2022-01-31T19:25:58.391 INFO:root:teuthology version: 1.1.0-7c0cb867 2022-01-31T19:25:58.393 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580 --verbose -- /home/teuthworker/archive/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580/orig.config.yaml 2022-01-31T19:25:58.453 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-01-31T19:25:58.528 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580 branch: master description: fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '6651580' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: master ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: i54081 sha1: e7bdcb25f366a81de92a121020949aeab8ce71c7 owner: scheduled_pdonnell@teuthology priority: 50 repo: https://github.com/ceph/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 sleep_before_teardown: 0 suite: fs:upgrade suite_branch: i54081 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: e7bdcb25f366a81de92a121020949aeab8ce71c7 targets: smithi167.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGxhyWoJedZHqDqtnUmz/IdrAavfF3CIWav1xZoVKtllgSXKD7LiWIEGa8XgM+SAHxBWgnO7GfhtzyN1BiKxdUAy2iO0L2IpEemOfMtUaobXSkHX5G+8vxXkFD55YXBBSnSVq8lbXzn7ZcGuKzsq2itMTBLPKCIVDp2/RPqBHrLc208uMBrLIPQPp/qLvKOyohQ/l+47WUKvZh1Koq/6HWV4vjsvsDgeVs2HDktak5js4hZC/1xRmxB/FChOxPC094+ZxfqfzEUWHsQzNIxSGVsDsl/a4jKwjTOJ5aUhsPHC07kpTlULcBN/JOiVI1ep9Q8crBOfclnAhkT20oPt1uATy55s5G1nzbNZPKxgv4b32K3QlDUmXcavWxBYIixRjwkdhA/Wx90C8pEOcR/0Q2ZAUQqohKTPyUirkMkNj7AtxXnCaKYyV7JWC92E2oH/5ZJEfN4tQNqt3KLtLYM/NkK4SyAflkX/hqPhMsVLAZa6gEybDghww+SlYMWk5HOBE= smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD57cD9bu+UfnykiF6UchwxeRan/zgt4DUjOtoarQYq9h1pOAv9lebTqcJsanMibe03vihUosTu60TwhuGWN5IIJGv++rAcEotoYSAFVjrq6JA9HbeFzmAV4SY/IxhUjb83ECzBwY4VaD1AsFlo8w/Q2AUKnWKCfaMgsqiWGPGQyz5Lnax8GSQ4sOXA+FSOla8NTEewMeUVMvIwD3YinjFP7nbHQrFCT+uh2f58tZcs5SvaBAN2t3PJu+OuwlDmc4GqLQO/uya24lXVdQPsAKvuyS31FH7EsRhyY5l0A3htCtVKERPzyyaPvbjVb3J61P/yK6vIKX06YV34Fj5gEJ+KBib8jMjqmAL1Lmx7NeTcPubzCCRF4BVMeYCi08D1oxcnNVfwi5L7fEB7af/PjqtM+6nADhY2fJU4k/U98CuGaDbLJR3ezFrHOYTqMNLb6Og7BmuwqkPCDP3DHBpVYjQqFyCnUku58PV2KhtjZqKcczJ6DPP/XMgnrCHW7Hy343c= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - ceph --format=json mds versions | jq -e ". | add == 4" - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology_branch: master teuthology_sha1: 7c0cb8672986d9dbe53078a123af65593653ef7a timestamp: 2022-01-31_19:13:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2022-01-31T19:25:58.529 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa; will attempt to use it 2022-01-31T19:25:58.530 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa/tasks 2022-01-31T19:25:58.530 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-01-31T19:25:58.532 INFO:teuthology.task.internal:Checking packages... 2022-01-31T19:25:58.552 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '29e1fc1722aa5915b44828a5ad02ec45ce760aa3' 2022-01-31T19:25:58.553 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:25:58.554 INFO:teuthology.packaging:ref: None 2022-01-31T19:25:58.554 INFO:teuthology.packaging:tag: None 2022-01-31T19:25:58.554 INFO:teuthology.packaging:branch: master 2022-01-31T19:25:58.554 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:25:58.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=master 2022-01-31T19:25:58.842 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10469.g29e1fc17 2022-01-31T19:25:58.843 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-01-31T19:25:58.854 INFO:teuthology.task.internal:no buildpackages task found 2022-01-31T19:25:58.854 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-01-31T19:25:58.867 INFO:teuthology.task.internal:Saving configuration 2022-01-31T19:25:58.893 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-01-31T19:25:58.904 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-31T19:25:58.929 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-01-31 19:20:35.309076', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIxou4kukrgjGvInCG0hBt+ls8WpYFezcp5guKkBVXTsgBHGGtS/3DYdlcABSgNVi5KupBkopNix6AzHIEjdmKc='} 2022-01-31T19:25:58.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-01-31 19:20:35.307723', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHsDKyAd0ujntFKyLBNqhclmfJGUdUnUHwV0Zu5jNfBd78jnmSylL5b702gjtJFSQrvqXvKOjd806efK4bOpDVw='} 2022-01-31T19:25:58.952 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-01-31T19:25:58.962 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-01-31T19:25:58.963 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-01-31T19:25:58.963 INFO:teuthology.run_tasks:Running task console_log... 2022-01-31T19:25:59.037 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee0d885d08>, signals=[15]) 2022-01-31T19:25:59.038 INFO:teuthology.run_tasks:Running task internal.connect... 2022-01-31T19:25:59.048 INFO:teuthology.task.internal:Opening connections... 2022-01-31T19:25:59.049 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:25:59.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:25:59.124 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2022-01-31T19:25:59.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:25:59.195 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-01-31T19:25:59.207 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2022-01-31T19:25:59.231 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2022-01-31T19:25:59.231 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2022-01-31T19:25:59.289 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2022-01-31T19:25:59.289 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="8" 2022-01-31T19:25:59.290 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2022-01-31T19:25:59.290 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2022-01-31T19:25:59.290 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="8" 2022-01-31T19:25:59.290 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el8" 2022-01-31T19:25:59.291 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-31T19:25:59.291 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2022-01-31T19:25:59.291 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-31T19:25:59.291 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2022-01-31T19:25:59.292 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-31T19:25:59.292 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-31T19:25:59.292 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-31T19:25:59.293 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2022-01-31T19:25:59.315 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2022-01-31T19:25:59.335 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2022-01-31T19:25:59.335 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2022-01-31T19:25:59.394 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2022-01-31T19:25:59.394 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8" 2022-01-31T19:25:59.395 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2022-01-31T19:25:59.395 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2022-01-31T19:25:59.395 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8" 2022-01-31T19:25:59.395 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8" 2022-01-31T19:25:59.396 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-31T19:25:59.396 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2022-01-31T19:25:59.396 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-31T19:25:59.396 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2022-01-31T19:25:59.396 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-31T19:25:59.397 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-31T19:25:59.397 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-31T19:25:59.397 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2022-01-31T19:25:59.415 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-01-31T19:25:59.431 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-01-31T19:25:59.442 INFO:teuthology.task.internal:Checking for old test directory... 2022-01-31T19:25:59.443 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2022-01-31T19:25:59.446 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2022-01-31T19:25:59.463 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-01-31T19:25:59.475 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-01-31T19:25:59.476 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2022-01-31T19:25:59.504 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2022-01-31T19:25:59.534 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-01-31T19:25:59.593 INFO:teuthology.run_tasks:Running task kernel... 2022-01-31T19:25:59.613 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-01-31T19:25:59.614 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-01-31T19:25:59.614 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2022-01-31T19:25:59.615 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2022-01-31T19:25:59.636 INFO:teuthology.orchestra.run.smithi167.stdout:4.18.0-348.2.1.el8_5.x86_64 2022-01-31T19:25:59.637 INFO:teuthology.task.kernel:Running kernel on smithi167: 4.18.0-348.2.1.el8_5.x86_64 2022-01-31T19:25:59.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2022-01-31T19:25:59.639 INFO:teuthology.orchestra.run.smithi171.stdout:4.18.0-348.2.1.el8_5.x86_64 2022-01-31T19:25:59.639 INFO:teuthology.task.kernel:Running kernel on smithi171: 4.18.0-348.2.1.el8_5.x86_64 2022-01-31T19:25:59.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum install -y kernel 2022-01-31T19:26:00.654 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:26:00.656 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:26:00.687 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:26:00.688 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:26:02.570 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - AppStream 30 MB/s | 19 MB 00:00 2022-01-31T19:26:02.662 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - AppStream 29 MB/s | 19 MB 00:00 2022-01-31T19:26:06.470 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - Base 36 MB/s | 18 MB 00:00 2022-01-31T19:26:06.562 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - Base 35 MB/s | 18 MB 00:00 2022-01-31T19:26:09.253 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - Extras 134 kB/s | 16 kB 00:00 2022-01-31T19:26:09.261 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - Extras 127 kB/s | 16 kB 00:00 2022-01-31T19:26:09.613 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - PowerTools 16 MB/s | 4.0 MB 00:00 2022-01-31T19:26:09.672 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - PowerTools 14 MB/s | 4.0 MB 00:00 2022-01-31T19:26:13.837 INFO:teuthology.orchestra.run.smithi171.stdout:Copr repo for python3-asyncssh owned by ceph 1.0 kB/s | 3.5 kB 00:03 2022-01-31T19:26:13.854 INFO:teuthology.orchestra.run.smithi167.stdout:Copr repo for python3-asyncssh owned by ceph 1.0 kB/s | 3.5 kB 00:03 2022-01-31T19:26:15.005 INFO:teuthology.orchestra.run.smithi167.stdout:Apache Arrow 180 kB/s | 184 kB 00:01 2022-01-31T19:26:15.013 INFO:teuthology.orchestra.run.smithi171.stdout:Apache Arrow 176 kB/s | 184 kB 00:01 2022-01-31T19:26:16.121 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2022-01-31T19:26:16.172 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 11 MB/s | 11 MB 00:01 2022-01-31T19:26:18.755 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 391 kB/s | 24 kB 00:00 2022-01-31T19:26:18.948 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 315 kB/s | 24 kB 00:00 2022-01-31T19:26:21.227 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-4.18.0-348.2.1.el8_5.x86_64 is already installed. 2022-01-31T19:26:21.227 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2022-01-31T19:26:21.321 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:26:21.321 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:26:21.322 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2022-01-31T19:26:21.322 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:26:21.322 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:26:21.322 INFO:teuthology.orchestra.run.smithi171.stdout: kernel x86_64 4.18.0-358.el8 CentOS-Base 7.5 M 2022-01-31T19:26:21.323 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:26:21.323 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core x86_64 4.18.0-358.el8 CentOS-Base 38 M 2022-01-31T19:26:21.323 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules x86_64 4.18.0-358.el8 CentOS-Base 31 M 2022-01-31T19:26:21.323 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:26:21.324 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:26:21.324 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:26:21.324 INFO:teuthology.orchestra.run.smithi171.stdout:Install 3 Packages 2022-01-31T19:26:21.324 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:26:21.325 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 77 M 2022-01-31T19:26:21.325 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 91 M 2022-01-31T19:26:21.325 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:26:21.494 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-348.2.1.el8_5.x86_64 is already installed. 2022-01-31T19:26:21.495 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2022-01-31T19:26:21.612 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:26:21.613 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:26:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2022-01-31T19:26:21.613 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:26:21.613 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:26:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 4.18.0-358.el8 CentOS-Base 7.5 M 2022-01-31T19:26:21.614 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:26:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 4.18.0-358.el8 CentOS-Base 38 M 2022-01-31T19:26:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 4.18.0-358.el8 CentOS-Base 31 M 2022-01-31T19:26:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:26:21.615 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:26:21.616 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:26:21.616 INFO:teuthology.orchestra.run.smithi167.stdout:Install 3 Packages 2022-01-31T19:26:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:26:21.617 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 77 M 2022-01-31T19:26:21.617 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 91 M 2022-01-31T19:26:21.618 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:26:21.890 INFO:teuthology.orchestra.run.smithi171.stdout:(1/3): kernel-4.18.0-358.el8.x86_64.rpm 13 MB/s | 7.5 MB 00:00 2022-01-31T19:26:22.299 INFO:teuthology.orchestra.run.smithi167.stdout:(1/3): kernel-4.18.0-358.el8.x86_64.rpm 11 MB/s | 7.5 MB 00:00 2022-01-31T19:26:22.498 INFO:teuthology.orchestra.run.smithi171.stdout:(2/3): kernel-core-4.18.0-358.el8.x86_64.rpm 33 MB/s | 38 MB 00:01 2022-01-31T19:26:22.907 INFO:teuthology.orchestra.run.smithi171.stdout:(3/3): kernel-modules-4.18.0-358.el8.x86_64.rpm 19 MB/s | 31 MB 00:01 2022-01-31T19:26:22.910 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:26:22.910 INFO:teuthology.orchestra.run.smithi171.stdout:Total 48 MB/s | 77 MB 00:01 2022-01-31T19:26:23.024 INFO:teuthology.orchestra.run.smithi167.stdout:(2/3): kernel-modules-4.18.0-358.el8.x86_64.rpm 22 MB/s | 31 MB 00:01 2022-01-31T19:26:23.399 INFO:teuthology.orchestra.run.smithi167.stdout:(3/3): kernel-core-4.18.0-358.el8.x86_64.rpm 22 MB/s | 38 MB 00:01 2022-01-31T19:26:23.402 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:26:23.403 INFO:teuthology.orchestra.run.smithi167.stdout:Total 43 MB/s | 77 MB 00:01 2022-01-31T19:26:23.489 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:26:23.852 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:26:23.852 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:26:23.982 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:26:24.352 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:26:24.352 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:26:25.122 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:26:25.137 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:26:25.639 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:26:25.655 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:26:27.606 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:26:28.063 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-core-4.18.0-358.el8.x86_64 1/3 2022-01-31T19:26:28.147 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:26:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-4.18.0-358.el8.x86_64 1/3 2022-01-31T19:26:31.817 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-4.18.0-358.el8.x86_64 1/3 2022-01-31T19:26:32.137 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-4.18.0-358.el8.x86_64 2/3 2022-01-31T19:26:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-4.18.0-358.el8.x86_64 1/3 2022-01-31T19:26:32.802 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-4.18.0-358.el8.x86_64 2/3 2022-01-31T19:26:35.241 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-4.18.0-358.el8.x86_64 2/3 2022-01-31T19:26:35.467 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:35.918 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-4.18.0-358.el8.x86_64 2/3 2022-01-31T19:26:36.190 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:56.798 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:57.128 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:57.128 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-4.18.0-358.el8.x86_64 1/3 2022-01-31T19:26:57.128 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-core-4.18.0-358.el8.x86_64 2/3 2022-01-31T19:26:57.525 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:57.586 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:57.586 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:26:57.586 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:26:57.587 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-4.18.0-358.el8.x86_64 kernel-core-4.18.0-358.el8.x86_64 2022-01-31T19:26:57.587 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-4.18.0-358.el8.x86_64 2022-01-31T19:26:57.587 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:26:57.587 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:26:57.707 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel --last | head -n 1 2022-01-31T19:26:57.873 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-4.18.0-358.el8.x86_64 Mon 31 Jan 2022 07:26:35 PM UTC 2022-01-31T19:26:57.874 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-358.el8.x86_64 2022-01-31T19:26:57.874 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-348.2.1.el8_5.x86_64 Expected: 4.18.0-358.el8.x86_64 2022-01-31T19:26:57.875 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-01-31T19:26:57.875 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-01-31T19:26:57.875 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-01-31T19:26:57.875 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2022-01-31T19:26:57.876 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel --last | head -n 1 2022-01-31T19:26:57.882 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:57.882 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-4.18.0-358.el8.x86_64 1/3 2022-01-31T19:26:57.883 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-4.18.0-358.el8.x86_64 2/3 2022-01-31T19:26:58.028 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-4.18.0-358.el8.x86_64 Mon 31 Jan 2022 07:26:35 PM UTC 2022-01-31T19:26:58.030 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-358.el8.x86_64 2022-01-31T19:26:58.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo rpm -qi grub2-tools 2022-01-31T19:26:58.063 INFO:teuthology.orchestra.run.smithi171.stdout:Name : grub2-tools 2022-01-31T19:26:58.064 INFO:teuthology.orchestra.run.smithi171.stdout:Epoch : 1 2022-01-31T19:26:58.064 INFO:teuthology.orchestra.run.smithi171.stdout:Version : 2.02 2022-01-31T19:26:58.064 INFO:teuthology.orchestra.run.smithi171.stdout:Release : 106.el8 2022-01-31T19:26:58.064 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: x86_64 2022-01-31T19:26:58.065 INFO:teuthology.orchestra.run.smithi171.stdout:Install Date: Thu 02 Dec 2021 10:09:41 PM UTC 2022-01-31T19:26:58.065 INFO:teuthology.orchestra.run.smithi171.stdout:Group : System Environment/Base 2022-01-31T19:26:58.065 INFO:teuthology.orchestra.run.smithi171.stdout:Size : 9265339 2022-01-31T19:26:58.065 INFO:teuthology.orchestra.run.smithi171.stdout:License : GPLv3+ 2022-01-31T19:26:58.066 INFO:teuthology.orchestra.run.smithi171.stdout:Signature : RSA/SHA256, Mon 08 Nov 2021 06:49:52 AM UTC, Key ID 05b555b38483c65d 2022-01-31T19:26:58.066 INFO:teuthology.orchestra.run.smithi171.stdout:Source RPM : grub2-2.02-106.el8.src.rpm 2022-01-31T19:26:58.066 INFO:teuthology.orchestra.run.smithi171.stdout:Build Date : Mon 08 Nov 2021 06:39:57 AM UTC 2022-01-31T19:26:58.066 INFO:teuthology.orchestra.run.smithi171.stdout:Build Host : kbuilder.bsys.centos.org 2022-01-31T19:26:58.066 INFO:teuthology.orchestra.run.smithi171.stdout:Relocations : (not relocatable) 2022-01-31T19:26:58.067 INFO:teuthology.orchestra.run.smithi171.stdout:Packager : CentOS BuildSystem 2022-01-31T19:26:58.067 INFO:teuthology.orchestra.run.smithi171.stdout:Vendor : CentOS 2022-01-31T19:26:58.067 INFO:teuthology.orchestra.run.smithi171.stdout:URL : http://www.gnu.org/software/grub/ 2022-01-31T19:26:58.067 INFO:teuthology.orchestra.run.smithi171.stdout:Summary : Support tools for GRUB. 2022-01-31T19:26:58.068 INFO:teuthology.orchestra.run.smithi171.stdout:Description : 2022-01-31T19:26:58.068 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:26:58.068 INFO:teuthology.orchestra.run.smithi171.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-01-31T19:26:58.068 INFO:teuthology.orchestra.run.smithi171.stdout:customizable bootloader with modular architecture. It supports a rich 2022-01-31T19:26:58.069 INFO:teuthology.orchestra.run.smithi171.stdout:variety of kernel formats, file systems, computer architectures and 2022-01-31T19:26:58.069 INFO:teuthology.orchestra.run.smithi171.stdout:hardware devices. 2022-01-31T19:26:58.069 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:26:58.069 INFO:teuthology.orchestra.run.smithi171.stdout:This subpackage provides tools for support of all platforms. 2022-01-31T19:26:58.070 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-01-31T19:26:58.071 INFO:teuthology.task.kernel:Updating grub on smithi171 to boot 4.18.0-358.el8.x86_64 2022-01-31T19:26:58.071 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-01-31T19:26:58.399 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-4.18.0-358.el8.x86_64 3/3 2022-01-31T19:26:58.400 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:26:58.400 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:26:58.400 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-4.18.0-358.el8.x86_64 kernel-core-4.18.0-358.el8.x86_64 2022-01-31T19:26:58.400 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-4.18.0-358.el8.x86_64 2022-01-31T19:26:58.401 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:26:58.401 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:26:58.530 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel --last | head -n 1 2022-01-31T19:26:58.698 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-4.18.0-358.el8.x86_64 Mon 31 Jan 2022 07:26:35 PM UTC 2022-01-31T19:26:58.699 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-358.el8.x86_64 2022-01-31T19:26:58.699 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-348.2.1.el8_5.x86_64 Expected: 4.18.0-358.el8.x86_64 2022-01-31T19:26:58.700 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-01-31T19:26:58.700 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-01-31T19:26:58.700 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-01-31T19:26:58.701 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2022-01-31T19:26:58.701 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel --last | head -n 1 2022-01-31T19:26:58.853 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-4.18.0-358.el8.x86_64 Mon 31 Jan 2022 07:26:35 PM UTC 2022-01-31T19:26:58.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-358.el8.x86_64 2022-01-31T19:26:58.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2022-01-31T19:26:58.888 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2022-01-31T19:26:58.888 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2022-01-31T19:26:58.888 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.02 2022-01-31T19:26:58.889 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 106.el8 2022-01-31T19:26:58.889 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2022-01-31T19:26:58.889 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Thu 02 Dec 2021 10:09:41 PM UTC 2022-01-31T19:26:58.889 INFO:teuthology.orchestra.run.smithi167.stdout:Group : System Environment/Base 2022-01-31T19:26:58.890 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 9265339 2022-01-31T19:26:58.890 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2022-01-31T19:26:58.890 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 08 Nov 2021 06:49:52 AM UTC, Key ID 05b555b38483c65d 2022-01-31T19:26:58.890 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.02-106.el8.src.rpm 2022-01-31T19:26:58.891 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Mon 08 Nov 2021 06:39:57 AM UTC 2022-01-31T19:26:58.891 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : kbuilder.bsys.centos.org 2022-01-31T19:26:58.891 INFO:teuthology.orchestra.run.smithi167.stdout:Relocations : (not relocatable) 2022-01-31T19:26:58.891 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : CentOS BuildSystem 2022-01-31T19:26:58.891 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2022-01-31T19:26:58.892 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2022-01-31T19:26:58.892 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2022-01-31T19:26:58.893 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2022-01-31T19:26:58.893 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:26:58.893 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-01-31T19:26:58.893 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2022-01-31T19:26:58.894 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2022-01-31T19:26:58.894 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2022-01-31T19:26:58.894 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:26:58.894 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2022-01-31T19:26:58.896 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-01-31T19:26:58.896 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 4.18.0-358.el8.x86_64 2022-01-31T19:26:58.897 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-01-31T19:26:59.753 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2022-01-31T19:27:00.767 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2022-01-31T19:27:01.560 INFO:teuthology.orchestra.run.smithi171.stderr:done 2022-01-31T19:27:01.561 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2022-01-31T19:27:01.593 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.FyhJ9NJLI3 2022-01-31T19:27:01.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FyhJ9NJLI3 2022-01-31T19:27:01.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.FyhJ9NJLI3 2022-01-31T19:27:01.828 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.FyhJ9NJLI3 is 6KB 2022-01-31T19:27:01.877 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.FyhJ9NJLI3 2022-01-31T19:27:01.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo /bin/ls /boot/loader/entries || true 2022-01-31T19:27:01.956 INFO:teuthology.orchestra.run.smithi171.stdout:54c8b3f95a884696922d0d687e6bf68d-0-rescue.conf 2022-01-31T19:27:01.956 INFO:teuthology.orchestra.run.smithi171.stdout:54c8b3f95a884696922d0d687e6bf68d-4.18.0-240.el8.x86_64.conf 2022-01-31T19:27:01.956 INFO:teuthology.orchestra.run.smithi171.stdout:54c8b3f95a884696922d0d687e6bf68d-4.18.0-348.2.1.el8_5.x86_64.conf 2022-01-31T19:27:01.957 INFO:teuthology.orchestra.run.smithi171.stdout:54c8b3f95a884696922d0d687e6bf68d-4.18.0-358.el8.x86_64.conf 2022-01-31T19:27:01.958 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-set-default 54c8b3f95a884696922d0d687e6bf68d-4.18.0-358.el8.x86_64 2022-01-31T19:27:02.108 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2022-01-31T19:27:02.687 INFO:teuthology.orchestra.run.smithi167.stderr:done 2022-01-31T19:27:02.688 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2022-01-31T19:27:02.715 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.tEIYItavyq 2022-01-31T19:27:02.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tEIYItavyq 2022-01-31T19:27:02.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.tEIYItavyq 2022-01-31T19:27:02.947 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.tEIYItavyq is 6KB 2022-01-31T19:27:02.995 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.tEIYItavyq 2022-01-31T19:27:03.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2022-01-31T19:27:03.091 INFO:teuthology.orchestra.run.smithi167.stdout:54c8b3f95a884696922d0d687e6bf68d-0-rescue.conf 2022-01-31T19:27:03.091 INFO:teuthology.orchestra.run.smithi167.stdout:54c8b3f95a884696922d0d687e6bf68d-4.18.0-240.el8.x86_64.conf 2022-01-31T19:27:03.091 INFO:teuthology.orchestra.run.smithi167.stdout:54c8b3f95a884696922d0d687e6bf68d-4.18.0-348.2.1.el8_5.x86_64.conf 2022-01-31T19:27:03.091 INFO:teuthology.orchestra.run.smithi167.stdout:54c8b3f95a884696922d0d687e6bf68d-4.18.0-358.el8.x86_64.conf 2022-01-31T19:27:03.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 54c8b3f95a884696922d0d687e6bf68d-4.18.0-358.el8.x86_64 2022-01-31T19:27:03.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2022-01-31T19:27:32.139 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-01-31T19:27:32.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-01-31T19:27:32.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:27:33.262 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-01-31T19:27:33.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2022-01-31T19:27:33.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:27:50.475 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2022-01-31T19:27:51.755 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2022-01-31T19:28:20.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-01-31T19:28:20.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:28:21.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2022-01-31T19:28:21.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:28:52.493 DEBUG:teuthology.orchestra.run.smithi171:> true 2022-01-31T19:28:52.821 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-01-31T19:28:52.821 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-358.el8.x86_64"... 2022-01-31T19:28:52.822 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2022-01-31T19:28:52.839 INFO:teuthology.orchestra.run.smithi171.stdout:4.18.0-358.el8.x86_64 2022-01-31T19:28:52.839 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-358.el8.x86_64 vs 4.18.0-358.el8.x86_64 2022-01-31T19:28:52.839 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-01-31T19:28:52.840 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2022-01-31T19:28:53.784 DEBUG:teuthology.orchestra.run.smithi167:> true 2022-01-31T19:28:53.841 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-01-31T19:28:53.842 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-01-31T19:28:53.910 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2022-01-31T19:28:53.935 DEBUG:teuthology.parallel:result is None 2022-01-31T19:28:54.077 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2022-01-31T19:28:54.078 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-358.el8.x86_64"... 2022-01-31T19:28:54.078 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2022-01-31T19:28:54.094 INFO:teuthology.orchestra.run.smithi167.stdout:4.18.0-358.el8.x86_64 2022-01-31T19:28:54.094 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-358.el8.x86_64 vs 4.18.0-358.el8.x86_64 2022-01-31T19:28:54.095 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-01-31T19:28:54.095 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2022-01-31T19:28:55.097 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-01-31T19:28:55.097 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-01-31T19:28:55.199 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2022-01-31T19:28:55.225 DEBUG:teuthology.parallel:result is None 2022-01-31T19:28:55.226 INFO:teuthology.run_tasks:Running task internal.base... 2022-01-31T19:28:55.240 INFO:teuthology.task.internal:Creating test directory... 2022-01-31T19:28:55.241 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-01-31T19:28:55.243 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-01-31T19:28:55.260 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-01-31T19:28:55.271 INFO:teuthology.run_tasks:Running task internal.archive... 2022-01-31T19:28:55.287 INFO:teuthology.task.internal:Creating archive directory... 2022-01-31T19:28:55.287 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-01-31T19:28:55.303 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-01-31T19:28:55.328 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-01-31T19:28:55.339 INFO:teuthology.task.internal:Enabling coredump saving... 2022-01-31T19:28:55.339 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-01-31T19:28:55.369 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-01-31T19:28:55.414 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-31T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-31T19:28:55.425 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-31T19:28:55.427 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-31T19:28:55.429 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-01-31T19:28:55.440 INFO:teuthology.task.internal:Configuring sudo... 2022-01-31T19:28:55.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-01-31T19:28:55.443 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-01-31T19:28:55.468 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-01-31T19:28:55.482 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-01-31T19:28:55.482 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-01-31T19:28:55.512 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-01-31T19:28:55.527 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-31T19:28:55.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-31T19:28:55.659 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-31T19:28:55.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-31T19:28:55.788 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:28:55.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-01-31T19:28:55.858 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-31T19:28:55.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-31T19:28:55.949 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-31T19:28:56.028 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-31T19:28:56.073 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:28:56.074 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-01-31T19:28:56.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2022-01-31T19:28:56.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2022-01-31T19:28:56.201 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-01-31T19:28:56.259 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-01-31T19:28:56.484 INFO:teuthology.run_tasks:Running task internal.timer... 2022-01-31T19:28:56.495 INFO:teuthology.task.internal:Starting timer... 2022-01-31T19:28:56.496 INFO:teuthology.run_tasks:Running task pcp... 2022-01-31T19:28:56.513 INFO:teuthology.run_tasks:Running task selinux... 2022-01-31T19:28:56.568 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-01-31T19:28:56.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2022-01-31T19:28:56.635 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: [ OK ] 2022-01-31T19:28:56.636 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2022-01-31T19:28:56.704 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: [ OK ] 2022-01-31T19:28:56.706 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-01-31T19:28:56.706 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2022-01-31T19:28:56.727 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2022-01-31T19:28:56.728 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2022-01-31T19:28:56.779 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2022-01-31T19:28:56.779 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi167.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2022-01-31T19:28:56.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-01-31T19:28:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-31T19:28:56.808 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-01-31T19:28:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-31T19:28:56.835 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-01-31T19:28:56.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2022-01-31T19:28:56.882 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2022-01-31T19:28:56.914 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-01-31T19:28:56.933 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-01-31T19:28:56.941 INFO:teuthology.repo_utils:Fetching master from origin 2022-01-31T19:28:56.979 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-01-31T19:33:47.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2022-01-31T19:33:47.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:33:47.607 DEBUG:teuthology.orchestra.run.smithi167:> true 2022-01-31T19:33:47.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-01-31T19:33:47.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-31T19:33:47.761 DEBUG:teuthology.orchestra.run.smithi171:> true 2022-01-31T19:33:47.835 INFO:teuthology.run_tasks:Running task clock... 2022-01-31T19:33:47.849 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-01-31T19:33:47.850 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-01-31T19:33:47.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-01-31T19:33:47.854 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-01-31T19:33:47.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-01-31T19:33:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-01-31T19:33:47.896 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-01-31T19:33:47.907 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-01-31T19:33:47.922 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-01-31T19:33:47.970 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2022-01-31T19:33:47.982 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2022-01-31T19:33:47.994 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2022-01-31T19:33:47.997 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-01-31T19:33:48.006 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2022-01-31T19:33:48.010 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-01-31T19:33:48.020 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-01-31T19:33:48.034 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-01-31T19:33:48.083 INFO:teuthology.orchestra.run.smithi167.stderr:bash: ntpq: command not found 2022-01-31T19:33:48.087 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-01-31T19:33:48.087 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2022-01-31T19:33:48.087 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.087 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.088 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.088 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.106 INFO:teuthology.orchestra.run.smithi171.stderr:bash: ntpq: command not found 2022-01-31T19:33:48.111 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-01-31T19:33:48.111 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2022-01-31T19:33:48.112 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.112 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.112 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.112 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-31T19:33:48.113 INFO:teuthology.run_tasks:Running task pexec... 2022-01-31T19:33:48.125 INFO:teuthology.task.pexec:Executing custom commands... 2022-01-31T19:33:48.126 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:33:48.126 DEBUG:teuthology.orchestra.run.smithi167:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-01-31T19:33:48.127 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi171.front.sepia.ceph.com 2022-01-31T19:33:48.127 DEBUG:teuthology.orchestra.run.smithi171:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-01-31T19:33:48.345 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:48.346 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:48.371 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:48.371 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:48.717 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:11 ago on Mon 31 Jan 2022 07:32:37 PM UTC. 2022-01-31T19:33:48.744 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:11 ago on Mon 31 Jan 2022 07:32:37 PM UTC. 2022-01-31T19:33:49.733 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:33:49.734 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:33:49.734 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2022-01-31T19:33:49.735 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:33:49.735 INFO:teuthology.orchestra.run.smithi167.stdout:Resetting modules: 2022-01-31T19:33:49.735 INFO:teuthology.orchestra.run.smithi167.stdout: container-tools 2022-01-31T19:33:49.735 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:49.736 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:33:49.736 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:33:49.736 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:49.748 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:33:49.749 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:33:49.750 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2022-01-31T19:33:49.750 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:33:49.750 INFO:teuthology.orchestra.run.smithi171.stdout:Resetting modules: 2022-01-31T19:33:49.751 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools 2022-01-31T19:33:49.751 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:49.751 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:33:49.751 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:33:49.751 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:49.955 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:33:49.966 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:33:50.209 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:50.210 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:50.242 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:50.242 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:50.578 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:13 ago on Mon 31 Jan 2022 07:32:37 PM UTC. 2022-01-31T19:33:50.614 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:13 ago on Mon 31 Jan 2022 07:32:37 PM UTC. 2022-01-31T19:33:51.537 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:33:51.538 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2022-01-31T19:33:51.538 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:33:51.539 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2022-01-31T19:33:51.539 INFO:teuthology.orchestra.run.smithi171.stdout:Installing group/module packages: 2022-01-31T19:33:51.539 INFO:teuthology.orchestra.run.smithi171.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2022-01-31T19:33:51.539 INFO:teuthology.orchestra.run.smithi171.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2022-01-31T19:33:51.540 INFO:teuthology.orchestra.run.smithi171.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2022-01-31T19:33:51.540 INFO:teuthology.orchestra.run.smithi171.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2022-01-31T19:33:51.540 INFO:teuthology.orchestra.run.smithi171.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2022-01-31T19:33:51.541 INFO:teuthology.orchestra.run.smithi171.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2022-01-31T19:33:51.541 INFO:teuthology.orchestra.run.smithi171.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2022-01-31T19:33:51.541 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:33:51.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2022-01-31T19:33:51.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2022-01-31T19:33:51.542 INFO:teuthology.orchestra.run.smithi171.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2022-01-31T19:33:51.542 INFO:teuthology.orchestra.run.smithi171.stdout:Installing module profiles: 2022-01-31T19:33:51.542 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools/common 2022-01-31T19:33:51.543 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:51.543 INFO:teuthology.orchestra.run.smithi171.stdout:Enabling module streams: 2022-01-31T19:33:51.543 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools rhel8 2022-01-31T19:33:51.543 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:51.543 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:33:51.544 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2022-01-31T19:33:51.544 INFO:teuthology.orchestra.run.smithi171.stdout:Install 10 Packages 2022-01-31T19:33:51.544 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:51.545 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 18 M 2022-01-31T19:33:51.546 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 64 M 2022-01-31T19:33:51.546 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:33:51.632 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:33:51.634 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2022-01-31T19:33:51.634 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:33:51.634 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2022-01-31T19:33:51.634 INFO:teuthology.orchestra.run.smithi167.stdout:Installing group/module packages: 2022-01-31T19:33:51.635 INFO:teuthology.orchestra.run.smithi167.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2022-01-31T19:33:51.635 INFO:teuthology.orchestra.run.smithi167.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2022-01-31T19:33:51.635 INFO:teuthology.orchestra.run.smithi167.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2022-01-31T19:33:51.635 INFO:teuthology.orchestra.run.smithi167.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2022-01-31T19:33:51.636 INFO:teuthology.orchestra.run.smithi167.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2022-01-31T19:33:51.636 INFO:teuthology.orchestra.run.smithi167.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2022-01-31T19:33:51.636 INFO:teuthology.orchestra.run.smithi167.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2022-01-31T19:33:51.636 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:33:51.636 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2022-01-31T19:33:51.637 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2022-01-31T19:33:51.637 INFO:teuthology.orchestra.run.smithi167.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2022-01-31T19:33:51.637 INFO:teuthology.orchestra.run.smithi167.stdout:Installing module profiles: 2022-01-31T19:33:51.637 INFO:teuthology.orchestra.run.smithi167.stdout: container-tools/common 2022-01-31T19:33:51.638 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:51.638 INFO:teuthology.orchestra.run.smithi167.stdout:Enabling module streams: 2022-01-31T19:33:51.638 INFO:teuthology.orchestra.run.smithi167.stdout: container-tools rhel8 2022-01-31T19:33:51.638 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:51.638 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:33:51.639 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2022-01-31T19:33:51.639 INFO:teuthology.orchestra.run.smithi167.stdout:Install 10 Packages 2022-01-31T19:33:51.639 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:51.640 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 18 M 2022-01-31T19:33:51.640 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 64 M 2022-01-31T19:33:51.641 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:33:51.729 INFO:teuthology.orchestra.run.smithi167.stdout:(1/10): cockpit-podman-35-1.module_el8.6.0+944+ 4.8 MB/s | 457 kB 00:00 2022-01-31T19:33:51.763 INFO:teuthology.orchestra.run.smithi167.stdout:(2/10): python3-podman-3.2.1-4.module_el8.6.0+9 4.4 MB/s | 148 kB 00:00 2022-01-31T19:33:51.789 INFO:teuthology.orchestra.run.smithi167.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 988 kB/s | 25 kB 00:00 2022-01-31T19:33:51.810 INFO:teuthology.orchestra.run.smithi171.stdout:(1/10): cockpit-podman-35-1.module_el8.6.0+944+ 1.7 MB/s | 457 kB 00:00 2022-01-31T19:33:51.813 INFO:teuthology.orchestra.run.smithi167.stdout:(4/10): python3-pyxdg-0.25-16.el8.noarch.rpm 3.7 MB/s | 94 kB 00:00 2022-01-31T19:33:51.844 INFO:teuthology.orchestra.run.smithi171.stdout:(2/10): python3-podman-3.2.1-4.module_el8.6.0+9 4.4 MB/s | 148 kB 00:00 2022-01-31T19:33:51.869 INFO:teuthology.orchestra.run.smithi171.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 999 kB/s | 25 kB 00:00 2022-01-31T19:33:51.903 INFO:teuthology.orchestra.run.smithi171.stdout:(4/10): crun-1.2-1.module_el8.6.0+954+963caf36. 536 kB/s | 193 kB 00:00 2022-01-31T19:33:51.936 INFO:teuthology.orchestra.run.smithi171.stdout:(5/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2022-01-31T19:33:51.981 INFO:teuthology.orchestra.run.smithi167.stdout:(5/10): buildah-1.23.1-2.module_el8.6.0+954+963 23 MB/s | 7.9 MB 00:00 2022-01-31T19:33:52.048 INFO:teuthology.orchestra.run.smithi167.stdout:(6/10): crun-1.2-1.module_el8.6.0+954+963caf36. 471 kB/s | 193 kB 00:00 2022-01-31T19:33:52.106 INFO:teuthology.orchestra.run.smithi167.stdout:(7/10): toolbox-0.0.99.3-1.module_el8.5.0+877+1 18 MB/s | 2.3 MB 00:00 2022-01-31T19:33:52.132 INFO:teuthology.orchestra.run.smithi167.stdout:(8/10): udica-0.2.6-2.module_el8.6.0+944+d413f9 581 kB/s | 48 kB 00:00 2022-01-31T19:33:52.182 INFO:teuthology.orchestra.run.smithi167.stdout:(9/10): yajl-2.1.0-10.el8.x86_64.rpm 539 kB/s | 41 kB 00:00 2022-01-31T19:33:52.295 INFO:teuthology.orchestra.run.smithi171.stdout:(6/10): buildah-1.23.1-2.module_el8.6.0+954+963 10 MB/s | 7.9 MB 00:00 2022-01-31T19:33:52.320 INFO:teuthology.orchestra.run.smithi171.stdout:(7/10): udica-0.2.6-2.module_el8.6.0+944+d413f9 1.9 MB/s | 48 kB 00:00 2022-01-31T19:33:52.323 INFO:teuthology.orchestra.run.smithi167.stdout:(10/10): skopeo-1.5.0-2.module_el8.6.0+954+963c 13 MB/s | 6.7 MB 00:00 2022-01-31T19:33:52.326 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:33:52.326 INFO:teuthology.orchestra.run.smithi167.stdout:Total 26 MB/s | 18 MB 00:00 2022-01-31T19:33:52.346 INFO:teuthology.orchestra.run.smithi171.stdout:(8/10): yajl-2.1.0-10.el8.x86_64.rpm 1.6 MB/s | 41 kB 00:00 2022-01-31T19:33:52.413 INFO:teuthology.orchestra.run.smithi171.stdout:(9/10): toolbox-0.0.99.3-1.module_el8.5.0+877+1 4.7 MB/s | 2.3 MB 00:00 2022-01-31T19:33:52.522 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:33:52.529 INFO:teuthology.orchestra.run.smithi171.stdout:(10/10): skopeo-1.5.0-2.module_el8.6.0+954+963c 11 MB/s | 6.7 MB 00:00 2022-01-31T19:33:52.531 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:33:52.532 INFO:teuthology.orchestra.run.smithi171.stdout:Total 18 MB/s | 18 MB 00:00 2022-01-31T19:33:52.540 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:33:52.540 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:33:52.728 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:33:52.731 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:33:52.734 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:33:52.746 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:33:52.746 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:33:52.933 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:33:52.936 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:33:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:33:53.424 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:33:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2022-01-31T19:33:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/10 2022-01-31T19:33:53.601 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2022-01-31T19:33:53.755 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/10 2022-01-31T19:33:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2022-01-31T19:33:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2022-01-31T19:33:54.017 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2022-01-31T19:33:54.155 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2022-01-31T19:33:54.199 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 5/10 2022-01-31T19:33:54.343 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 5/10 2022-01-31T19:33:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 6/10 2022-01-31T19:33:54.694 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 6/10 2022-01-31T19:33:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 7/10 2022-01-31T19:33:55.444 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 7/10 2022-01-31T19:33:55.526 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 8/10 2022-01-31T19:33:55.593 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 8/10 2022-01-31T19:33:56.328 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 9/10 2022-01-31T19:33:56.396 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 9/10 2022-01-31T19:33:56.511 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2022-01-31T19:33:56.574 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2022-01-31T19:33:56.811 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2022-01-31T19:33:56.811 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/10 2022-01-31T19:33:56.812 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/10 2022-01-31T19:33:56.812 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/10 2022-01-31T19:33:56.812 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2022-01-31T19:33:56.812 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/10 2022-01-31T19:33:56.813 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/10 2022-01-31T19:33:56.813 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/10 2022-01-31T19:33:56.813 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/10 2022-01-31T19:33:56.813 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/10 2022-01-31T19:33:56.885 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 10/10 2022-01-31T19:33:56.885 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/10 2022-01-31T19:33:56.885 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/10 2022-01-31T19:33:56.886 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/10 2022-01-31T19:33:56.886 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/10 2022-01-31T19:33:56.886 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/10 2022-01-31T19:33:56.886 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/10 2022-01-31T19:33:56.887 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/10 2022-01-31T19:33:56.887 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/10 2022-01-31T19:33:56.887 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/10 2022-01-31T19:33:57.352 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2022-01-31T19:33:57.353 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:57.353 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:33:57.354 INFO:teuthology.orchestra.run.smithi167.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2022-01-31T19:33:57.354 INFO:teuthology.orchestra.run.smithi167.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2022-01-31T19:33:57.354 INFO:teuthology.orchestra.run.smithi167.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2022-01-31T19:33:57.354 INFO:teuthology.orchestra.run.smithi167.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2022-01-31T19:33:57.354 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-01-31T19:33:57.355 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-01-31T19:33:57.355 INFO:teuthology.orchestra.run.smithi167.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2022-01-31T19:33:57.355 INFO:teuthology.orchestra.run.smithi167.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2022-01-31T19:33:57.355 INFO:teuthology.orchestra.run.smithi167.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2022-01-31T19:33:57.356 INFO:teuthology.orchestra.run.smithi167.stdout: yajl-2.1.0-10.el8.x86_64 2022-01-31T19:33:57.356 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:33:57.356 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:33:57.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2022-01-31T19:33:57.364 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:57.365 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:33:57.365 INFO:teuthology.orchestra.run.smithi171.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2022-01-31T19:33:57.365 INFO:teuthology.orchestra.run.smithi171.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2022-01-31T19:33:57.365 INFO:teuthology.orchestra.run.smithi171.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2022-01-31T19:33:57.366 INFO:teuthology.orchestra.run.smithi171.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2022-01-31T19:33:57.366 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-01-31T19:33:57.366 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-01-31T19:33:57.366 INFO:teuthology.orchestra.run.smithi171.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2022-01-31T19:33:57.367 INFO:teuthology.orchestra.run.smithi171.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2022-01-31T19:33:57.367 INFO:teuthology.orchestra.run.smithi171.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2022-01-31T19:33:57.367 INFO:teuthology.orchestra.run.smithi171.stdout: yajl-2.1.0-10.el8.x86_64 2022-01-31T19:33:57.367 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:33:57.367 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:33:57.559 DEBUG:teuthology.parallel:result is None 2022-01-31T19:33:57.560 DEBUG:teuthology.parallel:result is None 2022-01-31T19:33:57.560 INFO:teuthology.run_tasks:Running task install... 2022-01-31T19:33:57.577 DEBUG:teuthology.task.install:project ceph 2022-01-31T19:33:57.578 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '29e1fc1722aa5915b44828a5ad02ec45ce760aa3'}} 2022-01-31T19:33:57.579 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '29e1fc1722aa5915b44828a5ad02ec45ce760aa3'} 2022-01-31T19:33:57.579 INFO:teuthology.task.install:Using flavor: default 2022-01-31T19:33:57.592 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-01-31T19:33:57.592 INFO:teuthology.task.install:extra packages: [] 2022-01-31T19:33:57.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo lsb_release -is 2022-01-31T19:33:57.627 INFO:teuthology.orchestra.run.smithi167.stdout:CentOSStream 2022-01-31T19:33:57.629 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-01-31T19:33:57.629 DEBUG:teuthology.orchestra.run.smithi171:> sudo lsb_release -is 2022-01-31T19:33:57.630 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '29e1fc1722aa5915b44828a5ad02ec45ce760aa3', 'tag': None, 'wait_for_package': False} 2022-01-31T19:33:57.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo lsb_release -is 2022-01-31T19:33:57.666 INFO:teuthology.orchestra.run.smithi171.stdout:CentOSStream 2022-01-31T19:33:57.667 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-01-31T19:33:57.667 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '29e1fc1722aa5915b44828a5ad02ec45ce760aa3', 'tag': None, 'wait_for_package': False} 2022-01-31T19:33:57.668 DEBUG:teuthology.orchestra.run.smithi171:> sudo lsb_release -is 2022-01-31T19:33:57.703 INFO:teuthology.orchestra.run.smithi167.stdout:CentOSStream 2022-01-31T19:33:57.704 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-01-31T19:33:57.704 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:33:57.705 INFO:teuthology.packaging:ref: None 2022-01-31T19:33:57.705 INFO:teuthology.packaging:tag: None 2022-01-31T19:33:57.705 INFO:teuthology.packaging:branch: pacific 2022-01-31T19:33:57.706 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:33:57.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2022-01-31T19:33:57.744 INFO:teuthology.orchestra.run.smithi171.stdout:CentOSStream 2022-01-31T19:33:57.746 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-01-31T19:33:57.746 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:33:57.747 INFO:teuthology.packaging:ref: None 2022-01-31T19:33:57.747 INFO:teuthology.packaging:tag: None 2022-01-31T19:33:57.747 INFO:teuthology.packaging:branch: pacific 2022-01-31T19:33:57.747 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:33:57.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2022-01-31T19:33:58.027 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/ 2022-01-31T19:33:58.028 INFO:teuthology.task.install.rpm:Package version is 16.2.7-278.g9e931647 2022-01-31T19:33:58.049 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/ 2022-01-31T19:33:58.050 INFO:teuthology.task.install.rpm:Package version is 16.2.7-278.g9e931647 2022-01-31T19:33:58.197 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-01-31T19:33:58.198 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:33:58.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-01-31T19:33:58.220 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/9e9316478f690f97f4b84b62cb8253f68f19c8db/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-01-31T19:33:58.220 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:33:58.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-01-31T19:33:58.226 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-01-31T19:33:58.226 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:33:58.227 INFO:teuthology.packaging:ref: None 2022-01-31T19:33:58.227 INFO:teuthology.packaging:tag: None 2022-01-31T19:33:58.227 INFO:teuthology.packaging:branch: pacific 2022-01-31T19:33:58.228 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:33:58.228 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-01-31T19:33:58.249 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-01-31T19:33:58.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:33:58.250 INFO:teuthology.packaging:ref: None 2022-01-31T19:33:58.250 INFO:teuthology.packaging:tag: None 2022-01-31T19:33:58.250 INFO:teuthology.packaging:branch: pacific 2022-01-31T19:33:58.251 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:33:58.251 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-01-31T19:33:58.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-01-31T19:33:58.316 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-01-31T19:33:58.368 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-01-31T19:33:58.393 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-01-31T19:33:58.433 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2022-01-31T19:33:58.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2022-01-31T19:33:58.459 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2022-01-31T19:33:58.461 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2022-01-31T19:33:58.651 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.651 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.666 INFO:teuthology.orchestra.run.smithi167.stdout:76 files removed 2022-01-31T19:33:58.677 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.677 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw 2022-01-31T19:33:58.693 INFO:teuthology.orchestra.run.smithi171.stdout:76 files removed 2022-01-31T19:33:58.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw 2022-01-31T19:33:58.863 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.864 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.892 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:58.893 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:33:59.545 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 148 kB/s | 76 kB 00:00 2022-01-31T19:33:59.590 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 144 kB/s | 76 kB 00:00 2022-01-31T19:33:59.954 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 40 kB/s | 15 kB 00:00 2022-01-31T19:34:00.007 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 40 kB/s | 15 kB 00:00 2022-01-31T19:34:00.321 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2022-01-31T19:34:00.375 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2022-01-31T19:34:00.882 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - AppStream 36 MB/s | 19 MB 00:00 2022-01-31T19:34:00.993 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - AppStream 32 MB/s | 19 MB 00:00 2022-01-31T19:34:04.699 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - Base 36 MB/s | 18 MB 00:00 2022-01-31T19:34:04.835 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - Base 38 MB/s | 18 MB 00:00 2022-01-31T19:34:07.823 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - Extras 29 kB/s | 16 kB 00:00 2022-01-31T19:34:07.901 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - Extras 40 kB/s | 16 kB 00:00 2022-01-31T19:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-8 - PowerTools 14 MB/s | 4.0 MB 00:00 2022-01-31T19:34:10.262 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-8 - PowerTools 1.8 MB/s | 4.0 MB 00:02 2022-01-31T19:34:11.392 INFO:teuthology.orchestra.run.smithi167.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-01-31T19:34:12.509 INFO:teuthology.orchestra.run.smithi167.stdout:Apache Arrow 186 kB/s | 184 kB 00:00 2022-01-31T19:34:12.978 INFO:teuthology.orchestra.run.smithi171.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-01-31T19:34:13.628 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 11 MB/s | 11 MB 00:00 2022-01-31T19:34:14.054 INFO:teuthology.orchestra.run.smithi171.stdout:Apache Arrow 194 kB/s | 184 kB 00:00 2022-01-31T19:34:15.128 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2022-01-31T19:34:16.293 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 587 kB/s | 24 kB 00:00 2022-01-31T19:34:17.770 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 446 kB/s | 24 kB 00:00 2022-01-31T19:34:18.927 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:34:18.930 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:34:18.930 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:34:18.930 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:34:18.931 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:34:18.931 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:16.2.7-278.g9e931647.el8 ceph 12 M 2022-01-31T19:34:18.931 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2022-01-31T19:34:18.932 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:16.2.7-278.g9e931647.el8 ceph 3.7 M 2022-01-31T19:34:18.932 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:16.2.7-278.g9e931647.el8 ceph 4.2 M 2022-01-31T19:34:18.932 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:34:18.932 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:16.2.7-278.g9e931647.el8 ceph 6.0 M 2022-01-31T19:34:18.933 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:16.2.7-278.g9e931647.el8 ceph 24 M 2022-01-31T19:34:18.933 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:16.2.7-278.g9e931647.el8 ceph 26 k 2022-01-31T19:34:18.933 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-01-31T19:34:18.933 INFO:teuthology.orchestra.run.smithi167.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2022-01-31T19:34:18.933 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:16.2.7-278.g9e931647.el8 ceph 813 k 2022-01-31T19:34:18.934 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-01-31T19:34:18.934 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-01-31T19:34:18.934 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:16.2.7-278.g9e931647.el8 ceph 492 k 2022-01-31T19:34:18.934 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-01-31T19:34:18.935 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:16.2.7-278.g9e931647.el8 ceph 3.8 M 2022-01-31T19:34:18.935 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-01-31T19:34:18.935 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-01-31T19:34:18.935 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse x86_64 2:16.2.7-278.g9e931647.el8 ceph 45 k 2022-01-31T19:34:18.936 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:16.2.7-278.g9e931647.el8 ceph 92 k 2022-01-31T19:34:18.936 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:16.2.7-278.g9e931647.el8 ceph 214 k 2022-01-31T19:34:18.936 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-01-31T19:34:18.936 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:16.2.7-278.g9e931647.el8 ceph 387 k 2022-01-31T19:34:18.937 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:16.2.7-278.g9e931647.el8 ceph 368 k 2022-01-31T19:34:18.937 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:16.2.7-278.g9e931647.el8 ceph 114 k 2022-01-31T19:34:18.937 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:18.937 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:34:18.937 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:34:18.938 INFO:teuthology.orchestra.run.smithi167.stdout:Install 21 Packages 2022-01-31T19:34:18.938 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 2 Packages 2022-01-31T19:34:18.938 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:18.939 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 57 M 2022-01-31T19:34:18.939 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:34:19.728 INFO:teuthology.orchestra.run.smithi167.stdout:(1/23): ceph-base-16.2.7-278.g9e931647.el8.x86_ 7.6 MB/s | 6.0 MB 00:00 2022-01-31T19:34:19.812 INFO:teuthology.orchestra.run.smithi167.stdout:(2/23): ceph-selinux-16.2.7-278.g9e931647.el8.x 307 kB/s | 26 kB 00:00 2022-01-31T19:34:19.912 INFO:teuthology.orchestra.run.smithi167.stdout:(3/23): libcephfs2-16.2.7-278.g9e931647.el8.x86 8.0 MB/s | 813 kB 00:00 2022-01-31T19:34:20.004 INFO:teuthology.orchestra.run.smithi167.stdout:(4/23): libradosstriper1-16.2.7-278.g9e931647.e 5.2 MB/s | 492 kB 00:00 2022-01-31T19:34:20.379 INFO:teuthology.orchestra.run.smithi167.stdout:(5/23): librgw2-16.2.7-278.g9e931647.el8.x86_64 10 MB/s | 3.8 MB 00:00 2022-01-31T19:34:20.413 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:34:20.415 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:34:20.416 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:34:20.416 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:34:20.417 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:34:20.417 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:16.2.7-278.g9e931647.el8 ceph 12 M 2022-01-31T19:34:20.418 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2022-01-31T19:34:20.418 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:16.2.7-278.g9e931647.el8 ceph 3.7 M 2022-01-31T19:34:20.418 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:16.2.7-278.g9e931647.el8 ceph 4.2 M 2022-01-31T19:34:20.418 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:34:20.419 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:16.2.7-278.g9e931647.el8 ceph 6.0 M 2022-01-31T19:34:20.419 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:16.2.7-278.g9e931647.el8 ceph 24 M 2022-01-31T19:34:20.420 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:16.2.7-278.g9e931647.el8 ceph 26 k 2022-01-31T19:34:20.420 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-01-31T19:34:20.420 INFO:teuthology.orchestra.run.smithi171.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2022-01-31T19:34:20.421 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:16.2.7-278.g9e931647.el8 ceph 813 k 2022-01-31T19:34:20.421 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-01-31T19:34:20.421 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-01-31T19:34:20.421 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:16.2.7-278.g9e931647.el8 ceph 492 k 2022-01-31T19:34:20.422 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-01-31T19:34:20.422 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:16.2.7-278.g9e931647.el8 ceph 3.8 M 2022-01-31T19:34:20.422 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-01-31T19:34:20.422 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-01-31T19:34:20.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse x86_64 2:16.2.7-278.g9e931647.el8 ceph 45 k 2022-01-31T19:34:20.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:16.2.7-278.g9e931647.el8 ceph 92 k 2022-01-31T19:34:20.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:16.2.7-278.g9e931647.el8 ceph 214 k 2022-01-31T19:34:20.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-01-31T19:34:20.424 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:16.2.7-278.g9e931647.el8 ceph 387 k 2022-01-31T19:34:20.424 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:16.2.7-278.g9e931647.el8 ceph 368 k 2022-01-31T19:34:20.424 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:16.2.7-278.g9e931647.el8 ceph 114 k 2022-01-31T19:34:20.424 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:20.424 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:34:20.425 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:34:20.425 INFO:teuthology.orchestra.run.smithi171.stdout:Install 21 Packages 2022-01-31T19:34:20.425 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2022-01-31T19:34:20.425 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:20.426 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 57 M 2022-01-31T19:34:20.426 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:34:20.504 INFO:teuthology.orchestra.run.smithi167.stdout:(6/23): ceph-radosgw-16.2.7-278.g9e931647.el8.x 7.3 MB/s | 12 MB 00:01 2022-01-31T19:34:20.521 INFO:teuthology.orchestra.run.smithi167.stdout:(7/23): python3-ceph-argparse-16.2.7-278.g9e931 316 kB/s | 45 kB 00:00 2022-01-31T19:34:20.571 INFO:teuthology.orchestra.run.smithi167.stdout:(8/23): python3-ceph-common-16.2.7-278.g9e93164 1.4 MB/s | 92 kB 00:00 2022-01-31T19:34:20.622 INFO:teuthology.orchestra.run.smithi167.stdout:(9/23): python3-cephfs-16.2.7-278.g9e931647.el8 2.1 MB/s | 214 kB 00:00 2022-01-31T19:34:20.647 INFO:teuthology.orchestra.run.smithi167.stdout:(10/23): python3-rados-16.2.7-278.g9e931647.el8 5.0 MB/s | 387 kB 00:00 2022-01-31T19:34:20.689 INFO:teuthology.orchestra.run.smithi167.stdout:(11/23): python3-rbd-16.2.7-278.g9e931647.el8.x 5.4 MB/s | 368 kB 00:00 2022-01-31T19:34:20.714 INFO:teuthology.orchestra.run.smithi167.stdout:(12/23): python3-rgw-16.2.7-278.g9e931647.el8.x 1.7 MB/s | 114 kB 00:00 2022-01-31T19:34:21.741 INFO:teuthology.orchestra.run.smithi171.stdout:(1/23): ceph-base-16.2.7-278.g9e931647.el8.x86_ 4.6 MB/s | 6.0 MB 00:01 2022-01-31T19:34:21.822 INFO:teuthology.orchestra.run.smithi171.stdout:(2/23): ceph-selinux-16.2.7-278.g9e931647.el8.x 308 kB/s | 26 kB 00:00 2022-01-31T19:34:21.997 INFO:teuthology.orchestra.run.smithi171.stdout:(3/23): libcephfs2-16.2.7-278.g9e931647.el8.x86 4.5 MB/s | 813 kB 00:00 2022-01-31T19:34:22.181 INFO:teuthology.orchestra.run.smithi171.stdout:(4/23): libradosstriper1-16.2.7-278.g9e931647.e 2.6 MB/s | 492 kB 00:00 2022-01-31T19:34:22.225 INFO:teuthology.orchestra.run.smithi167.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 230 kB/s | 354 kB 00:01 2022-01-31T19:34:22.256 INFO:teuthology.orchestra.run.smithi167.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2022-01-31T19:34:22.282 INFO:teuthology.orchestra.run.smithi167.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-01-31T19:34:22.307 INFO:teuthology.orchestra.run.smithi167.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 163 kB/s | 259 kB 00:01 2022-01-31T19:34:22.502 INFO:teuthology.orchestra.run.smithi167.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 930 kB/s | 181 kB 00:00 2022-01-31T19:34:22.533 INFO:teuthology.orchestra.run.smithi167.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2022-01-31T19:34:22.539 INFO:teuthology.orchestra.run.smithi171.stdout:(5/23): ceph-radosgw-16.2.7-278.g9e931647.el8.x 5.4 MB/s | 12 MB 00:02 2022-01-31T19:34:22.558 INFO:teuthology.orchestra.run.smithi167.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2022-01-31T19:34:22.583 INFO:teuthology.orchestra.run.smithi167.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2022-01-31T19:34:22.818 INFO:teuthology.orchestra.run.smithi171.stdout:(6/23): ceph-common-16.2.7-278.g9e931647.el8.x8 9.9 MB/s | 24 MB 00:02 2022-01-31T19:34:23.008 INFO:teuthology.orchestra.run.smithi171.stdout:(7/23): python3-ceph-argparse-16.2.7-278.g9e931 96 kB/s | 45 kB 00:00 2022-01-31T19:34:23.157 INFO:teuthology.orchestra.run.smithi171.stdout:(8/23): python3-ceph-common-16.2.7-278.g9e93164 273 kB/s | 92 kB 00:00 2022-01-31T19:34:23.198 INFO:teuthology.orchestra.run.smithi171.stdout:(9/23): python3-cephfs-16.2.7-278.g9e931647.el8 1.1 MB/s | 214 kB 00:00 2022-01-31T19:34:23.208 INFO:teuthology.orchestra.run.smithi167.stdout:(21/23): ceph-common-16.2.7-278.g9e931647.el8.x 5.5 MB/s | 24 MB 00:04 2022-01-31T19:34:23.259 INFO:teuthology.orchestra.run.smithi171.stdout:(10/23): librgw2-16.2.7-278.g9e931647.el8.x86_6 3.5 MB/s | 3.8 MB 00:01 2022-01-31T19:34:23.275 INFO:teuthology.orchestra.run.smithi167.stdout:(22/23): librados2-16.2.7-278.g9e931647.el8.x86 5.2 MB/s | 3.7 MB 00:00 2022-01-31T19:34:23.284 INFO:teuthology.orchestra.run.smithi171.stdout:(11/23): python3-rados-16.2.7-278.g9e931647.el8 3.0 MB/s | 387 kB 00:00 2022-01-31T19:34:23.310 INFO:teuthology.orchestra.run.smithi171.stdout:(12/23): python3-rbd-16.2.7-278.g9e931647.el8.x 3.2 MB/s | 368 kB 00:00 2022-01-31T19:34:23.342 INFO:teuthology.orchestra.run.smithi167.stdout:(23/23): librbd1-16.2.7-278.g9e931647.el8.x86_6 5.5 MB/s | 4.2 MB 00:00 2022-01-31T19:34:23.343 INFO:teuthology.orchestra.run.smithi171.stdout:(13/23): python3-rgw-16.2.7-278.g9e931647.el8.x 1.3 MB/s | 114 kB 00:00 2022-01-31T19:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout:Total 13 MB/s | 57 MB 00:04 2022-01-31T19:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:34:23.402 INFO:teuthology.orchestra.run.smithi171.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.9 MB/s | 354 kB 00:00 2022-01-31T19:34:23.427 INFO:teuthology.orchestra.run.smithi171.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.9 MB/s | 47 kB 00:00 2022-01-31T19:34:23.433 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:34:23.434 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:34:23.452 INFO:teuthology.orchestra.run.smithi171.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 421 kB/s | 44 kB 00:00 2022-01-31T19:34:23.478 INFO:teuthology.orchestra.run.smithi171.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2022-01-31T19:34:23.612 INFO:teuthology.orchestra.run.smithi171.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 441 kB/s | 59 kB 00:00 2022-01-31T19:34:23.638 INFO:teuthology.orchestra.run.smithi171.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 983 kB/s | 181 kB 00:00 2022-01-31T19:34:23.663 INFO:teuthology.orchestra.run.smithi171.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2022-01-31T19:34:23.688 INFO:teuthology.orchestra.run.smithi171.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 991 kB/s | 75 kB 00:00 2022-01-31T19:34:23.763 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:34:23.766 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:34:23.971 INFO:teuthology.orchestra.run.smithi171.stdout:(22/23): librbd1-16.2.7-278.g9e931647.el8.x86_6 13 MB/s | 4.2 MB 00:00 2022-01-31T19:34:24.130 INFO:teuthology.orchestra.run.smithi171.stdout:(23/23): librados2-16.2.7-278.g9e931647.el8.x86 7.6 MB/s | 3.7 MB 00:00 2022-01-31T19:34:24.133 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:34:24.134 INFO:teuthology.orchestra.run.smithi171.stdout:Total 15 MB/s | 57 MB 00:03 2022-01-31T19:34:24.177 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:34:24.223 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:34:24.223 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:34:24.548 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:34:24.551 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:34:25.157 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:34:25.343 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2022-01-31T19:34:25.530 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-01-31T19:34:25.549 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:34:25.736 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2022-01-31T19:34:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-01-31T19:34:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-01-31T19:34:26.063 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:16.2.7-278.g9e931647.el8.x86_64 3/25 2022-01-31T19:34:26.157 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.7-278.g9e931647.el8.x86_64 3/25 2022-01-31T19:34:26.279 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-01-31T19:34:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:16.2.7-278.g9e931647.el8.x86_64 3/25 2022-01-31T19:34:26.461 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 4/25 2022-01-31T19:34:26.520 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.7-278.g9e931647.el8.x86_64 3/25 2022-01-31T19:34:26.638 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 5/25 2022-01-31T19:34:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 4/25 2022-01-31T19:34:26.900 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 5/25 2022-01-31T19:34:27.050 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 5/25 2022-01-31T19:34:27.238 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:16.2.7-278.g9e931647.el8.x86_64 6/25 2022-01-31T19:34:27.310 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 5/25 2022-01-31T19:34:27.330 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.7-278.g9e931647.el8.x86_64 6/25 2022-01-31T19:34:27.475 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:16.2.7-278.g9e931647.el8.x86_64 6/25 2022-01-31T19:34:27.511 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2022-01-31T19:34:27.544 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.7-278.g9e931647.el8.x86_64 6/25 2022-01-31T19:34:27.688 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2022-01-31T19:34:27.723 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2022-01-31T19:34:27.883 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2022-01-31T19:34:28.049 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2022-01-31T19:34:28.254 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2022-01-31T19:34:28.271 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:16.2.7-278.g9e931647.el8.x86_64 9/25 2022-01-31T19:34:28.337 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:16.2.7-278.g9e931647.el8.x86_64 9/25 2022-01-31T19:34:28.508 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:16.2.7-278.g9e931647.el8.x86_64 9/25 2022-01-31T19:34:28.519 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2022-01-31T19:34:28.634 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:16.2.7-278.g9e931647.el8.x 11/25 2022-01-31T19:34:28.858 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 12/25 2022-01-31T19:34:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 13/25 2022-01-31T19:34:29.180 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:16.2.7-278.g9e931647.el8.x86_64 9/25 2022-01-31T19:34:29.348 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2022-01-31T19:34:29.362 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 14/25 2022-01-31T19:34:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:16.2.7-278.g9e931647.el8.x 11/25 2022-01-31T19:34:29.679 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 12/25 2022-01-31T19:34:29.852 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 13/25 2022-01-31T19:34:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 14/25 2022-01-31T19:34:30.088 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 15/25 2022-01-31T19:34:30.149 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 15/25 2022-01-31T19:34:30.200 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 15/25 2022-01-31T19:34:30.273 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 15/25 2022-01-31T19:34:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2022-01-31T19:34:30.460 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2022-01-31T19:34:30.588 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2022-01-31T19:34:30.642 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2022-01-31T19:34:30.743 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-01-31T19:34:30.838 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-01-31T19:34:30.905 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:16.2.7-278.g9e931647.el8.x86 19/25 2022-01-31T19:34:30.959 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:16.2.7-278.g9e931647.el8.x86 19/25 2022-01-31T19:34:34.287 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 20/25 2022-01-31T19:34:34.294 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 20/25 2022-01-31T19:34:34.491 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 20/25 2022-01-31T19:34:34.504 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 20/25 2022-01-31T19:34:35.180 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 20/25 2022-01-31T19:34:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 20/25 2022-01-31T19:34:35.461 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 21/25 2022-01-31T19:34:35.491 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 21/25 2022-01-31T19:34:35.558 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 21/25 2022-01-31T19:34:35.624 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 21/25 2022-01-31T19:34:35.674 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 22/25 2022-01-31T19:34:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 22/25 2022-01-31T19:34:53.431 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 22/25 2022-01-31T19:34:53.432 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2022-01-31T19:34:53.432 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2022-01-31T19:34:53.432 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /dev 2022-01-31T19:34:53.433 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /run 2022-01-31T19:34:53.433 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2022-01-31T19:34:53.433 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2022-01-31T19:34:53.433 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2022-01-31T19:34:53.434 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2022-01-31T19:34:53.434 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /dev 2022-01-31T19:34:53.434 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:53.810 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 22/25 2022-01-31T19:34:53.810 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2022-01-31T19:34:53.811 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2022-01-31T19:34:53.811 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /dev 2022-01-31T19:34:53.811 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /run 2022-01-31T19:34:53.811 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2022-01-31T19:34:53.812 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2022-01-31T19:34:53.812 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2022-01-31T19:34:53.812 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2022-01-31T19:34:53.813 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /dev 2022-01-31T19:34:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:54.591 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 23/25 2022-01-31T19:34:54.661 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 23/25 2022-01-31T19:34:54.662 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-01-31T19:34:54.856 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-01-31T19:34:54.857 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-01-31T19:34:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 23/25 2022-01-31T19:34:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 23/25 2022-01-31T19:34:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-01-31T19:34:55.320 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-01-31T19:34:55.320 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-01-31T19:34:55.605 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-01-31T19:34:55.606 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 1/25 2022-01-31T19:34:55.606 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 2/25 2022-01-31T19:34:55.606 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 3/25 2022-01-31T19:34:55.606 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 4/25 2022-01-31T19:34:55.607 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 5/25 2022-01-31T19:34:55.607 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 6/25 2022-01-31T19:34:55.607 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:16.2.7-278.g9e931647.el8.x86_64 7/25 2022-01-31T19:34:55.607 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:16.2.7-278.g9e931647.el8.x 8/25 2022-01-31T19:34:55.608 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:16.2.7-278.g9e931647.el8.x86 9/25 2022-01-31T19:34:55.608 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 10/25 2022-01-31T19:34:55.608 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 11/25 2022-01-31T19:34:55.608 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 12/25 2022-01-31T19:34:55.609 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 13/25 2022-01-31T19:34:55.609 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-01-31T19:34:55.609 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-01-31T19:34:55.609 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-01-31T19:34:55.609 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-01-31T19:34:55.610 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-01-31T19:34:55.610 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2022-01-31T19:34:55.611 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2022-01-31T19:34:55.611 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2022-01-31T19:34:55.611 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.7-278.g9e931647.el8.x86_64 22/25 2022-01-31T19:34:55.611 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-01-31T19:34:55.612 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.7-278.g9e931647.el8.x86_64 24/25 2022-01-31T19:34:56.018 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-01-31T19:34:56.019 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 1/25 2022-01-31T19:34:56.019 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 2/25 2022-01-31T19:34:56.019 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 3/25 2022-01-31T19:34:56.020 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 4/25 2022-01-31T19:34:56.020 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 5/25 2022-01-31T19:34:56.021 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 6/25 2022-01-31T19:34:56.021 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:16.2.7-278.g9e931647.el8.x86_64 7/25 2022-01-31T19:34:56.022 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:16.2.7-278.g9e931647.el8.x 8/25 2022-01-31T19:34:56.022 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:16.2.7-278.g9e931647.el8.x86 9/25 2022-01-31T19:34:56.023 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 10/25 2022-01-31T19:34:56.023 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 11/25 2022-01-31T19:34:56.024 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 12/25 2022-01-31T19:34:56.024 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 13/25 2022-01-31T19:34:56.024 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-01-31T19:34:56.025 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-01-31T19:34:56.025 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-01-31T19:34:56.025 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-01-31T19:34:56.026 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-01-31T19:34:56.026 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2022-01-31T19:34:56.027 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2022-01-31T19:34:56.027 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2022-01-31T19:34:56.027 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.7-278.g9e931647.el8.x86_64 22/25 2022-01-31T19:34:56.028 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-01-31T19:34:56.028 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.7-278.g9e931647.el8.x86_64 24/25 2022-01-31T19:34:56.487 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-01-31T19:34:56.487 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:56.487 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2022-01-31T19:34:56.487 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.488 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.488 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:34:56.488 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.488 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.489 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.489 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.489 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-01-31T19:34:56.489 INFO:teuthology.orchestra.run.smithi171.stdout: leveldb-1.22-1.el8.x86_64 2022-01-31T19:34:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.2-3.el8.x86_64 2022-01-31T19:34:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-01-31T19:34:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.491 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-01-31T19:34:56.491 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.491 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.3.1-3.el8.x86_64 2022-01-31T19:34:56.491 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-01-31T19:34:56.491 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.492 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.492 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.492 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-01-31T19:34:56.492 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.493 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.493 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.493 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:56.493 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:34:56.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test 2022-01-31T19:34:56.897 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:34:56.897 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:34:56.916 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-01-31T19:34:56.917 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:56.917 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2022-01-31T19:34:56.917 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.917 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.918 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:34:56.918 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.918 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.918 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.918 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.919 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-01-31T19:34:56.919 INFO:teuthology.orchestra.run.smithi167.stdout: leveldb-1.22-1.el8.x86_64 2022-01-31T19:34:56.919 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.919 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.2-3.el8.x86_64 2022-01-31T19:34:56.920 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-01-31T19:34:56.920 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.920 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-01-31T19:34:56.920 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.921 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.3.1-3.el8.x86_64 2022-01-31T19:34:56.921 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-01-31T19:34:56.921 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.921 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.921 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.922 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-01-31T19:34:56.922 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.922 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.922 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:34:56.923 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:56.923 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:34:57.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test 2022-01-31T19:34:57.280 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:40 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:34:57.380 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:34:57.381 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:34:57.763 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:41 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:34:58.048 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:34:58.049 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:34:58.049 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:34:58.049 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:34:58.049 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:34:58.050 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:16.2.7-278.g9e931647.el8 ceph 52 M 2022-01-31T19:34:58.050 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:34:58.050 INFO:teuthology.orchestra.run.smithi171.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-01-31T19:34:58.050 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:16.2.7-278.g9e931647.el8 ceph 173 k 2022-01-31T19:34:58.051 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-01-31T19:34:58.051 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-01-31T19:34:58.051 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-01-31T19:34:58.051 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:58.052 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:34:58.052 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:34:58.052 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2022-01-31T19:34:58.052 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:34:58.053 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 53 M 2022-01-31T19:34:58.053 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 194 M 2022-01-31T19:34:58.054 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:34:58.115 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 3.1 MB/s | 202 kB 00:00 2022-01-31T19:34:58.149 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2022-01-31T19:34:58.200 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2022-01-31T19:34:58.425 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): libcephsqlite-16.2.7-278.g9e931647.el8.x 464 kB/s | 173 kB 00:00 2022-01-31T19:34:58.547 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:34:58.547 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:34:58.548 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:34:58.548 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:34:58.548 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:34:58.549 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:16.2.7-278.g9e931647.el8 ceph 52 M 2022-01-31T19:34:58.549 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:34:58.549 INFO:teuthology.orchestra.run.smithi167.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-01-31T19:34:58.549 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:16.2.7-278.g9e931647.el8 ceph 173 k 2022-01-31T19:34:58.550 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-01-31T19:34:58.550 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-01-31T19:34:58.550 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-01-31T19:34:58.551 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:58.551 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:34:58.551 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:34:58.551 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2022-01-31T19:34:58.552 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:34:58.552 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 53 M 2022-01-31T19:34:58.553 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 194 M 2022-01-31T19:34:58.553 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:34:58.625 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.7 MB/s | 202 kB 00:00 2022-01-31T19:34:58.659 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2022-01-31T19:34:58.692 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2022-01-31T19:34:58.917 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): libcephsqlite-16.2.7-278.g9e931647.el8.x 472 kB/s | 173 kB 00:00 2022-01-31T19:34:59.117 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 75 kB/s | 69 kB 00:00 2022-01-31T19:34:59.309 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 112 kB/s | 69 kB 00:00 2022-01-31T19:35:02.392 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): ceph-test-16.2.7-278.g9e931647.el8.x86_6 12 MB/s | 52 MB 00:04 2022-01-31T19:35:02.394 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:02.395 INFO:teuthology.orchestra.run.smithi171.stdout:Total 12 MB/s | 53 MB 00:04 2022-01-31T19:35:02.433 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:02.454 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:02.454 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:02.748 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:02.751 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:03.213 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:03.397 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2022-01-31T19:35:03.604 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-01-31T19:35:03.684 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): ceph-test-16.2.7-278.g9e931647.el8.x86_6 10 MB/s | 52 MB 00:05 2022-01-31T19:35:03.687 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:03.687 INFO:teuthology.orchestra.run.smithi167.stdout:Total 10 MB/s | 53 MB 00:05 2022-01-31T19:35:03.725 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:03.746 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:03.746 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:03.752 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-01-31T19:35:03.838 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-01-31T19:35:03.992 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-01-31T19:35:04.048 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:04.051 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:04.144 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 5/6 2022-01-31T19:35:04.523 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:04.719 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2022-01-31T19:35:04.935 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-01-31T19:35:05.124 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-01-31T19:35:05.204 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-01-31T19:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-01-31T19:35:05.516 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 5/6 2022-01-31T19:35:08.764 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 5/6 2022-01-31T19:35:09.008 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 6/6 2022-01-31T19:35:09.269 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 6/6 2022-01-31T19:35:09.270 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 1/6 2022-01-31T19:35:09.270 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 2/6 2022-01-31T19:35:09.271 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-01-31T19:35:09.271 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-01-31T19:35:09.272 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-01-31T19:35:09.758 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2022-01-31T19:35:09.759 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:09.759 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:09.759 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:09.759 INFO:teuthology.orchestra.run.smithi171.stdout: jq-1.6-3.el8.x86_64 2022-01-31T19:35:09.760 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:09.760 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-01-31T19:35:09.760 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-1.el8.x86_64 2022-01-31T19:35:09.760 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-01-31T19:35:09.761 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:09.761 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:09.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph 2022-01-31T19:35:10.098 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 5/6 2022-01-31T19:35:10.099 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:10.100 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 6/6 2022-01-31T19:35:10.490 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:53 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:10.600 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 6/6 2022-01-31T19:35:10.600 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 1/6 2022-01-31T19:35:10.601 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 2/6 2022-01-31T19:35:10.601 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-01-31T19:35:10.601 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-01-31T19:35:10.602 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-01-31T19:35:11.016 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2022-01-31T19:35:11.016 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:11.017 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:35:11.017 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:11.017 INFO:teuthology.orchestra.run.smithi167.stdout: jq-1.6-3.el8.x86_64 2022-01-31T19:35:11.017 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:11.018 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-01-31T19:35:11.018 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-1.el8.x86_64 2022-01-31T19:35:11.018 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-01-31T19:35:11.018 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:11.019 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:11.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph 2022-01-31T19:35:11.269 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:11.273 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2022-01-31T19:35:11.273 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:35:11.273 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2022-01-31T19:35:11.274 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:35:11.274 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:16.2.7-278.g9e931647.el8 ceph 6.4 k 2022-01-31T19:35:11.274 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:35:11.274 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:16.2.7-278.g9e931647.el8 ceph 2.4 M 2022-01-31T19:35:11.275 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:16.2.7-278.g9e931647.el8 ceph 1.6 M 2022-01-31T19:35:11.275 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 232 k 2022-01-31T19:35:11.275 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:16.2.7-278.g9e931647.el8 ceph 5.0 M 2022-01-31T19:35:11.275 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:16.2.7-278.g9e931647.el8 ceph 18 M 2022-01-31T19:35:11.276 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-01-31T19:35:11.276 INFO:teuthology.orchestra.run.smithi171.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-01-31T19:35:11.277 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-01-31T19:35:11.277 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-01-31T19:35:11.277 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-01-31T19:35:11.277 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-01-31T19:35:11.278 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-01-31T19:35:11.278 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-01-31T19:35:11.278 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-01-31T19:35:11.278 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-01-31T19:35:11.279 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-01-31T19:35:11.279 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-01-31T19:35:11.279 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-01-31T19:35:11.279 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-01-31T19:35:11.280 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-01-31T19:35:11.280 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-01-31T19:35:11.280 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-01-31T19:35:11.280 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-01-31T19:35:11.280 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-01-31T19:35:11.281 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-01-31T19:35:11.281 INFO:teuthology.orchestra.run.smithi171.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-01-31T19:35:11.281 INFO:teuthology.orchestra.run.smithi171.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-01-31T19:35:11.281 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-01-31T19:35:11.282 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-01-31T19:35:11.282 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-01-31T19:35:11.282 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-01-31T19:35:11.282 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:11.283 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:35:11.283 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2022-01-31T19:35:11.284 INFO:teuthology.orchestra.run.smithi171.stdout:Install 32 Packages 2022-01-31T19:35:11.284 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:11.285 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 31 M 2022-01-31T19:35:11.285 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 100 M 2022-01-31T19:35:11.285 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:35:11.404 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:11.404 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:11.456 INFO:teuthology.orchestra.run.smithi171.stdout:(1/32): ceph-16.2.7-278.g9e931647.el8.x86_64.rp 36 kB/s | 6.4 kB 00:00 2022-01-31T19:35:11.756 INFO:teuthology.orchestra.run.smithi171.stdout:(2/32): ceph-mds-16.2.7-278.g9e931647.el8.x86_6 5.0 MB/s | 2.4 MB 00:00 2022-01-31T19:35:11.789 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:55 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:11.831 INFO:teuthology.orchestra.run.smithi171.stdout:(3/32): ceph-mgr-16.2.7-278.g9e931647.el8.x86_6 2.9 MB/s | 1.6 MB 00:00 2022-01-31T19:35:11.940 INFO:teuthology.orchestra.run.smithi171.stdout:(4/32): ceph-mgr-modules-core-16.2.7-278.g9e931 2.1 MB/s | 232 kB 00:00 2022-01-31T19:35:11.990 INFO:teuthology.orchestra.run.smithi171.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2022-01-31T19:35:12.016 INFO:teuthology.orchestra.run.smithi171.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-01-31T19:35:12.041 INFO:teuthology.orchestra.run.smithi171.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2022-01-31T19:35:12.090 INFO:teuthology.orchestra.run.smithi171.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2022-01-31T19:35:12.125 INFO:teuthology.orchestra.run.smithi171.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2022-01-31T19:35:12.150 INFO:teuthology.orchestra.run.smithi171.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2022-01-31T19:35:12.192 INFO:teuthology.orchestra.run.smithi171.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2022-01-31T19:35:12.218 INFO:teuthology.orchestra.run.smithi171.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.2 MB/s | 109 kB 00:00 2022-01-31T19:35:12.293 INFO:teuthology.orchestra.run.smithi171.stdout:(13/32): ceph-mon-16.2.7-278.g9e931647.el8.x86_ 6.0 MB/s | 5.0 MB 00:00 2022-01-31T19:35:12.402 INFO:teuthology.orchestra.run.smithi171.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 238 kB/s | 44 kB 00:00 2022-01-31T19:35:12.461 INFO:teuthology.orchestra.run.smithi171.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2022-01-31T19:35:12.511 INFO:teuthology.orchestra.run.smithi171.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2022-01-31T19:35:12.544 INFO:teuthology.orchestra.run.smithi171.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2022-01-31T19:35:12.570 INFO:teuthology.orchestra.run.smithi171.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 188 kB/s | 11 kB 00:00 2022-01-31T19:35:12.586 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:12.589 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2022-01-31T19:35:12.589 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:35:12.590 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2022-01-31T19:35:12.590 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:35:12.590 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:16.2.7-278.g9e931647.el8 ceph 6.4 k 2022-01-31T19:35:12.590 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:35:12.591 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:16.2.7-278.g9e931647.el8 ceph 2.4 M 2022-01-31T19:35:12.591 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:16.2.7-278.g9e931647.el8 ceph 1.6 M 2022-01-31T19:35:12.591 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 232 k 2022-01-31T19:35:12.591 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:16.2.7-278.g9e931647.el8 ceph 5.0 M 2022-01-31T19:35:12.592 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:16.2.7-278.g9e931647.el8 ceph 18 M 2022-01-31T19:35:12.592 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-01-31T19:35:12.592 INFO:teuthology.orchestra.run.smithi167.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-01-31T19:35:12.592 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-01-31T19:35:12.593 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-01-31T19:35:12.593 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-01-31T19:35:12.593 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-01-31T19:35:12.593 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-01-31T19:35:12.593 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-01-31T19:35:12.594 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-01-31T19:35:12.594 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-01-31T19:35:12.594 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-01-31T19:35:12.594 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-01-31T19:35:12.595 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-01-31T19:35:12.595 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-01-31T19:35:12.595 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-01-31T19:35:12.595 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-01-31T19:35:12.596 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-01-31T19:35:12.596 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-01-31T19:35:12.596 INFO:teuthology.orchestra.run.smithi167.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-01-31T19:35:12.596 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-01-31T19:35:12.596 INFO:teuthology.orchestra.run.smithi167.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-01-31T19:35:12.597 INFO:teuthology.orchestra.run.smithi167.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-01-31T19:35:12.597 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-01-31T19:35:12.597 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-01-31T19:35:12.597 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-01-31T19:35:12.598 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-01-31T19:35:12.598 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:12.598 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:35:12.598 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2022-01-31T19:35:12.599 INFO:teuthology.orchestra.run.smithi167.stdout:Install 32 Packages 2022-01-31T19:35:12.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:12.600 INFO:teuthology.orchestra.run.smithi171.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 348 kB/s | 18 kB 00:00 2022-01-31T19:35:12.600 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 31 M 2022-01-31T19:35:12.600 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 100 M 2022-01-31T19:35:12.602 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:35:12.621 INFO:teuthology.orchestra.run.smithi171.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 967 kB/s | 49 kB 00:00 2022-01-31T19:35:12.786 INFO:teuthology.orchestra.run.smithi167.stdout:(1/32): ceph-16.2.7-278.g9e931647.el8.x86_64.rp 34 kB/s | 6.4 kB 00:00 2022-01-31T19:35:12.805 INFO:teuthology.orchestra.run.smithi171.stdout:(21/32): ceph-osd-16.2.7-278.g9e931647.el8.x86_ 17 MB/s | 18 MB 00:01 2022-01-31T19:35:12.830 INFO:teuthology.orchestra.run.smithi171.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 250 kB/s | 59 kB 00:00 2022-01-31T19:35:12.847 INFO:teuthology.orchestra.run.smithi171.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2022-01-31T19:35:12.881 INFO:teuthology.orchestra.run.smithi171.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 371 kB/s | 19 kB 00:00 2022-01-31T19:35:12.898 INFO:teuthology.orchestra.run.smithi171.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2022-01-31T19:35:12.923 INFO:teuthology.orchestra.run.smithi171.stdout:(26/32): python3-portend-2.6-1.el8.noarch.rpm 132 kB/s | 16 kB 00:00 2022-01-31T19:35:12.940 INFO:teuthology.orchestra.run.smithi171.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 494 kB/s | 29 kB 00:00 2022-01-31T19:35:12.957 INFO:teuthology.orchestra.run.smithi171.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 464 kB/s | 27 kB 00:00 2022-01-31T19:35:12.999 INFO:teuthology.orchestra.run.smithi171.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2022-01-31T19:35:13.025 INFO:teuthology.orchestra.run.smithi171.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2022-01-31T19:35:13.045 INFO:teuthology.orchestra.run.smithi167.stdout:(2/32): ceph-mgr-16.2.7-278.g9e931647.el8.x86_6 3.5 MB/s | 1.6 MB 00:00 2022-01-31T19:35:13.050 INFO:teuthology.orchestra.run.smithi171.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 456 kB/s | 23 kB 00:00 2022-01-31T19:35:13.075 INFO:teuthology.orchestra.run.smithi171.stdout:(32/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.6 MB/s | 241 kB 00:00 2022-01-31T19:35:13.078 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:13.078 INFO:teuthology.orchestra.run.smithi171.stdout:Total 17 MB/s | 31 MB 00:01 2022-01-31T19:35:13.094 INFO:teuthology.orchestra.run.smithi167.stdout:(3/32): ceph-mds-16.2.7-278.g9e931647.el8.x86_6 4.8 MB/s | 2.4 MB 00:00 2022-01-31T19:35:13.165 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:13.186 INFO:teuthology.orchestra.run.smithi167.stdout:(4/32): ceph-mgr-modules-core-16.2.7-278.g9e931 2.5 MB/s | 232 kB 00:00 2022-01-31T19:35:13.204 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:13.205 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:13.245 INFO:teuthology.orchestra.run.smithi167.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-01-31T19:35:13.271 INFO:teuthology.orchestra.run.smithi167.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-01-31T19:35:13.335 INFO:teuthology.orchestra.run.smithi167.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2022-01-31T19:35:13.354 INFO:teuthology.orchestra.run.smithi167.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-01-31T19:35:13.388 INFO:teuthology.orchestra.run.smithi167.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2022-01-31T19:35:13.413 INFO:teuthology.orchestra.run.smithi167.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2022-01-31T19:35:13.455 INFO:teuthology.orchestra.run.smithi167.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2022-01-31T19:35:13.481 INFO:teuthology.orchestra.run.smithi167.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-01-31T19:35:13.495 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:13.497 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:13.556 INFO:teuthology.orchestra.run.smithi167.stdout:(13/32): ceph-mon-16.2.7-278.g9e931647.el8.x86_ 6.5 MB/s | 5.0 MB 00:00 2022-01-31T19:35:13.673 INFO:teuthology.orchestra.run.smithi167.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 227 kB/s | 44 kB 00:00 2022-01-31T19:35:13.723 INFO:teuthology.orchestra.run.smithi167.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2022-01-31T19:35:13.782 INFO:teuthology.orchestra.run.smithi167.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2022-01-31T19:35:13.807 INFO:teuthology.orchestra.run.smithi167.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2022-01-31T19:35:13.833 INFO:teuthology.orchestra.run.smithi167.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2022-01-31T19:35:13.883 INFO:teuthology.orchestra.run.smithi167.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2022-01-31T19:35:13.908 INFO:teuthology.orchestra.run.smithi167.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2022-01-31T19:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2022-01-31T19:35:14.143 INFO:teuthology.orchestra.run.smithi167.stdout:(22/32): ceph-osd-16.2.7-278.g9e931647.el8.x86_ 17 MB/s | 18 MB 00:01 2022-01-31T19:35:14.176 INFO:teuthology.orchestra.run.smithi167.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 64 kB/s | 16 kB 00:00 2022-01-31T19:35:14.201 INFO:teuthology.orchestra.run.smithi167.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 966 kB/s | 283 kB 00:00 2022-01-31T19:35:14.227 INFO:teuthology.orchestra.run.smithi167.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2022-01-31T19:35:14.252 INFO:teuthology.orchestra.run.smithi167.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2022-01-31T19:35:14.277 INFO:teuthology.orchestra.run.smithi167.stdout:(27/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2022-01-31T19:35:14.303 INFO:teuthology.orchestra.run.smithi167.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2022-01-31T19:35:14.336 INFO:teuthology.orchestra.run.smithi167.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.8 MB/s | 241 kB 00:00 2022-01-31T19:35:14.353 INFO:teuthology.orchestra.run.smithi167.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2022-01-31T19:35:14.386 INFO:teuthology.orchestra.run.smithi167.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2022-01-31T19:35:14.437 INFO:teuthology.orchestra.run.smithi167.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2022-01-31T19:35:14.439 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:14.440 INFO:teuthology.orchestra.run.smithi167.stdout:Total 17 MB/s | 31 MB 00:01 2022-01-31T19:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:14.566 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:14.566 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:14.571 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:14.765 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-01-31T19:35:14.882 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:14.885 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:14.975 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-01-31T19:35:15.180 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-01-31T19:35:15.415 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-01-31T19:35:15.563 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-01-31T19:35:15.710 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-01-31T19:35:15.887 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-01-31T19:35:16.022 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-01-31T19:35:16.058 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:16.231 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-01-31T19:35:16.244 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-01-31T19:35:16.426 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-01-31T19:35:16.464 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-01-31T19:35:16.622 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-01-31T19:35:16.693 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-01-31T19:35:16.843 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-01-31T19:35:16.945 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-01-31T19:35:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-01-31T19:35:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-01-31T19:35:17.256 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-01-31T19:35:17.400 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-01-31T19:35:17.411 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-01-31T19:35:17.545 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-01-31T19:35:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-01-31T19:35:17.766 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-01-31T19:35:17.794 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-01-31T19:35:17.951 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-01-31T19:35:18.019 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-01-31T19:35:18.192 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-01-31T19:35:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-01-31T19:35:18.422 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-01-31T19:35:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-01-31T19:35:18.620 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-01-31T19:35:18.789 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-01-31T19:35:18.828 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-01-31T19:35:19.019 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-01-31T19:35:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-01-31T19:35:19.221 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-01-31T19:35:19.255 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-01-31T19:35:19.454 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-01-31T19:35:19.455 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-01-31T19:35:19.682 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-01-31T19:35:19.739 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-01-31T19:35:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-01-31T19:35:20.034 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-01-31T19:35:20.119 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-01-31T19:35:20.333 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-01-31T19:35:20.411 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:16.2.7-278.g9e931647.el8.n 27/32 2022-01-31T19:35:20.519 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-01-31T19:35:20.637 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 28/32 2022-01-31T19:35:20.731 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-01-31T19:35:20.943 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-01-31T19:35:21.187 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-01-31T19:35:21.510 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-01-31T19:35:21.835 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-01-31T19:35:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 28/32 2022-01-31T19:35:22.217 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:16.2.7-278.g9e931647.el8.n 27/32 2022-01-31T19:35:22.308 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 29/32 2022-01-31T19:35:22.409 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 28/32 2022-01-31T19:35:22.741 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 29/32 2022-01-31T19:35:22.892 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 30/32 2022-01-31T19:35:23.104 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 30/32 2022-01-31T19:35:23.316 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 31/32 2022-01-31T19:35:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 31/32 2022-01-31T19:35:23.449 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:16.2.7-278.g9e931647.el8.x86_64 32/32 2022-01-31T19:35:23.529 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-2:16.2.7-278.g9e931647.el8.x86_64 32/32 2022-01-31T19:35:23.530 INFO:teuthology.orchestra.run.smithi171.stdout:Couldn't write '0 2147483647' to '-net/ipv4/ping_group_range', ignoring: No such file or directory 2022-01-31T19:35:23.530 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:23.833 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:16.2.7-278.g9e931647.el8.x86_64 1/32 2022-01-31T19:35:23.834 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 2/32 2022-01-31T19:35:23.834 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 3/32 2022-01-31T19:35:23.835 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 4/32 2022-01-31T19:35:23.835 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 5/32 2022-01-31T19:35:23.835 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:16.2.7-278.g9e931647.el8.n 6/32 2022-01-31T19:35:23.835 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-01-31T19:35:23.836 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-01-31T19:35:23.836 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-01-31T19:35:23.836 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-01-31T19:35:23.836 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-01-31T19:35:23.837 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-01-31T19:35:23.837 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-01-31T19:35:23.837 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-01-31T19:35:23.837 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-01-31T19:35:23.838 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-01-31T19:35:23.838 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-01-31T19:35:23.838 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-01-31T19:35:23.838 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-01-31T19:35:23.839 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-01-31T19:35:23.839 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-01-31T19:35:23.839 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-01-31T19:35:23.840 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-01-31T19:35:23.840 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-01-31T19:35:23.840 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-01-31T19:35:23.840 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-01-31T19:35:23.841 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-01-31T19:35:23.841 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-01-31T19:35:23.841 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-01-31T19:35:23.841 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-01-31T19:35:23.842 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-01-31T19:35:23.889 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 28/32 2022-01-31T19:35:24.105 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 29/32 2022-01-31T19:35:24.542 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 29/32 2022-01-31T19:35:24.638 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-01-31T19:35:24.639 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:24.639 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:24.639 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:24.639 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:24.640 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:24.640 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:24.640 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:24.640 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:24.640 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-01-31T19:35:24.641 INFO:teuthology.orchestra.run.smithi171.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-01-31T19:35:24.641 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-01-31T19:35:24.641 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-01-31T19:35:24.641 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-01-31T19:35:24.642 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-01-31T19:35:24.642 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-6.2-6.el8.noarch 2022-01-31T19:35:24.642 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-01-31T19:35:24.642 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-01-31T19:35:24.643 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.0.6-13.el8.noarch 2022-01-31T19:35:24.643 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-01-31T19:35:24.643 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-01-31T19:35:24.643 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-01-31T19:35:24.644 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-2.6-1.el8.noarch 2022-01-31T19:35:24.644 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-01-31T19:35:24.644 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.14-14.el8.noarch 2022-01-31T19:35:24.644 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2017.2-9.el8.noarch 2022-01-31T19:35:24.645 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-01-31T19:35:24.645 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-01-31T19:35:24.645 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-01-31T19:35:24.645 INFO:teuthology.orchestra.run.smithi171.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-01-31T19:35:24.646 INFO:teuthology.orchestra.run.smithi171.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-01-31T19:35:24.646 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-01-31T19:35:24.646 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-01-31T19:35:24.646 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-01-31T19:35:24.646 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-01-31T19:35:24.647 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:24.647 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 30/32 2022-01-31T19:35:24.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base 2022-01-31T19:35:24.926 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 30/32 2022-01-31T19:35:25.039 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:25.040 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:25.155 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 31/32 2022-01-31T19:35:25.172 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 31/32 2022-01-31T19:35:25.296 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:16.2.7-278.g9e931647.el8.x86_64 32/32 2022-01-31T19:35:25.389 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-2:16.2.7-278.g9e931647.el8.x86_64 32/32 2022-01-31T19:35:25.389 INFO:teuthology.orchestra.run.smithi167.stdout:Couldn't write '0 2147483647' to '-net/ipv4/ping_group_range', ignoring: No such file or directory 2022-01-31T19:35:25.390 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:25.414 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:08 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:16.2.7-278.g9e931647.el8.x86_64 1/32 2022-01-31T19:35:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 2/32 2022-01-31T19:35:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 3/32 2022-01-31T19:35:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 4/32 2022-01-31T19:35:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 5/32 2022-01-31T19:35:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:16.2.7-278.g9e931647.el8.n 6/32 2022-01-31T19:35:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-01-31T19:35:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-01-31T19:35:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-01-31T19:35:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-01-31T19:35:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-01-31T19:35:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-01-31T19:35:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-01-31T19:35:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-01-31T19:35:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-01-31T19:35:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-01-31T19:35:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-01-31T19:35:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-01-31T19:35:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-01-31T19:35:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-01-31T19:35:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-01-31T19:35:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-01-31T19:35:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-01-31T19:35:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-01-31T19:35:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-01-31T19:35:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-01-31T19:35:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-01-31T19:35:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-01-31T19:35:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-01-31T19:35:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-01-31T19:35:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-01-31T19:35:26.173 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:35:26.218 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:26.219 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:35:26.219 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:26.292 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install cephadm 2022-01-31T19:35:26.471 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:26.544 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:26.632 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-01-31T19:35:26.633 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:26.633 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:35:26.633 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:26.633 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:26.634 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:26.634 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:26.634 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:26.634 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:26.635 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-01-31T19:35:26.635 INFO:teuthology.orchestra.run.smithi167.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-01-31T19:35:26.635 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-01-31T19:35:26.635 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-01-31T19:35:26.636 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-01-31T19:35:26.636 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-01-31T19:35:26.636 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-6.2-6.el8.noarch 2022-01-31T19:35:26.636 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-01-31T19:35:26.636 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-01-31T19:35:26.637 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.0.6-13.el8.noarch 2022-01-31T19:35:26.637 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-01-31T19:35:26.637 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-01-31T19:35:26.637 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-01-31T19:35:26.638 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-2.6-1.el8.noarch 2022-01-31T19:35:26.638 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-01-31T19:35:26.638 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.14-14.el8.noarch 2022-01-31T19:35:26.638 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2017.2-9.el8.noarch 2022-01-31T19:35:26.639 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-01-31T19:35:26.639 INFO:teuthology.orchestra.run.smithi167.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-01-31T19:35:26.639 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-01-31T19:35:26.639 INFO:teuthology.orchestra.run.smithi167.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-01-31T19:35:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-01-31T19:35:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-01-31T19:35:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-01-31T19:35:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-01-31T19:35:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-01-31T19:35:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:26.641 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:26.847 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:09 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:26.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base 2022-01-31T19:35:27.040 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:27.041 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:27.425 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:11 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:27.609 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:27.610 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:27.610 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:35:27.611 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:27.611 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:35:27.611 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 78 k 2022-01-31T19:35:27.611 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:27.612 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:35:27.612 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:27.612 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:35:27.612 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:27.612 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 78 k 2022-01-31T19:35:27.613 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 321 k 2022-01-31T19:35:27.613 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:35:27.848 INFO:teuthology.orchestra.run.smithi171.stdout:cephadm-16.2.7-278.g9e931647.el8.noarch.rpm 328 kB/s | 78 kB 00:00 2022-01-31T19:35:27.849 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:27.849 INFO:teuthology.orchestra.run.smithi171.stdout:Total 326 kB/s | 78 kB 00:00 2022-01-31T19:35:27.849 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:27.856 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:27.856 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:27.858 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:27.861 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:27.929 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:28.148 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:35:28.193 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:28.195 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:35:28.195 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:28.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install cephadm 2022-01-31T19:35:28.443 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:28.443 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:28.822 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:12 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:29.598 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:29.599 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:29.599 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:35:29.599 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:29.600 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:35:29.600 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 78 k 2022-01-31T19:35:29.600 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:29.600 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:35:29.601 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:29.601 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:35:29.601 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:29.601 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 78 k 2022-01-31T19:35:29.602 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 321 k 2022-01-31T19:35:29.602 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:35:29.854 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm-16.2.7-278.g9e931647.el8.noarch.rpm 305 kB/s | 78 kB 00:00 2022-01-31T19:35:29.855 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:29.855 INFO:teuthology.orchestra.run.smithi167.stdout:Total 304 kB/s | 78 kB 00:00 2022-01-31T19:35:29.856 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:29.863 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:29.863 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:29.864 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:29.867 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:29.914 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:30.014 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:30.190 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:30.387 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:30.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:30.745 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:30.745 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:30.745 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:30.746 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:30.746 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:30.952 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-immutable-object-cache 2022-01-31T19:35:31.149 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:31.150 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:31.445 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:31.534 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:14 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:31.637 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:31.833 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:32.201 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/1 2022-01-31T19:35:32.201 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:32.201 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:35:32.202 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:32.202 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:32.202 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:32.346 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:32.347 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:32.347 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2022-01-31T19:35:32.347 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:32.348 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:35:32.348 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:16.2.7-278.g9e931647.el8 ceph 161 k 2022-01-31T19:35:32.348 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:32.348 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:35:32.349 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:32.349 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:35:32.349 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:32.349 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 161 k 2022-01-31T19:35:32.349 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 471 k 2022-01-31T19:35:32.350 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:35:32.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache 2022-01-31T19:35:32.649 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-immutable-object-cache-16.2.7-278.g9e93164 535 kB/s | 161 kB 00:00 2022-01-31T19:35:32.651 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:32.651 INFO:teuthology.orchestra.run.smithi171.stdout:Total 532 kB/s | 161 kB 00:00 2022-01-31T19:35:32.651 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:32.661 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:32.662 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:32.708 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:32.711 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:32.760 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:32.760 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:32.840 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:16.2.7-278.g9e931647.e 1/1 2022-01-31T19:35:33.142 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:17 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:33.431 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.7-278.g9e931647.e 1/1 2022-01-31T19:35:33.793 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:16.2.7-278.g9e931647.e 1/1 2022-01-31T19:35:33.794 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:33.794 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:33.794 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:33.794 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:33.795 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:33.930 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:33.931 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:33.931 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2022-01-31T19:35:33.931 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:33.931 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:35:33.932 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:16.2.7-278.g9e931647.el8 ceph 161 k 2022-01-31T19:35:33.932 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:33.932 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:35:33.932 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:33.933 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:35:33.933 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:33.933 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 161 k 2022-01-31T19:35:33.933 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 471 k 2022-01-31T19:35:33.934 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:35:33.960 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr 2022-01-31T19:35:34.155 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:34.155 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-16.2.7-278.g9e93164 526 kB/s | 161 kB 00:00 2022-01-31T19:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout:Total 524 kB/s | 161 kB 00:00 2022-01-31T19:35:34.240 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:34.250 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:34.250 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:34.297 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:34.300 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:34.409 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:34.539 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:17 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:34.621 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:16.2.7-278.g9e931647.e 1/1 2022-01-31T19:35:34.995 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.7-278.g9e931647.e 1/1 2022-01-31T19:35:35.277 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:35:35.321 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:35.322 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:35:35.322 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:35.379 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:16.2.7-278.g9e931647.e 1/1 2022-01-31T19:35:35.379 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:35.379 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:35:35.380 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:35:35.380 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:35.380 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:35.386 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-dashboard 2022-01-31T19:35:35.563 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:35.564 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:35.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr 2022-01-31T19:35:35.761 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:35.762 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:35.943 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:18 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:36.143 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:20 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:36.740 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:36.740 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:36.741 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:35:36.741 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:36.741 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:35:36.742 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 2.9 M 2022-01-31T19:35:36.742 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:35:36.742 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 25 k 2022-01-31T19:35:36.742 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 15 k 2022-01-31T19:35:36.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-01-31T19:35:36.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-01-31T19:35:36.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-01-31T19:35:36.743 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:36.743 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:35:36.744 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:35:36.744 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2022-01-31T19:35:36.744 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:36.745 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.2 M 2022-01-31T19:35:36.745 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 15 M 2022-01-31T19:35:36.746 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:35:36.903 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-mgr-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:35:36.948 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:36.949 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:35:36.950 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:36.974 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): ceph-prometheus-alerts-16.2.7-278.g9e931 65 kB/s | 15 kB 00:00 2022-01-31T19:35:36.990 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): ceph-grafana-dashboards-16.2.7-278.g9e93 102 kB/s | 25 kB 00:00 2022-01-31T19:35:37.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-dashboard 2022-01-31T19:35:37.031 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 726 kB/s | 43 kB 00:00 2022-01-31T19:35:37.090 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2022-01-31T19:35:37.204 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:37.205 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:37.207 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2022-01-31T19:35:37.274 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): ceph-mgr-dashboard-16.2.7-278.g9e931647. 5.4 MB/s | 2.9 MB 00:00 2022-01-31T19:35:37.276 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:37.276 INFO:teuthology.orchestra.run.smithi171.stdout:Total 6.0 MB/s | 3.2 MB 00:00 2022-01-31T19:35:37.287 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:37.297 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:37.298 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:37.407 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:37.410 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:37.591 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:21 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:37.671 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:37.894 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-01-31T19:35:38.114 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-01-31T19:35:38.324 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-01-31T19:35:38.374 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:38.381 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:38.381 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:35:38.381 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:38.381 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:35:38.382 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 2.9 M 2022-01-31T19:35:38.382 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:35:38.382 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 25 k 2022-01-31T19:35:38.382 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 15 k 2022-01-31T19:35:38.383 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-01-31T19:35:38.383 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-01-31T19:35:38.383 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-01-31T19:35:38.383 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:38.384 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:35:38.384 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:35:38.384 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2022-01-31T19:35:38.384 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:38.385 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.2 M 2022-01-31T19:35:38.385 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 15 M 2022-01-31T19:35:38.385 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:35:38.437 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:16.2.7-278.g9e931647.el8.no 4/6 2022-01-31T19:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): ceph-grafana-dashboards-16.2.7-278.g9e93 105 kB/s | 25 kB 00:00 2022-01-31T19:35:38.642 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): ceph-prometheus-alerts-16.2.7-278.g9e931 57 kB/s | 15 kB 00:00 2022-01-31T19:35:38.692 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 566 kB/s | 43 kB 00:00 2022-01-31T19:35:38.768 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 269 kB/s | 34 kB 00:00 2022-01-31T19:35:38.876 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2022-01-31T19:35:38.926 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): ceph-mgr-dashboard-16.2.7-278.g9e931647. 5.3 MB/s | 2.9 MB 00:00 2022-01-31T19:35:38.928 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:38.929 INFO:teuthology.orchestra.run.smithi167.stdout:Total 5.8 MB/s | 3.2 MB 00:00 2022-01-31T19:35:38.940 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:38.949 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:38.949 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:39.066 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:39.069 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:39.375 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:16.2.7-278.g9e931647.el8.n 5/6 2022-01-31T19:35:40.039 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-01-31T19:35:40.869 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:35:41.404 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:35:41.404 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:16.2.7-278.g9e931647.el8.n 1/6 2022-01-31T19:35:41.405 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 2/6 2022-01-31T19:35:41.405 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:16.2.7-278.g9e931647.el8.no 3/6 2022-01-31T19:35:41.405 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-01-31T19:35:41.405 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-01-31T19:35:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-01-31T19:35:41.752 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-01-31T19:35:41.752 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:41.752 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:41.752 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:41.753 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:41.753 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:41.753 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-01-31T19:35:41.754 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-01-31T19:35:41.754 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.4.1-12.el8.noarch 2022-01-31T19:35:41.754 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:41.754 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:41.834 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-diskprediction-local 2022-01-31T19:35:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-01-31T19:35:42.014 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:42.014 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:42.061 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:16.2.7-278.g9e931647.el8.no 4/6 2022-01-31T19:35:42.394 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:25 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:43.105 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:16.2.7-278.g9e931647.el8.n 5/6 2022-01-31T19:35:43.209 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:43.210 INFO:teuthology.orchestra.run.smithi171.stdout:===================================================================================================== 2022-01-31T19:35:43.211 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:35:43.211 INFO:teuthology.orchestra.run.smithi171.stdout:===================================================================================================== 2022-01-31T19:35:43.211 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:35:43.212 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 8.6 M 2022-01-31T19:35:43.212 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:35:43.212 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 8.5.0-8.el8 CentOS-Base 643 k 2022-01-31T19:35:43.212 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 8.5.0-8.el8 CentOS-Base 171 k 2022-01-31T19:35:43.213 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-01-31T19:35:43.213 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-01-31T19:35:43.213 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-01-31T19:35:43.213 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-01-31T19:35:43.214 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-01-31T19:35:43.214 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:43.214 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:35:43.215 INFO:teuthology.orchestra.run.smithi171.stdout:===================================================================================================== 2022-01-31T19:35:43.215 INFO:teuthology.orchestra.run.smithi171.stdout:Install 8 Packages 2022-01-31T19:35:43.215 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:43.216 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 37 M 2022-01-31T19:35:43.216 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 210 M 2022-01-31T19:35:43.217 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:35:43.375 INFO:teuthology.orchestra.run.smithi171.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2022-01-31T19:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:35:43.483 INFO:teuthology.orchestra.run.smithi171.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 35 MB/s | 3.7 MB 00:00 2022-01-31T19:35:43.517 INFO:teuthology.orchestra.run.smithi171.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-01-31T19:35:43.626 INFO:teuthology.orchestra.run.smithi171.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2022-01-31T19:35:43.677 INFO:teuthology.orchestra.run.smithi171.stdout:(5/8): libgfortran-8.5.0-8.el8.x86_64.rpm 13 MB/s | 643 kB 00:00 2022-01-31T19:35:43.683 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:35:43.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:16.2.7-278.g9e931647.el8.n 1/6 2022-01-31T19:35:43.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 2/6 2022-01-31T19:35:43.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:16.2.7-278.g9e931647.el8.no 3/6 2022-01-31T19:35:43.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-01-31T19:35:43.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-01-31T19:35:43.726 INFO:teuthology.orchestra.run.smithi171.stdout:(6/8): libquadmath-8.5.0-8.el8.x86_64.rpm 3.4 MB/s | 171 kB 00:00 2022-01-31T19:35:43.910 INFO:teuthology.orchestra.run.smithi171.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 35 MB/s | 14 MB 00:00 2022-01-31T19:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-01-31T19:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-01-31T19:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-01-31T19:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.4.1-12.el8.noarch 2022-01-31T19:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:44.143 INFO:teuthology.orchestra.run.smithi171.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.7-278 9.2 MB/s | 8.6 MB 00:00 2022-01-31T19:35:44.146 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:44.146 INFO:teuthology.orchestra.run.smithi171.stdout:Total 39 MB/s | 37 MB 00:00 2022-01-31T19:35:44.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-diskprediction-local 2022-01-31T19:35:44.331 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:44.331 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:44.381 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:44.400 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:44.401 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:44.708 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:28 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:44.716 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:44.719 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:45.294 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:45.437 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-8.5.0-8.el8.x86_64 1/8 2022-01-31T19:35:45.495 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:45.496 INFO:teuthology.orchestra.run.smithi167.stdout:===================================================================================================== 2022-01-31T19:35:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:35:45.496 INFO:teuthology.orchestra.run.smithi167.stdout:===================================================================================================== 2022-01-31T19:35:45.497 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:35:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 8.6 M 2022-01-31T19:35:45.498 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:35:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 8.5.0-8.el8 CentOS-Base 643 k 2022-01-31T19:35:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 8.5.0-8.el8 CentOS-Base 171 k 2022-01-31T19:35:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-01-31T19:35:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-01-31T19:35:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-01-31T19:35:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-01-31T19:35:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-01-31T19:35:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:45.500 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:35:45.501 INFO:teuthology.orchestra.run.smithi167.stdout:===================================================================================================== 2022-01-31T19:35:45.501 INFO:teuthology.orchestra.run.smithi167.stdout:Install 8 Packages 2022-01-31T19:35:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:45.502 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 37 M 2022-01-31T19:35:45.503 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 210 M 2022-01-31T19:35:45.503 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:35:45.575 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libquadmath-8.5.0-8.el8.x86_64 1/8 2022-01-31T19:35:45.691 INFO:teuthology.orchestra.run.smithi167.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2022-01-31T19:35:45.721 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-8.5.0-8.el8.x86_64 2/8 2022-01-31T19:35:45.800 INFO:teuthology.orchestra.run.smithi167.stdout:(2/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2022-01-31T19:35:45.884 INFO:teuthology.orchestra.run.smithi167.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2022-01-31T19:35:45.909 INFO:teuthology.orchestra.run.smithi167.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2022-01-31T19:35:45.967 INFO:teuthology.orchestra.run.smithi167.stdout:(5/8): libgfortran-8.5.0-8.el8.x86_64.rpm 11 MB/s | 643 kB 00:00 2022-01-31T19:35:46.001 INFO:teuthology.orchestra.run.smithi167.stdout:(6/8): libquadmath-8.5.0-8.el8.x86_64.rpm 5.0 MB/s | 171 kB 00:00 2022-01-31T19:35:46.226 INFO:teuthology.orchestra.run.smithi167.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 40 MB/s | 14 MB 00:00 2022-01-31T19:35:46.319 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libgfortran-8.5.0-8.el8.x86_64 2/8 2022-01-31T19:35:46.452 INFO:teuthology.orchestra.run.smithi167.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.7-278 9.0 MB/s | 8.6 MB 00:00 2022-01-31T19:35:46.454 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:46.454 INFO:teuthology.orchestra.run.smithi167.stdout:Total 38 MB/s | 37 MB 00:00 2022-01-31T19:35:46.529 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-01-31T19:35:46.690 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:46.708 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:46.709 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:47.025 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:47.028 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:47.151 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-01-31T19:35:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-01-31T19:35:47.599 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-8.5.0-8.el8.x86_64 1/8 2022-01-31T19:35:47.898 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libquadmath-8.5.0-8.el8.x86_64 1/8 2022-01-31T19:35:48.038 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-01-31T19:35:48.076 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-8.5.0-8.el8.x86_64 2/8 2022-01-31T19:35:48.365 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-01-31T19:35:48.595 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-01-31T19:35:48.676 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libgfortran-8.5.0-8.el8.x86_64 2/8 2022-01-31T19:35:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-01-31T19:35:49.512 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-01-31T19:35:49.748 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-01-31T19:35:50.380 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-01-31T19:35:50.424 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-01-31T19:35:50.779 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-01-31T19:35:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-01-31T19:35:51.669 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-01-31T19:35:52.103 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647 8/8 2022-01-31T19:35:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647 8/8 2022-01-31T19:35:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647 1/8 2022-01-31T19:35:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-01-31T19:35:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-01-31T19:35:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-01-31T19:35:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-01-31T19:35:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-01-31T19:35:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-8.5.0-8.el8.x86_64 7/8 2022-01-31T19:35:52.727 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-01-31T19:35:52.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-8.5.0-8.el8.x86_64 8/8 2022-01-31T19:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-8.5.0-8.el8.x86_64 2022-01-31T19:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-8.5.0-8.el8.x86_64 2022-01-31T19:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.15-3.el8.x86_64 2022-01-31T19:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-01-31T19:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-01-31T19:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-01-31T19:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-01-31T19:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:52.781 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:52.874 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-rook 2022-01-31T19:35:53.052 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:53.052 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:53.434 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:36 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:35:54.027 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-01-31T19:35:54.220 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:35:54.221 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2022-01-31T19:35:54.222 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:35:54.222 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2022-01-31T19:35:54.222 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:35:54.222 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 27 k 2022-01-31T19:35:54.223 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:35:54.223 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-01-31T19:35:54.223 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-01-31T19:35:54.223 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-01-31T19:35:54.224 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-01-31T19:35:54.224 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-01-31T19:35:54.224 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-01-31T19:35:54.225 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-01-31T19:35:54.225 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-01-31T19:35:54.225 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-01-31T19:35:54.225 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-01-31T19:35:54.226 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-01-31T19:35:54.226 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-01-31T19:35:54.226 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:54.226 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:35:54.227 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2022-01-31T19:35:54.227 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2022-01-31T19:35:54.227 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:54.228 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 1.9 M 2022-01-31T19:35:54.228 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 24 M 2022-01-31T19:35:54.228 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:35:54.267 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647 8/8 2022-01-31T19:35:54.292 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 395 kB/s | 27 kB 00:00 2022-01-31T19:35:54.326 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.3 MB/s | 126 kB 00:00 2022-01-31T19:35:54.360 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 148 kB/s | 20 kB 00:00 2022-01-31T19:35:54.385 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.8 MB/s | 110 kB 00:00 2022-01-31T19:35:54.411 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2022-01-31T19:35:54.436 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2022-01-31T19:35:54.503 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): ceph-mgr-rook-16.2.7-278.g9e931647.el8. 98 kB/s | 27 kB 00:00 2022-01-31T19:35:54.537 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 162 kB/s | 16 kB 00:00 2022-01-31T19:35:54.554 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 231 kB/s | 33 kB 00:00 2022-01-31T19:35:54.592 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647 8/8 2022-01-31T19:35:54.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647 1/8 2022-01-31T19:35:54.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-01-31T19:35:54.593 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-01-31T19:35:54.594 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-01-31T19:35:54.594 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-01-31T19:35:54.594 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-01-31T19:35:54.594 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-8.5.0-8.el8.x86_64 7/8 2022-01-31T19:35:54.648 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): python3-rsa-4.8-1.el8.noarch.rpm 654 kB/s | 61 kB 00:00 2022-01-31T19:35:54.671 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 485 kB/s | 81 kB 00:00 2022-01-31T19:35:54.706 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.1 MB/s | 61 kB 00:00 2022-01-31T19:35:54.763 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.2 MB/s | 1.2 MB 00:00 2022-01-31T19:35:54.765 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:54.766 INFO:teuthology.orchestra.run.smithi171.stdout:Total 3.5 MB/s | 1.9 MB 00:00 2022-01-31T19:35:54.825 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:35:54.836 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:35:54.836 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:35:54.914 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:35:54.917 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:35:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-8.5.0-8.el8.x86_64 8/8 2022-01-31T19:35:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:55.009 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:35:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-8.5.0-8.el8.x86_64 2022-01-31T19:35:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-8.5.0-8.el8.x86_64 2022-01-31T19:35:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.15-3.el8.x86_64 2022-01-31T19:35:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-01-31T19:35:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-01-31T19:35:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-01-31T19:35:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-01-31T19:35:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:55.012 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:35:55.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-rook 2022-01-31T19:35:55.286 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:55.287 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:35:55.421 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:35:55.638 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-01-31T19:35:55.670 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:39 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:35:55.803 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-01-31T19:35:56.029 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-01-31T19:35:56.201 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-01-31T19:35:56.374 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-01-31T19:35:56.477 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:35:56.478 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2022-01-31T19:35:56.479 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:35:56.479 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2022-01-31T19:35:56.479 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:35:56.480 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 27 k 2022-01-31T19:35:56.480 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:35:56.480 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-01-31T19:35:56.480 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-01-31T19:35:56.480 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-01-31T19:35:56.481 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-01-31T19:35:56.481 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-01-31T19:35:56.481 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-01-31T19:35:56.481 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-01-31T19:35:56.482 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-01-31T19:35:56.482 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-01-31T19:35:56.482 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-01-31T19:35:56.482 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-01-31T19:35:56.483 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-01-31T19:35:56.483 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:56.483 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:35:56.483 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2022-01-31T19:35:56.484 INFO:teuthology.orchestra.run.smithi167.stdout:Install 13 Packages 2022-01-31T19:35:56.484 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:35:56.485 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.9 M 2022-01-31T19:35:56.485 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 24 M 2022-01-31T19:35:56.486 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 424 kB/s | 27 kB 00:00 2022-01-31T19:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.3 MB/s | 126 kB 00:00 2022-01-31T19:35:56.588 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-01-31T19:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 164 kB/s | 20 kB 00:00 2022-01-31T19:35:56.629 INFO:teuthology.orchestra.run.smithi167.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-01-31T19:35:56.655 INFO:teuthology.orchestra.run.smithi167.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2022-01-31T19:35:56.680 INFO:teuthology.orchestra.run.smithi167.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 846 kB/s | 43 kB 00:00 2022-01-31T19:35:56.747 INFO:teuthology.orchestra.run.smithi167.stdout:(7/13): ceph-mgr-rook-16.2.7-278.g9e931647.el8. 103 kB/s | 27 kB 00:00 2022-01-31T19:35:56.814 INFO:teuthology.orchestra.run.smithi167.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2022-01-31T19:35:56.839 INFO:teuthology.orchestra.run.smithi167.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2022-01-31T19:35:56.880 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-01-31T19:35:56.931 INFO:teuthology.orchestra.run.smithi167.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 441 kB/s | 81 kB 00:00 2022-01-31T19:35:57.056 INFO:teuthology.orchestra.run.smithi167.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 284 kB/s | 61 kB 00:00 2022-01-31T19:35:57.349 INFO:teuthology.orchestra.run.smithi167.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 147 kB/s | 61 kB 00:00 2022-01-31T19:35:57.350 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-01-31T19:35:57.608 INFO:teuthology.orchestra.run.smithi167.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.5 MB/s | 1.2 MB 00:00 2022-01-31T19:35:57.610 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:35:57.610 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2022-01-31T19:35:57.675 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:35:57.692 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:35:57.692 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:35:57.772 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:35:57.775 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:35:58.355 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:35:58.380 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-01-31T19:35:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-01-31T19:35:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-01-31T19:35:58.801 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-01-31T19:35:58.875 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-01-31T19:35:59.035 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-01-31T19:35:59.067 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-01-31T19:35:59.235 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 13/13 2022-01-31T19:35:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-01-31T19:35:59.447 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-01-31T19:35:59.491 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 13/13 2022-01-31T19:35:59.492 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 1/13 2022-01-31T19:35:59.492 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-01-31T19:35:59.492 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-01-31T19:35:59.493 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-01-31T19:35:59.493 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-01-31T19:35:59.493 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-01-31T19:35:59.493 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-01-31T19:35:59.494 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-01-31T19:35:59.494 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-01-31T19:35:59.494 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-01-31T19:35:59.494 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-01-31T19:35:59.495 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-01-31T19:35:59.645 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-01-31T19:35:59.933 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-01-31T19:35:59.934 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:59.934 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:35:59.934 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:35:59.935 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-01-31T19:35:59.935 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-01-31T19:35:59.935 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-01-31T19:35:59.935 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-01-31T19:35:59.935 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-01-31T19:35:59.936 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-01-31T19:35:59.936 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-01-31T19:35:59.936 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-01-31T19:35:59.936 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-01-31T19:35:59.937 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-01-31T19:35:59.937 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.8-1.el8.noarch 2022-01-31T19:35:59.937 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-01-31T19:35:59.938 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:35:59.938 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:35:59.938 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-01-31T19:36:00.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-cephadm 2022-01-31T19:36:00.163 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-01-31T19:36:00.183 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:00.184 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:00.567 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:43 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:01.231 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-01-31T19:36:01.352 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:01.353 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:01.353 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:36:01.354 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:01.354 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:01.354 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 106 k 2022-01-31T19:36:01.355 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:36:01.355 INFO:teuthology.orchestra.run.smithi171.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2022-01-31T19:36:01.356 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-01-31T19:36:01.356 INFO:teuthology.orchestra.run.smithi171.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2022-01-31T19:36:01.356 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-01-31T19:36:01.356 INFO:teuthology.orchestra.run.smithi171.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2022-01-31T19:36:01.357 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:01.357 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:01.357 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:01.357 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2022-01-31T19:36:01.358 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:01.358 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 5.8 M 2022-01-31T19:36:01.359 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 24 M 2022-01-31T19:36:01.359 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:01.442 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.1 MB/s | 538 kB 00:00 2022-01-31T19:36:01.551 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2022-01-31T19:36:01.573 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-01-31T19:36:01.576 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2022-01-31T19:36:01.660 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 585 kB/s | 49 kB 00:00 2022-01-31T19:36:01.711 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): ceph-mgr-cephadm-16.2.7-278.g9e931647.el 298 kB/s | 106 kB 00:00 2022-01-31T19:36:01.761 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.7 MB/s | 355 kB 00:00 2022-01-31T19:36:01.762 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:01.762 INFO:teuthology.orchestra.run.smithi171.stdout:Total 14 MB/s | 5.8 MB 00:00 2022-01-31T19:36:01.790 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-01-31T19:36:01.814 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:01.825 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:01.825 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:01.919 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:01.922 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:02.032 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-01-31T19:36:02.204 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:02.232 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 13/13 2022-01-31T19:36:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 13/13 2022-01-31T19:36:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 1/13 2022-01-31T19:36:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-01-31T19:36:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-01-31T19:36:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-01-31T19:36:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-01-31T19:36:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-01-31T19:36:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-01-31T19:36:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-01-31T19:36:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-01-31T19:36:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-01-31T19:36:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-01-31T19:36:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-01-31T19:36:02.488 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2022-01-31T19:36:02.960 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-01-31T19:36:02.961 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:02.961 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:02.961 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:36:02.961 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-01-31T19:36:02.962 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-01-31T19:36:02.962 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-01-31T19:36:02.962 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-01-31T19:36:02.962 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-01-31T19:36:02.963 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-01-31T19:36:02.963 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-01-31T19:36:02.963 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-01-31T19:36:02.963 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-01-31T19:36:02.964 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-01-31T19:36:02.964 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.8-1.el8.noarch 2022-01-31T19:36:02.964 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-01-31T19:36:02.964 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:02.965 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:03.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-cephadm 2022-01-31T19:36:03.237 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:03.237 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:03.525 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2022-01-31T19:36:03.624 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:47 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:04.408 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:04.409 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:36:04.409 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:04.409 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:16.2.7-278.g9e931647.el8 ceph-noarch 106 k 2022-01-31T19:36:04.410 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:36:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2022-01-31T19:36:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-01-31T19:36:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2022-01-31T19:36:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-01-31T19:36:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2022-01-31T19:36:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:04.412 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:04.412 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:04.412 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2022-01-31T19:36:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:04.413 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 5.8 M 2022-01-31T19:36:04.413 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 24 M 2022-01-31T19:36:04.414 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:04.454 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2022-01-31T19:36:04.500 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.0 MB/s | 538 kB 00:00 2022-01-31T19:36:04.625 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2022-01-31T19:36:04.651 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 145 kB/s | 21 kB 00:00 2022-01-31T19:36:04.734 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 586 kB/s | 49 kB 00:00 2022-01-31T19:36:04.818 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.8 MB/s | 355 kB 00:00 2022-01-31T19:36:04.830 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2022-01-31T19:36:04.951 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): ceph-mgr-cephadm-16.2.7-278.g9e931647.el 196 kB/s | 106 kB 00:00 2022-01-31T19:36:04.953 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:04.953 INFO:teuthology.orchestra.run.smithi167.stdout:Total 11 MB/s | 5.8 MB 00:00 2022-01-31T19:36:05.006 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:05.016 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:05.016 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:05.057 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2022-01-31T19:36:05.112 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:05.115 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:05.234 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:36:05.424 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:05.496 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:36:05.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/6 2022-01-31T19:36:05.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2022-01-31T19:36:05.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2022-01-31T19:36:05.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2022-01-31T19:36:05.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2022-01-31T19:36:05.690 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2022-01-31T19:36:05.849 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2022-01-31T19:36:05.850 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:05.850 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:05.850 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:36:05.851 INFO:teuthology.orchestra.run.smithi171.stdout: python3-apipkg-1.5-6.el8.noarch 2022-01-31T19:36:05.851 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.5.1-7.el8.noarch 2022-01-31T19:36:05.851 INFO:teuthology.orchestra.run.smithi171.stdout: python3-execnet-1.7.1-1.el8.noarch 2022-01-31T19:36:05.851 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-01-31T19:36:05.852 INFO:teuthology.orchestra.run.smithi171.stdout: python3-remoto-1.2.1-1.el8.noarch 2022-01-31T19:36:05.852 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:05.852 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:05.925 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse 2022-01-31T19:36:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2022-01-31T19:36:06.103 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:06.104 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:06.485 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:49 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:06.965 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2022-01-31T19:36:07.252 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:07.254 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:07.254 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2022-01-31T19:36:07.254 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:07.254 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:07.255 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:16.2.7-278.g9e931647.el8 ceph 896 k 2022-01-31T19:36:07.255 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2022-01-31T19:36:07.255 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2022-01-31T19:36:07.256 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:07.256 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:07.257 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:07.257 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2022-01-31T19:36:07.257 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:07.258 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 978 k 2022-01-31T19:36:07.258 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.8 M 2022-01-31T19:36:07.259 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:07.309 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.5 MB/s | 82 kB 00:00 2022-01-31T19:36:07.335 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2022-01-31T19:36:07.563 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2022-01-31T19:36:07.650 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-fuse-16.2.7-278.g9e931647.el8.x86_6 2.2 MB/s | 896 kB 00:00 2022-01-31T19:36:07.651 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:07.651 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.4 MB/s | 978 kB 00:00 2022-01-31T19:36:07.662 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:07.688 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:07.688 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:07.767 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:07.770 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:07.773 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:36:07.930 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:08.041 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 6/6 2022-01-31T19:36:08.042 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 1/6 2022-01-31T19:36:08.042 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2022-01-31T19:36:08.043 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2022-01-31T19:36:08.043 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2022-01-31T19:36:08.043 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2022-01-31T19:36:08.189 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2022-01-31T19:36:08.391 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2022-01-31T19:36:08.391 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:08.392 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:08.392 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:16.2.7-278.g9e931647.el8.noarch 2022-01-31T19:36:08.392 INFO:teuthology.orchestra.run.smithi167.stdout: python3-apipkg-1.5-6.el8.noarch 2022-01-31T19:36:08.392 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.5.1-7.el8.noarch 2022-01-31T19:36:08.393 INFO:teuthology.orchestra.run.smithi167.stdout: python3-execnet-1.7.1-1.el8.noarch 2022-01-31T19:36:08.393 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-01-31T19:36:08.393 INFO:teuthology.orchestra.run.smithi167.stdout: python3-remoto-1.2.1-1.el8.noarch 2022-01-31T19:36:08.394 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:08.394 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:08.395 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 2/2 2022-01-31T19:36:08.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse 2022-01-31T19:36:08.670 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:08.670 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:08.750 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 2/2 2022-01-31T19:36:08.750 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/2 2022-01-31T19:36:09.054 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:53 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:09.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2022-01-31T19:36:09.192 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:09.192 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:09.192 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2022-01-31T19:36:09.193 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:09.193 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:09.331 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel 2022-01-31T19:36:09.515 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:09.515 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:09.833 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:09.833 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2022-01-31T19:36:09.834 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:09.834 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:16.2.7-278.g9e931647.el8 ceph 896 k 2022-01-31T19:36:09.834 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2022-01-31T19:36:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2022-01-31T19:36:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:09.835 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:09.835 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:09.835 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2022-01-31T19:36:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:09.836 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 978 k 2022-01-31T19:36:09.836 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.8 M 2022-01-31T19:36:09.836 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:09.911 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.1 MB/s | 82 kB 00:00 2022-01-31T19:36:09.912 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:52 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:10.494 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-fuse-16.2.7-278.g9e931647.el8.x86_6 1.3 MB/s | 896 kB 00:00 2022-01-31T19:36:10.495 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:10.496 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.4 MB/s | 978 kB 00:00 2022-01-31T19:36:10.507 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:10.529 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:10.529 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:10.610 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:10.613 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:10.723 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:10.724 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:10.725 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2022-01-31T19:36:10.725 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:10.725 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:10.725 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:16.2.7-278.g9e931647.el8 ceph 131 k 2022-01-31T19:36:10.726 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:10.726 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:10.726 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:10.726 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:36:10.727 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:10.727 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 131 k 2022-01-31T19:36:10.727 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 432 k 2022-01-31T19:36:10.727 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:11.096 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2022-01-31T19:36:11.183 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-16.2.7-278.g9e931647.el8.x86_64. 287 kB/s | 131 kB 00:00 2022-01-31T19:36:11.184 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:11.184 INFO:teuthology.orchestra.run.smithi171.stdout:Total 286 kB/s | 131 kB 00:00 2022-01-31T19:36:11.184 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:11.190 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:11.190 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:11.240 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:11.242 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:11.314 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 2/2 2022-01-31T19:36:11.350 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:11.549 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 2/2 2022-01-31T19:36:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/2 2022-01-31T19:36:11.804 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2022-01-31T19:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:16.2.7-278.g9e931647.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2022-01-31T19:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:12.188 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:12.189 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:12.190 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:12.190 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:12.190 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:12.190 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:12.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel 2022-01-31T19:36:12.386 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2 2022-01-31T19:36:12.491 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:12.491 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:12.566 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:12.566 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:12.881 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:56 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:12.954 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:55 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:13.660 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:13.661 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:13.661 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2022-01-31T19:36:13.662 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:13.662 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:16.2.7-278.g9e931647.el8 ceph 131 k 2022-01-31T19:36:13.663 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:13.663 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:13.663 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:13.664 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:36:13.664 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:13.664 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 131 k 2022-01-31T19:36:13.664 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 432 k 2022-01-31T19:36:13.664 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:13.692 INFO:teuthology.orchestra.run.smithi171.stdout:Package libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:13.736 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:13.737 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:13.738 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:13.818 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs-devel 2022-01-31T19:36:14.002 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:14.002 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:14.052 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-16.2.7-278.g9e931647.el8.x86_64. 337 kB/s | 131 kB 00:00 2022-01-31T19:36:14.053 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:14.053 INFO:teuthology.orchestra.run.smithi167.stdout:Total 335 kB/s | 131 kB 00:00 2022-01-31T19:36:14.054 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:14.060 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:14.060 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:14.111 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:14.114 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:14.244 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:14.383 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:57 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:14.730 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:15.106 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:15.106 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:15.106 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:15.107 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:15.107 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:15.107 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:15.160 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:15.162 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:15.162 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2022-01-31T19:36:15.162 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:15.163 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:15.163 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:16.2.7-278.g9e931647.el8 ceph 25 k 2022-01-31T19:36:15.163 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:15.164 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:15.164 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:15.165 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:36:15.165 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:15.165 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 25 k 2022-01-31T19:36:15.166 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 104 k 2022-01-31T19:36:15.166 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:15.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2 2022-01-31T19:36:15.475 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs-devel-16.2.7-278.g9e931647.el8.x86_64 80 kB/s | 25 kB 00:00 2022-01-31T19:36:15.476 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:15.476 INFO:teuthology.orchestra.run.smithi171.stdout:Total 80 kB/s | 25 kB 00:00 2022-01-31T19:36:15.477 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:15.480 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:15.480 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:15.482 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:15.485 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:15.486 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:15.486 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:15.535 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:15.723 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:15.870 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:59 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:15.961 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:16.271 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:16.272 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:16.272 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:16.272 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:16.272 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:16.273 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:16.358 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2 2022-01-31T19:36:16.537 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:16.538 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:16.619 INFO:teuthology.orchestra.run.smithi167.stdout:Package libcephfs2-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:16.665 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:16.666 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:16.666 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:16.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs-devel 2022-01-31T19:36:16.922 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:16.922 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:16.923 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:59 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:17.309 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:01 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:17.690 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:18.503 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:18.504 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:18.504 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:18.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1 2022-01-31T19:36:18.507 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:18.507 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:18.507 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2022-01-31T19:36:18.508 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:18.508 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:18.508 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:16.2.7-278.g9e931647.el8 ceph 25 k 2022-01-31T19:36:18.508 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:18.509 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:18.509 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:18.509 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:36:18.509 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:18.510 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 25 k 2022-01-31T19:36:18.510 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 104 k 2022-01-31T19:36:18.510 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:18.510 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-devel-16.2.7-278.g9e931647.el8.x86_64 116 kB/s | 25 kB 00:00 2022-01-31T19:36:18.510 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:18.511 INFO:teuthology.orchestra.run.smithi167.stdout:Total 115 kB/s | 25 kB 00:00 2022-01-31T19:36:18.511 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:18.511 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:18.511 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:18.512 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:18.512 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:18.512 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:18.613 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:18.692 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:18.693 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:18.834 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:19.079 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:02 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:19.703 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:19.704 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:19.704 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:19.704 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:19.705 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:19.705 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:19.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2 2022-01-31T19:36:19.836 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:19.881 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:19.882 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:19.883 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:19.965 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:19.966 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:20.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rados 2022-01-31T19:36:20.349 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:04 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:20.422 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:20.423 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:20.806 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:03 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:21.092 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:21.137 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:21.138 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:21.139 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:21.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1 2022-01-31T19:36:21.423 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:21.545 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:21.589 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:21.590 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:21.590 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:21.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rgw 2022-01-31T19:36:21.810 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:05 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:21.836 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:21.836 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:22.219 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:05 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:22.557 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:22.602 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:22.603 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:22.603 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:22.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados 2022-01-31T19:36:22.858 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:22.859 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:22.976 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:23.021 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:23.024 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:23.024 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:23.093 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-cephfs 2022-01-31T19:36:23.239 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:07 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:23.272 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:23.273 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:23.656 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:06 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:23.960 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rados-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:24.004 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:24.005 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:24.006 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:24.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw 2022-01-31T19:36:24.257 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:24.257 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:24.413 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:24.459 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:24.460 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:24.460 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:24.535 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rbd 2022-01-31T19:36:24.640 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:08 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:24.714 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:24.715 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:25.095 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:08 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:25.378 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rgw-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:25.423 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:25.424 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:25.424 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:25.864 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:25.907 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:25.909 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2022-01-31T19:36:25.909 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:25.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs 2022-01-31T19:36:25.970 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse 2022-01-31T19:36:26.104 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:26.149 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:26.149 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:26.489 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:10 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:26.534 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:09 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:27.229 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-cephfs-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:27.273 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:27.274 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:27.274 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:27.347 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:27.349 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:27.349 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2022-01-31T19:36:27.349 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:27.350 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:27.350 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:16.2.7-278.g9e931647.el8 ceph 88 k 2022-01-31T19:36:27.350 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:27.351 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:27.351 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:27.351 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:36:27.352 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:27.352 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 88 k 2022-01-31T19:36:27.352 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 236 k 2022-01-31T19:36:27.352 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:27.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd 2022-01-31T19:36:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:27.592 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-16.2.7-278.g9e931647.el8.x86_64.rpm 363 kB/s | 88 kB 00:00 2022-01-31T19:36:27.593 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:27.593 INFO:teuthology.orchestra.run.smithi171.stdout:Total 361 kB/s | 88 kB 00:00 2022-01-31T19:36:27.593 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:27.602 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:27.602 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:27.651 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:27.653 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:27.775 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:27.920 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:11 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:27.963 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:28.225 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:28.577 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:28.578 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:28.578 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:28.578 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:28.578 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:28.579 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:28.634 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rbd-2:16.2.7-278.g9e931647.el8.x86_64 is already installed. 2022-01-31T19:36:28.679 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:28.680 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2022-01-31T19:36:28.680 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:28.746 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror 2022-01-31T19:36:28.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse 2022-01-31T19:36:28.925 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:28.926 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:28.934 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:28.934 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:29.318 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:13 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:29.321 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:12 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:30.127 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:30.128 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2022-01-31T19:36:30.128 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:30.129 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:16.2.7-278.g9e931647.el8 ceph 3.7 M 2022-01-31T19:36:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:30.129 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:30.130 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:30.130 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:36:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:30.130 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.7 M 2022-01-31T19:36:30.131 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 15 M 2022-01-31T19:36:30.131 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:30.140 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:30.141 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:30.141 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2022-01-31T19:36:30.141 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:30.142 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:30.142 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:16.2.7-278.g9e931647.el8 ceph 88 k 2022-01-31T19:36:30.142 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:30.142 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:30.143 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:30.143 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:36:30.143 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:30.143 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 88 k 2022-01-31T19:36:30.144 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 236 k 2022-01-31T19:36:30.144 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:30.379 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-16.2.7-278.g9e931647.el8.x86_64.rpm 370 kB/s | 88 kB 00:00 2022-01-31T19:36:30.380 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:30.381 INFO:teuthology.orchestra.run.smithi167.stdout:Total 368 kB/s | 88 kB 00:00 2022-01-31T19:36:30.381 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:30.390 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:30.391 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:30.439 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:30.441 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:30.583 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:30.680 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-16.2.7-278.g9e931647.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2022-01-31T19:36:30.681 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:30.682 INFO:teuthology.orchestra.run.smithi171.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2022-01-31T19:36:30.682 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:30.693 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:30.694 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:30.756 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:30.759 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:30.859 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:31.226 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:31.335 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:31.463 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:31.746 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:31.747 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:31.747 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:31.748 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:31.748 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:31.748 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:31.847 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:31.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror 2022-01-31T19:36:32.127 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:32.128 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:32.196 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:32.196 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:32.197 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:32.197 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:32.197 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:32.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd 2022-01-31T19:36:32.519 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:16 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:32.528 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:32.529 INFO:teuthology.orchestra.run.smithi171.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:32.917 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:15 ago on Mon 31 Jan 2022 07:34:17 PM UTC. 2022-01-31T19:36:33.307 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:33.309 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2022-01-31T19:36:33.310 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:33.310 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:16.2.7-278.g9e931647.el8 ceph 3.7 M 2022-01-31T19:36:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:33.312 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:33.312 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:33.312 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:36:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:33.314 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.7 M 2022-01-31T19:36:33.314 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 15 M 2022-01-31T19:36:33.315 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:33.697 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2022-01-31T19:36:33.698 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:33.698 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2022-01-31T19:36:33.698 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:33.699 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2022-01-31T19:36:33.699 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:16.2.7-278.g9e931647.el8 ceph 172 k 2022-01-31T19:36:33.699 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:33.699 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2022-01-31T19:36:33.699 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2022-01-31T19:36:33.700 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2022-01-31T19:36:33.700 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:33.700 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 172 k 2022-01-31T19:36:33.700 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 511 k 2022-01-31T19:36:33.701 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2022-01-31T19:36:33.836 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-16.2.7-278.g9e931647.el8.x86_64.rpm 7.0 MB/s | 3.7 MB 00:00 2022-01-31T19:36:33.837 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:33.837 INFO:teuthology.orchestra.run.smithi167.stdout:Total 6.9 MB/s | 3.7 MB 00:00 2022-01-31T19:36:33.838 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:33.849 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:33.850 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:33.913 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:33.915 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:33.974 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-16.2.7-278.g9e931647.el8.x86_64.rpm 624 kB/s | 172 kB 00:00 2022-01-31T19:36:33.975 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:33.975 INFO:teuthology.orchestra.run.smithi171.stdout:Total 621 kB/s | 172 kB 00:00 2022-01-31T19:36:33.975 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2022-01-31T19:36:33.986 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2022-01-31T19:36:33.986 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2022-01-31T19:36:34.035 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2022-01-31T19:36:34.037 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2022-01-31T19:36:34.150 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2022-01-31T19:36:34.353 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:34.358 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:34.610 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:34.985 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:34.986 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2022-01-31T19:36:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:36:34.986 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2022-01-31T19:36:35.163 DEBUG:teuthology.parallel:result is None 2022-01-31T19:36:35.322 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:35.322 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:35.323 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:35.323 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:35.323 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:35.323 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:35.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd 2022-01-31T19:36:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:35.681 INFO:teuthology.orchestra.run.smithi167.stderr:Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/epel-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist 2022-01-31T19:36:36.070 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:02:20 ago on Mon 31 Jan 2022 07:34:16 PM UTC. 2022-01-31T19:36:36.875 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2022-01-31T19:36:36.876 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:36.877 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2022-01-31T19:36:36.877 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:36.877 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2022-01-31T19:36:36.877 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:16.2.7-278.g9e931647.el8 ceph 172 k 2022-01-31T19:36:36.878 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:36.878 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2022-01-31T19:36:36.878 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2022-01-31T19:36:36.878 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2022-01-31T19:36:36.879 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:36.879 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 172 k 2022-01-31T19:36:36.879 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 511 k 2022-01-31T19:36:36.879 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2022-01-31T19:36:37.159 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-16.2.7-278.g9e931647.el8.x86_64.rpm 611 kB/s | 172 kB 00:00 2022-01-31T19:36:37.159 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2022-01-31T19:36:37.160 INFO:teuthology.orchestra.run.smithi167.stdout:Total 609 kB/s | 172 kB 00:00 2022-01-31T19:36:37.160 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2022-01-31T19:36:37.170 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2022-01-31T19:36:37.171 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2022-01-31T19:36:37.219 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2022-01-31T19:36:37.222 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2022-01-31T19:36:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2022-01-31T19:36:37.551 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:37.801 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:38.129 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 1/1 2022-01-31T19:36:38.129 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:38.129 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2022-01-31T19:36:38.130 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:16.2.7-278.g9e931647.el8.x86_64 2022-01-31T19:36:38.130 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:36:38.130 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2022-01-31T19:36:38.348 DEBUG:teuthology.parallel:result is None 2022-01-31T19:36:38.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo lsb_release -is 2022-01-31T19:36:38.384 INFO:teuthology.orchestra.run.smithi167.stdout:CentOSStream 2022-01-31T19:36:38.386 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-01-31T19:36:38.386 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:36:38.387 INFO:teuthology.packaging:ref: None 2022-01-31T19:36:38.387 INFO:teuthology.packaging:tag: None 2022-01-31T19:36:38.387 INFO:teuthology.packaging:branch: pacific 2022-01-31T19:36:38.388 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:36:38.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2022-01-31T19:36:38.708 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-01-31T19:36:38.730 INFO:teuthology.orchestra.run.smithi167.stdout:16.2.7-278.g9e931647.el8 2022-01-31T19:36:38.731 INFO:teuthology.packaging:The installed version of ceph is 16.2.7-278.g9e931647.el8 2022-01-31T19:36:38.731 INFO:teuthology.task.install:The correct ceph version 16.2.7-278.g9e931647 is installed. 2022-01-31T19:36:38.732 DEBUG:teuthology.orchestra.run.smithi171:> sudo lsb_release -is 2022-01-31T19:36:38.777 INFO:teuthology.orchestra.run.smithi171.stdout:CentOSStream 2022-01-31T19:36:38.779 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-01-31T19:36:38.780 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-31T19:36:38.780 INFO:teuthology.packaging:ref: None 2022-01-31T19:36:38.781 INFO:teuthology.packaging:tag: None 2022-01-31T19:36:38.781 INFO:teuthology.packaging:branch: pacific 2022-01-31T19:36:38.782 INFO:teuthology.packaging:sha1: 29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:36:38.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2022-01-31T19:36:39.125 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-01-31T19:36:39.151 INFO:teuthology.orchestra.run.smithi171.stdout:16.2.7-278.g9e931647.el8 2022-01-31T19:36:39.152 INFO:teuthology.packaging:The installed version of ceph is 16.2.7-278.g9e931647.el8 2022-01-31T19:36:39.152 INFO:teuthology.task.install:The correct ceph version 16.2.7-278.g9e931647 is installed. 2022-01-31T19:36:39.152 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-01-31T19:36:39.153 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:36:39.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-01-31T19:36:39.181 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:36:39.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-01-31T19:36:39.221 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-01-31T19:36:39.222 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:36:39.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2022-01-31T19:36:39.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-01-31T19:36:39.316 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:36:39.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2022-01-31T19:36:39.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-01-31T19:36:39.418 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-01-31T19:36:39.419 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:36:39.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2022-01-31T19:36:39.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-01-31T19:36:39.512 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:36:39.513 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2022-01-31T19:36:39.541 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-01-31T19:36:39.609 INFO:teuthology.run_tasks:Running task print... 2022-01-31T19:36:39.622 INFO:teuthology.task.print:**** done install task... 2022-01-31T19:36:39.622 INFO:teuthology.run_tasks:Running task cephadm... 2022-01-31T19:36:39.720 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '29e1fc1722aa5915b44828a5ad02ec45ce760aa3'} 2022-01-31T19:36:39.721 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2022-01-31T19:36:39.721 INFO:tasks.cephadm:Cluster fsid is 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:36:39.722 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-01-31T19:36:39.722 INFO:tasks.cephadm:No mon roles; fabricating mons 2022-01-31T19:36:39.722 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi167': '172.21.15.167', 'mon.smithi171': '172.21.15.171'} 2022-01-31T19:36:39.723 INFO:tasks.cephadm:Normalizing hostnames... 2022-01-31T19:36:39.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo hostname $(hostname -s) 2022-01-31T19:36:39.750 DEBUG:teuthology.orchestra.run.smithi171:> sudo hostname $(hostname -s) 2022-01-31T19:36:39.778 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2022-01-31T19:36:39.779 DEBUG:teuthology.orchestra.run.smithi167:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-01-31T19:36:39.964 INFO:teuthology.orchestra.run.smithi167.stdout:-rw-rw-r--. 1 ubuntu ubuntu 323421 Jan 31 19:36 /home/ubuntu/cephtest/cephadm 2022-01-31T19:36:39.964 DEBUG:teuthology.orchestra.run.smithi171:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-01-31T19:36:40.068 INFO:teuthology.orchestra.run.smithi171.stdout:-rw-rw-r--. 1 ubuntu ubuntu 323421 Jan 31 19:36 /home/ubuntu/cephtest/cephadm 2022-01-31T19:36:40.069 DEBUG:teuthology.orchestra.run.smithi167:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-01-31T19:36:40.087 DEBUG:teuthology.orchestra.run.smithi171:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-01-31T19:36:40.118 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2022-01-31T19:36:40.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2022-01-31T19:36:40.129 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2022-01-31T19:36:41.796 INFO:teuthology.orchestra.run.smithi171.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2022-01-31T19:36:42.220 INFO:teuthology.orchestra.run.smithi167.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2022-01-31T19:36:58.057 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-01-31T19:36:58.058 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2022-01-31T19:36:58.058 INFO:teuthology.orchestra.run.smithi171.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2022-01-31T19:36:58.058 INFO:teuthology.orchestra.run.smithi171.stdout: "repo_digests": [ 2022-01-31T19:36:58.058 INFO:teuthology.orchestra.run.smithi171.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2022-01-31T19:36:58.059 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2022-01-31T19:36:58.059 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-01-31T19:36:58.576 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:36:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2022-01-31T19:36:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2022-01-31T19:36:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "repo_digests": [ 2022-01-31T19:36:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2022-01-31T19:36:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2022-01-31T19:36:58.578 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:36:58.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph 2022-01-31T19:36:58.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph 2022-01-31T19:36:58.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /etc/ceph 2022-01-31T19:36:58.690 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /etc/ceph 2022-01-31T19:36:58.723 INFO:tasks.cephadm:Writing seed config... 2022-01-31T19:36:58.725 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2022-01-31T19:36:58.726 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2022-01-31T19:36:58.726 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2022-01-31T19:36:58.726 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2022-01-31T19:36:58.727 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2022-01-31T19:36:58.727 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2022-01-31T19:36:58.727 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2022-01-31T19:36:58.728 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2022-01-31T19:36:58.728 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2022-01-31T19:36:58.728 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-01-31T19:36:58.728 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-01-31T19:36:58.729 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2022-01-31T19:36:58.729 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2022-01-31T19:36:58.729 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2022-01-31T19:36:58.730 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2022-01-31T19:36:58.730 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2022-01-31T19:36:58.730 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2022-01-31T19:36:58.731 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2022-01-31T19:36:58.731 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2022-01-31T19:36:58.731 INFO:tasks.cephadm: override: [client] debug client = 20 2022-01-31T19:36:58.732 INFO:tasks.cephadm: override: [client] debug ms = 1 2022-01-31T19:36:58.732 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2022-01-31T19:36:58.732 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2022-01-31T19:36:58.733 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2022-01-31T19:36:58.733 INFO:tasks.cephadm: override: [mds] debug mds = 20 2022-01-31T19:36:58.733 INFO:tasks.cephadm: override: [mds] debug ms = 1 2022-01-31T19:36:58.733 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2022-01-31T19:36:58.734 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2022-01-31T19:36:58.734 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2022-01-31T19:36:58.734 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2022-01-31T19:36:58.735 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2022-01-31T19:36:58.735 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2022-01-31T19:36:58.735 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2022-01-31T19:36:58.736 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-01-31T19:36:58.736 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-01-31T19:36:58.736 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-01-31T19:36:58.737 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-01-31T19:36:58.737 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-01-31T19:36:58.737 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2022-01-31T19:36:58.738 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:36:58.739 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-01-31T19:36:58.755 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1c5a1df6-82cd-11ec-8c35-001a4aab830c mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2022-01-31T19:36:58.755 DEBUG:teuthology.orchestra.run.smithi167:mon.smithi167> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service 2022-01-31T19:36:58.798 INFO:tasks.cephadm:Bootstrapping... 2022-01-31T19:36:58.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.167 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-01-31T19:36:58.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:-- Logs begin at Mon 2022-01-31 19:28:37 UTC. -- 2022-01-31T19:36:58.991 INFO:teuthology.orchestra.run.smithi167.stderr:-------------------------------------------------------------------------------- 2022-01-31T19:36:58.992 INFO:teuthology.orchestra.run.smithi167.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', '1c5a1df6-82cd-11ec-8c35-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.167', '--skip-admin-label'] 2022-01-31T19:36:59.030 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: 3.4.1-dev 2022-01-31T19:36:59.131 INFO:teuthology.orchestra.run.smithi167.stderr:Verifying podman|docker is present... 2022-01-31T19:36:59.169 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: 3.4.1-dev 2022-01-31T19:36:59.272 INFO:teuthology.orchestra.run.smithi167.stderr:Verifying lvm2 is present... 2022-01-31T19:36:59.273 INFO:teuthology.orchestra.run.smithi167.stderr:Verifying time synchronization is in place... 2022-01-31T19:36:59.279 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2022-01-31T19:36:59.285 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: inactive 2022-01-31T19:36:59.291 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: enabled 2022-01-31T19:36:59.296 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: active 2022-01-31T19:36:59.297 INFO:teuthology.orchestra.run.smithi167.stderr:Unit chronyd.service is enabled and running 2022-01-31T19:36:59.297 INFO:teuthology.orchestra.run.smithi167.stderr:Repeating the final host check... 2022-01-31T19:36:59.334 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: 3.4.1-dev 2022-01-31T19:36:59.433 INFO:teuthology.orchestra.run.smithi167.stderr:podman (/bin/podman) version 3.4.1 is present 2022-01-31T19:36:59.434 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl is present 2022-01-31T19:36:59.434 INFO:teuthology.orchestra.run.smithi167.stderr:lvcreate is present 2022-01-31T19:36:59.438 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2022-01-31T19:36:59.444 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: inactive 2022-01-31T19:36:59.449 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: enabled 2022-01-31T19:36:59.455 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: active 2022-01-31T19:36:59.456 INFO:teuthology.orchestra.run.smithi167.stderr:Unit chronyd.service is enabled and running 2022-01-31T19:36:59.456 INFO:teuthology.orchestra.run.smithi167.stderr:Host looks OK 2022-01-31T19:36:59.457 INFO:teuthology.orchestra.run.smithi167.stderr:Cluster fsid: 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:36:59.457 INFO:teuthology.orchestra.run.smithi167.stderr:Acquiring lock 140238438324712 on /run/cephadm/1c5a1df6-82cd-11ec-8c35-001a4aab830c.lock 2022-01-31T19:36:59.457 INFO:teuthology.orchestra.run.smithi167.stderr:Lock 140238438324712 acquired on /run/cephadm/1c5a1df6-82cd-11ec-8c35-001a4aab830c.lock 2022-01-31T19:36:59.457 INFO:teuthology.orchestra.run.smithi167.stderr:Verifying IP 172.21.15.167 port 3300 ... 2022-01-31T19:36:59.458 INFO:teuthology.orchestra.run.smithi167.stderr:Verifying IP 172.21.15.167 port 6789 ... 2022-01-31T19:36:59.459 INFO:teuthology.orchestra.run.smithi167.stderr:Base mon IP is 172.21.15.167, final addrv is [v2:172.21.15.167:3300,v1:172.21.15.167:6789] 2022-01-31T19:36:59.459 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2022-01-31T19:36:59.459 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.167 metric 100 2022-01-31T19:36:59.461 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2022-01-31T19:36:59.462 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2022-01-31T19:36:59.464 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2022-01-31T19:36:59.464 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: inet6 ::1/128 scope host 2022-01-31T19:36:59.465 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2022-01-31T19:36:59.465 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2022-01-31T19:36:59.465 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7309/64 scope link noprefixroute 2022-01-31T19:36:59.465 INFO:teuthology.orchestra.run.smithi167.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2022-01-31T19:36:59.466 INFO:teuthology.orchestra.run.smithi167.stderr:Mon IP `172.21.15.167` is in CIDR network `172.21.0.0/20` 2022-01-31T19:36:59.467 INFO:teuthology.orchestra.run.smithi167.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-01-31T19:36:59.467 INFO:teuthology.orchestra.run.smithi167.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2022-01-31T19:36:59.535 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2022-01-31T19:36:59.853 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Getting image source signatures 2022-01-31T19:36:59.853 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Copying blob sha256:a997af56d24c50f31dbc8bd557a1c2adeb9ea675d7d7142f1a37cb629070c246 2022-01-31T19:36:59.853 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2022-01-31T19:36:59.854 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Copying config sha256:46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2022-01-31T19:36:59.911 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Writing manifest to image destination 2022-01-31T19:36:59.911 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: Storing signatures 2022-01-31T19:37:00.057 INFO:teuthology.orchestra.run.smithi167.stderr:/bin/podman: 46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2022-01-31T19:37:00.596 INFO:teuthology.orchestra.run.smithi167.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-01-31T19:37:00.960 INFO:teuthology.orchestra.run.smithi167.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-01-31T19:37:00.961 INFO:teuthology.orchestra.run.smithi167.stderr:Extracting ceph user uid/gid from container image... 2022-01-31T19:37:01.318 INFO:teuthology.orchestra.run.smithi167.stderr:stat: 167 167 2022-01-31T19:37:01.714 INFO:teuthology.orchestra.run.smithi167.stderr:Creating initial keys... 2022-01-31T19:37:02.035 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-authtool: AQBeOvhhVPGgARAAJ8HL0FfR243d7sayyC8YOg== 2022-01-31T19:37:02.769 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-authtool: AQBeOvhhWtkPLBAAp2o3/Hb2b9cvfVp+DL9WXw== 2022-01-31T19:37:03.652 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-authtool: AQBfOvhhgURyJhAAKIB39+RXCX2GdY3oUHuEPA== 2022-01-31T19:37:03.948 INFO:teuthology.orchestra.run.smithi167.stderr:Creating initial monmap... 2022-01-31T19:37:04.344 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2022-01-31T19:37:04.344 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:04.344 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-01-31T19:37:04.693 INFO:teuthology.orchestra.run.smithi167.stderr:monmaptool for smithi167 [v2:172.21.15.167:3300,v1:172.21.15.167:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2022-01-31T19:37:04.693 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/monmaptool: set fsid to 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:04.693 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-01-31T19:37:04.694 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:37:04.694 INFO:teuthology.orchestra.run.smithi167.stderr:Creating mon... 2022-01-31T19:37:05.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.050+0000 7fc62b047700 0 set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:37:05.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.051+0000 7fc62b047700 1 imported monmap: 2022-01-31T19:37:05.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: epoch 0 2022-01-31T19:37:05.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:05.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: last_changed 2022-01-31T19:37:04.335382+0000 2022-01-31T19:37:05.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: created 2022-01-31T19:37:04.335382+0000 2022-01-31T19:37:05.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2022-01-31T19:37:05.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: election_strategy: 1 2022-01-31T19:37:05.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.smithi167 2022-01-31T19:37:05.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.051+0000 7fc62b047700 0 /usr/bin/ceph-mon: set fsid to 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:05.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: RocksDB version: 6.8.1 2022-01-31T19:37:05.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:37:05.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Compile date Jul 8 2021 2022-01-31T19:37:05.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: DB SUMMARY 2022-01-31T19:37:05.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 0, files: 2022-01-31T19:37:05.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 2022-01-31T19:37:05.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.error_if_exists: 0 2022-01-31T19:37:05.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.create_if_missing: 1 2022-01-31T19:37:05.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.paranoid_checks: 1 2022-01-31T19:37:05.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.env: 0x5655151091c0 2022-01-31T19:37:05.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.fs: Posix File System 2022-01-31T19:37:05.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.info_log: 0x565516433000 2022-01-31T19:37:05.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:37:05.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.statistics: (nil) 2022-01-31T19:37:05.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.use_fsync: 0 2022-01-31T19:37:05.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_log_file_size: 0 2022-01-31T19:37:05.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:37:05.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:37:05.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:37:05.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:37:05.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.allow_fallocate: 1 2022-01-31T19:37:05.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:37:05.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:37:05.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.use_direct_reads: 0 2022-01-31T19:37:05.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:37:05.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:37:05.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.db_log_dir: 2022-01-31T19:37:05.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:37:05.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:37:05.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_subcompactions: 1 2022-01-31T19:37:05.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_background_flushes: -1 2022-01-31T19:37:05.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:37:05.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:37:05.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:37:05.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:37:05.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:37:05.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:37:05.094 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:37:05.094 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.write_buffer_manager: 0x56551643b530 2022-01-31T19:37:05.094 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:37:05.094 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:37:05.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:37:05.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:37:05.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:37:05.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:37:05.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:37:05.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:37:05.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:37:05.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.unordered_write: 0 2022-01-31T19:37:05.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:37:05.097 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:37:05.097 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:37:05.097 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:37:05.097 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.row_cache: None 2022-01-31T19:37:05.098 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.wal_filter: None 2022-01-31T19:37:05.098 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:37:05.098 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:37:05.098 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.preserve_deletes: 0 2022-01-31T19:37:05.099 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.two_write_queues: 0 2022-01-31T19:37:05.099 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:37:05.099 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.atomic_flush: 0 2022-01-31T19:37:05.099 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:37:05.100 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:37:05.100 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:37:05.100 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.log_readahead_size: 0 2022-01-31T19:37:05.100 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2022-01-31T19:37:05.100 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_background_jobs: 2 2022-01-31T19:37:05.101 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_background_compactions: -1 2022-01-31T19:37:05.101 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:37:05.101 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:37:05.101 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:37:05.102 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:37:05.102 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:37:05.102 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:37:05.102 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:37:05.103 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:37:05.103 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_open_files: -1 2022-01-31T19:37:05.103 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:37:05.103 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:37:05.103 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:37:05.104 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:37:05.104 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Compression algorithms supported: 2022-01-31T19:37:05.104 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:37:05.104 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kZSTD supported: 0 2022-01-31T19:37:05.105 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kXpressCompression supported: 0 2022-01-31T19:37:05.105 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:37:05.105 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kLZ4Compression supported: 1 2022-01-31T19:37:05.105 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kBZip2Compression supported: 0 2022-01-31T19:37:05.105 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kZlibCompression supported: 1 2022-01-31T19:37:05.106 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: kSnappyCompression supported: 1 2022-01-31T19:37:05.106 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:37:05.106 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2022-01-31T19:37:05.106 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.107 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000001 2022-01-31T19:37:05.107 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.107 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2022-01-31T19:37:05.107 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.107 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:37:05.108 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.merge_operator: 2022-01-31T19:37:05.108 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_filter: None 2022-01-31T19:37:05.108 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:37:05.108 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:37:05.109 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:37:05.109 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56551634c160) 2022-01-31T19:37:05.109 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2022-01-31T19:37:05.109 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:37:05.110 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:37:05.110 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2022-01-31T19:37:05.110 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: index_type: 0 2022-01-31T19:37:05.110 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2022-01-31T19:37:05.110 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: index_shortening: 1 2022-01-31T19:37:05.111 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:37:05.111 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2022-01-31T19:37:05.111 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: checksum: 1 2022-01-31T19:37:05.111 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2022-01-31T19:37:05.112 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_cache: 0x565516382d10 2022-01-31T19:37:05.112 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2022-01-31T19:37:05.112 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_cache_options: 2022-01-31T19:37:05.112 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: capacity : 536870912 2022-01-31T19:37:05.112 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2022-01-31T19:37:05.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2022-01-31T19:37:05.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2022-01-31T19:37:05.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2022-01-31T19:37:05.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2022-01-31T19:37:05.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_size: 4096 2022-01-31T19:37:05.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2022-01-31T19:37:05.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2022-01-31T19:37:05.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2022-01-31T19:37:05.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2022-01-31T19:37:05.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: partition_filters: 0 2022-01-31T19:37:05.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2022-01-31T19:37:05.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:37:05.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2022-01-31T19:37:05.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: verify_compression: 0 2022-01-31T19:37:05.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2022-01-31T19:37:05.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: format_version: 2 2022-01-31T19:37:05.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2022-01-31T19:37:05.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: block_align: 0 2022-01-31T19:37:05.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:37:05.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:37:05.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression: NoCompression 2022-01-31T19:37:05.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:37:05.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:37:05.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:37:05.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.num_levels: 7 2022-01-31T19:37:05.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:37:05.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:37:05.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:37:05.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:37:05.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:37:05.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:37:05.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:37:05.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:37:05.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:37:05.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:37:05.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:37:05.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:37:05.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:37:05.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:37:05.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:37:05.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:37:05.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:37:05.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:37:05.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:37:05.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:37:05.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:37:05.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:37:05.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:37:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:37:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:37:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:37:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:37:05.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:37:05.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:37:05.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:37:05.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:37:05.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:37:05.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:37:05.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:37:05.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:37:05.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:37:05.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:37:05.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:37:05.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:37:05.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:37:05.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:37:05.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:37:05.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:37:05.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:37:05.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:37:05.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:37:05.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:37:05.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.table_properties_collectors: 2022-01-31T19:37:05.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.inplace_update_support: 0 2022-01-31T19:37:05.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:37:05.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:37:05.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:37:05.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:37:05.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.bloom_locality: 0 2022-01-31T19:37:05.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.max_successive_merges: 0 2022-01-31T19:37:05.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:37:05.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:37:05.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.force_consistency_checks: 0 2022-01-31T19:37:05.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:37:05.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.ttl: 2592000 2022-01-31T19:37:05.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.052+0000 7fc62b047700 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:37:05.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.053+0000 7fc62b047700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:37:05.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.053+0000 7fc62b047700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2022-01-31T19:37:05.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.054+0000 7fc62b047700 4 rocksdb: DB pointer 0x565516447800 2022-01-31T19:37:05.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.054+0000 7fc613ed0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2022-01-31T19:37:05.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.054+0000 7fc613ed0700 4 rocksdb: [db_impl/db_impl.cc:851] 2022-01-31T19:37:05.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2022-01-31T19:37:05.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:37:05.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:37:05.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:37:05.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:37:05.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:37:05.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:37:05.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:37:05.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-01-31T19:37:05.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:05.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:05.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:37:05.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:37:05.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-01-31T19:37:05.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:05.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:05.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:37:05.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:05.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:05.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:37:05.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:37:05.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:37:05.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:05.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:05.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:37:05.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:37:05.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-01-31T19:37:05.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:05.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:05.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:37:05.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:37:05.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2022-01-31T19:37:05.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:05.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:05.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:37:05.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:05.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:05.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:37:05.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:37:05.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:37:05.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:05.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:05.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:37:05.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:37:05.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: 2022-01-31T19:37:05.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.055+0000 7fc62b047700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2022-01-31T19:37:05.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.055+0000 7fc62b047700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2022-01-31T19:37:05.146 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph-mon: debug 2022-01-31T19:37:05.055+0000 7fc62b047700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi167 for mon.smithi167 2022-01-31T19:37:05.448 INFO:teuthology.orchestra.run.smithi167.stderr:create mon.smithi167 on 2022-01-31T19:37:05.460 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2022-01-31T19:37:05.551 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2022-01-31T19:37:05.643 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c.target → /etc/systemd/system/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c.target. 2022-01-31T19:37:05.643 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c.target → /etc/systemd/system/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c.target. 2022-01-31T19:37:05.844 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Failed to reset failed state of unit ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Unit ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service not loaded. 2022-01-31T19:37:05.851 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Created symlink /etc/systemd/system/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c.target.wants/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service → /etc/systemd/system/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@.service. 2022-01-31T19:37:06.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:05 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:37:06.508 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:06 smithi167 podman[31327]: 2022-01-31T19:37:06.508 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:06 smithi167 bash[31181]: dfa2ea62b2fef07bdd83d06999c23a84f5eb55f7f8d5a5763b933829cfd20fa1 2022-01-31T19:37:06.542 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: disabled 2022-01-31T19:37:06.549 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: inactive 2022-01-31T19:37:06.549 INFO:teuthology.orchestra.run.smithi167.stderr:firewalld.service is not enabled 2022-01-31T19:37:06.550 INFO:teuthology.orchestra.run.smithi167.stderr:Not possible to enable service . firewalld.service is not available 2022-01-31T19:37:06.551 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for mon to start... 2022-01-31T19:37:06.551 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for mon... 2022-01-31T19:37:06.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:06 smithi167 conmon[31367]: cluster 2022-01-31T19:37:06.503574+0000 mon.smithi167 (mon.0) 0 : [INF] mkfs 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:06.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:06 smithi167 conmon[31367]: cluster 2022-01-31T19: 2022-01-31T19:37:06.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:06 smithi167 conmon[31367]: 37:06.488070+0000 mon.smithi167 (mon.0) 1 : cluster [INF] mon.smithi167 is new leader, mons smithi167 in quorum (ranks 0) 2022-01-31T19:37:06.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:06 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:37:06.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.987+0000 7f68f8285700 1 Processor -- start 2022-01-31T19:37:06.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.988+0000 7f68f8285700 1 -- start start 2022-01-31T19:37:06.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.988+0000 7f68f8285700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f0100f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:06.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.988+0000 7f68f8285700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f00fc900 con 0x7f68f006fc60 2022-01-31T19:37:06.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.988+0000 7f68f6021700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f0100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:06.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.988+0000 7f68f6021700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f0100f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60296/0 (socket says 172.21.15.167:60296) 2022-01-31T19:37:06.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.988+0000 7f68f6021700 1 -- 172.21.15.167:0/1224226570 learned_addr learned my addr 172.21.15.167:0/1224226570 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:06.996 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.989+0000 7f68f6021700 1 -- 172.21.15.167:0/1224226570 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f0101400 con 0x7f68f006fc60 2022-01-31T19:37:06.996 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.989+0000 7f68f6021700 1 --2- 172.21.15.167:0/1224226570 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f0100f10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f68e400b5f0 tx=0x7f68e4006ee0).ready entity=mon.0 client_cookie=ab5f9af5f5f16a16 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:06.996 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.989+0000 7f68f501f700 1 -- 172.21.15.167:0/1224226570 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e400d600 con 0x7f68f006fc60 2022-01-31T19:37:06.996 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.989+0000 7f68f501f700 1 -- 172.21.15.167:0/1224226570 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e400d760 con 0x7f68f006fc60 2022-01-31T19:37:06.997 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.989+0000 7f68f501f700 1 -- 172.21.15.167:0/1224226570 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e400da30 con 0x7f68f006fc60 2022-01-31T19:37:06.997 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 -- 172.21.15.167:0/1224226570 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 msgr2=0x7f68f0100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:06.997 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 --2- 172.21.15.167:0/1224226570 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f0100f10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f68e400b5f0 tx=0x7f68e4006ee0).stop 2022-01-31T19:37:06.997 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 -- 172.21.15.167:0/1224226570 shutdown_connections 2022-01-31T19:37:06.998 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 --2- 172.21.15.167:0/1224226570 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f0100f10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:06.998 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 -- 172.21.15.167:0/1224226570 >> 172.21.15.167:0/1224226570 conn(0x7f68f00f6530 msgr2=0x7f68f00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:06.998 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 -- 172.21.15.167:0/1224226570 shutdown_connections 2022-01-31T19:37:06.998 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.990+0000 7f68f8285700 1 -- 172.21.15.167:0/1224226570 wait complete. 2022-01-31T19:37:06.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f8285700 1 Processor -- start 2022-01-31T19:37:06.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f8285700 1 -- start start 2022-01-31T19:37:06.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f8285700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f010c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:06.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f8285700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f010cc30 con 0x7f68f006fc60 2022-01-31T19:37:06.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f6021700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f010c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:07.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f6021700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f010c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60298/0 (socket says 172.21.15.167:60298) 2022-01-31T19:37:07.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f6021700 1 -- 172.21.15.167:0/662535601 learned_addr learned my addr 172.21.15.167:0/662535601 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:07.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f6021700 1 -- 172.21.15.167:0/662535601 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e400b200 con 0x7f68f006fc60 2022-01-31T19:37:07.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.991+0000 7f68f6021700 1 --2- 172.21.15.167:0/662535601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f010c740 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f68e4006c70 tx=0x7f68e4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:07.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e4011b40 con 0x7f68f006fc60 2022-01-31T19:37:07.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e4011ca0 con 0x7f68f006fc60 2022-01-31T19:37:07.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f010ce30 con 0x7f68f006fc60 2022-01-31T19:37:07.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e401d430 con 0x7f68f006fc60 2022-01-31T19:37:07.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f010d2c0 con 0x7f68f006fc60 2022-01-31T19:37:07.002 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f68e400f070 con 0x7f68f006fc60 2022-01-31T19:37:07.002 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.992+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f68e401dbf0 con 0x7f68f006fc60 2022-01-31T19:37:07.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.993+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0045bb0 con 0x7f68f006fc60 2022-01-31T19:37:07.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:06.995+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f68e4016070 con 0x7f68f006fc60 2022-01-31T19:37:07.065 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.061+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f68f010e7d0 con 0x7f68f006fc60 2022-01-31T19:37:07.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.061+0000 7f68e2ffd700 1 -- 172.21.15.167:0/662535601 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f68e4036070 con 0x7f68f006fc60 2022-01-31T19:37:07.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: cluster: 2022-01-31T19:37:07.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: id: 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:07.067 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: health: HEALTH_OK 2022-01-31T19:37:07.067 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.067 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: services: 2022-01-31T19:37:07.067 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi167 (age 0.559779s) 2022-01-31T19:37:07.067 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mgr: no daemons active 2022-01-31T19:37:07.068 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2022-01-31T19:37:07.068 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.068 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: data: 2022-01-31T19:37:07.068 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2022-01-31T19:37:07.069 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2022-01-31T19:37:07.069 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2022-01-31T19:37:07.069 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: pgs: 2022-01-31T19:37:07.069 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.069 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 msgr2=0x7f68f010c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:07.070 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 --2- 172.21.15.167:0/662535601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f010c740 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f68e4006c70 tx=0x7f68e4016000).stop 2022-01-31T19:37:07.070 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 shutdown_connections 2022-01-31T19:37:07.070 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 --2- 172.21.15.167:0/662535601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f68f006fc60 0x7f68f010c740 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:07.070 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 >> 172.21.15.167:0/662535601 conn(0x7f68f00f6530 msgr2=0x7f68f00f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:07.071 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 shutdown_connections 2022-01-31T19:37:07.071 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:07.062+0000 7f68f8285700 1 -- 172.21.15.167:0/662535601 wait complete. 2022-01-31T19:37:07.389 INFO:teuthology.orchestra.run.smithi167.stderr:mon is available 2022-01-31T19:37:07.390 INFO:teuthology.orchestra.run.smithi167.stderr:Assimilating anything we can from ceph.conf... 2022-01-31T19:37:07.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:07 smithi167 conmon[31367]: cluster 2022-01-31T19:37:06.503012+0000 mon.smithi167 (mon.0) 2 : cluster [INF] mon.smithi167 is new leader, mons smithi167 in quorum (ranks 0) 2022-01-31T19:37:07.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:07 smithi167 conmon[31367]: cluster 2022-01-31T19:37:06.503189+0000 mon.smithi167 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi167=[v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0]} 2022-01-31T19:37:07.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:07 smithi167 conmon[31367]: cluster 2022-01-31T19:37:06.505526+0000 mon.smithi167 (mon.0) 4 : cluster [DBG] fsmap 2022-01-31T19:37:07.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:07 smithi167 conmon[31367]: cluster 2022-01-31T19:37:06.508666+0000 mon.smithi167 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2022-01-31T19:37:07.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:07 smithi167 conmon[31367]: cluster 2022-01-31T19:37:06.509045+0000 mon.smithi167 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2022-01-31T19:37:07.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:07 smithi167 conmon[31367]: audit 2022-01-31T19:37:07.062756+0000 mon.smithi167 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.167:0/662535601' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-01-31T19:37:08.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb6fdf700 1 Processor -- start 2022-01-31T19:37:08.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb6fdf700 1 -- start start 2022-01-31T19:37:08.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb6fdf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb00fc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:08.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb6fdf700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00fcd20 con 0x7f2cb00fc410 2022-01-31T19:37:08.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb4d7b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb00fc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:08.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb4d7b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb00fc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60300/0 (socket says 172.21.15.167:60300) 2022-01-31T19:37:08.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb4d7b700 1 -- 172.21.15.167:0/2634534189 learned_addr learned my addr 172.21.15.167:0/2634534189 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:08.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.115+0000 7f2cb4d7b700 1 -- 172.21.15.167:0/2634534189 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00fce60 con 0x7f2cb00fc410 2022-01-31T19:37:08.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb4d7b700 1 --2- 172.21.15.167:0/2634534189 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb00fc830 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f2ca400b5f0 tx=0x7f2ca4006ee0).ready entity=mon.0 client_cookie=5564a7db30ff3624 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:08.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2caf7fe700 1 -- 172.21.15.167:0/2634534189 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ca400d600 con 0x7f2cb00fc410 2022-01-31T19:37:08.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2caf7fe700 1 -- 172.21.15.167:0/2634534189 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca400d760 con 0x7f2cb00fc410 2022-01-31T19:37:08.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2caf7fe700 1 -- 172.21.15.167:0/2634534189 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ca400da30 con 0x7f2cb00fc410 2022-01-31T19:37:08.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/2634534189 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 msgr2=0x7f2cb00fc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:08.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 --2- 172.21.15.167:0/2634534189 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb00fc830 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f2ca400b5f0 tx=0x7f2ca4006ee0).stop 2022-01-31T19:37:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/2634534189 shutdown_connections 2022-01-31T19:37:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 --2- 172.21.15.167:0/2634534189 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb00fc830 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/2634534189 >> 172.21.15.167:0/2634534189 conn(0x7f2cb00f5ca0 msgr2=0x7f2cb00f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/2634534189 shutdown_connections 2022-01-31T19:37:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.116+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/2634534189 wait complete. 2022-01-31T19:37:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.117+0000 7f2cb6fdf700 1 Processor -- start 2022-01-31T19:37:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.117+0000 7f2cb6fdf700 1 -- start start 2022-01-31T19:37:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.117+0000 7f2cb6fdf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb010c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.117+0000 7f2cb6fdf700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb010cbf0 con 0x7f2cb00fc410 2022-01-31T19:37:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.117+0000 7f2cb4d7b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb010c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.117+0000 7f2cb4d7b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb010c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60302/0 (socket says 172.21.15.167:60302) 2022-01-31T19:37:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cb4d7b700 1 -- 172.21.15.167:0/385567018 learned_addr learned my addr 172.21.15.167:0/385567018 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cb4d7b700 1 -- 172.21.15.167:0/385567018 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca400b200 con 0x7f2cb00fc410 2022-01-31T19:37:08.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cb4d7b700 1 --2- 172.21.15.167:0/385567018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb010c700 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f2ca4006c70 tx=0x7f2ca4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:08.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ca4011b40 con 0x7f2cb00fc410 2022-01-31T19:37:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca4011ca0 con 0x7f2cb00fc410 2022-01-31T19:37:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb010cdf0 con 0x7f2cb00fc410 2022-01-31T19:37:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ca401d430 con 0x7f2cb00fc410 2022-01-31T19:37:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb010d280 con 0x7f2cb00fc410 2022-01-31T19:37:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f2ca400f070 con 0x7f2cb00fc410 2022-01-31T19:37:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.118+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2ca401dc60 con 0x7f2cb00fc410 2022-01-31T19:37:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.119+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb0045bb0 con 0x7f2cb00fc410 2022-01-31T19:37:08.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.122+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2ca4016070 con 0x7f2cb00fc410 2022-01-31T19:37:08.187 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.185+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2cb00fb1f0 con 0x7f2cb00fc410 2022-01-31T19:37:08.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.189+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 7 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f2ca40318e0 con 0x7f2cb00fc410 2022-01-31T19:37:08.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.189+0000 7f2cad7fa700 1 -- 172.21.15.167:0/385567018 <== mon.0 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+456 (secure 0 0 0) 0x7f2ca40276b0 con 0x7f2cb00fc410 2022-01-31T19:37:08.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: [global] 2022-01-31T19:37:08.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: fsid = 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:08.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: log_to_journald = false 2022-01-31T19:37:08.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.167:3300,v1:172.21.15.167:6789] 2022-01-31T19:37:08.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2022-01-31T19:37:08.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2022-01-31T19:37:08.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2022-01-31T19:37:08.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2022-01-31T19:37:08.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: [mgr] 2022-01-31T19:37:08.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2022-01-31T19:37:08.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: [osd] 2022-01-31T19:37:08.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2022-01-31T19:37:08.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2022-01-31T19:37:08.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2022-01-31T19:37:08.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2022-01-31T19:37:08.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2022-01-31T19:37:08.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 msgr2=0x7f2cb010c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:08.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 --2- 172.21.15.167:0/385567018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb010c700 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f2ca4006c70 tx=0x7f2ca4016000).stop 2022-01-31T19:37:08.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 shutdown_connections 2022-01-31T19:37:08.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 --2- 172.21.15.167:0/385567018 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2cb00fc410 0x7f2cb010c700 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:08.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 >> 172.21.15.167:0/385567018 conn(0x7f2cb00f5ca0 msgr2=0x7f2cb00f7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:08.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 shutdown_connections 2022-01-31T19:37:08.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.190+0000 7f2cb6fdf700 1 -- 172.21.15.167:0/385567018 wait complete. 2022-01-31T19:37:08.526 INFO:teuthology.orchestra.run.smithi167.stderr:Generating new minimal ceph.conf... 2022-01-31T19:37:08.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:08 smithi167 conmon[31367]: audit 2022-01-31T19:37:08.187583+0000 mon.smithi167 ( 2022-01-31T19:37:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:08 smithi167 conmon[31367]: mon.0) 8 : audit [INF] from='client.? 172.21.15.167:0/385567018' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-01-31T19:37:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:08 smithi167 conmon[31367]: audit 2022-01-31T19:37:08.190682+0000 mon.smithi167 (mon.0) 9 : audit [INF] from='client.? 172.21.15.167:0/385567018' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-01-31T19:37:08.993 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.989+0000 7f01d617d700 1 Processor -- start 2022-01-31T19:37:08.993 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.989+0000 7f01d617d700 1 -- start start 2022-01-31T19:37:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.989+0000 7f01d617d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d00fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.989+0000 7f01d617d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00fcd00 con 0x7f01d00fc3f0 2022-01-31T19:37:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01cf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d00fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01cf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d00fc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60304/0 (socket says 172.21.15.167:60304) 2022-01-31T19:37:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01cf7fe700 1 -- 172.21.15.167:0/2717777389 learned_addr learned my addr 172.21.15.167:0/2717777389 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01cf7fe700 1 -- 172.21.15.167:0/2717777389 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d00fce40 con 0x7f01d00fc3f0 2022-01-31T19:37:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01cf7fe700 1 --2- 172.21.15.167:0/2717777389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d00fc810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f01c400b5f0 tx=0x7f01c4006ee0).ready entity=mon.0 client_cookie=3cd818f92b3fa655 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01ce7fc700 1 -- 172.21.15.167:0/2717777389 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c400d600 con 0x7f01d00fc3f0 2022-01-31T19:37:08.996 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01ce7fc700 1 -- 172.21.15.167:0/2717777389 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f01c400dbc0 con 0x7f01d00fc3f0 2022-01-31T19:37:08.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.990+0000 7f01ce7fc700 1 -- 172.21.15.167:0/2717777389 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c4011b90 con 0x7f01d00fc3f0 2022-01-31T19:37:08.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 -- 172.21.15.167:0/2717777389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 msgr2=0x7f01d00fc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:09.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 --2- 172.21.15.167:0/2717777389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d00fc810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f01c400b5f0 tx=0x7f01c4006ee0).stop 2022-01-31T19:37:09.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 -- 172.21.15.167:0/2717777389 shutdown_connections 2022-01-31T19:37:09.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 --2- 172.21.15.167:0/2717777389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d00fc810 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:09.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 -- 172.21.15.167:0/2717777389 >> 172.21.15.167:0/2717777389 conn(0x7f01d00f6520 msgr2=0x7f01d00f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:09.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 -- 172.21.15.167:0/2717777389 shutdown_connections 2022-01-31T19:37:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.991+0000 7f01d617d700 1 -- 172.21.15.167:0/2717777389 wait complete. 2022-01-31T19:37:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01d617d700 1 Processor -- start 2022-01-31T19:37:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01d617d700 1 -- start start 2022-01-31T19:37:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01d617d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d010acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01d617d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d010b1c0 con 0x7f01d00fc3f0 2022-01-31T19:37:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01cf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d010acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01cf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d010acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60306/0 (socket says 172.21.15.167:60306) 2022-01-31T19:37:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.992+0000 7f01cf7fe700 1 -- 172.21.15.167:0/195950682 learned_addr learned my addr 172.21.15.167:0/195950682 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:09.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01cf7fe700 1 -- 172.21.15.167:0/195950682 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c400b200 con 0x7f01d00fc3f0 2022-01-31T19:37:09.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01cf7fe700 1 --2- 172.21.15.167:0/195950682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d010acd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f01c4000c00 tx=0x7f01c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:09.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c401b570 con 0x7f01d00fc3f0 2022-01-31T19:37:09.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f01c401bb30 con 0x7f01d00fc3f0 2022-01-31T19:37:09.003 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c401a6e0 con 0x7f01d00fc3f0 2022-01-31T19:37:09.004 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d010b3c0 con 0x7f01d00fc3f0 2022-01-31T19:37:09.004 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d010bf70 con 0x7f01d00fc3f0 2022-01-31T19:37:09.004 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.993+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f01c402b070 con 0x7f01d00fc3f0 2022-01-31T19:37:09.004 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.994+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f01c4024a60 con 0x7f01d00fc3f0 2022-01-31T19:37:09.005 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.994+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0046a20 con 0x7f01d00fc3f0 2022-01-31T19:37:09.005 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:08.997+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f01c401f020 con 0x7f01d00fc3f0 2022-01-31T19:37:09.062 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.060+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f01d00fb670 con 0x7f01d00fc3f0 2022-01-31T19:37:09.063 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.060+0000 7f01b7fff700 1 -- 172.21.15.167:0/195950682 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f01c4016070 con 0x7f01d00fc3f0 2022-01-31T19:37:09.065 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.061+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 msgr2=0x7f01d010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:09.065 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.061+0000 7f01d617d700 1 --2- 172.21.15.167:0/195950682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d010acd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f01c4000c00 tx=0x7f01c4006ee0).stop 2022-01-31T19:37:09.065 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.061+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 shutdown_connections 2022-01-31T19:37:09.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.061+0000 7f01d617d700 1 --2- 172.21.15.167:0/195950682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01d00fc3f0 0x7f01d010acd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:09.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.061+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 >> 172.21.15.167:0/195950682 conn(0x7f01d00f6520 msgr2=0x7f01d006f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:09.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.061+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 shutdown_connections 2022-01-31T19:37:09.066 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:09.062+0000 7f01d617d700 1 -- 172.21.15.167:0/195950682 wait complete. 2022-01-31T19:37:09.362 INFO:teuthology.orchestra.run.smithi167.stderr:Restarting the monitor... 2022-01-31T19:37:09.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 systemd[1]: Stopping Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:37:09.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 bash[31841]: Error: no container with name or ID "ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c-mon.smithi167" found: no such container 2022-01-31T19:37:09.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 conmon[31367]: audit 2022-01-31T19:37:09.062020+0000 mon.smithi167 ( 2022-01-31T19:37:09.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 conmon[31367]: mon.0) 10 : audit [DBG] from='client.? 172.21.15.167:0/195950682' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:37:09.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 conmon[31367]: debug 2022-01-31T19:37:09.526+0000 7f45872b0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi167 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2022-01-31T19:37:09.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 conmon[31367]: debug 2022-01-31T19:37:09.526+0000 7f45872b0700 -1 mon.smithi167@0(leader) e1 *** Got Signal Terminated *** 2022-01-31T19:37:10.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:09 smithi167 bash[31841]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c-mon-smithi167 2022-01-31T19:37:10.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 bash[31841]: Error: no container with name or ID "ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c-mon.smithi167" found: no such container 2022-01-31T19:37:10.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Succeeded. 2022-01-31T19:37:10.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:37:10.398 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:37:10.591 INFO:teuthology.orchestra.run.smithi167.stderr:Setting mon public_network to 172.21.0.0/20 2022-01-31T19:37:10.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 podman[32166]: 2022-01-31T19:37:10.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.566+0000 7f07e88d9700 0 set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:37:10.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.566+0000 7f07e88d9700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2022-01-31T19:37:10.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.566+0000 7f07e88d9700 0 pidfile_write: ignore empty --pid-file 2022-01-31T19:37:10.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.570+0000 7f07e88d9700 0 load: jerasure load: lrc load: isa 2022-01-31T19:37:10.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: RocksDB version: 6.8.1 2022-01-31T19:37:10.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:37:10.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Compile date Jul 8 2021 2022-01-31T19:37:10.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: DB SUMMARY 2022-01-31T19:37:10.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: CURRENT file: CURRENT 2022-01-31T19:37:10.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:37:10.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2022-01-31T19:37:10.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 1, files: 000004.sst 2022-01-31T19:37:10.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000006.log size: 98970 ; 2022-01-31T19:37:10.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.error_if_exists: 0 2022-01-31T19:37:10.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.create_if_missing: 0 2022-01-31T19:37:10.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.paranoid_checks: 1 2022-01-31T19:37:10.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.env: 0x561e472cf1c0 2022-01-31T19:37:10.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.fs: Posix File System 2022-01-31T19:37:10.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.info_log: 0x561e47e2ba40 2022-01-31T19:37:10.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:37:10.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.statistics: (nil) 2022-01-31T19:37:10.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.use_fsync: 0 2022-01-31T19:37:10.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_log_file_size: 0 2022-01-31T19:37:10.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:37:10.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:37:10.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:37:10.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:37:10.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.allow_fallocate: 1 2022-01-31T19:37:10.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:37:10.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:37:10.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.use_direct_reads: 0 2022-01-31T19:37:10.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:37:10.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:37:10.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.db_log_dir: 2022-01-31T19:37:10.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:37:10.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:37:10.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_subcompactions: 1 2022-01-31T19:37:10.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_background_flushes: -1 2022-01-31T19:37:10.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:37:10.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:37:10.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:37:10.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:37:10.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:37:10.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:37:10.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:37:10.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.write_buffer_manager: 0x561e47e33fb0 2022-01-31T19:37:10.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:37:10.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:37:10.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:37:10.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:37:10.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 bash[32013]: 0ebba348055ea29f8feb535287548154da626f7e10a74d6b7d99793d61a80a65 2022-01-31T19:37:10.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:37:10.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:37:10.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:37:10.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:37:10.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:37:10.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.unordered_write: 0 2022-01-31T19:37:10.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:37:10.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:37:10.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:37:10.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:37:10.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.row_cache: None 2022-01-31T19:37:10.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.wal_filter: None 2022-01-31T19:37:10.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:37:10.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:37:10.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.preserve_deletes: 0 2022-01-31T19:37:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.two_write_queues: 0 2022-01-31T19:37:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:37:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.atomic_flush: 0 2022-01-31T19:37:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:37:10.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:37:10.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:37:10.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.log_readahead_size: 0 2022-01-31T19:37:10.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2022-01-31T19:37:10.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_background_jobs: 2 2022-01-31T19:37:10.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_background_compactions: -1 2022-01-31T19:37:10.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:37:10.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:37:10.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:37:10.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:37:10.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:37:10.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:37:10.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:37:10.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:37:10.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.max_open_files: -1 2022-01-31T19:37:10.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:37:10.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:37:10.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:37:10.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:37:10.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Compression algorithms supported: 2022-01-31T19:37:10.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:37:10.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kZSTD supported: 0 2022-01-31T19:37:10.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kXpressCompression supported: 0 2022-01-31T19:37:10.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:37:10.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kLZ4Compression supported: 1 2022-01-31T19:37:10.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kBZip2Compression supported: 0 2022-01-31T19:37:10.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kZlibCompression supported: 1 2022-01-31T19:37:10.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: kSnappyCompression supported: 1 2022-01-31T19:37:10.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.571+0000 7f07e88d9700 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:37:10.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000005 2022-01-31T19:37:10.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2022-01-31T19:37:10.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:37:10.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.merge_operator: 2022-01-31T19:37:10.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_filter: None 2022-01-31T19:37:10.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:37:10.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:37:10.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:37:10.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561e47d45cc0) 2022-01-31T19:37:10.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cache_index_and_filter_blocks: 1 2022-01-31T19:37:10.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:37:10.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:37:10.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: pin_top_level_index_and_filter: 1 2022-01-31T19:37:10.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: index_type: 0 2022-01-31T19:37:10.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: data_block_index_type: 0 2022-01-31T19:37:10.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: index_shortening: 1 2022-01-31T19:37:10.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:37:10.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: hash_index_allow_collision: 1 2022-01-31T19:37:10.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: checksum: 1 2022-01-31T19:37:10.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: no_block_cache: 0 2022-01-31T19:37:10.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_cache: 0x561e47d7af10 2022-01-31T19:37:10.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_cache_name: BinnedLRUCache 2022-01-31T19:37:10.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_cache_options: 2022-01-31T19:37:10.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: capacity : 536870912 2022-01-31T19:37:10.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: num_shard_bits : 4 2022-01-31T19:37:10.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: strict_capacity_limit : 0 2022-01-31T19:37:10.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: high_pri_pool_ratio: 0.000 2022-01-31T19:37:10.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_cache_compressed: (nil) 2022-01-31T19:37:10.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: persistent_cache: (nil) 2022-01-31T19:37:10.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_size: 4096 2022-01-31T19:37:10.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_size_deviation: 10 2022-01-31T19:37:10.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_restart_interval: 16 2022-01-31T19:37:10.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: index_block_restart_interval: 1 2022-01-31T19:37:10.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: metadata_block_size: 4096 2022-01-31T19:37:10.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: partition_filters: 0 2022-01-31T19:37:10.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: use_delta_encoding: 1 2022-01-31T19:37:10.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:37:10.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: whole_key_filtering: 1 2022-01-31T19:37:10.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: verify_compression: 0 2022-01-31T19:37:10.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: read_amp_bytes_per_bit: 0 2022-01-31T19:37:10.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: format_version: 2 2022-01-31T19:37:10.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: enable_index_compression: 1 2022-01-31T19:37:10.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: block_align: 0 2022-01-31T19:37:10.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:37:10.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:37:10.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression: NoCompression 2022-01-31T19:37:10.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:37:10.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:37:10.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:37:10.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.num_levels: 7 2022-01-31T19:37:10.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:37:10.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:37:10.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:37:10.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:37:10.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:37:10.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:37:10.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:37:10.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:37:10.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:37:10.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:37:10.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:37:10.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:37:10.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:37:10.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:37:10.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:37:10.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:37:10.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:37:10.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:37:10.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:37:10.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:37:10.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:37:10.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:37:10.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:37:10.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:37:10.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:37:10.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:37:10.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:37:10.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:37:10.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:37:10.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:37:10.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:37:10.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:37:10.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:37:10.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:37:10.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:37:10.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:37:10.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:37:10.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:37:10.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:37:10.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:37:10.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:37:10.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:37:10.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:37:10.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:37:10.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:37:10.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:37:10.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:37:10.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.table_properties_collectors: 2022-01-31T19:37:10.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.inplace_update_support: 0 2022-01-31T19:37:10.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:37:10.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:37:10.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:37:10.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:37:10.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.bloom_locality: 0 2022-01-31T19:37:10.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.max_successive_merges: 0 2022-01-31T19:37:10.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:37:10.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:37:10.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.force_consistency_checks: 0 2022-01-31T19:37:10.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:37:10.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.ttl: 2592000 2022-01-31T19:37:10.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:37:10.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:37:10.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2022-01-31T19:37:10.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1643657830574371, "job": 1, "event": "recovery_started", "log_files": [6]} 2022-01-31T19:37:10.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.572+0000 7f07e88d9700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2022-01-31T19:37:10.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.573+0000 7f07e88d9700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-01-31T19:37:10.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.573+0000 7f07e88d9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1643657830575226, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94504, "table_properties": {"data_size": 92592, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12688, "raw_average_key_size": 49, "raw_value_size": 85548, "raw_average_value_size": 334, "num_data_blocks": 12, "num_entries": 256, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1643657830, "oldest_key_time": 3, "file_creation_time": 0}} 2022-01-31T19:37:10.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.573+0000 7f07e88d9700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2022-01-31T19:37:10.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.574+0000 7f07e88d9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1643657830575692, "job": 1, "event": "recovery_finished"} 2022-01-31T19:37:10.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.575+0000 7f07e88d9700 4 rocksdb: DB pointer 0x561e47e3f800 2022-01-31T19:37:10.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.575+0000 7f07cf6e1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2022-01-31T19:37:10.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.575+0000 7f07cf6e1700 4 rocksdb: [db_impl/db_impl.cc:851] 2022-01-31T19:37:10.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** DB Stats ** 2022-01-31T19:37:10.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:37:10.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:37:10.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:37:10.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:37:10.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:37:10.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:37:10.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:37:10.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** Compaction Stats [default] ** 2022-01-31T19:37:10.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:10.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:10.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: L0 2/0 93.98 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Sum 2/0 93.98 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** Compaction Stats [default] ** 2022-01-31T19:37:10.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:10.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:10.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:37:10.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:10.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:10.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:37:10.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:37:10.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:37:10.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Cumulative compaction: 0.00 GB write, 28.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:10.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Interval compaction: 0.00 GB write, 28.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:10.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:37:10.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:37:10.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** Compaction Stats [default] ** 2022-01-31T19:37:10.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:10.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:10.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: L0 2/0 93.98 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Sum 2/0 93.98 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:37:10.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** Compaction Stats [default] ** 2022-01-31T19:37:10.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:37:10.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:37:10.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.2 0.00 0.00 1 0.001 0 0 2022-01-31T19:37:10.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:37:10.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:10.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:37:10.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:37:10.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:37:10.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:37:10.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Cumulative compaction: 0.00 GB write, 27.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:10.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:37:10.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:37:10.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:37:10.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.576+0000 7f07e88d9700 0 starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:10.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.577+0000 7f07e88d9700 1 mon.smithi167@-1(???) e1 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:10.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.577+0000 7f07e88d9700 0 mon.smithi167@-1(???).mds e1 new map 2022-01-31T19:37:10.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.577+0000 7f07e88d9700 0 mon.smithi167@-1(???).mds e1 print_map 2022-01-31T19:37:10.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: e1 2022-01-31T19:37:10.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:37:10.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:37:10.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: legacy client fscid: -1 2022-01-31T19:37:10.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: No filesystems configured 2022-01-31T19:37:10.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.577+0000 7f07e88d9700 0 mon.smithi167@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-01-31T19:37:10.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.577+0000 7f07e88d9700 0 mon.smithi167@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-01-31T19:37:10.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.577+0000 7f07e88d9700 0 mon.smithi167@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-01-31T19:37:10.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.577+0000 7f07e88d9700 0 mon.smithi167@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-01-31T19:37:10.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: audit 2022-01-31T19:37:10.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19 2022-01-31T19:37:10.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: :37:09.062020+0000 mon.smithi167 2022-01-31T19:37:10.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: (mon.0) 10 : audit 2022-01-31T19:37:10.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: [DBG] from='client.? 172.21.15.167:0/195950682' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:37:10.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.578+0000 7f07e88d9700 1 mon.smithi167@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-01-31T19:37:10.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expand map: {default=false} 2022-01-31T19:37:10.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta from 'false' to 'false' 2022-01-31T19:37:10.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expanded map: {default=false} 2022-01-31T19:37:10.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expand map: {default=info} 2022-01-31T19:37:10.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta from 'info' to 'info' 2022-01-31T19:37:10.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 2022-01-31T19:37:10.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expanded map: {default=info} 2022-01-31T19:37:10.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expand map: {default=daemon} 2022-01-31T19:37:10.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta from 'daemon' to 'daemon' 2022-01-31T19:37:10.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expanded map: {default=daemon} 2022-01-31T19:37:10.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expand map: {default=debug} 2022-01-31T19:37:10.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta from 'debug' to 'debug' 2022-01-31T19:37:10.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: debug 2022-01-31T19:37:10.579+0000 7f07e88d9700 20 expand_channel_meta expanded map: {default=debug} 2022-01-31T19:37:10.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:37:10.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cluster 2022-01-31T19:37:10.595127+0000 mon.smithi167 (mon.0) 2022-01-31T19:37:10.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: 1 : cluster [INF] mon.smithi167 is new leader, mons smithi167 in quorum (ranks 0) 2022-01-31T19:37:10.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cluster 2022-01-31T19:37:10.595214+0000 mon.smithi167 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi167=[v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0]} 2022-01-31T19:37:10.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cluster 2022-01-31T19 2022-01-31T19:37:10.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: :37:10.595268+0000 mon.smithi167 (mon.0) 3 : cluster [DBG] fsmap 2022-01-31T19:37:10.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cluster 2022-01-31T19:37:10 2022-01-31T19:37:10.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: .595309+0000 mon.smithi167 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2022-01-31T19:37:10.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:10 smithi167 conmon[32206]: cluster 2022-01-31T19:37:10.595821+0000 mon.smithi167 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2022-01-31T19:37:11.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.117+0000 7f70dcfa7700 1 Processor -- start 2022-01-31T19:37:11.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.117+0000 7f70dcfa7700 1 -- start start 2022-01-31T19:37:11.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.117+0000 7f70dcfa7700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d80fd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:11.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.118+0000 7f70dcfa7700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d80fde30 con 0x7f70d80fd520 2022-01-31T19:37:11.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.118+0000 7f70d659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d80fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:11.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.118+0000 7f70d659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d80fd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60308/0 (socket says 172.21.15.167:60308) 2022-01-31T19:37:11.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.118+0000 7f70d659c700 1 -- 172.21.15.167:0/1092881598 learned_addr learned my addr 172.21.15.167:0/1092881598 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:11.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.118+0000 7f70d659c700 1 -- 172.21.15.167:0/1092881598 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d80fdf70 con 0x7f70d80fd520 2022-01-31T19:37:11.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.118+0000 7f70d659c700 1 --2- 172.21.15.167:0/1092881598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d80fd940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70cc00b5f0 tx=0x7f70cc006ee0).ready entity=mon.0 client_cookie=447211eaa8257a17 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:11.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70d559a700 1 -- 172.21.15.167:0/1092881598 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70cc00d600 con 0x7f70d80fd520 2022-01-31T19:37:11.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70d559a700 1 -- 172.21.15.167:0/1092881598 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f70cc00dbc0 con 0x7f70d80fd520 2022-01-31T19:37:11.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70d559a700 1 -- 172.21.15.167:0/1092881598 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70cc011b90 con 0x7f70d80fd520 2022-01-31T19:37:11.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1092881598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 msgr2=0x7f70d80fd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:11.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 --2- 172.21.15.167:0/1092881598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d80fd940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70cc00b5f0 tx=0x7f70cc006ee0).stop 2022-01-31T19:37:11.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1092881598 shutdown_connections 2022-01-31T19:37:11.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 --2- 172.21.15.167:0/1092881598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d80fd940 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:11.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1092881598 >> 172.21.15.167:0/1092881598 conn(0x7f70d80f7670 msgr2=0x7f70d80f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:11.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1092881598 shutdown_connections 2022-01-31T19:37:11.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.119+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1092881598 wait complete. 2022-01-31T19:37:11.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70dcfa7700 1 Processor -- start 2022-01-31T19:37:11.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70dcfa7700 1 -- start start 2022-01-31T19:37:11.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70dcfa7700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d810acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:11.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70dcfa7700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d810b1c0 con 0x7f70d80fd520 2022-01-31T19:37:11.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70d659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d810acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:11.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70d659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d810acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60310/0 (socket says 172.21.15.167:60310) 2022-01-31T19:37:11.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.120+0000 7f70d659c700 1 -- 172.21.15.167:0/1002020467 learned_addr learned my addr 172.21.15.167:0/1002020467 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:11.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70d659c700 1 -- 172.21.15.167:0/1002020467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc00b200 con 0x7f70d80fd520 2022-01-31T19:37:11.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70d659c700 1 --2- 172.21.15.167:0/1002020467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d810acd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f70cc00e040 tx=0x7f70cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:11.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70cc01b410 con 0x7f70d80fd520 2022-01-31T19:37:11.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f70cc01b9d0 con 0x7f70d80fd520 2022-01-31T19:37:11.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d810b3c0 con 0x7f70d80fd520 2022-01-31T19:37:11.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70cc01a7e0 con 0x7f70d80fd520 2022-01-31T19:37:11.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d810bf70 con 0x7f70d80fd520 2022-01-31T19:37:11.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f70cc00f070 con 0x7f70d80fd520 2022-01-31T19:37:11.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.121+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f70cc024ba0 con 0x7f70d80fd520 2022-01-31T19:37:11.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.122+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d8046a20 con 0x7f70d80fd520 2022-01-31T19:37:11.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.124+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f70cc01f030 con 0x7f70d80fd520 2022-01-31T19:37:11.189 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.188+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f70d810cda0 con 0x7f70d80fd520 2022-01-31T19:37:11.191 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.190+0000 7f70c6ffd700 1 -- 172.21.15.167:0/1002020467 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f70cc0330c0 con 0x7f70d80fd520 2022-01-31T19:37:11.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 msgr2=0x7f70d810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:11.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 --2- 172.21.15.167:0/1002020467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d810acd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f70cc00e040 tx=0x7f70cc006ee0).stop 2022-01-31T19:37:11.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 shutdown_connections 2022-01-31T19:37:11.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 --2- 172.21.15.167:0/1002020467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d80fd520 0x7f70d810acd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:11.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 >> 172.21.15.167:0/1002020467 conn(0x7f70d80f7670 msgr2=0x7f70d80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:11.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 shutdown_connections 2022-01-31T19:37:11.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:11.191+0000 7f70dcfa7700 1 -- 172.21.15.167:0/1002020467 wait complete. 2022-01-31T19:37:11.630 INFO:teuthology.orchestra.run.smithi167.stderr:Wrote config to /etc/ceph/ceph.conf 2022-01-31T19:37:11.630 INFO:teuthology.orchestra.run.smithi167.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-01-31T19:37:11.631 INFO:teuthology.orchestra.run.smithi167.stderr:Creating mgr... 2022-01-31T19:37:11.631 INFO:teuthology.orchestra.run.smithi167.stderr:Verifying port 9283 ... 2022-01-31T19:37:11.747 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Failed to reset failed state of unit ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mgr.smithi167.aciqpk.service: Unit ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mgr.smithi167.aciqpk.service not loaded. 2022-01-31T19:37:11.753 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: Created symlink /etc/systemd/system/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c.target.wants/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mgr.smithi167.aciqpk.service → /etc/systemd/system/ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@.service. 2022-01-31T19:37:12.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:12 smithi167 conmon[32206]: audit 2022-01-31T19:37:11.191658+0000 2022-01-31T19:37:12.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:12 smithi167 conmon[32206]: mon.smithi167 (mon.0) 6 : audit [INF] from='client.? 172.21.15.167:0/1002020467' entity='client.admin' 2022-01-31T19:37:12.551 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: disabled 2022-01-31T19:37:12.558 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: inactive 2022-01-31T19:37:12.559 INFO:teuthology.orchestra.run.smithi167.stderr:firewalld.service is not enabled 2022-01-31T19:37:12.560 INFO:teuthology.orchestra.run.smithi167.stderr:Not possible to enable service . firewalld.service is not available 2022-01-31T19:37:12.567 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: disabled 2022-01-31T19:37:12.574 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: inactive 2022-01-31T19:37:12.575 INFO:teuthology.orchestra.run.smithi167.stderr:firewalld.service is not enabled 2022-01-31T19:37:12.575 INFO:teuthology.orchestra.run.smithi167.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2022-01-31T19:37:12.576 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for mgr to start... 2022-01-31T19:37:12.576 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for mgr... 2022-01-31T19:37:13.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.109+0000 7f0294837700 1 Processor -- start 2022-01-31T19:37:13.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.109+0000 7f0294837700 1 -- start start 2022-01-31T19:37:13.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f0294837700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f02900feb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:13.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f0294837700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02900ff040 con 0x7f029006fc20 2022-01-31T19:37:13.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f028e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f02900feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:13.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f028e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f02900feb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60316/0 (socket says 172.21.15.167:60316) 2022-01-31T19:37:13.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f028e59c700 1 -- 172.21.15.167:0/849971051 learned_addr learned my addr 172.21.15.167:0/849971051 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:13.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f028e59c700 1 -- 172.21.15.167:0/849971051 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02900ff180 con 0x7f029006fc20 2022-01-31T19:37:13.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.110+0000 7f028e59c700 1 --2- 172.21.15.167:0/849971051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f02900feb50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f028000b5f0 tx=0x7f0280006ee0).ready entity=mon.0 client_cookie=1cf3a1a78a25d044 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:13.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.111+0000 7f028d59a700 1 -- 172.21.15.167:0/849971051 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f028000d600 con 0x7f029006fc20 2022-01-31T19:37:13.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.111+0000 7f028d59a700 1 -- 172.21.15.167:0/849971051 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f028000dbc0 con 0x7f029006fc20 2022-01-31T19:37:13.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.111+0000 7f028d59a700 1 -- 172.21.15.167:0/849971051 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0280011b90 con 0x7f029006fc20 2022-01-31T19:37:13.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.111+0000 7f0294837700 1 -- 172.21.15.167:0/849971051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 msgr2=0x7f02900feb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:13.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.111+0000 7f0294837700 1 --2- 172.21.15.167:0/849971051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f02900feb50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f028000b5f0 tx=0x7f0280006ee0).stop 2022-01-31T19:37:13.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.112+0000 7f0294837700 1 -- 172.21.15.167:0/849971051 shutdown_connections 2022-01-31T19:37:13.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.112+0000 7f0294837700 1 --2- 172.21.15.167:0/849971051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f02900feb50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:13.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.112+0000 7f0294837700 1 -- 172.21.15.167:0/849971051 >> 172.21.15.167:0/849971051 conn(0x7f02900f6530 msgr2=0x7f02900f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:13.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.112+0000 7f0294837700 1 -- 172.21.15.167:0/849971051 shutdown_connections 2022-01-31T19:37:13.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.112+0000 7f0294837700 1 -- 172.21.15.167:0/849971051 wait complete. 2022-01-31T19:37:13.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.112+0000 7f0294837700 1 Processor -- start 2022-01-31T19:37:13.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f0294837700 1 -- start start 2022-01-31T19:37:13.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f0294837700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f029010acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:13.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f0294837700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029010b1b0 con 0x7f029006fc20 2022-01-31T19:37:13.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f028e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f029010acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:13.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f028e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f029010acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60318/0 (socket says 172.21.15.167:60318) 2022-01-31T19:37:13.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f028e59c700 1 -- 172.21.15.167:0/196925753 learned_addr learned my addr 172.21.15.167:0/196925753 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:13.123 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f028e59c700 1 -- 172.21.15.167:0/196925753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028000b200 con 0x7f029006fc20 2022-01-31T19:37:13.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f028e59c700 1 --2- 172.21.15.167:0/196925753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f029010acc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0280006800 tx=0x7f0280006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:13.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.113+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f028001b410 con 0x7f029006fc20 2022-01-31T19:37:13.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.114+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029010b3b0 con 0x7f029006fc20 2022-01-31T19:37:13.124 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.114+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f028001b9d0 con 0x7f029006fc20 2022-01-31T19:37:13.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.114+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f028001a840 con 0x7f029006fc20 2022-01-31T19:37:13.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.114+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029010bf60 con 0x7f029006fc20 2022-01-31T19:37:13.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.114+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f028000f070 con 0x7f029006fc20 2022-01-31T19:37:13.125 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.114+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0280024ca0 con 0x7f029006fc20 2022-01-31T19:37:13.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.115+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0290046a20 con 0x7f029006fc20 2022-01-31T19:37:13.126 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.119+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f028001f030 con 0x7f029006fc20 2022-01-31T19:37:13.191 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.188+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f02900ffef0 con 0x7f029006fc20 2022-01-31T19:37:13.191 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.188+0000 7f027effd700 1 -- 172.21.15.167:0/196925753 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f0280032090 con 0x7f029006fc20 2022-01-31T19:37:13.192 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.192 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:13.192 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "fsid": "1c5a1df6-82cd-11ec-8c35-001a4aab830c", 2022-01-31T19:37:13.192 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "health": { 2022-01-31T19:37:13.193 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-01-31T19:37:13.193 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "checks": {}, 2022-01-31T19:37:13.193 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mutes": [] 2022-01-31T19:37:13.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-01-31T19:37:13.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum": [ 2022-01-31T19:37:13.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 0 2022-01-31T19:37:13.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:13.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum_names": [ 2022-01-31T19:37:13.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "smithi167" 2022-01-31T19:37:13.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:13.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum_age": 2, 2022-01-31T19:37:13.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "monmap": { 2022-01-31T19:37:13.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:13.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2022-01-31T19:37:13.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_mons": 1 2022-01-31T19:37:13.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osdmap": { 2022-01-31T19:37:13.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:13.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_osds": 0, 2022-01-31T19:37:13.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-01-31T19:37:13.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-01-31T19:37:13.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-01-31T19:37:13.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-01-31T19:37:13.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-01-31T19:37:13.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "pgmap": { 2022-01-31T19:37:13.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-01-31T19:37:13.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-01-31T19:37:13.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_pools": 0, 2022-01-31T19:37:13.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_objects": 0, 2022-01-31T19:37:13.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-01-31T19:37:13.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-01-31T19:37:13.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-01-31T19:37:13.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_total": 0 2022-01-31T19:37:13.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "fsmap": { 2022-01-31T19:37:13.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:13.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "by_rank": [], 2022-01-31T19:37:13.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "up:standby": 0 2022-01-31T19:37:13.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mgrmap": { 2022-01-31T19:37:13.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "available": false, 2022-01-31T19:37:13.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-01-31T19:37:13.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "modules": [ 2022-01-31T19:37:13.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "iostat", 2022-01-31T19:37:13.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "nfs", 2022-01-31T19:37:13.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "restful" 2022-01-31T19:37:13.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:13.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "services": {} 2022-01-31T19:37:13.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "servicemap": { 2022-01-31T19:37:13.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:13.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "modified": "2022-01-31T19:37:06.503929+0000", 2022-01-31T19:37:13.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "services": {} 2022-01-31T19:37:13.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:13.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "progress_events": {} 2022-01-31T19:37:13.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:13.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 msgr2=0x7f029010acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:13.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 --2- 172.21.15.167:0/196925753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f029010acc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0280006800 tx=0x7f0280006ee0).stop 2022-01-31T19:37:13.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 shutdown_connections 2022-01-31T19:37:13.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 --2- 172.21.15.167:0/196925753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f029006fc20 0x7f029010acc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:13.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 >> 172.21.15.167:0/196925753 conn(0x7f02900f6530 msgr2=0x7f02900f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:13.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 shutdown_connections 2022-01-31T19:37:13.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:13.190+0000 7f0294837700 1 -- 172.21.15.167:0/196925753 wait complete. 2022-01-31T19:37:13.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:13 smithi167 conmon[32206]: audit 2022-01-31T19:37:13.190304+0000 mon.smithi167 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.167:0/196925753' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-01-31T19:37:13.615 INFO:teuthology.orchestra.run.smithi167.stderr:mgr not available, waiting (1/15)... 2022-01-31T19:37:16.127 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.125+0000 7f30dfe73700 1 Processor -- start 2022-01-31T19:37:16.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.125+0000 7f30dfe73700 1 -- start start 2022-01-31T19:37:16.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30dfe73700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 0x7f30d80fa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:16.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30dfe73700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80fabb0 con 0x7f30d80fa280 2022-01-31T19:37:16.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30ddc0f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 0x7f30d80fa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:16.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30ddc0f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 0x7f30d80fa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60320/0 (socket says 172.21.15.167:60320) 2022-01-31T19:37:16.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30ddc0f700 1 -- 172.21.15.167:0/1162586200 learned_addr learned my addr 172.21.15.167:0/1162586200 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:16.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30ddc0f700 1 -- 172.21.15.167:0/1162586200 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80facf0 con 0x7f30d80fa280 2022-01-31T19:37:16.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.126+0000 7f30ddc0f700 1 --2- 172.21.15.167:0/1162586200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 0x7f30d80fa6c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f30d400b5f0 tx=0x7f30d4006ee0).ready entity=mon.0 client_cookie=136b312bf726104a server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:16.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.127+0000 7f30dcc0d700 1 -- 172.21.15.167:0/1162586200 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d400d600 con 0x7f30d80fa280 2022-01-31T19:37:16.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.127+0000 7f30dcc0d700 1 -- 172.21.15.167:0/1162586200 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f30d400dbc0 con 0x7f30d80fa280 2022-01-31T19:37:16.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.127+0000 7f30dcc0d700 1 -- 172.21.15.167:0/1162586200 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d4011b90 con 0x7f30d80fa280 2022-01-31T19:37:16.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 -- 172.21.15.167:0/1162586200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 msgr2=0x7f30d80fa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:16.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 --2- 172.21.15.167:0/1162586200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 0x7f30d80fa6c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f30d400b5f0 tx=0x7f30d4006ee0).stop 2022-01-31T19:37:16.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 -- 172.21.15.167:0/1162586200 shutdown_connections 2022-01-31T19:37:16.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 --2- 172.21.15.167:0/1162586200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d80fa280 0x7f30d80fa6c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:16.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 -- 172.21.15.167:0/1162586200 >> 172.21.15.167:0/1162586200 conn(0x7f30d80f5ca0 msgr2=0x7f30d80f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:16.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 -- 172.21.15.167:0/1162586200 shutdown_connections 2022-01-31T19:37:16.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.128+0000 7f30dfe73700 1 -- 172.21.15.167:0/1162586200 wait complete. 2022-01-31T19:37:16.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30dfe73700 1 Processor -- start 2022-01-31T19:37:16.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30dfe73700 1 -- start start 2022-01-31T19:37:16.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30dfe73700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 0x7f30d810b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:16.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30dfe73700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d810b6e0 con 0x7f30d810adb0 2022-01-31T19:37:16.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30ddc0f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 0x7f30d810b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:16.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30ddc0f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 0x7f30d810b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60322/0 (socket says 172.21.15.167:60322) 2022-01-31T19:37:16.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30ddc0f700 1 -- 172.21.15.167:0/3692144362 learned_addr learned my addr 172.21.15.167:0/3692144362 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:16.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30ddc0f700 1 -- 172.21.15.167:0/3692144362 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d400b200 con 0x7f30d810adb0 2022-01-31T19:37:16.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.129+0000 7f30ddc0f700 1 --2- 172.21.15.167:0/3692144362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 0x7f30d810b1f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f30d4013000 tx=0x7f30d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:16.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d400d890 con 0x7f30d810adb0 2022-01-31T19:37:16.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d810b8e0 con 0x7f30d810adb0 2022-01-31T19:37:16.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f30d400d9f0 con 0x7f30d810adb0 2022-01-31T19:37:16.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d401a5e0 con 0x7f30d810adb0 2022-01-31T19:37:16.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d810c350 con 0x7f30d810adb0 2022-01-31T19:37:16.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f30d402b070 con 0x7f30d810adb0 2022-01-31T19:37:16.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.130+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f30d4024a10 con 0x7f30d810adb0 2022-01-31T19:37:16.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.131+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d80591e0 con 0x7f30d810adb0 2022-01-31T19:37:16.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.133+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f30d401f020 con 0x7f30d810adb0 2022-01-31T19:37:16.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.203+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f30d80ffee0 con 0x7f30d810adb0 2022-01-31T19:37:16.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.203+0000 7f30c67fc700 1 -- 172.21.15.167:0/3692144362 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f30d4016070 con 0x7f30d810adb0 2022-01-31T19:37:16.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:16.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "fsid": "1c5a1df6-82cd-11ec-8c35-001a4aab830c", 2022-01-31T19:37:16.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "health": { 2022-01-31T19:37:16.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-01-31T19:37:16.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "checks": {}, 2022-01-31T19:37:16.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mutes": [] 2022-01-31T19:37:16.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-01-31T19:37:16.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum": [ 2022-01-31T19:37:16.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 0 2022-01-31T19:37:16.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:16.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum_names": [ 2022-01-31T19:37:16.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "smithi167" 2022-01-31T19:37:16.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:16.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum_age": 5, 2022-01-31T19:37:16.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "monmap": { 2022-01-31T19:37:16.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:16.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2022-01-31T19:37:16.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_mons": 1 2022-01-31T19:37:16.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osdmap": { 2022-01-31T19:37:16.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:16.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_osds": 0, 2022-01-31T19:37:16.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-01-31T19:37:16.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-01-31T19:37:16.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-01-31T19:37:16.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-01-31T19:37:16.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-01-31T19:37:16.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "pgmap": { 2022-01-31T19:37:16.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-01-31T19:37:16.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-01-31T19:37:16.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_pools": 0, 2022-01-31T19:37:16.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_objects": 0, 2022-01-31T19:37:16.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-01-31T19:37:16.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-01-31T19:37:16.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-01-31T19:37:16.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_total": 0 2022-01-31T19:37:16.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "fsmap": { 2022-01-31T19:37:16.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:16.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "by_rank": [], 2022-01-31T19:37:16.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "up:standby": 0 2022-01-31T19:37:16.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mgrmap": { 2022-01-31T19:37:16.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "available": false, 2022-01-31T19:37:16.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-01-31T19:37:16.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "modules": [ 2022-01-31T19:37:16.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "iostat", 2022-01-31T19:37:16.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "nfs", 2022-01-31T19:37:16.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "restful" 2022-01-31T19:37:16.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:16.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "services": {} 2022-01-31T19:37:16.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "servicemap": { 2022-01-31T19:37:16.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:16.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "modified": "2022-01-31T19:37:06.503929+0000", 2022-01-31T19:37:16.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "services": {} 2022-01-31T19:37:16.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:16.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "progress_events": {} 2022-01-31T19:37:16.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:16.224 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.204+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 msgr2=0x7f30d810b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:16.224 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.204+0000 7f30dfe73700 1 --2- 172.21.15.167:0/3692144362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 0x7f30d810b1f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f30d4013000 tx=0x7f30d4006ee0).stop 2022-01-31T19:37:16.224 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.204+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 shutdown_connections 2022-01-31T19:37:16.224 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.204+0000 7f30dfe73700 1 --2- 172.21.15.167:0/3692144362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d810adb0 0x7f30d810b1f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:16.225 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.204+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 >> 172.21.15.167:0/3692144362 conn(0x7f30d80f5ca0 msgr2=0x7f30d80f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:16.225 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.207+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 shutdown_connections 2022-01-31T19:37:16.225 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:16.207+0000 7f30dfe73700 1 -- 172.21.15.167:0/3692144362 wait complete. 2022-01-31T19:37:16.553 INFO:teuthology.orchestra.run.smithi167.stderr:mgr not available, waiting (2/15)... 2022-01-31T19:37:16.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:16 smithi167 conmon[32206]: audit 2022-01-31T19:37:16.204987+0000 mon.smithi167 (mon. 2022-01-31T19:37:16.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:16 smithi167 conmon[32206]: 0) 8 : audit [DBG] from='client.? 172.21.15.167:0/3692144362' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-01-31T19:37:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: cluster 2022-01-31T19:37:17.596568+0000 mon.smithi167 (mon.0 2022-01-31T19:37:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: ) 9 : cluster [INF] Activating manager daemon smithi167.aciqpk 2022-01-31T19:37:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: cluster 2022-01-31T19:37:17.599236+0000 mon.smithi167 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi167.aciqpk(active, starting, since 0.0027712s) 2022-01-31T19:37:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:37:17.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: 37:17.600107+0000 mon.smithi167 (mon.0) 11 2022-01-31T19:37:17.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: : audit [DBG] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:37:17.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:37:17.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: T19:37:17.600285+0000 mon.smithi167 ( 2022-01-31T19:37:17.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:37:17.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:17.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: -01-31T19:37:17 2022-01-31T19:37:17.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: .600463+0000 mon.smithi167 (mon.0) 13 : audit 2022-01-31T19:37:17.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: [DBG] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:37:17.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:37:17.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: T19:37:17.600655+0000 2022-01-31T19:37:17.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: mon.smithi167 (mon.0) 14 : audit [DBG] 2022-01-31T19:37:17.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:37:17.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:37:17.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: :37:17.600840+0000 mon.smithi167 (mon.0 2022-01-31T19:37:17.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: ) 15 : audit [DBG] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:37:17.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: cluster 2022- 2022-01-31T19:37:17.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: 01-31T19:37:17.613773 2022-01-31T19:37:17.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 16 : cluster [INF] 2022-01-31T19:37:17.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: Manager daemon smithi167.aciqpk is now available 2022-01-31T19:37:17.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:37:17.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: :37:17.621132+0000 mon.smithi167 (mon. 2022-01-31T19:37:17.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: 0) 17 : audit [INF] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:17.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:17.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: -01-31T19:37:17. 2022-01-31T19:37:17.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: 625768+0000 mon.smithi167 (mon.0) 18 : audit 2022-01-31T19:37:17.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: [INF] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:37:17.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:37:17.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: T19:37:17.626911+0000 mon.smithi167 ( 2022-01-31T19:37:17.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:37:17.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:17.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: -01-31T19:37:17.631919 2022-01-31T19:37:17.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 20 : audit [INF] 2022-01-31T19:37:17.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:17.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:37:17.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: :37:17.633765+0000 mon.smithi167 (mon. 2022-01-31T19:37:17.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:17 smithi167 conmon[32206]: 0) 21 : audit [INF] from='mgr.14100 172.21.15.167:0/1581026746' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:19.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:19 smithi167 conmon[32206]: cluster 2022-01-31T19:37:18.602496+0000 mon.smithi167 (mon.0) 22 : cluster [DBG] 2022-01-31T19:37:19.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:19 smithi167 conmon[32206]: mgrmap e3: smithi167.aciqpk(active, since 1.00603s) 2022-01-31T19:37:21.128 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.904+0000 7f7e21bf1700 1 Processor -- start 2022-01-31T19:37:21.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.904+0000 7f7e21bf1700 1 -- start start 2022-01-31T19:37:21.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.904+0000 7f7e21bf1700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c0fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:21.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.904+0000 7f7e21bf1700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0fcb20 con 0x7f7e1c0fc210 2022-01-31T19:37:21.129 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e20bef700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c0fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:21.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e20bef700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c0fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60342/0 (socket says 172.21.15.167:60342) 2022-01-31T19:37:21.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e20bef700 1 -- 172.21.15.167:0/2558508762 learned_addr learned my addr 172.21.15.167:0/2558508762 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:21.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e20bef700 1 -- 172.21.15.167:0/2558508762 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0fcc60 con 0x7f7e1c0fc210 2022-01-31T19:37:21.130 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e20bef700 1 --2- 172.21.15.167:0/2558508762 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c0fc630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7e1000b5f0 tx=0x7f7e10006ee0).ready entity=mon.0 client_cookie=35c1bd17adc4ab61 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:21.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e1b7fe700 1 -- 172.21.15.167:0/2558508762 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e1000d600 con 0x7f7e1c0fc210 2022-01-31T19:37:21.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e1b7fe700 1 -- 172.21.15.167:0/2558508762 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f7e1000dbc0 con 0x7f7e1c0fc210 2022-01-31T19:37:21.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.905+0000 7f7e1b7fe700 1 -- 172.21.15.167:0/2558508762 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e10011b90 con 0x7f7e1c0fc210 2022-01-31T19:37:21.131 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 -- 172.21.15.167:0/2558508762 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 msgr2=0x7f7e1c0fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:21.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 --2- 172.21.15.167:0/2558508762 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c0fc630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7e1000b5f0 tx=0x7f7e10006ee0).stop 2022-01-31T19:37:21.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 -- 172.21.15.167:0/2558508762 shutdown_connections 2022-01-31T19:37:21.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 --2- 172.21.15.167:0/2558508762 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c0fc630 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:21.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 -- 172.21.15.167:0/2558508762 >> 172.21.15.167:0/2558508762 conn(0x7f7e1c0f6340 msgr2=0x7f7e1c0f8780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:21.132 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 -- 172.21.15.167:0/2558508762 shutdown_connections 2022-01-31T19:37:21.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.906+0000 7f7e21bf1700 1 -- 172.21.15.167:0/2558508762 wait complete. 2022-01-31T19:37:21.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e21bf1700 1 Processor -- start 2022-01-31T19:37:21.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e21bf1700 1 -- start start 2022-01-31T19:37:21.133 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e21bf1700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c10d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:21.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e21bf1700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c06fc30 con 0x7f7e1c0fc210 2022-01-31T19:37:21.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e20bef700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c10d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:21.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e20bef700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c10d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60344/0 (socket says 172.21.15.167:60344) 2022-01-31T19:37:21.134 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e20bef700 1 -- 172.21.15.167:0/248562111 learned_addr learned my addr 172.21.15.167:0/248562111 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:21.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e20bef700 1 -- 172.21.15.167:0/248562111 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1000b200 con 0x7f7e1c0fc210 2022-01-31T19:37:21.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.907+0000 7f7e20bef700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c10d760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7e1000e040 tx=0x7f7e10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:21.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.908+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e1001b410 con 0x7f7e1c0fc210 2022-01-31T19:37:21.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.908+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f7e1001b9d0 con 0x7f7e1c0fc210 2022-01-31T19:37:21.135 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.908+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c06fe30 con 0x7f7e1c0fc210 2022-01-31T19:37:21.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.908+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e1001a7e0 con 0x7f7e1c0fc210 2022-01-31T19:37:21.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.908+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c10de10 con 0x7f7e1c0fc210 2022-01-31T19:37:21.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.908+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f7e1000f070 con 0x7f7e1c0fc210 2022-01-31T19:37:21.136 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.909+0000 7f7e197fa700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f7e00030f40 0x7f7e00033400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:21.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.909+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7e10042a20 con 0x7f7e1c0fc210 2022-01-31T19:37:21.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.909+0000 7f7e1bfff700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f7e00030f40 0x7f7e00033400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:21.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.909+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c046a20 con 0x7f7e1c0fc210 2022-01-31T19:37:21.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.909+0000 7f7e1bfff700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f7e00030f40 0x7f7e00033400 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7e080096a0 tx=0x7f7e08006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:21.137 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:20.913+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7e10016030 con 0x7f7e1c0fc210 2022-01-31T19:37:21.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.119+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7e1c0fb470 con 0x7f7e1c0fc210 2022-01-31T19:37:21.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.120+0000 7f7e197fa700 1 -- 172.21.15.167:0/248562111 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f7e10032090 con 0x7f7e1c0fc210 2022-01-31T19:37:21.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.138 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:21.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "fsid": "1c5a1df6-82cd-11ec-8c35-001a4aab830c", 2022-01-31T19:37:21.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "health": { 2022-01-31T19:37:21.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2022-01-31T19:37:21.139 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "checks": {}, 2022-01-31T19:37:21.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mutes": [] 2022-01-31T19:37:21.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "election_epoch": 5, 2022-01-31T19:37:21.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum": [ 2022-01-31T19:37:21.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 0 2022-01-31T19:37:21.141 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:21.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum_names": [ 2022-01-31T19:37:21.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "smithi167" 2022-01-31T19:37:21.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:21.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "quorum_age": 10, 2022-01-31T19:37:21.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "monmap": { 2022-01-31T19:37:21.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:21.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2022-01-31T19:37:21.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_mons": 1 2022-01-31T19:37:21.145 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.146 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osdmap": { 2022-01-31T19:37:21.146 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:21.146 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_osds": 0, 2022-01-31T19:37:21.147 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_up_osds": 0, 2022-01-31T19:37:21.147 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osd_up_since": 0, 2022-01-31T19:37:21.147 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_in_osds": 0, 2022-01-31T19:37:21.148 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "osd_in_since": 0, 2022-01-31T19:37:21.148 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2022-01-31T19:37:21.149 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.149 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "pgmap": { 2022-01-31T19:37:21.149 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "pgs_by_state": [], 2022-01-31T19:37:21.150 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_pgs": 0, 2022-01-31T19:37:21.150 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_pools": 0, 2022-01-31T19:37:21.150 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_objects": 0, 2022-01-31T19:37:21.151 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "data_bytes": 0, 2022-01-31T19:37:21.151 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_used": 0, 2022-01-31T19:37:21.151 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_avail": 0, 2022-01-31T19:37:21.151 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "bytes_total": 0 2022-01-31T19:37:21.152 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.152 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "fsmap": { 2022-01-31T19:37:21.152 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:21.152 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "by_rank": [], 2022-01-31T19:37:21.153 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "up:standby": 0 2022-01-31T19:37:21.153 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.153 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mgrmap": { 2022-01-31T19:37:21.153 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "available": true, 2022-01-31T19:37:21.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_standbys": 0, 2022-01-31T19:37:21.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "modules": [ 2022-01-31T19:37:21.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "iostat", 2022-01-31T19:37:21.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "nfs", 2022-01-31T19:37:21.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "restful" 2022-01-31T19:37:21.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ], 2022-01-31T19:37:21.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "services": {} 2022-01-31T19:37:21.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "servicemap": { 2022-01-31T19:37:21.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 1, 2022-01-31T19:37:21.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "modified": "2022-01-31T19:37:06.503929+0000", 2022-01-31T19:37:21.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "services": {} 2022-01-31T19:37:21.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: }, 2022-01-31T19:37:21.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "progress_events": {} 2022-01-31T19:37:21.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:21.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f7e00030f40 msgr2=0x7f7e00033400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:21.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f7e00030f40 0x7f7e00033400 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7e080096a0 tx=0x7f7e08006b40).stop 2022-01-31T19:37:21.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 msgr2=0x7f7e1c10d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:21.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c10d760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7e1000e040 tx=0x7f7e10006ee0).stop 2022-01-31T19:37:21.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 shutdown_connections 2022-01-31T19:37:21.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f7e00030f40 0x7f7e00033400 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:21.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 --2- 172.21.15.167:0/248562111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0fc210 0x7f7e1c10d760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:21.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.121+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 >> 172.21.15.167:0/248562111 conn(0x7f7e1c0f6340 msgr2=0x7f7e1c0f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:21.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.122+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 shutdown_connections 2022-01-31T19:37:21.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:21.122+0000 7f7e21bf1700 1 -- 172.21.15.167:0/248562111 wait complete. 2022-01-31T19:37:21.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:21 smithi167 conmon[32206]: cluster 2022-01-31T19:37:20.602464+0000 mon.smithi167 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi167.aciqpk(active, since 3s) 2022-01-31T19:37:21.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:21 smithi167 conmon[32206]: audit 2022-01-31T19:37:21.121422+0000 mon.smithi167 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.167:0/248562111' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-01-31T19:37:22.906 INFO:teuthology.orchestra.run.smithi167.stderr:mgr is available 2022-01-31T19:37:24.660 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.488+0000 7f1809be5700 1 Processor -- start 2022-01-31T19:37:24.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.488+0000 7f1809be5700 1 -- start start 2022-01-31T19:37:24.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.488+0000 7f1809be5700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f18040fc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:24.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f1809be5700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18040fccf0 con 0x7f18040fc3e0 2022-01-31T19:37:24.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f18040fc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:24.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f18040fc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60348/0 (socket says 172.21.15.167:60348) 2022-01-31T19:37:24.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18037fe700 1 -- 172.21.15.167:0/696269387 learned_addr learned my addr 172.21.15.167:0/696269387 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:24.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18037fe700 1 -- 172.21.15.167:0/696269387 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18040fce30 con 0x7f18040fc3e0 2022-01-31T19:37:24.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18037fe700 1 --2- 172.21.15.167:0/696269387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f18040fc800 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f17fc009960 tx=0x7f17fc0092d0).ready entity=mon.0 client_cookie=b843de787e78c288 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:24.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18027fc700 1 -- 172.21.15.167:0/696269387 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc00b9c0 con 0x7f18040fc3e0 2022-01-31T19:37:24.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18027fc700 1 -- 172.21.15.167:0/696269387 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f17fc011400 con 0x7f18040fc3e0 2022-01-31T19:37:24.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.489+0000 7f18027fc700 1 -- 172.21.15.167:0/696269387 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc010580 con 0x7f18040fc3e0 2022-01-31T19:37:24.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 -- 172.21.15.167:0/696269387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 msgr2=0x7f18040fc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:24.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 --2- 172.21.15.167:0/696269387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f18040fc800 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f17fc009960 tx=0x7f17fc0092d0).stop 2022-01-31T19:37:24.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 -- 172.21.15.167:0/696269387 shutdown_connections 2022-01-31T19:37:24.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 --2- 172.21.15.167:0/696269387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f18040fc800 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:24.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 -- 172.21.15.167:0/696269387 >> 172.21.15.167:0/696269387 conn(0x7f18040f6530 msgr2=0x7f18040f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:24.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 -- 172.21.15.167:0/696269387 shutdown_connections 2022-01-31T19:37:24.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 -- 172.21.15.167:0/696269387 wait complete. 2022-01-31T19:37:24.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.490+0000 7f1809be5700 1 Processor -- start 2022-01-31T19:37:24.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f1809be5700 1 -- start start 2022-01-31T19:37:24.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f1809be5700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f1804108a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:24.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f1809be5700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1804108f30 con 0x7f18040fc3e0 2022-01-31T19:37:24.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f18037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f1804108a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:24.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f18037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f1804108a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60350/0 (socket says 172.21.15.167:60350) 2022-01-31T19:37:24.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f18037fe700 1 -- 172.21.15.167:0/4268043523 learned_addr learned my addr 172.21.15.167:0/4268043523 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:24.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f18037fe700 1 -- 172.21.15.167:0/4268043523 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc009570 con 0x7f18040fc3e0 2022-01-31T19:37:24.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f18037fe700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f1804108a40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f17fc00b5c0 tx=0x7f17fc0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:24.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.491+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc010840 con 0x7f18040fc3e0 2022-01-31T19:37:24.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.492+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f17fc010e00 con 0x7f18040fc3e0 2022-01-31T19:37:24.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.492+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc01be90 con 0x7f18040fc3e0 2022-01-31T19:37:24.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.492+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1804109130 con 0x7f18040fc3e0 2022-01-31T19:37:24.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.492+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18041095c0 con 0x7f18040fc3e0 2022-01-31T19:37:24.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.493+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1804045bb0 con 0x7f18040fc3e0 2022-01-31T19:37:24.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.493+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f17fc00f070 con 0x7f18040fc3e0 2022-01-31T19:37:24.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.493+0000 7f17fbfff700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f17e40331e0 0x7f17e40356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:24.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.493+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f17fc043990 con 0x7f18040fc3e0 2022-01-31T19:37:24.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.493+0000 7f1802ffd700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f17e40331e0 0x7f17e40356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:24.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.494+0000 7f1802ffd700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f17e40331e0 0x7f17e40356a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f17ec0096a0 tx=0x7f17ec006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:24.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.497+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f17fc011e60 con 0x7f18040fc3e0 2022-01-31T19:37:24.670 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.654+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f18040fb660 con 0x7f18040fc3e0 2022-01-31T19:37:24.670 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.655+0000 7f17fbfff700 1 -- 172.21.15.167:0/4268043523 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f17fc02c340 con 0x7f18040fc3e0 2022-01-31T19:37:24.670 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.670 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: [global] 2022-01-31T19:37:24.671 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: fsid = 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:24.671 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: log_to_journald = false 2022-01-31T19:37:24.671 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2022-01-31T19:37:24.671 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2022-01-31T19:37:24.671 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2022-01-31T19:37:24.672 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2022-01-31T19:37:24.672 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.672 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: [mgr] 2022-01-31T19:37:24.672 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2022-01-31T19:37:24.673 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.673 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: [osd] 2022-01-31T19:37:24.673 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2022-01-31T19:37:24.673 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2022-01-31T19:37:24.673 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2022-01-31T19:37:24.674 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2022-01-31T19:37:24.674 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2022-01-31T19:37:24.674 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f17e40331e0 msgr2=0x7f17e40356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:24.674 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f17e40331e0 0x7f17e40356a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f17ec0096a0 tx=0x7f17ec006b40).stop 2022-01-31T19:37:24.675 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 msgr2=0x7f1804108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:24.675 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f1804108a40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f17fc00b5c0 tx=0x7f17fc0092d0).stop 2022-01-31T19:37:24.675 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 shutdown_connections 2022-01-31T19:37:24.675 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f17e40331e0 0x7f17e40356a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:24.675 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 --2- 172.21.15.167:0/4268043523 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18040fc3e0 0x7f1804108a40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:24.676 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 >> 172.21.15.167:0/4268043523 conn(0x7f18040f6530 msgr2=0x7f18040ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:24.676 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 shutdown_connections 2022-01-31T19:37:24.676 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:24.657+0000 7f1809be5700 1 -- 172.21.15.167:0/4268043523 wait complete. 2022-01-31T19:37:24.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:24 smithi167 conmon[32206]: audit 2022-01-31T19:37:24.656300+0000 2022-01-31T19:37:24.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:24 smithi167 conmon[32206]: mon.smithi167 (mon.0) 25 : audit [INF] from='client.? 172.21.15.167:0/4268043523' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-01-31T19:37:26.955 INFO:teuthology.orchestra.run.smithi167.stderr:Enabling cephadm module... 2022-01-31T19:37:28.835 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.737+0000 7ff8f8466700 1 Processor -- start 2022-01-31T19:37:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.737+0000 7ff8f8466700 1 -- start start 2022-01-31T19:37:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f8466700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f00fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f8466700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00fce30 con 0x7ff8f00fc470 2022-01-31T19:37:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f6202700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f00fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f6202700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f00fc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60354/0 (socket says 172.21.15.167:60354) 2022-01-31T19:37:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f6202700 1 -- 172.21.15.167:0/4182902357 learned_addr learned my addr 172.21.15.167:0/4182902357 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f6202700 1 -- 172.21.15.167:0/4182902357 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00fcf70 con 0x7ff8f00fc470 2022-01-31T19:37:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f6202700 1 --2- 172.21.15.167:0/4182902357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f00fc870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff8ec00b5f0 tx=0x7ff8ec006ee0).ready entity=mon.0 client_cookie=8431bc84a9da9ef1 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.738+0000 7ff8f5200700 1 -- 172.21.15.167:0/4182902357 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec00d600 con 0x7ff8f00fc470 2022-01-31T19:37:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f5200700 1 -- 172.21.15.167:0/4182902357 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff8ec00dbc0 con 0x7ff8f00fc470 2022-01-31T19:37:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f5200700 1 -- 172.21.15.167:0/4182902357 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec011b90 con 0x7ff8f00fc470 2022-01-31T19:37:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 -- 172.21.15.167:0/4182902357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 msgr2=0x7ff8f00fc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 --2- 172.21.15.167:0/4182902357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f00fc870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff8ec00b5f0 tx=0x7ff8ec006ee0).stop 2022-01-31T19:37:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 -- 172.21.15.167:0/4182902357 shutdown_connections 2022-01-31T19:37:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 --2- 172.21.15.167:0/4182902357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f00fc870 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 -- 172.21.15.167:0/4182902357 >> 172.21.15.167:0/4182902357 conn(0x7ff8f00f5ca0 msgr2=0x7ff8f00f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 -- 172.21.15.167:0/4182902357 shutdown_connections 2022-01-31T19:37:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.739+0000 7ff8f8466700 1 -- 172.21.15.167:0/4182902357 wait complete. 2022-01-31T19:37:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f8466700 1 Processor -- start 2022-01-31T19:37:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f8466700 1 -- start start 2022-01-31T19:37:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f8466700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f010cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f8466700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f010d410 con 0x7ff8f00fc470 2022-01-31T19:37:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f6202700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f010cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f6202700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f010cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60356/0 (socket says 172.21.15.167:60356) 2022-01-31T19:37:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.740+0000 7ff8f6202700 1 -- 172.21.15.167:0/319819611 learned_addr learned my addr 172.21.15.167:0/319819611 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8f6202700 1 -- 172.21.15.167:0/319819611 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec00b200 con 0x7ff8f00fc470 2022-01-31T19:37:28.842 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8f6202700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f010cf20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff8ec000c00 tx=0x7ff8ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:28.842 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec01b410 con 0x7ff8f00fc470 2022-01-31T19:37:28.842 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff8ec01b9d0 con 0x7ff8f00fc470 2022-01-31T19:37:28.843 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec01a720 con 0x7ff8f00fc470 2022-01-31T19:37:28.843 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f010d610 con 0x7ff8f00fc470 2022-01-31T19:37:28.843 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.741+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f010daa0 con 0x7ff8f00fc470 2022-01-31T19:37:28.843 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.742+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7ff8ec02b070 con 0x7ff8f00fc470 2022-01-31T19:37:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.742+0000 7ff8deffd700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7ff8d8030fa0 0x7ff8d8033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.742+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff8ec042c30 con 0x7ff8f00fc470 2022-01-31T19:37:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.742+0000 7ff8f5a01700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7ff8d8030fa0 0x7ff8d8033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.742+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f00591e0 con 0x7ff8f00fc470 2022-01-31T19:37:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.742+0000 7ff8f5a01700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7ff8d8030fa0 0x7ff8d8033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff8e00096a0 tx=0x7ff8e0006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:28.845 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.747+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff8ec016020 con 0x7ff8f00fc470 2022-01-31T19:37:28.942 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.940+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff8f01021e0 con 0x7ff8f00fc470 2022-01-31T19:37:28.994 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.993+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7ff8ec00f070 con 0x7ff8f00fc470 2022-01-31T19:37:28.995 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.994+0000 7ff8deffd700 1 -- 172.21.15.167:0/319819611 <== mon.0 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff8ec042570 con 0x7ff8f00fc470 2022-01-31T19:37:28.998 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7ff8d8030fa0 msgr2=0x7ff8d8033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7ff8d8030fa0 0x7ff8d8033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff8e00096a0 tx=0x7ff8e0006b40).stop 2022-01-31T19:37:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 msgr2=0x7ff8f010cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f010cf20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff8ec000c00 tx=0x7ff8ec006ee0).stop 2022-01-31T19:37:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 shutdown_connections 2022-01-31T19:37:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7ff8d8030fa0 0x7ff8d8033460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 --2- 172.21.15.167:0/319819611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8f00fc470 0x7ff8f010cf20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 >> 172.21.15.167:0/319819611 conn(0x7ff8f00f5ca0 msgr2=0x7ff8f00f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 shutdown_connections 2022-01-31T19:37:29.001 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:28.997+0000 7ff8f8466700 1 -- 172.21.15.167:0/319819611 wait complete. 2022-01-31T19:37:29.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:28 smithi167 conmon[32206]: audit 2022-01-31T19:37:28.941691+0000 mon.smithi167 (mon.0) 26 : audit [INF] from='client.? 172.21.15.167:0/319819611' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-01-31T19:37:30.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:29 smithi167 conmon[32206]: audit 2022-01-31T19:37:28.994918+0000 mon.smithi167 (mon.0) 27 : audit 2022-01-31T19:37:30.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:29 smithi167 conmon[32206]: [INF] from='client.? 172.21.15.167:0/319819611' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-01-31T19:37:30.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:29 smithi167 conmon[32206]: cluster 2022-01-31T19:37:28.994974+0000 mon.smithi167 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi167.aciqpk(active, since 11s) 2022-01-31T19:37:31.585 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.582+0000 7f675ca36700 1 Processor -- start 2022-01-31T19:37:31.585 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.582+0000 7f675ca36700 1 -- start start 2022-01-31T19:37:31.586 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.582+0000 7f675ca36700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f6758070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:31.586 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.582+0000 7f675ca36700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6758070740 con 0x7f6758071d80 2022-01-31T19:37:31.586 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f6758070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:31.586 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f6758070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60372/0 (socket says 172.21.15.167:60372) 2022-01-31T19:37:31.587 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675659c700 1 -- 172.21.15.167:0/627780440 learned_addr learned my addr 172.21.15.167:0/627780440 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:31.587 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675659c700 1 -- 172.21.15.167:0/627780440 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6758070880 con 0x7f6758071d80 2022-01-31T19:37:31.587 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675659c700 1 --2- 172.21.15.167:0/627780440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f6758070250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f674c00b5f0 tx=0x7f674c006ee0).ready entity=mon.0 client_cookie=f23a489157ff3929 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:31.587 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675559a700 1 -- 172.21.15.167:0/627780440 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674c00d600 con 0x7f6758071d80 2022-01-31T19:37:31.588 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675559a700 1 -- 172.21.15.167:0/627780440 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f674c00dbc0 con 0x7f6758071d80 2022-01-31T19:37:31.588 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.583+0000 7f675559a700 1 -- 172.21.15.167:0/627780440 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674c011b90 con 0x7f6758071d80 2022-01-31T19:37:31.588 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 -- 172.21.15.167:0/627780440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 msgr2=0x7f6758070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:31.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 --2- 172.21.15.167:0/627780440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f6758070250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f674c00b5f0 tx=0x7f674c006ee0).stop 2022-01-31T19:37:31.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 -- 172.21.15.167:0/627780440 shutdown_connections 2022-01-31T19:37:31.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 --2- 172.21.15.167:0/627780440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f6758070250 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:31.590 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 -- 172.21.15.167:0/627780440 >> 172.21.15.167:0/627780440 conn(0x7f67580f7640 msgr2=0x7f67580f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:31.591 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 -- 172.21.15.167:0/627780440 shutdown_connections 2022-01-31T19:37:31.591 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.584+0000 7f675ca36700 1 -- 172.21.15.167:0/627780440 wait complete. 2022-01-31T19:37:31.594 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675ca36700 1 Processor -- start 2022-01-31T19:37:31.594 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675ca36700 1 -- start start 2022-01-31T19:37:31.595 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675ca36700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f675810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:31.595 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675ca36700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675810d3b0 con 0x7f6758071d80 2022-01-31T19:37:31.595 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f675810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:31.596 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675659c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f675810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60374/0 (socket says 172.21.15.167:60374) 2022-01-31T19:37:31.596 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.585+0000 7f675659c700 1 -- 172.21.15.167:0/2362892996 learned_addr learned my addr 172.21.15.167:0/2362892996 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:31.596 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f675659c700 1 -- 172.21.15.167:0/2362892996 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c00b200 con 0x7f6758071d80 2022-01-31T19:37:31.596 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f675659c700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f675810cec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f674c006800 tx=0x7f674c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:31.596 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674c01b410 con 0x7f6758071d80 2022-01-31T19:37:31.597 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f674c01b9d0 con 0x7f6758071d80 2022-01-31T19:37:31.597 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674c01a840 con 0x7f6758071d80 2022-01-31T19:37:31.597 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675810d5b0 con 0x7f6758071d80 2022-01-31T19:37:31.597 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.586+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675810e1c0 con 0x7f6758071d80 2022-01-31T19:37:31.598 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.587+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f674c00f070 con 0x7f6758071d80 2022-01-31T19:37:31.598 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.587+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6758046a20 con 0x7f6758071d80 2022-01-31T19:37:31.598 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.587+0000 7f6746ffd700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 0x7f673c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:31.599 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.588+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f674c042bc0 con 0x7f6758071d80 2022-01-31T19:37:31.599 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.588+0000 7f6755d9b700 1 -- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 msgr2=0x7f673c033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/3497254897 2022-01-31T19:37:31.599 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.588+0000 7f6755d9b700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 0x7f673c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2022-01-31T19:37:31.600 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.592+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f674c01b570 con 0x7f6758071d80 2022-01-31T19:37:31.793 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.788+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f67580591e0 con 0x7f6758071d80 2022-01-31T19:37:31.794 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.788+0000 7f6746ffd700 1 -- 172.21.15.167:0/2362892996 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f674c016070 con 0x7f6758071d80 2022-01-31T19:37:31.794 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:31.795 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 5, 2022-01-31T19:37:31.795 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "available": true, 2022-01-31T19:37:31.796 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "active_name": "smithi167.aciqpk", 2022-01-31T19:37:31.796 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_standby": 0 2022-01-31T19:37:31.797 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:31.797 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.789+0000 7f6755d9b700 1 -- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 msgr2=0x7f673c033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/3497254897 2022-01-31T19:37:31.798 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.789+0000 7f6755d9b700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 0x7f673c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2022-01-31T19:37:31.798 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 msgr2=0x7f673c033460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-01-31T19:37:31.799 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 0x7f673c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:31.799 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 msgr2=0x7f675810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:31.799 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f675810cec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f674c006800 tx=0x7f674c006ee0).stop 2022-01-31T19:37:31.800 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 shutdown_connections 2022-01-31T19:37:31.800 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f673c030fa0 0x7f673c033460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:31.801 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 --2- 172.21.15.167:0/2362892996 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6758071d80 0x7f675810cec0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:31.801 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.790+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 >> 172.21.15.167:0/2362892996 conn(0x7f67580f7640 msgr2=0x7f67580f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:31.802 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.792+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 shutdown_connections 2022-01-31T19:37:31.802 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:31.792+0000 7f675ca36700 1 -- 172.21.15.167:0/2362892996 wait complete. 2022-01-31T19:37:32.138 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for the mgr to restart... 2022-01-31T19:37:32.139 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for mgr epoch 5... 2022-01-31T19:37:32.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:31 smithi167 conmon[32206]: audit 2022-01-31T19:37:31.789127+0000 mon.smithi167 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.167:0/2362892996' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-01-31T19:37:32.582 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.579+0000 7f9466572700 1 Processor -- start 2022-01-31T19:37:32.582 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.579+0000 7f9466572700 1 -- start start 2022-01-31T19:37:32.583 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.579+0000 7f9466572700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94600fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:32.583 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f9466572700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94600fc4a0 con 0x7f94600fbb90 2022-01-31T19:37:32.583 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94600fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:32.583 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94600fbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60382/0 (socket says 172.21.15.167:60382) 2022-01-31T19:37:32.584 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945ffff700 1 -- 172.21.15.167:0/406249929 learned_addr learned my addr 172.21.15.167:0/406249929 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:32.584 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945ffff700 1 -- 172.21.15.167:0/406249929 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94600fc5e0 con 0x7f94600fbb90 2022-01-31T19:37:32.584 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945ffff700 1 --2- 172.21.15.167:0/406249929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94600fbfb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f945000b5f0 tx=0x7f9450006ee0).ready entity=mon.0 client_cookie=1bfa112e8d14214a server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:32.584 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945effd700 1 -- 172.21.15.167:0/406249929 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f945000d600 con 0x7f94600fbb90 2022-01-31T19:37:32.584 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945effd700 1 -- 172.21.15.167:0/406249929 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f945000dbc0 con 0x7f94600fbb90 2022-01-31T19:37:32.585 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.580+0000 7f945effd700 1 -- 172.21.15.167:0/406249929 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9450011b90 con 0x7f94600fbb90 2022-01-31T19:37:32.585 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 -- 172.21.15.167:0/406249929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 msgr2=0x7f94600fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:32.588 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 --2- 172.21.15.167:0/406249929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94600fbfb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f945000b5f0 tx=0x7f9450006ee0).stop 2022-01-31T19:37:32.588 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 -- 172.21.15.167:0/406249929 shutdown_connections 2022-01-31T19:37:32.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 --2- 172.21.15.167:0/406249929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94600fbfb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:32.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 -- 172.21.15.167:0/406249929 >> 172.21.15.167:0/406249929 conn(0x7f94600f5ca0 msgr2=0x7f94600f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:32.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 -- 172.21.15.167:0/406249929 shutdown_connections 2022-01-31T19:37:32.589 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.581+0000 7f9466572700 1 -- 172.21.15.167:0/406249929 wait complete. 2022-01-31T19:37:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.582+0000 7f9466572700 1 Processor -- start 2022-01-31T19:37:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.582+0000 7f9466572700 1 -- start start 2022-01-31T19:37:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.582+0000 7f9466572700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94601089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.582+0000 7f9466572700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460108ee0 con 0x7f94600fbb90 2022-01-31T19:37:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94601089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94601089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60384/0 (socket says 172.21.15.167:60384) 2022-01-31T19:37:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945ffff700 1 -- 172.21.15.167:0/1440714972 learned_addr learned my addr 172.21.15.167:0/1440714972 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945ffff700 1 -- 172.21.15.167:0/1440714972 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945000b200 con 0x7f94600fbb90 2022-01-31T19:37:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945ffff700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94601089f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f945000e040 tx=0x7f9450006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:32.592 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f945001b410 con 0x7f94600fbb90 2022-01-31T19:37:32.592 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f945001b9d0 con 0x7f94600fbb90 2022-01-31T19:37:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.583+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f945001a7e0 con 0x7f94600fbb90 2022-01-31T19:37:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.584+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94601090e0 con 0x7f94600fbb90 2022-01-31T19:37:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.584+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9460109570 con 0x7f94600fbb90 2022-01-31T19:37:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.584+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f945000f070 con 0x7f94600fbb90 2022-01-31T19:37:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.585+0000 7f945cff9700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.585+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 --> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9460046a20 con 0x7f944c0320e0 2022-01-31T19:37:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.585+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9450042a10 con 0x7f94600fbb90 2022-01-31T19:37:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.585+0000 7f945f7fe700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 msgr2=0x7f944c0345a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/3497254897 2022-01-31T19:37:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.585+0000 7f945f7fe700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2022-01-31T19:37:32.787 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.786+0000 7f945f7fe700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 msgr2=0x7f944c0345a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/3497254897 2022-01-31T19:37:32.787 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:32.786+0000 7f945f7fe700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2022-01-31T19:37:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:33.187+0000 7f945f7fe700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 msgr2=0x7f944c0345a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/3497254897 2022-01-31T19:37:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:33.187+0000 7f945f7fe700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2022-01-31T19:37:33.989 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:33.988+0000 7f945f7fe700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 msgr2=0x7f944c0345a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/3497254897 2022-01-31T19:37:33.989 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:33.988+0000 7f945f7fe700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2022-01-31T19:37:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:34.191+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mgrmap(e 6) v1 ==== 36945+0+0 (secure 0 0 0) 0x7f9450023c90 con 0x7f94600fbb90 2022-01-31T19:37:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:34.191+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 msgr2=0x7f944c0345a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-01-31T19:37:34.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:34.191+0000 7f945cff9700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:34 smithi167 conmon[32206]: cluster 2022-01-31T19:37:34.139382+0000 mon.smithi167 (mon 2022-01-31T19:37:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:34 smithi167 conmon[32206]: .0) 30 : cluster [INF] Active manager daemon smithi167.aciqpk restarted 2022-01-31T19:37:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:34 smithi167 conmon[32206]: cluster 2022-01-31T19:37:34.139780+0000 mon.smithi167 (mon.0) 31 : cluster [INF] 2022-01-31T19:37:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:34 smithi167 conmon[32206]: Activating manager daemon smithi167.aciqpk 2022-01-31T19:37:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:34 smithi167 conmon[32206]: cluster 2022-01-31T19:37:34.141477+0000 mon.smithi167 (mon.0 2022-01-31T19:37:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:34 smithi167 conmon[32206]: ) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2022-01-31T19:37:35.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.199+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f9450019300 con 0x7f94600fbb90 2022-01-31T19:37:35.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.199+0000 7f945cff9700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f944c035990 0x7f944c037d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:35.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.199+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9460046a20 con 0x7f944c035990 2022-01-31T19:37:35.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.200+0000 7f945f7fe700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f944c035990 0x7f944c037d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:35.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.201+0000 7f945f7fe700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f944c035990 0x7f944c037d80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f94540036d0 tx=0x7f9454006ca0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:35.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.201+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f9460046a20 con 0x7f944c035990 2022-01-31T19:37:35.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.207+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f94600591e0 con 0x7f944c035990 2022-01-31T19:37:35.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f945cff9700 1 -- 172.21.15.167:0/1440714972 <== mgr.14116 v2:172.21.15.167:6800/2257398586 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f94600591e0 con 0x7f944c035990 2022-01-31T19:37:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2022-01-31T19:37:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "initialized": true 2022-01-31T19:37:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:35.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f944c035990 msgr2=0x7f944c037d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:35.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f944c035990 0x7f944c037d80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f94540036d0 tx=0x7f9454006ca0).stop 2022-01-31T19:37:35.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 msgr2=0x7f94601089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:35.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94601089f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f945000e040 tx=0x7f9450006ee0).stop 2022-01-31T19:37:35.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 shutdown_connections 2022-01-31T19:37:35.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f944c035990 0x7f944c037d80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:35.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94600fbb90 0x7f94601089f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:35.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 --2- 172.21.15.167:0/1440714972 >> [v2:172.21.15.167:6800/3497254897,v1:172.21.15.167:6801/3497254897] conn(0x7f944c0320e0 0x7f944c0345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.208+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 >> 172.21.15.167:0/1440714972 conn(0x7f94600f5ca0 msgr2=0x7f94600fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.209+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 shutdown_connections 2022-01-31T19:37:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:35.209+0000 7f9466572700 1 -- 172.21.15.167:0/1440714972 wait complete. 2022-01-31T19:37:35.581 INFO:teuthology.orchestra.run.smithi167.stderr:mgr epoch 5 is available 2022-01-31T19:37:35.582 INFO:teuthology.orchestra.run.smithi167.stderr:Setting orchestrator backend to cephadm... 2022-01-31T19:37:35.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: cluster 2022-01-31T19:37:34.193135+0000 mon.smithi167 (mon.0) 33 : 2022-01-31T19:37:35.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: cluster [DBG] mgrmap e6: smithi167.aciqpk(active, starting, since 0.0534327s) 2022-01-31T19:37:35.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022-01-31T19:37:34.195606+0000 mon.smithi167 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:37:35.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022-01-31T19:37:35.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 2022-01-31T19:37:34.196235+0000 mon.smithi167 ( 2022-01-31T19:37:35.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:37:35.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 2022-01-31T19:37:35.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022-01-31T19:37:35.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 2022-01-31T19:37: 2022-01-31T19:37:35.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 34.196950+0000 mon.smithi167 (mon.0) 2022-01-31T19:37:35.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 36 : audit [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:37:35.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:37:35.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 01-31T19:37:34.197145+0000 2022-01-31T19:37:35.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 37 : audit 2022-01-31T19:37:35.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:37:35.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:37:35.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 01-31T19:37:34.197275+0000 2022-01-31T19:37:35.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 38 : audit 2022-01-31T19:37:35.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:37:35.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: cluster 2022 2022-01-31T19:37:35.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: -01-31T19:37:34.210821 2022-01-31T19:37:35.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 39 : 2022-01-31T19:37:35.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: cluster [INF] Manager daemon smithi167.aciqpk is now available 2022-01-31T19:37:35.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:37:35.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 01-31T19:37:34.224282+0000 2022-01-31T19:37:35.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 40 : audit 2022-01-31T19:37:35.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:35.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:37:35.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 01-31T19:37:34.227231+0000 2022-01-31T19:37:35.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 41 : audit 2022-01-31T19:37:35.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:35.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:35.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: -01-31T19:37:34.232649+0000 2022-01-31T19:37:35.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 42 : audit 2022-01-31T19:37:35.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:35.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:37:35.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 01-31T19:37:34.320124+0000 2022-01-31T19:37:35.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 43 : audit [INF] 2022-01-31T19:37:35.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:35.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:37:35.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: -31T19:37:34.320839+0000 2022-01-31T19:37:35.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 44 : audit 2022-01-31T19:37:35.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:37:35.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:35.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: -01-31T19:37:34.322570 2022-01-31T19:37:35.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 45 2022-01-31T19:37:35.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: : audit [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:37:35.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:35.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: -01-31T19:37:34.324017 2022-01-31T19:37:35.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 46 : audit 2022-01-31T19:37:35.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:37:35.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:37:35.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 01-31T19:37:34.334521+0000 2022-01-31T19:37:35.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 47 : audit [INF] 2022-01-31T19:37:35.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:37:35.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: audit 2022 2022-01-31T19:37:35.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: -01-31T19:37:34. 2022-01-31T19:37:35.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: 335685+0000 mon.smithi167 (mon.0) 48 2022-01-31T19:37:35.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:35 smithi167 conmon[32206]: : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:37:36.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.081+0000 7f3588f0c700 1 Processor -- start 2022-01-31T19:37:36.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.081+0000 7f3588f0c700 1 -- start start 2022-01-31T19:37:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.081+0000 7f3588f0c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35840fc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f3588f0c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35840fccf0 con 0x7f35840fc3e0 2022-01-31T19:37:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35840fc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35840fc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60416/0 (socket says 172.21.15.167:60416) 2022-01-31T19:37:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358259c700 1 -- 172.21.15.167:0/3006979384 learned_addr learned my addr 172.21.15.167:0/3006979384 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358259c700 1 -- 172.21.15.167:0/3006979384 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35840fce30 con 0x7f35840fc3e0 2022-01-31T19:37:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358259c700 1 --2- 172.21.15.167:0/3006979384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35840fc800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f357400b5f0 tx=0x7f3574006ee0).ready entity=mon.0 client_cookie=6fe288e5c38aff7f server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358159a700 1 -- 172.21.15.167:0/3006979384 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357400d600 con 0x7f35840fc3e0 2022-01-31T19:37:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358159a700 1 -- 172.21.15.167:0/3006979384 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f357400dbc0 con 0x7f35840fc3e0 2022-01-31T19:37:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.082+0000 7f358159a700 1 -- 172.21.15.167:0/3006979384 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3574011b90 con 0x7f35840fc3e0 2022-01-31T19:37:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 -- 172.21.15.167:0/3006979384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 msgr2=0x7f35840fc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 --2- 172.21.15.167:0/3006979384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35840fc800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f357400b5f0 tx=0x7f3574006ee0).stop 2022-01-31T19:37:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 -- 172.21.15.167:0/3006979384 shutdown_connections 2022-01-31T19:37:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 --2- 172.21.15.167:0/3006979384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35840fc800 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 -- 172.21.15.167:0/3006979384 >> 172.21.15.167:0/3006979384 conn(0x7f35840f6530 msgr2=0x7f35840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 -- 172.21.15.167:0/3006979384 shutdown_connections 2022-01-31T19:37:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.083+0000 7f3588f0c700 1 -- 172.21.15.167:0/3006979384 wait complete. 2022-01-31T19:37:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f3588f0c700 1 Processor -- start 2022-01-31T19:37:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f3588f0c700 1 -- start start 2022-01-31T19:37:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f3588f0c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35841089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f3588f0c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3584108ed0 con 0x7f35840fc3e0 2022-01-31T19:37:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f358259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35841089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f358259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35841089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60418/0 (socket says 172.21.15.167:60418) 2022-01-31T19:37:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.084+0000 7f358259c700 1 -- 172.21.15.167:0/3752750465 learned_addr learned my addr 172.21.15.167:0/3752750465 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f358259c700 1 -- 172.21.15.167:0/3752750465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357400b200 con 0x7f35840fc3e0 2022-01-31T19:37:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f358259c700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35841089e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3574000c00 tx=0x7f3574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357401b570 con 0x7f35840fc3e0 2022-01-31T19:37:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f357401bb30 con 0x7f35840fc3e0 2022-01-31T19:37:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35841090d0 con 0x7f35840fc3e0 2022-01-31T19:37:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357401a6e0 con 0x7f35840fc3e0 2022-01-31T19:37:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.085+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3584109560 con 0x7f35840fc3e0 2022-01-31T19:37:36.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.086+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3584045bb0 con 0x7f35840fc3e0 2022-01-31T19:37:36.095 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.086+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f357402b070 con 0x7f35840fc3e0 2022-01-31T19:37:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.086+0000 7f357affd700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3564035340 0x7f3564037800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.086+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3574043040 con 0x7f35840fc3e0 2022-01-31T19:37:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.087+0000 7f3581d9b700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3564035340 0x7f3564037800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.087+0000 7f3581d9b700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3564035340 0x7f3564037800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f356c0096a0 tx=0x7f356c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:36.097 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.090+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3574023ba0 con 0x7f35840fc3e0 2022-01-31T19:37:36.255 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.254+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f3584109e80 con 0x7f3564035340 2022-01-31T19:37:36.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.259+0000 7f357affd700 1 -- 172.21.15.167:0/3752750465 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3584109e80 con 0x7f3564035340 2022-01-31T19:37:36.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.261+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3564035340 msgr2=0x7f3564037800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:36.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.261+0000 7f3588f0c700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3564035340 0x7f3564037800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f356c0096a0 tx=0x7f356c006b40).stop 2022-01-31T19:37:36.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.261+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 msgr2=0x7f35841089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:36.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.261+0000 7f3588f0c700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35841089e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3574000c00 tx=0x7f3574006ee0).stop 2022-01-31T19:37:36.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.262+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 shutdown_connections 2022-01-31T19:37:36.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.262+0000 7f3588f0c700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3564035340 0x7f3564037800 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:36.265 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.262+0000 7f3588f0c700 1 --2- 172.21.15.167:0/3752750465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840fc3e0 0x7f35841089e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:36.265 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.262+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 >> 172.21.15.167:0/3752750465 conn(0x7f35840f6530 msgr2=0x7f35840ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:36.265 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.262+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 shutdown_connections 2022-01-31T19:37:36.265 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:36.262+0000 7f3588f0c700 1 -- 172.21.15.167:0/3752750465 wait complete. 2022-01-31T19:37:36.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:36 smithi167 conmon[32206]: audit 2022-01-31T19:37:35.201737+0000 mgr.smithi167.aciqpk (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-01-31T19:37:36.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:36 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:37:36.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:36 smithi167 conmon[32206]: 37:35.202692+0000 mon.smithi167 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi167.aciqpk(active, since 1.06298s) 2022-01-31T19:37:36.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:36 smithi167 conmon[32206]: audit 2022-01-31T19:37:35.208667+0000 mgr.smithi167.aciqpk (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-01-31T19:37:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.067+0000 7f0bf4894700 1 Processor -- start 2022-01-31T19:37:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.067+0000 7f0bf4894700 1 -- start start 2022-01-31T19:37:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bf4894700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf00fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bf4894700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00fcd00 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf00fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf00fc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60422/0 (socket says 172.21.15.167:60422) 2022-01-31T19:37:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bee59c700 1 -- 172.21.15.167:0/663633888 learned_addr learned my addr 172.21.15.167:0/663633888 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bee59c700 1 -- 172.21.15.167:0/663633888 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00fce40 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bee59c700 1 --2- 172.21.15.167:0/663633888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf00fc810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0be000b5f0 tx=0x7f0be0006ee0).ready entity=mon.0 client_cookie=889e7a1296491887 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bed59a700 1 -- 172.21.15.167:0/663633888 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0be000d600 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bed59a700 1 -- 172.21.15.167:0/663633888 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0be000dbc0 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.068+0000 7f0bed59a700 1 -- 172.21.15.167:0/663633888 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0be0011b90 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 -- 172.21.15.167:0/663633888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 msgr2=0x7f0bf00fc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 --2- 172.21.15.167:0/663633888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf00fc810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0be000b5f0 tx=0x7f0be0006ee0).stop 2022-01-31T19:37:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 -- 172.21.15.167:0/663633888 shutdown_connections 2022-01-31T19:37:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 --2- 172.21.15.167:0/663633888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf00fc810 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 -- 172.21.15.167:0/663633888 >> 172.21.15.167:0/663633888 conn(0x7f0bf00f6520 msgr2=0x7f0bf00f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 -- 172.21.15.167:0/663633888 shutdown_connections 2022-01-31T19:37:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.069+0000 7f0bf4894700 1 -- 172.21.15.167:0/663633888 wait complete. 2022-01-31T19:37:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bf4894700 1 Processor -- start 2022-01-31T19:37:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bf4894700 1 -- start start 2022-01-31T19:37:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bf4894700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf010ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bf4894700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf010d3a0 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf010ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bee59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf010ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60424/0 (socket says 172.21.15.167:60424) 2022-01-31T19:37:37.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bee59c700 1 -- 172.21.15.167:0/1212373074 learned_addr learned my addr 172.21.15.167:0/1212373074 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:37.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.070+0000 7f0bee59c700 1 -- 172.21.15.167:0/1212373074 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be000b200 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.071+0000 7f0bee59c700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf010ceb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0be000e040 tx=0x7f0be0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.071+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0be001b410 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.071+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf010d5a0 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.071+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0be001b9d0 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.071+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0be001a7e0 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.071+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf010e1b0 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.072+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f0be000f070 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.072+0000 7f0be6ffd700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0bd0030f80 0x7f0bd0033440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:37.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.072+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0be0042b40 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.072+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf010da30 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.072+0000 7f0bedd9b700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0bd0030f80 0x7f0bd0033440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:37.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.072+0000 7f0bedd9b700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0bd0030f80 0x7f0bd0033440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0bd80096a0 tx=0x7f0bd8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:37.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.076+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0bf010da30 con 0x7f0bf00fc3f0 2022-01-31T19:37:37.232 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.231+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bf010da30 con 0x7f0bd0030f80 2022-01-31T19:37:37.232 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.231+0000 7f0be6ffd700 1 -- 172.21.15.167:0/1212373074 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0bf010da30 con 0x7f0bd0030f80 2022-01-31T19:37:37.233 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: value unchanged 2022-01-31T19:37:37.235 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0bd0030f80 msgr2=0x7f0bd0033440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:37.235 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0bd0030f80 0x7f0bd0033440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0bd80096a0 tx=0x7f0bd8006b40).stop 2022-01-31T19:37:37.235 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 msgr2=0x7f0bf010ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:37.236 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf010ceb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0be000e040 tx=0x7f0be0006ee0).stop 2022-01-31T19:37:37.236 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 shutdown_connections 2022-01-31T19:37:37.236 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0bd0030f80 0x7f0bd0033440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:37.237 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 --2- 172.21.15.167:0/1212373074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bf00fc3f0 0x7f0bf010ceb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:37.237 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 >> 172.21.15.167:0/1212373074 conn(0x7f0bf00f6520 msgr2=0x7f0bf00ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:37.237 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 shutdown_connections 2022-01-31T19:37:37.237 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:37.233+0000 7f0bf4894700 1 -- 172.21.15.167:0/1212373074 wait complete. 2022-01-31T19:37:37.414 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: audit 2022-01-31T19:37:36.255192+0000 mgr.smithi167.aciqpk (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:37.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: audit 2022-01-31T19:37:37.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: 2022-01-31T19:37:36.259658+0000 mon.smithi167 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:37.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: audit 2022-01-31T19:37:36.263774+0000 mon.smithi167 (mon.0) 2022-01-31T19:37:37.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: 51 : audit [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:37:37.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: cluster 2022-01- 2022-01-31T19:37:37.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: 31T19:37:36.592633+0000 2022-01-31T19:37:37.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: mon.smithi167 (mon.0) 52 : cluster 2022-01-31T19:37:37.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:37 smithi167 conmon[32206]: [DBG] mgrmap e8: smithi167.aciqpk(active, since 2s) 2022-01-31T19:37:37.552 INFO:teuthology.orchestra.run.smithi167.stderr:Generating ssh key... 2022-01-31T19:37:38.009 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.003+0000 7fa24c0b0700 1 Processor -- start 2022-01-31T19:37:38.009 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.003+0000 7fa24c0b0700 1 -- start start 2022-01-31T19:37:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.003+0000 7fa24c0b0700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa244100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.003+0000 7fa24c0b0700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440fc8c0 con 0x7fa24406fca0 2022-01-31T19:37:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa249e4c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa244100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa249e4c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa244100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60428/0 (socket says 172.21.15.167:60428) 2022-01-31T19:37:38.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa249e4c700 1 -- 172.21.15.167:0/532792594 learned_addr learned my addr 172.21.15.167:0/532792594 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:38.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa249e4c700 1 -- 172.21.15.167:0/532792594 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2441013c0 con 0x7fa24406fca0 2022-01-31T19:37:38.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa249e4c700 1 --2- 172.21.15.167:0/532792594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa244100ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa23400b5f0 tx=0x7fa234006ee0).ready entity=mon.0 client_cookie=1c9bbd6976f21bfe server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:38.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa248e4a700 1 -- 172.21.15.167:0/532792594 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23400d600 con 0x7fa24406fca0 2022-01-31T19:37:38.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa248e4a700 1 -- 172.21.15.167:0/532792594 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa23400dbc0 con 0x7fa24406fca0 2022-01-31T19:37:38.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.004+0000 7fa248e4a700 1 -- 172.21.15.167:0/532792594 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa234011b90 con 0x7fa24406fca0 2022-01-31T19:37:38.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 -- 172.21.15.167:0/532792594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 msgr2=0x7fa244100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:38.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 --2- 172.21.15.167:0/532792594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa244100ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa23400b5f0 tx=0x7fa234006ee0).stop 2022-01-31T19:37:38.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 -- 172.21.15.167:0/532792594 shutdown_connections 2022-01-31T19:37:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 --2- 172.21.15.167:0/532792594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa244100ed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 -- 172.21.15.167:0/532792594 >> 172.21.15.167:0/532792594 conn(0x7fa2440f6530 msgr2=0x7fa2440f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 -- 172.21.15.167:0/532792594 shutdown_connections 2022-01-31T19:37:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 -- 172.21.15.167:0/532792594 wait complete. 2022-01-31T19:37:38.014 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.005+0000 7fa24c0b0700 1 Processor -- start 2022-01-31T19:37:38.014 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa24c0b0700 1 -- start start 2022-01-31T19:37:38.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa24c0b0700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa24410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:38.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa24c0b0700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24410d3c0 con 0x7fa24406fca0 2022-01-31T19:37:38.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa249e4c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa24410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:38.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa249e4c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa24410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60430/0 (socket says 172.21.15.167:60430) 2022-01-31T19:37:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa249e4c700 1 -- 172.21.15.167:0/1829762648 learned_addr learned my addr 172.21.15.167:0/1829762648 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa249e4c700 1 -- 172.21.15.167:0/1829762648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23400b200 con 0x7fa24406fca0 2022-01-31T19:37:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa249e4c700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa24410ced0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa234000c00 tx=0x7fa234006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23401b410 con 0x7fa24406fca0 2022-01-31T19:37:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa23401b9d0 con 0x7fa24406fca0 2022-01-31T19:37:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24410d5c0 con 0x7fa24406fca0 2022-01-31T19:37:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23401a730 con 0x7fa24406fca0 2022-01-31T19:37:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.006+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24410e170 con 0x7fa24406fca0 2022-01-31T19:37:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.007+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa23402b070 con 0x7fa24406fca0 2022-01-31T19:37:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.007+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa244046a20 con 0x7fa24406fca0 2022-01-31T19:37:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.007+0000 7fa2327fc700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa22c030ff0 0x7fa22c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.008+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa234042c30 con 0x7fa24406fca0 2022-01-31T19:37:38.019 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.008+0000 7fa24964b700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa22c030ff0 0x7fa22c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:38.019 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.008+0000 7fa24964b700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa22c030ff0 0x7fa22c0334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa2400096a0 tx=0x7fa240006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:38.020 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.012+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa23401b570 con 0x7fa24406fca0 2022-01-31T19:37:38.164 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.163+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa24410e470 con 0x7fa22c030ff0 2022-01-31T19:37:38.446 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.445+0000 7fa2327fc700 1 -- 172.21.15.167:0/1829762648 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa24410e470 con 0x7fa22c030ff0 2022-01-31T19:37:38.450 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa22c030ff0 msgr2=0x7fa22c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:38.451 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa22c030ff0 0x7fa22c0334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa2400096a0 tx=0x7fa240006b40).stop 2022-01-31T19:37:38.451 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 msgr2=0x7fa24410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:38.451 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa24410ced0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa234000c00 tx=0x7fa234006ee0).stop 2022-01-31T19:37:38.452 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 shutdown_connections 2022-01-31T19:37:38.452 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa22c030ff0 0x7fa22c0334b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:38.452 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 --2- 172.21.15.167:0/1829762648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa24406fca0 0x7fa24410ced0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:38.453 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 >> 172.21.15.167:0/1829762648 conn(0x7fa2440f6530 msgr2=0x7fa2440f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:38.453 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 shutdown_connections 2022-01-31T19:37:38.453 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:38.447+0000 7fa24c0b0700 1 -- 172.21.15.167:0/1829762648 wait complete. 2022-01-31T19:37:38.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:38 smithi167 conmon[32206]: audit 2022-01-31T19:37:37.232179+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:37:38.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:38 smithi167 conmon[32206]: 14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:39.252 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb0832ea700 1 Processor -- start 2022-01-31T19:37:39.252 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb0832ea700 1 -- start start 2022-01-31T19:37:39.253 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb0832ea700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c0fd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:39.253 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb0832ea700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0fde30 con 0x7fb07c0fd520 2022-01-31T19:37:39.253 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb081086700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c0fd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:39.253 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb081086700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c0fd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60434/0 (socket says 172.21.15.167:60434) 2022-01-31T19:37:39.254 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.250+0000 7fb081086700 1 -- 172.21.15.167:0/1461097556 learned_addr learned my addr 172.21.15.167:0/1461097556 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:39.254 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb081086700 1 -- 172.21.15.167:0/1461097556 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c0fdf70 con 0x7fb07c0fd520 2022-01-31T19:37:39.254 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb081086700 1 --2- 172.21.15.167:0/1461097556 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c0fd940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb07800b5f0 tx=0x7fb078006ee0).ready entity=mon.0 client_cookie=42f9906b110bcb3a server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:39.254 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb06bfff700 1 -- 172.21.15.167:0/1461097556 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb07800d600 con 0x7fb07c0fd520 2022-01-31T19:37:39.254 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb06bfff700 1 -- 172.21.15.167:0/1461097556 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb07800dbc0 con 0x7fb07c0fd520 2022-01-31T19:37:39.255 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb06bfff700 1 -- 172.21.15.167:0/1461097556 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb078011b90 con 0x7fb07c0fd520 2022-01-31T19:37:39.258 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb0832ea700 1 -- 172.21.15.167:0/1461097556 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 msgr2=0x7fb07c0fd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:39.258 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb0832ea700 1 --2- 172.21.15.167:0/1461097556 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c0fd940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb07800b5f0 tx=0x7fb078006ee0).stop 2022-01-31T19:37:39.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb0832ea700 1 -- 172.21.15.167:0/1461097556 shutdown_connections 2022-01-31T19:37:39.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb0832ea700 1 --2- 172.21.15.167:0/1461097556 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c0fd940 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:39.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.251+0000 7fb0832ea700 1 -- 172.21.15.167:0/1461097556 >> 172.21.15.167:0/1461097556 conn(0x7fb07c0f7670 msgr2=0x7fb07c0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:39.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.252+0000 7fb0832ea700 1 -- 172.21.15.167:0/1461097556 shutdown_connections 2022-01-31T19:37:39.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.252+0000 7fb0832ea700 1 -- 172.21.15.167:0/1461097556 wait complete. 2022-01-31T19:37:39.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.252+0000 7fb0832ea700 1 Processor -- start 2022-01-31T19:37:39.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.252+0000 7fb0832ea700 1 -- start start 2022-01-31T19:37:39.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.252+0000 7fb0832ea700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c10acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:39.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb0832ea700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c10b1c0 con 0x7fb07c0fd520 2022-01-31T19:37:39.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb081086700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c10acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:39.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb081086700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c10acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60436/0 (socket says 172.21.15.167:60436) 2022-01-31T19:37:39.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb081086700 1 -- 172.21.15.167:0/56100168 learned_addr learned my addr 172.21.15.167:0/56100168 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:39.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb081086700 1 -- 172.21.15.167:0/56100168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07800b200 con 0x7fb07c0fd520 2022-01-31T19:37:39.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb081086700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c10acd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb078000c00 tx=0x7fb078006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:39.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb07801b570 con 0x7fb07c0fd520 2022-01-31T19:37:39.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb07801bb30 con 0x7fb07c0fd520 2022-01-31T19:37:39.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c10b3c0 con 0x7fb07c0fd520 2022-01-31T19:37:39.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb07801a6e0 con 0x7fb07c0fd520 2022-01-31T19:37:39.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.253+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c10bf70 con 0x7fb07c0fd520 2022-01-31T19:37:39.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.254+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fb07802b070 con 0x7fb07c0fd520 2022-01-31T19:37:39.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.254+0000 7fb069ffb700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fb064030ff0 0x7fb0640334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:39.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.254+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fb078042d30 con 0x7fb07c0fd520 2022-01-31T19:37:39.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.254+0000 7fb080885700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fb064030ff0 0x7fb0640334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:39.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.254+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c046a20 con 0x7fb07c0fd520 2022-01-31T19:37:39.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.255+0000 7fb080885700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fb064030ff0 0x7fb0640334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb06c0096a0 tx=0x7fb06c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:39.265 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.259+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb078016070 con 0x7fb07c0fd520 2022-01-31T19:37:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.414+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb07c0fc6e0 con 0x7fb064030ff0 2022-01-31T19:37:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.415+0000 7fb069ffb700 1 -- 172.21.15.167:0/56100168 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb07c0fc6e0 con 0x7fb064030ff0 2022-01-31T19:37:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDirFThKHfhqbBm9eATQx4qg0mmHakKo1NcRQs4EhKamQNIkFOKTI0etblzTQUX2AVZbEiGv6LEr19ZlqG7DNVPjCS7Lh4TDiMz43coDCyVLBXe5A1JY2hiqDtcJMy36I4kEAYn5P2PzutUrqIpTscGmSq1yoIi1c3EOTjNqWG43qGOnQDF5D19K9nEsfdomkw0hpdc7mQZaQrR1ru8RMAdsMGqQhTwI5kf4/HNIzn0c7xHb6sH9Qz3I9zpkt117rj658tcDgn12PDXtz0iYiza5MYXZeUhXu1KIlMik3c2Em1DMAfmFvdRmx7Oa2ujzFwfuR4QM1cbB19vnU0nu6gK9FG8S9TYSCfOQjd3HD7/1muM7hlLc+9ieAznGCUHRk5QiB34BuA5X+OHye0GpxtxbyW5v/R9v66nbXVYQQHaKsPJpmZFdh50AFUAEXJDSPq1i+QHpb3uvtoDEtnDnM+V0tZRAdEzY1N7qX/3TdKHXB6woa/joQb+6cRcnLEvlT0= ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:37:39.419 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fb064030ff0 msgr2=0x7fb0640334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:39.419 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fb064030ff0 0x7fb0640334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb06c0096a0 tx=0x7fb06c006b40).stop 2022-01-31T19:37:39.419 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 msgr2=0x7fb07c10acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:39.419 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c10acd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb078000c00 tx=0x7fb078006ee0).stop 2022-01-31T19:37:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 shutdown_connections 2022-01-31T19:37:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fb064030ff0 0x7fb0640334b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 --2- 172.21.15.167:0/56100168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb07c0fd520 0x7fb07c10acd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 >> 172.21.15.167:0/56100168 conn(0x7fb07c0f7670 msgr2=0x7fb07c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 shutdown_connections 2022-01-31T19:37:39.421 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:39.417+0000 7fb0832ea700 1 -- 172.21.15.167:0/56100168 wait complete. 2022-01-31T19:37:39.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:39 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:38.165014+0000 mgr.smithi167.aciqpk (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2022-01-31T19:37:39.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:39 smithi167 conmon[32206]: audit 2022-01-31T19:37:38.443432+0000 mon.smithi167 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:39.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:39 smithi167 conmon[32206]: audit 2022-01-31T19:37:38.445181+0000 mon.smithi167 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:39.826 INFO:teuthology.orchestra.run.smithi167.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-01-31T19:37:39.827 INFO:teuthology.orchestra.run.smithi167.stderr:Adding key to root@localhost authorized_keys... 2022-01-31T19:37:39.827 INFO:teuthology.orchestra.run.smithi167.stderr:Adding host smithi167... 2022-01-31T19:37:40.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f166bf4d700 1 Processor -- start 2022-01-31T19:37:40.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f166bf4d700 1 -- start start 2022-01-31T19:37:40.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f166bf4d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f1664070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:40.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f166bf4d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1664070740 con 0x7f1664071d80 2022-01-31T19:37:40.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f1669ce9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f1664070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:40.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f1669ce9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f1664070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60440/0 (socket says 172.21.15.167:60440) 2022-01-31T19:37:40.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.282+0000 7f1669ce9700 1 -- 172.21.15.167:0/1278126257 learned_addr learned my addr 172.21.15.167:0/1278126257 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:40.289 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f1669ce9700 1 -- 172.21.15.167:0/1278126257 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664070880 con 0x7f1664071d80 2022-01-31T19:37:40.289 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f1669ce9700 1 --2- 172.21.15.167:0/1278126257 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f1664070250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f166000b5f0 tx=0x7f1660006ee0).ready entity=mon.0 client_cookie=6a4da49df5b024f5 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:40.289 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f1668ce7700 1 -- 172.21.15.167:0/1278126257 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166000d600 con 0x7f1664071d80 2022-01-31T19:37:40.289 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f1668ce7700 1 -- 172.21.15.167:0/1278126257 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f166000dbc0 con 0x7f1664071d80 2022-01-31T19:37:40.290 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f1668ce7700 1 -- 172.21.15.167:0/1278126257 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1660011b90 con 0x7f1664071d80 2022-01-31T19:37:40.290 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f166bf4d700 1 -- 172.21.15.167:0/1278126257 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 msgr2=0x7f1664070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:40.290 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.283+0000 7f166bf4d700 1 --2- 172.21.15.167:0/1278126257 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f1664070250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f166000b5f0 tx=0x7f1660006ee0).stop 2022-01-31T19:37:40.290 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 -- 172.21.15.167:0/1278126257 shutdown_connections 2022-01-31T19:37:40.291 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 --2- 172.21.15.167:0/1278126257 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f1664070250 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:40.291 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 -- 172.21.15.167:0/1278126257 >> 172.21.15.167:0/1278126257 conn(0x7f16640f7670 msgr2=0x7f16640f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:40.291 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 -- 172.21.15.167:0/1278126257 shutdown_connections 2022-01-31T19:37:40.291 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 -- 172.21.15.167:0/1278126257 wait complete. 2022-01-31T19:37:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 Processor -- start 2022-01-31T19:37:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 -- start start 2022-01-31T19:37:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f166410cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.284+0000 7f166bf4d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166410d3e0 con 0x7f1664071d80 2022-01-31T19:37:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f1669ce9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f166410cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f1669ce9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f166410cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60442/0 (socket says 172.21.15.167:60442) 2022-01-31T19:37:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f1669ce9700 1 -- 172.21.15.167:0/1391445568 learned_addr learned my addr 172.21.15.167:0/1391445568 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f1669ce9700 1 -- 172.21.15.167:0/1391445568 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166000b200 con 0x7f1664071d80 2022-01-31T19:37:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f1669ce9700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f166410cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1660006800 tx=0x7f1660006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166000de50 con 0x7f1664071d80 2022-01-31T19:37:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166410d5e0 con 0x7f1664071d80 2022-01-31T19:37:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166410da70 con 0x7f1664071d80 2022-01-31T19:37:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f166001b870 con 0x7f1664071d80 2022-01-31T19:37:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.285+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166001a7a0 con 0x7f1664071d80 2022-01-31T19:37:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.286+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f166000f070 con 0x7f1664071d80 2022-01-31T19:37:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.286+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1644004fa0 con 0x7f1664071d80 2022-01-31T19:37:40.297 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.286+0000 7f16527fc700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f164c030ff0 0x7f164c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:40.297 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.286+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f1660042b50 con 0x7f1664071d80 2022-01-31T19:37:40.297 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.286+0000 7f16694e8700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f164c030ff0 0x7f164c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:40.297 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.287+0000 7f16694e8700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f164c030ff0 0x7f164c0334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f16540096a0 tx=0x7f1654006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:40.298 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.291+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f166002b4d0 con 0x7f1664071d80 2022-01-31T19:37:40.450 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:40.449+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi167", "addr": "172.21.15.167", "target": ["mon-mgr", ""]}) v1 -- 0x7f1644000bc0 con 0x7f164c030ff0 2022-01-31T19:37:40.543 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:40 smithi167 conmon[32206]: audit 2022-01-31T19:37:39.415955+0000 mgr.smithi167.aciqpk (mgr.14116 2022-01-31T19:37:40.544 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:40 smithi167 conmon[32206]: ) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:41.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Added host 'smithi167' with addr '172.21.15.167' 2022-01-31T19:37:41.140 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.137+0000 7f16527fc700 1 -- 172.21.15.167:0/1391445568 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1644000bc0 con 0x7f164c030ff0 2022-01-31T19:37:41.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f164c030ff0 msgr2=0x7f164c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:41.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f164c030ff0 0x7f164c0334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f16540096a0 tx=0x7f1654006b40).stop 2022-01-31T19:37:41.142 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 msgr2=0x7f166410cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:41.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f166410cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1660006800 tx=0x7f1660006ee0).stop 2022-01-31T19:37:41.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 shutdown_connections 2022-01-31T19:37:41.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f164c030ff0 0x7f164c0334b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:41.143 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 --2- 172.21.15.167:0/1391445568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1664071d80 0x7f166410cef0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:41.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 >> 172.21.15.167:0/1391445568 conn(0x7f16640f7670 msgr2=0x7f16640f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:41.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 shutdown_connections 2022-01-31T19:37:41.144 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:41.140+0000 7f166bf4d700 1 -- 172.21.15.167:0/1391445568 wait complete. 2022-01-31T19:37:41.464 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying mon service with default placement... 2022-01-31T19:37:41.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:41 smithi167 conmon[32206]: audit 2022-01-31T19:37:40.451080+0000 mgr.smithi167.aciqpk (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi167", "addr": "172.21.15.167", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:41.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:41 smithi167 conmon[32206]: audit 2022-01-31T19:37:41.137436+0000 mon.smithi167 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:41.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:41 smithi167 conmon[32206]: audit 2022-01-31T19:37:41.137961+0000 mon.smithi167 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:37:42.194 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.183+0000 7fa97fed6700 1 Processor -- start 2022-01-31T19:37:42.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.183+0000 7fa97fed6700 1 -- start start 2022-01-31T19:37:42.195 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.183+0000 7fa97fed6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:42.196 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.183+0000 7fa97fed6700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780fde20 con 0x7fa9780fd510 2022-01-31T19:37:42.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97dc72700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:42.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97dc72700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60454/0 (socket says 172.21.15.167:60454) 2022-01-31T19:37:42.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97dc72700 1 -- 172.21.15.167:0/2513058383 learned_addr learned my addr 172.21.15.167:0/2513058383 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:42.197 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97dc72700 1 -- 172.21.15.167:0/2513058383 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9780fdf60 con 0x7fa9780fd510 2022-01-31T19:37:42.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97dc72700 1 --2- 172.21.15.167:0/2513058383 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fd930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa97400b5f0 tx=0x7fa974006ee0).ready entity=mon.0 client_cookie=95847ec2052c4efd server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:42.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97cc70700 1 -- 172.21.15.167:0/2513058383 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa97400d600 con 0x7fa9780fd510 2022-01-31T19:37:42.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97cc70700 1 -- 172.21.15.167:0/2513058383 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa97400dbc0 con 0x7fa9780fd510 2022-01-31T19:37:42.198 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.184+0000 7fa97cc70700 1 -- 172.21.15.167:0/2513058383 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa974011b90 con 0x7fa9780fd510 2022-01-31T19:37:42.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 -- 172.21.15.167:0/2513058383 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 msgr2=0x7fa9780fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:42.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 --2- 172.21.15.167:0/2513058383 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fd930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa97400b5f0 tx=0x7fa974006ee0).stop 2022-01-31T19:37:42.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 -- 172.21.15.167:0/2513058383 shutdown_connections 2022-01-31T19:37:42.199 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 --2- 172.21.15.167:0/2513058383 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fd930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:42.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 -- 172.21.15.167:0/2513058383 >> 172.21.15.167:0/2513058383 conn(0x7fa9780f5ca0 msgr2=0x7fa9780f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:42.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 -- 172.21.15.167:0/2513058383 shutdown_connections 2022-01-31T19:37:42.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.185+0000 7fa97fed6700 1 -- 172.21.15.167:0/2513058383 wait complete. 2022-01-31T19:37:42.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97fed6700 1 Processor -- start 2022-01-31T19:37:42.200 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97fed6700 1 -- start start 2022-01-31T19:37:42.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97fed6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:42.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97fed6700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780fb1d0 con 0x7fa9780fd510 2022-01-31T19:37:42.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97dc72700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:42.201 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97dc72700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fcd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60456/0 (socket says 172.21.15.167:60456) 2022-01-31T19:37:42.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97dc72700 1 -- 172.21.15.167:0/1343446423 learned_addr learned my addr 172.21.15.167:0/1343446423 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:42.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97dc72700 1 -- 172.21.15.167:0/1343446423 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97400b200 con 0x7fa9780fd510 2022-01-31T19:37:42.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa97dc72700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fcd50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa97400e040 tx=0x7fa974006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:42.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.186+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa97401b410 con 0x7fa9780fd510 2022-01-31T19:37:42.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.187+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa97401b9d0 con 0x7fa9780fd510 2022-01-31T19:37:42.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.187+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9780fb370 con 0x7fa9780fd510 2022-01-31T19:37:42.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.187+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa97401a7e0 con 0x7fa9780fd510 2022-01-31T19:37:42.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.187+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9780fb800 con 0x7fa9780fd510 2022-01-31T19:37:42.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.187+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa97400f070 con 0x7fa9780fd510 2022-01-31T19:37:42.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.188+0000 7fa9667fc700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa960030ff0 0x7fa9600334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:42.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.188+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa974042b00 con 0x7fa9780fd510 2022-01-31T19:37:42.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.188+0000 7fa97d471700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa960030ff0 0x7fa9600334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:42.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.188+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa978046a20 con 0x7fa9780fd510 2022-01-31T19:37:42.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.188+0000 7fa97d471700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa960030ff0 0x7fa9600334b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa9680096a0 tx=0x7fa968006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:42.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.193+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa974016030 con 0x7fa9780fd510 2022-01-31T19:37:42.357 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.356+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9780fa990 con 0x7fa960030ff0 2022-01-31T19:37:42.361 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.360+0000 7fa9667fc700 1 -- 172.21.15.167:0/1343446423 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa9780fa990 con 0x7fa960030ff0 2022-01-31T19:37:42.361 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled mon update... 2022-01-31T19:37:42.362 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa960030ff0 msgr2=0x7fa9600334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:42.363 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa960030ff0 0x7fa9600334b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa9680096a0 tx=0x7fa968006b40).stop 2022-01-31T19:37:42.364 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 msgr2=0x7fa9780fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:42.364 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fcd50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa97400e040 tx=0x7fa974006ee0).stop 2022-01-31T19:37:42.364 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 shutdown_connections 2022-01-31T19:37:42.364 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fa960030ff0 0x7fa9600334b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:42.365 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 --2- 172.21.15.167:0/1343446423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9780fd510 0x7fa9780fcd50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:42.365 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 >> 172.21.15.167:0/1343446423 conn(0x7fa9780f5ca0 msgr2=0x7fa9780f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:42.365 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 shutdown_connections 2022-01-31T19:37:42.365 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:42.362+0000 7fa97fed6700 1 -- 172.21.15.167:0/1343446423 wait complete. 2022-01-31T19:37:42.699 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying mgr service with default placement... 2022-01-31T19:37:42.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:41.137720+0000 mgr.smithi167.aciqpk (mgr.14116) 9 : cephadm [INF] Added host smithi167 2022-01-31T19:37:42.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:42 smithi167 conmon[32206]: audit 2022-01-31T19:37:42.360473+0000 mon.smithi167 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:43.269 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.262+0000 7f6254479700 1 Processor -- start 2022-01-31T19:37:43.270 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.262+0000 7f6254479700 1 -- start start 2022-01-31T19:37:43.270 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.262+0000 7f6254479700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:43.270 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.262+0000 7f6254479700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c0fc8c0 con 0x7f624c06fca0 2022-01-31T19:37:43.271 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6252215700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:43.271 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6252215700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60460/0 (socket says 172.21.15.167:60460) 2022-01-31T19:37:43.271 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6252215700 1 -- 172.21.15.167:0/1531656809 learned_addr learned my addr 172.21.15.167:0/1531656809 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:43.272 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6252215700 1 -- 172.21.15.167:0/1531656809 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c1013c0 con 0x7f624c06fca0 2022-01-31T19:37:43.272 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6252215700 1 --2- 172.21.15.167:0/1531656809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c100ed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f624800b5f0 tx=0x7f6248006ee0).ready entity=mon.0 client_cookie=e4dc9bd68b878c65 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:43.272 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6251213700 1 -- 172.21.15.167:0/1531656809 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f624800d600 con 0x7f624c06fca0 2022-01-31T19:37:43.272 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6251213700 1 -- 172.21.15.167:0/1531656809 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f624800dbc0 con 0x7f624c06fca0 2022-01-31T19:37:43.273 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.263+0000 7f6251213700 1 -- 172.21.15.167:0/1531656809 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6248011b90 con 0x7f624c06fca0 2022-01-31T19:37:43.273 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 -- 172.21.15.167:0/1531656809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 msgr2=0x7f624c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:43.274 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 --2- 172.21.15.167:0/1531656809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c100ed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f624800b5f0 tx=0x7f6248006ee0).stop 2022-01-31T19:37:43.274 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 -- 172.21.15.167:0/1531656809 shutdown_connections 2022-01-31T19:37:43.274 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 --2- 172.21.15.167:0/1531656809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c100ed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:43.274 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 -- 172.21.15.167:0/1531656809 >> 172.21.15.167:0/1531656809 conn(0x7f624c0f6530 msgr2=0x7f624c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:43.275 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 -- 172.21.15.167:0/1531656809 shutdown_connections 2022-01-31T19:37:43.275 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.264+0000 7f6254479700 1 -- 172.21.15.167:0/1531656809 wait complete. 2022-01-31T19:37:43.275 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6254479700 1 Processor -- start 2022-01-31T19:37:43.275 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6254479700 1 -- start start 2022-01-31T19:37:43.275 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6254479700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:43.276 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6254479700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c10d3c0 con 0x7f624c06fca0 2022-01-31T19:37:43.276 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6252215700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:43.276 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6252215700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c10ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60462/0 (socket says 172.21.15.167:60462) 2022-01-31T19:37:43.277 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6252215700 1 -- 172.21.15.167:0/2868452334 learned_addr learned my addr 172.21.15.167:0/2868452334 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:43.277 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6252215700 1 -- 172.21.15.167:0/2868452334 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624800b200 con 0x7f624c06fca0 2022-01-31T19:37:43.277 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.265+0000 7f6252215700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c10ced0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6248011dd0 tx=0x7f6248006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:43.277 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.266+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f624801b410 con 0x7f624c06fca0 2022-01-31T19:37:43.278 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.266+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f624801b9d0 con 0x7f624c06fca0 2022-01-31T19:37:43.278 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.266+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c10d5c0 con 0x7f624c06fca0 2022-01-31T19:37:43.279 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.266+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f624801a720 con 0x7f624c06fca0 2022-01-31T19:37:43.279 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.266+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624c10e170 con 0x7f624c06fca0 2022-01-31T19:37:43.280 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.266+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f624802b070 con 0x7f624c06fca0 2022-01-31T19:37:43.281 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.267+0000 7f623affd700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f6234030fa0 0x7f6234033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:43.281 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.267+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6248042ce0 con 0x7f624c06fca0 2022-01-31T19:37:43.282 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.267+0000 7f6251a14700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f6234030fa0 0x7f6234033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:43.282 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.267+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624c046a20 con 0x7f624c06fca0 2022-01-31T19:37:43.282 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.267+0000 7f6251a14700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f6234030fa0 0x7f6234033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f623c0096a0 tx=0x7f623c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:43.283 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.272+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f624801f020 con 0x7f624c06fca0 2022-01-31T19:37:43.444 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.443+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f624c102120 con 0x7f6234030fa0 2022-01-31T19:37:43.448 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.447+0000 7f623affd700 1 -- 172.21.15.167:0/2868452334 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f624c102120 con 0x7f6234030fa0 2022-01-31T19:37:43.448 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled mgr update... 2022-01-31T19:37:43.451 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f6234030fa0 msgr2=0x7f6234033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:43.452 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f6234030fa0 0x7f6234033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f623c0096a0 tx=0x7f623c006b40).stop 2022-01-31T19:37:43.452 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 msgr2=0x7f624c10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:43.453 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c10ced0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6248011dd0 tx=0x7f6248006ee0).stop 2022-01-31T19:37:43.453 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 shutdown_connections 2022-01-31T19:37:43.454 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f6234030fa0 0x7f6234033460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:43.454 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 --2- 172.21.15.167:0/2868452334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624c06fca0 0x7f624c10ced0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:43.454 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 >> 172.21.15.167:0/2868452334 conn(0x7f624c0f6530 msgr2=0x7f624c0f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:43.455 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 shutdown_connections 2022-01-31T19:37:43.455 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:43.449+0000 7f6254479700 1 -- 172.21.15.167:0/2868452334 wait complete. 2022-01-31T19:37:43.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:43 smithi167 conmon[32206]: audit 2022-01-31T19:37:42.357457+0000 mgr.smithi167.aciqpk (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:43.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:43 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:42.358180+0000 mgr.smithi167.aciqpk (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2022-01-31T19:37:43.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:43 smithi167 conmon[32206]: audit 2022-01-31T19:37:43.447651+0000 mon.smithi167 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:43.836 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying crash service with default placement... 2022-01-31T19:37:44.379 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.376+0000 7fdbb64cc700 1 Processor -- start 2022-01-31T19:37:44.379 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.376+0000 7fdbb64cc700 1 -- start start 2022-01-31T19:37:44.379 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.376+0000 7fdbb64cc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb00fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:44.380 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbb64cc700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00fcd00 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.380 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb00fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:44.380 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb00fc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60466/0 (socket says 172.21.15.167:60466) 2022-01-31T19:37:44.380 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaffff700 1 -- 172.21.15.167:0/639796644 learned_addr learned my addr 172.21.15.167:0/639796644 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:44.381 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaffff700 1 -- 172.21.15.167:0/639796644 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00fce40 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.381 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaffff700 1 --2- 172.21.15.167:0/639796644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb00fc810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fdba000b870 tx=0x7fdba0006f70).ready entity=mon.0 client_cookie=f29032c6e7ae4d78 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:44.381 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaeffd700 1 -- 172.21.15.167:0/639796644 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdba000da20 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.381 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaeffd700 1 -- 172.21.15.167:0/639796644 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdba0010470 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.381 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.377+0000 7fdbaeffd700 1 -- 172.21.15.167:0/639796644 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdba000f5b0 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.386 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 -- 172.21.15.167:0/639796644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 msgr2=0x7fdbb00fc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:44.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 --2- 172.21.15.167:0/639796644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb00fc810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fdba000b870 tx=0x7fdba0006f70).stop 2022-01-31T19:37:44.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 -- 172.21.15.167:0/639796644 shutdown_connections 2022-01-31T19:37:44.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 --2- 172.21.15.167:0/639796644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb00fc810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:44.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 -- 172.21.15.167:0/639796644 >> 172.21.15.167:0/639796644 conn(0x7fdbb00f6520 msgr2=0x7fdbb00f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:44.388 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 -- 172.21.15.167:0/639796644 shutdown_connections 2022-01-31T19:37:44.388 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.378+0000 7fdbb64cc700 1 -- 172.21.15.167:0/639796644 wait complete. 2022-01-31T19:37:44.388 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.379+0000 7fdbb64cc700 1 Processor -- start 2022-01-31T19:37:44.388 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.379+0000 7fdbb64cc700 1 -- start start 2022-01-31T19:37:44.389 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.379+0000 7fdbb64cc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb010ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:44.389 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.379+0000 7fdbb64cc700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb010d370 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.389 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbaffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb010ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:44.389 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbaffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb010ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60468/0 (socket says 172.21.15.167:60468) 2022-01-31T19:37:44.389 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbaffff700 1 -- 172.21.15.167:0/3903503220 learned_addr learned my addr 172.21.15.167:0/3903503220 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:44.390 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbaffff700 1 -- 172.21.15.167:0/3903503220 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba000b480 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.390 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbaffff700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb010ce80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fdba0006800 tx=0x7fdba0006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:44.390 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdba000de80 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.390 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdba000fcb0 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.391 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdba0019ca0 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.391 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb010d570 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.391 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.380+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb010da00 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.392 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.381+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fdba0016030 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.392 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.381+0000 7fdbacff9700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fdb9c030ff0 0x7fdb9c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:44.393 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.381+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fdba00432c0 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.393 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.381+0000 7fdbaf7fe700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fdb9c030ff0 0x7fdb9c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:44.393 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.382+0000 7fdbaf7fe700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fdb9c030ff0 0x7fdb9c0334b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fdba40096a0 tx=0x7fdba4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:44.394 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.382+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb00591e0 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.394 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.386+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdba0014070 con 0x7fdbb00fc3f0 2022-01-31T19:37:44.557 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.556+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbb01027c0 con 0x7fdb9c030ff0 2022-01-31T19:37:44.561 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.560+0000 7fdbacff9700 1 -- 172.21.15.167:0/3903503220 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fdbb01027c0 con 0x7fdb9c030ff0 2022-01-31T19:37:44.561 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled crash update... 2022-01-31T19:37:44.563 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fdb9c030ff0 msgr2=0x7fdb9c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:44.564 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fdb9c030ff0 0x7fdb9c0334b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fdba40096a0 tx=0x7fdba4006b40).stop 2022-01-31T19:37:44.564 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 msgr2=0x7fdbb010ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:44.564 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb010ce80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fdba0006800 tx=0x7fdba0006f70).stop 2022-01-31T19:37:44.564 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 shutdown_connections 2022-01-31T19:37:44.565 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fdb9c030ff0 0x7fdb9c0334b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:44.565 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 --2- 172.21.15.167:0/3903503220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb00fc3f0 0x7fdbb010ce80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:44.565 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 >> 172.21.15.167:0/3903503220 conn(0x7fdbb00f6520 msgr2=0x7fdbb00ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:44.566 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 shutdown_connections 2022-01-31T19:37:44.566 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:44.562+0000 7fdbb64cc700 1 -- 172.21.15.167:0/3903503220 wait complete. 2022-01-31T19:37:45.075 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying prometheus service with default placement... 2022-01-31T19:37:45.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:44 smithi167 conmon[32206]: audit 2022-01-31T19:37:43.444835+0000 mgr.smithi167.aciqpk (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:44 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:43.445416+0000 mgr.smithi167.aciqpk (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2022-01-31T19:37:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:44 smithi167 conmon[32206]: audit 2022-01-31T19:37:43.998722+0000 mon.smithi167 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:45.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:44 smithi167 conmon[32206]: audit 2022-01-31T19:37:44.347461+0000 mon.smithi167 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:45.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:44 smithi167 conmon[32206]: audit 2022-01-31T19:37:44.560684+0000 mon.smithi167 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:45.547 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.543+0000 7efebaf43700 1 Processor -- start 2022-01-31T19:37:45.548 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.543+0000 7efebaf43700 1 -- start start 2022-01-31T19:37:45.548 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efebaf43700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb40feb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:45.548 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efebaf43700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40ff050 con 0x7efeb406fc20 2022-01-31T19:37:45.549 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efeb8cdf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb40feb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:45.549 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efeb8cdf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb40feb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60472/0 (socket says 172.21.15.167:60472) 2022-01-31T19:37:45.549 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efeb8cdf700 1 -- 172.21.15.167:0/879593966 learned_addr learned my addr 172.21.15.167:0/879593966 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:45.549 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efeb8cdf700 1 -- 172.21.15.167:0/879593966 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb40ff190 con 0x7efeb406fc20 2022-01-31T19:37:45.550 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.544+0000 7efeb8cdf700 1 --2- 172.21.15.167:0/879593966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb40feb60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7efea400b5f0 tx=0x7efea4006ee0).ready entity=mon.0 client_cookie=f963e8df370f810e server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:45.550 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efeb37fe700 1 -- 172.21.15.167:0/879593966 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea400d600 con 0x7efeb406fc20 2022-01-31T19:37:45.550 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efeb37fe700 1 -- 172.21.15.167:0/879593966 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efea400dbf0 con 0x7efeb406fc20 2022-01-31T19:37:45.550 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efeb37fe700 1 -- 172.21.15.167:0/879593966 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea4011bf0 con 0x7efeb406fc20 2022-01-31T19:37:45.551 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 -- 172.21.15.167:0/879593966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 msgr2=0x7efeb40feb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:45.551 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 --2- 172.21.15.167:0/879593966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb40feb60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7efea400b5f0 tx=0x7efea4006ee0).stop 2022-01-31T19:37:45.551 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 -- 172.21.15.167:0/879593966 shutdown_connections 2022-01-31T19:37:45.551 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 --2- 172.21.15.167:0/879593966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb40feb60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:45.551 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 -- 172.21.15.167:0/879593966 >> 172.21.15.167:0/879593966 conn(0x7efeb40f6520 msgr2=0x7efeb40f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:45.552 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 -- 172.21.15.167:0/879593966 shutdown_connections 2022-01-31T19:37:45.552 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.545+0000 7efebaf43700 1 -- 172.21.15.167:0/879593966 wait complete. 2022-01-31T19:37:45.552 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efebaf43700 1 Processor -- start 2022-01-31T19:37:45.552 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efebaf43700 1 -- start start 2022-01-31T19:37:45.553 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efebaf43700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb410acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:45.553 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efebaf43700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb410b1e0 con 0x7efeb406fc20 2022-01-31T19:37:45.553 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efeb8cdf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb410acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:45.553 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efeb8cdf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb410acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60474/0 (socket says 172.21.15.167:60474) 2022-01-31T19:37:45.554 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.546+0000 7efeb8cdf700 1 -- 172.21.15.167:0/3225010617 learned_addr learned my addr 172.21.15.167:0/3225010617 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:45.554 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efeb8cdf700 1 -- 172.21.15.167:0/3225010617 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea400b200 con 0x7efeb406fc20 2022-01-31T19:37:45.555 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efeb8cdf700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb410acf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7efea4006800 tx=0x7efea4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:45.555 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea401b410 con 0x7efeb406fc20 2022-01-31T19:37:45.556 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efea401ba00 con 0x7efeb406fc20 2022-01-31T19:37:45.556 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb410b3e0 con 0x7efeb406fc20 2022-01-31T19:37:45.556 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea401a6e0 con 0x7efeb406fc20 2022-01-31T19:37:45.556 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.547+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb410bf90 con 0x7efeb406fc20 2022-01-31T19:37:45.557 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.548+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7efea402b070 con 0x7efeb406fc20 2022-01-31T19:37:45.557 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.548+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4046a20 con 0x7efeb406fc20 2022-01-31T19:37:45.557 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.548+0000 7efeb17fa700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7efe94030ff0 0x7efe940334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:45.557 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.548+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7efea4042ca0 con 0x7efeb406fc20 2022-01-31T19:37:45.558 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.548+0000 7efeb3fff700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7efe94030ff0 0x7efe940334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:45.558 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.549+0000 7efeb3fff700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7efe94030ff0 0x7efe940334b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efe9c0096a0 tx=0x7efe9c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:45.558 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.553+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efea402b6c0 con 0x7efeb406fc20 2022-01-31T19:37:45.724 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.723+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7efeb410c1d0 con 0x7efe94030ff0 2022-01-31T19:37:45.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.727+0000 7efeb17fa700 1 -- 172.21.15.167:0/3225010617 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7efeb410c1d0 con 0x7efe94030ff0 2022-01-31T19:37:45.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled prometheus update... 2022-01-31T19:37:45.731 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7efe94030ff0 msgr2=0x7efe940334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:45.731 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7efe94030ff0 0x7efe940334b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efe9c0096a0 tx=0x7efe9c006b40).stop 2022-01-31T19:37:45.731 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 msgr2=0x7efeb410acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:45.731 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb410acf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7efea4006800 tx=0x7efea4006ee0).stop 2022-01-31T19:37:45.732 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 shutdown_connections 2022-01-31T19:37:45.732 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7efe94030ff0 0x7efe940334b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:45.732 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 --2- 172.21.15.167:0/3225010617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efeb406fc20 0x7efeb410acf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:45.732 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.729+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 >> 172.21.15.167:0/3225010617 conn(0x7efeb40f6520 msgr2=0x7efeb40f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:45.733 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.730+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 shutdown_connections 2022-01-31T19:37:45.733 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:45.730+0000 7efebaf43700 1 -- 172.21.15.167:0/3225010617 wait complete. 2022-01-31T19:37:46.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:45 smithi167 conmon[32206]: audit 2022-01-31T19:37:44.557777+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:37:46.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:45 smithi167 conmon[32206]: 14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:46.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:45 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:44.558417+0000 mgr.smithi167.aciqpk (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2022-01-31T19:37:46.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:45 smithi167 conmon[32206]: audit 2022-01-31T19:37:45.317066+0000 mon.smithi167 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:46.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:45 smithi167 conmon[32206]: audit 2022-01-31T19:37:45.727985+0000 mon.smithi167 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying grafana service with default placement... 2022-01-31T19:37:47.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:46 smithi167 conmon[32206]: audit 2022-01-31T19:37:45.725135+0000 mgr.smithi167.aciqpk (mgr.14116 2022-01-31T19:37:47.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:46 smithi167 conmon[32206]: ) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:47.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:46 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:45.725700+0000 mgr.smithi167.aciqpk (mgr.14116) 17 : cephadm [INF] Saving service prometheus spec with placement count:1 2022-01-31T19:37:47.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeb0a0c700 1 Processor -- start 2022-01-31T19:37:47.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeb0a0c700 1 -- start start 2022-01-31T19:37:47.079 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeb0a0c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac0fc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:47.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeb0a0c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0fccf0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeaa59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac0fc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:47.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeaa59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac0fc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60478/0 (socket says 172.21.15.167:60478) 2022-01-31T19:37:47.080 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.071+0000 7fbeaa59c700 1 -- 172.21.15.167:0/1384620470 learned_addr learned my addr 172.21.15.167:0/1384620470 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:47.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.072+0000 7fbeaa59c700 1 -- 172.21.15.167:0/1384620470 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac0fce30 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.072+0000 7fbeaa59c700 1 --2- 172.21.15.167:0/1384620470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac0fc800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbe9c00b5f0 tx=0x7fbe9c006ee0).ready entity=mon.0 client_cookie=41079739ae956c6c server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:47.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.072+0000 7fbea959a700 1 -- 172.21.15.167:0/1384620470 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9c00d600 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.081 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.072+0000 7fbea959a700 1 -- 172.21.15.167:0/1384620470 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe9c00dbf0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.072+0000 7fbea959a700 1 -- 172.21.15.167:0/1384620470 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9c011bf0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/1384620470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 msgr2=0x7fbeac0fc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:47.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 --2- 172.21.15.167:0/1384620470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac0fc800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbe9c00b5f0 tx=0x7fbe9c006ee0).stop 2022-01-31T19:37:47.082 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/1384620470 shutdown_connections 2022-01-31T19:37:47.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 --2- 172.21.15.167:0/1384620470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac0fc800 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:47.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/1384620470 >> 172.21.15.167:0/1384620470 conn(0x7fbeac0f6530 msgr2=0x7fbeac0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:47.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/1384620470 shutdown_connections 2022-01-31T19:37:47.083 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/1384620470 wait complete. 2022-01-31T19:37:47.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 Processor -- start 2022-01-31T19:37:47.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.073+0000 7fbeb0a0c700 1 -- start start 2022-01-31T19:37:47.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeb0a0c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac109b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:47.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeb0a0c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac10a030 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.084 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeaa59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac109b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:47.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeaa59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac109b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60480/0 (socket says 172.21.15.167:60480) 2022-01-31T19:37:47.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeaa59c700 1 -- 172.21.15.167:0/3428635926 learned_addr learned my addr 172.21.15.167:0/3428635926 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:47.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeaa59c700 1 -- 172.21.15.167:0/3428635926 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c00b200 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.085 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeaa59c700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac109b40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fbe9c011e30 tx=0x7fbe9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:47.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9c01b410 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe9c01ba00 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeac10a230 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.086 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9c01a7b0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.074+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeac10a6c0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.075+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fbe9c02b070 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.075+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeac045bb0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.075+0000 7fbe9affd700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fbe90030fa0 0x7fbe90033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:47.087 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.075+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fbe9c042ca0 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.076+0000 7fbea9d9b700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fbe90030fa0 0x7fbe90033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:47.088 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.076+0000 7fbea9d9b700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fbe90030fa0 0x7fbe90033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:47.090 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.080+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbe9c02b950 con 0x7fbeac0fc3e0 2022-01-31T19:37:47.241 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.241+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeac10af20 con 0x7fbe90030fa0 2022-01-31T19:37:47.246 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.245+0000 7fbe9affd700 1 -- 172.21.15.167:0/3428635926 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fbeac10af20 con 0x7fbe90030fa0 2022-01-31T19:37:47.246 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled grafana update... 2022-01-31T19:37:47.249 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fbe90030fa0 msgr2=0x7fbe90033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:47.249 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fbe90030fa0 0x7fbe90033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).stop 2022-01-31T19:37:47.249 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 msgr2=0x7fbeac109b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:47.250 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac109b40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fbe9c011e30 tx=0x7fbe9c006ee0).stop 2022-01-31T19:37:47.250 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 shutdown_connections 2022-01-31T19:37:47.250 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7fbe90030fa0 0x7fbe90033460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:47.251 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 --2- 172.21.15.167:0/3428635926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbeac0fc3e0 0x7fbeac109b40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:47.251 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 >> 172.21.15.167:0/3428635926 conn(0x7fbeac0f6530 msgr2=0x7fbeac0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:47.251 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 shutdown_connections 2022-01-31T19:37:47.251 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:47.247+0000 7fbeb0a0c700 1 -- 172.21.15.167:0/3428635926 wait complete. 2022-01-31T19:37:47.650 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying node-exporter service with default placement... 2022-01-31T19:37:48.112 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44ad462700 1 Processor -- start 2022-01-31T19:37:48.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44ad462700 1 -- start start 2022-01-31T19:37:48.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44ad462700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a80faac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:48.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44ad462700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a806fce0 con 0x7f44a80fa6a0 2022-01-31T19:37:48.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44a7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a80faac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:48.113 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44a7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a80faac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60484/0 (socket says 172.21.15.167:60484) 2022-01-31T19:37:48.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.108+0000 7f44a7fff700 1 -- 172.21.15.167:0/3210629679 learned_addr learned my addr 172.21.15.167:0/3210629679 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:48.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44a7fff700 1 -- 172.21.15.167:0/3210629679 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a806fe20 con 0x7f44a80fa6a0 2022-01-31T19:37:48.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44a7fff700 1 --2- 172.21.15.167:0/3210629679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a80faac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f449800b5f0 tx=0x7f4498006ee0).ready entity=mon.0 client_cookie=f8f2be4a2d8ef288 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:48.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44a6ffd700 1 -- 172.21.15.167:0/3210629679 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449800d600 con 0x7f44a80fa6a0 2022-01-31T19:37:48.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44a6ffd700 1 -- 172.21.15.167:0/3210629679 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f449800dbf0 con 0x7f44a80fa6a0 2022-01-31T19:37:48.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44a6ffd700 1 -- 172.21.15.167:0/3210629679 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4498011bf0 con 0x7f44a80fa6a0 2022-01-31T19:37:48.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44ad462700 1 -- 172.21.15.167:0/3210629679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 msgr2=0x7f44a80faac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:48.115 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44ad462700 1 --2- 172.21.15.167:0/3210629679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a80faac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f449800b5f0 tx=0x7f4498006ee0).stop 2022-01-31T19:37:48.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44ad462700 1 -- 172.21.15.167:0/3210629679 shutdown_connections 2022-01-31T19:37:48.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44ad462700 1 --2- 172.21.15.167:0/3210629679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a80faac0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:48.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.109+0000 7f44ad462700 1 -- 172.21.15.167:0/3210629679 >> 172.21.15.167:0/3210629679 conn(0x7f44a80f6340 msgr2=0x7f44a80f8780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:48.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.110+0000 7f44ad462700 1 -- 172.21.15.167:0/3210629679 shutdown_connections 2022-01-31T19:37:48.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.110+0000 7f44ad462700 1 -- 172.21.15.167:0/3210629679 wait complete. 2022-01-31T19:37:48.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.110+0000 7f44ad462700 1 Processor -- start 2022-01-31T19:37:48.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.110+0000 7f44ad462700 1 -- start start 2022-01-31T19:37:48.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44ad462700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a810cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:48.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44ad462700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a810d230 con 0x7f44a80fa6a0 2022-01-31T19:37:48.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a810cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:48.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a810cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60486/0 (socket says 172.21.15.167:60486) 2022-01-31T19:37:48.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a7fff700 1 -- 172.21.15.167:0/3780631766 learned_addr learned my addr 172.21.15.167:0/3780631766 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:48.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a7fff700 1 -- 172.21.15.167:0/3780631766 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449800b200 con 0x7f44a80fa6a0 2022-01-31T19:37:48.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a7fff700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a810cd40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f4498011de0 tx=0x7f4498006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:48.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449801b410 con 0x7f44a80fa6a0 2022-01-31T19:37:48.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f449801ba00 con 0x7f44a80fa6a0 2022-01-31T19:37:48.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a810d430 con 0x7f44a80fa6a0 2022-01-31T19:37:48.119 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449801a740 con 0x7f44a80fa6a0 2022-01-31T19:37:48.120 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.111+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a810dfe0 con 0x7f44a80fa6a0 2022-01-31T19:37:48.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.112+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f449802b070 con 0x7f44a80fa6a0 2022-01-31T19:37:48.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.112+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8046a20 con 0x7f44a80fa6a0 2022-01-31T19:37:48.121 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.112+0000 7f44a4ff9700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f4494030ff0 0x7f44940334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:48.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.112+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f4498042c50 con 0x7f44a80fa6a0 2022-01-31T19:37:48.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.113+0000 7f44a77fe700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f4494030ff0 0x7f44940334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:48.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.113+0000 7f44a77fe700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f4494030ff0 0x7f44940334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f449c0096a0 tx=0x7f449c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:48.122 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.117+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f449802b8f0 con 0x7f44a80fa6a0 2022-01-31T19:37:48.280 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.278+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f44a810e2e0 con 0x7f4494030ff0 2022-01-31T19:37:48.283 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.283+0000 7f44a4ff9700 1 -- 172.21.15.167:0/3780631766 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f44a810e2e0 con 0x7f4494030ff0 2022-01-31T19:37:48.284 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2022-01-31T19:37:48.286 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.284+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f4494030ff0 msgr2=0x7f44940334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:48.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.284+0000 7f44ad462700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f4494030ff0 0x7f44940334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f449c0096a0 tx=0x7f449c006b40).stop 2022-01-31T19:37:48.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 msgr2=0x7f44a810cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:48.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a810cd40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f4498011de0 tx=0x7f4498006ee0).stop 2022-01-31T19:37:48.287 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 shutdown_connections 2022-01-31T19:37:48.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f4494030ff0 0x7f44940334b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:48.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 --2- 172.21.15.167:0/3780631766 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a80fa6a0 0x7f44a810cd40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:48.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 >> 172.21.15.167:0/3780631766 conn(0x7f44a80f6340 msgr2=0x7f44a8101510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:48.288 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 shutdown_connections 2022-01-31T19:37:48.289 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:48.285+0000 7f44ad462700 1 -- 172.21.15.167:0/3780631766 wait complete. 2022-01-31T19:37:48.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:48 smithi167 conmon[32206]: audit 2022-01-31T19:37:47.242162+0000 mgr.smithi167.aciqpk (mgr.14116) 18 2022-01-31T19:37:48.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:48 smithi167 conmon[32206]: : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:48.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:48 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:47.242696+0000 mgr.smithi167.aciqpk (mgr.14116) 19 : cephadm [INF] Saving service grafana spec with placement count:1 2022-01-31T19:37:48.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:48 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:37:48.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:48 smithi167 conmon[32206]: T19:37:47.245733+0000 mon.smithi167 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:48.687 INFO:teuthology.orchestra.run.smithi167.stderr:Deploying alertmanager service with default placement... 2022-01-31T19:37:49.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.149+0000 7f064e0cf700 1 Processor -- start 2022-01-31T19:37:49.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.149+0000 7f064e0cf700 1 -- start start 2022-01-31T19:37:49.154 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f064e0cf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f0648100f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:49.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f064e0cf700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06480fc910 con 0x7f064806fc60 2022-01-31T19:37:49.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f06477fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f0648100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:49.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f06477fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f0648100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60490/0 (socket says 172.21.15.167:60490) 2022-01-31T19:37:49.155 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f06477fe700 1 -- 172.21.15.167:0/801890013 learned_addr learned my addr 172.21.15.167:0/801890013 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:49.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f06477fe700 1 -- 172.21.15.167:0/801890013 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0648101410 con 0x7f064806fc60 2022-01-31T19:37:49.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f06477fe700 1 --2- 172.21.15.167:0/801890013 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f0648100f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f063000b5f0 tx=0x7f0630006ee0).ready entity=mon.0 client_cookie=f42adc288311307 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:49.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.150+0000 7f06467fc700 1 -- 172.21.15.167:0/801890013 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063000d600 con 0x7f064806fc60 2022-01-31T19:37:49.156 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f06467fc700 1 -- 172.21.15.167:0/801890013 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f063000dbf0 con 0x7f064806fc60 2022-01-31T19:37:49.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f06467fc700 1 -- 172.21.15.167:0/801890013 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0630011bf0 con 0x7f064806fc60 2022-01-31T19:37:49.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 -- 172.21.15.167:0/801890013 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 msgr2=0x7f0648100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:49.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 --2- 172.21.15.167:0/801890013 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f0648100f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f063000b5f0 tx=0x7f0630006ee0).stop 2022-01-31T19:37:49.157 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 -- 172.21.15.167:0/801890013 shutdown_connections 2022-01-31T19:37:49.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 --2- 172.21.15.167:0/801890013 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f0648100f20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:49.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 -- 172.21.15.167:0/801890013 >> 172.21.15.167:0/801890013 conn(0x7f06480f6520 msgr2=0x7f06480f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:49.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 -- 172.21.15.167:0/801890013 shutdown_connections 2022-01-31T19:37:49.158 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.151+0000 7f064e0cf700 1 -- 172.21.15.167:0/801890013 wait complete. 2022-01-31T19:37:49.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.152+0000 7f064e0cf700 1 Processor -- start 2022-01-31T19:37:49.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.152+0000 7f064e0cf700 1 -- start start 2022-01-31T19:37:49.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.152+0000 7f064e0cf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f064810ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:49.159 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.152+0000 7f064e0cf700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064810d370 con 0x7f064806fc60 2022-01-31T19:37:49.160 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f06477fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f064810ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:49.160 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f06477fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f064810ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60492/0 (socket says 172.21.15.167:60492) 2022-01-31T19:37:49.160 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f06477fe700 1 -- 172.21.15.167:0/3945778917 learned_addr learned my addr 172.21.15.167:0/3945778917 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:49.160 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f06477fe700 1 -- 172.21.15.167:0/3945778917 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063000b200 con 0x7f064806fc60 2022-01-31T19:37:49.160 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f06477fe700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f064810ce80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0630011e00 tx=0x7f0630006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:49.162 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063001b410 con 0x7f064806fc60 2022-01-31T19:37:49.162 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f063001ba00 con 0x7f064806fc60 2022-01-31T19:37:49.162 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064810d570 con 0x7f064806fc60 2022-01-31T19:37:49.163 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063001a780 con 0x7f064806fc60 2022-01-31T19:37:49.163 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.153+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064810da00 con 0x7f064806fc60 2022-01-31T19:37:49.163 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.154+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f063002b070 con 0x7f064806fc60 2022-01-31T19:37:49.163 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.154+0000 7f063ffff700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0628030ff0 0x7f06280334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:49.163 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.154+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0630042c60 con 0x7f064806fc60 2022-01-31T19:37:49.164 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.154+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06480591e0 con 0x7f064806fc60 2022-01-31T19:37:49.164 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.154+0000 7f0646ffd700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0628030ff0 0x7f06280334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:49.164 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.155+0000 7f0646ffd700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0628030ff0 0x7f06280334b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f06380096a0 tx=0x7f0638006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:49.164 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.159+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f063001f020 con 0x7f064806fc60 2022-01-31T19:37:49.321 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.320+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f0648106e50 con 0x7f0628030ff0 2022-01-31T19:37:49.327 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.326+0000 7f063ffff700 1 -- 172.21.15.167:0/3945778917 <== mgr.14116 v2:172.21.15.167:6800/2257398586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f0648106e50 con 0x7f0628030ff0 2022-01-31T19:37:49.328 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2022-01-31T19:37:49.330 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0628030ff0 msgr2=0x7f06280334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:49.330 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0628030ff0 0x7f06280334b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f06380096a0 tx=0x7f0638006b40).stop 2022-01-31T19:37:49.330 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 msgr2=0x7f064810ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:49.330 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f064810ce80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0630011e00 tx=0x7f0630006ee0).stop 2022-01-31T19:37:49.331 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 shutdown_connections 2022-01-31T19:37:49.331 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f0628030ff0 0x7f06280334b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:49.331 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 --2- 172.21.15.167:0/3945778917 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f064806fc60 0x7f064810ce80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:49.331 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 >> 172.21.15.167:0/3945778917 conn(0x7f06480f6520 msgr2=0x7f06480f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:49.332 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 shutdown_connections 2022-01-31T19:37:49.332 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:49.328+0000 7f064e0cf700 1 -- 172.21.15.167:0/3945778917 wait complete. 2022-01-31T19:37:49.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:49 smithi167 conmon[32206]: audit 2022-01-31T19:37:48.280103+0000 mgr.smithi167.aciqpk (mgr.14116) 20 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:49.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:49 smithi167 conmon[32206]: cephadm 2022-01-31T19 2022-01-31T19:37:49.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:49 smithi167 conmon[32206]: :37:48.280623+0000 mgr.smithi167.aciqpk (mgr.14116) 21 : cephadm [INF] Saving service node-exporter spec with placement * 2022-01-31T19:37:49.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:49 smithi167 conmon[32206]: audit 2022-01-31T19:37:48.283446+0000 mon.smithi167 (mon.0) 65 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:50.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.197+0000 7f446b59e700 1 Processor -- start 2022-01-31T19:37:50.202 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.197+0000 7f446b59e700 1 -- start start 2022-01-31T19:37:50.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446b59e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c0fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:50.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446b59e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c0fcb20 con 0x7f446c0fc210 2022-01-31T19:37:50.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c0fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:50.203 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c0fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60496/0 (socket says 172.21.15.167:60496) 2022-01-31T19:37:50.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446a59c700 1 -- 172.21.15.167:0/1568026522 learned_addr learned my addr 172.21.15.167:0/1568026522 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:50.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446a59c700 1 -- 172.21.15.167:0/1568026522 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c0fcc60 con 0x7f446c0fc210 2022-01-31T19:37:50.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.198+0000 7f446a59c700 1 --2- 172.21.15.167:0/1568026522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c0fc630 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f445c009960 tx=0x7f445c0092d0).ready entity=mon.0 client_cookie=23efdab5a040c120 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:50.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446959a700 1 -- 172.21.15.167:0/1568026522 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f445c00b9c0 con 0x7f446c0fc210 2022-01-31T19:37:50.204 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446959a700 1 -- 172.21.15.167:0/1568026522 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f445c011400 con 0x7f446c0fc210 2022-01-31T19:37:50.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446959a700 1 -- 172.21.15.167:0/1568026522 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f445c010580 con 0x7f446c0fc210 2022-01-31T19:37:50.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 -- 172.21.15.167:0/1568026522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 msgr2=0x7f446c0fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:50.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 --2- 172.21.15.167:0/1568026522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c0fc630 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f445c009960 tx=0x7f445c0092d0).stop 2022-01-31T19:37:50.205 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 -- 172.21.15.167:0/1568026522 shutdown_connections 2022-01-31T19:37:50.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 --2- 172.21.15.167:0/1568026522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c0fc630 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:50.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 -- 172.21.15.167:0/1568026522 >> 172.21.15.167:0/1568026522 conn(0x7f446c0f6340 msgr2=0x7f446c0f8780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:50.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 -- 172.21.15.167:0/1568026522 shutdown_connections 2022-01-31T19:37:50.206 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.199+0000 7f446b59e700 1 -- 172.21.15.167:0/1568026522 wait complete. 2022-01-31T19:37:50.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446b59e700 1 Processor -- start 2022-01-31T19:37:50.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446b59e700 1 -- start start 2022-01-31T19:37:50.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446b59e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c1099d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:50.207 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446b59e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c109ec0 con 0x7f446c0fc210 2022-01-31T19:37:50.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c1099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:50.208 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c1099d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60498/0 (socket says 172.21.15.167:60498) 2022-01-31T19:37:50.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.200+0000 7f446a59c700 1 -- 172.21.15.167:0/3839041157 learned_addr learned my addr 172.21.15.167:0/3839041157 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:50.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f446a59c700 1 -- 172.21.15.167:0/3839041157 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c009570 con 0x7f446c0fc210 2022-01-31T19:37:50.209 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f446a59c700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c1099d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f445c009fd0 tx=0x7f445c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:50.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f445c010840 con 0x7f446c0fc210 2022-01-31T19:37:50.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f445c010e30 con 0x7f446c0fc210 2022-01-31T19:37:50.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f445c019d80 con 0x7f446c0fc210 2022-01-31T19:37:50.210 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c10a0c0 con 0x7f446c0fc210 2022-01-31T19:37:50.211 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.201+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c10a550 con 0x7f446c0fc210 2022-01-31T19:37:50.212 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.202+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f445c02a070 con 0x7f446c0fc210 2022-01-31T19:37:50.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.202+0000 7f4462ffd700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f444c030ff0 0x7f444c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:50.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.202+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f445c044ad0 con 0x7f446c0fc210 2022-01-31T19:37:50.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.202+0000 7f4469d9b700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f444c030ff0 0x7f444c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:50.213 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.202+0000 7f4469d9b700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f444c030ff0 0x7f444c0334b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f44540096a0 tx=0x7f4454006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:50.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.202+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c0591e0 con 0x7f446c0fc210 2022-01-31T19:37:50.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.207+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f445c022390 con 0x7f446c0fc210 2022-01-31T19:37:50.360 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.359+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f446c1038c0 con 0x7f446c0fc210 2022-01-31T19:37:50.362 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.361+0000 7f4462ffd700 1 -- 172.21.15.167:0/3839041157 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f445c0320d0 con 0x7f446c0fc210 2022-01-31T19:37:50.366 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f444c030ff0 msgr2=0x7f444c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:50.366 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f444c030ff0 0x7f444c0334b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f44540096a0 tx=0x7f4454006b40).stop 2022-01-31T19:37:50.366 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 msgr2=0x7f446c1099d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:50.367 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c1099d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f445c009fd0 tx=0x7f445c0092d0).stop 2022-01-31T19:37:50.367 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 shutdown_connections 2022-01-31T19:37:50.367 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f444c030ff0 0x7f444c0334b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:50.367 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 --2- 172.21.15.167:0/3839041157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0fc210 0x7f446c1099d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:50.368 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 >> 172.21.15.167:0/3839041157 conn(0x7f446c0f6340 msgr2=0x7f446c0f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:50.368 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 shutdown_connections 2022-01-31T19:37:50.368 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:50.363+0000 7f446b59e700 1 -- 172.21.15.167:0/3839041157 wait complete. 2022-01-31T19:37:50.545 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:50 smithi167 conmon[32206]: audit 2022-01-31T19:37:49.321389+0000 mgr.smithi167.aciqpk (mgr.14116) 22 : audit 2022-01-31T19:37:50.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:50 smithi167 conmon[32206]: [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:37:50.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:50 smithi167 conmon[32206]: cephadm 2022-01-31T19:37:49.321898+0000 mgr.smithi167.aciqpk (mgr.14116) 23 : cephadm [INF] Saving service alertmanager spec with placement count:1 2022-01-31T19:37:50.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:50 smithi167 conmon[32206]: audit 2022-01-31T19:37:49.326902+0000 mon.smithi167 (mon.0) 66 : audit [INF] from='mgr.14116 172.21.15.167:0/2761908003' entity='mgr.smithi167.aciqpk' 2022-01-31T19:37:51.214 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.208+0000 7f24808aa700 1 Processor -- start 2022-01-31T19:37:51.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.208+0000 7f24808aa700 1 -- start start 2022-01-31T19:37:51.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.208+0000 7f24808aa700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c0feae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:51.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.208+0000 7f24808aa700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0fefd0 con 0x7f247c06fca0 2022-01-31T19:37:51.215 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c0feae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:51.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c0feae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60502/0 (socket says 172.21.15.167:60502) 2022-01-31T19:37:51.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247a59c700 1 -- 172.21.15.167:0/1701961955 learned_addr learned my addr 172.21.15.167:0/1701961955 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:51.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247a59c700 1 -- 172.21.15.167:0/1701961955 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0ff110 con 0x7f247c06fca0 2022-01-31T19:37:51.216 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247a59c700 1 --2- 172.21.15.167:0/1701961955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c0feae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f246c00b5f0 tx=0x7f246c006ee0).ready entity=mon.0 client_cookie=b80d61cf0cedfcd1 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:51.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247959a700 1 -- 172.21.15.167:0/1701961955 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f246c00d600 con 0x7f247c06fca0 2022-01-31T19:37:51.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247959a700 1 -- 172.21.15.167:0/1701961955 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f246c00dbf0 con 0x7f247c06fca0 2022-01-31T19:37:51.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.209+0000 7f247959a700 1 -- 172.21.15.167:0/1701961955 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f246c011bf0 con 0x7f247c06fca0 2022-01-31T19:37:51.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 -- 172.21.15.167:0/1701961955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 msgr2=0x7f247c0feae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:51.217 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 --2- 172.21.15.167:0/1701961955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c0feae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f246c00b5f0 tx=0x7f246c006ee0).stop 2022-01-31T19:37:51.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 -- 172.21.15.167:0/1701961955 shutdown_connections 2022-01-31T19:37:51.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 --2- 172.21.15.167:0/1701961955 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c0feae0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:51.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 -- 172.21.15.167:0/1701961955 >> 172.21.15.167:0/1701961955 conn(0x7f247c0f6520 msgr2=0x7f247c0f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:51.218 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 -- 172.21.15.167:0/1701961955 shutdown_connections 2022-01-31T19:37:51.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.210+0000 7f24808aa700 1 -- 172.21.15.167:0/1701961955 wait complete. 2022-01-31T19:37:51.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f24808aa700 1 Processor -- start 2022-01-31T19:37:51.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f24808aa700 1 -- start start 2022-01-31T19:37:51.219 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f24808aa700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c10ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:51.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f24808aa700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c10b200 con 0x7f247c06fca0 2022-01-31T19:37:51.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f247a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c10ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:51.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f247a59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c10ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60504/0 (socket says 172.21.15.167:60504) 2022-01-31T19:37:51.220 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.211+0000 7f247a59c700 1 -- 172.21.15.167:0/2227764777 learned_addr learned my addr 172.21.15.167:0/2227764777 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:51.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f247a59c700 1 -- 172.21.15.167:0/2227764777 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c00b200 con 0x7f247c06fca0 2022-01-31T19:37:51.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f247a59c700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c10ad10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f246c00bf20 tx=0x7f246c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:51.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f246c01b410 con 0x7f247c06fca0 2022-01-31T19:37:51.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f246c01ba00 con 0x7f247c06fca0 2022-01-31T19:37:51.221 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f246c01a6c0 con 0x7f247c06fca0 2022-01-31T19:37:51.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c10b400 con 0x7f247c06fca0 2022-01-31T19:37:51.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.212+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c10bfb0 con 0x7f247c06fca0 2022-01-31T19:37:51.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.213+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f246c02b070 con 0x7f247c06fca0 2022-01-31T19:37:51.222 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.213+0000 7f2472ffd700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f245c030fa0 0x7f245c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:51.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.213+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c046a20 con 0x7f247c06fca0 2022-01-31T19:37:51.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.213+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f246c042bb0 con 0x7f247c06fca0 2022-01-31T19:37:51.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.213+0000 7f2479d9b700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f245c030fa0 0x7f245c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:51.223 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.213+0000 7f2479d9b700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f245c030fa0 0x7f245c033460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f24640096a0 tx=0x7f2464006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:51.224 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.218+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f246c01b570 con 0x7f247c06fca0 2022-01-31T19:37:51.371 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.370+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f247c0591e0 con 0x7f247c06fca0 2022-01-31T19:37:51.373 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.373+0000 7f2472ffd700 1 -- 172.21.15.167:0/2227764777 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f246c023e40 con 0x7f247c06fca0 2022-01-31T19:37:51.376 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f245c030fa0 msgr2=0x7f245c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:51.376 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f245c030fa0 0x7f245c033460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f24640096a0 tx=0x7f2464006b40).stop 2022-01-31T19:37:51.377 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 msgr2=0x7f247c10ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:51.377 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c10ad10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f246c00bf20 tx=0x7f246c006ee0).stop 2022-01-31T19:37:51.377 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 shutdown_connections 2022-01-31T19:37:51.377 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f245c030fa0 0x7f245c033460 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:51.378 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 --2- 172.21.15.167:0/2227764777 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f247c06fca0 0x7f247c10ad10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:51.378 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 >> 172.21.15.167:0/2227764777 conn(0x7f247c0f6520 msgr2=0x7f247c0f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:51.378 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 shutdown_connections 2022-01-31T19:37:51.378 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:51.375+0000 7f24808aa700 1 -- 172.21.15.167:0/2227764777 wait complete. 2022-01-31T19:37:51.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:51 smithi167 conmon[32206]: audit 2022-01-31T19:37:50.362054+0000 mon.smithi167 (mon.0) 67 2022-01-31T19:37:51.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:51 smithi167 conmon[32206]: : audit [INF] from='client.? 172.21.15.167:0/3839041157' entity='client.admin' 2022-01-31T19:37:51.992 INFO:teuthology.orchestra.run.smithi167.stderr:Enabling the dashboard module... 2022-01-31T19:37:52.438 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.434+0000 7f303726a700 1 Processor -- start 2022-01-31T19:37:52.438 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.434+0000 7f303726a700 1 -- start start 2022-01-31T19:37:52.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.434+0000 7f303726a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f30300fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:52.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.434+0000 7f303726a700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300fdcb0 con 0x7f30300fd3a0 2022-01-31T19:37:52.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.434+0000 7f3036268700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f30300fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:52.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3036268700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f30300fd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60508/0 (socket says 172.21.15.167:60508) 2022-01-31T19:37:52.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3036268700 1 -- 172.21.15.167:0/744131526 learned_addr learned my addr 172.21.15.167:0/744131526 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:52.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3036268700 1 -- 172.21.15.167:0/744131526 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300fddf0 con 0x7f30300fd3a0 2022-01-31T19:37:52.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3036268700 1 --2- 172.21.15.167:0/744131526 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f30300fd7c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f302000b5f0 tx=0x7f3020006ee0).ready entity=mon.0 client_cookie=b4e4705a2f899067 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:52.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3035266700 1 -- 172.21.15.167:0/744131526 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302000d600 con 0x7f30300fd3a0 2022-01-31T19:37:52.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3035266700 1 -- 172.21.15.167:0/744131526 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f302000dbf0 con 0x7f30300fd3a0 2022-01-31T19:37:52.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.435+0000 7f3035266700 1 -- 172.21.15.167:0/744131526 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3020011bf0 con 0x7f30300fd3a0 2022-01-31T19:37:52.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- 172.21.15.167:0/744131526 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 msgr2=0x7f30300fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:52.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 --2- 172.21.15.167:0/744131526 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f30300fd7c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f302000b5f0 tx=0x7f3020006ee0).stop 2022-01-31T19:37:52.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- 172.21.15.167:0/744131526 shutdown_connections 2022-01-31T19:37:52.442 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 --2- 172.21.15.167:0/744131526 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f30300fd7c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:52.442 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- 172.21.15.167:0/744131526 >> 172.21.15.167:0/744131526 conn(0x7f30300f5b00 msgr2=0x7f30300f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:52.442 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- 172.21.15.167:0/744131526 shutdown_connections 2022-01-31T19:37:52.442 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- 172.21.15.167:0/744131526 wait complete. 2022-01-31T19:37:52.443 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 Processor -- start 2022-01-31T19:37:52.443 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- start start 2022-01-31T19:37:52.444 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f303010ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:52.444 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f303726a700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303010b010 con 0x7f30300fd3a0 2022-01-31T19:37:52.444 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f3036268700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f303010ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:52.445 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f3036268700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f303010ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60510/0 (socket says 172.21.15.167:60510) 2022-01-31T19:37:52.445 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f3036268700 1 -- 172.21.15.167:0/1974783979 learned_addr learned my addr 172.21.15.167:0/1974783979 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:52.445 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f3036268700 1 -- 172.21.15.167:0/1974783979 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302000b200 con 0x7f30300fd3a0 2022-01-31T19:37:52.445 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f3036268700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f303010ab20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3020011e30 tx=0x7f3020006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:52.447 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.436+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302001b410 con 0x7f30300fd3a0 2022-01-31T19:37:52.447 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.437+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f302001ba00 con 0x7f30300fd3a0 2022-01-31T19:37:52.447 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.437+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302001a750 con 0x7f30300fd3a0 2022-01-31T19:37:52.447 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.437+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303010b210 con 0x7f30300fd3a0 2022-01-31T19:37:52.448 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.437+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303010b6a0 con 0x7f30300fd3a0 2022-01-31T19:37:52.448 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.437+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f302002b070 con 0x7f30300fd3a0 2022-01-31T19:37:52.448 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.437+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3030045bb0 con 0x7f30300fd3a0 2022-01-31T19:37:52.448 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.438+0000 7f301effd700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3018030ff0 0x7f30180334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:52.449 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.438+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3020042ac0 con 0x7f30300fd3a0 2022-01-31T19:37:52.449 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.438+0000 7f3035a67700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3018030ff0 0x7f30180334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:52.449 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.438+0000 7f3035a67700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3018030ff0 0x7f30180334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f302c0096a0 tx=0x7f302c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:52.449 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.442+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f302001b570 con 0x7f30300fd3a0 2022-01-31T19:37:52.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:52 smithi167 conmon[32206]: audit 2022-01-31T19:37:51.373527+0000 mon.smithi167 (mon.0) 2022-01-31T19:37:52.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:52 smithi167 conmon[32206]: 68 : audit [INF] from='client.? 172.21.15.167:0/2227764777' entity='client.admin' 2022-01-31T19:37:52.634 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:52.631+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f30300fa7e0 con 0x7f30300fd3a0 2022-01-31T19:37:53.380 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.378+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f3020042d70 con 0x7f30300fd3a0 2022-01-31T19:37:53.381 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.379+0000 7f301effd700 1 -- 172.21.15.167:0/1974783979 <== mon.0 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f302002cd20 con 0x7f30300fd3a0 2022-01-31T19:37:53.385 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3018030ff0 msgr2=0x7f30180334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:53.385 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3018030ff0 0x7f30180334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f302c0096a0 tx=0x7f302c006b40).stop 2022-01-31T19:37:53.385 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 msgr2=0x7f303010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:53.386 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f303010ab20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3020011e30 tx=0x7f3020006ee0).stop 2022-01-31T19:37:53.386 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 shutdown_connections 2022-01-31T19:37:53.386 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f3018030ff0 0x7f30180334b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:53.386 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 --2- 172.21.15.167:0/1974783979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30300fd3a0 0x7f303010ab20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:53.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 >> 172.21.15.167:0/1974783979 conn(0x7f30300f5b00 msgr2=0x7f30300f7570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:53.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 shutdown_connections 2022-01-31T19:37:53.387 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:53.382+0000 7f303726a700 1 -- 172.21.15.167:0/1974783979 wait complete. 2022-01-31T19:37:53.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:53 smithi167 conmon[32206]: audit 2022-01-31T19:37:52.633202+0000 mon.smithi167 (mon.0) 69 : audit [INF] from='client.? 172.21.15.167:0/1974783979' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-01-31T19:37:54.258 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.253+0000 7f5609a19700 1 Processor -- start 2022-01-31T19:37:54.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.253+0000 7f5609a19700 1 -- start start 2022-01-31T19:37:54.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.253+0000 7f5609a19700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 0x7f56040fd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:54.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.253+0000 7f5609a19700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56040fde20 con 0x7f56040fd510 2022-01-31T19:37:54.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.253+0000 7f5602ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 0x7f56040fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:54.259 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5602ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 0x7f56040fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60524/0 (socket says 172.21.15.167:60524) 2022-01-31T19:37:54.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5602ffd700 1 -- 172.21.15.167:0/2176069207 learned_addr learned my addr 172.21.15.167:0/2176069207 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:54.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5602ffd700 1 -- 172.21.15.167:0/2176069207 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56040fdf60 con 0x7f56040fd510 2022-01-31T19:37:54.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5602ffd700 1 --2- 172.21.15.167:0/2176069207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 0x7f56040fd930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f55f800b5f0 tx=0x7f55f8006ee0).ready entity=mon.0 client_cookie=87301d58b2ef1b4b server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:54.260 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5601ffb700 1 -- 172.21.15.167:0/2176069207 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f800d600 con 0x7f56040fd510 2022-01-31T19:37:54.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5601ffb700 1 -- 172.21.15.167:0/2176069207 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f55f800dbf0 con 0x7f56040fd510 2022-01-31T19:37:54.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.254+0000 7f5601ffb700 1 -- 172.21.15.167:0/2176069207 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f8011bf0 con 0x7f56040fd510 2022-01-31T19:37:54.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 -- 172.21.15.167:0/2176069207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 msgr2=0x7f56040fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:54.261 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 --2- 172.21.15.167:0/2176069207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 0x7f56040fd930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f55f800b5f0 tx=0x7f55f8006ee0).stop 2022-01-31T19:37:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 -- 172.21.15.167:0/2176069207 shutdown_connections 2022-01-31T19:37:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 --2- 172.21.15.167:0/2176069207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56040fd510 0x7f56040fd930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 -- 172.21.15.167:0/2176069207 >> 172.21.15.167:0/2176069207 conn(0x7f56040f7640 msgr2=0x7f56040f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 -- 172.21.15.167:0/2176069207 shutdown_connections 2022-01-31T19:37:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.255+0000 7f5609a19700 1 -- 172.21.15.167:0/2176069207 wait complete. 2022-01-31T19:37:54.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5609a19700 1 Processor -- start 2022-01-31T19:37:54.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5609a19700 1 -- start start 2022-01-31T19:37:54.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5609a19700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 0x7f560410b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:54.263 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5609a19700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560410b700 con 0x7f560410add0 2022-01-31T19:37:54.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5602ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 0x7f560410b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:54.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5602ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 0x7f560410b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60526/0 (socket says 172.21.15.167:60526) 2022-01-31T19:37:54.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.256+0000 7f5602ffd700 1 -- 172.21.15.167:0/1541618888 learned_addr learned my addr 172.21.15.167:0/1541618888 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:54.264 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.257+0000 7f5602ffd700 1 -- 172.21.15.167:0/1541618888 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f800b200 con 0x7f560410add0 2022-01-31T19:37:54.266 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.257+0000 7f5602ffd700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 0x7f560410b210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f55f8011fd0 tx=0x7f55f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:54.266 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.257+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f800d910 con 0x7f560410add0 2022-01-31T19:37:54.267 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.257+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560410b900 con 0x7f560410add0 2022-01-31T19:37:54.267 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.257+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f55f800da70 con 0x7f560410add0 2022-01-31T19:37:54.267 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.257+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f801a5e0 con 0x7f560410add0 2022-01-31T19:37:54.267 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.258+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560410c370 con 0x7f560410add0 2022-01-31T19:37:54.268 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.258+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f55f800f070 con 0x7f560410add0 2022-01-31T19:37:54.268 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.258+0000 7f55eb7fe700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 0x7f55f00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:54.268 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.258+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f55f8047560 con 0x7f560410add0 2022-01-31T19:37:54.269 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.258+0000 7f56027fc700 1 -- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 msgr2=0x7f55f00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:54.269 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.258+0000 7f56027fc700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 0x7f55f00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2022-01-31T19:37:54.269 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.259+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56040591e0 con 0x7f560410add0 2022-01-31T19:37:54.269 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.263+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f55f801f070 con 0x7f560410add0 2022-01-31T19:37:54.461 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.459+0000 7f56027fc700 1 -- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 msgr2=0x7f55f00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:54.461 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.459+0000 7f56027fc700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 0x7f55f00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2022-01-31T19:37:54.481 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.478+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5604104c90 con 0x7f560410add0 2022-01-31T19:37:54.481 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.479+0000 7f55eb7fe700 1 -- 172.21.15.167:0/1541618888 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f55f8016030 con 0x7f560410add0 2022-01-31T19:37:54.482 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:54.482 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "epoch": 9, 2022-01-31T19:37:54.482 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "available": true, 2022-01-31T19:37:54.482 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "active_name": "smithi167.aciqpk", 2022-01-31T19:37:54.482 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "num_standby": 0 2022-01-31T19:37:54.483 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:54.484 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.480+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 msgr2=0x7f55f00334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-01-31T19:37:54.484 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.480+0000 7f5609a19700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 0x7f55f00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:54.485 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.480+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 msgr2=0x7f560410b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:54.485 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.480+0000 7f5609a19700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 0x7f560410b210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f55f8011fd0 tx=0x7f55f8006ee0).stop 2022-01-31T19:37:54.485 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.481+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 shutdown_connections 2022-01-31T19:37:54.485 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.481+0000 7f5609a19700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f55f0030ff0 0x7f55f00334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:54.486 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.481+0000 7f5609a19700 1 --2- 172.21.15.167:0/1541618888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f560410add0 0x7f560410b210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:54.486 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.481+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 >> 172.21.15.167:0/1541618888 conn(0x7f56040f7640 msgr2=0x7f56040f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:54.486 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.481+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 shutdown_connections 2022-01-31T19:37:54.486 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:54.481+0000 7f5609a19700 1 -- 172.21.15.167:0/1541618888 wait complete. 2022-01-31T19:37:54.506 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:54 smithi167 conmon[32206]: audit 2022-01-31T19:37:53.380761+0000 mon.smithi167 (mon.0) 70 : audit [INF] from='client.? 172.21.15.167:0/1974783979' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-01-31T19:37:54.506 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:54 smithi167 conmon[32206]: cluster 2022-01-31T19:37:53.380857+0000 mon.smithi167 (mon.0) 71 : cluster [DBG] mgrmap e9: smithi167.aciqpk(active, since 19s) 2022-01-31T19:37:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for the mgr to restart... 2022-01-31T19:37:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:Waiting for mgr epoch 9... 2022-01-31T19:37:55.427 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.415+0000 7f5b81033700 1 Processor -- start 2022-01-31T19:37:55.427 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.415+0000 7f5b81033700 1 -- start start 2022-01-31T19:37:55.428 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.415+0000 7f5b81033700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c0fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:55.428 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.415+0000 7f5b81033700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0fc4a0 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.429 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.415+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c0fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:55.429 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c0fbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60534/0 (socket says 172.21.15.167:60534) 2022-01-31T19:37:55.430 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b7ad9d700 1 -- 172.21.15.167:0/3139648621 learned_addr learned my addr 172.21.15.167:0/3139648621 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:55.430 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b7ad9d700 1 -- 172.21.15.167:0/3139648621 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0fc5e0 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.431 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b7ad9d700 1 --2- 172.21.15.167:0/3139648621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c0fbfb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5b6400d900 tx=0x7f5b64006e70).ready entity=mon.0 client_cookie=866affa21bc02040 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:55.431 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b79d9b700 1 -- 172.21.15.167:0/3139648621 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b6400baf0 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.431 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b79d9b700 1 -- 172.21.15.167:0/3139648621 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5b6400bc50 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.432 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.416+0000 7f5b79d9b700 1 -- 172.21.15.167:0/3139648621 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b64015610 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.432 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 -- 172.21.15.167:0/3139648621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 msgr2=0x7f5b7c0fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:55.433 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 --2- 172.21.15.167:0/3139648621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c0fbfb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5b6400d900 tx=0x7f5b64006e70).stop 2022-01-31T19:37:55.433 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 -- 172.21.15.167:0/3139648621 shutdown_connections 2022-01-31T19:37:55.433 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 --2- 172.21.15.167:0/3139648621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c0fbfb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:55.434 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 -- 172.21.15.167:0/3139648621 >> 172.21.15.167:0/3139648621 conn(0x7f5b7c0f5ca0 msgr2=0x7f5b7c0f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:55.434 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 -- 172.21.15.167:0/3139648621 shutdown_connections 2022-01-31T19:37:55.435 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.417+0000 7f5b81033700 1 -- 172.21.15.167:0/3139648621 wait complete. 2022-01-31T19:37:55.436 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.418+0000 7f5b81033700 1 Processor -- start 2022-01-31T19:37:55.436 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.418+0000 7f5b81033700 1 -- start start 2022-01-31T19:37:55.436 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.418+0000 7f5b81033700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:55.437 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.418+0000 7f5b81033700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c108f00 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.437 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60536/0 (socket says 172.21.15.167:60536) 2022-01-31T19:37:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b7ad9d700 1 -- 172.21.15.167:0/1204007137 learned_addr learned my addr 172.21.15.167:0/1204007137 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:37:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b7ad9d700 1 -- 172.21.15.167:0/1204007137 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6400d510 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b7ad9d700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c108a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5b64006b70 tx=0x7f5b64006e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:55.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b640153e0 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.439 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5b64015540 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c109100 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b6401ec10 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.440 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.419+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c109590 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.420+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f5b64027410 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.420+0000 7f5b737fe700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:55.441 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.420+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f5b64045e50 con 0x7f5b7c0fbb90 2022-01-31T19:37:55.442 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.420+0000 7f5b7a59c700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 msgr2=0x7f5b5c033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:55.442 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.420+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2022-01-31T19:37:55.443 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.420+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 --> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b7c046a20 con 0x7f5b5c031040 2022-01-31T19:37:55.623 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.621+0000 7f5b7a59c700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 msgr2=0x7f5b5c033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:55.623 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:55.621+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2022-01-31T19:37:55.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:55 smithi167 conmon[32206]: audit 2022-01-31T19:37:54.480437+0000 mon.smithi167 (mon.0) 72 : 2022-01-31T19:37:55.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:55 smithi167 conmon[32206]: audit [DBG] from='client.? 172.21.15.167:0/1541618888' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-01-31T19:37:56.024 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:56.022+0000 7f5b7a59c700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 msgr2=0x7f5b5c033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:56.025 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:56.022+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2022-01-31T19:37:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:56.823+0000 7f5b7a59c700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 msgr2=0x7f5b5c033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:56.823+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2022-01-31T19:37:58.426 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:58.424+0000 7f5b7a59c700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 msgr2=0x7f5b5c033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6800/2257398586 2022-01-31T19:37:58.427 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:58.424+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2022-01-31T19:37:58.684 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:58.682+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mgrmap(e 10) v1 ==== 36958+0+0 (secure 0 0 0) 0x7f5b6401e420 con 0x7f5b7c0fbb90 2022-01-31T19:37:58.685 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:58.682+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 msgr2=0x7f5b5c033500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-01-31T19:37:58.685 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:58.682+0000 7f5b737fe700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:59.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:58 smithi167 conmon[32206]: cluster 2022-01-31T19:37:58.631396+0000 mon.smithi167 (mon.0) 73 : 2022-01-31T19:37:59.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:58 smithi167 conmon[32206]: cluster [INF] Active manager daemon smithi167.aciqpk restarted 2022-01-31T19:37:59.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:58 smithi167 conmon[32206]: cluster 2022-01-31T19:37:58.631804+0000 mon.smithi167 (mon.0) 74 : cluster [INF] Activating manager daemon smithi167.aciqpk 2022-01-31T19:37:59.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:58 smithi167 conmon[32206]: cluster 2022-01-31T19:37:58.633696+0000 mon.smithi167 (mon.0) 75 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2022-01-31T19:37:59.697 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.694+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f5b64026c70 con 0x7f5b7c0fbb90 2022-01-31T19:37:59.698 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.694+0000 7f5b737fe700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5b5c0349e0 0x7f5b5c036dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:37:59.698 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.695+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b7c046a20 con 0x7f5b5c0349e0 2022-01-31T19:37:59.698 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.695+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5b5c0349e0 0x7f5b5c036dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:37:59.699 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.695+0000 7f5b7a59c700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5b5c0349e0 0x7f5b5c036dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5b6c006fd0 tx=0x7f5b6c006db0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:37:59.699 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.696+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f5b7c046a20 con 0x7f5b5c0349e0 2022-01-31T19:37:59.704 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.702+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5b7c0591e0 con 0x7f5b5c0349e0 2022-01-31T19:37:59.704 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.702+0000 7f5b737fe700 1 -- 172.21.15.167:0/1204007137 <== mgr.14156 v2:172.21.15.167:6800/3192751116 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5b7c0591e0 con 0x7f5b5c0349e0 2022-01-31T19:37:59.705 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: { 2022-01-31T19:37:59.706 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2022-01-31T19:37:59.707 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: "initialized": true 2022-01-31T19:37:59.707 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: } 2022-01-31T19:37:59.707 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5b5c0349e0 msgr2=0x7f5b5c036dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:59.707 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5b5c0349e0 0x7f5b5c036dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5b6c006fd0 tx=0x7f5b6c006db0).stop 2022-01-31T19:37:59.708 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 msgr2=0x7f5b7c108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:37:59.708 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c108a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5b64006b70 tx=0x7f5b64006e70).stop 2022-01-31T19:37:59.709 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 shutdown_connections 2022-01-31T19:37:59.709 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5b5c0349e0 0x7f5b5c036dd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:59.709 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b7c0fbb90 0x7f5b7c108a10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:37:59.709 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 --2- 172.21.15.167:0/1204007137 >> [v2:172.21.15.167:6800/2257398586,v1:172.21.15.167:6801/2257398586] conn(0x7f5b5c031040 0x7f5b5c033500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:37:59.710 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 >> 172.21.15.167:0/1204007137 conn(0x7f5b7c0f5ca0 msgr2=0x7f5b7c0fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:37:59.710 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 shutdown_connections 2022-01-31T19:37:59.710 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:37:59.703+0000 7f5b81033700 1 -- 172.21.15.167:0/1204007137 wait complete. 2022-01-31T19:38:00.047 INFO:teuthology.orchestra.run.smithi167.stderr:mgr epoch 9 is available 2022-01-31T19:38:00.122 INFO:teuthology.orchestra.run.smithi167.stderr:Generating a dashboard self-signed certificate... 2022-01-31T19:38:00.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: cluster 2022-01-31T19:37:58.684745+0000 mon.smithi167 (mon.0) 76 : cluster [DBG] mgrmap e10: smithi167.aciqpk(active, starting, since 0.0530268s) 2022-01-31T19:38:00.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:37:58.687327+0000 mon.smithi167 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:38:00.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:37:58.687543+0000 mon.smithi167 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:38:00.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:37:58.687747+0000 mon.smithi167 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:38:00.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:37:58.687907+0000 mon.smithi167 (mon.0) 80 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:38:00.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:38:00.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: 37:58.688055+0000 mon.smithi167 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:38:00.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: cluster 2022-01-31T19:37:58 2022-01-31T19:38:00.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: .701925+0000 mon.smithi167 (mon.0) 82 : 2022-01-31T19:38:00.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: cluster [INF] Manager daemon smithi167.aciqpk is now available 2022-01-31T19:38:00.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: 2022-01-31T19:37 2022-01-31T19:38:00.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: :58.800145+0000 mon.smithi167 (mon 2022-01-31T19:38:00.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: .0) 83 : audit [DBG] 2022-01-31T19:38:00.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:38:00.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: 2022-01-31T19:37:58.802365 2022-01-31T19:38:00.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 84 2022-01-31T19:38:00.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:38:00.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: 2022-01-31T19:37 2022-01-31T19:38:00.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: :58.820898+0000 mon.smithi167 (mon.0 2022-01-31T19:38:00.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: ) 85 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:38:00.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: 2022-01-31T19:37:58.822854+0000 mon.smithi167 2022-01-31T19:38:00.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: (mon.0) 86 : audit [INF] 2022-01-31T19:38:00.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:38:00.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: 2022-01-31T19:37:59.613439 2022-01-31T19:38:00.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 87 : 2022-01-31T19:38:00.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:37:59 smithi167 conmon[32206]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:00.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.653+0000 7f8ac359e700 1 Processor -- start 2022-01-31T19:38:00.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.653+0000 7f8ac359e700 1 -- start start 2022-01-31T19:38:00.661 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.654+0000 7f8ac359e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac4100d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:00.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.654+0000 7f8ac359e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40fc740 con 0x7f8ac406fc80 2022-01-31T19:38:00.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.654+0000 7f8ac259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac4100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:00.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.654+0000 7f8ac259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac4100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60574/0 (socket says 172.21.15.167:60574) 2022-01-31T19:38:00.662 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.654+0000 7f8ac259c700 1 -- 172.21.15.167:0/4126573555 learned_addr learned my addr 172.21.15.167:0/4126573555 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:00.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.655+0000 7f8ac259c700 1 -- 172.21.15.167:0/4126573555 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4101240 con 0x7f8ac406fc80 2022-01-31T19:38:00.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.655+0000 7f8ac259c700 1 --2- 172.21.15.167:0/4126573555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac4100d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8ab400b5f0 tx=0x7f8ab4006ee0).ready entity=mon.0 client_cookie=28616b42860cb571 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:00.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.655+0000 7f8ac159a700 1 -- 172.21.15.167:0/4126573555 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab400d600 con 0x7f8ac406fc80 2022-01-31T19:38:00.663 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.655+0000 7f8ac159a700 1 -- 172.21.15.167:0/4126573555 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8ab400dbf0 con 0x7f8ac406fc80 2022-01-31T19:38:00.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.655+0000 7f8ac159a700 1 -- 172.21.15.167:0/4126573555 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab4011bf0 con 0x7f8ac406fc80 2022-01-31T19:38:00.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 -- 172.21.15.167:0/4126573555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 msgr2=0x7f8ac4100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:00.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 --2- 172.21.15.167:0/4126573555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac4100d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8ab400b5f0 tx=0x7f8ab4006ee0).stop 2022-01-31T19:38:00.664 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 -- 172.21.15.167:0/4126573555 shutdown_connections 2022-01-31T19:38:00.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 --2- 172.21.15.167:0/4126573555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac4100d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:00.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 -- 172.21.15.167:0/4126573555 >> 172.21.15.167:0/4126573555 conn(0x7f8ac40f6390 msgr2=0x7f8ac40f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:00.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 -- 172.21.15.167:0/4126573555 shutdown_connections 2022-01-31T19:38:00.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 -- 172.21.15.167:0/4126573555 wait complete. 2022-01-31T19:38:00.665 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 Processor -- start 2022-01-31T19:38:00.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.656+0000 7f8ac359e700 1 -- start start 2022-01-31T19:38:00.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac359e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:00.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac359e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac410d230 con 0x7f8ac406fc80 2022-01-31T19:38:00.666 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac410cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:00.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac410cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60576/0 (socket says 172.21.15.167:60576) 2022-01-31T19:38:00.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac259c700 1 -- 172.21.15.167:0/3360505635 learned_addr learned my addr 172.21.15.167:0/3360505635 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:00.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac259c700 1 -- 172.21.15.167:0/3360505635 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab400b200 con 0x7f8ac406fc80 2022-01-31T19:38:00.667 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac259c700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac410cd40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8ab400bf20 tx=0x7f8ab4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:00.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab401b410 con 0x7f8ac406fc80 2022-01-31T19:38:00.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8ab401ba00 con 0x7f8ac406fc80 2022-01-31T19:38:00.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab401a6c0 con 0x7f8ac406fc80 2022-01-31T19:38:00.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac410d430 con 0x7f8ac406fc80 2022-01-31T19:38:00.668 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.657+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac410e040 con 0x7f8ac406fc80 2022-01-31T19:38:00.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.658+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f8ab402b070 con 0x7f8ac406fc80 2022-01-31T19:38:00.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.658+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac4046a20 con 0x7f8ac406fc80 2022-01-31T19:38:00.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.658+0000 7f8ab2ffd700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8aa8030f90 0x7f8aa8033450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:00.669 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.658+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8ab4042ad0 con 0x7f8ac406fc80 2022-01-31T19:38:00.670 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.659+0000 7f8ac1d9b700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8aa8030f90 0x7f8aa8033450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:00.670 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.659+0000 7f8ac1d9b700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8aa8030f90 0x7f8aa8033450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8ab80096a0 tx=0x7f8ab8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:00.671 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.663+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8ab401b570 con 0x7f8ac406fc80 2022-01-31T19:38:00.828 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.826+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ac4102010 con 0x7f8aa8030f90 2022-01-31T19:38:00.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:00 smithi167 conmon[32206]: audit 2022-01-31T19:37:59.697621+0000 mgr.smithi167.aciqpk (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-01-31T19:38:00.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:00 smithi167 conmon[32206]: cluster 2022-01-31 2022-01-31T19:38:00.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:00 smithi167 conmon[32206]: T19:37:59.698559+0000 mon.smithi167 (mon.0) 88 : cluster [DBG] mgrmap e11: smithi167.aciqpk(active, since 1.06684s) 2022-01-31T19:38:00.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:00 smithi167 conmon[32206]: audit 2022-01-31T19:37:59.704123+0000 mgr.smithi167.aciqpk (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-01-31T19:38:00.988 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.986+0000 7f8ab2ffd700 1 -- 172.21.15.167:0/3360505635 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8ac4102010 con 0x7f8aa8030f90 2022-01-31T19:38:00.989 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: Self-signed certificate created 2022-01-31T19:38:00.991 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8aa8030f90 msgr2=0x7f8aa8033450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:00.991 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8aa8030f90 0x7f8aa8033450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8ab80096a0 tx=0x7f8ab8006b40).stop 2022-01-31T19:38:00.992 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 msgr2=0x7f8ac410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:00.992 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac410cd40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8ab400bf20 tx=0x7f8ab4006ee0).stop 2022-01-31T19:38:00.992 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 shutdown_connections 2022-01-31T19:38:00.992 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8aa8030f90 0x7f8aa8033450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:00.993 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 --2- 172.21.15.167:0/3360505635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ac406fc80 0x7f8ac410cd40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:00.993 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 >> 172.21.15.167:0/3360505635 conn(0x7f8ac40f6390 msgr2=0x7f8ac40f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:00.993 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 shutdown_connections 2022-01-31T19:38:00.993 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:00.988+0000 7f8ac359e700 1 -- 172.21.15.167:0/3360505635 wait complete. 2022-01-31T19:38:01.286 INFO:teuthology.orchestra.run.smithi167.stderr:Creating initial admin user... 2022-01-31T19:38:01.724 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.720+0000 7f842a422700 1 Processor -- start 2022-01-31T19:38:01.725 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.720+0000 7f842a422700 1 -- start start 2022-01-31T19:38:01.725 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.720+0000 7f842a422700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f8424100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:01.725 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.720+0000 7f842a422700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84240fc8e0 con 0x7f842406fc80 2022-01-31T19:38:01.725 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8423fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f8424100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:01.726 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8423fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f8424100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60580/0 (socket says 172.21.15.167:60580) 2022-01-31T19:38:01.726 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8423fff700 1 -- 172.21.15.167:0/1442302250 learned_addr learned my addr 172.21.15.167:0/1442302250 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:01.726 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8423fff700 1 -- 172.21.15.167:0/1442302250 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84241013e0 con 0x7f842406fc80 2022-01-31T19:38:01.726 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8423fff700 1 --2- 172.21.15.167:0/1442302250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f8424100ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f841800b5f0 tx=0x7f8418006ee0).ready entity=mon.0 client_cookie=edfd82d4853e552e server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:01.727 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8422ffd700 1 -- 172.21.15.167:0/1442302250 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f841800d600 con 0x7f842406fc80 2022-01-31T19:38:01.727 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8422ffd700 1 -- 172.21.15.167:0/1442302250 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f841800dbf0 con 0x7f842406fc80 2022-01-31T19:38:01.727 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.721+0000 7f8422ffd700 1 -- 172.21.15.167:0/1442302250 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8418011bf0 con 0x7f842406fc80 2022-01-31T19:38:01.727 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 -- 172.21.15.167:0/1442302250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 msgr2=0x7f8424100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:01.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 --2- 172.21.15.167:0/1442302250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f8424100ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f841800b5f0 tx=0x7f8418006ee0).stop 2022-01-31T19:38:01.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 -- 172.21.15.167:0/1442302250 shutdown_connections 2022-01-31T19:38:01.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 --2- 172.21.15.167:0/1442302250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f8424100ef0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:01.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 -- 172.21.15.167:0/1442302250 >> 172.21.15.167:0/1442302250 conn(0x7f84240f6530 msgr2=0x7f84240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:01.728 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 -- 172.21.15.167:0/1442302250 shutdown_connections 2022-01-31T19:38:01.729 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 -- 172.21.15.167:0/1442302250 wait complete. 2022-01-31T19:38:01.729 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.722+0000 7f842a422700 1 Processor -- start 2022-01-31T19:38:01.729 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f842a422700 1 -- start start 2022-01-31T19:38:01.729 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f842a422700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f842410ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:01.729 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f842a422700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842410d380 con 0x7f842406fc80 2022-01-31T19:38:01.732 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f8423fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f842410ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:01.733 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f8423fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f842410ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60582/0 (socket says 172.21.15.167:60582) 2022-01-31T19:38:01.733 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f8423fff700 1 -- 172.21.15.167:0/1723700776 learned_addr learned my addr 172.21.15.167:0/1723700776 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:01.733 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f8423fff700 1 -- 172.21.15.167:0/1723700776 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841800b200 con 0x7f842406fc80 2022-01-31T19:38:01.733 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.723+0000 7f8423fff700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f842410ce90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f841800b1b0 tx=0x7f8418006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:01.734 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.724+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f841801b410 con 0x7f842406fc80 2022-01-31T19:38:01.734 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.724+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f841801ba00 con 0x7f842406fc80 2022-01-31T19:38:01.734 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.724+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842410d580 con 0x7f842406fc80 2022-01-31T19:38:01.734 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.724+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f841801a7f0 con 0x7f842406fc80 2022-01-31T19:38:01.735 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.724+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842410da10 con 0x7f842406fc80 2022-01-31T19:38:01.735 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.724+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f841802b070 con 0x7f842406fc80 2022-01-31T19:38:01.735 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.725+0000 7f8420ff9700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8410031040 0x7f8410033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:01.735 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.725+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8418042c30 con 0x7f842406fc80 2022-01-31T19:38:01.735 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.725+0000 7f84237fe700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8410031040 0x7f8410033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:01.736 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.725+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8424046a20 con 0x7f842406fc80 2022-01-31T19:38:01.736 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.725+0000 7f84237fe700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8410031040 0x7f8410033500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f84140096a0 tx=0x7f8414006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:01.736 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.729+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f841801f070 con 0x7f842406fc80 2022-01-31T19:38:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:01.895+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8424059000 con 0x7f8410031040 2022-01-31T19:38:02.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.112+0000 7f8420ff9700 1 -- 172.21.15.167:0/1723700776 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f8424059000 con 0x7f8410031040 2022-01-31T19:38:02.114 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$HSTuHsLWwkSFGh76/fIG6uoXyP3VuWN8j3lf8AEiIGhX2iS3One0K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1643657882, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2022-01-31T19:38:02.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8410031040 msgr2=0x7f8410033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:02.116 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8410031040 0x7f8410033500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f84140096a0 tx=0x7f8414006b40).stop 2022-01-31T19:38:02.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 msgr2=0x7f842410ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:02.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f842410ce90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f841800b1b0 tx=0x7f8418006ee0).stop 2022-01-31T19:38:02.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 shutdown_connections 2022-01-31T19:38:02.117 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8410031040 0x7f8410033500 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:02.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 --2- 172.21.15.167:0/1723700776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f842406fc80 0x7f842410ce90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:02.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 >> 172.21.15.167:0/1723700776 conn(0x7f84240f6530 msgr2=0x7f84240f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:02.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 shutdown_connections 2022-01-31T19:38:02.118 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:02.114+0000 7f842a422700 1 -- 172.21.15.167:0/1723700776 wait complete. 2022-01-31T19:38:02.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:01 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.828344+0000 mgr.smithi167.aciqpk (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:38:02.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:01 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.982891+0000 mon.smithi167 (mon.0) 89 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:02.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:01 smithi167 conmon[32206]: audit 2022-01-31T19:38:00.987925+0000 mon.smithi167 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:02.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:01 smithi167 conmon[32206]: cluster 2022-01-31T19:38:01.594443+0000 mon.smithi167 (mon.0) 91 : cluster [DBG] mgrmap e12: smithi167.aciqpk(active, since 2s) 2022-01-31T19:38:02.523 INFO:teuthology.orchestra.run.smithi167.stderr:Fetching dashboard port number... 2022-01-31T19:38:03.006 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59c57d4700 1 Processor -- start 2022-01-31T19:38:03.006 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59c57d4700 1 -- start start 2022-01-31T19:38:03.006 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59c57d4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c00fe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:03.007 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59c57d4700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c00fedf0 con 0x7f59c006fca0 2022-01-31T19:38:03.007 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59bffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c00fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:03.007 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59bffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c00fe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60586/0 (socket says 172.21.15.167:60586) 2022-01-31T19:38:03.007 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59bffff700 1 -- 172.21.15.167:0/3358559682 learned_addr learned my addr 172.21.15.167:0/3358559682 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:03.008 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59bffff700 1 -- 172.21.15.167:0/3358559682 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c00fef30 con 0x7f59c006fca0 2022-01-31T19:38:03.008 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.001+0000 7f59bffff700 1 --2- 172.21.15.167:0/3358559682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c00fe900 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f59a800b5f0 tx=0x7f59a8006ee0).ready entity=mon.0 client_cookie=1eb721f0174e422e server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:03.008 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59beffd700 1 -- 172.21.15.167:0/3358559682 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a800d600 con 0x7f59c006fca0 2022-01-31T19:38:03.008 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59beffd700 1 -- 172.21.15.167:0/3358559682 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f59a800dbf0 con 0x7f59c006fca0 2022-01-31T19:38:03.008 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59beffd700 1 -- 172.21.15.167:0/3358559682 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a8011bf0 con 0x7f59c006fca0 2022-01-31T19:38:03.009 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 -- 172.21.15.167:0/3358559682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 msgr2=0x7f59c00fe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:03.009 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 --2- 172.21.15.167:0/3358559682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c00fe900 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f59a800b5f0 tx=0x7f59a8006ee0).stop 2022-01-31T19:38:03.009 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 -- 172.21.15.167:0/3358559682 shutdown_connections 2022-01-31T19:38:03.009 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 --2- 172.21.15.167:0/3358559682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c00fe900 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:03.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 -- 172.21.15.167:0/3358559682 >> 172.21.15.167:0/3358559682 conn(0x7f59c00f6340 msgr2=0x7f59c00f8780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:03.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 -- 172.21.15.167:0/3358559682 shutdown_connections 2022-01-31T19:38:03.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.002+0000 7f59c57d4700 1 -- 172.21.15.167:0/3358559682 wait complete. 2022-01-31T19:38:03.010 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.003+0000 7f59c57d4700 1 Processor -- start 2022-01-31T19:38:03.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.003+0000 7f59c57d4700 1 -- start start 2022-01-31T19:38:03.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59c57d4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c010aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:03.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59c57d4700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c010afb0 con 0x7f59c006fca0 2022-01-31T19:38:03.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c010aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:03.011 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c010aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60588/0 (socket says 172.21.15.167:60588) 2022-01-31T19:38:03.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bffff700 1 -- 172.21.15.167:0/2719638022 learned_addr learned my addr 172.21.15.167:0/2719638022 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:03.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bffff700 1 -- 172.21.15.167:0/2719638022 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a800b200 con 0x7f59c006fca0 2022-01-31T19:38:03.012 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bffff700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c010aac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f59a800b1b0 tx=0x7f59a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:03.014 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a801b410 con 0x7f59c006fca0 2022-01-31T19:38:03.014 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f59a801ba00 con 0x7f59c006fca0 2022-01-31T19:38:03.014 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a801a7f0 con 0x7f59c006fca0 2022-01-31T19:38:03.014 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c010b1b0 con 0x7f59c006fca0 2022-01-31T19:38:03.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.004+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c010b640 con 0x7f59c006fca0 2022-01-31T19:38:03.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.005+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f59a802b070 con 0x7f59c006fca0 2022-01-31T19:38:03.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.005+0000 7f59bcff9700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f59a0030ff0 0x7f59a00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:03.015 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.005+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f59a8042c70 con 0x7f59c006fca0 2022-01-31T19:38:03.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.005+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c00591e0 con 0x7f59c006fca0 2022-01-31T19:38:03.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.006+0000 7f59bf7fe700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f59a0030ff0 0x7f59a00334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:03.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.006+0000 7f59bf7fe700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f59a0030ff0 0x7f59a00334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f59b00096a0 tx=0x7f59b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:03.016 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.010+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f59a801f070 con 0x7f59c006fca0 2022-01-31T19:38:03.167 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.164+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f59c0045bb0 con 0x7f59c006fca0 2022-01-31T19:38:03.167 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.165+0000 7f59bcff9700 1 -- 172.21.15.167:0/2719638022 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f59a802b4e0 con 0x7f59c006fca0 2022-01-31T19:38:03.167 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 8443 2022-01-31T19:38:03.169 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f59a0030ff0 msgr2=0x7f59a00334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:03.169 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f59a0030ff0 0x7f59a00334b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f59b00096a0 tx=0x7f59b0006b40).stop 2022-01-31T19:38:03.170 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 msgr2=0x7f59c010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:03.170 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c010aac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f59a800b1b0 tx=0x7f59a8006ee0).stop 2022-01-31T19:38:03.170 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 shutdown_connections 2022-01-31T19:38:03.170 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f59a0030ff0 0x7f59a00334b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:03.171 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 --2- 172.21.15.167:0/2719638022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59c006fca0 0x7f59c010aac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:03.171 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.166+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 >> 172.21.15.167:0/2719638022 conn(0x7f59c00f6340 msgr2=0x7f59c00f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:03.171 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.167+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 shutdown_connections 2022-01-31T19:38:03.171 INFO:teuthology.orchestra.run.smithi167.stderr:/usr/bin/ceph: 2022-01-31T19:38:03.167+0000 7f59c57d4700 1 -- 172.21.15.167:0/2719638022 wait complete. 2022-01-31T19:38:03.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:03 smithi167 conmon[32206]: audit 2022-01-31T19:38:01.897914+0000 mgr.smithi167.aciqpk (mgr.14156) 4 : audit 2022-01-31T19:38:03.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:03 smithi167 conmon[32206]: [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:38:03.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:03 smithi167 conmon[32206]: audit 2022-01-31T19:38:02.113530+0000 mon.smithi167 (mon.0) 92 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:03.567 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: disabled 2022-01-31T19:38:03.574 INFO:teuthology.orchestra.run.smithi167.stderr:systemctl: inactive 2022-01-31T19:38:03.575 INFO:teuthology.orchestra.run.smithi167.stderr:firewalld.service is not enabled 2022-01-31T19:38:03.575 INFO:teuthology.orchestra.run.smithi167.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2022-01-31T19:38:03.577 INFO:teuthology.orchestra.run.smithi167.stderr:Ceph Dashboard is now available at: 2022-01-31T19:38:03.577 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.578 INFO:teuthology.orchestra.run.smithi167.stderr: URL: https://smithi167.front.sepia.ceph.com:8443/ 2022-01-31T19:38:03.578 INFO:teuthology.orchestra.run.smithi167.stderr: User: admin 2022-01-31T19:38:03.578 INFO:teuthology.orchestra.run.smithi167.stderr: Password: nl7xoxbo9t 2022-01-31T19:38:03.578 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.579 INFO:teuthology.orchestra.run.smithi167.stderr:You can access the Ceph CLI with: 2022-01-31T19:38:03.580 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.580 INFO:teuthology.orchestra.run.smithi167.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-01-31T19:38:03.580 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.580 INFO:teuthology.orchestra.run.smithi167.stderr:Please consider enabling telemetry to help improve Ceph: 2022-01-31T19:38:03.580 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.581 INFO:teuthology.orchestra.run.smithi167.stderr: ceph telemetry on 2022-01-31T19:38:03.581 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.581 INFO:teuthology.orchestra.run.smithi167.stderr:For more information see: 2022-01-31T19:38:03.581 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.581 INFO:teuthology.orchestra.run.smithi167.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2022-01-31T19:38:03.582 INFO:teuthology.orchestra.run.smithi167.stderr: 2022-01-31T19:38:03.582 INFO:teuthology.orchestra.run.smithi167.stderr:Bootstrap complete. 2022-01-31T19:38:03.612 INFO:tasks.cephadm:Fetching config... 2022-01-31T19:38:03.613 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:38:03.613 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-01-31T19:38:03.630 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-01-31T19:38:03.630 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:38:03.631 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-01-31T19:38:03.687 INFO:tasks.cephadm:Fetching mon keyring... 2022-01-31T19:38:03.687 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:38:03.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/keyring of=/dev/stdout 2022-01-31T19:38:03.753 INFO:tasks.cephadm:Fetching pub ssh key... 2022-01-31T19:38:03.753 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:38:03.754 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-01-31T19:38:03.810 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-01-31T19:38:03.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDirFThKHfhqbBm9eATQx4qg0mmHakKo1NcRQs4EhKamQNIkFOKTI0etblzTQUX2AVZbEiGv6LEr19ZlqG7DNVPjCS7Lh4TDiMz43coDCyVLBXe5A1JY2hiqDtcJMy36I4kEAYn5P2PzutUrqIpTscGmSq1yoIi1c3EOTjNqWG43qGOnQDF5D19K9nEsfdomkw0hpdc7mQZaQrR1ru8RMAdsMGqQhTwI5kf4/HNIzn0c7xHb6sH9Qz3I9zpkt117rj658tcDgn12PDXtz0iYiza5MYXZeUhXu1KIlMik3c2Em1DMAfmFvdRmx7Oa2ujzFwfuR4QM1cbB19vnU0nu6gK9FG8S9TYSCfOQjd3HD7/1muM7hlLc+9ieAznGCUHRk5QiB34BuA5X+OHye0GpxtxbyW5v/R9v66nbXVYQQHaKsPJpmZFdh50AFUAEXJDSPq1i+QHpb3uvtoDEtnDnM+V0tZRAdEzY1N7qX/3TdKHXB6woa/joQb+6cRcnLEvlT0= ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-01-31T19:38:03.891 INFO:teuthology.orchestra.run.smithi167.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDirFThKHfhqbBm9eATQx4qg0mmHakKo1NcRQs4EhKamQNIkFOKTI0etblzTQUX2AVZbEiGv6LEr19ZlqG7DNVPjCS7Lh4TDiMz43coDCyVLBXe5A1JY2hiqDtcJMy36I4kEAYn5P2PzutUrqIpTscGmSq1yoIi1c3EOTjNqWG43qGOnQDF5D19K9nEsfdomkw0hpdc7mQZaQrR1ru8RMAdsMGqQhTwI5kf4/HNIzn0c7xHb6sH9Qz3I9zpkt117rj658tcDgn12PDXtz0iYiza5MYXZeUhXu1KIlMik3c2Em1DMAfmFvdRmx7Oa2ujzFwfuR4QM1cbB19vnU0nu6gK9FG8S9TYSCfOQjd3HD7/1muM7hlLc+9ieAznGCUHRk5QiB34BuA5X+OHye0GpxtxbyW5v/R9v66nbXVYQQHaKsPJpmZFdh50AFUAEXJDSPq1i+QHpb3uvtoDEtnDnM+V0tZRAdEzY1N7qX/3TdKHXB6woa/joQb+6cRcnLEvlT0= ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:38:03.904 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDirFThKHfhqbBm9eATQx4qg0mmHakKo1NcRQs4EhKamQNIkFOKTI0etblzTQUX2AVZbEiGv6LEr19ZlqG7DNVPjCS7Lh4TDiMz43coDCyVLBXe5A1JY2hiqDtcJMy36I4kEAYn5P2PzutUrqIpTscGmSq1yoIi1c3EOTjNqWG43qGOnQDF5D19K9nEsfdomkw0hpdc7mQZaQrR1ru8RMAdsMGqQhTwI5kf4/HNIzn0c7xHb6sH9Qz3I9zpkt117rj658tcDgn12PDXtz0iYiza5MYXZeUhXu1KIlMik3c2Em1DMAfmFvdRmx7Oa2ujzFwfuR4QM1cbB19vnU0nu6gK9FG8S9TYSCfOQjd3HD7/1muM7hlLc+9ieAznGCUHRk5QiB34BuA5X+OHye0GpxtxbyW5v/R9v66nbXVYQQHaKsPJpmZFdh50AFUAEXJDSPq1i+QHpb3uvtoDEtnDnM+V0tZRAdEzY1N7qX/3TdKHXB6woa/joQb+6cRcnLEvlT0= ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-01-31T19:38:03.950 INFO:teuthology.orchestra.run.smithi171.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDirFThKHfhqbBm9eATQx4qg0mmHakKo1NcRQs4EhKamQNIkFOKTI0etblzTQUX2AVZbEiGv6LEr19ZlqG7DNVPjCS7Lh4TDiMz43coDCyVLBXe5A1JY2hiqDtcJMy36I4kEAYn5P2PzutUrqIpTscGmSq1yoIi1c3EOTjNqWG43qGOnQDF5D19K9nEsfdomkw0hpdc7mQZaQrR1ru8RMAdsMGqQhTwI5kf4/HNIzn0c7xHb6sH9Qz3I9zpkt117rj658tcDgn12PDXtz0iYiza5MYXZeUhXu1KIlMik3c2Em1DMAfmFvdRmx7Oa2ujzFwfuR4QM1cbB19vnU0nu6gK9FG8S9TYSCfOQjd3HD7/1muM7hlLc+9ieAznGCUHRk5QiB34BuA5X+OHye0GpxtxbyW5v/R9v66nbXVYQQHaKsPJpmZFdh50AFUAEXJDSPq1i+QHpb3uvtoDEtnDnM+V0tZRAdEzY1N7qX/3TdKHXB6woa/joQb+6cRcnLEvlT0= ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:38:03.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-01-31T19:38:04.412 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:04 smithi167 conmon[32206]: audit 2022-01-31T19:38:03.166483+0000 mon.smithi167 (mon.0) 93 : audit [DBG] from='client.? 172.21.15.167:0/2719638022' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-01-31T19:38:05.953 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.950+0000 7fd3a3156700 1 -- 172.21.15.167:0/703569749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 msgr2=0x7fd39c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:05.954 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.950+0000 7fd3a3156700 1 --2- 172.21.15.167:0/703569749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c0f58a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd390009fb0 tx=0x7fd390009b30).stop 2022-01-31T19:38:05.954 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.951+0000 7fd3a3156700 1 -- 172.21.15.167:0/703569749 shutdown_connections 2022-01-31T19:38:05.954 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.951+0000 7fd3a3156700 1 --2- 172.21.15.167:0/703569749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c0f58a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:05.955 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.951+0000 7fd3a3156700 1 -- 172.21.15.167:0/703569749 >> 172.21.15.167:0/703569749 conn(0x7fd39c0f0be0 msgr2=0x7fd39c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:05.955 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.951+0000 7fd3a3156700 1 -- 172.21.15.167:0/703569749 shutdown_connections 2022-01-31T19:38:05.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.951+0000 7fd3a3156700 1 -- 172.21.15.167:0/703569749 wait complete. 2022-01-31T19:38:05.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.952+0000 7fd3a3156700 1 Processor -- start 2022-01-31T19:38:05.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.952+0000 7fd3a3156700 1 -- start start 2022-01-31T19:38:05.962 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.952+0000 7fd3a3156700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:05.962 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.952+0000 7fd3a3156700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c060320 con 0x7fd39c0f5480 2022-01-31T19:38:05.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.952+0000 7fd3a0ef2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:05.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3a0ef2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60598/0 (socket says 172.21.15.167:60598) 2022-01-31T19:38:05.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3a0ef2700 1 -- 172.21.15.167:0/1268274652 learned_addr learned my addr 172.21.15.167:0/1268274652 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:05.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3a0ef2700 1 -- 172.21.15.167:0/1268274652 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390005040 con 0x7fd39c0f5480 2022-01-31T19:38:05.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3a0ef2700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c1094f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd390000c00 tx=0x7fd39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:05.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3900095f0 con 0x7fd39c0f5480 2022-01-31T19:38:05.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd390007e50 con 0x7fd39c0f5480 2022-01-31T19:38:05.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c060520 con 0x7fd39c0f5480 2022-01-31T19:38:05.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390014e70 con 0x7fd39c0f5480 2022-01-31T19:38:05.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.953+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c109ba0 con 0x7fd39c0f5480 2022-01-31T19:38:05.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.954+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fd390025070 con 0x7fd39c0f5480 2022-01-31T19:38:05.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.954+0000 7fd3997fa700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd38c031090 0x7fd38c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:05.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.954+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd390042c90 con 0x7fd39c0f5480 2022-01-31T19:38:05.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.954+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c045bb0 con 0x7fd39c0f5480 2022-01-31T19:38:05.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.957+0000 7fd39bfff700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd38c031090 0x7fd38c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:05.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.957+0000 7fd39bfff700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd38c031090 0x7fd38c033550 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fd3880096a0 tx=0x7fd388006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:05.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:05.962+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd390026350 con 0x7fd39c0f5480 2022-01-31T19:38:06.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.126+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd39c0fb060 con 0x7fd39c0f5480 2022-01-31T19:38:06.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.130+0000 7fd3997fa700 1 -- 172.21.15.167:0/1268274652 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd390012330 con 0x7fd39c0f5480 2022-01-31T19:38:06.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd38c031090 msgr2=0x7fd38c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:06.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd38c031090 0x7fd38c033550 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fd3880096a0 tx=0x7fd388006b40).stop 2022-01-31T19:38:06.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 msgr2=0x7fd39c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:06.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c1094f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd390000c00 tx=0x7fd39000b040).stop 2022-01-31T19:38:06.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 shutdown_connections 2022-01-31T19:38:06.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd38c031090 0x7fd38c033550 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:06.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 --2- 172.21.15.167:0/1268274652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd39c0f5480 0x7fd39c1094f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:06.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 >> 172.21.15.167:0/1268274652 conn(0x7fd39c0f0be0 msgr2=0x7fd39c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:06.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 shutdown_connections 2022-01-31T19:38:06.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:06.132+0000 7fd3a3156700 1 -- 172.21.15.167:0/1268274652 wait complete. 2022-01-31T19:38:06.408 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:06 smithi167 conmon[32206]: cluster 2022-01-31T19:38:05.116608+0000 mon.smithi167 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi167.aciqpk(active, since 6s) 2022-01-31T19:38:06.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:06 smithi167 conmon[32206]: audit 2022-01-31T19:38:05.655362+0000 mon.smithi167 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:06.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:06 smithi167 conmon[32206]: audit 2022-01-31T19:38:06.050163+0000 mon.smithi167 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi167", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:38:06.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:06 smithi167 conmon[32206]: audit 2022-01-31T19:38:06.053348+0000 mon.smithi167 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:06.466 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-01-31T19:38:06.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-01-31T19:38:07.402 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:07 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:06.060148+0000 mgr.smithi167.aciqpk (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi167 on smithi167 2022-01-31T19:38:07.403 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:07 smithi167 conmon[32206]: audit 2022-01-31T19:38:06.131888+0000 mon.smithi167 (mon.0) 98 : audit [INF] from='client.? 172.21.15.167:0/1268274652' entity='client.admin' 2022-01-31T19:38:08.081 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 -- 172.21.15.167:0/3218893204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 msgr2=0x7f48e80f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:08.082 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 --2- 172.21.15.167:0/3218893204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e80f6800 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f48d8009fb0 tx=0x7f48d8009b30).stop 2022-01-31T19:38:08.082 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 -- 172.21.15.167:0/3218893204 shutdown_connections 2022-01-31T19:38:08.082 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 --2- 172.21.15.167:0/3218893204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e80f6800 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:08.082 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 -- 172.21.15.167:0/3218893204 >> 172.21.15.167:0/3218893204 conn(0x7f48e80f1b40 msgr2=0x7f48e80f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 -- 172.21.15.167:0/3218893204 shutdown_connections 2022-01-31T19:38:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 -- 172.21.15.167:0/3218893204 wait complete. 2022-01-31T19:38:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.079+0000 7f48ed01a700 1 Processor -- start 2022-01-31T19:38:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48ed01a700 1 -- start start 2022-01-31T19:38:08.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48ed01a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e8106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:08.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48ed01a700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e8106b50 con 0x7f48e80f63e0 2022-01-31T19:38:08.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48e7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e8106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:08.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48e7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e8106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60624/0 (socket says 172.21.15.167:60624) 2022-01-31T19:38:08.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48e7fff700 1 -- 172.21.15.167:0/3167495948 learned_addr learned my addr 172.21.15.167:0/3167495948 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:08.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.080+0000 7f48e7fff700 1 -- 172.21.15.167:0/3167495948 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8005040 con 0x7f48e80f63e0 2022-01-31T19:38:08.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48e7fff700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e8106660 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f48d80089f0 tx=0x7f48d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:08.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48d8009320 con 0x7f48e80f63e0 2022-01-31T19:38:08.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e8106d50 con 0x7f48e80f63e0 2022-01-31T19:38:08.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f48d8013ec0 con 0x7f48e80f63e0 2022-01-31T19:38:08.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48d8014d30 con 0x7f48e80f63e0 2022-01-31T19:38:08.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e81071e0 con 0x7f48e80f63e0 2022-01-31T19:38:08.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.081+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f48d80263a0 con 0x7f48e80f63e0 2022-01-31T19:38:08.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.082+0000 7f48e4ff9700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f48d0031090 0x7f48d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:08.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.082+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f48d80423c0 con 0x7f48e80f63e0 2022-01-31T19:38:08.090 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.082+0000 7f48e77fe700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f48d0031090 0x7f48d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:08.090 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.082+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e8045bb0 con 0x7f48e80f63e0 2022-01-31T19:38:08.090 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.082+0000 7f48e77fe700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f48d0031090 0x7f48d0033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f48dc0096a0 tx=0x7f48dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:08.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.086+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f48d8012920 con 0x7f48e80f63e0 2022-01-31T19:38:08.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.261+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f48e80f2d90 con 0x7f48d0031090 2022-01-31T19:38:08.266 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.265+0000 7f48e4ff9700 1 -- 172.21.15.167:0/3167495948 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f48e80f2d90 con 0x7f48d0031090 2022-01-31T19:38:08.269 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f48d0031090 msgr2=0x7f48d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:08.269 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f48d0031090 0x7f48d0033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f48dc0096a0 tx=0x7f48dc006b40).stop 2022-01-31T19:38:08.270 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 msgr2=0x7f48e8106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:08.270 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e8106660 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f48d80089f0 tx=0x7f48d800b040).stop 2022-01-31T19:38:08.271 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 shutdown_connections 2022-01-31T19:38:08.271 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f48d0031090 0x7f48d0033550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:08.271 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 --2- 172.21.15.167:0/3167495948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48e80f63e0 0x7f48e8106660 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:08.271 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.267+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 >> 172.21.15.167:0/3167495948 conn(0x7f48e80f1b40 msgr2=0x7f48e80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:08.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.268+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 shutdown_connections 2022-01-31T19:38:08.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:08.268+0000 7f48ed01a700 1 -- 172.21.15.167:0/3167495948 wait complete. 2022-01-31T19:38:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:09 smithi167 conmon[32206]: audit 2022-01-31T19:38:08.263869+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:38:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:09 smithi167 conmon[32206]: .14156) 6 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:38:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:09 smithi167 conmon[32206]: audit 2022-01-31T19:38:08.266594+0000 mon.smithi167 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:09.515 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi171 2022-01-31T19:38:09.516 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:38:09.516 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.conf 2022-01-31T19:38:09.536 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:38:09.537 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:38:09.596 INFO:tasks.cephadm:Adding host smithi171 to orchestrator... 2022-01-31T19:38:09.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch host add smithi171 2022-01-31T19:38:11.702 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 -- 172.21.15.167:0/2766142867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 msgr2=0x7ff8b80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:11.702 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 --2- 172.21.15.167:0/2766142867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b80f69b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff8a8009fb0 tx=0x7ff8a8009b30).stop 2022-01-31T19:38:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 -- 172.21.15.167:0/2766142867 shutdown_connections 2022-01-31T19:38:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 --2- 172.21.15.167:0/2766142867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b80f69b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 -- 172.21.15.167:0/2766142867 >> 172.21.15.167:0/2766142867 conn(0x7ff8b80f1d10 msgr2=0x7ff8b80f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 -- 172.21.15.167:0/2766142867 shutdown_connections 2022-01-31T19:38:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.698+0000 7ff8be891700 1 -- 172.21.15.167:0/2766142867 wait complete. 2022-01-31T19:38:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8be891700 1 Processor -- start 2022-01-31T19:38:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8be891700 1 -- start start 2022-01-31T19:38:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8be891700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b8106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8be891700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b8106cf0 con 0x7ff8b80f6590 2022-01-31T19:38:11.705 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8b7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b8106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:11.705 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8b7fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b8106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60630/0 (socket says 172.21.15.167:60630) 2022-01-31T19:38:11.705 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8b7fff700 1 -- 172.21.15.167:0/453752676 learned_addr learned my addr 172.21.15.167:0/453752676 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:11.705 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.699+0000 7ff8b7fff700 1 -- 172.21.15.167:0/453752676 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a8005040 con 0x7ff8b80f6590 2022-01-31T19:38:11.705 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.700+0000 7ff8b7fff700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b8106800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff8a80089f0 tx=0x7ff8a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:11.706 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.700+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a8009320 con 0x7ff8b80f6590 2022-01-31T19:38:11.706 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.700+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff8a8013ec0 con 0x7ff8b80f6590 2022-01-31T19:38:11.706 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.700+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b8106ef0 con 0x7ff8b80f6590 2022-01-31T19:38:11.706 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.700+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a8014d30 con 0x7ff8b80f6590 2022-01-31T19:38:11.707 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.700+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b8107380 con 0x7ff8b80f6590 2022-01-31T19:38:11.708 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.701+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ff8a8014420 con 0x7ff8b80f6590 2022-01-31T19:38:11.708 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.701+0000 7ff8b4ff9700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff8a0031090 0x7ff8a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:11.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.701+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b8045bb0 con 0x7ff8b80f6590 2022-01-31T19:38:11.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.701+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff8a8042390 con 0x7ff8b80f6590 2022-01-31T19:38:11.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.701+0000 7ff8b77fe700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff8a0031090 0x7ff8a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:11.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.701+0000 7ff8b77fe700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff8a0031090 0x7ff8a0033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff8ac0096a0 tx=0x7ff8ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:11.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.705+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff8a801d7f0 con 0x7ff8b80f6590 2022-01-31T19:38:11.873 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:11.872+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8b80f2f10 con 0x7ff8a0031090 2022-01-31T19:38:12.601 INFO:teuthology.orchestra.run.smithi167.stdout:Added host 'smithi171' with addr '172.21.15.171' 2022-01-31T19:38:12.603 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.597+0000 7ff8b4ff9700 1 -- 172.21.15.167:0/453752676 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff8b80f2f10 con 0x7ff8a0031090 2022-01-31T19:38:12.604 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff8a0031090 msgr2=0x7ff8a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:12.604 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff8a0031090 0x7ff8a0033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff8ac0096a0 tx=0x7ff8ac006b40).stop 2022-01-31T19:38:12.604 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 msgr2=0x7ff8b8106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:12.604 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b8106800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff8a80089f0 tx=0x7ff8a800b040).stop 2022-01-31T19:38:12.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 shutdown_connections 2022-01-31T19:38:12.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff8a0031090 0x7ff8a0033550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:12.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 --2- 172.21.15.167:0/453752676 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8b80f6590 0x7ff8b8106800 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:12.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 >> 172.21.15.167:0/453752676 conn(0x7ff8b80f1d10 msgr2=0x7ff8b80f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:12.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 shutdown_connections 2022-01-31T19:38:12.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:12.600+0000 7ff8be891700 1 -- 172.21.15.167:0/453752676 wait complete. 2022-01-31T19:38:12.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:12 smithi167 conmon[32206]: audit 2022-01-31T19:38:11.874128+0000 mgr.smithi167.aciqpk (mgr.14156) 7 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:38:13.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch host ls --format=json 2022-01-31T19:38:13.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: audit 2022-01-31T19:38:12.598229+0000 2022-01-31T19:38:13.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: mon.smithi167 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:13.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:12.598511+0000 mgr.smithi167.aciqpk (mgr.14156) 8 : cephadm [INF] Added host smithi171 2022-01-31T19:38:13.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: audit 2022-01-31T19:38:13.125778+0000 mon.smithi167 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:13.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: audit 2022-01-31T19:38:13.127555+0000 mon.smithi167 (mon.0) 102 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi167", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2022-01-31T19:38:13.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: audit 2022-01-31T19:38:13.129247+0000 mon.smithi167 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi167", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2022-01-31T19:38:13.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: audit 2022-01-31T19:38:13.129869+0000 mon.smithi167 (mon.0) 104 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:13.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:13 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:13.130495+0000 mgr.smithi167.aciqpk (mgr.14156) 9 : cephadm [INF] Deploying daemon crash.smithi167 on smithi167 2022-01-31T19:38:14.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:14 smithi167 conmon[32206]: audit 2022-01-31T19:38:13.815260+0000 mon.smithi167 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:15.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 -- 172.21.15.167:0/1416457575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 msgr2=0x7ff15c0fd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:15.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 --2- 172.21.15.167:0/1416457575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c0fd130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff150004760 tx=0x7ff150009ad0).stop 2022-01-31T19:38:15.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 -- 172.21.15.167:0/1416457575 shutdown_connections 2022-01-31T19:38:15.093 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 --2- 172.21.15.167:0/1416457575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c0fd130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:15.093 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 -- 172.21.15.167:0/1416457575 >> 172.21.15.167:0/1416457575 conn(0x7ff15c06bd00 msgr2=0x7ff15c06e140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:15.093 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 -- 172.21.15.167:0/1416457575 shutdown_connections 2022-01-31T19:38:15.093 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 -- 172.21.15.167:0/1416457575 wait complete. 2022-01-31T19:38:15.094 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.087+0000 7ff16216b700 1 Processor -- start 2022-01-31T19:38:15.094 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff16216b700 1 -- start start 2022-01-31T19:38:15.094 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff16216b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c10ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:15.095 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff16216b700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c10d310 con 0x7ff15c0fcd10 2022-01-31T19:38:15.095 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff15b7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c10ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:15.095 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff15b7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c10ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60638/0 (socket says 172.21.15.167:60638) 2022-01-31T19:38:15.095 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff15b7fe700 1 -- 172.21.15.167:0/1172501973 learned_addr learned my addr 172.21.15.167:0/1172501973 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:15.096 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff15b7fe700 1 -- 172.21.15.167:0/1172501973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff150009d70 con 0x7ff15c0fcd10 2022-01-31T19:38:15.096 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff15b7fe700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c10ce20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff150004170 tx=0x7ff150009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:15.096 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.088+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff15000e040 con 0x7ff15c0fcd10 2022-01-31T19:38:15.096 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.089+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff150008ef0 con 0x7ff15c0fcd10 2022-01-31T19:38:15.097 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.089+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c10d510 con 0x7ff15c0fcd10 2022-01-31T19:38:15.097 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.089+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff150014c90 con 0x7ff15c0fcd10 2022-01-31T19:38:15.097 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.089+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c10e120 con 0x7ff15c0fcd10 2022-01-31T19:38:15.097 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.089+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7ff1500283f0 con 0x7ff15c0fcd10 2022-01-31T19:38:15.098 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.090+0000 7ff143fff700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff144031090 0x7ff144033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:15.098 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.090+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c10d9a0 con 0x7ff15c0fcd10 2022-01-31T19:38:15.098 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.090+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff150042250 con 0x7ff15c0fcd10 2022-01-31T19:38:15.098 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.090+0000 7ff15affd700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff144031090 0x7ff144033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:15.099 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.090+0000 7ff15affd700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff144031090 0x7ff144033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff14c0096a0 tx=0x7ff14c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:15.099 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.094+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff15c10d9a0 con 0x7ff15c0fcd10 2022-01-31T19:38:15.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.257+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff15c10d9a0 con 0x7ff144031090 2022-01-31T19:38:15.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.259+0000 7ff143fff700 1 -- 172.21.15.167:0/1172501973 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7ff15c10d9a0 con 0x7ff144031090 2022-01-31T19:38:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:38:15.261 INFO:teuthology.orchestra.run.smithi167.stdout:[{"addr": "172.21.15.167", "hostname": "smithi167", "labels": [], "status": ""}, {"addr": "172.21.15.171", "hostname": "smithi171", "labels": [], "status": ""}] 2022-01-31T19:38:15.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff144031090 msgr2=0x7ff144033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:15.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff144031090 0x7ff144033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff14c0096a0 tx=0x7ff14c006b40).stop 2022-01-31T19:38:15.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 msgr2=0x7ff15c10ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:15.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c10ce20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff150004170 tx=0x7ff150009ad0).stop 2022-01-31T19:38:15.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 shutdown_connections 2022-01-31T19:38:15.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff144031090 0x7ff144033550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:15.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 --2- 172.21.15.167:0/1172501973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff15c0fcd10 0x7ff15c10ce20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:15.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.260+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 >> 172.21.15.167:0/1172501973 conn(0x7ff15c06bd00 msgr2=0x7ff15c06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:15.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.261+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 shutdown_connections 2022-01-31T19:38:15.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:15.261+0000 7ff16216b700 1 -- 172.21.15.167:0/1172501973 wait complete. 2022-01-31T19:38:15.708 INFO:tasks.cephadm:Setting crush tunables to default 2022-01-31T19:38:15.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd crush tunables default 2022-01-31T19:38:15.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: audit 2022-01-31T19:38:15.260040 2022-01-31T19:38:15.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 10 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-01-31T19:38:15.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: audit 2022-01-31T19:38:15.519683+0000 mon.smithi167 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:15.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: audit 2022-01-31T19:38:15.789017+0000 mon.smithi167 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:15.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: audit 2022-01-31T19:38:15.791028+0000 mon.smithi167 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:15.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: audit 2022-01-31T19:38:15.791407+0000 mon.smithi167 (mon.0) 109 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-01-31T19:38:15.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: 2022-01-31T19:38:15.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:38:15.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: T19:38:15.797558+0000 mon.smithi167 ( 2022-01-31T19:38:15.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: mon.0) 110 : audit [INF] 2022-01-31T19:38:15.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:15 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:16.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:16 smithi167 conmon[32206]: audit 2022-01-31T19:38:15.791895+0000 mgr.smithi167.aciqpk (mgr.14156) 11 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2022-01-31T19:38:16.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:16 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:15.800808+0000 mgr.smithi167.aciqpk (mgr.14156) 12 : cephadm [INF] Deploying daemon grafana.smithi167 on smithi167 2022-01-31T19:38:17.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 -- 172.21.15.167:0/4174727815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 msgr2=0x7f98500f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:17.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 --2- 172.21.15.167:0/4174727815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f98500f6800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9840009fb0 tx=0x7f9840009b30).stop 2022-01-31T19:38:17.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 -- 172.21.15.167:0/4174727815 shutdown_connections 2022-01-31T19:38:17.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 --2- 172.21.15.167:0/4174727815 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f98500f6800 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:17.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 -- 172.21.15.167:0/4174727815 >> 172.21.15.167:0/4174727815 conn(0x7f98500f1b40 msgr2=0x7f98500f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:17.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 -- 172.21.15.167:0/4174727815 shutdown_connections 2022-01-31T19:38:17.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 -- 172.21.15.167:0/4174727815 wait complete. 2022-01-31T19:38:17.404 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.398+0000 7f98556b8700 1 Processor -- start 2022-01-31T19:38:17.404 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f98556b8700 1 -- start start 2022-01-31T19:38:17.404 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f98556b8700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f985006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:17.404 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f98556b8700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9850069f00 con 0x7f98500f63e0 2022-01-31T19:38:17.404 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f984ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f985006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:17.405 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f984ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f985006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60654/0 (socket says 172.21.15.167:60654) 2022-01-31T19:38:17.405 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f984ffff700 1 -- 172.21.15.167:0/3666228709 learned_addr learned my addr 172.21.15.167:0/3666228709 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:38:17.405 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f984ffff700 1 -- 172.21.15.167:0/3666228709 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840005040 con 0x7f98500f63e0 2022-01-31T19:38:17.405 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.399+0000 7f984ffff700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f985006ba80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f98400089f0 tx=0x7f984000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:17.406 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.400+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9840009320 con 0x7f98500f63e0 2022-01-31T19:38:17.406 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.400+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985006a0a0 con 0x7f98500f63e0 2022-01-31T19:38:17.406 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.400+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985006a530 con 0x7f98500f63e0 2022-01-31T19:38:17.407 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.400+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9840013ec0 con 0x7f98500f63e0 2022-01-31T19:38:17.407 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.400+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9840014d30 con 0x7f98500f63e0 2022-01-31T19:38:17.407 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.400+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f98400263a0 con 0x7f98500f63e0 2022-01-31T19:38:17.408 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.401+0000 7f984cff9700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f98380310e0 0x7f98380335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:17.408 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.401+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f98400432b0 con 0x7f98500f63e0 2022-01-31T19:38:17.409 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.401+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9830004fa0 con 0x7f98500f63e0 2022-01-31T19:38:17.409 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.401+0000 7f984f7fe700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f98380310e0 0x7f98380335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:17.409 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.401+0000 7f984f7fe700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f98380310e0 0x7f98380335a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f98440096a0 tx=0x7f9844006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:17.409 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.405+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9840012900 con 0x7f98500f63e0 2022-01-31T19:38:17.572 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.570+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9830005670 con 0x7f98500f63e0 2022-01-31T19:38:17.820 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.818+0000 7f984cff9700 1 -- 172.21.15.167:0/3666228709 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f984002b030 con 0x7f98500f63e0 2022-01-31T19:38:17.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f98380310e0 msgr2=0x7f98380335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:17.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f98380310e0 0x7f98380335a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f98440096a0 tx=0x7f9844006b40).stop 2022-01-31T19:38:17.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 msgr2=0x7f985006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:17.823 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f985006ba80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f98400089f0 tx=0x7f984000b040).stop 2022-01-31T19:38:17.823 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 shutdown_connections 2022-01-31T19:38:17.823 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f98380310e0 0x7f98380335a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:17.823 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 --2- 172.21.15.167:0/3666228709 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98500f63e0 0x7f985006ba80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:17.824 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 >> 172.21.15.167:0/3666228709 conn(0x7f98500f1b40 msgr2=0x7f98500f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:17.824 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 shutdown_connections 2022-01-31T19:38:17.824 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:38:17.820+0000 7f98556b8700 1 -- 172.21.15.167:0/3666228709 wait complete. 2022-01-31T19:38:17.825 INFO:teuthology.orchestra.run.smithi167.stderr:adjusted tunables profile to default 2022-01-31T19:38:18.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:17 smithi167 conmon[32206]: audit 2022-01-31T19:38:17.572527+0000 mon.smithi167 (mon 2022-01-31T19:38:18.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:17 smithi167 conmon[32206]: .0) 111 : audit [INF] from='client.? 172.21.15.167:0/3666228709' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-01-31T19:38:18.157 INFO:tasks.cephadm:Adding mon.smithi167 on smithi167 2022-01-31T19:38:18.157 INFO:tasks.cephadm:Adding mon.smithi171 on smithi171 2022-01-31T19:38:18.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch apply mon '2;smithi167:172.21.15.167=smithi167;smithi171:172.21.15.171=smithi171' 2022-01-31T19:38:19.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:18 smithi167 conmon[32206]: audit 2022-01-31T19:38:17.820076+0000 mon.smithi167 (mon. 2022-01-31T19:38:19.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:18 smithi167 conmon[32206]: 0) 112 : audit [INF] from='client.? 172.21.15.167:0/3666228709' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-01-31T19:38:19.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:18 smithi167 conmon[32206]: cluster 2022-01-31T19:38:17.820124+0000 mon.smithi167 (mon.0) 113 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-01-31T19:38:19.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:18 smithi167 conmon[32206]: audit 2022-01-31T19:38:18.817957+0000 mon.smithi167 (mon.0) 114 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:19.489 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.485+0000 7f148675d700 1 -- 172.21.15.171:0/1847704848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 msgr2=0x7f14800f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:19.490 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.485+0000 7f148675d700 1 --2- 172.21.15.171:0/1847704848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f14800f6800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f1470009fb0 tx=0x7f1470009b30).stop 2022-01-31T19:38:19.490 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.485+0000 7f148675d700 1 -- 172.21.15.171:0/1847704848 shutdown_connections 2022-01-31T19:38:19.490 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.485+0000 7f148675d700 1 --2- 172.21.15.171:0/1847704848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f14800f6800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:19.490 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.485+0000 7f148675d700 1 -- 172.21.15.171:0/1847704848 >> 172.21.15.171:0/1847704848 conn(0x7f14800f1b40 msgr2=0x7f14800f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:19.491 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.486+0000 7f148675d700 1 -- 172.21.15.171:0/1847704848 shutdown_connections 2022-01-31T19:38:19.491 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.486+0000 7f148675d700 1 -- 172.21.15.171:0/1847704848 wait complete. 2022-01-31T19:38:19.491 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.486+0000 7f148675d700 1 Processor -- start 2022-01-31T19:38:19.492 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.486+0000 7f148675d700 1 -- start start 2022-01-31T19:38:19.492 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148675d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f1480106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:19.493 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148675d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1480106b10 con 0x7f14800f63e0 2022-01-31T19:38:19.493 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148575b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f1480106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:19.493 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148575b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f1480106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46212/0 (socket says 172.21.15.171:46212) 2022-01-31T19:38:19.493 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148575b700 1 -- 172.21.15.171:0/893117814 learned_addr learned my addr 172.21.15.171:0/893117814 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:19.494 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148575b700 1 -- 172.21.15.171:0/893117814 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470005040 con 0x7f14800f63e0 2022-01-31T19:38:19.494 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148575b700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f1480106620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f14700089f0 tx=0x7f147000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:19.494 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1470009320 con 0x7f14800f63e0 2022-01-31T19:38:19.494 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1470013ec0 con 0x7f14800f63e0 2022-01-31T19:38:19.495 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1480106d10 con 0x7f14800f63e0 2022-01-31T19:38:19.495 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.487+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1470014d30 con 0x7f14800f63e0 2022-01-31T19:38:19.495 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.488+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14801071a0 con 0x7f14800f63e0 2022-01-31T19:38:19.495 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.488+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f14700263a0 con 0x7f14800f63e0 2022-01-31T19:38:19.496 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.488+0000 7f1475ffb700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f146c031090 0x7f146c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:19.496 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.488+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f14700424d0 con 0x7f14800f63e0 2022-01-31T19:38:19.496 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.489+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14800591e0 con 0x7f14800f63e0 2022-01-31T19:38:19.496 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.489+0000 7f1484f5a700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f146c031090 0x7f146c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:19.497 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.489+0000 7f1484f5a700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f146c031090 0x7f146c033550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f147c0096a0 tx=0x7f147c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:19.498 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.497+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f14700266d0 con 0x7f14800f63e0 2022-01-31T19:38:19.656 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.655+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi167:172.21.15.167=smithi167;smithi171:172.21.15.171=smithi171", "target": ["mon-mgr", ""]}) v1 -- 0x7f14800f2e60 con 0x7f146c031090 2022-01-31T19:38:19.660 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.659+0000 7f1475ffb700 1 -- 172.21.15.171:0/893117814 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f14800f2e60 con 0x7f146c031090 2022-01-31T19:38:19.661 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mon update... 2022-01-31T19:38:19.663 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f146c031090 msgr2=0x7f146c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:19.663 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f146c031090 0x7f146c033550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f147c0096a0 tx=0x7f147c006b40).stop 2022-01-31T19:38:19.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 msgr2=0x7f1480106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:19.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f1480106620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f14700089f0 tx=0x7f147000b040).stop 2022-01-31T19:38:19.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 shutdown_connections 2022-01-31T19:38:19.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f146c031090 0x7f146c033550 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:19.665 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 --2- 172.21.15.171:0/893117814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14800f63e0 0x7f1480106620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:19.665 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 >> 172.21.15.171:0/893117814 conn(0x7f14800f1b40 msgr2=0x7f14800f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:19.665 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 shutdown_connections 2022-01-31T19:38:19.665 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:19.661+0000 7f148675d700 1 -- 172.21.15.171:0/893117814 wait complete. 2022-01-31T19:38:20.009 DEBUG:teuthology.orchestra.run.smithi171:mon.smithi171> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi171.service 2022-01-31T19:38:20.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:20.013 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:20.042 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:-- Logs begin at Mon 2022-01-31 19:28:36 UTC. -- 2022-01-31T19:38:20.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:19 smithi167 conmon[32206]: cluster 2022-01-31T19:38:18.688432+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:38:20.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:19 smithi167 conmon[32206]: .14156) 13 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:20.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:19 smithi167 conmon[32206]: audit 2022-01-31T19:38:19.660041+0000 mon.smithi167 (mon.0) 115 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:21.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:20 smithi167 conmon[32206]: audit 2022-01-31T19:38:19.656644+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:38:21.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:20 smithi167 conmon[32206]: 14156) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi167:172.21.15.167=smithi167;smithi171:172.21.15.171=smithi171", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:38:21.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:20 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:19.657769+0000 mgr.smithi167.aciqpk (mgr.14156) 15 : cephadm [INF] Saving service mon spec with placement smithi167:172.21.15.167=smithi167;smithi171:172.21.15.171=smithi171;count:2 2022-01-31T19:38:21.450 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/2790587851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 msgr2=0x7fdd740f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:21.451 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 --2- 172.21.15.171:0/2790587851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd740f58d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdd64009fb0 tx=0x7fdd64009b30).stop 2022-01-31T19:38:21.451 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/2790587851 shutdown_connections 2022-01-31T19:38:21.451 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 --2- 172.21.15.171:0/2790587851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd740f58d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:21.452 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/2790587851 >> 172.21.15.171:0/2790587851 conn(0x7fdd740f0c10 msgr2=0x7fdd740f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:21.452 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/2790587851 shutdown_connections 2022-01-31T19:38:21.452 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.446+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/2790587851 wait complete. 2022-01-31T19:38:21.453 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd7bdd4700 1 Processor -- start 2022-01-31T19:38:21.453 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd7bdd4700 1 -- start start 2022-01-31T19:38:21.453 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd7bdd4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd74105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:21.453 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd7bdd4700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74105c10 con 0x7fdd740f54b0 2022-01-31T19:38:21.454 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd79b70700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd74105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:21.454 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd79b70700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd74105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46218/0 (socket says 172.21.15.171:46218) 2022-01-31T19:38:21.454 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd79b70700 1 -- 172.21.15.171:0/3677237515 learned_addr learned my addr 172.21.15.171:0/3677237515 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:21.455 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.447+0000 7fdd79b70700 1 -- 172.21.15.171:0/3677237515 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64005040 con 0x7fdd740f54b0 2022-01-31T19:38:21.455 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.448+0000 7fdd79b70700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd74105720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdd640089f0 tx=0x7fdd6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:21.455 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.448+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd64009320 con 0x7fdd740f54b0 2022-01-31T19:38:21.455 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.448+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd74105e10 con 0x7fdd740f54b0 2022-01-31T19:38:21.456 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.448+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdd64013ec0 con 0x7fdd740f54b0 2022-01-31T19:38:21.456 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.448+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd64014d30 con 0x7fdd740f54b0 2022-01-31T19:38:21.456 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.448+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd741062a0 con 0x7fdd740f54b0 2022-01-31T19:38:21.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.449+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fdd640095f0 con 0x7fdd740f54b0 2022-01-31T19:38:21.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.449+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd74045bb0 con 0x7fdd740f54b0 2022-01-31T19:38:21.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.449+0000 7fdd6a7fc700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdd60031090 0x7fdd60033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:21.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.449+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdd640423d0 con 0x7fdd740f54b0 2022-01-31T19:38:21.458 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.449+0000 7fdd7936f700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdd60031090 0x7fdd60033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:21.458 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.449+0000 7fdd7936f700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdd60031090 0x7fdd60033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fdd700096a0 tx=0x7fdd70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:21.459 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.454+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdd6401d7f0 con 0x7fdd740f54b0 2022-01-31T19:38:21.659 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.658+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd740fafa0 con 0x7fdd740f54b0 2022-01-31T19:38:21.660 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.659+0000 7fdd6a7fc700 1 -- 172.21.15.171:0/3677237515 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd6400fe40 con 0x7fdd740f54b0 2022-01-31T19:38:21.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:21.661 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:21.662 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdd60031090 msgr2=0x7fdd60033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:21.662 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdd60031090 0x7fdd60033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fdd700096a0 tx=0x7fdd70006b40).stop 2022-01-31T19:38:21.663 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 msgr2=0x7fdd74105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:21.663 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd74105720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdd640089f0 tx=0x7fdd6400b040).stop 2022-01-31T19:38:21.663 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 shutdown_connections 2022-01-31T19:38:21.663 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdd60031090 0x7fdd60033550 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:21.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 --2- 172.21.15.171:0/3677237515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd740f54b0 0x7fdd74105720 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:21.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 >> 172.21.15.171:0/3677237515 conn(0x7fdd740f0c10 msgr2=0x7fdd740f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:21.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 shutdown_connections 2022-01-31T19:38:21.664 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:21.661+0000 7fdd7bdd4700 1 -- 172.21.15.171:0/3677237515 wait complete. 2022-01-31T19:38:21.665 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:22.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:21 smithi167 conmon[32206]: cluster 2022-01-31T19:38:20.688604+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:38:22.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:21 smithi167 conmon[32206]: 14156) 16 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:22.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:21 smithi167 conmon[32206]: audit 2022-01-31T19:38:21.659857+0000 mon.smithi167 (mon.0) 116 : audit [DBG] from='client.? 172.21.15.171:0/3677237515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:23.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:23.086 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:24.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:23 smithi167 conmon[32206]: cluster 2022-01-31T19:38:22.688862+0000 mgr.smithi167.aciqpk (mgr.14156) 17 : cluster [DBG] 2022-01-31T19:38:24.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:23 smithi167 conmon[32206]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:24.465 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 -- 172.21.15.171:0/3166101484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 msgr2=0x7f848c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:24.465 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 --2- 172.21.15.171:0/3166101484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c0f69b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8480009fb0 tx=0x7f8480009b30).stop 2022-01-31T19:38:24.466 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 -- 172.21.15.171:0/3166101484 shutdown_connections 2022-01-31T19:38:24.466 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 --2- 172.21.15.171:0/3166101484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c0f69b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:24.466 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 -- 172.21.15.171:0/3166101484 >> 172.21.15.171:0/3166101484 conn(0x7f848c0f1cf0 msgr2=0x7f848c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:24.466 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 -- 172.21.15.171:0/3166101484 shutdown_connections 2022-01-31T19:38:24.467 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.461+0000 7f849324f700 1 -- 172.21.15.171:0/3166101484 wait complete. 2022-01-31T19:38:24.467 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.462+0000 7f849324f700 1 Processor -- start 2022-01-31T19:38:24.467 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.462+0000 7f849324f700 1 -- start start 2022-01-31T19:38:24.467 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.462+0000 7f849324f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:24.467 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.462+0000 7f849324f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c106cf0 con 0x7f848c0f6590 2022-01-31T19:38:24.468 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f8490feb700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:24.468 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f8490feb700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46224/0 (socket says 172.21.15.171:46224) 2022-01-31T19:38:24.468 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f8490feb700 1 -- 172.21.15.171:0/1402686664 learned_addr learned my addr 172.21.15.171:0/1402686664 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:24.468 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f8490feb700 1 -- 172.21.15.171:0/1402686664 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480005040 con 0x7f848c0f6590 2022-01-31T19:38:24.469 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f8490feb700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c106800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f8480000c00 tx=0x7f848000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:24.469 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84800095f0 con 0x7f848c0f6590 2022-01-31T19:38:24.469 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8480007e50 con 0x7f848c0f6590 2022-01-31T19:38:24.469 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c106ef0 con 0x7f848c0f6590 2022-01-31T19:38:24.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8480014e70 con 0x7f848c0f6590 2022-01-31T19:38:24.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.463+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c107380 con 0x7f848c0f6590 2022-01-31T19:38:24.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.464+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f8480025070 con 0x7f848c0f6590 2022-01-31T19:38:24.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.464+0000 7f84897fa700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f847c0310e0 0x7f847c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:24.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.464+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8480042c10 con 0x7f848c0f6590 2022-01-31T19:38:24.471 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.464+0000 7f848bfff700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f847c0310e0 0x7f847c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:24.471 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.464+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c0591e0 con 0x7f848c0f6590 2022-01-31T19:38:24.472 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.465+0000 7f848bfff700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f847c0310e0 0x7f847c0335a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f84780096a0 tx=0x7f8478006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:24.472 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.469+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8480026350 con 0x7f848c0f6590 2022-01-31T19:38:24.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.675+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f848c0fc080 con 0x7f848c0f6590 2022-01-31T19:38:24.677 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.676+0000 7f84897fa700 1 -- 172.21.15.171:0/1402686664 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8480012330 con 0x7f848c0f6590 2022-01-31T19:38:24.678 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:24.679 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:24.680 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f847c0310e0 msgr2=0x7f847c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:24.681 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f847c0310e0 0x7f847c0335a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f84780096a0 tx=0x7f8478006b40).stop 2022-01-31T19:38:24.681 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 msgr2=0x7f848c106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:24.681 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c106800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f8480000c00 tx=0x7f848000b040).stop 2022-01-31T19:38:24.681 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 shutdown_connections 2022-01-31T19:38:24.681 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f847c0310e0 0x7f847c0335a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:24.682 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 --2- 172.21.15.171:0/1402686664 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f848c0f6590 0x7f848c106800 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:24.682 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 >> 172.21.15.171:0/1402686664 conn(0x7f848c0f1cf0 msgr2=0x7f848c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:24.682 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 shutdown_connections 2022-01-31T19:38:24.683 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:24.678+0000 7f849324f700 1 -- 172.21.15.171:0/1402686664 wait complete. 2022-01-31T19:38:24.683 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:24.962 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:24 smithi167 conmon[32206]: audit 2022-01-31T19:38:24.676964+0000 mon.smithi167 (mon.0) 117 : audit [DBG] from='client.? 172.21.15.171:0/1402686664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:25.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:25 smithi167 conmon[32206]: cluster 2022-01-31T19:38:24.689162+0000 mgr.smithi167.aciqpk (mgr.14156) 18 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:26.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:26.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:27.530 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3917944007 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 msgr2=0x7f1d880f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:27.531 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 --2- 172.21.15.171:0/3917944007 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d880f91c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1d7c009fb0 tx=0x7f1d7c009b30).stop 2022-01-31T19:38:27.531 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3917944007 shutdown_connections 2022-01-31T19:38:27.531 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 --2- 172.21.15.171:0/3917944007 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d880f91c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:27.531 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3917944007 >> 172.21.15.171:0/3917944007 conn(0x7f1d880f0be0 msgr2=0x7f1d880f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:27.532 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3917944007 shutdown_connections 2022-01-31T19:38:27.532 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3917944007 wait complete. 2022-01-31T19:38:27.532 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.526+0000 7f1d8dad5700 1 Processor -- start 2022-01-31T19:38:27.532 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d8dad5700 1 -- start start 2022-01-31T19:38:27.533 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d8dad5700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d88106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:27.533 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d8dad5700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d88106c70 con 0x7f1d88060300 2022-01-31T19:38:27.533 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d877fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d88106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:27.533 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d877fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d88106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46230/0 (socket says 172.21.15.171:46230) 2022-01-31T19:38:27.533 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d877fe700 1 -- 172.21.15.171:0/3091680616 learned_addr learned my addr 172.21.15.171:0/3091680616 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:27.534 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d877fe700 1 -- 172.21.15.171:0/3091680616 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c005040 con 0x7f1d88060300 2022-01-31T19:38:27.534 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.527+0000 7f1d877fe700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d88106780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1d7c009f80 tx=0x7f1d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:27.534 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c009320 con 0x7f1d88060300 2022-01-31T19:38:27.534 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1d7c013680 con 0x7f1d88060300 2022-01-31T19:38:27.535 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d88106e70 con 0x7f1d88060300 2022-01-31T19:38:27.535 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c01eda0 con 0x7f1d88060300 2022-01-31T19:38:27.535 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d88107300 con 0x7f1d88060300 2022-01-31T19:38:27.535 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f1d7c014420 con 0x7f1d88060300 2022-01-31T19:38:27.536 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d6ffff700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1d70031090 0x7f1d70033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:27.536 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.528+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1d7c042360 con 0x7f1d88060300 2022-01-31T19:38:27.536 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.529+0000 7f1d86ffd700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1d70031090 0x7f1d70033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:27.536 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.529+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d880591e0 con 0x7f1d88060300 2022-01-31T19:38:27.537 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.529+0000 7f1d86ffd700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1d70031090 0x7f1d70033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1d780096a0 tx=0x7f1d78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:27.537 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.533+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1d7c014700 con 0x7f1d88060300 2022-01-31T19:38:27.745 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.744+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d88045bb0 con 0x7f1d88060300 2022-01-31T19:38:27.745 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.744+0000 7f1d6ffff700 1 -- 172.21.15.171:0/3091680616 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d7c01c320 con 0x7f1d88060300 2022-01-31T19:38:27.746 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:27.746 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:27.748 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1d70031090 msgr2=0x7f1d70033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:27.748 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1d70031090 0x7f1d70033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1d780096a0 tx=0x7f1d78006b40).stop 2022-01-31T19:38:27.748 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 msgr2=0x7f1d88106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:27.748 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d88106780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1d7c009f80 tx=0x7f1d7c00b040).stop 2022-01-31T19:38:27.749 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 shutdown_connections 2022-01-31T19:38:27.749 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1d70031090 0x7f1d70033550 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:27.749 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 --2- 172.21.15.171:0/3091680616 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d88060300 0x7f1d88106780 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:27.749 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 >> 172.21.15.171:0/3091680616 conn(0x7f1d880f0be0 msgr2=0x7f1d880f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:27.749 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 shutdown_connections 2022-01-31T19:38:27.750 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:27.746+0000 7f1d8dad5700 1 -- 172.21.15.171:0/3091680616 wait complete. 2022-01-31T19:38:27.750 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:27.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:27 smithi167 conmon[32206]: audit 2022-01-31T19:38:26.491255+0000 mon.smithi167 (mon.0) 118 2022-01-31T19:38:27.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:27 smithi167 conmon[32206]: : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:27.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:27 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:26.493101+0000 mgr.smithi167.aciqpk (mgr.14156) 19 : cephadm [INF] Deploying daemon node-exporter.smithi167 on smithi167 2022-01-31T19:38:27.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:27 smithi167 conmon[32206]: cluster 2022-01-31T19:38:26.689425+0000 mgr.smithi167.aciqpk (mgr.14156) 20 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:28.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:28 smithi167 conmon[32206]: audit 2022-01-31T19:38:27.745268+0000 mon.smithi167 (mon.0) 119 : audit [DBG] from='client.? 172.21.15.171:0/3091680616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:29.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:29.089 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:29 smithi167 conmon[32206]: cluster 2022-01-31T19:38:28.689645+0000 mgr.smithi167.aciqpk (mgr.14156) 21 : 2022-01-31T19:38:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:29 smithi167 conmon[32206]: cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:29 smithi167 conmon[32206]: audit 2022-01-31T19:38:28.821131+0000 mon.smithi167 (mon.0) 120 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:29 smithi167 conmon[32206]: audit 2022-01-31T19:38:29.361623+0000 mon.smithi167 (mon.0) 121 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:30.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:29 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:29.367243+0000 mgr.smithi167.aciqpk (mgr.14156) 22 : cephadm [INF] Deploying daemon prometheus.smithi167 on smithi167 2022-01-31T19:38:30.671 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 -- 172.21.15.171:0/3929310579 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f6a90 msgr2=0x7febfc0f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:30.672 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 --2- 172.21.15.171:0/3929310579 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f6a90 0x7febfc0f6eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7febf0009fb0 tx=0x7febf0009b30).stop 2022-01-31T19:38:30.672 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 -- 172.21.15.171:0/3929310579 shutdown_connections 2022-01-31T19:38:30.672 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 --2- 172.21.15.171:0/3929310579 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f6a90 0x7febfc0f6eb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:30.672 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 -- 172.21.15.171:0/3929310579 >> 172.21.15.171:0/3929310579 conn(0x7febfc0f0360 msgr2=0x7febfc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:30.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 -- 172.21.15.171:0/3929310579 shutdown_connections 2022-01-31T19:38:30.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.668+0000 7fec02994700 1 -- 172.21.15.171:0/3929310579 wait complete. 2022-01-31T19:38:30.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7fec02994700 1 Processor -- start 2022-01-31T19:38:30.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7fec02994700 1 -- start start 2022-01-31T19:38:30.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7fec02994700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 0x7febfc109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:30.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7fec02994700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc109b30 con 0x7febfc0f5390 2022-01-31T19:38:30.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7febfbfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 0x7febfc109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:30.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7febfbfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 0x7febfc109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46236/0 (socket says 172.21.15.171:46236) 2022-01-31T19:38:30.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.669+0000 7febfbfff700 1 -- 172.21.15.171:0/58467988 learned_addr learned my addr 172.21.15.171:0/58467988 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:30.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7febfbfff700 1 -- 172.21.15.171:0/58467988 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf0005040 con 0x7febfc0f5390 2022-01-31T19:38:30.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7febfbfff700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 0x7febfc109640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7febf00134d0 tx=0x7febf0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:30.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febf0013ee0 con 0x7febfc0f5390 2022-01-31T19:38:30.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc109c70 con 0x7febfc0f5390 2022-01-31T19:38:30.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7febf0013680 con 0x7febfc0f5390 2022-01-31T19:38:30.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc10a0a0 con 0x7febfc0f5390 2022-01-31T19:38:30.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.670+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febf001ee90 con 0x7febfc0f5390 2022-01-31T19:38:30.677 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.671+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7febf0025060 con 0x7febfc0f5390 2022-01-31T19:38:30.678 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.671+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc004fa0 con 0x7febfc0f5390 2022-01-31T19:38:30.678 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.671+0000 7febf8ff9700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7febe4031040 0x7febe4033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:30.678 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.671+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7febf0042260 con 0x7febfc0f5390 2022-01-31T19:38:30.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.671+0000 7febfb7fe700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7febe4031040 0x7febe4033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:30.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.672+0000 7febfb7fe700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7febe4031040 0x7febe4033500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7febec0096a0 tx=0x7febec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:30.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.675+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7febf001c930 con 0x7febfc0f5390 2022-01-31T19:38:30.884 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.883+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febdc005670 con 0x7febfc0f5390 2022-01-31T19:38:30.885 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.884+0000 7febf8ff9700 1 -- 172.21.15.171:0/58467988 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7febf001c340 con 0x7febfc0f5390 2022-01-31T19:38:30.885 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:30.886 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:30.887 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.885+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7febe4031040 msgr2=0x7febe4033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:30.887 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.885+0000 7fec02994700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7febe4031040 0x7febe4033500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7febec0096a0 tx=0x7febec006b40).stop 2022-01-31T19:38:30.887 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.885+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 msgr2=0x7febfc109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:30.888 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.885+0000 7fec02994700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 0x7febfc109640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7febf00134d0 tx=0x7febf0009b30).stop 2022-01-31T19:38:30.888 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.885+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 shutdown_connections 2022-01-31T19:38:30.888 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.885+0000 7fec02994700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7febe4031040 0x7febe4033500 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:30.888 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.886+0000 7fec02994700 1 --2- 172.21.15.171:0/58467988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febfc0f5390 0x7febfc109640 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:30.889 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.886+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 >> 172.21.15.171:0/58467988 conn(0x7febfc0f0360 msgr2=0x7febfc0f0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:30.889 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.886+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 shutdown_connections 2022-01-31T19:38:30.889 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:30.886+0000 7fec02994700 1 -- 172.21.15.171:0/58467988 wait complete. 2022-01-31T19:38:30.890 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:31.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:31 smithi167 conmon[32206]: cluster 2022-01-31T19:38:30.689876+0000 mgr.smithi167.aciqpk (mgr.14156) 23 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:31.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:31 smithi167 conmon[32206]: audit 2022-01-31T19:38:30.884764+0000 mon.smithi167 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.171:0/58467988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:32.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:32.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:33.720 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 -- 172.21.15.171:0/688205426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 msgr2=0x7fdcc40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:33.721 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 --2- 172.21.15.171:0/688205426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc40f56f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fdcb8009fb0 tx=0x7fdcb8009b30).stop 2022-01-31T19:38:33.721 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 -- 172.21.15.171:0/688205426 shutdown_connections 2022-01-31T19:38:33.721 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 --2- 172.21.15.171:0/688205426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc40f56f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:33.721 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 -- 172.21.15.171:0/688205426 >> 172.21.15.171:0/688205426 conn(0x7fdcc40f0a30 msgr2=0x7fdcc40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:33.722 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 -- 172.21.15.171:0/688205426 shutdown_connections 2022-01-31T19:38:33.722 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.717+0000 7fdcc8d70700 1 -- 172.21.15.171:0/688205426 wait complete. 2022-01-31T19:38:33.722 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.718+0000 7fdcc8d70700 1 Processor -- start 2022-01-31T19:38:33.722 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.718+0000 7fdcc8d70700 1 -- start start 2022-01-31T19:38:33.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.718+0000 7fdcc8d70700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc4109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:33.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.718+0000 7fdcc8d70700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc4060320 con 0x7fdcc40f52d0 2022-01-31T19:38:33.724 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc37fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc4109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:33.724 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc37fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc4109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46242/0 (socket says 172.21.15.171:46242) 2022-01-31T19:38:33.724 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc37fe700 1 -- 172.21.15.171:0/1918765433 learned_addr learned my addr 172.21.15.171:0/1918765433 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:33.724 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc37fe700 1 -- 172.21.15.171:0/1918765433 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8005040 con 0x7fdcc40f52d0 2022-01-31T19:38:33.724 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc37fe700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc4109310 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fdcb80089f0 tx=0x7fdcb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:33.725 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcb8009320 con 0x7fdcc40f52d0 2022-01-31T19:38:33.725 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdcb8013ec0 con 0x7fdcc40f52d0 2022-01-31T19:38:33.725 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcb8014d30 con 0x7fdcc40f52d0 2022-01-31T19:38:33.725 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc4060520 con 0x7fdcc40f52d0 2022-01-31T19:38:33.726 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.719+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc41099c0 con 0x7fdcc40f52d0 2022-01-31T19:38:33.727 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.720+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7fdcb80263a0 con 0x7fdcc40f52d0 2022-01-31T19:38:33.727 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.720+0000 7fdcabfff700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdcac0310e0 0x7fdcac0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:33.727 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.720+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdcb8042520 con 0x7fdcc40f52d0 2022-01-31T19:38:33.728 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.720+0000 7fdcc2ffd700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdcac0310e0 0x7fdcac0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:33.728 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.720+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc40591e0 con 0x7fdcc40f52d0 2022-01-31T19:38:33.728 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.721+0000 7fdcc2ffd700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdcac0310e0 0x7fdcac0335a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdcb40096a0 tx=0x7fdcb4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:33.728 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.725+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdcb80266d0 con 0x7fdcc40f52d0 2022-01-31T19:38:33.931 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.930+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcc4045bb0 con 0x7fdcc40f52d0 2022-01-31T19:38:33.931 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.930+0000 7fdcabfff700 1 -- 172.21.15.171:0/1918765433 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdcb802b020 con 0x7fdcc40f52d0 2022-01-31T19:38:33.932 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:33.932 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:33.934 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdcac0310e0 msgr2=0x7fdcac0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:33.934 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdcac0310e0 0x7fdcac0335a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdcb40096a0 tx=0x7fdcb4006b40).stop 2022-01-31T19:38:33.934 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 msgr2=0x7fdcc4109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:33.934 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc4109310 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fdcb80089f0 tx=0x7fdcb800b040).stop 2022-01-31T19:38:33.935 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 shutdown_connections 2022-01-31T19:38:33.935 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdcac0310e0 0x7fdcac0335a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:33.935 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 --2- 172.21.15.171:0/1918765433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcc40f52d0 0x7fdcc4109310 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:33.935 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 >> 172.21.15.171:0/1918765433 conn(0x7fdcc40f0a30 msgr2=0x7fdcc40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:33.936 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 shutdown_connections 2022-01-31T19:38:33.936 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:33.932+0000 7fdcc8d70700 1 -- 172.21.15.171:0/1918765433 wait complete. 2022-01-31T19:38:33.936 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:34.010 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:33 smithi167 conmon[32206]: cluster 2022-01-31T19:38:32.690126+0000 mgr.smithi167.aciqpk (mgr.14156) 24 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:35.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:34 smithi167 conmon[32206]: audit 2022-01-31T19:38:33.824046+0000 mon.smithi167 (mon.0) 123 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:35.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:34 smithi167 conmon[32206]: audit 2022-01-31T19:38:33.931379+0000 mon.smithi167 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.171:0/1918765433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:35.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:35.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:36.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:35 smithi167 conmon[32206]: cluster 2022-01-31T19:38:34.690392+0000 mgr.smithi167.aciqpk (mgr.14156) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:36.812 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 -- 172.21.15.171:0/3040627864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 msgr2=0x7f97000f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:36.812 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 --2- 172.21.15.171:0/3040627864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f97000f69b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f96fc004760 tx=0x7f96fc009ad0).stop 2022-01-31T19:38:36.813 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 -- 172.21.15.171:0/3040627864 shutdown_connections 2022-01-31T19:38:36.813 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 --2- 172.21.15.171:0/3040627864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f97000f69b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:36.813 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 -- 172.21.15.171:0/3040627864 >> 172.21.15.171:0/3040627864 conn(0x7f97000f1cd0 msgr2=0x7f97000f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:36.813 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 -- 172.21.15.171:0/3040627864 shutdown_connections 2022-01-31T19:38:36.814 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.809+0000 7f9707c57700 1 -- 172.21.15.171:0/3040627864 wait complete. 2022-01-31T19:38:36.814 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f9707c57700 1 Processor -- start 2022-01-31T19:38:36.814 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f9707c57700 1 -- start start 2022-01-31T19:38:36.814 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f9707c57700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f970006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:36.814 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f9707c57700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9700069f00 con 0x7f97000f6590 2022-01-31T19:38:36.815 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f97059f3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f970006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:36.815 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f97059f3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f970006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46248/0 (socket says 172.21.15.171:46248) 2022-01-31T19:38:36.815 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.810+0000 7f97059f3700 1 -- 172.21.15.171:0/3357973833 learned_addr learned my addr 172.21.15.171:0/3357973833 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:36.815 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f97059f3700 1 -- 172.21.15.171:0/3357973833 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc009d70 con 0x7f97000f6590 2022-01-31T19:38:36.816 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f97059f3700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f970006ba80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f96fc004170 tx=0x7f96fc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:36.816 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96fc00e040 con 0x7f97000f6590 2022-01-31T19:38:36.816 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970006a0a0 con 0x7f97000f6590 2022-01-31T19:38:36.816 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970006a530 con 0x7f97000f6590 2022-01-31T19:38:36.816 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f96fc008ef0 con 0x7f97000f6590 2022-01-31T19:38:36.817 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.811+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96fc014c90 con 0x7f97000f6590 2022-01-31T19:38:36.818 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.812+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f96fc0283f0 con 0x7f97000f6590 2022-01-31T19:38:36.818 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.812+0000 7f96f67fc700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96ec0310e0 0x7f96ec0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:36.818 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.812+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f96fc043070 con 0x7f97000f6590 2022-01-31T19:38:36.819 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.812+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e8004fa0 con 0x7f97000f6590 2022-01-31T19:38:36.819 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.812+0000 7f97051f2700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96ec0310e0 0x7f96ec0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:36.819 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.813+0000 7f97051f2700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96ec0310e0 0x7f96ec0335a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f96f00096a0 tx=0x7f96f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:36.819 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:36.817+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f96fc0268d0 con 0x7f97000f6590 2022-01-31T19:38:37.037 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.036+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96e8005670 con 0x7f97000f6590 2022-01-31T19:38:37.038 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.036+0000 7f96f67fc700 1 -- 172.21.15.171:0/3357973833 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f96fc026300 con 0x7f97000f6590 2022-01-31T19:38:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:37.038 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:37.039 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96ec0310e0 msgr2=0x7f96ec0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:37.040 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96ec0310e0 0x7f96ec0335a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f96f00096a0 tx=0x7f96f0006b40).stop 2022-01-31T19:38:37.040 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 msgr2=0x7f970006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:37.040 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f970006ba80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f96fc004170 tx=0x7f96fc009ad0).stop 2022-01-31T19:38:37.040 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 shutdown_connections 2022-01-31T19:38:37.041 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96ec0310e0 0x7f96ec0335a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:37.041 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 --2- 172.21.15.171:0/3357973833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000f6590 0x7f970006ba80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:37.041 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 >> 172.21.15.171:0/3357973833 conn(0x7f97000f1cd0 msgr2=0x7f97000f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:37.041 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 shutdown_connections 2022-01-31T19:38:37.042 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:37.038+0000 7f9707c57700 1 -- 172.21.15.171:0/3357973833 wait complete. 2022-01-31T19:38:37.042 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:37.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: audit 2022-01-31T19:38:36.404736+0000 mon.smithi167 (mon.0 2022-01-31T19:38:37.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: ) 125 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:37.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: audit 2022-01-31T19:38:36.405570+0000 mon.smithi167 (mon.0) 126 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:38:37.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: audit 2022-01-31T19:38:36.406497+0000 mon.smithi167 (mon.0) 127 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:37.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: audit 2022-01-31T19:38:36.407052+0000 mon.smithi167 (mon.0) 128 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:38:37.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: audit 2022-01-31T19:38:36.954355+0000 mon.smithi167 (mon.0) 129 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:37.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:37 smithi167 conmon[32206]: audit 2022-01-31T19:38:37.037156+0000 mon.smithi167 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.171:0/3357973833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:38.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:38.375 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:38.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:38 smithi167 conmon[32206]: cluster 2022-01-31T19:38:36.691047+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:38:38.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:38 smithi167 conmon[32206]: 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:38.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:38 smithi167 conmon[32206]: audit 2022-01-31T19:38:37.588147+0000 mon.smithi167 (mon.0) 131 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:40.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:39 smithi167 conmon[32206]: cluster 2022-01-31T19:38:38.691304+0000 mgr.smithi167.aciqpk (mgr.14156) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:40.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:39 smithi167 conmon[32206]: audit 2022-01-31T19:38:38.829763+0000 mon.smithi167 (mon.0) 132 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:40.227 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.224+0000 7f22c9586700 1 -- 172.21.15.171:0/1455792863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 msgr2=0x7f22c40f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:40.227 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.224+0000 7f22c9586700 1 --2- 172.21.15.171:0/1455792863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c40f9060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f22b4009fb0 tx=0x7f22b4009b30).stop 2022-01-31T19:38:40.229 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.224+0000 7f22c9586700 1 -- 172.21.15.171:0/1455792863 shutdown_connections 2022-01-31T19:38:40.229 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.224+0000 7f22c9586700 1 --2- 172.21.15.171:0/1455792863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c40f9060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:40.229 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.224+0000 7f22c9586700 1 -- 172.21.15.171:0/1455792863 >> 172.21.15.171:0/1455792863 conn(0x7f22c40f0a40 msgr2=0x7f22c40f2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:40.229 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.225+0000 7f22c9586700 1 -- 172.21.15.171:0/1455792863 shutdown_connections 2022-01-31T19:38:40.230 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.225+0000 7f22c9586700 1 -- 172.21.15.171:0/1455792863 wait complete. 2022-01-31T19:38:40.230 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.225+0000 7f22c9586700 1 Processor -- start 2022-01-31T19:38:40.230 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.225+0000 7f22c9586700 1 -- start start 2022-01-31T19:38:40.230 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c9586700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c4106540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:40.231 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c9586700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c4106a30 con 0x7f22c40602c0 2022-01-31T19:38:40.231 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c3fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c4106540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:40.231 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c3fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c4106540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46254/0 (socket says 172.21.15.171:46254) 2022-01-31T19:38:40.231 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c3fff700 1 -- 172.21.15.171:0/3301090753 learned_addr learned my addr 172.21.15.171:0/3301090753 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:40.232 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c3fff700 1 -- 172.21.15.171:0/3301090753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b4005040 con 0x7f22c40602c0 2022-01-31T19:38:40.232 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c3fff700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c4106540 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f22b4009d00 tx=0x7f22b4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:40.232 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.226+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22b4018070 con 0x7f22c40602c0 2022-01-31T19:38:40.232 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.227+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f22b401d420 con 0x7f22c40602c0 2022-01-31T19:38:40.232 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.227+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c4106c30 con 0x7f22c40602c0 2022-01-31T19:38:40.233 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.227+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22b401c5a0 con 0x7f22c40602c0 2022-01-31T19:38:40.233 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.227+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c41070c0 con 0x7f22c40602c0 2022-01-31T19:38:40.234 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.227+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f22b401c810 con 0x7f22c40602c0 2022-01-31T19:38:40.234 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.228+0000 7f22c0ff9700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f22ac0310e0 0x7f22ac0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:40.235 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.228+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f22b4042860 con 0x7f22c40602c0 2022-01-31T19:38:40.235 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.228+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c4045bb0 con 0x7f22c40602c0 2022-01-31T19:38:40.235 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.228+0000 7f22c37fe700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f22ac0310e0 0x7f22ac0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:40.235 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.229+0000 7f22c37fe700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f22ac0310e0 0x7f22ac0335a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f22b80096a0 tx=0x7f22b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:40.236 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.232+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f22b401bdb0 con 0x7f22c40602c0 2022-01-31T19:38:40.453 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.451+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22c4107d40 con 0x7f22c40602c0 2022-01-31T19:38:40.453 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.451+0000 7f22c0ff9700 1 -- 172.21.15.171:0/3301090753 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f22b401b370 con 0x7f22c40602c0 2022-01-31T19:38:40.454 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:40.454 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f22ac0310e0 msgr2=0x7f22ac0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f22ac0310e0 0x7f22ac0335a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f22b80096a0 tx=0x7f22b8006b40).stop 2022-01-31T19:38:40.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 msgr2=0x7f22c4106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:40.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c4106540 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f22b4009d00 tx=0x7f22b4009b30).stop 2022-01-31T19:38:40.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 shutdown_connections 2022-01-31T19:38:40.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f22ac0310e0 0x7f22ac0335a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:40.457 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 --2- 172.21.15.171:0/3301090753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22c40602c0 0x7f22c4106540 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:40.458 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 >> 172.21.15.171:0/3301090753 conn(0x7f22c40f0a40 msgr2=0x7f22c40f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:40.458 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 shutdown_connections 2022-01-31T19:38:40.458 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:40.453+0000 7f22c9586700 1 -- 172.21.15.171:0/3301090753 wait complete. 2022-01-31T19:38:40.458 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:41.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19:38:40.217170+0000 mon.smithi167 (mon.0) 133 : audit 2022-01-31T19:38:41.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:41.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19:38:40.453319+0000 mon.smithi167 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.171:0/3301090753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:41.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19:38 2022-01-31T19:38:41.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: :40.614000+0000 mon.smithi167 (mon.0) 135 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:41.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19:38:41.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 2022-01-31T19:38:40.614575+0000 mon.smithi167 (mon.0 2022-01-31T19:38:41.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: ) 136 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:38:41.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:38:41.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: :38:41.072016+0000 mon.smithi167 (mon.0) 2022-01-31T19:38:41.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 137 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:41.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:38:41.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: :38:41.120190+0000 mon.smithi167 (mon.0) 2022-01-31T19:38:41.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 138 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:41.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:38:41.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 19:38:41.122464+0000 mon.smithi167 (mon.0) 2022-01-31T19:38:41.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 139 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:41.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:38:41.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: :38:41.123789+0000 mon.smithi167 (mon.0) 2022-01-31T19:38:41.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 140 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi171", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2022-01-31T19:38:41.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19:38:41 2022-01-31T19:38:41.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: .125370+0000 mon.smithi167 (mon.0) 141 : audit [INF] 2022-01-31T19:38:41.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi171", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2022-01-31T19:38:41.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: audit 2022-01-31T19:38:41. 2022-01-31T19:38:41.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: 125976+0000 mon.smithi167 (mon.0) 142 : audit [DBG] 2022-01-31T19:38:41.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:41 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:41.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:41.978 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:42.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:42 smithi167 conmon[32206]: cluster 2022-01-31T19:38:40.691577+0000 mgr.smithi167.aciqpk (mgr.14156) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:42.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:41.072444+0000 mgr.smithi167.aciqpk (mgr.14156) 29 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:38:42.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:41.100862+0000 mgr.smithi167.aciqpk (mgr.14156) 30 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:38:42.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:41. 2022-01-31T19:38:42.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:42 smithi167 conmon[32206]: 126449+0000 mgr.smithi167.aciqpk (mgr.14156) 31 : cephadm [INF] Deploying daemon crash.smithi171 on smithi171 2022-01-31T19:38:43.672 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 -- 172.21.15.171:0/3290169087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 msgr2=0x7efe040f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:43.672 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 --2- 172.21.15.171:0/3290169087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe040f58a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7efdec009fb0 tx=0x7efdec009b30).stop 2022-01-31T19:38:43.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 -- 172.21.15.171:0/3290169087 shutdown_connections 2022-01-31T19:38:43.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 --2- 172.21.15.171:0/3290169087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe040f58a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:43.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 -- 172.21.15.171:0/3290169087 >> 172.21.15.171:0/3290169087 conn(0x7efe040f0be0 msgr2=0x7efe040f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:43.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 -- 172.21.15.171:0/3290169087 shutdown_connections 2022-01-31T19:38:43.673 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 -- 172.21.15.171:0/3290169087 wait complete. 2022-01-31T19:38:43.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.668+0000 7efe09b12700 1 Processor -- start 2022-01-31T19:38:43.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe09b12700 1 -- start start 2022-01-31T19:38:43.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe09b12700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe04109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:43.674 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe09b12700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04060300 con 0x7efe040f5480 2022-01-31T19:38:43.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe04109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:43.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe04109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46260/0 (socket says 172.21.15.171:46260) 2022-01-31T19:38:43.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe037fe700 1 -- 172.21.15.171:0/2081100973 learned_addr learned my addr 172.21.15.171:0/2081100973 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:43.675 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.669+0000 7efe037fe700 1 -- 172.21.15.171:0/2081100973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec005040 con 0x7efe040f5480 2022-01-31T19:38:43.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.670+0000 7efe037fe700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe04109510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7efdec0089f0 tx=0x7efdec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:43.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.670+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdec009320 con 0x7efe040f5480 2022-01-31T19:38:43.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.670+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efdec013ec0 con 0x7efe040f5480 2022-01-31T19:38:43.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.670+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe04060500 con 0x7efe040f5480 2022-01-31T19:38:43.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.670+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdec014d30 con 0x7efe040f5480 2022-01-31T19:38:43.677 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.670+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe04109bc0 con 0x7efe040f5480 2022-01-31T19:38:43.678 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.671+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7efdec0095f0 con 0x7efe040f5480 2022-01-31T19:38:43.678 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.671+0000 7efdfbfff700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efdf0031090 0x7efdf0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:43.678 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.671+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efdec0439e0 con 0x7efe040f5480 2022-01-31T19:38:43.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.671+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe040591e0 con 0x7efe040f5480 2022-01-31T19:38:43.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.671+0000 7efe02ffd700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efdf0031090 0x7efdf0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:43.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.672+0000 7efe02ffd700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efdf0031090 0x7efdf0033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7efdf40096a0 tx=0x7efdf4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:43.679 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.676+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efdec01b740 con 0x7efe040f5480 2022-01-31T19:38:43.897 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.895+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe04045bb0 con 0x7efe040f5480 2022-01-31T19:38:43.898 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.896+0000 7efdfbfff700 1 -- 172.21.15.171:0/2081100973 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe04045bb0 con 0x7efe040f5480 2022-01-31T19:38:43.899 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:43.899 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:43.900 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efdf0031090 msgr2=0x7efdf0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:43.900 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efdf0031090 0x7efdf0033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7efdf40096a0 tx=0x7efdf4006b40).stop 2022-01-31T19:38:43.901 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 msgr2=0x7efe04109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:43.901 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe04109510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7efdec0089f0 tx=0x7efdec00b040).stop 2022-01-31T19:38:43.901 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 shutdown_connections 2022-01-31T19:38:43.901 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efdf0031090 0x7efdf0033550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:43.902 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 --2- 172.21.15.171:0/2081100973 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe040f5480 0x7efe04109510 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:43.902 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 >> 172.21.15.171:0/2081100973 conn(0x7efe040f0be0 msgr2=0x7efe040f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:43.902 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 shutdown_connections 2022-01-31T19:38:43.902 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:43.898+0000 7efe09b12700 1 -- 172.21.15.171:0/2081100973 wait complete. 2022-01-31T19:38:43.903 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:44.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: cluster 2022-01-31T19:38:42.691914+0000 mgr.smithi167.aciqpk (mgr.14156) 32 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:44.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43.717414+0000 mon.smithi167 ( 2022-01-31T19:38:44.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: mon.0) 143 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:44.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43.719562+0000 mon.smithi167 (mon.0) 144 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:38:44.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43.722474+0000 mon.smithi167 (mon.0) 145 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-01-31T19:38:44.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43.723310+0000 mon.smithi167 (mon.0) 146 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:38:44.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43.724211+0000 mon.smithi167 (mon.0) 147 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:44.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43.833268+0000 mon.smithi167 (mon.0) 148 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:44.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: audit 2022-01-31T19:38:43. 2022-01-31T19:38:44.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: 898033+0000 mon.smithi167 (mon.0) 149 : audit [DBG] 2022-01-31T19:38:44.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:44 smithi167 conmon[32206]: from='client.? 172.21.15.171:0/2081100973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:45.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:45.348 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:45.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:45 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:43.725003+0000 mgr.smithi167.aciqpk (mgr.14156) 33 : cephadm [INF] Deploying daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:38:46.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: cluster 2022-01-31T19:38:44.692332+0000 mgr.smithi167.aciqpk (mgr.14156) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:46.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: 2022-01-31T19:38:46.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: audit 2022-01-31T19:38:45.742956+0000 mon.smithi167 (mon.0) 150 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:46.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: audit 2022-01-31T19:38:45.745150+0000 2022-01-31T19:38:46.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: mon.smithi167 (mon.0) 151 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:38:46.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:38:46.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: 19:38:45.746159+0000 mon.smithi167 (mon.0) 152 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:46.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:46 smithi167 conmon[32206]: 2022-01-31T19:38:47.201 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 -- 172.21.15.171:0/2544664483 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 msgr2=0x7f01740f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:47.202 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 --2- 172.21.15.171:0/2544664483 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f01740f91c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0170009fb0 tx=0x7f0170009b30).stop 2022-01-31T19:38:47.202 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 -- 172.21.15.171:0/2544664483 shutdown_connections 2022-01-31T19:38:47.202 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 --2- 172.21.15.171:0/2544664483 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f01740f91c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:47.202 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 -- 172.21.15.171:0/2544664483 >> 172.21.15.171:0/2544664483 conn(0x7f01740f0be0 msgr2=0x7f01740f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:47.203 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 -- 172.21.15.171:0/2544664483 shutdown_connections 2022-01-31T19:38:47.203 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.197+0000 7f017c4e2700 1 -- 172.21.15.171:0/2544664483 wait complete. 2022-01-31T19:38:47.203 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017c4e2700 1 Processor -- start 2022-01-31T19:38:47.203 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017c4e2700 1 -- start start 2022-01-31T19:38:47.203 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017c4e2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f0174106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:47.204 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017c4e2700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0174106c70 con 0x7f0174060300 2022-01-31T19:38:47.204 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017a27e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f0174106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:47.204 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017a27e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f0174106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46270/0 (socket says 172.21.15.171:46270) 2022-01-31T19:38:47.204 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.198+0000 7f017a27e700 1 -- 172.21.15.171:0/2098087543 learned_addr learned my addr 172.21.15.171:0/2098087543 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:47.205 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f017a27e700 1 -- 172.21.15.171:0/2098087543 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0170005040 con 0x7f0174060300 2022-01-31T19:38:47.205 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f017a27e700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f0174106780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0170009f80 tx=0x7f017000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:47.205 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0170009320 con 0x7f0174060300 2022-01-31T19:38:47.205 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0170013680 con 0x7f0174060300 2022-01-31T19:38:47.205 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0174106e70 con 0x7f0174060300 2022-01-31T19:38:47.206 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f017001eda0 con 0x7f0174060300 2022-01-31T19:38:47.206 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.199+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174107300 con 0x7f0174060300 2022-01-31T19:38:47.206 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.200+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 13) v1 ==== 37152+0+0 (secure 0 0 0) 0x7f0170014420 con 0x7f0174060300 2022-01-31T19:38:47.207 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.200+0000 7f016affd700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0160031090 0x7f0160033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:47.207 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.200+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f01700432a0 con 0x7f0174060300 2022-01-31T19:38:47.208 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.200+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174045bb0 con 0x7f0174060300 2022-01-31T19:38:47.208 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.200+0000 7f0179a7d700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0160031090 0x7f0160033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:47.208 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.201+0000 7f0179a7d700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0160031090 0x7f0160033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f01640096a0 tx=0x7f0164006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:47.208 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.205+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0170014700 con 0x7f0174060300 2022-01-31T19:38:47.430 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.428+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0174107f80 con 0x7f0174060300 2022-01-31T19:38:47.431 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.429+0000 7f016affd700 1 -- 172.21.15.171:0/2098087543 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f017001c320 con 0x7f0174060300 2022-01-31T19:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:37:04.335382Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-01-31T19:38:47.432 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0160031090 msgr2=0x7f0160033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:47.433 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0160031090 0x7f0160033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f01640096a0 tx=0x7f0164006b40).stop 2022-01-31T19:38:47.433 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 msgr2=0x7f0174106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:47.433 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f0174106780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0170009f80 tx=0x7f017000b040).stop 2022-01-31T19:38:47.433 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 shutdown_connections 2022-01-31T19:38:47.434 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0160031090 0x7f0160033550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:47.434 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 --2- 172.21.15.171:0/2098087543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0174060300 0x7f0174106780 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:47.434 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 >> 172.21.15.171:0/2098087543 conn(0x7f01740f0be0 msgr2=0x7f01740f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:47.434 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 shutdown_connections 2022-01-31T19:38:47.435 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:47.431+0000 7f017c4e2700 1 -- 172.21.15.171:0/2098087543 wait complete. 2022-01-31T19:38:47.435 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2022-01-31T19:38:47.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:47 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:45.746996+0000 mgr.smithi167.aciqpk (mgr.14156) 35 : cephadm [INF] Deploying daemon mon.smithi171 on smithi171 2022-01-31T19:38:48.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:48 smithi167 conmon[32206]: cluster 2022-01-31T19:38:46.692707+0000 mgr.smithi167.aciqpk (mgr.14156) 36 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:48.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:48 smithi167 conmon[32206]: audit 2022-01-31T19:38:47.430664+0000 mon.smithi167 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.171:0/2098087543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:48.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2022-01-31T19:38:48.798 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mon dump -f json 2022-01-31T19:38:48.828 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:48 smithi171 systemd[1]: Starting Ceph mon.smithi171 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:38:49.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:48 smithi171 podman[35265]: 2022-01-31T19:38:49.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.035+0000 7fb497bca700 0 set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:38:49.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.035+0000 7fb497bca700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2022-01-31T19:38:49.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.035+0000 7fb497bca700 0 pidfile_write: ignore empty --pid-file 2022-01-31T19:38:49.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 bash[35188]: 8f865af70de616fa3fca2fa84f7b5df3be83dd6053dc9051b340d0e15d0aa098 2022-01-31T19:38:49.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.039+0000 7fb497bca700 0 load: jerasure load: lrc load: isa 2022-01-31T19:38:49.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: RocksDB version: 6.8.1 2022-01-31T19:38:49.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:38:49.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Compile date Jul 8 2021 2022-01-31T19:38:49.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: DB SUMMARY 2022-01-31T19:38:49.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: CURRENT file: CURRENT 2022-01-31T19:38:49.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:38:49.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2022-01-31T19:38:49.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi171/store.db dir, Total Num: 0, files: 2022-01-31T19:38:49.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi171/store.db: 000003.log size: 511 ; 2022-01-31T19:38:49.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.error_if_exists: 0 2022-01-31T19:38:49.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.create_if_missing: 0 2022-01-31T19:38:49.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.paranoid_checks: 1 2022-01-31T19:38:49.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.env: 0x55a4715f31c0 2022-01-31T19:38:49.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.fs: Posix File System 2022-01-31T19:38:49.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.info_log: 0x55a4727cba40 2022-01-31T19:38:49.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:38:49.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.statistics: (nil) 2022-01-31T19:38:49.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.use_fsync: 0 2022-01-31T19:38:49.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_log_file_size: 0 2022-01-31T19:38:49.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:38:49.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:38:49.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:38:49.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:38:49.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.allow_fallocate: 1 2022-01-31T19:38:49.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:38:49.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:38:49.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.use_direct_reads: 0 2022-01-31T19:38:49.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:38:49.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:38:49.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.db_log_dir: 2022-01-31T19:38:49.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi171/store.db 2022-01-31T19:38:49.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:38:49.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_subcompactions: 1 2022-01-31T19:38:49.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_background_flushes: -1 2022-01-31T19:38:49.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:38:49.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:38:49.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:38:49.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:38:49.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:38:49.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:38:49.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:38:49.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.write_buffer_manager: 0x55a4727d3fb0 2022-01-31T19:38:49.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:38:49.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:38:49.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:38:49.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:38:49.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:38:49.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:38:49.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:38:49.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:38:49.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:38:49.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.unordered_write: 0 2022-01-31T19:38:49.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:38:49.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:38:49.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:38:49.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:38:49.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.row_cache: None 2022-01-31T19:38:49.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.wal_filter: None 2022-01-31T19:38:49.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:38:49.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:38:49.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.preserve_deletes: 0 2022-01-31T19:38:49.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.two_write_queues: 0 2022-01-31T19:38:49.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:38:49.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.atomic_flush: 0 2022-01-31T19:38:49.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:38:49.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:38:49.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:38:49.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.log_readahead_size: 0 2022-01-31T19:38:49.129 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2022-01-31T19:38:49.129 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_background_jobs: 2 2022-01-31T19:38:49.129 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_background_compactions: -1 2022-01-31T19:38:49.129 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:38:49.130 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:38:49.130 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:38:49.130 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:38:49.130 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:38:49.130 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:38:49.131 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:38:49.131 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:38:49.131 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_open_files: -1 2022-01-31T19:38:49.131 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:38:49.132 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:38:49.132 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:38:49.132 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:38:49.132 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Compression algorithms supported: 2022-01-31T19:38:49.132 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:38:49.133 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kZSTD supported: 0 2022-01-31T19:38:49.133 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kXpressCompression supported: 0 2022-01-31T19:38:49.133 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:38:49.133 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kLZ4Compression supported: 1 2022-01-31T19:38:49.134 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kBZip2Compression supported: 0 2022-01-31T19:38:49.134 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kZlibCompression supported: 1 2022-01-31T19:38:49.134 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: kSnappyCompression supported: 1 2022-01-31T19:38:49.134 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.134 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:38:49.135 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi171/store.db/MANIFEST-000001 2022-01-31T19:38:49.135 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.135 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2022-01-31T19:38:49.135 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.136 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:38:49.136 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.merge_operator: 2022-01-31T19:38:49.136 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_filter: None 2022-01-31T19:38:49.136 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:38:49.137 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:38:49.137 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:38:49.137 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a4726e5cc0) 2022-01-31T19:38:49.137 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: cache_index_and_filter_blocks: 1 2022-01-31T19:38:49.137 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:38:49.138 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:38:49.138 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: pin_top_level_index_and_filter: 1 2022-01-31T19:38:49.138 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: index_type: 0 2022-01-31T19:38:49.138 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: data_block_index_type: 0 2022-01-31T19:38:49.139 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: index_shortening: 1 2022-01-31T19:38:49.139 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:38:49.139 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: hash_index_allow_collision: 1 2022-01-31T19:38:49.139 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: checksum: 1 2022-01-31T19:38:49.139 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: no_block_cache: 0 2022-01-31T19:38:49.140 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_cache: 0x55a47271af10 2022-01-31T19:38:49.140 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_cache_name: BinnedLRUCache 2022-01-31T19:38:49.140 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_cache_options: 2022-01-31T19:38:49.140 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: capacity : 536870912 2022-01-31T19:38:49.141 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: num_shard_bits : 4 2022-01-31T19:38:49.141 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: strict_capacity_limit : 0 2022-01-31T19:38:49.141 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: high_pri_pool_ratio: 0.000 2022-01-31T19:38:49.141 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_cache_compressed: (nil) 2022-01-31T19:38:49.141 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: persistent_cache: (nil) 2022-01-31T19:38:49.142 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_size: 4096 2022-01-31T19:38:49.142 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_size_deviation: 10 2022-01-31T19:38:49.142 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_restart_interval: 16 2022-01-31T19:38:49.142 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: index_block_restart_interval: 1 2022-01-31T19:38:49.143 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: metadata_block_size: 4096 2022-01-31T19:38:49.143 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: partition_filters: 0 2022-01-31T19:38:49.143 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: use_delta_encoding: 1 2022-01-31T19:38:49.143 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:38:49.143 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: whole_key_filtering: 1 2022-01-31T19:38:49.144 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: verify_compression: 0 2022-01-31T19:38:49.144 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: read_amp_bytes_per_bit: 0 2022-01-31T19:38:49.144 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: format_version: 2 2022-01-31T19:38:49.144 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: enable_index_compression: 1 2022-01-31T19:38:49.145 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: block_align: 0 2022-01-31T19:38:49.145 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.145 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:38:49.145 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:38:49.145 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression: NoCompression 2022-01-31T19:38:49.146 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:38:49.146 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:38:49.146 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:38:49.146 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.num_levels: 7 2022-01-31T19:38:49.147 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.147 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:38:49.147 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:38:49.147 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:38:49.148 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:38:49.148 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:38:49.148 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:38:49.148 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:38:49.148 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:38:49.149 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:38:49.149 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:38:49.149 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:38:49.149 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:38:49.150 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:38:49.150 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:38:49.150 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:38:49.150 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:38:49.150 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:38:49.151 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:38:49.151 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:38:49.151 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:38:49.151 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:38:49.152 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:38:49.152 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:38:49.152 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:38:49.152 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:38:49.152 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:38:49.153 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:38:49.153 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:38:49.153 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:38:49.153 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:38:49.154 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:38:49.154 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:38:49.154 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:38:49.154 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:38:49.154 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:38:49.155 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:38:49.155 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:38:49.155 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:38:49.155 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:38:49.156 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:38:49.156 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:38:49.156 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:38:49.156 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:38:49.156 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:38:49.157 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:38:49.157 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:38:49.157 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:38:49.157 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.table_properties_collectors: 2022-01-31T19:38:49.158 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.inplace_update_support: 0 2022-01-31T19:38:49.158 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:38:49.158 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:38:49.158 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:38:49.159 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.040+0000 7fb497bca700 4 rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:38:49.159 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.bloom_locality: 0 2022-01-31T19:38:49.159 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.max_successive_merges: 0 2022-01-31T19:38:49.159 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:38:49.159 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:38:49.160 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.force_consistency_checks: 0 2022-01-31T19:38:49.160 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:38:49.160 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.ttl: 2592000 2022-01-31T19:38:49.160 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:38:49.161 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi171/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:38:49.161 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.161 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2022-01-31T19:38:49.161 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.161 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1643657929043397, "job": 1, "event": "recovery_started", "log_files": [3]} 2022-01-31T19:38:49.162 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.162 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.041+0000 7fb497bca700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2022-01-31T19:38:49.162 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.041+0000 7fb497bca700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-01-31T19:38:49.162 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.042+0000 7fb497bca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1643657929043882, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1643657929, "oldest_key_time": 3, "file_creation_time": 0}} 2022-01-31T19:38:49.163 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.042+0000 7fb497bca700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2022-01-31T19:38:49.163 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.163 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.042+0000 7fb497bca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1643657929044259, "job": 1, "event": "recovery_finished"} 2022-01-31T19:38:49.164 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.043+0000 7fb497bca700 4 rocksdb: DB pointer 0x55a4727df800 2022-01-31T19:38:49.164 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.043+0000 7fb47e9d2700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2022-01-31T19:38:49.165 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.043+0000 7fb47e9d2700 4 rocksdb: [db_impl/db_impl.cc:851] 2022-01-31T19:38:49.165 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** DB Stats ** 2022-01-31T19:38:49.165 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:38:49.165 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:38:49.166 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:38:49.166 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:38:49.166 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:38:49.166 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:38:49.166 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:38:49.167 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.167 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** Compaction Stats [default] ** 2022-01-31T19:38:49.167 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:38:49.167 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:38:49.168 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.168 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.168 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.168 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.169 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** Compaction Stats [default] ** 2022-01-31T19:38:49.169 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:38:49.169 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:38:49.169 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.169 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:38:49.170 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:38:49.170 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:38:49.170 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:38:49.170 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:38:49.171 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:38:49.171 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:38:49.171 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:38:49.172 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:38:49.172 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.172 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:38:49.172 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.173 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** Compaction Stats [default] ** 2022-01-31T19:38:49.173 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:38:49.173 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:38:49.173 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.173 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.174 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:38:49.174 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.174 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** Compaction Stats [default] ** 2022-01-31T19:38:49.174 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:38:49.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:38:49.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2022-01-31T19:38:49.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:38:49.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:38:49.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:38:49.176 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:38:49.176 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:38:49.176 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:38:49.176 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:38:49.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:38:49.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:38:49.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:38:49.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.178 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.043+0000 7fb497bca700 0 mon.smithi171 does not exist in monmap, will attempt to join an existing cluster 2022-01-31T19:38:49.178 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.043+0000 7fb497bca700 0 using public_addr v2:172.21.15.171:0/0 -> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2022-01-31T19:38:49.178 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.044+0000 7fb497bca700 0 starting mon.smithi171 rank -1 at public addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] at bind addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi171 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:38:49.178 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.045+0000 7fb497bca700 1 mon.smithi171@-1(???) e0 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:38:49.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 0 mon.smithi171@-1(synchronizing).mds e1 new map 2022-01-31T19:38:49.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 0 mon.smithi171@-1(synchronizing).mds e1 print_map 2022-01-31T19:38:49.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: e1 2022-01-31T19:38:49.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:38:49.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:38:49.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: legacy client fscid: -1 2022-01-31T19:38:49.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: No filesystems configured 2022-01-31T19:38:49.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-01-31T19:38:49.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-01-31T19:38:49.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-01-31T19:38:49.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-01-31T19:38:49.182 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-01-31T19:38:49.182 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-01-31T19:38:49.182 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 0 mon.smithi171@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-01-31T19:38:49.182 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.066+0000 7fb4811d7700 0 mon.smithi171@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-01-31T19:38:49.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 0 mon.smithi171@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-01-31T19:38:49.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.066+0000 7fb4811d7700 0 mon.smithi171@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-01-31T19:38:49.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: cluster 2022-01-31T19:38:49.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:46 2022-01-31T19:38:49.184 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: .692707+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:38:49.184 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 36 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:49.184 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:38:49.184 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 01-31T19:38:47.430664+0000 2022-01-31T19:38:49.185 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.171:0/2098087543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:49.185 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: 2022-01-31T19:38:49.185 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 conmon[35325]: debug 2022-01-31T19:38:49.067+0000 7fb4811d7700 1 mon.smithi171@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2022-01-31T19:38:49.396 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:49 smithi171 systemd[1]: Started Ceph mon.smithi171 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:38:54.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022-01-31T19:38:48.693015+0000 mgr.smithi167.aciqpk (mgr.14156) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:54.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022-01-31T19:38:49.075608+0000 mon.smithi167 (mon.0) 156 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:38:54.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022-01-31T19:38:49.075785+0000 mon.smithi167 (mon.0) 157 : cluster [INF] mon.smithi167 calling monitor election 2022-01-31T19:38:54.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022-01-31T19:38:49.088535+0000 2022-01-31T19:38:54.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 158 : audit [DBG] 2022-01-31T19:38:54.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022-01-31T19:38:54.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 2022-01-31T19:38:50 2022-01-31T19:38:54.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: .071989+0000 mon.smithi167 (mon.0) 159 2022-01-31T19:38:54.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022-01 2022-01-31T19:38:54.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -31T19:38:50.693368+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:38:54.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: .14156) 38 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:54.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:38:54.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:51.072144+0000 2022-01-31T19:38:54.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 160 : audit 2022-01-31T19:38:54.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022-01 2022-01-31T19:38:54.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -31T19:38:51.073330+0000 2022-01-31T19:38:54.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi171 (mon.1) 1 : cluster [INF] 2022-01-31T19:38:54.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi171 calling monitor election 2022-01-31T19:38:54.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022 2022-01-31T19:38:54.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:51.097623 2022-01-31T19:38:54.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 161 : audit 2022-01-31T19:38:54.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:38:54.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:38:54.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:52.072597+0000 2022-01-31T19:38:54.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 162 : audit [DBG] 2022-01-31T19:38:54.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022- 2022-01-31T19:38:54.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:52.693708 2022-01-31T19:38:54.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 39 : cluster 2022-01-31T19:38:54.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:54.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:38:54.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:53.072431+0000 2022-01-31T19:38:54.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 163 : audit [DBG] 2022-01-31T19:38:54.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:38:54.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:54.072779 2022-01-31T19:38:54.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 164 : audit 2022-01-31T19:38:54.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022 2022-01-31T19:38:54.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:54.102968 2022-01-31T19:38:54.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 165 : cluster 2022-01-31T19:38:54.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [INF] mon.smithi167 is new leader, mons smithi167,smithi171 in quorum (ranks 0,1) 2022-01-31T19:38:54.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022- 2022-01-31T19:38:54.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:54.106022 2022-01-31T19:38:54.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 166 : cluster 2022-01-31T19:38:54.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] monmap e2: 2 mons at {smithi167=[v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0],smithi171=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0]} 2022-01-31T19:38:54.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022 2022-01-31T19:38:54.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:54.106091+0000 2022-01-31T19:38:54.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 167 : cluster 2022-01-31T19:38:54.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] fsmap 2022-01-31T19:38:54.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022 2022-01-31T19:38:54.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:54.106124+0000 2022-01-31T19:38:54.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 168 : cluster 2022-01-31T19:38:54.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-01-31T19:38:54.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022- 2022-01-31T19:38:54.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:54.106355+0000 2022-01-31T19:38:54.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 169 : cluster [DBG] 2022-01-31T19:38:54.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mgrmap e13: smithi167.aciqpk(active, since 55s) 2022-01-31T19:38:54.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022 2022-01-31T19:38:54.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:54.107441 2022-01-31T19:38:54.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 170 : 2022-01-31T19:38:54.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster [DBG] Standby manager daemon smithi171.asyxnc started 2022-01-31T19:38:54.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022 2022-01-31T19:38:54.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:54.108531+0000 2022-01-31T19:38:54.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 171 : audit 2022-01-31T19:38:54.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:38:54.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: cluster 2022- 2022-01-31T19:38:54.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:54.110238+0000 2022-01-31T19:38:54.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 172 : cluster 2022-01-31T19:38:54.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [INF] overall HEALTH_OK 2022-01-31T19:38:54.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022 2022-01-31T19:38:54.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: -01-31T19:38:54. 2022-01-31T19:38:54.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 110679+0000 mon.smithi167 (mon.0) 173 2022-01-31T19:38:54.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: : audit [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:38:54.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:38:54.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:54.111068+0000 2022-01-31T19:38:54.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 174 : audit 2022-01-31T19:38:54.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:38:54.439 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:38:54.439 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: 01-31T19:38:54.112704+0000 2022-01-31T19:38:54.439 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: mon.smithi167 (mon.0) 175 : audit 2022-01-31T19:38:54.439 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:54 smithi167 conmon[32206]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:54.476 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:48.693015+0000 mgr.smithi167.aciqpk (mgr.14156) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:54.476 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:49.075608+0000 mon.smithi167 (mon.0) 156 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:38:54.477 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:49.075785+0000 mon.smithi167 (mon.0) 157 : cluster [INF] mon.smithi167 calling monitor election 2022-01-31T19:38:54.477 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:49.088535+0000 mon.smithi167 (mon.0) 158 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.477 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:50.071989+0000 mon.smithi167 (mon.0) 159 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.477 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:50.693368+0000 mgr.smithi167.aciqpk (mgr.14156) 38 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:54.477 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:51.072144+0000 mon.smithi167 (mon.0) 160 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.478 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:51.073330+0000 mon.smithi171 (mon.1) 1 : cluster [INF] mon.smithi171 calling monitor election 2022-01-31T19:38:54.478 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:51.097623+0000 mon.smithi167 (mon.0) 161 : audit [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:38:54.478 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:52.072597+0000 mon.smithi167 (mon.0) 162 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.478 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:52.693708+0000 mgr.smithi167.aciqpk (mgr.14156) 39 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:54.479 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:53.072431+0000 mon.smithi167 (mon.0) 163 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.479 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.072779+0000 mon.smithi167 (mon.0) 164 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:54.479 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.102968+0000 mon.smithi167 (mon.0) 165 : cluster [INF] mon.smithi167 is new leader, mons smithi167,smithi171 in quorum (ranks 0,1) 2022-01-31T19:38:54.479 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.106022+0000 mon.smithi167 (mon.0) 166 : cluster [DBG] monmap e2: 2 mons at {smithi167=[v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0],smithi171=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0]} 2022-01-31T19:38:54.480 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.106091+0000 mon.smithi167 (mon.0) 167 : cluster [DBG] fsmap 2022-01-31T19:38:54.480 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.106124+0000 mon.smithi167 (mon.0) 168 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2022-01-31T19:38:54.480 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.106355+0000 mon.smithi167 (mon.0) 169 : cluster [DBG] mgrmap e13: smithi167.aciqpk(active, since 55s) 2022-01-31T19:38:54.480 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.107441+0000 mon.smithi167 (mon.0) 170 : cluster [DBG] Standby manager daemon smithi171.asyxnc started 2022-01-31T19:38:54.480 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.108531+0000 mon.smithi167 (mon.0) 171 : audit [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:38:54.481 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.110238+0000 mon.smithi167 (mon.0) 172 : cluster [INF] overall HEALTH_OK 2022-01-31T19:38:54.481 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.110679+0000 mon.smithi167 (mon.0) 173 : audit [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:38:54.481 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.111068+0000 mon.smithi167 (mon.0) 174 : audit [DBG] from='mgr.? 172.21.15.171:0/127251532' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:38:54.481 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:54 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.112704+0000 mon.smithi167 (mon.0) 175 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:54.713 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 -- 172.21.15.171:0/1424118067 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4174003250 msgr2=0x7f41740056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:54.713 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 --2- 172.21.15.171:0/1424118067 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4174003250 0x7f41740056e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f418c0f6090 tx=0x7f417c002040).stop 2022-01-31T19:38:54.714 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 -- 172.21.15.171:0/1424118067 shutdown_connections 2022-01-31T19:38:54.714 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 --2- 172.21.15.171:0/1424118067 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4174003250 0x7f41740056e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:54.714 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 -- 172.21.15.171:0/1424118067 >> 172.21.15.171:0/1424118067 conn(0x7f418c0f0a40 msgr2=0x7f418c0f2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:54.714 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 -- 172.21.15.171:0/1424118067 shutdown_connections 2022-01-31T19:38:54.714 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.709+0000 7f418b59e700 1 -- 172.21.15.171:0/1424118067 wait complete. 2022-01-31T19:38:54.715 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.710+0000 7f418b59e700 1 Processor -- start 2022-01-31T19:38:54.715 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.710+0000 7f418b59e700 1 -- start start 2022-01-31T19:38:54.715 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418b59e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:54.715 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418b59e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f418c104890 0x7f418c105780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:54.716 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418b59e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c104df0 con 0x7f418c104890 2022-01-31T19:38:54.716 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418b59e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c104f30 con 0x7f4174003250 2022-01-31T19:38:54.716 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:54.716 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:49130/0 (socket says 172.21.15.171:49130) 2022-01-31T19:38:54.717 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 -- 172.21.15.171:0/222688893 learned_addr learned my addr 172.21.15.171:0/222688893 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:38:54.717 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f4189d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f418c104890 0x7f418c105780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:54.718 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 -- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 msgr2=0x7f418c1043a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-01-31T19:38:54.718 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 -- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 msgr2=0x7f418c1043a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-01-31T19:38:54.718 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-01-31T19:38:54.719 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f418a59c700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-01-31T19:38:54.719 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f4189d9b700 1 -- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 msgr2=0x7f418c1043a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-01-31T19:38:54.719 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f4189d9b700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:54.719 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.711+0000 7f4189d9b700 1 -- 172.21.15.171:0/222688893 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c005040 con 0x7f418c104890 2022-01-31T19:38:54.719 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.712+0000 7f4189d9b700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f418c104890 0x7f418c105780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f4180009390 tx=0x7f4180006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:54.720 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.712+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41800115e0 con 0x7f418c104890 2022-01-31T19:38:54.720 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.712+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4180011bd0 con 0x7f418c104890 2022-01-31T19:38:54.720 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.712+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c105d90 con 0x7f418c104890 2022-01-31T19:38:54.720 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.712+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4180010b50 con 0x7f418c104890 2022-01-31T19:38:54.721 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.712+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c106350 con 0x7f418c104890 2022-01-31T19:38:54.722 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.713+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c0591e0 con 0x7f418c104890 2022-01-31T19:38:54.722 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.713+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f4180018030 con 0x7f418c104890 2022-01-31T19:38:54.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.714+0000 7f417affd700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f416c05e710 0x7f416c060bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:38:54.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.714+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4180077470 con 0x7f418c104890 2022-01-31T19:38:54.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.714+0000 7f418a59c700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f416c05e710 0x7f416c060bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:38:54.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.714+0000 7f418a59c700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f416c05e710 0x7f416c060bd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f417c017750 tx=0x7f417c002040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:38:54.723 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.718+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f41800456b0 con 0x7f418c104890 2022-01-31T19:38:54.968 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.966+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f418c045bb0 con 0x7f418c104890 2022-01-31T19:38:54.968 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.967+0000 7f417affd700 1 -- 172.21.15.171:0/222688893 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+972 (secure 0 0 0) 0x7f4180016030 con 0x7f418c104890 2022-01-31T19:38:54.969 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-01-31T19:38:54.969 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":2,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","modified":"2022-01-31T19:38:49.072267Z","created":"2022-01-31T19:37:04.335382Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi167","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:3300","nonce":0},{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi171","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-01-31T19:38:54.971 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f416c05e710 msgr2=0x7f416c060bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:54.971 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f416c05e710 0x7f416c060bd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f417c017750 tx=0x7f417c002040).stop 2022-01-31T19:38:54.971 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f418c104890 msgr2=0x7f418c105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:38:54.971 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f418c104890 0x7f418c105780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f4180009390 tx=0x7f4180006e20).stop 2022-01-31T19:38:54.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 shutdown_connections 2022-01-31T19:38:54.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f416c05e710 0x7f416c060bd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:54.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4174003250 0x7f418c1043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:54.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 --2- 172.21.15.171:0/222688893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f418c104890 0x7f418c105780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:38:54.973 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 >> 172.21.15.171:0/222688893 conn(0x7f418c0f0a40 msgr2=0x7f418c0f8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:38:54.973 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.968+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 shutdown_connections 2022-01-31T19:38:54.973 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:38:54.969+0000 7f418b59e700 1 -- 172.21.15.171:0/222688893 wait complete. 2022-01-31T19:38:54.973 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 2 2022-01-31T19:38:55.311 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-01-31T19:38:55.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph config generate-minimal-conf 2022-01-31T19:38:55.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:55 smithi171 conmon[35325]: cephadm 2022-01-31T19:38:54.114395+0000 mgr.smithi167.aciqpk (mgr.14156) 40 : cephadm [INF] Deploying daemon node-exporter.smithi171 on smithi171 2022-01-31T19:38:55.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:55 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.168416+0000 mon.smithi167 (mon.0) 176 : cluster [DBG] mgrmap e14: smithi167.aciqpk(active, since 55s), standbys: smithi171.asyxnc 2022-01-31T19:38:55.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:55 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.168568+0000 mon.smithi167 (mon.0) 177 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi171.asyxnc", "id": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:38:55.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:55 smithi171 conmon[35325]: audit 2022-01-31T19:38:54.967561+0000 mon.smithi167 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.171:0/222688893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:55.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:55 smithi171 conmon[35325]: audit 2022-01-31T19:38:55.072668+0000 mon.smithi167 (mon.0) 179 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:55.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:54.114395+0000 mgr.smithi167.aciqpk (mgr.14156) 40 : cephadm [INF] Deploying daemon node-exporter.smithi171 on smithi171 2022-01-31T19:38:55.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: cluster 2022-01-31T19:38:54.168416+0000 mon.smithi167 (mon.0) 176 : cluster [DBG] mgrmap e14: smithi167.aciqpk(active, since 55s), standbys: smithi171.asyxnc 2022-01-31T19:38:55.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: audit 2022-01-31T19:38:54.168568+0000 mon.smithi167 (mon.0) 177 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi171.asyxnc", "id": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:38:55.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:38:55.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: -31T19:38:54.967561+0000 mon.smithi167 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.171:0/222688893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-01-31T19:38:55.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:38:55.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: 31T19:38:55.072668+0000 mon.smithi167 (mon.0) 2022-01-31T19:38:55.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:55 smithi167 conmon[32206]: 179 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:38:56.596 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:56 smithi171 conmon[35325]: cluster 2022-01-31T19:38:54.693993+0000 mgr.smithi167.aciqpk (mgr.14156) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:56.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:56 smithi167 conmon[32206]: cluster 2022-01-31T19:38:54.693993+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:38:56.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:56 smithi167 conmon[32206]: 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:57 smithi167 conmon[32206]: cluster 2022-01-31T19:38:56.694291+0000 mgr.smithi167.aciqpk (mgr.14156) 42 : 2022-01-31T19:38:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:57 smithi167 conmon[32206]: cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:57 smithi167 conmon[32206]: audit 2022-01-31T19:38:56.988466+0000 mon.smithi167 (mon.0) 180 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:57 smithi167 conmon[32206]: audit 2022-01-31T19:38:56.990227+0000 mon.smithi167 (mon.0) 181 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:38:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:57 smithi167 conmon[32206]: audit 2022-01-31T19:38:56.991428+0000 mon.smithi167 (mon.0) 182 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:57 smithi167 conmon[32206]: audit 2022-01-31T19:38:56.992414+0000 mon.smithi167 (mon.0) 183 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:38:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:57 smithi171 conmon[35325]: cluster 2022-01-31T19:38:56.694291+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:38:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:57 smithi171 conmon[35325]: 14156) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:38:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:57 smithi171 conmon[35325]: audit 2022-01-31T19:38:56.988466+0000 mon.smithi167 (mon.0) 180 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:57 smithi171 conmon[35325]: audit 2022-01-31T19:38:56.990227+0000 mon.smithi167 (mon.0) 181 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:38:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:57 smithi171 conmon[35325]: audit 2022-01-31T19:38:56.991428+0000 mon.smithi167 (mon.0) 182 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:38:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:57 smithi171 conmon[35325]: audit 2022-01-31T19:38:56.992414+0000 mon.smithi167 (mon.0) 183 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:38:59.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:58 smithi171 conmon[35325]: audit 2022-01-31T19:38:58.822652+0000 mon.smithi167 (mon.0) 184 : audit 2022-01-31T19:38:59.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:58 smithi171 conmon[35325]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:38:59.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:58 smithi171 conmon[35325]: audit 2022-01-31T19:38:58.823998+0000 mon.smithi167 (mon.0) 185 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:38:59.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:38:58 smithi171 conmon[35325]: audit 2022-01-31T19:38:58.841781+0000 mon.smithi167 (mon.0) 186 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:38:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:58 smithi167 conmon[32206]: audit 2022-01-31T19:38:58.822652+0000 mon.smithi167 (mon.0) 184 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:38:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:58 smithi167 conmon[32206]: 2022-01-31T19:38:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:58 smithi167 conmon[32206]: audit 2022-01-31T19:38:58.823998+0000 mon.smithi167 (mon.0) 185 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:38:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:58 smithi167 conmon[32206]: audit 2022-01-31T19:38:58.841781+0000 mon.smithi167 (mon 2022-01-31T19:38:59.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:38:58 smithi167 conmon[32206]: .0) 186 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:00.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 -- 172.21.15.167:0/4287408897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 msgr2=0x7efdb40fb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:00.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/4287408897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb40fb900 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7efda4009fb0 tx=0x7efda4009b30).stop 2022-01-31T19:39:00.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 -- 172.21.15.167:0/4287408897 shutdown_connections 2022-01-31T19:39:00.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/4287408897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb40fb900 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:00.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 -- 172.21.15.167:0/4287408897 >> 172.21.15.167:0/4287408897 conn(0x7efdb40f10f0 msgr2=0x7efdb40f3510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:00.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 -- 172.21.15.167:0/4287408897 shutdown_connections 2022-01-31T19:39:00.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 -- 172.21.15.167:0/4287408897 wait complete. 2022-01-31T19:39:00.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.120+0000 7efdb9e6f700 1 Processor -- start 2022-01-31T19:39:00.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb9e6f700 1 -- start start 2022-01-31T19:39:00.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb9e6f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:00.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb9e6f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb4105b20 0x7efdb4105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:00.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb9e6f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4112ca0 con 0x7efdb40fb520 2022-01-31T19:39:00.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb9e6f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4112de0 con 0x7efdb4105b20 2022-01-31T19:39:00.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:00.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60714/0 (socket says 172.21.15.167:60714) 2022-01-31T19:39:00.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 -- 172.21.15.167:0/1525357160 learned_addr learned my addr 172.21.15.167:0/1525357160 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:39:00.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 -- 172.21.15.167:0/1525357160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb4105b20 msgr2=0x7efdb4105fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:00.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb2ffd700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb4105b20 0x7efdb4105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:00.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb4105b20 0x7efdb4105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:00.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 -- 172.21.15.167:0/1525357160 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4005040 con 0x7efdb40fb520 2022-01-31T19:39:00.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.121+0000 7efdb37fe700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb4105630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7efda4013880 tx=0x7efda400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:00.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.122+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda4019070 con 0x7efdb40fb520 2022-01-31T19:39:00.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.122+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efda4007ba0 con 0x7efdb40fb520 2022-01-31T19:39:00.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.122+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb4112f20 con 0x7efdb40fb520 2022-01-31T19:39:00.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.122+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efda4014420 con 0x7efdb40fb520 2022-01-31T19:39:00.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.122+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb4113320 con 0x7efdb40fb520 2022-01-31T19:39:00.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.123+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb405e170 con 0x7efdb40fb520 2022-01-31T19:39:00.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.123+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7efda401e420 con 0x7efdb40fb520 2022-01-31T19:39:00.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.123+0000 7efd9bfff700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efd9c05e720 0x7efd9c060be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:00.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.123+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efda4078ff0 con 0x7efdb40fb520 2022-01-31T19:39:00.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.123+0000 7efdb2ffd700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efd9c05e720 0x7efd9c060be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:00.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.124+0000 7efdb2ffd700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efd9c05e720 0x7efd9c060be0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efda800a7a0 tx=0x7efda8008040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:00.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.127+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efda404bc80 con 0x7efdb40fb520 2022-01-31T19:39:00.294 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.292+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7efdb405ffb0 con 0x7efdb40fb520 2022-01-31T19:39:00.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:00 smithi167 conmon[32206]: cluster 2022-01-31T19:38:58.694589+0000 2022-01-31T19:39:00.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:00 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:00.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:00 smithi167 conmon[32206]: audit 2022-01-31T19:38:59.138525+0000 mon.smithi167 (mon.0) 187 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:00.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:00 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:59.139027+0000 mgr.smithi167.aciqpk (mgr.14156) 44 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2022-01-31T19:39:00.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:00 smithi167 conmon[32206]: cephadm 2022-01-31T19:38:59.157426+0000 mgr.smithi167.aciqpk (mgr.14156) 45 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:39:00.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:00 smithi167 conmon[32206]: audit 2022-01-31T19:38:59.186611+0000 mon.smithi167 (mon.0) 188 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:00.296 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.292+0000 7efd9bfff700 1 -- 172.21.15.167:0/1525357160 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+227 (secure 0 0 0) 0x7efda4004490 con 0x7efdb40fb520 2022-01-31T19:39:00.297 INFO:teuthology.orchestra.run.smithi167.stdout:# minimal ceph.conf for 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:39:00.297 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2022-01-31T19:39:00.298 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:39:00.298 INFO:teuthology.orchestra.run.smithi167.stdout: mon_host = [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2022-01-31T19:39:00.298 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efd9c05e720 msgr2=0x7efd9c060be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:00.299 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efd9c05e720 0x7efd9c060be0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efda800a7a0 tx=0x7efda8008040).stop 2022-01-31T19:39:00.299 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 msgr2=0x7efdb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:00.299 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb4105630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7efda4013880 tx=0x7efda400b040).stop 2022-01-31T19:39:00.299 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 shutdown_connections 2022-01-31T19:39:00.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7efd9c05e720 0x7efd9c060be0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:00.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb40fb520 0x7efdb4105630 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:00.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 --2- 172.21.15.167:0/1525357160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb4105b20 0x7efdb4105fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:00.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 >> 172.21.15.167:0/1525357160 conn(0x7efdb40f10f0 msgr2=0x7efdb40f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:00.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 shutdown_connections 2022-01-31T19:39:00.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:00.294+0000 7efdb9e6f700 1 -- 172.21.15.167:0/1525357160 wait complete. 2022-01-31T19:39:00.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:00 smithi171 conmon[35325]: cluster 2022-01-31T19:38:58.694589+0000 mgr.smithi167.aciqpk (mgr.14156) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:00.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:00 smithi171 conmon[35325]: audit 2022-01-31T19:38:59.138525+0000 mon.smithi167 (mon.0) 187 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:00.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:00 smithi171 conmon[35325]: cephadm 2022-01-31T19:38:59.139027+0000 mgr.smithi167.aciqpk (mgr.14156) 44 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2022-01-31T19:39:00.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:00 smithi171 conmon[35325]: cephadm 2022-01-31T19:38:59.157426+0000 mgr.smithi167.aciqpk (mgr.14156) 45 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:39:00.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:00 smithi171 conmon[35325]: audit 2022-01-31T19:39:00.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:00 smithi171 conmon[35325]: 2022-01-31T19:38:59.186611+0000 mon.smithi167 (mon.0) 188 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:00.784 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-01-31T19:39:00.784 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:39:00.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.conf 2022-01-31T19:39:00.817 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:39:00.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:39:00.882 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:39:00.883 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.conf 2022-01-31T19:39:00.920 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:39:00.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:39:00.990 INFO:tasks.cephadm:Deploying OSDs... 2022-01-31T19:39:00.991 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:39:00.991 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2022-01-31T19:39:01.007 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-01-31T19:39:01.008 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-01-31T19:39:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-01-31T19:39:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.064 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 151917 Links: 1 2022-01-31T19:39:01.064 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.064 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.064 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2022-01-31 19:38:35.787151859 +0000 2022-01-31T19:39:01.065 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2022-01-31 19:33:27.458766670 +0000 2022-01-31T19:39:01.065 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2022-01-31 19:33:27.458766670 +0000 2022-01-31T19:39:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2022-01-31T19:39:01.073 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2022-01-31T19:39:01.074 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2022-01-31T19:39:01.074 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000237296 s, 2.2 MB/s 2022-01-31T19:39:01.077 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-01-31T19:39:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-01-31T19:39:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.137 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 157168 Links: 1 2022-01-31T19:39:01.137 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.138 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.138 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2022-01-31 19:38:35.787151859 +0000 2022-01-31T19:39:01.138 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2022-01-31 19:33:27.794760363 +0000 2022-01-31T19:39:01.139 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2022-01-31 19:33:27.794760363 +0000 2022-01-31T19:39:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2022-01-31T19:39:01.142 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2022-01-31T19:39:01.142 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2022-01-31T19:39:01.142 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000207077 s, 2.5 MB/s 2022-01-31T19:39:01.145 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-01-31T19:39:01.202 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-01-31T19:39:01.203 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.203 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 156206 Links: 1 2022-01-31T19:39:01.203 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.204 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.204 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2022-01-31 19:38:35.787151859 +0000 2022-01-31T19:39:01.204 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2022-01-31 19:33:28.134753980 +0000 2022-01-31T19:39:01.204 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2022-01-31 19:33:28.134753980 +0000 2022-01-31T19:39:01.204 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2022-01-31T19:39:01.213 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2022-01-31T19:39:01.213 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2022-01-31T19:39:01.214 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000206839 s, 2.5 MB/s 2022-01-31T19:39:01.216 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-01-31T19:39:01.272 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-01-31T19:39:01.272 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.273 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 157233 Links: 1 2022-01-31T19:39:01.273 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.273 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.273 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2022-01-31 19:38:35.788151839 +0000 2022-01-31T19:39:01.274 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2022-01-31 19:33:28.503747053 +0000 2022-01-31T19:39:01.274 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2022-01-31 19:33:28.503747053 +0000 2022-01-31T19:39:01.274 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2022-01-31T19:39:01.282 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2022-01-31T19:39:01.282 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2022-01-31T19:39:01.283 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000180337 s, 2.8 MB/s 2022-01-31T19:39:01.285 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:39:01.285 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2022-01-31T19:39:01.306 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-01-31T19:39:01.306 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-01-31T19:39:01.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:01 smithi171 conmon[35325]: audit 2022-01-31T19:39:00.293825+0000 mon.smithi167 (mon.0) 189 : 2022-01-31T19:39:01.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:01 smithi171 conmon[35325]: audit [DBG] from='client.? 172.21.15.167:0/1525357160' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:01.364 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-01-31T19:39:01.364 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.364 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 156836 Links: 1 2022-01-31T19:39:01.365 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.365 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.365 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2022-01-31 19:38:54.652669205 +0000 2022-01-31T19:39:01.365 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2022-01-31 19:33:26.611122062 +0000 2022-01-31T19:39:01.365 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2022-01-31 19:33:26.611122062 +0000 2022-01-31T19:39:01.366 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2022-01-31T19:39:01.376 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2022-01-31T19:39:01.376 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2022-01-31T19:39:01.376 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000267982 s, 1.9 MB/s 2022-01-31T19:39:01.379 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-01-31T19:39:01.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:01 smithi167 conmon[32206]: audit 2022-01-31T19:39:00.293825+0000 mon.smithi167 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.167:0/1525357160' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:01.438 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-01-31T19:39:01.438 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.438 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 154089 Links: 1 2022-01-31T19:39:01.438 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.439 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.439 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2022-01-31 19:38:54.652669205 +0000 2022-01-31T19:39:01.439 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2022-01-31 19:33:26.948115311 +0000 2022-01-31T19:39:01.439 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2022-01-31 19:33:26.948115311 +0000 2022-01-31T19:39:01.440 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2022-01-31T19:39:01.449 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2022-01-31T19:39:01.449 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2022-01-31T19:39:01.449 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000251691 s, 2.0 MB/s 2022-01-31T19:39:01.452 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-01-31T19:39:01.510 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-01-31T19:39:01.511 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.511 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 153287 Links: 1 2022-01-31T19:39:01.511 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.511 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.512 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2022-01-31 19:38:54.652669205 +0000 2022-01-31T19:39:01.512 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2022-01-31 19:33:27.289108480 +0000 2022-01-31T19:39:01.512 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2022-01-31 19:33:27.289108480 +0000 2022-01-31T19:39:01.512 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2022-01-31T19:39:01.521 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2022-01-31T19:39:01.521 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2022-01-31T19:39:01.522 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000304135 s, 1.7 MB/s 2022-01-31T19:39:01.524 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-01-31T19:39:01.582 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-01-31T19:39:01.582 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-31T19:39:01.582 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 157880 Links: 1 2022-01-31T19:39:01.583 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:39:01.583 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2022-01-31T19:39:01.583 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2022-01-31 19:38:54.652669205 +0000 2022-01-31T19:39:01.583 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2022-01-31 19:33:27.646101329 +0000 2022-01-31T19:39:01.584 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2022-01-31 19:33:27.646101329 +0000 2022-01-31T19:39:01.584 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2022-01-31T19:39:01.592 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2022-01-31T19:39:01.593 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2022-01-31T19:39:01.593 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000238498 s, 2.1 MB/s 2022-01-31T19:39:01.596 INFO:tasks.cephadm:Deploying osd.0 on smithi167 with /dev/vg_nvme/lv_4... 2022-01-31T19:39:01.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-01-31T19:39:02.362 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:02 smithi167 conmon[32206]: cluster 2022-01-31T19:39:00.694939+0000 mgr.smithi167.aciqpk (mgr.14156) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:02.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:02 smithi171 conmon[35325]: cluster 2022-01-31T19:39:00.694939+0000 mgr.smithi167.aciqpk (mgr.14156) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:03.874 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-01-31T19:39:03.874 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-01-31T19:39:03.874 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2022-01-31T19:39:03.875 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2022-01-31T19:39:03.875 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011125 s, 943 MB/s 2022-01-31T19:39:03.875 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2022-01-31T19:39:04.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi167:vg_nvme/lv_4 2022-01-31T19:39:04.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: cluster 2022-01-31T19:39:02.695316+0000 mgr.smithi167.aciqpk (mgr.14156) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:04.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: audit 2022-01-31T19:39:03.967661+0000 mon.smithi167 (mon.0) 190 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:04.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: audit 2022-01-31T19:39:04.025857+0000 mon.smithi167 (mon.0) 191 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:04.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: audit 2022-01-31T19:39:04.028961+0000 mon.smithi167 (mon.0) 192 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:04.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: audit 2022-01-31T19:39 2022-01-31T19:39:04.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: :04.031362+0000 mon.smithi167 (mon.0) 193 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:39:04.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: audit 2022-01-31T19:39:04.031906+0000 mon.smithi167 (mon.0) 194 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-01-31T19:39:04.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:04 smithi167 conmon[32206]: audit 2022-01-31T19:39:04.032416+0000 mon.smithi167 (mon.0) 195 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:04.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: cluster 2022-01-31T19:39:02.695316+0000 mgr.smithi167.aciqpk (mgr.14156) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:04.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: audit 2022-01-31T 2022-01-31T19:39:04.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: 19:39:03.967661+0000 mon.smithi167 (mon.0) 190 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:04.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: audit 2022-01-31T19:39:04.025857+0000 mon.smithi167 (mon.0) 191 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:04.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: audit 2022-01-31T19:39 2022-01-31T19:39:04.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: :04.028961+0000 mon.smithi167 (mon.0) 192 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:04.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: audit 2022-01-31T19:39:04. 2022-01-31T19:39:04.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: 031362+0000 mon.smithi167 (mon.0) 193 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:39:04.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: audit 2022 2022-01-31T19:39:04.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: -01-31T19:39:04.031906+0000 mon.smithi167 (mon.0) 194 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-01-31T19:39:04.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: audit 2022-01-31T19:39:04 2022-01-31T19:39:04.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:04 smithi171 conmon[35325]: .032416+0000 mon.smithi167 (mon.0) 195 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:05.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:05 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:03.968118+0000 mgr.smithi167.aciqpk (mgr.14156) 48 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:39:05.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:05 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:04.031161+0000 mgr.smithi167.aciqpk (mgr.14156) 49 : cephadm [INF] Reconfiguring mon.smithi167 (unknown last config time)... 2022-01-31T19:39:05.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:05 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:04.032982+0000 mgr.smithi167.aciqpk (mgr.14156) 50 : cephadm [INF] Reconfiguring daemon mon.smithi167 on smithi167 2022-01-31T19:39:05.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:05 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:03.968118+0000 mgr.smithi167.aciqpk (mgr.14156) 48 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:39:05.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:05 smithi171 conmon[35325]: cephadm 2022-01-31 2022-01-31T19:39:05.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:05 smithi171 conmon[35325]: T19:39:04.031161+0000 mgr.smithi167.aciqpk (mgr.14156) 49 : cephadm [INF] Reconfiguring mon.smithi167 (unknown last config time)... 2022-01-31T19:39:05.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:05 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:04.032982+0000 mgr.smithi167.aciqpk (mgr.14156) 50 : cephadm [INF] Reconfiguring daemon mon.smithi167 on smithi167 2022-01-31T19:39:06.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: cluster 2022-01-31T19:39:04.695627+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:39:06.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: .14156) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:06.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.129044+0000 mon.smithi167 (mon.0) 196 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:06.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.129743+0000 mon.smithi167 (mon.0) 197 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi167.aciqpk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:39:06.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.130402+0000 mon.smithi167 (mon.0) 198 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:39:06.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:39:06.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:06 smithi167 conmon[32206]: 31T19:39:06.130970+0000 mon.smithi167 (mon.0) 199 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:06.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: cluster 2022-01-31T19:39:04.695627+0000 mgr.smithi167.aciqpk (mgr.14156) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:07.279 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:39:07.279 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: 01-31T19:39:06.129044+0000 mon.smithi167 (mon.0) 196 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:07.280 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: audit 2022-01-31T19:39:06.129743+0000 mon.smithi167 (mon.0) 197 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi167.aciqpk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:39:07.280 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: audit 2022-01-31T19:39:06.130402+0000 mon.smithi167 (mon.0) 198 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:39:07.280 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: audit 2022-01-31T19:39:06.130970+0000 mon.smithi167 (mon 2022-01-31T19:39:07.280 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:06 smithi171 conmon[35325]: .0) 199 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:07.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.718+0000 7f96e9c62700 1 -- 172.21.15.167:0/2258579820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 msgr2=0x7f96e40f5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:07.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.718+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2258579820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e40f5960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f96dc0099e0 tx=0x7f96dc004d10).stop 2022-01-31T19:39:07.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.718+0000 7f96e9c62700 1 -- 172.21.15.167:0/2258579820 shutdown_connections 2022-01-31T19:39:07.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2258579820 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 0x7f96e40f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:39:07.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2258579820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e40f5960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:07.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 -- 172.21.15.167:0/2258579820 >> 172.21.15.167:0/2258579820 conn(0x7f96e40f0d80 msgr2=0x7f96e40f31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:07.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 -- 172.21.15.167:0/2258579820 shutdown_connections 2022-01-31T19:39:07.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 -- 172.21.15.167:0/2258579820 wait complete. 2022-01-31T19:39:07.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 Processor -- start 2022-01-31T19:39:07.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.719+0000 7f96e9c62700 1 -- start start 2022-01-31T19:39:07.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e9c62700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e4109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:07.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e9c62700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 0x7f96e410c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:07.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e9c62700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4060530 con 0x7f96e40f5540 2022-01-31T19:39:07.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e9c62700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4060670 con 0x7f96e40f5e50 2022-01-31T19:39:07.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e4109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:07.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e4109d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60724/0 (socket says 172.21.15.167:60724) 2022-01-31T19:39:07.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 -- 172.21.15.167:0/2103827454 learned_addr learned my addr 172.21.15.167:0/2103827454 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:39:07.289 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 -- 172.21.15.167:0/2103827454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 msgr2=0x7f96e410c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:07.289 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e3fff700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 0x7f96e410c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:07.289 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 0x7f96e410c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:07.289 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 -- 172.21.15.167:0/2103827454 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc009640 con 0x7f96e40f5540 2022-01-31T19:39:07.290 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e3fff700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 0x7f96e410c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:39:07.290 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e8c60700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e4109d60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f96dc003380 tx=0x7f96dc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:07.290 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96dc00e070 con 0x7f96e40f5540 2022-01-31T19:39:07.291 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f96dc0049b0 con 0x7f96e40f5540 2022-01-31T19:39:07.291 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e410c810 con 0x7f96e40f5540 2022-01-31T19:39:07.291 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96dc018070 con 0x7f96e40f5540 2022-01-31T19:39:07.292 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.720+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e410ccf0 con 0x7f96e40f5540 2022-01-31T19:39:07.292 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.721+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e4045bb0 con 0x7f96e40f5540 2022-01-31T19:39:07.292 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.722+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f96dc013a20 con 0x7f96e40f5540 2022-01-31T19:39:07.292 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.722+0000 7f96e17fa700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96cc05e5b0 0x7f96cc060a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:07.292 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.722+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f96dc079590 con 0x7f96e40f5540 2022-01-31T19:39:07.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.722+0000 7f96e3fff700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96cc05e5b0 0x7f96cc060a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:07.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.723+0000 7f96e3fff700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96cc05e5b0 0x7f96cc060a70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f96d400a7e0 tx=0x7f96d4008040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:07.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.726+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f96dc04c170 con 0x7f96e40f5540 2022-01-31T19:39:07.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:06.897+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f96e4057ff0 con 0x7f96cc05e5b0 2022-01-31T19:39:07.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:07 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:06.129521+0000 mgr.smithi167.aciqpk (mgr.14156) 52 : cephadm [INF] Reconfiguring mgr.smithi167.aciqpk (unknown last config time)... 2022-01-31T19:39:07.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:07 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:06.131436+0000 mgr.smithi167.aciqpk (mgr.14156) 53 : cephadm [INF] Reconfiguring daemon mgr.smithi167.aciqpk on smithi167 2022-01-31T19:39:07.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:07 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.901136+0000 mon.smithi167 (mon.0) 200 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:39:07.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:07 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.902581+0000 mon.smithi167 (mon.0) 201 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:39:07.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:07 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.903097+0000 mon.smithi167 (mon.0) 202 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:07.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:06.129521+0000 mgr.smithi167.aciqpk (mgr.14156) 52 : cephadm [INF] Reconfiguring mgr.smithi167.aciqpk (unknown last config time)... 2022-01-31T19:39:07.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:06.131436+0000 mgr.smithi167.aciqpk (mgr.14156) 53 : cephadm [INF] Reconfiguring daemon mgr.smithi167.aciqpk on smithi167 2022-01-31T19:39:07.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:39:07.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: 01-31T19:39:06.901136+0000 mon.smithi167 (mon.0) 200 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:39:07.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: audit 2022 2022-01-31T19:39:07.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: -01-31T19:39:06.902581+0000 mon.smithi167 (mon.0) 201 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:39:07.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: audit 2022-01-31T19:39:06.903097+0000 mon.smithi167 (mon.0 2022-01-31T19:39:07.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:07 smithi171 conmon[35325]: ) 202 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:08.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:08 smithi167 conmon[32206]: cluster 2022-01-31T19:39:06.696111+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:39:08.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:08 smithi167 conmon[32206]: mgr.14156) 54 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:08.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:08 smithi167 conmon[32206]: audit 2022-01-31T19:39:06.900265+0000 mgr.smithi167.aciqpk (mgr.14156) 55 : audit [DBG] from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:39:08.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:08 smithi167 conmon[32206]: audit 2022-01-31T19:39:07.558277+0000 mon.smithi167 (mon.0) 203 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:08.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:08 smithi171 conmon[35325]: cluster 2022-01-31T19:39:06.696111+0000 mgr.smithi167.aciqpk (mgr.14156) 54 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:08.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:08 smithi171 conmon[35325]: audit 2022-01-31T19:39:06.900265+0000 mgr.smithi167.aciqpk (mgr.14156) 55 : audit [DBG] from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:39:08.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:08 smithi171 conmon[35325]: audit 2022-01-31T19:39:07.558277+0000 mon.smithi167 (mon.0) 203 : audit 2022-01-31T19:39:08.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:08 smithi171 conmon[35325]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:07.558785+0000 mgr.smithi167.aciqpk (mgr.14156) 56 : cephadm [INF] Reconfiguring alertmanager.smithi167 (dependencies changed)... 2022-01-31T19:39:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: cephadm 2022-01-31T 2022-01-31T19:39:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: 19:39:07.562058+0000 mgr.smithi167.aciqpk (mgr.14156) 57 : cephadm [INF] Reconfiguring daemon alertmanager.smithi167 on smithi167 2022-01-31T19:39:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: audit 2022-01-31T19:39:09.050427+0000 mon.smithi167 (mon.0) 204 : audit [INF] from='client.? 172.21.15.167:0/3320111963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b39ea02-b5a4-4a19-bd99-849c0378fad1"}]: dispatch 2022-01-31T19:39:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: audit 2022-01-31T19:39:09.054092+0000 mon.smithi167 (mon.0) 205 : audit [INF] from='client.? 172.21.15.167:0/3320111963' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b39ea02-b5a4-4a19-bd99-849c0378fad1"}]': finished 2022-01-31T19:39:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: cluster 2022-01-31T19:39:09.054140+0000 mon.smithi167 (mon.0) 206 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2022-01-31T19:39:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:09 smithi167 conmon[32206]: audit 2022-01-31T19:39:09.054227+0000 mon.smithi167 (mon.0) 207 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:07.558785+0000 mgr.smithi167.aciqpk (mgr.14156) 56 : cephadm [INF] Reconfiguring alertmanager.smithi167 (dependencies changed)... 2022-01-31T19:39:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:07.562058+0000 mgr.smithi167.aciqpk (mgr.14156) 57 : cephadm [INF] Reconfiguring daemon alertmanager.smithi167 on smithi167 2022-01-31T19:39:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: audit 2022-01-31T19:39:09.050427+0000 2022-01-31T19:39:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: mon.smithi167 (mon.0) 204 : audit [INF] from='client.? 172.21.15.167:0/3320111963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b39ea02-b5a4-4a19-bd99-849c0378fad1"}]: dispatch 2022-01-31T19:39:09.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: audit 2022-01-31T19:39:09.054092 2022-01-31T19:39:09.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 205 : audit [INF] from='client.? 172.21.15.167:0/3320111963' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b39ea02-b5a4-4a19-bd99-849c0378fad1"}]': finished 2022-01-31T19:39:09.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: cluster 2022-01- 2022-01-31T19:39:09.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: 31T19:39:09.054140+0000 mon.smithi167 (mon.0) 206 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2022-01-31T19:39:09.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: audit 2022-01-31T19:39:09.054227+0000 mon.smithi167 (mon.0 2022-01-31T19:39:09.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:09 smithi171 conmon[35325]: ) 207 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:10.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:10 smithi171 conmon[35325]: cluster 2022-01-31T19:39:08.696362+0000 mgr.smithi167.aciqpk (mgr.14156) 58 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:10.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:10 smithi171 conmon[35325]: audit 2022-01-31T19:39:09.479302+0000 mon.smithi167 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.167:0/818152058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:39:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:10 smithi167 conmon[32206]: cluster 2022-01-31T19:39:08.696362+0000 mgr.smithi167.aciqpk (mgr.14156) 58 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:10 smithi167 conmon[32206]: audit 2022-01-31T19:39:09. 2022-01-31T19:39:10.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:10 smithi167 conmon[32206]: 479302+0000 mon.smithi167 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.167:0/818152058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:39:11.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:11 smithi167 conmon[32206]: cluster 2022-01-31T19:39:10.696728+0000 mgr.smithi167.aciqpk (mgr.14156) 59 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:12.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:11 smithi171 conmon[35325]: cluster 2022-01-31T19:39:10.696728+0000 mgr.smithi167.aciqpk (mgr.14156) 59 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:14.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:13 smithi167 conmon[32206]: cluster 2022-01-31T19:39:12.697015+0000 mgr.smithi167.aciqpk 2022-01-31T19:39:14.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:13 smithi167 conmon[32206]: (mgr.14156) 60 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:13 smithi171 conmon[35325]: cluster 2022-01-31T19:39:12.697015+0000 mgr.smithi167.aciqpk (mgr.14156) 60 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:16.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:15 smithi167 conmon[32206]: cluster 2022-01-31T19:39:14.697390+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:39:16.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:15 smithi167 conmon[32206]: 61 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:16.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:15 smithi171 conmon[35325]: cluster 2022-01-31T19:39:14.697390+0000 mgr.smithi167.aciqpk (mgr.14156) 61 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:17.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:17 smithi167 conmon[32206]: audit 2022-01-31T19:39:16.498983+0000 mon.smithi167 (mon.0) 209 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:17.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:17 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:16.499477+0000 mgr.smithi167.aciqpk (mgr.14156) 62 : cephadm [INF] Reconfiguring crash.smithi167 (monmap changed)... 2022-01-31T19:39:17.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:17 smithi167 conmon[32206]: audit 2022-01-31T19:39:16.499700+0000 mon.smithi167 (mon.0) 210 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi167", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2022-01-31T19:39:17.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:17 smithi167 conmon[32206]: audit 2022-01-31T19:39:16.500270+0000 mon.smithi167 (mon.0) 211 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:17.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:17 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:16.500676+0000 mgr.smithi167.aciqpk (mgr.14156) 63 : cephadm [INF] Reconfiguring daemon crash.smithi167 on smithi167 2022-01-31T19:39:17.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:17 smithi167 conmon[32206]: cluster 2022-01-31T19:39:16.697673+0000 mgr.smithi167.aciqpk (mgr.14156) 64 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:17.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: audit 2022-01-31T19:39:16.498983+0000 mon.smithi167 (mon.0) 209 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:17.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:16.499477+0000 mgr.smithi167.aciqpk (mgr.14156) 62 : cephadm [INF] Reconfiguring crash.smithi167 (monmap changed)... 2022-01-31T19:39:17.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: audit 2022-01-31T19:39:16.499700+0000 mon.smithi167 2022-01-31T19:39:17.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: (mon.0) 210 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi167", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2022-01-31T19:39:17.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: audit 2022-01-31T19:39:16.500270+0000 mon.smithi167 2022-01-31T19:39:17.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: (mon.0) 211 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:17.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: cephadm 2022-01-31T 2022-01-31T19:39:17.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: 19:39:16.500676+0000 mgr.smithi167.aciqpk (mgr.14156) 63 : cephadm [INF] Reconfiguring daemon crash.smithi167 on smithi167 2022-01-31T19:39:17.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: cluster 2022-01-31T19:39:16.697673+0000 mgr.smithi167.aciqpk (mgr.14156) 64 : 2022-01-31T19:39:17.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:17 smithi171 conmon[35325]: cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:18.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: audit 2022-01-31T19:39:17.601837+0000 mon.smithi167 ( 2022-01-31T19:39:18.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: mon.0) 212 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-01-31T19:39:18.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: audit 2022-01-31T19:39:17.602452+0000 2022-01-31T19:39:18.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: mon.smithi167 (mon.0) 213 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:18.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:17 2022-01-31T19:39:18.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: .602951+0000 mgr.smithi167.aciqpk (mgr.14156) 65 : cephadm 2022-01-31T19:39:18.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:18 smithi167 conmon[32206]: [INF] Deploying daemon osd.0 on smithi167 2022-01-31T19:39:18.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:18 smithi171 conmon[35325]: audit 2022-01-31T19:39:17.601837+0000 mon.smithi167 (mon.0) 212 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-01-31T19:39:18.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:18 smithi171 conmon[35325]: audit 2022-01-31T19:39:17.602452+0000 mon.smithi167 (mon.0) 213 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:18.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:18 smithi171 conmon[35325]: cephadm 2022- 2022-01-31T19:39:18.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:18 smithi171 conmon[35325]: 01-31T19:39:17.602951+0000 mgr.smithi167.aciqpk (mgr.14156) 65 : cephadm [INF] Deploying daemon osd.0 on smithi167 2022-01-31T19:39:19.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:19 smithi171 conmon[35325]: audit 2022-01-31T19:39:18.597844+0000 mon.smithi167 (mon.0) 214 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:19.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:19 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:18.598435+0000 mgr.smithi167.aciqpk (mgr.14156) 66 : cephadm [INF] Reconfiguring grafana.smithi167 (dependencies changed)... 2022-01-31T19:39:19.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:19 smithi171 conmon[35325]: cephadm 2022-01-31T19:39: 2022-01-31T19:39:19.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:19 smithi171 conmon[35325]: 18.602857+0000 mgr.smithi167.aciqpk (mgr.14156) 67 : cephadm [INF] Reconfiguring daemon grafana.smithi167 on smithi167 2022-01-31T19:39:19.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:19 smithi171 conmon[35325]: cluster 2022-01-31T19:39: 2022-01-31T19:39:19.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:19 smithi171 conmon[35325]: 18.697939+0000 mgr.smithi167.aciqpk (mgr.14156) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:19.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:19 smithi167 conmon[32206]: audit 2022-01-31T19:39:18.597844+0000 mon.smithi167 (mon.0) 214 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:19.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:19 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:18.598435+0000 mgr.smithi167.aciqpk (mgr.14156) 66 : cephadm [INF] Reconfiguring grafana.smithi167 (dependencies changed)... 2022-01-31T19:39:19.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:19 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:18.602857+0000 mgr.smithi167.aciqpk (mgr.14156) 67 : cephadm [INF] Reconfiguring daemon grafana.smithi167 on smithi167 2022-01-31T19:39:19.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:19 smithi167 conmon[32206]: cluster 2022-01-31T19:39:18.697939+0000 mgr.smithi167.aciqpk (mgr.14156) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:21.456 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 0 on host 'smithi167' 2022-01-31T19:39:21.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.453+0000 7f96e17fa700 1 -- 172.21.15.167:0/2103827454 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f96e4057ff0 con 0x7f96cc05e5b0 2022-01-31T19:39:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96cc05e5b0 msgr2=0x7f96cc060a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96cc05e5b0 0x7f96cc060a70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f96d400a7e0 tx=0x7f96d4008040).stop 2022-01-31T19:39:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 msgr2=0x7f96e4109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e4109d60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f96dc003380 tx=0x7f96dc004d10).stop 2022-01-31T19:39:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 shutdown_connections 2022-01-31T19:39:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f96cc05e5b0 0x7f96cc060a70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96e40f5540 0x7f96e4109d60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 --2- 172.21.15.167:0/2103827454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f5e50 0x7f96e410c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 >> 172.21.15.167:0/2103827454 conn(0x7f96e40f0d80 msgr2=0x7f96e40fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.455+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 shutdown_connections 2022-01-31T19:39:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:21.456+0000 7f96e9c62700 1 -- 172.21.15.167:0/2103827454 wait complete. 2022-01-31T19:39:21.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:21 smithi167 conmon[32206]: cluster 2022-01-31T19:39:20.698160+0000 mgr.smithi167.aciqpk (mgr.14156) 69 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:21.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:21 smithi167 conmon[32206]: audit 2022-01-31T19:39:21.449000+0000 mon.smithi167 (mon.0) 215 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:21.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:21 smithi167 conmon[32206]: audit 2022-01-31T19:39:21.452882+0000 mon.smithi167 (mon.0) 216 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:21.880 DEBUG:teuthology.orchestra.run.smithi167:osd.0> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@osd.0.service 2022-01-31T19:39:21.882 INFO:tasks.cephadm:Deploying osd.1 on smithi167 with /dev/vg_nvme/lv_3... 2022-01-31T19:39:21.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-01-31T19:39:21.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:-- Logs begin at Mon 2022-01-31 19:28:37 UTC. -- 2022-01-31T19:39:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:21 smithi171 conmon[35325]: cluster 2022-01-31T19:39:20.698160+0000 mgr.smithi167.aciqpk (mgr.14156) 69 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:21 smithi171 conmon[35325]: audit 2022-01-31T19:39:21.449000+0000 mon.smithi167 (mon.0) 215 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:22.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:21 smithi171 conmon[35325]: audit 2022-01-31T19:39:21.452882+0000 mon.smithi167 (mon. 2022-01-31T19:39:22.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:21 smithi171 conmon[35325]: 0) 216 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:23.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:23 smithi167 conmon[32206]: cluster 2022-01-31T19:39:22.698407+0000 mgr.smithi167.aciqpk (mgr.14156) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:23 smithi171 conmon[35325]: cluster 2022-01-31T19:39:22.698407+0000 mgr.smithi167.aciqpk (mgr.14156) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:24.407 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-01-31T19:39:24.408 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-01-31T19:39:24.408 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2022-01-31T19:39:24.408 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2022-01-31T19:39:24.409 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127895 s, 820 MB/s 2022-01-31T19:39:24.409 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2022-01-31T19:39:25.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi167:vg_nvme/lv_3 2022-01-31T19:39:25.607 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[49112]: debug 2022-01-31T19:39:25.391+0000 7fc45a987080 -1 osd.0 0 log_to_monitors {default=true} 2022-01-31T19:39:25.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[32206]: cluster 2022-01-31T19:39:24.698823+0000 mgr.smithi167.aciqpk 2022-01-31T19:39:25.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[32206]: (mgr.14156) 71 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:25.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[32206]: audit 2022-01-31T19:39:25.394816+0000 mon.smithi167 (mon.0) 217 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-01-31T19:39:25.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[32206]: audit 2022-01-31T19:39:25.411627+0000 mon.smithi167 (mon.0) 218 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:25.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:25.412242+0000 mgr.smithi167.aciqpk (mgr.14156) 72 : cephadm [INF] Reconfiguring prometheus.smithi167 (dependencies changed)... 2022-01-31T19:39:25.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:25 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:25.414255+0000 mgr.smithi167.aciqpk (mgr.14156) 73 : cephadm [INF] Reconfiguring daemon prometheus.smithi167 on smithi167 2022-01-31T19:39:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: cluster 2022-01-31T19:39:24.698823+0000 mgr.smithi167.aciqpk (mgr.14156) 71 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: audit 2022-01-31T19:39:25.394816+0000 mon.smithi167 (mon.0) 217 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-01-31T19:39:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: audit 2022-01-31T19:39:25.411627+0000 mon.smithi167 ( 2022-01-31T19:39:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: mon.0) 218 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:25.412242+0000 mgr.smithi167.aciqpk 2022-01-31T19:39:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: (mgr.14156) 72 : cephadm [INF] Reconfiguring prometheus.smithi167 (dependencies changed)... 2022-01-31T19:39:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: cephadm 2022-01-31T19: 2022-01-31T19:39:26.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:25 smithi171 conmon[35325]: 39:25.414255+0000 mgr.smithi167.aciqpk (mgr.14156) 73 : cephadm [INF] Reconfiguring daemon prometheus.smithi167 on smithi167 2022-01-31T19:39:27.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 19:39:26 smithi167 conmon[49112]: debug 2022-01-31T19:39:26.766+0000 7fc450eea700 -1 osd.0 0 waiting for initial osdmap 2022-01-31T19:39:27.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 19:39:26 smithi167 conmon[49112]: debug 2022-01-31T19:39:26.769+0000 7fc44a660700 -1 osd.0 7 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-01-31T19:39:27.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:26 smithi167 conmon[32206]: audit 2022-01-31T19:39:25.760358+0000 mon.smithi167 (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-01-31T19:39:27.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:26 smithi167 conmon[32206]: cluster 2022-01-31T19:39:25.760407+0000 mon.smithi167 (mon.0) 220 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2022-01-31T19:39:27.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:26 smithi167 conmon[32206]: audit 2022-01-31T19:39:25.760524+0000 mon.smithi167 (mon.0) 221 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:27.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:26 smithi167 conmon[32206]: audit 2022-01-31T19:39:25.760915+0000 mon.smithi167 (mon.0) 222 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2022-01-31T19:39:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:26 smithi171 conmon[35325]: audit 2022-01-31T19:39:25.760358+0000 mon.smithi167 (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-01-31T19:39:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:26 smithi171 conmon[35325]: cluster 2022-01-31T19:39:25.760407+0000 mon.smithi167 (mon.0) 220 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2022-01-31T19:39:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:26 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:39:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:26 smithi171 conmon[35325]: -31T19:39:25.760524+0000 mon.smithi167 (mon.0) 221 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:27.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:26 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:39:27.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:26 smithi171 conmon[35325]: 31T19:39:25.760915+0000 mon.smithi167 (mon.0) 222 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2022-01-31T19:39:27.345 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 -- 172.21.15.167:0/1971758680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f8590 msgr2=0x7fc0180f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:27.346 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 --2- 172.21.15.167:0/1971758680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f8590 0x7fc0180f47a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc00c004660 tx=0x7fc00c009b30).stop 2022-01-31T19:39:27.346 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 -- 172.21.15.167:0/1971758680 shutdown_connections 2022-01-31T19:39:27.346 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 --2- 172.21.15.167:0/1971758680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f8590 0x7fc0180f47a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:27.347 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 --2- 172.21.15.167:0/1971758680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0180f7c80 0x7fc0180f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:27.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 -- 172.21.15.167:0/1971758680 >> 172.21.15.167:0/1971758680 conn(0x7fc0180f04e0 msgr2=0x7fc0180f2900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:27.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 -- 172.21.15.167:0/1971758680 shutdown_connections 2022-01-31T19:39:27.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.334+0000 7fc01c966700 1 -- 172.21.15.167:0/1971758680 wait complete. 2022-01-31T19:39:27.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01c966700 1 Processor -- start 2022-01-31T19:39:27.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01c966700 1 -- start start 2022-01-31T19:39:27.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01c966700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 0x7fc0180f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:27.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01c966700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0180f8590 0x7fc0180f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:27.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01c966700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180f5f30 con 0x7fc0180f7c80 2022-01-31T19:39:27.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01c966700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180f6070 con 0x7fc0180f8590 2022-01-31T19:39:27.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc016d9d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 0x7fc0180f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:27.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc016d9d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 0x7fc0180f7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60772/0 (socket says 172.21.15.167:60772) 2022-01-31T19:39:27.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc016d9d700 1 -- 172.21.15.167:0/2439971350 learned_addr learned my addr 172.21.15.167:0/2439971350 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:39:27.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.335+0000 7fc01659c700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0180f8590 0x7fc0180f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:27.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fc016d9d700 1 -- 172.21.15.167:0/2439971350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0180f8590 msgr2=0x7fc0180f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fc016d9d700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0180f8590 0x7fc0180f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fc016d9d700 1 -- 172.21.15.167:0/2439971350 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c005040 con 0x7fc0180f7c80 2022-01-31T19:39:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fc016d9d700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 0x7fc0180f7550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc00800c530 tx=0x7fc008007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00800f650 con 0x7fc0180f7c80 2022-01-31T19:39:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc00800fc40 con 0x7fc0180f7c80 2022-01-31T19:39:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0180f6350 con 0x7fc0180f7c80 2022-01-31T19:39:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc008012b90 con 0x7fc0180f7c80 2022-01-31T19:39:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.336+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01810eeb0 con 0x7fc0180f7c80 2022-01-31T19:39:27.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.337+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc018045bb0 con 0x7fc0180f7c80 2022-01-31T19:39:27.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.337+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fc008010070 con 0x7fc0180f7c80 2022-01-31T19:39:27.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.338+0000 7fbfff7fe700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc00005e5c0 0x7fc000060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.338+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1397+0+0 (secure 0 0 0) 0x7fc008077740 con 0x7fc0180f7c80 2022-01-31T19:39:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.338+0000 7fc01659c700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc00005e5c0 0x7fc000060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.338+0000 7fc01659c700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc00005e5c0 0x7fc000060a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc00c005010 tx=0x7fc00c009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.342+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc0080456b0 con 0x7fc0180f7c80 2022-01-31T19:39:27.517 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:27.517+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0180fef00 con 0x7fc00005e5c0 2022-01-31T19:39:27.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: cluster 2022-01-31T19:39:26.699066+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:39:27.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: mgr.14156) 74 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:27.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: audit 2022-01-31T19:39:26.764041+0000 mon.smithi167 (mon.0) 223 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2022-01-31T19:39:27.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: cluster 2022-01-31T19:39:26.764089+0000 mon.smithi167 (mon.0) 224 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2022-01-31T19:39:27.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: audit 2022-01-31T19:39:26.764282+0000 mon.smithi167 (mon.0) 225 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:27.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: audit 2022-01-31T19:39:26.767061+0000 mon.smithi167 (mon.0) 226 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:27.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: audit 2022-01-31T19:39:27.518989+0000 mon.smithi167 (mon.0) 227 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:39:27.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: audit 2022-01-31T19:39:27.520234+0000 mon.smithi167 (mon.0) 228 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:39:27.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:27 smithi167 conmon[32206]: audit 2022-01-31T19:39:27.520753+0000 mon.smithi167 (mon.0) 229 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: cluster 2022-01-31T19:39:26.699066+0000 mgr.smithi167.aciqpk (mgr.14156) 74 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-01-31T19:39:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: audit 2022-01-31T19:39:26.764041+0000 mon.smithi167 (mon.0) 223 : audit [INF] from='osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2022-01-31T19:39:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: cluster 2022-01-31T19:39:26. 2022-01-31T19:39:28.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: 764089+0000 mon.smithi167 (mon.0) 224 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2022-01-31T19:39:28.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: audit 2022-01-31T19:39:26.764282+0000 2022-01-31T19:39:28.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: mon.smithi167 (mon.0) 225 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:28.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: audit 2022-01-31T19:39:26.767061 2022-01-31T19:39:28.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 226 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:28.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:39:28.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: 39:27.518989+0000 mon.smithi167 (mon.0) 227 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:39:28.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: audit 2022-01-31T19:39:27.520234+0000 mon.smithi167 (mon.0) 228 : audit 2022-01-31T19:39:28.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:39:28.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: audit 2022 2022-01-31T19:39:28.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:27 smithi171 conmon[35325]: -01-31T19:39:27.520753+0000 mon.smithi167 (mon.0) 229 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:28.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: cluster 2022-01-31T19:39:26.422531+0000 osd.0 (osd.0 2022-01-31T19:39:28.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: ) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:39:28.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: cluster 2022-01-31T19:39:26.422628+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:39:28.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: audit 2022-01-31T19:39:27.518252+0000 mgr.smithi167.aciqpk (mgr.14156) 75 : audit [DBG] from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:39:28.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: cluster 2022-01-31T19:39:27.765202+0000 mon.smithi167 (mon.0) 230 : cluster [INF] osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] boot 2022-01-31T19:39:28.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: cluster 2022-01-31T19:39:27.765230+0000 mon.smithi167 (mon.0 2022-01-31T19:39:28.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: ) 231 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2022-01-31T19:39:28.908 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:28 smithi167 conmon[32206]: audit 2022-01-31T19:39:27.765671+0000 mon.smithi167 (mon.0) 232 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:29.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: cluster 2022-01-31T19:39:26.422531+0000 osd.0 (osd.0) 1 : cluster [DBG] 2022-01-31T19:39:29.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: purged_snaps scrub starts 2022-01-31T19:39:29.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: cluster 2022-01-31T19:39:26.422628+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:39:29.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: audit 2022-01-31T19:39:27.518252+0000 mgr.smithi167.aciqpk (mgr.14156) 75 : audit [DBG] from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:39:29.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: 2022-01-31T19:39:29.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: cluster 2022-01-31T19:39: 2022-01-31T19:39:29.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: 27.765202+0000 mon.smithi167 (mon.0) 230 : cluster [INF] osd.0 [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] boot 2022-01-31T19:39:29.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: cluster 2022-01-31T19:39:27.765230+0000 mon.smithi167 (mon.0) 231 : cluster 2022-01-31T19:39:29.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: [DBG] osdmap e8: 1 total, 1 up, 1 in 2022-01-31T19:39:29.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:39:29.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:28 smithi171 conmon[35325]: T19:39:27.765671+0000 mon.smithi167 (mon.0) 232 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:39:29.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:29 smithi167 conmon[32206]: cluster 2022-01-31T19:39:28.699532+0000 mgr.smithi167.aciqpk (mgr.14156) 76 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:29.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:29 smithi167 conmon[32206]: cluster 2022-01-31T19:39:28.775653+0000 mon.smithi167 (mon.0) 233 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2022-01-31T19:39:29.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:29 smithi167 conmon[32206]: audit 2022-01-31T19:39:29.584152+0000 mon.smithi167 (mon.0) 234 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:29.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:29 smithi167 conmon[32206]: audit 2022-01-31T19:39:29.584823+0000 mon.smithi167 (mon.0) 235 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi171", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2022-01-31T19:39:29.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:29 smithi167 conmon[32206]: audit 2022-01-31T19:39:29.585472+0000 mon.smithi167 (mon.0) 236 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:30.072 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: cluster 2022-01-31T19:39:28.699532+0000 mgr.smithi167.aciqpk (mgr.14156) 76 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:30.072 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: cluster 2022-01-31T19:39:28.775653+0000 mon.smithi167 (mon.0) 233 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2022-01-31T19:39:30.073 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: audit 2022-01-31T19:39:29 2022-01-31T19:39:30.073 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: .584152+0000 mon.smithi167 (mon.0) 234 : audit 2022-01-31T19:39:30.073 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:30.073 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:39:30.074 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: T19:39:29.584823+0000 mon.smithi167 ( 2022-01-31T19:39:30.074 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: mon.0) 235 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi171", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2022-01-31T19:39:30.074 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: 2022-01-31T19:39:30.074 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:39:30.075 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: 31T19:39:29.585472+0000 mon.smithi167 (mon.0) 236 : audit [DBG] 2022-01-31T19:39:30.075 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:29 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:30.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:29.584600 2022-01-31T19:39:30.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: +0000 mgr.smithi167.aciqpk (mgr.14156) 77 : cephadm [INF] Reconfiguring crash.smithi171 (monmap changed)... 2022-01-31T19:39:30.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:29.585905+0000 mgr.smithi167.aciqpk (mgr.14156) 78 : cephadm [INF] Reconfiguring daemon crash.smithi171 on smithi171 2022-01-31T19:39:30.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.558350+0000 mon.smithi167 (mon.0) 237 : audit [INF] from='client.? 172.21.15.167:0/94537626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "214aeaf2-f8c7-4638-96b6-4e6bb4bd802f"}]: dispatch 2022-01-31T19:39:30.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.561948+0000 mon.smithi167 (mon.0) 238 : audit [INF] from='client.? 172.21.15.167:0/94537626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "214aeaf2-f8c7-4638-96b6-4e6bb4bd802f"}]': finished 2022-01-31T19:39:30.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: cluster 2022-01-31T19:39:30.561992+0000 mon.smithi167 (mon.0) 239 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2022-01-31T19:39:30.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.562105+0000 mon.smithi167 (mon.0) 240 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:30.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.768275+0000 mon.smithi167 (mon.0) 241 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:30.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.769340+0000 mon.smithi167 (mon.0) 242 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:39:30.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:30 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.770245+0000 mon.smithi167 (mon.0) 243 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:39:31.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:29.584600+0000 mgr.smithi167.aciqpk (mgr.14156) 77 : 2022-01-31T19:39:31.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: cephadm [INF] Reconfiguring crash.smithi171 (monmap changed)... 2022-01-31T19:39:31.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:29.585905+0000 mgr.smithi167.aciqpk (mgr.14156) 78 : cephadm [INF] Reconfiguring daemon crash.smithi171 on smithi171 2022-01-31T19:39:31.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.558350+0000 mon.smithi167 (mon.0) 237 : audit [INF] from='client.? 172.21.15.167:0/94537626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "214aeaf2-f8c7-4638-96b6-4e6bb4bd802f"}]: dispatch 2022-01-31T19:39:31.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.561948+0000 mon.smithi167 (mon.0) 238 : audit [INF] from='client.? 172.21.15.167:0/94537626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "214aeaf2-f8c7-4638-96b6-4e6bb4bd802f"}]': finished 2022-01-31T19:39:31.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: cluster 2022-01-31T19:39:30.561992+0000 mon.smithi167 (mon.0) 239 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2022-01-31T19:39:31.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.562105+0000 mon.smithi167 (mon.0) 240 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:31.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.768275+0000 mon.smithi167 (mon.0) 241 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:31.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.769340+0000 mon.smithi167 (mon.0) 242 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:39:31.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:30 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.770245+0000 mon.smithi167 (mon.0) 243 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:39:32.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:31 smithi171 conmon[35325]: cluster 2022-01-31T19:39:30.699766+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:39:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:31 smithi171 conmon[35325]: mgr.14156) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:31 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:30.768847+0000 mgr.smithi167.aciqpk (mgr.14156) 80 : cephadm [INF] Reconfiguring mgr.smithi171.asyxnc (monmap changed)... 2022-01-31T19:39:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:31 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.775334+0000 mon.smithi167 (mon.0) 244 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:32.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:31 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:30.775871+0000 mgr.smithi167.aciqpk (mgr.14156) 81 : cephadm [INF] Reconfiguring daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:39:32.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:31 smithi171 conmon[35325]: audit 2022-01-31T19:39:30.983743+0000 mon.smithi167 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.167:0/4070402155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:39:32.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: cluster 2022-01-31T19:39:30.699766+0000 mgr.smithi167.aciqpk (mgr.14156) 79 : cluster 2022-01-31T19:39:32.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:32.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:30.768847+0000 mgr.smithi167.aciqpk (mgr.14156) 80 : cephadm [INF] Reconfiguring mgr.smithi171.asyxnc (monmap changed)... 2022-01-31T19:39:32.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: audit 2022-01-31T19:39:30.775334+0000 mon.smithi167 2022-01-31T19:39:32.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: (mon.0) 244 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:32.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: cephadm 2022- 2022-01-31T19:39:32.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: 01-31T19:39:30.775871+0000 mgr.smithi167.aciqpk 2022-01-31T19:39:32.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: (mgr.14156) 81 : cephadm [INF] 2022-01-31T19:39:32.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: Reconfiguring daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:39:32.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:39:32.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: 01-31T19:39:30. 2022-01-31T19:39:32.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: 983743+0000 mon.smithi167 (mon.0) 2022-01-31T19:39:32.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:31 smithi167 conmon[32206]: 245 : audit [DBG] from='client.? 172.21.15.167:0/4070402155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:39:33.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: audit 2022-01-31T19:39:31.953423+0000 mon.smithi167 (mon.0 2022-01-31T19:39:33.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: ) 246 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:33.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:31.953927+0000 mgr.smithi167.aciqpk (mgr.14156) 82 : cephadm [INF] Reconfiguring mon.smithi171 (monmap changed)... 2022-01-31T19:39:33.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: audit 2022-01-31T19:39:31.954255+0000 mon.smithi167 (mon.0) 247 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:39:33.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: audit 2022-01-31T19:39:31.954984+0000 mon.smithi167 (mon.0) 248 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-01-31T19:39:33.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: audit 2022-01-31T19:39:31.955650+0000 mon.smithi167 (mon.0) 249 : audit [DBG] 2022-01-31T19:39:33.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:33.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:31.956213+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:39:33.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:32 smithi171 conmon[35325]: 83 : cephadm [INF] Reconfiguring daemon mon.smithi171 on smithi171 2022-01-31T19:39:33.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: audit 2022-01-31T19:39:31.953423+0000 mon.smithi167 (mon.0) 246 : audit [INF] 2022-01-31T19:39:33.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:33.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:31.953927+0000 mgr.smithi167.aciqpk (mgr.14156) 82 : cephadm [INF] Reconfiguring mon.smithi171 (monmap changed)... 2022-01-31T19:39:33.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: audit 2022-01-31T19:39:31.954255+0000 2022-01-31T19:39:33.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: mon.smithi167 (mon.0) 247 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:39:33.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:39:33.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: -31T19:39:31.954984 2022-01-31T19:39:33.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 248 : audit 2022-01-31T19:39:33.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-01-31T19:39:33.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:39:33.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: 31T19:39:31.955650+0000 mon.smithi167 (mon.0) 249 2022-01-31T19:39:33.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:33.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: cephadm 2022- 2022-01-31T19:39:33.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: 01-31T19:39:31.956213 2022-01-31T19:39:33.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 83 2022-01-31T19:39:33.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:32 smithi167 conmon[32206]: : cephadm [INF] Reconfiguring daemon mon.smithi171 on smithi171 2022-01-31T19:39:34.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: cluster 2022-01-31T19:39:32.700258+0000 mgr.smithi167.aciqpk (mgr.14156) 84 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: audit 2022-01-31T19:39:33.122334+0000 mon.smithi167 (mon.0) 250 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: audit 2022-01-31T19:39:33 2022-01-31T19:39:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: .124613+0000 mon.smithi167 (mon.0) 251 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-01-31T19:39:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:39:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: 31T19:39:33.124946+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:39:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:34 smithi167 conmon[32206]: .14156) 85 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-01-31T19:39:34.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:34 smithi171 conmon[35325]: cluster 2022-01-31T19:39:32.700258+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:39:34.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:34 smithi171 conmon[35325]: 14156) 84 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:34.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:34 smithi171 conmon[35325]: audit 2022-01-31T19:39:33.122334+0000 mon.smithi167 (mon.0) 250 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:34.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:34 smithi171 conmon[35325]: audit 2022-01-31T19:39:33.124613+0000 mon.smithi167 (mon.0) 251 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-01-31T19:39:34.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:34 smithi171 conmon[35325]: audit 2022-01-31T19:39:33.124946+0000 mgr.smithi167.aciqpk (mgr.14156) 85 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2022-01-31T19:39:36.378 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:36 smithi167 conmon[32206]: cluster 2022-01-31T19:39:34.700599+0000 mgr.smithi167.aciqpk (mgr.14156) 86 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:36.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:36 smithi171 conmon[35325]: cluster 2022-01-31T19:39:34.700599+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:39:36.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:36 smithi171 conmon[35325]: 14156) 86 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:37.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:37 smithi167 conmon[32206]: audit 2022-01-31T19:39:37.095563+0000 mon.smithi167 (mon.0) 252 : audit [INF] 2022-01-31T19:39:37.540 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:37 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-01-31T19:39:37.540 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:37 smithi167 conmon[32206]: audit 2022-01-31T19:39:37.096265+0000 mon.smithi167 (mon.0) 253 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:37.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:37 smithi171 conmon[35325]: audit 2022-01-31T19:39:37.095563 2022-01-31T19:39:37.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:37 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 252 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-01-31T19:39:37.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:37 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:39:37.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:37 smithi171 conmon[35325]: 31T19:39:37.096265+0000 mon.smithi167 (mon.0) 253 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:38.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:38 smithi167 conmon[32206]: cluster 2022-01-31T19:39:36.700879+0000 mgr.smithi167.aciqpk (mgr.14156) 87 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:38.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:38 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:37.096833+0000 mgr.smithi167.aciqpk (mgr.14156) 88 : cephadm [INF] Deploying daemon osd.1 on smithi167 2022-01-31T19:39:38.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:38 smithi171 conmon[35325]: cluster 2022-01-31T19:39:36.700879+0000 2022-01-31T19:39:38.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:38 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 87 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:38.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:38 smithi171 conmon[35325]: cephadm 2022-01-31T 2022-01-31T19:39:38.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:38 smithi171 conmon[35325]: 19:39:37.096833+0000 mgr.smithi167.aciqpk (mgr.14156) 88 : cephadm [INF] Deploying daemon osd.1 on smithi167 2022-01-31T19:39:40.137 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 1 on host 'smithi167' 2022-01-31T19:39:40.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.133+0000 7fbfff7fe700 1 -- 172.21.15.167:0/2439971350 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc0180fef00 con 0x7fc00005e5c0 2022-01-31T19:39:40.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.135+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc00005e5c0 msgr2=0x7fc000060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:40.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.135+0000 7fc01c966700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc00005e5c0 0x7fc000060a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc00c005010 tx=0x7fc00c009b30).stop 2022-01-31T19:39:40.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 msgr2=0x7fc0180f7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:40.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 0x7fc0180f7550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc00800c530 tx=0x7fc008007e20).stop 2022-01-31T19:39:40.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 shutdown_connections 2022-01-31T19:39:40.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc00005e5c0 0x7fc000060a80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:40.140 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0180f7c80 0x7fc0180f7550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 --2- 172.21.15.167:0/2439971350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0180f8590 0x7fc0180f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 >> 172.21.15.167:0/2439971350 conn(0x7fc0180f04e0 msgr2=0x7fc0180fb510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 shutdown_connections 2022-01-31T19:39:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:40.136+0000 7fc01c966700 1 -- 172.21.15.167:0/2439971350 wait complete. 2022-01-31T19:39:40.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:40 smithi167 conmon[32206]: cluster 2022-01-31T19:39:38.701266+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:39:40.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:40 smithi167 conmon[32206]: 89 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:40.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:40 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.130421+0000 mon.smithi167 (mon.0) 254 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:40.504 DEBUG:teuthology.orchestra.run.smithi167:osd.1> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@osd.1.service 2022-01-31T19:39:40.507 INFO:tasks.cephadm:Deploying osd.2 on smithi167 with /dev/vg_nvme/lv_2... 2022-01-31T19:39:40.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-01-31T19:39:40.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:-- Logs begin at Mon 2022-01-31 19:28:37 UTC. -- 2022-01-31T19:39:40.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:40 smithi171 conmon[35325]: cluster 2022-01-31T19:39:38.701266+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:39:40.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:40 smithi171 conmon[35325]: .14156) 89 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:40.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:40 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.130421+0000 2022-01-31T19:39:40.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:40 smithi171 conmon[35325]: mon.smithi167 (mon.0) 254 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:40.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 19:39:40 smithi167 conmon[54076]: debug 2022-01-31T19:39:40.556+0000 7f546c8b3080 -1 Falling back to public interface 2022-01-31T19:39:41.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.134060+0000 2022-01-31T19:39:41.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: mon.smithi167 (mon.0) 255 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.145594+0000 mon.smithi167 (mon.0) 256 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.167:9093"}]: dispatch 2022-01-31T19:39:41.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.146010+0000 mgr.smithi167.aciqpk (mgr.14156) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.167:9093"}]: dispatch 2022-01-31T19:39:41.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.151304+0000 mon.smithi167 (mon.0) 257 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.154579+0000 mon.smithi167 (mon.0) 258 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-01-31T19:39:41.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.154906+0000 mgr.smithi167.aciqpk (mgr.14156) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-01-31T19:39:41.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.159660+0000 mon.smithi167 (mon.0) 259 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.167:3000"}]: dispatch 2022-01-31T19:39:41.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.160034+0000 mgr.smithi167.aciqpk (mgr.14156) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.167:3000"}]: dispatch 2022-01-31T19:39:41.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.164618+0000 mon.smithi167 (mon.0) 260 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.167576+0000 mon.smithi167 (mon.0) 261 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-01-31T19:39:41.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.167960+0000 mgr.smithi167.aciqpk (mgr.14156) 93 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-01-31T19:39:41.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.173618+0000 mon.smithi167 (mon.0) 262 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.167:9095"}]: dispatch 2022-01-31T19:39:41.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.173962+0000 mgr.smithi167.aciqpk (mgr.14156) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.167:9095"}]: dispatch 2022-01-31T19:39:41.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.178078+0000 mon.smithi167 (mon.0) 263 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.180424+0000 mon.smithi167 (mon.0) 264 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:39:41.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.181604+0000 mon.smithi167 (mon.0) 265 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:41.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:41 smithi167 conmon[32206]: audit 2022-01-31T19:39:40.182411+0000 mon.smithi167 (mon.0) 266 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:39:41.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.134060+0000 mon.smithi167 (mon.0) 255 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.145594+0000 mon.smithi167 (mon.0) 256 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.167:9093"}]: dispatch 2022-01-31T19:39:41.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.146010+0000 mgr.smithi167.aciqpk (mgr.14156) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.167:9093"}]: dispatch 2022-01-31T19:39:41.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.151304+0000 mon.smithi167 (mon.0) 257 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.154579+0000 mon.smithi167 (mon.0) 258 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-01-31T19:39:41.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.154906+0000 mgr.smithi167.aciqpk (mgr.14156) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2022-01-31T19:39:41.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.159660+0000 mon.smithi167 (mon.0) 259 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.167:3000"}]: dispatch 2022-01-31T19:39:41.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.160034+0000 mgr.smithi167.aciqpk (mgr.14156) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.167:3000"}]: dispatch 2022-01-31T19:39:41.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.164618+0000 mon.smithi167 (mon.0) 260 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.167576+0000 mon.smithi167 (mon.0) 261 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-01-31T19:39:41.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.167960+0000 mgr.smithi167.aciqpk (mgr.14156) 93 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2022-01-31T19:39:41.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.173618+0000 mon.smithi167 (mon.0) 262 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.167:9095"}]: dispatch 2022-01-31T19:39:41.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.173962+0000 mgr.smithi167.aciqpk (mgr.14156) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.167:9095"}]: dispatch 2022-01-31T19:39:41.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.178078+0000 mon.smithi167 (mon.0) 263 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:41.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.180424+0000 mon.smithi167 (mon.0) 264 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:39:41.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.181604+0000 mon.smithi167 (mon.0) 265 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:41.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:41 smithi171 conmon[35325]: audit 2022-01-31T19:39:40.182411+0000 mon.smithi167 (mon.0) 266 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:39:42.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:42 smithi167 conmon[32206]: cluster 2022-01-31T19:39:40.701583+0000 mgr.smithi167.aciqpk (mgr.14156) 95 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:42.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:42 smithi171 conmon[35325]: cluster 2022-01-31T19:39:40.701583+0000 mgr.smithi167.aciqpk (mgr.14156) 95 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:42.649 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-01-31T19:39:42.649 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-01-31T19:39:42.650 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2022-01-31T19:39:42.650 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2022-01-31T19:39:42.651 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120369 s, 871 MB/s 2022-01-31T19:39:42.651 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2022-01-31T19:39:43.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi167:vg_nvme/lv_2 2022-01-31T19:39:43.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:43 smithi171 conmon[35325]: audit 2022-01-31T19:39:42.328043+0000 mon.smithi167 (mon.0) 267 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:43.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:43 smithi167 conmon[32206]: audit 2022-01-31T19:39:42.328043+0000 mon.smithi167 (mon.0) 267 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:44.338 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 19:39:44 smithi167 conmon[54076]: debug 2022-01-31T19:39:44.053+0000 7f546c8b3080 -1 osd.1 0 log_to_monitors {default=true} 2022-01-31T19:39:44.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:44 smithi171 conmon[35325]: cluster 2022-01-31T19:39:42.702026+0000 2022-01-31T19:39:44.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:44 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 96 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:44.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:44 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:39:44.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:44 smithi171 conmon[35325]: :39:44.058148+0000 mon.smithi167 (mon.0) 268 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-01-31T19:39:44.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:44 smithi167 conmon[32206]: cluster 2022-01-31T19:39:42.702026+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:39:44.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:44 smithi167 conmon[32206]: .14156) 96 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:44.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:44 smithi167 conmon[32206]: audit 2022-01-31T19:39:44.058148+0000 mon.smithi167 (mon.0) 268 : audit [INF] 2022-01-31T19:39:44.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:44 smithi167 conmon[32206]: from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-01-31T19:39:45.082 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 -- 172.21.15.167:0/3770136707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 msgr2=0x7f67780f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:45.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 --2- 172.21.15.167:0/3770136707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f67780f5b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f67700099e0 tx=0x7f6770004d10).stop 2022-01-31T19:39:45.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 -- 172.21.15.167:0/3770136707 shutdown_connections 2022-01-31T19:39:45.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 --2- 172.21.15.167:0/3770136707 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 0x7f67780f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:39:45.083 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 --2- 172.21.15.167:0/3770136707 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f67780f5b10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:45.084 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 -- 172.21.15.167:0/3770136707 >> 172.21.15.167:0/3770136707 conn(0x7f67780f0ef0 msgr2=0x7f67780f3310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:45.084 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 -- 172.21.15.167:0/3770136707 shutdown_connections 2022-01-31T19:39:45.084 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.078+0000 7f677f493700 1 -- 172.21.15.167:0/3770136707 wait complete. 2022-01-31T19:39:45.084 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677f493700 1 Processor -- start 2022-01-31T19:39:45.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677f493700 1 -- start start 2022-01-31T19:39:45.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677f493700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f6778107230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:45.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677f493700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 0x7f6778107720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:45.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677f493700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778107d60 con 0x7f67780f56f0 2022-01-31T19:39:45.085 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677f493700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778108610 con 0x7f67780f6000 2022-01-31T19:39:45.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677d22f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f6778107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:45.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677d22f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f6778107230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:60820/0 (socket says 172.21.15.167:60820) 2022-01-31T19:39:45.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677d22f700 1 -- 172.21.15.167:0/4177372416 learned_addr learned my addr 172.21.15.167:0/4177372416 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:39:45.086 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.079+0000 7f677ca2e700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 0x7f6778107720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:45.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677d22f700 1 -- 172.21.15.167:0/4177372416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 msgr2=0x7f6778107720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:45.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677d22f700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 0x7f6778107720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:45.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677d22f700 1 -- 172.21.15.167:0/4177372416 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6770009640 con 0x7f67780f56f0 2022-01-31T19:39:45.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677ca2e700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 0x7f6778107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:39:45.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677d22f700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f6778107230 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f6770003380 tx=0x7f6770004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:45.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f677000e070 con 0x7f67780f56f0 2022-01-31T19:39:45.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6778108890 con 0x7f67780f56f0 2022-01-31T19:39:45.088 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f67700049b0 con 0x7f67780f56f0 2022-01-31T19:39:45.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6770018070 con 0x7f67780f56f0 2022-01-31T19:39:45.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.080+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6778108d70 con 0x7f67780f56f0 2022-01-31T19:39:45.090 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.081+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67780f4b80 con 0x7f67780f56f0 2022-01-31T19:39:45.090 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.081+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f677001c760 con 0x7f67780f56f0 2022-01-31T19:39:45.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.082+0000 7f676dffb700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f676405e5c0 0x7f6764060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:39:45.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.082+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1813+0+0 (secure 0 0 0) 0x7f67700796e0 con 0x7f67780f56f0 2022-01-31T19:39:45.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.082+0000 7f677ca2e700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f676405e5c0 0x7f6764060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:39:45.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.082+0000 7f677ca2e700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f676405e5c0 0x7f6764060a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f676800a7e0 tx=0x7f6768008040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:39:45.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.086+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f677004c000 con 0x7f67780f56f0 2022-01-31T19:39:45.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:45.254+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f67780008d0 con 0x7f676405e5c0 2022-01-31T19:39:45.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: audit 2022-01-31T19:39:44.336330+0000 mon.smithi167 2022-01-31T19:39:45.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: (mon.0) 269 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-01-31T19:39:45.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: cluster 2022-01-31T19: 2022-01-31T19:39:45.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 39:44.336380+0000 mon.smithi167 (mon.0) 270 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2022-01-31T19:39:45.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:39:45.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 31T19:39:44.336689+0000 mon.smithi167 (mon.0) 271 2022-01-31T19:39:45.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:45.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:39:45.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 31T19:39:44.337042+0000 mon.smithi167 (mon.0) 2022-01-31T19:39:45.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 272 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2022-01-31T19:39:45.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: audit 2022-01-31T19:39:45.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 2022-01-31T19:39:45.257095+0000 mon.smithi167 2022-01-31T19:39:45.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: (mon.0) 273 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:39:45.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: audit 2022-01-31T19:39:45.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 2022-01-31T 2022-01-31T19:39:45.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 19:39:45.258275+0000 mon.smithi167 (mon.0) 2022-01-31T19:39:45.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 274 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:39:45.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: audit 2022-01-31T19:39:45.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: 2022-01-31T19:39:45.258791+0000 2022-01-31T19:39:45.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:45 smithi171 conmon[35325]: mon.smithi167 (mon.0) 275 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:45.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[54076]: debug 2022-01-31T19:39:45.340+0000 7f5462e16700 -1 osd.1 0 waiting for initial osdmap 2022-01-31T19:39:45.671 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[54076]: debug 2022-01-31T19:39:45.345+0000 7f545f592700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-01-31T19:39:45.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: audit 2022-01-31T19:39:44.336330+0000 mon.smithi167 (mon.0) 269 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-01-31T19:39:45.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: cluster 2022-01-31T19:39:44.336380+0000 mon.smithi167 (mon.0) 270 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2022-01-31T19:39:45.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: audit 2022-01-31T19:39:44.336689+0000 mon.smithi167 (mon.0) 271 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:45.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: audit 2022-01-31T19:39:44.337042+0000 mon.smithi167 (mon.0) 272 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2022-01-31T19:39:45.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.257095+0000 mon.smithi167 (mon.0) 273 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:39:45.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.258275+0000 mon.smithi167 (mon.0) 274 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:39:45.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:45 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.258791+0000 mon.smithi167 (mon.0) 275 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:46.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: cluster 2022-01-31T19:39:44.702503+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:39:46.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: 97 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:46.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.256369+0000 mgr.smithi167.aciqpk (mgr.14156) 98 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:39:46.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.338107+0000 mon.smithi167 (mon.0) 276 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2022-01-31T19:39:46.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: cluster 2022-01-31T19:39:45.338163+0000 mon.smithi167 (mon.0) 277 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2022-01-31T19:39:46.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.338269+0000 mon.smithi167 (mon.0) 278 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:46.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.343682+0000 mon.smithi167 (mon.0) 279 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:46.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:46 smithi167 conmon[32206]: audit 2022-01-31T19:39:45.898724+0000 mon.smithi167 (mon.0) 280 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:46.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: cluster 2022-01-31T19:39:44.702503+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:39:46.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: mgr.14156) 97 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:46.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: audit 2022-01-31T19:39:45.256369+0000 2022-01-31T19:39:46.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 98 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:39:46.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: audit 2022-01-31T19:39:45.338107+0000 2022-01-31T19:39:46.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: mon.smithi167 (mon.0) 276 : audit [INF] from='osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2022-01-31T19:39:46.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: cluster 2022-01-31T19:39:46.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: 2022-01-31T19:39:45.338163+0000 mon.smithi167 2022-01-31T19:39:46.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: (mon.0) 277 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2022-01-31T19:39:46.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: audit 2022-01-31T19:39:46.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: 2022-01-31T19:39:45.338269+0000 2022-01-31T19:39:46.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: mon.smithi167 (mon.0) 278 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:46.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: audit 2022-01-31T19:39:46.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: 2022-01-31T19:39:45.343682+0000 2022-01-31T19:39:46.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: mon.smithi167 (mon.0) 279 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:46.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: audit 2022-01-31T19:39:46.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: 2022-01-31T19:39:45.898724+0000 2022-01-31T19:39:46.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:46 smithi171 conmon[35325]: mon.smithi167 (mon.0) 280 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:47.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: cluster 2022-01-31T19:39:45.100916+0000 2022-01-31T19:39:47.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:39:47.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: cluster 2022-01-31T19 2022-01-31T19:39:47.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: :39:45.101015+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:39:47.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:39:47.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: T19:39:46.341626+0000 mon.smithi167 (mon.0) 2022-01-31T19:39:47.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: 281 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:47.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: cluster 2022-01-31T19: 2022-01-31T19:39:47.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: 39:46.347665+0000 mon.smithi167 (mon.0) 282 : cluster 2022-01-31T19:39:47.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: [INF] osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] boot 2022-01-31T19:39:47.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: cluster 2022-01-31T 2022-01-31T19:39:47.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: 19:39:46.347694+0000 mon.smithi167 (mon.0) 2022-01-31T19:39:47.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: 283 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2022-01-31T19:39:47.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:39:47.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: 31T19:39:46.347892+0000 mon.smithi167 (mon.0) 284 : 2022-01-31T19:39:47.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:47 smithi171 conmon[35325]: audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:47.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:47 smithi167 conmon[32206]: cluster 2022-01-31T19:39:45.100916+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:39:47.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:47 smithi167 conmon[32206]: cluster 2022-01-31T19:39:45.101015+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:39:47.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:47 smithi167 conmon[32206]: audit 2022-01-31T19:39:46.341626+0000 mon.smithi167 (mon.0) 281 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:47.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:47 smithi167 conmon[32206]: cluster 2022-01-31T19:39:46.347665+0000 mon.smithi167 (mon.0) 282 : cluster [INF] osd.1 [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] boot 2022-01-31T19:39:47.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:47 smithi167 conmon[32206]: cluster 2022-01-31T19:39:46.347694+0000 mon.smithi167 (mon.0) 283 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2022-01-31T19:39:47.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:47 smithi167 conmon[32206]: audit 2022-01-31T19:39:46.347892+0000 mon.smithi167 (mon.0) 284 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:39:48.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: cluster 2022-01-31T19:39:46.702760+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:39:48.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: .14156) 99 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:48.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: cluster 2022-01-31T19:39:47.349292+0000 mon.smithi167 (mon.0) 285 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2022-01-31T19:39:48.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: audit 2022-01-31T19:39:47.926784+0000 mon.smithi167 (mon.0) 286 : audit [INF] from='client.? 172.21.15.167:0/2734618341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3536e17c-9f9a-4867-91f4-f84b2b5d53a4"}]: dispatch 2022-01-31T19:39:48.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: audit 2022-01-31T19:39:47.930781+0000 mon.smithi167 (mon.0) 287 : audit [INF] from='client.? 172.21.15.167:0/2734618341' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3536e17c-9f9a-4867-91f4-f84b2b5d53a4"}]': finished 2022-01-31T19:39:48.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: cluster 2022-01-31T19:39:47.930823+0000 mon.smithi167 (mon.0) 288 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2022-01-31T19:39:48.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: audit 2022-01-31T19:39:47.930905+0000 mon.smithi167 (mon.0) 289 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:39:48.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:48 smithi167 conmon[32206]: audit 2022-01-31T19:39:48.342015+0000 mon.smithi167 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.167:0/817104764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:39:48.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: cluster 2022-01-31T19:39:46.702760+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:39:48.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: mgr.14156) 99 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2022-01-31T19:39:48.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: cluster 2022-01-31T19:39:47.349292+0000 mon.smithi167 ( 2022-01-31T19:39:48.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: mon.0) 285 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2022-01-31T19:39:48.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: audit 2022-01-31T19:39:47.926784+0000 mon.smithi167 ( 2022-01-31T19:39:48.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: mon.0) 286 : audit [INF] from='client.? 172.21.15.167:0/2734618341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3536e17c-9f9a-4867-91f4-f84b2b5d53a4"}]: dispatch 2022-01-31T19:39:48.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: audit 2022-01-31T19:39:48.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: 2022-01-31T19:39:47.930781+0000 mon.smithi167 (mon. 2022-01-31T19:39:48.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: 0) 287 : audit [INF] from='client.? 172.21.15.167:0/2734618341' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3536e17c-9f9a-4867-91f4-f84b2b5d53a4"}]': finished 2022-01-31T19:39:48.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: cluster 2022-01-31T19:39:48.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: 2022-01-31T19:39:47.930823+0000 mon.smithi167 ( 2022-01-31T19:39:48.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: mon.0) 288 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2022-01-31T19:39:48.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: audit 2022-01-31T19:39:48.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: 2022-01-31T19:39:47.930905+0000 mon.smithi167 (mon.0 2022-01-31T19:39:48.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: ) 289 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:39:48.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: audit 2022-01-31T19:39:48.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: 2022-01-31T19:39:48. 2022-01-31T19:39:48.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: 342015+0000 mon.smithi167 (mon.0) 290 : audit [DBG] 2022-01-31T19:39:48.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:48 smithi171 conmon[35325]: from='client.? 172.21.15.167:0/817104764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:39:50.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:50 smithi171 conmon[35325]: cluster 2022-01-31T19:39:48.703206+0000 mgr.smithi167.aciqpk 2022-01-31T19:39:50.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:50 smithi171 conmon[35325]: (mgr.14156) 100 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:50.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:50 smithi167 conmon[32206]: cluster 2022-01-31T19:39:48.703206+0000 mgr.smithi167.aciqpk (mgr.14156) 100 : 2022-01-31T19:39:50.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:50 smithi167 conmon[32206]: cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:51.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:51 smithi167 conmon[32206]: cluster 2022-01-31T19:39:50.703537+0000 mgr.smithi167.aciqpk (mgr.14156) 101 : cluster 2022-01-31T19:39:51.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:51 smithi167 conmon[32206]: [DBG] pgmap v61: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:52.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:51 smithi171 conmon[35325]: cluster 2022-01-31T19:39:50.703537+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:39:52.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:51 smithi171 conmon[35325]: mgr.14156) 101 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:53 smithi171 conmon[35325]: cluster 2022-01-31T19:39:52.703812+0000 mgr.smithi167.aciqpk (mgr.14156) 102 : cluster [DBG] 2022-01-31T19:39:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:53 smithi171 conmon[35325]: pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:54.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:53 smithi167 conmon[32206]: cluster 2022-01-31T19:39:52.703812+0000 mgr.smithi167.aciqpk (mgr.14156) 102 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:55.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:53.999218+0000 mgr.smithi167.aciqpk (mgr.14156) 103 : cephadm [INF] Detected new or changed devices on smithi167 2022-01-31T19:39:55.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.003684+0000 mon.smithi167 (mon.0) 291 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:55.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.369504+0000 mon.smithi167 (mon.0) 292 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:39:55.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.370229+0000 mon.smithi167 (mon.0) 293 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:39:55.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:54.370702+0000 mgr.smithi167.aciqpk (mgr.14156) 104 : cephadm [INF] Adjusting osd_memory_target on smithi167 to 6472M 2022-01-31T19:39:55.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.374396+0000 mon.smithi167 (mon.0) 294 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:55.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.378180+0000 mon.smithi167 (mon.0) 295 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:55.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.381073+0000 mon.smithi167 (mon.0) 296 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:39:55.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.382009+0000 mon.smithi167 (mon.0) 297 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:55.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.382493+0000 mon.smithi167 (mon.0) 298 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:39:55.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.687127+0000 mon.smithi167 (mon.0) 299 : 2022-01-31T19:39:55.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-01-31T19:39:55.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:55 smithi167 conmon[32206]: audit 2022-01-31T19:39:54.687723+0000 mon.smithi167 (mon.0) 300 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:55.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:53.999218+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:39:55.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 14156) 103 : cephadm [INF] Detected new or changed devices on smithi167 2022-01-31T19:39:55.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:54.003684+0000 mon.smithi167 (mon.0) 291 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:55.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39 2022-01-31T19:39:55.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: :54.369504+0000 mon.smithi167 (mon.0) 292 : audit 2022-01-31T19:39:55.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:39:55.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:55.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 2022-01-31T19:39:54.370229+0000 2022-01-31T19:39:55.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: mon.smithi167 (mon.0) 293 : audit [INF] 2022-01-31T19:39:55.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:39:55.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: cephadm 2022- 2022-01-31T19:39:55.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 01-31T19:39:54.370702+0000 mgr.smithi167.aciqpk 2022-01-31T19:39:55.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: (mgr.14156) 104 : cephadm [INF] Adjusting osd_memory_target on smithi167 to 6472M 2022-01-31T19:39:55.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:55.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 2022-01-31T19:39:54.374396+0000 mon.smithi167 2022-01-31T19:39:55.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: (mon.0) 294 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:55.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:39:55.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: -31T19:39:54.378180+0000 mon.smithi167 (mon 2022-01-31T19:39:55.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: .0) 295 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:55.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:55.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 2022-01-31T19:39:54.381073 2022-01-31T19:39:55.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 296 : audit [DBG] 2022-01-31T19:39:55.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:39:55.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:39:55.364 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: -31T19:39:54.382009+0000 mon.smithi167 ( 2022-01-31T19:39:55.364 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: mon.0) 297 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:55.364 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:55.364 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 2022-01-31T19:39: 2022-01-31T19:39:55.364 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 54.382493+0000 mon.smithi167 (mon.0) 298 : 2022-01-31T19:39:55.365 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:39:55.365 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:55.365 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: 2022-01-31T19:39:54.687127+0000 2022-01-31T19:39:55.365 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: mon.smithi167 (mon.0) 299 : audit [INF] 2022-01-31T19:39:55.366 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-01-31T19:39:55.366 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: audit 2022-01-31T19:39:54 2022-01-31T19:39:55.366 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: .687723+0000 mon.smithi167 (mon.0) 300 : audit 2022-01-31T19:39:55.366 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:55 smithi171 conmon[35325]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:39:56.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:56 smithi167 conmon[32206]: cephadm 2022-01-31T19:39:54.688186+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:39:56.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:56 smithi167 conmon[32206]: ) 105 : cephadm [INF] Deploying daemon osd.2 on smithi167 2022-01-31T19:39:56.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:56 smithi167 conmon[32206]: cluster 2022-01-31T19:39:54.704101+0000 mgr.smithi167.aciqpk (mgr.14156) 106 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:56.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:56 smithi171 conmon[35325]: cephadm 2022-01-31T19:39:54.688186+0000 mgr.smithi167.aciqpk (mgr.14156) 105 : cephadm [INF] Deploying daemon osd.2 on smithi167 2022-01-31T19:39:56.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:56 smithi171 conmon[35325]: cluster 2022-01-31T19: 2022-01-31T19:39:56.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:56 smithi171 conmon[35325]: 39:54.704101+0000 mgr.smithi167.aciqpk (mgr.14156) 106 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:57.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:57 smithi171 conmon[35325]: audit 2022-01-31T19:39:56.532428+0000 mon.smithi167 (mon.0) 301 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:57.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:57 smithi171 conmon[35325]: 2022-01-31T19:39:57.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:57 smithi171 conmon[35325]: cluster 2022-01-31T19:39:56.704427+0000 mgr.smithi167.aciqpk (mgr.14156) 107 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:57.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:57 smithi167 conmon[32206]: audit 2022-01-31T19:39:56.532428+0000 mon.smithi167 ( 2022-01-31T19:39:57.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:57 smithi167 conmon[32206]: mon.0) 301 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:57.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:57 smithi167 conmon[32206]: cluster 2022-01-31T19:39:56.704427+0000 mgr.smithi167.aciqpk (mgr.14156) 107 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:39:58.271 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 2 on host 'smithi167' 2022-01-31T19:39:58.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.269+0000 7f676dffb700 1 -- 172.21.15.167:0/4177372416 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f67780008d0 con 0x7f676405e5c0 2022-01-31T19:39:58.273 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f676405e5c0 msgr2=0x7f6764060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:58.273 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f676405e5c0 0x7f6764060a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f676800a7e0 tx=0x7f6768008040).stop 2022-01-31T19:39:58.274 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 msgr2=0x7f6778107230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:39:58.274 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f6778107230 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f6770003380 tx=0x7f6770004d10).stop 2022-01-31T19:39:58.274 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 shutdown_connections 2022-01-31T19:39:58.275 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f676405e5c0 0x7f6764060a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:58.275 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67780f56f0 0x7f6778107230 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:58.275 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 --2- 172.21.15.167:0/4177372416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67780f6000 0x7f6778107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:39:58.275 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 >> 172.21.15.167:0/4177372416 conn(0x7f67780f0ef0 msgr2=0x7f67780fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:39:58.275 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 shutdown_connections 2022-01-31T19:39:58.276 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:39:58.271+0000 7f677f493700 1 -- 172.21.15.167:0/4177372416 wait complete. 2022-01-31T19:39:58.697 DEBUG:teuthology.orchestra.run.smithi167:osd.2> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@osd.2.service 2022-01-31T19:39:58.701 INFO:tasks.cephadm:Deploying osd.3 on smithi171 with /dev/vg_nvme/lv_4... 2022-01-31T19:39:58.702 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-01-31T19:39:58.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:-- Logs begin at Mon 2022-01-31 19:28:37 UTC. -- 2022-01-31T19:39:59.512 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:59 smithi171 conmon[35325]: audit 2022-01-31T19:39:58.265868+0000 mon.smithi167 (mon.0) 302 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:59.512 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:59 smithi171 conmon[35325]: audit 2022-01-31T19:39:58.269979+0000 mon.smithi167 (mon.0) 303 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:59.513 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:59 smithi171 conmon[35325]: audit 2022-01-31T19:39:58.823974+0000 mon.smithi167 (mon.0) 304 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:39:59.513 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:39:59 smithi171 conmon[35325]: audit 2022-01-31T19:39:58.824747+0000 mon.smithi167 (mon.0) 305 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:39:59.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:59 smithi167 conmon[32206]: audit 2022-01-31T19:39:58.265868+0000 mon.smithi167 (mon.0) 302 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:59.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:59 smithi167 conmon[32206]: audit 2022-01-31T19:39:58.269979+0000 mon.smithi167 (mon.0) 303 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:39:59.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:59 smithi167 conmon[32206]: audit 2022-01-31T19:39:58.823974+0000 mon.smithi167 (mon.0) 304 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:39:59.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:39:59 smithi167 conmon[32206]: audit 2022-01-31T19:39:58.824747+0000 mon.smithi167 (mon.0) 305 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:40:00.333 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-01-31T19:40:00.333 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-01-31T19:40:00.333 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2022-01-31T19:40:00.334 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2022-01-31T19:40:00.334 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112934 s, 928 MB/s 2022-01-31T19:40:00.334 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2022-01-31T19:40:00.463 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:00 smithi167 conmon[32206]: cluster 2022-01-31T19:39:58.704951+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:40:00.464 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:00 smithi167 conmon[32206]: ) 108 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:40:00.464 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:00 smithi167 conmon[32206]: cluster 2022-01-31T19:40:00.000141+0000 mon.smithi167 (mon.0) 306 : cluster [INF] overall HEALTH_OK 2022-01-31T19:40:00.465 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:00 smithi167 conmon[32206]: audit 2022-01-31T19:40:00.008346+0000 mon.smithi167 (mon.0) 307 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:00.529 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:00 smithi171 conmon[35325]: cluster 2022-01-31T19:39:58.704951+0000 mgr.smithi167.aciqpk (mgr.14156) 108 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:40:00.529 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:00 smithi171 conmon[35325]: cluster 2022-01-31T19:40:00 2022-01-31T19:40:00.529 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:00 smithi171 conmon[35325]: .000141+0000 mon.smithi167 (mon.0) 306 : cluster [INF] overall HEALTH_OK 2022-01-31T19:40:00.529 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:00 smithi171 conmon[35325]: audit 2022-01-31T19:40:00.008346+0000 mon.smithi167 (mon.0) 307 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:00.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi171:vg_nvme/lv_4 2022-01-31T19:40:01.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:01 smithi167 conmon[32206]: cluster 2022-01-31T19:40:00.705225+0000 mgr.smithi167.aciqpk (mgr.14156) 109 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:40:02.024 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:01 smithi171 conmon[35325]: cluster 2022-01-31T19:40:00.705225+0000 mgr.smithi167.aciqpk (mgr.14156) 109 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:40:02.280 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 -- 172.21.15.171:0/2402752552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f6980 msgr2=0x7f81f80f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:02.280 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 --2- 172.21.15.171:0/2402752552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f6980 0x7f81f80f6de0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f81f4004660 tx=0x7f81f4009b30).stop 2022-01-31T19:40:02.281 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 -- 172.21.15.171:0/2402752552 shutdown_connections 2022-01-31T19:40:02.281 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 --2- 172.21.15.171:0/2402752552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f6980 0x7f81f80f6de0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:02.281 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 --2- 172.21.15.171:0/2402752552 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f80f56e0 0x7f81f80f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:40:02.281 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 -- 172.21.15.171:0/2402752552 >> 172.21.15.171:0/2402752552 conn(0x7f81f80f0e60 msgr2=0x7f81f80f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:02.282 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 -- 172.21.15.171:0/2402752552 shutdown_connections 2022-01-31T19:40:02.282 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.277+0000 7f81ff762700 1 -- 172.21.15.171:0/2402752552 wait complete. 2022-01-31T19:40:02.282 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81ff762700 1 Processor -- start 2022-01-31T19:40:02.282 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81ff762700 1 -- start start 2022-01-31T19:40:02.282 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81ff762700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 0x7f81f8105fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:02.283 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81ff762700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f80f6980 0x7f81f81064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:02.283 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81ff762700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f8106b00 con 0x7f81f80f6980 2022-01-31T19:40:02.283 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81ff762700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81073b0 con 0x7f81f80f56e0 2022-01-31T19:40:02.283 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 0x7f81f8105fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:02.284 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 0x7f81f8105fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:49160/0 (socket says 172.21.15.171:49160) 2022-01-31T19:40:02.284 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 -- 172.21.15.171:0/304157693 learned_addr learned my addr 172.21.15.171:0/304157693 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:40:02.284 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fccfd700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f80f6980 0x7f81f81064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:02.284 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 -- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f80f6980 msgr2=0x7f81f81064c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:02.285 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f80f6980 0x7f81f81064c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:02.285 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 -- 172.21.15.171:0/304157693 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4005040 con 0x7f81f80f56e0 2022-01-31T19:40:02.285 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81fd4fe700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 0x7f81f8105fd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f81e80093f0 tx=0x7f81e8006ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:02.285 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e8013640 con 0x7f81f80f56e0 2022-01-31T19:40:02.285 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f81e8013c30 con 0x7f81f80f56e0 2022-01-31T19:40:02.286 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.278+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e8012be0 con 0x7f81f80f56e0 2022-01-31T19:40:02.286 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.279+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f8107610 con 0x7f81f80f56e0 2022-01-31T19:40:02.286 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.279+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f8107b80 con 0x7f81f80f56e0 2022-01-31T19:40:02.287 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.280+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f8045bb0 con 0x7f81f80f56e0 2022-01-31T19:40:02.287 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.280+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f81e801a060 con 0x7f81f80f56e0 2022-01-31T19:40:02.288 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.280+0000 7f81edffb700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f81e405e5c0 0x7f81e4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:02.288 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.280+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2224+0+0 (secure 0 0 0) 0x7f81e8077d00 con 0x7f81f80f56e0 2022-01-31T19:40:02.288 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.280+0000 7f81fccfd700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f81e405e5c0 0x7f81e4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:02.288 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.281+0000 7f81fccfd700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f81e405e5c0 0x7f81e4060a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f81f4005010 tx=0x7f81f4009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:02.289 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.284+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f81e807b030 con 0x7f81f80f56e0 2022-01-31T19:40:02.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 19:40:02 smithi167 conmon[60316]: debug 2022-01-31T19:40:02.077+0000 7f430c021080 -1 osd.2 0 log_to_monitors {default=true} 2022-01-31T19:40:02.443 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:02.442+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f81f80ff9c0 con 0x7f81e405e5c0 2022-01-31T19:40:02.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:02 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.080307+0000 mon.smithi167 (mon.0) 308 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-01-31T19:40:02.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:02 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.444271+0000 mon.smithi167 (mon.0) 309 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:40:02.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:02 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.445563+0000 mon.smithi167 (mon.0) 310 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:40:02.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:02 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.446073+0000 mon.smithi167 (mon.0) 311 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:02.899 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:02 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.080307+0000 mon.smithi167 (mon.0) 308 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-01-31T19:40:02.899 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:02 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.444271+0000 mon.smithi167 (mon.0) 309 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:40:02.900 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:02 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.445563+0000 mon.smithi167 (mon.0) 310 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:40:02.900 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:02 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.446073+0000 mon.smithi167 (mon.0) 311 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:02.900 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:02 smithi171 conmon[35325]: 2022-01-31T19:40:03.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.443520+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:40:03.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: mgr.14156) 110 : audit [DBG] from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:40:03.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.626096+0000 mon.smithi167 (mon.0) 312 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-01-31T19:40:03.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: cluster 2022-01-31T19:40:02.626149+0000 mon.smithi167 (mon.0) 313 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2022-01-31T19:40:03.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.626269+0000 mon.smithi167 (mon.0) 314 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:03.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.626618+0000 mon.smithi167 (mon.0) 315 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2022-01-31T19:40:03.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: cluster 2022-01-31T19:40:02.705550+0000 mgr.smithi167.aciqpk (mgr.14156) 111 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:40:03.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.921563+0000 mon.smithi167 (mon.0) 316 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:03.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.924654+0000 mon.smithi167 (mon.0) 317 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:03.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.927375+0000 mon.smithi167 (mon.0) 318 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:03.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.928292+0000 mon.smithi167 (mon.0) 319 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:03.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:03 smithi171 conmon[35325]: audit 2022-01-31T19:40:02.928737+0000 mon.smithi167 (mon.0) 320 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:03.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[60316]: debug 2022-01-31T19:40:03.630+0000 7f4302584700 -1 osd.2 0 waiting for initial osdmap 2022-01-31T19:40:03.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[60316]: debug 2022-01-31T19:40:03.635+0000 7f42fed00700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-01-31T19:40:03.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.443520+0000 mgr.smithi167.aciqpk (mgr.14156) 110 : 2022-01-31T19:40:03.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit [DBG] from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:40:03.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.626096+0000 mon.smithi167 (mon.0) 312 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-01-31T19:40:03.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: cluster 2022-01-31T19:40:02.626149+0000 mon.smithi167 (mon.0) 313 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2022-01-31T19:40:03.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.626269+0000 mon.smithi167 (mon.0) 314 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:03.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.626618+0000 mon.smithi167 (mon.0) 315 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2022-01-31T19:40:03.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: cluster 2022-01-31T19:40:02.705550+0000 mgr.smithi167.aciqpk (mgr.14156) 111 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2022-01-31T19:40:03.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19:40:02.921563+0000 mon.smithi167 (mon.0) 316 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:03.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:40:03.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: :40:02.924654+0000 mon.smithi167 (mon.0) 317 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:03.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: 2022-01-31T19:40:03.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:03.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: 01-31T19:40:02.927375+0000 mon.smithi167 (mon.0 2022-01-31T19:40:03.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: ) 318 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:03.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:40:03.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: :40:02.928292+0000 mon.smithi167 (mon.0 2022-01-31T19:40:03.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: ) 319 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:03.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:40:03.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: :40:02.928737+0000 mon.smithi167 (mon.0 2022-01-31T19:40:03.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:03 smithi167 conmon[32206]: ) 320 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:04.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:03.627725+0000 2022-01-31T19:40:04.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: mon.smithi167 (mon.0) 321 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2022-01-31T19:40:04.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: cluster 2022-01-31T19:40:03.627783+0000 mon.smithi167 (mon.0) 322 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2022-01-31T19:40:04.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:03.628097+0000 mon.smithi167 (mon.0) 323 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:04.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:03.631274+0000 mon.smithi167 (mon.0) 324 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:04.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:04.361326+0000 mon.smithi171 (mon.1) 2 : audit [INF] from='client.? 172.21.15.171:0/3785541606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e924c508-4baf-4ed0-9a32-42a51710ca8b"}]: dispatch 2022-01-31T19:40:04.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:04.361734+0000 mon.smithi167 (mon.0) 325 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e924c508-4baf-4ed0-9a32-42a51710ca8b"}]: dispatch 2022-01-31T19:40:04.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: cluster 2022-01-31T19:40:04.366109+0000 mon.smithi167 (mon.0) 326 : cluster [INF] osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] boot 2022-01-31T19:40:04.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:04.366152+0000 mon.smithi167 (mon.0) 327 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e924c508-4baf-4ed0-9a32-42a51710ca8b"}]': finished 2022-01-31T19:40:04.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: cluster 2022-01-31T19:40:04.366201+0000 mon.smithi167 (mon.0) 328 : cluster [DBG] osdmap e18: 4 total, 3 up, 4 in 2022-01-31T19:40:04.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:04.366311+0000 mon.smithi167 (mon.0) 329 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:04.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:04 smithi167 conmon[32206]: audit 2022-01-31T19:40:04.366736+0000 mon.smithi167 (mon.0) 330 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:05.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:03.627725+0000 mon.smithi167 (mon.0) 321 : audit [INF] from='osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2022-01-31T19:40:05.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: cluster 2022-01-31T19:40:03.627783+0000 mon.smithi167 (mon.0) 322 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2022-01-31T19:40:05.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:03.628097+0000 mon.smithi167 (mon.0) 323 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:05.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:03.631274+0000 mon.smithi167 (mon.0) 324 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:05.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:04.361326+0000 mon.smithi171 (mon.1) 2 : audit [INF] from='client.? 172.21.15.171:0/3785541606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e924c508-4baf-4ed0-9a32-42a51710ca8b"}]: dispatch 2022-01-31T19:40:05.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:04.361734+0000 mon.smithi167 (mon.0) 325 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e924c508-4baf-4ed0-9a32-42a51710ca8b"}]: dispatch 2022-01-31T19:40:05.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: cluster 2022-01-31T19:40:04.366109+0000 mon.smithi167 (mon.0) 326 : cluster [INF] osd.2 [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] boot 2022-01-31T19:40:05.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:04.366152+0000 mon.smithi167 (mon.0) 327 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e924c508-4baf-4ed0-9a32-42a51710ca8b"}]': finished 2022-01-31T19:40:05.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: cluster 2022-01-31T19:40:04.366201+0000 mon.smithi167 (mon.0) 328 : cluster [DBG] osdmap e18: 4 total, 3 up, 4 in 2022-01-31T19:40:05.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:40:05.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: 31T19:40:04.366311+0000 mon.smithi167 (mon.0) 329 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:40:05.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:04 smithi171 conmon[35325]: audit 2022-01-31T19:40:04.366736+0000 mon.smithi167 (mon.0) 330 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:05.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:05 smithi167 conmon[32206]: cluster 2022-01-31T19:40:03.040826+0000 osd.2 (osd.2) 1 : cluster 2022-01-31T19:40:05.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:05 smithi167 conmon[32206]: [DBG] purged_snaps scrub starts 2022-01-31T19:40:05.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:05 smithi167 conmon[32206]: cluster 2022-01-31T19:40:03.040925+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:05.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:05 smithi167 conmon[32206]: cluster 2022-01-31T19:40:04.705879+0000 mgr.smithi167.aciqpk (mgr.14156) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:05.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:05 smithi167 conmon[32206]: audit 2022-01-31T19:40:04.805342+0000 mon.smithi171 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.171:0/1224819566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:40:05.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:05 smithi167 conmon[32206]: audit 2022-01-31T19:40:05.259805+0000 mon.smithi167 (mon.0) 331 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:05 smithi171 conmon[35325]: cluster 2022-01-31T19:40:03.040826+0000 osd.2 (osd.2) 1 2022-01-31T19:40:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:05 smithi171 conmon[35325]: : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:05 smithi171 conmon[35325]: cluster 2022-01-31T19:40:03.040925+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:05 smithi171 conmon[35325]: cluster 2022-01-31T19:40:04.705879+0000 mgr.smithi167.aciqpk (mgr.14156) 112 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:06.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:05 smithi171 conmon[35325]: audit 2022-01-31T19:40:04.805342+0000 mon.smithi171 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.171:0/1224819566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:40:06.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:05 smithi171 conmon[35325]: audit 2022-01-31T19:40:05.259805+0000 mon.smithi167 (mon.0) 331 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:06.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:06 smithi167 conmon[32206]: cluster 2022-01-31T19:40:05.637294+0000 2022-01-31T19:40:06.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:06 smithi167 conmon[32206]: mon.smithi167 (mon.0) 332 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2022-01-31T19:40:06.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:06 smithi167 conmon[32206]: audit 2022-01-31T19:40:05.637579+0000 2022-01-31T19:40:06.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:06 smithi167 conmon[32206]: mon.smithi167 (mon.0) 333 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:07.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:06 smithi171 conmon[35325]: cluster 2022-01-31T19:40:05.637294+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:07.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:06 smithi171 conmon[35325]: 332 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2022-01-31T19:40:07.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:06 smithi171 conmon[35325]: audit 2022-01-31T19:40:05.637579+0000 mon.smithi167 (mon.0) 333 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:07.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:07 smithi167 conmon[32206]: cluster 2022-01-31T19:40:06.706199+0000 mgr.smithi167.aciqpk (mgr.14156) 113 : cluster [DBG] 2022-01-31T19:40:07.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:07 smithi167 conmon[32206]: pgmap v73: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:07.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:07 smithi167 conmon[32206]: audit 2022-01-31T19:40:07.517703+0000 mon.smithi167 (mon.0) 334 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:08.051 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:07 smithi171 conmon[35325]: cluster 2022-01-31T19:40:06.706199+0000 mgr.smithi167.aciqpk (mgr.14156) 113 : cluster 2022-01-31T19:40:08.445 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:07 smithi171 conmon[35325]: [DBG] pgmap v73: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:08.445 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:07 smithi171 conmon[35325]: audit 2022-01-31T19:40:07.517703+0000 mon.smithi167 (mon.0) 334 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:09 smithi171 conmon[35325]: cluster 2022-01-31T19:40:08.706562+0000 mgr.smithi167.aciqpk (mgr.14156) 114 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:10.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:09 smithi167 conmon[32206]: cluster 2022-01-31T19:40:08.706562+0000 mgr.smithi167.aciqpk (mgr.14156) 114 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:11.523 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:11 smithi171 conmon[35325]: audit 2022-01-31T19:40:10.422116+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:11.523 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:11 smithi171 conmon[35325]: 335 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:11.523 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:11 smithi171 conmon[35325]: audit 2022-01-31T19:40:10.427172+0000 mon.smithi167 (mon.0) 336 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:11.524 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:11 smithi171 conmon[35325]: audit 2022-01-31T19:40:10.427254+0000 mon.smithi167 (mon.0) 337 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-01-31T19:40:11.524 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:11 smithi171 conmon[35325]: audit 2022-01-31T19:40:10.428723+0000 mon.smithi167 (mon.0) 338 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:11.524 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:11 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:10.429907+0000 mgr.smithi167.aciqpk (mgr.14156) 115 : cephadm [INF] Deploying daemon osd.3 on smithi171 2022-01-31T19:40:11.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:11 smithi167 conmon[32206]: audit 2022-01-31T19:40:10.422116+0000 mon.smithi167 (mon.0) 335 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:11.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:11 smithi167 conmon[32206]: 2022-01-31T19:40:11.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:11 smithi167 conmon[32206]: audit 2022-01-31T19:40:10.427172+0000 mon.smithi167 (mon.0) 336 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:11.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:11 smithi167 conmon[32206]: audit 2022-01-31T19:40:10.427254+0000 mon.smithi167 (mon.0) 337 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-01-31T19:40:11.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:11 smithi167 conmon[32206]: audit 2022-01-31T19:40:10.428723+0000 mon.smithi167 (mon.0) 338 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:11.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:11 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:10.429907+0000 mgr.smithi167.aciqpk (mgr.14156) 115 : cephadm [INF] Deploying daemon osd.3 on smithi171 2022-01-31T19:40:12.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:12 smithi171 conmon[35325]: cluster 2022-01-31T19:40:10.706788+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:40:12.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:12 smithi171 conmon[35325]: 14156) 116 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:12.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:12 smithi167 conmon[32206]: cluster 2022-01-31T19:40:10.706788+0000 mgr.smithi167.aciqpk (mgr.14156) 116 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:13.127 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 3 on host 'smithi171' 2022-01-31T19:40:13.128 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.124+0000 7f81edffb700 1 -- 172.21.15.171:0/304157693 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f81f80ff9c0 con 0x7f81e405e5c0 2022-01-31T19:40:13.128 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.126+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f81e405e5c0 msgr2=0x7f81e4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:13.129 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.126+0000 7f81ff762700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f81e405e5c0 0x7f81e4060a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f81f4005010 tx=0x7f81f4009b30).stop 2022-01-31T19:40:13.129 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.126+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 msgr2=0x7f81f8105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:13.129 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.126+0000 7f81ff762700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 0x7f81f8105fd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f81e80093f0 tx=0x7f81e8006ec0).stop 2022-01-31T19:40:13.129 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 shutdown_connections 2022-01-31T19:40:13.130 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f81e405e5c0 0x7f81e4060a80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:13.130 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81f80f56e0 0x7f81f8105fd0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:13.130 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 --2- 172.21.15.171:0/304157693 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f80f6980 0x7f81f81064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:13.131 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 >> 172.21.15.171:0/304157693 conn(0x7f81f80f0e60 msgr2=0x7f81f80fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:13.131 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 shutdown_connections 2022-01-31T19:40:13.131 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:13.127+0000 7f81ff762700 1 -- 172.21.15.171:0/304157693 wait complete. 2022-01-31T19:40:13.464 DEBUG:teuthology.orchestra.run.smithi171:osd.3> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@osd.3.service 2022-01-31T19:40:13.466 INFO:tasks.cephadm:Deploying osd.4 on smithi171 with /dev/vg_nvme/lv_3... 2022-01-31T19:40:13.467 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-01-31T19:40:13.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:-- Logs begin at Mon 2022-01-31 19:28:36 UTC. -- 2022-01-31T19:40:13.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[41853]: debug 2022-01-31T19:40:13.524+0000 7f940d724080 -1 Falling back to public interface 2022-01-31T19:40:13.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: audit 2022-01-31T19:40:13.113358+0000 mon.smithi167 (mon.0) 339 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:13.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: audit 2022-01-31T19:40:13 2022-01-31T19:40:13.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: .116542+0000 mon.smithi167 (mon.0) 340 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:13.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: audit 2022-01-31T19:40:13.116796+0000 mon.smithi167 (mon.0) 341 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:13.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: audit 2022-01-31T19:40:13.118378+0000 mon.smithi167 (mon.0) 342 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:13.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: audit 2022-01-31T19:40:13.122550+0000 mon.smithi167 (mon.0) 343 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:13.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:13 smithi171 conmon[35325]: audit 2022-01-31T19:40:13.128059+0000 mon.smithi167 (mon.0) 344 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2022-01-31T19:40:13.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: audit 2022-01-31T19:40:13.113358+0000 mon.smithi167 (mon 2022-01-31T19:40:13.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: .0) 339 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:13.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: audit 2022-01-31T19:40:13.116542+0000 mon.smithi167 (mon.0 2022-01-31T19:40:13.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: ) 340 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:13.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:40:13.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: -31T19:40:13.116796+0000 mon.smithi167 (mon.0) 341 : audit 2022-01-31T19:40:13.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:13.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:40:13.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: 31T19:40:13.118378+0000 2022-01-31T19:40:13.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: mon.smithi167 (mon.0) 342 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:13.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: audit 2022-01-31T19:40:13.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: 2022-01-31T19:40:13.122550+0000 2022-01-31T19:40:13.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: mon.smithi167 (mon.0) 343 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:13.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: audit 2022-01-31T19:40:13.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: 2022-01-31T19:40:13. 2022-01-31T19:40:13.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:13 smithi167 conmon[32206]: 128059+0000 mon.smithi167 (mon.0) 344 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2022-01-31T19:40:14.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:14 smithi171 conmon[35325]: cluster 2022-01-31T19:40:12.707108+0000 mgr.smithi167.aciqpk (mgr.14156) 117 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:14.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:14 smithi171 conmon[35325]: audit 2022-01-31T19:40:14.123651+0000 mon.smithi167 (mon.0) 345 : 2022-01-31T19:40:14.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:14 smithi171 conmon[35325]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2022-01-31T19:40:14.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:14 smithi171 conmon[35325]: cluster 2022-01-31T19:40:14.123699+0000 mon.smithi167 (mon.0) 346 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2022-01-31T19:40:14.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:14 smithi171 conmon[35325]: audit 2022-01-31T19:40:14.123800+0000 mon.smithi167 (mon.0) 347 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:14.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:14 smithi171 conmon[35325]: audit 2022-01-31T19:40:14.124509+0000 mon.smithi167 (mon.0) 348 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2022-01-31T19:40:14.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:14 smithi167 conmon[32206]: cluster 2022-01-31T19:40:12.707108+0000 mgr.smithi167.aciqpk (mgr.14156) 117 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:14.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:14 smithi167 conmon[32206]: audit 2022-01-31T19:40:14.123651+0000 mon.smithi167 (mon.0) 345 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2022-01-31T19:40:14.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:14 smithi167 conmon[32206]: cluster 2022-01-31T19:40:14.123699+0000 mon.smithi167 (mon.0) 346 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2022-01-31T19:40:14.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:14 smithi167 conmon[32206]: audit 2022-01-31T19:40:14.123800+0000 mon.smithi167 (mon.0) 347 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:14.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:14 smithi167 conmon[32206]: audit 2022-01-31T19:40:14.124509+0000 mon.smithi167 (mon.0) 348 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2022-01-31T19:40:15.411 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-01-31T19:40:15.411 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-01-31T19:40:15.411 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2022-01-31T19:40:15.412 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2022-01-31T19:40:15.412 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127797 s, 821 MB/s 2022-01-31T19:40:15.412 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2022-01-31T19:40:15.830 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi171:vg_nvme/lv_3 2022-01-31T19:40:16.385 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:16 smithi171 conmon[35325]: cluster 2022-01-31T19:40:14.707376+0000 mgr.smithi167.aciqpk (mgr.14156) 118 : cluster [DBG] pgmap v78: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:16.386 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:16 smithi171 conmon[35325]: audit 2022-01-31T19:40:15.127600+0000 mon.smithi167 (mon.0) 349 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2022-01-31T19:40:16.386 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:16 smithi171 conmon[35325]: cluster 2022-01-31T19:40:15.127671+0000 mon.smithi167 (mon.0) 350 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2022-01-31T19:40:16.387 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:16 smithi171 conmon[35325]: audit 2022-01-31T19:40:15.127785+0000 mon.smithi167 (mon.0) 351 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:16.387 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:16 smithi171 conmon[35325]: audit 2022-01-31T19:40:16.081721+0000 mon.smithi167 (mon.0) 352 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:16.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: cluster 2022-01-31T19:40:14.707376+0000 mgr.smithi167.aciqpk (mgr.14156) 118 : cluster [DBG] pgmap v78: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:16.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: audit 2022-01-31T19:40:15.127600+0000 mon.smithi167 (mon.0) 349 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2022-01-31T19:40:16.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: cluster 2022-01-31T19:40:15.127671+0000 mon.smithi167 (mon.0) 350 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2022-01-31T19:40:16.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: audit 2022-01-31T19:40:15.127785+0000 mon.smithi167 (mon.0) 351 : audit 2022-01-31T19:40:16.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:16.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: audit 2022-01-31T19:40:16.081721+0000 mon.smithi167 (mon.0) 352 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:16.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:16 smithi167 conmon[32206]: 2022-01-31T19:40:17.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[41853]: debug 2022-01-31T19:40:17.044+0000 7f940d724080 -1 osd.3 0 log_to_monitors {default=true} 2022-01-31T19:40:17.558 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 -- 172.21.15.171:0/3379147349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f56f0 msgr2=0x7fde5c0f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:17.559 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 --2- 172.21.15.171:0/3379147349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f56f0 0x7fde5c0f5b10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fde50004660 tx=0x7fde50009b30).stop 2022-01-31T19:40:17.559 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 -- 172.21.15.171:0/3379147349 shutdown_connections 2022-01-31T19:40:17.559 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 --2- 172.21.15.171:0/3379147349 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f6990 0x7fde5c0f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:17.560 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 --2- 172.21.15.171:0/3379147349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f56f0 0x7fde5c0f5b10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:17.560 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 -- 172.21.15.171:0/3379147349 >> 172.21.15.171:0/3379147349 conn(0x7fde5c0f0e50 msgr2=0x7fde5c0f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:17.560 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 -- 172.21.15.171:0/3379147349 shutdown_connections 2022-01-31T19:40:17.561 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 -- 172.21.15.171:0/3379147349 wait complete. 2022-01-31T19:40:17.561 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.555+0000 7fde62efe700 1 Processor -- start 2022-01-31T19:40:17.561 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde62efe700 1 -- start start 2022-01-31T19:40:17.561 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde62efe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f56f0 0x7fde5c10c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:17.561 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde62efe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 0x7fde5c10e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:17.562 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde62efe700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c060620 con 0x7fde5c0f56f0 2022-01-31T19:40:17.562 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde62efe700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c060760 con 0x7fde5c0f6990 2022-01-31T19:40:17.562 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde60c9a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f56f0 0x7fde5c10c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:17.563 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde5bfff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 0x7fde5c10e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:17.563 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde60c9a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f56f0 0x7fde5c10c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.171:46388/0 (socket says 172.21.15.171:46388) 2022-01-31T19:40:17.563 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde5bfff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 0x7fde5c10e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:49202/0 (socket says 172.21.15.171:49202) 2022-01-31T19:40:17.563 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde60c9a700 1 -- 172.21.15.171:0/207017728 learned_addr learned my addr 172.21.15.171:0/207017728 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:40:17.563 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde5bfff700 1 -- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f56f0 msgr2=0x7fde5c10c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:17.564 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde5bfff700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f56f0 0x7fde5c10c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:17.564 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.556+0000 7fde5bfff700 1 -- 172.21.15.171:0/207017728 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde50005040 con 0x7fde5c0f6990 2022-01-31T19:40:17.564 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.557+0000 7fde5bfff700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 0x7fde5c10e6f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fde48002700 tx=0x7fde48009cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:17.564 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.557+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde480079c0 con 0x7fde5c0f6990 2022-01-31T19:40:17.565 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.557+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c10ebe0 con 0x7fde5c0f6990 2022-01-31T19:40:17.565 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.557+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fde4800f040 con 0x7fde5c0f6990 2022-01-31T19:40:17.567 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.557+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde480146a0 con 0x7fde5c0f6990 2022-01-31T19:40:17.568 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.557+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c10f060 con 0x7fde5c0f6990 2022-01-31T19:40:17.568 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.558+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5c045bb0 con 0x7fde5c0f6990 2022-01-31T19:40:17.568 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.558+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fde4801b060 con 0x7fde5c0f6990 2022-01-31T19:40:17.568 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.558+0000 7fde597fa700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fde4c05e5c0 0x7fde4c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:17.569 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.558+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3056+0+0 (secure 0 0 0) 0x7fde48077e00 con 0x7fde5c0f6990 2022-01-31T19:40:17.569 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.559+0000 7fde60c9a700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fde4c05e5c0 0x7fde4c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:17.569 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.559+0000 7fde60c9a700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fde4c05e5c0 0x7fde4c060a80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fde50013620 tx=0x7fde50009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:17.569 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.562+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fde4804a250 con 0x7fde5c0f6990 2022-01-31T19:40:17.743 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:17.739+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fde5c057ff0 con 0x7fde4c05e5c0 2022-01-31T19:40:17.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[35325]: cluster 2022-01-31T19:40:16.624351+0000 2022-01-31T19:40:17.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[35325]: mon.smithi167 (mon.0) 353 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2022-01-31T19:40:17.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[35325]: audit 2022-01-31T19:40:16.624479+0000 mon.smithi167 (mon.0) 354 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:17.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[35325]: cluster 2022-01-31T19:40:16.707707+0000 mgr.smithi167.aciqpk (mgr.14156) 119 : cluster [DBG] pgmap v81: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:17.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.048689+0000 mon.smithi171 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-01-31T19:40:17.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:17 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.048917+0000 mon.smithi167 (mon.0) 355 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-01-31T19:40:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: cluster 2022-01-31T19:40:16.624351+0000 mon.smithi167 (mon. 2022-01-31T19:40:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: 0) 353 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2022-01-31T19:40:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: audit 2022-01-31T19:40:16.624479+0000 mon.smithi167 (mon.0) 354 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:17.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: cluster 2022-01-31T19:40:16.707707+0000 mgr.smithi167.aciqpk (mgr.14156) 119 : cluster [DBG] pgmap v81: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:17.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: audit 2022-01-31T19:40:17.048689+0000 mon.smithi171 (mon.1) 4 : 2022-01-31T19:40:17.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: audit [INF] from='osd.3 [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-01-31T19:40:17.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: audit 2022-01-31T19:40:17.048917+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:17.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:17 smithi167 conmon[32206]: 355 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-01-31T19:40:18.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[41853]: debug 2022-01-31T19:40:18.636+0000 7f9403c87700 -1 osd.3 0 waiting for initial osdmap 2022-01-31T19:40:18.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[41853]: debug 2022-01-31T19:40:18.643+0000 7f9400403700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-01-31T19:40:18.777 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.629061+0000 mon.smithi167 (mon.0) 356 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-01-31T19:40:18.777 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: cluster 2022-01-31T19:40:17.629119+0000 mon.smithi167 (mon.0) 357 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2022-01-31T19:40:18.777 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.629243+0000 mon.smithi167 (mon.0) 358 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:18.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.630554+0000 mon.smithi171 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:18.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.630803+0000 mon.smithi167 (mon.0) 359 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:18.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.741855+0000 mgr.smithi167.aciqpk (mgr.14156) 120 : audit [DBG] from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:40:18.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.742644+0000 mon.smithi167 (mon.0) 360 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:40:18.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.743848+0000 mon.smithi167 (mon.0) 361 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:40:18.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.744391+0000 mon.smithi167 (mon.0) 362 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:18.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:18 smithi171 conmon[35325]: audit 2022-01-31T19:40:17.821536+0000 mon.smithi167 (mon.0) 363 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:18.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022-01-31T19:40:17.629061 2022-01-31T19:40:18.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 356 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-01-31T19:40:18.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: cluster 2022-01-31T19:40:18.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 2022-01-31T19:40:17.629119+0000 mon.smithi167 (mon.0 2022-01-31T19:40:18.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: ) 357 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2022-01-31T19:40:18.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:18.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 01-31T19:40:17.629243+0000 2022-01-31T19:40:18.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: mon.smithi167 (mon.0) 358 : audit 2022-01-31T19:40:18.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:18.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:18.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 01-31T19:40:17.630554 2022-01-31T19:40:18.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: +0000 mon.smithi171 (mon.1) 5 2022-01-31T19:40:18.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: : audit [INF] from='osd.3 [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:18.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:18.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 01-31T19:40:17.630803+0000 2022-01-31T19:40:18.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: mon.smithi167 (mon.0) 359 : audit 2022-01-31T19:40:18.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:18.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:18.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 01-31T19:40:17.741855 2022-01-31T19:40:18.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 120 2022-01-31T19:40:18.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: : audit [DBG] from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:40:18.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:18.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 01-31T19:40:17.742644 2022-01-31T19:40:18.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 360 : 2022-01-31T19:40:18.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:40:18.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:18.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 01-31T19:40:17.743848 2022-01-31T19:40:18.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 361 : 2022-01-31T19:40:18.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:40:18.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022-01-31T19:40:18.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 2022-01-31T19:40:17 2022-01-31T19:40:18.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: .744391+0000 mon.smithi167 (mon.0) 362 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:18.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 2022-01-31T19:40:18.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: audit 2022-01-31T19:40:18.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: 2022-01-31T19:40:17.821536+0000 mon.smithi167 (mon.0) 363 2022-01-31T19:40:18.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:18 smithi167 conmon[32206]: : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:19.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T19:40:18.633863+0000 mon.smithi167 (mon 2022-01-31T19:40:19.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: .0) 364 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2022-01-31T19:40:19.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cluster 2022-01-31T19:40:18.633967+0000 mon.smithi167 (mon.0) 365 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2022-01-31T19:40:19.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T19:40:18.634361+0000 mon.smithi167 2022-01-31T19:40:19.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: (mon.0) 366 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:19.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022 2022-01-31T19:40:19.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: -01-31T19:40:18. 2022-01-31T19:40:19.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 638571+0000 mon.smithi167 (mon.0) 367 : audit [DBG] 2022-01-31T19:40:19.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:19.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:40:19.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 40:18.707998+0000 mgr.smithi167.aciqpk (mgr.14156) 121 : 2022-01-31T19:40:19.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:19.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cephadm 2022-01-31 2022-01-31T19:40:19.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: T19:40:19.370277+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:40:19.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: mgr.14156) 122 : cephadm [INF] Detected new or changed devices on smithi171 2022-01-31T19:40:19.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022 2022-01-31T19:40:19.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: -01-31T19:40:19. 2022-01-31T19:40:19.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 375315+0000 mon.smithi167 (mon.0) 368 : audit [INF] 2022-01-31T19:40:19.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:19.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:40:19.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: :40:19.376273+0000 mon.smithi167 (mon.0 2022-01-31T19:40:19.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: ) 369 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:19.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cephadm 2022-01 2022-01-31T19:40:19.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: -31T19:40:19.376929+0000 2022-01-31T19:40:19.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 123 : cephadm [INF] 2022-01-31T19:40:19.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: Adjusting osd_memory_target on smithi171 to 16017M 2022-01-31T19:40:19.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T19:40:19.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 2022-01-31T19:40:19 2022-01-31T19:40:19.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: .381085+0000 mon.smithi167 (mon.0) 370 : audit 2022-01-31T19:40:19.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:19.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:40:19.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 19:40:19.384983+0000 mon.smithi167 (mon. 2022-01-31T19:40:19.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 0) 371 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:19.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:40:19.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 19:40:19.388651+0000 mon.smithi167 (mon.0 2022-01-31T19:40:19.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: ) 372 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:19.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:40:19.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 31T19:40:19.389987+0000 2022-01-31T19:40:19.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: mon.smithi167 (mon.0) 373 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:19.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:19.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 01-31T19:40:19. 2022-01-31T19:40:19.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 390756+0000 mon.smithi167 (mon.0) 374 : audit 2022-01-31T19:40:19.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:19.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:40:19.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 40:19.633004+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:19.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 375 : cluster [INF] osd.3 [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] boot 2022-01-31T19:40:19.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: cluster 2022-01- 2022-01-31T19:40:19.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 31T19:40:19.633054+0000 2022-01-31T19:40:19.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: mon.smithi167 (mon.0) 376 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2022-01-31T19:40:19.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 2022-01-31T19:40:19.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: audit 2022-01-31T19:40:19.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: 2022-01-31T19:40:19.633168 2022-01-31T19:40:19.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 377 : audit [DBG] 2022-01-31T19:40:19.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:19 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:18.633863+0000 mon.smithi167 (mon 2022-01-31T19:40:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: .0) 364 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2022-01-31T19:40:20.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: cluster 2022-01-31T19:40:18.633967+0000 mon.smithi167 (mon.0) 365 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2022-01-31T19:40:20.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:18.634361+0000 mon.smithi167 (mon.0) 366 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:20.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:18.638571+0000 mon.smithi167 (mon.0) 367 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:20.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: cluster 2022-01-31T19:40:18.707998+0000 mgr.smithi167.aciqpk (mgr.14156) 121 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-01-31T19:40:20.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:19.370277+0000 mgr.smithi167.aciqpk (mgr.14156) 122 : cephadm [INF] Detected new or changed devices on smithi171 2022-01-31T19:40:20.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.375315+0000 mon.smithi167 (mon.0) 368 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:20.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.376273+0000 mon.smithi167 (mon.0) 369 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:20.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:19.376929+0000 mgr.smithi167.aciqpk (mgr.14156) 123 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 16017M 2022-01-31T19:40:20.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.381085+0000 mon.smithi167 (mon.0) 370 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:20.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.384983+0000 mon.smithi167 (mon.0) 371 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:20.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.388651+0000 mon.smithi167 (mon.0) 372 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:20.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.389987+0000 mon.smithi167 (mon.0) 373 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:20.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.390756+0000 mon.smithi167 (mon.0) 374 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:20.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: cluster 2022-01-31T19:40:19.633004+0000 mon.smithi167 (mon.0) 375 : cluster [INF] osd.3 [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] boot 2022-01-31T19:40:20.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: cluster 2022-01-31T19:40:19.633054+0000 mon.smithi167 (mon.0) 376 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2022-01-31T19:40:20.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: audit 2022-01-31T19:40: 2022-01-31T19:40:20.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:19 smithi171 conmon[35325]: 19.633168+0000 mon.smithi167 (mon.0) 377 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:40:20.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: cluster 2022-01-31T19:40:18.082225+0000 osd.3 (osd.3) 1 2022-01-31T19:40:20.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:20.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: cluster 2022-01-31T19:40:18.082333+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:20.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: audit 2022-01-31T19:40:20.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: 2022-01-31T19:40:19.908919+0000 mon.smithi171 (mon.1) 6 : audit [INF] from='client.? 172.21.15.171:0/4015861064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4310782-5c0e-4962-b32e-45e28ac014ef"}]: dispatch 2022-01-31T19:40:20.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: audit 2022-01-31T19:40:20.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: 2022-01- 2022-01-31T19:40:20.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: 31T19:40:19.909285+0000 mon.smithi167 (mon 2022-01-31T19:40:20.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: .0) 378 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4310782-5c0e-4962-b32e-45e28ac014ef"}]: dispatch 2022-01-31T19:40:20.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:40:20.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: :40:19.914094+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:20.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: 379 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4310782-5c0e-4962-b32e-45e28ac014ef"}]': finished 2022-01-31T19:40:20.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: cluster 2022-01-31T19 2022-01-31T19:40:20.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: :40:19.914166+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:20.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: 380 : cluster [DBG] osdmap e26: 5 total, 4 up, 5 in 2022-01-31T19:40:20.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:40:20.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: 19:40:19.914298+0000 mon.smithi167 (mon.0 2022-01-31T19:40:20.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: ) 381 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:20.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:40:20.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: T19:40:20.337293+0000 mon.smithi171 (mon 2022-01-31T19:40:20.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:20 smithi167 conmon[32206]: .1) 7 : audit [DBG] from='client.? 172.21.15.171:0/1328050665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:40:21.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: cluster 2022-01-31T19:40:18.082225+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:21.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: cluster 2022-01-31T19:40:18.082333+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:21.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.908919+0000 mon.smithi171 (mon.1) 6 : audit [INF] from='client.? 172.21.15.171:0/4015861064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4310782-5c0e-4962-b32e-45e28ac014ef"}]: dispatch 2022-01-31T19:40:21.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.909285+0000 mon.smithi167 (mon.0) 378 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4310782-5c0e-4962-b32e-45e28ac014ef"}]: dispatch 2022-01-31T19:40:21.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.914094+0000 mon.smithi167 (mon.0) 379 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4310782-5c0e-4962-b32e-45e28ac014ef"}]': finished 2022-01-31T19:40:21.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: cluster 2022-01-31T19:40:19.914166+0000 mon.smithi167 (mon.0) 380 : cluster [DBG] osdmap e26: 5 total, 4 up, 5 in 2022-01-31T19:40:21.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: audit 2022-01-31T19:40:19.914298+0000 mon.smithi167 (mon.0) 381 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:21.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:20 smithi171 conmon[35325]: audit 2022-01-31T19:40:20.337293+0000 mon.smithi171 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.171:0/1328050665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:40:22.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:21 smithi167 conmon[32206]: cluster 2022-01-31T19:40:20.708490+0000 mgr.smithi167.aciqpk (mgr.14156) 124 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:22.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:21 smithi167 conmon[32206]: cluster 2022-01-31T19:40:20.914758+0000 mon.smithi167 (mon.0) 382 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2022-01-31T19:40:22.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:21 smithi167 conmon[32206]: audit 2022-01-31T19:40:22.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:21 smithi167 conmon[32206]: 2022-01-31T19:40:20.914867+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:22.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:21 smithi167 conmon[32206]: 383 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:22.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: cluster 2022-01-31T19:40:20.708490+0000 mgr.smithi167.aciqpk 2022-01-31T19:40:22.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: (mgr.14156) 124 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:22.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: cluster 2022-01-31T19:40:20.914758+0000 mon.smithi167 ( 2022-01-31T19:40:22.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: mon.0) 382 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2022-01-31T19:40:22.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: audit 2022-01-31T19:40:20.914867 2022-01-31T19:40:22.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 383 : audit [DBG] 2022-01-31T19:40:22.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:21 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:24.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:23 smithi167 conmon[32206]: cluster 2022-01-31T19:40:22.708792+0000 mgr.smithi167.aciqpk (mgr.14156) 125 : cluster [DBG] pgmap v89: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:24.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:23 smithi171 conmon[35325]: cluster 2022-01-31T19:40:22.708792+0000 mgr.smithi167.aciqpk 2022-01-31T19:40:24.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:23 smithi171 conmon[35325]: (mgr.14156) 125 : cluster [DBG] pgmap v89: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:25.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:25 smithi167 conmon[32206]: audit 2022-01-31T19:40:24.382822+0000 mon.smithi167 (mon.0) 384 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:25.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:25 smithi167 conmon[32206]: 2022-01-31T19:40:25.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:25 smithi167 conmon[32206]: audit 2022-01-31T19:40:24.388145+0000 mon.smithi167 (mon.0) 385 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:25.750 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:25 smithi171 conmon[35325]: audit 2022-01-31T19:40:24.382822+0000 mon.smithi167 (mon.0) 384 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:25.750 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:25 smithi171 conmon[35325]: audit 2022-01-31T19:40:24.388145+0000 mon.smithi167 (mon.0) 385 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:26.665 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:26 smithi171 conmon[35325]: cluster 2022-01-31T19:40:24.709098+0000 mgr.smithi167.aciqpk (mgr.14156) 126 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:26.665 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:26 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:40:26.665 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:26 smithi171 conmon[35325]: 31T19:40:25.939263+0000 mon.smithi167 (mon.0) 386 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-01-31T19:40:26.666 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:26 smithi171 conmon[35325]: audit 2022-01-31T19:40:25.940094+0000 mon.smithi167 (mon.0) 387 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:26.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:26 smithi167 conmon[32206]: cluster 2022-01-31T19:40:24.709098+0000 mgr.smithi167.aciqpk (mgr.14156) 126 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:26.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:26 smithi167 conmon[32206]: audit 2022-01-31T19:40:25.939263+0000 mon.smithi167 (mon.0) 386 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-01-31T19:40:26.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:26 smithi167 conmon[32206]: audit 2022-01-31T19:40:25.940094+0000 mon.smithi167 (mon.0) 387 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:27.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:27 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:25.940778+0000 mgr.smithi167.aciqpk (mgr.14156) 127 : cephadm [INF] Deploying daemon osd.4 on smithi171 2022-01-31T19:40:27.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:27 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:25.940778+0000 mgr.smithi167.aciqpk (mgr.14156) 127 : cephadm [INF] Deploying daemon osd.4 on smithi171 2022-01-31T19:40:28.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:28 smithi167 conmon[32206]: cluster 2022-01-31T19:40:26.709443+0000 mgr.smithi167.aciqpk (mgr.14156) 128 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:28.670 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:28 smithi171 conmon[35325]: cluster 2022-01-31T19:40:26.709443+0000 mgr.smithi167.aciqpk (mgr.14156) 128 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:28.857 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 4 on host 'smithi171' 2022-01-31T19:40:28.858 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.854+0000 7fde597fa700 1 -- 172.21.15.171:0/207017728 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde5c057ff0 con 0x7fde4c05e5c0 2022-01-31T19:40:28.858 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fde4c05e5c0 msgr2=0x7fde4c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:28.859 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fde4c05e5c0 0x7fde4c060a80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fde50013620 tx=0x7fde50009b30).stop 2022-01-31T19:40:28.859 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 msgr2=0x7fde5c10e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:28.859 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 0x7fde5c10e6f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fde48002700 tx=0x7fde48009cc0).stop 2022-01-31T19:40:28.859 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 shutdown_connections 2022-01-31T19:40:28.860 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fde4c05e5c0 0x7fde4c060a80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:28.860 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde5c0f56f0 0x7fde5c10c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:28.860 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 --2- 172.21.15.171:0/207017728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde5c0f6990 0x7fde5c10e6f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:28.860 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 >> 172.21.15.171:0/207017728 conn(0x7fde5c0f0e50 msgr2=0x7fde5c0fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:28.860 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 shutdown_connections 2022-01-31T19:40:28.861 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:28.856+0000 7fde62efe700 1 -- 172.21.15.171:0/207017728 wait complete. 2022-01-31T19:40:29.242 DEBUG:teuthology.orchestra.run.smithi171:osd.4> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@osd.4.service 2022-01-31T19:40:29.245 INFO:tasks.cephadm:Deploying osd.5 on smithi171 with /dev/vg_nvme/lv_2... 2022-01-31T19:40:29.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-01-31T19:40:29.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:-- Logs begin at Mon 2022-01-31 19:28:36 UTC. -- 2022-01-31T19:40:29.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[46715]: debug 2022-01-31T19:40:29.290+0000 7fa6a7415080 -1 Falling back to public interface 2022-01-31T19:40:29.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: audit 2022-01-31T19:40:28.840340+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:29.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: 388 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:29.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: audit 2022-01-31T19:40:28.842258+0000 mon.smithi167 (mon.0) 389 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:29.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: audit 2022-01-31T19:40:28.842981+0000 mon.smithi167 (mon.0) 390 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:29.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: audit 2022-01-31T19:40:28.846895+0000 mon.smithi167 (mon.0) 391 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:29.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: audit 2022-01-31T19:40:28.853805+0000 mon.smithi167 (mon.0) 392 : audit [INF] 2022-01-31T19:40:29.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:29 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:29.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: audit 2022-01-31T19:40:28.840340+0000 mon.smithi167 (mon.0 2022-01-31T19:40:29.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: ) 388 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:29.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: audit 2022-01-31T19:40:28.842258+0000 mon.smithi167 (mon.0) 389 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:29.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: audit 2022 2022-01-31T19:40:29.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: -01-31T19:40:28.842981+0000 mon.smithi167 (mon.0) 390 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:29.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:40:29.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: T19:40:28.846895+0000 mon.smithi167 (mon.0) 391 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:29.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: audit 2022-01-31T19:40:28.853805 2022-01-31T19:40:29.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:29 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 392 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:30.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:30 smithi167 conmon[32206]: cluster 2022-01-31T19:40:28.709916+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:40:30.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:30 smithi167 conmon[32206]: .14156) 129 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:30.678 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:30 smithi171 conmon[35325]: cluster 2022-01-31T19:40:28.709916+0000 mgr.smithi167.aciqpk (mgr.14156) 129 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:31.234 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-01-31T19:40:31.234 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-01-31T19:40:31.234 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2022-01-31T19:40:31.235 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2022-01-31T19:40:31.235 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123964 s, 846 MB/s 2022-01-31T19:40:31.235 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2022-01-31T19:40:31.901 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:31 smithi171 conmon[35325]: cluster 2022-01-31T19:40:30.710237+0000 mgr.smithi167.aciqpk (mgr.14156) 130 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:31.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi171:vg_nvme/lv_2 2022-01-31T19:40:31.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:31 smithi167 conmon[32206]: cluster 2022-01-31T19:40:30.710237+0000 mgr.smithi167.aciqpk (mgr.14156) 130 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:32.952 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 19:40:32 smithi171 conmon[46715]: debug 2022-01-31T19:40:32.834+0000 7fa6a7415080 -1 osd.4 0 log_to_monitors {default=true} 2022-01-31T19:40:33.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:33 smithi171 conmon[35325]: audit 2022-01-31T19:40:32.295001+0000 mon.smithi167 (mon.0) 393 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:33.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:33 smithi171 conmon[35325]: audit 2022-01-31T19:40:32.838122+0000 mon.smithi171 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-01-31T19:40:33.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:33 smithi171 conmon[35325]: audit 2022-01-31T19:40:32.838382+0000 mon.smithi167 (mon.0) 394 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-01-31T19:40:33.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:33 smithi167 conmon[32206]: audit 2022-01-31T19:40:32.295001+0000 mon.smithi167 (mon.0) 393 : 2022-01-31T19:40:33.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:33 smithi167 conmon[32206]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:33.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:33 smithi167 conmon[32206]: audit 2022-01-31T19:40:32.838122+0000 mon.smithi171 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-01-31T19:40:33.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:33 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:40:33.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:33 smithi167 conmon[32206]: 19:40:32.838382+0000 mon.smithi167 (mon.0) 394 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-01-31T19:40:33.701 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 -- 172.21.15.171:0/684022374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 msgr2=0x7fdca40f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:33.701 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 --2- 172.21.15.171:0/684022374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca40f7d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fdc90004660 tx=0x7fdc90009b30).stop 2022-01-31T19:40:33.701 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 -- 172.21.15.171:0/684022374 shutdown_connections 2022-01-31T19:40:33.701 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 --2- 172.21.15.171:0/684022374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca40f7d50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:33.702 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 --2- 172.21.15.171:0/684022374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca40f6650 0x7fdca40f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:33.702 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 -- 172.21.15.171:0/684022374 >> 172.21.15.171:0/684022374 conn(0x7fdca40f1db0 msgr2=0x7fdca40f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:33.702 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 -- 172.21.15.171:0/684022374 shutdown_connections 2022-01-31T19:40:33.702 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 -- 172.21.15.171:0/684022374 wait complete. 2022-01-31T19:40:33.703 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 Processor -- start 2022-01-31T19:40:33.703 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.699+0000 7fdca9f28700 1 -- start start 2022-01-31T19:40:33.703 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca9f28700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca40f6650 0x7fdca4106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:33.703 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca9f28700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca4107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:33.704 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca9f28700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4107940 con 0x7fdca40f6650 2022-01-31T19:40:33.704 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca9f28700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4107a80 con 0x7fdca40f78f0 2022-01-31T19:40:33.704 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca4107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:33.704 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca4107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:49264/0 (socket says 172.21.15.171:49264) 2022-01-31T19:40:33.705 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 -- 172.21.15.171:0/1061788125 learned_addr learned my addr 172.21.15.171:0/1061788125 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:40:33.705 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca8f26700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca40f6650 0x7fdca4106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:33.707 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 -- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca40f6650 msgr2=0x7fdca4106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:33.707 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca40f6650 0x7fdca4106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:33.707 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 -- 172.21.15.171:0/1061788125 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc90005040 con 0x7fdca40f78f0 2022-01-31T19:40:33.707 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca3fff700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca4107390 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fdc90009990 tx=0x7fdc90009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:33.708 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc90018070 con 0x7fdca40f78f0 2022-01-31T19:40:33.708 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdc90013410 con 0x7fdca40f78f0 2022-01-31T19:40:33.708 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca41083c0 con 0x7fdca40f78f0 2022-01-31T19:40:33.708 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4108920 con 0x7fdca40f78f0 2022-01-31T19:40:33.709 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.700+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc9001de80 con 0x7fdca40f78f0 2022-01-31T19:40:33.709 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.701+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca4045bb0 con 0x7fdca40f78f0 2022-01-31T19:40:33.709 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.702+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fdc9001d420 con 0x7fdca40f78f0 2022-01-31T19:40:33.709 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.702+0000 7fdca17fa700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdc9405e5c0 0x7fdc94060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:33.710 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.702+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3619+0+0 (secure 0 0 0) 0x7fdc9007a3b0 con 0x7fdca40f78f0 2022-01-31T19:40:33.710 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.703+0000 7fdca8f26700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdc9405e5c0 0x7fdc94060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:33.710 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.703+0000 7fdca8f26700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdc9405e5c0 0x7fdc94060a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdca40f7750 tx=0x7fdc9800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:33.711 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.706+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdc9004c5f0 con 0x7fdca40f78f0 2022-01-31T19:40:33.884 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:33.883+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdca4108b90 con 0x7fdc9405e5c0 2022-01-31T19:40:34.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[46715]: debug 2022-01-31T19:40:34.310+0000 7fa69d978700 -1 osd.4 0 waiting for initial osdmap 2022-01-31T19:40:34.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[46715]: debug 2022-01-31T19:40:34.318+0000 7fa6998f3700 -1 osd.4 29 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-01-31T19:40:34.583 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: cluster 2022-01-31T19:40:32.710557+0000 mgr.smithi167.aciqpk (mgr.14156) 131 : 2022-01-31T19:40:34.583 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:34.583 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.303341+0000 mon.smithi167 (mon.0) 395 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-01-31T19:40:34.584 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: cluster 2022-01-31T19:40:33.303414+0000 mon.smithi167 (mon.0) 396 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2022-01-31T19:40:34.584 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.304579+0000 mon.smithi171 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:34.584 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.304766+0000 mon.smithi167 (mon.0) 397 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:34.585 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.311651+0000 mon.smithi167 (mon.0) 398 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:34.585 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.334766+0000 mon.smithi167 (mon.0) 399 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:34.585 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.885322+0000 mon.smithi167 (mon.0) 400 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:40:34.586 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.886684+0000 mon.smithi167 (mon.0) 401 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:40:34.586 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:34 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.887246+0000 mon.smithi167 (mon.0) 402 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:34.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: cluster 2022-01-31T19:40:32.710557+0000 mgr.smithi167.aciqpk (mgr.14156) 131 2022-01-31T19:40:34.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:34.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022-01-31T19:40:33.303341+0000 mon.smithi167 (mon.0) 395 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-01-31T19:40:34.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: cluster 2022-01- 2022-01-31T19:40:34.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: 31T19:40:33.303414+0000 mon.smithi167 2022-01-31T19:40:34.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: (mon.0) 396 : cluster [DBG] 2022-01-31T19:40:34.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: osdmap e28: 5 total, 4 up, 5 in 2022-01-31T19:40:34.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022 2022-01-31T19:40:34.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: -01-31T19:40:33.304579 2022-01-31T19:40:34.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: +0000 mon.smithi171 (mon.1) 9 : audit 2022-01-31T19:40:34.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: [INF] from='osd.4 [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:34.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:40:34.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: 01-31T19:40:33.304766+0000 mon.smithi167 (mon.0) 397 : 2022-01-31T19:40:34.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:34.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022-01-31T19:40:34.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: 2022-01-31T19:40:33.311651+0000 mon.smithi167 (mon 2022-01-31T19:40:34.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: .0) 398 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:34.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022-01-31T19:40:33. 2022-01-31T19:40:34.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: 334766+0000 mon.smithi167 (mon.0) 399 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:34.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:40:34.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: 19:40:33.885322+0000 mon.smithi167 (mon.0) 400 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-01-31T19:40:34.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022-01-31T19:40:33.886684+0000 mon.smithi167 2022-01-31T19:40:34.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: (mon.0) 401 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-01-31T19:40:34.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: audit 2022-01-31T19:40:34.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:34 smithi167 conmon[32206]: 2022-01-31T19:40:33.887246+0000 mon.smithi167 (mon.0) 402 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:35.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:35 smithi171 conmon[35325]: audit 2022-01-31T19:40:33.884599+0000 mgr.smithi167.aciqpk (mgr.14156) 132 : audit [DBG] from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:40:35.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:35 smithi171 conmon[35325]: audit 2022-01-31T19:40:34.306005+0000 mon.smithi167 (mon.0) 403 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2022-01-31T19:40:35.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:35 smithi171 conmon[35325]: cluster 2022-01-31T19:40:34.306128+0000 mon.smithi167 (mon.0) 404 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2022-01-31T19:40:35.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:35 smithi171 conmon[35325]: audit 2022-01-31T19:40:34.306583+0000 mon.smithi167 (mon.0) 405 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:35.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:35 smithi171 conmon[35325]: audit 2022-01-31T19:40:34.310828+0000 mon.smithi167 (mon.0) 406 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:35.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: audit 2022-01-31T19:40:33.884599+0000 mgr.smithi167.aciqpk (mgr.14156) 132 2022-01-31T19:40:35.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: : audit [DBG] from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:40:35.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: audit 2022-01-31T19:40:34.306005+0000 mon.smithi167 (mon.0) 403 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2022-01-31T19:40:35.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: cluster 2022-01-31T19:40:34.306128+0000 2022-01-31T19:40:35.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: mon.smithi167 (mon.0) 404 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2022-01-31T19:40:35.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: audit 2022-01-31T19:40:34 2022-01-31T19:40:35.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: .306583+0000 mon.smithi167 (mon.0) 405 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:35.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: audit 2022-01-31T19:40:35.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: 2022-01-31T19:40:34.310828+0000 mon.smithi167 (mon.0) 406 : 2022-01-31T19:40:35.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:35 smithi167 conmon[32206]: audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:36.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: cluster 2022-01-31T19:40:33.822138+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:36.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: cluster 2022-01-31T19:40:33.822234+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:36.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: cluster 2022-01-31T19:40:34.710911+0000 mgr.smithi167.aciqpk (mgr.14156) 133 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:36.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.311414+0000 mon.smithi167 (mon.0) 407 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:36.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: cluster 2022-01-31T19:40:35.319266+0000 mon.smithi167 (mon.0) 408 : cluster [INF] osd.4 [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] boot 2022-01-31T19:40:36.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: cluster 2022-01-31T19:40:35.319316+0000 mon.smithi167 (mon.0) 409 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2022-01-31T19:40:36.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.324423+0000 mon.smithi167 (mon.0 2022-01-31T19:40:36.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: ) 410 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:36.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.525320+0000 mon.smithi167 (mon.0) 411 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:36.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.526320+0000 mon.smithi167 (mon.0) 412 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:36.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.527184+0000 mon.smithi167 (mon.0) 413 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:36.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.531529+0000 mon.smithi167 (mon.0) 414 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:36.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.535267+0000 mon.smithi167 (mon.0) 415 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:36.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.539078+0000 mon.smithi167 (mon.0) 416 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:36.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:35.540298+0000 mon.smithi167 (mon.0) 417 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:36.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:40:36.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: :40:35.541019+0000 mon.smithi167 (mon.0) 418 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:36.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:40:36.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: 40:36.087982+0000 mon.smithi171 (mon.1) 10 : audit 2022-01-31T19:40:36.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: [INF] from='client.? 172.21.15.171:0/863519148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ef7b4c2-8076-4866-b26a-8027fd249d2d"}]: dispatch 2022-01-31T19:40:36.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19:40:36.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: 2022-01-31T19:40:36.088342+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:36.614 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: 419 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ef7b4c2-8076-4866-b26a-8027fd249d2d"}]: dispatch 2022-01-31T19:40:36.614 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:40:36.614 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: :40:36.093077+0000 mon.smithi167 (mon.0) 420 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ef7b4c2-8076-4866-b26a-8027fd249d2d"}]': finished 2022-01-31T19:40:36.614 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: cluster 2022-01-31T19:40:36.093135 2022-01-31T19:40:36.614 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 421 : cluster [DBG] osdmap e31: 6 total, 5 up, 6 in 2022-01-31T19:40:36.615 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit 2022-01-31T 2022-01-31T19:40:36.615 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: 19:40:36.093241+0000 mon.smithi167 (mon.0) 422 : 2022-01-31T19:40:36.615 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:36 smithi171 conmon[35325]: audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:36.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: cluster 2022-01-31T19:40:33.822138+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:36.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 2022-01-31T19:40:36.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: cluster 2022-01-31T19:40:33.822234+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:36.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:40:36.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 40:34.710911+0000 mgr.smithi167.aciqpk (mgr.14156) 133 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2022-01-31T19:40:36.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:35 2022-01-31T19:40:36.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: .311414+0000 mon.smithi167 (mon.0) 407 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:36.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: cluster 2022-01-31T 2022-01-31T19:40:36.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 19:40:35.319266+0000 mon.smithi167 (mon.0) 408 : cluster [INF] osd.4 [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] boot 2022-01-31T19:40:36.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: cluster 2022-01-31T19:40:35.319316+0000 mon.smithi167 (mon 2022-01-31T19:40:36.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: .0) 409 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2022-01-31T19:40:36.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:36.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 2022-01-31T19:40:35.324423+0000 mon.smithi167 (mon.0) 410 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:40:36.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:35.525320+0000 mon.smithi167 (mon 2022-01-31T19:40:36.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: .0) 411 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:36.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022 2022-01-31T19:40:36.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: -01-31T19:40:35.526320+0000 mon.smithi167 (mon.0) 412 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:36.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:35.527184+0000 mon.smithi167 (mon 2022-01-31T19:40:36.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: .0) 413 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:36.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:36.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 2022-01-31T19:40:35.531529+0000 mon.smithi167 (mon.0) 414 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:36.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:35.535267+0000 mon.smithi167 2022-01-31T19:40:36.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: (mon.0) 415 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:36.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:35.539078+0000 mon.smithi167 2022-01-31T19:40:36.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: (mon.0) 416 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:36.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:35.540298+0000 mon.smithi167 (mon. 2022-01-31T19:40:36.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 0) 417 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:36.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:36.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: 2022-01-31T19:40:35.541019+0000 mon.smithi167 (mon.0) 418 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:36.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:40:36.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: T19:40:36.087982+0000 mon.smithi171 (mon.1) 10 : 2022-01-31T19:40:36.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit [INF] from='client.? 172.21.15.171:0/863519148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ef7b4c2-8076-4866-b26a-8027fd249d2d"}]: dispatch 2022-01-31T19:40:36.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022 2022-01-31T19:40:36.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: -01-31T19:40:36.088342+0000 mon.smithi167 (mon.0) 419 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ef7b4c2-8076-4866-b26a-8027fd249d2d"}]: dispatch 2022-01-31T19:40:36.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:36 2022-01-31T19:40:36.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: .093077+0000 mon.smithi167 (mon.0) 420 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ef7b4c2-8076-4866-b26a-8027fd249d2d"}]': finished 2022-01-31T19:40:36.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: cluster 2022-01-31T19:40:36.093135 2022-01-31T19:40:36.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 421 : cluster [DBG] osdmap e31: 6 total, 5 up, 6 in 2022-01-31T19:40:36.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: audit 2022-01-31T19:40:36.093241+0000 mon.smithi167 ( 2022-01-31T19:40:36.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:36 smithi167 conmon[32206]: mon.0) 422 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:37.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:37 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:35.520376+0000 mgr.smithi167.aciqpk (mgr.14156) 134 : cephadm [INF] Detected new or changed devices on smithi171 2022-01-31T19:40:37.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:37 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:35.527761+0000 mgr.smithi167.aciqpk (mgr.14156) 135 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 8008M 2022-01-31T19:40:37.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:37 smithi171 conmon[35325]: audit 2022-01-31T19:40:36.517080+0000 mon.smithi171 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.171:0/2485741411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:40:37.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:37 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:35.520376+0000 mgr.smithi167.aciqpk (mgr.14156) 134 : cephadm 2022-01-31T19:40:37.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:37 smithi167 conmon[32206]: [INF] Detected new or changed devices on smithi171 2022-01-31T19:40:37.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:37 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:35.527761+0000 mgr.smithi167.aciqpk (mgr.14156) 135 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 8008M 2022-01-31T19:40:37.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:37 smithi167 conmon[32206]: audit 2022-01-31T19:40:36.517080+0000 mon.smithi171 (mon.1) 11 : audit [DBG] 2022-01-31T19:40:37.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:37 smithi167 conmon[32206]: from='client.? 172.21.15.171:0/2485741411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-01-31T19:40:38.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:38 smithi171 conmon[35325]: cluster 2022-01-31T19:40:36.711206+0000 mgr.smithi167.aciqpk (mgr.14156) 136 : cluster [DBG] pgmap v100: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:38.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:38 smithi167 conmon[32206]: cluster 2022-01-31T19:40:36.711206+0000 mgr.smithi167.aciqpk (mgr.14156) 136 : cluster [DBG] pgmap v100: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:40.568 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:40 smithi171 conmon[35325]: cluster 2022-01-31T19:40:38.711532+0000 mgr.smithi167.aciqpk (mgr.14156) 137 2022-01-31T19:40:40.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:40 smithi171 conmon[35325]: : cluster [DBG] pgmap v101: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:40.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:40 smithi171 conmon[35325]: audit 2022-01-31T19:40:40.025245+0000 mon.smithi167 (mon.0) 423 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:40.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:40 smithi171 conmon[35325]: audit 2022-01-31T19:40:40.028435+0000 mon.smithi167 (mon.0) 424 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:40.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:40 smithi167 conmon[32206]: cluster 2022-01-31T19:40:38.711532+0000 mgr.smithi167.aciqpk (mgr.14156) 137 : 2022-01-31T19:40:40.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:40 smithi167 conmon[32206]: cluster [DBG] pgmap v101: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:40.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:40 smithi167 conmon[32206]: audit 2022-01-31T19:40:40.025245+0000 mon.smithi167 (mon.0) 423 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:40.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:40 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:40:40.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:40 smithi167 conmon[32206]: 40:40.028435+0000 mon.smithi167 (mon.0) 424 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:41.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:41 smithi171 conmon[35325]: cluster 2022-01-31T19:40:40.711831+0000 mgr.smithi167.aciqpk (mgr.14156) 138 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:41.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:41 smithi167 conmon[32206]: cluster 2022-01-31T19:40:40.711831+0000 mgr.smithi167.aciqpk (mgr.14156) 138 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:42.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:42 smithi167 conmon[32206]: audit 2022-01-31T19:40:42.251382+0000 2022-01-31T19:40:42.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:42 smithi167 conmon[32206]: mon.smithi167 (mon.0) 425 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-01-31T19:40:42.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:42 smithi167 conmon[32206]: audit 2022-01-31T19:40:42.252211+0000 mon.smithi167 (mon.0) 426 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:42.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:42.252852+0000 mgr.smithi167.aciqpk (mgr.14156) 139 : cephadm 2022-01-31T19:40:42.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:42 smithi167 conmon[32206]: [INF] Deploying daemon osd.5 on smithi171 2022-01-31T19:40:42.957 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:42 smithi171 conmon[35325]: audit 2022-01-31T19:40:42.251382+0000 mon.smithi167 (mon.0 2022-01-31T19:40:42.958 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:42 smithi171 conmon[35325]: ) 425 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-01-31T19:40:42.958 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:42 smithi171 conmon[35325]: audit 2022-01-31T19:40:42.252211+0000 mon.smithi167 (mon.0) 426 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:42.959 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:42 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:42.252852+0000 mgr.smithi167.aciqpk (mgr.14156) 139 : cephadm [INF] Deploying daemon osd.5 on smithi171 2022-01-31T19:40:43.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:43 smithi171 conmon[35325]: cluster 2022-01-31T19:40:42.712288+0000 mgr.smithi167.aciqpk (mgr.14156) 140 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:43.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:43 smithi167 conmon[32206]: cluster 2022-01-31T19:40:42.712288+0000 mgr.smithi167.aciqpk (mgr.14156) 140 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:43.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:43 smithi167 conmon[32206]: 2022-01-31T19:40:45.169 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 5 on host 'smithi171' 2022-01-31T19:40:45.170 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.166+0000 7fdca17fa700 1 -- 172.21.15.171:0/1061788125 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdca4108b90 con 0x7fdc9405e5c0 2022-01-31T19:40:45.170 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.168+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdc9405e5c0 msgr2=0x7fdc94060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:45.171 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.168+0000 7fdca9f28700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdc9405e5c0 0x7fdc94060a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdca40f7750 tx=0x7fdc9800b040).stop 2022-01-31T19:40:45.171 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.168+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 msgr2=0x7fdca4107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:45.171 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.168+0000 7fdca9f28700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca4107390 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fdc90009990 tx=0x7fdc90009b30).stop 2022-01-31T19:40:45.172 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 shutdown_connections 2022-01-31T19:40:45.172 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdc9405e5c0 0x7fdc94060a80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:45.172 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdca40f6650 0x7fdca4106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:45.172 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 --2- 172.21.15.171:0/1061788125 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdca40f78f0 0x7fdca4107390 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:45.173 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 >> 172.21.15.171:0/1061788125 conn(0x7fdca40f1db0 msgr2=0x7fdca40fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:45.173 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 shutdown_connections 2022-01-31T19:40:45.173 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:45.169+0000 7fdca9f28700 1 -- 172.21.15.171:0/1061788125 wait complete. 2022-01-31T19:40:45.562 DEBUG:teuthology.orchestra.run.smithi171:osd.5> sudo journalctl -f -n 0 -u ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@osd.5.service 2022-01-31T19:40:45.565 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2022-01-31T19:40:45.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd stat -f json 2022-01-31T19:40:45.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:-- Logs begin at Mon 2022-01-31 19:28:36 UTC. -- 2022-01-31T19:40:45.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[51620]: debug 2022-01-31T19:40:45.628+0000 7f84fa0c4080 -1 Falling back to public interface 2022-01-31T19:40:45.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: cluster 2022-01-31T19:40:44.712819+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:40:45.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: .14156) 141 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:45.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: audit 2022-01-31T19:40:45.157482+0000 mon.smithi167 (mon.0) 427 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:45.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: audit 2022-01-31T19:40:45.159472+0000 mon.smithi167 (mon.0) 428 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:45.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: audit 2022-01-31T19:40:45.161739+0000 mon.smithi167 (mon.0 2022-01-31T19:40:45.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: ) 429 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:45.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:40:45.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: -31T19:40:45.163706+0000 mon.smithi167 (mon 2022-01-31T19:40:45.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: .0) 430 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:45.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:40:45.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: 01-31T19:40:45.166271+0000 mon.smithi167 ( 2022-01-31T19:40:45.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:45 smithi171 conmon[35325]: mon.0) 431 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:45.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: cluster 2022-01-31T19:40:44.712819+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:40:45.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: 141 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:45.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: audit 2022-01-31T19:40:45.157482+0000 mon.smithi167 (mon.0) 427 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:45.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: audit 2022-01-31T19:40:45.159472+0000 mon.smithi167 (mon.0) 428 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:45.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: audit 2022-01-31T19:40:45.161739+0000 mon.smithi167 (mon.0) 429 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:45.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: audit 2022-01-31T19:40:45.163706+0000 mon.smithi167 (mon.0) 430 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:45.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:45 smithi167 conmon[32206]: audit 2022-01-31T19:40:45.166271+0000 mon.smithi167 (mon.0) 431 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:47.176 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 -- 172.21.15.167:0/3619475535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 msgr2=0x7f3e740fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:47.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 --2- 172.21.15.167:0/3619475535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e740fbd90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3e64004660 tx=0x7f3e64009b30).stop 2022-01-31T19:40:47.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 -- 172.21.15.167:0/3619475535 shutdown_connections 2022-01-31T19:40:47.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 --2- 172.21.15.167:0/3619475535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e740fbd90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:47.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 --2- 172.21.15.167:0/3619475535 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e74060390 0x7f3e740f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:47.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 -- 172.21.15.167:0/3619475535 >> 172.21.15.167:0/3619475535 conn(0x7f3e740f0e60 msgr2=0x7f3e740f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:47.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 -- 172.21.15.167:0/3619475535 shutdown_connections 2022-01-31T19:40:47.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.173+0000 7f3e789b6700 1 -- 172.21.15.167:0/3619475535 wait complete. 2022-01-31T19:40:47.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e789b6700 1 Processor -- start 2022-01-31T19:40:47.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e789b6700 1 -- start start 2022-01-31T19:40:47.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e789b6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e74060390 0x7f3e74107070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:47.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e789b6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e74107560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:47.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e789b6700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74107b10 con 0x7f3e740f99a0 2022-01-31T19:40:47.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e789b6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74107c50 con 0x7f3e74060390 2022-01-31T19:40:47.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e74107560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:47.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e7259c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e74060390 0x7f3e74107070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:47.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e74107560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32782/0 (socket says 172.21.15.167:32782) 2022-01-31T19:40:47.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 -- 172.21.15.167:0/1526925546 learned_addr learned my addr 172.21.15.167:0/1526925546 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:40:47.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 -- 172.21.15.167:0/1526925546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e74060390 msgr2=0x7f3e74107070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:47.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e74060390 0x7f3e74107070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:47.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 -- 172.21.15.167:0/1526925546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64005040 con 0x7f3e740f99a0 2022-01-31T19:40:47.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.174+0000 7f3e71d9b700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e74107560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3e64004e30 tx=0x7f3e64009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:47.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.175+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e64018070 con 0x7f3e740f99a0 2022-01-31T19:40:47.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.175+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3e64013410 con 0x7f3e740f99a0 2022-01-31T19:40:47.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.175+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e74108590 con 0x7f3e740f99a0 2022-01-31T19:40:47.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.175+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6401c3c0 con 0x7f3e740f99a0 2022-01-31T19:40:47.185 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.175+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e74108af0 con 0x7f3e740f99a0 2022-01-31T19:40:47.186 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.176+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e74045bb0 con 0x7f3e740f99a0 2022-01-31T19:40:47.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.176+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f3e6401d420 con 0x7f3e740f99a0 2022-01-31T19:40:47.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.176+0000 7f3e62ffd700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f3e5c05e5c0 0x7f3e5c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:47.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.176+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4030+0+0 (secure 0 0 0) 0x7f3e6407a050 con 0x7f3e740f99a0 2022-01-31T19:40:47.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.176+0000 7f3e7259c700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f3e5c05e5c0 0x7f3e5c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:47.188 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.177+0000 7f3e7259c700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f3e5c05e5c0 0x7f3e5c060a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3e68004c40 tx=0x7f3e6800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:47.188 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.181+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3e6404c0d0 con 0x7f3e740f99a0 2022-01-31T19:40:47.359 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.357+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3e74027ae0 con 0x7f3e740f99a0 2022-01-31T19:40:47.360 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.357+0000 7f3e62ffd700 1 -- 172.21.15.167:0/1526925546 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v31) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3e6404f6f0 con 0x7f3e740f99a0 2022-01-31T19:40:47.360 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:40:47.361 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f3e5c05e5c0 msgr2=0x7f3e5c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:47.361 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f3e5c05e5c0 0x7f3e5c060a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3e68004c40 tx=0x7f3e6800b040).stop 2022-01-31T19:40:47.362 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 msgr2=0x7f3e74107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:47.362 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e74107560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3e64004e30 tx=0x7f3e64009b30).stop 2022-01-31T19:40:47.362 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 shutdown_connections 2022-01-31T19:40:47.363 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f3e5c05e5c0 0x7f3e5c060a80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:47.363 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e74060390 0x7f3e74107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:47.363 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 --2- 172.21.15.167:0/1526925546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e740f99a0 0x7f3e74107560 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:47.363 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.359+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 >> 172.21.15.167:0/1526925546 conn(0x7f3e740f0e60 msgr2=0x7f3e740f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:47.363 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.360+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 shutdown_connections 2022-01-31T19:40:47.364 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:47.360+0000 7f3e789b6700 1 -- 172.21.15.167:0/1526925546 wait complete. 2022-01-31T19:40:47.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:47 smithi167 conmon[32206]: cluster 2022-01-31T19:40:46.713197+0000 mgr.smithi167.aciqpk (mgr.14156) 142 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:47.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:47 smithi167 conmon[32206]: audit 2022-01-31T19:40:47.359352+0000 mon.smithi167 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.167:0/1526925546' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-01-31T19:40:47.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:47 smithi167 conmon[32206]: audit 2022-01-31T19:40:47.675630+0000 mon.smithi167 (mon.0) 433 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:47.931 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":31,"num_osds":6,"num_up_osds":5,"osd_up_since":1643658035,"num_in_osds":6,"osd_in_since":1643658036,"num_remapped_pgs":0} 2022-01-31T19:40:48.084 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:47 smithi171 conmon[35325]: cluster 2022-01-31T19:40:46.713197+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:40:48.085 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:47 smithi171 conmon[35325]: 14156) 142 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:48.085 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:47 smithi171 conmon[35325]: audit 2022-01-31T19:40:47.359352+0000 mon.smithi167 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.167:0/1526925546' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-01-31T19:40:48.086 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:47 smithi171 conmon[35325]: audit 2022-01-31T19:40:47.675630+0000 mon.smithi167 (mon.0) 433 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:48.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd stat -f json 2022-01-31T19:40:49.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 19:40:49 smithi171 conmon[51620]: debug 2022-01-31T19:40:49.154+0000 7f84fa0c4080 -1 osd.5 0 log_to_monitors {default=true} 2022-01-31T19:40:50.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:49 smithi167 conmon[32206]: cluster 2022-01-31T19:40:48.713529+0000 mgr.smithi167.aciqpk (mgr.14156) 143 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:50.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:49 smithi167 conmon[32206]: audit 2022-01-31T19:40:49.158060+0000 mon.smithi171 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-01-31T19:40:50.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:49 smithi167 conmon[32206]: audit 2022-01-31T19:40:49.158410+0000 mon.smithi167 (mon.0) 434 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-01-31T19:40:50.045 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:49 smithi171 conmon[35325]: cluster 2022-01-31T19:40:48.713529+0000 mgr.smithi167.aciqpk (mgr.14156) 143 2022-01-31T19:40:50.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:49 smithi171 conmon[35325]: : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:50.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:49 smithi171 conmon[35325]: audit 2022-01-31T19:40:49.158060+0000 mon.smithi171 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-01-31T19:40:50.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:49 smithi171 conmon[35325]: audit 2022-01-31T19:40:49.158410+0000 mon.smithi167 (mon.0) 434 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-01-31T19:40:50.919 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 -- 172.21.15.167:0/3748740889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 msgr2=0x7f36040f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:50.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 --2- 172.21.15.167:0/3748740889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f36040f5970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f35f4004660 tx=0x7f35f4009b30).stop 2022-01-31T19:40:50.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 -- 172.21.15.167:0/3748740889 shutdown_connections 2022-01-31T19:40:50.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 --2- 172.21.15.167:0/3748740889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36040f67f0 0x7f36040f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:50.921 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 --2- 172.21.15.167:0/3748740889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f36040f5970 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:50.921 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 -- 172.21.15.167:0/3748740889 >> 172.21.15.167:0/3748740889 conn(0x7f36040f0cb0 msgr2=0x7f36040f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:50.921 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 -- 172.21.15.167:0/3748740889 shutdown_connections 2022-01-31T19:40:50.921 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.916+0000 7f3608abe700 1 -- 172.21.15.167:0/3748740889 wait complete. 2022-01-31T19:40:50.922 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.917+0000 7f3608abe700 1 Processor -- start 2022-01-31T19:40:50.922 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.917+0000 7f3608abe700 1 -- start start 2022-01-31T19:40:50.922 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3608abe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f3604105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:50.922 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3608abe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36040f67f0 0x7f36041063c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:50.923 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3608abe700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604106970 con 0x7f36040f5550 2022-01-31T19:40:50.923 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3608abe700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604107170 con 0x7f36040f67f0 2022-01-31T19:40:50.923 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f3604105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:50.923 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f3604105ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32794/0 (socket says 172.21.15.167:32794) 2022-01-31T19:40:50.923 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 -- 172.21.15.167:0/4132747371 learned_addr learned my addr 172.21.15.167:0/4132747371 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:40:50.924 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3602ffd700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36040f67f0 0x7f36041063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:50.924 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 -- 172.21.15.167:0/4132747371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36040f67f0 msgr2=0x7f36041063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:50.924 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36040f67f0 0x7f36041063c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:50.924 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 -- 172.21.15.167:0/4132747371 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f4005040 con 0x7f36040f5550 2022-01-31T19:40:50.925 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f36037fe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f3604105ed0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f35f4004e30 tx=0x7f35f4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:50.925 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f4018070 con 0x7f36040f5550 2022-01-31T19:40:50.925 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f35f4013410 con 0x7f36040f5550 2022-01-31T19:40:50.925 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36041073f0 con 0x7f36040f5550 2022-01-31T19:40:50.926 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f401c3c0 con 0x7f36040f5550 2022-01-31T19:40:50.926 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.918+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3604107950 con 0x7f36040f5550 2022-01-31T19:40:50.927 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.919+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36040591e0 con 0x7f36040f5550 2022-01-31T19:40:50.927 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.920+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f35f401d420 con 0x7f36040f5550 2022-01-31T19:40:50.928 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.921+0000 7f35ebfff700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f35ec05e5c0 0x7f35ec060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:50.928 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.921+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4067+0+0 (secure 0 0 0) 0x7f35f407a5c0 con 0x7f36040f5550 2022-01-31T19:40:50.928 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.921+0000 7f3602ffd700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f35ec05e5c0 0x7f35ec060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:50.928 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.921+0000 7f3602ffd700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f35ec05e5c0 0x7f35ec060a80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f35f8004c40 tx=0x7f35f800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:50.929 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:50.924+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f35f404c700 con 0x7f36040f5550 2022-01-31T19:40:51.084 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.082+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3604027ae0 con 0x7f36040f5550 2022-01-31T19:40:51.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40 2022-01-31T19:40:51.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: :49.779945+0000 mon.smithi167 (mon.0) 435 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-01-31T19:40:51.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: cluster 2022-01-31T19:40:49.779998+0000 mon.smithi167 (mon.0) 436 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2022-01-31T19:40:51.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:49.780159+0000 mon.smithi167 (mon.0) 437 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:51.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:49.780642+0000 mon.smithi171 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:51.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:49.780863+0000 mon.smithi167 (mon.0) 438 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:51.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:50.378785+0000 mgr.smithi167.aciqpk (mgr.14156) 144 : cephadm [INF] Detected new or changed devices on smithi171 2022-01-31T19:40:51.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.383254+0000 mon.smithi167 (mon.0) 439 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:51.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.726833+0000 mon.smithi167 (mon.0) 440 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:51.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.727641+0000 mon.smithi167 (mon.0) 441 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:51.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.728441+0000 mon.smithi167 (mon.0) 442 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:51.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:50 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.735143+0000 mon.smithi167 (mon.0) 443 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:51.087 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.083+0000 7f35ebfff700 1 -- 172.21.15.167:0/4132747371 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v33) v1 ==== 74+0+130 (secure 0 0 0) 0x7f35f404fd20 con 0x7f36040f5550 2022-01-31T19:40:51.088 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:40:51.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.085+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f35ec05e5c0 msgr2=0x7f35ec060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:51.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.085+0000 7f3608abe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f35ec05e5c0 0x7f35ec060a80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f35f8004c40 tx=0x7f35f800b040).stop 2022-01-31T19:40:51.089 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.085+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 msgr2=0x7f3604105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:51.090 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.085+0000 7f3608abe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f3604105ed0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f35f4004e30 tx=0x7f35f4009b30).stop 2022-01-31T19:40:51.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 shutdown_connections 2022-01-31T19:40:51.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f35ec05e5c0 0x7f35ec060a80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:51.091 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36040f5550 0x7f3604105ed0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:51.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 --2- 172.21.15.167:0/4132747371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36040f67f0 0x7f36041063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:51.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 >> 172.21.15.167:0/4132747371 conn(0x7f36040f0cb0 msgr2=0x7f36040fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:51.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 shutdown_connections 2022-01-31T19:40:51.092 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:51.086+0000 7f3608abe700 1 -- 172.21.15.167:0/4132747371 wait complete. 2022-01-31T19:40:51.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[51620]: debug 2022-01-31T19:40:50.790+0000 7f84f0627700 -1 osd.5 0 waiting for initial osdmap 2022-01-31T19:40:51.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[51620]: debug 2022-01-31T19:40:50.799+0000 7f84e959c700 -1 osd.5 33 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2022-01-31T19:40:51.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:49.779945+0000 mon.smithi167 (mon.0) 435 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-01-31T19:40:51.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: cluster 2022-01-31T19:40:49.779998+0000 mon.smithi167 (mon.0) 436 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2022-01-31T19:40:51.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:49.780159+0000 mon.smithi167 (mon.0) 437 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:51.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:49.780642+0000 mon.smithi171 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:51.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:49.780863+0000 mon.smithi167 (mon.0) 438 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2022-01-31T19:40:51.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:50.378785+0000 mgr.smithi167.aciqpk (mgr.14156) 144 : cephadm [INF] Detected new or changed devices on smithi171 2022-01-31T19:40:51.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:50.383254+0000 mon.smithi167 (mon.0) 439 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:51.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:50.726833+0000 mon.smithi167 (mon.0) 440 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:51.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:50.727641+0000 mon.smithi167 (mon.0) 441 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:51.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:50.728441+0000 mon.smithi167 (mon.0) 442 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:40:51.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:50 smithi171 conmon[35325]: audit 2022-01-31T19:40:50.735143+0000 mon.smithi167 (mon.0) 443 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:51.539 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":33,"num_osds":6,"num_up_osds":5,"osd_up_since":1643658035,"num_in_osds":6,"osd_in_since":1643658036,"num_remapped_pgs":0} 2022-01-31T19:40:52.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: cluster 2022-01-31T19:40:50.713822+0000 mgr.smithi167.aciqpk (mgr.14156) 145 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:52.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: cephadm 2022-01-31T19:40:50.729658+0000 mgr.smithi167.aciqpk (mgr.14156) 146 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 5339M 2022-01-31T19:40:52.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01-31T19:40:50.781479+0000 mon.smithi167 (mon 2022-01-31T19:40:52.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: .0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2022-01-31T19:40:52.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: cluster 2022- 2022-01-31T19:40:52.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 01-31T19:40:50.781574+0000 mon.smithi167 (mon.0) 445 : cluster 2022-01-31T19:40:52.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: [DBG] osdmap e33: 6 total, 5 up, 6 in 2022-01-31T19:40:52.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:40:52.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 40:50.781924+0000 mon.smithi167 (mon.0) 446 2022-01-31T19:40:52.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:52.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:40:52.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: :40:50.789968+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:52.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 447 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:52.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:40:52.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: :40:50.884995+0000 mon.smithi167 (mon.0) 2022-01-31T19:40:52.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 448 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:52.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:40:52.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: T19:40:50.888379+0000 mon.smithi167 (mon. 2022-01-31T19:40:52.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 0) 449 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:52.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:40:52.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: -31T19:40:50.893212+0000 mon.smithi167 2022-01-31T19:40:52.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: (mon.0) 450 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:52.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:40:52.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 01-31T19:40:50.894530+0000 mon.smithi167 2022-01-31T19:40:52.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: (mon.0) 451 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:52.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:40:52.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: 01-31T19:40:50.895304+0000 2022-01-31T19:40:52.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: mon.smithi167 (mon.0) 452 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:52.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:40:52.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: -31T19:40:51.083776+0000 mon.smithi167 ( 2022-01-31T19:40:52.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:51 smithi171 conmon[35325]: mon.0) 453 : audit [DBG] from='client.? 172.21.15.167:0/4132747371' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-01-31T19:40:52.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: cluster 2022-01-31T19:40:50.713822+0000 mgr.smithi167.aciqpk (mgr.14156) 145 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:52.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: cephadm 2022-01-31T19:40:50.729658+0000 mgr.smithi167.aciqpk (mgr.14156) 146 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 5339M 2022-01-31T19:40:52.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.781479+0000 mon.smithi167 (mon.0) 444 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2022-01-31T19:40:52.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: cluster 2022-01-31T19:40:50.781574+0000 mon.smithi167 (mon.0) 445 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2022-01-31T19:40:52.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.781924+0000 mon.smithi167 (mon.0) 446 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:52.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.789968+0000 mon.smithi167 (mon.0) 447 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:52.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.884995+0000 mon.smithi167 (mon.0 2022-01-31T19:40:52.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: ) 448 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:52.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.888379+0000 mon.smithi167 (mon.0) 449 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:52.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.893212+0000 mon.smithi167 (mon.0) 450 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:40:52.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.894530+0000 mon.smithi167 (mon.0) 451 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:40:52.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:50.895304+0000 mon.smithi167 (mon.0) 452 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:40:52.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: audit 2022-01-31T19:40:51.083776+0000 mon.smithi167 (mon 2022-01-31T19:40:52.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:51 smithi167 conmon[32206]: .0) 453 : audit [DBG] from='client.? 172.21.15.167:0/4132747371' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-01-31T19:40:52.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd stat -f json 2022-01-31T19:40:52.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: cluster 2022-01-31T19:40:50.200368+0000 2022-01-31T19:40:52.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:52.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: cluster 2022-01-31T19:40:50.200470+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:52.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: audit 2022-01-31T19:40:51.789293+0000 mon.smithi167 (mon.0) 454 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: cluster 2022-01-31T19:40:51.796098+0000 mon.smithi167 (mon.0) 455 : cluster [INF] osd.5 [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] boot 2022-01-31T19:40:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: cluster 2022-01-31T19:40:51.796133+0000 mon.smithi167 (mon.0) 456 : cluster [DBG] osdmap e34: 6 total, 6 up, 6 in 2022-01-31T19:40:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: audit 2022-01-31T19:40:51. 2022-01-31T19:40:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: 796773+0000 mon.smithi167 (mon.0) 457 : audit [DBG] 2022-01-31T19:40:52.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:52 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:53.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: cluster 2022-01-31T19:40:50.200368+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2022-01-31T19:40:53.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: cluster 2022-01-31T19:40:50.200470+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2022-01-31T19:40:53.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: audit 2022-01-31T19:40:51.789293+0000 mon.smithi167 (mon.0) 454 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: cluster 2022-01-31T19:40 2022-01-31T19:40:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: :51.796098+0000 mon.smithi167 (mon.0) 455 : cluster [INF] osd.5 [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] boot 2022-01-31T19:40:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: cluster 2022-01-31T19:40:51.796133+0000 mon.smithi167 ( 2022-01-31T19:40:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: mon.0) 456 : cluster [DBG] osdmap e34: 6 total, 6 up, 6 in 2022-01-31T19:40:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:40:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:52 smithi171 conmon[35325]: :40:51.796773+0000 mon.smithi167 (mon.0) 457 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:40:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:53 smithi171 conmon[35325]: cluster 2022-01-31T19:40:52.714154+0000 mgr.smithi167.aciqpk (mgr.14156) 147 : cluster [DBG] 2022-01-31T19:40:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:53 smithi171 conmon[35325]: pgmap v111: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:53 smithi171 conmon[35325]: cluster 2022-01-31T19:40:52.809359+0000 mon.smithi167 (mon.0) 458 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2022-01-31T19:40:54.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:53 smithi167 conmon[32206]: cluster 2022-01-31T19:40:52.714154+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:40:54.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:53 smithi167 conmon[32206]: ) 147 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2022-01-31T19:40:54.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:53 smithi167 conmon[32206]: cluster 2022-01-31T19:40:52.809359+0000 mon.smithi167 (mon.0) 458 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2022-01-31T19:40:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 -- 172.21.15.167:0/1528645303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 msgr2=0x7f868c0f5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 --2- 172.21.15.167:0/1528645303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c0f5120 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8680004660 tx=0x7f8680009b30).stop 2022-01-31T19:40:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 -- 172.21.15.167:0/1528645303 shutdown_connections 2022-01-31T19:40:55.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 --2- 172.21.15.167:0/1528645303 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f868c0f5fa0 0x7f868c0f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:55.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 --2- 172.21.15.167:0/1528645303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c0f5120 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:55.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 -- 172.21.15.167:0/1528645303 >> 172.21.15.167:0/1528645303 conn(0x7f868c0f0460 msgr2=0x7f868c0f2880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:55.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 -- 172.21.15.167:0/1528645303 shutdown_connections 2022-01-31T19:40:55.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.039+0000 7f86916a3700 1 -- 172.21.15.167:0/1528645303 wait complete. 2022-01-31T19:40:55.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f86916a3700 1 Processor -- start 2022-01-31T19:40:55.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f86916a3700 1 -- start start 2022-01-31T19:40:55.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f86916a3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c1055a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:55.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f86916a3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f868c0f5fa0 0x7f868c105a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f86916a3700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c106040 con 0x7f868c0f4d00 2022-01-31T19:40:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f86916a3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c106180 con 0x7f868c0f5fa0 2022-01-31T19:40:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868affd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c1055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868affd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c1055a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32834/0 (socket says 172.21.15.167:32834) 2022-01-31T19:40:55.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868affd700 1 -- 172.21.15.167:0/298899606 learned_addr learned my addr 172.21.15.167:0/298899606 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:40:55.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868a7fc700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f868c0f5fa0 0x7f868c105a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:55.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868affd700 1 -- 172.21.15.167:0/298899606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f868c0f5fa0 msgr2=0x7f868c105a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:55.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868affd700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f868c0f5fa0 0x7f868c105a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:55.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.040+0000 7f868affd700 1 -- 172.21.15.167:0/298899606 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680005040 con 0x7f868c0f4d00 2022-01-31T19:40:55.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.041+0000 7f868affd700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c1055a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8680014fd0 tx=0x7f8680009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:55.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.041+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8680019070 con 0x7f868c0f4d00 2022-01-31T19:40:55.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.041+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8680013410 con 0x7f868c0f4d00 2022-01-31T19:40:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.041+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c05ee70 con 0x7f868c0f4d00 2022-01-31T19:40:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.041+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8680012620 con 0x7f868c0f4d00 2022-01-31T19:40:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.041+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c05f3d0 con 0x7f868c0f4d00 2022-01-31T19:40:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.042+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c045bb0 con 0x7f868c0f4d00 2022-01-31T19:40:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.042+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f8680024030 con 0x7f868c0f4d00 2022-01-31T19:40:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.042+0000 7f86737fe700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f867405e5c0 0x7f8674060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.043+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f868007a7d0 con 0x7f868c0f4d00 2022-01-31T19:40:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.043+0000 7f868a7fc700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f867405e5c0 0x7f8674060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.043+0000 7f868a7fc700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f867405e5c0 0x7f8674060a80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f867c0041d0 tx=0x7f867c009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:55.051 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.047+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f868004c840 con 0x7f868c0f4d00 2022-01-31T19:40:55.204 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.202+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f868c0fd5a0 con 0x7f868c0f4d00 2022-01-31T19:40:55.204 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.203+0000 7f86737fe700 1 -- 172.21.15.167:0/298899606 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v35) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8680022020 con 0x7f868c0f4d00 2022-01-31T19:40:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:40:55.205 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.204+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f867405e5c0 msgr2=0x7f8674060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.204+0000 7f86916a3700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f867405e5c0 0x7f8674060a80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f867c0041d0 tx=0x7f867c009040).stop 2022-01-31T19:40:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.204+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 msgr2=0x7f868c1055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.204+0000 7f86916a3700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c1055a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8680014fd0 tx=0x7f8680009b30).stop 2022-01-31T19:40:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 shutdown_connections 2022-01-31T19:40:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f867405e5c0 0x7f8674060a80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f868c0f4d00 0x7f868c1055a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 --2- 172.21.15.167:0/298899606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f868c0f5fa0 0x7f868c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 >> 172.21.15.167:0/298899606 conn(0x7f868c0f0460 msgr2=0x7f868c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:55.208 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 shutdown_connections 2022-01-31T19:40:55.208 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:55.205+0000 7f86916a3700 1 -- 172.21.15.167:0/298899606 wait complete. 2022-01-31T19:40:55.622 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":35,"num_osds":6,"num_up_osds":6,"osd_up_since":1643658051,"num_in_osds":6,"osd_in_since":1643658036,"num_remapped_pgs":0} 2022-01-31T19:40:55.623 INFO:tasks.cephadm:Setting up client nodes... 2022-01-31T19:40:55.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-01-31T19:40:55.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:55 smithi167 conmon[32206]: cluster 2022-01-31T19:40:54.714451+0000 mgr.smithi167.aciqpk (mgr.14156) 148 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:40:55.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:55 smithi167 conmon[32206]: audit 2022-01-31T19:40:55.203651+0000 mon.smithi167 (mon.0) 459 : audit [DBG] from='client.? 172.21.15.167:0/298899606' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-01-31T19:40:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:55 smithi171 conmon[35325]: cluster 2022-01-31T19:40:54.714451+0000 mgr.smithi167.aciqpk (mgr.14156) 148 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:40:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:55 smithi171 conmon[35325]: 2022-01-31T19:40:56.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:55 smithi171 conmon[35325]: audit 2022-01-31T19:40:55.203651+0000 mon.smithi167 (mon.0) 459 : audit [DBG] from='client.? 172.21.15.167:0/298899606' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-01-31T19:40:57.333 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 -- 172.21.15.167:0/3650792121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 msgr2=0x7f151c0f71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:57.333 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 --2- 172.21.15.167:0/3650792121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f71c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1518004660 tx=0x7f1518009b30).stop 2022-01-31T19:40:57.334 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 -- 172.21.15.167:0/3650792121 shutdown_connections 2022-01-31T19:40:57.334 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 --2- 172.21.15.167:0/3650792121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f59d0 0x7f151c0f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:40:57.334 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 --2- 172.21.15.167:0/3650792121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f71c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:57.334 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 -- 172.21.15.167:0/3650792121 >> 172.21.15.167:0/3650792121 conn(0x7f151c0f05e0 msgr2=0x7f151c0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:57.335 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 -- 172.21.15.167:0/3650792121 shutdown_connections 2022-01-31T19:40:57.335 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.330+0000 7f1523796700 1 -- 172.21.15.167:0/3650792121 wait complete. 2022-01-31T19:40:57.335 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1523796700 1 Processor -- start 2022-01-31T19:40:57.335 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1523796700 1 -- start start 2022-01-31T19:40:57.336 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1523796700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f59d0 0x7f151c0fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:57.336 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1523796700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:57.336 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1523796700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0f8b30 con 0x7f151c0f6da0 2022-01-31T19:40:57.336 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1523796700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0f8c70 con 0x7f151c0f59d0 2022-01-31T19:40:57.337 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:57.337 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32852/0 (socket says 172.21.15.167:32852) 2022-01-31T19:40:57.337 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 -- 172.21.15.167:0/3307586262 learned_addr learned my addr 172.21.15.167:0/3307586262 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:40:57.337 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1521532700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f59d0 0x7f151c0fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:57.338 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 -- 172.21.15.167:0/3307586262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f59d0 msgr2=0x7f151c0fa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:57.338 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f59d0 0x7f151c0fa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:57.338 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 -- 172.21.15.167:0/3307586262 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1518005040 con 0x7f151c0f6da0 2022-01-31T19:40:57.339 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.331+0000 7f1520d31700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f8540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f15100093f0 tx=0x7f1510006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:57.339 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.332+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1510013640 con 0x7f151c0f6da0 2022-01-31T19:40:57.339 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.332+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1510013c30 con 0x7f151c0f6da0 2022-01-31T19:40:57.339 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.332+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c0f8f50 con 0x7f151c0f6da0 2022-01-31T19:40:57.339 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.332+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1510012be0 con 0x7f151c0f6da0 2022-01-31T19:40:57.341 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.332+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c10f1b0 con 0x7f151c0f6da0 2022-01-31T19:40:57.341 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.333+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c045bb0 con 0x7f151c0f6da0 2022-01-31T19:40:57.342 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.333+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f151001a060 con 0x7f151c0f6da0 2022-01-31T19:40:57.342 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.334+0000 7f150dffb700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f150805e5c0 0x7f1508060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:57.342 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.334+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f1510078500 con 0x7f151c0f6da0 2022-01-31T19:40:57.343 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.334+0000 7f1521532700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f150805e5c0 0x7f1508060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:57.343 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.334+0000 7f1521532700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f150805e5c0 0x7f1508060a80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1518013620 tx=0x7f1518009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:57.343 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.338+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f151004a440 con 0x7f151c0f6da0 2022-01-31T19:40:57.562 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.561+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f151c0fe050 con 0x7f151c0f6da0 2022-01-31T19:40:57.567 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.566+0000 7f150dffb700 1 -- 172.21.15.167:0/3307586262 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f151004da60 con 0x7f151c0f6da0 2022-01-31T19:40:57.568 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2022-01-31T19:40:57.568 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBJO/hhA6iCIRAArQogrr63GrjBe5Iu0xguMA== 2022-01-31T19:40:57.569 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f150805e5c0 msgr2=0x7f1508060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:57.569 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f150805e5c0 0x7f1508060a80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1518013620 tx=0x7f1518009b30).stop 2022-01-31T19:40:57.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 msgr2=0x7f151c0f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:57.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f8540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f15100093f0 tx=0x7f1510006ec0).stop 2022-01-31T19:40:57.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 shutdown_connections 2022-01-31T19:40:57.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f150805e5c0 0x7f1508060a80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:57.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f59d0 0x7f151c0fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:57.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 --2- 172.21.15.167:0/3307586262 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f151c0f6da0 0x7f151c0f8540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:57.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 >> 172.21.15.167:0/3307586262 conn(0x7f151c0f05e0 msgr2=0x7f151c0f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:57.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 shutdown_connections 2022-01-31T19:40:57.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:40:57.568+0000 7f1523796700 1 -- 172.21.15.167:0/3307586262 wait complete. 2022-01-31T19:40:57.967 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:40:57.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-01-31T19:40:57.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-01-31T19:40:58.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-01-31T19:40:58.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:58 smithi171 conmon[35325]: cluster 2022-01-31T19:40:56.714866+0000 mgr.smithi167.aciqpk (mgr.14156) 149 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:40:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:58 smithi171 conmon[35325]: audit 2022-01-31T19:40:57.050770+0000 mon.smithi167 (mon.0) 460 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:58 smithi171 conmon[35325]: audit 2022-01-31T19:40:57.477772+0000 mon.smithi167 (mon.0) 461 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:58 smithi171 conmon[35325]: audit 2022-01-31T19:40:57.562048+0000 mon.smithi167 (mon.0) 462 : audit [INF] from='client.? 172.21.15.167:0/3307586262' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-01-31T19:40:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:58 smithi171 conmon[35325]: audit 2022-01-31T19:40:57.567305+0000 mon.smithi167 (mon.0) 463 : audit [INF] from='client.? 172.21.15.167:0/3307586262' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-01-31T19:40:58.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:58 smithi167 conmon[32206]: cluster 2022-01-31T19:40:56.714866+0000 mgr.smithi167.aciqpk (mgr.14156) 149 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:40:58.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:58 smithi167 conmon[32206]: audit 2022-01-31T19:40:57.050770+0000 mon.smithi167 (mon.0) 460 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:58.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:58 smithi167 conmon[32206]: audit 2022-01-31T19:40:57.477772+0000 mon.smithi167 (mon.0) 461 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:40:58.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:58 smithi167 conmon[32206]: audit 2022-01-31T19:40:57.562048+0000 mon.smithi167 (mon.0) 462 : audit [INF] from='client.? 172.21.15.167:0/3307586262' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-01-31T19:40:58.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:58 smithi167 conmon[32206]: audit 2022-01-31T19:40:57.567305+0000 mon.smithi167 (mon.0) 463 : audit [INF] from='client.? 172.21.15.167:0/3307586262' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-01-31T19:40:59.316 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:59 smithi171 conmon[35325]: audit 2022-01-31T19:40:58.825037+0000 mon.smithi167 (mon.0) 464 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:40:59.317 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:40:59 smithi171 conmon[35325]: audit 2022-01-31T19:40:58.826055+0000 mon.smithi167 (mon.0) 465 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:40:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:59 smithi167 conmon[32206]: audit 2022-01-31T19:40:58.825037+0000 mon.smithi167 (mon.0) 464 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:40:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:59 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:40:59.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:40:59 smithi167 conmon[32206]: -31T19:40:58.826055+0000 mon.smithi167 (mon.0) 465 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:40:59.467 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 -- 172.21.15.171:0/2868844987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 msgr2=0x7fe2800f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:59.468 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 --2- 172.21.15.171:0/2868844987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe2800f6e10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fe270004660 tx=0x7fe270009b30).stop 2022-01-31T19:40:59.468 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 -- 172.21.15.171:0/2868844987 shutdown_connections 2022-01-31T19:40:59.469 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 --2- 172.21.15.171:0/2868844987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe2800f6e10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:59.469 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 --2- 172.21.15.171:0/2868844987 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2800f5710 0x7fe2800f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:59.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 -- 172.21.15.171:0/2868844987 >> 172.21.15.171:0/2868844987 conn(0x7fe2800f0e50 msgr2=0x7fe2800f3290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:59.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 -- 172.21.15.171:0/2868844987 shutdown_connections 2022-01-31T19:40:59.470 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.461+0000 7fe2851af700 1 -- 172.21.15.171:0/2868844987 wait complete. 2022-01-31T19:40:59.471 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe2851af700 1 Processor -- start 2022-01-31T19:40:59.471 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe2851af700 1 -- start start 2022-01-31T19:40:59.472 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe2851af700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2800f5710 0x7fe28010c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:59.472 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe2851af700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe28010e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:59.473 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe2851af700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280060570 con 0x7fe2800f5710 2022-01-31T19:40:59.473 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe2851af700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800606b0 con 0x7fe2800f69b0 2022-01-31T19:40:59.474 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27e59c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe28010e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:59.474 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27e59c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe28010e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:49388/0 (socket says 172.21.15.171:49388) 2022-01-31T19:40:59.475 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27e59c700 1 -- 172.21.15.171:0/3787011662 learned_addr learned my addr 172.21.15.171:0/3787011662 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-01-31T19:40:59.475 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27ed9d700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2800f5710 0x7fe28010c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:59.476 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27e59c700 1 -- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2800f5710 msgr2=0x7fe28010c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:59.476 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27e59c700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2800f5710 0x7fe28010c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:59.477 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.462+0000 7fe27e59c700 1 -- 172.21.15.171:0/3787011662 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270005040 con 0x7fe2800f69b0 2022-01-31T19:40:59.477 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.463+0000 7fe27e59c700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe28010e660 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe270009990 tx=0x7fe270009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:59.477 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.463+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe270018070 con 0x7fe2800f69b0 2022-01-31T19:40:59.478 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.463+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe270013410 con 0x7fe2800f69b0 2022-01-31T19:40:59.478 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.463+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28010eb50 con 0x7fe2800f69b0 2022-01-31T19:40:59.478 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.463+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27001c620 con 0x7fe2800f69b0 2022-01-31T19:40:59.479 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.463+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28010ef20 con 0x7fe2800f69b0 2022-01-31T19:40:59.479 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.464+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2800f4b00 con 0x7fe2800f69b0 2022-01-31T19:40:59.479 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.464+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fe27001c8e0 con 0x7fe2800f69b0 2022-01-31T19:40:59.479 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.464+0000 7fe2677fe700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe26805e5c0 0x7fe268060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:40:59.480 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.465+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fe27007a4c0 con 0x7fe2800f69b0 2022-01-31T19:40:59.480 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.465+0000 7fe27ed9d700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe26805e5c0 0x7fe268060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:40:59.480 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.465+0000 7fe27ed9d700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe26805e5c0 0x7fe268060a80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe280060300 tx=0x7fe274009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:40:59.481 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.468+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe27004c400 con 0x7fe2800f69b0 2022-01-31T19:40:59.676 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.674+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe280045bb0 con 0x7fe2800f69b0 2022-01-31T19:40:59.680 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.679+0000 7fe2677fe700 1 -- 172.21.15.171:0/3787011662 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe27004fa20 con 0x7fe2800f69b0 2022-01-31T19:40:59.681 INFO:teuthology.orchestra.run.smithi171.stdout:[client.1] 2022-01-31T19:40:59.681 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQBLO/hhq5FUKBAAL4bDJz32UhuhGmDgZeyNmg== 2022-01-31T19:40:59.682 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe26805e5c0 msgr2=0x7fe268060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:59.683 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe26805e5c0 0x7fe268060a80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe280060300 tx=0x7fe274009040).stop 2022-01-31T19:40:59.683 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 msgr2=0x7fe28010e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:40:59.683 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe28010e660 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe270009990 tx=0x7fe270009b30).stop 2022-01-31T19:40:59.683 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 shutdown_connections 2022-01-31T19:40:59.684 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe26805e5c0 0x7fe268060a80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:59.684 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2800f5710 0x7fe28010c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:59.684 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 --2- 172.21.15.171:0/3787011662 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe2800f69b0 0x7fe28010e660 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:40:59.684 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 >> 172.21.15.171:0/3787011662 conn(0x7fe2800f0e50 msgr2=0x7fe2800fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:40:59.685 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 shutdown_connections 2022-01-31T19:40:59.685 INFO:teuthology.orchestra.run.smithi171.stderr:2022-01-31T19:40:59.681+0000 7fe2851af700 1 -- 172.21.15.171:0/3787011662 wait complete. 2022-01-31T19:41:00.042 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:41:00.042 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2022-01-31T19:41:00.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-01-31T19:41:00.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd dump --format=json 2022-01-31T19:41:00.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:00 smithi171 conmon[35325]: cluster 2022-01-31T19:40:58.715193+0000 mgr.smithi167.aciqpk (mgr.14156) 150 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:00.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:00 smithi171 conmon[35325]: audit 2022-01-31T19:40:59.675943+0000 mon.smithi171 (mon.1) 14 : audit [INF] from='client.? 172.21.15.171:0/3787011662' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-01-31T19:41:00.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:00 smithi171 conmon[35325]: audit 2022-01-31T19:40:59.676424+0000 mon.smithi167 (mon.0) 466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-01-31T19:41:00.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:00 smithi171 conmon[35325]: audit 2022-01-31T19:40:59.680028+0000 mon.smithi167 (mon.0) 467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-01-31T19:41:00.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: cluster 2022-01-31T19:40:58.715193+0000 mgr.smithi167.aciqpk (mgr.14156) 150 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:00.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: audit 2022-01-31T19:40:59.675943+0000 mon.smithi171 (mon.1) 14 : audit [INF] from='client.? 172.21.15.171:0/3787011662' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-01-31T19:41:00.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: audit 2022 2022-01-31T19:41:00.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: -01-31T19:40:59.676424+0000 mon.smithi167 (mon.0) 466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-01-31T19:41:00.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: audit 2022 2022-01-31T19:41:00.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: -01-31T19:40:59.680028+0000 mon.smithi167 2022-01-31T19:41:00.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:00 smithi167 conmon[32206]: (mon.0) 467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-01-31T19:41:00.539 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:01.780 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 -- 172.21.15.167:0/3437447005 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 msgr2=0x7f5fdc0f6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:01.781 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 --2- 172.21.15.167:0/3437447005 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc0f6ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f5fcc004660 tx=0x7f5fcc009b30).stop 2022-01-31T19:41:01.781 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 -- 172.21.15.167:0/3437447005 shutdown_connections 2022-01-31T19:41:01.782 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 --2- 172.21.15.167:0/3437447005 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc0f6ef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:01.782 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 --2- 172.21.15.167:0/3437447005 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fdc0f57f0 0x7f5fdc0f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:01.782 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 -- 172.21.15.167:0/3437447005 >> 172.21.15.167:0/3437447005 conn(0x7f5fdc0f0f50 msgr2=0x7f5fdc0f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:01.782 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 -- 172.21.15.167:0/3437447005 shutdown_connections 2022-01-31T19:41:01.783 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.777+0000 7f5fe389e700 1 -- 172.21.15.167:0/3437447005 wait complete. 2022-01-31T19:41:01.783 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe389e700 1 Processor -- start 2022-01-31T19:41:01.784 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe389e700 1 -- start start 2022-01-31T19:41:01.784 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe389e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fdc0f57f0 0x7f5fdc1060b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:01.784 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe389e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc1065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:01.785 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe389e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc106b50 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.785 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe389e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc106c90 con 0x7f5fdc0f57f0 2022-01-31T19:41:01.785 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc1065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:01.786 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc1065a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32868/0 (socket says 172.21.15.167:32868) 2022-01-31T19:41:01.786 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 -- 172.21.15.167:0/1447646925 learned_addr learned my addr 172.21.15.167:0/1447646925 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:01.786 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe163a700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fdc0f57f0 0x7f5fdc1060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:01.787 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 -- 172.21.15.167:0/1447646925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fdc0f57f0 msgr2=0x7f5fdc1060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:01.787 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fdc0f57f0 0x7f5fdc1060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:01.787 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 -- 172.21.15.167:0/1447646925 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc005040 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.788 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.778+0000 7f5fe0e39700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc1065a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5fcc009990 tx=0x7f5fcc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:01.788 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.779+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fcc018070 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.788 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.779+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc1075d0 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.789 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.779+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5fcc013410 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.789 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.779+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fcc01de80 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.790 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.779+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc107b30 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.790 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.780+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc045bb0 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.791 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.780+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f5fcc01d420 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.791 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.780+0000 7f5fd1ffb700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5fc805e5c0 0x7f5fc8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:01.792 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.780+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f5fcc07a170 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.792 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.781+0000 7f5fe163a700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5fc805e5c0 0x7f5fc8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:01.792 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.781+0000 7f5fe163a700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5fc805e5c0 0x7f5fc8060a80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5fdc0f68f0 tx=0x7f5fd800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:01.793 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.784+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5fcc04c0b0 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:01 smithi167 conmon[32206]: cluster 2022-01-31T19:41:00.715722+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:41:01.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:01 smithi167 conmon[32206]: ) 151 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:01.936 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.935+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fdc027ae0 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.937 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.936+0000 7f5fd1ffb700 1 -- 172.21.15.167:0/1447646925 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+10694 (secure 0 0 0) 0x7f5fcc04f6d0 con 0x7f5fdc0f6a90 2022-01-31T19:41:01.938 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:01.939 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":35,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","created":"2022-01-31T19:37:06.503417+0000","modified":"2022-01-31T19:40:52.799933+0000","last_up_change":"2022-01-31T19:40:51.787533+0000","last_in_change":"2022-01-31T19:40:36.089120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-01-31T19:40:13.386038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"6b39ea02-b5a4-4a19-bd99-849c0378fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6803","nonce":2333495797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6805","nonce":2333495797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6809","nonce":2333495797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6807","nonce":2333495797}]},"public_addr":"172.21.15.167:6803/2333495797","cluster_addr":"172.21.15.167:6805/2333495797","heartbeat_back_addr":"172.21.15.167:6809/2333495797","heartbeat_front_addr":"172.21.15.167:6807/2333495797","state":["exists","up"]},{"osd":1,"uuid":"214aeaf2-f8c7-4638-96b6-4e6bb4bd802f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6811","nonce":3267421031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6813","nonce":3267421031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6817","nonce":3267421031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6815","nonce":3267421031}]},"public_addr":"172.21.15.167:6811/3267421031","cluster_addr":"172.21.15.167:6813/3267421031","heartbeat_back_addr":"172.21.15.167:6817/3267421031","heartbeat_front_addr":"172.21.15.167:6815/3267421031","state":["exists","up"]},{"osd":2,"uuid":"3536e17c-9f9a-4867-91f4-f84b2b5d53a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6819","nonce":2914232381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6821","nonce":2914232381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6825","nonce":2914232381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6823","nonce":2914232381}]},"public_addr":"172.21.15.167:6819/2914232381","cluster_addr":"172.21.15.167:6821/2914232381","heartbeat_back_addr":"172.21.15.167:6825/2914232381","heartbeat_front_addr":"172.21.15.167:6823/2914232381","state":["exists","up"]},{"osd":3,"uuid":"e924c508-4baf-4ed0-9a32-42a51710ca8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6801","nonce":1700293141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6803","nonce":1700293141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6807","nonce":1700293141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6805","nonce":1700293141}]},"public_addr":"172.21.15.171:6801/1700293141","cluster_addr":"172.21.15.171:6803/1700293141","heartbeat_back_addr":"172.21.15.171:6807/1700293141","heartbeat_front_addr":"172.21.15.171:6805/1700293141","state":["exists","up"]},{"osd":4,"uuid":"c4310782-5c0e-4962-b32e-45e28ac014ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6809","nonce":3746074472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6811","nonce":3746074472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6815","nonce":3746074472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6813","nonce":3746074472}]},"public_addr":"172.21.15.171:6809/3746074472","cluster_addr":"172.21.15.171:6811/3746074472","heartbeat_back_addr":"172.21.15.171:6815/3746074472","heartbeat_front_addr":"172.21.15.171:6813/3746074472","state":["exists","up"]},{"osd":5,"uuid":"2ef7b4c2-8076-4866-b26a-8027fd249d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6817","nonce":1956867870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6819","nonce":1956867870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6823","nonce":1956867870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6821","nonce":1956867870}]},"public_addr":"172.21.15.171:6817/1956867870","cluster_addr":"172.21.15.171:6819/1956867870","heartbeat_back_addr":"172.21.15.171:6823/1956867870","heartbeat_front_addr":"172.21.15.171:6821/1956867870","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:26.422630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:45.101018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:03.040927+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:18.082335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:33.822236+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:50.200473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2480743863":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6801/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2306410502":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6800/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2760109630":"2022-02-01T19:37:34.139403+0000","172.21.15.167:0/694084204":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6800/3497254897":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6801/3497254897":"2022-02-01T19:37:34.139403+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-31T19:41:01.939 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5fc805e5c0 msgr2=0x7f5fc8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:01.940 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5fc805e5c0 0x7f5fc8060a80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5fdc0f68f0 tx=0x7f5fd800b040).stop 2022-01-31T19:41:01.940 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 msgr2=0x7f5fdc1065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:01.940 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc1065a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5fcc009990 tx=0x7f5fcc009b30).stop 2022-01-31T19:41:01.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 shutdown_connections 2022-01-31T19:41:01.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5fc805e5c0 0x7f5fc8060a80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:01.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5fdc0f57f0 0x7f5fdc1060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:01.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 --2- 172.21.15.167:0/1447646925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fdc0f6a90 0x7f5fdc1065a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:01.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 >> 172.21.15.167:0/1447646925 conn(0x7f5fdc0f0f50 msgr2=0x7f5fdc0fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:01.942 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 shutdown_connections 2022-01-31T19:41:01.942 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:01.938+0000 7f5fe389e700 1 -- 172.21.15.167:0/1447646925 wait complete. 2022-01-31T19:41:02.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:01 smithi171 conmon[35325]: cluster 2022-01-31T19:41:00.715722+0000 mgr.smithi167.aciqpk (mgr.14156) 151 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:02.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:01 smithi171 conmon[35325]: 2022-01-31T19:41:02.358 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-01-31T19:40:13.386038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2022-01-31T19:41:02.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2022-01-31T19:41:02.799 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:02.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:02 smithi167 conmon[32206]: audit 2022-01-31T19:41:01.936651+0000 mon.smithi167 (mon.0) 468 : audit [DBG] from='client.? 172.21.15.167:0/1447646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:41:03.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:02 smithi171 conmon[35325]: audit 2022-01-31T19:41:01.936651+0000 mon.smithi167 (mon.0) 468 2022-01-31T19:41:03.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:02 smithi171 conmon[35325]: : audit [DBG] from='client.? 172.21.15.167:0/1447646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:41:03.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:03 smithi167 conmon[32206]: cluster 2022-01-31T19:41:02.715990+0000 mgr.smithi167.aciqpk (mgr.14156) 152 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:03.993 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 -- 172.21.15.167:0/3088541053 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 msgr2=0x7fb8580fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:03.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 --2- 172.21.15.167:0/3088541053 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb8580fbea0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb84c004660 tx=0x7fb84c009b30).stop 2022-01-31T19:41:03.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 -- 172.21.15.167:0/3088541053 shutdown_connections 2022-01-31T19:41:03.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 --2- 172.21.15.167:0/3088541053 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb8580fbea0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:03.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 --2- 172.21.15.167:0/3088541053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 0x7fb8580f94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:03.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 -- 172.21.15.167:0/3088541053 >> 172.21.15.167:0/3088541053 conn(0x7fb8580f0f50 msgr2=0x7fb8580f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:03.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.990+0000 7fb85cab5700 1 -- 172.21.15.167:0/3088541053 shutdown_connections 2022-01-31T19:41:03.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 -- 172.21.15.167:0/3088541053 wait complete. 2022-01-31T19:41:03.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 Processor -- start 2022-01-31T19:41:03.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 -- start start 2022-01-31T19:41:03.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 0x7fb858107130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:03.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb858107620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:03.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb858107bd0 con 0x7fb8580f9a50 2022-01-31T19:41:03.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb85cab5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb858107d10 con 0x7fb858060950 2022-01-31T19:41:03.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.991+0000 7fb855d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb858107620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:03.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb855d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb858107620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32886/0 (socket says 172.21.15.167:32886) 2022-01-31T19:41:03.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb855d9b700 1 -- 172.21.15.167:0/1999350801 learned_addr learned my addr 172.21.15.167:0/1999350801 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:03.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb85659c700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 0x7fb858107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:03.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb855d9b700 1 -- 172.21.15.167:0/1999350801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 msgr2=0x7fb858107130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:03.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb855d9b700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 0x7fb858107130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:03.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb855d9b700 1 -- 172.21.15.167:0/1999350801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c005040 con 0x7fb8580f9a50 2022-01-31T19:41:03.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb85659c700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 0x7fb858107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-01-31T19:41:03.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb855d9b700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb858107620 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb84c009820 tx=0x7fb84c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:03.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb84c019070 con 0x7fb8580f9a50 2022-01-31T19:41:04.000 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb84c007e20 con 0x7fb8580f9a50 2022-01-31T19:41:04.000 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb858108650 con 0x7fb8580f9a50 2022-01-31T19:41:04.000 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb84c01ecb0 con 0x7fb8580f9a50 2022-01-31T19:41:04.001 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.992+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb858108bb0 con 0x7fb8580f9a50 2022-01-31T19:41:04.002 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.993+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb858045bb0 con 0x7fb8580f9a50 2022-01-31T19:41:04.002 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.994+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fb84c01e520 con 0x7fb8580f9a50 2022-01-31T19:41:04.003 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.994+0000 7fb846ffd700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb84005e5c0 0x7fb840060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:04.003 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.994+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fb84c07a070 con 0x7fb8580f9a50 2022-01-31T19:41:04.003 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.994+0000 7fb85659c700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb84005e5c0 0x7fb840060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:04.003 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.995+0000 7fb85659c700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb84005e5c0 0x7fb840060a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb848007fc0 tx=0x7fb84800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:04.004 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:03.998+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb84c04c0a0 con 0x7fb8580f9a50 2022-01-31T19:41:04.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:03 smithi171 conmon[35325]: cluster 2022-01-31T19:41:02.715990+0000 mgr.smithi167.aciqpk (mgr.14156) 152 : cluster 2022-01-31T19:41:04.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:03 smithi171 conmon[35325]: [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:04.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.149+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb858027ae0 con 0x7fb8580f9a50 2022-01-31T19:41:04.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.150+0000 7fb846ffd700 1 -- 172.21.15.167:0/1999350801 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v35) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb84c04f6c0 con 0x7fb8580f9a50 2022-01-31T19:41:04.152 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 1 2022-01-31T19:41:04.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb84005e5c0 msgr2=0x7fb840060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:04.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb84005e5c0 0x7fb840060a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb848007fc0 tx=0x7fb84800b040).stop 2022-01-31T19:41:04.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 msgr2=0x7fb858107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:04.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb858107620 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb84c009820 tx=0x7fb84c00b040).stop 2022-01-31T19:41:04.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 shutdown_connections 2022-01-31T19:41:04.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb84005e5c0 0x7fb840060a80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:04.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb858060950 0x7fb858107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:04.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 --2- 172.21.15.167:0/1999350801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8580f9a50 0x7fb858107620 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:04.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 >> 172.21.15.167:0/1999350801 conn(0x7fb8580f0f50 msgr2=0x7fb8580f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:04.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 shutdown_connections 2022-01-31T19:41:04.158 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:04.151+0000 7fb85cab5700 1 -- 172.21.15.167:0/1999350801 wait complete. 2022-01-31T19:41:04.519 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-01-31T19:41:04.850 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-01-31T19:41:04.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph mgr dump --format=json 2022-01-31T19:41:04.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:04 smithi167 conmon[32206]: audit 2022-01-31T19:41:04.150335+0000 mon.smithi167 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.167:0/1999350801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-01-31T19:41:05.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:04 smithi171 conmon[35325]: audit 2022-01-31T19:41:04.150335+0000 mon.smithi167 (mon.0) 469 2022-01-31T19:41:05.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:04 smithi171 conmon[35325]: : audit [DBG] from='client.? 172.21.15.167:0/1999350801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-01-31T19:41:05.294 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:05.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:05 smithi167 conmon[32206]: cluster 2022-01-31T19:41:04.716538+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:41:05.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:05 smithi167 conmon[32206]: .14156) 153 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:05 smithi171 conmon[35325]: cluster 2022-01-31T19:41:04.716538+0000 mgr.smithi167.aciqpk (mgr.14156) 153 2022-01-31T19:41:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:05 smithi171 conmon[35325]: : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:06.429 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 -- 172.21.15.167:0/217008729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 msgr2=0x7f7f180f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:06.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/217008729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f180f93c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f7f0c004660 tx=0x7f7f0c009b30).stop 2022-01-31T19:41:06.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 -- 172.21.15.167:0/217008729 shutdown_connections 2022-01-31T19:41:06.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/217008729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f180f98b0 0x7f7f180fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:06.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/217008729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f180f93c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:06.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 -- 172.21.15.167:0/217008729 >> 172.21.15.167:0/217008729 conn(0x7f7f180f0dc0 msgr2=0x7f7f180f31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:06.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 -- 172.21.15.167:0/217008729 shutdown_connections 2022-01-31T19:41:06.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.424+0000 7f7f1d99f700 1 -- 172.21.15.167:0/217008729 wait complete. 2022-01-31T19:41:06.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1d99f700 1 Processor -- start 2022-01-31T19:41:06.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1d99f700 1 -- start start 2022-01-31T19:41:06.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1d99f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f18106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:06.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1d99f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f180f98b0 0x7f7f18107490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:06.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1d99f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18107a40 con 0x7f7f180608b0 2022-01-31T19:41:06.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1d99f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18107b80 con 0x7f7f180f98b0 2022-01-31T19:41:06.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.425+0000 7f7f1c99d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f18106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:06.434 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1c99d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f18106fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32902/0 (socket says 172.21.15.167:32902) 2022-01-31T19:41:06.434 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1c99d700 1 -- 172.21.15.167:0/1658524090 learned_addr learned my addr 172.21.15.167:0/1658524090 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:06.434 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f17fff700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f180f98b0 0x7f7f18107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:06.434 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1c99d700 1 -- 172.21.15.167:0/1658524090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f180f98b0 msgr2=0x7f7f18107490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:06.434 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1c99d700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f180f98b0 0x7f7f18107490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:06.435 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1c99d700 1 -- 172.21.15.167:0/1658524090 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c005040 con 0x7f7f180608b0 2022-01-31T19:41:06.435 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1c99d700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f18106fa0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f7f0c009990 tx=0x7f7f0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:06.435 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0c018070 con 0x7f7f180608b0 2022-01-31T19:41:06.435 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f181084c0 con 0x7f7f180608b0 2022-01-31T19:41:06.436 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7f0c013410 con 0x7f7f180608b0 2022-01-31T19:41:06.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0c01c3c0 con 0x7f7f180608b0 2022-01-31T19:41:06.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.426+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f18108a20 con 0x7f7f180608b0 2022-01-31T19:41:06.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.427+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18045bb0 con 0x7f7f180608b0 2022-01-31T19:41:06.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.428+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f7f0c01d420 con 0x7f7f180608b0 2022-01-31T19:41:06.438 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.428+0000 7f7f157fa700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f7f0805e610 0x7f7f08060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:06.438 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.428+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f7f0c07a250 con 0x7f7f180608b0 2022-01-31T19:41:06.439 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.428+0000 7f7f17fff700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f7f0805e610 0x7f7f08060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:06.439 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.428+0000 7f7f17fff700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f7f0805e610 0x7f7f08060ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7f18060680 tx=0x7f7f0400b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:06.440 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.432+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7f0c04c190 con 0x7f7f180608b0 2022-01-31T19:41:06.623 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.622+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7f18027ae0 con 0x7f7f180608b0 2022-01-31T19:41:06.625 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.625+0000 7f7f157fa700 1 -- 172.21.15.167:0/1658524090 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v14) v1 ==== 74+0+146954 (secure 0 0 0) 0x7f7f0c01d6b0 con 0x7f7f180608b0 2022-01-31T19:41:06.628 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:06.631 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f7f0805e610 msgr2=0x7f7f08060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:06.631 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f7f0805e610 0x7f7f08060ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7f18060680 tx=0x7f7f0400b040).stop 2022-01-31T19:41:06.631 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 msgr2=0x7f7f18106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:06.632 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f18106fa0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f7f0c009990 tx=0x7f7f0c009b30).stop 2022-01-31T19:41:06.632 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 shutdown_connections 2022-01-31T19:41:06.633 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f7f0805e610 0x7f7f08060ad0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:06.633 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f180608b0 0x7f7f18106fa0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:06.633 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 --2- 172.21.15.167:0/1658524090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f180f98b0 0x7f7f18107490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:06.633 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 >> 172.21.15.167:0/1658524090 conn(0x7f7f180f0dc0 msgr2=0x7f7f180f31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:06.634 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 shutdown_connections 2022-01-31T19:41:06.634 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:06.627+0000 7f7f1d99f700 1 -- 172.21.15.167:0/1658524090 wait complete. 2022-01-31T19:41:06.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:06 smithi167 conmon[32206]: audit 2022-01-31T19:41:06.623144+0000 mon.smithi167 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.167:0/1658524090' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-01-31T19:41:07.003 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":14,"active_gid":14156,"active_name":"smithi167.aciqpk","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":3192751116},{"type":"v1","addr":"172.21.15.167:6801","nonce":3192751116}]},"active_addr":"172.21.15.167:6801/3192751116","active_change":"2022-01-31T19:37:58.631708+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14200,"name":"smithi171.asyxnc","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.167:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":149344667}]},{"addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":858743625}]}]}} 2022-01-31T19:41:07.008 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-01-31T19:41:07.008 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-01-31T19:41:07.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd dump --format=json 2022-01-31T19:41:07.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:06 smithi171 conmon[35325]: audit 2022-01-31T19:41:06.623144+0000 mon.smithi167 (mon. 2022-01-31T19:41:07.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:06 smithi171 conmon[35325]: 0) 470 : audit [DBG] from='client.? 172.21.15.167:0/1658524090' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-01-31T19:41:07.485 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:07.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:07 smithi167 conmon[32206]: cluster 2022-01-31T19:41:06.716990+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:41:07.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:07 smithi167 conmon[32206]: .14156) 154 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:08.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:07 smithi171 conmon[35325]: cluster 2022-01-31T19:41:06.716990+0000 mgr.smithi167.aciqpk (mgr.14156) 154 : cluster [DBG] 2022-01-31T19:41:08.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:07 smithi171 conmon[35325]: pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:08.886 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 -- 172.21.15.167:0/288603623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 msgr2=0x7f38c00f7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:08.886 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 --2- 172.21.15.167:0/288603623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c00f7e50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f38ac004660 tx=0x7f38ac009b30).stop 2022-01-31T19:41:08.887 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 -- 172.21.15.167:0/288603623 shutdown_connections 2022-01-31T19:41:08.887 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 --2- 172.21.15.167:0/288603623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c00f7e50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:08.887 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 --2- 172.21.15.167:0/288603623 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 0x7f38c00f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:08.887 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 -- 172.21.15.167:0/288603623 >> 172.21.15.167:0/288603623 conn(0x7f38c00f1eb0 msgr2=0x7f38c00f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:08.888 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 -- 172.21.15.167:0/288603623 shutdown_connections 2022-01-31T19:41:08.888 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.882+0000 7f38c5af9700 1 -- 172.21.15.167:0/288603623 wait complete. 2022-01-31T19:41:08.888 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.883+0000 7f38c5af9700 1 Processor -- start 2022-01-31T19:41:08.888 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.883+0000 7f38c5af9700 1 -- start start 2022-01-31T19:41:08.889 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c5af9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 0x7f38c010d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:08.889 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c5af9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c010da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:08.889 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c5af9700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010e0b0 con 0x7f38c00f79f0 2022-01-31T19:41:08.889 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c5af9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010e960 con 0x7f38c00f6750 2022-01-31T19:41:08.890 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c010da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:08.890 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c010da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32918/0 (socket says 172.21.15.167:32918) 2022-01-31T19:41:08.890 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 -- 172.21.15.167:0/3743323857 learned_addr learned my addr 172.21.15.167:0/3743323857 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:08.890 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c4af7700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 0x7f38c010d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:08.891 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 -- 172.21.15.167:0/3743323857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 msgr2=0x7f38c010d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:08.891 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 0x7f38c010d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:08.891 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 -- 172.21.15.167:0/3743323857 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac005040 con 0x7f38c00f79f0 2022-01-31T19:41:08.891 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c4af7700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 0x7f38c010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:41:08.891 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bffff700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c010da70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f38ac009990 tx=0x7f38ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:08.892 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38ac018070 con 0x7f38c00f79f0 2022-01-31T19:41:08.892 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c010ebe0 con 0x7f38c00f79f0 2022-01-31T19:41:08.892 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f38ac013410 con 0x7f38c00f79f0 2022-01-31T19:41:08.892 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38ac01c620 con 0x7f38c00f79f0 2022-01-31T19:41:08.893 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.884+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c010f140 con 0x7f38c00f79f0 2022-01-31T19:41:08.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.886+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c00591e0 con 0x7f38c00f79f0 2022-01-31T19:41:08.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.886+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f38ac01c8e0 con 0x7f38c00f79f0 2022-01-31T19:41:08.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.886+0000 7f38bd7fa700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f38b005e610 0x7f38b0060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:08.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.886+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f38ac07a510 con 0x7f38c00f79f0 2022-01-31T19:41:08.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.886+0000 7f38c4af7700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f38b005e610 0x7f38b0060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:08.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.886+0000 7f38c4af7700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f38b005e610 0x7f38b0060ad0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f38c00f7850 tx=0x7f38b4008040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:08.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:08.890+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f38ac04c450 con 0x7f38c00f79f0 2022-01-31T19:41:09.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.040+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38c00008d0 con 0x7f38c00f79f0 2022-01-31T19:41:09.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.041+0000 7f38bd7fa700 1 -- 172.21.15.167:0/3743323857 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+10694 (secure 0 0 0) 0x7f38ac04fa70 con 0x7f38c00f79f0 2022-01-31T19:41:09.042 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:09.043 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":35,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","created":"2022-01-31T19:37:06.503417+0000","modified":"2022-01-31T19:40:52.799933+0000","last_up_change":"2022-01-31T19:40:51.787533+0000","last_in_change":"2022-01-31T19:40:36.089120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-01-31T19:40:13.386038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"6b39ea02-b5a4-4a19-bd99-849c0378fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6803","nonce":2333495797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6805","nonce":2333495797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6809","nonce":2333495797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6807","nonce":2333495797}]},"public_addr":"172.21.15.167:6803/2333495797","cluster_addr":"172.21.15.167:6805/2333495797","heartbeat_back_addr":"172.21.15.167:6809/2333495797","heartbeat_front_addr":"172.21.15.167:6807/2333495797","state":["exists","up"]},{"osd":1,"uuid":"214aeaf2-f8c7-4638-96b6-4e6bb4bd802f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6811","nonce":3267421031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6813","nonce":3267421031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6817","nonce":3267421031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6815","nonce":3267421031}]},"public_addr":"172.21.15.167:6811/3267421031","cluster_addr":"172.21.15.167:6813/3267421031","heartbeat_back_addr":"172.21.15.167:6817/3267421031","heartbeat_front_addr":"172.21.15.167:6815/3267421031","state":["exists","up"]},{"osd":2,"uuid":"3536e17c-9f9a-4867-91f4-f84b2b5d53a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6819","nonce":2914232381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6821","nonce":2914232381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6825","nonce":2914232381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6823","nonce":2914232381}]},"public_addr":"172.21.15.167:6819/2914232381","cluster_addr":"172.21.15.167:6821/2914232381","heartbeat_back_addr":"172.21.15.167:6825/2914232381","heartbeat_front_addr":"172.21.15.167:6823/2914232381","state":["exists","up"]},{"osd":3,"uuid":"e924c508-4baf-4ed0-9a32-42a51710ca8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6801","nonce":1700293141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6803","nonce":1700293141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6807","nonce":1700293141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6805","nonce":1700293141}]},"public_addr":"172.21.15.171:6801/1700293141","cluster_addr":"172.21.15.171:6803/1700293141","heartbeat_back_addr":"172.21.15.171:6807/1700293141","heartbeat_front_addr":"172.21.15.171:6805/1700293141","state":["exists","up"]},{"osd":4,"uuid":"c4310782-5c0e-4962-b32e-45e28ac014ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6809","nonce":3746074472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6811","nonce":3746074472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6815","nonce":3746074472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6813","nonce":3746074472}]},"public_addr":"172.21.15.171:6809/3746074472","cluster_addr":"172.21.15.171:6811/3746074472","heartbeat_back_addr":"172.21.15.171:6815/3746074472","heartbeat_front_addr":"172.21.15.171:6813/3746074472","state":["exists","up"]},{"osd":5,"uuid":"2ef7b4c2-8076-4866-b26a-8027fd249d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6817","nonce":1956867870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6819","nonce":1956867870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6823","nonce":1956867870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6821","nonce":1956867870}]},"public_addr":"172.21.15.171:6817/1956867870","cluster_addr":"172.21.15.171:6819/1956867870","heartbeat_back_addr":"172.21.15.171:6823/1956867870","heartbeat_front_addr":"172.21.15.171:6821/1956867870","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:26.422630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:45.101018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:03.040927+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:18.082335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:33.822236+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:50.200473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2480743863":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6801/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2306410502":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6800/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2760109630":"2022-02-01T19:37:34.139403+0000","172.21.15.167:0/694084204":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6800/3497254897":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6801/3497254897":"2022-02-01T19:37:34.139403+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-31T19:41:09.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.042+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f38b005e610 msgr2=0x7f38b0060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:09.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.042+0000 7f38c5af9700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f38b005e610 0x7f38b0060ad0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f38c00f7850 tx=0x7f38b4008040).stop 2022-01-31T19:41:09.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.042+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 msgr2=0x7f38c010da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:09.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.042+0000 7f38c5af9700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c010da70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f38ac009990 tx=0x7f38ac009b30).stop 2022-01-31T19:41:09.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 shutdown_connections 2022-01-31T19:41:09.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f38b005e610 0x7f38b0060ad0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:09.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38c00f6750 0x7f38c010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:09.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 --2- 172.21.15.167:0/3743323857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38c00f79f0 0x7f38c010da70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:09.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 >> 172.21.15.167:0/3743323857 conn(0x7f38c00f1eb0 msgr2=0x7f38c00f9430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:09.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 shutdown_connections 2022-01-31T19:41:09.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:09.043+0000 7f38c5af9700 1 -- 172.21.15.167:0/3743323857 wait complete. 2022-01-31T19:41:09.605 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-01-31T19:41:09.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd dump --format=json 2022-01-31T19:41:10.078 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:10.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:09 smithi171 conmon[35325]: cluster 2022-01-31T19:41:08.717275+0000 mgr.smithi167.aciqpk (mgr.14156) 155 : cluster [DBG] 2022-01-31T19:41:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:09 smithi171 conmon[35325]: pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:09 smithi171 conmon[35325]: audit 2022-01-31T19:41:09.041115+0000 mon.smithi167 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.167:0/3743323857' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:41:10.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:09 smithi167 conmon[32206]: cluster 2022-01-31T19:41:08.717275+0000 mgr.smithi167.aciqpk (mgr.14156) 155 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:10.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:09 smithi167 conmon[32206]: audit 2022-01-31T19:41:09.041115+0000 mon.smithi167 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.167:0/3743323857' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:41:11.436 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.433+0000 7f378759e700 1 -- 172.21.15.167:0/1286975499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f4de0 msgr2=0x7f37880f5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:11.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.433+0000 7f378759e700 1 --2- 172.21.15.167:0/1286975499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f4de0 0x7f37880f5200 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3770004660 tx=0x7f3770009b30).stop 2022-01-31T19:41:11.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.433+0000 7f378759e700 1 -- 172.21.15.167:0/1286975499 shutdown_connections 2022-01-31T19:41:11.437 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.433+0000 7f378759e700 1 --2- 172.21.15.167:0/1286975499 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37880f5fe0 0x7f37880f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:11.438 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 --2- 172.21.15.167:0/1286975499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f4de0 0x7f37880f5200 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:11.438 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 -- 172.21.15.167:0/1286975499 >> 172.21.15.167:0/1286975499 conn(0x7f37880f0540 msgr2=0x7f37880f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:11.438 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 -- 172.21.15.167:0/1286975499 shutdown_connections 2022-01-31T19:41:11.438 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 -- 172.21.15.167:0/1286975499 wait complete. 2022-01-31T19:41:11.439 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 Processor -- start 2022-01-31T19:41:11.439 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 -- start start 2022-01-31T19:41:11.439 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37880f4de0 0x7f3788104db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:11.439 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 0x7f37881052a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:11.440 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37881058a0 con 0x7f37880f5fe0 2022-01-31T19:41:11.440 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f378759e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37881060e0 con 0x7f37880f4de0 2022-01-31T19:41:11.440 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f3785d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 0x7f37881052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:11.441 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f3785d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 0x7f37881052a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32930/0 (socket says 172.21.15.167:32930) 2022-01-31T19:41:11.441 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.434+0000 7f3785d9b700 1 -- 172.21.15.167:0/49296440 learned_addr learned my addr 172.21.15.167:0/49296440 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:11.441 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f378659c700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37880f4de0 0x7f3788104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:11.441 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f3785d9b700 1 -- 172.21.15.167:0/49296440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37880f4de0 msgr2=0x7f3788104db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:11.442 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f3785d9b700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37880f4de0 0x7f3788104db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:11.442 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f3785d9b700 1 -- 172.21.15.167:0/49296440 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770005040 con 0x7f37880f5fe0 2022-01-31T19:41:11.442 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f3785d9b700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 0x7f37881052a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3778002770 tx=0x7f377800af10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:11.443 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3778008ad0 con 0x7f37880f5fe0 2022-01-31T19:41:11.443 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3778010040 con 0x7f37880f5fe0 2022-01-31T19:41:11.443 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37881063c0 con 0x7f37880f5fe0 2022-01-31T19:41:11.443 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3778015760 con 0x7f37880f5fe0 2022-01-31T19:41:11.444 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.435+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3788106900 con 0x7f37880f5fe0 2022-01-31T19:41:11.445 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.436+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37880591e0 con 0x7f37880f5fe0 2022-01-31T19:41:11.445 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.437+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f377800a110 con 0x7f37880f5fe0 2022-01-31T19:41:11.445 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.437+0000 7f377effd700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f377405e610 0x7f3774060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:11.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.437+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f3778079a50 con 0x7f37880f5fe0 2022-01-31T19:41:11.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.437+0000 7f378659c700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f377405e610 0x7f3774060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:11.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.437+0000 7f378659c700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f377405e610 0x7f3774060ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3770004630 tx=0x7f3770009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:11.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.441+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f377804b990 con 0x7f37880f5fe0 2022-01-31T19:41:11.594 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.593+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37880fceb0 con 0x7f37880f5fe0 2022-01-31T19:41:11.595 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.594+0000 7f377effd700 1 -- 172.21.15.167:0/49296440 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+10694 (secure 0 0 0) 0x7f37880fceb0 con 0x7f37880f5fe0 2022-01-31T19:41:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:11.596 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":35,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","created":"2022-01-31T19:37:06.503417+0000","modified":"2022-01-31T19:40:52.799933+0000","last_up_change":"2022-01-31T19:40:51.787533+0000","last_in_change":"2022-01-31T19:40:36.089120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-01-31T19:40:13.386038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"6b39ea02-b5a4-4a19-bd99-849c0378fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6803","nonce":2333495797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6805","nonce":2333495797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6809","nonce":2333495797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6807","nonce":2333495797}]},"public_addr":"172.21.15.167:6803/2333495797","cluster_addr":"172.21.15.167:6805/2333495797","heartbeat_back_addr":"172.21.15.167:6809/2333495797","heartbeat_front_addr":"172.21.15.167:6807/2333495797","state":["exists","up"]},{"osd":1,"uuid":"214aeaf2-f8c7-4638-96b6-4e6bb4bd802f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6811","nonce":3267421031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6813","nonce":3267421031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6817","nonce":3267421031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6815","nonce":3267421031}]},"public_addr":"172.21.15.167:6811/3267421031","cluster_addr":"172.21.15.167:6813/3267421031","heartbeat_back_addr":"172.21.15.167:6817/3267421031","heartbeat_front_addr":"172.21.15.167:6815/3267421031","state":["exists","up"]},{"osd":2,"uuid":"3536e17c-9f9a-4867-91f4-f84b2b5d53a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6819","nonce":2914232381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6821","nonce":2914232381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6825","nonce":2914232381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6823","nonce":2914232381}]},"public_addr":"172.21.15.167:6819/2914232381","cluster_addr":"172.21.15.167:6821/2914232381","heartbeat_back_addr":"172.21.15.167:6825/2914232381","heartbeat_front_addr":"172.21.15.167:6823/2914232381","state":["exists","up"]},{"osd":3,"uuid":"e924c508-4baf-4ed0-9a32-42a51710ca8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6801","nonce":1700293141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6803","nonce":1700293141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6807","nonce":1700293141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6805","nonce":1700293141}]},"public_addr":"172.21.15.171:6801/1700293141","cluster_addr":"172.21.15.171:6803/1700293141","heartbeat_back_addr":"172.21.15.171:6807/1700293141","heartbeat_front_addr":"172.21.15.171:6805/1700293141","state":["exists","up"]},{"osd":4,"uuid":"c4310782-5c0e-4962-b32e-45e28ac014ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6809","nonce":3746074472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6811","nonce":3746074472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6815","nonce":3746074472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6813","nonce":3746074472}]},"public_addr":"172.21.15.171:6809/3746074472","cluster_addr":"172.21.15.171:6811/3746074472","heartbeat_back_addr":"172.21.15.171:6815/3746074472","heartbeat_front_addr":"172.21.15.171:6813/3746074472","state":["exists","up"]},{"osd":5,"uuid":"2ef7b4c2-8076-4866-b26a-8027fd249d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6817","nonce":1956867870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6819","nonce":1956867870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6823","nonce":1956867870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6821","nonce":1956867870}]},"public_addr":"172.21.15.171:6817/1956867870","cluster_addr":"172.21.15.171:6819/1956867870","heartbeat_back_addr":"172.21.15.171:6823/1956867870","heartbeat_front_addr":"172.21.15.171:6821/1956867870","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:26.422630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:45.101018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:03.040927+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:18.082335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:33.822236+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:50.200473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2480743863":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6801/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2306410502":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6800/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2760109630":"2022-02-01T19:37:34.139403+0000","172.21.15.167:0/694084204":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6800/3497254897":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6801/3497254897":"2022-02-01T19:37:34.139403+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-31T19:41:11.597 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f377405e610 msgr2=0x7f3774060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:11.597 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f377405e610 0x7f3774060ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3770004630 tx=0x7f3770009b30).stop 2022-01-31T19:41:11.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 msgr2=0x7f37881052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:11.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 0x7f37881052a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3778002770 tx=0x7f377800af10).stop 2022-01-31T19:41:11.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 shutdown_connections 2022-01-31T19:41:11.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f377405e610 0x7f3774060ad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:11.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37880f4de0 0x7f3788104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:11.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 --2- 172.21.15.167:0/49296440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37880f5fe0 0x7f37881052a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:11.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 >> 172.21.15.167:0/49296440 conn(0x7f37880f0540 msgr2=0x7f37880f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:11.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 shutdown_connections 2022-01-31T19:41:11.600 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:11.596+0000 7f378759e700 1 -- 172.21.15.167:0/49296440 wait complete. 2022-01-31T19:41:11.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:11 smithi167 conmon[32206]: cluster 2022-01-31T19:41:10.717715+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:41:11.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:11 smithi167 conmon[32206]: 156 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:11.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:11 smithi167 conmon[32206]: audit 2022-01-31T19:41:11.594564+0000 mon.smithi167 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.167:0/49296440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:41:11.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-01-31T19:41:11.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-01-31T19:41:11.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-01-31T19:41:11.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-01-31T19:41:11.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-01-31T19:41:11.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-01-31T19:41:12.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:11 smithi171 conmon[35325]: cluster 2022-01-31T19:41:10.717715+0000 mgr.smithi167.aciqpk (mgr.14156) 156 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:11 smithi171 conmon[35325]: 2022-01-31T19:41:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:11 smithi171 conmon[35325]: audit 2022-01-31T19:41:11.594564+0000 mon.smithi167 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.167:0/49296440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:41:12.478 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:12.500 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:12.745 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:12.747 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:12.780 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:12.967 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:14.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:13 smithi167 conmon[32206]: cluster 2022-01-31T19:41:12.718004+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:41:14.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:13 smithi167 conmon[32206]: ) 157 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:13 smithi171 conmon[35325]: cluster 2022-01-31T19:41:12.718004+0000 mgr.smithi167.aciqpk (mgr.14156) 157 : cluster [DBG] 2022-01-31T19:41:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:13 smithi171 conmon[35325]: pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:15.174 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- 172.21.15.167:0/2836140645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f57f0 msgr2=0x7f0f180f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:15.175 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 --2- 172.21.15.167:0/2836140645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f57f0 0x7f0f180f5c10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f0f14004660 tx=0x7f0f14009b30).stop 2022-01-31T19:41:15.175 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- 172.21.15.167:0/2836140645 shutdown_connections 2022-01-31T19:41:15.175 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 --2- 172.21.15.167:0/2836140645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f180f6a90 0x7f0f180f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.175 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 --2- 172.21.15.167:0/2836140645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f57f0 0x7f0f180f5c10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.176 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- 172.21.15.167:0/2836140645 >> 172.21.15.167:0/2836140645 conn(0x7f0f180f0f50 msgr2=0x7f0f180f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:15.176 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- 172.21.15.167:0/2836140645 shutdown_connections 2022-01-31T19:41:15.176 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- 172.21.15.167:0/2836140645 wait complete. 2022-01-31T19:41:15.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 Processor -- start 2022-01-31T19:41:15.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- start start 2022-01-31T19:41:15.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f180f57f0 0x7f0f181093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:15.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 0x7f0f181098e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:15.177 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18109e90 con 0x7f0f180f6a90 2022-01-31T19:41:15.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.814+0000 7f0f20117700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18109fd0 con 0x7f0f180f57f0 2022-01-31T19:41:15.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 0x7f0f181098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:15.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 0x7f0f181098e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32954/0 (socket says 172.21.15.167:32954) 2022-01-31T19:41:15.178 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 -- 172.21.15.167:0/691537890 learned_addr learned my addr 172.21.15.167:0/691537890 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:15.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1deb3700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f180f57f0 0x7f0f181093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:15.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 -- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f180f57f0 msgr2=0x7f0f181093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:15.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f180f57f0 0x7f0f181093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 -- 172.21.15.167:0/691537890 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14005040 con 0x7f0f180f6a90 2022-01-31T19:41:15.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f1d6b2700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 0x7f0f181098e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0f080026d0 tx=0x7f0f08009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:15.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f080079c0 con 0x7f0f180f6a90 2022-01-31T19:41:15.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1810a970 con 0x7f0f180f6a90 2022-01-31T19:41:15.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0f0800f040 con 0x7f0f180f6a90 2022-01-31T19:41:15.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f08014820 con 0x7f0f180f6a90 2022-01-31T19:41:15.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.815+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1810af30 con 0x7f0f180f6a90 2022-01-31T19:41:15.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.816+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f1805dda0 con 0x7f0f180f6a90 2022-01-31T19:41:15.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.816+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f0f080090d0 con 0x7f0f180f6a90 2022-01-31T19:41:15.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f0e7fc700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0f0405e5c0 0x7f0f04060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:15.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f0f08078580 con 0x7f0f180f6a90 2022-01-31T19:41:15.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f1deb3700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0f0405e5c0 0x7f0f04060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:15.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f0e7fc700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] conn(0x7f0f04064210 0x7f0f04066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:15.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 --> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f04066cd0 con 0x7f0f04064210 2022-01-31T19:41:15.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f0807abf0 con 0x7f0f180f6a90 2022-01-31T19:41:15.184 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f1deb3700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0f0405e5c0 0x7f0f04060a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0f14005010 tx=0x7f0f14009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:15.184 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.817+0000 7f0f1e6b4700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] conn(0x7f0f04064210 0x7f0f04066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:15.184 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.818+0000 7f0f1e6b4700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] conn(0x7f0f04064210 0x7f0f04066650 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:15.184 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.818+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== osd.5 v2:172.21.15.171:6816/1956867870 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f0f04066cd0 con 0x7f0f04064210 2022-01-31T19:41:15.185 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.833+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 --> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f1810b3f0 con 0x7f0f04064210 2022-01-31T19:41:15.185 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.833+0000 7f0f0e7fc700 1 -- 172.21.15.167:0/691537890 <== osd.5 v2:172.21.15.171:6816/1956867870 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0f1810b3f0 con 0x7f0f04064210 2022-01-31T19:41:15.185 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] conn(0x7f0f04064210 msgr2=0x7f0f04066650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:15.186 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] conn(0x7f0f04064210 0x7f0f04066650 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.186 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0f0405e5c0 msgr2=0x7f0f04060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:15.186 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0f0405e5c0 0x7f0f04060a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0f14005010 tx=0x7f0f14009b30).stop 2022-01-31T19:41:15.186 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 msgr2=0x7f0f181098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:15.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 0x7f0f181098e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0f080026d0 tx=0x7f0f08009de0).stop 2022-01-31T19:41:15.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 shutdown_connections 2022-01-31T19:41:15.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:6816/1956867870,v1:172.21.15.171:6817/1956867870] conn(0x7f0f04064210 0x7f0f04066650 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.187 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0f0405e5c0 0x7f0f04060a80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.188 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f180f57f0 0x7f0f181093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.188 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 --2- 172.21.15.167:0/691537890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f180f6a90 0x7f0f181098e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:15.188 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 >> 172.21.15.167:0/691537890 conn(0x7f0f180f0f50 msgr2=0x7f0f18060480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:15.188 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 shutdown_connections 2022-01-31T19:41:15.189 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:14.834+0000 7f0f20117700 1 -- 172.21.15.167:0/691537890 wait complete. 2022-01-31T19:41:16.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:15 smithi167 conmon[32206]: cluster 2022-01-31T19:41:14.718372+0000 mgr.smithi167.aciqpk (mgr.14156) 158 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:15 smithi171 conmon[35325]: cluster 2022-01-31T19:41:14.718372+0000 mgr.smithi167.aciqpk (mgr.14156) 158 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:15 smithi171 conmon[35325]: 2022-01-31T19:41:16.483 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 -- 172.21.15.167:0/4167152510 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 msgr2=0x7ff2880f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:16.483 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 --2- 172.21.15.167:0/4167152510 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff2880f7cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7ff278004660 tx=0x7ff278009b30).stop 2022-01-31T19:41:16.484 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 -- 172.21.15.167:0/4167152510 shutdown_connections 2022-01-31T19:41:16.484 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 --2- 172.21.15.167:0/4167152510 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff2880f7cd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:16.484 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 --2- 172.21.15.167:0/4167152510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2880f4bb0 0x7ff2880f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:16.485 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 -- 172.21.15.167:0/4167152510 >> 172.21.15.167:0/4167152510 conn(0x7ff2880f06f0 msgr2=0x7ff2880f2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:16.485 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 -- 172.21.15.167:0/4167152510 shutdown_connections 2022-01-31T19:41:16.485 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.479+0000 7ff28fe52700 1 -- 172.21.15.167:0/4167152510 wait complete. 2022-01-31T19:41:16.485 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28fe52700 1 Processor -- start 2022-01-31T19:41:16.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28fe52700 1 -- start start 2022-01-31T19:41:16.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28fe52700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2880f4bb0 0x7ff288107190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:16.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28fe52700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff288107680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:16.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28fe52700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288107c30 con 0x7ff2880f5550 2022-01-31T19:41:16.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28fe52700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288107d70 con 0x7ff2880f4bb0 2022-01-31T19:41:16.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28d3ed700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff288107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:16.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28d3ed700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff288107680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32970/0 (socket says 172.21.15.167:32970) 2022-01-31T19:41:16.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.480+0000 7ff28d3ed700 1 -- 172.21.15.167:0/1380258355 learned_addr learned my addr 172.21.15.167:0/1380258355 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:16.488 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff28d3ed700 1 -- 172.21.15.167:0/1380258355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2880f4bb0 msgr2=0x7ff288107190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:16.488 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff28d3ed700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2880f4bb0 0x7ff288107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:16.488 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff28d3ed700 1 -- 172.21.15.167:0/1380258355 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278005040 con 0x7ff2880f5550 2022-01-31T19:41:16.488 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff28d3ed700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff288107680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff278007a60 tx=0x7ff278009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:16.488 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff278018070 con 0x7ff2880f5550 2022-01-31T19:41:16.489 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2881086b0 con 0x7ff2880f5550 2022-01-31T19:41:16.489 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff278013410 con 0x7ff2880f5550 2022-01-31T19:41:16.489 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27801c620 con 0x7ff2880f5550 2022-01-31T19:41:16.489 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.481+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288108c10 con 0x7ff2880f5550 2022-01-31T19:41:16.492 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.482+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff28805dda0 con 0x7ff2880f5550 2022-01-31T19:41:16.492 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.482+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7ff27801d9b0 con 0x7ff2880f5550 2022-01-31T19:41:16.492 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff27e7fc700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff27405e5c0 0x7ff274060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:16.493 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7ff27807a450 con 0x7ff2880f5550 2022-01-31T19:41:16.493 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff28dbee700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff27405e5c0 0x7ff274060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:16.493 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff27e7fc700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] conn(0x7ff274064210 0x7ff274066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:16.493 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 --> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff274066cd0 con 0x7ff274064210 2022-01-31T19:41:16.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff27807cac0 con 0x7ff2880f5550 2022-01-31T19:41:16.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff28e3ef700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] conn(0x7ff274064210 0x7ff274066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:16.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.483+0000 7ff28dbee700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff27405e5c0 0x7ff274060a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff28805f9c0 tx=0x7ff2840079e0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:16.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.484+0000 7ff28e3ef700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] conn(0x7ff274064210 0x7ff274066650 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:16.495 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.484+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== osd.1 v2:172.21.15.167:6810/3267421031 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7ff274066cd0 con 0x7ff274064210 2022-01-31T19:41:16.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 --> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2881091b0 con 0x7ff274064210 2022-01-31T19:41:16.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff27e7fc700 1 -- 172.21.15.167:0/1380258355 <== osd.1 v2:172.21.15.167:6810/3267421031 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2881091b0 con 0x7ff274064210 2022-01-31T19:41:16.502 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] conn(0x7ff274064210 msgr2=0x7ff274066650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:16.503 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] conn(0x7ff274064210 0x7ff274066650 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:16.503 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff27405e5c0 msgr2=0x7ff274060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:16.503 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff27405e5c0 0x7ff274060a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff28805f9c0 tx=0x7ff2840079e0).stop 2022-01-31T19:41:16.503 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 msgr2=0x7ff288107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:16.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff288107680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff278007a60 tx=0x7ff278009b30).stop 2022-01-31T19:41:16.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 shutdown_connections 2022-01-31T19:41:16.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6810/3267421031,v1:172.21.15.167:6811/3267421031] conn(0x7ff274064210 0x7ff274066650 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:16.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7ff27405e5c0 0x7ff274060a80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:16.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2880f4bb0 0x7ff288107190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:16.505 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 --2- 172.21.15.167:0/1380258355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2880f5550 0x7ff288107680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:16.505 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 >> 172.21.15.167:0/1380258355 conn(0x7ff2880f06f0 msgr2=0x7ff2880fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:16.505 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 shutdown_connections 2022-01-31T19:41:16.506 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.499+0000 7ff28fe52700 1 -- 172.21.15.167:0/1380258355 wait complete. 2022-01-31T19:41:17.102 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 -- 172.21.15.167:0/538778251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f68e0 msgr2=0x7f980c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.102 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 --2- 172.21.15.167:0/538778251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f68e0 0x7f980c0f6d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f97fc004660 tx=0x7f97fc009b30).stop 2022-01-31T19:41:17.103 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 -- 172.21.15.167:0/538778251 shutdown_connections 2022-01-31T19:41:17.103 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 --2- 172.21.15.167:0/538778251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f68e0 0x7f980c0f6d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.103 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 --2- 172.21.15.167:0/538778251 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f5640 0x7f980c0f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.103 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 -- 172.21.15.167:0/538778251 >> 172.21.15.167:0/538778251 conn(0x7f980c0f0dc0 msgr2=0x7f980c0f31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:17.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 -- 172.21.15.167:0/538778251 shutdown_connections 2022-01-31T19:41:17.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 -- 172.21.15.167:0/538778251 wait complete. 2022-01-31T19:41:17.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 Processor -- start 2022-01-31T19:41:17.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.911+0000 7f98125e6700 1 -- start start 2022-01-31T19:41:17.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98125e6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 0x7f980c1090b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98125e6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f68e0 0x7f980c1095a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98125e6700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c109b50 con 0x7f980c0f5640 2022-01-31T19:41:17.106 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98125e6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c109c90 con 0x7f980c0f68e0 2022-01-31T19:41:17.106 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 0x7f980c1090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.106 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 0x7f980c1090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32980/0 (socket says 172.21.15.167:32980) 2022-01-31T19:41:17.106 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 -- 172.21.15.167:0/2444015895 learned_addr learned my addr 172.21.15.167:0/2444015895 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:17.107 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f9810de3700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f68e0 0x7f980c1095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.107 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 -- 172.21.15.167:0/2444015895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f68e0 msgr2=0x7f980c1095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.107 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f68e0 0x7f980c1095a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.107 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 -- 172.21.15.167:0/2444015895 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc005040 con 0x7f980c0f5640 2022-01-31T19:41:17.108 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f9810de3700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f68e0 0x7f980c1095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-01-31T19:41:17.108 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98115e4700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 0x7f980c1090b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f980800e7c0 tx=0x7f980800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.108 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98080092e0 con 0x7f980c0f5640 2022-01-31T19:41:17.109 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f980800f040 con 0x7f980c0f5640 2022-01-31T19:41:17.109 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c10a630 con 0x7f980c0f5640 2022-01-31T19:41:17.109 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.912+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98080146c0 con 0x7f980c0f5640 2022-01-31T19:41:17.109 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.913+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c10abf0 con 0x7f980c0f5640 2022-01-31T19:41:17.110 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.913+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f980c05dc80 con 0x7f980c0f5640 2022-01-31T19:41:17.110 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f980801b030 con 0x7f980c0f5640 2022-01-31T19:41:17.110 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9801ffb700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f97f805e610 0x7f97f8060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.110 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f9808078680 con 0x7f980c0f5640 2022-01-31T19:41:17.111 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9810de3700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f97f805e610 0x7f97f8060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.111 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9801ffb700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] conn(0x7f97f8064260 0x7f97f80666a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.111 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 --> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97f8066d20 con 0x7f97f8064260 2022-01-31T19:41:17.111 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f980807acf0 con 0x7f980c0f5640 2022-01-31T19:41:17.112 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.914+0000 7f9811de5700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] conn(0x7f97f8064260 0x7f97f80666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.112 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.915+0000 7f9810de3700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f97f805e610 0x7f97f8060ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f97fc004d80 tx=0x7f97fc00b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.112 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.915+0000 7f9811de5700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] conn(0x7f97f8064260 0x7f97f80666a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.112 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.915+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== osd.2 v2:172.21.15.167:6818/2914232381 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f97f8066d20 con 0x7f97f8064260 2022-01-31T19:41:17.113 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.929+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 --> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f980c10b0d0 con 0x7f97f8064260 2022-01-31T19:41:17.113 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.930+0000 7f9801ffb700 1 -- 172.21.15.167:0/2444015895 <== osd.2 v2:172.21.15.167:6818/2914232381 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f980c10b0d0 con 0x7f97f8064260 2022-01-31T19:41:17.113 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] conn(0x7f97f8064260 msgr2=0x7f97f80666a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.114 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] conn(0x7f97f8064260 0x7f97f80666a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.114 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f97f805e610 msgr2=0x7f97f8060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.114 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f97f805e610 0x7f97f8060ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f97fc004d80 tx=0x7f97fc00b040).stop 2022-01-31T19:41:17.115 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 msgr2=0x7f980c1090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.115 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 0x7f980c1090b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f980800e7c0 tx=0x7f980800b040).stop 2022-01-31T19:41:17.115 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 shutdown_connections 2022-01-31T19:41:17.115 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f97f805e610 0x7f97f8060ad0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.116 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:6818/2914232381,v1:172.21.15.167:6819/2914232381] conn(0x7f97f8064260 0x7f97f80666a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.116 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f980c0f5640 0x7f980c1090b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.116 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 --2- 172.21.15.167:0/2444015895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f980c0f68e0 0x7f980c1095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.116 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.931+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 >> 172.21.15.167:0/2444015895 conn(0x7f980c0f0dc0 msgr2=0x7f980c0fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:17.117 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.932+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 shutdown_connections 2022-01-31T19:41:17.117 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:16.932+0000 7f98125e6700 1 -- 172.21.15.167:0/2444015895 wait complete. 2022-01-31T19:41:17.128 INFO:teuthology.orchestra.run.smithi167.stdout:146028888072 2022-01-31T19:41:17.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-01-31T19:41:17.148 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 -- 172.21.15.167:0/3078139166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 msgr2=0x7fa3340f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 --2- 172.21.15.167:0/3078139166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa3340f4fd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa328004660 tx=0x7fa328009b30).stop 2022-01-31T19:41:17.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 -- 172.21.15.167:0/3078139166 shutdown_connections 2022-01-31T19:41:17.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 --2- 172.21.15.167:0/3078139166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3340f5550 0x7fa3340f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 --2- 172.21.15.167:0/3078139166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa3340f4fd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 -- 172.21.15.167:0/3078139166 >> 172.21.15.167:0/3078139166 conn(0x7fa3340f06f0 msgr2=0x7fa3340f2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:17.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 -- 172.21.15.167:0/3078139166 shutdown_connections 2022-01-31T19:41:17.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.112+0000 7fa33af78700 1 -- 172.21.15.167:0/3078139166 wait complete. 2022-01-31T19:41:17.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa33af78700 1 Processor -- start 2022-01-31T19:41:17.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa33af78700 1 -- start start 2022-01-31T19:41:17.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa33af78700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa334107270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa33af78700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3340f5550 0x7fa334107760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa33af78700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334107d10 con 0x7fa3340f4bb0 2022-01-31T19:41:17.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa33af78700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334107e50 con 0x7fa3340f5550 2022-01-31T19:41:17.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.113+0000 7fa338d14700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa334107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa338d14700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa334107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:32992/0 (socket says 172.21.15.167:32992) 2022-01-31T19:41:17.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa338d14700 1 -- 172.21.15.167:0/1381723501 learned_addr learned my addr 172.21.15.167:0/1381723501 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:17.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa333fff700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3340f5550 0x7fa334107760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa338d14700 1 -- 172.21.15.167:0/1381723501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3340f5550 msgr2=0x7fa334107760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa338d14700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3340f5550 0x7fa334107760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa338d14700 1 -- 172.21.15.167:0/1381723501 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa328005040 con 0x7fa3340f4bb0 2022-01-31T19:41:17.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa338d14700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa334107270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa328004e30 tx=0x7fa328009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa328018070 con 0x7fa3340f4bb0 2022-01-31T19:41:17.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa328013410 con 0x7fa3340f4bb0 2022-01-31T19:41:17.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa334108790 con 0x7fa3340f4bb0 2022-01-31T19:41:17.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa32801c3c0 con 0x7fa3340f4bb0 2022-01-31T19:41:17.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.114+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa334108cf0 con 0x7fa3340f4bb0 2022-01-31T19:41:17.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.115+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa33405dda0 con 0x7fa3340f4bb0 2022-01-31T19:41:17.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.115+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fa32801d420 con 0x7fa3340f4bb0 2022-01-31T19:41:17.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa3317fa700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fa32405e5c0 0x7fa324060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fa32807a1a0 con 0x7fa3340f4bb0 2022-01-31T19:41:17.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa333fff700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fa32405e5c0 0x7fa324060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa3317fa700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] conn(0x7fa324064210 0x7fa324066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.158 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 --> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa324066cd0 con 0x7fa324064210 2022-01-31T19:41:17.159 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa32807c810 con 0x7fa3340f4bb0 2022-01-31T19:41:17.159 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa339515700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] conn(0x7fa324064210 0x7fa324066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.160 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.116+0000 7fa333fff700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fa32405e5c0 0x7fa324060a80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa3340f5990 tx=0x7fa32000b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.160 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.117+0000 7fa339515700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] conn(0x7fa324064210 0x7fa324066650 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.160 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.117+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== osd.0 v2:172.21.15.167:6802/2333495797 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fa324066cd0 con 0x7fa324064210 2022-01-31T19:41:17.160 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.135+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 --> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa334109230 con 0x7fa324064210 2022-01-31T19:41:17.161 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.135+0000 7fa3317fa700 1 -- 172.21.15.167:0/1381723501 <== osd.0 v2:172.21.15.167:6802/2333495797 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa334109230 con 0x7fa324064210 2022-01-31T19:41:17.161 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.135+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] conn(0x7fa324064210 msgr2=0x7fa324066650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.161 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.135+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] conn(0x7fa324064210 0x7fa324066650 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.161 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.135+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fa32405e5c0 msgr2=0x7fa324060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.162 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fa32405e5c0 0x7fa324060a80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa3340f5990 tx=0x7fa32000b040).stop 2022-01-31T19:41:17.162 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 msgr2=0x7fa334107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.162 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa334107270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa328004e30 tx=0x7fa328009b30).stop 2022-01-31T19:41:17.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 shutdown_connections 2022-01-31T19:41:17.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6802/2333495797,v1:172.21.15.167:6803/2333495797] conn(0x7fa324064210 0x7fa324066650 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fa32405e5c0 0x7fa324060a80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa3340f4bb0 0x7fa334107270 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 --2- 172.21.15.167:0/1381723501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3340f5550 0x7fa334107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.164 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 >> 172.21.15.167:0/1381723501 conn(0x7fa3340f06f0 msgr2=0x7fa3340fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:17.164 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 shutdown_connections 2022-01-31T19:41:17.164 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.136+0000 7fa33af78700 1 -- 172.21.15.167:0/1381723501 wait complete. 2022-01-31T19:41:17.180 INFO:teuthology.orchestra.run.smithi167.stdout:55834574868 2022-01-31T19:41:17.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-01-31T19:41:17.546 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.543+0000 7f97abd5e700 1 -- 172.21.15.167:0/1113779286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 msgr2=0x7f97a40f6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.546 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.543+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1113779286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a40f6030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f9794004660 tx=0x7f9794009b30).stop 2022-01-31T19:41:17.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- 172.21.15.167:0/1113779286 shutdown_connections 2022-01-31T19:41:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1113779286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a40f6030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1113779286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97a40f6e60 0x7f97a40f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- 172.21.15.167:0/1113779286 >> 172.21.15.167:0/1113779286 conn(0x7f97a40f0760 msgr2=0x7f97a40f2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- 172.21.15.167:0/1113779286 shutdown_connections 2022-01-31T19:41:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- 172.21.15.167:0/1113779286 wait complete. 2022-01-31T19:41:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 Processor -- start 2022-01-31T19:41:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- start start 2022-01-31T19:41:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a410c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97a40f6e60 0x7f97a410e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40f5730 con 0x7f97a40f5bb0 2022-01-31T19:41:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.545+0000 7f97abd5e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40f5870 con 0x7f97a40f6e60 2022-01-31T19:41:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a410c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a410c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33002/0 (socket says 172.21.15.167:33002) 2022-01-31T19:41:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 -- 172.21.15.167:0/1061314743 learned_addr learned my addr 172.21.15.167:0/1061314743 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a92f9700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97a40f6e60 0x7f97a410e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 -- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97a40f6e60 msgr2=0x7f97a410e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97a40f6e60 0x7f97a410e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 -- 172.21.15.167:0/1061314743 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9794005040 con 0x7f97a40f5bb0 2022-01-31T19:41:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f97a9afa700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a410c310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f97a0002700 tx=0x7f97a0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.546+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97a00079c0 con 0x7f97a40f5bb0 2022-01-31T19:41:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.547+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a410ed00 con 0x7f97a40f5bb0 2022-01-31T19:41:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.547+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f97a000f040 con 0x7f97a40f5bb0 2022-01-31T19:41:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.547+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97a0014860 con 0x7f97a40f5bb0 2022-01-31T19:41:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.547+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a410f100 con 0x7f97a40f5bb0 2022-01-31T19:41:17.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.547+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97a405ddc0 con 0x7f97a40f5bb0 2022-01-31T19:41:17.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.548+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f97a001b060 con 0x7f97a40f5bb0 2022-01-31T19:41:17.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.548+0000 7f979a7fc700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f979005e5c0 0x7f9790060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.548+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f97a0078400 con 0x7f97a40f5bb0 2022-01-31T19:41:17.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.548+0000 7f97a92f9700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f979005e5c0 0x7f9790060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.549+0000 7f979a7fc700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] conn(0x7f9790064210 0x7f9790066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:17.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.549+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 --> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9790066cd0 con 0x7f9790064210 2022-01-31T19:41:17.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.549+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97a007aa70 con 0x7f97a40f5bb0 2022-01-31T19:41:17.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.549+0000 7f97aa2fb700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] conn(0x7f9790064210 0x7f9790066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:17.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.549+0000 7f97a92f9700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f979005e5c0 0x7f9790060a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9794005010 tx=0x7f9794009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.549+0000 7f97aa2fb700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] conn(0x7f9790064210 0x7f9790066650 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:17.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.550+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== osd.3 v2:172.21.15.171:6800/1700293141 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f9790066cd0 con 0x7f9790064210 2022-01-31T19:41:17.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.566+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 --> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f97a40582b0 con 0x7f9790064210 2022-01-31T19:41:17.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f979a7fc700 1 -- 172.21.15.167:0/1061314743 <== osd.3 v2:172.21.15.171:6800/1700293141 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f97a40582b0 con 0x7f9790064210 2022-01-31T19:41:17.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] conn(0x7f9790064210 msgr2=0x7f9790066650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] conn(0x7f9790064210 0x7f9790066650 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.570 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f979005e5c0 msgr2=0x7f9790060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f979005e5c0 0x7f9790060a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9794005010 tx=0x7f9794009b30).stop 2022-01-31T19:41:17.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 msgr2=0x7f97a410c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:17.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a410c310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f97a0002700 tx=0x7f97a0009d80).stop 2022-01-31T19:41:17.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 shutdown_connections 2022-01-31T19:41:17.572 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:6800/1700293141,v1:172.21.15.171:6801/1700293141] conn(0x7f9790064210 0x7f9790066650 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.572 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f979005e5c0 0x7f9790060a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.572 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a40f5bb0 0x7f97a410c310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.572 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 --2- 172.21.15.167:0/1061314743 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97a40f6e60 0x7f97a410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:17.573 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 >> 172.21.15.167:0/1061314743 conn(0x7f97a40f0760 msgr2=0x7f97a40fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:17.573 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 shutdown_connections 2022-01-31T19:41:17.573 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:17.567+0000 7f97abd5e700 1 -- 172.21.15.167:0/1061314743 wait complete. 2022-01-31T19:41:17.736 INFO:teuthology.orchestra.run.smithi167.stdout:34359738392 2022-01-31T19:41:17.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-01-31T19:41:17.916 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:17 smithi167 conmon[32206]: cluster 2022-01-31T19:41:16.718745+0000 mgr.smithi167.aciqpk (mgr.14156) 159 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:17.983 INFO:teuthology.orchestra.run.smithi167.stdout:77309411344 2022-01-31T19:41:17.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-01-31T19:41:18.055 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:17 smithi171 conmon[35325]: cluster 2022-01-31T19:41:16.718745+0000 mgr.smithi167.aciqpk (mgr.14156) 159 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:18.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 -- 172.21.15.167:0/3489084268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 msgr2=0x7fd2f80f7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:18.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/3489084268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f80f7210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fd2e8004660 tx=0x7fd2e8009b30).stop 2022-01-31T19:41:18.133 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 -- 172.21.15.167:0/3489084268 shutdown_connections 2022-01-31T19:41:18.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/3489084268 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 0x7fd2f80f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/3489084268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f80f7210 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 -- 172.21.15.167:0/3489084268 >> 172.21.15.167:0/3489084268 conn(0x7fd2f80f06f0 msgr2=0x7fd2f80f2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:18.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 -- 172.21.15.167:0/3489084268 shutdown_connections 2022-01-31T19:41:18.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.124+0000 7fd2fe20b700 1 -- 172.21.15.167:0/3489084268 wait complete. 2022-01-31T19:41:18.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.125+0000 7fd2fe20b700 1 Processor -- start 2022-01-31T19:41:18.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.125+0000 7fd2fe20b700 1 -- start start 2022-01-31T19:41:18.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.125+0000 7fd2fe20b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 0x7fd2f8104f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:18.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2fe20b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f8105480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:18.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2fe20b700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f8105a30 con 0x7fd2f80f6df0 2022-01-31T19:41:18.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2fe20b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f8105b70 con 0x7fd2f80f5b40 2022-01-31T19:41:18.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f8105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:18.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f8105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33014/0 (socket says 172.21.15.167:33014) 2022-01-31T19:41:18.137 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 -- 172.21.15.167:0/2097127345 learned_addr learned my addr 172.21.15.167:0/2097127345 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:18.137 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f77fe700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 0x7fd2f8104f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:18.137 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 -- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 msgr2=0x7fd2f8104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:18.137 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 0x7fd2f8104f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 -- 172.21.15.167:0/2097127345 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8005040 con 0x7fd2f80f6df0 2022-01-31T19:41:18.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f77fe700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 0x7fd2f8104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-01-31T19:41:18.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2f6ffd700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f8105480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd2ec002700 tx=0x7fd2ec009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:18.138 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2ec009320 con 0x7fd2f80f6df0 2022-01-31T19:41:18.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f8106510 con 0x7fd2f80f6df0 2022-01-31T19:41:18.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd2ec00f040 con 0x7fd2f80f6df0 2022-01-31T19:41:18.139 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2ec014860 con 0x7fd2f80f6df0 2022-01-31T19:41:18.140 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.126+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f8106ad0 con 0x7fd2f80f6df0 2022-01-31T19:41:18.140 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.127+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2f805dda0 con 0x7fd2f80f6df0 2022-01-31T19:41:18.140 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.128+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fd2ec0149c0 con 0x7fd2f80f6df0 2022-01-31T19:41:18.140 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.128+0000 7fd2dffff700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd2e00607d0 0x7fd2e0062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:18.140 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fd2ec0793c0 con 0x7fd2f80f6df0 2022-01-31T19:41:18.141 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2f77fe700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd2e00607d0 0x7fd2e0062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:18.141 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2dffff700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] conn(0x7fd2e0066420 0x7fd2e0068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:18.141 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 --> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2e0068ee0 con 0x7fd2e0066420 2022-01-31T19:41:18.142 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2ec0797c0 con 0x7fd2f80f6df0 2022-01-31T19:41:18.142 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2f7fff700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] conn(0x7fd2e0066420 0x7fd2e0068860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:18.142 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2f77fe700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd2e00607d0 0x7fd2e0062c90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd2e8013620 tx=0x7fd2e8009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:18.142 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.129+0000 7fd2f7fff700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] conn(0x7fd2e0066420 0x7fd2e0068860 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:18.143 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.130+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== osd.4 v2:172.21.15.171:6808/3746074472 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fd2e0068ee0 con 0x7fd2e0066420 2022-01-31T19:41:18.146 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.144+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 --> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2f8106f90 con 0x7fd2e0066420 2022-01-31T19:41:18.147 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2dffff700 1 -- 172.21.15.167:0/2097127345 <== osd.4 v2:172.21.15.171:6808/3746074472 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd2f8106f90 con 0x7fd2e0066420 2022-01-31T19:41:18.148 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] conn(0x7fd2e0066420 msgr2=0x7fd2e0068860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:18.148 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] conn(0x7fd2e0066420 0x7fd2e0068860 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.148 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd2e00607d0 msgr2=0x7fd2e0062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:18.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd2e00607d0 0x7fd2e0062c90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd2e8013620 tx=0x7fd2e8009b30).stop 2022-01-31T19:41:18.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 msgr2=0x7fd2f8105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:18.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f8105480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd2ec002700 tx=0x7fd2ec009d80).stop 2022-01-31T19:41:18.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 shutdown_connections 2022-01-31T19:41:18.149 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:6808/3746074472,v1:172.21.15.171:6809/3746074472] conn(0x7fd2e0066420 0x7fd2e0068860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd2e00607d0 0x7fd2e0062c90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f80f5b40 0x7fd2f8104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 --2- 172.21.15.167:0/2097127345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2f80f6df0 0x7fd2f8105480 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:18.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 >> 172.21.15.167:0/2097127345 conn(0x7fd2f80f06f0 msgr2=0x7fd2f80fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:18.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 shutdown_connections 2022-01-31T19:41:18.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:18.145+0000 7fd2fe20b700 1 -- 172.21.15.167:0/2097127345 wait complete. 2022-01-31T19:41:18.402 INFO:teuthology.orchestra.run.smithi167.stdout:107374182414 2022-01-31T19:41:18.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-01-31T19:41:18.694 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:18.703 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:18.711 INFO:teuthology.orchestra.run.smithi167.stdout:128849018892 2022-01-31T19:41:18.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-01-31T19:41:18.721 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:19.185 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:20.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:19 smithi167 conmon[32206]: cluster 2022-01-31T19:41:18.719115+0000 mgr.smithi167.aciqpk (mgr.14156) 160 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:19 smithi171 conmon[35325]: cluster 2022-01-31T19:41:18.719115+0000 mgr.smithi167.aciqpk (mgr.14156) 160 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:19 smithi171 conmon[35325]: 2022-01-31T19:41:20.271 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:20.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 -- 172.21.15.167:0/2093159423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f6760 msgr2=0x7f85380f6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:20.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 --2- 172.21.15.167:0/2093159423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f6760 0x7f85380f6b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f852c004660 tx=0x7f852c009b30).stop 2022-01-31T19:41:20.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 -- 172.21.15.167:0/2093159423 shutdown_connections 2022-01-31T19:41:20.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 --2- 172.21.15.167:0/2093159423 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85380f7a00 0x7f85380f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:20.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 --2- 172.21.15.167:0/2093159423 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f6760 0x7f85380f6b80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:20.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 -- 172.21.15.167:0/2093159423 >> 172.21.15.167:0/2093159423 conn(0x7f85380f1ee0 msgr2=0x7f85380f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:20.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 -- 172.21.15.167:0/2093159423 shutdown_connections 2022-01-31T19:41:20.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.832+0000 7f853da0c700 1 -- 172.21.15.167:0/2093159423 wait complete. 2022-01-31T19:41:20.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853da0c700 1 Processor -- start 2022-01-31T19:41:20.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853da0c700 1 -- start start 2022-01-31T19:41:20.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853da0c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85380f6760 0x7f8538106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:20.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853da0c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 0x7f85381074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:20.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853da0c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538107a50 con 0x7f85380f7a00 2022-01-31T19:41:20.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853da0c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8538107b90 con 0x7f85380f6760 2022-01-31T19:41:20.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f8537fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 0x7f85381074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:20.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f8537fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 0x7f85381074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33032/0 (socket says 172.21.15.167:33032) 2022-01-31T19:41:20.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f8537fff700 1 -- 172.21.15.167:0/1905687759 learned_addr learned my addr 172.21.15.167:0/1905687759 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:20.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.833+0000 7f853ca0a700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85380f6760 0x7f8538106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:20.970 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f8537fff700 1 -- 172.21.15.167:0/1905687759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85380f6760 msgr2=0x7f8538106fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:20.970 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f8537fff700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85380f6760 0x7f8538106fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:20.970 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f8537fff700 1 -- 172.21.15.167:0/1905687759 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c005040 con 0x7f85380f7a00 2022-01-31T19:41:20.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f8537fff700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 0x7f85381074a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f852400c790 tx=0x7f8524007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:20.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8524002590 con 0x7f85380f7a00 2022-01-31T19:41:20.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8538108530 con 0x7f85380f7a00 2022-01-31T19:41:20.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8524002b80 con 0x7f85380f7a00 2022-01-31T19:41:20.972 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8524013d60 con 0x7f85380f7a00 2022-01-31T19:41:20.972 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.834+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8538108af0 con 0x7f85380f7a00 2022-01-31T19:41:20.972 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.835+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8538045bb0 con 0x7f85380f7a00 2022-01-31T19:41:20.972 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.835+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f852401a060 con 0x7f85380f7a00 2022-01-31T19:41:20.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.836+0000 7f85357fa700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f852805e610 0x7f8528060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:20.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.836+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f85240787f0 con 0x7f85380f7a00 2022-01-31T19:41:20.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.836+0000 7f853ca0a700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f852805e610 0x7f8528060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:20.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.836+0000 7f853ca0a700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f852805e610 0x7f8528060ad0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f852c004630 tx=0x7f852c009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:20.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:20.840+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f852404a730 con 0x7f85380f7a00 2022-01-31T19:41:21.006 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.004+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8538027ae0 con 0x7f85380f7a00 2022-01-31T19:41:21.006 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.004+0000 7f85357fa700 1 -- 172.21.15.167:0/1905687759 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f852404dd50 con 0x7f85380f7a00 2022-01-31T19:41:21.007 INFO:teuthology.orchestra.run.smithi167.stdout:146028888073 2022-01-31T19:41:21.008 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f852805e610 msgr2=0x7f8528060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:21.008 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f852805e610 0x7f8528060ad0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f852c004630 tx=0x7f852c009b30).stop 2022-01-31T19:41:21.008 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 msgr2=0x7f85381074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:21.009 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 0x7f85381074a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f852400c790 tx=0x7f8524007d70).stop 2022-01-31T19:41:21.010 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 shutdown_connections 2022-01-31T19:41:21.010 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f852805e610 0x7f8528060ad0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:21.010 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85380f6760 0x7f8538106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:21.010 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 --2- 172.21.15.167:0/1905687759 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85380f7a00 0x7f85381074a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:21.011 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 >> 172.21.15.167:0/1905687759 conn(0x7f85380f1ee0 msgr2=0x7f85380fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:21.011 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 shutdown_connections 2022-01-31T19:41:21.012 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:21.006+0000 7f853da0c700 1 -- 172.21.15.167:0/1905687759 wait complete. 2022-01-31T19:41:21.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:21 smithi167 conmon[32206]: cluster 2022-01-31T19:41:20.719457+0000 mgr.smithi167.aciqpk (mgr.14156) 161 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:21.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:21 smithi167 conmon[32206]: audit 2022-01-31T19:41:21.005832+0000 mon.smithi167 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.167:0/1905687759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-01-31T19:41:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:21 smithi171 conmon[35325]: cluster 2022-01-31T19:41:20.719457+0000 mgr.smithi167.aciqpk (mgr.14156) 161 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:21 smithi171 conmon[35325]: audit 2022-01-31T19:41:21.005832+0000 mon.smithi167 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.167:0/1905687759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-01-31T19:41:22.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888072 got 146028888073 for osd.5 2022-01-31T19:41:22.485 DEBUG:teuthology.parallel:result is None 2022-01-31T19:41:22.585 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 -- 172.21.15.167:0/2660637601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 msgr2=0x7fbe580f7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.586 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/2660637601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe580f7e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbe4c004660 tx=0x7fbe4c009b30).stop 2022-01-31T19:41:22.586 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 -- 172.21.15.167:0/2660637601 shutdown_connections 2022-01-31T19:41:22.586 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/2660637601 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe580f7e50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.586 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/2660637601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe580f6750 0x7fbe580f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 -- 172.21.15.167:0/2660637601 >> 172.21.15.167:0/2660637601 conn(0x7fbe580f1eb0 msgr2=0x7fbe580f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 -- 172.21.15.167:0/2660637601 shutdown_connections 2022-01-31T19:41:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 -- 172.21.15.167:0/2660637601 wait complete. 2022-01-31T19:41:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.583+0000 7fbe5d69a700 1 Processor -- start 2022-01-31T19:41:22.588 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe5d69a700 1 -- start start 2022-01-31T19:41:22.590 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe5d69a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe580f6750 0x7fbe5806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.591 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe5d69a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe58069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.591 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe5d69a700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5806a460 con 0x7fbe580f79f0 2022-01-31T19:41:22.591 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe5d69a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5806a5a0 con 0x7fbe580f6750 2022-01-31T19:41:22.591 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe58069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.592 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe58069f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33046/0 (socket says 172.21.15.167:33046) 2022-01-31T19:41:22.592 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 -- 172.21.15.167:0/3035995188 learned_addr learned my addr 172.21.15.167:0/3035995188 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:22.592 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe57fff700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe580f6750 0x7fbe5806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.592 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 -- 172.21.15.167:0/3035995188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe580f6750 msgr2=0x7fbe5806ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe580f6750 0x7fbe5806ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 -- 172.21.15.167:0/3035995188 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c005040 con 0x7fbe580f79f0 2022-01-31T19:41:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.584+0000 7fbe577fe700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe58069f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbe4c013fd0 tx=0x7fbe4c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.585+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4c018070 con 0x7fbe580f79f0 2022-01-31T19:41:22.594 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.585+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe4c013410 con 0x7fbe580f79f0 2022-01-31T19:41:22.594 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.585+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4c01c620 con 0x7fbe580f79f0 2022-01-31T19:41:22.594 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.585+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5806a820 con 0x7fbe580f79f0 2022-01-31T19:41:22.594 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.585+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5806ad80 con 0x7fbe580f79f0 2022-01-31T19:41:22.594 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.586+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe58045bb0 con 0x7fbe580f79f0 2022-01-31T19:41:22.595 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.586+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fbe4c01c8e0 con 0x7fbe580f79f0 2022-01-31T19:41:22.595 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.587+0000 7fbe54ff9700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbe4005e610 0x7fbe40060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.595 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.587+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fbe4c07a4e0 con 0x7fbe580f79f0 2022-01-31T19:41:22.595 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.587+0000 7fbe57fff700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbe4005e610 0x7fbe40060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.596 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.587+0000 7fbe57fff700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbe4005e610 0x7fbe40060ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbe5806b810 tx=0x7fbe48009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:22.596 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.591+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbe4c04c420 con 0x7fbe580f79f0 2022-01-31T19:41:22.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 -- 172.21.15.167:0/2465877587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 msgr2=0x7fdf640f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 --2- 172.21.15.167:0/2465877587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf640f70b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fdf54004660 tx=0x7fdf54009b30).stop 2022-01-31T19:41:22.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 -- 172.21.15.167:0/2465877587 shutdown_connections 2022-01-31T19:41:22.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 --2- 172.21.15.167:0/2465877587 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf640f7630 0x7fdf640f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 --2- 172.21.15.167:0/2465877587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf640f70b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 -- 172.21.15.167:0/2465877587 >> 172.21.15.167:0/2465877587 conn(0x7fdf640f0580 msgr2=0x7fdf640f29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:22.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 -- 172.21.15.167:0/2465877587 shutdown_connections 2022-01-31T19:41:22.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.662+0000 7fdf69506700 1 -- 172.21.15.167:0/2465877587 wait complete. 2022-01-31T19:41:22.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.663+0000 7fdf69506700 1 Processor -- start 2022-01-31T19:41:22.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.663+0000 7fdf69506700 1 -- start start 2022-01-31T19:41:22.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.663+0000 7fdf69506700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf64105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.663+0000 7fdf69506700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf640f7630 0x7fdf64106390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.663+0000 7fdf69506700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64106940 con 0x7fdf640f6c90 2022-01-31T19:41:22.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.663+0000 7fdf69506700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64106a80 con 0x7fdf640f7630 2022-01-31T19:41:22.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf64105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.670 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf64105ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33054/0 (socket says 172.21.15.167:33054) 2022-01-31T19:41:22.670 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 -- 172.21.15.167:0/3210281011 learned_addr learned my addr 172.21.15.167:0/3210281011 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:22.670 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf627fc700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf640f7630 0x7fdf64106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.670 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 -- 172.21.15.167:0/3210281011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf640f7630 msgr2=0x7fdf64106390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.671 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf640f7630 0x7fdf64106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.671 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 -- 172.21.15.167:0/3210281011 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54005040 con 0x7fdf640f6c90 2022-01-31T19:41:22.671 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf62ffd700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf64105ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fdf54004e30 tx=0x7fdf54009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:22.672 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf54018070 con 0x7fdf640f6c90 2022-01-31T19:41:22.672 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf54007ba0 con 0x7fdf640f6c90 2022-01-31T19:41:22.672 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf641073c0 con 0x7fdf640f6c90 2022-01-31T19:41:22.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf5401de80 con 0x7fdf640f6c90 2022-01-31T19:41:22.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.664+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf64107920 con 0x7fdf640f6c90 2022-01-31T19:41:22.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.666+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf64045bb0 con 0x7fdf640f6c90 2022-01-31T19:41:22.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.666+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fdf54007d10 con 0x7fdf640f6c90 2022-01-31T19:41:22.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.666+0000 7fdf4b7fe700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdf4c060820 0x7fdf4c062ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.674 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.666+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fdf5407a120 con 0x7fdf640f6c90 2022-01-31T19:41:22.674 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.666+0000 7fdf627fc700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdf4c060820 0x7fdf4c062ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.675 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.667+0000 7fdf627fc700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdf4c060820 0x7fdf4c062ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdf58009fd0 tx=0x7fdf5800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:22.676 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.674+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdf5404c0e0 con 0x7fdf640f6c90 2022-01-31T19:41:22.760 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.757+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbe58027ae0 con 0x7fbe580f79f0 2022-01-31T19:41:22.761 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.758+0000 7fbe54ff9700 1 -- 172.21.15.167:0/3035995188 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbe4c04fa40 con 0x7fbe580f79f0 2022-01-31T19:41:22.761 INFO:teuthology.orchestra.run.smithi167.stdout:107374182415 2022-01-31T19:41:22.762 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.759+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbe4005e610 msgr2=0x7fbe40060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.762 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.759+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbe4005e610 0x7fbe40060ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbe5806b810 tx=0x7fbe48009040).stop 2022-01-31T19:41:22.762 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 msgr2=0x7fbe58069f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe58069f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbe4c013fd0 tx=0x7fbe4c009b30).stop 2022-01-31T19:41:22.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 shutdown_connections 2022-01-31T19:41:22.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbe4005e610 0x7fbe40060ad0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe580f6750 0x7fbe5806ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 --2- 172.21.15.167:0/3035995188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe580f79f0 0x7fbe58069f00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 >> 172.21.15.167:0/3035995188 conn(0x7fbe580f1eb0 msgr2=0x7fbe580fb440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:22.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 shutdown_connections 2022-01-31T19:41:22.765 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.760+0000 7fbe5d69a700 1 -- 172.21.15.167:0/3035995188 wait complete. 2022-01-31T19:41:22.814 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 -- 172.21.15.167:0/1346582222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 msgr2=0x7f61840f5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.814 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 --2- 172.21.15.167:0/1346582222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f61840f5a80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f6174004660 tx=0x7f6174009b30).stop 2022-01-31T19:41:22.815 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 -- 172.21.15.167:0/1346582222 shutdown_connections 2022-01-31T19:41:22.815 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 --2- 172.21.15.167:0/1346582222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61840f6900 0x7f61840f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.815 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 --2- 172.21.15.167:0/1346582222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f61840f5a80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.815 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 -- 172.21.15.167:0/1346582222 >> 172.21.15.167:0/1346582222 conn(0x7f61840f0dc0 msgr2=0x7f61840f31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:22.816 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 -- 172.21.15.167:0/1346582222 shutdown_connections 2022-01-31T19:41:22.816 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.736+0000 7f61890cc700 1 -- 172.21.15.167:0/1346582222 wait complete. 2022-01-31T19:41:22.816 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61890cc700 1 Processor -- start 2022-01-31T19:41:22.817 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61890cc700 1 -- start start 2022-01-31T19:41:22.817 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61890cc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f6184105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.817 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61890cc700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61840f6900 0x7f61841063b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.817 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61890cc700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6184106960 con 0x7f61840f5660 2022-01-31T19:41:22.818 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61890cc700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6184106aa0 con 0x7f61840f6900 2022-01-31T19:41:22.818 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f6183fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f6184105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.818 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f6183fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f6184105ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33064/0 (socket says 172.21.15.167:33064) 2022-01-31T19:41:22.818 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f6183fff700 1 -- 172.21.15.167:0/1288526432 learned_addr learned my addr 172.21.15.167:0/1288526432 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:22.819 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.737+0000 7f61837fe700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61840f6900 0x7f61841063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.819 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6183fff700 1 -- 172.21.15.167:0/1288526432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61840f6900 msgr2=0x7f61841063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.819 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6183fff700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61840f6900 0x7f61841063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.819 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6183fff700 1 -- 172.21.15.167:0/1288526432 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174005040 con 0x7f61840f5660 2022-01-31T19:41:22.820 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6183fff700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f6184105ec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6174002040 tx=0x7f617400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:22.820 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6174019070 con 0x7f61840f5660 2022-01-31T19:41:22.820 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61841073e0 con 0x7f61840f5660 2022-01-31T19:41:22.820 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6184107940 con 0x7f61840f5660 2022-01-31T19:41:22.821 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6174007e50 con 0x7f61840f5660 2022-01-31T19:41:22.821 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.738+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6174014bb0 con 0x7f61840f5660 2022-01-31T19:41:22.821 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.739+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6164004fa0 con 0x7f61840f5660 2022-01-31T19:41:22.821 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.740+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f6174025070 con 0x7f61840f5660 2022-01-31T19:41:22.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.740+0000 7f6180ff9700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f616c05e610 0x7f616c060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:22.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.741+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f6174079920 con 0x7f61840f5660 2022-01-31T19:41:22.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.741+0000 7f61837fe700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f616c05e610 0x7f616c060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:22.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.741+0000 7f61837fe700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f616c05e610 0x7f616c060ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6178009fd0 tx=0x7f617800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:22.823 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.748+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f617404b950 con 0x7f61840f5660 2022-01-31T19:41:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.859+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdf64108070 con 0x7fdf640f6c90 2022-01-31T19:41:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.860+0000 7fdf4b7fe700 1 -- 172.21.15.167:0/3210281011 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf5404f700 con 0x7fdf640f6c90 2022-01-31T19:41:22.862 INFO:teuthology.orchestra.run.smithi167.stdout:77309411345 2022-01-31T19:41:22.863 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.861+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdf4c060820 msgr2=0x7fdf4c062ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.861+0000 7fdf69506700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdf4c060820 0x7fdf4c062ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdf58009fd0 tx=0x7fdf5800b040).stop 2022-01-31T19:41:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.861+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 msgr2=0x7fdf64105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.861+0000 7fdf69506700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf64105ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fdf54004e30 tx=0x7fdf54009b30).stop 2022-01-31T19:41:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 shutdown_connections 2022-01-31T19:41:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fdf4c060820 0x7fdf4c062ce0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640f6c90 0x7fdf64105ea0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 --2- 172.21.15.167:0/3210281011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf640f7630 0x7fdf64106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 >> 172.21.15.167:0/3210281011 conn(0x7fdf640f0580 msgr2=0x7fdf640fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:22.866 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 shutdown_connections 2022-01-31T19:41:22.866 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.862+0000 7fdf69506700 1 -- 172.21.15.167:0/3210281011 wait complete. 2022-01-31T19:41:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.936+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6164005940 con 0x7f61840f5660 2022-01-31T19:41:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.937+0000 7f6180ff9700 1 -- 172.21.15.167:0/1288526432 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6174023070 con 0x7f61840f5660 2022-01-31T19:41:22.939 INFO:teuthology.orchestra.run.smithi167.stdout:34359738393 2022-01-31T19:41:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f616c05e610 msgr2=0x7f616c060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f616c05e610 0x7f616c060ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6178009fd0 tx=0x7f617800b040).stop 2022-01-31T19:41:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 msgr2=0x7f6184105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f6184105ec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6174002040 tx=0x7f617400b040).stop 2022-01-31T19:41:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 shutdown_connections 2022-01-31T19:41:22.942 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f616c05e610 0x7f616c060ad0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.942 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61840f5660 0x7f6184105ec0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.942 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 --2- 172.21.15.167:0/1288526432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61840f6900 0x7f61841063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:22.942 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.938+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 >> 172.21.15.167:0/1288526432 conn(0x7f61840f0dc0 msgr2=0x7f61840fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:22.943 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.939+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 shutdown_connections 2022-01-31T19:41:22.943 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:22.939+0000 7f61890cc700 1 -- 172.21.15.167:0/1288526432 wait complete. 2022-01-31T19:41:23.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:22 smithi171 conmon[35325]: audit 2022-01-31T19:41:22.759591+0000 mon.smithi167 (mon.0) 474 : audit [DBG] from='client.? 172.21.15.167:0/3035995188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-01-31T19:41:23.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:22 smithi167 conmon[32206]: audit 2022-01-31T19:41:22.759591+0000 mon.smithi167 (mon.0) 474 2022-01-31T19:41:23.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:22 smithi167 conmon[32206]: : audit [DBG] from='client.? 172.21.15.167:0/3035995188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-01-31T19:41:23.250 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 -- 172.21.15.167:0/1733273694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 msgr2=0x7f91c00f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 --2- 172.21.15.167:0/1733273694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c00f72c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f91a8004660 tx=0x7f91a8009b30).stop 2022-01-31T19:41:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 -- 172.21.15.167:0/1733273694 shutdown_connections 2022-01-31T19:41:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 --2- 172.21.15.167:0/1733273694 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 0x7f91c00f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 --2- 172.21.15.167:0/1733273694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c00f72c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 -- 172.21.15.167:0/1733273694 >> 172.21.15.167:0/1733273694 conn(0x7f91c00f0f50 msgr2=0x7f91c00f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 -- 172.21.15.167:0/1733273694 shutdown_connections 2022-01-31T19:41:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.245+0000 7f91c5c20700 1 -- 172.21.15.167:0/1733273694 wait complete. 2022-01-31T19:41:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91c5c20700 1 Processor -- start 2022-01-31T19:41:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91c5c20700 1 -- start start 2022-01-31T19:41:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91c5c20700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 0x7f91c0109f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91c5c20700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c010c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91c5c20700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0060ab0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91c5c20700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0060bf0 con 0x7f91c00f54b0 2022-01-31T19:41:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91beffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c010c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:23.254 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91beffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c010c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33076/0 (socket says 172.21.15.167:33076) 2022-01-31T19:41:23.254 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91beffd700 1 -- 172.21.15.167:0/920349043 learned_addr learned my addr 172.21.15.167:0/920349043 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:23.254 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91bf7fe700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 0x7f91c0109f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:23.254 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91beffd700 1 -- 172.21.15.167:0/920349043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 msgr2=0x7f91c0109f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91beffd700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 0x7f91c0109f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91beffd700 1 -- 172.21.15.167:0/920349043 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a8005040 con 0x7f91c00f6ea0 2022-01-31T19:41:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.246+0000 7f91bf7fe700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 0x7f91c0109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:41:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.247+0000 7f91beffd700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c010c480 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f91b000b410 tx=0x7f91b0006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.247+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b00125f0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.247+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c010cae0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.247+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f91b0012be0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.247+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b0002a00 con 0x7f91c00f6ea0 2022-01-31T19:41:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.247+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c010d070 con 0x7f91c00f6ea0 2022-01-31T19:41:23.257 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.248+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c00f4be0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.257 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.248+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f91b000f030 con 0x7f91c00f6ea0 2022-01-31T19:41:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.249+0000 7f91b7fff700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f91ac05e5c0 0x7f91ac060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.249+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f91b0078af0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.249+0000 7f91bf7fe700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f91ac05e5c0 0x7f91ac060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:23.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.249+0000 7f91bf7fe700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f91ac05e5c0 0x7f91ac060a80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f91a8004e30 tx=0x7f91a8009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:23.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.253+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f91b004a9d0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.414+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f91c0027ae0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.414+0000 7f91b7fff700 1 -- 172.21.15.167:0/920349043 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91b004dff0 con 0x7f91c00f6ea0 2022-01-31T19:41:23.417 INFO:teuthology.orchestra.run.smithi167.stdout:55834574870 2022-01-31T19:41:23.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f91ac05e5c0 msgr2=0x7f91ac060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:23.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f91ac05e5c0 0x7f91ac060a80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f91a8004e30 tx=0x7f91a8009b30).stop 2022-01-31T19:41:23.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 msgr2=0x7f91c010c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:23.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c010c480 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f91b000b410 tx=0x7f91b0006ef0).stop 2022-01-31T19:41:23.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 shutdown_connections 2022-01-31T19:41:23.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f91ac05e5c0 0x7f91ac060a80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:23.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c00f54b0 0x7f91c0109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:23.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 --2- 172.21.15.167:0/920349043 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91c00f6ea0 0x7f91c010c480 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:23.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 >> 172.21.15.167:0/920349043 conn(0x7f91c00f0f50 msgr2=0x7f91c00fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:23.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 shutdown_connections 2022-01-31T19:41:23.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:23.416+0000 7f91c5c20700 1 -- 172.21.15.167:0/920349043 wait complete. 2022-01-31T19:41:23.513 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411344 got 77309411345 for osd.2 2022-01-31T19:41:23.513 DEBUG:teuthology.parallel:result is None 2022-01-31T19:41:23.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574868 got 55834574870 for osd.1 2022-01-31T19:41:23.914 DEBUG:teuthology.parallel:result is None 2022-01-31T19:41:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: cluster 2022-01-31T19:41:22.719821+0000 mgr.smithi167.aciqpk (mgr.14156) 162 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: 2022-01-31T19:41:24.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: audit 2022-01-31T19:41:22.861640+0000 mon.smithi167 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.167:0/3210281011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-01-31T19:41:24.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: audit 2022-01-31T19:41:22.938420+0000 2022-01-31T19:41:24.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: mon.smithi167 (mon.0) 476 : audit [DBG] from='client.? 172.21.15.167:0/1288526432' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-01-31T19:41:24.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:41:24.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: -31T19:41:23.415817+0000 mon.smithi167 (mon.0) 477 2022-01-31T19:41:24.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:23 smithi171 conmon[35325]: : audit [DBG] from='client.? 172.21.15.167:0/920349043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-01-31T19:41:24.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:23 smithi167 conmon[32206]: cluster 2022-01-31T19:41:22.719821+0000 2022-01-31T19:41:24.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:23 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 162 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:23 smithi167 conmon[32206]: audit 2022-01-31T19:41:22.861640+0000 mon.smithi167 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.167:0/3210281011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-01-31T19:41:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:23 smithi167 conmon[32206]: audit 2022-01-31T19:41:22.938420+0000 mon.smithi167 (mon.0) 476 : audit [DBG] from='client.? 172.21.15.167:0/1288526432' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-01-31T19:41:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:23 smithi167 conmon[32206]: audit 2022-01-31T19:41:23.415817+0000 mon.smithi167 (mon.0) 477 : audit [DBG] from='client.? 172.21.15.167:0/920349043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-01-31T19:41:24.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182414 got 107374182415 for osd.3 2022-01-31T19:41:24.223 DEBUG:teuthology.parallel:result is None 2022-01-31T19:41:24.372 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 -- 172.21.15.167:0/3790859670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 msgr2=0x7f84d40f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:24.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 --2- 172.21.15.167:0/3790859670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d40f6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f84c8004660 tx=0x7f84c8009b30).stop 2022-01-31T19:41:24.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 -- 172.21.15.167:0/3790859670 shutdown_connections 2022-01-31T19:41:24.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 --2- 172.21.15.167:0/3790859670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d40f6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:24.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 --2- 172.21.15.167:0/3790859670 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 0x7f84d40f7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:24.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 -- 172.21.15.167:0/3790859670 >> 172.21.15.167:0/3790859670 conn(0x7f84d40f0760 msgr2=0x7f84d40f2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:24.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 -- 172.21.15.167:0/3790859670 shutdown_connections 2022-01-31T19:41:24.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.282+0000 7f84daa61700 1 -- 172.21.15.167:0/3790859670 wait complete. 2022-01-31T19:41:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.283+0000 7f84daa61700 1 Processor -- start 2022-01-31T19:41:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.283+0000 7f84daa61700 1 -- start start 2022-01-31T19:41:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84daa61700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d4109e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84daa61700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 0x7f84d410c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84daa61700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40f58e0 con 0x7f84d40f5bf0 2022-01-31T19:41:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84daa61700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40f5a20 con 0x7f84d40f6ee0 2022-01-31T19:41:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d4109e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d4109e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33094/0 (socket says 172.21.15.167:33094) 2022-01-31T19:41:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 -- 172.21.15.167:0/3309206946 learned_addr learned my addr 172.21.15.167:0/3309206946 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d37fe700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 0x7f84d410c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:24.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 -- 172.21.15.167:0/3309206946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 msgr2=0x7f84d410c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:24.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 0x7f84d410c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:24.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 -- 172.21.15.167:0/3309206946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c8005040 con 0x7f84d40f5bf0 2022-01-31T19:41:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d37fe700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 0x7f84d410c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-01-31T19:41:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d3fff700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d4109e40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f84c400c530 tx=0x7f84c4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c400f650 con 0x7f84d40f5bf0 2022-01-31T19:41:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d410c950 con 0x7f84d40f5bf0 2022-01-31T19:41:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f84c400fc40 con 0x7f84d40f5bf0 2022-01-31T19:41:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c4012c40 con 0x7f84d40f5bf0 2022-01-31T19:41:24.381 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.284+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d410cee0 con 0x7f84d40f5bf0 2022-01-31T19:41:24.381 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.285+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d40f43d0 con 0x7f84d40f5bf0 2022-01-31T19:41:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.286+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f84c4010070 con 0x7f84d40f5bf0 2022-01-31T19:41:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.286+0000 7f84d0ff9700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f84bc05e5c0 0x7f84bc060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.286+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f84c4078350 con 0x7f84d40f5bf0 2022-01-31T19:41:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.286+0000 7f84d37fe700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f84bc05e5c0 0x7f84bc060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.287+0000 7f84d37fe700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f84bc05e5c0 0x7f84bc060a80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f84c8004e30 tx=0x7f84c8009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.290+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f84c40456b0 con 0x7f84d40f5bf0 2022-01-31T19:41:24.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.445+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f84d4045bb0 con 0x7f84d40f5bf0 2022-01-31T19:41:24.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.446+0000 7f84d0ff9700 1 -- 172.21.15.167:0/3309206946 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f84c404dcc0 con 0x7f84d40f5bf0 2022-01-31T19:41:24.448 INFO:teuthology.orchestra.run.smithi167.stdout:128849018893 2022-01-31T19:41:24.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f84bc05e5c0 msgr2=0x7f84bc060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:24.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f84bc05e5c0 0x7f84bc060a80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f84c8004e30 tx=0x7f84c8009b30).stop 2022-01-31T19:41:24.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 msgr2=0x7f84d4109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:24.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d4109e40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f84c400c530 tx=0x7f84c4007e20).stop 2022-01-31T19:41:24.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 shutdown_connections 2022-01-31T19:41:24.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f84bc05e5c0 0x7f84bc060a80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:24.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84d40f5bf0 0x7f84d4109e40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:24.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 --2- 172.21.15.167:0/3309206946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84d40f6ee0 0x7f84d410c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:24.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.447+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 >> 172.21.15.167:0/3309206946 conn(0x7f84d40f0760 msgr2=0x7f84d40fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:24.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.448+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 shutdown_connections 2022-01-31T19:41:24.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:24.448+0000 7f84daa61700 1 -- 172.21.15.167:0/3309206946 wait complete. 2022-01-31T19:41:25.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:24 smithi167 conmon[32206]: audit 2022-01-31T19:41:24.447608+0000 mon.smithi167 (mon. 2022-01-31T19:41:25.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:24 smithi167 conmon[32206]: 0) 478 : audit [DBG] from='client.? 172.21.15.167:0/3309206946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-01-31T19:41:25.063 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738392 got 34359738393 for osd.0 2022-01-31T19:41:25.063 DEBUG:teuthology.parallel:result is None 2022-01-31T19:41:25.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:24 smithi171 conmon[35325]: audit 2022-01-31T19:41:24.447608+0000 mon.smithi167 (mon.0) 478 : audit [DBG] from='client.? 172.21.15.167:0/3309206946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-01-31T19:41:25.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:24 smithi171 conmon[35325]: 2022-01-31T19:41:25.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018892 got 128849018893 for osd.4 2022-01-31T19:41:25.218 DEBUG:teuthology.parallel:result is None 2022-01-31T19:41:25.218 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-01-31T19:41:25.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph pg dump --format=json 2022-01-31T19:41:25.665 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:26.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:25 smithi167 conmon[32206]: cluster 2022-01-31T19:41:24.720212+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:41:26.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:25 smithi167 conmon[32206]: 163 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:25 smithi171 conmon[35325]: cluster 2022-01-31T19:41:24.720212+0000 mgr.smithi167.aciqpk (mgr.14156) 163 : 2022-01-31T19:41:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:25 smithi171 conmon[35325]: cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:27.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 -- 172.21.15.167:0/2722337414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 msgr2=0x7f89940f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2722337414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f89940f5c10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8988004660 tx=0x7f8988009b30).stop 2022-01-31T19:41:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 -- 172.21.15.167:0/2722337414 shutdown_connections 2022-01-31T19:41:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2722337414 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89940f6a90 0x7f89940f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2722337414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f89940f5c10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 -- 172.21.15.167:0/2722337414 >> 172.21.15.167:0/2722337414 conn(0x7f89940f0f50 msgr2=0x7f89940f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 -- 172.21.15.167:0/2722337414 shutdown_connections 2022-01-31T19:41:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 -- 172.21.15.167:0/2722337414 wait complete. 2022-01-31T19:41:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.039+0000 7f8999e6d700 1 Processor -- start 2022-01-31T19:41:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f8999e6d700 1 -- start start 2022-01-31T19:41:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f8999e6d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f899410c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f8999e6d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89940f6a90 0x7f899410e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f8999e6d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89940608f0 con 0x7f89940f57f0 2022-01-31T19:41:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f8999e6d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994060a30 con 0x7f89940f6a90 2022-01-31T19:41:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f899410c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f899410c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33112/0 (socket says 172.21.15.167:33112) 2022-01-31T19:41:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 -- 172.21.15.167:0/2685619546 learned_addr learned my addr 172.21.15.167:0/2685619546 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 -- 172.21.15.167:0/2685619546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89940f6a90 msgr2=0x7f899410e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89940f6a90 0x7f899410e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 -- 172.21.15.167:0/2685619546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988005040 con 0x7f89940f57f0 2022-01-31T19:41:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f89937fe700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f899410c300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8988007a60 tx=0x7f8988009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.040+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8988018070 con 0x7f89940f57f0 2022-01-31T19:41:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.041+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8988013410 con 0x7f89940f57f0 2022-01-31T19:41:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.041+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899410ecf0 con 0x7f89940f57f0 2022-01-31T19:41:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.041+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f898801c620 con 0x7f89940f57f0 2022-01-31T19:41:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.041+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899410f110 con 0x7f89940f57f0 2022-01-31T19:41:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.042+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8994045bb0 con 0x7f89940f57f0 2022-01-31T19:41:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.042+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f898801d940 con 0x7f89940f57f0 2022-01-31T19:41:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.042+0000 7f897bfff700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f897c05e5c0 0x7f897c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.042+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f898807a510 con 0x7f89940f57f0 2022-01-31T19:41:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.042+0000 7f8992ffd700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f897c05e5c0 0x7f897c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.043+0000 7f8992ffd700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f897c05e5c0 0x7f897c060a80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f8984009610 tx=0x7f8984006ab0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:27.055 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.046+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f898804c450 con 0x7f89940f57f0 2022-01-31T19:41:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.194+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8994057ff0 con 0x7f897c05e5c0 2022-01-31T19:41:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.196+0000 7f897bfff700 1 -- 172.21.15.167:0/2685619546 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24248 (secure 0 0 0) 0x7f8994057ff0 con 0x7f897c05e5c0 2022-01-31T19:41:27.198 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:27.200 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f897c05e5c0 msgr2=0x7f897c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:27.200 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f897c05e5c0 0x7f897c060a80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f8984009610 tx=0x7f8984006ab0).stop 2022-01-31T19:41:27.200 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 msgr2=0x7f899410c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:27.200 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f899410c300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8988007a60 tx=0x7f8988009b30).stop 2022-01-31T19:41:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 shutdown_connections 2022-01-31T19:41:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f897c05e5c0 0x7f897c060a80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89940f57f0 0x7f899410c300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 --2- 172.21.15.167:0/2685619546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89940f6a90 0x7f899410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 >> 172.21.15.167:0/2685619546 conn(0x7f89940f0f50 msgr2=0x7f89940fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 shutdown_connections 2022-01-31T19:41:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:27.198+0000 7f8999e6d700 1 -- 172.21.15.167:0/2685619546 wait complete. 2022-01-31T19:41:27.203 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2022-01-31T19:41:27.582 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2022-01-31T19:41:26.720366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":30812,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":28992,"kb_avail":562440100,"statfs":{"total":575970213888,"available":575938662400,"internally_reserved":0,"allocated":1814528,"data_stored":670475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29687808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2022-01-31T19:40:51.793186+0000","last_change":"2022-01-31T19:40:35.432015+0000","last_active":"2022-01-31T19:40:51.793186+0000","last_peered":"2022-01-31T19:40:51.793186+0000","last_clean":"2022-01-31T19:40:51.793186+0000","last_became_active":"2022-01-31T19:40:35.431631+0000","last_became_peered":"2022-01-31T19:40:35.431631+0000","last_unstale":"2022-01-31T19:40:51.793186+0000","last_undegraded":"2022-01-31T19:40:51.793186+0000","last_fullsized":"2022-01-31T19:40:51.793186+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-31T19:40:14.120439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-31T19:40:14.120439+0000","last_clean_scrub_stamp":"2022-01-31T19:40:14.120439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":34,"seq":146028888074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5040,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740112,"statfs":{"total":95995035648,"available":95989874688,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5104,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740048,"statfs":{"total":95995035648,"available":95989809152,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182416,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5100,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740052,"statfs":{"total":95995035648,"available":95989813248,"internally_reserved":0,"allocated":299008,"data_stored":111495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.57699999999999996}]},{"osd":1,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 31 19:41:07 2022","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Jan 31 19:41:07 2022","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.311},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.28699999999999998}]},{"osd":3,"last update":"Mon Jan 31 19:41:19 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5232,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 31 19:40:48 2022","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.316,"15min":0.316},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Jan 31 19:41:09 2022","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.438,"5min":0.438,"15min":0.438},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.29499999999999998}]},{"osd":3,"last update":"Mon Jan 31 19:41:20 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 31 19:40:47 2022","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28499999999999998}]},{"osd":2,"last update":"Mon Jan 31 19:41:05 2022","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":1.8879999999999999}]},{"osd":3,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.435},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-01-31T19:41:27.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph pg dump --format=json 2022-01-31T19:41:28.029 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:27 smithi171 conmon[35325]: cluster 2022-01-31T19:41:26.720485+0000 mgr.smithi167.aciqpk (mgr.14156) 164 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:27 smithi171 conmon[35325]: 2022-01-31T19:41:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:27 smithi171 conmon[35325]: audit 2022-01-31T19:41:27.196711+0000 mgr.smithi167.aciqpk (mgr.14156) 165 : audit [DBG] from='client.14396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-01-31T19:41:28.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:27 smithi167 conmon[32206]: cluster 2022-01-31T19:41:26.720485+0000 mgr.smithi167.aciqpk (mgr.14156) 164 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:28.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:27 smithi167 conmon[32206]: audit 2022-01-31T19:41:27.196711+0000 mgr.smithi167.aciqpk (mgr.14156) 165 : audit [DBG] from='client.14396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-01-31T19:41:29.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 -- 172.21.15.167:0/769639783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f6a90 msgr2=0x7f243c0f6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:29.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 --2- 172.21.15.167:0/769639783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f6a90 0x7f243c0f6ef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f242c004660 tx=0x7f242c009b30).stop 2022-01-31T19:41:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 -- 172.21.15.167:0/769639783 shutdown_connections 2022-01-31T19:41:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 --2- 172.21.15.167:0/769639783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f6a90 0x7f243c0f6ef0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 --2- 172.21.15.167:0/769639783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f243c0f57f0 0x7f243c0f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 -- 172.21.15.167:0/769639783 >> 172.21.15.167:0/769639783 conn(0x7f243c0f0f50 msgr2=0x7f243c0f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 -- 172.21.15.167:0/769639783 shutdown_connections 2022-01-31T19:41:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 -- 172.21.15.167:0/769639783 wait complete. 2022-01-31T19:41:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 Processor -- start 2022-01-31T19:41:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.254+0000 7f2443c9f700 1 -- start start 2022-01-31T19:41:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2443c9f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 0x7f243c1060b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2443c9f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f243c0f6a90 0x7f243c1065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2443c9f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c106b50 con 0x7f243c0f57f0 2022-01-31T19:41:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2443c9f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c106c90 con 0x7f243c0f6a90 2022-01-31T19:41:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 0x7f243c1060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 0x7f243c1060b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33126/0 (socket says 172.21.15.167:33126) 2022-01-31T19:41:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 -- 172.21.15.167:0/1351531605 learned_addr learned my addr 172.21.15.167:0/1351531605 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f244123a700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f243c0f6a90 0x7f243c1065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 -- 172.21.15.167:0/1351531605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f243c0f6a90 msgr2=0x7f243c1065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f243c0f6a90 0x7f243c1065a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 -- 172.21.15.167:0/1351531605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c005040 con 0x7f243c0f57f0 2022-01-31T19:41:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2441a3b700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 0x7f243c1060b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f243800c790 tx=0x7f2438007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2438013630 con 0x7f243c0f57f0 2022-01-31T19:41:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243c107630 con 0x7f243c0f57f0 2022-01-31T19:41:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2438012840 con 0x7f243c0f57f0 2022-01-31T19:41:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2438002760 con 0x7f243c0f57f0 2022-01-31T19:41:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.255+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243c107bf0 con 0x7f243c0f57f0 2022-01-31T19:41:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.256+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243c045bb0 con 0x7f243c0f57f0 2022-01-31T19:41:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.257+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f2438010070 con 0x7f243c0f57f0 2022-01-31T19:41:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.257+0000 7f24327fc700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f242805e5c0 0x7f2428060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.257+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f2438078660 con 0x7f243c0f57f0 2022-01-31T19:41:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.257+0000 7f244123a700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f242805e5c0 0x7f2428060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.257+0000 7f244123a700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f242805e5c0 0x7f2428060a80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f242c005010 tx=0x7f242c009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.261+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f243804a540 con 0x7f243c0f57f0 2022-01-31T19:41:29.410 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.408+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f243c0fe0f0 con 0x7f242805e5c0 2022-01-31T19:41:29.411 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.409+0000 7f24327fc700 1 -- 172.21.15.167:0/1351531605 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24221 (secure 0 0 0) 0x7f243c0fe0f0 con 0x7f242805e5c0 2022-01-31T19:41:29.412 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:29.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f242805e5c0 msgr2=0x7f2428060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:29.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f242805e5c0 0x7f2428060a80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f242c005010 tx=0x7f242c009b30).stop 2022-01-31T19:41:29.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 msgr2=0x7f243c1060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:29.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 0x7f243c1060b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f243800c790 tx=0x7f2438007d90).stop 2022-01-31T19:41:29.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 shutdown_connections 2022-01-31T19:41:29.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f242805e5c0 0x7f2428060a80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:29.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f243c0f57f0 0x7f243c1060b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:29.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 --2- 172.21.15.167:0/1351531605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f243c0f6a90 0x7f243c1065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:29.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.411+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 >> 172.21.15.167:0/1351531605 conn(0x7f243c0f0f50 msgr2=0x7f243c0fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:29.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.412+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 shutdown_connections 2022-01-31T19:41:29.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:29.412+0000 7f2443c9f700 1 -- 172.21.15.167:0/1351531605 wait complete. 2022-01-31T19:41:29.416 INFO:teuthology.orchestra.run.smithi167.stderr:dumped all 2022-01-31T19:41:29.747 INFO:teuthology.orchestra.run.smithi167.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2022-01-31T19:41:28.720684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":30812,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":28992,"kb_avail":562440100,"statfs":{"total":575970213888,"available":575938662400,"internally_reserved":0,"allocated":1814528,"data_stored":670475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29687808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2022-01-31T19:40:51.793186+0000","last_change":"2022-01-31T19:40:35.432015+0000","last_active":"2022-01-31T19:40:51.793186+0000","last_peered":"2022-01-31T19:40:51.793186+0000","last_clean":"2022-01-31T19:40:51.793186+0000","last_became_active":"2022-01-31T19:40:35.431631+0000","last_became_peered":"2022-01-31T19:40:35.431631+0000","last_unstale":"2022-01-31T19:40:51.793186+0000","last_undegraded":"2022-01-31T19:40:51.793186+0000","last_fullsized":"2022-01-31T19:40:51.793186+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-31T19:40:14.120439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-31T19:40:14.120439+0000","last_clean_scrub_stamp":"2022-01-31T19:40:14.120439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":34,"seq":146028888075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5040,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740112,"statfs":{"total":95995035648,"available":95989874688,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5104,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740048,"statfs":{"total":95995035648,"available":95989809152,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182416,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5100,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740052,"statfs":{"total":95995035648,"available":95989813248,"internally_reserved":0,"allocated":299008,"data_stored":111495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.57699999999999996}]},{"osd":1,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 31 19:41:07 2022","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Jan 31 19:41:07 2022","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.311},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.28699999999999998}]},{"osd":3,"last update":"Mon Jan 31 19:41:19 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5232,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739920,"statfs":{"total":95995035648,"available":95989678080,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 31 19:40:48 2022","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.316,"15min":0.316},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.317},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Jan 31 19:41:09 2022","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.438,"5min":0.438,"15min":0.438},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Jan 31 19:41:20 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.66100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":111796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 31 19:40:47 2022","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Jan 31 19:41:05 2022","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.252}]},{"osd":3,"last update":"Mon Jan 31 19:41:21 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-01-31T19:41:29.748 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-01-31T19:41:29.748 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-01-31T19:41:29.749 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-01-31T19:41:29.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph health --format=json 2022-01-31T19:41:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:29 smithi171 conmon[35325]: cluster 2022-01-31T19:41:28.720905+0000 mgr.smithi167.aciqpk (mgr.14156) 166 : cluster 2022-01-31T19:41:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:29 smithi171 conmon[35325]: [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:29 smithi171 conmon[35325]: audit 2022-01-31T19:41:29.410431+0000 mgr.smithi167.aciqpk (mgr.14156) 167 : audit [DBG] from='client.14400 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-01-31T19:41:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:29 smithi167 conmon[32206]: cluster 2022-01-31T19:41:28.720905+0000 mgr.smithi167.aciqpk (mgr.14156) 166 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:29 smithi167 conmon[32206]: audit 2022-01-31T19:41:29.410431+0000 mgr.smithi167.aciqpk (mgr.14156) 167 : audit [DBG] from='client.14400 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-01-31T19:41:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:41:31.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 -- 172.21.15.167:0/775284191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 msgr2=0x7f26080f5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:31.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 --2- 172.21.15.167:0/775284191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f26080f5910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f25fc004660 tx=0x7f25fc009b30).stop 2022-01-31T19:41:31.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 -- 172.21.15.167:0/775284191 shutdown_connections 2022-01-31T19:41:31.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 --2- 172.21.15.167:0/775284191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f26080f5910 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:31.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 --2- 172.21.15.167:0/775284191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26080f6ea0 0x7f26080f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:31.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 -- 172.21.15.167:0/775284191 >> 172.21.15.167:0/775284191 conn(0x7f26080f0f50 msgr2=0x7f26080f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:31.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 -- 172.21.15.167:0/775284191 shutdown_connections 2022-01-31T19:41:31.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.551+0000 7f260e098700 1 -- 172.21.15.167:0/775284191 wait complete. 2022-01-31T19:41:31.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f260e098700 1 Processor -- start 2022-01-31T19:41:31.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f260e098700 1 -- start start 2022-01-31T19:41:31.559 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f260e098700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f2608106180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:31.559 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f260e098700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26080f6ea0 0x7f2608106670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:31.560 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f260e098700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608106cb0 con 0x7f26080f54b0 2022-01-31T19:41:31.560 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f260e098700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608107560 con 0x7f26080f6ea0 2022-01-31T19:41:31.560 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f26077fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f2608106180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:31.561 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f26077fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f2608106180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33140/0 (socket says 172.21.15.167:33140) 2022-01-31T19:41:31.561 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f26077fe700 1 -- 172.21.15.167:0/333384100 learned_addr learned my addr 172.21.15.167:0/333384100 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:31.562 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.552+0000 7f2606ffd700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26080f6ea0 0x7f2608106670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:31.562 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f26077fe700 1 -- 172.21.15.167:0/333384100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26080f6ea0 msgr2=0x7f2608106670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:31.563 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f26077fe700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26080f6ea0 0x7f2608106670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:31.563 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f26077fe700 1 -- 172.21.15.167:0/333384100 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc005040 con 0x7f26080f54b0 2022-01-31T19:41:31.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f26077fe700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f2608106180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f25f80093f0 tx=0x7f25f8006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:31.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25f8013640 con 0x7f26080f54b0 2022-01-31T19:41:31.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2608107840 con 0x7f26080f54b0 2022-01-31T19:41:31.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f25f8013c30 con 0x7f26080f54b0 2022-01-31T19:41:31.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25f8012be0 con 0x7f26080f54b0 2022-01-31T19:41:31.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.553+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2608107e00 con 0x7f26080f54b0 2022-01-31T19:41:31.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.554+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26080f4be0 con 0x7f26080f54b0 2022-01-31T19:41:31.566 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.554+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f25f801a060 con 0x7f26080f54b0 2022-01-31T19:41:31.567 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.554+0000 7f25effff700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f25f005e5c0 0x7f25f0060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:31.567 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.555+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f25f8078530 con 0x7f26080f54b0 2022-01-31T19:41:31.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.555+0000 7f2606ffd700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f25f005e5c0 0x7f25f0060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:31.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.555+0000 7f2606ffd700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f25f005e5c0 0x7f25f0060a80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f25fc004e30 tx=0x7f25fc009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:31.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.559+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f25f804a470 con 0x7f26080f54b0 2022-01-31T19:41:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.770+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2608027ae0 con 0x7f26080f54b0 2022-01-31T19:41:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.770+0000 7f25effff700 1 -- 172.21.15.167:0/333384100 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f25f804da90 con 0x7f26080f54b0 2022-01-31T19:41:31.773 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:31.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f25f005e5c0 msgr2=0x7f25f0060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:31.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f25f005e5c0 0x7f25f0060a80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f25fc004e30 tx=0x7f25fc009b30).stop 2022-01-31T19:41:31.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 msgr2=0x7f2608106180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:31.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f2608106180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f25f80093f0 tx=0x7f25f8006ec0).stop 2022-01-31T19:41:31.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 shutdown_connections 2022-01-31T19:41:31.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f25f005e5c0 0x7f25f0060a80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:31.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26080f54b0 0x7f2608106180 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:31.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 --2- 172.21.15.167:0/333384100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26080f6ea0 0x7f2608106670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:31.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 >> 172.21.15.167:0/333384100 conn(0x7f26080f0f50 msgr2=0x7f26080fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:31.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 shutdown_connections 2022-01-31T19:41:31.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:31.772+0000 7f260e098700 1 -- 172.21.15.167:0/333384100 wait complete. 2022-01-31T19:41:31.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:31 smithi167 conmon[32206]: cluster 2022-01-31T19:41:30.721242+0000 mgr.smithi167.aciqpk (mgr.14156) 168 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:31 smithi171 conmon[35325]: cluster 2022-01-31T19:41:30.721242+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:41:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:31 smithi171 conmon[35325]: 168 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:32.212 INFO:teuthology.orchestra.run.smithi167.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-01-31T19:41:32.212 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-01-31T19:41:32.213 INFO:tasks.cephadm:Setup complete, yielding 2022-01-31T19:41:32.213 INFO:teuthology.run_tasks:Running task print... 2022-01-31T19:41:32.227 INFO:teuthology.task.print:**** done end installing pacific cephadm ... 2022-01-31T19:41:32.228 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:41:32.246 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:41:32.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2022-01-31T19:41:32.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:32 smithi167 conmon[32206]: audit 2022-01-31T19:41:31.772126+0000 mon.smithi167 (mon. 2022-01-31T19:41:32.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:32 smithi167 conmon[32206]: 0) 479 : audit [DBG] from='client.? 172.21.15.167:0/333384100' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-01-31T19:41:33.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:32 smithi171 conmon[35325]: audit 2022-01-31T19:41:31.772126+0000 mon.smithi167 (mon.0 2022-01-31T19:41:33.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:32 smithi171 conmon[35325]: ) 479 : audit [DBG] from='client.? 172.21.15.167:0/333384100' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-01-31T19:41:33.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:33 smithi167 conmon[32206]: cluster 2022-01-31T19:41:32.721747+0000 2022-01-31T19:41:33.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:33 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 169 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 -- 172.21.15.167:0/498814610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f6980 msgr2=0x7f82ec0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/498814610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f6980 0x7f82ec0f6de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f82d8004660 tx=0x7f82d8009b30).stop 2022-01-31T19:41:33.970 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 -- 172.21.15.167:0/498814610 shutdown_connections 2022-01-31T19:41:33.970 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/498814610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f6980 0x7f82ec0f6de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:33.970 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/498814610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f56e0 0x7f82ec0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 -- 172.21.15.167:0/498814610 >> 172.21.15.167:0/498814610 conn(0x7f82ec0f0e60 msgr2=0x7f82ec0f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 -- 172.21.15.167:0/498814610 shutdown_connections 2022-01-31T19:41:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 -- 172.21.15.167:0/498814610 wait complete. 2022-01-31T19:41:33.972 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.965+0000 7f82f2d1f700 1 Processor -- start 2022-01-31T19:41:33.972 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f2d1f700 1 -- start start 2022-01-31T19:41:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f2d1f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 0x7f82ec1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f2d1f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f6980 0x7f82ec107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f2d1f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec107bd0 con 0x7f82ec0f56e0 2022-01-31T19:41:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f2d1f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec108480 con 0x7f82ec0f6980 2022-01-31T19:41:33.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f0abb700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 0x7f82ec1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:33.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f0abb700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 0x7f82ec1070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33158/0 (socket says 172.21.15.167:33158) 2022-01-31T19:41:33.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f0abb700 1 -- 172.21.15.167:0/2800730879 learned_addr learned my addr 172.21.15.167:0/2800730879 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82ebfff700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f6980 0x7f82ec107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f0abb700 1 -- 172.21.15.167:0/2800730879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f6980 msgr2=0x7f82ec107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f0abb700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f6980 0x7f82ec107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82f0abb700 1 -- 172.21.15.167:0/2800730879 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8005040 con 0x7f82ec0f56e0 2022-01-31T19:41:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.966+0000 7f82ebfff700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f6980 0x7f82ec107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:41:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.967+0000 7f82f0abb700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 0x7f82ec1070a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f82e00093f0 tx=0x7f82e0006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.967+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e0009ea0 con 0x7f82ec0f56e0 2022-01-31T19:41:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.967+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec108760 con 0x7f82ec0f56e0 2022-01-31T19:41:33.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.967+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f82e0013ad0 con 0x7f82ec0f56e0 2022-01-31T19:41:33.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.967+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e000da80 con 0x7f82ec0f56e0 2022-01-31T19:41:33.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.967+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec108d20 con 0x7f82ec0f56e0 2022-01-31T19:41:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.968+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec0f4ad0 con 0x7f82ec0f56e0 2022-01-31T19:41:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.968+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f82e001a060 con 0x7f82ec0f56e0 2022-01-31T19:41:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.968+0000 7f82e97fa700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f82dc05e5c0 0x7f82dc060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.968+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f82e0078600 con 0x7f82ec0f56e0 2022-01-31T19:41:33.980 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.968+0000 7f82ebfff700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f82dc05e5c0 0x7f82dc060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:33.980 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.969+0000 7f82ebfff700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f82dc05e5c0 0x7f82dc060a80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f82d8004e30 tx=0x7f82d8009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:33.980 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:33.973+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f82e004a4e0 con 0x7f82ec0f56e0 2022-01-31T19:41:34.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:33 smithi171 conmon[35325]: cluster 2022-01-31T19:41:32.721747+0000 mgr.smithi167.aciqpk (mgr.14156) 169 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:34.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:33 smithi171 conmon[35325]: 2022-01-31T19:41:34.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.126+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f82ec045bb0 con 0x7f82ec0f56e0 2022-01-31T19:41:34.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.130+0000 7f82e97fa700 1 -- 172.21.15.167:0/2800730879 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f82e004db00 con 0x7f82ec0f56e0 2022-01-31T19:41:34.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f82dc05e5c0 msgr2=0x7f82dc060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:34.134 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f82dc05e5c0 0x7f82dc060a80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f82d8004e30 tx=0x7f82d8009b30).stop 2022-01-31T19:41:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 msgr2=0x7f82ec1070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 0x7f82ec1070a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f82e00093f0 tx=0x7f82e0006ec0).stop 2022-01-31T19:41:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 shutdown_connections 2022-01-31T19:41:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f82dc05e5c0 0x7f82dc060a80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82ec0f56e0 0x7f82ec1070a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 --2- 172.21.15.167:0/2800730879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82ec0f6980 0x7f82ec107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 >> 172.21.15.167:0/2800730879 conn(0x7f82ec0f0e60 msgr2=0x7f82ec05ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:34.137 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 shutdown_connections 2022-01-31T19:41:34.137 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:34.132+0000 7f82f2d1f700 1 -- 172.21.15.167:0/2800730879 wait complete. 2022-01-31T19:41:34.479 INFO:teuthology.run_tasks:Running task print... 2022-01-31T19:41:34.491 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2022-01-31T19:41:34.492 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:41:34.506 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:41:34.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph orch status' 2022-01-31T19:41:35.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:35 smithi167 conmon[32206]: audit 2022-01-31T19:41:34.131838+0000 mon.smithi167 (mon.0) 480 : audit [INF] from='client.? 172.21.15.167:0/2800730879' entity='client.admin' 2022-01-31T19:41:35.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:35 smithi167 conmon[32206]: audit 2022-01-31T19:41:34.135273+0000 mon.smithi167 (mon.0) 481 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:41:35.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:35 smithi167 conmon[32206]: audit 2022-01-31T19:41:34.136656+0000 mon.smithi167 (mon.0) 482 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:35.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:35 smithi167 conmon[32206]: audit 2022-01-31T19:41:34.137395+0000 mon.smithi167 (mon.0) 483 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:41:35.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:35 smithi167 conmon[32206]: audit 2022-01-31T19:41:34.142139+0000 mon.smithi167 (mon.0) 484 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:35.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: audit 2022-01-31T19:41:34.131838+0000 mon.smithi167 (mon.0) 480 : audit [INF] from='client.? 172.21.15.167:0/2800730879' entity='client.admin' 2022-01-31T19:41:35.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: 2022-01-31T19:41:35.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: audit 2022-01-31T19:41:34.135273+0000 mon.smithi167 (mon.0) 481 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:41:35.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: audit 2022-01-31T19:41:34.136656+0000 mon.smithi167 (mon.0) 482 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:35.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: audit 2022-01-31T19:41:34.137395+0000 mon.smithi167 (mon.0) 483 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:41:35.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: audit 2022-01-31T19:41:34.142139 2022-01-31T19:41:35.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:35 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 484 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:36.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:36 smithi167 conmon[32206]: cluster 2022-01-31T19:41:34.722059 2022-01-31T19:41:36.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:36 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 170 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:36.207 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 -- 172.21.15.167:0/2453368838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f6980 msgr2=0x7fc1500f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:36.207 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 --2- 172.21.15.167:0/2453368838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f6980 0x7fc1500f6de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fc14c004660 tx=0x7fc14c009b30).stop 2022-01-31T19:41:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 -- 172.21.15.167:0/2453368838 shutdown_connections 2022-01-31T19:41:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 --2- 172.21.15.167:0/2453368838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f6980 0x7fc1500f6de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 --2- 172.21.15.167:0/2453368838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1500f56e0 0x7fc1500f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:41:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 -- 172.21.15.167:0/2453368838 >> 172.21.15.167:0/2453368838 conn(0x7fc1500f0e60 msgr2=0x7fc1500f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 -- 172.21.15.167:0/2453368838 shutdown_connections 2022-01-31T19:41:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.203+0000 7fc15870d700 1 -- 172.21.15.167:0/2453368838 wait complete. 2022-01-31T19:41:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc15870d700 1 Processor -- start 2022-01-31T19:41:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc15870d700 1 -- start start 2022-01-31T19:41:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc15870d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 0x7fc150107b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:36.210 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc15870d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1500f6980 0x7fc15010a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:36.210 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc15870d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150060550 con 0x7fc1500f56e0 2022-01-31T19:41:36.210 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc15870d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150060690 con 0x7fc1500f6980 2022-01-31T19:41:36.210 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 0x7fc150107b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 0x7fc150107b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33174/0 (socket says 172.21.15.167:33174) 2022-01-31T19:41:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 -- 172.21.15.167:0/1598138133 learned_addr learned my addr 172.21.15.167:0/1598138133 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc155ca8700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1500f6980 0x7fc15010a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 -- 172.21.15.167:0/1598138133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1500f6980 msgr2=0x7fc15010a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1500f6980 0x7fc15010a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 -- 172.21.15.167:0/1598138133 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c005040 con 0x7fc1500f56e0 2022-01-31T19:41:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc1564a9700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 0x7fc150107b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc1400093f0 tx=0x7fc140006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.204+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc140013640 con 0x7fc1500f56e0 2022-01-31T19:41:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.205+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc140013c30 con 0x7fc1500f56e0 2022-01-31T19:41:36.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.205+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15010a6e0 con 0x7fc1500f56e0 2022-01-31T19:41:36.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.205+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc140012be0 con 0x7fc1500f56e0 2022-01-31T19:41:36.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.205+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15010aca0 con 0x7fc1500f56e0 2022-01-31T19:41:36.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.205+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1500f4ad0 con 0x7fc1500f56e0 2022-01-31T19:41:36.216 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.206+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fc14001a060 con 0x7fc1500f56e0 2022-01-31T19:41:36.216 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.206+0000 7fc146ffd700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc13c05e5c0 0x7fc13c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:36.216 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.206+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fc140078590 con 0x7fc1500f56e0 2022-01-31T19:41:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.206+0000 7fc155ca8700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc13c05e5c0 0x7fc13c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.207+0000 7fc155ca8700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc13c05e5c0 0x7fc13c060a80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc14c005010 tx=0x7fc14c009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.210+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc14004a4d0 con 0x7fc1500f56e0 2022-01-31T19:41:36.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.371+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1500ffed0 con 0x7fc13c05e5c0 2022-01-31T19:41:36.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.372+0000 7fc146ffd700 1 -- 172.21.15.167:0/1598138133 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc1500ffed0 con 0x7fc13c05e5c0 2022-01-31T19:41:36.375 INFO:teuthology.orchestra.run.smithi167.stdout:Backend: cephadm 2022-01-31T19:41:36.375 INFO:teuthology.orchestra.run.smithi167.stdout:Available: Yes 2022-01-31T19:41:36.375 INFO:teuthology.orchestra.run.smithi167.stdout:Paused: No 2022-01-31T19:41:36.376 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc13c05e5c0 msgr2=0x7fc13c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:36.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc13c05e5c0 0x7fc13c060a80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc14c005010 tx=0x7fc14c009b30).stop 2022-01-31T19:41:36.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 msgr2=0x7fc150107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:36.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 0x7fc150107b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc1400093f0 tx=0x7fc140006ec0).stop 2022-01-31T19:41:36.377 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 shutdown_connections 2022-01-31T19:41:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc13c05e5c0 0x7fc13c060a80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1500f56e0 0x7fc150107b80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 --2- 172.21.15.167:0/1598138133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1500f6980 0x7fc15010a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 >> 172.21.15.167:0/1598138133 conn(0x7fc1500f0e60 msgr2=0x7fc1500fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:36.379 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 shutdown_connections 2022-01-31T19:41:36.379 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:36.374+0000 7fc15870d700 1 -- 172.21.15.167:0/1598138133 wait complete. 2022-01-31T19:41:36.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:36 smithi171 conmon[35325]: cluster 2022-01-31T19:41:34.722059+0000 mgr.smithi167.aciqpk (mgr.14156) 170 2022-01-31T19:41:36.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:36 smithi171 conmon[35325]: : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:36.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph orch ps' 2022-01-31T19:41:37.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:37 smithi167 conmon[32206]: audit 2022-01-31T19:41:36.374098+0000 mgr.smithi167.aciqpk (mgr.14156) 171 2022-01-31T19:41:37.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:37 smithi167 conmon[32206]: : audit [DBG] from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:37.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:37 smithi171 conmon[35325]: audit 2022-01-31T19:41:36.374098+0000 mgr.smithi167.aciqpk (mgr.14156) 171 : 2022-01-31T19:41:37.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:37 smithi171 conmon[35325]: audit [DBG] from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:38.399 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 -- 172.21.15.167:0/647440297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 msgr2=0x7f45fc0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:38.400 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 --2- 172.21.15.167:0/647440297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc0f6de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f45ec004660 tx=0x7f45ec009b30).stop 2022-01-31T19:41:38.400 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 -- 172.21.15.167:0/647440297 shutdown_connections 2022-01-31T19:41:38.400 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 --2- 172.21.15.167:0/647440297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc0f6de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:38.400 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 --2- 172.21.15.167:0/647440297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45fc0f56e0 0x7f45fc0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 -- 172.21.15.167:0/647440297 >> 172.21.15.167:0/647440297 conn(0x7f45fc0f0e60 msgr2=0x7f45fc0f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 -- 172.21.15.167:0/647440297 shutdown_connections 2022-01-31T19:41:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.396+0000 7f46009f5700 1 -- 172.21.15.167:0/647440297 wait complete. 2022-01-31T19:41:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f46009f5700 1 Processor -- start 2022-01-31T19:41:38.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f46009f5700 1 -- start start 2022-01-31T19:41:38.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f46009f5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45fc0f56e0 0x7f45fc105f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:38.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f46009f5700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc106470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:38.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f46009f5700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc106a20 con 0x7f45fc0f6980 2022-01-31T19:41:38.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f46009f5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc106b60 con 0x7f45fc0f56e0 2022-01-31T19:41:38.404 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:38 smithi167 conmon[32206]: cluster 2022-01-31T19:41:36.722409+0000 mgr.smithi167.aciqpk (mgr.14156) 172 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:38.408 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f45f9d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc106470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:38.409 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f45f9d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc106470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33192/0 (socket says 172.21.15.167:33192) 2022-01-31T19:41:38.410 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f45f9d9b700 1 -- 172.21.15.167:0/3089598465 learned_addr learned my addr 172.21.15.167:0/3089598465 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:38.410 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.397+0000 7f45fa59c700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45fc0f56e0 0x7f45fc105f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:38.411 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f9d9b700 1 -- 172.21.15.167:0/3089598465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45fc0f56e0 msgr2=0x7f45fc105f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:38.411 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f9d9b700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45fc0f56e0 0x7f45fc105f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:38.412 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f9d9b700 1 -- 172.21.15.167:0/3089598465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec005040 con 0x7f45fc0f6980 2022-01-31T19:41:38.412 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f9d9b700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc106470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f45ec009990 tx=0x7f45ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:38.412 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45ec018070 con 0x7f45fc0f6980 2022-01-31T19:41:38.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc1074a0 con 0x7f45fc0f6980 2022-01-31T19:41:38.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f45ec013410 con 0x7f45fc0f6980 2022-01-31T19:41:38.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc107a00 con 0x7f45fc0f6980 2022-01-31T19:41:38.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.398+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45ec01c3c0 con 0x7f45fc0f6980 2022-01-31T19:41:38.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.399+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e0004fa0 con 0x7f45fc0f6980 2022-01-31T19:41:38.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.400+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f45ec01d420 con 0x7f45fc0f6980 2022-01-31T19:41:38.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.400+0000 7f45f2ffd700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f45e805e5c0 0x7f45e8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:38.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.400+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f45ec07a210 con 0x7f45fc0f6980 2022-01-31T19:41:38.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.400+0000 7f45fa59c700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f45e805e5c0 0x7f45e8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:38.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.400+0000 7f45fa59c700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f45e805e5c0 0x7f45e8060a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f45fc0f67e0 tx=0x7f45e4009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:38.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.404+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f45ec04c170 con 0x7f45fc0f6980 2022-01-31T19:41:38.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.562+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f45e0000bc0 con 0x7f45e805e5c0 2022-01-31T19:41:38.571 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.569+0000 7f45f2ffd700 1 -- 172.21.15.167:0/3089598465 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2520 (secure 0 0 0) 0x7f45e0000bc0 con 0x7f45e805e5c0 2022-01-31T19:41:38.572 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:41:38.572 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (2m) 41s ago 3m 16.4M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:41:38.572 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (3m) 41s ago 3m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:41:38.573 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (2m) 50s ago 2m 7147k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:41:38.573 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (2m) 41s ago 3m 30.3M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:41:38.573 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:9283 running 41s ago 4m - - 2022-01-31T19:41:38.573 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (2m) 50s ago 2m 373M - 16.2.5 46cf6318c64e 0e89e7ce37df 2022-01-31T19:41:38.573 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running 41s ago 4m - 2048M 2022-01-31T19:41:38.574 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (2m) 50s ago 2m 66.7M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:41:38.574 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (3m) 41s ago 3m 18.0M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:41:38.574 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (2m) 50s ago 2m 18.1M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:41:38.574 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (2m) 41s ago 2m 41.3M 6472M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:41:38.575 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (118s) 41s ago 118s 40.4M 6472M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:41:38.575 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (101s) 41s ago 100s 42.1M 6472M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:41:38.575 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (85s) 50s ago 85s 39.7M 5339M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:41:38.575 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (70s) 50s ago 69s 39.5M 5339M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:41:38.575 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (53s) 50s ago 53s 13.8M 5339M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:41:38.576 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (2m) 41s ago 3m 33.4M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:41:38.577 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f45e805e5c0 msgr2=0x7f45e8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:38.577 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f45e805e5c0 0x7f45e8060a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f45fc0f67e0 tx=0x7f45e4009040).stop 2022-01-31T19:41:38.577 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 msgr2=0x7f45fc106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:38.577 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc106470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f45ec009990 tx=0x7f45ec009b30).stop 2022-01-31T19:41:38.578 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 shutdown_connections 2022-01-31T19:41:38.578 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f45e805e5c0 0x7f45e8060a80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:38.578 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45fc0f56e0 0x7f45fc105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:38.578 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 --2- 172.21.15.167:0/3089598465 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45fc0f6980 0x7f45fc106470 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:38.579 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 >> 172.21.15.167:0/3089598465 conn(0x7f45fc0f0e60 msgr2=0x7f45fc0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:38.579 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 shutdown_connections 2022-01-31T19:41:38.579 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:38.571+0000 7f46009f5700 1 -- 172.21.15.167:0/3089598465 wait complete. 2022-01-31T19:41:38.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:38 smithi171 conmon[35325]: cluster 2022-01-31T19:41:36.722409+0000 mgr.smithi167.aciqpk (mgr.14156) 172 : 2022-01-31T19:41:38.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:38 smithi171 conmon[35325]: cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:38.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph orch ls' 2022-01-31T19:41:40.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:40 smithi167 conmon[32206]: audit 2022-01-31T19:41:38.564413+0000 mgr.smithi167.aciqpk (mgr.14156) 173 : audit [DBG] from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:40.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:40 smithi167 conmon[32206]: cluster 2022-01-31T19:41:38.722674+0000 mgr.smithi167.aciqpk (mgr.14156) 174 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:40.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 -- 172.21.15.167:0/3089934925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f56e0 msgr2=0x7fc0480f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 --2- 172.21.15.167:0/3089934925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f56e0 0x7fc0480f5b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc038004660 tx=0x7fc038009b30).stop 2022-01-31T19:41:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 -- 172.21.15.167:0/3089934925 shutdown_connections 2022-01-31T19:41:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 --2- 172.21.15.167:0/3089934925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0480f6980 0x7fc0480f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 --2- 172.21.15.167:0/3089934925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f56e0 0x7fc0480f5b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 -- 172.21.15.167:0/3089934925 >> 172.21.15.167:0/3089934925 conn(0x7fc0480f0e60 msgr2=0x7fc0480f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:40.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 -- 172.21.15.167:0/3089934925 shutdown_connections 2022-01-31T19:41:40.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.447+0000 7fc04f49a700 1 -- 172.21.15.167:0/3089934925 wait complete. 2022-01-31T19:41:40.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04f49a700 1 Processor -- start 2022-01-31T19:41:40.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04f49a700 1 -- start start 2022-01-31T19:41:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04f49a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0480f56e0 0x7fc048105fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04f49a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 0x7fc0481064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04f49a700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc048106a50 con 0x7fc0480f6980 2022-01-31T19:41:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04f49a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc048106b90 con 0x7fc0480f56e0 2022-01-31T19:41:40.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04ca35700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 0x7fc0481064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:40.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04ca35700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 0x7fc0481064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33206/0 (socket says 172.21.15.167:33206) 2022-01-31T19:41:40.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.448+0000 7fc04ca35700 1 -- 172.21.15.167:0/70177410 learned_addr learned my addr 172.21.15.167:0/70177410 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:40.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04d236700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0480f56e0 0x7fc048105fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04ca35700 1 -- 172.21.15.167:0/70177410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0480f56e0 msgr2=0x7fc048105fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04ca35700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0480f56e0 0x7fc048105fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04ca35700 1 -- 172.21.15.167:0/70177410 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc038005040 con 0x7fc0480f6980 2022-01-31T19:41:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04ca35700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 0x7fc0481064a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fc0440026d0 tx=0x7fc044009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:40.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0440079c0 con 0x7fc0480f6980 2022-01-31T19:41:40.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc04400f040 con 0x7fc0480f6980 2022-01-31T19:41:40.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc048107530 con 0x7fc0480f6980 2022-01-31T19:41:40.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc044014820 con 0x7fc0480f6980 2022-01-31T19:41:40.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.449+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc048107af0 con 0x7fc0480f6980 2022-01-31T19:41:40.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.450+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fc044014980 con 0x7fc0480f6980 2022-01-31T19:41:40.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.450+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc048045bb0 con 0x7fc0480f6980 2022-01-31T19:41:40.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.451+0000 7fc03dffb700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc03405e5c0 0x7fc034060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:40.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.451+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7fc044078450 con 0x7fc0480f6980 2022-01-31T19:41:40.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.451+0000 7fc04d236700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc03405e5c0 0x7fc034060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:40.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.451+0000 7fc04d236700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc03405e5c0 0x7fc034060a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc038005010 tx=0x7fc038009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:40.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.455+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc04404a410 con 0x7fc0480f6980 2022-01-31T19:41:40.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:40 smithi171 conmon[35325]: audit 2022-01-31T19:41:38.564413+0000 mgr.smithi167.aciqpk (mgr.14156) 173 : audit 2022-01-31T19:41:40.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:40 smithi171 conmon[35325]: [DBG] from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:40.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:40 smithi171 conmon[35325]: cluster 2022-01-31T19:41:38.722674+0000 mgr.smithi167.aciqpk (mgr.14156) 174 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.612+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc048107dc0 con 0x7fc03405e5c0 2022-01-31T19:41:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.616+0000 7fc03dffb700 1 -- 172.21.15.167:0/70177410 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7fc048107dc0 con 0x7fc03405e5c0 2022-01-31T19:41:40.618 INFO:teuthology.orchestra.run.smithi167.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-01-31T19:41:40.618 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager ?:9093,9094 1/1 43s ago 3m count:1 2022-01-31T19:41:40.619 INFO:teuthology.orchestra.run.smithi167.stdout:crash 2/2 52s ago 3m * 2022-01-31T19:41:40.619 INFO:teuthology.orchestra.run.smithi167.stdout:grafana ?:3000 1/1 43s ago 3m count:1 2022-01-31T19:41:40.619 INFO:teuthology.orchestra.run.smithi167.stdout:mgr 2/2 52s ago 3m count:2 2022-01-31T19:41:40.619 INFO:teuthology.orchestra.run.smithi167.stdout:mon 2/2 52s ago 3m smithi167:172.21.15.167=smithi167;smithi171:172.21.15.171=smithi171;count:2 2022-01-31T19:41:40.620 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter ?:9100 2/2 52s ago 3m * 2022-01-31T19:41:40.620 INFO:teuthology.orchestra.run.smithi167.stdout:osd 6/6 52s ago - 2022-01-31T19:41:40.620 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus ?:9095 1/1 43s ago 3m count:1 2022-01-31T19:41:40.621 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc03405e5c0 msgr2=0x7fc034060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:40.621 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc03405e5c0 0x7fc034060a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc038005010 tx=0x7fc038009b30).stop 2022-01-31T19:41:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 msgr2=0x7fc0481064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 0x7fc0481064a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fc0440026d0 tx=0x7fc044009de0).stop 2022-01-31T19:41:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 shutdown_connections 2022-01-31T19:41:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc03405e5c0 0x7fc034060a80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0480f56e0 0x7fc048105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 --2- 172.21.15.167:0/70177410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0480f6980 0x7fc0481064a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 >> 172.21.15.167:0/70177410 conn(0x7fc0480f0e60 msgr2=0x7fc0480fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 shutdown_connections 2022-01-31T19:41:40.624 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:40.618+0000 7fc04f49a700 1 -- 172.21.15.167:0/70177410 wait complete. 2022-01-31T19:41:40.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph orch host ls' 2022-01-31T19:41:41.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:41 smithi167 conmon[32206]: audit 2022-01-31T19:41:40. 2022-01-31T19:41:41.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:41 smithi167 conmon[32206]: 614770+0000 mgr.smithi167.aciqpk (mgr.14156) 175 : audit [DBG] from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:41.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:41 smithi167 conmon[32206]: cluster 2022-01-31T19:41:40.723115+0000 mgr.smithi167.aciqpk (mgr.14156) 176 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:41 smithi171 conmon[35325]: audit 2022-01-31T19:41:40.614770+0000 mgr.smithi167.aciqpk (mgr.14156) 175 : 2022-01-31T19:41:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:41 smithi171 conmon[35325]: audit [DBG] from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:41 smithi171 conmon[35325]: cluster 2022-01-31T19:41:40.723115+0000 mgr.smithi167.aciqpk (mgr.14156) 176 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:42.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 -- 172.21.15.167:0/2608114858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 msgr2=0x7f17000f7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:42.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 --2- 172.21.15.167:0/2608114858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f17000f7f10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f16f0004660 tx=0x7f16f0009b30).stop 2022-01-31T19:41:42.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 -- 172.21.15.167:0/2608114858 shutdown_connections 2022-01-31T19:41:42.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 --2- 172.21.15.167:0/2608114858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f17000f7f10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:42.607 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 --2- 172.21.15.167:0/2608114858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17000f6810 0x7f17000f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:42.607 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 -- 172.21.15.167:0/2608114858 >> 172.21.15.167:0/2608114858 conn(0x7f17000f1f90 msgr2=0x7f17000f43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:42.607 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 -- 172.21.15.167:0/2608114858 shutdown_connections 2022-01-31T19:41:42.608 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.602+0000 7f170493f700 1 -- 172.21.15.167:0/2608114858 wait complete. 2022-01-31T19:41:42.608 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f170493f700 1 Processor -- start 2022-01-31T19:41:42.608 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f170493f700 1 -- start start 2022-01-31T19:41:42.608 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f170493f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17000f6810 0x7f1700107060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:42.609 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f170493f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f1700107550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:42.609 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f170493f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700107b00 con 0x7f17000f7ab0 2022-01-31T19:41:42.609 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f170493f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700107c40 con 0x7f17000f6810 2022-01-31T19:41:42.609 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fdd9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f1700107550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:42.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fdd9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f1700107550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33218/0 (socket says 172.21.15.167:33218) 2022-01-31T19:41:42.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fdd9b700 1 -- 172.21.15.167:0/572655512 learned_addr learned my addr 172.21.15.167:0/572655512 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:42.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fe59c700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17000f6810 0x7f1700107060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:42.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fdd9b700 1 -- 172.21.15.167:0/572655512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17000f6810 msgr2=0x7f1700107060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:42.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fdd9b700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17000f6810 0x7f1700107060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:42.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.603+0000 7f16fdd9b700 1 -- 172.21.15.167:0/572655512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0005040 con 0x7f17000f7ab0 2022-01-31T19:41:42.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.604+0000 7f16fdd9b700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f1700107550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f16f0009990 tx=0x7f16f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:42.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.604+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f0018070 con 0x7f17000f7ab0 2022-01-31T19:41:42.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.604+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1700108580 con 0x7f17000f7ab0 2022-01-31T19:41:42.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.604+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f16f0013410 con 0x7f17000f7ab0 2022-01-31T19:41:42.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.604+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f001c3c0 con 0x7f17000f7ab0 2022-01-31T19:41:42.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.604+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1700108ae0 con 0x7f17000f7ab0 2022-01-31T19:41:42.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.605+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700045bb0 con 0x7f17000f7ab0 2022-01-31T19:41:42.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.605+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f16f001d420 con 0x7f17000f7ab0 2022-01-31T19:41:42.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.605+0000 7f16eeffd700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f16e805e5c0 0x7f16e8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:42.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.606+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f16f007a380 con 0x7f17000f7ab0 2022-01-31T19:41:42.615 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.606+0000 7f16fe59c700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f16e805e5c0 0x7f16e8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:42.615 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.606+0000 7f16fe59c700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f16e805e5c0 0x7f16e8060a80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f17000f7910 tx=0x7f16f4009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:42.615 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.610+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f16f004c2c0 con 0x7f17000f7ab0 2022-01-31T19:41:42.766 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.765+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f17000ff180 con 0x7f16e805e5c0 2022-01-31T19:41:42.767 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.766+0000 7f16eeffd700 1 -- 172.21.15.167:0/572655512 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+129 (secure 0 0 0) 0x7f16f400f650 con 0x7f16e805e5c0 2022-01-31T19:41:42.767 INFO:teuthology.orchestra.run.smithi167.stdout:HOST ADDR LABELS STATUS 2022-01-31T19:41:42.767 INFO:teuthology.orchestra.run.smithi167.stdout:smithi167 172.21.15.167 2022-01-31T19:41:42.768 INFO:teuthology.orchestra.run.smithi167.stdout:smithi171 172.21.15.171 2022-01-31T19:41:42.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f16e805e5c0 msgr2=0x7f16e8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:42.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f16e805e5c0 0x7f16e8060a80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f17000f7910 tx=0x7f16f4009040).stop 2022-01-31T19:41:42.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 msgr2=0x7f1700107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:42.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f1700107550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f16f0009990 tx=0x7f16f0009b30).stop 2022-01-31T19:41:42.770 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 shutdown_connections 2022-01-31T19:41:42.770 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f16e805e5c0 0x7f16e8060a80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:42.770 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17000f6810 0x7f1700107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:42.770 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 --2- 172.21.15.167:0/572655512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17000f7ab0 0x7f1700107550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:42.771 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 >> 172.21.15.167:0/572655512 conn(0x7f17000f1f90 msgr2=0x7f17000fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:42.771 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 shutdown_connections 2022-01-31T19:41:42.771 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:42.768+0000 7f170493f700 1 -- 172.21.15.167:0/572655512 wait complete. 2022-01-31T19:41:43.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph orch device ls' 2022-01-31T19:41:44.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:43 smithi171 conmon[35325]: cluster 2022-01-31T19:41:42.723455+0000 mgr.smithi167.aciqpk (mgr.14156) 177 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:44.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:43 smithi171 conmon[35325]: audit 2022-01-31T19:41:42.766313+0000 mgr.smithi167.aciqpk (mgr.14156) 178 : audit [DBG] from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:44.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:43 smithi167 conmon[32206]: cluster 2022-01-31T19:41:42.723455+0000 mgr.smithi167.aciqpk 2022-01-31T19:41:44.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:43 smithi167 conmon[32206]: (mgr.14156) 177 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:44.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:43 smithi167 conmon[32206]: audit 2022-01-31T19:41:42.766313+0000 mgr.smithi167.aciqpk (mgr.14156) 178 : audit [DBG] from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:44.604 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 -- 172.21.15.167:0/2450781459 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 msgr2=0x7f9fe80fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:44.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 --2- 172.21.15.167:0/2450781459 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe80fbd90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f9fd8004660 tx=0x7f9fd8009b30).stop 2022-01-31T19:41:44.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 -- 172.21.15.167:0/2450781459 shutdown_connections 2022-01-31T19:41:44.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 --2- 172.21.15.167:0/2450781459 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe80fbd90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:44.605 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 --2- 172.21.15.167:0/2450781459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9fe8060390 0x7f9fe80f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:44.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 -- 172.21.15.167:0/2450781459 >> 172.21.15.167:0/2450781459 conn(0x7f9fe80f0e60 msgr2=0x7f9fe80f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:44.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 -- 172.21.15.167:0/2450781459 shutdown_connections 2022-01-31T19:41:44.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 -- 172.21.15.167:0/2450781459 wait complete. 2022-01-31T19:41:44.606 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.603+0000 7f9fef770700 1 Processor -- start 2022-01-31T19:41:44.607 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fef770700 1 -- start start 2022-01-31T19:41:44.609 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fef770700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9fe8060390 0x7f9fe8107090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:44.609 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fef770700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe8107580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:44.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fef770700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe8107b30 con 0x7f9fe80f99a0 2022-01-31T19:41:44.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fef770700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe8107c70 con 0x7f9fe8060390 2022-01-31T19:41:44.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe8107580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:44.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe8107580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33234/0 (socket says 172.21.15.167:33234) 2022-01-31T19:41:44.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 -- 172.21.15.167:0/909838751 learned_addr learned my addr 172.21.15.167:0/909838751 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:44.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fed50c700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9fe8060390 0x7f9fe8107090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:44.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 -- 172.21.15.167:0/909838751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9fe8060390 msgr2=0x7f9fe8107090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:44.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9fe8060390 0x7f9fe8107090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:44.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 -- 172.21.15.167:0/909838751 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd8005040 con 0x7f9fe80f99a0 2022-01-31T19:41:44.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fecd0b700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe8107580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f9fd8007a60 tx=0x7f9fd8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:44.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.604+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fd8018070 con 0x7f9fe80f99a0 2022-01-31T19:41:44.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.605+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9fd8013410 con 0x7f9fe80f99a0 2022-01-31T19:41:44.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.605+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe81085b0 con 0x7f9fe80f99a0 2022-01-31T19:41:44.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.605+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fd801c400 con 0x7f9fe80f99a0 2022-01-31T19:41:44.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.605+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe8108b10 con 0x7f9fe80f99a0 2022-01-31T19:41:44.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.606+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe8045bb0 con 0x7f9fe80f99a0 2022-01-31T19:41:44.614 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.606+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f9fd801d420 con 0x7f9fe80f99a0 2022-01-31T19:41:44.615 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.607+0000 7f9fddffb700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f9fd405e5c0 0x7f9fd4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:44.615 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.607+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f9fd807a480 con 0x7f9fe80f99a0 2022-01-31T19:41:44.616 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.607+0000 7f9fed50c700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f9fd405e5c0 0x7f9fd4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:44.616 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.607+0000 7f9fed50c700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f9fd405e5c0 0x7f9fd4060a80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9fe40041d0 tx=0x7f9fe4009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:44.617 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.611+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9fd804c3c0 con 0x7f9fe80f99a0 2022-01-31T19:41:44.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.768+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fe8108d30 con 0x7f9fd405e5c0 2022-01-31T19:41:44.771 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.770+0000 7f9fddffb700 1 -- 172.21.15.167:0/909838751 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+279 (secure 0 0 0) 0x7f9fe8108d30 con 0x7f9fd405e5c0 2022-01-31T19:41:44.772 INFO:teuthology.orchestra.run.smithi167.stdout:Hostname Path Type Serial Size Health Ident Fault Available 2022-01-31T19:41:44.772 INFO:teuthology.orchestra.run.smithi167.stdout:smithi167 /dev/nvme0n1 ssd CVFT62330080400BGN 400G Unknown N/A N/A No 2022-01-31T19:41:44.772 INFO:teuthology.orchestra.run.smithi167.stdout:smithi171 /dev/nvme0n1 ssd CVFT6233001S400BGN 400G Unknown N/A N/A No 2022-01-31T19:41:44.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f9fd405e5c0 msgr2=0x7f9fd4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:44.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f9fd405e5c0 0x7f9fd4060a80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9fe40041d0 tx=0x7f9fe4009040).stop 2022-01-31T19:41:44.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 msgr2=0x7f9fe8107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:44.774 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe8107580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f9fd8007a60 tx=0x7f9fd8009b30).stop 2022-01-31T19:41:44.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 shutdown_connections 2022-01-31T19:41:44.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f9fd405e5c0 0x7f9fd4060a80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:44.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9fe8060390 0x7f9fe8107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:44.775 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 --2- 172.21.15.167:0/909838751 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fe80f99a0 0x7f9fe8107580 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:44.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 >> 172.21.15.167:0/909838751 conn(0x7f9fe80f0e60 msgr2=0x7f9fe80f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:44.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 shutdown_connections 2022-01-31T19:41:44.776 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:44.772+0000 7f9fef770700 1 -- 172.21.15.167:0/909838751 wait complete. 2022-01-31T19:41:45.142 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:41:45.157 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:41:45.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs volume create cephfs --placement=4' 2022-01-31T19:41:45.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:45 smithi167 conmon[32206]: cluster 2022-01-31T19:41:44.723750+0000 mgr.smithi167.aciqpk (mgr.14156) 179 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:45.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:45 smithi167 conmon[32206]: audit 2022-01-31T19:41:44.770071+0000 mgr.smithi167.aciqpk (mgr.14156) 180 : audit [DBG] from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:46.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:45 smithi171 conmon[35325]: cluster 2022-01-31T19:41:44.723750+0000 mgr.smithi167.aciqpk (mgr.14156) 179 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:46.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:45 smithi171 conmon[35325]: 2022-01-31T19:41:46.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:45 smithi171 conmon[35325]: audit 2022-01-31T19:41:44.770071+0000 mgr.smithi167.aciqpk (mgr.14156) 180 : audit [DBG] from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:46.753 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.750+0000 7f775cb42700 1 -- 172.21.15.167:0/2071201365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f6650 msgr2=0x7f77580f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:46.754 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 --2- 172.21.15.167:0/2071201365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f6650 0x7f77580f6a70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f7748004660 tx=0x7f7748009b30).stop 2022-01-31T19:41:46.754 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- 172.21.15.167:0/2071201365 shutdown_connections 2022-01-31T19:41:46.754 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 --2- 172.21.15.167:0/2071201365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77580f78f0 0x7f77580f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:46.755 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 --2- 172.21.15.167:0/2071201365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f6650 0x7f77580f6a70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:46.755 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- 172.21.15.167:0/2071201365 >> 172.21.15.167:0/2071201365 conn(0x7f77580f1db0 msgr2=0x7f77580f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:46.755 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- 172.21.15.167:0/2071201365 shutdown_connections 2022-01-31T19:41:46.755 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- 172.21.15.167:0/2071201365 wait complete. 2022-01-31T19:41:46.756 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 Processor -- start 2022-01-31T19:41:46.756 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- start start 2022-01-31T19:41:46.757 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77580f6650 0x7f7758106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:46.757 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 0x7f77581073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:46.757 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7758107960 con 0x7f77580f78f0 2022-01-31T19:41:46.757 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f775cb42700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7758107aa0 con 0x7f77580f6650 2022-01-31T19:41:46.758 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.751+0000 7f774ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 0x7f77581073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:46.758 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f774ffff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 0x7f77581073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33252/0 (socket says 172.21.15.167:33252) 2022-01-31T19:41:46.758 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f774ffff700 1 -- 172.21.15.167:0/3049338898 learned_addr learned my addr 172.21.15.167:0/3049338898 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:46.758 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f77577fe700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77580f6650 0x7f7758106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:46.759 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f774ffff700 1 -- 172.21.15.167:0/3049338898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77580f6650 msgr2=0x7f7758106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:46.759 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f774ffff700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77580f6650 0x7f7758106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:46.759 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f774ffff700 1 -- 172.21.15.167:0/3049338898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7748005040 con 0x7f77580f78f0 2022-01-31T19:41:46.759 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f774ffff700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 0x7f77581073b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f774000e7f0 tx=0x7f774000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:46.760 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77400092e0 con 0x7f77580f78f0 2022-01-31T19:41:46.760 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f774000f040 con 0x7f77580f78f0 2022-01-31T19:41:46.760 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77400146c0 con 0x7f77580f78f0 2022-01-31T19:41:46.760 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7758108440 con 0x7f77580f78f0 2022-01-31T19:41:46.761 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.752+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77581089b0 con 0x7f77580f78f0 2022-01-31T19:41:46.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.753+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7758045bb0 con 0x7f77580f78f0 2022-01-31T19:41:46.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.753+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f774001b030 con 0x7f77580f78f0 2022-01-31T19:41:46.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.754+0000 7f7754ff9700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f774405e5c0 0x7f7744060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:46.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.754+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4347+0+0 (secure 0 0 0) 0x7f7740078300 con 0x7f77580f78f0 2022-01-31T19:41:46.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.754+0000 7f77577fe700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f774405e5c0 0x7f7744060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:46.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.754+0000 7f77577fe700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f774405e5c0 0x7f7744060a80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7748000c00 tx=0x7f774800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:46.764 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.758+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f774004a290 con 0x7f77580f78f0 2022-01-31T19:41:46.940 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:46.939+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7758108c20 con 0x7f774405e5c0 2022-01-31T19:41:48.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:47 smithi171 conmon[35325]: cluster 2022-01-31T19:41:46.724070+0000 mgr.smithi167.aciqpk (mgr.14156) 181 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:48.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:47 smithi171 conmon[35325]: 2022-01-31T19:41:48.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:47 smithi171 conmon[35325]: audit 2022-01-31T19:41:46.940376+0000 mgr.smithi167.aciqpk (mgr.14156) 182 : audit [DBG] from='client.14432 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:48.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:47 smithi171 conmon[35325]: audit 2022-01-31T19:41:46.940927+0000 2022-01-31T19:41:48.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:47 smithi171 conmon[35325]: mon.smithi167 (mon.0) 485 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2022-01-31T19:41:48.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:47 smithi167 conmon[32206]: cluster 2022-01-31T19:41:46.724070+0000 mgr.smithi167.aciqpk (mgr.14156) 181 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:48.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:47 smithi167 conmon[32206]: audit 2022-01-31T19:41:46.940376+0000 mgr.smithi167.aciqpk (mgr.14156) 182 : audit [DBG] from='client.14432 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:41:48.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:47 smithi167 conmon[32206]: audit 2022-01-31T19:41:46.940927+0000 mon.smithi167 (mon.0) 485 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2022-01-31T19:41:48.828 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.827+0000 7f7754ff9700 1 -- 172.21.15.167:0/3049338898 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7758108c20 con 0x7f774405e5c0 2022-01-31T19:41:48.831 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f774405e5c0 msgr2=0x7f7744060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:48.832 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f774405e5c0 0x7f7744060a80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7748000c00 tx=0x7f774800b040).stop 2022-01-31T19:41:48.832 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 msgr2=0x7f77581073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:48.832 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 0x7f77581073b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f774000e7f0 tx=0x7f774000b040).stop 2022-01-31T19:41:48.833 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 shutdown_connections 2022-01-31T19:41:48.833 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f774405e5c0 0x7f7744060a80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:48.833 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77580f6650 0x7f7758106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:48.833 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 --2- 172.21.15.167:0/3049338898 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77580f78f0 0x7f77581073b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:48.834 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 >> 172.21.15.167:0/3049338898 conn(0x7f77580f1db0 msgr2=0x7f77580fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:48.834 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 shutdown_connections 2022-01-31T19:41:48.834 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:48.830+0000 7f775cb42700 1 -- 172.21.15.167:0/3049338898 wait complete. 2022-01-31T19:41:49.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:48 smithi171 conmon[35325]: audit 2022-01-31T19:41:47.788879+0000 mon.smithi167 (mon.0) 486 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2022-01-31T19:41:49.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:48 smithi171 conmon[35325]: cluster 2022-01-31T19:41:47.788925+0000 mon.smithi167 (mon.0) 487 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2022-01-31T19:41:49.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:48 smithi171 conmon[35325]: audit 2022-01-31T19:41:47.790248+0000 mon.smithi167 (mon.0) 488 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2022-01-31T19:41:49.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:48 smithi167 conmon[32206]: audit 2022-01-31T19:41:47.788879+0000 mon.smithi167 (mon.0) 486 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2022-01-31T19:41:49.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:48 smithi167 conmon[32206]: cluster 2022-01-31T19:41:47.788925+0000 mon.smithi167 (mon.0) 487 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2022-01-31T19:41:49.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:48 smithi167 conmon[32206]: audit 2022-01-31T19:41:47.790248+0000 mon.smithi167 (mon.0) 488 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2022-01-31T19:41:49.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:48 smithi167 conmon[32206]: debug 2022-01-31T19:41:48.802+0000 7f07d1ee6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-01-31T19:41:49.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs dump' 2022-01-31T19:41:50.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cluster 2022-01-31T19:41:48.724450+0000 mgr.smithi167.aciqpk (mgr.14156) 183 : cluster [DBG] pgmap v141: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:50.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.796909+0000 mon.smithi167 (mon.0) 489 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2022-01-31T19:41:50.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cluster 2022-01-31T19:41:48.797129+0000 mon.smithi167 (mon.0) 490 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2022-01-31T19:41:50.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.802393+0000 mon.smithi167 (mon.0) 491 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2022-01-31T19:41:50.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cluster 2022-01-31T19:41:48.803290+0000 mon.smithi167 (mon.0) 492 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-01-31T19:41:50.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cluster 2022-01-31T19:41:48.803317+0000 mon.smithi167 (mon.0) 493 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-01-31T19:41:50.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cluster 2022-01-31T19:41:48.818526+0000 mon.smithi167 (mon.0) 494 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2022-01-31T19:41:50.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.818704+0000 mon.smithi167 (mon.0) 495 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2022-01-31T19:41:50.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cluster 2022-01-31T 2022-01-31T19:41:50.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: 19:41:48.818800+0000 mon.smithi167 (mon.0) 496 : cluster [DBG] fsmap cephfs:0 2022-01-31T19:41:50.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cephadm 2022-01-31T19:41:48.823463+0000 mgr.smithi167.aciqpk (mgr.14156) 184 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2022-01-31T19:41:50.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.827550+0000 mon.smithi167 (mon.0) 497 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:50.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.828461+0000 mon.smithi167 (mon.0) 498 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:41:50.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.829954+0000 mon.smithi167 (mon.0) 499 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:50.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.831443+0000 mon.smithi167 (mon.0) 500 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:41:50.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.844369+0000 mon.smithi167 (mon.0) 501 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:50.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.851467+0000 mon.smithi167 (mon.0) 502 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:50.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.852240+0000 mon.smithi167 (mon.0) 503 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.sgnfie", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:50.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.855086+0000 mon.smithi167 (mon.0) 504 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.sgnfie", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:50.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: audit 2022-01-31T19:41:48.855704+0000 mon.smithi167 (mon.0) 505 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:50.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:49 smithi167 conmon[32206]: cephadm 2022-01-31T19:41:48.856311+0000 mgr.smithi167.aciqpk (mgr.14156) 185 : cephadm [INF] Deploying daemon mds.cephfs.smithi167.sgnfie on smithi167 2022-01-31T19:41:50.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cluster 2022-01-31T19:41:48.724450+0000 mgr.smithi167.aciqpk (mgr.14156) 183 : cluster 2022-01-31T19:41:50.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: [DBG] pgmap v141: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:50.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19:41:48.796909+0000 mon.smithi167 (mon.0) 489 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2022-01-31T19:41:50.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cluster 2022-01-31T19:41:48.797129+0000 mon.smithi167 2022-01-31T19:41:50.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: (mon.0) 490 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2022-01-31T19:41:50.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:41:50.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: :41:48.802393+0000 mon.smithi167 (mon.0) 491 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2022-01-31T19:41:50.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cluster 2022- 2022-01-31T19:41:50.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 01-31T19:41:48.803290+0000 mon.smithi167 ( 2022-01-31T19:41:50.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.0) 492 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-01-31T19:41:50.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cluster 2022- 2022-01-31T19:41:50.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 01-31T19:41:48.803317+0000 2022-01-31T19:41:50.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 493 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-01-31T19:41:50.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cluster 2022- 2022-01-31T19:41:50.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 01-31T19:41:48.818526+0000 mon.smithi167 ( 2022-01-31T19:41:50.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.0) 494 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2022-01-31T19:41:50.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:41:50.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 01-31T19:41:48.818704+0000 2022-01-31T19:41:50.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 495 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2022-01-31T19:41:50.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cluster 2022- 2022-01-31T19:41:50.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 01-31T19:41:48.818800+0000 2022-01-31T19:41:50.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 496 : cluster [DBG] fsmap cephfs:0 2022-01-31T19:41:50.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cephadm 2022- 2022-01-31T19:41:50.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 01-31T19:41:48.823463+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:41:50.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: .14156) 184 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2022-01-31T19:41:50.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:41:50.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 31T19:41:48.827550+0000 mon.smithi167 ( 2022-01-31T19:41:50.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.0) 497 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:50.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19:41:50.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 2022-01-31T19:41:48.828461+0000 2022-01-31T19:41:50.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 498 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:41:50.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 2022-01-31T19:41:50.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19:41:48 2022-01-31T19:41:50.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: .829954+0000 mon.smithi167 (mon.0) 499 : audit 2022-01-31T19:41:50.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:50.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19:41 2022-01-31T19:41:50.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: :48.831443+0000 mon.smithi167 (mon.0) 500 2022-01-31T19:41:50.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:41:50.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19:41:48.844369+0000 2022-01-31T19:41:50.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 501 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:50.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 2022-01-31T19:41:50.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19:41:48 2022-01-31T19:41:50.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: .851467+0000 mon.smithi167 (mon.0) 502 : audit 2022-01-31T19:41:50.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:50.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:41:50.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 41:48.852240+0000 mon.smithi167 (mon.0) 2022-01-31T19:41:50.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 503 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.sgnfie", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:50.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:41:50.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: 31T19:41:48.855086+0000 mon.smithi167 ( 2022-01-31T19:41:50.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mon.0) 504 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.sgnfie", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:50.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:41:50.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: -31T19:41:48.855704+0000 mon.smithi167 2022-01-31T19:41:50.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: (mon.0) 505 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:50.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: cephadm 2022 2022-01-31T19:41:50.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: -01-31T19:41:48.856311+0000 2022-01-31T19:41:50.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 185 : cephadm [INF] 2022-01-31T19:41:50.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:49 smithi171 conmon[35325]: Deploying daemon mds.cephfs.smithi167.sgnfie on smithi167 2022-01-31T19:41:51.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:50 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:41:51.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:50 smithi167 conmon[32206]: 41:49.821294+0000 mon.smithi167 (mon.0) 506 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2022-01-31T19:41:51.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:50 smithi171 conmon[35325]: cluster 2022-01-31T19:41:49.821294+0000 mon.smithi167 (mon.0) 506 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2022-01-31T19:41:51.343 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 -- 172.21.15.167:0/241629984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f5710 msgr2=0x7feb440f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:51.344 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 --2- 172.21.15.167:0/241629984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f5710 0x7feb440f5b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7feb40004660 tx=0x7feb40009b30).stop 2022-01-31T19:41:51.344 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 -- 172.21.15.167:0/241629984 shutdown_connections 2022-01-31T19:41:51.344 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 --2- 172.21.15.167:0/241629984 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f69b0 0x7feb440f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:51.345 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 --2- 172.21.15.167:0/241629984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f5710 0x7feb440f5b30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:51.345 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 -- 172.21.15.167:0/241629984 >> 172.21.15.167:0/241629984 conn(0x7feb440f0e50 msgr2=0x7feb440f3290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:51.345 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 -- 172.21.15.167:0/241629984 shutdown_connections 2022-01-31T19:41:51.345 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.342+0000 7feb4ba31700 1 -- 172.21.15.167:0/241629984 wait complete. 2022-01-31T19:41:51.347 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.343+0000 7feb4ba31700 1 Processor -- start 2022-01-31T19:41:51.348 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.343+0000 7feb4ba31700 1 -- start start 2022-01-31T19:41:51.348 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb4ba31700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f5710 0x7feb44109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:51.348 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb4ba31700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 0x7feb4410c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:51.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb4ba31700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440606f0 con 0x7feb440f69b0 2022-01-31T19:41:51.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb4ba31700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb44060830 con 0x7feb440f5710 2022-01-31T19:41:51.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 0x7feb4410c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:51.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 0x7feb4410c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33278/0 (socket says 172.21.15.167:33278) 2022-01-31T19:41:51.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 -- 172.21.15.167:0/288351467 learned_addr learned my addr 172.21.15.167:0/288351467 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:51.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb497cd700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f5710 0x7feb44109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:51.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 -- 172.21.15.167:0/288351467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f5710 msgr2=0x7feb44109db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:51.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f5710 0x7feb44109db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:51.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 -- 172.21.15.167:0/288351467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40005040 con 0x7feb440f69b0 2022-01-31T19:41:51.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb497cd700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f5710 0x7feb44109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:41:51.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.344+0000 7feb48fcc700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 0x7feb4410c2b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7feb3400c530 tx=0x7feb34007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:51.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.346+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb3400f650 con 0x7feb440f69b0 2022-01-31T19:41:51.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.346+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4410c860 con 0x7feb440f69b0 2022-01-31T19:41:51.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.347+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7feb3400fc40 con 0x7feb440f69b0 2022-01-31T19:41:51.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.347+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb34012c40 con 0x7feb440f69b0 2022-01-31T19:41:51.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.347+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4410ce20 con 0x7feb440f69b0 2022-01-31T19:41:51.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.348+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb44045bb0 con 0x7feb440f69b0 2022-01-31T19:41:51.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.348+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7feb34010070 con 0x7feb440f69b0 2022-01-31T19:41:51.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.348+0000 7feb39ffb700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7feb3005e5c0 0x7feb30060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:51.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.348+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7feb340785e0 con 0x7feb440f69b0 2022-01-31T19:41:51.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.348+0000 7feb497cd700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7feb3005e5c0 0x7feb30060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:51.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.349+0000 7feb497cd700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7feb3005e5c0 0x7feb30060a80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7feb40009830 tx=0x7feb40009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:51.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.353+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7feb340456b0 con 0x7feb440f69b0 2022-01-31T19:41:51.541 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.539+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feb44027ae0 con 0x7feb440f69b0 2022-01-31T19:41:51.541 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.539+0000 7feb39ffb700 1 -- 172.21.15.167:0/288351467 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1003 (secure 0 0 0) 0x7feb3404dc10 con 0x7feb440f69b0 2022-01-31T19:41:51.542 INFO:teuthology.orchestra.run.smithi167.stdout:e2 2022-01-31T19:41:51.542 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:41:51.543 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:41:51.543 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:41:51.543 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:51.543 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:41:51.544 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:41:51.544 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 2 2022-01-31T19:41:51.544 INFO:teuthology.orchestra.run.smithi167.stdout:flags 12 2022-01-31T19:41:51.544 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:41:51.545 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:41:48.803193+0000 2022-01-31T19:41:51.545 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:41:51.545 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:41:51.545 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:41:51.546 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:41:51.546 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:41:51.546 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:41:51.546 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:41:51.547 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:41:51.547 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:41:51.547 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 1 2022-01-31T19:41:51.547 INFO:teuthology.orchestra.run.smithi167.stdout:in 2022-01-31T19:41:51.548 INFO:teuthology.orchestra.run.smithi167.stdout:up {} 2022-01-31T19:41:51.548 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:41:51.548 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:41:51.548 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:41:51.549 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:41:51.549 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:41:51.549 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data disabled 2022-01-31T19:41:51.550 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:41:51.550 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 0 2022-01-31T19:41:51.550 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:51.550 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:41:51.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7feb3005e5c0 msgr2=0x7feb30060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:51.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7feb3005e5c0 0x7feb30060a80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7feb40009830 tx=0x7feb40009b30).stop 2022-01-31T19:41:51.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 msgr2=0x7feb4410c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:51.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 0x7feb4410c2b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7feb3400c530 tx=0x7feb34007e20).stop 2022-01-31T19:41:51.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 shutdown_connections 2022-01-31T19:41:51.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7feb3005e5c0 0x7feb30060a80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:51.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb440f5710 0x7feb44109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:51.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 --2- 172.21.15.167:0/288351467 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb440f69b0 0x7feb4410c2b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:51.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.541+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 >> 172.21.15.167:0/288351467 conn(0x7feb440f0e50 msgr2=0x7feb440fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:51.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.542+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 shutdown_connections 2022-01-31T19:41:51.555 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:51.542+0000 7feb4ba31700 1 -- 172.21.15.167:0/288351467 wait complete. 2022-01-31T19:41:51.555 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 2 2022-01-31T19:41:51.931 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:41:51.947 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:41:51.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs set cephfs max_mds 2' 2022-01-31T19:41:51.973 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: cluster 2022-01-31T19:41:50.724884+0000 mgr.smithi167.aciqpk (mgr.14156) 186 : cluster [DBG] pgmap v145: 65 pgs: 19 active+clean, 11 creating+peering, 35 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:51.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: cluster 2022-01-31T19:41:50.825957 2022-01-31T19:41:51.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 507 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2022-01-31T19:41:51.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: audit 2022-01-31T19:41:51.347062+0000 mon.smithi167 (mon.0) 508 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:51.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: audit 2022-01-31T19:41:51.348241+0000 mon.smithi167 (mon.0) 509 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.dyqqtj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:51.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: audit 2022-01-31T19:41:51.356202+0000 mon.smithi167 (mon.0) 510 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.dyqqtj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:51.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: audit 2022-01-31T19:41:51.357107+0000 mon.smithi167 (mon.0) 511 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:51.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: cephadm 2022-01-31T19:41:51.357791+0000 mgr.smithi167.aciqpk (mgr.14156) 187 : cephadm [INF] Deploying daemon mds.cephfs.smithi171.dyqqtj on smithi171 2022-01-31T19:41:51.976 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:51 smithi167 conmon[32206]: audit 2022-01-31T19:41:51.540211+0000 mon.smithi167 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.167:0/288351467' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:41:52.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: cluster 2022-01-31T19:41:50.724884+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:41:52.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: 186 : cluster [DBG] pgmap v145: 65 pgs: 19 active+clean, 11 creating+peering, 35 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:52.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: cluster 2022-01-31T19:41:50.825957+0000 mon.smithi167 (mon.0) 507 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2022-01-31T19:41:52.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: audit 2022-01-31T19:41:51.347062+0000 mon.smithi167 (mon.0) 508 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:52.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: audit 2022-01-31T19:41:51.348241+0000 mon.smithi167 (mon.0) 509 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.dyqqtj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:52.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: audit 2022-01-31T19:41:51.356202+0000 mon.smithi167 (mon.0) 510 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.dyqqtj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:52.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:41:52.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: T19:41:51.357107+0000 mon.smithi167 (mon.0) 511 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:52.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: cephadm 2022-01-31T 2022-01-31T19:41:52.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: 19:41:51.357791+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:41:52.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: 14156) 187 : cephadm [INF] Deploying daemon mds.cephfs.smithi171.dyqqtj on smithi171 2022-01-31T19:41:52.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: audit 2022-01-31T19:41:52.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: 2022-01-31T19:41: 2022-01-31T19:41:52.100 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: 51.540211+0000 mon.smithi167 (mon.0) 512 2022-01-31T19:41:52.100 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:51 smithi171 conmon[35325]: : audit [DBG] from='client.? 172.21.15.167:0/288351467' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:41:52.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.832519+0000 mon.smithi167 (mon.0) 513 : cluster [DBG] mds.? [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] up:boot 2022-01-31T19:41:52.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.832576+0000 mon.smithi167 (mon.0) 514 : cluster [INF] daemon mds.cephfs.smithi167.sgnfie assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2022-01-31T19:41:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.832640+0000 mon.smithi167 (mon.0) 515 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2022-01-31T19:41:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.832659+0000 mon.smithi167 (mon.0) 516 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-01-31T19:41:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.832681+0000 mon.smithi167 (mon.0) 517 : cluster [INF] Cluster is now healthy 2022-01-31T19:41:52.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.832843+0000 mon.smithi167 (mon.0) 518 : cluster [DBG] fsmap cephfs:0 1 up:standby 2022-01-31T19:41:52.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: audit 2022-01-31T19:41:51.832926+0000 mon.smithi167 (mon.0) 519 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.sgnfie"}]: dispatch 2022-01-31T19:41:52.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.835239+0000 mon.smithi167 (mon.0) 520 2022-01-31T19:41:52.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi167.sgnfie=up:creating} 2022-01-31T19:41:52.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:52 smithi167 conmon[32206]: cluster 2022-01-31T19:41:51.849161+0000 mon.smithi167 (mon.0) 521 : cluster [INF] daemon mds.cephfs.smithi167.sgnfie is now active in filesystem cephfs as rank 0 2022-01-31T19:41:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.832519+0000 mon.smithi167 (mon.0) 513 2022-01-31T19:41:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: : cluster [DBG] mds.? [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] up:boot 2022-01-31T19:41:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.832576+0000 mon.smithi167 (mon.0) 514 : cluster [INF] daemon mds.cephfs.smithi167.sgnfie assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2022-01-31T19:41:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.832640+0000 mon.smithi167 (mon.0) 515 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2022-01-31T19:41:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.832659+0000 mon.smithi167 (mon.0) 516 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-01-31T19:41:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.832681+0000 mon.smithi167 (mon.0) 517 : cluster [INF] Cluster is now healthy 2022-01-31T19:41:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.832843+0000 mon.smithi167 (mon.0) 518 : cluster [DBG] fsmap cephfs:0 1 up:standby 2022-01-31T19:41:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: audit 2022-01-31T19:41:51.832926+0000 mon.smithi167 (mon.0) 519 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.sgnfie"}]: dispatch 2022-01-31T19:41:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.835239+0000 mon.smithi167 (mon.0) 520 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi167.sgnfie=up:creating} 2022-01-31T19:41:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:52 smithi171 conmon[35325]: cluster 2022-01-31T19:41:51.849161+0000 mon.smithi167 (mon.0) 521 : cluster [INF] daemon mds.cephfs.smithi167.sgnfie is now active in filesystem cephfs as rank 0 2022-01-31T19:41:53.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 -- 172.21.15.167:0/3848245731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 msgr2=0x7f13ac0f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:53.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 --2- 172.21.15.167:0/3848245731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac0f7d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f1398004660 tx=0x7f1398009b30).stop 2022-01-31T19:41:53.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 -- 172.21.15.167:0/3848245731 shutdown_connections 2022-01-31T19:41:53.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 --2- 172.21.15.167:0/3848245731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac0f7d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:53.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 --2- 172.21.15.167:0/3848245731 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13ac0f6650 0x7f13ac0f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:53.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 -- 172.21.15.167:0/3848245731 >> 172.21.15.167:0/3848245731 conn(0x7f13ac0f1db0 msgr2=0x7f13ac0f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:53.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 -- 172.21.15.167:0/3848245731 shutdown_connections 2022-01-31T19:41:53.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.443+0000 7f13b1a77700 1 -- 172.21.15.167:0/3848245731 wait complete. 2022-01-31T19:41:53.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b1a77700 1 Processor -- start 2022-01-31T19:41:53.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b1a77700 1 -- start start 2022-01-31T19:41:53.449 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b1a77700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13ac0f6650 0x7f13ac06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:53.449 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b1a77700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:53.449 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b1a77700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac06a460 con 0x7f13ac0f78f0 2022-01-31T19:41:53.449 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b1a77700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac06a5a0 con 0x7f13ac0f6650 2022-01-31T19:41:53.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13abfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:53.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13abfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac069f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33302/0 (socket says 172.21.15.167:33302) 2022-01-31T19:41:53.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13abfff700 1 -- 172.21.15.167:0/2461569506 learned_addr learned my addr 172.21.15.167:0/2461569506 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:53.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.444+0000 7f13b0a75700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13ac0f6650 0x7f13ac06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:53.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13abfff700 1 -- 172.21.15.167:0/2461569506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13ac0f6650 msgr2=0x7f13ac06ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:53.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13abfff700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13ac0f6650 0x7f13ac06ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:53.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13abfff700 1 -- 172.21.15.167:0/2461569506 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1398005040 con 0x7f13ac0f78f0 2022-01-31T19:41:53.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13abfff700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac069f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1398009990 tx=0x7f1398009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:53.451 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1398018070 con 0x7f13ac0f78f0 2022-01-31T19:41:53.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac06a820 con 0x7f13ac0f78f0 2022-01-31T19:41:53.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1398013410 con 0x7f13ac0f78f0 2022-01-31T19:41:53.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139801de80 con 0x7f13ac0f78f0 2022-01-31T19:41:53.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.445+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac06ad80 con 0x7f13ac0f78f0 2022-01-31T19:41:53.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.446+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac045bb0 con 0x7f13ac0f78f0 2022-01-31T19:41:53.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.447+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f139801d420 con 0x7f13ac0f78f0 2022-01-31T19:41:53.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.447+0000 7f13a97fa700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f139c05e610 0x7f139c060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:53.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.447+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f139807a550 con 0x7f13ac0f78f0 2022-01-31T19:41:53.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.447+0000 7f13b0a75700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f139c05e610 0x7f139c060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:53.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.448+0000 7f13b0a75700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f139c05e610 0x7f139c060ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f13ac06b810 tx=0x7f13a000b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:53.455 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.451+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f139804c090 con 0x7f13ac0f78f0 2022-01-31T19:41:53.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.659+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f13ac027ae0 con 0x7f13ac0f78f0 2022-01-31T19:41:53.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.844+0000 7f13a97fa700 1 -- 172.21.15.167:0/2461569506 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f139804f6b0 con 0x7f13ac0f78f0 2022-01-31T19:41:53.848 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f139c05e610 msgr2=0x7f139c060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:53.849 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f139c05e610 0x7f139c060ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f13ac06b810 tx=0x7f13a000b040).stop 2022-01-31T19:41:53.849 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 msgr2=0x7f13ac069f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:53.849 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac069f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1398009990 tx=0x7f1398009b30).stop 2022-01-31T19:41:53.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 shutdown_connections 2022-01-31T19:41:53.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f139c05e610 0x7f139c060ad0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:53.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13ac0f6650 0x7f13ac06ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:53.851 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 --2- 172.21.15.167:0/2461569506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13ac0f78f0 0x7f13ac069f00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:53.851 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.848+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 >> 172.21.15.167:0/2461569506 conn(0x7f13ac0f1db0 msgr2=0x7f13ac0fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:53.851 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.849+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 shutdown_connections 2022-01-31T19:41:53.851 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:53.849+0000 7f13b1a77700 1 -- 172.21.15.167:0/2461569506 wait complete. 2022-01-31T19:41:54.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: cluster 2022-01-31T19:41:52.725288+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:41:54.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: 188 : cluster [DBG] pgmap v147: 65 pgs: 37 active+clean, 11 creating+peering, 17 unknown; 160 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:54.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: cluster 2022-01-31T19:41:52.839485+0000 mon.smithi167 (mon.0) 522 : cluster [DBG] mds.? [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] up:active 2022-01-31T19:41:54.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: cluster 2022-01-31T19:41:52.839550+0000 mon.smithi167 (mon.0) 523 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi167.sgnfie=up:active} 2022-01-31T19:41:54.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: audit 2022-01-31T19:41:53.148464+0000 mon.smithi167 (mon.0) 524 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:54.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: audit 2022-01-31T19:41:53.149405+0000 mon.smithi167 (mon.0) 2022-01-31T19:41:54.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: 525 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.rdkazj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:54.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:41:54.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: :41:53.154082+0000 mon.smithi167 (mon.0) 526 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.rdkazj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:54.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: audit 2022-01-31T19:41:53.154638 2022-01-31T19:41:54.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 527 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:54.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: cephadm 2022-01-31T19:41:53.155292+0000 mgr.smithi167.aciqpk 2022-01-31T19:41:54.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: (mgr.14156) 189 : cephadm [INF] Deploying daemon mds.cephfs.smithi167.rdkazj on smithi167 2022-01-31T19:41:54.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: audit 2022-01-31T19:41:53.659909+0000 mon.smithi167 ( 2022-01-31T19:41:54.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:53 smithi167 conmon[32206]: mon.0) 528 : audit [INF] from='client.? 172.21.15.167:0/2461569506' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2022-01-31T19:41:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: cluster 2022-01-31T19:41:52.725288+0000 mgr.smithi167.aciqpk (mgr.14156) 188 : cluster 2022-01-31T19:41:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: [DBG] pgmap v147: 65 pgs: 37 active+clean, 11 creating+peering, 17 unknown; 160 B data, 31 MiB used, 536 GiB / 536 GiB avail 2022-01-31T19:41:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: cluster 2022-01-31T19:41:52.839485+0000 mon.smithi167 (mon.0) 522 : cluster [DBG] mds.? [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] up:active 2022-01-31T19:41:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: cluster 2022-01-31T19:41:52.839550+0000 mon.smithi167 (mon.0) 523 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi167.sgnfie=up:active} 2022-01-31T19:41:54.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: audit 2022-01-31T19:41:53.148464+0000 mon.smithi167 (mon.0) 524 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:54.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: audit 2022-01-31T19:41:53.149405+0000 mon.smithi167 (mon.0) 525 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.rdkazj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:54.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:41:54.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: 41:53.154082+0000 mon.smithi167 (mon.0) 526 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi167.rdkazj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:54.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: audit 2022-01-31T19:41:53.154638+0000 mon.smithi167 (mon.0) 527 : audit [DBG] 2022-01-31T19:41:54.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:54.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: cephadm 2022- 2022-01-31T19:41:54.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: 01-31T19:41:53.155292+0000 mgr.smithi167.aciqpk (mgr.14156) 189 : 2022-01-31T19:41:54.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: cephadm [INF] Deploying daemon mds.cephfs.smithi167.rdkazj on smithi167 2022-01-31T19:41:54.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:41:54.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: 41:53.659909+0000 mon.smithi167 (mon.0) 528 2022-01-31T19:41:54.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:53 smithi171 conmon[35325]: : audit [INF] from='client.? 172.21.15.167:0/2461569506' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2022-01-31T19:41:54.381 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:41:54.395 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:41:54.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2022-01-31T19:41:54.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.838789+0000 mon.smithi167 (mon.0) 529 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-01-31T19:41:54.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.844611+0000 mon.smithi167 (mon.0) 530 : cluster [DBG] mds.? [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] up:boot 2022-01-31T19:41:54.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: audit 2022-01-31T19:41:53.844668+0000 mon.smithi167 (mon.0) 531 : audit [INF] from='client.? 172.21.15.167:0/2461569506' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2022-01-31T19:41:54.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.844723+0000 mon.smithi167 (mon.0) 532 : cluster [INF] daemon mds.cephfs.smithi171.dyqqtj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-01-31T19:41:54.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.844807+0000 mon.smithi167 (mon.0) 533 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2022-01-31T19:41:54.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.844837+0000 mon.smithi167 (mon.0) 534 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-01-31T19:41:54.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.845102+0000 mon.smithi167 (mon.0 2022-01-31T19:41:54.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: ) 535 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi167.sgnfie=up:active} 1 up:standby 2022-01-31T19:41:54.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: audit 2022-01-31T19:41:53.845232+0000 mon.smithi167 (mon.0) 536 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.dyqqtj"}]: dispatch 2022-01-31T19:41:54.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.848115+0000 mon.smithi167 (mon.0) 537 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:creating} 2022-01-31T19:41:54.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:54 smithi167 conmon[32206]: cluster 2022-01-31T19:41:53.860173+0000 mon.smithi167 (mon.0) 538 : cluster [INF] daemon mds.cephfs.smithi171.dyqqtj is now active in filesystem cephfs as rank 1 2022-01-31T19:41:55.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:53.838789+0000 mon.smithi167 (mon.0) 529 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-01-31T19:41:55.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: 2022-01-31T19:41:55.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:53.844611+0000 mon.smithi167 (mon.0) 530 : cluster [DBG] mds.? [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] up:boot 2022-01-31T19:41:55.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: audit 2022-01-31T19:41:53.844668+0000 mon.smithi167 (mon.0) 531 : audit [INF] from='client.? 172.21.15.167:0/2461569506' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2022-01-31T19:41:55.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:53.844723+0000 mon.smithi167 (mon.0) 532 : cluster [INF] daemon mds.cephfs.smithi171.dyqqtj assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-01-31T19:41:55.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01- 2022-01-31T19:41:55.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: 31T19:41:53.844807+0000 mon.smithi167 (mon.0) 533 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2022-01-31T19:41:55.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:55.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: 2022-01-31T19:41:53.844837+0000 mon.smithi167 (mon.0) 534 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-01-31T19:41:55.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:53.845102+0000 mon.smithi167 (mon.0 2022-01-31T19:41:55.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: ) 535 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi167.sgnfie=up:active} 1 up:standby 2022-01-31T19:41:55.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: audit 2022-01-31T19:41:55.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: 2022-01-31T19:41:53.845232+0000 mon.smithi167 (mon.0) 536 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.dyqqtj"}]: dispatch 2022-01-31T19:41:55.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:53. 2022-01-31T19:41:55.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: 848115+0000 mon.smithi167 (mon.0) 537 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:creating} 2022-01-31T19:41:55.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: cluster 2022-01-31T19:41:53.860173+0000 mon.smithi167 (mon. 2022-01-31T19:41:55.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:54 smithi171 conmon[35325]: 0) 538 : cluster [INF] daemon mds.cephfs.smithi171.dyqqtj is now active in filesystem cephfs as rank 1 2022-01-31T19:41:55.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: cluster 2022-01-31T19:41:54.725991+0000 mgr.smithi167.aciqpk (mgr.14156) 190 : cluster [DBG] pgmap v148: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 5.5 KiB/s wr, 26 op/s 2022-01-31T19:41:55.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: cluster 2022-01-31T19:41:54.851901+0000 mon.smithi167 (mon.0) 539 : cluster [DBG] mds.? [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] up:active 2022-01-31T19:41:55.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: cluster 2022-01-31T19:41:54.851960+0000 mon.smithi167 (mon.0) 540 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2022-01-31T19:41:55.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: audit 2022-01-31T19:41:55.746419+0000 mon.smithi167 (mon.0) 541 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:55.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: audit 2022-01-31T19:41:55.747318+0000 mon.smithi167 (mon.0) 542 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.erzfea", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:55.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: audit 2022-01-31T19:41:55.750313+0000 2022-01-31T19:41:55.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: mon.smithi167 (mon.0) 543 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.erzfea", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:55.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:55 smithi167 conmon[32206]: audit 2022-01-31T19:41:55.750877+0000 mon.smithi167 (mon.0) 544 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: cluster 2022-01-31T19:41:54.725991+0000 mgr.smithi167.aciqpk (mgr.14156) 190 : 2022-01-31T19:41:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: cluster [DBG] pgmap v148: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 5.5 KiB/s wr, 26 op/s 2022-01-31T19:41:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: cluster 2022-01-31T19:41:54.851901+0000 mon.smithi167 (mon.0) 539 : cluster [DBG] mds.? [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] up:active 2022-01-31T19:41:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: cluster 2022-01-31T19:41:54.851960+0000 mon.smithi167 (mon.0) 540 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2022-01-31T19:41:56.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: audit 2022-01-31T19:41:55.746419+0000 mon.smithi167 (mon.0) 541 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:56.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: audit 2022-01-31T19:41:55.747318+0000 mon.smithi167 (mon.0) 542 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.erzfea", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-01-31T19:41:56.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: audit 2022-01-31T19:41:55.750313+0000 mon.smithi167 (mon.0) 543 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi171.erzfea", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-01-31T19:41:56.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:55 smithi171 conmon[35325]: audit 2022-01-31T19:41:55.750877+0000 mon.smithi167 (mon.0) 544 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:56.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.411+0000 7f915f8bf700 1 -- 172.21.15.167:0/1934910447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 msgr2=0x7f91580f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:56.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.411+0000 7f915f8bf700 1 --2- 172.21.15.167:0/1934910447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f5b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f9154004660 tx=0x7f9154009b30).stop 2022-01-31T19:41:56.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- 172.21.15.167:0/1934910447 shutdown_connections 2022-01-31T19:41:56.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 --2- 172.21.15.167:0/1934910447 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91580f6990 0x7f91580f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:56.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 --2- 172.21.15.167:0/1934910447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f5b10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:56.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- 172.21.15.167:0/1934910447 >> 172.21.15.167:0/1934910447 conn(0x7f91580f0e30 msgr2=0x7f91580f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:56.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- 172.21.15.167:0/1934910447 shutdown_connections 2022-01-31T19:41:56.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- 172.21.15.167:0/1934910447 wait complete. 2022-01-31T19:41:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 Processor -- start 2022-01-31T19:41:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- start start 2022-01-31T19:41:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91580f6990 0x7f91580f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915810ad20 con 0x7f91580f56f0 2022-01-31T19:41:56.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915f8bf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915810ae60 con 0x7f91580f6990 2022-01-31T19:41:56.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915d65b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:56.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915d65b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33336/0 (socket says 172.21.15.167:33336) 2022-01-31T19:41:56.418 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.412+0000 7f915d65b700 1 -- 172.21.15.167:0/933583101 learned_addr learned my addr 172.21.15.167:0/933583101 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:56.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915ce5a700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91580f6990 0x7f91580f7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:56.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915d65b700 1 -- 172.21.15.167:0/933583101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91580f6990 msgr2=0x7f91580f7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:56.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915d65b700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91580f6990 0x7f91580f7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:56.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915d65b700 1 -- 172.21.15.167:0/933583101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154005040 con 0x7f91580f56f0 2022-01-31T19:41:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915d65b700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f9850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f9154007a60 tx=0x7f9154009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9154018070 con 0x7f91580f56f0 2022-01-31T19:41:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9154013410 con 0x7f91580f56f0 2022-01-31T19:41:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915810b0e0 con 0x7f91580f56f0 2022-01-31T19:41:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f915401c400 con 0x7f91580f56f0 2022-01-31T19:41:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.413+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915810b640 con 0x7f91580f56f0 2022-01-31T19:41:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.414+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9158045bb0 con 0x7f91580f56f0 2022-01-31T19:41:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.415+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f915401d420 con 0x7f91580f56f0 2022-01-31T19:41:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.415+0000 7f914dffb700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f914405e5c0 0x7f9144060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.415+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f915407a790 con 0x7f91580f56f0 2022-01-31T19:41:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.415+0000 7f915ce5a700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f914405e5c0 0x7f9144060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.416+0000 7f915ce5a700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f914405e5c0 0x7f9144060a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9148007fd0 tx=0x7f9148009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.419+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f915404c390 con 0x7f91580f56f0 2022-01-31T19:41:56.608 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.606+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f91580fdeb0 con 0x7f91580f56f0 2022-01-31T19:41:56.861 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.860+0000 7f914dffb700 1 -- 172.21.15.167:0/933583101 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7f915404f9b0 con 0x7f91580f56f0 2022-01-31T19:41:56.863 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f914405e5c0 msgr2=0x7f9144060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:56.863 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f914405e5c0 0x7f9144060a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9148007fd0 tx=0x7f9148009040).stop 2022-01-31T19:41:56.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 msgr2=0x7f91580f9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:56.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f9850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f9154007a60 tx=0x7f9154009b30).stop 2022-01-31T19:41:56.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 shutdown_connections 2022-01-31T19:41:56.864 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f914405e5c0 0x7f9144060a80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:56.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91580f56f0 0x7f91580f9850 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:56.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 --2- 172.21.15.167:0/933583101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91580f6990 0x7f91580f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:56.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 >> 172.21.15.167:0/933583101 conn(0x7f91580f0e30 msgr2=0x7f9158060170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:56.866 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 shutdown_connections 2022-01-31T19:41:56.866 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:56.862+0000 7f915f8bf700 1 -- 172.21.15.167:0/933583101 wait complete. 2022-01-31T19:41:57.081 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: cephadm 2022-01-31T19:41:55.751472+0000 mgr.smithi167.aciqpk (mgr.14156) 191 : cephadm [INF] Deploying daemon mds.cephfs.smithi171.erzfea on smithi171 2022-01-31T19:41:57.081 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: cluster 2022-01-31T19:41:55.850228+0000 mon.smithi167 (mon.0) 545 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2022-01-31T19:41:57.082 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: cluster 2022-01-31T19:41:55.850255+0000 mon.smithi167 (mon.0) 546 : cluster [INF] Cluster is now healthy 2022-01-31T19:41:57.082 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: cluster 2022-01-31T19:41:55.853523+0000 mon.smithi167 (mon.0) 547 : cluster [DBG] mds.? [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] up:boot 2022-01-31T19:41:57.082 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: cluster 2022-01-31T19:41:55.853587+0000 mon.smithi167 (mon.0) 548 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby 2022-01-31T19:41:57.083 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: audit 2022-01-31T19:41:55.853936+0000 mon.smithi167 (mon.0) 549 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.rdkazj"}]: dispatch 2022-01-31T19:41:57.083 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:56 smithi171 conmon[35325]: audit 2022-01-31T19:41:56.607714+0000 mon.smithi167 (mon.0) 550 : audit [INF] from='client.? 172.21.15.167:0/933583101' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-01-31T19:41:57.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: cephadm 2022-01-31T19:41:55.751472+0000 2022-01-31T19:41:57.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 191 : cephadm [INF] Deploying daemon mds.cephfs.smithi171.erzfea on smithi171 2022-01-31T19:41:57.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: cluster 2022-01-31 2022-01-31T19:41:57.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: T19:41:55.850228+0000 mon.smithi167 (mon.0) 545 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2022-01-31T19:41:57.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: cluster 2022-01-31 2022-01-31T19:41:57.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: T19:41:55.850255+0000 mon.smithi167 (mon.0 2022-01-31T19:41:57.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: ) 546 : cluster [INF] Cluster is now healthy 2022-01-31T19:41:57.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: cluster 2022-01-31T19:41: 2022-01-31T19:41:57.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: 55.853523+0000 mon.smithi167 (mon.0) 547 : cluster [DBG] mds.? [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] up:boot 2022-01-31T19:41:57.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: cluster 2022-01-31T19:41:55.853587+0000 2022-01-31T19:41:57.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: mon.smithi167 (mon.0) 548 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby 2022-01-31T19:41:57.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: audit 2022-01-31T19:41:55 2022-01-31T19:41:57.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: .853936+0000 mon.smithi167 (mon.0) 549 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.rdkazj"}]: dispatch 2022-01-31T19:41:57.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: audit 2022-01-31T19:41:56.607714+0000 mon.smithi167 ( 2022-01-31T19:41:57.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:56 smithi167 conmon[32206]: mon.0) 550 : audit [INF] from='client.? 172.21.15.167:0/933583101' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-01-31T19:41:57.254 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:41:57.267 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:41:57.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2022-01-31T19:41:58.002 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: cluster 2022-01-31T19:41:56.726364+0000 mgr.smithi167.aciqpk (mgr.14156) 192 : cluster [DBG] pgmap v149: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s wr, 14 op/s 2022-01-31T19:41:58.003 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: audit 2022-01-31T19:41:56.860956+0000 mon.smithi167 (mon.0) 551 : audit [INF] from='client.? 172.21.15.167:0/933583101' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-01-31T19:41:58.003 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: cluster 2022-01-31T19:41:56.861223+0000 mon.smithi167 (mon.0) 552 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby 2022-01-31T19:41:58.004 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: cluster 2022-01-31T19:41:56.865274+0000 mon.smithi167 (mon.0) 553 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby-replay 2022-01-31T19:41:58.004 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: audit 2022-01-31T19:41:57.642527+0000 mon.smithi167 (mon.0) 554 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:58.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: audit 2022-01-31T19:41:58.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: 2022-01-31T19:41:57.643442+0000 mon.smithi167 (mon.0) 555 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:41:58.006 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: audit 2022-01-31T19:41:57.644385+0000 mon.smithi167 (mon.0) 556 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:58.006 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:57 smithi167 conmon[32206]: audit 2022-01-31T19:41:57.644951+0000 mon.smithi167 (mon.0) 557 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:41:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: cluster 2022-01-31T19:41:56.726364+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:41:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: ) 192 : cluster [DBG] pgmap v149: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s wr, 14 op/s 2022-01-31T19:41:58.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: audit 2022-01-31T19:41:56.860956+0000 mon.smithi167 (mon.0) 551 : audit [INF] from='client.? 172.21.15.167:0/933583101' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-01-31T19:41:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: cluster 2022-01-31T19:41:56.861223+0000 mon.smithi167 (mon.0) 552 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby 2022-01-31T19:41:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: cluster 2022-01-31T19:41:56.865274+0000 mon.smithi167 (mon.0) 553 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby-replay 2022-01-31T19:41:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: audit 2022-01-31T19:41:57.642527+0000 mon.smithi167 (mon.0) 554 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: audit 2022-01-31T19:41:57.643442+0000 mon.smithi167 (mon.0) 555 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:41:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: audit 2022-01-31T19:41:57.644385+0000 mon.smithi167 (mon.0) 556 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:41:58.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:57 smithi171 conmon[35325]: audit 2022-01-31T19:41:57.644951+0000 mon.smithi167 (mon.0) 557 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:41:58.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 -- 172.21.15.167:0/917544650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f64d0 msgr2=0x7fb4e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:58.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/917544650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f64d0 0x7fb4e80f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fb4dc004660 tx=0x7fb4dc009b30).stop 2022-01-31T19:41:58.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 -- 172.21.15.167:0/917544650 shutdown_connections 2022-01-31T19:41:58.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/917544650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f64d0 0x7fb4e80f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:58.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/917544650 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e80f82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:58.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 -- 172.21.15.167:0/917544650 >> 172.21.15.167:0/917544650 conn(0x7fb4e80f1f70 msgr2=0x7fb4e80f4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:58.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 -- 172.21.15.167:0/917544650 shutdown_connections 2022-01-31T19:41:58.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.964+0000 7fb4ee04d700 1 -- 172.21.15.167:0/917544650 wait complete. 2022-01-31T19:41:58.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: cluster 2022-01-31T19:41:57.867433+0000 mon.smithi167 (mon.0) 2022-01-31T19:41:58.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: 558 : cluster [DBG] mds.? [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] up:boot 2022-01-31T19:41:58.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: cluster 2022-01-31T19:41:57.867756+0000 mon.smithi167 (mon.0) 559 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby-replay 1 up:standby 2022-01-31T19:41:58.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: audit 2022-01-31T19:41:57.867853+0000 mon.smithi167 (mon.0) 560 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.erzfea"}]: dispatch 2022-01-31T19:41:58.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: cluster 2022-01-31T19:41:57.871890+0000 mon.smithi167 (mon.0) 561 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:41:58.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: audit 2022-01-31T19:41:58.826389+0000 mon.smithi167 (mon.0) 562 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:41:58.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: audit 2022-01-31T19:41:58.827414+0000 mon.smithi167 (mon.0) 563 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:41:58.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:58 smithi167 conmon[32206]: audit 2022-01-31T19:41:58.852039+0000 mon.smithi167 (mon.0) 564 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:58.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4ee04d700 1 Processor -- start 2022-01-31T19:41:58.973 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4ee04d700 1 -- start start 2022-01-31T19:41:58.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4ee04d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4e80f64d0 0x7fb4e81070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:58.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4ee04d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e81075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:58.974 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4ee04d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e8107b90 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4ee04d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e8107cd0 con 0x7fb4e80f64d0 2022-01-31T19:41:58.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4e6ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e81075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:58.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4e6ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e81075e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33364/0 (socket says 172.21.15.167:33364) 2022-01-31T19:41:58.975 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4e6ffd700 1 -- 172.21.15.167:0/2916169114 learned_addr learned my addr 172.21.15.167:0/2916169114 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:41:58.976 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.965+0000 7fb4e77fe700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4e80f64d0 0x7fb4e81070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:58.976 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4e6ffd700 1 -- 172.21.15.167:0/2916169114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4e80f64d0 msgr2=0x7fb4e81070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:58.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4e6ffd700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4e80f64d0 0x7fb4e81070f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:58.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4e6ffd700 1 -- 172.21.15.167:0/2916169114 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc005040 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4e6ffd700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e81075e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb4dc000c00 tx=0x7fb4dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:58.977 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4dc019070 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.978 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb4dc007e50 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.978 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e8108610 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.978 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4dc014d00 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.978 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.966+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e8108b70 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.978 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.967+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e8045bb0 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.967+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fb4dc025070 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.968+0000 7fb4cffff700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb4d005e5c0 0x7fb4d0060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:41:58.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.968+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb4dc0799a0 con 0x7fb4e80f7ec0 2022-01-31T19:41:58.979 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.968+0000 7fb4e77fe700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb4d005e5c0 0x7fb4d0060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:41:58.980 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.968+0000 7fb4e77fe700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb4d005e5c0 0x7fb4d0060a80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fb4e80f7910 tx=0x7fb4d800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:41:58.980 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:58.972+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb4dc04b5a0 con 0x7fb4e80f7ec0 2022-01-31T19:41:59.172 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.171+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb4e8027ae0 con 0x7fb4e80f7ec0 2022-01-31T19:41:59.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: cluster 2022-01-31T19:41:57.867433+0000 mon.smithi167 (mon 2022-01-31T19:41:59.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: .0) 558 : cluster [DBG] mds.? [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] up:boot 2022-01-31T19:41:59.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: cluster 2022-01-31T19:41:57.867756+0000 mon.smithi167 (mon.0) 559 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 1 up:standby-replay 1 up:standby 2022-01-31T19:41:59.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: audit 2022-01-31T19:41:57.867853+0000 mon.smithi167 (mon.0) 560 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.erzfea"}]: dispatch 2022-01-31T19:41:59.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: cluster 2022-01-31T19:41:57.871890+0000 mon.smithi167 (mon.0) 561 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:41:59.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: audit 2022-01-31T19:41:58.826389+0000 mon.smithi167 (mon.0) 562 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:41:59.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: audit 2022-01-31T19:41:58.827414+0000 mon.smithi167 (mon.0) 563 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:41:59.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:58 smithi171 conmon[35325]: audit 2022-01-31T19:41:58.852039+0000 mon.smithi167 (mon.0) 564 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:41:59.878 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.877+0000 7fb4cffff700 1 -- 172.21.15.167:0/2916169114 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v14) v1 ==== 168+0+0 (secure 0 0 0) 0x7fb4dc04ebc0 con 0x7fb4e80f7ec0 2022-01-31T19:41:59.881 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.880+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb4d005e5c0 msgr2=0x7fb4d0060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:59.881 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.880+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb4d005e5c0 0x7fb4d0060a80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fb4e80f7910 tx=0x7fb4d800b040).stop 2022-01-31T19:41:59.882 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.880+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 msgr2=0x7fb4e81075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:41:59.882 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.880+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e81075e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb4dc000c00 tx=0x7fb4dc00b040).stop 2022-01-31T19:41:59.883 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 shutdown_connections 2022-01-31T19:41:59.883 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb4d005e5c0 0x7fb4d0060a80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:59.884 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4e80f64d0 0x7fb4e81070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:59.884 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 --2- 172.21.15.167:0/2916169114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4e80f7ec0 0x7fb4e81075e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:41:59.884 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 >> 172.21.15.167:0/2916169114 conn(0x7fb4e80f1f70 msgr2=0x7fb4e80fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:41:59.884 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 shutdown_connections 2022-01-31T19:41:59.885 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:41:59.881+0000 7fb4ee04d700 1 -- 172.21.15.167:0/2916169114 wait complete. 2022-01-31T19:41:59.885 INFO:teuthology.orchestra.run.smithi167.stderr:inline data enabled 2022-01-31T19:42:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:59 smithi167 conmon[32206]: cluster 2022-01-31T19:41:58.726819+0000 mgr.smithi167.aciqpk (mgr.14156) 193 : cluster [DBG] pgmap v150: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 3.7 KiB/s wr, 13 op/s 2022-01-31T19:42:00.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:41:59 smithi167 conmon[32206]: audit 2022-01-31T19:41:59.172357+0000 mon.smithi167 (mon.0) 565 : audit [INF] from='client.? 172.21.15.167:0/2916169114' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2022-01-31T19:42:00.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:59 smithi171 conmon[35325]: cluster 2022-01-31T19:41:58.726819+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:42:00.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:59 smithi171 conmon[35325]: ) 193 : cluster [DBG] pgmap v150: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 3.7 KiB/s wr, 13 op/s 2022-01-31T19:42:00.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:59 smithi171 conmon[35325]: audit 2022-01-31T19:41:59.172357+0000 mon.smithi167 (mon.0) 565 : audit [INF] 2022-01-31T19:42:00.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:41:59 smithi171 conmon[35325]: from='client.? 172.21.15.167:0/2916169114' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2022-01-31T19:42:00.399 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:42:00.417 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:42:00.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs dump' 2022-01-31T19:42:01.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:00 smithi167 conmon[32206]: cluster 2022-01-31T19:41:59.871421+0000 mon.smithi167 2022-01-31T19:42:01.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:00 smithi167 conmon[32206]: (mon.0) 566 : cluster [WRN] Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2022-01-31T19:42:01.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:00 smithi167 conmon[32206]: audit 2022-01-31T19:41:59.877772+0000 mon.smithi167 (mon.0) 567 : audit [INF] from='client.? 172.21.15.167:0/2916169114' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2022-01-31T19:42:01.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:00 smithi167 conmon[32206]: cluster 2022-01-31T19:41:59.877851+0000 mon.smithi167 (mon.0) 568 : cluster [DBG] mds.? [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] up:standby-replay 2022-01-31T19:42:01.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:00 smithi167 conmon[32206]: cluster 2022-01-31T19:41:59.877911+0000 mon.smithi167 (mon.0) 569 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:42:01.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:00 smithi167 conmon[32206]: audit 2022-01-31T19:42:00.509260+0000 mon.smithi167 (mon.0) 570 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:01.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: cluster 2022-01-31T19:41:59.871421+0000 mon.smithi167 (mon.0) 2022-01-31T19:42:01.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: 566 : cluster [WRN] Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2022-01-31T19:42:01.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: audit 2022-01-31T19:41:59.877772+0000 mon.smithi167 (mon.0) 567 : audit [INF] from='client.? 172.21.15.167:0/2916169114' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2022-01-31T19:42:01.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: cluster 2022-01-31T19:41:59.877851+0000 mon.smithi167 (mon.0) 568 : cluster [DBG] mds.? [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] up:standby-replay 2022-01-31T19:42:01.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: cluster 2022-01-31T19:41:59.877911+0000 mon.smithi167 (mon.0) 569 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:42:01.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: audit 2022-01-31T19:42:00 2022-01-31T19:42:01.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:00 smithi171 conmon[35325]: .509260+0000 mon.smithi167 (mon.0) 570 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:02.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:01 smithi167 conmon[32206]: cluster 2022-01-31T19:42:00.727330+0000 mgr.smithi167.aciqpk (mgr.14156) 194 : cluster [DBG] pgmap v151: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2022-01-31T19:42:02.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:01 smithi171 conmon[35325]: cluster 2022-01-31T19:42:00.727330+0000 mgr.smithi167.aciqpk (mgr.14156) 194 : cluster 2022-01-31T19:42:02.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:01 smithi171 conmon[35325]: [DBG] pgmap v151: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2022-01-31T19:42:02.477 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.474+0000 7fca6ea5c700 1 -- 172.21.15.167:0/2136745659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f7a80 msgr2=0x7fca680f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:02.478 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.474+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/2136745659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f7a80 0x7fca680f7ee0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fca5c004660 tx=0x7fca5c009b30).stop 2022-01-31T19:42:02.478 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- 172.21.15.167:0/2136745659 shutdown_connections 2022-01-31T19:42:02.478 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/2136745659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f7a80 0x7fca680f7ee0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:02.479 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/2136745659 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca680f67e0 0x7fca680f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:02.479 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- 172.21.15.167:0/2136745659 >> 172.21.15.167:0/2136745659 conn(0x7fca680f1f20 msgr2=0x7fca680f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:02.479 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- 172.21.15.167:0/2136745659 shutdown_connections 2022-01-31T19:42:02.480 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- 172.21.15.167:0/2136745659 wait complete. 2022-01-31T19:42:02.480 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 Processor -- start 2022-01-31T19:42:02.480 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- start start 2022-01-31T19:42:02.480 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 0x7fca68107070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:02.480 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca680f7a80 0x7fca68107560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:02.481 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca68107b10 con 0x7fca680f67e0 2022-01-31T19:42:02.481 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.475+0000 7fca6ea5c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca68107c50 con 0x7fca680f7a80 2022-01-31T19:42:02.481 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 0x7fca68107070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:02.481 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 0x7fca68107070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33374/0 (socket says 172.21.15.167:33374) 2022-01-31T19:42:02.482 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 -- 172.21.15.167:0/1826902215 learned_addr learned my addr 172.21.15.167:0/1826902215 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:02.482 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca677fe700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca680f7a80 0x7fca68107560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:02.482 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 -- 172.21.15.167:0/1826902215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca680f7a80 msgr2=0x7fca68107560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:02.482 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca680f7a80 0x7fca68107560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:02.483 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 -- 172.21.15.167:0/1826902215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c005040 con 0x7fca680f67e0 2022-01-31T19:42:02.483 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca67fff700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 0x7fca68107070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fca5800c530 tx=0x7fca58007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:02.483 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca5800f650 con 0x7fca680f67e0 2022-01-31T19:42:02.483 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fca5800fc40 con 0x7fca680f67e0 2022-01-31T19:42:02.484 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca681085f0 con 0x7fca680f67e0 2022-01-31T19:42:02.484 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca58012c90 con 0x7fca680f67e0 2022-01-31T19:42:02.484 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.476+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca68108bb0 con 0x7fca680f67e0 2022-01-31T19:42:02.485 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.477+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca68045bb0 con 0x7fca680f67e0 2022-01-31T19:42:02.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.478+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fca58019030 con 0x7fca680f67e0 2022-01-31T19:42:02.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.478+0000 7fca64ff9700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fca5005e5c0 0x7fca50060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:02.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.478+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fca58078540 con 0x7fca680f67e0 2022-01-31T19:42:02.486 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.478+0000 7fca677fe700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fca5005e5c0 0x7fca50060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:02.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.478+0000 7fca677fe700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fca5005e5c0 0x7fca50060a80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fca5c009d60 tx=0x7fca5c009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:02.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.482+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fca5804a190 con 0x7fca680f67e0 2022-01-31T19:42:02.687 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.686+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fca680ff0b0 con 0x7fca680f67e0 2022-01-31T19:42:02.688 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.686+0000 7fca64ff9700 1 -- 172.21.15.167:0/1826902215 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1672 (secure 0 0 0) 0x7fca580176a0 con 0x7fca680f67e0 2022-01-31T19:42:02.688 INFO:teuthology.orchestra.run.smithi167.stdout:e15 2022-01-31T19:42:02.688 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:42:02.689 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:42:02.689 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:42:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:02.689 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:42:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:42:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 15 2022-01-31T19:42:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:42:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:42:02.691 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:42:01.874180+0000 2022-01-31T19:42:02.691 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:42:02.691 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:42:02.691 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:42:02.691 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:42:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:42:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:42:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:42:02.692 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:42:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:42:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:42:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:42:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:42:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:42:02.694 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:42:02.694 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:42:02.694 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:42:02.694 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:42:02.695 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:42:02.695 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:42:02.695 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:42:02.695 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:42:02.695 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:42:02.696 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:42:02.696 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:42:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:02.696 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:02.697 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fca5005e5c0 msgr2=0x7fca50060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:02.697 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fca5005e5c0 0x7fca50060a80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fca5c009d60 tx=0x7fca5c009b30).stop 2022-01-31T19:42:02.698 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 msgr2=0x7fca68107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:02.698 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 0x7fca68107070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fca5800c530 tx=0x7fca58007e10).stop 2022-01-31T19:42:02.698 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 shutdown_connections 2022-01-31T19:42:02.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fca5005e5c0 0x7fca50060a80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:02.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca680f67e0 0x7fca68107070 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:02.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 --2- 172.21.15.167:0/1826902215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca680f7a80 0x7fca68107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:02.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.688+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 >> 172.21.15.167:0/1826902215 conn(0x7fca680f1f20 msgr2=0x7fca680fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:02.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.689+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 shutdown_connections 2022-01-31T19:42:02.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:02.689+0000 7fca6ea5c700 1 -- 172.21.15.167:0/1826902215 wait complete. 2022-01-31T19:42:02.700 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:03.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:02 smithi167 conmon[32206]: cluster 2022-01-31T19:42:01.884695+0000 mon.smithi167 (mon.0) 2022-01-31T19:42:03.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:02 smithi167 conmon[32206]: 571 : cluster [DBG] mds.? [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] up:standby-replay 2022-01-31T19:42:03.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:02 smithi167 conmon[32206]: cluster 2022-01-31T19:42:01.884753+0000 mon.smithi167 (mon.0) 572 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:42:03.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:02 smithi167 conmon[32206]: audit 2022-01-31T19:42:02.687177+0000 mon.smithi167 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.167:0/1826902215' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:42:03.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2022-01-31T19:42:03.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:02 smithi171 conmon[35325]: cluster 2022-01-31T19:42:01.884695+0000 mon.smithi167 (mon.0) 571 : cluster [DBG] mds.? [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] up:standby-replay 2022-01-31T19:42:03.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:02 smithi171 conmon[35325]: cluster 2022-01-31T19:42:01.884753+0000 mon.smithi167 (mon.0) 572 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:42:03.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:02 smithi171 conmon[35325]: audit 2022-01-31T19:42:02.687177+0000 mon.smithi167 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.167:0/1826902215' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:42:04.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:03 smithi167 conmon[32206]: cluster 2022-01-31T19:42:02.727788+0000 mgr.smithi167.aciqpk (mgr.14156) 195 : cluster [DBG] pgmap v152: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2022-01-31T19:42:04.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:03 smithi171 conmon[35325]: cluster 2022-01-31T19:42:02.727788+0000 mgr.smithi167.aciqpk (mgr.14156) 195 : cluster [DBG] pgmap v152: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2022-01-31T19:42:04.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 -- 172.21.15.167:0/1031213474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 msgr2=0x7f22500f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:04.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 --2- 172.21.15.167:0/1031213474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f6de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2240004660 tx=0x7f2240009b30).stop 2022-01-31T19:42:04.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 -- 172.21.15.167:0/1031213474 shutdown_connections 2022-01-31T19:42:04.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 --2- 172.21.15.167:0/1031213474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f6de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:04.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 --2- 172.21.15.167:0/1031213474 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22500f56e0 0x7f22500f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:04.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 -- 172.21.15.167:0/1031213474 >> 172.21.15.167:0/1031213474 conn(0x7f22500f0e60 msgr2=0x7f22500f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:04.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 -- 172.21.15.167:0/1031213474 shutdown_connections 2022-01-31T19:42:04.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 -- 172.21.15.167:0/1031213474 wait complete. 2022-01-31T19:42:04.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.838+0000 7f22566a3700 1 Processor -- start 2022-01-31T19:42:04.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f22566a3700 1 -- start start 2022-01-31T19:42:04.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f22566a3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22500f56e0 0x7f22500f9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:04.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f22566a3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:04.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f22566a3700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22500f8220 con 0x7f22500f6980 2022-01-31T19:42:04.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f22566a3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22500f8360 con 0x7f22500f56e0 2022-01-31T19:42:04.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224f7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:04.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224f7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33398/0 (socket says 172.21.15.167:33398) 2022-01-31T19:42:04.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224f7fe700 1 -- 172.21.15.167:0/1540983367 learned_addr learned my addr 172.21.15.167:0/1540983367 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:04.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224ffff700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22500f56e0 0x7f22500f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:04.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224f7fe700 1 -- 172.21.15.167:0/1540983367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22500f56e0 msgr2=0x7f22500f9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:04.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224f7fe700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22500f56e0 0x7f22500f9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:04.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.839+0000 7f224f7fe700 1 -- 172.21.15.167:0/1540983367 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240005040 con 0x7f22500f6980 2022-01-31T19:42:04.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.840+0000 7f224f7fe700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f7cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f2240009990 tx=0x7f2240009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:04.847 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.840+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2240018070 con 0x7f22500f6980 2022-01-31T19:42:04.847 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.840+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22500f85e0 con 0x7f22500f6980 2022-01-31T19:42:04.847 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.840+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2240013410 con 0x7f22500f6980 2022-01-31T19:42:04.847 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.840+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f224001de80 con 0x7f22500f6980 2022-01-31T19:42:04.848 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.840+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22500f8b40 con 0x7f22500f6980 2022-01-31T19:42:04.849 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.841+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2250045bb0 con 0x7f22500f6980 2022-01-31T19:42:04.849 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.842+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f224001d420 con 0x7f22500f6980 2022-01-31T19:42:04.849 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.842+0000 7f224cff9700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f223c05e5c0 0x7f223c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:04.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.842+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f224007a920 con 0x7f22500f6980 2022-01-31T19:42:04.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.842+0000 7f224ffff700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f223c05e5c0 0x7f223c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:04.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.843+0000 7f224ffff700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f223c05e5c0 0x7f223c060a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f22500f95d0 tx=0x7f223800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:04.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:04.846+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f224004c5f0 con 0x7f22500f6980 2022-01-31T19:42:05.060 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.059+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22500fdea0 con 0x7f22500f6980 2022-01-31T19:42:05.061 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.059+0000 7f224cff9700 1 -- 172.21.15.167:0/1540983367 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3041 (secure 0 0 0) 0x7f224004fc10 con 0x7f22500f6980 2022-01-31T19:42:05.062 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f223c05e5c0 msgr2=0x7f223c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:05.062 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f223c05e5c0 0x7f223c060a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f22500f95d0 tx=0x7f223800b040).stop 2022-01-31T19:42:05.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 msgr2=0x7f22500f7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:05.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f7cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f2240009990 tx=0x7f2240009b30).stop 2022-01-31T19:42:05.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 shutdown_connections 2022-01-31T19:42:05.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f223c05e5c0 0x7f223c060a80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:05.064 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22500f56e0 0x7f22500f9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:05.064 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 --2- 172.21.15.167:0/1540983367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22500f6980 0x7f22500f7cc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:05.064 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 >> 172.21.15.167:0/1540983367 conn(0x7f22500f0e60 msgr2=0x7f225005ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:05.064 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 shutdown_connections 2022-01-31T19:42:05.064 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:05.061+0000 7f22566a3700 1 -- 172.21.15.167:0/1540983367 wait complete. 2022-01-31T19:42:05.065 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:05.073 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:42:05.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:04 smithi167 conmon[32206]: audit 2022-01-31T19:42:03.947715+0000 mon.smithi167 (mon.0) 574 : 2022-01-31T19:42:05.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:04 smithi167 conmon[32206]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:05.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:04 smithi167 conmon[32206]: audit 2022-01-31T19:42:04.457672+0000 mon.smithi167 (mon.0) 575 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:05.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:04 smithi167 conmon[32206]: audit 2022-01-31T19:42:04.461728+0000 mon.smithi167 (mon.0) 576 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:42:05.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:04 smithi167 conmon[32206]: audit 2022-01-31T19:42:04.462731+0000 mon.smithi167 (mon.0) 577 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:42:05.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:04 smithi167 conmon[32206]: audit 2022-01-31T19:42:04.463300+0000 mon.smithi167 (mon.0) 578 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:42:05.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: audit 2022-01-31T19:42:03.947715+0000 mon.smithi167 (mon.0) 574 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:05.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: audit 2022-01-31T19:42 2022-01-31T19:42:05.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: :04.457672+0000 mon.smithi167 (mon.0) 575 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:05.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: audit 2022-01-31T19:42:04.461728+0000 mon.smithi167 (mon.0) 576 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:42:05.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: audit 2022 2022-01-31T19:42:05.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: -01-31T19:42:04.462731+0000 mon.smithi167 (mon.0) 577 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:42:05.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: 2022-01-31T19:42:05.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: audit 2022-01-31T19:42:05.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: 2022-01-31T19:42:04.463300+0000 mon.smithi167 (mon 2022-01-31T19:42:05.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:04 smithi171 conmon[35325]: .0) 578 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:42:05.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2022-01-31T19:42:06.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:05 smithi167 conmon[32206]: cluster 2022-01-31T19:42:04.728329+0000 mgr.smithi167.aciqpk (mgr.14156) 196 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2022-01-31T19:42:06.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:05 smithi167 conmon[32206]: audit 2022-01-31T19:42:05.059820+0000 mon.smithi167 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.167:0/1540983367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:06.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:05 smithi171 conmon[35325]: cluster 2022-01-31T19:42:04.728329+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:42:06.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:05 smithi171 conmon[35325]: mgr.14156) 196 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2022-01-31T19:42:06.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:05 smithi171 conmon[35325]: audit 2022-01-31T19:42:05.059820+0000 mon.smithi167 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.167:0/1540983367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:07.225 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 -- 172.21.15.167:0/983210248 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 msgr2=0x7f8c0c0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:07.225 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/983210248 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f6de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f8bfc004660 tx=0x7f8bfc009b30).stop 2022-01-31T19:42:07.225 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 -- 172.21.15.167:0/983210248 shutdown_connections 2022-01-31T19:42:07.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/983210248 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f6de0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:07.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/983210248 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 0x7f8c0c0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:07.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 -- 172.21.15.167:0/983210248 >> 172.21.15.167:0/983210248 conn(0x7f8c0c0f0e60 msgr2=0x7f8c0c0f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:07.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 -- 172.21.15.167:0/983210248 shutdown_connections 2022-01-31T19:42:07.227 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.221+0000 7f8c12b6d700 1 -- 172.21.15.167:0/983210248 wait complete. 2022-01-31T19:42:07.227 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c12b6d700 1 Processor -- start 2022-01-31T19:42:07.227 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c12b6d700 1 -- start start 2022-01-31T19:42:07.227 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c12b6d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 0x7f8c0c0fa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:07.228 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c12b6d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:07.228 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c12b6d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0f8a40 con 0x7f8c0c0f6980 2022-01-31T19:42:07.228 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c12b6d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0f8b80 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.228 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c0bfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:07.229 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c10909700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 0x7f8c0c0fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:07.229 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c0bfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33412/0 (socket says 172.21.15.167:33412) 2022-01-31T19:42:07.229 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.222+0000 7f8c0bfff700 1 -- 172.21.15.167:0/3536957137 learned_addr learned my addr 172.21.15.167:0/3536957137 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:07.229 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c10909700 1 -- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 msgr2=0x7f8c0c0f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:07.230 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c10909700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:07.230 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c10909700 1 -- 172.21.15.167:0/3536957137 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc005040 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.230 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c10909700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 0x7f8c0c0fa060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f8c0c0f67e0 tx=0x7f8bf400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:07.230 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf40092e0 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c0f8e60 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8bf400f040 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf4014750 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.223+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c10f030 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.232 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.224+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c045bb0 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.224+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f8bf401b030 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.225+0000 7f8c097fa700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8bf805e5c0 0x7f8bf8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:07.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.225+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8bf4078750 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.234 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.225+0000 7f8c0bfff700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8bf805e5c0 0x7f8bf8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:07.234 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.225+0000 7f8c0bfff700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8bf805e5c0 0x7f8bf8060a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8bfc005010 tx=0x7f8bfc016040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:07.234 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.229+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8bf404a350 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.453+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f8c0c10f740 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.454 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.453+0000 7f8c097fa700 1 -- 172.21.15.167:0/3536957137 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+86 (secure 0 0 0) 0x7f8bf4019090 con 0x7f8c0c0f56e0 2022-01-31T19:42:07.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8bf805e5c0 msgr2=0x7f8bf8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:07.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8bf805e5c0 0x7f8bf8060a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8bfc005010 tx=0x7f8bfc016040).stop 2022-01-31T19:42:07.457 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 msgr2=0x7f8c0c0fa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:07.458 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 0x7f8c0c0fa060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f8c0c0f67e0 tx=0x7f8bf400b040).stop 2022-01-31T19:42:07.458 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 shutdown_connections 2022-01-31T19:42:07.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8bf805e5c0 0x7f8bf8060a80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:07.459 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c0c0f56e0 0x7f8c0c0fa060 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:07.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 --2- 172.21.15.167:0/3536957137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c0c0f6980 0x7f8c0c0f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:07.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 >> 172.21.15.167:0/3536957137 conn(0x7f8c0c0f0e60 msgr2=0x7f8c0c05f700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:07.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 shutdown_connections 2022-01-31T19:42:07.460 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:07.455+0000 7f8c12b6d700 1 -- 172.21.15.167:0/3536957137 wait complete. 2022-01-31T19:42:07.466 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:42:07.808 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2022-01-31T19:42:07.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-01-31T19:42:08.037 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.033+0000 7fe197dd2700 1 -- 172.21.15.167:0/3807164385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f5af0 msgr2=0x7fe1900f5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.038 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.033+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3807164385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f5af0 0x7fe1900f5f10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe18c004660 tx=0x7fe18c009b30).stop 2022-01-31T19:42:08.038 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.033+0000 7fe197dd2700 1 -- 172.21.15.167:0/3807164385 shutdown_connections 2022-01-31T19:42:08.038 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.033+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3807164385 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1900f6d90 0x7fe1900f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.033+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3807164385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f5af0 0x7fe1900f5f10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.033+0000 7fe197dd2700 1 -- 172.21.15.167:0/3807164385 >> 172.21.15.167:0/3807164385 conn(0x7fe1900f1270 msgr2=0x7fe1900f3690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 -- 172.21.15.167:0/3807164385 shutdown_connections 2022-01-31T19:42:08.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 -- 172.21.15.167:0/3807164385 wait complete. 2022-01-31T19:42:08.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 Processor -- start 2022-01-31T19:42:08.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 -- start start 2022-01-31T19:42:08.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1900f5af0 0x7fe190106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 0x7fe190106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe190106cb0 con 0x7fe1900f6d90 2022-01-31T19:42:08.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe197dd2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe190106df0 con 0x7fe1900f5af0 2022-01-31T19:42:08.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe19536d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 0x7fe190106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe19536d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 0x7fe190106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33426/0 (socket says 172.21.15.167:33426) 2022-01-31T19:42:08.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.034+0000 7fe19536d700 1 -- 172.21.15.167:0/3099464645 learned_addr learned my addr 172.21.15.167:0/3099464645 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:08.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe195b6e700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1900f5af0 0x7fe190106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe19536d700 1 -- 172.21.15.167:0/3099464645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1900f5af0 msgr2=0x7fe190106210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe19536d700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1900f5af0 0x7fe190106210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe19536d700 1 -- 172.21.15.167:0/3099464645 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c005040 con 0x7fe1900f6d90 2022-01-31T19:42:08.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe19536d700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 0x7fe190106700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fe18000e7f0 tx=0x7fe18000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe180004570 con 0x7fe1900f6d90 2022-01-31T19:42:08.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe18000f040 con 0x7fe1900f6d90 2022-01-31T19:42:08.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19010b890 con 0x7fe1900f6d90 2022-01-31T19:42:08.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe180014830 con 0x7fe1900f6d90 2022-01-31T19:42:08.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.035+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19010be50 con 0x7fe1900f6d90 2022-01-31T19:42:08.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.036+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe190027ea0 con 0x7fe1900f6d90 2022-01-31T19:42:08.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.036+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fe1800077b0 con 0x7fe1900f6d90 2022-01-31T19:42:08.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.037+0000 7fe1867fc700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe17c05e5c0 0x7fe17c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.037+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe180078a40 con 0x7fe1900f6d90 2022-01-31T19:42:08.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.037+0000 7fe195b6e700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe17c05e5c0 0x7fe17c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.037+0000 7fe195b6e700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe17c05e5c0 0x7fe17c060a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe18c005010 tx=0x7fe18c00b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.049 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.041+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe1800476b0 con 0x7fe1900f6d90 2022-01-31T19:42:08.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.207+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe190058d10 con 0x7fe1900f6d90 2022-01-31T19:42:08.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.208+0000 7fe1867fc700 1 -- 172.21.15.167:0/3099464645 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+13869 (secure 0 0 0) 0x7fe18004a640 con 0x7fe1900f6d90 2022-01-31T19:42:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:08.210 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":40,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","created":"2022-01-31T19:37:06.503417+0000","modified":"2022-01-31T19:41:50.817316+0000","last_up_change":"2022-01-31T19:40:51.787533+0000","last_in_change":"2022-01-31T19:40:36.089120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-01-31T19:40:13.386038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2022-01-31T19:41:47.179870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2022-01-31T19:41:48.033371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b39ea02-b5a4-4a19-bd99-849c0378fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6803","nonce":2333495797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6805","nonce":2333495797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6809","nonce":2333495797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6807","nonce":2333495797}]},"public_addr":"172.21.15.167:6803/2333495797","cluster_addr":"172.21.15.167:6805/2333495797","heartbeat_back_addr":"172.21.15.167:6809/2333495797","heartbeat_front_addr":"172.21.15.167:6807/2333495797","state":["exists","up"]},{"osd":1,"uuid":"214aeaf2-f8c7-4638-96b6-4e6bb4bd802f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6811","nonce":3267421031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6813","nonce":3267421031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6817","nonce":3267421031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6815","nonce":3267421031}]},"public_addr":"172.21.15.167:6811/3267421031","cluster_addr":"172.21.15.167:6813/3267421031","heartbeat_back_addr":"172.21.15.167:6817/3267421031","heartbeat_front_addr":"172.21.15.167:6815/3267421031","state":["exists","up"]},{"osd":2,"uuid":"3536e17c-9f9a-4867-91f4-f84b2b5d53a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6819","nonce":2914232381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6821","nonce":2914232381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6825","nonce":2914232381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6823","nonce":2914232381}]},"public_addr":"172.21.15.167:6819/2914232381","cluster_addr":"172.21.15.167:6821/2914232381","heartbeat_back_addr":"172.21.15.167:6825/2914232381","heartbeat_front_addr":"172.21.15.167:6823/2914232381","state":["exists","up"]},{"osd":3,"uuid":"e924c508-4baf-4ed0-9a32-42a51710ca8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6801","nonce":1700293141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6803","nonce":1700293141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6807","nonce":1700293141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6805","nonce":1700293141}]},"public_addr":"172.21.15.171:6801/1700293141","cluster_addr":"172.21.15.171:6803/1700293141","heartbeat_back_addr":"172.21.15.171:6807/1700293141","heartbeat_front_addr":"172.21.15.171:6805/1700293141","state":["exists","up"]},{"osd":4,"uuid":"c4310782-5c0e-4962-b32e-45e28ac014ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6809","nonce":3746074472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6811","nonce":3746074472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6815","nonce":3746074472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6813","nonce":3746074472}]},"public_addr":"172.21.15.171:6809/3746074472","cluster_addr":"172.21.15.171:6811/3746074472","heartbeat_back_addr":"172.21.15.171:6815/3746074472","heartbeat_front_addr":"172.21.15.171:6813/3746074472","state":["exists","up"]},{"osd":5,"uuid":"2ef7b4c2-8076-4866-b26a-8027fd249d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6817","nonce":1956867870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6819","nonce":1956867870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6823","nonce":1956867870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6821","nonce":1956867870}]},"public_addr":"172.21.15.171:6817/1956867870","cluster_addr":"172.21.15.171:6819/1956867870","heartbeat_back_addr":"172.21.15.171:6823/1956867870","heartbeat_front_addr":"172.21.15.171:6821/1956867870","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:26.422630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:45.101018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:03.040927+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:18.082335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:33.822236+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:50.200473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2480743863":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6801/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2306410502":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6800/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2760109630":"2022-02-01T19:37:34.139403+0000","172.21.15.167:0/694084204":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6800/3497254897":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6801/3497254897":"2022-02-01T19:37:34.139403+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-31T19:42:08.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe17c05e5c0 msgr2=0x7fe17c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe17c05e5c0 0x7fe17c060a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe18c005010 tx=0x7fe18c00b040).stop 2022-01-31T19:42:08.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 msgr2=0x7fe190106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 0x7fe190106700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fe18000e7f0 tx=0x7fe18000b040).stop 2022-01-31T19:42:08.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 shutdown_connections 2022-01-31T19:42:08.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe17c05e5c0 0x7fe17c060a80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1900f5af0 0x7fe190106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 --2- 172.21.15.167:0/3099464645 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1900f6d90 0x7fe190106700 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 >> 172.21.15.167:0/3099464645 conn(0x7fe1900f1270 msgr2=0x7fe1900fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 shutdown_connections 2022-01-31T19:42:08.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.210+0000 7fe197dd2700 1 -- 172.21.15.167:0/3099464645 wait complete. 2022-01-31T19:42:08.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-01-31T19:40:13.386038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2022-01-31T19:41:47.179870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2022-01-31T19:41:48.033371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-01-31T19:42:08.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-01-31T19:42:08.365 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.360+0000 7f14789a3700 1 -- 172.21.15.167:0/608558739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 msgr2=0x7f14740f7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.365 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.360+0000 7f14789a3700 1 --2- 172.21.15.167:0/608558739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f14740f7000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f1464004660 tx=0x7f1464009b30).stop 2022-01-31T19:42:08.366 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- 172.21.15.167:0/608558739 shutdown_connections 2022-01-31T19:42:08.366 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 --2- 172.21.15.167:0/608558739 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14740f7e80 0x7f14740f82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.366 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 --2- 172.21.15.167:0/608558739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f14740f7000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.366 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- 172.21.15.167:0/608558739 >> 172.21.15.167:0/608558739 conn(0x7f14740f2340 msgr2=0x7f14740f4760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.366 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- 172.21.15.167:0/608558739 shutdown_connections 2022-01-31T19:42:08.367 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- 172.21.15.167:0/608558739 wait complete. 2022-01-31T19:42:08.367 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 Processor -- start 2022-01-31T19:42:08.367 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- start start 2022-01-31T19:42:08.367 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f1474107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.368 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14740f7e80 0x7f1474107840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.368 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474107df0 con 0x7f14740f6be0 2022-01-31T19:42:08.368 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f14789a3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474107f30 con 0x7f14740f7e80 2022-01-31T19:42:08.368 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f147259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f1474107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.369 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f147259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f1474107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33434/0 (socket says 172.21.15.167:33434) 2022-01-31T19:42:08.369 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.361+0000 7f147259c700 1 -- 172.21.15.167:0/2393432485 learned_addr learned my addr 172.21.15.167:0/2393432485 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:08.369 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f1471d9b700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14740f7e80 0x7f1474107840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.369 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f147259c700 1 -- 172.21.15.167:0/2393432485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14740f7e80 msgr2=0x7f1474107840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.370 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f147259c700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14740f7e80 0x7f1474107840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.370 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f147259c700 1 -- 172.21.15.167:0/2393432485 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464005040 con 0x7f14740f6be0 2022-01-31T19:42:08.370 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f147259c700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f1474107350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1464007a60 tx=0x7f1464009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.370 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1464018070 con 0x7f14740f6be0 2022-01-31T19:42:08.371 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147410c970 con 0x7f14740f6be0 2022-01-31T19:42:08.371 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147410cf30 con 0x7f14740f6be0 2022-01-31T19:42:08.371 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1464013410 con 0x7f14740f6be0 2022-01-31T19:42:08.371 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.362+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146401c440 con 0x7f14740f6be0 2022-01-31T19:42:08.372 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.363+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1458004fd0 con 0x7f14740f6be0 2022-01-31T19:42:08.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.363+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f146401d420 con 0x7f14740f6be0 2022-01-31T19:42:08.373 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.364+0000 7f1462ffd700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f145c05e5c0 0x7f145c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.364+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f146407a810 con 0x7f14740f6be0 2022-01-31T19:42:08.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.364+0000 7f1471d9b700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f145c05e5c0 0x7f145c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.364+0000 7f1471d9b700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f145c05e5c0 0x7f145c060a80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1468007fd0 tx=0x7f1468009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.374 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.368+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f146404c410 con 0x7f14740f6be0 2022-01-31T19:42:08.535 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.533+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1458005c20 con 0x7f14740f6be0 2022-01-31T19:42:08.536 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.533+0000 7f1462ffd700 1 -- 172.21.15.167:0/2393432485 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v40) v1 ==== 110+0+10 (secure 0 0 0) 0x7f146404fa30 con 0x7f14740f6be0 2022-01-31T19:42:08.536 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 1 2022-01-31T19:42:08.537 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f145c05e5c0 msgr2=0x7f145c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.538 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f145c05e5c0 0x7f145c060a80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1468007fd0 tx=0x7f1468009040).stop 2022-01-31T19:42:08.538 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 msgr2=0x7f1474107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.538 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f1474107350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1464007a60 tx=0x7f1464009b30).stop 2022-01-31T19:42:08.538 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 shutdown_connections 2022-01-31T19:42:08.539 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f145c05e5c0 0x7f145c060a80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.539 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14740f6be0 0x7f1474107350 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.539 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.535+0000 7f14789a3700 1 --2- 172.21.15.167:0/2393432485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14740f7e80 0x7f1474107840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.539 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.536+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 >> 172.21.15.167:0/2393432485 conn(0x7f14740f2340 msgr2=0x7f14740fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.540 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.536+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 shutdown_connections 2022-01-31T19:42:08.540 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.536+0000 7f14789a3700 1 -- 172.21.15.167:0/2393432485 wait complete. 2022-01-31T19:42:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:08 smithi167 conmon[32206]: cluster 2022-01-31T19:42:06.728717+0000 mgr.smithi167.aciqpk (mgr.14156) 197 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 938 B/s wr, 5 op/s 2022-01-31T19:42:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:08 smithi167 conmon[32206]: audit 2022-01-31T19:42:07.336132+0000 mon.smithi167 (mon.0) 580 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:08 smithi167 conmon[32206]: audit 2022-01-31T19:42:07.454067+0000 mon.smithi171 (mon.1) 15 : audit [DBG] from='client.? 172.21.15.167:0/3536957137' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2022-01-31T19:42:08.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:08 smithi167 conmon[32206]: audit 2022-01-31T19:42:08.209057+0000 mon.smithi167 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.167:0/3099464645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:42:08.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2022-01-31T19:42:08.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:08 smithi171 conmon[35325]: cluster 2022-01-31T19:42:06.728717+0000 mgr.smithi167.aciqpk (mgr.14156) 197 : cluster [DBG] 2022-01-31T19:42:08.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:08 smithi171 conmon[35325]: pgmap v154: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 938 B/s wr, 5 op/s 2022-01-31T19:42:08.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:08 smithi171 conmon[35325]: audit 2022-01-31T19:42:07.336132+0000 mon.smithi167 (mon.0) 580 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:08.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:08 smithi171 conmon[35325]: audit 2022-01-31T19:42:07.454067+0000 mon.smithi171 (mon.1) 15 : audit [DBG] from='client.? 172.21.15.167:0/3536957137' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2022-01-31T19:42:08.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:08 smithi171 conmon[35325]: audit 2022-01-31T19:42:08.209057+0000 mon.smithi167 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.167:0/3099464645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:42:08.678 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/4173428253 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f5b30 msgr2=0x7fe5d80f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.678 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/4173428253 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f5b30 0x7fe5d80f5f50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fe5d4004660 tx=0x7fe5d4009b30).stop 2022-01-31T19:42:08.679 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/4173428253 shutdown_connections 2022-01-31T19:42:08.679 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/4173428253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5d80f6440 0x7fe5d80f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.679 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/4173428253 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f5b30 0x7fe5d80f5f50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.680 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/4173428253 >> 172.21.15.167:0/4173428253 conn(0x7fe5d80f12e0 msgr2=0x7fe5d80f3700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.680 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/4173428253 shutdown_connections 2022-01-31T19:42:08.681 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.673+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/4173428253 wait complete. 2022-01-31T19:42:08.681 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dfc9f700 1 Processor -- start 2022-01-31T19:42:08.681 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dfc9f700 1 -- start start 2022-01-31T19:42:08.681 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dfc9f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5d80f5b30 0x7fe5d8107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.682 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dfc9f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 0x7fe5d8107860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.682 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dfc9f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8107e10 con 0x7fe5d80f6440 2022-01-31T19:42:08.682 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dfc9f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8107f50 con 0x7fe5d80f5b30 2022-01-31T19:42:08.682 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dd23a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 0x7fe5d8107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.683 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dd23a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 0x7fe5d8107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33444/0 (socket says 172.21.15.167:33444) 2022-01-31T19:42:08.683 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dd23a700 1 -- 172.21.15.167:0/927368945 learned_addr learned my addr 172.21.15.167:0/927368945 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:08.683 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dda3b700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5d80f5b30 0x7fe5d8107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.683 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dd23a700 1 -- 172.21.15.167:0/927368945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5d80f5b30 msgr2=0x7fe5d8107370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.684 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.674+0000 7fe5dd23a700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5d80f5b30 0x7fe5d8107370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.684 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5dd23a700 1 -- 172.21.15.167:0/927368945 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d4005040 con 0x7fe5d80f6440 2022-01-31T19:42:08.684 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5dd23a700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 0x7fe5d8107860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe5c8002700 tx=0x7fe5c8009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.684 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5c80079c0 con 0x7fe5d80f6440 2022-01-31T19:42:08.685 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe5c800f040 con 0x7fe5d80f6440 2022-01-31T19:42:08.685 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d810c9f0 con 0x7fe5d80f6440 2022-01-31T19:42:08.685 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5c80146e0 con 0x7fe5d80f6440 2022-01-31T19:42:08.685 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.675+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d810cfb0 con 0x7fe5d80f6440 2022-01-31T19:42:08.685 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.676+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d80fe2e0 con 0x7fe5d80f6440 2022-01-31T19:42:08.686 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.676+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fe5c801b060 con 0x7fe5d80f6440 2022-01-31T19:42:08.687 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.676+0000 7fe5ce7fc700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.687 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.677+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe5c8078710 con 0x7fe5d80f6440 2022-01-31T19:42:08.687 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.677+0000 7fe5dda3b700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.688 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.677+0000 7fe5dda3b700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe5d4005010 tx=0x7fe5d4009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.688 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.681+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe5c804a330 con 0x7fe5d80f6440 2022-01-31T19:42:08.840 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.838+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fe5d810d680 con 0x7fe5d80f6440 2022-01-31T19:42:08.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.839+0000 7fe5ce7fc700 1 -- 172.21.15.167:0/927368945 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v40) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe5c804d950 con 0x7fe5d80f6440 2022-01-31T19:42:08.841 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 32 2022-01-31T19:42:08.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.840+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 msgr2=0x7fe5c4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.840+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe5d4005010 tx=0x7fe5d4009b30).stop 2022-01-31T19:42:08.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.840+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 msgr2=0x7fe5d8107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.840+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 0x7fe5d8107860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe5c8002700 tx=0x7fe5c8009d60).stop 2022-01-31T19:42:08.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 shutdown_connections 2022-01-31T19:42:08.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5d80f5b30 0x7fe5d8107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 --2- 172.21.15.167:0/927368945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5d80f6440 0x7fe5d8107860 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 >> 172.21.15.167:0/927368945 conn(0x7fe5d80f12e0 msgr2=0x7fe5d80607b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 shutdown_connections 2022-01-31T19:42:08.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.841+0000 7fe5dfc9f700 1 -- 172.21.15.167:0/927368945 wait complete. 2022-01-31T19:42:08.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2022-01-31T19:42:08.993 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 -- 172.21.15.167:0/1911065312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f4ed0 msgr2=0x7fbf000f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.993 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 --2- 172.21.15.167:0/1911065312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f4ed0 0x7fbf000f52f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fbef8004660 tx=0x7fbef8009b30).stop 2022-01-31T19:42:08.993 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 -- 172.21.15.167:0/1911065312 shutdown_connections 2022-01-31T19:42:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 --2- 172.21.15.167:0/1911065312 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f5870 0x7fbf000f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 --2- 172.21.15.167:0/1911065312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f4ed0 0x7fbf000f52f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 -- 172.21.15.167:0/1911065312 >> 172.21.15.167:0/1911065312 conn(0x7fbf000f09f0 msgr2=0x7fbf000f2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:08.994 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 -- 172.21.15.167:0/1911065312 shutdown_connections 2022-01-31T19:42:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.988+0000 7fbf07638700 1 -- 172.21.15.167:0/1911065312 wait complete. 2022-01-31T19:42:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf07638700 1 Processor -- start 2022-01-31T19:42:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf07638700 1 -- start start 2022-01-31T19:42:08.995 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf07638700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f4ed0 0x7fbf001095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf07638700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 0x7fbf00109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:08.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf07638700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0010a070 con 0x7fbf000f5870 2022-01-31T19:42:08.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf07638700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0010a1b0 con 0x7fbf000f4ed0 2022-01-31T19:42:08.996 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf04bd3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 0x7fbf00109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf04bd3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 0x7fbf00109ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33454/0 (socket says 172.21.15.167:33454) 2022-01-31T19:42:08.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.989+0000 7fbf04bd3700 1 -- 172.21.15.167:0/3312913865 learned_addr learned my addr 172.21.15.167:0/3312913865 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:08.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf053d4700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f4ed0 0x7fbf001095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:08.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf04bd3700 1 -- 172.21.15.167:0/3312913865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f4ed0 msgr2=0x7fbf001095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:08.997 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf04bd3700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f4ed0 0x7fbf001095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:08.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf04bd3700 1 -- 172.21.15.167:0/3312913865 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8005040 con 0x7fbf000f5870 2022-01-31T19:42:08.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf053d4700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f4ed0 0x7fbf001095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:42:08.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf04bd3700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 0x7fbf00109ac0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fbef4002700 tx=0x7fbef4009da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:08.998 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbef40079c0 con 0x7fbf000f5870 2022-01-31T19:42:08.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0010ec50 con 0x7fbf000f5870 2022-01-31T19:42:08.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbef400f040 con 0x7fbf000f5870 2022-01-31T19:42:08.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbef4014840 con 0x7fbf000f5870 2022-01-31T19:42:08.999 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.990+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0010f210 con 0x7fbf000f5870 2022-01-31T19:42:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.991+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf00027ea0 con 0x7fbf000f5870 2022-01-31T19:42:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.992+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fbef4007b20 con 0x7fbf000f5870 2022-01-31T19:42:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.992+0000 7fbefdffb700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbee805e5c0 0x7fbee8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:09.001 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.992+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fbef4078a10 con 0x7fbf000f5870 2022-01-31T19:42:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.992+0000 7fbf053d4700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbee805e5c0 0x7fbee8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.992+0000 7fbf053d4700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbee805e5c0 0x7fbee8060a80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fbef8004e30 tx=0x7fbef8009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:08.996+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbef404a610 con 0x7fbf000f5870 2022-01-31T19:42:09.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.160+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fbf000ff430 con 0x7fbf000f5870 2022-01-31T19:42:09.163 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.161+0000 7fbefdffb700 1 -- 172.21.15.167:0/3312913865 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v40) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbef404dc30 con 0x7fbf000f5870 2022-01-31T19:42:09.163 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 32 2022-01-31T19:42:09.165 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.162+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbee805e5c0 msgr2=0x7fbee8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:09.165 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbee805e5c0 0x7fbee8060a80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fbef8004e30 tx=0x7fbef8009b30).stop 2022-01-31T19:42:09.165 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 msgr2=0x7fbf00109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:09.165 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 0x7fbf00109ac0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fbef4002700 tx=0x7fbef4009da0).stop 2022-01-31T19:42:09.166 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 shutdown_connections 2022-01-31T19:42:09.166 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fbee805e5c0 0x7fbee8060a80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.166 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf000f4ed0 0x7fbf001095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.166 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 --2- 172.21.15.167:0/3312913865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf000f5870 0x7fbf00109ac0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.167 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 >> 172.21.15.167:0/3312913865 conn(0x7fbf000f09f0 msgr2=0x7fbf000f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:09.167 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 shutdown_connections 2022-01-31T19:42:09.167 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.163+0000 7fbf07638700 1 -- 172.21.15.167:0/3312913865 wait complete. 2022-01-31T19:42:09.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-01-31T19:42:09.299 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 -- 172.21.15.167:0/966129682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f6d90 msgr2=0x7f1ee80f71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:09.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/966129682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f6d90 0x7f1ee80f71f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f1ed8004660 tx=0x7f1ed8009b30).stop 2022-01-31T19:42:09.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 -- 172.21.15.167:0/966129682 shutdown_connections 2022-01-31T19:42:09.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/966129682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f6d90 0x7f1ee80f71f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/966129682 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ee80f5af0 0x7f1ee80f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 -- 172.21.15.167:0/966129682 >> 172.21.15.167:0/966129682 conn(0x7f1ee80f1230 msgr2=0x7f1ee80f3670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:09.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 -- 172.21.15.167:0/966129682 shutdown_connections 2022-01-31T19:42:09.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 -- 172.21.15.167:0/966129682 wait complete. 2022-01-31T19:42:09.302 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 Processor -- start 2022-01-31T19:42:09.302 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 -- start start 2022-01-31T19:42:09.302 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 0x7f1ee8060490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:09.303 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.294+0000 7f1eef3ba700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ee80f6d90 0x7f1ee810c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:09.304 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eef3ba700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee810c640 con 0x7f1ee80f5af0 2022-01-31T19:42:09.305 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eef3ba700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee810c780 con 0x7f1ee80f6d90 2022-01-31T19:42:09.305 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 0x7f1ee8060490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:09.306 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 0x7f1ee8060490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33464/0 (socket says 172.21.15.167:33464) 2022-01-31T19:42:09.306 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 -- 172.21.15.167:0/19379525 learned_addr learned my addr 172.21.15.167:0/19379525 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:09.306 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eec955700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ee80f6d90 0x7f1ee810c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:09.307 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 -- 172.21.15.167:0/19379525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ee80f6d90 msgr2=0x7f1ee810c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:09.307 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ee80f6d90 0x7f1ee810c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.307 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 -- 172.21.15.167:0/19379525 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8005040 con 0x7f1ee80f5af0 2022-01-31T19:42:09.307 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eed156700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 0x7f1ee8060490 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f1ee400c540 tx=0x7f1ee4007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:09.308 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee400f650 con 0x7f1ee80f5af0 2022-01-31T19:42:09.308 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1ee400fc40 con 0x7f1ee80f5af0 2022-01-31T19:42:09.308 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee810c9e0 con 0x7f1ee80f5af0 2022-01-31T19:42:09.308 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee4012c40 con 0x7f1ee80f5af0 2022-01-31T19:42:09.309 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.295+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee810cfa0 con 0x7f1ee80f5af0 2022-01-31T19:42:09.309 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.296+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8027ea0 con 0x7f1ee80f5af0 2022-01-31T19:42:09.309 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.296+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f1ee4010070 con 0x7f1ee80f5af0 2022-01-31T19:42:09.310 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.297+0000 7f1eddffb700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ed405e5c0 0x7f1ed4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:09.310 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.297+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1ee40782b0 con 0x7f1ee80f5af0 2022-01-31T19:42:09.310 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.297+0000 7f1eec955700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ed405e5c0 0x7f1ed4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:09.310 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.297+0000 7f1eec955700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ed405e5c0 0x7f1ed4060a80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1ed8009d60 tx=0x7f1ed8009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:09.311 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.301+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1ee40456b0 con 0x7f1ee80f5af0 2022-01-31T19:42:09.513 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.511+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ee80fe2d0 con 0x7f1ee80f5af0 2022-01-31T19:42:09.514 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.511+0000 7f1eddffb700 1 -- 172.21.15.167:0/19379525 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3041 (secure 0 0 0) 0x7f1ee4017090 con 0x7f1ee80f5af0 2022-01-31T19:42:09.515 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:09.515 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-01-31T19:41:48.803120+0000","modified":"2022-01-31T19:42:01.874180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14436,"mds_1":24267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14436":{"gid":14436,"name":"cephfs.smithi167.sgnfie","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.167:6827/1333633577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1333633577},{"type":"v1","addr":"172.21.15.167:6827","nonce":1333633577}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14452":{"gid":14452,"name":"cephfs.smithi167.rdkazj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.167:6829/2852537983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":2852537983},{"type":"v1","addr":"172.21.15.167:6829","nonce":2852537983}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24267":{"gid":24267,"name":"cephfs.smithi171.dyqqtj","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.171:6825/4256455981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4256455981},{"type":"v1","addr":"172.21.15.171:6825","nonce":4256455981}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24279":{"gid":24279,"name":"cephfs.smithi171.erzfea","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.171:6827/1322376319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":1322376319},{"type":"v1","addr":"172.21.15.171:6827","nonce":1322376319}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2022-01-31T19:42:09.517 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.513+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ed405e5c0 msgr2=0x7f1ed4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:09.517 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.513+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ed405e5c0 0x7f1ed4060a80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1ed8009d60 tx=0x7f1ed8009b30).stop 2022-01-31T19:42:09.518 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.513+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 msgr2=0x7f1ee8060490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:09.518 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.513+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 0x7f1ee8060490 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f1ee400c540 tx=0x7f1ee4007df0).stop 2022-01-31T19:42:09.518 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.513+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 shutdown_connections 2022-01-31T19:42:09.518 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.514+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ed405e5c0 0x7f1ed4060a80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.514+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ee80f5af0 0x7f1ee8060490 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.514+0000 7f1eef3ba700 1 --2- 172.21.15.167:0/19379525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ee80f6d90 0x7f1ee810c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:09.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.514+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 >> 172.21.15.167:0/19379525 conn(0x7f1ee80f1230 msgr2=0x7f1ee80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:09.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.514+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 shutdown_connections 2022-01-31T19:42:09.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:09.514+0000 7f1eef3ba700 1 -- 172.21.15.167:0/19379525 wait complete. 2022-01-31T19:42:09.520 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:09.527 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:09 smithi167 conmon[32206]: audit 2022-01-31T19:42:08.535239 2022-01-31T19:42:09.527 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:09 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.167:0/2393432485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-01-31T19:42:09.527 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:09 smithi167 conmon[32206]: audit 2022-01-31T19:42:08.840322+0000 mon.smithi167 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.167:0/927368945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2022-01-31T19:42:09.528 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:09 smithi167 conmon[32206]: audit 2022-01-31T19:42:09.162631+0000 mon.smithi167 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.167:0/3312913865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2022-01-31T19:42:09.529 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 2, 'flags': 50} 2022-01-31T19:42:09.529 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-01-31T19:42:09.543 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:42:09.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph fs dump' 2022-01-31T19:42:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:09 smithi171 conmon[35325]: audit 2022-01-31T19:42:08.535239+0000 mon.smithi167 (mon.0) 582 : audit [DBG] 2022-01-31T19:42:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:09 smithi171 conmon[35325]: from='client.? 172.21.15.167:0/2393432485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-01-31T19:42:09.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:09 smithi171 conmon[35325]: audit 2022-01-31T19:42:08.840322+0000 mon.smithi167 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.167:0/927368945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2022-01-31T19:42:09.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:09 smithi171 conmon[35325]: audit 2022-01-31T19:42:09. 2022-01-31T19:42:09.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:09 smithi171 conmon[35325]: 162631+0000 mon.smithi167 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.167:0/3312913865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2022-01-31T19:42:10.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:10 smithi167 conmon[32206]: cluster 2022-01-31T19:42:08.729145+0000 mgr.smithi167.aciqpk (mgr.14156) 198 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2022-01-31T19:42:10.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:10 smithi167 conmon[32206]: audit 2022-01-31T19:42:09.513115+0000 mon.smithi167 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.167:0/19379525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:10.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:10 smithi171 conmon[35325]: cluster 2022-01-31T19:42:08.729145+0000 mgr.smithi167.aciqpk (mgr.14156) 198 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2022-01-31T19:42:10.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:10 smithi171 conmon[35325]: audit 2022-01-31T19:42:09.513115+0000 mon.smithi167 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.167:0/19379525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:11.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- 172.21.15.167:0/454660258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 msgr2=0x7fad840f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:11.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 --2- 172.21.15.167:0/454660258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad840f82e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fad74004660 tx=0x7fad74009b30).stop 2022-01-31T19:42:11.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- 172.21.15.167:0/454660258 shutdown_connections 2022-01-31T19:42:11.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 --2- 172.21.15.167:0/454660258 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad840f64d0 0x7fad840f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:11.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 --2- 172.21.15.167:0/454660258 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad840f82e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:11.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- 172.21.15.167:0/454660258 >> 172.21.15.167:0/454660258 conn(0x7fad840f1f70 msgr2=0x7fad840f4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:11.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- 172.21.15.167:0/454660258 shutdown_connections 2022-01-31T19:42:11.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- 172.21.15.167:0/454660258 wait complete. 2022-01-31T19:42:11.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 Processor -- start 2022-01-31T19:42:11.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- start start 2022-01-31T19:42:11.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad840f64d0 0x7fad84107170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:11.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad84107660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:11.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84107c10 con 0x7fad840f7ec0 2022-01-31T19:42:11.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.238+0000 7fad8b39b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84107d50 con 0x7fad840f64d0 2022-01-31T19:42:11.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad84107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:11.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad84107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33478/0 (socket says 172.21.15.167:33478) 2022-01-31T19:42:11.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 -- 172.21.15.167:0/2484407563 learned_addr learned my addr 172.21.15.167:0/2484407563 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:11.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad89137700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad840f64d0 0x7fad84107170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:11.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 -- 172.21.15.167:0/2484407563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad840f64d0 msgr2=0x7fad84107170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:11.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad840f64d0 0x7fad84107170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:11.247 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 -- 172.21.15.167:0/2484407563 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad74005040 con 0x7fad840f7ec0 2022-01-31T19:42:11.247 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad88936700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad84107660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fad8000c790 tx=0x7fad80007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:11.247 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad80002590 con 0x7fad840f7ec0 2022-01-31T19:42:11.247 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad80002b80 con 0x7fad840f7ec0 2022-01-31T19:42:11.248 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad841086f0 con 0x7fad840f7ec0 2022-01-31T19:42:11.248 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad80013d60 con 0x7fad840f7ec0 2022-01-31T19:42:11.248 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.239+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad84108cb0 con 0x7fad840f7ec0 2022-01-31T19:42:11.249 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.240+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad84045bb0 con 0x7fad840f7ec0 2022-01-31T19:42:11.250 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.241+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fad8001a060 con 0x7fad840f7ec0 2022-01-31T19:42:11.250 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.241+0000 7fad79ffb700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fad7005e5c0 0x7fad70060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:11.250 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.241+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fad80078b30 con 0x7fad840f7ec0 2022-01-31T19:42:11.250 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.241+0000 7fad89137700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fad7005e5c0 0x7fad70060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:11.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.241+0000 7fad89137700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fad7005e5c0 0x7fad70060a80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fad74004630 tx=0x7fad74009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:11.251 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.245+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad8004e160 con 0x7fad840f7ec0 2022-01-31T19:42:11.435 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.433+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fad84027ae0 con 0x7fad840f7ec0 2022-01-31T19:42:11.436 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.434+0000 7fad79ffb700 1 -- 172.21.15.167:0/2484407563 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1672 (secure 0 0 0) 0x7fad8004dd50 con 0x7fad840f7ec0 2022-01-31T19:42:11.437 INFO:teuthology.orchestra.run.smithi167.stdout:e15 2022-01-31T19:42:11.437 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:42:11.437 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:42:11.437 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:42:11.438 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:11.438 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:42:11.438 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:42:11.438 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 15 2022-01-31T19:42:11.439 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:42:11.439 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:42:11.439 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:42:01.874180+0000 2022-01-31T19:42:11.439 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:42:11.439 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:42:11.440 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:42:11.440 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:42:11.440 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:42:11.440 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:42:11.441 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:42:11.441 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:42:11.441 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:42:11.441 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:42:11.442 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:42:11.442 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:42:11.442 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:42:11.443 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:42:11.443 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:42:11.443 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:42:11.443 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:42:11.444 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:42:11.444 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:42:11.444 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:42:11.444 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:42:11.444 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:42:11.445 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:42:11.445 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:42:11.445 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:11.445 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:11.446 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fad7005e5c0 msgr2=0x7fad70060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:11.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fad7005e5c0 0x7fad70060a80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fad74004630 tx=0x7fad74009b30).stop 2022-01-31T19:42:11.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 msgr2=0x7fad84107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:11.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad84107660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fad8000c790 tx=0x7fad80007d70).stop 2022-01-31T19:42:11.447 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 shutdown_connections 2022-01-31T19:42:11.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fad7005e5c0 0x7fad70060a80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:11.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad840f64d0 0x7fad84107170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:11.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 --2- 172.21.15.167:0/2484407563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fad840f7ec0 0x7fad84107660 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:11.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.435+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 >> 172.21.15.167:0/2484407563 conn(0x7fad840f1f70 msgr2=0x7fad840fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:11.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.436+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 shutdown_connections 2022-01-31T19:42:11.449 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:11.436+0000 7fad8b39b700 1 -- 172.21.15.167:0/2484407563 wait complete. 2022-01-31T19:42:11.449 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:11.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:11 smithi167 conmon[32206]: cluster 2022-01-31T19:42:10.729580+0000 mgr.smithi167.aciqpk (mgr.14156) 199 : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2022-01-31T19:42:11.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:11 smithi167 conmon[32206]: audit 2022-01-31T19:42:10.758512+0000 mon.smithi167 (mon.0) 586 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:11.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:11 smithi167 conmon[32206]: audit 2022-01-31T19:42:10.764016+0000 mon.smithi167 (mon.0) 587 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:11.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:11 smithi167 conmon[32206]: audit 2022-01-31T19:42:11.435447+0000 mon.smithi167 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.167:0/2484407563' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:42:11.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2022-01-31T19:42:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: cluster 2022-01-31T19:42:10.729580+0000 mgr.smithi167.aciqpk (mgr.14156) 199 : cluster 2022-01-31T19:42:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: [DBG] pgmap v156: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2022-01-31T19:42:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: audit 2022-01-31T19:42:10.758512+0000 mon.smithi167 (mon.0) 586 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:12.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: audit 2022-01-31T19:42:10.764016 2022-01-31T19:42:12.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 587 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:12.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:42:12.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: :42:11.435447+0000 mon.smithi167 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.167:0/2484407563' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:42:12.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:11 smithi171 conmon[35325]: 2022-01-31T19:42:13.495 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.491+0000 7fb9ce83f700 1 -- 172.21.15.167:0/4229403624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 msgr2=0x7fb9c80f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:13.496 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.491+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/4229403624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c80f6de0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb9bc004660 tx=0x7fb9bc009b30).stop 2022-01-31T19:42:13.496 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 -- 172.21.15.167:0/4229403624 shutdown_connections 2022-01-31T19:42:13.496 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/4229403624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c80f6de0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:13.497 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/4229403624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c80f56e0 0x7fb9c80f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:13.497 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 -- 172.21.15.167:0/4229403624 >> 172.21.15.167:0/4229403624 conn(0x7fb9c80f0e60 msgr2=0x7fb9c80f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:13.497 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 -- 172.21.15.167:0/4229403624 shutdown_connections 2022-01-31T19:42:13.497 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 -- 172.21.15.167:0/4229403624 wait complete. 2022-01-31T19:42:13.497 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 Processor -- start 2022-01-31T19:42:13.498 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.492+0000 7fb9ce83f700 1 -- start start 2022-01-31T19:42:13.498 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9ce83f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c80f56e0 0x7fb9c8105f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:13.498 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9ce83f700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c8106470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:13.499 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9ce83f700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8106a20 con 0x7fb9c80f6980 2022-01-31T19:42:13.499 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9ce83f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8106b60 con 0x7fb9c80f56e0 2022-01-31T19:42:13.499 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c8106470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:13.499 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c8106470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33490/0 (socket says 172.21.15.167:33490) 2022-01-31T19:42:13.499 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 -- 172.21.15.167:0/3589049335 learned_addr learned my addr 172.21.15.167:0/3589049335 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:13.500 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 -- 172.21.15.167:0/3589049335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c80f56e0 msgr2=0x7fb9c8105f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:13.500 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c80f56e0 0x7fb9c8105f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:42:13.500 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 -- 172.21.15.167:0/3589049335 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc005040 con 0x7fb9c80f6980 2022-01-31T19:42:13.500 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c77fe700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c8106470 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fb9bc009990 tx=0x7fb9bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:13.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9bc018070 con 0x7fb9c80f6980 2022-01-31T19:42:13.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb9bc013410 con 0x7fb9c80f6980 2022-01-31T19:42:13.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.493+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c81074a0 con 0x7fb9c80f6980 2022-01-31T19:42:13.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.494+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9bc01c3c0 con 0x7fb9c80f6980 2022-01-31T19:42:13.501 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.494+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c8107a00 con 0x7fb9c80f6980 2022-01-31T19:42:13.503 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.495+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c8045bb0 con 0x7fb9c80f6980 2022-01-31T19:42:13.503 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.495+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fb9bc01d420 con 0x7fb9c80f6980 2022-01-31T19:42:13.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.495+0000 7fb9c4ff9700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb9b005e5c0 0x7fb9b0060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:13.504 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.495+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb9bc07a530 con 0x7fb9c80f6980 2022-01-31T19:42:13.505 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.495+0000 7fb9c7fff700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb9b005e5c0 0x7fb9b0060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:13.505 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.496+0000 7fb9c7fff700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb9b005e5c0 0x7fb9b0060a80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb9c80f67e0 tx=0x7fb9b80099e0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:13.505 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.499+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb9bc04c130 con 0x7fb9c80f6980 2022-01-31T19:42:13.688 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.686+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c8027ae0 con 0x7fb9c80f6980 2022-01-31T19:42:13.689 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.686+0000 7fb9c4ff9700 1 -- 172.21.15.167:0/3589049335 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3041 (secure 0 0 0) 0x7fb9bc01d6b0 con 0x7fb9c80f6980 2022-01-31T19:42:13.691 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb9b005e5c0 msgr2=0x7fb9b0060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:13.691 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb9b005e5c0 0x7fb9b0060a80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb9c80f67e0 tx=0x7fb9b80099e0).stop 2022-01-31T19:42:13.691 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 msgr2=0x7fb9c8106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:13.691 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c8106470 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fb9bc009990 tx=0x7fb9bc009b30).stop 2022-01-31T19:42:13.692 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 shutdown_connections 2022-01-31T19:42:13.692 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fb9b005e5c0 0x7fb9b0060a80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:13.692 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9c80f56e0 0x7fb9c8105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:42:13.692 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 --2- 172.21.15.167:0/3589049335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9c80f6980 0x7fb9c8106470 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:13.692 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.688+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 >> 172.21.15.167:0/3589049335 conn(0x7fb9c80f0e60 msgr2=0x7fb9c80fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:13.693 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.689+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 shutdown_connections 2022-01-31T19:42:13.693 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:13.689+0000 7fb9ce83f700 1 -- 172.21.15.167:0/3589049335 wait complete. 2022-01-31T19:42:13.693 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:13.700 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:42:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:13 smithi171 conmon[35325]: cluster 2022-01-31T19:42:12.730020+0000 mgr.smithi167.aciqpk (mgr.14156) 200 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 4 op/s 2022-01-31T19:42:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:13 smithi171 conmon[35325]: audit 2022-01-31T19:42:13.688092+0000 mon.smithi167 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.167:0/3589049335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:14.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:13 smithi167 conmon[32206]: cluster 2022-01-31T19:42:12.730020+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:42:14.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:13 smithi167 conmon[32206]: ) 200 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 4 op/s 2022-01-31T19:42:14.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:13 smithi167 conmon[32206]: audit 2022-01-31T19:42:13.688092+0000 mon.smithi167 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.167:0/3589049335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:14.190 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- bash -c 'ceph --format=json mds versions | jq -e ". | add == 4"' 2022-01-31T19:42:15.910 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 -- 172.21.15.167:0/1664842397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 msgr2=0x7fd7040f5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:15.910 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 --2- 172.21.15.167:0/1664842397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd7040f5ad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd6ec004660 tx=0x7fd6ec009b30).stop 2022-01-31T19:42:15.911 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:15 smithi167 conmon[32206]: cluster 2022-01-31T19:42:14.730706+0000 mgr.smithi167.aciqpk (mgr.14156) 201 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 4 op/s 2022-01-31T19:42:15.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 -- 172.21.15.167:0/1664842397 shutdown_connections 2022-01-31T19:42:15.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 --2- 172.21.15.167:0/1664842397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd7040f5ad0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:15.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 --2- 172.21.15.167:0/1664842397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7040f6d10 0x7fd7040f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:15.914 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 -- 172.21.15.167:0/1664842397 >> 172.21.15.167:0/1664842397 conn(0x7fd7040f05e0 msgr2=0x7fd7040f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:15.914 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 -- 172.21.15.167:0/1664842397 shutdown_connections 2022-01-31T19:42:15.914 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 -- 172.21.15.167:0/1664842397 wait complete. 2022-01-31T19:42:15.914 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.907+0000 7fd70a89d700 1 Processor -- start 2022-01-31T19:42:15.915 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd70a89d700 1 -- start start 2022-01-31T19:42:15.915 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd70a89d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7040f6d10 0x7fd704105f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:15.915 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd70a89d700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd704106450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:15.916 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd70a89d700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd704106a00 con 0x7fd7040f76b0 2022-01-31T19:42:15.916 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd70a89d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd704106b40 con 0x7fd7040f6d10 2022-01-31T19:42:15.916 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd704106450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:15.917 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd704106450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33512/0 (socket says 172.21.15.167:33512) 2022-01-31T19:42:15.917 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 -- 172.21.15.167:0/452393014 learned_addr learned my addr 172.21.15.167:0/452393014 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:15.917 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd703fff700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7040f6d10 0x7fd704105f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:15.918 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 -- 172.21.15.167:0/452393014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7040f6d10 msgr2=0x7fd704105f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:15.918 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7040f6d10 0x7fd704105f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:15.918 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 -- 172.21.15.167:0/452393014 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec005040 con 0x7fd7040f76b0 2022-01-31T19:42:15.918 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd7037fe700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd704106450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd6ec009990 tx=0x7fd6ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:15.919 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.908+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6ec018070 con 0x7fd7040f76b0 2022-01-31T19:42:15.919 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.909+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd704107480 con 0x7fd7040f76b0 2022-01-31T19:42:15.919 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.909+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd6ec013410 con 0x7fd7040f76b0 2022-01-31T19:42:15.919 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.909+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6ec01de80 con 0x7fd7040f76b0 2022-01-31T19:42:15.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.909+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7041079e0 con 0x7fd7040f76b0 2022-01-31T19:42:15.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.910+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd704045bb0 con 0x7fd7040f76b0 2022-01-31T19:42:15.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.910+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fd6ec01d420 con 0x7fd7040f76b0 2022-01-31T19:42:15.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.910+0000 7fd700ff9700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd6f005e5c0 0x7fd6f0060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:15.920 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.910+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd6ec07a450 con 0x7fd7040f76b0 2022-01-31T19:42:15.921 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.910+0000 7fd703fff700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd6f005e5c0 0x7fd6f0060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:15.921 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.911+0000 7fd703fff700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd6f005e5c0 0x7fd6f0060a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd70405f6d0 tx=0x7fd6f400b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:15.922 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:15.914+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd6ec04c090 con 0x7fd7040f76b0 2022-01-31T19:42:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:15 smithi171 conmon[35325]: cluster 2022-01-31T19:42:14.730706+0000 mgr.smithi167.aciqpk (mgr.14156) 201 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 4 op/s 2022-01-31T19:42:16.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.122+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fd704027ae0 con 0x7fd7040f76b0 2022-01-31T19:42:16.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.122+0000 7fd700ff9700 1 -- 172.21.15.167:0/452393014 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+86 (secure 0 0 0) 0x7fd6ec04f670 con 0x7fd7040f76b0 2022-01-31T19:42:16.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd6f005e5c0 msgr2=0x7fd6f0060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd6f005e5c0 0x7fd6f0060a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd70405f6d0 tx=0x7fd6f400b040).stop 2022-01-31T19:42:16.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 msgr2=0x7fd704106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd704106450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd6ec009990 tx=0x7fd6ec009b30).stop 2022-01-31T19:42:16.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 shutdown_connections 2022-01-31T19:42:16.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fd6f005e5c0 0x7fd6f0060a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7040f6d10 0x7fd704105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 --2- 172.21.15.167:0/452393014 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7040f76b0 0x7fd704106450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 >> 172.21.15.167:0/452393014 conn(0x7fd7040f05e0 msgr2=0x7fd7040fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:16.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 shutdown_connections 2022-01-31T19:42:16.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.124+0000 7fd70a89d700 1 -- 172.21.15.167:0/452393014 wait complete. 2022-01-31T19:42:16.136 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:42:16.523 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2022-01-31T19:42:16.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-01-31T19:42:16.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 -- 172.21.15.167:0/1490491371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 msgr2=0x7f4a4c0f5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.661 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 --2- 172.21.15.167:0/1490491371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c0f5f20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f4a3c004660 tx=0x7f4a3c009b30).stop 2022-01-31T19:42:16.661 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 -- 172.21.15.167:0/1490491371 shutdown_connections 2022-01-31T19:42:16.662 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 --2- 172.21.15.167:0/1490491371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a4c0f6da0 0x7f4a4c0f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.662 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 --2- 172.21.15.167:0/1490491371 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c0f5f20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.662 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 -- 172.21.15.167:0/1490491371 >> 172.21.15.167:0/1490491371 conn(0x7f4a4c0f1240 msgr2=0x7f4a4c0f3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:16.663 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 -- 172.21.15.167:0/1490491371 shutdown_connections 2022-01-31T19:42:16.663 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.655+0000 7f4a5461a700 1 -- 172.21.15.167:0/1490491371 wait complete. 2022-01-31T19:42:16.663 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a5461a700 1 Processor -- start 2022-01-31T19:42:16.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a5461a700 1 -- start start 2022-01-31T19:42:16.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a5461a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:16.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a5461a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a4c0f6da0 0x7f4a4c10c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:16.664 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a5461a700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c10c5f0 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a5461a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c10c730 con 0x7f4a4c0f6da0 2022-01-31T19:42:16.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:16.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c060530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33522/0 (socket says 172.21.15.167:33522) 2022-01-31T19:42:16.665 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 -- 172.21.15.167:0/2843557300 learned_addr learned my addr 172.21.15.167:0/2843557300 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:16.666 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a51bb5700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a4c0f6da0 0x7f4a4c10c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:16.666 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 -- 172.21.15.167:0/2843557300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a4c0f6da0 msgr2=0x7f4a4c10c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.666 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a4c0f6da0 0x7f4a4c10c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.666 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 -- 172.21.15.167:0/2843557300 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c005040 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.667 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.656+0000 7f4a523b6700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c060530 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f4a3c0097f0 tx=0x7f4a3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:16.667 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.657+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a3c019070 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.667 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.657+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c10c930 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.667 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.657+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4a3c009820 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.657+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a3c014c90 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.657+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c10ce90 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.658+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c027ea0 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.658+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f4a3c025070 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.668 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.658+0000 7f4a42ffd700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4a3805e5c0 0x7f4a38060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:16.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.658+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4a3c079940 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.658+0000 7f4a51bb5700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4a3805e5c0 0x7f4a38060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:16.669 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.659+0000 7f4a51bb5700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4a3805e5c0 0x7f4a38060a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f4a48007f80 tx=0x7f4a4800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:16.670 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.662+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4a3c04b540 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.821 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.819+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a4c0fe2e0 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.822 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.820+0000 7f4a42ffd700 1 -- 172.21.15.167:0/2843557300 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+13869 (secure 0 0 0) 0x7f4a3c04eb60 con 0x7f4a4c0f5b00 2022-01-31T19:42:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:16.823 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":40,"fsid":"1c5a1df6-82cd-11ec-8c35-001a4aab830c","created":"2022-01-31T19:37:06.503417+0000","modified":"2022-01-31T19:41:50.817316+0000","last_up_change":"2022-01-31T19:40:51.787533+0000","last_in_change":"2022-01-31T19:40:36.089120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-01-31T19:40:13.386038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2022-01-31T19:41:47.179870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2022-01-31T19:41:48.033371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6b39ea02-b5a4-4a19-bd99-849c0378fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6803","nonce":2333495797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6805","nonce":2333495797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6809","nonce":2333495797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2333495797},{"type":"v1","addr":"172.21.15.167:6807","nonce":2333495797}]},"public_addr":"172.21.15.167:6803/2333495797","cluster_addr":"172.21.15.167:6805/2333495797","heartbeat_back_addr":"172.21.15.167:6809/2333495797","heartbeat_front_addr":"172.21.15.167:6807/2333495797","state":["exists","up"]},{"osd":1,"uuid":"214aeaf2-f8c7-4638-96b6-4e6bb4bd802f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6811","nonce":3267421031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6813","nonce":3267421031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6817","nonce":3267421031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":3267421031},{"type":"v1","addr":"172.21.15.167:6815","nonce":3267421031}]},"public_addr":"172.21.15.167:6811/3267421031","cluster_addr":"172.21.15.167:6813/3267421031","heartbeat_back_addr":"172.21.15.167:6817/3267421031","heartbeat_front_addr":"172.21.15.167:6815/3267421031","state":["exists","up"]},{"osd":2,"uuid":"3536e17c-9f9a-4867-91f4-f84b2b5d53a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6819","nonce":2914232381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6821","nonce":2914232381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6825","nonce":2914232381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2914232381},{"type":"v1","addr":"172.21.15.167:6823","nonce":2914232381}]},"public_addr":"172.21.15.167:6819/2914232381","cluster_addr":"172.21.15.167:6821/2914232381","heartbeat_back_addr":"172.21.15.167:6825/2914232381","heartbeat_front_addr":"172.21.15.167:6823/2914232381","state":["exists","up"]},{"osd":3,"uuid":"e924c508-4baf-4ed0-9a32-42a51710ca8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6801","nonce":1700293141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6803","nonce":1700293141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6807","nonce":1700293141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":1700293141},{"type":"v1","addr":"172.21.15.171:6805","nonce":1700293141}]},"public_addr":"172.21.15.171:6801/1700293141","cluster_addr":"172.21.15.171:6803/1700293141","heartbeat_back_addr":"172.21.15.171:6807/1700293141","heartbeat_front_addr":"172.21.15.171:6805/1700293141","state":["exists","up"]},{"osd":4,"uuid":"c4310782-5c0e-4962-b32e-45e28ac014ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6809","nonce":3746074472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6811","nonce":3746074472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6815","nonce":3746074472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3746074472},{"type":"v1","addr":"172.21.15.171:6813","nonce":3746074472}]},"public_addr":"172.21.15.171:6809/3746074472","cluster_addr":"172.21.15.171:6811/3746074472","heartbeat_back_addr":"172.21.15.171:6815/3746074472","heartbeat_front_addr":"172.21.15.171:6813/3746074472","state":["exists","up"]},{"osd":5,"uuid":"2ef7b4c2-8076-4866-b26a-8027fd249d2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6817","nonce":1956867870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6819","nonce":1956867870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6823","nonce":1956867870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1956867870},{"type":"v1","addr":"172.21.15.171:6821","nonce":1956867870}]},"public_addr":"172.21.15.171:6817/1956867870","cluster_addr":"172.21.15.171:6819/1956867870","heartbeat_back_addr":"172.21.15.171:6823/1956867870","heartbeat_front_addr":"172.21.15.171:6821/1956867870","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:26.422630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:39:45.101018+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:03.040927+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:18.082335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:33.822236+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2022-01-31T19:40:50.200473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2480743863":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6801/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2306410502":"2022-02-01T19:37:58.631415+0000","172.21.15.167:6800/2257398586":"2022-02-01T19:37:58.631415+0000","172.21.15.167:0/2760109630":"2022-02-01T19:37:34.139403+0000","172.21.15.167:0/694084204":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6800/3497254897":"2022-02-01T19:37:34.139403+0000","172.21.15.167:6801/3497254897":"2022-02-01T19:37:34.139403+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-31T19:42:16.824 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4a3805e5c0 msgr2=0x7f4a38060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.824 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4a3805e5c0 0x7f4a38060a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f4a48007f80 tx=0x7f4a4800b040).stop 2022-01-31T19:42:16.825 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 msgr2=0x7f4a4c060530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.825 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c060530 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f4a3c0097f0 tx=0x7f4a3c00b040).stop 2022-01-31T19:42:16.825 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 shutdown_connections 2022-01-31T19:42:16.825 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4a3805e5c0 0x7f4a38060a80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.826 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4c0f5b00 0x7f4a4c060530 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.826 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 --2- 172.21.15.167:0/2843557300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a4c0f6da0 0x7f4a4c10c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.826 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 >> 172.21.15.167:0/2843557300 conn(0x7f4a4c0f1240 msgr2=0x7f4a4c0fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:16.826 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 shutdown_connections 2022-01-31T19:42:16.826 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.822+0000 7f4a5461a700 1 -- 172.21.15.167:0/2843557300 wait complete. 2022-01-31T19:42:16.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:16 smithi167 conmon[32206]: audit 2022-01-31T19:42:16.124119+0000 mon.smithi167 (mon.0) 590 : audit [DBG] 2022-01-31T19:42:16.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:16 smithi167 conmon[32206]: from='client.? 172.21.15.167:0/452393014' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2022-01-31T19:42:16.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-01-31T19:40:13.386038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2022-01-31T19:41:47.179870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2022-01-31T19:41:48.033371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-01-31T19:42:16.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-01-31T19:42:16.959 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2992684022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f7e80 msgr2=0x7f2c580f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.960 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2992684022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f7e80 0x7f2c580f82e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2c40004660 tx=0x7f2c40009b30).stop 2022-01-31T19:42:16.960 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2992684022 shutdown_connections 2022-01-31T19:42:16.960 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2992684022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f7e80 0x7f2c580f82e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.961 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2992684022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c580f6be0 0x7f2c580f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.961 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2992684022 >> 172.21.15.167:0/2992684022 conn(0x7f2c580f2340 msgr2=0x7f2c580f4760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:16.961 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2992684022 shutdown_connections 2022-01-31T19:42:16.961 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.955+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2992684022 wait complete. 2022-01-31T19:42:16.962 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c5d44c700 1 Processor -- start 2022-01-31T19:42:16.962 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c5d44c700 1 -- start start 2022-01-31T19:42:16.962 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c5d44c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 0x7f2c58107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:16.962 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c5d44c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c580f7e80 0x7f2c58107820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:16.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c5d44c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58107dd0 con 0x7f2c580f6be0 2022-01-31T19:42:16.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c5d44c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58107f10 con 0x7f2c580f7e80 2022-01-31T19:42:16.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 0x7f2c58107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:16.963 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 0x7f2c58107330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33532/0 (socket says 172.21.15.167:33532) 2022-01-31T19:42:16.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 -- 172.21.15.167:0/2984439920 learned_addr learned my addr 172.21.15.167:0/2984439920 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:16.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c567fc700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c580f7e80 0x7f2c58107820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:16.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 -- 172.21.15.167:0/2984439920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c580f7e80 msgr2=0x7f2c58107820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:16.964 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c580f7e80 0x7f2c58107820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:16.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 -- 172.21.15.167:0/2984439920 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c40005040 con 0x7f2c580f6be0 2022-01-31T19:42:16.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c56ffd700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 0x7f2c58107330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2c5000c810 tx=0x7f2c50004d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:16.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.956+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c5000a700 con 0x7f2c580f6be0 2022-01-31T19:42:16.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.957+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2c5000acf0 con 0x7f2c580f6be0 2022-01-31T19:42:16.965 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.957+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5810c9b0 con 0x7f2c580f6be0 2022-01-31T19:42:16.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.957+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c50012dc0 con 0x7f2c580f6be0 2022-01-31T19:42:16.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.957+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5810cf70 con 0x7f2c580f6be0 2022-01-31T19:42:16.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.957+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c58027ea0 con 0x7f2c580f6be0 2022-01-31T19:42:16.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.958+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f2c50019030 con 0x7f2c580f6be0 2022-01-31T19:42:16.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.958+0000 7f2c4f7fe700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f2c4405e5c0 0x7f2c44060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:16.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.958+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f2c50078a10 con 0x7f2c580f6be0 2022-01-31T19:42:16.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.958+0000 7f2c567fc700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f2c4405e5c0 0x7f2c44060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:16.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.959+0000 7f2c567fc700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f2c4405e5c0 0x7f2c44060a80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2c40004630 tx=0x7f2c40009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:16.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:16.962+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2c500466b0 con 0x7f2c580f6be0 2022-01-31T19:42:17.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:16 smithi171 conmon[35325]: audit 2022-01-31T19:42:16.124119+0000 mon.smithi167 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.167:0/452393014' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2022-01-31T19:42:17.119 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.117+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2c580ff3a0 con 0x7f2c580f6be0 2022-01-31T19:42:17.119 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.117+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/2984439920 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v40) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2c50017070 con 0x7f2c580f6be0 2022-01-31T19:42:17.120 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 1 2022-01-31T19:42:17.121 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f2c4405e5c0 msgr2=0x7f2c44060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f2c4405e5c0 0x7f2c44060a80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2c40004630 tx=0x7f2c40009b30).stop 2022-01-31T19:42:17.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 msgr2=0x7f2c58107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 0x7f2c58107330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2c5000c810 tx=0x7f2c50004d90).stop 2022-01-31T19:42:17.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 shutdown_connections 2022-01-31T19:42:17.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f2c4405e5c0 0x7f2c44060a80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2c580f6be0 0x7f2c58107330 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 --2- 172.21.15.167:0/2984439920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c580f7e80 0x7f2c58107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 >> 172.21.15.167:0/2984439920 conn(0x7f2c580f2340 msgr2=0x7f2c580fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:17.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 shutdown_connections 2022-01-31T19:42:17.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.119+0000 7f2c5d44c700 1 -- 172.21.15.167:0/2984439920 wait complete. 2022-01-31T19:42:17.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2022-01-31T19:42:17.254 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/1428056309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f6be0 msgr2=0x7fe5dc0f7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/1428056309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f6be0 0x7fe5dc0f7000 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fe5d0004660 tx=0x7fe5d0009b30).stop 2022-01-31T19:42:17.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/1428056309 shutdown_connections 2022-01-31T19:42:17.255 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/1428056309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc0f82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/1428056309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f6be0 0x7fe5dc0f7000 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/1428056309 >> 172.21.15.167:0/1428056309 conn(0x7fe5dc0f2340 msgr2=0x7fe5dc0f4760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:17.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/1428056309 shutdown_connections 2022-01-31T19:42:17.256 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/1428056309 wait complete. 2022-01-31T19:42:17.257 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 Processor -- start 2022-01-31T19:42:17.257 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.250+0000 7fe5e0a4e700 1 -- start start 2022-01-31T19:42:17.257 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5e0a4e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5dc0f6be0 0x7fe5dc107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5e0a4e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc107840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5e0a4e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc107df0 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5e0a4e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc107f30 con 0x7fe5dc0f6be0 2022-01-31T19:42:17.258 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc107840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc107840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33544/0 (socket says 172.21.15.167:33544) 2022-01-31T19:42:17.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5da59c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5dc0f6be0 0x7fe5dc107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.259 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 -- 172.21.15.167:0/398052586 learned_addr learned my addr 172.21.15.167:0/398052586 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:17.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 -- 172.21.15.167:0/398052586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5dc0f6be0 msgr2=0x7fe5dc107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.260 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5dc0f6be0 0x7fe5dc107350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 -- 172.21.15.167:0/398052586 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d0005040 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5d9d9b700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc107840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fe5cc0026d0 tx=0x7fe5cc009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:17.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc0079c0 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.261 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe5cc00f040 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5dc10c9d0 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.262 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc014820 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.251+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc10cf90 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.252+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5dc027ea0 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.263 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.253+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fe5cc0090d0 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.253+0000 7fe5caffd700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.264 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.253+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe5cc078840 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.265 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.253+0000 7fe5da59c700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.265 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.253+0000 7fe5da59c700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe5d0005010 tx=0x7fe5d0009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:17.265 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.257+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe5cc04a570 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.410 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.408+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fe5dc0ff3a0 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.410 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.408+0000 7fe5caffd700 1 -- 172.21.15.167:0/398052586 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v40) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe5cc04db90 con 0x7fe5dc0f7e80 2022-01-31T19:42:17.411 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 32 2022-01-31T19:42:17.412 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 msgr2=0x7fe5c4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.412 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe5d0005010 tx=0x7fe5d0009b30).stop 2022-01-31T19:42:17.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 msgr2=0x7fe5dc107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc107840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fe5cc0026d0 tx=0x7fe5cc009de0).stop 2022-01-31T19:42:17.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 shutdown_connections 2022-01-31T19:42:17.413 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe5c405e5c0 0x7fe5c4060a80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5dc0f6be0 0x7fe5dc107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 --2- 172.21.15.167:0/398052586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5dc0f7e80 0x7fe5dc107840 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 >> 172.21.15.167:0/398052586 conn(0x7fe5dc0f2340 msgr2=0x7fe5dc0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:17.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 shutdown_connections 2022-01-31T19:42:17.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.410+0000 7fe5e0a4e700 1 -- 172.21.15.167:0/398052586 wait complete. 2022-01-31T19:42:17.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2022-01-31T19:42:17.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 -- 172.21.15.167:0/1293160801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 msgr2=0x7f97600f5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 --2- 172.21.15.167:0/1293160801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f97600f5f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f975c004660 tx=0x7f975c009b30).stop 2022-01-31T19:42:17.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 -- 172.21.15.167:0/1293160801 shutdown_connections 2022-01-31T19:42:17.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 --2- 172.21.15.167:0/1293160801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97600f6da0 0x7f97600f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 --2- 172.21.15.167:0/1293160801 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f97600f5f20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 -- 172.21.15.167:0/1293160801 >> 172.21.15.167:0/1293160801 conn(0x7f97600f1260 msgr2=0x7f97600f3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:17.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 -- 172.21.15.167:0/1293160801 shutdown_connections 2022-01-31T19:42:17.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.540+0000 7f9767caa700 1 -- 172.21.15.167:0/1293160801 wait complete. 2022-01-31T19:42:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9767caa700 1 Processor -- start 2022-01-31T19:42:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9767caa700 1 -- start start 2022-01-31T19:42:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9767caa700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f9760060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9767caa700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97600f6da0 0x7f976010c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9767caa700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976010c640 con 0x7f97600f5b00 2022-01-31T19:42:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9767caa700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976010c780 con 0x7f97600f6da0 2022-01-31T19:42:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.541+0000 7f9765a46700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f9760060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765a46700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f9760060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33552/0 (socket says 172.21.15.167:33552) 2022-01-31T19:42:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765a46700 1 -- 172.21.15.167:0/2959090104 learned_addr learned my addr 172.21.15.167:0/2959090104 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765245700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97600f6da0 0x7f976010c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765a46700 1 -- 172.21.15.167:0/2959090104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97600f6da0 msgr2=0x7f976010c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.551 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765a46700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97600f6da0 0x7f976010c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765a46700 1 -- 172.21.15.167:0/2959090104 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c005040 con 0x7f97600f5b00 2022-01-31T19:42:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9765a46700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f9760060550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f975c004e30 tx=0x7f975c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975c018070 con 0x7f97600f5b00 2022-01-31T19:42:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f975c013410 con 0x7f97600f5b00 2022-01-31T19:42:17.552 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976010c9b0 con 0x7f97600f5b00 2022-01-31T19:42:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975c01c3c0 con 0x7f97600f5b00 2022-01-31T19:42:17.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.542+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976010cf10 con 0x7f97600f5b00 2022-01-31T19:42:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.543+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9760027ea0 con 0x7f97600f5b00 2022-01-31T19:42:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.543+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f975c01d420 con 0x7f97600f5b00 2022-01-31T19:42:17.555 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.544+0000 7f97567fc700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f974c05e5c0 0x7f974c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.555 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.544+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f975c07a4d0 con 0x7f97600f5b00 2022-01-31T19:42:17.555 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.544+0000 7f9765245700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f974c05e5c0 0x7f974c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.555 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.544+0000 7f9765245700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f974c05e5c0 0x7f974c060a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9750004c60 tx=0x7f975000b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:17.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.548+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f975c04c0d0 con 0x7f97600f5b00 2022-01-31T19:42:17.705 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.702+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f97600fe2e0 con 0x7f97600f5b00 2022-01-31T19:42:17.706 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.702+0000 7f97567fc700 1 -- 172.21.15.167:0/2959090104 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v40) v1 ==== 107+0+11 (secure 0 0 0) 0x7f975c04f6f0 con 0x7f97600f5b00 2022-01-31T19:42:17.706 INFO:teuthology.orchestra.run.smithi167.stdout:pg_num: 32 2022-01-31T19:42:17.708 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f974c05e5c0 msgr2=0x7f974c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f974c05e5c0 0x7f974c060a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9750004c60 tx=0x7f975000b040).stop 2022-01-31T19:42:17.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 msgr2=0x7f9760060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.709 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f9760060550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f975c004e30 tx=0x7f975c009b30).stop 2022-01-31T19:42:17.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 shutdown_connections 2022-01-31T19:42:17.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f974c05e5c0 0x7f974c060a80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97600f5b00 0x7f9760060550 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.711 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 --2- 172.21.15.167:0/2959090104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97600f6da0 0x7f976010c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.711 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 >> 172.21.15.167:0/2959090104 conn(0x7f97600f1260 msgr2=0x7f97600fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:17.712 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 shutdown_connections 2022-01-31T19:42:17.712 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.704+0000 7f9767caa700 1 -- 172.21.15.167:0/2959090104 wait complete. 2022-01-31T19:42:17.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-01-31T19:42:17.837 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 -- 172.21.15.167:0/3285283343 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 msgr2=0x7f03400f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.838 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 --2- 172.21.15.167:0/3285283343 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03400f62a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f033c004660 tx=0x7f033c009b30).stop 2022-01-31T19:42:17.838 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 -- 172.21.15.167:0/3285283343 shutdown_connections 2022-01-31T19:42:17.838 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 --2- 172.21.15.167:0/3285283343 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03400f62a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.838 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 --2- 172.21.15.167:0/3285283343 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03400f70d0 0x7f03400f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.839 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 -- 172.21.15.167:0/3285283343 >> 172.21.15.167:0/3285283343 conn(0x7f03400f0990 msgr2=0x7f03400f2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:17.839 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 -- 172.21.15.167:0/3285283343 shutdown_connections 2022-01-31T19:42:17.839 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.833+0000 7f0347980700 1 -- 172.21.15.167:0/3285283343 wait complete. 2022-01-31T19:42:17.839 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0347980700 1 Processor -- start 2022-01-31T19:42:17.839 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0347980700 1 -- start start 2022-01-31T19:42:17.840 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0347980700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03401073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.840 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0347980700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03400f70d0 0x7f0340107890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.840 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0347980700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340107e40 con 0x7f03400f5e20 2022-01-31T19:42:17.840 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0347980700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340107f80 con 0x7f03400f70d0 2022-01-31T19:42:17.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f034571c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03401073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f034571c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03401073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33562/0 (socket says 172.21.15.167:33562) 2022-01-31T19:42:17.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f034571c700 1 -- 172.21.15.167:0/812350588 learned_addr learned my addr 172.21.15.167:0/812350588 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:17.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f034571c700 1 -- 172.21.15.167:0/812350588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03400f70d0 msgr2=0x7f0340107890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:17.841 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f0344f1b700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03400f70d0 0x7f0340107890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f034571c700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03400f70d0 0x7f0340107890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:17.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.834+0000 7f034571c700 1 -- 172.21.15.167:0/812350588 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c005040 con 0x7f03400f5e20 2022-01-31T19:42:17.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.835+0000 7f034571c700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03401073a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0330002700 tx=0x7f0330009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:17.842 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.835+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0330009320 con 0x7f03400f5e20 2022-01-31T19:42:17.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.835+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f033000f040 con 0x7f03400f5e20 2022-01-31T19:42:17.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.835+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034010ca20 con 0x7f03400f5e20 2022-01-31T19:42:17.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.835+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03300146f0 con 0x7f03400f5e20 2022-01-31T19:42:17.843 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.835+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034010cfe0 con 0x7f03400f5e20 2022-01-31T19:42:17.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.836+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03400fe2f0 con 0x7f03400f5e20 2022-01-31T19:42:17.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.836+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f0330009480 con 0x7f03400f5e20 2022-01-31T19:42:17.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.836+0000 7f0335ffb700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f032c05e5c0 0x7f032c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:17.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.836+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0330078f60 con 0x7f03400f5e20 2022-01-31T19:42:17.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.837+0000 7f0344f1b700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f032c05e5c0 0x7f032c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:17.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.837+0000 7f0344f1b700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f032c05e5c0 0x7f032c060a80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f033c009d60 tx=0x7f033c009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:17.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:17.840+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f033004ab00 con 0x7f03400f5e20 2022-01-31T19:42:18.035 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.033+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f034010d9b0 con 0x7f03400f5e20 2022-01-31T19:42:18.036 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.034+0000 7f0335ffb700 1 -- 172.21.15.167:0/812350588 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3041 (secure 0 0 0) 0x7f033004e120 con 0x7f03400f5e20 2022-01-31T19:42:18.037 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:18.037 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-01-31T19:41:48.803120+0000","modified":"2022-01-31T19:42:01.874180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14436,"mds_1":24267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14436":{"gid":14436,"name":"cephfs.smithi167.sgnfie","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.167:6827/1333633577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1333633577},{"type":"v1","addr":"172.21.15.167:6827","nonce":1333633577}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14452":{"gid":14452,"name":"cephfs.smithi167.rdkazj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.167:6829/2852537983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":2852537983},{"type":"v1","addr":"172.21.15.167:6829","nonce":2852537983}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24267":{"gid":24267,"name":"cephfs.smithi171.dyqqtj","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.171:6825/4256455981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4256455981},{"type":"v1","addr":"172.21.15.171:6825","nonce":4256455981}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24279":{"gid":24279,"name":"cephfs.smithi171.erzfea","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.171:6827/1322376319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":1322376319},{"type":"v1","addr":"172.21.15.171:6827","nonce":1322376319}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2022-01-31T19:42:18.038 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f032c05e5c0 msgr2=0x7f032c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:18.038 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f032c05e5c0 0x7f032c060a80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f033c009d60 tx=0x7f033c009b30).stop 2022-01-31T19:42:18.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 msgr2=0x7f03401073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:18.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03401073a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0330002700 tx=0x7f0330009d60).stop 2022-01-31T19:42:18.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 shutdown_connections 2022-01-31T19:42:18.039 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f032c05e5c0 0x7f032c060a80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:18.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03400f5e20 0x7f03401073a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:18.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 --2- 172.21.15.167:0/812350588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03400f70d0 0x7f0340107890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:18.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 >> 172.21.15.167:0/812350588 conn(0x7f03400f0990 msgr2=0x7f03400f5530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:18.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 shutdown_connections 2022-01-31T19:42:18.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:18.036+0000 7f0347980700 1 -- 172.21.15.167:0/812350588 wait complete. 2022-01-31T19:42:18.041 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:18.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: cluster 2022-01-31T19:42:16.731055 2022-01-31T19:42:18.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 202 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:18.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: audit 2022-01-31T19:42:16.821530+0000 mon.smithi167 (mon.0) 591 : audit 2022-01-31T19:42:18.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: [DBG] from='client.? 172.21.15.167:0/2843557300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:42:18.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:42:18.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: T19:42:17.118775+0000 2022-01-31T19:42:18.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: mon.smithi167 (mon.0) 592 : audit [DBG] 2022-01-31T19:42:18.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: from='client.? 172.21.15.167:0/2984439920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-01-31T19:42:18.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: audit 2022-01-31T19:42:17.410083 2022-01-31T19:42:18.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 593 : audit [DBG] 2022-01-31T19:42:18.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: from='client.? 172.21.15.167:0/398052586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2022-01-31T19:42:18.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: audit 2022-01-31T19:42:18.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: 2022-01-31T19:42:17.704162+0000 mon.smithi167 (mon. 2022-01-31T19:42:18.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:17 smithi167 conmon[32206]: 0) 594 : audit [DBG] from='client.? 172.21.15.167:0/2959090104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2022-01-31T19:42:18.053 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 2, 'flags': 50} 2022-01-31T19:42:18.053 INFO:teuthology.run_tasks:Running task kclient... 2022-01-31T19:42:18.080 INFO:tasks.kclient:Mounting kernel clients... 2022-01-31T19:42:18.081 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2022-01-31T19:42:18.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:18.082 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2022-01-31T19:42:18.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:18.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link delete ceph-brx 2022-01-31T19:42:18.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: cluster 2022-01-31T19:42:16.731055+0000 mgr.smithi167.aciqpk (mgr.14156) 202 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: audit 2022-01-31T19:42:16.821530+0000 mon.smithi167 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.167:0/2843557300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-01-31T19:42:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: audit 2022-01-31T19:42:17.118775+0000 mon.smithi167 (mon.0) 592 : audit 2022-01-31T19:42:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: [DBG] from='client.? 172.21.15.167:0/2984439920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2022-01-31T19:42:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:42:18.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: :42:17.410083+0000 mon.smithi167 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.167:0/398052586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2022-01-31T19:42:18.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:42:18.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: 01-31T19:42:17.704162+0000 mon.smithi167 2022-01-31T19:42:18.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:17 smithi171 conmon[35325]: (mon.0) 594 : audit [DBG] from='client.? 172.21.15.167:0/2959090104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2022-01-31T19:42:18.167 INFO:teuthology.orchestra.run.smithi167.stderr:Cannot find device "ceph-brx" 2022-01-31T19:42:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-31T19:42:18.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:18.169 DEBUG:teuthology.orchestra.run.smithi171:> ip netns list 2022-01-31T19:42:18.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:18.192 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip link delete ceph-brx 2022-01-31T19:42:18.269 INFO:teuthology.orchestra.run.smithi171.stderr:Cannot find device "ceph-brx" 2022-01-31T19:42:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-31T19:42:18.271 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2022-01-31T19:42:18.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph fs ls 2022-01-31T19:42:18.715 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:42:19.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:18 smithi171 conmon[35325]: audit 2022-01-31T19:42:18.035730+0000 mon.smithi167 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.167:0/812350588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:19.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:18 smithi167 conmon[32206]: audit 2022-01-31T19:42:18.035730+0000 mon.smithi167 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.167:0/812350588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-01-31T19:42:19.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:19 smithi167 conmon[32206]: cluster 2022-01-31T19:42:18.731492+0000 mgr.smithi167.aciqpk (mgr.14156) 203 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:19.981 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 -- 172.21.15.167:0/3288924175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f6ea0 msgr2=0x7f5bc00f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:19.981 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/3288924175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc00f72c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5ba8004660 tx=0x7f5ba8009b30).stop 2022-01-31T19:42:19.982 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 -- 172.21.15.167:0/3288924175 shutdown_connections 2022-01-31T19:42:19.982 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/3288924175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f54b0 0x7f5bc00f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:19.982 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/3288924175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc00f72c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:19.982 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 -- 172.21.15.167:0/3288924175 >> 172.21.15.167:0/3288924175 conn(0x7f5bc00f0f50 msgr2=0x7f5bc00f3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:19.983 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 -- 172.21.15.167:0/3288924175 shutdown_connections 2022-01-31T19:42:19.983 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.977+0000 7f5bc48c2700 1 -- 172.21.15.167:0/3288924175 wait complete. 2022-01-31T19:42:19.983 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bc48c2700 1 Processor -- start 2022-01-31T19:42:19.983 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bc48c2700 1 -- start start 2022-01-31T19:42:19.984 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bc48c2700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 0x7f5bc0109f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:19.984 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bc48c2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc010c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:19.984 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bc48c2700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0060a00 con 0x7f5bc00f54b0 2022-01-31T19:42:19.984 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bc48c2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0060b40 con 0x7f5bc00f6ea0 2022-01-31T19:42:19.984 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbe59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 0x7f5bc0109f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:19.985 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbe59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 0x7f5bc0109f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33574/0 (socket says 172.21.15.167:33574) 2022-01-31T19:42:19.985 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbe59c700 1 -- 172.21.15.167:0/1415391194 learned_addr learned my addr 172.21.15.167:0/1415391194 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:19.985 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbdd9b700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc010c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:19.985 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbe59c700 1 -- 172.21.15.167:0/1415391194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f6ea0 msgr2=0x7f5bc010c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:19.986 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbe59c700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc010c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:19.986 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbe59c700 1 -- 172.21.15.167:0/1415391194 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba8005040 con 0x7f5bc00f54b0 2022-01-31T19:42:19.986 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.978+0000 7f5bbdd9b700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc010c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:42:19.986 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.979+0000 7f5bbe59c700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 0x7f5bc0109f90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5ba8007a60 tx=0x7f5ba8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:19.987 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.979+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba8018070 con 0x7f5bc00f54b0 2022-01-31T19:42:19.987 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.979+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc010c9e0 con 0x7f5bc00f54b0 2022-01-31T19:42:19.987 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.979+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5ba8013410 con 0x7f5bc00f54b0 2022-01-31T19:42:19.987 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.979+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba801c620 con 0x7f5bc00f54b0 2022-01-31T19:42:19.987 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.979+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc010cf10 con 0x7f5bc00f54b0 2022-01-31T19:42:19.989 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.980+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc00f4be0 con 0x7f5bc00f54b0 2022-01-31T19:42:19.989 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.980+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f5ba801d940 con 0x7f5bc00f54b0 2022-01-31T19:42:19.989 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.980+0000 7f5bb6ffd700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5bac05e5c0 0x7f5bac060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:19.990 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.980+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5ba807a810 con 0x7f5bc00f54b0 2022-01-31T19:42:19.990 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.980+0000 7f5bbdd9b700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5bac05e5c0 0x7f5bac060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:19.990 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.981+0000 7f5bbdd9b700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5bac05e5c0 0x7f5bac060a80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5bb0004a70 tx=0x7f5bb0009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:19.990 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:19.984+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5ba804c410 con 0x7f5bc00f54b0 2022-01-31T19:42:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:19 smithi171 conmon[35325]: cluster 2022-01-31T19:42:18.731492+0000 mgr.smithi167.aciqpk (mgr.14156) 203 : cluster 2022-01-31T19:42:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:19 smithi171 conmon[35325]: [DBG] pgmap v160: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:20.175 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.173+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5bc00fe090 con 0x7f5bc00f54b0 2022-01-31T19:42:20.176 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.173+0000 7f5bb6ffd700 1 -- 172.21.15.167:0/1415391194 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f5ba804fa30 con 0x7f5bc00f54b0 2022-01-31T19:42:20.177 INFO:teuthology.orchestra.run.smithi167.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2022-01-31T19:42:20.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5bac05e5c0 msgr2=0x7f5bac060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:20.179 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5bac05e5c0 0x7f5bac060a80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5bb0004a70 tx=0x7f5bb0009040).stop 2022-01-31T19:42:20.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 msgr2=0x7f5bc0109f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:20.180 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 0x7f5bc0109f90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5ba8007a60 tx=0x7f5ba8009b30).stop 2022-01-31T19:42:20.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 shutdown_connections 2022-01-31T19:42:20.181 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5bac05e5c0 0x7f5bac060a80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:20.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bc00f54b0 0x7f5bc0109f90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:20.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 --2- 172.21.15.167:0/1415391194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5bc00f6ea0 0x7f5bc010c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:20.182 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 >> 172.21.15.167:0/1415391194 conn(0x7f5bc00f0f50 msgr2=0x7f5bc00fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 shutdown_connections 2022-01-31T19:42:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:20.175+0000 7f5bc48c2700 1 -- 172.21.15.167:0/1415391194 wait complete. 2022-01-31T19:42:20.719 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-01-31T19:42:20.720 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-01-31T19:42:20.721 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi167.front.sepia.ceph.com 2022-01-31T19:42:20.722 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-01-31T19:42:20.722 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:20.723 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-01-31T19:42:20.723 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-01-31T19:42:20.724 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-01-31T19:42:20.725 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-01-31T19:42:20.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:20.726 DEBUG:teuthology.orchestra.run.smithi167:> ip addr 2022-01-31T19:42:20.745 INFO:teuthology.orchestra.run.smithi167.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-01-31T19:42:20.753 INFO:teuthology.orchestra.run.smithi167.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-01-31T19:42:20.754 INFO:teuthology.orchestra.run.smithi167.stdout: inet 127.0.0.1/8 scope host lo 2022-01-31T19:42:20.754 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2022-01-31T19:42:20.754 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 ::1/128 scope host 2022-01-31T19:42:20.754 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2022-01-31T19:42:20.755 INFO:teuthology.orchestra.run.smithi167.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-01-31T19:42:20.755 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:d9:35:e6 brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:20.755 INFO:teuthology.orchestra.run.smithi167.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-01-31T19:42:20.755 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:d9:35:e7 brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:20.756 INFO:teuthology.orchestra.run.smithi167.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-01-31T19:42:20.756 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:88:73:08 brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:20.756 INFO:teuthology.orchestra.run.smithi167.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-01-31T19:42:20.756 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:88:73:09 brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:20.757 INFO:teuthology.orchestra.run.smithi167.stdout: inet 172.21.15.167/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-01-31T19:42:20.757 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft 42389sec preferred_lft 42389sec 2022-01-31T19:42:20.757 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::ec4:7aff:fe88:7309/64 scope link noprefixroute 2022-01-31T19:42:20.757 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2022-01-31T19:42:20.758 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-01-31T19:42:20.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:20.759 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec bash -c ' 2022-01-31T19:42:20.759 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2022-01-31T19:42:20.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link add name ceph-brx type bridge 2022-01-31T19:42:20.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip addr flush dev ceph-brx 2022-01-31T19:42:20.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set ceph-brx up 2022-01-31T19:42:20.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-01-31T19:42:20.762 DEBUG:teuthology.orchestra.run.smithi167:> ') 2022-01-31T19:42:20.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:20.913 DEBUG:teuthology.orchestra.run.smithi167:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-01-31T19:42:20.940 INFO:teuthology.orchestra.run.smithi167.stdout:1 2022-01-31T19:42:20.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:20.942 DEBUG:teuthology.orchestra.run.smithi167:> route 2022-01-31T19:42:21.007 INFO:teuthology.orchestra.run.smithi167.stdout:Kernel IP routing table 2022-01-31T19:42:21.007 INFO:teuthology.orchestra.run.smithi167.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-01-31T19:42:21.008 INFO:teuthology.orchestra.run.smithi167.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-01-31T19:42:21.008 INFO:teuthology.orchestra.run.smithi167.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-01-31T19:42:21.008 INFO:teuthology.orchestra.run.smithi167.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-01-31T19:42:21.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:21.011 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec bash -c ' 2022-01-31T19:42:21.011 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2022-01-31T19:42:21.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-01-31T19:42:21.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-01-31T19:42:21.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-01-31T19:42:21.012 DEBUG:teuthology.orchestra.run.smithi167:> ') 2022-01-31T19:42:21.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:20 smithi167 conmon[32206]: audit 2022-01-31T19:42:20.174998+0000 mon.smithi167 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.167:0/1415391194' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-01-31T19:42:21.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:20 smithi171 conmon[35325]: audit 2022-01-31T19:42:20.174998+0000 mon.smithi167 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.167:0/1415391194' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-01-31T19:42:21.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:20 smithi171 conmon[35325]: 2022-01-31T19:42:21.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:21.324 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2022-01-31T19:42:21.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:21.381 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list-id 2022-01-31T19:42:21.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:21.437 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec bash -c ' 2022-01-31T19:42:21.438 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2022-01-31T19:42:21.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-01-31T19:42:21.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-01-31T19:42:21.439 DEBUG:teuthology.orchestra.run.smithi167:> ') 2022-01-31T19:42:21.530 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-01-31T19:42:21.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:21.532 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec bash -c ' 2022-01-31T19:42:21.532 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2022-01-31T19:42:21.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-01-31T19:42:21.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-01-31T19:42:21.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-01-31T19:42:21.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-01-31T19:42:21.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-01-31T19:42:21.534 DEBUG:teuthology.orchestra.run.smithi167:> ') 2022-01-31T19:42:21.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:21.759 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec bash -c ' 2022-01-31T19:42:21.760 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2022-01-31T19:42:21.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set brx.0 up 2022-01-31T19:42:21.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set dev brx.0 master ceph-brx 2022-01-31T19:42:21.761 DEBUG:teuthology.orchestra.run.smithi167:> ') 2022-01-31T19:42:21.878 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-01-31T19:42:21.879 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:21.899 INFO:teuthology.orchestra.run.smithi167.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-01-31T19:42:21.900 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-01-31T19:42:21.901 DEBUG:teuthology.orchestra.run.smithi167:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:21.956 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2022-01-31T19:42:21.957 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-01-31T19:42:21.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-01-31T19:42:22.104 INFO:teuthology.orchestra.run.smithi167.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-01-31T19:42:22.105 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs" will pass to kernel. 2022-01-31T19:42:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:21 smithi171 conmon[35325]: cluster 2022-01-31T19:42:20.731964+0000 mgr.smithi167.aciqpk (mgr.14156) 204 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:22.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:21 smithi167 conmon[32206]: cluster 2022-01-31T19:42:20.731964+0000 mgr.smithi167.aciqpk (mgr.14156) 204 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:22.229 INFO:teuthology.orchestra.run.smithi167.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-01-31T19:42:22.229 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs" will pass to kernel. 2022-01-31T19:42:22.234 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-01-31T19:42:22.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:22.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:22.267 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2022-01-31T19:42:22.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -- ceph fs ls 2022-01-31T19:42:22.749 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/1c5a1df6-82cd-11ec-8c35-001a4aab830c/mon.smithi167/config 2022-01-31T19:42:23.946 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 -- 172.21.15.167:0/3567229405 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f68f0 msgr2=0x7f8adc0f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:23.946 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/3567229405 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f68f0 0x7f8adc0f6d10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f8ad0004660 tx=0x7f8ad0009b30).stop 2022-01-31T19:42:23.947 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 -- 172.21.15.167:0/3567229405 shutdown_connections 2022-01-31T19:42:23.947 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/3567229405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8adc0f7b90 0x7f8adc0f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:23.947 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/3567229405 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f68f0 0x7f8adc0f6d10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:23.947 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 -- 172.21.15.167:0/3567229405 >> 172.21.15.167:0/3567229405 conn(0x7f8adc0f2090 msgr2=0x7f8adc0f44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:23.948 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 -- 172.21.15.167:0/3567229405 shutdown_connections 2022-01-31T19:42:23.948 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 -- 172.21.15.167:0/3567229405 wait complete. 2022-01-31T19:42:23.948 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.942+0000 7f8ae1faf700 1 Processor -- start 2022-01-31T19:42:23.948 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8ae1faf700 1 -- start start 2022-01-31T19:42:23.949 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8ae1faf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8adc0f68f0 0x7f8adc107160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:23.949 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8ae1faf700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 0x7f8adc107650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:23.950 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8ae1faf700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc107c00 con 0x7f8adc0f7b90 2022-01-31T19:42:23.950 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8ae1faf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc107d40 con 0x7f8adc0f68f0 2022-01-31T19:42:23.950 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adaffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 0x7f8adc107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:23.950 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adaffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 0x7f8adc107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33600/0 (socket says 172.21.15.167:33600) 2022-01-31T19:42:23.951 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adaffd700 1 -- 172.21.15.167:0/2904618633 learned_addr learned my addr 172.21.15.167:0/2904618633 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:23.951 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adb7fe700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8adc0f68f0 0x7f8adc107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:23.951 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adaffd700 1 -- 172.21.15.167:0/2904618633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8adc0f68f0 msgr2=0x7f8adc107160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adaffd700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8adc0f68f0 0x7f8adc107160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.943+0000 7f8adaffd700 1 -- 172.21.15.167:0/2904618633 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0005040 con 0x7f8adc0f7b90 2022-01-31T19:42:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.944+0000 7f8adaffd700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 0x7f8adc107650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8acc00c530 tx=0x7f8acc007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.944+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8acc00f650 con 0x7f8adc0f7b90 2022-01-31T19:42:23.953 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.944+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc1086e0 con 0x7f8adc0f7b90 2022-01-31T19:42:23.953 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.944+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8acc00fc40 con 0x7f8adc0f7b90 2022-01-31T19:42:23.953 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.944+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8acc012c40 con 0x7f8adc0f7b90 2022-01-31T19:42:23.953 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.944+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc108ca0 con 0x7f8adc0f7b90 2022-01-31T19:42:23.955 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.945+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc045bb0 con 0x7f8adc0f7b90 2022-01-31T19:42:23.955 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.945+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f8acc010070 con 0x7f8adc0f7b90 2022-01-31T19:42:23.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.945+0000 7f8ac3fff700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8ac405e5c0 0x7f8ac4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:23.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.945+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8acc078680 con 0x7f8adc0f7b90 2022-01-31T19:42:23.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.946+0000 7f8adb7fe700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8ac405e5c0 0x7f8ac4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:23.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.946+0000 7f8adb7fe700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8ac405e5c0 0x7f8ac4060a80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8ad0013580 tx=0x7f8ad0009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:23.957 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:23.950+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8acc0456b0 con 0x7f8adc0f7b90 2022-01-31T19:42:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:23 smithi171 conmon[35325]: cluster 2022-01-31T19:42:22.732506+0000 mgr.smithi167.aciqpk (mgr.14156) 205 : cluster [DBG] 2022-01-31T19:42:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:23 smithi171 conmon[35325]: pgmap v162: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:24.146 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.144+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8adc027ae0 con 0x7f8adc0f7b90 2022-01-31T19:42:24.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:23 smithi167 conmon[32206]: cluster 2022-01-31T19:42:22 2022-01-31T19:42:24.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:23 smithi167 conmon[32206]: .732506+0000 mgr.smithi167.aciqpk (mgr.14156) 205 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:24.147 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.144+0000 7f8ac3fff700 1 -- 172.21.15.167:0/2904618633 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f8acc04dcb0 con 0x7f8adc0f7b90 2022-01-31T19:42:24.148 INFO:teuthology.orchestra.run.smithi167.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2022-01-31T19:42:24.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8ac405e5c0 msgr2=0x7f8ac4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:24.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8ac405e5c0 0x7f8ac4060a80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8ad0013580 tx=0x7f8ad0009b30).stop 2022-01-31T19:42:24.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 msgr2=0x7f8adc107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:24.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 0x7f8adc107650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8acc00c530 tx=0x7f8acc007e20).stop 2022-01-31T19:42:24.151 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 shutdown_connections 2022-01-31T19:42:24.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f8ac405e5c0 0x7f8ac4060a80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:24.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8adc0f68f0 0x7f8adc107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:24.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 --2- 172.21.15.167:0/2904618633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8adc0f7b90 0x7f8adc107650 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:24.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 >> 172.21.15.167:0/2904618633 conn(0x7f8adc0f2090 msgr2=0x7f8adc0fb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:24.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 shutdown_connections 2022-01-31T19:42:24.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:24.147+0000 7f8ae1faf700 1 -- 172.21.15.167:0/2904618633 wait complete. 2022-01-31T19:42:24.569 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-01-31T19:42:24.569 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-01-31T19:42:24.570 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi171.front.sepia.ceph.com 2022-01-31T19:42:24.570 INFO:tasks.cephfs.mount:self.client.name = client.1 2022-01-31T19:42:24.570 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:24.570 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-01-31T19:42:24.571 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-01-31T19:42:24.571 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-01-31T19:42:24.571 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2022-01-31T19:42:24.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:24.572 DEBUG:teuthology.orchestra.run.smithi171:> ip addr 2022-01-31T19:42:24.593 INFO:teuthology.orchestra.run.smithi171.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-01-31T19:42:24.594 INFO:teuthology.orchestra.run.smithi171.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-01-31T19:42:24.594 INFO:teuthology.orchestra.run.smithi171.stdout: inet 127.0.0.1/8 scope host lo 2022-01-31T19:42:24.595 INFO:teuthology.orchestra.run.smithi171.stdout: valid_lft forever preferred_lft forever 2022-01-31T19:42:24.595 INFO:teuthology.orchestra.run.smithi171.stdout: inet6 ::1/128 scope host 2022-01-31T19:42:24.595 INFO:teuthology.orchestra.run.smithi171.stdout: valid_lft forever preferred_lft forever 2022-01-31T19:42:24.595 INFO:teuthology.orchestra.run.smithi171.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-01-31T19:42:24.596 INFO:teuthology.orchestra.run.smithi171.stdout: link/ether 0c:c4:7a:d9:39:a2 brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:24.596 INFO:teuthology.orchestra.run.smithi171.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-01-31T19:42:24.596 INFO:teuthology.orchestra.run.smithi171.stdout: link/ether 0c:c4:7a:d9:39:a3 brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:24.596 INFO:teuthology.orchestra.run.smithi171.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-01-31T19:42:24.597 INFO:teuthology.orchestra.run.smithi171.stdout: link/ether 0c:c4:7a:88:6f:6c brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:24.597 INFO:teuthology.orchestra.run.smithi171.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-01-31T19:42:24.597 INFO:teuthology.orchestra.run.smithi171.stdout: link/ether 0c:c4:7a:88:6f:6d brd ff:ff:ff:ff:ff:ff 2022-01-31T19:42:24.597 INFO:teuthology.orchestra.run.smithi171.stdout: inet 172.21.15.171/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-01-31T19:42:24.597 INFO:teuthology.orchestra.run.smithi171.stdout: valid_lft 42384sec preferred_lft 42384sec 2022-01-31T19:42:24.598 INFO:teuthology.orchestra.run.smithi171.stdout: inet6 fe80::ec4:7aff:fe88:6f6d/64 scope link noprefixroute 2022-01-31T19:42:24.598 INFO:teuthology.orchestra.run.smithi171.stdout: valid_lft forever preferred_lft forever 2022-01-31T19:42:24.598 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-01-31T19:42:24.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:24.600 DEBUG:teuthology.orchestra.run.smithi171:> (cd / && exec bash -c ' 2022-01-31T19:42:24.600 DEBUG:teuthology.orchestra.run.smithi171:> set -e 2022-01-31T19:42:24.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip link add name ceph-brx type bridge 2022-01-31T19:42:24.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip addr flush dev ceph-brx 2022-01-31T19:42:24.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip link set ceph-brx up 2022-01-31T19:42:24.602 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-01-31T19:42:24.602 DEBUG:teuthology.orchestra.run.smithi171:> ') 2022-01-31T19:42:24.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:24.772 DEBUG:teuthology.orchestra.run.smithi171:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-01-31T19:42:24.801 INFO:teuthology.orchestra.run.smithi171.stdout:1 2022-01-31T19:42:24.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:24.802 DEBUG:teuthology.orchestra.run.smithi171:> route 2022-01-31T19:42:24.873 INFO:teuthology.orchestra.run.smithi171.stdout:Kernel IP routing table 2022-01-31T19:42:24.873 INFO:teuthology.orchestra.run.smithi171.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-01-31T19:42:24.873 INFO:teuthology.orchestra.run.smithi171.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-01-31T19:42:24.874 INFO:teuthology.orchestra.run.smithi171.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-01-31T19:42:24.874 INFO:teuthology.orchestra.run.smithi171.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-01-31T19:42:24.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:24.876 DEBUG:teuthology.orchestra.run.smithi171:> (cd / && exec bash -c ' 2022-01-31T19:42:24.876 DEBUG:teuthology.orchestra.run.smithi171:> set -e 2022-01-31T19:42:24.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-01-31T19:42:24.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-01-31T19:42:24.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-01-31T19:42:24.877 DEBUG:teuthology.orchestra.run.smithi171:> ') 2022-01-31T19:42:24.951 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:24 smithi171 conmon[35325]: audit 2022-01-31T19:42:24.146012+0000 mon.smithi167 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.167:0/2904618633' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-01-31T19:42:25.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:25.167 DEBUG:teuthology.orchestra.run.smithi171:> ip netns list 2022-01-31T19:42:25.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:24 smithi167 conmon[32206]: audit 2022-01-31T19:42:24.146012+0000 mon.smithi167 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.167:0/2904618633' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-01-31T19:42:25.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:25.223 DEBUG:teuthology.orchestra.run.smithi171:> ip netns list-id 2022-01-31T19:42:25.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:25.278 DEBUG:teuthology.orchestra.run.smithi171:> (cd / && exec bash -c ' 2022-01-31T19:42:25.279 DEBUG:teuthology.orchestra.run.smithi171:> set -e 2022-01-31T19:42:25.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2022-01-31T19:42:25.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2022-01-31T19:42:25.280 DEBUG:teuthology.orchestra.run.smithi171:> ') 2022-01-31T19:42:25.370 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2022-01-31T19:42:25.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:25.371 DEBUG:teuthology.orchestra.run.smithi171:> (cd / && exec bash -c ' 2022-01-31T19:42:25.372 DEBUG:teuthology.orchestra.run.smithi171:> set -e 2022-01-31T19:42:25.372 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2022-01-31T19:42:25.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-01-31T19:42:25.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2022-01-31T19:42:25.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2022-01-31T19:42:25.374 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2022-01-31T19:42:25.374 DEBUG:teuthology.orchestra.run.smithi171:> ') 2022-01-31T19:42:25.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:25.572 DEBUG:teuthology.orchestra.run.smithi171:> (cd / && exec bash -c ' 2022-01-31T19:42:25.572 DEBUG:teuthology.orchestra.run.smithi171:> set -e 2022-01-31T19:42:25.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip link set brx.0 up 2022-01-31T19:42:25.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo ip link set dev brx.0 master ceph-brx 2022-01-31T19:42:25.573 DEBUG:teuthology.orchestra.run.smithi171:> ') 2022-01-31T19:42:25.683 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-01-31T19:42:25.684 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:25.702 INFO:teuthology.orchestra.run.smithi171.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2022-01-31T19:42:25.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-01-31T19:42:25.704 DEBUG:teuthology.orchestra.run.smithi171:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:25.758 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2022-01-31T19:42:25.759 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-01-31T19:42:25.759 DEBUG:teuthology.orchestra.run.smithi171:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2022-01-31T19:42:25.889 INFO:teuthology.orchestra.run.smithi171.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2022-01-31T19:42:25.889 INFO:teuthology.orchestra.run.smithi171.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs" will pass to kernel. 2022-01-31T19:42:25.905 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:25 smithi171 conmon[35325]: cluster 2022-01-31T19:42:24.732929+0000 2022-01-31T19:42:25.905 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:25 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 206 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 4 op/s 2022-01-31T19:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2022-01-31T19:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs" will pass to kernel. 2022-01-31T19:42:26.022 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-01-31T19:42:26.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-01-31T19:42:26.023 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:26.049 INFO:teuthology.run_tasks:Running task print... 2022-01-31T19:42:26.065 INFO:teuthology.task.print:**** done client 2022-01-31T19:42:26.065 INFO:teuthology.run_tasks:Running task parallel... 2022-01-31T19:42:26.081 INFO:teuthology.task.parallel:starting parallel... 2022-01-31T19:42:26.082 INFO:teuthology.task.parallel:In parallel, running task sequential... 2022-01-31T19:42:26.082 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2022-01-31T19:42:26.083 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:42:26.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2022-01-31T19:42:26.084 INFO:teuthology.task.parallel:In parallel, running task sequential... 2022-01-31T19:42:26.085 INFO:teuthology.task.sequential:In sequential, running task workunit... 2022-01-31T19:42:26.088 INFO:tasks.workunit:Pulling workunits from ref e7bdcb25f366a81de92a121020949aeab8ce71c7 2022-01-31T19:42:26.090 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-01-31T19:42:26.091 INFO:tasks.workunit:timeout=3h 2022-01-31T19:42:26.091 INFO:tasks.workunit:cleanup=True 2022-01-31T19:42:26.091 DEBUG:teuthology.orchestra.run.smithi167:> stat -- /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:26.114 INFO:teuthology.orchestra.run.smithi167.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:26.114 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-01-31T19:42:26.114 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 8eh/142d Inode: 1 Links: 2 2022-01-31T19:42:26.115 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:42:26.115 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:cephfs_t:s0 2022-01-31T19:42:26.115 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-01-31T19:42:26.116 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2022-01-31 19:41:51.836990669 +0000 2022-01-31T19:42:26.116 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2022-01-31 19:42:22.263713285 +0000 2022-01-31T19:42:26.116 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2022-01-31 19:41:51.836990669 +0000 2022-01-31T19:42:26.116 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-01-31T19:42:26.117 DEBUG:teuthology.orchestra.run.smithi167:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-01-31T19:42:26.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:25 smithi167 conmon[32206]: cluster 2022-01-31T19:42:24.732929+0000 2022-01-31T19:42:26.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:25 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 206 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 4 op/s 2022-01-31T19:42:26.157 DEBUG:teuthology.orchestra.run.smithi171:> stat -- /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:26.175 INFO:teuthology.orchestra.run.smithi171.stdout: File: /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:26.175 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2022-01-31T19:42:26.176 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 78h/120d Inode: 1 Links: 3 2022-01-31T19:42:26.176 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-31T19:42:26.176 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:cephfs_t:s0 2022-01-31T19:42:26.176 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-01-31T19:42:26.177 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2022-01-31 19:42:26.150637876 +0000 2022-01-31T19:42:26.177 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2022-01-31 19:42:26.150637876 +0000 2022-01-31T19:42:26.178 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2022-01-31 19:41:51.836990669 +0000 2022-01-31T19:42:26.178 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2022-01-31T19:42:26.178 DEBUG:teuthology.orchestra.run.smithi171:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2022-01-31T19:42:26.245 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e7bdcb25f366a81de92a121020949aeab8ce71c7 2022-01-31T19:42:26.246 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout e7bdcb25f366a81de92a121020949aeab8ce71c7 2022-01-31T19:42:26.262 INFO:tasks.workunit.client.0.smithi167.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-01-31T19:42:26.533 INFO:tasks.workunit.client.1.smithi171.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2022-01-31T19:42:27.731 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.727+0000 7fcccb777700 1 -- 172.21.15.167:0/2566222974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 msgr2=0x7fccc40f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:27.732 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.727+0000 7fcccb777700 1 --2- 172.21.15.167:0/2566222974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc40f6de0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fccb4004660 tx=0x7fccb4009b30).stop 2022-01-31T19:42:27.732 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.727+0000 7fcccb777700 1 -- 172.21.15.167:0/2566222974 shutdown_connections 2022-01-31T19:42:27.733 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.727+0000 7fcccb777700 1 --2- 172.21.15.167:0/2566222974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc40f6de0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:27.733 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.727+0000 7fcccb777700 1 --2- 172.21.15.167:0/2566222974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fccc40f56e0 0x7fccc40f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:27.733 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.727+0000 7fcccb777700 1 -- 172.21.15.167:0/2566222974 >> 172.21.15.167:0/2566222974 conn(0x7fccc40f0e60 msgr2=0x7fccc40f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:27.733 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 -- 172.21.15.167:0/2566222974 shutdown_connections 2022-01-31T19:42:27.734 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 -- 172.21.15.167:0/2566222974 wait complete. 2022-01-31T19:42:27.734 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 Processor -- start 2022-01-31T19:42:27.734 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 -- start start 2022-01-31T19:42:27.734 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fccc40f56e0 0x7fccc4107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:27.735 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc4107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:27.735 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4107b20 con 0x7fccc40f6980 2022-01-31T19:42:27.735 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fcccb777700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4107c60 con 0x7fccc40f56e0 2022-01-31T19:42:27.735 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fccc8d12700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc4107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fccc8d12700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc4107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33624/0 (socket says 172.21.15.167:33624) 2022-01-31T19:42:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.728+0000 7fccc8d12700 1 -- 172.21.15.167:0/1727283640 learned_addr learned my addr 172.21.15.167:0/1727283640 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccc9513700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fccc40f56e0 0x7fccc4107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccc8d12700 1 -- 172.21.15.167:0/1727283640 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fccc40f56e0 msgr2=0x7fccc4107080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccc8d12700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fccc40f56e0 0x7fccc4107080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccc8d12700 1 -- 172.21.15.167:0/1727283640 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4005040 con 0x7fccc40f6980 2022-01-31T19:42:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccc8d12700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc4107570 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fccb4009990 tx=0x7fccb4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccb4018070 con 0x7fccc40f6980 2022-01-31T19:42:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc41085a0 con 0x7fccc40f6980 2022-01-31T19:42:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fccb4013410 con 0x7fccc40f6980 2022-01-31T19:42:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccb401de80 con 0x7fccc40f6980 2022-01-31T19:42:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.729+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4108b00 con 0x7fccc40f6980 2022-01-31T19:42:27.739 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.730+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc4045bb0 con 0x7fccc40f6980 2022-01-31T19:42:27.740 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.731+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fccb401d420 con 0x7fccc40f6980 2022-01-31T19:42:27.740 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.731+0000 7fccb9ffb700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fccb005e5c0 0x7fccb0060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:27.740 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.731+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fccb407a480 con 0x7fccc40f6980 2022-01-31T19:42:27.740 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.731+0000 7fccc9513700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fccb005e5c0 0x7fccb0060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:27.741 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.732+0000 7fccc9513700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fccb005e5c0 0x7fccb0060a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fccc40f67e0 tx=0x7fccc000b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:27.741 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.735+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fccb404c080 con 0x7fccc40f6980 2022-01-31T19:42:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.888+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fccc40fe020 con 0x7fccc40f6980 2022-01-31T19:42:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.888+0000 7fccb9ffb700 1 -- 172.21.15.167:0/1727283640 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fccb404f6a0 con 0x7fccc40f6980 2022-01-31T19:42:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fccb005e5c0 msgr2=0x7fccb0060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fccb005e5c0 0x7fccb0060a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fccc40f67e0 tx=0x7fccc000b040).stop 2022-01-31T19:42:27.893 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 msgr2=0x7fccc4107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:27.893 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc4107570 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fccb4009990 tx=0x7fccb4009b30).stop 2022-01-31T19:42:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 shutdown_connections 2022-01-31T19:42:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fccb005e5c0 0x7fccb0060a80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fccc40f56e0 0x7fccc4107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 --2- 172.21.15.167:0/1727283640 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fccc40f6980 0x7fccc4107570 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 >> 172.21.15.167:0/1727283640 conn(0x7fccc40f0e60 msgr2=0x7fccc405f700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 shutdown_connections 2022-01-31T19:42:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:27.890+0000 7fcccb777700 1 -- 172.21.15.167:0/1727283640 wait complete. 2022-01-31T19:42:28.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:27 smithi167 conmon[32206]: cluster 2022-01-31T19:42:26.733461+0000 mgr.smithi167.aciqpk (mgr.14156) 207 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2022-01-31T19:42:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:27 smithi171 conmon[35325]: cluster 2022-01-31T19:42:26.733461+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:42:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:27 smithi171 conmon[35325]: ) 207 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2022-01-31T19:42:28.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2022-01-31T19:42:30.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:29 smithi167 conmon[32206]: cluster 2022-01-31T19:42:28.733991+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:42:30.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:29 smithi167 conmon[32206]: ) 208 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-01-31T19:42:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.043+0000 7f39b1a5c700 1 -- 172.21.15.167:0/2071983836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 msgr2=0x7f39ac0f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.043+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/2071983836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac0f6a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f399c0099e0 tx=0x7f399c004d10).stop 2022-01-31T19:42:30.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.043+0000 7f39b1a5c700 1 -- 172.21.15.167:0/2071983836 shutdown_connections 2022-01-31T19:42:30.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.043+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/2071983836 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac0f78f0 0x7f39ac0f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:30.053 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.043+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/2071983836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac0f6a70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:30.054 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.043+0000 7f39b1a5c700 1 -- 172.21.15.167:0/2071983836 >> 172.21.15.167:0/2071983836 conn(0x7f39ac0f1db0 msgr2=0x7f39ac0f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:30.054 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.044+0000 7f39b1a5c700 1 -- 172.21.15.167:0/2071983836 shutdown_connections 2022-01-31T19:42:30.055 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.044+0000 7f39b1a5c700 1 -- 172.21.15.167:0/2071983836 wait complete. 2022-01-31T19:42:30.055 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.044+0000 7f39b1a5c700 1 Processor -- start 2022-01-31T19:42:30.056 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.044+0000 7f39b1a5c700 1 -- start start 2022-01-31T19:42:30.056 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b1a5c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:30.057 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b1a5c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac0f78f0 0x7f39ac069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:30.057 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b1a5c700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac06a460 con 0x7f39ac0f6650 2022-01-31T19:42:30.057 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b1a5c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac06a5a0 con 0x7f39ac0f78f0 2022-01-31T19:42:30.058 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:30.058 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33634/0 (socket says 172.21.15.167:33634) 2022-01-31T19:42:30.058 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 -- 172.21.15.167:0/870174066 learned_addr learned my addr 172.21.15.167:0/870174066 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:30.059 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39abfff700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac0f78f0 0x7f39ac069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:30.059 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 -- 172.21.15.167:0/870174066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac0f78f0 msgr2=0x7f39ac069f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:30.059 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac0f78f0 0x7f39ac069f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:30.059 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 -- 172.21.15.167:0/870174066 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c009640 con 0x7f39ac0f6650 2022-01-31T19:42:30.060 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39b0a5a700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac06ba80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f399c00b3b0 tx=0x7f399c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:30.060 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.045+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f399c00e030 con 0x7f39ac0f6650 2022-01-31T19:42:30.060 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.046+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac06a820 con 0x7f39ac0f6650 2022-01-31T19:42:30.061 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.046+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f399c007d40 con 0x7f39ac0f6650 2022-01-31T19:42:30.061 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.046+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f399c018070 con 0x7f39ac0f6650 2022-01-31T19:42:30.062 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.046+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac06ad00 con 0x7f39ac0f6650 2022-01-31T19:42:30.062 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.047+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f399c01d420 con 0x7f39ac0f6650 2022-01-31T19:42:30.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.047+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac045bb0 con 0x7f39ac0f6650 2022-01-31T19:42:30.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.047+0000 7f39a97fa700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f399805e5c0 0x7f3998060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:30.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.047+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f399c07a5a0 con 0x7f39ac0f6650 2022-01-31T19:42:30.063 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.047+0000 7f39abfff700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f399805e5c0 0x7f3998060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:30.064 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.048+0000 7f39abfff700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f399805e5c0 0x7f3998060a80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3994004e20 tx=0x7f399400b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:30.065 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.052+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f399c04c1a0 con 0x7f39ac0f6650 2022-01-31T19:42:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:29 smithi171 conmon[35325]: cluster 2022-01-31T19:42:28.733991+0000 mgr.smithi167.aciqpk (mgr.14156) 208 : cluster [DBG] 2022-01-31T19:42:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:29 smithi171 conmon[35325]: pgmap v165: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2022-01-31T19:42:30.209 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.206+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f39ac027ae0 con 0x7f39ac0f6650 2022-01-31T19:42:30.210 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.207+0000 7f39a97fa700 1 -- 172.21.15.167:0/870174066 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f399c04f7c0 con 0x7f39ac0f6650 2022-01-31T19:42:30.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f399805e5c0 msgr2=0x7f3998060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:30.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f399805e5c0 0x7f3998060a80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3994004e20 tx=0x7f399400b040).stop 2022-01-31T19:42:30.212 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 msgr2=0x7f39ac06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac06ba80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f399c00b3b0 tx=0x7f399c004d10).stop 2022-01-31T19:42:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 shutdown_connections 2022-01-31T19:42:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f399805e5c0 0x7f3998060a80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39ac0f6650 0x7f39ac06ba80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:30.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 --2- 172.21.15.167:0/870174066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39ac0f78f0 0x7f39ac069f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:30.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 >> 172.21.15.167:0/870174066 conn(0x7f39ac0f1db0 msgr2=0x7f39ac0fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:30.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 shutdown_connections 2022-01-31T19:42:30.214 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:30.209+0000 7f39b1a5c700 1 -- 172.21.15.167:0/870174066 wait complete. 2022-01-31T19:42:30.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph config set global log_to_journald false --force' 2022-01-31T19:42:31.912 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:31 smithi167 conmon[32206]: cluster 2022-01-31T19:42:30.734402+0000 mgr.smithi167.aciqpk (mgr.14156) 209 : cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2022-01-31T19:42:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:31 smithi171 conmon[35325]: cluster 2022-01-31T19:42:30.734402+0000 mgr.smithi167.aciqpk (mgr.14156) 209 : cluster [DBG] 2022-01-31T19:42:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:31 smithi171 conmon[35325]: pgmap v166: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2022-01-31T19:42:32.238 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- 172.21.15.167:0/1794363100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 msgr2=0x7fefc80f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:32.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 --2- 172.21.15.167:0/1794363100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc80f5b00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fefc4004660 tx=0x7fefc4009b30).stop 2022-01-31T19:42:32.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- 172.21.15.167:0/1794363100 shutdown_connections 2022-01-31T19:42:32.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 --2- 172.21.15.167:0/1794363100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc80f6980 0x7fefc80f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:32.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 --2- 172.21.15.167:0/1794363100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc80f5b00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- 172.21.15.167:0/1794363100 >> 172.21.15.167:0/1794363100 conn(0x7fefc80f0e60 msgr2=0x7fefc80f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- 172.21.15.167:0/1794363100 shutdown_connections 2022-01-31T19:42:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- 172.21.15.167:0/1794363100 wait complete. 2022-01-31T19:42:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 Processor -- start 2022-01-31T19:42:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- start start 2022-01-31T19:42:32.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc8107090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:32.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc80f6980 0x7fefc8107580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:32.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc8107b30 con 0x7fefc80f56e0 2022-01-31T19:42:32.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.229+0000 7fefcfd38700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc8107c70 con 0x7fefc80f6980 2022-01-31T19:42:32.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc8107090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:32.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc8107090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33648/0 (socket says 172.21.15.167:33648) 2022-01-31T19:42:32.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 -- 172.21.15.167:0/2087279548 learned_addr learned my addr 172.21.15.167:0/2087279548 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:32.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcd2d3700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc80f6980 0x7fefc8107580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 -- 172.21.15.167:0/2087279548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc80f6980 msgr2=0x7fefc8107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc80f6980 0x7fefc8107580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 -- 172.21.15.167:0/2087279548 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4005040 con 0x7fefc80f56e0 2022-01-31T19:42:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcdad4700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc8107090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fefc4007a60 tx=0x7fefc4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:32.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefc4018070 con 0x7fefc80f56e0 2022-01-31T19:42:32.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fefc4013410 con 0x7fefc80f56e0 2022-01-31T19:42:32.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc81085b0 con 0x7fefc80f56e0 2022-01-31T19:42:32.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefc401c400 con 0x7fefc80f56e0 2022-01-31T19:42:32.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.230+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc8108b10 con 0x7fefc80f56e0 2022-01-31T19:42:32.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.231+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc8045bb0 con 0x7fefc80f56e0 2022-01-31T19:42:32.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.232+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fefc401d420 con 0x7fefc80f56e0 2022-01-31T19:42:32.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.232+0000 7fefbe7fc700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fefb405e5c0 0x7fefb4060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:32.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.232+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fefc407a7c0 con 0x7fefc80f56e0 2022-01-31T19:42:32.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.232+0000 7fefcd2d3700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fefb405e5c0 0x7fefb4060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:32.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.232+0000 7fefcd2d3700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fefb405e5c0 0x7fefb4060a80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fefb8007fd0 tx=0x7fefb8009040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:32.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.237+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fefc404c3c0 con 0x7fefc80f56e0 2022-01-31T19:42:32.393 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.391+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fefc81092c0 con 0x7fefc80f56e0 2022-01-31T19:42:32.398 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.396+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 7 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fefc404c3c0 con 0x7fefc80f56e0 2022-01-31T19:42:32.398 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.396+0000 7fefbe7fc700 1 -- 172.21.15.167:0/2087279548 <== mon.0 v2:172.21.15.167:3300/0 8 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fefc404f9e0 con 0x7fefc80f56e0 2022-01-31T19:42:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fefb405e5c0 msgr2=0x7fefb4060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fefb405e5c0 0x7fefb4060a80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fefb8007fd0 tx=0x7fefb8009040).stop 2022-01-31T19:42:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 msgr2=0x7fefc8107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:32.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc8107090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fefc4007a60 tx=0x7fefc4009b30).stop 2022-01-31T19:42:32.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 shutdown_connections 2022-01-31T19:42:32.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fefb405e5c0 0x7fefb4060a80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:32.402 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc80f56e0 0x7fefc8107090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:32.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 --2- 172.21.15.167:0/2087279548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc80f6980 0x7fefc8107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:32.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 >> 172.21.15.167:0/2087279548 conn(0x7fefc80f0e60 msgr2=0x7fefc805ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:32.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 shutdown_connections 2022-01-31T19:42:32.403 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:32.399+0000 7fefcfd38700 1 -- 172.21.15.167:0/2087279548 wait complete. 2022-01-31T19:42:32.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2022-01-31T19:42:33.541 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:33 smithi167 conmon[32206]: audit 2022-01-31T19:42:32.397872+0000 mon.smithi167 (mon.0) 598 : audit [INF] from='client.? 172.21.15.167:0/2087279548' entity='client.admin' 2022-01-31T19:42:33.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:33 smithi167 conmon[32206]: audit 2022-01-31T19:42:32.401802+0000 mon.smithi167 (mon.0) 599 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:42:33.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:33 smithi167 conmon[32206]: audit 2022-01-31T19:42:32.403159+0000 mon.smithi167 (mon.0) 600 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:42:33.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:33 smithi167 conmon[32206]: audit 2022-01-31T19:42:32.403925+0000 mon.smithi167 (mon.0) 601 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:42:33.543 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:33 smithi167 conmon[32206]: audit 2022-01-31T19:42:32.408924+0000 mon.smithi167 (mon.0) 602 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:33.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:33 smithi171 conmon[35325]: audit 2022-01-31T19:42:32.397872+0000 mon.smithi167 (mon.0) 2022-01-31T19:42:33.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:33 smithi171 conmon[35325]: 598 : audit [INF] from='client.? 172.21.15.167:0/2087279548' entity='client.admin' 2022-01-31T19:42:33.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:33 smithi171 conmon[35325]: audit 2022-01-31T19:42:32.401802+0000 mon.smithi167 (mon.0) 599 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:42:33.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:33 smithi171 conmon[35325]: audit 2022-01-31T19:42:32.403159+0000 mon.smithi167 (mon.0) 600 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:42:33.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:33 smithi171 conmon[35325]: audit 2022-01-31T19:42:32.403925+0000 mon.smithi167 (mon.0) 601 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:42:33.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:33 smithi171 conmon[35325]: audit 2022-01-31T19:42:32.408924+0000 mon.smithi167 (mon.0) 602 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:34.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2473648204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f4ac0 msgr2=0x7f1fc00f4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:34.349 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2473648204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc00f4ee0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1fb0004660 tx=0x7f1fb0009b30).stop 2022-01-31T19:42:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2473648204 shutdown_connections 2022-01-31T19:42:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2473648204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f5460 0x7f1fc00f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2473648204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc00f4ee0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2473648204 >> 172.21.15.167:0/2473648204 conn(0x7f1fc00f05e0 msgr2=0x7f1fc00f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2473648204 shutdown_connections 2022-01-31T19:42:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.346+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2473648204 wait complete. 2022-01-31T19:42:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fc66ef700 1 Processor -- start 2022-01-31T19:42:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fc66ef700 1 -- start start 2022-01-31T19:42:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fc66ef700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc01070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:34.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fc66ef700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 0x7f1fc01075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:34.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fc66ef700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0107b80 con 0x7f1fc00f5460 2022-01-31T19:42:34.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fc66ef700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0107cc0 con 0x7f1fc00f4ac0 2022-01-31T19:42:34.353 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 0x7f1fc01075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:34.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 0x7f1fc01075d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33670/0 (socket says 172.21.15.167:33670) 2022-01-31T19:42:34.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc01070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:34.354 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbf7fe700 1 -- 172.21.15.167:0/2111288768 learned_addr learned my addr 172.21.15.167:0/2111288768 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbf7fe700 1 -- 172.21.15.167:0/2111288768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f4ac0 msgr2=0x7f1fc01070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbf7fe700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc01070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbf7fe700 1 -- 172.21.15.167:0/2111288768 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0005040 con 0x7f1fc00f5460 2022-01-31T19:42:34.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.347+0000 7f1fbffff700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc01070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-01-31T19:42:34.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.348+0000 7f1fbf7fe700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 0x7f1fc01075d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1fb400c510 tx=0x7f1fb4007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:34.356 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.348+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fb400f650 con 0x7f1fc00f5460 2022-01-31T19:42:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.348+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc0108660 con 0x7f1fc00f5460 2022-01-31T19:42:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.348+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f1fb400fc60 con 0x7f1fc00f5460 2022-01-31T19:42:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.348+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fb4012c20 con 0x7f1fc00f5460 2022-01-31T19:42:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.348+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc0108c20 con 0x7f1fc00f5460 2022-01-31T19:42:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.349+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0045bb0 con 0x7f1fc00f5460 2022-01-31T19:42:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.349+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f1fb4010070 con 0x7f1fc00f5460 2022-01-31T19:42:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.349+0000 7f1fbcff9700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1fa805e5c0 0x7f1fa8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.350+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1fb4078510 con 0x7f1fc00f5460 2022-01-31T19:42:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.350+0000 7f1fbffff700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1fa805e5c0 0x7f1fa8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.350+0000 7f1fbffff700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1fa805e5c0 0x7f1fa8060a80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1fb0004630 tx=0x7f1fb0009b30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.353+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1fb40456b0 con 0x7f1fc00f5460 2022-01-31T19:42:34.528 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.527+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fc0108fa0 con 0x7f1fa805e5c0 2022-01-31T19:42:34.533 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.532+0000 7f1fbcff9700 1 -- 172.21.15.167:0/2111288768 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f1fc0108fa0 con 0x7f1fa805e5c0 2022-01-31T19:42:34.533 INFO:teuthology.orchestra.run.smithi167.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:42:34.535 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1fa805e5c0 msgr2=0x7f1fa8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:34.535 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1fa805e5c0 0x7f1fa8060a80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1fb0004630 tx=0x7f1fb0009b30).stop 2022-01-31T19:42:34.536 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 msgr2=0x7f1fc01075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:34.536 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 0x7f1fc01075d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1fb400c510 tx=0x7f1fb4007df0).stop 2022-01-31T19:42:34.536 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 shutdown_connections 2022-01-31T19:42:34.536 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1fa805e5c0 0x7f1fa8060a80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc00f4ac0 0x7f1fc01070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 --2- 172.21.15.167:0/2111288768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fc00f5460 0x7f1fc01075d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 >> 172.21.15.167:0/2111288768 conn(0x7f1fc00f05e0 msgr2=0x7f1fc00fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 shutdown_connections 2022-01-31T19:42:34.538 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:34.534+0000 7f1fc66ef700 1 -- 172.21.15.167:0/2111288768 wait complete. 2022-01-31T19:42:34.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:34 smithi167 conmon[32206]: cluster 2022-01-31T19:42:32.734803+0000 mgr.smithi167.aciqpk (mgr.14156) 210 : cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2022-01-31T19:42:34.813 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:34 smithi171 conmon[35325]: cluster 2022-01-31T19:42:32.734803+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:42:34.813 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:34 smithi171 conmon[35325]: 210 : cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2022-01-31T19:42:34.964 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2022-01-31T19:42:34.965 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi167.front.sepia.ceph.com 2022-01-31T19:42:34.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done' 2022-01-31T19:42:35.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: audit 2022-01-31T19:42:34.528267+0000 2022-01-31T19:42:35.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 211 : audit [DBG] from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:42:35.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: cephadm 2022-01-31T19:42:34.528895+0000 mgr.smithi167.aciqpk (mgr.14156) 212 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:42:35.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: audit 2022-01-31T19:42:34.532537+0000 mon.smithi167 (mon.0) 603 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:35.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: audit 2022-01-31T19:42:34.533012+0000 mon.smithi167 (mon.0) 604 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:42:35.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: audit 2022-01-31T19:42:34.534137+0000 mon.smithi167 (mon.0) 605 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:42:35.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: audit 2022-01-31T19:42:34.534977+0000 mon.smithi167 (mon.0) 606 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:42:35.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: audit 2022-01-31T19:42:34.542788+0000 mon.smithi167 (mon.0) 607 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:35.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: cephadm 2022-01-31T19:42:34.546577+0000 mgr.smithi167.aciqpk (mgr.14156) 213 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:42:35.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:35 smithi167 conmon[32206]: cluster 2022-01-31T19:42:34.735246+0000 mgr.smithi167.aciqpk (mgr.14156) 214 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2022-01-31T19:42:35.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit 2022-01-31T19:42:34.528267+0000 mgr.smithi167.aciqpk (mgr.14156) 211 : audit [DBG] from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:42:35.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: cephadm 2022-01-31T19:42: 2022-01-31T19:42:35.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 34.528895+0000 mgr.smithi167.aciqpk (mgr.14156) 212 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:42:35.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit 2022-01-31T19:42:34.532537+0000 mon.smithi167 (mon.0) 603 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:35.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit 2022-01-31T19:42:34.533012+0000 2022-01-31T19:42:35.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: mon.smithi167 (mon.0) 604 : audit [DBG] 2022-01-31T19:42:35.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:42:35.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit 2022-01-31T19:42:35.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 2022-01-31T19:42: 2022-01-31T19:42:35.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 34.534137+0000 mon.smithi167 (mon. 2022-01-31T19:42:35.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 0) 605 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:42:35.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:42:35.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 01-31T19:42:34.534977+0000 2022-01-31T19:42:35.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: mon.smithi167 (mon.0) 606 : 2022-01-31T19:42:35.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:42:35.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: audit 2022 2022-01-31T19:42:35.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: -01-31T19:42:34 2022-01-31T19:42:35.863 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: .542788+0000 mon.smithi167 (mon. 2022-01-31T19:42:35.863 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 0) 607 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:42:35.863 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: cephadm 2022 2022-01-31T19:42:35.863 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: -01-31T19:42:34. 2022-01-31T19:42:35.864 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 546577+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:42:35.864 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: 213 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:29e1fc1722aa5915b44828a5ad02ec45ce760aa3 2022-01-31T19:42:35.864 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: cluster 2022 2022-01-31T19:42:35.864 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: -01-31T19:42:34 2022-01-31T19:42:35.865 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: .735246+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:42:35.865 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:35 smithi171 conmon[35325]: ) 214 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2022-01-31T19:42:36.512 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.510+0000 7f747359e700 1 -- 172.21.15.167:0/993348366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f67f0 msgr2=0x7f74740f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:36.513 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.510+0000 7f747359e700 1 --2- 172.21.15.167:0/993348366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f67f0 0x7f74740f6c50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f7468004660 tx=0x7f7468009b30).stop 2022-01-31T19:42:36.513 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 -- 172.21.15.167:0/993348366 shutdown_connections 2022-01-31T19:42:36.513 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 --2- 172.21.15.167:0/993348366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f67f0 0x7f74740f6c50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.514 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 --2- 172.21.15.167:0/993348366 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f5550 0x7f74740f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.514 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 -- 172.21.15.167:0/993348366 >> 172.21.15.167:0/993348366 conn(0x7f74740f0cb0 msgr2=0x7f74740f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:36.514 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 -- 172.21.15.167:0/993348366 shutdown_connections 2022-01-31T19:42:36.514 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 -- 172.21.15.167:0/993348366 wait complete. 2022-01-31T19:42:36.515 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 Processor -- start 2022-01-31T19:42:36.515 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.511+0000 7f747359e700 1 -- start start 2022-01-31T19:42:36.515 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f747359e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f5550 0x7f7474068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:36.515 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f747359e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 0x7f74740692a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:36.516 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f747359e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74740698a0 con 0x7f74740f5550 2022-01-31T19:42:36.516 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f747359e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747410f960 con 0x7f74740f67f0 2022-01-31T19:42:36.516 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 0x7f74740692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:36.516 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 0x7f74740692a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40016/0 (socket says 172.21.15.167:40016) 2022-01-31T19:42:36.516 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 -- 172.21.15.167:0/832653438 learned_addr learned my addr 172.21.15.167:0/832653438 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:36.518 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f747259c700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f5550 0x7f7474068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:36.518 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 -- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f5550 msgr2=0x7f7474068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:36.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f5550 0x7f7474068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 -- 172.21.15.167:0/832653438 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7468005040 con 0x7f74740f67f0 2022-01-31T19:42:36.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f747259c700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f5550 0x7f7474068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:42:36.519 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.512+0000 7f7471d9b700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 0x7f74740692a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7468009990 tx=0x7f7468009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:36.520 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.513+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7468007ba0 con 0x7f74740f67f0 2022-01-31T19:42:36.520 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.513+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747410fbe0 con 0x7f74740f67f0 2022-01-31T19:42:36.520 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.513+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74741100e0 con 0x7f74740f67f0 2022-01-31T19:42:36.520 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.513+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f7468007d00 con 0x7f74740f67f0 2022-01-31T19:42:36.520 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.513+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f746801bea0 con 0x7f74740f67f0 2022-01-31T19:42:36.521 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.514+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7458004fa0 con 0x7f74740f67f0 2022-01-31T19:42:36.521 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.514+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f746801b480 con 0x7f74740f67f0 2022-01-31T19:42:36.521 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.515+0000 7f7462ffd700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f745c05e5c0 0x7f745c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:36.521 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.515+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f746807c5c0 con 0x7f74740f67f0 2022-01-31T19:42:36.522 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.515+0000 7f747259c700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f745c05e5c0 0x7f745c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:36.522 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.515+0000 7f747259c700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f745c05e5c0 0x7f745c060a80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f74740f6650 tx=0x7f746400b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:36.522 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.519+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f746804e1e0 con 0x7f74740f67f0 2022-01-31T19:42:36.763 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.762+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7458000bc0 con 0x7f745c05e5c0 2022-01-31T19:42:36.765 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.763+0000 7f7462ffd700 1 -- 172.21.15.167:0/832653438 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+278 (secure 0 0 0) 0x7f7458000bc0 con 0x7f745c05e5c0 2022-01-31T19:42:36.766 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f745c05e5c0 msgr2=0x7f745c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:36.767 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f745c05e5c0 0x7f745c060a80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f74740f6650 tx=0x7f746400b040).stop 2022-01-31T19:42:36.767 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 msgr2=0x7f74740692a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:36.767 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 0x7f74740692a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7468009990 tx=0x7f7468009b30).stop 2022-01-31T19:42:36.768 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 shutdown_connections 2022-01-31T19:42:36.768 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f745c05e5c0 0x7f745c060a80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.768 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740f5550 0x7f7474068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.768 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 --2- 172.21.15.167:0/832653438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74740f67f0 0x7f74740692a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 >> 172.21.15.167:0/832653438 conn(0x7f74740f0cb0 msgr2=0x7f74740fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:36.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 shutdown_connections 2022-01-31T19:42:36.769 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.765+0000 7f747359e700 1 -- 172.21.15.167:0/832653438 wait complete. 2022-01-31T19:42:36.777 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:42:36.902 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 -- 172.21.15.167:0/1825096294 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f5560 msgr2=0x7f950c0f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:36.903 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 --2- 172.21.15.167:0/1825096294 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f5560 0x7f950c0f5980 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f9508004660 tx=0x7f9508009b30).stop 2022-01-31T19:42:36.903 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 -- 172.21.15.167:0/1825096294 shutdown_connections 2022-01-31T19:42:36.903 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 --2- 172.21.15.167:0/1825096294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f6800 0x7f950c0f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.903 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 --2- 172.21.15.167:0/1825096294 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f5560 0x7f950c0f5980 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 -- 172.21.15.167:0/1825096294 >> 172.21.15.167:0/1825096294 conn(0x7f950c0f0ce0 msgr2=0x7f950c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.900+0000 7f9513483700 1 -- 172.21.15.167:0/1825096294 shutdown_connections 2022-01-31T19:42:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 -- 172.21.15.167:0/1825096294 wait complete. 2022-01-31T19:42:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 Processor -- start 2022-01-31T19:42:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 -- start start 2022-01-31T19:42:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 0x7f950c10bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f6800 0x7f950c10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c060390 con 0x7f950c0f6800 2022-01-31T19:42:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.901+0000 7f9513483700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0604d0 con 0x7f950c0f5560 2022-01-31T19:42:36.907 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 0x7f950c10bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9511c80700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f6800 0x7f950c10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 0x7f950c10bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40026/0 (socket says 172.21.15.167:40026) 2022-01-31T19:42:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 -- 172.21.15.167:0/2143221155 learned_addr learned my addr 172.21.15.167:0/2143221155 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 -- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f6800 msgr2=0x7f950c10e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f6800 0x7f950c10e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 -- 172.21.15.167:0/2143221155 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508005040 con 0x7f950c0f5560 2022-01-31T19:42:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9512481700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 0x7f950c10bfb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9508014040 tx=0x7f950800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9508019070 con 0x7f950c0f5560 2022-01-31T19:42:36.910 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c10e9a0 con 0x7f950c0f5560 2022-01-31T19:42:36.910 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f9508007e70 con 0x7f950c0f5560 2022-01-31T19:42:36.910 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.902+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9508014d10 con 0x7f950c0f5560 2022-01-31T19:42:36.911 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.903+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c10ed70 con 0x7f950c0f5560 2022-01-31T19:42:36.911 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.904+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f9508025070 con 0x7f950c0f5560 2022-01-31T19:42:36.912 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.904+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c045bb0 con 0x7f950c0f5560 2022-01-31T19:42:36.912 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.904+0000 7f9502ffd700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f94f805e5c0 0x7f94f8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:36.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.904+0000 7f9511c80700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f94f805e5c0 0x7f94f8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:36.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.904+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f9508079810 con 0x7f950c0f5560 2022-01-31T19:42:36.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.905+0000 7f9511c80700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f94f805e5c0 0x7f94f8060a80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f94fc009fd0 tx=0x7f94fc00b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:36.914 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:36.908+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f950804b490 con 0x7f950c0f5560 2022-01-31T19:42:37.133 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.132+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f950c057ff0 con 0x7f94f805e5c0 2022-01-31T19:42:37.145 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.144+0000 7f9502ffd700 1 -- 172.21.15.167:0/2143221155 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3168 (secure 0 0 0) 0x7f950c057ff0 con 0x7f94f805e5c0 2022-01-31T19:42:37.145 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:42:37.146 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (3m) 26s ago 4m 22.0M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:42:37.146 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (4m) 26s ago 4m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:42:37.146 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (3m) 29s ago 3m 7155k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:42:37.146 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (3m) 26s ago 4m 31.6M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:42:37.147 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.rdkazj smithi167 running (41s) 26s ago 41s 14.4M - 16.2.5 46cf6318c64e 29f596ef1666 2022-01-31T19:42:37.147 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.sgnfie smithi167 running (46s) 26s ago 45s 16.6M - 16.2.5 46cf6318c64e 8c462c58dec4 2022-01-31T19:42:37.147 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.dyqqtj smithi171 running (44s) 29s ago 44s 18.4M - 16.2.5 46cf6318c64e 5e84b5044af1 2022-01-31T19:42:37.147 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.erzfea smithi171 running (39s) 29s ago 39s 14.6M - 16.2.5 46cf6318c64e 802065eafb38 2022-01-31T19:42:37.147 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:9283 running 26s ago 5m - - 2022-01-31T19:42:37.148 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (3m) 29s ago 3m 374M - 16.2.5 46cf6318c64e 0e89e7ce37df 2022-01-31T19:42:37.148 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running 26s ago 5m - 2048M 2022-01-31T19:42:37.148 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (3m) 29s ago 3m 83.9M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:42:37.148 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (4m) 26s ago 4m 18.1M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:42:37.149 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (3m) 29s ago 3m 18.2M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:42:37.149 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (3m) 26s ago 3m 49.6M 6472M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:42:37.149 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (2m) 26s ago 2m 49.4M 6472M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:42:37.149 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (2m) 26s ago 2m 49.0M 6472M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:42:37.150 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (2m) 29s ago 2m 49.9M 5339M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:42:37.150 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (2m) 29s ago 2m 47.2M 5339M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:42:37.150 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (112s) 29s ago 112s 47.3M 5339M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:42:37.150 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (3m) 26s ago 4m 40.1M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:42:37.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f94f805e5c0 msgr2=0x7f94f8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f94f805e5c0 0x7f94f8060a80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f94fc009fd0 tx=0x7f94fc00b040).stop 2022-01-31T19:42:37.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 msgr2=0x7f950c10bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 0x7f950c10bfb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9508014040 tx=0x7f950800b040).stop 2022-01-31T19:42:37.153 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 shutdown_connections 2022-01-31T19:42:37.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f94f805e5c0 0x7f94f8060a80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f950c0f5560 0x7f950c10bfb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.154 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 --2- 172.21.15.167:0/2143221155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c0f6800 0x7f950c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 >> 172.21.15.167:0/2143221155 conn(0x7f950c0f0ce0 msgr2=0x7f950c0fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:37.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 shutdown_connections 2022-01-31T19:42:37.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.146+0000 7f9513483700 1 -- 172.21.15.167:0/2143221155 wait complete. 2022-01-31T19:42:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 -- 172.21.15.167:0/774731393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 msgr2=0x7fe0800f49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 --2- 172.21.15.167:0/774731393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f49a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fe070004660 tx=0x7fe070009b30).stop 2022-01-31T19:42:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 -- 172.21.15.167:0/774731393 shutdown_connections 2022-01-31T19:42:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 --2- 172.21.15.167:0/774731393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f49a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.273 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 --2- 172.21.15.167:0/774731393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 0x7fe0800f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.273 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 -- 172.21.15.167:0/774731393 >> 172.21.15.167:0/774731393 conn(0x7fe0800f0640 msgr2=0x7fe0800f2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:37.280 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 -- 172.21.15.167:0/774731393 shutdown_connections 2022-01-31T19:42:37.281 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.270+0000 7fe087b88700 1 -- 172.21.15.167:0/774731393 wait complete. 2022-01-31T19:42:37.281 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe087b88700 1 Processor -- start 2022-01-31T19:42:37.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe087b88700 1 -- start start 2022-01-31T19:42:37.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe087b88700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 0x7fe0800f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:37.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe087b88700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:37.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe087b88700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08010ac90 con 0x7fe0800f8820 2022-01-31T19:42:37.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe087b88700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08010add0 con 0x7fe0800f7e80 2022-01-31T19:42:37.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085123700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:37.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085924700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 0x7fe0800f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:37.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085123700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33720/0 (socket says 172.21.15.167:33720) 2022-01-31T19:42:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085123700 1 -- 172.21.15.167:0/2671249333 learned_addr learned my addr 172.21.15.167:0/2671249333 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085924700 1 -- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 msgr2=0x7fe0800f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085924700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.271+0000 7fe085924700 1 -- 172.21.15.167:0/2671249333 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070005040 con 0x7fe0800f7e80 2022-01-31T19:42:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.272+0000 7fe085924700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 0x7fe0800f7750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe08005ebc0 tx=0x7fe07c00b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.272+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe07c0092e0 con 0x7fe0800f7e80 2022-01-31T19:42:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.272+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fe07c00f040 con 0x7fe0800f7e80 2022-01-31T19:42:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.272+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe07c0147a0 con 0x7fe0800f7e80 2022-01-31T19:42:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.272+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0800f6180 con 0x7fe0800f7e80 2022-01-31T19:42:37.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.272+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0800f66f0 con 0x7fe0800f7e80 2022-01-31T19:42:37.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.273+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe080045bb0 con 0x7fe0800f7e80 2022-01-31T19:42:37.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.273+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fe07c01b030 con 0x7fe0800f7e80 2022-01-31T19:42:37.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.274+0000 7fe0767fc700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe06c05e5c0 0x7fe06c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.274+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe07c078760 con 0x7fe0800f7e80 2022-01-31T19:42:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.274+0000 7fe085123700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe06c05e5c0 0x7fe06c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.274+0000 7fe085123700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe06c05e5c0 0x7fe06c060a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe070009d60 tx=0x7fe070016040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.278+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe07c04a360 con 0x7fe0800f7e80 2022-01-31T19:42:37.590 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.589+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe0800fefb0 con 0x7fe0800f7e80 2022-01-31T19:42:37.591 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.590+0000 7fe0767fc700 1 -- 172.21.15.167:0/2671249333 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+573 (secure 0 0 0) 0x7fe07c019250 con 0x7fe0800f7e80 2022-01-31T19:42:37.592 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:42:37.592 INFO:teuthology.orchestra.run.smithi167.stdout: "mon": { 2022-01-31T19:42:37.592 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:42:37.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:42:37.593 INFO:teuthology.orchestra.run.smithi167.stdout: "mgr": { 2022-01-31T19:42:37.593 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:42:37.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:42:37.594 INFO:teuthology.orchestra.run.smithi167.stdout: "osd": { 2022-01-31T19:42:37.594 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6 2022-01-31T19:42:37.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:42:37.594 INFO:teuthology.orchestra.run.smithi167.stdout: "mds": { 2022-01-31T19:42:37.595 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4 2022-01-31T19:42:37.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:42:37.595 INFO:teuthology.orchestra.run.smithi167.stdout: "overall": { 2022-01-31T19:42:37.595 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 14 2022-01-31T19:42:37.596 INFO:teuthology.orchestra.run.smithi167.stdout: } 2022-01-31T19:42:37.596 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:42:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.592+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe06c05e5c0 msgr2=0x7fe06c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.592+0000 7fe087b88700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe06c05e5c0 0x7fe06c060a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe070009d60 tx=0x7fe070016040).stop 2022-01-31T19:42:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.592+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 msgr2=0x7fe0800f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.592+0000 7fe087b88700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 0x7fe0800f7750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe08005ebc0 tx=0x7fe07c00b040).stop 2022-01-31T19:42:37.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 shutdown_connections 2022-01-31T19:42:37.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe06c05e5c0 0x7fe06c060a80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.598 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0800f7e80 0x7fe0800f7750 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0x7fe08005ebc0 tx=0x7fe07c00b040).stop 2022-01-31T19:42:37.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 --2- 172.21.15.167:0/2671249333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0800f8820 0x7fe0800f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 >> 172.21.15.167:0/2671249333 conn(0x7fe0800f0640 msgr2=0x7fe0800fb710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:37.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 shutdown_connections 2022-01-31T19:42:37.599 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.593+0000 7fe087b88700 1 -- 172.21.15.167:0/2671249333 wait complete. 2022-01-31T19:42:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 -- 172.21.15.167:0/1019501365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 msgr2=0x7f20140fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 --2- 172.21.15.167:0/1019501365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140fbd90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f2004004660 tx=0x7f2004009b30).stop 2022-01-31T19:42:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 -- 172.21.15.167:0/1019501365 shutdown_connections 2022-01-31T19:42:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 --2- 172.21.15.167:0/1019501365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140fbd90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 --2- 172.21.15.167:0/1019501365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20140603b0 0x7f20140f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 -- 172.21.15.167:0/1019501365 >> 172.21.15.167:0/1019501365 conn(0x7f20140f0e60 msgr2=0x7f20140f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 -- 172.21.15.167:0/1019501365 shutdown_connections 2022-01-31T19:42:37.718 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.714+0000 7f2018d9e700 1 -- 172.21.15.167:0/1019501365 wait complete. 2022-01-31T19:42:37.718 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.715+0000 7f2018d9e700 1 Processor -- start 2022-01-31T19:42:37.718 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.715+0000 7f2018d9e700 1 -- start start 2022-01-31T19:42:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.715+0000 7f2018d9e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20140603b0 0x7f20140f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.715+0000 7f2018d9e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.715+0000 7f2018d9e700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20140f5860 con 0x7f20140f9980 2022-01-31T19:42:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.715+0000 7f2018d9e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20140f59a0 con 0x7f20140603b0 2022-01-31T19:42:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140f5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33730/0 (socket says 172.21.15.167:33730) 2022-01-31T19:42:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 -- 172.21.15.167:0/3866816320 learned_addr learned my addr 172.21.15.167:0/3866816320 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:42:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 -- 172.21.15.167:0/3866816320 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20140603b0 msgr2=0x7f20140f6e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-01-31T19:42:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20140603b0 0x7f20140f6e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:42:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 -- 172.21.15.167:0/3866816320 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004005040 con 0x7f20140f9980 2022-01-31T19:42:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2011d9b700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140f5300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2004014040 tx=0x7f200400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2004019070 con 0x7f20140f9980 2022-01-31T19:42:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20140f5c20 con 0x7f20140f9980 2022-01-31T19:42:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.716+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20140f6180 con 0x7f20140f9980 2022-01-31T19:42:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.717+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f2004007e70 con 0x7f20140f9980 2022-01-31T19:42:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.717+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2004014dc0 con 0x7f20140f9980 2022-01-31T19:42:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.718+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f2004025070 con 0x7f20140f9980 2022-01-31T19:42:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.719+0000 7f2002ffd700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ffc05e5c0 0x7f1ffc060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:42:37.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.719+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f2004079780 con 0x7f20140f9980 2022-01-31T19:42:37.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.719+0000 7f201259c700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ffc05e5c0 0x7f1ffc060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:42:37.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.719+0000 7f201259c700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ffc05e5c0 0x7f1ffc060a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f200c004fd0 tx=0x7f200c00b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:42:37.726 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.720+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8004fa0 con 0x7f20140f9980 2022-01-31T19:42:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:37.726+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2004023020 con 0x7f20140f9980 2022-01-31T19:42:37.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:37 smithi167 conmon[32206]: cluster 2022-01-31T19:42:36.735598+0000 mgr.smithi167.aciqpk (mgr.14156) 215 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 597 B/s wr, 3 op/s 2022-01-31T19:42:37.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:37 smithi167 conmon[32206]: audit 2022-01-31T19:42:36.763713+0000 mgr.smithi167.aciqpk (mgr.14156) 216 : audit [DBG] from='client.24345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:42:37.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:37 smithi167 conmon[32206]: audit 2022-01-31T19:42:37.134010+0000 mgr.smithi167.aciqpk (mgr.14156) 217 : audit [DBG] from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:42:37.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:37 smithi167 conmon[32206]: audit 2022-01-31T19:42:37.590824+0000 mon.smithi171 (mon.1) 16 : audit [DBG] from='client.? 172.21.15.167:0/2671249333' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:42:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.002+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1ff8005e80 con 0x7f20140f9980 2022-01-31T19:42:38.005 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.003+0000 7f2002ffd700 1 -- 172.21.15.167:0/3866816320 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1672 (secure 0 0 0) 0x7f200404b3a0 con 0x7f20140f9980 2022-01-31T19:42:38.005 INFO:teuthology.orchestra.run.smithi167.stdout:e15 2022-01-31T19:42:38.005 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:42:38.006 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:42:38.006 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:42:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:38.006 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:42:38.007 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:42:38.007 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 15 2022-01-31T19:42:38.007 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:42:38.007 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:42:38.008 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:42:01.874180+0000 2022-01-31T19:42:38.008 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:42:38.008 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:42:38.008 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:42:38.009 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:42:38.009 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:42:38.009 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:42:38.009 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:42:38.009 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:42:38.010 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:42:38.010 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:42:38.010 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:42:38.011 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:42:38.011 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:42:38.011 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:42:38.011 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:42:38.011 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:42:38.012 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:42:38.012 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:42:38.012 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:42:38.012 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:42:38.013 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:42:38.014 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:42:38.014 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:42:38.014 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:42:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:42:38.015 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ffc05e5c0 msgr2=0x7f1ffc060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ffc05e5c0 0x7f1ffc060a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f200c004fd0 tx=0x7f200c00b040).stop 2022-01-31T19:42:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 msgr2=0x7f20140f5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:42:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140f5300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2004014040 tx=0x7f200400b040).stop 2022-01-31T19:42:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 shutdown_connections 2022-01-31T19:42:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f1ffc05e5c0 0x7f1ffc060a80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20140603b0 0x7f20140f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:42:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 --2- 172.21.15.167:0/3866816320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20140f9980 0x7f20140f5300 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:42:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 >> 172.21.15.167:0/3866816320 conn(0x7f20140f0e60 msgr2=0x7f20140f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:42:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 shutdown_connections 2022-01-31T19:42:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:42:38.005+0000 7f2018d9e700 1 -- 172.21.15.167:0/3866816320 wait complete. 2022-01-31T19:42:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:42:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: cluster 2022-01-31T19:42:36.735598+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:42:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: 215 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 597 B/s wr, 3 op/s 2022-01-31T19:42:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: audit 2022-01-31T19:42:36.763713+0000 mgr.smithi167.aciqpk (mgr.14156) 216 : audit [DBG] from='client.24345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:42:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:42:38.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: :42:37.134010+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:42:38.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: .14156) 217 : audit [DBG] from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:42:38.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: 2022-01-31T19:42:38.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:42:38.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: 01-31T19:42:37. 2022-01-31T19:42:38.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: 590824+0000 mon.smithi171 (mon.1) 16 2022-01-31T19:42:38.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:37 smithi171 conmon[35325]: : audit [DBG] from='client.? 172.21.15.167:0/2671249333' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:42:39.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:38 smithi171 conmon[35325]: audit 2022-01-31T19:42:38.003632+0000 mon.smithi167 (mon.0) 608 : audit [DBG] from='client.? 172.21.15.167:0/3866816320' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:42:39.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:38 smithi167 conmon[32206]: audit 2022-01-31T19:42:38.003632+0000 mon.smithi167 2022-01-31T19:42:39.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:38 smithi167 conmon[32206]: (mon.0) 608 : audit [DBG] from='client.? 172.21.15.167:0/3866816320' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:42:40.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:39 smithi171 conmon[35325]: cluster 2022-01-31T19:42:38.736042+0000 mgr.smithi167.aciqpk (mgr.14156) 218 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 597 B/s wr, 4 op/s 2022-01-31T19:42:40.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:39 smithi167 conmon[32206]: cluster 2022-01-31T19:42:38.736042+0000 mgr.smithi167.aciqpk (mgr.14156) 218 2022-01-31T19:42:40.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:39 smithi167 conmon[32206]: : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 597 B/s wr, 4 op/s 2022-01-31T19:42:41.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:41 smithi167 conmon[32206]: cluster 2022-01-31T19:42:40.736459+0000 2022-01-31T19:42:41.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:41 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 219 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 511 B/s wr, 4 op/s 2022-01-31T19:42:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:41 smithi171 conmon[35325]: cluster 2022-01-31T19:42:40.736459+0000 mgr.smithi167.aciqpk (mgr.14156) 219 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 511 B/s wr, 4 op/s 2022-01-31T19:42:42.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:41 smithi171 conmon[35325]: 2022-01-31T19:42:44.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:43 smithi171 conmon[35325]: cluster 2022-01-31T19:42:42.736892+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:42:44.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:43 smithi171 conmon[35325]: mgr.14156) 220 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:44.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:43 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:42:44.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:43 smithi167 conmon[32206]: 42:42.736892+0000 mgr.smithi167.aciqpk (mgr.14156) 220 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:46.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:45 smithi171 conmon[35325]: cluster 2022-01-31T19:42:44.737321+0000 mgr.smithi167.aciqpk 2022-01-31T19:42:46.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:45 smithi171 conmon[35325]: (mgr.14156) 221 : cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-01-31T19:42:46.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:45 smithi167 conmon[32206]: cluster 2022-01-31T19:42:44.737321+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:42:46.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:45 smithi167 conmon[32206]: .14156) 221 : cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-01-31T19:42:48.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:47 smithi171 conmon[35325]: cluster 2022-01-31T19:42:46 2022-01-31T19:42:48.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:47 smithi171 conmon[35325]: .737688+0000 mgr.smithi167.aciqpk (mgr.14156) 222 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:48.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:47 smithi167 conmon[32206]: cluster 2022-01-31T19:42:46.737688+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:42:48.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:47 smithi167 conmon[32206]: ) 222 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:50.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:49 smithi171 conmon[35325]: cluster 2022-01-31T19:42:48.738103+0000 mgr.smithi167.aciqpk (mgr.14156) 223 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:50.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:49 smithi167 conmon[32206]: cluster 2022-01-31T19:42:48.738103+0000 mgr.smithi167.aciqpk (mgr.14156) 223 : cluster 2022-01-31T19:42:50.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:49 smithi167 conmon[32206]: [DBG] pgmap v175: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:51.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:51 smithi167 conmon[32206]: cluster 2022-01-31T19:42:50. 2022-01-31T19:42:52.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:51 smithi167 conmon[32206]: 738564+0000 mgr.smithi167.aciqpk (mgr.14156) 224 : cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:52.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:51 smithi171 conmon[35325]: cluster 2022-01-31T19:42:50.738564+0000 mgr.smithi167.aciqpk (mgr.14156) 224 : cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:42:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:53 smithi171 conmon[35325]: cluster 2022-01-31T19:42:52.738978+0000 mgr.smithi167.aciqpk (mgr.14156) 225 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:54.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:53 smithi167 conmon[32206]: cluster 2022-01-31T19:42:52. 2022-01-31T19:42:54.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:53 smithi167 conmon[32206]: 738978+0000 mgr.smithi167.aciqpk (mgr.14156) 225 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:56.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:55 smithi171 conmon[35325]: cluster 2022-01-31T19:42:54.739398+0000 mgr.smithi167.aciqpk (mgr.14156) 226 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-01-31T19:42:56.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:55 smithi167 conmon[32206]: cluster 2022-01-31T19:42:54.739398 2022-01-31T19:42:56.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:55 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 226 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-01-31T19:42:58.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:57 smithi171 conmon[35325]: cluster 2022-01-31T19:42:56.739719+0000 mgr.smithi167.aciqpk (mgr.14156) 227 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:42:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:57 smithi167 conmon[32206]: cluster 2022-01-31T19:42:56.739719 2022-01-31T19:42:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:57 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 227 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2022-01-31T19:43:00.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:59 smithi171 conmon[35325]: cluster 2022-01-31T19:42:58.740129+0000 mgr.smithi167.aciqpk (mgr.14156) 228 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:43:00.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:59 smithi171 conmon[35325]: audit 2022-01-31T19:42:58.827742+0000 mon.smithi167 (mon.0) 609 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:43:00.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:59 smithi171 conmon[35325]: audit 2022-01-31T19:42:58.828801+0000 mon.smithi167 2022-01-31T19:43:00.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:42:59 smithi171 conmon[35325]: (mon.0) 610 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:43:00.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:59 smithi167 conmon[32206]: cluster 2022-01-31T19:42:58.740129+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:00.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:59 smithi167 conmon[32206]: 14156) 228 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 9.0 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2022-01-31T19:43:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:59 smithi167 conmon[32206]: audit 2022-01-31T19:42:58.827742+0000 mon.smithi167 (mon.0) 609 : audit [INF] 2022-01-31T19:43:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:59 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:43:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:59 smithi167 conmon[32206]: audit 2022-01-31T19:42:58.828801+0000 mon.smithi167 (mon. 2022-01-31T19:43:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:42:59 smithi167 conmon[32206]: 0) 610 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:43:01.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:01 smithi167 conmon[32206]: cluster 2022-01-31T19:43:00.740573+0000 2022-01-31T19:43:01.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:01 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 229 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 11 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2022-01-31T19:43:02.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:01 smithi171 conmon[35325]: cluster 2022-01-31T19:43:00. 2022-01-31T19:43:02.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:01 smithi171 conmon[35325]: 740573+0000 mgr.smithi167.aciqpk (mgr.14156) 229 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 11 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2022-01-31T19:43:04.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:03 smithi171 conmon[35325]: cluster 2022-01-31T19:43:02.741035+0000 mgr.smithi167.aciqpk (mgr.14156) 230 : cluster [DBG] 2022-01-31T19:43:04.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:03 smithi171 conmon[35325]: pgmap v182: 65 pgs: 65 active+clean; 11 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 4 op/s 2022-01-31T19:43:04.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:03 smithi167 conmon[32206]: cluster 2022-01-31T19:43:02.741035+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:04.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:03 smithi167 conmon[32206]: 14156) 230 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 11 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 4 op/s 2022-01-31T19:43:05.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: audit 2022-01-31T19:43:04. 2022-01-31T19:43:05.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: 515235+0000 mon.smithi167 (mon.0) 611 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:05.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:04.515732+0000 mgr.smithi167.aciqpk (mgr.14156) 231 : cephadm [INF] Upgrade: Target is version 17.0.0-10469-g29e1fc17 (unknown) 2022-01-31T19:43:05.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:05.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: 2022-01-31T19:43:04.515820+0000 mgr.smithi167.aciqpk (mgr.14156) 232 : cephadm 2022-01-31T19:43:05.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:02d03897d173962c893f849232e4bded4c918b1691a3512b5dad3dbbcf75bad5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02d03897d173962c893f849232e4bded4c918b1691a3512b5dad3dbbcf75bad5'] 2022-01-31T19:43:05.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:43:05.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: -31T19:43:04.516786+0000 mon.smithi167 2022-01-31T19:43:05.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: (mon.0) 612 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:05.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: 2022-01-31T19:43:05.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: cephadm 2022-01-31T 2022-01-31T19:43:05.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: 19:43:04.518036+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:43:05.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: .14156) 233 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi167.aciqpk) 2022-01-31T19:43:05.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: cluster 2022-01-31T19:43:05.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: 2022-01-31T19:43:04.741465 2022-01-31T19:43:05.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: +0000 mgr.smithi167.aciqpk (mgr.14156) 234 : cluster 2022-01-31T19:43:05.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: [DBG] pgmap v183: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2022-01-31T19:43:05.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:05.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: 2022-01-31T19:43:04.897467 2022-01-31T19:43:05.862 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: +0000 mgr.smithi167.aciqpk (mgr.14156) 235 : cephadm [INF] 2022-01-31T19:43:05.863 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:05 smithi171 conmon[35325]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:02d03897d173962c893f849232e4bded4c918b1691a3512b5dad3dbbcf75bad5 on smithi171 2022-01-31T19:43:05.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: audit 2022-01-31T19:43: 2022-01-31T19:43:05.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: 04.515235+0000 mon.smithi167 (mon.0) 611 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:05.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: cephadm 2022-01-31 2022-01-31T19:43:05.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: T19:43:04.515732+0000 mgr.smithi167.aciqpk (mgr.14156) 231 : cephadm [INF] 2022-01-31T19:43:05.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: Upgrade: Target is version 17.0.0-10469-g29e1fc17 (unknown) 2022-01-31T19:43:05.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: cephadm 2022-01- 2022-01-31T19:43:05.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: 31T19:43:04.515820+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:43:05.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: .14156) 232 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:02d03897d173962c893f849232e4bded4c918b1691a3512b5dad3dbbcf75bad5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02d03897d173962c893f849232e4bded4c918b1691a3512b5dad3dbbcf75bad5'] 2022-01-31T19:43:05.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: 2022-01-31T19:43:05.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: audit 2022-01-31T19:43 2022-01-31T19:43:05.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: :04.516786+0000 mon.smithi167 (mon.0) 2022-01-31T19:43:05.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: 612 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:05.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: cephadm 2022 2022-01-31T19:43:05.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: -01-31T19:43:04.518036+0000 2022-01-31T19:43:05.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 233 : cephadm 2022-01-31T19:43:05.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: [INF] Upgrade: Need to upgrade myself (mgr.smithi167.aciqpk) 2022-01-31T19:43:05.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: cluster 2022-01- 2022-01-31T19:43:05.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: 31T19:43:04.741465+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:43:05.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: mgr.14156) 234 : cluster [DBG] 2022-01-31T19:43:05.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: pgmap v183: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2022-01-31T19:43:05.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: cephadm 2022-01-31T19 2022-01-31T19:43:05.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: :43:04.897467+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:05.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:05 smithi167 conmon[32206]: 14156) 235 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:02d03897d173962c893f849232e4bded4c918b1691a3512b5dad3dbbcf75bad5 on smithi171 2022-01-31T19:43:08.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:07 smithi171 conmon[35325]: cluster 2022-01-31T19:43:06.741858+0000 mgr.smithi167.aciqpk (mgr.14156) 236 : cluster 2022-01-31T19:43:08.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:07 smithi171 conmon[35325]: [DBG] pgmap v184: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2022-01-31T19:43:08.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:07 smithi167 conmon[32206]: cluster 2022-01-31T19:43:06.741858 2022-01-31T19:43:08.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:07 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 236 : cluster [DBG] pgmap v184: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2022-01-31T19:43:08.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 -- 172.21.15.167:0/2877552971 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f6420 msgr2=0x7f67340f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 --2- 172.21.15.167:0/2877552971 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f6420 0x7f67340f68a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f6728004660 tx=0x7f6728009b30).stop 2022-01-31T19:43:08.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 -- 172.21.15.167:0/2877552971 shutdown_connections 2022-01-31T19:43:08.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 --2- 172.21.15.167:0/2877552971 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f6420 0x7f67340f68a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 --2- 172.21.15.167:0/2877552971 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f5b10 0x7f67340f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 -- 172.21.15.167:0/2877552971 >> 172.21.15.167:0/2877552971 conn(0x7f67340f04e0 msgr2=0x7f67340f2900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.115+0000 7f67396f1700 1 -- 172.21.15.167:0/2877552971 shutdown_connections 2022-01-31T19:43:08.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.116+0000 7f67396f1700 1 -- 172.21.15.167:0/2877552971 wait complete. 2022-01-31T19:43:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.116+0000 7f67396f1700 1 Processor -- start 2022-01-31T19:43:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.116+0000 7f67396f1700 1 -- start start 2022-01-31T19:43:08.125 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67396f1700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f5b10 0x7f67340fa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67396f1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 0x7f67340f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67396f1700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673410aab0 con 0x7f67340f5b10 2022-01-31T19:43:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67396f1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673410abf0 con 0x7f67340f6420 2022-01-31T19:43:08.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67337fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 0x7f67340f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67337fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 0x7f67340f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40134/0 (socket says 172.21.15.167:40134) 2022-01-31T19:43:08.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67337fe700 1 -- 172.21.15.167:0/541376714 learned_addr learned my addr 172.21.15.167:0/541376714 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:08.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f6733fff700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f5b10 0x7f67340fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67337fe700 1 -- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f5b10 msgr2=0x7f67340fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67337fe700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f5b10 0x7f67340fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.117+0000 7f67337fe700 1 -- 172.21.15.167:0/541376714 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728005040 con 0x7f67340f6420 2022-01-31T19:43:08.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f6733fff700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f5b10 0x7f67340fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-01-31T19:43:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f67337fe700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 0x7f67340f84c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6728000c00 tx=0x7f672800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6728019070 con 0x7f67340f6420 2022-01-31T19:43:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f6728007e70 con 0x7f67340f6420 2022-01-31T19:43:08.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6728014d90 con 0x7f67340f6420 2022-01-31T19:43:08.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673410ae70 con 0x7f67340f6420 2022-01-31T19:43:08.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.118+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673410b3d0 con 0x7f67340f6420 2022-01-31T19:43:08.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.119+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6734045bb0 con 0x7f67340f6420 2022-01-31T19:43:08.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.120+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f6728025070 con 0x7f67340f6420 2022-01-31T19:43:08.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.120+0000 7f6730ff9700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f671c05e5c0 0x7f671c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.121+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f67280798f0 con 0x7f67340f6420 2022-01-31T19:43:08.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.121+0000 7f6733fff700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f671c05e5c0 0x7f671c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.121+0000 7f6733fff700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f671c05e5c0 0x7f671c060a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6724009da0 tx=0x7f672400b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:08.133 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.126+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f672804b490 con 0x7f67340f6420 2022-01-31T19:43:08.311 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.309+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67340f8f40 con 0x7f671c05e5c0 2022-01-31T19:43:08.312 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.310+0000 7f6730ff9700 1 -- 172.21.15.167:0/541376714 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+346 (secure 0 0 0) 0x7f67340f8f40 con 0x7f671c05e5c0 2022-01-31T19:43:08.315 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f671c05e5c0 msgr2=0x7f671c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.315 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f671c05e5c0 0x7f671c060a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6724009da0 tx=0x7f672400b040).stop 2022-01-31T19:43:08.315 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 msgr2=0x7f67340f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.315 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 0x7f67340f84c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6728000c00 tx=0x7f672800b040).stop 2022-01-31T19:43:08.316 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 shutdown_connections 2022-01-31T19:43:08.316 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f671c05e5c0 0x7f671c060a80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.316 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67340f5b10 0x7f67340fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.316 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 --2- 172.21.15.167:0/541376714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67340f6420 0x7f67340f84c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.317 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 >> 172.21.15.167:0/541376714 conn(0x7f67340f04e0 msgr2=0x7f67340f4840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:08.317 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 shutdown_connections 2022-01-31T19:43:08.317 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.313+0000 7f67396f1700 1 -- 172.21.15.167:0/541376714 wait complete. 2022-01-31T19:43:08.329 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:43:08.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 -- 172.21.15.167:0/379894426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f4d60 msgr2=0x7f49d00f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 --2- 172.21.15.167:0/379894426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f4d60 0x7f49d00f5180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f49c0004770 tx=0x7f49c000ac40).stop 2022-01-31T19:43:08.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 -- 172.21.15.167:0/379894426 shutdown_connections 2022-01-31T19:43:08.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 --2- 172.21.15.167:0/379894426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49d00f5f60 0x7f49d00f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 --2- 172.21.15.167:0/379894426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f4d60 0x7f49d00f5180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 -- 172.21.15.167:0/379894426 >> 172.21.15.167:0/379894426 conn(0x7f49d00f04e0 msgr2=0x7f49d00f2900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:08.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 -- 172.21.15.167:0/379894426 shutdown_connections 2022-01-31T19:43:08.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.421+0000 7f49d5668700 1 -- 172.21.15.167:0/379894426 wait complete. 2022-01-31T19:43:08.427 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49d5668700 1 Processor -- start 2022-01-31T19:43:08.427 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49d5668700 1 -- start start 2022-01-31T19:43:08.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49d5668700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49d00f4d60 0x7f49d0106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49d5668700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 0x7f49d01073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49d5668700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0107960 con 0x7f49d00f5f60 2022-01-31T19:43:08.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49d5668700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0107aa0 con 0x7f49d00f4d60 2022-01-31T19:43:08.429 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49cf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 0x7f49d01073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.429 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49cf7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 0x7f49d01073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33830/0 (socket says 172.21.15.167:33830) 2022-01-31T19:43:08.429 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.422+0000 7f49cf7fe700 1 -- 172.21.15.167:0/3391634101 learned_addr learned my addr 172.21.15.167:0/3391634101 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:08.429 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49cffff700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49d00f4d60 0x7f49d0106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49cf7fe700 1 -- 172.21.15.167:0/3391634101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49d00f4d60 msgr2=0x7f49d0106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49cf7fe700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49d00f4d60 0x7f49d0106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49cf7fe700 1 -- 172.21.15.167:0/3391634101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0005040 con 0x7f49d00f5f60 2022-01-31T19:43:08.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49cf7fe700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 0x7f49d01073b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f49c40026d0 tx=0x7f49c4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:08.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49c40079c0 con 0x7f49d00f5f60 2022-01-31T19:43:08.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d0108440 con 0x7f49d00f5f60 2022-01-31T19:43:08.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f49c400f040 con 0x7f49d00f5f60 2022-01-31T19:43:08.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49c40148b0 con 0x7f49d00f5f60 2022-01-31T19:43:08.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.423+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d0108a00 con 0x7f49d00f5f60 2022-01-31T19:43:08.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.425+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f49c4007b20 con 0x7f49d00f5f60 2022-01-31T19:43:08.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.425+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d0045bb0 con 0x7f49d00f5f60 2022-01-31T19:43:08.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.426+0000 7f49ccff9700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f49b805e5c0 0x7f49b8060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.426+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f49c40788a0 con 0x7f49d00f5f60 2022-01-31T19:43:08.433 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.426+0000 7f49cffff700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f49b805e5c0 0x7f49b8060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.434 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.426+0000 7f49cffff700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f49b805e5c0 0x7f49b8060a80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f49c0014450 tx=0x7f49c0016040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:08.435 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.434+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f49c404a4a0 con 0x7f49d00f5f60 2022-01-31T19:43:08.602 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.599+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f49d0108c80 con 0x7f49b805e5c0 2022-01-31T19:43:08.610 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.608+0000 7f49ccff9700 1 -- 172.21.15.167:0/3391634101 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3124 (secure 0 0 0) 0x7f49d0108c80 con 0x7f49b805e5c0 2022-01-31T19:43:08.611 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:43:08.611 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (3m) 57s ago 4m 22.0M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:43:08.611 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (4m) 57s ago 4m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:43:08.612 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (4m) 61s ago 4m 7155k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:43:08.612 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (3m) 57s ago 4m 31.6M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:43:08.612 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.rdkazj smithi167 running (73s) 57s ago 72s 14.4M - 16.2.5 46cf6318c64e 29f596ef1666 2022-01-31T19:43:08.613 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.sgnfie smithi167 running (77s) 57s ago 77s 16.6M - 16.2.5 46cf6318c64e 8c462c58dec4 2022-01-31T19:43:08.613 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.dyqqtj smithi171 running (75s) 61s ago 75s 18.4M - 16.2.5 46cf6318c64e 5e84b5044af1 2022-01-31T19:43:08.613 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.erzfea smithi171 running (71s) 61s ago 71s 14.6M - 16.2.5 46cf6318c64e 802065eafb38 2022-01-31T19:43:08.614 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:9283 running 57s ago 5m - - 2022-01-31T19:43:08.614 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (4m) 61s ago 4m 374M - 16.2.5 46cf6318c64e 0e89e7ce37df 2022-01-31T19:43:08.614 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running 57s ago 6m - 2048M 2022-01-31T19:43:08.615 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (4m) 61s ago 4m 83.9M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:43:08.615 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (4m) 57s ago 4m 18.1M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:43:08.615 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (4m) 61s ago 4m 18.2M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:43:08.615 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (3m) 57s ago 3m 49.6M 6472M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:43:08.616 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (3m) 57s ago 3m 49.4M 6472M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:43:08.616 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (3m) 57s ago 3m 49.0M 6472M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:43:08.616 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (2m) 61s ago 2m 49.9M 5339M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:43:08.616 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (2m) 61s ago 2m 47.2M 5339M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:43:08.616 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (2m) 61s ago 2m 47.3M 5339M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:43:08.617 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (3m) 57s ago 4m 40.1M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:43:08.618 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f49b805e5c0 msgr2=0x7f49b8060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.618 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f49b805e5c0 0x7f49b8060a80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f49c0014450 tx=0x7f49c0016040).stop 2022-01-31T19:43:08.618 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 msgr2=0x7f49d01073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.619 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 0x7f49d01073b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f49c40026d0 tx=0x7f49c4009de0).stop 2022-01-31T19:43:08.619 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 shutdown_connections 2022-01-31T19:43:08.619 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f49b805e5c0 0x7f49b8060a80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.619 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49d00f4d60 0x7f49d0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.619 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 --2- 172.21.15.167:0/3391634101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49d00f5f60 0x7f49d01073b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.620 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.611+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 >> 172.21.15.167:0/3391634101 conn(0x7f49d00f04e0 msgr2=0x7f49d00f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:08.620 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.612+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 shutdown_connections 2022-01-31T19:43:08.620 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.612+0000 7f49d5668700 1 -- 172.21.15.167:0/3391634101 wait complete. 2022-01-31T19:43:08.715 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 -- 172.21.15.167:0/2976550436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f7ec0 msgr2=0x7fe3140f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.715 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 --2- 172.21.15.167:0/2976550436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f7ec0 0x7fe3140f82e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fe304004660 tx=0x7fe304009b30).stop 2022-01-31T19:43:08.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 -- 172.21.15.167:0/2976550436 shutdown_connections 2022-01-31T19:43:08.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 --2- 172.21.15.167:0/2976550436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3140f64d0 0x7fe3140f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 --2- 172.21.15.167:0/2976550436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f7ec0 0x7fe3140f82e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 -- 172.21.15.167:0/2976550436 >> 172.21.15.167:0/2976550436 conn(0x7fe3140f1f70 msgr2=0x7fe3140f4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:08.719 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 -- 172.21.15.167:0/2976550436 shutdown_connections 2022-01-31T19:43:08.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.713+0000 7fe319574700 1 -- 172.21.15.167:0/2976550436 wait complete. 2022-01-31T19:43:08.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.714+0000 7fe319574700 1 Processor -- start 2022-01-31T19:43:08.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.714+0000 7fe319574700 1 -- start start 2022-01-31T19:43:08.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe319574700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 0x7fe31406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe319574700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3140f7ec0 0x7fe314069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe319574700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31406a460 con 0x7fe3140f64d0 2022-01-31T19:43:08.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe319574700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31406a5a0 con 0x7fe3140f7ec0 2022-01-31T19:43:08.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe3127fc700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3140f7ec0 0x7fe314069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe312ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 0x7fe31406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe312ffd700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 0x7fe31406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33838/0 (socket says 172.21.15.167:33838) 2022-01-31T19:43:08.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe312ffd700 1 -- 172.21.15.167:0/431913386 learned_addr learned my addr 172.21.15.167:0/431913386 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:08.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe312ffd700 1 -- 172.21.15.167:0/431913386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3140f7ec0 msgr2=0x7fe314069f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe312ffd700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3140f7ec0 0x7fe314069f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.715+0000 7fe312ffd700 1 -- 172.21.15.167:0/431913386 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304005040 con 0x7fe3140f64d0 2022-01-31T19:43:08.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.716+0000 7fe312ffd700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 0x7fe31406ba80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fe304013a70 tx=0x7fe304009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:08.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.716+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe304018070 con 0x7fe3140f64d0 2022-01-31T19:43:08.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.716+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31406a820 con 0x7fe3140f64d0 2022-01-31T19:43:08.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.716+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31406ad80 con 0x7fe3140f64d0 2022-01-31T19:43:08.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.716+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fe3040097d0 con 0x7fe3140f64d0 2022-01-31T19:43:08.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.716+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30401de60 con 0x7fe3140f64d0 2022-01-31T19:43:08.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.717+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f4004fa0 con 0x7fe3140f64d0 2022-01-31T19:43:08.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.718+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fe304007ba0 con 0x7fe3140f64d0 2022-01-31T19:43:08.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.718+0000 7fe2fb7fe700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe2fc05e5c0 0x7fe2fc060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:08.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.718+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe30407a460 con 0x7fe3140f64d0 2022-01-31T19:43:08.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.718+0000 7fe3127fc700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe2fc05e5c0 0x7fe2fc060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:08.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.719+0000 7fe3127fc700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe2fc05e5c0 0x7fe2fc060a80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe3080077b0 tx=0x7fe308009ce0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:08.728 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.722+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe30404c0e0 con 0x7fe3140f64d0 2022-01-31T19:43:08.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.952+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe2f4005e80 con 0x7fe3140f64d0 2022-01-31T19:43:08.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.953+0000 7fe2fb7fe700 1 -- 172.21.15.167:0/431913386 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+573 (secure 0 0 0) 0x7fe30404f700 con 0x7fe3140f64d0 2022-01-31T19:43:08.958 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:43:08.958 INFO:teuthology.orchestra.run.smithi167.stdout: "mon": { 2022-01-31T19:43:08.959 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:43:08.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:08.960 INFO:teuthology.orchestra.run.smithi167.stdout: "mgr": { 2022-01-31T19:43:08.961 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:43:08.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:08.962 INFO:teuthology.orchestra.run.smithi167.stdout: "osd": { 2022-01-31T19:43:08.962 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6 2022-01-31T19:43:08.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:08.963 INFO:teuthology.orchestra.run.smithi167.stdout: "mds": { 2022-01-31T19:43:08.964 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4 2022-01-31T19:43:08.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:08.964 INFO:teuthology.orchestra.run.smithi167.stdout: "overall": { 2022-01-31T19:43:08.964 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 14 2022-01-31T19:43:08.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2022-01-31T19:43:08.965 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:43:08.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe2fc05e5c0 msgr2=0x7fe2fc060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.966 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe2fc05e5c0 0x7fe2fc060a80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe3080077b0 tx=0x7fe308009ce0).stop 2022-01-31T19:43:08.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 msgr2=0x7fe31406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:08.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 0x7fe31406ba80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fe304013a70 tx=0x7fe304009b30).stop 2022-01-31T19:43:08.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 shutdown_connections 2022-01-31T19:43:08.967 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fe2fc05e5c0 0x7fe2fc060a80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3140f64d0 0x7fe31406ba80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.968 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 --2- 172.21.15.167:0/431913386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3140f7ec0 0x7fe314069f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:08.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.954+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 >> 172.21.15.167:0/431913386 conn(0x7fe3140f1f70 msgr2=0x7fe3140fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:08.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.955+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 shutdown_connections 2022-01-31T19:43:08.969 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:08.955+0000 7fe319574700 1 -- 172.21.15.167:0/431913386 wait complete. 2022-01-31T19:43:09.066 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 -- 172.21.15.167:0/3860990285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f6800 msgr2=0x7f0a680f6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:09.067 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 --2- 172.21.15.167:0/3860990285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f6800 0x7f0a680f6c60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f0a58004660 tx=0x7f0a58009b30).stop 2022-01-31T19:43:09.067 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 -- 172.21.15.167:0/3860990285 shutdown_connections 2022-01-31T19:43:09.067 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 --2- 172.21.15.167:0/3860990285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f6800 0x7f0a680f6c60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:09.067 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 --2- 172.21.15.167:0/3860990285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a680f5560 0x7f0a680f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:09.068 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 -- 172.21.15.167:0/3860990285 >> 172.21.15.167:0/3860990285 conn(0x7f0a680f0ce0 msgr2=0x7f0a680f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:09.068 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 -- 172.21.15.167:0/3860990285 shutdown_connections 2022-01-31T19:43:09.068 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.063+0000 7f0a6f136700 1 -- 172.21.15.167:0/3860990285 wait complete. 2022-01-31T19:43:09.068 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6f136700 1 Processor -- start 2022-01-31T19:43:09.068 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6f136700 1 -- start start 2022-01-31T19:43:09.069 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6f136700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 0x7f0a680f96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:09.069 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6f136700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a680f6800 0x7f0a680f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:09.069 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6f136700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6810ab00 con 0x7f0a680f5560 2022-01-31T19:43:09.069 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6f136700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6810ac40 con 0x7f0a680f6800 2022-01-31T19:43:09.070 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6e134700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 0x7f0a680f96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:09.070 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6e134700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 0x7f0a680f96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33848/0 (socket says 172.21.15.167:33848) 2022-01-31T19:43:09.071 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.064+0000 7f0a6e134700 1 -- 172.21.15.167:0/4246991700 learned_addr learned my addr 172.21.15.167:0/4246991700 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:09.071 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a6e134700 1 -- 172.21.15.167:0/4246991700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a680f6800 msgr2=0x7f0a680f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:09.071 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a6e134700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a680f6800 0x7f0a680f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:43:09.071 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a6e134700 1 -- 172.21.15.167:0/4246991700 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a58005040 con 0x7f0a680f5560 2022-01-31T19:43:09.072 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a6e134700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 0x7f0a680f96c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0a6400e7c0 tx=0x7f0a6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:09.072 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a640092e0 con 0x7f0a680f5560 2022-01-31T19:43:09.072 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f0a6400f040 con 0x7f0a680f5560 2022-01-31T19:43:09.072 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.065+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a64014750 con 0x7f0a680f5560 2022-01-31T19:43:09.073 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.066+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a680f80f0 con 0x7f0a680f5560 2022-01-31T19:43:09.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:08 smithi167 conmon[32206]: audit 2022-01-31T19:43:08.311938 2022-01-31T19:43:09.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:08 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 237 : audit [DBG] from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:09.075 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.066+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a680f8680 con 0x7f0a680f5560 2022-01-31T19:43:09.075 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.067+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a680591e0 con 0x7f0a680f5560 2022-01-31T19:43:09.075 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.068+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f0a640077b0 con 0x7f0a680f5560 2022-01-31T19:43:09.076 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.068+0000 7f0a5effd700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0a5405e610 0x7f0a54060ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:09.076 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.068+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0a64078670 con 0x7f0a680f5560 2022-01-31T19:43:09.076 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.068+0000 7f0a6d933700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0a5405e610 0x7f0a54060ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:09.077 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.069+0000 7f0a6d933700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0a5405e610 0x7f0a54060ad0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f0a58004d80 tx=0x7f0a5800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:09.078 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.076+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0a6404a2f0 con 0x7f0a680f5560 2022-01-31T19:43:09.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:08 smithi171 conmon[35325]: audit 2022-01-31T19:43:08.311938+0000 2022-01-31T19:43:09.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:08 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 237 : audit [DBG] from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:09.290 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.288+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0a68045bb0 con 0x7f0a680f5560 2022-01-31T19:43:09.290 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.288+0000 7f0a5effd700 1 -- 172.21.15.167:0/4246991700 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1672 (secure 0 0 0) 0x7f0a64019070 con 0x7f0a680f5560 2022-01-31T19:43:09.291 INFO:teuthology.orchestra.run.smithi167.stdout:e15 2022-01-31T19:43:09.291 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:43:09.292 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:43:09.292 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:43:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:43:09.292 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:43:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:43:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 15 2022-01-31T19:43:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:43:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:43:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:42:01.874180+0000 2022-01-31T19:43:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:43:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:43:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:43:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:43:09.295 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:43:09.295 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:43:09.295 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:43:09.295 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:43:09.296 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:43:09.296 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:43:09.296 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:43:09.296 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:43:09.296 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:43:09.297 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:43:09.297 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:43:09.297 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:43:09.297 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:43:09.298 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:43:09.298 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:43:09.298 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:43:09.298 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:43:09.299 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:43:09.299 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:43:09.299 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:43:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:43:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:43:09.300 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0a5405e610 msgr2=0x7f0a54060ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:09.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0a5405e610 0x7f0a54060ad0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f0a58004d80 tx=0x7f0a5800b040).stop 2022-01-31T19:43:09.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 msgr2=0x7f0a680f96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:09.301 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 0x7f0a680f96c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0a6400e7c0 tx=0x7f0a6400b040).stop 2022-01-31T19:43:09.302 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 shutdown_connections 2022-01-31T19:43:09.302 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f0a5405e610 0x7f0a54060ad0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:09.302 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a680f5560 0x7f0a680f96c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:09.303 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 --2- 172.21.15.167:0/4246991700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a680f6800 0x7f0a680f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:43:09.303 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 >> 172.21.15.167:0/4246991700 conn(0x7f0a680f0ce0 msgr2=0x7f0a6805ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:09.303 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 shutdown_connections 2022-01-31T19:43:09.303 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:09.291+0000 7f0a6f136700 1 -- 172.21.15.167:0/4246991700 wait complete. 2022-01-31T19:43:09.304 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 15 2022-01-31T19:43:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: audit 2022-01-31T19:43:08.602160+0000 mgr.smithi167.aciqpk 2022-01-31T19:43:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: (mgr.14156) 238 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: cluster 2022-01-31T19:43:08.742340+0000 mgr.smithi167.aciqpk (mgr.14156) 239 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2022-01-31T19:43:10.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: audit 2022-01-31T19:43:08.954234+0000 mon.smithi167 (mon 2022-01-31T19:43:10.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: .0) 613 : audit [DBG] from='client.? 172.21.15.167:0/431913386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:43:10.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:43:10.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: 43:09.290019+0000 mon.smithi167 (mon.0) 614 2022-01-31T19:43:10.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:09 smithi171 conmon[35325]: : audit [DBG] from='client.? 172.21.15.167:0/4246991700' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:43:10.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: audit 2022-01-31T19:43:08.602160+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:43:10.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: mgr.14156) 238 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:10.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: cluster 2022-01-31T19:43:08.742340+0000 mgr.smithi167.aciqpk (mgr.14156) 239 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2022-01-31T19:43:10.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: audit 2022-01-31T19:43:08.954234+0000 mon.smithi167 (mon.0) 613 : audit 2022-01-31T19:43:10.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: [DBG] from='client.? 172.21.15.167:0/431913386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:43:10.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: audit 2022-01-31T19:43:09.290019+0000 2022-01-31T19:43:10.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:09 smithi167 conmon[32206]: mon.smithi167 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.167:0/4246991700' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:43:11.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:11 smithi167 conmon[32206]: cluster 2022-01-31T19:43:10.742801+0000 mgr.smithi167.aciqpk (mgr.14156) 240 : cluster [DBG] 2022-01-31T19:43:11.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:11 smithi167 conmon[32206]: pgmap v186: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2022-01-31T19:43:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:11 smithi171 conmon[35325]: cluster 2022-01-31T19:43:10. 2022-01-31T19:43:12.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:11 smithi171 conmon[35325]: 742801+0000 mgr.smithi167.aciqpk (mgr.14156) 240 : cluster [DBG] pgmap v186: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2022-01-31T19:43:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:13 smithi171 conmon[35325]: cluster 2022-01-31T19:43:12. 2022-01-31T19:43:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:13 smithi171 conmon[35325]: 743237+0000 mgr.smithi167.aciqpk (mgr.14156) 241 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 426 B/s wr, 4 op/s 2022-01-31T19:43:14.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:13 smithi167 conmon[32206]: cluster 2022-01-31T19:43:12.743237+0000 2022-01-31T19:43:14.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:13 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 241 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 426 B/s wr, 4 op/s 2022-01-31T19:43:16.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:15 smithi171 conmon[35325]: cluster 2022-01-31T19:43:14.743738+0000 2022-01-31T19:43:16.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:15 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 242 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2022-01-31T19:43:16.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:15 smithi167 conmon[32206]: cluster 2022-01-31T19:43:14 2022-01-31T19:43:16.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:15 smithi167 conmon[32206]: .743738+0000 mgr.smithi167.aciqpk (mgr.14156) 242 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2022-01-31T19:43:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:17 smithi171 conmon[35325]: cluster 2022-01-31T19:43:16.744141+0000 2022-01-31T19:43:18.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:17 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 243 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2022-01-31T19:43:18.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:17 smithi167 conmon[32206]: cluster 2022-01-31T19:43:16.744141 2022-01-31T19:43:18.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:17 smithi167 conmon[32206]: +0000 mgr.smithi167.aciqpk (mgr.14156) 243 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2022-01-31T19:43:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:19 smithi171 conmon[35325]: cluster 2022-01-31T19:43:18.744644+0000 2022-01-31T19:43:20.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:19 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 244 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2022-01-31T19:43:20.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:19 smithi167 conmon[32206]: cluster 2022-01-31T19:43:18.744644+0000 mgr.smithi167.aciqpk (mgr.14156) 244 : 2022-01-31T19:43:20.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:19 smithi167 conmon[32206]: cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2022-01-31T19:43:21.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:21 smithi167 conmon[32206]: cluster 2022-01-31T19:43: 2022-01-31T19:43:21.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:21 smithi167 conmon[32206]: 20.745020+0000 mgr.smithi167.aciqpk (mgr.14156) 245 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2022-01-31T19:43:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:21 smithi171 conmon[35325]: cluster 2022-01-31T19:43:20.745020+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:43:22.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:21 smithi171 conmon[35325]: mgr.14156) 245 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2022-01-31T19:43:22.729 INFO:tasks.workunit.client.0.smithi167.stderr:Note: switching to 'e7bdcb25f366a81de92a121020949aeab8ce71c7'. 2022-01-31T19:43:22.729 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.730 INFO:tasks.workunit.client.0.smithi167.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-01-31T19:43:22.730 INFO:tasks.workunit.client.0.smithi167.stderr:changes and commit them, and you can discard any commits you make in this 2022-01-31T19:43:22.732 INFO:tasks.workunit.client.0.smithi167.stderr:state without impacting any branches by switching back to a branch. 2022-01-31T19:43:22.732 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.732 INFO:tasks.workunit.client.0.smithi167.stderr:If you want to create a new branch to retain commits you create, you may 2022-01-31T19:43:22.732 INFO:tasks.workunit.client.0.smithi167.stderr:do so (now or later) by using -c with the switch command. Example: 2022-01-31T19:43:22.733 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.733 INFO:tasks.workunit.client.0.smithi167.stderr: git switch -c 2022-01-31T19:43:22.733 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.733 INFO:tasks.workunit.client.0.smithi167.stderr:Or undo this operation with: 2022-01-31T19:43:22.734 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.734 INFO:tasks.workunit.client.0.smithi167.stderr: git switch - 2022-01-31T19:43:22.734 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.734 INFO:tasks.workunit.client.0.smithi167.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-01-31T19:43:22.735 INFO:tasks.workunit.client.0.smithi167.stderr: 2022-01-31T19:43:22.735 INFO:tasks.workunit.client.0.smithi167.stderr:HEAD is now at e7bdcb25f36 qa: test inline compat set on older MDSMap 2022-01-31T19:43:22.736 DEBUG:teuthology.orchestra.run.smithi167:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-01-31T19:43:22.758 INFO:tasks.workunit.client.0.smithi167.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-01-31T19:43:22.760 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-01-31T19:43:22.760 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-01-31T19:43:23.315 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-01-31T19:43:23.372 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-01-31T19:43:23.408 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-01-31T19:43:23.410 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-01-31T19:43:23.410 INFO:tasks.workunit.client.0.smithi167.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-01-31T19:43:23.446 INFO:tasks.workunit.client.0.smithi167.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-01-31T19:43:23.450 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2022-01-31T19:43:23.450 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-01-31T19:43:23.507 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2022-01-31T19:43:23.509 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2022-01-31T19:43:23.509 DEBUG:teuthology.orchestra.run.smithi167:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7bdcb25f366a81de92a121020949aeab8ce71c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2022-01-31T19:43:23.570 INFO:tasks.workunit.client.0.smithi167.stderr:+ mkdir -p fsstress 2022-01-31T19:43:23.571 INFO:tasks.workunit.client.0.smithi167.stderr:+ pushd fsstress 2022-01-31T19:43:23.572 INFO:tasks.workunit.client.0.smithi167.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2022-01-31T19:43:23.572 INFO:tasks.workunit.client.0.smithi167.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2022-01-31T19:43:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:23 smithi171 conmon[35325]: cluster 2022-01-31T19:43:22.745391 2022-01-31T19:43:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:23 smithi171 conmon[35325]: +0000 mgr.smithi167.aciqpk (mgr.14156) 246 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2022-01-31T19:43:24.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:23 smithi167 conmon[32206]: cluster 2022-01-31T19:43:22.745391+0000 mgr.smithi167.aciqpk (mgr.14156) 246 : 2022-01-31T19:43:24.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:23 smithi167 conmon[32206]: cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 12 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2022-01-31T19:43:24.351 INFO:tasks.workunit.client.0.smithi167.stderr:+ tar xzf ltp-full.tgz 2022-01-31T19:43:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:25 smithi171 conmon[35325]: cluster 2022-01-31T19:43:24.745843+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:43:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:25 smithi171 conmon[35325]: mgr.14156) 247 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 25 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2022-01-31T19:43:26.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:25 smithi167 conmon[32206]: cluster 2022-01-31T19:43:24.745843+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:26.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:25 smithi167 conmon[32206]: 14156) 247 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 25 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2022-01-31T19:43:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:27 smithi171 conmon[35325]: cluster 2022-01-31T19:43:26.746247+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:27 smithi171 conmon[35325]: 14156) 248 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 25 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2022-01-31T19:43:28.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:27 smithi167 conmon[32206]: cluster 2022-01-31T19:43:26.746247+0000 2022-01-31T19:43:28.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:27 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 248 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 25 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2022-01-31T19:43:28.948 INFO:tasks.workunit.client.1.smithi171.stderr:Note: switching to 'e7bdcb25f366a81de92a121020949aeab8ce71c7'. 2022-01-31T19:43:28.949 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.949 INFO:tasks.workunit.client.1.smithi171.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-01-31T19:43:28.949 INFO:tasks.workunit.client.1.smithi171.stderr:changes and commit them, and you can discard any commits you make in this 2022-01-31T19:43:28.949 INFO:tasks.workunit.client.1.smithi171.stderr:state without impacting any branches by switching back to a branch. 2022-01-31T19:43:28.949 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.950 INFO:tasks.workunit.client.1.smithi171.stderr:If you want to create a new branch to retain commits you create, you may 2022-01-31T19:43:28.950 INFO:tasks.workunit.client.1.smithi171.stderr:do so (now or later) by using -c with the switch command. Example: 2022-01-31T19:43:28.950 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.950 INFO:tasks.workunit.client.1.smithi171.stderr: git switch -c 2022-01-31T19:43:28.951 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.951 INFO:tasks.workunit.client.1.smithi171.stderr:Or undo this operation with: 2022-01-31T19:43:28.951 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.951 INFO:tasks.workunit.client.1.smithi171.stderr: git switch - 2022-01-31T19:43:28.952 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.952 INFO:tasks.workunit.client.1.smithi171.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-01-31T19:43:28.952 INFO:tasks.workunit.client.1.smithi171.stderr: 2022-01-31T19:43:28.952 INFO:tasks.workunit.client.1.smithi171.stderr:HEAD is now at e7bdcb25f36 qa: test inline compat set on older MDSMap 2022-01-31T19:43:28.953 DEBUG:teuthology.orchestra.run.smithi171:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2022-01-31T19:43:28.976 INFO:tasks.workunit.client.1.smithi171.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-01-31T19:43:28.978 INFO:tasks.workunit.client.1.smithi171.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2022-01-31T19:43:28.978 INFO:tasks.workunit.client.1.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-01-31T19:43:29.875 INFO:tasks.workunit.client.1.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-01-31T19:43:29.939 INFO:tasks.workunit.client.1.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-01-31T19:43:29.977 INFO:tasks.workunit.client.1.smithi171.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2022-01-31T19:43:29.979 INFO:tasks.workunit.client.1.smithi171.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2022-01-31T19:43:29.979 INFO:tasks.workunit.client.1.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-01-31T19:43:30.017 INFO:tasks.workunit.client.1.smithi171.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2022-01-31T19:43:30.020 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-01-31T19:43:30.020 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2022-01-31T19:43:30.076 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2022-01-31T19:43:30.077 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2022-01-31T19:43:30.078 DEBUG:teuthology.orchestra.run.smithi171:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7bdcb25f366a81de92a121020949aeab8ce71c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2022-01-31T19:43:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:29 smithi171 conmon[35325]: cluster 2022-01-31T19:43:28.746668+0000 mgr.smithi167.aciqpk (mgr.14156) 249 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 2.9 MiB data, 47 MiB used, 536 GiB / 536 GiB avail; 245 KiB/s rd, 255 KiB/s wr, 5 op/s 2022-01-31T19:43:30.150 INFO:tasks.workunit.client.1.smithi171.stderr:+ mkdir -p fsstress 2022-01-31T19:43:30.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:29 smithi167 conmon[32206]: cluster 2022-01-31T19:43:28.746668+0000 mgr.smithi167.aciqpk (mgr.14156) 249 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 2.9 MiB data, 47 MiB used, 536 GiB / 536 GiB avail; 245 KiB/s rd, 255 KiB/s wr, 5 op/s 2022-01-31T19:43:30.173 INFO:tasks.workunit.client.1.smithi171.stderr:+ pushd fsstress 2022-01-31T19:43:30.173 INFO:tasks.workunit.client.1.smithi171.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2022-01-31T19:43:30.174 INFO:tasks.workunit.client.1.smithi171.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2022-01-31T19:43:31.732 INFO:tasks.workunit.client.1.smithi171.stderr:+ tar xzf ltp-full.tgz 2022-01-31T19:43:31.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:31 smithi167 conmon[32206]: cluster 2022-01-31T19:43:30.747109+0000 2022-01-31T19:43:31.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:31 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14156) 250 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 18 MiB data, 71 MiB used, 536 GiB / 536 GiB avail; 694 KiB/s rd, 1.6 MiB/s wr, 114 op/s 2022-01-31T19:43:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:31 smithi171 conmon[35325]: cluster 2022-01-31T19:43:30.747109+0000 mgr.smithi167.aciqpk (mgr.14156) 250 : 2022-01-31T19:43:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:31 smithi171 conmon[35325]: cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 18 MiB data, 71 MiB used, 536 GiB / 536 GiB avail; 694 KiB/s rd, 1.6 MiB/s wr, 114 op/s 2022-01-31T19:43:34.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:33 smithi171 conmon[35325]: cluster 2022-01-31T19:43:32.747488+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:43:34.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:33 smithi171 conmon[35325]: ) 251 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 25 MiB data, 131 MiB used, 536 GiB / 536 GiB avail; 793 KiB/s rd, 2.2 MiB/s wr, 217 op/s 2022-01-31T19:43:34.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:33 smithi167 conmon[32206]: cluster 2022-01-31T19:43:32.747488+0000 mgr.smithi167.aciqpk (mgr.14156) 251 : 2022-01-31T19:43:34.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:33 smithi167 conmon[32206]: cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 25 MiB data, 131 MiB used, 536 GiB / 536 GiB avail; 793 KiB/s rd, 2.2 MiB/s wr, 217 op/s 2022-01-31T19:43:36.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:35 smithi171 conmon[35325]: cluster 2022-01-31T19:43:34.747872+0000 2022-01-31T19:43:36.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:35 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 252 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 58 MiB data, 238 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 420 op/s 2022-01-31T19:43:36.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:35 smithi167 conmon[32206]: cluster 2022-01-31T19:43:34.747872+0000 mgr.smithi167.aciqpk (mgr.14156) 252 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 58 MiB data, 238 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 420 op/s 2022-01-31T19:43:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:37 smithi171 conmon[35325]: cluster 2022-01-31T19:43:36.748255+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:38.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:37 smithi171 conmon[35325]: 14156) 253 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 59 MiB data, 247 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 431 op/s 2022-01-31T19:43:38.175 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:37 smithi171 conmon[35325]: cluster 2022-01-31T19:43:36.823324+0000 mon.smithi167 (mon.0) 615 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:43:38.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:37 smithi167 conmon[32206]: cluster 2022-01-31T19:43:36.748255+0000 mgr.smithi167.aciqpk (mgr.14156 2022-01-31T19:43:38.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:37 smithi167 conmon[32206]: ) 253 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 59 MiB data, 247 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 431 op/s 2022-01-31T19:43:38.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:37 smithi167 conmon[32206]: cluster 2022-01-31T19:43:36.823324+0000 mon.smithi167 (mon.0) 615 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:43:39.414 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 -- 172.21.15.167:0/1569407487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f7ec0 msgr2=0x7fc8200f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 --2- 172.21.15.167:0/1569407487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f7ec0 0x7fc8200f82e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fc810004660 tx=0x7fc810009b30).stop 2022-01-31T19:43:39.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 -- 172.21.15.167:0/1569407487 shutdown_connections 2022-01-31T19:43:39.415 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 --2- 172.21.15.167:0/1569407487 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8200f8860 0x7fc8200f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 --2- 172.21.15.167:0/1569407487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f7ec0 0x7fc8200f82e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 -- 172.21.15.167:0/1569407487 >> 172.21.15.167:0/1569407487 conn(0x7fc8200f0640 msgr2=0x7fc8200f2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 -- 172.21.15.167:0/1569407487 shutdown_connections 2022-01-31T19:43:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.413+0000 7fc826361700 1 -- 172.21.15.167:0/1569407487 wait complete. 2022-01-31T19:43:39.416 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc826361700 1 Processor -- start 2022-01-31T19:43:39.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc826361700 1 -- start start 2022-01-31T19:43:39.419 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc826361700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8200f7ec0 0x7fc8200f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc826361700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 0x7fc8200f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc826361700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8200f6170 con 0x7fc8200f8860 2022-01-31T19:43:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc826361700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8200f62b0 con 0x7fc8200f7ec0 2022-01-31T19:43:39.420 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc81f7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 0x7fc8200f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:39.421 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc81f7fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 0x7fc8200f5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33958/0 (socket says 172.21.15.167:33958) 2022-01-31T19:43:39.421 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc81f7fe700 1 -- 172.21.15.167:0/245515589 learned_addr learned my addr 172.21.15.167:0/245515589 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:39.421 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.414+0000 7fc81f7fe700 1 -- 172.21.15.167:0/245515589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8200f7ec0 msgr2=0x7fc8200f7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.421 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc81f7fe700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8200f7ec0 0x7fc8200f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:43:39.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc81f7fe700 1 -- 172.21.15.167:0/245515589 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810005040 con 0x7fc8200f8860 2022-01-31T19:43:39.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc81f7fe700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 0x7fc8200f5c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc81400e7f0 tx=0x7fc81400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:39.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc814004570 con 0x7fc8200f8860 2022-01-31T19:43:39.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8200f6590 con 0x7fc8200f8860 2022-01-31T19:43:39.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fc81400f040 con 0x7fc8200f8860 2022-01-31T19:43:39.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8140148d0 con 0x7fc8200f8860 2022-01-31T19:43:39.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.415+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82010f0c0 con 0x7fc8200f8860 2022-01-31T19:43:39.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.416+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc820045bb0 con 0x7fc8200f8860 2022-01-31T19:43:39.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.416+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fc8140077b0 con 0x7fc8200f8860 2022-01-31T19:43:39.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.417+0000 7fc81cff9700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc80805e5c0 0x7fc808060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:39.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.417+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc8140789e0 con 0x7fc8200f8860 2022-01-31T19:43:39.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.417+0000 7fc81ffff700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc80805e5c0 0x7fc808060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:39.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.417+0000 7fc81ffff700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc80805e5c0 0x7fc808060a80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc810004d80 tx=0x7fc81000b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:39.426 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.425+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc8140476b0 con 0x7fc8200f8860 2022-01-31T19:43:39.607 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.606+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc820057ff0 con 0x7fc80805e5c0 2022-01-31T19:43:39.608 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.608+0000 7fc81cff9700 1 -- 172.21.15.167:0/245515589 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+346 (secure 0 0 0) 0x7fc820057ff0 con 0x7fc80805e5c0 2022-01-31T19:43:39.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc80805e5c0 msgr2=0x7fc808060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc80805e5c0 0x7fc808060a80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc810004d80 tx=0x7fc81000b040).stop 2022-01-31T19:43:39.611 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 msgr2=0x7fc8200f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 0x7fc8200f5c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc81400e7f0 tx=0x7fc81400b040).stop 2022-01-31T19:43:39.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 shutdown_connections 2022-01-31T19:43:39.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc80805e5c0 0x7fc808060a80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.612 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8200f7ec0 0x7fc8200f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:43:39.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 --2- 172.21.15.167:0/245515589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8200f8860 0x7fc8200f5c10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.609+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 >> 172.21.15.167:0/245515589 conn(0x7fc8200f0640 msgr2=0x7fc8200fb750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:39.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.610+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 shutdown_connections 2022-01-31T19:43:39.613 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.610+0000 7fc826361700 1 -- 172.21.15.167:0/245515589 wait complete. 2022-01-31T19:43:39.621 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:43:39.719 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 -- 172.21.15.167:0/2930460663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f7ec0 msgr2=0x7f4cac0f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.719 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/2930460663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f82e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f4ca8004660 tx=0x7f4ca8009b30).stop 2022-01-31T19:43:39.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 -- 172.21.15.167:0/2930460663 shutdown_connections 2022-01-31T19:43:39.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/2930460663 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/2930460663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f82e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 -- 172.21.15.167:0/2930460663 >> 172.21.15.167:0/2930460663 conn(0x7f4cac0f0640 msgr2=0x7f4cac0f2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:39.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.716+0000 7f4cb46c3700 1 -- 172.21.15.167:0/2930460663 shutdown_connections 2022-01-31T19:43:39.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 -- 172.21.15.167:0/2930460663 wait complete. 2022-01-31T19:43:39.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 Processor -- start 2022-01-31T19:43:39.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 -- start start 2022-01-31T19:43:39.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:39.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:39.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0f6170 con 0x7f4cac0f8860 2022-01-31T19:43:39.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb46c3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0f62b0 con 0x7f4cac0f7ec0 2022-01-31T19:43:39.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.717+0000 7f4cb1c5e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:39.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb1c5e700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33968/0 (socket says 172.21.15.167:33968) 2022-01-31T19:43:39.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb1c5e700 1 -- 172.21.15.167:0/240316000 learned_addr learned my addr 172.21.15.167:0/240316000 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:39.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb245f700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:39.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb1c5e700 1 -- 172.21.15.167:0/240316000 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f7ec0 msgr2=0x7f4cac0f7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb1c5e700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb1c5e700 1 -- 172.21.15.167:0/240316000 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8005040 con 0x7f4cac0f8860 2022-01-31T19:43:39.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb245f700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-01-31T19:43:39.726 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb1c5e700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f5c10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4c9c00e7f0 tx=0x7f4c9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:39.726 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c9c0092e0 con 0x7f4cac0f8860 2022-01-31T19:43:39.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f4c9c00f040 con 0x7f4cac0f8860 2022-01-31T19:43:39.728 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac0f6590 con 0x7f4cac0f8860 2022-01-31T19:43:39.728 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c9c014750 con 0x7f4cac0f8860 2022-01-31T19:43:39.728 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.718+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac10f060 con 0x7f4cac0f8860 2022-01-31T19:43:39.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.719+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac045bb0 con 0x7f4cac0f8860 2022-01-31T19:43:39.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.720+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f4c9c01b030 con 0x7f4cac0f8860 2022-01-31T19:43:39.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.720+0000 7f4ca2ffd700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4c9805e5c0 0x7f4c98060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:39.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.720+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4c9c0786d0 con 0x7f4cac0f8860 2022-01-31T19:43:39.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.720+0000 7f4cb245f700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4c9805e5c0 0x7f4c98060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:39.730 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.721+0000 7f4cb245f700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4c9805e5c0 0x7f4c98060a80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f4ca8000c00 tx=0x7f4ca8016040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:39.730 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.726+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4c9c04a380 con 0x7f4cac0f8860 2022-01-31T19:43:39.913 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.912+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 --> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cac057ff0 con 0x7f4c9805e5c0 2022-01-31T19:43:39.923 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.922+0000 7f4ca2ffd700 1 -- 172.21.15.167:0/240316000 <== mgr.14156 v2:172.21.15.167:6800/3192751116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3168 (secure 0 0 0) 0x7f4cac057ff0 con 0x7f4c9805e5c0 2022-01-31T19:43:39.923 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:43:39.924 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (4m) 89s ago 5m 22.0M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:43:39.924 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (5m) 89s ago 5m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:43:39.924 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (4m) 92s ago 4m 7155k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:43:39.924 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (4m) 89s ago 5m 31.6M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:43:39.925 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.rdkazj smithi167 running (104s) 89s ago 104s 14.4M - 16.2.5 46cf6318c64e 29f596ef1666 2022-01-31T19:43:39.925 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.sgnfie smithi167 running (109s) 89s ago 108s 16.6M - 16.2.5 46cf6318c64e 8c462c58dec4 2022-01-31T19:43:39.925 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.dyqqtj smithi171 running (107s) 92s ago 106s 18.4M - 16.2.5 46cf6318c64e 5e84b5044af1 2022-01-31T19:43:39.925 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.erzfea smithi171 running (102s) 92s ago 102s 14.6M - 16.2.5 46cf6318c64e 802065eafb38 2022-01-31T19:43:39.926 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:9283 running 89s ago 6m - - 2022-01-31T19:43:39.926 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (4m) 92s ago 4m 374M - 16.2.5 46cf6318c64e 0e89e7ce37df 2022-01-31T19:43:39.926 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running 89s ago 6m - 2048M 2022-01-31T19:43:39.926 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (4m) 92s ago 4m 83.9M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:43:39.927 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (5m) 89s ago 5m 18.1M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:43:39.927 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (4m) 92s ago 4m 18.2M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:43:39.927 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (4m) 89s ago 4m 49.6M 6472M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:43:39.927 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (4m) 89s ago 3m 49.4M 6472M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:43:39.927 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (3m) 89s ago 3m 49.0M 6472M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:43:39.928 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (3m) 92s ago 3m 49.9M 5339M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:43:39.928 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (3m) 92s ago 3m 47.2M 5339M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:43:39.928 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (2m) 92s ago 2m 47.3M 5339M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:43:39.928 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (4m) 89s ago 5m 40.1M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:43:39.929 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4c9805e5c0 msgr2=0x7f4c98060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.930 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4c9805e5c0 0x7f4c98060a80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f4ca8000c00 tx=0x7f4ca8016040).stop 2022-01-31T19:43:39.930 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 msgr2=0x7f4cac0f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:39.930 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f5c10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4c9c00e7f0 tx=0x7f4c9c00b040).stop 2022-01-31T19:43:39.930 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 shutdown_connections 2022-01-31T19:43:39.931 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f4c9805e5c0 0x7f4c98060a80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.931 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cac0f7ec0 0x7f4cac0f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.931 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 --2- 172.21.15.167:0/240316000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cac0f8860 0x7f4cac0f5c10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:39.931 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 >> 172.21.15.167:0/240316000 conn(0x7f4cac0f0640 msgr2=0x7f4cac0fb750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:39.932 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 shutdown_connections 2022-01-31T19:43:39.932 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:39.924+0000 7f4cb46c3700 1 -- 172.21.15.167:0/240316000 wait complete. 2022-01-31T19:43:40.035 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.034+0000 7f5a78bb5700 1 -- 172.21.15.167:0/808379153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f4b20 msgr2=0x7f5a740f4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.036 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.034+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/808379153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f4b20 0x7f5a740f4f40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f5a64004660 tx=0x7f5a64009b30).stop 2022-01-31T19:43:40.036 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.034+0000 7f5a78bb5700 1 -- 172.21.15.167:0/808379153 shutdown_connections 2022-01-31T19:43:40.036 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.034+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/808379153 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f54c0 0x7f5a740f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.037 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.034+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/808379153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f4b20 0x7f5a740f4f40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.037 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.034+0000 7f5a78bb5700 1 -- 172.21.15.167:0/808379153 >> 172.21.15.167:0/808379153 conn(0x7f5a740f0640 msgr2=0x7f5a740f2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:40.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:39 smithi167 conmon[32206]: cluster 2022-01-31T19:43:38.748658+0000 mgr.smithi167.aciqpk (mgr.14156) 254 : cluster 2022-01-31T19:43:40.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:39 smithi167 conmon[32206]: [DBG] pgmap v200: 65 pgs: 65 active+clean; 72 MiB data, 308 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 505 op/s 2022-01-31T19:43:40.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 -- 172.21.15.167:0/808379153 shutdown_connections 2022-01-31T19:43:40.040 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 -- 172.21.15.167:0/808379153 wait complete. 2022-01-31T19:43:40.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 Processor -- start 2022-01-31T19:43:40.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 -- start start 2022-01-31T19:43:40.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 0x7f5a740f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:40.041 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f54c0 0x7f5a740f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:40.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740f60b0 con 0x7f5a740f54c0 2022-01-31T19:43:40.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.036+0000 7f5a78bb5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740f61f0 con 0x7f5a740f4b20 2022-01-31T19:43:40.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 0x7f5a740f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:40.042 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 0x7f5a740f76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40296/0 (socket says 172.21.15.167:40296) 2022-01-31T19:43:40.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 -- 172.21.15.167:0/3345489962 learned_addr learned my addr 172.21.15.167:0/3345489962 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:40.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a71d9b700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f54c0 0x7f5a740f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:40.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 -- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f54c0 msgr2=0x7f5a740f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f54c0 0x7f5a740f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.043 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 -- 172.21.15.167:0/3345489962 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64005040 con 0x7f5a740f4b20 2022-01-31T19:43:40.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.037+0000 7f5a7259c700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 0x7f5a740f76d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5a64013a70 tx=0x7f5a64009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:40.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.038+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a64018070 con 0x7f5a740f4b20 2022-01-31T19:43:40.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.038+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f5a640097d0 con 0x7f5a740f4b20 2022-01-31T19:43:40.044 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.038+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6401c3c0 con 0x7f5a740f4b20 2022-01-31T19:43:40.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.038+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a740f6470 con 0x7f5a740f4b20 2022-01-31T19:43:40.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.038+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a740f69d0 con 0x7f5a740f4b20 2022-01-31T19:43:40.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.039+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74045bb0 con 0x7f5a740f4b20 2022-01-31T19:43:40.045 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.039+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7f5a64007ba0 con 0x7f5a740f4b20 2022-01-31T19:43:40.046 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.040+0000 7f5a62ffd700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5a5c05e5c0 0x7f5a5c060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:40.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.040+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5a6407a580 con 0x7f5a740f4b20 2022-01-31T19:43:40.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.040+0000 7f5a71d9b700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5a5c05e5c0 0x7f5a5c060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:40.047 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.040+0000 7f5a71d9b700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5a5c05e5c0 0x7f5a5c060a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5a740f5900 tx=0x7f5a68009c60).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:40.048 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.044+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5a6404c180 con 0x7f5a740f4b20 2022-01-31T19:43:40.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:39 smithi171 conmon[35325]: cluster 2022-01-31T19:43:38.748658+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:43:40.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:39 smithi171 conmon[35325]: mgr.14156) 254 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 72 MiB data, 308 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 505 op/s 2022-01-31T19:43:40.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.284+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5a740587c0 con 0x7f5a740f4b20 2022-01-31T19:43:40.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.285+0000 7f5a62ffd700 1 -- 172.21.15.167:0/3345489962 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+573 (secure 0 0 0) 0x7f5a6404f7a0 con 0x7f5a740f4b20 2022-01-31T19:43:40.288 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:43:40.288 INFO:teuthology.orchestra.run.smithi167.stdout: "mon": { 2022-01-31T19:43:40.288 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:43:40.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:40.289 INFO:teuthology.orchestra.run.smithi167.stdout: "mgr": { 2022-01-31T19:43:40.289 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:43:40.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:40.289 INFO:teuthology.orchestra.run.smithi167.stdout: "osd": { 2022-01-31T19:43:40.289 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6 2022-01-31T19:43:40.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:40.290 INFO:teuthology.orchestra.run.smithi167.stdout: "mds": { 2022-01-31T19:43:40.290 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4 2022-01-31T19:43:40.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:43:40.291 INFO:teuthology.orchestra.run.smithi167.stdout: "overall": { 2022-01-31T19:43:40.291 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 14 2022-01-31T19:43:40.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2022-01-31T19:43:40.292 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:43:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5a5c05e5c0 msgr2=0x7f5a5c060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5a5c05e5c0 0x7f5a5c060a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5a740f5900 tx=0x7f5a68009c60).stop 2022-01-31T19:43:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 msgr2=0x7f5a740f76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 0x7f5a740f76d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5a64013a70 tx=0x7f5a64009b30).stop 2022-01-31T19:43:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 shutdown_connections 2022-01-31T19:43:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7f5a5c05e5c0 0x7f5a5c060a80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a740f4b20 0x7f5a740f76d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 --2- 172.21.15.167:0/3345489962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a740f54c0 0x7f5a740f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 >> 172.21.15.167:0/3345489962 conn(0x7f5a740f0640 msgr2=0x7f5a740fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 shutdown_connections 2022-01-31T19:43:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.287+0000 7f5a78bb5700 1 -- 172.21.15.167:0/3345489962 wait complete. 2022-01-31T19:43:40.422 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.421+0000 7fc8cc846700 1 -- 172.21.15.167:0/1353268049 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f6980 msgr2=0x7fc8c80f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.421+0000 7fc8cc846700 1 --2- 172.21.15.167:0/1353268049 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f6980 0x7fc8c80f6de0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc8bc004660 tx=0x7fc8bc009b30).stop 2022-01-31T19:43:40.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.421+0000 7fc8cc846700 1 -- 172.21.15.167:0/1353268049 shutdown_connections 2022-01-31T19:43:40.423 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.421+0000 7fc8cc846700 1 --2- 172.21.15.167:0/1353268049 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f6980 0x7fc8c80f6de0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.421+0000 7fc8cc846700 1 --2- 172.21.15.167:0/1353268049 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f56e0 0x7fc8c80f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.424 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.421+0000 7fc8cc846700 1 -- 172.21.15.167:0/1353268049 >> 172.21.15.167:0/1353268049 conn(0x7fc8c80f0e60 msgr2=0x7fc8c80f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:40.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.422+0000 7fc8cc846700 1 -- 172.21.15.167:0/1353268049 shutdown_connections 2022-01-31T19:43:40.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.422+0000 7fc8cc846700 1 -- 172.21.15.167:0/1353268049 wait complete. 2022-01-31T19:43:40.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.422+0000 7fc8cc846700 1 Processor -- start 2022-01-31T19:43:40.425 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.422+0000 7fc8cc846700 1 -- start start 2022-01-31T19:43:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.422+0000 7fc8cc846700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f56e0 0x7fc8c8105ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8cc846700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 0x7fc8c81063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8cc846700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8106990 con 0x7fc8c80f6980 2022-01-31T19:43:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8cc846700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8106ad0 con 0x7fc8c80f56e0 2022-01-31T19:43:40.427 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 0x7fc8c81063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:40.427 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 0x7fc8c81063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:33990/0 (socket says 172.21.15.167:33990) 2022-01-31T19:43:40.427 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 -- 172.21.15.167:0/3965992528 learned_addr learned my addr 172.21.15.167:0/3965992528 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:43:40.427 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c659c700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f56e0 0x7fc8c8105ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 -- 172.21.15.167:0/3965992528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f56e0 msgr2=0x7fc8c8105ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f56e0 0x7fc8c8105ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 -- 172.21.15.167:0/3965992528 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc005040 con 0x7fc8c80f6980 2022-01-31T19:43:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8c5d9b700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 0x7fc8c81063e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc8bc000c00 tx=0x7fc8bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:40.429 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8bc019070 con 0x7fc8c80f6980 2022-01-31T19:43:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fc8bc007e70 con 0x7fc8c80f6980 2022-01-31T19:43:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8bc014ce0 con 0x7fc8c80f6980 2022-01-31T19:43:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.423+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c8107410 con 0x7fc8c80f6980 2022-01-31T19:43:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.424+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c8107970 con 0x7fc8c80f6980 2022-01-31T19:43:40.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.425+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c8045bb0 con 0x7fc8c80f6980 2022-01-31T19:43:40.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.425+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 14) v1 ==== 74051+0+0 (secure 0 0 0) 0x7fc8bc025070 con 0x7fc8c80f6980 2022-01-31T19:43:40.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.426+0000 7fc8b6ffd700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc8b005e5c0 0x7fc8b0060a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:43:40.431 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.426+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc8bc079790 con 0x7fc8c80f6980 2022-01-31T19:43:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.426+0000 7fc8c659c700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc8b005e5c0 0x7fc8b0060a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:43:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.427+0000 7fc8c659c700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc8b005e5c0 0x7fc8b0060a80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc8b8009fd0 tx=0x7fc8b800b040).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:43:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.430+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc8bc04b410 con 0x7fc8c80f6980 2022-01-31T19:43:40.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.658+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc8c8027ae0 con 0x7fc8c80f6980 2022-01-31T19:43:40.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.659+0000 7fc8b6ffd700 1 -- 172.21.15.167:0/3965992528 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1689 (secure 0 0 0) 0x7fc8bc04ea30 con 0x7fc8c80f6980 2022-01-31T19:43:40.661 INFO:teuthology.orchestra.run.smithi167.stdout:e16 2022-01-31T19:43:40.661 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:43:40.661 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:43:40.661 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:43:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:43:40.662 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:43:40.663 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:43:40.663 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 16 2022-01-31T19:43:40.663 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:43:40.663 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:43:40.664 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:43:36.817533+0000 2022-01-31T19:43:40.664 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:43:40.664 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:43:40.664 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:43:40.664 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:43:40.665 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:43:40.665 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:43:40.665 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:43:40.666 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:43:40.666 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:43:40.666 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:43:40.667 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:43:40.667 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:43:40.667 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:43:40.667 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:43:40.668 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:43:40.668 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:43:40.668 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:43:40.668 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:43:40.668 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:43:40.669 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:43:40.669 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:43:40.669 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:43:40.669 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:43:40.670 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:43:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:43:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:43:40.671 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc8b005e5c0 msgr2=0x7fc8b0060a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.671 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc8b005e5c0 0x7fc8b0060a80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc8b8009fd0 tx=0x7fc8b800b040).stop 2022-01-31T19:43:40.672 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 msgr2=0x7fc8c81063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:43:40.672 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 0x7fc8c81063e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc8bc000c00 tx=0x7fc8bc00b040).stop 2022-01-31T19:43:40.672 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 shutdown_connections 2022-01-31T19:43:40.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:6800/3192751116,v1:172.21.15.167:6801/3192751116] conn(0x7fc8b005e5c0 0x7fc8b0060a80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc8c80f56e0 0x7fc8c8105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 --2- 172.21.15.167:0/3965992528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8c80f6980 0x7fc8c81063e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:43:40.673 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 >> 172.21.15.167:0/3965992528 conn(0x7fc8c80f0e60 msgr2=0x7fc8c80fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:43:40.674 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 shutdown_connections 2022-01-31T19:43:40.674 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:43:40.662+0000 7fc8cc846700 1 -- 172.21.15.167:0/3965992528 wait complete. 2022-01-31T19:43:40.674 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 16 2022-01-31T19:43:40.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: audit 2022-01-31T19:43:39.608206+0000 mgr.smithi167.aciqpk (mgr.14156) 255 : 2022-01-31T19:43:40.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:40.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: audit 2022-01-31T19:43:39.913278+0000 mgr.smithi167.aciqpk (mgr.14156) 256 : audit [DBG] from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:40.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: audit 2022-01-31T19:43:40.285532+0000 mon.smithi171 2022-01-31T19:43:40.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: (mon.1) 17 : audit [DBG] from='client.? 172.21.15.167:0/3345489962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:43:40.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: audit 2022-01-31T19:43:40.659758+0000 mon.smithi167 (mon 2022-01-31T19:43:40.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:40 smithi167 conmon[32206]: .0) 616 : audit [DBG] from='client.? 172.21.15.167:0/3965992528' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:43:41.047 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:40 smithi171 conmon[35325]: audit 2022-01-31T19:43:39.608206+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:43:41.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:40 smithi171 conmon[35325]: mgr.14156) 255 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:41.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:40 smithi171 conmon[35325]: audit 2022-01-31T19:43:39.913278+0000 mgr.smithi167.aciqpk (mgr.14156) 256 : audit [DBG] from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:43:41.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:40 smithi171 conmon[35325]: audit 2022-01-31T19:43:40.285532+0000 mon.smithi171 (mon.1) 17 : audit [DBG] from='client.? 172.21.15.167:0/3345489962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:43:41.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:40 smithi171 conmon[35325]: audit 2022-01-31T19:43:40.659758+0000 mon.smithi167 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.167:0/3965992528' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:43:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:41 smithi171 conmon[35325]: cluster 2022-01-31T19:43:40.749040+0000 mgr.smithi167.aciqpk (mgr.14156) 257 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 85 MiB data, 334 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 7.2 MiB/s wr, 550 op/s 2022-01-31T19:43:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:41 smithi171 conmon[35325]: audit 2022-01-31T19:43:41.693993+0000 mon.smithi167 (mon.0) 617 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:42.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:41 smithi171 conmon[35325]: audit 2022-01-31T19:43:41.694710+0000 mon.smithi167 (mon.0) 618 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:43:42.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:41 smithi171 conmon[35325]: audit 2022-01-31T19:43:41.695461+0000 mon.smithi167 (mon.0) 619 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:43:42.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:41 smithi171 conmon[35325]: audit 2022-01-31T19:43:41.696060+0000 mon.smithi167 (mon.0) 620 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:43:42.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: cluster 2022-01-31T19:43:40. 2022-01-31T19:43:42.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: 749040+0000 mgr.smithi167.aciqpk (mgr.14156) 257 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 85 MiB data, 334 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 7.2 MiB/s wr, 550 op/s 2022-01-31T19:43:42.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:43:42.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: 43:41.693993+0000 mon.smithi167 (mon.0) 617 : audit 2022-01-31T19:43:42.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:42.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: audit 2022-01-31T19:43:42.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: 2022-01-31T19: 2022-01-31T19:43:42.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: 43:41.694710+0000 mon.smithi167 ( 2022-01-31T19:43:42.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: mon.0) 618 : audit [INF] 2022-01-31T19:43:42.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:43:42.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: audit 2022-01-31T19:43:42.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: 2022-01-31T19:43 2022-01-31T19:43:42.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: :41.695461+0000 mon.smithi167 (mon. 2022-01-31T19:43:42.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: 0) 619 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:43:42.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:42.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: T19:43:41.696060+0000 mon.smithi167 2022-01-31T19:43:42.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: (mon.0) 620 : audit [DBG] 2022-01-31T19:43:42.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:41 smithi167 conmon[32206]: from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:43:43.059 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:42 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:41.688509+0000 2022-01-31T19:43:43.060 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:42 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 258 : cephadm [INF] Upgrade: Updating mgr.smithi171.asyxnc 2022-01-31T19:43:43.060 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:42 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:41.696751+0000 mgr.smithi167.aciqpk (mgr.14156) 259 : cephadm [INF] Deploying daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:43:43.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:43:41.688509+0000 mgr.smithi167.aciqpk (mgr.14156) 258 : cephadm 2022-01-31T19:43:43.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:42 smithi167 conmon[32206]: [INF] Upgrade: Updating mgr.smithi171.asyxnc 2022-01-31T19:43:43.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:43:41.696751+0000 mgr.smithi167.aciqpk (mgr.14156) 259 : cephadm [INF] Deploying daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:43:43.988 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:43 smithi171 conmon[35325]: cluster 2022-01-31T19:43:42.749489+0000 2022-01-31T19:43:43.988 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:43 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14156) 260 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 87 MiB data, 350 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.1 MiB/s wr, 441 op/s 2022-01-31T19:43:44.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:43 smithi167 conmon[32206]: cluster 2022-01-31T19:43:42. 2022-01-31T19:43:44.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:43 smithi167 conmon[32206]: 749489+0000 mgr.smithi167.aciqpk (mgr.14156) 260 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 87 MiB data, 350 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.1 MiB/s wr, 441 op/s 2022-01-31T19:43:44.417 INFO:tasks.workunit.client.0.smithi167.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2022-01-31T19:43:44.418 INFO:tasks.workunit.client.0.smithi167.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2022-01-31T19:43:44.418 INFO:tasks.workunit.client.0.smithi167.stderr:+ make 2022-01-31T19:43:45.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:45 smithi171 conmon[35325]: audit 2022-01-31T19:43:44.037757+0000 mon.smithi167 (mon.0) 621 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:45.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:45 smithi171 conmon[35325]: audit 2022-01-31T19:43:44.038519+0000 mon.smithi167 (mon.0) 622 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:45.250 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:45 smithi171 conmon[35325]: audit 2022-01-31T19:43:44.039735+0000 mon.smithi167 (mon.0) 623 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:43:45.250 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:45 smithi171 conmon[35325]: audit 2022-01-31T19:43:44.040549+0000 mon.smithi167 (mon.0) 624 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:43:45.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:45 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:43:45.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:45 smithi167 conmon[32206]: 19:43:44.037757+0000 mon.smithi167 (mon.0) 621 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:45.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:45 smithi167 conmon[32206]: audit 2022-01-31T19:43:44.038519+0000 mon.smithi167 (mon.0) 622 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:45.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:45 smithi167 conmon[32206]: audit 2022-01-31T19:43:44.039735+0000 mon.smithi167 (mon.0) 623 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:43:45.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:45 smithi167 conmon[32206]: audit 2022-01-31T19:43:44.040549+0000 mon.smithi167 (mon.0) 624 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:43:46.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:46 smithi171 conmon[35325]: cluster 2022-01-31T19:43:44.750025+0000 mgr.smithi167.aciqpk (mgr.14156) 261 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 93 MiB data, 373 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 346 op/s 2022-01-31T19:43:46.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:46 smithi167 conmon[32206]: cluster 2022-01-31T19:43:44.750025+0000 mgr.smithi167.aciqpk (mgr.14156) 261 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 93 MiB data, 373 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 346 op/s 2022-01-31T19:43:48.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:48 smithi171 conmon[35325]: cluster 2022-01-31T19:43:46.750425+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:43:48.476 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:48 smithi171 conmon[35325]: .14156) 262 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 102 MiB data, 423 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 272 op/s 2022-01-31T19:43:48.477 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:48 smithi167 conmon[32206]: cluster 2022-01-31T19:43:46.750425+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:43:48.477 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:48 smithi167 conmon[32206]: .14156) 262 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 102 MiB data, 423 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 272 op/s 2022-01-31T19:43:49.478 INFO:tasks.workunit.client.0.smithi167.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2022-01-31T19:43:49.696 INFO:tasks.workunit.client.0.smithi167.stderr:++ readlink -f fsstress 2022-01-31T19:43:49.697 INFO:tasks.workunit.client.0.smithi167.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2022-01-31T19:43:49.698 INFO:tasks.workunit.client.0.smithi167.stderr:+ popd 2022-01-31T19:43:49.698 INFO:tasks.workunit.client.0.smithi167.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2022-01-31T19:43:49.699 INFO:tasks.workunit.client.0.smithi167.stderr:+ popd 2022-01-31T19:43:49.699 INFO:tasks.workunit.client.0.smithi167.stdout:~/cephtest/mnt.0/client.0/tmp 2022-01-31T19:43:49.700 INFO:tasks.workunit.client.0.smithi167.stderr:++ mktemp -d -p . 2022-01-31T19:43:49.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ T=./tmp.orDGTlWuTY 2022-01-31T19:43:49.700 INFO:tasks.workunit.client.0.smithi167.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.orDGTlWuTY -l 1 -n 1000 -p 10 -v 2022-01-31T19:43:49.701 INFO:tasks.workunit.client.0.smithi167.stdout:seed = 1643287892 2022-01-31T19:43:49.709 INFO:tasks.workunit.client.0.smithi167.stdout:4/0: dread - no filename 2022-01-31T19:43:49.710 INFO:tasks.workunit.client.0.smithi167.stdout:5/0: dread - no filename 2022-01-31T19:43:49.710 INFO:tasks.workunit.client.0.smithi167.stdout:2/0: dread - no filename 2022-01-31T19:43:49.710 INFO:tasks.workunit.client.0.smithi167.stdout:5/1: stat - no entries 2022-01-31T19:43:49.711 INFO:tasks.workunit.client.0.smithi167.stdout:4/1: chown . 581 1 2022-01-31T19:43:49.711 INFO:tasks.workunit.client.0.smithi167.stdout:2/1: chown . 8712 1 2022-01-31T19:43:49.711 INFO:tasks.workunit.client.0.smithi167.stdout:2/2: fsync - no filename 2022-01-31T19:43:49.711 INFO:tasks.workunit.client.0.smithi167.stdout:5/2: getdents . 0 2022-01-31T19:43:49.717 INFO:tasks.workunit.client.0.smithi167.stdout:6/0: rename - no filename 2022-01-31T19:43:49.717 INFO:tasks.workunit.client.0.smithi167.stdout:6/1: chown . 13 1 2022-01-31T19:43:49.718 INFO:tasks.workunit.client.0.smithi167.stdout:1/0: mkdir d0 0 2022-01-31T19:43:49.718 INFO:tasks.workunit.client.0.smithi167.stdout:1/1: chown d0 26817185 1 2022-01-31T19:43:49.719 INFO:tasks.workunit.client.0.smithi167.stdout:3/0: creat f0 x:0 0 0 2022-01-31T19:43:49.719 INFO:tasks.workunit.client.0.smithi167.stdout:4/2: mkdir d0 0 2022-01-31T19:43:49.720 INFO:tasks.workunit.client.0.smithi167.stdout:4/3: dwrite - no filename 2022-01-31T19:43:49.720 INFO:tasks.workunit.client.0.smithi167.stdout:4/4: write - no filename 2022-01-31T19:43:49.720 INFO:tasks.workunit.client.0.smithi167.stdout:4/5: read - no filename 2022-01-31T19:43:49.720 INFO:tasks.workunit.client.0.smithi167.stdout:4/6: dwrite - no filename 2022-01-31T19:43:49.721 INFO:tasks.workunit.client.0.smithi167.stdout:4/7: write - no filename 2022-01-31T19:43:49.721 INFO:tasks.workunit.client.0.smithi167.stdout:2/3: creat f0 x:0 0 0 2022-01-31T19:43:49.722 INFO:tasks.workunit.client.0.smithi167.stdout:2/4: creat f1 x:0 0 0 2022-01-31T19:43:49.722 INFO:tasks.workunit.client.0.smithi167.stdout:5/3: symlink l0 0 2022-01-31T19:43:49.723 INFO:tasks.workunit.client.0.smithi167.stdout:7/0: dread - no filename 2022-01-31T19:43:49.723 INFO:tasks.workunit.client.0.smithi167.stdout:7/1: link - no file 2022-01-31T19:43:49.723 INFO:tasks.workunit.client.0.smithi167.stdout:7/2: link - no file 2022-01-31T19:43:49.724 INFO:tasks.workunit.client.0.smithi167.stdout:6/2: creat f0 x:0 0 0 2022-01-31T19:43:49.724 INFO:tasks.workunit.client.0.smithi167.stdout:6/3: write f0 [1029203,109889] 0 2022-01-31T19:43:49.725 INFO:tasks.workunit.client.0.smithi167.stdout:1/2: creat d0/f1 x:0 0 0 2022-01-31T19:43:49.725 INFO:tasks.workunit.client.0.smithi167.stdout:1/3: dread - d0/f1 zero size 2022-01-31T19:43:49.726 INFO:tasks.workunit.client.0.smithi167.stdout:3/1: rename f0 to f1 0 2022-01-31T19:43:49.726 INFO:tasks.workunit.client.0.smithi167.stdout:3/2: dread - f1 zero size 2022-01-31T19:43:49.726 INFO:tasks.workunit.client.0.smithi167.stdout:3/3: chown f1 1828 1 2022-01-31T19:43:49.726 INFO:tasks.workunit.client.0.smithi167.stdout:3/4: write f1 [632485,17680] 0 2022-01-31T19:43:49.727 INFO:tasks.workunit.client.0.smithi167.stdout:4/8: symlink d0/l1 0 2022-01-31T19:43:49.727 INFO:tasks.workunit.client.0.smithi167.stdout:4/9: write - no filename 2022-01-31T19:43:49.728 INFO:tasks.workunit.client.0.smithi167.stdout:5/4: creat f1 x:0 0 0 2022-01-31T19:43:49.728 INFO:tasks.workunit.client.0.smithi167.stdout:5/5: chown f1 0 1 2022-01-31T19:43:49.729 INFO:tasks.workunit.client.0.smithi167.stdout:5/6: read - f1 zero size 2022-01-31T19:43:49.729 INFO:tasks.workunit.client.0.smithi167.stdout:5/7: write f1 [682188,7562] 0 2022-01-31T19:43:49.729 INFO:tasks.workunit.client.0.smithi167.stdout:8/0: readlink - no filename 2022-01-31T19:43:49.731 INFO:tasks.workunit.client.0.smithi167.stdout:7/3: symlink l0 0 2022-01-31T19:43:49.731 INFO:tasks.workunit.client.0.smithi167.stdout:7/4: chown l0 36 1 2022-01-31T19:43:49.731 INFO:tasks.workunit.client.0.smithi167.stdout:7/5: fdatasync - no filename 2022-01-31T19:43:49.732 INFO:tasks.workunit.client.0.smithi167.stdout:7/6: write - no filename 2022-01-31T19:43:49.732 INFO:tasks.workunit.client.0.smithi167.stdout:7/7: dread - no filename 2022-01-31T19:43:49.732 INFO:tasks.workunit.client.0.smithi167.stdout:7/8: chown l0 2559383 1 2022-01-31T19:43:49.732 INFO:tasks.workunit.client.0.smithi167.stdout:7/9: dwrite - no filename 2022-01-31T19:43:49.732 INFO:tasks.workunit.client.0.smithi167.stdout:6/4: symlink l1 0 2022-01-31T19:43:49.733 INFO:tasks.workunit.client.0.smithi167.stdout:6/5: readlink l1 0 2022-01-31T19:43:49.733 INFO:tasks.workunit.client.0.smithi167.stdout:1/4: mknod d0/c2 0 2022-01-31T19:43:49.733 INFO:tasks.workunit.client.0.smithi167.stdout:1/5: rename d0 to d0/d3 22 2022-01-31T19:43:49.733 INFO:tasks.workunit.client.0.smithi167.stdout:1/6: creat d0/f4 x:0 0 0 2022-01-31T19:43:49.734 INFO:tasks.workunit.client.0.smithi167.stdout:1/7: readlink - no filename 2022-01-31T19:43:49.734 INFO:tasks.workunit.client.0.smithi167.stdout:1/8: write d0/f1 [725146,29330] 0 2022-01-31T19:43:49.734 INFO:tasks.workunit.client.0.smithi167.stdout:4/10: symlink d0/l2 0 2022-01-31T19:43:49.734 INFO:tasks.workunit.client.0.smithi167.stdout:4/11: dwrite - no filename 2022-01-31T19:43:49.736 INFO:tasks.workunit.client.0.smithi167.stdout:2/5: link f1 f2 0 2022-01-31T19:43:49.736 INFO:tasks.workunit.client.0.smithi167.stdout:5/8: mknod c2 0 2022-01-31T19:43:49.736 INFO:tasks.workunit.client.0.smithi167.stdout:5/9: write f1 [1008055,114552] 0 2022-01-31T19:43:49.737 INFO:tasks.workunit.client.0.smithi167.stdout:9/0: chown . 27333864 1 2022-01-31T19:43:49.737 INFO:tasks.workunit.client.0.smithi167.stdout:8/1: creat f0 x:0 0 0 2022-01-31T19:43:49.737 INFO:tasks.workunit.client.0.smithi167.stdout:8/2: chown f0 52015 1 2022-01-31T19:43:49.738 INFO:tasks.workunit.client.0.smithi167.stdout:8/3: fdatasync f0 0 2022-01-31T19:43:49.738 INFO:tasks.workunit.client.0.smithi167.stdout:8/4: truncate f0 395967 0 2022-01-31T19:43:49.738 INFO:tasks.workunit.client.0.smithi167.stdout:8/5: readlink - no filename 2022-01-31T19:43:49.739 INFO:tasks.workunit.client.0.smithi167.stdout:7/10: symlink l1 0 2022-01-31T19:43:49.739 INFO:tasks.workunit.client.0.smithi167.stdout:6/6: mknod c2 0 2022-01-31T19:43:49.739 INFO:tasks.workunit.client.0.smithi167.stdout:6/7: creat f3 x:0 0 0 2022-01-31T19:43:49.740 INFO:tasks.workunit.client.0.smithi167.stdout:4/12: mkdir d0/d3 0 2022-01-31T19:43:49.740 INFO:tasks.workunit.client.0.smithi167.stdout:4/13: dwrite - no filename 2022-01-31T19:43:49.741 INFO:tasks.workunit.client.0.smithi167.stdout:2/6: rename f1 to f3 0 2022-01-31T19:43:49.741 INFO:tasks.workunit.client.0.smithi167.stdout:2/7: dread - f3 zero size 2022-01-31T19:43:49.742 INFO:tasks.workunit.client.0.smithi167.stdout:9/1: symlink l0 0 2022-01-31T19:43:49.743 INFO:tasks.workunit.client.0.smithi167.stdout:7/11: creat f2 x:0 0 0 2022-01-31T19:43:49.743 INFO:tasks.workunit.client.0.smithi167.stdout:7/12: stat f2 0 2022-01-31T19:43:49.744 INFO:tasks.workunit.client.0.smithi167.stdout:6/8: unlink f0 0 2022-01-31T19:43:49.744 INFO:tasks.workunit.client.0.smithi167.stdout:6/9: rmdir - no directory 2022-01-31T19:43:49.745 INFO:tasks.workunit.client.0.smithi167.stdout:6/10: unlink c2 0 2022-01-31T19:43:49.745 INFO:tasks.workunit.client.0.smithi167.stdout:1/9: link d0/f4 d0/f5 0 2022-01-31T19:43:49.746 INFO:tasks.workunit.client.0.smithi167.stdout:1/10: dread - d0/f4 zero size 2022-01-31T19:43:49.746 INFO:tasks.workunit.client.0.smithi167.stdout:1/11: dread - d0/f5 zero size 2022-01-31T19:43:49.746 INFO:tasks.workunit.client.0.smithi167.stdout:1/12: creat d0/f6 x:0 0 0 2022-01-31T19:43:49.747 INFO:tasks.workunit.client.0.smithi167.stdout:1/13: write d0/f1 [1106838,57418] 0 2022-01-31T19:43:49.747 INFO:tasks.workunit.client.0.smithi167.stdout:4/14: symlink d0/d3/l4 0 2022-01-31T19:43:49.748 INFO:tasks.workunit.client.0.smithi167.stdout:2/8: mknod c4 0 2022-01-31T19:43:49.748 INFO:tasks.workunit.client.0.smithi167.stdout:2/9: truncate f2 129776 0 2022-01-31T19:43:49.748 INFO:tasks.workunit.client.0.smithi167.stdout:2/10: fdatasync f2 0 2022-01-31T19:43:49.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/2: creat f1 x:0 0 0 2022-01-31T19:43:49.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/3: truncate f1 912525 0 2022-01-31T19:43:49.750 INFO:tasks.workunit.client.0.smithi167.stdout:7/13: mknod c3 0 2022-01-31T19:43:49.750 INFO:tasks.workunit.client.0.smithi167.stdout:7/14: write f2 [967198,70362] 0 2022-01-31T19:43:49.752 INFO:tasks.workunit.client.0.smithi167.stdout:6/11: symlink l4 0 2022-01-31T19:43:49.754 INFO:tasks.workunit.client.0.smithi167.stdout:4/15: link d0/l1 d0/l5 0 2022-01-31T19:43:49.755 INFO:tasks.workunit.client.0.smithi167.stdout:2/11: link c4 c5 0 2022-01-31T19:43:49.755 INFO:tasks.workunit.client.0.smithi167.stdout:2/12: fdatasync f3 0 2022-01-31T19:43:49.756 INFO:tasks.workunit.client.0.smithi167.stdout:4/16: rmdir d0 39 2022-01-31T19:43:49.758 INFO:tasks.workunit.client.0.smithi167.stdout:8/6: dwrite f0 [0,4194304] 0 2022-01-31T19:43:49.758 INFO:tasks.workunit.client.0.smithi167.stdout:8/7: creat f1 x:0 0 0 2022-01-31T19:43:49.760 INFO:tasks.workunit.client.0.smithi167.stdout:3/5: dwrite f1 [0,4194304] 0 2022-01-31T19:43:49.761 INFO:tasks.workunit.client.0.smithi167.stdout:7/15: dread f2 [0,4194304] 0 2022-01-31T19:43:49.761 INFO:tasks.workunit.client.0.smithi167.stdout:8/8: symlink l2 0 2022-01-31T19:43:49.762 INFO:tasks.workunit.client.0.smithi167.stdout:7/16: dread f2 [0,4194304] 0 2022-01-31T19:43:49.763 INFO:tasks.workunit.client.0.smithi167.stdout:7/17: write f2 [1271994,60026] 0 2022-01-31T19:43:49.763 INFO:tasks.workunit.client.0.smithi167.stdout:7/18: chown l1 2040917 1 2022-01-31T19:43:49.763 INFO:tasks.workunit.client.0.smithi167.stdout:3/6: symlink l2 0 2022-01-31T19:43:49.764 INFO:tasks.workunit.client.0.smithi167.stdout:3/7: link l2 l3 0 2022-01-31T19:43:49.766 INFO:tasks.workunit.client.0.smithi167.stdout:3/8: rename l2 to l4 0 2022-01-31T19:43:49.766 INFO:tasks.workunit.client.0.smithi167.stdout:8/9: write f0 [2059778,12109] 0 2022-01-31T19:43:49.767 INFO:tasks.workunit.client.0.smithi167.stdout:8/10: symlink l3 0 2022-01-31T19:43:49.767 INFO:tasks.workunit.client.0.smithi167.stdout:7/19: fdatasync f2 0 2022-01-31T19:43:49.768 INFO:tasks.workunit.client.0.smithi167.stdout:8/11: mknod c4 0 2022-01-31T19:43:49.768 INFO:tasks.workunit.client.0.smithi167.stdout:8/12: dread - f1 zero size 2022-01-31T19:43:49.771 INFO:tasks.workunit.client.0.smithi167.stdout:8/13: rename f1 to f5 0 2022-01-31T19:43:49.773 INFO:tasks.workunit.client.0.smithi167.stdout:1/14: dwrite d0/f4 [0,4194304] 0 2022-01-31T19:43:49.774 INFO:tasks.workunit.client.0.smithi167.stdout:8/14: mkdir d6 0 2022-01-31T19:43:49.779 INFO:tasks.workunit.client.0.smithi167.stdout:9/4: dwrite f1 [0,4194304] 0 2022-01-31T19:43:49.781 INFO:tasks.workunit.client.0.smithi167.stdout:8/15: mknod d6/c7 0 2022-01-31T19:43:49.781 INFO:tasks.workunit.client.0.smithi167.stdout:8/16: chown f5 13301439 1 2022-01-31T19:43:49.781 INFO:tasks.workunit.client.0.smithi167.stdout:8/17: chown d6 12856833 1 2022-01-31T19:43:49.782 INFO:tasks.workunit.client.0.smithi167.stdout:8/18: chown c4 183261824 1 2022-01-31T19:43:49.782 INFO:tasks.workunit.client.0.smithi167.stdout:2/13: dwrite f3 [0,4194304] 0 2022-01-31T19:43:49.786 INFO:tasks.workunit.client.0.smithi167.stdout:7/20: dwrite f2 [0,4194304] 0 2022-01-31T19:43:49.788 INFO:tasks.workunit.client.0.smithi167.stdout:8/19: creat d6/f8 x:0 0 0 2022-01-31T19:43:49.789 INFO:tasks.workunit.client.0.smithi167.stdout:9/5: dread f1 [0,4194304] 0 2022-01-31T19:43:49.789 INFO:tasks.workunit.client.0.smithi167.stdout:9/6: chown l0 125074729 1 2022-01-31T19:43:49.792 INFO:tasks.workunit.client.0.smithi167.stdout:3/9: fsync f1 0 2022-01-31T19:43:49.793 INFO:tasks.workunit.client.0.smithi167.stdout:3/10: readlink l4 0 2022-01-31T19:43:49.794 INFO:tasks.workunit.client.0.smithi167.stdout:5/10: fsync f1 0 2022-01-31T19:43:49.794 INFO:tasks.workunit.client.0.smithi167.stdout:5/11: chown l0 0 1 2022-01-31T19:43:49.794 INFO:tasks.workunit.client.0.smithi167.stdout:5/12: fsync f1 0 2022-01-31T19:43:49.796 INFO:tasks.workunit.client.0.smithi167.stdout:8/20: symlink d6/l9 0 2022-01-31T19:43:49.796 INFO:tasks.workunit.client.0.smithi167.stdout:8/21: dread - f5 zero size 2022-01-31T19:43:49.797 INFO:tasks.workunit.client.0.smithi167.stdout:8/22: chown l2 882 1 2022-01-31T19:43:49.798 INFO:tasks.workunit.client.0.smithi167.stdout:3/11: symlink l5 0 2022-01-31T19:43:49.799 INFO:tasks.workunit.client.0.smithi167.stdout:3/12: fdatasync f1 0 2022-01-31T19:43:49.799 INFO:tasks.workunit.client.0.smithi167.stdout:5/13: rename c2 to c3 0 2022-01-31T19:43:49.803 INFO:tasks.workunit.client.0.smithi167.stdout:5/14: symlink l4 0 2022-01-31T19:43:49.804 INFO:tasks.workunit.client.0.smithi167.stdout:6/12: fsync f3 0 2022-01-31T19:43:49.804 INFO:tasks.workunit.client.0.smithi167.stdout:6/13: chown l1 1755030 1 2022-01-31T19:43:49.808 INFO:tasks.workunit.client.0.smithi167.stdout:6/14: symlink l5 0 2022-01-31T19:43:49.808 INFO:tasks.workunit.client.0.smithi167.stdout:6/15: write f3 [113330,94084] 0 2022-01-31T19:43:49.808 INFO:tasks.workunit.client.0.smithi167.stdout:6/16: read f3 [152567,91763] 0 2022-01-31T19:43:49.810 INFO:tasks.workunit.client.0.smithi167.stdout:2/14: dread f2 [0,4194304] 0 2022-01-31T19:43:49.810 INFO:tasks.workunit.client.0.smithi167.stdout:1/15: dwrite d0/f1 [0,4194304] 0 2022-01-31T19:43:49.810 INFO:tasks.workunit.client.0.smithi167.stdout:6/17: rename l5 to l6 0 2022-01-31T19:43:49.811 INFO:tasks.workunit.client.0.smithi167.stdout:2/15: rename c4 to c6 0 2022-01-31T19:43:49.811 INFO:tasks.workunit.client.0.smithi167.stdout:2/16: chown f3 2843 1 2022-01-31T19:43:49.812 INFO:tasks.workunit.client.0.smithi167.stdout:1/16: write d0/f1 [58783,68922] 0 2022-01-31T19:43:49.812 INFO:tasks.workunit.client.0.smithi167.stdout:6/18: mknod c7 0 2022-01-31T19:43:49.813 INFO:tasks.workunit.client.0.smithi167.stdout:6/19: unlink l4 0 2022-01-31T19:43:49.813 INFO:tasks.workunit.client.0.smithi167.stdout:1/17: write d0/f1 [1171981,4260] 0 2022-01-31T19:43:49.818 INFO:tasks.workunit.client.0.smithi167.stdout:5/15: dwrite f1 [0,4194304] 0 2022-01-31T19:43:49.818 INFO:tasks.workunit.client.0.smithi167.stdout:3/13: write f1 [4251149,107963] 0 2022-01-31T19:43:49.818 INFO:tasks.workunit.client.0.smithi167.stdout:3/14: chown l4 0 1 2022-01-31T19:43:49.822 INFO:tasks.workunit.client.0.smithi167.stdout:1/18: mknod d0/c7 0 2022-01-31T19:43:49.824 INFO:tasks.workunit.client.0.smithi167.stdout:8/23: unlink f5 0 2022-01-31T19:43:49.825 INFO:tasks.workunit.client.0.smithi167.stdout:8/24: fdatasync d6/f8 0 2022-01-31T19:43:49.825 INFO:tasks.workunit.client.0.smithi167.stdout:8/25: stat l2 0 2022-01-31T19:43:49.826 INFO:tasks.workunit.client.0.smithi167.stdout:9/7: write f1 [5055630,83318] 0 2022-01-31T19:43:49.828 INFO:tasks.workunit.client.0.smithi167.stdout:2/17: truncate f3 3876969 0 2022-01-31T19:43:49.828 INFO:tasks.workunit.client.0.smithi167.stdout:2/18: readlink - no filename 2022-01-31T19:43:49.828 INFO:tasks.workunit.client.0.smithi167.stdout:2/19: truncate f0 768419 0 2022-01-31T19:43:49.829 INFO:tasks.workunit.client.0.smithi167.stdout:7/21: write f2 [4987676,51006] 0 2022-01-31T19:43:49.832 INFO:tasks.workunit.client.0.smithi167.stdout:3/15: dread f1 [0,4194304] 0 2022-01-31T19:43:49.835 INFO:tasks.workunit.client.0.smithi167.stdout:9/8: symlink l2 0 2022-01-31T19:43:49.838 INFO:tasks.workunit.client.0.smithi167.stdout:3/16: write f1 [4177871,116271] 0 2022-01-31T19:43:49.840 INFO:tasks.workunit.client.0.smithi167.stdout:3/17: write f1 [1101079,28517] 0 2022-01-31T19:43:49.840 INFO:tasks.workunit.client.0.smithi167.stdout:1/19: truncate d0/f4 1530224 0 2022-01-31T19:43:49.841 INFO:tasks.workunit.client.0.smithi167.stdout:1/20: stat d0/c2 0 2022-01-31T19:43:49.842 INFO:tasks.workunit.client.0.smithi167.stdout:3/18: symlink l6 0 2022-01-31T19:43:49.843 INFO:tasks.workunit.client.0.smithi167.stdout:1/21: write d0/f1 [1056118,69162] 0 2022-01-31T19:43:49.845 INFO:tasks.workunit.client.0.smithi167.stdout:1/22: symlink d0/l8 0 2022-01-31T19:43:49.847 INFO:tasks.workunit.client.0.smithi167.stdout:9/9: dread f1 [0,4194304] 0 2022-01-31T19:43:49.847 INFO:tasks.workunit.client.0.smithi167.stdout:9/10: write f1 [5897619,59515] 0 2022-01-31T19:43:49.848 INFO:tasks.workunit.client.0.smithi167.stdout:5/16: truncate f1 3503041 0 2022-01-31T19:43:49.852 INFO:tasks.workunit.client.0.smithi167.stdout:3/19: truncate f1 1831537 0 2022-01-31T19:43:49.853 INFO:tasks.workunit.client.0.smithi167.stdout:3/20: creat f7 x:0 0 0 2022-01-31T19:43:49.853 INFO:tasks.workunit.client.0.smithi167.stdout:3/21: creat f8 x:0 0 0 2022-01-31T19:43:49.854 INFO:tasks.workunit.client.0.smithi167.stdout:5/17: fsync f1 0 2022-01-31T19:43:49.855 INFO:tasks.workunit.client.0.smithi167.stdout:7/22: dwrite f2 [4194304,4194304] 0 2022-01-31T19:43:49.856 INFO:tasks.workunit.client.0.smithi167.stdout:1/23: truncate d0/f1 4175147 0 2022-01-31T19:43:49.856 INFO:tasks.workunit.client.0.smithi167.stdout:1/24: fsync d0/f5 0 2022-01-31T19:43:49.857 INFO:tasks.workunit.client.0.smithi167.stdout:8/26: dwrite d6/f8 [0,4194304] 0 2022-01-31T19:43:49.857 INFO:tasks.workunit.client.0.smithi167.stdout:9/11: fdatasync f1 0 2022-01-31T19:43:49.859 INFO:tasks.workunit.client.0.smithi167.stdout:1/25: write d0/f4 [1271009,88838] 0 2022-01-31T19:43:49.859 INFO:tasks.workunit.client.0.smithi167.stdout:3/22: mkdir d9 0 2022-01-31T19:43:49.860 INFO:tasks.workunit.client.0.smithi167.stdout:2/20: read f2 [1637611,6590] 0 2022-01-31T19:43:49.860 INFO:tasks.workunit.client.0.smithi167.stdout:7/23: symlink l4 0 2022-01-31T19:43:49.862 INFO:tasks.workunit.client.0.smithi167.stdout:7/24: write f2 [5231903,94924] 0 2022-01-31T19:43:49.862 INFO:tasks.workunit.client.0.smithi167.stdout:7/25: readlink l0 0 2022-01-31T19:43:49.863 INFO:tasks.workunit.client.0.smithi167.stdout:9/12: mknod c3 0 2022-01-31T19:43:49.863 INFO:tasks.workunit.client.0.smithi167.stdout:9/13: fsync f1 0 2022-01-31T19:43:49.864 INFO:tasks.workunit.client.0.smithi167.stdout:1/26: mknod d0/c9 0 2022-01-31T19:43:49.865 INFO:tasks.workunit.client.0.smithi167.stdout:2/21: link f2 f7 0 2022-01-31T19:43:49.865 INFO:tasks.workunit.client.0.smithi167.stdout:2/22: creat f8 x:0 0 0 2022-01-31T19:43:49.868 INFO:tasks.workunit.client.0.smithi167.stdout:2/23: unlink c5 0 2022-01-31T19:43:49.868 INFO:tasks.workunit.client.0.smithi167.stdout:2/24: truncate f8 70556 0 2022-01-31T19:43:49.870 INFO:tasks.workunit.client.0.smithi167.stdout:2/25: truncate f7 214143 0 2022-01-31T19:43:49.877 INFO:tasks.workunit.client.0.smithi167.stdout:5/18: dwrite f1 [0,4194304] 0 2022-01-31T19:43:49.877 INFO:tasks.workunit.client.0.smithi167.stdout:5/19: fsync f1 0 2022-01-31T19:43:49.891 INFO:tasks.workunit.client.0.smithi167.stdout:9/14: dwrite f1 [4194304,4194304] 0 2022-01-31T19:43:49.893 INFO:tasks.workunit.client.0.smithi167.stdout:9/15: rename f1 to f4 0 2022-01-31T19:43:49.894 INFO:tasks.workunit.client.0.smithi167.stdout:8/27: dwrite d6/f8 [0,4194304] 0 2022-01-31T19:43:49.894 INFO:tasks.workunit.client.0.smithi167.stdout:8/28: creat d6/fa x:0 0 0 2022-01-31T19:43:49.894 INFO:tasks.workunit.client.0.smithi167.stdout:8/29: readlink l3 0 2022-01-31T19:43:49.895 INFO:tasks.workunit.client.0.smithi167.stdout:5/20: dwrite f1 [0,4194304] 0 2022-01-31T19:43:49.896 INFO:tasks.workunit.client.0.smithi167.stdout:8/30: read f0 [1853477,62891] 0 2022-01-31T19:43:49.896 INFO:tasks.workunit.client.0.smithi167.stdout:1/27: read d0/f1 [969677,42251] 0 2022-01-31T19:43:49.897 INFO:tasks.workunit.client.0.smithi167.stdout:5/21: mkdir d5 0 2022-01-31T19:43:49.898 INFO:tasks.workunit.client.0.smithi167.stdout:5/22: chown d5 10046 1 2022-01-31T19:43:49.900 INFO:tasks.workunit.client.0.smithi167.stdout:8/31: mkdir d6/db 0 2022-01-31T19:43:49.904 INFO:tasks.workunit.client.0.smithi167.stdout:7/26: truncate f2 1049410 0 2022-01-31T19:43:49.910 INFO:tasks.workunit.client.0.smithi167.stdout:8/32: dread f0 [0,4194304] 0 2022-01-31T19:43:49.910 INFO:tasks.workunit.client.0.smithi167.stdout:8/33: stat l3 0 2022-01-31T19:43:49.911 INFO:tasks.workunit.client.0.smithi167.stdout:8/34: readlink l3 0 2022-01-31T19:43:49.911 INFO:tasks.workunit.client.0.smithi167.stdout:8/35: creat d6/db/fc x:0 0 0 2022-01-31T19:43:49.912 INFO:tasks.workunit.client.0.smithi167.stdout:8/36: rename d6/db/fc to d6/db/fd 0 2022-01-31T19:43:49.915 INFO:tasks.workunit.client.0.smithi167.stdout:3/23: dwrite f1 [0,4194304] 0 2022-01-31T19:43:49.915 INFO:tasks.workunit.client.0.smithi167.stdout:8/37: dread f0 [0,4194304] 0 2022-01-31T19:43:49.916 INFO:tasks.workunit.client.0.smithi167.stdout:8/38: mkdir d6/db/de 0 2022-01-31T19:43:49.918 INFO:tasks.workunit.client.0.smithi167.stdout:3/24: write f1 [694731,74882] 0 2022-01-31T19:43:49.919 INFO:tasks.workunit.client.0.smithi167.stdout:3/25: write f7 [154495,86333] 0 2022-01-31T19:43:49.919 INFO:tasks.workunit.client.0.smithi167.stdout:3/26: truncate f8 163054 0 2022-01-31T19:43:49.921 INFO:tasks.workunit.client.0.smithi167.stdout:5/23: dwrite f1 [4194304,4194304] 0 2022-01-31T19:43:49.922 INFO:tasks.workunit.client.0.smithi167.stdout:5/24: fdatasync f1 0 2022-01-31T19:43:49.922 INFO:tasks.workunit.client.0.smithi167.stdout:5/25: chown l4 63592709 1 2022-01-31T19:43:49.922 INFO:tasks.workunit.client.0.smithi167.stdout:1/28: dwrite d0/f4 [0,4194304] 0 2022-01-31T19:43:49.923 INFO:tasks.workunit.client.0.smithi167.stdout:9/16: link f4 f5 0 2022-01-31T19:43:49.923 INFO:tasks.workunit.client.0.smithi167.stdout:5/26: write f1 [7218871,14392] 0 2022-01-31T19:43:49.925 INFO:tasks.workunit.client.0.smithi167.stdout:1/29: rename d0/c7 to d0/ca 0 2022-01-31T19:43:49.926 INFO:tasks.workunit.client.0.smithi167.stdout:1/30: read - d0/f6 zero size 2022-01-31T19:43:49.927 INFO:tasks.workunit.client.0.smithi167.stdout:5/27: mkdir d5/d6 0 2022-01-31T19:43:49.929 INFO:tasks.workunit.client.0.smithi167.stdout:1/31: mknod d0/cb 0 2022-01-31T19:43:49.930 INFO:tasks.workunit.client.0.smithi167.stdout:7/27: write f2 [1830803,86745] 0 2022-01-31T19:43:49.931 INFO:tasks.workunit.client.0.smithi167.stdout:7/28: stat c3 0 2022-01-31T19:43:49.933 INFO:tasks.workunit.client.0.smithi167.stdout:5/28: creat d5/f7 x:0 0 0 2022-01-31T19:43:49.934 INFO:tasks.workunit.client.0.smithi167.stdout:5/29: truncate d5/f7 620348 0 2022-01-31T19:43:49.934 INFO:tasks.workunit.client.0.smithi167.stdout:5/30: fdatasync d5/f7 0 2022-01-31T19:43:49.935 INFO:tasks.workunit.client.0.smithi167.stdout:1/32: rename d0/f4 to d0/fc 0 2022-01-31T19:43:49.936 INFO:tasks.workunit.client.0.smithi167.stdout:1/33: creat d0/fd x:0 0 0 2022-01-31T19:43:49.936 INFO:tasks.workunit.client.0.smithi167.stdout:1/34: dread - d0/f6 zero size 2022-01-31T19:43:49.936 INFO:tasks.workunit.client.0.smithi167.stdout:1/35: stat d0/fc 0 2022-01-31T19:43:49.936 INFO:tasks.workunit.client.0.smithi167.stdout:1/36: creat d0/fe x:0 0 0 2022-01-31T19:43:49.937 INFO:tasks.workunit.client.0.smithi167.stdout:1/37: fdatasync d0/fe 0 2022-01-31T19:43:49.939 INFO:tasks.workunit.client.0.smithi167.stdout:7/29: mknod c5 0 2022-01-31T19:43:49.939 INFO:tasks.workunit.client.0.smithi167.stdout:7/30: stat c5 0 2022-01-31T19:43:49.939 INFO:tasks.workunit.client.0.smithi167.stdout:7/31: creat f6 x:0 0 0 2022-01-31T19:43:49.940 INFO:tasks.workunit.client.0.smithi167.stdout:7/32: write f6 [1026500,58714] 0 2022-01-31T19:43:49.940 INFO:tasks.workunit.client.0.smithi167.stdout:7/33: chown f6 115714520 1 2022-01-31T19:43:49.940 INFO:tasks.workunit.client.0.smithi167.stdout:7/34: creat f7 x:0 0 0 2022-01-31T19:43:49.941 INFO:tasks.workunit.client.0.smithi167.stdout:9/17: dwrite f4 [0,4194304] 0 2022-01-31T19:43:49.943 INFO:tasks.workunit.client.0.smithi167.stdout:1/38: mknod d0/cf 0 2022-01-31T19:43:49.943 INFO:tasks.workunit.client.0.smithi167.stdout:1/39: truncate d0/fe 565918 0 2022-01-31T19:43:49.943 INFO:tasks.workunit.client.0.smithi167.stdout:1/40: dread - d0/f6 zero size 2022-01-31T19:43:49.943 INFO:tasks.workunit.client.0.smithi167.stdout:1/41: dread - d0/f6 zero size 2022-01-31T19:43:49.946 INFO:tasks.workunit.client.0.smithi167.stdout:7/35: mkdir d8 0 2022-01-31T19:43:49.946 INFO:tasks.workunit.client.0.smithi167.stdout:3/27: dread f1 [0,4194304] 0 2022-01-31T19:43:49.946 INFO:tasks.workunit.client.0.smithi167.stdout:7/36: write f2 [1598850,14135] 0 2022-01-31T19:43:49.947 INFO:tasks.workunit.client.0.smithi167.stdout:8/39: dwrite d6/f8 [0,4194304] 0 2022-01-31T19:43:49.947 INFO:tasks.workunit.client.0.smithi167.stdout:9/18: unlink c3 0 2022-01-31T19:43:49.949 INFO:tasks.workunit.client.0.smithi167.stdout:3/28: creat d9/fa x:0 0 0 2022-01-31T19:43:49.949 INFO:tasks.workunit.client.0.smithi167.stdout:7/37: symlink d8/l9 0 2022-01-31T19:43:49.952 INFO:tasks.workunit.client.0.smithi167.stdout:5/31: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:43:49.953 INFO:tasks.workunit.client.0.smithi167.stdout:5/32: chown f1 43259 1 2022-01-31T19:43:49.968 INFO:tasks.workunit.client.0.smithi167.stdout:1/42: dread d0/f5 [0,4194304] 0 2022-01-31T19:43:49.968 INFO:tasks.workunit.client.0.smithi167.stdout:1/43: readlink d0/l8 0 2022-01-31T19:43:49.986 INFO:tasks.workunit.client.0.smithi167.stdout:8/40: dwrite f0 [0,4194304] 0 2022-01-31T19:43:49.987 INFO:tasks.workunit.client.0.smithi167.stdout:8/41: creat d6/ff x:0 0 0 2022-01-31T19:43:49.987 INFO:tasks.workunit.client.0.smithi167.stdout:8/42: dread - d6/db/fd zero size 2022-01-31T19:43:49.989 INFO:tasks.workunit.client.0.smithi167.stdout:8/43: creat d6/db/de/f10 x:0 0 0 2022-01-31T19:43:49.989 INFO:tasks.workunit.client.0.smithi167.stdout:8/44: fsync d6/db/fd 0 2022-01-31T19:43:49.989 INFO:tasks.workunit.client.0.smithi167.stdout:8/45: mknod d6/db/c11 0 2022-01-31T19:43:49.989 INFO:tasks.workunit.client.0.smithi167.stdout:8/46: creat d6/f12 x:0 0 0 2022-01-31T19:43:49.990 INFO:tasks.workunit.client.0.smithi167.stdout:8/47: chown d6/fa 493 1 2022-01-31T19:43:49.990 INFO:tasks.workunit.client.0.smithi167.stdout:8/48: fdatasync d6/ff 0 2022-01-31T19:43:49.990 INFO:tasks.workunit.client.0.smithi167.stdout:8/49: dread - d6/db/de/f10 zero size 2022-01-31T19:43:49.990 INFO:tasks.workunit.client.0.smithi167.stdout:8/50: chown d6/f8 1753532 1 2022-01-31T19:43:49.991 INFO:tasks.workunit.client.0.smithi167.stdout:8/51: truncate d6/ff 768791 0 2022-01-31T19:43:49.991 INFO:tasks.workunit.client.0.smithi167.stdout:8/52: creat d6/f13 x:0 0 0 2022-01-31T19:43:49.991 INFO:tasks.workunit.client.0.smithi167.stdout:8/53: creat d6/f14 x:0 0 0 2022-01-31T19:43:49.992 INFO:tasks.workunit.client.0.smithi167.stdout:8/54: mknod d6/db/de/c15 0 2022-01-31T19:43:49.995 INFO:tasks.workunit.client.0.smithi167.stdout:9/19: dwrite f4 [0,4194304] 0 2022-01-31T19:43:49.996 INFO:tasks.workunit.client.0.smithi167.stdout:7/38: dwrite f7 [0,4194304] 0 2022-01-31T19:43:49.996 INFO:tasks.workunit.client.0.smithi167.stdout:7/39: chown f6 25366 1 2022-01-31T19:43:49.997 INFO:tasks.workunit.client.0.smithi167.stdout:9/20: write f5 [1681974,129725] 0 2022-01-31T19:43:49.998 INFO:tasks.workunit.client.0.smithi167.stdout:5/33: dwrite f1 [4194304,4194304] 0 2022-01-31T19:43:49.999 INFO:tasks.workunit.client.0.smithi167.stdout:7/40: creat d8/fa x:0 0 0 2022-01-31T19:43:49.999 INFO:tasks.workunit.client.0.smithi167.stdout:3/29: dwrite f7 [0,4194304] 0 2022-01-31T19:43:50.000 INFO:tasks.workunit.client.0.smithi167.stdout:3/30: chown f7 120057124 1 2022-01-31T19:43:50.000 INFO:tasks.workunit.client.0.smithi167.stdout:3/31: write d9/fa [68361,110219] 0 2022-01-31T19:43:50.001 INFO:tasks.workunit.client.0.smithi167.stdout:9/21: symlink l6 0 2022-01-31T19:43:50.001 INFO:tasks.workunit.client.0.smithi167.stdout:9/22: creat f7 x:0 0 0 2022-01-31T19:43:50.001 INFO:tasks.workunit.client.0.smithi167.stdout:9/23: creat f8 x:0 0 0 2022-01-31T19:43:50.001 INFO:tasks.workunit.client.0.smithi167.stdout:9/24: chown f5 13767953 1 2022-01-31T19:43:50.001 INFO:tasks.workunit.client.0.smithi167.stdout:9/25: dread - f7 zero size 2022-01-31T19:43:50.002 INFO:tasks.workunit.client.0.smithi167.stdout:9/26: stat f7 0 2022-01-31T19:43:50.002 INFO:tasks.workunit.client.0.smithi167.stdout:9/27: chown l6 7257 1 2022-01-31T19:43:50.002 INFO:tasks.workunit.client.0.smithi167.stdout:9/28: rmdir - no directory 2022-01-31T19:43:50.002 INFO:tasks.workunit.client.0.smithi167.stdout:9/29: creat f9 x:0 0 0 2022-01-31T19:43:50.003 INFO:tasks.workunit.client.0.smithi167.stdout:9/30: dread - f7 zero size 2022-01-31T19:43:50.003 INFO:tasks.workunit.client.0.smithi167.stdout:9/31: dread - f8 zero size 2022-01-31T19:43:50.003 INFO:tasks.workunit.client.0.smithi167.stdout:1/44: dwrite d0/f6 [0,4194304] 0 2022-01-31T19:43:50.004 INFO:tasks.workunit.client.0.smithi167.stdout:7/41: symlink d8/lb 0 2022-01-31T19:43:50.004 INFO:tasks.workunit.client.0.smithi167.stdout:7/42: truncate f7 4628681 0 2022-01-31T19:43:50.005 INFO:tasks.workunit.client.0.smithi167.stdout:9/32: write f5 [1889594,49308] 0 2022-01-31T19:43:50.012 INFO:tasks.workunit.client.0.smithi167.stdout:1/45: mkdir d0/d10 0 2022-01-31T19:43:50.012 INFO:tasks.workunit.client.0.smithi167.stdout:3/32: dread f1 [0,4194304] 0 2022-01-31T19:43:50.012 INFO:tasks.workunit.client.0.smithi167.stdout:3/33: fdatasync f1 0 2022-01-31T19:43:50.013 INFO:tasks.workunit.client.0.smithi167.stdout:3/34: write f8 [1100783,45725] 0 2022-01-31T19:43:50.013 INFO:tasks.workunit.client.0.smithi167.stdout:7/43: mkdir d8/dc 0 2022-01-31T19:43:50.015 INFO:tasks.workunit.client.0.smithi167.stdout:9/33: mknod ca 0 2022-01-31T19:43:50.015 INFO:tasks.workunit.client.0.smithi167.stdout:9/34: write f4 [9268652,30221] 0 2022-01-31T19:43:50.030 INFO:tasks.workunit.client.0.smithi167.stdout:1/46: dread d0/f1 [0,4194304] 0 2022-01-31T19:43:50.030 INFO:tasks.workunit.client.0.smithi167.stdout:8/55: dwrite d6/f14 [0,4194304] 0 2022-01-31T19:43:50.030 INFO:tasks.workunit.client.0.smithi167.stdout:8/56: chown d6/db/c11 327237 1 2022-01-31T19:43:50.031 INFO:tasks.workunit.client.0.smithi167.stdout:1/47: mkdir d0/d11 0 2022-01-31T19:43:50.032 INFO:tasks.workunit.client.0.smithi167.stdout:1/48: write d0/fe [1437144,11661] 0 2022-01-31T19:43:50.033 INFO:tasks.workunit.client.0.smithi167.stdout:8/57: symlink d6/l16 0 2022-01-31T19:43:50.033 INFO:tasks.workunit.client.0.smithi167.stdout:8/58: write d6/f13 [763461,45226] 0 2022-01-31T19:43:50.033 INFO:tasks.workunit.client.0.smithi167.stdout:8/59: stat d6/c7 0 2022-01-31T19:43:50.033 INFO:tasks.workunit.client.0.smithi167.stdout:8/60: write d6/db/de/f10 [760690,113886] 0 2022-01-31T19:43:50.034 INFO:tasks.workunit.client.0.smithi167.stdout:1/49: symlink d0/d10/l12 0 2022-01-31T19:43:50.035 INFO:tasks.workunit.client.0.smithi167.stdout:1/50: chown d0/f6 24 1 2022-01-31T19:43:50.035 INFO:tasks.workunit.client.0.smithi167.stdout:1/51: creat d0/d10/f13 x:0 0 0 2022-01-31T19:43:50.035 INFO:tasks.workunit.client.0.smithi167.stdout:7/44: dwrite f6 [0,4194304] 0 2022-01-31T19:43:50.036 INFO:tasks.workunit.client.0.smithi167.stdout:3/35: dwrite f7 [0,4194304] 0 2022-01-31T19:43:50.036 INFO:tasks.workunit.client.0.smithi167.stdout:1/52: mkdir d0/d10/d14 0 2022-01-31T19:43:50.036 INFO:tasks.workunit.client.0.smithi167.stdout:1/53: fdatasync d0/f5 0 2022-01-31T19:43:50.040 INFO:tasks.workunit.client.0.smithi167.stdout:3/36: write f1 [1395981,115949] 0 2022-01-31T19:43:50.042 INFO:tasks.workunit.client.0.smithi167.stdout:1/54: mknod d0/d11/c15 0 2022-01-31T19:43:50.043 INFO:tasks.workunit.client.0.smithi167.stdout:3/37: mknod d9/cb 0 2022-01-31T19:43:50.043 INFO:tasks.workunit.client.0.smithi167.stdout:3/38: truncate f8 1842703 0 2022-01-31T19:43:50.043 INFO:tasks.workunit.client.0.smithi167.stdout:3/39: write d9/fa [552439,40057] 0 2022-01-31T19:43:50.044 INFO:tasks.workunit.client.0.smithi167.stdout:1/55: mkdir d0/d10/d14/d16 0 2022-01-31T19:43:50.044 INFO:tasks.workunit.client.0.smithi167.stdout:3/40: link f8 d9/fc 0 2022-01-31T19:43:50.046 INFO:tasks.workunit.client.0.smithi167.stdout:1/56: rename d0/d10/f13 to d0/f17 0 2022-01-31T19:43:50.047 INFO:tasks.workunit.client.0.smithi167.stdout:3/41: mknod d9/cd 0 2022-01-31T19:43:50.054 INFO:tasks.workunit.client.0.smithi167.stdout:7/45: dwrite f2 [0,4194304] 0 2022-01-31T19:43:50.055 INFO:tasks.workunit.client.0.smithi167.stdout:8/61: dwrite d6/f8 [0,4194304] 0 2022-01-31T19:43:50.055 INFO:tasks.workunit.client.0.smithi167.stdout:8/62: truncate d6/f13 1215609 0 2022-01-31T19:43:50.056 INFO:tasks.workunit.client.0.smithi167.stdout:7/46: link d8/lb d8/ld 0 2022-01-31T19:43:50.056 INFO:tasks.workunit.client.0.smithi167.stdout:7/47: stat l4 0 2022-01-31T19:43:50.056 INFO:tasks.workunit.client.0.smithi167.stdout:7/48: chown f2 37791 1 2022-01-31T19:43:50.061 INFO:tasks.workunit.client.0.smithi167.stdout:1/57: rename d0/d10/d14 to d0/d10/d18 0 2022-01-31T19:43:50.061 INFO:tasks.workunit.client.0.smithi167.stdout:1/58: creat d0/d10/f19 x:0 0 0 2022-01-31T19:43:50.064 INFO:tasks.workunit.client.0.smithi167.stdout:7/49: mknod d8/ce 0 2022-01-31T19:43:50.066 INFO:tasks.workunit.client.0.smithi167.stdout:3/42: truncate d9/fc 1014364 0 2022-01-31T19:43:50.067 INFO:tasks.workunit.client.0.smithi167.stdout:7/50: mkdir d8/dc/df 0 2022-01-31T19:43:50.068 INFO:tasks.workunit.client.0.smithi167.stdout:7/51: mknod d8/c10 0 2022-01-31T19:43:50.070 INFO:tasks.workunit.client.0.smithi167.stdout:7/52: unlink d8/ce 0 2022-01-31T19:43:50.070 INFO:tasks.workunit.client.0.smithi167.stdout:7/53: readlink d8/ld 0 2022-01-31T19:43:50.071 INFO:tasks.workunit.client.0.smithi167.stdout:7/54: truncate f6 5212269 0 2022-01-31T19:43:50.071 INFO:tasks.workunit.client.0.smithi167.stdout:7/55: readlink d8/ld 0 2022-01-31T19:43:50.072 INFO:tasks.workunit.client.0.smithi167.stdout:7/56: mknod d8/dc/c11 0 2022-01-31T19:43:50.073 INFO:tasks.workunit.client.0.smithi167.stdout:8/63: dwrite d6/db/fd [0,4194304] 0 2022-01-31T19:43:50.073 INFO:tasks.workunit.client.0.smithi167.stdout:8/64: truncate d6/fa 713207 0 2022-01-31T19:43:50.074 INFO:tasks.workunit.client.0.smithi167.stdout:7/57: rename d8/dc/df to d8/dc/d12 0 2022-01-31T19:43:50.075 INFO:tasks.workunit.client.0.smithi167.stdout:8/65: unlink d6/l9 0 2022-01-31T19:43:50.076 INFO:tasks.workunit.client.0.smithi167.stdout:7/58: link l4 d8/l13 0 2022-01-31T19:43:50.082 INFO:tasks.workunit.client.0.smithi167.stdout:7/59: mknod d8/dc/d12/c14 0 2022-01-31T19:43:50.084 INFO:tasks.workunit.client.0.smithi167.stdout:1/59: dwrite d0/fd [0,4194304] 0 2022-01-31T19:43:50.089 INFO:tasks.workunit.client.0.smithi167.stdout:1/60: rmdir d0/d10 39 2022-01-31T19:43:50.097 INFO:tasks.workunit.client.0.smithi167.stdout:8/66: dwrite d6/f13 [0,4194304] 0 2022-01-31T19:43:50.099 INFO:tasks.workunit.client.0.smithi167.stdout:8/67: truncate d6/db/fd 2053243 0 2022-01-31T19:43:50.101 INFO:tasks.workunit.client.0.smithi167.stdout:8/68: rename d6/db/de/c15 to d6/db/c17 0 2022-01-31T19:43:50.102 INFO:tasks.workunit.client.0.smithi167.stdout:8/69: dread - d6/f12 zero size 2022-01-31T19:43:50.102 INFO:tasks.workunit.client.0.smithi167.stdout:8/70: write d6/db/de/f10 [1297595,104604] 0 2022-01-31T19:43:50.103 INFO:tasks.workunit.client.0.smithi167.stdout:8/71: write d6/db/de/f10 [645021,18861] 0 2022-01-31T19:43:50.103 INFO:tasks.workunit.client.0.smithi167.stdout:8/72: mkdir d6/db/de/d18 0 2022-01-31T19:43:50.104 INFO:tasks.workunit.client.0.smithi167.stdout:8/73: fdatasync d6/ff 0 2022-01-31T19:43:50.107 INFO:tasks.workunit.client.0.smithi167.stdout:7/60: dwrite f7 [0,4194304] 0 2022-01-31T19:43:50.107 INFO:tasks.workunit.client.0.smithi167.stdout:7/61: fdatasync f6 0 2022-01-31T19:43:50.108 INFO:tasks.workunit.client.0.smithi167.stdout:1/61: dwrite d0/f6 [0,4194304] 0 2022-01-31T19:43:50.109 INFO:tasks.workunit.client.0.smithi167.stdout:7/62: creat d8/dc/f15 x:0 0 0 2022-01-31T19:43:50.111 INFO:tasks.workunit.client.0.smithi167.stdout:1/62: rmdir d0/d11 39 2022-01-31T19:43:50.113 INFO:tasks.workunit.client.0.smithi167.stdout:1/63: creat d0/d10/f1a x:0 0 0 2022-01-31T19:43:50.114 INFO:tasks.workunit.client.0.smithi167.stdout:1/64: creat d0/f1b x:0 0 0 2022-01-31T19:43:50.116 INFO:tasks.workunit.client.0.smithi167.stdout:1/65: rename d0/d11/c15 to d0/c1c 0 2022-01-31T19:43:50.118 INFO:tasks.workunit.client.0.smithi167.stdout:1/66: write d0/f5 [2710869,106499] 0 2022-01-31T19:43:50.118 INFO:tasks.workunit.client.0.smithi167.stdout:1/67: stat d0/d11 0 2022-01-31T19:43:50.133 INFO:tasks.workunit.client.0.smithi167.stdout:7/63: dwrite d8/dc/f15 [0,4194304] 0 2022-01-31T19:43:50.134 INFO:tasks.workunit.client.0.smithi167.stdout:8/74: dwrite d6/f14 [0,4194304] 0 2022-01-31T19:43:50.138 INFO:tasks.workunit.client.0.smithi167.stdout:8/75: dread d6/db/de/f10 [0,4194304] 0 2022-01-31T19:43:50.138 INFO:tasks.workunit.client.0.smithi167.stdout:8/76: chown f0 21 1 2022-01-31T19:43:50.138 INFO:tasks.workunit.client.0.smithi167.stdout:8/77: chown d6/f8 2979 1 2022-01-31T19:43:50.139 INFO:tasks.workunit.client.0.smithi167.stdout:8/78: chown d6/f14 647 1 2022-01-31T19:43:50.141 INFO:tasks.workunit.client.0.smithi167.stdout:8/79: rename f0 to d6/db/de/d18/f19 0 2022-01-31T19:43:50.141 INFO:tasks.workunit.client.0.smithi167.stdout:8/80: fsync d6/f14 0 2022-01-31T19:43:50.142 INFO:tasks.workunit.client.0.smithi167.stdout:7/64: getdents d8/dc 0 2022-01-31T19:43:50.144 INFO:tasks.workunit.client.0.smithi167.stdout:7/65: mknod d8/dc/d12/c16 0 2022-01-31T19:43:50.146 INFO:tasks.workunit.client.0.smithi167.stdout:7/66: creat d8/dc/f17 x:0 0 0 2022-01-31T19:43:50.147 INFO:tasks.workunit.client.0.smithi167.stdout:7/67: mkdir d8/dc/d18 0 2022-01-31T19:43:50.149 INFO:tasks.workunit.client.0.smithi167.stdout:7/68: mkdir d8/dc/d19 0 2022-01-31T19:43:50.149 INFO:tasks.workunit.client.0.smithi167.stdout:7/69: truncate f7 5438697 0 2022-01-31T19:43:50.149 INFO:tasks.workunit.client.0.smithi167.stdout:7/70: write f6 [5192742,104809] 0 2022-01-31T19:43:50.151 INFO:tasks.workunit.client.0.smithi167.stdout:8/81: dread d6/f8 [0,4194304] 0 2022-01-31T19:43:50.152 INFO:tasks.workunit.client.0.smithi167.stdout:7/71: creat d8/dc/d19/f1a x:0 0 0 2022-01-31T19:43:50.175 INFO:tasks.workunit.client.0.smithi167.stdout:8/82: dwrite d6/f14 [0,4194304] 0 2022-01-31T19:43:50.189 INFO:tasks.workunit.client.0.smithi167.stdout:8/83: dwrite d6/db/de/d18/f19 [0,4194304] 0 2022-01-31T19:43:50.189 INFO:tasks.workunit.client.0.smithi167.stdout:8/84: creat d6/db/f1a x:0 0 0 2022-01-31T19:43:50.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:50 smithi171 conmon[35325]: cluster 2022-01-31T19:43:48.750777+0000 mgr.smithi167.aciqpk (mgr.14156) 263 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 145 MiB data, 588 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 7.7 MiB/s wr, 743 op/s 2022-01-31T19:43:50.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:50 smithi167 conmon[32206]: cluster 2022-01-31T19:43:48.750777+0000 mgr.smithi167.aciqpk (mgr.14156) 263 : cluster 2022-01-31T19:43:50.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:50 smithi167 conmon[32206]: [DBG] pgmap v205: 65 pgs: 65 active+clean; 145 MiB data, 588 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 7.7 MiB/s wr, 743 op/s 2022-01-31T19:43:52.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: cluster 2022-01-31T19:43:50.751254+0000 mgr.smithi167.aciqpk (mgr.14156) 264 : cluster 2022-01-31T19:43:52.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: [DBG] pgmap v206: 65 pgs: 65 active+clean; 175 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 9.4 MiB/s wr, 864 op/s 2022-01-31T19:43:52.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: audit 2022-01-31T19:43:50.941275+0000 mon.smithi167 (mon.0) 625 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:52.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: audit 2022-01-31T19:43:51.412566+0000 mon.smithi167 ( 2022-01-31T19:43:52.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: mon.0) 626 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:52.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: audit 2022-01-31T19:43:51.417740+0000 mon.smithi167 (mon.0) 627 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:52.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:51.418793+0000 mgr.smithi167.aciqpk (mgr.14156) 265 : cephadm [INF] 2022-01-31T19:43:52.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: Upgrade: Need to upgrade myself (mgr.smithi167.aciqpk) 2022-01-31T19:43:52.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:51. 2022-01-31T19:43:52.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: 420899+0000 mgr.smithi167.aciqpk (mgr.14156) 266 : cephadm [INF] Failing over to other MGR 2022-01-31T19:43:52.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:43:52.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: T19:43:51.421166+0000 mon.smithi167 (mon.0) 628 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr fail", "who": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:43:52.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: cluster 2022-01-31T19:43:51.425157+0000 mon.smithi167 (mon. 2022-01-31T19:43:52.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:51 smithi171 conmon[35325]: 0) 629 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2022-01-31T19:43:52.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: cluster 2022-01-31T19:43:50.751254+0000 mgr.smithi167.aciqpk (mgr.14156) 264 : cluster 2022-01-31T19:43:52.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: [DBG] pgmap v206: 65 pgs: 65 active+clean; 175 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 9.4 MiB/s wr, 864 op/s 2022-01-31T19:43:52.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: audit 2022-01-31T19:43:50.941275+0000 mon.smithi167 (mon.0) 625 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:52.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: audit 2022-01-31T19:43:51.412566+0000 mon.smithi167 (mon.0 2022-01-31T19:43:52.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: ) 626 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' 2022-01-31T19:43:52.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:43:52.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: -31T19:43:51.417740+0000 mon.smithi167 ( 2022-01-31T19:43:52.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: mon.0) 627 : audit [DBG] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:52.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: cephadm 2022-01 2022-01-31T19:43:52.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: -31T19:43:51.418793+0000 mgr.smithi167.aciqpk (mgr.14156) 2022-01-31T19:43:52.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: 265 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi167.aciqpk) 2022-01-31T19:43:52.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: cephadm 2022-01-31 2022-01-31T19:43:52.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: T19:43:51.420899+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:43:52.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: 14156) 266 : cephadm [INF] Failing over to other MGR 2022-01-31T19:43:52.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:43:52.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: 31T19:43:51.421166+0000 mon.smithi167 (mon. 2022-01-31T19:43:52.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: 0) 628 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr fail", "who": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:43:52.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: cluster 2022-01-31T19: 2022-01-31T19:43:52.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: 43:51.425157+0000 mon.smithi167 (mon.0) 629 : 2022-01-31T19:43:52.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:51 smithi167 conmon[32206]: cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2022-01-31T19:43:53.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cluster 2022-01-31T19:43:52.118359+0000 mon.smithi167 (mon.0) 630 : cluster [INF] Active manager daemon smithi171.asyxnc restarted 2022-01-31T19:43:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.119061+0000 mon.smithi171 (mon.1) 18 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:43:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cluster 2022-01-31T19:43:52.119081+0000 mon.smithi167 (mon.0) 631 : cluster [INF] Activating manager daemon smithi171.asyxnc 2022-01-31T19:43:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.119531+0000 mon.smithi171 (mon.1) 19 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:43:53.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cluster 2022-01-31T19:43:52.123735+0000 mon.smithi167 (mon.0) 632 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2022-01-31T19:43:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.128487+0000 mon.smithi171 (mon.1) 20 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:43:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.128813+0000 mon.smithi171 (mon.1) 21 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:43:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cluster 2022-01-31T19:43:52.270398+0000 mon.smithi167 (mon.0) 633 : cluster [DBG] Standby manager daemon smithi167.aciqpk started 2022-01-31T19:43:53.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.420961+0000 mon.smithi167 (mon.0) 634 : audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "mgr fail", "who": "smithi167.aciqpk"}]': finished 2022-01-31T19:43:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cluster 2022-01-31T19:43:52.421022+0000 mon.smithi167 (mon.0) 635 : cluster [DBG] mgrmap e15: smithi171.asyxnc(active, starting, since 0.302184s), standbys: smithi167.aciqpk 2022-01-31T19:43:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52. 2022-01-31T19:43:53.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 421199+0000 mon.smithi171 (mon.1) 22 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:43:53.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.421367 2022-01-31T19:43:53.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: +0000 mon.smithi171 (mon.1) 23 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:43:53.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:43:53.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 31T19:43:52.421661+0000 mon.smithi171 (mon.1 2022-01-31T19:43:53.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: ) 24 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.sgnfie"}]: dispatch 2022-01-31T19:43:53.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43: 2022-01-31T19:43:53.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 52.421906+0000 mon.smithi171 (mon.1) 25 : audit 2022-01-31T19:43:53.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.rdkazj"}]: dispatch 2022-01-31T19:43:53.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52.422125+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 26 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.dyqqtj"}]: dispatch 2022-01-31T19:43:53.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52 2022-01-31T19:43:53.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: .422305+0000 mon.smithi171 (mon.1) 27 : audit [DBG] 2022-01-31T19:43:53.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.erzfea"}]: dispatch 2022-01-31T19:43:53.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52.422648+0000 mon.smithi171 (mon.1) 28 : 2022-01-31T19:43:53.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr metadata", "who": "smithi171.asyxnc", "id": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:43:53.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:43:53.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 43:52.422845+0000 mon.smithi171 (mon.1) 29 2022-01-31T19:43:53.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:43:53.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43: 2022-01-31T19:43:53.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 52.423077+0000 mon.smithi171 (mon.1) 30 : audit 2022-01-31T19:43:53.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:43:53.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52 2022-01-31T19:43:53.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: .423341+0000 mon.smithi171 (mon.1) 31 : audit [DBG] 2022-01-31T19:43:53.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:43:53.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:43:53.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: -31T19:43:52.423593+0000 mon.smithi171 (mon.1) 32 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:43:53.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.423894+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 33 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:43:53.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:43:53.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: T19:43:52.424117+0000 mon.smithi171 (mon.1) 34 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:43:53.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52.424341+0000 mon.smithi171 (mon.1) 35 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:43:53.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.425209+0000 mon.smithi171 (mon. 2022-01-31T19:43:53.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 1) 36 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:43:53.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:43:53.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 31T19:43:52.425597+0000 mon.smithi171 (mon.1) 37 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:43:53.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52.426089+0000 mon.smithi171 (mon.1) 38 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:43:53.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cluster 2022-01-31T19:43:52.469305+0000 2022-01-31T19:43:53.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: mon.smithi167 (mon.0) 636 : cluster [INF] Manager daemon smithi171.asyxnc is now available 2022-01-31T19:43:53.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.486661+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.122 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 39 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:53.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:43:53.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: T19:43:52.489146+0000 mon.smithi171 (mon.1) 40 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:53.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T 2022-01-31T19:43:53.123 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 19:43:52.490868+0000 mon.smithi171 (mon.1) 41 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:43:53.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43: 2022-01-31T19:43:53.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 52.491915+0000 mon.smithi171 (mon.1) 42 : audit [INF] 2022-01-31T19:43:53.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:43:53.124 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52.497501+0000 mon.smithi171 (mon.1 2022-01-31T19:43:53.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: ) 43 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2022-01-31T19:43:53.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022 2022-01-31T19:43:53.125 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: -01-31T19:43:52.497770+0000 mon.smithi167 (mon.0) 637 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2022-01-31T19:43:53.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.512949+0000 mon.smithi171 2022-01-31T19:43:53.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: (mon.1) 44 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:43:53.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.126 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 2022-01-31T19:43:52.513342+0000 mon.smithi167 (mon.0) 638 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:43:53.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.516179+0000 2022-01-31T19:43:53.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: mon.smithi171 (mon.1) 45 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/trash_purge_schedule"}]: dispatch 2022-01-31T19:43:53.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31T19:43:52.516491+0000 mon.smithi167 (mon.0 2022-01-31T19:43:53.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: ) 639 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/trash_purge_schedule"}]: dispatch 2022-01-31T19:43:53.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:43:53.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: T19:43:52.534693+0000 mon.smithi167 (mon.0) 640 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:53.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:52.824309+0000 mgr.smithi171.asyxnc (mgr.24387) 2022-01-31T19:43:53.128 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: 1 : cephadm [INF] Deploying cephadm binary to smithi167 2022-01-31T19:43:53.129 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: cephadm 2022-01-31 2022-01-31T19:43:53.129 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:52 smithi171 conmon[35325]: T19:43:52.871041+0000 mgr.smithi171.asyxnc (mgr.24387) 2 : cephadm [INF] Deploying cephadm binary to smithi171 2022-01-31T19:43:53.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster 2022-01-31T19:43:52.118359+0000 mon.smithi167 (mon.0) 630 : 2022-01-31T19:43:53.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster [INF] Active manager daemon smithi171.asyxnc restarted 2022-01-31T19:43:53.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:52.119061+0000 mon.smithi171 (mon.1) 18 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:43:53.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster 2022-01-31T19:43:52.119081+0000 mon.smithi167 (mon.0) 631 : cluster [INF] Activating manager daemon smithi171.asyxnc 2022-01-31T19:43:53.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:52.119531+0000 mon.smithi171 (mon.1) 19 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:43:53.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster 2022-01-31T19:43:52.123735+0000 mon.smithi167 (mon.0) 632 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2022-01-31T19:43:53.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:52.128487+0000 mon.smithi171 (mon.1) 20 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:43:53.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:52.128813+0000 mon.smithi171 (mon.1) 21 : audit [DBG] from='mgr.? 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:43:53.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster 2022-01-31T19:43:52.270398+0000 mon.smithi167 (mon.0) 633 : cluster [DBG] Standby manager daemon smithi167.aciqpk started 2022-01-31T19:43:53.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:52.420961+0000 mon.smithi167 (mon.0) 634 : 2022-01-31T19:43:53.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit [INF] from='mgr.14156 172.21.15.167:0/2564848990' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "mgr fail", "who": "smithi167.aciqpk"}]': finished 2022-01-31T19:43:53.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster 2022-01-31T19:43:52.421022+0000 mon.smithi167 (mon.0) 635 : cluster [DBG] mgrmap e15: smithi171.asyxnc(active, starting, since 0.302184s), standbys: smithi167.aciqpk 2022-01-31T19:43:53.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43 2022-01-31T19:43:53.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :52.421199+0000 mon.smithi171 (mon.1) 22 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:43:53.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:52 2022-01-31T19:43:53.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .421367+0000 mon.smithi171 (mon.1) 23 : audit 2022-01-31T19:43:53.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:43:53.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43:53.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 2022-01-31T19:43:52.421661+0000 mon.smithi171 (mon 2022-01-31T19:43:53.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .1) 24 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.sgnfie"}]: dispatch 2022-01-31T19:43:53.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.421906+0000 mon.smithi171 (mon. 2022-01-31T19:43:53.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 1) 25 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.rdkazj"}]: dispatch 2022-01-31T19:43:53.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:43:53.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 19:43:52.422125+0000 mon.smithi171 (mon.1 2022-01-31T19:43:53.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: ) 26 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.dyqqtj"}]: dispatch 2022-01-31T19:43:53.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:43:53.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :43:52.422305+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 27 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.erzfea"}]: dispatch 2022-01-31T19:43:53.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:43:53.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 31T19:43:52.422648+0000 mon.smithi171 (mon 2022-01-31T19:43:53.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .1) 28 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr metadata", "who": "smithi171.asyxnc", "id": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:43:53.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.422845+0000 mon.smithi171 (mon 2022-01-31T19:43:53.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .1) 29 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:43:53.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.423077+0000 mon.smithi171 (mon 2022-01-31T19:43:53.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .1) 30 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:43:53.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.423341+0000 mon.smithi171 (mon. 2022-01-31T19:43:53.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 1) 31 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:43:53.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:43:53.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :43:52.423593+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 32 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:43:53.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:43:53.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 43:52.423894+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 33 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:43:53.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:43:53.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 43:52.424117+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 34 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:43:53.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:43:53.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 43:52.424341+0000 mon.smithi171 (mon.1) 35 2022-01-31T19:43:53.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:43:53.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43 2022-01-31T19:43:53.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :52.425209+0000 mon.smithi171 (mon.1) 36 2022-01-31T19:43:53.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:43:53.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19:43 2022-01-31T19:43:53.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :52.425597+0000 mon.smithi171 (mon.1) 37 2022-01-31T19:43:53.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:43:53.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:43:53.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 43:52.426089+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 38 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:43:53.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cluster 2022-01-31T19 2022-01-31T19:43:53.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :43:52.469305+0000 mon.smithi167 (mon.0) 2022-01-31T19:43:53.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 636 : cluster [INF] Manager daemon smithi171.asyxnc is now available 2022-01-31T19:43:53.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:43:53.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 19:43:52.486661+0000 mon.smithi171 (mon.1) 2022-01-31T19:43:53.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 39 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:53.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:43:53.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: :43:52.489146+0000 mon.smithi171 (mon.1 2022-01-31T19:43:53.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: ) 40 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:43:53.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:43:53.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 19:43:52.490868+0000 mon.smithi171 (mon. 2022-01-31T19:43:53.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 1) 41 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:43:53.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:43:53.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 19:43:52.491915+0000 mon.smithi171 (mon. 2022-01-31T19:43:53.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 1) 42 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:43:53.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:43:53.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 19:43:52.497501+0000 mon.smithi171 (mon. 2022-01-31T19:43:53.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 1) 43 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2022-01-31T19:43:53.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.497770+0000 mon.smithi167 (mon 2022-01-31T19:43:53.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .0) 637 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]: dispatch 2022-01-31T19:43:53.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:43:53.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 31T19:43:52.512949+0000 mon.smithi171 ( 2022-01-31T19:43:53.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: mon.1) 44 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:43:53.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01- 2022-01-31T19:43:53.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: 31T19:43:52.513342+0000 mon.smithi167 (mon 2022-01-31T19:43:53.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .0) 638 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:43:53.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.516179+0000 mon.smithi171 (mon 2022-01-31T19:43:53.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .1) 45 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/trash_purge_schedule"}]: dispatch 2022-01-31T19:43:53.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.516491+0000 mon.smithi167 (mon 2022-01-31T19:43:53.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .0) 639 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi171.asyxnc/trash_purge_schedule"}]: dispatch 2022-01-31T19:43:53.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:43:53.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: T19:43:52.534693+0000 mon.smithi167 (mon 2022-01-31T19:43:53.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: .0) 640 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:53.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cephadm 2022-01 2022-01-31T19:43:53.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: -31T19:43:52.824309+0000 mgr.smithi171.asyxnc ( 2022-01-31T19:43:53.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: mgr.24387) 1 : cephadm [INF] Deploying cephadm binary to smithi167 2022-01-31T19:43:53.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: cephadm 2022-01 2022-01-31T19:43:53.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: -31T19:43:52.871041+0000 mgr.smithi171.asyxnc 2022-01-31T19:43:53.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:52 smithi167 conmon[32206]: (mgr.24387) 2 : cephadm [INF] Deploying cephadm binary to smithi171 2022-01-31T19:43:54.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: audit 2022-01-31T19:43:53 2022-01-31T19:43:54.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: .416992+0000 mon.smithi167 (mon.0) 641 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]': finished 2022-01-31T19:43:54.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: cluster 2022-01-31T19:43:53.417070+0000 mon.smithi167 (mon.0) 642 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2022-01-31T19:43:54.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: audit 2022-01-31T19:43:53.420492+0000 mon.smithi171 (mon.1) 46 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-01-31T19:43:54.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: cluster 2022-01-31T19:43:53.425713+0000 mon.smithi167 (mon.0) 643 : cluster [DBG] mgrmap e16: smithi171.asyxnc(active, since 1.30687s), standbys: smithi167.aciqpk 2022-01-31T19:43:54.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: audit 2022-01-31T19:43:53.428023+0000 mon.smithi167 (mon.0) 644 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-01-31T19:43:54.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:54 smithi167 conmon[32206]: cluster 2022-01-31T19:43:53.446460+0000 mgr.smithi171.asyxnc (mgr.24387) 3 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 268 MiB data, 1019 MiB used, 535 GiB / 536 GiB avail 2022-01-31T19:43:54.837 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.416992+0000 2022-01-31T19:43:54.837 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: mon.smithi167 (mon.0) 641 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr"}]': finished 2022-01-31T19:43:54.837 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: cluster 2022-01-31T19:43:53.417070+0000 mon.smithi167 (mon.0) 642 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2022-01-31T19:43:54.837 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.420492+0000 mon.smithi171 (mon.1) 46 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-01-31T19:43:54.838 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: cluster 2022-01-31T19:43:53.425713+0000 mon.smithi167 (mon.0) 643 : cluster [DBG] mgrmap e16: smithi171.asyxnc(active, since 1.30687s), standbys: smithi167.aciqpk 2022-01-31T19:43:54.838 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: audit 2022-01-31T19:43:53.428023+0000 mon.smithi167 (mon.0) 644 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-01-31T19:43:54.838 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:54 smithi171 conmon[35325]: cluster 2022-01-31T19:43:53.446460+0000 mgr.smithi171.asyxnc (mgr.24387) 3 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 268 MiB data, 1019 MiB used, 535 GiB / 536 GiB avail 2022-01-31T19:43:55.769 INFO:tasks.workunit.client.1.smithi171.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2022-01-31T19:43:55.776 INFO:tasks.workunit.client.1.smithi171.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2022-01-31T19:43:55.777 INFO:tasks.workunit.client.1.smithi171.stderr:+ make 2022-01-31T19:43:55.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: cluster 2022-01-31T19:43:54.425320+0000 mgr.smithi171.asyxnc (mgr.24387) 4 2022-01-31T19:43:55.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 268 MiB data, 1019 MiB used, 535 GiB / 536 GiB avail 2022-01-31T19:43:55.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: audit 2022-01-31T19:43:54.433475+0000 mon.smithi167 (mon.0) 645 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-01-31T19:43:55.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: cluster 2022-01-31T19:43:54.433568+0000 mon.smithi167 (mon.0) 646 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2022-01-31T19:43:55.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: cluster 2022-01-31T19:43:54.433924+0000 mon.smithi167 ( 2022-01-31T19:43:55.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: mon.0) 647 : cluster [DBG] mgrmap e17: smithi171.asyxnc(active, since 2s), standbys: smithi167.aciqpk 2022-01-31T19:43:55.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:54.836021+0000 mgr.smithi171.asyxnc (mgr.24387) 5 : cephadm [INF] [31/Jan/2022:19:43:54] ENGINE Bus STARTING 2022-01-31T19:43:55.860 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:43:55.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: -31T19:43:54.955603+0000 mon.smithi167 (mon.0) 648 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:55.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: audit 2022-01-31T 2022-01-31T19:43:55.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:55 smithi171 conmon[35325]: 19:43:55.122431+0000 mon.smithi167 (mon.0) 649 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:55.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: cluster 2022-01-31T19:43:54.425320+0000 mgr.smithi171.asyxnc (mgr.24387 2022-01-31T19:43:55.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: ) 4 : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 268 MiB data, 1019 MiB used, 535 GiB / 536 GiB avail 2022-01-31T19:43:55.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: audit 2022-01-31T19:43:54.433475+0000 mon.smithi167 (mon.0) 645 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-01-31T19:43:55.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: cluster 2022-01-31T19:43:54.433568+0000 mon.smithi167 (mon.0) 646 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2022-01-31T19:43:55.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: cluster 2022-01-31T19:43:54.433924+0000 mon.smithi167 (mon.0) 647 : cluster [DBG] mgrmap e17: smithi171.asyxnc(active, since 2s), standbys: smithi167.aciqpk 2022-01-31T19:43:55.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: cephadm 2022-01-31T19:43:54.836021+0000 mgr.smithi171.asyxnc (mgr.24387) 5 : cephadm [INF] [31/Jan/2022:19:43:54] ENGINE Bus STARTING 2022-01-31T19:43:55.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: audit 2022-01-31T19:43:54.955603+0000 mon.smithi167 (mon.0) 648 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:55.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:55 smithi167 conmon[32206]: audit 2022-01-31T19:43:55.122431+0000 mon.smithi167 (mon.0) 649 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:55.937 INFO:tasks.workunit.client.1.smithi171.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2022-01-31T19:43:56.174 INFO:tasks.workunit.client.1.smithi171.stderr:++ readlink -f fsstress 2022-01-31T19:43:56.175 INFO:tasks.workunit.client.1.smithi171.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2022-01-31T19:43:56.176 INFO:tasks.workunit.client.1.smithi171.stderr:+ popd 2022-01-31T19:43:56.177 INFO:tasks.workunit.client.1.smithi171.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2022-01-31T19:43:56.177 INFO:tasks.workunit.client.1.smithi171.stderr:+ popd 2022-01-31T19:43:56.178 INFO:tasks.workunit.client.1.smithi171.stdout:~/cephtest/mnt.1/client.1/tmp 2022-01-31T19:43:56.179 INFO:tasks.workunit.client.1.smithi171.stderr:++ mktemp -d -p . 2022-01-31T19:43:56.180 INFO:tasks.workunit.client.1.smithi171.stderr:+ T=./tmp.TTnjqkluUa 2022-01-31T19:43:56.180 INFO:tasks.workunit.client.1.smithi171.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.TTnjqkluUa -l 1 -n 1000 -p 10 -v 2022-01-31T19:43:56.180 INFO:tasks.workunit.client.1.smithi171.stdout:seed = 1643808645 2022-01-31T19:43:56.182 INFO:tasks.workunit.client.1.smithi171.stdout:0/0: write - no filename 2022-01-31T19:43:56.183 INFO:tasks.workunit.client.1.smithi171.stdout:1/0: fdatasync - no filename 2022-01-31T19:43:56.183 INFO:tasks.workunit.client.1.smithi171.stdout:1/1: rename - no filename 2022-01-31T19:43:56.183 INFO:tasks.workunit.client.1.smithi171.stdout:1/2: getdents . 0 2022-01-31T19:43:56.184 INFO:tasks.workunit.client.1.smithi171.stdout:1/3: truncate - no filename 2022-01-31T19:43:56.184 INFO:tasks.workunit.client.1.smithi171.stdout:1/4: dread - no filename 2022-01-31T19:43:56.185 INFO:tasks.workunit.client.1.smithi171.stdout:1/5: dread - no filename 2022-01-31T19:43:56.185 INFO:tasks.workunit.client.1.smithi171.stdout:1/6: truncate - no filename 2022-01-31T19:43:56.185 INFO:tasks.workunit.client.1.smithi171.stdout:1/7: dwrite - no filename 2022-01-31T19:43:56.185 INFO:tasks.workunit.client.1.smithi171.stdout:1/8: stat - no entries 2022-01-31T19:43:56.186 INFO:tasks.workunit.client.1.smithi171.stdout:0/1: creat f0 x:0 0 0 2022-01-31T19:43:56.186 INFO:tasks.workunit.client.1.smithi171.stdout:0/2: write f0 [405277,16563] 0 2022-01-31T19:43:56.187 INFO:tasks.workunit.client.1.smithi171.stdout:0/3: chown f0 2039200 1 2022-01-31T19:43:56.187 INFO:tasks.workunit.client.1.smithi171.stdout:0/4: rmdir - no directory 2022-01-31T19:43:56.187 INFO:tasks.workunit.client.1.smithi171.stdout:0/5: rmdir - no directory 2022-01-31T19:43:56.187 INFO:tasks.workunit.client.1.smithi171.stdout:0/6: rename f0 to f1 0 2022-01-31T19:43:56.188 INFO:tasks.workunit.client.1.smithi171.stdout:3/0: link - no file 2022-01-31T19:43:56.188 INFO:tasks.workunit.client.1.smithi171.stdout:2/0: creat f0 x:0 0 0 2022-01-31T19:43:56.189 INFO:tasks.workunit.client.1.smithi171.stdout:2/1: chown f0 102908779 1 2022-01-31T19:43:56.189 INFO:tasks.workunit.client.1.smithi171.stdout:5/0: rename - no filename 2022-01-31T19:43:56.190 INFO:tasks.workunit.client.1.smithi171.stdout:3/1: creat f0 x:0 0 0 2022-01-31T19:43:56.192 INFO:tasks.workunit.client.1.smithi171.stdout:2/2: rename f0 to f1 0 2022-01-31T19:43:56.192 INFO:tasks.workunit.client.1.smithi171.stdout:2/3: fdatasync f1 0 2022-01-31T19:43:56.192 INFO:tasks.workunit.client.1.smithi171.stdout:2/4: creat f2 x:0 0 0 2022-01-31T19:43:56.193 INFO:tasks.workunit.client.1.smithi171.stdout:2/5: truncate f2 839838 0 2022-01-31T19:43:56.194 INFO:tasks.workunit.client.1.smithi171.stdout:5/1: mknod c0 0 2022-01-31T19:43:56.194 INFO:tasks.workunit.client.1.smithi171.stdout:5/2: dread - no filename 2022-01-31T19:43:56.194 INFO:tasks.workunit.client.1.smithi171.stdout:5/3: chown c0 361 1 2022-01-31T19:43:56.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/4: read - no filename 2022-01-31T19:43:56.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/5: write - no filename 2022-01-31T19:43:56.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/6: chown c0 7 1 2022-01-31T19:43:56.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/7: write - no filename 2022-01-31T19:43:56.196 INFO:tasks.workunit.client.1.smithi171.stdout:3/2: mkdir d1 0 2022-01-31T19:43:56.196 INFO:tasks.workunit.client.1.smithi171.stdout:3/3: readlink - no filename 2022-01-31T19:43:56.196 INFO:tasks.workunit.client.1.smithi171.stdout:3/4: truncate f0 486117 0 2022-01-31T19:43:56.197 INFO:tasks.workunit.client.1.smithi171.stdout:3/5: write f0 [1018068,45462] 0 2022-01-31T19:43:56.198 INFO:tasks.workunit.client.1.smithi171.stdout:4/0: creat f0 x:0 0 0 2022-01-31T19:43:56.198 INFO:tasks.workunit.client.1.smithi171.stdout:4/1: truncate f0 1018052 0 2022-01-31T19:43:56.199 INFO:tasks.workunit.client.1.smithi171.stdout:5/8: creat f1 x:0 0 0 2022-01-31T19:43:56.201 INFO:tasks.workunit.client.1.smithi171.stdout:7/0: readlink - no filename 2022-01-31T19:43:56.209 INFO:tasks.workunit.client.1.smithi171.stdout:4/2: dread f0 [0,4194304] 0 2022-01-31T19:43:56.209 INFO:tasks.workunit.client.1.smithi171.stdout:7/1: symlink l0 0 2022-01-31T19:43:56.221 INFO:tasks.workunit.client.1.smithi171.stdout:4/3: dread f0 [0,4194304] 0 2022-01-31T19:43:56.221 INFO:tasks.workunit.client.1.smithi171.stdout:7/2: mkdir d1 0 2022-01-31T19:43:56.224 INFO:tasks.workunit.client.1.smithi171.stdout:8/0: creat f0 x:0 0 0 2022-01-31T19:43:56.224 INFO:tasks.workunit.client.1.smithi171.stdout:8/1: dread - f0 zero size 2022-01-31T19:43:56.224 INFO:tasks.workunit.client.1.smithi171.stdout:8/2: creat f1 x:0 0 0 2022-01-31T19:43:56.224 INFO:tasks.workunit.client.1.smithi171.stdout:7/3: mkdir d1/d2 0 2022-01-31T19:43:56.225 INFO:tasks.workunit.client.1.smithi171.stdout:4/4: dread f0 [0,4194304] 0 2022-01-31T19:43:56.248 INFO:tasks.workunit.client.1.smithi171.stdout:7/4: mknod d1/c3 0 2022-01-31T19:43:56.248 INFO:tasks.workunit.client.1.smithi171.stdout:7/5: fsync - no filename 2022-01-31T19:43:56.249 INFO:tasks.workunit.client.1.smithi171.stdout:7/6: write - no filename 2022-01-31T19:43:56.249 INFO:tasks.workunit.client.1.smithi171.stdout:7/7: write - no filename 2022-01-31T19:43:56.249 INFO:tasks.workunit.client.1.smithi171.stdout:7/8: dread - no filename 2022-01-31T19:43:56.250 INFO:tasks.workunit.client.1.smithi171.stdout:7/9: dwrite - no filename 2022-01-31T19:43:56.265 INFO:tasks.workunit.client.1.smithi171.stdout:8/3: link f1 f2 0 2022-01-31T19:43:56.265 INFO:tasks.workunit.client.1.smithi171.stdout:8/4: fdatasync f2 0 2022-01-31T19:43:56.265 INFO:tasks.workunit.client.1.smithi171.stdout:7/10: symlink d1/d2/l4 0 2022-01-31T19:43:56.266 INFO:tasks.workunit.client.1.smithi171.stdout:7/11: creat d1/f5 x:0 0 0 2022-01-31T19:43:56.266 INFO:tasks.workunit.client.1.smithi171.stdout:7/12: truncate d1/f5 730323 0 2022-01-31T19:43:56.267 INFO:tasks.workunit.client.1.smithi171.stdout:7/13: symlink d1/l6 0 2022-01-31T19:43:56.270 INFO:tasks.workunit.client.1.smithi171.stdout:7/14: mkdir d1/d2/d7 0 2022-01-31T19:43:56.859 INFO:tasks.workunit.client.1.smithi171.stdout:8/5: dwrite f1 [0,4194304] 0 2022-01-31T19:43:57.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:56 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:54.950542+0000 mgr.smithi171.asyxnc ( 2022-01-31T19:43:57.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:56 smithi171 conmon[35325]: mgr.24387) 6 : cephadm [INF] [31/Jan/2022:19:43:54] ENGINE Serving on https://172.21.15.171:7150 2022-01-31T19:43:57.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:56 smithi171 conmon[35325]: cephadm 2022-01-31T19:43:54.950718+0000 mgr.smithi171.asyxnc (mgr.24387) 7 : cephadm [INF] [31/Jan/2022:19:43:54] ENGINE Bus STARTED 2022-01-31T19:43:57.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:56 smithi167 conmon[32206]: cephadm 2022-01-31T19:43:54.950542+0000 mgr.smithi171.asyxnc (mgr.24387) 6 : cephadm [INF] [31/Jan/2022:19:43:54] ENGINE Serving on https://172.21.15.171:7150 2022-01-31T19:43:57.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:56 smithi167 conmon[32206]: cephadm 2022-01-31T19:43:54.950718+0000 mgr.smithi171.asyxnc (mgr.24387) 7 : cephadm [INF] [31/Jan/2022:19:43:54] ENGINE Bus STARTED 2022-01-31T19:43:57.188 INFO:tasks.workunit.client.1.smithi171.stdout:3/6: dwrite f0 [0,4194304] 0 2022-01-31T19:43:57.267 INFO:tasks.workunit.client.1.smithi171.stdout:3/7: dread f0 [0,4194304] 0 2022-01-31T19:43:57.314 INFO:tasks.workunit.client.1.smithi171.stdout:0/7: dwrite f1 [0,4194304] 0 2022-01-31T19:43:57.314 INFO:tasks.workunit.client.1.smithi171.stdout:0/8: readlink - no filename 2022-01-31T19:43:57.473 INFO:tasks.workunit.client.1.smithi171.stdout:8/6: truncate f0 235486 0 2022-01-31T19:43:57.474 INFO:tasks.workunit.client.1.smithi171.stdout:9/0: symlink l0 0 2022-01-31T19:43:57.588 INFO:tasks.workunit.client.1.smithi171.stdout:7/15: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:43:57.588 INFO:tasks.workunit.client.1.smithi171.stdout:7/16: chown d1/c3 342 1 2022-01-31T19:43:57.589 INFO:tasks.workunit.client.1.smithi171.stdout:7/17: creat d1/f8 x:0 0 0 2022-01-31T19:43:57.650 INFO:tasks.workunit.client.1.smithi171.stdout:2/6: getdents . 0 2022-01-31T19:43:57.672 INFO:tasks.workunit.client.1.smithi171.stdout:5/9: getdents . 0 2022-01-31T19:43:57.673 INFO:tasks.workunit.client.1.smithi171.stdout:5/10: rmdir - no directory 2022-01-31T19:43:57.691 INFO:tasks.workunit.client.1.smithi171.stdout:5/11: dwrite f1 [0,4194304] 0 2022-01-31T19:43:57.692 INFO:tasks.workunit.client.1.smithi171.stdout:5/12: chown f1 14 1 2022-01-31T19:43:57.692 INFO:tasks.workunit.client.1.smithi171.stdout:5/13: read f1 [2371792,26767] 0 2022-01-31T19:43:58.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:57 smithi171 conmon[35325]: cluster 2022-01-31T19:43:56.428852+0000 mgr.smithi171.asyxnc ( 2022-01-31T19:43:58.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:57 smithi171 conmon[35325]: mgr.24387) 8 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 268 MiB data, 1019 MiB used, 535 GiB / 536 GiB avail 2022-01-31T19:43:58.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:57 smithi171 conmon[35325]: cluster 2022-01-31T19:43:56.723704+0000 mon.smithi167 (mon.0) 650 : cluster [DBG] mgrmap e18: smithi171.asyxnc(active, since 4s), standbys: smithi167.aciqpk 2022-01-31T19:43:58.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:57 smithi171 conmon[35325]: audit 2022-01-31T19:43:58.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:57 smithi171 conmon[35325]: 2022-01-31T19:43:57.424595+0000 mon.smithi167 (mon.0) 651 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: cluster 2022-01-31T19:43:56.428852+0000 mgr.smithi171.asyxnc ( 2022-01-31T19:43:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: mgr.24387) 8 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 268 MiB data, 1019 MiB used, 535 GiB / 536 GiB avail 2022-01-31T19:43:58.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: cluster 2022-01-31T19:43:56.723704+0000 mon.smithi167 (mon.0) 650 : cluster [DBG] 2022-01-31T19:43:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: mgrmap e18: smithi171.asyxnc(active, since 4s), standbys: smithi167.aciqpk 2022-01-31T19:43:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:43:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: -31T19:43:57.424595+0000 mon.smithi167 (mon.0) 651 : audit 2022-01-31T19:43:58.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:57 smithi167 conmon[32206]: [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:43:59.373 INFO:tasks.workunit.client.1.smithi171.stdout:0/9: mknod c2 0 2022-01-31T19:43:59.709 INFO:tasks.workunit.client.1.smithi171.stdout:9/1: rename l0 to l1 0 2022-01-31T19:43:59.709 INFO:tasks.workunit.client.1.smithi171.stdout:9/2: chown l1 5115795 1 2022-01-31T19:43:59.710 INFO:tasks.workunit.client.1.smithi171.stdout:9/3: write - no filename 2022-01-31T19:44:00.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: cluster 2022-01-31T19:43:58 2022-01-31T19:44:00.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: .311385+0000 mon.smithi167 (mon.0) 652 : cluster [DBG] Standby manager daemon smithi167.aciqpk restarted 2022-01-31T19:44:00.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: cluster 2022-01- 2022-01-31T19:44:00.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 31T19:43:58.311482+0000 mon.smithi167 (mon.0 2022-01-31T19:44:00.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: ) 653 : cluster [DBG] Standby manager daemon smithi167.aciqpk started 2022-01-31T19:44:00.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:00.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: -01-31T19:43:58 2022-01-31T19:44:00.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: .312524+0000 mon.smithi167 (mon.0) 2022-01-31T19:44:00.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 654 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/crt"}]: dispatch 2022-01-31T19:44:00.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:00.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 01-31T19:43:58. 2022-01-31T19:44:00.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 314478+0000 mon.smithi167 (mon.0) 2022-01-31T19:44:00.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 655 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:44:00.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:00.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: -01-31T19:43:58 2022-01-31T19:44:00.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: .315408+0000 mon.smithi167 (mon.0 2022-01-31T19:44:00.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: ) 656 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/key"}]: dispatch 2022-01-31T19:44:00.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:00.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: -01-31T19:43:58 2022-01-31T19:44:00.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: .315658+0000 mon.smithi167 (mon. 2022-01-31T19:44:00.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 0) 657 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:44:00.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 2022-01-31T19:44:00.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: audit 2022-01-31T19:44:00.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 2022-01-31T19:43:58. 2022-01-31T19:44:00.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: 663010+0000 mon.smithi167 (mon.0) 658 2022-01-31T19:44:00.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:43:59 smithi171 conmon[35325]: : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-01-31T19:44:00.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: cluster 2022-01-31T19:43:58.311385+0000 mon.smithi167 (mon.0) 652 : cluster [DBG] Standby manager daemon smithi167.aciqpk restarted 2022-01-31T19:44:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: cluster 2022-01-31T19:43:58.311482+0000 mon.smithi167 (mon.0) 653 : cluster [DBG] Standby manager daemon smithi167.aciqpk started 2022-01-31T19:44:00.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: audit 2022-01-31T19:43:58.312524+0000 mon.smithi167 (mon.0) 654 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/crt"}]: dispatch 2022-01-31T19:44:00.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: audit 2022-01-31T19:43:58.314478+0000 mon.smithi167 (mon.0) 655 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:44:00.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: audit 2022-01-31T19:43:58.315408+0000 mon.smithi167 (mon.0) 656 : audit 2022-01-31T19:44:00.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/key"}]: dispatch 2022-01-31T19:44:00.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: audit 2022-01-31T19:43:58.315658+0000 mon.smithi167 (mon.0) 657 : audit [DBG] from='mgr.? 172.21.15.167:0/2704288234' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:44:00.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: audit 2022-01-31T19:43 2022-01-31T19:44:00.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:43:59 smithi167 conmon[32206]: :58.663010+0000 mon.smithi167 (mon.0) 658 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-01-31T19:44:00.237 INFO:tasks.workunit.client.1.smithi171.stdout:2/7: rename f1 to f3 0 2022-01-31T19:44:00.239 INFO:tasks.workunit.client.1.smithi171.stdout:2/8: write f2 [666095,52177] 0 2022-01-31T19:44:00.261 INFO:tasks.workunit.client.1.smithi171.stdout:5/14: mknod c2 0 2022-01-31T19:44:00.261 INFO:tasks.workunit.client.1.smithi171.stdout:5/15: chown c2 0 1 2022-01-31T19:44:00.506 INFO:tasks.workunit.client.1.smithi171.stdout:8/7: truncate f2 3785418 0 2022-01-31T19:44:00.507 INFO:tasks.workunit.client.1.smithi171.stdout:8/8: creat f3 x:0 0 0 2022-01-31T19:44:00.507 INFO:tasks.workunit.client.1.smithi171.stdout:3/8: truncate f0 52755 0 2022-01-31T19:44:00.511 INFO:tasks.workunit.client.1.smithi171.stdout:7/18: unlink d1/f8 0 2022-01-31T19:44:00.514 INFO:tasks.workunit.client.1.smithi171.stdout:2/9: rename f2 to f4 0 2022-01-31T19:44:00.514 INFO:tasks.workunit.client.1.smithi171.stdout:2/10: chown f4 34 1 2022-01-31T19:44:00.516 INFO:tasks.workunit.client.1.smithi171.stdout:5/16: symlink l3 0 2022-01-31T19:44:00.516 INFO:tasks.workunit.client.1.smithi171.stdout:5/17: fsync f1 0 2022-01-31T19:44:00.517 INFO:tasks.workunit.client.1.smithi171.stdout:5/18: fsync f1 0 2022-01-31T19:44:00.518 INFO:tasks.workunit.client.1.smithi171.stdout:0/10: rename f1 to f3 0 2022-01-31T19:44:00.519 INFO:tasks.workunit.client.1.smithi171.stdout:0/11: creat f4 x:0 0 0 2022-01-31T19:44:00.520 INFO:tasks.workunit.client.1.smithi171.stdout:0/12: dread - f4 zero size 2022-01-31T19:44:00.520 INFO:tasks.workunit.client.1.smithi171.stdout:2/11: fdatasync f4 0 2022-01-31T19:44:00.521 INFO:tasks.workunit.client.1.smithi171.stdout:7/19: symlink d1/l9 0 2022-01-31T19:44:00.521 INFO:tasks.workunit.client.1.smithi171.stdout:7/20: fsync d1/f5 0 2022-01-31T19:44:00.524 INFO:tasks.workunit.client.1.smithi171.stdout:0/13: symlink l5 0 2022-01-31T19:44:00.524 INFO:tasks.workunit.client.1.smithi171.stdout:0/14: chown f3 135704 1 2022-01-31T19:44:00.524 INFO:tasks.workunit.client.1.smithi171.stdout:0/15: creat f6 x:0 0 0 2022-01-31T19:44:00.525 INFO:tasks.workunit.client.1.smithi171.stdout:0/16: stat f3 0 2022-01-31T19:44:00.525 INFO:tasks.workunit.client.1.smithi171.stdout:0/17: creat f7 x:0 0 0 2022-01-31T19:44:00.525 INFO:tasks.workunit.client.1.smithi171.stdout:0/18: chown f4 2429792 1 2022-01-31T19:44:00.525 INFO:tasks.workunit.client.1.smithi171.stdout:0/19: creat f8 x:0 0 0 2022-01-31T19:44:00.526 INFO:tasks.workunit.client.1.smithi171.stdout:0/20: dread - f4 zero size 2022-01-31T19:44:00.527 INFO:tasks.workunit.client.1.smithi171.stdout:0/21: mkdir d9 0 2022-01-31T19:44:00.529 INFO:tasks.workunit.client.1.smithi171.stdout:8/9: dread f1 [0,4194304] 0 2022-01-31T19:44:00.529 INFO:tasks.workunit.client.1.smithi171.stdout:0/22: creat d9/fa x:0 0 0 2022-01-31T19:44:00.530 INFO:tasks.workunit.client.1.smithi171.stdout:0/23: read - f6 zero size 2022-01-31T19:44:00.530 INFO:tasks.workunit.client.1.smithi171.stdout:0/24: dread - f4 zero size 2022-01-31T19:44:00.531 INFO:tasks.workunit.client.1.smithi171.stdout:0/25: mkdir d9/db 0 2022-01-31T19:44:00.531 INFO:tasks.workunit.client.1.smithi171.stdout:0/26: write f6 [824776,119391] 0 2022-01-31T19:44:00.535 INFO:tasks.workunit.client.1.smithi171.stdout:8/10: dread f2 [0,4194304] 0 2022-01-31T19:44:00.536 INFO:tasks.workunit.client.1.smithi171.stdout:0/27: write f6 [84749,91903] 0 2022-01-31T19:44:00.536 INFO:tasks.workunit.client.1.smithi171.stdout:0/28: dread - f7 zero size 2022-01-31T19:44:00.543 INFO:tasks.workunit.client.1.smithi171.stdout:8/11: dread f1 [0,4194304] 0 2022-01-31T19:44:00.544 INFO:tasks.workunit.client.1.smithi171.stdout:3/9: dwrite f0 [0,4194304] 0 2022-01-31T19:44:00.544 INFO:tasks.workunit.client.1.smithi171.stdout:5/19: dwrite f1 [4194304,4194304] 0 2022-01-31T19:44:00.544 INFO:tasks.workunit.client.1.smithi171.stdout:5/20: fsync f1 0 2022-01-31T19:44:00.545 INFO:tasks.workunit.client.1.smithi171.stdout:7/21: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:44:00.545 INFO:tasks.workunit.client.1.smithi171.stdout:7/22: fsync d1/f5 0 2022-01-31T19:44:00.546 INFO:tasks.workunit.client.1.smithi171.stdout:8/12: symlink l4 0 2022-01-31T19:44:00.546 INFO:tasks.workunit.client.1.smithi171.stdout:8/13: rmdir - no directory 2022-01-31T19:44:00.547 INFO:tasks.workunit.client.1.smithi171.stdout:3/10: write f0 [1867679,67560] 0 2022-01-31T19:44:00.547 INFO:tasks.workunit.client.1.smithi171.stdout:2/12: getdents . 0 2022-01-31T19:44:00.551 INFO:tasks.workunit.client.1.smithi171.stdout:5/21: rename c0 to c4 0 2022-01-31T19:44:00.551 INFO:tasks.workunit.client.1.smithi171.stdout:0/29: fsync f7 0 2022-01-31T19:44:00.556 INFO:tasks.workunit.client.1.smithi171.stdout:3/11: rmdir d1 0 2022-01-31T19:44:00.560 INFO:tasks.workunit.client.1.smithi171.stdout:0/30: mkdir d9/db/dc 0 2022-01-31T19:44:00.564 INFO:tasks.workunit.client.1.smithi171.stdout:2/13: dwrite f4 [0,4194304] 0 2022-01-31T19:44:00.564 INFO:tasks.workunit.client.1.smithi171.stdout:2/14: chown f4 12 1 2022-01-31T19:44:00.565 INFO:tasks.workunit.client.1.smithi171.stdout:0/31: dread f6 [0,4194304] 0 2022-01-31T19:44:00.565 INFO:tasks.workunit.client.1.smithi171.stdout:0/32: write f7 [101771,28868] 0 2022-01-31T19:44:00.566 INFO:tasks.workunit.client.1.smithi171.stdout:0/33: getdents d9/db/dc 0 2022-01-31T19:44:00.567 INFO:tasks.workunit.client.1.smithi171.stdout:2/15: symlink l5 0 2022-01-31T19:44:00.567 INFO:tasks.workunit.client.1.smithi171.stdout:2/16: creat f6 x:0 0 0 2022-01-31T19:44:00.568 INFO:tasks.workunit.client.1.smithi171.stdout:8/14: dwrite f1 [0,4194304] 0 2022-01-31T19:44:00.568 INFO:tasks.workunit.client.1.smithi171.stdout:8/15: dread - f3 zero size 2022-01-31T19:44:00.569 INFO:tasks.workunit.client.1.smithi171.stdout:8/16: link f1 f5 0 2022-01-31T19:44:00.570 INFO:tasks.workunit.client.1.smithi171.stdout:5/22: dwrite f1 [8388608,4194304] 0 2022-01-31T19:44:00.570 INFO:tasks.workunit.client.1.smithi171.stdout:5/23: fsync f1 0 2022-01-31T19:44:00.571 INFO:tasks.workunit.client.1.smithi171.stdout:5/24: stat c2 0 2022-01-31T19:44:00.575 INFO:tasks.workunit.client.1.smithi171.stdout:2/17: getdents . 0 2022-01-31T19:44:00.575 INFO:tasks.workunit.client.1.smithi171.stdout:2/18: fsync f4 0 2022-01-31T19:44:00.576 INFO:tasks.workunit.client.1.smithi171.stdout:5/25: mkdir d5 0 2022-01-31T19:44:00.578 INFO:tasks.workunit.client.1.smithi171.stdout:3/12: dread f0 [0,4194304] 0 2022-01-31T19:44:00.580 INFO:tasks.workunit.client.1.smithi171.stdout:0/34: dwrite f3 [0,4194304] 0 2022-01-31T19:44:00.581 INFO:tasks.workunit.client.1.smithi171.stdout:2/19: mkdir d7 0 2022-01-31T19:44:00.581 INFO:tasks.workunit.client.1.smithi171.stdout:5/26: write f1 [5251050,115652] 0 2022-01-31T19:44:00.582 INFO:tasks.workunit.client.1.smithi171.stdout:8/17: dread f1 [0,4194304] 0 2022-01-31T19:44:00.583 INFO:tasks.workunit.client.1.smithi171.stdout:0/35: mknod d9/db/cd 0 2022-01-31T19:44:00.583 INFO:tasks.workunit.client.1.smithi171.stdout:0/36: readlink l5 0 2022-01-31T19:44:00.586 INFO:tasks.workunit.client.1.smithi171.stdout:3/13: dread f0 [0,4194304] 0 2022-01-31T19:44:00.586 INFO:tasks.workunit.client.1.smithi171.stdout:3/14: fsync f0 0 2022-01-31T19:44:00.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/27: mkdir d5/d6 0 2022-01-31T19:44:00.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/28: fdatasync f1 0 2022-01-31T19:44:00.591 INFO:tasks.workunit.client.1.smithi171.stdout:5/29: symlink d5/d6/l7 0 2022-01-31T19:44:00.592 INFO:tasks.workunit.client.1.smithi171.stdout:5/30: stat l3 0 2022-01-31T19:44:00.611 INFO:tasks.workunit.client.1.smithi171.stdout:5/31: truncate f1 5564443 0 2022-01-31T19:44:00.614 INFO:tasks.workunit.client.1.smithi171.stdout:0/37: dwrite f3 [0,4194304] 0 2022-01-31T19:44:00.614 INFO:tasks.workunit.client.1.smithi171.stdout:0/38: readlink l5 0 2022-01-31T19:44:00.615 INFO:tasks.workunit.client.1.smithi171.stdout:0/39: dread f6 [0,4194304] 0 2022-01-31T19:44:00.616 INFO:tasks.workunit.client.1.smithi171.stdout:0/40: symlink d9/db/le 0 2022-01-31T19:44:00.617 INFO:tasks.workunit.client.1.smithi171.stdout:0/41: rename d9/db/le to d9/db/lf 0 2022-01-31T19:44:00.618 INFO:tasks.workunit.client.1.smithi171.stdout:0/42: fdatasync f6 0 2022-01-31T19:44:00.618 INFO:tasks.workunit.client.1.smithi171.stdout:2/20: dwrite f4 [0,4194304] 0 2022-01-31T19:44:00.619 INFO:tasks.workunit.client.1.smithi171.stdout:0/43: mknod d9/c10 0 2022-01-31T19:44:00.620 INFO:tasks.workunit.client.1.smithi171.stdout:2/21: write f3 [361690,91714] 0 2022-01-31T19:44:00.620 INFO:tasks.workunit.client.1.smithi171.stdout:0/44: rmdir d9/db 39 2022-01-31T19:44:00.620 INFO:tasks.workunit.client.1.smithi171.stdout:3/15: dwrite f0 [0,4194304] 0 2022-01-31T19:44:00.621 INFO:tasks.workunit.client.1.smithi171.stdout:2/22: mkdir d7/d8 0 2022-01-31T19:44:00.623 INFO:tasks.workunit.client.1.smithi171.stdout:3/16: mknod c2 0 2022-01-31T19:44:00.624 INFO:tasks.workunit.client.1.smithi171.stdout:0/45: symlink d9/db/dc/l11 0 2022-01-31T19:44:00.624 INFO:tasks.workunit.client.1.smithi171.stdout:0/46: chown d9/db 1383 1 2022-01-31T19:44:00.625 INFO:tasks.workunit.client.1.smithi171.stdout:2/23: chown f6 608604551 1 2022-01-31T19:44:00.625 INFO:tasks.workunit.client.1.smithi171.stdout:3/17: link f0 f3 0 2022-01-31T19:44:00.628 INFO:tasks.workunit.client.1.smithi171.stdout:2/24: rename d7/d8 to d7/d9 0 2022-01-31T19:44:00.629 INFO:tasks.workunit.client.1.smithi171.stdout:3/18: unlink f3 0 2022-01-31T19:44:00.634 INFO:tasks.workunit.client.1.smithi171.stdout:3/19: write f0 [2971526,20996] 0 2022-01-31T19:44:00.635 INFO:tasks.workunit.client.1.smithi171.stdout:3/20: mkdir d4 0 2022-01-31T19:44:00.636 INFO:tasks.workunit.client.1.smithi171.stdout:3/21: creat d4/f5 x:0 0 0 2022-01-31T19:44:00.639 INFO:tasks.workunit.client.1.smithi171.stdout:0/47: getdents d9 0 2022-01-31T19:44:00.640 INFO:tasks.workunit.client.1.smithi171.stdout:0/48: link f4 d9/db/f12 0 2022-01-31T19:44:00.641 INFO:tasks.workunit.client.1.smithi171.stdout:0/49: creat d9/f13 x:0 0 0 2022-01-31T19:44:00.644 INFO:tasks.workunit.client.1.smithi171.stdout:0/50: dread f6 [0,4194304] 0 2022-01-31T19:44:00.645 INFO:tasks.workunit.client.1.smithi171.stdout:0/51: creat d9/db/f14 x:0 0 0 2022-01-31T19:44:00.645 INFO:tasks.workunit.client.1.smithi171.stdout:0/52: readlink l5 0 2022-01-31T19:44:00.646 INFO:tasks.workunit.client.1.smithi171.stdout:5/32: dwrite f1 [0,4194304] 0 2022-01-31T19:44:00.647 INFO:tasks.workunit.client.1.smithi171.stdout:2/25: dwrite f6 [0,4194304] 0 2022-01-31T19:44:00.647 INFO:tasks.workunit.client.1.smithi171.stdout:2/26: chown f3 225 1 2022-01-31T19:44:00.648 INFO:tasks.workunit.client.1.smithi171.stdout:0/53: symlink d9/db/l15 0 2022-01-31T19:44:00.648 INFO:tasks.workunit.client.1.smithi171.stdout:0/54: dread - d9/db/f14 zero size 2022-01-31T19:44:00.649 INFO:tasks.workunit.client.1.smithi171.stdout:0/55: write d9/db/f14 [54715,12697] 0 2022-01-31T19:44:00.649 INFO:tasks.workunit.client.1.smithi171.stdout:0/56: creat d9/f16 x:0 0 0 2022-01-31T19:44:00.649 INFO:tasks.workunit.client.1.smithi171.stdout:0/57: creat d9/f17 x:0 0 0 2022-01-31T19:44:00.649 INFO:tasks.workunit.client.1.smithi171.stdout:0/58: dread - d9/f17 zero size 2022-01-31T19:44:00.651 INFO:tasks.workunit.client.1.smithi171.stdout:5/33: creat d5/f8 x:0 0 0 2022-01-31T19:44:00.651 INFO:tasks.workunit.client.1.smithi171.stdout:5/34: dread - d5/f8 zero size 2022-01-31T19:44:00.652 INFO:tasks.workunit.client.1.smithi171.stdout:3/22: dread f0 [0,4194304] 0 2022-01-31T19:44:00.655 INFO:tasks.workunit.client.1.smithi171.stdout:3/23: write f0 [1660194,28843] 0 2022-01-31T19:44:00.655 INFO:tasks.workunit.client.1.smithi171.stdout:3/24: truncate f0 4743041 0 2022-01-31T19:44:00.657 INFO:tasks.workunit.client.1.smithi171.stdout:3/25: mknod d4/c6 0 2022-01-31T19:44:00.658 INFO:tasks.workunit.client.1.smithi171.stdout:3/26: read - d4/f5 zero size 2022-01-31T19:44:00.658 INFO:tasks.workunit.client.1.smithi171.stdout:3/27: symlink d4/l7 0 2022-01-31T19:44:00.663 INFO:tasks.workunit.client.1.smithi171.stdout:3/28: fsync f0 0 2022-01-31T19:44:00.663 INFO:tasks.workunit.client.1.smithi171.stdout:2/27: dwrite f3 [0,4194304] 0 2022-01-31T19:44:00.665 INFO:tasks.workunit.client.1.smithi171.stdout:3/29: mknod d4/c8 0 2022-01-31T19:44:00.666 INFO:tasks.workunit.client.1.smithi171.stdout:3/30: creat d4/f9 x:0 0 0 2022-01-31T19:44:00.666 INFO:tasks.workunit.client.1.smithi171.stdout:3/31: creat d4/fa x:0 0 0 2022-01-31T19:44:00.666 INFO:tasks.workunit.client.1.smithi171.stdout:3/32: readlink d4/l7 0 2022-01-31T19:44:00.667 INFO:tasks.workunit.client.1.smithi171.stdout:2/28: creat d7/fa x:0 0 0 2022-01-31T19:44:00.667 INFO:tasks.workunit.client.1.smithi171.stdout:2/29: write f3 [1914884,73431] 0 2022-01-31T19:44:00.668 INFO:tasks.workunit.client.1.smithi171.stdout:2/30: stat f4 0 2022-01-31T19:44:00.674 INFO:tasks.workunit.client.1.smithi171.stdout:3/33: link d4/f5 d4/fb 0 2022-01-31T19:44:00.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/34: unlink d4/l7 0 2022-01-31T19:44:00.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/35: dread - d4/f5 zero size 2022-01-31T19:44:00.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/36: chown f0 2250 1 2022-01-31T19:44:00.679 INFO:tasks.workunit.client.1.smithi171.stdout:3/37: rename f0 to d4/fc 0 2022-01-31T19:44:00.679 INFO:tasks.workunit.client.1.smithi171.stdout:3/38: write d4/fa [711139,25428] 0 2022-01-31T19:44:00.680 INFO:tasks.workunit.client.1.smithi171.stdout:3/39: read d4/fa [454395,6839] 0 2022-01-31T19:44:00.681 INFO:tasks.workunit.client.1.smithi171.stdout:3/40: write d4/fb [689439,127728] 0 2022-01-31T19:44:00.681 INFO:tasks.workunit.client.1.smithi171.stdout:3/41: mkdir d4/dd 0 2022-01-31T19:44:00.681 INFO:tasks.workunit.client.1.smithi171.stdout:3/42: truncate d4/fa 1688649 0 2022-01-31T19:44:00.681 INFO:tasks.workunit.client.1.smithi171.stdout:3/43: chown d4/fa 457513469 1 2022-01-31T19:44:00.682 INFO:tasks.workunit.client.1.smithi171.stdout:3/44: rename d4/f5 to d4/dd/fe 0 2022-01-31T19:44:00.682 INFO:tasks.workunit.client.1.smithi171.stdout:3/45: chown d4/c8 8026 1 2022-01-31T19:44:00.683 INFO:tasks.workunit.client.1.smithi171.stdout:3/46: mknod d4/dd/cf 0 2022-01-31T19:44:00.683 INFO:tasks.workunit.client.1.smithi171.stdout:3/47: chown d4/fb 12862 1 2022-01-31T19:44:00.684 INFO:tasks.workunit.client.1.smithi171.stdout:2/31: dwrite f3 [0,4194304] 0 2022-01-31T19:44:00.685 INFO:tasks.workunit.client.1.smithi171.stdout:2/32: write d7/fa [906144,95108] 0 2022-01-31T19:44:00.686 INFO:tasks.workunit.client.1.smithi171.stdout:2/33: link f6 d7/d9/fb 0 2022-01-31T19:44:00.688 INFO:tasks.workunit.client.1.smithi171.stdout:3/48: dread d4/fa [0,4194304] 0 2022-01-31T19:44:00.695 INFO:tasks.workunit.client.1.smithi171.stdout:2/34: dread f4 [0,4194304] 0 2022-01-31T19:44:00.695 INFO:tasks.workunit.client.1.smithi171.stdout:2/35: chown f6 602140392 1 2022-01-31T19:44:00.705 INFO:tasks.workunit.client.1.smithi171.stdout:2/36: truncate f3 1302224 0 2022-01-31T19:44:00.706 INFO:tasks.workunit.client.1.smithi171.stdout:2/37: symlink d7/lc 0 2022-01-31T19:44:00.711 INFO:tasks.workunit.client.1.smithi171.stdout:3/49: dwrite d4/dd/fe [0,4194304] 0 2022-01-31T19:44:00.711 INFO:tasks.workunit.client.1.smithi171.stdout:3/50: write d4/f9 [615869,125735] 0 2022-01-31T19:44:00.711 INFO:tasks.workunit.client.1.smithi171.stdout:3/51: fdatasync d4/dd/fe 0 2022-01-31T19:44:00.712 INFO:tasks.workunit.client.1.smithi171.stdout:3/52: dread d4/fa [0,4194304] 0 2022-01-31T19:44:00.714 INFO:tasks.workunit.client.1.smithi171.stdout:2/38: dread d7/fa [0,4194304] 0 2022-01-31T19:44:00.714 INFO:tasks.workunit.client.1.smithi171.stdout:3/53: mknod d4/dd/c10 0 2022-01-31T19:44:00.715 INFO:tasks.workunit.client.1.smithi171.stdout:2/39: mkdir d7/dd 0 2022-01-31T19:44:00.716 INFO:tasks.workunit.client.1.smithi171.stdout:3/54: truncate d4/fc 4457263 0 2022-01-31T19:44:00.716 INFO:tasks.workunit.client.1.smithi171.stdout:3/55: chown d4/c6 6397 1 2022-01-31T19:44:00.717 INFO:tasks.workunit.client.1.smithi171.stdout:3/56: symlink d4/dd/l11 0 2022-01-31T19:44:00.729 INFO:tasks.workunit.client.1.smithi171.stdout:2/40: dwrite f3 [0,4194304] 0 2022-01-31T19:44:00.731 INFO:tasks.workunit.client.1.smithi171.stdout:2/41: creat d7/d9/fe x:0 0 0 2022-01-31T19:44:00.731 INFO:tasks.workunit.client.1.smithi171.stdout:2/42: write f3 [3621760,1594] 0 2022-01-31T19:44:00.732 INFO:tasks.workunit.client.1.smithi171.stdout:2/43: write d7/fa [521407,89983] 0 2022-01-31T19:44:00.734 INFO:tasks.workunit.client.1.smithi171.stdout:2/44: creat d7/d9/ff x:0 0 0 2022-01-31T19:44:00.734 INFO:tasks.workunit.client.1.smithi171.stdout:2/45: fdatasync d7/d9/ff 0 2022-01-31T19:44:00.747 INFO:tasks.workunit.client.1.smithi171.stdout:2/46: dwrite f6 [4194304,4194304] 0 2022-01-31T19:44:00.749 INFO:tasks.workunit.client.1.smithi171.stdout:2/47: symlink d7/d9/l10 0 2022-01-31T19:44:00.769 INFO:tasks.workunit.client.1.smithi171.stdout:2/48: dwrite d7/fa [0,4194304] 0 2022-01-31T19:44:00.770 INFO:tasks.workunit.client.1.smithi171.stdout:2/49: read - d7/d9/ff zero size 2022-01-31T19:44:00.770 INFO:tasks.workunit.client.1.smithi171.stdout:2/50: creat d7/f11 x:0 0 0 2022-01-31T19:44:01.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: cluster 2022-01-31T19:43:58.429204+0000 mgr.smithi171.asyxnc (mgr.24387 2022-01-31T19:44:01.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: ) 9 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 287 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 822 KiB/s rd, 4.0 MiB/s wr, 326 op/s 2022-01-31T19:44:01.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022-01-31T19:43:58.663298+0000 mon.smithi171 (mon.1) 47 : 2022-01-31T19:44:01.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:44:01.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:01.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: -01-31T19:43:58.663520+0000 mon.smithi171 (mon.1) 48 : audit [DBG] 2022-01-31T19:44:01.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:44:01.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:44:01.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: 43:59.158460+0000 mon.smithi167 (mon.0) 659 2022-01-31T19:44:01.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-01-31T19:44:01.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:01.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: 01-31T19:43:59.162190+0000 2022-01-31T19:44:01.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: mon.smithi171 (mon.1) 49 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-01-31T19:44:01.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:44:01.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: T19:43:59.673829+0000 mon.smithi171 (mon.1) 2022-01-31T19:44:01.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: 50 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-01-31T19:44:01.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:01.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: -01-31T19:43:59.674890+0000 2022-01-31T19:44:01.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: mon.smithi171 (mon.1) 51 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:44:01.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: 2022-01-31T19:44:01.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: audit 2022-01-31T19:43:59 2022-01-31T19:44:01.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: .675102+0000 mon.smithi171 (mon.1) 52 : audit 2022-01-31T19:44:01.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:44:01.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: cluster 2022-01-31T19 2022-01-31T19:44:01.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: :43:59.682945+0000 mon.smithi167 (mon.0 2022-01-31T19:44:01.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:00 smithi171 conmon[35325]: ) 660 : cluster [DBG] mgrmap e19: smithi171.asyxnc(active, since 7s), standbys: smithi167.aciqpk 2022-01-31T19:44:01.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: cluster 2022-01-31T19:43:58.429204 2022-01-31T19:44:01.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: +0000 mgr.smithi171.asyxnc (mgr.24387) 9 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 287 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 822 KiB/s rd, 4.0 MiB/s wr, 326 op/s 2022-01-31T19:44:01.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022-01-31T19:43:58.663298+0000 mon.smithi171 (mon 2022-01-31T19:44:01.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: .1) 47 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:44:01.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022-01-31T19:44:01.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 2022-01-31T19 2022-01-31T19:44:01.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: :43:58.663520+0000 mon.smithi171 ( 2022-01-31T19:44:01.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: mon.1) 48 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:44:01.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 2022-01-31T19:44:01.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:44:01.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: -31T19:43:59.158460+0000 2022-01-31T19:44:01.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: mon.smithi167 (mon.0) 659 : audit 2022-01-31T19:44:01.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-01-31T19:44:01.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022-01-31T19:44:01.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 2022-01-31T19: 2022-01-31T19:44:01.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 43:59.162190+0000 mon.smithi171 ( 2022-01-31T19:44:01.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: mon.1) 49 : audit [INF] 2022-01-31T19:44:01.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-01-31T19:44:01.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022-01-31T19:44:01.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 2022-01-31T19:43: 2022-01-31T19:44:01.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 59.673829+0000 mon.smithi171 (mon. 2022-01-31T19:44:01.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 1) 50 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-01-31T19:44:01.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:44:01.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 01-31T19:43:59.674890+0000 2022-01-31T19:44:01.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: mon.smithi171 (mon.1) 51 : audit 2022-01-31T19:44:01.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:44:01.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: audit 2022-01-31T19:44:01.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 2022-01-31T19:43 2022-01-31T19:44:01.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: :59.675102+0000 mon.smithi171 (mon. 2022-01-31T19:44:01.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 1) 52 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:44:01.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: cluster 2022 2022-01-31T19:44:01.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: -01-31T19:43:59. 2022-01-31T19:44:01.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: 682945+0000 mon.smithi167 (mon.0) 660 2022-01-31T19:44:01.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:00 smithi167 conmon[32206]: : cluster [DBG] mgrmap e19: smithi171.asyxnc(active, since 7s), standbys: smithi167.aciqpk 2022-01-31T19:44:02.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:01 smithi171 conmon[35325]: cluster 2022-01-31T19:44:00.429693+0000 mgr.smithi171.asyxnc (mgr.24387) 10 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 330 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 9.4 MiB/s wr, 817 op/s 2022-01-31T19:44:02.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:01 smithi167 conmon[32206]: cluster 2022-01-31T19:44:00.429693+0000 mgr.smithi171.asyxnc (mgr.24387) 10 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 330 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 9.4 MiB/s wr, 817 op/s 2022-01-31T19:44:03.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:03 smithi167 conmon[32206]: cluster 2022-01-31T19:44:02.430075+0000 mgr.smithi171.asyxnc (mgr.24387) 11 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 330 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 635 op/s 2022-01-31T19:44:04.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:03 smithi171 conmon[35325]: cluster 2022-01-31T19:44:02.430075+0000 mgr.smithi171.asyxnc (mgr.24387) 11 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 330 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 635 op/s 2022-01-31T19:44:04.828 INFO:tasks.workunit.client.0.smithi167.stdout:2/26: sync 2022-01-31T19:44:04.828 INFO:tasks.workunit.client.0.smithi167.stdout:3/43: sync 2022-01-31T19:44:04.828 INFO:tasks.workunit.client.0.smithi167.stdout:9/35: sync 2022-01-31T19:44:04.829 INFO:tasks.workunit.client.0.smithi167.stdout:8/85: sync 2022-01-31T19:44:04.831 INFO:tasks.workunit.client.0.smithi167.stdout:9/36: dread - f8 zero size 2022-01-31T19:44:04.831 INFO:tasks.workunit.client.0.smithi167.stdout:3/44: truncate f1 4785124 0 2022-01-31T19:44:04.831 INFO:tasks.workunit.client.0.smithi167.stdout:8/86: dread - d6/f12 zero size 2022-01-31T19:44:04.832 INFO:tasks.workunit.client.0.smithi167.stdout:1/68: sync 2022-01-31T19:44:04.832 INFO:tasks.workunit.client.0.smithi167.stdout:5/34: sync 2022-01-31T19:44:04.832 INFO:tasks.workunit.client.0.smithi167.stdout:6/20: sync 2022-01-31T19:44:04.832 INFO:tasks.workunit.client.0.smithi167.stdout:4/17: sync 2022-01-31T19:44:04.833 INFO:tasks.workunit.client.0.smithi167.stdout:4/18: dwrite - no filename 2022-01-31T19:44:04.833 INFO:tasks.workunit.client.0.smithi167.stdout:4/19: dwrite - no filename 2022-01-31T19:44:04.833 INFO:tasks.workunit.client.0.smithi167.stdout:7/72: sync 2022-01-31T19:44:04.833 INFO:tasks.workunit.client.0.smithi167.stdout:4/20: readlink d0/l5 0 2022-01-31T19:44:04.834 INFO:tasks.workunit.client.0.smithi167.stdout:0/0: sync 2022-01-31T19:44:04.834 INFO:tasks.workunit.client.0.smithi167.stdout:3/45: read f8 [378288,16847] 0 2022-01-31T19:44:04.834 INFO:tasks.workunit.client.0.smithi167.stdout:1/69: write d0/f5 [3714465,26107] 0 2022-01-31T19:44:04.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/87: read d6/db/fd [440829,11420] 0 2022-01-31T19:44:04.835 INFO:tasks.workunit.client.0.smithi167.stdout:9/37: rename l6 to lb 0 2022-01-31T19:44:04.835 INFO:tasks.workunit.client.0.smithi167.stdout:9/38: fdatasync f8 0 2022-01-31T19:44:04.835 INFO:tasks.workunit.client.0.smithi167.stdout:1/70: write d0/fe [412261,26243] 0 2022-01-31T19:44:04.836 INFO:tasks.workunit.client.0.smithi167.stdout:5/35: rename f1 to d5/f8 0 2022-01-31T19:44:04.836 INFO:tasks.workunit.client.0.smithi167.stdout:6/21: mkdir d8 0 2022-01-31T19:44:04.837 INFO:tasks.workunit.client.0.smithi167.stdout:6/22: write f3 [385454,18390] 0 2022-01-31T19:44:04.837 INFO:tasks.workunit.client.0.smithi167.stdout:7/73: mkdir d8/d1b 0 2022-01-31T19:44:04.837 INFO:tasks.workunit.client.0.smithi167.stdout:4/21: rename d0/d3 to d0/d6 0 2022-01-31T19:44:04.837 INFO:tasks.workunit.client.0.smithi167.stdout:4/22: write - no filename 2022-01-31T19:44:04.838 INFO:tasks.workunit.client.0.smithi167.stdout:4/23: read - no filename 2022-01-31T19:44:04.838 INFO:tasks.workunit.client.0.smithi167.stdout:0/1: creat f0 x:0 0 0 2022-01-31T19:44:04.839 INFO:tasks.workunit.client.0.smithi167.stdout:2/27: dread f0 [0,4194304] 0 2022-01-31T19:44:04.840 INFO:tasks.workunit.client.0.smithi167.stdout:9/39: symlink lc 0 2022-01-31T19:44:04.840 INFO:tasks.workunit.client.0.smithi167.stdout:9/40: dread - f7 zero size 2022-01-31T19:44:04.840 INFO:tasks.workunit.client.0.smithi167.stdout:1/71: mknod d0/d10/c1d 0 2022-01-31T19:44:04.841 INFO:tasks.workunit.client.0.smithi167.stdout:1/72: readlink d0/l8 0 2022-01-31T19:44:04.843 INFO:tasks.workunit.client.0.smithi167.stdout:6/23: creat d8/f9 x:0 0 0 2022-01-31T19:44:04.843 INFO:tasks.workunit.client.0.smithi167.stdout:6/24: creat d8/fa x:0 0 0 2022-01-31T19:44:04.843 INFO:tasks.workunit.client.0.smithi167.stdout:6/25: creat d8/fb x:0 0 0 2022-01-31T19:44:04.844 INFO:tasks.workunit.client.0.smithi167.stdout:6/26: write d8/f9 [776134,125777] 0 2022-01-31T19:44:04.844 INFO:tasks.workunit.client.0.smithi167.stdout:6/27: dread - d8/fa zero size 2022-01-31T19:44:04.844 INFO:tasks.workunit.client.0.smithi167.stdout:6/28: dread - d8/fb zero size 2022-01-31T19:44:04.844 INFO:tasks.workunit.client.0.smithi167.stdout:6/29: stat d8/f9 0 2022-01-31T19:44:04.845 INFO:tasks.workunit.client.0.smithi167.stdout:7/74: unlink d8/dc/f15 0 2022-01-31T19:44:04.846 INFO:tasks.workunit.client.0.smithi167.stdout:4/24: mkdir d0/d6/d7 0 2022-01-31T19:44:04.846 INFO:tasks.workunit.client.0.smithi167.stdout:4/25: chown d0 158420380 1 2022-01-31T19:44:04.846 INFO:tasks.workunit.client.0.smithi167.stdout:4/26: write - no filename 2022-01-31T19:44:04.846 INFO:tasks.workunit.client.0.smithi167.stdout:4/27: chown d0/d6 1825 1 2022-01-31T19:44:04.847 INFO:tasks.workunit.client.0.smithi167.stdout:0/2: unlink f0 0 2022-01-31T19:44:04.847 INFO:tasks.workunit.client.0.smithi167.stdout:0/3: write - no filename 2022-01-31T19:44:04.848 INFO:tasks.workunit.client.0.smithi167.stdout:0/4: creat f1 x:0 0 0 2022-01-31T19:44:04.848 INFO:tasks.workunit.client.0.smithi167.stdout:0/5: chown f1 1 1 2022-01-31T19:44:04.849 INFO:tasks.workunit.client.0.smithi167.stdout:1/73: creat d0/d10/d18/f1e x:0 0 0 2022-01-31T19:44:04.849 INFO:tasks.workunit.client.0.smithi167.stdout:1/74: fsync d0/f1 0 2022-01-31T19:44:04.849 INFO:tasks.workunit.client.0.smithi167.stdout:1/75: chown d0/f17 7874778 1 2022-01-31T19:44:04.852 INFO:tasks.workunit.client.0.smithi167.stdout:9/41: dread f4 [4194304,4194304] 0 2022-01-31T19:44:04.852 INFO:tasks.workunit.client.0.smithi167.stdout:7/75: creat d8/dc/f1c x:0 0 0 2022-01-31T19:44:04.852 INFO:tasks.workunit.client.0.smithi167.stdout:7/76: chown d8/dc/d12/c14 21092961 1 2022-01-31T19:44:04.853 INFO:tasks.workunit.client.0.smithi167.stdout:9/42: write f5 [7233371,85102] 0 2022-01-31T19:44:04.853 INFO:tasks.workunit.client.0.smithi167.stdout:4/28: creat d0/d6/d7/f8 x:0 0 0 2022-01-31T19:44:04.853 INFO:tasks.workunit.client.0.smithi167.stdout:4/29: write d0/d6/d7/f8 [610621,49725] 0 2022-01-31T19:44:04.854 INFO:tasks.workunit.client.0.smithi167.stdout:1/76: rmdir d0/d10 39 2022-01-31T19:44:04.855 INFO:tasks.workunit.client.0.smithi167.stdout:7/77: mknod d8/dc/c1d 0 2022-01-31T19:44:04.856 INFO:tasks.workunit.client.0.smithi167.stdout:7/78: fsync d8/fa 0 2022-01-31T19:44:04.856 INFO:tasks.workunit.client.0.smithi167.stdout:9/43: read f4 [2439792,68937] 0 2022-01-31T19:44:04.857 INFO:tasks.workunit.client.0.smithi167.stdout:3/46: dwrite d9/fa [0,4194304] 0 2022-01-31T19:44:04.857 INFO:tasks.workunit.client.0.smithi167.stdout:5/36: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:44:04.857 INFO:tasks.workunit.client.0.smithi167.stdout:5/37: chown d5 3 1 2022-01-31T19:44:04.858 INFO:tasks.workunit.client.0.smithi167.stdout:0/6: rename f1 to f2 0 2022-01-31T19:44:04.859 INFO:tasks.workunit.client.0.smithi167.stdout:0/7: fdatasync f2 0 2022-01-31T19:44:04.859 INFO:tasks.workunit.client.0.smithi167.stdout:3/47: dread d9/fc [0,4194304] 0 2022-01-31T19:44:04.859 INFO:tasks.workunit.client.0.smithi167.stdout:1/77: mkdir d0/d10/d18/d16/d1f 0 2022-01-31T19:44:04.859 INFO:tasks.workunit.client.0.smithi167.stdout:1/78: write d0/d10/d18/f1e [838494,127992] 0 2022-01-31T19:44:04.861 INFO:tasks.workunit.client.0.smithi167.stdout:7/79: mknod d8/dc/d12/c1e 0 2022-01-31T19:44:04.868 INFO:tasks.workunit.client.0.smithi167.stdout:8/88: write d6/fa [35229,46622] 0 2022-01-31T19:44:04.872 INFO:tasks.workunit.client.0.smithi167.stdout:0/8: symlink l3 0 2022-01-31T19:44:04.872 INFO:tasks.workunit.client.0.smithi167.stdout:0/9: dread - f2 zero size 2022-01-31T19:44:04.872 INFO:tasks.workunit.client.0.smithi167.stdout:0/10: creat f4 x:0 0 0 2022-01-31T19:44:04.873 INFO:tasks.workunit.client.0.smithi167.stdout:0/11: read - f4 zero size 2022-01-31T19:44:04.878 INFO:tasks.workunit.client.0.smithi167.stdout:7/80: rename d8/dc/d12/c14 to d8/d1b/c1f 0 2022-01-31T19:44:04.881 INFO:tasks.workunit.client.0.smithi167.stdout:3/48: getdents d9 0 2022-01-31T19:44:04.884 INFO:tasks.workunit.client.0.smithi167.stdout:5/38: dread d5/f8 [4194304,4194304] 0 2022-01-31T19:44:04.884 INFO:tasks.workunit.client.0.smithi167.stdout:5/39: creat d5/f9 x:0 0 0 2022-01-31T19:44:04.884 INFO:tasks.workunit.client.0.smithi167.stdout:5/40: write d5/f9 [512983,97078] 0 2022-01-31T19:44:04.887 INFO:tasks.workunit.client.0.smithi167.stdout:5/41: symlink d5/d6/la 0 2022-01-31T19:44:04.888 INFO:tasks.workunit.client.0.smithi167.stdout:9/44: dwrite f4 [4194304,4194304] 0 2022-01-31T19:44:04.888 INFO:tasks.workunit.client.0.smithi167.stdout:9/45: chown l0 2014 1 2022-01-31T19:44:04.889 INFO:tasks.workunit.client.0.smithi167.stdout:5/42: symlink d5/lb 0 2022-01-31T19:44:04.902 INFO:tasks.workunit.client.0.smithi167.stdout:4/30: dwrite d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:04.902 INFO:tasks.workunit.client.0.smithi167.stdout:3/49: truncate d9/fa 2645023 0 2022-01-31T19:44:04.903 INFO:tasks.workunit.client.0.smithi167.stdout:3/50: write f1 [3274162,76854] 0 2022-01-31T19:44:04.904 INFO:tasks.workunit.client.0.smithi167.stdout:3/51: read f1 [3128737,6301] 0 2022-01-31T19:44:04.904 INFO:tasks.workunit.client.0.smithi167.stdout:3/52: read f1 [2431672,48039] 0 2022-01-31T19:44:04.905 INFO:tasks.workunit.client.0.smithi167.stdout:4/31: rename d0/l5 to d0/l9 0 2022-01-31T19:44:04.906 INFO:tasks.workunit.client.0.smithi167.stdout:3/53: symlink d9/le 0 2022-01-31T19:44:04.907 INFO:tasks.workunit.client.0.smithi167.stdout:4/32: mknod d0/ca 0 2022-01-31T19:44:04.907 INFO:tasks.workunit.client.0.smithi167.stdout:3/54: rmdir d9 39 2022-01-31T19:44:04.908 INFO:tasks.workunit.client.0.smithi167.stdout:1/79: dwrite d0/fe [0,4194304] 0 2022-01-31T19:44:04.912 INFO:tasks.workunit.client.0.smithi167.stdout:0/12: dwrite f2 [0,4194304] 0 2022-01-31T19:44:04.912 INFO:tasks.workunit.client.0.smithi167.stdout:1/80: mkdir d0/d10/d20 0 2022-01-31T19:44:04.912 INFO:tasks.workunit.client.0.smithi167.stdout:8/89: dwrite d6/db/f1a [0,4194304] 0 2022-01-31T19:44:04.913 INFO:tasks.workunit.client.0.smithi167.stdout:7/81: dwrite d8/dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:04.914 INFO:tasks.workunit.client.0.smithi167.stdout:2/28: dwrite f8 [0,4194304] 0 2022-01-31T19:44:04.915 INFO:tasks.workunit.client.0.smithi167.stdout:5/43: rmdir d5 39 2022-01-31T19:44:04.916 INFO:tasks.workunit.client.0.smithi167.stdout:5/44: chown l0 1757831 1 2022-01-31T19:44:04.917 INFO:tasks.workunit.client.0.smithi167.stdout:5/45: read d5/f7 [1439679,24284] 0 2022-01-31T19:44:04.917 INFO:tasks.workunit.client.0.smithi167.stdout:5/46: chown l4 504496047 1 2022-01-31T19:44:04.917 INFO:tasks.workunit.client.0.smithi167.stdout:9/46: truncate f5 8697132 0 2022-01-31T19:44:04.920 INFO:tasks.workunit.client.0.smithi167.stdout:1/81: truncate d0/fd 600962 0 2022-01-31T19:44:04.921 INFO:tasks.workunit.client.0.smithi167.stdout:4/33: truncate d0/d6/d7/f8 2235484 0 2022-01-31T19:44:04.921 INFO:tasks.workunit.client.0.smithi167.stdout:4/34: chown d0/d6/d7/f8 22243832 1 2022-01-31T19:44:04.922 INFO:tasks.workunit.client.0.smithi167.stdout:4/35: fdatasync d0/d6/d7/f8 0 2022-01-31T19:44:04.926 INFO:tasks.workunit.client.0.smithi167.stdout:8/90: creat d6/db/de/d18/f1b x:0 0 0 2022-01-31T19:44:04.927 INFO:tasks.workunit.client.0.smithi167.stdout:0/13: write f2 [4969513,17443] 0 2022-01-31T19:44:04.927 INFO:tasks.workunit.client.0.smithi167.stdout:0/14: rmdir - no directory 2022-01-31T19:44:04.927 INFO:tasks.workunit.client.0.smithi167.stdout:0/15: chown l3 106 1 2022-01-31T19:44:04.929 INFO:tasks.workunit.client.0.smithi167.stdout:7/82: symlink d8/dc/d12/l20 0 2022-01-31T19:44:04.933 INFO:tasks.workunit.client.0.smithi167.stdout:5/47: mknod d5/cc 0 2022-01-31T19:44:04.934 INFO:tasks.workunit.client.0.smithi167.stdout:9/47: mkdir dd 0 2022-01-31T19:44:04.936 INFO:tasks.workunit.client.0.smithi167.stdout:1/82: symlink d0/d11/l21 0 2022-01-31T19:44:04.936 INFO:tasks.workunit.client.0.smithi167.stdout:1/83: rename d0 to d0/d10/d18/d22 22 2022-01-31T19:44:04.938 INFO:tasks.workunit.client.0.smithi167.stdout:8/91: rmdir d6 39 2022-01-31T19:44:04.939 INFO:tasks.workunit.client.0.smithi167.stdout:0/16: mknod c5 0 2022-01-31T19:44:04.940 INFO:tasks.workunit.client.0.smithi167.stdout:2/29: dwrite f3 [0,4194304] 0 2022-01-31T19:44:04.940 INFO:tasks.workunit.client.0.smithi167.stdout:2/30: readlink - no filename 2022-01-31T19:44:04.941 INFO:tasks.workunit.client.0.smithi167.stdout:3/55: dwrite f1 [0,4194304] 0 2022-01-31T19:44:04.941 INFO:tasks.workunit.client.0.smithi167.stdout:0/17: write f2 [106585,93196] 0 2022-01-31T19:44:04.942 INFO:tasks.workunit.client.0.smithi167.stdout:0/18: chown c5 3507070 1 2022-01-31T19:44:04.942 INFO:tasks.workunit.client.0.smithi167.stdout:0/19: chown c5 2042279009 1 2022-01-31T19:44:04.942 INFO:tasks.workunit.client.0.smithi167.stdout:5/48: mknod d5/d6/cd 0 2022-01-31T19:44:04.943 INFO:tasks.workunit.client.0.smithi167.stdout:2/31: truncate f0 1662515 0 2022-01-31T19:44:04.944 INFO:tasks.workunit.client.0.smithi167.stdout:1/84: write d0/f6 [3976925,21698] 0 2022-01-31T19:44:04.944 INFO:tasks.workunit.client.0.smithi167.stdout:1/85: dread - d0/d10/f1a zero size 2022-01-31T19:44:04.944 INFO:tasks.workunit.client.0.smithi167.stdout:3/56: getdents d9 0 2022-01-31T19:44:04.944 INFO:tasks.workunit.client.0.smithi167.stdout:3/57: write d9/fa [2844377,19044] 0 2022-01-31T19:44:04.945 INFO:tasks.workunit.client.0.smithi167.stdout:3/58: chown f8 1 1 2022-01-31T19:44:04.945 INFO:tasks.workunit.client.0.smithi167.stdout:3/59: chown f8 2 1 2022-01-31T19:44:04.945 INFO:tasks.workunit.client.0.smithi167.stdout:0/20: mknod c6 0 2022-01-31T19:44:04.948 INFO:tasks.workunit.client.0.smithi167.stdout:3/60: rename d9/fa to d9/ff 0 2022-01-31T19:44:04.949 INFO:tasks.workunit.client.0.smithi167.stdout:3/61: chown l4 49262433 1 2022-01-31T19:44:04.949 INFO:tasks.workunit.client.0.smithi167.stdout:3/62: fdatasync f1 0 2022-01-31T19:44:04.950 INFO:tasks.workunit.client.0.smithi167.stdout:0/21: mknod c7 0 2022-01-31T19:44:04.954 INFO:tasks.workunit.client.0.smithi167.stdout:0/22: link f4 f8 0 2022-01-31T19:44:04.958 INFO:tasks.workunit.client.0.smithi167.stdout:0/23: link f4 f9 0 2022-01-31T19:44:04.958 INFO:tasks.workunit.client.0.smithi167.stdout:0/24: creat fa x:0 0 0 2022-01-31T19:44:04.958 INFO:tasks.workunit.client.0.smithi167.stdout:7/83: truncate d8/dc/d19/f1a 1452051 0 2022-01-31T19:44:04.959 INFO:tasks.workunit.client.0.smithi167.stdout:7/84: write d8/dc/f1c [717620,21766] 0 2022-01-31T19:44:04.959 INFO:tasks.workunit.client.0.smithi167.stdout:7/85: truncate d8/dc/f17 208833 0 2022-01-31T19:44:04.961 INFO:tasks.workunit.client.0.smithi167.stdout:7/86: truncate f6 2489 0 2022-01-31T19:44:04.961 INFO:tasks.workunit.client.0.smithi167.stdout:7/87: creat d8/dc/f21 x:0 0 0 2022-01-31T19:44:04.961 INFO:tasks.workunit.client.0.smithi167.stdout:7/88: chown l4 4471 1 2022-01-31T19:44:04.962 INFO:tasks.workunit.client.0.smithi167.stdout:1/86: dread d0/d10/d18/f1e [0,4194304] 0 2022-01-31T19:44:04.963 INFO:tasks.workunit.client.0.smithi167.stdout:1/87: chown d0 3068 1 2022-01-31T19:44:04.965 INFO:tasks.workunit.client.0.smithi167.stdout:4/36: dwrite d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:04.965 INFO:tasks.workunit.client.0.smithi167.stdout:4/37: stat d0/d6/l4 0 2022-01-31T19:44:04.967 INFO:tasks.workunit.client.0.smithi167.stdout:9/48: dwrite f5 [4194304,4194304] 0 2022-01-31T19:44:04.967 INFO:tasks.workunit.client.0.smithi167.stdout:9/49: readlink lc 0 2022-01-31T19:44:04.971 INFO:tasks.workunit.client.0.smithi167.stdout:4/38: getdents d0 0 2022-01-31T19:44:04.973 INFO:tasks.workunit.client.0.smithi167.stdout:4/39: read d0/d6/d7/f8 [3174788,7467] 0 2022-01-31T19:44:04.976 INFO:tasks.workunit.client.0.smithi167.stdout:1/88: dread d0/f1 [0,4194304] 0 2022-01-31T19:44:04.976 INFO:tasks.workunit.client.0.smithi167.stdout:1/89: dread - d0/f1b zero size 2022-01-31T19:44:04.976 INFO:tasks.workunit.client.0.smithi167.stdout:3/63: dread d9/ff [0,4194304] 0 2022-01-31T19:44:04.977 INFO:tasks.workunit.client.0.smithi167.stdout:8/92: dwrite d6/f14 [0,4194304] 0 2022-01-31T19:44:04.978 INFO:tasks.workunit.client.0.smithi167.stdout:5/49: write d5/f8 [5226998,99381] 0 2022-01-31T19:44:04.979 INFO:tasks.workunit.client.0.smithi167.stdout:4/40: write d0/d6/d7/f8 [2086828,107436] 0 2022-01-31T19:44:04.980 INFO:tasks.workunit.client.0.smithi167.stdout:1/90: symlink d0/l23 0 2022-01-31T19:44:04.981 INFO:tasks.workunit.client.0.smithi167.stdout:3/64: chown d9/cd 22 1 2022-01-31T19:44:04.983 INFO:tasks.workunit.client.0.smithi167.stdout:8/93: mknod d6/db/de/d18/c1c 0 2022-01-31T19:44:04.987 INFO:tasks.workunit.client.0.smithi167.stdout:1/91: mkdir d0/d11/d24 0 2022-01-31T19:44:04.990 INFO:tasks.workunit.client.0.smithi167.stdout:8/94: rename d6/db/f1a to d6/f1d 0 2022-01-31T19:44:04.992 INFO:tasks.workunit.client.0.smithi167.stdout:1/92: creat d0/f25 x:0 0 0 2022-01-31T19:44:04.992 INFO:tasks.workunit.client.0.smithi167.stdout:1/93: chown d0/c1c 2677074 1 2022-01-31T19:44:04.993 INFO:tasks.workunit.client.0.smithi167.stdout:1/94: mkdir d0/d10/d18/d16/d26 0 2022-01-31T19:44:04.996 INFO:tasks.workunit.client.0.smithi167.stdout:0/25: dwrite f8 [0,4194304] 0 2022-01-31T19:44:05.000 INFO:tasks.workunit.client.0.smithi167.stdout:0/26: read f8 [885986,99449] 0 2022-01-31T19:44:05.000 INFO:tasks.workunit.client.0.smithi167.stdout:0/27: unlink f2 0 2022-01-31T19:44:05.001 INFO:tasks.workunit.client.0.smithi167.stdout:7/89: truncate f6 385756 0 2022-01-31T19:44:05.001 INFO:tasks.workunit.client.0.smithi167.stdout:7/90: fdatasync f2 0 2022-01-31T19:44:05.005 INFO:tasks.workunit.client.0.smithi167.stdout:7/91: rmdir d8/dc/d12 39 2022-01-31T19:44:05.008 INFO:tasks.workunit.client.0.smithi167.stdout:4/41: truncate d0/d6/d7/f8 1743794 0 2022-01-31T19:44:05.008 INFO:tasks.workunit.client.0.smithi167.stdout:4/42: chown d0/l2 51682190 1 2022-01-31T19:44:05.009 INFO:tasks.workunit.client.0.smithi167.stdout:7/92: mkdir d8/d1b/d22 0 2022-01-31T19:44:05.013 INFO:tasks.workunit.client.0.smithi167.stdout:2/32: dwrite f0 [0,4194304] 0 2022-01-31T19:44:05.014 INFO:tasks.workunit.client.0.smithi167.stdout:5/50: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:44:05.014 INFO:tasks.workunit.client.0.smithi167.stdout:3/65: dwrite f1 [0,4194304] 0 2022-01-31T19:44:05.015 INFO:tasks.workunit.client.0.smithi167.stdout:0/28: dread f9 [0,4194304] 0 2022-01-31T19:44:05.015 INFO:tasks.workunit.client.0.smithi167.stdout:8/95: write d6/f13 [4744677,89416] 0 2022-01-31T19:44:05.015 INFO:tasks.workunit.client.0.smithi167.stdout:8/96: chown d6/db/c11 56 1 2022-01-31T19:44:05.016 INFO:tasks.workunit.client.0.smithi167.stdout:8/97: write d6/f1d [865757,17422] 0 2022-01-31T19:44:05.016 INFO:tasks.workunit.client.0.smithi167.stdout:8/98: stat d6/f13 0 2022-01-31T19:44:05.017 INFO:tasks.workunit.client.0.smithi167.stdout:2/33: rename f3 to f9 0 2022-01-31T19:44:05.019 INFO:tasks.workunit.client.0.smithi167.stdout:0/29: rename c5 to cb 0 2022-01-31T19:44:05.020 INFO:tasks.workunit.client.0.smithi167.stdout:0/30: stat c6 0 2022-01-31T19:44:05.020 INFO:tasks.workunit.client.0.smithi167.stdout:0/31: readlink l3 0 2022-01-31T19:44:05.021 INFO:tasks.workunit.client.0.smithi167.stdout:7/93: link f6 d8/dc/f23 0 2022-01-31T19:44:05.024 INFO:tasks.workunit.client.0.smithi167.stdout:3/66: creat d9/f10 x:0 0 0 2022-01-31T19:44:05.025 INFO:tasks.workunit.client.0.smithi167.stdout:3/67: chown l5 249611760 1 2022-01-31T19:44:05.025 INFO:tasks.workunit.client.0.smithi167.stdout:3/68: read - d9/f10 zero size 2022-01-31T19:44:05.027 INFO:tasks.workunit.client.0.smithi167.stdout:5/51: truncate d5/f8 7547723 0 2022-01-31T19:44:05.028 INFO:tasks.workunit.client.0.smithi167.stdout:2/34: dread f7 [0,4194304] 0 2022-01-31T19:44:05.028 INFO:tasks.workunit.client.0.smithi167.stdout:2/35: write f0 [4711997,1520] 0 2022-01-31T19:44:05.029 INFO:tasks.workunit.client.0.smithi167.stdout:7/94: mkdir d8/dc/d18/d24 0 2022-01-31T19:44:05.031 INFO:tasks.workunit.client.0.smithi167.stdout:4/43: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:05.033 INFO:tasks.workunit.client.0.smithi167.stdout:3/69: creat d9/f11 x:0 0 0 2022-01-31T19:44:05.033 INFO:tasks.workunit.client.0.smithi167.stdout:3/70: chown f1 3914 1 2022-01-31T19:44:05.035 INFO:tasks.workunit.client.0.smithi167.stdout:5/52: mkdir d5/d6/de 0 2022-01-31T19:44:05.037 INFO:tasks.workunit.client.0.smithi167.stdout:7/95: dread f2 [0,4194304] 0 2022-01-31T19:44:05.040 INFO:tasks.workunit.client.0.smithi167.stdout:2/36: rename f0 to fa 0 2022-01-31T19:44:05.040 INFO:tasks.workunit.client.0.smithi167.stdout:2/37: creat fb x:0 0 0 2022-01-31T19:44:05.041 INFO:tasks.workunit.client.0.smithi167.stdout:4/44: getdents d0/d6/d7 0 2022-01-31T19:44:05.042 INFO:tasks.workunit.client.0.smithi167.stdout:3/71: stat l3 0 2022-01-31T19:44:05.043 INFO:tasks.workunit.client.0.smithi167.stdout:2/38: write f8 [3786034,123710] 0 2022-01-31T19:44:05.043 INFO:tasks.workunit.client.0.smithi167.stdout:4/45: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:05.044 INFO:tasks.workunit.client.0.smithi167.stdout:5/53: creat d5/ff x:0 0 0 2022-01-31T19:44:05.049 INFO:tasks.workunit.client.0.smithi167.stdout:4/46: creat d0/d6/d7/fb x:0 0 0 2022-01-31T19:44:05.051 INFO:tasks.workunit.client.0.smithi167.stdout:5/54: mkdir d5/d10 0 2022-01-31T19:44:05.052 INFO:tasks.workunit.client.0.smithi167.stdout:3/72: mknod d9/c12 0 2022-01-31T19:44:05.054 INFO:tasks.workunit.client.0.smithi167.stdout:4/47: mkdir d0/d6/dc 0 2022-01-31T19:44:05.055 INFO:tasks.workunit.client.0.smithi167.stdout:7/96: write d8/dc/f23 [540037,75328] 0 2022-01-31T19:44:05.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/73: creat d9/f13 x:0 0 0 2022-01-31T19:44:05.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/74: dread - d9/f10 zero size 2022-01-31T19:44:05.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/75: readlink l5 0 2022-01-31T19:44:05.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/76: write f8 [1982706,18489] 0 2022-01-31T19:44:05.057 INFO:tasks.workunit.client.0.smithi167.stdout:7/97: read d8/dc/d19/f1a [1115613,41717] 0 2022-01-31T19:44:05.057 INFO:tasks.workunit.client.0.smithi167.stdout:4/48: mkdir d0/d6/dc/dd 0 2022-01-31T19:44:05.057 INFO:tasks.workunit.client.0.smithi167.stdout:4/49: chown d0/d6/d7 47 1 2022-01-31T19:44:05.058 INFO:tasks.workunit.client.0.smithi167.stdout:7/98: link f6 d8/d1b/f25 0 2022-01-31T19:44:05.059 INFO:tasks.workunit.client.0.smithi167.stdout:4/50: creat d0/d6/dc/dd/fe x:0 0 0 2022-01-31T19:44:05.060 INFO:tasks.workunit.client.0.smithi167.stdout:3/77: rename l3 to d9/l14 0 2022-01-31T19:44:05.061 INFO:tasks.workunit.client.0.smithi167.stdout:3/78: truncate f7 2832863 0 2022-01-31T19:44:05.062 INFO:tasks.workunit.client.0.smithi167.stdout:5/55: dread d5/f8 [0,4194304] 0 2022-01-31T19:44:05.062 INFO:tasks.workunit.client.0.smithi167.stdout:5/56: stat d5/d6/cd 0 2022-01-31T19:44:05.063 INFO:tasks.workunit.client.0.smithi167.stdout:3/79: creat d9/f15 x:0 0 0 2022-01-31T19:44:05.063 INFO:tasks.workunit.client.0.smithi167.stdout:3/80: chown d9/ff 1 1 2022-01-31T19:44:05.064 INFO:tasks.workunit.client.0.smithi167.stdout:3/81: fdatasync d9/f13 0 2022-01-31T19:44:05.066 INFO:tasks.workunit.client.0.smithi167.stdout:8/99: dwrite d6/ff [0,4194304] 0 2022-01-31T19:44:05.067 INFO:tasks.workunit.client.0.smithi167.stdout:2/39: dwrite f8 [4194304,4194304] 0 2022-01-31T19:44:05.067 INFO:tasks.workunit.client.0.smithi167.stdout:2/40: creat fc x:0 0 0 2022-01-31T19:44:05.071 INFO:tasks.workunit.client.0.smithi167.stdout:5/57: dread d5/f7 [0,4194304] 0 2022-01-31T19:44:05.071 INFO:tasks.workunit.client.0.smithi167.stdout:5/58: readlink l0 0 2022-01-31T19:44:05.071 INFO:tasks.workunit.client.0.smithi167.stdout:5/59: chown d5/lb 59137950 1 2022-01-31T19:44:05.075 INFO:tasks.workunit.client.0.smithi167.stdout:8/100: mkdir d6/d1e 0 2022-01-31T19:44:05.075 INFO:tasks.workunit.client.0.smithi167.stdout:8/101: truncate d6/fa 1416374 0 2022-01-31T19:44:05.076 INFO:tasks.workunit.client.0.smithi167.stdout:8/102: chown d6/f12 0 1 2022-01-31T19:44:05.076 INFO:tasks.workunit.client.0.smithi167.stdout:4/51: fsync d0/d6/dc/dd/fe 0 2022-01-31T19:44:05.076 INFO:tasks.workunit.client.0.smithi167.stdout:4/52: stat d0/d6/d7/f8 0 2022-01-31T19:44:05.077 INFO:tasks.workunit.client.0.smithi167.stdout:3/82: dread f8 [0,4194304] 0 2022-01-31T19:44:05.080 INFO:tasks.workunit.client.0.smithi167.stdout:2/41: mkdir dd 0 2022-01-31T19:44:05.083 INFO:tasks.workunit.client.0.smithi167.stdout:2/42: symlink dd/le 0 2022-01-31T19:44:05.084 INFO:tasks.workunit.client.0.smithi167.stdout:3/83: link d9/cd d9/c16 0 2022-01-31T19:44:05.084 INFO:tasks.workunit.client.0.smithi167.stdout:3/84: write d9/f15 [433139,120967] 0 2022-01-31T19:44:05.084 INFO:tasks.workunit.client.0.smithi167.stdout:3/85: dread - d9/f13 zero size 2022-01-31T19:44:05.085 INFO:tasks.workunit.client.0.smithi167.stdout:3/86: creat d9/f17 x:0 0 0 2022-01-31T19:44:05.085 INFO:tasks.workunit.client.0.smithi167.stdout:3/87: write d9/f13 [650059,12649] 0 2022-01-31T19:44:05.086 INFO:tasks.workunit.client.0.smithi167.stdout:3/88: creat d9/f18 x:0 0 0 2022-01-31T19:44:05.086 INFO:tasks.workunit.client.0.smithi167.stdout:8/103: dread d6/f1d [0,4194304] 0 2022-01-31T19:44:05.088 INFO:tasks.workunit.client.0.smithi167.stdout:8/104: symlink d6/db/de/d18/l1f 0 2022-01-31T19:44:05.088 INFO:tasks.workunit.client.0.smithi167.stdout:8/105: readlink l2 0 2022-01-31T19:44:05.090 INFO:tasks.workunit.client.0.smithi167.stdout:8/106: creat d6/db/de/f20 x:0 0 0 2022-01-31T19:44:05.090 INFO:tasks.workunit.client.0.smithi167.stdout:8/107: creat d6/db/de/d18/f21 x:0 0 0 2022-01-31T19:44:05.099 INFO:tasks.workunit.client.0.smithi167.stdout:5/60: dwrite d5/f8 [0,4194304] 0 2022-01-31T19:44:05.100 INFO:tasks.workunit.client.0.smithi167.stdout:7/99: dwrite d8/dc/f23 [0,4194304] 0 2022-01-31T19:44:05.103 INFO:tasks.workunit.client.0.smithi167.stdout:5/61: dread d5/f7 [0,4194304] 0 2022-01-31T19:44:05.103 INFO:tasks.workunit.client.0.smithi167.stdout:5/62: fsync d5/f7 0 2022-01-31T19:44:05.104 INFO:tasks.workunit.client.0.smithi167.stdout:2/43: dwrite fa [0,4194304] 0 2022-01-31T19:44:05.105 INFO:tasks.workunit.client.0.smithi167.stdout:3/89: rmdir d9 39 2022-01-31T19:44:05.108 INFO:tasks.workunit.client.0.smithi167.stdout:5/63: symlink d5/d10/l11 0 2022-01-31T19:44:05.108 INFO:tasks.workunit.client.0.smithi167.stdout:5/64: fdatasync d5/ff 0 2022-01-31T19:44:05.108 INFO:tasks.workunit.client.0.smithi167.stdout:5/65: write d5/ff [74833,81329] 0 2022-01-31T19:44:05.109 INFO:tasks.workunit.client.0.smithi167.stdout:8/108: dwrite d6/db/de/d18/f19 [0,4194304] 0 2022-01-31T19:44:05.109 INFO:tasks.workunit.client.0.smithi167.stdout:8/109: chown d6/c7 1 1 2022-01-31T19:44:05.110 INFO:tasks.workunit.client.0.smithi167.stdout:2/44: mkdir dd/df 0 2022-01-31T19:44:05.112 INFO:tasks.workunit.client.0.smithi167.stdout:5/66: rename d5/ff to d5/d6/f12 0 2022-01-31T19:44:05.112 INFO:tasks.workunit.client.0.smithi167.stdout:5/67: chown d5/d6/cd 142327896 1 2022-01-31T19:44:05.113 INFO:tasks.workunit.client.0.smithi167.stdout:2/45: stat c6 0 2022-01-31T19:44:05.120 INFO:tasks.workunit.client.0.smithi167.stdout:7/100: dwrite d8/dc/f23 [0,4194304] 0 2022-01-31T19:44:05.123 INFO:tasks.workunit.client.0.smithi167.stdout:3/90: write f7 [1474728,82419] 0 2022-01-31T19:44:05.123 INFO:tasks.workunit.client.0.smithi167.stdout:7/101: creat d8/dc/d18/d24/f26 x:0 0 0 2022-01-31T19:44:05.124 INFO:tasks.workunit.client.0.smithi167.stdout:7/102: chown d8/dc/f21 3562585 1 2022-01-31T19:44:05.124 INFO:tasks.workunit.client.0.smithi167.stdout:3/91: getdents d9 0 2022-01-31T19:44:05.125 INFO:tasks.workunit.client.0.smithi167.stdout:3/92: dread - d9/f17 zero size 2022-01-31T19:44:05.125 INFO:tasks.workunit.client.0.smithi167.stdout:3/93: fdatasync d9/fc 0 2022-01-31T19:44:05.125 INFO:tasks.workunit.client.0.smithi167.stdout:3/94: write f8 [2666520,51564] 0 2022-01-31T19:44:05.126 INFO:tasks.workunit.client.0.smithi167.stdout:7/103: symlink d8/l27 0 2022-01-31T19:44:05.127 INFO:tasks.workunit.client.0.smithi167.stdout:3/95: rmdir d9 39 2022-01-31T19:44:05.127 INFO:tasks.workunit.client.0.smithi167.stdout:3/96: chown l5 375 1 2022-01-31T19:44:05.129 INFO:tasks.workunit.client.0.smithi167.stdout:7/104: symlink d8/dc/d12/l28 0 2022-01-31T19:44:05.129 INFO:tasks.workunit.client.0.smithi167.stdout:7/105: creat d8/dc/d18/d24/f29 x:0 0 0 2022-01-31T19:44:05.129 INFO:tasks.workunit.client.0.smithi167.stdout:7/106: write d8/fa [725324,29218] 0 2022-01-31T19:44:05.130 INFO:tasks.workunit.client.0.smithi167.stdout:3/97: symlink d9/l19 0 2022-01-31T19:44:05.133 INFO:tasks.workunit.client.0.smithi167.stdout:8/110: dwrite d6/f8 [4194304,4194304] 0 2022-01-31T19:44:05.134 INFO:tasks.workunit.client.0.smithi167.stdout:7/107: read f6 [3401513,35865] 0 2022-01-31T19:44:05.135 INFO:tasks.workunit.client.0.smithi167.stdout:8/111: creat d6/d1e/f22 x:0 0 0 2022-01-31T19:44:05.136 INFO:tasks.workunit.client.0.smithi167.stdout:8/112: write d6/db/de/d18/f1b [414644,47853] 0 2022-01-31T19:44:05.136 INFO:tasks.workunit.client.0.smithi167.stdout:8/113: dread - d6/db/de/d18/f21 zero size 2022-01-31T19:44:05.137 INFO:tasks.workunit.client.0.smithi167.stdout:7/108: creat d8/dc/d12/f2a x:0 0 0 2022-01-31T19:44:05.137 INFO:tasks.workunit.client.0.smithi167.stdout:8/114: mknod d6/c23 0 2022-01-31T19:44:05.137 INFO:tasks.workunit.client.0.smithi167.stdout:8/115: creat d6/d1e/f24 x:0 0 0 2022-01-31T19:44:05.138 INFO:tasks.workunit.client.0.smithi167.stdout:7/109: rmdir d8/d1b/d22 0 2022-01-31T19:44:05.138 INFO:tasks.workunit.client.0.smithi167.stdout:7/110: creat d8/dc/f2b x:0 0 0 2022-01-31T19:44:05.140 INFO:tasks.workunit.client.0.smithi167.stdout:8/116: symlink d6/db/l25 0 2022-01-31T19:44:05.145 INFO:tasks.workunit.client.0.smithi167.stdout:8/117: write d6/ff [2802642,80947] 0 2022-01-31T19:44:05.145 INFO:tasks.workunit.client.0.smithi167.stdout:5/68: dwrite d5/d6/f12 [0,4194304] 0 2022-01-31T19:44:05.148 INFO:tasks.workunit.client.0.smithi167.stdout:5/69: read d5/f8 [3050271,97399] 0 2022-01-31T19:44:05.150 INFO:tasks.workunit.client.0.smithi167.stdout:5/70: getdents d5/d10 0 2022-01-31T19:44:05.150 INFO:tasks.workunit.client.0.smithi167.stdout:8/118: dread d6/db/fd [0,4194304] 0 2022-01-31T19:44:05.151 INFO:tasks.workunit.client.0.smithi167.stdout:8/119: chown d6/ff 30773 1 2022-01-31T19:44:05.151 INFO:tasks.workunit.client.0.smithi167.stdout:8/120: chown d6/d1e/f24 22 1 2022-01-31T19:44:05.152 INFO:tasks.workunit.client.0.smithi167.stdout:5/71: creat d5/d6/f13 x:0 0 0 2022-01-31T19:44:05.153 INFO:tasks.workunit.client.0.smithi167.stdout:5/72: mkdir d5/d10/d14 0 2022-01-31T19:44:05.154 INFO:tasks.workunit.client.0.smithi167.stdout:2/46: dwrite f2 [0,4194304] 0 2022-01-31T19:44:05.155 INFO:tasks.workunit.client.0.smithi167.stdout:5/73: symlink d5/d10/l15 0 2022-01-31T19:44:05.156 INFO:tasks.workunit.client.0.smithi167.stdout:2/47: read fa [1668167,89011] 0 2022-01-31T19:44:05.157 INFO:tasks.workunit.client.0.smithi167.stdout:5/74: rmdir d5/d10/d14 0 2022-01-31T19:44:05.160 INFO:tasks.workunit.client.0.smithi167.stdout:3/98: dwrite d9/f10 [0,4194304] 0 2022-01-31T19:44:05.163 INFO:tasks.workunit.client.0.smithi167.stdout:3/99: creat d9/f1a x:0 0 0 2022-01-31T19:44:05.164 INFO:tasks.workunit.client.0.smithi167.stdout:5/75: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:05.165 INFO:tasks.workunit.client.0.smithi167.stdout:7/111: dwrite d8/dc/f21 [0,4194304] 0 2022-01-31T19:44:05.167 INFO:tasks.workunit.client.0.smithi167.stdout:7/112: write f7 [905269,66649] 0 2022-01-31T19:44:05.168 INFO:tasks.workunit.client.0.smithi167.stdout:2/48: dread f8 [0,4194304] 0 2022-01-31T19:44:05.168 INFO:tasks.workunit.client.0.smithi167.stdout:2/49: read - fb zero size 2022-01-31T19:44:05.171 INFO:tasks.workunit.client.0.smithi167.stdout:2/50: link c6 dd/df/c10 0 2022-01-31T19:44:05.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/76: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:05.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/77: stat d5/d10/l11 0 2022-01-31T19:44:05.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/78: fdatasync d5/d6/f12 0 2022-01-31T19:44:05.173 INFO:tasks.workunit.client.0.smithi167.stdout:5/79: readlink d5/lb 0 2022-01-31T19:44:05.173 INFO:tasks.workunit.client.0.smithi167.stdout:5/80: readlink l0 0 2022-01-31T19:44:05.174 INFO:tasks.workunit.client.0.smithi167.stdout:2/51: unlink c6 0 2022-01-31T19:44:05.174 INFO:tasks.workunit.client.0.smithi167.stdout:5/81: mkdir d5/d6/d16 0 2022-01-31T19:44:05.174 INFO:tasks.workunit.client.0.smithi167.stdout:8/121: dwrite d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:05.175 INFO:tasks.workunit.client.0.smithi167.stdout:2/52: mknod dd/df/c11 0 2022-01-31T19:44:05.175 INFO:tasks.workunit.client.0.smithi167.stdout:2/53: readlink dd/le 0 2022-01-31T19:44:05.176 INFO:tasks.workunit.client.0.smithi167.stdout:8/122: write d6/fa [287087,27926] 0 2022-01-31T19:44:05.176 INFO:tasks.workunit.client.0.smithi167.stdout:5/82: creat d5/d10/f17 x:0 0 0 2022-01-31T19:44:05.180 INFO:tasks.workunit.client.0.smithi167.stdout:7/113: dread f6 [0,4194304] 0 2022-01-31T19:44:05.180 INFO:tasks.workunit.client.0.smithi167.stdout:7/114: write d8/dc/f23 [4456664,122964] 0 2022-01-31T19:44:05.184 INFO:tasks.workunit.client.0.smithi167.stdout:5/83: mkdir d5/d6/d16/d18 0 2022-01-31T19:44:05.185 INFO:tasks.workunit.client.0.smithi167.stdout:8/123: dread d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:05.187 INFO:tasks.workunit.client.0.smithi167.stdout:3/100: dwrite d9/f11 [0,4194304] 0 2022-01-31T19:44:05.189 INFO:tasks.workunit.client.0.smithi167.stdout:5/84: symlink d5/d6/d16/l19 0 2022-01-31T19:44:05.189 INFO:tasks.workunit.client.0.smithi167.stdout:5/85: fdatasync d5/f9 0 2022-01-31T19:44:05.190 INFO:tasks.workunit.client.0.smithi167.stdout:5/86: creat d5/d6/f1a x:0 0 0 2022-01-31T19:44:05.190 INFO:tasks.workunit.client.0.smithi167.stdout:2/54: truncate f8 1632809 0 2022-01-31T19:44:05.202 INFO:tasks.workunit.client.0.smithi167.stdout:3/101: unlink l6 0 2022-01-31T19:44:05.205 INFO:tasks.workunit.client.0.smithi167.stdout:3/102: dread d9/fc [0,4194304] 0 2022-01-31T19:44:05.206 INFO:tasks.workunit.client.0.smithi167.stdout:3/103: chown d9/cd 37335 1 2022-01-31T19:44:05.207 INFO:tasks.workunit.client.0.smithi167.stdout:5/87: symlink d5/d6/de/l1b 0 2022-01-31T19:44:05.207 INFO:tasks.workunit.client.0.smithi167.stdout:5/88: creat d5/d6/f1c x:0 0 0 2022-01-31T19:44:05.208 INFO:tasks.workunit.client.0.smithi167.stdout:2/55: mknod dd/c12 0 2022-01-31T19:44:05.210 INFO:tasks.workunit.client.0.smithi167.stdout:3/104: mkdir d9/d1b 0 2022-01-31T19:44:05.211 INFO:tasks.workunit.client.0.smithi167.stdout:3/105: creat d9/f1c x:0 0 0 2022-01-31T19:44:05.211 INFO:tasks.workunit.client.0.smithi167.stdout:8/124: dwrite d6/f8 [4194304,4194304] 0 2022-01-31T19:44:05.213 INFO:tasks.workunit.client.0.smithi167.stdout:5/89: mknod d5/d10/c1d 0 2022-01-31T19:44:05.213 INFO:tasks.workunit.client.0.smithi167.stdout:5/90: chown d5 9632 1 2022-01-31T19:44:05.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/106: rename d9/f17 to d9/f1d 0 2022-01-31T19:44:05.219 INFO:tasks.workunit.client.0.smithi167.stdout:8/125: rename d6/d1e/f24 to d6/db/de/f26 0 2022-01-31T19:44:05.220 INFO:tasks.workunit.client.0.smithi167.stdout:2/56: dread f7 [0,4194304] 0 2022-01-31T19:44:05.221 INFO:tasks.workunit.client.0.smithi167.stdout:8/126: getdents d6 0 2022-01-31T19:44:05.226 INFO:tasks.workunit.client.0.smithi167.stdout:8/127: link d6/db/c17 d6/db/de/d18/c27 0 2022-01-31T19:44:05.228 INFO:tasks.workunit.client.0.smithi167.stdout:8/128: mkdir d6/db/d28 0 2022-01-31T19:44:05.228 INFO:tasks.workunit.client.0.smithi167.stdout:5/91: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:44:05.228 INFO:tasks.workunit.client.0.smithi167.stdout:5/92: truncate d5/f8 7983440 0 2022-01-31T19:44:05.229 INFO:tasks.workunit.client.0.smithi167.stdout:8/129: rmdir d6/db 39 2022-01-31T19:44:05.229 INFO:tasks.workunit.client.0.smithi167.stdout:8/130: creat d6/d1e/f29 x:0 0 0 2022-01-31T19:44:05.230 INFO:tasks.workunit.client.0.smithi167.stdout:8/131: truncate d6/db/de/d18/f1b 968821 0 2022-01-31T19:44:05.230 INFO:tasks.workunit.client.0.smithi167.stdout:8/132: chown d6/db/de 202823 1 2022-01-31T19:44:05.230 INFO:tasks.workunit.client.0.smithi167.stdout:8/133: read d6/db/de/f10 [1152120,115509] 0 2022-01-31T19:44:05.230 INFO:tasks.workunit.client.0.smithi167.stdout:8/134: fdatasync d6/f13 0 2022-01-31T19:44:05.231 INFO:tasks.workunit.client.0.smithi167.stdout:8/135: rename d6/db to d6/db/d2a 22 2022-01-31T19:44:05.231 INFO:tasks.workunit.client.0.smithi167.stdout:8/136: chown d6/db/d28 862 1 2022-01-31T19:44:05.232 INFO:tasks.workunit.client.0.smithi167.stdout:5/93: creat d5/d6/de/f1e x:0 0 0 2022-01-31T19:44:05.233 INFO:tasks.workunit.client.0.smithi167.stdout:5/94: stat l4 0 2022-01-31T19:44:05.233 INFO:tasks.workunit.client.0.smithi167.stdout:3/107: getdents d9 0 2022-01-31T19:44:05.234 INFO:tasks.workunit.client.0.smithi167.stdout:3/108: readlink d9/l19 0 2022-01-31T19:44:05.236 INFO:tasks.workunit.client.0.smithi167.stdout:3/109: truncate f1 3044514 0 2022-01-31T19:44:05.236 INFO:tasks.workunit.client.0.smithi167.stdout:5/95: creat d5/d6/d16/d18/f1f x:0 0 0 2022-01-31T19:44:05.237 INFO:tasks.workunit.client.0.smithi167.stdout:5/96: truncate d5/d10/f17 689861 0 2022-01-31T19:44:05.240 INFO:tasks.workunit.client.0.smithi167.stdout:3/110: rename d9/f1d to d9/d1b/f1e 0 2022-01-31T19:44:05.240 INFO:tasks.workunit.client.0.smithi167.stdout:3/111: read d9/f13 [444730,66649] 0 2022-01-31T19:44:05.240 INFO:tasks.workunit.client.0.smithi167.stdout:5/97: mknod d5/d6/d16/c20 0 2022-01-31T19:44:05.240 INFO:tasks.workunit.client.0.smithi167.stdout:5/98: creat d5/d10/f21 x:0 0 0 2022-01-31T19:44:05.241 INFO:tasks.workunit.client.0.smithi167.stdout:5/99: truncate d5/d10/f21 653923 0 2022-01-31T19:44:05.241 INFO:tasks.workunit.client.0.smithi167.stdout:5/100: mkdir d5/d6/de/d22 0 2022-01-31T19:44:05.242 INFO:tasks.workunit.client.0.smithi167.stdout:5/101: truncate d5/d6/f1a 660396 0 2022-01-31T19:44:05.245 INFO:tasks.workunit.client.0.smithi167.stdout:8/137: dwrite d6/db/fd [0,4194304] 0 2022-01-31T19:44:05.248 INFO:tasks.workunit.client.0.smithi167.stdout:8/138: link d6/f1d d6/db/f2b 0 2022-01-31T19:44:05.249 INFO:tasks.workunit.client.0.smithi167.stdout:8/139: mknod d6/db/c2c 0 2022-01-31T19:44:05.250 INFO:tasks.workunit.client.0.smithi167.stdout:8/140: dread - d6/db/de/f20 zero size 2022-01-31T19:44:05.250 INFO:tasks.workunit.client.0.smithi167.stdout:8/141: fdatasync d6/db/de/f26 0 2022-01-31T19:44:05.250 INFO:tasks.workunit.client.0.smithi167.stdout:8/142: stat d6/db/de 0 2022-01-31T19:44:05.251 INFO:tasks.workunit.client.0.smithi167.stdout:8/143: chown d6/l16 495 1 2022-01-31T19:44:05.252 INFO:tasks.workunit.client.0.smithi167.stdout:3/112: dread d9/f11 [0,4194304] 0 2022-01-31T19:44:05.252 INFO:tasks.workunit.client.0.smithi167.stdout:3/113: truncate d9/d1b/f1e 413207 0 2022-01-31T19:44:05.252 INFO:tasks.workunit.client.0.smithi167.stdout:3/114: chown d9/d1b 923141 1 2022-01-31T19:44:05.252 INFO:tasks.workunit.client.0.smithi167.stdout:5/102: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:05.253 INFO:tasks.workunit.client.0.smithi167.stdout:5/103: fsync d5/d10/f17 0 2022-01-31T19:44:05.253 INFO:tasks.workunit.client.0.smithi167.stdout:5/104: write d5/d6/f1c [282029,124752] 0 2022-01-31T19:44:05.254 INFO:tasks.workunit.client.0.smithi167.stdout:5/105: symlink d5/l23 0 2022-01-31T19:44:05.254 INFO:tasks.workunit.client.0.smithi167.stdout:5/106: write d5/d6/d16/d18/f1f [931003,96194] 0 2022-01-31T19:44:05.257 INFO:tasks.workunit.client.0.smithi167.stdout:5/107: rmdir d5/d6/d16/d18 39 2022-01-31T19:44:05.259 INFO:tasks.workunit.client.0.smithi167.stdout:5/108: mkdir d5/d6/d16/d24 0 2022-01-31T19:44:05.260 INFO:tasks.workunit.client.0.smithi167.stdout:5/109: read d5/d6/f1a [132386,67102] 0 2022-01-31T19:44:05.261 INFO:tasks.workunit.client.0.smithi167.stdout:8/144: dread d6/f8 [4194304,4194304] 0 2022-01-31T19:44:05.261 INFO:tasks.workunit.client.0.smithi167.stdout:8/145: read - d6/f12 zero size 2022-01-31T19:44:05.264 INFO:tasks.workunit.client.0.smithi167.stdout:5/110: creat d5/d6/de/d22/f25 x:0 0 0 2022-01-31T19:44:05.264 INFO:tasks.workunit.client.0.smithi167.stdout:5/111: write d5/f9 [977940,4211] 0 2022-01-31T19:44:05.265 INFO:tasks.workunit.client.0.smithi167.stdout:5/112: chown d5/d10/c1d 111 1 2022-01-31T19:44:05.266 INFO:tasks.workunit.client.0.smithi167.stdout:8/146: truncate d6/db/de/d18/f21 3972858 0 2022-01-31T19:44:05.267 INFO:tasks.workunit.client.0.smithi167.stdout:5/113: creat d5/d6/d16/f26 x:0 0 0 2022-01-31T19:44:05.268 INFO:tasks.workunit.client.0.smithi167.stdout:5/114: write d5/d6/d16/d18/f1f [945468,108741] 0 2022-01-31T19:44:05.268 INFO:tasks.workunit.client.0.smithi167.stdout:8/147: mkdir d6/d1e/d2d 0 2022-01-31T19:44:05.269 INFO:tasks.workunit.client.0.smithi167.stdout:5/115: rename c3 to d5/d6/d16/d24/c27 0 2022-01-31T19:44:05.270 INFO:tasks.workunit.client.0.smithi167.stdout:5/116: fsync d5/d6/f12 0 2022-01-31T19:44:05.271 INFO:tasks.workunit.client.0.smithi167.stdout:8/148: rename c4 to d6/db/de/d18/c2e 0 2022-01-31T19:44:05.274 INFO:tasks.workunit.client.0.smithi167.stdout:8/149: truncate d6/f8 7157604 0 2022-01-31T19:44:05.275 INFO:tasks.workunit.client.0.smithi167.stdout:5/117: dread d5/d6/d16/d18/f1f [0,4194304] 0 2022-01-31T19:44:05.275 INFO:tasks.workunit.client.0.smithi167.stdout:8/150: mkdir d6/db/d2f 0 2022-01-31T19:44:05.276 INFO:tasks.workunit.client.0.smithi167.stdout:8/151: fdatasync d6/db/de/f10 0 2022-01-31T19:44:05.277 INFO:tasks.workunit.client.0.smithi167.stdout:8/152: write d6/db/de/d18/f21 [1066686,39350] 0 2022-01-31T19:44:05.292 INFO:tasks.workunit.client.0.smithi167.stdout:5/118: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:44:05.292 INFO:tasks.workunit.client.0.smithi167.stdout:8/153: dread d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:05.293 INFO:tasks.workunit.client.0.smithi167.stdout:5/119: read d5/d6/f1a [631545,22998] 0 2022-01-31T19:44:05.295 INFO:tasks.workunit.client.0.smithi167.stdout:5/120: write d5/d6/f12 [1219996,91035] 0 2022-01-31T19:44:05.296 INFO:tasks.workunit.client.0.smithi167.stdout:5/121: chown d5/d6/cd 861 1 2022-01-31T19:44:05.303 INFO:tasks.workunit.client.0.smithi167.stdout:8/154: dread d6/f13 [0,4194304] 0 2022-01-31T19:44:05.337 INFO:tasks.workunit.client.0.smithi167.stdout:5/122: dwrite d5/f9 [0,4194304] 0 2022-01-31T19:44:05.337 INFO:tasks.workunit.client.0.smithi167.stdout:8/155: dwrite d6/db/de/f20 [0,4194304] 0 2022-01-31T19:44:05.365 INFO:tasks.workunit.client.0.smithi167.stdout:8/156: dwrite d6/ff [4194304,4194304] 0 2022-01-31T19:44:05.366 INFO:tasks.workunit.client.0.smithi167.stdout:8/157: chown d6/l16 269 1 2022-01-31T19:44:05.366 INFO:tasks.workunit.client.0.smithi167.stdout:8/158: readlink d6/l16 0 2022-01-31T19:44:05.366 INFO:tasks.workunit.client.0.smithi167.stdout:5/123: dwrite d5/d6/f1c [0,4194304] 0 2022-01-31T19:44:05.367 INFO:tasks.workunit.client.0.smithi167.stdout:5/124: creat d5/d6/f28 x:0 0 0 2022-01-31T19:44:05.367 INFO:tasks.workunit.client.0.smithi167.stdout:8/159: mknod d6/db/de/d18/c30 0 2022-01-31T19:44:05.368 INFO:tasks.workunit.client.0.smithi167.stdout:8/160: dread - d6/db/de/f26 zero size 2022-01-31T19:44:05.368 INFO:tasks.workunit.client.0.smithi167.stdout:8/161: chown d6/f14 148 1 2022-01-31T19:44:05.369 INFO:tasks.workunit.client.0.smithi167.stdout:5/125: getdents d5/d6/d16/d18 0 2022-01-31T19:44:05.370 INFO:tasks.workunit.client.0.smithi167.stdout:8/162: mknod d6/db/c31 0 2022-01-31T19:44:05.372 INFO:tasks.workunit.client.0.smithi167.stdout:8/163: creat d6/f32 x:0 0 0 2022-01-31T19:44:05.373 INFO:tasks.workunit.client.0.smithi167.stdout:5/126: dread d5/d10/f17 [0,4194304] 0 2022-01-31T19:44:05.373 INFO:tasks.workunit.client.0.smithi167.stdout:5/127: stat d5/d10/c1d 0 2022-01-31T19:44:05.377 INFO:tasks.workunit.client.0.smithi167.stdout:5/128: getdents d5/d6 0 2022-01-31T19:44:05.378 INFO:tasks.workunit.client.0.smithi167.stdout:5/129: symlink d5/d6/d16/l29 0 2022-01-31T19:44:05.378 INFO:tasks.workunit.client.0.smithi167.stdout:5/130: chown d5/d6/cd 0 1 2022-01-31T19:44:05.380 INFO:tasks.workunit.client.0.smithi167.stdout:5/131: rmdir d5/d6/d16 39 2022-01-31T19:44:05.396 INFO:tasks.workunit.client.0.smithi167.stdout:5/132: dwrite d5/d10/f17 [0,4194304] 0 2022-01-31T19:44:05.398 INFO:tasks.workunit.client.0.smithi167.stdout:5/133: rmdir d5/d6/d16 39 2022-01-31T19:44:05.398 INFO:tasks.workunit.client.0.smithi167.stdout:5/134: stat d5/d6 0 2022-01-31T19:44:05.405 INFO:tasks.workunit.client.0.smithi167.stdout:5/135: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:05.406 INFO:tasks.workunit.client.0.smithi167.stdout:5/136: read d5/d6/f1c [1242609,45151] 0 2022-01-31T19:44:05.407 INFO:tasks.workunit.client.0.smithi167.stdout:5/137: dread d5/d6/d16/d18/f1f [0,4194304] 0 2022-01-31T19:44:05.409 INFO:tasks.workunit.client.0.smithi167.stdout:5/138: rename d5/d6/f12 to d5/d6/de/f2a 0 2022-01-31T19:44:05.410 INFO:tasks.workunit.client.0.smithi167.stdout:5/139: fdatasync d5/d6/de/f2a 0 2022-01-31T19:44:05.410 INFO:tasks.workunit.client.0.smithi167.stdout:5/140: creat d5/d6/d16/f2b x:0 0 0 2022-01-31T19:44:05.411 INFO:tasks.workunit.client.0.smithi167.stdout:5/141: dread - d5/d6/d16/f26 zero size 2022-01-31T19:44:05.427 INFO:tasks.workunit.client.0.smithi167.stdout:5/142: dwrite d5/f9 [4194304,4194304] 0 2022-01-31T19:44:05.428 INFO:tasks.workunit.client.0.smithi167.stdout:5/143: write d5/f8 [51227,75637] 0 2022-01-31T19:44:05.430 INFO:tasks.workunit.client.0.smithi167.stdout:5/144: creat d5/d6/d16/d18/f2c x:0 0 0 2022-01-31T19:44:05.434 INFO:tasks.workunit.client.0.smithi167.stdout:5/145: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:05.435 INFO:tasks.workunit.client.0.smithi167.stdout:5/146: mkdir d5/d6/de/d22/d2d 0 2022-01-31T19:44:05.435 INFO:tasks.workunit.client.0.smithi167.stdout:5/147: write d5/f9 [1373215,107674] 0 2022-01-31T19:44:05.436 INFO:tasks.workunit.client.0.smithi167.stdout:5/148: creat d5/d6/d16/d18/f2e x:0 0 0 2022-01-31T19:44:05.437 INFO:tasks.workunit.client.0.smithi167.stdout:5/149: mknod d5/d6/c2f 0 2022-01-31T19:44:05.438 INFO:tasks.workunit.client.0.smithi167.stdout:5/150: truncate d5/d6/d16/d18/f1f 706222 0 2022-01-31T19:44:05.439 INFO:tasks.workunit.client.0.smithi167.stdout:5/151: mknod d5/d6/d16/d18/c30 0 2022-01-31T19:44:05.441 INFO:tasks.workunit.client.0.smithi167.stdout:5/152: unlink d5/d6/d16/d24/c27 0 2022-01-31T19:44:05.442 INFO:tasks.workunit.client.0.smithi167.stdout:5/153: write d5/f9 [2747872,98430] 0 2022-01-31T19:44:05.457 INFO:tasks.workunit.client.0.smithi167.stdout:5/154: dread d5/d6/de/f2a [0,4194304] 0 2022-01-31T19:44:05.458 INFO:tasks.workunit.client.0.smithi167.stdout:5/155: unlink d5/d6/de/f1e 0 2022-01-31T19:44:05.459 INFO:tasks.workunit.client.0.smithi167.stdout:5/156: chown d5/d6/d16/f26 14025 1 2022-01-31T19:44:05.459 INFO:tasks.workunit.client.0.smithi167.stdout:5/157: truncate d5/d6/d16/d18/f2e 358787 0 2022-01-31T19:44:05.493 INFO:tasks.workunit.client.0.smithi167.stdout:1/95: sync 2022-01-31T19:44:05.494 INFO:tasks.workunit.client.0.smithi167.stdout:6/30: sync 2022-01-31T19:44:05.494 INFO:tasks.workunit.client.0.smithi167.stdout:0/32: sync 2022-01-31T19:44:05.494 INFO:tasks.workunit.client.0.smithi167.stdout:9/50: sync 2022-01-31T19:44:05.495 INFO:tasks.workunit.client.0.smithi167.stdout:4/53: sync 2022-01-31T19:44:05.495 INFO:tasks.workunit.client.0.smithi167.stdout:1/96: creat d0/d11/f27 x:0 0 0 2022-01-31T19:44:05.496 INFO:tasks.workunit.client.0.smithi167.stdout:0/33: rename f4 to fc 0 2022-01-31T19:44:05.496 INFO:tasks.workunit.client.0.smithi167.stdout:9/51: truncate f5 4203893 0 2022-01-31T19:44:05.497 INFO:tasks.workunit.client.0.smithi167.stdout:4/54: mknod d0/d6/dc/dd/cf 0 2022-01-31T19:44:05.497 INFO:tasks.workunit.client.0.smithi167.stdout:4/55: dread - d0/d6/d7/fb zero size 2022-01-31T19:44:05.498 INFO:tasks.workunit.client.0.smithi167.stdout:1/97: rename d0/fe to d0/d10/d20/f28 0 2022-01-31T19:44:05.499 INFO:tasks.workunit.client.0.smithi167.stdout:1/98: fdatasync d0/f6 0 2022-01-31T19:44:05.501 INFO:tasks.workunit.client.0.smithi167.stdout:0/34: mknod cd 0 2022-01-31T19:44:05.502 INFO:tasks.workunit.client.0.smithi167.stdout:0/35: creat fe x:0 0 0 2022-01-31T19:44:05.502 INFO:tasks.workunit.client.0.smithi167.stdout:0/36: creat ff x:0 0 0 2022-01-31T19:44:05.502 INFO:tasks.workunit.client.0.smithi167.stdout:0/37: creat f10 x:0 0 0 2022-01-31T19:44:05.502 INFO:tasks.workunit.client.0.smithi167.stdout:0/38: write ff [16713,41762] 0 2022-01-31T19:44:05.503 INFO:tasks.workunit.client.0.smithi167.stdout:4/56: symlink d0/d6/dc/l10 0 2022-01-31T19:44:05.503 INFO:tasks.workunit.client.0.smithi167.stdout:4/57: dread - d0/d6/dc/dd/fe zero size 2022-01-31T19:44:05.504 INFO:tasks.workunit.client.0.smithi167.stdout:1/99: creat d0/d10/d18/d16/d1f/f29 x:0 0 0 2022-01-31T19:44:05.506 INFO:tasks.workunit.client.0.smithi167.stdout:1/100: read d0/f6 [4080859,80079] 0 2022-01-31T19:44:05.509 INFO:tasks.workunit.client.0.smithi167.stdout:0/39: rename c7 to c11 0 2022-01-31T19:44:05.510 INFO:tasks.workunit.client.0.smithi167.stdout:0/40: unlink c11 0 2022-01-31T19:44:05.510 INFO:tasks.workunit.client.0.smithi167.stdout:0/41: chown cb 137980 1 2022-01-31T19:44:05.510 INFO:tasks.workunit.client.0.smithi167.stdout:0/42: write f10 [362033,123882] 0 2022-01-31T19:44:05.511 INFO:tasks.workunit.client.0.smithi167.stdout:0/43: write f10 [544536,126331] 0 2022-01-31T19:44:05.511 INFO:tasks.workunit.client.0.smithi167.stdout:4/58: rename d0/d6/d7/fb to d0/d6/dc/f11 0 2022-01-31T19:44:05.512 INFO:tasks.workunit.client.0.smithi167.stdout:6/31: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:05.512 INFO:tasks.workunit.client.0.smithi167.stdout:4/59: chown d0 13553177 1 2022-01-31T19:44:05.512 INFO:tasks.workunit.client.0.smithi167.stdout:4/60: getdents d0 0 2022-01-31T19:44:05.512 INFO:tasks.workunit.client.0.smithi167.stdout:6/32: write d8/fb [1006155,91692] 0 2022-01-31T19:44:05.517 INFO:tasks.workunit.client.0.smithi167.stdout:6/33: symlink d8/lc 0 2022-01-31T19:44:05.517 INFO:tasks.workunit.client.0.smithi167.stdout:6/34: creat d8/fd x:0 0 0 2022-01-31T19:44:05.529 INFO:tasks.workunit.client.0.smithi167.stdout:1/101: dwrite d0/d11/f27 [0,4194304] 0 2022-01-31T19:44:05.530 INFO:tasks.workunit.client.0.smithi167.stdout:1/102: stat d0/d10/f1a 0 2022-01-31T19:44:05.530 INFO:tasks.workunit.client.0.smithi167.stdout:4/61: dwrite d0/d6/dc/dd/fe [0,4194304] 0 2022-01-31T19:44:05.532 INFO:tasks.workunit.client.0.smithi167.stdout:6/35: getdents d8 0 2022-01-31T19:44:05.538 INFO:tasks.workunit.client.0.smithi167.stdout:6/36: symlink d8/le 0 2022-01-31T19:44:05.540 INFO:tasks.workunit.client.0.smithi167.stdout:1/103: rmdir d0 39 2022-01-31T19:44:05.541 INFO:tasks.workunit.client.0.smithi167.stdout:6/37: mknod d8/cf 0 2022-01-31T19:44:05.541 INFO:tasks.workunit.client.0.smithi167.stdout:6/38: fsync f3 0 2022-01-31T19:44:05.542 INFO:tasks.workunit.client.0.smithi167.stdout:1/104: rename d0/c9 to d0/d10/d18/d16/c2a 0 2022-01-31T19:44:05.543 INFO:tasks.workunit.client.0.smithi167.stdout:1/105: write d0/fc [3256439,126071] 0 2022-01-31T19:44:05.543 INFO:tasks.workunit.client.0.smithi167.stdout:1/106: chown d0/d10/d18/d16/c2a 2870272 1 2022-01-31T19:44:05.544 INFO:tasks.workunit.client.0.smithi167.stdout:6/39: mknod d8/c10 0 2022-01-31T19:44:05.546 INFO:tasks.workunit.client.0.smithi167.stdout:1/107: creat d0/d10/d18/f2b x:0 0 0 2022-01-31T19:44:05.547 INFO:tasks.workunit.client.0.smithi167.stdout:6/40: creat d8/f11 x:0 0 0 2022-01-31T19:44:05.547 INFO:tasks.workunit.client.0.smithi167.stdout:6/41: truncate d8/fd 522960 0 2022-01-31T19:44:05.547 INFO:tasks.workunit.client.0.smithi167.stdout:6/42: chown d8 3492 1 2022-01-31T19:44:05.548 INFO:tasks.workunit.client.0.smithi167.stdout:0/44: dwrite ff [0,4194304] 0 2022-01-31T19:44:05.550 INFO:tasks.workunit.client.0.smithi167.stdout:0/45: symlink l12 0 2022-01-31T19:44:05.551 INFO:tasks.workunit.client.0.smithi167.stdout:4/62: dwrite d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:05.557 INFO:tasks.workunit.client.0.smithi167.stdout:4/63: write d0/d6/d7/f8 [2461457,91065] 0 2022-01-31T19:44:05.561 INFO:tasks.workunit.client.0.smithi167.stdout:1/108: dwrite d0/f1b [0,4194304] 0 2022-01-31T19:44:05.563 INFO:tasks.workunit.client.0.smithi167.stdout:1/109: write d0/f1 [2206345,38745] 0 2022-01-31T19:44:05.563 INFO:tasks.workunit.client.0.smithi167.stdout:1/110: chown d0/ca 59 1 2022-01-31T19:44:05.567 INFO:tasks.workunit.client.0.smithi167.stdout:6/43: rmdir d8 39 2022-01-31T19:44:05.568 INFO:tasks.workunit.client.0.smithi167.stdout:6/44: truncate f3 454643 0 2022-01-31T19:44:05.568 INFO:tasks.workunit.client.0.smithi167.stdout:6/45: write d8/fb [1238609,27737] 0 2022-01-31T19:44:05.568 INFO:tasks.workunit.client.0.smithi167.stdout:6/46: chown c7 1236662583 1 2022-01-31T19:44:05.569 INFO:tasks.workunit.client.0.smithi167.stdout:1/111: mknod d0/d10/d20/c2c 0 2022-01-31T19:44:05.569 INFO:tasks.workunit.client.0.smithi167.stdout:6/47: symlink d8/l12 0 2022-01-31T19:44:05.570 INFO:tasks.workunit.client.0.smithi167.stdout:4/64: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:05.570 INFO:tasks.workunit.client.0.smithi167.stdout:1/112: symlink d0/d10/d18/d16/l2d 0 2022-01-31T19:44:05.571 INFO:tasks.workunit.client.0.smithi167.stdout:6/48: dread d8/fd [0,4194304] 0 2022-01-31T19:44:05.571 INFO:tasks.workunit.client.0.smithi167.stdout:6/49: write d8/f11 [1005556,10591] 0 2022-01-31T19:44:05.571 INFO:tasks.workunit.client.0.smithi167.stdout:1/113: link d0/f5 d0/d10/d20/f2e 0 2022-01-31T19:44:05.572 INFO:tasks.workunit.client.0.smithi167.stdout:6/50: symlink d8/l13 0 2022-01-31T19:44:05.573 INFO:tasks.workunit.client.0.smithi167.stdout:4/65: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:05.573 INFO:tasks.workunit.client.0.smithi167.stdout:4/66: rename d0/d6/d7 to d0/d6/d7/d12 22 2022-01-31T19:44:05.574 INFO:tasks.workunit.client.0.smithi167.stdout:4/67: read - d0/d6/dc/f11 zero size 2022-01-31T19:44:05.574 INFO:tasks.workunit.client.0.smithi167.stdout:4/68: getdents d0/d6/dc/dd 0 2022-01-31T19:44:05.575 INFO:tasks.workunit.client.0.smithi167.stdout:4/69: chown d0/d6/dc/dd 108984 1 2022-01-31T19:44:05.575 INFO:tasks.workunit.client.0.smithi167.stdout:4/70: chown d0/d6/dc 1822248604 1 2022-01-31T19:44:05.576 INFO:tasks.workunit.client.0.smithi167.stdout:4/71: creat d0/d6/dc/f13 x:0 0 0 2022-01-31T19:44:05.577 INFO:tasks.workunit.client.0.smithi167.stdout:4/72: creat d0/d6/d7/f14 x:0 0 0 2022-01-31T19:44:05.577 INFO:tasks.workunit.client.0.smithi167.stdout:4/73: chown d0/l1 13556 1 2022-01-31T19:44:05.577 INFO:tasks.workunit.client.0.smithi167.stdout:4/74: write d0/d6/dc/f11 [395279,80798] 0 2022-01-31T19:44:05.577 INFO:tasks.workunit.client.0.smithi167.stdout:4/75: readlink d0/d6/dc/l10 0 2022-01-31T19:44:05.578 INFO:tasks.workunit.client.0.smithi167.stdout:0/46: dwrite f10 [0,4194304] 0 2022-01-31T19:44:05.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/76: unlink d0/l9 0 2022-01-31T19:44:05.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/77: read - d0/d6/dc/f13 zero size 2022-01-31T19:44:05.584 INFO:tasks.workunit.client.0.smithi167.stdout:6/51: getdents d8 0 2022-01-31T19:44:05.585 INFO:tasks.workunit.client.0.smithi167.stdout:4/78: mknod d0/d6/c15 0 2022-01-31T19:44:05.585 INFO:tasks.workunit.client.0.smithi167.stdout:4/79: chown d0 13188363 1 2022-01-31T19:44:05.585 INFO:tasks.workunit.client.0.smithi167.stdout:4/80: write d0/d6/d7/f14 [113504,41627] 0 2022-01-31T19:44:05.586 INFO:tasks.workunit.client.0.smithi167.stdout:6/52: creat d8/f14 x:0 0 0 2022-01-31T19:44:05.587 INFO:tasks.workunit.client.0.smithi167.stdout:4/81: write d0/d6/d7/f8 [4019214,37294] 0 2022-01-31T19:44:05.587 INFO:tasks.workunit.client.0.smithi167.stdout:4/82: fdatasync d0/d6/dc/f13 0 2022-01-31T19:44:05.588 INFO:tasks.workunit.client.0.smithi167.stdout:1/114: dwrite d0/f6 [0,4194304] 0 2022-01-31T19:44:05.588 INFO:tasks.workunit.client.0.smithi167.stdout:4/83: creat d0/d6/f16 x:0 0 0 2022-01-31T19:44:05.588 INFO:tasks.workunit.client.0.smithi167.stdout:4/84: creat d0/d6/dc/f17 x:0 0 0 2022-01-31T19:44:05.588 INFO:tasks.workunit.client.0.smithi167.stdout:6/53: truncate d8/f9 599546 0 2022-01-31T19:44:05.592 INFO:tasks.workunit.client.0.smithi167.stdout:6/54: dread d8/fd [0,4194304] 0 2022-01-31T19:44:05.592 INFO:tasks.workunit.client.0.smithi167.stdout:6/55: dread d8/fd [0,4194304] 0 2022-01-31T19:44:05.597 INFO:tasks.workunit.client.0.smithi167.stdout:0/47: dwrite fe [0,4194304] 0 2022-01-31T19:44:05.597 INFO:tasks.workunit.client.0.smithi167.stdout:0/48: creat f13 x:0 0 0 2022-01-31T19:44:05.604 INFO:tasks.workunit.client.0.smithi167.stdout:1/115: write d0/fd [726743,69580] 0 2022-01-31T19:44:05.605 INFO:tasks.workunit.client.0.smithi167.stdout:4/85: getdents d0/d6 0 2022-01-31T19:44:05.605 INFO:tasks.workunit.client.0.smithi167.stdout:4/86: write d0/d6/f16 [137600,60912] 0 2022-01-31T19:44:05.605 INFO:tasks.workunit.client.0.smithi167.stdout:4/87: getdents d0/d6/dc/dd 0 2022-01-31T19:44:05.607 INFO:tasks.workunit.client.0.smithi167.stdout:1/116: mknod d0/d11/d24/c2f 0 2022-01-31T19:44:05.613 INFO:tasks.workunit.client.0.smithi167.stdout:6/56: dwrite f3 [0,4194304] 0 2022-01-31T19:44:05.615 INFO:tasks.workunit.client.0.smithi167.stdout:6/57: chown d8/f11 0 1 2022-01-31T19:44:05.616 INFO:tasks.workunit.client.0.smithi167.stdout:6/58: stat d8/f9 0 2022-01-31T19:44:05.616 INFO:tasks.workunit.client.0.smithi167.stdout:6/59: creat d8/f15 x:0 0 0 2022-01-31T19:44:05.616 INFO:tasks.workunit.client.0.smithi167.stdout:6/60: chown d8/f9 313 1 2022-01-31T19:44:05.624 INFO:tasks.workunit.client.0.smithi167.stdout:0/49: dwrite f13 [0,4194304] 0 2022-01-31T19:44:05.635 INFO:tasks.workunit.client.0.smithi167.stdout:0/50: dread f10 [0,4194304] 0 2022-01-31T19:44:05.635 INFO:tasks.workunit.client.0.smithi167.stdout:1/117: dwrite d0/d11/f27 [0,4194304] 0 2022-01-31T19:44:05.635 INFO:tasks.workunit.client.0.smithi167.stdout:4/88: dwrite d0/d6/f16 [0,4194304] 0 2022-01-31T19:44:05.636 INFO:tasks.workunit.client.0.smithi167.stdout:4/89: dread - d0/d6/dc/f13 zero size 2022-01-31T19:44:05.636 INFO:tasks.workunit.client.0.smithi167.stdout:0/51: creat f14 x:0 0 0 2022-01-31T19:44:05.640 INFO:tasks.workunit.client.0.smithi167.stdout:6/61: dread d8/fb [0,4194304] 0 2022-01-31T19:44:05.650 INFO:tasks.workunit.client.0.smithi167.stdout:0/52: dread ff [0,4194304] 0 2022-01-31T19:44:05.650 INFO:tasks.workunit.client.0.smithi167.stdout:1/118: creat d0/f30 x:0 0 0 2022-01-31T19:44:05.651 INFO:tasks.workunit.client.0.smithi167.stdout:7/115: sync 2022-01-31T19:44:05.651 INFO:tasks.workunit.client.0.smithi167.stdout:3/115: sync 2022-01-31T19:44:05.653 INFO:tasks.workunit.client.0.smithi167.stdout:0/53: rename f9 to f15 0 2022-01-31T19:44:05.653 INFO:tasks.workunit.client.0.smithi167.stdout:0/54: chown f8 14520 1 2022-01-31T19:44:05.655 INFO:tasks.workunit.client.0.smithi167.stdout:1/119: mknod d0/d11/d24/c31 0 2022-01-31T19:44:05.655 INFO:tasks.workunit.client.0.smithi167.stdout:4/90: dwrite d0/d6/dc/f11 [0,4194304] 0 2022-01-31T19:44:05.655 INFO:tasks.workunit.client.0.smithi167.stdout:4/91: stat d0/d6/dc/f17 0 2022-01-31T19:44:05.655 INFO:tasks.workunit.client.0.smithi167.stdout:4/92: creat d0/d6/d7/f18 x:0 0 0 2022-01-31T19:44:05.655 INFO:tasks.workunit.client.0.smithi167.stdout:4/93: stat d0/l2 0 2022-01-31T19:44:05.656 INFO:tasks.workunit.client.0.smithi167.stdout:0/55: read f15 [282074,75523] 0 2022-01-31T19:44:05.656 INFO:tasks.workunit.client.0.smithi167.stdout:6/62: link d8/f11 d8/f16 0 2022-01-31T19:44:05.659 INFO:tasks.workunit.client.0.smithi167.stdout:3/116: unlink d9/ff 0 2022-01-31T19:44:05.659 INFO:tasks.workunit.client.0.smithi167.stdout:3/117: stat d9/c12 0 2022-01-31T19:44:05.659 INFO:tasks.workunit.client.0.smithi167.stdout:1/120: mknod d0/d10/d20/c32 0 2022-01-31T19:44:05.660 INFO:tasks.workunit.client.0.smithi167.stdout:1/121: chown d0/d11/d24/c31 4597 1 2022-01-31T19:44:05.662 INFO:tasks.workunit.client.0.smithi167.stdout:7/116: dread d8/d1b/f25 [0,4194304] 0 2022-01-31T19:44:05.678 INFO:tasks.workunit.client.0.smithi167.stdout:3/118: dwrite d9/f10 [0,4194304] 0 2022-01-31T19:44:05.679 INFO:tasks.workunit.client.0.smithi167.stdout:3/119: chown d9/fc 1549852 1 2022-01-31T19:44:05.679 INFO:tasks.workunit.client.0.smithi167.stdout:3/120: truncate d9/f18 865566 0 2022-01-31T19:44:05.695 INFO:tasks.workunit.client.0.smithi167.stdout:4/94: rename d0/d6/dc/f13 to d0/d6/dc/f19 0 2022-01-31T19:44:05.695 INFO:tasks.workunit.client.0.smithi167.stdout:2/57: sync 2022-01-31T19:44:05.699 INFO:tasks.workunit.client.0.smithi167.stdout:1/122: unlink d0/d10/l12 0 2022-01-31T19:44:05.700 INFO:tasks.workunit.client.0.smithi167.stdout:7/117: rename d8/d1b to d8/d2c 0 2022-01-31T19:44:05.701 INFO:tasks.workunit.client.0.smithi167.stdout:7/118: write f7 [1199702,32937] 0 2022-01-31T19:44:05.702 INFO:tasks.workunit.client.0.smithi167.stdout:4/95: mkdir d0/d6/dc/dd/d1a 0 2022-01-31T19:44:05.702 INFO:tasks.workunit.client.0.smithi167.stdout:4/96: dread - d0/d6/dc/f19 zero size 2022-01-31T19:44:05.703 INFO:tasks.workunit.client.0.smithi167.stdout:2/58: mknod dd/c13 0 2022-01-31T19:44:05.704 INFO:tasks.workunit.client.0.smithi167.stdout:6/63: symlink d8/l17 0 2022-01-31T19:44:05.705 INFO:tasks.workunit.client.0.smithi167.stdout:1/123: mkdir d0/d11/d24/d33 0 2022-01-31T19:44:05.708 INFO:tasks.workunit.client.0.smithi167.stdout:3/121: getdents d9 0 2022-01-31T19:44:05.708 INFO:tasks.workunit.client.0.smithi167.stdout:3/122: chown d9/l19 3026 1 2022-01-31T19:44:05.708 INFO:tasks.workunit.client.0.smithi167.stdout:7/119: symlink d8/dc/d18/l2d 0 2022-01-31T19:44:05.709 INFO:tasks.workunit.client.0.smithi167.stdout:2/59: truncate f8 238230 0 2022-01-31T19:44:05.711 INFO:tasks.workunit.client.0.smithi167.stdout:6/64: mknod d8/c18 0 2022-01-31T19:44:05.712 INFO:tasks.workunit.client.0.smithi167.stdout:6/65: truncate d8/fa 581910 0 2022-01-31T19:44:05.714 INFO:tasks.workunit.client.0.smithi167.stdout:6/66: chown d8/lc 928404996 1 2022-01-31T19:44:05.715 INFO:tasks.workunit.client.0.smithi167.stdout:7/120: dread d8/dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:05.716 INFO:tasks.workunit.client.0.smithi167.stdout:6/67: creat d8/f19 x:0 0 0 2022-01-31T19:44:05.718 INFO:tasks.workunit.client.0.smithi167.stdout:7/121: rmdir d8/d2c 39 2022-01-31T19:44:05.718 INFO:tasks.workunit.client.0.smithi167.stdout:7/122: write d8/d2c/f25 [4575225,56984] 0 2022-01-31T19:44:05.724 INFO:tasks.workunit.client.0.smithi167.stdout:6/68: write d8/fa [1406399,71727] 0 2022-01-31T19:44:05.726 INFO:tasks.workunit.client.0.smithi167.stdout:6/69: creat d8/f1a x:0 0 0 2022-01-31T19:44:05.727 INFO:tasks.workunit.client.0.smithi167.stdout:6/70: mknod d8/c1b 0 2022-01-31T19:44:05.732 INFO:tasks.workunit.client.0.smithi167.stdout:1/124: dwrite d0/f5 [4194304,4194304] 0 2022-01-31T19:44:05.735 INFO:tasks.workunit.client.0.smithi167.stdout:1/125: unlink d0/c1c 0 2022-01-31T19:44:05.735 INFO:tasks.workunit.client.0.smithi167.stdout:1/126: chown d0/f1b 377337509 1 2022-01-31T19:44:05.735 INFO:tasks.workunit.client.0.smithi167.stdout:1/127: dread - d0/f30 zero size 2022-01-31T19:44:05.736 INFO:tasks.workunit.client.0.smithi167.stdout:1/128: dread - d0/d10/d18/d16/d1f/f29 zero size 2022-01-31T19:44:05.737 INFO:tasks.workunit.client.0.smithi167.stdout:3/123: dwrite d9/f1c [0,4194304] 0 2022-01-31T19:44:05.737 INFO:tasks.workunit.client.0.smithi167.stdout:3/124: write f1 [3068600,1488] 0 2022-01-31T19:44:05.738 INFO:tasks.workunit.client.0.smithi167.stdout:7/123: dwrite d8/dc/d18/d24/f29 [0,4194304] 0 2022-01-31T19:44:05.738 INFO:tasks.workunit.client.0.smithi167.stdout:1/129: creat d0/d10/f34 x:0 0 0 2022-01-31T19:44:05.744 INFO:tasks.workunit.client.0.smithi167.stdout:6/71: truncate d8/fa 256322 0 2022-01-31T19:44:05.748 INFO:tasks.workunit.client.0.smithi167.stdout:6/72: creat d8/f1c x:0 0 0 2022-01-31T19:44:05.748 INFO:tasks.workunit.client.0.smithi167.stdout:6/73: chown d8/f19 178 1 2022-01-31T19:44:05.749 INFO:tasks.workunit.client.0.smithi167.stdout:6/74: creat d8/f1d x:0 0 0 2022-01-31T19:44:05.750 INFO:tasks.workunit.client.0.smithi167.stdout:6/75: chown d8/le 827 1 2022-01-31T19:44:05.751 INFO:tasks.workunit.client.0.smithi167.stdout:6/76: truncate d8/f9 458022 0 2022-01-31T19:44:05.753 INFO:tasks.workunit.client.0.smithi167.stdout:6/77: creat d8/f1e x:0 0 0 2022-01-31T19:44:05.753 INFO:tasks.workunit.client.0.smithi167.stdout:6/78: truncate d8/f1c 5702 0 2022-01-31T19:44:05.754 INFO:tasks.workunit.client.0.smithi167.stdout:6/79: creat d8/f1f x:0 0 0 2022-01-31T19:44:05.755 INFO:tasks.workunit.client.0.smithi167.stdout:6/80: chown d8/f11 406 1 2022-01-31T19:44:05.755 INFO:tasks.workunit.client.0.smithi167.stdout:6/81: chown d8/le 1768233 1 2022-01-31T19:44:05.755 INFO:tasks.workunit.client.0.smithi167.stdout:6/82: fsync d8/fd 0 2022-01-31T19:44:05.756 INFO:tasks.workunit.client.0.smithi167.stdout:6/83: mkdir d8/d20 0 2022-01-31T19:44:05.756 INFO:tasks.workunit.client.0.smithi167.stdout:3/125: dwrite d9/f15 [0,4194304] 0 2022-01-31T19:44:05.758 INFO:tasks.workunit.client.0.smithi167.stdout:7/124: dwrite d8/dc/f21 [0,4194304] 0 2022-01-31T19:44:05.759 INFO:tasks.workunit.client.0.smithi167.stdout:1/130: truncate d0/f5 775742 0 2022-01-31T19:44:05.759 INFO:tasks.workunit.client.0.smithi167.stdout:1/131: readlink d0/l8 0 2022-01-31T19:44:05.769 INFO:tasks.workunit.client.0.smithi167.stdout:7/125: truncate d8/dc/d19/f1a 491452 0 2022-01-31T19:44:05.771 INFO:tasks.workunit.client.0.smithi167.stdout:1/132: mknod d0/d11/c35 0 2022-01-31T19:44:05.771 INFO:tasks.workunit.client.0.smithi167.stdout:6/84: rename d8/f1d to d8/d20/f21 0 2022-01-31T19:44:05.773 INFO:tasks.workunit.client.0.smithi167.stdout:7/126: mkdir d8/d2e 0 2022-01-31T19:44:05.773 INFO:tasks.workunit.client.0.smithi167.stdout:7/127: chown d8 29292408 1 2022-01-31T19:44:05.774 INFO:tasks.workunit.client.0.smithi167.stdout:6/85: rmdir d8/d20 39 2022-01-31T19:44:05.774 INFO:tasks.workunit.client.0.smithi167.stdout:3/126: dwrite f1 [0,4194304] 0 2022-01-31T19:44:05.775 INFO:tasks.workunit.client.0.smithi167.stdout:1/133: write d0/f6 [2194969,49747] 0 2022-01-31T19:44:05.776 INFO:tasks.workunit.client.0.smithi167.stdout:7/128: creat d8/d2e/f2f x:0 0 0 2022-01-31T19:44:05.777 INFO:tasks.workunit.client.0.smithi167.stdout:6/86: creat d8/d20/f22 x:0 0 0 2022-01-31T19:44:05.778 INFO:tasks.workunit.client.0.smithi167.stdout:6/87: stat d8/f15 0 2022-01-31T19:44:05.778 INFO:tasks.workunit.client.0.smithi167.stdout:7/129: dread d8/fa [0,4194304] 0 2022-01-31T19:44:05.779 INFO:tasks.workunit.client.0.smithi167.stdout:3/127: mknod d9/d1b/c1f 0 2022-01-31T19:44:05.781 INFO:tasks.workunit.client.0.smithi167.stdout:1/134: symlink d0/d11/d24/d33/l36 0 2022-01-31T19:44:05.784 INFO:tasks.workunit.client.0.smithi167.stdout:7/130: symlink d8/dc/d12/l30 0 2022-01-31T19:44:05.785 INFO:tasks.workunit.client.0.smithi167.stdout:3/128: mknod d9/d1b/c20 0 2022-01-31T19:44:05.785 INFO:tasks.workunit.client.0.smithi167.stdout:3/129: truncate f7 3767766 0 2022-01-31T19:44:05.785 INFO:tasks.workunit.client.0.smithi167.stdout:3/130: chown l4 155430 1 2022-01-31T19:44:05.787 INFO:tasks.workunit.client.0.smithi167.stdout:5/158: sync 2022-01-31T19:44:05.787 INFO:tasks.workunit.client.0.smithi167.stdout:8/164: sync 2022-01-31T19:44:05.788 INFO:tasks.workunit.client.0.smithi167.stdout:8/165: stat d6/db 0 2022-01-31T19:44:05.788 INFO:tasks.workunit.client.0.smithi167.stdout:8/166: chown d6/db 9916 1 2022-01-31T19:44:05.788 INFO:tasks.workunit.client.0.smithi167.stdout:5/159: write d5/d6/de/d22/f25 [392079,121927] 0 2022-01-31T19:44:05.789 INFO:tasks.workunit.client.0.smithi167.stdout:1/135: rename d0/d10/f1a to d0/d10/d20/f37 0 2022-01-31T19:44:05.791 INFO:tasks.workunit.client.0.smithi167.stdout:6/88: getdents d8/d20 0 2022-01-31T19:44:05.792 INFO:tasks.workunit.client.0.smithi167.stdout:6/89: chown d8/f14 761361162 1 2022-01-31T19:44:05.793 INFO:tasks.workunit.client.0.smithi167.stdout:7/131: mkdir d8/d2e/d31 0 2022-01-31T19:44:05.797 INFO:tasks.workunit.client.0.smithi167.stdout:8/167: rmdir d6/db 39 2022-01-31T19:44:05.798 INFO:tasks.workunit.client.0.smithi167.stdout:5/160: rmdir d5/d6/de/d22/d2d 0 2022-01-31T19:44:05.799 INFO:tasks.workunit.client.0.smithi167.stdout:1/136: dread d0/f1 [0,4194304] 0 2022-01-31T19:44:05.800 INFO:tasks.workunit.client.0.smithi167.stdout:7/132: rename c3 to d8/dc/d18/d24/c32 0 2022-01-31T19:44:05.801 INFO:tasks.workunit.client.0.smithi167.stdout:5/161: truncate d5/d10/f17 2003426 0 2022-01-31T19:44:05.802 INFO:tasks.workunit.client.0.smithi167.stdout:5/162: fdatasync d5/f9 0 2022-01-31T19:44:05.802 INFO:tasks.workunit.client.0.smithi167.stdout:1/137: rmdir d0/d11 39 2022-01-31T19:44:05.803 INFO:tasks.workunit.client.0.smithi167.stdout:5/163: truncate d5/d6/de/f2a 3253784 0 2022-01-31T19:44:05.804 INFO:tasks.workunit.client.0.smithi167.stdout:1/138: rename d0/fd to d0/d10/d18/f38 0 2022-01-31T19:44:05.806 INFO:tasks.workunit.client.0.smithi167.stdout:1/139: mkdir d0/d10/d18/d39 0 2022-01-31T19:44:05.808 INFO:tasks.workunit.client.0.smithi167.stdout:7/133: dread d8/dc/f17 [0,4194304] 0 2022-01-31T19:44:05.808 INFO:tasks.workunit.client.0.smithi167.stdout:7/134: chown f6 3890561 1 2022-01-31T19:44:05.813 INFO:tasks.workunit.client.0.smithi167.stdout:3/131: rename d9/d1b to d9/d21 0 2022-01-31T19:44:05.814 INFO:tasks.workunit.client.0.smithi167.stdout:3/132: write d9/fc [2370931,46882] 0 2022-01-31T19:44:05.816 INFO:tasks.workunit.client.0.smithi167.stdout:3/133: dread d9/d21/f1e [0,4194304] 0 2022-01-31T19:44:05.816 INFO:tasks.workunit.client.0.smithi167.stdout:6/90: dwrite f3 [0,4194304] 0 2022-01-31T19:44:05.817 INFO:tasks.workunit.client.0.smithi167.stdout:6/91: truncate d8/d20/f22 926649 0 2022-01-31T19:44:05.817 INFO:tasks.workunit.client.0.smithi167.stdout:3/134: unlink d9/f15 0 2022-01-31T19:44:05.821 INFO:tasks.workunit.client.0.smithi167.stdout:6/92: symlink d8/l23 0 2022-01-31T19:44:05.823 INFO:tasks.workunit.client.0.smithi167.stdout:6/93: dread d8/fb [0,4194304] 0 2022-01-31T19:44:05.827 INFO:tasks.workunit.client.0.smithi167.stdout:8/168: dwrite d6/db/de/f20 [0,4194304] 0 2022-01-31T19:44:05.828 INFO:tasks.workunit.client.0.smithi167.stdout:8/169: fdatasync d6/d1e/f29 0 2022-01-31T19:44:05.828 INFO:tasks.workunit.client.0.smithi167.stdout:8/170: fdatasync d6/d1e/f22 0 2022-01-31T19:44:05.828 INFO:tasks.workunit.client.0.smithi167.stdout:8/171: chown d6/fa 116 1 2022-01-31T19:44:05.829 INFO:tasks.workunit.client.0.smithi167.stdout:8/172: read d6/db/de/d18/f1b [862455,42320] 0 2022-01-31T19:44:05.829 INFO:tasks.workunit.client.0.smithi167.stdout:1/140: dwrite d0/fc [0,4194304] 0 2022-01-31T19:44:05.831 INFO:tasks.workunit.client.0.smithi167.stdout:8/173: link d6/db/de/f20 d6/db/d2f/f33 0 2022-01-31T19:44:05.832 INFO:tasks.workunit.client.0.smithi167.stdout:8/174: write d6/f8 [1670664,48732] 0 2022-01-31T19:44:05.833 INFO:tasks.workunit.client.0.smithi167.stdout:8/175: readlink l3 0 2022-01-31T19:44:05.833 INFO:tasks.workunit.client.0.smithi167.stdout:8/176: chown d6/f13 0 1 2022-01-31T19:44:05.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/177: truncate d6/db/f2b 3629800 0 2022-01-31T19:44:05.836 INFO:tasks.workunit.client.0.smithi167.stdout:8/178: mknod d6/d1e/d2d/c34 0 2022-01-31T19:44:05.836 INFO:tasks.workunit.client.0.smithi167.stdout:8/179: chown d6/d1e/d2d 13709 1 2022-01-31T19:44:05.837 INFO:tasks.workunit.client.0.smithi167.stdout:8/180: mknod d6/d1e/d2d/c35 0 2022-01-31T19:44:05.839 INFO:tasks.workunit.client.0.smithi167.stdout:8/181: unlink d6/d1e/d2d/c35 0 2022-01-31T19:44:05.840 INFO:tasks.workunit.client.0.smithi167.stdout:6/94: dwrite d8/d20/f21 [0,4194304] 0 2022-01-31T19:44:05.843 INFO:tasks.workunit.client.0.smithi167.stdout:6/95: read d8/fd [501957,104137] 0 2022-01-31T19:44:05.843 INFO:tasks.workunit.client.0.smithi167.stdout:6/96: chown d8/l17 0 1 2022-01-31T19:44:05.844 INFO:tasks.workunit.client.0.smithi167.stdout:5/164: dread d5/d10/f17 [0,4194304] 0 2022-01-31T19:44:05.844 INFO:tasks.workunit.client.0.smithi167.stdout:1/141: rename d0/d10/d18/f38 to d0/f3a 0 2022-01-31T19:44:05.845 INFO:tasks.workunit.client.0.smithi167.stdout:6/97: mknod d8/d20/c24 0 2022-01-31T19:44:05.845 INFO:tasks.workunit.client.0.smithi167.stdout:6/98: write d8/f1f [958690,114380] 0 2022-01-31T19:44:05.849 INFO:tasks.workunit.client.0.smithi167.stdout:3/135: dwrite d9/fc [0,4194304] 0 2022-01-31T19:44:05.849 INFO:tasks.workunit.client.0.smithi167.stdout:3/136: truncate f1 5044595 0 2022-01-31T19:44:05.849 INFO:tasks.workunit.client.0.smithi167.stdout:3/137: readlink d9/le 0 2022-01-31T19:44:05.850 INFO:tasks.workunit.client.0.smithi167.stdout:6/99: fdatasync d8/f1f 0 2022-01-31T19:44:05.850 INFO:tasks.workunit.client.0.smithi167.stdout:6/100: truncate d8/f19 520055 0 2022-01-31T19:44:05.860 INFO:tasks.workunit.client.0.smithi167.stdout:6/101: mknod d8/c25 0 2022-01-31T19:44:05.861 INFO:tasks.workunit.client.0.smithi167.stdout:6/102: chown d8/f19 8 1 2022-01-31T19:44:05.861 INFO:tasks.workunit.client.0.smithi167.stdout:3/138: dread d9/fc [0,4194304] 0 2022-01-31T19:44:05.862 INFO:tasks.workunit.client.0.smithi167.stdout:6/103: read f3 [2917908,21765] 0 2022-01-31T19:44:05.863 INFO:tasks.workunit.client.0.smithi167.stdout:3/139: write f7 [1428314,3210] 0 2022-01-31T19:44:05.863 INFO:tasks.workunit.client.0.smithi167.stdout:6/104: stat d8/c10 0 2022-01-31T19:44:05.864 INFO:tasks.workunit.client.0.smithi167.stdout:1/142: dwrite d0/f1 [0,4194304] 0 2022-01-31T19:44:05.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/140: creat d9/d21/f22 x:0 0 0 2022-01-31T19:44:05.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/141: unlink l4 0 2022-01-31T19:44:05.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/142: truncate d9/d21/f1e 488457 0 2022-01-31T19:44:05.866 INFO:tasks.workunit.client.0.smithi167.stdout:6/105: creat d8/d20/f26 x:0 0 0 2022-01-31T19:44:05.870 INFO:tasks.workunit.client.0.smithi167.stdout:1/143: write d0/f6 [1108084,121350] 0 2022-01-31T19:44:05.870 INFO:tasks.workunit.client.0.smithi167.stdout:1/144: truncate d0/d10/d18/f1e 1132032 0 2022-01-31T19:44:05.871 INFO:tasks.workunit.client.0.smithi167.stdout:1/145: unlink d0/d10/c1d 0 2022-01-31T19:44:05.872 INFO:tasks.workunit.client.0.smithi167.stdout:6/106: mknod d8/d20/c27 0 2022-01-31T19:44:05.875 INFO:tasks.workunit.client.0.smithi167.stdout:8/182: dwrite d6/f8 [4194304,4194304] 0 2022-01-31T19:44:05.876 INFO:tasks.workunit.client.0.smithi167.stdout:1/146: rename d0/ca to d0/d11/d24/c3b 0 2022-01-31T19:44:05.877 INFO:tasks.workunit.client.0.smithi167.stdout:3/143: getdents d9 0 2022-01-31T19:44:05.878 INFO:tasks.workunit.client.0.smithi167.stdout:1/147: mknod d0/d10/d20/c3c 0 2022-01-31T19:44:05.880 INFO:tasks.workunit.client.0.smithi167.stdout:3/144: rename d9/f1c to d9/d21/f23 0 2022-01-31T19:44:05.881 INFO:tasks.workunit.client.0.smithi167.stdout:1/148: creat d0/d10/d18/d39/f3d x:0 0 0 2022-01-31T19:44:05.881 INFO:tasks.workunit.client.0.smithi167.stdout:1/149: write d0/d10/d18/d16/d1f/f29 [763654,36242] 0 2022-01-31T19:44:05.882 INFO:tasks.workunit.client.0.smithi167.stdout:1/150: creat d0/d10/d18/f3e x:0 0 0 2022-01-31T19:44:05.882 INFO:tasks.workunit.client.0.smithi167.stdout:1/151: write d0/f17 [598471,72745] 0 2022-01-31T19:44:05.882 INFO:tasks.workunit.client.0.smithi167.stdout:1/152: dread - d0/f30 zero size 2022-01-31T19:44:05.882 INFO:tasks.workunit.client.0.smithi167.stdout:1/153: read - d0/f25 zero size 2022-01-31T19:44:05.907 INFO:tasks.workunit.client.0.smithi167.stdout:3/145: dwrite d9/fc [0,4194304] 0 2022-01-31T19:44:05.908 INFO:tasks.workunit.client.0.smithi167.stdout:8/183: dwrite d6/ff [4194304,4194304] 0 2022-01-31T19:44:05.908 INFO:tasks.workunit.client.0.smithi167.stdout:8/184: chown d6/f1d 2 1 2022-01-31T19:44:05.909 INFO:tasks.workunit.client.0.smithi167.stdout:1/154: dwrite d0/d10/d20/f28 [0,4194304] 0 2022-01-31T19:44:05.909 INFO:tasks.workunit.client.0.smithi167.stdout:8/185: write d6/f14 [3386609,114974] 0 2022-01-31T19:44:05.910 INFO:tasks.workunit.client.0.smithi167.stdout:1/155: creat d0/d11/d24/f3f x:0 0 0 2022-01-31T19:44:05.910 INFO:tasks.workunit.client.0.smithi167.stdout:3/146: dread d9/f13 [0,4194304] 0 2022-01-31T19:44:05.911 INFO:tasks.workunit.client.0.smithi167.stdout:8/186: mkdir d6/db/d36 0 2022-01-31T19:44:05.911 INFO:tasks.workunit.client.0.smithi167.stdout:3/147: chown d9/cb 1426183 1 2022-01-31T19:44:05.912 INFO:tasks.workunit.client.0.smithi167.stdout:8/187: link d6/db/l25 d6/db/de/l37 0 2022-01-31T19:44:05.917 INFO:tasks.workunit.client.0.smithi167.stdout:8/188: symlink d6/db/d28/l38 0 2022-01-31T19:44:05.919 INFO:tasks.workunit.client.0.smithi167.stdout:1/156: dread d0/d10/d20/f2e [0,4194304] 0 2022-01-31T19:44:05.922 INFO:tasks.workunit.client.0.smithi167.stdout:1/157: symlink d0/d10/d20/l40 0 2022-01-31T19:44:05.922 INFO:tasks.workunit.client.0.smithi167.stdout:1/158: fdatasync d0/d10/f19 0 2022-01-31T19:44:05.926 INFO:tasks.workunit.client.0.smithi167.stdout:3/148: truncate d9/fc 489281 0 2022-01-31T19:44:05.926 INFO:tasks.workunit.client.0.smithi167.stdout:3/149: creat d9/d21/f24 x:0 0 0 2022-01-31T19:44:05.928 INFO:tasks.workunit.client.0.smithi167.stdout:3/150: truncate d9/f11 1525531 0 2022-01-31T19:44:05.930 INFO:tasks.workunit.client.0.smithi167.stdout:3/151: rename d9/d21/f22 to d9/f25 0 2022-01-31T19:44:05.930 INFO:tasks.workunit.client.0.smithi167.stdout:3/152: dread - d9/f25 zero size 2022-01-31T19:44:05.931 INFO:tasks.workunit.client.0.smithi167.stdout:3/153: mkdir d9/d26 0 2022-01-31T19:44:05.934 INFO:tasks.workunit.client.0.smithi167.stdout:3/154: rename d9/d21/f24 to d9/f27 0 2022-01-31T19:44:05.935 INFO:tasks.workunit.client.0.smithi167.stdout:8/189: dwrite d6/fa [0,4194304] 0 2022-01-31T19:44:05.935 INFO:tasks.workunit.client.0.smithi167.stdout:1/159: dread d0/f6 [0,4194304] 0 2022-01-31T19:44:05.935 INFO:tasks.workunit.client.0.smithi167.stdout:8/190: chown d6/db/c2c 1126 1 2022-01-31T19:44:05.935 INFO:tasks.workunit.client.0.smithi167.stdout:1/160: write d0/d11/d24/f3f [526698,33717] 0 2022-01-31T19:44:05.940 INFO:tasks.workunit.client.0.smithi167.stdout:9/52: sync 2022-01-31T19:44:05.940 INFO:tasks.workunit.client.0.smithi167.stdout:8/191: rename d6/db/c2c to d6/db/d36/c39 0 2022-01-31T19:44:05.942 INFO:tasks.workunit.client.0.smithi167.stdout:9/53: creat dd/fe x:0 0 0 2022-01-31T19:44:05.942 INFO:tasks.workunit.client.0.smithi167.stdout:9/54: dread - f7 zero size 2022-01-31T19:44:05.942 INFO:tasks.workunit.client.0.smithi167.stdout:9/55: creat dd/ff x:0 0 0 2022-01-31T19:44:05.942 INFO:tasks.workunit.client.0.smithi167.stdout:9/56: creat dd/f10 x:0 0 0 2022-01-31T19:44:05.943 INFO:tasks.workunit.client.0.smithi167.stdout:9/57: creat dd/f11 x:0 0 0 2022-01-31T19:44:05.951 INFO:tasks.workunit.client.0.smithi167.stdout:8/192: write d6/db/de/f20 [3173194,110772] 0 2022-01-31T19:44:05.962 INFO:tasks.workunit.client.0.smithi167.stdout:1/161: dwrite d0/fc [0,4194304] 0 2022-01-31T19:44:05.962 INFO:tasks.workunit.client.0.smithi167.stdout:3/155: dwrite d9/d21/f1e [0,4194304] 0 2022-01-31T19:44:05.962 INFO:tasks.workunit.client.0.smithi167.stdout:1/162: write d0/f25 [820698,85066] 0 2022-01-31T19:44:05.962 INFO:tasks.workunit.client.0.smithi167.stdout:1/163: dread - d0/f30 zero size 2022-01-31T19:44:06.038 INFO:tasks.workunit.client.0.smithi167.stdout:8/193: rename d6/db/c31 to d6/db/de/c3a 0 2022-01-31T19:44:06.043 INFO:tasks.workunit.client.0.smithi167.stdout:3/156: link d9/f11 d9/d21/f28 0 2022-01-31T19:44:06.043 INFO:tasks.workunit.client.0.smithi167.stdout:9/58: getdents dd 0 2022-01-31T19:44:06.044 INFO:tasks.workunit.client.0.smithi167.stdout:9/59: dread - f8 zero size 2022-01-31T19:44:06.044 INFO:tasks.workunit.client.0.smithi167.stdout:9/60: write dd/ff [501511,86861] 0 2022-01-31T19:44:06.044 INFO:tasks.workunit.client.0.smithi167.stdout:3/157: write f1 [1339093,47319] 0 2022-01-31T19:44:06.045 INFO:tasks.workunit.client.0.smithi167.stdout:1/164: mknod d0/d10/d18/d16/c41 0 2022-01-31T19:44:06.045 INFO:tasks.workunit.client.0.smithi167.stdout:1/165: chown d0/d10/d18/d16/d1f/f29 5199616 1 2022-01-31T19:44:06.046 INFO:tasks.workunit.client.0.smithi167.stdout:1/166: creat d0/d10/f42 x:0 0 0 2022-01-31T19:44:06.050 INFO:tasks.workunit.client.0.smithi167.stdout:8/194: rename d6/db/d2f/f33 to d6/db/d36/f3b 0 2022-01-31T19:44:06.051 INFO:tasks.workunit.client.0.smithi167.stdout:8/195: creat d6/d1e/f3c x:0 0 0 2022-01-31T19:44:06.051 INFO:tasks.workunit.client.0.smithi167.stdout:9/61: mknod dd/c12 0 2022-01-31T19:44:06.056 INFO:tasks.workunit.client.0.smithi167.stdout:1/167: unlink d0/d11/f27 0 2022-01-31T19:44:06.056 INFO:tasks.workunit.client.0.smithi167.stdout:1/168: fdatasync d0/f25 0 2022-01-31T19:44:06.057 INFO:tasks.workunit.client.0.smithi167.stdout:8/196: mkdir d6/db/de/d3d 0 2022-01-31T19:44:06.058 INFO:tasks.workunit.client.0.smithi167.stdout:8/197: truncate d6/d1e/f22 623415 0 2022-01-31T19:44:06.058 INFO:tasks.workunit.client.0.smithi167.stdout:8/198: readlink d6/db/l25 0 2022-01-31T19:44:06.058 INFO:tasks.workunit.client.0.smithi167.stdout:8/199: creat d6/db/de/f3e x:0 0 0 2022-01-31T19:44:06.060 INFO:tasks.workunit.client.0.smithi167.stdout:9/62: mknod dd/c13 0 2022-01-31T19:44:06.060 INFO:tasks.workunit.client.0.smithi167.stdout:9/63: unlink lb 0 2022-01-31T19:44:06.061 INFO:tasks.workunit.client.0.smithi167.stdout:3/158: dread d9/f18 [0,4194304] 0 2022-01-31T19:44:06.063 INFO:tasks.workunit.client.0.smithi167.stdout:9/64: creat dd/f14 x:0 0 0 2022-01-31T19:44:06.064 INFO:tasks.workunit.client.0.smithi167.stdout:9/65: rename dd/c13 to dd/c15 0 2022-01-31T19:44:06.066 INFO:tasks.workunit.client.0.smithi167.stdout:9/66: creat dd/f16 x:0 0 0 2022-01-31T19:44:06.067 INFO:tasks.workunit.client.0.smithi167.stdout:9/67: mkdir dd/d17 0 2022-01-31T19:44:06.068 INFO:tasks.workunit.client.0.smithi167.stdout:9/68: getdents dd/d17 0 2022-01-31T19:44:06.068 INFO:tasks.workunit.client.0.smithi167.stdout:9/69: creat dd/d17/f18 x:0 0 0 2022-01-31T19:44:06.069 INFO:tasks.workunit.client.0.smithi167.stdout:9/70: write f7 [450084,53544] 0 2022-01-31T19:44:06.069 INFO:tasks.workunit.client.0.smithi167.stdout:9/71: fsync dd/fe 0 2022-01-31T19:44:06.099 INFO:tasks.workunit.client.0.smithi167.stdout:1/169: dwrite d0/d10/d18/d39/f3d [0,4194304] 0 2022-01-31T19:44:06.099 INFO:tasks.workunit.client.0.smithi167.stdout:1/170: dread - d0/f30 zero size 2022-01-31T19:44:06.100 INFO:tasks.workunit.client.0.smithi167.stdout:0/56: sync 2022-01-31T19:44:06.101 INFO:tasks.workunit.client.0.smithi167.stdout:3/159: dwrite d9/d21/f23 [4194304,4194304] 0 2022-01-31T19:44:06.102 INFO:tasks.workunit.client.0.smithi167.stdout:8/200: dwrite d6/db/de/d18/f1b [0,4194304] 0 2022-01-31T19:44:06.103 INFO:tasks.workunit.client.0.smithi167.stdout:9/72: dwrite dd/f16 [0,4194304] 0 2022-01-31T19:44:06.105 INFO:tasks.workunit.client.0.smithi167.stdout:8/201: symlink d6/l3f 0 2022-01-31T19:44:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:05 smithi171 conmon[35325]: cluster 2022-01-31T19:44:04.430653+0000 mgr.smithi171.asyxnc (mgr.24387) 2022-01-31T19:44:06.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:05 smithi171 conmon[35325]: 12 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 365 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 5.8 MiB/s rd, 15 MiB/s wr, 601 op/s 2022-01-31T19:44:06.115 INFO:tasks.workunit.client.0.smithi167.stdout:3/160: mkdir d9/d29 0 2022-01-31T19:44:06.116 INFO:tasks.workunit.client.0.smithi167.stdout:1/171: truncate d0/d10/d18/d39/f3d 4042306 0 2022-01-31T19:44:06.123 INFO:tasks.workunit.client.0.smithi167.stdout:8/202: rename d6/f12 to d6/d1e/d2d/f40 0 2022-01-31T19:44:06.127 INFO:tasks.workunit.client.0.smithi167.stdout:8/203: dread d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:06.127 INFO:tasks.workunit.client.0.smithi167.stdout:8/204: creat d6/d1e/f41 x:0 0 0 2022-01-31T19:44:06.129 INFO:tasks.workunit.client.0.smithi167.stdout:8/205: symlink d6/db/d36/l42 0 2022-01-31T19:44:06.130 INFO:tasks.workunit.client.0.smithi167.stdout:8/206: dread - d6/db/de/f3e zero size 2022-01-31T19:44:06.137 INFO:tasks.workunit.client.0.smithi167.stdout:0/57: dwrite f8 [0,4194304] 0 2022-01-31T19:44:06.139 INFO:tasks.workunit.client.0.smithi167.stdout:3/161: dwrite d9/d21/f23 [0,4194304] 0 2022-01-31T19:44:06.139 INFO:tasks.workunit.client.0.smithi167.stdout:0/58: mknod c16 0 2022-01-31T19:44:06.140 INFO:tasks.workunit.client.0.smithi167.stdout:0/59: chown fe 39588 1 2022-01-31T19:44:06.143 INFO:tasks.workunit.client.0.smithi167.stdout:9/73: dwrite dd/f11 [0,4194304] 0 2022-01-31T19:44:06.143 INFO:tasks.workunit.client.0.smithi167.stdout:9/74: write f9 [573314,58346] 0 2022-01-31T19:44:06.155 INFO:tasks.workunit.client.0.smithi167.stdout:8/207: dwrite d6/db/de/f20 [0,4194304] 0 2022-01-31T19:44:06.158 INFO:tasks.workunit.client.0.smithi167.stdout:3/162: dread d9/d21/f23 [0,4194304] 0 2022-01-31T19:44:06.159 INFO:tasks.workunit.client.0.smithi167.stdout:3/163: creat d9/d26/f2a x:0 0 0 2022-01-31T19:44:06.166 INFO:tasks.workunit.client.0.smithi167.stdout:9/75: dwrite dd/f14 [0,4194304] 0 2022-01-31T19:44:06.170 INFO:tasks.workunit.client.0.smithi167.stdout:1/172: dread d0/d10/d18/d39/f3d [0,4194304] 0 2022-01-31T19:44:06.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:05 smithi167 conmon[32206]: cluster 2022-01-31T19:44:04.430653+0000 2022-01-31T19:44:06.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:05 smithi167 conmon[32206]: mgr.smithi171.asyxnc (mgr.24387) 12 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 365 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 5.8 MiB/s rd, 15 MiB/s wr, 601 op/s 2022-01-31T19:44:06.171 INFO:tasks.workunit.client.0.smithi167.stdout:1/173: chown d0/d10/d20/c3c 2124232063 1 2022-01-31T19:44:06.171 INFO:tasks.workunit.client.0.smithi167.stdout:1/174: truncate d0/fc 1619121 0 2022-01-31T19:44:06.181 INFO:tasks.workunit.client.0.smithi167.stdout:0/60: dwrite ff [0,4194304] 0 2022-01-31T19:44:06.181 INFO:tasks.workunit.client.0.smithi167.stdout:0/61: fsync f10 0 2022-01-31T19:44:06.181 INFO:tasks.workunit.client.0.smithi167.stdout:0/62: chown f14 2444460 1 2022-01-31T19:44:06.182 INFO:tasks.workunit.client.0.smithi167.stdout:3/164: dwrite d9/d21/f28 [0,4194304] 0 2022-01-31T19:44:06.182 INFO:tasks.workunit.client.0.smithi167.stdout:8/208: dwrite d6/f1d [4194304,4194304] 0 2022-01-31T19:44:06.183 INFO:tasks.workunit.client.0.smithi167.stdout:8/209: stat d6/db/de/f3e 0 2022-01-31T19:44:06.186 INFO:tasks.workunit.client.0.smithi167.stdout:0/63: mkdir d17 0 2022-01-31T19:44:06.187 INFO:tasks.workunit.client.0.smithi167.stdout:3/165: creat d9/d29/f2b x:0 0 0 2022-01-31T19:44:06.189 INFO:tasks.workunit.client.0.smithi167.stdout:8/210: creat d6/db/d28/f43 x:0 0 0 2022-01-31T19:44:06.191 INFO:tasks.workunit.client.0.smithi167.stdout:8/211: symlink d6/db/de/d3d/l44 0 2022-01-31T19:44:06.193 INFO:tasks.workunit.client.0.smithi167.stdout:8/212: write d6/db/de/d18/f1b [647420,57676] 0 2022-01-31T19:44:06.193 INFO:tasks.workunit.client.0.smithi167.stdout:8/213: dread - d6/db/de/f26 zero size 2022-01-31T19:44:06.194 INFO:tasks.workunit.client.0.smithi167.stdout:8/214: dread d6/d1e/f22 [0,4194304] 0 2022-01-31T19:44:06.198 INFO:tasks.workunit.client.0.smithi167.stdout:8/215: write d6/f14 [1547414,44998] 0 2022-01-31T19:44:06.200 INFO:tasks.workunit.client.0.smithi167.stdout:8/216: link d6/db/de/d18/c30 d6/db/c45 0 2022-01-31T19:44:06.203 INFO:tasks.workunit.client.0.smithi167.stdout:1/175: dwrite d0/fc [0,4194304] 0 2022-01-31T19:44:06.206 INFO:tasks.workunit.client.0.smithi167.stdout:1/176: read d0/d10/d20/f28 [1291080,3072] 0 2022-01-31T19:44:06.220 INFO:tasks.workunit.client.0.smithi167.stdout:8/217: dwrite d6/db/de/d18/f1b [0,4194304] 0 2022-01-31T19:44:06.222 INFO:tasks.workunit.client.0.smithi167.stdout:8/218: rename d6/db/de/d18/c30 to d6/db/d28/c46 0 2022-01-31T19:44:06.222 INFO:tasks.workunit.client.0.smithi167.stdout:8/219: fsync d6/db/de/f26 0 2022-01-31T19:44:06.223 INFO:tasks.workunit.client.0.smithi167.stdout:8/220: dread d6/d1e/f22 [0,4194304] 0 2022-01-31T19:44:06.223 INFO:tasks.workunit.client.0.smithi167.stdout:8/221: rmdir d6/db/de/d3d 39 2022-01-31T19:44:06.224 INFO:tasks.workunit.client.0.smithi167.stdout:8/222: write d6/db/fd [2283785,28848] 0 2022-01-31T19:44:06.224 INFO:tasks.workunit.client.0.smithi167.stdout:8/223: chown d6 32627 1 2022-01-31T19:44:06.225 INFO:tasks.workunit.client.0.smithi167.stdout:8/224: symlink d6/d1e/d2d/l47 0 2022-01-31T19:44:06.225 INFO:tasks.workunit.client.0.smithi167.stdout:8/225: mkdir d6/db/d36/d48 0 2022-01-31T19:44:06.226 INFO:tasks.workunit.client.0.smithi167.stdout:8/226: link d6/db/d36/l42 d6/d1e/d2d/l49 0 2022-01-31T19:44:06.228 INFO:tasks.workunit.client.0.smithi167.stdout:8/227: link d6/db/de/f10 d6/db/d36/f4a 0 2022-01-31T19:44:06.229 INFO:tasks.workunit.client.0.smithi167.stdout:8/228: write d6/db/de/d18/f19 [3469669,77161] 0 2022-01-31T19:44:06.246 INFO:tasks.workunit.client.0.smithi167.stdout:8/229: dwrite d6/d1e/f41 [0,4194304] 0 2022-01-31T19:44:06.248 INFO:tasks.workunit.client.0.smithi167.stdout:8/230: symlink d6/db/de/d18/l4b 0 2022-01-31T19:44:06.248 INFO:tasks.workunit.client.0.smithi167.stdout:8/231: fdatasync d6/db/de/f20 0 2022-01-31T19:44:06.256 INFO:tasks.workunit.client.0.smithi167.stdout:8/232: dread d6/d1e/f41 [0,4194304] 0 2022-01-31T19:44:06.268 INFO:tasks.workunit.client.0.smithi167.stdout:4/97: sync 2022-01-31T19:44:06.269 INFO:tasks.workunit.client.0.smithi167.stdout:4/98: rename d0/d6/l4 to d0/l1b 0 2022-01-31T19:44:06.270 INFO:tasks.workunit.client.0.smithi167.stdout:4/99: mkdir d0/d6/dc/d1c 0 2022-01-31T19:44:06.270 INFO:tasks.workunit.client.0.smithi167.stdout:4/100: creat d0/d6/dc/f1d x:0 0 0 2022-01-31T19:44:06.271 INFO:tasks.workunit.client.0.smithi167.stdout:4/101: write d0/d6/d7/f8 [5073942,38144] 0 2022-01-31T19:44:06.271 INFO:tasks.workunit.client.0.smithi167.stdout:4/102: fdatasync d0/d6/d7/f18 0 2022-01-31T19:44:06.272 INFO:tasks.workunit.client.0.smithi167.stdout:5/165: sync 2022-01-31T19:44:06.273 INFO:tasks.workunit.client.0.smithi167.stdout:2/60: sync 2022-01-31T19:44:06.273 INFO:tasks.workunit.client.0.smithi167.stdout:7/135: sync 2022-01-31T19:44:06.273 INFO:tasks.workunit.client.0.smithi167.stdout:2/61: unlink fb 0 2022-01-31T19:44:06.273 INFO:tasks.workunit.client.0.smithi167.stdout:7/136: dread - d8/dc/f2b zero size 2022-01-31T19:44:06.274 INFO:tasks.workunit.client.0.smithi167.stdout:7/137: creat d8/dc/f33 x:0 0 0 2022-01-31T19:44:06.274 INFO:tasks.workunit.client.0.smithi167.stdout:7/138: read d8/dc/d18/d24/f29 [17917,8952] 0 2022-01-31T19:44:06.277 INFO:tasks.workunit.client.0.smithi167.stdout:5/166: rename d5/d6/f1c to d5/d6/d16/d18/f31 0 2022-01-31T19:44:06.279 INFO:tasks.workunit.client.0.smithi167.stdout:2/62: symlink dd/l14 0 2022-01-31T19:44:06.279 INFO:tasks.workunit.client.0.smithi167.stdout:2/63: readlink dd/le 0 2022-01-31T19:44:06.280 INFO:tasks.workunit.client.0.smithi167.stdout:5/167: dread d5/d6/de/d22/f25 [0,4194304] 0 2022-01-31T19:44:06.282 INFO:tasks.workunit.client.0.smithi167.stdout:7/139: symlink d8/dc/l34 0 2022-01-31T19:44:06.284 INFO:tasks.workunit.client.0.smithi167.stdout:5/168: rename d5/d6/d16/d18/c30 to d5/d6/d16/c32 0 2022-01-31T19:44:06.285 INFO:tasks.workunit.client.0.smithi167.stdout:4/103: dwrite d0/d6/dc/f11 [4194304,4194304] 0 2022-01-31T19:44:06.285 INFO:tasks.workunit.client.0.smithi167.stdout:4/104: stat d0/l2 0 2022-01-31T19:44:06.285 INFO:tasks.workunit.client.0.smithi167.stdout:2/64: dread f7 [0,4194304] 0 2022-01-31T19:44:06.285 INFO:tasks.workunit.client.0.smithi167.stdout:7/140: mkdir d8/d35 0 2022-01-31T19:44:06.286 INFO:tasks.workunit.client.0.smithi167.stdout:5/169: unlink d5/d6/d16/f2b 0 2022-01-31T19:44:06.287 INFO:tasks.workunit.client.0.smithi167.stdout:4/105: symlink d0/d6/d7/l1e 0 2022-01-31T19:44:06.289 INFO:tasks.workunit.client.0.smithi167.stdout:2/65: creat dd/f15 x:0 0 0 2022-01-31T19:44:06.290 INFO:tasks.workunit.client.0.smithi167.stdout:7/141: creat d8/d35/f36 x:0 0 0 2022-01-31T19:44:06.290 INFO:tasks.workunit.client.0.smithi167.stdout:7/142: write d8/dc/f1c [920496,121103] 0 2022-01-31T19:44:06.294 INFO:tasks.workunit.client.0.smithi167.stdout:2/66: rename f2 to dd/f16 0 2022-01-31T19:44:06.300 INFO:tasks.workunit.client.0.smithi167.stdout:2/67: mknod dd/c17 0 2022-01-31T19:44:06.300 INFO:tasks.workunit.client.0.smithi167.stdout:2/68: unlink fc 0 2022-01-31T19:44:06.301 INFO:tasks.workunit.client.0.smithi167.stdout:2/69: readlink dd/l14 0 2022-01-31T19:44:06.304 INFO:tasks.workunit.client.0.smithi167.stdout:2/70: chown dd/df/c10 52 1 2022-01-31T19:44:06.311 INFO:tasks.workunit.client.0.smithi167.stdout:7/143: dwrite d8/d2c/f25 [0,4194304] 0 2022-01-31T19:44:06.311 INFO:tasks.workunit.client.0.smithi167.stdout:5/170: dwrite d5/f9 [4194304,4194304] 0 2022-01-31T19:44:06.312 INFO:tasks.workunit.client.0.smithi167.stdout:5/171: creat d5/d6/d16/d18/f33 x:0 0 0 2022-01-31T19:44:06.312 INFO:tasks.workunit.client.0.smithi167.stdout:5/172: chown d5/f8 621 1 2022-01-31T19:44:06.312 INFO:tasks.workunit.client.0.smithi167.stdout:5/173: fsync d5/f9 0 2022-01-31T19:44:06.313 INFO:tasks.workunit.client.0.smithi167.stdout:7/144: mknod d8/dc/d12/c37 0 2022-01-31T19:44:06.320 INFO:tasks.workunit.client.0.smithi167.stdout:7/145: symlink d8/dc/d19/l38 0 2022-01-31T19:44:06.321 INFO:tasks.workunit.client.0.smithi167.stdout:7/146: symlink d8/dc/d12/l39 0 2022-01-31T19:44:06.331 INFO:tasks.workunit.client.0.smithi167.stdout:2/71: dwrite dd/f15 [0,4194304] 0 2022-01-31T19:44:06.335 INFO:tasks.workunit.client.0.smithi167.stdout:5/174: dwrite d5/d6/d16/d18/f1f [0,4194304] 0 2022-01-31T19:44:06.335 INFO:tasks.workunit.client.0.smithi167.stdout:5/175: creat d5/f34 x:0 0 0 2022-01-31T19:44:06.336 INFO:tasks.workunit.client.0.smithi167.stdout:2/72: getdents dd/df 0 2022-01-31T19:44:06.336 INFO:tasks.workunit.client.0.smithi167.stdout:5/176: write d5/f7 [708224,39796] 0 2022-01-31T19:44:06.337 INFO:tasks.workunit.client.0.smithi167.stdout:7/147: dwrite d8/dc/d18/d24/f26 [0,4194304] 0 2022-01-31T19:44:06.337 INFO:tasks.workunit.client.0.smithi167.stdout:5/177: write d5/d10/f17 [226913,74343] 0 2022-01-31T19:44:06.337 INFO:tasks.workunit.client.0.smithi167.stdout:5/178: creat d5/d6/de/d22/f35 x:0 0 0 2022-01-31T19:44:06.338 INFO:tasks.workunit.client.0.smithi167.stdout:5/179: write d5/d6/de/d22/f25 [354876,44457] 0 2022-01-31T19:44:06.340 INFO:tasks.workunit.client.0.smithi167.stdout:2/73: mkdir dd/d18 0 2022-01-31T19:44:06.341 INFO:tasks.workunit.client.0.smithi167.stdout:2/74: chown dd/df/c10 703188 1 2022-01-31T19:44:06.341 INFO:tasks.workunit.client.0.smithi167.stdout:5/180: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:06.343 INFO:tasks.workunit.client.0.smithi167.stdout:7/148: link d8/dc/d19/l38 d8/dc/d19/l3a 0 2022-01-31T19:44:06.346 INFO:tasks.workunit.client.0.smithi167.stdout:5/181: rename d5/d6/f13 to d5/d6/d16/d24/f36 0 2022-01-31T19:44:06.362 INFO:tasks.workunit.client.0.smithi167.stdout:5/182: dread d5/d6/d16/d18/f1f [0,4194304] 0 2022-01-31T19:44:06.362 INFO:tasks.workunit.client.0.smithi167.stdout:5/183: creat d5/d6/de/f37 x:0 0 0 2022-01-31T19:44:06.362 INFO:tasks.workunit.client.0.smithi167.stdout:5/184: creat d5/d6/d16/f38 x:0 0 0 2022-01-31T19:44:06.364 INFO:tasks.workunit.client.0.smithi167.stdout:5/185: symlink d5/d6/d16/l39 0 2022-01-31T19:44:06.365 INFO:tasks.workunit.client.0.smithi167.stdout:2/75: dwrite dd/f16 [0,4194304] 0 2022-01-31T19:44:06.366 INFO:tasks.workunit.client.0.smithi167.stdout:2/76: symlink dd/l19 0 2022-01-31T19:44:06.367 INFO:tasks.workunit.client.0.smithi167.stdout:2/77: chown dd/l14 56905 1 2022-01-31T19:44:06.367 INFO:tasks.workunit.client.0.smithi167.stdout:7/149: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:06.368 INFO:tasks.workunit.client.0.smithi167.stdout:5/186: rename d5/d10 to d5/d6/d3a 0 2022-01-31T19:44:06.368 INFO:tasks.workunit.client.0.smithi167.stdout:5/187: creat d5/d6/d16/f3b x:0 0 0 2022-01-31T19:44:06.369 INFO:tasks.workunit.client.0.smithi167.stdout:5/188: stat l4 0 2022-01-31T19:44:06.380 INFO:tasks.workunit.client.0.smithi167.stdout:2/78: rename dd/f16 to dd/f1a 0 2022-01-31T19:44:06.381 INFO:tasks.workunit.client.0.smithi167.stdout:2/79: write dd/f1a [1146901,69125] 0 2022-01-31T19:44:06.381 INFO:tasks.workunit.client.0.smithi167.stdout:7/150: getdents d8/d35 0 2022-01-31T19:44:06.384 INFO:tasks.workunit.client.0.smithi167.stdout:6/107: sync 2022-01-31T19:44:06.385 INFO:tasks.workunit.client.0.smithi167.stdout:2/80: rmdir dd/df 39 2022-01-31T19:44:06.388 INFO:tasks.workunit.client.0.smithi167.stdout:7/151: creat d8/dc/d18/f3b x:0 0 0 2022-01-31T19:44:06.388 INFO:tasks.workunit.client.0.smithi167.stdout:7/152: creat d8/dc/d19/f3c x:0 0 0 2022-01-31T19:44:06.388 INFO:tasks.workunit.client.0.smithi167.stdout:7/153: readlink d8/l9 0 2022-01-31T19:44:06.391 INFO:tasks.workunit.client.0.smithi167.stdout:5/189: dwrite d5/f9 [0,4194304] 0 2022-01-31T19:44:06.391 INFO:tasks.workunit.client.0.smithi167.stdout:5/190: unlink d5/d6/d16/c32 0 2022-01-31T19:44:06.391 INFO:tasks.workunit.client.0.smithi167.stdout:6/108: mknod d8/c28 0 2022-01-31T19:44:06.394 INFO:tasks.workunit.client.0.smithi167.stdout:5/191: symlink d5/d6/d16/d18/l3c 0 2022-01-31T19:44:06.395 INFO:tasks.workunit.client.0.smithi167.stdout:6/109: symlink d8/d20/l29 0 2022-01-31T19:44:06.397 INFO:tasks.workunit.client.0.smithi167.stdout:6/110: truncate d8/fd 255575 0 2022-01-31T19:44:06.399 INFO:tasks.workunit.client.0.smithi167.stdout:6/111: truncate d8/fd 923253 0 2022-01-31T19:44:06.405 INFO:tasks.workunit.client.0.smithi167.stdout:7/154: dwrite d8/dc/d12/f2a [0,4194304] 0 2022-01-31T19:44:06.409 INFO:tasks.workunit.client.0.smithi167.stdout:2/81: dread f9 [0,4194304] 0 2022-01-31T19:44:06.411 INFO:tasks.workunit.client.0.smithi167.stdout:7/155: creat d8/d35/f3d x:0 0 0 2022-01-31T19:44:06.413 INFO:tasks.workunit.client.0.smithi167.stdout:7/156: symlink d8/dc/l3e 0 2022-01-31T19:44:06.413 INFO:tasks.workunit.client.0.smithi167.stdout:7/157: symlink d8/d2e/l3f 0 2022-01-31T19:44:06.413 INFO:tasks.workunit.client.0.smithi167.stdout:7/158: creat d8/dc/f40 x:0 0 0 2022-01-31T19:44:06.415 INFO:tasks.workunit.client.0.smithi167.stdout:7/159: symlink d8/dc/d12/l41 0 2022-01-31T19:44:06.416 INFO:tasks.workunit.client.0.smithi167.stdout:7/160: creat d8/dc/d19/f42 x:0 0 0 2022-01-31T19:44:06.424 INFO:tasks.workunit.client.0.smithi167.stdout:5/192: dwrite d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:06.424 INFO:tasks.workunit.client.0.smithi167.stdout:5/193: creat d5/d6/d16/d18/f3d x:0 0 0 2022-01-31T19:44:06.424 INFO:tasks.workunit.client.0.smithi167.stdout:5/194: chown d5/d6/d3a/c1d 0 1 2022-01-31T19:44:06.425 INFO:tasks.workunit.client.0.smithi167.stdout:5/195: creat d5/d6/de/f3e x:0 0 0 2022-01-31T19:44:06.425 INFO:tasks.workunit.client.0.smithi167.stdout:6/112: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:06.428 INFO:tasks.workunit.client.0.smithi167.stdout:5/196: symlink d5/d6/d3a/l3f 0 2022-01-31T19:44:06.428 INFO:tasks.workunit.client.0.smithi167.stdout:5/197: read - d5/d6/d16/d24/f36 zero size 2022-01-31T19:44:06.428 INFO:tasks.workunit.client.0.smithi167.stdout:5/198: chown d5/d6/de/l1b 3360 1 2022-01-31T19:44:06.429 INFO:tasks.workunit.client.0.smithi167.stdout:5/199: symlink d5/d6/de/l40 0 2022-01-31T19:44:06.430 INFO:tasks.workunit.client.0.smithi167.stdout:5/200: creat d5/d6/f41 x:0 0 0 2022-01-31T19:44:06.430 INFO:tasks.workunit.client.0.smithi167.stdout:5/201: creat d5/d6/f42 x:0 0 0 2022-01-31T19:44:06.430 INFO:tasks.workunit.client.0.smithi167.stdout:5/202: fsync d5/d6/d3a/f17 0 2022-01-31T19:44:06.431 INFO:tasks.workunit.client.0.smithi167.stdout:5/203: write d5/f34 [484487,9456] 0 2022-01-31T19:44:06.431 INFO:tasks.workunit.client.0.smithi167.stdout:5/204: chown d5/lb 412 1 2022-01-31T19:44:06.431 INFO:tasks.workunit.client.0.smithi167.stdout:5/205: write d5/d6/d16/d18/f2c [747885,109604] 0 2022-01-31T19:44:06.432 INFO:tasks.workunit.client.0.smithi167.stdout:5/206: mkdir d5/d6/d16/d18/d43 0 2022-01-31T19:44:06.432 INFO:tasks.workunit.client.0.smithi167.stdout:5/207: write d5/f34 [390622,10008] 0 2022-01-31T19:44:06.432 INFO:tasks.workunit.client.0.smithi167.stdout:5/208: write d5/d6/f42 [483825,86374] 0 2022-01-31T19:44:06.433 INFO:tasks.workunit.client.0.smithi167.stdout:5/209: truncate d5/f7 4392241 0 2022-01-31T19:44:06.434 INFO:tasks.workunit.client.0.smithi167.stdout:2/82: dwrite fa [4194304,4194304] 0 2022-01-31T19:44:06.434 INFO:tasks.workunit.client.0.smithi167.stdout:5/210: write d5/d6/d16/d18/f31 [3324947,15057] 0 2022-01-31T19:44:06.434 INFO:tasks.workunit.client.0.smithi167.stdout:5/211: creat d5/f44 x:0 0 0 2022-01-31T19:44:06.434 INFO:tasks.workunit.client.0.smithi167.stdout:2/83: truncate f7 1152850 0 2022-01-31T19:44:06.435 INFO:tasks.workunit.client.0.smithi167.stdout:2/84: write f8 [324265,77090] 0 2022-01-31T19:44:06.437 INFO:tasks.workunit.client.0.smithi167.stdout:7/161: rename d8/d2e to d8/d2c/d43 0 2022-01-31T19:44:06.438 INFO:tasks.workunit.client.0.smithi167.stdout:7/162: truncate f2 3085805 0 2022-01-31T19:44:06.440 INFO:tasks.workunit.client.0.smithi167.stdout:5/212: dread d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:06.441 INFO:tasks.workunit.client.0.smithi167.stdout:5/213: dread d5/d6/de/d22/f25 [0,4194304] 0 2022-01-31T19:44:06.442 INFO:tasks.workunit.client.0.smithi167.stdout:7/163: getdents d8/dc/d19 0 2022-01-31T19:44:06.443 INFO:tasks.workunit.client.0.smithi167.stdout:5/214: dread d5/d6/de/d22/f25 [0,4194304] 0 2022-01-31T19:44:06.443 INFO:tasks.workunit.client.0.smithi167.stdout:5/215: truncate d5/f34 1320249 0 2022-01-31T19:44:06.443 INFO:tasks.workunit.client.0.smithi167.stdout:5/216: fsync d5/d6/de/f3e 0 2022-01-31T19:44:06.443 INFO:tasks.workunit.client.0.smithi167.stdout:5/217: write d5/d6/f42 [1109229,50440] 0 2022-01-31T19:44:06.444 INFO:tasks.workunit.client.0.smithi167.stdout:5/218: mkdir d5/d6/d16/d18/d43/d45 0 2022-01-31T19:44:06.445 INFO:tasks.workunit.client.0.smithi167.stdout:5/219: truncate d5/d6/f42 1682562 0 2022-01-31T19:44:06.450 INFO:tasks.workunit.client.0.smithi167.stdout:2/85: dread f9 [0,4194304] 0 2022-01-31T19:44:06.454 INFO:tasks.workunit.client.0.smithi167.stdout:7/164: dread d8/dc/d18/d24/f29 [0,4194304] 0 2022-01-31T19:44:06.455 INFO:tasks.workunit.client.0.smithi167.stdout:7/165: fsync d8/dc/f23 0 2022-01-31T19:44:06.460 INFO:tasks.workunit.client.0.smithi167.stdout:7/166: dread f2 [0,4194304] 0 2022-01-31T19:44:06.461 INFO:tasks.workunit.client.0.smithi167.stdout:5/220: dwrite d5/d6/d16/d18/f2e [0,4194304] 0 2022-01-31T19:44:06.464 INFO:tasks.workunit.client.0.smithi167.stdout:5/221: rename d5/d6/d3a/l15 to d5/d6/d3a/l46 0 2022-01-31T19:44:06.464 INFO:tasks.workunit.client.0.smithi167.stdout:5/222: chown d5/f7 510184 1 2022-01-31T19:44:06.465 INFO:tasks.workunit.client.0.smithi167.stdout:5/223: truncate d5/d6/de/f3e 256160 0 2022-01-31T19:44:06.465 INFO:tasks.workunit.client.0.smithi167.stdout:5/224: chown d5/d6/d16/d18/l3c 25096 1 2022-01-31T19:44:06.494 INFO:tasks.workunit.client.0.smithi167.stdout:2/86: dwrite f8 [0,4194304] 0 2022-01-31T19:44:06.495 INFO:tasks.workunit.client.0.smithi167.stdout:2/87: mkdir dd/df/d1b 0 2022-01-31T19:44:06.495 INFO:tasks.workunit.client.0.smithi167.stdout:2/88: truncate f7 1715298 0 2022-01-31T19:44:06.496 INFO:tasks.workunit.client.0.smithi167.stdout:5/225: dwrite d5/d6/f28 [0,4194304] 0 2022-01-31T19:44:06.496 INFO:tasks.workunit.client.0.smithi167.stdout:7/167: dwrite d8/dc/f17 [0,4194304] 0 2022-01-31T19:44:06.497 INFO:tasks.workunit.client.0.smithi167.stdout:2/89: read fa [4735864,96758] 0 2022-01-31T19:44:06.497 INFO:tasks.workunit.client.0.smithi167.stdout:2/90: chown dd/df/c11 42554103 1 2022-01-31T19:44:06.501 INFO:tasks.workunit.client.0.smithi167.stdout:7/168: unlink l1 0 2022-01-31T19:44:06.502 INFO:tasks.workunit.client.0.smithi167.stdout:2/91: creat dd/df/f1c x:0 0 0 2022-01-31T19:44:06.502 INFO:tasks.workunit.client.0.smithi167.stdout:2/92: fsync dd/f1a 0 2022-01-31T19:44:06.503 INFO:tasks.workunit.client.0.smithi167.stdout:2/93: fsync f8 0 2022-01-31T19:44:06.503 INFO:tasks.workunit.client.0.smithi167.stdout:2/94: creat dd/df/f1d x:0 0 0 2022-01-31T19:44:06.512 INFO:tasks.workunit.client.0.smithi167.stdout:5/226: dwrite d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:06.517 INFO:tasks.workunit.client.0.smithi167.stdout:2/95: dread f8 [0,4194304] 0 2022-01-31T19:44:06.543 INFO:tasks.workunit.client.0.smithi167.stdout:5/227: dwrite d5/f7 [4194304,4194304] 0 2022-01-31T19:44:06.543 INFO:tasks.workunit.client.0.smithi167.stdout:2/96: dwrite f8 [0,4194304] 0 2022-01-31T19:44:06.544 INFO:tasks.workunit.client.0.smithi167.stdout:5/228: rename d5/d6/c2f to d5/d6/c47 0 2022-01-31T19:44:06.569 INFO:tasks.workunit.client.0.smithi167.stdout:2/97: dwrite f9 [0,4194304] 0 2022-01-31T19:44:06.569 INFO:tasks.workunit.client.0.smithi167.stdout:2/98: creat dd/f1e x:0 0 0 2022-01-31T19:44:06.569 INFO:tasks.workunit.client.0.smithi167.stdout:2/99: read - dd/df/f1c zero size 2022-01-31T19:44:06.570 INFO:tasks.workunit.client.0.smithi167.stdout:5/229: dwrite d5/d6/d16/d18/f31 [0,4194304] 0 2022-01-31T19:44:06.570 INFO:tasks.workunit.client.0.smithi167.stdout:2/100: symlink dd/d18/l1f 0 2022-01-31T19:44:06.571 INFO:tasks.workunit.client.0.smithi167.stdout:2/101: creat dd/df/f20 x:0 0 0 2022-01-31T19:44:06.572 INFO:tasks.workunit.client.0.smithi167.stdout:2/102: mknod dd/c21 0 2022-01-31T19:44:06.573 INFO:tasks.workunit.client.0.smithi167.stdout:2/103: mkdir dd/df/d22 0 2022-01-31T19:44:06.573 INFO:tasks.workunit.client.0.smithi167.stdout:2/104: chown dd/c21 150124078 1 2022-01-31T19:44:06.574 INFO:tasks.workunit.client.0.smithi167.stdout:2/105: rename dd/df/f1d to dd/f23 0 2022-01-31T19:44:06.590 INFO:tasks.workunit.client.0.smithi167.stdout:5/230: dwrite d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:06.591 INFO:tasks.workunit.client.0.smithi167.stdout:2/106: dwrite dd/f1e [0,4194304] 0 2022-01-31T19:44:06.591 INFO:tasks.workunit.client.0.smithi167.stdout:2/107: stat dd/f1a 0 2022-01-31T19:44:06.591 INFO:tasks.workunit.client.0.smithi167.stdout:2/108: rename dd to dd/df/d1b/d24 22 2022-01-31T19:44:06.592 INFO:tasks.workunit.client.0.smithi167.stdout:5/231: creat d5/d6/d3a/f48 x:0 0 0 2022-01-31T19:44:06.592 INFO:tasks.workunit.client.0.smithi167.stdout:2/109: write f7 [491929,64590] 0 2022-01-31T19:44:06.593 INFO:tasks.workunit.client.0.smithi167.stdout:5/232: mknod d5/d6/de/d22/c49 0 2022-01-31T19:44:06.594 INFO:tasks.workunit.client.0.smithi167.stdout:5/233: symlink d5/d6/d16/d18/d43/l4a 0 2022-01-31T19:44:06.596 INFO:tasks.workunit.client.0.smithi167.stdout:2/110: getdents dd/d18 0 2022-01-31T19:44:06.615 INFO:tasks.workunit.client.0.smithi167.stdout:2/111: dwrite dd/f23 [0,4194304] 0 2022-01-31T19:44:06.616 INFO:tasks.workunit.client.0.smithi167.stdout:2/112: write dd/df/f1c [488940,28457] 0 2022-01-31T19:44:06.616 INFO:tasks.workunit.client.0.smithi167.stdout:5/234: dwrite d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:06.616 INFO:tasks.workunit.client.0.smithi167.stdout:5/235: chown d5/d6/cd 62412 1 2022-01-31T19:44:06.617 INFO:tasks.workunit.client.0.smithi167.stdout:5/236: write d5/d6/d16/f38 [488322,19011] 0 2022-01-31T19:44:06.619 INFO:tasks.workunit.client.0.smithi167.stdout:5/237: truncate d5/d6/f42 148664 0 2022-01-31T19:44:06.626 INFO:tasks.workunit.client.0.smithi167.stdout:5/238: rmdir d5/d6/de 39 2022-01-31T19:44:06.635 INFO:tasks.workunit.client.0.smithi167.stdout:9/76: sync 2022-01-31T19:44:06.635 INFO:tasks.workunit.client.0.smithi167.stdout:3/166: sync 2022-01-31T19:44:06.635 INFO:tasks.workunit.client.0.smithi167.stdout:0/64: sync 2022-01-31T19:44:06.635 INFO:tasks.workunit.client.0.smithi167.stdout:3/167: dread - d9/f27 zero size 2022-01-31T19:44:06.636 INFO:tasks.workunit.client.0.smithi167.stdout:0/65: symlink d17/l18 0 2022-01-31T19:44:06.639 INFO:tasks.workunit.client.0.smithi167.stdout:2/113: dwrite f9 [0,4194304] 0 2022-01-31T19:44:06.639 INFO:tasks.workunit.client.0.smithi167.stdout:3/168: creat d9/f2c x:0 0 0 2022-01-31T19:44:06.641 INFO:tasks.workunit.client.0.smithi167.stdout:3/169: symlink d9/d29/l2d 0 2022-01-31T19:44:06.643 INFO:tasks.workunit.client.0.smithi167.stdout:3/170: mkdir d9/d29/d2e 0 2022-01-31T19:44:06.652 INFO:tasks.workunit.client.0.smithi167.stdout:3/171: write d9/d21/f1e [2712501,52579] 0 2022-01-31T19:44:06.652 INFO:tasks.workunit.client.0.smithi167.stdout:3/172: readlink l5 0 2022-01-31T19:44:06.655 INFO:tasks.workunit.client.0.smithi167.stdout:3/173: dread d9/fc [0,4194304] 0 2022-01-31T19:44:06.662 INFO:tasks.workunit.client.0.smithi167.stdout:0/66: dwrite f10 [0,4194304] 0 2022-01-31T19:44:06.663 INFO:tasks.workunit.client.0.smithi167.stdout:9/77: dwrite dd/fe [0,4194304] 0 2022-01-31T19:44:06.663 INFO:tasks.workunit.client.0.smithi167.stdout:9/78: stat dd/fe 0 2022-01-31T19:44:06.664 INFO:tasks.workunit.client.0.smithi167.stdout:0/67: mkdir d17/d19 0 2022-01-31T19:44:06.666 INFO:tasks.workunit.client.0.smithi167.stdout:9/79: rmdir dd/d17 39 2022-01-31T19:44:06.668 INFO:tasks.workunit.client.0.smithi167.stdout:9/80: mkdir dd/d19 0 2022-01-31T19:44:06.669 INFO:tasks.workunit.client.0.smithi167.stdout:9/81: truncate f7 671791 0 2022-01-31T19:44:06.669 INFO:tasks.workunit.client.0.smithi167.stdout:9/82: truncate f7 733433 0 2022-01-31T19:44:06.672 INFO:tasks.workunit.client.0.smithi167.stdout:9/83: dread dd/ff [0,4194304] 0 2022-01-31T19:44:06.674 INFO:tasks.workunit.client.0.smithi167.stdout:9/84: creat dd/f1a x:0 0 0 2022-01-31T19:44:06.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/114: dwrite dd/f1e [0,4194304] 0 2022-01-31T19:44:06.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/115: fsync dd/f1e 0 2022-01-31T19:44:06.676 INFO:tasks.workunit.client.0.smithi167.stdout:9/85: rename f8 to dd/d19/f1b 0 2022-01-31T19:44:06.692 INFO:tasks.workunit.client.0.smithi167.stdout:0/68: dwrite f10 [0,4194304] 0 2022-01-31T19:44:06.694 INFO:tasks.workunit.client.0.smithi167.stdout:2/116: dwrite dd/f23 [4194304,4194304] 0 2022-01-31T19:44:06.695 INFO:tasks.workunit.client.0.smithi167.stdout:9/86: getdents dd/d19 0 2022-01-31T19:44:06.695 INFO:tasks.workunit.client.0.smithi167.stdout:3/174: dwrite d9/f1a [0,4194304] 0 2022-01-31T19:44:06.696 INFO:tasks.workunit.client.0.smithi167.stdout:3/175: chown d9/f2c 25 1 2022-01-31T19:44:06.696 INFO:tasks.workunit.client.0.smithi167.stdout:9/87: getdents dd/d17 0 2022-01-31T19:44:06.703 INFO:tasks.workunit.client.0.smithi167.stdout:0/69: dread f13 [0,4194304] 0 2022-01-31T19:44:06.706 INFO:tasks.workunit.client.0.smithi167.stdout:2/117: dread f9 [0,4194304] 0 2022-01-31T19:44:06.707 INFO:tasks.workunit.client.0.smithi167.stdout:9/88: dread f9 [0,4194304] 0 2022-01-31T19:44:06.711 INFO:tasks.workunit.client.0.smithi167.stdout:3/176: dwrite d9/f25 [0,4194304] 0 2022-01-31T19:44:06.714 INFO:tasks.workunit.client.0.smithi167.stdout:3/177: mknod d9/d29/c2f 0 2022-01-31T19:44:06.716 INFO:tasks.workunit.client.0.smithi167.stdout:3/178: unlink d9/d29/l2d 0 2022-01-31T19:44:06.717 INFO:tasks.workunit.client.0.smithi167.stdout:3/179: creat d9/f30 x:0 0 0 2022-01-31T19:44:06.725 INFO:tasks.workunit.client.0.smithi167.stdout:9/89: dwrite dd/f1a [0,4194304] 0 2022-01-31T19:44:06.734 INFO:tasks.workunit.client.0.smithi167.stdout:4/106: sync 2022-01-31T19:44:06.735 INFO:tasks.workunit.client.0.smithi167.stdout:8/233: sync 2022-01-31T19:44:06.735 INFO:tasks.workunit.client.0.smithi167.stdout:1/177: sync 2022-01-31T19:44:06.735 INFO:tasks.workunit.client.0.smithi167.stdout:3/180: dwrite d9/f10 [0,4194304] 0 2022-01-31T19:44:06.736 INFO:tasks.workunit.client.0.smithi167.stdout:1/178: mknod d0/d10/d18/d16/c43 0 2022-01-31T19:44:06.737 INFO:tasks.workunit.client.0.smithi167.stdout:1/179: mknod d0/d11/d24/c44 0 2022-01-31T19:44:06.737 INFO:tasks.workunit.client.0.smithi167.stdout:8/234: write d6/ff [2918688,894] 0 2022-01-31T19:44:06.738 INFO:tasks.workunit.client.0.smithi167.stdout:4/107: write d0/d6/dc/f11 [5617577,30849] 0 2022-01-31T19:44:06.738 INFO:tasks.workunit.client.0.smithi167.stdout:4/108: write d0/d6/dc/f1d [231903,33630] 0 2022-01-31T19:44:06.738 INFO:tasks.workunit.client.0.smithi167.stdout:1/180: mkdir d0/d11/d45 0 2022-01-31T19:44:06.740 INFO:tasks.workunit.client.0.smithi167.stdout:8/235: mkdir d6/d1e/d4c 0 2022-01-31T19:44:06.741 INFO:tasks.workunit.client.0.smithi167.stdout:4/109: write d0/d6/dc/dd/fe [334774,34262] 0 2022-01-31T19:44:06.741 INFO:tasks.workunit.client.0.smithi167.stdout:1/181: mkdir d0/d11/d46 0 2022-01-31T19:44:06.742 INFO:tasks.workunit.client.0.smithi167.stdout:8/236: creat d6/db/d36/f4d x:0 0 0 2022-01-31T19:44:06.745 INFO:tasks.workunit.client.0.smithi167.stdout:3/181: dread d9/d21/f1e [0,4194304] 0 2022-01-31T19:44:06.746 INFO:tasks.workunit.client.0.smithi167.stdout:3/182: chown d9/f30 16461 1 2022-01-31T19:44:06.748 INFO:tasks.workunit.client.0.smithi167.stdout:3/183: link d9/f25 d9/d26/f31 0 2022-01-31T19:44:06.748 INFO:tasks.workunit.client.0.smithi167.stdout:3/184: dread - d9/f27 zero size 2022-01-31T19:44:06.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/90: dwrite dd/d17/f18 [0,4194304] 0 2022-01-31T19:44:06.749 INFO:tasks.workunit.client.0.smithi167.stdout:3/185: mknod d9/d29/c32 0 2022-01-31T19:44:06.749 INFO:tasks.workunit.client.0.smithi167.stdout:4/110: dread d0/d6/f16 [0,4194304] 0 2022-01-31T19:44:06.750 INFO:tasks.workunit.client.0.smithi167.stdout:4/111: chown d0/d6/dc/f11 24909 1 2022-01-31T19:44:06.750 INFO:tasks.workunit.client.0.smithi167.stdout:3/186: read d9/f10 [2642629,14294] 0 2022-01-31T19:44:06.751 INFO:tasks.workunit.client.0.smithi167.stdout:3/187: write d9/f25 [3011192,124701] 0 2022-01-31T19:44:06.754 INFO:tasks.workunit.client.0.smithi167.stdout:9/91: unlink f5 0 2022-01-31T19:44:06.754 INFO:tasks.workunit.client.0.smithi167.stdout:4/112: dread d0/d6/f16 [0,4194304] 0 2022-01-31T19:44:06.754 INFO:tasks.workunit.client.0.smithi167.stdout:3/188: write d9/f11 [1164445,129295] 0 2022-01-31T19:44:06.756 INFO:tasks.workunit.client.0.smithi167.stdout:8/237: dread d6/ff [4194304,4194304] 0 2022-01-31T19:44:06.757 INFO:tasks.workunit.client.0.smithi167.stdout:4/113: symlink d0/d6/dc/dd/l1f 0 2022-01-31T19:44:06.760 INFO:tasks.workunit.client.0.smithi167.stdout:3/189: link d9/d29/f2b d9/d29/f33 0 2022-01-31T19:44:06.760 INFO:tasks.workunit.client.0.smithi167.stdout:3/190: chown d9/d29/d2e 1 1 2022-01-31T19:44:06.762 INFO:tasks.workunit.client.0.smithi167.stdout:4/114: mknod d0/d6/dc/dd/d1a/c20 0 2022-01-31T19:44:06.763 INFO:tasks.workunit.client.0.smithi167.stdout:4/115: readlink d0/d6/dc/dd/l1f 0 2022-01-31T19:44:06.763 INFO:tasks.workunit.client.0.smithi167.stdout:9/92: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:06.764 INFO:tasks.workunit.client.0.smithi167.stdout:4/116: mkdir d0/d6/d21 0 2022-01-31T19:44:06.766 INFO:tasks.workunit.client.0.smithi167.stdout:9/93: creat dd/f1c x:0 0 0 2022-01-31T19:44:06.766 INFO:tasks.workunit.client.0.smithi167.stdout:9/94: write dd/ff [1629139,27157] 0 2022-01-31T19:44:06.768 INFO:tasks.workunit.client.0.smithi167.stdout:4/117: rename d0/d6/f16 to d0/f22 0 2022-01-31T19:44:06.769 INFO:tasks.workunit.client.0.smithi167.stdout:4/118: read - d0/d6/dc/f17 zero size 2022-01-31T19:44:06.770 INFO:tasks.workunit.client.0.smithi167.stdout:9/95: symlink dd/l1d 0 2022-01-31T19:44:06.772 INFO:tasks.workunit.client.0.smithi167.stdout:3/191: dread d9/f25 [0,4194304] 0 2022-01-31T19:44:06.772 INFO:tasks.workunit.client.0.smithi167.stdout:3/192: chown d9/f30 32009 1 2022-01-31T19:44:06.773 INFO:tasks.workunit.client.0.smithi167.stdout:3/193: symlink d9/d21/l34 0 2022-01-31T19:44:06.773 INFO:tasks.workunit.client.0.smithi167.stdout:3/194: write d9/fc [728243,38417] 0 2022-01-31T19:44:06.774 INFO:tasks.workunit.client.0.smithi167.stdout:3/195: fsync f7 0 2022-01-31T19:44:06.774 INFO:tasks.workunit.client.0.smithi167.stdout:3/196: chown d9/f10 228717810 1 2022-01-31T19:44:06.776 INFO:tasks.workunit.client.0.smithi167.stdout:4/119: fsync d0/f22 0 2022-01-31T19:44:06.777 INFO:tasks.workunit.client.0.smithi167.stdout:4/120: symlink d0/d6/d7/l23 0 2022-01-31T19:44:06.778 INFO:tasks.workunit.client.0.smithi167.stdout:4/121: fdatasync d0/d6/d7/f8 0 2022-01-31T19:44:06.778 INFO:tasks.workunit.client.0.smithi167.stdout:4/122: fsync d0/d6/d7/f14 0 2022-01-31T19:44:06.779 INFO:tasks.workunit.client.0.smithi167.stdout:4/123: creat d0/d6/dc/d1c/f24 x:0 0 0 2022-01-31T19:44:06.779 INFO:tasks.workunit.client.0.smithi167.stdout:4/124: readlink d0/d6/d7/l1e 0 2022-01-31T19:44:06.780 INFO:tasks.workunit.client.0.smithi167.stdout:8/238: dwrite d6/d1e/f41 [0,4194304] 0 2022-01-31T19:44:06.780 INFO:tasks.workunit.client.0.smithi167.stdout:8/239: chown d6/d1e/f29 80 1 2022-01-31T19:44:06.780 INFO:tasks.workunit.client.0.smithi167.stdout:8/240: chown l2 1599 1 2022-01-31T19:44:06.781 INFO:tasks.workunit.client.0.smithi167.stdout:4/125: symlink d0/d6/dc/l25 0 2022-01-31T19:44:06.781 INFO:tasks.workunit.client.0.smithi167.stdout:3/197: fsync f8 0 2022-01-31T19:44:06.781 INFO:tasks.workunit.client.0.smithi167.stdout:3/198: write d9/f13 [1158468,121695] 0 2022-01-31T19:44:06.782 INFO:tasks.workunit.client.0.smithi167.stdout:8/241: mkdir d6/db/d4e 0 2022-01-31T19:44:06.782 INFO:tasks.workunit.client.0.smithi167.stdout:8/242: readlink d6/db/d36/l42 0 2022-01-31T19:44:06.783 INFO:tasks.workunit.client.0.smithi167.stdout:4/126: dread d0/d6/d7/f14 [0,4194304] 0 2022-01-31T19:44:06.783 INFO:tasks.workunit.client.0.smithi167.stdout:4/127: chown d0/d6/dc/dd/l1f 2728 1 2022-01-31T19:44:06.784 INFO:tasks.workunit.client.0.smithi167.stdout:8/243: creat d6/db/d4e/f4f x:0 0 0 2022-01-31T19:44:06.784 INFO:tasks.workunit.client.0.smithi167.stdout:3/199: read f1 [3453448,44467] 0 2022-01-31T19:44:06.784 INFO:tasks.workunit.client.0.smithi167.stdout:3/200: stat d9/d21/f1e 0 2022-01-31T19:44:06.785 INFO:tasks.workunit.client.0.smithi167.stdout:8/244: mknod d6/d1e/d4c/c50 0 2022-01-31T19:44:06.786 INFO:tasks.workunit.client.0.smithi167.stdout:8/245: chown d6/f8 0 1 2022-01-31T19:44:06.786 INFO:tasks.workunit.client.0.smithi167.stdout:8/246: fsync d6/d1e/d2d/f40 0 2022-01-31T19:44:06.786 INFO:tasks.workunit.client.0.smithi167.stdout:9/96: dread dd/d17/f18 [0,4194304] 0 2022-01-31T19:44:06.787 INFO:tasks.workunit.client.0.smithi167.stdout:3/201: unlink d9/d21/f28 0 2022-01-31T19:44:06.789 INFO:tasks.workunit.client.0.smithi167.stdout:9/97: dread f9 [0,4194304] 0 2022-01-31T19:44:06.789 INFO:tasks.workunit.client.0.smithi167.stdout:9/98: fdatasync dd/ff 0 2022-01-31T19:44:06.790 INFO:tasks.workunit.client.0.smithi167.stdout:9/99: chown dd/d19/f1b 16994137 1 2022-01-31T19:44:06.791 INFO:tasks.workunit.client.0.smithi167.stdout:3/202: getdents d9/d29 0 2022-01-31T19:44:06.792 INFO:tasks.workunit.client.0.smithi167.stdout:3/203: write d9/f25 [3932215,20851] 0 2022-01-31T19:44:06.799 INFO:tasks.workunit.client.0.smithi167.stdout:8/247: dread d6/ff [4194304,4194304] 0 2022-01-31T19:44:06.800 INFO:tasks.workunit.client.0.smithi167.stdout:8/248: write d6/db/d36/f4a [1973522,1762] 0 2022-01-31T19:44:06.800 INFO:tasks.workunit.client.0.smithi167.stdout:8/249: stat d6/db/de/d18/l4b 0 2022-01-31T19:44:06.800 INFO:tasks.workunit.client.0.smithi167.stdout:8/250: truncate d6/d1e/d2d/f40 28203 0 2022-01-31T19:44:06.801 INFO:tasks.workunit.client.0.smithi167.stdout:8/251: readlink l3 0 2022-01-31T19:44:06.809 INFO:tasks.workunit.client.0.smithi167.stdout:4/128: dwrite d0/d6/dc/d1c/f24 [0,4194304] 0 2022-01-31T19:44:06.811 INFO:tasks.workunit.client.0.smithi167.stdout:4/129: mkdir d0/d6/d7/d26 0 2022-01-31T19:44:06.817 INFO:tasks.workunit.client.0.smithi167.stdout:9/100: dwrite dd/d17/f18 [4194304,4194304] 0 2022-01-31T19:44:06.819 INFO:tasks.workunit.client.0.smithi167.stdout:9/101: mknod dd/d19/c1e 0 2022-01-31T19:44:06.820 INFO:tasks.workunit.client.0.smithi167.stdout:9/102: read f9 [40592,13677] 0 2022-01-31T19:44:06.821 INFO:tasks.workunit.client.0.smithi167.stdout:3/204: dwrite d9/d29/f33 [0,4194304] 0 2022-01-31T19:44:06.822 INFO:tasks.workunit.client.0.smithi167.stdout:8/252: dwrite d6/db/d36/f4d [0,4194304] 0 2022-01-31T19:44:06.824 INFO:tasks.workunit.client.0.smithi167.stdout:9/103: creat dd/d17/f1f x:0 0 0 2022-01-31T19:44:06.826 INFO:tasks.workunit.client.0.smithi167.stdout:3/205: chown d9/l19 913269923 1 2022-01-31T19:44:06.828 INFO:tasks.workunit.client.0.smithi167.stdout:8/253: mkdir d6/db/d2f/d51 0 2022-01-31T19:44:06.830 INFO:tasks.workunit.client.0.smithi167.stdout:9/104: rename dd/f11 to dd/d17/f20 0 2022-01-31T19:44:06.832 INFO:tasks.workunit.client.0.smithi167.stdout:3/206: truncate d9/f10 1329207 0 2022-01-31T19:44:06.832 INFO:tasks.workunit.client.0.smithi167.stdout:4/130: dwrite d0/d6/dc/d1c/f24 [0,4194304] 0 2022-01-31T19:44:06.833 INFO:tasks.workunit.client.0.smithi167.stdout:9/105: creat dd/d19/f21 x:0 0 0 2022-01-31T19:44:06.834 INFO:tasks.workunit.client.0.smithi167.stdout:9/106: fsync dd/f1c 0 2022-01-31T19:44:06.834 INFO:tasks.workunit.client.0.smithi167.stdout:9/107: read - dd/d19/f1b zero size 2022-01-31T19:44:06.834 INFO:tasks.workunit.client.0.smithi167.stdout:3/207: creat d9/d29/d2e/f35 x:0 0 0 2022-01-31T19:44:06.837 INFO:tasks.workunit.client.0.smithi167.stdout:3/208: getdents d9 0 2022-01-31T19:44:06.840 INFO:tasks.workunit.client.0.smithi167.stdout:3/209: link d9/f18 d9/d29/f36 0 2022-01-31T19:44:06.843 INFO:tasks.workunit.client.0.smithi167.stdout:9/108: dread f4 [0,4194304] 0 2022-01-31T19:44:06.845 INFO:tasks.workunit.client.0.smithi167.stdout:9/109: rename lc to dd/d19/l22 0 2022-01-31T19:44:06.845 INFO:tasks.workunit.client.0.smithi167.stdout:9/110: stat f9 0 2022-01-31T19:44:06.853 INFO:tasks.workunit.client.0.smithi167.stdout:9/111: dread dd/ff [0,4194304] 0 2022-01-31T19:44:06.853 INFO:tasks.workunit.client.0.smithi167.stdout:4/131: dread d0/d6/dc/d1c/f24 [0,4194304] 0 2022-01-31T19:44:06.854 INFO:tasks.workunit.client.0.smithi167.stdout:8/254: dwrite d6/db/d28/f43 [0,4194304] 0 2022-01-31T19:44:06.854 INFO:tasks.workunit.client.0.smithi167.stdout:4/132: truncate d0/d6/dc/f1d 471945 0 2022-01-31T19:44:06.854 INFO:tasks.workunit.client.0.smithi167.stdout:4/133: truncate d0/d6/dc/f19 355473 0 2022-01-31T19:44:06.854 INFO:tasks.workunit.client.0.smithi167.stdout:4/134: write d0/d6/dc/f11 [9060470,113511] 0 2022-01-31T19:44:06.855 INFO:tasks.workunit.client.0.smithi167.stdout:3/210: dread d9/d29/f33 [0,4194304] 0 2022-01-31T19:44:06.858 INFO:tasks.workunit.client.0.smithi167.stdout:8/255: rmdir d6/db/d36/d48 0 2022-01-31T19:44:06.859 INFO:tasks.workunit.client.0.smithi167.stdout:4/135: rename d0/d6/dc/f1d to d0/d6/dc/d1c/f27 0 2022-01-31T19:44:06.860 INFO:tasks.workunit.client.0.smithi167.stdout:8/256: mkdir d6/d52 0 2022-01-31T19:44:06.862 INFO:tasks.workunit.client.0.smithi167.stdout:4/136: rename d0/d6/d7/l23 to d0/l28 0 2022-01-31T19:44:06.865 INFO:tasks.workunit.client.0.smithi167.stdout:4/137: creat d0/d6/dc/dd/f29 x:0 0 0 2022-01-31T19:44:06.868 INFO:tasks.workunit.client.0.smithi167.stdout:3/211: dwrite d9/f27 [0,4194304] 0 2022-01-31T19:44:06.868 INFO:tasks.workunit.client.0.smithi167.stdout:3/212: fsync d9/f10 0 2022-01-31T19:44:06.869 INFO:tasks.workunit.client.0.smithi167.stdout:8/257: dread d6/f14 [0,4194304] 0 2022-01-31T19:44:06.873 INFO:tasks.workunit.client.0.smithi167.stdout:6/113: sync 2022-01-31T19:44:06.873 INFO:tasks.workunit.client.0.smithi167.stdout:5/239: sync 2022-01-31T19:44:06.873 INFO:tasks.workunit.client.0.smithi167.stdout:7/169: sync 2022-01-31T19:44:06.874 INFO:tasks.workunit.client.0.smithi167.stdout:7/170: creat d8/dc/d12/f44 x:0 0 0 2022-01-31T19:44:06.874 INFO:tasks.workunit.client.0.smithi167.stdout:2/118: sync 2022-01-31T19:44:06.875 INFO:tasks.workunit.client.0.smithi167.stdout:0/70: sync 2022-01-31T19:44:06.875 INFO:tasks.workunit.client.0.smithi167.stdout:2/119: fdatasync dd/f1a 0 2022-01-31T19:44:06.875 INFO:tasks.workunit.client.0.smithi167.stdout:0/71: unlink l3 0 2022-01-31T19:44:06.876 INFO:tasks.workunit.client.0.smithi167.stdout:3/213: link d9/d29/c2f d9/d29/c37 0 2022-01-31T19:44:06.877 INFO:tasks.workunit.client.0.smithi167.stdout:5/240: rmdir d5/d6/d16/d18/d43 39 2022-01-31T19:44:06.883 INFO:tasks.workunit.client.0.smithi167.stdout:1/182: sync 2022-01-31T19:44:06.883 INFO:tasks.workunit.client.0.smithi167.stdout:4/138: dread d0/d6/dc/dd/fe [0,4194304] 0 2022-01-31T19:44:06.883 INFO:tasks.workunit.client.0.smithi167.stdout:4/139: readlink d0/l1b 0 2022-01-31T19:44:06.884 INFO:tasks.workunit.client.0.smithi167.stdout:4/140: creat d0/d6/dc/f2a x:0 0 0 2022-01-31T19:44:06.884 INFO:tasks.workunit.client.0.smithi167.stdout:4/141: chown d0/d6/d7 7673 1 2022-01-31T19:44:06.885 INFO:tasks.workunit.client.0.smithi167.stdout:0/72: mknod d17/c1a 0 2022-01-31T19:44:06.885 INFO:tasks.workunit.client.0.smithi167.stdout:0/73: chown f15 476594 1 2022-01-31T19:44:06.885 INFO:tasks.workunit.client.0.smithi167.stdout:0/74: write f14 [834626,49164] 0 2022-01-31T19:44:06.886 INFO:tasks.workunit.client.0.smithi167.stdout:2/120: mkdir dd/df/d22/d25 0 2022-01-31T19:44:06.887 INFO:tasks.workunit.client.0.smithi167.stdout:5/241: creat d5/d6/d16/d24/f4b x:0 0 0 2022-01-31T19:44:06.888 INFO:tasks.workunit.client.0.smithi167.stdout:1/183: mknod d0/d10/d18/d16/d26/c47 0 2022-01-31T19:44:06.890 INFO:tasks.workunit.client.0.smithi167.stdout:4/142: truncate d0/d6/dc/dd/fe 3423694 0 2022-01-31T19:44:06.892 INFO:tasks.workunit.client.0.smithi167.stdout:0/75: creat d17/d19/f1b x:0 0 0 2022-01-31T19:44:06.893 INFO:tasks.workunit.client.0.smithi167.stdout:2/121: rmdir dd/d18 39 2022-01-31T19:44:06.896 INFO:tasks.workunit.client.0.smithi167.stdout:5/242: creat d5/d6/de/f4c x:0 0 0 2022-01-31T19:44:06.898 INFO:tasks.workunit.client.0.smithi167.stdout:1/184: mknod d0/d10/d18/d16/c48 0 2022-01-31T19:44:06.900 INFO:tasks.workunit.client.0.smithi167.stdout:5/243: mkdir d5/d6/de/d22/d4d 0 2022-01-31T19:44:06.900 INFO:tasks.workunit.client.0.smithi167.stdout:5/244: creat d5/d6/d3a/f4e x:0 0 0 2022-01-31T19:44:06.905 INFO:tasks.workunit.client.0.smithi167.stdout:1/185: unlink d0/d10/d18/d16/d1f/f29 0 2022-01-31T19:44:06.905 INFO:tasks.workunit.client.0.smithi167.stdout:1/186: readlink d0/d10/d20/l40 0 2022-01-31T19:44:06.905 INFO:tasks.workunit.client.0.smithi167.stdout:3/214: dread f1 [0,4194304] 0 2022-01-31T19:44:06.905 INFO:tasks.workunit.client.0.smithi167.stdout:8/258: dwrite d6/f32 [0,4194304] 0 2022-01-31T19:44:06.907 INFO:tasks.workunit.client.0.smithi167.stdout:0/76: write fc [4036017,100574] 0 2022-01-31T19:44:06.907 INFO:tasks.workunit.client.0.smithi167.stdout:0/77: chown f15 14 1 2022-01-31T19:44:06.907 INFO:tasks.workunit.client.0.smithi167.stdout:0/78: readlink l12 0 2022-01-31T19:44:06.909 INFO:tasks.workunit.client.0.smithi167.stdout:5/245: rename d5/d6/d16/d18/d43/l4a to d5/d6/d16/d18/l4f 0 2022-01-31T19:44:06.911 INFO:tasks.workunit.client.0.smithi167.stdout:6/114: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:06.912 INFO:tasks.workunit.client.0.smithi167.stdout:3/215: truncate d9/d26/f31 3005897 0 2022-01-31T19:44:06.912 INFO:tasks.workunit.client.0.smithi167.stdout:0/79: getdents d17 0 2022-01-31T19:44:06.913 INFO:tasks.workunit.client.0.smithi167.stdout:0/80: readlink l12 0 2022-01-31T19:44:06.913 INFO:tasks.workunit.client.0.smithi167.stdout:0/81: unlink f13 0 2022-01-31T19:44:06.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/82: mkdir d17/d19/d1c 0 2022-01-31T19:44:06.918 INFO:tasks.workunit.client.0.smithi167.stdout:6/115: mkdir d8/d2a 0 2022-01-31T19:44:06.920 INFO:tasks.workunit.client.0.smithi167.stdout:3/216: write d9/d29/f2b [3539643,54911] 0 2022-01-31T19:44:06.922 INFO:tasks.workunit.client.0.smithi167.stdout:7/171: dwrite d8/dc/d19/f42 [0,4194304] 0 2022-01-31T19:44:06.925 INFO:tasks.workunit.client.0.smithi167.stdout:4/143: dwrite d0/d6/dc/f19 [0,4194304] 0 2022-01-31T19:44:06.925 INFO:tasks.workunit.client.0.smithi167.stdout:2/122: dwrite dd/df/f20 [0,4194304] 0 2022-01-31T19:44:06.938 INFO:tasks.workunit.client.0.smithi167.stdout:2/123: dread fa [0,4194304] 0 2022-01-31T19:44:06.938 INFO:tasks.workunit.client.0.smithi167.stdout:2/124: chown dd/df/c10 154 1 2022-01-31T19:44:06.939 INFO:tasks.workunit.client.0.smithi167.stdout:1/187: dwrite d0/d11/d24/f3f [0,4194304] 0 2022-01-31T19:44:06.939 INFO:tasks.workunit.client.0.smithi167.stdout:8/259: dwrite d6/db/de/f3e [0,4194304] 0 2022-01-31T19:44:06.944 INFO:tasks.workunit.client.0.smithi167.stdout:2/125: dread f7 [0,4194304] 0 2022-01-31T19:44:06.944 INFO:tasks.workunit.client.0.smithi167.stdout:8/260: mknod d6/db/de/d3d/c53 0 2022-01-31T19:44:06.948 INFO:tasks.workunit.client.0.smithi167.stdout:2/126: read f8 [1642058,102256] 0 2022-01-31T19:44:06.948 INFO:tasks.workunit.client.0.smithi167.stdout:2/127: stat dd/df/d1b 0 2022-01-31T19:44:06.948 INFO:tasks.workunit.client.0.smithi167.stdout:2/128: stat fa 0 2022-01-31T19:44:06.948 INFO:tasks.workunit.client.0.smithi167.stdout:7/172: dwrite d8/dc/d18/d24/f26 [0,4194304] 0 2022-01-31T19:44:06.949 INFO:tasks.workunit.client.0.smithi167.stdout:7/173: fdatasync d8/fa 0 2022-01-31T19:44:06.949 INFO:tasks.workunit.client.0.smithi167.stdout:7/174: write d8/dc/d12/f44 [371360,55880] 0 2022-01-31T19:44:06.949 INFO:tasks.workunit.client.0.smithi167.stdout:7/175: write f7 [5516544,67937] 0 2022-01-31T19:44:06.949 INFO:tasks.workunit.client.0.smithi167.stdout:6/116: dwrite d8/fb [0,4194304] 0 2022-01-31T19:44:06.949 INFO:tasks.workunit.client.0.smithi167.stdout:6/117: read - d8/f1e zero size 2022-01-31T19:44:06.950 INFO:tasks.workunit.client.0.smithi167.stdout:2/129: getdents dd/df/d22 0 2022-01-31T19:44:06.950 INFO:tasks.workunit.client.0.smithi167.stdout:3/217: dwrite f1 [0,4194304] 0 2022-01-31T19:44:06.952 INFO:tasks.workunit.client.0.smithi167.stdout:6/118: dread d8/f16 [0,4194304] 0 2022-01-31T19:44:06.953 INFO:tasks.workunit.client.0.smithi167.stdout:6/119: readlink d8/l17 0 2022-01-31T19:44:06.958 INFO:tasks.workunit.client.0.smithi167.stdout:7/176: dread d8/dc/d18/d24/f26 [0,4194304] 0 2022-01-31T19:44:06.961 INFO:tasks.workunit.client.0.smithi167.stdout:4/144: truncate d0/d6/dc/f19 1747507 0 2022-01-31T19:44:06.961 INFO:tasks.workunit.client.0.smithi167.stdout:2/130: symlink dd/df/d22/l26 0 2022-01-31T19:44:06.963 INFO:tasks.workunit.client.0.smithi167.stdout:1/188: dwrite d0/d10/f42 [0,4194304] 0 2022-01-31T19:44:06.963 INFO:tasks.workunit.client.0.smithi167.stdout:8/261: dread d6/db/de/f3e [0,4194304] 0 2022-01-31T19:44:06.963 INFO:tasks.workunit.client.0.smithi167.stdout:8/262: chown d6/db/d2f 599 1 2022-01-31T19:44:06.964 INFO:tasks.workunit.client.0.smithi167.stdout:8/263: chown d6/f1d 1015 1 2022-01-31T19:44:06.964 INFO:tasks.workunit.client.0.smithi167.stdout:8/264: write d6/db/de/f26 [670336,63853] 0 2022-01-31T19:44:06.966 INFO:tasks.workunit.client.0.smithi167.stdout:2/131: write dd/f1a [2755064,39160] 0 2022-01-31T19:44:06.967 INFO:tasks.workunit.client.0.smithi167.stdout:4/145: link d0/d6/dc/d1c/f24 d0/d6/d21/f2b 0 2022-01-31T19:44:06.972 INFO:tasks.workunit.client.0.smithi167.stdout:4/146: rename d0/d6/dc/dd/cf to d0/d6/dc/d1c/c2c 0 2022-01-31T19:44:06.972 INFO:tasks.workunit.client.0.smithi167.stdout:4/147: truncate d0/d6/dc/d1c/f27 808328 0 2022-01-31T19:44:06.973 INFO:tasks.workunit.client.0.smithi167.stdout:4/148: write d0/f22 [1160166,69363] 0 2022-01-31T19:44:06.973 INFO:tasks.workunit.client.0.smithi167.stdout:2/132: link dd/l14 dd/df/d22/d25/l27 0 2022-01-31T19:44:06.984 INFO:tasks.workunit.client.0.smithi167.stdout:3/218: dwrite d9/f25 [0,4194304] 0 2022-01-31T19:44:06.984 INFO:tasks.workunit.client.0.smithi167.stdout:3/219: write d9/f30 [295899,120921] 0 2022-01-31T19:44:06.986 INFO:tasks.workunit.client.0.smithi167.stdout:6/120: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:06.987 INFO:tasks.workunit.client.0.smithi167.stdout:2/133: dread dd/df/f20 [0,4194304] 0 2022-01-31T19:44:06.987 INFO:tasks.workunit.client.0.smithi167.stdout:2/134: readlink dd/l14 0 2022-01-31T19:44:06.987 INFO:tasks.workunit.client.0.smithi167.stdout:2/135: stat dd/f15 0 2022-01-31T19:44:06.988 INFO:tasks.workunit.client.0.smithi167.stdout:1/189: dwrite d0/f3a [0,4194304] 0 2022-01-31T19:44:06.990 INFO:tasks.workunit.client.0.smithi167.stdout:4/149: write d0/d6/dc/f19 [817167,8157] 0 2022-01-31T19:44:06.992 INFO:tasks.workunit.client.0.smithi167.stdout:6/121: read d8/f19 [394906,77171] 0 2022-01-31T19:44:06.992 INFO:tasks.workunit.client.0.smithi167.stdout:6/122: fsync d8/fd 0 2022-01-31T19:44:06.993 INFO:tasks.workunit.client.0.smithi167.stdout:8/265: dwrite d6/db/d36/f4a [0,4194304] 0 2022-01-31T19:44:06.993 INFO:tasks.workunit.client.0.smithi167.stdout:2/136: symlink dd/df/d1b/l28 0 2022-01-31T19:44:06.999 INFO:tasks.workunit.client.0.smithi167.stdout:6/123: read d8/fa [2200007,5909] 0 2022-01-31T19:44:07.001 INFO:tasks.workunit.client.0.smithi167.stdout:8/266: rename d6/d1e/d2d/l49 to d6/l54 0 2022-01-31T19:44:07.001 INFO:tasks.workunit.client.0.smithi167.stdout:8/267: creat d6/db/de/d18/f55 x:0 0 0 2022-01-31T19:44:07.002 INFO:tasks.workunit.client.0.smithi167.stdout:2/137: unlink dd/df/f20 0 2022-01-31T19:44:07.005 INFO:tasks.workunit.client.0.smithi167.stdout:6/124: getdents d8/d20 0 2022-01-31T19:44:07.008 INFO:tasks.workunit.client.0.smithi167.stdout:6/125: symlink d8/d20/l2b 0 2022-01-31T19:44:07.008 INFO:tasks.workunit.client.0.smithi167.stdout:6/126: chown f3 28543 1 2022-01-31T19:44:07.009 INFO:tasks.workunit.client.0.smithi167.stdout:3/220: dwrite d9/f27 [0,4194304] 0 2022-01-31T19:44:07.010 INFO:tasks.workunit.client.0.smithi167.stdout:3/221: fdatasync d9/f11 0 2022-01-31T19:44:07.010 INFO:tasks.workunit.client.0.smithi167.stdout:3/222: symlink d9/d26/l38 0 2022-01-31T19:44:07.011 INFO:tasks.workunit.client.0.smithi167.stdout:3/223: creat d9/d29/d2e/f39 x:0 0 0 2022-01-31T19:44:07.033 INFO:tasks.workunit.client.0.smithi167.stdout:1/190: dwrite d0/f25 [0,4194304] 0 2022-01-31T19:44:07.036 INFO:tasks.workunit.client.0.smithi167.stdout:1/191: rename d0/d11/d24/c2f to d0/c49 0 2022-01-31T19:44:07.036 INFO:tasks.workunit.client.0.smithi167.stdout:1/192: dread - d0/d10/d20/f37 zero size 2022-01-31T19:44:07.041 INFO:tasks.workunit.client.0.smithi167.stdout:4/150: dwrite d0/d6/dc/f11 [4194304,4194304] 0 2022-01-31T19:44:07.041 INFO:tasks.workunit.client.0.smithi167.stdout:8/268: dwrite d6/d1e/f41 [0,4194304] 0 2022-01-31T19:44:07.042 INFO:tasks.workunit.client.0.smithi167.stdout:2/138: dwrite dd/df/f1c [0,4194304] 0 2022-01-31T19:44:07.043 INFO:tasks.workunit.client.0.smithi167.stdout:4/151: getdents d0/d6/d21 0 2022-01-31T19:44:07.043 INFO:tasks.workunit.client.0.smithi167.stdout:6/127: dwrite d8/fb [0,4194304] 0 2022-01-31T19:44:07.045 INFO:tasks.workunit.client.0.smithi167.stdout:8/269: creat d6/db/de/d3d/f56 x:0 0 0 2022-01-31T19:44:07.046 INFO:tasks.workunit.client.0.smithi167.stdout:1/193: dread d0/f5 [0,4194304] 0 2022-01-31T19:44:07.052 INFO:tasks.workunit.client.0.smithi167.stdout:3/224: truncate d9/f27 2258877 0 2022-01-31T19:44:07.052 INFO:tasks.workunit.client.0.smithi167.stdout:3/225: creat d9/d21/f3a x:0 0 0 2022-01-31T19:44:07.053 INFO:tasks.workunit.client.0.smithi167.stdout:2/139: link dd/df/f1c dd/f29 0 2022-01-31T19:44:07.054 INFO:tasks.workunit.client.0.smithi167.stdout:4/152: mknod d0/d6/d21/c2d 0 2022-01-31T19:44:07.055 INFO:tasks.workunit.client.0.smithi167.stdout:6/128: symlink d8/d2a/l2c 0 2022-01-31T19:44:07.059 INFO:tasks.workunit.client.0.smithi167.stdout:6/129: creat d8/f2d x:0 0 0 2022-01-31T19:44:07.060 INFO:tasks.workunit.client.0.smithi167.stdout:6/130: fdatasync d8/fd 0 2022-01-31T19:44:07.060 INFO:tasks.workunit.client.0.smithi167.stdout:6/131: readlink l1 0 2022-01-31T19:44:07.062 INFO:tasks.workunit.client.0.smithi167.stdout:6/132: creat d8/d2a/f2e x:0 0 0 2022-01-31T19:44:07.067 INFO:tasks.workunit.client.0.smithi167.stdout:4/153: dread d0/d6/dc/d1c/f27 [0,4194304] 0 2022-01-31T19:44:07.067 INFO:tasks.workunit.client.0.smithi167.stdout:4/154: creat d0/d6/dc/dd/f2e x:0 0 0 2022-01-31T19:44:07.067 INFO:tasks.workunit.client.0.smithi167.stdout:4/155: stat d0/d6/dc/l10 0 2022-01-31T19:44:07.068 INFO:tasks.workunit.client.0.smithi167.stdout:1/194: dwrite d0/d10/d18/f2b [0,4194304] 0 2022-01-31T19:44:07.070 INFO:tasks.workunit.client.0.smithi167.stdout:4/156: creat d0/d6/d7/d26/f2f x:0 0 0 2022-01-31T19:44:07.071 INFO:tasks.workunit.client.0.smithi167.stdout:4/157: creat d0/d6/d7/f30 x:0 0 0 2022-01-31T19:44:07.073 INFO:tasks.workunit.client.0.smithi167.stdout:1/195: symlink d0/d10/d18/d16/d26/l4a 0 2022-01-31T19:44:07.081 INFO:tasks.workunit.client.0.smithi167.stdout:8/270: dwrite d6/f8 [4194304,4194304] 0 2022-01-31T19:44:07.082 INFO:tasks.workunit.client.0.smithi167.stdout:3/226: dwrite d9/f1a [4194304,4194304] 0 2022-01-31T19:44:07.083 INFO:tasks.workunit.client.0.smithi167.stdout:3/227: dread - d9/d21/f3a zero size 2022-01-31T19:44:07.083 INFO:tasks.workunit.client.0.smithi167.stdout:2/140: dwrite dd/f29 [0,4194304] 0 2022-01-31T19:44:07.084 INFO:tasks.workunit.client.0.smithi167.stdout:3/228: read d9/d21/f1e [2335893,38269] 0 2022-01-31T19:44:07.084 INFO:tasks.workunit.client.0.smithi167.stdout:3/229: stat d9/d29/d2e/f35 0 2022-01-31T19:44:07.093 INFO:tasks.workunit.client.0.smithi167.stdout:4/158: rmdir d0/d6/dc 39 2022-01-31T19:44:07.093 INFO:tasks.workunit.client.0.smithi167.stdout:4/159: chown d0/d6/dc/f17 108827 1 2022-01-31T19:44:07.094 INFO:tasks.workunit.client.0.smithi167.stdout:4/160: chown d0/d6/d7/d26 7345613 1 2022-01-31T19:44:07.094 INFO:tasks.workunit.client.0.smithi167.stdout:4/161: chown d0/d6/dc/f19 146434746 1 2022-01-31T19:44:07.117 INFO:tasks.workunit.client.0.smithi167.stdout:2/141: dwrite dd/f23 [4194304,4194304] 0 2022-01-31T19:44:07.117 INFO:tasks.workunit.client.0.smithi167.stdout:3/230: dwrite d9/d29/d2e/f35 [0,4194304] 0 2022-01-31T19:44:07.118 INFO:tasks.workunit.client.0.smithi167.stdout:8/271: dwrite d6/db/d36/f3b [0,4194304] 0 2022-01-31T19:44:07.119 INFO:tasks.workunit.client.0.smithi167.stdout:3/231: mkdir d9/d29/d3b 0 2022-01-31T19:44:07.119 INFO:tasks.workunit.client.0.smithi167.stdout:4/162: dwrite d0/d6/dc/f2a [0,4194304] 0 2022-01-31T19:44:07.120 INFO:tasks.workunit.client.0.smithi167.stdout:4/163: read d0/d6/dc/d1c/f24 [1191285,88484] 0 2022-01-31T19:44:07.120 INFO:tasks.workunit.client.0.smithi167.stdout:2/142: getdents dd/df 0 2022-01-31T19:44:07.123 INFO:tasks.workunit.client.0.smithi167.stdout:8/272: link d6/db/de/f3e d6/f57 0 2022-01-31T19:44:07.125 INFO:tasks.workunit.client.0.smithi167.stdout:3/232: rename f1 to d9/d29/f3c 0 2022-01-31T19:44:07.127 INFO:tasks.workunit.client.0.smithi167.stdout:4/164: getdents d0/d6 0 2022-01-31T19:44:07.129 INFO:tasks.workunit.client.0.smithi167.stdout:2/143: creat dd/df/d22/d25/f2a x:0 0 0 2022-01-31T19:44:07.129 INFO:tasks.workunit.client.0.smithi167.stdout:2/144: fsync f8 0 2022-01-31T19:44:07.131 INFO:tasks.workunit.client.0.smithi167.stdout:2/145: mknod dd/df/d22/c2b 0 2022-01-31T19:44:07.133 INFO:tasks.workunit.client.0.smithi167.stdout:2/146: mkdir dd/df/d2c 0 2022-01-31T19:44:07.147 INFO:tasks.workunit.client.0.smithi167.stdout:3/233: dwrite f8 [0,4194304] 0 2022-01-31T19:44:07.210 INFO:tasks.workunit.client.0.smithi167.stdout:3/234: dread d9/d29/f33 [0,4194304] 0 2022-01-31T19:44:07.210 INFO:tasks.workunit.client.0.smithi167.stdout:3/235: write d9/d21/f3a [573469,15235] 0 2022-01-31T19:44:07.211 INFO:tasks.workunit.client.0.smithi167.stdout:8/273: dwrite d6/db/d36/f4d [0,4194304] 0 2022-01-31T19:44:07.212 INFO:tasks.workunit.client.0.smithi167.stdout:3/236: symlink d9/d29/d3b/l3d 0 2022-01-31T19:44:07.212 INFO:tasks.workunit.client.0.smithi167.stdout:3/237: creat d9/d29/d2e/f3e x:0 0 0 2022-01-31T19:44:07.212 INFO:tasks.workunit.client.0.smithi167.stdout:3/238: chown d9/f18 3422298 1 2022-01-31T19:44:07.212 INFO:tasks.workunit.client.0.smithi167.stdout:3/239: creat d9/d21/f3f x:0 0 0 2022-01-31T19:44:07.213 INFO:tasks.workunit.client.0.smithi167.stdout:8/274: rename d6/f13 to d6/db/d4e/f58 0 2022-01-31T19:44:07.214 INFO:tasks.workunit.client.0.smithi167.stdout:3/240: creat d9/f40 x:0 0 0 2022-01-31T19:44:07.215 INFO:tasks.workunit.client.0.smithi167.stdout:8/275: symlink d6/db/d2f/d51/l59 0 2022-01-31T19:44:07.216 INFO:tasks.workunit.client.0.smithi167.stdout:8/276: creat d6/db/d28/f5a x:0 0 0 2022-01-31T19:44:07.216 INFO:tasks.workunit.client.0.smithi167.stdout:8/277: stat d6/db/de/d3d/f56 0 2022-01-31T19:44:07.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/241: dread d9/d29/f36 [0,4194304] 0 2022-01-31T19:44:07.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/242: fsync d9/d26/f31 0 2022-01-31T19:44:07.217 INFO:tasks.workunit.client.0.smithi167.stdout:2/147: dwrite f9 [0,4194304] 0 2022-01-31T19:44:07.217 INFO:tasks.workunit.client.0.smithi167.stdout:8/278: rmdir d6/db/d2f 39 2022-01-31T19:44:07.218 INFO:tasks.workunit.client.0.smithi167.stdout:8/279: chown d6/db/d2f 4361229 1 2022-01-31T19:44:07.218 INFO:tasks.workunit.client.0.smithi167.stdout:2/148: link dd/f15 dd/df/d2c/f2d 0 2022-01-31T19:44:07.232 INFO:tasks.workunit.client.0.smithi167.stdout:2/149: write dd/f29 [2302106,7972] 0 2022-01-31T19:44:07.233 INFO:tasks.workunit.client.0.smithi167.stdout:2/150: link dd/c17 dd/df/d22/d25/c2e 0 2022-01-31T19:44:07.234 INFO:tasks.workunit.client.0.smithi167.stdout:2/151: mkdir dd/df/d1b/d2f 0 2022-01-31T19:44:07.236 INFO:tasks.workunit.client.0.smithi167.stdout:2/152: creat dd/d18/f30 x:0 0 0 2022-01-31T19:44:07.236 INFO:tasks.workunit.client.0.smithi167.stdout:2/153: getdents dd/df/d1b/d2f 0 2022-01-31T19:44:07.236 INFO:tasks.workunit.client.0.smithi167.stdout:2/154: creat dd/d18/f31 x:0 0 0 2022-01-31T19:44:07.237 INFO:tasks.workunit.client.0.smithi167.stdout:3/243: dwrite d9/f18 [0,4194304] 0 2022-01-31T19:44:07.237 INFO:tasks.workunit.client.0.smithi167.stdout:3/244: readlink d9/le 0 2022-01-31T19:44:07.238 INFO:tasks.workunit.client.0.smithi167.stdout:2/155: getdents dd/df/d22 0 2022-01-31T19:44:07.239 INFO:tasks.workunit.client.0.smithi167.stdout:2/156: dread - dd/df/d22/d25/f2a zero size 2022-01-31T19:44:07.239 INFO:tasks.workunit.client.0.smithi167.stdout:2/157: write dd/df/d22/d25/f2a [395090,57003] 0 2022-01-31T19:44:07.240 INFO:tasks.workunit.client.0.smithi167.stdout:3/245: getdents d9/d26 0 2022-01-31T19:44:07.242 INFO:tasks.workunit.client.0.smithi167.stdout:3/246: rmdir d9/d21 39 2022-01-31T19:44:07.243 INFO:tasks.workunit.client.0.smithi167.stdout:3/247: dread d9/f30 [0,4194304] 0 2022-01-31T19:44:07.256 INFO:tasks.workunit.client.0.smithi167.stdout:8/280: dwrite d6/db/de/d3d/f56 [0,4194304] 0 2022-01-31T19:44:07.256 INFO:tasks.workunit.client.0.smithi167.stdout:2/158: dread f7 [0,4194304] 0 2022-01-31T19:44:07.256 INFO:tasks.workunit.client.0.smithi167.stdout:8/281: stat d6/db/f2b 0 2022-01-31T19:44:07.260 INFO:tasks.workunit.client.0.smithi167.stdout:8/282: creat d6/db/f5b x:0 0 0 2022-01-31T19:44:07.260 INFO:tasks.workunit.client.0.smithi167.stdout:8/283: symlink d6/db/de/l5c 0 2022-01-31T19:44:07.261 INFO:tasks.workunit.client.0.smithi167.stdout:8/284: truncate d6/d1e/f29 337659 0 2022-01-31T19:44:07.263 INFO:tasks.workunit.client.0.smithi167.stdout:8/285: rename d6/d52 to d6/db/d36/d5d 0 2022-01-31T19:44:07.265 INFO:tasks.workunit.client.0.smithi167.stdout:2/159: dread f9 [0,4194304] 0 2022-01-31T19:44:07.266 INFO:tasks.workunit.client.0.smithi167.stdout:8/286: read d6/d1e/f41 [2094283,88450] 0 2022-01-31T19:44:07.266 INFO:tasks.workunit.client.0.smithi167.stdout:3/248: dwrite d9/f18 [0,4194304] 0 2022-01-31T19:44:07.266 INFO:tasks.workunit.client.0.smithi167.stdout:2/160: creat dd/df/d2c/f32 x:0 0 0 2022-01-31T19:44:07.267 INFO:tasks.workunit.client.0.smithi167.stdout:8/287: getdents d6/d1e/d4c 0 2022-01-31T19:44:07.268 INFO:tasks.workunit.client.0.smithi167.stdout:3/249: write d9/f10 [148469,3317] 0 2022-01-31T19:44:07.268 INFO:tasks.workunit.client.0.smithi167.stdout:8/288: truncate d6/db/de/f26 526899 0 2022-01-31T19:44:07.270 INFO:tasks.workunit.client.0.smithi167.stdout:3/250: mkdir d9/d29/d3b/d41 0 2022-01-31T19:44:07.271 INFO:tasks.workunit.client.0.smithi167.stdout:8/289: mknod d6/db/de/d3d/c5e 0 2022-01-31T19:44:07.273 INFO:tasks.workunit.client.0.smithi167.stdout:8/290: unlink d6/db/de/d3d/c53 0 2022-01-31T19:44:07.273 INFO:tasks.workunit.client.0.smithi167.stdout:8/291: chown d6/db/de 0 1 2022-01-31T19:44:07.274 INFO:tasks.workunit.client.0.smithi167.stdout:8/292: stat d6/f8 0 2022-01-31T19:44:07.275 INFO:tasks.workunit.client.0.smithi167.stdout:9/112: sync 2022-01-31T19:44:07.276 INFO:tasks.workunit.client.0.smithi167.stdout:6/133: sync 2022-01-31T19:44:07.276 INFO:tasks.workunit.client.0.smithi167.stdout:5/246: sync 2022-01-31T19:44:07.277 INFO:tasks.workunit.client.0.smithi167.stdout:7/177: sync 2022-01-31T19:44:07.277 INFO:tasks.workunit.client.0.smithi167.stdout:0/83: sync 2022-01-31T19:44:07.277 INFO:tasks.workunit.client.0.smithi167.stdout:7/178: write d8/dc/d18/d24/f26 [4629305,16029] 0 2022-01-31T19:44:07.279 INFO:tasks.workunit.client.0.smithi167.stdout:5/247: symlink d5/d6/de/l50 0 2022-01-31T19:44:07.279 INFO:tasks.workunit.client.0.smithi167.stdout:5/248: stat d5/d6 0 2022-01-31T19:44:07.279 INFO:tasks.workunit.client.0.smithi167.stdout:5/249: write d5/d6/d16/d18/f3d [782771,129864] 0 2022-01-31T19:44:07.280 INFO:tasks.workunit.client.0.smithi167.stdout:5/250: chown d5/d6/d16/d18/f2c 64 1 2022-01-31T19:44:07.280 INFO:tasks.workunit.client.0.smithi167.stdout:5/251: readlink d5/d6/d3a/l11 0 2022-01-31T19:44:07.280 INFO:tasks.workunit.client.0.smithi167.stdout:6/134: write d8/d20/f21 [99153,61615] 0 2022-01-31T19:44:07.281 INFO:tasks.workunit.client.0.smithi167.stdout:9/113: getdents dd 0 2022-01-31T19:44:07.281 INFO:tasks.workunit.client.0.smithi167.stdout:5/252: dread d5/d6/d16/d18/f2c [0,4194304] 0 2022-01-31T19:44:07.285 INFO:tasks.workunit.client.0.smithi167.stdout:6/135: creat d8/d20/f2f x:0 0 0 2022-01-31T19:44:07.296 INFO:tasks.workunit.client.0.smithi167.stdout:9/114: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:07.299 INFO:tasks.workunit.client.0.smithi167.stdout:9/115: dread dd/ff [0,4194304] 0 2022-01-31T19:44:07.299 INFO:tasks.workunit.client.0.smithi167.stdout:9/116: chown dd/d19/f21 230081 1 2022-01-31T19:44:07.299 INFO:tasks.workunit.client.0.smithi167.stdout:9/117: chown dd/f1a 1830 1 2022-01-31T19:44:07.300 INFO:tasks.workunit.client.0.smithi167.stdout:6/136: write d8/fb [594605,7047] 0 2022-01-31T19:44:07.301 INFO:tasks.workunit.client.0.smithi167.stdout:2/161: dwrite fa [4194304,4194304] 0 2022-01-31T19:44:07.301 INFO:tasks.workunit.client.0.smithi167.stdout:0/84: dread f15 [0,4194304] 0 2022-01-31T19:44:07.301 INFO:tasks.workunit.client.0.smithi167.stdout:2/162: truncate dd/d18/f30 1002161 0 2022-01-31T19:44:07.307 INFO:tasks.workunit.client.0.smithi167.stdout:7/179: dwrite d8/dc/d19/f42 [0,4194304] 0 2022-01-31T19:44:07.322 INFO:tasks.workunit.client.0.smithi167.stdout:9/118: dwrite dd/d19/f21 [0,4194304] 0 2022-01-31T19:44:07.322 INFO:tasks.workunit.client.0.smithi167.stdout:7/180: dread d8/dc/d19/f42 [0,4194304] 0 2022-01-31T19:44:07.328 INFO:tasks.workunit.client.0.smithi167.stdout:0/85: dwrite f15 [0,4194304] 0 2022-01-31T19:44:07.337 INFO:tasks.workunit.client.0.smithi167.stdout:7/181: dwrite f7 [4194304,4194304] 0 2022-01-31T19:44:07.354 INFO:tasks.workunit.client.0.smithi167.stdout:7/182: dwrite d8/dc/f23 [0,4194304] 0 2022-01-31T19:44:07.358 INFO:tasks.workunit.client.0.smithi167.stdout:1/196: sync 2022-01-31T19:44:07.358 INFO:tasks.workunit.client.0.smithi167.stdout:4/165: sync 2022-01-31T19:44:07.360 INFO:tasks.workunit.client.0.smithi167.stdout:0/86: truncate f14 20363 0 2022-01-31T19:44:07.361 INFO:tasks.workunit.client.0.smithi167.stdout:0/87: fsync fc 0 2022-01-31T19:44:07.364 INFO:tasks.workunit.client.0.smithi167.stdout:7/183: creat d8/f45 x:0 0 0 2022-01-31T19:44:07.365 INFO:tasks.workunit.client.0.smithi167.stdout:2/163: rmdir dd/df/d22/d25 39 2022-01-31T19:44:07.365 INFO:tasks.workunit.client.0.smithi167.stdout:1/197: dread d0/fc [0,4194304] 0 2022-01-31T19:44:07.366 INFO:tasks.workunit.client.0.smithi167.stdout:2/164: write dd/d18/f30 [255050,89619] 0 2022-01-31T19:44:07.367 INFO:tasks.workunit.client.0.smithi167.stdout:4/166: rename d0/d6/dc to d0/d6/d7/d26/d31 0 2022-01-31T19:44:07.367 INFO:tasks.workunit.client.0.smithi167.stdout:4/167: chown d0/d6/d7/d26/d31/d1c 228593435 1 2022-01-31T19:44:07.368 INFO:tasks.workunit.client.0.smithi167.stdout:7/184: rename d8 to d8/d2c/d43/d31/d46 22 2022-01-31T19:44:07.368 INFO:tasks.workunit.client.0.smithi167.stdout:9/119: getdents dd/d17 0 2022-01-31T19:44:07.369 INFO:tasks.workunit.client.0.smithi167.stdout:2/165: write dd/f29 [1665338,22634] 0 2022-01-31T19:44:07.371 INFO:tasks.workunit.client.0.smithi167.stdout:2/166: read dd/df/d2c/f2d [3186064,85813] 0 2022-01-31T19:44:07.371 INFO:tasks.workunit.client.0.smithi167.stdout:4/168: creat d0/d6/d7/d26/d31/d1c/f32 x:0 0 0 2022-01-31T19:44:07.372 INFO:tasks.workunit.client.0.smithi167.stdout:9/120: dread f7 [0,4194304] 0 2022-01-31T19:44:07.374 INFO:tasks.workunit.client.0.smithi167.stdout:7/185: unlink d8/dc/d18/d24/f26 0 2022-01-31T19:44:07.374 INFO:tasks.workunit.client.0.smithi167.stdout:7/186: dread - d8/dc/f2b zero size 2022-01-31T19:44:07.376 INFO:tasks.workunit.client.0.smithi167.stdout:2/167: mknod dd/df/c33 0 2022-01-31T19:44:07.383 INFO:tasks.workunit.client.0.smithi167.stdout:2/168: rename dd/f1e to dd/df/d1b/d2f/f34 0 2022-01-31T19:44:07.383 INFO:tasks.workunit.client.0.smithi167.stdout:4/169: truncate d0/d6/d21/f2b 1008814 0 2022-01-31T19:44:07.384 INFO:tasks.workunit.client.0.smithi167.stdout:9/121: getdents dd/d19 0 2022-01-31T19:44:07.386 INFO:tasks.workunit.client.0.smithi167.stdout:4/170: creat d0/d6/d7/d26/d31/d1c/f33 x:0 0 0 2022-01-31T19:44:07.386 INFO:tasks.workunit.client.0.smithi167.stdout:2/169: write dd/df/d1b/d2f/f34 [4005667,21461] 0 2022-01-31T19:44:07.388 INFO:tasks.workunit.client.0.smithi167.stdout:9/122: rename f9 to dd/d17/f23 0 2022-01-31T19:44:07.388 INFO:tasks.workunit.client.0.smithi167.stdout:9/123: readlink l2 0 2022-01-31T19:44:07.389 INFO:tasks.workunit.client.0.smithi167.stdout:0/88: dwrite f15 [0,4194304] 0 2022-01-31T19:44:07.389 INFO:tasks.workunit.client.0.smithi167.stdout:4/171: rmdir d0/d6 39 2022-01-31T19:44:07.390 INFO:tasks.workunit.client.0.smithi167.stdout:4/172: unlink d0/d6/d7/d26/d31/f2a 0 2022-01-31T19:44:07.391 INFO:tasks.workunit.client.0.smithi167.stdout:2/170: write dd/f15 [3663770,32152] 0 2022-01-31T19:44:07.391 INFO:tasks.workunit.client.0.smithi167.stdout:4/173: mknod d0/d6/d7/d26/d31/dd/c34 0 2022-01-31T19:44:07.392 INFO:tasks.workunit.client.0.smithi167.stdout:1/198: dwrite d0/d10/f34 [0,4194304] 0 2022-01-31T19:44:07.393 INFO:tasks.workunit.client.0.smithi167.stdout:1/199: chown d0/d10/f42 1 1 2022-01-31T19:44:07.393 INFO:tasks.workunit.client.0.smithi167.stdout:2/171: link dd/df/d2c/f2d dd/df/f35 0 2022-01-31T19:44:07.395 INFO:tasks.workunit.client.0.smithi167.stdout:4/174: mknod d0/c35 0 2022-01-31T19:44:07.395 INFO:tasks.workunit.client.0.smithi167.stdout:4/175: dread - d0/d6/d7/f30 zero size 2022-01-31T19:44:07.396 INFO:tasks.workunit.client.0.smithi167.stdout:7/187: dwrite d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:07.396 INFO:tasks.workunit.client.0.smithi167.stdout:7/188: write d8/dc/d18/f3b [190555,77460] 0 2022-01-31T19:44:07.405 INFO:tasks.workunit.client.0.smithi167.stdout:7/189: symlink d8/dc/d12/l47 0 2022-01-31T19:44:07.406 INFO:tasks.workunit.client.0.smithi167.stdout:7/190: fdatasync f6 0 2022-01-31T19:44:07.406 INFO:tasks.workunit.client.0.smithi167.stdout:7/191: truncate d8/d35/f3d 225622 0 2022-01-31T19:44:07.406 INFO:tasks.workunit.client.0.smithi167.stdout:7/192: unlink l4 0 2022-01-31T19:44:07.409 INFO:tasks.workunit.client.0.smithi167.stdout:4/176: rename d0/d6/d7/d26/d31/dd/fe to d0/d6/d21/f36 0 2022-01-31T19:44:07.411 INFO:tasks.workunit.client.0.smithi167.stdout:7/193: symlink d8/d2c/d43/l48 0 2022-01-31T19:44:07.411 INFO:tasks.workunit.client.0.smithi167.stdout:7/194: creat d8/dc/d12/f49 x:0 0 0 2022-01-31T19:44:07.411 INFO:tasks.workunit.client.0.smithi167.stdout:7/195: stat d8/d35/f3d 0 2022-01-31T19:44:07.411 INFO:tasks.workunit.client.0.smithi167.stdout:7/196: fdatasync f2 0 2022-01-31T19:44:07.412 INFO:tasks.workunit.client.0.smithi167.stdout:7/197: dread - d8/d35/f36 zero size 2022-01-31T19:44:07.412 INFO:tasks.workunit.client.0.smithi167.stdout:4/177: creat d0/d6/d21/f37 x:0 0 0 2022-01-31T19:44:07.413 INFO:tasks.workunit.client.0.smithi167.stdout:4/178: truncate d0/d6/d7/d26/d31/dd/f2e 101952 0 2022-01-31T19:44:07.413 INFO:tasks.workunit.client.0.smithi167.stdout:7/198: mknod d8/d35/c4a 0 2022-01-31T19:44:07.414 INFO:tasks.workunit.client.0.smithi167.stdout:4/179: mknod d0/d6/d7/d26/c38 0 2022-01-31T19:44:07.414 INFO:tasks.workunit.client.0.smithi167.stdout:4/180: chown d0/d6/d7 113 1 2022-01-31T19:44:07.415 INFO:tasks.workunit.client.0.smithi167.stdout:0/89: dwrite fe [0,4194304] 0 2022-01-31T19:44:07.415 INFO:tasks.workunit.client.0.smithi167.stdout:0/90: fsync f15 0 2022-01-31T19:44:07.415 INFO:tasks.workunit.client.0.smithi167.stdout:9/124: dwrite f4 [0,4194304] 0 2022-01-31T19:44:07.416 INFO:tasks.workunit.client.0.smithi167.stdout:7/199: rename d8/dc/f17 to d8/d35/f4b 0 2022-01-31T19:44:07.417 INFO:tasks.workunit.client.0.smithi167.stdout:7/200: read d8/dc/f1c [604638,29069] 0 2022-01-31T19:44:07.417 INFO:tasks.workunit.client.0.smithi167.stdout:7/201: write d8/d2c/d43/f2f [1012747,128698] 0 2022-01-31T19:44:07.417 INFO:tasks.workunit.client.0.smithi167.stdout:7/202: stat d8/dc 0 2022-01-31T19:44:07.418 INFO:tasks.workunit.client.0.smithi167.stdout:4/181: unlink d0/f22 0 2022-01-31T19:44:07.423 INFO:tasks.workunit.client.0.smithi167.stdout:9/125: mkdir dd/d19/d24 0 2022-01-31T19:44:07.423 INFO:tasks.workunit.client.0.smithi167.stdout:9/126: rename dd to dd/d17/d25 22 2022-01-31T19:44:07.423 INFO:tasks.workunit.client.0.smithi167.stdout:9/127: getdents dd 0 2022-01-31T19:44:07.425 INFO:tasks.workunit.client.0.smithi167.stdout:7/203: symlink d8/dc/l4c 0 2022-01-31T19:44:07.429 INFO:tasks.workunit.client.0.smithi167.stdout:2/172: truncate dd/df/d1b/d2f/f34 212239 0 2022-01-31T19:44:07.430 INFO:tasks.workunit.client.0.smithi167.stdout:2/173: write dd/f23 [1792960,8295] 0 2022-01-31T19:44:07.431 INFO:tasks.workunit.client.0.smithi167.stdout:2/174: read dd/f23 [3289032,46795] 0 2022-01-31T19:44:07.431 INFO:tasks.workunit.client.0.smithi167.stdout:9/128: symlink dd/d19/d24/l26 0 2022-01-31T19:44:07.433 INFO:tasks.workunit.client.0.smithi167.stdout:7/204: mkdir d8/dc/d12/d4d 0 2022-01-31T19:44:07.433 INFO:tasks.workunit.client.0.smithi167.stdout:7/205: fdatasync d8/d2c/f25 0 2022-01-31T19:44:07.433 INFO:tasks.workunit.client.0.smithi167.stdout:7/206: read d8/dc/d12/f44 [62981,13992] 0 2022-01-31T19:44:07.438 INFO:tasks.workunit.client.0.smithi167.stdout:9/129: rmdir dd/d19/d24 39 2022-01-31T19:44:07.439 INFO:tasks.workunit.client.0.smithi167.stdout:9/130: write dd/f10 [885631,92074] 0 2022-01-31T19:44:07.439 INFO:tasks.workunit.client.0.smithi167.stdout:7/207: mknod d8/d2c/d43/c4e 0 2022-01-31T19:44:07.445 INFO:tasks.workunit.client.0.smithi167.stdout:2/175: rmdir dd/d18 39 2022-01-31T19:44:07.448 INFO:tasks.workunit.client.0.smithi167.stdout:9/131: creat dd/d19/f27 x:0 0 0 2022-01-31T19:44:07.448 INFO:tasks.workunit.client.0.smithi167.stdout:7/208: read - d8/f45 zero size 2022-01-31T19:44:07.451 INFO:tasks.workunit.client.0.smithi167.stdout:9/132: mkdir dd/d28 0 2022-01-31T19:44:07.453 INFO:tasks.workunit.client.0.smithi167.stdout:9/133: mknod dd/c29 0 2022-01-31T19:44:07.453 INFO:tasks.workunit.client.0.smithi167.stdout:9/134: truncate dd/d19/f27 360510 0 2022-01-31T19:44:07.455 INFO:tasks.workunit.client.0.smithi167.stdout:9/135: write dd/fe [2233589,129770] 0 2022-01-31T19:44:07.456 INFO:tasks.workunit.client.0.smithi167.stdout:9/136: readlink dd/d19/d24/l26 0 2022-01-31T19:44:07.457 INFO:tasks.workunit.client.0.smithi167.stdout:9/137: unlink dd/d17/f23 0 2022-01-31T19:44:07.458 INFO:tasks.workunit.client.0.smithi167.stdout:9/138: write dd/f1a [396242,65728] 0 2022-01-31T19:44:07.461 INFO:tasks.workunit.client.0.smithi167.stdout:9/139: mknod dd/d19/c2a 0 2022-01-31T19:44:07.463 INFO:tasks.workunit.client.0.smithi167.stdout:7/209: dread d8/dc/d19/f42 [0,4194304] 0 2022-01-31T19:44:07.464 INFO:tasks.workunit.client.0.smithi167.stdout:2/176: dwrite dd/df/f1c [0,4194304] 0 2022-01-31T19:44:07.466 INFO:tasks.workunit.client.0.smithi167.stdout:7/210: mkdir d8/d2c/d43/d31/d4f 0 2022-01-31T19:44:07.467 INFO:tasks.workunit.client.0.smithi167.stdout:7/211: fsync d8/fa 0 2022-01-31T19:44:07.468 INFO:tasks.workunit.client.0.smithi167.stdout:7/212: symlink d8/d2c/d43/l50 0 2022-01-31T19:44:07.497 INFO:tasks.workunit.client.0.smithi167.stdout:9/140: dwrite dd/d19/f21 [0,4194304] 0 2022-01-31T19:44:07.498 INFO:tasks.workunit.client.0.smithi167.stdout:2/177: dwrite dd/f15 [0,4194304] 0 2022-01-31T19:44:07.499 INFO:tasks.workunit.client.0.smithi167.stdout:7/213: dwrite f7 [0,4194304] 0 2022-01-31T19:44:07.500 INFO:tasks.workunit.client.0.smithi167.stdout:9/141: mknod dd/d19/c2b 0 2022-01-31T19:44:07.500 INFO:tasks.workunit.client.0.smithi167.stdout:9/142: write dd/d17/f1f [189361,52202] 0 2022-01-31T19:44:07.507 INFO:tasks.workunit.client.0.smithi167.stdout:7/214: link d8/d2c/d43/l48 d8/dc/d12/l51 0 2022-01-31T19:44:07.507 INFO:tasks.workunit.client.0.smithi167.stdout:9/143: rmdir dd/d19/d24 39 2022-01-31T19:44:07.508 INFO:tasks.workunit.client.0.smithi167.stdout:7/215: truncate d8/dc/d19/f1a 565122 0 2022-01-31T19:44:07.510 INFO:tasks.workunit.client.0.smithi167.stdout:9/144: creat dd/d19/d24/f2c x:0 0 0 2022-01-31T19:44:07.510 INFO:tasks.workunit.client.0.smithi167.stdout:9/145: unlink f4 0 2022-01-31T19:44:07.512 INFO:tasks.workunit.client.0.smithi167.stdout:9/146: dread f7 [0,4194304] 0 2022-01-31T19:44:07.513 INFO:tasks.workunit.client.0.smithi167.stdout:9/147: read dd/f1a [694230,105006] 0 2022-01-31T19:44:07.515 INFO:tasks.workunit.client.0.smithi167.stdout:9/148: getdents dd/d17 0 2022-01-31T19:44:07.517 INFO:tasks.workunit.client.0.smithi167.stdout:9/149: link dd/d17/f18 dd/d19/f2d 0 2022-01-31T19:44:07.518 INFO:tasks.workunit.client.0.smithi167.stdout:9/150: write dd/d17/f1f [1169369,111745] 0 2022-01-31T19:44:07.519 INFO:tasks.workunit.client.0.smithi167.stdout:9/151: rename dd/f1c to dd/d17/f2e 0 2022-01-31T19:44:07.520 INFO:tasks.workunit.client.0.smithi167.stdout:9/152: chown dd/d19/f27 15 1 2022-01-31T19:44:07.526 INFO:tasks.workunit.client.0.smithi167.stdout:2/178: dwrite dd/df/d2c/f2d [0,4194304] 0 2022-01-31T19:44:07.529 INFO:tasks.workunit.client.0.smithi167.stdout:7/216: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:07.529 INFO:tasks.workunit.client.0.smithi167.stdout:2/179: rename dd/df/d22/d25 to dd/df/d1b/d2f/d36 0 2022-01-31T19:44:07.532 INFO:tasks.workunit.client.0.smithi167.stdout:7/217: symlink d8/dc/d18/d24/l52 0 2022-01-31T19:44:07.532 INFO:tasks.workunit.client.0.smithi167.stdout:7/218: write d8/dc/f40 [195303,31526] 0 2022-01-31T19:44:07.533 INFO:tasks.workunit.client.0.smithi167.stdout:7/219: write d8/dc/f33 [821552,103797] 0 2022-01-31T19:44:07.534 INFO:tasks.workunit.client.0.smithi167.stdout:2/180: creat dd/df/d22/f37 x:0 0 0 2022-01-31T19:44:07.534 INFO:tasks.workunit.client.0.smithi167.stdout:2/181: readlink dd/df/d1b/l28 0 2022-01-31T19:44:07.534 INFO:tasks.workunit.client.0.smithi167.stdout:7/220: link d8/d2c/c1f d8/dc/d12/d4d/c53 0 2022-01-31T19:44:07.535 INFO:tasks.workunit.client.0.smithi167.stdout:7/221: symlink d8/d35/l54 0 2022-01-31T19:44:07.535 INFO:tasks.workunit.client.0.smithi167.stdout:7/222: dread - d8/dc/f2b zero size 2022-01-31T19:44:07.536 INFO:tasks.workunit.client.0.smithi167.stdout:7/223: mkdir d8/d2c/d43/d31/d55 0 2022-01-31T19:44:07.536 INFO:tasks.workunit.client.0.smithi167.stdout:7/224: truncate d8/dc/d19/f3c 602610 0 2022-01-31T19:44:07.537 INFO:tasks.workunit.client.0.smithi167.stdout:7/225: chown d8/d2c/d43/d31/d55 11041 1 2022-01-31T19:44:07.538 INFO:tasks.workunit.client.0.smithi167.stdout:7/226: symlink d8/d2c/d43/d31/d4f/l56 0 2022-01-31T19:44:07.538 INFO:tasks.workunit.client.0.smithi167.stdout:7/227: symlink d8/dc/d18/l57 0 2022-01-31T19:44:07.539 INFO:tasks.workunit.client.0.smithi167.stdout:7/228: chown d8/dc/d12/f2a 238 1 2022-01-31T19:44:07.540 INFO:tasks.workunit.client.0.smithi167.stdout:7/229: mknod d8/c58 0 2022-01-31T19:44:07.540 INFO:tasks.workunit.client.0.smithi167.stdout:7/230: write d8/d35/f3d [34498,22142] 0 2022-01-31T19:44:07.540 INFO:tasks.workunit.client.0.smithi167.stdout:7/231: chown d8/dc/d18/d24/f29 4361442 1 2022-01-31T19:44:07.542 INFO:tasks.workunit.client.0.smithi167.stdout:7/232: rename d8/dc/d18/l57 to d8/dc/d18/d24/l59 0 2022-01-31T19:44:07.543 INFO:tasks.workunit.client.0.smithi167.stdout:7/233: creat d8/d2c/d43/d31/f5a x:0 0 0 2022-01-31T19:44:07.560 INFO:tasks.workunit.client.0.smithi167.stdout:2/182: dwrite f9 [0,4194304] 0 2022-01-31T19:44:07.561 INFO:tasks.workunit.client.0.smithi167.stdout:7/234: dwrite d8/dc/d19/f3c [0,4194304] 0 2022-01-31T19:44:07.562 INFO:tasks.workunit.client.0.smithi167.stdout:2/183: rename dd/df/d22/l26 to dd/df/l38 0 2022-01-31T19:44:07.583 INFO:tasks.workunit.client.0.smithi167.stdout:3/251: sync 2022-01-31T19:44:07.589 INFO:tasks.workunit.client.0.smithi167.stdout:7/235: dwrite d8/d35/f4b [0,4194304] 0 2022-01-31T19:44:07.589 INFO:tasks.workunit.client.0.smithi167.stdout:7/236: creat d8/d2c/d43/d31/f5b x:0 0 0 2022-01-31T19:44:07.589 INFO:tasks.workunit.client.0.smithi167.stdout:7/237: write d8/dc/f2b [249076,79204] 0 2022-01-31T19:44:07.590 INFO:tasks.workunit.client.0.smithi167.stdout:7/238: write d8/f45 [30481,8142] 0 2022-01-31T19:44:07.590 INFO:tasks.workunit.client.0.smithi167.stdout:2/184: dwrite dd/df/d2c/f2d [4194304,4194304] 0 2022-01-31T19:44:07.592 INFO:tasks.workunit.client.0.smithi167.stdout:2/185: mknod dd/df/c39 0 2022-01-31T19:44:07.593 INFO:tasks.workunit.client.0.smithi167.stdout:2/186: creat dd/d18/f3a x:0 0 0 2022-01-31T19:44:07.593 INFO:tasks.workunit.client.0.smithi167.stdout:2/187: readlink dd/d18/l1f 0 2022-01-31T19:44:07.594 INFO:tasks.workunit.client.0.smithi167.stdout:2/188: mknod dd/df/d1b/d2f/d36/c3b 0 2022-01-31T19:44:07.597 INFO:tasks.workunit.client.0.smithi167.stdout:3/252: dwrite d9/d29/d2e/f39 [0,4194304] 0 2022-01-31T19:44:07.598 INFO:tasks.workunit.client.0.smithi167.stdout:2/189: write dd/df/f1c [4184595,27520] 0 2022-01-31T19:44:07.598 INFO:tasks.workunit.client.0.smithi167.stdout:2/190: dread - dd/d18/f31 zero size 2022-01-31T19:44:07.598 INFO:tasks.workunit.client.0.smithi167.stdout:2/191: chown dd/d18/f3a 89 1 2022-01-31T19:44:07.599 INFO:tasks.workunit.client.0.smithi167.stdout:7/239: dread d8/dc/f33 [0,4194304] 0 2022-01-31T19:44:07.600 INFO:tasks.workunit.client.0.smithi167.stdout:7/240: write f2 [2473019,12755] 0 2022-01-31T19:44:07.600 INFO:tasks.workunit.client.0.smithi167.stdout:7/241: fdatasync d8/dc/f33 0 2022-01-31T19:44:07.602 INFO:tasks.workunit.client.0.smithi167.stdout:7/242: truncate d8/dc/d19/f3c 2874069 0 2022-01-31T19:44:07.605 INFO:tasks.workunit.client.0.smithi167.stdout:7/243: link d8/f45 d8/dc/d19/f5c 0 2022-01-31T19:44:07.606 INFO:tasks.workunit.client.0.smithi167.stdout:7/244: readlink d8/d2c/d43/l48 0 2022-01-31T19:44:07.629 INFO:tasks.workunit.client.0.smithi167.stdout:2/192: dwrite dd/df/f35 [0,4194304] 0 2022-01-31T19:44:07.629 INFO:tasks.workunit.client.0.smithi167.stdout:7/245: dwrite d8/dc/f21 [4194304,4194304] 0 2022-01-31T19:44:07.629 INFO:tasks.workunit.client.0.smithi167.stdout:7/246: creat d8/d35/f5d x:0 0 0 2022-01-31T19:44:07.630 INFO:tasks.workunit.client.0.smithi167.stdout:2/193: write dd/f23 [4034235,103474] 0 2022-01-31T19:44:07.630 INFO:tasks.workunit.client.0.smithi167.stdout:7/247: dread d8/d35/f3d [0,4194304] 0 2022-01-31T19:44:07.635 INFO:tasks.workunit.client.0.smithi167.stdout:7/248: write d8/dc/d18/d24/f29 [3864008,96130] 0 2022-01-31T19:44:07.635 INFO:tasks.workunit.client.0.smithi167.stdout:7/249: dread - d8/d35/f5d zero size 2022-01-31T19:44:07.636 INFO:tasks.workunit.client.0.smithi167.stdout:7/250: mkdir d8/dc/d12/d4d/d5e 0 2022-01-31T19:44:07.649 INFO:tasks.workunit.client.0.smithi167.stdout:2/194: dwrite f7 [0,4194304] 0 2022-01-31T19:44:07.649 INFO:tasks.workunit.client.0.smithi167.stdout:7/251: dread d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:07.649 INFO:tasks.workunit.client.0.smithi167.stdout:2/195: chown dd/df/l38 47307777 1 2022-01-31T19:44:07.652 INFO:tasks.workunit.client.0.smithi167.stdout:7/252: mknod d8/d2c/d43/c5f 0 2022-01-31T19:44:07.653 INFO:tasks.workunit.client.0.smithi167.stdout:2/196: mknod dd/df/d2c/c3c 0 2022-01-31T19:44:07.654 INFO:tasks.workunit.client.0.smithi167.stdout:7/253: link f7 d8/d35/f60 0 2022-01-31T19:44:07.656 INFO:tasks.workunit.client.0.smithi167.stdout:2/197: mkdir dd/df/d1b/d3d 0 2022-01-31T19:44:07.656 INFO:tasks.workunit.client.0.smithi167.stdout:2/198: write fa [8693579,40336] 0 2022-01-31T19:44:07.656 INFO:tasks.workunit.client.0.smithi167.stdout:2/199: chown dd/c17 383735 1 2022-01-31T19:44:07.658 INFO:tasks.workunit.client.0.smithi167.stdout:7/254: mknod d8/dc/d18/c61 0 2022-01-31T19:44:07.660 INFO:tasks.workunit.client.0.smithi167.stdout:6/137: sync 2022-01-31T19:44:07.660 INFO:tasks.workunit.client.0.smithi167.stdout:8/293: sync 2022-01-31T19:44:07.661 INFO:tasks.workunit.client.0.smithi167.stdout:5/253: sync 2022-01-31T19:44:07.661 INFO:tasks.workunit.client.0.smithi167.stdout:5/254: fdatasync d5/d6/de/f3e 0 2022-01-31T19:44:07.661 INFO:tasks.workunit.client.0.smithi167.stdout:2/200: link dd/c17 dd/c3e 0 2022-01-31T19:44:07.662 INFO:tasks.workunit.client.0.smithi167.stdout:2/201: dread - dd/d18/f31 zero size 2022-01-31T19:44:07.662 INFO:tasks.workunit.client.0.smithi167.stdout:2/202: dread - dd/df/d2c/f32 zero size 2022-01-31T19:44:07.662 INFO:tasks.workunit.client.0.smithi167.stdout:2/203: write dd/d18/f30 [1047805,43685] 0 2022-01-31T19:44:07.662 INFO:tasks.workunit.client.0.smithi167.stdout:2/204: write f9 [4875946,106514] 0 2022-01-31T19:44:07.671 INFO:tasks.workunit.client.0.smithi167.stdout:8/294: read d6/db/fd [2164271,122994] 0 2022-01-31T19:44:07.672 INFO:tasks.workunit.client.0.smithi167.stdout:8/295: readlink d6/l54 0 2022-01-31T19:44:07.672 INFO:tasks.workunit.client.0.smithi167.stdout:7/255: rename f2 to d8/dc/d12/f62 0 2022-01-31T19:44:07.672 INFO:tasks.workunit.client.0.smithi167.stdout:2/205: truncate f8 3341987 0 2022-01-31T19:44:07.672 INFO:tasks.workunit.client.0.smithi167.stdout:8/296: symlink d6/d1e/d2d/l5f 0 2022-01-31T19:44:07.673 INFO:tasks.workunit.client.0.smithi167.stdout:2/206: unlink dd/df/d1b/d2f/d36/l27 0 2022-01-31T19:44:07.673 INFO:tasks.workunit.client.0.smithi167.stdout:2/207: stat dd/f23 0 2022-01-31T19:44:07.673 INFO:tasks.workunit.client.0.smithi167.stdout:8/297: truncate d6/db/d36/f4a 658227 0 2022-01-31T19:44:07.674 INFO:tasks.workunit.client.0.smithi167.stdout:6/138: rename d8/d2a/l2c to d8/l30 0 2022-01-31T19:44:07.674 INFO:tasks.workunit.client.0.smithi167.stdout:7/256: getdents d8/dc/d18/d24 0 2022-01-31T19:44:07.674 INFO:tasks.workunit.client.0.smithi167.stdout:7/257: chown f7 1503498061 1 2022-01-31T19:44:07.675 INFO:tasks.workunit.client.0.smithi167.stdout:7/258: write d8/dc/f2b [145198,84706] 0 2022-01-31T19:44:07.675 INFO:tasks.workunit.client.0.smithi167.stdout:7/259: fsync d8/d35/f3d 0 2022-01-31T19:44:07.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/208: mkdir dd/df/d3f 0 2022-01-31T19:44:07.676 INFO:tasks.workunit.client.0.smithi167.stdout:2/209: fdatasync dd/df/d1b/d2f/d36/f2a 0 2022-01-31T19:44:07.676 INFO:tasks.workunit.client.0.smithi167.stdout:6/139: creat d8/f31 x:0 0 0 2022-01-31T19:44:07.678 INFO:tasks.workunit.client.0.smithi167.stdout:7/260: link d8/dc/d19/f42 d8/d2c/f63 0 2022-01-31T19:44:07.680 INFO:tasks.workunit.client.0.smithi167.stdout:2/210: mknod dd/df/c40 0 2022-01-31T19:44:07.682 INFO:tasks.workunit.client.0.smithi167.stdout:2/211: creat dd/df/d3f/f41 x:0 0 0 2022-01-31T19:44:07.682 INFO:tasks.workunit.client.0.smithi167.stdout:2/212: chown dd/df/d2c/f2d 2 1 2022-01-31T19:44:07.682 INFO:tasks.workunit.client.0.smithi167.stdout:2/213: write dd/d18/f3a [335250,94809] 0 2022-01-31T19:44:07.689 INFO:tasks.workunit.client.0.smithi167.stdout:6/140: dwrite d8/f1a [0,4194304] 0 2022-01-31T19:44:07.690 INFO:tasks.workunit.client.0.smithi167.stdout:7/261: fsync d8/dc/f2b 0 2022-01-31T19:44:07.699 INFO:tasks.workunit.client.0.smithi167.stdout:6/141: link d8/d20/f22 d8/d2a/f32 0 2022-01-31T19:44:07.700 INFO:tasks.workunit.client.0.smithi167.stdout:7/262: getdents d8/dc 0 2022-01-31T19:44:07.700 INFO:tasks.workunit.client.0.smithi167.stdout:7/263: chown d8/dc/d18 179 1 2022-01-31T19:44:07.727 INFO:tasks.workunit.client.0.smithi167.stdout:6/142: dwrite d8/f15 [0,4194304] 0 2022-01-31T19:44:07.727 INFO:tasks.workunit.client.0.smithi167.stdout:7/264: dwrite d8/d35/f60 [8388608,4194304] 0 2022-01-31T19:44:07.730 INFO:tasks.workunit.client.0.smithi167.stdout:7/265: rename d8/d2c/d43/l3f to d8/dc/d12/l64 0 2022-01-31T19:44:07.731 INFO:tasks.workunit.client.0.smithi167.stdout:6/143: dread d8/d2a/f32 [0,4194304] 0 2022-01-31T19:44:07.737 INFO:tasks.workunit.client.0.smithi167.stdout:6/144: rename d8/d2a to d8/d33 0 2022-01-31T19:44:07.738 INFO:tasks.workunit.client.0.smithi167.stdout:6/145: mknod d8/d33/c34 0 2022-01-31T19:44:07.747 INFO:tasks.workunit.client.0.smithi167.stdout:7/266: dwrite d8/dc/d19/f5c [0,4194304] 0 2022-01-31T19:44:07.748 INFO:tasks.workunit.client.0.smithi167.stdout:6/146: rmdir d8/d33 39 2022-01-31T19:44:07.748 INFO:tasks.workunit.client.0.smithi167.stdout:6/147: chown d8/l23 49 1 2022-01-31T19:44:07.751 INFO:tasks.workunit.client.0.smithi167.stdout:7/267: link d8/dc/d18/c61 d8/dc/c65 0 2022-01-31T19:44:07.752 INFO:tasks.workunit.client.0.smithi167.stdout:6/148: dread d8/d20/f22 [0,4194304] 0 2022-01-31T19:44:07.752 INFO:tasks.workunit.client.0.smithi167.stdout:6/149: fdatasync d8/f15 0 2022-01-31T19:44:07.752 INFO:tasks.workunit.client.0.smithi167.stdout:6/150: fdatasync d8/f31 0 2022-01-31T19:44:07.753 INFO:tasks.workunit.client.0.smithi167.stdout:6/151: chown l6 1857 1 2022-01-31T19:44:07.753 INFO:tasks.workunit.client.0.smithi167.stdout:6/152: fsync d8/d33/f32 0 2022-01-31T19:44:07.753 INFO:tasks.workunit.client.0.smithi167.stdout:6/153: write d8/f1c [923719,80939] 0 2022-01-31T19:44:07.754 INFO:tasks.workunit.client.0.smithi167.stdout:6/154: mkdir d8/d33/d35 0 2022-01-31T19:44:07.754 INFO:tasks.workunit.client.0.smithi167.stdout:6/155: mknod d8/c36 0 2022-01-31T19:44:07.755 INFO:tasks.workunit.client.0.smithi167.stdout:6/156: write d8/f1f [1521805,102911] 0 2022-01-31T19:44:07.755 INFO:tasks.workunit.client.0.smithi167.stdout:6/157: symlink d8/d33/l37 0 2022-01-31T19:44:07.757 INFO:tasks.workunit.client.0.smithi167.stdout:6/158: truncate d8/d33/f32 835086 0 2022-01-31T19:44:07.757 INFO:tasks.workunit.client.0.smithi167.stdout:6/159: chown d8/c28 28 1 2022-01-31T19:44:07.757 INFO:tasks.workunit.client.0.smithi167.stdout:6/160: write d8/f31 [121222,3363] 0 2022-01-31T19:44:07.758 INFO:tasks.workunit.client.0.smithi167.stdout:6/161: symlink d8/l38 0 2022-01-31T19:44:07.760 INFO:tasks.workunit.client.0.smithi167.stdout:6/162: unlink d8/c28 0 2022-01-31T19:44:07.761 INFO:tasks.workunit.client.0.smithi167.stdout:6/163: link d8/f2d d8/d33/f39 0 2022-01-31T19:44:07.761 INFO:tasks.workunit.client.0.smithi167.stdout:6/164: chown d8/f11 23093 1 2022-01-31T19:44:07.771 INFO:tasks.workunit.client.0.smithi167.stdout:7/268: dwrite d8/dc/d12/f49 [0,4194304] 0 2022-01-31T19:44:07.775 INFO:tasks.workunit.client.0.smithi167.stdout:7/269: symlink d8/d2c/d43/d31/l66 0 2022-01-31T19:44:07.776 INFO:tasks.workunit.client.0.smithi167.stdout:7/270: mkdir d8/dc/d12/d4d/d67 0 2022-01-31T19:44:07.777 INFO:tasks.workunit.client.0.smithi167.stdout:7/271: creat d8/dc/d19/f68 x:0 0 0 2022-01-31T19:44:07.779 INFO:tasks.workunit.client.0.smithi167.stdout:6/165: dwrite d8/f1a [0,4194304] 0 2022-01-31T19:44:07.782 INFO:tasks.workunit.client.0.smithi167.stdout:6/166: rename d8/l23 to d8/d33/l3a 0 2022-01-31T19:44:07.785 INFO:tasks.workunit.client.0.smithi167.stdout:6/167: creat d8/d20/f3b x:0 0 0 2022-01-31T19:44:07.785 INFO:tasks.workunit.client.0.smithi167.stdout:6/168: write d8/d20/f22 [1589883,102661] 0 2022-01-31T19:44:07.787 INFO:tasks.workunit.client.0.smithi167.stdout:6/169: rename d8/f2d to d8/d33/f3c 0 2022-01-31T19:44:07.789 INFO:tasks.workunit.client.0.smithi167.stdout:6/170: rename d8/f11 to d8/d33/d35/f3d 0 2022-01-31T19:44:07.790 INFO:tasks.workunit.client.0.smithi167.stdout:6/171: link d8/c18 d8/d33/c3e 0 2022-01-31T19:44:07.793 INFO:tasks.workunit.client.0.smithi167.stdout:7/272: dwrite d8/dc/d12/f62 [0,4194304] 0 2022-01-31T19:44:07.794 INFO:tasks.workunit.client.0.smithi167.stdout:7/273: stat d8/d2c/d43/d31/d4f/l56 0 2022-01-31T19:44:07.811 INFO:tasks.workunit.client.0.smithi167.stdout:6/172: rename d8/d33/f39 to d8/f3f 0 2022-01-31T19:44:07.815 INFO:tasks.workunit.client.0.smithi167.stdout:7/274: rename d8/c10 to d8/dc/c69 0 2022-01-31T19:44:07.817 INFO:tasks.workunit.client.0.smithi167.stdout:6/173: rename d8/d20/f3b to d8/d33/f40 0 2022-01-31T19:44:07.819 INFO:tasks.workunit.client.0.smithi167.stdout:7/275: truncate d8/dc/f23 2247531 0 2022-01-31T19:44:07.819 INFO:tasks.workunit.client.0.smithi167.stdout:7/276: creat d8/d2c/d43/d31/f6a x:0 0 0 2022-01-31T19:44:07.820 INFO:tasks.workunit.client.0.smithi167.stdout:7/277: read d8/dc/d19/f5c [1220945,60674] 0 2022-01-31T19:44:07.826 INFO:tasks.workunit.client.0.smithi167.stdout:1/200: sync 2022-01-31T19:44:07.827 INFO:tasks.workunit.client.0.smithi167.stdout:4/182: sync 2022-01-31T19:44:07.827 INFO:tasks.workunit.client.0.smithi167.stdout:0/91: sync 2022-01-31T19:44:07.828 INFO:tasks.workunit.client.0.smithi167.stdout:7/278: mkdir d8/dc/d12/d4d/d6b 0 2022-01-31T19:44:07.830 INFO:tasks.workunit.client.0.smithi167.stdout:4/183: truncate d0/d6/d7/d26/d31/f19 1308593 0 2022-01-31T19:44:07.832 INFO:tasks.workunit.client.0.smithi167.stdout:0/92: symlink d17/d19/d1c/l1d 0 2022-01-31T19:44:07.835 INFO:tasks.workunit.client.0.smithi167.stdout:1/201: rename d0/d10/d18/d16 to d0/d11/d24/d33/d4b 0 2022-01-31T19:44:07.835 INFO:tasks.workunit.client.0.smithi167.stdout:1/202: symlink d0/d11/d24/d33/d4b/d1f/l4c 0 2022-01-31T19:44:07.836 INFO:tasks.workunit.client.0.smithi167.stdout:4/184: dread d0/d6/d21/f36 [0,4194304] 0 2022-01-31T19:44:07.836 INFO:tasks.workunit.client.0.smithi167.stdout:4/185: readlink d0/d6/d7/d26/d31/dd/l1f 0 2022-01-31T19:44:07.837 INFO:tasks.workunit.client.0.smithi167.stdout:7/279: rename d8/d2c/d43/d31/d55 to d8/d2c/d6c 0 2022-01-31T19:44:07.837 INFO:tasks.workunit.client.0.smithi167.stdout:7/280: chown d8/d2c/d43/d31 665274 1 2022-01-31T19:44:07.838 INFO:tasks.workunit.client.0.smithi167.stdout:6/174: dwrite d8/f16 [0,4194304] 0 2022-01-31T19:44:07.839 INFO:tasks.workunit.client.0.smithi167.stdout:1/203: creat d0/d11/f4d x:0 0 0 2022-01-31T19:44:07.839 INFO:tasks.workunit.client.0.smithi167.stdout:1/204: chown d0/d11/d24/d33/d4b/c43 0 1 2022-01-31T19:44:07.843 INFO:tasks.workunit.client.0.smithi167.stdout:7/281: link f6 d8/d2c/d6c/f6d 0 2022-01-31T19:44:07.846 INFO:tasks.workunit.client.0.smithi167.stdout:6/175: link d8/d33/f40 d8/d33/f41 0 2022-01-31T19:44:07.847 INFO:tasks.workunit.client.0.smithi167.stdout:0/93: dread fe [0,4194304] 0 2022-01-31T19:44:07.851 INFO:tasks.workunit.client.0.smithi167.stdout:6/176: creat d8/d20/f42 x:0 0 0 2022-01-31T19:44:07.851 INFO:tasks.workunit.client.0.smithi167.stdout:4/186: rename d0/d6/d21 to d0/d6/d7/d39 0 2022-01-31T19:44:07.851 INFO:tasks.workunit.client.0.smithi167.stdout:4/187: write d0/d6/d7/d26/d31/f11 [8476659,98308] 0 2022-01-31T19:44:07.853 INFO:tasks.workunit.client.0.smithi167.stdout:7/282: rmdir d8/dc/d19 39 2022-01-31T19:44:07.855 INFO:tasks.workunit.client.0.smithi167.stdout:1/205: rename d0/d10/f34 to d0/d10/d20/f4e 0 2022-01-31T19:44:07.857 INFO:tasks.workunit.client.0.smithi167.stdout:4/188: truncate d0/d6/d7/d26/d31/d1c/f27 755410 0 2022-01-31T19:44:07.859 INFO:tasks.workunit.client.0.smithi167.stdout:1/206: truncate d0/d10/d18/f1e 776606 0 2022-01-31T19:44:07.859 INFO:tasks.workunit.client.0.smithi167.stdout:1/207: stat d0/d10/d20/f28 0 2022-01-31T19:44:07.860 INFO:tasks.workunit.client.0.smithi167.stdout:4/189: getdents d0/d6/d7/d26/d31/dd 0 2022-01-31T19:44:07.861 INFO:tasks.workunit.client.0.smithi167.stdout:1/208: write d0/d10/d20/f2e [2262249,59710] 0 2022-01-31T19:44:07.861 INFO:tasks.workunit.client.0.smithi167.stdout:7/283: dread d8/dc/d12/f62 [0,4194304] 0 2022-01-31T19:44:07.867 INFO:tasks.workunit.client.0.smithi167.stdout:4/190: mkdir d0/d6/d7/d39/d3a 0 2022-01-31T19:44:07.867 INFO:tasks.workunit.client.0.smithi167.stdout:6/177: dwrite d8/fb [0,4194304] 0 2022-01-31T19:44:07.867 INFO:tasks.workunit.client.0.smithi167.stdout:6/178: truncate d8/d20/f2f 616143 0 2022-01-31T19:44:07.867 INFO:tasks.workunit.client.0.smithi167.stdout:6/179: truncate d8/f31 1112980 0 2022-01-31T19:44:07.868 INFO:tasks.workunit.client.0.smithi167.stdout:1/209: rmdir d0/d11 39 2022-01-31T19:44:07.872 INFO:tasks.workunit.client.0.smithi167.stdout:4/191: getdents d0/d6/d7/d26/d31 0 2022-01-31T19:44:07.873 INFO:tasks.workunit.client.0.smithi167.stdout:6/180: creat d8/d20/f43 x:0 0 0 2022-01-31T19:44:07.874 INFO:tasks.workunit.client.0.smithi167.stdout:1/210: creat d0/d10/d20/f4f x:0 0 0 2022-01-31T19:44:07.874 INFO:tasks.workunit.client.0.smithi167.stdout:1/211: chown d0/f1b 402856532 1 2022-01-31T19:44:07.875 INFO:tasks.workunit.client.0.smithi167.stdout:7/284: rename d8/dc/d12 to d8/d2c/d6e 0 2022-01-31T19:44:07.876 INFO:tasks.workunit.client.0.smithi167.stdout:4/192: creat d0/d6/f3b x:0 0 0 2022-01-31T19:44:07.876 INFO:tasks.workunit.client.0.smithi167.stdout:4/193: stat d0/d6/d7/d26/d31/d1c/f33 0 2022-01-31T19:44:07.876 INFO:tasks.workunit.client.0.smithi167.stdout:4/194: truncate d0/d6/d7/d26/d31/d1c/f32 747273 0 2022-01-31T19:44:07.878 INFO:tasks.workunit.client.0.smithi167.stdout:7/285: link d8/dc/l34 d8/d2c/d6e/d4d/l6f 0 2022-01-31T19:44:07.879 INFO:tasks.workunit.client.0.smithi167.stdout:4/195: read d0/d6/d7/d26/d31/d1c/f32 [119804,14837] 0 2022-01-31T19:44:07.881 INFO:tasks.workunit.client.0.smithi167.stdout:6/181: rename d8/fa to d8/d20/f44 0 2022-01-31T19:44:07.881 INFO:tasks.workunit.client.0.smithi167.stdout:6/182: chown d8/f1e 47 1 2022-01-31T19:44:07.885 INFO:tasks.workunit.client.0.smithi167.stdout:6/183: rename d8/c36 to d8/c45 0 2022-01-31T19:44:07.885 INFO:tasks.workunit.client.0.smithi167.stdout:6/184: stat d8/d33/c34 0 2022-01-31T19:44:07.885 INFO:tasks.workunit.client.0.smithi167.stdout:6/185: readlink d8/d33/l3a 0 2022-01-31T19:44:07.888 INFO:tasks.workunit.client.0.smithi167.stdout:6/186: mknod d8/d20/c46 0 2022-01-31T19:44:07.889 INFO:tasks.workunit.client.0.smithi167.stdout:6/187: rename d8/d33/f2e to d8/d20/f47 0 2022-01-31T19:44:07.891 INFO:tasks.workunit.client.0.smithi167.stdout:7/286: dread d8/d35/f4b [0,4194304] 0 2022-01-31T19:44:07.898 INFO:tasks.workunit.client.0.smithi167.stdout:4/196: dwrite d0/d6/d7/d26/d31/d1c/f33 [0,4194304] 0 2022-01-31T19:44:07.898 INFO:tasks.workunit.client.0.smithi167.stdout:4/197: fsync d0/d6/d7/d26/d31/d1c/f27 0 2022-01-31T19:44:07.899 INFO:tasks.workunit.client.0.smithi167.stdout:1/212: dwrite d0/d10/d18/f3e [0,4194304] 0 2022-01-31T19:44:07.899 INFO:tasks.workunit.client.0.smithi167.stdout:1/213: write d0/fc [4112949,16670] 0 2022-01-31T19:44:07.915 INFO:tasks.workunit.client.0.smithi167.stdout:7/287: dwrite d8/dc/d18/f3b [0,4194304] 0 2022-01-31T19:44:07.915 INFO:tasks.workunit.client.0.smithi167.stdout:7/288: chown d8/d2c/d6e 1 1 2022-01-31T19:44:07.916 INFO:tasks.workunit.client.0.smithi167.stdout:4/198: dwrite d0/d6/f3b [0,4194304] 0 2022-01-31T19:44:07.969 INFO:tasks.workunit.client.0.smithi167.stdout:1/214: rename d0/d11/d24/f3f to d0/d11/d24/f50 0 2022-01-31T19:44:07.970 INFO:tasks.workunit.client.0.smithi167.stdout:1/215: write d0/d10/d18/f2b [426718,128984] 0 2022-01-31T19:44:07.972 INFO:tasks.workunit.client.0.smithi167.stdout:7/289: mknod d8/dc/d18/d24/c70 0 2022-01-31T19:44:07.975 INFO:tasks.workunit.client.0.smithi167.stdout:4/199: rename d0/d6/d7/f14 to d0/d6/d7/d39/d3a/f3c 0 2022-01-31T19:44:07.975 INFO:tasks.workunit.client.0.smithi167.stdout:4/200: write d0/d6/d7/f30 [180077,84085] 0 2022-01-31T19:44:07.975 INFO:tasks.workunit.client.0.smithi167.stdout:4/201: readlink d0/d6/d7/d26/d31/l10 0 2022-01-31T19:44:07.982 INFO:tasks.workunit.client.0.smithi167.stdout:7/290: rmdir d8/d2c/d6e/d4d 39 2022-01-31T19:44:07.983 INFO:tasks.workunit.client.0.smithi167.stdout:7/291: fdatasync d8/d35/f4b 0 2022-01-31T19:44:07.985 INFO:tasks.workunit.client.0.smithi167.stdout:4/202: link d0/d6/d7/d26/d31/dd/f2e d0/d6/d7/d26/d31/dd/f3d 0 2022-01-31T19:44:07.985 INFO:tasks.workunit.client.0.smithi167.stdout:4/203: readlink d0/d6/d7/d26/d31/l25 0 2022-01-31T19:44:07.985 INFO:tasks.workunit.client.0.smithi167.stdout:4/204: dread - d0/d6/d7/d26/f2f zero size 2022-01-31T19:44:07.987 INFO:tasks.workunit.client.0.smithi167.stdout:1/216: dwrite d0/d10/f42 [0,4194304] 0 2022-01-31T19:44:07.987 INFO:tasks.workunit.client.0.smithi167.stdout:1/217: stat d0/d11/l21 0 2022-01-31T19:44:07.987 INFO:tasks.workunit.client.0.smithi167.stdout:4/205: symlink d0/d6/d7/d26/d31/dd/l3e 0 2022-01-31T19:44:07.988 INFO:tasks.workunit.client.0.smithi167.stdout:7/292: rmdir d8/d35 39 2022-01-31T19:44:07.988 INFO:tasks.workunit.client.0.smithi167.stdout:1/218: rmdir d0/d11/d24/d33/d4b/d26 39 2022-01-31T19:44:07.994 INFO:tasks.workunit.client.0.smithi167.stdout:3/253: sync 2022-01-31T19:44:07.994 INFO:tasks.workunit.client.0.smithi167.stdout:5/255: sync 2022-01-31T19:44:07.994 INFO:tasks.workunit.client.0.smithi167.stdout:9/153: sync 2022-01-31T19:44:07.995 INFO:tasks.workunit.client.0.smithi167.stdout:3/254: readlink d9/l19 0 2022-01-31T19:44:07.995 INFO:tasks.workunit.client.0.smithi167.stdout:3/255: read - d9/d26/f2a zero size 2022-01-31T19:44:07.995 INFO:tasks.workunit.client.0.smithi167.stdout:5/256: write d5/d6/d16/f3b [599998,128900] 0 2022-01-31T19:44:07.996 INFO:tasks.workunit.client.0.smithi167.stdout:3/256: read d9/f10 [294746,1046] 0 2022-01-31T19:44:07.996 INFO:tasks.workunit.client.0.smithi167.stdout:3/257: creat d9/f42 x:0 0 0 2022-01-31T19:44:07.996 INFO:tasks.workunit.client.0.smithi167.stdout:5/257: mknod d5/d6/d16/d24/c51 0 2022-01-31T19:44:07.997 INFO:tasks.workunit.client.0.smithi167.stdout:5/258: dread d5/d6/d16/f38 [0,4194304] 0 2022-01-31T19:44:08.004 INFO:tasks.workunit.client.0.smithi167.stdout:4/206: rename d0/d6/d7/d26/d31/dd to d0/d3f 0 2022-01-31T19:44:08.004 INFO:tasks.workunit.client.0.smithi167.stdout:4/207: chown d0/d6/d7/d39/d3a 605009160 1 2022-01-31T19:44:08.004 INFO:tasks.workunit.client.0.smithi167.stdout:3/258: mknod d9/d29/c43 0 2022-01-31T19:44:08.005 INFO:tasks.workunit.client.0.smithi167.stdout:3/259: mkdir d9/d29/d44 0 2022-01-31T19:44:08.006 INFO:tasks.workunit.client.0.smithi167.stdout:3/260: truncate d9/d29/d2e/f39 2308357 0 2022-01-31T19:44:08.016 INFO:tasks.workunit.client.0.smithi167.stdout:4/208: dread d0/d6/d7/d26/d31/d1c/f33 [0,4194304] 0 2022-01-31T19:44:08.016 INFO:tasks.workunit.client.0.smithi167.stdout:4/209: chown d0/d6/d7/d26/d31/f17 119112 1 2022-01-31T19:44:08.017 INFO:tasks.workunit.client.0.smithi167.stdout:4/210: fdatasync d0/d6/d7/d26/d31/f11 0 2022-01-31T19:44:08.017 INFO:tasks.workunit.client.0.smithi167.stdout:4/211: unlink d0/d6/d7/d39/d3a/f3c 0 2022-01-31T19:44:08.018 INFO:tasks.workunit.client.0.smithi167.stdout:4/212: truncate d0/d6/d7/d39/f36 638523 0 2022-01-31T19:44:08.020 INFO:tasks.workunit.client.0.smithi167.stdout:4/213: rename d0/d6/d7/d26/d31/d1c to d0/d6/d7/d26/d40 0 2022-01-31T19:44:08.021 INFO:tasks.workunit.client.0.smithi167.stdout:4/214: rmdir d0/d6/d7/d39 39 2022-01-31T19:44:08.021 INFO:tasks.workunit.client.0.smithi167.stdout:4/215: getdents d0/d6/d7/d26/d40 0 2022-01-31T19:44:08.022 INFO:tasks.workunit.client.0.smithi167.stdout:4/216: dread d0/d3f/f2e [0,4194304] 0 2022-01-31T19:44:08.023 INFO:tasks.workunit.client.0.smithi167.stdout:4/217: fdatasync d0/d6/d7/d26/d40/f24 0 2022-01-31T19:44:08.023 INFO:tasks.workunit.client.0.smithi167.stdout:7/293: dwrite d8/d2c/d43/d31/f6a [0,4194304] 0 2022-01-31T19:44:08.024 INFO:tasks.workunit.client.0.smithi167.stdout:1/219: dwrite d0/f1b [4194304,4194304] 0 2022-01-31T19:44:08.024 INFO:tasks.workunit.client.0.smithi167.stdout:9/154: dwrite dd/d19/f21 [0,4194304] 0 2022-01-31T19:44:08.025 INFO:tasks.workunit.client.0.smithi167.stdout:5/259: dwrite d5/f9 [4194304,4194304] 0 2022-01-31T19:44:08.026 INFO:tasks.workunit.client.0.smithi167.stdout:4/218: creat d0/d3f/d1a/f41 x:0 0 0 2022-01-31T19:44:08.028 INFO:tasks.workunit.client.0.smithi167.stdout:1/220: symlink d0/d11/d24/l51 0 2022-01-31T19:44:08.030 INFO:tasks.workunit.client.0.smithi167.stdout:4/219: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:08.031 INFO:tasks.workunit.client.0.smithi167.stdout:4/220: stat d0 0 2022-01-31T19:44:08.031 INFO:tasks.workunit.client.0.smithi167.stdout:4/221: getdents d0/d6/d7/d26/d40 0 2022-01-31T19:44:08.031 INFO:tasks.workunit.client.0.smithi167.stdout:4/222: truncate d0/d3f/d1a/f41 247385 0 2022-01-31T19:44:08.032 INFO:tasks.workunit.client.0.smithi167.stdout:3/261: rmdir d9 39 2022-01-31T19:44:08.032 INFO:tasks.workunit.client.0.smithi167.stdout:9/155: rename dd/f14 to dd/d17/f2f 0 2022-01-31T19:44:08.033 INFO:tasks.workunit.client.0.smithi167.stdout:7/294: dread d8/d2c/d6e/f2a [0,4194304] 0 2022-01-31T19:44:08.034 INFO:tasks.workunit.client.0.smithi167.stdout:5/260: mkdir d5/d6/d16/d24/d52 0 2022-01-31T19:44:08.036 INFO:tasks.workunit.client.0.smithi167.stdout:1/221: rmdir d0/d11/d24/d33/d4b/d1f 39 2022-01-31T19:44:08.036 INFO:tasks.workunit.client.0.smithi167.stdout:1/222: creat d0/d10/d18/d39/f52 x:0 0 0 2022-01-31T19:44:08.037 INFO:tasks.workunit.client.0.smithi167.stdout:4/223: unlink d0/d3f/l1f 0 2022-01-31T19:44:08.038 INFO:tasks.workunit.client.0.smithi167.stdout:9/156: rmdir dd 39 2022-01-31T19:44:08.047 INFO:tasks.workunit.client.0.smithi167.stdout:7/295: mkdir d8/d2c/d6e/d4d/d5e/d71 0 2022-01-31T19:44:08.048 INFO:tasks.workunit.client.0.smithi167.stdout:1/223: symlink d0/d10/d20/l53 0 2022-01-31T19:44:08.050 INFO:tasks.workunit.client.0.smithi167.stdout:9/157: rename dd/d19/c2b to dd/d17/c30 0 2022-01-31T19:44:08.050 INFO:tasks.workunit.client.0.smithi167.stdout:9/158: fdatasync dd/d19/f1b 0 2022-01-31T19:44:08.051 INFO:tasks.workunit.client.0.smithi167.stdout:7/296: link d8/d2c/d6e/l64 d8/d2c/d6e/d4d/d67/l72 0 2022-01-31T19:44:08.053 INFO:tasks.workunit.client.0.smithi167.stdout:3/262: dwrite d9/d26/f2a [0,4194304] 0 2022-01-31T19:44:08.053 INFO:tasks.workunit.client.0.smithi167.stdout:1/224: unlink d0/d10/d18/f2b 0 2022-01-31T19:44:08.054 INFO:tasks.workunit.client.0.smithi167.stdout:1/225: dread - d0/d11/f4d zero size 2022-01-31T19:44:08.054 INFO:tasks.workunit.client.0.smithi167.stdout:5/261: dwrite d5/d6/d16/d18/f2e [0,4194304] 0 2022-01-31T19:44:08.054 INFO:tasks.workunit.client.0.smithi167.stdout:5/262: fsync d5/d6/d16/f26 0 2022-01-31T19:44:08.055 INFO:tasks.workunit.client.0.smithi167.stdout:4/224: mknod d0/d6/d7/d39/c42 0 2022-01-31T19:44:08.055 INFO:tasks.workunit.client.0.smithi167.stdout:7/297: creat d8/dc/d18/d24/f73 x:0 0 0 2022-01-31T19:44:08.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/263: write f7 [278897,99245] 0 2022-01-31T19:44:08.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/264: chown d9/d29/d2e/f39 451135 1 2022-01-31T19:44:08.056 INFO:tasks.workunit.client.0.smithi167.stdout:3/265: chown d9/d26/f31 128515 1 2022-01-31T19:44:08.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:07 smithi171 conmon[35325]: cluster 2022-01-31T19:44:06.431017+0000 mgr.smithi171.asyxnc (mgr. 2022-01-31T19:44:08.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:07 smithi171 conmon[35325]: 24387) 13 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 365 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 501 op/s 2022-01-31T19:44:08.119 INFO:tasks.workunit.client.0.smithi167.stdout:1/226: symlink d0/d10/d20/l54 0 2022-01-31T19:44:08.127 INFO:tasks.workunit.client.0.smithi167.stdout:5/263: rename d5/d6/la to d5/d6/d16/l53 0 2022-01-31T19:44:08.127 INFO:tasks.workunit.client.0.smithi167.stdout:7/298: symlink d8/d2c/d43/l74 0 2022-01-31T19:44:08.129 INFO:tasks.workunit.client.0.smithi167.stdout:7/299: dread d8/dc/f33 [0,4194304] 0 2022-01-31T19:44:08.129 INFO:tasks.workunit.client.0.smithi167.stdout:1/227: creat d0/d11/d24/d33/f55 x:0 0 0 2022-01-31T19:44:08.130 INFO:tasks.workunit.client.0.smithi167.stdout:1/228: chown d0/f25 14 1 2022-01-31T19:44:08.130 INFO:tasks.workunit.client.0.smithi167.stdout:1/229: creat d0/d10/d20/f56 x:0 0 0 2022-01-31T19:44:08.130 INFO:tasks.workunit.client.0.smithi167.stdout:1/230: stat d0/d11/d24/d33/l36 0 2022-01-31T19:44:08.135 INFO:tasks.workunit.client.0.smithi167.stdout:1/231: creat d0/d11/d24/d33/d4b/f57 x:0 0 0 2022-01-31T19:44:08.137 INFO:tasks.workunit.client.0.smithi167.stdout:4/225: rename d0/d6/d7/d26/d40/f33 to d0/d6/d7/d39/f43 0 2022-01-31T19:44:08.138 INFO:tasks.workunit.client.0.smithi167.stdout:3/266: dread d9/f11 [0,4194304] 0 2022-01-31T19:44:08.138 INFO:tasks.workunit.client.0.smithi167.stdout:3/267: write d9/d29/d2e/f3e [694270,68940] 0 2022-01-31T19:44:08.139 INFO:tasks.workunit.client.0.smithi167.stdout:1/232: mknod d0/d11/d46/c58 0 2022-01-31T19:44:08.139 INFO:tasks.workunit.client.0.smithi167.stdout:5/264: dread d5/f8 [0,4194304] 0 2022-01-31T19:44:08.139 INFO:tasks.workunit.client.0.smithi167.stdout:1/233: stat d0/d10/d20 0 2022-01-31T19:44:08.139 INFO:tasks.workunit.client.0.smithi167.stdout:5/265: write d5/d6/de/f4c [241779,4221] 0 2022-01-31T19:44:08.140 INFO:tasks.workunit.client.0.smithi167.stdout:1/234: read - d0/d11/d24/d33/f55 zero size 2022-01-31T19:44:08.140 INFO:tasks.workunit.client.0.smithi167.stdout:5/266: chown d5/d6/d16/d18/f1f 201837895 1 2022-01-31T19:44:08.140 INFO:tasks.workunit.client.0.smithi167.stdout:3/268: write d9/fc [973474,113928] 0 2022-01-31T19:44:08.141 INFO:tasks.workunit.client.0.smithi167.stdout:4/226: creat d0/d6/d7/d39/f44 x:0 0 0 2022-01-31T19:44:08.142 INFO:tasks.workunit.client.0.smithi167.stdout:4/227: truncate d0/d6/d7/d26/d40/f24 500935 0 2022-01-31T19:44:08.142 INFO:tasks.workunit.client.0.smithi167.stdout:4/228: write d0/d6/d7/f30 [693069,100727] 0 2022-01-31T19:44:08.143 INFO:tasks.workunit.client.0.smithi167.stdout:3/269: write d9/d26/f31 [3826073,126600] 0 2022-01-31T19:44:08.143 INFO:tasks.workunit.client.0.smithi167.stdout:3/270: getdents d9/d29/d44 0 2022-01-31T19:44:08.143 INFO:tasks.workunit.client.0.smithi167.stdout:3/271: chown d9/cd 185412 1 2022-01-31T19:44:08.144 INFO:tasks.workunit.client.0.smithi167.stdout:4/229: symlink d0/d6/d7/d26/d40/l45 0 2022-01-31T19:44:08.144 INFO:tasks.workunit.client.0.smithi167.stdout:4/230: stat d0/d3f/d1a/f41 0 2022-01-31T19:44:08.146 INFO:tasks.workunit.client.0.smithi167.stdout:4/231: mknod d0/d6/d7/d26/d31/c46 0 2022-01-31T19:44:08.150 INFO:tasks.workunit.client.0.smithi167.stdout:1/235: dread d0/f6 [0,4194304] 0 2022-01-31T19:44:08.150 INFO:tasks.workunit.client.0.smithi167.stdout:1/236: chown d0/d10/d20/c32 11 1 2022-01-31T19:44:08.150 INFO:tasks.workunit.client.0.smithi167.stdout:5/267: dread d5/d6/f28 [0,4194304] 0 2022-01-31T19:44:08.152 INFO:tasks.workunit.client.0.smithi167.stdout:7/300: dwrite d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:08.152 INFO:tasks.workunit.client.0.smithi167.stdout:1/237: creat d0/d11/d24/d33/d4b/d26/f59 x:0 0 0 2022-01-31T19:44:08.153 INFO:tasks.workunit.client.0.smithi167.stdout:5/268: link d5/d6/de/f2a d5/d6/de/d22/f54 0 2022-01-31T19:44:08.161 INFO:tasks.workunit.client.0.smithi167.stdout:3/272: dread f8 [0,4194304] 0 2022-01-31T19:44:08.163 INFO:tasks.workunit.client.0.smithi167.stdout:5/269: dread d5/f9 [0,4194304] 0 2022-01-31T19:44:08.164 INFO:tasks.workunit.client.0.smithi167.stdout:1/238: getdents d0/d11/d24/d33 0 2022-01-31T19:44:08.165 INFO:tasks.workunit.client.0.smithi167.stdout:5/270: mknod d5/d6/c55 0 2022-01-31T19:44:08.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:07 smithi167 conmon[32206]: cluster 2022-01-31T19:44:06.431017+0000 mgr.smithi171.asyxnc (mgr. 2022-01-31T19:44:08.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:07 smithi167 conmon[32206]: 24387) 13 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 365 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 501 op/s 2022-01-31T19:44:08.172 INFO:tasks.workunit.client.0.smithi167.stdout:1/239: rename d0/d11/d24 to d0/d10/d5a 0 2022-01-31T19:44:08.173 INFO:tasks.workunit.client.0.smithi167.stdout:1/240: creat d0/d10/d5a/d33/d4b/f5b x:0 0 0 2022-01-31T19:44:08.174 INFO:tasks.workunit.client.0.smithi167.stdout:1/241: dread - d0/d10/d20/f56 zero size 2022-01-31T19:44:08.174 INFO:tasks.workunit.client.0.smithi167.stdout:1/242: rename d0/d10/d5a/c3b to d0/d10/d5a/d33/d4b/d26/c5c 0 2022-01-31T19:44:08.175 INFO:tasks.workunit.client.0.smithi167.stdout:1/243: mkdir d0/d10/d18/d39/d5d 0 2022-01-31T19:44:08.176 INFO:tasks.workunit.client.0.smithi167.stdout:1/244: symlink d0/d10/d5a/d33/d4b/d26/l5e 0 2022-01-31T19:44:08.177 INFO:tasks.workunit.client.0.smithi167.stdout:1/245: mknod d0/d10/d5a/d33/d4b/d26/c5f 0 2022-01-31T19:44:08.177 INFO:tasks.workunit.client.0.smithi167.stdout:1/246: write d0/d10/d18/d39/f52 [58537,69101] 0 2022-01-31T19:44:08.194 INFO:tasks.workunit.client.0.smithi167.stdout:1/247: dwrite d0/f1b [4194304,4194304] 0 2022-01-31T19:44:08.197 INFO:tasks.workunit.client.0.smithi167.stdout:5/271: dwrite d5/d6/d16/f38 [0,4194304] 0 2022-01-31T19:44:08.198 INFO:tasks.workunit.client.0.smithi167.stdout:1/248: symlink d0/d10/d5a/d33/d4b/l60 0 2022-01-31T19:44:08.198 INFO:tasks.workunit.client.0.smithi167.stdout:1/249: fdatasync d0/d10/d20/f37 0 2022-01-31T19:44:08.198 INFO:tasks.workunit.client.0.smithi167.stdout:5/272: write d5/d6/d16/d18/f1f [2472617,70932] 0 2022-01-31T19:44:08.201 INFO:tasks.workunit.client.0.smithi167.stdout:1/250: mknod d0/d11/d45/c61 0 2022-01-31T19:44:08.201 INFO:tasks.workunit.client.0.smithi167.stdout:1/251: mkdir d0/d62 0 2022-01-31T19:44:08.202 INFO:tasks.workunit.client.0.smithi167.stdout:1/252: write d0/d10/f42 [4476421,67735] 0 2022-01-31T19:44:08.202 INFO:tasks.workunit.client.0.smithi167.stdout:5/273: getdents d5/d6 0 2022-01-31T19:44:08.203 INFO:tasks.workunit.client.0.smithi167.stdout:1/253: write d0/f1b [4948613,66361] 0 2022-01-31T19:44:08.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/274: mknod d5/d6/de/d22/c56 0 2022-01-31T19:44:08.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/275: readlink d5/lb 0 2022-01-31T19:44:08.206 INFO:tasks.workunit.client.0.smithi167.stdout:1/254: symlink d0/d11/l63 0 2022-01-31T19:44:08.206 INFO:tasks.workunit.client.0.smithi167.stdout:1/255: mknod d0/c64 0 2022-01-31T19:44:08.228 INFO:tasks.workunit.client.0.smithi167.stdout:5/276: dwrite d5/d6/d16/d24/f4b [0,4194304] 0 2022-01-31T19:44:08.229 INFO:tasks.workunit.client.0.smithi167.stdout:5/277: chown d5/d6/d16/d18/f2c 4 1 2022-01-31T19:44:08.229 INFO:tasks.workunit.client.0.smithi167.stdout:5/278: fdatasync d5/d6/d16/f38 0 2022-01-31T19:44:08.229 INFO:tasks.workunit.client.0.smithi167.stdout:1/256: dwrite d0/d10/d20/f2e [0,4194304] 0 2022-01-31T19:44:08.232 INFO:tasks.workunit.client.0.smithi167.stdout:1/257: link d0/f6 d0/d10/d5a/d33/d4b/f65 0 2022-01-31T19:44:08.233 INFO:tasks.workunit.client.0.smithi167.stdout:1/258: creat d0/d10/d5a/d33/d4b/d26/f66 x:0 0 0 2022-01-31T19:44:08.246 INFO:tasks.workunit.client.0.smithi167.stdout:8/298: sync 2022-01-31T19:44:08.246 INFO:tasks.workunit.client.0.smithi167.stdout:8/299: mkdir d6/db/d2f/d51/d60 0 2022-01-31T19:44:08.247 INFO:tasks.workunit.client.0.smithi167.stdout:8/300: truncate d6/db/f2b 8730954 0 2022-01-31T19:44:08.247 INFO:tasks.workunit.client.0.smithi167.stdout:8/301: mkdir d6/db/d28/d61 0 2022-01-31T19:44:08.249 INFO:tasks.workunit.client.0.smithi167.stdout:5/279: dwrite d5/d6/d16/d18/f3d [0,4194304] 0 2022-01-31T19:44:08.252 INFO:tasks.workunit.client.0.smithi167.stdout:5/280: creat d5/d6/d16/d18/d43/f57 x:0 0 0 2022-01-31T19:44:08.253 INFO:tasks.workunit.client.0.smithi167.stdout:5/281: stat d5/d6/de/l40 0 2022-01-31T19:44:08.254 INFO:tasks.workunit.client.0.smithi167.stdout:5/282: dread d5/d6/d16/f3b [0,4194304] 0 2022-01-31T19:44:08.255 INFO:tasks.workunit.client.0.smithi167.stdout:5/283: creat d5/d6/d16/d18/f58 x:0 0 0 2022-01-31T19:44:08.255 INFO:tasks.workunit.client.0.smithi167.stdout:8/302: dread d6/db/de/d18/f19 [0,4194304] 0 2022-01-31T19:44:08.256 INFO:tasks.workunit.client.0.smithi167.stdout:5/284: rename d5/f34 to d5/d6/d16/d18/f59 0 2022-01-31T19:44:08.257 INFO:tasks.workunit.client.0.smithi167.stdout:8/303: symlink d6/db/l62 0 2022-01-31T19:44:08.258 INFO:tasks.workunit.client.0.smithi167.stdout:8/304: creat d6/db/d36/d5d/f63 x:0 0 0 2022-01-31T19:44:08.259 INFO:tasks.workunit.client.0.smithi167.stdout:8/305: chown d6/db/l25 225 1 2022-01-31T19:44:08.266 INFO:tasks.workunit.client.0.smithi167.stdout:1/259: dwrite d0/d10/d20/f2e [0,4194304] 0 2022-01-31T19:44:08.267 INFO:tasks.workunit.client.0.smithi167.stdout:1/260: dread - d0/f30 zero size 2022-01-31T19:44:08.267 INFO:tasks.workunit.client.0.smithi167.stdout:1/261: write d0/f17 [250844,95998] 0 2022-01-31T19:44:08.273 INFO:tasks.workunit.client.0.smithi167.stdout:5/285: dread d5/f9 [4194304,4194304] 0 2022-01-31T19:44:08.273 INFO:tasks.workunit.client.0.smithi167.stdout:5/286: stat d5/d6/d3a/f21 0 2022-01-31T19:44:08.275 INFO:tasks.workunit.client.0.smithi167.stdout:5/287: truncate d5/f9 2628247 0 2022-01-31T19:44:08.276 INFO:tasks.workunit.client.0.smithi167.stdout:5/288: fdatasync d5/d6/d16/d18/f3d 0 2022-01-31T19:44:08.287 INFO:tasks.workunit.client.0.smithi167.stdout:8/306: dwrite d6/f8 [4194304,4194304] 0 2022-01-31T19:44:08.290 INFO:tasks.workunit.client.0.smithi167.stdout:1/262: dwrite d0/d10/f42 [0,4194304] 0 2022-01-31T19:44:08.290 INFO:tasks.workunit.client.0.smithi167.stdout:1/263: chown d0/f1 14932 1 2022-01-31T19:44:08.290 INFO:tasks.workunit.client.0.smithi167.stdout:1/264: fsync d0/d10/d5a/f50 0 2022-01-31T19:44:08.290 INFO:tasks.workunit.client.0.smithi167.stdout:5/289: rename d5/f9 to d5/d6/d16/f5a 0 2022-01-31T19:44:08.291 INFO:tasks.workunit.client.0.smithi167.stdout:1/265: getdents d0/d11/d46 0 2022-01-31T19:44:08.292 INFO:tasks.workunit.client.0.smithi167.stdout:8/307: rename d6/db/de/l5c to d6/db/de/d18/l64 0 2022-01-31T19:44:08.293 INFO:tasks.workunit.client.0.smithi167.stdout:1/266: mknod d0/d10/d5a/d33/c67 0 2022-01-31T19:44:08.297 INFO:tasks.workunit.client.0.smithi167.stdout:8/308: mknod d6/c65 0 2022-01-31T19:44:08.298 INFO:tasks.workunit.client.0.smithi167.stdout:5/290: rmdir d5/d6/d16/d18/d43 39 2022-01-31T19:44:08.298 INFO:tasks.workunit.client.0.smithi167.stdout:8/309: chown d6/db/l25 319 1 2022-01-31T19:44:08.298 INFO:tasks.workunit.client.0.smithi167.stdout:8/310: truncate d6/db/d4e/f4f 644327 0 2022-01-31T19:44:08.300 INFO:tasks.workunit.client.0.smithi167.stdout:5/291: truncate d5/d6/d16/d18/f59 1261762 0 2022-01-31T19:44:08.303 INFO:tasks.workunit.client.0.smithi167.stdout:5/292: rename d5/d6/d16/d24/d52 to d5/d6/d16/d18/d5b 0 2022-01-31T19:44:08.303 INFO:tasks.workunit.client.0.smithi167.stdout:5/293: chown d5/d6/d16/d18/f3d 226 1 2022-01-31T19:44:08.305 INFO:tasks.workunit.client.0.smithi167.stdout:5/294: creat d5/d6/d16/d18/d43/f5c x:0 0 0 2022-01-31T19:44:08.305 INFO:tasks.workunit.client.0.smithi167.stdout:5/295: readlink d5/d6/d16/l39 0 2022-01-31T19:44:08.308 INFO:tasks.workunit.client.0.smithi167.stdout:5/296: rename d5/d6/d16/d18/l4f to d5/d6/l5d 0 2022-01-31T19:44:08.308 INFO:tasks.workunit.client.0.smithi167.stdout:5/297: fdatasync d5/d6/d16/d18/f2c 0 2022-01-31T19:44:08.309 INFO:tasks.workunit.client.0.smithi167.stdout:5/298: symlink d5/l5e 0 2022-01-31T19:44:08.311 INFO:tasks.workunit.client.0.smithi167.stdout:5/299: link d5/d6/d16/d18/d43/f57 d5/d6/de/d22/f5f 0 2022-01-31T19:44:08.313 INFO:tasks.workunit.client.0.smithi167.stdout:5/300: creat d5/d6/d16/d18/d43/d45/f60 x:0 0 0 2022-01-31T19:44:08.313 INFO:tasks.workunit.client.0.smithi167.stdout:5/301: chown d5/d6/de/f3e 6 1 2022-01-31T19:44:08.315 INFO:tasks.workunit.client.0.smithi167.stdout:8/311: dwrite d6/f8 [8388608,4194304] 0 2022-01-31T19:44:08.317 INFO:tasks.workunit.client.0.smithi167.stdout:1/267: dwrite d0/d10/d20/f28 [0,4194304] 0 2022-01-31T19:44:08.319 INFO:tasks.workunit.client.0.smithi167.stdout:8/312: write d6/db/d36/f3b [2728690,123231] 0 2022-01-31T19:44:08.319 INFO:tasks.workunit.client.0.smithi167.stdout:8/313: dread - d6/d1e/f3c zero size 2022-01-31T19:44:08.320 INFO:tasks.workunit.client.0.smithi167.stdout:8/314: fsync d6/ff 0 2022-01-31T19:44:08.320 INFO:tasks.workunit.client.0.smithi167.stdout:1/268: write d0/d10/d20/f4e [3440146,4071] 0 2022-01-31T19:44:08.321 INFO:tasks.workunit.client.0.smithi167.stdout:1/269: write d0/d10/d18/d39/f3d [480399,53557] 0 2022-01-31T19:44:08.321 INFO:tasks.workunit.client.0.smithi167.stdout:1/270: chown d0/c2 33210383 1 2022-01-31T19:44:08.325 INFO:tasks.workunit.client.0.smithi167.stdout:1/271: dread d0/f6 [0,4194304] 0 2022-01-31T19:44:08.332 INFO:tasks.workunit.client.0.smithi167.stdout:8/315: mkdir d6/db/d28/d66 0 2022-01-31T19:44:08.335 INFO:tasks.workunit.client.0.smithi167.stdout:5/302: rename d5/d6/d16/d24 to d5/d61 0 2022-01-31T19:44:08.340 INFO:tasks.workunit.client.0.smithi167.stdout:1/272: rename d0/fc to d0/d10/d5a/d33/d4b/f68 0 2022-01-31T19:44:08.346 INFO:tasks.workunit.client.0.smithi167.stdout:8/316: dread d6/fa [0,4194304] 0 2022-01-31T19:44:08.352 INFO:tasks.workunit.client.0.smithi167.stdout:5/303: dwrite d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:08.352 INFO:tasks.workunit.client.0.smithi167.stdout:5/304: chown d5/d6/d16/f38 438602 1 2022-01-31T19:44:08.353 INFO:tasks.workunit.client.0.smithi167.stdout:5/305: truncate d5/f8 8302209 0 2022-01-31T19:44:08.353 INFO:tasks.workunit.client.0.smithi167.stdout:5/306: fdatasync d5/d6/d16/d18/f31 0 2022-01-31T19:44:08.368 INFO:tasks.workunit.client.0.smithi167.stdout:5/307: dwrite d5/d6/d3a/f21 [0,4194304] 0 2022-01-31T19:44:08.369 INFO:tasks.workunit.client.0.smithi167.stdout:8/317: dwrite d6/db/d36/f4a [0,4194304] 0 2022-01-31T19:44:08.376 INFO:tasks.workunit.client.0.smithi167.stdout:5/308: dread d5/d6/d3a/f21 [0,4194304] 0 2022-01-31T19:44:08.376 INFO:tasks.workunit.client.0.smithi167.stdout:5/309: stat d5/d6/d16/d18 0 2022-01-31T19:44:08.377 INFO:tasks.workunit.client.0.smithi167.stdout:5/310: creat d5/d6/f62 x:0 0 0 2022-01-31T19:44:08.378 INFO:tasks.workunit.client.0.smithi167.stdout:5/311: write d5/f8 [5558809,124946] 0 2022-01-31T19:44:08.385 INFO:tasks.workunit.client.0.smithi167.stdout:8/318: dwrite d6/d1e/f22 [0,4194304] 0 2022-01-31T19:44:08.388 INFO:tasks.workunit.client.0.smithi167.stdout:8/319: creat d6/db/d2f/d51/f67 x:0 0 0 2022-01-31T19:44:08.394 INFO:tasks.workunit.client.0.smithi167.stdout:5/312: dwrite d5/d6/f62 [0,4194304] 0 2022-01-31T19:44:08.396 INFO:tasks.workunit.client.0.smithi167.stdout:8/320: rmdir d6/db/d2f/d51 39 2022-01-31T19:44:08.397 INFO:tasks.workunit.client.0.smithi167.stdout:8/321: dread - d6/db/d2f/d51/f67 zero size 2022-01-31T19:44:08.397 INFO:tasks.workunit.client.0.smithi167.stdout:8/322: read d6/db/fd [3245996,47888] 0 2022-01-31T19:44:08.400 INFO:tasks.workunit.client.0.smithi167.stdout:8/323: link d6/db/d36/f4a d6/db/d36/d5d/f68 0 2022-01-31T19:44:08.401 INFO:tasks.workunit.client.0.smithi167.stdout:8/324: write d6/ff [4623291,85013] 0 2022-01-31T19:44:08.404 INFO:tasks.workunit.client.0.smithi167.stdout:5/313: rmdir d5 39 2022-01-31T19:44:08.408 INFO:tasks.workunit.client.0.smithi167.stdout:8/325: rename d6/db/d36/f4d to d6/d1e/f69 0 2022-01-31T19:44:08.408 INFO:tasks.workunit.client.0.smithi167.stdout:8/326: stat d6/f32 0 2022-01-31T19:44:08.408 INFO:tasks.workunit.client.0.smithi167.stdout:8/327: chown d6/f32 38362007 1 2022-01-31T19:44:08.409 INFO:tasks.workunit.client.0.smithi167.stdout:5/314: mkdir d5/d6/d16/d18/d5b/d63 0 2022-01-31T19:44:08.410 INFO:tasks.workunit.client.0.smithi167.stdout:5/315: write d5/d6/d16/d18/f2e [152384,38121] 0 2022-01-31T19:44:08.410 INFO:tasks.workunit.client.0.smithi167.stdout:8/328: mkdir d6/db/d6a 0 2022-01-31T19:44:08.411 INFO:tasks.workunit.client.0.smithi167.stdout:8/329: mkdir d6/d1e/d2d/d6b 0 2022-01-31T19:44:08.412 INFO:tasks.workunit.client.0.smithi167.stdout:8/330: dread d6/d1e/d2d/f40 [0,4194304] 0 2022-01-31T19:44:08.428 INFO:tasks.workunit.client.0.smithi167.stdout:8/331: dwrite d6/db/d28/f5a [0,4194304] 0 2022-01-31T19:44:08.429 INFO:tasks.workunit.client.0.smithi167.stdout:5/316: dwrite d5/d6/d3a/f4e [0,4194304] 0 2022-01-31T19:44:08.431 INFO:tasks.workunit.client.0.smithi167.stdout:2/214: sync 2022-01-31T19:44:08.431 INFO:tasks.workunit.client.0.smithi167.stdout:0/94: sync 2022-01-31T19:44:08.431 INFO:tasks.workunit.client.0.smithi167.stdout:6/188: sync 2022-01-31T19:44:08.432 INFO:tasks.workunit.client.0.smithi167.stdout:6/189: dread - d8/f14 zero size 2022-01-31T19:44:08.436 INFO:tasks.workunit.client.0.smithi167.stdout:5/317: link d5/f8 d5/d6/d16/d18/f64 0 2022-01-31T19:44:08.437 INFO:tasks.workunit.client.0.smithi167.stdout:5/318: dread - d5/d6/de/d22/f35 zero size 2022-01-31T19:44:08.437 INFO:tasks.workunit.client.0.smithi167.stdout:5/319: write d5/d6/d16/d18/f2c [1303884,86757] 0 2022-01-31T19:44:08.438 INFO:tasks.workunit.client.0.smithi167.stdout:0/95: unlink d17/d19/f1b 0 2022-01-31T19:44:08.439 INFO:tasks.workunit.client.0.smithi167.stdout:0/96: chown d17/c1a 10 1 2022-01-31T19:44:08.439 INFO:tasks.workunit.client.0.smithi167.stdout:0/97: fsync f8 0 2022-01-31T19:44:08.439 INFO:tasks.workunit.client.0.smithi167.stdout:5/320: getdents d5/d6/de/d22 0 2022-01-31T19:44:08.440 INFO:tasks.workunit.client.0.smithi167.stdout:0/98: mknod d17/d19/c1e 0 2022-01-31T19:44:08.441 INFO:tasks.workunit.client.0.smithi167.stdout:5/321: mkdir d5/d6/de/d22/d4d/d65 0 2022-01-31T19:44:08.442 INFO:tasks.workunit.client.0.smithi167.stdout:5/322: read d5/d6/f62 [3185096,40261] 0 2022-01-31T19:44:08.443 INFO:tasks.workunit.client.0.smithi167.stdout:5/323: symlink d5/d6/d3a/l66 0 2022-01-31T19:44:08.443 INFO:tasks.workunit.client.0.smithi167.stdout:5/324: fdatasync d5/d6/d16/f26 0 2022-01-31T19:44:08.457 INFO:tasks.workunit.client.0.smithi167.stdout:0/99: dread fc [0,4194304] 0 2022-01-31T19:44:08.457 INFO:tasks.workunit.client.0.smithi167.stdout:0/100: write f14 [668448,98475] 0 2022-01-31T19:44:08.461 INFO:tasks.workunit.client.0.smithi167.stdout:0/101: dread f8 [0,4194304] 0 2022-01-31T19:44:08.471 INFO:tasks.workunit.client.0.smithi167.stdout:6/190: dwrite d8/fd [0,4194304] 0 2022-01-31T19:44:08.472 INFO:tasks.workunit.client.0.smithi167.stdout:2/215: dwrite f7 [0,4194304] 0 2022-01-31T19:44:08.472 INFO:tasks.workunit.client.0.smithi167.stdout:8/332: dwrite d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:08.472 INFO:tasks.workunit.client.0.smithi167.stdout:6/191: read d8/d33/d35/f3d [549611,30742] 0 2022-01-31T19:44:08.476 INFO:tasks.workunit.client.0.smithi167.stdout:0/102: dread ff [0,4194304] 0 2022-01-31T19:44:08.476 INFO:tasks.workunit.client.0.smithi167.stdout:0/103: fsync fa 0 2022-01-31T19:44:08.476 INFO:tasks.workunit.client.0.smithi167.stdout:0/104: readlink d17/d19/d1c/l1d 0 2022-01-31T19:44:08.477 INFO:tasks.workunit.client.0.smithi167.stdout:0/105: write f14 [243706,11781] 0 2022-01-31T19:44:08.481 INFO:tasks.workunit.client.0.smithi167.stdout:2/216: mknod dd/df/d1b/c42 0 2022-01-31T19:44:08.481 INFO:tasks.workunit.client.0.smithi167.stdout:8/333: creat d6/db/d2f/d51/d60/f6c x:0 0 0 2022-01-31T19:44:08.483 INFO:tasks.workunit.client.0.smithi167.stdout:6/192: symlink d8/d33/d35/l48 0 2022-01-31T19:44:08.483 INFO:tasks.workunit.client.0.smithi167.stdout:8/334: dread d6/d1e/f29 [0,4194304] 0 2022-01-31T19:44:08.483 INFO:tasks.workunit.client.0.smithi167.stdout:8/335: dread - d6/db/de/d18/f55 zero size 2022-01-31T19:44:08.483 INFO:tasks.workunit.client.0.smithi167.stdout:0/106: truncate fc 1973460 0 2022-01-31T19:44:08.484 INFO:tasks.workunit.client.0.smithi167.stdout:2/217: rename dd/df/c11 to dd/c43 0 2022-01-31T19:44:08.484 INFO:tasks.workunit.client.0.smithi167.stdout:2/218: fdatasync dd/df/d22/f37 0 2022-01-31T19:44:08.485 INFO:tasks.workunit.client.0.smithi167.stdout:6/193: truncate f3 1017873 0 2022-01-31T19:44:08.486 INFO:tasks.workunit.client.0.smithi167.stdout:8/336: symlink d6/d1e/d4c/l6d 0 2022-01-31T19:44:08.488 INFO:tasks.workunit.client.0.smithi167.stdout:2/219: creat dd/df/d1b/d3d/f44 x:0 0 0 2022-01-31T19:44:08.488 INFO:tasks.workunit.client.0.smithi167.stdout:2/220: unlink dd/c3e 0 2022-01-31T19:44:08.501 INFO:tasks.workunit.client.0.smithi167.stdout:8/337: dread d6/ff [4194304,4194304] 0 2022-01-31T19:44:08.513 INFO:tasks.workunit.client.0.smithi167.stdout:0/107: dwrite f14 [0,4194304] 0 2022-01-31T19:44:08.514 INFO:tasks.workunit.client.0.smithi167.stdout:0/108: chown d17/c1a 33194413 1 2022-01-31T19:44:08.514 INFO:tasks.workunit.client.0.smithi167.stdout:0/109: dread - fa zero size 2022-01-31T19:44:08.514 INFO:tasks.workunit.client.0.smithi167.stdout:0/110: creat d17/d19/f1f x:0 0 0 2022-01-31T19:44:08.515 INFO:tasks.workunit.client.0.smithi167.stdout:0/111: symlink d17/d19/l20 0 2022-01-31T19:44:08.517 INFO:tasks.workunit.client.0.smithi167.stdout:0/112: creat d17/d19/d1c/f21 x:0 0 0 2022-01-31T19:44:08.517 INFO:tasks.workunit.client.0.smithi167.stdout:0/113: getdents d17 0 2022-01-31T19:44:08.517 INFO:tasks.workunit.client.0.smithi167.stdout:0/114: fdatasync ff 0 2022-01-31T19:44:08.518 INFO:tasks.workunit.client.0.smithi167.stdout:0/115: creat d17/f22 x:0 0 0 2022-01-31T19:44:08.519 INFO:tasks.workunit.client.0.smithi167.stdout:0/116: truncate f10 3070203 0 2022-01-31T19:44:08.520 INFO:tasks.workunit.client.0.smithi167.stdout:0/117: write d17/d19/d1c/f21 [373207,25570] 0 2022-01-31T19:44:08.520 INFO:tasks.workunit.client.0.smithi167.stdout:0/118: chown d17/d19/d1c/f21 4 1 2022-01-31T19:44:08.521 INFO:tasks.workunit.client.0.smithi167.stdout:6/194: dwrite d8/f19 [0,4194304] 0 2022-01-31T19:44:08.523 INFO:tasks.workunit.client.0.smithi167.stdout:8/338: dwrite d6/db/d36/f3b [0,4194304] 0 2022-01-31T19:44:08.527 INFO:tasks.workunit.client.0.smithi167.stdout:0/119: fsync d17/d19/d1c/f21 0 2022-01-31T19:44:08.528 INFO:tasks.workunit.client.0.smithi167.stdout:8/339: dread d6/db/d4e/f58 [0,4194304] 0 2022-01-31T19:44:08.530 INFO:tasks.workunit.client.0.smithi167.stdout:6/195: rename d8/d33/f41 to d8/d33/f49 0 2022-01-31T19:44:08.534 INFO:tasks.workunit.client.0.smithi167.stdout:6/196: chown d8/d20/c24 8531 1 2022-01-31T19:44:08.534 INFO:tasks.workunit.client.0.smithi167.stdout:6/197: dread - d8/f1e zero size 2022-01-31T19:44:08.537 INFO:tasks.workunit.client.0.smithi167.stdout:6/198: rename d8/l30 to d8/d33/l4a 0 2022-01-31T19:44:08.538 INFO:tasks.workunit.client.0.smithi167.stdout:6/199: write d8/d20/f26 [978828,46224] 0 2022-01-31T19:44:08.545 INFO:tasks.workunit.client.0.smithi167.stdout:9/159: sync 2022-01-31T19:44:08.545 INFO:tasks.workunit.client.0.smithi167.stdout:4/232: sync 2022-01-31T19:44:08.545 INFO:tasks.workunit.client.0.smithi167.stdout:3/273: sync 2022-01-31T19:44:08.545 INFO:tasks.workunit.client.0.smithi167.stdout:7/301: sync 2022-01-31T19:44:08.546 INFO:tasks.workunit.client.0.smithi167.stdout:9/160: fsync dd/d17/f18 0 2022-01-31T19:44:08.546 INFO:tasks.workunit.client.0.smithi167.stdout:9/161: write dd/f1a [4463358,126759] 0 2022-01-31T19:44:08.546 INFO:tasks.workunit.client.0.smithi167.stdout:9/162: readlink dd/d19/d24/l26 0 2022-01-31T19:44:08.546 INFO:tasks.workunit.client.0.smithi167.stdout:9/163: fdatasync f7 0 2022-01-31T19:44:08.547 INFO:tasks.workunit.client.0.smithi167.stdout:9/164: creat dd/d19/d24/f31 x:0 0 0 2022-01-31T19:44:08.547 INFO:tasks.workunit.client.0.smithi167.stdout:9/165: creat dd/d19/d24/f32 x:0 0 0 2022-01-31T19:44:08.547 INFO:tasks.workunit.client.0.smithi167.stdout:3/274: write d9/f30 [283743,65669] 0 2022-01-31T19:44:08.548 INFO:tasks.workunit.client.0.smithi167.stdout:4/233: mkdir d0/d6/d7/d26/d47 0 2022-01-31T19:44:08.549 INFO:tasks.workunit.client.0.smithi167.stdout:7/302: dread d8/dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:08.549 INFO:tasks.workunit.client.0.smithi167.stdout:7/303: write d8/d35/f3d [618237,111325] 0 2022-01-31T19:44:08.549 INFO:tasks.workunit.client.0.smithi167.stdout:7/304: dread d8/dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:08.549 INFO:tasks.workunit.client.0.smithi167.stdout:7/305: write d8/d2c/d6e/f44 [1248374,112965] 0 2022-01-31T19:44:08.550 INFO:tasks.workunit.client.0.smithi167.stdout:3/275: rename d9/l19 to d9/d29/d44/l45 0 2022-01-31T19:44:08.550 INFO:tasks.workunit.client.0.smithi167.stdout:3/276: read d9/f11 [47637,26559] 0 2022-01-31T19:44:08.551 INFO:tasks.workunit.client.0.smithi167.stdout:3/277: fsync d9/d21/f1e 0 2022-01-31T19:44:08.551 INFO:tasks.workunit.client.0.smithi167.stdout:7/306: mknod d8/d2c/d6e/d4d/d67/c75 0 2022-01-31T19:44:08.552 INFO:tasks.workunit.client.0.smithi167.stdout:7/307: creat d8/dc/d18/f76 x:0 0 0 2022-01-31T19:44:08.553 INFO:tasks.workunit.client.0.smithi167.stdout:3/278: rmdir d9/d29/d3b 39 2022-01-31T19:44:08.553 INFO:tasks.workunit.client.0.smithi167.stdout:0/120: dwrite d17/d19/d1c/f21 [0,4194304] 0 2022-01-31T19:44:08.555 INFO:tasks.workunit.client.0.smithi167.stdout:7/308: mknod d8/d2c/d6e/d4d/d5e/d71/c77 0 2022-01-31T19:44:08.556 INFO:tasks.workunit.client.0.smithi167.stdout:4/234: dread d0/d6/d7/f30 [0,4194304] 0 2022-01-31T19:44:08.557 INFO:tasks.workunit.client.0.smithi167.stdout:4/235: write d0/d6/d7/d26/d40/f32 [160902,104938] 0 2022-01-31T19:44:08.557 INFO:tasks.workunit.client.0.smithi167.stdout:4/236: readlink d0/d6/d7/l1e 0 2022-01-31T19:44:08.558 INFO:tasks.workunit.client.0.smithi167.stdout:4/237: creat d0/d6/d7/d39/d3a/f48 x:0 0 0 2022-01-31T19:44:08.559 INFO:tasks.workunit.client.0.smithi167.stdout:4/238: write d0/d6/d7/f8 [5746610,8419] 0 2022-01-31T19:44:08.560 INFO:tasks.workunit.client.0.smithi167.stdout:4/239: symlink d0/d3f/l49 0 2022-01-31T19:44:08.560 INFO:tasks.workunit.client.0.smithi167.stdout:4/240: read d0/d6/d7/d26/d40/f32 [4783,90595] 0 2022-01-31T19:44:08.561 INFO:tasks.workunit.client.0.smithi167.stdout:8/340: dwrite d6/db/f5b [0,4194304] 0 2022-01-31T19:44:08.563 INFO:tasks.workunit.client.0.smithi167.stdout:4/241: dread d0/d6/d7/d26/d40/f27 [0,4194304] 0 2022-01-31T19:44:08.563 INFO:tasks.workunit.client.0.smithi167.stdout:4/242: creat d0/d6/d7/f4a x:0 0 0 2022-01-31T19:44:08.563 INFO:tasks.workunit.client.0.smithi167.stdout:4/243: readlink d0/d6/d7/l1e 0 2022-01-31T19:44:08.564 INFO:tasks.workunit.client.0.smithi167.stdout:8/341: rmdir d6 39 2022-01-31T19:44:08.565 INFO:tasks.workunit.client.0.smithi167.stdout:4/244: truncate d0/d6/d7/d39/f43 3609580 0 2022-01-31T19:44:08.570 INFO:tasks.workunit.client.0.smithi167.stdout:8/342: write d6/db/de/f20 [3377635,35060] 0 2022-01-31T19:44:08.571 INFO:tasks.workunit.client.0.smithi167.stdout:6/200: dwrite d8/d20/f22 [0,4194304] 0 2022-01-31T19:44:08.571 INFO:tasks.workunit.client.0.smithi167.stdout:8/343: symlink d6/db/d4e/l6e 0 2022-01-31T19:44:08.578 INFO:tasks.workunit.client.0.smithi167.stdout:6/201: rename d8/d33/c34 to d8/d20/c4b 0 2022-01-31T19:44:08.578 INFO:tasks.workunit.client.0.smithi167.stdout:7/309: getdents d8/d2c/d6e/d4d/d67 0 2022-01-31T19:44:08.580 INFO:tasks.workunit.client.0.smithi167.stdout:9/166: dwrite dd/fe [0,4194304] 0 2022-01-31T19:44:08.581 INFO:tasks.workunit.client.0.smithi167.stdout:3/279: dwrite d9/f11 [0,4194304] 0 2022-01-31T19:44:08.582 INFO:tasks.workunit.client.0.smithi167.stdout:3/280: dread - d9/f42 zero size 2022-01-31T19:44:08.582 INFO:tasks.workunit.client.0.smithi167.stdout:0/121: dwrite d17/f22 [0,4194304] 0 2022-01-31T19:44:08.582 INFO:tasks.workunit.client.0.smithi167.stdout:0/122: fdatasync ff 0 2022-01-31T19:44:08.583 INFO:tasks.workunit.client.0.smithi167.stdout:8/344: rename d6/db/f5b to d6/db/de/d3d/f6f 0 2022-01-31T19:44:08.585 INFO:tasks.workunit.client.0.smithi167.stdout:4/245: rmdir d0/d6/d7/d39/d3a 39 2022-01-31T19:44:08.588 INFO:tasks.workunit.client.0.smithi167.stdout:9/167: truncate dd/f10 623623 0 2022-01-31T19:44:08.589 INFO:tasks.workunit.client.0.smithi167.stdout:3/281: dread d9/d26/f31 [0,4194304] 0 2022-01-31T19:44:08.591 INFO:tasks.workunit.client.0.smithi167.stdout:0/123: symlink d17/d19/d1c/l23 0 2022-01-31T19:44:08.597 INFO:tasks.workunit.client.0.smithi167.stdout:8/345: symlink d6/db/d28/l70 0 2022-01-31T19:44:08.597 INFO:tasks.workunit.client.0.smithi167.stdout:8/346: creat d6/db/de/d3d/f71 x:0 0 0 2022-01-31T19:44:08.598 INFO:tasks.workunit.client.0.smithi167.stdout:6/202: mkdir d8/d20/d4c 0 2022-01-31T19:44:08.599 INFO:tasks.workunit.client.0.smithi167.stdout:4/246: mknod d0/d3f/d1a/c4b 0 2022-01-31T19:44:08.600 INFO:tasks.workunit.client.0.smithi167.stdout:9/168: creat dd/d28/f33 x:0 0 0 2022-01-31T19:44:08.602 INFO:tasks.workunit.client.0.smithi167.stdout:3/282: mknod d9/d29/d44/c46 0 2022-01-31T19:44:08.603 INFO:tasks.workunit.client.0.smithi167.stdout:3/283: fdatasync d9/f40 0 2022-01-31T19:44:08.603 INFO:tasks.workunit.client.0.smithi167.stdout:0/124: symlink d17/d19/l24 0 2022-01-31T19:44:08.604 INFO:tasks.workunit.client.0.smithi167.stdout:7/310: rename c5 to d8/c78 0 2022-01-31T19:44:08.604 INFO:tasks.workunit.client.0.smithi167.stdout:7/311: read - d8/dc/d18/d24/f73 zero size 2022-01-31T19:44:08.605 INFO:tasks.workunit.client.0.smithi167.stdout:8/347: symlink d6/l72 0 2022-01-31T19:44:08.606 INFO:tasks.workunit.client.0.smithi167.stdout:6/203: creat d8/d33/d35/f4d x:0 0 0 2022-01-31T19:44:08.607 INFO:tasks.workunit.client.0.smithi167.stdout:4/247: getdents d0/d6/d7/d26/d40 0 2022-01-31T19:44:08.607 INFO:tasks.workunit.client.0.smithi167.stdout:9/169: mkdir dd/d34 0 2022-01-31T19:44:08.609 INFO:tasks.workunit.client.0.smithi167.stdout:3/284: mknod d9/d21/c47 0 2022-01-31T19:44:08.609 INFO:tasks.workunit.client.0.smithi167.stdout:3/285: stat d9/d29/f3c 0 2022-01-31T19:44:08.610 INFO:tasks.workunit.client.0.smithi167.stdout:0/125: symlink d17/l25 0 2022-01-31T19:44:08.611 INFO:tasks.workunit.client.0.smithi167.stdout:7/312: creat d8/d2c/d6c/f79 x:0 0 0 2022-01-31T19:44:08.612 INFO:tasks.workunit.client.0.smithi167.stdout:8/348: rmdir d6/d1e/d4c 39 2022-01-31T19:44:08.614 INFO:tasks.workunit.client.0.smithi167.stdout:6/204: rename d8/f14 to d8/d33/f4e 0 2022-01-31T19:44:08.615 INFO:tasks.workunit.client.0.smithi167.stdout:4/248: mkdir d0/d6/d7/d26/d40/d4c 0 2022-01-31T19:44:08.619 INFO:tasks.workunit.client.0.smithi167.stdout:3/286: symlink d9/d29/d2e/l48 0 2022-01-31T19:44:08.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/126: link fa d17/f26 0 2022-01-31T19:44:08.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/127: stat d17/d19/d1c/f21 0 2022-01-31T19:44:08.622 INFO:tasks.workunit.client.0.smithi167.stdout:0/128: fdatasync fa 0 2022-01-31T19:44:08.622 INFO:tasks.workunit.client.0.smithi167.stdout:0/129: write d17/d19/d1c/f21 [2885877,27304] 0 2022-01-31T19:44:08.623 INFO:tasks.workunit.client.0.smithi167.stdout:7/313: mkdir d8/d2c/d7a 0 2022-01-31T19:44:08.624 INFO:tasks.workunit.client.0.smithi167.stdout:8/349: mkdir d6/db/d4e/d73 0 2022-01-31T19:44:08.626 INFO:tasks.workunit.client.0.smithi167.stdout:6/205: mkdir d8/d33/d4f 0 2022-01-31T19:44:08.627 INFO:tasks.workunit.client.0.smithi167.stdout:6/206: write d8/d20/f21 [1145264,70898] 0 2022-01-31T19:44:08.628 INFO:tasks.workunit.client.0.smithi167.stdout:4/249: write d0/d6/d7/d26/d40/f32 [1185124,87684] 0 2022-01-31T19:44:08.628 INFO:tasks.workunit.client.0.smithi167.stdout:4/250: stat d0/d6/d7/f18 0 2022-01-31T19:44:08.629 INFO:tasks.workunit.client.0.smithi167.stdout:3/287: creat d9/d29/d44/f49 x:0 0 0 2022-01-31T19:44:08.632 INFO:tasks.workunit.client.0.smithi167.stdout:1/273: sync 2022-01-31T19:44:08.633 INFO:tasks.workunit.client.0.smithi167.stdout:0/130: rename d17/d19/c1e to d17/d19/d1c/c27 0 2022-01-31T19:44:08.633 INFO:tasks.workunit.client.0.smithi167.stdout:7/314: symlink d8/d2c/d43/d31/d4f/l7b 0 2022-01-31T19:44:08.634 INFO:tasks.workunit.client.0.smithi167.stdout:6/207: truncate d8/f1a 1112488 0 2022-01-31T19:44:08.636 INFO:tasks.workunit.client.0.smithi167.stdout:1/274: dread d0/d10/d18/f1e [0,4194304] 0 2022-01-31T19:44:08.637 INFO:tasks.workunit.client.0.smithi167.stdout:4/251: rmdir d0/d6/d7/d26/d47 0 2022-01-31T19:44:08.638 INFO:tasks.workunit.client.0.smithi167.stdout:9/170: rename dd/d28 to dd/d35 0 2022-01-31T19:44:08.642 INFO:tasks.workunit.client.0.smithi167.stdout:0/131: symlink d17/d19/l28 0 2022-01-31T19:44:08.643 INFO:tasks.workunit.client.0.smithi167.stdout:0/132: write f15 [2395297,101371] 0 2022-01-31T19:44:08.643 INFO:tasks.workunit.client.0.smithi167.stdout:6/208: unlink d8/f16 0 2022-01-31T19:44:08.643 INFO:tasks.workunit.client.0.smithi167.stdout:1/275: symlink d0/d10/l69 0 2022-01-31T19:44:08.643 INFO:tasks.workunit.client.0.smithi167.stdout:7/315: dread d8/d2c/d6e/f62 [0,4194304] 0 2022-01-31T19:44:08.644 INFO:tasks.workunit.client.0.smithi167.stdout:4/252: mknod d0/d6/d7/d26/c4d 0 2022-01-31T19:44:08.645 INFO:tasks.workunit.client.0.smithi167.stdout:9/171: creat dd/d17/f36 x:0 0 0 2022-01-31T19:44:08.646 INFO:tasks.workunit.client.0.smithi167.stdout:7/316: dread d8/dc/f2b [0,4194304] 0 2022-01-31T19:44:08.649 INFO:tasks.workunit.client.0.smithi167.stdout:0/133: mknod d17/d19/c29 0 2022-01-31T19:44:08.651 INFO:tasks.workunit.client.0.smithi167.stdout:6/209: rename d8/d33/f49 to d8/d33/d4f/f50 0 2022-01-31T19:44:08.651 INFO:tasks.workunit.client.0.smithi167.stdout:8/350: dwrite d6/f8 [4194304,4194304] 0 2022-01-31T19:44:08.651 INFO:tasks.workunit.client.0.smithi167.stdout:0/134: write ff [1670602,3879] 0 2022-01-31T19:44:08.652 INFO:tasks.workunit.client.0.smithi167.stdout:0/135: read ff [1114822,89506] 0 2022-01-31T19:44:08.652 INFO:tasks.workunit.client.0.smithi167.stdout:9/172: link dd/f1a dd/d35/f37 0 2022-01-31T19:44:08.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/317: symlink d8/d2c/d6c/l7c 0 2022-01-31T19:44:08.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/318: write d8/d2c/d43/d31/f5b [894317,128113] 0 2022-01-31T19:44:08.659 INFO:tasks.workunit.client.0.smithi167.stdout:8/351: mknod d6/db/d28/d66/c74 0 2022-01-31T19:44:08.661 INFO:tasks.workunit.client.0.smithi167.stdout:7/319: mknod d8/d35/c7d 0 2022-01-31T19:44:08.662 INFO:tasks.workunit.client.0.smithi167.stdout:9/173: truncate dd/d17/f1f 131463 0 2022-01-31T19:44:08.663 INFO:tasks.workunit.client.0.smithi167.stdout:7/320: write f7 [690551,112009] 0 2022-01-31T19:44:08.667 INFO:tasks.workunit.client.0.smithi167.stdout:5/325: sync 2022-01-31T19:44:08.668 INFO:tasks.workunit.client.0.smithi167.stdout:5/326: truncate d5/d6/f1a 1109335 0 2022-01-31T19:44:08.668 INFO:tasks.workunit.client.0.smithi167.stdout:5/327: fdatasync d5/d6/d3a/f21 0 2022-01-31T19:44:08.670 INFO:tasks.workunit.client.0.smithi167.stdout:5/328: symlink d5/d6/d16/d18/l67 0 2022-01-31T19:44:08.670 INFO:tasks.workunit.client.0.smithi167.stdout:5/329: dread - d5/d6/de/f37 zero size 2022-01-31T19:44:08.672 INFO:tasks.workunit.client.0.smithi167.stdout:4/253: dwrite d0/d6/d7/d39/f37 [0,4194304] 0 2022-01-31T19:44:08.672 INFO:tasks.workunit.client.0.smithi167.stdout:0/136: dread f14 [0,4194304] 0 2022-01-31T19:44:08.672 INFO:tasks.workunit.client.0.smithi167.stdout:9/174: read dd/d17/f20 [1979618,84100] 0 2022-01-31T19:44:08.673 INFO:tasks.workunit.client.0.smithi167.stdout:9/175: readlink l2 0 2022-01-31T19:44:08.673 INFO:tasks.workunit.client.0.smithi167.stdout:6/210: rmdir d8 39 2022-01-31T19:44:08.676 INFO:tasks.workunit.client.0.smithi167.stdout:9/176: dread dd/f10 [0,4194304] 0 2022-01-31T19:44:08.678 INFO:tasks.workunit.client.0.smithi167.stdout:5/330: mknod d5/c68 0 2022-01-31T19:44:08.679 INFO:tasks.workunit.client.0.smithi167.stdout:8/352: dwrite d6/db/d36/f4a [0,4194304] 0 2022-01-31T19:44:08.683 INFO:tasks.workunit.client.0.smithi167.stdout:6/211: mkdir d8/d33/d51 0 2022-01-31T19:44:08.683 INFO:tasks.workunit.client.0.smithi167.stdout:6/212: write d8/d33/f40 [964772,88924] 0 2022-01-31T19:44:08.684 INFO:tasks.workunit.client.0.smithi167.stdout:5/331: rmdir d5/d6/de/d22 39 2022-01-31T19:44:08.685 INFO:tasks.workunit.client.0.smithi167.stdout:5/332: chown d5/d6/f28 59 1 2022-01-31T19:44:08.686 INFO:tasks.workunit.client.0.smithi167.stdout:0/137: rmdir d17 39 2022-01-31T19:44:08.687 INFO:tasks.workunit.client.0.smithi167.stdout:0/138: dread - d17/d19/f1f zero size 2022-01-31T19:44:08.688 INFO:tasks.workunit.client.0.smithi167.stdout:5/333: mkdir d5/d6/d16/d18/d43/d45/d69 0 2022-01-31T19:44:08.689 INFO:tasks.workunit.client.0.smithi167.stdout:6/213: mknod d8/d20/c52 0 2022-01-31T19:44:08.689 INFO:tasks.workunit.client.0.smithi167.stdout:6/214: write d8/f9 [1133175,90122] 0 2022-01-31T19:44:08.690 INFO:tasks.workunit.client.0.smithi167.stdout:5/334: mknod d5/d6/c6a 0 2022-01-31T19:44:08.690 INFO:tasks.workunit.client.0.smithi167.stdout:5/335: chown d5/d6/d16/d18/f64 16081276 1 2022-01-31T19:44:08.692 INFO:tasks.workunit.client.0.smithi167.stdout:5/336: getdents d5/d6/d16/d18/d5b 0 2022-01-31T19:44:08.693 INFO:tasks.workunit.client.0.smithi167.stdout:5/337: mknod d5/d6/d16/d18/d43/d45/c6b 0 2022-01-31T19:44:08.693 INFO:tasks.workunit.client.0.smithi167.stdout:4/254: dwrite d0/d3f/f2e [0,4194304] 0 2022-01-31T19:44:08.694 INFO:tasks.workunit.client.0.smithi167.stdout:5/338: link d5/d6/de/d22/f25 d5/d6/d16/d18/d5b/f6c 0 2022-01-31T19:44:08.696 INFO:tasks.workunit.client.0.smithi167.stdout:5/339: mkdir d5/d6/de/d22/d6d 0 2022-01-31T19:44:08.697 INFO:tasks.workunit.client.0.smithi167.stdout:4/255: mkdir d0/d6/d7/d39/d4e 0 2022-01-31T19:44:08.698 INFO:tasks.workunit.client.0.smithi167.stdout:4/256: write d0/d6/d7/d39/f43 [57753,101153] 0 2022-01-31T19:44:08.700 INFO:tasks.workunit.client.0.smithi167.stdout:4/257: unlink d0/d6/d7/d26/c38 0 2022-01-31T19:44:08.704 INFO:tasks.workunit.client.0.smithi167.stdout:4/258: rename d0/d6/d7/d26/d31/f17 to d0/d6/d7/d26/d40/d4c/f4f 0 2022-01-31T19:44:08.705 INFO:tasks.workunit.client.0.smithi167.stdout:6/215: dread d8/f15 [0,4194304] 0 2022-01-31T19:44:08.706 INFO:tasks.workunit.client.0.smithi167.stdout:4/259: mkdir d0/d6/d7/d26/d50 0 2022-01-31T19:44:08.707 INFO:tasks.workunit.client.0.smithi167.stdout:4/260: chown d0/d3f/d1a/c4b 950015 1 2022-01-31T19:44:08.707 INFO:tasks.workunit.client.0.smithi167.stdout:6/216: getdents d8/d33/d35 0 2022-01-31T19:44:08.709 INFO:tasks.workunit.client.0.smithi167.stdout:6/217: creat d8/d33/d4f/f53 x:0 0 0 2022-01-31T19:44:08.710 INFO:tasks.workunit.client.0.smithi167.stdout:6/218: chown d8/d20/f44 11367 1 2022-01-31T19:44:08.711 INFO:tasks.workunit.client.0.smithi167.stdout:6/219: rename d8/d20/d4c to d8/d20/d54 0 2022-01-31T19:44:08.713 INFO:tasks.workunit.client.0.smithi167.stdout:6/220: rename d8/d33/d4f/f50 to d8/d20/f55 0 2022-01-31T19:44:08.726 INFO:tasks.workunit.client.0.smithi167.stdout:4/261: dread d0/d3f/f3d [0,4194304] 0 2022-01-31T19:44:08.728 INFO:tasks.workunit.client.0.smithi167.stdout:4/262: mkdir d0/d51 0 2022-01-31T19:44:08.728 INFO:tasks.workunit.client.0.smithi167.stdout:4/263: chown d0/d6 0 1 2022-01-31T19:44:08.728 INFO:tasks.workunit.client.0.smithi167.stdout:4/264: truncate d0/d3f/f29 928257 0 2022-01-31T19:44:08.729 INFO:tasks.workunit.client.0.smithi167.stdout:8/353: dwrite d6/db/d36/f3b [0,4194304] 0 2022-01-31T19:44:08.730 INFO:tasks.workunit.client.0.smithi167.stdout:8/354: mknod d6/db/d6a/c75 0 2022-01-31T19:44:08.730 INFO:tasks.workunit.client.0.smithi167.stdout:8/355: fsync d6/db/d4e/f4f 0 2022-01-31T19:44:08.730 INFO:tasks.workunit.client.0.smithi167.stdout:8/356: stat d6/l16 0 2022-01-31T19:44:08.730 INFO:tasks.workunit.client.0.smithi167.stdout:8/357: creat d6/db/d28/f76 x:0 0 0 2022-01-31T19:44:08.731 INFO:tasks.workunit.client.0.smithi167.stdout:9/177: dwrite dd/f1a [0,4194304] 0 2022-01-31T19:44:08.732 INFO:tasks.workunit.client.0.smithi167.stdout:9/178: creat dd/d17/f38 x:0 0 0 2022-01-31T19:44:08.736 INFO:tasks.workunit.client.0.smithi167.stdout:5/340: dwrite d5/d6/d16/d18/d5b/f6c [0,4194304] 0 2022-01-31T19:44:08.738 INFO:tasks.workunit.client.0.smithi167.stdout:6/221: dwrite d8/d33/f3c [0,4194304] 0 2022-01-31T19:44:08.739 INFO:tasks.workunit.client.0.smithi167.stdout:6/222: chown d8/d33/f4e 154758 1 2022-01-31T19:44:08.740 INFO:tasks.workunit.client.0.smithi167.stdout:0/139: dwrite f8 [0,4194304] 0 2022-01-31T19:44:08.740 INFO:tasks.workunit.client.0.smithi167.stdout:8/358: dread d6/ff [0,4194304] 0 2022-01-31T19:44:08.741 INFO:tasks.workunit.client.0.smithi167.stdout:6/223: symlink d8/d33/d51/l56 0 2022-01-31T19:44:08.742 INFO:tasks.workunit.client.0.smithi167.stdout:8/359: link d6/db/de/d3d/f56 d6/db/d36/d5d/f77 0 2022-01-31T19:44:08.744 INFO:tasks.workunit.client.0.smithi167.stdout:4/265: dwrite d0/d6/d7/d26/d31/f19 [0,4194304] 0 2022-01-31T19:44:08.748 INFO:tasks.workunit.client.0.smithi167.stdout:5/341: dread d5/d6/de/d22/f54 [0,4194304] 0 2022-01-31T19:44:08.749 INFO:tasks.workunit.client.0.smithi167.stdout:5/342: read d5/d6/de/f3e [110338,80570] 0 2022-01-31T19:44:08.751 INFO:tasks.workunit.client.0.smithi167.stdout:9/179: getdents dd/d17 0 2022-01-31T19:44:08.752 INFO:tasks.workunit.client.0.smithi167.stdout:9/180: dread - dd/d17/f38 zero size 2022-01-31T19:44:08.752 INFO:tasks.workunit.client.0.smithi167.stdout:9/181: truncate dd/d19/d24/f32 472948 0 2022-01-31T19:44:08.756 INFO:tasks.workunit.client.0.smithi167.stdout:6/224: rename d8/f19 to d8/d20/d54/f57 0 2022-01-31T19:44:08.758 INFO:tasks.workunit.client.0.smithi167.stdout:8/360: mknod d6/db/d4e/d73/c78 0 2022-01-31T19:44:08.759 INFO:tasks.workunit.client.0.smithi167.stdout:8/361: chown d6/db/d28 27603 1 2022-01-31T19:44:08.759 INFO:tasks.workunit.client.0.smithi167.stdout:8/362: readlink d6/db/de/d18/l64 0 2022-01-31T19:44:08.759 INFO:tasks.workunit.client.0.smithi167.stdout:8/363: readlink d6/l16 0 2022-01-31T19:44:08.759 INFO:tasks.workunit.client.0.smithi167.stdout:8/364: write d6/fa [2142190,52579] 0 2022-01-31T19:44:08.760 INFO:tasks.workunit.client.0.smithi167.stdout:4/266: creat d0/d6/d7/d39/d4e/f52 x:0 0 0 2022-01-31T19:44:08.762 INFO:tasks.workunit.client.0.smithi167.stdout:6/225: mknod d8/d33/d35/c58 0 2022-01-31T19:44:08.762 INFO:tasks.workunit.client.0.smithi167.stdout:6/226: creat d8/d33/d4f/f59 x:0 0 0 2022-01-31T19:44:08.763 INFO:tasks.workunit.client.0.smithi167.stdout:5/343: unlink d5/d6/d16/d18/d43/f5c 0 2022-01-31T19:44:08.766 INFO:tasks.workunit.client.0.smithi167.stdout:6/227: dread d8/d33/f40 [0,4194304] 0 2022-01-31T19:44:08.768 INFO:tasks.workunit.client.0.smithi167.stdout:2/221: sync 2022-01-31T19:44:08.769 INFO:tasks.workunit.client.0.smithi167.stdout:2/222: readlink dd/df/l38 0 2022-01-31T19:44:08.769 INFO:tasks.workunit.client.0.smithi167.stdout:2/223: fdatasync fa 0 2022-01-31T19:44:08.769 INFO:tasks.workunit.client.0.smithi167.stdout:8/365: stat d6/db/de/d18/c27 0 2022-01-31T19:44:08.769 INFO:tasks.workunit.client.0.smithi167.stdout:8/366: creat d6/db/d2f/d51/d60/f79 x:0 0 0 2022-01-31T19:44:08.770 INFO:tasks.workunit.client.0.smithi167.stdout:5/344: symlink d5/d6/de/d22/d4d/l6e 0 2022-01-31T19:44:08.770 INFO:tasks.workunit.client.0.smithi167.stdout:5/345: dread - d5/d6/d3a/f48 zero size 2022-01-31T19:44:08.771 INFO:tasks.workunit.client.0.smithi167.stdout:6/228: symlink d8/d33/d4f/l5a 0 2022-01-31T19:44:08.772 INFO:tasks.workunit.client.0.smithi167.stdout:2/224: rename dd/df/d22/c2b to dd/df/d1b/c45 0 2022-01-31T19:44:08.773 INFO:tasks.workunit.client.0.smithi167.stdout:8/367: mknod d6/db/d28/d61/c7a 0 2022-01-31T19:44:08.774 INFO:tasks.workunit.client.0.smithi167.stdout:5/346: creat d5/d6/de/f6f x:0 0 0 2022-01-31T19:44:08.776 INFO:tasks.workunit.client.0.smithi167.stdout:6/229: unlink d8/f31 0 2022-01-31T19:44:08.777 INFO:tasks.workunit.client.0.smithi167.stdout:8/368: creat d6/db/d4e/d73/f7b x:0 0 0 2022-01-31T19:44:08.778 INFO:tasks.workunit.client.0.smithi167.stdout:5/347: unlink d5/d6/de/f6f 0 2022-01-31T19:44:08.780 INFO:tasks.workunit.client.0.smithi167.stdout:8/369: read d6/db/d36/f4a [1160454,89636] 0 2022-01-31T19:44:08.780 INFO:tasks.workunit.client.0.smithi167.stdout:8/370: write d6/db/d4e/d73/f7b [183683,40151] 0 2022-01-31T19:44:08.780 INFO:tasks.workunit.client.0.smithi167.stdout:6/230: creat d8/f5b x:0 0 0 2022-01-31T19:44:08.780 INFO:tasks.workunit.client.0.smithi167.stdout:6/231: write d8/fd [4691269,57075] 0 2022-01-31T19:44:08.782 INFO:tasks.workunit.client.0.smithi167.stdout:8/371: rename d6/d1e/f69 to d6/db/d6a/f7c 0 2022-01-31T19:44:08.782 INFO:tasks.workunit.client.0.smithi167.stdout:8/372: chown d6/d1e/f29 3145121 1 2022-01-31T19:44:08.783 INFO:tasks.workunit.client.0.smithi167.stdout:9/182: dwrite dd/d19/d24/f32 [0,4194304] 0 2022-01-31T19:44:08.784 INFO:tasks.workunit.client.0.smithi167.stdout:9/183: dread - dd/d19/d24/f31 zero size 2022-01-31T19:44:08.784 INFO:tasks.workunit.client.0.smithi167.stdout:8/373: mkdir d6/db/d36/d5d/d7d 0 2022-01-31T19:44:08.789 INFO:tasks.workunit.client.0.smithi167.stdout:8/374: symlink d6/db/d2f/d51/l7e 0 2022-01-31T19:44:08.790 INFO:tasks.workunit.client.0.smithi167.stdout:9/184: symlink dd/d34/l39 0 2022-01-31T19:44:08.791 INFO:tasks.workunit.client.0.smithi167.stdout:8/375: symlink d6/db/d2f/d51/d60/l7f 0 2022-01-31T19:44:08.792 INFO:tasks.workunit.client.0.smithi167.stdout:8/376: write d6/db/d2f/d51/d60/f6c [174984,16951] 0 2022-01-31T19:44:08.792 INFO:tasks.workunit.client.0.smithi167.stdout:8/377: read d6/d1e/f29 [180381,67318] 0 2022-01-31T19:44:08.793 INFO:tasks.workunit.client.0.smithi167.stdout:8/378: truncate d6/f32 1815268 0 2022-01-31T19:44:08.793 INFO:tasks.workunit.client.0.smithi167.stdout:8/379: truncate d6/db/de/d18/f55 456350 0 2022-01-31T19:44:08.794 INFO:tasks.workunit.client.0.smithi167.stdout:8/380: truncate d6/db/d36/f3b 3313028 0 2022-01-31T19:44:08.798 INFO:tasks.workunit.client.0.smithi167.stdout:4/267: dwrite d0/d6/d7/d26/d31/f19 [0,4194304] 0 2022-01-31T19:44:08.799 INFO:tasks.workunit.client.0.smithi167.stdout:4/268: creat d0/d6/d7/d39/f53 x:0 0 0 2022-01-31T19:44:08.799 INFO:tasks.workunit.client.0.smithi167.stdout:4/269: read d0/d6/d7/f30 [286827,33660] 0 2022-01-31T19:44:08.800 INFO:tasks.workunit.client.0.smithi167.stdout:4/270: creat d0/d6/d7/d39/d3a/f54 x:0 0 0 2022-01-31T19:44:08.801 INFO:tasks.workunit.client.0.smithi167.stdout:4/271: creat d0/d6/d7/d39/d3a/f55 x:0 0 0 2022-01-31T19:44:08.802 INFO:tasks.workunit.client.0.smithi167.stdout:2/225: dwrite dd/f23 [0,4194304] 0 2022-01-31T19:44:08.802 INFO:tasks.workunit.client.0.smithi167.stdout:2/226: chown dd/df/d22/f37 0 1 2022-01-31T19:44:08.803 INFO:tasks.workunit.client.0.smithi167.stdout:4/272: creat d0/d3f/d1a/f56 x:0 0 0 2022-01-31T19:44:08.804 INFO:tasks.workunit.client.0.smithi167.stdout:4/273: truncate d0/d3f/f3d 415739 0 2022-01-31T19:44:08.805 INFO:tasks.workunit.client.0.smithi167.stdout:4/274: stat d0/ca 0 2022-01-31T19:44:08.805 INFO:tasks.workunit.client.0.smithi167.stdout:2/227: rmdir dd/d18 39 2022-01-31T19:44:08.806 INFO:tasks.workunit.client.0.smithi167.stdout:4/275: mknod d0/d6/d7/d26/d40/d4c/c57 0 2022-01-31T19:44:08.811 INFO:tasks.workunit.client.0.smithi167.stdout:4/276: fdatasync d0/d6/d7/d26/d40/f27 0 2022-01-31T19:44:08.811 INFO:tasks.workunit.client.0.smithi167.stdout:4/277: chown d0/d6/d7/d26/d31/f19 2371940 1 2022-01-31T19:44:08.818 INFO:tasks.workunit.client.0.smithi167.stdout:6/232: dwrite d8/d20/f43 [0,4194304] 0 2022-01-31T19:44:08.819 INFO:tasks.workunit.client.0.smithi167.stdout:5/348: dwrite d5/d6/f28 [0,4194304] 0 2022-01-31T19:44:08.823 INFO:tasks.workunit.client.0.smithi167.stdout:5/349: mknod d5/d6/c70 0 2022-01-31T19:44:08.827 INFO:tasks.workunit.client.0.smithi167.stdout:9/185: dwrite dd/ff [0,4194304] 0 2022-01-31T19:44:08.827 INFO:tasks.workunit.client.0.smithi167.stdout:9/186: truncate dd/d19/d24/f2c 470167 0 2022-01-31T19:44:08.827 INFO:tasks.workunit.client.0.smithi167.stdout:9/187: fdatasync dd/f1a 0 2022-01-31T19:44:08.827 INFO:tasks.workunit.client.0.smithi167.stdout:6/233: rename d8/d20/f43 to d8/f5c 0 2022-01-31T19:44:08.828 INFO:tasks.workunit.client.0.smithi167.stdout:8/381: dwrite d6/ff [4194304,4194304] 0 2022-01-31T19:44:08.828 INFO:tasks.workunit.client.0.smithi167.stdout:2/228: dwrite f9 [0,4194304] 0 2022-01-31T19:44:08.828 INFO:tasks.workunit.client.0.smithi167.stdout:9/188: read dd/d17/f2f [1126024,51985] 0 2022-01-31T19:44:08.829 INFO:tasks.workunit.client.0.smithi167.stdout:5/350: link l0 d5/d6/de/d22/l71 0 2022-01-31T19:44:08.831 INFO:tasks.workunit.client.0.smithi167.stdout:6/234: mkdir d8/d33/d51/d5d 0 2022-01-31T19:44:08.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/382: mkdir d6/d1e/d4c/d80 0 2022-01-31T19:44:08.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/383: chown d6/d1e/d2d/l5f 24854918 1 2022-01-31T19:44:08.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/384: chown d6/db/d36/c39 368 1 2022-01-31T19:44:08.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/385: fdatasync d6/db/d36/d5d/f77 0 2022-01-31T19:44:08.837 INFO:tasks.workunit.client.0.smithi167.stdout:9/189: creat dd/d35/f3a x:0 0 0 2022-01-31T19:44:08.840 INFO:tasks.workunit.client.0.smithi167.stdout:4/278: write d0/d3f/f2e [1078673,128419] 0 2022-01-31T19:44:08.843 INFO:tasks.workunit.client.0.smithi167.stdout:5/351: getdents d5/d61 0 2022-01-31T19:44:08.847 INFO:tasks.workunit.client.0.smithi167.stdout:5/352: dread d5/d6/d3a/f21 [0,4194304] 0 2022-01-31T19:44:08.848 INFO:tasks.workunit.client.0.smithi167.stdout:5/353: chown d5/d6/d3a/f4e 0 1 2022-01-31T19:44:08.848 INFO:tasks.workunit.client.0.smithi167.stdout:8/386: creat d6/db/d2f/f81 x:0 0 0 2022-01-31T19:44:08.848 INFO:tasks.workunit.client.0.smithi167.stdout:8/387: fdatasync d6/f8 0 2022-01-31T19:44:08.849 INFO:tasks.workunit.client.0.smithi167.stdout:8/388: write d6/db/de/f3e [2537362,13274] 0 2022-01-31T19:44:08.849 INFO:tasks.workunit.client.0.smithi167.stdout:8/389: write d6/db/de/d3d/f71 [299893,120059] 0 2022-01-31T19:44:08.850 INFO:tasks.workunit.client.0.smithi167.stdout:4/279: unlink d0/d3f/d1a/f56 0 2022-01-31T19:44:08.850 INFO:tasks.workunit.client.0.smithi167.stdout:4/280: creat d0/d6/f58 x:0 0 0 2022-01-31T19:44:08.850 INFO:tasks.workunit.client.0.smithi167.stdout:5/354: symlink d5/d6/de/d22/d4d/d65/l72 0 2022-01-31T19:44:08.851 INFO:tasks.workunit.client.0.smithi167.stdout:4/281: fsync d0/d6/d7/d26/d40/f24 0 2022-01-31T19:44:08.853 INFO:tasks.workunit.client.0.smithi167.stdout:8/390: read d6/db/de/d18/f1b [1483516,58773] 0 2022-01-31T19:44:08.853 INFO:tasks.workunit.client.0.smithi167.stdout:5/355: link d5/d6/d16/l29 d5/d6/de/d22/d6d/l73 0 2022-01-31T19:44:08.854 INFO:tasks.workunit.client.0.smithi167.stdout:2/229: dwrite dd/d18/f30 [0,4194304] 0 2022-01-31T19:44:08.855 INFO:tasks.workunit.client.0.smithi167.stdout:4/282: unlink d0/d6/d7/d39/f53 0 2022-01-31T19:44:08.856 INFO:tasks.workunit.client.0.smithi167.stdout:8/391: mknod d6/db/d4e/c82 0 2022-01-31T19:44:08.860 INFO:tasks.workunit.client.0.smithi167.stdout:4/283: dread d0/d6/d7/d26/d40/f32 [0,4194304] 0 2022-01-31T19:44:08.860 INFO:tasks.workunit.client.0.smithi167.stdout:6/235: dwrite d8/d20/d54/f57 [0,4194304] 0 2022-01-31T19:44:08.862 INFO:tasks.workunit.client.0.smithi167.stdout:4/284: symlink d0/d6/d7/d26/d31/l59 0 2022-01-31T19:44:08.863 INFO:tasks.workunit.client.0.smithi167.stdout:6/236: mknod d8/d33/c5e 0 2022-01-31T19:44:08.865 INFO:tasks.workunit.client.0.smithi167.stdout:9/190: dwrite dd/fe [4194304,4194304] 0 2022-01-31T19:44:08.866 INFO:tasks.workunit.client.0.smithi167.stdout:2/230: dread f8 [0,4194304] 0 2022-01-31T19:44:08.866 INFO:tasks.workunit.client.0.smithi167.stdout:2/231: read - dd/d18/f31 zero size 2022-01-31T19:44:08.866 INFO:tasks.workunit.client.0.smithi167.stdout:2/232: fsync dd/d18/f3a 0 2022-01-31T19:44:08.867 INFO:tasks.workunit.client.0.smithi167.stdout:6/237: creat d8/d33/f5f x:0 0 0 2022-01-31T19:44:08.867 INFO:tasks.workunit.client.0.smithi167.stdout:4/285: dread d0/d6/d7/d39/f2b [0,4194304] 0 2022-01-31T19:44:08.868 INFO:tasks.workunit.client.0.smithi167.stdout:4/286: unlink d0/d3f/d1a/c20 0 2022-01-31T19:44:08.868 INFO:tasks.workunit.client.0.smithi167.stdout:4/287: write d0/d6/d7/d39/f2b [626232,38679] 0 2022-01-31T19:44:08.870 INFO:tasks.workunit.client.0.smithi167.stdout:9/191: mkdir dd/d19/d24/d3b 0 2022-01-31T19:44:08.872 INFO:tasks.workunit.client.0.smithi167.stdout:9/192: read dd/d19/f21 [1332719,32090] 0 2022-01-31T19:44:08.873 INFO:tasks.workunit.client.0.smithi167.stdout:9/193: symlink dd/d34/l3c 0 2022-01-31T19:44:08.874 INFO:tasks.workunit.client.0.smithi167.stdout:9/194: rename dd/d17/f20 to dd/d19/f3d 0 2022-01-31T19:44:08.876 INFO:tasks.workunit.client.0.smithi167.stdout:2/233: dread dd/df/f1c [0,4194304] 0 2022-01-31T19:44:08.877 INFO:tasks.workunit.client.0.smithi167.stdout:9/195: dread dd/f10 [0,4194304] 0 2022-01-31T19:44:08.881 INFO:tasks.workunit.client.0.smithi167.stdout:5/356: dwrite d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:08.882 INFO:tasks.workunit.client.0.smithi167.stdout:8/392: dwrite d6/d1e/f3c [0,4194304] 0 2022-01-31T19:44:08.882 INFO:tasks.workunit.client.0.smithi167.stdout:8/393: fsync d6/db/d4e/d73/f7b 0 2022-01-31T19:44:08.883 INFO:tasks.workunit.client.0.smithi167.stdout:2/234: rename dd/df/d1b/d2f/d36/c2e to dd/df/d1b/d2f/d36/c46 0 2022-01-31T19:44:08.887 INFO:tasks.workunit.client.0.smithi167.stdout:5/357: creat d5/d61/f74 x:0 0 0 2022-01-31T19:44:08.895 INFO:tasks.workunit.client.0.smithi167.stdout:9/196: getdents dd/d19 0 2022-01-31T19:44:08.898 INFO:tasks.workunit.client.0.smithi167.stdout:5/358: getdents d5/d6/d16/d18 0 2022-01-31T19:44:08.899 INFO:tasks.workunit.client.0.smithi167.stdout:8/394: rmdir d6/db/d4e/d73 39 2022-01-31T19:44:08.900 INFO:tasks.workunit.client.0.smithi167.stdout:9/197: getdents dd 0 2022-01-31T19:44:08.901 INFO:tasks.workunit.client.0.smithi167.stdout:5/359: creat d5/d6/d3a/f75 x:0 0 0 2022-01-31T19:44:08.902 INFO:tasks.workunit.client.0.smithi167.stdout:9/198: link dd/d35/f3a dd/d17/f3e 0 2022-01-31T19:44:08.903 INFO:tasks.workunit.client.0.smithi167.stdout:2/235: dwrite f7 [0,4194304] 0 2022-01-31T19:44:08.903 INFO:tasks.workunit.client.0.smithi167.stdout:5/360: truncate d5/d6/de/d22/f25 2246046 0 2022-01-31T19:44:08.905 INFO:tasks.workunit.client.0.smithi167.stdout:9/199: mknod dd/d34/c3f 0 2022-01-31T19:44:08.907 INFO:tasks.workunit.client.0.smithi167.stdout:2/236: rename dd/f1a to dd/df/d1b/d3d/f47 0 2022-01-31T19:44:08.908 INFO:tasks.workunit.client.0.smithi167.stdout:2/237: write dd/d18/f31 [100585,16147] 0 2022-01-31T19:44:08.908 INFO:tasks.workunit.client.0.smithi167.stdout:5/361: mkdir d5/d6/de/d22/d76 0 2022-01-31T19:44:08.909 INFO:tasks.workunit.client.0.smithi167.stdout:9/200: mknod dd/c40 0 2022-01-31T19:44:08.911 INFO:tasks.workunit.client.0.smithi167.stdout:5/362: symlink d5/d6/de/d22/l77 0 2022-01-31T19:44:08.915 INFO:tasks.workunit.client.0.smithi167.stdout:9/201: rename dd/d34/l3c to dd/d19/l41 0 2022-01-31T19:44:08.917 INFO:tasks.workunit.client.0.smithi167.stdout:5/363: unlink d5/d6/d16/f3b 0 2022-01-31T19:44:08.918 INFO:tasks.workunit.client.0.smithi167.stdout:5/364: rmdir d5/d6/d3a 39 2022-01-31T19:44:08.919 INFO:tasks.workunit.client.0.smithi167.stdout:5/365: stat d5/d6/d3a/l11 0 2022-01-31T19:44:08.919 INFO:tasks.workunit.client.0.smithi167.stdout:5/366: fsync d5/d6/d16/d18/f64 0 2022-01-31T19:44:08.920 INFO:tasks.workunit.client.0.smithi167.stdout:5/367: rename d5/d6/d16/f38 to d5/d6/de/d22/d4d/f78 0 2022-01-31T19:44:08.920 INFO:tasks.workunit.client.0.smithi167.stdout:5/368: chown d5/f7 27 1 2022-01-31T19:44:08.921 INFO:tasks.workunit.client.0.smithi167.stdout:5/369: truncate d5/d6/de/f2a 855557 0 2022-01-31T19:44:08.921 INFO:tasks.workunit.client.0.smithi167.stdout:5/370: rmdir d5/d6/de/d22 39 2022-01-31T19:44:08.922 INFO:tasks.workunit.client.0.smithi167.stdout:5/371: write d5/d6/d3a/f75 [580881,110944] 0 2022-01-31T19:44:08.925 INFO:tasks.workunit.client.0.smithi167.stdout:2/238: dwrite dd/d18/f31 [0,4194304] 0 2022-01-31T19:44:08.925 INFO:tasks.workunit.client.0.smithi167.stdout:2/239: readlink dd/d18/l1f 0 2022-01-31T19:44:08.926 INFO:tasks.workunit.client.0.smithi167.stdout:5/372: fdatasync d5/d6/d3a/f75 0 2022-01-31T19:44:08.926 INFO:tasks.workunit.client.0.smithi167.stdout:5/373: write d5/d6/de/f4c [1191178,122105] 0 2022-01-31T19:44:08.926 INFO:tasks.workunit.client.0.smithi167.stdout:5/374: chown d5/d6/d3a/l11 778626858 1 2022-01-31T19:44:08.927 INFO:tasks.workunit.client.0.smithi167.stdout:5/375: chown d5/d6/de/f37 54 1 2022-01-31T19:44:08.927 INFO:tasks.workunit.client.0.smithi167.stdout:5/376: dread - d5/d6/d16/d18/d43/d45/f60 zero size 2022-01-31T19:44:08.932 INFO:tasks.workunit.client.0.smithi167.stdout:5/377: read d5/d6/d16/d18/f31 [1554033,125659] 0 2022-01-31T19:44:08.932 INFO:tasks.workunit.client.0.smithi167.stdout:2/240: mknod dd/df/d1b/c48 0 2022-01-31T19:44:08.933 INFO:tasks.workunit.client.0.smithi167.stdout:2/241: symlink dd/d18/l49 0 2022-01-31T19:44:08.934 INFO:tasks.workunit.client.0.smithi167.stdout:2/242: mkdir dd/df/d1b/d3d/d4a 0 2022-01-31T19:44:08.934 INFO:tasks.workunit.client.0.smithi167.stdout:2/243: chown dd/c43 349 1 2022-01-31T19:44:08.937 INFO:tasks.workunit.client.0.smithi167.stdout:2/244: getdents dd/df/d3f 0 2022-01-31T19:44:08.938 INFO:tasks.workunit.client.0.smithi167.stdout:2/245: mkdir dd/df/d1b/d3d/d4a/d4b 0 2022-01-31T19:44:08.938 INFO:tasks.workunit.client.0.smithi167.stdout:2/246: readlink dd/df/d1b/l28 0 2022-01-31T19:44:08.939 INFO:tasks.workunit.client.0.smithi167.stdout:9/202: dwrite dd/f16 [0,4194304] 0 2022-01-31T19:44:08.939 INFO:tasks.workunit.client.0.smithi167.stdout:9/203: creat dd/d19/d24/f42 x:0 0 0 2022-01-31T19:44:08.939 INFO:tasks.workunit.client.0.smithi167.stdout:2/247: creat dd/d18/f4c x:0 0 0 2022-01-31T19:44:08.941 INFO:tasks.workunit.client.0.smithi167.stdout:9/204: creat dd/d19/d24/d3b/f43 x:0 0 0 2022-01-31T19:44:08.953 INFO:tasks.workunit.client.0.smithi167.stdout:9/205: dread dd/d17/f2f [0,4194304] 0 2022-01-31T19:44:08.954 INFO:tasks.workunit.client.0.smithi167.stdout:5/378: dwrite d5/d6/d16/d18/f58 [0,4194304] 0 2022-01-31T19:44:08.954 INFO:tasks.workunit.client.0.smithi167.stdout:9/206: creat dd/d35/f44 x:0 0 0 2022-01-31T19:44:08.954 INFO:tasks.workunit.client.0.smithi167.stdout:9/207: fsync dd/ff 0 2022-01-31T19:44:08.954 INFO:tasks.workunit.client.0.smithi167.stdout:9/208: creat dd/d19/d24/f45 x:0 0 0 2022-01-31T19:44:08.955 INFO:tasks.workunit.client.0.smithi167.stdout:9/209: read dd/ff [3435729,95321] 0 2022-01-31T19:44:08.959 INFO:tasks.workunit.client.0.smithi167.stdout:9/210: rename dd/d19/d24/f32 to dd/d19/d24/d3b/f46 0 2022-01-31T19:44:08.959 INFO:tasks.workunit.client.0.smithi167.stdout:9/211: chown dd/d17/f38 0 1 2022-01-31T19:44:08.963 INFO:tasks.workunit.client.0.smithi167.stdout:9/212: mknod dd/d17/c47 0 2022-01-31T19:44:08.963 INFO:tasks.workunit.client.0.smithi167.stdout:9/213: chown dd/fe 141256 1 2022-01-31T19:44:08.970 INFO:tasks.workunit.client.0.smithi167.stdout:5/379: dwrite d5/d6/d3a/f21 [0,4194304] 0 2022-01-31T19:44:08.971 INFO:tasks.workunit.client.0.smithi167.stdout:9/214: getdents dd/d17 0 2022-01-31T19:44:08.973 INFO:tasks.workunit.client.0.smithi167.stdout:5/380: truncate d5/d6/d16/d18/f3d 2585260 0 2022-01-31T19:44:08.974 INFO:tasks.workunit.client.0.smithi167.stdout:9/215: symlink dd/d19/d24/d3b/l48 0 2022-01-31T19:44:08.974 INFO:tasks.workunit.client.0.smithi167.stdout:9/216: truncate dd/d17/f3e 759492 0 2022-01-31T19:44:08.975 INFO:tasks.workunit.client.0.smithi167.stdout:5/381: creat d5/d6/de/d22/d4d/d65/f79 x:0 0 0 2022-01-31T19:44:08.975 INFO:tasks.workunit.client.0.smithi167.stdout:5/382: write d5/d6/d16/d18/f2e [4571610,11846] 0 2022-01-31T19:44:08.976 INFO:tasks.workunit.client.0.smithi167.stdout:9/217: mknod dd/d19/d24/c49 0 2022-01-31T19:44:08.976 INFO:tasks.workunit.client.0.smithi167.stdout:5/383: symlink d5/d6/de/d22/d4d/l7a 0 2022-01-31T19:44:08.977 INFO:tasks.workunit.client.0.smithi167.stdout:9/218: getdents dd/d34 0 2022-01-31T19:44:08.979 INFO:tasks.workunit.client.0.smithi167.stdout:5/384: rename d5/d6/d3a/l66 to d5/d6/de/d22/d4d/l7b 0 2022-01-31T19:44:08.980 INFO:tasks.workunit.client.0.smithi167.stdout:5/385: stat d5/d6/d16/d18/d5b/d63 0 2022-01-31T19:44:08.980 INFO:tasks.workunit.client.0.smithi167.stdout:5/386: write d5/d6/d16/d18/d43/f57 [256219,44973] 0 2022-01-31T19:44:08.981 INFO:tasks.workunit.client.0.smithi167.stdout:9/219: dread dd/d19/f27 [0,4194304] 0 2022-01-31T19:44:08.988 INFO:tasks.workunit.client.0.smithi167.stdout:5/387: dread d5/d6/de/d22/d4d/f78 [0,4194304] 0 2022-01-31T19:44:08.988 INFO:tasks.workunit.client.0.smithi167.stdout:5/388: creat d5/d6/de/d22/d4d/d65/f7c x:0 0 0 2022-01-31T19:44:08.989 INFO:tasks.workunit.client.0.smithi167.stdout:5/389: write d5/f7 [5243643,125486] 0 2022-01-31T19:44:09.006 INFO:tasks.workunit.client.0.smithi167.stdout:5/390: dwrite d5/d6/d16/d18/d43/d45/f60 [0,4194304] 0 2022-01-31T19:44:09.009 INFO:tasks.workunit.client.0.smithi167.stdout:5/391: link d5/d6/d16/d18/f59 d5/d6/d16/d18/f7d 0 2022-01-31T19:44:09.011 INFO:tasks.workunit.client.0.smithi167.stdout:5/392: rename d5/d6/de/d22/d4d/f78 to d5/d6/d16/f7e 0 2022-01-31T19:44:09.011 INFO:tasks.workunit.client.0.smithi167.stdout:5/393: fsync d5/d6/d16/d18/f58 0 2022-01-31T19:44:09.012 INFO:tasks.workunit.client.0.smithi167.stdout:5/394: stat d5/d6/de/d22/d4d/d65/l72 0 2022-01-31T19:44:09.019 INFO:tasks.workunit.client.0.smithi167.stdout:5/395: dread d5/d61/f4b [0,4194304] 0 2022-01-31T19:44:09.033 INFO:tasks.workunit.client.0.smithi167.stdout:5/396: dwrite d5/d6/f62 [4194304,4194304] 0 2022-01-31T19:44:09.033 INFO:tasks.workunit.client.0.smithi167.stdout:5/397: creat d5/d61/f7f x:0 0 0 2022-01-31T19:44:09.049 INFO:tasks.workunit.client.0.smithi167.stdout:5/398: dwrite d5/d6/d16/d18/d43/f57 [0,4194304] 0 2022-01-31T19:44:09.050 INFO:tasks.workunit.client.0.smithi167.stdout:5/399: stat d5/d6/de/d22/d4d/l7a 0 2022-01-31T19:44:09.050 INFO:tasks.workunit.client.0.smithi167.stdout:5/400: chown d5/d6/c55 4674 1 2022-01-31T19:44:09.072 INFO:tasks.workunit.client.0.smithi167.stdout:5/401: dwrite d5/d6/f42 [0,4194304] 0 2022-01-31T19:44:09.075 INFO:tasks.workunit.client.0.smithi167.stdout:5/402: rename d5/d6/d16/d18/f2e to d5/d6/de/d22/d76/f80 0 2022-01-31T19:44:09.076 INFO:tasks.workunit.client.0.smithi167.stdout:5/403: mknod d5/d6/d3a/c81 0 2022-01-31T19:44:09.077 INFO:tasks.workunit.client.0.smithi167.stdout:5/404: rename d5/d6/f62 to d5/d6/d16/f82 0 2022-01-31T19:44:09.078 INFO:tasks.workunit.client.0.smithi167.stdout:5/405: write d5/d6/de/d22/f54 [487270,70681] 0 2022-01-31T19:44:09.079 INFO:tasks.workunit.client.0.smithi167.stdout:5/406: read - d5/d61/f36 zero size 2022-01-31T19:44:09.102 INFO:tasks.workunit.client.0.smithi167.stdout:5/407: dwrite d5/d6/de/f3e [0,4194304] 0 2022-01-31T19:44:09.104 INFO:tasks.workunit.client.0.smithi167.stdout:5/408: creat d5/d6/d16/d18/d43/d45/f83 x:0 0 0 2022-01-31T19:44:09.105 INFO:tasks.workunit.client.0.smithi167.stdout:5/409: dread - d5/d6/de/d22/f35 zero size 2022-01-31T19:44:09.105 INFO:tasks.workunit.client.0.smithi167.stdout:5/410: creat d5/d6/de/d22/d6d/f84 x:0 0 0 2022-01-31T19:44:09.107 INFO:tasks.workunit.client.0.smithi167.stdout:5/411: truncate d5/d6/d16/f26 3098275 0 2022-01-31T19:44:09.123 INFO:tasks.workunit.client.0.smithi167.stdout:5/412: dwrite d5/d6/f28 [0,4194304] 0 2022-01-31T19:44:09.125 INFO:tasks.workunit.client.0.smithi167.stdout:5/413: dread d5/d6/d16/d18/f2c [0,4194304] 0 2022-01-31T19:44:09.126 INFO:tasks.workunit.client.0.smithi167.stdout:5/414: write d5/d6/d16/d18/d5b/f6c [1802184,85863] 0 2022-01-31T19:44:09.128 INFO:tasks.workunit.client.0.smithi167.stdout:5/415: creat d5/d6/de/d22/d4d/f85 x:0 0 0 2022-01-31T19:44:09.128 INFO:tasks.workunit.client.0.smithi167.stdout:5/416: unlink d5/d6/d16/f82 0 2022-01-31T19:44:09.161 INFO:tasks.workunit.client.0.smithi167.stdout:5/417: dwrite d5/d6/d16/d18/f58 [0,4194304] 0 2022-01-31T19:44:09.162 INFO:tasks.workunit.client.0.smithi167.stdout:5/418: mknod d5/d6/de/d22/d4d/c86 0 2022-01-31T19:44:09.162 INFO:tasks.workunit.client.0.smithi167.stdout:5/419: readlink d5/d6/d16/d18/l67 0 2022-01-31T19:44:09.164 INFO:tasks.workunit.client.0.smithi167.stdout:5/420: getdents d5/d6/d16/d18/d43/d45 0 2022-01-31T19:44:09.164 INFO:tasks.workunit.client.0.smithi167.stdout:5/421: creat d5/d6/de/d22/d4d/f87 x:0 0 0 2022-01-31T19:44:09.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/422: dread d5/d6/d3a/f21 [0,4194304] 0 2022-01-31T19:44:09.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/423: unlink d5/d6/d16/f5a 0 2022-01-31T19:44:09.187 INFO:tasks.workunit.client.0.smithi167.stdout:5/424: dwrite d5/d6/d16/d18/f64 [4194304,4194304] 0 2022-01-31T19:44:09.187 INFO:tasks.workunit.client.0.smithi167.stdout:5/425: mkdir d5/d6/d16/d18/d5b/d63/d88 0 2022-01-31T19:44:09.188 INFO:tasks.workunit.client.0.smithi167.stdout:5/426: getdents d5/d6/d16/d18/d5b 0 2022-01-31T19:44:09.189 INFO:tasks.workunit.client.0.smithi167.stdout:5/427: dread d5/d6/f1a [0,4194304] 0 2022-01-31T19:44:09.190 INFO:tasks.workunit.client.0.smithi167.stdout:5/428: stat d5/d6/f28 0 2022-01-31T19:44:09.190 INFO:tasks.workunit.client.0.smithi167.stdout:5/429: mknod d5/d6/de/d22/c89 0 2022-01-31T19:44:09.203 INFO:tasks.workunit.client.0.smithi167.stdout:5/430: dwrite d5/d6/d16/d18/d43/f57 [4194304,4194304] 0 2022-01-31T19:44:09.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/431: readlink d5/d6/de/d22/d4d/l7a 0 2022-01-31T19:44:09.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/432: write d5/d6/de/d22/d4d/f85 [588820,62513] 0 2022-01-31T19:44:09.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/433: symlink d5/d6/d16/d18/d43/l8a 0 2022-01-31T19:44:09.206 INFO:tasks.workunit.client.0.smithi167.stdout:5/434: creat d5/f8b x:0 0 0 2022-01-31T19:44:09.207 INFO:tasks.workunit.client.0.smithi167.stdout:5/435: write d5/d6/d3a/f48 [324143,83986] 0 2022-01-31T19:44:09.212 INFO:tasks.workunit.client.0.smithi167.stdout:5/436: mknod d5/d6/d16/d18/d43/d45/c8c 0 2022-01-31T19:44:09.212 INFO:tasks.workunit.client.0.smithi167.stdout:5/437: readlink d5/d6/de/d22/d6d/l73 0 2022-01-31T19:44:09.213 INFO:tasks.workunit.client.0.smithi167.stdout:5/438: creat d5/d6/d16/d18/f8d x:0 0 0 2022-01-31T19:44:09.229 INFO:tasks.workunit.client.0.smithi167.stdout:5/439: dwrite d5/d6/de/d22/d4d/d65/f7c [0,4194304] 0 2022-01-31T19:44:09.232 INFO:tasks.workunit.client.0.smithi167.stdout:5/440: dread d5/d6/d16/f7e [0,4194304] 0 2022-01-31T19:44:09.233 INFO:tasks.workunit.client.0.smithi167.stdout:5/441: creat d5/d6/d3a/f8e x:0 0 0 2022-01-31T19:44:09.234 INFO:tasks.workunit.client.0.smithi167.stdout:5/442: creat d5/d6/d3a/f8f x:0 0 0 2022-01-31T19:44:09.243 INFO:tasks.workunit.client.0.smithi167.stdout:1/276: sync 2022-01-31T19:44:09.243 INFO:tasks.workunit.client.0.smithi167.stdout:7/321: sync 2022-01-31T19:44:09.243 INFO:tasks.workunit.client.0.smithi167.stdout:0/140: sync 2022-01-31T19:44:09.244 INFO:tasks.workunit.client.0.smithi167.stdout:3/288: sync 2022-01-31T19:44:09.244 INFO:tasks.workunit.client.0.smithi167.stdout:7/322: readlink d8/dc/l34 0 2022-01-31T19:44:09.244 INFO:tasks.workunit.client.0.smithi167.stdout:0/141: fsync ff 0 2022-01-31T19:44:09.244 INFO:tasks.workunit.client.0.smithi167.stdout:3/289: getdents d9/d26 0 2022-01-31T19:44:09.245 INFO:tasks.workunit.client.0.smithi167.stdout:0/142: rename c6 to d17/d19/c2a 0 2022-01-31T19:44:09.246 INFO:tasks.workunit.client.0.smithi167.stdout:1/277: dread d0/d10/d18/d39/f52 [0,4194304] 0 2022-01-31T19:44:09.246 INFO:tasks.workunit.client.0.smithi167.stdout:1/278: dread - d0/d10/d5a/d33/d4b/d26/f59 zero size 2022-01-31T19:44:09.246 INFO:tasks.workunit.client.0.smithi167.stdout:3/290: mkdir d9/d21/d4a 0 2022-01-31T19:44:09.246 INFO:tasks.workunit.client.0.smithi167.stdout:3/291: stat d9/d26/l38 0 2022-01-31T19:44:09.247 INFO:tasks.workunit.client.0.smithi167.stdout:1/279: symlink d0/d10/d5a/d33/d4b/d26/l6a 0 2022-01-31T19:44:09.248 INFO:tasks.workunit.client.0.smithi167.stdout:3/292: mknod d9/d29/d3b/d41/c4b 0 2022-01-31T19:44:09.250 INFO:tasks.workunit.client.0.smithi167.stdout:3/293: rmdir d9/d21/d4a 0 2022-01-31T19:44:09.251 INFO:tasks.workunit.client.0.smithi167.stdout:3/294: rmdir d9/d29/d3b/d41 39 2022-01-31T19:44:09.253 INFO:tasks.workunit.client.0.smithi167.stdout:3/295: rmdir d9 39 2022-01-31T19:44:09.253 INFO:tasks.workunit.client.0.smithi167.stdout:3/296: stat f7 0 2022-01-31T19:44:09.253 INFO:tasks.workunit.client.0.smithi167.stdout:3/297: chown d9/d29/d3b/d41 7061712 1 2022-01-31T19:44:09.254 INFO:tasks.workunit.client.0.smithi167.stdout:3/298: stat d9/d29/c37 0 2022-01-31T19:44:09.254 INFO:tasks.workunit.client.0.smithi167.stdout:7/323: dread d8/d2c/d43/d31/f6a [0,4194304] 0 2022-01-31T19:44:09.255 INFO:tasks.workunit.client.0.smithi167.stdout:3/299: creat d9/d29/f4c x:0 0 0 2022-01-31T19:44:09.256 INFO:tasks.workunit.client.0.smithi167.stdout:3/300: chown f7 183 1 2022-01-31T19:44:09.271 INFO:tasks.workunit.client.0.smithi167.stdout:7/324: dread d8/dc/d19/f5c [0,4194304] 0 2022-01-31T19:44:09.273 INFO:tasks.workunit.client.0.smithi167.stdout:3/301: dread d9/d21/f23 [0,4194304] 0 2022-01-31T19:44:09.273 INFO:tasks.workunit.client.0.smithi167.stdout:7/325: write d8/dc/f1c [316539,1527] 0 2022-01-31T19:44:09.273 INFO:tasks.workunit.client.0.smithi167.stdout:0/143: dwrite fc [0,4194304] 0 2022-01-31T19:44:09.274 INFO:tasks.workunit.client.0.smithi167.stdout:1/280: dwrite d0/f25 [0,4194304] 0 2022-01-31T19:44:09.276 INFO:tasks.workunit.client.0.smithi167.stdout:0/144: rename d17/d19/d1c/l1d to d17/d19/l2b 0 2022-01-31T19:44:09.278 INFO:tasks.workunit.client.0.smithi167.stdout:1/281: stat d0/d10/d5a/d33/d4b/c2a 0 2022-01-31T19:44:09.278 INFO:tasks.workunit.client.0.smithi167.stdout:1/282: chown d0/d10/d5a/d33/d4b/d26/l4a 28 1 2022-01-31T19:44:09.282 INFO:tasks.workunit.client.0.smithi167.stdout:3/302: dread d9/f1a [4194304,4194304] 0 2022-01-31T19:44:09.284 INFO:tasks.workunit.client.0.smithi167.stdout:3/303: mknod d9/d29/d3b/d41/c4d 0 2022-01-31T19:44:09.306 INFO:tasks.workunit.client.0.smithi167.stdout:7/326: dwrite d8/d2c/d6e/f2a [0,4194304] 0 2022-01-31T19:44:09.306 INFO:tasks.workunit.client.0.smithi167.stdout:7/327: chown d8/d2c/d43/d31/d4f 1678017 1 2022-01-31T19:44:09.308 INFO:tasks.workunit.client.0.smithi167.stdout:7/328: creat d8/d2c/d6e/d4d/d67/f7e x:0 0 0 2022-01-31T19:44:09.308 INFO:tasks.workunit.client.0.smithi167.stdout:0/145: dwrite fa [0,4194304] 0 2022-01-31T19:44:09.308 INFO:tasks.workunit.client.0.smithi167.stdout:1/283: dwrite d0/d10/f42 [0,4194304] 0 2022-01-31T19:44:09.308 INFO:tasks.workunit.client.0.smithi167.stdout:1/284: dread - d0/d10/d5a/d33/d4b/f5b zero size 2022-01-31T19:44:09.309 INFO:tasks.workunit.client.0.smithi167.stdout:1/285: stat d0/d10/d5a/d33 0 2022-01-31T19:44:09.310 INFO:tasks.workunit.client.0.smithi167.stdout:0/146: mkdir d17/d2c 0 2022-01-31T19:44:09.310 INFO:tasks.workunit.client.0.smithi167.stdout:0/147: chown fe 143 1 2022-01-31T19:44:09.311 INFO:tasks.workunit.client.0.smithi167.stdout:3/304: dwrite d9/fc [0,4194304] 0 2022-01-31T19:44:09.311 INFO:tasks.workunit.client.0.smithi167.stdout:3/305: fsync d9/f11 0 2022-01-31T19:44:09.312 INFO:tasks.workunit.client.0.smithi167.stdout:1/286: link d0/d10/d5a/d33/d4b/l60 d0/d10/d5a/d33/d4b/l6b 0 2022-01-31T19:44:09.312 INFO:tasks.workunit.client.0.smithi167.stdout:1/287: read - d0/d10/d20/f4f zero size 2022-01-31T19:44:09.314 INFO:tasks.workunit.client.0.smithi167.stdout:1/288: mkdir d0/d6c 0 2022-01-31T19:44:09.324 INFO:tasks.workunit.client.0.smithi167.stdout:3/306: dread d9/d29/f33 [0,4194304] 0 2022-01-31T19:44:09.324 INFO:tasks.workunit.client.0.smithi167.stdout:3/307: rename d9/d29/d3b/d41 to d9/d29/d3b/d41/d4e 22 2022-01-31T19:44:09.324 INFO:tasks.workunit.client.0.smithi167.stdout:3/308: fsync f8 0 2022-01-31T19:44:09.324 INFO:tasks.workunit.client.0.smithi167.stdout:3/309: truncate d9/d29/f36 4611663 0 2022-01-31T19:44:09.340 INFO:tasks.workunit.client.0.smithi167.stdout:1/289: dwrite d0/d10/d5a/d33/d4b/f5b [0,4194304] 0 2022-01-31T19:44:09.340 INFO:tasks.workunit.client.0.smithi167.stdout:3/310: dwrite d9/d29/d2e/f35 [0,4194304] 0 2022-01-31T19:44:09.341 INFO:tasks.workunit.client.0.smithi167.stdout:3/311: stat d9/f11 0 2022-01-31T19:44:09.341 INFO:tasks.workunit.client.0.smithi167.stdout:7/329: dwrite d8/dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:09.342 INFO:tasks.workunit.client.0.smithi167.stdout:1/290: truncate d0/f6 2206879 0 2022-01-31T19:44:09.342 INFO:tasks.workunit.client.0.smithi167.stdout:1/291: chown d0/d10/d5a/d33 95532 1 2022-01-31T19:44:09.345 INFO:tasks.workunit.client.0.smithi167.stdout:3/312: mknod d9/d29/d3b/d41/c4f 0 2022-01-31T19:44:09.349 INFO:tasks.workunit.client.0.smithi167.stdout:7/330: rename d8/dc/d18/d24/l59 to d8/d2c/d6c/l7f 0 2022-01-31T19:44:09.350 INFO:tasks.workunit.client.0.smithi167.stdout:7/331: chown d8/d2c/d6e/f2a 123120 1 2022-01-31T19:44:09.351 INFO:tasks.workunit.client.0.smithi167.stdout:7/332: mknod d8/dc/d19/c80 0 2022-01-31T19:44:09.353 INFO:tasks.workunit.client.0.smithi167.stdout:7/333: write d8/dc/d19/f42 [3927840,83357] 0 2022-01-31T19:44:09.356 INFO:tasks.workunit.client.0.smithi167.stdout:7/334: rename d8/d2c/d43/d31/f5a to d8/d2c/d6e/d4d/d6b/f81 0 2022-01-31T19:44:09.357 INFO:tasks.workunit.client.0.smithi167.stdout:7/335: symlink d8/d2c/l82 0 2022-01-31T19:44:09.358 INFO:tasks.workunit.client.0.smithi167.stdout:7/336: mkdir d8/d2c/d43/d31/d4f/d83 0 2022-01-31T19:44:09.359 INFO:tasks.workunit.client.0.smithi167.stdout:7/337: creat d8/d2c/d6e/d4d/d5e/f84 x:0 0 0 2022-01-31T19:44:09.361 INFO:tasks.workunit.client.0.smithi167.stdout:7/338: rename d8/dc/c1d to d8/d35/c85 0 2022-01-31T19:44:09.361 INFO:tasks.workunit.client.0.smithi167.stdout:7/339: readlink d8/d2c/d43/l50 0 2022-01-31T19:44:09.362 INFO:tasks.workunit.client.0.smithi167.stdout:7/340: truncate d8/d35/f5d 393827 0 2022-01-31T19:44:09.366 INFO:tasks.workunit.client.0.smithi167.stdout:3/313: dwrite d9/f2c [0,4194304] 0 2022-01-31T19:44:09.367 INFO:tasks.workunit.client.0.smithi167.stdout:1/292: dwrite d0/f6 [0,4194304] 0 2022-01-31T19:44:09.370 INFO:tasks.workunit.client.0.smithi167.stdout:1/293: creat d0/d10/d5a/d33/d4b/f6d x:0 0 0 2022-01-31T19:44:09.370 INFO:tasks.workunit.client.0.smithi167.stdout:1/294: chown d0/c49 289 1 2022-01-31T19:44:09.371 INFO:tasks.workunit.client.0.smithi167.stdout:1/295: mknod d0/d11/d46/c6e 0 2022-01-31T19:44:09.372 INFO:tasks.workunit.client.0.smithi167.stdout:1/296: mknod d0/d10/d5a/d33/d4b/d1f/c6f 0 2022-01-31T19:44:09.375 INFO:tasks.workunit.client.0.smithi167.stdout:1/297: dread d0/f17 [0,4194304] 0 2022-01-31T19:44:09.375 INFO:tasks.workunit.client.0.smithi167.stdout:1/298: stat d0/l23 0 2022-01-31T19:44:09.377 INFO:tasks.workunit.client.0.smithi167.stdout:1/299: mknod d0/d10/d5a/d33/d4b/c70 0 2022-01-31T19:44:09.377 INFO:tasks.workunit.client.0.smithi167.stdout:1/300: dread - d0/d10/d5a/d33/f55 zero size 2022-01-31T19:44:09.377 INFO:tasks.workunit.client.0.smithi167.stdout:1/301: write d0/d10/d5a/d33/f55 [489058,2019] 0 2022-01-31T19:44:09.378 INFO:tasks.workunit.client.0.smithi167.stdout:1/302: truncate d0/d10/d5a/d33/d4b/f65 4410537 0 2022-01-31T19:44:09.378 INFO:tasks.workunit.client.0.smithi167.stdout:1/303: stat d0/d10/d20/c32 0 2022-01-31T19:44:09.378 INFO:tasks.workunit.client.0.smithi167.stdout:7/341: rmdir d8/d2c/d6e/d4d/d5e 39 2022-01-31T19:44:09.378 INFO:tasks.workunit.client.0.smithi167.stdout:3/314: dread f8 [0,4194304] 0 2022-01-31T19:44:09.379 INFO:tasks.workunit.client.0.smithi167.stdout:3/315: write f7 [578250,50721] 0 2022-01-31T19:44:09.379 INFO:tasks.workunit.client.0.smithi167.stdout:3/316: chown d9/d29/d2e/f3e 12046559 1 2022-01-31T19:44:09.380 INFO:tasks.workunit.client.0.smithi167.stdout:1/304: rmdir d0/d6c 0 2022-01-31T19:44:09.380 INFO:tasks.workunit.client.0.smithi167.stdout:7/342: mknod d8/d2c/d6e/d4d/d5e/c86 0 2022-01-31T19:44:09.382 INFO:tasks.workunit.client.0.smithi167.stdout:1/305: mknod d0/d10/d5a/d33/d4b/c71 0 2022-01-31T19:44:09.383 INFO:tasks.workunit.client.0.smithi167.stdout:7/343: getdents d8/dc/d19 0 2022-01-31T19:44:09.383 INFO:tasks.workunit.client.0.smithi167.stdout:1/306: dread d0/d10/d18/f1e [0,4194304] 0 2022-01-31T19:44:09.384 INFO:tasks.workunit.client.0.smithi167.stdout:1/307: creat d0/f72 x:0 0 0 2022-01-31T19:44:09.384 INFO:tasks.workunit.client.0.smithi167.stdout:7/344: write d8/dc/d19/f5c [1328286,127881] 0 2022-01-31T19:44:09.387 INFO:tasks.workunit.client.0.smithi167.stdout:7/345: truncate d8/fa 3699016 0 2022-01-31T19:44:09.389 INFO:tasks.workunit.client.0.smithi167.stdout:7/346: truncate d8/d2c/d43/d31/f6a 176009 0 2022-01-31T19:44:09.390 INFO:tasks.workunit.client.0.smithi167.stdout:7/347: readlink d8/d2c/d43/l48 0 2022-01-31T19:44:09.390 INFO:tasks.workunit.client.0.smithi167.stdout:7/348: write d8/d2c/d6e/d4d/d5e/f84 [1032445,81902] 0 2022-01-31T19:44:09.398 INFO:tasks.workunit.client.0.smithi167.stdout:7/349: dread d8/d35/f60 [8388608,4194304] 0 2022-01-31T19:44:09.398 INFO:tasks.workunit.client.0.smithi167.stdout:7/350: mknod d8/d2c/d43/c87 0 2022-01-31T19:44:09.399 INFO:tasks.workunit.client.0.smithi167.stdout:7/351: chown d8/dc/d19/f42 460810 1 2022-01-31T19:44:09.403 INFO:tasks.workunit.client.0.smithi167.stdout:3/317: dwrite d9/f25 [0,4194304] 0 2022-01-31T19:44:09.403 INFO:tasks.workunit.client.0.smithi167.stdout:1/308: dwrite d0/d10/d5a/d33/d4b/f65 [0,4194304] 0 2022-01-31T19:44:09.403 INFO:tasks.workunit.client.0.smithi167.stdout:1/309: chown d0/d10/d20/f28 37704315 1 2022-01-31T19:44:09.409 INFO:tasks.workunit.client.0.smithi167.stdout:1/310: rmdir d0/d10/d18 39 2022-01-31T19:44:09.410 INFO:tasks.workunit.client.0.smithi167.stdout:6/238: sync 2022-01-31T19:44:09.414 INFO:tasks.workunit.client.0.smithi167.stdout:6/239: creat d8/d33/d51/d5d/f60 x:0 0 0 2022-01-31T19:44:09.415 INFO:tasks.workunit.client.0.smithi167.stdout:6/240: stat d8/d20/c46 0 2022-01-31T19:44:09.415 INFO:tasks.workunit.client.0.smithi167.stdout:1/311: getdents d0/d10/d5a/d33/d4b/d26 0 2022-01-31T19:44:09.416 INFO:tasks.workunit.client.0.smithi167.stdout:1/312: creat d0/f73 x:0 0 0 2022-01-31T19:44:09.416 INFO:tasks.workunit.client.0.smithi167.stdout:6/241: creat d8/d33/d35/f61 x:0 0 0 2022-01-31T19:44:09.420 INFO:tasks.workunit.client.0.smithi167.stdout:7/352: dwrite d8/d2c/d6e/f44 [0,4194304] 0 2022-01-31T19:44:09.422 INFO:tasks.workunit.client.0.smithi167.stdout:7/353: getdents d8/d2c/d43/d31 0 2022-01-31T19:44:09.424 INFO:tasks.workunit.client.0.smithi167.stdout:7/354: creat d8/d2c/d6e/d4d/d6b/f88 x:0 0 0 2022-01-31T19:44:09.426 INFO:tasks.workunit.client.0.smithi167.stdout:3/318: dwrite d9/f11 [0,4194304] 0 2022-01-31T19:44:09.426 INFO:tasks.workunit.client.0.smithi167.stdout:1/313: dread d0/d10/d5a/d33/d4b/f5b [0,4194304] 0 2022-01-31T19:44:09.429 INFO:tasks.workunit.client.0.smithi167.stdout:1/314: read d0/d10/d5a/d33/d4b/f68 [2667666,84133] 0 2022-01-31T19:44:09.431 INFO:tasks.workunit.client.0.smithi167.stdout:7/355: creat d8/dc/d19/f89 x:0 0 0 2022-01-31T19:44:09.431 INFO:tasks.workunit.client.0.smithi167.stdout:3/319: mknod d9/d29/d3b/c50 0 2022-01-31T19:44:09.433 INFO:tasks.workunit.client.0.smithi167.stdout:1/315: rename d0/d10/d18/f1e to d0/d10/d5a/d33/d4b/d26/f74 0 2022-01-31T19:44:09.435 INFO:tasks.workunit.client.0.smithi167.stdout:3/320: truncate d9/d29/f2b 4016164 0 2022-01-31T19:44:09.436 INFO:tasks.workunit.client.0.smithi167.stdout:3/321: creat d9/d29/d2e/f51 x:0 0 0 2022-01-31T19:44:09.437 INFO:tasks.workunit.client.0.smithi167.stdout:3/322: write d9/d21/f3a [502331,41524] 0 2022-01-31T19:44:09.437 INFO:tasks.workunit.client.0.smithi167.stdout:3/323: creat d9/d29/f52 x:0 0 0 2022-01-31T19:44:09.438 INFO:tasks.workunit.client.0.smithi167.stdout:7/356: rename d8/dc/l34 to d8/l8a 0 2022-01-31T19:44:09.438 INFO:tasks.workunit.client.0.smithi167.stdout:7/357: fsync d8/d2c/d6e/d4d/d6b/f88 0 2022-01-31T19:44:09.439 INFO:tasks.workunit.client.0.smithi167.stdout:7/358: truncate d8/d2c/d6e/f2a 4475293 0 2022-01-31T19:44:09.439 INFO:tasks.workunit.client.0.smithi167.stdout:7/359: write d8/dc/f40 [230991,88720] 0 2022-01-31T19:44:09.444 INFO:tasks.workunit.client.0.smithi167.stdout:1/316: rename d0/d10/d18/d39 to d0/d75 0 2022-01-31T19:44:09.444 INFO:tasks.workunit.client.0.smithi167.stdout:1/317: chown d0/d10/d5a/d33/d4b/d26/l5e 419 1 2022-01-31T19:44:09.445 INFO:tasks.workunit.client.0.smithi167.stdout:1/318: stat d0/d11/d46/c58 0 2022-01-31T19:44:09.446 INFO:tasks.workunit.client.0.smithi167.stdout:3/324: dread d9/f2c [0,4194304] 0 2022-01-31T19:44:09.446 INFO:tasks.workunit.client.0.smithi167.stdout:3/325: readlink d9/l14 0 2022-01-31T19:44:09.446 INFO:tasks.workunit.client.0.smithi167.stdout:1/319: mknod d0/d10/d5a/d33/c76 0 2022-01-31T19:44:09.447 INFO:tasks.workunit.client.0.smithi167.stdout:1/320: unlink d0/c49 0 2022-01-31T19:44:09.447 INFO:tasks.workunit.client.0.smithi167.stdout:1/321: dread - d0/f73 zero size 2022-01-31T19:44:09.447 INFO:tasks.workunit.client.0.smithi167.stdout:3/326: symlink d9/d21/l53 0 2022-01-31T19:44:09.448 INFO:tasks.workunit.client.0.smithi167.stdout:1/322: mknod d0/d11/d45/c77 0 2022-01-31T19:44:09.448 INFO:tasks.workunit.client.0.smithi167.stdout:1/323: readlink d0/d10/d5a/d33/d4b/d26/l4a 0 2022-01-31T19:44:09.449 INFO:tasks.workunit.client.0.smithi167.stdout:3/327: rename d9/d29/c32 to d9/d29/d3b/d41/c54 0 2022-01-31T19:44:09.450 INFO:tasks.workunit.client.0.smithi167.stdout:1/324: write d0/f1 [1871024,121393] 0 2022-01-31T19:44:09.450 INFO:tasks.workunit.client.0.smithi167.stdout:1/325: dread - d0/d10/d5a/d33/d4b/f6d zero size 2022-01-31T19:44:09.451 INFO:tasks.workunit.client.0.smithi167.stdout:3/328: rename d9/d29/f3c to d9/d26/f55 0 2022-01-31T19:44:09.451 INFO:tasks.workunit.client.0.smithi167.stdout:3/329: dread - d9/d29/f4c zero size 2022-01-31T19:44:09.452 INFO:tasks.workunit.client.0.smithi167.stdout:1/326: rmdir d0/d75/d5d 0 2022-01-31T19:44:09.453 INFO:tasks.workunit.client.0.smithi167.stdout:3/330: mkdir d9/d56 0 2022-01-31T19:44:09.453 INFO:tasks.workunit.client.0.smithi167.stdout:3/331: readlink d9/le 0 2022-01-31T19:44:09.455 INFO:tasks.workunit.client.0.smithi167.stdout:1/327: link d0/f3a d0/d10/d5a/d33/d4b/d1f/f78 0 2022-01-31T19:44:09.456 INFO:tasks.workunit.client.0.smithi167.stdout:1/328: truncate d0/d10/d5a/d33/d4b/d26/f66 640487 0 2022-01-31T19:44:09.456 INFO:tasks.workunit.client.0.smithi167.stdout:1/329: chown d0/d11/l63 0 1 2022-01-31T19:44:09.458 INFO:tasks.workunit.client.0.smithi167.stdout:3/332: creat d9/d29/d3b/f57 x:0 0 0 2022-01-31T19:44:09.459 INFO:tasks.workunit.client.0.smithi167.stdout:3/333: chown d9/d29/d2e/f51 3446698 1 2022-01-31T19:44:09.459 INFO:tasks.workunit.client.0.smithi167.stdout:3/334: creat d9/d29/d44/f58 x:0 0 0 2022-01-31T19:44:09.459 INFO:tasks.workunit.client.0.smithi167.stdout:1/330: symlink d0/d10/d20/l79 0 2022-01-31T19:44:09.461 INFO:tasks.workunit.client.0.smithi167.stdout:3/335: creat d9/d56/f59 x:0 0 0 2022-01-31T19:44:09.461 INFO:tasks.workunit.client.0.smithi167.stdout:3/336: creat d9/d29/d3b/f5a x:0 0 0 2022-01-31T19:44:09.461 INFO:tasks.workunit.client.0.smithi167.stdout:3/337: creat d9/d29/d44/f5b x:0 0 0 2022-01-31T19:44:09.461 INFO:tasks.workunit.client.0.smithi167.stdout:7/360: dwrite d8/dc/d19/f3c [0,4194304] 0 2022-01-31T19:44:09.462 INFO:tasks.workunit.client.0.smithi167.stdout:1/331: rename d0/d10/d5a/d33/d4b/d26 to d0/d75/d7a 0 2022-01-31T19:44:09.464 INFO:tasks.workunit.client.0.smithi167.stdout:1/332: write d0/f25 [3491831,84814] 0 2022-01-31T19:44:09.464 INFO:tasks.workunit.client.0.smithi167.stdout:1/333: write d0/d75/d7a/f59 [977580,89128] 0 2022-01-31T19:44:09.470 INFO:tasks.workunit.client.0.smithi167.stdout:3/338: dread d9/d29/d2e/f35 [0,4194304] 0 2022-01-31T19:44:09.470 INFO:tasks.workunit.client.0.smithi167.stdout:3/339: chown d9/d26/f55 0 1 2022-01-31T19:44:09.471 INFO:tasks.workunit.client.0.smithi167.stdout:3/340: write d9/d29/f2b [273900,71140] 0 2022-01-31T19:44:09.472 INFO:tasks.workunit.client.0.smithi167.stdout:3/341: readlink d9/d29/d3b/l3d 0 2022-01-31T19:44:09.472 INFO:tasks.workunit.client.0.smithi167.stdout:1/334: creat d0/d10/d5a/d33/d4b/d1f/f7b x:0 0 0 2022-01-31T19:44:09.474 INFO:tasks.workunit.client.0.smithi167.stdout:3/342: link d9/f18 d9/d29/d3b/d41/f5c 0 2022-01-31T19:44:09.476 INFO:tasks.workunit.client.0.smithi167.stdout:1/335: mkdir d0/d10/d20/d7c 0 2022-01-31T19:44:09.476 INFO:tasks.workunit.client.0.smithi167.stdout:1/336: write d0/d10/d20/f37 [804344,93276] 0 2022-01-31T19:44:09.477 INFO:tasks.workunit.client.0.smithi167.stdout:3/343: link d9/c16 d9/d21/c5d 0 2022-01-31T19:44:09.478 INFO:tasks.workunit.client.0.smithi167.stdout:1/337: truncate d0/d10/f42 2151253 0 2022-01-31T19:44:09.480 INFO:tasks.workunit.client.0.smithi167.stdout:7/361: dwrite d8/dc/d18/d24/f29 [0,4194304] 0 2022-01-31T19:44:09.480 INFO:tasks.workunit.client.0.smithi167.stdout:3/344: unlink d9/d21/c20 0 2022-01-31T19:44:09.486 INFO:tasks.workunit.client.0.smithi167.stdout:7/362: symlink d8/d35/l8b 0 2022-01-31T19:44:09.491 INFO:tasks.workunit.client.0.smithi167.stdout:7/363: unlink d8/dc/d18/f76 0 2022-01-31T19:44:09.492 INFO:tasks.workunit.client.0.smithi167.stdout:7/364: symlink d8/dc/d18/l8c 0 2022-01-31T19:44:09.495 INFO:tasks.workunit.client.0.smithi167.stdout:1/338: dwrite d0/d10/d20/f37 [0,4194304] 0 2022-01-31T19:44:09.497 INFO:tasks.workunit.client.0.smithi167.stdout:7/365: rename d8/dc/d19 to d8/d2c/d43/d31/d8d 0 2022-01-31T19:44:09.498 INFO:tasks.workunit.client.0.smithi167.stdout:1/339: mknod d0/c7d 0 2022-01-31T19:44:09.500 INFO:tasks.workunit.client.0.smithi167.stdout:7/366: mkdir d8/d2c/d6e/d4d/d6b/d8e 0 2022-01-31T19:44:09.500 INFO:tasks.workunit.client.0.smithi167.stdout:7/367: chown d8/d2c/f63 4621 1 2022-01-31T19:44:09.502 INFO:tasks.workunit.client.0.smithi167.stdout:7/368: mkdir d8/d2c/d6e/d4d/d67/d8f 0 2022-01-31T19:44:09.502 INFO:tasks.workunit.client.0.smithi167.stdout:7/369: chown d8/dc/f23 1871673350 1 2022-01-31T19:44:09.507 INFO:tasks.workunit.client.0.smithi167.stdout:7/370: rename d8/dc/l4c to d8/d2c/d6e/d4d/d5e/d71/l90 0 2022-01-31T19:44:09.507 INFO:tasks.workunit.client.0.smithi167.stdout:7/371: write d8/dc/d18/f3b [4478904,106471] 0 2022-01-31T19:44:09.510 INFO:tasks.workunit.client.0.smithi167.stdout:7/372: creat d8/d2c/d6e/d4d/d5e/f91 x:0 0 0 2022-01-31T19:44:09.511 INFO:tasks.workunit.client.0.smithi167.stdout:7/373: dread d8/d35/f5d [0,4194304] 0 2022-01-31T19:44:09.512 INFO:tasks.workunit.client.0.smithi167.stdout:7/374: link d8/dc/l3e d8/d2c/d43/l92 0 2022-01-31T19:44:09.512 INFO:tasks.workunit.client.0.smithi167.stdout:7/375: truncate d8/d2c/d6e/f2a 127259 0 2022-01-31T19:44:09.514 INFO:tasks.workunit.client.0.smithi167.stdout:7/376: unlink d8/d2c/d43/d31/d8d/f89 0 2022-01-31T19:44:09.515 INFO:tasks.workunit.client.0.smithi167.stdout:7/377: read d8/dc/f33 [898688,8455] 0 2022-01-31T19:44:09.515 INFO:tasks.workunit.client.0.smithi167.stdout:7/378: stat d8/d2c/d43/f2f 0 2022-01-31T19:44:09.517 INFO:tasks.workunit.client.0.smithi167.stdout:7/379: write d8/dc/f21 [6463414,99468] 0 2022-01-31T19:44:09.517 INFO:tasks.workunit.client.0.smithi167.stdout:7/380: creat d8/d2c/d6c/f93 x:0 0 0 2022-01-31T19:44:09.518 INFO:tasks.workunit.client.0.smithi167.stdout:7/381: write d8/d2c/d6e/f62 [4859893,111376] 0 2022-01-31T19:44:09.518 INFO:tasks.workunit.client.0.smithi167.stdout:7/382: rmdir d8/d2c/d43 39 2022-01-31T19:44:09.519 INFO:tasks.workunit.client.0.smithi167.stdout:7/383: read - d8/d2c/d6c/f93 zero size 2022-01-31T19:44:09.519 INFO:tasks.workunit.client.0.smithi167.stdout:7/384: mkdir d8/d2c/d43/d94 0 2022-01-31T19:44:09.520 INFO:tasks.workunit.client.0.smithi167.stdout:7/385: stat d8/d2c/d6e/c16 0 2022-01-31T19:44:09.520 INFO:tasks.workunit.client.0.smithi167.stdout:7/386: symlink d8/d2c/d6c/l95 0 2022-01-31T19:44:09.534 INFO:tasks.workunit.client.0.smithi167.stdout:8/395: sync 2022-01-31T19:44:09.535 INFO:tasks.workunit.client.0.smithi167.stdout:4/288: sync 2022-01-31T19:44:09.535 INFO:tasks.workunit.client.0.smithi167.stdout:9/220: sync 2022-01-31T19:44:09.535 INFO:tasks.workunit.client.0.smithi167.stdout:2/248: sync 2022-01-31T19:44:09.535 INFO:tasks.workunit.client.0.smithi167.stdout:4/289: chown d0/d6/d7/d26/d40/d4c 148 1 2022-01-31T19:44:09.537 INFO:tasks.workunit.client.0.smithi167.stdout:9/221: read dd/d17/f1f [19842,92921] 0 2022-01-31T19:44:09.537 INFO:tasks.workunit.client.0.smithi167.stdout:9/222: truncate dd/d17/f36 791912 0 2022-01-31T19:44:09.537 INFO:tasks.workunit.client.0.smithi167.stdout:9/223: creat dd/d19/d24/d3b/f4a x:0 0 0 2022-01-31T19:44:09.537 INFO:tasks.workunit.client.0.smithi167.stdout:9/224: read - dd/d35/f33 zero size 2022-01-31T19:44:09.538 INFO:tasks.workunit.client.0.smithi167.stdout:9/225: chown dd/f10 3833717 1 2022-01-31T19:44:09.538 INFO:tasks.workunit.client.0.smithi167.stdout:8/396: mknod d6/db/d4e/c83 0 2022-01-31T19:44:09.538 INFO:tasks.workunit.client.0.smithi167.stdout:8/397: chown d6/db/de/d18/l64 231983 1 2022-01-31T19:44:09.538 INFO:tasks.workunit.client.0.smithi167.stdout:2/249: creat dd/df/d1b/d2f/f4d x:0 0 0 2022-01-31T19:44:09.539 INFO:tasks.workunit.client.0.smithi167.stdout:7/387: dwrite d8/d35/f5d [0,4194304] 0 2022-01-31T19:44:09.539 INFO:tasks.workunit.client.0.smithi167.stdout:7/388: fsync d8/d2c/d43/d31/d8d/f68 0 2022-01-31T19:44:09.540 INFO:tasks.workunit.client.0.smithi167.stdout:4/290: symlink d0/d6/d7/d26/d50/l5a 0 2022-01-31T19:44:09.541 INFO:tasks.workunit.client.0.smithi167.stdout:1/340: dwrite d0/f1 [0,4194304] 0 2022-01-31T19:44:09.547 INFO:tasks.workunit.client.0.smithi167.stdout:8/398: truncate d6/db/de/f10 1022145 0 2022-01-31T19:44:09.548 INFO:tasks.workunit.client.0.smithi167.stdout:8/399: fsync d6/db/de/d3d/f56 0 2022-01-31T19:44:09.548 INFO:tasks.workunit.client.0.smithi167.stdout:8/400: getdents d6/d1e/d2d/d6b 0 2022-01-31T19:44:09.548 INFO:tasks.workunit.client.0.smithi167.stdout:8/401: truncate d6/db/d28/f76 327446 0 2022-01-31T19:44:09.548 INFO:tasks.workunit.client.0.smithi167.stdout:8/402: chown d6/db/d6a/c75 45 1 2022-01-31T19:44:09.549 INFO:tasks.workunit.client.0.smithi167.stdout:7/389: creat d8/d2c/d6e/d4d/d67/d8f/f96 x:0 0 0 2022-01-31T19:44:09.552 INFO:tasks.workunit.client.0.smithi167.stdout:1/341: symlink d0/d75/l7e 0 2022-01-31T19:44:09.555 INFO:tasks.workunit.client.0.smithi167.stdout:2/250: getdents dd/df/d1b/d3d 0 2022-01-31T19:44:09.555 INFO:tasks.workunit.client.0.smithi167.stdout:2/251: write fa [9205424,4060] 0 2022-01-31T19:44:09.555 INFO:tasks.workunit.client.0.smithi167.stdout:2/252: chown dd/d18/f3a 133304764 1 2022-01-31T19:44:09.556 INFO:tasks.workunit.client.0.smithi167.stdout:8/403: rmdir d6/db/d36/d5d/d7d 0 2022-01-31T19:44:09.558 INFO:tasks.workunit.client.0.smithi167.stdout:2/253: rename dd/df/d22/f37 to dd/d18/f4e 0 2022-01-31T19:44:09.560 INFO:tasks.workunit.client.0.smithi167.stdout:1/342: write d0/d10/d5a/d33/d4b/f68 [348293,100366] 0 2022-01-31T19:44:09.560 INFO:tasks.workunit.client.0.smithi167.stdout:1/343: read - d0/d10/d20/f4f zero size 2022-01-31T19:44:09.562 INFO:tasks.workunit.client.0.smithi167.stdout:9/226: dwrite dd/d19/d24/d3b/f4a [0,4194304] 0 2022-01-31T19:44:09.562 INFO:tasks.workunit.client.0.smithi167.stdout:9/227: chown dd/d19/d24/c49 46143 1 2022-01-31T19:44:09.565 INFO:tasks.workunit.client.0.smithi167.stdout:1/344: read d0/d10/d20/f4e [1335067,128631] 0 2022-01-31T19:44:09.568 INFO:tasks.workunit.client.0.smithi167.stdout:1/345: mkdir d0/d62/d7f 0 2022-01-31T19:44:09.570 INFO:tasks.workunit.client.0.smithi167.stdout:1/346: creat d0/d10/d18/f80 x:0 0 0 2022-01-31T19:44:09.572 INFO:tasks.workunit.client.0.smithi167.stdout:1/347: rmdir d0/d11/d46 39 2022-01-31T19:44:09.572 INFO:tasks.workunit.client.0.smithi167.stdout:1/348: truncate d0/d11/f4d 1000650 0 2022-01-31T19:44:09.572 INFO:tasks.workunit.client.0.smithi167.stdout:2/254: dread dd/f23 [4194304,4194304] 0 2022-01-31T19:44:09.573 INFO:tasks.workunit.client.0.smithi167.stdout:4/291: dwrite d0/d6/d7/d39/f37 [0,4194304] 0 2022-01-31T19:44:09.574 INFO:tasks.workunit.client.0.smithi167.stdout:1/349: rename d0/d10/d5a/d33/c67 to d0/d11/c81 0 2022-01-31T19:44:09.575 INFO:tasks.workunit.client.0.smithi167.stdout:1/350: chown d0/d10/d5a/d33/d4b 721 1 2022-01-31T19:44:09.575 INFO:tasks.workunit.client.0.smithi167.stdout:1/351: write d0/f17 [947125,33000] 0 2022-01-31T19:44:09.575 INFO:tasks.workunit.client.0.smithi167.stdout:1/352: readlink d0/d10/d20/l54 0 2022-01-31T19:44:09.579 INFO:tasks.workunit.client.0.smithi167.stdout:7/390: dwrite f7 [0,4194304] 0 2022-01-31T19:44:09.581 INFO:tasks.workunit.client.0.smithi167.stdout:8/404: dwrite d6/db/d36/d5d/f77 [0,4194304] 0 2022-01-31T19:44:09.581 INFO:tasks.workunit.client.0.smithi167.stdout:8/405: chown d6/db/de/d3d/f6f 661 1 2022-01-31T19:44:09.596 INFO:tasks.workunit.client.0.smithi167.stdout:7/391: dread d8/d2c/d43/d31/d8d/f42 [0,4194304] 0 2022-01-31T19:44:09.597 INFO:tasks.workunit.client.0.smithi167.stdout:7/392: readlink d8/l27 0 2022-01-31T19:44:09.606 INFO:tasks.workunit.client.0.smithi167.stdout:2/255: dwrite dd/d18/f3a [0,4194304] 0 2022-01-31T19:44:09.607 INFO:tasks.workunit.client.0.smithi167.stdout:2/256: write dd/d18/f4c [759152,115227] 0 2022-01-31T19:44:09.607 INFO:tasks.workunit.client.0.smithi167.stdout:2/257: chown dd/df/d1b/l28 4694614 1 2022-01-31T19:44:09.607 INFO:tasks.workunit.client.0.smithi167.stdout:2/258: chown dd/le 18 1 2022-01-31T19:44:09.608 INFO:tasks.workunit.client.0.smithi167.stdout:4/292: dwrite d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:09.608 INFO:tasks.workunit.client.0.smithi167.stdout:4/293: readlink d0/d6/d7/d26/d31/l59 0 2022-01-31T19:44:09.609 INFO:tasks.workunit.client.0.smithi167.stdout:4/294: mknod d0/d6/d7/d26/c5b 0 2022-01-31T19:44:09.612 INFO:tasks.workunit.client.0.smithi167.stdout:1/353: dwrite d0/d10/d20/f4f [0,4194304] 0 2022-01-31T19:44:09.614 INFO:tasks.workunit.client.0.smithi167.stdout:1/354: mkdir d0/d62/d7f/d82 0 2022-01-31T19:44:09.615 INFO:tasks.workunit.client.0.smithi167.stdout:1/355: creat d0/d62/d7f/d82/f83 x:0 0 0 2022-01-31T19:44:09.616 INFO:tasks.workunit.client.0.smithi167.stdout:1/356: mknod d0/d10/d5a/d33/d4b/c84 0 2022-01-31T19:44:09.617 INFO:tasks.workunit.client.0.smithi167.stdout:1/357: dread d0/d75/d7a/f74 [0,4194304] 0 2022-01-31T19:44:09.622 INFO:tasks.workunit.client.0.smithi167.stdout:4/295: dread d0/d6/f3b [0,4194304] 0 2022-01-31T19:44:09.623 INFO:tasks.workunit.client.0.smithi167.stdout:4/296: rmdir d0/d6/d7/d26/d50 39 2022-01-31T19:44:09.624 INFO:tasks.workunit.client.0.smithi167.stdout:8/406: dwrite d6/db/de/d3d/f71 [0,4194304] 0 2022-01-31T19:44:09.625 INFO:tasks.workunit.client.0.smithi167.stdout:4/297: rmdir d0/d6/d7/d26/d31 39 2022-01-31T19:44:09.625 INFO:tasks.workunit.client.0.smithi167.stdout:7/393: dwrite d8/d2c/d6e/d4d/d67/d8f/f96 [0,4194304] 0 2022-01-31T19:44:09.625 INFO:tasks.workunit.client.0.smithi167.stdout:2/259: dwrite dd/df/d1b/d2f/f4d [0,4194304] 0 2022-01-31T19:44:09.628 INFO:tasks.workunit.client.0.smithi167.stdout:8/407: dread d6/db/de/f10 [0,4194304] 0 2022-01-31T19:44:09.628 INFO:tasks.workunit.client.0.smithi167.stdout:8/408: creat d6/db/d36/f84 x:0 0 0 2022-01-31T19:44:09.628 INFO:tasks.workunit.client.0.smithi167.stdout:8/409: getdents d6/d1e/d2d/d6b 0 2022-01-31T19:44:09.629 INFO:tasks.workunit.client.0.smithi167.stdout:8/410: write d6/d1e/f22 [560214,36617] 0 2022-01-31T19:44:09.632 INFO:tasks.workunit.client.0.smithi167.stdout:4/298: mkdir d0/d6/d7/d26/d31/d5c 0 2022-01-31T19:44:09.633 INFO:tasks.workunit.client.0.smithi167.stdout:4/299: write d0/d6/d7/f30 [345698,12414] 0 2022-01-31T19:44:09.635 INFO:tasks.workunit.client.0.smithi167.stdout:1/358: getdents d0/d62/d7f/d82 0 2022-01-31T19:44:09.636 INFO:tasks.workunit.client.0.smithi167.stdout:7/394: creat d8/d2c/d43/f97 x:0 0 0 2022-01-31T19:44:09.641 INFO:tasks.workunit.client.0.smithi167.stdout:2/260: unlink dd/df/d1b/l28 0 2022-01-31T19:44:09.641 INFO:tasks.workunit.client.0.smithi167.stdout:2/261: chown dd/df/c10 71517 1 2022-01-31T19:44:09.643 INFO:tasks.workunit.client.0.smithi167.stdout:4/300: mkdir d0/d5d 0 2022-01-31T19:44:09.643 INFO:tasks.workunit.client.0.smithi167.stdout:4/301: dread - d0/d6/d7/d26/d40/d4c/f4f zero size 2022-01-31T19:44:09.645 INFO:tasks.workunit.client.0.smithi167.stdout:7/395: rename d8/d2c/l82 to d8/d2c/d6e/d4d/d5e/d71/l98 0 2022-01-31T19:44:09.645 INFO:tasks.workunit.client.0.smithi167.stdout:2/262: rename dd/d18 to dd/d18/d4f 22 2022-01-31T19:44:09.645 INFO:tasks.workunit.client.0.smithi167.stdout:7/396: chown d8/d2c/d6c/l7f 31336 1 2022-01-31T19:44:09.646 INFO:tasks.workunit.client.0.smithi167.stdout:2/263: truncate dd/df/d2c/f2d 6043835 0 2022-01-31T19:44:09.647 INFO:tasks.workunit.client.0.smithi167.stdout:7/397: unlink d8/dc/l3e 0 2022-01-31T19:44:09.648 INFO:tasks.workunit.client.0.smithi167.stdout:2/264: creat dd/df/f50 x:0 0 0 2022-01-31T19:44:09.649 INFO:tasks.workunit.client.0.smithi167.stdout:7/398: mkdir d8/d2c/d7a/d99 0 2022-01-31T19:44:09.650 INFO:tasks.workunit.client.0.smithi167.stdout:2/265: mkdir dd/df/d3f/d51 0 2022-01-31T19:44:09.650 INFO:tasks.workunit.client.0.smithi167.stdout:2/266: fdatasync dd/f15 0 2022-01-31T19:44:09.651 INFO:tasks.workunit.client.0.smithi167.stdout:4/302: dread d0/d6/d7/d26/d31/f19 [0,4194304] 0 2022-01-31T19:44:09.651 INFO:tasks.workunit.client.0.smithi167.stdout:7/399: mkdir d8/d2c/d6e/d9a 0 2022-01-31T19:44:09.652 INFO:tasks.workunit.client.0.smithi167.stdout:7/400: dread d8/dc/f33 [0,4194304] 0 2022-01-31T19:44:09.653 INFO:tasks.workunit.client.0.smithi167.stdout:7/401: stat d8/d2c/d43/d31/d8d/f1a 0 2022-01-31T19:44:09.653 INFO:tasks.workunit.client.0.smithi167.stdout:7/402: readlink l0 0 2022-01-31T19:44:09.653 INFO:tasks.workunit.client.0.smithi167.stdout:4/303: rename d0/d6/d7/d39/f44 to d0/d6/d7/d26/d31/f5e 0 2022-01-31T19:44:09.653 INFO:tasks.workunit.client.0.smithi167.stdout:7/403: truncate f7 966941 0 2022-01-31T19:44:09.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/404: unlink d8/d2c/d6e/c16 0 2022-01-31T19:44:09.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/405: fsync d8/d2c/d6e/d4d/d6b/f88 0 2022-01-31T19:44:09.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/406: chown d8/c78 61632 1 2022-01-31T19:44:09.656 INFO:tasks.workunit.client.0.smithi167.stdout:7/407: truncate d8/dc/f21 6086710 0 2022-01-31T19:44:09.656 INFO:tasks.workunit.client.0.smithi167.stdout:7/408: chown d8/d2c 99 1 2022-01-31T19:44:09.656 INFO:tasks.workunit.client.0.smithi167.stdout:4/304: dread d0/d6/d7/d39/f2b [0,4194304] 0 2022-01-31T19:44:09.657 INFO:tasks.workunit.client.0.smithi167.stdout:4/305: symlink d0/d6/d7/d26/d50/l5f 0 2022-01-31T19:44:09.658 INFO:tasks.workunit.client.0.smithi167.stdout:4/306: mkdir d0/d3f/d60 0 2022-01-31T19:44:09.659 INFO:tasks.workunit.client.0.smithi167.stdout:1/359: dwrite d0/d10/d20/f2e [0,4194304] 0 2022-01-31T19:44:09.660 INFO:tasks.workunit.client.0.smithi167.stdout:8/411: fsync d6/db/d36/f84 0 2022-01-31T19:44:09.661 INFO:tasks.workunit.client.0.smithi167.stdout:8/412: chown d6/db/de/d18/c2e 54 1 2022-01-31T19:44:09.665 INFO:tasks.workunit.client.0.smithi167.stdout:1/360: mknod d0/d10/d5a/d33/d4b/d1f/c85 0 2022-01-31T19:44:09.666 INFO:tasks.workunit.client.0.smithi167.stdout:1/361: chown d0/d11/d45 0 1 2022-01-31T19:44:09.666 INFO:tasks.workunit.client.0.smithi167.stdout:1/362: fdatasync d0/d10/f42 0 2022-01-31T19:44:09.666 INFO:tasks.workunit.client.0.smithi167.stdout:1/363: write d0/d10/d18/f80 [751033,33782] 0 2022-01-31T19:44:09.666 INFO:tasks.workunit.client.0.smithi167.stdout:1/364: truncate d0/d10/d5a/d33/d4b/f57 881754 0 2022-01-31T19:44:09.669 INFO:tasks.workunit.client.0.smithi167.stdout:1/365: read d0/f6 [2766795,3818] 0 2022-01-31T19:44:09.669 INFO:tasks.workunit.client.0.smithi167.stdout:7/409: write f7 [1906545,72091] 0 2022-01-31T19:44:09.682 INFO:tasks.workunit.client.0.smithi167.stdout:4/307: rmdir d0 39 2022-01-31T19:44:09.683 INFO:tasks.workunit.client.0.smithi167.stdout:4/308: truncate d0/d6/d7/d39/d3a/f48 264548 0 2022-01-31T19:44:09.683 INFO:tasks.workunit.client.0.smithi167.stdout:8/413: mknod d6/db/d2f/c85 0 2022-01-31T19:44:09.683 INFO:tasks.workunit.client.0.smithi167.stdout:8/414: chown d6/db/d28/d61/c7a 262590272 1 2022-01-31T19:44:09.684 INFO:tasks.workunit.client.0.smithi167.stdout:8/415: creat d6/db/de/f86 x:0 0 0 2022-01-31T19:44:09.684 INFO:tasks.workunit.client.0.smithi167.stdout:8/416: chown d6/fa 20621606 1 2022-01-31T19:44:09.684 INFO:tasks.workunit.client.0.smithi167.stdout:8/417: mknod d6/db/d6a/c87 0 2022-01-31T19:44:09.684 INFO:tasks.workunit.client.0.smithi167.stdout:1/366: rmdir d0/d10/d5a/d33/d4b 39 2022-01-31T19:44:09.685 INFO:tasks.workunit.client.0.smithi167.stdout:1/367: write d0/f30 [737686,62498] 0 2022-01-31T19:44:09.685 INFO:tasks.workunit.client.0.smithi167.stdout:2/267: dwrite dd/f15 [0,4194304] 0 2022-01-31T19:44:09.686 INFO:tasks.workunit.client.0.smithi167.stdout:7/410: rename d8/d2c/d43/d31/d8d/l38 to d8/d2c/d6e/d9a/l9b 0 2022-01-31T19:44:09.686 INFO:tasks.workunit.client.0.smithi167.stdout:7/411: truncate d8/d2c/d6c/f93 27541 0 2022-01-31T19:44:09.687 INFO:tasks.workunit.client.0.smithi167.stdout:8/418: mkdir d6/db/d4e/d73/d88 0 2022-01-31T19:44:09.689 INFO:tasks.workunit.client.0.smithi167.stdout:1/368: unlink d0/d10/d20/f2e 0 2022-01-31T19:44:09.689 INFO:tasks.workunit.client.0.smithi167.stdout:1/369: creat d0/f86 x:0 0 0 2022-01-31T19:44:09.689 INFO:tasks.workunit.client.0.smithi167.stdout:1/370: write d0/d10/d20/f28 [5011265,67608] 0 2022-01-31T19:44:09.690 INFO:tasks.workunit.client.0.smithi167.stdout:2/268: symlink dd/df/l52 0 2022-01-31T19:44:09.691 INFO:tasks.workunit.client.0.smithi167.stdout:7/412: dread d8/d2c/d43/d31/d8d/f42 [0,4194304] 0 2022-01-31T19:44:09.691 INFO:tasks.workunit.client.0.smithi167.stdout:7/413: chown d8/d2c/d6e/d4d/d6b/f81 793 1 2022-01-31T19:44:09.691 INFO:tasks.workunit.client.0.smithi167.stdout:8/419: creat d6/db/d28/d61/f89 x:0 0 0 2022-01-31T19:44:09.692 INFO:tasks.workunit.client.0.smithi167.stdout:1/371: mknod d0/d11/d45/c87 0 2022-01-31T19:44:09.693 INFO:tasks.workunit.client.0.smithi167.stdout:7/414: creat d8/d2c/d7a/f9c x:0 0 0 2022-01-31T19:44:09.693 INFO:tasks.workunit.client.0.smithi167.stdout:8/420: mknod d6/db/d4e/d73/c8a 0 2022-01-31T19:44:09.696 INFO:tasks.workunit.client.0.smithi167.stdout:4/309: dwrite d0/d6/d7/f4a [0,4194304] 0 2022-01-31T19:44:09.696 INFO:tasks.workunit.client.0.smithi167.stdout:4/310: read - d0/d6/d7/f18 zero size 2022-01-31T19:44:09.697 INFO:tasks.workunit.client.0.smithi167.stdout:1/372: read d0/d10/d18/f3e [2655483,36963] 0 2022-01-31T19:44:09.699 INFO:tasks.workunit.client.0.smithi167.stdout:5/443: sync 2022-01-31T19:44:09.699 INFO:tasks.workunit.client.0.smithi167.stdout:0/148: sync 2022-01-31T19:44:09.700 INFO:tasks.workunit.client.0.smithi167.stdout:5/444: fsync d5/d6/de/d22/d4d/d65/f79 0 2022-01-31T19:44:09.701 INFO:tasks.workunit.client.0.smithi167.stdout:0/149: write f14 [429991,28192] 0 2022-01-31T19:44:09.702 INFO:tasks.workunit.client.0.smithi167.stdout:5/445: mkdir d5/d6/d16/d18/d43/d45/d69/d90 0 2022-01-31T19:44:09.706 INFO:tasks.workunit.client.0.smithi167.stdout:8/421: dread d6/db/fd [0,4194304] 0 2022-01-31T19:44:09.710 INFO:tasks.workunit.client.0.smithi167.stdout:4/311: dread d0/d3f/f3d [0,4194304] 0 2022-01-31T19:44:09.710 INFO:tasks.workunit.client.0.smithi167.stdout:4/312: write d0/d6/d7/d26/d40/f27 [1605081,118243] 0 2022-01-31T19:44:09.714 INFO:tasks.workunit.client.0.smithi167.stdout:8/422: rename d6/db/d36/d5d/f77 to d6/d1e/d4c/f8b 0 2022-01-31T19:44:09.716 INFO:tasks.workunit.client.0.smithi167.stdout:2/269: dwrite dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:09.725 INFO:tasks.workunit.client.0.smithi167.stdout:5/446: getdents d5/d6/de/d22/d6d 0 2022-01-31T19:44:09.727 INFO:tasks.workunit.client.0.smithi167.stdout:4/313: mkdir d0/d6/d7/d39/d4e/d61 0 2022-01-31T19:44:09.728 INFO:tasks.workunit.client.0.smithi167.stdout:8/423: creat d6/db/d28/d66/f8c x:0 0 0 2022-01-31T19:44:09.729 INFO:tasks.workunit.client.0.smithi167.stdout:8/424: dread - d6/db/de/f86 zero size 2022-01-31T19:44:09.730 INFO:tasks.workunit.client.0.smithi167.stdout:2/270: symlink dd/df/d22/l53 0 2022-01-31T19:44:09.731 INFO:tasks.workunit.client.0.smithi167.stdout:2/271: write dd/df/f1c [1051305,112520] 0 2022-01-31T19:44:09.732 INFO:tasks.workunit.client.0.smithi167.stdout:4/314: creat d0/d6/d7/d26/f62 x:0 0 0 2022-01-31T19:44:09.733 INFO:tasks.workunit.client.0.smithi167.stdout:2/272: truncate dd/d18/f3a 2998820 0 2022-01-31T19:44:09.734 INFO:tasks.workunit.client.0.smithi167.stdout:4/315: symlink d0/d3f/d60/l63 0 2022-01-31T19:44:09.735 INFO:tasks.workunit.client.0.smithi167.stdout:4/316: creat d0/d6/d7/d26/d40/f64 x:0 0 0 2022-01-31T19:44:09.737 INFO:tasks.workunit.client.0.smithi167.stdout:4/317: write d0/d6/d7/d39/f43 [2127870,28135] 0 2022-01-31T19:44:09.740 INFO:tasks.workunit.client.0.smithi167.stdout:4/318: mkdir d0/d3f/d60/d65 0 2022-01-31T19:44:09.742 INFO:tasks.workunit.client.0.smithi167.stdout:1/373: dwrite d0/f5 [4194304,4194304] 0 2022-01-31T19:44:09.743 INFO:tasks.workunit.client.0.smithi167.stdout:1/374: unlink d0/f6 0 2022-01-31T19:44:09.743 INFO:tasks.workunit.client.0.smithi167.stdout:7/415: dwrite d8/d2c/d6e/d4d/d5e/f91 [0,4194304] 0 2022-01-31T19:44:09.743 INFO:tasks.workunit.client.0.smithi167.stdout:0/150: dwrite fa [0,4194304] 0 2022-01-31T19:44:09.748 INFO:tasks.workunit.client.0.smithi167.stdout:1/375: rmdir d0/d75/d7a 39 2022-01-31T19:44:09.752 INFO:tasks.workunit.client.0.smithi167.stdout:7/416: rmdir d8/d35 39 2022-01-31T19:44:09.752 INFO:tasks.workunit.client.0.smithi167.stdout:7/417: truncate d8/d2c/d6e/d4d/d67/f7e 731801 0 2022-01-31T19:44:09.753 INFO:tasks.workunit.client.0.smithi167.stdout:0/151: creat d17/d19/f2d x:0 0 0 2022-01-31T19:44:09.753 INFO:tasks.workunit.client.0.smithi167.stdout:0/152: creat d17/d19/f2e x:0 0 0 2022-01-31T19:44:09.754 INFO:tasks.workunit.client.0.smithi167.stdout:4/319: mknod d0/d6/d7/d26/d40/c66 0 2022-01-31T19:44:09.755 INFO:tasks.workunit.client.0.smithi167.stdout:1/376: link d0/d75/d7a/f66 d0/d62/d7f/d82/f88 0 2022-01-31T19:44:09.755 INFO:tasks.workunit.client.0.smithi167.stdout:1/377: readlink d0/d10/d5a/l51 0 2022-01-31T19:44:09.756 INFO:tasks.workunit.client.0.smithi167.stdout:7/418: unlink d8/d2c/d6c/f93 0 2022-01-31T19:44:09.758 INFO:tasks.workunit.client.0.smithi167.stdout:4/320: symlink d0/d5d/l67 0 2022-01-31T19:44:09.758 INFO:tasks.workunit.client.0.smithi167.stdout:4/321: stat d0/d6/d7/d39/c2d 0 2022-01-31T19:44:09.759 INFO:tasks.workunit.client.0.smithi167.stdout:7/419: mknod d8/d2c/d43/d31/d4f/c9d 0 2022-01-31T19:44:09.759 INFO:tasks.workunit.client.0.smithi167.stdout:7/420: unlink d8/dc/c69 0 2022-01-31T19:44:09.760 INFO:tasks.workunit.client.0.smithi167.stdout:0/153: link d17/d19/f2e d17/f2f 0 2022-01-31T19:44:09.760 INFO:tasks.workunit.client.0.smithi167.stdout:0/154: chown d17/d19/d1c 233017 1 2022-01-31T19:44:09.760 INFO:tasks.workunit.client.0.smithi167.stdout:4/322: symlink d0/d51/l68 0 2022-01-31T19:44:09.761 INFO:tasks.workunit.client.0.smithi167.stdout:7/421: chown d8/d2c/d43/d94 363 1 2022-01-31T19:44:09.762 INFO:tasks.workunit.client.0.smithi167.stdout:7/422: write d8/d35/f36 [900521,78917] 0 2022-01-31T19:44:09.762 INFO:tasks.workunit.client.0.smithi167.stdout:0/155: creat d17/d19/d1c/f30 x:0 0 0 2022-01-31T19:44:09.763 INFO:tasks.workunit.client.0.smithi167.stdout:4/323: symlink d0/d51/l69 0 2022-01-31T19:44:09.763 INFO:tasks.workunit.client.0.smithi167.stdout:4/324: readlink d0/d3f/l3e 0 2022-01-31T19:44:09.764 INFO:tasks.workunit.client.0.smithi167.stdout:4/325: stat d0/d5d 0 2022-01-31T19:44:09.764 INFO:tasks.workunit.client.0.smithi167.stdout:4/326: chown d0/l1 14081972 1 2022-01-31T19:44:09.764 INFO:tasks.workunit.client.0.smithi167.stdout:4/327: dread - d0/d6/f58 zero size 2022-01-31T19:44:09.764 INFO:tasks.workunit.client.0.smithi167.stdout:4/328: write d0/d6/d7/d39/d3a/f54 [361480,83263] 0 2022-01-31T19:44:09.766 INFO:tasks.workunit.client.0.smithi167.stdout:7/423: mkdir d8/d2c/d6c/d9e 0 2022-01-31T19:44:09.766 INFO:tasks.workunit.client.0.smithi167.stdout:5/447: dwrite d5/d61/f4b [0,4194304] 0 2022-01-31T19:44:09.766 INFO:tasks.workunit.client.0.smithi167.stdout:8/425: dwrite d6/db/d2f/f81 [0,4194304] 0 2022-01-31T19:44:09.767 INFO:tasks.workunit.client.0.smithi167.stdout:8/426: fdatasync d6/db/de/f20 0 2022-01-31T19:44:09.767 INFO:tasks.workunit.client.0.smithi167.stdout:2/273: dwrite dd/df/d1b/d3d/f44 [0,4194304] 0 2022-01-31T19:44:09.767 INFO:tasks.workunit.client.0.smithi167.stdout:2/274: chown dd/df/l38 295 1 2022-01-31T19:44:09.767 INFO:tasks.workunit.client.0.smithi167.stdout:0/156: creat d17/f31 x:0 0 0 2022-01-31T19:44:09.768 INFO:tasks.workunit.client.0.smithi167.stdout:0/157: creat d17/d19/d1c/f32 x:0 0 0 2022-01-31T19:44:09.768 INFO:tasks.workunit.client.0.smithi167.stdout:2/275: write dd/df/d1b/d3d/f47 [4201195,69988] 0 2022-01-31T19:44:09.768 INFO:tasks.workunit.client.0.smithi167.stdout:4/329: creat d0/d3f/d1a/f6a x:0 0 0 2022-01-31T19:44:09.769 INFO:tasks.workunit.client.0.smithi167.stdout:7/424: write d8/d2c/d43/d31/d8d/f1a [2716968,56416] 0 2022-01-31T19:44:09.769 INFO:tasks.workunit.client.0.smithi167.stdout:5/448: dread d5/f8 [0,4194304] 0 2022-01-31T19:44:09.773 INFO:tasks.workunit.client.0.smithi167.stdout:8/427: dread d6/fa [0,4194304] 0 2022-01-31T19:44:09.775 INFO:tasks.workunit.client.0.smithi167.stdout:2/276: mkdir dd/df/d1b/d3d/d4a/d4b/d54 0 2022-01-31T19:44:09.775 INFO:tasks.workunit.client.0.smithi167.stdout:2/277: chown dd/df/f50 2783 1 2022-01-31T19:44:09.776 INFO:tasks.workunit.client.0.smithi167.stdout:2/278: creat dd/f55 x:0 0 0 2022-01-31T19:44:09.777 INFO:tasks.workunit.client.0.smithi167.stdout:8/428: dread d6/db/d36/d5d/f68 [0,4194304] 0 2022-01-31T19:44:09.777 INFO:tasks.workunit.client.0.smithi167.stdout:8/429: write d6/d1e/d2d/f40 [848349,84134] 0 2022-01-31T19:44:09.777 INFO:tasks.workunit.client.0.smithi167.stdout:4/330: symlink d0/l6b 0 2022-01-31T19:44:09.777 INFO:tasks.workunit.client.0.smithi167.stdout:4/331: dread - d0/d6/d7/d26/f62 zero size 2022-01-31T19:44:09.778 INFO:tasks.workunit.client.0.smithi167.stdout:4/332: readlink d0/d5d/l67 0 2022-01-31T19:44:09.778 INFO:tasks.workunit.client.0.smithi167.stdout:1/378: dwrite d0/d75/f52 [0,4194304] 0 2022-01-31T19:44:09.779 INFO:tasks.workunit.client.0.smithi167.stdout:1/379: write d0/d10/d20/f56 [500169,47056] 0 2022-01-31T19:44:09.779 INFO:tasks.workunit.client.0.smithi167.stdout:5/449: mkdir d5/d6/d16/d18/d91 0 2022-01-31T19:44:09.779 INFO:tasks.workunit.client.0.smithi167.stdout:5/450: readlink d5/d6/de/d22/d4d/l6e 0 2022-01-31T19:44:09.794 INFO:tasks.workunit.client.0.smithi167.stdout:8/430: mkdir d6/db/d6a/d8d 0 2022-01-31T19:44:09.794 INFO:tasks.workunit.client.0.smithi167.stdout:0/158: rmdir d17/d19/d1c 39 2022-01-31T19:44:09.794 INFO:tasks.workunit.client.0.smithi167.stdout:0/159: chown d17/f22 101 1 2022-01-31T19:44:09.794 INFO:tasks.workunit.client.0.smithi167.stdout:0/160: write f8 [4887074,45549] 0 2022-01-31T19:44:09.795 INFO:tasks.workunit.client.0.smithi167.stdout:4/333: mknod d0/d6/d7/d39/d4e/c6c 0 2022-01-31T19:44:09.796 INFO:tasks.workunit.client.0.smithi167.stdout:5/451: dwrite d5/d61/f7f [0,4194304] 0 2022-01-31T19:44:09.799 INFO:tasks.workunit.client.0.smithi167.stdout:7/425: getdents d8/d2c/d6e 0 2022-01-31T19:44:09.799 INFO:tasks.workunit.client.0.smithi167.stdout:7/426: stat d8/d35/f60 0 2022-01-31T19:44:09.799 INFO:tasks.workunit.client.0.smithi167.stdout:7/427: dread - d8/d2c/d6c/f79 zero size 2022-01-31T19:44:09.800 INFO:tasks.workunit.client.0.smithi167.stdout:1/380: getdents d0/d62/d7f/d82 0 2022-01-31T19:44:09.801 INFO:tasks.workunit.client.0.smithi167.stdout:2/279: getdents dd/d18 0 2022-01-31T19:44:09.804 INFO:tasks.workunit.client.0.smithi167.stdout:0/161: mkdir d17/d33 0 2022-01-31T19:44:09.805 INFO:tasks.workunit.client.0.smithi167.stdout:4/334: mknod d0/d6/d7/d39/d4e/c6d 0 2022-01-31T19:44:09.814 INFO:tasks.workunit.client.0.smithi167.stdout:5/452: unlink d5/d6/f1a 0 2022-01-31T19:44:09.814 INFO:tasks.workunit.client.0.smithi167.stdout:5/453: chown d5/d6/de/d22/d4d/f85 250643 1 2022-01-31T19:44:09.818 INFO:tasks.workunit.client.0.smithi167.stdout:2/280: dread fa [4194304,4194304] 0 2022-01-31T19:44:09.819 INFO:tasks.workunit.client.0.smithi167.stdout:7/428: mknod d8/d2c/d43/d94/c9f 0 2022-01-31T19:44:09.820 INFO:tasks.workunit.client.0.smithi167.stdout:1/381: symlink d0/d62/l89 0 2022-01-31T19:44:09.820 INFO:tasks.workunit.client.0.smithi167.stdout:1/382: write d0/d10/d18/f80 [1402272,85912] 0 2022-01-31T19:44:09.821 INFO:tasks.workunit.client.0.smithi167.stdout:8/431: getdents d6/db/d4e 0 2022-01-31T19:44:09.821 INFO:tasks.workunit.client.0.smithi167.stdout:8/432: fdatasync d6/d1e/f41 0 2022-01-31T19:44:09.824 INFO:tasks.workunit.client.0.smithi167.stdout:0/162: creat d17/d19/d1c/f34 x:0 0 0 2022-01-31T19:44:09.824 INFO:tasks.workunit.client.0.smithi167.stdout:6/242: sync 2022-01-31T19:44:09.824 INFO:tasks.workunit.client.0.smithi167.stdout:6/243: write d8/f1c [975430,3524] 0 2022-01-31T19:44:09.825 INFO:tasks.workunit.client.0.smithi167.stdout:4/335: truncate d0/d6/d7/f4a 2364538 0 2022-01-31T19:44:09.826 INFO:tasks.workunit.client.0.smithi167.stdout:5/454: creat d5/d6/f92 x:0 0 0 2022-01-31T19:44:09.827 INFO:tasks.workunit.client.0.smithi167.stdout:5/455: truncate d5/d6/d3a/f8f 897794 0 2022-01-31T19:44:09.827 INFO:tasks.workunit.client.0.smithi167.stdout:1/383: mknod d0/d75/c8a 0 2022-01-31T19:44:09.830 INFO:tasks.workunit.client.0.smithi167.stdout:6/244: mkdir d8/d62 0 2022-01-31T19:44:09.832 INFO:tasks.workunit.client.0.smithi167.stdout:6/245: dread d8/f9 [0,4194304] 0 2022-01-31T19:44:09.834 INFO:tasks.workunit.client.0.smithi167.stdout:6/246: dread d8/d20/f2f [0,4194304] 0 2022-01-31T19:44:09.836 INFO:tasks.workunit.client.0.smithi167.stdout:6/247: creat d8/d20/d54/f63 x:0 0 0 2022-01-31T19:44:09.836 INFO:tasks.workunit.client.0.smithi167.stdout:6/248: chown d8/d33/d4f/l5a 13392 1 2022-01-31T19:44:09.838 INFO:tasks.workunit.client.0.smithi167.stdout:6/249: getdents d8/d33/d35 0 2022-01-31T19:44:09.839 INFO:tasks.workunit.client.0.smithi167.stdout:7/429: dread d8/d35/f5d [0,4194304] 0 2022-01-31T19:44:09.841 INFO:tasks.workunit.client.0.smithi167.stdout:7/430: mknod d8/d2c/d6e/ca0 0 2022-01-31T19:44:09.842 INFO:tasks.workunit.client.0.smithi167.stdout:6/250: mkdir d8/d33/d51/d5d/d64 0 2022-01-31T19:44:09.842 INFO:tasks.workunit.client.0.smithi167.stdout:6/251: creat d8/d33/d4f/f65 x:0 0 0 2022-01-31T19:44:09.843 INFO:tasks.workunit.client.0.smithi167.stdout:7/431: unlink d8/d35/f60 0 2022-01-31T19:44:09.843 INFO:tasks.workunit.client.0.smithi167.stdout:7/432: chown d8/d2c/d6e/f2a 0 1 2022-01-31T19:44:09.844 INFO:tasks.workunit.client.0.smithi167.stdout:7/433: mkdir d8/d2c/d43/da1 0 2022-01-31T19:44:09.845 INFO:tasks.workunit.client.0.smithi167.stdout:7/434: chown d8/d35/c7d 16284251 1 2022-01-31T19:44:09.851 INFO:tasks.workunit.client.0.smithi167.stdout:2/281: dwrite dd/df/d1b/d2f/f34 [0,4194304] 0 2022-01-31T19:44:09.852 INFO:tasks.workunit.client.0.smithi167.stdout:6/252: write d8/d33/f3c [486202,110571] 0 2022-01-31T19:44:09.853 INFO:tasks.workunit.client.0.smithi167.stdout:6/253: symlink d8/d62/l66 0 2022-01-31T19:44:09.857 INFO:tasks.workunit.client.0.smithi167.stdout:0/163: dwrite d17/f22 [0,4194304] 0 2022-01-31T19:44:09.858 INFO:tasks.workunit.client.0.smithi167.stdout:0/164: chown d17/f31 154425026 1 2022-01-31T19:44:09.859 INFO:tasks.workunit.client.0.smithi167.stdout:0/165: truncate f14 1121110 0 2022-01-31T19:44:09.860 INFO:tasks.workunit.client.0.smithi167.stdout:0/166: mknod d17/c35 0 2022-01-31T19:44:09.861 INFO:tasks.workunit.client.0.smithi167.stdout:9/228: sync 2022-01-31T19:44:09.861 INFO:tasks.workunit.client.0.smithi167.stdout:9/229: read - dd/d35/f33 zero size 2022-01-31T19:44:09.861 INFO:tasks.workunit.client.0.smithi167.stdout:3/345: sync 2022-01-31T19:44:09.863 INFO:tasks.workunit.client.0.smithi167.stdout:1/384: dwrite d0/f72 [0,4194304] 0 2022-01-31T19:44:09.864 INFO:tasks.workunit.client.0.smithi167.stdout:2/282: dread dd/df/f35 [0,4194304] 0 2022-01-31T19:44:09.864 INFO:tasks.workunit.client.0.smithi167.stdout:0/167: creat d17/d33/f36 x:0 0 0 2022-01-31T19:44:09.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/346: symlink d9/d26/l5e 0 2022-01-31T19:44:09.867 INFO:tasks.workunit.client.0.smithi167.stdout:1/385: symlink d0/d62/d7f/l8b 0 2022-01-31T19:44:09.868 INFO:tasks.workunit.client.0.smithi167.stdout:0/168: creat d17/d2c/f37 x:0 0 0 2022-01-31T19:44:09.869 INFO:tasks.workunit.client.0.smithi167.stdout:0/169: write d17/d2c/f37 [811867,43887] 0 2022-01-31T19:44:09.870 INFO:tasks.workunit.client.0.smithi167.stdout:3/347: truncate d9/d29/d2e/f3e 637837 0 2022-01-31T19:44:09.870 INFO:tasks.workunit.client.0.smithi167.stdout:3/348: dread - d9/d29/d2e/f51 zero size 2022-01-31T19:44:09.870 INFO:tasks.workunit.client.0.smithi167.stdout:3/349: unlink d9/d21/f3a 0 2022-01-31T19:44:09.871 INFO:tasks.workunit.client.0.smithi167.stdout:3/350: chown f8 6101367 1 2022-01-31T19:44:09.871 INFO:tasks.workunit.client.0.smithi167.stdout:3/351: truncate d9/f27 3046524 0 2022-01-31T19:44:09.871 INFO:tasks.workunit.client.0.smithi167.stdout:1/386: mknod d0/d10/d5a/d33/c8c 0 2022-01-31T19:44:09.872 INFO:tasks.workunit.client.0.smithi167.stdout:0/170: mknod d17/d19/c38 0 2022-01-31T19:44:09.875 INFO:tasks.workunit.client.0.smithi167.stdout:3/352: mknod d9/d29/d2e/c5f 0 2022-01-31T19:44:09.875 INFO:tasks.workunit.client.0.smithi167.stdout:3/353: stat d9/d21 0 2022-01-31T19:44:09.875 INFO:tasks.workunit.client.0.smithi167.stdout:1/387: write d0/d75/f3d [3629816,85459] 0 2022-01-31T19:44:09.877 INFO:tasks.workunit.client.0.smithi167.stdout:1/388: unlink d0/d62/d7f/d82/f83 0 2022-01-31T19:44:09.878 INFO:tasks.workunit.client.0.smithi167.stdout:1/389: chown d0/d10/d5a/d33/l36 3564300 1 2022-01-31T19:44:09.878 INFO:tasks.workunit.client.0.smithi167.stdout:7/435: dwrite d8/d2c/d6e/d4d/d5e/f84 [0,4194304] 0 2022-01-31T19:44:09.879 INFO:tasks.workunit.client.0.smithi167.stdout:7/436: dread - d8/d2c/d6c/f79 zero size 2022-01-31T19:44:09.879 INFO:tasks.workunit.client.0.smithi167.stdout:5/456: dwrite d5/d6/d16/d18/d43/d45/f83 [0,4194304] 0 2022-01-31T19:44:09.880 INFO:tasks.workunit.client.0.smithi167.stdout:1/390: truncate d0/f1b 2107271 0 2022-01-31T19:44:09.880 INFO:tasks.workunit.client.0.smithi167.stdout:1/391: stat d0/f72 0 2022-01-31T19:44:09.881 INFO:tasks.workunit.client.0.smithi167.stdout:4/336: write d0/d6/d7/f4a [2232767,8507] 0 2022-01-31T19:44:09.882 INFO:tasks.workunit.client.0.smithi167.stdout:4/337: fsync d0/d6/d7/d26/d40/d4c/f4f 0 2022-01-31T19:44:09.883 INFO:tasks.workunit.client.0.smithi167.stdout:2/283: dread dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:09.887 INFO:tasks.workunit.client.0.smithi167.stdout:5/457: dread d5/d6/d16/d18/f3d [0,4194304] 0 2022-01-31T19:44:09.888 INFO:tasks.workunit.client.0.smithi167.stdout:7/437: unlink d8/d2c/d43/d31/f5b 0 2022-01-31T19:44:09.897 INFO:tasks.workunit.client.0.smithi167.stdout:2/284: dread dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:09.898 INFO:tasks.workunit.client.0.smithi167.stdout:4/338: rename d0/ca to d0/d3f/d1a/c6e 0 2022-01-31T19:44:09.899 INFO:tasks.workunit.client.0.smithi167.stdout:7/438: symlink d8/d2c/d43/d31/d4f/d83/la2 0 2022-01-31T19:44:09.899 INFO:tasks.workunit.client.0.smithi167.stdout:7/439: write d8/dc/d18/d24/f73 [528452,83028] 0 2022-01-31T19:44:09.901 INFO:tasks.workunit.client.0.smithi167.stdout:1/392: dread d0/d10/d5a/f50 [0,4194304] 0 2022-01-31T19:44:09.902 INFO:tasks.workunit.client.0.smithi167.stdout:4/339: unlink d0/d3f/d1a/f41 0 2022-01-31T19:44:09.904 INFO:tasks.workunit.client.0.smithi167.stdout:4/340: write d0/d6/d7/d39/f43 [3114974,109237] 0 2022-01-31T19:44:09.904 INFO:tasks.workunit.client.0.smithi167.stdout:4/341: chown d0/d5d/l67 2 1 2022-01-31T19:44:09.904 INFO:tasks.workunit.client.0.smithi167.stdout:7/440: symlink d8/d2c/d6e/d4d/d5e/la3 0 2022-01-31T19:44:09.905 INFO:tasks.workunit.client.0.smithi167.stdout:4/342: read d0/d6/d7/d39/f36 [226954,53719] 0 2022-01-31T19:44:09.906 INFO:tasks.workunit.client.0.smithi167.stdout:1/393: write d0/d10/d5a/d33/d4b/d1f/f7b [585437,89399] 0 2022-01-31T19:44:09.906 INFO:tasks.workunit.client.0.smithi167.stdout:1/394: chown d0/d10/d5a/d33/c8c 19433336 1 2022-01-31T19:44:09.906 INFO:tasks.workunit.client.0.smithi167.stdout:4/343: read d0/d6/d7/d39/f36 [18873,83292] 0 2022-01-31T19:44:09.906 INFO:tasks.workunit.client.0.smithi167.stdout:4/344: unlink d0/d3f/d1a/f6a 0 2022-01-31T19:44:09.907 INFO:tasks.workunit.client.0.smithi167.stdout:1/395: creat d0/d75/f8d x:0 0 0 2022-01-31T19:44:09.907 INFO:tasks.workunit.client.0.smithi167.stdout:1/396: readlink d0/d10/l69 0 2022-01-31T19:44:09.909 INFO:tasks.workunit.client.0.smithi167.stdout:4/345: mknod d0/d6/d7/d39/d4e/c6f 0 2022-01-31T19:44:09.909 INFO:tasks.workunit.client.0.smithi167.stdout:4/346: write d0/d6/d7/f30 [1661855,95463] 0 2022-01-31T19:44:09.909 INFO:tasks.workunit.client.0.smithi167.stdout:4/347: fsync d0/d6/d7/d39/d3a/f54 0 2022-01-31T19:44:09.910 INFO:tasks.workunit.client.0.smithi167.stdout:4/348: mknod d0/c70 0 2022-01-31T19:44:09.910 INFO:tasks.workunit.client.0.smithi167.stdout:4/349: write d0/d6/f58 [709749,13572] 0 2022-01-31T19:44:09.912 INFO:tasks.workunit.client.0.smithi167.stdout:4/350: creat d0/d6/d7/d26/d31/f71 x:0 0 0 2022-01-31T19:44:09.912 INFO:tasks.workunit.client.0.smithi167.stdout:4/351: write d0/d6/d7/d26/d31/f71 [813279,31068] 0 2022-01-31T19:44:09.913 INFO:tasks.workunit.client.0.smithi167.stdout:2/285: dread dd/df/d1b/d3d/f44 [0,4194304] 0 2022-01-31T19:44:09.917 INFO:tasks.workunit.client.0.smithi167.stdout:3/354: dwrite d9/d29/f36 [0,4194304] 0 2022-01-31T19:44:09.918 INFO:tasks.workunit.client.0.smithi167.stdout:3/355: chown d9/f30 1267 1 2022-01-31T19:44:09.918 INFO:tasks.workunit.client.0.smithi167.stdout:6/254: dwrite d8/d33/f3c [4194304,4194304] 0 2022-01-31T19:44:09.918 INFO:tasks.workunit.client.0.smithi167.stdout:7/441: dread d8/d2c/d6e/f44 [0,4194304] 0 2022-01-31T19:44:09.918 INFO:tasks.workunit.client.0.smithi167.stdout:7/442: chown d8/d2c/d43/da1 612 1 2022-01-31T19:44:09.920 INFO:tasks.workunit.client.0.smithi167.stdout:0/171: dwrite d17/d19/f2e [0,4194304] 0 2022-01-31T19:44:09.920 INFO:tasks.workunit.client.0.smithi167.stdout:0/172: dread - d17/d19/f1f zero size 2022-01-31T19:44:09.921 INFO:tasks.workunit.client.0.smithi167.stdout:1/397: dread d0/d10/d5a/d33/d4b/d1f/f78 [0,4194304] 0 2022-01-31T19:44:09.921 INFO:tasks.workunit.client.0.smithi167.stdout:2/286: creat dd/df/d1b/f56 x:0 0 0 2022-01-31T19:44:09.921 INFO:tasks.workunit.client.0.smithi167.stdout:2/287: readlink dd/d18/l1f 0 2022-01-31T19:44:09.922 INFO:tasks.workunit.client.0.smithi167.stdout:7/443: write d8/d2c/d6e/d4d/d5e/f84 [1166011,65023] 0 2022-01-31T19:44:09.922 INFO:tasks.workunit.client.0.smithi167.stdout:7/444: readlink d8/d2c/d6c/l95 0 2022-01-31T19:44:09.931 INFO:tasks.workunit.client.0.smithi167.stdout:1/398: write d0/d75/f3d [1534690,90689] 0 2022-01-31T19:44:09.931 INFO:tasks.workunit.client.0.smithi167.stdout:1/399: read - d0/f73 zero size 2022-01-31T19:44:09.931 INFO:tasks.workunit.client.0.smithi167.stdout:7/445: read d8/d2c/f25 [2246107,84521] 0 2022-01-31T19:44:09.932 INFO:tasks.workunit.client.0.smithi167.stdout:0/173: dread fe [0,4194304] 0 2022-01-31T19:44:09.932 INFO:tasks.workunit.client.0.smithi167.stdout:0/174: write d17/d19/d1c/f34 [865586,80486] 0 2022-01-31T19:44:09.933 INFO:tasks.workunit.client.0.smithi167.stdout:1/400: write d0/d75/d7a/f74 [195028,31367] 0 2022-01-31T19:44:09.933 INFO:tasks.workunit.client.0.smithi167.stdout:1/401: dread - d0/d10/f19 zero size 2022-01-31T19:44:09.933 INFO:tasks.workunit.client.0.smithi167.stdout:6/255: rename d8 to d8/d20/d67 22 2022-01-31T19:44:09.944 INFO:tasks.workunit.client.0.smithi167.stdout:6/256: dread d8/d20/f44 [0,4194304] 0 2022-01-31T19:44:09.946 INFO:tasks.workunit.client.0.smithi167.stdout:0/175: mkdir d17/d19/d1c/d39 0 2022-01-31T19:44:09.946 INFO:tasks.workunit.client.0.smithi167.stdout:0/176: write d17/f22 [4757479,30826] 0 2022-01-31T19:44:09.947 INFO:tasks.workunit.client.0.smithi167.stdout:1/402: symlink d0/d75/d7a/l8e 0 2022-01-31T19:44:09.947 INFO:tasks.workunit.client.0.smithi167.stdout:2/288: dwrite dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:09.948 INFO:tasks.workunit.client.0.smithi167.stdout:2/289: read dd/df/f1c [753107,91369] 0 2022-01-31T19:44:09.950 INFO:tasks.workunit.client.0.smithi167.stdout:3/356: rename d9/d29/d44 to d9/d29/d60 0 2022-01-31T19:44:09.953 INFO:tasks.workunit.client.0.smithi167.stdout:6/257: getdents d8/d62 0 2022-01-31T19:44:09.953 INFO:tasks.workunit.client.0.smithi167.stdout:6/258: write d8/f1e [808543,99248] 0 2022-01-31T19:44:09.954 INFO:tasks.workunit.client.0.smithi167.stdout:2/290: dread f8 [0,4194304] 0 2022-01-31T19:44:09.955 INFO:tasks.workunit.client.0.smithi167.stdout:0/177: symlink d17/d19/d1c/l3a 0 2022-01-31T19:44:09.956 INFO:tasks.workunit.client.0.smithi167.stdout:0/178: readlink d17/d19/d1c/l3a 0 2022-01-31T19:44:09.956 INFO:tasks.workunit.client.0.smithi167.stdout:0/179: write d17/f2f [358878,50078] 0 2022-01-31T19:44:09.958 INFO:tasks.workunit.client.0.smithi167.stdout:1/403: mknod d0/d10/d5a/d33/d4b/d1f/c8f 0 2022-01-31T19:44:09.960 INFO:tasks.workunit.client.0.smithi167.stdout:3/357: mknod d9/d56/c61 0 2022-01-31T19:44:09.961 INFO:tasks.workunit.client.0.smithi167.stdout:6/259: getdents d8/d20/d54 0 2022-01-31T19:44:09.962 INFO:tasks.workunit.client.0.smithi167.stdout:6/260: getdents d8/d33/d51/d5d/d64 0 2022-01-31T19:44:09.962 INFO:tasks.workunit.client.0.smithi167.stdout:6/261: write d8/f1e [1952011,83942] 0 2022-01-31T19:44:09.963 INFO:tasks.workunit.client.0.smithi167.stdout:1/404: mknod d0/d10/d5a/d33/d4b/d1f/c90 0 2022-01-31T19:44:09.964 INFO:tasks.workunit.client.0.smithi167.stdout:1/405: creat d0/d10/d18/f91 x:0 0 0 2022-01-31T19:44:09.965 INFO:tasks.workunit.client.0.smithi167.stdout:3/358: symlink d9/d29/d60/l62 0 2022-01-31T19:44:09.966 INFO:tasks.workunit.client.0.smithi167.stdout:6/262: creat d8/d33/d51/d5d/d64/f68 x:0 0 0 2022-01-31T19:44:09.970 INFO:tasks.workunit.client.0.smithi167.stdout:3/359: rename d9/d29/d3b/f5a to d9/d56/f63 0 2022-01-31T19:44:09.979 INFO:tasks.workunit.client.0.smithi167.stdout:2/291: dwrite dd/f55 [0,4194304] 0 2022-01-31T19:44:09.979 INFO:tasks.workunit.client.0.smithi167.stdout:2/292: write dd/df/d1b/f56 [549059,116628] 0 2022-01-31T19:44:09.985 INFO:tasks.workunit.client.0.smithi167.stdout:0/180: dwrite d17/d19/d1c/f32 [0,4194304] 0 2022-01-31T19:44:09.990 INFO:tasks.workunit.client.0.smithi167.stdout:0/181: read fa [1742025,47604] 0 2022-01-31T19:44:09.992 INFO:tasks.workunit.client.0.smithi167.stdout:1/406: dwrite d0/d75/f3d [0,4194304] 0 2022-01-31T19:44:09.993 INFO:tasks.workunit.client.0.smithi167.stdout:1/407: stat d0/d10/d5a/d33/d4b/c2a 0 2022-01-31T19:44:09.993 INFO:tasks.workunit.client.0.smithi167.stdout:1/408: chown d0/d10/d5a/d33/d4b/f5b 29701015 1 2022-01-31T19:44:09.994 INFO:tasks.workunit.client.0.smithi167.stdout:1/409: symlink d0/d11/d45/l92 0 2022-01-31T19:44:09.995 INFO:tasks.workunit.client.0.smithi167.stdout:1/410: rename d0/f86 to d0/f93 0 2022-01-31T19:44:09.996 INFO:tasks.workunit.client.0.smithi167.stdout:1/411: getdents d0/d62/d7f 0 2022-01-31T19:44:09.996 INFO:tasks.workunit.client.0.smithi167.stdout:1/412: rename d0/d10/d5a/d33 to d0/d10/d5a/d33/d4b/d1f/d94 22 2022-01-31T19:44:09.999 INFO:tasks.workunit.client.0.smithi167.stdout:3/360: dwrite d9/d29/f33 [0,4194304] 0 2022-01-31T19:44:10.000 INFO:tasks.workunit.client.0.smithi167.stdout:3/361: symlink d9/d29/d60/l64 0 2022-01-31T19:44:10.001 INFO:tasks.workunit.client.0.smithi167.stdout:3/362: write d9/f40 [930886,2298] 0 2022-01-31T19:44:10.001 INFO:tasks.workunit.client.0.smithi167.stdout:2/293: dwrite dd/f29 [0,4194304] 0 2022-01-31T19:44:10.002 INFO:tasks.workunit.client.0.smithi167.stdout:2/294: symlink dd/df/d3f/d51/l57 0 2022-01-31T19:44:10.005 INFO:tasks.workunit.client.0.smithi167.stdout:3/363: dread d9/f2c [0,4194304] 0 2022-01-31T19:44:10.005 INFO:tasks.workunit.client.0.smithi167.stdout:3/364: fsync d9/d29/f52 0 2022-01-31T19:44:10.008 INFO:tasks.workunit.client.0.smithi167.stdout:3/365: rmdir d9/d29/d2e 39 2022-01-31T19:44:10.009 INFO:tasks.workunit.client.0.smithi167.stdout:1/413: dread d0/f72 [0,4194304] 0 2022-01-31T19:44:10.010 INFO:tasks.workunit.client.0.smithi167.stdout:3/366: rename d9/d21/f3f to d9/d26/f65 0 2022-01-31T19:44:10.011 INFO:tasks.workunit.client.0.smithi167.stdout:1/414: creat d0/d10/d5a/d33/f95 x:0 0 0 2022-01-31T19:44:10.012 INFO:tasks.workunit.client.0.smithi167.stdout:1/415: fdatasync d0/d10/d5a/d33/d4b/d1f/f7b 0 2022-01-31T19:44:10.012 INFO:tasks.workunit.client.0.smithi167.stdout:1/416: fdatasync d0/f72 0 2022-01-31T19:44:10.013 INFO:tasks.workunit.client.0.smithi167.stdout:1/417: symlink d0/l96 0 2022-01-31T19:44:10.014 INFO:tasks.workunit.client.0.smithi167.stdout:3/367: getdents d9/d29 0 2022-01-31T19:44:10.014 INFO:tasks.workunit.client.0.smithi167.stdout:3/368: chown d9/d26/f65 10085744 1 2022-01-31T19:44:10.015 INFO:tasks.workunit.client.0.smithi167.stdout:1/418: mkdir d0/d10/d5a/d33/d97 0 2022-01-31T19:44:10.018 INFO:tasks.workunit.client.0.smithi167.stdout:3/369: creat d9/d29/d2e/f66 x:0 0 0 2022-01-31T19:44:10.019 INFO:tasks.workunit.client.0.smithi167.stdout:3/370: unlink d9/f13 0 2022-01-31T19:44:10.019 INFO:tasks.workunit.client.0.smithi167.stdout:3/371: chown d9/d21/l53 0 1 2022-01-31T19:44:10.024 INFO:tasks.workunit.client.0.smithi167.stdout:2/295: dwrite dd/df/d1b/f56 [0,4194304] 0 2022-01-31T19:44:10.027 INFO:tasks.workunit.client.0.smithi167.stdout:2/296: dread f9 [4194304,4194304] 0 2022-01-31T19:44:10.043 INFO:tasks.workunit.client.0.smithi167.stdout:1/419: dwrite d0/f25 [0,4194304] 0 2022-01-31T19:44:10.043 INFO:tasks.workunit.client.0.smithi167.stdout:1/420: write d0/f1 [5215880,538] 0 2022-01-31T19:44:10.045 INFO:tasks.workunit.client.0.smithi167.stdout:3/372: dwrite d9/d29/d2e/f3e [0,4194304] 0 2022-01-31T19:44:10.048 INFO:tasks.workunit.client.0.smithi167.stdout:1/421: getdents d0/d75 0 2022-01-31T19:44:10.048 INFO:tasks.workunit.client.0.smithi167.stdout:1/422: write d0/f17 [1849112,91351] 0 2022-01-31T19:44:10.048 INFO:tasks.workunit.client.0.smithi167.stdout:1/423: getdents d0/d75 0 2022-01-31T19:44:10.048 INFO:tasks.workunit.client.0.smithi167.stdout:2/297: dwrite dd/d18/f30 [0,4194304] 0 2022-01-31T19:44:10.049 INFO:tasks.workunit.client.0.smithi167.stdout:1/424: chown d0/d10/d5a/d33/d4b/c43 328286622 1 2022-01-31T19:44:10.049 INFO:tasks.workunit.client.0.smithi167.stdout:1/425: fdatasync d0/f1b 0 2022-01-31T19:44:10.050 INFO:tasks.workunit.client.0.smithi167.stdout:1/426: chown d0/d10/d18/f80 5 1 2022-01-31T19:44:10.051 INFO:tasks.workunit.client.0.smithi167.stdout:1/427: creat d0/d10/d5a/d33/d4b/f98 x:0 0 0 2022-01-31T19:44:10.052 INFO:tasks.workunit.client.0.smithi167.stdout:1/428: rmdir d0/d11/d45 39 2022-01-31T19:44:10.054 INFO:tasks.workunit.client.0.smithi167.stdout:3/373: dread d9/f25 [0,4194304] 0 2022-01-31T19:44:10.054 INFO:tasks.workunit.client.0.smithi167.stdout:1/429: truncate d0/d10/d5a/d33/d4b/d1f/f7b 262678 0 2022-01-31T19:44:10.055 INFO:tasks.workunit.client.0.smithi167.stdout:1/430: creat d0/d11/d46/f99 x:0 0 0 2022-01-31T19:44:10.057 INFO:tasks.workunit.client.0.smithi167.stdout:3/374: read d9/f10 [331748,70262] 0 2022-01-31T19:44:10.058 INFO:tasks.workunit.client.0.smithi167.stdout:3/375: write d9/d29/d2e/f66 [888323,109993] 0 2022-01-31T19:44:10.061 INFO:tasks.workunit.client.0.smithi167.stdout:1/431: mknod d0/d11/d46/c9a 0 2022-01-31T19:44:10.061 INFO:tasks.workunit.client.0.smithi167.stdout:1/432: fdatasync d0/d10/d5a/d33/d4b/f98 0 2022-01-31T19:44:10.063 INFO:tasks.workunit.client.0.smithi167.stdout:3/376: dread d9/fc [0,4194304] 0 2022-01-31T19:44:10.064 INFO:tasks.workunit.client.0.smithi167.stdout:3/377: creat d9/d29/d2e/f67 x:0 0 0 2022-01-31T19:44:10.067 INFO:tasks.workunit.client.0.smithi167.stdout:3/378: creat d9/d29/d60/f68 x:0 0 0 2022-01-31T19:44:10.086 INFO:tasks.workunit.client.0.smithi167.stdout:1/433: dwrite d0/f93 [0,4194304] 0 2022-01-31T19:44:10.087 INFO:tasks.workunit.client.0.smithi167.stdout:3/379: dwrite d9/d29/d60/f68 [0,4194304] 0 2022-01-31T19:44:10.087 INFO:tasks.workunit.client.0.smithi167.stdout:3/380: write d9/d29/d2e/f51 [514754,50519] 0 2022-01-31T19:44:10.089 INFO:tasks.workunit.client.0.smithi167.stdout:1/434: symlink d0/d10/d5a/d33/d4b/d1f/l9b 0 2022-01-31T19:44:10.089 INFO:tasks.workunit.client.0.smithi167.stdout:1/435: chown d0/d10/d5a/d33/d4b 0 1 2022-01-31T19:44:10.093 INFO:tasks.workunit.client.0.smithi167.stdout:1/436: mknod d0/d10/d5a/d33/d4b/c9c 0 2022-01-31T19:44:10.094 INFO:tasks.workunit.client.0.smithi167.stdout:1/437: write d0/d10/d20/f4e [4791813,30878] 0 2022-01-31T19:44:10.107 INFO:tasks.workunit.client.0.smithi167.stdout:1/438: dread d0/d10/d5a/f50 [0,4194304] 0 2022-01-31T19:44:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:09 smithi171 conmon[35325]: cluster 2022-01-31T19:44:08.431413+0000 mgr.smithi171.asyxnc (mgr.24387) 14 2022-01-31T19:44:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:09 smithi171 conmon[35325]: : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 525 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 16 MiB/s rd, 39 MiB/s wr, 525 op/s 2022-01-31T19:44:10.108 INFO:tasks.workunit.client.0.smithi167.stdout:1/439: symlink d0/d10/d5a/d33/d97/l9d 0 2022-01-31T19:44:10.117 INFO:tasks.workunit.client.0.smithi167.stdout:3/381: dwrite d9/d29/d2e/f51 [0,4194304] 0 2022-01-31T19:44:10.124 INFO:tasks.workunit.client.0.smithi167.stdout:1/440: dwrite d0/d10/d20/f56 [0,4194304] 0 2022-01-31T19:44:10.127 INFO:tasks.workunit.client.0.smithi167.stdout:1/441: mknod d0/c9e 0 2022-01-31T19:44:10.127 INFO:tasks.workunit.client.0.smithi167.stdout:1/442: write d0/d10/d18/f91 [53626,35111] 0 2022-01-31T19:44:10.129 INFO:tasks.workunit.client.0.smithi167.stdout:1/443: link d0/d10/d5a/d33/d4b/d1f/c90 d0/d10/d5a/d33/d97/c9f 0 2022-01-31T19:44:10.129 INFO:tasks.workunit.client.0.smithi167.stdout:1/444: read - d0/d10/d5a/d33/d4b/f98 zero size 2022-01-31T19:44:10.129 INFO:tasks.workunit.client.0.smithi167.stdout:1/445: fdatasync d0/d10/d18/f80 0 2022-01-31T19:44:10.130 INFO:tasks.workunit.client.0.smithi167.stdout:1/446: creat d0/d11/d45/fa0 x:0 0 0 2022-01-31T19:44:10.134 INFO:tasks.workunit.client.0.smithi167.stdout:1/447: rename d0/d10/d18/f80 to d0/d10/d5a/d33/d4b/d1f/fa1 0 2022-01-31T19:44:10.154 INFO:tasks.workunit.client.0.smithi167.stdout:1/448: dwrite d0/f30 [0,4194304] 0 2022-01-31T19:44:10.156 INFO:tasks.workunit.client.0.smithi167.stdout:1/449: creat d0/d10/d5a/d33/d4b/fa2 x:0 0 0 2022-01-31T19:44:10.157 INFO:tasks.workunit.client.0.smithi167.stdout:1/450: dread d0/d10/d18/f91 [0,4194304] 0 2022-01-31T19:44:10.157 INFO:tasks.workunit.client.0.smithi167.stdout:1/451: stat d0/d11/d46/c58 0 2022-01-31T19:44:10.157 INFO:tasks.workunit.client.0.smithi167.stdout:1/452: stat d0/d10/d20/f37 0 2022-01-31T19:44:10.158 INFO:tasks.workunit.client.0.smithi167.stdout:1/453: truncate d0/f72 2624688 0 2022-01-31T19:44:10.158 INFO:tasks.workunit.client.0.smithi167.stdout:1/454: fsync d0/d75/d7a/f74 0 2022-01-31T19:44:10.161 INFO:tasks.workunit.client.0.smithi167.stdout:1/455: unlink d0/d10/d5a/d33/l36 0 2022-01-31T19:44:10.162 INFO:tasks.workunit.client.0.smithi167.stdout:1/456: creat d0/d62/fa3 x:0 0 0 2022-01-31T19:44:10.163 INFO:tasks.workunit.client.0.smithi167.stdout:1/457: chown d0/d11/d46/f99 22 1 2022-01-31T19:44:10.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:09 smithi167 conmon[32206]: cluster 2022-01-31T19:44:08.431413+0000 mgr.smithi171.asyxnc (mgr.24387) 14 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 525 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 16 MiB/s rd, 39 MiB/s wr, 525 op/s 2022-01-31T19:44:10.172 INFO:tasks.workunit.client.0.smithi167.stdout:4/352: sync 2022-01-31T19:44:10.172 INFO:tasks.workunit.client.0.smithi167.stdout:8/433: sync 2022-01-31T19:44:10.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/458: sync 2022-01-31T19:44:10.172 INFO:tasks.workunit.client.0.smithi167.stdout:9/230: sync 2022-01-31T19:44:10.173 INFO:tasks.workunit.client.0.smithi167.stdout:8/434: chown d6/db/de/d18/l1f 81463893 1 2022-01-31T19:44:10.173 INFO:tasks.workunit.client.0.smithi167.stdout:8/435: creat d6/db/d28/d66/f8e x:0 0 0 2022-01-31T19:44:10.174 INFO:tasks.workunit.client.0.smithi167.stdout:4/353: dread d0/d6/d7/d26/d31/f71 [0,4194304] 0 2022-01-31T19:44:10.175 INFO:tasks.workunit.client.0.smithi167.stdout:5/459: rename d5/d6/d3a/l11 to d5/d6/de/l93 0 2022-01-31T19:44:10.180 INFO:tasks.workunit.client.0.smithi167.stdout:4/354: truncate d0/d6/d7/d26/d31/f19 3988590 0 2022-01-31T19:44:10.180 INFO:tasks.workunit.client.0.smithi167.stdout:4/355: fdatasync d0/d6/f3b 0 2022-01-31T19:44:10.180 INFO:tasks.workunit.client.0.smithi167.stdout:4/356: chown d0/d6/d7/d26/d31/l10 100091329 1 2022-01-31T19:44:10.181 INFO:tasks.workunit.client.0.smithi167.stdout:4/357: rmdir d0/d6 39 2022-01-31T19:44:10.181 INFO:tasks.workunit.client.0.smithi167.stdout:4/358: write d0/d6/d7/d39/f36 [1557911,68544] 0 2022-01-31T19:44:10.182 INFO:tasks.workunit.client.0.smithi167.stdout:4/359: write d0/d6/d7/d39/d3a/f48 [923301,82054] 0 2022-01-31T19:44:10.183 INFO:tasks.workunit.client.0.smithi167.stdout:1/458: dwrite d0/d10/d5a/d33/d4b/f68 [8388608,4194304] 0 2022-01-31T19:44:10.192 INFO:tasks.workunit.client.0.smithi167.stdout:1/459: read d0/f25 [7699,130273] 0 2022-01-31T19:44:10.193 INFO:tasks.workunit.client.0.smithi167.stdout:9/231: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:10.195 INFO:tasks.workunit.client.0.smithi167.stdout:9/232: unlink dd/d17/f38 0 2022-01-31T19:44:10.195 INFO:tasks.workunit.client.0.smithi167.stdout:9/233: chown dd/c40 101909 1 2022-01-31T19:44:10.196 INFO:tasks.workunit.client.0.smithi167.stdout:9/234: truncate dd/f16 3739598 0 2022-01-31T19:44:10.196 INFO:tasks.workunit.client.0.smithi167.stdout:9/235: chown dd/f10 244267 1 2022-01-31T19:44:10.198 INFO:tasks.workunit.client.0.smithi167.stdout:1/460: write d0/d10/d18/f3e [964429,43319] 0 2022-01-31T19:44:10.210 INFO:tasks.workunit.client.0.smithi167.stdout:8/436: dwrite d6/db/de/d18/f1b [0,4194304] 0 2022-01-31T19:44:10.214 INFO:tasks.workunit.client.0.smithi167.stdout:5/460: dwrite d5/d6/de/d22/f54 [0,4194304] 0 2022-01-31T19:44:10.215 INFO:tasks.workunit.client.0.smithi167.stdout:4/360: dwrite d0/d6/d7/d39/f37 [0,4194304] 0 2022-01-31T19:44:10.215 INFO:tasks.workunit.client.0.smithi167.stdout:4/361: creat d0/d6/d7/d26/f72 x:0 0 0 2022-01-31T19:44:10.215 INFO:tasks.workunit.client.0.smithi167.stdout:4/362: chown d0/d6/d7/f30 52020 1 2022-01-31T19:44:10.216 INFO:tasks.workunit.client.0.smithi167.stdout:9/236: dwrite dd/d35/f37 [4194304,4194304] 0 2022-01-31T19:44:10.216 INFO:tasks.workunit.client.0.smithi167.stdout:9/237: chown dd/c29 1 1 2022-01-31T19:44:10.229 INFO:tasks.workunit.client.0.smithi167.stdout:5/461: dwrite d5/d61/f74 [0,4194304] 0 2022-01-31T19:44:10.292 INFO:tasks.workunit.client.0.smithi167.stdout:1/461: link d0/d10/d5a/d33/d97/c9f d0/d75/d7a/ca4 0 2022-01-31T19:44:10.295 INFO:tasks.workunit.client.0.smithi167.stdout:8/437: rename d6/db/d36/f4a to d6/d1e/d4c/d80/f8f 0 2022-01-31T19:44:10.295 INFO:tasks.workunit.client.0.smithi167.stdout:8/438: chown d6/db/d2f/d51/d60/f79 36 1 2022-01-31T19:44:10.295 INFO:tasks.workunit.client.0.smithi167.stdout:8/439: creat d6/db/d2f/f90 x:0 0 0 2022-01-31T19:44:10.338 INFO:tasks.workunit.client.0.smithi167.stdout:4/363: getdents d0/d6 0 2022-01-31T19:44:10.339 INFO:tasks.workunit.client.0.smithi167.stdout:9/238: mknod dd/d19/d24/c4b 0 2022-01-31T19:44:10.343 INFO:tasks.workunit.client.0.smithi167.stdout:5/462: creat d5/d6/d16/d18/d43/d45/f94 x:0 0 0 2022-01-31T19:44:10.345 INFO:tasks.workunit.client.0.smithi167.stdout:5/463: dread d5/d6/d3a/f75 [0,4194304] 0 2022-01-31T19:44:10.346 INFO:tasks.workunit.client.0.smithi167.stdout:5/464: stat d5/d6/de/d22/f25 0 2022-01-31T19:44:10.349 INFO:tasks.workunit.client.0.smithi167.stdout:8/440: symlink d6/d1e/d2d/d6b/l91 0 2022-01-31T19:44:10.349 INFO:tasks.workunit.client.0.smithi167.stdout:8/441: dread - d6/db/d28/d61/f89 zero size 2022-01-31T19:44:10.351 INFO:tasks.workunit.client.0.smithi167.stdout:4/364: creat d0/f73 x:0 0 0 2022-01-31T19:44:10.352 INFO:tasks.workunit.client.0.smithi167.stdout:9/239: mkdir dd/d19/d24/d3b/d4c 0 2022-01-31T19:44:10.352 INFO:tasks.workunit.client.0.smithi167.stdout:4/365: read d0/d6/d7/d26/d40/f27 [705830,123998] 0 2022-01-31T19:44:10.355 INFO:tasks.workunit.client.0.smithi167.stdout:4/366: dread d0/d6/d7/d26/d40/f27 [0,4194304] 0 2022-01-31T19:44:10.358 INFO:tasks.workunit.client.0.smithi167.stdout:8/442: rename d6/db/d4e/d73/c8a to d6/db/d4e/d73/c92 0 2022-01-31T19:44:10.360 INFO:tasks.workunit.client.0.smithi167.stdout:4/367: mknod d0/d6/d7/d39/d3a/c74 0 2022-01-31T19:44:10.362 INFO:tasks.workunit.client.0.smithi167.stdout:1/462: rename d0/d11/d45 to d0/d10/d5a/d33/d4b/da5 0 2022-01-31T19:44:10.363 INFO:tasks.workunit.client.0.smithi167.stdout:4/368: mkdir d0/d3f/d75 0 2022-01-31T19:44:10.363 INFO:tasks.workunit.client.0.smithi167.stdout:4/369: write d0/d6/d7/d39/d3a/f55 [828029,1820] 0 2022-01-31T19:44:10.363 INFO:tasks.workunit.client.0.smithi167.stdout:4/370: write d0/d3f/f2e [1664684,14516] 0 2022-01-31T19:44:10.364 INFO:tasks.workunit.client.0.smithi167.stdout:5/465: dwrite d5/d6/d3a/f4e [0,4194304] 0 2022-01-31T19:44:10.365 INFO:tasks.workunit.client.0.smithi167.stdout:8/443: write d6/db/d36/f3b [582744,73898] 0 2022-01-31T19:44:10.365 INFO:tasks.workunit.client.0.smithi167.stdout:8/444: fsync d6/f57 0 2022-01-31T19:44:10.365 INFO:tasks.workunit.client.0.smithi167.stdout:8/445: chown d6/db/d36/d5d/f63 31640 1 2022-01-31T19:44:10.366 INFO:tasks.workunit.client.0.smithi167.stdout:9/240: rename dd/c15 to dd/d34/c4d 0 2022-01-31T19:44:10.366 INFO:tasks.workunit.client.0.smithi167.stdout:9/241: read dd/d17/f1f [63598,49094] 0 2022-01-31T19:44:10.366 INFO:tasks.workunit.client.0.smithi167.stdout:1/463: creat d0/d10/d5a/d33/d4b/d1f/fa6 x:0 0 0 2022-01-31T19:44:10.367 INFO:tasks.workunit.client.0.smithi167.stdout:1/464: write d0/f25 [328599,28199] 0 2022-01-31T19:44:10.371 INFO:tasks.workunit.client.0.smithi167.stdout:8/446: link d6/db/d6a/c75 d6/db/c93 0 2022-01-31T19:44:10.372 INFO:tasks.workunit.client.0.smithi167.stdout:9/242: mknod dd/d19/c4e 0 2022-01-31T19:44:10.372 INFO:tasks.workunit.client.0.smithi167.stdout:4/371: dread d0/d3f/f29 [0,4194304] 0 2022-01-31T19:44:10.373 INFO:tasks.workunit.client.0.smithi167.stdout:4/372: write d0/d3f/f2e [1759363,53557] 0 2022-01-31T19:44:10.374 INFO:tasks.workunit.client.0.smithi167.stdout:7/446: sync 2022-01-31T19:44:10.375 INFO:tasks.workunit.client.0.smithi167.stdout:7/447: chown d8/d2c/d6e/l30 28628 1 2022-01-31T19:44:10.375 INFO:tasks.workunit.client.0.smithi167.stdout:7/448: chown d8/d2c/d6e/l41 226508 1 2022-01-31T19:44:10.375 INFO:tasks.workunit.client.0.smithi167.stdout:5/466: rmdir d5/d6/d16 39 2022-01-31T19:44:10.379 INFO:tasks.workunit.client.0.smithi167.stdout:8/447: truncate d6/db/d28/f43 2192049 0 2022-01-31T19:44:10.381 INFO:tasks.workunit.client.0.smithi167.stdout:9/243: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:10.382 INFO:tasks.workunit.client.0.smithi167.stdout:9/244: dread - dd/d35/f44 zero size 2022-01-31T19:44:10.382 INFO:tasks.workunit.client.0.smithi167.stdout:9/245: stat dd/d34/l39 0 2022-01-31T19:44:10.383 INFO:tasks.workunit.client.0.smithi167.stdout:4/373: unlink d0/d6/d7/d39/c2d 0 2022-01-31T19:44:10.383 INFO:tasks.workunit.client.0.smithi167.stdout:4/374: chown d0/d3f/d60 249 1 2022-01-31T19:44:10.383 INFO:tasks.workunit.client.0.smithi167.stdout:7/449: link d8/dc/d18/d24/c70 d8/d2c/d6e/d4d/d6b/ca4 0 2022-01-31T19:44:10.384 INFO:tasks.workunit.client.0.smithi167.stdout:7/450: stat d8/d2c/d6e/d4d/d67/f7e 0 2022-01-31T19:44:10.384 INFO:tasks.workunit.client.0.smithi167.stdout:5/467: rename d5/d6/de/f2a to d5/d6/de/d22/d76/f95 0 2022-01-31T19:44:10.386 INFO:tasks.workunit.client.0.smithi167.stdout:1/465: dwrite d0/d10/d5a/f50 [0,4194304] 0 2022-01-31T19:44:10.387 INFO:tasks.workunit.client.0.smithi167.stdout:8/448: getdents d6 0 2022-01-31T19:44:10.387 INFO:tasks.workunit.client.0.smithi167.stdout:8/449: creat d6/db/d28/f94 x:0 0 0 2022-01-31T19:44:10.388 INFO:tasks.workunit.client.0.smithi167.stdout:9/246: unlink dd/d19/d24/f31 0 2022-01-31T19:44:10.389 INFO:tasks.workunit.client.0.smithi167.stdout:4/375: creat d0/d6/f76 x:0 0 0 2022-01-31T19:44:10.389 INFO:tasks.workunit.client.0.smithi167.stdout:7/451: stat d8/d2c/d43/d31/d8d/l3a 0 2022-01-31T19:44:10.391 INFO:tasks.workunit.client.0.smithi167.stdout:5/468: rmdir d5/d6/de/d22/d6d 39 2022-01-31T19:44:10.391 INFO:tasks.workunit.client.0.smithi167.stdout:5/469: write d5/d6/d16/d18/f8d [435174,10284] 0 2022-01-31T19:44:10.391 INFO:tasks.workunit.client.0.smithi167.stdout:4/376: write d0/d6/d7/f8 [444999,126917] 0 2022-01-31T19:44:10.391 INFO:tasks.workunit.client.0.smithi167.stdout:4/377: truncate d0/d6/d7/d39/d3a/f55 1273381 0 2022-01-31T19:44:10.393 INFO:tasks.workunit.client.0.smithi167.stdout:1/466: write d0/d10/d5a/d33/d4b/f6d [355546,76529] 0 2022-01-31T19:44:10.393 INFO:tasks.workunit.client.0.smithi167.stdout:1/467: creat d0/d10/d5a/d33/fa7 x:0 0 0 2022-01-31T19:44:10.393 INFO:tasks.workunit.client.0.smithi167.stdout:1/468: dread - d0/d62/fa3 zero size 2022-01-31T19:44:10.394 INFO:tasks.workunit.client.0.smithi167.stdout:1/469: write d0/d10/d5a/d33/d4b/d1f/f7b [1243986,75858] 0 2022-01-31T19:44:10.394 INFO:tasks.workunit.client.0.smithi167.stdout:8/450: mknod d6/d1e/d2d/d6b/c95 0 2022-01-31T19:44:10.395 INFO:tasks.workunit.client.0.smithi167.stdout:5/470: getdents d5/d6/d16 0 2022-01-31T19:44:10.397 INFO:tasks.workunit.client.0.smithi167.stdout:4/378: rename d0/f73 to d0/d3f/d60/f77 0 2022-01-31T19:44:10.399 INFO:tasks.workunit.client.0.smithi167.stdout:9/247: dread dd/d19/f3d [0,4194304] 0 2022-01-31T19:44:10.403 INFO:tasks.workunit.client.0.smithi167.stdout:5/471: mknod d5/d6/d16/d18/d43/d45/d69/d90/c96 0 2022-01-31T19:44:10.404 INFO:tasks.workunit.client.0.smithi167.stdout:4/379: link d0/d6/d7/d26/d40/c2c d0/d6/d7/d39/c78 0 2022-01-31T19:44:10.407 INFO:tasks.workunit.client.0.smithi167.stdout:1/470: dread d0/d10/d5a/f50 [0,4194304] 0 2022-01-31T19:44:10.409 INFO:tasks.workunit.client.0.smithi167.stdout:4/380: dread d0/d6/d7/d26/d40/f24 [0,4194304] 0 2022-01-31T19:44:10.410 INFO:tasks.workunit.client.0.smithi167.stdout:7/452: dwrite d8/d2c/d6e/d4d/d67/d8f/f96 [0,4194304] 0 2022-01-31T19:44:10.415 INFO:tasks.workunit.client.0.smithi167.stdout:5/472: dread d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:10.418 INFO:tasks.workunit.client.0.smithi167.stdout:8/451: dwrite d6/db/d4e/d73/f7b [0,4194304] 0 2022-01-31T19:44:10.418 INFO:tasks.workunit.client.0.smithi167.stdout:8/452: truncate d6/db/d36/d5d/f68 1357662 0 2022-01-31T19:44:10.418 INFO:tasks.workunit.client.0.smithi167.stdout:8/453: chown d6/db/d2f/d51/d60/f79 6325 1 2022-01-31T19:44:10.453 INFO:tasks.workunit.client.0.smithi167.stdout:1/471: unlink d0/d75/f3d 0 2022-01-31T19:44:10.453 INFO:tasks.workunit.client.0.smithi167.stdout:4/381: mkdir d0/d6/d7/d39/d3a/d79 0 2022-01-31T19:44:10.454 INFO:tasks.workunit.client.0.smithi167.stdout:4/382: truncate d0/d6/d7/d26/d40/f24 1106903 0 2022-01-31T19:44:10.454 INFO:tasks.workunit.client.0.smithi167.stdout:4/383: creat d0/f7a x:0 0 0 2022-01-31T19:44:10.455 INFO:tasks.workunit.client.0.smithi167.stdout:7/453: rename d8/d2c/d43/d31/d8d/c80 to d8/d2c/d43/da1/ca5 0 2022-01-31T19:44:10.457 INFO:tasks.workunit.client.0.smithi167.stdout:5/473: creat d5/d6/de/d22/f97 x:0 0 0 2022-01-31T19:44:10.458 INFO:tasks.workunit.client.0.smithi167.stdout:5/474: write d5/f7 [5965346,12524] 0 2022-01-31T19:44:10.460 INFO:tasks.workunit.client.0.smithi167.stdout:1/472: mknod d0/d10/d20/ca8 0 2022-01-31T19:44:10.462 INFO:tasks.workunit.client.0.smithi167.stdout:4/384: creat d0/d6/d7/f7b x:0 0 0 2022-01-31T19:44:10.462 INFO:tasks.workunit.client.0.smithi167.stdout:4/385: dread - d0/d3f/d60/f77 zero size 2022-01-31T19:44:10.463 INFO:tasks.workunit.client.0.smithi167.stdout:7/454: creat d8/d2c/d43/fa6 x:0 0 0 2022-01-31T19:44:10.466 INFO:tasks.workunit.client.0.smithi167.stdout:5/475: creat d5/d6/d16/d18/d5b/d63/f98 x:0 0 0 2022-01-31T19:44:10.466 INFO:tasks.workunit.client.0.smithi167.stdout:5/476: stat d5/d6/de/d22/d4d/l7a 0 2022-01-31T19:44:10.467 INFO:tasks.workunit.client.0.smithi167.stdout:5/477: chown d5/d6/cd 6 1 2022-01-31T19:44:10.470 INFO:tasks.workunit.client.0.smithi167.stdout:1/473: unlink d0/d10/d5a/d33/d97/c9f 0 2022-01-31T19:44:10.474 INFO:tasks.workunit.client.0.smithi167.stdout:5/478: mknod d5/d6/d16/d18/c99 0 2022-01-31T19:44:10.475 INFO:tasks.workunit.client.0.smithi167.stdout:4/386: dwrite d0/d6/d7/f18 [0,4194304] 0 2022-01-31T19:44:10.475 INFO:tasks.workunit.client.0.smithi167.stdout:4/387: chown d0/d6/d7/d39/f43 1524 1 2022-01-31T19:44:10.480 INFO:tasks.workunit.client.0.smithi167.stdout:6/263: sync 2022-01-31T19:44:10.480 INFO:tasks.workunit.client.0.smithi167.stdout:0/182: sync 2022-01-31T19:44:10.481 INFO:tasks.workunit.client.0.smithi167.stdout:7/455: truncate d8/d2c/d6e/d4d/d67/d8f/f96 2062872 0 2022-01-31T19:44:10.484 INFO:tasks.workunit.client.0.smithi167.stdout:4/388: rename d0/l1 to d0/d6/l7c 0 2022-01-31T19:44:10.486 INFO:tasks.workunit.client.0.smithi167.stdout:1/474: unlink d0/d10/d5a/d33/d4b/f57 0 2022-01-31T19:44:10.486 INFO:tasks.workunit.client.0.smithi167.stdout:1/475: chown d0/d10/d5a/d33/c8c 24 1 2022-01-31T19:44:10.487 INFO:tasks.workunit.client.0.smithi167.stdout:6/264: symlink d8/d33/d51/l69 0 2022-01-31T19:44:10.487 INFO:tasks.workunit.client.0.smithi167.stdout:6/265: dread - d8/d33/d51/d5d/f60 zero size 2022-01-31T19:44:10.488 INFO:tasks.workunit.client.0.smithi167.stdout:6/266: getdents d8/d33/d51/d5d 0 2022-01-31T19:44:10.493 INFO:tasks.workunit.client.0.smithi167.stdout:5/479: dwrite d5/d6/d16/f7e [0,4194304] 0 2022-01-31T19:44:10.493 INFO:tasks.workunit.client.0.smithi167.stdout:5/480: fsync d5/d6/d16/d18/f8d 0 2022-01-31T19:44:10.494 INFO:tasks.workunit.client.0.smithi167.stdout:5/481: mknod d5/d6/de/d22/d6d/c9a 0 2022-01-31T19:44:10.495 INFO:tasks.workunit.client.0.smithi167.stdout:5/482: symlink d5/d6/d16/d18/d43/l9b 0 2022-01-31T19:44:10.495 INFO:tasks.workunit.client.0.smithi167.stdout:5/483: dread - d5/d6/d16/d18/f33 zero size 2022-01-31T19:44:10.496 INFO:tasks.workunit.client.0.smithi167.stdout:5/484: mknod d5/d61/c9c 0 2022-01-31T19:44:10.497 INFO:tasks.workunit.client.0.smithi167.stdout:5/485: creat d5/d6/d16/f9d x:0 0 0 2022-01-31T19:44:10.498 INFO:tasks.workunit.client.0.smithi167.stdout:5/486: truncate d5/d6/de/d22/d4d/f87 292983 0 2022-01-31T19:44:10.504 INFO:tasks.workunit.client.0.smithi167.stdout:7/456: dwrite d8/dc/f40 [0,4194304] 0 2022-01-31T19:44:10.505 INFO:tasks.workunit.client.0.smithi167.stdout:7/457: fdatasync d8/d35/f5d 0 2022-01-31T19:44:10.507 INFO:tasks.workunit.client.0.smithi167.stdout:7/458: rmdir d8/d2c/d43/da1 39 2022-01-31T19:44:10.508 INFO:tasks.workunit.client.0.smithi167.stdout:7/459: write d8/d2c/f25 [1240026,69293] 0 2022-01-31T19:44:10.510 INFO:tasks.workunit.client.0.smithi167.stdout:7/460: unlink d8/dc/c65 0 2022-01-31T19:44:10.511 INFO:tasks.workunit.client.0.smithi167.stdout:7/461: readlink d8/d2c/d6e/l20 0 2022-01-31T19:44:10.512 INFO:tasks.workunit.client.0.smithi167.stdout:5/487: dread d5/d61/f74 [0,4194304] 0 2022-01-31T19:44:10.512 INFO:tasks.workunit.client.0.smithi167.stdout:5/488: chown d5/d6/de/d22/d76/f80 10120926 1 2022-01-31T19:44:10.513 INFO:tasks.workunit.client.0.smithi167.stdout:5/489: fsync d5/d6/d16/d18/f3d 0 2022-01-31T19:44:10.513 INFO:tasks.workunit.client.0.smithi167.stdout:5/490: creat d5/d6/de/d22/d4d/f9e x:0 0 0 2022-01-31T19:44:10.513 INFO:tasks.workunit.client.0.smithi167.stdout:5/491: stat d5/d6/de/d22/c49 0 2022-01-31T19:44:10.513 INFO:tasks.workunit.client.0.smithi167.stdout:7/462: read d8/dc/f23 [565857,41489] 0 2022-01-31T19:44:10.514 INFO:tasks.workunit.client.0.smithi167.stdout:7/463: write d8/d35/f5d [1226096,107932] 0 2022-01-31T19:44:10.519 INFO:tasks.workunit.client.0.smithi167.stdout:5/492: dread d5/d6/de/f3e [0,4194304] 0 2022-01-31T19:44:10.520 INFO:tasks.workunit.client.0.smithi167.stdout:5/493: write d5/d6/de/d22/d6d/f84 [851618,33620] 0 2022-01-31T19:44:10.520 INFO:tasks.workunit.client.0.smithi167.stdout:5/494: mkdir d5/d6/d16/d18/d5b/d9f 0 2022-01-31T19:44:10.526 INFO:tasks.workunit.client.0.smithi167.stdout:7/464: dwrite d8/d2c/d6e/f2a [0,4194304] 0 2022-01-31T19:44:10.541 INFO:tasks.workunit.client.0.smithi167.stdout:5/495: dwrite d5/d6/de/d22/d4d/d65/f7c [4194304,4194304] 0 2022-01-31T19:44:10.541 INFO:tasks.workunit.client.0.smithi167.stdout:7/465: dread d8/d2c/d6c/f6d [0,4194304] 0 2022-01-31T19:44:10.541 INFO:tasks.workunit.client.0.smithi167.stdout:5/496: fsync d5/d6/d16/f26 0 2022-01-31T19:44:10.542 INFO:tasks.workunit.client.0.smithi167.stdout:5/497: mkdir d5/d6/de/d22/d6d/da0 0 2022-01-31T19:44:10.561 INFO:tasks.workunit.client.0.smithi167.stdout:7/466: dwrite d8/d2c/d43/d31/d8d/f5c [0,4194304] 0 2022-01-31T19:44:10.562 INFO:tasks.workunit.client.0.smithi167.stdout:5/498: dwrite d5/d6/de/d22/d4d/d65/f79 [0,4194304] 0 2022-01-31T19:44:10.565 INFO:tasks.workunit.client.0.smithi167.stdout:5/499: dread d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:10.582 INFO:tasks.workunit.client.0.smithi167.stdout:5/500: dwrite d5/d6/d3a/f8e [0,4194304] 0 2022-01-31T19:44:10.582 INFO:tasks.workunit.client.0.smithi167.stdout:7/467: dwrite d8/d35/f3d [0,4194304] 0 2022-01-31T19:44:10.588 INFO:tasks.workunit.client.0.smithi167.stdout:5/501: dread d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:10.589 INFO:tasks.workunit.client.0.smithi167.stdout:7/468: creat d8/fa7 x:0 0 0 2022-01-31T19:44:10.589 INFO:tasks.workunit.client.0.smithi167.stdout:7/469: fdatasync d8/d2c/d6e/d4d/d67/d8f/f96 0 2022-01-31T19:44:10.590 INFO:tasks.workunit.client.0.smithi167.stdout:5/502: link d5/f7 d5/d6/de/d22/d4d/fa1 0 2022-01-31T19:44:10.591 INFO:tasks.workunit.client.0.smithi167.stdout:5/503: chown d5/d6/de/d22/d4d/f9e 0 1 2022-01-31T19:44:10.591 INFO:tasks.workunit.client.0.smithi167.stdout:5/504: write d5/d6/d16/d18/d43/d45/f60 [5155579,86092] 0 2022-01-31T19:44:10.591 INFO:tasks.workunit.client.0.smithi167.stdout:7/470: write d8/d35/f36 [332666,74904] 0 2022-01-31T19:44:10.593 INFO:tasks.workunit.client.0.smithi167.stdout:5/505: rename l0 to d5/d6/de/d22/d76/la2 0 2022-01-31T19:44:10.594 INFO:tasks.workunit.client.0.smithi167.stdout:5/506: write d5/d6/de/d22/d4d/f85 [125047,33944] 0 2022-01-31T19:44:10.595 INFO:tasks.workunit.client.0.smithi167.stdout:5/507: stat d5/d6/de/d22/d6d/f84 0 2022-01-31T19:44:10.595 INFO:tasks.workunit.client.0.smithi167.stdout:5/508: fdatasync d5/d6/de/d22/f35 0 2022-01-31T19:44:10.595 INFO:tasks.workunit.client.0.smithi167.stdout:7/471: symlink d8/d2c/la8 0 2022-01-31T19:44:10.597 INFO:tasks.workunit.client.0.smithi167.stdout:2/298: sync 2022-01-31T19:44:10.597 INFO:tasks.workunit.client.0.smithi167.stdout:3/382: sync 2022-01-31T19:44:10.597 INFO:tasks.workunit.client.0.smithi167.stdout:3/383: creat d9/d29/d60/f69 x:0 0 0 2022-01-31T19:44:10.600 INFO:tasks.workunit.client.0.smithi167.stdout:3/384: symlink d9/d29/l6a 0 2022-01-31T19:44:10.600 INFO:tasks.workunit.client.0.smithi167.stdout:2/299: write dd/df/d1b/d2f/f4d [1151774,93114] 0 2022-01-31T19:44:10.603 INFO:tasks.workunit.client.0.smithi167.stdout:3/385: rename d9/d29/d2e to d9/d29/d3b/d41/d6b 0 2022-01-31T19:44:10.604 INFO:tasks.workunit.client.0.smithi167.stdout:2/300: rename dd/c13 to dd/df/d1b/d3d/c58 0 2022-01-31T19:44:10.605 INFO:tasks.workunit.client.0.smithi167.stdout:3/386: rename d9/le to d9/d21/l6c 0 2022-01-31T19:44:10.607 INFO:tasks.workunit.client.0.smithi167.stdout:2/301: unlink dd/d18/l49 0 2022-01-31T19:44:10.608 INFO:tasks.workunit.client.0.smithi167.stdout:3/387: rename d9/f40 to d9/d56/f6d 0 2022-01-31T19:44:10.609 INFO:tasks.workunit.client.0.smithi167.stdout:2/302: unlink dd/c17 0 2022-01-31T19:44:10.610 INFO:tasks.workunit.client.0.smithi167.stdout:3/388: rename d9/d56/f59 to d9/d26/f6e 0 2022-01-31T19:44:10.611 INFO:tasks.workunit.client.0.smithi167.stdout:3/389: symlink d9/d29/d60/l6f 0 2022-01-31T19:44:10.641 INFO:tasks.workunit.client.0.smithi167.stdout:3/390: getdents d9/d29/d3b/d41 0 2022-01-31T19:44:10.641 INFO:tasks.workunit.client.0.smithi167.stdout:3/391: chown d9/f30 781919 1 2022-01-31T19:44:10.642 INFO:tasks.workunit.client.0.smithi167.stdout:3/392: mkdir d9/d29/d3b/d70 0 2022-01-31T19:44:10.644 INFO:tasks.workunit.client.0.smithi167.stdout:3/393: link d9/cb d9/d29/d3b/c71 0 2022-01-31T19:44:10.644 INFO:tasks.workunit.client.0.smithi167.stdout:3/394: readlink d9/d29/d3b/d41/d6b/l48 0 2022-01-31T19:44:10.645 INFO:tasks.workunit.client.0.smithi167.stdout:3/395: readlink d9/d29/d60/l45 0 2022-01-31T19:44:10.645 INFO:tasks.workunit.client.0.smithi167.stdout:3/396: stat d9/d29/f33 0 2022-01-31T19:44:10.648 INFO:tasks.workunit.client.0.smithi167.stdout:5/509: dwrite d5/d6/de/f37 [0,4194304] 0 2022-01-31T19:44:10.651 INFO:tasks.workunit.client.0.smithi167.stdout:2/303: dwrite dd/d18/f4e [4194304,4194304] 0 2022-01-31T19:44:10.651 INFO:tasks.workunit.client.0.smithi167.stdout:2/304: fdatasync dd/d18/f30 0 2022-01-31T19:44:10.651 INFO:tasks.workunit.client.0.smithi167.stdout:2/305: chown dd/df/d1b/d2f/f34 2868 1 2022-01-31T19:44:10.653 INFO:tasks.workunit.client.0.smithi167.stdout:2/306: symlink dd/df/d1b/d3d/l59 0 2022-01-31T19:44:10.653 INFO:tasks.workunit.client.0.smithi167.stdout:7/472: dwrite d8/d2c/d43/f97 [0,4194304] 0 2022-01-31T19:44:10.654 INFO:tasks.workunit.client.0.smithi167.stdout:7/473: fdatasync d8/dc/f2b 0 2022-01-31T19:44:10.654 INFO:tasks.workunit.client.0.smithi167.stdout:3/397: dread d9/f27 [0,4194304] 0 2022-01-31T19:44:10.654 INFO:tasks.workunit.client.0.smithi167.stdout:7/474: creat d8/d2c/d6e/d4d/d5e/fa9 x:0 0 0 2022-01-31T19:44:10.654 INFO:tasks.workunit.client.0.smithi167.stdout:3/398: chown d9/d29/d3b/d41/d6b/f3e 44 1 2022-01-31T19:44:10.655 INFO:tasks.workunit.client.0.smithi167.stdout:2/307: write fa [1633347,621] 0 2022-01-31T19:44:10.657 INFO:tasks.workunit.client.0.smithi167.stdout:7/475: mkdir d8/d2c/d43/d31/d4f/d83/daa 0 2022-01-31T19:44:10.657 INFO:tasks.workunit.client.0.smithi167.stdout:7/476: chown d8/d2c/d43/d31/d4f/l7b 13523136 1 2022-01-31T19:44:10.659 INFO:tasks.workunit.client.0.smithi167.stdout:3/399: creat d9/d29/d3b/d41/d6b/f72 x:0 0 0 2022-01-31T19:44:10.661 INFO:tasks.workunit.client.0.smithi167.stdout:3/400: link d9/d29/f36 d9/d56/f73 0 2022-01-31T19:44:10.662 INFO:tasks.workunit.client.0.smithi167.stdout:3/401: readlink d9/d21/l34 0 2022-01-31T19:44:10.665 INFO:tasks.workunit.client.0.smithi167.stdout:7/477: dread d8/d2c/d6e/d4d/d67/f7e [0,4194304] 0 2022-01-31T19:44:10.665 INFO:tasks.workunit.client.0.smithi167.stdout:7/478: chown d8/d2c/d43/d31/d4f/d83/la2 670807 1 2022-01-31T19:44:10.667 INFO:tasks.workunit.client.0.smithi167.stdout:7/479: mkdir d8/dab 0 2022-01-31T19:44:10.667 INFO:tasks.workunit.client.0.smithi167.stdout:7/480: fsync d8/d2c/d6e/f49 0 2022-01-31T19:44:10.668 INFO:tasks.workunit.client.0.smithi167.stdout:7/481: link d8/dc/c11 d8/dab/cac 0 2022-01-31T19:44:10.669 INFO:tasks.workunit.client.0.smithi167.stdout:7/482: truncate d8/fa 70675 0 2022-01-31T19:44:10.675 INFO:tasks.workunit.client.0.smithi167.stdout:5/510: dwrite d5/d6/d16/d18/f8d [0,4194304] 0 2022-01-31T19:44:10.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/308: dwrite dd/f29 [4194304,4194304] 0 2022-01-31T19:44:10.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/309: chown dd/d18/f31 174515121 1 2022-01-31T19:44:10.680 INFO:tasks.workunit.client.0.smithi167.stdout:5/511: mkdir d5/d6/de/d22/d76/da3 0 2022-01-31T19:44:10.684 INFO:tasks.workunit.client.0.smithi167.stdout:5/512: unlink d5/d6/d16/f7e 0 2022-01-31T19:44:10.687 INFO:tasks.workunit.client.0.smithi167.stdout:5/513: dread d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:10.687 INFO:tasks.workunit.client.0.smithi167.stdout:3/402: dwrite d9/d29/d60/f58 [0,4194304] 0 2022-01-31T19:44:10.688 INFO:tasks.workunit.client.0.smithi167.stdout:7/483: dwrite d8/d2c/d6c/f79 [0,4194304] 0 2022-01-31T19:44:10.688 INFO:tasks.workunit.client.0.smithi167.stdout:5/514: mkdir d5/d6/de/d22/d6d/da0/da4 0 2022-01-31T19:44:10.693 INFO:tasks.workunit.client.0.smithi167.stdout:7/484: creat d8/d2c/d43/d31/fad x:0 0 0 2022-01-31T19:44:10.697 INFO:tasks.workunit.client.0.smithi167.stdout:7/485: mknod d8/d2c/d6e/cae 0 2022-01-31T19:44:10.697 INFO:tasks.workunit.client.0.smithi167.stdout:7/486: fdatasync d8/d2c/d6e/d4d/d67/d8f/f96 0 2022-01-31T19:44:10.699 INFO:tasks.workunit.client.0.smithi167.stdout:7/487: mkdir d8/d2c/d43/d94/daf 0 2022-01-31T19:44:10.699 INFO:tasks.workunit.client.0.smithi167.stdout:7/488: write d8/fa7 [39733,102814] 0 2022-01-31T19:44:10.699 INFO:tasks.workunit.client.0.smithi167.stdout:7/489: readlink d8/d2c/d6e/l20 0 2022-01-31T19:44:10.700 INFO:tasks.workunit.client.0.smithi167.stdout:0/183: sync 2022-01-31T19:44:10.701 INFO:tasks.workunit.client.0.smithi167.stdout:1/476: sync 2022-01-31T19:44:10.701 INFO:tasks.workunit.client.0.smithi167.stdout:6/267: sync 2022-01-31T19:44:10.701 INFO:tasks.workunit.client.0.smithi167.stdout:0/184: fdatasync d17/f31 0 2022-01-31T19:44:10.702 INFO:tasks.workunit.client.0.smithi167.stdout:8/454: sync 2022-01-31T19:44:10.703 INFO:tasks.workunit.client.0.smithi167.stdout:1/477: getdents d0/d11 0 2022-01-31T19:44:10.703 INFO:tasks.workunit.client.0.smithi167.stdout:9/248: sync 2022-01-31T19:44:10.703 INFO:tasks.workunit.client.0.smithi167.stdout:4/389: sync 2022-01-31T19:44:10.703 INFO:tasks.workunit.client.0.smithi167.stdout:4/390: truncate d0/d6/d7/d26/d40/f64 537594 0 2022-01-31T19:44:10.704 INFO:tasks.workunit.client.0.smithi167.stdout:4/391: read - d0/d6/d7/d26/f2f zero size 2022-01-31T19:44:10.704 INFO:tasks.workunit.client.0.smithi167.stdout:8/455: creat d6/db/d4e/d73/d88/f96 x:0 0 0 2022-01-31T19:44:10.705 INFO:tasks.workunit.client.0.smithi167.stdout:1/478: mkdir d0/d10/d5a/d33/d4b/da5/da9 0 2022-01-31T19:44:10.705 INFO:tasks.workunit.client.0.smithi167.stdout:9/249: creat dd/d34/f4f x:0 0 0 2022-01-31T19:44:10.705 INFO:tasks.workunit.client.0.smithi167.stdout:4/392: unlink d0/d6/d7/d39/d3a/f55 0 2022-01-31T19:44:10.707 INFO:tasks.workunit.client.0.smithi167.stdout:8/456: rename d6/db/d2f/f81 to d6/db/de/d3d/f97 0 2022-01-31T19:44:10.707 INFO:tasks.workunit.client.0.smithi167.stdout:8/457: creat d6/db/d28/d61/f98 x:0 0 0 2022-01-31T19:44:10.708 INFO:tasks.workunit.client.0.smithi167.stdout:5/515: dread d5/d6/de/f3e [0,4194304] 0 2022-01-31T19:44:10.708 INFO:tasks.workunit.client.0.smithi167.stdout:5/516: creat d5/d6/de/d22/d4d/fa5 x:0 0 0 2022-01-31T19:44:10.708 INFO:tasks.workunit.client.0.smithi167.stdout:5/517: fdatasync d5/d6/de/d22/d4d/f9e 0 2022-01-31T19:44:10.709 INFO:tasks.workunit.client.0.smithi167.stdout:5/518: write d5/f44 [840015,36897] 0 2022-01-31T19:44:10.709 INFO:tasks.workunit.client.0.smithi167.stdout:5/519: write d5/d6/d3a/f48 [836008,109825] 0 2022-01-31T19:44:10.709 INFO:tasks.workunit.client.0.smithi167.stdout:1/479: unlink d0/d10/d20/l53 0 2022-01-31T19:44:10.710 INFO:tasks.workunit.client.0.smithi167.stdout:4/393: write d0/d6/d7/f18 [641481,101187] 0 2022-01-31T19:44:10.713 INFO:tasks.workunit.client.0.smithi167.stdout:9/250: dread dd/d19/f21 [0,4194304] 0 2022-01-31T19:44:10.713 INFO:tasks.workunit.client.0.smithi167.stdout:1/480: creat d0/d10/d20/d7c/faa x:0 0 0 2022-01-31T19:44:10.714 INFO:tasks.workunit.client.0.smithi167.stdout:4/394: creat d0/d6/d7/d26/d40/d4c/f7d x:0 0 0 2022-01-31T19:44:10.715 INFO:tasks.workunit.client.0.smithi167.stdout:5/520: getdents d5/d6/d16 0 2022-01-31T19:44:10.715 INFO:tasks.workunit.client.0.smithi167.stdout:5/521: dread - d5/d6/de/d22/d4d/fa5 zero size 2022-01-31T19:44:10.718 INFO:tasks.workunit.client.0.smithi167.stdout:1/481: creat d0/d11/fab x:0 0 0 2022-01-31T19:44:10.718 INFO:tasks.workunit.client.0.smithi167.stdout:1/482: chown d0/d10/d20 9524 1 2022-01-31T19:44:10.719 INFO:tasks.workunit.client.0.smithi167.stdout:4/395: link d0/d6/d7/d39/c42 d0/d5d/c7e 0 2022-01-31T19:44:10.719 INFO:tasks.workunit.client.0.smithi167.stdout:1/483: chown d0/d10/d5a/d33/d4b/l6b 6369 1 2022-01-31T19:44:10.721 INFO:tasks.workunit.client.0.smithi167.stdout:4/396: symlink d0/d3f/l7f 0 2022-01-31T19:44:10.721 INFO:tasks.workunit.client.0.smithi167.stdout:4/397: write d0/d6/d7/d26/f72 [582038,15166] 0 2022-01-31T19:44:10.722 INFO:tasks.workunit.client.0.smithi167.stdout:9/251: getdents dd 0 2022-01-31T19:44:10.723 INFO:tasks.workunit.client.0.smithi167.stdout:4/398: symlink d0/d6/d7/l80 0 2022-01-31T19:44:10.724 INFO:tasks.workunit.client.0.smithi167.stdout:9/252: mkdir dd/d19/d24/d50 0 2022-01-31T19:44:10.727 INFO:tasks.workunit.client.0.smithi167.stdout:0/185: dread d17/f26 [0,4194304] 0 2022-01-31T19:44:10.729 INFO:tasks.workunit.client.0.smithi167.stdout:9/253: truncate dd/d35/f3a 603221 0 2022-01-31T19:44:10.730 INFO:tasks.workunit.client.0.smithi167.stdout:9/254: read dd/d19/d24/d3b/f46 [2024555,69107] 0 2022-01-31T19:44:10.731 INFO:tasks.workunit.client.0.smithi167.stdout:9/255: chown dd/f16 758693 1 2022-01-31T19:44:10.731 INFO:tasks.workunit.client.0.smithi167.stdout:9/256: write dd/d19/f3d [5229503,74362] 0 2022-01-31T19:44:10.731 INFO:tasks.workunit.client.0.smithi167.stdout:9/257: write dd/d19/d24/d3b/f4a [4237712,111944] 0 2022-01-31T19:44:10.731 INFO:tasks.workunit.client.0.smithi167.stdout:0/186: unlink d17/d19/c38 0 2022-01-31T19:44:10.732 INFO:tasks.workunit.client.0.smithi167.stdout:1/484: write d0/d10/d5a/d33/d4b/f65 [3784643,58284] 0 2022-01-31T19:44:10.733 INFO:tasks.workunit.client.0.smithi167.stdout:9/258: rename dd/d34/f4f to dd/d34/f51 0 2022-01-31T19:44:10.736 INFO:tasks.workunit.client.0.smithi167.stdout:9/259: symlink dd/d19/d24/d3b/d4c/l52 0 2022-01-31T19:44:10.737 INFO:tasks.workunit.client.0.smithi167.stdout:0/187: rmdir d17/d19/d1c 39 2022-01-31T19:44:10.737 INFO:tasks.workunit.client.0.smithi167.stdout:0/188: chown f8 2550 1 2022-01-31T19:44:10.740 INFO:tasks.workunit.client.0.smithi167.stdout:0/189: getdents d17/d2c 0 2022-01-31T19:44:10.740 INFO:tasks.workunit.client.0.smithi167.stdout:0/190: stat d17/d19/l24 0 2022-01-31T19:44:10.742 INFO:tasks.workunit.client.0.smithi167.stdout:5/522: dwrite d5/d6/d16/d18/d5b/f6c [0,4194304] 0 2022-01-31T19:44:10.742 INFO:tasks.workunit.client.0.smithi167.stdout:0/191: rename d17/c1a to d17/d19/d1c/d39/c3b 0 2022-01-31T19:44:10.742 INFO:tasks.workunit.client.0.smithi167.stdout:0/192: dread - d17/f31 zero size 2022-01-31T19:44:10.748 INFO:tasks.workunit.client.0.smithi167.stdout:4/399: dwrite d0/d6/d7/d26/d31/f71 [0,4194304] 0 2022-01-31T19:44:10.751 INFO:tasks.workunit.client.0.smithi167.stdout:7/490: dwrite d8/d2c/d6c/f6d [0,4194304] 0 2022-01-31T19:44:10.763 INFO:tasks.workunit.client.0.smithi167.stdout:4/400: rename d0/d6/d7/l80 to d0/l81 0 2022-01-31T19:44:10.764 INFO:tasks.workunit.client.0.smithi167.stdout:7/491: mknod d8/d2c/d7a/d99/cb0 0 2022-01-31T19:44:10.765 INFO:tasks.workunit.client.0.smithi167.stdout:4/401: mknod d0/d3f/d75/c82 0 2022-01-31T19:44:10.768 INFO:tasks.workunit.client.0.smithi167.stdout:4/402: symlink d0/d6/d7/d39/l83 0 2022-01-31T19:44:10.769 INFO:tasks.workunit.client.0.smithi167.stdout:7/492: rename d8/d2c/d43/d31 to d8/d2c/d6e/d4d/d5e/d71/db1 0 2022-01-31T19:44:10.769 INFO:tasks.workunit.client.0.smithi167.stdout:7/493: chown d8/d2c/d43/d94/c9f 483983076 1 2022-01-31T19:44:10.770 INFO:tasks.workunit.client.0.smithi167.stdout:4/403: mknod d0/d6/d7/d26/d31/c84 0 2022-01-31T19:44:10.770 INFO:tasks.workunit.client.0.smithi167.stdout:9/260: dread dd/f1a [4194304,4194304] 0 2022-01-31T19:44:10.771 INFO:tasks.workunit.client.0.smithi167.stdout:7/494: link d8/dc/d18/c61 d8/d2c/d6e/d4d/d5e/d71/db1/cb2 0 2022-01-31T19:44:10.772 INFO:tasks.workunit.client.0.smithi167.stdout:4/404: symlink d0/l85 0 2022-01-31T19:44:10.772 INFO:tasks.workunit.client.0.smithi167.stdout:4/405: stat d0/d6/d7/d39/d4e/c6c 0 2022-01-31T19:44:10.772 INFO:tasks.workunit.client.0.smithi167.stdout:4/406: creat d0/d6/d7/d26/d40/d4c/f86 x:0 0 0 2022-01-31T19:44:10.772 INFO:tasks.workunit.client.0.smithi167.stdout:4/407: write d0/d6/d7/f7b [10668,45438] 0 2022-01-31T19:44:10.773 INFO:tasks.workunit.client.0.smithi167.stdout:9/261: mknod dd/d17/c53 0 2022-01-31T19:44:10.773 INFO:tasks.workunit.client.0.smithi167.stdout:9/262: creat dd/d19/d24/d3b/f54 x:0 0 0 2022-01-31T19:44:10.774 INFO:tasks.workunit.client.0.smithi167.stdout:7/495: chown d8/d2c/d43/l92 6399118 1 2022-01-31T19:44:10.774 INFO:tasks.workunit.client.0.smithi167.stdout:6/268: dwrite d8/d33/f40 [0,4194304] 0 2022-01-31T19:44:10.775 INFO:tasks.workunit.client.0.smithi167.stdout:4/408: symlink d0/d6/l87 0 2022-01-31T19:44:10.775 INFO:tasks.workunit.client.0.smithi167.stdout:9/263: write dd/fe [7713575,41184] 0 2022-01-31T19:44:10.776 INFO:tasks.workunit.client.0.smithi167.stdout:9/264: write dd/d19/f27 [1329087,69529] 0 2022-01-31T19:44:10.776 INFO:tasks.workunit.client.0.smithi167.stdout:8/458: dwrite d6/db/de/f3e [0,4194304] 0 2022-01-31T19:44:10.777 INFO:tasks.workunit.client.0.smithi167.stdout:8/459: write d6/db/fd [1214097,94646] 0 2022-01-31T19:44:10.777 INFO:tasks.workunit.client.0.smithi167.stdout:8/460: fsync d6/db/d28/d66/f8e 0 2022-01-31T19:44:10.778 INFO:tasks.workunit.client.0.smithi167.stdout:7/496: mkdir d8/d2c/d6e/d4d/d5e/d71/db3 0 2022-01-31T19:44:10.778 INFO:tasks.workunit.client.0.smithi167.stdout:1/485: dwrite d0/d11/f4d [0,4194304] 0 2022-01-31T19:44:10.778 INFO:tasks.workunit.client.0.smithi167.stdout:1/486: write d0/d62/fa3 [198451,70468] 0 2022-01-31T19:44:10.779 INFO:tasks.workunit.client.0.smithi167.stdout:5/523: dwrite d5/d61/f74 [4194304,4194304] 0 2022-01-31T19:44:10.781 INFO:tasks.workunit.client.0.smithi167.stdout:0/193: dwrite d17/f22 [0,4194304] 0 2022-01-31T19:44:10.786 INFO:tasks.workunit.client.0.smithi167.stdout:6/269: mkdir d8/d33/d51/d6a 0 2022-01-31T19:44:10.787 INFO:tasks.workunit.client.0.smithi167.stdout:6/270: truncate d8/fb 5155202 0 2022-01-31T19:44:10.787 INFO:tasks.workunit.client.0.smithi167.stdout:4/409: creat d0/d3f/d60/d65/f88 x:0 0 0 2022-01-31T19:44:10.788 INFO:tasks.workunit.client.0.smithi167.stdout:4/410: dread - d0/f7a zero size 2022-01-31T19:44:10.790 INFO:tasks.workunit.client.0.smithi167.stdout:8/461: truncate d6/db/d4e/d73/f7b 545832 0 2022-01-31T19:44:10.790 INFO:tasks.workunit.client.0.smithi167.stdout:0/194: dread f14 [0,4194304] 0 2022-01-31T19:44:10.796 INFO:tasks.workunit.client.0.smithi167.stdout:5/524: unlink d5/d6/de/d22/f25 0 2022-01-31T19:44:10.798 INFO:tasks.workunit.client.0.smithi167.stdout:6/271: creat d8/d20/f6b x:0 0 0 2022-01-31T19:44:10.800 INFO:tasks.workunit.client.0.smithi167.stdout:5/525: symlink d5/d6/d16/la6 0 2022-01-31T19:44:10.804 INFO:tasks.workunit.client.0.smithi167.stdout:6/272: mknod d8/d33/d35/c6c 0 2022-01-31T19:44:10.806 INFO:tasks.workunit.client.0.smithi167.stdout:1/487: dwrite d0/f1 [4194304,4194304] 0 2022-01-31T19:44:10.808 INFO:tasks.workunit.client.0.smithi167.stdout:5/526: unlink d5/d6/d16/d18/f31 0 2022-01-31T19:44:10.808 INFO:tasks.workunit.client.0.smithi167.stdout:5/527: truncate d5/d6/d16/d18/f33 856137 0 2022-01-31T19:44:10.808 INFO:tasks.workunit.client.0.smithi167.stdout:5/528: read d5/d6/d16/d18/f2c [1382685,116887] 0 2022-01-31T19:44:10.811 INFO:tasks.workunit.client.0.smithi167.stdout:5/529: mknod d5/d6/de/d22/d76/ca7 0 2022-01-31T19:44:10.813 INFO:tasks.workunit.client.0.smithi167.stdout:5/530: getdents d5/d6/d16/d18/d43/d45 0 2022-01-31T19:44:10.814 INFO:tasks.workunit.client.0.smithi167.stdout:4/411: getdents d0/d6/d7/d26/d40/d4c 0 2022-01-31T19:44:10.815 INFO:tasks.workunit.client.0.smithi167.stdout:4/412: chown d0/d6/d7/d39/d3a 11043957 1 2022-01-31T19:44:10.817 INFO:tasks.workunit.client.0.smithi167.stdout:5/531: write d5/d6/d3a/f17 [1459443,88189] 0 2022-01-31T19:44:10.817 INFO:tasks.workunit.client.0.smithi167.stdout:5/532: write d5/d6/de/d22/d4d/f9e [176780,50585] 0 2022-01-31T19:44:10.820 INFO:tasks.workunit.client.0.smithi167.stdout:6/273: dread d8/fd [0,4194304] 0 2022-01-31T19:44:10.821 INFO:tasks.workunit.client.0.smithi167.stdout:6/274: write d8/f5c [3616823,110087] 0 2022-01-31T19:44:10.824 INFO:tasks.workunit.client.0.smithi167.stdout:7/497: dwrite d8/d2c/d6e/d4d/d5e/d71/db1/f6a [0,4194304] 0 2022-01-31T19:44:10.824 INFO:tasks.workunit.client.0.smithi167.stdout:1/488: dread d0/f5 [4194304,4194304] 0 2022-01-31T19:44:10.825 INFO:tasks.workunit.client.0.smithi167.stdout:8/462: dwrite d6/db/de/f3e [0,4194304] 0 2022-01-31T19:44:10.825 INFO:tasks.workunit.client.0.smithi167.stdout:9/265: dwrite dd/d17/f18 [4194304,4194304] 0 2022-01-31T19:44:10.826 INFO:tasks.workunit.client.0.smithi167.stdout:9/266: read dd/d17/f18 [3849042,108128] 0 2022-01-31T19:44:10.826 INFO:tasks.workunit.client.0.smithi167.stdout:0/195: dwrite fa [0,4194304] 0 2022-01-31T19:44:10.827 INFO:tasks.workunit.client.0.smithi167.stdout:1/489: mknod d0/d10/d5a/d33/d4b/da5/cac 0 2022-01-31T19:44:10.828 INFO:tasks.workunit.client.0.smithi167.stdout:5/533: dread d5/d6/d16/d18/f59 [0,4194304] 0 2022-01-31T19:44:10.829 INFO:tasks.workunit.client.0.smithi167.stdout:7/498: symlink d8/d2c/d6e/d4d/d6b/d8e/lb4 0 2022-01-31T19:44:10.831 INFO:tasks.workunit.client.0.smithi167.stdout:9/267: creat dd/d17/f55 x:0 0 0 2022-01-31T19:44:10.832 INFO:tasks.workunit.client.0.smithi167.stdout:1/490: mkdir d0/d10/d5a/d33/d4b/da5/da9/dad 0 2022-01-31T19:44:10.832 INFO:tasks.workunit.client.0.smithi167.stdout:6/275: dread d8/fd [0,4194304] 0 2022-01-31T19:44:10.833 INFO:tasks.workunit.client.0.smithi167.stdout:7/499: symlink d8/d2c/d6e/d9a/lb5 0 2022-01-31T19:44:10.834 INFO:tasks.workunit.client.0.smithi167.stdout:7/500: truncate d8/d2c/d6e/d4d/d6b/f81 151988 0 2022-01-31T19:44:10.835 INFO:tasks.workunit.client.0.smithi167.stdout:4/413: dwrite d0/d6/d7/d26/d31/f71 [0,4194304] 0 2022-01-31T19:44:10.836 INFO:tasks.workunit.client.0.smithi167.stdout:4/414: truncate d0/d6/d7/d26/d40/f64 1199215 0 2022-01-31T19:44:10.837 INFO:tasks.workunit.client.0.smithi167.stdout:5/534: symlink d5/d6/d16/d18/d5b/la8 0 2022-01-31T19:44:10.838 INFO:tasks.workunit.client.0.smithi167.stdout:9/268: mkdir dd/d19/d24/d56 0 2022-01-31T19:44:10.839 INFO:tasks.workunit.client.0.smithi167.stdout:4/415: write d0/d6/d7/f30 [6845,102089] 0 2022-01-31T19:44:10.839 INFO:tasks.workunit.client.0.smithi167.stdout:5/535: read d5/d6/d3a/f4e [2226368,98740] 0 2022-01-31T19:44:10.840 INFO:tasks.workunit.client.0.smithi167.stdout:9/269: write dd/ff [2217129,34248] 0 2022-01-31T19:44:10.840 INFO:tasks.workunit.client.0.smithi167.stdout:0/196: dread fa [0,4194304] 0 2022-01-31T19:44:10.841 INFO:tasks.workunit.client.0.smithi167.stdout:0/197: stat f8 0 2022-01-31T19:44:10.841 INFO:tasks.workunit.client.0.smithi167.stdout:1/491: link d0/d11/d46/c9a d0/d11/d46/cae 0 2022-01-31T19:44:10.842 INFO:tasks.workunit.client.0.smithi167.stdout:6/276: creat d8/d62/f6d x:0 0 0 2022-01-31T19:44:10.842 INFO:tasks.workunit.client.0.smithi167.stdout:6/277: truncate d8/d20/d54/f63 574622 0 2022-01-31T19:44:10.842 INFO:tasks.workunit.client.0.smithi167.stdout:5/536: read d5/d61/f7f [2688734,127136] 0 2022-01-31T19:44:10.843 INFO:tasks.workunit.client.0.smithi167.stdout:7/501: mknod d8/d2c/d6c/cb6 0 2022-01-31T19:44:10.843 INFO:tasks.workunit.client.0.smithi167.stdout:7/502: dread - d8/d2c/d6e/d4d/d6b/f88 zero size 2022-01-31T19:44:10.843 INFO:tasks.workunit.client.0.smithi167.stdout:4/416: mkdir d0/d6/d7/d89 0 2022-01-31T19:44:10.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.841+0000 7fac448a3700 1 -- 172.21.15.167:0/844988625 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 msgr2=0x7fac400f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:10.844 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.841+0000 7fac448a3700 1 --2- 172.21.15.167:0/844988625 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400f5b30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fac30004660 tx=0x7fac30009b30).stop 2022-01-31T19:44:10.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.841+0000 7fac448a3700 1 -- 172.21.15.167:0/844988625 shutdown_connections 2022-01-31T19:44:10.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.841+0000 7fac448a3700 1 --2- 172.21.15.167:0/844988625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac400f69b0 0x7fac400f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:10.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.841+0000 7fac448a3700 1 --2- 172.21.15.167:0/844988625 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400f5b30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:10.845 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.841+0000 7fac448a3700 1 -- 172.21.15.167:0/844988625 >> 172.21.15.167:0/844988625 conn(0x7fac400f0e90 msgr2=0x7fac400f32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:10.846 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.842+0000 7fac448a3700 1 -- 172.21.15.167:0/844988625 shutdown_connections 2022-01-31T19:44:10.846 INFO:tasks.workunit.client.0.smithi167.stdout:1/492: write d0/f3a [3468954,88685] 0 2022-01-31T19:44:10.847 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.842+0000 7fac448a3700 1 -- 172.21.15.167:0/844988625 wait complete. 2022-01-31T19:44:10.847 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.843+0000 7fac448a3700 1 Processor -- start 2022-01-31T19:44:10.847 INFO:tasks.workunit.client.0.smithi167.stdout:0/198: unlink d17/d19/d1c/f30 0 2022-01-31T19:44:10.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.846+0000 7fac448a3700 1 -- start start 2022-01-31T19:44:10.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.846+0000 7fac448a3700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:10.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.846+0000 7fac448a3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac400f69b0 0x7fac400fbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:10.850 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.846+0000 7fac448a3700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401149a0 con 0x7fac400f5710 2022-01-31T19:44:10.851 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.846+0000 7fac448a3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40114ae0 con 0x7fac400f69b0 2022-01-31T19:44:10.852 INFO:tasks.workunit.client.0.smithi167.stdout:6/278: rename d8/d20/d54/f57 to d8/d33/d51/d5d/f6e 0 2022-01-31T19:44:10.852 INFO:tasks.workunit.client.0.smithi167.stdout:5/537: mknod d5/d6/d16/d18/d43/ca9 0 2022-01-31T19:44:10.852 INFO:tasks.workunit.client.0.smithi167.stdout:5/538: truncate d5/d6/d3a/f48 1334971 0 2022-01-31T19:44:10.852 INFO:tasks.workunit.client.0.smithi167.stdout:7/503: symlink d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/daa/lb7 0 2022-01-31T19:44:10.853 INFO:tasks.workunit.client.0.smithi167.stdout:4/417: creat d0/d6/d7/d26/d40/f8a x:0 0 0 2022-01-31T19:44:10.853 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.849+0000 7fac3e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:10.854 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.849+0000 7fac3e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34100/0 (socket says 172.21.15.167:34100) 2022-01-31T19:44:10.854 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.849+0000 7fac3e59c700 1 -- 172.21.15.167:0/1526068842 learned_addr learned my addr 172.21.15.167:0/1526068842 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:10.854 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.849+0000 7fac3dd9b700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac400f69b0 0x7fac400fbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:10.855 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.850+0000 7fac3e59c700 1 -- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac400f69b0 msgr2=0x7fac400fbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:10.855 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.850+0000 7fac3e59c700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac400f69b0 0x7fac400fbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:10.855 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.850+0000 7fac3e59c700 1 -- 172.21.15.167:0/1526068842 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac30005040 con 0x7fac400f5710 2022-01-31T19:44:10.855 INFO:tasks.workunit.client.0.smithi167.stdout:0/199: creat d17/d2c/f3c x:0 0 0 2022-01-31T19:44:10.856 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.850+0000 7fac3e59c700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400fd470 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fac30000c00 tx=0x7fac3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:10.856 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.851+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac30019070 con 0x7fac400f5710 2022-01-31T19:44:10.857 INFO:tasks.workunit.client.0.smithi167.stdout:5/539: mkdir d5/d6/de/d22/d6d/da0/da4/daa 0 2022-01-31T19:44:10.857 INFO:tasks.workunit.client.0.smithi167.stdout:0/200: rename d17/d19/f2e to d17/f3d 0 2022-01-31T19:44:10.858 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.852+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac400fda20 con 0x7fac400f5710 2022-01-31T19:44:10.858 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.852+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fac30013380 con 0x7fac400f5710 2022-01-31T19:44:10.858 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.852+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac30014cd0 con 0x7fac400f5710 2022-01-31T19:44:10.859 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.852+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac400fc6d0 con 0x7fac400f5710 2022-01-31T19:44:10.859 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.853+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac40045bb0 con 0x7fac400f5710 2022-01-31T19:44:10.859 INFO:tasks.workunit.client.0.smithi167.stdout:5/540: truncate d5/d6/de/d22/d4d/d65/f79 598407 0 2022-01-31T19:44:10.860 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.854+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 19) v1 ==== 78465+0+0 (secure 0 0 0) 0x7fac30025070 con 0x7fac400f5710 2022-01-31T19:44:10.860 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.855+0000 7fac2effd700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fac28061a10 0x7fac28063ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:10.860 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.855+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5343+0+0 (secure 0 0 0) 0x7fac3007e280 con 0x7fac400f5710 2022-01-31T19:44:10.861 INFO:tasks.workunit.client.0.smithi167.stdout:5/541: symlink d5/d6/de/d22/lab 0 2022-01-31T19:44:10.862 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.855+0000 7fac3dd9b700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fac28061a10 0x7fac28063ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:10.862 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.856+0000 7fac3dd9b700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fac28061a10 0x7fac28063ed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fac400fbc10 tx=0x7fac3400b040).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:10.864 INFO:tasks.workunit.client.0.smithi167.stdout:4/418: dread d0/d6/d7/f7b [0,4194304] 0 2022-01-31T19:44:10.864 INFO:tasks.workunit.client.0.smithi167.stdout:4/419: readlink d0/d3f/l3e 0 2022-01-31T19:44:10.864 INFO:tasks.workunit.client.0.smithi167.stdout:4/420: creat d0/d6/d7/d26/d31/f8b x:0 0 0 2022-01-31T19:44:10.865 INFO:tasks.workunit.client.0.smithi167.stdout:4/421: stat d0/d3f/d60/d65/f88 0 2022-01-31T19:44:10.865 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:10.862+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7fac3004cae0 con 0x7fac400f5710 2022-01-31T19:44:10.866 INFO:tasks.workunit.client.0.smithi167.stdout:7/504: dread d8/d2c/d6e/f44 [0,4194304] 0 2022-01-31T19:44:10.866 INFO:tasks.workunit.client.0.smithi167.stdout:0/201: read d17/f26 [1692150,70713] 0 2022-01-31T19:44:10.866 INFO:tasks.workunit.client.0.smithi167.stdout:9/270: dwrite dd/d19/f1b [0,4194304] 0 2022-01-31T19:44:10.867 INFO:tasks.workunit.client.0.smithi167.stdout:8/463: dwrite d6/d1e/f22 [4194304,4194304] 0 2022-01-31T19:44:10.867 INFO:tasks.workunit.client.0.smithi167.stdout:8/464: chown d6/d1e/d2d/d6b/l91 217165985 1 2022-01-31T19:44:10.868 INFO:tasks.workunit.client.0.smithi167.stdout:8/465: chown d6/c7 9289 1 2022-01-31T19:44:10.871 INFO:tasks.workunit.client.0.smithi167.stdout:6/279: fdatasync d8/f5c 0 2022-01-31T19:44:10.875 INFO:tasks.workunit.client.0.smithi167.stdout:4/422: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:10.877 INFO:tasks.workunit.client.0.smithi167.stdout:4/423: dread d0/d6/d7/d39/d3a/f54 [0,4194304] 0 2022-01-31T19:44:10.877 INFO:tasks.workunit.client.0.smithi167.stdout:4/424: getdents d0/d6/d7/d26/d40/d4c 0 2022-01-31T19:44:10.878 INFO:tasks.workunit.client.0.smithi167.stdout:5/542: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:44:10.878 INFO:tasks.workunit.client.0.smithi167.stdout:5/543: stat d5/d6/de/d22/d4d/d65 0 2022-01-31T19:44:10.879 INFO:tasks.workunit.client.0.smithi167.stdout:7/505: creat d8/d2c/d6e/d4d/d5e/d71/db3/fb8 x:0 0 0 2022-01-31T19:44:10.883 INFO:tasks.workunit.client.0.smithi167.stdout:0/202: mkdir d17/d3e 0 2022-01-31T19:44:10.883 INFO:tasks.workunit.client.0.smithi167.stdout:0/203: unlink c16 0 2022-01-31T19:44:10.885 INFO:tasks.workunit.client.0.smithi167.stdout:8/466: dwrite d6/fa [0,4194304] 0 2022-01-31T19:44:10.885 INFO:tasks.workunit.client.0.smithi167.stdout:8/467: chown d6/d1e/f3c 41924188 1 2022-01-31T19:44:10.889 INFO:tasks.workunit.client.0.smithi167.stdout:9/271: mknod dd/d19/c57 0 2022-01-31T19:44:10.890 INFO:tasks.workunit.client.0.smithi167.stdout:8/468: dread d6/f32 [0,4194304] 0 2022-01-31T19:44:10.891 INFO:tasks.workunit.client.0.smithi167.stdout:9/272: write dd/f1a [8097938,35313] 0 2022-01-31T19:44:10.891 INFO:tasks.workunit.client.0.smithi167.stdout:9/273: stat dd/d19/d24/d56 0 2022-01-31T19:44:10.895 INFO:tasks.workunit.client.0.smithi167.stdout:8/469: dread d6/db/d28/f43 [0,4194304] 0 2022-01-31T19:44:10.895 INFO:tasks.workunit.client.0.smithi167.stdout:4/425: rename d0/d6/d7/d26/d31/c46 to d0/d3f/d60/c8c 0 2022-01-31T19:44:10.896 INFO:tasks.workunit.client.0.smithi167.stdout:4/426: chown d0/d6/d7/d39/d3a/c74 144024676 1 2022-01-31T19:44:10.897 INFO:tasks.workunit.client.0.smithi167.stdout:4/427: read d0/d6/d7/d26/d31/f71 [59014,54031] 0 2022-01-31T19:44:10.897 INFO:tasks.workunit.client.0.smithi167.stdout:4/428: chown d0/d6/d7/d26/d40/d4c/f86 54364796 1 2022-01-31T19:44:10.897 INFO:tasks.workunit.client.0.smithi167.stdout:4/429: getdents d0/d6/d7/d26/d40/d4c 0 2022-01-31T19:44:10.898 INFO:tasks.workunit.client.0.smithi167.stdout:5/544: symlink d5/d6/de/d22/d76/lac 0 2022-01-31T19:44:10.898 INFO:tasks.workunit.client.0.smithi167.stdout:5/545: chown d5/d6/d3a/f8f 126 1 2022-01-31T19:44:10.900 INFO:tasks.workunit.client.0.smithi167.stdout:6/280: write d8/d33/d35/f3d [1244177,15125] 0 2022-01-31T19:44:10.902 INFO:tasks.workunit.client.0.smithi167.stdout:7/506: truncate d8/d2c/d6c/f79 3655502 0 2022-01-31T19:44:10.903 INFO:tasks.workunit.client.0.smithi167.stdout:7/507: creat d8/d2c/d6e/d4d/d67/fb9 x:0 0 0 2022-01-31T19:44:10.904 INFO:tasks.workunit.client.0.smithi167.stdout:0/204: rmdir d17/d19/d1c/d39 39 2022-01-31T19:44:10.904 INFO:tasks.workunit.client.0.smithi167.stdout:0/205: write d17/d19/d1c/f32 [5179939,111876] 0 2022-01-31T19:44:10.905 INFO:tasks.workunit.client.0.smithi167.stdout:9/274: symlink dd/d19/d24/l58 0 2022-01-31T19:44:10.907 INFO:tasks.workunit.client.0.smithi167.stdout:8/470: mkdir d6/d1e/d4c/d80/d99 0 2022-01-31T19:44:10.912 INFO:tasks.workunit.client.0.smithi167.stdout:7/508: unlink d8/d2c/d6e/d4d/l6f 0 2022-01-31T19:44:10.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/206: symlink d17/d19/d1c/l3f 0 2022-01-31T19:44:10.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/207: chown d17/d19/d1c/c27 2 1 2022-01-31T19:44:10.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/208: chown d17/d19/d1c/f32 0 1 2022-01-31T19:44:10.918 INFO:tasks.workunit.client.0.smithi167.stdout:0/209: unlink d17/d19/d1c/l3a 0 2022-01-31T19:44:10.918 INFO:tasks.workunit.client.0.smithi167.stdout:8/471: symlink d6/db/d4e/d73/d88/l9a 0 2022-01-31T19:44:10.918 INFO:tasks.workunit.client.0.smithi167.stdout:8/472: readlink d6/l72 0 2022-01-31T19:44:10.918 INFO:tasks.workunit.client.0.smithi167.stdout:5/546: mknod d5/d6/d16/d18/d5b/d63/d88/cad 0 2022-01-31T19:44:10.919 INFO:tasks.workunit.client.0.smithi167.stdout:0/210: rename d17/c35 to d17/d3e/c40 0 2022-01-31T19:44:10.920 INFO:tasks.workunit.client.0.smithi167.stdout:8/473: mknod d6/db/de/c9b 0 2022-01-31T19:44:10.921 INFO:tasks.workunit.client.0.smithi167.stdout:8/474: mknod d6/db/d4e/d73/d88/c9c 0 2022-01-31T19:44:10.932 INFO:tasks.workunit.client.0.smithi167.stdout:9/275: dwrite dd/d19/f27 [0,4194304] 0 2022-01-31T19:44:10.933 INFO:tasks.workunit.client.0.smithi167.stdout:6/281: dwrite d8/fb [4194304,4194304] 0 2022-01-31T19:44:10.934 INFO:tasks.workunit.client.0.smithi167.stdout:9/276: mkdir dd/d19/d59 0 2022-01-31T19:44:10.934 INFO:tasks.workunit.client.0.smithi167.stdout:9/277: chown dd/d19/d24/l58 1849489423 1 2022-01-31T19:44:10.937 INFO:tasks.workunit.client.0.smithi167.stdout:6/282: link d8/d33/d51/d5d/f6e d8/d62/f6f 0 2022-01-31T19:44:10.937 INFO:tasks.workunit.client.0.smithi167.stdout:6/283: unlink d8/f1e 0 2022-01-31T19:44:10.938 INFO:tasks.workunit.client.0.smithi167.stdout:9/278: rename dd/d19/d24/f2c to dd/d34/f5a 0 2022-01-31T19:44:10.939 INFO:tasks.workunit.client.0.smithi167.stdout:9/279: write dd/d35/f33 [248598,18910] 0 2022-01-31T19:44:10.941 INFO:tasks.workunit.client.0.smithi167.stdout:7/509: dread d8/d2c/f25 [0,4194304] 0 2022-01-31T19:44:10.943 INFO:tasks.workunit.client.0.smithi167.stdout:4/430: dwrite d0/d6/d7/f4a [0,4194304] 0 2022-01-31T19:44:10.943 INFO:tasks.workunit.client.0.smithi167.stdout:5/547: dread d5/d6/d3a/f4e [0,4194304] 0 2022-01-31T19:44:10.944 INFO:tasks.workunit.client.0.smithi167.stdout:8/475: dwrite d6/db/de/d3d/f56 [0,4194304] 0 2022-01-31T19:44:10.944 INFO:tasks.workunit.client.0.smithi167.stdout:8/476: chown d6/db/de 5392 1 2022-01-31T19:44:10.945 INFO:tasks.workunit.client.0.smithi167.stdout:9/280: mkdir dd/d19/d24/d56/d5b 0 2022-01-31T19:44:10.945 INFO:tasks.workunit.client.0.smithi167.stdout:9/281: fsync dd/d17/f2e 0 2022-01-31T19:44:10.949 INFO:tasks.workunit.client.0.smithi167.stdout:6/284: rename l1 to d8/d33/d4f/l70 0 2022-01-31T19:44:10.949 INFO:tasks.workunit.client.0.smithi167.stdout:6/285: write d8/d33/f5f [670224,61344] 0 2022-01-31T19:44:10.952 INFO:tasks.workunit.client.0.smithi167.stdout:7/510: unlink d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f42 0 2022-01-31T19:44:10.952 INFO:tasks.workunit.client.0.smithi167.stdout:7/511: fdatasync d8/d2c/d6e/f49 0 2022-01-31T19:44:10.952 INFO:tasks.workunit.client.0.smithi167.stdout:7/512: fdatasync d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f3c 0 2022-01-31T19:44:10.955 INFO:tasks.workunit.client.0.smithi167.stdout:5/548: creat d5/d6/d16/d18/fae x:0 0 0 2022-01-31T19:44:10.955 INFO:tasks.workunit.client.0.smithi167.stdout:5/549: dread - d5/d6/de/d22/f97 zero size 2022-01-31T19:44:10.955 INFO:tasks.workunit.client.0.smithi167.stdout:8/477: dread d6/ff [0,4194304] 0 2022-01-31T19:44:10.957 INFO:tasks.workunit.client.0.smithi167.stdout:9/282: link dd/d19/f1b dd/d19/d24/d56/d5b/f5c 0 2022-01-31T19:44:10.961 INFO:tasks.workunit.client.0.smithi167.stdout:6/286: link d8/c45 d8/d33/c71 0 2022-01-31T19:44:10.961 INFO:tasks.workunit.client.0.smithi167.stdout:6/287: truncate d8/d20/f22 4564063 0 2022-01-31T19:44:10.963 INFO:tasks.workunit.client.0.smithi167.stdout:9/283: write dd/ff [3868601,58042] 0 2022-01-31T19:44:10.964 INFO:tasks.workunit.client.0.smithi167.stdout:9/284: fdatasync dd/d34/f51 0 2022-01-31T19:44:10.964 INFO:tasks.workunit.client.0.smithi167.stdout:9/285: truncate dd/d19/d24/d3b/f43 922725 0 2022-01-31T19:44:10.964 INFO:tasks.workunit.client.0.smithi167.stdout:4/431: link d0/d6/d7/d26/d31/l10 d0/d6/d7/d26/d40/l8d 0 2022-01-31T19:44:10.964 INFO:tasks.workunit.client.0.smithi167.stdout:4/432: stat d0/d6/d7/d26/f72 0 2022-01-31T19:44:10.965 INFO:tasks.workunit.client.0.smithi167.stdout:8/478: dread d6/db/d4e/f58 [0,4194304] 0 2022-01-31T19:44:10.966 INFO:tasks.workunit.client.0.smithi167.stdout:7/513: link d8/d2c/d43/l50 d8/d2c/d6e/d4d/d5e/d71/db3/lba 0 2022-01-31T19:44:10.967 INFO:tasks.workunit.client.0.smithi167.stdout:9/286: rmdir dd/d19/d24/d3b/d4c 39 2022-01-31T19:44:10.968 INFO:tasks.workunit.client.0.smithi167.stdout:4/433: mknod d0/d6/d7/d39/d4e/d61/c8e 0 2022-01-31T19:44:10.968 INFO:tasks.workunit.client.0.smithi167.stdout:4/434: write d0/d6/d7/d26/f2f [240789,46498] 0 2022-01-31T19:44:10.969 INFO:tasks.workunit.client.0.smithi167.stdout:8/479: rmdir d6/d1e/d4c 39 2022-01-31T19:44:10.969 INFO:tasks.workunit.client.0.smithi167.stdout:8/480: chown d6/db/d28 0 1 2022-01-31T19:44:10.970 INFO:tasks.workunit.client.0.smithi167.stdout:7/514: dread d8/fa [0,4194304] 0 2022-01-31T19:44:10.970 INFO:tasks.workunit.client.0.smithi167.stdout:9/287: mkdir dd/d19/d24/d50/d5d 0 2022-01-31T19:44:10.971 INFO:tasks.workunit.client.0.smithi167.stdout:9/288: creat dd/d19/d24/d3b/f5e x:0 0 0 2022-01-31T19:44:10.971 INFO:tasks.workunit.client.0.smithi167.stdout:4/435: link d0/d51/l68 d0/d6/l8f 0 2022-01-31T19:44:10.975 INFO:tasks.workunit.client.0.smithi167.stdout:8/481: symlink d6/db/d6a/d8d/l9d 0 2022-01-31T19:44:10.975 INFO:tasks.workunit.client.0.smithi167.stdout:7/515: rename d8/d2c/d6c to d8/dc/d18/d24/dbb 0 2022-01-31T19:44:10.978 INFO:tasks.workunit.client.0.smithi167.stdout:9/289: creat dd/d19/d24/d3b/d4c/f5f x:0 0 0 2022-01-31T19:44:10.979 INFO:tasks.workunit.client.0.smithi167.stdout:9/290: truncate dd/fe 1584525 0 2022-01-31T19:44:10.980 INFO:tasks.workunit.client.0.smithi167.stdout:7/516: symlink d8/d2c/d43/lbc 0 2022-01-31T19:44:10.981 INFO:tasks.workunit.client.0.smithi167.stdout:7/517: fsync d8/d2c/d6e/d4d/d67/f7e 0 2022-01-31T19:44:10.982 INFO:tasks.workunit.client.0.smithi167.stdout:9/291: rename dd/d19/d24/d3b/f46 to dd/d35/f60 0 2022-01-31T19:44:10.989 INFO:tasks.workunit.client.0.smithi167.stdout:8/482: dread d6/f57 [0,4194304] 0 2022-01-31T19:44:10.992 INFO:tasks.workunit.client.0.smithi167.stdout:8/483: dread d6/db/d4e/d73/f7b [0,4194304] 0 2022-01-31T19:44:11.008 INFO:tasks.workunit.client.0.smithi167.stdout:8/484: dwrite d6/d1e/d2d/f40 [0,4194304] 0 2022-01-31T19:44:11.008 INFO:tasks.workunit.client.0.smithi167.stdout:8/485: fdatasync d6/f14 0 2022-01-31T19:44:11.011 INFO:tasks.workunit.client.0.smithi167.stdout:8/486: rename d6/db/d2f/d51 to d6/db/d6a/d8d/d9e 0 2022-01-31T19:44:11.013 INFO:tasks.workunit.client.0.smithi167.stdout:8/487: symlink d6/db/d6a/d8d/d9e/l9f 0 2022-01-31T19:44:11.013 INFO:tasks.workunit.client.0.smithi167.stdout:8/488: fdatasync d6/db/d6a/f7c 0 2022-01-31T19:44:11.014 INFO:tasks.workunit.client.0.smithi167.stdout:8/489: symlink d6/la0 0 2022-01-31T19:44:11.014 INFO:tasks.workunit.client.0.smithi167.stdout:8/490: mknod d6/db/d6a/d8d/ca1 0 2022-01-31T19:44:11.015 INFO:tasks.workunit.client.0.smithi167.stdout:8/491: truncate d6/f8 5426205 0 2022-01-31T19:44:11.022 INFO:tasks.workunit.client.0.smithi167.stdout:8/492: dread d6/db/d28/f5a [0,4194304] 0 2022-01-31T19:44:11.026 INFO:tasks.workunit.client.0.smithi167.stdout:6/288: dwrite d8/f3f [8388608,4194304] 0 2022-01-31T19:44:11.027 INFO:tasks.workunit.client.0.smithi167.stdout:5/550: dwrite d5/d6/de/f37 [4194304,4194304] 0 2022-01-31T19:44:11.028 INFO:tasks.workunit.client.0.smithi167.stdout:5/551: fsync d5/d6/de/d22/d76/f80 0 2022-01-31T19:44:11.028 INFO:tasks.workunit.client.0.smithi167.stdout:5/552: readlink d5/d6/de/d22/l77 0 2022-01-31T19:44:11.029 INFO:tasks.workunit.client.0.smithi167.stdout:5/553: read d5/f8 [5254752,8626] 0 2022-01-31T19:44:11.032 INFO:tasks.workunit.client.0.smithi167.stdout:6/289: write d8/f1a [20642,10164] 0 2022-01-31T19:44:11.038 INFO:tasks.workunit.client.0.smithi167.stdout:5/554: dread d5/d6/d16/d18/d43/d45/f83 [0,4194304] 0 2022-01-31T19:44:11.038 INFO:tasks.workunit.client.0.smithi167.stdout:5/555: chown d5/d6/d16/d18/f2c 3572298 1 2022-01-31T19:44:11.043 INFO:tasks.workunit.client.0.smithi167.stdout:6/290: mkdir d8/d33/d51/d72 0 2022-01-31T19:44:11.044 INFO:tasks.workunit.client.0.smithi167.stdout:6/291: readlink d8/lc 0 2022-01-31T19:44:11.045 INFO:tasks.workunit.client.0.smithi167.stdout:7/518: dwrite d8/d2c/d6e/d4d/d67/d8f/f96 [0,4194304] 0 2022-01-31T19:44:11.047 INFO:tasks.workunit.client.0.smithi167.stdout:8/493: dwrite d6/db/de/d18/f1b [0,4194304] 0 2022-01-31T19:44:11.049 INFO:tasks.workunit.client.0.smithi167.stdout:5/556: symlink d5/d61/laf 0 2022-01-31T19:44:11.049 INFO:tasks.workunit.client.0.smithi167.stdout:5/557: fdatasync d5/d6/d16/d18/f59 0 2022-01-31T19:44:11.077 INFO:tasks.workunit.client.0.smithi167.stdout:8/494: dread d6/f8 [0,4194304] 0 2022-01-31T19:44:11.078 INFO:tasks.workunit.client.0.smithi167.stdout:6/292: dwrite d8/f5b [0,4194304] 0 2022-01-31T19:44:11.078 INFO:tasks.workunit.client.0.smithi167.stdout:7/519: dwrite d8/dc/f21 [4194304,4194304] 0 2022-01-31T19:44:11.078 INFO:tasks.workunit.client.0.smithi167.stdout:5/558: dwrite d5/d6/de/d22/d4d/d65/f7c [4194304,4194304] 0 2022-01-31T19:44:11.079 INFO:tasks.workunit.client.0.smithi167.stdout:8/495: creat d6/db/d4e/d73/fa2 x:0 0 0 2022-01-31T19:44:11.079 INFO:tasks.workunit.client.0.smithi167.stdout:8/496: chown d6/db/d28/c46 96530 1 2022-01-31T19:44:11.081 INFO:tasks.workunit.client.0.smithi167.stdout:5/559: getdents d5/d6/d16/d18/d43/d45/d69/d90 0 2022-01-31T19:44:11.081 INFO:tasks.workunit.client.0.smithi167.stdout:5/560: readlink d5/d6/l5d 0 2022-01-31T19:44:11.082 INFO:tasks.workunit.client.0.smithi167.stdout:8/497: symlink d6/db/d36/la3 0 2022-01-31T19:44:11.082 INFO:tasks.workunit.client.0.smithi167.stdout:8/498: write d6/db/de/d18/f21 [4914645,33969] 0 2022-01-31T19:44:11.083 INFO:tasks.workunit.client.0.smithi167.stdout:7/520: truncate d8/d2c/d43/f97 437562 0 2022-01-31T19:44:11.085 INFO:tasks.workunit.client.0.smithi167.stdout:7/521: write d8/d2c/d6e/f2a [2179083,44902] 0 2022-01-31T19:44:11.088 INFO:tasks.workunit.client.0.smithi167.stdout:8/499: dread d6/db/fd [0,4194304] 0 2022-01-31T19:44:11.090 INFO:tasks.workunit.client.0.smithi167.stdout:5/561: dread d5/d6/de/d22/f5f [4194304,4194304] 0 2022-01-31T19:44:11.093 INFO:tasks.workunit.client.0.smithi167.stdout:5/562: write d5/d6/de/f37 [6467862,29606] 0 2022-01-31T19:44:11.096 INFO:tasks.workunit.client.0.smithi167.stdout:5/563: unlink d5/d6/d3a/f8e 0 2022-01-31T19:44:11.097 INFO:tasks.workunit.client.0.smithi167.stdout:5/564: mkdir d5/d6/de/d22/d76/db0 0 2022-01-31T19:44:11.101 INFO:tasks.workunit.client.0.smithi167.stdout:6/293: dwrite d8/f1c [0,4194304] 0 2022-01-31T19:44:11.109 INFO:tasks.workunit.client.0.smithi167.stdout:6/294: rmdir d8 39 2022-01-31T19:44:11.114 INFO:tasks.workunit.client.0.smithi167.stdout:8/500: dwrite d6/d1e/f41 [0,4194304] 0 2022-01-31T19:44:11.116 INFO:tasks.workunit.client.0.smithi167.stdout:8/501: unlink d6/db/de/d18/f55 0 2022-01-31T19:44:11.118 INFO:tasks.workunit.client.0.smithi167.stdout:8/502: mkdir d6/db/de/d18/da4 0 2022-01-31T19:44:11.120 INFO:tasks.workunit.client.0.smithi167.stdout:7/522: dwrite d8/dc/f21 [0,4194304] 0 2022-01-31T19:44:11.123 INFO:tasks.workunit.client.0.smithi167.stdout:7/523: rename d8/dc/d18/d24/dbb/l7f to d8/d2c/d6e/d4d/d5e/d71/lbd 0 2022-01-31T19:44:11.123 INFO:tasks.workunit.client.0.smithi167.stdout:5/565: dwrite d5/d6/de/d22/d4d/fa1 [4194304,4194304] 0 2022-01-31T19:44:11.130 INFO:tasks.workunit.client.0.smithi167.stdout:6/295: dwrite d8/f1a [0,4194304] 0 2022-01-31T19:44:11.132 INFO:tasks.workunit.client.0.smithi167.stdout:7/524: rmdir d8/dab 39 2022-01-31T19:44:11.133 INFO:tasks.workunit.client.0.smithi167.stdout:5/566: symlink d5/d6/d16/d18/d43/d45/lb1 0 2022-01-31T19:44:11.134 INFO:tasks.workunit.client.0.smithi167.stdout:6/296: creat d8/f73 x:0 0 0 2022-01-31T19:44:11.135 INFO:tasks.workunit.client.0.smithi167.stdout:5/567: truncate d5/d6/d16/d18/f64 6988257 0 2022-01-31T19:44:11.136 INFO:tasks.workunit.client.0.smithi167.stdout:5/568: dread - d5/d6/d16/d18/d43/d45/f94 zero size 2022-01-31T19:44:11.136 INFO:tasks.workunit.client.0.smithi167.stdout:6/297: dread d8/d33/f32 [4194304,4194304] 0 2022-01-31T19:44:11.137 INFO:tasks.workunit.client.0.smithi167.stdout:6/298: rmdir d8/d33/d51/d5d 39 2022-01-31T19:44:11.138 INFO:tasks.workunit.client.0.smithi167.stdout:6/299: chown d8/d20/c27 111241 1 2022-01-31T19:44:11.141 INFO:tasks.workunit.client.0.smithi167.stdout:6/300: write d8/d20/f55 [3306549,43217] 0 2022-01-31T19:44:11.143 INFO:tasks.workunit.client.0.smithi167.stdout:7/525: dread d8/d2c/d6e/d4d/d67/d8f/f96 [0,4194304] 0 2022-01-31T19:44:11.144 INFO:tasks.workunit.client.0.smithi167.stdout:6/301: symlink d8/d33/d4f/l74 0 2022-01-31T19:44:11.145 INFO:tasks.workunit.client.0.smithi167.stdout:6/302: stat d8/d33/l37 0 2022-01-31T19:44:11.145 INFO:tasks.workunit.client.0.smithi167.stdout:6/303: write d8/d33/d35/f3d [5177288,79873] 0 2022-01-31T19:44:11.147 INFO:tasks.workunit.client.0.smithi167.stdout:6/304: creat d8/d33/d51/d72/f75 x:0 0 0 2022-01-31T19:44:11.150 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.148+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 --> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac40068ad0 con 0x7fac28061a10 2022-01-31T19:44:11.152 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.150+0000 7fac2effd700 1 -- 172.21.15.167:0/1526068842 <== mgr.24387 v2:172.21.15.171:6828/1254375478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+217 (secure 0 0 0) 0x7fac40068ad0 con 0x7fac28061a10 2022-01-31T19:44:11.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fac28061a10 msgr2=0x7fac28063ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.155 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fac28061a10 0x7fac28063ed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fac400fbc10 tx=0x7fac3400b040).stop 2022-01-31T19:44:11.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 msgr2=0x7fac400fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400fd470 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fac30000c00 tx=0x7fac3000b040).stop 2022-01-31T19:44:11.156 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 shutdown_connections 2022-01-31T19:44:11.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fac28061a10 0x7fac28063ed0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac400f5710 0x7fac400fd470 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 --2- 172.21.15.167:0/1526068842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac400f69b0 0x7fac400fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.157 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.153+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 >> 172.21.15.167:0/1526068842 conn(0x7fac400f0e90 msgr2=0x7fac400fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:11.158 INFO:tasks.workunit.client.0.smithi167.stdout:6/305: creat d8/d33/d4f/f76 x:0 0 0 2022-01-31T19:44:11.159 INFO:tasks.workunit.client.0.smithi167.stdout:6/306: chown d8/d20 4 1 2022-01-31T19:44:11.159 INFO:tasks.workunit.client.0.smithi167.stdout:6/307: write d8/d33/f4e [371861,96712] 0 2022-01-31T19:44:11.159 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.154+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 shutdown_connections 2022-01-31T19:44:11.160 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.154+0000 7fac448a3700 1 -- 172.21.15.167:0/1526068842 wait complete. 2022-01-31T19:44:11.161 INFO:tasks.workunit.client.0.smithi167.stdout:6/308: read d8/d20/f21 [898189,1360] 0 2022-01-31T19:44:11.161 INFO:tasks.workunit.client.0.smithi167.stdout:6/309: chown d8/d33/d51/d72/f75 44576 1 2022-01-31T19:44:11.161 INFO:tasks.workunit.client.0.smithi167.stdout:5/569: dwrite d5/d6/de/d22/d4d/d65/f7c [0,4194304] 0 2022-01-31T19:44:11.162 INFO:tasks.workunit.client.0.smithi167.stdout:6/310: link d8/d33/f5f d8/d62/f77 0 2022-01-31T19:44:11.162 INFO:tasks.workunit.client.0.smithi167.stdout:5/570: mknod d5/d6/d3a/cb2 0 2022-01-31T19:44:11.162 INFO:tasks.workunit.client.0.smithi167.stdout:6/311: rename d8/d20/c46 to d8/d20/d54/c78 0 2022-01-31T19:44:11.162 INFO:tasks.workunit.client.0.smithi167.stdout:6/312: readlink d8/d33/d4f/l70 0 2022-01-31T19:44:11.163 INFO:tasks.workunit.client.0.smithi167.stdout:6/313: chown d8/f5c 6711 1 2022-01-31T19:44:11.163 INFO:tasks.workunit.client.0.smithi167.stdout:6/314: chown d8/f1f 7 1 2022-01-31T19:44:11.164 INFO:tasks.workunit.client.0.smithi167.stdout:6/315: creat d8/d20/f79 x:0 0 0 2022-01-31T19:44:11.164 INFO:tasks.workunit.client.0.smithi167.stdout:7/526: dwrite d8/d35/f5d [0,4194304] 0 2022-01-31T19:44:11.164 INFO:tasks.workunit.client.0.smithi167.stdout:6/316: creat d8/f7a x:0 0 0 2022-01-31T19:44:11.172 INFO:tasks.workunit.client.0.smithi167.stdout:7/527: link d8/d2c/d6e/l30 d8/lbe 0 2022-01-31T19:44:11.173 INFO:tasks.workunit.client.0.smithi167.stdout:7/528: readlink d8/d2c/d43/l50 0 2022-01-31T19:44:11.173 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:44:11.174 INFO:tasks.workunit.client.0.smithi167.stdout:7/529: creat d8/d35/fbf x:0 0 0 2022-01-31T19:44:11.174 INFO:tasks.workunit.client.0.smithi167.stdout:7/530: chown d8/d2c/d6e/d4d/d5e/d71/db1/d4f 635 1 2022-01-31T19:44:11.176 INFO:tasks.workunit.client.0.smithi167.stdout:7/531: creat d8/d2c/d43/da1/fc0 x:0 0 0 2022-01-31T19:44:11.177 INFO:tasks.workunit.client.0.smithi167.stdout:7/532: dread d8/d2c/d6e/d4d/d67/f7e [0,4194304] 0 2022-01-31T19:44:11.177 INFO:tasks.workunit.client.0.smithi167.stdout:7/533: dread - d8/d2c/d6e/d4d/d5e/d71/db1/fad zero size 2022-01-31T19:44:11.189 INFO:tasks.workunit.client.0.smithi167.stdout:2/310: sync 2022-01-31T19:44:11.190 INFO:tasks.workunit.client.0.smithi167.stdout:3/403: sync 2022-01-31T19:44:11.190 INFO:tasks.workunit.client.0.smithi167.stdout:3/404: unlink d9/f1a 0 2022-01-31T19:44:11.191 INFO:tasks.workunit.client.0.smithi167.stdout:2/311: read f9 [4564800,75482] 0 2022-01-31T19:44:11.192 INFO:tasks.workunit.client.0.smithi167.stdout:2/312: chown dd/d18/f4e 18614 1 2022-01-31T19:44:11.192 INFO:tasks.workunit.client.0.smithi167.stdout:5/571: dwrite d5/d6/d3a/f4e [4194304,4194304] 0 2022-01-31T19:44:11.192 INFO:tasks.workunit.client.0.smithi167.stdout:6/317: dwrite d8/d33/d4f/f76 [0,4194304] 0 2022-01-31T19:44:11.193 INFO:tasks.workunit.client.0.smithi167.stdout:6/318: fdatasync d8/f3f 0 2022-01-31T19:44:11.193 INFO:tasks.workunit.client.0.smithi167.stdout:6/319: write d8/d20/f26 [2054502,78034] 0 2022-01-31T19:44:11.193 INFO:tasks.workunit.client.0.smithi167.stdout:3/405: rmdir d9/d29 39 2022-01-31T19:44:11.193 INFO:tasks.workunit.client.0.smithi167.stdout:3/406: chown d9/l14 7 1 2022-01-31T19:44:11.194 INFO:tasks.workunit.client.0.smithi167.stdout:7/534: dwrite d8/d2c/d6e/d4d/d67/fb9 [0,4194304] 0 2022-01-31T19:44:11.194 INFO:tasks.workunit.client.0.smithi167.stdout:7/535: readlink d8/d2c/d6e/d4d/d5e/d71/db1/l66 0 2022-01-31T19:44:11.195 INFO:tasks.workunit.client.0.smithi167.stdout:5/572: rmdir d5/d6/d16/d18/d91 0 2022-01-31T19:44:11.196 INFO:tasks.workunit.client.0.smithi167.stdout:2/313: dread dd/df/d1b/d2f/d36/f2a [0,4194304] 0 2022-01-31T19:44:11.196 INFO:tasks.workunit.client.0.smithi167.stdout:2/314: read - dd/df/d2c/f32 zero size 2022-01-31T19:44:11.196 INFO:tasks.workunit.client.0.smithi167.stdout:6/320: rename d8/d20/c24 to d8/d33/d51/d5d/d64/c7b 0 2022-01-31T19:44:11.199 INFO:tasks.workunit.client.0.smithi167.stdout:3/407: symlink d9/l74 0 2022-01-31T19:44:11.200 INFO:tasks.workunit.client.0.smithi167.stdout:3/408: write d9/f2c [3917952,10165] 0 2022-01-31T19:44:11.202 INFO:tasks.workunit.client.0.smithi167.stdout:7/536: mkdir d8/dc/dc1 0 2022-01-31T19:44:11.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/573: symlink d5/d6/d16/d18/d43/lb3 0 2022-01-31T19:44:11.204 INFO:tasks.workunit.client.0.smithi167.stdout:5/574: creat d5/d6/de/d22/d4d/d65/fb4 x:0 0 0 2022-01-31T19:44:11.205 INFO:tasks.workunit.client.0.smithi167.stdout:2/315: truncate dd/d18/f4e 1344802 0 2022-01-31T19:44:11.211 INFO:tasks.workunit.client.0.smithi167.stdout:6/321: mknod d8/d20/c7c 0 2022-01-31T19:44:11.212 INFO:tasks.workunit.client.0.smithi167.stdout:6/322: chown d8 15118240 1 2022-01-31T19:44:11.212 INFO:tasks.workunit.client.0.smithi167.stdout:3/409: getdents d9 0 2022-01-31T19:44:11.213 INFO:tasks.workunit.client.0.smithi167.stdout:5/575: dread d5/d61/f4b [0,4194304] 0 2022-01-31T19:44:11.213 INFO:tasks.workunit.client.0.smithi167.stdout:6/323: mknod d8/d33/c7d 0 2022-01-31T19:44:11.214 INFO:tasks.workunit.client.0.smithi167.stdout:6/324: dread - d8/d62/f6d zero size 2022-01-31T19:44:11.214 INFO:tasks.workunit.client.0.smithi167.stdout:2/316: dread dd/d18/f31 [0,4194304] 0 2022-01-31T19:44:11.215 INFO:tasks.workunit.client.0.smithi167.stdout:3/410: rename d9/d29/d3b/d41/d6b/l48 to d9/d29/l75 0 2022-01-31T19:44:11.215 INFO:tasks.workunit.client.0.smithi167.stdout:3/411: chown d9/d56/f6d 186522114 1 2022-01-31T19:44:11.215 INFO:tasks.workunit.client.0.smithi167.stdout:3/412: readlink d9/d29/d60/l45 0 2022-01-31T19:44:11.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/413: chown f7 997 1 2022-01-31T19:44:11.216 INFO:tasks.workunit.client.0.smithi167.stdout:6/325: dread f3 [0,4194304] 0 2022-01-31T19:44:11.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/414: creat d9/d29/f76 x:0 0 0 2022-01-31T19:44:11.217 INFO:tasks.workunit.client.0.smithi167.stdout:2/317: getdents dd/df/d1b/d2f 0 2022-01-31T19:44:11.218 INFO:tasks.workunit.client.0.smithi167.stdout:2/318: dread - dd/df/f50 zero size 2022-01-31T19:44:11.218 INFO:tasks.workunit.client.0.smithi167.stdout:2/319: stat dd/df/d3f/d51 0 2022-01-31T19:44:11.219 INFO:tasks.workunit.client.0.smithi167.stdout:6/326: mkdir d8/d33/d51/d7e 0 2022-01-31T19:44:11.221 INFO:tasks.workunit.client.0.smithi167.stdout:2/320: rename dd/df/d1b/d3d/c58 to dd/df/d1b/d2f/c5a 0 2022-01-31T19:44:11.221 INFO:tasks.workunit.client.0.smithi167.stdout:2/321: fsync dd/f15 0 2022-01-31T19:44:11.222 INFO:tasks.workunit.client.0.smithi167.stdout:6/327: creat d8/d33/d51/d7e/f7f x:0 0 0 2022-01-31T19:44:11.222 INFO:tasks.workunit.client.0.smithi167.stdout:6/328: stat d8/d33/d35/f3d 0 2022-01-31T19:44:11.223 INFO:tasks.workunit.client.0.smithi167.stdout:6/329: chown l6 8653919 1 2022-01-31T19:44:11.223 INFO:tasks.workunit.client.0.smithi167.stdout:6/330: write d8/d20/f42 [743852,10521] 0 2022-01-31T19:44:11.223 INFO:tasks.workunit.client.0.smithi167.stdout:2/322: truncate dd/df/d1b/d2f/d36/f2a 1086796 0 2022-01-31T19:44:11.233 INFO:tasks.workunit.client.0.smithi167.stdout:7/537: dwrite d8/d35/f3d [0,4194304] 0 2022-01-31T19:44:11.279 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3323504234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f5ce0 msgr2=0x7ff9540f6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.279 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3323504234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f5ce0 0x7ff9540f6100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7ff944004660 tx=0x7ff944009b30).stop 2022-01-31T19:44:11.279 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3323504234 shutdown_connections 2022-01-31T19:44:11.280 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3323504234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9540f6680 0x7ff9540f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.280 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3323504234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f5ce0 0x7ff9540f6100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.280 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3323504234 >> 172.21.15.167:0/3323504234 conn(0x7ff9540f05f0 msgr2=0x7ff9540f2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:11.280 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3323504234 shutdown_connections 2022-01-31T19:44:11.281 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.275+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3323504234 wait complete. 2022-01-31T19:44:11.281 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95c7ae700 1 Processor -- start 2022-01-31T19:44:11.281 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95c7ae700 1 -- start start 2022-01-31T19:44:11.281 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95c7ae700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9540f5ce0 0x7ff954109e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:11.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95c7ae700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 0x7ff95410c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:11.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95c7ae700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9540f57d0 con 0x7ff9540f6680 2022-01-31T19:44:11.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95c7ae700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9540f5910 con 0x7ff9540f5ce0 2022-01-31T19:44:11.282 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 0x7ff95410c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:11.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 0x7ff95410c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34112/0 (socket says 172.21.15.167:34112) 2022-01-31T19:44:11.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 -- 172.21.15.167:0/3276428305 learned_addr learned my addr 172.21.15.167:0/3276428305 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:11.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff95a54a700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9540f5ce0 0x7ff954109e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:11.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 -- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9540f5ce0 msgr2=0x7ff954109e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.283 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9540f5ce0 0x7ff954109e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 -- 172.21.15.167:0/3276428305 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944005040 con 0x7ff9540f6680 2022-01-31T19:44:11.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.276+0000 7ff959d49700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 0x7ff95410c350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff95000c790 tx=0x7ff950007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:11.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.277+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff950002590 con 0x7ff9540f6680 2022-01-31T19:44:11.284 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.277+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95410c900 con 0x7ff9540f6680 2022-01-31T19:44:11.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.277+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7ff950002ba0 con 0x7ff9540f6680 2022-01-31T19:44:11.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.277+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff950013d90 con 0x7ff9540f6680 2022-01-31T19:44:11.285 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.277+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95410cec0 con 0x7ff9540f6680 2022-01-31T19:44:11.286 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.278+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff954045bb0 con 0x7ff9540f6680 2022-01-31T19:44:11.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.278+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 19) v1 ==== 78465+0+0 (secure 0 0 0) 0x7ff95001a060 con 0x7ff9540f6680 2022-01-31T19:44:11.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.278+0000 7ff94affd700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7ff940061a00 0x7ff940063ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:11.287 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.279+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5343+0+0 (secure 0 0 0) 0x7ff95007d1f0 con 0x7ff9540f6680 2022-01-31T19:44:11.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.279+0000 7ff95a54a700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7ff940061a00 0x7ff940063ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:11.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.279+0000 7ff95a54a700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7ff940061a00 0x7ff940063ec0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7ff944004630 tx=0x7ff944009b30).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:11.288 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.283+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7ff95004b890 con 0x7ff9540f6680 2022-01-31T19:44:11.319 INFO:tasks.workunit.client.0.smithi167.stdout:6/331: creat d8/d33/d51/d5d/f80 x:0 0 0 2022-01-31T19:44:11.320 INFO:tasks.workunit.client.0.smithi167.stdout:2/323: creat dd/df/d1b/d3d/d4a/f5b x:0 0 0 2022-01-31T19:44:11.320 INFO:tasks.workunit.client.0.smithi167.stdout:2/324: unlink dd/c21 0 2022-01-31T19:44:11.402 INFO:tasks.workunit.client.0.smithi167.stdout:5/576: truncate d5/d6/d3a/f4e 4772921 0 2022-01-31T19:44:11.409 INFO:tasks.workunit.client.0.smithi167.stdout:7/538: truncate d8/d2c/d6e/d4d/d67/f7e 349625 0 2022-01-31T19:44:11.413 INFO:tasks.workunit.client.0.smithi167.stdout:6/332: link d8/d33/d51/l56 d8/d33/d51/d7e/l81 0 2022-01-31T19:44:11.420 INFO:tasks.workunit.client.0.smithi167.stdout:6/333: dread d8/d62/f6f [0,4194304] 0 2022-01-31T19:44:11.439 INFO:tasks.workunit.client.0.smithi167.stdout:6/334: dwrite d8/d33/d51/d7e/f7f [0,4194304] 0 2022-01-31T19:44:11.440 INFO:tasks.workunit.client.0.smithi167.stdout:6/335: write d8/d20/f42 [713951,18489] 0 2022-01-31T19:44:11.440 INFO:tasks.workunit.client.0.smithi167.stdout:6/336: chown d8/d20/f21 1650 1 2022-01-31T19:44:11.463 INFO:tasks.workunit.client.0.smithi167.stdout:1/493: sync 2022-01-31T19:44:11.493 INFO:tasks.workunit.client.0.smithi167.stdout:5/577: symlink d5/d6/d16/lb5 0 2022-01-31T19:44:11.494 INFO:tasks.workunit.client.0.smithi167.stdout:7/539: creat d8/d2c/d6e/d4d/d5e/d71/db1/d4f/fc2 x:0 0 0 2022-01-31T19:44:11.494 INFO:tasks.workunit.client.0.smithi167.stdout:7/540: readlink d8/dc/d18/l2d 0 2022-01-31T19:44:11.494 INFO:tasks.workunit.client.0.smithi167.stdout:7/541: chown d8/d2c/d6e/d4d/d67/d8f 5521689 1 2022-01-31T19:44:11.496 INFO:tasks.workunit.client.0.smithi167.stdout:1/494: mknod d0/d10/d5a/d33/caf 0 2022-01-31T19:44:11.499 INFO:tasks.workunit.client.0.smithi167.stdout:5/578: creat d5/d6/de/d22/d76/db0/fb6 x:0 0 0 2022-01-31T19:44:11.500 INFO:tasks.workunit.client.0.smithi167.stdout:5/579: read d5/d6/d16/d18/f33 [297675,48736] 0 2022-01-31T19:44:11.501 INFO:tasks.workunit.client.0.smithi167.stdout:6/337: dread d8/d20/d54/f63 [0,4194304] 0 2022-01-31T19:44:11.501 INFO:tasks.workunit.client.0.smithi167.stdout:7/542: dread d8/d35/f4b [0,4194304] 0 2022-01-31T19:44:11.503 INFO:tasks.workunit.client.0.smithi167.stdout:1/495: creat d0/d11/fb0 x:0 0 0 2022-01-31T19:44:11.503 INFO:tasks.workunit.client.0.smithi167.stdout:1/496: fdatasync d0/d10/d5a/d33/f95 0 2022-01-31T19:44:11.504 INFO:tasks.workunit.client.0.smithi167.stdout:5/580: creat d5/d6/de/d22/d6d/fb7 x:0 0 0 2022-01-31T19:44:11.506 INFO:tasks.workunit.client.0.smithi167.stdout:6/338: unlink d8/d33/d35/l48 0 2022-01-31T19:44:11.506 INFO:tasks.workunit.client.0.smithi167.stdout:6/339: dread - d8/d33/d4f/f53 zero size 2022-01-31T19:44:11.507 INFO:tasks.workunit.client.0.smithi167.stdout:6/340: creat d8/d33/f82 x:0 0 0 2022-01-31T19:44:11.508 INFO:tasks.workunit.client.0.smithi167.stdout:6/341: fsync d8/d33/f40 0 2022-01-31T19:44:11.508 INFO:tasks.workunit.client.0.smithi167.stdout:6/342: creat d8/d33/d51/d72/f83 x:0 0 0 2022-01-31T19:44:11.510 INFO:tasks.workunit.client.0.smithi167.stdout:1/497: rmdir d0/d62 39 2022-01-31T19:44:11.510 INFO:tasks.workunit.client.0.smithi167.stdout:1/498: write d0/d10/d5a/d33/d4b/f65 [152545,3089] 0 2022-01-31T19:44:11.510 INFO:tasks.workunit.client.0.smithi167.stdout:1/499: write d0/d10/d5a/d33/d4b/f5b [4207617,5547] 0 2022-01-31T19:44:11.512 INFO:tasks.workunit.client.0.smithi167.stdout:6/343: mknod d8/d33/c84 0 2022-01-31T19:44:11.512 INFO:tasks.workunit.client.0.smithi167.stdout:6/344: write d8/d33/f82 [872543,102650] 0 2022-01-31T19:44:11.514 INFO:tasks.workunit.client.0.smithi167.stdout:5/581: dread d5/d6/d3a/f4e [0,4194304] 0 2022-01-31T19:44:11.514 INFO:tasks.workunit.client.0.smithi167.stdout:5/582: chown d5/f8 917 1 2022-01-31T19:44:11.515 INFO:tasks.workunit.client.0.smithi167.stdout:5/583: chown d5/d6/cd 28488146 1 2022-01-31T19:44:11.515 INFO:tasks.workunit.client.0.smithi167.stdout:5/584: chown d5/d6/de/d22/d76 162 1 2022-01-31T19:44:11.517 INFO:tasks.workunit.client.0.smithi167.stdout:1/500: link d0/d75/d7a/ca4 d0/d75/d7a/cb1 0 2022-01-31T19:44:11.520 INFO:tasks.workunit.client.0.smithi167.stdout:1/501: write d0/d10/d5a/f50 [683559,23037] 0 2022-01-31T19:44:11.521 INFO:tasks.workunit.client.0.smithi167.stdout:1/502: creat d0/d10/d20/fb2 x:0 0 0 2022-01-31T19:44:11.521 INFO:tasks.workunit.client.0.smithi167.stdout:5/585: truncate d5/d61/f74 4329441 0 2022-01-31T19:44:11.521 INFO:tasks.workunit.client.0.smithi167.stdout:5/586: fdatasync d5/f7 0 2022-01-31T19:44:11.522 INFO:tasks.workunit.client.0.smithi167.stdout:5/587: read d5/d6/d3a/f4e [282614,41543] 0 2022-01-31T19:44:11.522 INFO:tasks.workunit.client.0.smithi167.stdout:5/588: stat d5/d6/d16/d18/f58 0 2022-01-31T19:44:11.523 INFO:tasks.workunit.client.0.smithi167.stdout:5/589: creat d5/d6/de/d22/d4d/fb8 x:0 0 0 2022-01-31T19:44:11.523 INFO:tasks.workunit.client.0.smithi167.stdout:5/590: chown d5/d6/de/d22/d76/lac 9 1 2022-01-31T19:44:11.524 INFO:tasks.workunit.client.0.smithi167.stdout:1/503: mkdir d0/db3 0 2022-01-31T19:44:11.525 INFO:tasks.workunit.client.0.smithi167.stdout:5/591: getdents d5/d6/d16/d18/d5b/d9f 0 2022-01-31T19:44:11.525 INFO:tasks.workunit.client.0.smithi167.stdout:7/543: dwrite d8/d2c/d6e/d4d/d67/f7e [0,4194304] 0 2022-01-31T19:44:11.526 INFO:tasks.workunit.client.0.smithi167.stdout:1/504: mkdir d0/d75/db4 0 2022-01-31T19:44:11.527 INFO:tasks.workunit.client.0.smithi167.stdout:1/505: write d0/d10/d5a/d33/d4b/da5/fa0 [796356,41214] 0 2022-01-31T19:44:11.530 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.528+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 --> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9540fdfa0 con 0x7ff940061a00 2022-01-31T19:44:11.531 INFO:tasks.workunit.client.0.smithi167.stdout:1/506: rmdir d0/d10/d5a/d33/d4b/da5/da9 39 2022-01-31T19:44:11.531 INFO:tasks.workunit.client.0.smithi167.stdout:7/544: rmdir d8/d2c/d6e/d4d/d5e/d71/db3 39 2022-01-31T19:44:11.532 INFO:tasks.workunit.client.0.smithi167.stdout:6/345: dwrite d8/d20/f79 [0,4194304] 0 2022-01-31T19:44:11.535 INFO:tasks.workunit.client.0.smithi167.stdout:6/346: read d8/d33/f3c [2157391,79542] 0 2022-01-31T19:44:11.537 INFO:tasks.workunit.client.0.smithi167.stdout:9/292: sync 2022-01-31T19:44:11.538 INFO:tasks.workunit.client.0.smithi167.stdout:4/436: sync 2022-01-31T19:44:11.538 INFO:tasks.workunit.client.0.smithi167.stdout:0/211: sync 2022-01-31T19:44:11.538 INFO:tasks.workunit.client.0.smithi167.stdout:4/437: dread - d0/d6/d7/d39/d4e/f52 zero size 2022-01-31T19:44:11.539 INFO:tasks.workunit.client.0.smithi167.stdout:4/438: chown d0/c70 1 1 2022-01-31T19:44:11.539 INFO:tasks.workunit.client.0.smithi167.stdout:9/293: write dd/d19/f3d [5529611,112023] 0 2022-01-31T19:44:11.539 INFO:tasks.workunit.client.0.smithi167.stdout:4/439: creat d0/d3f/d60/d65/f90 x:0 0 0 2022-01-31T19:44:11.540 INFO:tasks.workunit.client.0.smithi167.stdout:9/294: creat dd/d34/f61 x:0 0 0 2022-01-31T19:44:11.540 INFO:tasks.workunit.client.0.smithi167.stdout:0/212: write d17/f22 [4122679,86058] 0 2022-01-31T19:44:11.541 INFO:tasks.workunit.client.0.smithi167.stdout:0/213: truncate d17/d33/f36 128924 0 2022-01-31T19:44:11.541 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.538+0000 7ff94affd700 1 -- 172.21.15.167:0/3276428305 <== mgr.24387 v2:172.21.15.171:6828/1254375478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7ff9540fdfa0 con 0x7ff940061a00 2022-01-31T19:44:11.541 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:44:11.542 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (4m) 2m ago 5m 22.0M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:44:11.542 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (5m) 2m ago 5m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:44:11.542 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (5m) 20s ago 5m 7151k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:44:11.542 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (4m) 2m ago 5m 31.6M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:44:11.543 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.rdkazj smithi167 running (2m) 2m ago 2m 14.4M - 16.2.5 46cf6318c64e 29f596ef1666 2022-01-31T19:44:11.543 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.sgnfie smithi167 running (2m) 2m ago 2m 16.6M - 16.2.5 46cf6318c64e 8c462c58dec4 2022-01-31T19:44:11.543 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.dyqqtj smithi171 running (2m) 20s ago 2m 234M - 16.2.5 46cf6318c64e 5e84b5044af1 2022-01-31T19:44:11.543 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.erzfea smithi171 running (2m) 20s ago 2m 81.2M - 16.2.5 46cf6318c64e 802065eafb38 2022-01-31T19:44:11.544 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:9283 running 2m ago 6m - - 2022-01-31T19:44:11.544 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (27s) 20s ago 5m 48.9M - 17.0.0-10469-g29e1fc17 e02b80923a35 56429afd72ba 2022-01-31T19:44:11.544 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running 2m ago 7m - 2048M 2022-01-31T19:44:11.544 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (5m) 20s ago 5m 101M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:44:11.545 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (5m) 2m ago 5m 18.1M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:44:11.545 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (5m) 20s ago 5m 18.2M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:44:11.545 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (4m) 2m ago 4m 49.6M 6472M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:44:11.545 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (4m) 2m ago 4m 49.4M 6472M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:44:11.546 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (4m) 2m ago 4m 49.0M 6472M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:44:11.546 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (3m) 20s ago 3m 164M 5339M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:44:11.546 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (3m) 20s ago 3m 133M 5339M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:44:11.546 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (3m) 20s ago 3m 159M 5339M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:44:11.546 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (4m) 2m ago 5m 40.1M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:44:11.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.540+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7ff940061a00 msgr2=0x7ff940063ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7ff940061a00 0x7ff940063ec0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7ff944004630 tx=0x7ff944009b30).stop 2022-01-31T19:44:11.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 msgr2=0x7ff95410c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 0x7ff95410c350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff95000c790 tx=0x7ff950007d70).stop 2022-01-31T19:44:11.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 shutdown_connections 2022-01-31T19:44:11.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7ff940061a00 0x7ff940063ec0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9540f5ce0 0x7ff954109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 --2- 172.21.15.167:0/3276428305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9540f6680 0x7ff95410c350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 >> 172.21.15.167:0/3276428305 conn(0x7ff9540f05f0 msgr2=0x7ff9540fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:11.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 shutdown_connections 2022-01-31T19:44:11.550 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.541+0000 7ff95c7ae700 1 -- 172.21.15.167:0/3276428305 wait complete. 2022-01-31T19:44:11.552 INFO:tasks.workunit.client.0.smithi167.stdout:1/507: creat d0/d10/d5a/fb5 x:0 0 0 2022-01-31T19:44:11.553 INFO:tasks.workunit.client.0.smithi167.stdout:1/508: write d0/d10/d5a/d33/d4b/f6d [40143,15630] 0 2022-01-31T19:44:11.553 INFO:tasks.workunit.client.0.smithi167.stdout:1/509: readlink d0/d10/d5a/d33/d4b/d1f/l4c 0 2022-01-31T19:44:11.554 INFO:tasks.workunit.client.0.smithi167.stdout:1/510: write d0/d10/d18/f91 [761140,27722] 0 2022-01-31T19:44:11.556 INFO:tasks.workunit.client.0.smithi167.stdout:7/545: mkdir d8/d2c/d6e/d4d/d6b/d8e/dc3 0 2022-01-31T19:44:11.557 INFO:tasks.workunit.client.0.smithi167.stdout:7/546: readlink d8/d2c/d6e/l51 0 2022-01-31T19:44:11.557 INFO:tasks.workunit.client.0.smithi167.stdout:7/547: write d8/dc/d18/d24/dbb/f79 [3395578,127258] 0 2022-01-31T19:44:11.562 INFO:tasks.workunit.client.0.smithi167.stdout:6/347: creat d8/d20/d54/f85 x:0 0 0 2022-01-31T19:44:11.565 INFO:tasks.workunit.client.0.smithi167.stdout:4/440: creat d0/d6/d7/d39/d3a/d79/f91 x:0 0 0 2022-01-31T19:44:11.565 INFO:tasks.workunit.client.0.smithi167.stdout:4/441: truncate d0/d3f/d60/d65/f88 211071 0 2022-01-31T19:44:11.566 INFO:tasks.workunit.client.0.smithi167.stdout:9/295: rmdir dd/d19/d24/d56 39 2022-01-31T19:44:11.566 INFO:tasks.workunit.client.0.smithi167.stdout:0/214: symlink d17/d2c/l41 0 2022-01-31T19:44:11.569 INFO:tasks.workunit.client.0.smithi167.stdout:7/548: truncate f7 1013127 0 2022-01-31T19:44:11.569 INFO:tasks.workunit.client.0.smithi167.stdout:7/549: dread - d8/d2c/d6e/d4d/d5e/d71/db1/fad zero size 2022-01-31T19:44:11.570 INFO:tasks.workunit.client.0.smithi167.stdout:6/348: rmdir d8/d33/d51/d72 39 2022-01-31T19:44:11.571 INFO:tasks.workunit.client.0.smithi167.stdout:7/550: dread d8/dc/f2b [0,4194304] 0 2022-01-31T19:44:11.571 INFO:tasks.workunit.client.0.smithi167.stdout:7/551: truncate d8/dc/d18/d24/f73 1351144 0 2022-01-31T19:44:11.572 INFO:tasks.workunit.client.0.smithi167.stdout:1/511: dwrite d0/d10/d5a/d33/d4b/f6d [0,4194304] 0 2022-01-31T19:44:11.572 INFO:tasks.workunit.client.0.smithi167.stdout:9/296: dread dd/d17/f36 [0,4194304] 0 2022-01-31T19:44:11.573 INFO:tasks.workunit.client.0.smithi167.stdout:0/215: write ff [1022804,123174] 0 2022-01-31T19:44:11.573 INFO:tasks.workunit.client.0.smithi167.stdout:1/512: creat d0/d10/d20/fb6 x:0 0 0 2022-01-31T19:44:11.579 INFO:tasks.workunit.client.0.smithi167.stdout:0/216: rename d17/f26 to d17/d3e/f42 0 2022-01-31T19:44:11.579 INFO:tasks.workunit.client.0.smithi167.stdout:7/552: fsync d8/dc/d18/d24/dbb/f79 0 2022-01-31T19:44:11.581 INFO:tasks.workunit.client.0.smithi167.stdout:9/297: mknod dd/c62 0 2022-01-31T19:44:11.582 INFO:tasks.workunit.client.0.smithi167.stdout:9/298: truncate dd/d19/d24/d3b/f4a 4369030 0 2022-01-31T19:44:11.584 INFO:tasks.workunit.client.0.smithi167.stdout:1/513: dread d0/d62/d7f/d82/f88 [0,4194304] 0 2022-01-31T19:44:11.585 INFO:tasks.workunit.client.0.smithi167.stdout:0/217: getdents d17/d19 0 2022-01-31T19:44:11.585 INFO:tasks.workunit.client.0.smithi167.stdout:1/514: truncate d0/f25 338948 0 2022-01-31T19:44:11.587 INFO:tasks.workunit.client.0.smithi167.stdout:0/218: link d17/d2c/f3c d17/f43 0 2022-01-31T19:44:11.589 INFO:tasks.workunit.client.0.smithi167.stdout:1/515: mkdir d0/d62/db7 0 2022-01-31T19:44:11.589 INFO:tasks.workunit.client.0.smithi167.stdout:0/219: read fa [2862122,110927] 0 2022-01-31T19:44:11.589 INFO:tasks.workunit.client.0.smithi167.stdout:0/220: chown d17/d19/f1f 10340897 1 2022-01-31T19:44:11.594 INFO:tasks.workunit.client.0.smithi167.stdout:6/349: dwrite d8/f1a [0,4194304] 0 2022-01-31T19:44:11.594 INFO:tasks.workunit.client.0.smithi167.stdout:6/350: write d8/f73 [951520,104482] 0 2022-01-31T19:44:11.595 INFO:tasks.workunit.client.0.smithi167.stdout:6/351: mkdir d8/d33/d51/d5d/d86 0 2022-01-31T19:44:11.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/299: dread dd/d19/f27 [0,4194304] 0 2022-01-31T19:44:11.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/300: write dd/d19/d24/f45 [1022861,95412] 0 2022-01-31T19:44:11.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/301: truncate dd/d17/f1f 1091607 0 2022-01-31T19:44:11.599 INFO:tasks.workunit.client.0.smithi167.stdout:1/516: write d0/d11/f4d [3451432,35135] 0 2022-01-31T19:44:11.600 INFO:tasks.workunit.client.0.smithi167.stdout:1/517: dread - d0/d75/f8d zero size 2022-01-31T19:44:11.600 INFO:tasks.workunit.client.0.smithi167.stdout:0/221: dread d17/d3e/f42 [0,4194304] 0 2022-01-31T19:44:11.600 INFO:tasks.workunit.client.0.smithi167.stdout:1/518: stat d0/d10/d5a/f50 0 2022-01-31T19:44:11.600 INFO:tasks.workunit.client.0.smithi167.stdout:0/222: chown d17/d19/d1c/f34 1516084 1 2022-01-31T19:44:11.601 INFO:tasks.workunit.client.0.smithi167.stdout:0/223: fsync d17/f2f 0 2022-01-31T19:44:11.601 INFO:tasks.workunit.client.0.smithi167.stdout:1/519: creat d0/d10/d20/d7c/fb8 x:0 0 0 2022-01-31T19:44:11.601 INFO:tasks.workunit.client.0.smithi167.stdout:1/520: fdatasync d0/d75/d7a/f59 0 2022-01-31T19:44:11.601 INFO:tasks.workunit.client.0.smithi167.stdout:9/302: rename dd/d17/f55 to dd/d19/d24/f63 0 2022-01-31T19:44:11.601 INFO:tasks.workunit.client.0.smithi167.stdout:7/553: dwrite d8/dc/f40 [4194304,4194304] 0 2022-01-31T19:44:11.602 INFO:tasks.workunit.client.0.smithi167.stdout:7/554: chown d8/d2c/d6e/d4d/d5e/d71/db1/d4f/c9d 475909712 1 2022-01-31T19:44:11.602 INFO:tasks.workunit.client.0.smithi167.stdout:7/555: chown d8/fa7 8 1 2022-01-31T19:44:11.603 INFO:tasks.workunit.client.0.smithi167.stdout:4/442: dwrite d0/d6/d7/f30 [0,4194304] 0 2022-01-31T19:44:11.607 INFO:tasks.workunit.client.0.smithi167.stdout:1/521: link d0/d10/f42 d0/d10/d5a/fb9 0 2022-01-31T19:44:11.609 INFO:tasks.workunit.client.0.smithi167.stdout:9/303: creat dd/f64 x:0 0 0 2022-01-31T19:44:11.611 INFO:tasks.workunit.client.0.smithi167.stdout:7/556: mknod d8/d2c/d6e/d4d/d6b/cc4 0 2022-01-31T19:44:11.613 INFO:tasks.workunit.client.0.smithi167.stdout:4/443: symlink d0/d6/d7/d39/d3a/l92 0 2022-01-31T19:44:11.617 INFO:tasks.workunit.client.0.smithi167.stdout:9/304: rmdir dd/d19 39 2022-01-31T19:44:11.617 INFO:tasks.workunit.client.0.smithi167.stdout:9/305: write dd/d35/f33 [818799,125783] 0 2022-01-31T19:44:11.618 INFO:tasks.workunit.client.0.smithi167.stdout:9/306: creat dd/d35/f65 x:0 0 0 2022-01-31T19:44:11.618 INFO:tasks.workunit.client.0.smithi167.stdout:9/307: chown l2 367614 1 2022-01-31T19:44:11.618 INFO:tasks.workunit.client.0.smithi167.stdout:9/308: fsync dd/f10 0 2022-01-31T19:44:11.619 INFO:tasks.workunit.client.0.smithi167.stdout:9/309: stat dd/l1d 0 2022-01-31T19:44:11.619 INFO:tasks.workunit.client.0.smithi167.stdout:9/310: stat dd/d19/d24/c49 0 2022-01-31T19:44:11.620 INFO:tasks.workunit.client.0.smithi167.stdout:7/557: truncate d8/d2c/d6e/d4d/d5e/f91 3391985 0 2022-01-31T19:44:11.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/224: rmdir d17 39 2022-01-31T19:44:11.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/225: dread - d17/f43 zero size 2022-01-31T19:44:11.621 INFO:tasks.workunit.client.0.smithi167.stdout:4/444: write d0/d3f/f3d [1471369,52770] 0 2022-01-31T19:44:11.623 INFO:tasks.workunit.client.0.smithi167.stdout:9/311: symlink dd/d35/l66 0 2022-01-31T19:44:11.624 INFO:tasks.workunit.client.0.smithi167.stdout:7/558: stat d8/d2c/d6e/d4d/d5e/d71/l98 0 2022-01-31T19:44:11.626 INFO:tasks.workunit.client.0.smithi167.stdout:6/352: dwrite d8/f1f [0,4194304] 0 2022-01-31T19:44:11.626 INFO:tasks.workunit.client.0.smithi167.stdout:9/312: read dd/d17/f2f [3959595,99763] 0 2022-01-31T19:44:11.627 INFO:tasks.workunit.client.0.smithi167.stdout:0/226: mkdir d17/d19/d1c/d39/d44 0 2022-01-31T19:44:11.627 INFO:tasks.workunit.client.0.smithi167.stdout:1/522: dwrite d0/d10/d5a/d33/d4b/f98 [0,4194304] 0 2022-01-31T19:44:11.628 INFO:tasks.workunit.client.0.smithi167.stdout:7/559: symlink d8/d2c/d6e/d4d/d6b/d8e/lc5 0 2022-01-31T19:44:11.628 INFO:tasks.workunit.client.0.smithi167.stdout:7/560: readlink d8/dc/d18/d24/dbb/l95 0 2022-01-31T19:44:11.636 INFO:tasks.workunit.client.0.smithi167.stdout:6/353: write d8/d20/f44 [2712658,91267] 0 2022-01-31T19:44:11.642 INFO:tasks.workunit.client.0.smithi167.stdout:0/227: mknod d17/d19/d1c/d39/d44/c45 0 2022-01-31T19:44:11.646 INFO:tasks.workunit.client.0.smithi167.stdout:1/523: creat d0/d62/d7f/d82/fba x:0 0 0 2022-01-31T19:44:11.646 INFO:tasks.workunit.client.0.smithi167.stdout:1/524: readlink d0/d10/d5a/d33/d97/l9d 0 2022-01-31T19:44:11.648 INFO:tasks.workunit.client.0.smithi167.stdout:9/313: rmdir dd/d17 39 2022-01-31T19:44:11.648 INFO:tasks.workunit.client.0.smithi167.stdout:0/228: truncate d17/f3d 2367241 0 2022-01-31T19:44:11.649 INFO:tasks.workunit.client.0.smithi167.stdout:6/354: getdents d8/d33/d51/d7e 0 2022-01-31T19:44:11.651 INFO:tasks.workunit.client.0.smithi167.stdout:9/314: getdents dd/d17 0 2022-01-31T19:44:11.651 INFO:tasks.workunit.client.0.smithi167.stdout:9/315: creat dd/d19/d24/d3b/d4c/f67 x:0 0 0 2022-01-31T19:44:11.651 INFO:tasks.workunit.client.0.smithi167.stdout:9/316: chown dd/d35/f44 387305 1 2022-01-31T19:44:11.652 INFO:tasks.workunit.client.0.smithi167.stdout:9/317: write dd/d19/f21 [3605677,128828] 0 2022-01-31T19:44:11.652 INFO:tasks.workunit.client.0.smithi167.stdout:6/355: mkdir d8/d33/d4f/d87 0 2022-01-31T19:44:11.654 INFO:tasks.workunit.client.0.smithi167.stdout:7/561: dread d8/d2c/d43/f2f [0,4194304] 0 2022-01-31T19:44:11.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/562: write d8/fa [134060,63774] 0 2022-01-31T19:44:11.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/563: write d8/d2c/d6e/d4d/d5e/d71/db3/fb8 [437668,112268] 0 2022-01-31T19:44:11.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/564: readlink d8/d2c/d43/lbc 0 2022-01-31T19:44:11.655 INFO:tasks.workunit.client.0.smithi167.stdout:7/565: chown d8/d35/f36 6254296 1 2022-01-31T19:44:11.656 INFO:tasks.workunit.client.0.smithi167.stdout:6/356: write d8/f3f [2857799,65139] 0 2022-01-31T19:44:11.657 INFO:tasks.workunit.client.0.smithi167.stdout:7/566: link d8/d2c/d6e/d4d/d5e/d71/db1/l66 d8/d2c/d6e/d4d/lc6 0 2022-01-31T19:44:11.658 INFO:tasks.workunit.client.0.smithi167.stdout:4/445: dwrite d0/d6/d7/d39/d4e/f52 [0,4194304] 0 2022-01-31T19:44:11.659 INFO:tasks.workunit.client.0.smithi167.stdout:7/567: write d8/d2c/d6e/d4d/d5e/d71/db1/f6a [516067,31201] 0 2022-01-31T19:44:11.663 INFO:tasks.workunit.client.0.smithi167.stdout:7/568: symlink d8/d2c/d6e/d4d/d5e/d71/db1/d8d/lc7 0 2022-01-31T19:44:11.665 INFO:tasks.workunit.client.0.smithi167.stdout:7/569: creat d8/d2c/d7a/fc8 x:0 0 0 2022-01-31T19:44:11.665 INFO:tasks.workunit.client.0.smithi167.stdout:7/570: fsync d8/fa7 0 2022-01-31T19:44:11.678 INFO:tasks.workunit.client.0.smithi167.stdout:6/357: dread d8/f3f [4194304,4194304] 0 2022-01-31T19:44:11.679 INFO:tasks.workunit.client.0.smithi167.stdout:0/229: write d17/f2f [2987915,125557] 0 2022-01-31T19:44:11.679 INFO:tasks.workunit.client.0.smithi167.stdout:0/230: chown d17/d19/d1c/d39/d44/c45 12 1 2022-01-31T19:44:11.680 INFO:tasks.workunit.client.0.smithi167.stdout:6/358: creat d8/d33/d51/d5d/d64/f88 x:0 0 0 2022-01-31T19:44:11.684 INFO:tasks.workunit.client.0.smithi167.stdout:0/231: unlink f15 0 2022-01-31T19:44:11.685 INFO:tasks.workunit.client.0.smithi167.stdout:6/359: symlink d8/d33/d51/d5d/d86/l89 0 2022-01-31T19:44:11.691 INFO:tasks.workunit.client.0.smithi167.stdout:0/232: dread f10 [0,4194304] 0 2022-01-31T19:44:11.701 INFO:tasks.workunit.client.0.smithi167.stdout:9/318: dwrite f7 [0,4194304] 0 2022-01-31T19:44:11.701 INFO:tasks.workunit.client.0.smithi167.stdout:1/525: dwrite d0/d10/d5a/d33/d4b/d1f/fa6 [0,4194304] 0 2022-01-31T19:44:11.702 INFO:tasks.workunit.client.0.smithi167.stdout:1/526: creat d0/d10/d5a/d33/fbb x:0 0 0 2022-01-31T19:44:11.702 INFO:tasks.workunit.client.0.smithi167.stdout:1/527: chown d0/d10/d20/d7c/fb8 4274 1 2022-01-31T19:44:11.702 INFO:tasks.workunit.client.0.smithi167.stdout:9/319: unlink dd/d19/d24/d3b/d4c/l52 0 2022-01-31T19:44:11.702 INFO:tasks.workunit.client.0.smithi167.stdout:9/320: chown dd/d19/f27 0 1 2022-01-31T19:44:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.697+0000 7f99688ce700 1 -- 172.21.15.167:0/308926930 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9964069270 msgr2=0x7f9964102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.697+0000 7f99688ce700 1 --2- 172.21.15.167:0/308926930 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9964069270 0x7f9964102cb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f9958004660 tx=0x7f9958009b30).stop 2022-01-31T19:44:11.703 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.697+0000 7f99688ce700 1 -- 172.21.15.167:0/308926930 shutdown_connections 2022-01-31T19:44:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.697+0000 7f99688ce700 1 --2- 172.21.15.167:0/308926930 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9964069270 0x7f9964102cb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.697+0000 7f99688ce700 1 --2- 172.21.15.167:0/308926930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99640688d0 0x7f9964068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:44:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.697+0000 7f99688ce700 1 -- 172.21.15.167:0/308926930 >> 172.21.15.167:0/308926930 conn(0x7f99640637e0 msgr2=0x7f9964065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:11.704 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.698+0000 7f99688ce700 1 -- 172.21.15.167:0/308926930 shutdown_connections 2022-01-31T19:44:11.705 INFO:tasks.workunit.client.0.smithi167.stdout:9/321: rename dd/l1d to dd/d34/l68 0 2022-01-31T19:44:11.705 INFO:tasks.workunit.client.0.smithi167.stdout:0/233: dread d17/d3e/f42 [0,4194304] 0 2022-01-31T19:44:11.705 INFO:tasks.workunit.client.0.smithi167.stdout:0/234: truncate d17/d19/f1f 250975 0 2022-01-31T19:44:11.706 INFO:tasks.workunit.client.0.smithi167.stdout:4/446: dwrite d0/d6/d7/d39/f37 [4194304,4194304] 0 2022-01-31T19:44:11.706 INFO:tasks.workunit.client.0.smithi167.stdout:4/447: read - d0/d6/d7/d26/d31/f5e zero size 2022-01-31T19:44:11.706 INFO:tasks.workunit.client.0.smithi167.stdout:4/448: fdatasync d0/d3f/d60/f77 0 2022-01-31T19:44:11.706 INFO:tasks.workunit.client.0.smithi167.stdout:7/571: dwrite d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f1a [0,4194304] 0 2022-01-31T19:44:11.707 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.698+0000 7f99688ce700 1 -- 172.21.15.167:0/308926930 wait complete. 2022-01-31T19:44:11.707 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.699+0000 7f99688ce700 1 Processor -- start 2022-01-31T19:44:11.708 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.699+0000 7f99688ce700 1 -- start start 2022-01-31T19:44:11.708 INFO:tasks.workunit.client.0.smithi167.stdout:6/360: dread d8/fb [0,4194304] 0 2022-01-31T19:44:11.709 INFO:tasks.workunit.client.0.smithi167.stdout:6/361: chown d8/f5b 7150637 1 2022-01-31T19:44:11.709 INFO:tasks.workunit.client.0.smithi167.stdout:6/362: readlink d8/d33/d51/d5d/d86/l89 0 2022-01-31T19:44:11.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.699+0000 7f99688ce700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 0x7f9964110380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:11.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.699+0000 7f99688ce700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964069270 0x7f9964112880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:11.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.699+0000 7f99688ce700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99640ffb20 con 0x7f99640688d0 2022-01-31T19:44:11.710 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.699+0000 7f99688ce700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99640ffc60 con 0x7f9964069270 2022-01-31T19:44:11.711 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.700+0000 7f996259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 0x7f9964110380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:11.711 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.700+0000 7f996259c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 0x7f9964110380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34122/0 (socket says 172.21.15.167:34122) 2022-01-31T19:44:11.711 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.700+0000 7f996259c700 1 -- 172.21.15.167:0/847153096 learned_addr learned my addr 172.21.15.167:0/847153096 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:11.712 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.700+0000 7f9961d9b700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964069270 0x7f9964112880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:11.712 INFO:tasks.workunit.client.0.smithi167.stdout:9/322: read dd/d19/f1b [3526074,11423] 0 2022-01-31T19:44:11.712 INFO:tasks.workunit.client.0.smithi167.stdout:9/323: read - dd/d34/f51 zero size 2022-01-31T19:44:11.713 INFO:tasks.workunit.client.0.smithi167.stdout:9/324: write dd/d34/f51 [1024836,94522] 0 2022-01-31T19:44:11.714 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.701+0000 7f996259c700 1 -- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964069270 msgr2=0x7f9964112880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:11.714 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.701+0000 7f996259c700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964069270 0x7f9964112880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:11.714 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.701+0000 7f996259c700 1 -- 172.21.15.167:0/847153096 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958005040 con 0x7f99640688d0 2022-01-31T19:44:11.715 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.702+0000 7f996259c700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 0x7f9964110380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f99540093f0 tx=0x7f9954006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:11.715 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.702+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9954013640 con 0x7f99640688d0 2022-01-31T19:44:11.715 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.702+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f9954013c50 con 0x7f99640688d0 2022-01-31T19:44:11.715 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.702+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9954012c70 con 0x7f99640688d0 2022-01-31T19:44:11.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.702+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9964112ee0 con 0x7f99640688d0 2022-01-31T19:44:11.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.703+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99641134a0 con 0x7f99640688d0 2022-01-31T19:44:11.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.705+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 19) v1 ==== 78465+0+0 (secure 0 0 0) 0x7f995401a060 con 0x7f99640688d0 2022-01-31T19:44:11.716 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.706+0000 7f9952ffd700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7f994c061a60 0x7f994c063f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:11.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.706+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5343+0+0 (secure 0 0 0) 0x7f995407c450 con 0x7f99640688d0 2022-01-31T19:44:11.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.706+0000 7f9961d9b700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7f994c061a60 0x7f994c063f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:11.717 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.707+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948004fa0 con 0x7f99640688d0 2022-01-31T19:44:11.718 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.707+0000 7f9961d9b700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7f994c061a60 0x7f994c063f20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9958004630 tx=0x7f9958009b30).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:11.718 INFO:tasks.workunit.client.0.smithi167.stdout:9/325: dread dd/d34/f51 [0,4194304] 0 2022-01-31T19:44:11.719 INFO:tasks.workunit.client.0.smithi167.stdout:4/449: mkdir d0/d6/d7/d39/d93 0 2022-01-31T19:44:11.719 INFO:tasks.workunit.client.0.smithi167.stdout:6/363: mknod d8/c8a 0 2022-01-31T19:44:11.719 INFO:tasks.workunit.client.0.smithi167.stdout:6/364: fsync d8/fd 0 2022-01-31T19:44:11.719 INFO:tasks.workunit.client.0.smithi167.stdout:9/326: symlink dd/d19/d24/d3b/d4c/l69 0 2022-01-31T19:44:11.720 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:11.714+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7f995404abc0 con 0x7f99640688d0 2022-01-31T19:44:11.721 INFO:tasks.workunit.client.0.smithi167.stdout:6/365: creat d8/d33/d35/f8b x:0 0 0 2022-01-31T19:44:11.722 INFO:tasks.workunit.client.0.smithi167.stdout:9/327: mknod dd/d19/d24/d50/c6a 0 2022-01-31T19:44:11.722 INFO:tasks.workunit.client.0.smithi167.stdout:9/328: mkdir dd/d6b 0 2022-01-31T19:44:11.722 INFO:tasks.workunit.client.0.smithi167.stdout:9/329: read dd/d17/f36 [74657,72917] 0 2022-01-31T19:44:11.723 INFO:tasks.workunit.client.0.smithi167.stdout:6/366: dread d8/d33/f4e [0,4194304] 0 2022-01-31T19:44:11.723 INFO:tasks.workunit.client.0.smithi167.stdout:0/235: dwrite f14 [0,4194304] 0 2022-01-31T19:44:11.723 INFO:tasks.workunit.client.0.smithi167.stdout:0/236: stat d17/d19/d1c/f21 0 2022-01-31T19:44:11.724 INFO:tasks.workunit.client.0.smithi167.stdout:0/237: chown ff 154 1 2022-01-31T19:44:11.724 INFO:tasks.workunit.client.0.smithi167.stdout:1/528: dwrite d0/d10/d18/f91 [0,4194304] 0 2022-01-31T19:44:11.725 INFO:tasks.workunit.client.0.smithi167.stdout:7/572: truncate d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f1a 3706659 0 2022-01-31T19:44:11.725 INFO:tasks.workunit.client.0.smithi167.stdout:0/238: link d17/d19/d1c/c27 d17/d19/d1c/d39/c46 0 2022-01-31T19:44:11.725 INFO:tasks.workunit.client.0.smithi167.stdout:7/573: write d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f68 [321399,10754] 0 2022-01-31T19:44:11.728 INFO:tasks.workunit.client.0.smithi167.stdout:7/574: symlink d8/d2c/d6e/d4d/d5e/d71/lc9 0 2022-01-31T19:44:11.730 INFO:tasks.workunit.client.0.smithi167.stdout:0/239: getdents d17/d2c 0 2022-01-31T19:44:11.731 INFO:tasks.workunit.client.0.smithi167.stdout:7/575: rename d8/d2c/d6e/f49 to d8/dc/d18/fca 0 2022-01-31T19:44:11.734 INFO:tasks.workunit.client.0.smithi167.stdout:7/576: mkdir d8/d2c/d6e/d4d/d67/d8f/dcb 0 2022-01-31T19:44:11.736 INFO:tasks.workunit.client.0.smithi167.stdout:7/577: mkdir d8/dc/d18/d24/dcc 0 2022-01-31T19:44:11.736 INFO:tasks.workunit.client.0.smithi167.stdout:7/578: write d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f68 [858492,84746] 0 2022-01-31T19:44:11.736 INFO:tasks.workunit.client.0.smithi167.stdout:7/579: creat d8/d2c/d7a/fcd x:0 0 0 2022-01-31T19:44:11.737 INFO:tasks.workunit.client.0.smithi167.stdout:7/580: chown d8/d2c/d7a/f9c 1279836544 1 2022-01-31T19:44:11.738 INFO:tasks.workunit.client.0.smithi167.stdout:7/581: mkdir d8/d2c/d6e/d4d/d5e/dce 0 2022-01-31T19:44:11.738 INFO:tasks.workunit.client.0.smithi167.stdout:7/582: write d8/d2c/d6e/d4d/d6b/f88 [629760,23507] 0 2022-01-31T19:44:11.741 INFO:tasks.workunit.client.0.smithi167.stdout:8/503: sync 2022-01-31T19:44:11.742 INFO:tasks.workunit.client.0.smithi167.stdout:2/325: sync 2022-01-31T19:44:11.747 INFO:tasks.workunit.client.0.smithi167.stdout:0/240: dread d17/d19/d1c/f32 [0,4194304] 0 2022-01-31T19:44:11.748 INFO:tasks.workunit.client.0.smithi167.stdout:0/241: truncate d17/d3e/f42 4122741 0 2022-01-31T19:44:11.749 INFO:tasks.workunit.client.0.smithi167.stdout:0/242: truncate f10 1394140 0 2022-01-31T19:44:11.753 INFO:tasks.workunit.client.0.smithi167.stdout:2/326: dread f9 [0,4194304] 0 2022-01-31T19:44:11.755 INFO:tasks.workunit.client.0.smithi167.stdout:3/415: sync 2022-01-31T19:44:11.763 INFO:tasks.workunit.client.0.smithi167.stdout:8/504: dread d6/db/d28/f5a [0,4194304] 0 2022-01-31T19:44:11.764 INFO:tasks.workunit.client.0.smithi167.stdout:8/505: rmdir d6/d1e/d4c 39 2022-01-31T19:44:11.767 INFO:tasks.workunit.client.0.smithi167.stdout:3/416: write d9/d29/d3b/d41/d6b/f35 [3990823,120576] 0 2022-01-31T19:44:11.775 INFO:tasks.workunit.client.0.smithi167.stdout:9/330: dwrite dd/d35/f65 [0,4194304] 0 2022-01-31T19:44:11.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:11 smithi167 conmon[32206]: cluster 2022-01-31T19:44:10. 2022-01-31T19:44:11.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:11 smithi167 conmon[32206]: 431891+0000 mgr.smithi171.asyxnc (mgr.24387) 15 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 803 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 MiB/s rd, 90 MiB/s wr, 477 op/s 2022-01-31T19:44:11.787 INFO:tasks.workunit.client.0.smithi167.stdout:6/367: dwrite d8/d33/d51/d5d/d64/f68 [0,4194304] 0 2022-01-31T19:44:11.787 INFO:tasks.workunit.client.0.smithi167.stdout:1/529: dwrite d0/d10/d20/d7c/faa [0,4194304] 0 2022-01-31T19:44:11.787 INFO:tasks.workunit.client.0.smithi167.stdout:1/530: readlink d0/l96 0 2022-01-31T19:44:11.788 INFO:tasks.workunit.client.0.smithi167.stdout:1/531: write d0/d75/d7a/f66 [110030,111998] 0 2022-01-31T19:44:11.790 INFO:tasks.workunit.client.0.smithi167.stdout:6/368: symlink d8/d33/l8c 0 2022-01-31T19:44:11.790 INFO:tasks.workunit.client.0.smithi167.stdout:6/369: dread - d8/d33/d51/d5d/f60 zero size 2022-01-31T19:44:11.791 INFO:tasks.workunit.client.0.smithi167.stdout:8/506: dwrite d6/db/d6a/d8d/d9e/f67 [0,4194304] 0 2022-01-31T19:44:11.791 INFO:tasks.workunit.client.0.smithi167.stdout:8/507: rename d6/db to d6/db/d6a/da5 22 2022-01-31T19:44:11.794 INFO:tasks.workunit.client.0.smithi167.stdout:4/450: dwrite d0/d3f/f3d [0,4194304] 0 2022-01-31T19:44:11.795 INFO:tasks.workunit.client.0.smithi167.stdout:7/583: dwrite d8/d2c/d6e/d4d/d5e/fa9 [0,4194304] 0 2022-01-31T19:44:11.796 INFO:tasks.workunit.client.0.smithi167.stdout:3/417: dwrite d9/f27 [0,4194304] 0 2022-01-31T19:44:11.797 INFO:tasks.workunit.client.0.smithi167.stdout:1/532: link d0/d10/d5a/d33/d4b/da5/c61 d0/d11/cbc 0 2022-01-31T19:44:11.797 INFO:tasks.workunit.client.0.smithi167.stdout:9/331: truncate dd/d19/f3d 3157943 0 2022-01-31T19:44:11.798 INFO:tasks.workunit.client.0.smithi167.stdout:6/370: creat d8/d33/d51/d5d/d86/f8d x:0 0 0 2022-01-31T19:44:11.799 INFO:tasks.workunit.client.0.smithi167.stdout:6/371: creat d8/d33/d4f/f8e x:0 0 0 2022-01-31T19:44:11.799 INFO:tasks.workunit.client.0.smithi167.stdout:6/372: chown d8/d20/d54/f63 250982415 1 2022-01-31T19:44:11.799 INFO:tasks.workunit.client.0.smithi167.stdout:6/373: dread - d8/d33/d51/d72/f75 zero size 2022-01-31T19:44:11.800 INFO:tasks.workunit.client.0.smithi167.stdout:9/332: write f7 [3920623,70334] 0 2022-01-31T19:44:11.801 INFO:tasks.workunit.client.0.smithi167.stdout:8/508: rename d6/l72 to d6/db/d36/la6 0 2022-01-31T19:44:11.803 INFO:tasks.workunit.client.0.smithi167.stdout:9/333: dread dd/d35/f33 [0,4194304] 0 2022-01-31T19:44:11.804 INFO:tasks.workunit.client.0.smithi167.stdout:4/451: creat d0/d3f/d60/f94 x:0 0 0 2022-01-31T19:44:11.805 INFO:tasks.workunit.client.0.smithi167.stdout:2/327: dwrite dd/df/d1b/f56 [4194304,4194304] 0 2022-01-31T19:44:11.810 INFO:tasks.workunit.client.0.smithi167.stdout:7/584: mknod d8/dc/d18/d24/dbb/ccf 0 2022-01-31T19:44:11.810 INFO:tasks.workunit.client.0.smithi167.stdout:7/585: fdatasync d8/d2c/d7a/fc8 0 2022-01-31T19:44:11.811 INFO:tasks.workunit.client.0.smithi167.stdout:1/533: mkdir d0/d75/dbd 0 2022-01-31T19:44:11.812 INFO:tasks.workunit.client.0.smithi167.stdout:9/334: mknod dd/d17/c6c 0 2022-01-31T19:44:11.816 INFO:tasks.workunit.client.0.smithi167.stdout:8/509: dread d6/db/de/d3d/f71 [0,4194304] 0 2022-01-31T19:44:11.816 INFO:tasks.workunit.client.0.smithi167.stdout:8/510: readlink d6/l54 0 2022-01-31T19:44:11.816 INFO:tasks.workunit.client.0.smithi167.stdout:8/511: creat d6/db/de/d3d/fa7 x:0 0 0 2022-01-31T19:44:11.817 INFO:tasks.workunit.client.0.smithi167.stdout:7/586: write d8/d2c/d6e/f44 [2999971,75050] 0 2022-01-31T19:44:11.817 INFO:tasks.workunit.client.0.smithi167.stdout:4/452: link d0/d6/d7/d39/f37 d0/d6/d7/d39/d4e/d61/f95 0 2022-01-31T19:44:11.830 INFO:tasks.workunit.client.0.smithi167.stdout:1/534: symlink d0/d62/lbe 0 2022-01-31T19:44:11.831 INFO:tasks.workunit.client.0.smithi167.stdout:3/418: dwrite d9/f42 [0,4194304] 0 2022-01-31T19:44:11.831 INFO:tasks.workunit.client.0.smithi167.stdout:3/419: readlink l5 0 2022-01-31T19:44:11.832 INFO:tasks.workunit.client.0.smithi167.stdout:9/335: creat dd/d19/f6d x:0 0 0 2022-01-31T19:44:11.832 INFO:tasks.workunit.client.0.smithi167.stdout:9/336: write dd/d35/f44 [1001453,119722] 0 2022-01-31T19:44:11.833 INFO:tasks.workunit.client.0.smithi167.stdout:9/337: unlink dd/d19/d24/f42 0 2022-01-31T19:44:11.833 INFO:tasks.workunit.client.0.smithi167.stdout:9/338: truncate dd/f64 329315 0 2022-01-31T19:44:11.833 INFO:tasks.workunit.client.0.smithi167.stdout:7/587: mkdir d8/dab/dd0 0 2022-01-31T19:44:11.834 INFO:tasks.workunit.client.0.smithi167.stdout:0/243: dwrite d17/f22 [0,4194304] 0 2022-01-31T19:44:11.834 INFO:tasks.workunit.client.0.smithi167.stdout:0/244: dread - d17/d19/f2d zero size 2022-01-31T19:44:11.835 INFO:tasks.workunit.client.0.smithi167.stdout:6/374: dwrite d8/d20/f21 [0,4194304] 0 2022-01-31T19:44:11.835 INFO:tasks.workunit.client.0.smithi167.stdout:6/375: stat d8/d33/d35/f8b 0 2022-01-31T19:44:11.835 INFO:tasks.workunit.client.0.smithi167.stdout:1/535: mknod d0/d10/d18/cbf 0 2022-01-31T19:44:11.839 INFO:tasks.workunit.client.0.smithi167.stdout:3/420: rmdir d9/d29/d3b/d70 0 2022-01-31T19:44:11.839 INFO:tasks.workunit.client.0.smithi167.stdout:3/421: readlink d9/d29/d60/l62 0 2022-01-31T19:44:11.841 INFO:tasks.workunit.client.0.smithi167.stdout:9/339: mknod dd/d19/d24/d3b/c6e 0 2022-01-31T19:44:11.842 INFO:tasks.workunit.client.0.smithi167.stdout:4/453: dread d0/d3f/d60/d65/f88 [0,4194304] 0 2022-01-31T19:44:11.843 INFO:tasks.workunit.client.0.smithi167.stdout:4/454: fdatasync d0/d6/d7/f4a 0 2022-01-31T19:44:11.843 INFO:tasks.workunit.client.0.smithi167.stdout:4/455: write d0/d6/d7/d26/d31/f19 [4772056,105225] 0 2022-01-31T19:44:11.843 INFO:tasks.workunit.client.0.smithi167.stdout:4/456: stat d0/d6/d7/f7b 0 2022-01-31T19:44:11.844 INFO:tasks.workunit.client.0.smithi167.stdout:1/536: mknod d0/d11/cc0 0 2022-01-31T19:44:11.844 INFO:tasks.workunit.client.0.smithi167.stdout:4/457: symlink d0/d6/d7/d39/d4e/l96 0 2022-01-31T19:44:11.845 INFO:tasks.workunit.client.0.smithi167.stdout:1/537: write d0/f5 [1295030,62946] 0 2022-01-31T19:44:11.846 INFO:tasks.workunit.client.0.smithi167.stdout:4/458: creat d0/d6/d7/d89/f97 x:0 0 0 2022-01-31T19:44:11.848 INFO:tasks.workunit.client.0.smithi167.stdout:0/245: dread d17/d19/d1c/f21 [0,4194304] 0 2022-01-31T19:44:11.848 INFO:tasks.workunit.client.0.smithi167.stdout:3/422: dread d9/f30 [0,4194304] 0 2022-01-31T19:44:11.848 INFO:tasks.workunit.client.0.smithi167.stdout:8/512: dwrite d6/d1e/d2d/f40 [4194304,4194304] 0 2022-01-31T19:44:11.852 INFO:tasks.workunit.client.0.smithi167.stdout:3/423: rename f8 to d9/d21/f77 0 2022-01-31T19:44:11.857 INFO:tasks.workunit.client.0.smithi167.stdout:9/340: dread dd/d17/f2f [0,4194304] 0 2022-01-31T19:44:11.857 INFO:tasks.workunit.client.0.smithi167.stdout:0/246: mknod d17/d2c/c47 0 2022-01-31T19:44:11.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/424: creat d9/f78 x:0 0 0 2022-01-31T19:44:11.866 INFO:tasks.workunit.client.0.smithi167.stdout:3/425: chown d9/d26/l5e 90023 1 2022-01-31T19:44:11.867 INFO:tasks.workunit.client.0.smithi167.stdout:9/341: symlink dd/d19/d24/d56/d5b/l6f 0 2022-01-31T19:44:11.867 INFO:tasks.workunit.client.0.smithi167.stdout:9/342: fsync dd/d19/d24/d3b/f54 0 2022-01-31T19:44:11.868 INFO:tasks.workunit.client.0.smithi167.stdout:0/247: mknod d17/d3e/c48 0 2022-01-31T19:44:11.871 INFO:tasks.workunit.client.0.smithi167.stdout:9/343: getdents dd/d34 0 2022-01-31T19:44:11.872 INFO:tasks.workunit.client.0.smithi167.stdout:9/344: dread - dd/d19/d24/d3b/d4c/f5f zero size 2022-01-31T19:44:11.874 INFO:tasks.workunit.client.0.smithi167.stdout:9/345: getdents dd/d35 0 2022-01-31T19:44:11.875 INFO:tasks.workunit.client.0.smithi167.stdout:9/346: mknod dd/d19/d59/c70 0 2022-01-31T19:44:11.878 INFO:tasks.workunit.client.0.smithi167.stdout:9/347: write dd/d35/f37 [7084376,44774] 0 2022-01-31T19:44:11.878 INFO:tasks.workunit.client.0.smithi167.stdout:9/348: chown dd/d17/c30 866 1 2022-01-31T19:44:11.879 INFO:tasks.workunit.client.0.smithi167.stdout:9/349: rename dd/d19/c4e to dd/d19/c71 0 2022-01-31T19:44:11.880 INFO:tasks.workunit.client.0.smithi167.stdout:1/538: dwrite d0/d75/f8d [0,4194304] 0 2022-01-31T19:44:11.880 INFO:tasks.workunit.client.0.smithi167.stdout:1/539: chown d0/d10/d5a/c44 2182302 1 2022-01-31T19:44:11.881 INFO:tasks.workunit.client.0.smithi167.stdout:8/513: dwrite d6/d1e/f29 [0,4194304] 0 2022-01-31T19:44:11.881 INFO:tasks.workunit.client.0.smithi167.stdout:7/588: dwrite d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:11.882 INFO:tasks.workunit.client.0.smithi167.stdout:9/350: dread dd/d35/f44 [0,4194304] 0 2022-01-31T19:44:11.882 INFO:tasks.workunit.client.0.smithi167.stdout:9/351: truncate dd/d19/f2d 8760134 0 2022-01-31T19:44:11.884 INFO:tasks.workunit.client.0.smithi167.stdout:7/589: rmdir d8/d2c/d6e/d4d/d5e/d71/db1 39 2022-01-31T19:44:11.885 INFO:tasks.workunit.client.0.smithi167.stdout:3/426: dwrite d9/d56/f6d [0,4194304] 0 2022-01-31T19:44:11.889 INFO:tasks.workunit.client.0.smithi167.stdout:7/590: mkdir d8/d2c/d7a/dd1 0 2022-01-31T19:44:11.890 INFO:tasks.workunit.client.0.smithi167.stdout:7/591: rename d8 to d8/dc/d18/d24/dcc/dd2 22 2022-01-31T19:44:11.892 INFO:tasks.workunit.client.0.smithi167.stdout:3/427: truncate d9/f18 3003197 0 2022-01-31T19:44:11.892 INFO:tasks.workunit.client.0.smithi167.stdout:3/428: stat d9/d29/d60/f5b 0 2022-01-31T19:44:11.893 INFO:tasks.workunit.client.0.smithi167.stdout:3/429: creat d9/f79 x:0 0 0 2022-01-31T19:44:11.893 INFO:tasks.workunit.client.0.smithi167.stdout:7/592: mkdir d8/d2c/d7a/dd3 0 2022-01-31T19:44:11.895 INFO:tasks.workunit.client.0.smithi167.stdout:3/430: creat d9/d26/f7a x:0 0 0 2022-01-31T19:44:11.896 INFO:tasks.workunit.client.0.smithi167.stdout:3/431: symlink d9/d56/l7b 0 2022-01-31T19:44:11.896 INFO:tasks.workunit.client.0.smithi167.stdout:3/432: creat d9/d56/f7c x:0 0 0 2022-01-31T19:44:11.898 INFO:tasks.workunit.client.0.smithi167.stdout:7/593: read f7 [769944,130350] 0 2022-01-31T19:44:11.902 INFO:tasks.workunit.client.0.smithi167.stdout:7/594: creat d8/d2c/d6e/d9a/fd4 x:0 0 0 2022-01-31T19:44:11.904 INFO:tasks.workunit.client.0.smithi167.stdout:0/248: dwrite fc [0,4194304] 0 2022-01-31T19:44:11.904 INFO:tasks.workunit.client.0.smithi167.stdout:9/352: dwrite dd/d35/f33 [0,4194304] 0 2022-01-31T19:44:11.905 INFO:tasks.workunit.client.0.smithi167.stdout:9/353: stat dd/d19/d24 0 2022-01-31T19:44:11.905 INFO:tasks.workunit.client.0.smithi167.stdout:8/514: dwrite d6/db/d4e/d73/f7b [0,4194304] 0 2022-01-31T19:44:11.906 INFO:tasks.workunit.client.0.smithi167.stdout:7/595: symlink d8/d2c/d6e/d4d/d5e/dce/ld5 0 2022-01-31T19:44:11.906 INFO:tasks.workunit.client.0.smithi167.stdout:0/249: mkdir d17/d19/d1c/d39/d44/d49 0 2022-01-31T19:44:11.906 INFO:tasks.workunit.client.0.smithi167.stdout:0/250: fdatasync d17/f22 0 2022-01-31T19:44:11.910 INFO:tasks.workunit.client.0.smithi167.stdout:9/354: mkdir dd/d6b/d72 0 2022-01-31T19:44:11.911 INFO:tasks.workunit.client.0.smithi167.stdout:9/355: write dd/d19/f3d [288169,129033] 0 2022-01-31T19:44:11.913 INFO:tasks.workunit.client.0.smithi167.stdout:8/515: getdents d6/d1e/d4c/d80 0 2022-01-31T19:44:11.913 INFO:tasks.workunit.client.0.smithi167.stdout:8/516: stat d6/d1e/f41 0 2022-01-31T19:44:11.914 INFO:tasks.workunit.client.0.smithi167.stdout:8/517: fsync d6/db/de/f86 0 2022-01-31T19:44:11.914 INFO:tasks.workunit.client.0.smithi167.stdout:8/518: write d6/db/d28/f76 [1267671,102548] 0 2022-01-31T19:44:11.916 INFO:tasks.workunit.client.0.smithi167.stdout:7/596: link d8/d2c/d6e/d4d/d5e/d71/db3/lba d8/d2c/d7a/dd3/ld6 0 2022-01-31T19:44:11.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/251: creat d17/d3e/f4a x:0 0 0 2022-01-31T19:44:11.918 INFO:tasks.workunit.client.0.smithi167.stdout:0/252: chown d17/f22 0 1 2022-01-31T19:44:11.918 INFO:tasks.workunit.client.0.smithi167.stdout:0/253: read fa [723796,13220] 0 2022-01-31T19:44:11.919 INFO:tasks.workunit.client.0.smithi167.stdout:9/356: symlink dd/d19/d24/d56/l73 0 2022-01-31T19:44:11.922 INFO:tasks.workunit.client.0.smithi167.stdout:0/254: unlink d17/d19/d1c/f34 0 2022-01-31T19:44:11.923 INFO:tasks.workunit.client.0.smithi167.stdout:9/357: mknod dd/d19/d24/c74 0 2022-01-31T19:44:11.924 INFO:tasks.workunit.client.0.smithi167.stdout:7/597: getdents d8/d2c/d43 0 2022-01-31T19:44:11.925 INFO:tasks.workunit.client.0.smithi167.stdout:0/255: link d17/d3e/c40 d17/d19/d1c/d39/d44/d49/c4b 0 2022-01-31T19:44:11.926 INFO:tasks.workunit.client.0.smithi167.stdout:0/256: write d17/d33/f36 [965705,59411] 0 2022-01-31T19:44:11.927 INFO:tasks.workunit.client.0.smithi167.stdout:9/358: creat dd/d19/d24/d56/f75 x:0 0 0 2022-01-31T19:44:11.928 INFO:tasks.workunit.client.0.smithi167.stdout:9/359: creat dd/d19/d24/f76 x:0 0 0 2022-01-31T19:44:11.930 INFO:tasks.workunit.client.0.smithi167.stdout:8/519: dwrite d6/d1e/d4c/f8b [4194304,4194304] 0 2022-01-31T19:44:11.932 INFO:tasks.workunit.client.0.smithi167.stdout:9/360: rmdir dd/d35 39 2022-01-31T19:44:11.935 INFO:tasks.workunit.client.0.smithi167.stdout:9/361: truncate dd/d19/f1b 189311 0 2022-01-31T19:44:11.936 INFO:tasks.workunit.client.0.smithi167.stdout:9/362: mknod dd/d19/d24/d50/c77 0 2022-01-31T19:44:11.938 INFO:tasks.workunit.client.0.smithi167.stdout:9/363: creat dd/d19/d24/d50/d5d/f78 x:0 0 0 2022-01-31T19:44:11.938 INFO:tasks.workunit.client.0.smithi167.stdout:9/364: truncate f7 5132127 0 2022-01-31T19:44:11.938 INFO:tasks.workunit.client.0.smithi167.stdout:9/365: creat dd/d19/d24/f79 x:0 0 0 2022-01-31T19:44:11.959 INFO:tasks.workunit.client.0.smithi167.stdout:8/520: dwrite d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:11.959 INFO:tasks.workunit.client.0.smithi167.stdout:8/521: truncate d6/db/d28/d66/f8e 597985 0 2022-01-31T19:44:11.960 INFO:tasks.workunit.client.0.smithi167.stdout:8/522: write d6/db/d4e/d73/fa2 [709325,93041] 0 2022-01-31T19:44:11.960 INFO:tasks.workunit.client.0.smithi167.stdout:8/523: symlink d6/db/d6a/d8d/d9e/d60/la8 0 2022-01-31T19:44:11.961 INFO:tasks.workunit.client.0.smithi167.stdout:8/524: link d6/fa d6/db/d6a/d8d/d9e/d60/fa9 0 2022-01-31T19:44:11.965 INFO:tasks.workunit.client.0.smithi167.stdout:8/525: dread d6/db/d28/f5a [0,4194304] 0 2022-01-31T19:44:11.965 INFO:tasks.workunit.client.0.smithi167.stdout:8/526: readlink l2 0 2022-01-31T19:44:11.966 INFO:tasks.workunit.client.0.smithi167.stdout:9/366: dwrite dd/d17/f2e [0,4194304] 0 2022-01-31T19:44:11.967 INFO:tasks.workunit.client.0.smithi167.stdout:0/257: dwrite f8 [0,4194304] 0 2022-01-31T19:44:11.968 INFO:tasks.workunit.client.0.smithi167.stdout:7/598: dwrite f6 [0,4194304] 0 2022-01-31T19:44:11.968 INFO:tasks.workunit.client.0.smithi167.stdout:7/599: chown d8/d2c/d7a/fc8 162412526 1 2022-01-31T19:44:11.983 INFO:tasks.workunit.client.0.smithi167.stdout:8/527: rename d6/db/c93 to d6/db/d4e/caa 0 2022-01-31T19:44:11.984 INFO:tasks.workunit.client.0.smithi167.stdout:8/528: write d6/db/d36/f84 [47080,63502] 0 2022-01-31T19:44:11.984 INFO:tasks.workunit.client.0.smithi167.stdout:8/529: stat l3 0 2022-01-31T19:44:11.984 INFO:tasks.workunit.client.0.smithi167.stdout:8/530: fdatasync d6/db/de/f10 0 2022-01-31T19:44:11.988 INFO:tasks.workunit.client.0.smithi167.stdout:9/367: creat dd/d6b/f7a x:0 0 0 2022-01-31T19:44:11.989 INFO:tasks.workunit.client.0.smithi167.stdout:9/368: read dd/d19/f27 [4113678,94123] 0 2022-01-31T19:44:11.989 INFO:tasks.workunit.client.0.smithi167.stdout:9/369: write dd/d34/f51 [2057575,118734] 0 2022-01-31T19:44:12.000 INFO:tasks.workunit.client.0.smithi167.stdout:0/258: mknod d17/d19/c4c 0 2022-01-31T19:44:12.001 INFO:tasks.workunit.client.0.smithi167.stdout:8/531: creat d6/fab x:0 0 0 2022-01-31T19:44:12.003 INFO:tasks.workunit.client.0.smithi167.stdout:9/370: rename dd/d19/d24/f76 to dd/d6b/d72/f7b 0 2022-01-31T19:44:12.004 INFO:tasks.workunit.client.0.smithi167.stdout:7/600: rmdir d8/d2c/d43/da1 39 2022-01-31T19:44:12.004 INFO:tasks.workunit.client.0.smithi167.stdout:7/601: creat d8/d2c/d6e/d4d/d5e/fd7 x:0 0 0 2022-01-31T19:44:12.005 INFO:tasks.workunit.client.0.smithi167.stdout:8/532: creat d6/d1e/fac x:0 0 0 2022-01-31T19:44:12.006 INFO:tasks.workunit.client.0.smithi167.stdout:9/371: link dd/d19/d24/d56/d5b/l6f dd/d19/d24/d3b/d4c/l7c 0 2022-01-31T19:44:12.007 INFO:tasks.workunit.client.0.smithi167.stdout:9/372: read - dd/d6b/d72/f7b zero size 2022-01-31T19:44:12.010 INFO:tasks.workunit.client.0.smithi167.stdout:7/602: unlink d8/d2c/d6e/l41 0 2022-01-31T19:44:12.011 INFO:tasks.workunit.client.0.smithi167.stdout:7/603: write d8/d2c/d7a/fc8 [945852,16205] 0 2022-01-31T19:44:12.012 INFO:tasks.workunit.client.0.smithi167.stdout:0/259: dread d17/f22 [0,4194304] 0 2022-01-31T19:44:12.013 INFO:tasks.workunit.client.0.smithi167.stdout:9/373: symlink dd/d19/d24/d3b/l7d 0 2022-01-31T19:44:12.013 INFO:tasks.workunit.client.0.smithi167.stdout:9/374: chown dd/d19/d24/d3b/d4c/f5f 6 1 2022-01-31T19:44:12.014 INFO:tasks.workunit.client.0.smithi167.stdout:7/604: mkdir d8/d2c/d6e/d9a/dd8 0 2022-01-31T19:44:12.014 INFO:tasks.workunit.client.0.smithi167.stdout:9/375: write dd/d35/f37 [4718773,115211] 0 2022-01-31T19:44:12.015 INFO:tasks.workunit.client.0.smithi167.stdout:0/260: truncate ff 3746399 0 2022-01-31T19:44:12.016 INFO:tasks.workunit.client.0.smithi167.stdout:7/605: mknod d8/d2c/d6e/d9a/dd8/cd9 0 2022-01-31T19:44:12.016 INFO:tasks.workunit.client.0.smithi167.stdout:7/606: fdatasync d8/d2c/d6e/d4d/d5e/d71/db1/fad 0 2022-01-31T19:44:12.017 INFO:tasks.workunit.client.0.smithi167.stdout:9/376: mknod dd/d19/d24/c7e 0 2022-01-31T19:44:12.018 INFO:tasks.workunit.client.0.smithi167.stdout:0/261: creat d17/d19/f4d x:0 0 0 2022-01-31T19:44:12.018 INFO:tasks.workunit.client.0.smithi167.stdout:7/607: getdents d8/d2c/d6e/d4d/d6b/d8e 0 2022-01-31T19:44:12.019 INFO:tasks.workunit.client.0.smithi167.stdout:8/533: dwrite d6/db/d28/d66/f8e [0,4194304] 0 2022-01-31T19:44:12.019 INFO:tasks.workunit.client.0.smithi167.stdout:8/534: readlink d6/db/d6a/d8d/l9d 0 2022-01-31T19:44:12.020 INFO:tasks.workunit.client.0.smithi167.stdout:8/535: truncate d6/db/d28/f94 705297 0 2022-01-31T19:44:12.021 INFO:tasks.workunit.client.0.smithi167.stdout:9/377: truncate dd/d19/f2d 6007196 0 2022-01-31T19:44:12.027 INFO:tasks.workunit.client.0.smithi167.stdout:0/262: truncate fe 3040314 0 2022-01-31T19:44:12.027 INFO:tasks.workunit.client.0.smithi167.stdout:0/263: creat d17/d19/f4e x:0 0 0 2022-01-31T19:44:12.031 INFO:tasks.workunit.client.0.smithi167.stdout:7/608: unlink d8/d2c/d43/fa6 0 2022-01-31T19:44:12.034 INFO:tasks.workunit.client.0.smithi167.stdout:8/536: rename d6/db/de/d18/l4b to d6/db/d28/d61/lad 0 2022-01-31T19:44:12.034 INFO:tasks.workunit.client.0.smithi167.stdout:7/609: creat d8/d2c/fda x:0 0 0 2022-01-31T19:44:12.034 INFO:tasks.workunit.client.0.smithi167.stdout:8/537: write d6/d1e/f3c [4440838,7238] 0 2022-01-31T19:44:12.035 INFO:tasks.workunit.client.0.smithi167.stdout:8/538: fsync d6/db/d4e/f4f 0 2022-01-31T19:44:12.045 INFO:tasks.workunit.client.0.smithi167.stdout:8/539: rmdir d6/db/d36/d5d 39 2022-01-31T19:44:12.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/378: dwrite dd/d19/d24/d3b/f4a [0,4194304] 0 2022-01-31T19:44:12.047 INFO:tasks.workunit.client.0.smithi167.stdout:8/540: write d6/db/de/d18/f19 [175109,68684] 0 2022-01-31T19:44:12.047 INFO:tasks.workunit.client.0.smithi167.stdout:8/541: stat d6/f32 0 2022-01-31T19:44:12.050 INFO:tasks.workunit.client.0.smithi167.stdout:9/379: mkdir dd/d17/d7f 0 2022-01-31T19:44:12.050 INFO:tasks.workunit.client.0.smithi167.stdout:8/542: dread d6/f32 [0,4194304] 0 2022-01-31T19:44:12.050 INFO:tasks.workunit.client.0.smithi167.stdout:8/543: stat d6/db/d4e/d73/d88/l9a 0 2022-01-31T19:44:12.053 INFO:tasks.workunit.client.0.smithi167.stdout:8/544: symlink d6/d1e/d2d/lae 0 2022-01-31T19:44:12.053 INFO:tasks.workunit.client.0.smithi167.stdout:7/610: dwrite d8/d2c/d6e/d4d/d5e/d71/db3/fb8 [0,4194304] 0 2022-01-31T19:44:12.054 INFO:tasks.workunit.client.0.smithi167.stdout:7/611: read - d8/d2c/d6e/d4d/d5e/d71/db1/d4f/fc2 zero size 2022-01-31T19:44:12.059 INFO:tasks.workunit.client.0.smithi167.stdout:8/545: mkdir d6/db/d28/daf 0 2022-01-31T19:44:12.063 INFO:tasks.workunit.client.0.smithi167.stdout:9/380: rename dd/d19/d24/d3b to dd/d17/d80 0 2022-01-31T19:44:12.063 INFO:tasks.workunit.client.0.smithi167.stdout:9/381: truncate dd/d19/f1b 858730 0 2022-01-31T19:44:12.065 INFO:tasks.workunit.client.0.smithi167.stdout:7/612: rename d8/d35/l8b to d8/dc/d18/d24/ldb 0 2022-01-31T19:44:12.067 INFO:tasks.workunit.client.0.smithi167.stdout:9/382: symlink dd/d19/d24/d50/d5d/l81 0 2022-01-31T19:44:12.068 INFO:tasks.workunit.client.0.smithi167.stdout:9/383: readlink dd/d17/d80/d4c/l69 0 2022-01-31T19:44:12.068 INFO:tasks.workunit.client.0.smithi167.stdout:9/384: fsync dd/d17/d80/f54 0 2022-01-31T19:44:12.069 INFO:tasks.workunit.client.0.smithi167.stdout:8/546: rename d6/db/de/d3d/f56 to d6/db/d6a/d8d/fb0 0 2022-01-31T19:44:12.071 INFO:tasks.workunit.client.0.smithi167.stdout:9/385: mknod dd/d19/d24/d50/d5d/c82 0 2022-01-31T19:44:12.073 INFO:tasks.workunit.client.0.smithi167.stdout:8/547: rename d6/db/de/d3d/l44 to d6/d1e/d4c/d80/d99/lb1 0 2022-01-31T19:44:12.074 INFO:tasks.workunit.client.0.smithi167.stdout:8/548: fsync d6/db/de/f26 0 2022-01-31T19:44:12.074 INFO:tasks.workunit.client.0.smithi167.stdout:7/613: mknod d8/d2c/cdc 0 2022-01-31T19:44:12.076 INFO:tasks.workunit.client.0.smithi167.stdout:9/386: link dd/d6b/f7a dd/d6b/f83 0 2022-01-31T19:44:12.076 INFO:tasks.workunit.client.0.smithi167.stdout:5/592: sync 2022-01-31T19:44:12.076 INFO:tasks.workunit.client.0.smithi167.stdout:9/387: write dd/d35/f3a [1323504,94355] 0 2022-01-31T19:44:12.077 INFO:tasks.workunit.client.0.smithi167.stdout:9/388: creat dd/f84 x:0 0 0 2022-01-31T19:44:12.077 INFO:tasks.workunit.client.0.smithi167.stdout:9/389: creat dd/d19/d24/d56/f85 x:0 0 0 2022-01-31T19:44:12.077 INFO:tasks.workunit.client.0.smithi167.stdout:9/390: write dd/f84 [762824,43472] 0 2022-01-31T19:44:12.077 INFO:tasks.workunit.client.0.smithi167.stdout:8/549: creat d6/db/de/d18/da4/fb2 x:0 0 0 2022-01-31T19:44:12.078 INFO:tasks.workunit.client.0.smithi167.stdout:8/550: stat d6/db/de/d3d 0 2022-01-31T19:44:12.078 INFO:tasks.workunit.client.0.smithi167.stdout:7/614: symlink d8/d2c/d43/d94/daf/ldd 0 2022-01-31T19:44:12.079 INFO:tasks.workunit.client.0.smithi167.stdout:5/593: unlink d5/d6/d16/la6 0 2022-01-31T19:44:12.079 INFO:tasks.workunit.client.0.smithi167.stdout:7/615: dread d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f68 [0,4194304] 0 2022-01-31T19:44:12.083 INFO:tasks.workunit.client.0.smithi167.stdout:9/391: read dd/d34/f5a [282518,46317] 0 2022-01-31T19:44:12.098 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.095+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9948005e80 con 0x7f99640688d0 2022-01-31T19:44:12.098 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.096+0000 7f9952ffd700 1 -- 172.21.15.167:0/847153096 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+785 (secure 0 0 0) 0x7f9954018070 con 0x7f99640688d0 2022-01-31T19:44:12.098 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:44:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "mon": { 2022-01-31T19:44:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:44:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "mgr": { 2022-01-31T19:44:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 1, 2022-01-31T19:44:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev)": 1 2022-01-31T19:44:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "osd": { 2022-01-31T19:44:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6 2022-01-31T19:44:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "mds": { 2022-01-31T19:44:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4 2022-01-31T19:44:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "overall": { 2022-01-31T19:44:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 13, 2022-01-31T19:44:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev)": 1 2022-01-31T19:44:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2022-01-31T19:44:12.103 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:44:12.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7f994c061a60 msgr2=0x7f994c063f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:12.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7f994c061a60 0x7f994c063f20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9958004630 tx=0x7f9958009b30).stop 2022-01-31T19:44:12.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 msgr2=0x7f9964110380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:12.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 0x7f9964110380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f99540093f0 tx=0x7f9954006ec0).stop 2022-01-31T19:44:12.104 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 shutdown_connections 2022-01-31T19:44:12.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7f994c061a60 0x7f994c063f20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99640688d0 0x7f9964110380 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.098+0000 7f99688ce700 1 --2- 172.21.15.167:0/847153096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964069270 0x7f9964112880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.105 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.099+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 >> 172.21.15.167:0/847153096 conn(0x7f99640637e0 msgr2=0x7f9964067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:12.106 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.099+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 shutdown_connections 2022-01-31T19:44:12.106 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.099+0000 7f99688ce700 1 -- 172.21.15.167:0/847153096 wait complete. 2022-01-31T19:44:12.106 INFO:tasks.workunit.client.0.smithi167.stdout:9/392: dwrite dd/d17/f36 [0,4194304] 0 2022-01-31T19:44:12.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:11 smithi171 conmon[35325]: cluster 2022-01-31T19:44:10.431891+0000 mgr.smithi171.asyxnc (mgr 2022-01-31T19:44:12.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:11 smithi171 conmon[35325]: .24387) 15 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 803 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 MiB/s rd, 90 MiB/s wr, 477 op/s 2022-01-31T19:44:12.111 INFO:tasks.workunit.client.0.smithi167.stdout:5/594: dwrite d5/d6/d16/d18/d5b/d63/f98 [0,4194304] 0 2022-01-31T19:44:12.114 INFO:tasks.workunit.client.0.smithi167.stdout:5/595: dread d5/d6/d16/d18/f3d [0,4194304] 0 2022-01-31T19:44:12.174 INFO:tasks.workunit.client.0.smithi167.stdout:8/551: mknod d6/db/d6a/d8d/d9e/d60/cb3 0 2022-01-31T19:44:12.175 INFO:tasks.workunit.client.0.smithi167.stdout:7/616: rmdir d8/d2c/d6e/d9a/dd8 39 2022-01-31T19:44:12.179 INFO:tasks.workunit.client.0.smithi167.stdout:9/393: rename dd/d19/d24/f79 to dd/d19/d24/d56/f86 0 2022-01-31T19:44:12.186 INFO:tasks.workunit.client.0.smithi167.stdout:5/596: rename d5/d6/d16/d18/f8d to d5/fb9 0 2022-01-31T19:44:12.193 INFO:tasks.workunit.client.0.smithi167.stdout:9/394: rename dd/d34/c3f to dd/d6b/c87 0 2022-01-31T19:44:12.193 INFO:tasks.workunit.client.0.smithi167.stdout:5/597: rename d5 to d5/d6/d16/d18/d43/d45/dba 22 2022-01-31T19:44:12.196 INFO:tasks.workunit.client.0.smithi167.stdout:8/552: link d6/d1e/d4c/d80/d99/lb1 d6/db/d28/lb4 0 2022-01-31T19:44:12.199 INFO:tasks.workunit.client.0.smithi167.stdout:9/395: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:12.199 INFO:tasks.workunit.client.0.smithi167.stdout:7/617: dwrite d8/d2c/d6e/d4d/d5e/d71/db1/f6a [0,4194304] 0 2022-01-31T19:44:12.200 INFO:tasks.workunit.client.0.smithi167.stdout:8/553: dread d6/db/d6a/d8d/d9e/d60/f6c [0,4194304] 0 2022-01-31T19:44:12.200 INFO:tasks.workunit.client.0.smithi167.stdout:7/618: write d8/d2c/d6e/d4d/d6b/f81 [447484,108172] 0 2022-01-31T19:44:12.200 INFO:tasks.workunit.client.0.smithi167.stdout:7/619: read - d8/d2c/d7a/fcd zero size 2022-01-31T19:44:12.203 INFO:tasks.workunit.client.0.smithi167.stdout:9/396: rename dd/fe to dd/d35/f88 0 2022-01-31T19:44:12.208 INFO:tasks.workunit.client.0.smithi167.stdout:8/554: symlink d6/d1e/d2d/lb5 0 2022-01-31T19:44:12.209 INFO:tasks.workunit.client.0.smithi167.stdout:8/555: dread - d6/db/d28/d66/f8c zero size 2022-01-31T19:44:12.209 INFO:tasks.workunit.client.0.smithi167.stdout:7/620: creat d8/d2c/d6e/d4d/d6b/d8e/fde x:0 0 0 2022-01-31T19:44:12.210 INFO:tasks.workunit.client.0.smithi167.stdout:9/397: mknod dd/d19/d24/d56/d5b/c89 0 2022-01-31T19:44:12.211 INFO:tasks.workunit.client.0.smithi167.stdout:9/398: fsync dd/d17/f1f 0 2022-01-31T19:44:12.211 INFO:tasks.workunit.client.0.smithi167.stdout:5/598: dread d5/d6/f42 [0,4194304] 0 2022-01-31T19:44:12.211 INFO:tasks.workunit.client.0.smithi167.stdout:8/556: truncate d6/db/d28/f76 803403 0 2022-01-31T19:44:12.212 INFO:tasks.workunit.client.0.smithi167.stdout:9/399: write dd/d19/f2d [5989756,117396] 0 2022-01-31T19:44:12.212 INFO:tasks.workunit.client.0.smithi167.stdout:9/400: fdatasync dd/f1a 0 2022-01-31T19:44:12.212 INFO:tasks.workunit.client.0.smithi167.stdout:7/621: truncate f6 3185715 0 2022-01-31T19:44:12.213 INFO:tasks.workunit.client.0.smithi167.stdout:8/557: write d6/d1e/d4c/f8b [6038789,36991] 0 2022-01-31T19:44:12.214 INFO:tasks.workunit.client.0.smithi167.stdout:5/599: getdents d5/d6/de/d22/d4d/d65 0 2022-01-31T19:44:12.214 INFO:tasks.workunit.client.0.smithi167.stdout:5/600: truncate d5/d61/f74 4929511 0 2022-01-31T19:44:12.214 INFO:tasks.workunit.client.0.smithi167.stdout:5/601: readlink l4 0 2022-01-31T19:44:12.214 INFO:tasks.workunit.client.0.smithi167.stdout:7/622: mknod d8/d2c/d7a/cdf 0 2022-01-31T19:44:12.215 INFO:tasks.workunit.client.0.smithi167.stdout:7/623: chown d8/d2c/d6e/d4d/d5e/d71/db1/d8d/lc7 1562 1 2022-01-31T19:44:12.217 INFO:tasks.workunit.client.0.smithi167.stdout:5/602: mkdir d5/d6/de/d22/d4d/d65/dbb 0 2022-01-31T19:44:12.222 INFO:tasks.workunit.client.0.smithi167.stdout:7/624: dread d8/d2c/d6e/d4d/d67/d8f/f96 [0,4194304] 0 2022-01-31T19:44:12.223 INFO:tasks.workunit.client.0.smithi167.stdout:9/401: rmdir dd/d19 39 2022-01-31T19:44:12.223 INFO:tasks.workunit.client.0.smithi167.stdout:9/402: write dd/d17/d80/d4c/f67 [1008978,127167] 0 2022-01-31T19:44:12.227 INFO:tasks.workunit.client.0.smithi167.stdout:5/603: mkdir d5/d6/d16/d18/d5b/d63/d88/dbc 0 2022-01-31T19:44:12.228 INFO:tasks.workunit.client.0.smithi167.stdout:7/625: getdents d8/d2c/d43/da1 0 2022-01-31T19:44:12.229 INFO:tasks.workunit.client.0.smithi167.stdout:5/604: creat d5/d6/de/fbd x:0 0 0 2022-01-31T19:44:12.229 INFO:tasks.workunit.client.0.smithi167.stdout:5/605: dread - d5/d6/de/fbd zero size 2022-01-31T19:44:12.230 INFO:tasks.workunit.client.0.smithi167.stdout:5/606: symlink d5/d6/d16/d18/d43/lbe 0 2022-01-31T19:44:12.232 INFO:tasks.workunit.client.0.smithi167.stdout:5/607: creat d5/d6/de/d22/d4d/d65/fbf x:0 0 0 2022-01-31T19:44:12.233 INFO:tasks.workunit.client.0.smithi167.stdout:5/608: getdents d5/d6/d16/d18/d43/d45/d69 0 2022-01-31T19:44:12.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.233+0000 7fb6f3fff700 1 -- 172.21.15.167:0/3198866945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 msgr2=0x7fb6f40fbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:12.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/3198866945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40fbe90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb6e4004660 tx=0x7fb6e4009b30).stop 2022-01-31T19:44:12.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 -- 172.21.15.167:0/3198866945 shutdown_connections 2022-01-31T19:44:12.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/3198866945 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40fbe90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/3198866945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6f40f42c0 0x7fb6f40f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 -- 172.21.15.167:0/3198866945 >> 172.21.15.167:0/3198866945 conn(0x7fb6f40f0020 msgr2=0x7fb6f40f2440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:12.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 -- 172.21.15.167:0/3198866945 shutdown_connections 2022-01-31T19:44:12.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 -- 172.21.15.167:0/3198866945 wait complete. 2022-01-31T19:44:12.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.234+0000 7fb6f3fff700 1 Processor -- start 2022-01-31T19:44:12.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f3fff700 1 -- start start 2022-01-31T19:44:12.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f3fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6f40f42c0 0x7fb6f40f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:12.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f3fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:12.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f3fff700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f40f62e0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f3fff700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f40f6420 con 0x7fb6f40f42c0 2022-01-31T19:44:12.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:12.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40f5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34132/0 (socket says 172.21.15.167:34132) 2022-01-31T19:44:12.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 -- 172.21.15.167:0/1317149099 learned_addr learned my addr 172.21.15.167:0/1317149099 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:12.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f2ffd700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6f40f42c0 0x7fb6f40f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:12.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 -- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6f40f42c0 msgr2=0x7fb6f40f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:12.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6f40f42c0 0x7fb6f40f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 -- 172.21.15.167:0/1317149099 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4005040 con 0x7fb6f40f4c60 2022-01-31T19:44:12.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6f27fc700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40f5d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fb6e4009990 tx=0x7fb6e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:12.242 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.235+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e4018070 con 0x7fb6f40f4c60 2022-01-31T19:44:12.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.236+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f40f66a0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.243 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.236+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f40f6c00 con 0x7fb6f40f4c60 2022-01-31T19:44:12.243 INFO:tasks.workunit.client.0.smithi167.stdout:8/558: dwrite d6/f57 [4194304,4194304] 0 2022-01-31T19:44:12.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.237+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fb6e4007ba0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.244 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.237+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e401ddf0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.245 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.239+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d4004fa0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.245 INFO:tasks.workunit.client.0.smithi167.stdout:8/559: creat d6/d1e/d4c/d80/fb6 x:0 0 0 2022-01-31T19:44:12.245 INFO:tasks.workunit.client.0.smithi167.stdout:9/403: dwrite dd/d19/d24/f45 [0,4194304] 0 2022-01-31T19:44:12.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.240+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 19) v1 ==== 78465+0+0 (secure 0 0 0) 0x7fb6e401d420 con 0x7fb6f40f4c60 2022-01-31T19:44:12.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.241+0000 7fb6db7fe700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fb6dc0619a0 0x7fb6dc063e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:12.246 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.241+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5343+0+0 (secure 0 0 0) 0x7fb6e407ef00 con 0x7fb6f40f4c60 2022-01-31T19:44:12.247 INFO:tasks.workunit.client.0.smithi167.stdout:9/404: dread dd/d19/f1b [0,4194304] 0 2022-01-31T19:44:12.247 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.245+0000 7fb6f2ffd700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fb6dc0619a0 0x7fb6dc063e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:12.248 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.245+0000 7fb6f2ffd700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fb6dc0619a0 0x7fb6dc063e60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb6f40f7690 tx=0x7fb6e800b040).ready entity=mgr.24387 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:12.249 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.247+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7fb6e404d7d0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.251 INFO:tasks.workunit.client.0.smithi167.stdout:8/560: rename d6/l3f to d6/db/lb7 0 2022-01-31T19:44:12.252 INFO:tasks.workunit.client.0.smithi167.stdout:5/609: dwrite d5/d6/de/d22/d76/db0/fb6 [0,4194304] 0 2022-01-31T19:44:12.253 INFO:tasks.workunit.client.0.smithi167.stdout:5/610: stat d5/d61 0 2022-01-31T19:44:12.257 INFO:tasks.workunit.client.0.smithi167.stdout:5/611: dread d5/f44 [0,4194304] 0 2022-01-31T19:44:12.258 INFO:tasks.workunit.client.0.smithi167.stdout:2/328: sync 2022-01-31T19:44:12.258 INFO:tasks.workunit.client.0.smithi167.stdout:2/329: fsync dd/f29 0 2022-01-31T19:44:12.258 INFO:tasks.workunit.client.0.smithi167.stdout:9/405: getdents dd 0 2022-01-31T19:44:12.258 INFO:tasks.workunit.client.0.smithi167.stdout:9/406: write dd/ff [4446041,14323] 0 2022-01-31T19:44:12.259 INFO:tasks.workunit.client.0.smithi167.stdout:9/407: write dd/d34/f5a [397112,93381] 0 2022-01-31T19:44:12.262 INFO:tasks.workunit.client.0.smithi167.stdout:2/330: truncate dd/f29 3326067 0 2022-01-31T19:44:12.263 INFO:tasks.workunit.client.0.smithi167.stdout:2/331: chown dd/df/l52 53916 1 2022-01-31T19:44:12.264 INFO:tasks.workunit.client.0.smithi167.stdout:2/332: mknod dd/df/d3f/c5c 0 2022-01-31T19:44:12.264 INFO:tasks.workunit.client.0.smithi167.stdout:2/333: fsync dd/df/d2c/f2d 0 2022-01-31T19:44:12.303 INFO:tasks.workunit.client.0.smithi167.stdout:5/612: dwrite d5/d6/d16/d18/d43/d45/f94 [0,4194304] 0 2022-01-31T19:44:12.303 INFO:tasks.workunit.client.0.smithi167.stdout:8/561: dwrite d6/db/de/d18/f21 [0,4194304] 0 2022-01-31T19:44:12.304 INFO:tasks.workunit.client.0.smithi167.stdout:9/408: dwrite dd/d19/f3d [0,4194304] 0 2022-01-31T19:44:12.304 INFO:tasks.workunit.client.0.smithi167.stdout:9/409: fsync dd/f1a 0 2022-01-31T19:44:12.304 INFO:tasks.workunit.client.0.smithi167.stdout:2/334: dwrite f9 [0,4194304] 0 2022-01-31T19:44:12.305 INFO:tasks.workunit.client.0.smithi167.stdout:5/613: mkdir d5/d6/d16/d18/d5b/d63/d88/dbc/dc0 0 2022-01-31T19:44:12.305 INFO:tasks.workunit.client.0.smithi167.stdout:8/562: mknod d6/db/d28/d61/cb8 0 2022-01-31T19:44:12.306 INFO:tasks.workunit.client.0.smithi167.stdout:8/563: dread - d6/fab zero size 2022-01-31T19:44:12.308 INFO:tasks.workunit.client.0.smithi167.stdout:2/335: rename dd/df/f35 to dd/df/f5d 0 2022-01-31T19:44:12.311 INFO:tasks.workunit.client.0.smithi167.stdout:5/614: fsync d5/d6/de/d22/d4d/d65/fb4 0 2022-01-31T19:44:12.311 INFO:tasks.workunit.client.0.smithi167.stdout:5/615: fsync d5/f44 0 2022-01-31T19:44:12.312 INFO:tasks.workunit.client.0.smithi167.stdout:5/616: write d5/d6/d16/d18/f3d [2929768,103492] 0 2022-01-31T19:44:12.313 INFO:tasks.workunit.client.0.smithi167.stdout:8/564: rename d6/d1e/d4c/d80/f8f to d6/db/d6a/fb9 0 2022-01-31T19:44:12.317 INFO:tasks.workunit.client.0.smithi167.stdout:2/336: rename dd/l14 to dd/df/d1b/d3d/d4a/d4b/l5e 0 2022-01-31T19:44:12.317 INFO:tasks.workunit.client.0.smithi167.stdout:8/565: mkdir d6/d1e/d4c/dba 0 2022-01-31T19:44:12.319 INFO:tasks.workunit.client.0.smithi167.stdout:5/617: rename d5/d6/de/l40 to d5/d6/de/d22/lc1 0 2022-01-31T19:44:12.320 INFO:tasks.workunit.client.0.smithi167.stdout:9/410: dwrite dd/f10 [0,4194304] 0 2022-01-31T19:44:12.329 INFO:tasks.workunit.client.0.smithi167.stdout:9/411: write dd/d35/f44 [207903,22466] 0 2022-01-31T19:44:12.329 INFO:tasks.workunit.client.0.smithi167.stdout:8/566: rename d6/c7 to d6/db/d28/daf/cbb 0 2022-01-31T19:44:12.329 INFO:tasks.workunit.client.0.smithi167.stdout:9/412: write dd/d35/f88 [189773,80425] 0 2022-01-31T19:44:12.333 INFO:tasks.workunit.client.0.smithi167.stdout:5/618: rmdir d5/d6/de 39 2022-01-31T19:44:12.337 INFO:tasks.workunit.client.0.smithi167.stdout:5/619: unlink d5/d61/laf 0 2022-01-31T19:44:12.340 INFO:tasks.workunit.client.0.smithi167.stdout:5/620: rename d5/d6/f92 to d5/d6/d16/d18/d5b/fc2 0 2022-01-31T19:44:12.340 INFO:tasks.workunit.client.0.smithi167.stdout:9/413: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:12.340 INFO:tasks.workunit.client.0.smithi167.stdout:9/414: read dd/d19/f2d [2827063,83694] 0 2022-01-31T19:44:12.341 INFO:tasks.workunit.client.0.smithi167.stdout:9/415: write dd/ff [4670868,35640] 0 2022-01-31T19:44:12.342 INFO:tasks.workunit.client.0.smithi167.stdout:5/621: mknod d5/d6/d16/d18/d5b/d63/cc3 0 2022-01-31T19:44:12.342 INFO:tasks.workunit.client.0.smithi167.stdout:5/622: chown d5/d6/de/d22/d6d/da0 58588218 1 2022-01-31T19:44:12.342 INFO:tasks.workunit.client.0.smithi167.stdout:5/623: fsync d5/d6/de/d22/f5f 0 2022-01-31T19:44:12.342 INFO:tasks.workunit.client.0.smithi167.stdout:5/624: write d5/d6/de/d22/d4d/d65/fbf [510469,120400] 0 2022-01-31T19:44:12.343 INFO:tasks.workunit.client.0.smithi167.stdout:5/625: write d5/d6/de/d22/d4d/f87 [11142,119359] 0 2022-01-31T19:44:12.343 INFO:tasks.workunit.client.0.smithi167.stdout:2/337: dwrite dd/df/d1b/d2f/f4d [0,4194304] 0 2022-01-31T19:44:12.344 INFO:tasks.workunit.client.0.smithi167.stdout:2/338: unlink dd/d18/f30 0 2022-01-31T19:44:12.345 INFO:tasks.workunit.client.0.smithi167.stdout:2/339: mkdir dd/df/d3f/d51/d5f 0 2022-01-31T19:44:12.346 INFO:tasks.workunit.client.0.smithi167.stdout:2/340: creat dd/df/d1b/d3d/d4a/d4b/d54/f60 x:0 0 0 2022-01-31T19:44:12.347 INFO:tasks.workunit.client.0.smithi167.stdout:2/341: truncate dd/df/f1c 2324521 0 2022-01-31T19:44:12.347 INFO:tasks.workunit.client.0.smithi167.stdout:9/416: write dd/d19/f1b [408926,91779] 0 2022-01-31T19:44:12.348 INFO:tasks.workunit.client.0.smithi167.stdout:2/342: rmdir dd/df/d3f/d51 39 2022-01-31T19:44:12.348 INFO:tasks.workunit.client.0.smithi167.stdout:9/417: fsync dd/d35/f3a 0 2022-01-31T19:44:12.352 INFO:tasks.workunit.client.0.smithi167.stdout:8/567: dwrite d6/db/de/d3d/f71 [0,4194304] 0 2022-01-31T19:44:12.353 INFO:tasks.workunit.client.0.smithi167.stdout:5/626: dread d5/d6/d16/d18/d43/d45/f94 [0,4194304] 0 2022-01-31T19:44:12.353 INFO:tasks.workunit.client.0.smithi167.stdout:5/627: fdatasync d5/d6/de/d22/f97 0 2022-01-31T19:44:12.360 INFO:tasks.workunit.client.0.smithi167.stdout:2/343: link dd/d18/l1f dd/df/d22/l61 0 2022-01-31T19:44:12.361 INFO:tasks.workunit.client.0.smithi167.stdout:9/418: link dd/d17/d80/f5e dd/d6b/d72/f8a 0 2022-01-31T19:44:12.367 INFO:tasks.workunit.client.0.smithi167.stdout:8/568: dwrite d6/f1d [4194304,4194304] 0 2022-01-31T19:44:12.376 INFO:tasks.workunit.client.0.smithi167.stdout:5/628: rename d5/d6/de/d22/d4d/d65 to d5/d6/d16/d18/d43/d45/d69/dc4 0 2022-01-31T19:44:12.376 INFO:tasks.workunit.client.0.smithi167.stdout:5/629: chown d5/d6/d16/d18/f1f 519761 1 2022-01-31T19:44:12.376 INFO:tasks.workunit.client.0.smithi167.stdout:2/344: mkdir dd/d18/d62 0 2022-01-31T19:44:12.377 INFO:tasks.workunit.client.0.smithi167.stdout:5/630: symlink d5/d6/d16/d18/d5b/d63/d88/dbc/dc0/lc5 0 2022-01-31T19:44:12.379 INFO:tasks.workunit.client.0.smithi167.stdout:2/345: read dd/f55 [2912763,55118] 0 2022-01-31T19:44:12.380 INFO:tasks.workunit.client.0.smithi167.stdout:2/346: readlink dd/df/d22/l53 0 2022-01-31T19:44:12.381 INFO:tasks.workunit.client.0.smithi167.stdout:2/347: dread dd/df/d1b/d2f/d36/f2a [0,4194304] 0 2022-01-31T19:44:12.381 INFO:tasks.workunit.client.0.smithi167.stdout:2/348: fdatasync dd/f55 0 2022-01-31T19:44:12.381 INFO:tasks.workunit.client.0.smithi167.stdout:9/419: dwrite dd/d6b/f7a [0,4194304] 0 2022-01-31T19:44:12.385 INFO:tasks.workunit.client.0.smithi167.stdout:9/420: mknod dd/c8b 0 2022-01-31T19:44:12.385 INFO:tasks.workunit.client.0.smithi167.stdout:9/421: stat dd/c12 0 2022-01-31T19:44:12.389 INFO:tasks.workunit.client.0.smithi167.stdout:6/376: sync 2022-01-31T19:44:12.390 INFO:tasks.workunit.client.0.smithi167.stdout:9/422: dread dd/d17/f18 [0,4194304] 0 2022-01-31T19:44:12.392 INFO:tasks.workunit.client.0.smithi167.stdout:2/349: rename dd/df/d2c to dd/d18/d63 0 2022-01-31T19:44:12.395 INFO:tasks.workunit.client.0.smithi167.stdout:6/377: mknod d8/d33/d51/d5d/d64/c8f 0 2022-01-31T19:44:12.396 INFO:tasks.workunit.client.0.smithi167.stdout:6/378: fsync d8/d62/f6d 0 2022-01-31T19:44:12.396 INFO:tasks.workunit.client.0.smithi167.stdout:6/379: chown d8/lc 21103 1 2022-01-31T19:44:12.397 INFO:tasks.workunit.client.0.smithi167.stdout:2/350: creat dd/d18/d62/f64 x:0 0 0 2022-01-31T19:44:12.397 INFO:tasks.workunit.client.0.smithi167.stdout:5/631: dwrite d5/d6/d16/d18/d5b/fc2 [0,4194304] 0 2022-01-31T19:44:12.400 INFO:tasks.workunit.client.0.smithi167.stdout:2/351: rename f9 to dd/df/d3f/d51/f65 0 2022-01-31T19:44:12.402 INFO:tasks.workunit.client.0.smithi167.stdout:5/632: creat d5/d6/d16/d18/d43/d45/fc6 x:0 0 0 2022-01-31T19:44:12.403 INFO:tasks.workunit.client.0.smithi167.stdout:5/633: truncate d5/d6/f28 5180296 0 2022-01-31T19:44:12.405 INFO:tasks.workunit.client.0.smithi167.stdout:2/352: creat dd/df/d1b/d2f/f66 x:0 0 0 2022-01-31T19:44:12.405 INFO:tasks.workunit.client.0.smithi167.stdout:9/423: dwrite dd/d19/d24/d56/f85 [0,4194304] 0 2022-01-31T19:44:12.408 INFO:tasks.workunit.client.0.smithi167.stdout:2/353: symlink dd/df/d1b/l67 0 2022-01-31T19:44:12.409 INFO:tasks.workunit.client.0.smithi167.stdout:6/380: rmdir d8/d33/d51/d5d 39 2022-01-31T19:44:12.410 INFO:tasks.workunit.client.0.smithi167.stdout:6/381: stat d8/le 0 2022-01-31T19:44:12.410 INFO:tasks.workunit.client.0.smithi167.stdout:6/382: dread - d8/d62/f6d zero size 2022-01-31T19:44:12.411 INFO:tasks.workunit.client.0.smithi167.stdout:9/424: getdents dd/d34 0 2022-01-31T19:44:12.411 INFO:tasks.workunit.client.0.smithi167.stdout:9/425: chown dd/d19/c71 4672549 1 2022-01-31T19:44:12.415 INFO:tasks.workunit.client.0.smithi167.stdout:2/354: mknod dd/df/d1b/d2f/d36/c68 0 2022-01-31T19:44:12.419 INFO:tasks.workunit.client.0.smithi167.stdout:6/383: dread d8/d33/d51/d5d/d64/f68 [0,4194304] 0 2022-01-31T19:44:12.421 INFO:tasks.workunit.client.0.smithi167.stdout:2/355: creat dd/df/d3f/d51/d5f/f69 x:0 0 0 2022-01-31T19:44:12.421 INFO:tasks.workunit.client.0.smithi167.stdout:6/384: getdents d8/d20 0 2022-01-31T19:44:12.422 INFO:tasks.workunit.client.0.smithi167.stdout:2/356: write dd/df/d1b/d3d/f44 [4075840,52148] 0 2022-01-31T19:44:12.422 INFO:tasks.workunit.client.0.smithi167.stdout:2/357: creat dd/df/d1b/f6a x:0 0 0 2022-01-31T19:44:12.423 INFO:tasks.workunit.client.0.smithi167.stdout:2/358: creat dd/df/d1b/d3d/d4a/f6b x:0 0 0 2022-01-31T19:44:12.425 INFO:tasks.workunit.client.0.smithi167.stdout:2/359: mknod dd/df/d1b/d2f/c6c 0 2022-01-31T19:44:12.427 INFO:tasks.workunit.client.0.smithi167.stdout:2/360: truncate dd/df/d1b/d2f/f4d 1740693 0 2022-01-31T19:44:12.427 INFO:tasks.workunit.client.0.smithi167.stdout:2/361: chown dd/df/d1b/d2f 13728011 1 2022-01-31T19:44:12.430 INFO:tasks.workunit.client.0.smithi167.stdout:9/426: dwrite dd/d17/f1f [0,4194304] 0 2022-01-31T19:44:12.430 INFO:tasks.workunit.client.0.smithi167.stdout:9/427: truncate dd/d34/f61 641303 0 2022-01-31T19:44:12.435 INFO:tasks.workunit.client.0.smithi167.stdout:6/385: dread d8/fb [4194304,4194304] 0 2022-01-31T19:44:12.446 INFO:tasks.workunit.client.0.smithi167.stdout:6/386: dread d8/d20/f44 [0,4194304] 0 2022-01-31T19:44:12.446 INFO:tasks.workunit.client.0.smithi167.stdout:6/387: dread d8/d20/f2f [0,4194304] 0 2022-01-31T19:44:12.452 INFO:tasks.workunit.client.0.smithi167.stdout:9/428: dwrite dd/d17/d80/d4c/f67 [0,4194304] 0 2022-01-31T19:44:12.452 INFO:tasks.workunit.client.0.smithi167.stdout:2/362: dwrite dd/df/f50 [0,4194304] 0 2022-01-31T19:44:12.452 INFO:tasks.workunit.client.0.smithi167.stdout:2/363: chown dd/df/c10 1916459638 1 2022-01-31T19:44:12.456 INFO:tasks.workunit.client.0.smithi167.stdout:9/429: symlink dd/d6b/d72/l8c 0 2022-01-31T19:44:12.457 INFO:tasks.workunit.client.0.smithi167.stdout:2/364: mkdir dd/df/d1b/d6d 0 2022-01-31T19:44:12.457 INFO:tasks.workunit.client.0.smithi167.stdout:2/365: stat dd/df/d1b/d3d/d4a/f6b 0 2022-01-31T19:44:12.458 INFO:tasks.workunit.client.0.smithi167.stdout:9/430: getdents dd/d35 0 2022-01-31T19:44:12.458 INFO:tasks.workunit.client.0.smithi167.stdout:9/431: stat dd/d17/d80/d4c 0 2022-01-31T19:44:12.459 INFO:tasks.workunit.client.0.smithi167.stdout:2/366: stat dd/df/d1b/d2f/d36/c3b 0 2022-01-31T19:44:12.459 INFO:tasks.workunit.client.0.smithi167.stdout:9/432: rmdir dd/d19/d24/d50 39 2022-01-31T19:44:12.459 INFO:tasks.workunit.client.0.smithi167.stdout:9/433: fdatasync dd/d19/d24/d56/f85 0 2022-01-31T19:44:12.459 INFO:tasks.workunit.client.0.smithi167.stdout:2/367: truncate dd/df/d1b/f6a 899397 0 2022-01-31T19:44:12.462 INFO:tasks.workunit.client.0.smithi167.stdout:9/434: write f7 [669597,80422] 0 2022-01-31T19:44:12.463 INFO:tasks.workunit.client.0.smithi167.stdout:6/388: dwrite d8/d20/f21 [0,4194304] 0 2022-01-31T19:44:12.463 INFO:tasks.workunit.client.0.smithi167.stdout:6/389: dread - d8/d33/d4f/f59 zero size 2022-01-31T19:44:12.463 INFO:tasks.workunit.client.0.smithi167.stdout:6/390: creat d8/d33/f90 x:0 0 0 2022-01-31T19:44:12.463 INFO:tasks.workunit.client.0.smithi167.stdout:2/368: fsync dd/df/d1b/d3d/f44 0 2022-01-31T19:44:12.464 INFO:tasks.workunit.client.0.smithi167.stdout:2/369: creat dd/d18/d62/f6e x:0 0 0 2022-01-31T19:44:12.464 INFO:tasks.workunit.client.0.smithi167.stdout:2/370: truncate dd/d18/d62/f6e 791302 0 2022-01-31T19:44:12.464 INFO:tasks.workunit.client.0.smithi167.stdout:2/371: chown dd/df/d1b/d3d/f44 1936401 1 2022-01-31T19:44:12.464 INFO:tasks.workunit.client.0.smithi167.stdout:2/372: dread - dd/df/d3f/d51/d5f/f69 zero size 2022-01-31T19:44:12.465 INFO:tasks.workunit.client.0.smithi167.stdout:2/373: dread - dd/df/d3f/f41 zero size 2022-01-31T19:44:12.465 INFO:tasks.workunit.client.0.smithi167.stdout:6/391: symlink d8/d20/l91 0 2022-01-31T19:44:12.466 INFO:tasks.workunit.client.0.smithi167.stdout:2/374: mknod dd/df/d3f/d51/c6f 0 2022-01-31T19:44:12.467 INFO:tasks.workunit.client.0.smithi167.stdout:6/392: mkdir d8/d62/d92 0 2022-01-31T19:44:12.468 INFO:tasks.workunit.client.0.smithi167.stdout:2/375: mknod dd/c70 0 2022-01-31T19:44:12.468 INFO:tasks.workunit.client.0.smithi167.stdout:6/393: symlink d8/d62/l93 0 2022-01-31T19:44:12.470 INFO:tasks.workunit.client.0.smithi167.stdout:6/394: mkdir d8/d33/d51/d5d/d86/d94 0 2022-01-31T19:44:12.471 INFO:tasks.workunit.client.0.smithi167.stdout:2/376: mknod dd/df/d1b/d2f/d36/c71 0 2022-01-31T19:44:12.472 INFO:tasks.workunit.client.0.smithi167.stdout:6/395: dread d8/d33/f5f [0,4194304] 0 2022-01-31T19:44:12.473 INFO:tasks.workunit.client.0.smithi167.stdout:6/396: truncate d8/d20/f6b 918310 0 2022-01-31T19:44:12.473 INFO:tasks.workunit.client.0.smithi167.stdout:6/397: stat d8/d62 0 2022-01-31T19:44:12.473 INFO:tasks.workunit.client.0.smithi167.stdout:6/398: read - d8/d33/d51/d5d/d86/f8d zero size 2022-01-31T19:44:12.474 INFO:tasks.workunit.client.0.smithi167.stdout:6/399: creat d8/d33/d51/d72/f95 x:0 0 0 2022-01-31T19:44:12.474 INFO:tasks.workunit.client.0.smithi167.stdout:6/400: truncate d8/f7a 411539 0 2022-01-31T19:44:12.475 INFO:tasks.workunit.client.0.smithi167.stdout:6/401: creat d8/d33/d51/d5d/d86/d94/f96 x:0 0 0 2022-01-31T19:44:12.475 INFO:tasks.workunit.client.0.smithi167.stdout:6/402: creat d8/d33/d51/d5d/d86/d94/f97 x:0 0 0 2022-01-31T19:44:12.476 INFO:tasks.workunit.client.0.smithi167.stdout:6/403: mknod d8/d33/d51/d5d/d86/d94/c98 0 2022-01-31T19:44:12.481 INFO:tasks.workunit.client.0.smithi167.stdout:2/377: dread dd/df/d1b/d3d/f47 [0,4194304] 0 2022-01-31T19:44:12.497 INFO:tasks.workunit.client.0.smithi167.stdout:6/404: dwrite d8/d33/d35/f61 [0,4194304] 0 2022-01-31T19:44:12.498 INFO:tasks.workunit.client.0.smithi167.stdout:2/378: dwrite dd/df/d1b/d2f/d36/f2a [0,4194304] 0 2022-01-31T19:44:12.498 INFO:tasks.workunit.client.0.smithi167.stdout:2/379: fdatasync dd/d18/f4c 0 2022-01-31T19:44:12.499 INFO:tasks.workunit.client.0.smithi167.stdout:2/380: write dd/d18/d63/f2d [2431227,23890] 0 2022-01-31T19:44:12.501 INFO:tasks.workunit.client.0.smithi167.stdout:2/381: link dd/l19 dd/df/d3f/d51/d5f/l72 0 2022-01-31T19:44:12.503 INFO:tasks.workunit.client.0.smithi167.stdout:2/382: getdents dd/df/d1b/d2f 0 2022-01-31T19:44:12.503 INFO:tasks.workunit.client.0.smithi167.stdout:2/383: chown dd/df/d1b/d2f/c6c 686 1 2022-01-31T19:44:12.504 INFO:tasks.workunit.client.0.smithi167.stdout:2/384: read dd/f55 [4193609,112751] 0 2022-01-31T19:44:12.504 INFO:tasks.workunit.client.0.smithi167.stdout:2/385: write dd/df/d3f/d51/d5f/f69 [447138,77901] 0 2022-01-31T19:44:12.504 INFO:tasks.workunit.client.0.smithi167.stdout:2/386: mknod dd/df/d1b/d3d/d4a/d4b/c73 0 2022-01-31T19:44:12.507 INFO:tasks.workunit.client.0.smithi167.stdout:2/387: rename dd/df/d1b/d6d to dd/df/d1b/d3d/d4a/d4b/d74 0 2022-01-31T19:44:12.507 INFO:tasks.workunit.client.0.smithi167.stdout:2/388: chown dd/df/d1b/d2f/d36/c68 8883 1 2022-01-31T19:44:12.508 INFO:tasks.workunit.client.0.smithi167.stdout:6/405: dread d8/d20/f79 [0,4194304] 0 2022-01-31T19:44:12.508 INFO:tasks.workunit.client.0.smithi167.stdout:2/389: mkdir dd/d18/d62/d75 0 2022-01-31T19:44:12.509 INFO:tasks.workunit.client.0.smithi167.stdout:2/390: write dd/df/d1b/d2f/f4d [2259327,114399] 0 2022-01-31T19:44:12.510 INFO:tasks.workunit.client.0.smithi167.stdout:6/406: rename d8/d33/d51/l56 to d8/d33/d51/d5d/l99 0 2022-01-31T19:44:12.512 INFO:tasks.workunit.client.0.smithi167.stdout:2/391: symlink dd/df/d1b/d3d/d4a/d4b/d74/l76 0 2022-01-31T19:44:12.514 INFO:tasks.workunit.client.0.smithi167.stdout:6/407: mknod d8/d33/d51/d72/c9a 0 2022-01-31T19:44:12.515 INFO:tasks.workunit.client.0.smithi167.stdout:2/392: link dd/f55 dd/df/d22/f77 0 2022-01-31T19:44:12.516 INFO:tasks.workunit.client.0.smithi167.stdout:6/408: creat d8/d33/d51/d7e/f9b x:0 0 0 2022-01-31T19:44:12.518 INFO:tasks.workunit.client.0.smithi167.stdout:2/393: symlink dd/df/l78 0 2022-01-31T19:44:12.519 INFO:tasks.workunit.client.0.smithi167.stdout:2/394: truncate dd/df/d1b/d3d/f47 4130133 0 2022-01-31T19:44:12.520 INFO:tasks.workunit.client.0.smithi167.stdout:2/395: write dd/d18/d62/f64 [708543,31053] 0 2022-01-31T19:44:12.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.550+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb6d4005940 con 0x7fb6f40f4c60 2022-01-31T19:44:12.553 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.551+0000 7fb6db7fe700 1 -- 172.21.15.167:0/1317149099 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1689 (secure 0 0 0) 0x7fb6e4050df0 con 0x7fb6f40f4c60 2022-01-31T19:44:12.553 INFO:teuthology.orchestra.run.smithi167.stdout:e16 2022-01-31T19:44:12.554 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:44:12.554 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:44:12.554 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:44:12.554 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:44:12.555 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:44:12.555 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:44:12.555 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 16 2022-01-31T19:44:12.555 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:44:12.556 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:44:12.556 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:43:36.817533+0000 2022-01-31T19:44:12.556 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:44:12.556 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:44:12.557 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:44:12.557 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:44:12.557 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:44:12.557 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:44:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:44:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:44:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:44:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:44:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:44:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:44:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:44:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:44:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:44:12.560 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:44:12.560 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:44:12.560 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:44:12.560 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:44:12.561 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:44:12.561 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:44:12.561 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:44:12.561 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:44:12.561 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:44:12.562 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:44:12.562 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:44:12.563 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.552+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fb6dc0619a0 msgr2=0x7fb6dc063e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:12.563 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.552+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fb6dc0619a0 0x7fb6dc063e60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb6f40f7690 tx=0x7fb6e800b040).stop 2022-01-31T19:44:12.563 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.552+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 msgr2=0x7fb6f40f5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:12.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.552+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40f5d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fb6e4009990 tx=0x7fb6e4009b30).stop 2022-01-31T19:44:12.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 shutdown_connections 2022-01-31T19:44:12.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:6828/1254375478,v1:172.21.15.171:6829/1254375478] conn(0x7fb6dc0619a0 0x7fb6dc063e60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.564 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6f40f42c0 0x7fb6f40f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 --2- 172.21.15.167:0/1317149099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6f40f4c60 0x7fb6f40f5d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:12.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 >> 172.21.15.167:0/1317149099 conn(0x7fb6f40f0020 msgr2=0x7fb6f40f8690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:12.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 shutdown_connections 2022-01-31T19:44:12.565 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:12.553+0000 7fb6f3fff700 1 -- 172.21.15.167:0/1317149099 wait complete. 2022-01-31T19:44:12.566 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 16 2022-01-31T19:44:12.608 INFO:tasks.workunit.client.0.smithi167.stdout:3/433: sync 2022-01-31T19:44:12.608 INFO:tasks.workunit.client.0.smithi167.stdout:1/540: sync 2022-01-31T19:44:12.608 INFO:tasks.workunit.client.0.smithi167.stdout:4/459: sync 2022-01-31T19:44:12.609 INFO:tasks.workunit.client.0.smithi167.stdout:3/434: chown d9/d29/c2f 1806445111 1 2022-01-31T19:44:12.611 INFO:tasks.workunit.client.0.smithi167.stdout:1/541: mkdir d0/d10/d5a/d33/d4b/da5/dc1 0 2022-01-31T19:44:12.612 INFO:tasks.workunit.client.0.smithi167.stdout:4/460: creat d0/d3f/d75/f98 x:0 0 0 2022-01-31T19:44:12.614 INFO:tasks.workunit.client.0.smithi167.stdout:3/435: link d9/d21/c1f d9/c7d 0 2022-01-31T19:44:12.614 INFO:tasks.workunit.client.0.smithi167.stdout:3/436: stat d9/d29/c43 0 2022-01-31T19:44:12.614 INFO:tasks.workunit.client.0.smithi167.stdout:3/437: truncate d9/f30 1283084 0 2022-01-31T19:44:12.615 INFO:tasks.workunit.client.0.smithi167.stdout:3/438: chown d9/d21/l34 0 1 2022-01-31T19:44:12.615 INFO:tasks.workunit.client.0.smithi167.stdout:1/542: link d0/d11/fb0 d0/d75/d7a/fc2 0 2022-01-31T19:44:12.616 INFO:tasks.workunit.client.0.smithi167.stdout:4/461: getdents d0/d6/d7/d39/d4e 0 2022-01-31T19:44:12.617 INFO:tasks.workunit.client.0.smithi167.stdout:4/462: rename d0/d6/d7/d89 to d0/d6/d7/d89/d99 22 2022-01-31T19:44:12.617 INFO:tasks.workunit.client.0.smithi167.stdout:4/463: creat d0/d6/d7/f9a x:0 0 0 2022-01-31T19:44:12.617 INFO:tasks.workunit.client.0.smithi167.stdout:3/439: mknod d9/d21/c7e 0 2022-01-31T19:44:12.619 INFO:tasks.workunit.client.0.smithi167.stdout:4/464: rmdir d0/d6 39 2022-01-31T19:44:12.619 INFO:tasks.workunit.client.0.smithi167.stdout:4/465: unlink d0/d3f/d1a/c4b 0 2022-01-31T19:44:12.620 INFO:tasks.workunit.client.0.smithi167.stdout:4/466: readlink d0/d3f/d60/l63 0 2022-01-31T19:44:12.623 INFO:tasks.workunit.client.0.smithi167.stdout:4/467: unlink d0/d6/d7/d89/f97 0 2022-01-31T19:44:12.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/440: rmdir d9/d26 39 2022-01-31T19:44:12.625 INFO:tasks.workunit.client.0.smithi167.stdout:4/468: creat d0/f9b x:0 0 0 2022-01-31T19:44:12.625 INFO:tasks.workunit.client.0.smithi167.stdout:4/469: chown d0/d6/d7/d26/d31/f8b 30 1 2022-01-31T19:44:12.625 INFO:tasks.workunit.client.0.smithi167.stdout:4/470: chown d0/d6/d7/d39/l83 104707648 1 2022-01-31T19:44:12.626 INFO:tasks.workunit.client.0.smithi167.stdout:4/471: readlink d0/d6/l8f 0 2022-01-31T19:44:12.626 INFO:tasks.workunit.client.0.smithi167.stdout:4/472: fsync d0/d3f/d60/f77 0 2022-01-31T19:44:12.626 INFO:tasks.workunit.client.0.smithi167.stdout:1/543: dread d0/d10/d18/f3e [0,4194304] 0 2022-01-31T19:44:12.627 INFO:tasks.workunit.client.0.smithi167.stdout:3/441: mkdir d9/d7f 0 2022-01-31T19:44:12.627 INFO:tasks.workunit.client.0.smithi167.stdout:4/473: link d0/d6/f76 d0/d6/d7/d26/d40/d4c/f9c 0 2022-01-31T19:44:12.632 INFO:tasks.workunit.client.0.smithi167.stdout:1/544: rename d0/d11/l63 to d0/d10/d5a/d33/d4b/da5/da9/dad/lc3 0 2022-01-31T19:44:12.632 INFO:tasks.workunit.client.0.smithi167.stdout:1/545: write d0/d62/d7f/d82/f88 [144932,21034] 0 2022-01-31T19:44:12.633 INFO:tasks.workunit.client.0.smithi167.stdout:3/442: unlink d9/d26/f31 0 2022-01-31T19:44:12.633 INFO:tasks.workunit.client.0.smithi167.stdout:3/443: fdatasync d9/d29/f2b 0 2022-01-31T19:44:12.634 INFO:tasks.workunit.client.0.smithi167.stdout:3/444: creat d9/d29/d3b/d41/d6b/f80 x:0 0 0 2022-01-31T19:44:12.634 INFO:tasks.workunit.client.0.smithi167.stdout:3/445: creat d9/f81 x:0 0 0 2022-01-31T19:44:12.634 INFO:tasks.workunit.client.0.smithi167.stdout:1/546: write d0/d10/d5a/d33/d4b/f68 [9777549,89288] 0 2022-01-31T19:44:12.656 INFO:tasks.workunit.client.0.smithi167.stdout:3/446: dwrite d9/f27 [0,4194304] 0 2022-01-31T19:44:12.656 INFO:tasks.workunit.client.0.smithi167.stdout:3/447: fdatasync d9/f78 0 2022-01-31T19:44:12.657 INFO:tasks.workunit.client.0.smithi167.stdout:1/547: dwrite d0/f72 [0,4194304] 0 2022-01-31T19:44:12.657 INFO:tasks.workunit.client.0.smithi167.stdout:4/474: dwrite d0/d6/d7/d26/d31/f11 [4194304,4194304] 0 2022-01-31T19:44:12.658 INFO:tasks.workunit.client.0.smithi167.stdout:4/475: fsync d0/d3f/f2e 0 2022-01-31T19:44:12.658 INFO:tasks.workunit.client.0.smithi167.stdout:3/448: link d9/c7d d9/d29/d3b/c82 0 2022-01-31T19:44:12.659 INFO:tasks.workunit.client.0.smithi167.stdout:1/548: mknod d0/cc4 0 2022-01-31T19:44:12.661 INFO:tasks.workunit.client.0.smithi167.stdout:4/476: rename d0/d6/d7/d26/d40/f27 to d0/d3f/d60/d65/f9d 0 2022-01-31T19:44:12.662 INFO:tasks.workunit.client.0.smithi167.stdout:1/549: link d0/f93 d0/db3/fc5 0 2022-01-31T19:44:12.663 INFO:tasks.workunit.client.0.smithi167.stdout:1/550: write d0/d10/d5a/d33/d4b/da5/fa0 [748272,47178] 0 2022-01-31T19:44:12.663 INFO:tasks.workunit.client.0.smithi167.stdout:3/449: getdents d9/d29 0 2022-01-31T19:44:12.664 INFO:tasks.workunit.client.0.smithi167.stdout:3/450: rmdir d9/d29/d3b/d41 39 2022-01-31T19:44:12.666 INFO:tasks.workunit.client.0.smithi167.stdout:3/451: symlink d9/d29/l83 0 2022-01-31T19:44:12.666 INFO:tasks.workunit.client.0.smithi167.stdout:3/452: chown d9/d29/d3b/d41/d6b/f39 25629 1 2022-01-31T19:44:12.667 INFO:tasks.workunit.client.0.smithi167.stdout:3/453: chown d9/d29/d60/l62 1969 1 2022-01-31T19:44:12.667 INFO:tasks.workunit.client.0.smithi167.stdout:3/454: truncate d9/f10 791735 0 2022-01-31T19:44:12.673 INFO:tasks.workunit.client.0.smithi167.stdout:1/551: dread d0/d10/d20/f37 [0,4194304] 0 2022-01-31T19:44:12.676 INFO:tasks.workunit.client.0.smithi167.stdout:3/455: getdents d9/d26 0 2022-01-31T19:44:12.679 INFO:tasks.workunit.client.0.smithi167.stdout:1/552: creat d0/d62/fc6 x:0 0 0 2022-01-31T19:44:12.681 INFO:tasks.workunit.client.0.smithi167.stdout:3/456: rename d9/d29/d3b/d41/d6b/f67 to d9/d29/d3b/f84 0 2022-01-31T19:44:12.681 INFO:tasks.workunit.client.0.smithi167.stdout:4/477: dwrite d0/d6/d7/f7b [0,4194304] 0 2022-01-31T19:44:12.684 INFO:tasks.workunit.client.0.smithi167.stdout:3/457: write d9/d56/f73 [978488,115389] 0 2022-01-31T19:44:12.685 INFO:tasks.workunit.client.0.smithi167.stdout:4/478: write d0/d6/d7/f30 [2428635,103578] 0 2022-01-31T19:44:12.697 INFO:tasks.workunit.client.0.smithi167.stdout:1/553: dwrite d0/d10/d5a/d33/fbb [0,4194304] 0 2022-01-31T19:44:12.698 INFO:tasks.workunit.client.0.smithi167.stdout:3/458: dread d9/d56/f6d [0,4194304] 0 2022-01-31T19:44:12.700 INFO:tasks.workunit.client.0.smithi167.stdout:1/554: write d0/d10/d5a/d33/fbb [3952840,128523] 0 2022-01-31T19:44:12.702 INFO:tasks.workunit.client.0.smithi167.stdout:1/555: symlink d0/d75/dbd/lc7 0 2022-01-31T19:44:12.702 INFO:tasks.workunit.client.0.smithi167.stdout:1/556: chown d0/d62/d7f 14930664 1 2022-01-31T19:44:12.703 INFO:tasks.workunit.client.0.smithi167.stdout:4/479: dwrite d0/d3f/f2e [0,4194304] 0 2022-01-31T19:44:12.704 INFO:tasks.workunit.client.0.smithi167.stdout:4/480: chown d0/d6/d7/d39/d4e/d61/c8e 1 1 2022-01-31T19:44:12.704 INFO:tasks.workunit.client.0.smithi167.stdout:1/557: rmdir d0/d10/d5a/d33/d4b/da5/da9 39 2022-01-31T19:44:12.704 INFO:tasks.workunit.client.0.smithi167.stdout:1/558: creat d0/d10/d20/d7c/fc8 x:0 0 0 2022-01-31T19:44:12.704 INFO:tasks.workunit.client.0.smithi167.stdout:1/559: fsync d0/d10/f42 0 2022-01-31T19:44:12.706 INFO:tasks.workunit.client.0.smithi167.stdout:4/481: unlink d0/f9b 0 2022-01-31T19:44:12.709 INFO:tasks.workunit.client.0.smithi167.stdout:1/560: truncate d0/d10/d5a/d33/d4b/f5b 2716984 0 2022-01-31T19:44:12.709 INFO:tasks.workunit.client.0.smithi167.stdout:1/561: truncate d0/d75/d7a/f59 1305483 0 2022-01-31T19:44:12.710 INFO:tasks.workunit.client.0.smithi167.stdout:1/562: mknod d0/d10/d20/cc9 0 2022-01-31T19:44:12.711 INFO:tasks.workunit.client.0.smithi167.stdout:4/482: dread d0/d3f/d60/d65/f9d [0,4194304] 0 2022-01-31T19:44:12.711 INFO:tasks.workunit.client.0.smithi167.stdout:1/563: write d0/d75/d7a/f74 [1507630,85751] 0 2022-01-31T19:44:12.711 INFO:tasks.workunit.client.0.smithi167.stdout:4/483: write d0/d6/d7/d39/d3a/f54 [1211112,71534] 0 2022-01-31T19:44:12.711 INFO:tasks.workunit.client.0.smithi167.stdout:4/484: dread - d0/d6/d7/d26/d40/d4c/f7d zero size 2022-01-31T19:44:12.712 INFO:tasks.workunit.client.0.smithi167.stdout:4/485: mknod d0/d6/d7/d39/d3a/c9e 0 2022-01-31T19:44:12.713 INFO:tasks.workunit.client.0.smithi167.stdout:4/486: mknod d0/d6/d7/d39/d3a/d79/c9f 0 2022-01-31T19:44:12.713 INFO:tasks.workunit.client.0.smithi167.stdout:4/487: chown d0/d3f/d75/c82 1179148 1 2022-01-31T19:44:12.713 INFO:tasks.workunit.client.0.smithi167.stdout:4/488: write d0/d6/d7/d26/d40/d4c/f86 [768626,7140] 0 2022-01-31T19:44:12.716 INFO:tasks.workunit.client.0.smithi167.stdout:4/489: mknod d0/d6/d7/d26/d31/d5c/ca0 0 2022-01-31T19:44:12.727 INFO:tasks.workunit.client.0.smithi167.stdout:1/564: dwrite d0/d10/d5a/d33/f95 [0,4194304] 0 2022-01-31T19:44:12.727 INFO:tasks.workunit.client.0.smithi167.stdout:1/565: dread - d0/d11/fab zero size 2022-01-31T19:44:12.727 INFO:tasks.workunit.client.0.smithi167.stdout:1/566: readlink d0/d75/d7a/l6a 0 2022-01-31T19:44:12.736 INFO:tasks.workunit.client.0.smithi167.stdout:4/490: dwrite d0/d6/d7/d39/d4e/d61/f95 [4194304,4194304] 0 2022-01-31T19:44:12.742 INFO:tasks.workunit.client.0.smithi167.stdout:1/567: dwrite d0/f17 [0,4194304] 0 2022-01-31T19:44:12.742 INFO:tasks.workunit.client.0.smithi167.stdout:1/568: truncate d0/d10/d5a/d33/fa7 1033058 0 2022-01-31T19:44:12.742 INFO:tasks.workunit.client.0.smithi167.stdout:1/569: fsync d0/f25 0 2022-01-31T19:44:12.745 INFO:tasks.workunit.client.0.smithi167.stdout:1/570: write d0/d10/d5a/d33/d4b/d1f/fa6 [2900349,114410] 0 2022-01-31T19:44:12.746 INFO:tasks.workunit.client.0.smithi167.stdout:1/571: symlink d0/d11/lca 0 2022-01-31T19:44:12.747 INFO:tasks.workunit.client.0.smithi167.stdout:1/572: mknod d0/d10/d20/d7c/ccb 0 2022-01-31T19:44:12.747 INFO:tasks.workunit.client.0.smithi167.stdout:1/573: dread - d0/d10/d5a/d33/d4b/fa2 zero size 2022-01-31T19:44:12.748 INFO:tasks.workunit.client.0.smithi167.stdout:1/574: creat d0/d10/d18/fcc x:0 0 0 2022-01-31T19:44:12.749 INFO:tasks.workunit.client.0.smithi167.stdout:1/575: fsync d0/d62/d7f/d82/f88 0 2022-01-31T19:44:12.750 INFO:tasks.workunit.client.0.smithi167.stdout:1/576: write d0/d10/d5a/d33/d4b/d1f/f78 [1851327,91894] 0 2022-01-31T19:44:12.750 INFO:tasks.workunit.client.0.smithi167.stdout:1/577: chown d0/d10/d5a/d33/d4b/da5/dc1 1 1 2022-01-31T19:44:12.760 INFO:tasks.workunit.client.0.smithi167.stdout:4/491: dwrite d0/d6/d7/d39/f2b [0,4194304] 0 2022-01-31T19:44:12.760 INFO:tasks.workunit.client.0.smithi167.stdout:4/492: dread - d0/f7a zero size 2022-01-31T19:44:12.771 INFO:tasks.workunit.client.0.smithi167.stdout:1/578: dwrite d0/d10/d18/fcc [0,4194304] 0 2022-01-31T19:44:12.771 INFO:tasks.workunit.client.0.smithi167.stdout:1/579: chown d0/d10/d20/l40 1745802 1 2022-01-31T19:44:12.771 INFO:tasks.workunit.client.0.smithi167.stdout:1/580: write d0/d10/d5a/d33/d4b/f65 [2735208,105023] 0 2022-01-31T19:44:12.786 INFO:tasks.workunit.client.0.smithi167.stdout:1/581: dwrite d0/d10/d5a/d33/d4b/f6d [0,4194304] 0 2022-01-31T19:44:12.791 INFO:tasks.workunit.client.0.smithi167.stdout:1/582: dread d0/d10/d18/f3e [0,4194304] 0 2022-01-31T19:44:12.791 INFO:tasks.workunit.client.0.smithi167.stdout:1/583: fdatasync d0/d10/f42 0 2022-01-31T19:44:12.792 INFO:tasks.workunit.client.0.smithi167.stdout:1/584: chown d0/d75/d7a/f66 18827 1 2022-01-31T19:44:12.792 INFO:tasks.workunit.client.0.smithi167.stdout:1/585: readlink d0/d62/lbe 0 2022-01-31T19:44:12.792 INFO:tasks.workunit.client.0.smithi167.stdout:1/586: write d0/d10/d20/d7c/fc8 [212921,81239] 0 2022-01-31T19:44:12.792 INFO:tasks.workunit.client.0.smithi167.stdout:1/587: fsync d0/d10/f42 0 2022-01-31T19:44:12.793 INFO:tasks.workunit.client.0.smithi167.stdout:1/588: link d0/d10/d20/f28 d0/d10/d20/fcd 0 2022-01-31T19:44:12.794 INFO:tasks.workunit.client.0.smithi167.stdout:1/589: dread - d0/d10/d20/fb2 zero size 2022-01-31T19:44:12.794 INFO:tasks.workunit.client.0.smithi167.stdout:1/590: truncate d0/d62/d7f/d82/f88 1310304 0 2022-01-31T19:44:12.794 INFO:tasks.workunit.client.0.smithi167.stdout:1/591: write d0/d10/d5a/fb5 [522308,73085] 0 2022-01-31T19:44:12.794 INFO:tasks.workunit.client.0.smithi167.stdout:1/592: read d0/d10/d5a/d33/f55 [36544,28332] 0 2022-01-31T19:44:12.795 INFO:tasks.workunit.client.0.smithi167.stdout:1/593: write d0/d62/d7f/d82/f88 [978935,3898] 0 2022-01-31T19:44:12.796 INFO:tasks.workunit.client.0.smithi167.stdout:1/594: fdatasync d0/d10/d5a/d33/d4b/f65 0 2022-01-31T19:44:12.797 INFO:tasks.workunit.client.0.smithi167.stdout:1/595: mknod d0/d62/d7f/d82/cce 0 2022-01-31T19:44:12.797 INFO:tasks.workunit.client.0.smithi167.stdout:1/596: dread - d0/d75/d7a/fc2 zero size 2022-01-31T19:44:12.798 INFO:tasks.workunit.client.0.smithi167.stdout:1/597: read d0/d10/d20/f28 [3288180,1368] 0 2022-01-31T19:44:12.798 INFO:tasks.workunit.client.0.smithi167.stdout:1/598: chown d0/d10/d5a/d33/d97 16397 1 2022-01-31T19:44:12.798 INFO:tasks.workunit.client.0.smithi167.stdout:1/599: symlink d0/d10/d5a/d33/d4b/da5/dc1/lcf 0 2022-01-31T19:44:12.800 INFO:tasks.workunit.client.0.smithi167.stdout:1/600: creat d0/d10/d5a/d33/d97/fd0 x:0 0 0 2022-01-31T19:44:12.816 INFO:tasks.workunit.client.0.smithi167.stdout:1/601: dwrite d0/d10/d5a/d33/d4b/fa2 [0,4194304] 0 2022-01-31T19:44:12.816 INFO:tasks.workunit.client.0.smithi167.stdout:1/602: creat d0/d10/d20/d7c/fd1 x:0 0 0 2022-01-31T19:44:12.820 INFO:tasks.workunit.client.0.smithi167.stdout:1/603: symlink d0/d11/ld2 0 2022-01-31T19:44:12.820 INFO:tasks.workunit.client.0.smithi167.stdout:1/604: stat d0/d10/d18/fcc 0 2022-01-31T19:44:12.826 INFO:tasks.workunit.client.0.smithi167.stdout:1/605: dread d0/d10/d20/f4e [0,4194304] 0 2022-01-31T19:44:12.828 INFO:tasks.workunit.client.0.smithi167.stdout:1/606: rename d0/d10/d5a/d33/d4b/c41 to d0/d10/d20/d7c/cd3 0 2022-01-31T19:44:12.835 INFO:tasks.workunit.client.0.smithi167.stdout:1/607: dread d0/d10/d5a/d33/fbb [0,4194304] 0 2022-01-31T19:44:12.836 INFO:tasks.workunit.client.0.smithi167.stdout:1/608: creat d0/d10/d20/d7c/fd4 x:0 0 0 2022-01-31T19:44:12.848 INFO:tasks.workunit.client.0.smithi167.stdout:1/609: dwrite d0/d11/d46/f99 [0,4194304] 0 2022-01-31T19:44:12.850 INFO:tasks.workunit.client.0.smithi167.stdout:1/610: mknod d0/d10/d5a/d33/d4b/da5/cd5 0 2022-01-31T19:44:12.850 INFO:tasks.workunit.client.0.smithi167.stdout:1/611: truncate d0/d75/d7a/f59 1344856 0 2022-01-31T19:44:12.863 INFO:tasks.workunit.client.0.smithi167.stdout:1/612: dwrite d0/d10/d5a/d33/d4b/da5/fa0 [0,4194304] 0 2022-01-31T19:44:12.863 INFO:tasks.workunit.client.0.smithi167.stdout:1/613: fdatasync d0/f3a 0 2022-01-31T19:44:12.877 INFO:tasks.workunit.client.0.smithi167.stdout:1/614: dwrite d0/d10/d5a/d33/fbb [0,4194304] 0 2022-01-31T19:44:12.886 INFO:tasks.workunit.client.0.smithi167.stdout:0/264: sync 2022-01-31T19:44:12.887 INFO:tasks.workunit.client.0.smithi167.stdout:7/626: sync 2022-01-31T19:44:12.887 INFO:tasks.workunit.client.0.smithi167.stdout:8/569: sync 2022-01-31T19:44:12.888 INFO:tasks.workunit.client.0.smithi167.stdout:0/265: mknod d17/d19/d1c/d39/c4f 0 2022-01-31T19:44:12.888 INFO:tasks.workunit.client.0.smithi167.stdout:0/266: fsync d17/f2f 0 2022-01-31T19:44:12.888 INFO:tasks.workunit.client.0.smithi167.stdout:0/267: write d17/d19/f1f [284454,37241] 0 2022-01-31T19:44:12.889 INFO:tasks.workunit.client.0.smithi167.stdout:8/570: symlink d6/db/d6a/d8d/lbc 0 2022-01-31T19:44:12.889 INFO:tasks.workunit.client.0.smithi167.stdout:7/627: symlink d8/d2c/d6e/d4d/d67/d8f/le0 0 2022-01-31T19:44:12.890 INFO:tasks.workunit.client.0.smithi167.stdout:0/268: symlink d17/d2c/l50 0 2022-01-31T19:44:12.890 INFO:tasks.workunit.client.0.smithi167.stdout:0/269: write d17/d19/f1f [895032,122889] 0 2022-01-31T19:44:12.891 INFO:tasks.workunit.client.0.smithi167.stdout:8/571: mkdir d6/db/d4e/dbd 0 2022-01-31T19:44:12.891 INFO:tasks.workunit.client.0.smithi167.stdout:8/572: read d6/db/d4e/d73/fa2 [783689,878] 0 2022-01-31T19:44:12.891 INFO:tasks.workunit.client.0.smithi167.stdout:7/628: getdents d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83 0 2022-01-31T19:44:12.892 INFO:tasks.workunit.client.0.smithi167.stdout:0/270: rename d17/d19/d1c/f21 to d17/d19/f51 0 2022-01-31T19:44:12.893 INFO:tasks.workunit.client.0.smithi167.stdout:0/271: mknod d17/d3e/c52 0 2022-01-31T19:44:12.894 INFO:tasks.workunit.client.0.smithi167.stdout:7/629: getdents d8/d2c/d6e/d4d/d6b 0 2022-01-31T19:44:12.894 INFO:tasks.workunit.client.0.smithi167.stdout:7/630: rmdir d8/dc 39 2022-01-31T19:44:12.895 INFO:tasks.workunit.client.0.smithi167.stdout:7/631: mknod d8/d2c/d6e/d9a/ce1 0 2022-01-31T19:44:12.898 INFO:tasks.workunit.client.0.smithi167.stdout:0/272: dread d17/d19/f51 [0,4194304] 0 2022-01-31T19:44:12.898 INFO:tasks.workunit.client.0.smithi167.stdout:0/273: fdatasync d17/f3d 0 2022-01-31T19:44:12.899 INFO:tasks.workunit.client.0.smithi167.stdout:1/615: dwrite d0/d10/d18/f3e [0,4194304] 0 2022-01-31T19:44:12.899 INFO:tasks.workunit.client.0.smithi167.stdout:1/616: chown d0/d10/d20/cc9 1816988933 1 2022-01-31T19:44:12.903 INFO:tasks.workunit.client.0.smithi167.stdout:1/617: rmdir d0/d10/d5a/d33/d97 39 2022-01-31T19:44:12.904 INFO:tasks.workunit.client.0.smithi167.stdout:7/632: dread d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f3c [0,4194304] 0 2022-01-31T19:44:12.905 INFO:tasks.workunit.client.0.smithi167.stdout:0/274: write d17/f22 [2617395,49259] 0 2022-01-31T19:44:12.906 INFO:tasks.workunit.client.0.smithi167.stdout:1/618: creat d0/fd6 x:0 0 0 2022-01-31T19:44:12.906 INFO:tasks.workunit.client.0.smithi167.stdout:1/619: chown d0/d75/d7a/c47 25637371 1 2022-01-31T19:44:12.906 INFO:tasks.workunit.client.0.smithi167.stdout:1/620: dread - d0/d10/f19 zero size 2022-01-31T19:44:12.907 INFO:tasks.workunit.client.0.smithi167.stdout:0/275: fsync d17/d19/f51 0 2022-01-31T19:44:12.907 INFO:tasks.workunit.client.0.smithi167.stdout:1/621: write d0/d10/d5a/fb9 [740253,54014] 0 2022-01-31T19:44:12.910 INFO:tasks.workunit.client.0.smithi167.stdout:0/276: getdents d17/d19 0 2022-01-31T19:44:12.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: audit 2022-01-31T19:44:11.151267+0000 mgr.smithi171.asyxnc (mgr.24387) 2022-01-31T19:44:12.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: 16 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:12.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: audit 2022-01-31T19:44:11.531135+0000 mgr.smithi171.asyxnc (mgr.24387) 17 : audit [DBG] from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:12.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:44:12.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: 44:12.097584+0000 mon.smithi167 (mon.0) 661 : audit [DBG] from='client.? 172.21.15.167:0/847153096' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:44:12.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: audit 2022-01-31T19:44:12.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: 2022-01-31T19:44:12.552399+0000 mon.smithi167 (mon.0 2022-01-31T19:44:12.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:12 smithi167 conmon[32206]: ) 662 : audit [DBG] from='client.? 172.21.15.167:0/1317149099' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:44:12.922 INFO:tasks.workunit.client.0.smithi167.stdout:7/633: dwrite d8/d2c/f25 [0,4194304] 0 2022-01-31T19:44:12.925 INFO:tasks.workunit.client.0.smithi167.stdout:7/634: dread d8/dc/d18/d24/f73 [0,4194304] 0 2022-01-31T19:44:12.929 INFO:tasks.workunit.client.0.smithi167.stdout:7/635: rename d8/d2c/d6e/d4d/d6b/d8e to d8/d2c/d7a/d99/de2 0 2022-01-31T19:44:12.931 INFO:tasks.workunit.client.0.smithi167.stdout:7/636: link d8/dab/cac d8/dc/ce3 0 2022-01-31T19:44:12.931 INFO:tasks.workunit.client.0.smithi167.stdout:7/637: write d8/dc/d18/d24/dbb/f6d [4875992,128233] 0 2022-01-31T19:44:12.932 INFO:tasks.workunit.client.0.smithi167.stdout:7/638: dread d8/d2c/d6e/d4d/d6b/f81 [0,4194304] 0 2022-01-31T19:44:12.933 INFO:tasks.workunit.client.0.smithi167.stdout:7/639: creat d8/dab/fe4 x:0 0 0 2022-01-31T19:44:12.934 INFO:tasks.workunit.client.0.smithi167.stdout:7/640: mknod d8/d2c/d7a/dd3/ce5 0 2022-01-31T19:44:12.934 INFO:tasks.workunit.client.0.smithi167.stdout:7/641: chown d8/d2c/d6e/c37 40 1 2022-01-31T19:44:12.935 INFO:tasks.workunit.client.0.smithi167.stdout:7/642: creat d8/d2c/d43/d94/daf/fe6 x:0 0 0 2022-01-31T19:44:12.952 INFO:tasks.workunit.client.0.smithi167.stdout:7/643: dwrite d8/d2c/d6e/d9a/fd4 [0,4194304] 0 2022-01-31T19:44:12.952 INFO:tasks.workunit.client.0.smithi167.stdout:7/644: chown d8/d2c/d43/d94/daf 253137 1 2022-01-31T19:44:12.952 INFO:tasks.workunit.client.0.smithi167.stdout:7/645: read d8/d35/f4b [1243573,2263] 0 2022-01-31T19:44:12.953 INFO:tasks.workunit.client.0.smithi167.stdout:7/646: symlink d8/dc/dc1/le7 0 2022-01-31T19:44:12.955 INFO:tasks.workunit.client.0.smithi167.stdout:7/647: write d8/d2c/d6e/d4d/d5e/f91 [380548,90429] 0 2022-01-31T19:44:12.955 INFO:tasks.workunit.client.0.smithi167.stdout:7/648: truncate d8/d2c/d7a/d99/de2/fde 88565 0 2022-01-31T19:44:12.955 INFO:tasks.workunit.client.0.smithi167.stdout:7/649: write d8/d2c/f25 [5354132,120016] 0 2022-01-31T19:44:12.956 INFO:tasks.workunit.client.0.smithi167.stdout:7/650: link d8/d35/c4a d8/d2c/d7a/dd3/ce8 0 2022-01-31T19:44:12.958 INFO:tasks.workunit.client.0.smithi167.stdout:7/651: mkdir d8/d2c/d7a/d99/de2/dc3/de9 0 2022-01-31T19:44:12.958 INFO:tasks.workunit.client.0.smithi167.stdout:7/652: stat d8/dc/d18/l8c 0 2022-01-31T19:44:12.959 INFO:tasks.workunit.client.0.smithi167.stdout:7/653: write d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f68 [573942,938] 0 2022-01-31T19:44:12.959 INFO:tasks.workunit.client.0.smithi167.stdout:7/654: chown d8/fa 6582 1 2022-01-31T19:44:12.960 INFO:tasks.workunit.client.0.smithi167.stdout:7/655: symlink d8/d2c/d6e/d4d/d5e/lea 0 2022-01-31T19:44:12.960 INFO:tasks.workunit.client.0.smithi167.stdout:7/656: read d8/d2c/d6e/d4d/d5e/d71/db3/fb8 [3090235,14574] 0 2022-01-31T19:44:12.963 INFO:tasks.workunit.client.0.smithi167.stdout:7/657: creat d8/feb x:0 0 0 2022-01-31T19:44:12.964 INFO:tasks.workunit.client.0.smithi167.stdout:7/658: readlink d8/d2c/d6e/d4d/d67/d8f/le0 0 2022-01-31T19:44:12.964 INFO:tasks.workunit.client.0.smithi167.stdout:7/659: creat d8/d2c/d6e/d4d/d5e/d71/db1/d8d/fec x:0 0 0 2022-01-31T19:44:12.964 INFO:tasks.workunit.client.0.smithi167.stdout:7/660: creat d8/dab/fed x:0 0 0 2022-01-31T19:44:12.966 INFO:tasks.workunit.client.0.smithi167.stdout:7/661: truncate d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f1a 2202024 0 2022-01-31T19:44:12.968 INFO:tasks.workunit.client.0.smithi167.stdout:7/662: mkdir d8/d2c/d43/d94/daf/dee 0 2022-01-31T19:44:12.968 INFO:tasks.workunit.client.0.smithi167.stdout:7/663: dread - d8/dab/fe4 zero size 2022-01-31T19:44:12.969 INFO:tasks.workunit.client.0.smithi167.stdout:7/664: mknod d8/d2c/d7a/d99/de2/cef 0 2022-01-31T19:44:12.971 INFO:tasks.workunit.client.0.smithi167.stdout:7/665: creat d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/ff0 x:0 0 0 2022-01-31T19:44:12.973 INFO:tasks.workunit.client.0.smithi167.stdout:7/666: link d8/d2c/d43/d94/c9f d8/d2c/d7a/dd3/cf1 0 2022-01-31T19:44:12.973 INFO:tasks.workunit.client.0.smithi167.stdout:7/667: write d8/dc/f2b [690878,88194] 0 2022-01-31T19:44:12.974 INFO:tasks.workunit.client.0.smithi167.stdout:9/435: sync 2022-01-31T19:44:12.974 INFO:tasks.workunit.client.0.smithi167.stdout:5/634: sync 2022-01-31T19:44:12.974 INFO:tasks.workunit.client.0.smithi167.stdout:5/635: chown d5/d6/d16/d18/d43/d45/d69/dc4/fb4 54 1 2022-01-31T19:44:12.975 INFO:tasks.workunit.client.0.smithi167.stdout:9/436: rename dd/d19/f27 to dd/d35/f8d 0 2022-01-31T19:44:13.054 INFO:tasks.workunit.client.0.smithi167.stdout:7/668: rmdir d8 39 2022-01-31T19:44:13.056 INFO:tasks.workunit.client.0.smithi167.stdout:5/636: rename d5/d6/f42 to d5/d6/d16/d18/d43/fc7 0 2022-01-31T19:44:13.057 INFO:tasks.workunit.client.0.smithi167.stdout:9/437: creat dd/d19/d24/d56/f8e x:0 0 0 2022-01-31T19:44:13.057 INFO:tasks.workunit.client.0.smithi167.stdout:5/637: write d5/d6/d16/d18/d5b/d63/f98 [1450679,62479] 0 2022-01-31T19:44:13.058 INFO:tasks.workunit.client.0.smithi167.stdout:5/638: truncate d5/d6/de/d22/f35 1039752 0 2022-01-31T19:44:13.059 INFO:tasks.workunit.client.0.smithi167.stdout:7/669: creat d8/d2c/d7a/d99/de2/dc3/ff2 x:0 0 0 2022-01-31T19:44:13.059 INFO:tasks.workunit.client.0.smithi167.stdout:7/670: fdatasync d8/d2c/d6e/f44 0 2022-01-31T19:44:13.059 INFO:tasks.workunit.client.0.smithi167.stdout:7/671: fdatasync d8/d2c/d6e/d4d/d5e/d71/db1/d8d/fec 0 2022-01-31T19:44:13.059 INFO:tasks.workunit.client.0.smithi167.stdout:7/672: readlink d8/dc/d18/d24/dbb/l7c 0 2022-01-31T19:44:13.060 INFO:tasks.workunit.client.0.smithi167.stdout:7/673: mkdir d8/d2c/d43/df3 0 2022-01-31T19:44:13.063 INFO:tasks.workunit.client.0.smithi167.stdout:5/639: dread d5/d6/d16/d18/d43/d45/f83 [0,4194304] 0 2022-01-31T19:44:13.079 INFO:tasks.workunit.client.0.smithi167.stdout:9/438: dwrite dd/d6b/d72/f8a [0,4194304] 0 2022-01-31T19:44:13.080 INFO:tasks.workunit.client.0.smithi167.stdout:7/674: dwrite d8/d2c/f25 [4194304,4194304] 0 2022-01-31T19:44:13.081 INFO:tasks.workunit.client.0.smithi167.stdout:9/439: mkdir dd/d35/d8f 0 2022-01-31T19:44:13.081 INFO:tasks.workunit.client.0.smithi167.stdout:9/440: stat dd/d19/d24/l26 0 2022-01-31T19:44:13.082 INFO:tasks.workunit.client.0.smithi167.stdout:7/675: chown d8/d2c/c1f 172476 1 2022-01-31T19:44:13.082 INFO:tasks.workunit.client.0.smithi167.stdout:7/676: symlink d8/d35/lf4 0 2022-01-31T19:44:13.083 INFO:tasks.workunit.client.0.smithi167.stdout:9/441: dread dd/f84 [0,4194304] 0 2022-01-31T19:44:13.083 INFO:tasks.workunit.client.0.smithi167.stdout:9/442: write dd/d19/d24/d50/d5d/f78 [1046745,38195] 0 2022-01-31T19:44:13.084 INFO:tasks.workunit.client.0.smithi167.stdout:7/677: write d8/d2c/d6e/d4d/d5e/d71/db3/fb8 [1962458,24703] 0 2022-01-31T19:44:13.084 INFO:tasks.workunit.client.0.smithi167.stdout:9/443: dread f7 [4194304,4194304] 0 2022-01-31T19:44:13.085 INFO:tasks.workunit.client.0.smithi167.stdout:9/444: write dd/d17/d80/d4c/f5f [1044476,47869] 0 2022-01-31T19:44:13.086 INFO:tasks.workunit.client.0.smithi167.stdout:7/678: getdents d8/dc 0 2022-01-31T19:44:13.087 INFO:tasks.workunit.client.0.smithi167.stdout:9/445: symlink dd/d19/l90 0 2022-01-31T19:44:13.088 INFO:tasks.workunit.client.0.smithi167.stdout:7/679: truncate d8/fa7 116694 0 2022-01-31T19:44:13.089 INFO:tasks.workunit.client.0.smithi167.stdout:7/680: dread - d8/d2c/fda zero size 2022-01-31T19:44:13.089 INFO:tasks.workunit.client.0.smithi167.stdout:7/681: truncate d8/d2c/fda 579347 0 2022-01-31T19:44:13.090 INFO:tasks.workunit.client.0.smithi167.stdout:9/446: creat dd/d17/d80/d4c/f91 x:0 0 0 2022-01-31T19:44:13.090 INFO:tasks.workunit.client.0.smithi167.stdout:9/447: chown dd/c8b 18 1 2022-01-31T19:44:13.091 INFO:tasks.workunit.client.0.smithi167.stdout:9/448: creat dd/d34/f92 x:0 0 0 2022-01-31T19:44:13.091 INFO:tasks.workunit.client.0.smithi167.stdout:9/449: readlink l0 0 2022-01-31T19:44:13.091 INFO:tasks.workunit.client.0.smithi167.stdout:9/450: creat dd/d6b/f93 x:0 0 0 2022-01-31T19:44:13.092 INFO:tasks.workunit.client.0.smithi167.stdout:7/682: write d8/d2c/d43/f2f [154181,47862] 0 2022-01-31T19:44:13.092 INFO:tasks.workunit.client.0.smithi167.stdout:9/451: read dd/d19/f21 [880787,125736] 0 2022-01-31T19:44:13.094 INFO:tasks.workunit.client.0.smithi167.stdout:7/683: creat d8/d2c/d6e/d4d/d67/ff5 x:0 0 0 2022-01-31T19:44:13.097 INFO:tasks.workunit.client.0.smithi167.stdout:7/684: dread d8/d35/f36 [0,4194304] 0 2022-01-31T19:44:13.097 INFO:tasks.workunit.client.0.smithi167.stdout:7/685: fsync f6 0 2022-01-31T19:44:13.097 INFO:tasks.workunit.client.0.smithi167.stdout:7/686: read d8/d2c/d6e/d4d/d67/f7e [1848362,101441] 0 2022-01-31T19:44:13.099 INFO:tasks.workunit.client.0.smithi167.stdout:7/687: rename d8/d2c/d6e/d9a/fd4 to d8/d2c/d43/df3/ff6 0 2022-01-31T19:44:13.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: audit 2022-01-31T19:44:11.151267+0000 mgr.smithi171.asyxnc (mgr.24387) 16 : audit 2022-01-31T19:44:13.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:13.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: audit 2022-01-31T19:44:11.531135+0000 mgr.smithi171.asyxnc (mgr.24387) 17 : audit [DBG] from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:13.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: audit 2022-01-31T19:44:12.097584+0000 mon.smithi167 ( 2022-01-31T19:44:13.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: mon.0) 661 : audit [DBG] from='client.? 172.21.15.167:0/847153096' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:44:13.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:13.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: 01-31T19:44:12.552399+0000 mon.smithi167 (mon.0) 662 : audit [DBG] 2022-01-31T19:44:13.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:12 smithi171 conmon[35325]: from='client.? 172.21.15.167:0/1317149099' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:44:13.176 INFO:tasks.workunit.client.0.smithi167.stdout:6/409: sync 2022-01-31T19:44:13.176 INFO:tasks.workunit.client.0.smithi167.stdout:6/410: read d8/f1a [3429164,90110] 0 2022-01-31T19:44:13.177 INFO:tasks.workunit.client.0.smithi167.stdout:6/411: write d8/d20/f21 [4956798,34725] 0 2022-01-31T19:44:13.190 INFO:tasks.workunit.client.0.smithi167.stdout:6/412: dwrite d8/d33/d35/f4d [0,4194304] 0 2022-01-31T19:44:13.191 INFO:tasks.workunit.client.0.smithi167.stdout:6/413: write d8/d20/f26 [2140569,57998] 0 2022-01-31T19:44:13.192 INFO:tasks.workunit.client.0.smithi167.stdout:6/414: dread d8/d33/f82 [0,4194304] 0 2022-01-31T19:44:13.201 INFO:tasks.workunit.client.0.smithi167.stdout:6/415: dread d8/f1c [0,4194304] 0 2022-01-31T19:44:13.202 INFO:tasks.workunit.client.0.smithi167.stdout:6/416: write d8/d33/f40 [4598271,1359] 0 2022-01-31T19:44:13.202 INFO:tasks.workunit.client.0.smithi167.stdout:6/417: mknod d8/d33/d4f/d87/c9c 0 2022-01-31T19:44:13.203 INFO:tasks.workunit.client.0.smithi167.stdout:6/418: link d8/f1c d8/d33/d51/d6a/f9d 0 2022-01-31T19:44:13.217 INFO:tasks.workunit.client.0.smithi167.stdout:6/419: dwrite d8/f5b [0,4194304] 0 2022-01-31T19:44:13.218 INFO:tasks.workunit.client.0.smithi167.stdout:6/420: read f3 [898094,89627] 0 2022-01-31T19:44:13.225 INFO:tasks.workunit.client.0.smithi167.stdout:6/421: dread d8/d62/f6f [0,4194304] 0 2022-01-31T19:44:13.226 INFO:tasks.workunit.client.0.smithi167.stdout:6/422: write d8/d33/d51/d5d/d86/d94/f97 [397235,106447] 0 2022-01-31T19:44:13.226 INFO:tasks.workunit.client.0.smithi167.stdout:6/423: creat d8/d33/d51/d5d/d86/d94/f9e x:0 0 0 2022-01-31T19:44:13.229 INFO:tasks.workunit.client.0.smithi167.stdout:6/424: unlink d8/d20/f21 0 2022-01-31T19:44:13.230 INFO:tasks.workunit.client.0.smithi167.stdout:6/425: creat d8/d33/d51/d5d/f9f x:0 0 0 2022-01-31T19:44:13.237 INFO:tasks.workunit.client.0.smithi167.stdout:6/426: dread d8/f5b [0,4194304] 0 2022-01-31T19:44:13.238 INFO:tasks.workunit.client.0.smithi167.stdout:6/427: symlink d8/d62/d92/la0 0 2022-01-31T19:44:13.239 INFO:tasks.workunit.client.0.smithi167.stdout:6/428: truncate d8/d33/d4f/f53 351081 0 2022-01-31T19:44:13.239 INFO:tasks.workunit.client.0.smithi167.stdout:6/429: write d8/d33/f40 [1225096,53257] 0 2022-01-31T19:44:13.240 INFO:tasks.workunit.client.0.smithi167.stdout:6/430: getdents d8/d62 0 2022-01-31T19:44:13.240 INFO:tasks.workunit.client.0.smithi167.stdout:6/431: readlink d8/l38 0 2022-01-31T19:44:13.242 INFO:tasks.workunit.client.0.smithi167.stdout:6/432: unlink d8/l13 0 2022-01-31T19:44:13.243 INFO:tasks.workunit.client.0.smithi167.stdout:6/433: creat d8/d62/d92/fa1 x:0 0 0 2022-01-31T19:44:13.243 INFO:tasks.workunit.client.0.smithi167.stdout:6/434: stat d8/d33/d51/d5d/f80 0 2022-01-31T19:44:13.244 INFO:tasks.workunit.client.0.smithi167.stdout:6/435: readlink d8/l12 0 2022-01-31T19:44:13.244 INFO:tasks.workunit.client.0.smithi167.stdout:6/436: rename d8/d33 to d8/d33/d51/d5d/da2 22 2022-01-31T19:44:13.245 INFO:tasks.workunit.client.0.smithi167.stdout:6/437: rename d8/f5c to d8/d33/d51/d6a/fa3 0 2022-01-31T19:44:13.259 INFO:tasks.workunit.client.0.smithi167.stdout:6/438: dwrite d8/f73 [0,4194304] 0 2022-01-31T19:44:13.259 INFO:tasks.workunit.client.0.smithi167.stdout:6/439: write d8/f1f [4523592,32228] 0 2022-01-31T19:44:13.261 INFO:tasks.workunit.client.0.smithi167.stdout:6/440: mkdir d8/d33/d51/d5d/d64/da4 0 2022-01-31T19:44:13.261 INFO:tasks.workunit.client.0.smithi167.stdout:6/441: rmdir d8/d33/d51 39 2022-01-31T19:44:13.262 INFO:tasks.workunit.client.0.smithi167.stdout:6/442: truncate d8/d33/d35/f61 88321 0 2022-01-31T19:44:13.263 INFO:tasks.workunit.client.0.smithi167.stdout:6/443: fdatasync d8/d33/d51/d7e/f7f 0 2022-01-31T19:44:13.263 INFO:tasks.workunit.client.0.smithi167.stdout:6/444: creat d8/d33/d51/d72/fa5 x:0 0 0 2022-01-31T19:44:13.263 INFO:tasks.workunit.client.0.smithi167.stdout:6/445: fdatasync d8/d33/f3c 0 2022-01-31T19:44:13.263 INFO:tasks.workunit.client.0.smithi167.stdout:6/446: chown d8/d33/c7d 124602 1 2022-01-31T19:44:13.273 INFO:tasks.workunit.client.0.smithi167.stdout:6/447: dread d8/d33/f3c [0,4194304] 0 2022-01-31T19:44:13.282 INFO:tasks.workunit.client.0.smithi167.stdout:1/622: sync 2022-01-31T19:44:13.283 INFO:tasks.workunit.client.0.smithi167.stdout:4/493: sync 2022-01-31T19:44:13.283 INFO:tasks.workunit.client.0.smithi167.stdout:2/396: sync 2022-01-31T19:44:13.283 INFO:tasks.workunit.client.0.smithi167.stdout:3/459: sync 2022-01-31T19:44:13.283 INFO:tasks.workunit.client.0.smithi167.stdout:2/397: chown f8 13 1 2022-01-31T19:44:13.284 INFO:tasks.workunit.client.0.smithi167.stdout:3/460: fdatasync d9/f79 0 2022-01-31T19:44:13.284 INFO:tasks.workunit.client.0.smithi167.stdout:8/573: sync 2022-01-31T19:44:13.284 INFO:tasks.workunit.client.0.smithi167.stdout:2/398: creat dd/df/f79 x:0 0 0 2022-01-31T19:44:13.284 INFO:tasks.workunit.client.0.smithi167.stdout:1/623: creat d0/d10/d5a/fd7 x:0 0 0 2022-01-31T19:44:13.285 INFO:tasks.workunit.client.0.smithi167.stdout:2/399: write f8 [4247331,78981] 0 2022-01-31T19:44:13.285 INFO:tasks.workunit.client.0.smithi167.stdout:0/277: sync 2022-01-31T19:44:13.285 INFO:tasks.workunit.client.0.smithi167.stdout:5/640: sync 2022-01-31T19:44:13.285 INFO:tasks.workunit.client.0.smithi167.stdout:5/641: fsync d5/f44 0 2022-01-31T19:44:13.286 INFO:tasks.workunit.client.0.smithi167.stdout:8/574: dread d6/db/d36/f84 [0,4194304] 0 2022-01-31T19:44:13.287 INFO:tasks.workunit.client.0.smithi167.stdout:1/624: dread d0/d75/d7a/f59 [0,4194304] 0 2022-01-31T19:44:13.287 INFO:tasks.workunit.client.0.smithi167.stdout:1/625: creat d0/d11/fd8 x:0 0 0 2022-01-31T19:44:13.287 INFO:tasks.workunit.client.0.smithi167.stdout:2/400: truncate dd/d18/f4e 1075663 0 2022-01-31T19:44:13.287 INFO:tasks.workunit.client.0.smithi167.stdout:5/642: dread d5/f44 [0,4194304] 0 2022-01-31T19:44:13.288 INFO:tasks.workunit.client.0.smithi167.stdout:5/643: creat d5/d6/d16/d18/d5b/d63/fc8 x:0 0 0 2022-01-31T19:44:13.288 INFO:tasks.workunit.client.0.smithi167.stdout:4/494: mknod d0/d6/d7/d26/d40/ca1 0 2022-01-31T19:44:13.289 INFO:tasks.workunit.client.0.smithi167.stdout:1/626: rename d0/d11/d46/c9a to d0/d62/d7f/d82/cd9 0 2022-01-31T19:44:13.290 INFO:tasks.workunit.client.0.smithi167.stdout:1/627: dread - d0/f73 zero size 2022-01-31T19:44:13.290 INFO:tasks.workunit.client.0.smithi167.stdout:1/628: truncate d0/d75/d7a/f59 2043654 0 2022-01-31T19:44:13.290 INFO:tasks.workunit.client.0.smithi167.stdout:2/401: stat dd/df/d1b/c45 0 2022-01-31T19:44:13.291 INFO:tasks.workunit.client.0.smithi167.stdout:4/495: read d0/d3f/f2e [561480,102433] 0 2022-01-31T19:44:13.292 INFO:tasks.workunit.client.0.smithi167.stdout:2/402: symlink dd/df/d3f/d51/l7a 0 2022-01-31T19:44:13.294 INFO:tasks.workunit.client.0.smithi167.stdout:2/403: mknod dd/df/d1b/d3d/d4a/d4b/d54/c7b 0 2022-01-31T19:44:13.294 INFO:tasks.workunit.client.0.smithi167.stdout:2/404: write dd/df/d22/f77 [4725240,116523] 0 2022-01-31T19:44:13.295 INFO:tasks.workunit.client.0.smithi167.stdout:6/448: dwrite d8/d20/d54/f63 [0,4194304] 0 2022-01-31T19:44:13.295 INFO:tasks.workunit.client.0.smithi167.stdout:3/461: dread d9/f42 [0,4194304] 0 2022-01-31T19:44:13.296 INFO:tasks.workunit.client.0.smithi167.stdout:3/462: fsync d9/d26/f55 0 2022-01-31T19:44:13.305 INFO:tasks.workunit.client.0.smithi167.stdout:8/575: dread d6/f8 [0,4194304] 0 2022-01-31T19:44:13.306 INFO:tasks.workunit.client.0.smithi167.stdout:5/644: dread d5/d6/d16/d18/f58 [0,4194304] 0 2022-01-31T19:44:13.306 INFO:tasks.workunit.client.0.smithi167.stdout:6/449: getdents d8/d33/d51/d5d 0 2022-01-31T19:44:13.310 INFO:tasks.workunit.client.0.smithi167.stdout:3/463: rmdir d9/d21 39 2022-01-31T19:44:13.310 INFO:tasks.workunit.client.0.smithi167.stdout:3/464: fsync d9/f42 0 2022-01-31T19:44:13.311 INFO:tasks.workunit.client.0.smithi167.stdout:6/450: creat d8/d20/fa6 x:0 0 0 2022-01-31T19:44:13.311 INFO:tasks.workunit.client.0.smithi167.stdout:6/451: creat d8/d20/d54/fa7 x:0 0 0 2022-01-31T19:44:13.311 INFO:tasks.workunit.client.0.smithi167.stdout:6/452: readlink d8/d33/d51/d5d/d86/l89 0 2022-01-31T19:44:13.312 INFO:tasks.workunit.client.0.smithi167.stdout:5/645: read d5/d6/d16/d18/d43/d45/d69/dc4/f79 [154890,14265] 0 2022-01-31T19:44:13.312 INFO:tasks.workunit.client.0.smithi167.stdout:5/646: chown d5/d6/de/d22/d4d/l6e 192 1 2022-01-31T19:44:13.315 INFO:tasks.workunit.client.0.smithi167.stdout:0/278: dwrite d17/d33/f36 [0,4194304] 0 2022-01-31T19:44:13.317 INFO:tasks.workunit.client.0.smithi167.stdout:5/647: rename d5/d6/cd to d5/d6/d16/d18/d5b/d9f/cc9 0 2022-01-31T19:44:13.318 INFO:tasks.workunit.client.0.smithi167.stdout:6/453: mknod d8/ca8 0 2022-01-31T19:44:13.318 INFO:tasks.workunit.client.0.smithi167.stdout:6/454: chown d8/d33/d51/d6a/fa3 6522377 1 2022-01-31T19:44:13.319 INFO:tasks.workunit.client.0.smithi167.stdout:6/455: truncate d8/d33/f32 4677016 0 2022-01-31T19:44:13.321 INFO:tasks.workunit.client.0.smithi167.stdout:0/279: unlink d17/d2c/l50 0 2022-01-31T19:44:13.323 INFO:tasks.workunit.client.0.smithi167.stdout:6/456: unlink d8/d33/f4e 0 2022-01-31T19:44:13.325 INFO:tasks.workunit.client.0.smithi167.stdout:0/280: symlink d17/d19/d1c/d39/d44/d49/l53 0 2022-01-31T19:44:13.326 INFO:tasks.workunit.client.0.smithi167.stdout:5/648: dread d5/d6/d16/d18/f58 [0,4194304] 0 2022-01-31T19:44:13.326 INFO:tasks.workunit.client.0.smithi167.stdout:0/281: fsync d17/d2c/f3c 0 2022-01-31T19:44:13.326 INFO:tasks.workunit.client.0.smithi167.stdout:0/282: chown f10 9493587 1 2022-01-31T19:44:13.326 INFO:tasks.workunit.client.0.smithi167.stdout:0/283: chown d17/d2c/f37 86 1 2022-01-31T19:44:13.329 INFO:tasks.workunit.client.0.smithi167.stdout:0/284: symlink d17/d19/d1c/d39/d44/l54 0 2022-01-31T19:44:13.330 INFO:tasks.workunit.client.0.smithi167.stdout:6/457: rmdir d8/d33/d4f 39 2022-01-31T19:44:13.333 INFO:tasks.workunit.client.0.smithi167.stdout:6/458: symlink d8/d33/d4f/la9 0 2022-01-31T19:44:13.334 INFO:tasks.workunit.client.0.smithi167.stdout:6/459: chown d8/d20/fa6 8626 1 2022-01-31T19:44:13.335 INFO:tasks.workunit.client.0.smithi167.stdout:4/496: dwrite d0/d6/d7/d39/f36 [0,4194304] 0 2022-01-31T19:44:13.336 INFO:tasks.workunit.client.0.smithi167.stdout:4/497: truncate d0/d6/d7/f4a 3861487 0 2022-01-31T19:44:13.337 INFO:tasks.workunit.client.0.smithi167.stdout:1/629: dwrite d0/f93 [0,4194304] 0 2022-01-31T19:44:13.337 INFO:tasks.workunit.client.0.smithi167.stdout:2/405: dwrite fa [0,4194304] 0 2022-01-31T19:44:13.337 INFO:tasks.workunit.client.0.smithi167.stdout:1/630: write d0/f73 [843072,14327] 0 2022-01-31T19:44:13.338 INFO:tasks.workunit.client.0.smithi167.stdout:4/498: rmdir d0/d6/d7/d26/d40/d4c 39 2022-01-31T19:44:13.340 INFO:tasks.workunit.client.0.smithi167.stdout:2/406: truncate dd/df/d1b/f56 1406094 0 2022-01-31T19:44:13.340 INFO:tasks.workunit.client.0.smithi167.stdout:2/407: write dd/df/d1b/f6a [1205070,52942] 0 2022-01-31T19:44:13.340 INFO:tasks.workunit.client.0.smithi167.stdout:2/408: chown dd/f29 14472412 1 2022-01-31T19:44:13.340 INFO:tasks.workunit.client.0.smithi167.stdout:4/499: write d0/d6/d7/d26/d40/f24 [1409437,724] 0 2022-01-31T19:44:13.341 INFO:tasks.workunit.client.0.smithi167.stdout:1/631: link d0/d10/d5a/c44 d0/d10/d5a/d33/d4b/da5/dc1/cda 0 2022-01-31T19:44:13.342 INFO:tasks.workunit.client.0.smithi167.stdout:0/285: dread d17/f22 [0,4194304] 0 2022-01-31T19:44:13.342 INFO:tasks.workunit.client.0.smithi167.stdout:2/409: mknod dd/df/d1b/d2f/c7c 0 2022-01-31T19:44:13.343 INFO:tasks.workunit.client.0.smithi167.stdout:4/500: mkdir d0/d3f/da2 0 2022-01-31T19:44:13.343 INFO:tasks.workunit.client.0.smithi167.stdout:4/501: creat d0/d6/d7/d89/fa3 x:0 0 0 2022-01-31T19:44:13.344 INFO:tasks.workunit.client.0.smithi167.stdout:8/576: dwrite d6/d1e/fac [0,4194304] 0 2022-01-31T19:44:13.344 INFO:tasks.workunit.client.0.smithi167.stdout:0/286: creat d17/d19/d1c/d39/f55 x:0 0 0 2022-01-31T19:44:13.345 INFO:tasks.workunit.client.0.smithi167.stdout:2/410: mknod dd/c7d 0 2022-01-31T19:44:13.345 INFO:tasks.workunit.client.0.smithi167.stdout:3/465: dwrite d9/d29/f2b [0,4194304] 0 2022-01-31T19:44:13.351 INFO:tasks.workunit.client.0.smithi167.stdout:8/577: creat d6/d1e/d2d/fbe x:0 0 0 2022-01-31T19:44:13.351 INFO:tasks.workunit.client.0.smithi167.stdout:8/578: chown d6/db/d6a/d8d/d9e/d60/f79 1282 1 2022-01-31T19:44:13.351 INFO:tasks.workunit.client.0.smithi167.stdout:8/579: fsync d6/db/d28/d66/f8c 0 2022-01-31T19:44:13.353 INFO:tasks.workunit.client.0.smithi167.stdout:4/502: read d0/d3f/f3d [1026053,52800] 0 2022-01-31T19:44:13.354 INFO:tasks.workunit.client.0.smithi167.stdout:0/287: creat d17/d19/d1c/d39/d44/f56 x:0 0 0 2022-01-31T19:44:13.359 INFO:tasks.workunit.client.0.smithi167.stdout:5/649: dwrite d5/d6/d16/d18/d43/d45/fc6 [0,4194304] 0 2022-01-31T19:44:13.360 INFO:tasks.workunit.client.0.smithi167.stdout:3/466: dread d9/fc [0,4194304] 0 2022-01-31T19:44:13.363 INFO:tasks.workunit.client.0.smithi167.stdout:6/460: dwrite d8/d62/f77 [0,4194304] 0 2022-01-31T19:44:13.363 INFO:tasks.workunit.client.0.smithi167.stdout:6/461: chown d8/d62/f77 0 1 2022-01-31T19:44:13.363 INFO:tasks.workunit.client.0.smithi167.stdout:1/632: dwrite d0/f93 [4194304,4194304] 0 2022-01-31T19:44:13.364 INFO:tasks.workunit.client.0.smithi167.stdout:2/411: symlink dd/d18/l7e 0 2022-01-31T19:44:13.364 INFO:tasks.workunit.client.0.smithi167.stdout:0/288: mknod d17/d19/d1c/d39/d44/d49/c57 0 2022-01-31T19:44:13.366 INFO:tasks.workunit.client.0.smithi167.stdout:5/650: mkdir d5/d6/de/d22/d6d/da0/da4/dca 0 2022-01-31T19:44:13.368 INFO:tasks.workunit.client.0.smithi167.stdout:3/467: creat d9/d7f/f85 x:0 0 0 2022-01-31T19:44:13.370 INFO:tasks.workunit.client.0.smithi167.stdout:1/633: mkdir d0/d75/db4/ddb 0 2022-01-31T19:44:13.372 INFO:tasks.workunit.client.0.smithi167.stdout:0/289: creat d17/d33/f58 x:0 0 0 2022-01-31T19:44:13.374 INFO:tasks.workunit.client.0.smithi167.stdout:8/580: dwrite d6/f57 [0,4194304] 0 2022-01-31T19:44:13.374 INFO:tasks.workunit.client.0.smithi167.stdout:3/468: rename d9/d29/d3b/d41/c4d to d9/c86 0 2022-01-31T19:44:13.375 INFO:tasks.workunit.client.0.smithi167.stdout:6/462: getdents d8/d33/d51/d5d/d86/d94 0 2022-01-31T19:44:13.377 INFO:tasks.workunit.client.0.smithi167.stdout:1/634: truncate d0/f30 3768386 0 2022-01-31T19:44:13.377 INFO:tasks.workunit.client.0.smithi167.stdout:1/635: write d0/f72 [4387884,56847] 0 2022-01-31T19:44:13.378 INFO:tasks.workunit.client.0.smithi167.stdout:1/636: creat d0/d10/d5a/d33/d4b/d1f/fdc x:0 0 0 2022-01-31T19:44:13.378 INFO:tasks.workunit.client.0.smithi167.stdout:2/412: getdents dd/df/d22 0 2022-01-31T19:44:13.380 INFO:tasks.workunit.client.0.smithi167.stdout:0/290: symlink d17/d2c/l59 0 2022-01-31T19:44:13.380 INFO:tasks.workunit.client.0.smithi167.stdout:0/291: chown fe 69453 1 2022-01-31T19:44:13.381 INFO:tasks.workunit.client.0.smithi167.stdout:4/503: dwrite d0/d6/d7/d39/f43 [0,4194304] 0 2022-01-31T19:44:13.381 INFO:tasks.workunit.client.0.smithi167.stdout:4/504: truncate d0/d6/d7/d26/d40/d4c/f9c 463674 0 2022-01-31T19:44:13.381 INFO:tasks.workunit.client.0.smithi167.stdout:4/505: truncate d0/d6/d7/d26/d31/f11 9767415 0 2022-01-31T19:44:13.381 INFO:tasks.workunit.client.0.smithi167.stdout:8/581: mkdir d6/db/d2f/dbf 0 2022-01-31T19:44:13.382 INFO:tasks.workunit.client.0.smithi167.stdout:2/413: write dd/df/d1b/f56 [98990,105287] 0 2022-01-31T19:44:13.386 INFO:tasks.workunit.client.0.smithi167.stdout:1/637: creat d0/d62/d7f/fdd x:0 0 0 2022-01-31T19:44:13.386 INFO:tasks.workunit.client.0.smithi167.stdout:5/651: dread d5/d6/d16/d18/d43/d45/d69/dc4/f7c [4194304,4194304] 0 2022-01-31T19:44:13.387 INFO:tasks.workunit.client.0.smithi167.stdout:0/292: getdents d17/d19/d1c/d39/d44/d49 0 2022-01-31T19:44:13.387 INFO:tasks.workunit.client.0.smithi167.stdout:0/293: stat d17/d33/f36 0 2022-01-31T19:44:13.399 INFO:tasks.workunit.client.0.smithi167.stdout:4/506: link d0/d6/d7/d26/d31/l10 d0/d6/d7/d39/d3a/la4 0 2022-01-31T19:44:13.399 INFO:tasks.workunit.client.0.smithi167.stdout:4/507: read - d0/d6/d7/d26/d31/f5e zero size 2022-01-31T19:44:13.399 INFO:tasks.workunit.client.0.smithi167.stdout:4/508: chown d0/d6/d7/d26/d40/f64 27627302 1 2022-01-31T19:44:13.400 INFO:tasks.workunit.client.0.smithi167.stdout:4/509: chown d0/d6/d7/d39/f2b 130491790 1 2022-01-31T19:44:13.400 INFO:tasks.workunit.client.0.smithi167.stdout:2/414: creat dd/df/d1b/d3d/d4a/d4b/f7f x:0 0 0 2022-01-31T19:44:13.401 INFO:tasks.workunit.client.0.smithi167.stdout:3/469: rmdir d9/d56 39 2022-01-31T19:44:13.404 INFO:tasks.workunit.client.0.smithi167.stdout:1/638: mkdir d0/d10/d5a/d33/d4b/da5/dde 0 2022-01-31T19:44:13.405 INFO:tasks.workunit.client.0.smithi167.stdout:5/652: getdents d5/d6/de/d22/d4d 0 2022-01-31T19:44:13.408 INFO:tasks.workunit.client.0.smithi167.stdout:0/294: mkdir d17/d19/d5a 0 2022-01-31T19:44:13.409 INFO:tasks.workunit.client.0.smithi167.stdout:7/688: sync 2022-01-31T19:44:13.409 INFO:tasks.workunit.client.0.smithi167.stdout:9/452: sync 2022-01-31T19:44:13.412 INFO:tasks.workunit.client.0.smithi167.stdout:4/510: creat d0/d5d/fa5 x:0 0 0 2022-01-31T19:44:13.414 INFO:tasks.workunit.client.0.smithi167.stdout:2/415: creat dd/df/d3f/d51/f80 x:0 0 0 2022-01-31T19:44:13.415 INFO:tasks.workunit.client.0.smithi167.stdout:1/639: truncate d0/d10/f42 1040675 0 2022-01-31T19:44:13.415 INFO:tasks.workunit.client.0.smithi167.stdout:1/640: creat d0/d10/d5a/d33/d4b/d1f/fdf x:0 0 0 2022-01-31T19:44:13.416 INFO:tasks.workunit.client.0.smithi167.stdout:5/653: creat d5/d6/de/d22/d6d/fcb x:0 0 0 2022-01-31T19:44:13.417 INFO:tasks.workunit.client.0.smithi167.stdout:5/654: fdatasync d5/d6/d16/d18/f3d 0 2022-01-31T19:44:13.417 INFO:tasks.workunit.client.0.smithi167.stdout:3/470: dwrite d9/d26/f2a [0,4194304] 0 2022-01-31T19:44:13.417 INFO:tasks.workunit.client.0.smithi167.stdout:3/471: chown d9/c7d 3896797 1 2022-01-31T19:44:13.418 INFO:tasks.workunit.client.0.smithi167.stdout:0/295: symlink d17/l5b 0 2022-01-31T19:44:13.418 INFO:tasks.workunit.client.0.smithi167.stdout:7/689: creat d8/d2c/d6e/d4d/d5e/d71/ff7 x:0 0 0 2022-01-31T19:44:13.419 INFO:tasks.workunit.client.0.smithi167.stdout:7/690: write d8/d2c/d6e/d4d/d67/d8f/f96 [3916197,78622] 0 2022-01-31T19:44:13.419 INFO:tasks.workunit.client.0.smithi167.stdout:7/691: write d8/d2c/d7a/fcd [283829,117073] 0 2022-01-31T19:44:13.424 INFO:tasks.workunit.client.0.smithi167.stdout:9/453: creat dd/d19/d24/d56/d5b/f94 x:0 0 0 2022-01-31T19:44:13.424 INFO:tasks.workunit.client.0.smithi167.stdout:9/454: creat dd/d19/d24/d56/d5b/f95 x:0 0 0 2022-01-31T19:44:13.425 INFO:tasks.workunit.client.0.smithi167.stdout:9/455: write dd/d19/f2d [148846,88101] 0 2022-01-31T19:44:13.425 INFO:tasks.workunit.client.0.smithi167.stdout:9/456: stat dd/d6b/d72/f8a 0 2022-01-31T19:44:13.425 INFO:tasks.workunit.client.0.smithi167.stdout:9/457: fsync dd/d34/f92 0 2022-01-31T19:44:13.425 INFO:tasks.workunit.client.0.smithi167.stdout:9/458: stat dd/d17/f2f 0 2022-01-31T19:44:13.426 INFO:tasks.workunit.client.0.smithi167.stdout:9/459: dread - dd/d19/d24/d56/d5b/f95 zero size 2022-01-31T19:44:13.426 INFO:tasks.workunit.client.0.smithi167.stdout:2/416: mknod dd/df/d1b/d3d/d4a/c81 0 2022-01-31T19:44:13.429 INFO:tasks.workunit.client.0.smithi167.stdout:5/655: unlink d5/d6/d16/d18/d43/f57 0 2022-01-31T19:44:13.429 INFO:tasks.workunit.client.0.smithi167.stdout:5/656: chown d5/d6/de/d22/d6d/fb7 5065 1 2022-01-31T19:44:13.430 INFO:tasks.workunit.client.0.smithi167.stdout:0/296: rename d17/d33/f58 to d17/d33/f5c 0 2022-01-31T19:44:13.430 INFO:tasks.workunit.client.0.smithi167.stdout:0/297: chown fa 63 1 2022-01-31T19:44:13.431 INFO:tasks.workunit.client.0.smithi167.stdout:7/692: mkdir d8/d2c/d43/d94/daf/dee/df8 0 2022-01-31T19:44:13.434 INFO:tasks.workunit.client.0.smithi167.stdout:9/460: mkdir dd/d19/d24/d56/d96 0 2022-01-31T19:44:13.435 INFO:tasks.workunit.client.0.smithi167.stdout:4/511: rmdir d0/d6/d7/d89 39 2022-01-31T19:44:13.436 INFO:tasks.workunit.client.0.smithi167.stdout:2/417: rmdir dd/df/d1b/d2f/d36 39 2022-01-31T19:44:13.437 INFO:tasks.workunit.client.0.smithi167.stdout:2/418: read - dd/df/d1b/d2f/f66 zero size 2022-01-31T19:44:13.437 INFO:tasks.workunit.client.0.smithi167.stdout:2/419: readlink dd/d18/l1f 0 2022-01-31T19:44:13.437 INFO:tasks.workunit.client.0.smithi167.stdout:3/472: dwrite d9/d29/f52 [0,4194304] 0 2022-01-31T19:44:13.438 INFO:tasks.workunit.client.0.smithi167.stdout:3/473: fdatasync d9/d26/f2a 0 2022-01-31T19:44:13.439 INFO:tasks.workunit.client.0.smithi167.stdout:5/657: dread d5/f7 [4194304,4194304] 0 2022-01-31T19:44:13.439 INFO:tasks.workunit.client.0.smithi167.stdout:5/658: dread - d5/d6/d16/d18/fae zero size 2022-01-31T19:44:13.445 INFO:tasks.workunit.client.0.smithi167.stdout:1/641: rmdir d0/d10/d5a/d33 39 2022-01-31T19:44:13.450 INFO:tasks.workunit.client.0.smithi167.stdout:7/693: link d8/d2c/d43/f2f d8/d2c/d6e/ff9 0 2022-01-31T19:44:13.450 INFO:tasks.workunit.client.0.smithi167.stdout:4/512: symlink d0/d5d/la6 0 2022-01-31T19:44:13.452 INFO:tasks.workunit.client.0.smithi167.stdout:4/513: write d0/d6/d7/d26/d31/f19 [4302672,77149] 0 2022-01-31T19:44:13.454 INFO:tasks.workunit.client.0.smithi167.stdout:2/420: unlink dd/df/d3f/d51/l7a 0 2022-01-31T19:44:13.455 INFO:tasks.workunit.client.0.smithi167.stdout:1/642: write d0/d10/d5a/d33/d4b/fa2 [44381,94354] 0 2022-01-31T19:44:13.455 INFO:tasks.workunit.client.0.smithi167.stdout:1/643: chown d0/d10/d5a/d33/d4b/da5/da9/dad/lc3 28652 1 2022-01-31T19:44:13.456 INFO:tasks.workunit.client.0.smithi167.stdout:5/659: dread d5/d6/d16/d18/d43/d45/fc6 [0,4194304] 0 2022-01-31T19:44:13.456 INFO:tasks.workunit.client.0.smithi167.stdout:1/644: truncate d0/d10/d5a/d33/fa7 1870906 0 2022-01-31T19:44:13.456 INFO:tasks.workunit.client.0.smithi167.stdout:1/645: read - d0/d10/f19 zero size 2022-01-31T19:44:13.457 INFO:tasks.workunit.client.0.smithi167.stdout:4/514: write d0/d6/d7/d39/d3a/f54 [953030,45872] 0 2022-01-31T19:44:13.457 INFO:tasks.workunit.client.0.smithi167.stdout:4/515: truncate d0/d6/d7/d26/d31/f5e 814840 0 2022-01-31T19:44:13.457 INFO:tasks.workunit.client.0.smithi167.stdout:3/474: mknod d9/d29/c87 0 2022-01-31T19:44:13.458 INFO:tasks.workunit.client.0.smithi167.stdout:7/694: link d8/dc/c11 d8/d2c/d7a/d99/cfa 0 2022-01-31T19:44:13.461 INFO:tasks.workunit.client.0.smithi167.stdout:2/421: truncate dd/df/d22/f77 3242004 0 2022-01-31T19:44:13.462 INFO:tasks.workunit.client.0.smithi167.stdout:2/422: chown dd/d18/d62 17163 1 2022-01-31T19:44:13.462 INFO:tasks.workunit.client.0.smithi167.stdout:2/423: truncate dd/df/d1b/d3d/d4a/f6b 156320 0 2022-01-31T19:44:13.462 INFO:tasks.workunit.client.0.smithi167.stdout:2/424: chown dd/df/d1b/d3d/l59 14595355 1 2022-01-31T19:44:13.464 INFO:tasks.workunit.client.0.smithi167.stdout:0/298: dwrite d17/f2f [0,4194304] 0 2022-01-31T19:44:13.465 INFO:tasks.workunit.client.0.smithi167.stdout:4/516: symlink d0/d3f/d1a/la7 0 2022-01-31T19:44:13.465 INFO:tasks.workunit.client.0.smithi167.stdout:3/475: mknod d9/c88 0 2022-01-31T19:44:13.466 INFO:tasks.workunit.client.0.smithi167.stdout:3/476: chown d9/l74 3631976 1 2022-01-31T19:44:13.466 INFO:tasks.workunit.client.0.smithi167.stdout:3/477: fdatasync d9/d29/f33 0 2022-01-31T19:44:13.466 INFO:tasks.workunit.client.0.smithi167.stdout:3/478: truncate d9/f18 4041029 0 2022-01-31T19:44:13.466 INFO:tasks.workunit.client.0.smithi167.stdout:9/461: dwrite dd/f10 [0,4194304] 0 2022-01-31T19:44:13.467 INFO:tasks.workunit.client.0.smithi167.stdout:5/660: mkdir d5/d6/d16/d18/d5b/d63/d88/dbc/dcc 0 2022-01-31T19:44:13.467 INFO:tasks.workunit.client.0.smithi167.stdout:1/646: dread d0/f30 [0,4194304] 0 2022-01-31T19:44:13.472 INFO:tasks.workunit.client.0.smithi167.stdout:0/299: rename d17/d19/d1c/d39/c46 to d17/d33/c5d 0 2022-01-31T19:44:13.476 INFO:tasks.workunit.client.0.smithi167.stdout:3/479: unlink d9/d29/d3b/d41/d6b/c5f 0 2022-01-31T19:44:13.477 INFO:tasks.workunit.client.0.smithi167.stdout:9/462: mknod dd/d17/d80/d4c/c97 0 2022-01-31T19:44:13.478 INFO:tasks.workunit.client.0.smithi167.stdout:9/463: truncate dd/d17/d80/f54 426420 0 2022-01-31T19:44:13.478 INFO:tasks.workunit.client.0.smithi167.stdout:5/661: mkdir d5/dcd 0 2022-01-31T19:44:13.481 INFO:tasks.workunit.client.0.smithi167.stdout:7/695: mkdir d8/dfb 0 2022-01-31T19:44:13.482 INFO:tasks.workunit.client.0.smithi167.stdout:1/647: chown d0/d10/d5a/d33/d4b/c71 8855627 1 2022-01-31T19:44:13.482 INFO:tasks.workunit.client.0.smithi167.stdout:1/648: fdatasync d0/d10/d20/d7c/faa 0 2022-01-31T19:44:13.482 INFO:tasks.workunit.client.0.smithi167.stdout:1/649: truncate d0/d10/d5a/d33/d97/fd0 473074 0 2022-01-31T19:44:13.484 INFO:tasks.workunit.client.0.smithi167.stdout:4/517: dwrite d0/d6/d7/d26/d40/f32 [0,4194304] 0 2022-01-31T19:44:13.486 INFO:tasks.workunit.client.0.smithi167.stdout:0/300: creat d17/f5e x:0 0 0 2022-01-31T19:44:13.487 INFO:tasks.workunit.client.0.smithi167.stdout:0/301: creat d17/d3e/f5f x:0 0 0 2022-01-31T19:44:13.488 INFO:tasks.workunit.client.0.smithi167.stdout:9/464: link dd/d35/f44 dd/d19/d59/f98 0 2022-01-31T19:44:13.489 INFO:tasks.workunit.client.0.smithi167.stdout:5/662: readlink d5/d6/de/d22/d76/lac 0 2022-01-31T19:44:13.491 INFO:tasks.workunit.client.0.smithi167.stdout:7/696: symlink d8/d2c/d6e/d4d/d5e/d71/lfc 0 2022-01-31T19:44:13.493 INFO:tasks.workunit.client.0.smithi167.stdout:4/518: rmdir d0/d6/d7/d89 39 2022-01-31T19:44:13.495 INFO:tasks.workunit.client.0.smithi167.stdout:0/302: rmdir d17/d19/d5a 0 2022-01-31T19:44:13.496 INFO:tasks.workunit.client.0.smithi167.stdout:9/465: mknod dd/d19/d59/c99 0 2022-01-31T19:44:13.498 INFO:tasks.workunit.client.0.smithi167.stdout:3/480: dwrite d9/d56/f7c [0,4194304] 0 2022-01-31T19:44:13.498 INFO:tasks.workunit.client.0.smithi167.stdout:5/663: write d5/d6/de/d22/d4d/f85 [401499,126896] 0 2022-01-31T19:44:13.499 INFO:tasks.workunit.client.0.smithi167.stdout:5/664: write d5/d6/d3a/f17 [2407858,116409] 0 2022-01-31T19:44:13.504 INFO:tasks.workunit.client.0.smithi167.stdout:1/650: dwrite d0/d10/d5a/d33/d97/fd0 [0,4194304] 0 2022-01-31T19:44:13.504 INFO:tasks.workunit.client.0.smithi167.stdout:1/651: chown d0/d10/d5a/d33/d97/l9d 11 1 2022-01-31T19:44:13.505 INFO:tasks.workunit.client.0.smithi167.stdout:7/697: mknod d8/dc/cfd 0 2022-01-31T19:44:13.510 INFO:tasks.workunit.client.0.smithi167.stdout:0/303: rename d17/f43 to d17/d2c/f60 0 2022-01-31T19:44:13.510 INFO:tasks.workunit.client.0.smithi167.stdout:0/304: creat d17/d3e/f61 x:0 0 0 2022-01-31T19:44:13.511 INFO:tasks.workunit.client.0.smithi167.stdout:0/305: unlink fe 0 2022-01-31T19:44:13.512 INFO:tasks.workunit.client.0.smithi167.stdout:3/481: mkdir d9/d29/d3b/d41/d6b/d89 0 2022-01-31T19:44:13.515 INFO:tasks.workunit.client.0.smithi167.stdout:1/652: link d0/f30 d0/d10/d5a/d33/d4b/da5/fe0 0 2022-01-31T19:44:13.516 INFO:tasks.workunit.client.0.smithi167.stdout:7/698: link d8/d2c/d6e/d4d/d5e/d71/db1/d8d/fec d8/dab/dd0/ffe 0 2022-01-31T19:44:13.517 INFO:tasks.workunit.client.0.smithi167.stdout:4/519: getdents d0/d3f/d60/d65 0 2022-01-31T19:44:13.517 INFO:tasks.workunit.client.0.smithi167.stdout:4/520: stat d0/d6/d7/d26/d40/f32 0 2022-01-31T19:44:13.517 INFO:tasks.workunit.client.0.smithi167.stdout:4/521: write d0/d6/f76 [335276,48035] 0 2022-01-31T19:44:13.519 INFO:tasks.workunit.client.0.smithi167.stdout:4/522: read d0/d6/f58 [446373,16610] 0 2022-01-31T19:44:13.522 INFO:tasks.workunit.client.0.smithi167.stdout:3/482: symlink d9/d7f/l8a 0 2022-01-31T19:44:13.523 INFO:tasks.workunit.client.0.smithi167.stdout:1/653: symlink d0/d10/d5a/d33/d4b/d1f/le1 0 2022-01-31T19:44:13.524 INFO:tasks.workunit.client.0.smithi167.stdout:1/654: chown d0/d10/d18/f91 1 1 2022-01-31T19:44:13.524 INFO:tasks.workunit.client.0.smithi167.stdout:1/655: write d0/f1 [8891898,103860] 0 2022-01-31T19:44:13.524 INFO:tasks.workunit.client.0.smithi167.stdout:1/656: stat d0/d10/d5a/fd7 0 2022-01-31T19:44:13.525 INFO:tasks.workunit.client.0.smithi167.stdout:5/665: dwrite d5/d6/de/fbd [0,4194304] 0 2022-01-31T19:44:13.525 INFO:tasks.workunit.client.0.smithi167.stdout:1/657: stat d0/d10/d20/d7c 0 2022-01-31T19:44:13.525 INFO:tasks.workunit.client.0.smithi167.stdout:1/658: write d0/d10/d18/f91 [4824663,6895] 0 2022-01-31T19:44:13.527 INFO:tasks.workunit.client.0.smithi167.stdout:9/466: dwrite f7 [0,4194304] 0 2022-01-31T19:44:13.528 INFO:tasks.workunit.client.0.smithi167.stdout:1/659: dread d0/f25 [0,4194304] 0 2022-01-31T19:44:13.528 INFO:tasks.workunit.client.0.smithi167.stdout:1/660: creat d0/d62/d7f/d82/fe2 x:0 0 0 2022-01-31T19:44:13.529 INFO:tasks.workunit.client.0.smithi167.stdout:4/523: symlink d0/d6/d7/d39/d93/la8 0 2022-01-31T19:44:13.533 INFO:tasks.workunit.client.0.smithi167.stdout:3/483: unlink d9/d21/l53 0 2022-01-31T19:44:13.534 INFO:tasks.workunit.client.0.smithi167.stdout:3/484: fdatasync d9/d29/f33 0 2022-01-31T19:44:13.534 INFO:tasks.workunit.client.0.smithi167.stdout:3/485: write d9/d29/d3b/d41/d6b/f35 [4365294,53240] 0 2022-01-31T19:44:13.541 INFO:tasks.workunit.client.0.smithi167.stdout:0/306: rmdir d17/d3e 39 2022-01-31T19:44:13.542 INFO:tasks.workunit.client.0.smithi167.stdout:7/699: dwrite d8/d2c/d6e/d4d/d5e/fa9 [0,4194304] 0 2022-01-31T19:44:13.542 INFO:tasks.workunit.client.0.smithi167.stdout:5/666: dread - d5/d6/de/d22/d6d/fb7 zero size 2022-01-31T19:44:13.542 INFO:tasks.workunit.client.0.smithi167.stdout:2/425: truncate dd/f55 1494770 0 2022-01-31T19:44:13.545 INFO:tasks.workunit.client.0.smithi167.stdout:7/700: truncate d8/d2c/d6e/d4d/d6b/f81 483017 0 2022-01-31T19:44:13.546 INFO:tasks.workunit.client.0.smithi167.stdout:5/667: getdents d5 0 2022-01-31T19:44:13.547 INFO:tasks.workunit.client.0.smithi167.stdout:2/426: getdents dd/df/d1b/d3d 0 2022-01-31T19:44:13.547 INFO:tasks.workunit.client.0.smithi167.stdout:2/427: chown dd/df/d1b/l67 177338 1 2022-01-31T19:44:13.549 INFO:tasks.workunit.client.0.smithi167.stdout:5/668: rename d5/d6/d16/d18/d43/d45/d69/dc4/f79 to d5/d6/de/d22/d6d/da0/da4/fce 0 2022-01-31T19:44:13.550 INFO:tasks.workunit.client.0.smithi167.stdout:2/428: mknod dd/df/d22/c82 0 2022-01-31T19:44:13.550 INFO:tasks.workunit.client.0.smithi167.stdout:2/429: write dd/df/f1c [2451452,85235] 0 2022-01-31T19:44:13.552 INFO:tasks.workunit.client.0.smithi167.stdout:2/430: rename dd/d18/l1f to dd/l83 0 2022-01-31T19:44:13.555 INFO:tasks.workunit.client.0.smithi167.stdout:5/669: rename d5/d6/d16/d18/d5b/la8 to d5/d6/d16/d18/d5b/lcf 0 2022-01-31T19:44:13.561 INFO:tasks.workunit.client.0.smithi167.stdout:0/307: dread fc [0,4194304] 0 2022-01-31T19:44:13.565 INFO:tasks.workunit.client.0.smithi167.stdout:7/701: dread d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:13.566 INFO:tasks.workunit.client.0.smithi167.stdout:7/702: creat d8/d2c/d6e/d4d/d67/fff x:0 0 0 2022-01-31T19:44:13.572 INFO:tasks.workunit.client.0.smithi167.stdout:5/670: dread d5/d6/de/d22/d76/f80 [0,4194304] 0 2022-01-31T19:44:13.575 INFO:tasks.workunit.client.0.smithi167.stdout:5/671: write d5/d6/d16/d18/d43/d45/f60 [1227706,59378] 0 2022-01-31T19:44:13.576 INFO:tasks.workunit.client.0.smithi167.stdout:9/467: dwrite dd/d17/d80/f5e [0,4194304] 0 2022-01-31T19:44:13.577 INFO:tasks.workunit.client.0.smithi167.stdout:5/672: unlink d5/d6/d16/d18/d43/d45/lb1 0 2022-01-31T19:44:13.582 INFO:tasks.workunit.client.0.smithi167.stdout:9/468: rmdir dd/d17/d80 39 2022-01-31T19:44:13.583 INFO:tasks.workunit.client.0.smithi167.stdout:9/469: stat dd/d34/f5a 0 2022-01-31T19:44:13.586 INFO:tasks.workunit.client.0.smithi167.stdout:1/661: dwrite d0/d10/f42 [0,4194304] 0 2022-01-31T19:44:13.587 INFO:tasks.workunit.client.0.smithi167.stdout:5/673: rename d5/d6/d16/d18/d43/d45/c6b to d5/d6/de/d22/d6d/cd0 0 2022-01-31T19:44:13.588 INFO:tasks.workunit.client.0.smithi167.stdout:4/524: dwrite d0/d6/f76 [0,4194304] 0 2022-01-31T19:44:13.588 INFO:tasks.workunit.client.0.smithi167.stdout:0/308: dwrite d17/d2c/f60 [0,4194304] 0 2022-01-31T19:44:13.588 INFO:tasks.workunit.client.0.smithi167.stdout:1/662: rmdir d0/d10/d5a/d33/d4b/d1f 39 2022-01-31T19:44:13.589 INFO:tasks.workunit.client.0.smithi167.stdout:5/674: symlink d5/d6/de/d22/d6d/da0/da4/ld1 0 2022-01-31T19:44:13.589 INFO:tasks.workunit.client.0.smithi167.stdout:5/675: readlink d5/lb 0 2022-01-31T19:44:13.590 INFO:tasks.workunit.client.0.smithi167.stdout:5/676: stat d5/d6/d16/d18/d43/d45/f83 0 2022-01-31T19:44:13.590 INFO:tasks.workunit.client.0.smithi167.stdout:2/431: dwrite dd/df/d1b/f56 [0,4194304] 0 2022-01-31T19:44:13.595 INFO:tasks.workunit.client.0.smithi167.stdout:0/309: dread d17/f22 [0,4194304] 0 2022-01-31T19:44:13.595 INFO:tasks.workunit.client.0.smithi167.stdout:7/703: dwrite d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/ff0 [0,4194304] 0 2022-01-31T19:44:13.598 INFO:tasks.workunit.client.0.smithi167.stdout:2/432: dread dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:13.603 INFO:tasks.workunit.client.0.smithi167.stdout:1/663: symlink d0/db3/le3 0 2022-01-31T19:44:13.603 INFO:tasks.workunit.client.0.smithi167.stdout:1/664: stat d0/cf 0 2022-01-31T19:44:13.605 INFO:tasks.workunit.client.0.smithi167.stdout:7/704: write d8/d2c/d6e/d4d/d5e/fa9 [71376,61957] 0 2022-01-31T19:44:13.607 INFO:tasks.workunit.client.0.smithi167.stdout:0/310: dread d17/d2c/f3c [0,4194304] 0 2022-01-31T19:44:13.609 INFO:tasks.workunit.client.0.smithi167.stdout:1/665: dread d0/d62/d7f/d82/f88 [0,4194304] 0 2022-01-31T19:44:13.609 INFO:tasks.workunit.client.0.smithi167.stdout:1/666: chown d0/d10/d5a/d33/d4b/d1f/fdf 0 1 2022-01-31T19:44:13.609 INFO:tasks.workunit.client.0.smithi167.stdout:1/667: dread - d0/d10/d20/d7c/fd4 zero size 2022-01-31T19:44:13.624 INFO:tasks.workunit.client.0.smithi167.stdout:2/433: unlink dd/c7d 0 2022-01-31T19:44:13.625 INFO:tasks.workunit.client.0.smithi167.stdout:0/311: rmdir d17/d19/d1c/d39/d44/d49 39 2022-01-31T19:44:13.628 INFO:tasks.workunit.client.0.smithi167.stdout:5/677: dwrite d5/f44 [0,4194304] 0 2022-01-31T19:44:13.628 INFO:tasks.workunit.client.0.smithi167.stdout:5/678: fsync d5/d6/de/f4c 0 2022-01-31T19:44:13.629 INFO:tasks.workunit.client.0.smithi167.stdout:9/470: dwrite dd/d19/d24/d56/f75 [0,4194304] 0 2022-01-31T19:44:13.629 INFO:tasks.workunit.client.0.smithi167.stdout:9/471: fsync dd/d19/d24/d56/f86 0 2022-01-31T19:44:13.629 INFO:tasks.workunit.client.0.smithi167.stdout:1/668: link d0/d75/d7a/f74 d0/fe4 0 2022-01-31T19:44:13.629 INFO:tasks.workunit.client.0.smithi167.stdout:1/669: write d0/d10/d5a/d33/d4b/da5/fa0 [4854509,74753] 0 2022-01-31T19:44:13.629 INFO:tasks.workunit.client.0.smithi167.stdout:1/670: chown d0/d10/d5a/d33/d4b/da5/fe0 203606 1 2022-01-31T19:44:13.630 INFO:tasks.workunit.client.0.smithi167.stdout:7/705: dwrite d8/feb [0,4194304] 0 2022-01-31T19:44:13.630 INFO:tasks.workunit.client.0.smithi167.stdout:4/525: dwrite d0/d6/d7/d26/d40/f24 [0,4194304] 0 2022-01-31T19:44:13.631 INFO:tasks.workunit.client.0.smithi167.stdout:2/434: mknod dd/df/d1b/d2f/c84 0 2022-01-31T19:44:13.631 INFO:tasks.workunit.client.0.smithi167.stdout:2/435: write dd/f29 [1844537,28121] 0 2022-01-31T19:44:13.632 INFO:tasks.workunit.client.0.smithi167.stdout:0/312: symlink d17/d19/d1c/d39/d44/l62 0 2022-01-31T19:44:13.633 INFO:tasks.workunit.client.0.smithi167.stdout:1/671: dread d0/d10/d20/f37 [0,4194304] 0 2022-01-31T19:44:13.635 INFO:tasks.workunit.client.0.smithi167.stdout:5/679: link d5/d6/d16/d18/fae d5/d6/de/d22/d6d/da0/da4/fd2 0 2022-01-31T19:44:13.640 INFO:tasks.workunit.client.0.smithi167.stdout:9/472: rmdir dd/d19/d24/d56/d96 0 2022-01-31T19:44:13.641 INFO:tasks.workunit.client.0.smithi167.stdout:5/680: dread d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:13.641 INFO:tasks.workunit.client.0.smithi167.stdout:9/473: write dd/f84 [632241,88609] 0 2022-01-31T19:44:13.646 INFO:tasks.workunit.client.0.smithi167.stdout:7/706: mkdir d8/d2c/d6e/d4d/d5e/d71/db1/d8d/d100 0 2022-01-31T19:44:13.649 INFO:tasks.workunit.client.0.smithi167.stdout:0/313: unlink d17/d19/d1c/d39/f55 0 2022-01-31T19:44:13.650 INFO:tasks.workunit.client.0.smithi167.stdout:0/314: chown d17/d19/d1c/l23 6046 1 2022-01-31T19:44:13.653 INFO:tasks.workunit.client.0.smithi167.stdout:5/681: creat d5/d6/d16/d18/d5b/d63/d88/dbc/fd3 x:0 0 0 2022-01-31T19:44:13.654 INFO:tasks.workunit.client.0.smithi167.stdout:4/526: dwrite d0/d6/d7/d39/d3a/d79/f91 [0,4194304] 0 2022-01-31T19:44:13.654 INFO:tasks.workunit.client.0.smithi167.stdout:2/436: dwrite dd/f55 [0,4194304] 0 2022-01-31T19:44:13.655 INFO:tasks.workunit.client.0.smithi167.stdout:2/437: chown dd/df/d22 2258869 1 2022-01-31T19:44:13.655 INFO:tasks.workunit.client.0.smithi167.stdout:9/474: rename dd/d19/d24/f63 to dd/d35/f9a 0 2022-01-31T19:44:13.658 INFO:tasks.workunit.client.0.smithi167.stdout:0/315: symlink d17/d2c/l63 0 2022-01-31T19:44:13.660 INFO:tasks.workunit.client.0.smithi167.stdout:1/672: rmdir d0/d62/d7f/d82 39 2022-01-31T19:44:13.661 INFO:tasks.workunit.client.0.smithi167.stdout:5/682: getdents d5/d6/de/d22/d6d/da0/da4 0 2022-01-31T19:44:13.664 INFO:tasks.workunit.client.0.smithi167.stdout:5/683: read d5/f7 [1413479,4061] 0 2022-01-31T19:44:13.664 INFO:tasks.workunit.client.0.smithi167.stdout:5/684: truncate d5/d6/de/d22/d76/f80 4748017 0 2022-01-31T19:44:13.664 INFO:tasks.workunit.client.0.smithi167.stdout:5/685: write d5/d6/de/d22/d4d/fb8 [729092,63947] 0 2022-01-31T19:44:13.665 INFO:tasks.workunit.client.0.smithi167.stdout:5/686: dread - d5/d6/d16/d18/d5b/d63/d88/dbc/fd3 zero size 2022-01-31T19:44:13.665 INFO:tasks.workunit.client.0.smithi167.stdout:2/438: mkdir dd/df/d1b/d3d/d4a/d4b/d74/d85 0 2022-01-31T19:44:13.665 INFO:tasks.workunit.client.0.smithi167.stdout:2/439: write f8 [4227772,45232] 0 2022-01-31T19:44:13.665 INFO:tasks.workunit.client.0.smithi167.stdout:2/440: write dd/df/d3f/d51/d5f/f69 [1507330,103800] 0 2022-01-31T19:44:13.666 INFO:tasks.workunit.client.0.smithi167.stdout:7/707: dwrite d8/dc/d18/fca [0,4194304] 0 2022-01-31T19:44:13.667 INFO:tasks.workunit.client.0.smithi167.stdout:7/708: truncate d8/d2c/d6e/d4d/d5e/d71/ff7 638541 0 2022-01-31T19:44:13.667 INFO:tasks.workunit.client.0.smithi167.stdout:9/475: rename dd/d19/c71 to dd/d35/d8f/c9b 0 2022-01-31T19:44:13.667 INFO:tasks.workunit.client.0.smithi167.stdout:0/316: creat d17/d2c/f64 x:0 0 0 2022-01-31T19:44:13.667 INFO:tasks.workunit.client.0.smithi167.stdout:9/476: write dd/d17/d80/f54 [280030,86592] 0 2022-01-31T19:44:13.668 INFO:tasks.workunit.client.0.smithi167.stdout:9/477: write dd/f64 [1217636,44777] 0 2022-01-31T19:44:13.668 INFO:tasks.workunit.client.0.smithi167.stdout:0/317: write d17/d3e/f42 [1732907,43597] 0 2022-01-31T19:44:13.669 INFO:tasks.workunit.client.0.smithi167.stdout:5/687: truncate d5/d6/de/fbd 3702471 0 2022-01-31T19:44:13.669 INFO:tasks.workunit.client.0.smithi167.stdout:5/688: chown d5/d6/de/d22/d76/db0/fb6 24 1 2022-01-31T19:44:13.669 INFO:tasks.workunit.client.0.smithi167.stdout:4/527: write d0/d6/d7/d39/d3a/d79/f91 [4812655,120898] 0 2022-01-31T19:44:13.670 INFO:tasks.workunit.client.0.smithi167.stdout:2/441: rmdir dd/df/d1b/d2f/d36 39 2022-01-31T19:44:13.670 INFO:tasks.workunit.client.0.smithi167.stdout:2/442: creat dd/d18/d62/f86 x:0 0 0 2022-01-31T19:44:13.672 INFO:tasks.workunit.client.0.smithi167.stdout:9/478: rename dd/d6b/d72/l8c to dd/d19/d24/d56/d5b/l9c 0 2022-01-31T19:44:13.674 INFO:tasks.workunit.client.0.smithi167.stdout:2/443: symlink dd/d18/l87 0 2022-01-31T19:44:13.674 INFO:tasks.workunit.client.0.smithi167.stdout:2/444: write dd/f55 [4240509,87785] 0 2022-01-31T19:44:13.676 INFO:tasks.workunit.client.0.smithi167.stdout:9/479: creat dd/d17/d7f/f9d x:0 0 0 2022-01-31T19:44:13.676 INFO:tasks.workunit.client.0.smithi167.stdout:9/480: chown dd/d35/f8d 251 1 2022-01-31T19:44:13.678 INFO:tasks.workunit.client.0.smithi167.stdout:4/528: write d0/d6/d7/f8 [2786989,117052] 0 2022-01-31T19:44:13.679 INFO:tasks.workunit.client.0.smithi167.stdout:2/445: link dd/df/d1b/d3d/d4a/d4b/l5e dd/df/d1b/d3d/d4a/d4b/d54/l88 0 2022-01-31T19:44:13.681 INFO:tasks.workunit.client.0.smithi167.stdout:9/481: mknod dd/d19/d24/d50/d5d/c9e 0 2022-01-31T19:44:13.682 INFO:tasks.workunit.client.0.smithi167.stdout:4/529: creat d0/d6/d7/d89/fa9 x:0 0 0 2022-01-31T19:44:13.684 INFO:tasks.workunit.client.0.smithi167.stdout:1/673: dwrite d0/d10/d20/f56 [0,4194304] 0 2022-01-31T19:44:13.684 INFO:tasks.workunit.client.0.smithi167.stdout:7/709: dread d8/dc/d18/d24/dbb/f79 [0,4194304] 0 2022-01-31T19:44:13.688 INFO:tasks.workunit.client.0.smithi167.stdout:4/530: rename d0/d6/d7/d26/d31/f8b to d0/d6/d7/d39/d4e/faa 0 2022-01-31T19:44:13.688 INFO:tasks.workunit.client.0.smithi167.stdout:4/531: truncate d0/d6/d7/d39/d3a/f54 1845428 0 2022-01-31T19:44:13.689 INFO:tasks.workunit.client.0.smithi167.stdout:2/446: dread dd/df/d1b/d3d/f47 [0,4194304] 0 2022-01-31T19:44:13.691 INFO:tasks.workunit.client.0.smithi167.stdout:1/674: unlink d0/cf 0 2022-01-31T19:44:13.691 INFO:tasks.workunit.client.0.smithi167.stdout:1/675: readlink d0/d62/d7f/l8b 0 2022-01-31T19:44:13.692 INFO:tasks.workunit.client.0.smithi167.stdout:7/710: rename d8/d2c/d6e/d4d/d67/c75 to d8/d2c/d6e/d4d/d67/d8f/c101 0 2022-01-31T19:44:13.694 INFO:tasks.workunit.client.0.smithi167.stdout:2/447: link dd/df/d1b/d3d/d4a/f6b dd/d18/d62/f89 0 2022-01-31T19:44:13.694 INFO:tasks.workunit.client.0.smithi167.stdout:2/448: chown dd/df/d1b/d2f/d36/c3b 20295 1 2022-01-31T19:44:13.695 INFO:tasks.workunit.client.0.smithi167.stdout:1/676: truncate d0/d62/fa3 13226 0 2022-01-31T19:44:13.695 INFO:tasks.workunit.client.0.smithi167.stdout:1/677: write d0/d11/fab [576918,42503] 0 2022-01-31T19:44:13.695 INFO:tasks.workunit.client.0.smithi167.stdout:7/711: link d8/d2c/d7a/fc8 d8/d2c/d43/d94/daf/f102 0 2022-01-31T19:44:13.697 INFO:tasks.workunit.client.0.smithi167.stdout:2/449: creat dd/df/d22/f8a x:0 0 0 2022-01-31T19:44:13.697 INFO:tasks.workunit.client.0.smithi167.stdout:9/482: dread dd/d35/f60 [0,4194304] 0 2022-01-31T19:44:13.698 INFO:tasks.workunit.client.0.smithi167.stdout:0/318: dwrite d17/d3e/f61 [0,4194304] 0 2022-01-31T19:44:13.698 INFO:tasks.workunit.client.0.smithi167.stdout:1/678: symlink d0/d75/le5 0 2022-01-31T19:44:13.698 INFO:tasks.workunit.client.0.smithi167.stdout:1/679: getdents d0/d75/db4/ddb 0 2022-01-31T19:44:13.699 INFO:tasks.workunit.client.0.smithi167.stdout:1/680: truncate d0/d10/d20/d7c/faa 5123281 0 2022-01-31T19:44:13.700 INFO:tasks.workunit.client.0.smithi167.stdout:7/712: mknod d8/dc/c103 0 2022-01-31T19:44:13.701 INFO:tasks.workunit.client.0.smithi167.stdout:7/713: getdents d8/d2c/d7a/d99/de2/dc3/de9 0 2022-01-31T19:44:13.701 INFO:tasks.workunit.client.0.smithi167.stdout:2/450: unlink dd/df/d1b/d3d/d4a/f6b 0 2022-01-31T19:44:13.701 INFO:tasks.workunit.client.0.smithi167.stdout:2/451: getdents dd/d18/d62/d75 0 2022-01-31T19:44:13.705 INFO:tasks.workunit.client.0.smithi167.stdout:9/483: rename dd/d17/d80/f4a to dd/d19/d59/f9f 0 2022-01-31T19:44:13.705 INFO:tasks.workunit.client.0.smithi167.stdout:9/484: chown dd/d19/d24/d56/l73 11183041 1 2022-01-31T19:44:13.706 INFO:tasks.workunit.client.0.smithi167.stdout:9/485: symlink dd/la0 0 2022-01-31T19:44:13.706 INFO:tasks.workunit.client.0.smithi167.stdout:9/486: fdatasync dd/d19/f3d 0 2022-01-31T19:44:13.707 INFO:tasks.workunit.client.0.smithi167.stdout:9/487: chown dd/d19/d24/c49 8 1 2022-01-31T19:44:13.717 INFO:tasks.workunit.client.0.smithi167.stdout:9/488: symlink dd/d19/d24/d50/d5d/la1 0 2022-01-31T19:44:13.717 INFO:tasks.workunit.client.0.smithi167.stdout:4/532: dwrite d0/d6/d7/d26/f62 [0,4194304] 0 2022-01-31T19:44:13.717 INFO:tasks.workunit.client.0.smithi167.stdout:4/533: chown d0/d6/d7/f9a 1 1 2022-01-31T19:44:13.718 INFO:tasks.workunit.client.0.smithi167.stdout:7/714: dread d8/d2c/d6e/f44 [0,4194304] 0 2022-01-31T19:44:13.718 INFO:tasks.workunit.client.0.smithi167.stdout:4/534: write d0/d3f/f2e [120977,5008] 0 2022-01-31T19:44:13.718 INFO:tasks.workunit.client.0.smithi167.stdout:4/535: dread - d0/d3f/d60/d65/f90 zero size 2022-01-31T19:44:13.718 INFO:tasks.workunit.client.0.smithi167.stdout:4/536: write d0/d6/d7/d26/d40/f64 [491764,87902] 0 2022-01-31T19:44:13.719 INFO:tasks.workunit.client.0.smithi167.stdout:1/681: dwrite d0/d10/d5a/d33/f55 [0,4194304] 0 2022-01-31T19:44:13.719 INFO:tasks.workunit.client.0.smithi167.stdout:7/715: readlink d8/lb 0 2022-01-31T19:44:13.726 INFO:tasks.workunit.client.0.smithi167.stdout:9/489: dwrite dd/d17/f18 [4194304,4194304] 0 2022-01-31T19:44:13.727 INFO:tasks.workunit.client.0.smithi167.stdout:1/682: symlink d0/d10/d20/d7c/le6 0 2022-01-31T19:44:13.727 INFO:tasks.workunit.client.0.smithi167.stdout:1/683: fsync d0/d10/d5a/d33/d4b/f68 0 2022-01-31T19:44:13.728 INFO:tasks.workunit.client.0.smithi167.stdout:1/684: chown d0/d10/d5a/d33/d4b/da5/l92 3 1 2022-01-31T19:44:13.728 INFO:tasks.workunit.client.0.smithi167.stdout:7/716: creat d8/d2c/d7a/dd1/f104 x:0 0 0 2022-01-31T19:44:13.732 INFO:tasks.workunit.client.0.smithi167.stdout:9/490: link dd/d19/d24/c7e dd/d17/d80/d4c/ca2 0 2022-01-31T19:44:13.733 INFO:tasks.workunit.client.0.smithi167.stdout:1/685: mknod d0/d11/ce7 0 2022-01-31T19:44:13.734 INFO:tasks.workunit.client.0.smithi167.stdout:1/686: read d0/d62/fa3 [8713,107469] 0 2022-01-31T19:44:13.735 INFO:tasks.workunit.client.0.smithi167.stdout:9/491: creat dd/fa3 x:0 0 0 2022-01-31T19:44:13.735 INFO:tasks.workunit.client.0.smithi167.stdout:9/492: readlink dd/d19/d24/d50/d5d/la1 0 2022-01-31T19:44:13.736 INFO:tasks.workunit.client.0.smithi167.stdout:1/687: mkdir d0/d10/d20/de8 0 2022-01-31T19:44:13.737 INFO:tasks.workunit.client.0.smithi167.stdout:9/493: creat dd/d19/d24/d50/d5d/fa4 x:0 0 0 2022-01-31T19:44:13.738 INFO:tasks.workunit.client.0.smithi167.stdout:9/494: chown dd/d19 55048327 1 2022-01-31T19:44:13.740 INFO:tasks.workunit.client.0.smithi167.stdout:9/495: rename dd/d35/d8f to dd/d19/d24/d56/d5b/da5 0 2022-01-31T19:44:13.742 INFO:tasks.workunit.client.0.smithi167.stdout:1/688: dread d0/fe4 [0,4194304] 0 2022-01-31T19:44:13.744 INFO:tasks.workunit.client.0.smithi167.stdout:9/496: dread dd/f84 [0,4194304] 0 2022-01-31T19:44:13.745 INFO:tasks.workunit.client.0.smithi167.stdout:9/497: chown dd/d19/d24/d56/l73 5141926 1 2022-01-31T19:44:13.745 INFO:tasks.workunit.client.0.smithi167.stdout:9/498: chown dd/d19/d24/d56/d5b/c89 196 1 2022-01-31T19:44:13.745 INFO:tasks.workunit.client.0.smithi167.stdout:9/499: stat dd/d6b/d72 0 2022-01-31T19:44:13.746 INFO:tasks.workunit.client.0.smithi167.stdout:1/689: read d0/d10/d20/f28 [2453533,127580] 0 2022-01-31T19:44:13.746 INFO:tasks.workunit.client.0.smithi167.stdout:1/690: write d0/d62/d7f/fdd [619242,60984] 0 2022-01-31T19:44:13.746 INFO:tasks.workunit.client.0.smithi167.stdout:9/500: creat dd/d6b/d72/fa6 x:0 0 0 2022-01-31T19:44:13.747 INFO:tasks.workunit.client.0.smithi167.stdout:9/501: rename dd/c62 to dd/d6b/d72/ca7 0 2022-01-31T19:44:13.747 INFO:tasks.workunit.client.0.smithi167.stdout:4/537: dwrite d0/d6/d7/d26/d40/f64 [0,4194304] 0 2022-01-31T19:44:13.748 INFO:tasks.workunit.client.0.smithi167.stdout:4/538: stat d0/d6/d7/d26/d31/f5e 0 2022-01-31T19:44:13.748 INFO:tasks.workunit.client.0.smithi167.stdout:7/717: dwrite d8/d2c/d43/f2f [0,4194304] 0 2022-01-31T19:44:13.754 INFO:tasks.workunit.client.0.smithi167.stdout:7/718: symlink d8/d2c/d6e/d4d/d5e/d71/db1/l105 0 2022-01-31T19:44:13.754 INFO:tasks.workunit.client.0.smithi167.stdout:4/539: rmdir d0/d6/d7/d26 39 2022-01-31T19:44:13.756 INFO:tasks.workunit.client.0.smithi167.stdout:4/540: unlink d0/d6/d7/d26/d40/c66 0 2022-01-31T19:44:13.760 INFO:tasks.workunit.client.0.smithi167.stdout:4/541: write d0/d6/d7/f30 [2281064,106367] 0 2022-01-31T19:44:13.760 INFO:tasks.workunit.client.0.smithi167.stdout:4/542: chown d0/d6/d7/f4a 6679044 1 2022-01-31T19:44:13.761 INFO:tasks.workunit.client.0.smithi167.stdout:4/543: readlink d0/d6/d7/d26/d40/l8d 0 2022-01-31T19:44:13.761 INFO:tasks.workunit.client.0.smithi167.stdout:4/544: mkdir d0/d6/dab 0 2022-01-31T19:44:13.777 INFO:tasks.workunit.client.0.smithi167.stdout:4/545: dread d0/d6/d7/d26/d40/f64 [0,4194304] 0 2022-01-31T19:44:13.778 INFO:tasks.workunit.client.0.smithi167.stdout:4/546: creat d0/fac x:0 0 0 2022-01-31T19:44:13.778 INFO:tasks.workunit.client.0.smithi167.stdout:4/547: creat d0/d6/d7/d89/fad x:0 0 0 2022-01-31T19:44:13.778 INFO:tasks.workunit.client.0.smithi167.stdout:4/548: fsync d0/d6/d7/d39/d3a/d79/f91 0 2022-01-31T19:44:13.779 INFO:tasks.workunit.client.0.smithi167.stdout:4/549: fsync d0/d3f/d60/d65/f88 0 2022-01-31T19:44:13.786 INFO:tasks.workunit.client.0.smithi167.stdout:1/691: dwrite d0/d75/f52 [0,4194304] 0 2022-01-31T19:44:13.787 INFO:tasks.workunit.client.0.smithi167.stdout:1/692: fsync d0/d62/d7f/d82/fba 0 2022-01-31T19:44:13.787 INFO:tasks.workunit.client.0.smithi167.stdout:7/719: dwrite d8/fa [0,4194304] 0 2022-01-31T19:44:13.788 INFO:tasks.workunit.client.0.smithi167.stdout:7/720: chown d8/dc/d18/d24/f29 63 1 2022-01-31T19:44:13.788 INFO:tasks.workunit.client.0.smithi167.stdout:7/721: fdatasync d8/d2c/d6e/d4d/d5e/d71/ff7 0 2022-01-31T19:44:13.789 INFO:tasks.workunit.client.0.smithi167.stdout:1/693: unlink d0/d10/d5a/l51 0 2022-01-31T19:44:13.789 INFO:tasks.workunit.client.0.smithi167.stdout:1/694: dread - d0/d10/d5a/fd7 zero size 2022-01-31T19:44:13.790 INFO:tasks.workunit.client.0.smithi167.stdout:7/722: symlink d8/d2c/d6e/d4d/d5e/d71/db3/l106 0 2022-01-31T19:44:13.790 INFO:tasks.workunit.client.0.smithi167.stdout:7/723: write d8/d2c/d43/f2f [4540087,59567] 0 2022-01-31T19:44:13.790 INFO:tasks.workunit.client.0.smithi167.stdout:1/695: mknod d0/d10/d5a/d33/d4b/d1f/ce9 0 2022-01-31T19:44:13.794 INFO:tasks.workunit.client.0.smithi167.stdout:9/502: dwrite dd/f84 [0,4194304] 0 2022-01-31T19:44:13.799 INFO:tasks.workunit.client.0.smithi167.stdout:4/550: dwrite d0/d6/d7/d26/f72 [0,4194304] 0 2022-01-31T19:44:13.800 INFO:tasks.workunit.client.0.smithi167.stdout:7/724: dread d8/feb [0,4194304] 0 2022-01-31T19:44:13.803 INFO:tasks.workunit.client.0.smithi167.stdout:7/725: creat d8/dc/dc1/f107 x:0 0 0 2022-01-31T19:44:13.804 INFO:tasks.workunit.client.0.smithi167.stdout:7/726: chown d8/d2c/d6e/d4d/d5e/d71/lbd 2029 1 2022-01-31T19:44:13.805 INFO:tasks.workunit.client.0.smithi167.stdout:7/727: truncate f6 4124561 0 2022-01-31T19:44:13.805 INFO:tasks.workunit.client.0.smithi167.stdout:4/551: dread d0/d6/d7/d39/d3a/f54 [0,4194304] 0 2022-01-31T19:44:13.811 INFO:tasks.workunit.client.0.smithi167.stdout:9/503: dwrite dd/d19/d24/d56/f85 [4194304,4194304] 0 2022-01-31T19:44:13.812 INFO:tasks.workunit.client.0.smithi167.stdout:7/728: dread d8/d2c/d6e/d4d/d5e/fa9 [0,4194304] 0 2022-01-31T19:44:13.813 INFO:tasks.workunit.client.0.smithi167.stdout:9/504: mknod dd/d19/d24/d50/ca8 0 2022-01-31T19:44:13.813 INFO:tasks.workunit.client.0.smithi167.stdout:9/505: dread - dd/d19/d24/d56/f8e zero size 2022-01-31T19:44:13.814 INFO:tasks.workunit.client.0.smithi167.stdout:9/506: write dd/d19/d24/d56/f85 [1340000,81623] 0 2022-01-31T19:44:13.815 INFO:tasks.workunit.client.0.smithi167.stdout:7/729: mknod d8/d2c/d6e/d4d/d67/d8f/c108 0 2022-01-31T19:44:13.817 INFO:tasks.workunit.client.0.smithi167.stdout:9/507: link dd/d17/c47 dd/d17/d80/d4c/ca9 0 2022-01-31T19:44:13.818 INFO:tasks.workunit.client.0.smithi167.stdout:7/730: rename d8/d2c/d6e/d4d/d5e/dce/ld5 to d8/d2c/d6e/d9a/l109 0 2022-01-31T19:44:13.819 INFO:tasks.workunit.client.0.smithi167.stdout:7/731: truncate d8/d2c/d43/d94/daf/fe6 121668 0 2022-01-31T19:44:13.820 INFO:tasks.workunit.client.0.smithi167.stdout:9/508: symlink dd/d19/d24/laa 0 2022-01-31T19:44:13.820 INFO:tasks.workunit.client.0.smithi167.stdout:9/509: stat dd/d35 0 2022-01-31T19:44:13.820 INFO:tasks.workunit.client.0.smithi167.stdout:9/510: readlink l2 0 2022-01-31T19:44:13.820 INFO:tasks.workunit.client.0.smithi167.stdout:7/732: truncate d8/d2c/d6e/d4d/d5e/fa9 1593257 0 2022-01-31T19:44:13.821 INFO:tasks.workunit.client.0.smithi167.stdout:7/733: write d8/d2c/fda [1169233,113244] 0 2022-01-31T19:44:13.822 INFO:tasks.workunit.client.0.smithi167.stdout:9/511: link dd/d34/l68 dd/d6b/d72/lab 0 2022-01-31T19:44:13.822 INFO:tasks.workunit.client.0.smithi167.stdout:9/512: fsync dd/d35/f9a 0 2022-01-31T19:44:13.822 INFO:tasks.workunit.client.0.smithi167.stdout:7/734: mknod d8/dab/c10a 0 2022-01-31T19:44:13.824 INFO:tasks.workunit.client.0.smithi167.stdout:9/513: mknod dd/d19/d24/d56/cac 0 2022-01-31T19:44:13.824 INFO:tasks.workunit.client.0.smithi167.stdout:4/552: truncate d0/d6/d7/f30 3785785 0 2022-01-31T19:44:13.827 INFO:tasks.workunit.client.0.smithi167.stdout:7/735: rename d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f3c to d8/d2c/d43/d94/f10b 0 2022-01-31T19:44:13.827 INFO:tasks.workunit.client.0.smithi167.stdout:7/736: readlink d8/d2c/d43/l92 0 2022-01-31T19:44:13.827 INFO:tasks.workunit.client.0.smithi167.stdout:7/737: chown d8/dc/d18/l2d 2044521 1 2022-01-31T19:44:13.828 INFO:tasks.workunit.client.0.smithi167.stdout:4/553: creat d0/d3f/fae x:0 0 0 2022-01-31T19:44:13.830 INFO:tasks.workunit.client.0.smithi167.stdout:7/738: link d8/d2c/d7a/d99/de2/dc3/ff2 d8/d2c/d6e/d9a/f10c 0 2022-01-31T19:44:13.832 INFO:tasks.workunit.client.0.smithi167.stdout:4/554: symlink d0/d6/d7/d89/laf 0 2022-01-31T19:44:13.832 INFO:tasks.workunit.client.0.smithi167.stdout:4/555: chown d0/d6/d7/d26/d31/d5c/ca0 7 1 2022-01-31T19:44:13.832 INFO:tasks.workunit.client.0.smithi167.stdout:4/556: write d0/d3f/d60/d65/f90 [386673,48058] 0 2022-01-31T19:44:13.833 INFO:tasks.workunit.client.0.smithi167.stdout:6/463: sync 2022-01-31T19:44:13.833 INFO:tasks.workunit.client.0.smithi167.stdout:8/582: sync 2022-01-31T19:44:13.833 INFO:tasks.workunit.client.0.smithi167.stdout:6/464: fsync d8/f5b 0 2022-01-31T19:44:13.833 INFO:tasks.workunit.client.0.smithi167.stdout:8/583: fdatasync d6/db/de/f10 0 2022-01-31T19:44:13.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/584: write d6/db/d28/d66/f8c [158125,89479] 0 2022-01-31T19:44:13.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/585: truncate d6/db/d6a/d8d/d9e/d60/f6c 673051 0 2022-01-31T19:44:13.834 INFO:tasks.workunit.client.0.smithi167.stdout:7/739: getdents d8/d2c/d6e/d9a/dd8 0 2022-01-31T19:44:13.835 INFO:tasks.workunit.client.0.smithi167.stdout:4/557: unlink d0/d3f/d75/c82 0 2022-01-31T19:44:13.836 INFO:tasks.workunit.client.0.smithi167.stdout:6/465: creat d8/d33/d51/d5d/faa x:0 0 0 2022-01-31T19:44:13.838 INFO:tasks.workunit.client.0.smithi167.stdout:8/586: mknod d6/db/d4e/dbd/cc0 0 2022-01-31T19:44:13.839 INFO:tasks.workunit.client.0.smithi167.stdout:7/740: symlink d8/d35/l10d 0 2022-01-31T19:44:13.840 INFO:tasks.workunit.client.0.smithi167.stdout:9/514: dwrite dd/d17/d80/d4c/f5f [0,4194304] 0 2022-01-31T19:44:13.847 INFO:tasks.workunit.client.0.smithi167.stdout:4/558: link d0/d6/l7c d0/d6/d7/d39/d93/lb0 0 2022-01-31T19:44:13.848 INFO:tasks.workunit.client.0.smithi167.stdout:4/559: stat d0/d6/d7/d26/c5b 0 2022-01-31T19:44:13.848 INFO:tasks.workunit.client.0.smithi167.stdout:4/560: truncate d0/d6/d7/d89/fa9 880646 0 2022-01-31T19:44:13.856 INFO:tasks.workunit.client.0.smithi167.stdout:6/466: unlink d8/d33/d51/d5d/d64/f68 0 2022-01-31T19:44:13.859 INFO:tasks.workunit.client.0.smithi167.stdout:8/587: getdents d6/db/d6a/d8d/d9e 0 2022-01-31T19:44:13.859 INFO:tasks.workunit.client.0.smithi167.stdout:8/588: chown d6/l16 16615 1 2022-01-31T19:44:13.859 INFO:tasks.workunit.client.0.smithi167.stdout:8/589: fsync d6/f32 0 2022-01-31T19:44:13.860 INFO:tasks.workunit.client.0.smithi167.stdout:8/590: creat d6/db/de/d3d/fc1 x:0 0 0 2022-01-31T19:44:13.860 INFO:tasks.workunit.client.0.smithi167.stdout:8/591: truncate d6/db/de/f10 2320631 0 2022-01-31T19:44:13.860 INFO:tasks.workunit.client.0.smithi167.stdout:8/592: dread - d6/db/d6a/d8d/d9e/d60/f79 zero size 2022-01-31T19:44:13.861 INFO:tasks.workunit.client.0.smithi167.stdout:7/741: symlink d8/d2c/d6e/l10e 0 2022-01-31T19:44:13.864 INFO:tasks.workunit.client.0.smithi167.stdout:9/515: creat dd/d19/d59/fad x:0 0 0 2022-01-31T19:44:13.864 INFO:tasks.workunit.client.0.smithi167.stdout:9/516: creat dd/d19/d59/fae x:0 0 0 2022-01-31T19:44:13.864 INFO:tasks.workunit.client.0.smithi167.stdout:9/517: chown dd/d19/d24/c49 1263949 1 2022-01-31T19:44:13.864 INFO:tasks.workunit.client.0.smithi167.stdout:9/518: write dd/d35/f9a [990846,105320] 0 2022-01-31T19:44:13.865 INFO:tasks.workunit.client.0.smithi167.stdout:4/561: mkdir d0/d6/d7/d89/db1 0 2022-01-31T19:44:13.867 INFO:tasks.workunit.client.0.smithi167.stdout:6/467: mkdir d8/d33/d51/d5d/d64/dab 0 2022-01-31T19:44:13.869 INFO:tasks.workunit.client.0.smithi167.stdout:8/593: mkdir d6/db/de/d18/da4/dc2 0 2022-01-31T19:44:13.869 INFO:tasks.workunit.client.0.smithi167.stdout:8/594: readlink d6/db/d28/d61/lad 0 2022-01-31T19:44:13.870 INFO:tasks.workunit.client.0.smithi167.stdout:7/742: readlink d8/l13 0 2022-01-31T19:44:13.870 INFO:tasks.workunit.client.0.smithi167.stdout:8/595: read d6/db/d28/f76 [341831,122138] 0 2022-01-31T19:44:13.870 INFO:tasks.workunit.client.0.smithi167.stdout:8/596: stat d6/db/d4e/d73 0 2022-01-31T19:44:13.871 INFO:tasks.workunit.client.0.smithi167.stdout:8/597: chown d6/db/d36/l42 7884351 1 2022-01-31T19:44:13.871 INFO:tasks.workunit.client.0.smithi167.stdout:8/598: fdatasync d6/f1d 0 2022-01-31T19:44:13.871 INFO:tasks.workunit.client.0.smithi167.stdout:9/519: symlink dd/d19/d24/laf 0 2022-01-31T19:44:13.872 INFO:tasks.workunit.client.0.smithi167.stdout:4/562: mknod d0/d6/d7/d39/d4e/cb2 0 2022-01-31T19:44:13.874 INFO:tasks.workunit.client.0.smithi167.stdout:7/743: truncate d8/d2c/d43/f97 127071 0 2022-01-31T19:44:13.877 INFO:tasks.workunit.client.0.smithi167.stdout:6/468: rename d8/d33/d4f/f53 to d8/fac 0 2022-01-31T19:44:13.877 INFO:tasks.workunit.client.0.smithi167.stdout:6/469: fsync d8/f9 0 2022-01-31T19:44:13.879 INFO:tasks.workunit.client.0.smithi167.stdout:7/744: mkdir d8/d2c/d7a/d10f 0 2022-01-31T19:44:13.881 INFO:tasks.workunit.client.0.smithi167.stdout:4/563: dread d0/d6/d7/d89/fa9 [0,4194304] 0 2022-01-31T19:44:13.884 INFO:tasks.workunit.client.0.smithi167.stdout:6/470: mkdir d8/d33/d51/d72/dad 0 2022-01-31T19:44:13.887 INFO:tasks.workunit.client.0.smithi167.stdout:7/745: creat d8/d2c/d6e/d9a/f110 x:0 0 0 2022-01-31T19:44:13.887 INFO:tasks.workunit.client.0.smithi167.stdout:7/746: write d8/dc/f23 [4629748,113987] 0 2022-01-31T19:44:13.887 INFO:tasks.workunit.client.0.smithi167.stdout:7/747: chown d8/d2c/d6e/d4d/d5e/d71/db1/d4f/fc2 1607 1 2022-01-31T19:44:13.889 INFO:tasks.workunit.client.0.smithi167.stdout:7/748: mknod d8/dc/d18/d24/dcc/c111 0 2022-01-31T19:44:13.890 INFO:tasks.workunit.client.0.smithi167.stdout:9/520: dwrite dd/d17/f36 [4194304,4194304] 0 2022-01-31T19:44:13.892 INFO:tasks.workunit.client.0.smithi167.stdout:9/521: symlink dd/d19/d24/d56/d5b/da5/lb0 0 2022-01-31T19:44:13.892 INFO:tasks.workunit.client.0.smithi167.stdout:9/522: chown dd/f64 2812083 1 2022-01-31T19:44:13.892 INFO:tasks.workunit.client.0.smithi167.stdout:9/523: read dd/d34/f61 [29472,64462] 0 2022-01-31T19:44:13.893 INFO:tasks.workunit.client.0.smithi167.stdout:9/524: mknod dd/d19/d24/d50/cb1 0 2022-01-31T19:44:13.894 INFO:tasks.workunit.client.0.smithi167.stdout:9/525: link dd/d19/d59/f98 dd/d17/d7f/fb2 0 2022-01-31T19:44:13.911 INFO:tasks.workunit.client.0.smithi167.stdout:4/564: dwrite d0/d6/d7/d39/f43 [0,4194304] 0 2022-01-31T19:44:13.911 INFO:tasks.workunit.client.0.smithi167.stdout:4/565: write d0/d6/d7/d89/fa3 [754204,112842] 0 2022-01-31T19:44:13.911 INFO:tasks.workunit.client.0.smithi167.stdout:6/471: dwrite d8/d62/f6d [0,4194304] 0 2022-01-31T19:44:13.912 INFO:tasks.workunit.client.0.smithi167.stdout:9/526: dwrite dd/d34/f61 [0,4194304] 0 2022-01-31T19:44:13.912 INFO:tasks.workunit.client.0.smithi167.stdout:4/566: mkdir d0/d6/d7/d26/d50/db3 0 2022-01-31T19:44:13.913 INFO:tasks.workunit.client.0.smithi167.stdout:4/567: fdatasync d0/d6/d7/f7b 0 2022-01-31T19:44:13.913 INFO:tasks.workunit.client.0.smithi167.stdout:6/472: creat d8/d33/d51/d5d/d64/dab/fae x:0 0 0 2022-01-31T19:44:13.915 INFO:tasks.workunit.client.0.smithi167.stdout:4/568: mkdir d0/d6/d7/d26/d40/d4c/db4 0 2022-01-31T19:44:13.915 INFO:tasks.workunit.client.0.smithi167.stdout:4/569: getdents d0/d3f/d60/d65 0 2022-01-31T19:44:13.916 INFO:tasks.workunit.client.0.smithi167.stdout:4/570: mkdir d0/d3f/d60/db5 0 2022-01-31T19:44:13.916 INFO:tasks.workunit.client.0.smithi167.stdout:4/571: chown d0/d6/f58 6663 1 2022-01-31T19:44:13.917 INFO:tasks.workunit.client.0.smithi167.stdout:4/572: mkdir d0/d6/d7/d39/d3a/db6 0 2022-01-31T19:44:13.918 INFO:tasks.workunit.client.0.smithi167.stdout:4/573: rename d0/d6/d7/d39/f2b to d0/d6/d7/d39/d93/fb7 0 2022-01-31T19:44:13.920 INFO:tasks.workunit.client.0.smithi167.stdout:9/527: dread dd/d35/f60 [0,4194304] 0 2022-01-31T19:44:13.920 INFO:tasks.workunit.client.0.smithi167.stdout:9/528: read - dd/d19/f6d zero size 2022-01-31T19:44:13.921 INFO:tasks.workunit.client.0.smithi167.stdout:9/529: chown dd/d17/c6c 15737 1 2022-01-31T19:44:13.921 INFO:tasks.workunit.client.0.smithi167.stdout:9/530: read - dd/fa3 zero size 2022-01-31T19:44:13.922 INFO:tasks.workunit.client.0.smithi167.stdout:9/531: mkdir dd/d17/d80/d4c/db3 0 2022-01-31T19:44:13.922 INFO:tasks.workunit.client.0.smithi167.stdout:7/749: dwrite d8/dc/d18/d24/dbb/f6d [0,4194304] 0 2022-01-31T19:44:13.922 INFO:tasks.workunit.client.0.smithi167.stdout:7/750: readlink d8/d2c/d6e/d9a/l109 0 2022-01-31T19:44:13.923 INFO:tasks.workunit.client.0.smithi167.stdout:9/532: creat dd/d19/d24/d56/d5b/da5/fb4 x:0 0 0 2022-01-31T19:44:13.933 INFO:tasks.workunit.client.0.smithi167.stdout:4/574: dread d0/d6/d7/d39/d4e/f52 [0,4194304] 0 2022-01-31T19:44:13.935 INFO:tasks.workunit.client.0.smithi167.stdout:4/575: mknod d0/cb8 0 2022-01-31T19:44:13.936 INFO:tasks.workunit.client.0.smithi167.stdout:4/576: mknod d0/d3f/d60/d65/cb9 0 2022-01-31T19:44:13.937 INFO:tasks.workunit.client.0.smithi167.stdout:4/577: symlink d0/d3f/d75/lba 0 2022-01-31T19:44:13.938 INFO:tasks.workunit.client.0.smithi167.stdout:6/473: dwrite d8/f1c [0,4194304] 0 2022-01-31T19:44:13.940 INFO:tasks.workunit.client.0.smithi167.stdout:6/474: truncate d8/f7a 287522 0 2022-01-31T19:44:13.940 INFO:tasks.workunit.client.0.smithi167.stdout:6/475: write d8/d33/d51/d72/f83 [606154,96371] 0 2022-01-31T19:44:13.940 INFO:tasks.workunit.client.0.smithi167.stdout:6/476: readlink d8/d33/l3a 0 2022-01-31T19:44:13.940 INFO:tasks.workunit.client.0.smithi167.stdout:6/477: dread - d8/d33/d51/d5d/faa zero size 2022-01-31T19:44:13.946 INFO:tasks.workunit.client.0.smithi167.stdout:4/578: write d0/d6/d7/f30 [81807,47552] 0 2022-01-31T19:44:13.947 INFO:tasks.workunit.client.0.smithi167.stdout:7/751: dwrite d8/dab/fed [0,4194304] 0 2022-01-31T19:44:13.948 INFO:tasks.workunit.client.0.smithi167.stdout:9/533: dwrite dd/d17/f2f [0,4194304] 0 2022-01-31T19:44:13.949 INFO:tasks.workunit.client.0.smithi167.stdout:9/534: fsync dd/d34/f61 0 2022-01-31T19:44:13.949 INFO:tasks.workunit.client.0.smithi167.stdout:9/535: stat dd/d19/d24/d50/d5d/fa4 0 2022-01-31T19:44:13.951 INFO:tasks.workunit.client.0.smithi167.stdout:7/752: truncate d8/dc/f1c 2927506 0 2022-01-31T19:44:13.953 INFO:tasks.workunit.client.0.smithi167.stdout:9/536: rename dd/d17/d80/d4c/ca9 to dd/d17/d80/d4c/db3/cb5 0 2022-01-31T19:44:13.954 INFO:tasks.workunit.client.0.smithi167.stdout:9/537: chown dd/d19/d24/d56/f75 6935232 1 2022-01-31T19:44:13.954 INFO:tasks.workunit.client.0.smithi167.stdout:9/538: write dd/d19/d24/d56/f8e [141573,37390] 0 2022-01-31T19:44:13.958 INFO:tasks.workunit.client.0.smithi167.stdout:7/753: creat d8/d35/f112 x:0 0 0 2022-01-31T19:44:13.958 INFO:tasks.workunit.client.0.smithi167.stdout:9/539: creat dd/d19/d24/fb6 x:0 0 0 2022-01-31T19:44:13.960 INFO:tasks.workunit.client.0.smithi167.stdout:6/478: dwrite d8/d20/d54/f85 [0,4194304] 0 2022-01-31T19:44:13.961 INFO:tasks.workunit.client.0.smithi167.stdout:4/579: dread d0/d6/d7/f4a [0,4194304] 0 2022-01-31T19:44:13.961 INFO:tasks.workunit.client.0.smithi167.stdout:7/754: getdents d8/d2c/d6e/d4d/d67 0 2022-01-31T19:44:13.962 INFO:tasks.workunit.client.0.smithi167.stdout:7/755: readlink d8/d35/lf4 0 2022-01-31T19:44:13.963 INFO:tasks.workunit.client.0.smithi167.stdout:7/756: write d8/d2c/d6e/d4d/d67/fb9 [2014764,115578] 0 2022-01-31T19:44:13.964 INFO:tasks.workunit.client.0.smithi167.stdout:4/580: symlink d0/d6/d7/d39/d93/lbb 0 2022-01-31T19:44:13.967 INFO:tasks.workunit.client.0.smithi167.stdout:7/757: truncate d8/d2c/d7a/fcd 106002 0 2022-01-31T19:44:13.969 INFO:tasks.workunit.client.0.smithi167.stdout:6/479: dread d8/d62/f6f [0,4194304] 0 2022-01-31T19:44:13.970 INFO:tasks.workunit.client.0.smithi167.stdout:7/758: mknod d8/dc/d18/d24/dcc/c113 0 2022-01-31T19:44:13.980 INFO:tasks.workunit.client.0.smithi167.stdout:4/581: dread d0/d6/d7/d26/d40/f32 [0,4194304] 0 2022-01-31T19:44:13.982 INFO:tasks.workunit.client.0.smithi167.stdout:4/582: symlink d0/d6/d7/d26/d40/lbc 0 2022-01-31T19:44:13.983 INFO:tasks.workunit.client.0.smithi167.stdout:4/583: truncate d0/d6/d7/d26/d40/d4c/f9c 4845897 0 2022-01-31T19:44:13.985 INFO:tasks.workunit.client.0.smithi167.stdout:4/584: write d0/d6/f76 [2730739,47277] 0 2022-01-31T19:44:13.986 INFO:tasks.workunit.client.0.smithi167.stdout:4/585: mknod d0/d6/d7/d26/d50/cbd 0 2022-01-31T19:44:13.987 INFO:tasks.workunit.client.0.smithi167.stdout:4/586: dread - d0/d5d/fa5 zero size 2022-01-31T19:44:13.987 INFO:tasks.workunit.client.0.smithi167.stdout:4/587: creat d0/d6/d7/d26/d40/d4c/fbe x:0 0 0 2022-01-31T19:44:13.988 INFO:tasks.workunit.client.0.smithi167.stdout:4/588: creat d0/d6/d7/d39/d3a/d79/fbf x:0 0 0 2022-01-31T19:44:13.994 INFO:tasks.workunit.client.0.smithi167.stdout:7/759: dwrite d8/d2c/d6e/d4d/d67/ff5 [0,4194304] 0 2022-01-31T19:44:13.996 INFO:tasks.workunit.client.0.smithi167.stdout:7/760: symlink d8/dc/d18/l114 0 2022-01-31T19:44:13.996 INFO:tasks.workunit.client.0.smithi167.stdout:7/761: fdatasync d8/dc/f21 0 2022-01-31T19:44:13.997 INFO:tasks.workunit.client.0.smithi167.stdout:6/480: dwrite d8/d33/f82 [0,4194304] 0 2022-01-31T19:44:13.997 INFO:tasks.workunit.client.0.smithi167.stdout:7/762: symlink d8/d2c/d43/df3/l115 0 2022-01-31T19:44:13.997 INFO:tasks.workunit.client.0.smithi167.stdout:7/763: chown d8/d2c/d43/d94/daf/ldd 25499878 1 2022-01-31T19:44:13.998 INFO:tasks.workunit.client.0.smithi167.stdout:7/764: mkdir d8/d2c/d6e/d9a/dd8/d116 0 2022-01-31T19:44:14.000 INFO:tasks.workunit.client.0.smithi167.stdout:7/765: creat d8/d2c/d6e/d4d/d5e/d71/db1/f117 x:0 0 0 2022-01-31T19:44:14.001 INFO:tasks.workunit.client.0.smithi167.stdout:7/766: write d8/feb [1880282,104459] 0 2022-01-31T19:44:14.003 INFO:tasks.workunit.client.0.smithi167.stdout:7/767: mkdir d8/d2c/d43/df3/d118 0 2022-01-31T19:44:14.007 INFO:tasks.workunit.client.0.smithi167.stdout:7/768: dread d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:14.008 INFO:tasks.workunit.client.0.smithi167.stdout:7/769: mkdir d8/d35/d119 0 2022-01-31T19:44:14.014 INFO:tasks.workunit.client.0.smithi167.stdout:3/486: sync 2022-01-31T19:44:14.016 INFO:tasks.workunit.client.0.smithi167.stdout:3/487: creat d9/d26/f8b x:0 0 0 2022-01-31T19:44:14.016 INFO:tasks.workunit.client.0.smithi167.stdout:3/488: creat d9/d7f/f8c x:0 0 0 2022-01-31T19:44:14.019 INFO:tasks.workunit.client.0.smithi167.stdout:3/489: mkdir d9/d29/d3b/d8d 0 2022-01-31T19:44:14.019 INFO:tasks.workunit.client.0.smithi167.stdout:3/490: stat d9/l74 0 2022-01-31T19:44:14.020 INFO:tasks.workunit.client.0.smithi167.stdout:3/491: readlink d9/d29/d60/l6f 0 2022-01-31T19:44:14.020 INFO:tasks.workunit.client.0.smithi167.stdout:3/492: chown d9/c12 0 1 2022-01-31T19:44:14.021 INFO:tasks.workunit.client.0.smithi167.stdout:3/493: dread d9/d29/d3b/d41/d6b/f66 [0,4194304] 0 2022-01-31T19:44:14.038 INFO:tasks.workunit.client.0.smithi167.stdout:3/494: dwrite d9/d56/f73 [0,4194304] 0 2022-01-31T19:44:14.038 INFO:tasks.workunit.client.0.smithi167.stdout:3/495: stat d9/d56/f63 0 2022-01-31T19:44:14.039 INFO:tasks.workunit.client.0.smithi167.stdout:3/496: mknod d9/d56/c8e 0 2022-01-31T19:44:14.040 INFO:tasks.workunit.client.0.smithi167.stdout:3/497: write d9/d21/f77 [150479,15118] 0 2022-01-31T19:44:14.041 INFO:tasks.workunit.client.0.smithi167.stdout:3/498: read - d9/d26/f6e zero size 2022-01-31T19:44:14.041 INFO:tasks.workunit.client.0.smithi167.stdout:3/499: write d9/d29/d3b/f84 [10257,96050] 0 2022-01-31T19:44:14.041 INFO:tasks.workunit.client.0.smithi167.stdout:3/500: write d9/f78 [166733,33806] 0 2022-01-31T19:44:14.041 INFO:tasks.workunit.client.0.smithi167.stdout:3/501: truncate d9/d29/f4c 670616 0 2022-01-31T19:44:14.042 INFO:tasks.workunit.client.0.smithi167.stdout:3/502: mknod d9/d29/d3b/d41/d6b/d89/c8f 0 2022-01-31T19:44:14.044 INFO:tasks.workunit.client.0.smithi167.stdout:3/503: creat d9/d56/f90 x:0 0 0 2022-01-31T19:44:14.044 INFO:tasks.workunit.client.0.smithi167.stdout:3/504: truncate d9/d29/d60/f5b 291718 0 2022-01-31T19:44:14.044 INFO:tasks.workunit.client.0.smithi167.stdout:3/505: creat d9/d56/f91 x:0 0 0 2022-01-31T19:44:14.046 INFO:tasks.workunit.client.0.smithi167.stdout:3/506: mknod d9/d56/c92 0 2022-01-31T19:44:14.046 INFO:tasks.workunit.client.0.smithi167.stdout:3/507: creat d9/d7f/f93 x:0 0 0 2022-01-31T19:44:14.046 INFO:tasks.workunit.client.0.smithi167.stdout:3/508: fsync d9/d29/d3b/f57 0 2022-01-31T19:44:14.049 INFO:tasks.workunit.client.0.smithi167.stdout:3/509: link d9/d29/d60/f68 d9/d7f/f94 0 2022-01-31T19:44:14.049 INFO:tasks.workunit.client.0.smithi167.stdout:3/510: chown d9/d29/d3b/d41/d6b/f35 312339207 1 2022-01-31T19:44:14.050 INFO:tasks.workunit.client.0.smithi167.stdout:3/511: link d9/c88 d9/d29/d3b/d8d/c95 0 2022-01-31T19:44:14.052 INFO:tasks.workunit.client.0.smithi167.stdout:3/512: rename d9/d29/d3b/d41/d6b/f35 to d9/d29/d60/f96 0 2022-01-31T19:44:14.054 INFO:tasks.workunit.client.0.smithi167.stdout:3/513: creat d9/d29/d3b/d41/d6b/d89/f97 x:0 0 0 2022-01-31T19:44:14.054 INFO:tasks.workunit.client.0.smithi167.stdout:3/514: creat d9/d26/f98 x:0 0 0 2022-01-31T19:44:14.054 INFO:tasks.workunit.client.0.smithi167.stdout:3/515: fsync d9/d29/d60/f69 0 2022-01-31T19:44:14.055 INFO:tasks.workunit.client.0.smithi167.stdout:3/516: fdatasync d9/d29/d3b/d41/f5c 0 2022-01-31T19:44:14.055 INFO:tasks.workunit.client.0.smithi167.stdout:3/517: write d9/d26/f55 [2522217,35002] 0 2022-01-31T19:44:14.057 INFO:tasks.workunit.client.0.smithi167.stdout:3/518: unlink d9/d29/d3b/f57 0 2022-01-31T19:44:14.058 INFO:tasks.workunit.client.0.smithi167.stdout:3/519: write d9/f2c [838576,59413] 0 2022-01-31T19:44:14.058 INFO:tasks.workunit.client.0.smithi167.stdout:3/520: chown d9/f11 16 1 2022-01-31T19:44:14.059 INFO:tasks.workunit.client.0.smithi167.stdout:3/521: creat d9/d21/f99 x:0 0 0 2022-01-31T19:44:14.063 INFO:tasks.workunit.client.0.smithi167.stdout:3/522: dread d9/d29/d3b/d41/d6b/f39 [0,4194304] 0 2022-01-31T19:44:14.065 INFO:tasks.workunit.client.0.smithi167.stdout:3/523: mknod d9/d29/d60/c9a 0 2022-01-31T19:44:14.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:13 smithi171 conmon[35325]: cluster 2022-01-31T19:44:12.432313+0000 mgr.smithi171.asyxnc ( 2022-01-31T19:44:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:13 smithi171 conmon[35325]: mgr.24387) 18 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 803 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 38 MiB/s rd, 86 MiB/s wr, 137 op/s 2022-01-31T19:44:14.158 INFO:tasks.workunit.client.0.smithi167.stdout:5/689: sync 2022-01-31T19:44:14.158 INFO:tasks.workunit.client.0.smithi167.stdout:0/319: sync 2022-01-31T19:44:14.158 INFO:tasks.workunit.client.0.smithi167.stdout:0/320: fsync f14 0 2022-01-31T19:44:14.159 INFO:tasks.workunit.client.0.smithi167.stdout:0/321: creat d17/d19/d1c/d39/d44/f65 x:0 0 0 2022-01-31T19:44:14.161 INFO:tasks.workunit.client.0.smithi167.stdout:0/322: rmdir d17/d19/d1c 39 2022-01-31T19:44:14.161 INFO:tasks.workunit.client.0.smithi167.stdout:0/323: fdatasync fc 0 2022-01-31T19:44:14.162 INFO:tasks.workunit.client.0.smithi167.stdout:0/324: symlink d17/l66 0 2022-01-31T19:44:14.162 INFO:tasks.workunit.client.0.smithi167.stdout:0/325: read d17/d2c/f37 [203458,56954] 0 2022-01-31T19:44:14.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:13 smithi167 conmon[32206]: cluster 2022-01-31T19:44:12.432313 2022-01-31T19:44:14.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:13 smithi167 conmon[32206]: +0000 mgr.smithi171.asyxnc (mgr.24387) 18 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 803 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 38 MiB/s rd, 86 MiB/s wr, 137 op/s 2022-01-31T19:44:14.171 INFO:tasks.workunit.client.0.smithi167.stdout:0/326: dread f14 [0,4194304] 0 2022-01-31T19:44:14.172 INFO:tasks.workunit.client.0.smithi167.stdout:0/327: read d17/d2c/f3c [2738172,121023] 0 2022-01-31T19:44:14.172 INFO:tasks.workunit.client.0.smithi167.stdout:0/328: chown d17/d19/f4e 815 1 2022-01-31T19:44:14.172 INFO:tasks.workunit.client.0.smithi167.stdout:0/329: chown d17/d19/d1c/d39/d44/f56 55125 1 2022-01-31T19:44:14.173 INFO:tasks.workunit.client.0.smithi167.stdout:0/330: mknod d17/d3e/c67 0 2022-01-31T19:44:14.174 INFO:tasks.workunit.client.0.smithi167.stdout:0/331: creat d17/d3e/f68 x:0 0 0 2022-01-31T19:44:14.174 INFO:tasks.workunit.client.0.smithi167.stdout:0/332: chown d17/d19/f1f 16953 1 2022-01-31T19:44:14.174 INFO:tasks.workunit.client.0.smithi167.stdout:0/333: dread - d17/d19/f2d zero size 2022-01-31T19:44:14.175 INFO:tasks.workunit.client.0.smithi167.stdout:0/334: read d17/d19/d1c/f32 [5082008,107] 0 2022-01-31T19:44:14.175 INFO:tasks.workunit.client.0.smithi167.stdout:0/335: mkdir d17/d69 0 2022-01-31T19:44:14.179 INFO:tasks.workunit.client.0.smithi167.stdout:0/336: dread f14 [0,4194304] 0 2022-01-31T19:44:14.194 INFO:tasks.workunit.client.0.smithi167.stdout:0/337: dwrite d17/d2c/f60 [0,4194304] 0 2022-01-31T19:44:14.195 INFO:tasks.workunit.client.0.smithi167.stdout:0/338: symlink d17/d19/d1c/d39/d44/l6a 0 2022-01-31T19:44:14.195 INFO:tasks.workunit.client.0.smithi167.stdout:0/339: write d17/d19/d1c/d39/d44/f65 [945790,11447] 0 2022-01-31T19:44:14.212 INFO:tasks.workunit.client.0.smithi167.stdout:0/340: dwrite f10 [0,4194304] 0 2022-01-31T19:44:14.213 INFO:tasks.workunit.client.0.smithi167.stdout:0/341: fsync d17/d33/f5c 0 2022-01-31T19:44:14.213 INFO:tasks.workunit.client.0.smithi167.stdout:0/342: fsync d17/d2c/f60 0 2022-01-31T19:44:14.329 INFO:tasks.workunit.client.0.smithi167.stdout:2/452: sync 2022-01-31T19:44:14.329 INFO:tasks.workunit.client.0.smithi167.stdout:1/696: sync 2022-01-31T19:44:14.329 INFO:tasks.workunit.client.0.smithi167.stdout:8/599: sync 2022-01-31T19:44:14.330 INFO:tasks.workunit.client.0.smithi167.stdout:1/697: chown d0/d10/d20/cc9 505924577 1 2022-01-31T19:44:14.330 INFO:tasks.workunit.client.0.smithi167.stdout:2/453: creat dd/df/d1b/d3d/d4a/d4b/d54/f8b x:0 0 0 2022-01-31T19:44:14.331 INFO:tasks.workunit.client.0.smithi167.stdout:8/600: mkdir d6/db/de/d18/da4/dc2/dc3 0 2022-01-31T19:44:14.331 INFO:tasks.workunit.client.0.smithi167.stdout:8/601: chown d6/db/d6a/fb9 7618094 1 2022-01-31T19:44:14.332 INFO:tasks.workunit.client.0.smithi167.stdout:8/602: write d6/db/d28/d61/f89 [45221,1576] 0 2022-01-31T19:44:14.332 INFO:tasks.workunit.client.0.smithi167.stdout:1/698: mknod d0/d10/cea 0 2022-01-31T19:44:14.332 INFO:tasks.workunit.client.0.smithi167.stdout:1/699: readlink d0/d10/d20/l40 0 2022-01-31T19:44:14.333 INFO:tasks.workunit.client.0.smithi167.stdout:2/454: truncate dd/f15 1905034 0 2022-01-31T19:44:14.333 INFO:tasks.workunit.client.0.smithi167.stdout:8/603: mkdir d6/db/d28/daf/dc4 0 2022-01-31T19:44:14.334 INFO:tasks.workunit.client.0.smithi167.stdout:1/700: creat d0/db3/feb x:0 0 0 2022-01-31T19:44:14.334 INFO:tasks.workunit.client.0.smithi167.stdout:1/701: chown d0/d62/lbe 15409 1 2022-01-31T19:44:14.334 INFO:tasks.workunit.client.0.smithi167.stdout:8/604: write d6/d1e/d4c/f8b [7190986,53208] 0 2022-01-31T19:44:14.335 INFO:tasks.workunit.client.0.smithi167.stdout:2/455: write dd/d18/d62/f89 [101255,68603] 0 2022-01-31T19:44:14.336 INFO:tasks.workunit.client.0.smithi167.stdout:7/770: sync 2022-01-31T19:44:14.337 INFO:tasks.workunit.client.0.smithi167.stdout:3/524: sync 2022-01-31T19:44:14.337 INFO:tasks.workunit.client.0.smithi167.stdout:6/481: sync 2022-01-31T19:44:14.337 INFO:tasks.workunit.client.0.smithi167.stdout:3/525: dread - d9/d7f/f85 zero size 2022-01-31T19:44:14.338 INFO:tasks.workunit.client.0.smithi167.stdout:9/540: sync 2022-01-31T19:44:14.338 INFO:tasks.workunit.client.0.smithi167.stdout:4/589: sync 2022-01-31T19:44:14.339 INFO:tasks.workunit.client.0.smithi167.stdout:1/702: mkdir d0/d75/dec 0 2022-01-31T19:44:14.341 INFO:tasks.workunit.client.0.smithi167.stdout:8/605: creat d6/db/de/d18/da4/dc2/dc3/fc5 x:0 0 0 2022-01-31T19:44:14.341 INFO:tasks.workunit.client.0.smithi167.stdout:8/606: readlink d6/db/d6a/d8d/d9e/d60/la8 0 2022-01-31T19:44:14.343 INFO:tasks.workunit.client.0.smithi167.stdout:2/456: mknod dd/df/d1b/d3d/d4a/c8c 0 2022-01-31T19:44:14.345 INFO:tasks.workunit.client.0.smithi167.stdout:7/771: rename d8/d2c/d6e/d4d/d5e/d71/db1/l105 to d8/d2c/d6e/d9a/dd8/l11a 0 2022-01-31T19:44:14.347 INFO:tasks.workunit.client.0.smithi167.stdout:3/526: mkdir d9/d29/d3b/d9b 0 2022-01-31T19:44:14.348 INFO:tasks.workunit.client.0.smithi167.stdout:9/541: truncate dd/d19/f21 3476323 0 2022-01-31T19:44:14.349 INFO:tasks.workunit.client.0.smithi167.stdout:4/590: symlink d0/d6/d7/d39/d4e/d61/lc0 0 2022-01-31T19:44:14.352 INFO:tasks.workunit.client.0.smithi167.stdout:8/607: creat d6/db/d36/d5d/fc6 x:0 0 0 2022-01-31T19:44:14.352 INFO:tasks.workunit.client.0.smithi167.stdout:7/772: mknod d8/d35/d119/c11b 0 2022-01-31T19:44:14.352 INFO:tasks.workunit.client.0.smithi167.stdout:4/591: dread d0/d6/f58 [0,4194304] 0 2022-01-31T19:44:14.354 INFO:tasks.workunit.client.0.smithi167.stdout:3/527: rename d9/d56/c8e to d9/d29/d3b/c9c 0 2022-01-31T19:44:14.354 INFO:tasks.workunit.client.0.smithi167.stdout:3/528: chown d9/d21/f23 37 1 2022-01-31T19:44:14.354 INFO:tasks.workunit.client.0.smithi167.stdout:3/529: chown d9/d56/c92 175 1 2022-01-31T19:44:14.355 INFO:tasks.workunit.client.0.smithi167.stdout:9/542: mknod dd/d17/cb7 0 2022-01-31T19:44:14.355 INFO:tasks.workunit.client.0.smithi167.stdout:9/543: dread - dd/d19/d24/d56/d5b/f95 zero size 2022-01-31T19:44:14.356 INFO:tasks.workunit.client.0.smithi167.stdout:8/608: mkdir d6/db/de/d18/da4/dc2/dc3/dc7 0 2022-01-31T19:44:14.357 INFO:tasks.workunit.client.0.smithi167.stdout:7/773: creat d8/d2c/d43/d94/f11c x:0 0 0 2022-01-31T19:44:14.357 INFO:tasks.workunit.client.0.smithi167.stdout:7/774: write d8/d2c/fda [1597220,100113] 0 2022-01-31T19:44:14.358 INFO:tasks.workunit.client.0.smithi167.stdout:7/775: readlink d8/l13 0 2022-01-31T19:44:14.359 INFO:tasks.workunit.client.0.smithi167.stdout:3/530: unlink d9/l74 0 2022-01-31T19:44:14.361 INFO:tasks.workunit.client.0.smithi167.stdout:8/609: getdents d6/d1e/d4c 0 2022-01-31T19:44:14.363 INFO:tasks.workunit.client.0.smithi167.stdout:7/776: link d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/daa/lb7 d8/d2c/d7a/d99/de2/dc3/l11d 0 2022-01-31T19:44:14.363 INFO:tasks.workunit.client.0.smithi167.stdout:7/777: fdatasync d8/d2c/d6e/ff9 0 2022-01-31T19:44:14.363 INFO:tasks.workunit.client.0.smithi167.stdout:7/778: fsync d8/d2c/d43/da1/fc0 0 2022-01-31T19:44:14.364 INFO:tasks.workunit.client.0.smithi167.stdout:7/779: stat d8/d2c/d7a/d99/de2 0 2022-01-31T19:44:14.365 INFO:tasks.workunit.client.0.smithi167.stdout:4/592: mknod d0/d6/d7/d39/cc1 0 2022-01-31T19:44:14.365 INFO:tasks.workunit.client.0.smithi167.stdout:4/593: chown d0/d3f/d1a/c6e 254 1 2022-01-31T19:44:14.365 INFO:tasks.workunit.client.0.smithi167.stdout:3/531: mkdir d9/d29/d9d 0 2022-01-31T19:44:14.366 INFO:tasks.workunit.client.0.smithi167.stdout:3/532: fsync d9/d29/d3b/d41/d6b/f51 0 2022-01-31T19:44:14.366 INFO:tasks.workunit.client.0.smithi167.stdout:3/533: creat d9/d29/d3b/f9e x:0 0 0 2022-01-31T19:44:14.366 INFO:tasks.workunit.client.0.smithi167.stdout:3/534: stat d9/d29/l75 0 2022-01-31T19:44:14.366 INFO:tasks.workunit.client.0.smithi167.stdout:3/535: chown d9/d29/d3b/d41/c4b 0 1 2022-01-31T19:44:14.367 INFO:tasks.workunit.client.0.smithi167.stdout:9/544: rmdir dd/d19/d24/d56 39 2022-01-31T19:44:14.367 INFO:tasks.workunit.client.0.smithi167.stdout:4/594: dread d0/d6/d7/d39/d3a/f48 [0,4194304] 0 2022-01-31T19:44:14.368 INFO:tasks.workunit.client.0.smithi167.stdout:8/610: creat d6/d1e/d2d/d6b/fc8 x:0 0 0 2022-01-31T19:44:14.369 INFO:tasks.workunit.client.0.smithi167.stdout:8/611: dread - d6/db/d36/d5d/fc6 zero size 2022-01-31T19:44:14.372 INFO:tasks.workunit.client.0.smithi167.stdout:7/780: creat d8/dc/f11e x:0 0 0 2022-01-31T19:44:14.372 INFO:tasks.workunit.client.0.smithi167.stdout:7/781: write d8/d2c/d43/d94/f11c [206092,63473] 0 2022-01-31T19:44:14.372 INFO:tasks.workunit.client.0.smithi167.stdout:6/482: dwrite d8/f1f [0,4194304] 0 2022-01-31T19:44:14.374 INFO:tasks.workunit.client.0.smithi167.stdout:3/536: link d9/c12 d9/d29/d3b/c9f 0 2022-01-31T19:44:14.374 INFO:tasks.workunit.client.0.smithi167.stdout:3/537: creat d9/d29/d60/fa0 x:0 0 0 2022-01-31T19:44:14.375 INFO:tasks.workunit.client.0.smithi167.stdout:1/703: dwrite d0/d10/d5a/fb5 [0,4194304] 0 2022-01-31T19:44:14.375 INFO:tasks.workunit.client.0.smithi167.stdout:2/457: dwrite dd/df/d1b/d3d/d4a/d4b/d54/f60 [0,4194304] 0 2022-01-31T19:44:14.376 INFO:tasks.workunit.client.0.smithi167.stdout:9/545: symlink dd/d35/lb8 0 2022-01-31T19:44:14.379 INFO:tasks.workunit.client.0.smithi167.stdout:4/595: mkdir d0/d6/d7/dc2 0 2022-01-31T19:44:14.380 INFO:tasks.workunit.client.0.smithi167.stdout:8/612: creat d6/db/de/d18/da4/dc2/dc3/dc7/fc9 x:0 0 0 2022-01-31T19:44:14.381 INFO:tasks.workunit.client.0.smithi167.stdout:7/782: dread d8/dc/f40 [0,4194304] 0 2022-01-31T19:44:14.382 INFO:tasks.workunit.client.0.smithi167.stdout:6/483: symlink d8/d62/d92/laf 0 2022-01-31T19:44:14.384 INFO:tasks.workunit.client.0.smithi167.stdout:3/538: dread d9/f18 [0,4194304] 0 2022-01-31T19:44:14.385 INFO:tasks.workunit.client.0.smithi167.stdout:3/539: chown d9/d7f 2621 1 2022-01-31T19:44:14.385 INFO:tasks.workunit.client.0.smithi167.stdout:3/540: chown d9/d7f/f94 1382584 1 2022-01-31T19:44:14.389 INFO:tasks.workunit.client.0.smithi167.stdout:2/458: unlink dd/d18/d62/f89 0 2022-01-31T19:44:14.389 INFO:tasks.workunit.client.0.smithi167.stdout:2/459: readlink dd/d18/l87 0 2022-01-31T19:44:14.391 INFO:tasks.workunit.client.0.smithi167.stdout:9/546: link dd/c29 dd/d19/d24/d50/cb9 0 2022-01-31T19:44:14.391 INFO:tasks.workunit.client.0.smithi167.stdout:9/547: fdatasync dd/d35/f44 0 2022-01-31T19:44:14.391 INFO:tasks.workunit.client.0.smithi167.stdout:9/548: fdatasync dd/d17/f1f 0 2022-01-31T19:44:14.391 INFO:tasks.workunit.client.0.smithi167.stdout:9/549: dread - dd/d19/d59/fae zero size 2022-01-31T19:44:14.392 INFO:tasks.workunit.client.0.smithi167.stdout:9/550: readlink dd/d17/d80/d4c/l69 0 2022-01-31T19:44:14.401 INFO:tasks.workunit.client.0.smithi167.stdout:8/613: dwrite d6/db/de/d18/da4/fb2 [0,4194304] 0 2022-01-31T19:44:14.401 INFO:tasks.workunit.client.0.smithi167.stdout:8/614: readlink d6/db/d28/d61/lad 0 2022-01-31T19:44:14.401 INFO:tasks.workunit.client.0.smithi167.stdout:8/615: write d6/db/d36/d5d/fc6 [574796,72591] 0 2022-01-31T19:44:14.402 INFO:tasks.workunit.client.0.smithi167.stdout:7/783: truncate d8/d2c/d6e/d4d/d5e/d71/db1/f6a 2041947 0 2022-01-31T19:44:14.402 INFO:tasks.workunit.client.0.smithi167.stdout:8/616: write d6/db/d4e/d73/fa2 [40209,60917] 0 2022-01-31T19:44:14.403 INFO:tasks.workunit.client.0.smithi167.stdout:6/484: rmdir d8/d33/d51/d5d/d64/da4 0 2022-01-31T19:44:14.403 INFO:tasks.workunit.client.0.smithi167.stdout:3/541: unlink d9/c16 0 2022-01-31T19:44:14.406 INFO:tasks.workunit.client.0.smithi167.stdout:4/596: getdents d0/d6/d7/d39 0 2022-01-31T19:44:14.407 INFO:tasks.workunit.client.0.smithi167.stdout:4/597: write d0/d6/f58 [65709,117241] 0 2022-01-31T19:44:14.410 INFO:tasks.workunit.client.0.smithi167.stdout:8/617: link d6/d1e/d2d/l47 d6/db/d36/d5d/lca 0 2022-01-31T19:44:14.410 INFO:tasks.workunit.client.0.smithi167.stdout:6/485: rmdir d8/d33/d51/d5d/d86 39 2022-01-31T19:44:14.412 INFO:tasks.workunit.client.0.smithi167.stdout:3/542: rename d9/f42 to d9/d29/fa1 0 2022-01-31T19:44:14.412 INFO:tasks.workunit.client.0.smithi167.stdout:3/543: getdents d9/d29/d9d 0 2022-01-31T19:44:14.412 INFO:tasks.workunit.client.0.smithi167.stdout:3/544: write d9/d29/d60/f49 [262622,6116] 0 2022-01-31T19:44:14.413 INFO:tasks.workunit.client.0.smithi167.stdout:3/545: write d9/d29/d60/f49 [137299,114887] 0 2022-01-31T19:44:14.414 INFO:tasks.workunit.client.0.smithi167.stdout:4/598: write d0/d6/d7/d39/f43 [2022711,115965] 0 2022-01-31T19:44:14.414 INFO:tasks.workunit.client.0.smithi167.stdout:8/618: unlink d6/db/d6a/d8d/d9e/d60/fa9 0 2022-01-31T19:44:14.415 INFO:tasks.workunit.client.0.smithi167.stdout:4/599: symlink d0/d6/d7/d39/d3a/d79/lc3 0 2022-01-31T19:44:14.417 INFO:tasks.workunit.client.0.smithi167.stdout:2/460: dwrite dd/df/d3f/d51/f80 [0,4194304] 0 2022-01-31T19:44:14.417 INFO:tasks.workunit.client.0.smithi167.stdout:2/461: write dd/df/d3f/f41 [726590,43949] 0 2022-01-31T19:44:14.419 INFO:tasks.workunit.client.0.smithi167.stdout:8/619: rename d6/db/de/d18/da4/fb2 to d6/db/d4e/d73/d88/fcb 0 2022-01-31T19:44:14.422 INFO:tasks.workunit.client.0.smithi167.stdout:3/546: dread d9/d21/f77 [0,4194304] 0 2022-01-31T19:44:14.422 INFO:tasks.workunit.client.0.smithi167.stdout:3/547: chown d9/c7d 919710405 1 2022-01-31T19:44:14.422 INFO:tasks.workunit.client.0.smithi167.stdout:7/784: dread d8/d2c/f63 [0,4194304] 0 2022-01-31T19:44:14.422 INFO:tasks.workunit.client.0.smithi167.stdout:7/785: readlink d8/d2c/d6e/d4d/d67/d8f/le0 0 2022-01-31T19:44:14.425 INFO:tasks.workunit.client.0.smithi167.stdout:9/551: dwrite dd/f64 [0,4194304] 0 2022-01-31T19:44:14.425 INFO:tasks.workunit.client.0.smithi167.stdout:2/462: mkdir dd/d18/d8d 0 2022-01-31T19:44:14.425 INFO:tasks.workunit.client.0.smithi167.stdout:2/463: write dd/df/d1b/d3d/d4a/d4b/d54/f8b [680043,30300] 0 2022-01-31T19:44:14.426 INFO:tasks.workunit.client.0.smithi167.stdout:2/464: chown dd/c43 597 1 2022-01-31T19:44:14.428 INFO:tasks.workunit.client.0.smithi167.stdout:8/620: link d6/d1e/d2d/d6b/fc8 d6/db/d6a/d8d/d9e/fcc 0 2022-01-31T19:44:14.430 INFO:tasks.workunit.client.0.smithi167.stdout:3/548: rename d9/d26/l5e to d9/d29/d9d/la2 0 2022-01-31T19:44:14.431 INFO:tasks.workunit.client.0.smithi167.stdout:3/549: dread - d9/d26/f8b zero size 2022-01-31T19:44:14.431 INFO:tasks.workunit.client.0.smithi167.stdout:3/550: write d9/d56/f91 [551927,41920] 0 2022-01-31T19:44:14.433 INFO:tasks.workunit.client.0.smithi167.stdout:7/786: symlink d8/d2c/d6e/d9a/l11f 0 2022-01-31T19:44:14.435 INFO:tasks.workunit.client.0.smithi167.stdout:2/465: dread dd/df/d1b/d2f/f4d [0,4194304] 0 2022-01-31T19:44:14.435 INFO:tasks.workunit.client.0.smithi167.stdout:2/466: readlink dd/df/d1b/d3d/d4a/d4b/l5e 0 2022-01-31T19:44:14.438 INFO:tasks.workunit.client.0.smithi167.stdout:6/486: dwrite d8/d20/f44 [0,4194304] 0 2022-01-31T19:44:14.439 INFO:tasks.workunit.client.0.smithi167.stdout:4/600: dwrite d0/d6/d7/d39/f36 [0,4194304] 0 2022-01-31T19:44:14.441 INFO:tasks.workunit.client.0.smithi167.stdout:9/552: symlink dd/d19/d24/lba 0 2022-01-31T19:44:14.441 INFO:tasks.workunit.client.0.smithi167.stdout:9/553: creat dd/d17/d7f/fbb x:0 0 0 2022-01-31T19:44:14.441 INFO:tasks.workunit.client.0.smithi167.stdout:9/554: creat dd/d19/d24/d50/d5d/fbc x:0 0 0 2022-01-31T19:44:14.442 INFO:tasks.workunit.client.0.smithi167.stdout:8/621: symlink d6/d1e/d4c/lcd 0 2022-01-31T19:44:14.442 INFO:tasks.workunit.client.0.smithi167.stdout:8/622: creat d6/db/de/d18/fce x:0 0 0 2022-01-31T19:44:14.446 INFO:tasks.workunit.client.0.smithi167.stdout:3/551: mkdir d9/d29/d3b/d9b/da3 0 2022-01-31T19:44:14.446 INFO:tasks.workunit.client.0.smithi167.stdout:3/552: dread - d9/d26/f65 zero size 2022-01-31T19:44:14.446 INFO:tasks.workunit.client.0.smithi167.stdout:3/553: creat d9/d21/fa4 x:0 0 0 2022-01-31T19:44:14.447 INFO:tasks.workunit.client.0.smithi167.stdout:7/787: getdents d8/d2c/d6e/d4d/d5e/dce 0 2022-01-31T19:44:14.447 INFO:tasks.workunit.client.0.smithi167.stdout:7/788: fdatasync d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f5c 0 2022-01-31T19:44:14.456 INFO:tasks.workunit.client.0.smithi167.stdout:8/623: rename d6/d1e/d4c/d80/d99 to d6/d1e/dcf 0 2022-01-31T19:44:14.458 INFO:tasks.workunit.client.0.smithi167.stdout:3/554: getdents d9/d29/d3b/d8d 0 2022-01-31T19:44:14.458 INFO:tasks.workunit.client.0.smithi167.stdout:8/624: write d6/db/de/d18/f21 [2311519,39891] 0 2022-01-31T19:44:14.461 INFO:tasks.workunit.client.0.smithi167.stdout:2/467: link dd/d18/d62/f6e dd/df/d3f/f8e 0 2022-01-31T19:44:14.462 INFO:tasks.workunit.client.0.smithi167.stdout:6/487: rmdir d8/d20 39 2022-01-31T19:44:14.462 INFO:tasks.workunit.client.0.smithi167.stdout:6/488: chown d8/f1a 12 1 2022-01-31T19:44:14.462 INFO:tasks.workunit.client.0.smithi167.stdout:6/489: chown d8/d33/d51/d72/c9a 7 1 2022-01-31T19:44:14.463 INFO:tasks.workunit.client.0.smithi167.stdout:9/555: rmdir dd/d6b 39 2022-01-31T19:44:14.465 INFO:tasks.workunit.client.0.smithi167.stdout:8/625: creat d6/db/d4e/fd0 x:0 0 0 2022-01-31T19:44:14.465 INFO:tasks.workunit.client.0.smithi167.stdout:8/626: fsync d6/d1e/d2d/fbe 0 2022-01-31T19:44:14.466 INFO:tasks.workunit.client.0.smithi167.stdout:2/468: symlink dd/df/d3f/l8f 0 2022-01-31T19:44:14.467 INFO:tasks.workunit.client.0.smithi167.stdout:2/469: write dd/f23 [1769980,119398] 0 2022-01-31T19:44:14.467 INFO:tasks.workunit.client.0.smithi167.stdout:8/627: write d6/db/fd [626376,115981] 0 2022-01-31T19:44:14.468 INFO:tasks.workunit.client.0.smithi167.stdout:6/490: link d8/d33/d35/c58 d8/d33/d51/d5d/d86/d94/cb0 0 2022-01-31T19:44:14.468 INFO:tasks.workunit.client.0.smithi167.stdout:6/491: rename d8/d33/d51/d5d to d8/d33/d51/d5d/db1 22 2022-01-31T19:44:14.468 INFO:tasks.workunit.client.0.smithi167.stdout:2/470: symlink dd/df/d22/l90 0 2022-01-31T19:44:14.469 INFO:tasks.workunit.client.0.smithi167.stdout:8/628: symlink d6/db/d28/d66/ld1 0 2022-01-31T19:44:14.469 INFO:tasks.workunit.client.0.smithi167.stdout:8/629: fsync d6/db/d28/f43 0 2022-01-31T19:44:14.471 INFO:tasks.workunit.client.0.smithi167.stdout:6/492: rmdir d8/d33/d51/d6a 39 2022-01-31T19:44:14.472 INFO:tasks.workunit.client.0.smithi167.stdout:6/493: dread d8/d20/f22 [4194304,4194304] 0 2022-01-31T19:44:14.472 INFO:tasks.workunit.client.0.smithi167.stdout:6/494: chown d8/d33/l37 0 1 2022-01-31T19:44:14.472 INFO:tasks.workunit.client.0.smithi167.stdout:3/555: dread d9/d29/d60/f58 [0,4194304] 0 2022-01-31T19:44:14.473 INFO:tasks.workunit.client.0.smithi167.stdout:7/789: dwrite d8/d2c/d43/d94/f10b [4194304,4194304] 0 2022-01-31T19:44:14.473 INFO:tasks.workunit.client.0.smithi167.stdout:3/556: creat d9/d29/d3b/d41/d6b/d89/fa5 x:0 0 0 2022-01-31T19:44:14.474 INFO:tasks.workunit.client.0.smithi167.stdout:4/601: dwrite d0/d6/d7/f18 [4194304,4194304] 0 2022-01-31T19:44:14.474 INFO:tasks.workunit.client.0.smithi167.stdout:6/495: creat d8/d33/d4f/d87/fb2 x:0 0 0 2022-01-31T19:44:14.479 INFO:tasks.workunit.client.0.smithi167.stdout:4/602: truncate d0/d6/d7/d39/d3a/f54 1783564 0 2022-01-31T19:44:14.480 INFO:tasks.workunit.client.0.smithi167.stdout:6/496: symlink d8/d33/d51/d7e/lb3 0 2022-01-31T19:44:14.480 INFO:tasks.workunit.client.0.smithi167.stdout:7/790: read d8/dc/d18/d24/f29 [2581370,66980] 0 2022-01-31T19:44:14.481 INFO:tasks.workunit.client.0.smithi167.stdout:4/603: creat d0/d3f/da2/fc4 x:0 0 0 2022-01-31T19:44:14.482 INFO:tasks.workunit.client.0.smithi167.stdout:3/557: dread d9/d56/f6d [0,4194304] 0 2022-01-31T19:44:14.483 INFO:tasks.workunit.client.0.smithi167.stdout:3/558: stat d9/d29/f4c 0 2022-01-31T19:44:14.483 INFO:tasks.workunit.client.0.smithi167.stdout:6/497: mkdir d8/d33/d51/d5d/d86/d94/db4 0 2022-01-31T19:44:14.484 INFO:tasks.workunit.client.0.smithi167.stdout:5/690: sync 2022-01-31T19:44:14.484 INFO:tasks.workunit.client.0.smithi167.stdout:5/691: truncate d5/d6/de/d22/d4d/f85 1386132 0 2022-01-31T19:44:14.485 INFO:tasks.workunit.client.0.smithi167.stdout:9/556: dwrite dd/d17/d80/f54 [0,4194304] 0 2022-01-31T19:44:14.486 INFO:tasks.workunit.client.0.smithi167.stdout:9/557: chown dd/d19/d24/d56/d5b 29018964 1 2022-01-31T19:44:14.486 INFO:tasks.workunit.client.0.smithi167.stdout:9/558: fdatasync dd/d35/f65 0 2022-01-31T19:44:14.486 INFO:tasks.workunit.client.0.smithi167.stdout:9/559: truncate dd/fa3 577156 0 2022-01-31T19:44:14.487 INFO:tasks.workunit.client.0.smithi167.stdout:7/791: rename d8/d2c/d6e/d4d/d5e/d71/db1/d4f/l7b to d8/d2c/d6e/d4d/d67/d8f/dcb/l120 0 2022-01-31T19:44:14.487 INFO:tasks.workunit.client.0.smithi167.stdout:4/604: creat d0/d6/dab/fc5 x:0 0 0 2022-01-31T19:44:14.488 INFO:tasks.workunit.client.0.smithi167.stdout:4/605: chown d0/d3f/d60/d65 4 1 2022-01-31T19:44:14.488 INFO:tasks.workunit.client.0.smithi167.stdout:4/606: write d0/d6/d7/d26/d40/d4c/f4f [479348,116860] 0 2022-01-31T19:44:14.488 INFO:tasks.workunit.client.0.smithi167.stdout:3/559: mknod d9/d29/d3b/d9b/ca6 0 2022-01-31T19:44:14.489 INFO:tasks.workunit.client.0.smithi167.stdout:3/560: stat d9/d29/f52 0 2022-01-31T19:44:14.489 INFO:tasks.workunit.client.0.smithi167.stdout:6/498: mknod d8/d33/d51/d72/cb5 0 2022-01-31T19:44:14.490 INFO:tasks.workunit.client.0.smithi167.stdout:5/692: creat d5/d6/d16/d18/d43/fd4 x:0 0 0 2022-01-31T19:44:14.494 INFO:tasks.workunit.client.0.smithi167.stdout:9/560: rmdir dd/d6b/d72 39 2022-01-31T19:44:14.494 INFO:tasks.workunit.client.0.smithi167.stdout:9/561: fdatasync dd/d19/d24/d56/f86 0 2022-01-31T19:44:14.495 INFO:tasks.workunit.client.0.smithi167.stdout:9/562: chown dd/d34/l39 448482 1 2022-01-31T19:44:14.496 INFO:tasks.workunit.client.0.smithi167.stdout:9/563: write dd/d34/f61 [2818223,35203] 0 2022-01-31T19:44:14.496 INFO:tasks.workunit.client.0.smithi167.stdout:9/564: creat dd/d17/d80/d4c/fbd x:0 0 0 2022-01-31T19:44:14.497 INFO:tasks.workunit.client.0.smithi167.stdout:7/792: mkdir d8/d2c/d43/d94/daf/dee/df8/d121 0 2022-01-31T19:44:14.497 INFO:tasks.workunit.client.0.smithi167.stdout:7/793: getdents d8/dc/d18/d24/dbb/d9e 0 2022-01-31T19:44:14.497 INFO:tasks.workunit.client.0.smithi167.stdout:7/794: fsync d8/d2c/d6e/d4d/d6b/f88 0 2022-01-31T19:44:14.497 INFO:tasks.workunit.client.0.smithi167.stdout:8/630: dwrite d6/db/d28/d61/f89 [0,4194304] 0 2022-01-31T19:44:14.498 INFO:tasks.workunit.client.0.smithi167.stdout:8/631: readlink d6/d1e/d2d/d6b/l91 0 2022-01-31T19:44:14.498 INFO:tasks.workunit.client.0.smithi167.stdout:4/607: creat d0/d6/fc6 x:0 0 0 2022-01-31T19:44:14.498 INFO:tasks.workunit.client.0.smithi167.stdout:2/471: dwrite f8 [4194304,4194304] 0 2022-01-31T19:44:14.499 INFO:tasks.workunit.client.0.smithi167.stdout:3/561: creat d9/d29/d9d/fa7 x:0 0 0 2022-01-31T19:44:14.502 INFO:tasks.workunit.client.0.smithi167.stdout:6/499: creat d8/d33/d51/d5d/d86/fb6 x:0 0 0 2022-01-31T19:44:14.503 INFO:tasks.workunit.client.0.smithi167.stdout:6/500: write d8/f7a [1311779,69628] 0 2022-01-31T19:44:14.508 INFO:tasks.workunit.client.0.smithi167.stdout:9/565: link dd/d17/d7f/fbb dd/d19/d24/d56/d5b/fbe 0 2022-01-31T19:44:14.510 INFO:tasks.workunit.client.0.smithi167.stdout:8/632: creat d6/db/fd2 x:0 0 0 2022-01-31T19:44:14.511 INFO:tasks.workunit.client.0.smithi167.stdout:4/608: creat d0/d6/d7/d26/d31/fc7 x:0 0 0 2022-01-31T19:44:14.512 INFO:tasks.workunit.client.0.smithi167.stdout:6/501: creat d8/fb7 x:0 0 0 2022-01-31T19:44:14.513 INFO:tasks.workunit.client.0.smithi167.stdout:6/502: creat d8/d62/d92/fb8 x:0 0 0 2022-01-31T19:44:14.514 INFO:tasks.workunit.client.0.smithi167.stdout:5/693: rename d5/d6/d16/d18/d5b/d63/d88 to d5/d6/d16/d18/d43/d45/dd5 0 2022-01-31T19:44:14.514 INFO:tasks.workunit.client.0.smithi167.stdout:9/566: dread dd/d17/f18 [0,4194304] 0 2022-01-31T19:44:14.515 INFO:tasks.workunit.client.0.smithi167.stdout:4/609: mkdir d0/d6/d7/d39/d3a/dc8 0 2022-01-31T19:44:14.517 INFO:tasks.workunit.client.0.smithi167.stdout:6/503: mkdir d8/d33/d51/d5d/d64/dab/db9 0 2022-01-31T19:44:14.518 INFO:tasks.workunit.client.0.smithi167.stdout:4/610: dread d0/d6/d7/d39/d3a/f54 [0,4194304] 0 2022-01-31T19:44:14.518 INFO:tasks.workunit.client.0.smithi167.stdout:4/611: stat d0/d6/d7/d39/d4e/d61 0 2022-01-31T19:44:14.519 INFO:tasks.workunit.client.0.smithi167.stdout:7/795: rename d8/dc/d18/fca to d8/dc/d18/d24/dcc/f122 0 2022-01-31T19:44:14.519 INFO:tasks.workunit.client.0.smithi167.stdout:7/796: write d8/d2c/d7a/fcd [359336,19643] 0 2022-01-31T19:44:14.519 INFO:tasks.workunit.client.0.smithi167.stdout:5/694: creat d5/d6/d16/d18/d43/d45/d69/dc4/fd6 x:0 0 0 2022-01-31T19:44:14.520 INFO:tasks.workunit.client.0.smithi167.stdout:9/567: mkdir dd/d17/d7f/dbf 0 2022-01-31T19:44:14.521 INFO:tasks.workunit.client.0.smithi167.stdout:9/568: chown dd/d17/d80/f54 42524 1 2022-01-31T19:44:14.521 INFO:tasks.workunit.client.0.smithi167.stdout:6/504: getdents d8/d33/d4f 0 2022-01-31T19:44:14.522 INFO:tasks.workunit.client.0.smithi167.stdout:8/633: rename d6/db/d6a/d8d/d9e/f67 to d6/d1e/fd3 0 2022-01-31T19:44:14.522 INFO:tasks.workunit.client.0.smithi167.stdout:8/634: readlink d6/d1e/dcf/lb1 0 2022-01-31T19:44:14.523 INFO:tasks.workunit.client.0.smithi167.stdout:7/797: creat d8/d2c/d6e/d4d/d67/d8f/dcb/f123 x:0 0 0 2022-01-31T19:44:14.524 INFO:tasks.workunit.client.0.smithi167.stdout:5/695: read d5/d6/de/d22/d76/db0/fb6 [1056715,37916] 0 2022-01-31T19:44:14.525 INFO:tasks.workunit.client.0.smithi167.stdout:9/569: mknod dd/d17/d80/d4c/db3/cc0 0 2022-01-31T19:44:14.525 INFO:tasks.workunit.client.0.smithi167.stdout:3/562: dwrite d9/d29/f52 [0,4194304] 0 2022-01-31T19:44:14.526 INFO:tasks.workunit.client.0.smithi167.stdout:2/472: dwrite dd/df/d22/f77 [4194304,4194304] 0 2022-01-31T19:44:14.526 INFO:tasks.workunit.client.0.smithi167.stdout:3/563: write d9/d29/f4c [1333461,20139] 0 2022-01-31T19:44:14.526 INFO:tasks.workunit.client.0.smithi167.stdout:6/505: creat d8/d33/d51/d5d/d86/d94/fba x:0 0 0 2022-01-31T19:44:14.531 INFO:tasks.workunit.client.0.smithi167.stdout:4/612: rename d0/d6/f3b to d0/d6/d7/d26/fc9 0 2022-01-31T19:44:14.531 INFO:tasks.workunit.client.0.smithi167.stdout:4/613: creat d0/d3f/d1a/fca x:0 0 0 2022-01-31T19:44:14.532 INFO:tasks.workunit.client.0.smithi167.stdout:7/798: truncate d8/dc/f1c 2710021 0 2022-01-31T19:44:14.534 INFO:tasks.workunit.client.0.smithi167.stdout:5/696: dread d5/d6/d16/d18/d43/d45/d69/dc4/f7c [4194304,4194304] 0 2022-01-31T19:44:14.534 INFO:tasks.workunit.client.0.smithi167.stdout:9/570: mkdir dd/d6b/dc1 0 2022-01-31T19:44:14.534 INFO:tasks.workunit.client.0.smithi167.stdout:5/697: fdatasync d5/fb9 0 2022-01-31T19:44:14.535 INFO:tasks.workunit.client.0.smithi167.stdout:9/571: write dd/d35/f3a [1410195,130390] 0 2022-01-31T19:44:14.535 INFO:tasks.workunit.client.0.smithi167.stdout:9/572: fsync dd/fa3 0 2022-01-31T19:44:14.537 INFO:tasks.workunit.client.0.smithi167.stdout:4/614: dread d0/d6/d7/d89/fa3 [0,4194304] 0 2022-01-31T19:44:14.538 INFO:tasks.workunit.client.0.smithi167.stdout:4/615: read d0/d6/f58 [166349,84912] 0 2022-01-31T19:44:14.538 INFO:tasks.workunit.client.0.smithi167.stdout:4/616: fdatasync d0/d6/d7/d39/d3a/f54 0 2022-01-31T19:44:14.538 INFO:tasks.workunit.client.0.smithi167.stdout:4/617: dread - d0/d5d/fa5 zero size 2022-01-31T19:44:14.550 INFO:tasks.workunit.client.0.smithi167.stdout:8/635: dwrite d6/d1e/d2d/fbe [0,4194304] 0 2022-01-31T19:44:14.551 INFO:tasks.workunit.client.0.smithi167.stdout:2/473: creat dd/d18/d62/f91 x:0 0 0 2022-01-31T19:44:14.552 INFO:tasks.workunit.client.0.smithi167.stdout:2/474: truncate dd/d18/f4e 1683047 0 2022-01-31T19:44:14.552 INFO:tasks.workunit.client.0.smithi167.stdout:2/475: creat dd/df/d3f/d51/f92 x:0 0 0 2022-01-31T19:44:14.552 INFO:tasks.workunit.client.0.smithi167.stdout:2/476: creat dd/df/d1b/d3d/d4a/d4b/f93 x:0 0 0 2022-01-31T19:44:14.553 INFO:tasks.workunit.client.0.smithi167.stdout:6/506: mknod d8/d20/d54/cbb 0 2022-01-31T19:44:14.553 INFO:tasks.workunit.client.0.smithi167.stdout:5/698: mknod d5/dcd/cd7 0 2022-01-31T19:44:14.556 INFO:tasks.workunit.client.0.smithi167.stdout:9/573: getdents dd/d35 0 2022-01-31T19:44:14.556 INFO:tasks.workunit.client.0.smithi167.stdout:9/574: readlink dd/d19/d24/d56/d5b/da5/lb0 0 2022-01-31T19:44:14.558 INFO:tasks.workunit.client.0.smithi167.stdout:3/564: dwrite d9/d29/f76 [0,4194304] 0 2022-01-31T19:44:14.559 INFO:tasks.workunit.client.0.smithi167.stdout:3/565: truncate d9/f81 822711 0 2022-01-31T19:44:14.562 INFO:tasks.workunit.client.0.smithi167.stdout:0/343: sync 2022-01-31T19:44:14.562 INFO:tasks.workunit.client.0.smithi167.stdout:1/704: sync 2022-01-31T19:44:14.562 INFO:tasks.workunit.client.0.smithi167.stdout:9/575: read dd/d17/f3e [868612,106061] 0 2022-01-31T19:44:14.563 INFO:tasks.workunit.client.0.smithi167.stdout:9/576: chown dd/d19/d24/d56/d5b/da5 2793203 1 2022-01-31T19:44:14.563 INFO:tasks.workunit.client.0.smithi167.stdout:7/799: dwrite d8/d35/f4b [0,4194304] 0 2022-01-31T19:44:14.563 INFO:tasks.workunit.client.0.smithi167.stdout:4/618: dwrite d0/d6/d7/d26/d31/f5e [0,4194304] 0 2022-01-31T19:44:14.564 INFO:tasks.workunit.client.0.smithi167.stdout:0/344: read d17/d3e/f42 [3524853,28420] 0 2022-01-31T19:44:14.569 INFO:tasks.workunit.client.0.smithi167.stdout:8/636: rename d6/f14 to d6/d1e/d4c/dba/fd4 0 2022-01-31T19:44:14.569 INFO:tasks.workunit.client.0.smithi167.stdout:2/477: truncate dd/df/d3f/d51/f80 3799923 0 2022-01-31T19:44:14.570 INFO:tasks.workunit.client.0.smithi167.stdout:8/637: read d6/db/d4e/d73/fa2 [215759,53566] 0 2022-01-31T19:44:14.571 INFO:tasks.workunit.client.0.smithi167.stdout:3/566: mkdir d9/d7f/da8 0 2022-01-31T19:44:14.574 INFO:tasks.workunit.client.0.smithi167.stdout:0/345: unlink d17/d19/d1c/d39/d44/l62 0 2022-01-31T19:44:14.574 INFO:tasks.workunit.client.0.smithi167.stdout:0/346: write d17/d33/f5c [504257,13997] 0 2022-01-31T19:44:14.577 INFO:tasks.workunit.client.0.smithi167.stdout:6/507: rename d8/fb to d8/d33/d51/d5d/d86/d94/fbc 0 2022-01-31T19:44:14.577 INFO:tasks.workunit.client.0.smithi167.stdout:1/705: dread d0/d10/d5a/d33/d4b/d1f/fa6 [0,4194304] 0 2022-01-31T19:44:14.577 INFO:tasks.workunit.client.0.smithi167.stdout:1/706: chown d0/d75/dec 918596596 1 2022-01-31T19:44:14.578 INFO:tasks.workunit.client.0.smithi167.stdout:3/567: mknod d9/d29/d3b/d9b/da3/ca9 0 2022-01-31T19:44:14.578 INFO:tasks.workunit.client.0.smithi167.stdout:0/347: mknod d17/d3e/c6b 0 2022-01-31T19:44:14.581 INFO:tasks.workunit.client.0.smithi167.stdout:6/508: mkdir d8/d33/d51/d5d/d86/dbd 0 2022-01-31T19:44:14.582 INFO:tasks.workunit.client.0.smithi167.stdout:5/699: dwrite d5/d6/d16/f26 [0,4194304] 0 2022-01-31T19:44:14.583 INFO:tasks.workunit.client.0.smithi167.stdout:8/638: dread d6/d1e/f22 [4194304,4194304] 0 2022-01-31T19:44:14.583 INFO:tasks.workunit.client.0.smithi167.stdout:1/707: creat d0/d75/dbd/fed x:0 0 0 2022-01-31T19:44:14.586 INFO:tasks.workunit.client.0.smithi167.stdout:3/568: rename d9/d7f/f94 to d9/d7f/faa 0 2022-01-31T19:44:14.590 INFO:tasks.workunit.client.0.smithi167.stdout:7/800: dwrite d8/dc/f33 [0,4194304] 0 2022-01-31T19:44:14.591 INFO:tasks.workunit.client.0.smithi167.stdout:5/700: mkdir d5/d6/de/d22/d76/db0/dd8 0 2022-01-31T19:44:14.591 INFO:tasks.workunit.client.0.smithi167.stdout:5/701: fsync d5/d6/de/d22/d76/f80 0 2022-01-31T19:44:14.591 INFO:tasks.workunit.client.0.smithi167.stdout:5/702: unlink d5/d6/d16/d18/l67 0 2022-01-31T19:44:14.593 INFO:tasks.workunit.client.0.smithi167.stdout:8/639: unlink d6/db/d6a/d8d/d9e/l7e 0 2022-01-31T19:44:14.595 INFO:tasks.workunit.client.0.smithi167.stdout:1/708: creat d0/d10/d5a/d33/d4b/d1f/fee x:0 0 0 2022-01-31T19:44:14.596 INFO:tasks.workunit.client.0.smithi167.stdout:1/709: truncate d0/d10/d5a/d33/d4b/d1f/f78 4437194 0 2022-01-31T19:44:14.596 INFO:tasks.workunit.client.0.smithi167.stdout:0/348: dwrite d17/d19/f4d [0,4194304] 0 2022-01-31T19:44:14.597 INFO:tasks.workunit.client.0.smithi167.stdout:0/349: fdatasync fa 0 2022-01-31T19:44:14.597 INFO:tasks.workunit.client.0.smithi167.stdout:1/710: write d0/d10/d5a/fb9 [1637504,24029] 0 2022-01-31T19:44:14.598 INFO:tasks.workunit.client.0.smithi167.stdout:3/569: truncate d9/d29/fa1 835412 0 2022-01-31T19:44:14.599 INFO:tasks.workunit.client.0.smithi167.stdout:6/509: rename d8/d33/f82 to d8/fbe 0 2022-01-31T19:44:14.600 INFO:tasks.workunit.client.0.smithi167.stdout:3/570: write d9/d29/f76 [1464703,128069] 0 2022-01-31T19:44:14.600 INFO:tasks.workunit.client.0.smithi167.stdout:3/571: creat d9/d56/fab x:0 0 0 2022-01-31T19:44:14.602 INFO:tasks.workunit.client.0.smithi167.stdout:7/801: mkdir d8/dc/dc1/d124 0 2022-01-31T19:44:14.605 INFO:tasks.workunit.client.0.smithi167.stdout:0/350: dread d17/d33/f36 [0,4194304] 0 2022-01-31T19:44:14.606 INFO:tasks.workunit.client.0.smithi167.stdout:0/351: chown d17/d19/d1c/d39/d44/d49/c4b 96741038 1 2022-01-31T19:44:14.606 INFO:tasks.workunit.client.0.smithi167.stdout:5/703: creat d5/d6/de/d22/d76/fd9 x:0 0 0 2022-01-31T19:44:14.611 INFO:tasks.workunit.client.0.smithi167.stdout:0/352: dread d17/d3e/f42 [0,4194304] 0 2022-01-31T19:44:14.618 INFO:tasks.workunit.client.0.smithi167.stdout:1/711: unlink d0/db3/le3 0 2022-01-31T19:44:14.621 INFO:tasks.workunit.client.0.smithi167.stdout:6/510: unlink d8/f3f 0 2022-01-31T19:44:14.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/572: link d9/d56/f7c d9/d29/d60/fac 0 2022-01-31T19:44:14.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/573: chown d9/d26/f55 10404598 1 2022-01-31T19:44:14.625 INFO:tasks.workunit.client.0.smithi167.stdout:5/704: symlink d5/d6/de/d22/d76/lda 0 2022-01-31T19:44:14.625 INFO:tasks.workunit.client.0.smithi167.stdout:5/705: write d5/d6/de/d22/d4d/f85 [2123257,93325] 0 2022-01-31T19:44:14.629 INFO:tasks.workunit.client.0.smithi167.stdout:1/712: link d0/d10/d20/f28 d0/d62/d7f/d82/fef 0 2022-01-31T19:44:14.629 INFO:tasks.workunit.client.0.smithi167.stdout:0/353: dwrite d17/d19/f51 [0,4194304] 0 2022-01-31T19:44:14.630 INFO:tasks.workunit.client.0.smithi167.stdout:6/511: truncate d8/d33/d51/d5d/d86/d94/fbc 2433484 0 2022-01-31T19:44:14.631 INFO:tasks.workunit.client.0.smithi167.stdout:7/802: symlink d8/d2c/l125 0 2022-01-31T19:44:14.631 INFO:tasks.workunit.client.0.smithi167.stdout:5/706: mknod d5/d6/d16/cdb 0 2022-01-31T19:44:14.632 INFO:tasks.workunit.client.0.smithi167.stdout:5/707: chown d5/d6/d16/d18 45894 1 2022-01-31T19:44:14.632 INFO:tasks.workunit.client.0.smithi167.stdout:1/713: symlink d0/d62/d7f/lf0 0 2022-01-31T19:44:14.634 INFO:tasks.workunit.client.0.smithi167.stdout:8/640: rename d6/db/d2f to d6/db/d28/d61/dd5 0 2022-01-31T19:44:14.635 INFO:tasks.workunit.client.0.smithi167.stdout:5/708: dread d5/d6/d16/d18/f59 [0,4194304] 0 2022-01-31T19:44:14.635 INFO:tasks.workunit.client.0.smithi167.stdout:5/709: stat d5/d6/de/d22/d76/f80 0 2022-01-31T19:44:14.635 INFO:tasks.workunit.client.0.smithi167.stdout:8/641: write d6/db/d28/f5a [3468708,127005] 0 2022-01-31T19:44:14.637 INFO:tasks.workunit.client.0.smithi167.stdout:5/710: write d5/d6/d3a/f17 [442402,49228] 0 2022-01-31T19:44:14.643 INFO:tasks.workunit.client.0.smithi167.stdout:3/574: dwrite d9/f27 [0,4194304] 0 2022-01-31T19:44:14.643 INFO:tasks.workunit.client.0.smithi167.stdout:3/575: chown d9/d29/d60/l45 256 1 2022-01-31T19:44:14.645 INFO:tasks.workunit.client.0.smithi167.stdout:7/803: unlink d8/dc/d18/d24/dbb/cb6 0 2022-01-31T19:44:14.647 INFO:tasks.workunit.client.0.smithi167.stdout:1/714: rename d0/d10/d5a/d33/d4b/da5/c77 to d0/d10/d20/cf1 0 2022-01-31T19:44:14.652 INFO:tasks.workunit.client.0.smithi167.stdout:5/711: creat d5/d6/d16/d18/d5b/d9f/fdc x:0 0 0 2022-01-31T19:44:14.652 INFO:tasks.workunit.client.0.smithi167.stdout:0/354: getdents d17/d19/d1c/d39/d44 0 2022-01-31T19:44:14.653 INFO:tasks.workunit.client.0.smithi167.stdout:5/712: read d5/d61/f4b [3242099,46025] 0 2022-01-31T19:44:14.653 INFO:tasks.workunit.client.0.smithi167.stdout:7/804: read d8/d2c/d6e/d4d/d6b/f81 [383933,112279] 0 2022-01-31T19:44:14.653 INFO:tasks.workunit.client.0.smithi167.stdout:3/576: dread d9/d29/d3b/d41/f5c [0,4194304] 0 2022-01-31T19:44:14.654 INFO:tasks.workunit.client.0.smithi167.stdout:7/805: truncate d8/d35/f36 1966005 0 2022-01-31T19:44:14.654 INFO:tasks.workunit.client.0.smithi167.stdout:6/512: getdents d8/d33/d51/d7e 0 2022-01-31T19:44:14.657 INFO:tasks.workunit.client.0.smithi167.stdout:1/715: mkdir d0/d10/d5a/df2 0 2022-01-31T19:44:14.658 INFO:tasks.workunit.client.0.smithi167.stdout:8/642: rmdir d6/d1e/d2d 39 2022-01-31T19:44:14.658 INFO:tasks.workunit.client.0.smithi167.stdout:8/643: chown d6/db/d6a/d8d/d9e/d60/f6c 3540 1 2022-01-31T19:44:14.660 INFO:tasks.workunit.client.0.smithi167.stdout:0/355: symlink d17/l6c 0 2022-01-31T19:44:14.661 INFO:tasks.workunit.client.0.smithi167.stdout:8/644: write d6/db/d6a/f7c [3769348,84490] 0 2022-01-31T19:44:14.661 INFO:tasks.workunit.client.0.smithi167.stdout:3/577: symlink d9/d7f/da8/lad 0 2022-01-31T19:44:14.661 INFO:tasks.workunit.client.0.smithi167.stdout:7/806: mkdir d8/d2c/d6e/d4d/d5e/d71/db1/d126 0 2022-01-31T19:44:14.663 INFO:tasks.workunit.client.0.smithi167.stdout:6/513: rename d8/c25 to d8/d33/d51/d5d/d86/cbf 0 2022-01-31T19:44:14.664 INFO:tasks.workunit.client.0.smithi167.stdout:0/356: symlink d17/d2c/l6d 0 2022-01-31T19:44:14.664 INFO:tasks.workunit.client.0.smithi167.stdout:0/357: chown d17/d19/d1c/d39/d44/d49/c57 75387927 1 2022-01-31T19:44:14.664 INFO:tasks.workunit.client.0.smithi167.stdout:0/358: chown d17/d19/d1c/d39/d44/d49 28779 1 2022-01-31T19:44:14.665 INFO:tasks.workunit.client.0.smithi167.stdout:1/716: dread d0/d11/d46/f99 [0,4194304] 0 2022-01-31T19:44:14.665 INFO:tasks.workunit.client.0.smithi167.stdout:8/645: mkdir d6/db/de/d3d/dd6 0 2022-01-31T19:44:14.667 INFO:tasks.workunit.client.0.smithi167.stdout:3/578: link d9/d29/d60/l62 d9/d26/lae 0 2022-01-31T19:44:14.669 INFO:tasks.workunit.client.0.smithi167.stdout:7/807: rmdir d8/d2c/d6e/d9a/dd8/d116 0 2022-01-31T19:44:14.669 INFO:tasks.workunit.client.0.smithi167.stdout:7/808: chown d8/dc/f21 758349287 1 2022-01-31T19:44:14.670 INFO:tasks.workunit.client.0.smithi167.stdout:7/809: fdatasync d8/d2c/d7a/fcd 0 2022-01-31T19:44:14.670 INFO:tasks.workunit.client.0.smithi167.stdout:5/713: dread d5/d6/de/fbd [0,4194304] 0 2022-01-31T19:44:14.670 INFO:tasks.workunit.client.0.smithi167.stdout:8/646: getdents d6/d1e/dcf 0 2022-01-31T19:44:14.670 INFO:tasks.workunit.client.0.smithi167.stdout:8/647: chown d6/d1e/f22 195 1 2022-01-31T19:44:14.671 INFO:tasks.workunit.client.0.smithi167.stdout:7/810: mknod d8/dc/dc1/c127 0 2022-01-31T19:44:14.671 INFO:tasks.workunit.client.0.smithi167.stdout:5/714: rmdir d5/d6/de/d22/d6d/da0/da4 39 2022-01-31T19:44:14.672 INFO:tasks.workunit.client.0.smithi167.stdout:5/715: write d5/d6/de/d22/d4d/f9e [382301,79959] 0 2022-01-31T19:44:14.672 INFO:tasks.workunit.client.0.smithi167.stdout:5/716: write d5/d6/de/d22/d4d/f87 [754754,35195] 0 2022-01-31T19:44:14.673 INFO:tasks.workunit.client.0.smithi167.stdout:8/648: unlink d6/db/d6a/d8d/fb0 0 2022-01-31T19:44:14.673 INFO:tasks.workunit.client.0.smithi167.stdout:0/359: dread d17/d2c/f3c [0,4194304] 0 2022-01-31T19:44:14.673 INFO:tasks.workunit.client.0.smithi167.stdout:7/811: symlink d8/dc/d18/d24/dbb/d9e/l128 0 2022-01-31T19:44:14.674 INFO:tasks.workunit.client.0.smithi167.stdout:7/812: dread - d8/d2c/d6e/d4d/d5e/d71/db1/f117 zero size 2022-01-31T19:44:14.675 INFO:tasks.workunit.client.0.smithi167.stdout:1/717: dread d0/d10/d5a/d33/d4b/f65 [0,4194304] 0 2022-01-31T19:44:14.676 INFO:tasks.workunit.client.0.smithi167.stdout:5/717: truncate d5/d6/de/f4c 856889 0 2022-01-31T19:44:14.676 INFO:tasks.workunit.client.0.smithi167.stdout:5/718: dread - d5/d6/d16/d18/d43/d45/d69/dc4/fd6 zero size 2022-01-31T19:44:14.676 INFO:tasks.workunit.client.0.smithi167.stdout:0/360: creat d17/d19/d1c/d39/d44/d49/f6e x:0 0 0 2022-01-31T19:44:14.677 INFO:tasks.workunit.client.0.smithi167.stdout:0/361: creat d17/d33/f6f x:0 0 0 2022-01-31T19:44:14.677 INFO:tasks.workunit.client.0.smithi167.stdout:0/362: dread - d17/f5e zero size 2022-01-31T19:44:14.678 INFO:tasks.workunit.client.0.smithi167.stdout:1/718: truncate d0/fe4 1306354 0 2022-01-31T19:44:14.678 INFO:tasks.workunit.client.0.smithi167.stdout:1/719: fsync d0/d10/d18/f91 0 2022-01-31T19:44:14.680 INFO:tasks.workunit.client.0.smithi167.stdout:5/719: link d5/d6/de/d22/d4d/f9e d5/fdd 0 2022-01-31T19:44:14.681 INFO:tasks.workunit.client.0.smithi167.stdout:3/579: dread d9/d21/f1e [0,4194304] 0 2022-01-31T19:44:14.681 INFO:tasks.workunit.client.0.smithi167.stdout:1/720: dread d0/d10/d5a/d33/d4b/da5/fe0 [0,4194304] 0 2022-01-31T19:44:14.681 INFO:tasks.workunit.client.0.smithi167.stdout:3/580: write d9/d29/d3b/d41/d6b/f72 [204936,95953] 0 2022-01-31T19:44:14.682 INFO:tasks.workunit.client.0.smithi167.stdout:1/721: dread - d0/d62/fc6 zero size 2022-01-31T19:44:14.687 INFO:tasks.workunit.client.0.smithi167.stdout:0/363: symlink d17/d19/d1c/d39/d44/d49/l70 0 2022-01-31T19:44:14.690 INFO:tasks.workunit.client.0.smithi167.stdout:5/720: symlink d5/d6/d16/d18/d43/d45/d69/d90/lde 0 2022-01-31T19:44:14.692 INFO:tasks.workunit.client.0.smithi167.stdout:8/649: dread d6/d1e/d2d/f40 [0,4194304] 0 2022-01-31T19:44:14.692 INFO:tasks.workunit.client.0.smithi167.stdout:7/813: dwrite d8/dc/d18/d24/f29 [0,4194304] 0 2022-01-31T19:44:14.692 INFO:tasks.workunit.client.0.smithi167.stdout:8/650: truncate d6/db/d4e/d73/d88/f96 754336 0 2022-01-31T19:44:14.693 INFO:tasks.workunit.client.0.smithi167.stdout:7/814: chown d8/d2c/d7a/d99/de2/cef 12573196 1 2022-01-31T19:44:14.693 INFO:tasks.workunit.client.0.smithi167.stdout:7/815: fsync d8/dc/d18/d24/f29 0 2022-01-31T19:44:14.693 INFO:tasks.workunit.client.0.smithi167.stdout:3/581: rename d9/d29/d3b/d41/c54 to d9/d29/d60/caf 0 2022-01-31T19:44:14.694 INFO:tasks.workunit.client.0.smithi167.stdout:1/722: mknod d0/d10/d5a/d33/d97/cf3 0 2022-01-31T19:44:14.701 INFO:tasks.workunit.client.0.smithi167.stdout:7/816: unlink d8/dc/d18/l8c 0 2022-01-31T19:44:14.701 INFO:tasks.workunit.client.0.smithi167.stdout:8/651: dread d6/f1d [4194304,4194304] 0 2022-01-31T19:44:14.702 INFO:tasks.workunit.client.0.smithi167.stdout:8/652: creat d6/db/d36/d5d/fd7 x:0 0 0 2022-01-31T19:44:14.703 INFO:tasks.workunit.client.0.smithi167.stdout:5/721: truncate d5/d6/de/d22/f5f 7159976 0 2022-01-31T19:44:14.703 INFO:tasks.workunit.client.0.smithi167.stdout:5/722: read - d5/d6/de/d22/d6d/da0/da4/fd2 zero size 2022-01-31T19:44:14.703 INFO:tasks.workunit.client.0.smithi167.stdout:5/723: chown d5/d6/d16/lb5 36 1 2022-01-31T19:44:14.704 INFO:tasks.workunit.client.0.smithi167.stdout:1/723: link d0/d11/fd8 d0/d10/d20/d7c/ff4 0 2022-01-31T19:44:14.704 INFO:tasks.workunit.client.0.smithi167.stdout:3/582: write d9/d26/f2a [3415635,30955] 0 2022-01-31T19:44:14.707 INFO:tasks.workunit.client.0.smithi167.stdout:0/364: dwrite d17/d33/f6f [0,4194304] 0 2022-01-31T19:44:14.710 INFO:tasks.workunit.client.0.smithi167.stdout:8/653: symlink d6/db/de/d3d/dd6/ld8 0 2022-01-31T19:44:14.710 INFO:tasks.workunit.client.0.smithi167.stdout:8/654: chown d6/db/de/d3d/dd6 109885681 1 2022-01-31T19:44:14.712 INFO:tasks.workunit.client.0.smithi167.stdout:7/817: dread d8/d2c/d43/df3/ff6 [0,4194304] 0 2022-01-31T19:44:14.712 INFO:tasks.workunit.client.0.smithi167.stdout:7/818: dread - d8/dc/dc1/f107 zero size 2022-01-31T19:44:14.713 INFO:tasks.workunit.client.0.smithi167.stdout:5/724: mkdir d5/d6/d16/d18/d43/d45/dd5/dbc/dc0/ddf 0 2022-01-31T19:44:14.714 INFO:tasks.workunit.client.0.smithi167.stdout:3/583: mkdir d9/d29/d3b/d9b/da3/db0 0 2022-01-31T19:44:14.714 INFO:tasks.workunit.client.0.smithi167.stdout:3/584: write d9/d29/d60/f5b [808500,103095] 0 2022-01-31T19:44:14.721 INFO:tasks.workunit.client.0.smithi167.stdout:8/655: truncate d6/db/de/f3e 1965346 0 2022-01-31T19:44:14.722 INFO:tasks.workunit.client.0.smithi167.stdout:7/819: creat d8/d2c/d6e/d4d/d5e/d71/db3/f129 x:0 0 0 2022-01-31T19:44:14.723 INFO:tasks.workunit.client.0.smithi167.stdout:5/725: mknod d5/d6/d16/d18/d43/d45/dd5/dbc/dcc/ce0 0 2022-01-31T19:44:14.725 INFO:tasks.workunit.client.0.smithi167.stdout:3/585: link d9/d29/d3b/d41/d6b/f72 d9/d26/fb1 0 2022-01-31T19:44:14.725 INFO:tasks.workunit.client.0.smithi167.stdout:5/726: mkdir d5/d6/de/d22/d76/db0/de1 0 2022-01-31T19:44:14.728 INFO:tasks.workunit.client.0.smithi167.stdout:1/724: dwrite d0/d10/d20/f56 [0,4194304] 0 2022-01-31T19:44:14.729 INFO:tasks.workunit.client.0.smithi167.stdout:1/725: creat d0/ff5 x:0 0 0 2022-01-31T19:44:14.729 INFO:tasks.workunit.client.0.smithi167.stdout:5/727: read d5/d6/de/d22/d4d/f87 [287985,99071] 0 2022-01-31T19:44:14.731 INFO:tasks.workunit.client.0.smithi167.stdout:0/365: dwrite f14 [0,4194304] 0 2022-01-31T19:44:14.731 INFO:tasks.workunit.client.0.smithi167.stdout:1/726: getdents d0/d10/d18 0 2022-01-31T19:44:14.732 INFO:tasks.workunit.client.0.smithi167.stdout:0/366: symlink d17/d3e/l71 0 2022-01-31T19:44:14.734 INFO:tasks.workunit.client.0.smithi167.stdout:1/727: unlink d0/d11/l21 0 2022-01-31T19:44:14.735 INFO:tasks.workunit.client.0.smithi167.stdout:1/728: read d0/d11/d46/f99 [276796,73938] 0 2022-01-31T19:44:14.736 INFO:tasks.workunit.client.0.smithi167.stdout:8/656: dread d6/db/d36/f3b [0,4194304] 0 2022-01-31T19:44:14.736 INFO:tasks.workunit.client.0.smithi167.stdout:3/586: write d9/d21/f23 [4851331,83152] 0 2022-01-31T19:44:14.736 INFO:tasks.workunit.client.0.smithi167.stdout:3/587: write d9/d26/f6e [513593,51011] 0 2022-01-31T19:44:14.736 INFO:tasks.workunit.client.0.smithi167.stdout:3/588: dread - d9/d29/d3b/f9e zero size 2022-01-31T19:44:14.737 INFO:tasks.workunit.client.0.smithi167.stdout:0/367: mkdir d17/d72 0 2022-01-31T19:44:14.738 INFO:tasks.workunit.client.0.smithi167.stdout:0/368: write d17/f2f [1652039,114805] 0 2022-01-31T19:44:14.743 INFO:tasks.workunit.client.0.smithi167.stdout:8/657: getdents d6/db/de/d18/da4/dc2 0 2022-01-31T19:44:14.747 INFO:tasks.workunit.client.0.smithi167.stdout:8/658: rename d6/db/d36/d5d/lca to d6/db/de/d18/da4/dc2/ld9 0 2022-01-31T19:44:14.748 INFO:tasks.workunit.client.0.smithi167.stdout:8/659: rename d6/db/d6a/d8d/lbc to d6/db/de/d3d/dd6/lda 0 2022-01-31T19:44:14.749 INFO:tasks.workunit.client.0.smithi167.stdout:8/660: unlink d6/db/de/d3d/f71 0 2022-01-31T19:44:14.751 INFO:tasks.workunit.client.0.smithi167.stdout:3/589: dread d9/d29/f52 [0,4194304] 0 2022-01-31T19:44:14.755 INFO:tasks.workunit.client.0.smithi167.stdout:3/590: write f7 [375573,53520] 0 2022-01-31T19:44:14.755 INFO:tasks.workunit.client.0.smithi167.stdout:3/591: stat d9/d29/d3b/d41/d6b/f80 0 2022-01-31T19:44:14.756 INFO:tasks.workunit.client.0.smithi167.stdout:3/592: fdatasync d9/d7f/f93 0 2022-01-31T19:44:14.758 INFO:tasks.workunit.client.0.smithi167.stdout:5/728: dwrite d5/d6/d16/d18/f59 [0,4194304] 0 2022-01-31T19:44:14.758 INFO:tasks.workunit.client.0.smithi167.stdout:5/729: chown d5/d6/d3a 1850052587 1 2022-01-31T19:44:14.758 INFO:tasks.workunit.client.0.smithi167.stdout:5/730: chown d5/d6/d16/d18/d43/d45/dd5/dbc/dc0 7033 1 2022-01-31T19:44:14.759 INFO:tasks.workunit.client.0.smithi167.stdout:3/593: creat d9/d29/fb2 x:0 0 0 2022-01-31T19:44:14.759 INFO:tasks.workunit.client.0.smithi167.stdout:1/729: dwrite d0/f73 [0,4194304] 0 2022-01-31T19:44:14.759 INFO:tasks.workunit.client.0.smithi167.stdout:1/730: chown d0/d10/d5a/d33/d4b 8011 1 2022-01-31T19:44:14.761 INFO:tasks.workunit.client.0.smithi167.stdout:3/594: truncate d9/d29/d60/fac 3212506 0 2022-01-31T19:44:14.765 INFO:tasks.workunit.client.0.smithi167.stdout:1/731: creat d0/d75/d7a/ff6 x:0 0 0 2022-01-31T19:44:14.765 INFO:tasks.workunit.client.0.smithi167.stdout:1/732: unlink d0/d10/d20/c2c 0 2022-01-31T19:44:14.766 INFO:tasks.workunit.client.0.smithi167.stdout:3/595: write d9/d26/f55 [3171039,106530] 0 2022-01-31T19:44:14.768 INFO:tasks.workunit.client.0.smithi167.stdout:3/596: stat d9/f25 0 2022-01-31T19:44:14.771 INFO:tasks.workunit.client.0.smithi167.stdout:5/731: dread d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:14.771 INFO:tasks.workunit.client.0.smithi167.stdout:8/661: dwrite d6/db/d6a/f7c [0,4194304] 0 2022-01-31T19:44:14.774 INFO:tasks.workunit.client.0.smithi167.stdout:8/662: rename d6/d1e/d2d/c34 to d6/db/d4e/d73/cdb 0 2022-01-31T19:44:14.774 INFO:tasks.workunit.client.0.smithi167.stdout:8/663: stat d6/db/d36/d5d/f68 0 2022-01-31T19:44:14.777 INFO:tasks.workunit.client.0.smithi167.stdout:5/732: write d5/d6/de/d22/d76/f80 [3833398,64829] 0 2022-01-31T19:44:14.781 INFO:tasks.workunit.client.0.smithi167.stdout:5/733: symlink d5/d6/de/d22/d76/db0/de1/le2 0 2022-01-31T19:44:14.804 INFO:tasks.workunit.client.0.smithi167.stdout:1/733: dwrite d0/d62/fa3 [0,4194304] 0 2022-01-31T19:44:14.805 INFO:tasks.workunit.client.0.smithi167.stdout:0/369: dwrite d17/f2f [0,4194304] 0 2022-01-31T19:44:14.805 INFO:tasks.workunit.client.0.smithi167.stdout:1/734: dread - d0/d62/d7f/d82/fba zero size 2022-01-31T19:44:14.805 INFO:tasks.workunit.client.0.smithi167.stdout:1/735: write d0/d10/d20/f4f [4797919,114070] 0 2022-01-31T19:44:14.805 INFO:tasks.workunit.client.0.smithi167.stdout:1/736: chown d0/d62/l89 238910 1 2022-01-31T19:44:14.807 INFO:tasks.workunit.client.0.smithi167.stdout:0/370: rmdir d17/d3e 39 2022-01-31T19:44:14.808 INFO:tasks.workunit.client.0.smithi167.stdout:0/371: creat d17/d72/f73 x:0 0 0 2022-01-31T19:44:14.808 INFO:tasks.workunit.client.0.smithi167.stdout:0/372: write d17/d19/d1c/d39/d44/d49/f6e [956255,41936] 0 2022-01-31T19:44:14.810 INFO:tasks.workunit.client.0.smithi167.stdout:0/373: creat d17/d19/d1c/d39/f74 x:0 0 0 2022-01-31T19:44:14.810 INFO:tasks.workunit.client.0.smithi167.stdout:0/374: readlink l12 0 2022-01-31T19:44:14.815 INFO:tasks.workunit.client.0.smithi167.stdout:3/597: dwrite d9/d29/d3b/d41/d6b/f66 [0,4194304] 0 2022-01-31T19:44:14.816 INFO:tasks.workunit.client.0.smithi167.stdout:8/664: dwrite d6/db/de/d18/f19 [0,4194304] 0 2022-01-31T19:44:14.817 INFO:tasks.workunit.client.0.smithi167.stdout:3/598: dread d9/d26/fb1 [0,4194304] 0 2022-01-31T19:44:14.818 INFO:tasks.workunit.client.0.smithi167.stdout:3/599: chown d9/d29/d3b/c9c 66070 1 2022-01-31T19:44:14.818 INFO:tasks.workunit.client.0.smithi167.stdout:5/734: dwrite d5/d6/de/d22/d4d/f87 [0,4194304] 0 2022-01-31T19:44:14.819 INFO:tasks.workunit.client.0.smithi167.stdout:5/735: fdatasync d5/d6/f41 0 2022-01-31T19:44:14.823 INFO:tasks.workunit.client.0.smithi167.stdout:3/600: rename d9/d21/c5d to d9/d29/d3b/d9b/da3/cb3 0 2022-01-31T19:44:14.823 INFO:tasks.workunit.client.0.smithi167.stdout:3/601: chown d9/d26/f98 1090842 1 2022-01-31T19:44:14.826 INFO:tasks.workunit.client.0.smithi167.stdout:0/375: dwrite d17/d19/d1c/f32 [4194304,4194304] 0 2022-01-31T19:44:14.827 INFO:tasks.workunit.client.0.smithi167.stdout:8/665: dread d6/db/d4e/d73/d88/f96 [0,4194304] 0 2022-01-31T19:44:14.827 INFO:tasks.workunit.client.0.smithi167.stdout:8/666: truncate d6/d1e/d2d/f40 8801667 0 2022-01-31T19:44:14.828 INFO:tasks.workunit.client.0.smithi167.stdout:3/602: truncate d9/d29/d60/f58 3125252 0 2022-01-31T19:44:14.828 INFO:tasks.workunit.client.0.smithi167.stdout:3/603: read - d9/d7f/f85 zero size 2022-01-31T19:44:14.828 INFO:tasks.workunit.client.0.smithi167.stdout:3/604: stat d9/d7f 0 2022-01-31T19:44:14.829 INFO:tasks.workunit.client.0.smithi167.stdout:3/605: chown d9/f78 18335034 1 2022-01-31T19:44:14.829 INFO:tasks.workunit.client.0.smithi167.stdout:0/376: mknod d17/d33/c75 0 2022-01-31T19:44:14.831 INFO:tasks.workunit.client.0.smithi167.stdout:5/736: dread d5/d6/f28 [0,4194304] 0 2022-01-31T19:44:14.832 INFO:tasks.workunit.client.0.smithi167.stdout:8/667: rename d6/db/d28/d66/ld1 to d6/d1e/d2d/d6b/ldc 0 2022-01-31T19:44:14.834 INFO:tasks.workunit.client.0.smithi167.stdout:8/668: symlink d6/db/d28/d61/dd5/dbf/ldd 0 2022-01-31T19:44:14.835 INFO:tasks.workunit.client.0.smithi167.stdout:8/669: truncate d6/d1e/f41 3832824 0 2022-01-31T19:44:14.835 INFO:tasks.workunit.client.0.smithi167.stdout:8/670: creat d6/db/d36/d5d/fde x:0 0 0 2022-01-31T19:44:14.835 INFO:tasks.workunit.client.0.smithi167.stdout:8/671: fsync d6/db/de/f10 0 2022-01-31T19:44:14.836 INFO:tasks.workunit.client.0.smithi167.stdout:8/672: chown d6/db/d4e/f4f 0 1 2022-01-31T19:44:14.836 INFO:tasks.workunit.client.0.smithi167.stdout:8/673: write d6/db/d4e/fd0 [63842,101382] 0 2022-01-31T19:44:14.841 INFO:tasks.workunit.client.0.smithi167.stdout:8/674: truncate d6/db/d28/d66/f8e 1448667 0 2022-01-31T19:44:14.848 INFO:tasks.workunit.client.0.smithi167.stdout:9/577: sync 2022-01-31T19:44:14.850 INFO:tasks.workunit.client.0.smithi167.stdout:9/578: unlink dd/d19/d59/c70 0 2022-01-31T19:44:14.851 INFO:tasks.workunit.client.0.smithi167.stdout:9/579: rmdir dd/d19 39 2022-01-31T19:44:14.853 INFO:tasks.workunit.client.0.smithi167.stdout:9/580: rename f7 to dd/d6b/d72/fc2 0 2022-01-31T19:44:14.854 INFO:tasks.workunit.client.0.smithi167.stdout:9/581: unlink dd/d35/f60 0 2022-01-31T19:44:14.862 INFO:tasks.workunit.client.0.smithi167.stdout:0/377: dwrite d17/f2f [4194304,4194304] 0 2022-01-31T19:44:14.864 INFO:tasks.workunit.client.0.smithi167.stdout:3/606: dwrite d9/d29/d3b/d41/d6b/f3e [0,4194304] 0 2022-01-31T19:44:14.864 INFO:tasks.workunit.client.0.smithi167.stdout:3/607: write d9/d7f/f93 [1015911,30617] 0 2022-01-31T19:44:14.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/608: chown d9/d29/f2b 684137 1 2022-01-31T19:44:14.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/609: readlink d9/d56/l7b 0 2022-01-31T19:44:14.865 INFO:tasks.workunit.client.0.smithi167.stdout:5/737: dwrite d5/d6/de/d22/d4d/f87 [0,4194304] 0 2022-01-31T19:44:14.867 INFO:tasks.workunit.client.0.smithi167.stdout:0/378: write ff [3171416,39896] 0 2022-01-31T19:44:14.873 INFO:tasks.workunit.client.0.smithi167.stdout:0/379: read d17/d33/f6f [3260144,78952] 0 2022-01-31T19:44:14.874 INFO:tasks.workunit.client.0.smithi167.stdout:0/380: creat d17/d2c/f76 x:0 0 0 2022-01-31T19:44:14.878 INFO:tasks.workunit.client.0.smithi167.stdout:0/381: read d17/d19/f1f [98030,96478] 0 2022-01-31T19:44:14.879 INFO:tasks.workunit.client.0.smithi167.stdout:0/382: symlink d17/d69/l77 0 2022-01-31T19:44:14.881 INFO:tasks.workunit.client.0.smithi167.stdout:0/383: creat d17/d69/f78 x:0 0 0 2022-01-31T19:44:14.883 INFO:tasks.workunit.client.0.smithi167.stdout:0/384: unlink d17/d3e/f4a 0 2022-01-31T19:44:14.883 INFO:tasks.workunit.client.0.smithi167.stdout:0/385: dread - d17/d19/f2d zero size 2022-01-31T19:44:14.883 INFO:tasks.workunit.client.0.smithi167.stdout:0/386: creat d17/d2c/f79 x:0 0 0 2022-01-31T19:44:14.883 INFO:tasks.workunit.client.0.smithi167.stdout:0/387: creat d17/f7a x:0 0 0 2022-01-31T19:44:14.884 INFO:tasks.workunit.client.0.smithi167.stdout:0/388: write d17/d19/f4d [4447063,35396] 0 2022-01-31T19:44:14.886 INFO:tasks.workunit.client.0.smithi167.stdout:0/389: write d17/f22 [1637515,94641] 0 2022-01-31T19:44:14.886 INFO:tasks.workunit.client.0.smithi167.stdout:0/390: fdatasync d17/d19/d1c/d39/f74 0 2022-01-31T19:44:14.886 INFO:tasks.workunit.client.0.smithi167.stdout:0/391: write d17/d3e/f68 [689775,78263] 0 2022-01-31T19:44:14.887 INFO:tasks.workunit.client.0.smithi167.stdout:0/392: mkdir d17/d19/d1c/d39/d7b 0 2022-01-31T19:44:14.888 INFO:tasks.workunit.client.0.smithi167.stdout:0/393: rename d17/d19/d1c/d39/d44/l6a to d17/l7c 0 2022-01-31T19:44:14.888 INFO:tasks.workunit.client.0.smithi167.stdout:0/394: fdatasync d17/d33/f36 0 2022-01-31T19:44:14.890 INFO:tasks.workunit.client.0.smithi167.stdout:5/738: dwrite d5/d6/d16/d18/d43/fd4 [0,4194304] 0 2022-01-31T19:44:14.890 INFO:tasks.workunit.client.0.smithi167.stdout:3/610: dwrite d9/d26/f55 [0,4194304] 0 2022-01-31T19:44:14.891 INFO:tasks.workunit.client.0.smithi167.stdout:9/582: dwrite dd/d19/d59/f9f [4194304,4194304] 0 2022-01-31T19:44:14.898 INFO:tasks.workunit.client.0.smithi167.stdout:9/583: symlink dd/d17/d80/d4c/lc3 0 2022-01-31T19:44:14.912 INFO:tasks.workunit.client.0.smithi167.stdout:9/584: dread dd/d19/f2d [0,4194304] 0 2022-01-31T19:44:14.912 INFO:tasks.workunit.client.0.smithi167.stdout:9/585: fdatasync dd/d19/d59/f9f 0 2022-01-31T19:44:14.913 INFO:tasks.workunit.client.0.smithi167.stdout:9/586: getdents dd/d6b/dc1 0 2022-01-31T19:44:14.915 INFO:tasks.workunit.client.0.smithi167.stdout:9/587: dread dd/d19/d24/d50/d5d/f78 [0,4194304] 0 2022-01-31T19:44:14.916 INFO:tasks.workunit.client.0.smithi167.stdout:0/395: dwrite d17/d2c/f79 [0,4194304] 0 2022-01-31T19:44:14.919 INFO:tasks.workunit.client.0.smithi167.stdout:0/396: mknod d17/d19/d1c/d39/c7d 0 2022-01-31T19:44:14.920 INFO:tasks.workunit.client.0.smithi167.stdout:3/611: dwrite d9/d29/d3b/d41/d6b/f72 [0,4194304] 0 2022-01-31T19:44:14.920 INFO:tasks.workunit.client.0.smithi167.stdout:3/612: write d9/d29/d3b/f9e [157718,122163] 0 2022-01-31T19:44:14.920 INFO:tasks.workunit.client.0.smithi167.stdout:3/613: write d9/d7f/f85 [672348,128772] 0 2022-01-31T19:44:14.920 INFO:tasks.workunit.client.0.smithi167.stdout:3/614: write d9/f2c [4487860,70889] 0 2022-01-31T19:44:14.921 INFO:tasks.workunit.client.0.smithi167.stdout:5/739: dwrite d5/d6/de/d22/d4d/fa5 [0,4194304] 0 2022-01-31T19:44:14.922 INFO:tasks.workunit.client.0.smithi167.stdout:9/588: rmdir dd/d17/d80/d4c 39 2022-01-31T19:44:14.922 INFO:tasks.workunit.client.0.smithi167.stdout:9/589: dread - dd/d19/d24/d56/d5b/f95 zero size 2022-01-31T19:44:14.923 INFO:tasks.workunit.client.0.smithi167.stdout:9/590: write dd/d19/f3d [2088932,64591] 0 2022-01-31T19:44:14.924 INFO:tasks.workunit.client.0.smithi167.stdout:9/591: stat dd/d19/d24/d56/d5b 0 2022-01-31T19:44:14.927 INFO:tasks.workunit.client.0.smithi167.stdout:5/740: mknod d5/d6/de/d22/d6d/da0/da4/ce3 0 2022-01-31T19:44:14.927 INFO:tasks.workunit.client.0.smithi167.stdout:5/741: write d5/d6/d3a/f75 [1394520,93520] 0 2022-01-31T19:44:14.928 INFO:tasks.workunit.client.0.smithi167.stdout:5/742: read d5/d6/d16/d18/f64 [738196,82222] 0 2022-01-31T19:44:14.928 INFO:tasks.workunit.client.0.smithi167.stdout:5/743: chown d5/d6/de/d22/d4d/f9e 7 1 2022-01-31T19:44:14.934 INFO:tasks.workunit.client.0.smithi167.stdout:0/397: rmdir d17 39 2022-01-31T19:44:14.936 INFO:tasks.workunit.client.0.smithi167.stdout:3/615: dread d9/d29/f33 [0,4194304] 0 2022-01-31T19:44:14.937 INFO:tasks.workunit.client.0.smithi167.stdout:9/592: mkdir dd/d17/dc4 0 2022-01-31T19:44:14.937 INFO:tasks.workunit.client.0.smithi167.stdout:5/744: write d5/d6/d16/d18/d43/d45/f60 [1770629,15714] 0 2022-01-31T19:44:14.937 INFO:tasks.workunit.client.0.smithi167.stdout:0/398: getdents d17/d2c 0 2022-01-31T19:44:14.938 INFO:tasks.workunit.client.0.smithi167.stdout:3/616: stat d9/d21/c1f 0 2022-01-31T19:44:14.938 INFO:tasks.workunit.client.0.smithi167.stdout:9/593: symlink dd/d17/d80/d4c/lc5 0 2022-01-31T19:44:14.940 INFO:tasks.workunit.client.0.smithi167.stdout:5/745: link d5/d6/de/f4c d5/d6/de/fe4 0 2022-01-31T19:44:14.941 INFO:tasks.workunit.client.0.smithi167.stdout:3/617: truncate d9/d29/d60/f58 239038 0 2022-01-31T19:44:14.943 INFO:tasks.workunit.client.0.smithi167.stdout:9/594: link dd/d19/d24/d50/d5d/c82 dd/d17/dc4/cc6 0 2022-01-31T19:44:14.945 INFO:tasks.workunit.client.0.smithi167.stdout:0/399: dread d17/f3d [4194304,4194304] 0 2022-01-31T19:44:14.946 INFO:tasks.workunit.client.0.smithi167.stdout:5/746: rename d5/c68 to d5/d6/d16/d18/ce5 0 2022-01-31T19:44:14.949 INFO:tasks.workunit.client.0.smithi167.stdout:0/400: truncate d17/d33/f5c 208045 0 2022-01-31T19:44:14.950 INFO:tasks.workunit.client.0.smithi167.stdout:0/401: readlink d17/d2c/l41 0 2022-01-31T19:44:14.951 INFO:tasks.workunit.client.0.smithi167.stdout:0/402: rename d17/d72 to d17/d3e/d7e 0 2022-01-31T19:44:14.967 INFO:tasks.workunit.client.0.smithi167.stdout:5/747: dread d5/d6/d16/d18/d5b/fc2 [0,4194304] 0 2022-01-31T19:44:14.968 INFO:tasks.workunit.client.0.smithi167.stdout:5/748: getdents d5/d6/de/d22 0 2022-01-31T19:44:14.971 INFO:tasks.workunit.client.0.smithi167.stdout:3/618: dwrite d9/d29/d60/fa0 [0,4194304] 0 2022-01-31T19:44:14.971 INFO:tasks.workunit.client.0.smithi167.stdout:3/619: read d9/d7f/f85 [714909,26146] 0 2022-01-31T19:44:14.972 INFO:tasks.workunit.client.0.smithi167.stdout:3/620: creat d9/d29/d3b/d9b/fb4 x:0 0 0 2022-01-31T19:44:14.975 INFO:tasks.workunit.client.0.smithi167.stdout:0/403: dwrite d17/d2c/f64 [0,4194304] 0 2022-01-31T19:44:14.979 INFO:tasks.workunit.client.0.smithi167.stdout:0/404: rmdir d17/d19 39 2022-01-31T19:44:14.979 INFO:tasks.workunit.client.0.smithi167.stdout:0/405: fsync d17/f3d 0 2022-01-31T19:44:14.980 INFO:tasks.workunit.client.0.smithi167.stdout:0/406: rename d17/d19/f4e to d17/d19/f7f 0 2022-01-31T19:44:14.981 INFO:tasks.workunit.client.0.smithi167.stdout:0/407: rmdir d17/d19/d1c/d39 39 2022-01-31T19:44:14.990 INFO:tasks.workunit.client.0.smithi167.stdout:5/749: dwrite d5/d6/de/d22/d4d/f9e [0,4194304] 0 2022-01-31T19:44:14.993 INFO:tasks.workunit.client.0.smithi167.stdout:3/621: dwrite d9/d26/fb1 [0,4194304] 0 2022-01-31T19:44:15.001 INFO:tasks.workunit.client.0.smithi167.stdout:3/622: dread d9/d26/f2a [0,4194304] 0 2022-01-31T19:44:15.003 INFO:tasks.workunit.client.0.smithi167.stdout:3/623: truncate d9/d29/d3b/d41/f5c 4096130 0 2022-01-31T19:44:15.004 INFO:tasks.workunit.client.0.smithi167.stdout:2/478: sync 2022-01-31T19:44:15.004 INFO:tasks.workunit.client.0.smithi167.stdout:4/619: sync 2022-01-31T19:44:15.005 INFO:tasks.workunit.client.0.smithi167.stdout:4/620: write d0/d3f/d60/d65/f9d [2401756,2986] 0 2022-01-31T19:44:15.005 INFO:tasks.workunit.client.0.smithi167.stdout:5/750: dread d5/d6/de/d22/f5f [0,4194304] 0 2022-01-31T19:44:15.006 INFO:tasks.workunit.client.0.smithi167.stdout:5/751: fsync d5/d6/d16/d18/d43/d45/dd5/dbc/fd3 0 2022-01-31T19:44:15.006 INFO:tasks.workunit.client.0.smithi167.stdout:5/752: truncate d5/d6/d16/f9d 123752 0 2022-01-31T19:44:15.006 INFO:tasks.workunit.client.0.smithi167.stdout:2/479: write dd/df/d3f/f8e [603920,79964] 0 2022-01-31T19:44:15.007 INFO:tasks.workunit.client.0.smithi167.stdout:4/621: unlink d0/d6/d7/d39/d4e/l96 0 2022-01-31T19:44:15.007 INFO:tasks.workunit.client.0.smithi167.stdout:5/753: truncate d5/d6/de/d22/d76/db0/fb6 3617811 0 2022-01-31T19:44:15.008 INFO:tasks.workunit.client.0.smithi167.stdout:2/480: symlink dd/df/l94 0 2022-01-31T19:44:15.010 INFO:tasks.workunit.client.0.smithi167.stdout:4/622: creat d0/d3f/d60/db5/fcb x:0 0 0 2022-01-31T19:44:15.011 INFO:tasks.workunit.client.0.smithi167.stdout:5/754: creat d5/d6/de/d22/d6d/da0/fe6 x:0 0 0 2022-01-31T19:44:15.012 INFO:tasks.workunit.client.0.smithi167.stdout:2/481: mknod dd/df/d1b/d3d/d4a/d4b/d74/d85/c95 0 2022-01-31T19:44:15.014 INFO:tasks.workunit.client.0.smithi167.stdout:4/623: unlink d0/d6/d7/d39/cc1 0 2022-01-31T19:44:15.016 INFO:tasks.workunit.client.0.smithi167.stdout:5/755: unlink d5/fdd 0 2022-01-31T19:44:15.018 INFO:tasks.workunit.client.0.smithi167.stdout:4/624: write d0/d6/d7/d39/f37 [1875322,51045] 0 2022-01-31T19:44:15.019 INFO:tasks.workunit.client.0.smithi167.stdout:2/482: mkdir dd/df/d22/d96 0 2022-01-31T19:44:15.020 INFO:tasks.workunit.client.0.smithi167.stdout:4/625: creat d0/d6/d7/d39/d4e/d61/fcc x:0 0 0 2022-01-31T19:44:15.021 INFO:tasks.workunit.client.0.smithi167.stdout:2/483: rename dd/df/d1b/d2f/c6c to dd/df/d1b/d3d/c97 0 2022-01-31T19:44:15.022 INFO:tasks.workunit.client.0.smithi167.stdout:4/626: mknod d0/d3f/d75/ccd 0 2022-01-31T19:44:15.023 INFO:tasks.workunit.client.0.smithi167.stdout:0/408: dwrite d17/d2c/f64 [4194304,4194304] 0 2022-01-31T19:44:15.024 INFO:tasks.workunit.client.0.smithi167.stdout:0/409: chown cb 3274623 1 2022-01-31T19:44:15.024 INFO:tasks.workunit.client.0.smithi167.stdout:0/410: dread - d17/d19/d1c/d39/f74 zero size 2022-01-31T19:44:15.024 INFO:tasks.workunit.client.0.smithi167.stdout:2/484: symlink dd/df/d1b/l98 0 2022-01-31T19:44:15.025 INFO:tasks.workunit.client.0.smithi167.stdout:2/485: write f8 [5588132,82467] 0 2022-01-31T19:44:15.025 INFO:tasks.workunit.client.0.smithi167.stdout:4/627: mkdir d0/d3f/d75/dce 0 2022-01-31T19:44:15.026 INFO:tasks.workunit.client.0.smithi167.stdout:2/486: write dd/df/d1b/d3d/d4a/d4b/d54/f8b [118007,85828] 0 2022-01-31T19:44:15.026 INFO:tasks.workunit.client.0.smithi167.stdout:0/411: creat d17/d19/d1c/d39/d44/d49/f80 x:0 0 0 2022-01-31T19:44:15.027 INFO:tasks.workunit.client.0.smithi167.stdout:3/624: dwrite d9/d29/d9d/fa7 [0,4194304] 0 2022-01-31T19:44:15.027 INFO:tasks.workunit.client.0.smithi167.stdout:2/487: creat dd/d18/f99 x:0 0 0 2022-01-31T19:44:15.029 INFO:tasks.workunit.client.0.smithi167.stdout:2/488: dread dd/df/d3f/f8e [0,4194304] 0 2022-01-31T19:44:15.030 INFO:tasks.workunit.client.0.smithi167.stdout:0/412: creat d17/d19/d1c/d39/d7b/f81 x:0 0 0 2022-01-31T19:44:15.035 INFO:tasks.workunit.client.0.smithi167.stdout:3/625: mknod d9/d29/d9d/cb5 0 2022-01-31T19:44:15.039 INFO:tasks.workunit.client.0.smithi167.stdout:2/489: link dd/d18/f99 dd/df/f9a 0 2022-01-31T19:44:15.039 INFO:tasks.workunit.client.0.smithi167.stdout:2/490: fdatasync dd/f55 0 2022-01-31T19:44:15.039 INFO:tasks.workunit.client.0.smithi167.stdout:2/491: write fa [9204193,98359] 0 2022-01-31T19:44:15.040 INFO:tasks.workunit.client.0.smithi167.stdout:0/413: symlink d17/d2c/l82 0 2022-01-31T19:44:15.040 INFO:tasks.workunit.client.0.smithi167.stdout:0/414: fsync d17/d2c/f76 0 2022-01-31T19:44:15.041 INFO:tasks.workunit.client.0.smithi167.stdout:3/626: symlink d9/d29/d3b/d41/d6b/d89/lb6 0 2022-01-31T19:44:15.043 INFO:tasks.workunit.client.0.smithi167.stdout:2/492: creat dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b x:0 0 0 2022-01-31T19:44:15.063 INFO:tasks.workunit.client.0.smithi167.stdout:3/627: dwrite d9/fc [0,4194304] 0 2022-01-31T19:44:15.064 INFO:tasks.workunit.client.0.smithi167.stdout:3/628: write d9/d29/fb2 [412200,40909] 0 2022-01-31T19:44:15.064 INFO:tasks.workunit.client.0.smithi167.stdout:0/415: dwrite d17/d3e/f5f [0,4194304] 0 2022-01-31T19:44:15.064 INFO:tasks.workunit.client.0.smithi167.stdout:0/416: fdatasync d17/d3e/f42 0 2022-01-31T19:44:15.065 INFO:tasks.workunit.client.0.smithi167.stdout:3/629: rename d9/d26/lae to d9/d29/d60/lb7 0 2022-01-31T19:44:15.066 INFO:tasks.workunit.client.0.smithi167.stdout:3/630: creat d9/d26/fb8 x:0 0 0 2022-01-31T19:44:15.066 INFO:tasks.workunit.client.0.smithi167.stdout:3/631: truncate d9/d29/d3b/d41/f5c 4927783 0 2022-01-31T19:44:15.066 INFO:tasks.workunit.client.0.smithi167.stdout:3/632: write d9/d29/f4c [1943639,45361] 0 2022-01-31T19:44:15.067 INFO:tasks.workunit.client.0.smithi167.stdout:0/417: truncate d17/d19/d1c/d39/d44/f65 921796 0 2022-01-31T19:44:15.069 INFO:tasks.workunit.client.0.smithi167.stdout:3/633: link d9/d21/f77 d9/d29/d3b/d9b/da3/db0/fb9 0 2022-01-31T19:44:15.070 INFO:tasks.workunit.client.0.smithi167.stdout:3/634: getdents d9/d7f 0 2022-01-31T19:44:15.071 INFO:tasks.workunit.client.0.smithi167.stdout:3/635: read - d9/d29/d3b/d41/d6b/f80 zero size 2022-01-31T19:44:15.071 INFO:tasks.workunit.client.0.smithi167.stdout:3/636: truncate d9/d26/f65 688147 0 2022-01-31T19:44:15.071 INFO:tasks.workunit.client.0.smithi167.stdout:3/637: fdatasync d9/f2c 0 2022-01-31T19:44:15.072 INFO:tasks.workunit.client.0.smithi167.stdout:2/493: dwrite dd/df/d3f/d51/d5f/f69 [0,4194304] 0 2022-01-31T19:44:15.077 INFO:tasks.workunit.client.0.smithi167.stdout:2/494: truncate dd/df/d1b/d2f/f34 3778533 0 2022-01-31T19:44:15.093 INFO:tasks.workunit.client.0.smithi167.stdout:3/638: dwrite d9/d56/f6d [0,4194304] 0 2022-01-31T19:44:15.093 INFO:tasks.workunit.client.0.smithi167.stdout:2/495: dwrite dd/d18/f31 [0,4194304] 0 2022-01-31T19:44:15.098 INFO:tasks.workunit.client.0.smithi167.stdout:3/639: rename d9/d29/f52 to d9/d29/d60/fba 0 2022-01-31T19:44:15.099 INFO:tasks.workunit.client.0.smithi167.stdout:3/640: dread d9/f78 [0,4194304] 0 2022-01-31T19:44:15.099 INFO:tasks.workunit.client.0.smithi167.stdout:2/496: unlink dd/df/d1b/d2f/f66 0 2022-01-31T19:44:15.100 INFO:tasks.workunit.client.0.smithi167.stdout:3/641: truncate d9/f25 4160422 0 2022-01-31T19:44:15.101 INFO:tasks.workunit.client.0.smithi167.stdout:2/497: symlink dd/df/d22/d96/l9c 0 2022-01-31T19:44:15.101 INFO:tasks.workunit.client.0.smithi167.stdout:2/498: write dd/d18/d63/f2d [2744854,111019] 0 2022-01-31T19:44:15.102 INFO:tasks.workunit.client.0.smithi167.stdout:2/499: chown dd/df/c39 1273 1 2022-01-31T19:44:15.102 INFO:tasks.workunit.client.0.smithi167.stdout:2/500: chown dd/d18/l7e 2011 1 2022-01-31T19:44:15.105 INFO:tasks.workunit.client.0.smithi167.stdout:3/642: mkdir d9/d7f/dbb 0 2022-01-31T19:44:15.107 INFO:tasks.workunit.client.0.smithi167.stdout:1/737: sync 2022-01-31T19:44:15.107 INFO:tasks.workunit.client.0.smithi167.stdout:7/820: sync 2022-01-31T19:44:15.108 INFO:tasks.workunit.client.0.smithi167.stdout:6/514: sync 2022-01-31T19:44:15.108 INFO:tasks.workunit.client.0.smithi167.stdout:2/501: link dd/df/d1b/d2f/f34 dd/df/d22/f9d 0 2022-01-31T19:44:15.109 INFO:tasks.workunit.client.0.smithi167.stdout:2/502: write dd/df/f79 [309170,122729] 0 2022-01-31T19:44:15.110 INFO:tasks.workunit.client.0.smithi167.stdout:7/821: creat d8/dc/d18/d24/dbb/f12a x:0 0 0 2022-01-31T19:44:15.111 INFO:tasks.workunit.client.0.smithi167.stdout:3/643: rmdir d9/d29/d9d 39 2022-01-31T19:44:15.111 INFO:tasks.workunit.client.0.smithi167.stdout:3/644: dread - d9/d29/d60/f69 zero size 2022-01-31T19:44:15.112 INFO:tasks.workunit.client.0.smithi167.stdout:3/645: chown d9/d29/d60/l6f 5820941 1 2022-01-31T19:44:15.113 INFO:tasks.workunit.client.0.smithi167.stdout:7/822: creat d8/d2c/d6e/d4d/d67/d8f/f12b x:0 0 0 2022-01-31T19:44:15.116 INFO:tasks.workunit.client.0.smithi167.stdout:3/646: getdents d9/d29/d3b/d9b/da3 0 2022-01-31T19:44:15.117 INFO:tasks.workunit.client.0.smithi167.stdout:7/823: getdents d8/d2c/d43/df3 0 2022-01-31T19:44:15.117 INFO:tasks.workunit.client.0.smithi167.stdout:7/824: truncate d8/d2c/d6e/d9a/f110 970588 0 2022-01-31T19:44:15.117 INFO:tasks.workunit.client.0.smithi167.stdout:7/825: stat d8/d2c/d6e/f2a 0 2022-01-31T19:44:15.118 INFO:tasks.workunit.client.0.smithi167.stdout:7/826: dread - d8/dc/f11e zero size 2022-01-31T19:44:15.118 INFO:tasks.workunit.client.0.smithi167.stdout:7/827: chown d8/d2c/d43/da1/fc0 7956716 1 2022-01-31T19:44:15.120 INFO:tasks.workunit.client.0.smithi167.stdout:3/647: unlink d9/d29/d3b/d9b/da3/cb3 0 2022-01-31T19:44:15.120 INFO:tasks.workunit.client.0.smithi167.stdout:3/648: chown d9/d29/d60/l6f 1175 1 2022-01-31T19:44:15.120 INFO:tasks.workunit.client.0.smithi167.stdout:1/738: dread d0/d10/d18/f3e [0,4194304] 0 2022-01-31T19:44:15.122 INFO:tasks.workunit.client.0.smithi167.stdout:3/649: mknod d9/d29/d3b/cbc 0 2022-01-31T19:44:15.127 INFO:tasks.workunit.client.0.smithi167.stdout:2/503: dread dd/df/d1b/d3d/f44 [0,4194304] 0 2022-01-31T19:44:15.130 INFO:tasks.workunit.client.0.smithi167.stdout:6/515: dwrite d8/d33/d51/d5d/d86/d94/f96 [0,4194304] 0 2022-01-31T19:44:15.130 INFO:tasks.workunit.client.0.smithi167.stdout:6/516: write d8/d33/d35/f61 [728554,7375] 0 2022-01-31T19:44:15.130 INFO:tasks.workunit.client.0.smithi167.stdout:2/504: mkdir dd/df/d22/d96/d9e 0 2022-01-31T19:44:15.134 INFO:tasks.workunit.client.0.smithi167.stdout:6/517: symlink d8/d33/d51/d5d/d86/lc0 0 2022-01-31T19:44:15.140 INFO:tasks.workunit.client.0.smithi167.stdout:6/518: dread d8/d62/f6f [0,4194304] 0 2022-01-31T19:44:15.141 INFO:tasks.workunit.client.0.smithi167.stdout:6/519: chown d8/d33/l3a 130537588 1 2022-01-31T19:44:15.141 INFO:tasks.workunit.client.0.smithi167.stdout:6/520: chown d8/d20/d54/f63 5259 1 2022-01-31T19:44:15.142 INFO:tasks.workunit.client.0.smithi167.stdout:2/505: dread dd/f23 [0,4194304] 0 2022-01-31T19:44:15.144 INFO:tasks.workunit.client.0.smithi167.stdout:2/506: mknod dd/d18/d62/d75/c9f 0 2022-01-31T19:44:15.146 INFO:tasks.workunit.client.0.smithi167.stdout:7/828: getdents d8/d2c/d6e/d4d/d67/d8f 0 2022-01-31T19:44:15.147 INFO:tasks.workunit.client.0.smithi167.stdout:7/829: read d8/d2c/d6e/f44 [3073211,123663] 0 2022-01-31T19:44:15.150 INFO:tasks.workunit.client.0.smithi167.stdout:7/830: symlink d8/dc/d18/d24/dcc/l12c 0 2022-01-31T19:44:15.150 INFO:tasks.workunit.client.0.smithi167.stdout:6/521: dread d8/f1a [0,4194304] 0 2022-01-31T19:44:15.151 INFO:tasks.workunit.client.0.smithi167.stdout:6/522: dread - d8/d33/d4f/f59 zero size 2022-01-31T19:44:15.159 INFO:tasks.workunit.client.0.smithi167.stdout:1/739: dwrite d0/d10/f19 [0,4194304] 0 2022-01-31T19:44:15.160 INFO:tasks.workunit.client.0.smithi167.stdout:1/740: truncate d0/d10/d20/d7c/ff4 166377 0 2022-01-31T19:44:15.161 INFO:tasks.workunit.client.0.smithi167.stdout:2/507: rmdir dd 39 2022-01-31T19:44:15.162 INFO:tasks.workunit.client.0.smithi167.stdout:3/650: dwrite d9/d29/d60/f49 [0,4194304] 0 2022-01-31T19:44:15.163 INFO:tasks.workunit.client.0.smithi167.stdout:6/523: truncate d8/d62/f6d 3339354 0 2022-01-31T19:44:15.164 INFO:tasks.workunit.client.0.smithi167.stdout:1/741: symlink d0/d62/d7f/lf7 0 2022-01-31T19:44:15.164 INFO:tasks.workunit.client.0.smithi167.stdout:7/831: dwrite d8/dc/f23 [4194304,4194304] 0 2022-01-31T19:44:15.167 INFO:tasks.workunit.client.0.smithi167.stdout:6/524: rmdir d8/d33/d51/d5d/d86/d94/db4 0 2022-01-31T19:44:15.167 INFO:tasks.workunit.client.0.smithi167.stdout:6/525: chown d8/d33/d51/d5d 12746149 1 2022-01-31T19:44:15.168 INFO:tasks.workunit.client.0.smithi167.stdout:1/742: mknod d0/d62/db7/cf8 0 2022-01-31T19:44:15.168 INFO:tasks.workunit.client.0.smithi167.stdout:1/743: creat d0/d75/d7a/ff9 x:0 0 0 2022-01-31T19:44:15.169 INFO:tasks.workunit.client.0.smithi167.stdout:7/832: mkdir d8/d2c/d6e/d4d/d12d 0 2022-01-31T19:44:15.172 INFO:tasks.workunit.client.0.smithi167.stdout:3/651: rmdir d9/d29/d3b/d41/d6b/d89 39 2022-01-31T19:44:15.174 INFO:tasks.workunit.client.0.smithi167.stdout:3/652: dread d9/f30 [0,4194304] 0 2022-01-31T19:44:15.177 INFO:tasks.workunit.client.0.smithi167.stdout:1/744: rename d0/d10/d5a/c31 to d0/db3/cfa 0 2022-01-31T19:44:15.177 INFO:tasks.workunit.client.0.smithi167.stdout:1/745: readlink d0/d75/d7a/l6a 0 2022-01-31T19:44:15.178 INFO:tasks.workunit.client.0.smithi167.stdout:1/746: fdatasync d0/d62/d7f/fdd 0 2022-01-31T19:44:15.178 INFO:tasks.workunit.client.0.smithi167.stdout:7/833: mkdir d8/d2c/d43/d94/d12e 0 2022-01-31T19:44:15.179 INFO:tasks.workunit.client.0.smithi167.stdout:7/834: chown d8/dc/dc1 125774135 1 2022-01-31T19:44:15.179 INFO:tasks.workunit.client.0.smithi167.stdout:7/835: getdents d8/d2c/d7a 0 2022-01-31T19:44:15.182 INFO:tasks.workunit.client.0.smithi167.stdout:7/836: mkdir d8/d2c/d6e/d4d/d67/d12f 0 2022-01-31T19:44:15.183 INFO:tasks.workunit.client.0.smithi167.stdout:1/747: dread d0/d11/fab [0,4194304] 0 2022-01-31T19:44:15.185 INFO:tasks.workunit.client.0.smithi167.stdout:1/748: mknod d0/d10/d5a/d33/cfb 0 2022-01-31T19:44:15.185 INFO:tasks.workunit.client.0.smithi167.stdout:1/749: read - d0/d75/d7a/ff9 zero size 2022-01-31T19:44:15.185 INFO:tasks.workunit.client.0.smithi167.stdout:7/837: mkdir d8/d2c/d6e/d4d/d5e/d71/db3/d130 0 2022-01-31T19:44:15.187 INFO:tasks.workunit.client.0.smithi167.stdout:7/838: symlink d8/dab/dd0/l131 0 2022-01-31T19:44:15.190 INFO:tasks.workunit.client.0.smithi167.stdout:2/508: write dd/df/d22/f9d [273958,45934] 0 2022-01-31T19:44:15.190 INFO:tasks.workunit.client.0.smithi167.stdout:2/509: read - dd/d18/d63/f32 zero size 2022-01-31T19:44:15.191 INFO:tasks.workunit.client.0.smithi167.stdout:1/750: read d0/f3a [289591,110762] 0 2022-01-31T19:44:15.191 INFO:tasks.workunit.client.0.smithi167.stdout:3/653: dwrite d9/f79 [0,4194304] 0 2022-01-31T19:44:15.192 INFO:tasks.workunit.client.0.smithi167.stdout:6/526: dwrite d8/d20/f26 [0,4194304] 0 2022-01-31T19:44:15.193 INFO:tasks.workunit.client.0.smithi167.stdout:2/510: mknod dd/df/d1b/d2f/ca0 0 2022-01-31T19:44:15.193 INFO:tasks.workunit.client.0.smithi167.stdout:2/511: chown dd/df 2 1 2022-01-31T19:44:15.194 INFO:tasks.workunit.client.0.smithi167.stdout:1/751: rename d0/d10/f42 to d0/d10/d5a/d33/d4b/da5/da9/ffc 0 2022-01-31T19:44:15.195 INFO:tasks.workunit.client.0.smithi167.stdout:1/752: creat d0/d62/ffd x:0 0 0 2022-01-31T19:44:15.195 INFO:tasks.workunit.client.0.smithi167.stdout:6/527: mknod d8/d33/d51/d7e/cc1 0 2022-01-31T19:44:15.198 INFO:tasks.workunit.client.0.smithi167.stdout:2/512: link dd/df/d3f/f41 dd/d18/d63/fa1 0 2022-01-31T19:44:15.198 INFO:tasks.workunit.client.0.smithi167.stdout:2/513: getdents dd/df/d22/d96/d9e 0 2022-01-31T19:44:15.199 INFO:tasks.workunit.client.0.smithi167.stdout:1/753: symlink d0/d10/d18/lfe 0 2022-01-31T19:44:15.199 INFO:tasks.workunit.client.0.smithi167.stdout:1/754: truncate d0/f72 4904983 0 2022-01-31T19:44:15.202 INFO:tasks.workunit.client.0.smithi167.stdout:3/654: dread d9/d21/f77 [0,4194304] 0 2022-01-31T19:44:15.203 INFO:tasks.workunit.client.0.smithi167.stdout:2/514: symlink dd/df/d1b/d3d/d4a/la2 0 2022-01-31T19:44:15.205 INFO:tasks.workunit.client.0.smithi167.stdout:1/755: rmdir d0/d10/d5a/d33/d4b/da5/da9/dad 39 2022-01-31T19:44:15.206 INFO:tasks.workunit.client.0.smithi167.stdout:1/756: stat d0/d10/d5a/d33/d4b/f65 0 2022-01-31T19:44:15.206 INFO:tasks.workunit.client.0.smithi167.stdout:1/757: chown d0/d10/d5a/d33/d4b/da5/dc1 8770 1 2022-01-31T19:44:15.206 INFO:tasks.workunit.client.0.smithi167.stdout:1/758: truncate d0/f72 5780960 0 2022-01-31T19:44:15.207 INFO:tasks.workunit.client.0.smithi167.stdout:1/759: write d0/d11/fab [213084,81455] 0 2022-01-31T19:44:15.208 INFO:tasks.workunit.client.0.smithi167.stdout:1/760: write d0/d10/d5a/d33/d4b/d1f/f7b [1839377,7652] 0 2022-01-31T19:44:15.208 INFO:tasks.workunit.client.0.smithi167.stdout:1/761: fsync d0/d10/d20/f28 0 2022-01-31T19:44:15.208 INFO:tasks.workunit.client.0.smithi167.stdout:1/762: fsync d0/d10/d5a/f50 0 2022-01-31T19:44:15.208 INFO:tasks.workunit.client.0.smithi167.stdout:7/839: dwrite d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f1a [0,4194304] 0 2022-01-31T19:44:15.209 INFO:tasks.workunit.client.0.smithi167.stdout:3/655: mkdir d9/d29/dbd 0 2022-01-31T19:44:15.209 INFO:tasks.workunit.client.0.smithi167.stdout:3/656: write d9/d56/f73 [5404241,116475] 0 2022-01-31T19:44:15.209 INFO:tasks.workunit.client.0.smithi167.stdout:2/515: chown dd/d18/d62/f86 10784818 1 2022-01-31T19:44:15.210 INFO:tasks.workunit.client.0.smithi167.stdout:1/763: creat d0/d10/d5a/d33/d4b/da5/da9/dad/fff x:0 0 0 2022-01-31T19:44:15.215 INFO:tasks.workunit.client.0.smithi167.stdout:6/528: dwrite d8/d33/d51/d5d/d86/d94/f96 [4194304,4194304] 0 2022-01-31T19:44:15.216 INFO:tasks.workunit.client.0.smithi167.stdout:7/840: rmdir d8/d2c/d6e/d4d/d67/d8f 39 2022-01-31T19:44:15.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/657: mkdir d9/d7f/dbe 0 2022-01-31T19:44:15.217 INFO:tasks.workunit.client.0.smithi167.stdout:2/516: rename dd/df/d1b/d3d/d4a/d4b/d54/f60 to dd/d18/d63/fa3 0 2022-01-31T19:44:15.219 INFO:tasks.workunit.client.0.smithi167.stdout:6/529: mkdir d8/d33/d51/d72/dc2 0 2022-01-31T19:44:15.219 INFO:tasks.workunit.client.0.smithi167.stdout:6/530: write d8/d33/d51/d5d/f80 [287731,103104] 0 2022-01-31T19:44:15.220 INFO:tasks.workunit.client.0.smithi167.stdout:7/841: mkdir d8/d2c/d43/d94/daf/dee/df8/d132 0 2022-01-31T19:44:15.221 INFO:tasks.workunit.client.0.smithi167.stdout:7/842: write d8/d2c/d6e/f62 [5243078,24333] 0 2022-01-31T19:44:15.223 INFO:tasks.workunit.client.0.smithi167.stdout:2/517: rename dd/df/l94 to dd/d18/d63/la4 0 2022-01-31T19:44:15.225 INFO:tasks.workunit.client.0.smithi167.stdout:7/843: rename d8/d2c/d43/d94/daf/fe6 to d8/dc/d18/d24/dcc/f133 0 2022-01-31T19:44:15.228 INFO:tasks.workunit.client.0.smithi167.stdout:3/658: dread d9/d29/f2b [0,4194304] 0 2022-01-31T19:44:15.229 INFO:tasks.workunit.client.0.smithi167.stdout:3/659: mknod d9/d29/d3b/d41/d6b/cbf 0 2022-01-31T19:44:15.230 INFO:tasks.workunit.client.0.smithi167.stdout:7/844: dread d8/d2c/d43/f97 [0,4194304] 0 2022-01-31T19:44:15.230 INFO:tasks.workunit.client.0.smithi167.stdout:1/764: dwrite d0/d10/d5a/d33/d4b/d1f/fdc [0,4194304] 0 2022-01-31T19:44:15.235 INFO:tasks.workunit.client.0.smithi167.stdout:9/595: sync 2022-01-31T19:44:15.235 INFO:tasks.workunit.client.0.smithi167.stdout:8/675: sync 2022-01-31T19:44:15.235 INFO:tasks.workunit.client.0.smithi167.stdout:5/756: sync 2022-01-31T19:44:15.239 INFO:tasks.workunit.client.0.smithi167.stdout:7/845: creat d8/d2c/d7a/dd3/f134 x:0 0 0 2022-01-31T19:44:15.242 INFO:tasks.workunit.client.0.smithi167.stdout:1/765: truncate d0/d62/fa3 713985 0 2022-01-31T19:44:15.243 INFO:tasks.workunit.client.0.smithi167.stdout:8/676: creat d6/db/d28/d61/dd5/fdf x:0 0 0 2022-01-31T19:44:15.245 INFO:tasks.workunit.client.0.smithi167.stdout:5/757: link d5/d6/d3a/cb2 d5/ce7 0 2022-01-31T19:44:15.247 INFO:tasks.workunit.client.0.smithi167.stdout:7/846: link d8/d35/f4b d8/d2c/d43/d94/daf/f135 0 2022-01-31T19:44:15.249 INFO:tasks.workunit.client.0.smithi167.stdout:1/766: mkdir d0/d100 0 2022-01-31T19:44:15.251 INFO:tasks.workunit.client.0.smithi167.stdout:5/758: rename d5/d6/d16/d18/d43/fd4 to d5/d6/d16/d18/d43/fe8 0 2022-01-31T19:44:15.253 INFO:tasks.workunit.client.0.smithi167.stdout:9/596: dread dd/f10 [0,4194304] 0 2022-01-31T19:44:15.254 INFO:tasks.workunit.client.0.smithi167.stdout:7/847: dread d8/d2c/d6e/d4d/d6b/f81 [0,4194304] 0 2022-01-31T19:44:15.259 INFO:tasks.workunit.client.0.smithi167.stdout:7/848: read d8/d2c/d6e/d4d/d67/d8f/f96 [649463,124394] 0 2022-01-31T19:44:15.259 INFO:tasks.workunit.client.0.smithi167.stdout:2/518: dwrite dd/d18/d62/f64 [0,4194304] 0 2022-01-31T19:44:15.260 INFO:tasks.workunit.client.0.smithi167.stdout:7/849: dread - d8/d2c/d6e/d4d/d5e/d71/db3/f129 zero size 2022-01-31T19:44:15.260 INFO:tasks.workunit.client.0.smithi167.stdout:7/850: creat d8/dc/d18/d24/dbb/f136 x:0 0 0 2022-01-31T19:44:15.260 INFO:tasks.workunit.client.0.smithi167.stdout:7/851: fdatasync d8/d2c/d6e/d4d/d67/fb9 0 2022-01-31T19:44:15.260 INFO:tasks.workunit.client.0.smithi167.stdout:7/852: chown d8/d2c/d7a/dd3 0 1 2022-01-31T19:44:15.260 INFO:tasks.workunit.client.0.smithi167.stdout:7/853: read d8/fa7 [58448,16111] 0 2022-01-31T19:44:15.261 INFO:tasks.workunit.client.0.smithi167.stdout:7/854: chown d8/d2c/d43/df3 9694772 1 2022-01-31T19:44:15.261 INFO:tasks.workunit.client.0.smithi167.stdout:9/597: rmdir dd/d17/d7f/dbf 0 2022-01-31T19:44:15.267 INFO:tasks.workunit.client.0.smithi167.stdout:6/531: dwrite d8/d20/f42 [0,4194304] 0 2022-01-31T19:44:15.268 INFO:tasks.workunit.client.0.smithi167.stdout:6/532: chown d8/d33/d51/d5d/d64/dab/db9 11436742 1 2022-01-31T19:44:15.268 INFO:tasks.workunit.client.0.smithi167.stdout:6/533: write d8/d33/d4f/d87/fb2 [219499,43847] 0 2022-01-31T19:44:15.268 INFO:tasks.workunit.client.0.smithi167.stdout:6/534: dread - d8/d20/fa6 zero size 2022-01-31T19:44:15.268 INFO:tasks.workunit.client.0.smithi167.stdout:5/759: dread d5/d6/d16/d18/d43/fe8 [0,4194304] 0 2022-01-31T19:44:15.269 INFO:tasks.workunit.client.0.smithi167.stdout:4/628: sync 2022-01-31T19:44:15.269 INFO:tasks.workunit.client.0.smithi167.stdout:4/629: stat d0/d6/d7/d26/d40/d4c/c57 0 2022-01-31T19:44:15.269 INFO:tasks.workunit.client.0.smithi167.stdout:2/519: unlink dd/df/d1b/d3d/d4a/c81 0 2022-01-31T19:44:15.270 INFO:tasks.workunit.client.0.smithi167.stdout:3/660: dwrite d9/d21/fa4 [0,4194304] 0 2022-01-31T19:44:15.271 INFO:tasks.workunit.client.0.smithi167.stdout:9/598: creat dd/d19/d24/d56/fc7 x:0 0 0 2022-01-31T19:44:15.271 INFO:tasks.workunit.client.0.smithi167.stdout:9/599: write dd/d35/f9a [1235716,119709] 0 2022-01-31T19:44:15.274 INFO:tasks.workunit.client.0.smithi167.stdout:8/677: dwrite d6/db/de/d18/da4/dc2/dc3/fc5 [0,4194304] 0 2022-01-31T19:44:15.275 INFO:tasks.workunit.client.0.smithi167.stdout:6/535: rename d8/d33/d35/f3d to d8/d33/d51/d5d/fc3 0 2022-01-31T19:44:15.275 INFO:tasks.workunit.client.0.smithi167.stdout:6/536: chown d8/fac 623250551 1 2022-01-31T19:44:15.275 INFO:tasks.workunit.client.0.smithi167.stdout:5/760: mknod d5/d6/d16/d18/d5b/d9f/ce9 0 2022-01-31T19:44:15.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/520: creat dd/d18/d8d/fa5 x:0 0 0 2022-01-31T19:44:15.280 INFO:tasks.workunit.client.0.smithi167.stdout:0/418: sync 2022-01-31T19:44:15.281 INFO:tasks.workunit.client.0.smithi167.stdout:0/419: chown d17/d19/d1c/d39/d7b/f81 169 1 2022-01-31T19:44:15.281 INFO:tasks.workunit.client.0.smithi167.stdout:0/420: readlink d17/d19/l20 0 2022-01-31T19:44:15.285 INFO:tasks.workunit.client.0.smithi167.stdout:7/855: dwrite d8/dc/d18/d24/dcc/f133 [0,4194304] 0 2022-01-31T19:44:15.287 INFO:tasks.workunit.client.0.smithi167.stdout:2/521: unlink dd/c43 0 2022-01-31T19:44:15.287 INFO:tasks.workunit.client.0.smithi167.stdout:2/522: chown dd/df/d1b/d2f/ca0 1802638 1 2022-01-31T19:44:15.288 INFO:tasks.workunit.client.0.smithi167.stdout:1/767: dwrite d0/d10/d20/fb2 [0,4194304] 0 2022-01-31T19:44:15.288 INFO:tasks.workunit.client.0.smithi167.stdout:0/421: creat d17/d3e/d7e/f83 x:0 0 0 2022-01-31T19:44:15.289 INFO:tasks.workunit.client.0.smithi167.stdout:8/678: rename d6/db/de/d3d/f6f to d6/db/de/d3d/dd6/fe0 0 2022-01-31T19:44:15.293 INFO:tasks.workunit.client.0.smithi167.stdout:8/679: write d6/db/de/d18/f19 [4180219,27980] 0 2022-01-31T19:44:15.294 INFO:tasks.workunit.client.0.smithi167.stdout:7/856: mknod d8/d2c/d6e/d4d/d5e/d71/db3/c137 0 2022-01-31T19:44:15.294 INFO:tasks.workunit.client.0.smithi167.stdout:7/857: truncate d8/d2c/d6e/d4d/d67/d8f/f12b 593078 0 2022-01-31T19:44:15.297 INFO:tasks.workunit.client.0.smithi167.stdout:7/858: read d8/dc/f40 [5151062,19542] 0 2022-01-31T19:44:15.300 INFO:tasks.workunit.client.0.smithi167.stdout:2/523: truncate f7 1261079 0 2022-01-31T19:44:15.307 INFO:tasks.workunit.client.0.smithi167.stdout:5/761: dwrite d5/d6/de/d22/f97 [0,4194304] 0 2022-01-31T19:44:15.307 INFO:tasks.workunit.client.0.smithi167.stdout:4/630: dwrite d0/d6/d7/d39/d4e/d61/f95 [0,4194304] 0 2022-01-31T19:44:15.307 INFO:tasks.workunit.client.0.smithi167.stdout:0/422: rename d17/d69/l77 to d17/d19/d1c/d39/d44/l84 0 2022-01-31T19:44:15.308 INFO:tasks.workunit.client.0.smithi167.stdout:8/680: rmdir d6/db/d4e/dbd 39 2022-01-31T19:44:15.308 INFO:tasks.workunit.client.0.smithi167.stdout:8/681: fsync d6/db/d36/f84 0 2022-01-31T19:44:15.308 INFO:tasks.workunit.client.0.smithi167.stdout:2/524: truncate dd/df/d1b/d2f/d36/f2a 3958463 0 2022-01-31T19:44:15.310 INFO:tasks.workunit.client.0.smithi167.stdout:5/762: creat d5/d6/d16/d18/d43/d45/dd5/dbc/dc0/fea x:0 0 0 2022-01-31T19:44:15.312 INFO:tasks.workunit.client.0.smithi167.stdout:0/423: mkdir d17/d19/d1c/d39/d7b/d85 0 2022-01-31T19:44:15.314 INFO:tasks.workunit.client.0.smithi167.stdout:8/682: rename d6/db/d4e/d73/c92 to d6/d1e/d2d/d6b/ce1 0 2022-01-31T19:44:15.314 INFO:tasks.workunit.client.0.smithi167.stdout:8/683: write d6/db/d4e/fd0 [502814,21299] 0 2022-01-31T19:44:15.315 INFO:tasks.workunit.client.0.smithi167.stdout:2/525: creat dd/d18/d62/fa6 x:0 0 0 2022-01-31T19:44:15.315 INFO:tasks.workunit.client.0.smithi167.stdout:2/526: chown dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b 996639971 1 2022-01-31T19:44:15.316 INFO:tasks.workunit.client.0.smithi167.stdout:5/763: symlink d5/dcd/leb 0 2022-01-31T19:44:15.316 INFO:tasks.workunit.client.0.smithi167.stdout:4/631: getdents d0/d6/d7/d39/d4e/d61 0 2022-01-31T19:44:15.317 INFO:tasks.workunit.client.0.smithi167.stdout:8/684: symlink d6/db/d36/d5d/le2 0 2022-01-31T19:44:15.319 INFO:tasks.workunit.client.0.smithi167.stdout:2/527: mkdir dd/df/d1b/d3d/d4a/da7 0 2022-01-31T19:44:15.319 INFO:tasks.workunit.client.0.smithi167.stdout:2/528: fsync dd/d18/d62/f64 0 2022-01-31T19:44:15.321 INFO:tasks.workunit.client.0.smithi167.stdout:5/764: rename d5/d6/d16/d18/d43/fe8 to d5/d6/d16/d18/d43/d45/fec 0 2022-01-31T19:44:15.323 INFO:tasks.workunit.client.0.smithi167.stdout:8/685: mkdir d6/db/d28/d61/de3 0 2022-01-31T19:44:15.324 INFO:tasks.workunit.client.0.smithi167.stdout:8/686: creat d6/db/de/d18/da4/dc2/dc3/fe4 x:0 0 0 2022-01-31T19:44:15.324 INFO:tasks.workunit.client.0.smithi167.stdout:8/687: chown d6/db/d28/f76 20 1 2022-01-31T19:44:15.324 INFO:tasks.workunit.client.0.smithi167.stdout:8/688: fdatasync d6/db/de/d3d/fa7 0 2022-01-31T19:44:15.325 INFO:tasks.workunit.client.0.smithi167.stdout:8/689: creat d6/db/de/d18/da4/dc2/dc3/fe5 x:0 0 0 2022-01-31T19:44:15.325 INFO:tasks.workunit.client.0.smithi167.stdout:8/690: write d6/db/f2b [9435287,74276] 0 2022-01-31T19:44:15.325 INFO:tasks.workunit.client.0.smithi167.stdout:8/691: stat d6/db/d36/d5d/fd7 0 2022-01-31T19:44:15.327 INFO:tasks.workunit.client.0.smithi167.stdout:8/692: creat d6/d1e/dcf/fe6 x:0 0 0 2022-01-31T19:44:15.328 INFO:tasks.workunit.client.0.smithi167.stdout:2/529: dread dd/d18/d62/f64 [0,4194304] 0 2022-01-31T19:44:15.330 INFO:tasks.workunit.client.0.smithi167.stdout:2/530: mknod dd/df/d1b/d2f/d36/ca8 0 2022-01-31T19:44:15.330 INFO:tasks.workunit.client.0.smithi167.stdout:6/537: dwrite d8/d33/d51/d5d/f6e [0,4194304] 0 2022-01-31T19:44:15.331 INFO:tasks.workunit.client.0.smithi167.stdout:8/693: write d6/d1e/d4c/dba/fd4 [1046957,113097] 0 2022-01-31T19:44:15.331 INFO:tasks.workunit.client.0.smithi167.stdout:8/694: write d6/db/de/f3e [2500963,62491] 0 2022-01-31T19:44:15.332 INFO:tasks.workunit.client.0.smithi167.stdout:1/768: dwrite d0/f30 [0,4194304] 0 2022-01-31T19:44:15.334 INFO:tasks.workunit.client.0.smithi167.stdout:3/661: dwrite d9/d29/f36 [0,4194304] 0 2022-01-31T19:44:15.337 INFO:tasks.workunit.client.0.smithi167.stdout:2/531: read dd/d18/d63/fa1 [588957,42392] 0 2022-01-31T19:44:15.337 INFO:tasks.workunit.client.0.smithi167.stdout:5/765: dread d5/d6/d16/d18/d43/d45/d69/dc4/f7c [4194304,4194304] 0 2022-01-31T19:44:15.337 INFO:tasks.workunit.client.0.smithi167.stdout:7/859: dwrite d8/d2c/d6e/d4d/d67/d8f/dcb/f123 [0,4194304] 0 2022-01-31T19:44:15.338 INFO:tasks.workunit.client.0.smithi167.stdout:3/662: truncate d9/d21/f77 1211396 0 2022-01-31T19:44:15.342 INFO:tasks.workunit.client.0.smithi167.stdout:6/538: rename d8/d33/d51/d5d/d86/d94 to d8/d33/d51/d7e/dc4 0 2022-01-31T19:44:15.343 INFO:tasks.workunit.client.0.smithi167.stdout:5/766: write d5/d6/d3a/f17 [233814,105040] 0 2022-01-31T19:44:15.344 INFO:tasks.workunit.client.0.smithi167.stdout:5/767: readlink d5/d6/de/d22/d6d/l73 0 2022-01-31T19:44:15.344 INFO:tasks.workunit.client.0.smithi167.stdout:5/768: read - d5/d6/de/d22/d6d/fcb zero size 2022-01-31T19:44:15.345 INFO:tasks.workunit.client.0.smithi167.stdout:7/860: unlink d8/d2c/d7a/cdf 0 2022-01-31T19:44:15.345 INFO:tasks.workunit.client.0.smithi167.stdout:3/663: write d9/d21/f1e [2223301,55441] 0 2022-01-31T19:44:15.345 INFO:tasks.workunit.client.0.smithi167.stdout:3/664: chown d9/fc 8074 1 2022-01-31T19:44:15.345 INFO:tasks.workunit.client.0.smithi167.stdout:3/665: dread - d9/d29/d3b/d41/d6b/d89/f97 zero size 2022-01-31T19:44:15.347 INFO:tasks.workunit.client.0.smithi167.stdout:6/539: rename d8/d33/c3e to d8/d33/cc5 0 2022-01-31T19:44:15.347 INFO:tasks.workunit.client.0.smithi167.stdout:4/632: dwrite d0/d6/d7/f30 [0,4194304] 0 2022-01-31T19:44:15.347 INFO:tasks.workunit.client.0.smithi167.stdout:4/633: dread - d0/d6/d7/d89/fad zero size 2022-01-31T19:44:15.347 INFO:tasks.workunit.client.0.smithi167.stdout:7/861: read d8/dc/f33 [1608201,97026] 0 2022-01-31T19:44:15.348 INFO:tasks.workunit.client.0.smithi167.stdout:7/862: chown d8/d2c/d6e/d9a/lb5 342331 1 2022-01-31T19:44:15.348 INFO:tasks.workunit.client.0.smithi167.stdout:7/863: fsync d8/d2c/d6e/d9a/f10c 0 2022-01-31T19:44:15.350 INFO:tasks.workunit.client.0.smithi167.stdout:5/769: mkdir d5/d6/d16/d18/d43/d45/dd5/dbc/dcc/ded 0 2022-01-31T19:44:15.351 INFO:tasks.workunit.client.0.smithi167.stdout:4/634: write d0/d6/d7/f30 [648413,29485] 0 2022-01-31T19:44:15.353 INFO:tasks.workunit.client.0.smithi167.stdout:7/864: link d8/dc/f2b d8/d2c/d6e/d4d/d5e/d71/db3/f138 0 2022-01-31T19:44:15.354 INFO:tasks.workunit.client.0.smithi167.stdout:5/770: mkdir d5/d6/de/d22/d76/db0/dee 0 2022-01-31T19:44:15.356 INFO:tasks.workunit.client.0.smithi167.stdout:7/865: unlink d8/d2c/d7a/d99/cfa 0 2022-01-31T19:44:15.357 INFO:tasks.workunit.client.0.smithi167.stdout:7/866: write d8/d2c/d43/d94/daf/f102 [483787,78618] 0 2022-01-31T19:44:15.357 INFO:tasks.workunit.client.0.smithi167.stdout:7/867: chown d8/d2c/d6e/ff9 110985070 1 2022-01-31T19:44:15.358 INFO:tasks.workunit.client.0.smithi167.stdout:5/771: rmdir d5/d6/de 39 2022-01-31T19:44:15.358 INFO:tasks.workunit.client.0.smithi167.stdout:5/772: symlink d5/d6/de/d22/lef 0 2022-01-31T19:44:15.363 INFO:tasks.workunit.client.0.smithi167.stdout:5/773: unlink d5/d6/de/d22/d6d/da0/da4/fce 0 2022-01-31T19:44:15.365 INFO:tasks.workunit.client.0.smithi167.stdout:4/635: dread d0/d6/d7/d39/f37 [0,4194304] 0 2022-01-31T19:44:15.365 INFO:tasks.workunit.client.0.smithi167.stdout:4/636: truncate d0/d6/d7/d39/d3a/f48 1934788 0 2022-01-31T19:44:15.366 INFO:tasks.workunit.client.0.smithi167.stdout:5/774: mknod d5/d6/d16/cf0 0 2022-01-31T19:44:15.366 INFO:tasks.workunit.client.0.smithi167.stdout:5/775: fdatasync d5/d6/de/d22/d4d/fa5 0 2022-01-31T19:44:15.372 INFO:tasks.workunit.client.0.smithi167.stdout:4/637: rename d0/d3f/d60 to d0/d3f/da2/dcf 0 2022-01-31T19:44:15.372 INFO:tasks.workunit.client.0.smithi167.stdout:4/638: write d0/d3f/da2/dcf/db5/fcb [714106,2683] 0 2022-01-31T19:44:15.374 INFO:tasks.workunit.client.0.smithi167.stdout:8/695: dwrite d6/db/de/f20 [0,4194304] 0 2022-01-31T19:44:15.374 INFO:tasks.workunit.client.0.smithi167.stdout:8/696: readlink d6/db/d6a/d8d/d9e/d60/la8 0 2022-01-31T19:44:15.375 INFO:tasks.workunit.client.0.smithi167.stdout:2/532: dwrite dd/df/f50 [0,4194304] 0 2022-01-31T19:44:15.375 INFO:tasks.workunit.client.0.smithi167.stdout:5/776: rename d5/d6/de/d22/d76/ca7 to d5/d6/de/d22/d6d/da0/da4/dca/cf1 0 2022-01-31T19:44:15.376 INFO:tasks.workunit.client.0.smithi167.stdout:6/540: dwrite d8/d20/f2f [0,4194304] 0 2022-01-31T19:44:15.376 INFO:tasks.workunit.client.0.smithi167.stdout:8/697: symlink d6/db/d28/d61/dd5/le7 0 2022-01-31T19:44:15.376 INFO:tasks.workunit.client.0.smithi167.stdout:8/698: write d6/db/d6a/d8d/d9e/fcc [925130,15831] 0 2022-01-31T19:44:15.377 INFO:tasks.workunit.client.0.smithi167.stdout:7/868: dwrite d8/d2c/d6e/f2a [0,4194304] 0 2022-01-31T19:44:15.381 INFO:tasks.workunit.client.0.smithi167.stdout:3/666: dread d9/d21/f1e [0,4194304] 0 2022-01-31T19:44:15.383 INFO:tasks.workunit.client.0.smithi167.stdout:2/533: link dd/d18/f4e dd/d18/fa9 0 2022-01-31T19:44:15.387 INFO:tasks.workunit.client.0.smithi167.stdout:5/777: rename d5/d6/de/d22/d4d/fa5 to d5/d6/d16/d18/d43/d45/d69/dc4/dbb/ff2 0 2022-01-31T19:44:15.388 INFO:tasks.workunit.client.0.smithi167.stdout:5/778: stat d5/d6/de/d22/d4d/l7a 0 2022-01-31T19:44:15.389 INFO:tasks.workunit.client.0.smithi167.stdout:4/639: dwrite d0/d6/d7/d26/d40/f24 [4194304,4194304] 0 2022-01-31T19:44:15.391 INFO:tasks.workunit.client.0.smithi167.stdout:6/541: creat d8/d62/fc6 x:0 0 0 2022-01-31T19:44:15.392 INFO:tasks.workunit.client.0.smithi167.stdout:7/869: creat d8/d2c/d6e/d9a/dd8/f139 x:0 0 0 2022-01-31T19:44:15.406 INFO:tasks.workunit.client.0.smithi167.stdout:3/667: creat d9/d29/dbd/fc0 x:0 0 0 2022-01-31T19:44:15.407 INFO:tasks.workunit.client.0.smithi167.stdout:3/668: creat d9/fc1 x:0 0 0 2022-01-31T19:44:15.407 INFO:tasks.workunit.client.0.smithi167.stdout:3/669: chown d9/f79 14 1 2022-01-31T19:44:15.407 INFO:tasks.workunit.client.0.smithi167.stdout:3/670: unlink d9/d21/f1e 0 2022-01-31T19:44:15.408 INFO:tasks.workunit.client.0.smithi167.stdout:5/779: unlink d5/d6/de/d22/f54 0 2022-01-31T19:44:15.410 INFO:tasks.workunit.client.0.smithi167.stdout:4/640: rename d0/d6/d7/d39/d93/lb0 to d0/d6/d7/d89/ld0 0 2022-01-31T19:44:15.412 INFO:tasks.workunit.client.0.smithi167.stdout:8/699: truncate d6/db/d36/f3b 3392875 0 2022-01-31T19:44:15.418 INFO:tasks.workunit.client.0.smithi167.stdout:3/671: link d9/d29/c87 d9/d29/d3b/d41/d6b/cc2 0 2022-01-31T19:44:15.420 INFO:tasks.workunit.client.0.smithi167.stdout:5/780: symlink d5/d6/de/d22/d76/db0/dd8/lf3 0 2022-01-31T19:44:15.421 INFO:tasks.workunit.client.0.smithi167.stdout:6/542: getdents d8/d33/d51/d5d/d64/dab 0 2022-01-31T19:44:15.421 INFO:tasks.workunit.client.0.smithi167.stdout:6/543: dread - d8/d33/d35/f8b zero size 2022-01-31T19:44:15.422 INFO:tasks.workunit.client.0.smithi167.stdout:3/672: write d9/d29/f33 [306937,74808] 0 2022-01-31T19:44:15.423 INFO:tasks.workunit.client.0.smithi167.stdout:5/781: write d5/d6/f28 [3552566,48547] 0 2022-01-31T19:44:15.423 INFO:tasks.workunit.client.0.smithi167.stdout:5/782: fsync d5/d6/d16/d18/f2c 0 2022-01-31T19:44:15.423 INFO:tasks.workunit.client.0.smithi167.stdout:5/783: chown d5/d6/d16/d18/f2c 711 1 2022-01-31T19:44:15.428 INFO:tasks.workunit.client.0.smithi167.stdout:7/870: rmdir d8/d2c/d6e/d4d/d5e/d71 39 2022-01-31T19:44:15.430 INFO:tasks.workunit.client.0.smithi167.stdout:6/544: mkdir d8/d33/d51/d72/dc7 0 2022-01-31T19:44:15.431 INFO:tasks.workunit.client.0.smithi167.stdout:3/673: truncate d9/d29/d3b/d41/d6b/f39 709399 0 2022-01-31T19:44:15.431 INFO:tasks.workunit.client.0.smithi167.stdout:3/674: dread - d9/d29/d3b/d41/d6b/d89/fa5 zero size 2022-01-31T19:44:15.432 INFO:tasks.workunit.client.0.smithi167.stdout:6/545: symlink d8/d33/lc8 0 2022-01-31T19:44:15.434 INFO:tasks.workunit.client.0.smithi167.stdout:4/641: dwrite d0/d3f/f3d [0,4194304] 0 2022-01-31T19:44:15.435 INFO:tasks.workunit.client.0.smithi167.stdout:4/642: write d0/d6/d7/d26/d40/d4c/f9c [5472746,21652] 0 2022-01-31T19:44:15.435 INFO:tasks.workunit.client.0.smithi167.stdout:4/643: getdents d0/d6/d7/d39/d3a/dc8 0 2022-01-31T19:44:15.435 INFO:tasks.workunit.client.0.smithi167.stdout:4/644: stat d0/d3f/f2e 0 2022-01-31T19:44:15.436 INFO:tasks.workunit.client.0.smithi167.stdout:3/675: unlink d9/d29/d9d/la2 0 2022-01-31T19:44:15.436 INFO:tasks.workunit.client.0.smithi167.stdout:3/676: creat d9/d26/fc3 x:0 0 0 2022-01-31T19:44:15.437 INFO:tasks.workunit.client.0.smithi167.stdout:8/700: dwrite d6/db/de/d18/da4/dc2/dc3/fe5 [0,4194304] 0 2022-01-31T19:44:15.437 INFO:tasks.workunit.client.0.smithi167.stdout:8/701: stat d6/f1d 0 2022-01-31T19:44:15.438 INFO:tasks.workunit.client.0.smithi167.stdout:6/546: link d8/d33/d51/d5d/d64/dab/fae d8/d33/d51/d72/fc9 0 2022-01-31T19:44:15.439 INFO:tasks.workunit.client.0.smithi167.stdout:4/645: mknod d0/d6/dab/cd1 0 2022-01-31T19:44:15.440 INFO:tasks.workunit.client.0.smithi167.stdout:3/677: rmdir d9/d29/d3b/d9b/da3 39 2022-01-31T19:44:15.440 INFO:tasks.workunit.client.0.smithi167.stdout:3/678: chown d9/d7f/f93 122196 1 2022-01-31T19:44:15.444 INFO:tasks.workunit.client.0.smithi167.stdout:4/646: mkdir d0/d6/d7/d26/dd2 0 2022-01-31T19:44:15.448 INFO:tasks.workunit.client.0.smithi167.stdout:8/702: dread d6/db/d6a/fb9 [0,4194304] 0 2022-01-31T19:44:15.449 INFO:tasks.workunit.client.0.smithi167.stdout:8/703: fsync d6/fa 0 2022-01-31T19:44:15.449 INFO:tasks.workunit.client.0.smithi167.stdout:8/704: dread - d6/db/d28/d61/dd5/fdf zero size 2022-01-31T19:44:15.452 INFO:tasks.workunit.client.0.smithi167.stdout:5/784: dwrite d5/d6/de/fbd [0,4194304] 0 2022-01-31T19:44:15.453 INFO:tasks.workunit.client.0.smithi167.stdout:5/785: truncate d5/d6/d16/d18/d43/d45/f83 2697554 0 2022-01-31T19:44:15.454 INFO:tasks.workunit.client.0.smithi167.stdout:7/871: dwrite d8/dc/f1c [0,4194304] 0 2022-01-31T19:44:15.455 INFO:tasks.workunit.client.0.smithi167.stdout:5/786: rename d5/d6/d16/d18/d5b/lcf to d5/d6/d16/d18/d43/d45/dd5/dbc/lf4 0 2022-01-31T19:44:15.458 INFO:tasks.workunit.client.0.smithi167.stdout:7/872: unlink d8/d2c/d7a/dd3/cf1 0 2022-01-31T19:44:15.458 INFO:tasks.workunit.client.0.smithi167.stdout:7/873: chown d8/d2c 12141 1 2022-01-31T19:44:15.459 INFO:tasks.workunit.client.0.smithi167.stdout:4/647: fdatasync d0/d6/d7/d26/d40/d4c/f9c 0 2022-01-31T19:44:15.460 INFO:tasks.workunit.client.0.smithi167.stdout:4/648: dread - d0/d6/d7/d26/d40/d4c/f7d zero size 2022-01-31T19:44:15.461 INFO:tasks.workunit.client.0.smithi167.stdout:4/649: rename d0/d3f/da2/dcf/f77 to d0/d51/fd3 0 2022-01-31T19:44:15.468 INFO:tasks.workunit.client.0.smithi167.stdout:7/874: write d8/dc/f21 [307327,48470] 0 2022-01-31T19:44:15.469 INFO:tasks.workunit.client.0.smithi167.stdout:7/875: dread - d8/dc/f11e zero size 2022-01-31T19:44:15.471 INFO:tasks.workunit.client.0.smithi167.stdout:6/547: dwrite d8/d33/f40 [0,4194304] 0 2022-01-31T19:44:15.472 INFO:tasks.workunit.client.0.smithi167.stdout:6/548: write d8/d33/d51/d72/f75 [26471,89077] 0 2022-01-31T19:44:15.475 INFO:tasks.workunit.client.0.smithi167.stdout:3/679: dwrite d9/d29/d60/f96 [0,4194304] 0 2022-01-31T19:44:15.476 INFO:tasks.workunit.client.0.smithi167.stdout:6/549: mkdir d8/dca 0 2022-01-31T19:44:15.476 INFO:tasks.workunit.client.0.smithi167.stdout:6/550: dread - d8/d33/d4f/f8e zero size 2022-01-31T19:44:15.477 INFO:tasks.workunit.client.0.smithi167.stdout:6/551: chown d8/d33/l3a 12 1 2022-01-31T19:44:15.479 INFO:tasks.workunit.client.0.smithi167.stdout:3/680: symlink d9/d26/lc4 0 2022-01-31T19:44:15.480 INFO:tasks.workunit.client.0.smithi167.stdout:6/552: mknod d8/ccb 0 2022-01-31T19:44:15.480 INFO:tasks.workunit.client.0.smithi167.stdout:6/553: creat d8/d33/d51/d5d/d86/fcc x:0 0 0 2022-01-31T19:44:15.481 INFO:tasks.workunit.client.0.smithi167.stdout:3/681: dread d9/d29/d3b/d9b/da3/db0/fb9 [0,4194304] 0 2022-01-31T19:44:15.482 INFO:tasks.workunit.client.0.smithi167.stdout:6/554: creat d8/d33/d51/d5d/d64/fcd x:0 0 0 2022-01-31T19:44:15.483 INFO:tasks.workunit.client.0.smithi167.stdout:3/682: unlink d9/d56/f63 0 2022-01-31T19:44:15.485 INFO:tasks.workunit.client.0.smithi167.stdout:6/555: write d8/f1c [3747335,105113] 0 2022-01-31T19:44:15.485 INFO:tasks.workunit.client.0.smithi167.stdout:3/683: mknod d9/d29/d3b/d9b/cc5 0 2022-01-31T19:44:15.485 INFO:tasks.workunit.client.0.smithi167.stdout:6/556: dread - d8/d20/fa6 zero size 2022-01-31T19:44:15.485 INFO:tasks.workunit.client.0.smithi167.stdout:3/684: write d9/d29/dbd/fc0 [63469,71083] 0 2022-01-31T19:44:15.487 INFO:tasks.workunit.client.0.smithi167.stdout:3/685: creat d9/d29/d3b/d8d/fc6 x:0 0 0 2022-01-31T19:44:15.488 INFO:tasks.workunit.client.0.smithi167.stdout:3/686: symlink d9/d29/dbd/lc7 0 2022-01-31T19:44:15.488 INFO:tasks.workunit.client.0.smithi167.stdout:3/687: truncate d9/d29/fb2 1222842 0 2022-01-31T19:44:15.489 INFO:tasks.workunit.client.0.smithi167.stdout:3/688: link d9/d29/d3b/d41/d6b/d89/lb6 d9/d7f/dbe/lc8 0 2022-01-31T19:44:15.491 INFO:tasks.workunit.client.0.smithi167.stdout:3/689: mkdir d9/d29/d60/dc9 0 2022-01-31T19:44:15.492 INFO:tasks.workunit.client.0.smithi167.stdout:8/705: dwrite d6/db/d4e/f4f [0,4194304] 0 2022-01-31T19:44:15.492 INFO:tasks.workunit.client.0.smithi167.stdout:8/706: fsync d6/db/d4e/d73/fa2 0 2022-01-31T19:44:15.493 INFO:tasks.workunit.client.0.smithi167.stdout:3/690: rename d9/d29/d3b/cbc to d9/d29/d60/dc9/cca 0 2022-01-31T19:44:15.500 INFO:tasks.workunit.client.0.smithi167.stdout:5/787: dwrite d5/d6/d16/d18/f2c [0,4194304] 0 2022-01-31T19:44:15.501 INFO:tasks.workunit.client.0.smithi167.stdout:5/788: dread - d5/d6/d16/d18/d43/d45/d69/dc4/fb4 zero size 2022-01-31T19:44:15.501 INFO:tasks.workunit.client.0.smithi167.stdout:5/789: chown d5/d61/f4b 493092409 1 2022-01-31T19:44:15.501 INFO:tasks.workunit.client.0.smithi167.stdout:5/790: stat d5/d6/d16/d18/d43/d45/dd5/dbc/fd3 0 2022-01-31T19:44:15.501 INFO:tasks.workunit.client.0.smithi167.stdout:4/650: dwrite d0/d6/fc6 [0,4194304] 0 2022-01-31T19:44:15.502 INFO:tasks.workunit.client.0.smithi167.stdout:4/651: read - d0/d6/d7/f9a zero size 2022-01-31T19:44:15.502 INFO:tasks.workunit.client.0.smithi167.stdout:7/876: dwrite d8/d35/f4b [4194304,4194304] 0 2022-01-31T19:44:15.502 INFO:tasks.workunit.client.0.smithi167.stdout:7/877: fsync d8/d2c/d6e/d4d/d5e/d71/db1/d8d/fec 0 2022-01-31T19:44:15.503 INFO:tasks.workunit.client.0.smithi167.stdout:3/691: mknod d9/d7f/da8/ccb 0 2022-01-31T19:44:15.503 INFO:tasks.workunit.client.0.smithi167.stdout:4/652: rename d0/cb8 to d0/d6/d7/d26/d50/db3/cd4 0 2022-01-31T19:44:15.503 INFO:tasks.workunit.client.0.smithi167.stdout:7/878: mkdir d8/dab/d13a 0 2022-01-31T19:44:15.504 INFO:tasks.workunit.client.0.smithi167.stdout:7/879: chown d8/d2c/d6e/l51 0 1 2022-01-31T19:44:15.505 INFO:tasks.workunit.client.0.smithi167.stdout:3/692: creat d9/d7f/dbe/fcc x:0 0 0 2022-01-31T19:44:15.505 INFO:tasks.workunit.client.0.smithi167.stdout:3/693: stat d9/d29/d60/c46 0 2022-01-31T19:44:15.506 INFO:tasks.workunit.client.0.smithi167.stdout:3/694: readlink d9/d29/d60/l45 0 2022-01-31T19:44:15.506 INFO:tasks.workunit.client.0.smithi167.stdout:3/695: fsync d9/d29/d9d/fa7 0 2022-01-31T19:44:15.507 INFO:tasks.workunit.client.0.smithi167.stdout:7/880: mkdir d8/dc/d18/d24/d13b 0 2022-01-31T19:44:15.510 INFO:tasks.workunit.client.0.smithi167.stdout:3/696: write d9/f2c [2607285,38434] 0 2022-01-31T19:44:15.510 INFO:tasks.workunit.client.0.smithi167.stdout:6/557: dwrite d8/d33/d51/d7e/dc4/f96 [4194304,4194304] 0 2022-01-31T19:44:15.511 INFO:tasks.workunit.client.0.smithi167.stdout:3/697: unlink d9/c86 0 2022-01-31T19:44:15.513 INFO:tasks.workunit.client.0.smithi167.stdout:8/707: rmdir d6/d1e/d4c/d80 39 2022-01-31T19:44:15.513 INFO:tasks.workunit.client.0.smithi167.stdout:8/708: creat d6/db/d4e/fe8 x:0 0 0 2022-01-31T19:44:15.515 INFO:tasks.workunit.client.0.smithi167.stdout:7/881: mknod d8/d2c/d43/d94/c13c 0 2022-01-31T19:44:15.515 INFO:tasks.workunit.client.0.smithi167.stdout:7/882: chown d8/d2c/d6e/d4d/d67/d8f/dcb/l120 66434191 1 2022-01-31T19:44:15.521 INFO:tasks.workunit.client.0.smithi167.stdout:5/791: dwrite d5/d6/d16/d18/f3d [0,4194304] 0 2022-01-31T19:44:15.523 INFO:tasks.workunit.client.0.smithi167.stdout:4/653: dwrite d0/d6/f76 [4194304,4194304] 0 2022-01-31T19:44:15.524 INFO:tasks.workunit.client.0.smithi167.stdout:6/558: dread d8/d33/d51/d5d/f6e [0,4194304] 0 2022-01-31T19:44:15.526 INFO:tasks.workunit.client.0.smithi167.stdout:8/709: rename d6/ff to d6/db/de/d3d/fe9 0 2022-01-31T19:44:15.529 INFO:tasks.workunit.client.0.smithi167.stdout:7/883: symlink d8/d2c/d6e/d4d/d5e/d71/db1/d4f/l13d 0 2022-01-31T19:44:15.529 INFO:tasks.workunit.client.0.smithi167.stdout:7/884: getdents d8/d2c/d43/df3/d118 0 2022-01-31T19:44:15.534 INFO:tasks.workunit.client.0.smithi167.stdout:3/698: getdents d9 0 2022-01-31T19:44:15.534 INFO:tasks.workunit.client.0.smithi167.stdout:3/699: chown d9/d29/d60/f5b 240 1 2022-01-31T19:44:15.538 INFO:tasks.workunit.client.0.smithi167.stdout:6/559: creat d8/d33/d51/d72/dc2/fce x:0 0 0 2022-01-31T19:44:15.538 INFO:tasks.workunit.client.0.smithi167.stdout:8/710: creat d6/db/d28/d61/de3/fea x:0 0 0 2022-01-31T19:44:15.539 INFO:tasks.workunit.client.0.smithi167.stdout:8/711: chown d6/db/d36/d5d/le2 2581073 1 2022-01-31T19:44:15.539 INFO:tasks.workunit.client.0.smithi167.stdout:8/712: write d6/db/de/f10 [1980444,30664] 0 2022-01-31T19:44:15.540 INFO:tasks.workunit.client.0.smithi167.stdout:5/792: dread d5/d6/de/d22/d4d/f9e [0,4194304] 0 2022-01-31T19:44:15.549 INFO:tasks.workunit.client.0.smithi167.stdout:7/885: mknod d8/dc/d18/d24/dcc/c13e 0 2022-01-31T19:44:15.549 INFO:tasks.workunit.client.0.smithi167.stdout:4/654: symlink d0/d6/d7/d26/d40/ld5 0 2022-01-31T19:44:15.552 INFO:tasks.workunit.client.0.smithi167.stdout:3/700: rmdir d9/d7f/da8 39 2022-01-31T19:44:15.555 INFO:tasks.workunit.client.0.smithi167.stdout:8/713: stat d6/d1e/d4c/c50 0 2022-01-31T19:44:15.556 INFO:tasks.workunit.client.0.smithi167.stdout:5/793: mknod d5/d6/d16/d18/d43/d45/dd5/dbc/dcc/cf5 0 2022-01-31T19:44:15.558 INFO:tasks.workunit.client.0.smithi167.stdout:4/655: mkdir d0/d3f/d75/dd6 0 2022-01-31T19:44:15.559 INFO:tasks.workunit.client.0.smithi167.stdout:3/701: symlink d9/d29/d60/dc9/lcd 0 2022-01-31T19:44:15.559 INFO:tasks.workunit.client.0.smithi167.stdout:3/702: creat d9/d21/fce x:0 0 0 2022-01-31T19:44:15.560 INFO:tasks.workunit.client.0.smithi167.stdout:3/703: readlink d9/d29/l6a 0 2022-01-31T19:44:15.560 INFO:tasks.workunit.client.0.smithi167.stdout:8/714: mkdir d6/db/de/d18/deb 0 2022-01-31T19:44:15.561 INFO:tasks.workunit.client.0.smithi167.stdout:6/560: dread d8/d33/f3c [4194304,4194304] 0 2022-01-31T19:44:15.561 INFO:tasks.workunit.client.0.smithi167.stdout:8/715: write d6/db/d4e/fd0 [51347,85016] 0 2022-01-31T19:44:15.562 INFO:tasks.workunit.client.0.smithi167.stdout:5/794: mknod d5/d6/de/d22/d76/db0/cf6 0 2022-01-31T19:44:15.564 INFO:tasks.workunit.client.0.smithi167.stdout:4/656: creat d0/d6/d7/d26/d31/d5c/fd7 x:0 0 0 2022-01-31T19:44:15.565 INFO:tasks.workunit.client.0.smithi167.stdout:4/657: stat d0/d6/d7/d39/d4e/d61/c8e 0 2022-01-31T19:44:15.565 INFO:tasks.workunit.client.0.smithi167.stdout:4/658: stat d0/d6/d7/d26/d50/db3 0 2022-01-31T19:44:15.565 INFO:tasks.workunit.client.0.smithi167.stdout:7/886: dwrite d8/d2c/d6e/d4d/d67/d8f/f96 [0,4194304] 0 2022-01-31T19:44:15.566 INFO:tasks.workunit.client.0.smithi167.stdout:3/704: stat d9/d29/d60/dc9/cca 0 2022-01-31T19:44:15.566 INFO:tasks.workunit.client.0.smithi167.stdout:3/705: fdatasync d9/d29/d3b/d9b/fb4 0 2022-01-31T19:44:15.567 INFO:tasks.workunit.client.0.smithi167.stdout:8/716: mkdir d6/d1e/d4c/dba/dec 0 2022-01-31T19:44:15.572 INFO:tasks.workunit.client.0.smithi167.stdout:4/659: dread d0/d6/d7/f8 [4194304,4194304] 0 2022-01-31T19:44:15.574 INFO:tasks.workunit.client.0.smithi167.stdout:7/887: dread d8/dc/d18/d24/dcc/f133 [0,4194304] 0 2022-01-31T19:44:15.575 INFO:tasks.workunit.client.0.smithi167.stdout:3/706: mknod d9/d29/d60/ccf 0 2022-01-31T19:44:15.575 INFO:tasks.workunit.client.0.smithi167.stdout:5/795: mknod d5/d6/d16/d18/d5b/cf7 0 2022-01-31T19:44:15.579 INFO:tasks.workunit.client.0.smithi167.stdout:6/561: dwrite d8/f73 [0,4194304] 0 2022-01-31T19:44:15.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/660: mkdir d0/d6/dab/dd8 0 2022-01-31T19:44:15.580 INFO:tasks.workunit.client.0.smithi167.stdout:9/600: sync 2022-01-31T19:44:15.580 INFO:tasks.workunit.client.0.smithi167.stdout:0/424: sync 2022-01-31T19:44:15.581 INFO:tasks.workunit.client.0.smithi167.stdout:6/562: symlink d8/d33/d51/d72/dad/lcf 0 2022-01-31T19:44:15.582 INFO:tasks.workunit.client.0.smithi167.stdout:1/769: sync 2022-01-31T19:44:15.583 INFO:tasks.workunit.client.0.smithi167.stdout:4/661: unlink d0/d3f/fae 0 2022-01-31T19:44:15.584 INFO:tasks.workunit.client.0.smithi167.stdout:7/888: getdents d8/d2c/d6e/d9a/dd8 0 2022-01-31T19:44:15.585 INFO:tasks.workunit.client.0.smithi167.stdout:7/889: write d8/f45 [4771635,81758] 0 2022-01-31T19:44:15.585 INFO:tasks.workunit.client.0.smithi167.stdout:7/890: creat d8/dc/f13f x:0 0 0 2022-01-31T19:44:15.585 INFO:tasks.workunit.client.0.smithi167.stdout:7/891: chown d8/d2c/d43/df3/l115 160 1 2022-01-31T19:44:15.587 INFO:tasks.workunit.client.0.smithi167.stdout:0/425: rename d17/d33/c75 to d17/d33/c86 0 2022-01-31T19:44:15.587 INFO:tasks.workunit.client.0.smithi167.stdout:3/707: dread d9/f18 [0,4194304] 0 2022-01-31T19:44:15.587 INFO:tasks.workunit.client.0.smithi167.stdout:3/708: truncate d9/d26/fb8 988091 0 2022-01-31T19:44:15.587 INFO:tasks.workunit.client.0.smithi167.stdout:3/709: chown d9/d29/d60/l64 25282 1 2022-01-31T19:44:15.588 INFO:tasks.workunit.client.0.smithi167.stdout:3/710: dread - d9/d29/d3b/d41/d6b/f80 zero size 2022-01-31T19:44:15.588 INFO:tasks.workunit.client.0.smithi167.stdout:9/601: mknod dd/d17/d7f/cc8 0 2022-01-31T19:44:15.589 INFO:tasks.workunit.client.0.smithi167.stdout:1/770: truncate d0/d10/d20/f4e 2918699 0 2022-01-31T19:44:15.590 INFO:tasks.workunit.client.0.smithi167.stdout:4/662: symlink d0/d6/d7/ld9 0 2022-01-31T19:44:15.591 INFO:tasks.workunit.client.0.smithi167.stdout:5/796: dread d5/d6/d16/d18/d43/d45/f94 [0,4194304] 0 2022-01-31T19:44:15.591 INFO:tasks.workunit.client.0.smithi167.stdout:7/892: symlink d8/d2c/d43/d94/daf/dee/l140 0 2022-01-31T19:44:15.592 INFO:tasks.workunit.client.0.smithi167.stdout:7/893: chown d8/d2c/d6e/d4d/d5e/f91 0 1 2022-01-31T19:44:15.593 INFO:tasks.workunit.client.0.smithi167.stdout:0/426: rename d17/d19/d1c/d39/d7b/d85 to d17/d2c/d87 0 2022-01-31T19:44:15.595 INFO:tasks.workunit.client.0.smithi167.stdout:9/602: dread dd/f10 [0,4194304] 0 2022-01-31T19:44:15.595 INFO:tasks.workunit.client.0.smithi167.stdout:3/711: write d9/d29/f2b [835609,127113] 0 2022-01-31T19:44:15.596 INFO:tasks.workunit.client.0.smithi167.stdout:1/771: truncate d0/d75/f8d 2161367 0 2022-01-31T19:44:15.599 INFO:tasks.workunit.client.0.smithi167.stdout:7/894: dread d8/d2c/d43/df3/ff6 [0,4194304] 0 2022-01-31T19:44:15.600 INFO:tasks.workunit.client.0.smithi167.stdout:4/663: creat d0/d6/d7/d26/dd2/fda x:0 0 0 2022-01-31T19:44:15.600 INFO:tasks.workunit.client.0.smithi167.stdout:4/664: creat d0/d6/d7/d26/d31/d5c/fdb x:0 0 0 2022-01-31T19:44:15.602 INFO:tasks.workunit.client.0.smithi167.stdout:8/717: dwrite d6/d1e/f22 [4194304,4194304] 0 2022-01-31T19:44:15.602 INFO:tasks.workunit.client.0.smithi167.stdout:5/797: dread d5/d6/d16/d18/f58 [0,4194304] 0 2022-01-31T19:44:15.603 INFO:tasks.workunit.client.0.smithi167.stdout:5/798: fsync d5/d6/d3a/f17 0 2022-01-31T19:44:15.603 INFO:tasks.workunit.client.0.smithi167.stdout:6/563: dwrite d8/d20/f2f [0,4194304] 0 2022-01-31T19:44:15.604 INFO:tasks.workunit.client.0.smithi167.stdout:6/564: fsync d8/d62/f6d 0 2022-01-31T19:44:15.604 INFO:tasks.workunit.client.0.smithi167.stdout:0/427: mkdir d17/d19/d1c/d39/d44/d88 0 2022-01-31T19:44:15.607 INFO:tasks.workunit.client.0.smithi167.stdout:8/718: dread d6/db/d6a/d8d/d9e/d60/f6c [0,4194304] 0 2022-01-31T19:44:15.607 INFO:tasks.workunit.client.0.smithi167.stdout:8/719: fdatasync d6/db/d4e/f58 0 2022-01-31T19:44:15.607 INFO:tasks.workunit.client.0.smithi167.stdout:8/720: chown d6/db/de/d18 7606329 1 2022-01-31T19:44:15.607 INFO:tasks.workunit.client.0.smithi167.stdout:3/712: symlink d9/d21/ld0 0 2022-01-31T19:44:15.608 INFO:tasks.workunit.client.0.smithi167.stdout:3/713: fdatasync d9/d29/d3b/d41/d6b/d89/fa5 0 2022-01-31T19:44:15.608 INFO:tasks.workunit.client.0.smithi167.stdout:3/714: chown d9/d29/d60/dc9/lcd 230533077 1 2022-01-31T19:44:15.609 INFO:tasks.workunit.client.0.smithi167.stdout:7/895: mknod d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/daa/c141 0 2022-01-31T19:44:15.611 INFO:tasks.workunit.client.0.smithi167.stdout:4/665: symlink d0/d6/d7/d26/dd2/ldc 0 2022-01-31T19:44:15.615 INFO:tasks.workunit.client.0.smithi167.stdout:1/772: dwrite d0/d10/d5a/d33/d4b/da5/da9/ffc [0,4194304] 0 2022-01-31T19:44:15.615 INFO:tasks.workunit.client.0.smithi167.stdout:5/799: rename d5/d6/d16/l19 to d5/d6/de/d22/d76/lf8 0 2022-01-31T19:44:15.618 INFO:tasks.workunit.client.0.smithi167.stdout:0/428: link d17/d19/d1c/d39/d44/f56 d17/d33/f89 0 2022-01-31T19:44:15.619 INFO:tasks.workunit.client.0.smithi167.stdout:0/429: write d17/f22 [4352142,59263] 0 2022-01-31T19:44:15.619 INFO:tasks.workunit.client.0.smithi167.stdout:9/603: rmdir dd/d35 39 2022-01-31T19:44:15.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/430: write d17/d19/d1c/f32 [3420485,66922] 0 2022-01-31T19:44:15.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/431: write d17/d19/d1c/d39/f74 [672794,65631] 0 2022-01-31T19:44:15.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/432: stat d17/d69/f78 0 2022-01-31T19:44:15.621 INFO:tasks.workunit.client.0.smithi167.stdout:0/433: fsync d17/d2c/f64 0 2022-01-31T19:44:15.622 INFO:tasks.workunit.client.0.smithi167.stdout:0/434: fsync d17/d2c/f60 0 2022-01-31T19:44:15.622 INFO:tasks.workunit.client.0.smithi167.stdout:8/721: creat d6/db/de/d18/deb/fed x:0 0 0 2022-01-31T19:44:15.623 INFO:tasks.workunit.client.0.smithi167.stdout:0/435: write d17/d33/f6f [1928594,29814] 0 2022-01-31T19:44:15.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/715: creat d9/d29/d3b/d41/d6b/d89/fd1 x:0 0 0 2022-01-31T19:44:15.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/716: chown d9/d29/d3b/d41/c4f 2789 1 2022-01-31T19:44:15.624 INFO:tasks.workunit.client.0.smithi167.stdout:7/896: rmdir d8/d2c/d43/d94/daf/dee/df8/d121 0 2022-01-31T19:44:15.625 INFO:tasks.workunit.client.0.smithi167.stdout:2/534: sync 2022-01-31T19:44:15.625 INFO:tasks.workunit.client.0.smithi167.stdout:2/535: write dd/d18/d62/fa6 [821244,82005] 0 2022-01-31T19:44:15.625 INFO:tasks.workunit.client.0.smithi167.stdout:2/536: truncate dd/d18/d63/fa1 1581502 0 2022-01-31T19:44:15.626 INFO:tasks.workunit.client.0.smithi167.stdout:8/722: write d6/db/de/d18/f1b [2742758,96212] 0 2022-01-31T19:44:15.626 INFO:tasks.workunit.client.0.smithi167.stdout:8/723: write d6/db/d4e/d73/fa2 [1095835,82743] 0 2022-01-31T19:44:15.628 INFO:tasks.workunit.client.0.smithi167.stdout:6/565: rename d8/c8a to d8/d33/d51/d72/dad/cd0 0 2022-01-31T19:44:15.629 INFO:tasks.workunit.client.0.smithi167.stdout:6/566: getdents d8/dca 0 2022-01-31T19:44:15.629 INFO:tasks.workunit.client.0.smithi167.stdout:5/800: rmdir d5/d6/de/d22/d76/db0/dd8 39 2022-01-31T19:44:15.634 INFO:tasks.workunit.client.0.smithi167.stdout:3/717: mkdir d9/d29/d3b/d41/d6b/dd2 0 2022-01-31T19:44:15.635 INFO:tasks.workunit.client.0.smithi167.stdout:8/724: link d6/db/d28/daf/cbb d6/d1e/d4c/dba/cee 0 2022-01-31T19:44:15.638 INFO:tasks.workunit.client.0.smithi167.stdout:4/666: rename d0/d6/d7/d39/f37 to d0/d6/d7/d26/d31/d5c/fdd 0 2022-01-31T19:44:15.639 INFO:tasks.workunit.client.0.smithi167.stdout:6/567: mkdir d8/d33/d4f/dd1 0 2022-01-31T19:44:15.640 INFO:tasks.workunit.client.0.smithi167.stdout:5/801: mkdir d5/d6/d16/d18/d43/d45/d69/d90/df9 0 2022-01-31T19:44:15.640 INFO:tasks.workunit.client.0.smithi167.stdout:9/604: dread dd/d17/d80/f43 [0,4194304] 0 2022-01-31T19:44:15.641 INFO:tasks.workunit.client.0.smithi167.stdout:0/436: rmdir d17/d19/d1c/d39/d44/d49 39 2022-01-31T19:44:15.642 INFO:tasks.workunit.client.0.smithi167.stdout:3/718: rmdir d9 39 2022-01-31T19:44:15.642 INFO:tasks.workunit.client.0.smithi167.stdout:3/719: stat d9/d56 0 2022-01-31T19:44:15.643 INFO:tasks.workunit.client.0.smithi167.stdout:2/537: symlink dd/d18/d62/laa 0 2022-01-31T19:44:15.643 INFO:tasks.workunit.client.0.smithi167.stdout:2/538: truncate dd/d18/d62/fa6 1027563 0 2022-01-31T19:44:15.643 INFO:tasks.workunit.client.0.smithi167.stdout:8/725: symlink d6/db/d28/d61/de3/lef 0 2022-01-31T19:44:15.644 INFO:tasks.workunit.client.0.smithi167.stdout:8/726: write d6/db/fd2 [58436,32207] 0 2022-01-31T19:44:15.644 INFO:tasks.workunit.client.0.smithi167.stdout:6/568: rename d8/fac to d8/d33/fd2 0 2022-01-31T19:44:15.645 INFO:tasks.workunit.client.0.smithi167.stdout:9/605: mkdir dd/d19/d24/d56/dc9 0 2022-01-31T19:44:15.647 INFO:tasks.workunit.client.0.smithi167.stdout:2/539: rmdir dd/df/d22 39 2022-01-31T19:44:15.647 INFO:tasks.workunit.client.0.smithi167.stdout:2/540: readlink dd/df/d1b/d3d/d4a/d4b/d54/l88 0 2022-01-31T19:44:15.649 INFO:tasks.workunit.client.0.smithi167.stdout:4/667: dread d0/d6/d7/f30 [0,4194304] 0 2022-01-31T19:44:15.649 INFO:tasks.workunit.client.0.smithi167.stdout:3/720: rename d9/d29/d3b/d41/d6b/d89/fd1 to d9/fd3 0 2022-01-31T19:44:15.650 INFO:tasks.workunit.client.0.smithi167.stdout:3/721: read - d9/d26/f8b zero size 2022-01-31T19:44:15.650 INFO:tasks.workunit.client.0.smithi167.stdout:6/569: rename d8/d33/d35/f4d to d8/d33/d35/fd3 0 2022-01-31T19:44:15.653 INFO:tasks.workunit.client.0.smithi167.stdout:9/606: unlink dd/d19/d24/d56/l73 0 2022-01-31T19:44:15.653 INFO:tasks.workunit.client.0.smithi167.stdout:9/607: creat dd/d19/d24/fca x:0 0 0 2022-01-31T19:44:15.653 INFO:tasks.workunit.client.0.smithi167.stdout:9/608: chown dd/f16 3 1 2022-01-31T19:44:15.654 INFO:tasks.workunit.client.0.smithi167.stdout:9/609: creat dd/d17/d7f/fcb x:0 0 0 2022-01-31T19:44:15.654 INFO:tasks.workunit.client.0.smithi167.stdout:5/802: dread d5/d6/de/f3e [0,4194304] 0 2022-01-31T19:44:15.654 INFO:tasks.workunit.client.0.smithi167.stdout:0/437: getdents d17/d19/d1c/d39/d7b 0 2022-01-31T19:44:15.656 INFO:tasks.workunit.client.0.smithi167.stdout:2/541: mkdir dd/d18/d63/dab 0 2022-01-31T19:44:15.657 INFO:tasks.workunit.client.0.smithi167.stdout:4/668: chown d0/d6/d7/d26/d40/l45 4733321 1 2022-01-31T19:44:15.659 INFO:tasks.workunit.client.0.smithi167.stdout:3/722: mkdir d9/d29/dd4 0 2022-01-31T19:44:15.661 INFO:tasks.workunit.client.0.smithi167.stdout:6/570: creat d8/d33/d51/d5d/d86/dbd/fd4 x:0 0 0 2022-01-31T19:44:15.661 INFO:tasks.workunit.client.0.smithi167.stdout:8/727: dwrite d6/db/fd [0,4194304] 0 2022-01-31T19:44:15.662 INFO:tasks.workunit.client.0.smithi167.stdout:8/728: chown d6/db/d28 3570027 1 2022-01-31T19:44:15.662 INFO:tasks.workunit.client.0.smithi167.stdout:8/729: write d6/db/de/d18/da4/dc2/dc3/dc7/fc9 [827801,113026] 0 2022-01-31T19:44:15.665 INFO:tasks.workunit.client.0.smithi167.stdout:5/803: mknod d5/d6/de/d22/d76/cfa 0 2022-01-31T19:44:15.665 INFO:tasks.workunit.client.0.smithi167.stdout:5/804: fsync d5/d6/d16/d18/d5b/d63/fc8 0 2022-01-31T19:44:15.665 INFO:tasks.workunit.client.0.smithi167.stdout:0/438: rmdir d17/d19/d1c/d39/d44/d49 39 2022-01-31T19:44:15.667 INFO:tasks.workunit.client.0.smithi167.stdout:2/542: rename dd/d18/d62/f86 to dd/df/d1b/d3d/d4a/fac 0 2022-01-31T19:44:15.667 INFO:tasks.workunit.client.0.smithi167.stdout:2/543: write dd/df/f79 [1343013,72655] 0 2022-01-31T19:44:15.668 INFO:tasks.workunit.client.0.smithi167.stdout:6/571: link d8/d33/d51/d5d/d86/l89 d8/d33/d51/d72/dc2/ld5 0 2022-01-31T19:44:15.668 INFO:tasks.workunit.client.0.smithi167.stdout:6/572: creat d8/d33/d51/d72/fd6 x:0 0 0 2022-01-31T19:44:15.668 INFO:tasks.workunit.client.0.smithi167.stdout:9/610: dwrite dd/fa3 [0,4194304] 0 2022-01-31T19:44:15.669 INFO:tasks.workunit.client.0.smithi167.stdout:9/611: unlink dd/d19/d59/c99 0 2022-01-31T19:44:15.670 INFO:tasks.workunit.client.0.smithi167.stdout:0/439: creat d17/d19/d1c/d39/d44/d88/f8a x:0 0 0 2022-01-31T19:44:15.672 INFO:tasks.workunit.client.0.smithi167.stdout:8/730: read d6/fa [1933635,35038] 0 2022-01-31T19:44:15.673 INFO:tasks.workunit.client.0.smithi167.stdout:8/731: truncate d6/db/de/d3d/fa7 357357 0 2022-01-31T19:44:15.675 INFO:tasks.workunit.client.0.smithi167.stdout:3/723: rename d9/d29/d3b/d9b/cc5 to d9/d29/d3b/d41/d6b/dd2/cd5 0 2022-01-31T19:44:15.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/544: creat dd/df/d3f/d51/fad x:0 0 0 2022-01-31T19:44:15.677 INFO:tasks.workunit.client.0.smithi167.stdout:6/573: creat d8/d33/d51/d5d/d64/dab/fd7 x:0 0 0 2022-01-31T19:44:15.677 INFO:tasks.workunit.client.0.smithi167.stdout:6/574: creat d8/d33/d51/d5d/d64/dab/fd8 x:0 0 0 2022-01-31T19:44:15.677 INFO:tasks.workunit.client.0.smithi167.stdout:6/575: creat d8/d33/d51/d72/dc2/fd9 x:0 0 0 2022-01-31T19:44:15.679 INFO:tasks.workunit.client.0.smithi167.stdout:5/805: rmdir d5/d6/d16/d18/d5b/d63 39 2022-01-31T19:44:15.679 INFO:tasks.workunit.client.0.smithi167.stdout:5/806: stat d5/d6/de/d22/c56 0 2022-01-31T19:44:15.679 INFO:tasks.workunit.client.0.smithi167.stdout:5/807: creat d5/d6/d16/d18/ffb x:0 0 0 2022-01-31T19:44:15.680 INFO:tasks.workunit.client.0.smithi167.stdout:5/808: write d5/d6/d16/d18/d43/d45/fec [1541922,12945] 0 2022-01-31T19:44:15.681 INFO:tasks.workunit.client.0.smithi167.stdout:8/732: unlink d6/db/de/f3e 0 2022-01-31T19:44:15.683 INFO:tasks.workunit.client.0.smithi167.stdout:3/724: link d9/d7f/f93 d9/d29/d3b/d41/d6b/d89/fd6 0 2022-01-31T19:44:15.684 INFO:tasks.workunit.client.0.smithi167.stdout:2/545: creat dd/df/d1b/d3d/d4a/fae x:0 0 0 2022-01-31T19:44:15.687 INFO:tasks.workunit.client.0.smithi167.stdout:2/546: dread dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:15.688 INFO:tasks.workunit.client.0.smithi167.stdout:6/576: link d8/l17 d8/d33/d51/d5d/d86/dbd/lda 0 2022-01-31T19:44:15.689 INFO:tasks.workunit.client.0.smithi167.stdout:4/669: dwrite d0/d3f/f3d [0,4194304] 0 2022-01-31T19:44:15.689 INFO:tasks.workunit.client.0.smithi167.stdout:4/670: creat d0/d6/dab/fde x:0 0 0 2022-01-31T19:44:15.689 INFO:tasks.workunit.client.0.smithi167.stdout:4/671: chown d0/d6/l8f 2637 1 2022-01-31T19:44:15.690 INFO:tasks.workunit.client.0.smithi167.stdout:4/672: getdents d0/d6/d7/d39/d3a/dc8 0 2022-01-31T19:44:15.690 INFO:tasks.workunit.client.0.smithi167.stdout:4/673: chown d0/d3f/da2/dcf/d65/f9d 713 1 2022-01-31T19:44:15.690 INFO:tasks.workunit.client.0.smithi167.stdout:4/674: truncate d0/d6/d7/d39/d4e/f52 4396201 0 2022-01-31T19:44:15.690 INFO:tasks.workunit.client.0.smithi167.stdout:4/675: dread - d0/d3f/da2/fc4 zero size 2022-01-31T19:44:15.691 INFO:tasks.workunit.client.0.smithi167.stdout:9/612: dwrite dd/d19/f21 [0,4194304] 0 2022-01-31T19:44:15.691 INFO:tasks.workunit.client.0.smithi167.stdout:2/547: dread dd/df/d3f/f8e [0,4194304] 0 2022-01-31T19:44:15.691 INFO:tasks.workunit.client.0.smithi167.stdout:5/809: rename d5/d6/d16/d18/f7d to d5/d6/de/d22/d76/da3/ffc 0 2022-01-31T19:44:15.701 INFO:tasks.workunit.client.0.smithi167.stdout:8/733: mkdir d6/db/d28/d61/dd5/df0 0 2022-01-31T19:44:15.703 INFO:tasks.workunit.client.0.smithi167.stdout:3/725: mknod d9/d7f/dbe/cd7 0 2022-01-31T19:44:15.704 INFO:tasks.workunit.client.0.smithi167.stdout:8/734: dread d6/db/de/f26 [0,4194304] 0 2022-01-31T19:44:15.704 INFO:tasks.workunit.client.0.smithi167.stdout:8/735: creat d6/db/d28/d66/ff1 x:0 0 0 2022-01-31T19:44:15.705 INFO:tasks.workunit.client.0.smithi167.stdout:8/736: chown d6/db/de/d3d/dd6 151131 1 2022-01-31T19:44:15.705 INFO:tasks.workunit.client.0.smithi167.stdout:8/737: fsync d6/db/de/d3d/fc1 0 2022-01-31T19:44:15.706 INFO:tasks.workunit.client.0.smithi167.stdout:6/577: unlink d8/d20/f79 0 2022-01-31T19:44:15.706 INFO:tasks.workunit.client.0.smithi167.stdout:6/578: dread - d8/d33/d4f/f59 zero size 2022-01-31T19:44:15.706 INFO:tasks.workunit.client.0.smithi167.stdout:9/613: symlink dd/d17/d7f/lcc 0 2022-01-31T19:44:15.707 INFO:tasks.workunit.client.0.smithi167.stdout:8/738: dread d6/db/d6a/fb9 [0,4194304] 0 2022-01-31T19:44:15.709 INFO:tasks.workunit.client.0.smithi167.stdout:5/810: creat d5/d6/ffd x:0 0 0 2022-01-31T19:44:15.709 INFO:tasks.workunit.client.0.smithi167.stdout:5/811: dread - d5/d6/d16/d18/d43/d45/dd5/dbc/fd3 zero size 2022-01-31T19:44:15.711 INFO:tasks.workunit.client.0.smithi167.stdout:2/548: dwrite dd/d18/d63/f32 [0,4194304] 0 2022-01-31T19:44:15.711 INFO:tasks.workunit.client.0.smithi167.stdout:2/549: dread - dd/df/d3f/d51/fad zero size 2022-01-31T19:44:15.712 INFO:tasks.workunit.client.0.smithi167.stdout:6/579: creat d8/d33/d51/d72/dc7/fdb x:0 0 0 2022-01-31T19:44:15.712 INFO:tasks.workunit.client.0.smithi167.stdout:6/580: chown d8/d33/l4a 262768793 1 2022-01-31T19:44:15.713 INFO:tasks.workunit.client.0.smithi167.stdout:6/581: write d8/d33/d51/d5d/d64/dab/fd7 [240809,73106] 0 2022-01-31T19:44:15.713 INFO:tasks.workunit.client.0.smithi167.stdout:8/739: getdents d6/db/de/d3d/dd6 0 2022-01-31T19:44:15.713 INFO:tasks.workunit.client.0.smithi167.stdout:8/740: chown d6/db/d36/d5d/f68 201692 1 2022-01-31T19:44:15.733 INFO:tasks.workunit.client.0.smithi167.stdout:2/550: dwrite dd/df/d1b/d3d/d4a/d4b/d54/f8b [0,4194304] 0 2022-01-31T19:44:15.733 INFO:tasks.workunit.client.0.smithi167.stdout:5/812: mknod d5/d6/de/d22/d76/db0/dee/cfe 0 2022-01-31T19:44:15.734 INFO:tasks.workunit.client.0.smithi167.stdout:3/726: getdents d9/d29/d60 0 2022-01-31T19:44:15.736 INFO:tasks.workunit.client.0.smithi167.stdout:6/582: creat d8/d20/fdc x:0 0 0 2022-01-31T19:44:15.739 INFO:tasks.workunit.client.0.smithi167.stdout:2/551: mknod dd/d18/d8d/caf 0 2022-01-31T19:44:15.740 INFO:tasks.workunit.client.0.smithi167.stdout:2/552: chown dd/d18/d63 33 1 2022-01-31T19:44:15.740 INFO:tasks.workunit.client.0.smithi167.stdout:2/553: fsync dd/df/f5d 0 2022-01-31T19:44:15.740 INFO:tasks.workunit.client.0.smithi167.stdout:2/554: dread - dd/df/d1b/d3d/d4a/f5b zero size 2022-01-31T19:44:15.743 INFO:tasks.workunit.client.0.smithi167.stdout:4/676: rename d0/d3f to d0/d6/d7/d26/d31/ddf 0 2022-01-31T19:44:15.744 INFO:tasks.workunit.client.0.smithi167.stdout:9/614: rename dd/d19 to dd/d19/d24/d50/dcd 22 2022-01-31T19:44:15.747 INFO:tasks.workunit.client.0.smithi167.stdout:5/813: unlink d5/d6/de/d22/d6d/cd0 0 2022-01-31T19:44:15.749 INFO:tasks.workunit.client.0.smithi167.stdout:3/727: link d9/d56/f90 d9/d29/dbd/fd8 0 2022-01-31T19:44:15.751 INFO:tasks.workunit.client.0.smithi167.stdout:6/583: link d8/d33/d51/d72/fd6 d8/d33/d51/d5d/d86/dbd/fdd 0 2022-01-31T19:44:15.752 INFO:tasks.workunit.client.0.smithi167.stdout:8/741: getdents d6/db/de/d18 0 2022-01-31T19:44:15.755 INFO:tasks.workunit.client.0.smithi167.stdout:9/615: rename dd/d19/f21 to dd/d19/d59/fce 0 2022-01-31T19:44:15.757 INFO:tasks.workunit.client.0.smithi167.stdout:5/814: creat d5/fff x:0 0 0 2022-01-31T19:44:15.757 INFO:tasks.workunit.client.0.smithi167.stdout:5/815: dread - d5/d6/de/d22/d6d/fb7 zero size 2022-01-31T19:44:15.758 INFO:tasks.workunit.client.0.smithi167.stdout:6/584: mknod d8/d33/d51/d5d/d86/dbd/cde 0 2022-01-31T19:44:15.759 INFO:tasks.workunit.client.0.smithi167.stdout:9/616: mkdir dd/d19/d24/d56/d5b/dcf 0 2022-01-31T19:44:15.760 INFO:tasks.workunit.client.0.smithi167.stdout:5/816: mknod d5/d6/de/d22/d76/db0/dd8/c100 0 2022-01-31T19:44:15.761 INFO:tasks.workunit.client.0.smithi167.stdout:5/817: unlink d5/d6/d16/d18/f33 0 2022-01-31T19:44:15.762 INFO:tasks.workunit.client.0.smithi167.stdout:3/728: mkdir d9/d29/d3b/d9b/da3/dd9 0 2022-01-31T19:44:15.762 INFO:tasks.workunit.client.0.smithi167.stdout:3/729: write d9/d29/d3b/d41/d6b/f39 [1481084,32037] 0 2022-01-31T19:44:15.763 INFO:tasks.workunit.client.0.smithi167.stdout:6/585: getdents d8/d33/d51 0 2022-01-31T19:44:15.764 INFO:tasks.workunit.client.0.smithi167.stdout:8/742: symlink d6/db/d6a/d8d/d9e/lf2 0 2022-01-31T19:44:15.764 INFO:tasks.workunit.client.0.smithi167.stdout:8/743: write d6/db/d6a/d8d/d9e/fcc [1954634,98840] 0 2022-01-31T19:44:15.765 INFO:tasks.workunit.client.0.smithi167.stdout:5/818: read d5/d6/de/fbd [3781689,89661] 0 2022-01-31T19:44:15.765 INFO:tasks.workunit.client.0.smithi167.stdout:5/819: truncate d5/d6/de/fe4 1215513 0 2022-01-31T19:44:15.765 INFO:tasks.workunit.client.0.smithi167.stdout:9/617: link dd/fa3 dd/d6b/dc1/fd0 0 2022-01-31T19:44:15.768 INFO:tasks.workunit.client.0.smithi167.stdout:3/730: rmdir d9/d29/dd4 0 2022-01-31T19:44:15.769 INFO:tasks.workunit.client.0.smithi167.stdout:8/744: write d6/db/de/d18/da4/dc2/dc3/fc5 [1606787,63616] 0 2022-01-31T19:44:15.769 INFO:tasks.workunit.client.0.smithi167.stdout:3/731: write d9/d29/d9d/fa7 [454588,108404] 0 2022-01-31T19:44:15.771 INFO:tasks.workunit.client.0.smithi167.stdout:5/820: creat d5/d6/d16/d18/d5b/d63/f101 x:0 0 0 2022-01-31T19:44:15.772 INFO:tasks.workunit.client.0.smithi167.stdout:2/555: dwrite f8 [4194304,4194304] 0 2022-01-31T19:44:15.772 INFO:tasks.workunit.client.0.smithi167.stdout:9/618: rename dd/d6b/d72 to dd/d19/d24/d50/dd1 0 2022-01-31T19:44:15.773 INFO:tasks.workunit.client.0.smithi167.stdout:4/677: dwrite d0/d6/d7/d26/d31/d5c/fdd [4194304,4194304] 0 2022-01-31T19:44:15.778 INFO:tasks.workunit.client.0.smithi167.stdout:3/732: mknod d9/d29/d60/dc9/cda 0 2022-01-31T19:44:15.779 INFO:tasks.workunit.client.0.smithi167.stdout:3/733: dread - d9/d29/d3b/d8d/fc6 zero size 2022-01-31T19:44:15.780 INFO:tasks.workunit.client.0.smithi167.stdout:3/734: dread d9/d29/d60/f58 [0,4194304] 0 2022-01-31T19:44:15.780 INFO:tasks.workunit.client.0.smithi167.stdout:5/821: mknod d5/d6/d16/d18/d43/d45/dd5/dbc/dcc/ded/c102 0 2022-01-31T19:44:15.780 INFO:tasks.workunit.client.0.smithi167.stdout:5/822: chown d5/d6/de/l1b 71 1 2022-01-31T19:44:15.782 INFO:tasks.workunit.client.0.smithi167.stdout:6/586: rmdir d8/d33/d51/d72 39 2022-01-31T19:44:15.788 INFO:tasks.workunit.client.0.smithi167.stdout:2/556: dwrite dd/df/d1b/d2f/d36/f2a [0,4194304] 0 2022-01-31T19:44:15.789 INFO:tasks.workunit.client.0.smithi167.stdout:9/619: link dd/d19/c2a dd/d19/d59/cd2 0 2022-01-31T19:44:15.792 INFO:tasks.workunit.client.0.smithi167.stdout:4/678: symlink d0/d6/d7/d26/d31/ddf/le0 0 2022-01-31T19:44:15.793 INFO:tasks.workunit.client.0.smithi167.stdout:8/745: truncate d6/d1e/f29 3225361 0 2022-01-31T19:44:15.793 INFO:tasks.workunit.client.0.smithi167.stdout:8/746: write d6/db/de/d18/fce [775153,76485] 0 2022-01-31T19:44:15.794 INFO:tasks.workunit.client.0.smithi167.stdout:3/735: rmdir d9/d7f/da8 39 2022-01-31T19:44:15.797 INFO:tasks.workunit.client.0.smithi167.stdout:3/736: write d9/d29/fa1 [339543,45972] 0 2022-01-31T19:44:15.798 INFO:tasks.workunit.client.0.smithi167.stdout:3/737: chown d9/d29/d3b/f84 33946 1 2022-01-31T19:44:15.798 INFO:tasks.workunit.client.0.smithi167.stdout:3/738: write d9/d26/fb8 [1644021,35780] 0 2022-01-31T19:44:15.798 INFO:tasks.workunit.client.0.smithi167.stdout:3/739: fdatasync d9/fc1 0 2022-01-31T19:44:15.798 INFO:tasks.workunit.client.0.smithi167.stdout:3/740: write d9/d29/d60/fac [3927348,94976] 0 2022-01-31T19:44:15.799 INFO:tasks.workunit.client.0.smithi167.stdout:3/741: creat d9/d26/fdb x:0 0 0 2022-01-31T19:44:15.799 INFO:tasks.workunit.client.0.smithi167.stdout:5/823: link d5/d6/d3a/f21 d5/d6/d16/d18/d43/d45/dd5/dbc/f103 0 2022-01-31T19:44:15.800 INFO:tasks.workunit.client.0.smithi167.stdout:2/557: mknod dd/df/d22/d96/d9e/cb0 0 2022-01-31T19:44:15.800 INFO:tasks.workunit.client.0.smithi167.stdout:2/558: fdatasync dd/df/d1b/d3d/d4a/d4b/f7f 0 2022-01-31T19:44:15.800 INFO:tasks.workunit.client.0.smithi167.stdout:9/620: getdents dd/d19/d24/d56/d5b/da5 0 2022-01-31T19:44:15.801 INFO:tasks.workunit.client.0.smithi167.stdout:6/587: dwrite d8/d33/d35/f8b [0,4194304] 0 2022-01-31T19:44:15.801 INFO:tasks.workunit.client.0.smithi167.stdout:8/747: link d6/db/d4e/fe8 d6/db/de/d3d/dd6/ff3 0 2022-01-31T19:44:15.810 INFO:tasks.workunit.client.0.smithi167.stdout:5/824: link d5/fff d5/d6/de/d22/d6d/da0/da4/dca/f104 0 2022-01-31T19:44:15.811 INFO:tasks.workunit.client.0.smithi167.stdout:4/679: dwrite d0/d6/d7/d26/d31/ddf/da2/dcf/d65/f88 [0,4194304] 0 2022-01-31T19:44:15.811 INFO:tasks.workunit.client.0.smithi167.stdout:4/680: readlink d0/l81 0 2022-01-31T19:44:15.812 INFO:tasks.workunit.client.0.smithi167.stdout:2/559: symlink dd/df/d1b/d3d/d4a/d4b/d74/lb1 0 2022-01-31T19:44:15.812 INFO:tasks.workunit.client.0.smithi167.stdout:4/681: read d0/d6/d7/d26/d31/f11 [1642077,60709] 0 2022-01-31T19:44:15.813 INFO:tasks.workunit.client.0.smithi167.stdout:9/621: mknod dd/d19/d24/d50/dd1/cd3 0 2022-01-31T19:44:15.815 INFO:tasks.workunit.client.0.smithi167.stdout:8/748: dread d6/db/de/d18/da4/dc2/dc3/fe5 [0,4194304] 0 2022-01-31T19:44:15.819 INFO:tasks.workunit.client.0.smithi167.stdout:3/742: getdents d9/d29/d3b 0 2022-01-31T19:44:15.827 INFO:tasks.workunit.client.0.smithi167.stdout:2/560: rename dd/d18/f4c to dd/df/d22/d96/d9e/fb2 0 2022-01-31T19:44:15.829 INFO:tasks.workunit.client.0.smithi167.stdout:9/622: creat dd/d19/d24/d56/dc9/fd4 x:0 0 0 2022-01-31T19:44:15.830 INFO:tasks.workunit.client.0.smithi167.stdout:8/749: stat d6/f1d 0 2022-01-31T19:44:15.831 INFO:tasks.workunit.client.0.smithi167.stdout:8/750: dread - d6/db/d28/d61/de3/fea zero size 2022-01-31T19:44:15.838 INFO:tasks.workunit.client.0.smithi167.stdout:6/588: rmdir d8 39 2022-01-31T19:44:15.839 INFO:tasks.workunit.client.0.smithi167.stdout:3/743: rmdir d9/d29/d3b/d41/d6b/dd2 39 2022-01-31T19:44:15.839 INFO:tasks.workunit.client.0.smithi167.stdout:9/623: mkdir dd/d6b/dd5 0 2022-01-31T19:44:15.840 INFO:tasks.workunit.client.0.smithi167.stdout:9/624: dread - dd/d19/d24/d50/dd1/f7b zero size 2022-01-31T19:44:15.840 INFO:tasks.workunit.client.0.smithi167.stdout:9/625: stat dd/d19/d24/d56/d5b/fbe 0 2022-01-31T19:44:15.841 INFO:tasks.workunit.client.0.smithi167.stdout:3/744: read d9/d26/fb1 [1017003,42908] 0 2022-01-31T19:44:15.841 INFO:tasks.workunit.client.0.smithi167.stdout:6/589: mknod d8/d33/d51/d72/cdf 0 2022-01-31T19:44:15.845 INFO:tasks.workunit.client.0.smithi167.stdout:3/745: rename d9/d29/d60/l62 to d9/d29/d3b/ldc 0 2022-01-31T19:44:15.845 INFO:tasks.workunit.client.0.smithi167.stdout:3/746: read d9/d26/f65 [153031,68260] 0 2022-01-31T19:44:15.846 INFO:tasks.workunit.client.0.smithi167.stdout:3/747: chown d9/d26/f98 1107771 1 2022-01-31T19:44:15.846 INFO:tasks.workunit.client.0.smithi167.stdout:3/748: write d9/d29/f4c [1425084,24947] 0 2022-01-31T19:44:15.848 INFO:tasks.workunit.client.0.smithi167.stdout:5/825: dwrite d5/f7 [0,4194304] 0 2022-01-31T19:44:15.848 INFO:tasks.workunit.client.0.smithi167.stdout:4/682: dwrite d0/d6/d7/f9a [0,4194304] 0 2022-01-31T19:44:15.848 INFO:tasks.workunit.client.0.smithi167.stdout:3/749: mkdir d9/d21/ddd 0 2022-01-31T19:44:15.848 INFO:tasks.workunit.client.0.smithi167.stdout:3/750: read - d9/fd3 zero size 2022-01-31T19:44:15.853 INFO:tasks.workunit.client.0.smithi167.stdout:5/826: getdents d5/d6/d16/d18/d43/d45/dd5 0 2022-01-31T19:44:15.855 INFO:tasks.workunit.client.0.smithi167.stdout:3/751: link d9/d29/fb2 d9/d29/d3b/d9b/da3/fde 0 2022-01-31T19:44:15.855 INFO:tasks.workunit.client.0.smithi167.stdout:3/752: write d9/f81 [1598776,128768] 0 2022-01-31T19:44:15.856 INFO:tasks.workunit.client.0.smithi167.stdout:5/827: creat d5/d6/de/f105 x:0 0 0 2022-01-31T19:44:15.858 INFO:tasks.workunit.client.0.smithi167.stdout:2/561: dwrite dd/df/d1b/d3d/d4a/f5b [0,4194304] 0 2022-01-31T19:44:15.858 INFO:tasks.workunit.client.0.smithi167.stdout:8/751: dwrite d6/db/d4e/f58 [0,4194304] 0 2022-01-31T19:44:15.860 INFO:tasks.workunit.client.0.smithi167.stdout:3/753: link d9/d26/f6e d9/d29/d60/dc9/fdf 0 2022-01-31T19:44:15.866 INFO:tasks.workunit.client.0.smithi167.stdout:2/562: rmdir dd/df/d1b/d3d/d4a/d4b/d54 39 2022-01-31T19:44:15.867 INFO:tasks.workunit.client.0.smithi167.stdout:2/563: fsync dd/f15 0 2022-01-31T19:44:15.868 INFO:tasks.workunit.client.0.smithi167.stdout:3/754: symlink d9/le0 0 2022-01-31T19:44:15.868 INFO:tasks.workunit.client.0.smithi167.stdout:2/564: creat dd/df/d1b/d2f/fb3 x:0 0 0 2022-01-31T19:44:15.871 INFO:tasks.workunit.client.0.smithi167.stdout:3/755: symlink d9/d21/le1 0 2022-01-31T19:44:15.872 INFO:tasks.workunit.client.0.smithi167.stdout:3/756: getdents d9/d29/d3b/d41 0 2022-01-31T19:44:15.884 INFO:tasks.workunit.client.0.smithi167.stdout:6/590: dwrite d8/d33/d4f/d87/fb2 [0,4194304] 0 2022-01-31T19:44:15.884 INFO:tasks.workunit.client.0.smithi167.stdout:9/626: dwrite dd/d35/f37 [0,4194304] 0 2022-01-31T19:44:15.884 INFO:tasks.workunit.client.0.smithi167.stdout:9/627: chown dd/d17/f36 132372500 1 2022-01-31T19:44:15.885 INFO:tasks.workunit.client.0.smithi167.stdout:5/828: dwrite d5/f7 [4194304,4194304] 0 2022-01-31T19:44:15.886 INFO:tasks.workunit.client.0.smithi167.stdout:2/565: dwrite dd/df/d3f/d51/f92 [0,4194304] 0 2022-01-31T19:44:15.886 INFO:tasks.workunit.client.0.smithi167.stdout:3/757: dread d9/f27 [0,4194304] 0 2022-01-31T19:44:15.887 INFO:tasks.workunit.client.0.smithi167.stdout:5/829: dread d5/d6/de/fe4 [0,4194304] 0 2022-01-31T19:44:15.887 INFO:tasks.workunit.client.0.smithi167.stdout:6/591: unlink d8/d33/d35/fd3 0 2022-01-31T19:44:15.893 INFO:tasks.workunit.client.0.smithi167.stdout:9/628: dread dd/f16 [0,4194304] 0 2022-01-31T19:44:15.893 INFO:tasks.workunit.client.0.smithi167.stdout:9/629: chown dd/d19/l90 13 1 2022-01-31T19:44:15.898 INFO:tasks.workunit.client.0.smithi167.stdout:0/440: sync 2022-01-31T19:44:15.899 INFO:tasks.workunit.client.0.smithi167.stdout:3/758: rename d9/d29/c37 to d9/d29/d60/dc9/ce2 0 2022-01-31T19:44:15.900 INFO:tasks.workunit.client.0.smithi167.stdout:1/773: sync 2022-01-31T19:44:15.900 INFO:tasks.workunit.client.0.smithi167.stdout:7/897: sync 2022-01-31T19:44:15.900 INFO:tasks.workunit.client.0.smithi167.stdout:2/566: truncate dd/f29 1362736 0 2022-01-31T19:44:15.902 INFO:tasks.workunit.client.0.smithi167.stdout:5/830: mkdir d5/d6/de/d22/d76/db0/d106 0 2022-01-31T19:44:15.903 INFO:tasks.workunit.client.0.smithi167.stdout:6/592: write d8/d33/d51/d5d/f6e [215283,71222] 0 2022-01-31T19:44:15.904 INFO:tasks.workunit.client.0.smithi167.stdout:0/441: mknod d17/d3e/c8b 0 2022-01-31T19:44:15.906 INFO:tasks.workunit.client.0.smithi167.stdout:3/759: unlink d9/c88 0 2022-01-31T19:44:15.906 INFO:tasks.workunit.client.0.smithi167.stdout:1/774: symlink d0/d75/l101 0 2022-01-31T19:44:15.907 INFO:tasks.workunit.client.0.smithi167.stdout:7/898: rename d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/daa/c141 to d8/d2c/d6e/d9a/c142 0 2022-01-31T19:44:15.909 INFO:tasks.workunit.client.0.smithi167.stdout:2/567: mknod dd/d18/cb4 0 2022-01-31T19:44:15.910 INFO:tasks.workunit.client.0.smithi167.stdout:0/442: creat d17/d19/d1c/d39/d7b/f8c x:0 0 0 2022-01-31T19:44:15.912 INFO:tasks.workunit.client.0.smithi167.stdout:6/593: dread d8/d20/d54/f63 [0,4194304] 0 2022-01-31T19:44:15.912 INFO:tasks.workunit.client.0.smithi167.stdout:7/899: write d8/d2c/d6e/d4d/d5e/f91 [1106587,38355] 0 2022-01-31T19:44:15.912 INFO:tasks.workunit.client.0.smithi167.stdout:3/760: creat d9/d29/d3b/d9b/da3/dd9/fe3 x:0 0 0 2022-01-31T19:44:15.913 INFO:tasks.workunit.client.0.smithi167.stdout:2/568: rename dd/d18/d63/fa1 to dd/df/d1b/d3d/d4a/da7/fb5 0 2022-01-31T19:44:15.913 INFO:tasks.workunit.client.0.smithi167.stdout:2/569: fsync dd/df/f79 0 2022-01-31T19:44:15.915 INFO:tasks.workunit.client.0.smithi167.stdout:0/443: link d17/f2f d17/d2c/d87/f8d 0 2022-01-31T19:44:15.915 INFO:tasks.workunit.client.0.smithi167.stdout:0/444: unlink cd 0 2022-01-31T19:44:15.915 INFO:tasks.workunit.client.0.smithi167.stdout:0/445: chown d17/d19/d1c/d39/d44/d49/l53 11270407 1 2022-01-31T19:44:15.918 INFO:tasks.workunit.client.0.smithi167.stdout:6/594: mknod d8/d33/d51/d5d/d64/dab/db9/ce0 0 2022-01-31T19:44:15.919 INFO:tasks.workunit.client.0.smithi167.stdout:6/595: dread - d8/d33/d51/d5d/d64/f88 zero size 2022-01-31T19:44:15.919 INFO:tasks.workunit.client.0.smithi167.stdout:9/630: dwrite dd/d19/d59/f9f [0,4194304] 0 2022-01-31T19:44:15.920 INFO:tasks.workunit.client.0.smithi167.stdout:3/761: mknod d9/d29/d9d/ce4 0 2022-01-31T19:44:15.921 INFO:tasks.workunit.client.0.smithi167.stdout:9/631: dread dd/d35/f44 [0,4194304] 0 2022-01-31T19:44:15.922 INFO:tasks.workunit.client.0.smithi167.stdout:5/831: dwrite d5/d6/de/d22/d76/da3/ffc [0,4194304] 0 2022-01-31T19:44:15.922 INFO:tasks.workunit.client.0.smithi167.stdout:5/832: fsync d5/d6/d16/d18/ffb 0 2022-01-31T19:44:15.923 INFO:tasks.workunit.client.0.smithi167.stdout:2/570: rename dd/df/d1b/d3d/f47 to dd/fb6 0 2022-01-31T19:44:15.923 INFO:tasks.workunit.client.0.smithi167.stdout:2/571: fdatasync dd/df/d3f/f41 0 2022-01-31T19:44:15.923 INFO:tasks.workunit.client.0.smithi167.stdout:1/775: dread d0/d10/d5a/fb9 [0,4194304] 0 2022-01-31T19:44:15.924 INFO:tasks.workunit.client.0.smithi167.stdout:0/446: mknod d17/d19/d1c/d39/d7b/c8e 0 2022-01-31T19:44:15.924 INFO:tasks.workunit.client.0.smithi167.stdout:1/776: write d0/d10/d20/f4e [2612868,60418] 0 2022-01-31T19:44:15.929 INFO:tasks.workunit.client.0.smithi167.stdout:3/762: symlink d9/d29/d3b/d8d/le5 0 2022-01-31T19:44:15.930 INFO:tasks.workunit.client.0.smithi167.stdout:9/632: mkdir dd/d19/d24/d50/dd6 0 2022-01-31T19:44:15.931 INFO:tasks.workunit.client.0.smithi167.stdout:3/763: write d9/d29/d3b/d41/d6b/d89/fd6 [298814,97518] 0 2022-01-31T19:44:15.932 INFO:tasks.workunit.client.0.smithi167.stdout:5/833: link d5/d6/d3a/f48 d5/d61/f107 0 2022-01-31T19:44:15.933 INFO:tasks.workunit.client.0.smithi167.stdout:2/572: mkdir dd/d18/db7 0 2022-01-31T19:44:15.935 INFO:tasks.workunit.client.0.smithi167.stdout:0/447: rename d17/d33/f5c to d17/d19/d1c/f8f 0 2022-01-31T19:44:15.935 INFO:tasks.workunit.client.0.smithi167.stdout:0/448: truncate d17/d3e/f68 1353395 0 2022-01-31T19:44:15.936 INFO:tasks.workunit.client.0.smithi167.stdout:0/449: readlink d17/d19/l28 0 2022-01-31T19:44:15.936 INFO:tasks.workunit.client.0.smithi167.stdout:0/450: fdatasync d17/d19/f2d 0 2022-01-31T19:44:15.937 INFO:tasks.workunit.client.0.smithi167.stdout:0/451: write d17/d2c/d87/f8d [7299390,101054] 0 2022-01-31T19:44:15.937 INFO:tasks.workunit.client.0.smithi167.stdout:0/452: write d17/d2c/f79 [4444532,73601] 0 2022-01-31T19:44:15.937 INFO:tasks.workunit.client.0.smithi167.stdout:7/900: dwrite d8/d2c/d6e/d4d/d67/fff [0,4194304] 0 2022-01-31T19:44:15.937 INFO:tasks.workunit.client.0.smithi167.stdout:5/834: mknod d5/d6/de/d22/d6d/da0/c108 0 2022-01-31T19:44:15.939 INFO:tasks.workunit.client.0.smithi167.stdout:1/777: rename d0/d75/db4 to d0/d11/d102 0 2022-01-31T19:44:15.941 INFO:tasks.workunit.client.0.smithi167.stdout:5/835: symlink d5/d6/de/d22/d4d/l109 0 2022-01-31T19:44:15.942 INFO:tasks.workunit.client.0.smithi167.stdout:6/596: dwrite d8/d33/f40 [4194304,4194304] 0 2022-01-31T19:44:15.942 INFO:tasks.workunit.client.0.smithi167.stdout:9/633: dread dd/d17/f1f [0,4194304] 0 2022-01-31T19:44:15.943 INFO:tasks.workunit.client.0.smithi167.stdout:9/634: write dd/d17/f3e [1080401,33715] 0 2022-01-31T19:44:15.943 INFO:tasks.workunit.client.0.smithi167.stdout:5/836: write d5/d6/de/d22/d76/f80 [2664043,101522] 0 2022-01-31T19:44:15.948 INFO:tasks.workunit.client.0.smithi167.stdout:6/597: dread d8/f1a [0,4194304] 0 2022-01-31T19:44:15.949 INFO:tasks.workunit.client.0.smithi167.stdout:3/764: dread d9/d29/d3b/d41/d6b/d89/fd6 [0,4194304] 0 2022-01-31T19:44:15.951 INFO:tasks.workunit.client.0.smithi167.stdout:3/765: read d9/f2c [300038,62260] 0 2022-01-31T19:44:15.951 INFO:tasks.workunit.client.0.smithi167.stdout:3/766: fdatasync d9/d56/f6d 0 2022-01-31T19:44:15.957 INFO:tasks.workunit.client.0.smithi167.stdout:6/598: read d8/d33/d35/f61 [21944,75323] 0 2022-01-31T19:44:15.958 INFO:tasks.workunit.client.0.smithi167.stdout:1/778: dread d0/f3a [0,4194304] 0 2022-01-31T19:44:15.958 INFO:tasks.workunit.client.0.smithi167.stdout:1/779: dread - d0/d62/d7f/d82/fe2 zero size 2022-01-31T19:44:15.958 INFO:tasks.workunit.client.0.smithi167.stdout:1/780: getdents d0/d100 0 2022-01-31T19:44:15.958 INFO:tasks.workunit.client.0.smithi167.stdout:1/781: fdatasync d0/d75/dbd/fed 0 2022-01-31T19:44:15.960 INFO:tasks.workunit.client.0.smithi167.stdout:5/837: rename d5/d6/d16/d18/d43/ca9 to d5/d6/d16/d18/d5b/d63/c10a 0 2022-01-31T19:44:15.960 INFO:tasks.workunit.client.0.smithi167.stdout:5/838: chown d5/d6/d16/d18/d43/d45/d69/dc4/f7c 5 1 2022-01-31T19:44:15.962 INFO:tasks.workunit.client.0.smithi167.stdout:9/635: dread dd/d19/d24/d50/dd1/fc2 [0,4194304] 0 2022-01-31T19:44:15.962 INFO:tasks.workunit.client.0.smithi167.stdout:5/839: write d5/d6/d3a/f4e [3854115,9692] 0 2022-01-31T19:44:15.971 INFO:tasks.workunit.client.0.smithi167.stdout:1/782: creat d0/d10/d5a/df2/f103 x:0 0 0 2022-01-31T19:44:15.973 INFO:tasks.workunit.client.0.smithi167.stdout:7/901: dwrite d8/d35/f5d [0,4194304] 0 2022-01-31T19:44:15.973 INFO:tasks.workunit.client.0.smithi167.stdout:9/636: rename dd/d19/f6d to dd/d19/d24/d56/dc9/fd7 0 2022-01-31T19:44:15.974 INFO:tasks.workunit.client.0.smithi167.stdout:0/453: dwrite d17/f2f [4194304,4194304] 0 2022-01-31T19:44:15.975 INFO:tasks.workunit.client.0.smithi167.stdout:3/767: dwrite d9/d56/f91 [0,4194304] 0 2022-01-31T19:44:15.975 INFO:tasks.workunit.client.0.smithi167.stdout:3/768: fsync d9/fc 0 2022-01-31T19:44:15.976 INFO:tasks.workunit.client.0.smithi167.stdout:7/902: creat d8/dc/d18/d24/dcc/f143 x:0 0 0 2022-01-31T19:44:15.979 INFO:tasks.workunit.client.0.smithi167.stdout:9/637: mknod dd/d19/d24/d56/d5b/da5/cd8 0 2022-01-31T19:44:15.985 INFO:tasks.workunit.client.0.smithi167.stdout:1/783: dread d0/d10/d5a/d33/d4b/d1f/fdc [0,4194304] 0 2022-01-31T19:44:15.985 INFO:tasks.workunit.client.0.smithi167.stdout:1/784: dread - d0/db3/feb zero size 2022-01-31T19:44:15.987 INFO:tasks.workunit.client.0.smithi167.stdout:1/785: mknod d0/db3/c104 0 2022-01-31T19:44:15.987 INFO:tasks.workunit.client.0.smithi167.stdout:9/638: dread dd/d35/f9a [0,4194304] 0 2022-01-31T19:44:15.987 INFO:tasks.workunit.client.0.smithi167.stdout:9/639: chown dd 163 1 2022-01-31T19:44:15.988 INFO:tasks.workunit.client.0.smithi167.stdout:1/786: creat d0/d10/d5a/d33/d4b/da5/da9/f105 x:0 0 0 2022-01-31T19:44:15.989 INFO:tasks.workunit.client.0.smithi167.stdout:9/640: mkdir dd/dd9 0 2022-01-31T19:44:15.989 INFO:tasks.workunit.client.0.smithi167.stdout:9/641: fsync dd/d34/f61 0 2022-01-31T19:44:15.989 INFO:tasks.workunit.client.0.smithi167.stdout:9/642: readlink l2 0 2022-01-31T19:44:15.990 INFO:tasks.workunit.client.0.smithi167.stdout:9/643: stat dd/d19/d59/fae 0 2022-01-31T19:44:15.997 INFO:tasks.workunit.client.0.smithi167.stdout:5/840: dwrite d5/d6/d16/d18/d43/d45/dd5/dbc/f103 [0,4194304] 0 2022-01-31T19:44:15.997 INFO:tasks.workunit.client.0.smithi167.stdout:5/841: readlink d5/d6/de/d22/d76/lf8 0 2022-01-31T19:44:15.997 INFO:tasks.workunit.client.0.smithi167.stdout:5/842: chown d5/d6/d16/d18/d43/d45/dd5 2339 1 2022-01-31T19:44:15.999 INFO:tasks.workunit.client.0.smithi167.stdout:3/769: dwrite d9/d21/f99 [0,4194304] 0 2022-01-31T19:44:15.999 INFO:tasks.workunit.client.0.smithi167.stdout:3/770: chown d9/d29/d3b/d41/c4f 20 1 2022-01-31T19:44:16.000 INFO:tasks.workunit.client.0.smithi167.stdout:3/771: read - d9/d26/fdb zero size 2022-01-31T19:44:16.000 INFO:tasks.workunit.client.0.smithi167.stdout:6/599: dwrite d8/d33/d51/d7e/dc4/f97 [0,4194304] 0 2022-01-31T19:44:16.001 INFO:tasks.workunit.client.0.smithi167.stdout:5/843: dread d5/d61/f107 [0,4194304] 0 2022-01-31T19:44:16.001 INFO:tasks.workunit.client.0.smithi167.stdout:5/844: readlink d5/d6/d16/l39 0 2022-01-31T19:44:16.004 INFO:tasks.workunit.client.0.smithi167.stdout:1/787: dread d0/d10/d20/fb2 [0,4194304] 0 2022-01-31T19:44:16.005 INFO:tasks.workunit.client.0.smithi167.stdout:0/454: truncate d17/d2c/d87/f8d 7368198 0 2022-01-31T19:44:16.006 INFO:tasks.workunit.client.0.smithi167.stdout:9/644: dwrite dd/d34/f51 [0,4194304] 0 2022-01-31T19:44:16.006 INFO:tasks.workunit.client.0.smithi167.stdout:9/645: chown dd/d19/d24 8 1 2022-01-31T19:44:16.006 INFO:tasks.workunit.client.0.smithi167.stdout:9/646: write dd/d17/f2e [4389079,1374] 0 2022-01-31T19:44:16.006 INFO:tasks.workunit.client.0.smithi167.stdout:9/647: truncate dd/d19/d59/f98 1726933 0 2022-01-31T19:44:16.007 INFO:tasks.workunit.client.0.smithi167.stdout:3/772: creat d9/d7f/da8/fe6 x:0 0 0 2022-01-31T19:44:16.010 INFO:tasks.workunit.client.0.smithi167.stdout:6/600: write d8/d33/d51/d6a/fa3 [1010847,105074] 0 2022-01-31T19:44:16.010 INFO:tasks.workunit.client.0.smithi167.stdout:6/601: truncate d8/d33/d51/d5d/d64/dab/fd7 1076565 0 2022-01-31T19:44:16.012 INFO:tasks.workunit.client.0.smithi167.stdout:5/845: truncate d5/f7 8000340 0 2022-01-31T19:44:16.013 INFO:tasks.workunit.client.0.smithi167.stdout:5/846: chown d5/d6/de/d22/d76/fd9 33949178 1 2022-01-31T19:44:16.013 INFO:tasks.workunit.client.0.smithi167.stdout:5/847: read d5/d6/f28 [264620,64403] 0 2022-01-31T19:44:16.014 INFO:tasks.workunit.client.0.smithi167.stdout:5/848: chown d5/d6/de/f4c 7826 1 2022-01-31T19:44:16.014 INFO:tasks.workunit.client.0.smithi167.stdout:5/849: chown d5/d6/de/d22/l71 14906 1 2022-01-31T19:44:16.014 INFO:tasks.workunit.client.0.smithi167.stdout:0/455: mknod d17/d2c/d87/c90 0 2022-01-31T19:44:16.015 INFO:tasks.workunit.client.0.smithi167.stdout:9/648: mkdir dd/d17/dda 0 2022-01-31T19:44:16.016 INFO:tasks.workunit.client.0.smithi167.stdout:3/773: mkdir d9/de7 0 2022-01-31T19:44:16.017 INFO:tasks.workunit.client.0.smithi167.stdout:6/602: creat d8/d62/fe1 x:0 0 0 2022-01-31T19:44:16.018 INFO:tasks.workunit.client.0.smithi167.stdout:5/850: creat d5/d6/d16/d18/d43/d45/dd5/dbc/dcc/ded/f10b x:0 0 0 2022-01-31T19:44:16.019 INFO:tasks.workunit.client.0.smithi167.stdout:1/788: dwrite d0/d10/d5a/d33/d4b/d1f/fa1 [0,4194304] 0 2022-01-31T19:44:16.020 INFO:tasks.workunit.client.0.smithi167.stdout:0/456: mknod d17/d2c/c91 0 2022-01-31T19:44:16.022 INFO:tasks.workunit.client.0.smithi167.stdout:3/774: rename d9/le0 to d9/d29/d3b/d41/le8 0 2022-01-31T19:44:16.022 INFO:tasks.workunit.client.0.smithi167.stdout:3/775: creat d9/d26/fe9 x:0 0 0 2022-01-31T19:44:16.022 INFO:tasks.workunit.client.0.smithi167.stdout:3/776: fdatasync d9/d26/f8b 0 2022-01-31T19:44:16.024 INFO:tasks.workunit.client.0.smithi167.stdout:6/603: mknod d8/d33/d51/d72/ce2 0 2022-01-31T19:44:16.024 INFO:tasks.workunit.client.0.smithi167.stdout:6/604: fdatasync d8/d33/d51/d72/dc2/fce 0 2022-01-31T19:44:16.025 INFO:tasks.workunit.client.0.smithi167.stdout:6/605: write d8/d62/fc6 [514508,124716] 0 2022-01-31T19:44:16.025 INFO:tasks.workunit.client.0.smithi167.stdout:5/851: symlink d5/d6/de/d22/d76/db0/dee/l10c 0 2022-01-31T19:44:16.028 INFO:tasks.workunit.client.0.smithi167.stdout:0/457: unlink d17/d19/d1c/d39/d44/d49/l53 0 2022-01-31T19:44:16.028 INFO:tasks.workunit.client.0.smithi167.stdout:0/458: write d17/d2c/f76 [702619,39778] 0 2022-01-31T19:44:16.028 INFO:tasks.workunit.client.0.smithi167.stdout:0/459: write d17/d3e/d7e/f73 [807383,107940] 0 2022-01-31T19:44:16.029 INFO:tasks.workunit.client.0.smithi167.stdout:1/789: rename d0/d75/d7a/cb1 to d0/d75/dec/c106 0 2022-01-31T19:44:16.030 INFO:tasks.workunit.client.0.smithi167.stdout:3/777: getdents d9/d7f 0 2022-01-31T19:44:16.031 INFO:tasks.workunit.client.0.smithi167.stdout:5/852: mkdir d5/d6/de/d22/d4d/d10d 0 2022-01-31T19:44:16.033 INFO:tasks.workunit.client.0.smithi167.stdout:9/649: dwrite dd/d17/f2f [4194304,4194304] 0 2022-01-31T19:44:16.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/460: symlink d17/d69/l92 0 2022-01-31T19:44:16.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/461: dread - d17/d3e/d7e/f83 zero size 2022-01-31T19:44:16.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/462: write d17/d3e/f68 [1581677,35180] 0 2022-01-31T19:44:16.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/463: chown d17/d19/d1c/d39/d7b/c8e 6092 1 2022-01-31T19:44:16.039 INFO:tasks.workunit.client.0.smithi167.stdout:6/606: rename d8/d33/d51/d5d/d86/dbd/fdd to d8/d33/d51/d5d/d64/dab/fe3 0 2022-01-31T19:44:16.039 INFO:tasks.workunit.client.0.smithi167.stdout:6/607: truncate d8/d62/fe1 633773 0 2022-01-31T19:44:16.041 INFO:tasks.workunit.client.0.smithi167.stdout:3/778: link d9/l14 d9/d29/d60/lea 0 2022-01-31T19:44:16.042 INFO:tasks.workunit.client.0.smithi167.stdout:3/779: dread - d9/d56/fab zero size 2022-01-31T19:44:16.045 INFO:tasks.workunit.client.0.smithi167.stdout:6/608: creat d8/d33/fe4 x:0 0 0 2022-01-31T19:44:16.045 INFO:tasks.workunit.client.0.smithi167.stdout:6/609: write d8/d33/d51/d72/fc9 [571469,16378] 0 2022-01-31T19:44:16.046 INFO:tasks.workunit.client.0.smithi167.stdout:3/780: write d9/d21/f99 [3980908,126411] 0 2022-01-31T19:44:16.046 INFO:tasks.workunit.client.0.smithi167.stdout:6/610: rmdir d8 39 2022-01-31T19:44:16.047 INFO:tasks.workunit.client.0.smithi167.stdout:3/781: mkdir d9/d29/d3b/d41/deb 0 2022-01-31T19:44:16.063 INFO:tasks.workunit.client.0.smithi167.stdout:1/790: dwrite d0/d10/d20/d7c/fb8 [0,4194304] 0 2022-01-31T19:44:16.064 INFO:tasks.workunit.client.0.smithi167.stdout:1/791: dread - d0/d10/d5a/d33/d4b/d1f/fee zero size 2022-01-31T19:44:16.064 INFO:tasks.workunit.client.0.smithi167.stdout:1/792: unlink d0/d11/c35 0 2022-01-31T19:44:16.064 INFO:tasks.workunit.client.0.smithi167.stdout:1/793: write d0/d10/d5a/d33/d4b/da5/da9/dad/fff [768325,22751] 0 2022-01-31T19:44:16.065 INFO:tasks.workunit.client.0.smithi167.stdout:5/853: dwrite d5/d6/d16/d18/fae [0,4194304] 0 2022-01-31T19:44:16.066 INFO:tasks.workunit.client.0.smithi167.stdout:1/794: mknod d0/d75/dec/c107 0 2022-01-31T19:44:16.067 INFO:tasks.workunit.client.0.smithi167.stdout:1/795: creat d0/d10/d5a/d33/d4b/da5/da9/dad/f108 x:0 0 0 2022-01-31T19:44:16.067 INFO:tasks.workunit.client.0.smithi167.stdout:1/796: dread - d0/d62/fc6 zero size 2022-01-31T19:44:16.070 INFO:tasks.workunit.client.0.smithi167.stdout:5/854: unlink d5/d6/d16/d18/d43/d45/d69/d90/lde 0 2022-01-31T19:44:16.070 INFO:tasks.workunit.client.0.smithi167.stdout:5/855: readlink d5/d6/de/l50 0 2022-01-31T19:44:16.071 INFO:tasks.workunit.client.0.smithi167.stdout:1/797: rmdir d0/d10/d5a/d33 39 2022-01-31T19:44:16.071 INFO:tasks.workunit.client.0.smithi167.stdout:1/798: dread - d0/d62/d7f/d82/fe2 zero size 2022-01-31T19:44:16.072 INFO:tasks.workunit.client.0.smithi167.stdout:1/799: chown d0/d10/d5a/d33/d4b/da5/dc1/cda 248675 1 2022-01-31T19:44:16.073 INFO:tasks.workunit.client.0.smithi167.stdout:1/800: mknod d0/d62/db7/c109 0 2022-01-31T19:44:16.074 INFO:tasks.workunit.client.0.smithi167.stdout:5/856: rmdir d5/d6/de/d22/d76 39 2022-01-31T19:44:16.080 INFO:tasks.workunit.client.0.smithi167.stdout:1/801: dread d0/d10/d20/f4e [0,4194304] 0 2022-01-31T19:44:16.080 INFO:tasks.workunit.client.0.smithi167.stdout:1/802: chown d0/d10/d20/f4e 2125786838 1 2022-01-31T19:44:16.082 INFO:tasks.workunit.client.0.smithi167.stdout:9/650: dwrite dd/d19/d59/f9f [8388608,4194304] 0 2022-01-31T19:44:16.084 INFO:tasks.workunit.client.0.smithi167.stdout:6/611: dwrite d8/f7a [0,4194304] 0 2022-01-31T19:44:16.084 INFO:tasks.workunit.client.0.smithi167.stdout:9/651: mknod dd/d34/cdb 0 2022-01-31T19:44:16.085 INFO:tasks.workunit.client.0.smithi167.stdout:9/652: chown dd/d19/l22 1 1 2022-01-31T19:44:16.085 INFO:tasks.workunit.client.0.smithi167.stdout:3/782: dwrite d9/f11 [0,4194304] 0 2022-01-31T19:44:16.086 INFO:tasks.workunit.client.0.smithi167.stdout:3/783: fdatasync d9/d56/f6d 0 2022-01-31T19:44:16.087 INFO:tasks.workunit.client.0.smithi167.stdout:0/464: dwrite ff [0,4194304] 0 2022-01-31T19:44:16.088 INFO:tasks.workunit.client.0.smithi167.stdout:5/857: dread d5/d6/de/d22/d76/db0/fb6 [0,4194304] 0 2022-01-31T19:44:16.088 INFO:tasks.workunit.client.0.smithi167.stdout:5/858: chown d5/d6/d16/d18/d5b/d9f/cc9 2238 1 2022-01-31T19:44:16.088 INFO:tasks.workunit.client.0.smithi167.stdout:3/784: dread d9/d29/d3b/d41/d6b/f39 [0,4194304] 0 2022-01-31T19:44:16.090 INFO:tasks.workunit.client.0.smithi167.stdout:6/612: creat d8/d33/d51/d72/dad/fe5 x:0 0 0 2022-01-31T19:44:16.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:15 smithi167 conmon[32206]: cluster 2022-01-31T19:44:14.432785+0000 mgr.smithi171.asyxnc (mgr 2022-01-31T19:44:16.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:15 smithi167 conmon[32206]: .24387) 19 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 1.4 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 89 MiB/s rd, 183 MiB/s wr, 284 op/s 2022-01-31T19:44:16.094 INFO:tasks.workunit.client.0.smithi167.stdout:3/785: read d9/d29/d3b/d9b/da3/db0/fb9 [1182505,35126] 0 2022-01-31T19:44:16.100 INFO:tasks.workunit.client.0.smithi167.stdout:9/653: creat dd/d19/fdc x:0 0 0 2022-01-31T19:44:16.104 INFO:tasks.workunit.client.0.smithi167.stdout:6/613: dread d8/d20/f26 [0,4194304] 0 2022-01-31T19:44:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:15 smithi171 conmon[35325]: cluster 2022-01-31T19:44:14.432785+0000 2022-01-31T19:44:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:15 smithi171 conmon[35325]: mgr.smithi171.asyxnc (mgr.24387) 19 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 1.4 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 89 MiB/s rd, 183 MiB/s wr, 284 op/s 2022-01-31T19:44:16.112 INFO:tasks.workunit.client.0.smithi167.stdout:3/786: rename d9/d29/d3b/d41/d6b/f51 to d9/d21/fec 0 2022-01-31T19:44:16.113 INFO:tasks.workunit.client.0.smithi167.stdout:1/803: dwrite d0/d10/d5a/d33/d4b/fa2 [0,4194304] 0 2022-01-31T19:44:16.113 INFO:tasks.workunit.client.0.smithi167.stdout:9/654: symlink dd/d17/d80/ldd 0 2022-01-31T19:44:16.113 INFO:tasks.workunit.client.0.smithi167.stdout:1/804: chown d0/d11/lca 66 1 2022-01-31T19:44:16.114 INFO:tasks.workunit.client.0.smithi167.stdout:6/614: creat d8/d33/d51/d5d/d64/dab/db9/fe6 x:0 0 0 2022-01-31T19:44:16.115 INFO:tasks.workunit.client.0.smithi167.stdout:3/787: creat d9/d29/d60/fed x:0 0 0 2022-01-31T19:44:16.116 INFO:tasks.workunit.client.0.smithi167.stdout:6/615: write d8/fd [3257034,66019] 0 2022-01-31T19:44:16.117 INFO:tasks.workunit.client.0.smithi167.stdout:6/616: write d8/d33/fe4 [85696,120929] 0 2022-01-31T19:44:16.117 INFO:tasks.workunit.client.0.smithi167.stdout:6/617: write d8/d20/d54/fa7 [658623,106862] 0 2022-01-31T19:44:16.117 INFO:tasks.workunit.client.0.smithi167.stdout:5/859: dwrite d5/d6/de/d22/f35 [0,4194304] 0 2022-01-31T19:44:16.118 INFO:tasks.workunit.client.0.smithi167.stdout:1/805: rename d0/db3/fc5 to d0/d62/d7f/d82/f10a 0 2022-01-31T19:44:16.119 INFO:tasks.workunit.client.0.smithi167.stdout:0/465: dwrite d17/d19/d1c/d39/d7b/f81 [0,4194304] 0 2022-01-31T19:44:16.125 INFO:tasks.workunit.client.0.smithi167.stdout:5/860: link d5/d61/f7f d5/f10e 0 2022-01-31T19:44:16.130 INFO:tasks.workunit.client.0.smithi167.stdout:3/788: dread d9/d29/d60/f49 [0,4194304] 0 2022-01-31T19:44:16.130 INFO:tasks.workunit.client.0.smithi167.stdout:9/655: rename dd/d17/d80/d4c/f91 to dd/d19/d24/d50/fde 0 2022-01-31T19:44:16.133 INFO:tasks.workunit.client.0.smithi167.stdout:1/806: dwrite d0/d75/d7a/f66 [0,4194304] 0 2022-01-31T19:44:16.134 INFO:tasks.workunit.client.0.smithi167.stdout:1/807: readlink d0/d10/d5a/d33/d4b/d1f/le1 0 2022-01-31T19:44:16.138 INFO:tasks.workunit.client.0.smithi167.stdout:0/466: creat d17/d19/d1c/d39/d44/f93 x:0 0 0 2022-01-31T19:44:16.145 INFO:tasks.workunit.client.0.smithi167.stdout:5/861: dwrite d5/d6/d16/d18/d43/d45/d69/dc4/fbf [0,4194304] 0 2022-01-31T19:44:16.145 INFO:tasks.workunit.client.0.smithi167.stdout:5/862: creat d5/d6/d3a/f10f x:0 0 0 2022-01-31T19:44:16.145 INFO:tasks.workunit.client.0.smithi167.stdout:5/863: getdents d5/d6/de/d22/d4d/d10d 0 2022-01-31T19:44:16.145 INFO:tasks.workunit.client.0.smithi167.stdout:5/864: write d5/d6/ffd [744522,13034] 0 2022-01-31T19:44:16.148 INFO:tasks.workunit.client.0.smithi167.stdout:3/789: link d9/d29/f33 d9/d29/d3b/fee 0 2022-01-31T19:44:16.148 INFO:tasks.workunit.client.0.smithi167.stdout:3/790: fsync f7 0 2022-01-31T19:44:16.149 INFO:tasks.workunit.client.0.smithi167.stdout:9/656: rename dd/d19/d24/c4b to dd/d6b/dc1/cdf 0 2022-01-31T19:44:16.152 INFO:tasks.workunit.client.0.smithi167.stdout:5/865: creat d5/d6/d16/d18/d5b/f110 x:0 0 0 2022-01-31T19:44:16.152 INFO:tasks.workunit.client.0.smithi167.stdout:5/866: dread - d5/d6/de/f105 zero size 2022-01-31T19:44:16.152 INFO:tasks.workunit.client.0.smithi167.stdout:5/867: fdatasync d5/d6/d16/d18/d5b/f110 0 2022-01-31T19:44:16.152 INFO:tasks.workunit.client.0.smithi167.stdout:5/868: creat d5/d6/f111 x:0 0 0 2022-01-31T19:44:16.153 INFO:tasks.workunit.client.0.smithi167.stdout:3/791: mkdir d9/d29/d3b/d9b/da3/dd9/def 0 2022-01-31T19:44:16.154 INFO:tasks.workunit.client.0.smithi167.stdout:9/657: rename dd/d19/d24/d50/cb1 to dd/d17/ce0 0 2022-01-31T19:44:16.155 INFO:tasks.workunit.client.0.smithi167.stdout:5/869: creat d5/dcd/f112 x:0 0 0 2022-01-31T19:44:16.157 INFO:tasks.workunit.client.0.smithi167.stdout:0/467: dwrite d17/d19/d1c/f8f [0,4194304] 0 2022-01-31T19:44:16.158 INFO:tasks.workunit.client.0.smithi167.stdout:1/808: dwrite d0/d75/dbd/fed [0,4194304] 0 2022-01-31T19:44:16.158 INFO:tasks.workunit.client.0.smithi167.stdout:1/809: fdatasync d0/d10/d20/f28 0 2022-01-31T19:44:16.159 INFO:tasks.workunit.client.0.smithi167.stdout:4/683: sync 2022-01-31T19:44:16.159 INFO:tasks.workunit.client.0.smithi167.stdout:8/752: sync 2022-01-31T19:44:16.159 INFO:tasks.workunit.client.0.smithi167.stdout:8/753: fsync d6/db/d28/d61/dd5/f90 0 2022-01-31T19:44:16.160 INFO:tasks.workunit.client.0.smithi167.stdout:3/792: creat d9/d7f/dbb/ff0 x:0 0 0 2022-01-31T19:44:16.160 INFO:tasks.workunit.client.0.smithi167.stdout:5/870: creat d5/d6/de/d22/d76/db0/f113 x:0 0 0 2022-01-31T19:44:16.165 INFO:tasks.workunit.client.0.smithi167.stdout:8/754: mknod d6/d1e/d2d/cf4 0 2022-01-31T19:44:16.172 INFO:tasks.workunit.client.0.smithi167.stdout:4/684: write d0/d6/d7/d39/d93/fb7 [2424186,36637] 0 2022-01-31T19:44:16.172 INFO:tasks.workunit.client.0.smithi167.stdout:1/810: dread d0/d10/d5a/fb9 [0,4194304] 0 2022-01-31T19:44:16.173 INFO:tasks.workunit.client.0.smithi167.stdout:1/811: creat d0/d10/d5a/d33/d4b/d1f/f10b x:0 0 0 2022-01-31T19:44:16.173 INFO:tasks.workunit.client.0.smithi167.stdout:5/871: dread d5/d6/de/d22/f35 [0,4194304] 0 2022-01-31T19:44:16.175 INFO:tasks.workunit.client.0.smithi167.stdout:8/755: getdents d6/db/d28/d61/de3 0 2022-01-31T19:44:16.175 INFO:tasks.workunit.client.0.smithi167.stdout:4/685: mknod d0/d6/d7/d26/d31/ddf/d75/ce1 0 2022-01-31T19:44:16.177 INFO:tasks.workunit.client.0.smithi167.stdout:1/812: creat d0/d10/d5a/d33/d4b/da5/dc1/f10c x:0 0 0 2022-01-31T19:44:16.178 INFO:tasks.workunit.client.0.smithi167.stdout:5/872: write d5/d6/d16/d18/d43/d45/d69/dc4/dbb/ff2 [3053585,9962] 0 2022-01-31T19:44:16.178 INFO:tasks.workunit.client.0.smithi167.stdout:5/873: truncate d5/d6/de/d22/d76/db0/f113 130415 0 2022-01-31T19:44:16.179 INFO:tasks.workunit.client.0.smithi167.stdout:1/813: rename d0/d10/d5a/d33/d4b/f65 to d0/d100/f10d 0 2022-01-31T19:44:16.180 INFO:tasks.workunit.client.0.smithi167.stdout:4/686: dread d0/d6/d7/d26/d40/f32 [0,4194304] 0 2022-01-31T19:44:16.180 INFO:tasks.workunit.client.0.smithi167.stdout:4/687: truncate d0/d6/d7/d26/d31/ddf/da2/fc4 963493 0 2022-01-31T19:44:16.181 INFO:tasks.workunit.client.0.smithi167.stdout:4/688: chown d0/d6/d7/d26/d31/f5e 2603 1 2022-01-31T19:44:16.182 INFO:tasks.workunit.client.0.smithi167.stdout:5/874: unlink d5/d6/de/d22/f35 0 2022-01-31T19:44:16.183 INFO:tasks.workunit.client.0.smithi167.stdout:1/814: creat d0/d62/db7/f10e x:0 0 0 2022-01-31T19:44:16.183 INFO:tasks.workunit.client.0.smithi167.stdout:1/815: chown d0/d10/d5a/d33/d4b/d1f/fdc 195404 1 2022-01-31T19:44:16.185 INFO:tasks.workunit.client.0.smithi167.stdout:4/689: rename d0/d6/dab/dd8 to d0/d6/d7/d26/d31/d5c/de2 0 2022-01-31T19:44:16.185 INFO:tasks.workunit.client.0.smithi167.stdout:4/690: write d0/d6/d7/d26/f2f [1200614,62891] 0 2022-01-31T19:44:16.185 INFO:tasks.workunit.client.0.smithi167.stdout:4/691: write d0/d6/d7/d26/d40/f32 [4340213,56963] 0 2022-01-31T19:44:16.185 INFO:tasks.workunit.client.0.smithi167.stdout:4/692: write d0/d6/d7/f8 [6286376,34588] 0 2022-01-31T19:44:16.186 INFO:tasks.workunit.client.0.smithi167.stdout:4/693: read d0/d6/f58 [219374,50618] 0 2022-01-31T19:44:16.186 INFO:tasks.workunit.client.0.smithi167.stdout:5/875: mkdir d5/d6/de/d22/d4d/d114 0 2022-01-31T19:44:16.186 INFO:tasks.workunit.client.0.smithi167.stdout:5/876: dread d5/d6/d3a/f8f [0,4194304] 0 2022-01-31T19:44:16.188 INFO:tasks.workunit.client.0.smithi167.stdout:5/877: getdents d5/d6/d16/d18 0 2022-01-31T19:44:16.188 INFO:tasks.workunit.client.0.smithi167.stdout:5/878: readlink d5/d6/de/d22/d4d/l109 0 2022-01-31T19:44:16.189 INFO:tasks.workunit.client.0.smithi167.stdout:5/879: dread - d5/d6/d16/d18/d5b/d9f/fdc zero size 2022-01-31T19:44:16.189 INFO:tasks.workunit.client.0.smithi167.stdout:5/880: chown d5/d6/de/d22/d76/db0/dd8/lf3 0 1 2022-01-31T19:44:16.189 INFO:tasks.workunit.client.0.smithi167.stdout:5/881: creat d5/d6/d16/d18/d43/d45/d69/dc4/f115 x:0 0 0 2022-01-31T19:44:16.190 INFO:tasks.workunit.client.0.smithi167.stdout:5/882: chown d5/d6/f41 3029225 1 2022-01-31T19:44:16.190 INFO:tasks.workunit.client.0.smithi167.stdout:0/468: dwrite d17/d19/d1c/f8f [0,4194304] 0 2022-01-31T19:44:16.192 INFO:tasks.workunit.client.0.smithi167.stdout:3/793: dwrite d9/d26/fe9 [0,4194304] 0 2022-01-31T19:44:16.192 INFO:tasks.workunit.client.0.smithi167.stdout:3/794: fdatasync d9/d7f/dbb/ff0 0 2022-01-31T19:44:16.193 INFO:tasks.workunit.client.0.smithi167.stdout:9/658: dwrite dd/d19/d24/d50/dd1/fc2 [4194304,4194304] 0 2022-01-31T19:44:16.194 INFO:tasks.workunit.client.0.smithi167.stdout:8/756: dread d6/d1e/d4c/dba/fd4 [0,4194304] 0 2022-01-31T19:44:16.195 INFO:tasks.workunit.client.0.smithi167.stdout:8/757: creat d6/d1e/dcf/ff5 x:0 0 0 2022-01-31T19:44:16.199 INFO:tasks.workunit.client.0.smithi167.stdout:0/469: creat d17/d69/f94 x:0 0 0 2022-01-31T19:44:16.201 INFO:tasks.workunit.client.0.smithi167.stdout:4/694: dread d0/d6/d7/d39/d4e/d61/f95 [4194304,4194304] 0 2022-01-31T19:44:16.207 INFO:tasks.workunit.client.0.smithi167.stdout:8/758: mknod d6/d1e/d2d/d6b/cf6 0 2022-01-31T19:44:16.208 INFO:tasks.workunit.client.0.smithi167.stdout:0/470: creat d17/d33/f95 x:0 0 0 2022-01-31T19:44:16.208 INFO:tasks.workunit.client.0.smithi167.stdout:0/471: write d17/d19/f2d [997212,112445] 0 2022-01-31T19:44:16.209 INFO:tasks.workunit.client.0.smithi167.stdout:8/759: write d6/d1e/d4c/dba/fd4 [2727587,8914] 0 2022-01-31T19:44:16.212 INFO:tasks.workunit.client.0.smithi167.stdout:4/695: dread d0/d6/d7/d26/fc9 [0,4194304] 0 2022-01-31T19:44:16.214 INFO:tasks.workunit.client.0.smithi167.stdout:0/472: rename d17/d19/f2d to d17/d19/d1c/d39/d7b/f96 0 2022-01-31T19:44:16.214 INFO:tasks.workunit.client.0.smithi167.stdout:0/473: chown d17/d3e/c6b 39 1 2022-01-31T19:44:16.214 INFO:tasks.workunit.client.0.smithi167.stdout:0/474: fsync d17/d19/d1c/f32 0 2022-01-31T19:44:16.215 INFO:tasks.workunit.client.0.smithi167.stdout:8/760: link d6/db/d6a/c87 d6/db/de/d3d/cf7 0 2022-01-31T19:44:16.217 INFO:tasks.workunit.client.0.smithi167.stdout:4/696: mkdir d0/d6/d7/d26/d31/ddf/d75/de3 0 2022-01-31T19:44:16.217 INFO:tasks.workunit.client.0.smithi167.stdout:4/697: chown d0/d6/d7/d26/d31/ddf/d75 72 1 2022-01-31T19:44:16.217 INFO:tasks.workunit.client.0.smithi167.stdout:0/475: creat d17/d2c/d87/f97 x:0 0 0 2022-01-31T19:44:16.219 INFO:tasks.workunit.client.0.smithi167.stdout:8/761: rename d6/db/l25 to d6/db/d28/daf/lf8 0 2022-01-31T19:44:16.220 INFO:tasks.workunit.client.0.smithi167.stdout:0/476: mkdir d17/d19/d1c/d39/d44/d88/d98 0 2022-01-31T19:44:16.220 INFO:tasks.workunit.client.0.smithi167.stdout:8/762: truncate d6/fab 380029 0 2022-01-31T19:44:16.220 INFO:tasks.workunit.client.0.smithi167.stdout:8/763: dread - d6/db/d36/d5d/fde zero size 2022-01-31T19:44:16.221 INFO:tasks.workunit.client.0.smithi167.stdout:8/764: truncate d6/db/d28/f76 1326257 0 2022-01-31T19:44:16.222 INFO:tasks.workunit.client.0.smithi167.stdout:0/477: getdents d17 0 2022-01-31T19:44:16.222 INFO:tasks.workunit.client.0.smithi167.stdout:8/765: rmdir d6/d1e/d2d/d6b 39 2022-01-31T19:44:16.222 INFO:tasks.workunit.client.0.smithi167.stdout:8/766: fdatasync d6/db/d28/f43 0 2022-01-31T19:44:16.223 INFO:tasks.workunit.client.0.smithi167.stdout:0/478: unlink d17/d2c/l82 0 2022-01-31T19:44:16.224 INFO:tasks.workunit.client.0.smithi167.stdout:0/479: creat d17/d2c/f99 x:0 0 0 2022-01-31T19:44:16.224 INFO:tasks.workunit.client.0.smithi167.stdout:0/480: fdatasync d17/d3e/f61 0 2022-01-31T19:44:16.224 INFO:tasks.workunit.client.0.smithi167.stdout:8/767: write d6/f32 [650121,72565] 0 2022-01-31T19:44:16.226 INFO:tasks.workunit.client.0.smithi167.stdout:0/481: mknod d17/d19/c9a 0 2022-01-31T19:44:16.226 INFO:tasks.workunit.client.0.smithi167.stdout:0/482: truncate d17/d69/f78 379806 0 2022-01-31T19:44:16.226 INFO:tasks.workunit.client.0.smithi167.stdout:0/483: dread - d17/d19/f7f zero size 2022-01-31T19:44:16.226 INFO:tasks.workunit.client.0.smithi167.stdout:0/484: readlink d17/l25 0 2022-01-31T19:44:16.227 INFO:tasks.workunit.client.0.smithi167.stdout:0/485: dread - d17/f5e zero size 2022-01-31T19:44:16.227 INFO:tasks.workunit.client.0.smithi167.stdout:0/486: read d17/d19/d1c/d39/d44/d49/f6e [553723,36487] 0 2022-01-31T19:44:16.227 INFO:tasks.workunit.client.0.smithi167.stdout:0/487: creat d17/d3e/f9b x:0 0 0 2022-01-31T19:44:16.227 INFO:tasks.workunit.client.0.smithi167.stdout:0/488: chown fc 21074 1 2022-01-31T19:44:16.228 INFO:tasks.workunit.client.0.smithi167.stdout:0/489: truncate d17/d19/d1c/d39/d44/d49/f80 106143 0 2022-01-31T19:44:16.228 INFO:tasks.workunit.client.0.smithi167.stdout:0/490: truncate d17/d19/d1c/d39/d44/d88/f8a 972179 0 2022-01-31T19:44:16.230 INFO:tasks.workunit.client.0.smithi167.stdout:4/698: dread d0/d6/d7/d26/d40/d4c/f9c [4194304,4194304] 0 2022-01-31T19:44:16.230 INFO:tasks.workunit.client.0.smithi167.stdout:9/659: dwrite dd/d35/f44 [0,4194304] 0 2022-01-31T19:44:16.231 INFO:tasks.workunit.client.0.smithi167.stdout:3/795: dwrite d9/d29/dbd/fd8 [0,4194304] 0 2022-01-31T19:44:16.231 INFO:tasks.workunit.client.0.smithi167.stdout:1/816: dwrite d0/d10/f19 [0,4194304] 0 2022-01-31T19:44:16.233 INFO:tasks.workunit.client.0.smithi167.stdout:8/768: dread d6/db/d28/f43 [0,4194304] 0 2022-01-31T19:44:16.239 INFO:tasks.workunit.client.0.smithi167.stdout:9/660: link dd/d34/f61 dd/d17/d7f/fe1 0 2022-01-31T19:44:16.241 INFO:tasks.workunit.client.0.smithi167.stdout:9/661: dread dd/d19/f1b [0,4194304] 0 2022-01-31T19:44:16.242 INFO:tasks.workunit.client.0.smithi167.stdout:9/662: chown dd/d19/d24/d56/d5b/f94 46031505 1 2022-01-31T19:44:16.246 INFO:tasks.workunit.client.0.smithi167.stdout:4/699: link d0/d6/d7/d39/d3a/f54 d0/d6/d7/d26/d40/fe4 0 2022-01-31T19:44:16.246 INFO:tasks.workunit.client.0.smithi167.stdout:3/796: mknod d9/d29/d3b/d9b/da3/db0/cf1 0 2022-01-31T19:44:16.253 INFO:tasks.workunit.client.0.smithi167.stdout:0/491: dwrite d17/f5e [0,4194304] 0 2022-01-31T19:44:16.260 INFO:tasks.workunit.client.0.smithi167.stdout:1/817: rmdir d0/d75/dec 39 2022-01-31T19:44:16.260 INFO:tasks.workunit.client.0.smithi167.stdout:1/818: stat d0/d10/d5a/d33/d4b/l6b 0 2022-01-31T19:44:16.267 INFO:tasks.workunit.client.0.smithi167.stdout:1/819: creat d0/d10/d5a/d33/d4b/f10f x:0 0 0 2022-01-31T19:44:16.267 INFO:tasks.workunit.client.0.smithi167.stdout:1/820: truncate d0/d10/d20/fb6 701802 0 2022-01-31T19:44:16.267 INFO:tasks.workunit.client.0.smithi167.stdout:1/821: creat d0/d10/d5a/d33/d4b/d1f/f110 x:0 0 0 2022-01-31T19:44:16.268 INFO:tasks.workunit.client.0.smithi167.stdout:1/822: creat d0/d10/d20/d7c/f111 x:0 0 0 2022-01-31T19:44:16.268 INFO:tasks.workunit.client.0.smithi167.stdout:0/492: dread d17/d3e/f61 [0,4194304] 0 2022-01-31T19:44:16.269 INFO:tasks.workunit.client.0.smithi167.stdout:8/769: getdents d6/d1e 0 2022-01-31T19:44:16.269 INFO:tasks.workunit.client.0.smithi167.stdout:8/770: creat d6/db/de/d18/da4/dc2/dc3/ff9 x:0 0 0 2022-01-31T19:44:16.273 INFO:tasks.workunit.client.0.smithi167.stdout:1/823: mkdir d0/d10/d5a/d33/d4b/da5/da9/dad/d112 0 2022-01-31T19:44:16.274 INFO:tasks.workunit.client.0.smithi167.stdout:0/493: unlink d17/d19/f7f 0 2022-01-31T19:44:16.275 INFO:tasks.workunit.client.0.smithi167.stdout:2/573: sync 2022-01-31T19:44:16.275 INFO:tasks.workunit.client.0.smithi167.stdout:7/903: sync 2022-01-31T19:44:16.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/574: fdatasync dd/df/d1b/d3d/d4a/d4b/d54/f8b 0 2022-01-31T19:44:16.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/575: write dd/df/f5d [3347824,24829] 0 2022-01-31T19:44:16.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/576: fsync dd/d18/fa9 0 2022-01-31T19:44:16.277 INFO:tasks.workunit.client.0.smithi167.stdout:2/577: read dd/d18/f31 [493169,125357] 0 2022-01-31T19:44:16.279 INFO:tasks.workunit.client.0.smithi167.stdout:4/700: rename d0/d6/d7/d39/d3a/d79 to d0/d6/d7/d26/dd2/de5 0 2022-01-31T19:44:16.279 INFO:tasks.workunit.client.0.smithi167.stdout:2/578: dread dd/d18/d62/f6e [0,4194304] 0 2022-01-31T19:44:16.280 INFO:tasks.workunit.client.0.smithi167.stdout:2/579: write dd/df/d22/f8a [221451,114598] 0 2022-01-31T19:44:16.280 INFO:tasks.workunit.client.0.smithi167.stdout:0/494: mknod d17/d19/d1c/d39/d7b/c9c 0 2022-01-31T19:44:16.280 INFO:tasks.workunit.client.0.smithi167.stdout:7/904: link d8/d2c/d6e/d4d/d67/d8f/f12b d8/d35/d119/f144 0 2022-01-31T19:44:16.281 INFO:tasks.workunit.client.0.smithi167.stdout:4/701: mkdir d0/d6/d7/d26/dd2/de5/de6 0 2022-01-31T19:44:16.282 INFO:tasks.workunit.client.0.smithi167.stdout:7/905: symlink d8/d2c/d6e/d9a/l145 0 2022-01-31T19:44:16.283 INFO:tasks.workunit.client.0.smithi167.stdout:7/906: chown d8/d2c/d7a/d99/de2/dc3/de9 1 1 2022-01-31T19:44:16.283 INFO:tasks.workunit.client.0.smithi167.stdout:0/495: write d17/f5e [1007446,116804] 0 2022-01-31T19:44:16.284 INFO:tasks.workunit.client.0.smithi167.stdout:4/702: mkdir d0/d6/d7/d39/d4e/de7 0 2022-01-31T19:44:16.284 INFO:tasks.workunit.client.0.smithi167.stdout:4/703: dread - d0/d6/d7/d26/d31/ddf/da2/dcf/f94 zero size 2022-01-31T19:44:16.284 INFO:tasks.workunit.client.0.smithi167.stdout:2/580: getdents dd 0 2022-01-31T19:44:16.285 INFO:tasks.workunit.client.0.smithi167.stdout:7/907: creat d8/d2c/d6e/f146 x:0 0 0 2022-01-31T19:44:16.286 INFO:tasks.workunit.client.0.smithi167.stdout:7/908: creat d8/d2c/d7a/d99/de2/dc3/f147 x:0 0 0 2022-01-31T19:44:16.286 INFO:tasks.workunit.client.0.smithi167.stdout:0/496: mknod d17/d19/d1c/c9d 0 2022-01-31T19:44:16.287 INFO:tasks.workunit.client.0.smithi167.stdout:0/497: creat d17/d69/f9e x:0 0 0 2022-01-31T19:44:16.287 INFO:tasks.workunit.client.0.smithi167.stdout:0/498: chown d17/d19/f4d 11 1 2022-01-31T19:44:16.287 INFO:tasks.workunit.client.0.smithi167.stdout:4/704: getdents d0/d6/d7/d26/d31/ddf/da2/dcf/db5 0 2022-01-31T19:44:16.288 INFO:tasks.workunit.client.0.smithi167.stdout:4/705: chown d0/d6/d7/d26/d31/l25 44532 1 2022-01-31T19:44:16.290 INFO:tasks.workunit.client.0.smithi167.stdout:7/909: mknod d8/dc/d18/d24/dbb/c148 0 2022-01-31T19:44:16.292 INFO:tasks.workunit.client.0.smithi167.stdout:1/824: dwrite d0/d10/d20/f37 [0,4194304] 0 2022-01-31T19:44:16.298 INFO:tasks.workunit.client.0.smithi167.stdout:7/910: unlink d8/d2c/d6e/d4d/c53 0 2022-01-31T19:44:16.299 INFO:tasks.workunit.client.0.smithi167.stdout:1/825: getdents d0/d10/d5a/d33/d97 0 2022-01-31T19:44:16.300 INFO:tasks.workunit.client.0.smithi167.stdout:1/826: fsync d0/d10/d20/d7c/ff4 0 2022-01-31T19:44:16.300 INFO:tasks.workunit.client.0.smithi167.stdout:0/499: rmdir d17/d19/d1c/d39/d7b 39 2022-01-31T19:44:16.302 INFO:tasks.workunit.client.0.smithi167.stdout:7/911: mknod d8/dc/dc1/c149 0 2022-01-31T19:44:16.303 INFO:tasks.workunit.client.0.smithi167.stdout:0/500: mknod d17/d3e/d7e/c9f 0 2022-01-31T19:44:16.304 INFO:tasks.workunit.client.0.smithi167.stdout:0/501: creat d17/d69/fa0 x:0 0 0 2022-01-31T19:44:16.304 INFO:tasks.workunit.client.0.smithi167.stdout:0/502: chown d17/d33/f95 34 1 2022-01-31T19:44:16.308 INFO:tasks.workunit.client.0.smithi167.stdout:7/912: dread d8/d2c/d6e/d9a/f110 [0,4194304] 0 2022-01-31T19:44:16.309 INFO:tasks.workunit.client.0.smithi167.stdout:7/913: truncate d8/d2c/d43/f2f 1811650 0 2022-01-31T19:44:16.309 INFO:tasks.workunit.client.0.smithi167.stdout:7/914: dread - d8/d2c/d7a/dd3/f134 zero size 2022-01-31T19:44:16.311 INFO:tasks.workunit.client.0.smithi167.stdout:7/915: rename d8/d2c/d43/d94/daf to d8/dc/d18/d14a 0 2022-01-31T19:44:16.313 INFO:tasks.workunit.client.0.smithi167.stdout:7/916: creat d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d83/f14b x:0 0 0 2022-01-31T19:44:16.314 INFO:tasks.workunit.client.0.smithi167.stdout:2/581: dwrite dd/d18/fa9 [0,4194304] 0 2022-01-31T19:44:16.315 INFO:tasks.workunit.client.0.smithi167.stdout:4/706: dwrite d0/d6/d7/d26/d31/ddf/d1a/fca [0,4194304] 0 2022-01-31T19:44:16.316 INFO:tasks.workunit.client.0.smithi167.stdout:2/582: symlink dd/df/d3f/d51/d5f/lb8 0 2022-01-31T19:44:16.318 INFO:tasks.workunit.client.0.smithi167.stdout:2/583: creat dd/df/d1b/d2f/d36/fb9 x:0 0 0 2022-01-31T19:44:16.320 INFO:tasks.workunit.client.0.smithi167.stdout:2/584: link dd/df/d1b/d3d/d4a/c8c dd/d18/d8d/cba 0 2022-01-31T19:44:16.329 INFO:tasks.workunit.client.0.smithi167.stdout:1/827: dwrite d0/d10/d5a/d33/d4b/d1f/fdc [0,4194304] 0 2022-01-31T19:44:16.331 INFO:tasks.workunit.client.0.smithi167.stdout:0/503: dwrite d17/d33/f95 [0,4194304] 0 2022-01-31T19:44:16.344 INFO:tasks.workunit.client.0.smithi167.stdout:7/917: dread d8/d2c/d6e/d4d/d67/fb9 [0,4194304] 0 2022-01-31T19:44:16.345 INFO:tasks.workunit.client.0.smithi167.stdout:4/707: dwrite d0/d6/d7/d26/d40/f64 [0,4194304] 0 2022-01-31T19:44:16.347 INFO:tasks.workunit.client.0.smithi167.stdout:2/585: creat dd/df/d1b/d3d/d4a/da7/fbb x:0 0 0 2022-01-31T19:44:16.348 INFO:tasks.workunit.client.0.smithi167.stdout:2/586: chown dd/df/d1b/d3d/d4a/da7 41526 1 2022-01-31T19:44:16.363 INFO:tasks.workunit.client.0.smithi167.stdout:1/828: dwrite d0/fe4 [0,4194304] 0 2022-01-31T19:44:16.364 INFO:tasks.workunit.client.0.smithi167.stdout:1/829: write d0/d75/d7a/f59 [2125596,70483] 0 2022-01-31T19:44:16.366 INFO:tasks.workunit.client.0.smithi167.stdout:7/918: dwrite d8/d35/f36 [0,4194304] 0 2022-01-31T19:44:16.370 INFO:tasks.workunit.client.0.smithi167.stdout:2/587: dwrite dd/df/d1b/d3d/d4a/d4b/f93 [0,4194304] 0 2022-01-31T19:44:16.370 INFO:tasks.workunit.client.0.smithi167.stdout:2/588: readlink dd/df/l78 0 2022-01-31T19:44:16.393 INFO:tasks.workunit.client.0.smithi167.stdout:0/504: mkdir d17/d3e/da1 0 2022-01-31T19:44:16.394 INFO:tasks.workunit.client.0.smithi167.stdout:1/830: dwrite d0/f1b [0,4194304] 0 2022-01-31T19:44:16.396 INFO:tasks.workunit.client.0.smithi167.stdout:2/589: dwrite dd/d18/d63/fa3 [0,4194304] 0 2022-01-31T19:44:16.416 INFO:tasks.workunit.client.0.smithi167.stdout:2/590: dwrite dd/df/d1b/f56 [0,4194304] 0 2022-01-31T19:44:16.443 INFO:tasks.workunit.client.0.smithi167.stdout:4/708: symlink d0/d6/d7/d26/d31/ddf/d1a/le8 0 2022-01-31T19:44:16.446 INFO:tasks.workunit.client.0.smithi167.stdout:4/709: dread d0/d6/d7/d26/fc9 [0,4194304] 0 2022-01-31T19:44:16.447 INFO:tasks.workunit.client.0.smithi167.stdout:7/919: creat d8/dfb/f14c x:0 0 0 2022-01-31T19:44:16.454 INFO:tasks.workunit.client.0.smithi167.stdout:0/505: link d17/d19/d1c/d39/d44/l54 d17/d3e/d7e/la2 0 2022-01-31T19:44:16.455 INFO:tasks.workunit.client.0.smithi167.stdout:0/506: write d17/d2c/f79 [3025234,55543] 0 2022-01-31T19:44:16.455 INFO:tasks.workunit.client.0.smithi167.stdout:0/507: dread - d17/f31 zero size 2022-01-31T19:44:16.456 INFO:tasks.workunit.client.0.smithi167.stdout:1/831: rename d0/cb to d0/d62/c113 0 2022-01-31T19:44:16.463 INFO:tasks.workunit.client.0.smithi167.stdout:4/710: mknod d0/d6/d7/d26/d31/ddf/d75/dd6/ce9 0 2022-01-31T19:44:16.463 INFO:tasks.workunit.client.0.smithi167.stdout:4/711: fsync d0/d6/d7/d26/d40/f24 0 2022-01-31T19:44:16.463 INFO:tasks.workunit.client.0.smithi167.stdout:4/712: stat d0/d6/d7/d26/f72 0 2022-01-31T19:44:16.463 INFO:tasks.workunit.client.0.smithi167.stdout:4/713: readlink d0/d6/d7/d26/d50/l5a 0 2022-01-31T19:44:16.464 INFO:tasks.workunit.client.0.smithi167.stdout:4/714: creat d0/d6/d7/d26/d31/d5c/fea x:0 0 0 2022-01-31T19:44:16.464 INFO:tasks.workunit.client.0.smithi167.stdout:7/920: symlink d8/d2c/d6e/d4d/d5e/d71/db1/d126/l14d 0 2022-01-31T19:44:16.465 INFO:tasks.workunit.client.0.smithi167.stdout:7/921: write d8/dab/fe4 [352899,9739] 0 2022-01-31T19:44:16.465 INFO:tasks.workunit.client.0.smithi167.stdout:0/508: unlink d17/d3e/c8b 0 2022-01-31T19:44:16.465 INFO:tasks.workunit.client.0.smithi167.stdout:0/509: creat d17/d33/fa3 x:0 0 0 2022-01-31T19:44:16.470 INFO:tasks.workunit.client.0.smithi167.stdout:1/832: mknod d0/d10/d5a/d33/d4b/da5/da9/dad/d112/c114 0 2022-01-31T19:44:16.475 INFO:tasks.workunit.client.0.smithi167.stdout:0/510: mkdir d17/d33/da4 0 2022-01-31T19:44:16.475 INFO:tasks.workunit.client.0.smithi167.stdout:0/511: write d17/f31 [1011903,55857] 0 2022-01-31T19:44:16.487 INFO:tasks.workunit.client.0.smithi167.stdout:4/715: dwrite d0/d6/d7/f7b [0,4194304] 0 2022-01-31T19:44:16.489 INFO:tasks.workunit.client.0.smithi167.stdout:4/716: rename d0/d6/d7/d39/l83 to d0/d6/d7/d39/d4e/d61/leb 0 2022-01-31T19:44:16.490 INFO:tasks.workunit.client.0.smithi167.stdout:4/717: stat d0/d6/d7/d26/d40/f32 0 2022-01-31T19:44:16.490 INFO:tasks.workunit.client.0.smithi167.stdout:4/718: fsync d0/d6/d7/d26/d31/f11 0 2022-01-31T19:44:16.490 INFO:tasks.workunit.client.0.smithi167.stdout:4/719: creat d0/d6/d7/d39/fec x:0 0 0 2022-01-31T19:44:16.492 INFO:tasks.workunit.client.0.smithi167.stdout:4/720: creat d0/d6/d7/d26/d31/ddf/da2/dcf/db5/fed x:0 0 0 2022-01-31T19:44:16.493 INFO:tasks.workunit.client.0.smithi167.stdout:4/721: creat d0/d6/d7/d39/d4e/de7/fee x:0 0 0 2022-01-31T19:44:16.494 INFO:tasks.workunit.client.0.smithi167.stdout:4/722: stat d0/d6/d7/d26/d31/ddf/da2/dcf/f94 0 2022-01-31T19:44:16.494 INFO:tasks.workunit.client.0.smithi167.stdout:4/723: fdatasync d0/d6/d7/d89/fa3 0 2022-01-31T19:44:16.494 INFO:tasks.workunit.client.0.smithi167.stdout:7/922: dwrite d8/d2c/d7a/f9c [0,4194304] 0 2022-01-31T19:44:16.495 INFO:tasks.workunit.client.0.smithi167.stdout:4/724: rmdir d0/d6/d7/d39/d3a/dc8 0 2022-01-31T19:44:16.495 INFO:tasks.workunit.client.0.smithi167.stdout:4/725: chown d0/d6/d7/d26/d50/cbd 5 1 2022-01-31T19:44:16.496 INFO:tasks.workunit.client.0.smithi167.stdout:4/726: chown d0/d6/d7/d26/d40/ld5 62968816 1 2022-01-31T19:44:16.497 INFO:tasks.workunit.client.0.smithi167.stdout:0/512: dwrite d17/d2c/d87/f8d [4194304,4194304] 0 2022-01-31T19:44:16.501 INFO:tasks.workunit.client.0.smithi167.stdout:0/513: creat d17/d19/d1c/d39/d44/d49/fa5 x:0 0 0 2022-01-31T19:44:16.504 INFO:tasks.workunit.client.0.smithi167.stdout:4/727: getdents d0/d6/dab 0 2022-01-31T19:44:16.505 INFO:tasks.workunit.client.0.smithi167.stdout:4/728: read d0/d6/d7/d26/d31/ddf/da2/fc4 [884075,100559] 0 2022-01-31T19:44:16.508 INFO:tasks.workunit.client.0.smithi167.stdout:4/729: link d0/d6/d7/d26/d31/ddf/f3d d0/d6/d7/d39/d3a/fef 0 2022-01-31T19:44:16.509 INFO:tasks.workunit.client.0.smithi167.stdout:0/514: dread d17/f3d [0,4194304] 0 2022-01-31T19:44:16.511 INFO:tasks.workunit.client.0.smithi167.stdout:4/730: mkdir d0/d6/d7/d39/df0 0 2022-01-31T19:44:16.511 INFO:tasks.workunit.client.0.smithi167.stdout:4/731: chown d0/d6/d7/d26/d31/l25 0 1 2022-01-31T19:44:16.512 INFO:tasks.workunit.client.0.smithi167.stdout:0/515: symlink d17/d19/la6 0 2022-01-31T19:44:16.512 INFO:tasks.workunit.client.0.smithi167.stdout:0/516: readlink d17/d19/l2b 0 2022-01-31T19:44:16.514 INFO:tasks.workunit.client.0.smithi167.stdout:0/517: dread d17/d2c/f79 [4194304,4194304] 0 2022-01-31T19:44:16.515 INFO:tasks.workunit.client.1.smithi171.stdout:8/18: sync 2022-01-31T19:44:16.515 INFO:tasks.workunit.client.1.smithi171.stdout:5/35: sync 2022-01-31T19:44:16.515 INFO:tasks.workunit.client.1.smithi171.stdout:9/4: sync 2022-01-31T19:44:16.515 INFO:tasks.workunit.client.1.smithi171.stdout:7/23: sync 2022-01-31T19:44:16.516 INFO:tasks.workunit.client.1.smithi171.stdout:0/59: sync 2022-01-31T19:44:16.516 INFO:tasks.workunit.client.1.smithi171.stdout:2/51: sync 2022-01-31T19:44:16.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/5: sync 2022-01-31T19:44:16.516 INFO:tasks.workunit.client.1.smithi171.stdout:0/60: chown d9/f16 0 1 2022-01-31T19:44:16.516 INFO:tasks.workunit.client.1.smithi171.stdout:1/9: sync 2022-01-31T19:44:16.517 INFO:tasks.workunit.client.1.smithi171.stdout:6/0: sync 2022-01-31T19:44:16.517 INFO:tasks.workunit.client.1.smithi171.stdout:1/10: chown . 85598 1 2022-01-31T19:44:16.517 INFO:tasks.workunit.client.1.smithi171.stdout:1/11: chown . 440500 1 2022-01-31T19:44:16.517 INFO:tasks.workunit.client.1.smithi171.stdout:6/1: chown . 1694435927 1 2022-01-31T19:44:16.518 INFO:tasks.workunit.client.1.smithi171.stdout:6/2: fdatasync - no filename 2022-01-31T19:44:16.518 INFO:tasks.workunit.client.1.smithi171.stdout:6/3: link - no file 2022-01-31T19:44:16.518 INFO:tasks.workunit.client.1.smithi171.stdout:6/4: chown . 41505215 1 2022-01-31T19:44:16.518 INFO:tasks.workunit.client.1.smithi171.stdout:6/5: chown . 1 1 2022-01-31T19:44:16.519 INFO:tasks.workunit.client.1.smithi171.stdout:6/6: link - no file 2022-01-31T19:44:16.519 INFO:tasks.workunit.client.1.smithi171.stdout:6/7: stat - no entries 2022-01-31T19:44:16.519 INFO:tasks.workunit.client.1.smithi171.stdout:6/8: fsync - no filename 2022-01-31T19:44:16.519 INFO:tasks.workunit.client.1.smithi171.stdout:6/9: truncate - no filename 2022-01-31T19:44:16.520 INFO:tasks.workunit.client.1.smithi171.stdout:6/10: fsync - no filename 2022-01-31T19:44:16.520 INFO:tasks.workunit.client.1.smithi171.stdout:3/57: sync 2022-01-31T19:44:16.520 INFO:tasks.workunit.client.1.smithi171.stdout:3/58: chown d4/c8 54810534 1 2022-01-31T19:44:16.521 INFO:tasks.workunit.client.0.smithi167.stdout:0/518: truncate d17/d3e/f42 2966102 0 2022-01-31T19:44:16.521 INFO:tasks.workunit.client.0.smithi167.stdout:0/519: mknod d17/d19/d1c/d39/d44/d88/d98/ca7 0 2022-01-31T19:44:16.522 INFO:tasks.workunit.client.0.smithi167.stdout:0/520: mknod d17/d19/d1c/d39/d44/ca8 0 2022-01-31T19:44:16.522 INFO:tasks.workunit.client.0.smithi167.stdout:0/521: mkdir d17/d19/d1c/d39/d7b/da9 0 2022-01-31T19:44:16.522 INFO:tasks.workunit.client.0.smithi167.stdout:0/522: truncate d17/d69/f94 808688 0 2022-01-31T19:44:16.523 INFO:tasks.workunit.client.1.smithi171.stdout:7/24: link d1/d2/l4 d1/d2/d7/la 0 2022-01-31T19:44:16.523 INFO:tasks.workunit.client.1.smithi171.stdout:9/5: link l1 l2 0 2022-01-31T19:44:16.523 INFO:tasks.workunit.client.1.smithi171.stdout:7/25: readlink d1/l6 0 2022-01-31T19:44:16.524 INFO:tasks.workunit.client.1.smithi171.stdout:7/26: write d1/f5 [4291298,67638] 0 2022-01-31T19:44:16.524 INFO:tasks.workunit.client.1.smithi171.stdout:7/27: truncate d1/f5 4771812 0 2022-01-31T19:44:16.524 INFO:tasks.workunit.client.1.smithi171.stdout:0/61: rename l5 to d9/l18 0 2022-01-31T19:44:16.525 INFO:tasks.workunit.client.1.smithi171.stdout:4/6: dread f0 [0,4194304] 0 2022-01-31T19:44:16.525 INFO:tasks.workunit.client.1.smithi171.stdout:4/7: creat f1 x:0 0 0 2022-01-31T19:44:16.525 INFO:tasks.workunit.client.1.smithi171.stdout:2/52: truncate f3 21984 0 2022-01-31T19:44:16.526 INFO:tasks.workunit.client.1.smithi171.stdout:6/11: mknod c0 0 2022-01-31T19:44:16.526 INFO:tasks.workunit.client.1.smithi171.stdout:6/12: stat c0 0 2022-01-31T19:44:16.527 INFO:tasks.workunit.client.1.smithi171.stdout:6/13: dread - no filename 2022-01-31T19:44:16.527 INFO:tasks.workunit.client.1.smithi171.stdout:6/14: write - no filename 2022-01-31T19:44:16.527 INFO:tasks.workunit.client.1.smithi171.stdout:1/12: mknod c0 0 2022-01-31T19:44:16.527 INFO:tasks.workunit.client.1.smithi171.stdout:1/13: dread - no filename 2022-01-31T19:44:16.527 INFO:tasks.workunit.client.1.smithi171.stdout:3/59: truncate d4/fa 1463508 0 2022-01-31T19:44:16.528 INFO:tasks.workunit.client.1.smithi171.stdout:3/60: readlink d4/dd/l11 0 2022-01-31T19:44:16.528 INFO:tasks.workunit.client.1.smithi171.stdout:5/36: dread f1 [0,4194304] 0 2022-01-31T19:44:16.528 INFO:tasks.workunit.client.1.smithi171.stdout:5/37: dread - d5/f8 zero size 2022-01-31T19:44:16.528 INFO:tasks.workunit.client.1.smithi171.stdout:5/38: rename d5 to d5/d6/d9 22 2022-01-31T19:44:16.529 INFO:tasks.workunit.client.1.smithi171.stdout:9/6: link l2 l3 0 2022-01-31T19:44:16.529 INFO:tasks.workunit.client.1.smithi171.stdout:0/62: symlink d9/l19 0 2022-01-31T19:44:16.530 INFO:tasks.workunit.client.1.smithi171.stdout:0/63: write f8 [566629,63793] 0 2022-01-31T19:44:16.532 INFO:tasks.workunit.client.1.smithi171.stdout:4/8: mkdir d2 0 2022-01-31T19:44:16.532 INFO:tasks.workunit.client.1.smithi171.stdout:5/39: dread f1 [0,4194304] 0 2022-01-31T19:44:16.532 INFO:tasks.workunit.client.1.smithi171.stdout:2/53: creat d7/d9/f12 x:0 0 0 2022-01-31T19:44:16.532 INFO:tasks.workunit.client.1.smithi171.stdout:2/54: write d7/d9/ff [725094,66149] 0 2022-01-31T19:44:16.533 INFO:tasks.workunit.client.1.smithi171.stdout:6/15: mknod c1 0 2022-01-31T19:44:16.534 INFO:tasks.workunit.client.0.smithi167.stdout:4/732: dwrite d0/d6/d7/d39/d3a/f54 [0,4194304] 0 2022-01-31T19:44:16.534 INFO:tasks.workunit.client.0.smithi167.stdout:4/733: fsync d0/d6/d7/d26/d31/f19 0 2022-01-31T19:44:16.534 INFO:tasks.workunit.client.0.smithi167.stdout:4/734: creat d0/d6/d7/d89/ff1 x:0 0 0 2022-01-31T19:44:16.535 INFO:tasks.workunit.client.0.smithi167.stdout:4/735: fdatasync d0/d6/d7/d26/d31/ddf/f2e 0 2022-01-31T19:44:16.537 INFO:tasks.workunit.client.0.smithi167.stdout:4/736: read d0/d6/d7/d26/d31/ddf/da2/dcf/d65/f90 [197076,124196] 0 2022-01-31T19:44:16.539 INFO:tasks.workunit.client.0.smithi167.stdout:0/523: dread d17/f5e [0,4194304] 0 2022-01-31T19:44:16.539 INFO:tasks.workunit.client.0.smithi167.stdout:0/524: fsync d17/d2c/d87/f97 0 2022-01-31T19:44:16.539 INFO:tasks.workunit.client.0.smithi167.stdout:0/525: chown d17/d3e/c67 49281255 1 2022-01-31T19:44:16.540 INFO:tasks.workunit.client.0.smithi167.stdout:0/526: chown f10 505 1 2022-01-31T19:44:16.540 INFO:tasks.workunit.client.0.smithi167.stdout:4/737: rmdir d0/d6/d7/d26/d40 39 2022-01-31T19:44:16.541 INFO:tasks.workunit.client.1.smithi171.stdout:9/7: creat f4 x:0 0 0 2022-01-31T19:44:16.542 INFO:tasks.workunit.client.0.smithi167.stdout:4/738: symlink d0/d6/d7/d39/d4e/d61/lf2 0 2022-01-31T19:44:16.544 INFO:tasks.workunit.client.1.smithi171.stdout:0/64: rename d9/l19 to d9/db/dc/l1a 0 2022-01-31T19:44:16.546 INFO:tasks.workunit.client.1.smithi171.stdout:4/9: mkdir d2/d3 0 2022-01-31T19:44:16.546 INFO:tasks.workunit.client.1.smithi171.stdout:4/10: read f0 [976425,17145] 0 2022-01-31T19:44:16.547 INFO:tasks.workunit.client.1.smithi171.stdout:4/11: chown f1 108410 1 2022-01-31T19:44:16.547 INFO:tasks.workunit.client.1.smithi171.stdout:4/12: write f0 [743188,44449] 0 2022-01-31T19:44:16.548 INFO:tasks.workunit.client.0.smithi167.stdout:4/739: creat d0/d6/dab/ff3 x:0 0 0 2022-01-31T19:44:16.549 INFO:tasks.workunit.client.1.smithi171.stdout:5/40: link d5/d6/l7 d5/d6/la 0 2022-01-31T19:44:16.549 INFO:tasks.workunit.client.0.smithi167.stdout:4/740: creat d0/d6/d7/d26/d40/d4c/db4/ff4 x:0 0 0 2022-01-31T19:44:16.551 INFO:tasks.workunit.client.1.smithi171.stdout:6/16: creat f2 x:0 0 0 2022-01-31T19:44:16.552 INFO:tasks.workunit.client.1.smithi171.stdout:6/17: fdatasync f2 0 2022-01-31T19:44:16.552 INFO:tasks.workunit.client.1.smithi171.stdout:6/18: write f2 [882733,115777] 0 2022-01-31T19:44:16.552 INFO:tasks.workunit.client.0.smithi167.stdout:4/741: creat d0/d6/d7/d26/ff5 x:0 0 0 2022-01-31T19:44:16.553 INFO:tasks.workunit.client.0.smithi167.stdout:4/742: getdents d0/d6/d7/d26/d31/ddf/da2/dcf/d65 0 2022-01-31T19:44:16.554 INFO:tasks.workunit.client.0.smithi167.stdout:4/743: mkdir d0/d51/df6 0 2022-01-31T19:44:16.555 INFO:tasks.workunit.client.1.smithi171.stdout:3/61: write d4/fa [1123323,35545] 0 2022-01-31T19:44:16.555 INFO:tasks.workunit.client.1.smithi171.stdout:9/8: mknod c5 0 2022-01-31T19:44:16.556 INFO:tasks.workunit.client.1.smithi171.stdout:2/55: getdents d7/d9 0 2022-01-31T19:44:16.556 INFO:tasks.workunit.client.1.smithi171.stdout:5/41: stat d5/d6/l7 0 2022-01-31T19:44:16.557 INFO:tasks.workunit.client.1.smithi171.stdout:6/19: rename c0 to c3 0 2022-01-31T19:44:16.558 INFO:tasks.workunit.client.1.smithi171.stdout:6/20: chown c3 324 1 2022-01-31T19:44:16.558 INFO:tasks.workunit.client.1.smithi171.stdout:6/21: write f2 [1401535,97003] 0 2022-01-31T19:44:16.559 INFO:tasks.workunit.client.1.smithi171.stdout:9/9: rename l1 to l6 0 2022-01-31T19:44:16.559 INFO:tasks.workunit.client.1.smithi171.stdout:9/10: chown l3 0 1 2022-01-31T19:44:16.559 INFO:tasks.workunit.client.1.smithi171.stdout:9/11: write f4 [931502,68498] 0 2022-01-31T19:44:16.561 INFO:tasks.workunit.client.0.smithi167.stdout:4/744: mkdir d0/d6/d7/d89/db1/df7 0 2022-01-31T19:44:16.561 INFO:tasks.workunit.client.0.smithi167.stdout:4/745: chown d0/d6/d7/d39/d4e/d61 27 1 2022-01-31T19:44:16.565 INFO:tasks.workunit.client.1.smithi171.stdout:2/56: dread d7/d9/fb [4194304,4194304] 0 2022-01-31T19:44:16.565 INFO:tasks.workunit.client.1.smithi171.stdout:2/57: chown d7/d9/l10 415846853 1 2022-01-31T19:44:16.566 INFO:tasks.workunit.client.1.smithi171.stdout:2/58: fsync f3 0 2022-01-31T19:44:16.568 INFO:tasks.workunit.client.0.smithi167.stdout:4/746: dread d0/d6/f58 [0,4194304] 0 2022-01-31T19:44:16.569 INFO:tasks.workunit.client.0.smithi167.stdout:4/747: chown d0/d6/d7/d26/dd2/de5/f91 862480186 1 2022-01-31T19:44:16.570 INFO:tasks.workunit.client.0.smithi167.stdout:4/748: symlink d0/d6/d7/d26/d50/lf8 0 2022-01-31T19:44:16.570 INFO:tasks.workunit.client.0.smithi167.stdout:4/749: dread - d0/d6/d7/d26/d31/ddf/d75/f98 zero size 2022-01-31T19:44:16.571 INFO:tasks.workunit.client.0.smithi167.stdout:4/750: dread - d0/d6/dab/ff3 zero size 2022-01-31T19:44:16.571 INFO:tasks.workunit.client.0.smithi167.stdout:4/751: fdatasync d0/d6/d7/d39/d3a/fef 0 2022-01-31T19:44:16.571 INFO:tasks.workunit.client.1.smithi171.stdout:7/28: truncate d1/f5 1922165 0 2022-01-31T19:44:16.574 INFO:tasks.workunit.client.1.smithi171.stdout:3/62: dread d4/fc [0,4194304] 0 2022-01-31T19:44:16.575 INFO:tasks.workunit.client.1.smithi171.stdout:3/63: unlink d4/dd/fe 0 2022-01-31T19:44:16.576 INFO:tasks.workunit.client.1.smithi171.stdout:3/64: mknod d4/c12 0 2022-01-31T19:44:16.577 INFO:tasks.workunit.client.1.smithi171.stdout:3/65: creat d4/f13 x:0 0 0 2022-01-31T19:44:16.578 INFO:tasks.workunit.client.1.smithi171.stdout:3/66: unlink d4/dd/l11 0 2022-01-31T19:44:16.578 INFO:tasks.workunit.client.0.smithi167.stdout:0/527: dwrite d17/d19/d1c/f8f [4194304,4194304] 0 2022-01-31T19:44:16.580 INFO:tasks.workunit.client.1.smithi171.stdout:8/19: dwrite f2 [0,4194304] 0 2022-01-31T19:44:16.581 INFO:tasks.workunit.client.1.smithi171.stdout:8/20: creat f6 x:0 0 0 2022-01-31T19:44:16.581 INFO:tasks.workunit.client.1.smithi171.stdout:8/21: creat f7 x:0 0 0 2022-01-31T19:44:16.581 INFO:tasks.workunit.client.1.smithi171.stdout:8/22: fdatasync f7 0 2022-01-31T19:44:16.581 INFO:tasks.workunit.client.1.smithi171.stdout:8/23: stat l4 0 2022-01-31T19:44:16.582 INFO:tasks.workunit.client.1.smithi171.stdout:8/24: creat f8 x:0 0 0 2022-01-31T19:44:16.582 INFO:tasks.workunit.client.1.smithi171.stdout:8/25: creat f9 x:0 0 0 2022-01-31T19:44:16.582 INFO:tasks.workunit.client.1.smithi171.stdout:8/26: creat fa x:0 0 0 2022-01-31T19:44:16.582 INFO:tasks.workunit.client.1.smithi171.stdout:8/27: creat fb x:0 0 0 2022-01-31T19:44:16.583 INFO:tasks.workunit.client.1.smithi171.stdout:8/28: creat fc x:0 0 0 2022-01-31T19:44:16.583 INFO:tasks.workunit.client.1.smithi171.stdout:4/13: dwrite f1 [0,4194304] 0 2022-01-31T19:44:16.583 INFO:tasks.workunit.client.1.smithi171.stdout:0/65: dwrite f6 [0,4194304] 0 2022-01-31T19:44:16.585 INFO:tasks.workunit.client.0.smithi167.stdout:0/528: creat d17/d33/da4/faa x:0 0 0 2022-01-31T19:44:16.586 INFO:tasks.workunit.client.0.smithi167.stdout:0/529: truncate d17/d19/d1c/d39/d44/f93 890210 0 2022-01-31T19:44:16.586 INFO:tasks.workunit.client.0.smithi167.stdout:0/530: creat d17/d3e/d7e/fab x:0 0 0 2022-01-31T19:44:16.586 INFO:tasks.workunit.client.0.smithi167.stdout:0/531: fsync d17/d2c/f37 0 2022-01-31T19:44:16.586 INFO:tasks.workunit.client.0.smithi167.stdout:0/532: write d17/d19/d1c/d39/d44/d49/f6e [854872,116440] 0 2022-01-31T19:44:16.587 INFO:tasks.workunit.client.0.smithi167.stdout:0/533: symlink d17/d2c/lac 0 2022-01-31T19:44:16.588 INFO:tasks.workunit.client.1.smithi171.stdout:3/67: read d4/fb [1162220,29402] 0 2022-01-31T19:44:16.588 INFO:tasks.workunit.client.1.smithi171.stdout:0/66: dread d9/db/f14 [0,4194304] 0 2022-01-31T19:44:16.588 INFO:tasks.workunit.client.1.smithi171.stdout:0/67: dread - f4 zero size 2022-01-31T19:44:16.589 INFO:tasks.workunit.client.1.smithi171.stdout:9/12: dread f4 [0,4194304] 0 2022-01-31T19:44:16.589 INFO:tasks.workunit.client.1.smithi171.stdout:5/42: write f1 [1925326,118151] 0 2022-01-31T19:44:16.589 INFO:tasks.workunit.client.1.smithi171.stdout:5/43: readlink d5/d6/l7 0 2022-01-31T19:44:16.589 INFO:tasks.workunit.client.1.smithi171.stdout:5/44: chown d5/d6/la 113 1 2022-01-31T19:44:16.590 INFO:tasks.workunit.client.1.smithi171.stdout:5/45: dread - d5/f8 zero size 2022-01-31T19:44:16.590 INFO:tasks.workunit.client.1.smithi171.stdout:2/59: write f3 [615632,108254] 0 2022-01-31T19:44:16.590 INFO:tasks.workunit.client.1.smithi171.stdout:2/60: stat d7/fa 0 2022-01-31T19:44:16.590 INFO:tasks.workunit.client.1.smithi171.stdout:2/61: write d7/d9/fb [7384354,50283] 0 2022-01-31T19:44:16.591 INFO:tasks.workunit.client.1.smithi171.stdout:2/62: write d7/d9/ff [1146636,108492] 0 2022-01-31T19:44:16.591 INFO:tasks.workunit.client.1.smithi171.stdout:7/29: write d1/f5 [2275794,38150] 0 2022-01-31T19:44:16.592 INFO:tasks.workunit.client.1.smithi171.stdout:8/29: symlink ld 0 2022-01-31T19:44:16.592 INFO:tasks.workunit.client.1.smithi171.stdout:6/22: dread f2 [0,4194304] 0 2022-01-31T19:44:16.592 INFO:tasks.workunit.client.1.smithi171.stdout:6/23: chown f2 304702 1 2022-01-31T19:44:16.594 INFO:tasks.workunit.client.1.smithi171.stdout:5/46: link c2 d5/d6/cb 0 2022-01-31T19:44:16.595 INFO:tasks.workunit.client.1.smithi171.stdout:3/68: dread d4/fa [0,4194304] 0 2022-01-31T19:44:16.596 INFO:tasks.workunit.client.1.smithi171.stdout:2/63: rename f6 to d7/f13 0 2022-01-31T19:44:16.596 INFO:tasks.workunit.client.1.smithi171.stdout:2/64: read d7/f13 [4277523,90531] 0 2022-01-31T19:44:16.599 INFO:tasks.workunit.client.1.smithi171.stdout:6/24: symlink l4 0 2022-01-31T19:44:16.601 INFO:tasks.workunit.client.1.smithi171.stdout:8/30: symlink le 0 2022-01-31T19:44:16.601 INFO:tasks.workunit.client.1.smithi171.stdout:3/69: mknod d4/dd/c14 0 2022-01-31T19:44:16.602 INFO:tasks.workunit.client.1.smithi171.stdout:2/65: link d7/d9/f12 d7/f14 0 2022-01-31T19:44:16.603 INFO:tasks.workunit.client.1.smithi171.stdout:6/25: dread f2 [0,4194304] 0 2022-01-31T19:44:16.604 INFO:tasks.workunit.client.1.smithi171.stdout:6/26: chown c3 4 1 2022-01-31T19:44:16.604 INFO:tasks.workunit.client.1.smithi171.stdout:6/27: chown c1 1 1 2022-01-31T19:44:16.604 INFO:tasks.workunit.client.1.smithi171.stdout:5/47: fsync f1 0 2022-01-31T19:44:16.622 INFO:tasks.workunit.client.1.smithi171.stdout:7/30: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:44:16.623 INFO:tasks.workunit.client.1.smithi171.stdout:7/31: readlink d1/l6 0 2022-01-31T19:44:16.623 INFO:tasks.workunit.client.1.smithi171.stdout:0/68: dwrite f7 [0,4194304] 0 2022-01-31T19:44:16.625 INFO:tasks.workunit.client.1.smithi171.stdout:2/66: dread d7/f13 [4194304,4194304] 0 2022-01-31T19:44:16.625 INFO:tasks.workunit.client.1.smithi171.stdout:7/32: mknod d1/d2/d7/cb 0 2022-01-31T19:44:16.626 INFO:tasks.workunit.client.1.smithi171.stdout:4/14: dwrite f0 [0,4194304] 0 2022-01-31T19:44:16.629 INFO:tasks.workunit.client.1.smithi171.stdout:4/15: read f1 [279928,11161] 0 2022-01-31T19:44:16.631 INFO:tasks.workunit.client.1.smithi171.stdout:0/69: getdents d9/db 0 2022-01-31T19:44:16.631 INFO:tasks.workunit.client.1.smithi171.stdout:7/33: rmdir d1 39 2022-01-31T19:44:16.632 INFO:tasks.workunit.client.1.smithi171.stdout:4/16: symlink d2/l4 0 2022-01-31T19:44:16.635 INFO:tasks.workunit.client.1.smithi171.stdout:2/67: dread d7/d9/fb [0,4194304] 0 2022-01-31T19:44:16.637 INFO:tasks.workunit.client.1.smithi171.stdout:7/34: creat d1/d2/fc x:0 0 0 2022-01-31T19:44:16.637 INFO:tasks.workunit.client.1.smithi171.stdout:4/17: mkdir d2/d3/d5 0 2022-01-31T19:44:16.639 INFO:tasks.workunit.client.1.smithi171.stdout:4/18: creat d2/d3/d5/f6 x:0 0 0 2022-01-31T19:44:16.640 INFO:tasks.workunit.client.1.smithi171.stdout:4/19: chown f1 23355379 1 2022-01-31T19:44:16.642 INFO:tasks.workunit.client.1.smithi171.stdout:4/20: creat d2/d3/f7 x:0 0 0 2022-01-31T19:44:16.645 INFO:tasks.workunit.client.1.smithi171.stdout:5/48: truncate f1 2285790 0 2022-01-31T19:44:16.646 INFO:tasks.workunit.client.1.smithi171.stdout:6/28: dwrite f2 [0,4194304] 0 2022-01-31T19:44:16.648 INFO:tasks.workunit.client.1.smithi171.stdout:9/13: dwrite f4 [0,4194304] 0 2022-01-31T19:44:16.653 INFO:tasks.workunit.client.1.smithi171.stdout:2/68: dread d7/d9/ff [0,4194304] 0 2022-01-31T19:44:16.654 INFO:tasks.workunit.client.1.smithi171.stdout:2/69: truncate d7/d9/ff 1768769 0 2022-01-31T19:44:16.655 INFO:tasks.workunit.client.1.smithi171.stdout:2/70: rename d7/d9/fb to d7/dd/f15 0 2022-01-31T19:44:16.656 INFO:tasks.workunit.client.1.smithi171.stdout:2/71: chown d7/dd 3204 1 2022-01-31T19:44:16.658 INFO:tasks.workunit.client.1.smithi171.stdout:2/72: link f4 d7/dd/f16 0 2022-01-31T19:44:16.659 INFO:tasks.workunit.client.1.smithi171.stdout:2/73: symlink d7/dd/l17 0 2022-01-31T19:44:16.661 INFO:tasks.workunit.client.1.smithi171.stdout:4/21: dread f0 [0,4194304] 0 2022-01-31T19:44:16.668 INFO:tasks.workunit.client.1.smithi171.stdout:2/74: dread d7/d9/ff [0,4194304] 0 2022-01-31T19:44:16.669 INFO:tasks.workunit.client.1.smithi171.stdout:8/31: dwrite f2 [4194304,4194304] 0 2022-01-31T19:44:16.670 INFO:tasks.workunit.client.1.smithi171.stdout:4/22: dread f1 [0,4194304] 0 2022-01-31T19:44:16.670 INFO:tasks.workunit.client.1.smithi171.stdout:2/75: unlink d7/d9/ff 0 2022-01-31T19:44:16.680 INFO:tasks.workunit.client.0.smithi167.stdout:6/618: sync 2022-01-31T19:44:16.680 INFO:tasks.workunit.client.0.smithi167.stdout:5/883: sync 2022-01-31T19:44:16.681 INFO:tasks.workunit.client.1.smithi171.stdout:7/35: dwrite d1/d2/fc [0,4194304] 0 2022-01-31T19:44:16.681 INFO:tasks.workunit.client.1.smithi171.stdout:6/29: dwrite f2 [0,4194304] 0 2022-01-31T19:44:16.682 INFO:tasks.workunit.client.0.smithi167.stdout:6/619: creat d8/d33/d51/d5d/d86/fe7 x:0 0 0 2022-01-31T19:44:16.683 INFO:tasks.workunit.client.0.smithi167.stdout:6/620: mknod d8/d33/d51/d5d/d86/ce8 0 2022-01-31T19:44:16.684 INFO:tasks.workunit.client.1.smithi171.stdout:8/32: dread f5 [0,4194304] 0 2022-01-31T19:44:16.684 INFO:tasks.workunit.client.1.smithi171.stdout:8/33: stat l4 0 2022-01-31T19:44:16.685 INFO:tasks.workunit.client.1.smithi171.stdout:0/70: dwrite f6 [0,4194304] 0 2022-01-31T19:44:16.685 INFO:tasks.workunit.client.1.smithi171.stdout:0/71: stat d9/db/l15 0 2022-01-31T19:44:16.686 INFO:tasks.workunit.client.1.smithi171.stdout:0/72: write d9/f13 [252842,66587] 0 2022-01-31T19:44:16.686 INFO:tasks.workunit.client.1.smithi171.stdout:0/73: fsync d9/fa 0 2022-01-31T19:44:16.687 INFO:tasks.workunit.client.1.smithi171.stdout:6/30: rename c1 to c5 0 2022-01-31T19:44:16.690 INFO:tasks.workunit.client.1.smithi171.stdout:6/31: symlink l6 0 2022-01-31T19:44:16.690 INFO:tasks.workunit.client.1.smithi171.stdout:6/32: chown c5 281743 1 2022-01-31T19:44:16.691 INFO:tasks.workunit.client.1.smithi171.stdout:2/76: dwrite f4 [0,4194304] 0 2022-01-31T19:44:16.691 INFO:tasks.workunit.client.1.smithi171.stdout:2/77: stat f3 0 2022-01-31T19:44:16.692 INFO:tasks.workunit.client.1.smithi171.stdout:8/34: getdents . 0 2022-01-31T19:44:16.695 INFO:tasks.workunit.client.1.smithi171.stdout:6/33: rename l4 to l7 0 2022-01-31T19:44:16.695 INFO:tasks.workunit.client.1.smithi171.stdout:6/34: rmdir - no directory 2022-01-31T19:44:16.696 INFO:tasks.workunit.client.1.smithi171.stdout:2/78: getdents d7 0 2022-01-31T19:44:16.697 INFO:tasks.workunit.client.1.smithi171.stdout:8/35: mkdir df 0 2022-01-31T19:44:16.697 INFO:tasks.workunit.client.1.smithi171.stdout:2/79: write d7/f13 [2467687,18243] 0 2022-01-31T19:44:16.698 INFO:tasks.workunit.client.1.smithi171.stdout:6/35: mknod c8 0 2022-01-31T19:44:16.706 INFO:tasks.workunit.client.1.smithi171.stdout:4/23: rmdir d2/d3 39 2022-01-31T19:44:16.707 INFO:tasks.workunit.client.1.smithi171.stdout:2/80: unlink f4 0 2022-01-31T19:44:16.707 INFO:tasks.workunit.client.1.smithi171.stdout:4/24: write d2/d3/f7 [208124,16224] 0 2022-01-31T19:44:16.707 INFO:tasks.workunit.client.1.smithi171.stdout:4/25: chown f0 228268680 1 2022-01-31T19:44:16.708 INFO:tasks.workunit.client.1.smithi171.stdout:4/26: write f1 [137989,20559] 0 2022-01-31T19:44:16.708 INFO:tasks.workunit.client.1.smithi171.stdout:8/36: dread f0 [0,4194304] 0 2022-01-31T19:44:16.709 INFO:tasks.workunit.client.1.smithi171.stdout:2/81: mknod d7/dd/c18 0 2022-01-31T19:44:16.711 INFO:tasks.workunit.client.1.smithi171.stdout:4/27: mknod d2/c8 0 2022-01-31T19:44:16.711 INFO:tasks.workunit.client.1.smithi171.stdout:4/28: readlink d2/l4 0 2022-01-31T19:44:16.712 INFO:tasks.workunit.client.1.smithi171.stdout:4/29: write d2/d3/d5/f6 [560267,99944] 0 2022-01-31T19:44:16.712 INFO:tasks.workunit.client.1.smithi171.stdout:4/30: write d2/d3/d5/f6 [364845,95295] 0 2022-01-31T19:44:16.713 INFO:tasks.workunit.client.0.smithi167.stdout:6/621: dwrite d8/d33/d4f/f65 [0,4194304] 0 2022-01-31T19:44:16.714 INFO:tasks.workunit.client.1.smithi171.stdout:8/37: read - f6 zero size 2022-01-31T19:44:16.714 INFO:tasks.workunit.client.1.smithi171.stdout:2/82: creat d7/f19 x:0 0 0 2022-01-31T19:44:16.714 INFO:tasks.workunit.client.1.smithi171.stdout:4/31: mknod d2/c9 0 2022-01-31T19:44:16.715 INFO:tasks.workunit.client.0.smithi167.stdout:5/884: dwrite d5/d6/d16/d18/f1f [0,4194304] 0 2022-01-31T19:44:16.716 INFO:tasks.workunit.client.0.smithi167.stdout:5/885: write d5/d6/de/d22/d6d/fcb [787908,9737] 0 2022-01-31T19:44:16.716 INFO:tasks.workunit.client.0.smithi167.stdout:5/886: read - d5/d6/d3a/f10f zero size 2022-01-31T19:44:16.716 INFO:tasks.workunit.client.0.smithi167.stdout:5/887: chown d5/d6/d16/d18/f1f 18358647 1 2022-01-31T19:44:16.716 INFO:tasks.workunit.client.0.smithi167.stdout:6/622: symlink d8/d33/d51/d72/dad/le9 0 2022-01-31T19:44:16.717 INFO:tasks.workunit.client.0.smithi167.stdout:6/623: write d8/d62/d92/fb8 [870458,126007] 0 2022-01-31T19:44:16.717 INFO:tasks.workunit.client.0.smithi167.stdout:6/624: fdatasync d8/d33/fd2 0 2022-01-31T19:44:16.718 INFO:tasks.workunit.client.1.smithi171.stdout:8/38: creat df/f10 x:0 0 0 2022-01-31T19:44:16.718 INFO:tasks.workunit.client.1.smithi171.stdout:6/36: dread f2 [0,4194304] 0 2022-01-31T19:44:16.719 INFO:tasks.workunit.client.0.smithi167.stdout:5/888: write d5/d6/d16/d18/f1f [767503,12087] 0 2022-01-31T19:44:16.719 INFO:tasks.workunit.client.0.smithi167.stdout:6/625: truncate d8/d33/d51/d5d/d64/dab/fae 148293 0 2022-01-31T19:44:16.719 INFO:tasks.workunit.client.0.smithi167.stdout:6/626: truncate d8/d62/d92/fa1 659816 0 2022-01-31T19:44:16.720 INFO:tasks.workunit.client.1.smithi171.stdout:7/36: dwrite d1/d2/fc [0,4194304] 0 2022-01-31T19:44:16.720 INFO:tasks.workunit.client.1.smithi171.stdout:7/37: fdatasync d1/d2/fc 0 2022-01-31T19:44:16.721 INFO:tasks.workunit.client.0.smithi167.stdout:5/889: creat d5/d6/d16/d18/d43/d45/dd5/dbc/f116 x:0 0 0 2022-01-31T19:44:16.721 INFO:tasks.workunit.client.0.smithi167.stdout:5/890: readlink d5/d6/d16/d18/d43/d45/dd5/dbc/dc0/lc5 0 2022-01-31T19:44:16.722 INFO:tasks.workunit.client.0.smithi167.stdout:6/627: mknod d8/d33/d51/d7e/cea 0 2022-01-31T19:44:16.723 INFO:tasks.workunit.client.1.smithi171.stdout:6/37: write f2 [2553886,83849] 0 2022-01-31T19:44:16.723 INFO:tasks.workunit.client.1.smithi171.stdout:6/38: creat f9 x:0 0 0 2022-01-31T19:44:16.723 INFO:tasks.workunit.client.1.smithi171.stdout:6/39: creat fa x:0 0 0 2022-01-31T19:44:16.723 INFO:tasks.workunit.client.1.smithi171.stdout:7/38: rename l0 to d1/d2/ld 0 2022-01-31T19:44:16.724 INFO:tasks.workunit.client.0.smithi167.stdout:5/891: rename d5/ce7 to d5/d6/d16/d18/d5b/d63/c117 0 2022-01-31T19:44:16.724 INFO:tasks.workunit.client.0.smithi167.stdout:5/892: creat d5/f118 x:0 0 0 2022-01-31T19:44:16.724 INFO:tasks.workunit.client.0.smithi167.stdout:5/893: write d5/d6/d16/d18/d43/d45/d69/dc4/fb4 [689305,22676] 0 2022-01-31T19:44:16.725 INFO:tasks.workunit.client.0.smithi167.stdout:6/628: mkdir d8/d33/d51/d7e/dc4/deb 0 2022-01-31T19:44:16.725 INFO:tasks.workunit.client.1.smithi171.stdout:6/40: write f2 [1931501,110653] 0 2022-01-31T19:44:16.726 INFO:tasks.workunit.client.1.smithi171.stdout:6/41: rmdir - no directory 2022-01-31T19:44:16.726 INFO:tasks.workunit.client.1.smithi171.stdout:6/42: dread - fa zero size 2022-01-31T19:44:16.727 INFO:tasks.workunit.client.1.smithi171.stdout:4/32: fdatasync f1 0 2022-01-31T19:44:16.727 INFO:tasks.workunit.client.1.smithi171.stdout:0/74: dwrite f3 [0,4194304] 0 2022-01-31T19:44:16.727 INFO:tasks.workunit.client.1.smithi171.stdout:7/39: creat d1/d2/d7/fe x:0 0 0 2022-01-31T19:44:16.728 INFO:tasks.workunit.client.0.smithi167.stdout:6/629: unlink c7 0 2022-01-31T19:44:16.729 INFO:tasks.workunit.client.1.smithi171.stdout:4/33: unlink d2/l4 0 2022-01-31T19:44:16.730 INFO:tasks.workunit.client.1.smithi171.stdout:0/75: mkdir d9/d1b 0 2022-01-31T19:44:16.740 INFO:tasks.workunit.client.1.smithi171.stdout:7/40: write d1/d2/fc [843682,81142] 0 2022-01-31T19:44:16.740 INFO:tasks.workunit.client.1.smithi171.stdout:7/41: dread - d1/d2/d7/fe zero size 2022-01-31T19:44:16.740 INFO:tasks.workunit.client.1.smithi171.stdout:7/42: creat d1/d2/d7/ff x:0 0 0 2022-01-31T19:44:16.743 INFO:tasks.workunit.client.1.smithi171.stdout:7/43: unlink d1/l9 0 2022-01-31T19:44:16.744 INFO:tasks.workunit.client.1.smithi171.stdout:7/44: creat d1/d2/f10 x:0 0 0 2022-01-31T19:44:16.754 INFO:tasks.workunit.client.0.smithi167.stdout:5/894: dread d5/d61/f7f [0,4194304] 0 2022-01-31T19:44:16.755 INFO:tasks.workunit.client.0.smithi167.stdout:5/895: creat d5/d6/d16/f119 x:0 0 0 2022-01-31T19:44:16.756 INFO:tasks.workunit.client.0.smithi167.stdout:5/896: mknod d5/d6/d16/d18/d5b/c11a 0 2022-01-31T19:44:16.758 INFO:tasks.workunit.client.0.smithi167.stdout:5/897: unlink d5/dcd/f112 0 2022-01-31T19:44:16.760 INFO:tasks.workunit.client.0.smithi167.stdout:5/898: mknod d5/d6/d16/d18/d43/d45/dd5/c11b 0 2022-01-31T19:44:16.760 INFO:tasks.workunit.client.0.smithi167.stdout:5/899: stat d5/d6/d16/d18/d43/d45/dd5/c11b 0 2022-01-31T19:44:16.760 INFO:tasks.workunit.client.0.smithi167.stdout:5/900: mkdir d5/d6/d16/d18/d5b/d63/d11c 0 2022-01-31T19:44:16.761 INFO:tasks.workunit.client.0.smithi167.stdout:5/901: mkdir d5/d6/d16/d11d 0 2022-01-31T19:44:16.762 INFO:tasks.workunit.client.0.smithi167.stdout:5/902: mknod d5/d6/de/d22/d6d/da0/da4/daa/c11e 0 2022-01-31T19:44:16.763 INFO:tasks.workunit.client.1.smithi171.stdout:8/39: dwrite df/f10 [0,4194304] 0 2022-01-31T19:44:16.763 INFO:tasks.workunit.client.0.smithi167.stdout:5/903: truncate d5/d61/f74 78390 0 2022-01-31T19:44:16.764 INFO:tasks.workunit.client.1.smithi171.stdout:2/83: dwrite d7/dd/f15 [4194304,4194304] 0 2022-01-31T19:44:16.764 INFO:tasks.workunit.client.1.smithi171.stdout:2/84: write d7/f19 [989542,102937] 0 2022-01-31T19:44:16.765 INFO:tasks.workunit.client.1.smithi171.stdout:2/85: chown f3 131225 1 2022-01-31T19:44:16.765 INFO:tasks.workunit.client.1.smithi171.stdout:4/34: dread d2/d3/f7 [0,4194304] 0 2022-01-31T19:44:16.766 INFO:tasks.workunit.client.1.smithi171.stdout:0/76: dwrite d9/f16 [0,4194304] 0 2022-01-31T19:44:16.768 INFO:tasks.workunit.client.0.smithi167.stdout:5/904: rename d5/d6/de/fe4 to d5/d6/d16/d18/d43/d45/d69/d90/f11f 0 2022-01-31T19:44:16.768 INFO:tasks.workunit.client.1.smithi171.stdout:8/40: truncate f7 317094 0 2022-01-31T19:44:16.769 INFO:tasks.workunit.client.1.smithi171.stdout:6/43: dread f2 [0,4194304] 0 2022-01-31T19:44:16.770 INFO:tasks.workunit.client.1.smithi171.stdout:6/44: write fa [300219,128887] 0 2022-01-31T19:44:16.770 INFO:tasks.workunit.client.1.smithi171.stdout:6/45: chown c5 132 1 2022-01-31T19:44:16.770 INFO:tasks.workunit.client.1.smithi171.stdout:2/86: mkdir d7/d9/d1a 0 2022-01-31T19:44:16.771 INFO:tasks.workunit.client.1.smithi171.stdout:7/45: getdents d1/d2 0 2022-01-31T19:44:16.774 INFO:tasks.workunit.client.1.smithi171.stdout:4/35: mkdir d2/d3/da 0 2022-01-31T19:44:16.779 INFO:tasks.workunit.client.1.smithi171.stdout:0/77: mknod d9/db/c1c 0 2022-01-31T19:44:16.780 INFO:tasks.workunit.client.1.smithi171.stdout:0/78: fdatasync f3 0 2022-01-31T19:44:16.782 INFO:tasks.workunit.client.1.smithi171.stdout:8/41: readlink ld 0 2022-01-31T19:44:16.782 INFO:tasks.workunit.client.1.smithi171.stdout:2/87: creat d7/dd/f1b x:0 0 0 2022-01-31T19:44:16.782 INFO:tasks.workunit.client.1.smithi171.stdout:6/46: symlink lb 0 2022-01-31T19:44:16.783 INFO:tasks.workunit.client.1.smithi171.stdout:0/79: creat d9/f1d x:0 0 0 2022-01-31T19:44:16.784 INFO:tasks.workunit.client.1.smithi171.stdout:8/42: chown f2 267029995 1 2022-01-31T19:44:16.785 INFO:tasks.workunit.client.1.smithi171.stdout:0/80: unlink c2 0 2022-01-31T19:44:16.793 INFO:tasks.workunit.client.0.smithi167.stdout:5/905: dwrite d5/d6/d16/d18/d5b/f6c [0,4194304] 0 2022-01-31T19:44:16.793 INFO:tasks.workunit.client.1.smithi171.stdout:7/46: dwrite d1/d2/f10 [0,4194304] 0 2022-01-31T19:44:16.793 INFO:tasks.workunit.client.1.smithi171.stdout:7/47: chown d1/d2/d7/la 9184816 1 2022-01-31T19:44:16.794 INFO:tasks.workunit.client.0.smithi167.stdout:5/906: creat d5/d6/de/d22/d4d/d10d/f120 x:0 0 0 2022-01-31T19:44:16.798 INFO:tasks.workunit.client.0.smithi167.stdout:5/907: mknod d5/d6/d16/d18/d43/d45/dd5/dbc/dcc/ded/c121 0 2022-01-31T19:44:16.798 INFO:tasks.workunit.client.0.smithi167.stdout:5/908: write d5/d6/d16/d18/f1f [4281636,112102] 0 2022-01-31T19:44:16.798 INFO:tasks.workunit.client.0.smithi167.stdout:5/909: getdents d5/d6/d16/d18/d5b/d63/d11c 0 2022-01-31T19:44:16.799 INFO:tasks.workunit.client.1.smithi171.stdout:7/48: read d1/d2/fc [1629128,101061] 0 2022-01-31T19:44:16.800 INFO:tasks.workunit.client.1.smithi171.stdout:7/49: rename d1/d2/d7/la to d1/d2/d7/l11 0 2022-01-31T19:44:16.801 INFO:tasks.workunit.client.0.smithi167.stdout:5/910: write d5/d6/d16/d18/f2c [1044727,75483] 0 2022-01-31T19:44:16.804 INFO:tasks.workunit.client.1.smithi171.stdout:2/88: dwrite d7/f13 [4194304,4194304] 0 2022-01-31T19:44:16.805 INFO:tasks.workunit.client.0.smithi167.stdout:5/911: rmdir d5/d6/d16/d18/d43/d45/dd5/dbc/dc0/ddf 0 2022-01-31T19:44:16.812 INFO:tasks.workunit.client.0.smithi167.stdout:5/912: getdents d5/d6/de/d22/d76/db0/de1 0 2022-01-31T19:44:16.812 INFO:tasks.workunit.client.0.smithi167.stdout:5/913: truncate d5/d6/d16/d18/f2c 4949415 0 2022-01-31T19:44:16.822 INFO:tasks.workunit.client.1.smithi171.stdout:6/47: dwrite f9 [0,4194304] 0 2022-01-31T19:44:16.825 INFO:tasks.workunit.client.1.smithi171.stdout:2/89: dread d7/dd/f16 [0,4194304] 0 2022-01-31T19:44:16.826 INFO:tasks.workunit.client.1.smithi171.stdout:2/90: fsync d7/dd/f16 0 2022-01-31T19:44:16.826 INFO:tasks.workunit.client.1.smithi171.stdout:2/91: fdatasync d7/d9/f12 0 2022-01-31T19:44:16.826 INFO:tasks.workunit.client.1.smithi171.stdout:2/92: creat d7/f1c x:0 0 0 2022-01-31T19:44:16.826 INFO:tasks.workunit.client.1.smithi171.stdout:0/81: dwrite f4 [0,4194304] 0 2022-01-31T19:44:16.826 INFO:tasks.workunit.client.1.smithi171.stdout:0/82: chown d9/f16 304047198 1 2022-01-31T19:44:16.835 INFO:tasks.workunit.client.1.smithi171.stdout:6/48: dread f9 [0,4194304] 0 2022-01-31T19:44:16.835 INFO:tasks.workunit.client.1.smithi171.stdout:6/49: read f9 [2660546,108958] 0 2022-01-31T19:44:16.836 INFO:tasks.workunit.client.1.smithi171.stdout:6/50: fdatasync fa 0 2022-01-31T19:44:16.839 INFO:tasks.workunit.client.1.smithi171.stdout:6/51: dread fa [0,4194304] 0 2022-01-31T19:44:16.841 INFO:tasks.workunit.client.1.smithi171.stdout:8/43: dwrite f1 [4194304,4194304] 0 2022-01-31T19:44:16.842 INFO:tasks.workunit.client.0.smithi167.stdout:5/914: dwrite d5/d6/ffd [0,4194304] 0 2022-01-31T19:44:16.842 INFO:tasks.workunit.client.0.smithi167.stdout:5/915: readlink d5/d6/de/d22/l77 0 2022-01-31T19:44:16.843 INFO:tasks.workunit.client.1.smithi171.stdout:2/93: dwrite d7/f14 [0,4194304] 0 2022-01-31T19:44:16.843 INFO:tasks.workunit.client.1.smithi171.stdout:8/44: mknod df/c11 0 2022-01-31T19:44:16.846 INFO:tasks.workunit.client.1.smithi171.stdout:6/52: dread f2 [0,4194304] 0 2022-01-31T19:44:16.847 INFO:tasks.workunit.client.0.smithi167.stdout:5/916: write d5/f8 [3169868,108265] 0 2022-01-31T19:44:16.847 INFO:tasks.workunit.client.0.smithi167.stdout:5/917: chown l4 982028 1 2022-01-31T19:44:16.848 INFO:tasks.workunit.client.1.smithi171.stdout:6/53: mkdir dc 0 2022-01-31T19:44:16.849 INFO:tasks.workunit.client.0.smithi167.stdout:5/918: mkdir d5/d6/d16/d18/d122 0 2022-01-31T19:44:16.849 INFO:tasks.workunit.client.0.smithi167.stdout:5/919: read d5/d6/de/d22/d4d/fb8 [752401,19586] 0 2022-01-31T19:44:16.849 INFO:tasks.workunit.client.0.smithi167.stdout:5/920: dread - d5/d6/de/d22/d6d/fb7 zero size 2022-01-31T19:44:16.850 INFO:tasks.workunit.client.0.smithi167.stdout:5/921: write d5/d6/de/d22/d4d/fa1 [5044779,127381] 0 2022-01-31T19:44:16.851 INFO:tasks.workunit.client.1.smithi171.stdout:6/54: mkdir dc/dd 0 2022-01-31T19:44:16.851 INFO:tasks.workunit.client.1.smithi171.stdout:6/55: write f2 [293218,19861] 0 2022-01-31T19:44:16.852 INFO:tasks.workunit.client.1.smithi171.stdout:6/56: mknod dc/ce 0 2022-01-31T19:44:16.852 INFO:tasks.workunit.client.1.smithi171.stdout:6/57: fsync f9 0 2022-01-31T19:44:16.852 INFO:tasks.workunit.client.1.smithi171.stdout:6/58: symlink dc/lf 0 2022-01-31T19:44:16.853 INFO:tasks.workunit.client.0.smithi167.stdout:5/922: creat d5/d6/d16/d11d/f123 x:0 0 0 2022-01-31T19:44:16.854 INFO:tasks.workunit.client.1.smithi171.stdout:2/94: dread d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:16.855 INFO:tasks.workunit.client.1.smithi171.stdout:2/95: creat d7/d9/f1d x:0 0 0 2022-01-31T19:44:16.856 INFO:tasks.workunit.client.1.smithi171.stdout:2/96: creat d7/dd/f1e x:0 0 0 2022-01-31T19:44:16.856 INFO:tasks.workunit.client.1.smithi171.stdout:2/97: fsync d7/f19 0 2022-01-31T19:44:16.857 INFO:tasks.workunit.client.1.smithi171.stdout:0/83: dwrite d9/f13 [0,4194304] 0 2022-01-31T19:44:16.857 INFO:tasks.workunit.client.0.smithi167.stdout:5/923: mkdir d5/d6/de/d124 0 2022-01-31T19:44:16.858 INFO:tasks.workunit.client.1.smithi171.stdout:2/98: mknod d7/dd/c1f 0 2022-01-31T19:44:16.859 INFO:tasks.workunit.client.1.smithi171.stdout:2/99: write d7/f19 [1192610,13467] 0 2022-01-31T19:44:16.859 INFO:tasks.workunit.client.1.smithi171.stdout:2/100: write d7/f1c [802698,53995] 0 2022-01-31T19:44:16.860 INFO:tasks.workunit.client.0.smithi167.stdout:5/924: mkdir d5/d125 0 2022-01-31T19:44:16.860 INFO:tasks.workunit.client.0.smithi167.stdout:5/925: fdatasync d5/d6/de/d22/d4d/f85 0 2022-01-31T19:44:16.861 INFO:tasks.workunit.client.1.smithi171.stdout:8/45: getdents df 0 2022-01-31T19:44:16.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/797: sync 2022-01-31T19:44:16.866 INFO:tasks.workunit.client.0.smithi167.stdout:9/663: sync 2022-01-31T19:44:16.867 INFO:tasks.workunit.client.0.smithi167.stdout:3/798: symlink d9/d29/d3b/d41/deb/lf2 0 2022-01-31T19:44:16.867 INFO:tasks.workunit.client.0.smithi167.stdout:9/664: dread dd/d19/d24/d56/d5b/f5c [0,4194304] 0 2022-01-31T19:44:16.867 INFO:tasks.workunit.client.0.smithi167.stdout:9/665: chown dd/d35/f44 3 1 2022-01-31T19:44:16.868 INFO:tasks.workunit.client.1.smithi171.stdout:8/46: mknod df/c12 0 2022-01-31T19:44:16.868 INFO:tasks.workunit.client.1.smithi171.stdout:0/84: rmdir d9 39 2022-01-31T19:44:16.868 INFO:tasks.workunit.client.1.smithi171.stdout:0/85: readlink d9/l18 0 2022-01-31T19:44:16.869 INFO:tasks.workunit.client.0.smithi167.stdout:9/666: mkdir dd/d17/dc4/de2 0 2022-01-31T19:44:16.869 INFO:tasks.workunit.client.0.smithi167.stdout:9/667: chown dd/d17/d7f/fb2 11928467 1 2022-01-31T19:44:16.870 INFO:tasks.workunit.client.0.smithi167.stdout:9/668: dread - dd/d19/d59/fad zero size 2022-01-31T19:44:16.870 INFO:tasks.workunit.client.0.smithi167.stdout:9/669: rename dd/d17/d80 to dd/d17/d80/d4c/de3 22 2022-01-31T19:44:16.870 INFO:tasks.workunit.client.1.smithi171.stdout:8/47: mknod df/c13 0 2022-01-31T19:44:16.871 INFO:tasks.workunit.client.1.smithi171.stdout:8/48: fsync f6 0 2022-01-31T19:44:16.871 INFO:tasks.workunit.client.0.smithi167.stdout:9/670: unlink dd/d35/f3a 0 2022-01-31T19:44:16.872 INFO:tasks.workunit.client.0.smithi167.stdout:3/799: dread d9/f27 [0,4194304] 0 2022-01-31T19:44:16.874 INFO:tasks.workunit.client.0.smithi167.stdout:5/926: dread d5/d6/f41 [0,4194304] 0 2022-01-31T19:44:16.874 INFO:tasks.workunit.client.0.smithi167.stdout:3/800: unlink d9/d26/l38 0 2022-01-31T19:44:16.875 INFO:tasks.workunit.client.0.smithi167.stdout:3/801: getdents d9/d21/ddd 0 2022-01-31T19:44:16.875 INFO:tasks.workunit.client.1.smithi171.stdout:8/49: rmdir df 39 2022-01-31T19:44:16.876 INFO:tasks.workunit.client.0.smithi167.stdout:5/927: read d5/d6/d16/d18/d43/d45/d69/dc4/dbb/ff2 [52311,77972] 0 2022-01-31T19:44:16.876 INFO:tasks.workunit.client.0.smithi167.stdout:3/802: rmdir d9/d29/d3b/d41/d6b/d89 39 2022-01-31T19:44:16.878 INFO:tasks.workunit.client.0.smithi167.stdout:5/928: truncate d5/d6/de/d22/d76/f95 2577067 0 2022-01-31T19:44:16.879 INFO:tasks.workunit.client.1.smithi171.stdout:6/59: dwrite f9 [4194304,4194304] 0 2022-01-31T19:44:16.922 INFO:tasks.workunit.client.1.smithi171.stdout:2/101: dwrite d7/f1c [0,4194304] 0 2022-01-31T19:44:16.925 INFO:tasks.workunit.client.0.smithi167.stdout:3/803: dwrite d9/d26/f6e [0,4194304] 0 2022-01-31T19:44:16.925 INFO:tasks.workunit.client.0.smithi167.stdout:5/929: dwrite d5/d6/d16/d18/d43/d45/d69/dc4/f7c [8388608,4194304] 0 2022-01-31T19:44:16.925 INFO:tasks.workunit.client.0.smithi167.stdout:9/671: dwrite dd/d6b/f7a [0,4194304] 0 2022-01-31T19:44:16.926 INFO:tasks.workunit.client.0.smithi167.stdout:9/672: fsync dd/d19/d59/f98 0 2022-01-31T19:44:16.926 INFO:tasks.workunit.client.0.smithi167.stdout:9/673: creat dd/d19/d24/d56/dc9/fe4 x:0 0 0 2022-01-31T19:44:16.926 INFO:tasks.workunit.client.1.smithi171.stdout:0/86: dwrite d9/f16 [0,4194304] 0 2022-01-31T19:44:16.927 INFO:tasks.workunit.client.1.smithi171.stdout:0/87: creat d9/f1e x:0 0 0 2022-01-31T19:44:16.927 INFO:tasks.workunit.client.1.smithi171.stdout:0/88: chown f8 98717214 1 2022-01-31T19:44:16.928 INFO:tasks.workunit.client.1.smithi171.stdout:8/50: getdents df 0 2022-01-31T19:44:16.929 INFO:tasks.workunit.client.1.smithi171.stdout:6/60: dread f2 [0,4194304] 0 2022-01-31T19:44:16.930 INFO:tasks.workunit.client.0.smithi167.stdout:3/804: rename d9/d29/l83 to d9/d29/d3b/d41/lf3 0 2022-01-31T19:44:16.930 INFO:tasks.workunit.client.1.smithi171.stdout:0/89: rename f7 to d9/f1f 0 2022-01-31T19:44:16.931 INFO:tasks.workunit.client.0.smithi167.stdout:5/930: symlink d5/d6/d16/d18/d43/d45/d69/d90/df9/l126 0 2022-01-31T19:44:16.931 INFO:tasks.workunit.client.0.smithi167.stdout:3/805: write d9/d29/d60/fba [4015263,45954] 0 2022-01-31T19:44:16.933 INFO:tasks.workunit.client.1.smithi171.stdout:2/102: truncate d7/f19 354836 0 2022-01-31T19:44:16.933 INFO:tasks.workunit.client.1.smithi171.stdout:2/103: read - d7/f11 zero size 2022-01-31T19:44:16.934 INFO:tasks.workunit.client.1.smithi171.stdout:6/61: rmdir dc/dd 0 2022-01-31T19:44:16.934 INFO:tasks.workunit.client.0.smithi167.stdout:9/674: rmdir dd/dd9 0 2022-01-31T19:44:16.935 INFO:tasks.workunit.client.0.smithi167.stdout:3/806: dread d9/d29/d3b/d41/d6b/d89/fd6 [0,4194304] 0 2022-01-31T19:44:16.939 INFO:tasks.workunit.client.0.smithi167.stdout:5/931: rename d5/d6/de/f3e to d5/d6/de/d22/d76/da3/f127 0 2022-01-31T19:44:16.940 INFO:tasks.workunit.client.1.smithi171.stdout:2/104: mknod d7/c20 0 2022-01-31T19:44:16.941 INFO:tasks.workunit.client.0.smithi167.stdout:9/675: link dd/d19/d24/d56/d5b/da5/fb4 dd/d35/fe5 0 2022-01-31T19:44:16.941 INFO:tasks.workunit.client.1.smithi171.stdout:2/105: unlink d7/dd/f16 0 2022-01-31T19:44:16.942 INFO:tasks.workunit.client.1.smithi171.stdout:2/106: stat d7/f14 0 2022-01-31T19:44:16.942 INFO:tasks.workunit.client.0.smithi167.stdout:3/807: symlink d9/d7f/lf4 0 2022-01-31T19:44:16.943 INFO:tasks.workunit.client.0.smithi167.stdout:9/676: link dd/d34/f61 dd/d19/d24/d56/d5b/dcf/fe6 0 2022-01-31T19:44:16.943 INFO:tasks.workunit.client.0.smithi167.stdout:9/677: fsync dd/d19/d24/d56/dc9/fd7 0 2022-01-31T19:44:16.944 INFO:tasks.workunit.client.0.smithi167.stdout:9/678: dread - dd/d19/d24/d56/dc9/fd7 zero size 2022-01-31T19:44:16.944 INFO:tasks.workunit.client.0.smithi167.stdout:3/808: mkdir d9/d29/d3b/d41/df5 0 2022-01-31T19:44:16.945 INFO:tasks.workunit.client.0.smithi167.stdout:3/809: dread - d9/d29/d3b/d9b/da3/dd9/fe3 zero size 2022-01-31T19:44:16.946 INFO:tasks.workunit.client.0.smithi167.stdout:9/679: mkdir dd/de7 0 2022-01-31T19:44:16.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/107: rename d7/dd/c18 to d7/dd/c21 0 2022-01-31T19:44:16.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/108: readlink d7/lc 0 2022-01-31T19:44:16.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/109: truncate d7/fa 5117730 0 2022-01-31T19:44:16.951 INFO:tasks.workunit.client.1.smithi171.stdout:2/110: link d7/f19 d7/f22 0 2022-01-31T19:44:16.952 INFO:tasks.workunit.client.0.smithi167.stdout:5/932: dread d5/d6/d16/d18/f2c [0,4194304] 0 2022-01-31T19:44:16.953 INFO:tasks.workunit.client.1.smithi171.stdout:8/51: dwrite f7 [0,4194304] 0 2022-01-31T19:44:16.953 INFO:tasks.workunit.client.1.smithi171.stdout:6/62: getdents dc 0 2022-01-31T19:44:16.953 INFO:tasks.workunit.client.1.smithi171.stdout:2/111: write d7/f1c [2784861,101734] 0 2022-01-31T19:44:16.955 INFO:tasks.workunit.client.0.smithi167.stdout:5/933: mkdir d5/d6/de/d22/d4d/d10d/d128 0 2022-01-31T19:44:16.955 INFO:tasks.workunit.client.0.smithi167.stdout:5/934: truncate d5/d6/d16/f119 920726 0 2022-01-31T19:44:16.955 INFO:tasks.workunit.client.0.smithi167.stdout:9/680: write dd/d6b/f83 [38818,32338] 0 2022-01-31T19:44:16.958 INFO:tasks.workunit.client.1.smithi171.stdout:2/112: dread d7/f22 [0,4194304] 0 2022-01-31T19:44:16.958 INFO:tasks.workunit.client.1.smithi171.stdout:2/113: creat d7/d9/f23 x:0 0 0 2022-01-31T19:44:16.959 INFO:tasks.workunit.client.0.smithi167.stdout:5/935: rename d5/d6/d16/d18/d43/l9b to d5/d6/d16/d18/d43/d45/d69/l129 0 2022-01-31T19:44:16.960 INFO:tasks.workunit.client.1.smithi171.stdout:8/52: dread - fa zero size 2022-01-31T19:44:16.960 INFO:tasks.workunit.client.1.smithi171.stdout:8/53: write f1 [8695808,99567] 0 2022-01-31T19:44:16.960 INFO:tasks.workunit.client.1.smithi171.stdout:8/54: truncate f0 816512 0 2022-01-31T19:44:16.961 INFO:tasks.workunit.client.1.smithi171.stdout:6/63: rmdir dc 39 2022-01-31T19:44:16.961 INFO:tasks.workunit.client.1.smithi171.stdout:6/64: readlink dc/lf 0 2022-01-31T19:44:16.962 INFO:tasks.workunit.client.0.smithi167.stdout:9/681: read - dd/d17/d7f/fbb zero size 2022-01-31T19:44:16.962 INFO:tasks.workunit.client.0.smithi167.stdout:3/810: dwrite d9/fd3 [0,4194304] 0 2022-01-31T19:44:16.963 INFO:tasks.workunit.client.0.smithi167.stdout:3/811: chown d9/d29/d3b/d41/df5 0 1 2022-01-31T19:44:16.963 INFO:tasks.workunit.client.1.smithi171.stdout:2/114: fdatasync d7/dd/f15 0 2022-01-31T19:44:16.964 INFO:tasks.workunit.client.1.smithi171.stdout:6/65: write f2 [3761194,127909] 0 2022-01-31T19:44:16.964 INFO:tasks.workunit.client.1.smithi171.stdout:2/115: mknod d7/d9/c24 0 2022-01-31T19:44:16.964 INFO:tasks.workunit.client.1.smithi171.stdout:2/116: unlink f3 0 2022-01-31T19:44:16.964 INFO:tasks.workunit.client.1.smithi171.stdout:2/117: creat d7/f25 x:0 0 0 2022-01-31T19:44:16.965 INFO:tasks.workunit.client.1.smithi171.stdout:2/118: write d7/dd/f1b [891905,89384] 0 2022-01-31T19:44:16.965 INFO:tasks.workunit.client.1.smithi171.stdout:6/66: mknod dc/c10 0 2022-01-31T19:44:16.966 INFO:tasks.workunit.client.1.smithi171.stdout:2/119: write d7/dd/f15 [6836973,46213] 0 2022-01-31T19:44:16.967 INFO:tasks.workunit.client.0.smithi167.stdout:3/812: dread d9/d29/fb2 [0,4194304] 0 2022-01-31T19:44:16.967 INFO:tasks.workunit.client.0.smithi167.stdout:3/813: write d9/d7f/dbe/fcc [791227,31531] 0 2022-01-31T19:44:16.967 INFO:tasks.workunit.client.0.smithi167.stdout:3/814: chown d9/d29/d3b/d9b/da3 76 1 2022-01-31T19:44:16.968 INFO:tasks.workunit.client.0.smithi167.stdout:9/682: write dd/d35/f33 [2204278,3075] 0 2022-01-31T19:44:16.968 INFO:tasks.workunit.client.0.smithi167.stdout:9/683: getdents dd/d19/d24/d50 0 2022-01-31T19:44:16.968 INFO:tasks.workunit.client.0.smithi167.stdout:9/684: creat dd/d19/d24/d56/d5b/dcf/fe8 x:0 0 0 2022-01-31T19:44:16.971 INFO:tasks.workunit.client.0.smithi167.stdout:5/936: dread d5/d6/de/d22/f97 [0,4194304] 0 2022-01-31T19:44:16.974 INFO:tasks.workunit.client.0.smithi167.stdout:9/685: mkdir dd/d19/d24/d56/d5b/da5/de9 0 2022-01-31T19:44:16.979 INFO:tasks.workunit.client.0.smithi167.stdout:9/686: read dd/d19/d24/d50/dd1/f8a [1568269,18451] 0 2022-01-31T19:44:16.985 INFO:tasks.workunit.client.0.smithi167.stdout:9/687: creat dd/d17/d80/d4c/fea x:0 0 0 2022-01-31T19:44:16.985 INFO:tasks.workunit.client.0.smithi167.stdout:9/688: dread - dd/d19/d24/fb6 zero size 2022-01-31T19:44:16.986 INFO:tasks.workunit.client.0.smithi167.stdout:5/937: write d5/d61/f74 [281719,30590] 0 2022-01-31T19:44:16.988 INFO:tasks.workunit.client.0.smithi167.stdout:9/689: rmdir dd/d17/dc4 39 2022-01-31T19:44:17.000 INFO:tasks.workunit.client.0.smithi167.stdout:5/938: dread d5/f44 [0,4194304] 0 2022-01-31T19:44:17.001 INFO:tasks.workunit.client.0.smithi167.stdout:5/939: chown d5/d6/de/d22/d6d/da0/da4/ce3 78832957 1 2022-01-31T19:44:17.011 INFO:tasks.workunit.client.0.smithi167.stdout:3/815: dwrite d9/d29/d60/dc9/fdf [0,4194304] 0 2022-01-31T19:44:17.011 INFO:tasks.workunit.client.0.smithi167.stdout:3/816: write d9/d29/d3b/d41/d6b/d89/f97 [561061,114218] 0 2022-01-31T19:44:17.012 INFO:tasks.workunit.client.1.smithi171.stdout:2/120: dwrite d7/d9/f23 [0,4194304] 0 2022-01-31T19:44:17.012 INFO:tasks.workunit.client.1.smithi171.stdout:6/67: dwrite f2 [0,4194304] 0 2022-01-31T19:44:17.013 INFO:tasks.workunit.client.0.smithi167.stdout:9/690: dwrite dd/d17/d80/d4c/f67 [0,4194304] 0 2022-01-31T19:44:17.013 INFO:tasks.workunit.client.0.smithi167.stdout:9/691: readlink dd/d19/d24/l26 0 2022-01-31T19:44:17.013 INFO:tasks.workunit.client.0.smithi167.stdout:5/940: dread d5/d6/d16/d18/d43/fc7 [0,4194304] 0 2022-01-31T19:44:17.014 INFO:tasks.workunit.client.0.smithi167.stdout:5/941: chown d5/d6/de/d22/d76/db0/dee 24882 1 2022-01-31T19:44:17.014 INFO:tasks.workunit.client.1.smithi171.stdout:6/68: mknod dc/c11 0 2022-01-31T19:44:17.015 INFO:tasks.workunit.client.1.smithi171.stdout:2/121: rmdir d7/dd 39 2022-01-31T19:44:17.015 INFO:tasks.workunit.client.1.smithi171.stdout:2/122: readlink d7/d9/l10 0 2022-01-31T19:44:17.017 INFO:tasks.workunit.client.1.smithi171.stdout:2/123: mkdir d7/d26 0 2022-01-31T19:44:17.018 INFO:tasks.workunit.client.0.smithi167.stdout:3/817: rename d9/d29/c2f to d9/de7/cf6 0 2022-01-31T19:44:17.019 INFO:tasks.workunit.client.0.smithi167.stdout:3/818: fdatasync d9/d56/f90 0 2022-01-31T19:44:17.019 INFO:tasks.workunit.client.0.smithi167.stdout:3/819: dread - d9/d26/f98 zero size 2022-01-31T19:44:17.020 INFO:tasks.workunit.client.0.smithi167.stdout:5/942: creat d5/d6/d16/d18/f12a x:0 0 0 2022-01-31T19:44:17.021 INFO:tasks.workunit.client.0.smithi167.stdout:3/820: symlink d9/d7f/dbb/lf7 0 2022-01-31T19:44:17.023 INFO:tasks.workunit.client.1.smithi171.stdout:6/69: getdents dc 0 2022-01-31T19:44:17.027 INFO:tasks.workunit.client.0.smithi167.stdout:3/821: mkdir d9/d29/d60/df8 0 2022-01-31T19:44:17.028 INFO:tasks.workunit.client.0.smithi167.stdout:9/692: dread dd/d19/d24/d50/dd1/f8a [0,4194304] 0 2022-01-31T19:44:17.028 INFO:tasks.workunit.client.0.smithi167.stdout:9/693: truncate dd/d19/d24/d56/fc7 363500 0 2022-01-31T19:44:17.029 INFO:tasks.workunit.client.0.smithi167.stdout:9/694: stat dd/d19/d24/d50/d5d/c9e 0 2022-01-31T19:44:17.029 INFO:tasks.workunit.client.0.smithi167.stdout:9/695: fdatasync dd/d19/d24/d56/dc9/fd7 0 2022-01-31T19:44:17.031 INFO:tasks.workunit.client.1.smithi171.stdout:6/70: unlink lb 0 2022-01-31T19:44:17.031 INFO:tasks.workunit.client.1.smithi171.stdout:6/71: mknod dc/c12 0 2022-01-31T19:44:17.032 INFO:tasks.workunit.client.1.smithi171.stdout:6/72: write f9 [9267012,106315] 0 2022-01-31T19:44:17.032 INFO:tasks.workunit.client.1.smithi171.stdout:6/73: fsync f2 0 2022-01-31T19:44:17.033 INFO:tasks.workunit.client.1.smithi171.stdout:6/74: chown c5 91818394 1 2022-01-31T19:44:17.033 INFO:tasks.workunit.client.1.smithi171.stdout:6/75: chown c5 24402571 1 2022-01-31T19:44:17.034 INFO:tasks.workunit.client.0.smithi167.stdout:3/822: mknod d9/d29/d3b/d41/df5/cf9 0 2022-01-31T19:44:17.034 INFO:tasks.workunit.client.0.smithi167.stdout:3/823: chown d9/d29/d3b 2471279 1 2022-01-31T19:44:17.036 INFO:tasks.workunit.client.1.smithi171.stdout:6/76: creat dc/f13 x:0 0 0 2022-01-31T19:44:17.037 INFO:tasks.workunit.client.0.smithi167.stdout:1/833: sync 2022-01-31T19:44:17.038 INFO:tasks.workunit.client.0.smithi167.stdout:4/752: sync 2022-01-31T19:44:17.038 INFO:tasks.workunit.client.0.smithi167.stdout:4/753: truncate d0/d6/d7/d26/d31/ddf/da2/dcf/db5/fcb 927171 0 2022-01-31T19:44:17.039 INFO:tasks.workunit.client.0.smithi167.stdout:2/591: sync 2022-01-31T19:44:17.039 INFO:tasks.workunit.client.0.smithi167.stdout:8/771: sync 2022-01-31T19:44:17.039 INFO:tasks.workunit.client.0.smithi167.stdout:8/772: truncate d6/db/f2b 9601412 0 2022-01-31T19:44:17.040 INFO:tasks.workunit.client.0.smithi167.stdout:8/773: write d6/db/de/f26 [1042098,101365] 0 2022-01-31T19:44:17.040 INFO:tasks.workunit.client.0.smithi167.stdout:0/534: sync 2022-01-31T19:44:17.041 INFO:tasks.workunit.client.0.smithi167.stdout:7/923: sync 2022-01-31T19:44:17.041 INFO:tasks.workunit.client.0.smithi167.stdout:6/630: sync 2022-01-31T19:44:17.041 INFO:tasks.workunit.client.0.smithi167.stdout:1/834: dread d0/d10/d20/d7c/ff4 [0,4194304] 0 2022-01-31T19:44:17.042 INFO:tasks.workunit.client.0.smithi167.stdout:6/631: readlink d8/d33/d51/d5d/d86/dbd/lda 0 2022-01-31T19:44:17.042 INFO:tasks.workunit.client.0.smithi167.stdout:6/632: truncate d8/d33/d51/d72/dc2/fd9 259058 0 2022-01-31T19:44:17.043 INFO:tasks.workunit.client.0.smithi167.stdout:9/696: rename dd/d19/d24/d56/d5b/f94 to dd/d6b/dc1/feb 0 2022-01-31T19:44:17.043 INFO:tasks.workunit.client.0.smithi167.stdout:9/697: getdents dd/de7 0 2022-01-31T19:44:17.044 INFO:tasks.workunit.client.0.smithi167.stdout:7/924: dread d8/d2c/d43/f97 [0,4194304] 0 2022-01-31T19:44:17.044 INFO:tasks.workunit.client.0.smithi167.stdout:4/754: mknod d0/d6/d7/d26/d31/ddf/d75/cf9 0 2022-01-31T19:44:17.045 INFO:tasks.workunit.client.0.smithi167.stdout:8/774: creat d6/d1e/ffa x:0 0 0 2022-01-31T19:44:17.045 INFO:tasks.workunit.client.0.smithi167.stdout:6/633: rmdir d8/d33/d51/d5d/d64 39 2022-01-31T19:44:17.045 INFO:tasks.workunit.client.0.smithi167.stdout:9/698: rename dd/d34/c4d to dd/d35/cec 0 2022-01-31T19:44:17.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/699: stat dd/d6b/dd5 0 2022-01-31T19:44:17.048 INFO:tasks.workunit.client.1.smithi171.stdout:6/77: dread fa [0,4194304] 0 2022-01-31T19:44:17.048 INFO:tasks.workunit.client.1.smithi171.stdout:6/78: stat f9 0 2022-01-31T19:44:17.051 INFO:tasks.workunit.client.0.smithi167.stdout:6/634: write d8/d33/f40 [5174374,98300] 0 2022-01-31T19:44:17.052 INFO:tasks.workunit.client.0.smithi167.stdout:2/592: rename dd/d18/d62/laa to dd/d18/db7/lbc 0 2022-01-31T19:44:17.052 INFO:tasks.workunit.client.0.smithi167.stdout:6/635: creat d8/d33/d4f/dd1/fec x:0 0 0 2022-01-31T19:44:17.053 INFO:tasks.workunit.client.0.smithi167.stdout:7/925: rename d8/dab/fe4 to d8/d2c/d6e/d9a/f14e 0 2022-01-31T19:44:17.053 INFO:tasks.workunit.client.0.smithi167.stdout:9/700: write dd/d17/f2f [4892852,26428] 0 2022-01-31T19:44:17.053 INFO:tasks.workunit.client.0.smithi167.stdout:2/593: creat dd/df/d1b/d3d/d4a/d4b/d54/fbd x:0 0 0 2022-01-31T19:44:17.054 INFO:tasks.workunit.client.0.smithi167.stdout:1/835: dread d0/d10/d20/f37 [0,4194304] 0 2022-01-31T19:44:17.054 INFO:tasks.workunit.client.0.smithi167.stdout:5/943: dwrite d5/d6/de/d22/d76/db0/fb6 [0,4194304] 0 2022-01-31T19:44:17.055 INFO:tasks.workunit.client.0.smithi167.stdout:4/755: rename d0/d6/d7/d39/d4e/c6d to d0/d6/d7/d26/d40/d4c/cfa 0 2022-01-31T19:44:17.055 INFO:tasks.workunit.client.0.smithi167.stdout:9/701: link dd/d19/d24/d56/d5b/fbe dd/d19/d24/d56/d5b/da5/fed 0 2022-01-31T19:44:17.056 INFO:tasks.workunit.client.0.smithi167.stdout:1/836: creat d0/d100/f115 x:0 0 0 2022-01-31T19:44:17.056 INFO:tasks.workunit.client.0.smithi167.stdout:8/775: rename d6/d1e/d2d/d6b/cf6 to d6/db/d36/d5d/cfb 0 2022-01-31T19:44:17.057 INFO:tasks.workunit.client.0.smithi167.stdout:8/776: read d6/db/de/d18/fce [111230,36881] 0 2022-01-31T19:44:17.057 INFO:tasks.workunit.client.0.smithi167.stdout:4/756: creat d0/d6/d7/d39/d4e/ffb x:0 0 0 2022-01-31T19:44:17.057 INFO:tasks.workunit.client.0.smithi167.stdout:4/757: chown d0/d6/d7/d26/d40/d4c/fbe 9553 1 2022-01-31T19:44:17.059 INFO:tasks.workunit.client.1.smithi171.stdout:6/79: dread f2 [0,4194304] 0 2022-01-31T19:44:17.060 INFO:tasks.workunit.client.1.smithi171.stdout:6/80: creat dc/f14 x:0 0 0 2022-01-31T19:44:17.064 INFO:tasks.workunit.client.0.smithi167.stdout:1/837: creat d0/d10/d5a/d33/d97/f116 x:0 0 0 2022-01-31T19:44:17.065 INFO:tasks.workunit.client.0.smithi167.stdout:7/926: rename d8/dc/f40 to d8/d2c/d7a/d99/de2/dc3/de9/f14f 0 2022-01-31T19:44:17.065 INFO:tasks.workunit.client.0.smithi167.stdout:4/758: mknod d0/cfc 0 2022-01-31T19:44:17.065 INFO:tasks.workunit.client.0.smithi167.stdout:4/759: dread - d0/d6/d7/d26/d31/ddf/da2/dcf/db5/fed zero size 2022-01-31T19:44:17.066 INFO:tasks.workunit.client.0.smithi167.stdout:4/760: chown d0/d6/d7/d26/d31/d5c/fd7 276461555 1 2022-01-31T19:44:17.066 INFO:tasks.workunit.client.0.smithi167.stdout:4/761: write d0/d6/d7/d39/d4e/f52 [4963167,57620] 0 2022-01-31T19:44:17.067 INFO:tasks.workunit.client.0.smithi167.stdout:4/762: read - d0/d6/d7/d39/d4e/de7/fee zero size 2022-01-31T19:44:17.067 INFO:tasks.workunit.client.0.smithi167.stdout:4/763: write d0/d6/d7/d89/fad [865492,45343] 0 2022-01-31T19:44:17.068 INFO:tasks.workunit.client.0.smithi167.stdout:4/764: fdatasync d0/d6/d7/d39/d4e/ffb 0 2022-01-31T19:44:17.068 INFO:tasks.workunit.client.0.smithi167.stdout:4/765: creat d0/d6/d7/d26/d40/d4c/ffd x:0 0 0 2022-01-31T19:44:17.068 INFO:tasks.workunit.client.0.smithi167.stdout:4/766: chown d0/d6/dab/fde 5 1 2022-01-31T19:44:17.069 INFO:tasks.workunit.client.0.smithi167.stdout:1/838: mkdir d0/d10/d5a/d33/d97/d117 0 2022-01-31T19:44:17.069 INFO:tasks.workunit.client.0.smithi167.stdout:1/839: creat d0/db3/f118 x:0 0 0 2022-01-31T19:44:17.070 INFO:tasks.workunit.client.0.smithi167.stdout:1/840: fsync d0/d10/d5a/d33/f55 0 2022-01-31T19:44:17.070 INFO:tasks.workunit.client.0.smithi167.stdout:1/841: read d0/d11/fd8 [21307,26501] 0 2022-01-31T19:44:17.071 INFO:tasks.workunit.client.0.smithi167.stdout:5/944: dread d5/d6/de/d22/d76/da3/ffc [0,4194304] 0 2022-01-31T19:44:17.071 INFO:tasks.workunit.client.0.smithi167.stdout:2/594: rename dd/c12 to dd/df/d1b/d3d/d4a/cbe 0 2022-01-31T19:44:17.071 INFO:tasks.workunit.client.0.smithi167.stdout:6/636: dwrite d8/d33/d4f/f59 [0,4194304] 0 2022-01-31T19:44:17.072 INFO:tasks.workunit.client.0.smithi167.stdout:7/927: creat d8/dab/d13a/f150 x:0 0 0 2022-01-31T19:44:17.072 INFO:tasks.workunit.client.0.smithi167.stdout:1/842: mknod d0/d10/d5a/d33/d4b/d1f/c119 0 2022-01-31T19:44:17.073 INFO:tasks.workunit.client.0.smithi167.stdout:5/945: unlink d5/f118 0 2022-01-31T19:44:17.073 INFO:tasks.workunit.client.0.smithi167.stdout:9/702: rename dd/d35/f9a to dd/fee 0 2022-01-31T19:44:17.073 INFO:tasks.workunit.client.0.smithi167.stdout:2/595: getdents dd/df/d1b/d3d/d4a/d4b 0 2022-01-31T19:44:17.075 INFO:tasks.workunit.client.0.smithi167.stdout:6/637: unlink d8/d33/d51/d5d/d64/dab/fae 0 2022-01-31T19:44:17.075 INFO:tasks.workunit.client.0.smithi167.stdout:6/638: write d8/d33/d51/d7e/dc4/f9e [252854,103296] 0 2022-01-31T19:44:17.075 INFO:tasks.workunit.client.0.smithi167.stdout:7/928: mknod d8/d2c/d6e/c151 0 2022-01-31T19:44:17.075 INFO:tasks.workunit.client.0.smithi167.stdout:1/843: creat d0/d62/f11a x:0 0 0 2022-01-31T19:44:17.076 INFO:tasks.workunit.client.0.smithi167.stdout:1/844: fsync d0/d62/d7f/d82/fe2 0 2022-01-31T19:44:17.076 INFO:tasks.workunit.client.0.smithi167.stdout:1/845: dread - d0/d10/d5a/d33/d4b/da5/da9/dad/f108 zero size 2022-01-31T19:44:17.076 INFO:tasks.workunit.client.0.smithi167.stdout:2/596: dread dd/d18/f3a [0,4194304] 0 2022-01-31T19:44:17.076 INFO:tasks.workunit.client.0.smithi167.stdout:5/946: mkdir d5/d6/d12b 0 2022-01-31T19:44:17.077 INFO:tasks.workunit.client.0.smithi167.stdout:5/947: fdatasync d5/d6/d16/d18/d43/d45/fec 0 2022-01-31T19:44:17.077 INFO:tasks.workunit.client.0.smithi167.stdout:7/929: creat d8/d2c/d6e/f152 x:0 0 0 2022-01-31T19:44:17.077 INFO:tasks.workunit.client.0.smithi167.stdout:7/930: write d8/d2c/d6e/d4d/d67/d8f/f12b [1500667,123902] 0 2022-01-31T19:44:17.078 INFO:tasks.workunit.client.0.smithi167.stdout:2/597: creat dd/fbf x:0 0 0 2022-01-31T19:44:17.078 INFO:tasks.workunit.client.0.smithi167.stdout:9/703: dread dd/d35/f65 [0,4194304] 0 2022-01-31T19:44:17.079 INFO:tasks.workunit.client.0.smithi167.stdout:5/948: write d5/d6/de/d22/f97 [2753203,17534] 0 2022-01-31T19:44:17.079 INFO:tasks.workunit.client.0.smithi167.stdout:8/777: rename d6/l54 to d6/db/d36/lfc 0 2022-01-31T19:44:17.081 INFO:tasks.workunit.client.0.smithi167.stdout:2/598: dread dd/fb6 [0,4194304] 0 2022-01-31T19:44:17.082 INFO:tasks.workunit.client.0.smithi167.stdout:2/599: fdatasync dd/df/d1b/d3d/d4a/da7/fbb 0 2022-01-31T19:44:17.084 INFO:tasks.workunit.client.0.smithi167.stdout:4/767: rename d0/d6/d7/d39/d93 to d0/d6/d7/d26/d40/d4c/dfe 0 2022-01-31T19:44:17.087 INFO:tasks.workunit.client.0.smithi167.stdout:8/778: rename d6/db/d36/la3 to d6/d1e/dcf/lfd 0 2022-01-31T19:44:17.088 INFO:tasks.workunit.client.0.smithi167.stdout:1/846: write d0/f1b [1130597,14168] 0 2022-01-31T19:44:17.089 INFO:tasks.workunit.client.0.smithi167.stdout:4/768: dread d0/d6/d7/d26/d31/ddf/f29 [0,4194304] 0 2022-01-31T19:44:17.089 INFO:tasks.workunit.client.0.smithi167.stdout:4/769: creat d0/d6/d7/d26/d31/ddf/da2/dcf/db5/fff x:0 0 0 2022-01-31T19:44:17.090 INFO:tasks.workunit.client.0.smithi167.stdout:4/770: write d0/d6/d7/d39/d3a/fef [773600,9114] 0 2022-01-31T19:44:17.093 INFO:tasks.workunit.client.0.smithi167.stdout:8/779: creat d6/d1e/d2d/d6b/ffe x:0 0 0 2022-01-31T19:44:17.093 INFO:tasks.workunit.client.0.smithi167.stdout:9/704: dread dd/d35/f44 [0,4194304] 0 2022-01-31T19:44:17.097 INFO:tasks.workunit.client.0.smithi167.stdout:7/931: dread f6 [4194304,4194304] 0 2022-01-31T19:44:17.098 INFO:tasks.workunit.client.0.smithi167.stdout:1/847: rmdir d0/d10 39 2022-01-31T19:44:17.098 INFO:tasks.workunit.client.0.smithi167.stdout:1/848: dread - d0/d10/d5a/d33/d4b/da5/da9/dad/f108 zero size 2022-01-31T19:44:17.100 INFO:tasks.workunit.client.0.smithi167.stdout:5/949: dread d5/d6/d16/d18/d5b/f6c [0,4194304] 0 2022-01-31T19:44:17.102 INFO:tasks.workunit.client.0.smithi167.stdout:6/639: dwrite f3 [0,4194304] 0 2022-01-31T19:44:17.102 INFO:tasks.workunit.client.0.smithi167.stdout:6/640: chown d8/d33/f5f 30653838 1 2022-01-31T19:44:17.103 INFO:tasks.workunit.client.1.smithi171.stdout:6/81: dwrite f9 [0,4194304] 0 2022-01-31T19:44:17.103 INFO:tasks.workunit.client.1.smithi171.stdout:6/82: write f2 [1701337,50794] 0 2022-01-31T19:44:17.106 INFO:tasks.workunit.client.0.smithi167.stdout:8/780: rename d6/db/d6a/c87 to d6/d1e/dcf/cff 0 2022-01-31T19:44:17.106 INFO:tasks.workunit.client.0.smithi167.stdout:7/932: mknod d8/d2c/d6e/c153 0 2022-01-31T19:44:17.107 INFO:tasks.workunit.client.0.smithi167.stdout:5/950: getdents d5/d6/de/d22 0 2022-01-31T19:44:17.108 INFO:tasks.workunit.client.0.smithi167.stdout:5/951: chown d5/d6/de/fbd 0 1 2022-01-31T19:44:17.108 INFO:tasks.workunit.client.0.smithi167.stdout:7/933: write d8/d2c/d6e/f44 [3448178,49632] 0 2022-01-31T19:44:17.109 INFO:tasks.workunit.client.0.smithi167.stdout:6/641: symlink d8/d20/d54/led 0 2022-01-31T19:44:17.110 INFO:tasks.workunit.client.0.smithi167.stdout:6/642: fsync d8/d33/d51/d72/dc2/fd9 0 2022-01-31T19:44:17.110 INFO:tasks.workunit.client.0.smithi167.stdout:6/643: truncate d8/d33/d51/d72/f83 887329 0 2022-01-31T19:44:17.113 INFO:tasks.workunit.client.0.smithi167.stdout:4/771: dwrite d0/d6/d7/d26/d31/f71 [0,4194304] 0 2022-01-31T19:44:17.114 INFO:tasks.workunit.client.0.smithi167.stdout:5/952: rmdir d5/d6/d16/d18/d43/d45 39 2022-01-31T19:44:17.115 INFO:tasks.workunit.client.0.smithi167.stdout:7/934: creat d8/dc/d18/f154 x:0 0 0 2022-01-31T19:44:17.115 INFO:tasks.workunit.client.0.smithi167.stdout:7/935: readlink d8/d2c/d43/l92 0 2022-01-31T19:44:17.120 INFO:tasks.workunit.client.0.smithi167.stdout:5/953: dread d5/d6/d16/d18/f59 [0,4194304] 0 2022-01-31T19:44:17.121 INFO:tasks.workunit.client.0.smithi167.stdout:6/644: creat d8/d33/d51/d6a/fee x:0 0 0 2022-01-31T19:44:17.121 INFO:tasks.workunit.client.0.smithi167.stdout:6/645: fdatasync d8/d33/d51/d7e/dc4/f96 0 2022-01-31T19:44:17.121 INFO:tasks.workunit.client.0.smithi167.stdout:6/646: readlink d8/d33/d4f/l74 0 2022-01-31T19:44:17.121 INFO:tasks.workunit.client.0.smithi167.stdout:6/647: dread - d8/d33/d51/d5d/d64/dab/fe3 zero size 2022-01-31T19:44:17.137 INFO:tasks.workunit.client.0.smithi167.stdout:4/772: mknod d0/d6/d7/d26/d31/ddf/da2/c100 0 2022-01-31T19:44:17.137 INFO:tasks.workunit.client.0.smithi167.stdout:4/773: chown d0/d6/d7/d26/d31/ddf/f29 20915 1 2022-01-31T19:44:17.138 INFO:tasks.workunit.client.0.smithi167.stdout:4/774: stat d0/d6/dab/cd1 0 2022-01-31T19:44:17.138 INFO:tasks.workunit.client.0.smithi167.stdout:4/775: truncate d0/d6/d7/d39/fec 986933 0 2022-01-31T19:44:17.138 INFO:tasks.workunit.client.0.smithi167.stdout:4/776: write d0/d6/d7/d39/d4e/faa [339746,47203] 0 2022-01-31T19:44:17.139 INFO:tasks.workunit.client.0.smithi167.stdout:5/954: mknod d5/d6/d16/d18/d43/d45/dd5/dbc/dc0/c12c 0 2022-01-31T19:44:17.139 INFO:tasks.workunit.client.0.smithi167.stdout:5/955: dread - d5/d6/d16/d18/d43/d45/dd5/dbc/f116 zero size 2022-01-31T19:44:17.140 INFO:tasks.workunit.client.0.smithi167.stdout:5/956: chown d5/d6/d16/f9d 1158 1 2022-01-31T19:44:17.140 INFO:tasks.workunit.client.0.smithi167.stdout:9/705: rename dd/d17/d80/d4c to dd/d17/d7f/def 0 2022-01-31T19:44:17.143 INFO:tasks.workunit.client.0.smithi167.stdout:1/849: rename d0/d10/d20/ca8 to d0/d10/c11b 0 2022-01-31T19:44:17.145 INFO:tasks.workunit.client.0.smithi167.stdout:9/706: creat dd/d35/ff0 x:0 0 0 2022-01-31T19:44:17.146 INFO:tasks.workunit.client.0.smithi167.stdout:9/707: fdatasync dd/d17/d80/f54 0 2022-01-31T19:44:17.146 INFO:tasks.workunit.client.0.smithi167.stdout:8/781: rename d6/db/l62 to d6/db/d28/daf/dc4/l100 0 2022-01-31T19:44:17.147 INFO:tasks.workunit.client.0.smithi167.stdout:9/708: mknod dd/d6b/cf1 0 2022-01-31T19:44:17.150 INFO:tasks.workunit.client.0.smithi167.stdout:8/782: rename d6/db/d6a/d8d/d9e/d60/l7f to d6/db/d4e/dbd/l101 0 2022-01-31T19:44:17.150 INFO:tasks.workunit.client.0.smithi167.stdout:1/850: dread d0/d10/d5a/d33/d4b/da5/fe0 [0,4194304] 0 2022-01-31T19:44:17.151 INFO:tasks.workunit.client.0.smithi167.stdout:9/709: symlink dd/d17/d7f/def/db3/lf2 0 2022-01-31T19:44:17.151 INFO:tasks.workunit.client.0.smithi167.stdout:9/710: read - dd/d19/d59/fae zero size 2022-01-31T19:44:17.153 INFO:tasks.workunit.client.0.smithi167.stdout:8/783: unlink d6/db/de/d3d/dd6/fe0 0 2022-01-31T19:44:17.154 INFO:tasks.workunit.client.0.smithi167.stdout:4/777: dread d0/d6/d7/f4a [0,4194304] 0 2022-01-31T19:44:17.156 INFO:tasks.workunit.client.0.smithi167.stdout:1/851: unlink d0/d10/d18/fcc 0 2022-01-31T19:44:17.157 INFO:tasks.workunit.client.0.smithi167.stdout:9/711: creat dd/d19/d24/d56/d5b/ff3 x:0 0 0 2022-01-31T19:44:17.159 INFO:tasks.workunit.client.0.smithi167.stdout:1/852: mkdir d0/d10/d5a/d33/d4b/da5/dde/d11c 0 2022-01-31T19:44:17.160 INFO:tasks.workunit.client.0.smithi167.stdout:9/712: truncate dd/d17/f3e 1735182 0 2022-01-31T19:44:17.160 INFO:tasks.workunit.client.0.smithi167.stdout:9/713: truncate dd/d6b/f93 722318 0 2022-01-31T19:44:17.161 INFO:tasks.workunit.client.0.smithi167.stdout:9/714: write dd/d17/f3e [1934542,104398] 0 2022-01-31T19:44:17.161 INFO:tasks.workunit.client.0.smithi167.stdout:9/715: dread - dd/d19/d59/fae zero size 2022-01-31T19:44:17.161 INFO:tasks.workunit.client.0.smithi167.stdout:7/936: dwrite d8/fa7 [0,4194304] 0 2022-01-31T19:44:17.162 INFO:tasks.workunit.client.0.smithi167.stdout:6/648: dwrite d8/d33/f32 [0,4194304] 0 2022-01-31T19:44:17.163 INFO:tasks.workunit.client.0.smithi167.stdout:5/957: dwrite d5/d6/d3a/f4e [0,4194304] 0 2022-01-31T19:44:17.163 INFO:tasks.workunit.client.0.smithi167.stdout:5/958: readlink d5/d6/de/d22/lc1 0 2022-01-31T19:44:17.165 INFO:tasks.workunit.client.0.smithi167.stdout:9/716: stat dd/d19/d59/cd2 0 2022-01-31T19:44:17.165 INFO:tasks.workunit.client.0.smithi167.stdout:9/717: rename dd/d19/d24 to dd/d19/d24/df4 22 2022-01-31T19:44:17.169 INFO:tasks.workunit.client.0.smithi167.stdout:7/937: creat d8/d2c/d6e/d4d/d5e/dce/f155 x:0 0 0 2022-01-31T19:44:17.169 INFO:tasks.workunit.client.0.smithi167.stdout:7/938: fdatasync d8/dc/f21 0 2022-01-31T19:44:17.169 INFO:tasks.workunit.client.0.smithi167.stdout:7/939: chown d8/d2c/d43/df3/ff6 117955 1 2022-01-31T19:44:17.169 INFO:tasks.workunit.client.0.smithi167.stdout:7/940: chown d8/d2c/d6e/d4d/d67/d8f/c101 194159984 1 2022-01-31T19:44:17.170 INFO:tasks.workunit.client.0.smithi167.stdout:7/941: chown d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f1a 3 1 2022-01-31T19:44:17.170 INFO:tasks.workunit.client.0.smithi167.stdout:6/649: creat d8/d33/d51/fef x:0 0 0 2022-01-31T19:44:17.171 INFO:tasks.workunit.client.0.smithi167.stdout:6/650: truncate d8/d33/d51/d5d/f80 638130 0 2022-01-31T19:44:17.172 INFO:tasks.workunit.client.0.smithi167.stdout:5/959: getdents d5/d6/d16/d18/d43/d45/d69/d90/df9 0 2022-01-31T19:44:17.175 INFO:tasks.workunit.client.0.smithi167.stdout:6/651: link d8/f73 d8/d33/ff0 0 2022-01-31T19:44:17.175 INFO:tasks.workunit.client.0.smithi167.stdout:6/652: write d8/d33/d35/f61 [1006429,56390] 0 2022-01-31T19:44:17.175 INFO:tasks.workunit.client.0.smithi167.stdout:6/653: chown d8/d33/d51/d72/dad/le9 1634297624 1 2022-01-31T19:44:17.177 INFO:tasks.workunit.client.0.smithi167.stdout:6/654: mkdir d8/d33/d51/d6a/df1 0 2022-01-31T19:44:17.178 INFO:tasks.workunit.client.0.smithi167.stdout:1/853: dwrite d0/d10/d5a/d33/d4b/d1f/fa1 [0,4194304] 0 2022-01-31T19:44:17.178 INFO:tasks.workunit.client.0.smithi167.stdout:4/778: dwrite d0/d6/d7/d26/d31/d5c/fd7 [0,4194304] 0 2022-01-31T19:44:17.178 INFO:tasks.workunit.client.0.smithi167.stdout:5/960: dread d5/d6/d16/d18/f59 [0,4194304] 0 2022-01-31T19:44:17.179 INFO:tasks.workunit.client.0.smithi167.stdout:5/961: fdatasync d5/d6/f28 0 2022-01-31T19:44:17.179 INFO:tasks.workunit.client.0.smithi167.stdout:6/655: creat d8/d33/d51/ff2 x:0 0 0 2022-01-31T19:44:17.180 INFO:tasks.workunit.client.0.smithi167.stdout:6/656: dread - d8/d33/d51/d6a/fee zero size 2022-01-31T19:44:17.187 INFO:tasks.workunit.client.0.smithi167.stdout:1/854: symlink d0/d11/d102/l11d 0 2022-01-31T19:44:17.188 INFO:tasks.workunit.client.0.smithi167.stdout:1/855: chown d0/fe4 23880 1 2022-01-31T19:44:17.189 INFO:tasks.workunit.client.0.smithi167.stdout:4/779: symlink d0/d6/d7/d26/d31/ddf/d75/dd6/l101 0 2022-01-31T19:44:17.189 INFO:tasks.workunit.client.0.smithi167.stdout:4/780: write d0/d6/fc6 [4335216,116748] 0 2022-01-31T19:44:17.189 INFO:tasks.workunit.client.0.smithi167.stdout:7/942: dwrite d8/d2c/d6e/f152 [0,4194304] 0 2022-01-31T19:44:17.193 INFO:tasks.workunit.client.0.smithi167.stdout:6/657: rmdir d8/d33 39 2022-01-31T19:44:17.194 INFO:tasks.workunit.client.0.smithi167.stdout:4/781: rmdir d0/d6/d7/d26/d31/ddf/d75/dd6 39 2022-01-31T19:44:17.196 INFO:tasks.workunit.client.0.smithi167.stdout:7/943: mkdir d8/d2c/d6e/d4d/d5e/d71/d156 0 2022-01-31T19:44:17.197 INFO:tasks.workunit.client.0.smithi167.stdout:7/944: fdatasync d8/d2c/d6e/d4d/d5e/fd7 0 2022-01-31T19:44:17.199 INFO:tasks.workunit.client.0.smithi167.stdout:5/962: rename d5/d6/de/d22/d4d/f87 to d5/d6/d16/d18/d43/d45/d69/dc4/f12d 0 2022-01-31T19:44:17.200 INFO:tasks.workunit.client.0.smithi167.stdout:4/782: getdents d0/d6/d7/d26/d31/ddf/da2/dcf 0 2022-01-31T19:44:17.203 INFO:tasks.workunit.client.0.smithi167.stdout:5/963: link d5/d6/d16/d18/d43/d45/d69/dc4/fb4 d5/d6/d16/d18/d43/d45/d69/d90/df9/f12e 0 2022-01-31T19:44:17.204 INFO:tasks.workunit.client.0.smithi167.stdout:4/783: mkdir d0/d6/d7/d39/d4e/d61/d102 0 2022-01-31T19:44:17.205 INFO:tasks.workunit.client.0.smithi167.stdout:5/964: truncate d5/d6/d16/d18/d5b/fc2 1852843 0 2022-01-31T19:44:17.207 INFO:tasks.workunit.client.0.smithi167.stdout:4/784: truncate d0/d6/d7/d39/d3a/f54 3070547 0 2022-01-31T19:44:17.208 INFO:tasks.workunit.client.0.smithi167.stdout:5/965: symlink d5/d6/de/d22/d6d/l12f 0 2022-01-31T19:44:17.209 INFO:tasks.workunit.client.0.smithi167.stdout:4/785: link d0/d6/d7/d89/ld0 d0/d6/d7/d89/db1/l103 0 2022-01-31T19:44:17.219 INFO:tasks.workunit.client.0.smithi167.stdout:6/658: dwrite d8/d33/d51/d5d/d64/f88 [0,4194304] 0 2022-01-31T19:44:17.220 INFO:tasks.workunit.client.0.smithi167.stdout:7/945: dwrite d8/d2c/d43/f97 [0,4194304] 0 2022-01-31T19:44:17.221 INFO:tasks.workunit.client.1.smithi171.stdout:5/49: sync 2022-01-31T19:44:17.222 INFO:tasks.workunit.client.1.smithi171.stdout:1/14: sync 2022-01-31T19:44:17.222 INFO:tasks.workunit.client.1.smithi171.stdout:1/15: write - no filename 2022-01-31T19:44:17.222 INFO:tasks.workunit.client.1.smithi171.stdout:4/36: sync 2022-01-31T19:44:17.223 INFO:tasks.workunit.client.0.smithi167.stdout:7/946: write d8/dc/f21 [7530887,17311] 0 2022-01-31T19:44:17.224 INFO:tasks.workunit.client.0.smithi167.stdout:5/966: dread d5/d6/de/d22/d76/db0/fb6 [0,4194304] 0 2022-01-31T19:44:17.224 INFO:tasks.workunit.client.0.smithi167.stdout:6/659: dread - d8/d33/d51/d72/dc2/fce zero size 2022-01-31T19:44:17.225 INFO:tasks.workunit.client.1.smithi171.stdout:7/50: sync 2022-01-31T19:44:17.225 INFO:tasks.workunit.client.1.smithi171.stdout:9/14: sync 2022-01-31T19:44:17.225 INFO:tasks.workunit.client.1.smithi171.stdout:3/70: sync 2022-01-31T19:44:17.225 INFO:tasks.workunit.client.1.smithi171.stdout:7/51: write d1/d2/d7/fe [492307,51547] 0 2022-01-31T19:44:17.226 INFO:tasks.workunit.client.1.smithi171.stdout:1/16: rename c0 to c1 0 2022-01-31T19:44:17.226 INFO:tasks.workunit.client.1.smithi171.stdout:1/17: truncate - no filename 2022-01-31T19:44:17.226 INFO:tasks.workunit.client.1.smithi171.stdout:1/18: fdatasync - no filename 2022-01-31T19:44:17.227 INFO:tasks.workunit.client.1.smithi171.stdout:1/19: fsync - no filename 2022-01-31T19:44:17.227 INFO:tasks.workunit.client.1.smithi171.stdout:1/20: chown c1 0 1 2022-01-31T19:44:17.227 INFO:tasks.workunit.client.1.smithi171.stdout:1/21: dread - no filename 2022-01-31T19:44:17.227 INFO:tasks.workunit.client.1.smithi171.stdout:5/50: symlink d5/lc 0 2022-01-31T19:44:17.228 INFO:tasks.workunit.client.1.smithi171.stdout:3/71: chown d4/c6 29 1 2022-01-31T19:44:17.229 INFO:tasks.workunit.client.1.smithi171.stdout:5/51: mkdir d5/d6/dd 0 2022-01-31T19:44:17.229 INFO:tasks.workunit.client.1.smithi171.stdout:5/52: dread - d5/f8 zero size 2022-01-31T19:44:17.229 INFO:tasks.workunit.client.0.smithi167.stdout:7/947: mkdir d8/d2c/d7a/d99/de2/dc3/d157 0 2022-01-31T19:44:17.230 INFO:tasks.workunit.client.0.smithi167.stdout:1/856: dwrite d0/d10/d5a/d33/d4b/da5/da9/dad/f108 [0,4194304] 0 2022-01-31T19:44:17.230 INFO:tasks.workunit.client.0.smithi167.stdout:1/857: fsync d0/d75/d7a/f59 0 2022-01-31T19:44:17.230 INFO:tasks.workunit.client.0.smithi167.stdout:6/660: dread d8/d33/d51/d5d/f6e [0,4194304] 0 2022-01-31T19:44:17.230 INFO:tasks.workunit.client.0.smithi167.stdout:6/661: stat d8/d33/d51/d72/dad 0 2022-01-31T19:44:17.231 INFO:tasks.workunit.client.0.smithi167.stdout:7/948: unlink d8/dc/c11 0 2022-01-31T19:44:17.231 INFO:tasks.workunit.client.0.smithi167.stdout:7/949: creat d8/d2c/d43/d94/f158 x:0 0 0 2022-01-31T19:44:17.231 INFO:tasks.workunit.client.0.smithi167.stdout:7/950: chown d8/dab/d13a/f150 230712 1 2022-01-31T19:44:17.232 INFO:tasks.workunit.client.1.smithi171.stdout:3/72: dread d4/f9 [0,4194304] 0 2022-01-31T19:44:17.232 INFO:tasks.workunit.client.0.smithi167.stdout:1/858: creat d0/d11/d102/ddb/f11e x:0 0 0 2022-01-31T19:44:17.234 INFO:tasks.workunit.client.1.smithi171.stdout:3/73: symlink d4/dd/l15 0 2022-01-31T19:44:17.235 INFO:tasks.workunit.client.0.smithi167.stdout:6/662: truncate d8/d33/d51/d72/f95 551720 0 2022-01-31T19:44:17.235 INFO:tasks.workunit.client.0.smithi167.stdout:6/663: chown d8/d33/d4f/f8e 222 1 2022-01-31T19:44:17.236 INFO:tasks.workunit.client.1.smithi171.stdout:3/74: unlink d4/c8 0 2022-01-31T19:44:17.237 INFO:tasks.workunit.client.0.smithi167.stdout:4/786: dwrite d0/d6/d7/d26/dd2/de5/fbf [0,4194304] 0 2022-01-31T19:44:17.238 INFO:tasks.workunit.client.1.smithi171.stdout:3/75: mknod d4/dd/c16 0 2022-01-31T19:44:17.238 INFO:tasks.workunit.client.1.smithi171.stdout:3/76: readlink d4/dd/l15 0 2022-01-31T19:44:17.243 INFO:tasks.workunit.client.0.smithi167.stdout:7/951: link d8/d2c/d6e/d9a/l145 d8/dc/dc1/d124/l159 0 2022-01-31T19:44:17.243 INFO:tasks.workunit.client.0.smithi167.stdout:7/952: chown d8/d2c/d6e/d4d/d5e/f91 80296592 1 2022-01-31T19:44:17.243 INFO:tasks.workunit.client.0.smithi167.stdout:7/953: write d8/dc/d18/d14a/f135 [8582105,41571] 0 2022-01-31T19:44:17.244 INFO:tasks.workunit.client.1.smithi171.stdout:3/77: write d4/fc [3671544,5043] 0 2022-01-31T19:44:17.244 INFO:tasks.workunit.client.0.smithi167.stdout:1/859: mknod d0/d10/d5a/d33/d4b/da5/c11f 0 2022-01-31T19:44:17.248 INFO:tasks.workunit.client.0.smithi167.stdout:7/954: getdents d8/dab 0 2022-01-31T19:44:17.251 INFO:tasks.workunit.client.0.smithi167.stdout:1/860: rename d0/f17 to d0/d75/d7a/f120 0 2022-01-31T19:44:17.255 INFO:tasks.workunit.client.0.smithi167.stdout:4/787: dread d0/d6/d7/d26/d31/f5e [0,4194304] 0 2022-01-31T19:44:17.256 INFO:tasks.workunit.client.0.smithi167.stdout:4/788: creat d0/d6/d7/d89/db1/f104 x:0 0 0 2022-01-31T19:44:17.257 INFO:tasks.workunit.client.0.smithi167.stdout:4/789: chown d0/d6/d7/d26/dd2/de5/c9f 3 1 2022-01-31T19:44:17.257 INFO:tasks.workunit.client.0.smithi167.stdout:4/790: read - d0/fac zero size 2022-01-31T19:44:17.258 INFO:tasks.workunit.client.0.smithi167.stdout:1/861: write d0/d10/d5a/fb5 [647680,97892] 0 2022-01-31T19:44:17.258 INFO:tasks.workunit.client.0.smithi167.stdout:1/862: fdatasync d0/d10/d5a/df2/f103 0 2022-01-31T19:44:17.258 INFO:tasks.workunit.client.0.smithi167.stdout:4/791: symlink d0/d6/d7/d26/d31/d5c/l105 0 2022-01-31T19:44:17.259 INFO:tasks.workunit.client.0.smithi167.stdout:4/792: readlink d0/d6/d7/d26/d50/l5f 0 2022-01-31T19:44:17.259 INFO:tasks.workunit.client.0.smithi167.stdout:4/793: write d0/d6/d7/d26/d40/d4c/f86 [38966,86505] 0 2022-01-31T19:44:17.259 INFO:tasks.workunit.client.0.smithi167.stdout:1/863: creat d0/d10/f121 x:0 0 0 2022-01-31T19:44:17.261 INFO:tasks.workunit.client.0.smithi167.stdout:4/794: unlink d0/d6/d7/d26/d31/ddf/da2/dcf/c8c 0 2022-01-31T19:44:17.263 INFO:tasks.workunit.client.0.smithi167.stdout:4/795: creat d0/d6/d7/d26/d31/ddf/d75/dd6/f106 x:0 0 0 2022-01-31T19:44:17.263 INFO:tasks.workunit.client.0.smithi167.stdout:4/796: creat d0/d6/d7/d26/dd2/f107 x:0 0 0 2022-01-31T19:44:17.264 INFO:tasks.workunit.client.0.smithi167.stdout:4/797: truncate d0/d6/d7/d26/d40/d4c/f4f 1281668 0 2022-01-31T19:44:17.264 INFO:tasks.workunit.client.0.smithi167.stdout:4/798: rename d0/d6/d7/d89/db1 to d0/d6/d7/d89/db1/df7/d108 22 2022-01-31T19:44:17.266 INFO:tasks.workunit.client.0.smithi167.stdout:7/955: dread d8/d2c/d6e/d4d/d67/ff5 [0,4194304] 0 2022-01-31T19:44:17.266 INFO:tasks.workunit.client.0.smithi167.stdout:4/799: mknod d0/d6/d7/d26/d31/d5c/c109 0 2022-01-31T19:44:17.267 INFO:tasks.workunit.client.1.smithi171.stdout:7/52: dwrite d1/d2/fc [0,4194304] 0 2022-01-31T19:44:17.267 INFO:tasks.workunit.client.1.smithi171.stdout:4/37: dwrite d2/d3/d5/f6 [0,4194304] 0 2022-01-31T19:44:17.269 INFO:tasks.workunit.client.0.smithi167.stdout:5/967: dwrite d5/f7 [8388608,4194304] 0 2022-01-31T19:44:17.269 INFO:tasks.workunit.client.0.smithi167.stdout:5/968: chown d5/d6/d16/d18/d43/d45/dd5/dbc 0 1 2022-01-31T19:44:17.270 INFO:tasks.workunit.client.1.smithi171.stdout:4/38: symlink d2/d3/lb 0 2022-01-31T19:44:17.271 INFO:tasks.workunit.client.1.smithi171.stdout:4/39: write f0 [3959954,16627] 0 2022-01-31T19:44:17.272 INFO:tasks.workunit.client.1.smithi171.stdout:5/53: dwrite f1 [0,4194304] 0 2022-01-31T19:44:17.272 INFO:tasks.workunit.client.0.smithi167.stdout:6/664: truncate d8/d33/d51/d5d/d64/f88 3060161 0 2022-01-31T19:44:17.273 INFO:tasks.workunit.client.0.smithi167.stdout:6/665: dread - d8/d33/d51/d5d/d86/fcc zero size 2022-01-31T19:44:17.275 INFO:tasks.workunit.client.1.smithi171.stdout:4/40: mkdir d2/d3/d5/dc 0 2022-01-31T19:44:17.275 INFO:tasks.workunit.client.0.smithi167.stdout:6/666: unlink d8/d20/f6b 0 2022-01-31T19:44:17.280 INFO:tasks.workunit.client.1.smithi171.stdout:4/41: link d2/d3/lb d2/ld 0 2022-01-31T19:44:17.281 INFO:tasks.workunit.client.0.smithi167.stdout:6/667: mknod d8/d33/d51/d5d/d64/cf3 0 2022-01-31T19:44:17.286 INFO:tasks.workunit.client.1.smithi171.stdout:7/53: dread d1/d2/fc [0,4194304] 0 2022-01-31T19:44:17.286 INFO:tasks.workunit.client.1.smithi171.stdout:7/54: unlink d1/l6 0 2022-01-31T19:44:17.286 INFO:tasks.workunit.client.1.smithi171.stdout:4/42: mknod d2/d3/d5/dc/ce 0 2022-01-31T19:44:17.287 INFO:tasks.workunit.client.0.smithi167.stdout:6/668: rename d8/d33/d4f/f65 to d8/d33/d51/ff4 0 2022-01-31T19:44:17.287 INFO:tasks.workunit.client.0.smithi167.stdout:6/669: chown d8/d33/f3c 15 1 2022-01-31T19:44:17.288 INFO:tasks.workunit.client.0.smithi167.stdout:6/670: write d8/d33/d51/d6a/fee [710465,101568] 0 2022-01-31T19:44:17.288 INFO:tasks.workunit.client.0.smithi167.stdout:6/671: chown d8/f73 0 1 2022-01-31T19:44:17.288 INFO:tasks.workunit.client.1.smithi171.stdout:4/43: symlink d2/d3/d5/lf 0 2022-01-31T19:44:17.289 INFO:tasks.workunit.client.0.smithi167.stdout:6/672: mknod d8/d33/d51/d72/cf5 0 2022-01-31T19:44:17.290 INFO:tasks.workunit.client.1.smithi171.stdout:4/44: creat d2/d3/f10 x:0 0 0 2022-01-31T19:44:17.290 INFO:tasks.workunit.client.1.smithi171.stdout:4/45: chown d2/c9 59070678 1 2022-01-31T19:44:17.290 INFO:tasks.workunit.client.1.smithi171.stdout:4/46: truncate d2/d3/f10 649981 0 2022-01-31T19:44:17.302 INFO:tasks.workunit.client.1.smithi171.stdout:4/47: write f0 [3789188,53036] 0 2022-01-31T19:44:17.303 INFO:tasks.workunit.client.1.smithi171.stdout:5/54: dwrite d5/f8 [0,4194304] 0 2022-01-31T19:44:17.303 INFO:tasks.workunit.client.1.smithi171.stdout:4/48: dread d2/d3/f7 [0,4194304] 0 2022-01-31T19:44:17.304 INFO:tasks.workunit.client.1.smithi171.stdout:4/49: fdatasync f1 0 2022-01-31T19:44:17.306 INFO:tasks.workunit.client.1.smithi171.stdout:4/50: rename d2/d3/da to d2/d3/d5/d11 0 2022-01-31T19:44:17.307 INFO:tasks.workunit.client.0.smithi167.stdout:5/969: dwrite d5/d6/d16/d18/d5b/f110 [0,4194304] 0 2022-01-31T19:44:17.307 INFO:tasks.workunit.client.0.smithi167.stdout:5/970: stat d5/d125 0 2022-01-31T19:44:17.307 INFO:tasks.workunit.client.0.smithi167.stdout:5/971: creat d5/d6/d3a/f130 x:0 0 0 2022-01-31T19:44:17.308 INFO:tasks.workunit.client.1.smithi171.stdout:4/51: dread d2/d3/f7 [0,4194304] 0 2022-01-31T19:44:17.309 INFO:tasks.workunit.client.1.smithi171.stdout:4/52: mkdir d2/d3/d5/d11/d12 0 2022-01-31T19:44:17.311 INFO:tasks.workunit.client.1.smithi171.stdout:4/53: creat d2/f13 x:0 0 0 2022-01-31T19:44:17.311 INFO:tasks.workunit.client.1.smithi171.stdout:4/54: write d2/f13 [300033,22921] 0 2022-01-31T19:44:17.311 INFO:tasks.workunit.client.1.smithi171.stdout:4/55: creat d2/f14 x:0 0 0 2022-01-31T19:44:17.313 INFO:tasks.workunit.client.1.smithi171.stdout:4/56: mkdir d2/d3/d5/d11/d12/d15 0 2022-01-31T19:44:17.315 INFO:tasks.workunit.client.0.smithi167.stdout:6/673: dwrite d8/d33/d51/ff4 [0,4194304] 0 2022-01-31T19:44:17.315 INFO:tasks.workunit.client.0.smithi167.stdout:6/674: write d8/d20/f47 [299228,115866] 0 2022-01-31T19:44:17.316 INFO:tasks.workunit.client.1.smithi171.stdout:7/55: dwrite d1/d2/d7/ff [0,4194304] 0 2022-01-31T19:44:17.321 INFO:tasks.workunit.client.0.smithi167.stdout:5/972: dread d5/d6/d16/d18/d43/d45/f60 [0,4194304] 0 2022-01-31T19:44:17.322 INFO:tasks.workunit.client.0.smithi167.stdout:7/956: dwrite d8/dc/d18/d14a/f135 [0,4194304] 0 2022-01-31T19:44:17.324 INFO:tasks.workunit.client.0.smithi167.stdout:5/973: symlink d5/d6/de/d22/d6d/da0/da4/daa/l131 0 2022-01-31T19:44:17.324 INFO:tasks.workunit.client.0.smithi167.stdout:5/974: chown d5/d6/d16/d18/d43/d45/d69/dc4/dbb 15536 1 2022-01-31T19:44:17.325 INFO:tasks.workunit.client.0.smithi167.stdout:5/975: mknod d5/d6/de/c132 0 2022-01-31T19:44:17.327 INFO:tasks.workunit.client.0.smithi167.stdout:5/976: mknod d5/d6/d3a/c133 0 2022-01-31T19:44:17.334 INFO:tasks.workunit.client.0.smithi167.stdout:5/977: fdatasync d5/d6/d3a/f130 0 2022-01-31T19:44:17.335 INFO:tasks.workunit.client.0.smithi167.stdout:5/978: creat d5/d6/d16/d18/d43/d45/d69/dc4/f134 x:0 0 0 2022-01-31T19:44:17.338 INFO:tasks.workunit.client.1.smithi171.stdout:4/57: dwrite d2/f14 [0,4194304] 0 2022-01-31T19:44:17.338 INFO:tasks.workunit.client.1.smithi171.stdout:4/58: rename d2 to d2/d3/d5/dc/d16 22 2022-01-31T19:44:17.340 INFO:tasks.workunit.client.1.smithi171.stdout:4/59: symlink d2/d3/d5/d11/d12/l17 0 2022-01-31T19:44:17.342 INFO:tasks.workunit.client.1.smithi171.stdout:9/15: dwrite f4 [4194304,4194304] 0 2022-01-31T19:44:17.347 INFO:tasks.workunit.client.1.smithi171.stdout:1/22: getdents . 0 2022-01-31T19:44:17.348 INFO:tasks.workunit.client.0.smithi167.stdout:1/864: dread d0/d75/d7a/f120 [0,4194304] 0 2022-01-31T19:44:17.351 INFO:tasks.workunit.client.0.smithi167.stdout:6/675: dwrite d8/d33/d51/d72/f75 [0,4194304] 0 2022-01-31T19:44:17.352 INFO:tasks.workunit.client.1.smithi171.stdout:5/55: getdents d5 0 2022-01-31T19:44:17.352 INFO:tasks.workunit.client.1.smithi171.stdout:7/56: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:44:17.352 INFO:tasks.workunit.client.1.smithi171.stdout:5/56: creat d5/fe x:0 0 0 2022-01-31T19:44:17.353 INFO:tasks.workunit.client.1.smithi171.stdout:7/57: mkdir d1/d2/d12 0 2022-01-31T19:44:17.353 INFO:tasks.workunit.client.1.smithi171.stdout:7/58: readlink d1/d2/ld 0 2022-01-31T19:44:17.353 INFO:tasks.workunit.client.0.smithi167.stdout:7/957: dwrite d8/dc/d18/d14a/f135 [0,4194304] 0 2022-01-31T19:44:17.354 INFO:tasks.workunit.client.0.smithi167.stdout:7/958: stat d8/d2c/d6e/d4d/d5e/d71/lfc 0 2022-01-31T19:44:17.354 INFO:tasks.workunit.client.0.smithi167.stdout:4/800: fsync d0/d6/fc6 0 2022-01-31T19:44:17.354 INFO:tasks.workunit.client.0.smithi167.stdout:1/865: link d0/d10/d5a/d33/d4b/fa2 d0/d10/d5a/d33/d4b/da5/da9/f122 0 2022-01-31T19:44:17.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:17 smithi171 conmon[35325]: audit 2022-01-31T19:44:16.012412+0000 mon.smithi167 ( 2022-01-31T19:44:17.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:17 smithi171 conmon[35325]: mon.0) 663 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:17.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:17 smithi171 conmon[35325]: cluster 2022-01-31T19:44:16.433186+0000 mgr.smithi171.asyxnc (mgr.24387) 20 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 1.4 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 85 MiB/s rd, 177 MiB/s wr, 258 op/s 2022-01-31T19:44:17.361 INFO:tasks.workunit.client.1.smithi171.stdout:7/59: getdents d1 0 2022-01-31T19:44:17.361 INFO:tasks.workunit.client.1.smithi171.stdout:7/60: stat d1/d2/d7 0 2022-01-31T19:44:17.362 INFO:tasks.workunit.client.1.smithi171.stdout:9/16: dread f4 [0,4194304] 0 2022-01-31T19:44:17.363 INFO:tasks.workunit.client.0.smithi167.stdout:7/959: chown d8/dc/f13f 559517916 1 2022-01-31T19:44:17.364 INFO:tasks.workunit.client.1.smithi171.stdout:7/61: mknod d1/d2/d12/c13 0 2022-01-31T19:44:17.365 INFO:tasks.workunit.client.0.smithi167.stdout:1/866: link d0/d62/db7/f10e d0/d62/f123 0 2022-01-31T19:44:17.366 INFO:tasks.workunit.client.1.smithi171.stdout:9/17: mknod c7 0 2022-01-31T19:44:17.367 INFO:tasks.workunit.client.1.smithi171.stdout:7/62: stat d1/d2/l4 0 2022-01-31T19:44:17.367 INFO:tasks.workunit.client.0.smithi167.stdout:4/801: dread d0/d6/f76 [4194304,4194304] 0 2022-01-31T19:44:17.368 INFO:tasks.workunit.client.0.smithi167.stdout:4/802: dread - d0/d6/d7/d39/d4e/d61/fcc zero size 2022-01-31T19:44:17.368 INFO:tasks.workunit.client.0.smithi167.stdout:4/803: truncate d0/d6/d7/d26/d31/d5c/fea 731903 0 2022-01-31T19:44:17.368 INFO:tasks.workunit.client.0.smithi167.stdout:4/804: chown d0/d6/d7/d26/d31/ddf/f2e 154565863 1 2022-01-31T19:44:17.370 INFO:tasks.workunit.client.1.smithi171.stdout:3/78: dwrite d4/fc [4194304,4194304] 0 2022-01-31T19:44:17.371 INFO:tasks.workunit.client.1.smithi171.stdout:4/60: dwrite d2/f13 [0,4194304] 0 2022-01-31T19:44:17.371 INFO:tasks.workunit.client.1.smithi171.stdout:7/63: rename d1/d2/d7/l11 to d1/d2/d12/l14 0 2022-01-31T19:44:17.372 INFO:tasks.workunit.client.1.smithi171.stdout:4/61: write d2/f14 [2300560,4147] 0 2022-01-31T19:44:17.372 INFO:tasks.workunit.client.1.smithi171.stdout:4/62: readlink d2/d3/d5/lf 0 2022-01-31T19:44:17.376 INFO:tasks.workunit.client.1.smithi171.stdout:4/63: fdatasync d2/f14 0 2022-01-31T19:44:17.376 INFO:tasks.workunit.client.1.smithi171.stdout:4/64: write d2/d3/d5/f6 [4573609,110250] 0 2022-01-31T19:44:17.376 INFO:tasks.workunit.client.1.smithi171.stdout:4/65: stat d2/f13 0 2022-01-31T19:44:17.377 INFO:tasks.workunit.client.1.smithi171.stdout:7/64: symlink d1/d2/l15 0 2022-01-31T19:44:17.377 INFO:tasks.workunit.client.0.smithi167.stdout:6/676: mknod d8/d33/d51/cf6 0 2022-01-31T19:44:17.378 INFO:tasks.workunit.client.0.smithi167.stdout:6/677: chown d8/d33/d51/d6a/df1 142574 1 2022-01-31T19:44:17.378 INFO:tasks.workunit.client.0.smithi167.stdout:7/960: unlink d8/d2c/d6e/l10e 0 2022-01-31T19:44:17.379 INFO:tasks.workunit.client.0.smithi167.stdout:7/961: read - d8/d2c/d6e/f146 zero size 2022-01-31T19:44:17.380 INFO:tasks.workunit.client.0.smithi167.stdout:1/867: getdents d0/d10/d5a/d33/d4b/da5 0 2022-01-31T19:44:17.380 INFO:tasks.workunit.client.0.smithi167.stdout:1/868: stat d0/d10/d5a/d33/cfb 0 2022-01-31T19:44:17.380 INFO:tasks.workunit.client.0.smithi167.stdout:1/869: unlink d0/f1b 0 2022-01-31T19:44:17.380 INFO:tasks.workunit.client.0.smithi167.stdout:1/870: dread - d0/d10/d5a/d33/d4b/da5/dc1/f10c zero size 2022-01-31T19:44:17.381 INFO:tasks.workunit.client.0.smithi167.stdout:4/805: creat d0/d51/df6/f10a x:0 0 0 2022-01-31T19:44:17.382 INFO:tasks.workunit.client.0.smithi167.stdout:4/806: write d0/d6/d7/f4a [4011587,7985] 0 2022-01-31T19:44:17.382 INFO:tasks.workunit.client.0.smithi167.stdout:4/807: chown d0/d6/d7/d26/d31/d5c/de2 0 1 2022-01-31T19:44:17.383 INFO:tasks.workunit.client.1.smithi171.stdout:7/65: creat d1/f16 x:0 0 0 2022-01-31T19:44:17.383 INFO:tasks.workunit.client.1.smithi171.stdout:7/66: dread - d1/f16 zero size 2022-01-31T19:44:17.384 INFO:tasks.workunit.client.0.smithi167.stdout:6/678: creat d8/d33/d51/d5d/d64/ff7 x:0 0 0 2022-01-31T19:44:17.387 INFO:tasks.workunit.client.0.smithi167.stdout:7/962: write d8/dc/f2b [480297,91333] 0 2022-01-31T19:44:17.387 INFO:tasks.workunit.client.0.smithi167.stdout:7/963: fsync d8/d2c/d6e/d9a/f110 0 2022-01-31T19:44:17.388 INFO:tasks.workunit.client.0.smithi167.stdout:4/808: dread d0/d6/d7/d26/d40/d4c/f4f [0,4194304] 0 2022-01-31T19:44:17.390 INFO:tasks.workunit.client.1.smithi171.stdout:7/67: getdents d1/d2/d7 0 2022-01-31T19:44:17.392 INFO:tasks.workunit.client.1.smithi171.stdout:6/83: sync 2022-01-31T19:44:17.392 INFO:tasks.workunit.client.1.smithi171.stdout:2/124: sync 2022-01-31T19:44:17.392 INFO:tasks.workunit.client.1.smithi171.stdout:8/55: sync 2022-01-31T19:44:17.393 INFO:tasks.workunit.client.1.smithi171.stdout:2/125: fsync d7/f25 0 2022-01-31T19:44:17.393 INFO:tasks.workunit.client.1.smithi171.stdout:0/90: sync 2022-01-31T19:44:17.393 INFO:tasks.workunit.client.1.smithi171.stdout:6/84: creat dc/f15 x:0 0 0 2022-01-31T19:44:17.393 INFO:tasks.workunit.client.1.smithi171.stdout:9/18: dwrite f4 [0,4194304] 0 2022-01-31T19:44:17.393 INFO:tasks.workunit.client.1.smithi171.stdout:9/19: fsync f4 0 2022-01-31T19:44:17.394 INFO:tasks.workunit.client.1.smithi171.stdout:8/56: rmdir df 39 2022-01-31T19:44:17.395 INFO:tasks.workunit.client.0.smithi167.stdout:1/871: rename d0/d11/fb0 to d0/d62/db7/f124 0 2022-01-31T19:44:17.395 INFO:tasks.workunit.client.0.smithi167.stdout:3/824: sync 2022-01-31T19:44:17.395 INFO:tasks.workunit.client.0.smithi167.stdout:2/600: sync 2022-01-31T19:44:17.396 INFO:tasks.workunit.client.0.smithi167.stdout:0/535: sync 2022-01-31T19:44:17.396 INFO:tasks.workunit.client.0.smithi167.stdout:0/536: chown l12 13939 1 2022-01-31T19:44:17.396 INFO:tasks.workunit.client.0.smithi167.stdout:0/537: readlink d17/d19/l20 0 2022-01-31T19:44:17.396 INFO:tasks.workunit.client.0.smithi167.stdout:6/679: symlink d8/d33/d51/d5d/lf8 0 2022-01-31T19:44:17.397 INFO:tasks.workunit.client.0.smithi167.stdout:6/680: fsync d8/d62/fc6 0 2022-01-31T19:44:17.398 INFO:tasks.workunit.client.1.smithi171.stdout:9/20: link l6 l8 0 2022-01-31T19:44:17.399 INFO:tasks.workunit.client.0.smithi167.stdout:7/964: mkdir d8/d2c/d6e/d4d/d5e/d71/d156/d15a 0 2022-01-31T19:44:17.399 INFO:tasks.workunit.client.0.smithi167.stdout:1/872: mknod d0/d11/d102/ddb/c125 0 2022-01-31T19:44:17.399 INFO:tasks.workunit.client.0.smithi167.stdout:1/873: chown d0/c9e 12197572 1 2022-01-31T19:44:17.399 INFO:tasks.workunit.client.0.smithi167.stdout:2/601: mknod dd/d18/d8d/cc0 0 2022-01-31T19:44:17.400 INFO:tasks.workunit.client.0.smithi167.stdout:2/602: creat dd/fc1 x:0 0 0 2022-01-31T19:44:17.400 INFO:tasks.workunit.client.0.smithi167.stdout:2/603: fsync dd/df/d22/f9d 0 2022-01-31T19:44:17.400 INFO:tasks.workunit.client.0.smithi167.stdout:3/825: creat d9/d29/d3b/d9b/da3/db0/ffa x:0 0 0 2022-01-31T19:44:17.401 INFO:tasks.workunit.client.0.smithi167.stdout:0/538: unlink d17/d19/d1c/d39/c7d 0 2022-01-31T19:44:17.403 INFO:tasks.workunit.client.1.smithi171.stdout:4/66: dwrite d2/f14 [0,4194304] 0 2022-01-31T19:44:17.403 INFO:tasks.workunit.client.1.smithi171.stdout:8/57: mknod df/c14 0 2022-01-31T19:44:17.403 INFO:tasks.workunit.client.1.smithi171.stdout:2/126: mkdir d7/dd/d27 0 2022-01-31T19:44:17.404 INFO:tasks.workunit.client.1.smithi171.stdout:2/127: chown d7/fa 5 1 2022-01-31T19:44:17.404 INFO:tasks.workunit.client.1.smithi171.stdout:2/128: getdents d7/dd/d27 0 2022-01-31T19:44:17.404 INFO:tasks.workunit.client.1.smithi171.stdout:0/91: write d9/f13 [3186584,111551] 0 2022-01-31T19:44:17.404 INFO:tasks.workunit.client.1.smithi171.stdout:6/85: write f9 [5165653,48160] 0 2022-01-31T19:44:17.405 INFO:tasks.workunit.client.1.smithi171.stdout:6/86: write f2 [4698797,58068] 0 2022-01-31T19:44:17.405 INFO:tasks.workunit.client.1.smithi171.stdout:6/87: write dc/f15 [537154,117945] 0 2022-01-31T19:44:17.405 INFO:tasks.workunit.client.1.smithi171.stdout:8/58: symlink df/l15 0 2022-01-31T19:44:17.405 INFO:tasks.workunit.client.1.smithi171.stdout:9/21: write f4 [806960,9836] 0 2022-01-31T19:44:17.405 INFO:tasks.workunit.client.1.smithi171.stdout:2/129: symlink d7/dd/d27/l28 0 2022-01-31T19:44:17.406 INFO:tasks.workunit.client.1.smithi171.stdout:2/130: fdatasync d7/f25 0 2022-01-31T19:44:17.406 INFO:tasks.workunit.client.1.smithi171.stdout:2/131: chown d7/d9/l10 28053 1 2022-01-31T19:44:17.406 INFO:tasks.workunit.client.1.smithi171.stdout:2/132: dread - d7/f25 zero size 2022-01-31T19:44:17.407 INFO:tasks.workunit.client.0.smithi167.stdout:1/874: rename d0/d62/d7f/d82/cce to d0/d75/dbd/c126 0 2022-01-31T19:44:17.408 INFO:tasks.workunit.client.0.smithi167.stdout:1/875: readlink d0/d62/l89 0 2022-01-31T19:44:17.408 INFO:tasks.workunit.client.0.smithi167.stdout:2/604: rmdir dd/df/d3f/d51/d5f 39 2022-01-31T19:44:17.408 INFO:tasks.workunit.client.0.smithi167.stdout:3/826: mknod d9/d29/d3b/d41/d6b/dd2/cfb 0 2022-01-31T19:44:17.409 INFO:tasks.workunit.client.1.smithi171.stdout:6/88: read f2 [536629,68292] 0 2022-01-31T19:44:17.409 INFO:tasks.workunit.client.1.smithi171.stdout:2/133: write d7/f14 [1741253,41149] 0 2022-01-31T19:44:17.409 INFO:tasks.workunit.client.1.smithi171.stdout:2/134: readlink d7/lc 0 2022-01-31T19:44:17.410 INFO:tasks.workunit.client.1.smithi171.stdout:0/92: link d9/c10 d9/db/c20 0 2022-01-31T19:44:17.410 INFO:tasks.workunit.client.1.smithi171.stdout:0/93: truncate f8 914966 0 2022-01-31T19:44:17.410 INFO:tasks.workunit.client.1.smithi171.stdout:6/89: rename dc/c10 to dc/c16 0 2022-01-31T19:44:17.410 INFO:tasks.workunit.client.1.smithi171.stdout:8/59: write df/f10 [2702990,13470] 0 2022-01-31T19:44:17.411 INFO:tasks.workunit.client.1.smithi171.stdout:2/135: symlink d7/dd/l29 0 2022-01-31T19:44:17.412 INFO:tasks.workunit.client.0.smithi167.stdout:7/965: rename d8/d2c/d6e/d4d/d5e/d71/db1/d4f/l13d to d8/dc/dc1/l15b 0 2022-01-31T19:44:17.412 INFO:tasks.workunit.client.0.smithi167.stdout:0/539: getdents d17/d33 0 2022-01-31T19:44:17.413 INFO:tasks.workunit.client.1.smithi171.stdout:0/94: getdents d9 0 2022-01-31T19:44:17.413 INFO:tasks.workunit.client.1.smithi171.stdout:0/95: chown d9/f1d 9 1 2022-01-31T19:44:17.413 INFO:tasks.workunit.client.1.smithi171.stdout:0/96: chown d9/db/f12 52529 1 2022-01-31T19:44:17.414 INFO:tasks.workunit.client.1.smithi171.stdout:4/67: dread f0 [0,4194304] 0 2022-01-31T19:44:17.414 INFO:tasks.workunit.client.1.smithi171.stdout:4/68: write f1 [3776563,57966] 0 2022-01-31T19:44:17.414 INFO:tasks.workunit.client.1.smithi171.stdout:4/69: write d2/d3/f10 [1624232,75929] 0 2022-01-31T19:44:17.414 INFO:tasks.workunit.client.1.smithi171.stdout:8/60: read f5 [4121926,17210] 0 2022-01-31T19:44:17.415 INFO:tasks.workunit.client.0.smithi167.stdout:1/876: link d0/d75/d7a/c5f d0/d62/d7f/d82/c127 0 2022-01-31T19:44:17.415 INFO:tasks.workunit.client.1.smithi171.stdout:4/70: rmdir d2/d3 39 2022-01-31T19:44:17.416 INFO:tasks.workunit.client.0.smithi167.stdout:0/540: creat d17/fad x:0 0 0 2022-01-31T19:44:17.417 INFO:tasks.workunit.client.0.smithi167.stdout:7/966: mkdir d8/dab/dd0/d15c 0 2022-01-31T19:44:17.417 INFO:tasks.workunit.client.1.smithi171.stdout:4/71: symlink d2/d3/d5/l18 0 2022-01-31T19:44:17.418 INFO:tasks.workunit.client.1.smithi171.stdout:4/72: stat d2/d3/d5/dc 0 2022-01-31T19:44:17.419 INFO:tasks.workunit.client.1.smithi171.stdout:4/73: read f0 [1743481,1825] 0 2022-01-31T19:44:17.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:17 smithi167 conmon[32206]: audit 2022-01-31T19:44:16.012412+0000 mon.smithi167 (mon.0) 663 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:17.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:17 smithi167 conmon[32206]: cluster 2022-01-31T19:44:16.433186+0000 mgr.smithi171.asyxnc (mgr.24387) 20 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 1.4 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 85 MiB/s rd, 177 MiB/s wr, 258 op/s 2022-01-31T19:44:17.422 INFO:tasks.workunit.client.0.smithi167.stdout:0/541: dread d17/d19/d1c/d39/d44/f93 [0,4194304] 0 2022-01-31T19:44:17.422 INFO:tasks.workunit.client.0.smithi167.stdout:7/967: rename d8/d2c/d6e/d9a/l11f to d8/dc/dc1/l15d 0 2022-01-31T19:44:17.423 INFO:tasks.workunit.client.0.smithi167.stdout:1/877: write d0/d10/d5a/d33/d4b/da5/da9/ffc [3682790,14341] 0 2022-01-31T19:44:17.423 INFO:tasks.workunit.client.0.smithi167.stdout:1/878: chown d0/d100/f115 3 1 2022-01-31T19:44:17.423 INFO:tasks.workunit.client.0.smithi167.stdout:1/879: write d0/d10/d5a/d33/d4b/da5/dc1/f10c [908464,101335] 0 2022-01-31T19:44:17.423 INFO:tasks.workunit.client.0.smithi167.stdout:1/880: creat d0/d11/d46/f128 x:0 0 0 2022-01-31T19:44:17.424 INFO:tasks.workunit.client.0.smithi167.stdout:1/881: chown d0/c7d 3872 1 2022-01-31T19:44:17.424 INFO:tasks.workunit.client.1.smithi171.stdout:9/22: fdatasync f4 0 2022-01-31T19:44:17.426 INFO:tasks.workunit.client.0.smithi167.stdout:1/882: mknod d0/d10/d5a/d33/d4b/da5/da9/dad/c129 0 2022-01-31T19:44:17.426 INFO:tasks.workunit.client.0.smithi167.stdout:1/883: dread - d0/d62/ffd zero size 2022-01-31T19:44:17.430 INFO:tasks.workunit.client.0.smithi167.stdout:1/884: link d0/f3a d0/d11/d102/f12a 0 2022-01-31T19:44:17.431 INFO:tasks.workunit.client.0.smithi167.stdout:1/885: chown d0/d10/d18/cbf 15371 1 2022-01-31T19:44:17.431 INFO:tasks.workunit.client.0.smithi167.stdout:3/827: write d9/d29/d60/fa0 [3542389,92832] 0 2022-01-31T19:44:17.432 INFO:tasks.workunit.client.0.smithi167.stdout:0/542: dread d17/f22 [0,4194304] 0 2022-01-31T19:44:17.432 INFO:tasks.workunit.client.0.smithi167.stdout:0/543: chown d17/d2c/f79 9735128 1 2022-01-31T19:44:17.436 INFO:tasks.workunit.client.0.smithi167.stdout:1/886: mkdir d0/d10/d5a/d33/d4b/da5/d12b 0 2022-01-31T19:44:17.436 INFO:tasks.workunit.client.0.smithi167.stdout:1/887: fdatasync d0/d10/d5a/d33/d4b/da5/da9/dad/f108 0 2022-01-31T19:44:17.437 INFO:tasks.workunit.client.0.smithi167.stdout:3/828: mkdir d9/dfc 0 2022-01-31T19:44:17.437 INFO:tasks.workunit.client.0.smithi167.stdout:0/544: getdents d17/d19/d1c/d39/d44/d88/d98 0 2022-01-31T19:44:17.437 INFO:tasks.workunit.client.0.smithi167.stdout:0/545: creat d17/d3e/fae x:0 0 0 2022-01-31T19:44:17.437 INFO:tasks.workunit.client.0.smithi167.stdout:0/546: chown d17/d2c/f76 0 1 2022-01-31T19:44:17.438 INFO:tasks.workunit.client.0.smithi167.stdout:0/547: fdatasync d17/f7a 0 2022-01-31T19:44:17.438 INFO:tasks.workunit.client.0.smithi167.stdout:0/548: creat d17/d19/d1c/d39/d44/d88/faf x:0 0 0 2022-01-31T19:44:17.438 INFO:tasks.workunit.client.0.smithi167.stdout:3/829: rmdir d9/d29 39 2022-01-31T19:44:17.439 INFO:tasks.workunit.client.1.smithi171.stdout:7/68: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:44:17.439 INFO:tasks.workunit.client.0.smithi167.stdout:4/809: dwrite d0/d6/d7/d89/fad [0,4194304] 0 2022-01-31T19:44:17.440 INFO:tasks.workunit.client.0.smithi167.stdout:1/888: write d0/d10/d5a/d33/d4b/da5/da9/ffc [3804417,32686] 0 2022-01-31T19:44:17.441 INFO:tasks.workunit.client.0.smithi167.stdout:0/549: getdents d17/d19 0 2022-01-31T19:44:17.441 INFO:tasks.workunit.client.0.smithi167.stdout:0/550: chown f8 832576 1 2022-01-31T19:44:17.444 INFO:tasks.workunit.client.1.smithi171.stdout:2/136: dread d7/f14 [0,4194304] 0 2022-01-31T19:44:17.445 INFO:tasks.workunit.client.1.smithi171.stdout:2/137: stat d7/dd/c21 0 2022-01-31T19:44:17.446 INFO:tasks.workunit.client.0.smithi167.stdout:1/889: mkdir d0/d10/d12c 0 2022-01-31T19:44:17.447 INFO:tasks.workunit.client.1.smithi171.stdout:2/138: unlink d7/f1c 0 2022-01-31T19:44:17.449 INFO:tasks.workunit.client.1.smithi171.stdout:2/139: mknod d7/d26/c2a 0 2022-01-31T19:44:17.449 INFO:tasks.workunit.client.1.smithi171.stdout:2/140: truncate d7/f11 539987 0 2022-01-31T19:44:17.449 INFO:tasks.workunit.client.1.smithi171.stdout:2/141: read - d7/d9/f1d zero size 2022-01-31T19:44:17.449 INFO:tasks.workunit.client.1.smithi171.stdout:2/142: readlink d7/dd/d27/l28 0 2022-01-31T19:44:17.450 INFO:tasks.workunit.client.1.smithi171.stdout:7/69: dread d1/d2/f10 [0,4194304] 0 2022-01-31T19:44:17.452 INFO:tasks.workunit.client.0.smithi167.stdout:4/810: write d0/d6/d7/d26/dd2/de5/f91 [3618343,4063] 0 2022-01-31T19:44:17.453 INFO:tasks.workunit.client.0.smithi167.stdout:3/830: dread d9/d29/dbd/fd8 [0,4194304] 0 2022-01-31T19:44:17.453 INFO:tasks.workunit.client.0.smithi167.stdout:3/831: write d9/d29/d3b/d9b/da3/dd9/fe3 [517991,57738] 0 2022-01-31T19:44:17.453 INFO:tasks.workunit.client.0.smithi167.stdout:3/832: chown d9/d29/d3b/d9b/da3/dd9/def 3087 1 2022-01-31T19:44:17.453 INFO:tasks.workunit.client.0.smithi167.stdout:3/833: chown d9/d29/d60/c9a 961449 1 2022-01-31T19:44:17.456 INFO:tasks.workunit.client.0.smithi167.stdout:4/811: rename d0/d6/d7/d26/f72 to d0/d6/d7/d26/d40/d4c/f10b 0 2022-01-31T19:44:17.456 INFO:tasks.workunit.client.0.smithi167.stdout:4/812: creat d0/d6/d7/d39/d4e/f10c x:0 0 0 2022-01-31T19:44:17.457 INFO:tasks.workunit.client.0.smithi167.stdout:4/813: write d0/d6/d7/d26/d31/ddf/f2e [4492719,56222] 0 2022-01-31T19:44:17.459 INFO:tasks.workunit.client.0.smithi167.stdout:3/834: rename d9/d26/fe9 to d9/d7f/da8/ffd 0 2022-01-31T19:44:17.460 INFO:tasks.workunit.client.1.smithi171.stdout:3/79: dwrite d4/fb [4194304,4194304] 0 2022-01-31T19:44:17.460 INFO:tasks.workunit.client.1.smithi171.stdout:3/80: chown d4/dd/c16 112715560 1 2022-01-31T19:44:17.461 INFO:tasks.workunit.client.1.smithi171.stdout:3/81: unlink d4/f9 0 2022-01-31T19:44:17.461 INFO:tasks.workunit.client.1.smithi171.stdout:9/23: getdents . 0 2022-01-31T19:44:17.463 INFO:tasks.workunit.client.0.smithi167.stdout:4/814: rename d0/d6/d7/d26/dd2/de5/fbf to d0/d6/d7/d89/f10d 0 2022-01-31T19:44:17.463 INFO:tasks.workunit.client.1.smithi171.stdout:3/82: read d4/fb [6151333,118471] 0 2022-01-31T19:44:17.463 INFO:tasks.workunit.client.1.smithi171.stdout:3/83: chown d4/c6 35 1 2022-01-31T19:44:17.464 INFO:tasks.workunit.client.0.smithi167.stdout:2/605: dwrite dd/df/d1b/d3d/d4a/da7/fbb [0,4194304] 0 2022-01-31T19:44:17.467 INFO:tasks.workunit.client.0.smithi167.stdout:2/606: mknod dd/d18/d62/cc2 0 2022-01-31T19:44:17.467 INFO:tasks.workunit.client.0.smithi167.stdout:2/607: readlink dd/df/d1b/d3d/d4a/d4b/d54/l88 0 2022-01-31T19:44:17.468 INFO:tasks.workunit.client.0.smithi167.stdout:2/608: rmdir dd/df/d22 39 2022-01-31T19:44:17.468 INFO:tasks.workunit.client.1.smithi171.stdout:2/143: dread d7/d9/f23 [0,4194304] 0 2022-01-31T19:44:17.471 INFO:tasks.workunit.client.1.smithi171.stdout:9/24: creat f9 x:0 0 0 2022-01-31T19:44:17.473 INFO:tasks.workunit.client.1.smithi171.stdout:9/25: chown f4 16 1 2022-01-31T19:44:17.474 INFO:tasks.workunit.client.1.smithi171.stdout:3/84: truncate d4/fa 259941 0 2022-01-31T19:44:17.475 INFO:tasks.workunit.client.0.smithi167.stdout:6/681: dwrite d8/d33/f5f [4194304,4194304] 0 2022-01-31T19:44:17.475 INFO:tasks.workunit.client.1.smithi171.stdout:0/97: dwrite f4 [0,4194304] 0 2022-01-31T19:44:17.476 INFO:tasks.workunit.client.1.smithi171.stdout:6/90: dwrite fa [0,4194304] 0 2022-01-31T19:44:17.476 INFO:tasks.workunit.client.1.smithi171.stdout:6/91: creat dc/f17 x:0 0 0 2022-01-31T19:44:17.477 INFO:tasks.workunit.client.1.smithi171.stdout:3/85: mknod d4/dd/c17 0 2022-01-31T19:44:17.477 INFO:tasks.workunit.client.0.smithi167.stdout:7/968: dwrite d8/d2c/d6e/d4d/d67/ff5 [4194304,4194304] 0 2022-01-31T19:44:17.477 INFO:tasks.workunit.client.0.smithi167.stdout:4/815: dread d0/d6/d7/f7b [0,4194304] 0 2022-01-31T19:44:17.478 INFO:tasks.workunit.client.0.smithi167.stdout:4/816: read - d0/f7a zero size 2022-01-31T19:44:17.478 INFO:tasks.workunit.client.0.smithi167.stdout:6/682: rename d8/d33/d51/d5d/d86/ce8 to d8/d33/d51/d5d/d64/dab/db9/cf9 0 2022-01-31T19:44:17.479 INFO:tasks.workunit.client.0.smithi167.stdout:4/817: write d0/d6/d7/d26/f62 [1426182,100597] 0 2022-01-31T19:44:17.479 INFO:tasks.workunit.client.1.smithi171.stdout:2/144: dread d7/f13 [0,4194304] 0 2022-01-31T19:44:17.480 INFO:tasks.workunit.client.1.smithi171.stdout:4/74: dwrite d2/d3/f10 [0,4194304] 0 2022-01-31T19:44:17.480 INFO:tasks.workunit.client.1.smithi171.stdout:6/92: mkdir dc/d18 0 2022-01-31T19:44:17.481 INFO:tasks.workunit.client.1.smithi171.stdout:6/93: dread - dc/f13 zero size 2022-01-31T19:44:17.481 INFO:tasks.workunit.client.1.smithi171.stdout:6/94: unlink l6 0 2022-01-31T19:44:17.482 INFO:tasks.workunit.client.0.smithi167.stdout:7/969: dread d8/d2c/d6e/d4d/d5e/d71/db1/d8d/f68 [0,4194304] 0 2022-01-31T19:44:17.483 INFO:tasks.workunit.client.1.smithi171.stdout:6/95: mkdir dc/d19 0 2022-01-31T19:44:17.484 INFO:tasks.workunit.client.1.smithi171.stdout:6/96: chown dc/ce 291 1 2022-01-31T19:44:17.484 INFO:tasks.workunit.client.1.smithi171.stdout:7/70: dwrite d1/d2/d7/fe [0,4194304] 0 2022-01-31T19:44:17.485 INFO:tasks.workunit.client.1.smithi171.stdout:2/145: dread d7/d9/f23 [0,4194304] 0 2022-01-31T19:44:17.486 INFO:tasks.workunit.client.1.smithi171.stdout:2/146: stat d7/dd/c21 0 2022-01-31T19:44:17.486 INFO:tasks.workunit.client.1.smithi171.stdout:2/147: fdatasync d7/fa 0 2022-01-31T19:44:17.486 INFO:tasks.workunit.client.1.smithi171.stdout:2/148: fsync d7/fa 0 2022-01-31T19:44:17.486 INFO:tasks.workunit.client.1.smithi171.stdout:2/149: chown d7/dd 1327743 1 2022-01-31T19:44:17.487 INFO:tasks.workunit.client.1.smithi171.stdout:6/97: creat dc/d19/f1a x:0 0 0 2022-01-31T19:44:17.487 INFO:tasks.workunit.client.1.smithi171.stdout:6/98: fdatasync dc/d19/f1a 0 2022-01-31T19:44:17.488 INFO:tasks.workunit.client.1.smithi171.stdout:7/71: symlink d1/d2/d12/l17 0 2022-01-31T19:44:17.489 INFO:tasks.workunit.client.1.smithi171.stdout:4/75: dread f1 [0,4194304] 0 2022-01-31T19:44:17.490 INFO:tasks.workunit.client.1.smithi171.stdout:9/26: dread f4 [0,4194304] 0 2022-01-31T19:44:17.491 INFO:tasks.workunit.client.0.smithi167.stdout:4/818: write d0/d6/d7/d39/d4e/f52 [2627308,80836] 0 2022-01-31T19:44:17.492 INFO:tasks.workunit.client.1.smithi171.stdout:6/99: creat dc/d18/f1b x:0 0 0 2022-01-31T19:44:17.492 INFO:tasks.workunit.client.1.smithi171.stdout:7/72: creat d1/d2/d12/f18 x:0 0 0 2022-01-31T19:44:17.493 INFO:tasks.workunit.client.0.smithi167.stdout:0/551: rmdir d17/d3e 39 2022-01-31T19:44:17.497 INFO:tasks.workunit.client.0.smithi167.stdout:2/609: dwrite dd/df/d1b/d2f/f34 [0,4194304] 0 2022-01-31T19:44:17.498 INFO:tasks.workunit.client.0.smithi167.stdout:6/683: read - d8/d33/d51/d72/dc7/fdb zero size 2022-01-31T19:44:17.498 INFO:tasks.workunit.client.0.smithi167.stdout:6/684: fdatasync d8/d33/d51/d5d/f9f 0 2022-01-31T19:44:17.498 INFO:tasks.workunit.client.0.smithi167.stdout:2/610: fsync dd/df/d1b/d3d/d4a/fac 0 2022-01-31T19:44:17.498 INFO:tasks.workunit.client.0.smithi167.stdout:2/611: truncate dd/fbf 406561 0 2022-01-31T19:44:17.499 INFO:tasks.workunit.client.0.smithi167.stdout:7/970: creat d8/dc/d18/d14a/dee/df8/d132/f15e x:0 0 0 2022-01-31T19:44:17.499 INFO:tasks.workunit.client.0.smithi167.stdout:7/971: write d8/d2c/d7a/d99/de2/fde [469465,115292] 0 2022-01-31T19:44:17.500 INFO:tasks.workunit.client.1.smithi171.stdout:9/27: creat fa x:0 0 0 2022-01-31T19:44:17.502 INFO:tasks.workunit.client.1.smithi171.stdout:7/73: creat d1/d2/d7/f19 x:0 0 0 2022-01-31T19:44:17.502 INFO:tasks.workunit.client.1.smithi171.stdout:7/74: fdatasync d1/d2/d7/ff 0 2022-01-31T19:44:17.507 INFO:tasks.workunit.client.0.smithi167.stdout:6/685: write d8/d62/f6f [1006723,5095] 0 2022-01-31T19:44:17.509 INFO:tasks.workunit.client.0.smithi167.stdout:7/972: rename d8/d2c/d43/d94/f10b to d8/dc/dc1/f15f 0 2022-01-31T19:44:17.509 INFO:tasks.workunit.client.0.smithi167.stdout:4/819: mkdir d0/d6/d7/d26/d40/d10e 0 2022-01-31T19:44:17.511 INFO:tasks.workunit.client.0.smithi167.stdout:6/686: stat d8/d33/d51/d72/fd6 0 2022-01-31T19:44:17.513 INFO:tasks.workunit.client.1.smithi171.stdout:0/98: dwrite f3 [4194304,4194304] 0 2022-01-31T19:44:17.515 INFO:tasks.workunit.client.0.smithi167.stdout:3/835: write d9/f25 [417406,105983] 0 2022-01-31T19:44:17.515 INFO:tasks.workunit.client.0.smithi167.stdout:0/552: dread fa [0,4194304] 0 2022-01-31T19:44:17.515 INFO:tasks.workunit.client.0.smithi167.stdout:0/553: readlink d17/d19/l24 0 2022-01-31T19:44:17.515 INFO:tasks.workunit.client.0.smithi167.stdout:0/554: fdatasync d17/d19/d1c/d39/d44/f56 0 2022-01-31T19:44:17.516 INFO:tasks.workunit.client.0.smithi167.stdout:0/555: fdatasync d17/d69/f94 0 2022-01-31T19:44:17.516 INFO:tasks.workunit.client.0.smithi167.stdout:0/556: dread - d17/d19/d1c/d39/d44/d88/faf zero size 2022-01-31T19:44:17.516 INFO:tasks.workunit.client.0.smithi167.stdout:0/557: stat d17/l18 0 2022-01-31T19:44:17.517 INFO:tasks.workunit.client.0.smithi167.stdout:7/973: mknod d8/d2c/d6e/d4d/d67/c160 0 2022-01-31T19:44:17.517 INFO:tasks.workunit.client.1.smithi171.stdout:0/99: creat d9/db/f21 x:0 0 0 2022-01-31T19:44:17.517 INFO:tasks.workunit.client.1.smithi171.stdout:3/86: truncate d4/fb 3871582 0 2022-01-31T19:44:17.518 INFO:tasks.workunit.client.0.smithi167.stdout:6/687: getdents d8/d33/d51/d72/dc2 0 2022-01-31T19:44:17.518 INFO:tasks.workunit.client.0.smithi167.stdout:6/688: creat d8/d33/d51/d6a/ffa x:0 0 0 2022-01-31T19:44:17.519 INFO:tasks.workunit.client.1.smithi171.stdout:0/100: mkdir d9/db/d22 0 2022-01-31T19:44:17.520 INFO:tasks.workunit.client.0.smithi167.stdout:3/836: creat d9/de7/ffe x:0 0 0 2022-01-31T19:44:17.522 INFO:tasks.workunit.client.1.smithi171.stdout:0/101: rename d9/db/cd to d9/db/c23 0 2022-01-31T19:44:17.523 INFO:tasks.workunit.client.0.smithi167.stdout:0/558: rename d17/d2c/l63 to d17/d19/d1c/d39/d7b/lb0 0 2022-01-31T19:44:17.523 INFO:tasks.workunit.client.0.smithi167.stdout:0/559: read - d17/d33/da4/faa zero size 2022-01-31T19:44:17.527 INFO:tasks.workunit.client.0.smithi167.stdout:0/560: dread fc [4194304,4194304] 0 2022-01-31T19:44:17.528 INFO:tasks.workunit.client.1.smithi171.stdout:4/76: dwrite d2/f14 [0,4194304] 0 2022-01-31T19:44:17.530 INFO:tasks.workunit.client.0.smithi167.stdout:0/561: rename d17/d2c/f99 to d17/d19/fb1 0 2022-01-31T19:44:17.530 INFO:tasks.workunit.client.0.smithi167.stdout:4/820: dread d0/d6/d7/d39/d3a/f54 [0,4194304] 0 2022-01-31T19:44:17.531 INFO:tasks.workunit.client.0.smithi167.stdout:0/562: mknod d17/d2c/cb2 0 2022-01-31T19:44:17.531 INFO:tasks.workunit.client.0.smithi167.stdout:0/563: getdents d17/d3e/da1 0 2022-01-31T19:44:17.533 INFO:tasks.workunit.client.0.smithi167.stdout:0/564: symlink d17/d19/d1c/d39/d7b/lb3 0 2022-01-31T19:44:17.533 INFO:tasks.workunit.client.0.smithi167.stdout:0/565: creat d17/d33/da4/fb4 x:0 0 0 2022-01-31T19:44:17.533 INFO:tasks.workunit.client.0.smithi167.stdout:0/566: write d17/d19/d1c/d39/d44/f65 [1795800,27070] 0 2022-01-31T19:44:17.533 INFO:tasks.workunit.client.0.smithi167.stdout:0/567: creat d17/d2c/fb5 x:0 0 0 2022-01-31T19:44:17.534 INFO:tasks.workunit.client.0.smithi167.stdout:2/612: dwrite fa [8388608,4194304] 0 2022-01-31T19:44:17.536 INFO:tasks.workunit.client.1.smithi171.stdout:4/77: creat d2/d3/d5/dc/f19 x:0 0 0 2022-01-31T19:44:17.536 INFO:tasks.workunit.client.1.smithi171.stdout:4/78: link d2/f14 d2/d3/d5/f1a 0 2022-01-31T19:44:17.536 INFO:tasks.workunit.client.0.smithi167.stdout:0/568: mknod d17/d33/cb6 0 2022-01-31T19:44:17.537 INFO:tasks.workunit.client.0.smithi167.stdout:2/613: symlink dd/df/d1b/d3d/d4a/d4b/lc3 0 2022-01-31T19:44:17.538 INFO:tasks.workunit.client.1.smithi171.stdout:7/75: dwrite d1/f16 [0,4194304] 0 2022-01-31T19:44:17.538 INFO:tasks.workunit.client.1.smithi171.stdout:7/76: creat d1/d2/d12/f1a x:0 0 0 2022-01-31T19:44:17.544 INFO:tasks.workunit.client.0.smithi167.stdout:2/614: mkdir dd/df/d1b/d3d/d4a/d4b/d74/d85/dc4 0 2022-01-31T19:44:17.544 INFO:tasks.workunit.client.0.smithi167.stdout:2/615: creat dd/df/d3f/d51/fc5 x:0 0 0 2022-01-31T19:44:17.544 INFO:tasks.workunit.client.0.smithi167.stdout:2/616: creat dd/df/d1b/d3d/d4a/da7/fc6 x:0 0 0 2022-01-31T19:44:17.549 INFO:tasks.workunit.client.1.smithi171.stdout:6/100: dwrite f9 [0,4194304] 0 2022-01-31T19:44:17.550 INFO:tasks.workunit.client.1.smithi171.stdout:6/101: read - dc/f17 zero size 2022-01-31T19:44:17.550 INFO:tasks.workunit.client.1.smithi171.stdout:0/102: dread d9/f16 [0,4194304] 0 2022-01-31T19:44:17.551 INFO:tasks.workunit.client.0.smithi167.stdout:2/617: dread dd/df/d3f/f8e [0,4194304] 0 2022-01-31T19:44:17.551 INFO:tasks.workunit.client.1.smithi171.stdout:4/79: write f0 [2048310,78853] 0 2022-01-31T19:44:17.551 INFO:tasks.workunit.client.1.smithi171.stdout:4/80: write f1 [4799140,56391] 0 2022-01-31T19:44:17.553 INFO:tasks.workunit.client.1.smithi171.stdout:2/150: write d7/fa [1534800,10941] 0 2022-01-31T19:44:17.553 INFO:tasks.workunit.client.1.smithi171.stdout:0/103: mkdir d9/d24 0 2022-01-31T19:44:17.554 INFO:tasks.workunit.client.1.smithi171.stdout:2/151: read d7/dd/f15 [3456556,61204] 0 2022-01-31T19:44:17.555 INFO:tasks.workunit.client.1.smithi171.stdout:2/152: chown d7/d9/fe 868 1 2022-01-31T19:44:17.555 INFO:tasks.workunit.client.1.smithi171.stdout:6/102: mknod dc/d18/c1c 0 2022-01-31T19:44:17.555 INFO:tasks.workunit.client.1.smithi171.stdout:6/103: dread - dc/d18/f1b zero size 2022-01-31T19:44:17.555 INFO:tasks.workunit.client.1.smithi171.stdout:6/104: creat dc/d19/f1d x:0 0 0 2022-01-31T19:44:17.556 INFO:tasks.workunit.client.1.smithi171.stdout:6/105: mkdir dc/d1e 0 2022-01-31T19:44:17.558 INFO:tasks.workunit.client.1.smithi171.stdout:6/106: link dc/c11 dc/d18/c1f 0 2022-01-31T19:44:17.559 INFO:tasks.workunit.client.1.smithi171.stdout:6/107: mknod dc/d18/c20 0 2022-01-31T19:44:17.559 INFO:tasks.workunit.client.1.smithi171.stdout:6/108: getdents dc/d1e 0 2022-01-31T19:44:17.560 INFO:tasks.workunit.client.1.smithi171.stdout:6/109: mkdir dc/d21 0 2022-01-31T19:44:17.561 INFO:tasks.workunit.client.1.smithi171.stdout:6/110: chown dc/f13 17745634 1 2022-01-31T19:44:17.573 INFO:tasks.workunit.client.0.smithi167.stdout:7/974: dwrite d8/d2c/d7a/f9c [4194304,4194304] 0 2022-01-31T19:44:17.573 INFO:tasks.workunit.client.0.smithi167.stdout:7/975: chown d8/d2c/d7a/dd1/f104 3490 1 2022-01-31T19:44:17.575 INFO:tasks.workunit.client.0.smithi167.stdout:3/837: dwrite d9/d21/f23 [0,4194304] 0 2022-01-31T19:44:17.576 INFO:tasks.workunit.client.0.smithi167.stdout:0/569: dwrite d17/f3d [4194304,4194304] 0 2022-01-31T19:44:17.577 INFO:tasks.workunit.client.1.smithi171.stdout:9/28: getdents . 0 2022-01-31T19:44:17.578 INFO:tasks.workunit.client.0.smithi167.stdout:6/689: dwrite d8/d33/f90 [0,4194304] 0 2022-01-31T19:44:17.578 INFO:tasks.workunit.client.0.smithi167.stdout:7/976: mkdir d8/d2c/d6e/d4d/d5e/d71/db1/d4f/d161 0 2022-01-31T19:44:17.579 INFO:tasks.workunit.client.0.smithi167.stdout:6/690: truncate d8/d33/d51/fef 1009871 0 2022-01-31T19:44:17.579 INFO:tasks.workunit.client.0.smithi167.stdout:7/977: getdents d8/d2c/d7a/d99/de2/dc3/d157 0 2022-01-31T19:44:17.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/821: dwrite d0/d6/d7/d26/f62 [0,4194304] 0 2022-01-31T19:44:17.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/822: truncate d0/d5d/fa5 364551 0 2022-01-31T19:44:17.580 INFO:tasks.workunit.client.0.smithi167.stdout:2/618: dwrite dd/df/f50 [0,4194304] 0 2022-01-31T19:44:17.581 INFO:tasks.workunit.client.1.smithi171.stdout:0/104: dwrite f6 [0,4194304] 0 2022-01-31T19:44:17.582 INFO:tasks.workunit.client.0.smithi167.stdout:7/978: write d8/d2c/d7a/f9c [5675475,1477] 0 2022-01-31T19:44:17.583 INFO:tasks.workunit.client.1.smithi171.stdout:9/29: creat fb x:0 0 0 2022-01-31T19:44:17.584 INFO:tasks.workunit.client.0.smithi167.stdout:8/784: sync 2022-01-31T19:44:17.586 INFO:tasks.workunit.client.1.smithi171.stdout:0/105: mkdir d9/d1b/d25 0 2022-01-31T19:44:17.589 INFO:tasks.workunit.client.0.smithi167.stdout:3/838: creat d9/d29/d3b/d9b/da3/dd9/def/fff x:0 0 0 2022-01-31T19:44:17.589 INFO:tasks.workunit.client.0.smithi167.stdout:3/839: write d9/d7f/dbb/ff0 [205027,51930] 0 2022-01-31T19:44:17.592 INFO:tasks.workunit.client.0.smithi167.stdout:0/570: stat d17/d19/d1c/d39/d44/l54 0 2022-01-31T19:44:17.592 INFO:tasks.workunit.client.0.smithi167.stdout:0/571: dread - d17/d3e/f9b zero size 2022-01-31T19:44:17.594 INFO:tasks.workunit.client.0.smithi167.stdout:6/691: rmdir d8/d33/d51/d72/dc7 39 2022-01-31T19:44:17.595 INFO:tasks.workunit.client.1.smithi171.stdout:0/106: creat d9/d24/f26 x:0 0 0 2022-01-31T19:44:17.595 INFO:tasks.workunit.client.1.smithi171.stdout:7/77: rename d1/d2 to d1/d1b 0 2022-01-31T19:44:17.596 INFO:tasks.workunit.client.1.smithi171.stdout:7/78: fsync d1/d1b/d7/f19 0 2022-01-31T19:44:17.596 INFO:tasks.workunit.client.0.smithi167.stdout:4/823: mknod d0/d6/d7/d26/d31/d5c/de2/c10f 0 2022-01-31T19:44:17.597 INFO:tasks.workunit.client.0.smithi167.stdout:7/979: creat d8/dab/dd0/f162 x:0 0 0 2022-01-31T19:44:17.597 INFO:tasks.workunit.client.0.smithi167.stdout:7/980: fsync d8/d2c/d43/df3/ff6 0 2022-01-31T19:44:17.599 INFO:tasks.workunit.client.0.smithi167.stdout:8/785: unlink d6/db/d28/d61/dd5/le7 0 2022-01-31T19:44:17.600 INFO:tasks.workunit.client.0.smithi167.stdout:3/840: mkdir d9/d29/d3b/d41/d6b/d100 0 2022-01-31T19:44:17.602 INFO:tasks.workunit.client.0.smithi167.stdout:0/572: creat d17/d19/d1c/d39/d7b/da9/fb7 x:0 0 0 2022-01-31T19:44:17.602 INFO:tasks.workunit.client.0.smithi167.stdout:4/824: mkdir d0/d6/d7/d39/df0/d110 0 2022-01-31T19:44:17.603 INFO:tasks.workunit.client.0.smithi167.stdout:4/825: stat d0/d6/d7/d26/dd2/de5/f91 0 2022-01-31T19:44:17.603 INFO:tasks.workunit.client.0.smithi167.stdout:7/981: symlink d8/d2c/d7a/dd3/l163 0 2022-01-31T19:44:17.604 INFO:tasks.workunit.client.0.smithi167.stdout:7/982: write d8/dab/dd0/f162 [969469,12369] 0 2022-01-31T19:44:17.604 INFO:tasks.workunit.client.0.smithi167.stdout:7/983: dread - d8/d2c/d6e/d4d/d5e/d71/db1/f117 zero size 2022-01-31T19:44:17.604 INFO:tasks.workunit.client.0.smithi167.stdout:0/573: mknod d17/cb8 0 2022-01-31T19:44:17.605 INFO:tasks.workunit.client.0.smithi167.stdout:3/841: write d9/f2c [3593845,112859] 0 2022-01-31T19:44:17.605 INFO:tasks.workunit.client.0.smithi167.stdout:6/692: dread d8/d33/d51/d7e/dc4/fbc [0,4194304] 0 2022-01-31T19:44:17.609 INFO:tasks.workunit.client.0.smithi167.stdout:4/826: mkdir d0/d6/d7/d26/d31/ddf/d75/dd6/d111 0 2022-01-31T19:44:17.610 INFO:tasks.workunit.client.0.smithi167.stdout:7/984: link d8/d35/f112 d8/d2c/d7a/d99/de2/dc3/f164 0 2022-01-31T19:44:17.610 INFO:tasks.workunit.client.0.smithi167.stdout:0/574: creat d17/d19/d1c/d39/d7b/fb9 x:0 0 0 2022-01-31T19:44:17.610 INFO:tasks.workunit.client.0.smithi167.stdout:0/575: readlink d17/d19/d1c/d39/d44/l54 0 2022-01-31T19:44:17.610 INFO:tasks.workunit.client.0.smithi167.stdout:0/576: truncate fa 3441358 0 2022-01-31T19:44:17.611 INFO:tasks.workunit.client.0.smithi167.stdout:0/577: dread - d17/d3e/fae zero size 2022-01-31T19:44:17.612 INFO:tasks.workunit.client.0.smithi167.stdout:3/842: write d9/d26/f6e [3341076,13719] 0 2022-01-31T19:44:17.612 INFO:tasks.workunit.client.0.smithi167.stdout:3/843: write d9/d29/d60/f5b [1068134,10653] 0 2022-01-31T19:44:17.612 INFO:tasks.workunit.client.0.smithi167.stdout:8/786: dread d6/db/de/d18/f19 [0,4194304] 0 2022-01-31T19:44:17.612 INFO:tasks.workunit.client.0.smithi167.stdout:4/827: creat d0/d6/d7/d26/d31/d5c/de2/f112 x:0 0 0 2022-01-31T19:44:17.612 INFO:tasks.workunit.client.0.smithi167.stdout:4/828: write d0/d6/d7/f9a [4973835,45835] 0 2022-01-31T19:44:17.614 INFO:tasks.workunit.client.1.smithi171.stdout:0/107: dread d9/db/f12 [0,4194304] 0 2022-01-31T19:44:17.615 INFO:tasks.workunit.client.0.smithi167.stdout:8/787: creat d6/db/d28/d61/dd5/dbf/f102 x:0 0 0 2022-01-31T19:44:17.615 INFO:tasks.workunit.client.0.smithi167.stdout:8/788: truncate d6/db/d28/d61/f98 374245 0 2022-01-31T19:44:17.617 INFO:tasks.workunit.client.0.smithi167.stdout:7/985: rename d8/d2c/d6e/l28 to d8/dc/l165 0 2022-01-31T19:44:17.617 INFO:tasks.workunit.client.0.smithi167.stdout:2/619: dwrite dd/df/d1b/d3d/d4a/da7/fbb [0,4194304] 0 2022-01-31T19:44:17.618 INFO:tasks.workunit.client.0.smithi167.stdout:7/986: creat d8/d2c/d43/d94/f166 x:0 0 0 2022-01-31T19:44:17.618 INFO:tasks.workunit.client.1.smithi171.stdout:4/81: dwrite f1 [4194304,4194304] 0 2022-01-31T19:44:17.619 INFO:tasks.workunit.client.0.smithi167.stdout:8/789: creat d6/db/d28/d61/dd5/f103 x:0 0 0 2022-01-31T19:44:17.620 INFO:tasks.workunit.client.1.smithi171.stdout:0/108: dread d9/db/f12 [0,4194304] 0 2022-01-31T19:44:17.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/111: rmdir dc/d18 39 2022-01-31T19:44:17.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/112: fsync dc/f17 0 2022-01-31T19:44:17.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/113: fsync dc/f14 0 2022-01-31T19:44:17.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/844: rename d9/d7f/da8/fe6 to d9/d56/f101 0 2022-01-31T19:44:17.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/845: write d9/d29/d3b/d41/d6b/f3e [4844168,21702] 0 2022-01-31T19:44:17.623 INFO:tasks.workunit.client.0.smithi167.stdout:3/846: write d9/d26/f65 [1169819,7251] 0 2022-01-31T19:44:17.624 INFO:tasks.workunit.client.0.smithi167.stdout:3/847: write f7 [4504766,44388] 0 2022-01-31T19:44:17.624 INFO:tasks.workunit.client.0.smithi167.stdout:7/987: mknod d8/d2c/d7a/dd3/c167 0 2022-01-31T19:44:17.625 INFO:tasks.workunit.client.0.smithi167.stdout:0/578: dread d17/d3e/f5f [0,4194304] 0 2022-01-31T19:44:17.626 INFO:tasks.workunit.client.0.smithi167.stdout:4/829: rename d0/d6/d7/d26/d31/ddf/da2/dcf/d65 to d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d113 0 2022-01-31T19:44:17.626 INFO:tasks.workunit.client.0.smithi167.stdout:4/830: chown d0/d6/d7/d39/d3a/c9e 21 1 2022-01-31T19:44:17.626 INFO:tasks.workunit.client.0.smithi167.stdout:4/831: readlink d0/d6/d7/d39/d3a/la4 0 2022-01-31T19:44:17.627 INFO:tasks.workunit.client.0.smithi167.stdout:7/988: creat d8/d2c/d43/f168 x:0 0 0 2022-01-31T19:44:17.628 INFO:tasks.workunit.client.0.smithi167.stdout:7/989: fdatasync d8/dc/d18/f3b 0 2022-01-31T19:44:17.628 INFO:tasks.workunit.client.0.smithi167.stdout:0/579: mkdir d17/d33/da4/dba 0 2022-01-31T19:44:17.629 INFO:tasks.workunit.client.0.smithi167.stdout:0/580: stat l12 0 2022-01-31T19:44:17.629 INFO:tasks.workunit.client.1.smithi171.stdout:4/82: unlink d2/d3/d5/f6 0 2022-01-31T19:44:17.630 INFO:tasks.workunit.client.0.smithi167.stdout:4/832: unlink d0/d6/d7/d26/d31/ddf/d75/cf9 0 2022-01-31T19:44:17.630 INFO:tasks.workunit.client.0.smithi167.stdout:4/833: fsync d0/d6/dab/fc5 0 2022-01-31T19:44:17.631 INFO:tasks.workunit.client.1.smithi171.stdout:9/30: dwrite f4 [4194304,4194304] 0 2022-01-31T19:44:17.631 INFO:tasks.workunit.client.1.smithi171.stdout:9/31: creat fc x:0 0 0 2022-01-31T19:44:17.631 INFO:tasks.workunit.client.1.smithi171.stdout:9/32: truncate f9 988947 0 2022-01-31T19:44:17.631 INFO:tasks.workunit.client.1.smithi171.stdout:9/33: chown fa 0 1 2022-01-31T19:44:17.632 INFO:tasks.workunit.client.1.smithi171.stdout:9/34: write fa [351942,6193] 0 2022-01-31T19:44:17.632 INFO:tasks.workunit.client.0.smithi167.stdout:7/990: creat d8/d2c/d43/d94/d12e/f169 x:0 0 0 2022-01-31T19:44:17.632 INFO:tasks.workunit.client.0.smithi167.stdout:0/581: rmdir d17/d3e 39 2022-01-31T19:44:17.634 INFO:tasks.workunit.client.1.smithi171.stdout:7/79: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:44:17.637 INFO:tasks.workunit.client.0.smithi167.stdout:0/582: link d17/d2c/d87/f8d d17/d19/d1c/d39/fbb 0 2022-01-31T19:44:17.637 INFO:tasks.workunit.client.0.smithi167.stdout:3/848: dread d9/d56/f73 [0,4194304] 0 2022-01-31T19:44:17.637 INFO:tasks.workunit.client.0.smithi167.stdout:3/849: creat d9/d29/d3b/d9b/f102 x:0 0 0 2022-01-31T19:44:17.638 INFO:tasks.workunit.client.0.smithi167.stdout:3/850: write d9/fc1 [884027,122443] 0 2022-01-31T19:44:17.638 INFO:tasks.workunit.client.1.smithi171.stdout:9/35: unlink l2 0 2022-01-31T19:44:17.639 INFO:tasks.workunit.client.0.smithi167.stdout:0/583: mknod d17/d19/d1c/d39/d44/d88/cbc 0 2022-01-31T19:44:17.640 INFO:tasks.workunit.client.0.smithi167.stdout:0/584: mkdir d17/d19/d1c/d39/d7b/dbd 0 2022-01-31T19:44:17.641 INFO:tasks.workunit.client.0.smithi167.stdout:0/585: symlink d17/d19/d1c/d39/d7b/lbe 0 2022-01-31T19:44:17.642 INFO:tasks.workunit.client.0.smithi167.stdout:0/586: write d17/d69/fa0 [235405,110936] 0 2022-01-31T19:44:17.642 INFO:tasks.workunit.client.0.smithi167.stdout:0/587: chown d17/d2c/f79 249561 1 2022-01-31T19:44:17.643 INFO:tasks.workunit.client.0.smithi167.stdout:0/588: rename d17/d19/l24 to d17/d69/lbf 0 2022-01-31T19:44:17.644 INFO:tasks.workunit.client.1.smithi171.stdout:1/23: sync 2022-01-31T19:44:17.644 INFO:tasks.workunit.client.1.smithi171.stdout:8/61: sync 2022-01-31T19:44:17.645 INFO:tasks.workunit.client.1.smithi171.stdout:8/62: write f7 [4758799,20640] 0 2022-01-31T19:44:17.645 INFO:tasks.workunit.client.1.smithi171.stdout:3/87: dwrite d4/fb [0,4194304] 0 2022-01-31T19:44:17.646 INFO:tasks.workunit.client.0.smithi167.stdout:0/589: link d17/d33/c5d d17/cc0 0 2022-01-31T19:44:17.646 INFO:tasks.workunit.client.1.smithi171.stdout:6/114: getdents dc/d19 0 2022-01-31T19:44:17.647 INFO:tasks.workunit.client.1.smithi171.stdout:5/57: sync 2022-01-31T19:44:17.647 INFO:tasks.workunit.client.1.smithi171.stdout:5/58: chown d5/d6/l7 695683023 1 2022-01-31T19:44:17.647 INFO:tasks.workunit.client.1.smithi171.stdout:5/59: creat d5/ff x:0 0 0 2022-01-31T19:44:17.648 INFO:tasks.workunit.client.1.smithi171.stdout:7/80: mknod d1/d1b/c1c 0 2022-01-31T19:44:17.648 INFO:tasks.workunit.client.1.smithi171.stdout:0/109: getdents d9/d1b 0 2022-01-31T19:44:17.648 INFO:tasks.workunit.client.0.smithi167.stdout:0/590: write d17/d3e/f68 [629180,117386] 0 2022-01-31T19:44:17.650 INFO:tasks.workunit.client.1.smithi171.stdout:1/24: rename c1 to c2 0 2022-01-31T19:44:17.651 INFO:tasks.workunit.client.1.smithi171.stdout:1/25: chown c2 24151 1 2022-01-31T19:44:17.651 INFO:tasks.workunit.client.1.smithi171.stdout:8/63: creat df/f16 x:0 0 0 2022-01-31T19:44:17.651 INFO:tasks.workunit.client.1.smithi171.stdout:8/64: truncate f6 123640 0 2022-01-31T19:44:17.655 INFO:tasks.workunit.client.0.smithi167.stdout:2/620: dread dd/d18/d63/f32 [0,4194304] 0 2022-01-31T19:44:17.655 INFO:tasks.workunit.client.0.smithi167.stdout:2/621: fsync dd/d18/fa9 0 2022-01-31T19:44:17.656 INFO:tasks.workunit.client.1.smithi171.stdout:6/115: truncate f9 6420341 0 2022-01-31T19:44:17.656 INFO:tasks.workunit.client.1.smithi171.stdout:5/60: symlink d5/d6/l10 0 2022-01-31T19:44:17.657 INFO:tasks.workunit.client.0.smithi167.stdout:2/622: symlink dd/d18/d63/dab/lc7 0 2022-01-31T19:44:17.661 INFO:tasks.workunit.client.1.smithi171.stdout:7/81: rmdir d1/d1b 39 2022-01-31T19:44:17.662 INFO:tasks.workunit.client.1.smithi171.stdout:7/82: chown d1/d1b/d7/cb 450 1 2022-01-31T19:44:17.662 INFO:tasks.workunit.client.1.smithi171.stdout:1/26: creat f3 x:0 0 0 2022-01-31T19:44:17.662 INFO:tasks.workunit.client.0.smithi167.stdout:0/591: dread d17/d3e/f5f [0,4194304] 0 2022-01-31T19:44:17.663 INFO:tasks.workunit.client.0.smithi167.stdout:0/592: getdents d17/d3e/d7e 0 2022-01-31T19:44:17.665 INFO:tasks.workunit.client.0.smithi167.stdout:0/593: write d17/d19/d1c/d39/d44/d88/f8a [974859,101178] 0 2022-01-31T19:44:17.666 INFO:tasks.workunit.client.1.smithi171.stdout:3/88: symlink d4/l18 0 2022-01-31T19:44:17.666 INFO:tasks.workunit.client.1.smithi171.stdout:0/110: mknod d9/d1b/c27 0 2022-01-31T19:44:17.667 INFO:tasks.workunit.client.1.smithi171.stdout:6/116: link dc/f15 dc/d18/f22 0 2022-01-31T19:44:17.667 INFO:tasks.workunit.client.1.smithi171.stdout:6/117: stat dc/d18 0 2022-01-31T19:44:17.667 INFO:tasks.workunit.client.1.smithi171.stdout:1/27: mknod c4 0 2022-01-31T19:44:17.667 INFO:tasks.workunit.client.1.smithi171.stdout:1/28: dread - f3 zero size 2022-01-31T19:44:17.668 INFO:tasks.workunit.client.1.smithi171.stdout:3/89: dread d4/fa [0,4194304] 0 2022-01-31T19:44:17.668 INFO:tasks.workunit.client.1.smithi171.stdout:0/111: rmdir d9/d1b 39 2022-01-31T19:44:17.668 INFO:tasks.workunit.client.1.smithi171.stdout:6/118: rmdir dc/d18 39 2022-01-31T19:44:17.668 INFO:tasks.workunit.client.1.smithi171.stdout:6/119: write dc/d18/f1b [502699,69020] 0 2022-01-31T19:44:17.669 INFO:tasks.workunit.client.1.smithi171.stdout:1/29: mknod c5 0 2022-01-31T19:44:17.670 INFO:tasks.workunit.client.1.smithi171.stdout:3/90: read - d4/f13 zero size 2022-01-31T19:44:17.670 INFO:tasks.workunit.client.0.smithi167.stdout:8/790: dwrite d6/db/d28/f94 [0,4194304] 0 2022-01-31T19:44:17.670 INFO:tasks.workunit.client.0.smithi167.stdout:8/791: creat d6/db/d28/d61/dd5/f104 x:0 0 0 2022-01-31T19:44:17.671 INFO:tasks.workunit.client.0.smithi167.stdout:8/792: dread - d6/db/de/d18/deb/fed zero size 2022-01-31T19:44:17.671 INFO:tasks.workunit.client.0.smithi167.stdout:8/793: chown d6/db/de/d18/da4/dc2/dc3/fe4 10536640 1 2022-01-31T19:44:17.671 INFO:tasks.workunit.client.0.smithi167.stdout:0/594: creat d17/d33/da4/dba/fc1 x:0 0 0 2022-01-31T19:44:17.672 INFO:tasks.workunit.client.1.smithi171.stdout:6/120: mkdir dc/d21/d23 0 2022-01-31T19:44:17.672 INFO:tasks.workunit.client.1.smithi171.stdout:1/30: symlink l6 0 2022-01-31T19:44:17.673 INFO:tasks.workunit.client.1.smithi171.stdout:1/31: creat f7 x:0 0 0 2022-01-31T19:44:17.673 INFO:tasks.workunit.client.1.smithi171.stdout:1/32: creat f8 x:0 0 0 2022-01-31T19:44:17.673 INFO:tasks.workunit.client.1.smithi171.stdout:1/33: write f3 [780502,24913] 0 2022-01-31T19:44:17.673 INFO:tasks.workunit.client.1.smithi171.stdout:1/34: rmdir - no directory 2022-01-31T19:44:17.674 INFO:tasks.workunit.client.0.smithi167.stdout:0/595: getdents d17/d19/d1c 0 2022-01-31T19:44:17.674 INFO:tasks.workunit.client.1.smithi171.stdout:0/112: write d9/f1f [3268650,117045] 0 2022-01-31T19:44:17.676 INFO:tasks.workunit.client.1.smithi171.stdout:3/91: creat d4/dd/f19 x:0 0 0 2022-01-31T19:44:17.676 INFO:tasks.workunit.client.1.smithi171.stdout:6/121: creat dc/d21/d23/f24 x:0 0 0 2022-01-31T19:44:17.676 INFO:tasks.workunit.client.1.smithi171.stdout:6/122: fdatasync dc/f15 0 2022-01-31T19:44:17.676 INFO:tasks.workunit.client.1.smithi171.stdout:6/123: readlink dc/lf 0 2022-01-31T19:44:17.677 INFO:tasks.workunit.client.1.smithi171.stdout:6/124: write dc/d21/d23/f24 [456796,93828] 0 2022-01-31T19:44:17.677 INFO:tasks.workunit.client.1.smithi171.stdout:6/125: write dc/f14 [699892,3271] 0 2022-01-31T19:44:17.677 INFO:tasks.workunit.client.1.smithi171.stdout:6/126: dread - dc/f13 zero size 2022-01-31T19:44:17.678 INFO:tasks.workunit.client.0.smithi167.stdout:2/623: write dd/df/d1b/d3d/f44 [2978221,118507] 0 2022-01-31T19:44:17.678 INFO:tasks.workunit.client.0.smithi167.stdout:2/624: chown dd/df/d1b/d2f/d36/ca8 3 1 2022-01-31T19:44:17.678 INFO:tasks.workunit.client.0.smithi167.stdout:2/625: chown dd/l19 3136829 1 2022-01-31T19:44:17.678 INFO:tasks.workunit.client.0.smithi167.stdout:2/626: fdatasync dd/df/d3f/f8e 0 2022-01-31T19:44:17.679 INFO:tasks.workunit.client.0.smithi167.stdout:2/627: mkdir dd/d18/d62/d75/dc8 0 2022-01-31T19:44:17.679 INFO:tasks.workunit.client.0.smithi167.stdout:2/628: chown dd/df/d1b/d2f/d36/c3b 8 1 2022-01-31T19:44:17.681 INFO:tasks.workunit.client.0.smithi167.stdout:2/629: rmdir dd/df/d1b/d3d/d4a/d4b/d54 39 2022-01-31T19:44:17.681 INFO:tasks.workunit.client.0.smithi167.stdout:2/630: stat dd/df/d22/l53 0 2022-01-31T19:44:17.683 INFO:tasks.workunit.client.0.smithi167.stdout:2/631: mknod dd/df/d22/d96/d9e/cc9 0 2022-01-31T19:44:17.684 INFO:tasks.workunit.client.0.smithi167.stdout:2/632: fdatasync dd/df/d1b/d2f/f34 0 2022-01-31T19:44:17.684 INFO:tasks.workunit.client.1.smithi171.stdout:3/92: mkdir d4/dd/d1a 0 2022-01-31T19:44:17.685 INFO:tasks.workunit.client.1.smithi171.stdout:6/127: truncate fa 528959 0 2022-01-31T19:44:17.685 INFO:tasks.workunit.client.1.smithi171.stdout:0/113: mknod d9/db/c28 0 2022-01-31T19:44:17.685 INFO:tasks.workunit.client.1.smithi171.stdout:0/114: chown d9/d24/f26 0 1 2022-01-31T19:44:17.686 INFO:tasks.workunit.client.0.smithi167.stdout:2/633: mkdir dd/df/d1b/d3d/d4a/d4b/d54/dca 0 2022-01-31T19:44:17.686 INFO:tasks.workunit.client.1.smithi171.stdout:4/83: dwrite d2/d3/d5/f1a [0,4194304] 0 2022-01-31T19:44:17.689 INFO:tasks.workunit.client.1.smithi171.stdout:6/128: creat dc/d18/f25 x:0 0 0 2022-01-31T19:44:17.689 INFO:tasks.workunit.client.1.smithi171.stdout:6/129: creat dc/d21/d23/f26 x:0 0 0 2022-01-31T19:44:17.689 INFO:tasks.workunit.client.1.smithi171.stdout:6/130: write dc/f13 [522774,49488] 0 2022-01-31T19:44:17.690 INFO:tasks.workunit.client.1.smithi171.stdout:4/84: symlink d2/d3/d5/d11/l1b 0 2022-01-31T19:44:17.697 INFO:tasks.workunit.client.1.smithi171.stdout:6/131: mkdir dc/d21/d23/d27 0 2022-01-31T19:44:17.697 INFO:tasks.workunit.client.1.smithi171.stdout:6/132: readlink l7 0 2022-01-31T19:44:17.697 INFO:tasks.workunit.client.1.smithi171.stdout:6/133: write dc/d19/f1a [246992,121186] 0 2022-01-31T19:44:17.698 INFO:tasks.workunit.client.1.smithi171.stdout:4/85: rename f0 to d2/d3/f1c 0 2022-01-31T19:44:17.698 INFO:tasks.workunit.client.1.smithi171.stdout:6/134: readlink dc/lf 0 2022-01-31T19:44:17.701 INFO:tasks.workunit.client.1.smithi171.stdout:4/86: creat d2/d3/d5/d11/d12/f1d x:0 0 0 2022-01-31T19:44:17.701 INFO:tasks.workunit.client.1.smithi171.stdout:4/87: chown d2/d3/d5/l18 1931 1 2022-01-31T19:44:17.703 INFO:tasks.workunit.client.1.smithi171.stdout:4/88: truncate d2/d3/f1c 436454 0 2022-01-31T19:44:17.704 INFO:tasks.workunit.client.0.smithi167.stdout:4/834: dwrite d0/d6/d7/d26/d40/f24 [4194304,4194304] 0 2022-01-31T19:44:17.704 INFO:tasks.workunit.client.0.smithi167.stdout:2/634: dread dd/f29 [0,4194304] 0 2022-01-31T19:44:17.704 INFO:tasks.workunit.client.0.smithi167.stdout:4/835: chown d0/d6/d7/d39 2 1 2022-01-31T19:44:17.706 INFO:tasks.workunit.client.0.smithi167.stdout:4/836: symlink d0/d6/d7/d39/df0/d110/l114 0 2022-01-31T19:44:17.707 INFO:tasks.workunit.client.0.smithi167.stdout:4/837: dread - d0/fac zero size 2022-01-31T19:44:17.707 INFO:tasks.workunit.client.0.smithi167.stdout:2/635: mkdir dd/dcb 0 2022-01-31T19:44:17.707 INFO:tasks.workunit.client.0.smithi167.stdout:2/636: dread - dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b zero size 2022-01-31T19:44:17.707 INFO:tasks.workunit.client.0.smithi167.stdout:2/637: write dd/df/d1b/d3d/d4a/d4b/f7f [232380,52008] 0 2022-01-31T19:44:17.708 INFO:tasks.workunit.client.0.smithi167.stdout:2/638: fsync dd/d18/d63/fa3 0 2022-01-31T19:44:17.710 INFO:tasks.workunit.client.0.smithi167.stdout:4/838: symlink d0/d6/d7/d26/dd2/l115 0 2022-01-31T19:44:17.711 INFO:tasks.workunit.client.1.smithi171.stdout:5/61: dwrite d5/ff [0,4194304] 0 2022-01-31T19:44:17.711 INFO:tasks.workunit.client.1.smithi171.stdout:9/36: dwrite fc [0,4194304] 0 2022-01-31T19:44:17.712 INFO:tasks.workunit.client.0.smithi167.stdout:3/851: dwrite d9/d26/fb8 [0,4194304] 0 2022-01-31T19:44:17.712 INFO:tasks.workunit.client.0.smithi167.stdout:3/852: fsync d9/d26/f6e 0 2022-01-31T19:44:17.712 INFO:tasks.workunit.client.0.smithi167.stdout:3/853: chown d9/d29/fb2 115515908 1 2022-01-31T19:44:17.713 INFO:tasks.workunit.client.0.smithi167.stdout:2/639: mkdir dd/df/d1b/d3d/d4a/da7/dcc 0 2022-01-31T19:44:17.713 INFO:tasks.workunit.client.0.smithi167.stdout:2/640: fsync dd/df/d22/f77 0 2022-01-31T19:44:17.713 INFO:tasks.workunit.client.0.smithi167.stdout:2/641: stat dd/df/c40 0 2022-01-31T19:44:17.714 INFO:tasks.workunit.client.0.smithi167.stdout:4/839: creat d0/d6/d7/f116 x:0 0 0 2022-01-31T19:44:17.714 INFO:tasks.workunit.client.0.smithi167.stdout:4/840: chown d0/d6/d7/d26/dd2/l115 4 1 2022-01-31T19:44:17.714 INFO:tasks.workunit.client.0.smithi167.stdout:4/841: chown d0/d6/d7/d26/d40 29340578 1 2022-01-31T19:44:17.715 INFO:tasks.workunit.client.1.smithi171.stdout:5/62: rename c2 to d5/c11 0 2022-01-31T19:44:17.715 INFO:tasks.workunit.client.1.smithi171.stdout:5/63: rmdir d5/d6 39 2022-01-31T19:44:17.728 INFO:tasks.workunit.client.0.smithi167.stdout:2/642: symlink dd/d18/d63/lcd 0 2022-01-31T19:44:17.729 INFO:tasks.workunit.client.1.smithi171.stdout:8/65: dwrite f2 [0,4194304] 0 2022-01-31T19:44:17.729 INFO:tasks.workunit.client.1.smithi171.stdout:8/66: creat df/f17 x:0 0 0 2022-01-31T19:44:17.731 INFO:tasks.workunit.client.1.smithi171.stdout:8/67: mkdir df/d18 0 2022-01-31T19:44:17.732 INFO:tasks.workunit.client.1.smithi171.stdout:8/68: mkdir df/d18/d19 0 2022-01-31T19:44:17.733 INFO:tasks.workunit.client.1.smithi171.stdout:8/69: mkdir df/d1a 0 2022-01-31T19:44:17.739 INFO:tasks.workunit.client.1.smithi171.stdout:7/83: dwrite d1/d1b/f10 [0,4194304] 0 2022-01-31T19:44:17.748 INFO:tasks.workunit.client.1.smithi171.stdout:1/35: dwrite f7 [0,4194304] 0 2022-01-31T19:44:17.761 INFO:tasks.workunit.client.1.smithi171.stdout:1/36: dread f7 [0,4194304] 0 2022-01-31T19:44:17.761 INFO:tasks.workunit.client.1.smithi171.stdout:1/37: readlink l6 0 2022-01-31T19:44:17.763 INFO:tasks.workunit.client.0.smithi167.stdout:8/794: dwrite d6/db/d28/d61/f89 [4194304,4194304] 0 2022-01-31T19:44:17.763 INFO:tasks.workunit.client.0.smithi167.stdout:8/795: fdatasync d6/db/de/f26 0 2022-01-31T19:44:17.764 INFO:tasks.workunit.client.1.smithi171.stdout:3/93: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:17.764 INFO:tasks.workunit.client.0.smithi167.stdout:8/796: chown d6/la0 1219 1 2022-01-31T19:44:17.765 INFO:tasks.workunit.client.0.smithi167.stdout:8/797: getdents d6/db/d28/d61/de3 0 2022-01-31T19:44:17.765 INFO:tasks.workunit.client.0.smithi167.stdout:8/798: fdatasync d6/db/de/d18/f21 0 2022-01-31T19:44:17.766 INFO:tasks.workunit.client.0.smithi167.stdout:0/596: dwrite d17/d19/f51 [0,4194304] 0 2022-01-31T19:44:17.766 INFO:tasks.workunit.client.0.smithi167.stdout:0/597: creat d17/d19/d1c/d39/d44/d49/fc2 x:0 0 0 2022-01-31T19:44:17.767 INFO:tasks.workunit.client.0.smithi167.stdout:4/842: dwrite d0/d51/df6/f10a [0,4194304] 0 2022-01-31T19:44:17.767 INFO:tasks.workunit.client.0.smithi167.stdout:4/843: fsync d0/d6/d7/d26/d40/d4c/f10b 0 2022-01-31T19:44:17.767 INFO:tasks.workunit.client.1.smithi171.stdout:7/84: dwrite d1/f16 [4194304,4194304] 0 2022-01-31T19:44:17.768 INFO:tasks.workunit.client.0.smithi167.stdout:3/854: dwrite d9/d26/f65 [0,4194304] 0 2022-01-31T19:44:17.768 INFO:tasks.workunit.client.1.smithi171.stdout:3/94: creat d4/dd/d1a/f1b x:0 0 0 2022-01-31T19:44:17.769 INFO:tasks.workunit.client.1.smithi171.stdout:3/95: stat d4/dd/c16 0 2022-01-31T19:44:17.769 INFO:tasks.workunit.client.1.smithi171.stdout:0/115: dwrite d9/f1e [0,4194304] 0 2022-01-31T19:44:17.770 INFO:tasks.workunit.client.1.smithi171.stdout:0/116: truncate d9/f17 432837 0 2022-01-31T19:44:17.772 INFO:tasks.workunit.client.1.smithi171.stdout:5/64: dwrite d5/ff [0,4194304] 0 2022-01-31T19:44:17.773 INFO:tasks.workunit.client.0.smithi167.stdout:2/643: dwrite dd/df/f1c [0,4194304] 0 2022-01-31T19:44:17.774 INFO:tasks.workunit.client.0.smithi167.stdout:7/991: fsync d8/d35/f112 0 2022-01-31T19:44:17.774 INFO:tasks.workunit.client.0.smithi167.stdout:7/992: write d8/d2c/d7a/dd3/f134 [768061,82353] 0 2022-01-31T19:44:17.774 INFO:tasks.workunit.client.0.smithi167.stdout:7/993: fdatasync d8/f45 0 2022-01-31T19:44:17.775 INFO:tasks.workunit.client.1.smithi171.stdout:9/37: dread fa [0,4194304] 0 2022-01-31T19:44:17.775 INFO:tasks.workunit.client.1.smithi171.stdout:9/38: write fb [389110,38070] 0 2022-01-31T19:44:17.777 INFO:tasks.workunit.client.0.smithi167.stdout:3/855: write d9/fc1 [103185,47565] 0 2022-01-31T19:44:17.778 INFO:tasks.workunit.client.0.smithi167.stdout:0/598: creat d17/d19/d1c/fc3 x:0 0 0 2022-01-31T19:44:17.779 INFO:tasks.workunit.client.1.smithi171.stdout:9/39: write f4 [4190785,86571] 0 2022-01-31T19:44:17.779 INFO:tasks.workunit.client.1.smithi171.stdout:9/40: write fb [577657,52788] 0 2022-01-31T19:44:17.791 INFO:tasks.workunit.client.1.smithi171.stdout:6/135: rmdir dc 39 2022-01-31T19:44:17.796 INFO:tasks.workunit.client.1.smithi171.stdout:0/117: creat d9/d1b/d25/f29 x:0 0 0 2022-01-31T19:44:17.798 INFO:tasks.workunit.client.1.smithi171.stdout:9/41: mkdir dd 0 2022-01-31T19:44:17.799 INFO:tasks.workunit.client.1.smithi171.stdout:9/42: fdatasync f9 0 2022-01-31T19:44:17.799 INFO:tasks.workunit.client.1.smithi171.stdout:6/136: symlink dc/l28 0 2022-01-31T19:44:17.800 INFO:tasks.workunit.client.0.smithi167.stdout:4/844: dwrite d0/d6/d7/d26/d31/d5c/fd7 [0,4194304] 0 2022-01-31T19:44:17.800 INFO:tasks.workunit.client.1.smithi171.stdout:0/118: symlink d9/d1b/d25/l2a 0 2022-01-31T19:44:17.802 INFO:tasks.workunit.client.1.smithi171.stdout:6/137: rename dc/d21/d23 to dc/d19/d29 0 2022-01-31T19:44:17.803 INFO:tasks.workunit.client.1.smithi171.stdout:1/38: dwrite f7 [0,4194304] 0 2022-01-31T19:44:17.803 INFO:tasks.workunit.client.1.smithi171.stdout:1/39: write f3 [879019,59564] 0 2022-01-31T19:44:17.803 INFO:tasks.workunit.client.1.smithi171.stdout:1/40: truncate f8 197566 0 2022-01-31T19:44:17.804 INFO:tasks.workunit.client.1.smithi171.stdout:1/41: creat f9 x:0 0 0 2022-01-31T19:44:17.804 INFO:tasks.workunit.client.1.smithi171.stdout:5/65: dwrite d5/f8 [0,4194304] 0 2022-01-31T19:44:17.805 INFO:tasks.workunit.client.1.smithi171.stdout:6/138: creat dc/d18/f2a x:0 0 0 2022-01-31T19:44:17.805 INFO:tasks.workunit.client.1.smithi171.stdout:3/96: dwrite d4/fb [0,4194304] 0 2022-01-31T19:44:17.806 INFO:tasks.workunit.client.1.smithi171.stdout:0/119: dread f8 [0,4194304] 0 2022-01-31T19:44:17.807 INFO:tasks.workunit.client.1.smithi171.stdout:0/120: chown d9/d1b/d25/l2a 320762398 1 2022-01-31T19:44:17.808 INFO:tasks.workunit.client.1.smithi171.stdout:5/66: rename d5/d6/la to d5/l12 0 2022-01-31T19:44:17.808 INFO:tasks.workunit.client.1.smithi171.stdout:5/67: dread - d5/fe zero size 2022-01-31T19:44:17.809 INFO:tasks.workunit.client.1.smithi171.stdout:5/68: rename d5 to d5/d13 22 2022-01-31T19:44:17.809 INFO:tasks.workunit.client.0.smithi167.stdout:5/979: sync 2022-01-31T19:44:17.810 INFO:tasks.workunit.client.0.smithi167.stdout:5/980: getdents d5/d6/de/d22 0 2022-01-31T19:44:17.810 INFO:tasks.workunit.client.0.smithi167.stdout:5/981: read - d5/d6/de/d22/d76/fd9 zero size 2022-01-31T19:44:17.810 INFO:tasks.workunit.client.0.smithi167.stdout:5/982: chown d5/d6/de/d22/d76/lac 13977865 1 2022-01-31T19:44:17.810 INFO:tasks.workunit.client.0.smithi167.stdout:5/983: readlink d5/dcd/leb 0 2022-01-31T19:44:17.811 INFO:tasks.workunit.client.0.smithi167.stdout:9/718: sync 2022-01-31T19:44:17.811 INFO:tasks.workunit.client.0.smithi167.stdout:2/644: link dd/df/d1b/c45 dd/df/d1b/d3d/d4a/da7/dcc/cce 0 2022-01-31T19:44:17.811 INFO:tasks.workunit.client.0.smithi167.stdout:3/856: symlink d9/d7f/l103 0 2022-01-31T19:44:17.811 INFO:tasks.workunit.client.0.smithi167.stdout:3/857: readlink d9/d29/d60/lea 0 2022-01-31T19:44:17.812 INFO:tasks.workunit.client.0.smithi167.stdout:3/858: chown d9/d29/d3b/d9b/da3/fde 631962 1 2022-01-31T19:44:17.812 INFO:tasks.workunit.client.0.smithi167.stdout:3/859: chown d9/d29/d3b/d41/deb/lf2 43 1 2022-01-31T19:44:17.813 INFO:tasks.workunit.client.0.smithi167.stdout:0/599: creat d17/d2c/d87/fc4 x:0 0 0 2022-01-31T19:44:17.813 INFO:tasks.workunit.client.0.smithi167.stdout:8/799: rename d6/db/de/d18/da4 to d6/db/d28/d61/dd5/dbf/d105 0 2022-01-31T19:44:17.813 INFO:tasks.workunit.client.0.smithi167.stdout:8/800: chown d6/d1e/fd3 433098118 1 2022-01-31T19:44:17.814 INFO:tasks.workunit.client.0.smithi167.stdout:8/801: write d6/db/d28/d66/f8e [2147682,99214] 0 2022-01-31T19:44:17.814 INFO:tasks.workunit.client.0.smithi167.stdout:8/802: readlink d6/db/d36/la6 0 2022-01-31T19:44:17.815 INFO:tasks.workunit.client.1.smithi171.stdout:1/42: write f7 [57672,17487] 0 2022-01-31T19:44:17.815 INFO:tasks.workunit.client.1.smithi171.stdout:5/69: creat d5/d6/f14 x:0 0 0 2022-01-31T19:44:17.815 INFO:tasks.workunit.client.1.smithi171.stdout:1/43: rename f8 to fa 0 2022-01-31T19:44:17.815 INFO:tasks.workunit.client.1.smithi171.stdout:0/121: write f6 [2981468,96324] 0 2022-01-31T19:44:17.816 INFO:tasks.workunit.client.1.smithi171.stdout:1/44: rename c5 to cb 0 2022-01-31T19:44:17.816 INFO:tasks.workunit.client.0.smithi167.stdout:4/845: truncate d0/d6/d7/d89/fa9 83722 0 2022-01-31T19:44:17.817 INFO:tasks.workunit.client.1.smithi171.stdout:7/85: dwrite d1/d1b/d7/fe [0,4194304] 0 2022-01-31T19:44:17.817 INFO:tasks.workunit.client.1.smithi171.stdout:7/86: fsync d1/f16 0 2022-01-31T19:44:17.817 INFO:tasks.workunit.client.1.smithi171.stdout:7/87: fsync d1/d1b/d7/f19 0 2022-01-31T19:44:17.818 INFO:tasks.workunit.client.1.smithi171.stdout:1/45: mknod cc 0 2022-01-31T19:44:17.819 INFO:tasks.workunit.client.0.smithi167.stdout:4/846: getdents d0/d6/d7/d26/dd2 0 2022-01-31T19:44:17.820 INFO:tasks.workunit.client.1.smithi171.stdout:0/122: creat d9/f2b x:0 0 0 2022-01-31T19:44:17.820 INFO:tasks.workunit.client.0.smithi167.stdout:5/984: write d5/d6/de/d22/d76/da3/f127 [2722284,57244] 0 2022-01-31T19:44:17.822 INFO:tasks.workunit.client.0.smithi167.stdout:7/994: rename d8/dc/d18/d14a/dee/df8/d132 to d8/dfb/d16a 0 2022-01-31T19:44:17.822 INFO:tasks.workunit.client.0.smithi167.stdout:7/995: dread - d8/d35/f112 zero size 2022-01-31T19:44:17.824 INFO:tasks.workunit.client.1.smithi171.stdout:1/46: mkdir dd 0 2022-01-31T19:44:17.824 INFO:tasks.workunit.client.1.smithi171.stdout:1/47: write f7 [4882863,106735] 0 2022-01-31T19:44:17.826 INFO:tasks.workunit.client.0.smithi167.stdout:4/847: write d0/d6/d7/d89/fa3 [624524,64756] 0 2022-01-31T19:44:17.826 INFO:tasks.workunit.client.0.smithi167.stdout:2/645: dread dd/d18/fa9 [0,4194304] 0 2022-01-31T19:44:17.827 INFO:tasks.workunit.client.0.smithi167.stdout:2/646: chown dd/df/d3f/d51/l57 80123 1 2022-01-31T19:44:17.828 INFO:tasks.workunit.client.0.smithi167.stdout:5/985: mknod d5/d6/d16/d18/d5b/d63/c135 0 2022-01-31T19:44:17.828 INFO:tasks.workunit.client.0.smithi167.stdout:5/986: unlink d5/f8 0 2022-01-31T19:44:17.829 INFO:tasks.workunit.client.0.smithi167.stdout:5/987: truncate d5/d6/de/f37 8415859 0 2022-01-31T19:44:17.829 INFO:tasks.workunit.client.0.smithi167.stdout:4/848: mkdir d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d117 0 2022-01-31T19:44:17.829 INFO:tasks.workunit.client.0.smithi167.stdout:2/647: mknod dd/d18/d63/ccf 0 2022-01-31T19:44:17.832 INFO:tasks.workunit.client.0.smithi167.stdout:5/988: rename d5/d6/d16/d18/d43/d45/dd5/dbc/f116 to d5/dcd/f136 0 2022-01-31T19:44:17.833 INFO:tasks.workunit.client.0.smithi167.stdout:5/989: fdatasync d5/f10e 0 2022-01-31T19:44:17.833 INFO:tasks.workunit.client.0.smithi167.stdout:5/990: fdatasync d5/d6/f28 0 2022-01-31T19:44:17.834 INFO:tasks.workunit.client.0.smithi167.stdout:7/996: read d8/d2c/d6e/d4d/d5e/d71/db1/f6a [1890632,54324] 0 2022-01-31T19:44:17.834 INFO:tasks.workunit.client.0.smithi167.stdout:2/648: creat dd/d18/d63/fd0 x:0 0 0 2022-01-31T19:44:17.834 INFO:tasks.workunit.client.0.smithi167.stdout:2/649: read - dd/fc1 zero size 2022-01-31T19:44:17.835 INFO:tasks.workunit.client.0.smithi167.stdout:2/650: read - dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b zero size 2022-01-31T19:44:17.836 INFO:tasks.workunit.client.0.smithi167.stdout:5/991: mkdir d5/d6/d16/d18/d43/d45/d69/d90/d137 0 2022-01-31T19:44:17.836 INFO:tasks.workunit.client.0.smithi167.stdout:5/992: chown d5/d6/de/c132 4115758 1 2022-01-31T19:44:17.836 INFO:tasks.workunit.client.0.smithi167.stdout:5/993: creat d5/d6/de/f138 x:0 0 0 2022-01-31T19:44:17.837 INFO:tasks.workunit.client.0.smithi167.stdout:7/997: truncate d8/d2c/d6e/d4d/d5e/fa9 1252453 0 2022-01-31T19:44:17.840 INFO:tasks.workunit.client.1.smithi171.stdout:7/88: truncate d1/f5 2131635 0 2022-01-31T19:44:17.841 INFO:tasks.workunit.client.1.smithi171.stdout:7/89: chown d1/d1b/f10 8 1 2022-01-31T19:44:17.841 INFO:tasks.workunit.client.1.smithi171.stdout:1/48: mknod dd/ce 0 2022-01-31T19:44:17.841 INFO:tasks.workunit.client.1.smithi171.stdout:1/49: truncate fa 1000277 0 2022-01-31T19:44:17.842 INFO:tasks.workunit.client.1.smithi171.stdout:7/90: mknod d1/c1d 0 2022-01-31T19:44:17.843 INFO:tasks.workunit.client.0.smithi167.stdout:5/994: rename d5/d6/de/d22/d4d/c86 to d5/d6/d16/d18/d43/d45/dd5/dbc/dc0/c139 0 2022-01-31T19:44:17.843 INFO:tasks.workunit.client.0.smithi167.stdout:4/849: dread d0/d6/d7/d26/d31/f71 [0,4194304] 0 2022-01-31T19:44:17.843 INFO:tasks.workunit.client.0.smithi167.stdout:4/850: truncate d0/d6/d7/d26/d31/ddf/d75/dd6/f106 515041 0 2022-01-31T19:44:17.843 INFO:tasks.workunit.client.0.smithi167.stdout:8/803: dread d6/db/d4e/d73/f7b [0,4194304] 0 2022-01-31T19:44:17.844 INFO:tasks.workunit.client.1.smithi171.stdout:1/50: dread fa [0,4194304] 0 2022-01-31T19:44:17.844 INFO:tasks.workunit.client.1.smithi171.stdout:1/51: mkdir dd/df 0 2022-01-31T19:44:17.847 INFO:tasks.workunit.client.0.smithi167.stdout:8/804: getdents d6/d1e/d4c 0 2022-01-31T19:44:17.847 INFO:tasks.workunit.client.1.smithi171.stdout:2/153: sync 2022-01-31T19:44:17.848 INFO:tasks.workunit.client.1.smithi171.stdout:2/154: symlink d7/dd/l2b 0 2022-01-31T19:44:17.849 INFO:tasks.workunit.client.1.smithi171.stdout:2/155: write d7/f19 [120180,73294] 0 2022-01-31T19:44:17.851 INFO:tasks.workunit.client.1.smithi171.stdout:2/156: getdents d7/d26 0 2022-01-31T19:44:17.851 INFO:tasks.workunit.client.1.smithi171.stdout:2/157: chown d7/d9/f23 54 1 2022-01-31T19:44:17.852 INFO:tasks.workunit.client.1.smithi171.stdout:2/158: creat d7/d9/f2c x:0 0 0 2022-01-31T19:44:17.853 INFO:tasks.workunit.client.0.smithi167.stdout:7/998: rename d8/dfb/f14c to d8/dc/f16b 0 2022-01-31T19:44:17.853 INFO:tasks.workunit.client.0.smithi167.stdout:7/999: chown d8/d35/fbf 1540382966 1 2022-01-31T19:44:17.854 INFO:tasks.workunit.client.0.smithi167.stdout:4/851: truncate d0/d6/d7/d26/d40/d4c/dfe/fb7 5421092 0 2022-01-31T19:44:17.857 INFO:tasks.workunit.client.0.smithi167.stdout:4/852: mkdir d0/d6/d7/d26/dd2/de5/de6/d118 0 2022-01-31T19:44:17.857 INFO:tasks.workunit.client.0.smithi167.stdout:4/853: dread - d0/d6/d7/d26/ff5 zero size 2022-01-31T19:44:17.858 INFO:tasks.workunit.client.0.smithi167.stdout:4/854: mkdir d0/d6/d7/d26/d31/d5c/de2/d119 0 2022-01-31T19:44:17.858 INFO:tasks.workunit.client.0.smithi167.stdout:4/855: dread - d0/d6/dab/fde zero size 2022-01-31T19:44:17.859 INFO:tasks.workunit.client.1.smithi171.stdout:7/91: dread d1/d1b/fc [0,4194304] 0 2022-01-31T19:44:17.860 INFO:tasks.workunit.client.0.smithi167.stdout:4/856: rmdir d0/d6/d7/d26/d31/ddf/d75 39 2022-01-31T19:44:17.862 INFO:tasks.workunit.client.0.smithi167.stdout:4/857: mkdir d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d113/d11a 0 2022-01-31T19:44:17.862 INFO:tasks.workunit.client.0.smithi167.stdout:4/858: write d0/d6/dab/ff3 [844919,40033] 0 2022-01-31T19:44:17.863 INFO:tasks.workunit.client.0.smithi167.stdout:4/859: mknod d0/d6/dab/c11b 0 2022-01-31T19:44:17.865 INFO:tasks.workunit.client.0.smithi167.stdout:4/860: mkdir d0/d6/d7/d89/d11c 0 2022-01-31T19:44:17.867 INFO:tasks.workunit.client.0.smithi167.stdout:4/861: rename d0/l85 to d0/d6/d7/d26/d31/ddf/d75/dd6/l11d 0 2022-01-31T19:44:17.868 INFO:tasks.workunit.client.0.smithi167.stdout:4/862: stat d0/d6/d7/d26/d50/db3/cd4 0 2022-01-31T19:44:17.868 INFO:tasks.workunit.client.0.smithi167.stdout:4/863: mknod d0/d6/d7/d26/d31/ddf/d75/dd6/c11e 0 2022-01-31T19:44:17.869 INFO:tasks.workunit.client.0.smithi167.stdout:4/864: write d0/d5d/fa5 [815385,28753] 0 2022-01-31T19:44:17.870 INFO:tasks.workunit.client.0.smithi167.stdout:4/865: creat d0/d6/d7/d26/dd2/de5/f11f x:0 0 0 2022-01-31T19:44:17.871 INFO:tasks.workunit.client.0.smithi167.stdout:9/719: dwrite dd/d19/d24/d56/d5b/fbe [0,4194304] 0 2022-01-31T19:44:17.871 INFO:tasks.workunit.client.0.smithi167.stdout:9/720: chown dd/d34 76578363 1 2022-01-31T19:44:17.872 INFO:tasks.workunit.client.0.smithi167.stdout:4/866: creat d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d113/f120 x:0 0 0 2022-01-31T19:44:17.872 INFO:tasks.workunit.client.0.smithi167.stdout:4/867: dread - d0/d6/d7/d26/d40/d4c/ffd zero size 2022-01-31T19:44:17.873 INFO:tasks.workunit.client.0.smithi167.stdout:0/600: dwrite d17/d19/d1c/d39/d44/d49/f80 [0,4194304] 0 2022-01-31T19:44:17.873 INFO:tasks.workunit.client.0.smithi167.stdout:0/601: dread - d17/d3e/fae zero size 2022-01-31T19:44:17.873 INFO:tasks.workunit.client.0.smithi167.stdout:0/602: readlink l12 0 2022-01-31T19:44:17.873 INFO:tasks.workunit.client.0.smithi167.stdout:0/603: chown d17/d2c/f76 52153 1 2022-01-31T19:44:17.874 INFO:tasks.workunit.client.0.smithi167.stdout:3/860: dwrite d9/d7f/f85 [0,4194304] 0 2022-01-31T19:44:17.875 INFO:tasks.workunit.client.1.smithi171.stdout:3/97: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:17.875 INFO:tasks.workunit.client.1.smithi171.stdout:3/98: readlink d4/dd/l15 0 2022-01-31T19:44:17.875 INFO:tasks.workunit.client.1.smithi171.stdout:3/99: unlink d4/dd/c10 0 2022-01-31T19:44:17.876 INFO:tasks.workunit.client.1.smithi171.stdout:3/100: write d4/dd/d1a/f1b [680741,67505] 0 2022-01-31T19:44:17.876 INFO:tasks.workunit.client.0.smithi167.stdout:1/890: sync 2022-01-31T19:44:17.876 INFO:tasks.workunit.client.0.smithi167.stdout:6/693: sync 2022-01-31T19:44:17.877 INFO:tasks.workunit.client.1.smithi171.stdout:9/43: dwrite fb [0,4194304] 0 2022-01-31T19:44:17.878 INFO:tasks.workunit.client.0.smithi167.stdout:9/721: mknod dd/d19/d24/d50/d5d/cf5 0 2022-01-31T19:44:17.879 INFO:tasks.workunit.client.0.smithi167.stdout:9/722: write dd/d19/d24/d56/dc9/fe4 [807061,129818] 0 2022-01-31T19:44:17.879 INFO:tasks.workunit.client.1.smithi171.stdout:2/159: dwrite d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:17.880 INFO:tasks.workunit.client.1.smithi171.stdout:6/139: rmdir dc/d18 39 2022-01-31T19:44:17.880 INFO:tasks.workunit.client.1.smithi171.stdout:5/70: dwrite d5/d6/f14 [0,4194304] 0 2022-01-31T19:44:17.881 INFO:tasks.workunit.client.1.smithi171.stdout:0/123: dwrite d9/f2b [0,4194304] 0 2022-01-31T19:44:17.881 INFO:tasks.workunit.client.0.smithi167.stdout:5/995: dwrite d5/d6/d16/f9d [0,4194304] 0 2022-01-31T19:44:17.883 INFO:tasks.workunit.client.1.smithi171.stdout:9/44: creat dd/fe x:0 0 0 2022-01-31T19:44:17.884 INFO:tasks.workunit.client.1.smithi171.stdout:9/45: dread - dd/fe zero size 2022-01-31T19:44:17.885 INFO:tasks.workunit.client.1.smithi171.stdout:2/160: mknod d7/d9/c2d 0 2022-01-31T19:44:17.887 INFO:tasks.workunit.client.1.smithi171.stdout:6/140: mknod dc/d21/c2b 0 2022-01-31T19:44:17.892 INFO:tasks.workunit.client.1.smithi171.stdout:6/141: write fa [386195,66625] 0 2022-01-31T19:44:17.893 INFO:tasks.workunit.client.0.smithi167.stdout:0/604: rename d17/d3e/d7e/fab to d17/d19/d1c/d39/fc5 0 2022-01-31T19:44:17.894 INFO:tasks.workunit.client.1.smithi171.stdout:5/71: mkdir d5/d6/d15 0 2022-01-31T19:44:17.894 INFO:tasks.workunit.client.1.smithi171.stdout:5/72: readlink d5/lc 0 2022-01-31T19:44:17.895 INFO:tasks.workunit.client.1.smithi171.stdout:1/52: getdents dd 0 2022-01-31T19:44:17.897 INFO:tasks.workunit.client.0.smithi167.stdout:3/861: creat d9/d29/d60/df8/f104 x:0 0 0 2022-01-31T19:44:17.902 INFO:tasks.workunit.client.0.smithi167.stdout:1/891: mknod d0/d10/d5a/d33/d4b/da5/da9/dad/c12d 0 2022-01-31T19:44:17.903 INFO:tasks.workunit.client.0.smithi167.stdout:4/868: fsync d0/d6/d7/d26/d40/f24 0 2022-01-31T19:44:17.903 INFO:tasks.workunit.client.0.smithi167.stdout:9/723: unlink dd/d19/c1e 0 2022-01-31T19:44:17.905 INFO:tasks.workunit.client.1.smithi171.stdout:3/101: rmdir d4 39 2022-01-31T19:44:17.907 INFO:tasks.workunit.client.1.smithi171.stdout:6/142: creat dc/d18/f2c x:0 0 0 2022-01-31T19:44:17.907 INFO:tasks.workunit.client.1.smithi171.stdout:6/143: stat dc/d18/c20 0 2022-01-31T19:44:17.907 INFO:tasks.workunit.client.1.smithi171.stdout:6/144: stat dc/c12 0 2022-01-31T19:44:17.908 INFO:tasks.workunit.client.1.smithi171.stdout:5/73: truncate d5/f8 4061864 0 2022-01-31T19:44:17.910 INFO:tasks.workunit.client.1.smithi171.stdout:3/102: mknod d4/dd/d1a/c1c 0 2022-01-31T19:44:17.910 INFO:tasks.workunit.client.1.smithi171.stdout:6/145: rename dc/lf to dc/d19/l2d 0 2022-01-31T19:44:17.911 INFO:tasks.workunit.client.1.smithi171.stdout:6/146: chown dc/d1e 1986046532 1 2022-01-31T19:44:17.912 INFO:tasks.workunit.client.1.smithi171.stdout:5/74: rename d5/fe to d5/d6/f16 0 2022-01-31T19:44:17.913 INFO:tasks.workunit.client.0.smithi167.stdout:1/892: write d0/d10/f19 [3401141,80071] 0 2022-01-31T19:44:17.913 INFO:tasks.workunit.client.1.smithi171.stdout:6/147: creat dc/d21/f2e x:0 0 0 2022-01-31T19:44:17.914 INFO:tasks.workunit.client.1.smithi171.stdout:5/75: symlink d5/d6/dd/l17 0 2022-01-31T19:44:17.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/605: link d17/d19/d1c/d39/d7b/lb3 d17/d2c/lc6 0 2022-01-31T19:44:17.918 INFO:tasks.workunit.client.0.smithi167.stdout:3/862: mknod d9/d29/d3b/d41/d6b/c105 0 2022-01-31T19:44:17.919 INFO:tasks.workunit.client.0.smithi167.stdout:3/863: dread - d9/d29/d3b/d9b/f102 zero size 2022-01-31T19:44:17.920 INFO:tasks.workunit.client.0.smithi167.stdout:6/694: mknod d8/d33/d51/d72/cfb 0 2022-01-31T19:44:17.920 INFO:tasks.workunit.client.0.smithi167.stdout:6/695: read - d8/d33/d51/d5d/d86/dbd/fd4 zero size 2022-01-31T19:44:17.920 INFO:tasks.workunit.client.0.smithi167.stdout:6/696: write d8/d33/d51/d7e/dc4/fba [801751,59829] 0 2022-01-31T19:44:17.921 INFO:tasks.workunit.client.0.smithi167.stdout:9/724: getdents dd/d17/d80 0 2022-01-31T19:44:17.921 INFO:tasks.workunit.client.0.smithi167.stdout:4/869: link d0/d6/d7/d26/d31/ddf/d1a/c6e d0/d6/d7/d26/d31/ddf/da2/c121 0 2022-01-31T19:44:17.922 INFO:tasks.workunit.client.0.smithi167.stdout:4/870: creat d0/d6/d7/d26/f122 x:0 0 0 2022-01-31T19:44:17.922 INFO:tasks.workunit.client.0.smithi167.stdout:4/871: chown d0/d6/d7/d26/d31/fc7 227790025 1 2022-01-31T19:44:17.923 INFO:tasks.workunit.client.0.smithi167.stdout:0/606: dread d17/f31 [0,4194304] 0 2022-01-31T19:44:17.923 INFO:tasks.workunit.client.0.smithi167.stdout:0/607: fdatasync d17/d3e/d7e/f73 0 2022-01-31T19:44:17.924 INFO:tasks.workunit.client.0.smithi167.stdout:1/893: rename d0/d75/f52 to d0/d62/d7f/d82/f12e 0 2022-01-31T19:44:17.925 INFO:tasks.workunit.client.0.smithi167.stdout:6/697: unlink d8/d62/d92/laf 0 2022-01-31T19:44:17.927 INFO:tasks.workunit.client.1.smithi171.stdout:5/76: dread f1 [0,4194304] 0 2022-01-31T19:44:17.928 INFO:tasks.workunit.client.0.smithi167.stdout:1/894: rename d0/db3/feb to d0/d10/d20/f12f 0 2022-01-31T19:44:17.929 INFO:tasks.workunit.client.0.smithi167.stdout:1/895: dread - d0/d62/d7f/d82/fe2 zero size 2022-01-31T19:44:17.929 INFO:tasks.workunit.client.0.smithi167.stdout:1/896: chown d0/d11/d102/ddb 887597 1 2022-01-31T19:44:17.929 INFO:tasks.workunit.client.0.smithi167.stdout:1/897: fsync d0/d62/d7f/fdd 0 2022-01-31T19:44:17.929 INFO:tasks.workunit.client.0.smithi167.stdout:1/898: write d0/d10/d5a/df2/f103 [147445,123059] 0 2022-01-31T19:44:17.930 INFO:tasks.workunit.client.0.smithi167.stdout:1/899: chown d0/d62/fa3 15 1 2022-01-31T19:44:17.930 INFO:tasks.workunit.client.0.smithi167.stdout:1/900: chown d0/d10/d20/l54 2 1 2022-01-31T19:44:17.930 INFO:tasks.workunit.client.0.smithi167.stdout:1/901: chown d0/d10/d5a/d33/d4b/da5/da9/dad/f108 1667985210 1 2022-01-31T19:44:17.931 INFO:tasks.workunit.client.0.smithi167.stdout:4/872: unlink d0/d6/d7/d26/f122 0 2022-01-31T19:44:17.931 INFO:tasks.workunit.client.0.smithi167.stdout:8/805: dwrite d6/db/d28/d66/f8e [0,4194304] 0 2022-01-31T19:44:17.932 INFO:tasks.workunit.client.0.smithi167.stdout:8/806: chown d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/ff9 169820 1 2022-01-31T19:44:17.932 INFO:tasks.workunit.client.0.smithi167.stdout:4/873: rename d0/d6/d7/d26/d31/ddf/d75/de3 to d0/d6/d7/d26/d31/ddf/d1a/d123 0 2022-01-31T19:44:17.932 INFO:tasks.workunit.client.0.smithi167.stdout:4/874: fdatasync d0/d6/d7/d26/d40/d4c/ffd 0 2022-01-31T19:44:17.932 INFO:tasks.workunit.client.0.smithi167.stdout:4/875: dread - d0/d6/d7/f116 zero size 2022-01-31T19:44:17.933 INFO:tasks.workunit.client.1.smithi171.stdout:5/77: dread f1 [0,4194304] 0 2022-01-31T19:44:17.933 INFO:tasks.workunit.client.0.smithi167.stdout:1/902: rmdir d0/d75/d7a 39 2022-01-31T19:44:17.934 INFO:tasks.workunit.client.0.smithi167.stdout:3/864: dread d9/d7f/da8/ffd [0,4194304] 0 2022-01-31T19:44:17.934 INFO:tasks.workunit.client.0.smithi167.stdout:3/865: chown d9/d56/c61 225462907 1 2022-01-31T19:44:17.935 INFO:tasks.workunit.client.0.smithi167.stdout:8/807: rename d6/db/d28/c46 to d6/db/de/d18/deb/c106 0 2022-01-31T19:44:17.935 INFO:tasks.workunit.client.0.smithi167.stdout:8/808: stat d6/db/d4e/fd0 0 2022-01-31T19:44:17.936 INFO:tasks.workunit.client.0.smithi167.stdout:4/876: symlink d0/d6/d7/d26/d50/db3/l124 0 2022-01-31T19:44:17.936 INFO:tasks.workunit.client.0.smithi167.stdout:4/877: write d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d113/f120 [394917,97218] 0 2022-01-31T19:44:17.937 INFO:tasks.workunit.client.1.smithi171.stdout:9/46: dwrite f9 [0,4194304] 0 2022-01-31T19:44:17.938 INFO:tasks.workunit.client.0.smithi167.stdout:3/866: rename d9/d29/d60/f69 to d9/d7f/da8/f106 0 2022-01-31T19:44:17.939 INFO:tasks.workunit.client.0.smithi167.stdout:8/809: unlink d6/db/d4e/d73/fa2 0 2022-01-31T19:44:17.940 INFO:tasks.workunit.client.1.smithi171.stdout:9/47: unlink c5 0 2022-01-31T19:44:17.941 INFO:tasks.workunit.client.1.smithi171.stdout:0/124: dwrite d9/f13 [0,4194304] 0 2022-01-31T19:44:17.941 INFO:tasks.workunit.client.1.smithi171.stdout:2/161: dwrite d7/dd/f15 [0,4194304] 0 2022-01-31T19:44:17.942 INFO:tasks.workunit.client.1.smithi171.stdout:9/48: mkdir dd/df 0 2022-01-31T19:44:17.943 INFO:tasks.workunit.client.0.smithi167.stdout:4/878: link d0/d6/d7/d26/d31/ddf/da2/fc4 d0/d6/d7/d26/d31/ddf/da2/f125 0 2022-01-31T19:44:17.943 INFO:tasks.workunit.client.0.smithi167.stdout:4/879: read - d0/d6/d7/d26/dd2/de5/f11f zero size 2022-01-31T19:44:17.943 INFO:tasks.workunit.client.0.smithi167.stdout:4/880: truncate d0/d6/d7/d26/f2f 2067759 0 2022-01-31T19:44:17.943 INFO:tasks.workunit.client.0.smithi167.stdout:4/881: fsync d0/d6/d7/f116 0 2022-01-31T19:44:17.944 INFO:tasks.workunit.client.1.smithi171.stdout:0/125: mkdir d9/db/d2c 0 2022-01-31T19:44:17.945 INFO:tasks.workunit.client.1.smithi171.stdout:0/126: chown d9 128763 1 2022-01-31T19:44:17.945 INFO:tasks.workunit.client.1.smithi171.stdout:9/49: mknod dd/c10 0 2022-01-31T19:44:17.946 INFO:tasks.workunit.client.0.smithi167.stdout:3/867: rmdir d9/d29/d3b 39 2022-01-31T19:44:17.946 INFO:tasks.workunit.client.0.smithi167.stdout:3/868: fsync d9/d29/d3b/d9b/f102 0 2022-01-31T19:44:17.946 INFO:tasks.workunit.client.0.smithi167.stdout:5/996: dwrite d5/d6/d16/d18/d43/d45/f94 [0,4194304] 0 2022-01-31T19:44:17.946 INFO:tasks.workunit.client.0.smithi167.stdout:4/882: creat d0/d6/d7/d26/d31/ddf/d75/dd6/d111/f126 x:0 0 0 2022-01-31T19:44:17.948 INFO:tasks.workunit.client.0.smithi167.stdout:1/903: dread d0/d10/d5a/fb9 [0,4194304] 0 2022-01-31T19:44:17.948 INFO:tasks.workunit.client.0.smithi167.stdout:3/869: creat d9/d29/d3b/d41/d6b/d100/f107 x:0 0 0 2022-01-31T19:44:17.948 INFO:tasks.workunit.client.0.smithi167.stdout:9/725: dwrite dd/ff [0,4194304] 0 2022-01-31T19:44:17.949 INFO:tasks.workunit.client.1.smithi171.stdout:3/103: dwrite d4/dd/d1a/f1b [0,4194304] 0 2022-01-31T19:44:17.949 INFO:tasks.workunit.client.1.smithi171.stdout:3/104: truncate d4/dd/d1a/f1b 4523576 0 2022-01-31T19:44:17.949 INFO:tasks.workunit.client.1.smithi171.stdout:0/127: rename d9/db/f12 to d9/d1b/f2d 0 2022-01-31T19:44:17.949 INFO:tasks.workunit.client.1.smithi171.stdout:1/53: truncate f7 354808 0 2022-01-31T19:44:17.950 INFO:tasks.workunit.client.1.smithi171.stdout:2/162: write d7/dd/f15 [4282430,124006] 0 2022-01-31T19:44:17.954 INFO:tasks.workunit.client.1.smithi171.stdout:9/50: rename fb to dd/f11 0 2022-01-31T19:44:17.954 INFO:tasks.workunit.client.1.smithi171.stdout:4/89: sync 2022-01-31T19:44:17.955 INFO:tasks.workunit.client.1.smithi171.stdout:8/70: sync 2022-01-31T19:44:17.955 INFO:tasks.workunit.client.1.smithi171.stdout:8/71: dread - df/f16 zero size 2022-01-31T19:44:17.958 INFO:tasks.workunit.client.0.smithi167.stdout:3/870: read d9/d29/d3b/d41/d6b/d89/f97 [484886,97973] 0 2022-01-31T19:44:17.958 INFO:tasks.workunit.client.0.smithi167.stdout:3/871: chown d9/d7f/da8/ffd 1852806 1 2022-01-31T19:44:17.958 INFO:tasks.workunit.client.0.smithi167.stdout:3/872: chown d9/d29/d60/lb7 4 1 2022-01-31T19:44:17.960 INFO:tasks.workunit.client.1.smithi171.stdout:1/54: link f3 dd/df/f10 0 2022-01-31T19:44:17.960 INFO:tasks.workunit.client.1.smithi171.stdout:1/55: chown dd/df 14245351 1 2022-01-31T19:44:17.961 INFO:tasks.workunit.client.0.smithi167.stdout:3/873: dread d9/fc1 [0,4194304] 0 2022-01-31T19:44:17.962 INFO:tasks.workunit.client.0.smithi167.stdout:1/904: mknod d0/d75/dbd/c130 0 2022-01-31T19:44:17.963 INFO:tasks.workunit.client.1.smithi171.stdout:2/163: mknod d7/d26/c2e 0 2022-01-31T19:44:17.964 INFO:tasks.workunit.client.1.smithi171.stdout:9/51: creat dd/df/f12 x:0 0 0 2022-01-31T19:44:17.965 INFO:tasks.workunit.client.1.smithi171.stdout:4/90: rename f1 to d2/f1e 0 2022-01-31T19:44:17.966 INFO:tasks.workunit.client.1.smithi171.stdout:4/91: stat d2/f1e 0 2022-01-31T19:44:17.966 INFO:tasks.workunit.client.1.smithi171.stdout:4/92: readlink d2/d3/d5/d11/d12/l17 0 2022-01-31T19:44:17.966 INFO:tasks.workunit.client.1.smithi171.stdout:0/128: getdents d9/d24 0 2022-01-31T19:44:17.969 INFO:tasks.workunit.client.1.smithi171.stdout:2/164: creat d7/dd/d27/f2f x:0 0 0 2022-01-31T19:44:17.969 INFO:tasks.workunit.client.1.smithi171.stdout:3/105: dread d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:17.970 INFO:tasks.workunit.client.1.smithi171.stdout:3/106: readlink d4/l18 0 2022-01-31T19:44:17.970 INFO:tasks.workunit.client.1.smithi171.stdout:0/129: rename d9/db/d2c to d9/db/dc/d2e 0 2022-01-31T19:44:17.971 INFO:tasks.workunit.client.1.smithi171.stdout:0/130: readlink d9/l18 0 2022-01-31T19:44:17.972 INFO:tasks.workunit.client.1.smithi171.stdout:2/165: truncate d7/f14 1244994 0 2022-01-31T19:44:17.974 INFO:tasks.workunit.client.1.smithi171.stdout:3/107: creat d4/f1d x:0 0 0 2022-01-31T19:44:17.974 INFO:tasks.workunit.client.1.smithi171.stdout:3/108: readlink d4/dd/l15 0 2022-01-31T19:44:17.975 INFO:tasks.workunit.client.1.smithi171.stdout:0/131: link d9/db/c28 d9/db/d22/c2f 0 2022-01-31T19:44:17.976 INFO:tasks.workunit.client.1.smithi171.stdout:0/132: readlink d9/d1b/d25/l2a 0 2022-01-31T19:44:17.982 INFO:tasks.workunit.client.0.smithi167.stdout:5/997: dwrite d5/dcd/f136 [0,4194304] 0 2022-01-31T19:44:17.988 INFO:tasks.workunit.client.0.smithi167.stdout:1/905: rename d0/f5 to d0/d10/d5a/d33/d97/f131 0 2022-01-31T19:44:17.991 INFO:tasks.workunit.client.0.smithi167.stdout:5/998: symlink d5/l13a 0 2022-01-31T19:44:17.992 INFO:tasks.workunit.client.0.smithi167.stdout:1/906: symlink d0/d11/l132 0 2022-01-31T19:44:17.993 INFO:tasks.workunit.client.0.smithi167.stdout:5/999: creat d5/d125/f13b x:0 0 0 2022-01-31T19:44:17.994 INFO:tasks.workunit.client.0.smithi167.stdout:1/907: truncate d0/d10/d5a/d33/d97/fd0 3194199 0 2022-01-31T19:44:17.995 INFO:tasks.workunit.client.0.smithi167.stdout:1/908: creat d0/d11/d46/f133 x:0 0 0 2022-01-31T19:44:17.995 INFO:tasks.workunit.client.0.smithi167.stdout:1/909: write d0/d62/f11a [689448,118674] 0 2022-01-31T19:44:17.996 INFO:tasks.workunit.client.0.smithi167.stdout:1/910: symlink d0/d10/d20/d7c/l134 0 2022-01-31T19:44:17.997 INFO:tasks.workunit.client.0.smithi167.stdout:1/911: chown d0/d62/d7f/d82/f12e 0 1 2022-01-31T19:44:17.998 INFO:tasks.workunit.client.0.smithi167.stdout:1/912: creat d0/d11/d102/f135 x:0 0 0 2022-01-31T19:44:17.998 INFO:tasks.workunit.client.0.smithi167.stdout:1/913: readlink d0/d62/lbe 0 2022-01-31T19:44:18.000 INFO:tasks.workunit.client.0.smithi167.stdout:1/914: rename d0/d10/d20/fcd to d0/d11/d46/f136 0 2022-01-31T19:44:18.001 INFO:tasks.workunit.client.1.smithi171.stdout:6/148: dwrite fa [0,4194304] 0 2022-01-31T19:44:18.001 INFO:tasks.workunit.client.1.smithi171.stdout:6/149: chown dc/c16 707071149 1 2022-01-31T19:44:18.002 INFO:tasks.workunit.client.0.smithi167.stdout:9/726: dwrite dd/d19/d24/d56/dc9/fd4 [0,4194304] 0 2022-01-31T19:44:18.003 INFO:tasks.workunit.client.1.smithi171.stdout:6/150: mkdir dc/d19/d29/d2f 0 2022-01-31T19:44:18.004 INFO:tasks.workunit.client.0.smithi167.stdout:9/727: mkdir dd/d19/df6 0 2022-01-31T19:44:18.005 INFO:tasks.workunit.client.1.smithi171.stdout:6/151: mkdir dc/d19/d30 0 2022-01-31T19:44:18.005 INFO:tasks.workunit.client.0.smithi167.stdout:9/728: link dd/d19/d24/d50/dd1/cd3 dd/d6b/dd5/cf7 0 2022-01-31T19:44:18.006 INFO:tasks.workunit.client.0.smithi167.stdout:9/729: creat dd/d35/ff8 x:0 0 0 2022-01-31T19:44:18.007 INFO:tasks.workunit.client.0.smithi167.stdout:9/730: mkdir dd/d19/d59/df9 0 2022-01-31T19:44:18.008 INFO:tasks.workunit.client.0.smithi167.stdout:9/731: mkdir dd/d19/d24/d50/d5d/dfa 0 2022-01-31T19:44:18.009 INFO:tasks.workunit.client.1.smithi171.stdout:6/152: read f2 [698185,63655] 0 2022-01-31T19:44:18.022 INFO:tasks.workunit.client.0.smithi167.stdout:4/883: dwrite d0/d6/d7/d89/fa9 [0,4194304] 0 2022-01-31T19:44:18.025 INFO:tasks.workunit.client.0.smithi167.stdout:3/874: dwrite d9/d29/d3b/d41/d6b/d89/f97 [0,4194304] 0 2022-01-31T19:44:18.025 INFO:tasks.workunit.client.0.smithi167.stdout:3/875: creat d9/d26/f108 x:0 0 0 2022-01-31T19:44:18.025 INFO:tasks.workunit.client.0.smithi167.stdout:3/876: chown d9/d29/d3b/d41/deb 5 1 2022-01-31T19:44:18.026 INFO:tasks.workunit.client.0.smithi167.stdout:4/884: creat d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d117/f127 x:0 0 0 2022-01-31T19:44:18.027 INFO:tasks.workunit.client.0.smithi167.stdout:4/885: read - d0/d6/d7/d26/d40/d4c/ffd zero size 2022-01-31T19:44:18.027 INFO:tasks.workunit.client.1.smithi171.stdout:1/56: dwrite fa [0,4194304] 0 2022-01-31T19:44:18.028 INFO:tasks.workunit.client.1.smithi171.stdout:1/57: chown fa 428196 1 2022-01-31T19:44:18.028 INFO:tasks.workunit.client.1.smithi171.stdout:6/153: dread f9 [0,4194304] 0 2022-01-31T19:44:18.028 INFO:tasks.workunit.client.1.smithi171.stdout:2/166: dwrite d7/f19 [0,4194304] 0 2022-01-31T19:44:18.029 INFO:tasks.workunit.client.1.smithi171.stdout:1/58: rmdir dd/df 39 2022-01-31T19:44:18.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/154: mknod dc/d19/d29/c31 0 2022-01-31T19:44:18.030 INFO:tasks.workunit.client.0.smithi167.stdout:4/886: link d0/d6/d7/d26/dd2/l115 d0/d6/d7/d39/df0/l128 0 2022-01-31T19:44:18.031 INFO:tasks.workunit.client.1.smithi171.stdout:1/59: dread f7 [0,4194304] 0 2022-01-31T19:44:18.031 INFO:tasks.workunit.client.1.smithi171.stdout:1/60: rename dd/df to dd/df/d11 22 2022-01-31T19:44:18.032 INFO:tasks.workunit.client.0.smithi167.stdout:4/887: symlink d0/d6/d7/d89/db1/l129 0 2022-01-31T19:44:18.033 INFO:tasks.workunit.client.0.smithi167.stdout:4/888: mknod d0/d6/d7/d26/d40/d4c/c12a 0 2022-01-31T19:44:18.033 INFO:tasks.workunit.client.0.smithi167.stdout:4/889: fdatasync d0/d6/d7/d39/d4e/f52 0 2022-01-31T19:44:18.034 INFO:tasks.workunit.client.1.smithi171.stdout:9/52: dwrite f4 [0,4194304] 0 2022-01-31T19:44:18.035 INFO:tasks.workunit.client.1.smithi171.stdout:8/72: dwrite fb [0,4194304] 0 2022-01-31T19:44:18.035 INFO:tasks.workunit.client.1.smithi171.stdout:3/109: dwrite d4/fb [4194304,4194304] 0 2022-01-31T19:44:18.035 INFO:tasks.workunit.client.1.smithi171.stdout:0/133: dwrite d9/f13 [4194304,4194304] 0 2022-01-31T19:44:18.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/110: chown d4/f13 94 1 2022-01-31T19:44:18.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/111: read d4/fa [130729,23348] 0 2022-01-31T19:44:18.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/112: chown d4/dd 257572 1 2022-01-31T19:44:18.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/113: stat c2 0 2022-01-31T19:44:18.038 INFO:tasks.workunit.client.0.smithi167.stdout:1/915: dwrite d0/d11/fd8 [0,4194304] 0 2022-01-31T19:44:18.038 INFO:tasks.workunit.client.0.smithi167.stdout:9/732: dwrite dd/d19/f3d [0,4194304] 0 2022-01-31T19:44:18.038 INFO:tasks.workunit.client.0.smithi167.stdout:9/733: fdatasync dd/d17/d7f/def/fea 0 2022-01-31T19:44:18.039 INFO:tasks.workunit.client.0.smithi167.stdout:4/890: rename d0/d6/d7/d26/d31/d5c/de2/d119 to d0/d6/d7/dc2/d12b 0 2022-01-31T19:44:18.039 INFO:tasks.workunit.client.0.smithi167.stdout:4/891: chown d0/d6/c15 106660083 1 2022-01-31T19:44:18.039 INFO:tasks.workunit.client.0.smithi167.stdout:3/877: dread d9/d29/f76 [0,4194304] 0 2022-01-31T19:44:18.039 INFO:tasks.workunit.client.0.smithi167.stdout:9/734: write dd/f84 [2365352,97424] 0 2022-01-31T19:44:18.040 INFO:tasks.workunit.client.0.smithi167.stdout:9/735: chown dd/d17/d7f/fbb 1 1 2022-01-31T19:44:18.041 INFO:tasks.workunit.client.1.smithi171.stdout:6/155: creat dc/d19/f32 x:0 0 0 2022-01-31T19:44:18.041 INFO:tasks.workunit.client.1.smithi171.stdout:6/156: chown f9 162280840 1 2022-01-31T19:44:18.041 INFO:tasks.workunit.client.1.smithi171.stdout:6/157: write dc/f14 [334752,3020] 0 2022-01-31T19:44:18.042 INFO:tasks.workunit.client.1.smithi171.stdout:6/158: write f9 [652054,57213] 0 2022-01-31T19:44:18.042 INFO:tasks.workunit.client.1.smithi171.stdout:6/159: stat dc/d19/d30 0 2022-01-31T19:44:18.042 INFO:tasks.workunit.client.1.smithi171.stdout:6/160: write dc/d18/f1b [590457,10659] 0 2022-01-31T19:44:18.042 INFO:tasks.workunit.client.1.smithi171.stdout:1/61: creat dd/df/f12 x:0 0 0 2022-01-31T19:44:18.042 INFO:tasks.workunit.client.1.smithi171.stdout:1/62: truncate f9 652887 0 2022-01-31T19:44:18.043 INFO:tasks.workunit.client.0.smithi167.stdout:1/916: rmdir d0/d10/d20/de8 0 2022-01-31T19:44:18.044 INFO:tasks.workunit.client.0.smithi167.stdout:3/878: truncate d9/d26/fb8 1872078 0 2022-01-31T19:44:18.045 INFO:tasks.workunit.client.0.smithi167.stdout:3/879: fdatasync d9/d29/d3b/d41/d6b/f39 0 2022-01-31T19:44:18.045 INFO:tasks.workunit.client.0.smithi167.stdout:3/880: fsync d9/d29/d3b/d41/d6b/d89/f97 0 2022-01-31T19:44:18.045 INFO:tasks.workunit.client.0.smithi167.stdout:3/881: fsync d9/d56/f101 0 2022-01-31T19:44:18.045 INFO:tasks.workunit.client.0.smithi167.stdout:3/882: fsync d9/d7f/f8c 0 2022-01-31T19:44:18.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/736: mkdir dd/d17/d7f/dfb 0 2022-01-31T19:44:18.047 INFO:tasks.workunit.client.0.smithi167.stdout:9/737: chown dd/d17/d80 2 1 2022-01-31T19:44:18.047 INFO:tasks.workunit.client.0.smithi167.stdout:9/738: truncate dd/d19/d24/fca 533418 0 2022-01-31T19:44:18.048 INFO:tasks.workunit.client.1.smithi171.stdout:8/73: creat df/d18/d19/f1b x:0 0 0 2022-01-31T19:44:18.049 INFO:tasks.workunit.client.0.smithi167.stdout:3/883: write d9/d29/fa1 [41448,47315] 0 2022-01-31T19:44:18.049 INFO:tasks.workunit.client.0.smithi167.stdout:9/739: rmdir dd/d19/d24/d56/d5b/da5/de9 0 2022-01-31T19:44:18.049 INFO:tasks.workunit.client.0.smithi167.stdout:9/740: truncate dd/d19/d24/d56/f86 857930 0 2022-01-31T19:44:18.049 INFO:tasks.workunit.client.0.smithi167.stdout:9/741: fdatasync dd/d17/d7f/fcb 0 2022-01-31T19:44:18.050 INFO:tasks.workunit.client.1.smithi171.stdout:0/134: mkdir d9/d1b/d30 0 2022-01-31T19:44:18.050 INFO:tasks.workunit.client.1.smithi171.stdout:0/135: creat d9/db/f31 x:0 0 0 2022-01-31T19:44:18.051 INFO:tasks.workunit.client.0.smithi167.stdout:9/742: unlink dd/d6b/f7a 0 2022-01-31T19:44:18.052 INFO:tasks.workunit.client.1.smithi171.stdout:3/114: creat d4/dd/f1e x:0 0 0 2022-01-31T19:44:18.053 INFO:tasks.workunit.client.1.smithi171.stdout:2/167: symlink d7/dd/l30 0 2022-01-31T19:44:18.054 INFO:tasks.workunit.client.0.smithi167.stdout:4/892: dwrite d0/d6/d7/d26/d40/f32 [0,4194304] 0 2022-01-31T19:44:18.055 INFO:tasks.workunit.client.0.smithi167.stdout:4/893: write d0/d6/d7/d26/d31/ddf/da2/dcf/db5/d117/f127 [707838,25403] 0 2022-01-31T19:44:18.055 INFO:tasks.workunit.client.1.smithi171.stdout:0/136: write d9/d1b/f2d [3665251,105473] 0 2022-01-31T19:44:18.057 INFO:tasks.workunit.client.1.smithi171.stdout:6/161: rename dc/f15 to dc/d19/d29/d27/f33 0 2022-01-31T19:44:18.057 INFO:tasks.workunit.client.1.smithi171.stdout:6/162: write dc/d19/d29/d27/f33 [1058816,68696] 0 2022-01-31T19:44:18.057 INFO:tasks.workunit.client.1.smithi171.stdout:6/163: stat dc/l28 0 2022-01-31T19:44:18.058 INFO:tasks.workunit.client.1.smithi171.stdout:6/164: chown dc/d18/c1f 732189 1 2022-01-31T19:44:18.059 INFO:tasks.workunit.client.0.smithi167.stdout:3/884: rename d9/d56/f73 to d9/d29/d3b/d9b/da3/f109 0 2022-01-31T19:44:18.059 INFO:tasks.workunit.client.0.smithi167.stdout:9/743: symlink dd/d17/dc4/lfc 0 2022-01-31T19:44:18.060 INFO:tasks.workunit.client.1.smithi171.stdout:1/63: link cc dd/df/c13 0 2022-01-31T19:44:18.060 INFO:tasks.workunit.client.1.smithi171.stdout:1/64: truncate fa 4981175 0 2022-01-31T19:44:18.061 INFO:tasks.workunit.client.1.smithi171.stdout:9/53: rmdir dd/df 39 2022-01-31T19:44:18.061 INFO:tasks.workunit.client.1.smithi171.stdout:6/165: dread dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:18.061 INFO:tasks.workunit.client.1.smithi171.stdout:6/166: write dc/d18/f22 [1299928,14015] 0 2022-01-31T19:44:18.061 INFO:tasks.workunit.client.1.smithi171.stdout:6/167: fsync dc/d19/f1d 0 2022-01-31T19:44:18.062 INFO:tasks.workunit.client.1.smithi171.stdout:6/168: getdents dc/d19/d29/d2f 0 2022-01-31T19:44:18.062 INFO:tasks.workunit.client.1.smithi171.stdout:1/65: dread f9 [0,4194304] 0 2022-01-31T19:44:18.064 INFO:tasks.workunit.client.1.smithi171.stdout:0/137: mknod d9/db/c32 0 2022-01-31T19:44:18.069 INFO:tasks.workunit.client.1.smithi171.stdout:9/54: mknod dd/c13 0 2022-01-31T19:44:18.070 INFO:tasks.workunit.client.1.smithi171.stdout:8/74: rmdir df/d18/d19 39 2022-01-31T19:44:18.073 INFO:tasks.workunit.client.1.smithi171.stdout:0/138: truncate d9/f1f 4090550 0 2022-01-31T19:44:18.073 INFO:tasks.workunit.client.1.smithi171.stdout:0/139: stat f6 0 2022-01-31T19:44:18.074 INFO:tasks.workunit.client.1.smithi171.stdout:0/140: stat d9/db/f31 0 2022-01-31T19:44:18.074 INFO:tasks.workunit.client.1.smithi171.stdout:0/141: stat d9/d1b/f2d 0 2022-01-31T19:44:18.075 INFO:tasks.workunit.client.1.smithi171.stdout:8/75: rmdir df/d18 39 2022-01-31T19:44:18.075 INFO:tasks.workunit.client.1.smithi171.stdout:8/76: read - df/f16 zero size 2022-01-31T19:44:18.075 INFO:tasks.workunit.client.1.smithi171.stdout:8/77: readlink le 0 2022-01-31T19:44:18.076 INFO:tasks.workunit.client.1.smithi171.stdout:8/78: dread f0 [0,4194304] 0 2022-01-31T19:44:18.078 INFO:tasks.workunit.client.1.smithi171.stdout:8/79: creat df/d18/f1c x:0 0 0 2022-01-31T19:44:18.078 INFO:tasks.workunit.client.1.smithi171.stdout:8/80: getdents df/d1a 0 2022-01-31T19:44:18.079 INFO:tasks.workunit.client.1.smithi171.stdout:7/92: sync 2022-01-31T19:44:18.080 INFO:tasks.workunit.client.1.smithi171.stdout:7/93: unlink d1/d1b/d12/l14 0 2022-01-31T19:44:18.081 INFO:tasks.workunit.client.1.smithi171.stdout:7/94: write d1/d1b/d7/f19 [128407,49173] 0 2022-01-31T19:44:18.082 INFO:tasks.workunit.client.1.smithi171.stdout:7/95: mkdir d1/d1e 0 2022-01-31T19:44:18.086 INFO:tasks.workunit.client.1.smithi171.stdout:7/96: dread d1/d1b/fc [0,4194304] 0 2022-01-31T19:44:18.088 INFO:tasks.workunit.client.1.smithi171.stdout:7/97: symlink d1/d1b/d12/l1f 0 2022-01-31T19:44:18.089 INFO:tasks.workunit.client.1.smithi171.stdout:7/98: mkdir d1/d1e/d20 0 2022-01-31T19:44:18.090 INFO:tasks.workunit.client.1.smithi171.stdout:7/99: chown d1/d1e 314837 1 2022-01-31T19:44:18.093 INFO:tasks.workunit.client.1.smithi171.stdout:1/66: dwrite fa [0,4194304] 0 2022-01-31T19:44:18.097 INFO:tasks.workunit.client.0.smithi167.stdout:9/744: dwrite dd/d35/f37 [4194304,4194304] 0 2022-01-31T19:44:18.098 INFO:tasks.workunit.client.1.smithi171.stdout:1/67: dread dd/df/f10 [0,4194304] 0 2022-01-31T19:44:18.098 INFO:tasks.workunit.client.1.smithi171.stdout:1/68: dread - dd/df/f12 zero size 2022-01-31T19:44:18.099 INFO:tasks.workunit.client.1.smithi171.stdout:1/69: link f3 dd/f14 0 2022-01-31T19:44:18.099 INFO:tasks.workunit.client.1.smithi171.stdout:1/70: chown c2 174174681 1 2022-01-31T19:44:18.100 INFO:tasks.workunit.client.1.smithi171.stdout:1/71: symlink dd/df/l15 0 2022-01-31T19:44:18.101 INFO:tasks.workunit.client.1.smithi171.stdout:9/55: dwrite dd/fe [0,4194304] 0 2022-01-31T19:44:18.102 INFO:tasks.workunit.client.1.smithi171.stdout:0/142: dwrite d9/f1d [0,4194304] 0 2022-01-31T19:44:18.102 INFO:tasks.workunit.client.1.smithi171.stdout:8/81: dwrite f0 [0,4194304] 0 2022-01-31T19:44:18.107 INFO:tasks.workunit.client.1.smithi171.stdout:9/56: rename c7 to dd/df/c14 0 2022-01-31T19:44:18.115 INFO:tasks.workunit.client.1.smithi171.stdout:0/143: mknod d9/d1b/d25/c33 0 2022-01-31T19:44:18.116 INFO:tasks.workunit.client.1.smithi171.stdout:0/144: chown d9/db/c28 110 1 2022-01-31T19:44:18.117 INFO:tasks.workunit.client.1.smithi171.stdout:8/82: symlink df/d18/l1d 0 2022-01-31T19:44:18.118 INFO:tasks.workunit.client.1.smithi171.stdout:9/57: dread fc [0,4194304] 0 2022-01-31T19:44:18.119 INFO:tasks.workunit.client.1.smithi171.stdout:0/145: creat d9/d1b/f34 x:0 0 0 2022-01-31T19:44:18.121 INFO:tasks.workunit.client.1.smithi171.stdout:8/83: link df/d18/f1c df/d18/f1e 0 2022-01-31T19:44:18.121 INFO:tasks.workunit.client.1.smithi171.stdout:8/84: fdatasync df/f16 0 2022-01-31T19:44:18.122 INFO:tasks.workunit.client.1.smithi171.stdout:1/72: dwrite dd/df/f10 [0,4194304] 0 2022-01-31T19:44:18.122 INFO:tasks.workunit.client.1.smithi171.stdout:9/58: mkdir dd/df/d15 0 2022-01-31T19:44:18.123 INFO:tasks.workunit.client.1.smithi171.stdout:9/59: chown f4 24029 1 2022-01-31T19:44:18.124 INFO:tasks.workunit.client.1.smithi171.stdout:8/85: write f2 [6350487,43758] 0 2022-01-31T19:44:18.128 INFO:tasks.workunit.client.1.smithi171.stdout:7/100: dwrite d1/d1b/d7/f19 [0,4194304] 0 2022-01-31T19:44:18.130 INFO:tasks.workunit.client.1.smithi171.stdout:0/146: rename d9/db/dc/d2e to d9/d1b/d30/d35 0 2022-01-31T19:44:18.130 INFO:tasks.workunit.client.1.smithi171.stdout:9/60: creat dd/df/d15/f16 x:0 0 0 2022-01-31T19:44:18.130 INFO:tasks.workunit.client.1.smithi171.stdout:9/61: chown f4 3345 1 2022-01-31T19:44:18.131 INFO:tasks.workunit.client.1.smithi171.stdout:9/62: truncate dd/df/f12 1018473 0 2022-01-31T19:44:18.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/101: write d1/d1b/d7/ff [3225428,130875] 0 2022-01-31T19:44:18.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/102: fsync d1/d1b/d12/f1a 0 2022-01-31T19:44:18.133 INFO:tasks.workunit.client.1.smithi171.stdout:7/103: read d1/d1b/d7/fe [2576471,38063] 0 2022-01-31T19:44:18.133 INFO:tasks.workunit.client.1.smithi171.stdout:8/86: symlink df/d18/d19/l1f 0 2022-01-31T19:44:18.133 INFO:tasks.workunit.client.1.smithi171.stdout:8/87: creat df/f20 x:0 0 0 2022-01-31T19:44:18.133 INFO:tasks.workunit.client.1.smithi171.stdout:8/88: readlink ld 0 2022-01-31T19:44:18.134 INFO:tasks.workunit.client.1.smithi171.stdout:0/147: unlink d9/db/dc/l11 0 2022-01-31T19:44:18.136 INFO:tasks.workunit.client.1.smithi171.stdout:7/104: truncate d1/d1b/d7/fe 3442466 0 2022-01-31T19:44:18.137 INFO:tasks.workunit.client.1.smithi171.stdout:8/89: mkdir df/d18/d19/d21 0 2022-01-31T19:44:18.137 INFO:tasks.workunit.client.1.smithi171.stdout:0/148: symlink d9/d1b/d25/l36 0 2022-01-31T19:44:18.138 INFO:tasks.workunit.client.1.smithi171.stdout:9/63: dread dd/df/f12 [0,4194304] 0 2022-01-31T19:44:18.143 INFO:tasks.workunit.client.1.smithi171.stdout:7/105: truncate d1/d1b/fc 3476417 0 2022-01-31T19:44:18.143 INFO:tasks.workunit.client.1.smithi171.stdout:8/90: creat df/d18/d19/f22 x:0 0 0 2022-01-31T19:44:18.143 INFO:tasks.workunit.client.1.smithi171.stdout:7/106: fsync d1/f16 0 2022-01-31T19:44:18.144 INFO:tasks.workunit.client.1.smithi171.stdout:9/64: creat dd/df/f17 x:0 0 0 2022-01-31T19:44:18.145 INFO:tasks.workunit.client.1.smithi171.stdout:1/73: dwrite f7 [0,4194304] 0 2022-01-31T19:44:18.145 INFO:tasks.workunit.client.1.smithi171.stdout:1/74: fdatasync f3 0 2022-01-31T19:44:18.147 INFO:tasks.workunit.client.1.smithi171.stdout:8/91: mknod df/d18/d19/d21/c23 0 2022-01-31T19:44:18.147 INFO:tasks.workunit.client.1.smithi171.stdout:8/92: write df/f16 [69442,93175] 0 2022-01-31T19:44:18.149 INFO:tasks.workunit.client.0.smithi167.stdout:4/894: unlink d0/d6/d7/d89/fad 0 2022-01-31T19:44:18.149 INFO:tasks.workunit.client.0.smithi167.stdout:4/895: write d0/d6/d7/d26/d31/ddf/d75/dd6/d111/f126 [639128,111470] 0 2022-01-31T19:44:18.150 INFO:tasks.workunit.client.0.smithi167.stdout:3/885: mkdir d9/d7f/d10a 0 2022-01-31T19:44:18.151 INFO:tasks.workunit.client.0.smithi167.stdout:3/886: stat d9/d21/c47 0 2022-01-31T19:44:18.152 INFO:tasks.workunit.client.0.smithi167.stdout:9/745: symlink dd/d19/lfd 0 2022-01-31T19:44:18.152 INFO:tasks.workunit.client.0.smithi167.stdout:3/887: write d9/d26/f2a [4107897,7014] 0 2022-01-31T19:44:18.152 INFO:tasks.workunit.client.1.smithi171.stdout:9/65: dread dd/df/f12 [0,4194304] 0 2022-01-31T19:44:18.153 INFO:tasks.workunit.client.1.smithi171.stdout:9/66: fsync fc 0 2022-01-31T19:44:18.158 INFO:tasks.workunit.client.0.smithi167.stdout:3/888: dread d9/fc [0,4194304] 0 2022-01-31T19:44:18.165 INFO:tasks.workunit.client.1.smithi171.stdout:0/149: dwrite d9/db/f21 [0,4194304] 0 2022-01-31T19:44:18.165 INFO:tasks.workunit.client.1.smithi171.stdout:7/107: dread d1/d1b/d7/f19 [0,4194304] 0 2022-01-31T19:44:18.165 INFO:tasks.workunit.client.1.smithi171.stdout:1/75: rename c4 to dd/df/c16 0 2022-01-31T19:44:18.168 INFO:tasks.workunit.client.0.smithi167.stdout:3/889: creat d9/d29/d60/dc9/f10b x:0 0 0 2022-01-31T19:44:18.168 INFO:tasks.workunit.client.1.smithi171.stdout:8/93: symlink df/d18/d19/l24 0 2022-01-31T19:44:18.170 INFO:tasks.workunit.client.1.smithi171.stdout:0/150: mknod d9/d1b/d30/d35/c37 0 2022-01-31T19:44:18.172 INFO:tasks.workunit.client.1.smithi171.stdout:7/108: creat d1/d1b/d12/f21 x:0 0 0 2022-01-31T19:44:18.173 INFO:tasks.workunit.client.0.smithi167.stdout:3/890: getdents d9/d56 0 2022-01-31T19:44:18.174 INFO:tasks.workunit.client.1.smithi171.stdout:1/76: mkdir dd/df/d17 0 2022-01-31T19:44:18.174 INFO:tasks.workunit.client.1.smithi171.stdout:1/77: chown c2 4081 1 2022-01-31T19:44:18.175 INFO:tasks.workunit.client.1.smithi171.stdout:7/109: write d1/d1b/f10 [227357,27517] 0 2022-01-31T19:44:18.175 INFO:tasks.workunit.client.1.smithi171.stdout:1/78: fsync dd/f14 0 2022-01-31T19:44:18.176 INFO:tasks.workunit.client.1.smithi171.stdout:8/94: write f7 [1482382,113024] 0 2022-01-31T19:44:18.176 INFO:tasks.workunit.client.1.smithi171.stdout:7/110: creat d1/d1b/f22 x:0 0 0 2022-01-31T19:44:18.177 INFO:tasks.workunit.client.1.smithi171.stdout:8/95: symlink df/d18/d19/l25 0 2022-01-31T19:44:18.180 INFO:tasks.workunit.client.1.smithi171.stdout:8/96: creat df/d18/d19/d21/f26 x:0 0 0 2022-01-31T19:44:18.182 INFO:tasks.workunit.client.1.smithi171.stdout:8/97: mkdir df/d27 0 2022-01-31T19:44:18.191 INFO:tasks.workunit.client.1.smithi171.stdout:5/78: sync 2022-01-31T19:44:18.191 INFO:tasks.workunit.client.1.smithi171.stdout:5/79: chown f1 87539073 1 2022-01-31T19:44:18.192 INFO:tasks.workunit.client.1.smithi171.stdout:5/80: truncate d5/f8 950972 0 2022-01-31T19:44:18.193 INFO:tasks.workunit.client.1.smithi171.stdout:5/81: write f1 [4123774,72035] 0 2022-01-31T19:44:18.193 INFO:tasks.workunit.client.1.smithi171.stdout:5/82: dread - d5/d6/f16 zero size 2022-01-31T19:44:18.193 INFO:tasks.workunit.client.1.smithi171.stdout:7/111: dread d1/d1b/d7/ff [0,4194304] 0 2022-01-31T19:44:18.194 INFO:tasks.workunit.client.1.smithi171.stdout:5/83: unlink d5/l12 0 2022-01-31T19:44:18.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/84: symlink d5/d6/dd/l18 0 2022-01-31T19:44:18.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/85: chown d5 2903896 1 2022-01-31T19:44:18.196 INFO:tasks.workunit.client.1.smithi171.stdout:5/86: creat d5/d6/dd/f19 x:0 0 0 2022-01-31T19:44:18.197 INFO:tasks.workunit.client.0.smithi167.stdout:4/896: dwrite d0/d6/d7/f116 [0,4194304] 0 2022-01-31T19:44:18.198 INFO:tasks.workunit.client.0.smithi167.stdout:4/897: truncate d0/d6/d7/d89/db1/f104 286874 0 2022-01-31T19:44:18.204 INFO:tasks.workunit.client.0.smithi167.stdout:9/746: dwrite dd/d34/f51 [0,4194304] 0 2022-01-31T19:44:18.205 INFO:tasks.workunit.client.0.smithi167.stdout:9/747: rename dd/d19/d24/d50/dd1/fa6 to dd/d19/d24/d56/dc9/ffe 0 2022-01-31T19:44:18.215 INFO:tasks.workunit.client.1.smithi171.stdout:9/67: dwrite f9 [0,4194304] 0 2022-01-31T19:44:18.215 INFO:tasks.workunit.client.1.smithi171.stdout:0/151: dwrite f6 [0,4194304] 0 2022-01-31T19:44:18.215 INFO:tasks.workunit.client.1.smithi171.stdout:0/152: dread - d9/d1b/d25/f29 zero size 2022-01-31T19:44:18.216 INFO:tasks.workunit.client.0.smithi167.stdout:3/891: dwrite d9/d29/d3b/d41/d6b/d89/fd6 [0,4194304] 0 2022-01-31T19:44:18.221 INFO:tasks.workunit.client.0.smithi167.stdout:3/892: truncate d9/f11 1159463 0 2022-01-31T19:44:18.223 INFO:tasks.workunit.client.1.smithi171.stdout:5/87: rmdir d5/d6 39 2022-01-31T19:44:18.224 INFO:tasks.workunit.client.0.smithi167.stdout:3/893: rename d9/d29/f76 to d9/d29/d3b/d9b/da3/f10c 0 2022-01-31T19:44:18.225 INFO:tasks.workunit.client.0.smithi167.stdout:3/894: creat d9/d29/d3b/d41/d6b/f10d x:0 0 0 2022-01-31T19:44:18.225 INFO:tasks.workunit.client.0.smithi167.stdout:3/895: write d9/d21/fce [1027804,65579] 0 2022-01-31T19:44:18.226 INFO:tasks.workunit.client.1.smithi171.stdout:9/68: truncate fa 12600 0 2022-01-31T19:44:18.226 INFO:tasks.workunit.client.1.smithi171.stdout:9/69: symlink dd/l18 0 2022-01-31T19:44:18.227 INFO:tasks.workunit.client.0.smithi167.stdout:3/896: link d9/d29/d3b/d41/d6b/f66 d9/d7f/da8/f10e 0 2022-01-31T19:44:18.228 INFO:tasks.workunit.client.1.smithi171.stdout:9/70: rename l6 to dd/l19 0 2022-01-31T19:44:18.229 INFO:tasks.workunit.client.1.smithi171.stdout:9/71: chown l8 5 1 2022-01-31T19:44:18.229 INFO:tasks.workunit.client.1.smithi171.stdout:9/72: chown l3 521938 1 2022-01-31T19:44:18.230 INFO:tasks.workunit.client.1.smithi171.stdout:9/73: getdents dd/df/d15 0 2022-01-31T19:44:18.231 INFO:tasks.workunit.client.1.smithi171.stdout:1/79: dwrite dd/f14 [0,4194304] 0 2022-01-31T19:44:18.231 INFO:tasks.workunit.client.1.smithi171.stdout:9/74: mkdir dd/df/d15/d1a 0 2022-01-31T19:44:18.232 INFO:tasks.workunit.client.1.smithi171.stdout:9/75: chown dd/c13 64 1 2022-01-31T19:44:18.233 INFO:tasks.workunit.client.0.smithi167.stdout:3/897: link d9/d29/d60/dc9/ce2 d9/d29/d60/df8/c10f 0 2022-01-31T19:44:18.233 INFO:tasks.workunit.client.0.smithi167.stdout:3/898: write d9/d29/d3b/f9e [717349,81679] 0 2022-01-31T19:44:18.234 INFO:tasks.workunit.client.0.smithi167.stdout:9/748: dwrite dd/d19/d24/d50/d5d/fa4 [0,4194304] 0 2022-01-31T19:44:18.235 INFO:tasks.workunit.client.1.smithi171.stdout:1/80: rename cc to dd/df/d17/c18 0 2022-01-31T19:44:18.235 INFO:tasks.workunit.client.1.smithi171.stdout:1/81: chown f7 3564244 1 2022-01-31T19:44:18.235 INFO:tasks.workunit.client.0.smithi167.stdout:3/899: dread d9/d29/f36 [0,4194304] 0 2022-01-31T19:44:18.236 INFO:tasks.workunit.client.0.smithi167.stdout:3/900: truncate d9/d56/f90 4404592 0 2022-01-31T19:44:18.236 INFO:tasks.workunit.client.0.smithi167.stdout:3/901: read - d9/d26/fdb zero size 2022-01-31T19:44:18.237 INFO:tasks.workunit.client.1.smithi171.stdout:7/112: dwrite d1/d1b/f10 [4194304,4194304] 0 2022-01-31T19:44:18.237 INFO:tasks.workunit.client.1.smithi171.stdout:7/113: chown d1/d1b/f10 30450748 1 2022-01-31T19:44:18.238 INFO:tasks.workunit.client.1.smithi171.stdout:1/82: mkdir dd/df/d19 0 2022-01-31T19:44:18.238 INFO:tasks.workunit.client.0.smithi167.stdout:4/898: dwrite d0/d6/d7/d89/f10d [4194304,4194304] 0 2022-01-31T19:44:18.239 INFO:tasks.workunit.client.0.smithi167.stdout:4/899: stat d0/d6/d7/d26/d40/d4c/dfe/la8 0 2022-01-31T19:44:18.239 INFO:tasks.workunit.client.0.smithi167.stdout:4/900: creat d0/d6/d7/d39/f12c x:0 0 0 2022-01-31T19:44:18.240 INFO:tasks.workunit.client.0.smithi167.stdout:9/749: unlink dd/d17/f3e 0 2022-01-31T19:44:18.240 INFO:tasks.workunit.client.0.smithi167.stdout:9/750: readlink dd/d19/l90 0 2022-01-31T19:44:18.241 INFO:tasks.workunit.client.1.smithi171.stdout:7/114: link d1/d1b/d12/c13 d1/d1b/d12/c23 0 2022-01-31T19:44:18.241 INFO:tasks.workunit.client.1.smithi171.stdout:1/83: mkdir dd/df/d1a 0 2022-01-31T19:44:18.244 INFO:tasks.workunit.client.0.smithi167.stdout:4/901: truncate d0/d6/d7/d26/d31/f19 623060 0 2022-01-31T19:44:18.245 INFO:tasks.workunit.client.1.smithi171.stdout:1/84: write fa [3275562,109790] 0 2022-01-31T19:44:18.246 INFO:tasks.workunit.client.1.smithi171.stdout:9/76: rename fa to dd/df/d15/f1b 0 2022-01-31T19:44:18.247 INFO:tasks.workunit.client.1.smithi171.stdout:1/85: stat c2 0 2022-01-31T19:44:18.247 INFO:tasks.workunit.client.0.smithi167.stdout:9/751: creat dd/d17/d7f/def/fff x:0 0 0 2022-01-31T19:44:18.248 INFO:tasks.workunit.client.0.smithi167.stdout:9/752: write dd/d19/d59/fce [5027972,112780] 0 2022-01-31T19:44:18.251 INFO:tasks.workunit.client.1.smithi171.stdout:9/77: mkdir dd/d1c 0 2022-01-31T19:44:18.251 INFO:tasks.workunit.client.0.smithi167.stdout:9/753: mknod dd/d19/d59/c100 0 2022-01-31T19:44:18.252 INFO:tasks.workunit.client.0.smithi167.stdout:9/754: symlink dd/d19/d24/l101 0 2022-01-31T19:44:18.253 INFO:tasks.workunit.client.0.smithi167.stdout:9/755: rename dd/d19/d24/l58 to dd/d19/d24/d50/d5d/dfa/l102 0 2022-01-31T19:44:18.255 INFO:tasks.workunit.client.0.smithi167.stdout:9/756: creat dd/d19/d24/d50/dd6/f103 x:0 0 0 2022-01-31T19:44:18.258 INFO:tasks.workunit.client.0.smithi167.stdout:9/757: rename dd/d19/d59/c100 to dd/d17/c104 0 2022-01-31T19:44:18.258 INFO:tasks.workunit.client.0.smithi167.stdout:9/758: truncate dd/d17/d7f/fcb 918130 0 2022-01-31T19:44:18.262 INFO:tasks.workunit.client.0.smithi167.stdout:3/902: dwrite d9/de7/ffe [0,4194304] 0 2022-01-31T19:44:18.263 INFO:tasks.workunit.client.0.smithi167.stdout:3/903: write d9/d26/f98 [33384,108108] 0 2022-01-31T19:44:18.263 INFO:tasks.workunit.client.0.smithi167.stdout:9/759: rename dd/d6b/dd5 to dd/d35/d105 0 2022-01-31T19:44:18.263 INFO:tasks.workunit.client.0.smithi167.stdout:9/760: dread - dd/d34/f92 zero size 2022-01-31T19:44:18.265 INFO:tasks.workunit.client.0.smithi167.stdout:3/904: mknod d9/de7/c110 0 2022-01-31T19:44:18.267 INFO:tasks.workunit.client.0.smithi167.stdout:9/761: creat dd/d17/d80/f106 x:0 0 0 2022-01-31T19:44:18.267 INFO:tasks.workunit.client.1.smithi171.stdout:7/115: rename d1/d1b/d12 to d1/d24 0 2022-01-31T19:44:18.268 INFO:tasks.workunit.client.1.smithi171.stdout:7/116: truncate d1/d24/f1a 556066 0 2022-01-31T19:44:18.269 INFO:tasks.workunit.client.1.smithi171.stdout:7/117: link d1/d24/c13 d1/d1b/d7/c25 0 2022-01-31T19:44:18.269 INFO:tasks.workunit.client.1.smithi171.stdout:7/118: readlink d1/d24/l17 0 2022-01-31T19:44:18.270 INFO:tasks.workunit.client.0.smithi167.stdout:9/762: write dd/d17/d80/f54 [2671882,38514] 0 2022-01-31T19:44:18.270 INFO:tasks.workunit.client.0.smithi167.stdout:9/763: symlink dd/d17/dc4/l107 0 2022-01-31T19:44:18.276 INFO:tasks.workunit.client.1.smithi171.stdout:4/93: sync 2022-01-31T19:44:18.276 INFO:tasks.workunit.client.0.smithi167.stdout:4/902: dwrite d0/d6/d7/f9a [0,4194304] 0 2022-01-31T19:44:18.277 INFO:tasks.workunit.client.1.smithi171.stdout:9/78: dwrite dd/df/f12 [0,4194304] 0 2022-01-31T19:44:18.278 INFO:tasks.workunit.client.1.smithi171.stdout:9/79: truncate f4 9356066 0 2022-01-31T19:44:18.278 INFO:tasks.workunit.client.1.smithi171.stdout:4/94: symlink d2/d3/d5/d11/l1f 0 2022-01-31T19:44:18.280 INFO:tasks.workunit.client.1.smithi171.stdout:9/80: mknod dd/c1d 0 2022-01-31T19:44:18.281 INFO:tasks.workunit.client.0.smithi167.stdout:9/764: write dd/d35/f88 [591809,53477] 0 2022-01-31T19:44:18.281 INFO:tasks.workunit.client.0.smithi167.stdout:4/903: rmdir d0/d6/d7/dc2/d12b 0 2022-01-31T19:44:18.281 INFO:tasks.workunit.client.0.smithi167.stdout:4/904: write d0/d6/d7/d26/d31/ddf/d75/f98 [806215,87311] 0 2022-01-31T19:44:18.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/81: truncate dd/fe 5996 0 2022-01-31T19:44:18.284 INFO:tasks.workunit.client.1.smithi171.stdout:9/82: symlink dd/df/d15/l1e 0 2022-01-31T19:44:18.286 INFO:tasks.workunit.client.0.smithi167.stdout:9/765: dread dd/d19/d24/d56/fc7 [0,4194304] 0 2022-01-31T19:44:18.288 INFO:tasks.workunit.client.0.smithi167.stdout:9/766: mknod dd/d19/d59/df9/c108 0 2022-01-31T19:44:18.288 INFO:tasks.workunit.client.0.smithi167.stdout:9/767: fdatasync dd/f84 0 2022-01-31T19:44:18.290 INFO:tasks.workunit.client.1.smithi171.stdout:9/83: dread fc [0,4194304] 0 2022-01-31T19:44:18.291 INFO:tasks.workunit.client.1.smithi171.stdout:9/84: chown dd/f11 110 1 2022-01-31T19:44:18.292 INFO:tasks.workunit.client.1.smithi171.stdout:1/86: dwrite fa [0,4194304] 0 2022-01-31T19:44:18.292 INFO:tasks.workunit.client.0.smithi167.stdout:9/768: symlink dd/d17/dda/l109 0 2022-01-31T19:44:18.292 INFO:tasks.workunit.client.0.smithi167.stdout:9/769: symlink dd/d19/df6/l10a 0 2022-01-31T19:44:18.293 INFO:tasks.workunit.client.0.smithi167.stdout:9/770: mknod dd/d19/d59/df9/c10b 0 2022-01-31T19:44:18.294 INFO:tasks.workunit.client.1.smithi171.stdout:1/87: getdents dd/df/d17 0 2022-01-31T19:44:18.296 INFO:tasks.workunit.client.0.smithi167.stdout:3/905: dwrite d9/d7f/da8/f106 [0,4194304] 0 2022-01-31T19:44:18.296 INFO:tasks.workunit.client.1.smithi171.stdout:7/119: rmdir d1 39 2022-01-31T19:44:18.299 INFO:tasks.workunit.client.1.smithi171.stdout:7/120: mkdir d1/d26 0 2022-01-31T19:44:18.313 INFO:tasks.workunit.client.1.smithi171.stdout:1/88: dread f3 [0,4194304] 0 2022-01-31T19:44:18.314 INFO:tasks.workunit.client.1.smithi171.stdout:1/89: truncate fa 5107673 0 2022-01-31T19:44:18.314 INFO:tasks.workunit.client.1.smithi171.stdout:1/90: truncate dd/df/f10 188629 0 2022-01-31T19:44:18.315 INFO:tasks.workunit.client.1.smithi171.stdout:1/91: mkdir dd/df/d19/d1b 0 2022-01-31T19:44:18.321 INFO:tasks.workunit.client.1.smithi171.stdout:4/95: dwrite d2/d3/f7 [0,4194304] 0 2022-01-31T19:44:18.327 INFO:tasks.workunit.client.0.smithi167.stdout:4/905: dwrite d0/d6/d7/d39/d4e/f52 [0,4194304] 0 2022-01-31T19:44:18.328 INFO:tasks.workunit.client.0.smithi167.stdout:4/906: truncate d0/d6/d7/d26/d31/d5c/de2/f112 929357 0 2022-01-31T19:44:18.328 INFO:tasks.workunit.client.0.smithi167.stdout:4/907: dread - d0/fac zero size 2022-01-31T19:44:18.328 INFO:tasks.workunit.client.1.smithi171.stdout:9/85: dwrite dd/df/f17 [0,4194304] 0 2022-01-31T19:44:18.329 INFO:tasks.workunit.client.0.smithi167.stdout:3/906: dwrite d9/d56/f91 [0,4194304] 0 2022-01-31T19:44:18.330 INFO:tasks.workunit.client.0.smithi167.stdout:3/907: chown d9/d56/fab 0 1 2022-01-31T19:44:18.330 INFO:tasks.workunit.client.0.smithi167.stdout:3/908: write d9/d29/d3b/d9b/da3/db0/ffa [704373,45382] 0 2022-01-31T19:44:18.330 INFO:tasks.workunit.client.0.smithi167.stdout:3/909: write d9/f2c [5173001,118004] 0 2022-01-31T19:44:18.331 INFO:tasks.workunit.client.1.smithi171.stdout:9/86: creat dd/d1c/f1f x:0 0 0 2022-01-31T19:44:18.333 INFO:tasks.workunit.client.0.smithi167.stdout:3/910: getdents d9/d7f 0 2022-01-31T19:44:18.334 INFO:tasks.workunit.client.0.smithi167.stdout:4/908: dread d0/d6/d7/d89/db1/f104 [0,4194304] 0 2022-01-31T19:44:18.335 INFO:tasks.workunit.client.1.smithi171.stdout:4/96: dread d2/d3/d5/f1a [0,4194304] 0 2022-01-31T19:44:18.335 INFO:tasks.workunit.client.0.smithi167.stdout:4/909: mkdir d0/d6/d7/d26/dd2/de5/de6/d12d 0 2022-01-31T19:44:18.336 INFO:tasks.workunit.client.0.smithi167.stdout:4/910: readlink d0/d6/d7/d26/d31/ddf/l49 0 2022-01-31T19:44:18.337 INFO:tasks.workunit.client.1.smithi171.stdout:4/97: link d2/ld d2/d3/d5/d11/d12/l20 0 2022-01-31T19:44:18.337 INFO:tasks.workunit.client.0.smithi167.stdout:4/911: read d0/d6/d7/d26/d31/ddf/f2e [2039089,27066] 0 2022-01-31T19:44:18.340 INFO:tasks.workunit.client.1.smithi171.stdout:1/92: dwrite f7 [0,4194304] 0 2022-01-31T19:44:18.343 INFO:tasks.workunit.client.0.smithi167.stdout:2/651: sync 2022-01-31T19:44:18.344 INFO:tasks.workunit.client.0.smithi167.stdout:2/652: truncate dd/df/d22/d96/d9e/fb2 1064268 0 2022-01-31T19:44:18.344 INFO:tasks.workunit.client.0.smithi167.stdout:2/653: readlink dd/df/d1b/d3d/d4a/d4b/d54/l88 0 2022-01-31T19:44:18.344 INFO:tasks.workunit.client.0.smithi167.stdout:2/654: readlink dd/df/l78 0 2022-01-31T19:44:18.345 INFO:tasks.workunit.client.0.smithi167.stdout:2/655: mknod dd/df/d1b/d3d/d4a/da7/cd1 0 2022-01-31T19:44:18.345 INFO:tasks.workunit.client.1.smithi171.stdout:1/93: rename cb to dd/df/d19/c1c 0 2022-01-31T19:44:18.345 INFO:tasks.workunit.client.1.smithi171.stdout:1/94: chown dd/df/d19 1853233 1 2022-01-31T19:44:18.346 INFO:tasks.workunit.client.0.smithi167.stdout:2/656: mknod dd/df/d1b/d3d/d4a/da7/dcc/cd2 0 2022-01-31T19:44:18.347 INFO:tasks.workunit.client.1.smithi171.stdout:1/95: unlink dd/df/c16 0 2022-01-31T19:44:18.348 INFO:tasks.workunit.client.1.smithi171.stdout:9/87: truncate dd/df/f17 973843 0 2022-01-31T19:44:18.351 INFO:tasks.workunit.client.1.smithi171.stdout:1/96: read f7 [3217994,93261] 0 2022-01-31T19:44:18.351 INFO:tasks.workunit.client.1.smithi171.stdout:1/97: truncate f3 990723 0 2022-01-31T19:44:18.352 INFO:tasks.workunit.client.1.smithi171.stdout:9/88: dread dd/df/d15/f1b [0,4194304] 0 2022-01-31T19:44:18.352 INFO:tasks.workunit.client.1.smithi171.stdout:1/98: creat dd/df/d17/f1d x:0 0 0 2022-01-31T19:44:18.353 INFO:tasks.workunit.client.1.smithi171.stdout:9/89: write dd/df/d15/f16 [248772,124904] 0 2022-01-31T19:44:18.354 INFO:tasks.workunit.client.1.smithi171.stdout:1/99: dread f3 [0,4194304] 0 2022-01-31T19:44:18.356 INFO:tasks.workunit.client.1.smithi171.stdout:1/100: read f3 [132016,101213] 0 2022-01-31T19:44:18.358 INFO:tasks.workunit.client.1.smithi171.stdout:1/101: rename dd/df/d1a to dd/df/d19/d1b/d1e 0 2022-01-31T19:44:18.360 INFO:tasks.workunit.client.1.smithi171.stdout:1/102: read f3 [545034,38022] 0 2022-01-31T19:44:18.360 INFO:tasks.workunit.client.1.smithi171.stdout:1/103: rename dd/df to dd/df/d19/d1b/d1f 22 2022-01-31T19:44:18.360 INFO:tasks.workunit.client.1.smithi171.stdout:1/104: write f9 [1263704,32846] 0 2022-01-31T19:44:18.362 INFO:tasks.workunit.client.1.smithi171.stdout:1/105: write dd/df/f10 [354963,80147] 0 2022-01-31T19:44:18.362 INFO:tasks.workunit.client.1.smithi171.stdout:1/106: creat dd/df/d17/f20 x:0 0 0 2022-01-31T19:44:18.362 INFO:tasks.workunit.client.1.smithi171.stdout:1/107: write dd/f14 [1865278,858] 0 2022-01-31T19:44:18.364 INFO:tasks.workunit.client.1.smithi171.stdout:1/108: creat dd/df/d19/f21 x:0 0 0 2022-01-31T19:44:18.379 INFO:tasks.workunit.client.0.smithi167.stdout:3/911: dwrite d9/d56/f6d [0,4194304] 0 2022-01-31T19:44:18.379 INFO:tasks.workunit.client.0.smithi167.stdout:3/912: creat d9/d29/d60/df8/f111 x:0 0 0 2022-01-31T19:44:18.380 INFO:tasks.workunit.client.0.smithi167.stdout:3/913: fsync d9/d29/d3b/d41/d6b/f3e 0 2022-01-31T19:44:18.380 INFO:tasks.workunit.client.0.smithi167.stdout:4/912: dwrite d0/d6/d7/d26/d31/ddf/da2/dcf/f94 [0,4194304] 0 2022-01-31T19:44:18.381 INFO:tasks.workunit.client.1.smithi171.stdout:4/98: dwrite d2/d3/f10 [0,4194304] 0 2022-01-31T19:44:18.381 INFO:tasks.workunit.client.0.smithi167.stdout:3/914: truncate d9/d26/f65 1197523 0 2022-01-31T19:44:18.382 INFO:tasks.workunit.client.0.smithi167.stdout:3/915: readlink d9/d29/d60/lb7 0 2022-01-31T19:44:18.382 INFO:tasks.workunit.client.0.smithi167.stdout:3/916: creat d9/d7f/da8/f112 x:0 0 0 2022-01-31T19:44:18.383 INFO:tasks.workunit.client.1.smithi171.stdout:4/99: read d2/d3/f1c [110604,69582] 0 2022-01-31T19:44:18.383 INFO:tasks.workunit.client.1.smithi171.stdout:4/100: dread - d2/d3/d5/dc/f19 zero size 2022-01-31T19:44:18.384 INFO:tasks.workunit.client.0.smithi167.stdout:4/913: creat d0/d6/d7/d26/d31/ddf/d75/dce/f12e x:0 0 0 2022-01-31T19:44:18.384 INFO:tasks.workunit.client.0.smithi167.stdout:4/914: chown d0/d6/d7/d26/d31/ddf/da2/f125 205249 1 2022-01-31T19:44:18.384 INFO:tasks.workunit.client.0.smithi167.stdout:4/915: dread - d0/f7a zero size 2022-01-31T19:44:18.384 INFO:tasks.workunit.client.0.smithi167.stdout:4/916: chown d0/d6/d7/d26/dd2/de5/de6/d118 0 1 2022-01-31T19:44:18.385 INFO:tasks.workunit.client.1.smithi171.stdout:4/101: rmdir d2 39 2022-01-31T19:44:18.385 INFO:tasks.workunit.client.0.smithi167.stdout:2/657: dwrite dd/d18/fa9 [0,4194304] 0 2022-01-31T19:44:18.387 INFO:tasks.workunit.client.1.smithi171.stdout:9/90: dwrite f4 [8388608,4194304] 0 2022-01-31T19:44:18.387 INFO:tasks.workunit.client.1.smithi171.stdout:9/91: chown dd/d1c/f1f 4069 1 2022-01-31T19:44:18.387 INFO:tasks.workunit.client.1.smithi171.stdout:9/92: creat dd/df/f20 x:0 0 0 2022-01-31T19:44:18.387 INFO:tasks.workunit.client.1.smithi171.stdout:9/93: creat dd/df/f21 x:0 0 0 2022-01-31T19:44:18.388 INFO:tasks.workunit.client.0.smithi167.stdout:2/658: creat dd/df/fd3 x:0 0 0 2022-01-31T19:44:18.388 INFO:tasks.workunit.client.0.smithi167.stdout:2/659: read dd/d18/d62/fa6 [250205,99213] 0 2022-01-31T19:44:18.391 INFO:tasks.workunit.client.1.smithi171.stdout:4/102: dread d2/d3/d5/f1a [0,4194304] 0 2022-01-31T19:44:18.398 INFO:tasks.workunit.client.1.smithi171.stdout:1/109: dread dd/df/f10 [0,4194304] 0 2022-01-31T19:44:18.409 INFO:tasks.workunit.client.1.smithi171.stdout:3/115: sync 2022-01-31T19:44:18.410 INFO:tasks.workunit.client.1.smithi171.stdout:6/169: sync 2022-01-31T19:44:18.410 INFO:tasks.workunit.client.1.smithi171.stdout:2/168: sync 2022-01-31T19:44:18.410 INFO:tasks.workunit.client.1.smithi171.stdout:6/170: fsync dc/d19/f1a 0 2022-01-31T19:44:18.411 INFO:tasks.workunit.client.1.smithi171.stdout:6/171: write dc/d18/f25 [643303,82611] 0 2022-01-31T19:44:18.411 INFO:tasks.workunit.client.1.smithi171.stdout:3/116: unlink d4/dd/c16 0 2022-01-31T19:44:18.412 INFO:tasks.workunit.client.1.smithi171.stdout:6/172: creat dc/d19/d29/f34 x:0 0 0 2022-01-31T19:44:18.412 INFO:tasks.workunit.client.1.smithi171.stdout:3/117: mknod d4/dd/d1a/c1f 0 2022-01-31T19:44:18.413 INFO:tasks.workunit.client.1.smithi171.stdout:3/118: truncate d4/f13 684488 0 2022-01-31T19:44:18.415 INFO:tasks.workunit.client.1.smithi171.stdout:6/173: rmdir dc 39 2022-01-31T19:44:18.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:18 smithi167 conmon[32206]: audit 2022-01-31T19:44:17. 2022-01-31T19:44:18.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:18 smithi167 conmon[32206]: 159731+0000 mon.smithi167 (mon.0) 664 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:18.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:18 smithi167 conmon[32206]: audit 2022-01-31T19:44:17.878438+0000 mon.smithi167 (mon.0) 665 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:18.427 INFO:tasks.workunit.client.1.smithi171.stdout:4/103: dwrite d2/f14 [0,4194304] 0 2022-01-31T19:44:18.428 INFO:tasks.workunit.client.0.smithi167.stdout:3/917: dwrite d9/f78 [0,4194304] 0 2022-01-31T19:44:18.428 INFO:tasks.workunit.client.0.smithi167.stdout:3/918: unlink d9/l14 0 2022-01-31T19:44:18.428 INFO:tasks.workunit.client.0.smithi167.stdout:3/919: write d9/d56/fab [4948,29074] 0 2022-01-31T19:44:18.429 INFO:tasks.workunit.client.1.smithi171.stdout:4/104: symlink d2/d3/d5/d11/d12/l21 0 2022-01-31T19:44:18.432 INFO:tasks.workunit.client.1.smithi171.stdout:9/94: dwrite f9 [0,4194304] 0 2022-01-31T19:44:18.432 INFO:tasks.workunit.client.0.smithi167.stdout:3/920: symlink d9/dfc/l113 0 2022-01-31T19:44:18.433 INFO:tasks.workunit.client.1.smithi171.stdout:1/110: dwrite f3 [0,4194304] 0 2022-01-31T19:44:18.434 INFO:tasks.workunit.client.1.smithi171.stdout:1/111: write dd/df/d17/f20 [78331,57212] 0 2022-01-31T19:44:18.437 INFO:tasks.workunit.client.1.smithi171.stdout:1/112: stat dd/df/d19/c1c 0 2022-01-31T19:44:18.439 INFO:tasks.workunit.client.1.smithi171.stdout:1/113: write f7 [1257511,881] 0 2022-01-31T19:44:18.440 INFO:tasks.workunit.client.0.smithi167.stdout:3/921: mknod d9/d7f/dbb/c114 0 2022-01-31T19:44:18.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/95: link dd/fe dd/df/d15/f22 0 2022-01-31T19:44:18.441 INFO:tasks.workunit.client.1.smithi171.stdout:9/96: stat dd/c10 0 2022-01-31T19:44:18.441 INFO:tasks.workunit.client.1.smithi171.stdout:9/97: creat dd/d1c/f23 x:0 0 0 2022-01-31T19:44:18.441 INFO:tasks.workunit.client.1.smithi171.stdout:9/98: read - dd/df/f20 zero size 2022-01-31T19:44:18.442 INFO:tasks.workunit.client.0.smithi167.stdout:4/917: rename d0/d6/d7/d26/d31 to d0/d5d/d12f 0 2022-01-31T19:44:18.442 INFO:tasks.workunit.client.0.smithi167.stdout:4/918: mkdir d0/d6/d7/d89/d11c/d130 0 2022-01-31T19:44:18.459 INFO:tasks.workunit.client.1.smithi171.stdout:2/169: dwrite d7/d9/f2c [0,4194304] 0 2022-01-31T19:44:18.463 INFO:tasks.workunit.client.1.smithi171.stdout:2/170: rename d7/d9/f1d to d7/d26/f31 0 2022-01-31T19:44:18.470 INFO:tasks.workunit.client.1.smithi171.stdout:2/171: dread d7/f14 [0,4194304] 0 2022-01-31T19:44:18.472 INFO:tasks.workunit.client.1.smithi171.stdout:2/172: rename d7/f19 to d7/d9/d1a/f32 0 2022-01-31T19:44:18.472 INFO:tasks.workunit.client.1.smithi171.stdout:2/173: unlink d7/c20 0 2022-01-31T19:44:18.472 INFO:tasks.workunit.client.1.smithi171.stdout:2/174: fsync d7/d9/f2c 0 2022-01-31T19:44:18.473 INFO:tasks.workunit.client.1.smithi171.stdout:4/105: dwrite d2/d3/d5/dc/f19 [0,4194304] 0 2022-01-31T19:44:18.474 INFO:tasks.workunit.client.1.smithi171.stdout:4/106: truncate d2/d3/d5/d11/d12/f1d 733651 0 2022-01-31T19:44:18.474 INFO:tasks.workunit.client.1.smithi171.stdout:6/174: dwrite dc/d18/f2a [0,4194304] 0 2022-01-31T19:44:18.474 INFO:tasks.workunit.client.1.smithi171.stdout:3/119: dwrite d4/dd/d1a/f1b [0,4194304] 0 2022-01-31T19:44:18.474 INFO:tasks.workunit.client.1.smithi171.stdout:3/120: truncate d4/f1d 666796 0 2022-01-31T19:44:18.475 INFO:tasks.workunit.client.1.smithi171.stdout:3/121: creat d4/dd/f20 x:0 0 0 2022-01-31T19:44:18.475 INFO:tasks.workunit.client.1.smithi171.stdout:1/114: dwrite f3 [0,4194304] 0 2022-01-31T19:44:18.475 INFO:tasks.workunit.client.1.smithi171.stdout:2/175: symlink d7/l33 0 2022-01-31T19:44:18.476 INFO:tasks.workunit.client.1.smithi171.stdout:9/99: dwrite dd/df/f20 [0,4194304] 0 2022-01-31T19:44:18.479 INFO:tasks.workunit.client.0.smithi167.stdout:4/919: dwrite d0/d6/d7/d26/d40/d4c/ffd [0,4194304] 0 2022-01-31T19:44:18.480 INFO:tasks.workunit.client.1.smithi171.stdout:4/107: link d2/d3/d5/d11/d12/l20 d2/d3/l22 0 2022-01-31T19:44:18.482 INFO:tasks.workunit.client.1.smithi171.stdout:6/175: symlink dc/d19/d30/l35 0 2022-01-31T19:44:18.483 INFO:tasks.workunit.client.0.smithi167.stdout:4/920: unlink d0/d5d/d12f/f71 0 2022-01-31T19:44:18.484 INFO:tasks.workunit.client.1.smithi171.stdout:1/115: creat dd/df/d19/d1b/f22 x:0 0 0 2022-01-31T19:44:18.485 INFO:tasks.workunit.client.1.smithi171.stdout:2/176: truncate d7/f22 3406313 0 2022-01-31T19:44:18.486 INFO:tasks.workunit.client.1.smithi171.stdout:2/177: fdatasync d7/dd/f1b 0 2022-01-31T19:44:18.486 INFO:tasks.workunit.client.1.smithi171.stdout:2/178: truncate d7/d9/f12 2080468 0 2022-01-31T19:44:18.489 INFO:tasks.workunit.client.1.smithi171.stdout:4/108: rmdir d2/d3/d5 39 2022-01-31T19:44:18.489 INFO:tasks.workunit.client.1.smithi171.stdout:4/109: stat d2/c9 0 2022-01-31T19:44:18.495 INFO:tasks.workunit.client.1.smithi171.stdout:1/116: mkdir dd/d23 0 2022-01-31T19:44:18.497 INFO:tasks.workunit.client.1.smithi171.stdout:2/179: mknod d7/c34 0 2022-01-31T19:44:18.497 INFO:tasks.workunit.client.1.smithi171.stdout:3/122: dwrite d4/fb [0,4194304] 0 2022-01-31T19:44:18.498 INFO:tasks.workunit.client.1.smithi171.stdout:6/176: rmdir dc/d18 39 2022-01-31T19:44:18.498 INFO:tasks.workunit.client.1.smithi171.stdout:6/177: dread - dc/d19/d29/f26 zero size 2022-01-31T19:44:18.499 INFO:tasks.workunit.client.1.smithi171.stdout:1/117: mkdir dd/df/d19/d24 0 2022-01-31T19:44:18.500 INFO:tasks.workunit.client.1.smithi171.stdout:1/118: chown dd/df/f10 764350005 1 2022-01-31T19:44:18.500 INFO:tasks.workunit.client.1.smithi171.stdout:4/110: dread d2/f13 [0,4194304] 0 2022-01-31T19:44:18.500 INFO:tasks.workunit.client.1.smithi171.stdout:9/100: dread f9 [0,4194304] 0 2022-01-31T19:44:18.505 INFO:tasks.workunit.client.1.smithi171.stdout:6/178: fsync dc/d18/f25 0 2022-01-31T19:44:18.506 INFO:tasks.workunit.client.0.smithi167.stdout:4/921: dwrite d0/f7a [0,4194304] 0 2022-01-31T19:44:18.507 INFO:tasks.workunit.client.1.smithi171.stdout:1/119: mknod dd/d23/c25 0 2022-01-31T19:44:18.507 INFO:tasks.workunit.client.1.smithi171.stdout:4/111: mknod d2/d3/d5/d11/d12/c23 0 2022-01-31T19:44:18.509 INFO:tasks.workunit.client.0.smithi167.stdout:4/922: symlink d0/d6/d7/d39/d4e/d61/d102/l131 0 2022-01-31T19:44:18.511 INFO:tasks.workunit.client.0.smithi167.stdout:4/923: mknod d0/d6/d7/d89/db1/c132 0 2022-01-31T19:44:18.511 INFO:tasks.workunit.client.0.smithi167.stdout:4/924: truncate d0/d5d/d12f/ddf/da2/dcf/db5/fed 656858 0 2022-01-31T19:44:18.512 INFO:tasks.workunit.client.1.smithi171.stdout:4/112: link d2/f14 d2/d3/d5/d11/d12/d15/f24 0 2022-01-31T19:44:18.514 INFO:tasks.workunit.client.1.smithi171.stdout:9/101: rmdir dd 39 2022-01-31T19:44:18.515 INFO:tasks.workunit.client.1.smithi171.stdout:9/102: dread - dd/d1c/f23 zero size 2022-01-31T19:44:18.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/113: creat d2/d3/f25 x:0 0 0 2022-01-31T19:44:18.517 INFO:tasks.workunit.client.1.smithi171.stdout:3/123: dwrite d4/fa [0,4194304] 0 2022-01-31T19:44:18.517 INFO:tasks.workunit.client.1.smithi171.stdout:3/124: chown d4/l18 261329 1 2022-01-31T19:44:18.517 INFO:tasks.workunit.client.1.smithi171.stdout:3/125: write d4/dd/f1e [254195,2936] 0 2022-01-31T19:44:18.518 INFO:tasks.workunit.client.1.smithi171.stdout:4/114: write d2/d3/f10 [3374400,122462] 0 2022-01-31T19:44:18.518 INFO:tasks.workunit.client.1.smithi171.stdout:4/115: write d2/d3/f1c [671406,102731] 0 2022-01-31T19:44:18.519 INFO:tasks.workunit.client.1.smithi171.stdout:9/103: mknod dd/df/c24 0 2022-01-31T19:44:18.519 INFO:tasks.workunit.client.1.smithi171.stdout:9/104: chown fc 782772106 1 2022-01-31T19:44:18.520 INFO:tasks.workunit.client.1.smithi171.stdout:0/153: sync 2022-01-31T19:44:18.520 INFO:tasks.workunit.client.1.smithi171.stdout:5/88: sync 2022-01-31T19:44:18.521 INFO:tasks.workunit.client.1.smithi171.stdout:8/98: sync 2022-01-31T19:44:18.521 INFO:tasks.workunit.client.1.smithi171.stdout:7/121: sync 2022-01-31T19:44:18.521 INFO:tasks.workunit.client.1.smithi171.stdout:8/99: chown df/f17 4605 1 2022-01-31T19:44:18.523 INFO:tasks.workunit.client.0.smithi167.stdout:4/925: rmdir d0 39 2022-01-31T19:44:18.524 INFO:tasks.workunit.client.0.smithi167.stdout:4/926: mkdir d0/d6/d7/d26/dd2/de5/de6/d118/d133 0 2022-01-31T19:44:18.524 INFO:tasks.workunit.client.0.smithi167.stdout:4/927: write d0/d6/d7/d26/d40/d4c/f4f [2256353,53394] 0 2022-01-31T19:44:18.524 INFO:tasks.workunit.client.1.smithi171.stdout:7/122: write d1/d1b/d7/ff [711582,125928] 0 2022-01-31T19:44:18.525 INFO:tasks.workunit.client.1.smithi171.stdout:7/123: fdatasync d1/d1b/d7/f19 0 2022-01-31T19:44:18.525 INFO:tasks.workunit.client.1.smithi171.stdout:7/124: stat d1/d1b/c1c 0 2022-01-31T19:44:18.525 INFO:tasks.workunit.client.1.smithi171.stdout:8/100: dread f6 [0,4194304] 0 2022-01-31T19:44:18.526 INFO:tasks.workunit.client.1.smithi171.stdout:8/101: write f5 [6006952,25266] 0 2022-01-31T19:44:18.526 INFO:tasks.workunit.client.1.smithi171.stdout:4/116: creat d2/d3/d5/dc/f26 x:0 0 0 2022-01-31T19:44:18.526 INFO:tasks.workunit.client.1.smithi171.stdout:4/117: write d2/d3/d5/dc/f26 [811175,107470] 0 2022-01-31T19:44:18.528 INFO:tasks.workunit.client.1.smithi171.stdout:9/105: creat dd/df/f25 x:0 0 0 2022-01-31T19:44:18.530 INFO:tasks.workunit.client.1.smithi171.stdout:5/89: creat d5/d6/f1a x:0 0 0 2022-01-31T19:44:18.532 INFO:tasks.workunit.client.1.smithi171.stdout:0/154: rename d9/c10 to d9/d1b/d25/c38 0 2022-01-31T19:44:18.532 INFO:tasks.workunit.client.1.smithi171.stdout:0/155: read - d9/d1b/f34 zero size 2022-01-31T19:44:18.532 INFO:tasks.workunit.client.1.smithi171.stdout:0/156: fdatasync f8 0 2022-01-31T19:44:18.534 INFO:tasks.workunit.client.0.smithi167.stdout:0/608: sync 2022-01-31T19:44:18.534 INFO:tasks.workunit.client.0.smithi167.stdout:6/698: sync 2022-01-31T19:44:18.535 INFO:tasks.workunit.client.0.smithi167.stdout:0/609: dread d17/d19/d1c/d39/d44/f93 [0,4194304] 0 2022-01-31T19:44:18.536 INFO:tasks.workunit.client.1.smithi171.stdout:5/90: dread f1 [0,4194304] 0 2022-01-31T19:44:18.537 INFO:tasks.workunit.client.0.smithi167.stdout:0/610: mkdir d17/d3e/d7e/dc7 0 2022-01-31T19:44:18.537 INFO:tasks.workunit.client.0.smithi167.stdout:0/611: creat d17/d69/fc8 x:0 0 0 2022-01-31T19:44:18.537 INFO:tasks.workunit.client.0.smithi167.stdout:0/612: creat d17/d19/d1c/d39/d44/d88/fc9 x:0 0 0 2022-01-31T19:44:18.538 INFO:tasks.workunit.client.0.smithi167.stdout:0/613: creat d17/d69/fca x:0 0 0 2022-01-31T19:44:18.538 INFO:tasks.workunit.client.0.smithi167.stdout:0/614: readlink d17/d2c/lac 0 2022-01-31T19:44:18.539 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:18 smithi171 conmon[35325]: audit 2022-01-31T19:44:17.159731 2022-01-31T19:44:18.539 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:18 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 664 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:18.539 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:18 smithi171 conmon[35325]: audit 2022-01-31T19:44:17.878438+0000 mon.smithi167 (mon.0) 665 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:18.540 INFO:tasks.workunit.client.0.smithi167.stdout:0/615: write d17/d3e/f5f [3626420,3783] 0 2022-01-31T19:44:18.540 INFO:tasks.workunit.client.1.smithi171.stdout:7/125: symlink d1/d24/l27 0 2022-01-31T19:44:18.541 INFO:tasks.workunit.client.1.smithi171.stdout:3/126: mkdir d4/d21 0 2022-01-31T19:44:18.541 INFO:tasks.workunit.client.1.smithi171.stdout:3/127: getdents d4/d21 0 2022-01-31T19:44:18.541 INFO:tasks.workunit.client.1.smithi171.stdout:6/179: dread f2 [0,4194304] 0 2022-01-31T19:44:18.541 INFO:tasks.workunit.client.1.smithi171.stdout:6/180: chown c5 7468050 1 2022-01-31T19:44:18.542 INFO:tasks.workunit.client.0.smithi167.stdout:0/616: creat d17/d3e/d7e/dc7/fcb x:0 0 0 2022-01-31T19:44:18.543 INFO:tasks.workunit.client.1.smithi171.stdout:0/157: mknod d9/db/d22/c39 0 2022-01-31T19:44:18.543 INFO:tasks.workunit.client.1.smithi171.stdout:9/106: creat dd/df/d15/f26 x:0 0 0 2022-01-31T19:44:18.543 INFO:tasks.workunit.client.1.smithi171.stdout:8/102: creat df/d1a/f28 x:0 0 0 2022-01-31T19:44:18.544 INFO:tasks.workunit.client.1.smithi171.stdout:6/181: write f9 [1452527,36863] 0 2022-01-31T19:44:18.545 INFO:tasks.workunit.client.1.smithi171.stdout:5/91: rename d5/d6/cb to d5/d6/d15/c1b 0 2022-01-31T19:44:18.545 INFO:tasks.workunit.client.1.smithi171.stdout:0/158: mkdir d9/db/d22/d3a 0 2022-01-31T19:44:18.545 INFO:tasks.workunit.client.1.smithi171.stdout:1/120: dread f3 [0,4194304] 0 2022-01-31T19:44:18.549 INFO:tasks.workunit.client.0.smithi167.stdout:6/699: dread d8/f73 [0,4194304] 0 2022-01-31T19:44:18.549 INFO:tasks.workunit.client.1.smithi171.stdout:5/92: creat d5/d6/d15/f1c x:0 0 0 2022-01-31T19:44:18.550 INFO:tasks.workunit.client.1.smithi171.stdout:0/159: write f4 [2599209,54656] 0 2022-01-31T19:44:18.550 INFO:tasks.workunit.client.1.smithi171.stdout:0/160: creat d9/f3b x:0 0 0 2022-01-31T19:44:18.550 INFO:tasks.workunit.client.1.smithi171.stdout:0/161: stat d9/f1f 0 2022-01-31T19:44:18.551 INFO:tasks.workunit.client.1.smithi171.stdout:8/103: rmdir df/d18 39 2022-01-31T19:44:18.556 INFO:tasks.workunit.client.1.smithi171.stdout:7/126: dread d1/d1b/d7/ff [0,4194304] 0 2022-01-31T19:44:18.557 INFO:tasks.workunit.client.1.smithi171.stdout:7/127: chown d1/d24/c13 196 1 2022-01-31T19:44:18.557 INFO:tasks.workunit.client.1.smithi171.stdout:0/162: creat d9/d24/f3c x:0 0 0 2022-01-31T19:44:18.557 INFO:tasks.workunit.client.1.smithi171.stdout:8/104: chown f8 15685 1 2022-01-31T19:44:18.557 INFO:tasks.workunit.client.1.smithi171.stdout:8/105: chown ld 1610 1 2022-01-31T19:44:18.558 INFO:tasks.workunit.client.1.smithi171.stdout:8/106: read - f8 zero size 2022-01-31T19:44:18.561 INFO:tasks.workunit.client.1.smithi171.stdout:4/118: dwrite d2/d3/d5/d11/d12/d15/f24 [0,4194304] 0 2022-01-31T19:44:18.563 INFO:tasks.workunit.client.1.smithi171.stdout:4/119: creat d2/d3/d5/d11/f27 x:0 0 0 2022-01-31T19:44:18.563 INFO:tasks.workunit.client.1.smithi171.stdout:4/120: creat d2/d3/f28 x:0 0 0 2022-01-31T19:44:18.566 INFO:tasks.workunit.client.1.smithi171.stdout:4/121: mknod d2/c29 0 2022-01-31T19:44:18.567 INFO:tasks.workunit.client.1.smithi171.stdout:7/128: dread d1/d1b/d7/ff [0,4194304] 0 2022-01-31T19:44:18.568 INFO:tasks.workunit.client.1.smithi171.stdout:9/107: dread f4 [0,4194304] 0 2022-01-31T19:44:18.569 INFO:tasks.workunit.client.1.smithi171.stdout:9/108: chown dd/d1c 451626 1 2022-01-31T19:44:18.569 INFO:tasks.workunit.client.1.smithi171.stdout:9/109: fdatasync dd/d1c/f1f 0 2022-01-31T19:44:18.569 INFO:tasks.workunit.client.1.smithi171.stdout:7/129: creat d1/d24/f28 x:0 0 0 2022-01-31T19:44:18.570 INFO:tasks.workunit.client.1.smithi171.stdout:4/122: read d2/d3/d5/dc/f19 [1879291,78695] 0 2022-01-31T19:44:18.570 INFO:tasks.workunit.client.1.smithi171.stdout:7/130: truncate d1/d1b/d7/f19 316901 0 2022-01-31T19:44:18.572 INFO:tasks.workunit.client.1.smithi171.stdout:4/123: creat d2/d3/d5/d11/d12/f2a x:0 0 0 2022-01-31T19:44:18.573 INFO:tasks.workunit.client.1.smithi171.stdout:7/131: mkdir d1/d1e/d20/d29 0 2022-01-31T19:44:18.574 INFO:tasks.workunit.client.1.smithi171.stdout:6/182: dread fa [0,4194304] 0 2022-01-31T19:44:18.574 INFO:tasks.workunit.client.0.smithi167.stdout:4/928: dwrite d0/d6/d7/d39/d4e/f52 [4194304,4194304] 0 2022-01-31T19:44:18.575 INFO:tasks.workunit.client.1.smithi171.stdout:4/124: write d2/d3/d5/dc/f26 [454040,34222] 0 2022-01-31T19:44:18.576 INFO:tasks.workunit.client.0.smithi167.stdout:4/929: creat d0/d6/d7/d26/f134 x:0 0 0 2022-01-31T19:44:18.576 INFO:tasks.workunit.client.1.smithi171.stdout:7/132: unlink d1/d1b/f10 0 2022-01-31T19:44:18.577 INFO:tasks.workunit.client.0.smithi167.stdout:4/930: creat d0/d6/d7/d39/d4e/d61/d102/f135 x:0 0 0 2022-01-31T19:44:18.578 INFO:tasks.workunit.client.1.smithi171.stdout:4/125: rmdir d2/d3/d5 39 2022-01-31T19:44:18.578 INFO:tasks.workunit.client.1.smithi171.stdout:6/183: truncate f2 1561662 0 2022-01-31T19:44:18.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/931: symlink d0/d5d/d12f/ddf/d75/dd6/l136 0 2022-01-31T19:44:18.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/932: chown d0/d5d/d12f/l10 673 1 2022-01-31T19:44:18.579 INFO:tasks.workunit.client.0.smithi167.stdout:4/933: write d0/d5d/d12f/d5c/de2/f112 [850759,21580] 0 2022-01-31T19:44:18.580 INFO:tasks.workunit.client.1.smithi171.stdout:3/128: dwrite d4/f1d [0,4194304] 0 2022-01-31T19:44:18.580 INFO:tasks.workunit.client.1.smithi171.stdout:3/129: chown d4/f1d 3 1 2022-01-31T19:44:18.580 INFO:tasks.workunit.client.1.smithi171.stdout:3/130: chown d4/dd/c14 6527 1 2022-01-31T19:44:18.581 INFO:tasks.workunit.client.1.smithi171.stdout:3/131: stat d4/fb 0 2022-01-31T19:44:18.581 INFO:tasks.workunit.client.0.smithi167.stdout:4/934: unlink d0/d5d/d12f/ddf/d75/lba 0 2022-01-31T19:44:18.582 INFO:tasks.workunit.client.1.smithi171.stdout:4/126: rename d2/d3/l22 to d2/d3/d5/l2b 0 2022-01-31T19:44:18.582 INFO:tasks.workunit.client.0.smithi167.stdout:4/935: unlink d0/d5d/d12f/ddf/l3e 0 2022-01-31T19:44:18.583 INFO:tasks.workunit.client.0.smithi167.stdout:0/617: dwrite d17/d2c/d87/f8d [4194304,4194304] 0 2022-01-31T19:44:18.584 INFO:tasks.workunit.client.0.smithi167.stdout:6/700: dwrite d8/d62/f77 [0,4194304] 0 2022-01-31T19:44:18.584 INFO:tasks.workunit.client.1.smithi171.stdout:3/132: mknod d4/d21/c22 0 2022-01-31T19:44:18.585 INFO:tasks.workunit.client.1.smithi171.stdout:3/133: readlink d4/l18 0 2022-01-31T19:44:18.586 INFO:tasks.workunit.client.0.smithi167.stdout:0/618: rename d17/d19/d1c/d39/d44/l54 to d17/d33/da4/lcc 0 2022-01-31T19:44:18.586 INFO:tasks.workunit.client.0.smithi167.stdout:0/619: truncate d17/d33/fa3 1039508 0 2022-01-31T19:44:18.587 INFO:tasks.workunit.client.1.smithi171.stdout:0/163: fdatasync d9/d1b/f2d 0 2022-01-31T19:44:18.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/93: dwrite d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:18.588 INFO:tasks.workunit.client.1.smithi171.stdout:9/110: getdents dd/df 0 2022-01-31T19:44:18.589 INFO:tasks.workunit.client.0.smithi167.stdout:0/620: write d17/d3e/f5f [3911549,83759] 0 2022-01-31T19:44:18.591 INFO:tasks.workunit.client.1.smithi171.stdout:0/164: creat d9/d1b/d30/f3d x:0 0 0 2022-01-31T19:44:18.592 INFO:tasks.workunit.client.0.smithi167.stdout:6/701: write d8/fd [3105664,65191] 0 2022-01-31T19:44:18.593 INFO:tasks.workunit.client.0.smithi167.stdout:4/936: rmdir d0/d5d/d12f/ddf/d75 39 2022-01-31T19:44:18.594 INFO:tasks.workunit.client.0.smithi167.stdout:0/621: write d17/d2c/f3c [3471909,43651] 0 2022-01-31T19:44:18.596 INFO:tasks.workunit.client.0.smithi167.stdout:6/702: symlink d8/d62/d92/lfc 0 2022-01-31T19:44:18.596 INFO:tasks.workunit.client.0.smithi167.stdout:6/703: chown d8/d33/d51/d5d/d86/fcc 7 1 2022-01-31T19:44:18.597 INFO:tasks.workunit.client.0.smithi167.stdout:0/622: mkdir d17/d2c/dcd 0 2022-01-31T19:44:18.598 INFO:tasks.workunit.client.1.smithi171.stdout:3/134: dread d4/f1d [0,4194304] 0 2022-01-31T19:44:18.599 INFO:tasks.workunit.client.1.smithi171.stdout:3/135: fsync d4/fa 0 2022-01-31T19:44:18.599 INFO:tasks.workunit.client.1.smithi171.stdout:3/136: chown d4/dd/d1a/c1c 91652040 1 2022-01-31T19:44:18.599 INFO:tasks.workunit.client.1.smithi171.stdout:8/107: dwrite f1 [0,4194304] 0 2022-01-31T19:44:18.599 INFO:tasks.workunit.client.1.smithi171.stdout:8/108: creat df/d1a/f29 x:0 0 0 2022-01-31T19:44:18.600 INFO:tasks.workunit.client.1.smithi171.stdout:9/111: creat dd/df/d15/d1a/f27 x:0 0 0 2022-01-31T19:44:18.601 INFO:tasks.workunit.client.0.smithi167.stdout:6/704: unlink d8/d33/d51/d72/dad/fe5 0 2022-01-31T19:44:18.601 INFO:tasks.workunit.client.0.smithi167.stdout:0/623: creat d17/d3e/d7e/fce x:0 0 0 2022-01-31T19:44:18.601 INFO:tasks.workunit.client.0.smithi167.stdout:6/705: symlink d8/d33/d51/d6a/df1/lfd 0 2022-01-31T19:44:18.602 INFO:tasks.workunit.client.0.smithi167.stdout:6/706: readlink d8/d33/l8c 0 2022-01-31T19:44:18.602 INFO:tasks.workunit.client.0.smithi167.stdout:6/707: read d8/d33/d51/d5d/f6e [306645,51750] 0 2022-01-31T19:44:18.603 INFO:tasks.workunit.client.0.smithi167.stdout:0/624: rename d17/d19/d1c/d39/d44/d88/cbc to d17/d69/ccf 0 2022-01-31T19:44:18.603 INFO:tasks.workunit.client.0.smithi167.stdout:6/708: symlink d8/d33/d51/d5d/d64/dab/lfe 0 2022-01-31T19:44:18.604 INFO:tasks.workunit.client.1.smithi171.stdout:0/165: creat d9/db/d22/d3a/f3e x:0 0 0 2022-01-31T19:44:18.604 INFO:tasks.workunit.client.1.smithi171.stdout:5/94: getdents d5 0 2022-01-31T19:44:18.604 INFO:tasks.workunit.client.1.smithi171.stdout:5/95: chown d5/d6/f16 338165 1 2022-01-31T19:44:18.605 INFO:tasks.workunit.client.1.smithi171.stdout:5/96: chown d5/d6 6 1 2022-01-31T19:44:18.605 INFO:tasks.workunit.client.1.smithi171.stdout:5/97: creat d5/d6/d15/f1d x:0 0 0 2022-01-31T19:44:18.605 INFO:tasks.workunit.client.1.smithi171.stdout:9/112: read dd/df/f20 [271821,38939] 0 2022-01-31T19:44:18.606 INFO:tasks.workunit.client.0.smithi167.stdout:4/937: dread d0/d6/d7/d26/dd2/de5/f91 [0,4194304] 0 2022-01-31T19:44:18.606 INFO:tasks.workunit.client.0.smithi167.stdout:0/625: mkdir d17/d33/da4/dd0 0 2022-01-31T19:44:18.607 INFO:tasks.workunit.client.0.smithi167.stdout:0/626: creat d17/d69/fd1 x:0 0 0 2022-01-31T19:44:18.607 INFO:tasks.workunit.client.0.smithi167.stdout:6/709: mknod d8/d62/cff 0 2022-01-31T19:44:18.607 INFO:tasks.workunit.client.0.smithi167.stdout:4/938: symlink d0/d6/d7/d26/d40/d4c/l137 0 2022-01-31T19:44:18.607 INFO:tasks.workunit.client.0.smithi167.stdout:4/939: write d0/d5d/d12f/ddf/da2/f125 [787293,79880] 0 2022-01-31T19:44:18.608 INFO:tasks.workunit.client.0.smithi167.stdout:0/627: getdents d17/d33 0 2022-01-31T19:44:18.609 INFO:tasks.workunit.client.0.smithi167.stdout:4/940: unlink d0/d6/f76 0 2022-01-31T19:44:18.610 INFO:tasks.workunit.client.1.smithi171.stdout:8/109: mkdir df/d2a 0 2022-01-31T19:44:18.611 INFO:tasks.workunit.client.1.smithi171.stdout:4/127: dwrite d2/d3/d5/f1a [0,4194304] 0 2022-01-31T19:44:18.612 INFO:tasks.workunit.client.0.smithi167.stdout:0/628: truncate fa 1659446 0 2022-01-31T19:44:18.613 INFO:tasks.workunit.client.1.smithi171.stdout:5/98: symlink d5/d6/dd/l1e 0 2022-01-31T19:44:18.613 INFO:tasks.workunit.client.1.smithi171.stdout:5/99: write d5/d6/f1a [541847,73102] 0 2022-01-31T19:44:18.614 INFO:tasks.workunit.client.1.smithi171.stdout:5/100: write d5/d6/dd/f19 [33572,38866] 0 2022-01-31T19:44:18.614 INFO:tasks.workunit.client.0.smithi167.stdout:0/629: write d17/d2c/f60 [3565050,100763] 0 2022-01-31T19:44:18.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/133: dwrite d1/d1b/fc [0,4194304] 0 2022-01-31T19:44:18.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/134: dread - d1/d24/f21 zero size 2022-01-31T19:44:18.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/135: chown d1/d1b/d7/cb 901389 1 2022-01-31T19:44:18.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/136: write d1/d24/f28 [170657,80254] 0 2022-01-31T19:44:18.620 INFO:tasks.workunit.client.1.smithi171.stdout:4/128: rmdir d2/d3/d5/dc 39 2022-01-31T19:44:18.620 INFO:tasks.workunit.client.1.smithi171.stdout:4/129: write d2/f14 [5198696,35995] 0 2022-01-31T19:44:18.620 INFO:tasks.workunit.client.0.smithi167.stdout:1/917: sync 2022-01-31T19:44:18.621 INFO:tasks.workunit.client.0.smithi167.stdout:8/810: sync 2022-01-31T19:44:18.625 INFO:tasks.workunit.client.0.smithi167.stdout:8/811: link d6/f8 d6/db/de/d3d/f107 0 2022-01-31T19:44:18.625 INFO:tasks.workunit.client.0.smithi167.stdout:8/812: write d6/db/d28/d61/dd5/f90 [948208,52716] 0 2022-01-31T19:44:18.626 INFO:tasks.workunit.client.1.smithi171.stdout:4/130: mknod d2/d3/d5/d11/d12/c2c 0 2022-01-31T19:44:18.627 INFO:tasks.workunit.client.0.smithi167.stdout:1/918: getdents d0/d10/d5a/d33/d4b 0 2022-01-31T19:44:18.627 INFO:tasks.workunit.client.0.smithi167.stdout:1/919: stat d0/d10/d5a/d33/d4b/da5/dde/d11c 0 2022-01-31T19:44:18.629 INFO:tasks.workunit.client.1.smithi171.stdout:3/137: dread d4/fc [4194304,4194304] 0 2022-01-31T19:44:18.629 INFO:tasks.workunit.client.1.smithi171.stdout:5/101: dread d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:18.629 INFO:tasks.workunit.client.1.smithi171.stdout:5/102: creat d5/d6/d15/f1f x:0 0 0 2022-01-31T19:44:18.630 INFO:tasks.workunit.client.1.smithi171.stdout:4/131: mknod d2/d3/d5/dc/c2d 0 2022-01-31T19:44:18.631 INFO:tasks.workunit.client.0.smithi167.stdout:8/813: write d6/db/fd [1958909,32223] 0 2022-01-31T19:44:18.633 INFO:tasks.workunit.client.0.smithi167.stdout:8/814: link d6/d1e/f3c d6/db/d28/d61/dd5/dbf/d105/f108 0 2022-01-31T19:44:18.633 INFO:tasks.workunit.client.1.smithi171.stdout:4/132: creat d2/d3/d5/f2e x:0 0 0 2022-01-31T19:44:18.636 INFO:tasks.workunit.client.0.smithi167.stdout:8/815: rename d6/db/de/d18/deb to d6/d1e/d4c/d109 0 2022-01-31T19:44:18.638 INFO:tasks.workunit.client.0.smithi167.stdout:1/920: write d0/d62/d7f/d82/fef [2898757,75088] 0 2022-01-31T19:44:18.638 INFO:tasks.workunit.client.0.smithi167.stdout:1/921: creat d0/d11/d102/ddb/f137 x:0 0 0 2022-01-31T19:44:18.639 INFO:tasks.workunit.client.0.smithi167.stdout:8/816: symlink d6/db/d28/d61/l10a 0 2022-01-31T19:44:18.640 INFO:tasks.workunit.client.1.smithi171.stdout:3/138: write d4/fa [903711,35342] 0 2022-01-31T19:44:18.640 INFO:tasks.workunit.client.1.smithi171.stdout:3/139: stat d4/dd 0 2022-01-31T19:44:18.640 INFO:tasks.workunit.client.1.smithi171.stdout:9/113: dwrite fc [0,4194304] 0 2022-01-31T19:44:18.640 INFO:tasks.workunit.client.1.smithi171.stdout:9/114: write f4 [13222709,69474] 0 2022-01-31T19:44:18.641 INFO:tasks.workunit.client.1.smithi171.stdout:9/115: write dd/d1c/f23 [584050,32858] 0 2022-01-31T19:44:18.641 INFO:tasks.workunit.client.0.smithi167.stdout:8/817: getdents d6/db/d6a/d8d/d9e/d60 0 2022-01-31T19:44:18.642 INFO:tasks.workunit.client.0.smithi167.stdout:8/818: write d6/db/d4e/d73/d88/f96 [783542,27671] 0 2022-01-31T19:44:18.642 INFO:tasks.workunit.client.1.smithi171.stdout:0/166: rename d9/d1b to d9/db/dc/d3f 0 2022-01-31T19:44:18.643 INFO:tasks.workunit.client.0.smithi167.stdout:1/922: rmdir d0/d10/d5a/d33/d4b/da5/da9/dad 39 2022-01-31T19:44:18.644 INFO:tasks.workunit.client.1.smithi171.stdout:3/140: truncate d4/dd/d1a/f1b 78594 0 2022-01-31T19:44:18.645 INFO:tasks.workunit.client.1.smithi171.stdout:3/141: chown d4/dd/c17 2670147 1 2022-01-31T19:44:18.645 INFO:tasks.workunit.client.1.smithi171.stdout:3/142: chown d4/dd/f20 0 1 2022-01-31T19:44:18.646 INFO:tasks.workunit.client.1.smithi171.stdout:9/116: rename dd/c1d to dd/df/c28 0 2022-01-31T19:44:18.647 INFO:tasks.workunit.client.1.smithi171.stdout:9/117: write dd/df/d15/f1b [1042102,31998] 0 2022-01-31T19:44:18.647 INFO:tasks.workunit.client.1.smithi171.stdout:3/143: mkdir d4/d23 0 2022-01-31T19:44:18.649 INFO:tasks.workunit.client.1.smithi171.stdout:3/144: symlink d4/dd/l24 0 2022-01-31T19:44:18.649 INFO:tasks.workunit.client.1.smithi171.stdout:3/145: link d4/dd/c17 d4/dd/c25 0 2022-01-31T19:44:18.653 INFO:tasks.workunit.client.0.smithi167.stdout:1/923: dread d0/d10/d5a/d33/d4b/f6d [0,4194304] 0 2022-01-31T19:44:18.653 INFO:tasks.workunit.client.0.smithi167.stdout:1/924: chown d0/d10/cea 123700 1 2022-01-31T19:44:18.654 INFO:tasks.workunit.client.1.smithi171.stdout:9/118: dread dd/df/f17 [0,4194304] 0 2022-01-31T19:44:18.655 INFO:tasks.workunit.client.1.smithi171.stdout:9/119: chown dd/df/c24 13111639 1 2022-01-31T19:44:18.660 INFO:tasks.workunit.client.0.smithi167.stdout:8/819: dread d6/db/d4e/f4f [0,4194304] 0 2022-01-31T19:44:18.660 INFO:tasks.workunit.client.0.smithi167.stdout:6/710: dwrite d8/d33/d51/d7e/dc4/fbc [0,4194304] 0 2022-01-31T19:44:18.660 INFO:tasks.workunit.client.0.smithi167.stdout:6/711: write d8/d33/d51/d5d/d64/ff7 [660121,139] 0 2022-01-31T19:44:18.661 INFO:tasks.workunit.client.0.smithi167.stdout:1/925: dread d0/d10/d5a/d33/fbb [0,4194304] 0 2022-01-31T19:44:18.662 INFO:tasks.workunit.client.0.smithi167.stdout:8/820: symlink d6/db/d28/daf/dc4/l10b 0 2022-01-31T19:44:18.662 INFO:tasks.workunit.client.0.smithi167.stdout:4/941: dwrite d0/d6/d7/d26/d40/f32 [4194304,4194304] 0 2022-01-31T19:44:18.662 INFO:tasks.workunit.client.0.smithi167.stdout:4/942: chown d0/d6/d7/d26/dd2/l115 31600950 1 2022-01-31T19:44:18.662 INFO:tasks.workunit.client.0.smithi167.stdout:4/943: write d0/d6/d7/d39/d4e/ffb [347453,10859] 0 2022-01-31T19:44:18.663 INFO:tasks.workunit.client.0.smithi167.stdout:4/944: chown d0/d5d/d12f/ddf/da2/dcf/db5/d113/f90 681 1 2022-01-31T19:44:18.663 INFO:tasks.workunit.client.0.smithi167.stdout:0/630: dwrite d17/d19/f4d [0,4194304] 0 2022-01-31T19:44:18.664 INFO:tasks.workunit.client.0.smithi167.stdout:6/712: unlink d8/fb7 0 2022-01-31T19:44:18.666 INFO:tasks.workunit.client.1.smithi171.stdout:8/110: dwrite df/d18/f1e [0,4194304] 0 2022-01-31T19:44:18.666 INFO:tasks.workunit.client.1.smithi171.stdout:4/133: rmdir d2/d3/d5/d11 39 2022-01-31T19:44:18.667 INFO:tasks.workunit.client.0.smithi167.stdout:1/926: symlink d0/d11/l138 0 2022-01-31T19:44:18.669 INFO:tasks.workunit.client.1.smithi171.stdout:8/111: symlink df/d2a/l2b 0 2022-01-31T19:44:18.670 INFO:tasks.workunit.client.1.smithi171.stdout:8/112: readlink df/d18/d19/l1f 0 2022-01-31T19:44:18.671 INFO:tasks.workunit.client.1.smithi171.stdout:4/134: link d2/c9 d2/d3/c2f 0 2022-01-31T19:44:18.672 INFO:tasks.workunit.client.0.smithi167.stdout:4/945: dread d0/d6/d7/f9a [0,4194304] 0 2022-01-31T19:44:18.673 INFO:tasks.workunit.client.1.smithi171.stdout:4/135: getdents d2/d3/d5/d11/d12/d15 0 2022-01-31T19:44:18.676 INFO:tasks.workunit.client.0.smithi167.stdout:6/713: write d8/d62/f77 [6582659,130547] 0 2022-01-31T19:44:18.677 INFO:tasks.workunit.client.1.smithi171.stdout:5/103: dwrite d5/d6/d15/f1d [0,4194304] 0 2022-01-31T19:44:18.677 INFO:tasks.workunit.client.1.smithi171.stdout:5/104: write d5/d6/d15/f1f [123135,89809] 0 2022-01-31T19:44:18.677 INFO:tasks.workunit.client.1.smithi171.stdout:5/105: chown l3 0 1 2022-01-31T19:44:18.684 INFO:tasks.workunit.client.1.smithi171.stdout:0/167: dwrite f8 [0,4194304] 0 2022-01-31T19:44:18.689 INFO:tasks.workunit.client.1.smithi171.stdout:9/120: dwrite dd/d1c/f1f [0,4194304] 0 2022-01-31T19:44:18.689 INFO:tasks.workunit.client.1.smithi171.stdout:0/168: write f4 [4156655,114993] 0 2022-01-31T19:44:18.690 INFO:tasks.workunit.client.0.smithi167.stdout:0/631: rename d17/d2c/l6d to d17/d19/ld2 0 2022-01-31T19:44:18.692 INFO:tasks.workunit.client.0.smithi167.stdout:1/927: link d0/d11/l132 d0/d10/d5a/df2/l139 0 2022-01-31T19:44:18.692 INFO:tasks.workunit.client.0.smithi167.stdout:1/928: dread - d0/d11/d46/f128 zero size 2022-01-31T19:44:18.692 INFO:tasks.workunit.client.0.smithi167.stdout:1/929: chown d0/d62/db7/cf8 631 1 2022-01-31T19:44:18.693 INFO:tasks.workunit.client.0.smithi167.stdout:1/930: chown d0/c2 97 1 2022-01-31T19:44:18.693 INFO:tasks.workunit.client.0.smithi167.stdout:1/931: chown d0/d62/db7 7330 1 2022-01-31T19:44:18.693 INFO:tasks.workunit.client.0.smithi167.stdout:1/932: fdatasync d0/d10/d5a/d33/d4b/da5/fe0 0 2022-01-31T19:44:18.694 INFO:tasks.workunit.client.0.smithi167.stdout:1/933: write d0/d10/d5a/d33/d4b/da5/fa0 [3836625,54834] 0 2022-01-31T19:44:18.694 INFO:tasks.workunit.client.1.smithi171.stdout:9/121: link dd/df/d15/d1a/f27 dd/d1c/f29 0 2022-01-31T19:44:18.695 INFO:tasks.workunit.client.1.smithi171.stdout:0/169: mknod d9/db/dc/c40 0 2022-01-31T19:44:18.695 INFO:tasks.workunit.client.0.smithi167.stdout:4/946: rename d0/d6/d7/d39/d4e/d61/fcc to d0/d6/d7/d89/d11c/d130/f138 0 2022-01-31T19:44:18.700 INFO:tasks.workunit.client.0.smithi167.stdout:0/632: rename d17/d19/d1c/d39/f74 to d17/d3e/d7e/fd3 0 2022-01-31T19:44:18.701 INFO:tasks.workunit.client.0.smithi167.stdout:8/821: dwrite d6/d1e/f41 [0,4194304] 0 2022-01-31T19:44:18.702 INFO:tasks.workunit.client.0.smithi167.stdout:0/633: mknod d17/d19/d1c/d39/d7b/cd4 0 2022-01-31T19:44:18.704 INFO:tasks.workunit.client.0.smithi167.stdout:0/634: symlink d17/d3e/d7e/ld5 0 2022-01-31T19:44:18.705 INFO:tasks.workunit.client.1.smithi171.stdout:0/170: getdents d9/db/d22/d3a 0 2022-01-31T19:44:18.705 INFO:tasks.workunit.client.1.smithi171.stdout:3/146: dread d4/dd/d1a/f1b [0,4194304] 0 2022-01-31T19:44:18.705 INFO:tasks.workunit.client.1.smithi171.stdout:3/147: write d4/dd/d1a/f1b [763082,85365] 0 2022-01-31T19:44:18.710 INFO:tasks.workunit.client.0.smithi167.stdout:6/714: fdatasync d8/d62/f77 0 2022-01-31T19:44:18.710 INFO:tasks.workunit.client.0.smithi167.stdout:6/715: chown d8/d62/fc6 28283554 1 2022-01-31T19:44:18.712 INFO:tasks.workunit.client.0.smithi167.stdout:8/822: write d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/fc5 [1920548,38438] 0 2022-01-31T19:44:18.717 INFO:tasks.workunit.client.1.smithi171.stdout:0/171: write d9/db/f21 [3183545,44132] 0 2022-01-31T19:44:18.722 INFO:tasks.workunit.client.1.smithi171.stdout:4/136: dwrite d2/f1e [4194304,4194304] 0 2022-01-31T19:44:18.722 INFO:tasks.workunit.client.1.smithi171.stdout:4/137: dread - d2/d3/d5/d11/d12/f2a zero size 2022-01-31T19:44:18.723 INFO:tasks.workunit.client.0.smithi167.stdout:6/716: dread d8/d33/d51/ff4 [0,4194304] 0 2022-01-31T19:44:18.723 INFO:tasks.workunit.client.0.smithi167.stdout:6/717: chown d8/d62/d92/fb8 1273827032 1 2022-01-31T19:44:18.724 INFO:tasks.workunit.client.1.smithi171.stdout:5/106: dwrite d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:18.726 INFO:tasks.workunit.client.0.smithi167.stdout:6/718: rename d8/lc to d8/d33/d51/d7e/dc4/l100 0 2022-01-31T19:44:18.726 INFO:tasks.workunit.client.0.smithi167.stdout:6/719: stat d8/c10 0 2022-01-31T19:44:18.726 INFO:tasks.workunit.client.0.smithi167.stdout:6/720: creat d8/d33/d51/d5d/d64/f101 x:0 0 0 2022-01-31T19:44:18.727 INFO:tasks.workunit.client.1.smithi171.stdout:5/107: read d5/d6/d15/f1d [3124510,34386] 0 2022-01-31T19:44:18.728 INFO:tasks.workunit.client.1.smithi171.stdout:5/108: link d5/d6/f16 d5/d6/dd/f20 0 2022-01-31T19:44:18.738 INFO:tasks.workunit.client.1.smithi171.stdout:5/109: dread d5/f8 [0,4194304] 0 2022-01-31T19:44:18.738 INFO:tasks.workunit.client.1.smithi171.stdout:5/110: creat d5/d6/f21 x:0 0 0 2022-01-31T19:44:18.739 INFO:tasks.workunit.client.1.smithi171.stdout:4/138: dread d2/d3/d5/f1a [0,4194304] 0 2022-01-31T19:44:18.740 INFO:tasks.workunit.client.1.smithi171.stdout:4/139: truncate d2/d3/d5/d11/d12/f1d 1398775 0 2022-01-31T19:44:18.741 INFO:tasks.workunit.client.1.smithi171.stdout:4/140: getdents d2/d3/d5/dc 0 2022-01-31T19:44:18.742 INFO:tasks.workunit.client.0.smithi167.stdout:2/660: sync 2022-01-31T19:44:18.743 INFO:tasks.workunit.client.1.smithi171.stdout:4/141: creat d2/d3/d5/d11/d12/f30 x:0 0 0 2022-01-31T19:44:18.743 INFO:tasks.workunit.client.1.smithi171.stdout:4/142: stat d2/d3/d5/l18 0 2022-01-31T19:44:18.744 INFO:tasks.workunit.client.1.smithi171.stdout:0/172: write d9/db/dc/d3f/f2d [461567,127294] 0 2022-01-31T19:44:18.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/771: sync 2022-01-31T19:44:18.750 INFO:tasks.workunit.client.0.smithi167.stdout:9/772: chown dd/d19/lfd 16268 1 2022-01-31T19:44:18.750 INFO:tasks.workunit.client.0.smithi167.stdout:9/773: stat dd/d19/d59/fce 0 2022-01-31T19:44:18.750 INFO:tasks.workunit.client.0.smithi167.stdout:9/774: creat dd/d19/d24/d56/dc9/f10c x:0 0 0 2022-01-31T19:44:18.750 INFO:tasks.workunit.client.0.smithi167.stdout:3/922: sync 2022-01-31T19:44:18.751 INFO:tasks.workunit.client.0.smithi167.stdout:9/775: getdents dd/d19/d24/d50/d5d/dfa 0 2022-01-31T19:44:18.753 INFO:tasks.workunit.client.0.smithi167.stdout:3/923: mknod d9/d21/ddd/c115 0 2022-01-31T19:44:18.762 INFO:tasks.workunit.client.0.smithi167.stdout:1/934: dwrite d0/d62/f11a [0,4194304] 0 2022-01-31T19:44:18.763 INFO:tasks.workunit.client.0.smithi167.stdout:1/935: creat d0/d62/f13a x:0 0 0 2022-01-31T19:44:18.763 INFO:tasks.workunit.client.0.smithi167.stdout:8/823: dwrite d6/db/d36/f84 [0,4194304] 0 2022-01-31T19:44:18.764 INFO:tasks.workunit.client.1.smithi171.stdout:3/148: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:18.764 INFO:tasks.workunit.client.0.smithi167.stdout:0/635: dwrite d17/d19/d1c/d39/d44/d88/f8a [0,4194304] 0 2022-01-31T19:44:18.766 INFO:tasks.workunit.client.1.smithi171.stdout:3/149: rmdir d4/d23 0 2022-01-31T19:44:18.766 INFO:tasks.workunit.client.1.smithi171.stdout:5/111: dread d5/d6/f1a [0,4194304] 0 2022-01-31T19:44:18.766 INFO:tasks.workunit.client.1.smithi171.stdout:9/122: dwrite dd/df/f21 [0,4194304] 0 2022-01-31T19:44:18.767 INFO:tasks.workunit.client.0.smithi167.stdout:1/936: creat d0/d10/d20/f13b x:0 0 0 2022-01-31T19:44:18.770 INFO:tasks.workunit.client.1.smithi171.stdout:4/143: rmdir d2/d3/d5/d11/d12 39 2022-01-31T19:44:18.770 INFO:tasks.workunit.client.1.smithi171.stdout:4/144: fsync d2/f13 0 2022-01-31T19:44:18.771 INFO:tasks.workunit.client.0.smithi167.stdout:8/824: mkdir d6/db/d28/d10c 0 2022-01-31T19:44:18.771 INFO:tasks.workunit.client.0.smithi167.stdout:6/721: dwrite d8/d33/d51/ff4 [0,4194304] 0 2022-01-31T19:44:18.774 INFO:tasks.workunit.client.0.smithi167.stdout:2/661: dwrite dd/df/d1b/d3d/f44 [0,4194304] 0 2022-01-31T19:44:18.775 INFO:tasks.workunit.client.0.smithi167.stdout:9/776: dwrite dd/f64 [0,4194304] 0 2022-01-31T19:44:18.775 INFO:tasks.workunit.client.0.smithi167.stdout:9/777: chown dd/d35/d105 3377 1 2022-01-31T19:44:18.775 INFO:tasks.workunit.client.0.smithi167.stdout:8/825: link d6/db/d4e/f58 d6/d1e/d2d/f10d 0 2022-01-31T19:44:18.775 INFO:tasks.workunit.client.0.smithi167.stdout:9/778: write dd/d17/d7f/fcb [560494,86950] 0 2022-01-31T19:44:18.777 INFO:tasks.workunit.client.1.smithi171.stdout:5/112: creat d5/d6/dd/f22 x:0 0 0 2022-01-31T19:44:18.777 INFO:tasks.workunit.client.1.smithi171.stdout:0/173: dread d9/f13 [0,4194304] 0 2022-01-31T19:44:18.777 INFO:tasks.workunit.client.1.smithi171.stdout:9/123: creat dd/d1c/f2a x:0 0 0 2022-01-31T19:44:18.777 INFO:tasks.workunit.client.0.smithi167.stdout:8/826: write d6/db/de/d3d/f107 [1970942,536] 0 2022-01-31T19:44:18.778 INFO:tasks.workunit.client.0.smithi167.stdout:1/937: write d0/d10/d5a/d33/d4b/fa2 [1921379,44836] 0 2022-01-31T19:44:18.778 INFO:tasks.workunit.client.0.smithi167.stdout:2/662: rename dd/df/d1b/d2f/f4d to dd/df/d3f/d51/fd4 0 2022-01-31T19:44:18.779 INFO:tasks.workunit.client.0.smithi167.stdout:1/938: rename d0/d10/d5a/d33/d4b/da5/fa0 to d0/d10/d5a/d33/d97/d117/f13c 0 2022-01-31T19:44:18.779 INFO:tasks.workunit.client.1.smithi171.stdout:4/145: rename d2/d3/d5/l18 to d2/d3/d5/d11/d12/d15/l31 0 2022-01-31T19:44:18.779 INFO:tasks.workunit.client.1.smithi171.stdout:0/174: write f3 [3777547,50848] 0 2022-01-31T19:44:18.781 INFO:tasks.workunit.client.0.smithi167.stdout:2/663: dread dd/fbf [0,4194304] 0 2022-01-31T19:44:18.781 INFO:tasks.workunit.client.0.smithi167.stdout:2/664: dread - dd/df/f9a zero size 2022-01-31T19:44:18.781 INFO:tasks.workunit.client.0.smithi167.stdout:1/939: rename d0/d10/d5a/d33/d4b/d1f/fee to d0/d62/d7f/f13d 0 2022-01-31T19:44:18.782 INFO:tasks.workunit.client.1.smithi171.stdout:3/150: link d4/dd/l15 d4/l26 0 2022-01-31T19:44:18.782 INFO:tasks.workunit.client.1.smithi171.stdout:5/113: truncate d5/d6/f16 212085 0 2022-01-31T19:44:18.782 INFO:tasks.workunit.client.1.smithi171.stdout:9/124: truncate fc 3612839 0 2022-01-31T19:44:18.782 INFO:tasks.workunit.client.1.smithi171.stdout:9/125: write dd/df/d15/f22 [533769,51325] 0 2022-01-31T19:44:18.783 INFO:tasks.workunit.client.1.smithi171.stdout:3/151: mkdir d4/dd/d27 0 2022-01-31T19:44:18.783 INFO:tasks.workunit.client.0.smithi167.stdout:1/940: getdents d0/d10/d5a/d33/d97 0 2022-01-31T19:44:18.784 INFO:tasks.workunit.client.1.smithi171.stdout:5/114: mkdir d5/d6/d15/d23 0 2022-01-31T19:44:18.784 INFO:tasks.workunit.client.1.smithi171.stdout:5/115: fdatasync d5/d6/f21 0 2022-01-31T19:44:18.785 INFO:tasks.workunit.client.1.smithi171.stdout:3/152: mkdir d4/dd/d27/d28 0 2022-01-31T19:44:18.785 INFO:tasks.workunit.client.1.smithi171.stdout:5/116: symlink d5/d6/d15/d23/l24 0 2022-01-31T19:44:18.786 INFO:tasks.workunit.client.1.smithi171.stdout:5/117: write d5/d6/dd/f19 [130754,73020] 0 2022-01-31T19:44:18.787 INFO:tasks.workunit.client.1.smithi171.stdout:5/118: creat d5/d6/d15/d23/f25 x:0 0 0 2022-01-31T19:44:18.788 INFO:tasks.workunit.client.0.smithi167.stdout:9/779: dread dd/d19/d24/d56/f85 [0,4194304] 0 2022-01-31T19:44:18.788 INFO:tasks.workunit.client.0.smithi167.stdout:9/780: truncate dd/d17/d7f/def/fea 299615 0 2022-01-31T19:44:18.788 INFO:tasks.workunit.client.0.smithi167.stdout:9/781: chown dd/d17/dc4 5248 1 2022-01-31T19:44:18.789 INFO:tasks.workunit.client.1.smithi171.stdout:5/119: mkdir d5/d6/d26 0 2022-01-31T19:44:18.790 INFO:tasks.workunit.client.1.smithi171.stdout:5/120: creat d5/d6/f27 x:0 0 0 2022-01-31T19:44:18.791 INFO:tasks.workunit.client.1.smithi171.stdout:5/121: mknod d5/d6/d26/c28 0 2022-01-31T19:44:18.793 INFO:tasks.workunit.client.0.smithi167.stdout:8/827: dread d6/db/d28/d61/dd5/dbf/d105/f108 [0,4194304] 0 2022-01-31T19:44:18.795 INFO:tasks.workunit.client.1.smithi171.stdout:3/153: dread d4/fa [0,4194304] 0 2022-01-31T19:44:18.795 INFO:tasks.workunit.client.1.smithi171.stdout:3/154: write d4/dd/f20 [432049,81665] 0 2022-01-31T19:44:18.795 INFO:tasks.workunit.client.1.smithi171.stdout:9/126: dread dd/df/f17 [0,4194304] 0 2022-01-31T19:44:18.796 INFO:tasks.workunit.client.1.smithi171.stdout:9/127: creat dd/df/d15/d1a/f2b x:0 0 0 2022-01-31T19:44:18.796 INFO:tasks.workunit.client.1.smithi171.stdout:9/128: truncate dd/df/d15/f26 114327 0 2022-01-31T19:44:18.796 INFO:tasks.workunit.client.1.smithi171.stdout:3/155: mkdir d4/d21/d29 0 2022-01-31T19:44:18.798 INFO:tasks.workunit.client.1.smithi171.stdout:9/129: creat dd/df/d15/f2c x:0 0 0 2022-01-31T19:44:18.798 INFO:tasks.workunit.client.1.smithi171.stdout:9/130: fsync f4 0 2022-01-31T19:44:18.799 INFO:tasks.workunit.client.0.smithi167.stdout:2/665: dread dd/df/d1b/d3d/d4a/d4b/d54/f8b [0,4194304] 0 2022-01-31T19:44:18.799 INFO:tasks.workunit.client.0.smithi167.stdout:0/636: dwrite d17/d19/d1c/d39/d44/d88/fc9 [0,4194304] 0 2022-01-31T19:44:18.799 INFO:tasks.workunit.client.0.smithi167.stdout:0/637: write d17/d69/f78 [583115,70127] 0 2022-01-31T19:44:18.800 INFO:tasks.workunit.client.1.smithi171.stdout:3/156: creat d4/d21/f2a x:0 0 0 2022-01-31T19:44:18.803 INFO:tasks.workunit.client.0.smithi167.stdout:0/638: creat d17/d2c/dcd/fd6 x:0 0 0 2022-01-31T19:44:18.803 INFO:tasks.workunit.client.0.smithi167.stdout:0/639: chown d17/d33/da4/faa 20 1 2022-01-31T19:44:18.809 INFO:tasks.workunit.client.1.smithi171.stdout:5/122: rename d5/d6/d26 to d5/d6/d15/d29 0 2022-01-31T19:44:18.810 INFO:tasks.workunit.client.1.smithi171.stdout:5/123: chown d5/d6/d15/d23 9 1 2022-01-31T19:44:18.811 INFO:tasks.workunit.client.1.smithi171.stdout:0/175: dread f4 [0,4194304] 0 2022-01-31T19:44:18.813 INFO:tasks.workunit.client.1.smithi171.stdout:9/131: mknod dd/df/d15/d1a/c2d 0 2022-01-31T19:44:18.815 INFO:tasks.workunit.client.1.smithi171.stdout:5/124: truncate d5/d6/d15/f1d 2321654 0 2022-01-31T19:44:18.816 INFO:tasks.workunit.client.1.smithi171.stdout:3/157: dread d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:18.817 INFO:tasks.workunit.client.1.smithi171.stdout:0/176: rmdir d9/db/dc 39 2022-01-31T19:44:18.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/177: creat d9/db/f41 x:0 0 0 2022-01-31T19:44:18.819 INFO:tasks.workunit.client.1.smithi171.stdout:3/158: truncate d4/fb 1196442 0 2022-01-31T19:44:18.819 INFO:tasks.workunit.client.1.smithi171.stdout:3/159: write d4/d21/f2a [58397,6645] 0 2022-01-31T19:44:18.819 INFO:tasks.workunit.client.1.smithi171.stdout:3/160: chown d4/d21/d29 18 1 2022-01-31T19:44:18.825 INFO:tasks.workunit.client.0.smithi167.stdout:9/782: dwrite dd/d19/d24/d56/dc9/fd7 [0,4194304] 0 2022-01-31T19:44:18.826 INFO:tasks.workunit.client.0.smithi167.stdout:9/783: creat dd/d19/d24/d56/dc9/f10d x:0 0 0 2022-01-31T19:44:18.826 INFO:tasks.workunit.client.0.smithi167.stdout:9/784: dread - dd/d19/d59/fae zero size 2022-01-31T19:44:18.827 INFO:tasks.workunit.client.0.smithi167.stdout:1/941: dwrite d0/d10/d5a/df2/f103 [0,4194304] 0 2022-01-31T19:44:18.832 INFO:tasks.workunit.client.0.smithi167.stdout:1/942: mkdir d0/d62/d7f/d13e 0 2022-01-31T19:44:18.833 INFO:tasks.workunit.client.0.smithi167.stdout:1/943: chown d0/d100/f115 199147247 1 2022-01-31T19:44:18.834 INFO:tasks.workunit.client.0.smithi167.stdout:9/785: symlink dd/d6b/l10e 0 2022-01-31T19:44:18.836 INFO:tasks.workunit.client.0.smithi167.stdout:2/666: dwrite dd/df/d1b/d3d/d4a/da7/fbb [0,4194304] 0 2022-01-31T19:44:18.837 INFO:tasks.workunit.client.0.smithi167.stdout:1/944: mkdir d0/d10/d5a/df2/d13f 0 2022-01-31T19:44:18.837 INFO:tasks.workunit.client.0.smithi167.stdout:1/945: creat d0/d11/f140 x:0 0 0 2022-01-31T19:44:18.837 INFO:tasks.workunit.client.0.smithi167.stdout:9/786: symlink dd/d17/d7f/def/l10f 0 2022-01-31T19:44:18.839 INFO:tasks.workunit.client.1.smithi171.stdout:5/125: getdents d5/d6/d15 0 2022-01-31T19:44:18.840 INFO:tasks.workunit.client.0.smithi167.stdout:8/828: dwrite d6/f1d [4194304,4194304] 0 2022-01-31T19:44:18.841 INFO:tasks.workunit.client.1.smithi171.stdout:0/178: dwrite d9/db/f31 [0,4194304] 0 2022-01-31T19:44:18.841 INFO:tasks.workunit.client.1.smithi171.stdout:9/132: dwrite dd/df/f17 [0,4194304] 0 2022-01-31T19:44:18.841 INFO:tasks.workunit.client.1.smithi171.stdout:9/133: truncate dd/df/d15/f16 1081650 0 2022-01-31T19:44:18.842 INFO:tasks.workunit.client.1.smithi171.stdout:5/126: symlink d5/d6/dd/l2a 0 2022-01-31T19:44:18.845 INFO:tasks.workunit.client.0.smithi167.stdout:2/667: symlink dd/df/d1b/d3d/ld5 0 2022-01-31T19:44:18.847 INFO:tasks.workunit.client.1.smithi171.stdout:9/134: stat dd/df/c28 0 2022-01-31T19:44:18.847 INFO:tasks.workunit.client.1.smithi171.stdout:9/135: truncate dd/d1c/f2a 192524 0 2022-01-31T19:44:18.847 INFO:tasks.workunit.client.1.smithi171.stdout:5/127: symlink d5/d6/dd/l2b 0 2022-01-31T19:44:18.847 INFO:tasks.workunit.client.1.smithi171.stdout:5/128: dread - d5/d6/f27 zero size 2022-01-31T19:44:18.848 INFO:tasks.workunit.client.1.smithi171.stdout:5/129: symlink d5/d6/d15/l2c 0 2022-01-31T19:44:18.850 INFO:tasks.workunit.client.0.smithi167.stdout:2/668: link dd/df/d1b/d2f/d36/ca8 dd/df/d3f/cd6 0 2022-01-31T19:44:18.853 INFO:tasks.workunit.client.1.smithi171.stdout:5/130: mknod d5/d6/d15/d29/c2d 0 2022-01-31T19:44:18.853 INFO:tasks.workunit.client.0.smithi167.stdout:2/669: symlink dd/df/d1b/d3d/d4a/d4b/d74/d85/dc4/ld7 0 2022-01-31T19:44:18.854 INFO:tasks.workunit.client.0.smithi167.stdout:2/670: unlink dd/df/d3f/d51/d5f/l72 0 2022-01-31T19:44:18.855 INFO:tasks.workunit.client.0.smithi167.stdout:9/787: dread dd/ff [0,4194304] 0 2022-01-31T19:44:18.856 INFO:tasks.workunit.client.1.smithi171.stdout:0/179: dread d9/f1d [0,4194304] 0 2022-01-31T19:44:18.857 INFO:tasks.workunit.client.0.smithi167.stdout:2/671: rename dd/d18/f31 to dd/d18/db7/fd8 0 2022-01-31T19:44:18.859 INFO:tasks.workunit.client.0.smithi167.stdout:2/672: mknod dd/df/d1b/d3d/d4a/d4b/d54/cd9 0 2022-01-31T19:44:18.865 INFO:tasks.workunit.client.1.smithi171.stdout:3/161: dwrite d4/d21/f2a [0,4194304] 0 2022-01-31T19:44:18.866 INFO:tasks.workunit.client.1.smithi171.stdout:3/162: chown d4/dd/l15 0 1 2022-01-31T19:44:18.868 INFO:tasks.workunit.client.0.smithi167.stdout:8/829: dwrite d6/db/de/d18/f1b [0,4194304] 0 2022-01-31T19:44:18.868 INFO:tasks.workunit.client.0.smithi167.stdout:9/788: unlink dd/d6b/dc1/feb 0 2022-01-31T19:44:18.869 INFO:tasks.workunit.client.0.smithi167.stdout:2/673: truncate dd/df/d3f/f8e 294702 0 2022-01-31T19:44:18.869 INFO:tasks.workunit.client.0.smithi167.stdout:1/946: getdents d0/d11 0 2022-01-31T19:44:18.869 INFO:tasks.workunit.client.0.smithi167.stdout:1/947: write d0/f3a [3376829,113485] 0 2022-01-31T19:44:18.869 INFO:tasks.workunit.client.0.smithi167.stdout:1/948: chown d0/d10/d5a/d33/d4b/d1f/f7b 45177 1 2022-01-31T19:44:18.870 INFO:tasks.workunit.client.0.smithi167.stdout:1/949: creat d0/d11/d102/f141 x:0 0 0 2022-01-31T19:44:18.870 INFO:tasks.workunit.client.0.smithi167.stdout:9/789: getdents dd/d17/dc4 0 2022-01-31T19:44:18.870 INFO:tasks.workunit.client.0.smithi167.stdout:2/674: getdents dd/df/d1b/d3d/d4a/d4b/d74/d85 0 2022-01-31T19:44:18.871 INFO:tasks.workunit.client.0.smithi167.stdout:2/675: truncate dd/df/d1b/d3d/d4a/d4b/d54/fbd 613212 0 2022-01-31T19:44:18.871 INFO:tasks.workunit.client.0.smithi167.stdout:2/676: dread - dd/df/d1b/d3d/d4a/fac zero size 2022-01-31T19:44:18.871 INFO:tasks.workunit.client.0.smithi167.stdout:8/830: link d6/db/d6a/d8d/d9e/lf2 d6/l10e 0 2022-01-31T19:44:18.878 INFO:tasks.workunit.client.0.smithi167.stdout:1/950: mkdir d0/d62/db7/d142 0 2022-01-31T19:44:18.878 INFO:tasks.workunit.client.0.smithi167.stdout:1/951: write d0/d75/d7a/ff6 [76237,83794] 0 2022-01-31T19:44:18.880 INFO:tasks.workunit.client.0.smithi167.stdout:8/831: getdents d6/db/de/d18 0 2022-01-31T19:44:18.886 INFO:tasks.workunit.client.0.smithi167.stdout:1/952: mkdir d0/d10/d5a/d33/d4b/da5/dde/d143 0 2022-01-31T19:44:18.887 INFO:tasks.workunit.client.0.smithi167.stdout:1/953: readlink d0/d10/d5a/d33/d4b/d1f/l9b 0 2022-01-31T19:44:18.892 INFO:tasks.workunit.client.0.smithi167.stdout:2/677: write dd/df/d22/d96/d9e/fb2 [618129,10985] 0 2022-01-31T19:44:19.017 INFO:tasks.workunit.client.0.smithi167.stdout:2/678: readlink dd/d18/db7/lbc 0 2022-01-31T19:44:19.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/131: rename d5/d6/d15/d29/c2d to d5/d6/d15/d23/c2e 0 2022-01-31T19:44:19.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/132: write d5/d6/d15/f1d [3124226,123771] 0 2022-01-31T19:44:19.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/133: readlink d5/d6/l7 0 2022-01-31T19:44:19.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/134: getdents d5 0 2022-01-31T19:44:19.029 INFO:tasks.workunit.client.1.smithi171.stdout:5/135: truncate f1 4876772 0 2022-01-31T19:44:19.029 INFO:tasks.workunit.client.1.smithi171.stdout:5/136: truncate d5/d6/f16 242241 0 2022-01-31T19:44:19.029 INFO:tasks.workunit.client.1.smithi171.stdout:0/180: creat d9/db/d22/d3a/f42 x:0 0 0 2022-01-31T19:44:19.029 INFO:tasks.workunit.client.1.smithi171.stdout:0/181: fdatasync d9/f17 0 2022-01-31T19:44:19.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/163: rename d4/f13 to d4/dd/d1a/f2b 0 2022-01-31T19:44:19.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/164: read d4/dd/f1e [118844,128065] 0 2022-01-31T19:44:19.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/165: unlink d4/l18 0 2022-01-31T19:44:19.030 INFO:tasks.workunit.client.1.smithi171.stdout:0/182: getdents d9 0 2022-01-31T19:44:19.031 INFO:tasks.workunit.client.1.smithi171.stdout:3/166: link d4/l26 d4/l2c 0 2022-01-31T19:44:19.031 INFO:tasks.workunit.client.1.smithi171.stdout:3/167: creat d4/dd/d27/d28/f2d x:0 0 0 2022-01-31T19:44:19.031 INFO:tasks.workunit.client.1.smithi171.stdout:3/168: mkdir d4/dd/d27/d2e 0 2022-01-31T19:44:19.031 INFO:tasks.workunit.client.1.smithi171.stdout:3/169: rename d4/dd/c17 to d4/dd/d27/d28/c2f 0 2022-01-31T19:44:19.032 INFO:tasks.workunit.client.1.smithi171.stdout:5/137: dwrite f1 [0,4194304] 0 2022-01-31T19:44:19.032 INFO:tasks.workunit.client.1.smithi171.stdout:3/170: write d4/fc [3386471,60509] 0 2022-01-31T19:44:19.032 INFO:tasks.workunit.client.1.smithi171.stdout:3/171: dread d4/fb [0,4194304] 0 2022-01-31T19:44:19.032 INFO:tasks.workunit.client.1.smithi171.stdout:3/172: chown d4/dd/d27/d28/f2d 0 1 2022-01-31T19:44:19.032 INFO:tasks.workunit.client.1.smithi171.stdout:5/138: dwrite d5/f8 [0,4194304] 0 2022-01-31T19:44:19.033 INFO:tasks.workunit.client.1.smithi171.stdout:5/139: rename d5/d6/dd/l1e to d5/d6/d15/d29/l2f 0 2022-01-31T19:44:19.033 INFO:tasks.workunit.client.1.smithi171.stdout:5/140: unlink d5/d6/d15/d23/c2e 0 2022-01-31T19:44:19.033 INFO:tasks.workunit.client.1.smithi171.stdout:2/180: sync 2022-01-31T19:44:19.033 INFO:tasks.workunit.client.1.smithi171.stdout:2/181: creat d7/dd/f35 x:0 0 0 2022-01-31T19:44:19.034 INFO:tasks.workunit.client.1.smithi171.stdout:2/182: link d7/dd/c21 d7/dd/d27/c36 0 2022-01-31T19:44:19.034 INFO:tasks.workunit.client.1.smithi171.stdout:2/183: truncate d7/f11 842591 0 2022-01-31T19:44:19.034 INFO:tasks.workunit.client.1.smithi171.stdout:2/184: symlink d7/l37 0 2022-01-31T19:44:19.034 INFO:tasks.workunit.client.1.smithi171.stdout:0/183: dwrite d9/f17 [0,4194304] 0 2022-01-31T19:44:19.034 INFO:tasks.workunit.client.1.smithi171.stdout:3/173: dwrite d4/dd/d27/d28/f2d [0,4194304] 0 2022-01-31T19:44:19.035 INFO:tasks.workunit.client.1.smithi171.stdout:0/184: symlink d9/db/d22/l43 0 2022-01-31T19:44:19.035 INFO:tasks.workunit.client.1.smithi171.stdout:0/185: readlink d9/l18 0 2022-01-31T19:44:19.035 INFO:tasks.workunit.client.1.smithi171.stdout:3/174: rename d4/dd/d1a/c1c to d4/dd/d27/d2e/c30 0 2022-01-31T19:44:19.035 INFO:tasks.workunit.client.1.smithi171.stdout:3/175: creat d4/d21/f31 x:0 0 0 2022-01-31T19:44:19.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/176: chown d4/dd/f19 44388738 1 2022-01-31T19:44:19.036 INFO:tasks.workunit.client.1.smithi171.stdout:2/185: dread d7/dd/f15 [4194304,4194304] 0 2022-01-31T19:44:19.036 INFO:tasks.workunit.client.1.smithi171.stdout:2/186: fsync d7/dd/f15 0 2022-01-31T19:44:19.037 INFO:tasks.workunit.client.1.smithi171.stdout:2/187: fsync d7/f25 0 2022-01-31T19:44:19.037 INFO:tasks.workunit.client.1.smithi171.stdout:0/186: symlink d9/db/dc/l44 0 2022-01-31T19:44:19.037 INFO:tasks.workunit.client.1.smithi171.stdout:0/187: truncate d9/db/f14 159408 0 2022-01-31T19:44:19.037 INFO:tasks.workunit.client.1.smithi171.stdout:0/188: dread - d9/db/d22/d3a/f3e zero size 2022-01-31T19:44:19.038 INFO:tasks.workunit.client.1.smithi171.stdout:2/188: mknod d7/dd/d27/c38 0 2022-01-31T19:44:19.038 INFO:tasks.workunit.client.1.smithi171.stdout:0/189: unlink d9/db/f31 0 2022-01-31T19:44:19.038 INFO:tasks.workunit.client.1.smithi171.stdout:0/190: truncate d9/db/f21 2496306 0 2022-01-31T19:44:19.038 INFO:tasks.workunit.client.1.smithi171.stdout:2/189: dread d7/dd/f15 [4194304,4194304] 0 2022-01-31T19:44:19.039 INFO:tasks.workunit.client.1.smithi171.stdout:0/191: symlink d9/db/dc/l45 0 2022-01-31T19:44:19.039 INFO:tasks.workunit.client.1.smithi171.stdout:2/190: dwrite d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:19.040 INFO:tasks.workunit.client.0.smithi167.stdout:2/679: dread dd/fbf [0,4194304] 0 2022-01-31T19:44:19.041 INFO:tasks.workunit.client.0.smithi167.stdout:2/680: mkdir dd/df/d22/d96/d9e/dda 0 2022-01-31T19:44:19.041 INFO:tasks.workunit.client.0.smithi167.stdout:9/790: dwrite dd/d19/d24/d56/d5b/da5/fed [0,4194304] 0 2022-01-31T19:44:19.041 INFO:tasks.workunit.client.0.smithi167.stdout:9/791: dread dd/d17/f1f [0,4194304] 0 2022-01-31T19:44:19.041 INFO:tasks.workunit.client.0.smithi167.stdout:1/954: dwrite d0/d10/d20/d7c/fb8 [4194304,4194304] 0 2022-01-31T19:44:19.042 INFO:tasks.workunit.client.0.smithi167.stdout:1/955: write d0/d11/d102/ddb/f137 [626773,80260] 0 2022-01-31T19:44:19.042 INFO:tasks.workunit.client.0.smithi167.stdout:2/681: dread dd/df/d1b/d3d/d4a/d4b/f93 [0,4194304] 0 2022-01-31T19:44:19.042 INFO:tasks.workunit.client.0.smithi167.stdout:8/832: dwrite d6/db/d28/d61/f89 [0,4194304] 0 2022-01-31T19:44:19.042 INFO:tasks.workunit.client.0.smithi167.stdout:2/682: write dd/df/d1b/d2f/d36/fb9 [438685,107697] 0 2022-01-31T19:44:19.042 INFO:tasks.workunit.client.0.smithi167.stdout:1/956: symlink d0/d100/l144 0 2022-01-31T19:44:19.043 INFO:tasks.workunit.client.0.smithi167.stdout:3/924: sync 2022-01-31T19:44:19.043 INFO:tasks.workunit.client.0.smithi167.stdout:4/947: sync 2022-01-31T19:44:19.043 INFO:tasks.workunit.client.0.smithi167.stdout:4/948: creat d0/d5d/f139 x:0 0 0 2022-01-31T19:44:19.043 INFO:tasks.workunit.client.0.smithi167.stdout:4/949: truncate d0/d6/d7/d39/d4e/ffb 364973 0 2022-01-31T19:44:19.044 INFO:tasks.workunit.client.0.smithi167.stdout:4/950: truncate d0/d6/d7/d26/d40/d4c/f4f 584159 0 2022-01-31T19:44:19.044 INFO:tasks.workunit.client.0.smithi167.stdout:4/951: rmdir d0/d6/d7 39 2022-01-31T19:44:19.044 INFO:tasks.workunit.client.0.smithi167.stdout:4/952: readlink d0/d6/d7/d89/db1/l103 0 2022-01-31T19:44:19.044 INFO:tasks.workunit.client.0.smithi167.stdout:4/953: write d0/d5d/d12f/ddf/f29 [972216,51237] 0 2022-01-31T19:44:19.044 INFO:tasks.workunit.client.0.smithi167.stdout:8/833: dread d6/d1e/fac [0,4194304] 0 2022-01-31T19:44:19.045 INFO:tasks.workunit.client.0.smithi167.stdout:8/834: unlink d6/db/d6a/d8d/d9e/d60/la8 0 2022-01-31T19:44:19.045 INFO:tasks.workunit.client.0.smithi167.stdout:8/835: mkdir d6/db/d36/d10f 0 2022-01-31T19:44:19.045 INFO:tasks.workunit.client.0.smithi167.stdout:8/836: readlink d6/db/d36/lfc 0 2022-01-31T19:44:19.045 INFO:tasks.workunit.client.0.smithi167.stdout:9/792: dwrite dd/d19/d24/f45 [0,4194304] 0 2022-01-31T19:44:19.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/793: stat dd/d17/dc4/lfc 0 2022-01-31T19:44:19.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/794: truncate dd/d19/d24/d50/fde 661718 0 2022-01-31T19:44:19.046 INFO:tasks.workunit.client.0.smithi167.stdout:8/837: mknod d6/db/d28/d61/dd5/df0/c110 0 2022-01-31T19:44:19.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/795: creat dd/d6b/dc1/f110 x:0 0 0 2022-01-31T19:44:19.046 INFO:tasks.workunit.client.0.smithi167.stdout:9/796: creat dd/d19/d24/f111 x:0 0 0 2022-01-31T19:44:19.047 INFO:tasks.workunit.client.0.smithi167.stdout:4/954: dwrite d0/d6/d7/d39/f12c [0,4194304] 0 2022-01-31T19:44:19.047 INFO:tasks.workunit.client.0.smithi167.stdout:4/955: chown d0/l81 0 1 2022-01-31T19:44:19.047 INFO:tasks.workunit.client.0.smithi167.stdout:2/683: dwrite f8 [4194304,4194304] 0 2022-01-31T19:44:19.047 INFO:tasks.workunit.client.0.smithi167.stdout:2/684: chown dd/df/d22/f77 2355 1 2022-01-31T19:44:19.048 INFO:tasks.workunit.client.0.smithi167.stdout:2/685: stat dd/df/d22/l90 0 2022-01-31T19:44:19.048 INFO:tasks.workunit.client.0.smithi167.stdout:1/957: dwrite d0/d10/f19 [0,4194304] 0 2022-01-31T19:44:19.048 INFO:tasks.workunit.client.0.smithi167.stdout:9/797: dwrite dd/d19/d24/d56/dc9/fd7 [0,4194304] 0 2022-01-31T19:44:19.048 INFO:tasks.workunit.client.0.smithi167.stdout:3/925: dwrite d9/d7f/f8c [0,4194304] 0 2022-01-31T19:44:19.048 INFO:tasks.workunit.client.0.smithi167.stdout:4/956: rename d0/d6/d7/d89/d11c/d130/f138 to d0/d5d/d12f/d5c/de2/f13a 0 2022-01-31T19:44:19.049 INFO:tasks.workunit.client.0.smithi167.stdout:2/686: mknod dd/d18/d63/dab/cdb 0 2022-01-31T19:44:19.049 INFO:tasks.workunit.client.0.smithi167.stdout:9/798: rmdir dd/d6b 39 2022-01-31T19:44:19.049 INFO:tasks.workunit.client.0.smithi167.stdout:3/926: creat d9/d29/d3b/d9b/da3/f116 x:0 0 0 2022-01-31T19:44:19.049 INFO:tasks.workunit.client.0.smithi167.stdout:3/927: write d9/d26/f8b [589103,49395] 0 2022-01-31T19:44:19.050 INFO:tasks.workunit.client.0.smithi167.stdout:2/687: unlink dd/d18/db7/fd8 0 2022-01-31T19:44:19.050 INFO:tasks.workunit.client.0.smithi167.stdout:2/688: creat dd/df/d1b/d3d/d4a/da7/fdc x:0 0 0 2022-01-31T19:44:19.050 INFO:tasks.workunit.client.0.smithi167.stdout:9/799: dread dd/d19/d24/d56/fc7 [0,4194304] 0 2022-01-31T19:44:19.050 INFO:tasks.workunit.client.0.smithi167.stdout:9/800: dread dd/d6b/dc1/fd0 [0,4194304] 0 2022-01-31T19:44:19.050 INFO:tasks.workunit.client.0.smithi167.stdout:9/801: getdents dd/d19/d24/d50/dd1 0 2022-01-31T19:44:19.051 INFO:tasks.workunit.client.0.smithi167.stdout:9/802: getdents dd/d19/d24/d56/dc9 0 2022-01-31T19:44:19.051 INFO:tasks.workunit.client.0.smithi167.stdout:9/803: unlink dd/d19/l41 0 2022-01-31T19:44:19.051 INFO:tasks.workunit.client.0.smithi167.stdout:9/804: truncate dd/f84 1606722 0 2022-01-31T19:44:19.051 INFO:tasks.workunit.client.0.smithi167.stdout:9/805: stat dd/d19/d24/d50/d5d/la1 0 2022-01-31T19:44:19.098 INFO:tasks.workunit.client.1.smithi171.stdout:3/177: dwrite d4/dd/f20 [0,4194304] 0 2022-01-31T19:44:19.099 INFO:tasks.workunit.client.1.smithi171.stdout:2/191: dread d7/f11 [0,4194304] 0 2022-01-31T19:44:19.100 INFO:tasks.workunit.client.0.smithi167.stdout:2/689: dread dd/df/d22/f8a [0,4194304] 0 2022-01-31T19:44:19.101 INFO:tasks.workunit.client.1.smithi171.stdout:3/178: symlink d4/dd/d27/d2e/l32 0 2022-01-31T19:44:19.101 INFO:tasks.workunit.client.1.smithi171.stdout:3/179: stat d4/dd/f1e 0 2022-01-31T19:44:19.101 INFO:tasks.workunit.client.1.smithi171.stdout:3/180: chown d4/c6 5373452 1 2022-01-31T19:44:19.102 INFO:tasks.workunit.client.1.smithi171.stdout:3/181: readlink d4/l2c 0 2022-01-31T19:44:19.102 INFO:tasks.workunit.client.1.smithi171.stdout:2/192: creat d7/dd/f39 x:0 0 0 2022-01-31T19:44:19.103 INFO:tasks.workunit.client.1.smithi171.stdout:2/193: creat d7/f3a x:0 0 0 2022-01-31T19:44:19.103 INFO:tasks.workunit.client.1.smithi171.stdout:3/182: symlink d4/dd/l33 0 2022-01-31T19:44:19.104 INFO:tasks.workunit.client.0.smithi167.stdout:2/690: symlink dd/d18/d63/dab/ldd 0 2022-01-31T19:44:19.105 INFO:tasks.workunit.client.1.smithi171.stdout:2/194: rename d7/d9/l10 to d7/d26/l3b 0 2022-01-31T19:44:19.105 INFO:tasks.workunit.client.1.smithi171.stdout:2/195: dread - d7/d26/f31 zero size 2022-01-31T19:44:19.106 INFO:tasks.workunit.client.1.smithi171.stdout:3/183: mknod d4/d21/c34 0 2022-01-31T19:44:19.106 INFO:tasks.workunit.client.1.smithi171.stdout:2/196: rmdir d7/d26 39 2022-01-31T19:44:19.107 INFO:tasks.workunit.client.1.smithi171.stdout:2/197: chown d7/d9 11 1 2022-01-31T19:44:19.107 INFO:tasks.workunit.client.1.smithi171.stdout:2/198: read d7/dd/f1b [312844,109289] 0 2022-01-31T19:44:19.108 INFO:tasks.workunit.client.1.smithi171.stdout:2/199: write d7/fa [5937863,73368] 0 2022-01-31T19:44:19.108 INFO:tasks.workunit.client.1.smithi171.stdout:3/184: dread d4/dd/f1e [0,4194304] 0 2022-01-31T19:44:19.109 INFO:tasks.workunit.client.0.smithi167.stdout:3/928: dread d9/f27 [0,4194304] 0 2022-01-31T19:44:19.109 INFO:tasks.workunit.client.0.smithi167.stdout:3/929: creat d9/d29/d3b/d41/d6b/f117 x:0 0 0 2022-01-31T19:44:19.109 INFO:tasks.workunit.client.0.smithi167.stdout:9/806: dwrite dd/d19/d59/fad [0,4194304] 0 2022-01-31T19:44:19.113 INFO:tasks.workunit.client.0.smithi167.stdout:9/807: symlink dd/d19/d24/d56/d5b/l112 0 2022-01-31T19:44:19.114 INFO:tasks.workunit.client.0.smithi167.stdout:9/808: dread - dd/d19/d24/d56/dc9/f10c zero size 2022-01-31T19:44:19.115 INFO:tasks.workunit.client.1.smithi171.stdout:2/200: dread d7/dd/f15 [0,4194304] 0 2022-01-31T19:44:19.116 INFO:tasks.workunit.client.0.smithi167.stdout:9/809: getdents dd/d17/d7f 0 2022-01-31T19:44:19.117 INFO:tasks.workunit.client.0.smithi167.stdout:9/810: link dd/d19/d24/d56/cac dd/d19/d24/d56/d5b/da5/c113 0 2022-01-31T19:44:19.120 INFO:tasks.workunit.client.0.smithi167.stdout:9/811: mknod dd/d19/d24/c114 0 2022-01-31T19:44:19.120 INFO:tasks.workunit.client.0.smithi167.stdout:9/812: write dd/d19/d24/d56/dc9/f10c [140472,127255] 0 2022-01-31T19:44:19.121 INFO:tasks.workunit.client.0.smithi167.stdout:9/813: read - dd/d19/d24/f111 zero size 2022-01-31T19:44:19.121 INFO:tasks.workunit.client.0.smithi167.stdout:9/814: readlink dd/d19/d24/d50/dd1/lab 0 2022-01-31T19:44:19.121 INFO:tasks.workunit.client.0.smithi167.stdout:9/815: truncate dd/d35/fe5 166944 0 2022-01-31T19:44:19.122 INFO:tasks.workunit.client.0.smithi167.stdout:3/930: dread d9/d29/d60/f49 [0,4194304] 0 2022-01-31T19:44:19.122 INFO:tasks.workunit.client.0.smithi167.stdout:3/931: chown d9/d26/f98 601 1 2022-01-31T19:44:19.123 INFO:tasks.workunit.client.0.smithi167.stdout:3/932: link d9/d29/d3b/l3d d9/d7f/dbe/l118 0 2022-01-31T19:44:19.124 INFO:tasks.workunit.client.0.smithi167.stdout:3/933: mkdir d9/d29/d3b/d9b/da3/db0/d119 0 2022-01-31T19:44:19.125 INFO:tasks.workunit.client.0.smithi167.stdout:3/934: truncate d9/d29/d3b/d41/d6b/d100/f107 330066 0 2022-01-31T19:44:19.125 INFO:tasks.workunit.client.1.smithi171.stdout:2/201: dread d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:19.126 INFO:tasks.workunit.client.0.smithi167.stdout:3/935: mkdir d9/d29/d3b/d41/d6b/d11a 0 2022-01-31T19:44:19.126 INFO:tasks.workunit.client.0.smithi167.stdout:3/936: write d9/d26/f7a [110836,48015] 0 2022-01-31T19:44:19.127 INFO:tasks.workunit.client.0.smithi167.stdout:3/937: mkdir d9/d7f/dbe/d11b 0 2022-01-31T19:44:19.127 INFO:tasks.workunit.client.1.smithi171.stdout:2/202: creat d7/dd/f3c x:0 0 0 2022-01-31T19:44:19.128 INFO:tasks.workunit.client.0.smithi167.stdout:3/938: write d9/fd3 [2451083,20433] 0 2022-01-31T19:44:19.129 INFO:tasks.workunit.client.0.smithi167.stdout:3/939: rename d9/d29/d60/fac to d9/d7f/f11c 0 2022-01-31T19:44:19.134 INFO:tasks.workunit.client.0.smithi167.stdout:9/816: dread dd/d19/d24/d56/dc9/fd4 [0,4194304] 0 2022-01-31T19:44:19.136 INFO:tasks.workunit.client.0.smithi167.stdout:9/817: stat dd/d6b/f93 0 2022-01-31T19:44:19.150 INFO:tasks.workunit.client.1.smithi171.stdout:2/203: dwrite d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:19.150 INFO:tasks.workunit.client.1.smithi171.stdout:2/204: unlink d7/d9/c24 0 2022-01-31T19:44:19.152 INFO:tasks.workunit.client.1.smithi171.stdout:1/121: sync 2022-01-31T19:44:19.152 INFO:tasks.workunit.client.1.smithi171.stdout:1/122: creat dd/df/d17/f26 x:0 0 0 2022-01-31T19:44:19.152 INFO:tasks.workunit.client.1.smithi171.stdout:1/123: chown dd 742 1 2022-01-31T19:44:19.153 INFO:tasks.workunit.client.1.smithi171.stdout:1/124: creat dd/df/d19/f27 x:0 0 0 2022-01-31T19:44:19.153 INFO:tasks.workunit.client.1.smithi171.stdout:1/125: creat dd/df/d19/d1b/f28 x:0 0 0 2022-01-31T19:44:19.153 INFO:tasks.workunit.client.1.smithi171.stdout:2/205: mkdir d7/d26/d3d 0 2022-01-31T19:44:19.156 INFO:tasks.workunit.client.1.smithi171.stdout:1/126: write f7 [801522,4092] 0 2022-01-31T19:44:19.156 INFO:tasks.workunit.client.1.smithi171.stdout:1/127: write dd/df/f12 [366695,100657] 0 2022-01-31T19:44:19.157 INFO:tasks.workunit.client.1.smithi171.stdout:1/128: creat dd/d23/f29 x:0 0 0 2022-01-31T19:44:19.157 INFO:tasks.workunit.client.1.smithi171.stdout:1/129: fsync dd/f14 0 2022-01-31T19:44:19.158 INFO:tasks.workunit.client.1.smithi171.stdout:1/130: write dd/df/d19/f21 [11451,92007] 0 2022-01-31T19:44:19.158 INFO:tasks.workunit.client.0.smithi167.stdout:9/818: dwrite dd/d19/d24/d50/dd6/f103 [0,4194304] 0 2022-01-31T19:44:19.159 INFO:tasks.workunit.client.0.smithi167.stdout:9/819: read - dd/d19/d24/d56/dc9/f10d zero size 2022-01-31T19:44:19.161 INFO:tasks.workunit.client.1.smithi171.stdout:2/206: getdents d7/d9 0 2022-01-31T19:44:19.161 INFO:tasks.workunit.client.1.smithi171.stdout:2/207: stat d7/d26/c2a 0 2022-01-31T19:44:19.162 INFO:tasks.workunit.client.1.smithi171.stdout:2/208: write d7/d9/f12 [1465021,119892] 0 2022-01-31T19:44:19.164 INFO:tasks.workunit.client.1.smithi171.stdout:2/209: creat d7/d9/f3e x:0 0 0 2022-01-31T19:44:19.165 INFO:tasks.workunit.client.1.smithi171.stdout:2/210: chown d7/d9/c2d 453921 1 2022-01-31T19:44:19.165 INFO:tasks.workunit.client.0.smithi167.stdout:9/820: dread dd/d19/d59/fad [0,4194304] 0 2022-01-31T19:44:19.166 INFO:tasks.workunit.client.0.smithi167.stdout:9/821: write dd/d35/f33 [4529760,120872] 0 2022-01-31T19:44:19.166 INFO:tasks.workunit.client.1.smithi171.stdout:2/211: write d7/d9/f2c [594088,60370] 0 2022-01-31T19:44:19.167 INFO:tasks.workunit.client.1.smithi171.stdout:1/131: truncate f7 3801338 0 2022-01-31T19:44:19.173 INFO:tasks.workunit.client.1.smithi171.stdout:1/132: fdatasync f7 0 2022-01-31T19:44:19.173 INFO:tasks.workunit.client.1.smithi171.stdout:1/133: creat dd/df/f2a x:0 0 0 2022-01-31T19:44:19.175 INFO:tasks.workunit.client.1.smithi171.stdout:1/134: symlink dd/df/d19/d1b/l2b 0 2022-01-31T19:44:19.175 INFO:tasks.workunit.client.1.smithi171.stdout:1/135: fsync dd/d23/f29 0 2022-01-31T19:44:19.176 INFO:tasks.workunit.client.1.smithi171.stdout:1/136: mknod dd/df/d17/c2c 0 2022-01-31T19:44:19.184 INFO:tasks.workunit.client.1.smithi171.stdout:2/212: dwrite d7/f13 [4194304,4194304] 0 2022-01-31T19:44:19.188 INFO:tasks.workunit.client.0.smithi167.stdout:9/822: dwrite dd/d35/f8d [0,4194304] 0 2022-01-31T19:44:19.189 INFO:tasks.workunit.client.1.smithi171.stdout:1/137: dread dd/df/d19/f21 [0,4194304] 0 2022-01-31T19:44:19.191 INFO:tasks.workunit.client.1.smithi171.stdout:1/138: unlink dd/df/d19/c1c 0 2022-01-31T19:44:19.208 INFO:tasks.workunit.client.1.smithi171.stdout:2/213: dwrite d7/d26/f31 [0,4194304] 0 2022-01-31T19:44:19.208 INFO:tasks.workunit.client.1.smithi171.stdout:2/214: rename d7 to d7/d3f 22 2022-01-31T19:44:19.208 INFO:tasks.workunit.client.1.smithi171.stdout:2/215: fdatasync d7/fa 0 2022-01-31T19:44:19.210 INFO:tasks.workunit.client.1.smithi171.stdout:2/216: unlink d7/d9/f3e 0 2022-01-31T19:44:19.211 INFO:tasks.workunit.client.1.smithi171.stdout:2/217: fdatasync d7/d9/fe 0 2022-01-31T19:44:19.212 INFO:tasks.workunit.client.1.smithi171.stdout:2/218: readlink d7/dd/l17 0 2022-01-31T19:44:19.212 INFO:tasks.workunit.client.1.smithi171.stdout:2/219: stat d7/dd/d27/l28 0 2022-01-31T19:44:19.212 INFO:tasks.workunit.client.1.smithi171.stdout:2/220: dread - d7/dd/f39 zero size 2022-01-31T19:44:19.213 INFO:tasks.workunit.client.1.smithi171.stdout:2/221: readlink d7/dd/d27/l28 0 2022-01-31T19:44:19.213 INFO:tasks.workunit.client.1.smithi171.stdout:2/222: mkdir d7/d26/d3d/d40 0 2022-01-31T19:44:19.213 INFO:tasks.workunit.client.1.smithi171.stdout:2/223: truncate d7/dd/f39 124079 0 2022-01-31T19:44:19.214 INFO:tasks.workunit.client.0.smithi167.stdout:9/823: dwrite dd/d19/d24/d50/dd6/f103 [4194304,4194304] 0 2022-01-31T19:44:19.215 INFO:tasks.workunit.client.1.smithi171.stdout:2/224: mknod d7/dd/c41 0 2022-01-31T19:44:19.215 INFO:tasks.workunit.client.1.smithi171.stdout:1/139: dwrite f9 [0,4194304] 0 2022-01-31T19:44:19.215 INFO:tasks.workunit.client.1.smithi171.stdout:1/140: readlink dd/df/l15 0 2022-01-31T19:44:19.215 INFO:tasks.workunit.client.1.smithi171.stdout:1/141: creat dd/df/d19/f2d x:0 0 0 2022-01-31T19:44:19.216 INFO:tasks.workunit.client.0.smithi167.stdout:9/824: truncate dd/d35/f65 505182 0 2022-01-31T19:44:19.216 INFO:tasks.workunit.client.0.smithi167.stdout:9/825: write dd/d19/d24/d50/d5d/fbc [185112,2051] 0 2022-01-31T19:44:19.217 INFO:tasks.workunit.client.0.smithi167.stdout:9/826: chown dd/d19/d59/df9/c108 982934859 1 2022-01-31T19:44:19.219 INFO:tasks.workunit.client.1.smithi171.stdout:1/142: mkdir dd/d23/d2e 0 2022-01-31T19:44:19.219 INFO:tasks.workunit.client.1.smithi171.stdout:1/143: chown dd/df/d19/d24 269 1 2022-01-31T19:44:19.219 INFO:tasks.workunit.client.1.smithi171.stdout:1/144: unlink dd/df/d19/f2d 0 2022-01-31T19:44:19.221 INFO:tasks.workunit.client.1.smithi171.stdout:1/145: creat dd/d23/d2e/f2f x:0 0 0 2022-01-31T19:44:19.222 INFO:tasks.workunit.client.1.smithi171.stdout:1/146: symlink dd/df/d19/l30 0 2022-01-31T19:44:19.222 INFO:tasks.workunit.client.1.smithi171.stdout:1/147: chown dd/df/d17/f26 19 1 2022-01-31T19:44:19.223 INFO:tasks.workunit.client.1.smithi171.stdout:1/148: rename dd/df/d19/d1b/f22 to dd/df/d17/f31 0 2022-01-31T19:44:19.229 INFO:tasks.workunit.client.1.smithi171.stdout:2/225: dwrite d7/dd/f39 [0,4194304] 0 2022-01-31T19:44:19.230 INFO:tasks.workunit.client.0.smithi167.stdout:9/827: dread dd/d35/f37 [4194304,4194304] 0 2022-01-31T19:44:19.232 INFO:tasks.workunit.client.1.smithi171.stdout:1/149: dread dd/df/f12 [0,4194304] 0 2022-01-31T19:44:19.233 INFO:tasks.workunit.client.1.smithi171.stdout:1/150: dread dd/df/f12 [0,4194304] 0 2022-01-31T19:44:19.233 INFO:tasks.workunit.client.0.smithi167.stdout:9/828: dread dd/d19/d59/fad [0,4194304] 0 2022-01-31T19:44:19.238 INFO:tasks.workunit.client.1.smithi171.stdout:2/226: truncate d7/fa 2761691 0 2022-01-31T19:44:19.238 INFO:tasks.workunit.client.1.smithi171.stdout:2/227: truncate d7/d9/f23 168129 0 2022-01-31T19:44:19.248 INFO:tasks.workunit.client.1.smithi171.stdout:1/151: dread f3 [0,4194304] 0 2022-01-31T19:44:19.250 INFO:tasks.workunit.client.1.smithi171.stdout:2/228: rename d7/d9/f23 to d7/d26/d3d/f42 0 2022-01-31T19:44:19.250 INFO:tasks.workunit.client.1.smithi171.stdout:2/229: truncate d7/f22 3365673 0 2022-01-31T19:44:19.252 INFO:tasks.workunit.client.1.smithi171.stdout:2/230: symlink d7/dd/l43 0 2022-01-31T19:44:19.254 INFO:tasks.workunit.client.0.smithi167.stdout:9/829: dwrite dd/d35/f33 [4194304,4194304] 0 2022-01-31T19:44:19.255 INFO:tasks.workunit.client.1.smithi171.stdout:1/152: write f9 [2676944,87564] 0 2022-01-31T19:44:19.257 INFO:tasks.workunit.client.0.smithi167.stdout:9/830: symlink dd/d19/d24/l115 0 2022-01-31T19:44:19.259 INFO:tasks.workunit.client.1.smithi171.stdout:1/153: unlink dd/df/d17/f31 0 2022-01-31T19:44:19.259 INFO:tasks.workunit.client.1.smithi171.stdout:1/154: write dd/df/d19/f21 [98998,22813] 0 2022-01-31T19:44:19.259 INFO:tasks.workunit.client.1.smithi171.stdout:1/155: dread - dd/df/d17/f26 zero size 2022-01-31T19:44:19.259 INFO:tasks.workunit.client.1.smithi171.stdout:1/156: fsync dd/df/d19/d1b/f28 0 2022-01-31T19:44:19.259 INFO:tasks.workunit.client.1.smithi171.stdout:1/157: chown dd/df/d19/d1b 77 1 2022-01-31T19:44:19.260 INFO:tasks.workunit.client.1.smithi171.stdout:1/158: write dd/d23/d2e/f2f [464983,47075] 0 2022-01-31T19:44:19.263 INFO:tasks.workunit.client.1.smithi171.stdout:1/159: dread dd/df/d17/f20 [0,4194304] 0 2022-01-31T19:44:19.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/231: dread d7/d26/f31 [0,4194304] 0 2022-01-31T19:44:19.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/232: readlink d7/lc 0 2022-01-31T19:44:19.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/233: stat d7/dd/d27 0 2022-01-31T19:44:19.267 INFO:tasks.workunit.client.1.smithi171.stdout:2/234: fdatasync d7/d9/d1a/f32 0 2022-01-31T19:44:19.268 INFO:tasks.workunit.client.1.smithi171.stdout:2/235: link d7/dd/f15 d7/d9/f44 0 2022-01-31T19:44:19.274 INFO:tasks.workunit.client.1.smithi171.stdout:2/236: write d7/d9/f44 [8295289,23077] 0 2022-01-31T19:44:19.278 INFO:tasks.workunit.client.1.smithi171.stdout:2/237: rename d7/dd/d27 to d7/d9/d1a/d45 0 2022-01-31T19:44:19.278 INFO:tasks.workunit.client.1.smithi171.stdout:2/238: fdatasync d7/d9/f2c 0 2022-01-31T19:44:19.278 INFO:tasks.workunit.client.1.smithi171.stdout:2/239: creat d7/d9/f46 x:0 0 0 2022-01-31T19:44:19.283 INFO:tasks.workunit.client.0.smithi167.stdout:9/831: dwrite dd/d17/d7f/fbb [0,4194304] 0 2022-01-31T19:44:19.283 INFO:tasks.workunit.client.0.smithi167.stdout:9/832: creat dd/d19/d59/f116 x:0 0 0 2022-01-31T19:44:19.285 INFO:tasks.workunit.client.0.smithi167.stdout:9/833: mkdir dd/d19/d24/d50/d5d/d117 0 2022-01-31T19:44:19.286 INFO:tasks.workunit.client.0.smithi167.stdout:9/834: write dd/d34/f92 [638550,123519] 0 2022-01-31T19:44:19.286 INFO:tasks.workunit.client.0.smithi167.stdout:9/835: write dd/d6b/f83 [4596697,21364] 0 2022-01-31T19:44:19.287 INFO:tasks.workunit.client.0.smithi167.stdout:9/836: rename dd/la0 to dd/d19/d24/d56/d5b/dcf/l118 0 2022-01-31T19:44:19.289 INFO:tasks.workunit.client.0.smithi167.stdout:9/837: rename dd/d17/d7f/def/db3/lf2 to dd/d19/d24/d50/d5d/l119 0 2022-01-31T19:44:19.290 INFO:tasks.workunit.client.1.smithi171.stdout:1/160: dwrite dd/d23/f29 [0,4194304] 0 2022-01-31T19:44:19.290 INFO:tasks.workunit.client.1.smithi171.stdout:1/161: dread - dd/df/d17/f26 zero size 2022-01-31T19:44:19.294 INFO:tasks.workunit.client.1.smithi171.stdout:6/184: sync 2022-01-31T19:44:19.294 INFO:tasks.workunit.client.1.smithi171.stdout:7/137: sync 2022-01-31T19:44:19.294 INFO:tasks.workunit.client.1.smithi171.stdout:6/185: stat dc/d18/f2c 0 2022-01-31T19:44:19.294 INFO:tasks.workunit.client.1.smithi171.stdout:7/138: chown d1/d24/f18 1996 1 2022-01-31T19:44:19.295 INFO:tasks.workunit.client.1.smithi171.stdout:7/139: readlink d1/d24/l17 0 2022-01-31T19:44:19.295 INFO:tasks.workunit.client.1.smithi171.stdout:6/186: symlink dc/l36 0 2022-01-31T19:44:19.296 INFO:tasks.workunit.client.1.smithi171.stdout:6/187: fsync dc/d19/d29/f34 0 2022-01-31T19:44:19.296 INFO:tasks.workunit.client.1.smithi171.stdout:6/188: chown c3 172 1 2022-01-31T19:44:19.296 INFO:tasks.workunit.client.1.smithi171.stdout:6/189: chown dc/f13 13546846 1 2022-01-31T19:44:19.297 INFO:tasks.workunit.client.1.smithi171.stdout:6/190: mkdir dc/d19/d29/d27/d37 0 2022-01-31T19:44:19.297 INFO:tasks.workunit.client.1.smithi171.stdout:6/191: read dc/f14 [385026,33689] 0 2022-01-31T19:44:19.297 INFO:tasks.workunit.client.1.smithi171.stdout:6/192: chown dc/d18/f2a 53147716 1 2022-01-31T19:44:19.298 INFO:tasks.workunit.client.1.smithi171.stdout:6/193: chown dc/d19 4 1 2022-01-31T19:44:19.299 INFO:tasks.workunit.client.1.smithi171.stdout:6/194: dread dc/d19/f1a [0,4194304] 0 2022-01-31T19:44:19.300 INFO:tasks.workunit.client.1.smithi171.stdout:2/240: dwrite d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:19.301 INFO:tasks.workunit.client.0.smithi167.stdout:9/838: dread dd/d17/d80/f54 [0,4194304] 0 2022-01-31T19:44:19.302 INFO:tasks.workunit.client.0.smithi167.stdout:9/839: mknod dd/d19/d24/d50/d5d/dfa/c11a 0 2022-01-31T19:44:19.303 INFO:tasks.workunit.client.0.smithi167.stdout:9/840: mkdir dd/d19/d24/d50/dd6/d11b 0 2022-01-31T19:44:19.304 INFO:tasks.workunit.client.1.smithi171.stdout:6/195: dread f2 [0,4194304] 0 2022-01-31T19:44:19.305 INFO:tasks.workunit.client.0.smithi167.stdout:9/841: symlink dd/d19/d24/d50/d5d/dfa/l11c 0 2022-01-31T19:44:19.306 INFO:tasks.workunit.client.1.smithi171.stdout:6/196: truncate dc/d18/f2a 2253393 0 2022-01-31T19:44:19.306 INFO:tasks.workunit.client.0.smithi167.stdout:9/842: write dd/d17/f36 [1129260,31118] 0 2022-01-31T19:44:19.309 INFO:tasks.workunit.client.0.smithi167.stdout:9/843: link dd/d19/d24/fca dd/d19/d24/d50/dd6/f11d 0 2022-01-31T19:44:19.311 INFO:tasks.workunit.client.0.smithi167.stdout:9/844: creat dd/d17/dc4/de2/f11e x:0 0 0 2022-01-31T19:44:19.313 INFO:tasks.workunit.client.0.smithi167.stdout:9/845: symlink dd/d19/d24/d50/d5d/dfa/l11f 0 2022-01-31T19:44:19.314 INFO:tasks.workunit.client.0.smithi167.stdout:9/846: creat dd/d17/d7f/dfb/f120 x:0 0 0 2022-01-31T19:44:19.314 INFO:tasks.workunit.client.0.smithi167.stdout:9/847: chown dd/d19/d24/fca 507537480 1 2022-01-31T19:44:19.318 INFO:tasks.workunit.client.0.smithi167.stdout:0/640: sync 2022-01-31T19:44:19.319 INFO:tasks.workunit.client.0.smithi167.stdout:6/722: sync 2022-01-31T19:44:19.319 INFO:tasks.workunit.client.0.smithi167.stdout:6/723: dread - d8/d20/fa6 zero size 2022-01-31T19:44:19.320 INFO:tasks.workunit.client.0.smithi167.stdout:0/641: creat d17/d3e/d7e/dc7/fd7 x:0 0 0 2022-01-31T19:44:19.321 INFO:tasks.workunit.client.0.smithi167.stdout:6/724: readlink d8/d33/d51/l69 0 2022-01-31T19:44:19.321 INFO:tasks.workunit.client.1.smithi171.stdout:1/162: dwrite dd/d23/f29 [0,4194304] 0 2022-01-31T19:44:19.322 INFO:tasks.workunit.client.1.smithi171.stdout:1/163: fsync dd/df/d19/d1b/f28 0 2022-01-31T19:44:19.322 INFO:tasks.workunit.client.1.smithi171.stdout:1/164: fdatasync dd/df/d17/f1d 0 2022-01-31T19:44:19.322 INFO:tasks.workunit.client.1.smithi171.stdout:1/165: fsync dd/df/d19/d1b/f28 0 2022-01-31T19:44:19.322 INFO:tasks.workunit.client.1.smithi171.stdout:1/166: creat dd/df/d19/f32 x:0 0 0 2022-01-31T19:44:19.323 INFO:tasks.workunit.client.1.smithi171.stdout:1/167: chown dd/d23/f29 1569 1 2022-01-31T19:44:19.323 INFO:tasks.workunit.client.1.smithi171.stdout:1/168: mknod dd/df/c33 0 2022-01-31T19:44:19.324 INFO:tasks.workunit.client.0.smithi167.stdout:0/642: write f14 [1852947,103268] 0 2022-01-31T19:44:19.324 INFO:tasks.workunit.client.0.smithi167.stdout:0/643: symlink d17/d19/d1c/d39/d7b/ld8 0 2022-01-31T19:44:19.324 INFO:tasks.workunit.client.0.smithi167.stdout:0/644: write d17/fad [765972,76997] 0 2022-01-31T19:44:19.325 INFO:tasks.workunit.client.0.smithi167.stdout:0/645: mkdir d17/d19/d1c/d39/d7b/dbd/dd9 0 2022-01-31T19:44:19.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/169: rename dd/df/d19/d24 to dd/d23/d34 0 2022-01-31T19:44:19.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/170: stat dd/df/d19/d1b 0 2022-01-31T19:44:19.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/171: truncate dd/df/d19/f27 771617 0 2022-01-31T19:44:19.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/172: write dd/df/d17/f26 [266768,85171] 0 2022-01-31T19:44:19.327 INFO:tasks.workunit.client.1.smithi171.stdout:1/173: chown dd/d23/c25 32034 1 2022-01-31T19:44:19.327 INFO:tasks.workunit.client.1.smithi171.stdout:2/241: dwrite d7/dd/f1b [0,4194304] 0 2022-01-31T19:44:19.327 INFO:tasks.workunit.client.1.smithi171.stdout:1/174: mkdir dd/df/d19/d1b/d1e/d35 0 2022-01-31T19:44:19.327 INFO:tasks.workunit.client.1.smithi171.stdout:1/175: stat f7 0 2022-01-31T19:44:19.328 INFO:tasks.workunit.client.0.smithi167.stdout:0/646: dread d17/f31 [0,4194304] 0 2022-01-31T19:44:19.328 INFO:tasks.workunit.client.0.smithi167.stdout:0/647: creat d17/d2c/dcd/fda x:0 0 0 2022-01-31T19:44:19.329 INFO:tasks.workunit.client.0.smithi167.stdout:6/725: dread d8/d33/ff0 [0,4194304] 0 2022-01-31T19:44:19.334 INFO:tasks.workunit.client.0.smithi167.stdout:6/726: rename d8/d33/d51/d5d/d64/f88 to d8/d33/d51/d5d/d64/dab/f102 0 2022-01-31T19:44:19.334 INFO:tasks.workunit.client.0.smithi167.stdout:6/727: chown d8/d33/d51/d5d/d64/cf3 63 1 2022-01-31T19:44:19.335 INFO:tasks.workunit.client.0.smithi167.stdout:6/728: mkdir d8/d20/d103 0 2022-01-31T19:44:19.336 INFO:tasks.workunit.client.0.smithi167.stdout:6/729: mknod d8/d33/d51/d5d/d64/dab/db9/c104 0 2022-01-31T19:44:19.338 INFO:tasks.workunit.client.1.smithi171.stdout:5/141: sync 2022-01-31T19:44:19.339 INFO:tasks.workunit.client.1.smithi171.stdout:0/192: sync 2022-01-31T19:44:19.339 INFO:tasks.workunit.client.1.smithi171.stdout:0/193: dread - d9/fa zero size 2022-01-31T19:44:19.339 INFO:tasks.workunit.client.1.smithi171.stdout:0/194: truncate d9/d24/f3c 584013 0 2022-01-31T19:44:19.339 INFO:tasks.workunit.client.1.smithi171.stdout:8/113: sync 2022-01-31T19:44:19.340 INFO:tasks.workunit.client.1.smithi171.stdout:4/146: sync 2022-01-31T19:44:19.340 INFO:tasks.workunit.client.1.smithi171.stdout:4/147: stat d2/d3 0 2022-01-31T19:44:19.340 INFO:tasks.workunit.client.1.smithi171.stdout:9/136: sync 2022-01-31T19:44:19.341 INFO:tasks.workunit.client.1.smithi171.stdout:5/142: link d5/d6/l7 d5/d6/d15/l30 0 2022-01-31T19:44:19.341 INFO:tasks.workunit.client.1.smithi171.stdout:5/143: fsync d5/d6/d15/f1d 0 2022-01-31T19:44:19.341 INFO:tasks.workunit.client.1.smithi171.stdout:0/195: mknod d9/c46 0 2022-01-31T19:44:19.342 INFO:tasks.workunit.client.1.smithi171.stdout:8/114: symlink df/d2a/l2c 0 2022-01-31T19:44:19.342 INFO:tasks.workunit.client.1.smithi171.stdout:4/148: mknod d2/d3/d5/d11/d12/d15/c32 0 2022-01-31T19:44:19.343 INFO:tasks.workunit.client.1.smithi171.stdout:9/137: creat dd/f2e x:0 0 0 2022-01-31T19:44:19.344 INFO:tasks.workunit.client.1.smithi171.stdout:8/115: rmdir df/d18 39 2022-01-31T19:44:19.345 INFO:tasks.workunit.client.1.smithi171.stdout:4/149: mknod d2/d3/d5/dc/c33 0 2022-01-31T19:44:19.346 INFO:tasks.workunit.client.1.smithi171.stdout:0/196: dread d9/db/f14 [0,4194304] 0 2022-01-31T19:44:19.348 INFO:tasks.workunit.client.1.smithi171.stdout:8/116: mknod df/d18/c2d 0 2022-01-31T19:44:19.348 INFO:tasks.workunit.client.1.smithi171.stdout:5/144: rmdir d5/d6/d15/d23 39 2022-01-31T19:44:19.349 INFO:tasks.workunit.client.1.smithi171.stdout:8/117: symlink df/d18/l2e 0 2022-01-31T19:44:19.349 INFO:tasks.workunit.client.1.smithi171.stdout:8/118: truncate df/d18/d19/f22 487063 0 2022-01-31T19:44:19.353 INFO:tasks.workunit.client.1.smithi171.stdout:5/145: link d5/d6/d15/l2c d5/d6/d15/l31 0 2022-01-31T19:44:19.354 INFO:tasks.workunit.client.1.smithi171.stdout:0/197: dread f4 [0,4194304] 0 2022-01-31T19:44:19.355 INFO:tasks.workunit.client.1.smithi171.stdout:9/138: write f4 [9867498,116808] 0 2022-01-31T19:44:19.356 INFO:tasks.workunit.client.1.smithi171.stdout:9/139: chown dd/c10 0 1 2022-01-31T19:44:19.357 INFO:tasks.workunit.client.0.smithi167.stdout:6/730: dread d8/d33/d35/f8b [0,4194304] 0 2022-01-31T19:44:19.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:19 smithi171 conmon[35325]: cluster 2022-01-31T19:44:18.433725+0000 mgr.smithi171.asyxnc (mgr.24387) 21 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 1.7 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 112 MiB/s rd, 230 MiB/s wr, 331 op/s 2022-01-31T19:44:19.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:19 smithi171 conmon[35325]: audit 2022-01-31T19:44:19.091196+0000 mon.smithi167 (mon.0) 666 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:19.362 INFO:tasks.workunit.client.0.smithi167.stdout:6/731: write d8/f15 [2191877,2260] 0 2022-01-31T19:44:19.363 INFO:tasks.workunit.client.0.smithi167.stdout:6/732: chown d8/d33/d51/d7e/dc4 8884944 1 2022-01-31T19:44:19.363 INFO:tasks.workunit.client.0.smithi167.stdout:6/733: write d8/d33/d51/d5d/d86/fe7 [601550,47434] 0 2022-01-31T19:44:19.364 INFO:tasks.workunit.client.1.smithi171.stdout:1/176: dwrite dd/df/d19/f32 [0,4194304] 0 2022-01-31T19:44:19.364 INFO:tasks.workunit.client.0.smithi167.stdout:0/648: dwrite d17/d3e/f42 [0,4194304] 0 2022-01-31T19:44:19.365 INFO:tasks.workunit.client.0.smithi167.stdout:0/649: creat d17/d33/da4/dd0/fdb x:0 0 0 2022-01-31T19:44:19.367 INFO:tasks.workunit.client.0.smithi167.stdout:0/650: link d17/l7c d17/d19/d1c/d39/d7b/dbd/ldc 0 2022-01-31T19:44:19.367 INFO:tasks.workunit.client.0.smithi167.stdout:0/651: truncate f14 2702031 0 2022-01-31T19:44:19.368 INFO:tasks.workunit.client.0.smithi167.stdout:0/652: rmdir d17/d19/d1c/d39/d7b/dbd 39 2022-01-31T19:44:19.372 INFO:tasks.workunit.client.1.smithi171.stdout:8/119: dread f2 [4194304,4194304] 0 2022-01-31T19:44:19.375 INFO:tasks.workunit.client.0.smithi167.stdout:6/734: fdatasync d8/f15 0 2022-01-31T19:44:19.376 INFO:tasks.workunit.client.0.smithi167.stdout:6/735: creat d8/d33/d51/d5d/d86/dbd/f105 x:0 0 0 2022-01-31T19:44:19.380 INFO:tasks.workunit.client.1.smithi171.stdout:5/146: dwrite d5/f8 [4194304,4194304] 0 2022-01-31T19:44:19.381 INFO:tasks.workunit.client.1.smithi171.stdout:5/147: write d5/d6/f21 [336426,71624] 0 2022-01-31T19:44:19.381 INFO:tasks.workunit.client.1.smithi171.stdout:5/148: readlink d5/d6/dd/l2a 0 2022-01-31T19:44:19.382 INFO:tasks.workunit.client.1.smithi171.stdout:4/150: dwrite d2/d3/d5/d11/f27 [0,4194304] 0 2022-01-31T19:44:19.383 INFO:tasks.workunit.client.1.smithi171.stdout:4/151: readlink d2/d3/d5/d11/d12/l21 0 2022-01-31T19:44:19.384 INFO:tasks.workunit.client.1.smithi171.stdout:5/149: link d5/d6/d15/l30 d5/d6/l32 0 2022-01-31T19:44:19.385 INFO:tasks.workunit.client.1.smithi171.stdout:2/242: dwrite d7/dd/f1b [0,4194304] 0 2022-01-31T19:44:19.386 INFO:tasks.workunit.client.1.smithi171.stdout:2/243: rename d7 to d7/d26/d47 22 2022-01-31T19:44:19.386 INFO:tasks.workunit.client.1.smithi171.stdout:9/140: dread dd/df/f17 [0,4194304] 0 2022-01-31T19:44:19.388 INFO:tasks.workunit.client.1.smithi171.stdout:8/120: rmdir df 39 2022-01-31T19:44:19.389 INFO:tasks.workunit.client.1.smithi171.stdout:8/121: fsync df/d18/d19/f1b 0 2022-01-31T19:44:19.390 INFO:tasks.workunit.client.1.smithi171.stdout:5/150: creat d5/d6/d15/f33 x:0 0 0 2022-01-31T19:44:19.391 INFO:tasks.workunit.client.1.smithi171.stdout:0/198: dread f6 [0,4194304] 0 2022-01-31T19:44:19.391 INFO:tasks.workunit.client.1.smithi171.stdout:0/199: creat d9/db/f47 x:0 0 0 2022-01-31T19:44:19.391 INFO:tasks.workunit.client.1.smithi171.stdout:0/200: stat d9/db/d22/d3a/f42 0 2022-01-31T19:44:19.392 INFO:tasks.workunit.client.1.smithi171.stdout:1/177: read fa [54671,110512] 0 2022-01-31T19:44:19.392 INFO:tasks.workunit.client.1.smithi171.stdout:9/141: creat dd/df/d15/f2f x:0 0 0 2022-01-31T19:44:19.393 INFO:tasks.workunit.client.1.smithi171.stdout:4/152: dread d2/d3/f1c [0,4194304] 0 2022-01-31T19:44:19.393 INFO:tasks.workunit.client.1.smithi171.stdout:4/153: dread - d2/d3/f28 zero size 2022-01-31T19:44:19.393 INFO:tasks.workunit.client.1.smithi171.stdout:8/122: mknod df/d2a/c2f 0 2022-01-31T19:44:19.393 INFO:tasks.workunit.client.1.smithi171.stdout:9/142: write dd/df/d15/f1b [537052,20767] 0 2022-01-31T19:44:19.394 INFO:tasks.workunit.client.1.smithi171.stdout:9/143: chown dd/df/f17 7669 1 2022-01-31T19:44:19.395 INFO:tasks.workunit.client.1.smithi171.stdout:5/151: link d5/d6/dd/f22 d5/d6/d15/d29/f34 0 2022-01-31T19:44:19.395 INFO:tasks.workunit.client.1.smithi171.stdout:5/152: write d5/d6/f1a [508428,86141] 0 2022-01-31T19:44:19.395 INFO:tasks.workunit.client.1.smithi171.stdout:5/153: creat d5/d6/dd/f35 x:0 0 0 2022-01-31T19:44:19.396 INFO:tasks.workunit.client.1.smithi171.stdout:2/244: dread d7/d9/f2c [0,4194304] 0 2022-01-31T19:44:19.397 INFO:tasks.workunit.client.1.smithi171.stdout:0/201: rename d9/fa to d9/db/d22/d3a/f48 0 2022-01-31T19:44:19.397 INFO:tasks.workunit.client.1.smithi171.stdout:0/202: creat d9/d24/f49 x:0 0 0 2022-01-31T19:44:19.399 INFO:tasks.workunit.client.1.smithi171.stdout:1/178: mkdir dd/df/d19/d36 0 2022-01-31T19:44:19.401 INFO:tasks.workunit.client.1.smithi171.stdout:4/154: symlink d2/d3/d5/l34 0 2022-01-31T19:44:19.402 INFO:tasks.workunit.client.1.smithi171.stdout:4/155: write d2/d3/d5/d11/d12/f30 [687951,44336] 0 2022-01-31T19:44:19.402 INFO:tasks.workunit.client.0.smithi167.stdout:0/653: dwrite f8 [0,4194304] 0 2022-01-31T19:44:19.403 INFO:tasks.workunit.client.1.smithi171.stdout:8/123: getdents df/d18/d19/d21 0 2022-01-31T19:44:19.403 INFO:tasks.workunit.client.1.smithi171.stdout:9/144: symlink dd/l30 0 2022-01-31T19:44:19.407 INFO:tasks.workunit.client.1.smithi171.stdout:1/179: rename dd/d23/c25 to dd/c37 0 2022-01-31T19:44:19.407 INFO:tasks.workunit.client.1.smithi171.stdout:1/180: creat dd/df/f38 x:0 0 0 2022-01-31T19:44:19.409 INFO:tasks.workunit.client.1.smithi171.stdout:4/156: mknod d2/d3/d5/d11/d12/d15/c35 0 2022-01-31T19:44:19.410 INFO:tasks.workunit.client.1.smithi171.stdout:9/145: write dd/f11 [2874089,55542] 0 2022-01-31T19:44:19.410 INFO:tasks.workunit.client.1.smithi171.stdout:8/124: mkdir df/d2a/d30 0 2022-01-31T19:44:19.410 INFO:tasks.workunit.client.1.smithi171.stdout:8/125: dread - f8 zero size 2022-01-31T19:44:19.411 INFO:tasks.workunit.client.1.smithi171.stdout:9/146: mknod dd/c31 0 2022-01-31T19:44:19.411 INFO:tasks.workunit.client.1.smithi171.stdout:0/203: dread d9/f16 [0,4194304] 0 2022-01-31T19:44:19.412 INFO:tasks.workunit.client.1.smithi171.stdout:8/126: rmdir df/d18/d19/d21 39 2022-01-31T19:44:19.412 INFO:tasks.workunit.client.1.smithi171.stdout:8/127: chown df/c14 5085658 1 2022-01-31T19:44:19.415 INFO:tasks.workunit.client.1.smithi171.stdout:0/204: rename d9/f2b to d9/db/dc/d3f/f4a 0 2022-01-31T19:44:19.418 INFO:tasks.workunit.client.1.smithi171.stdout:0/205: mknod d9/d24/c4b 0 2022-01-31T19:44:19.418 INFO:tasks.workunit.client.1.smithi171.stdout:0/206: readlink d9/db/lf 0 2022-01-31T19:44:19.419 INFO:tasks.workunit.client.1.smithi171.stdout:0/207: read d9/f16 [2328914,116110] 0 2022-01-31T19:44:19.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:19 smithi167 conmon[32206]: cluster 2022-01-31T19:44:18 2022-01-31T19:44:19.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:19 smithi167 conmon[32206]: .433725+0000 mgr.smithi171.asyxnc (mgr.24387) 21 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 1.7 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 112 MiB/s rd, 230 MiB/s wr, 331 op/s 2022-01-31T19:44:19.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:19 smithi167 conmon[32206]: audit 2022-01-31T19:44:19.091196+0000 mon.smithi167 (mon.0) 666 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:19.421 INFO:tasks.workunit.client.1.smithi171.stdout:0/208: rename d9/db/dc/d3f/d30 to d9/d24/d4c 0 2022-01-31T19:44:19.421 INFO:tasks.workunit.client.1.smithi171.stdout:0/209: creat d9/d24/f4d x:0 0 0 2022-01-31T19:44:19.423 INFO:tasks.workunit.client.1.smithi171.stdout:4/157: dread d2/d3/d5/d11/f27 [0,4194304] 0 2022-01-31T19:44:19.423 INFO:tasks.workunit.client.1.smithi171.stdout:4/158: chown d2/d3/d5/d11/d12/l20 7134054 1 2022-01-31T19:44:19.424 INFO:tasks.workunit.client.1.smithi171.stdout:2/245: dwrite d7/f22 [0,4194304] 0 2022-01-31T19:44:19.424 INFO:tasks.workunit.client.1.smithi171.stdout:4/159: write d2/d3/d5/dc/f26 [60992,44172] 0 2022-01-31T19:44:19.424 INFO:tasks.workunit.client.1.smithi171.stdout:4/160: fsync d2/d3/d5/dc/f19 0 2022-01-31T19:44:19.426 INFO:tasks.workunit.client.1.smithi171.stdout:8/128: dread f2 [0,4194304] 0 2022-01-31T19:44:19.428 INFO:tasks.workunit.client.1.smithi171.stdout:2/246: creat d7/d26/d3d/d40/f48 x:0 0 0 2022-01-31T19:44:19.432 INFO:tasks.workunit.client.1.smithi171.stdout:5/154: dwrite d5/d6/d15/f1f [0,4194304] 0 2022-01-31T19:44:19.433 INFO:tasks.workunit.client.1.smithi171.stdout:5/155: readlink d5/d6/dd/l2b 0 2022-01-31T19:44:19.434 INFO:tasks.workunit.client.1.smithi171.stdout:9/147: rmdir dd 39 2022-01-31T19:44:19.435 INFO:tasks.workunit.client.1.smithi171.stdout:4/161: mkdir d2/d3/d5/d11/d36 0 2022-01-31T19:44:19.436 INFO:tasks.workunit.client.1.smithi171.stdout:4/162: fsync d2/d3/d5/dc/f19 0 2022-01-31T19:44:19.436 INFO:tasks.workunit.client.1.smithi171.stdout:4/163: truncate d2/d3/f7 4233557 0 2022-01-31T19:44:19.436 INFO:tasks.workunit.client.1.smithi171.stdout:1/181: dwrite dd/df/d17/f20 [0,4194304] 0 2022-01-31T19:44:19.437 INFO:tasks.workunit.client.1.smithi171.stdout:8/129: creat df/d18/d19/d21/f31 x:0 0 0 2022-01-31T19:44:19.437 INFO:tasks.workunit.client.1.smithi171.stdout:8/130: creat df/d18/d19/d21/f32 x:0 0 0 2022-01-31T19:44:19.438 INFO:tasks.workunit.client.1.smithi171.stdout:8/131: fsync f2 0 2022-01-31T19:44:19.438 INFO:tasks.workunit.client.1.smithi171.stdout:8/132: write f0 [4822706,128348] 0 2022-01-31T19:44:19.438 INFO:tasks.workunit.client.1.smithi171.stdout:0/210: dread d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:19.439 INFO:tasks.workunit.client.1.smithi171.stdout:5/156: rename d5/d6/l32 to d5/d6/d15/d29/l36 0 2022-01-31T19:44:19.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/148: rmdir dd/df 39 2022-01-31T19:44:19.441 INFO:tasks.workunit.client.1.smithi171.stdout:9/149: chown dd/df/f25 129959 1 2022-01-31T19:44:19.441 INFO:tasks.workunit.client.1.smithi171.stdout:4/164: truncate d2/d3/d5/f1a 1341386 0 2022-01-31T19:44:19.442 INFO:tasks.workunit.client.1.smithi171.stdout:4/165: dread - d2/d3/d5/d11/d12/f2a zero size 2022-01-31T19:44:19.445 INFO:tasks.workunit.client.1.smithi171.stdout:1/182: mknod dd/df/d17/c39 0 2022-01-31T19:44:19.445 INFO:tasks.workunit.client.1.smithi171.stdout:1/183: chown dd/d23/d2e 1313 1 2022-01-31T19:44:19.445 INFO:tasks.workunit.client.1.smithi171.stdout:1/184: chown dd/df/d19/f21 1 1 2022-01-31T19:44:19.445 INFO:tasks.workunit.client.1.smithi171.stdout:1/185: write f3 [5144035,74000] 0 2022-01-31T19:44:19.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/186: creat dd/df/f3a x:0 0 0 2022-01-31T19:44:19.446 INFO:tasks.workunit.client.1.smithi171.stdout:8/133: mknod df/d2a/d30/c33 0 2022-01-31T19:44:19.449 INFO:tasks.workunit.client.1.smithi171.stdout:0/211: symlink d9/db/l4e 0 2022-01-31T19:44:19.449 INFO:tasks.workunit.client.1.smithi171.stdout:0/212: write d9/d24/d4c/f3d [1010860,67271] 0 2022-01-31T19:44:19.450 INFO:tasks.workunit.client.1.smithi171.stdout:5/157: unlink c4 0 2022-01-31T19:44:19.450 INFO:tasks.workunit.client.1.smithi171.stdout:9/150: dread dd/f11 [0,4194304] 0 2022-01-31T19:44:19.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/166: dread d2/d3/d5/dc/f19 [0,4194304] 0 2022-01-31T19:44:19.451 INFO:tasks.workunit.client.1.smithi171.stdout:9/151: write dd/df/f17 [724316,23952] 0 2022-01-31T19:44:19.453 INFO:tasks.workunit.client.1.smithi171.stdout:1/187: rename dd/df/d19/l30 to dd/df/d17/l3b 0 2022-01-31T19:44:19.453 INFO:tasks.workunit.client.1.smithi171.stdout:1/188: chown dd/d23/d2e 134 1 2022-01-31T19:44:19.454 INFO:tasks.workunit.client.1.smithi171.stdout:8/134: symlink df/d1a/l34 0 2022-01-31T19:44:19.454 INFO:tasks.workunit.client.1.smithi171.stdout:8/135: chown df/c11 0 1 2022-01-31T19:44:19.455 INFO:tasks.workunit.client.1.smithi171.stdout:0/213: symlink d9/db/dc/l4f 0 2022-01-31T19:44:19.456 INFO:tasks.workunit.client.1.smithi171.stdout:9/152: stat l3 0 2022-01-31T19:44:19.458 INFO:tasks.workunit.client.1.smithi171.stdout:1/189: symlink dd/d23/l3c 0 2022-01-31T19:44:19.460 INFO:tasks.workunit.client.1.smithi171.stdout:1/190: mknod dd/d23/c3d 0 2022-01-31T19:44:19.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/167: read d2/d3/f7 [630730,102457] 0 2022-01-31T19:44:19.462 INFO:tasks.workunit.client.1.smithi171.stdout:1/191: link dd/df/d19/f21 dd/df/d19/d1b/f3e 0 2022-01-31T19:44:19.464 INFO:tasks.workunit.client.1.smithi171.stdout:4/168: symlink d2/l37 0 2022-01-31T19:44:19.464 INFO:tasks.workunit.client.1.smithi171.stdout:4/169: chown d2/d3/d5/d11 524010 1 2022-01-31T19:44:19.464 INFO:tasks.workunit.client.1.smithi171.stdout:4/170: chown d2/d3/d5/l2b 9535 1 2022-01-31T19:44:19.467 INFO:tasks.workunit.client.1.smithi171.stdout:4/171: write d2/d3/d5/dc/f26 [697765,61779] 0 2022-01-31T19:44:19.467 INFO:tasks.workunit.client.1.smithi171.stdout:4/172: read - d2/d3/d5/d11/d12/f2a zero size 2022-01-31T19:44:19.467 INFO:tasks.workunit.client.1.smithi171.stdout:4/173: creat d2/d3/d5/f38 x:0 0 0 2022-01-31T19:44:19.468 INFO:tasks.workunit.client.1.smithi171.stdout:1/192: rename dd/df/d17/f1d to dd/d23/f3f 0 2022-01-31T19:44:19.480 INFO:tasks.workunit.client.1.smithi171.stdout:5/158: dwrite d5/d6/f14 [4194304,4194304] 0 2022-01-31T19:44:19.485 INFO:tasks.workunit.client.0.smithi167.stdout:8/838: sync 2022-01-31T19:44:19.485 INFO:tasks.workunit.client.0.smithi167.stdout:1/958: sync 2022-01-31T19:44:19.486 INFO:tasks.workunit.client.1.smithi171.stdout:0/214: dwrite d9/d24/d4c/f3d [0,4194304] 0 2022-01-31T19:44:19.487 INFO:tasks.workunit.client.1.smithi171.stdout:0/215: mknod d9/d24/d4c/d35/c50 0 2022-01-31T19:44:19.487 INFO:tasks.workunit.client.1.smithi171.stdout:0/216: write d9/db/d22/d3a/f3e [778374,107834] 0 2022-01-31T19:44:19.488 INFO:tasks.workunit.client.0.smithi167.stdout:8/839: symlink d6/db/d28/d61/dd5/dbf/l111 0 2022-01-31T19:44:19.488 INFO:tasks.workunit.client.0.smithi167.stdout:1/959: write d0/d10/d5a/d33/d4b/da5/da9/f122 [3481903,83964] 0 2022-01-31T19:44:19.489 INFO:tasks.workunit.client.0.smithi167.stdout:8/840: creat d6/db/d28/daf/dc4/f112 x:0 0 0 2022-01-31T19:44:19.490 INFO:tasks.workunit.client.1.smithi171.stdout:0/217: mknod d9/d24/c51 0 2022-01-31T19:44:19.490 INFO:tasks.workunit.client.0.smithi167.stdout:1/960: mkdir d0/d10/d5a/df2/d13f/d145 0 2022-01-31T19:44:19.492 INFO:tasks.workunit.client.1.smithi171.stdout:0/218: creat d9/db/dc/d3f/d25/f52 x:0 0 0 2022-01-31T19:44:19.492 INFO:tasks.workunit.client.0.smithi167.stdout:1/961: write d0/d10/d5a/d33/fa7 [583298,36695] 0 2022-01-31T19:44:19.493 INFO:tasks.workunit.client.1.smithi171.stdout:0/219: write d9/db/dc/d3f/f4a [2121806,84779] 0 2022-01-31T19:44:19.493 INFO:tasks.workunit.client.0.smithi167.stdout:8/841: creat d6/f113 x:0 0 0 2022-01-31T19:44:19.494 INFO:tasks.workunit.client.0.smithi167.stdout:8/842: stat d6/db/d28/d66/ff1 0 2022-01-31T19:44:19.494 INFO:tasks.workunit.client.0.smithi167.stdout:8/843: chown d6/d1e/d2d/cf4 9234480 1 2022-01-31T19:44:19.495 INFO:tasks.workunit.client.0.smithi167.stdout:8/844: write d6/db/d6a/d8d/d9e/d60/f79 [104455,67402] 0 2022-01-31T19:44:19.495 INFO:tasks.workunit.client.0.smithi167.stdout:1/962: creat d0/d10/d5a/d33/d4b/f146 x:0 0 0 2022-01-31T19:44:19.496 INFO:tasks.workunit.client.0.smithi167.stdout:1/963: chown d0/f25 1 1 2022-01-31T19:44:19.496 INFO:tasks.workunit.client.0.smithi167.stdout:8/845: creat d6/db/d28/d61/de3/f114 x:0 0 0 2022-01-31T19:44:19.497 INFO:tasks.workunit.client.0.smithi167.stdout:1/964: write d0/d10/d5a/d33/fbb [2997660,69772] 0 2022-01-31T19:44:19.497 INFO:tasks.workunit.client.1.smithi171.stdout:0/220: creat d9/f53 x:0 0 0 2022-01-31T19:44:19.498 INFO:tasks.workunit.client.1.smithi171.stdout:0/221: symlink d9/db/dc/d3f/d25/l54 0 2022-01-31T19:44:19.499 INFO:tasks.workunit.client.0.smithi167.stdout:1/965: mknod d0/d75/d7a/c147 0 2022-01-31T19:44:19.499 INFO:tasks.workunit.client.0.smithi167.stdout:1/966: chown d0/d10/d5a/d33/d4b/f68 22082888 1 2022-01-31T19:44:19.501 INFO:tasks.workunit.client.0.smithi167.stdout:1/967: mkdir d0/d10/d5a/d33/d97/d148 0 2022-01-31T19:44:19.508 INFO:tasks.workunit.client.1.smithi171.stdout:1/193: dwrite dd/df/d19/f27 [0,4194304] 0 2022-01-31T19:44:19.508 INFO:tasks.workunit.client.1.smithi171.stdout:1/194: creat dd/df/d17/f40 x:0 0 0 2022-01-31T19:44:19.508 INFO:tasks.workunit.client.1.smithi171.stdout:1/195: creat dd/d23/f41 x:0 0 0 2022-01-31T19:44:19.509 INFO:tasks.workunit.client.1.smithi171.stdout:1/196: chown c2 3 1 2022-01-31T19:44:19.510 INFO:tasks.workunit.client.1.smithi171.stdout:1/197: write dd/d23/d2e/f2f [242323,80022] 0 2022-01-31T19:44:19.511 INFO:tasks.workunit.client.1.smithi171.stdout:1/198: dread - dd/d23/f41 zero size 2022-01-31T19:44:19.511 INFO:tasks.workunit.client.1.smithi171.stdout:8/136: dwrite fb [0,4194304] 0 2022-01-31T19:44:19.514 INFO:tasks.workunit.client.0.smithi167.stdout:8/846: dread d6/d1e/d4c/dba/fd4 [0,4194304] 0 2022-01-31T19:44:19.514 INFO:tasks.workunit.client.0.smithi167.stdout:8/847: truncate d6/fab 1237851 0 2022-01-31T19:44:19.514 INFO:tasks.workunit.client.0.smithi167.stdout:8/848: stat d6/db/d28/d61/dd5/dbf 0 2022-01-31T19:44:19.514 INFO:tasks.workunit.client.0.smithi167.stdout:8/849: chown d6/db/d4e/fd0 109 1 2022-01-31T19:44:19.515 INFO:tasks.workunit.client.0.smithi167.stdout:1/968: dread d0/d10/d20/f37 [0,4194304] 0 2022-01-31T19:44:19.518 INFO:tasks.workunit.client.0.smithi167.stdout:1/969: unlink d0/d10/d5a/d33/d4b/da5/da9/f105 0 2022-01-31T19:44:19.519 INFO:tasks.workunit.client.0.smithi167.stdout:1/970: mknod d0/d75/d7a/c149 0 2022-01-31T19:44:19.519 INFO:tasks.workunit.client.0.smithi167.stdout:1/971: read - d0/d62/d7f/f13d zero size 2022-01-31T19:44:19.521 INFO:tasks.workunit.client.0.smithi167.stdout:1/972: truncate d0/d10/d5a/d33/d97/fd0 813638 0 2022-01-31T19:44:19.521 INFO:tasks.workunit.client.0.smithi167.stdout:1/973: creat d0/db3/f14a x:0 0 0 2022-01-31T19:44:19.521 INFO:tasks.workunit.client.0.smithi167.stdout:1/974: dread - d0/d11/d102/f141 zero size 2022-01-31T19:44:19.522 INFO:tasks.workunit.client.0.smithi167.stdout:1/975: readlink d0/d10/d5a/d33/d4b/d1f/le1 0 2022-01-31T19:44:19.524 INFO:tasks.workunit.client.0.smithi167.stdout:1/976: mkdir d0/d10/d5a/d33/d4b/da5/dde/d11c/d14b 0 2022-01-31T19:44:19.525 INFO:tasks.workunit.client.1.smithi171.stdout:8/137: rename f9 to df/d2a/d30/f35 0 2022-01-31T19:44:19.526 INFO:tasks.workunit.client.1.smithi171.stdout:8/138: mkdir df/d18/d19/d36 0 2022-01-31T19:44:19.526 INFO:tasks.workunit.client.0.smithi167.stdout:1/977: creat d0/d75/d7a/f14c x:0 0 0 2022-01-31T19:44:19.527 INFO:tasks.workunit.client.1.smithi171.stdout:9/153: dwrite dd/df/f20 [0,4194304] 0 2022-01-31T19:44:19.527 INFO:tasks.workunit.client.0.smithi167.stdout:8/850: dread d6/db/d36/f84 [0,4194304] 0 2022-01-31T19:44:19.528 INFO:tasks.workunit.client.0.smithi167.stdout:8/851: mkdir d6/db/d28/d61/dd5/dbf/d105/d115 0 2022-01-31T19:44:19.529 INFO:tasks.workunit.client.1.smithi171.stdout:9/154: write dd/d1c/f1f [1973857,82186] 0 2022-01-31T19:44:19.530 INFO:tasks.workunit.client.0.smithi167.stdout:8/852: read d6/db/de/f10 [675245,83497] 0 2022-01-31T19:44:19.530 INFO:tasks.workunit.client.0.smithi167.stdout:8/853: chown d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/fe4 3265 1 2022-01-31T19:44:19.532 INFO:tasks.workunit.client.1.smithi171.stdout:9/155: creat dd/df/f32 x:0 0 0 2022-01-31T19:44:19.538 INFO:tasks.workunit.client.1.smithi171.stdout:5/159: dwrite d5/d6/d15/f1d [0,4194304] 0 2022-01-31T19:44:19.541 INFO:tasks.workunit.client.0.smithi167.stdout:8/854: dread d6/db/de/d3d/f107 [0,4194304] 0 2022-01-31T19:44:19.542 INFO:tasks.workunit.client.1.smithi171.stdout:1/199: dwrite fa [0,4194304] 0 2022-01-31T19:44:19.542 INFO:tasks.workunit.client.1.smithi171.stdout:5/160: creat d5/f37 x:0 0 0 2022-01-31T19:44:19.542 INFO:tasks.workunit.client.1.smithi171.stdout:4/174: dwrite d2/d3/d5/dc/f26 [0,4194304] 0 2022-01-31T19:44:19.543 INFO:tasks.workunit.client.1.smithi171.stdout:4/175: truncate d2/d3/d5/d11/d12/f30 943148 0 2022-01-31T19:44:19.543 INFO:tasks.workunit.client.1.smithi171.stdout:4/176: write d2/d3/f7 [5210099,16368] 0 2022-01-31T19:44:19.544 INFO:tasks.workunit.client.1.smithi171.stdout:4/177: write d2/d3/d5/dc/f19 [3258044,91293] 0 2022-01-31T19:44:19.544 INFO:tasks.workunit.client.1.smithi171.stdout:1/200: write dd/d23/d2e/f2f [49787,6059] 0 2022-01-31T19:44:19.544 INFO:tasks.workunit.client.1.smithi171.stdout:1/201: write f7 [4680078,19310] 0 2022-01-31T19:44:19.545 INFO:tasks.workunit.client.1.smithi171.stdout:8/139: dwrite f8 [0,4194304] 0 2022-01-31T19:44:19.546 INFO:tasks.workunit.client.0.smithi167.stdout:8/855: write d6/d1e/d2d/f10d [2768774,123016] 0 2022-01-31T19:44:19.546 INFO:tasks.workunit.client.0.smithi167.stdout:8/856: write d6/db/d28/d61/de3/fea [18403,11479] 0 2022-01-31T19:44:19.546 INFO:tasks.workunit.client.0.smithi167.stdout:8/857: read - d6/db/de/d3d/fc1 zero size 2022-01-31T19:44:19.551 INFO:tasks.workunit.client.0.smithi167.stdout:8/858: write d6/d1e/d2d/f40 [3310009,73324] 0 2022-01-31T19:44:19.554 INFO:tasks.workunit.client.0.smithi167.stdout:1/978: getdents d0/d10/d5a/d33/d4b 0 2022-01-31T19:44:19.556 INFO:tasks.workunit.client.0.smithi167.stdout:1/979: dread d0/d62/d7f/d82/fef [4194304,4194304] 0 2022-01-31T19:44:19.557 INFO:tasks.workunit.client.0.smithi167.stdout:1/980: read d0/d10/d5a/d33/fa7 [384072,62347] 0 2022-01-31T19:44:19.558 INFO:tasks.workunit.client.0.smithi167.stdout:1/981: chown d0/db3/cfa 77731 1 2022-01-31T19:44:19.564 INFO:tasks.workunit.client.0.smithi167.stdout:8/859: dread d6/db/d6a/d8d/d9e/d60/f79 [0,4194304] 0 2022-01-31T19:44:19.565 INFO:tasks.workunit.client.0.smithi167.stdout:8/860: chown d6/db/d4e/d73/d88/fcb 2075471 1 2022-01-31T19:44:19.565 INFO:tasks.workunit.client.0.smithi167.stdout:8/861: dread - d6/d1e/dcf/ff5 zero size 2022-01-31T19:44:19.567 INFO:tasks.workunit.client.0.smithi167.stdout:1/982: dread d0/f30 [0,4194304] 0 2022-01-31T19:44:19.570 INFO:tasks.workunit.client.0.smithi167.stdout:8/862: fdatasync d6/db/d4e/f58 0 2022-01-31T19:44:19.570 INFO:tasks.workunit.client.0.smithi167.stdout:8/863: fdatasync d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/fc5 0 2022-01-31T19:44:19.571 INFO:tasks.workunit.client.0.smithi167.stdout:1/983: dread d0/d10/d5a/d33/d4b/d1f/f78 [4194304,4194304] 0 2022-01-31T19:44:19.573 INFO:tasks.workunit.client.0.smithi167.stdout:1/984: unlink d0/d10/d5a/d33/d4b/d1f/fa6 0 2022-01-31T19:44:19.579 INFO:tasks.workunit.client.0.smithi167.stdout:1/985: getdents d0/d11/d102 0 2022-01-31T19:44:19.579 INFO:tasks.workunit.client.0.smithi167.stdout:1/986: creat d0/d10/d20/d7c/f14d x:0 0 0 2022-01-31T19:44:19.590 INFO:tasks.workunit.client.0.smithi167.stdout:1/987: dread d0/fe4 [0,4194304] 0 2022-01-31T19:44:19.590 INFO:tasks.workunit.client.0.smithi167.stdout:1/988: readlink d0/d10/d5a/d33/d4b/d1f/l4c 0 2022-01-31T19:44:19.591 INFO:tasks.workunit.client.0.smithi167.stdout:1/989: symlink d0/d10/d5a/d33/d4b/da5/da9/dad/d112/l14e 0 2022-01-31T19:44:19.592 INFO:tasks.workunit.client.0.smithi167.stdout:1/990: fsync d0/d10/d5a/d33/d4b/d1f/fdf 0 2022-01-31T19:44:19.592 INFO:tasks.workunit.client.0.smithi167.stdout:1/991: readlink d0/d10/d20/l79 0 2022-01-31T19:44:19.606 INFO:tasks.workunit.client.1.smithi171.stdout:5/161: mknod d5/d6/d15/d29/c38 0 2022-01-31T19:44:19.608 INFO:tasks.workunit.client.1.smithi171.stdout:4/178: creat d2/d3/d5/dc/f39 x:0 0 0 2022-01-31T19:44:19.611 INFO:tasks.workunit.client.1.smithi171.stdout:1/202: creat dd/f42 x:0 0 0 2022-01-31T19:44:19.612 INFO:tasks.workunit.client.1.smithi171.stdout:8/140: rename df/d18/l1d to df/d18/l37 0 2022-01-31T19:44:19.612 INFO:tasks.workunit.client.1.smithi171.stdout:8/141: fsync fb 0 2022-01-31T19:44:19.613 INFO:tasks.workunit.client.1.smithi171.stdout:5/162: creat d5/d6/d15/d29/f39 x:0 0 0 2022-01-31T19:44:19.613 INFO:tasks.workunit.client.1.smithi171.stdout:5/163: chown d5/d6/dd/f19 0 1 2022-01-31T19:44:19.614 INFO:tasks.workunit.client.1.smithi171.stdout:4/179: creat d2/d3/d5/d11/d12/d15/f3a x:0 0 0 2022-01-31T19:44:19.622 INFO:tasks.workunit.client.1.smithi171.stdout:1/203: rename dd/ce to dd/df/d19/d1b/c43 0 2022-01-31T19:44:19.624 INFO:tasks.workunit.client.1.smithi171.stdout:4/180: mkdir d2/d3/d5/d3b 0 2022-01-31T19:44:19.625 INFO:tasks.workunit.client.1.smithi171.stdout:4/181: fdatasync d2/d3/d5/dc/f19 0 2022-01-31T19:44:19.625 INFO:tasks.workunit.client.1.smithi171.stdout:4/182: fdatasync d2/d3/d5/d11/d12/f2a 0 2022-01-31T19:44:19.627 INFO:tasks.workunit.client.1.smithi171.stdout:3/185: sync 2022-01-31T19:44:19.628 INFO:tasks.workunit.client.0.smithi167.stdout:4/957: sync 2022-01-31T19:44:19.628 INFO:tasks.workunit.client.0.smithi167.stdout:4/958: fdatasync d0/d6/d7/d26/ff5 0 2022-01-31T19:44:19.630 INFO:tasks.workunit.client.1.smithi171.stdout:3/186: mkdir d4/dd/d27/d28/d35 0 2022-01-31T19:44:19.630 INFO:tasks.workunit.client.1.smithi171.stdout:3/187: stat d4/dd/c25 0 2022-01-31T19:44:19.638 INFO:tasks.workunit.client.1.smithi171.stdout:3/188: write d4/d21/f2a [2429552,9435] 0 2022-01-31T19:44:19.639 INFO:tasks.workunit.client.1.smithi171.stdout:3/189: link d4/dd/cf d4/dd/c36 0 2022-01-31T19:44:19.639 INFO:tasks.workunit.client.0.smithi167.stdout:4/959: dread d0/d6/d7/f18 [4194304,4194304] 0 2022-01-31T19:44:19.642 INFO:tasks.workunit.client.1.smithi171.stdout:5/164: dwrite d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:19.642 INFO:tasks.workunit.client.1.smithi171.stdout:5/165: fdatasync d5/d6/f27 0 2022-01-31T19:44:19.643 INFO:tasks.workunit.client.0.smithi167.stdout:4/960: mknod d0/d6/d7/d89/d11c/c13b 0 2022-01-31T19:44:19.644 INFO:tasks.workunit.client.1.smithi171.stdout:3/190: read d4/f1d [1093200,108482] 0 2022-01-31T19:44:19.644 INFO:tasks.workunit.client.1.smithi171.stdout:3/191: fdatasync d4/dd/f1e 0 2022-01-31T19:44:19.644 INFO:tasks.workunit.client.1.smithi171.stdout:8/142: dwrite df/d18/d19/d21/f31 [0,4194304] 0 2022-01-31T19:44:19.644 INFO:tasks.workunit.client.1.smithi171.stdout:8/143: fdatasync df/d18/f1e 0 2022-01-31T19:44:19.645 INFO:tasks.workunit.client.1.smithi171.stdout:4/183: dwrite d2/d3/d5/d11/d12/f1d [0,4194304] 0 2022-01-31T19:44:19.645 INFO:tasks.workunit.client.1.smithi171.stdout:4/184: chown d2/d3/d5/lf 1418 1 2022-01-31T19:44:19.645 INFO:tasks.workunit.client.0.smithi167.stdout:4/961: symlink d0/d5d/d12f/d5c/de2/l13c 0 2022-01-31T19:44:19.646 INFO:tasks.workunit.client.0.smithi167.stdout:4/962: getdents d0/d6/d7/d89/d11c/d130 0 2022-01-31T19:44:19.648 INFO:tasks.workunit.client.1.smithi171.stdout:5/166: rmdir d5/d6/d15/d29 39 2022-01-31T19:44:19.648 INFO:tasks.workunit.client.1.smithi171.stdout:5/167: write d5/d6/d15/f1f [4583230,96757] 0 2022-01-31T19:44:19.648 INFO:tasks.workunit.client.1.smithi171.stdout:5/168: chown d5/d6/d15/l30 12 1 2022-01-31T19:44:19.650 INFO:tasks.workunit.client.1.smithi171.stdout:5/169: read d5/d6/dd/f20 [3157077,126095] 0 2022-01-31T19:44:19.650 INFO:tasks.workunit.client.1.smithi171.stdout:8/144: mkdir df/d2a/d38 0 2022-01-31T19:44:19.650 INFO:tasks.workunit.client.1.smithi171.stdout:5/170: chown f1 71498870 1 2022-01-31T19:44:19.651 INFO:tasks.workunit.client.1.smithi171.stdout:8/145: write df/f20 [86789,44723] 0 2022-01-31T19:44:19.651 INFO:tasks.workunit.client.1.smithi171.stdout:1/204: rmdir dd/df/d19/d1b 39 2022-01-31T19:44:19.652 INFO:tasks.workunit.client.0.smithi167.stdout:4/963: dread d0/d6/d7/f8 [0,4194304] 0 2022-01-31T19:44:19.653 INFO:tasks.workunit.client.1.smithi171.stdout:1/205: write dd/f14 [437517,49937] 0 2022-01-31T19:44:19.654 INFO:tasks.workunit.client.0.smithi167.stdout:4/964: symlink d0/l13d 0 2022-01-31T19:44:19.655 INFO:tasks.workunit.client.1.smithi171.stdout:3/192: getdents d4/dd 0 2022-01-31T19:44:19.657 INFO:tasks.workunit.client.1.smithi171.stdout:5/171: mknod d5/d6/d15/d23/c3a 0 2022-01-31T19:44:19.657 INFO:tasks.workunit.client.1.smithi171.stdout:5/172: creat d5/f3b x:0 0 0 2022-01-31T19:44:19.658 INFO:tasks.workunit.client.1.smithi171.stdout:5/173: dread - d5/d6/d15/d23/f25 zero size 2022-01-31T19:44:19.659 INFO:tasks.workunit.client.1.smithi171.stdout:4/185: dwrite d2/d3/d5/d11/d12/f30 [0,4194304] 0 2022-01-31T19:44:19.663 INFO:tasks.workunit.client.1.smithi171.stdout:4/186: rename d2/d3/d5/d11/d12/f30 to d2/f3c 0 2022-01-31T19:44:19.665 INFO:tasks.workunit.client.1.smithi171.stdout:4/187: dread d2/f14 [0,4194304] 0 2022-01-31T19:44:19.666 INFO:tasks.workunit.client.1.smithi171.stdout:4/188: link d2/d3/c2f d2/d3/c3d 0 2022-01-31T19:44:19.668 INFO:tasks.workunit.client.1.smithi171.stdout:4/189: rename d2/d3/f10 to d2/d3/d5/d11/d12/d15/f3e 0 2022-01-31T19:44:19.669 INFO:tasks.workunit.client.1.smithi171.stdout:8/146: getdents df/d2a 0 2022-01-31T19:44:19.670 INFO:tasks.workunit.client.1.smithi171.stdout:4/190: truncate d2/d3/d5/d11/f27 3635046 0 2022-01-31T19:44:19.671 INFO:tasks.workunit.client.1.smithi171.stdout:4/191: dread - d2/d3/d5/d11/d12/f2a zero size 2022-01-31T19:44:19.671 INFO:tasks.workunit.client.1.smithi171.stdout:4/192: dread - d2/d3/f28 zero size 2022-01-31T19:44:19.673 INFO:tasks.workunit.client.1.smithi171.stdout:8/147: write f7 [1981181,110063] 0 2022-01-31T19:44:19.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/193: write d2/d3/d5/d11/d12/d15/f3e [1305228,29880] 0 2022-01-31T19:44:19.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/194: write d2/d3/d5/f38 [571046,18041] 0 2022-01-31T19:44:19.676 INFO:tasks.workunit.client.1.smithi171.stdout:4/195: chown d2/ld 310 1 2022-01-31T19:44:19.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/193: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:19.683 INFO:tasks.workunit.client.1.smithi171.stdout:5/174: dwrite d5/d6/f27 [0,4194304] 0 2022-01-31T19:44:19.685 INFO:tasks.workunit.client.1.smithi171.stdout:4/196: fsync d2/d3/d5/f38 0 2022-01-31T19:44:19.685 INFO:tasks.workunit.client.1.smithi171.stdout:4/197: stat d2/d3/d5/d11/d12/c23 0 2022-01-31T19:44:19.687 INFO:tasks.workunit.client.1.smithi171.stdout:5/175: write d5/ff [1492245,12027] 0 2022-01-31T19:44:19.695 INFO:tasks.workunit.client.1.smithi171.stdout:8/148: dwrite df/f10 [0,4194304] 0 2022-01-31T19:44:19.696 INFO:tasks.workunit.client.1.smithi171.stdout:8/149: creat df/d18/f39 x:0 0 0 2022-01-31T19:44:19.697 INFO:tasks.workunit.client.1.smithi171.stdout:8/150: write fa [895041,46832] 0 2022-01-31T19:44:19.697 INFO:tasks.workunit.client.1.smithi171.stdout:8/151: rmdir df/d2a 39 2022-01-31T19:44:19.699 INFO:tasks.workunit.client.1.smithi171.stdout:8/152: symlink df/l3a 0 2022-01-31T19:44:19.699 INFO:tasks.workunit.client.1.smithi171.stdout:8/153: write df/d18/d19/d21/f32 [228164,119816] 0 2022-01-31T19:44:19.701 INFO:tasks.workunit.client.1.smithi171.stdout:8/154: link f1 df/d1a/f3b 0 2022-01-31T19:44:19.702 INFO:tasks.workunit.client.1.smithi171.stdout:4/198: dread d2/d3/d5/d11/d12/f1d [0,4194304] 0 2022-01-31T19:44:19.703 INFO:tasks.workunit.client.1.smithi171.stdout:4/199: truncate d2/f3c 4912184 0 2022-01-31T19:44:19.703 INFO:tasks.workunit.client.1.smithi171.stdout:8/155: mkdir df/d18/d3c 0 2022-01-31T19:44:19.704 INFO:tasks.workunit.client.1.smithi171.stdout:5/176: dwrite d5/f8 [4194304,4194304] 0 2022-01-31T19:44:19.704 INFO:tasks.workunit.client.1.smithi171.stdout:4/200: symlink d2/d3/d5/d11/d12/d15/l3f 0 2022-01-31T19:44:19.705 INFO:tasks.workunit.client.1.smithi171.stdout:4/201: stat d2/d3/d5/d11/d12/d15/l3f 0 2022-01-31T19:44:19.706 INFO:tasks.workunit.client.1.smithi171.stdout:8/156: mkdir df/d2a/d38/d3d 0 2022-01-31T19:44:19.708 INFO:tasks.workunit.client.1.smithi171.stdout:8/157: creat df/d18/d19/d36/f3e x:0 0 0 2022-01-31T19:44:19.708 INFO:tasks.workunit.client.1.smithi171.stdout:4/202: getdents d2/d3/d5 0 2022-01-31T19:44:19.710 INFO:tasks.workunit.client.1.smithi171.stdout:8/158: symlink df/d27/l3f 0 2022-01-31T19:44:19.710 INFO:tasks.workunit.client.1.smithi171.stdout:8/159: write df/d2a/d30/f35 [675024,94090] 0 2022-01-31T19:44:19.715 INFO:tasks.workunit.client.1.smithi171.stdout:8/160: rename df/d18/f39 to df/d2a/f40 0 2022-01-31T19:44:19.716 INFO:tasks.workunit.client.1.smithi171.stdout:8/161: readlink df/l15 0 2022-01-31T19:44:19.716 INFO:tasks.workunit.client.1.smithi171.stdout:8/162: readlink df/d18/l2e 0 2022-01-31T19:44:19.717 INFO:tasks.workunit.client.1.smithi171.stdout:7/140: sync 2022-01-31T19:44:19.718 INFO:tasks.workunit.client.1.smithi171.stdout:4/203: rmdir d2/d3 39 2022-01-31T19:44:19.721 INFO:tasks.workunit.client.1.smithi171.stdout:7/141: rename d1/d1e/d20 to d1/d1e/d2a 0 2022-01-31T19:44:19.726 INFO:tasks.workunit.client.1.smithi171.stdout:7/142: dread d1/d24/f28 [0,4194304] 0 2022-01-31T19:44:19.726 INFO:tasks.workunit.client.1.smithi171.stdout:7/143: read d1/d1b/d7/f19 [276783,98958] 0 2022-01-31T19:44:19.728 INFO:tasks.workunit.client.1.smithi171.stdout:7/144: symlink d1/d1b/l2b 0 2022-01-31T19:44:19.728 INFO:tasks.workunit.client.1.smithi171.stdout:7/145: getdents d1/d26 0 2022-01-31T19:44:19.729 INFO:tasks.workunit.client.1.smithi171.stdout:7/146: symlink d1/d1b/l2c 0 2022-01-31T19:44:19.730 INFO:tasks.workunit.client.1.smithi171.stdout:8/163: dread f2 [0,4194304] 0 2022-01-31T19:44:19.731 INFO:tasks.workunit.client.1.smithi171.stdout:8/164: write df/d18/d19/f1b [357390,51817] 0 2022-01-31T19:44:19.731 INFO:tasks.workunit.client.1.smithi171.stdout:8/165: creat df/d18/f41 x:0 0 0 2022-01-31T19:44:19.731 INFO:tasks.workunit.client.1.smithi171.stdout:7/147: mknod d1/d1e/c2d 0 2022-01-31T19:44:19.731 INFO:tasks.workunit.client.1.smithi171.stdout:7/148: fsync d1/d1b/f22 0 2022-01-31T19:44:19.732 INFO:tasks.workunit.client.1.smithi171.stdout:5/177: dwrite d5/d6/f1a [0,4194304] 0 2022-01-31T19:44:19.732 INFO:tasks.workunit.client.1.smithi171.stdout:7/149: read d1/d1b/d7/f19 [12714,79138] 0 2022-01-31T19:44:19.734 INFO:tasks.workunit.client.1.smithi171.stdout:5/178: symlink d5/d6/d15/l3c 0 2022-01-31T19:44:19.734 INFO:tasks.workunit.client.1.smithi171.stdout:5/179: write d5/d6/f14 [9435862,31904] 0 2022-01-31T19:44:19.735 INFO:tasks.workunit.client.1.smithi171.stdout:7/150: link d1/d1b/f22 d1/d1b/d7/f2e 0 2022-01-31T19:44:19.736 INFO:tasks.workunit.client.1.smithi171.stdout:4/204: dwrite d2/d3/d5/f1a [0,4194304] 0 2022-01-31T19:44:19.736 INFO:tasks.workunit.client.1.smithi171.stdout:4/205: fdatasync d2/d3/d5/dc/f19 0 2022-01-31T19:44:19.737 INFO:tasks.workunit.client.1.smithi171.stdout:5/180: rmdir d5/d6/d15/d29 39 2022-01-31T19:44:19.738 INFO:tasks.workunit.client.1.smithi171.stdout:7/151: creat d1/d1e/d2a/f2f x:0 0 0 2022-01-31T19:44:19.740 INFO:tasks.workunit.client.1.smithi171.stdout:5/181: mkdir d5/d6/d15/d23/d3d 0 2022-01-31T19:44:19.742 INFO:tasks.workunit.client.1.smithi171.stdout:7/152: mkdir d1/d1b/d7/d30 0 2022-01-31T19:44:19.743 INFO:tasks.workunit.client.1.smithi171.stdout:7/153: write d1/d24/f21 [320897,14502] 0 2022-01-31T19:44:19.743 INFO:tasks.workunit.client.1.smithi171.stdout:5/182: symlink d5/d6/l3e 0 2022-01-31T19:44:19.754 INFO:tasks.workunit.client.1.smithi171.stdout:8/166: dwrite df/d18/f1c [0,4194304] 0 2022-01-31T19:44:19.756 INFO:tasks.workunit.client.1.smithi171.stdout:8/167: getdents df/d18/d19 0 2022-01-31T19:44:19.759 INFO:tasks.workunit.client.1.smithi171.stdout:7/154: rename d1/d1b/d7 to d1/d1e/d2a/d29/d31 0 2022-01-31T19:44:19.760 INFO:tasks.workunit.client.1.smithi171.stdout:4/206: dwrite d2/d3/d5/dc/f26 [0,4194304] 0 2022-01-31T19:44:19.761 INFO:tasks.workunit.client.1.smithi171.stdout:8/168: rename df/f10 to df/d18/f42 0 2022-01-31T19:44:19.762 INFO:tasks.workunit.client.1.smithi171.stdout:5/183: truncate d5/d6/f14 6394180 0 2022-01-31T19:44:19.763 INFO:tasks.workunit.client.1.smithi171.stdout:7/155: symlink d1/d26/l32 0 2022-01-31T19:44:19.764 INFO:tasks.workunit.client.1.smithi171.stdout:4/207: symlink d2/d3/d5/d11/d12/l40 0 2022-01-31T19:44:19.764 INFO:tasks.workunit.client.1.smithi171.stdout:4/208: stat d2/d3/f1c 0 2022-01-31T19:44:19.768 INFO:tasks.workunit.client.1.smithi171.stdout:5/184: fsync d5/d6/f14 0 2022-01-31T19:44:19.771 INFO:tasks.workunit.client.1.smithi171.stdout:7/156: symlink d1/d1e/d2a/d29/l33 0 2022-01-31T19:44:19.771 INFO:tasks.workunit.client.1.smithi171.stdout:4/209: mkdir d2/d3/d5/d41 0 2022-01-31T19:44:19.772 INFO:tasks.workunit.client.1.smithi171.stdout:8/169: rmdir df/d18 39 2022-01-31T19:44:19.773 INFO:tasks.workunit.client.1.smithi171.stdout:8/170: getdents df/d18/d3c 0 2022-01-31T19:44:19.774 INFO:tasks.workunit.client.1.smithi171.stdout:7/157: creat d1/d1e/d2a/d29/d31/f34 x:0 0 0 2022-01-31T19:44:19.774 INFO:tasks.workunit.client.1.smithi171.stdout:7/158: read d1/d24/f21 [141456,110665] 0 2022-01-31T19:44:19.776 INFO:tasks.workunit.client.1.smithi171.stdout:4/210: unlink d2/d3/d5/d11/d12/d15/l3f 0 2022-01-31T19:44:19.776 INFO:tasks.workunit.client.1.smithi171.stdout:4/211: chown d2/d3/d5/l34 283490876 1 2022-01-31T19:44:19.778 INFO:tasks.workunit.client.1.smithi171.stdout:8/171: rename df/d18/d19/d21/f31 to df/d2a/d38/f43 0 2022-01-31T19:44:19.778 INFO:tasks.workunit.client.1.smithi171.stdout:8/172: chown df/d18/d19/d36 3755 1 2022-01-31T19:44:19.779 INFO:tasks.workunit.client.1.smithi171.stdout:4/212: creat d2/d3/d5/dc/f42 x:0 0 0 2022-01-31T19:44:19.780 INFO:tasks.workunit.client.1.smithi171.stdout:4/213: read d2/d3/d5/f1a [2604139,121695] 0 2022-01-31T19:44:19.782 INFO:tasks.workunit.client.1.smithi171.stdout:4/214: write d2/d3/d5/d11/d12/f1d [1976159,114109] 0 2022-01-31T19:44:19.784 INFO:tasks.workunit.client.1.smithi171.stdout:8/173: dread df/d2a/d30/f35 [0,4194304] 0 2022-01-31T19:44:19.785 INFO:tasks.workunit.client.1.smithi171.stdout:7/159: dread d1/f16 [0,4194304] 0 2022-01-31T19:44:19.785 INFO:tasks.workunit.client.1.smithi171.stdout:4/215: creat d2/f43 x:0 0 0 2022-01-31T19:44:19.786 INFO:tasks.workunit.client.1.smithi171.stdout:8/174: write f2 [3456087,39602] 0 2022-01-31T19:44:19.789 INFO:tasks.workunit.client.1.smithi171.stdout:8/175: symlink df/d27/l44 0 2022-01-31T19:44:19.801 INFO:tasks.workunit.client.1.smithi171.stdout:5/185: dwrite d5/d6/d15/f1f [0,4194304] 0 2022-01-31T19:44:19.802 INFO:tasks.workunit.client.1.smithi171.stdout:7/160: dwrite d1/d24/f1a [0,4194304] 0 2022-01-31T19:44:19.803 INFO:tasks.workunit.client.1.smithi171.stdout:7/161: unlink d1/d1b/l2c 0 2022-01-31T19:44:19.803 INFO:tasks.workunit.client.1.smithi171.stdout:7/162: creat d1/d24/f35 x:0 0 0 2022-01-31T19:44:19.803 INFO:tasks.workunit.client.1.smithi171.stdout:5/186: link d5/f8 d5/d6/d15/f3f 0 2022-01-31T19:44:19.807 INFO:tasks.workunit.client.1.smithi171.stdout:8/176: dread df/d1a/f3b [8388608,4194304] 0 2022-01-31T19:44:19.812 INFO:tasks.workunit.client.1.smithi171.stdout:7/163: rename d1/d1e/d2a/d29/d31/f2e to d1/f36 0 2022-01-31T19:44:19.812 INFO:tasks.workunit.client.1.smithi171.stdout:7/164: creat d1/d24/f37 x:0 0 0 2022-01-31T19:44:19.813 INFO:tasks.workunit.client.1.smithi171.stdout:7/165: write d1/d1b/fc [4576806,54657] 0 2022-01-31T19:44:19.813 INFO:tasks.workunit.client.1.smithi171.stdout:7/166: readlink d1/d24/l27 0 2022-01-31T19:44:19.813 INFO:tasks.workunit.client.1.smithi171.stdout:7/167: creat d1/d1e/d2a/d29/d31/f38 x:0 0 0 2022-01-31T19:44:19.819 INFO:tasks.workunit.client.1.smithi171.stdout:5/187: dwrite d5/d6/d15/f1c [0,4194304] 0 2022-01-31T19:44:19.819 INFO:tasks.workunit.client.1.smithi171.stdout:5/188: write d5/d6/dd/f22 [1048285,56580] 0 2022-01-31T19:44:19.819 INFO:tasks.workunit.client.1.smithi171.stdout:5/189: chown d5/d6/l7 5 1 2022-01-31T19:44:19.819 INFO:tasks.workunit.client.1.smithi171.stdout:8/177: symlink df/d1a/l45 0 2022-01-31T19:44:19.821 INFO:tasks.workunit.client.1.smithi171.stdout:7/168: creat d1/d1e/d2a/d29/d31/d30/f39 x:0 0 0 2022-01-31T19:44:19.824 INFO:tasks.workunit.client.1.smithi171.stdout:8/178: rmdir df/d2a/d38/d3d 0 2022-01-31T19:44:19.824 INFO:tasks.workunit.client.1.smithi171.stdout:5/190: dread d5/d6/dd/f22 [0,4194304] 0 2022-01-31T19:44:19.827 INFO:tasks.workunit.client.1.smithi171.stdout:2/247: sync 2022-01-31T19:44:19.827 INFO:tasks.workunit.client.1.smithi171.stdout:6/197: sync 2022-01-31T19:44:19.827 INFO:tasks.workunit.client.1.smithi171.stdout:2/248: dread - d7/d9/f46 zero size 2022-01-31T19:44:19.830 INFO:tasks.workunit.client.1.smithi171.stdout:8/179: mkdir df/d18/d19/d46 0 2022-01-31T19:44:19.830 INFO:tasks.workunit.client.1.smithi171.stdout:8/180: fdatasync df/d1a/f29 0 2022-01-31T19:44:19.832 INFO:tasks.workunit.client.1.smithi171.stdout:5/191: unlink d5/lc 0 2022-01-31T19:44:19.834 INFO:tasks.workunit.client.1.smithi171.stdout:8/181: creat df/d18/d19/d36/f47 x:0 0 0 2022-01-31T19:44:19.834 INFO:tasks.workunit.client.1.smithi171.stdout:8/182: fsync fa 0 2022-01-31T19:44:19.835 INFO:tasks.workunit.client.1.smithi171.stdout:5/192: rename d5/d6/d15/l3c to d5/l40 0 2022-01-31T19:44:19.835 INFO:tasks.workunit.client.1.smithi171.stdout:8/183: rmdir df/d2a/d30 39 2022-01-31T19:44:19.836 INFO:tasks.workunit.client.1.smithi171.stdout:6/198: write f9 [2165655,30243] 0 2022-01-31T19:44:19.845 INFO:tasks.workunit.client.1.smithi171.stdout:2/249: dwrite d7/d26/d3d/f42 [0,4194304] 0 2022-01-31T19:44:19.851 INFO:tasks.workunit.client.1.smithi171.stdout:5/193: dwrite d5/d6/dd/f19 [0,4194304] 0 2022-01-31T19:44:19.948 INFO:tasks.workunit.client.1.smithi171.stdout:6/199: rename dc/d19/f32 to dc/d19/d29/d2f/f38 0 2022-01-31T19:44:19.949 INFO:tasks.workunit.client.1.smithi171.stdout:8/184: rmdir df/d18/d3c 0 2022-01-31T19:44:19.950 INFO:tasks.workunit.client.1.smithi171.stdout:6/200: dread dc/d18/f22 [0,4194304] 0 2022-01-31T19:44:19.951 INFO:tasks.workunit.client.1.smithi171.stdout:6/201: readlink dc/d19/d30/l35 0 2022-01-31T19:44:19.951 INFO:tasks.workunit.client.1.smithi171.stdout:6/202: write f9 [4607569,73153] 0 2022-01-31T19:44:20.020 INFO:tasks.workunit.client.1.smithi171.stdout:5/194: link d5/d6/dd/f35 d5/d6/d15/d23/f41 0 2022-01-31T19:44:20.020 INFO:tasks.workunit.client.1.smithi171.stdout:5/195: creat d5/d6/d15/f42 x:0 0 0 2022-01-31T19:44:20.021 INFO:tasks.workunit.client.1.smithi171.stdout:5/196: unlink d5/c11 0 2022-01-31T19:44:20.021 INFO:tasks.workunit.client.1.smithi171.stdout:5/197: chown d5/d6/d15/d29/c38 32431514 1 2022-01-31T19:44:20.021 INFO:tasks.workunit.client.1.smithi171.stdout:5/198: write d5/d6/d15/f1d [4436137,101092] 0 2022-01-31T19:44:20.022 INFO:tasks.workunit.client.1.smithi171.stdout:5/199: write f1 [97392,88248] 0 2022-01-31T19:44:20.023 INFO:tasks.workunit.client.1.smithi171.stdout:5/200: write d5/f37 [1001898,109465] 0 2022-01-31T19:44:20.023 INFO:tasks.workunit.client.1.smithi171.stdout:5/201: write d5/d6/d15/d23/f41 [641442,108107] 0 2022-01-31T19:44:20.023 INFO:tasks.workunit.client.1.smithi171.stdout:2/250: rename d7/d9/c2d to d7/d26/d3d/d40/c49 0 2022-01-31T19:44:20.023 INFO:tasks.workunit.client.1.smithi171.stdout:2/251: write d7/f3a [931396,38989] 0 2022-01-31T19:44:20.024 INFO:tasks.workunit.client.1.smithi171.stdout:2/252: dread - d7/d26/d3d/d40/f48 zero size 2022-01-31T19:44:20.024 INFO:tasks.workunit.client.1.smithi171.stdout:8/185: write f3 [924122,63854] 0 2022-01-31T19:44:20.025 INFO:tasks.workunit.client.1.smithi171.stdout:6/203: creat dc/d18/f39 x:0 0 0 2022-01-31T19:44:20.025 INFO:tasks.workunit.client.1.smithi171.stdout:6/204: chown dc/ce 44207434 1 2022-01-31T19:44:20.026 INFO:tasks.workunit.client.1.smithi171.stdout:6/205: write dc/d18/f2a [3289637,92981] 0 2022-01-31T19:44:20.026 INFO:tasks.workunit.client.1.smithi171.stdout:6/206: stat dc/d18 0 2022-01-31T19:44:20.032 INFO:tasks.workunit.client.1.smithi171.stdout:5/202: symlink d5/l43 0 2022-01-31T19:44:20.033 INFO:tasks.workunit.client.1.smithi171.stdout:2/253: creat d7/dd/f4a x:0 0 0 2022-01-31T19:44:20.034 INFO:tasks.workunit.client.1.smithi171.stdout:8/186: symlink df/d2a/l48 0 2022-01-31T19:44:20.040 INFO:tasks.workunit.client.1.smithi171.stdout:5/203: unlink d5/d6/f14 0 2022-01-31T19:44:20.041 INFO:tasks.workunit.client.1.smithi171.stdout:2/254: symlink d7/d9/d1a/l4b 0 2022-01-31T19:44:20.042 INFO:tasks.workunit.client.1.smithi171.stdout:8/187: creat df/d2a/f49 x:0 0 0 2022-01-31T19:44:20.050 INFO:tasks.workunit.client.1.smithi171.stdout:2/255: creat d7/dd/f4c x:0 0 0 2022-01-31T19:44:20.052 INFO:tasks.workunit.client.1.smithi171.stdout:5/204: truncate d5/d6/d15/f1c 340539 0 2022-01-31T19:44:20.052 INFO:tasks.workunit.client.1.smithi171.stdout:5/205: rename d5 to d5/d6/d44 22 2022-01-31T19:44:20.052 INFO:tasks.workunit.client.1.smithi171.stdout:5/206: creat d5/d6/dd/f45 x:0 0 0 2022-01-31T19:44:20.052 INFO:tasks.workunit.client.1.smithi171.stdout:5/207: creat d5/d6/f46 x:0 0 0 2022-01-31T19:44:20.053 INFO:tasks.workunit.client.1.smithi171.stdout:2/256: symlink d7/d26/d3d/l4d 0 2022-01-31T19:44:20.056 INFO:tasks.workunit.client.1.smithi171.stdout:5/208: rename d5/d6/l7 to d5/l47 0 2022-01-31T19:44:20.059 INFO:tasks.workunit.client.1.smithi171.stdout:2/257: rename d7/dd/f15 to d7/d9/f4e 0 2022-01-31T19:44:20.059 INFO:tasks.workunit.client.1.smithi171.stdout:2/258: dread - d7/dd/f4c zero size 2022-01-31T19:44:20.061 INFO:tasks.workunit.client.1.smithi171.stdout:2/259: truncate d7/d9/f12 1656621 0 2022-01-31T19:44:20.061 INFO:tasks.workunit.client.1.smithi171.stdout:2/260: creat d7/d26/d3d/d40/f4f x:0 0 0 2022-01-31T19:44:20.062 INFO:tasks.workunit.client.1.smithi171.stdout:2/261: readlink d7/lc 0 2022-01-31T19:44:20.062 INFO:tasks.workunit.client.1.smithi171.stdout:2/262: write d7/d9/f46 [905399,32416] 0 2022-01-31T19:44:20.062 INFO:tasks.workunit.client.1.smithi171.stdout:2/263: dread - d7/f25 zero size 2022-01-31T19:44:20.063 INFO:tasks.workunit.client.1.smithi171.stdout:2/264: readlink l5 0 2022-01-31T19:44:20.063 INFO:tasks.workunit.client.1.smithi171.stdout:2/265: readlink d7/d9/d1a/d45/l28 0 2022-01-31T19:44:20.064 INFO:tasks.workunit.client.1.smithi171.stdout:5/209: link d5/d6/d15/f1f d5/d6/d15/d29/f48 0 2022-01-31T19:44:20.067 INFO:tasks.workunit.client.1.smithi171.stdout:8/188: dwrite f1 [4194304,4194304] 0 2022-01-31T19:44:20.073 INFO:tasks.workunit.client.1.smithi171.stdout:5/210: creat d5/d6/d15/d23/d3d/f49 x:0 0 0 2022-01-31T19:44:20.074 INFO:tasks.workunit.client.1.smithi171.stdout:0/222: sync 2022-01-31T19:44:20.074 INFO:tasks.workunit.client.1.smithi171.stdout:2/266: rmdir d7 39 2022-01-31T19:44:20.076 INFO:tasks.workunit.client.1.smithi171.stdout:8/189: getdents df/d18/d19/d21 0 2022-01-31T19:44:20.076 INFO:tasks.workunit.client.1.smithi171.stdout:8/190: write df/d1a/f29 [168611,64587] 0 2022-01-31T19:44:20.076 INFO:tasks.workunit.client.1.smithi171.stdout:5/211: symlink d5/l4a 0 2022-01-31T19:44:20.080 INFO:tasks.workunit.client.1.smithi171.stdout:5/212: write d5/d6/f1a [1390267,95499] 0 2022-01-31T19:44:20.081 INFO:tasks.workunit.client.1.smithi171.stdout:5/213: fsync d5/d6/f27 0 2022-01-31T19:44:20.083 INFO:tasks.workunit.client.1.smithi171.stdout:5/214: mknod d5/d6/d15/d29/c4b 0 2022-01-31T19:44:20.084 INFO:tasks.workunit.client.1.smithi171.stdout:5/215: fdatasync d5/d6/d15/d29/f48 0 2022-01-31T19:44:20.085 INFO:tasks.workunit.client.1.smithi171.stdout:2/267: dread d7/dd/f1b [0,4194304] 0 2022-01-31T19:44:20.098 INFO:tasks.workunit.client.1.smithi171.stdout:0/223: dwrite f8 [0,4194304] 0 2022-01-31T19:44:20.100 INFO:tasks.workunit.client.1.smithi171.stdout:8/191: dwrite df/d2a/d38/f43 [0,4194304] 0 2022-01-31T19:44:20.104 INFO:tasks.workunit.client.1.smithi171.stdout:0/224: dread d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:20.105 INFO:tasks.workunit.client.1.smithi171.stdout:0/225: mknod d9/db/d22/d3a/c55 0 2022-01-31T19:44:20.124 INFO:tasks.workunit.client.1.smithi171.stdout:5/216: dwrite d5/d6/d15/f3f [8388608,4194304] 0 2022-01-31T19:44:20.125 INFO:tasks.workunit.client.1.smithi171.stdout:2/268: dwrite d7/dd/f1b [0,4194304] 0 2022-01-31T19:44:20.125 INFO:tasks.workunit.client.1.smithi171.stdout:2/269: rmdir d7/d9/d1a 39 2022-01-31T19:44:20.126 INFO:tasks.workunit.client.1.smithi171.stdout:2/270: chown d7/dd/l30 107 1 2022-01-31T19:44:20.126 INFO:tasks.workunit.client.1.smithi171.stdout:0/226: dwrite d9/db/dc/d3f/f34 [0,4194304] 0 2022-01-31T19:44:20.140 INFO:tasks.workunit.client.1.smithi171.stdout:2/271: dwrite d7/d9/d1a/f32 [0,4194304] 0 2022-01-31T19:44:20.151 INFO:tasks.workunit.client.1.smithi171.stdout:2/272: dread d7/d26/d3d/f42 [0,4194304] 0 2022-01-31T19:44:20.152 INFO:tasks.workunit.client.1.smithi171.stdout:2/273: write d7/dd/f1e [227751,106921] 0 2022-01-31T19:44:20.152 INFO:tasks.workunit.client.1.smithi171.stdout:2/274: dread - d7/d26/d3d/d40/f4f zero size 2022-01-31T19:44:20.153 INFO:tasks.workunit.client.1.smithi171.stdout:2/275: mknod d7/c50 0 2022-01-31T19:44:20.153 INFO:tasks.workunit.client.1.smithi171.stdout:0/227: dwrite d9/db/dc/d3f/f34 [4194304,4194304] 0 2022-01-31T19:44:20.154 INFO:tasks.workunit.client.1.smithi171.stdout:2/276: creat d7/d26/d3d/d40/f51 x:0 0 0 2022-01-31T19:44:20.154 INFO:tasks.workunit.client.1.smithi171.stdout:0/228: symlink d9/db/d22/d3a/l56 0 2022-01-31T19:44:20.155 INFO:tasks.workunit.client.1.smithi171.stdout:2/277: creat d7/dd/f52 x:0 0 0 2022-01-31T19:44:20.155 INFO:tasks.workunit.client.1.smithi171.stdout:2/278: write d7/f22 [4871394,23959] 0 2022-01-31T19:44:20.156 INFO:tasks.workunit.client.1.smithi171.stdout:2/279: chown d7/d26/d3d/d40/f51 0 1 2022-01-31T19:44:20.156 INFO:tasks.workunit.client.1.smithi171.stdout:2/280: read d7/f22 [4246444,82354] 0 2022-01-31T19:44:20.156 INFO:tasks.workunit.client.1.smithi171.stdout:0/229: symlink d9/db/dc/d3f/l57 0 2022-01-31T19:44:20.157 INFO:tasks.workunit.client.1.smithi171.stdout:0/230: mknod d9/db/d22/c58 0 2022-01-31T19:44:20.157 INFO:tasks.workunit.client.1.smithi171.stdout:0/231: write f4 [4984004,103806] 0 2022-01-31T19:44:20.158 INFO:tasks.workunit.client.1.smithi171.stdout:2/281: dread d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:20.158 INFO:tasks.workunit.client.1.smithi171.stdout:0/232: mkdir d9/db/d59 0 2022-01-31T19:44:20.159 INFO:tasks.workunit.client.1.smithi171.stdout:2/282: creat d7/d9/d1a/f53 x:0 0 0 2022-01-31T19:44:20.160 INFO:tasks.workunit.client.1.smithi171.stdout:0/233: mknod d9/db/d22/d3a/c5a 0 2022-01-31T19:44:20.160 INFO:tasks.workunit.client.1.smithi171.stdout:0/234: write d9/db/f14 [1167686,77383] 0 2022-01-31T19:44:20.161 INFO:tasks.workunit.client.1.smithi171.stdout:2/283: mknod d7/d9/d1a/c54 0 2022-01-31T19:44:20.161 INFO:tasks.workunit.client.1.smithi171.stdout:2/284: dread - d7/f25 zero size 2022-01-31T19:44:20.162 INFO:tasks.workunit.client.1.smithi171.stdout:2/285: mkdir d7/d55 0 2022-01-31T19:44:20.176 INFO:tasks.workunit.client.1.smithi171.stdout:0/235: dwrite d9/f1e [0,4194304] 0 2022-01-31T19:44:20.177 INFO:tasks.workunit.client.1.smithi171.stdout:0/236: creat d9/db/d22/f5b x:0 0 0 2022-01-31T19:44:20.178 INFO:tasks.workunit.client.1.smithi171.stdout:0/237: rename d9/f17 to d9/f5c 0 2022-01-31T19:44:20.179 INFO:tasks.workunit.client.1.smithi171.stdout:0/238: mkdir d9/db/d22/d5d 0 2022-01-31T19:44:20.181 INFO:tasks.workunit.client.1.smithi171.stdout:0/239: read d9/f1f [3779398,11687] 0 2022-01-31T19:44:20.181 INFO:tasks.workunit.client.1.smithi171.stdout:0/240: dread d9/d24/f3c [0,4194304] 0 2022-01-31T19:44:20.182 INFO:tasks.workunit.client.1.smithi171.stdout:0/241: creat d9/d24/d4c/f5e x:0 0 0 2022-01-31T19:44:20.183 INFO:tasks.workunit.client.1.smithi171.stdout:0/242: write d9/db/d22/d3a/f42 [920958,13065] 0 2022-01-31T19:44:20.185 INFO:tasks.workunit.client.1.smithi171.stdout:2/286: dwrite d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:20.187 INFO:tasks.workunit.client.1.smithi171.stdout:2/287: unlink d7/f25 0 2022-01-31T19:44:20.188 INFO:tasks.workunit.client.1.smithi171.stdout:2/288: read d7/d9/f12 [1460074,116887] 0 2022-01-31T19:44:20.189 INFO:tasks.workunit.client.1.smithi171.stdout:2/289: write d7/d9/f4e [6883990,8122] 0 2022-01-31T19:44:20.190 INFO:tasks.workunit.client.1.smithi171.stdout:2/290: dread d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:20.209 INFO:tasks.workunit.client.1.smithi171.stdout:2/291: dwrite d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:20.219 INFO:tasks.workunit.client.1.smithi171.stdout:1/206: sync 2022-01-31T19:44:20.219 INFO:tasks.workunit.client.1.smithi171.stdout:9/156: sync 2022-01-31T19:44:20.221 INFO:tasks.workunit.client.1.smithi171.stdout:1/207: creat dd/df/d19/d1b/d1e/f44 x:0 0 0 2022-01-31T19:44:20.221 INFO:tasks.workunit.client.1.smithi171.stdout:1/208: creat dd/df/d17/f45 x:0 0 0 2022-01-31T19:44:20.222 INFO:tasks.workunit.client.1.smithi171.stdout:2/292: dwrite d7/d9/d1a/f53 [0,4194304] 0 2022-01-31T19:44:20.222 INFO:tasks.workunit.client.1.smithi171.stdout:2/293: stat d7/c50 0 2022-01-31T19:44:20.223 INFO:tasks.workunit.client.1.smithi171.stdout:9/157: symlink dd/df/d15/l33 0 2022-01-31T19:44:20.223 INFO:tasks.workunit.client.1.smithi171.stdout:9/158: fdatasync dd/df/f20 0 2022-01-31T19:44:20.223 INFO:tasks.workunit.client.1.smithi171.stdout:9/159: fsync dd/d1c/f23 0 2022-01-31T19:44:20.226 INFO:tasks.workunit.client.1.smithi171.stdout:2/294: unlink d7/d26/d3d/d40/f48 0 2022-01-31T19:44:20.228 INFO:tasks.workunit.client.1.smithi171.stdout:9/160: rename dd/df/d15/l33 to dd/d1c/l34 0 2022-01-31T19:44:20.229 INFO:tasks.workunit.client.1.smithi171.stdout:9/161: read dd/df/d15/f1b [294980,20569] 0 2022-01-31T19:44:20.229 INFO:tasks.workunit.client.1.smithi171.stdout:9/162: rename dd/df/d15/d1a to dd/df/d15/d1a/d35 22 2022-01-31T19:44:20.229 INFO:tasks.workunit.client.1.smithi171.stdout:2/295: creat d7/f56 x:0 0 0 2022-01-31T19:44:20.230 INFO:tasks.workunit.client.1.smithi171.stdout:9/163: creat dd/df/d15/f36 x:0 0 0 2022-01-31T19:44:20.231 INFO:tasks.workunit.client.1.smithi171.stdout:2/296: truncate d7/f3a 472996 0 2022-01-31T19:44:20.231 INFO:tasks.workunit.client.1.smithi171.stdout:2/297: creat d7/d26/d3d/d40/f57 x:0 0 0 2022-01-31T19:44:20.234 INFO:tasks.workunit.client.1.smithi171.stdout:2/298: truncate d7/d9/f44 7103505 0 2022-01-31T19:44:20.235 INFO:tasks.workunit.client.1.smithi171.stdout:1/209: dwrite dd/df/f12 [0,4194304] 0 2022-01-31T19:44:20.235 INFO:tasks.workunit.client.1.smithi171.stdout:1/210: fsync dd/df/f3a 0 2022-01-31T19:44:20.236 INFO:tasks.workunit.client.1.smithi171.stdout:2/299: mknod d7/d26/d3d/c58 0 2022-01-31T19:44:20.236 INFO:tasks.workunit.client.1.smithi171.stdout:2/300: write d7/d9/d1a/f32 [2442119,55519] 0 2022-01-31T19:44:20.237 INFO:tasks.workunit.client.1.smithi171.stdout:2/301: stat d7/dd/f1b 0 2022-01-31T19:44:20.238 INFO:tasks.workunit.client.1.smithi171.stdout:2/302: read d7/d26/d3d/f42 [354545,6684] 0 2022-01-31T19:44:20.238 INFO:tasks.workunit.client.1.smithi171.stdout:2/303: fsync d7/d26/d3d/f42 0 2022-01-31T19:44:20.238 INFO:tasks.workunit.client.1.smithi171.stdout:2/304: chown d7/d9/d1a/d45 13 1 2022-01-31T19:44:20.239 INFO:tasks.workunit.client.1.smithi171.stdout:2/305: chown d7/d9 126 1 2022-01-31T19:44:20.239 INFO:tasks.workunit.client.1.smithi171.stdout:1/211: creat dd/df/d19/d1b/d1e/f46 x:0 0 0 2022-01-31T19:44:20.239 INFO:tasks.workunit.client.1.smithi171.stdout:1/212: creat dd/df/d19/f47 x:0 0 0 2022-01-31T19:44:20.240 INFO:tasks.workunit.client.1.smithi171.stdout:9/164: getdents dd/d1c 0 2022-01-31T19:44:20.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/165: chown fc 444 1 2022-01-31T19:44:20.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/166: chown dd/df/d15/f16 0 1 2022-01-31T19:44:20.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/167: read - dd/df/d15/d1a/f27 zero size 2022-01-31T19:44:20.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/168: fdatasync dd/df/d15/f36 0 2022-01-31T19:44:20.242 INFO:tasks.workunit.client.1.smithi171.stdout:9/169: read - dd/f2e zero size 2022-01-31T19:44:20.242 INFO:tasks.workunit.client.1.smithi171.stdout:2/306: link d7/d26/d3d/f42 d7/d55/f59 0 2022-01-31T19:44:20.245 INFO:tasks.workunit.client.1.smithi171.stdout:1/213: mknod dd/d23/c48 0 2022-01-31T19:44:20.246 INFO:tasks.workunit.client.1.smithi171.stdout:1/214: stat dd/df/d17/f26 0 2022-01-31T19:44:20.249 INFO:tasks.workunit.client.1.smithi171.stdout:9/170: dread dd/df/f12 [0,4194304] 0 2022-01-31T19:44:20.249 INFO:tasks.workunit.client.1.smithi171.stdout:1/215: creat dd/df/d19/d1b/f49 x:0 0 0 2022-01-31T19:44:20.249 INFO:tasks.workunit.client.1.smithi171.stdout:1/216: readlink dd/d23/l3c 0 2022-01-31T19:44:20.249 INFO:tasks.workunit.client.1.smithi171.stdout:1/217: dread - dd/d23/f41 zero size 2022-01-31T19:44:20.250 INFO:tasks.workunit.client.1.smithi171.stdout:1/218: dread - dd/df/f38 zero size 2022-01-31T19:44:20.251 INFO:tasks.workunit.client.1.smithi171.stdout:9/171: creat dd/f37 x:0 0 0 2022-01-31T19:44:20.251 INFO:tasks.workunit.client.1.smithi171.stdout:9/172: write dd/f11 [5188357,53153] 0 2022-01-31T19:44:20.251 INFO:tasks.workunit.client.1.smithi171.stdout:9/173: stat dd/df/c24 0 2022-01-31T19:44:20.251 INFO:tasks.workunit.client.1.smithi171.stdout:9/174: write dd/d1c/f1f [5051845,112285] 0 2022-01-31T19:44:20.255 INFO:tasks.workunit.client.1.smithi171.stdout:9/175: mkdir dd/df/d15/d38 0 2022-01-31T19:44:20.256 INFO:tasks.workunit.client.1.smithi171.stdout:9/176: symlink dd/d1c/l39 0 2022-01-31T19:44:20.256 INFO:tasks.workunit.client.1.smithi171.stdout:9/177: readlink dd/l18 0 2022-01-31T19:44:20.259 INFO:tasks.workunit.client.1.smithi171.stdout:2/307: dwrite d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:20.259 INFO:tasks.workunit.client.1.smithi171.stdout:2/308: write d7/dd/f1e [1109938,32885] 0 2022-01-31T19:44:20.264 INFO:tasks.workunit.client.1.smithi171.stdout:9/178: dread dd/df/f17 [0,4194304] 0 2022-01-31T19:44:20.266 INFO:tasks.workunit.client.1.smithi171.stdout:1/219: rmdir dd/df 39 2022-01-31T19:44:20.278 INFO:tasks.workunit.client.1.smithi171.stdout:9/179: dwrite fc [0,4194304] 0 2022-01-31T19:44:20.281 INFO:tasks.workunit.client.1.smithi171.stdout:9/180: link dd/df/f12 dd/df/d15/f3a 0 2022-01-31T19:44:20.281 INFO:tasks.workunit.client.1.smithi171.stdout:9/181: write dd/df/d15/f2f [562285,102083] 0 2022-01-31T19:44:20.281 INFO:tasks.workunit.client.1.smithi171.stdout:9/182: chown fc 4341 1 2022-01-31T19:44:20.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/183: chown l3 518087535 1 2022-01-31T19:44:20.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/184: write dd/df/f32 [811887,24937] 0 2022-01-31T19:44:20.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/185: readlink dd/df/d15/l1e 0 2022-01-31T19:44:20.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/186: rename dd to dd/df/d15/d38/d3b 22 2022-01-31T19:44:20.283 INFO:tasks.workunit.client.1.smithi171.stdout:9/187: stat dd/df/d15/f22 0 2022-01-31T19:44:20.284 INFO:tasks.workunit.client.1.smithi171.stdout:9/188: rename dd/d1c/f2a to dd/f3c 0 2022-01-31T19:44:20.284 INFO:tasks.workunit.client.1.smithi171.stdout:9/189: creat dd/d1c/f3d x:0 0 0 2022-01-31T19:44:20.284 INFO:tasks.workunit.client.1.smithi171.stdout:9/190: write dd/df/d15/f26 [624805,89461] 0 2022-01-31T19:44:20.285 INFO:tasks.workunit.client.1.smithi171.stdout:9/191: write dd/df/f20 [1519601,124972] 0 2022-01-31T19:44:20.285 INFO:tasks.workunit.client.1.smithi171.stdout:9/192: mknod dd/d1c/c3e 0 2022-01-31T19:44:20.286 INFO:tasks.workunit.client.1.smithi171.stdout:9/193: mkdir dd/df/d3f 0 2022-01-31T19:44:20.287 INFO:tasks.workunit.client.1.smithi171.stdout:9/194: read dd/f11 [883729,57594] 0 2022-01-31T19:44:20.288 INFO:tasks.workunit.client.1.smithi171.stdout:9/195: mknod dd/df/d15/d38/c40 0 2022-01-31T19:44:20.288 INFO:tasks.workunit.client.1.smithi171.stdout:9/196: dread - dd/d1c/f29 zero size 2022-01-31T19:44:20.288 INFO:tasks.workunit.client.1.smithi171.stdout:9/197: readlink dd/l18 0 2022-01-31T19:44:20.289 INFO:tasks.workunit.client.1.smithi171.stdout:9/198: mkdir dd/df/d15/d38/d41 0 2022-01-31T19:44:20.289 INFO:tasks.workunit.client.1.smithi171.stdout:9/199: truncate dd/df/d15/f2f 1408362 0 2022-01-31T19:44:20.290 INFO:tasks.workunit.client.1.smithi171.stdout:9/200: write fc [3316236,16857] 0 2022-01-31T19:44:20.290 INFO:tasks.workunit.client.1.smithi171.stdout:9/201: link l3 dd/df/d15/d38/d41/l42 0 2022-01-31T19:44:20.304 INFO:tasks.workunit.client.1.smithi171.stdout:9/202: dwrite dd/df/f12 [0,4194304] 0 2022-01-31T19:44:20.304 INFO:tasks.workunit.client.1.smithi171.stdout:9/203: chown dd/c31 445058 1 2022-01-31T19:44:20.305 INFO:tasks.workunit.client.1.smithi171.stdout:9/204: getdents dd/df/d15/d1a 0 2022-01-31T19:44:20.319 INFO:tasks.workunit.client.1.smithi171.stdout:9/205: dwrite dd/f2e [0,4194304] 0 2022-01-31T19:44:20.320 INFO:tasks.workunit.client.1.smithi171.stdout:9/206: stat dd/df/c24 0 2022-01-31T19:44:20.335 INFO:tasks.workunit.client.1.smithi171.stdout:9/207: dwrite dd/df/d15/f36 [0,4194304] 0 2022-01-31T19:44:20.336 INFO:tasks.workunit.client.1.smithi171.stdout:9/208: write dd/df/d15/d1a/f2b [403041,64780] 0 2022-01-31T19:44:20.336 INFO:tasks.workunit.client.1.smithi171.stdout:9/209: stat dd/df/d15/d38/d41 0 2022-01-31T19:44:20.336 INFO:tasks.workunit.client.1.smithi171.stdout:9/210: chown dd/df/f20 97 1 2022-01-31T19:44:20.337 INFO:tasks.workunit.client.1.smithi171.stdout:9/211: write dd/df/d15/f3a [2205328,86613] 0 2022-01-31T19:44:20.338 INFO:tasks.workunit.client.1.smithi171.stdout:9/212: dread dd/df/d15/f22 [0,4194304] 0 2022-01-31T19:44:20.344 INFO:tasks.workunit.client.1.smithi171.stdout:9/213: dread dd/df/d15/f36 [0,4194304] 0 2022-01-31T19:44:20.345 INFO:tasks.workunit.client.1.smithi171.stdout:9/214: read dd/df/d15/f36 [1806624,59763] 0 2022-01-31T19:44:20.346 INFO:tasks.workunit.client.1.smithi171.stdout:9/215: symlink dd/df/d15/d1a/l43 0 2022-01-31T19:44:20.347 INFO:tasks.workunit.client.1.smithi171.stdout:9/216: symlink dd/df/d3f/l44 0 2022-01-31T19:44:20.347 INFO:tasks.workunit.client.1.smithi171.stdout:9/217: chown dd/l30 91255 1 2022-01-31T19:44:20.350 INFO:tasks.workunit.client.1.smithi171.stdout:9/218: rmdir dd 39 2022-01-31T19:44:20.351 INFO:tasks.workunit.client.1.smithi171.stdout:9/219: symlink dd/df/d15/l45 0 2022-01-31T19:44:20.352 INFO:tasks.workunit.client.1.smithi171.stdout:9/220: truncate dd/f2e 1663528 0 2022-01-31T19:44:20.352 INFO:tasks.workunit.client.1.smithi171.stdout:9/221: write dd/d1c/f3d [145106,40998] 0 2022-01-31T19:44:20.353 INFO:tasks.workunit.client.1.smithi171.stdout:9/222: symlink dd/l46 0 2022-01-31T19:44:20.354 INFO:tasks.workunit.client.1.smithi171.stdout:9/223: symlink dd/df/d15/d38/d41/l47 0 2022-01-31T19:44:20.357 INFO:tasks.workunit.client.1.smithi171.stdout:9/224: dread dd/fe [0,4194304] 0 2022-01-31T19:44:20.357 INFO:tasks.workunit.client.1.smithi171.stdout:9/225: chown dd/c31 105 1 2022-01-31T19:44:20.358 INFO:tasks.workunit.client.1.smithi171.stdout:9/226: readlink l8 0 2022-01-31T19:44:20.359 INFO:tasks.workunit.client.1.smithi171.stdout:9/227: write dd/df/f17 [3379507,96508] 0 2022-01-31T19:44:20.360 INFO:tasks.workunit.client.1.smithi171.stdout:9/228: write dd/df/d15/f36 [2416417,130223] 0 2022-01-31T19:44:20.361 INFO:tasks.workunit.client.1.smithi171.stdout:9/229: write dd/df/d15/f36 [1573153,109152] 0 2022-01-31T19:44:20.362 INFO:tasks.workunit.client.1.smithi171.stdout:9/230: dread dd/df/d15/f16 [0,4194304] 0 2022-01-31T19:44:20.363 INFO:tasks.workunit.client.1.smithi171.stdout:9/231: truncate f4 8419936 0 2022-01-31T19:44:20.364 INFO:tasks.workunit.client.1.smithi171.stdout:9/232: read dd/fe [71727,19401] 0 2022-01-31T19:44:20.381 INFO:tasks.workunit.client.1.smithi171.stdout:9/233: dwrite dd/df/d15/f36 [0,4194304] 0 2022-01-31T19:44:20.382 INFO:tasks.workunit.client.1.smithi171.stdout:9/234: mknod dd/c48 0 2022-01-31T19:44:20.383 INFO:tasks.workunit.client.1.smithi171.stdout:9/235: chown dd/df/f25 7968428 1 2022-01-31T19:44:20.383 INFO:tasks.workunit.client.1.smithi171.stdout:9/236: chown dd/df/d15/f2c 1 1 2022-01-31T19:44:20.384 INFO:tasks.workunit.client.1.smithi171.stdout:9/237: creat dd/d1c/f49 x:0 0 0 2022-01-31T19:44:20.385 INFO:tasks.workunit.client.1.smithi171.stdout:9/238: unlink dd/df/d3f/l44 0 2022-01-31T19:44:20.386 INFO:tasks.workunit.client.1.smithi171.stdout:9/239: creat dd/df/d15/f4a x:0 0 0 2022-01-31T19:44:20.387 INFO:tasks.workunit.client.1.smithi171.stdout:9/240: chown dd/df/d15/d1a/c2d 22 1 2022-01-31T19:44:20.388 INFO:tasks.workunit.client.1.smithi171.stdout:9/241: creat dd/df/d15/d1a/f4b x:0 0 0 2022-01-31T19:44:20.390 INFO:tasks.workunit.client.1.smithi171.stdout:3/194: sync 2022-01-31T19:44:20.391 INFO:tasks.workunit.client.1.smithi171.stdout:4/216: sync 2022-01-31T19:44:20.393 INFO:tasks.workunit.client.1.smithi171.stdout:9/242: getdents dd/df/d3f 0 2022-01-31T19:44:20.395 INFO:tasks.workunit.client.1.smithi171.stdout:9/243: creat dd/df/d3f/f4c x:0 0 0 2022-01-31T19:44:20.406 INFO:tasks.workunit.client.1.smithi171.stdout:3/195: dread d4/dd/d27/d28/f2d [0,4194304] 0 2022-01-31T19:44:20.407 INFO:tasks.workunit.client.1.smithi171.stdout:3/196: creat d4/f37 x:0 0 0 2022-01-31T19:44:20.408 INFO:tasks.workunit.client.1.smithi171.stdout:3/197: creat d4/dd/d27/d28/d35/f38 x:0 0 0 2022-01-31T19:44:20.408 INFO:tasks.workunit.client.1.smithi171.stdout:3/198: truncate d4/dd/d1a/f2b 1019282 0 2022-01-31T19:44:20.415 INFO:tasks.workunit.client.1.smithi171.stdout:9/244: dwrite dd/f37 [0,4194304] 0 2022-01-31T19:44:20.430 INFO:tasks.workunit.client.1.smithi171.stdout:4/217: dwrite d2/d3/d5/dc/f42 [0,4194304] 0 2022-01-31T19:44:20.430 INFO:tasks.workunit.client.1.smithi171.stdout:4/218: write d2/d3/d5/d11/d12/d15/f3e [4798675,107661] 0 2022-01-31T19:44:20.431 INFO:tasks.workunit.client.1.smithi171.stdout:4/219: readlink d2/d3/d5/d11/d12/l40 0 2022-01-31T19:44:20.433 INFO:tasks.workunit.client.1.smithi171.stdout:3/199: dwrite d4/d21/f31 [0,4194304] 0 2022-01-31T19:44:20.433 INFO:tasks.workunit.client.1.smithi171.stdout:3/200: chown d4/d21/c34 20977 1 2022-01-31T19:44:20.433 INFO:tasks.workunit.client.1.smithi171.stdout:3/201: readlink d4/dd/l24 0 2022-01-31T19:44:20.434 INFO:tasks.workunit.client.1.smithi171.stdout:4/220: truncate d2/d3/f1c 540096 0 2022-01-31T19:44:20.434 INFO:tasks.workunit.client.1.smithi171.stdout:9/245: dwrite dd/df/d15/d1a/f4b [0,4194304] 0 2022-01-31T19:44:20.435 INFO:tasks.workunit.client.1.smithi171.stdout:9/246: write dd/df/d15/f1b [483016,3594] 0 2022-01-31T19:44:20.436 INFO:tasks.workunit.client.1.smithi171.stdout:3/202: mkdir d4/dd/d39 0 2022-01-31T19:44:20.437 INFO:tasks.workunit.client.1.smithi171.stdout:4/221: creat d2/d3/f44 x:0 0 0 2022-01-31T19:44:20.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/247: mkdir dd/df/d15/d38/d4d 0 2022-01-31T19:44:20.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/248: chown dd/c31 336443283 1 2022-01-31T19:44:20.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/249: fsync dd/f37 0 2022-01-31T19:44:20.441 INFO:tasks.workunit.client.1.smithi171.stdout:9/250: chown dd/df/d15/d38/d41/l42 102 1 2022-01-31T19:44:20.442 INFO:tasks.workunit.client.1.smithi171.stdout:9/251: dread dd/d1c/f23 [0,4194304] 0 2022-01-31T19:44:20.442 INFO:tasks.workunit.client.1.smithi171.stdout:9/252: chown dd/l19 45341 1 2022-01-31T19:44:20.442 INFO:tasks.workunit.client.1.smithi171.stdout:3/203: mkdir d4/d21/d3a 0 2022-01-31T19:44:20.442 INFO:tasks.workunit.client.1.smithi171.stdout:3/204: stat d4/dd/f20 0 2022-01-31T19:44:20.443 INFO:tasks.workunit.client.1.smithi171.stdout:4/222: creat d2/d3/d5/d11/d12/f45 x:0 0 0 2022-01-31T19:44:20.443 INFO:tasks.workunit.client.1.smithi171.stdout:4/223: dread - d2/f43 zero size 2022-01-31T19:44:20.444 INFO:tasks.workunit.client.1.smithi171.stdout:3/205: truncate d4/dd/d27/d28/f2d 2273267 0 2022-01-31T19:44:20.453 INFO:tasks.workunit.client.1.smithi171.stdout:4/224: dread d2/d3/d5/dc/f26 [0,4194304] 0 2022-01-31T19:44:20.455 INFO:tasks.workunit.client.1.smithi171.stdout:4/225: read d2/d3/d5/f1a [2395251,80165] 0 2022-01-31T19:44:20.455 INFO:tasks.workunit.client.1.smithi171.stdout:4/226: fsync d2/d3/d5/f1a 0 2022-01-31T19:44:20.457 INFO:tasks.workunit.client.1.smithi171.stdout:4/227: creat d2/d3/d5/f46 x:0 0 0 2022-01-31T19:44:20.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/228: read d2/d3/d5/dc/f42 [318245,25286] 0 2022-01-31T19:44:20.463 INFO:tasks.workunit.client.1.smithi171.stdout:9/253: dwrite dd/d1c/f3d [0,4194304] 0 2022-01-31T19:44:20.463 INFO:tasks.workunit.client.1.smithi171.stdout:9/254: chown dd/df/d15/d38/d41/l42 14873 1 2022-01-31T19:44:20.463 INFO:tasks.workunit.client.1.smithi171.stdout:9/255: write dd/f2e [2217213,79027] 0 2022-01-31T19:44:20.468 INFO:tasks.workunit.client.1.smithi171.stdout:3/206: dwrite d4/d21/f2a [0,4194304] 0 2022-01-31T19:44:20.469 INFO:tasks.workunit.client.1.smithi171.stdout:9/256: link dd/df/d15/d1a/f27 dd/df/d15/f4e 0 2022-01-31T19:44:20.471 INFO:tasks.workunit.client.1.smithi171.stdout:9/257: rmdir dd/d1c 39 2022-01-31T19:44:20.472 INFO:tasks.workunit.client.1.smithi171.stdout:9/258: rmdir dd/df/d15/d38/d41 39 2022-01-31T19:44:20.473 INFO:tasks.workunit.client.1.smithi171.stdout:3/207: dread d4/dd/d1a/f1b [0,4194304] 0 2022-01-31T19:44:20.473 INFO:tasks.workunit.client.1.smithi171.stdout:9/259: symlink dd/d1c/l4f 0 2022-01-31T19:44:20.475 INFO:tasks.workunit.client.1.smithi171.stdout:9/260: creat dd/df/d15/d38/d41/f50 x:0 0 0 2022-01-31T19:44:20.475 INFO:tasks.workunit.client.1.smithi171.stdout:9/261: fsync dd/df/f20 0 2022-01-31T19:44:20.476 INFO:tasks.workunit.client.1.smithi171.stdout:3/208: write d4/d21/f31 [1461585,47994] 0 2022-01-31T19:44:20.477 INFO:tasks.workunit.client.1.smithi171.stdout:3/209: creat d4/d21/d3a/f3b x:0 0 0 2022-01-31T19:44:20.479 INFO:tasks.workunit.client.1.smithi171.stdout:3/210: rename d4/f37 to d4/d21/d3a/f3c 0 2022-01-31T19:44:20.480 INFO:tasks.workunit.client.1.smithi171.stdout:4/229: dwrite d2/f13 [0,4194304] 0 2022-01-31T19:44:20.481 INFO:tasks.workunit.client.1.smithi171.stdout:3/211: mknod d4/dd/d27/d2e/c3d 0 2022-01-31T19:44:20.481 INFO:tasks.workunit.client.1.smithi171.stdout:3/212: readlink d4/l2c 0 2022-01-31T19:44:20.487 INFO:tasks.workunit.client.1.smithi171.stdout:3/213: mknod d4/dd/d39/c3e 0 2022-01-31T19:44:20.489 INFO:tasks.workunit.client.1.smithi171.stdout:4/230: rmdir d2/d3/d5 39 2022-01-31T19:44:20.491 INFO:tasks.workunit.client.1.smithi171.stdout:3/214: truncate d4/dd/f19 2698589 0 2022-01-31T19:44:20.493 INFO:tasks.workunit.client.1.smithi171.stdout:4/231: rename d2/f43 to d2/d3/d5/d41/f47 0 2022-01-31T19:44:20.493 INFO:tasks.workunit.client.1.smithi171.stdout:4/232: read - d2/d3/d5/d11/d12/d15/f3a zero size 2022-01-31T19:44:20.496 INFO:tasks.workunit.client.1.smithi171.stdout:4/233: dread d2/d3/d5/f38 [0,4194304] 0 2022-01-31T19:44:20.498 INFO:tasks.workunit.client.1.smithi171.stdout:4/234: creat d2/d3/d5/d11/f48 x:0 0 0 2022-01-31T19:44:20.499 INFO:tasks.workunit.client.1.smithi171.stdout:9/262: dwrite dd/f2e [0,4194304] 0 2022-01-31T19:44:20.500 INFO:tasks.workunit.client.1.smithi171.stdout:9/263: fsync dd/df/d15/f26 0 2022-01-31T19:44:20.500 INFO:tasks.workunit.client.1.smithi171.stdout:9/264: fsync dd/df/d15/d1a/f4b 0 2022-01-31T19:44:20.509 INFO:tasks.workunit.client.1.smithi171.stdout:4/235: dread d2/d3/d5/dc/f42 [0,4194304] 0 2022-01-31T19:44:20.510 INFO:tasks.workunit.client.1.smithi171.stdout:4/236: mknod d2/d3/c49 0 2022-01-31T19:44:20.510 INFO:tasks.workunit.client.1.smithi171.stdout:4/237: stat d2/d3/d5/d11/d12/f2a 0 2022-01-31T19:44:20.511 INFO:tasks.workunit.client.1.smithi171.stdout:8/192: sync 2022-01-31T19:44:20.511 INFO:tasks.workunit.client.1.smithi171.stdout:6/207: sync 2022-01-31T19:44:20.511 INFO:tasks.workunit.client.1.smithi171.stdout:7/169: sync 2022-01-31T19:44:20.512 INFO:tasks.workunit.client.1.smithi171.stdout:7/170: creat d1/d1b/f3a x:0 0 0 2022-01-31T19:44:20.514 INFO:tasks.workunit.client.1.smithi171.stdout:6/208: creat dc/d19/d29/d27/f3a x:0 0 0 2022-01-31T19:44:20.514 INFO:tasks.workunit.client.1.smithi171.stdout:6/209: dread - dc/d19/d29/d27/f3a zero size 2022-01-31T19:44:20.514 INFO:tasks.workunit.client.1.smithi171.stdout:6/210: fsync dc/d21/f2e 0 2022-01-31T19:44:20.514 INFO:tasks.workunit.client.1.smithi171.stdout:6/211: fsync dc/d19/f1a 0 2022-01-31T19:44:20.515 INFO:tasks.workunit.client.1.smithi171.stdout:4/238: write d2/d3/d5/dc/f26 [3518356,32677] 0 2022-01-31T19:44:20.515 INFO:tasks.workunit.client.1.smithi171.stdout:7/171: creat d1/d1e/d2a/f3b x:0 0 0 2022-01-31T19:44:20.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/239: write d2/d3/d5/d11/f27 [2349323,12726] 0 2022-01-31T19:44:20.516 INFO:tasks.workunit.client.1.smithi171.stdout:6/212: mkdir dc/d19/d29/d27/d37/d3b 0 2022-01-31T19:44:20.517 INFO:tasks.workunit.client.1.smithi171.stdout:8/193: dread df/d18/d19/f22 [0,4194304] 0 2022-01-31T19:44:20.517 INFO:tasks.workunit.client.1.smithi171.stdout:8/194: dread - df/d2a/f40 zero size 2022-01-31T19:44:20.518 INFO:tasks.workunit.client.1.smithi171.stdout:7/172: mkdir d1/d1e/d2a/d29/d31/d30/d3c 0 2022-01-31T19:44:20.518 INFO:tasks.workunit.client.1.smithi171.stdout:7/173: write d1/d1b/fc [5161895,76707] 0 2022-01-31T19:44:20.518 INFO:tasks.workunit.client.1.smithi171.stdout:3/215: dwrite d4/d21/d3a/f3c [0,4194304] 0 2022-01-31T19:44:20.519 INFO:tasks.workunit.client.1.smithi171.stdout:4/240: mknod d2/d3/d5/c4a 0 2022-01-31T19:44:20.519 INFO:tasks.workunit.client.1.smithi171.stdout:9/265: dwrite dd/df/d3f/f4c [0,4194304] 0 2022-01-31T19:44:20.520 INFO:tasks.workunit.client.1.smithi171.stdout:8/195: mknod df/d1a/c4a 0 2022-01-31T19:44:20.522 INFO:tasks.workunit.client.1.smithi171.stdout:5/217: sync 2022-01-31T19:44:20.522 INFO:tasks.workunit.client.1.smithi171.stdout:1/220: sync 2022-01-31T19:44:20.523 INFO:tasks.workunit.client.1.smithi171.stdout:2/309: sync 2022-01-31T19:44:20.523 INFO:tasks.workunit.client.1.smithi171.stdout:5/218: dread - d5/f3b zero size 2022-01-31T19:44:20.523 INFO:tasks.workunit.client.1.smithi171.stdout:0/243: sync 2022-01-31T19:44:20.523 INFO:tasks.workunit.client.1.smithi171.stdout:0/244: chown d9/d24/d4c/d35 32512668 1 2022-01-31T19:44:20.524 INFO:tasks.workunit.client.1.smithi171.stdout:7/174: truncate d1/f5 1934681 0 2022-01-31T19:44:20.527 INFO:tasks.workunit.client.1.smithi171.stdout:3/216: truncate d4/dd/f20 1385788 0 2022-01-31T19:44:20.529 INFO:tasks.workunit.client.1.smithi171.stdout:8/196: write df/d1a/f3b [7262665,58173] 0 2022-01-31T19:44:20.533 INFO:tasks.workunit.client.1.smithi171.stdout:2/310: rename l5 to d7/d26/l5a 0 2022-01-31T19:44:20.533 INFO:tasks.workunit.client.1.smithi171.stdout:2/311: write d7/f3a [540101,127620] 0 2022-01-31T19:44:20.534 INFO:tasks.workunit.client.1.smithi171.stdout:5/219: creat d5/d6/d15/d29/f4c x:0 0 0 2022-01-31T19:44:20.536 INFO:tasks.workunit.client.1.smithi171.stdout:0/245: creat d9/db/d22/d5d/f5f x:0 0 0 2022-01-31T19:44:20.537 INFO:tasks.workunit.client.1.smithi171.stdout:1/221: dread fa [0,4194304] 0 2022-01-31T19:44:20.537 INFO:tasks.workunit.client.1.smithi171.stdout:7/175: creat d1/d1e/f3d x:0 0 0 2022-01-31T19:44:20.539 INFO:tasks.workunit.client.1.smithi171.stdout:5/220: write d5/d6/d15/f3f [6367935,14151] 0 2022-01-31T19:44:20.540 INFO:tasks.workunit.client.1.smithi171.stdout:5/221: dread - d5/d6/d15/d29/f39 zero size 2022-01-31T19:44:20.540 INFO:tasks.workunit.client.1.smithi171.stdout:5/222: creat d5/d6/d15/d29/f4d x:0 0 0 2022-01-31T19:44:20.542 INFO:tasks.workunit.client.1.smithi171.stdout:8/197: symlink df/d18/d19/d21/l4b 0 2022-01-31T19:44:20.544 INFO:tasks.workunit.client.1.smithi171.stdout:4/241: rmdir d2/d3/d5/d11 39 2022-01-31T19:44:20.545 INFO:tasks.workunit.client.1.smithi171.stdout:8/198: dread df/d18/d19/f1b [0,4194304] 0 2022-01-31T19:44:20.546 INFO:tasks.workunit.client.1.smithi171.stdout:2/312: mknod d7/c5b 0 2022-01-31T19:44:20.547 INFO:tasks.workunit.client.1.smithi171.stdout:6/213: dwrite dc/d19/d29/d2f/f38 [0,4194304] 0 2022-01-31T19:44:20.547 INFO:tasks.workunit.client.1.smithi171.stdout:9/266: dwrite dd/df/d15/f22 [0,4194304] 0 2022-01-31T19:44:20.550 INFO:tasks.workunit.client.1.smithi171.stdout:1/222: rename dd/df/f10 to dd/df/d19/d1b/d1e/f4a 0 2022-01-31T19:44:20.550 INFO:tasks.workunit.client.1.smithi171.stdout:1/223: stat dd/df/d19/d1b/d1e/d35 0 2022-01-31T19:44:20.550 INFO:tasks.workunit.client.1.smithi171.stdout:1/224: read - dd/df/f38 zero size 2022-01-31T19:44:20.550 INFO:tasks.workunit.client.1.smithi171.stdout:7/176: symlink d1/d1b/l3e 0 2022-01-31T19:44:20.551 INFO:tasks.workunit.client.1.smithi171.stdout:7/177: dread - d1/d24/f18 zero size 2022-01-31T19:44:20.553 INFO:tasks.workunit.client.1.smithi171.stdout:3/217: rmdir d4/d21 39 2022-01-31T19:44:20.554 INFO:tasks.workunit.client.1.smithi171.stdout:1/225: dread fa [0,4194304] 0 2022-01-31T19:44:20.556 INFO:tasks.workunit.client.1.smithi171.stdout:1/226: write dd/d23/f29 [590213,62546] 0 2022-01-31T19:44:20.556 INFO:tasks.workunit.client.1.smithi171.stdout:4/242: rmdir d2/d3/d5/d41 39 2022-01-31T19:44:20.557 INFO:tasks.workunit.client.1.smithi171.stdout:4/243: read d2/d3/d5/d11/d12/d15/f24 [1809438,18658] 0 2022-01-31T19:44:20.560 INFO:tasks.workunit.client.1.smithi171.stdout:8/199: unlink df/c11 0 2022-01-31T19:44:20.561 INFO:tasks.workunit.client.1.smithi171.stdout:2/313: dread d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:20.561 INFO:tasks.workunit.client.1.smithi171.stdout:2/314: creat d7/d26/d3d/d40/f5c x:0 0 0 2022-01-31T19:44:20.562 INFO:tasks.workunit.client.1.smithi171.stdout:0/246: getdents d9/db/d22/d3a 0 2022-01-31T19:44:20.562 INFO:tasks.workunit.client.1.smithi171.stdout:9/267: dread dd/df/f20 [0,4194304] 0 2022-01-31T19:44:20.563 INFO:tasks.workunit.client.1.smithi171.stdout:9/268: dread - dd/df/d15/f2c zero size 2022-01-31T19:44:20.563 INFO:tasks.workunit.client.1.smithi171.stdout:9/269: chown dd/df/f12 51 1 2022-01-31T19:44:20.564 INFO:tasks.workunit.client.1.smithi171.stdout:9/270: write dd/df/d15/f22 [1293861,47297] 0 2022-01-31T19:44:20.564 INFO:tasks.workunit.client.1.smithi171.stdout:6/214: mkdir dc/d19/d30/d3c 0 2022-01-31T19:44:20.564 INFO:tasks.workunit.client.1.smithi171.stdout:6/215: dread - dc/d19/d29/d27/f3a zero size 2022-01-31T19:44:20.571 INFO:tasks.workunit.client.1.smithi171.stdout:7/178: unlink d1/d1e/d2a/d29/d31/cb 0 2022-01-31T19:44:20.571 INFO:tasks.workunit.client.1.smithi171.stdout:7/179: creat d1/d1e/d2a/d29/d31/d30/f3f x:0 0 0 2022-01-31T19:44:20.571 INFO:tasks.workunit.client.1.smithi171.stdout:3/218: creat d4/dd/d27/f3f x:0 0 0 2022-01-31T19:44:20.572 INFO:tasks.workunit.client.1.smithi171.stdout:1/227: getdents dd/df/d19/d1b/d1e 0 2022-01-31T19:44:20.572 INFO:tasks.workunit.client.1.smithi171.stdout:1/228: readlink dd/d23/l3c 0 2022-01-31T19:44:20.573 INFO:tasks.workunit.client.1.smithi171.stdout:4/244: mkdir d2/d3/d5/dc/d4b 0 2022-01-31T19:44:20.573 INFO:tasks.workunit.client.1.smithi171.stdout:4/245: read - d2/d3/f28 zero size 2022-01-31T19:44:20.574 INFO:tasks.workunit.client.1.smithi171.stdout:4/246: chown d2/d3/d5/dc 3018 1 2022-01-31T19:44:20.574 INFO:tasks.workunit.client.1.smithi171.stdout:8/200: symlink df/d2a/l4c 0 2022-01-31T19:44:20.575 INFO:tasks.workunit.client.1.smithi171.stdout:2/315: mknod d7/c5d 0 2022-01-31T19:44:20.576 INFO:tasks.workunit.client.1.smithi171.stdout:9/271: creat dd/df/d3f/f51 x:0 0 0 2022-01-31T19:44:20.577 INFO:tasks.workunit.client.1.smithi171.stdout:7/180: symlink d1/d1e/d2a/d29/d31/d30/d3c/l40 0 2022-01-31T19:44:20.578 INFO:tasks.workunit.client.1.smithi171.stdout:7/181: dread - d1/d1b/f3a zero size 2022-01-31T19:44:20.578 INFO:tasks.workunit.client.1.smithi171.stdout:3/219: mkdir d4/dd/d40 0 2022-01-31T19:44:20.580 INFO:tasks.workunit.client.1.smithi171.stdout:4/247: unlink d2/d3/d5/d11/d12/d15/c35 0 2022-01-31T19:44:20.580 INFO:tasks.workunit.client.1.smithi171.stdout:4/248: write d2/f1e [8828267,64965] 0 2022-01-31T19:44:20.581 INFO:tasks.workunit.client.1.smithi171.stdout:4/249: truncate d2/d3/d5/d11/f48 221477 0 2022-01-31T19:44:20.581 INFO:tasks.workunit.client.1.smithi171.stdout:3/220: dread d4/dd/f1e [0,4194304] 0 2022-01-31T19:44:20.582 INFO:tasks.workunit.client.1.smithi171.stdout:9/272: creat dd/d1c/f52 x:0 0 0 2022-01-31T19:44:20.582 INFO:tasks.workunit.client.1.smithi171.stdout:9/273: truncate dd/df/d15/d1a/f2b 1049329 0 2022-01-31T19:44:20.585 INFO:tasks.workunit.client.1.smithi171.stdout:4/250: symlink d2/d3/d5/d11/d12/l4c 0 2022-01-31T19:44:20.585 INFO:tasks.workunit.client.1.smithi171.stdout:4/251: dread - d2/d3/d5/d11/d12/f2a zero size 2022-01-31T19:44:20.586 INFO:tasks.workunit.client.1.smithi171.stdout:8/201: dread df/d18/f42 [0,4194304] 0 2022-01-31T19:44:20.586 INFO:tasks.workunit.client.1.smithi171.stdout:2/316: mkdir d7/dd/d5e 0 2022-01-31T19:44:20.588 INFO:tasks.workunit.client.1.smithi171.stdout:4/252: symlink d2/l4d 0 2022-01-31T19:44:20.589 INFO:tasks.workunit.client.1.smithi171.stdout:4/253: write d2/d3/d5/d11/f48 [712324,78621] 0 2022-01-31T19:44:20.589 INFO:tasks.workunit.client.1.smithi171.stdout:4/254: dread - d2/d3/d5/f2e zero size 2022-01-31T19:44:20.591 INFO:tasks.workunit.client.1.smithi171.stdout:4/255: dread d2/d3/f1c [0,4194304] 0 2022-01-31T19:44:20.591 INFO:tasks.workunit.client.1.smithi171.stdout:9/274: mkdir dd/df/d15/d53 0 2022-01-31T19:44:20.592 INFO:tasks.workunit.client.1.smithi171.stdout:9/275: write dd/df/d15/f26 [1083847,126313] 0 2022-01-31T19:44:20.592 INFO:tasks.workunit.client.1.smithi171.stdout:9/276: read - dd/df/d15/d38/d41/f50 zero size 2022-01-31T19:44:20.596 INFO:tasks.workunit.client.1.smithi171.stdout:0/247: dwrite d9/db/f41 [0,4194304] 0 2022-01-31T19:44:20.596 INFO:tasks.workunit.client.1.smithi171.stdout:9/277: symlink dd/df/d15/d38/d4d/l54 0 2022-01-31T19:44:20.596 INFO:tasks.workunit.client.1.smithi171.stdout:9/278: rename dd to dd/df/d15/d38/d55 22 2022-01-31T19:44:20.598 INFO:tasks.workunit.client.1.smithi171.stdout:0/248: mknod d9/d24/c60 0 2022-01-31T19:44:20.599 INFO:tasks.workunit.client.1.smithi171.stdout:8/202: dread f2 [4194304,4194304] 0 2022-01-31T19:44:20.600 INFO:tasks.workunit.client.1.smithi171.stdout:9/279: rename dd/f2e to dd/df/d15/d1a/f56 0 2022-01-31T19:44:20.601 INFO:tasks.workunit.client.1.smithi171.stdout:8/203: creat df/d18/f4d x:0 0 0 2022-01-31T19:44:20.602 INFO:tasks.workunit.client.1.smithi171.stdout:9/280: mknod dd/df/d15/d38/d4d/c57 0 2022-01-31T19:44:20.603 INFO:tasks.workunit.client.1.smithi171.stdout:7/182: dwrite d1/d1e/d2a/d29/d31/f19 [0,4194304] 0 2022-01-31T19:44:20.603 INFO:tasks.workunit.client.1.smithi171.stdout:7/183: readlink d1/d1e/d2a/d29/l33 0 2022-01-31T19:44:20.604 INFO:tasks.workunit.client.1.smithi171.stdout:6/216: dwrite dc/d19/d29/d27/f3a [0,4194304] 0 2022-01-31T19:44:20.604 INFO:tasks.workunit.client.1.smithi171.stdout:9/281: mknod dd/df/d15/d53/c58 0 2022-01-31T19:44:20.604 INFO:tasks.workunit.client.1.smithi171.stdout:9/282: write dd/d1c/f49 [232997,78504] 0 2022-01-31T19:44:20.605 INFO:tasks.workunit.client.1.smithi171.stdout:7/184: symlink d1/d1b/l41 0 2022-01-31T19:44:20.605 INFO:tasks.workunit.client.1.smithi171.stdout:7/185: dread - d1/d24/f18 zero size 2022-01-31T19:44:20.606 INFO:tasks.workunit.client.1.smithi171.stdout:3/221: dwrite d4/fb [0,4194304] 0 2022-01-31T19:44:20.606 INFO:tasks.workunit.client.1.smithi171.stdout:3/222: fdatasync d4/dd/d27/d28/d35/f38 0 2022-01-31T19:44:20.610 INFO:tasks.workunit.client.1.smithi171.stdout:6/217: getdents dc/d19/d29/d27/d37 0 2022-01-31T19:44:20.613 INFO:tasks.workunit.client.1.smithi171.stdout:9/283: link dd/df/d15/l1e dd/d1c/l59 0 2022-01-31T19:44:20.614 INFO:tasks.workunit.client.1.smithi171.stdout:7/186: symlink d1/d26/l42 0 2022-01-31T19:44:20.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/187: readlink d1/d24/l27 0 2022-01-31T19:44:20.617 INFO:tasks.workunit.client.1.smithi171.stdout:4/256: getdents d2/d3/d5/d11/d12 0 2022-01-31T19:44:20.619 INFO:tasks.workunit.client.1.smithi171.stdout:2/317: rmdir d7 39 2022-01-31T19:44:20.619 INFO:tasks.workunit.client.1.smithi171.stdout:2/318: stat d7/d9/f2c 0 2022-01-31T19:44:20.619 INFO:tasks.workunit.client.1.smithi171.stdout:7/188: dread d1/f16 [0,4194304] 0 2022-01-31T19:44:20.620 INFO:tasks.workunit.client.1.smithi171.stdout:3/223: link d4/dd/cf d4/d21/c41 0 2022-01-31T19:44:20.620 INFO:tasks.workunit.client.1.smithi171.stdout:3/224: fsync d4/d21/d3a/f3c 0 2022-01-31T19:44:20.623 INFO:tasks.workunit.client.1.smithi171.stdout:9/284: symlink dd/df/d15/d38/l5a 0 2022-01-31T19:44:20.624 INFO:tasks.workunit.client.1.smithi171.stdout:4/257: symlink d2/d3/d5/d11/d12/d15/l4e 0 2022-01-31T19:44:20.626 INFO:tasks.workunit.client.1.smithi171.stdout:0/249: getdents d9/d24 0 2022-01-31T19:44:20.626 INFO:tasks.workunit.client.1.smithi171.stdout:0/250: fsync d9/db/f14 0 2022-01-31T19:44:20.627 INFO:tasks.workunit.client.1.smithi171.stdout:7/189: stat d1/d24/c23 0 2022-01-31T19:44:20.627 INFO:tasks.workunit.client.1.smithi171.stdout:6/218: dwrite f2 [0,4194304] 0 2022-01-31T19:44:20.627 INFO:tasks.workunit.client.1.smithi171.stdout:6/219: write dc/d18/f2c [731948,95588] 0 2022-01-31T19:44:20.629 INFO:tasks.workunit.client.1.smithi171.stdout:3/225: link d4/d21/d3a/f3b d4/dd/d40/f42 0 2022-01-31T19:44:20.629 INFO:tasks.workunit.client.1.smithi171.stdout:3/226: stat d4/d21/c22 0 2022-01-31T19:44:20.632 INFO:tasks.workunit.client.1.smithi171.stdout:4/258: creat d2/d3/d5/dc/f4f x:0 0 0 2022-01-31T19:44:20.633 INFO:tasks.workunit.client.1.smithi171.stdout:0/251: symlink d9/db/dc/d3f/l61 0 2022-01-31T19:44:20.634 INFO:tasks.workunit.client.1.smithi171.stdout:0/252: write d9/d24/f3c [389627,34423] 0 2022-01-31T19:44:20.634 INFO:tasks.workunit.client.1.smithi171.stdout:0/253: fsync d9/f53 0 2022-01-31T19:44:20.635 INFO:tasks.workunit.client.1.smithi171.stdout:7/190: creat d1/d1e/d2a/d29/f43 x:0 0 0 2022-01-31T19:44:20.635 INFO:tasks.workunit.client.1.smithi171.stdout:3/227: truncate d4/dd/f19 232839 0 2022-01-31T19:44:20.636 INFO:tasks.workunit.client.1.smithi171.stdout:9/285: rmdir dd/df/d3f 39 2022-01-31T19:44:20.638 INFO:tasks.workunit.client.1.smithi171.stdout:4/259: unlink d2/d3/d5/dc/ce 0 2022-01-31T19:44:20.641 INFO:tasks.workunit.client.1.smithi171.stdout:3/228: truncate d4/dd/d27/d28/f2d 769753 0 2022-01-31T19:44:20.641 INFO:tasks.workunit.client.1.smithi171.stdout:3/229: rename d4/d21 to d4/d21/d29/d43 22 2022-01-31T19:44:20.641 INFO:tasks.workunit.client.1.smithi171.stdout:3/230: stat d4/c6 0 2022-01-31T19:44:20.642 INFO:tasks.workunit.client.1.smithi171.stdout:9/286: link dd/df/d15/f2c dd/f5b 0 2022-01-31T19:44:20.643 INFO:tasks.workunit.client.1.smithi171.stdout:7/191: rmdir d1/d1e/d2a/d29/d31 39 2022-01-31T19:44:20.643 INFO:tasks.workunit.client.1.smithi171.stdout:7/192: read - d1/d1e/d2a/f3b zero size 2022-01-31T19:44:20.644 INFO:tasks.workunit.client.1.smithi171.stdout:7/193: write d1/d24/f21 [676344,98846] 0 2022-01-31T19:44:20.644 INFO:tasks.workunit.client.1.smithi171.stdout:3/231: symlink d4/d21/d3a/l44 0 2022-01-31T19:44:20.645 INFO:tasks.workunit.client.1.smithi171.stdout:3/232: readlink d4/dd/d27/d2e/l32 0 2022-01-31T19:44:20.645 INFO:tasks.workunit.client.1.smithi171.stdout:9/287: symlink dd/d1c/l5c 0 2022-01-31T19:44:20.645 INFO:tasks.workunit.client.1.smithi171.stdout:9/288: chown dd/df/d15/d38/l5a 5897 1 2022-01-31T19:44:20.646 INFO:tasks.workunit.client.1.smithi171.stdout:7/194: rename d1/d24/f37 to d1/d24/f44 0 2022-01-31T19:44:20.646 INFO:tasks.workunit.client.1.smithi171.stdout:7/195: fdatasync d1/d1e/d2a/d29/d31/f19 0 2022-01-31T19:44:20.647 INFO:tasks.workunit.client.1.smithi171.stdout:3/233: rmdir d4/d21/d29 0 2022-01-31T19:44:20.648 INFO:tasks.workunit.client.1.smithi171.stdout:9/289: symlink dd/l5d 0 2022-01-31T19:44:20.648 INFO:tasks.workunit.client.1.smithi171.stdout:9/290: fdatasync f4 0 2022-01-31T19:44:20.649 INFO:tasks.workunit.client.1.smithi171.stdout:9/291: fdatasync fc 0 2022-01-31T19:44:20.649 INFO:tasks.workunit.client.1.smithi171.stdout:9/292: chown dd 87846017 1 2022-01-31T19:44:20.650 INFO:tasks.workunit.client.1.smithi171.stdout:7/196: creat d1/d1e/d2a/d29/d31/d30/d3c/f45 x:0 0 0 2022-01-31T19:44:20.650 INFO:tasks.workunit.client.1.smithi171.stdout:7/197: dread - d1/d24/f18 zero size 2022-01-31T19:44:20.653 INFO:tasks.workunit.client.1.smithi171.stdout:9/293: symlink dd/df/d3f/l5e 0 2022-01-31T19:44:20.653 INFO:tasks.workunit.client.1.smithi171.stdout:9/294: rmdir dd 39 2022-01-31T19:44:20.655 INFO:tasks.workunit.client.1.smithi171.stdout:9/295: read dd/df/d15/f22 [3387103,8405] 0 2022-01-31T19:44:20.655 INFO:tasks.workunit.client.1.smithi171.stdout:2/319: dwrite d7/d55/f59 [0,4194304] 0 2022-01-31T19:44:20.656 INFO:tasks.workunit.client.1.smithi171.stdout:9/296: truncate dd/df/d15/d1a/f4b 2106833 0 2022-01-31T19:44:20.658 INFO:tasks.workunit.client.1.smithi171.stdout:2/320: creat d7/d26/f5f x:0 0 0 2022-01-31T19:44:20.659 INFO:tasks.workunit.client.1.smithi171.stdout:9/297: truncate fc 203306 0 2022-01-31T19:44:20.660 INFO:tasks.workunit.client.1.smithi171.stdout:9/298: symlink dd/d1c/l5f 0 2022-01-31T19:44:20.660 INFO:tasks.workunit.client.1.smithi171.stdout:9/299: dread - dd/df/d15/f4a zero size 2022-01-31T19:44:20.665 INFO:tasks.workunit.client.1.smithi171.stdout:7/198: dread d1/d1e/d2a/d29/d31/fe [0,4194304] 0 2022-01-31T19:44:20.665 INFO:tasks.workunit.client.1.smithi171.stdout:7/199: write d1/d24/f21 [1547004,22266] 0 2022-01-31T19:44:20.666 INFO:tasks.workunit.client.1.smithi171.stdout:7/200: chown d1/d1e/d2a/d29/d31/d30/f3f 16 1 2022-01-31T19:44:20.670 INFO:tasks.workunit.client.1.smithi171.stdout:6/220: dwrite f9 [4194304,4194304] 0 2022-01-31T19:44:20.673 INFO:tasks.workunit.client.1.smithi171.stdout:6/221: creat dc/d19/d30/f3d x:0 0 0 2022-01-31T19:44:20.674 INFO:tasks.workunit.client.1.smithi171.stdout:6/222: truncate dc/d18/f2c 863492 0 2022-01-31T19:44:20.675 INFO:tasks.workunit.client.1.smithi171.stdout:6/223: mknod dc/d19/d30/c3e 0 2022-01-31T19:44:20.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/260: dwrite d2/d3/d5/d11/d12/d15/f24 [4194304,4194304] 0 2022-01-31T19:44:20.678 INFO:tasks.workunit.client.1.smithi171.stdout:3/234: rmdir d4/dd/d27 39 2022-01-31T19:44:20.681 INFO:tasks.workunit.client.1.smithi171.stdout:4/261: truncate d2/f13 1686317 0 2022-01-31T19:44:20.682 INFO:tasks.workunit.client.1.smithi171.stdout:4/262: rename d2/d3/d5/dc to d2/d3/d5/dc/d50 22 2022-01-31T19:44:20.682 INFO:tasks.workunit.client.1.smithi171.stdout:7/201: dread d1/d1e/d2a/d29/d31/f19 [0,4194304] 0 2022-01-31T19:44:20.682 INFO:tasks.workunit.client.1.smithi171.stdout:7/202: chown d1/d24/f35 15223656 1 2022-01-31T19:44:20.685 INFO:tasks.workunit.client.1.smithi171.stdout:4/263: link d2/d3/c49 d2/d3/c51 0 2022-01-31T19:44:20.687 INFO:tasks.workunit.client.1.smithi171.stdout:7/203: unlink d1/d1e/d2a/f2f 0 2022-01-31T19:44:20.688 INFO:tasks.workunit.client.1.smithi171.stdout:3/235: dread d4/dd/d1a/f2b [0,4194304] 0 2022-01-31T19:44:20.689 INFO:tasks.workunit.client.1.smithi171.stdout:3/236: readlink d4/dd/d27/d2e/l32 0 2022-01-31T19:44:20.692 INFO:tasks.workunit.client.1.smithi171.stdout:4/264: rename d2/f1e to d2/d3/d5/d11/d12/d15/f52 0 2022-01-31T19:44:20.693 INFO:tasks.workunit.client.1.smithi171.stdout:4/265: mknod d2/d3/d5/dc/d4b/c53 0 2022-01-31T19:44:20.695 INFO:tasks.workunit.client.1.smithi171.stdout:4/266: mknod d2/d3/d5/d11/d12/c54 0 2022-01-31T19:44:20.695 INFO:tasks.workunit.client.1.smithi171.stdout:9/300: dwrite dd/f37 [0,4194304] 0 2022-01-31T19:44:20.695 INFO:tasks.workunit.client.1.smithi171.stdout:0/254: dwrite d9/d24/f3c [0,4194304] 0 2022-01-31T19:44:20.696 INFO:tasks.workunit.client.1.smithi171.stdout:0/255: getdents d9/db/d22/d3a 0 2022-01-31T19:44:20.697 INFO:tasks.workunit.client.1.smithi171.stdout:9/301: mknod dd/df/c60 0 2022-01-31T19:44:20.697 INFO:tasks.workunit.client.1.smithi171.stdout:0/256: write d9/f1e [799240,125372] 0 2022-01-31T19:44:20.697 INFO:tasks.workunit.client.1.smithi171.stdout:0/257: truncate d9/db/dc/d3f/d25/f52 209958 0 2022-01-31T19:44:20.698 INFO:tasks.workunit.client.1.smithi171.stdout:2/321: dwrite d7/d26/d3d/d40/f4f [0,4194304] 0 2022-01-31T19:44:20.698 INFO:tasks.workunit.client.1.smithi171.stdout:0/258: creat d9/f62 x:0 0 0 2022-01-31T19:44:20.698 INFO:tasks.workunit.client.1.smithi171.stdout:2/322: write d7/d26/d3d/d40/f5c [890677,23136] 0 2022-01-31T19:44:20.701 INFO:tasks.workunit.client.1.smithi171.stdout:0/259: dread d9/db/d22/d3a/f42 [0,4194304] 0 2022-01-31T19:44:20.709 INFO:tasks.workunit.client.1.smithi171.stdout:0/260: link d9/db/dc/d3f/d25/f52 d9/db/dc/d3f/d25/f63 0 2022-01-31T19:44:20.709 INFO:tasks.workunit.client.1.smithi171.stdout:2/323: rmdir d7/d26 39 2022-01-31T19:44:20.711 INFO:tasks.workunit.client.1.smithi171.stdout:0/261: rename d9/db/d22/d3a/c55 to d9/d24/d4c/d35/c64 0 2022-01-31T19:44:20.712 INFO:tasks.workunit.client.1.smithi171.stdout:6/224: dwrite fa [0,4194304] 0 2022-01-31T19:44:20.712 INFO:tasks.workunit.client.1.smithi171.stdout:6/225: fdatasync dc/f17 0 2022-01-31T19:44:20.715 INFO:tasks.workunit.client.1.smithi171.stdout:4/267: dread d2/d3/d5/d11/d12/f1d [0,4194304] 0 2022-01-31T19:44:20.723 INFO:tasks.workunit.client.1.smithi171.stdout:9/302: dwrite dd/f5b [0,4194304] 0 2022-01-31T19:44:20.730 INFO:tasks.workunit.client.1.smithi171.stdout:4/268: dwrite d2/d3/d5/d11/f27 [4194304,4194304] 0 2022-01-31T19:44:20.734 INFO:tasks.workunit.client.1.smithi171.stdout:2/324: rmdir d7/dd 39 2022-01-31T19:44:20.735 INFO:tasks.workunit.client.1.smithi171.stdout:2/325: chown d7/d26/f5f 5860164 1 2022-01-31T19:44:20.736 INFO:tasks.workunit.client.1.smithi171.stdout:0/262: truncate f3 1219785 0 2022-01-31T19:44:20.736 INFO:tasks.workunit.client.1.smithi171.stdout:0/263: chown f4 2586 1 2022-01-31T19:44:20.736 INFO:tasks.workunit.client.1.smithi171.stdout:0/264: dread - d9/d24/f26 zero size 2022-01-31T19:44:20.747 INFO:tasks.workunit.client.1.smithi171.stdout:9/303: truncate dd/df/d15/f22 2320718 0 2022-01-31T19:44:20.750 INFO:tasks.workunit.client.1.smithi171.stdout:4/269: creat d2/d3/d5/d3b/f55 x:0 0 0 2022-01-31T19:44:20.750 INFO:tasks.workunit.client.1.smithi171.stdout:4/270: fdatasync d2/d3/d5/d11/d12/f45 0 2022-01-31T19:44:20.751 INFO:tasks.workunit.client.1.smithi171.stdout:2/326: symlink d7/d55/l60 0 2022-01-31T19:44:20.751 INFO:tasks.workunit.client.1.smithi171.stdout:2/327: truncate d7/f11 1562682 0 2022-01-31T19:44:20.751 INFO:tasks.workunit.client.1.smithi171.stdout:4/271: write d2/f3c [219667,22407] 0 2022-01-31T19:44:20.752 INFO:tasks.workunit.client.1.smithi171.stdout:4/272: read d2/d3/d5/dc/f26 [3476877,27217] 0 2022-01-31T19:44:20.753 INFO:tasks.workunit.client.1.smithi171.stdout:0/265: symlink d9/db/dc/d3f/d25/l65 0 2022-01-31T19:44:20.755 INFO:tasks.workunit.client.1.smithi171.stdout:9/304: symlink dd/df/d15/d38/d41/l61 0 2022-01-31T19:44:20.758 INFO:tasks.workunit.client.1.smithi171.stdout:2/328: mkdir d7/d55/d61 0 2022-01-31T19:44:20.760 INFO:tasks.workunit.client.1.smithi171.stdout:4/273: unlink d2/d3/d5/d11/d12/l17 0 2022-01-31T19:44:20.762 INFO:tasks.workunit.client.1.smithi171.stdout:2/329: link d7/d26/d3d/d40/f57 d7/f62 0 2022-01-31T19:44:20.763 INFO:tasks.workunit.client.1.smithi171.stdout:2/330: fdatasync d7/f22 0 2022-01-31T19:44:20.763 INFO:tasks.workunit.client.1.smithi171.stdout:4/274: mkdir d2/d3/d5/dc/d4b/d56 0 2022-01-31T19:44:20.781 INFO:tasks.workunit.client.1.smithi171.stdout:0/266: dwrite d9/d24/d4c/f3d [4194304,4194304] 0 2022-01-31T19:44:20.782 INFO:tasks.workunit.client.1.smithi171.stdout:0/267: creat d9/d24/d4c/f66 x:0 0 0 2022-01-31T19:44:20.782 INFO:tasks.workunit.client.1.smithi171.stdout:9/305: dwrite dd/df/d15/f4a [0,4194304] 0 2022-01-31T19:44:20.782 INFO:tasks.workunit.client.1.smithi171.stdout:2/331: dwrite d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:20.784 INFO:tasks.workunit.client.1.smithi171.stdout:4/275: rmdir d2/d3/d5/d3b 39 2022-01-31T19:44:20.789 INFO:tasks.workunit.client.1.smithi171.stdout:0/268: dread d9/f1d [0,4194304] 0 2022-01-31T19:44:20.790 INFO:tasks.workunit.client.1.smithi171.stdout:0/269: dread - d9/f62 zero size 2022-01-31T19:44:20.794 INFO:tasks.workunit.client.1.smithi171.stdout:0/270: mkdir d9/d24/d4c/d35/d67 0 2022-01-31T19:44:20.794 INFO:tasks.workunit.client.1.smithi171.stdout:9/306: rmdir dd/df/d15/d1a 39 2022-01-31T19:44:20.796 INFO:tasks.workunit.client.1.smithi171.stdout:9/307: unlink dd/df/d15/f26 0 2022-01-31T19:44:20.802 INFO:tasks.workunit.client.1.smithi171.stdout:2/332: dwrite d7/f13 [0,4194304] 0 2022-01-31T19:44:20.804 INFO:tasks.workunit.client.1.smithi171.stdout:2/333: dread d7/d26/d3d/d40/f5c [0,4194304] 0 2022-01-31T19:44:20.805 INFO:tasks.workunit.client.1.smithi171.stdout:2/334: truncate d7/d26/d3d/f42 3877518 0 2022-01-31T19:44:20.805 INFO:tasks.workunit.client.1.smithi171.stdout:2/335: write d7/dd/f3c [662660,18614] 0 2022-01-31T19:44:20.806 INFO:tasks.workunit.client.1.smithi171.stdout:2/336: rmdir d7/d9/d1a/d45 39 2022-01-31T19:44:20.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/271: dwrite d9/db/d22/d3a/f42 [0,4194304] 0 2022-01-31T19:44:20.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/272: dread - d9/d24/f26 zero size 2022-01-31T19:44:20.819 INFO:tasks.workunit.client.1.smithi171.stdout:0/273: write d9/db/dc/d3f/f4a [1915335,4764] 0 2022-01-31T19:44:20.819 INFO:tasks.workunit.client.1.smithi171.stdout:0/274: read - d9/db/dc/d3f/d25/f29 zero size 2022-01-31T19:44:20.820 INFO:tasks.workunit.client.1.smithi171.stdout:0/275: read d9/db/dc/d3f/f2d [3804626,75676] 0 2022-01-31T19:44:20.820 INFO:tasks.workunit.client.1.smithi171.stdout:0/276: chown f4 49 1 2022-01-31T19:44:20.821 INFO:tasks.workunit.client.1.smithi171.stdout:0/277: symlink d9/db/d22/d5d/l68 0 2022-01-31T19:44:20.821 INFO:tasks.workunit.client.1.smithi171.stdout:0/278: fsync d9/d24/d4c/f5e 0 2022-01-31T19:44:20.821 INFO:tasks.workunit.client.1.smithi171.stdout:0/279: creat d9/db/d22/d5d/f69 x:0 0 0 2022-01-31T19:44:20.829 INFO:tasks.workunit.client.1.smithi171.stdout:2/337: dwrite d7/d9/f2c [4194304,4194304] 0 2022-01-31T19:44:20.829 INFO:tasks.workunit.client.1.smithi171.stdout:9/308: dwrite dd/df/f20 [0,4194304] 0 2022-01-31T19:44:20.829 INFO:tasks.workunit.client.1.smithi171.stdout:9/309: readlink dd/d1c/l39 0 2022-01-31T19:44:20.832 INFO:tasks.workunit.client.1.smithi171.stdout:0/280: dread d9/d24/d4c/f3d [0,4194304] 0 2022-01-31T19:44:20.837 INFO:tasks.workunit.client.1.smithi171.stdout:9/310: creat dd/df/f62 x:0 0 0 2022-01-31T19:44:20.838 INFO:tasks.workunit.client.1.smithi171.stdout:0/281: creat d9/db/d22/d3a/f6a x:0 0 0 2022-01-31T19:44:20.841 INFO:tasks.workunit.client.1.smithi171.stdout:9/311: creat dd/df/d15/d38/d4d/f63 x:0 0 0 2022-01-31T19:44:20.841 INFO:tasks.workunit.client.1.smithi171.stdout:0/282: symlink d9/d24/l6b 0 2022-01-31T19:44:20.841 INFO:tasks.workunit.client.1.smithi171.stdout:9/312: write dd/d1c/f52 [183458,42718] 0 2022-01-31T19:44:20.842 INFO:tasks.workunit.client.1.smithi171.stdout:1/229: sync 2022-01-31T19:44:20.842 INFO:tasks.workunit.client.1.smithi171.stdout:5/223: sync 2022-01-31T19:44:20.842 INFO:tasks.workunit.client.1.smithi171.stdout:1/230: dread - dd/d23/f41 zero size 2022-01-31T19:44:20.843 INFO:tasks.workunit.client.1.smithi171.stdout:1/231: dread dd/df/d19/f21 [0,4194304] 0 2022-01-31T19:44:20.844 INFO:tasks.workunit.client.1.smithi171.stdout:1/232: fdatasync f7 0 2022-01-31T19:44:20.845 INFO:tasks.workunit.client.1.smithi171.stdout:0/283: mknod d9/db/d22/d3a/c6c 0 2022-01-31T19:44:20.846 INFO:tasks.workunit.client.1.smithi171.stdout:9/313: mkdir dd/df/d15/d1a/d64 0 2022-01-31T19:44:20.858 INFO:tasks.workunit.client.1.smithi171.stdout:1/233: dread dd/df/d19/f32 [0,4194304] 0 2022-01-31T19:44:20.860 INFO:tasks.workunit.client.1.smithi171.stdout:1/234: mkdir dd/d23/d34/d4b 0 2022-01-31T19:44:20.862 INFO:tasks.workunit.client.1.smithi171.stdout:1/235: link dd/df/d17/l3b dd/d23/d34/d4b/l4c 0 2022-01-31T19:44:20.867 INFO:tasks.workunit.client.1.smithi171.stdout:9/314: dwrite dd/df/d15/f1b [0,4194304] 0 2022-01-31T19:44:20.869 INFO:tasks.workunit.client.1.smithi171.stdout:9/315: mknod dd/c65 0 2022-01-31T19:44:20.871 INFO:tasks.workunit.client.1.smithi171.stdout:9/316: mkdir dd/df/d15/d1a/d64/d66 0 2022-01-31T19:44:20.873 INFO:tasks.workunit.client.1.smithi171.stdout:9/317: creat dd/df/d15/d1a/d64/f67 x:0 0 0 2022-01-31T19:44:20.877 INFO:tasks.workunit.client.1.smithi171.stdout:0/284: dwrite d9/f53 [0,4194304] 0 2022-01-31T19:44:20.879 INFO:tasks.workunit.client.1.smithi171.stdout:5/224: dwrite d5/d6/d15/d29/f4c [0,4194304] 0 2022-01-31T19:44:20.883 INFO:tasks.workunit.client.1.smithi171.stdout:1/236: dwrite dd/d23/f29 [4194304,4194304] 0 2022-01-31T19:44:20.886 INFO:tasks.workunit.client.1.smithi171.stdout:1/237: unlink dd/df/d19/d1b/d1e/f46 0 2022-01-31T19:44:20.892 INFO:tasks.workunit.client.1.smithi171.stdout:1/238: getdents dd/d23/d34 0 2022-01-31T19:44:20.894 INFO:tasks.workunit.client.1.smithi171.stdout:1/239: mknod dd/c4d 0 2022-01-31T19:44:20.898 INFO:tasks.workunit.client.1.smithi171.stdout:9/318: dwrite dd/d1c/f1f [4194304,4194304] 0 2022-01-31T19:44:20.900 INFO:tasks.workunit.client.1.smithi171.stdout:5/225: dwrite d5/d6/d15/f1c [0,4194304] 0 2022-01-31T19:44:20.900 INFO:tasks.workunit.client.1.smithi171.stdout:5/226: chown d5/d6/d15/d23/l24 4219042 1 2022-01-31T19:44:20.900 INFO:tasks.workunit.client.1.smithi171.stdout:5/227: chown d5/d6/d15/d29/l36 199844940 1 2022-01-31T19:44:20.902 INFO:tasks.workunit.client.1.smithi171.stdout:9/319: rename dd/c10 to dd/df/d15/d38/c68 0 2022-01-31T19:44:20.902 INFO:tasks.workunit.client.1.smithi171.stdout:9/320: chown dd/df/d15/f1b 102 1 2022-01-31T19:44:20.905 INFO:tasks.workunit.client.1.smithi171.stdout:1/240: dread dd/df/d19/d1b/d1e/f4a [0,4194304] 0 2022-01-31T19:44:20.906 INFO:tasks.workunit.client.1.smithi171.stdout:5/228: rename d5/d6/d15/f1f to d5/f4e 0 2022-01-31T19:44:20.906 INFO:tasks.workunit.client.1.smithi171.stdout:5/229: write d5/d6/d15/d29/f4d [29178,5740] 0 2022-01-31T19:44:20.906 INFO:tasks.workunit.client.1.smithi171.stdout:5/230: chown l3 2 1 2022-01-31T19:44:20.907 INFO:tasks.workunit.client.1.smithi171.stdout:5/231: write d5/d6/d15/d29/f34 [544865,14384] 0 2022-01-31T19:44:20.908 INFO:tasks.workunit.client.1.smithi171.stdout:1/241: creat dd/df/d19/d1b/f4e x:0 0 0 2022-01-31T19:44:20.908 INFO:tasks.workunit.client.1.smithi171.stdout:1/242: chown dd/df/d19/d1b/f49 110906 1 2022-01-31T19:44:20.909 INFO:tasks.workunit.client.1.smithi171.stdout:1/243: stat f3 0 2022-01-31T19:44:20.909 INFO:tasks.workunit.client.1.smithi171.stdout:1/244: fdatasync fa 0 2022-01-31T19:44:20.909 INFO:tasks.workunit.client.1.smithi171.stdout:1/245: write dd/df/f2a [65791,63587] 0 2022-01-31T19:44:20.910 INFO:tasks.workunit.client.1.smithi171.stdout:5/232: write d5/d6/d15/f42 [717130,104922] 0 2022-01-31T19:44:20.910 INFO:tasks.workunit.client.1.smithi171.stdout:5/233: dread - d5/d6/d15/f33 zero size 2022-01-31T19:44:20.912 INFO:tasks.workunit.client.1.smithi171.stdout:5/234: truncate d5/f8 2368760 0 2022-01-31T19:44:20.920 INFO:tasks.workunit.client.1.smithi171.stdout:9/321: dwrite dd/d1c/f29 [0,4194304] 0 2022-01-31T19:44:20.923 INFO:tasks.workunit.client.1.smithi171.stdout:9/322: rename dd/df/d15/f1b to dd/df/d15/f69 0 2022-01-31T19:44:20.923 INFO:tasks.workunit.client.1.smithi171.stdout:9/323: chown dd/df/d15/d1a 1 1 2022-01-31T19:44:20.924 INFO:tasks.workunit.client.1.smithi171.stdout:9/324: creat dd/f6a x:0 0 0 2022-01-31T19:44:20.925 INFO:tasks.workunit.client.1.smithi171.stdout:1/246: dwrite dd/df/d19/f21 [0,4194304] 0 2022-01-31T19:44:20.926 INFO:tasks.workunit.client.1.smithi171.stdout:9/325: link dd/df/c60 dd/df/c6b 0 2022-01-31T19:44:20.929 INFO:tasks.workunit.client.1.smithi171.stdout:9/326: mknod dd/df/d15/d38/d4d/c6c 0 2022-01-31T19:44:20.932 INFO:tasks.workunit.client.1.smithi171.stdout:5/235: dread d5/d6/d15/f3f [0,4194304] 0 2022-01-31T19:44:20.934 INFO:tasks.workunit.client.1.smithi171.stdout:5/236: rename d5/d6/f27 to d5/f4f 0 2022-01-31T19:44:20.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/247: dread f7 [0,4194304] 0 2022-01-31T19:44:20.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/248: write dd/df/f2a [263007,11479] 0 2022-01-31T19:44:20.936 INFO:tasks.workunit.client.1.smithi171.stdout:5/237: mknod d5/d6/d15/d23/c50 0 2022-01-31T19:44:20.937 INFO:tasks.workunit.client.1.smithi171.stdout:5/238: read f1 [292374,104786] 0 2022-01-31T19:44:20.937 INFO:tasks.workunit.client.1.smithi171.stdout:5/239: chown d5/l40 599339 1 2022-01-31T19:44:20.937 INFO:tasks.workunit.client.1.smithi171.stdout:5/240: dread - d5/d6/dd/f45 zero size 2022-01-31T19:44:20.937 INFO:tasks.workunit.client.1.smithi171.stdout:5/241: fsync d5/d6/d15/d29/f4c 0 2022-01-31T19:44:20.938 INFO:tasks.workunit.client.1.smithi171.stdout:1/249: link dd/d23/l3c dd/d23/l4f 0 2022-01-31T19:44:20.938 INFO:tasks.workunit.client.1.smithi171.stdout:1/250: mkdir dd/d23/d34/d50 0 2022-01-31T19:44:20.939 INFO:tasks.workunit.client.1.smithi171.stdout:1/251: symlink dd/d23/d2e/l51 0 2022-01-31T19:44:20.940 INFO:tasks.workunit.client.1.smithi171.stdout:1/252: creat dd/d23/d34/d4b/f52 x:0 0 0 2022-01-31T19:44:20.941 INFO:tasks.workunit.client.1.smithi171.stdout:1/253: write dd/df/d19/d1b/f49 [907992,84789] 0 2022-01-31T19:44:20.941 INFO:tasks.workunit.client.1.smithi171.stdout:1/254: creat dd/d23/d34/d50/f53 x:0 0 0 2022-01-31T19:44:20.943 INFO:tasks.workunit.client.1.smithi171.stdout:1/255: symlink dd/d23/l54 0 2022-01-31T19:44:20.943 INFO:tasks.workunit.client.1.smithi171.stdout:1/256: chown dd/df/d19/f21 4006 1 2022-01-31T19:44:20.944 INFO:tasks.workunit.client.1.smithi171.stdout:1/257: chown dd/d23/d2e/f2f 338 1 2022-01-31T19:44:20.944 INFO:tasks.workunit.client.1.smithi171.stdout:1/258: write dd/df/d19/d1b/f49 [912492,52896] 0 2022-01-31T19:44:20.944 INFO:tasks.workunit.client.1.smithi171.stdout:5/242: dread d5/ff [0,4194304] 0 2022-01-31T19:44:20.944 INFO:tasks.workunit.client.1.smithi171.stdout:5/243: creat d5/d6/d15/d23/d3d/f51 x:0 0 0 2022-01-31T19:44:20.945 INFO:tasks.workunit.client.1.smithi171.stdout:1/259: mknod dd/d23/d34/d4b/c55 0 2022-01-31T19:44:20.945 INFO:tasks.workunit.client.1.smithi171.stdout:1/260: creat dd/f56 x:0 0 0 2022-01-31T19:44:20.946 INFO:tasks.workunit.client.1.smithi171.stdout:5/244: truncate d5/d6/dd/f35 167592 0 2022-01-31T19:44:20.946 INFO:tasks.workunit.client.1.smithi171.stdout:5/245: readlink d5/d6/d15/d29/l2f 0 2022-01-31T19:44:20.946 INFO:tasks.workunit.client.1.smithi171.stdout:5/246: chown d5/d6/l3e 29464753 1 2022-01-31T19:44:20.946 INFO:tasks.workunit.client.1.smithi171.stdout:1/261: truncate dd/df/f12 1467613 0 2022-01-31T19:44:20.950 INFO:tasks.workunit.client.1.smithi171.stdout:9/327: dwrite f9 [0,4194304] 0 2022-01-31T19:44:20.954 INFO:tasks.workunit.client.1.smithi171.stdout:1/262: truncate dd/df/d19/f21 2422987 0 2022-01-31T19:44:20.954 INFO:tasks.workunit.client.1.smithi171.stdout:1/263: fdatasync dd/df/d17/f45 0 2022-01-31T19:44:20.954 INFO:tasks.workunit.client.1.smithi171.stdout:1/264: creat dd/d23/f57 x:0 0 0 2022-01-31T19:44:20.959 INFO:tasks.workunit.client.1.smithi171.stdout:1/265: dread dd/df/d19/d1b/d1e/f4a [0,4194304] 0 2022-01-31T19:44:20.964 INFO:tasks.workunit.client.1.smithi171.stdout:1/266: link dd/d23/d2e/f2f dd/d23/d34/d50/f58 0 2022-01-31T19:44:20.964 INFO:tasks.workunit.client.1.smithi171.stdout:5/247: dwrite d5/d6/d15/d29/f39 [0,4194304] 0 2022-01-31T19:44:20.966 INFO:tasks.workunit.client.1.smithi171.stdout:9/328: dread dd/df/d15/d1a/f56 [0,4194304] 0 2022-01-31T19:44:20.966 INFO:tasks.workunit.client.1.smithi171.stdout:9/329: write dd/df/d15/f4a [4266326,111539] 0 2022-01-31T19:44:20.966 INFO:tasks.workunit.client.1.smithi171.stdout:9/330: readlink dd/d1c/l5f 0 2022-01-31T19:44:20.967 INFO:tasks.workunit.client.1.smithi171.stdout:1/267: rename dd/df/f2a to dd/d23/d34/d4b/f59 0 2022-01-31T19:44:20.971 INFO:tasks.workunit.client.1.smithi171.stdout:5/248: symlink d5/d6/dd/l52 0 2022-01-31T19:44:20.972 INFO:tasks.workunit.client.1.smithi171.stdout:9/331: symlink dd/df/d15/d53/l6d 0 2022-01-31T19:44:20.979 INFO:tasks.workunit.client.1.smithi171.stdout:9/332: mkdir dd/df/d15/d1a/d64/d6e 0 2022-01-31T19:44:20.980 INFO:tasks.workunit.client.1.smithi171.stdout:1/268: rmdir dd/d23/d2e 39 2022-01-31T19:44:20.980 INFO:tasks.workunit.client.1.smithi171.stdout:9/333: read dd/df/f17 [1715080,1713] 0 2022-01-31T19:44:20.981 INFO:tasks.workunit.client.1.smithi171.stdout:1/269: symlink dd/d23/d2e/l5a 0 2022-01-31T19:44:20.982 INFO:tasks.workunit.client.1.smithi171.stdout:9/334: symlink dd/df/d15/d1a/d64/d6e/l6f 0 2022-01-31T19:44:20.985 INFO:tasks.workunit.client.1.smithi171.stdout:1/270: creat dd/df/d19/d36/f5b x:0 0 0 2022-01-31T19:44:20.986 INFO:tasks.workunit.client.1.smithi171.stdout:9/335: rename dd/d1c/f23 to dd/d1c/f70 0 2022-01-31T19:44:20.994 INFO:tasks.workunit.client.1.smithi171.stdout:9/336: dread dd/df/d15/f69 [0,4194304] 0 2022-01-31T19:44:20.995 INFO:tasks.workunit.client.1.smithi171.stdout:5/249: dwrite d5/d6/d15/f1d [0,4194304] 0 2022-01-31T19:44:20.996 INFO:tasks.workunit.client.1.smithi171.stdout:5/250: mknod d5/d6/d15/d29/c53 0 2022-01-31T19:44:20.997 INFO:tasks.workunit.client.1.smithi171.stdout:5/251: unlink d5/f3b 0 2022-01-31T19:44:21.003 INFO:tasks.workunit.client.1.smithi171.stdout:5/252: dread d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:21.010 INFO:tasks.workunit.client.1.smithi171.stdout:9/337: dwrite dd/f37 [4194304,4194304] 0 2022-01-31T19:44:21.014 INFO:tasks.workunit.client.1.smithi171.stdout:9/338: symlink dd/df/d15/d1a/d64/d6e/l71 0 2022-01-31T19:44:21.014 INFO:tasks.workunit.client.1.smithi171.stdout:9/339: fdatasync dd/df/d15/d38/d4d/f63 0 2022-01-31T19:44:21.014 INFO:tasks.workunit.client.1.smithi171.stdout:9/340: write dd/df/f21 [4569542,1734] 0 2022-01-31T19:44:21.017 INFO:tasks.workunit.client.1.smithi171.stdout:9/341: link f4 dd/df/f72 0 2022-01-31T19:44:21.018 INFO:tasks.workunit.client.1.smithi171.stdout:9/342: fsync dd/d1c/f29 0 2022-01-31T19:44:21.018 INFO:tasks.workunit.client.1.smithi171.stdout:9/343: truncate dd/f6a 596455 0 2022-01-31T19:44:21.018 INFO:tasks.workunit.client.1.smithi171.stdout:9/344: fdatasync dd/df/d15/d1a/f4b 0 2022-01-31T19:44:21.020 INFO:tasks.workunit.client.1.smithi171.stdout:9/345: mkdir dd/df/d3f/d73 0 2022-01-31T19:44:21.020 INFO:tasks.workunit.client.1.smithi171.stdout:9/346: stat dd/c13 0 2022-01-31T19:44:21.024 INFO:tasks.workunit.client.1.smithi171.stdout:9/347: getdents dd/df/d15/d38/d4d 0 2022-01-31T19:44:21.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/348: dread dd/d1c/f52 [0,4194304] 0 2022-01-31T19:44:21.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/349: creat dd/f74 x:0 0 0 2022-01-31T19:44:21.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/350: creat dd/d1c/f75 x:0 0 0 2022-01-31T19:44:21.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/351: creat dd/df/d15/d38/d41/f76 x:0 0 0 2022-01-31T19:44:21.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/253: dwrite d5/f4e [0,4194304] 0 2022-01-31T19:44:21.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/254: read d5/d6/dd/f35 [126884,30225] 0 2022-01-31T19:44:21.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/226: sync 2022-01-31T19:44:21.029 INFO:tasks.workunit.client.1.smithi171.stdout:3/237: sync 2022-01-31T19:44:21.030 INFO:tasks.workunit.client.1.smithi171.stdout:8/204: sync 2022-01-31T19:44:21.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/238: chown d4/dd/d40/f42 70979855 1 2022-01-31T19:44:21.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/239: chown d4/dd/d27/d2e 926867201 1 2022-01-31T19:44:21.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/240: truncate d4/dd/d27/f3f 932031 0 2022-01-31T19:44:21.031 INFO:tasks.workunit.client.1.smithi171.stdout:9/352: unlink dd/d1c/l5c 0 2022-01-31T19:44:21.031 INFO:tasks.workunit.client.1.smithi171.stdout:9/353: write dd/d1c/f52 [755873,113243] 0 2022-01-31T19:44:21.034 INFO:tasks.workunit.client.1.smithi171.stdout:5/255: symlink d5/d6/d15/d29/l54 0 2022-01-31T19:44:21.034 INFO:tasks.workunit.client.1.smithi171.stdout:5/256: creat d5/f55 x:0 0 0 2022-01-31T19:44:21.036 INFO:tasks.workunit.client.1.smithi171.stdout:8/205: link df/d18/d19/d21/f26 df/d18/f4e 0 2022-01-31T19:44:21.036 INFO:tasks.workunit.client.1.smithi171.stdout:8/206: fdatasync fa 0 2022-01-31T19:44:21.036 INFO:tasks.workunit.client.1.smithi171.stdout:8/207: fdatasync df/d1a/f29 0 2022-01-31T19:44:21.036 INFO:tasks.workunit.client.1.smithi171.stdout:8/208: chown df/d2a/d30/f35 45 1 2022-01-31T19:44:21.037 INFO:tasks.workunit.client.1.smithi171.stdout:3/241: unlink d4/d21/d3a/f3c 0 2022-01-31T19:44:21.037 INFO:tasks.workunit.client.1.smithi171.stdout:9/354: rename dd/df/d15/d1a/c2d to dd/df/d15/d1a/d64/d6e/c77 0 2022-01-31T19:44:21.039 INFO:tasks.workunit.client.1.smithi171.stdout:9/355: write dd/df/d15/f2f [724645,69842] 0 2022-01-31T19:44:21.039 INFO:tasks.workunit.client.1.smithi171.stdout:9/356: creat dd/df/d15/d38/d41/f78 x:0 0 0 2022-01-31T19:44:21.039 INFO:tasks.workunit.client.1.smithi171.stdout:9/357: creat dd/f79 x:0 0 0 2022-01-31T19:44:21.039 INFO:tasks.workunit.client.1.smithi171.stdout:9/358: creat dd/df/d15/d38/d41/f7a x:0 0 0 2022-01-31T19:44:21.040 INFO:tasks.workunit.client.1.smithi171.stdout:8/209: creat df/f4f x:0 0 0 2022-01-31T19:44:21.040 INFO:tasks.workunit.client.1.smithi171.stdout:3/242: creat d4/dd/f45 x:0 0 0 2022-01-31T19:44:21.041 INFO:tasks.workunit.client.1.smithi171.stdout:3/243: write d4/dd/d1a/f1b [174774,38762] 0 2022-01-31T19:44:21.041 INFO:tasks.workunit.client.1.smithi171.stdout:3/244: write d4/dd/d40/f42 [482197,53661] 0 2022-01-31T19:44:21.043 INFO:tasks.workunit.client.1.smithi171.stdout:9/359: mknod dd/df/d15/d38/c7b 0 2022-01-31T19:44:21.043 INFO:tasks.workunit.client.1.smithi171.stdout:9/360: dread - dd/df/f25 zero size 2022-01-31T19:44:21.043 INFO:tasks.workunit.client.1.smithi171.stdout:9/361: chown dd/df/d15/d1a/f27 21350777 1 2022-01-31T19:44:21.044 INFO:tasks.workunit.client.1.smithi171.stdout:8/210: write f7 [2175308,23470] 0 2022-01-31T19:44:21.045 INFO:tasks.workunit.client.1.smithi171.stdout:6/227: dwrite dc/d18/f2c [0,4194304] 0 2022-01-31T19:44:21.045 INFO:tasks.workunit.client.1.smithi171.stdout:8/211: truncate df/f4f 925027 0 2022-01-31T19:44:21.045 INFO:tasks.workunit.client.1.smithi171.stdout:8/212: chown df/d18/d19/d21/f32 830 1 2022-01-31T19:44:21.045 INFO:tasks.workunit.client.1.smithi171.stdout:5/257: mknod d5/d6/d15/c56 0 2022-01-31T19:44:21.045 INFO:tasks.workunit.client.1.smithi171.stdout:3/245: write d4/d21/f2a [1198759,64999] 0 2022-01-31T19:44:21.047 INFO:tasks.workunit.client.1.smithi171.stdout:5/258: write d5/d6/d15/f3f [1467336,92553] 0 2022-01-31T19:44:21.047 INFO:tasks.workunit.client.1.smithi171.stdout:6/228: getdents dc/d21 0 2022-01-31T19:44:21.063 INFO:tasks.workunit.client.1.smithi171.stdout:8/213: getdents df 0 2022-01-31T19:44:21.064 INFO:tasks.workunit.client.1.smithi171.stdout:3/246: rmdir d4/d21/d3a 39 2022-01-31T19:44:21.066 INFO:tasks.workunit.client.1.smithi171.stdout:5/259: unlink d5/d6/dd/l2b 0 2022-01-31T19:44:21.069 INFO:tasks.workunit.client.1.smithi171.stdout:3/247: symlink d4/d21/l46 0 2022-01-31T19:44:21.070 INFO:tasks.workunit.client.1.smithi171.stdout:3/248: write d4/d21/f2a [1985608,21867] 0 2022-01-31T19:44:21.073 INFO:tasks.workunit.client.1.smithi171.stdout:3/249: creat d4/dd/d27/f47 x:0 0 0 2022-01-31T19:44:21.073 INFO:tasks.workunit.client.1.smithi171.stdout:3/250: write d4/d21/d3a/f3b [1103526,118616] 0 2022-01-31T19:44:21.073 INFO:tasks.workunit.client.1.smithi171.stdout:3/251: readlink d4/dd/l33 0 2022-01-31T19:44:21.074 INFO:tasks.workunit.client.1.smithi171.stdout:3/252: read d4/dd/d27/d28/f2d [312483,60391] 0 2022-01-31T19:44:21.087 INFO:tasks.workunit.client.1.smithi171.stdout:5/260: dwrite d5/d6/d15/f1d [0,4194304] 0 2022-01-31T19:44:21.089 INFO:tasks.workunit.client.1.smithi171.stdout:5/261: rename d5/d6/f1a to d5/f57 0 2022-01-31T19:44:21.090 INFO:tasks.workunit.client.1.smithi171.stdout:5/262: mkdir d5/d6/d15/d23/d58 0 2022-01-31T19:44:21.090 INFO:tasks.workunit.client.1.smithi171.stdout:5/263: unlink f1 0 2022-01-31T19:44:21.100 INFO:tasks.workunit.client.1.smithi171.stdout:3/253: dwrite d4/fc [0,4194304] 0 2022-01-31T19:44:21.100 INFO:tasks.workunit.client.1.smithi171.stdout:3/254: chown d4/dd/c25 1 1 2022-01-31T19:44:21.106 INFO:tasks.workunit.client.1.smithi171.stdout:5/264: dwrite d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:21.111 INFO:tasks.workunit.client.1.smithi171.stdout:3/255: dread d4/fa [0,4194304] 0 2022-01-31T19:44:21.111 INFO:tasks.workunit.client.1.smithi171.stdout:3/256: chown d4/d21 79968 1 2022-01-31T19:44:21.111 INFO:tasks.workunit.client.1.smithi171.stdout:3/257: write d4/dd/f1e [23362,98044] 0 2022-01-31T19:44:21.112 INFO:tasks.workunit.client.1.smithi171.stdout:5/265: getdents d5/d6 0 2022-01-31T19:44:21.114 INFO:tasks.workunit.client.1.smithi171.stdout:5/266: mkdir d5/d59 0 2022-01-31T19:44:21.114 INFO:tasks.workunit.client.1.smithi171.stdout:5/267: read d5/d6/d15/d23/f41 [136512,59405] 0 2022-01-31T19:44:21.115 INFO:tasks.workunit.client.1.smithi171.stdout:5/268: dread - d5/d6/d15/d23/d3d/f49 zero size 2022-01-31T19:44:21.116 INFO:tasks.workunit.client.1.smithi171.stdout:5/269: mknod d5/c5a 0 2022-01-31T19:44:21.116 INFO:tasks.workunit.client.1.smithi171.stdout:5/270: write d5/d6/d15/f1c [5241539,128770] 0 2022-01-31T19:44:21.116 INFO:tasks.workunit.client.1.smithi171.stdout:5/271: stat d5/d6/d15/d29/c28 0 2022-01-31T19:44:21.117 INFO:tasks.workunit.client.1.smithi171.stdout:5/272: symlink d5/d59/l5b 0 2022-01-31T19:44:21.118 INFO:tasks.workunit.client.1.smithi171.stdout:5/273: link d5/d6/d15/f1c d5/d6/d15/d29/f5c 0 2022-01-31T19:44:21.119 INFO:tasks.workunit.client.1.smithi171.stdout:3/258: dread d4/dd/d40/f42 [0,4194304] 0 2022-01-31T19:44:21.119 INFO:tasks.workunit.client.1.smithi171.stdout:3/259: dread d4/dd/d27/f3f [0,4194304] 0 2022-01-31T19:44:21.123 INFO:tasks.workunit.client.1.smithi171.stdout:3/260: dread d4/dd/f1e [0,4194304] 0 2022-01-31T19:44:21.124 INFO:tasks.workunit.client.1.smithi171.stdout:3/261: creat d4/dd/d27/d28/d35/f48 x:0 0 0 2022-01-31T19:44:21.125 INFO:tasks.workunit.client.1.smithi171.stdout:3/262: mknod d4/dd/d40/c49 0 2022-01-31T19:44:21.126 INFO:tasks.workunit.client.1.smithi171.stdout:3/263: mknod d4/dd/d27/c4a 0 2022-01-31T19:44:21.143 INFO:tasks.workunit.client.1.smithi171.stdout:3/264: dwrite d4/dd/f20 [0,4194304] 0 2022-01-31T19:44:21.152 INFO:tasks.workunit.client.1.smithi171.stdout:3/265: dread d4/fc [0,4194304] 0 2022-01-31T19:44:21.153 INFO:tasks.workunit.client.1.smithi171.stdout:3/266: getdents d4/dd/d27/d28 0 2022-01-31T19:44:21.153 INFO:tasks.workunit.client.1.smithi171.stdout:3/267: mkdir d4/dd/d4b 0 2022-01-31T19:44:21.154 INFO:tasks.workunit.client.1.smithi171.stdout:3/268: symlink d4/d21/d3a/l4c 0 2022-01-31T19:44:21.174 INFO:tasks.workunit.client.1.smithi171.stdout:3/269: dwrite d4/fb [4194304,4194304] 0 2022-01-31T19:44:21.175 INFO:tasks.workunit.client.1.smithi171.stdout:3/270: mknod d4/d21/c4d 0 2022-01-31T19:44:21.176 INFO:tasks.workunit.client.1.smithi171.stdout:3/271: write d4/d21/f2a [3261814,74956] 0 2022-01-31T19:44:21.177 INFO:tasks.workunit.client.1.smithi171.stdout:3/272: mkdir d4/dd/d27/d4e 0 2022-01-31T19:44:21.179 INFO:tasks.workunit.client.1.smithi171.stdout:3/273: dread d4/dd/f1e [0,4194304] 0 2022-01-31T19:44:21.180 INFO:tasks.workunit.client.1.smithi171.stdout:3/274: dread - d4/dd/f45 zero size 2022-01-31T19:44:21.180 INFO:tasks.workunit.client.1.smithi171.stdout:3/275: truncate d4/dd/d27/d28/d35/f48 526289 0 2022-01-31T19:44:21.196 INFO:tasks.workunit.client.1.smithi171.stdout:3/276: dwrite d4/dd/d1a/f2b [0,4194304] 0 2022-01-31T19:44:21.196 INFO:tasks.workunit.client.1.smithi171.stdout:3/277: chown d4/dd/c25 1783451 1 2022-01-31T19:44:21.241 INFO:tasks.workunit.client.1.smithi171.stdout:7/204: sync 2022-01-31T19:44:21.242 INFO:tasks.workunit.client.1.smithi171.stdout:2/338: sync 2022-01-31T19:44:21.243 INFO:tasks.workunit.client.1.smithi171.stdout:2/339: mkdir d7/d26/d63 0 2022-01-31T19:44:21.260 INFO:tasks.workunit.client.1.smithi171.stdout:2/340: dwrite d7/dd/f4a [0,4194304] 0 2022-01-31T19:44:21.261 INFO:tasks.workunit.client.1.smithi171.stdout:2/341: fsync d7/dd/f3c 0 2022-01-31T19:44:21.262 INFO:tasks.workunit.client.1.smithi171.stdout:2/342: creat d7/d26/d63/f64 x:0 0 0 2022-01-31T19:44:21.263 INFO:tasks.workunit.client.1.smithi171.stdout:2/343: rename d7/dd/f4a to d7/d55/f65 0 2022-01-31T19:44:21.328 INFO:tasks.workunit.client.1.smithi171.stdout:2/344: link d7/d9/d1a/f53 d7/d26/d3d/f66 0 2022-01-31T19:44:21.328 INFO:tasks.workunit.client.1.smithi171.stdout:2/345: creat d7/d26/d63/f67 x:0 0 0 2022-01-31T19:44:21.328 INFO:tasks.workunit.client.1.smithi171.stdout:2/346: write d7/d26/d3d/f42 [4538743,28228] 0 2022-01-31T19:44:21.330 INFO:tasks.workunit.client.1.smithi171.stdout:2/347: read d7/d9/f4e [1017966,72038] 0 2022-01-31T19:44:21.331 INFO:tasks.workunit.client.1.smithi171.stdout:2/348: mkdir d7/d9/d68 0 2022-01-31T19:44:21.332 INFO:tasks.workunit.client.1.smithi171.stdout:2/349: fsync d7/d9/f4e 0 2022-01-31T19:44:21.332 INFO:tasks.workunit.client.1.smithi171.stdout:2/350: creat d7/dd/d5e/f69 x:0 0 0 2022-01-31T19:44:21.364 INFO:tasks.workunit.client.0.smithi167.stdout:3/940: sync 2022-01-31T19:44:21.365 INFO:tasks.workunit.client.0.smithi167.stdout:6/736: sync 2022-01-31T19:44:21.365 INFO:tasks.workunit.client.0.smithi167.stdout:0/654: sync 2022-01-31T19:44:21.365 INFO:tasks.workunit.client.0.smithi167.stdout:6/737: fdatasync d8/d33/d51/d72/dc2/fce 0 2022-01-31T19:44:21.366 INFO:tasks.workunit.client.0.smithi167.stdout:9/848: sync 2022-01-31T19:44:21.366 INFO:tasks.workunit.client.0.smithi167.stdout:3/941: write d9/d21/f23 [8881357,121640] 0 2022-01-31T19:44:21.366 INFO:tasks.workunit.client.0.smithi167.stdout:4/965: sync 2022-01-31T19:44:21.366 INFO:tasks.workunit.client.0.smithi167.stdout:1/992: sync 2022-01-31T19:44:21.367 INFO:tasks.workunit.client.0.smithi167.stdout:8/864: sync 2022-01-31T19:44:21.367 INFO:tasks.workunit.client.0.smithi167.stdout:1/993: chown d0/f93 28 1 2022-01-31T19:44:21.367 INFO:tasks.workunit.client.0.smithi167.stdout:8/865: write d6/db/fd2 [234847,12924] 0 2022-01-31T19:44:21.367 INFO:tasks.workunit.client.0.smithi167.stdout:2/691: sync 2022-01-31T19:44:21.368 INFO:tasks.workunit.client.0.smithi167.stdout:4/966: dread d0/d5d/d12f/f19 [0,4194304] 0 2022-01-31T19:44:21.381 INFO:tasks.workunit.client.0.smithi167.stdout:1/994: dread d0/f93 [4194304,4194304] 0 2022-01-31T19:44:21.381 INFO:tasks.workunit.client.0.smithi167.stdout:1/995: creat d0/d62/db7/f14f x:0 0 0 2022-01-31T19:44:21.381 INFO:tasks.workunit.client.0.smithi167.stdout:1/996: truncate d0/d10/d5a/d33/d4b/d1f/f110 768207 0 2022-01-31T19:44:21.382 INFO:tasks.workunit.client.0.smithi167.stdout:1/997: chown d0/d75/dbd 252 1 2022-01-31T19:44:21.382 INFO:tasks.workunit.client.0.smithi167.stdout:1/998: write d0/d62/d7f/d82/fe2 [2016,51195] 0 2022-01-31T19:44:21.461 INFO:tasks.workunit.client.0.smithi167.stdout:0/655: mknod d17/d19/d1c/d39/d44/d49/cdd 0 2022-01-31T19:44:21.461 INFO:tasks.workunit.client.0.smithi167.stdout:0/656: write d17/d69/f9e [152394,100921] 0 2022-01-31T19:44:21.462 INFO:tasks.workunit.client.0.smithi167.stdout:0/657: chown d17/d3e/c40 1540 1 2022-01-31T19:44:21.462 INFO:tasks.workunit.client.0.smithi167.stdout:0/658: dread - d17/d19/d1c/d39/d44/f56 zero size 2022-01-31T19:44:21.462 INFO:tasks.workunit.client.0.smithi167.stdout:0/659: dread - d17/d19/fb1 zero size 2022-01-31T19:44:21.463 INFO:tasks.workunit.client.0.smithi167.stdout:6/738: mkdir d8/d33/d51/d6a/df1/d106 0 2022-01-31T19:44:21.463 INFO:tasks.workunit.client.0.smithi167.stdout:9/849: mkdir dd/d19/d59/df9/d121 0 2022-01-31T19:44:21.465 INFO:tasks.workunit.client.0.smithi167.stdout:3/942: chown d9/d29/d60/caf 23 1 2022-01-31T19:44:21.465 INFO:tasks.workunit.client.0.smithi167.stdout:3/943: write d9/d7f/da8/f112 [599105,43280] 0 2022-01-31T19:44:21.467 INFO:tasks.workunit.client.0.smithi167.stdout:8/866: truncate d6/db/d6a/d8d/d9e/fcc 104219 0 2022-01-31T19:44:21.467 INFO:tasks.workunit.client.0.smithi167.stdout:8/867: dread - d6/d1e/d2d/d6b/ffe zero size 2022-01-31T19:44:21.469 INFO:tasks.workunit.client.0.smithi167.stdout:2/692: unlink dd/df/d1b/d3d/d4a/d4b/d74/lb1 0 2022-01-31T19:44:21.471 INFO:tasks.workunit.client.0.smithi167.stdout:4/967: creat d0/d5d/d12f/ddf/d75/dce/f13e x:0 0 0 2022-01-31T19:44:21.473 INFO:tasks.workunit.client.0.smithi167.stdout:1/999: mknod d0/d10/d5a/d33/d4b/da5/dde/d11c/d14b/c150 0 2022-01-31T19:44:21.474 INFO:tasks.workunit.client.0.smithi167.stdout:4/968: write d0/d5d/d12f/d5c/fea [654357,123877] 0 2022-01-31T19:44:21.475 INFO:tasks.workunit.client.0.smithi167.stdout:3/944: dread d9/fd3 [0,4194304] 0 2022-01-31T19:44:21.475 INFO:tasks.workunit.client.0.smithi167.stdout:0/660: mkdir d17/d3e/d7e/dc7/dde 0 2022-01-31T19:44:21.475 INFO:tasks.workunit.client.0.smithi167.stdout:6/739: getdents d8/d33/d51/d5d/d64 0 2022-01-31T19:44:21.478 INFO:tasks.workunit.client.0.smithi167.stdout:8/868: unlink l2 0 2022-01-31T19:44:21.479 INFO:tasks.workunit.client.0.smithi167.stdout:2/693: symlink dd/dcb/lde 0 2022-01-31T19:44:21.479 INFO:tasks.workunit.client.0.smithi167.stdout:2/694: chown dd/df/d3f/d51/l57 1136 1 2022-01-31T19:44:21.481 INFO:tasks.workunit.client.0.smithi167.stdout:3/945: creat d9/d7f/f11d x:0 0 0 2022-01-31T19:44:21.483 INFO:tasks.workunit.client.0.smithi167.stdout:6/740: creat d8/d33/d51/d7e/dc4/f107 x:0 0 0 2022-01-31T19:44:21.484 INFO:tasks.workunit.client.0.smithi167.stdout:0/661: dread d17/d2c/f60 [0,4194304] 0 2022-01-31T19:44:21.484 INFO:tasks.workunit.client.0.smithi167.stdout:2/695: mkdir dd/df/d1b/d3d/d4a/d4b/d54/dca/ddf 0 2022-01-31T19:44:21.486 INFO:tasks.workunit.client.0.smithi167.stdout:6/741: mkdir d8/d20/d103/d108 0 2022-01-31T19:44:21.487 INFO:tasks.workunit.client.0.smithi167.stdout:6/742: chown d8/d33/d51/d6a/df1 10557773 1 2022-01-31T19:44:21.487 INFO:tasks.workunit.client.0.smithi167.stdout:6/743: fdatasync d8/d33/d51/d7e/dc4/fba 0 2022-01-31T19:44:21.487 INFO:tasks.workunit.client.0.smithi167.stdout:6/744: readlink d8/d33/d51/d72/dad/le9 0 2022-01-31T19:44:21.488 INFO:tasks.workunit.client.0.smithi167.stdout:4/969: creat d0/d6/d7/d89/d11c/d130/f13f x:0 0 0 2022-01-31T19:44:21.488 INFO:tasks.workunit.client.0.smithi167.stdout:6/745: creat d8/d33/d35/f109 x:0 0 0 2022-01-31T19:44:21.489 INFO:tasks.workunit.client.0.smithi167.stdout:6/746: truncate d8/d33/d51/d72/dc2/fce 732765 0 2022-01-31T19:44:21.489 INFO:tasks.workunit.client.0.smithi167.stdout:6/747: readlink d8/d33/d51/d72/dad/le9 0 2022-01-31T19:44:21.489 INFO:tasks.workunit.client.0.smithi167.stdout:6/748: write d8/d33/d35/f109 [515852,809] 0 2022-01-31T19:44:21.490 INFO:tasks.workunit.client.0.smithi167.stdout:4/970: mkdir d0/d6/d140 0 2022-01-31T19:44:21.490 INFO:tasks.workunit.client.0.smithi167.stdout:4/971: chown d0/d5d/d12f/ddf/d75/dce/f12e 7544598 1 2022-01-31T19:44:21.490 INFO:tasks.workunit.client.0.smithi167.stdout:4/972: fdatasync d0/d5d/d12f/f19 0 2022-01-31T19:44:21.491 INFO:tasks.workunit.client.0.smithi167.stdout:6/749: rmdir d8/d20/d103 39 2022-01-31T19:44:21.498 INFO:tasks.workunit.client.0.smithi167.stdout:9/850: dwrite dd/d19/d24/d56/f75 [0,4194304] 0 2022-01-31T19:44:21.499 INFO:tasks.workunit.client.0.smithi167.stdout:9/851: chown dd/d17/f36 64620 1 2022-01-31T19:44:21.499 INFO:tasks.workunit.client.0.smithi167.stdout:9/852: fsync dd/d35/f8d 0 2022-01-31T19:44:21.507 INFO:tasks.workunit.client.0.smithi167.stdout:6/750: write d8/d20/f22 [4187977,87450] 0 2022-01-31T19:44:21.510 INFO:tasks.workunit.client.0.smithi167.stdout:8/869: dwrite d6/d1e/d4c/d80/fb6 [0,4194304] 0 2022-01-31T19:44:21.514 INFO:tasks.workunit.client.0.smithi167.stdout:9/853: dread dd/d17/d7f/fe1 [0,4194304] 0 2022-01-31T19:44:21.514 INFO:tasks.workunit.client.0.smithi167.stdout:9/854: write dd/d17/d80/f43 [1790746,103828] 0 2022-01-31T19:44:21.514 INFO:tasks.workunit.client.0.smithi167.stdout:6/751: dread d8/d20/f47 [0,4194304] 0 2022-01-31T19:44:21.514 INFO:tasks.workunit.client.0.smithi167.stdout:6/752: fdatasync d8/d33/d51/d5d/d64/f101 0 2022-01-31T19:44:21.515 INFO:tasks.workunit.client.0.smithi167.stdout:8/870: unlink d6/l10e 0 2022-01-31T19:44:21.515 INFO:tasks.workunit.client.0.smithi167.stdout:8/871: chown d6/db/d28/d61/dd5/df0/c110 1695703 1 2022-01-31T19:44:21.516 INFO:tasks.workunit.client.0.smithi167.stdout:8/872: chown d6/d1e/d2d/lae 71086 1 2022-01-31T19:44:21.517 INFO:tasks.workunit.client.0.smithi167.stdout:9/855: symlink dd/d19/d24/d50/dd1/l122 0 2022-01-31T19:44:21.526 INFO:tasks.workunit.client.0.smithi167.stdout:3/946: dwrite d9/d29/d3b/d9b/da3/f10c [4194304,4194304] 0 2022-01-31T19:44:21.527 INFO:tasks.workunit.client.0.smithi167.stdout:4/973: dwrite d0/d6/d7/d39/d4e/ffb [0,4194304] 0 2022-01-31T19:44:21.529 INFO:tasks.workunit.client.0.smithi167.stdout:3/947: rename d9/d21/ld0 to d9/d29/d3b/d9b/l11e 0 2022-01-31T19:44:21.530 INFO:tasks.workunit.client.0.smithi167.stdout:4/974: mkdir d0/d6/d140/d141 0 2022-01-31T19:44:21.531 INFO:tasks.workunit.client.0.smithi167.stdout:3/948: getdents d9/d29/d3b/d41 0 2022-01-31T19:44:21.532 INFO:tasks.workunit.client.0.smithi167.stdout:4/975: creat d0/d51/df6/f142 x:0 0 0 2022-01-31T19:44:21.532 INFO:tasks.workunit.client.0.smithi167.stdout:3/949: creat d9/d29/d60/dc9/f11f x:0 0 0 2022-01-31T19:44:21.533 INFO:tasks.workunit.client.0.smithi167.stdout:0/662: dwrite d17/d19/d1c/d39/d44/f65 [0,4194304] 0 2022-01-31T19:44:21.533 INFO:tasks.workunit.client.0.smithi167.stdout:0/663: write d17/d33/fa3 [416130,14255] 0 2022-01-31T19:44:21.534 INFO:tasks.workunit.client.0.smithi167.stdout:4/976: rename d0/d5d/fa5 to d0/d5d/d12f/ddf/da2/dcf/db5/d117/f143 0 2022-01-31T19:44:21.536 INFO:tasks.workunit.client.0.smithi167.stdout:3/950: mknod d9/d7f/d10a/c120 0 2022-01-31T19:44:21.538 INFO:tasks.workunit.client.0.smithi167.stdout:0/664: getdents d17 0 2022-01-31T19:44:21.538 INFO:tasks.workunit.client.0.smithi167.stdout:0/665: read - d17/d2c/d87/fc4 zero size 2022-01-31T19:44:21.539 INFO:tasks.workunit.client.0.smithi167.stdout:0/666: mkdir d17/d19/d1c/d39/d44/d88/ddf 0 2022-01-31T19:44:21.540 INFO:tasks.workunit.client.0.smithi167.stdout:0/667: creat d17/d3e/da1/fe0 x:0 0 0 2022-01-31T19:44:21.540 INFO:tasks.workunit.client.0.smithi167.stdout:3/951: write d9/d29/d3b/fee [4007009,28032] 0 2022-01-31T19:44:21.541 INFO:tasks.workunit.client.0.smithi167.stdout:3/952: creat d9/d29/d3b/d41/d6b/d100/f121 x:0 0 0 2022-01-31T19:44:21.542 INFO:tasks.workunit.client.0.smithi167.stdout:2/696: dwrite dd/df/d1b/d3d/d4a/d4b/d54/fbd [0,4194304] 0 2022-01-31T19:44:21.542 INFO:tasks.workunit.client.0.smithi167.stdout:3/953: dread d9/d29/d3b/d9b/da3/db0/fb9 [0,4194304] 0 2022-01-31T19:44:21.543 INFO:tasks.workunit.client.0.smithi167.stdout:6/753: dwrite d8/d20/f42 [0,4194304] 0 2022-01-31T19:44:21.547 INFO:tasks.workunit.client.0.smithi167.stdout:4/977: dread d0/d51/df6/f10a [0,4194304] 0 2022-01-31T19:44:21.547 INFO:tasks.workunit.client.0.smithi167.stdout:4/978: chown d0/d5d/d12f/ddf/d1a/c6e 5911 1 2022-01-31T19:44:21.550 INFO:tasks.workunit.client.0.smithi167.stdout:2/697: write dd/d18/d63/fa3 [3399156,27341] 0 2022-01-31T19:44:21.550 INFO:tasks.workunit.client.0.smithi167.stdout:2/698: write dd/d18/f3a [3427508,60562] 0 2022-01-31T19:44:21.551 INFO:tasks.workunit.client.0.smithi167.stdout:3/954: mknod d9/d7f/dbb/c122 0 2022-01-31T19:44:21.552 INFO:tasks.workunit.client.0.smithi167.stdout:6/754: dread d8/d20/d54/f63 [0,4194304] 0 2022-01-31T19:44:21.555 INFO:tasks.workunit.client.0.smithi167.stdout:4/979: mkdir d0/d5d/d144 0 2022-01-31T19:44:21.556 INFO:tasks.workunit.client.0.smithi167.stdout:2/699: symlink dd/df/d1b/d3d/le0 0 2022-01-31T19:44:21.556 INFO:tasks.workunit.client.0.smithi167.stdout:2/700: read - dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b zero size 2022-01-31T19:44:21.557 INFO:tasks.workunit.client.0.smithi167.stdout:2/701: write dd/df/d3f/d51/fad [546564,41814] 0 2022-01-31T19:44:21.557 INFO:tasks.workunit.client.0.smithi167.stdout:6/755: creat d8/d33/d51/d72/dc7/f10a x:0 0 0 2022-01-31T19:44:21.560 INFO:tasks.workunit.client.0.smithi167.stdout:4/980: unlink d0/d6/d7/d89/d11c/c13b 0 2022-01-31T19:44:21.566 INFO:tasks.workunit.client.0.smithi167.stdout:3/955: dwrite d9/d29/d3b/d41/f5c [4194304,4194304] 0 2022-01-31T19:44:21.569 INFO:tasks.workunit.client.0.smithi167.stdout:3/956: fdatasync d9/d29/d60/fed 0 2022-01-31T19:44:21.569 INFO:tasks.workunit.client.0.smithi167.stdout:0/668: dwrite d17/d3e/fae [0,4194304] 0 2022-01-31T19:44:21.572 INFO:tasks.workunit.client.0.smithi167.stdout:0/669: read d17/d19/f4d [2396393,59639] 0 2022-01-31T19:44:21.579 INFO:tasks.workunit.client.0.smithi167.stdout:0/670: write d17/d3e/f42 [497863,124115] 0 2022-01-31T19:44:21.580 INFO:tasks.workunit.client.0.smithi167.stdout:0/671: mkdir d17/d19/d1c/de1 0 2022-01-31T19:44:21.582 INFO:tasks.workunit.client.0.smithi167.stdout:2/702: dwrite dd/df/d1b/d3d/d4a/fae [0,4194304] 0 2022-01-31T19:44:21.583 INFO:tasks.workunit.client.0.smithi167.stdout:0/672: rename d17/d2c/d87/c90 to d17/d19/d1c/d39/d7b/dbd/dd9/ce2 0 2022-01-31T19:44:21.583 INFO:tasks.workunit.client.0.smithi167.stdout:0/673: chown d17/d3e/da1 13964 1 2022-01-31T19:44:21.583 INFO:tasks.workunit.client.0.smithi167.stdout:0/674: write d17/d3e/d7e/dc7/fd7 [487332,773] 0 2022-01-31T19:44:21.595 INFO:tasks.workunit.client.0.smithi167.stdout:9/856: dwrite dd/d19/d24/f45 [0,4194304] 0 2022-01-31T19:44:21.595 INFO:tasks.workunit.client.0.smithi167.stdout:2/703: dread dd/df/d22/f9d [0,4194304] 0 2022-01-31T19:44:21.595 INFO:tasks.workunit.client.0.smithi167.stdout:2/704: chown dd/df/d1b/d3d/d4a/d4b 3465 1 2022-01-31T19:44:21.603 INFO:tasks.workunit.client.0.smithi167.stdout:6/756: dwrite d8/d33/d4f/f8e [0,4194304] 0 2022-01-31T19:44:21.603 INFO:tasks.workunit.client.0.smithi167.stdout:6/757: chown d8/c1b 28879 1 2022-01-31T19:44:21.604 INFO:tasks.workunit.client.0.smithi167.stdout:0/675: dwrite d17/d69/f94 [0,4194304] 0 2022-01-31T19:44:21.605 INFO:tasks.workunit.client.0.smithi167.stdout:4/981: dwrite d0/d6/d7/d39/d4e/f52 [0,4194304] 0 2022-01-31T19:44:21.605 INFO:tasks.workunit.client.0.smithi167.stdout:4/982: chown d0/d6/d7/d39/f12c 410 1 2022-01-31T19:44:21.613 INFO:tasks.workunit.client.0.smithi167.stdout:2/705: dwrite dd/d18/d8d/fa5 [0,4194304] 0 2022-01-31T19:44:21.621 INFO:tasks.workunit.client.0.smithi167.stdout:2/706: dread f7 [0,4194304] 0 2022-01-31T19:44:21.622 INFO:tasks.workunit.client.0.smithi167.stdout:0/676: dwrite d17/d2c/dcd/fd6 [0,4194304] 0 2022-01-31T19:44:21.622 INFO:tasks.workunit.client.0.smithi167.stdout:0/677: read - d17/d3e/f9b zero size 2022-01-31T19:44:21.622 INFO:tasks.workunit.client.0.smithi167.stdout:0/678: read - d17/d3e/d7e/fce zero size 2022-01-31T19:44:21.634 INFO:tasks.workunit.client.0.smithi167.stdout:9/857: mkdir dd/d34/d123 0 2022-01-31T19:44:21.634 INFO:tasks.workunit.client.0.smithi167.stdout:9/858: fsync dd/d19/d24/d56/dc9/fd4 0 2022-01-31T19:44:21.636 INFO:tasks.workunit.client.0.smithi167.stdout:2/707: dwrite dd/df/f50 [0,4194304] 0 2022-01-31T19:44:21.638 INFO:tasks.workunit.client.0.smithi167.stdout:2/708: write dd/d18/f3a [828966,107345] 0 2022-01-31T19:44:21.646 INFO:tasks.workunit.client.0.smithi167.stdout:4/983: getdents d0/d5d/d12f/d5c/de2 0 2022-01-31T19:44:21.649 INFO:tasks.workunit.client.0.smithi167.stdout:9/859: dwrite dd/d17/d7f/fb2 [0,4194304] 0 2022-01-31T19:44:21.662 INFO:tasks.workunit.client.0.smithi167.stdout:2/709: mknod dd/df/d1b/d3d/d4a/da7/ce1 0 2022-01-31T19:44:21.665 INFO:tasks.workunit.client.0.smithi167.stdout:4/984: creat d0/d5d/d12f/ddf/d75/f145 x:0 0 0 2022-01-31T19:44:21.665 INFO:tasks.workunit.client.0.smithi167.stdout:4/985: write d0/d6/d7/d26/fc9 [1819709,10950] 0 2022-01-31T19:44:21.667 INFO:tasks.workunit.client.0.smithi167.stdout:9/860: link dd/d19/d24/d50/c6a dd/d17/d7f/def/c124 0 2022-01-31T19:44:21.672 INFO:tasks.workunit.client.0.smithi167.stdout:0/679: getdents d17/d33/da4/dd0 0 2022-01-31T19:44:21.674 INFO:tasks.workunit.client.0.smithi167.stdout:2/710: creat dd/df/d1b/d3d/d4a/d4b/d74/d85/fe2 x:0 0 0 2022-01-31T19:44:21.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/711: fdatasync dd/df/d1b/d2f/d36/fb9 0 2022-01-31T19:44:21.675 INFO:tasks.workunit.client.0.smithi167.stdout:2/712: chown dd/d18/d63/f2d 989015 1 2022-01-31T19:44:21.676 INFO:tasks.workunit.client.0.smithi167.stdout:9/861: unlink dd/d6b/f93 0 2022-01-31T19:44:21.680 INFO:tasks.workunit.client.0.smithi167.stdout:2/713: mknod dd/d18/ce3 0 2022-01-31T19:44:21.681 INFO:tasks.workunit.client.0.smithi167.stdout:2/714: fdatasync dd/df/d22/f9d 0 2022-01-31T19:44:21.681 INFO:tasks.workunit.client.0.smithi167.stdout:2/715: readlink dd/df/d22/l90 0 2022-01-31T19:44:21.682 INFO:tasks.workunit.client.0.smithi167.stdout:2/716: rename dd/d18/ce3 to dd/df/d3f/d51/d5f/ce4 0 2022-01-31T19:44:21.683 INFO:tasks.workunit.client.0.smithi167.stdout:2/717: creat dd/df/d1b/d3d/d4a/d4b/d54/dca/fe5 x:0 0 0 2022-01-31T19:44:21.685 INFO:tasks.workunit.client.0.smithi167.stdout:2/718: mkdir dd/df/d22/d96/d9e/dda/de6 0 2022-01-31T19:44:21.698 INFO:tasks.workunit.client.0.smithi167.stdout:9/862: dwrite dd/d19/d24/d56/f75 [4194304,4194304] 0 2022-01-31T19:44:21.698 INFO:tasks.workunit.client.0.smithi167.stdout:9/863: chown dd/d19/d24/f45 98623047 1 2022-01-31T19:44:21.698 INFO:tasks.workunit.client.0.smithi167.stdout:9/864: truncate dd/d35/ff0 275573 0 2022-01-31T19:44:21.699 INFO:tasks.workunit.client.0.smithi167.stdout:0/680: dwrite d17/d19/d1c/fc3 [0,4194304] 0 2022-01-31T19:44:21.700 INFO:tasks.workunit.client.0.smithi167.stdout:0/681: read - d17/d19/d1c/d39/fc5 zero size 2022-01-31T19:44:21.700 INFO:tasks.workunit.client.0.smithi167.stdout:2/719: dread dd/d18/f3a [0,4194304] 0 2022-01-31T19:44:21.701 INFO:tasks.workunit.client.0.smithi167.stdout:9/865: truncate dd/d34/f51 3160164 0 2022-01-31T19:44:21.701 INFO:tasks.workunit.client.0.smithi167.stdout:9/866: write dd/d19/d24/fb6 [912342,27076] 0 2022-01-31T19:44:21.704 INFO:tasks.workunit.client.0.smithi167.stdout:0/682: symlink d17/d33/le3 0 2022-01-31T19:44:21.704 INFO:tasks.workunit.client.0.smithi167.stdout:2/720: creat dd/df/d3f/d51/d5f/fe7 x:0 0 0 2022-01-31T19:44:21.706 INFO:tasks.workunit.client.0.smithi167.stdout:0/683: rename d17/d33/f6f to d17/d19/d1c/d39/d44/d88/d98/fe4 0 2022-01-31T19:44:21.726 INFO:tasks.workunit.client.0.smithi167.stdout:9/867: dwrite dd/d34/f61 [0,4194304] 0 2022-01-31T19:44:21.727 INFO:tasks.workunit.client.0.smithi167.stdout:2/721: dwrite dd/df/d1b/d3d/d4a/f5b [0,4194304] 0 2022-01-31T19:44:21.730 INFO:tasks.workunit.client.0.smithi167.stdout:2/722: mknod dd/df/d1b/d2f/ce8 0 2022-01-31T19:44:21.730 INFO:tasks.workunit.client.0.smithi167.stdout:2/723: stat dd/df/d1b/d2f/ca0 0 2022-01-31T19:44:21.731 INFO:tasks.workunit.client.0.smithi167.stdout:2/724: rmdir dd/df/d1b 39 2022-01-31T19:44:21.732 INFO:tasks.workunit.client.0.smithi167.stdout:2/725: write dd/f55 [8260463,100094] 0 2022-01-31T19:44:21.734 INFO:tasks.workunit.client.0.smithi167.stdout:2/726: rename dd/d18/d63/dab/lc7 to dd/df/d1b/d3d/d4a/d4b/d74/d85/le9 0 2022-01-31T19:44:21.734 INFO:tasks.workunit.client.0.smithi167.stdout:2/727: truncate dd/d18/d62/fa6 2032988 0 2022-01-31T19:44:21.735 INFO:tasks.workunit.client.0.smithi167.stdout:2/728: mkdir dd/d18/d62/dea 0 2022-01-31T19:44:21.743 INFO:tasks.workunit.client.0.smithi167.stdout:9/868: dwrite dd/d19/d24/d56/d5b/da5/fed [0,4194304] 0 2022-01-31T19:44:21.745 INFO:tasks.workunit.client.0.smithi167.stdout:9/869: unlink dd/d17/d7f/fcb 0 2022-01-31T19:44:21.746 INFO:tasks.workunit.client.0.smithi167.stdout:9/870: mkdir dd/d19/d24/d56/dc9/d125 0 2022-01-31T19:44:21.747 INFO:tasks.workunit.client.0.smithi167.stdout:9/871: rename dd/c8b to dd/d19/d24/d56/dc9/d125/c126 0 2022-01-31T19:44:21.747 INFO:tasks.workunit.client.0.smithi167.stdout:9/872: chown dd/d34/cdb 4 1 2022-01-31T19:44:21.748 INFO:tasks.workunit.client.0.smithi167.stdout:9/873: mknod dd/d19/d24/d50/dd1/c127 0 2022-01-31T19:44:21.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/874: creat dd/d34/f128 x:0 0 0 2022-01-31T19:44:21.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/875: chown dd/d19/d24/d56/d5b/c89 3 1 2022-01-31T19:44:21.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/876: chown dd/d19/d24/fca 179668246 1 2022-01-31T19:44:21.749 INFO:tasks.workunit.client.0.smithi167.stdout:9/877: readlink dd/d17/dda/l109 0 2022-01-31T19:44:21.750 INFO:tasks.workunit.client.0.smithi167.stdout:9/878: creat dd/f129 x:0 0 0 2022-01-31T19:44:21.750 INFO:tasks.workunit.client.0.smithi167.stdout:9/879: chown dd/d34/l39 7333 1 2022-01-31T19:44:21.751 INFO:tasks.workunit.client.0.smithi167.stdout:9/880: truncate dd/d17/d80/f43 762679 0 2022-01-31T19:44:21.751 INFO:tasks.workunit.client.0.smithi167.stdout:9/881: mknod dd/d19/d24/d50/dd6/c12a 0 2022-01-31T19:44:21.752 INFO:tasks.workunit.client.0.smithi167.stdout:9/882: readlink dd/d19/d24/d50/d5d/l81 0 2022-01-31T19:44:21.752 INFO:tasks.workunit.client.0.smithi167.stdout:2/729: dwrite dd/df/d1b/d3d/d4a/da7/fbb [4194304,4194304] 0 2022-01-31T19:44:21.753 INFO:tasks.workunit.client.0.smithi167.stdout:2/730: stat dd/d18 0 2022-01-31T19:44:21.753 INFO:tasks.workunit.client.0.smithi167.stdout:2/731: write dd/f29 [2644856,69480] 0 2022-01-31T19:44:21.757 INFO:tasks.workunit.client.0.smithi167.stdout:2/732: rename dd/df/d1b/d3d/f44 to dd/d18/db7/feb 0 2022-01-31T19:44:21.773 INFO:tasks.workunit.client.0.smithi167.stdout:9/883: dwrite dd/d35/f65 [0,4194304] 0 2022-01-31T19:44:21.774 INFO:tasks.workunit.client.0.smithi167.stdout:9/884: getdents dd/d19/d24/d50/dd6/d11b 0 2022-01-31T19:44:21.774 INFO:tasks.workunit.client.0.smithi167.stdout:9/885: unlink dd/d6b/l10e 0 2022-01-31T19:44:21.774 INFO:tasks.workunit.client.0.smithi167.stdout:9/886: read - dd/d17/dc4/de2/f11e zero size 2022-01-31T19:44:21.776 INFO:tasks.workunit.client.0.smithi167.stdout:2/733: dwrite dd/d18/f99 [0,4194304] 0 2022-01-31T19:44:21.780 INFO:tasks.workunit.client.0.smithi167.stdout:2/734: symlink dd/df/d22/d96/d9e/dda/lec 0 2022-01-31T19:44:21.782 INFO:tasks.workunit.client.0.smithi167.stdout:2/735: symlink dd/df/d22/d96/d9e/dda/de6/led 0 2022-01-31T19:44:21.788 INFO:tasks.workunit.client.0.smithi167.stdout:9/887: dwrite dd/d35/ff0 [0,4194304] 0 2022-01-31T19:44:21.791 INFO:tasks.workunit.client.0.smithi167.stdout:2/736: truncate dd/df/f9a 735144 0 2022-01-31T19:44:21.791 INFO:tasks.workunit.client.0.smithi167.stdout:2/737: creat dd/df/d1b/d3d/d4a/d4b/d74/d85/fee x:0 0 0 2022-01-31T19:44:21.793 INFO:tasks.workunit.client.0.smithi167.stdout:9/888: creat dd/d19/d24/d50/dd6/d11b/f12b x:0 0 0 2022-01-31T19:44:21.794 INFO:tasks.workunit.client.0.smithi167.stdout:9/889: write dd/d19/d24/d56/dc9/fd7 [3139388,15978] 0 2022-01-31T19:44:21.795 INFO:tasks.workunit.client.0.smithi167.stdout:2/738: truncate dd/d18/fa9 3456454 0 2022-01-31T19:44:21.797 INFO:tasks.workunit.client.0.smithi167.stdout:9/890: creat dd/d6b/f12c x:0 0 0 2022-01-31T19:44:21.798 INFO:tasks.workunit.client.0.smithi167.stdout:9/891: write dd/d35/f44 [2135636,91130] 0 2022-01-31T19:44:21.799 INFO:tasks.workunit.client.0.smithi167.stdout:2/739: rename dd/df/d1b/d3d/d4a/d4b/d54/dca/fe5 to dd/df/d1b/d3d/d4a/da7/dcc/fef 0 2022-01-31T19:44:21.801 INFO:tasks.workunit.client.0.smithi167.stdout:2/740: mknod dd/d18/d63/cf0 0 2022-01-31T19:44:21.802 INFO:tasks.workunit.client.0.smithi167.stdout:2/741: read dd/df/d1b/d2f/f34 [167804,57684] 0 2022-01-31T19:44:21.807 INFO:tasks.workunit.client.0.smithi167.stdout:9/892: dread dd/d19/d59/fce [0,4194304] 0 2022-01-31T19:44:21.807 INFO:tasks.workunit.client.0.smithi167.stdout:9/893: chown dd/d17 14834 1 2022-01-31T19:44:21.809 INFO:tasks.workunit.client.0.smithi167.stdout:9/894: link dd/d19/d24/c74 dd/d19/d24/d50/d5d/d117/c12d 0 2022-01-31T19:44:21.810 INFO:tasks.workunit.client.0.smithi167.stdout:9/895: mkdir dd/d19/d24/d56/dc9/d125/d12e 0 2022-01-31T19:44:21.811 INFO:tasks.workunit.client.0.smithi167.stdout:2/742: dread dd/f23 [0,4194304] 0 2022-01-31T19:44:21.812 INFO:tasks.workunit.client.0.smithi167.stdout:2/743: creat dd/d18/d63/ff1 x:0 0 0 2022-01-31T19:44:21.813 INFO:tasks.workunit.client.0.smithi167.stdout:2/744: write dd/df/d3f/d51/fd4 [1666180,120223] 0 2022-01-31T19:44:21.813 INFO:tasks.workunit.client.0.smithi167.stdout:2/745: creat dd/df/ff2 x:0 0 0 2022-01-31T19:44:21.814 INFO:tasks.workunit.client.0.smithi167.stdout:2/746: mknod dd/d18/cf3 0 2022-01-31T19:44:21.832 INFO:tasks.workunit.client.0.smithi167.stdout:9/896: dwrite dd/d19/d24/d50/dd6/f103 [4194304,4194304] 0 2022-01-31T19:44:21.833 INFO:tasks.workunit.client.0.smithi167.stdout:2/747: dwrite dd/df/d1b/d3d/d4a/d4b/d54/f8b [0,4194304] 0 2022-01-31T19:44:21.834 INFO:tasks.workunit.client.0.smithi167.stdout:9/897: truncate dd/f84 139188 0 2022-01-31T19:44:21.836 INFO:tasks.workunit.client.0.smithi167.stdout:9/898: rename dd/f1a to dd/d19/d24/d56/f12f 0 2022-01-31T19:44:21.836 INFO:tasks.workunit.client.0.smithi167.stdout:9/899: dread - dd/d17/d80/f106 zero size 2022-01-31T19:44:21.837 INFO:tasks.workunit.client.0.smithi167.stdout:9/900: dread dd/d19/d24/d56/d5b/da5/fb4 [0,4194304] 0 2022-01-31T19:44:21.838 INFO:tasks.workunit.client.0.smithi167.stdout:9/901: creat dd/d19/d24/d56/dc9/d125/f130 x:0 0 0 2022-01-31T19:44:21.839 INFO:tasks.workunit.client.0.smithi167.stdout:9/902: dread - dd/d19/d24/f111 zero size 2022-01-31T19:44:21.844 INFO:tasks.workunit.client.0.smithi167.stdout:9/903: rename dd/d19/d59/cd2 to dd/d19/d24/d56/d5b/da5/c131 0 2022-01-31T19:44:21.844 INFO:tasks.workunit.client.0.smithi167.stdout:9/904: truncate dd/d19/d24/d56/dc9/f10d 724286 0 2022-01-31T19:44:21.844 INFO:tasks.workunit.client.0.smithi167.stdout:9/905: fsync dd/d6b/dc1/fd0 0 2022-01-31T19:44:21.845 INFO:tasks.workunit.client.0.smithi167.stdout:9/906: write dd/d17/d7f/def/fbd [373028,25266] 0 2022-01-31T19:44:21.846 INFO:tasks.workunit.client.0.smithi167.stdout:9/907: rename dd/d6b/dc1/f110 to dd/d19/d24/d56/dc9/d125/f132 0 2022-01-31T19:44:21.846 INFO:tasks.workunit.client.0.smithi167.stdout:9/908: chown dd/d19/df6 47 1 2022-01-31T19:44:21.846 INFO:tasks.workunit.client.0.smithi167.stdout:9/909: fsync dd/d17/f2f 0 2022-01-31T19:44:21.847 INFO:tasks.workunit.client.0.smithi167.stdout:9/910: link dd/d17/f1f dd/d17/dda/f133 0 2022-01-31T19:44:21.856 INFO:tasks.workunit.client.0.smithi167.stdout:2/748: dwrite dd/f55 [0,4194304] 0 2022-01-31T19:44:21.856 INFO:tasks.workunit.client.0.smithi167.stdout:2/749: fdatasync dd/f23 0 2022-01-31T19:44:21.859 INFO:tasks.workunit.client.0.smithi167.stdout:2/750: write dd/d18/d8d/fa5 [3204772,7488] 0 2022-01-31T19:44:21.875 INFO:tasks.workunit.client.0.smithi167.stdout:2/751: dwrite dd/df/d1b/d3d/d4a/d4b/d54/f8b [0,4194304] 0 2022-01-31T19:44:21.875 INFO:tasks.workunit.client.0.smithi167.stdout:2/752: readlink dd/l19 0 2022-01-31T19:44:21.882 INFO:tasks.workunit.client.0.smithi167.stdout:2/753: dread dd/d18/f4e [0,4194304] 0 2022-01-31T19:44:21.883 INFO:tasks.workunit.client.0.smithi167.stdout:2/754: mknod dd/df/d22/d96/d9e/dda/cf4 0 2022-01-31T19:44:21.884 INFO:tasks.workunit.client.0.smithi167.stdout:2/755: mkdir dd/df/d1b/d3d/d4a/da7/dcc/df5 0 2022-01-31T19:44:21.884 INFO:tasks.workunit.client.0.smithi167.stdout:2/756: write dd/d18/d62/f6e [306914,82357] 0 2022-01-31T19:44:21.885 INFO:tasks.workunit.client.0.smithi167.stdout:2/757: link dd/d18/d8d/cc0 dd/df/d1b/d3d/d4a/d4b/d74/cf6 0 2022-01-31T19:44:21.886 INFO:tasks.workunit.client.0.smithi167.stdout:2/758: mknod dd/df/d1b/d3d/d4a/da7/dcc/cf7 0 2022-01-31T19:44:21.887 INFO:tasks.workunit.client.0.smithi167.stdout:2/759: truncate dd/df/d3f/d51/fad 207170 0 2022-01-31T19:44:21.888 INFO:tasks.workunit.client.0.smithi167.stdout:2/760: mkdir dd/d18/d62/dea/df8 0 2022-01-31T19:44:21.902 INFO:tasks.workunit.client.0.smithi167.stdout:2/761: dwrite dd/df/f9a [0,4194304] 0 2022-01-31T19:44:21.903 INFO:tasks.workunit.client.0.smithi167.stdout:2/762: getdents dd/df/d1b/d3d 0 2022-01-31T19:44:21.904 INFO:tasks.workunit.client.0.smithi167.stdout:2/763: write dd/d18/f4e [4392343,118647] 0 2022-01-31T19:44:21.905 INFO:tasks.workunit.client.0.smithi167.stdout:2/764: unlink dd/df/d1b/d3d/d4a/da7/ce1 0 2022-01-31T19:44:21.906 INFO:tasks.workunit.client.0.smithi167.stdout:2/765: write dd/df/d22/f8a [216224,102093] 0 2022-01-31T19:44:21.924 INFO:tasks.workunit.client.0.smithi167.stdout:2/766: dwrite dd/fc1 [0,4194304] 0 2022-01-31T19:44:21.925 INFO:tasks.workunit.client.0.smithi167.stdout:2/767: chown dd/df/d22/d96/d9e 11627287 1 2022-01-31T19:44:21.939 INFO:tasks.workunit.client.0.smithi167.stdout:2/768: mkdir dd/df/d1b/df9 0 2022-01-31T19:44:21.939 INFO:tasks.workunit.client.0.smithi167.stdout:2/769: read - dd/d18/d63/ff1 zero size 2022-01-31T19:44:21.941 INFO:tasks.workunit.client.0.smithi167.stdout:2/770: getdents dd/df/d22/d96/d9e/dda 0 2022-01-31T19:44:21.941 INFO:tasks.workunit.client.0.smithi167.stdout:2/771: truncate dd/df/fd3 855063 0 2022-01-31T19:44:21.942 INFO:tasks.workunit.client.0.smithi167.stdout:2/772: dread dd/df/d3f/f8e [0,4194304] 0 2022-01-31T19:44:21.943 INFO:tasks.workunit.client.0.smithi167.stdout:2/773: symlink dd/df/d1b/d3d/d4a/da7/lfa 0 2022-01-31T19:44:21.959 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:21 smithi171 conmon[35325]: cluster 2022-01-31T19:44:20.434249+0000 2022-01-31T19:44:21.959 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:21 smithi171 conmon[35325]: mgr.smithi171.asyxnc (mgr.24387) 22 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 2.2 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 144 MiB/s rd, 290 MiB/s wr, 432 op/s 2022-01-31T19:44:21.960 INFO:tasks.workunit.client.0.smithi167.stdout:2/774: dwrite dd/f55 [4194304,4194304] 0 2022-01-31T19:44:21.960 INFO:tasks.workunit.client.0.smithi167.stdout:2/775: write fa [502439,283] 0 2022-01-31T19:44:21.962 INFO:tasks.workunit.client.0.smithi167.stdout:2/776: rename dd/df/d3f/d51/d5f/fe7 to dd/d18/db7/ffb 0 2022-01-31T19:44:21.962 INFO:tasks.workunit.client.0.smithi167.stdout:2/777: creat dd/df/d22/d96/d9e/dda/ffc x:0 0 0 2022-01-31T19:44:21.978 INFO:tasks.workunit.client.0.smithi167.stdout:2/778: dwrite dd/df/d3f/d51/f65 [0,4194304] 0 2022-01-31T19:44:21.978 INFO:tasks.workunit.client.0.smithi167.stdout:2/779: fsync dd/df/f9a 0 2022-01-31T19:44:21.980 INFO:tasks.workunit.client.0.smithi167.stdout:2/780: rename dd/df/c40 to dd/d18/d62/d75/cfd 0 2022-01-31T19:44:21.981 INFO:tasks.workunit.client.0.smithi167.stdout:2/781: mkdir dd/df/d1b/d3d/d4a/da7/dfe 0 2022-01-31T19:44:21.981 INFO:tasks.workunit.client.0.smithi167.stdout:2/782: write dd/df/d1b/d3d/d4a/d4b/d74/d85/fee [867497,119257] 0 2022-01-31T19:44:21.982 INFO:tasks.workunit.client.0.smithi167.stdout:2/783: symlink dd/df/d22/d96/d9e/dda/de6/lff 0 2022-01-31T19:44:21.982 INFO:tasks.workunit.client.0.smithi167.stdout:2/784: rmdir dd/df/d3f 39 2022-01-31T19:44:21.984 INFO:tasks.workunit.client.0.smithi167.stdout:2/785: rename dd/df/ff2 to dd/df/d22/d96/d9e/dda/f100 0 2022-01-31T19:44:21.984 INFO:tasks.workunit.client.0.smithi167.stdout:2/786: truncate dd/df/d1b/f6a 1310994 0 2022-01-31T19:44:21.985 INFO:tasks.workunit.client.0.smithi167.stdout:2/787: creat dd/df/d1b/d2f/f101 x:0 0 0 2022-01-31T19:44:21.985 INFO:tasks.workunit.client.0.smithi167.stdout:2/788: mknod dd/df/d1b/d3d/d4a/d4b/d54/c102 0 2022-01-31T19:44:21.986 INFO:tasks.workunit.client.0.smithi167.stdout:2/789: write dd/d18/d63/f2d [3089329,80948] 0 2022-01-31T19:44:21.995 INFO:tasks.workunit.client.0.smithi167.stdout:2/790: dread f7 [0,4194304] 0 2022-01-31T19:44:21.995 INFO:tasks.workunit.client.0.smithi167.stdout:2/791: readlink dd/df/d22/d96/d9e/dda/de6/led 0 2022-01-31T19:44:21.996 INFO:tasks.workunit.client.0.smithi167.stdout:2/792: creat dd/d18/d63/dab/f103 x:0 0 0 2022-01-31T19:44:22.001 INFO:tasks.workunit.client.0.smithi167.stdout:2/793: dread dd/df/d3f/d51/fd4 [0,4194304] 0 2022-01-31T19:44:22.001 INFO:tasks.workunit.client.0.smithi167.stdout:2/794: creat dd/df/f104 x:0 0 0 2022-01-31T19:44:22.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:21 smithi167 conmon[32206]: cluster 2022-01-31T19:44:20.434249+0000 mgr.smithi171.asyxnc (mgr.24387) 22 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 2.2 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 144 MiB/s rd, 290 MiB/s wr, 432 op/s 2022-01-31T19:44:22.025 INFO:tasks.workunit.client.0.smithi167.stdout:2/795: dwrite dd/d18/d63/f32 [0,4194304] 0 2022-01-31T19:44:22.035 INFO:tasks.workunit.client.0.smithi167.stdout:2/796: rename dd/df/d3f/d51 to dd/d18/d62/d75/d105 0 2022-01-31T19:44:22.039 INFO:tasks.workunit.client.0.smithi167.stdout:2/797: dread dd/df/d1b/f6a [0,4194304] 0 2022-01-31T19:44:22.041 INFO:tasks.workunit.client.0.smithi167.stdout:2/798: creat dd/f106 x:0 0 0 2022-01-31T19:44:22.042 INFO:tasks.workunit.client.0.smithi167.stdout:2/799: creat dd/df/d1b/d3d/d4a/da7/dcc/f107 x:0 0 0 2022-01-31T19:44:22.045 INFO:tasks.workunit.client.0.smithi167.stdout:2/800: creat dd/df/d22/f108 x:0 0 0 2022-01-31T19:44:22.048 INFO:tasks.workunit.client.0.smithi167.stdout:2/801: unlink dd/df/d1b/d3d/d4a/da7/dcc/fef 0 2022-01-31T19:44:22.049 INFO:tasks.workunit.client.0.smithi167.stdout:2/802: mknod dd/df/d22/c109 0 2022-01-31T19:44:22.050 INFO:tasks.workunit.client.0.smithi167.stdout:2/803: mknod dd/df/d1b/d3d/d4a/c10a 0 2022-01-31T19:44:22.051 INFO:tasks.workunit.client.0.smithi167.stdout:2/804: mknod dd/df/d22/d96/d9e/dda/c10b 0 2022-01-31T19:44:22.052 INFO:tasks.workunit.client.0.smithi167.stdout:2/805: symlink dd/df/d22/d96/d9e/dda/de6/l10c 0 2022-01-31T19:44:22.066 INFO:tasks.workunit.client.0.smithi167.stdout:2/806: dwrite dd/df/d1b/d3d/d4a/d4b/d54/f8b [0,4194304] 0 2022-01-31T19:44:22.067 INFO:tasks.workunit.client.0.smithi167.stdout:2/807: read - dd/df/d22/d96/d9e/dda/f100 zero size 2022-01-31T19:44:22.067 INFO:tasks.workunit.client.0.smithi167.stdout:2/808: symlink dd/d18/l10d 0 2022-01-31T19:44:22.081 INFO:tasks.workunit.client.0.smithi167.stdout:2/809: dwrite dd/df/f1c [4194304,4194304] 0 2022-01-31T19:44:22.094 INFO:tasks.workunit.client.0.smithi167.stdout:2/810: dwrite dd/df/d1b/d3d/d4a/d4b/d74/d85/fee [0,4194304] 0 2022-01-31T19:44:22.095 INFO:tasks.workunit.client.0.smithi167.stdout:2/811: chown dd/d18/d62/d75/d105/fad 30542105 1 2022-01-31T19:44:22.097 INFO:tasks.workunit.client.0.smithi167.stdout:2/812: getdents dd/dcb 0 2022-01-31T19:44:22.097 INFO:tasks.workunit.client.0.smithi167.stdout:2/813: write dd/df/d1b/d2f/f34 [1020711,36083] 0 2022-01-31T19:44:22.099 INFO:tasks.workunit.client.0.smithi167.stdout:2/814: link dd/d18/f4e dd/df/d1b/d3d/d4a/da7/dcc/f10e 0 2022-01-31T19:44:22.100 INFO:tasks.workunit.client.0.smithi167.stdout:2/815: write dd/df/d1b/d3d/d4a/f5b [417886,80457] 0 2022-01-31T19:44:22.103 INFO:tasks.workunit.client.0.smithi167.stdout:2/816: creat dd/d18/d62/dea/f10f x:0 0 0 2022-01-31T19:44:22.103 INFO:tasks.workunit.client.0.smithi167.stdout:2/817: readlink dd/d18/l87 0 2022-01-31T19:44:22.105 INFO:tasks.workunit.client.0.smithi167.stdout:2/818: mkdir dd/df/d110 0 2022-01-31T19:44:22.106 INFO:tasks.workunit.client.0.smithi167.stdout:2/819: fdatasync dd/d18/d63/f2d 0 2022-01-31T19:44:22.106 INFO:tasks.workunit.client.0.smithi167.stdout:2/820: write dd/d18/d62/f91 [244834,6179] 0 2022-01-31T19:44:22.107 INFO:tasks.workunit.client.0.smithi167.stdout:2/821: mknod dd/df/d1b/d3d/d4a/d4b/d54/dca/c111 0 2022-01-31T19:44:22.108 INFO:tasks.workunit.client.0.smithi167.stdout:2/822: symlink dd/df/l112 0 2022-01-31T19:44:22.109 INFO:tasks.workunit.client.0.smithi167.stdout:2/823: symlink dd/df/d1b/d3d/d4a/l113 0 2022-01-31T19:44:22.110 INFO:tasks.workunit.client.0.smithi167.stdout:2/824: write f8 [199054,78044] 0 2022-01-31T19:44:22.125 INFO:tasks.workunit.client.0.smithi167.stdout:2/825: dwrite dd/d18/f99 [0,4194304] 0 2022-01-31T19:44:22.140 INFO:tasks.workunit.client.0.smithi167.stdout:2/826: dwrite dd/d18/d62/f64 [0,4194304] 0 2022-01-31T19:44:22.141 INFO:tasks.workunit.client.0.smithi167.stdout:2/827: fdatasync dd/d18/f99 0 2022-01-31T19:44:22.158 INFO:tasks.workunit.client.0.smithi167.stdout:2/828: dwrite dd/fbf [0,4194304] 0 2022-01-31T19:44:22.158 INFO:tasks.workunit.client.0.smithi167.stdout:2/829: creat dd/df/d1b/d3d/d4a/da7/f114 x:0 0 0 2022-01-31T19:44:22.158 INFO:tasks.workunit.client.0.smithi167.stdout:2/830: write dd/df/d22/d96/d9e/dda/f100 [180298,91178] 0 2022-01-31T19:44:22.159 INFO:tasks.workunit.client.0.smithi167.stdout:2/831: mknod dd/d18/d62/d75/d105/d5f/c115 0 2022-01-31T19:44:22.178 INFO:tasks.workunit.client.0.smithi167.stdout:2/832: dwrite dd/df/d1b/d3d/d4a/d4b/f93 [0,4194304] 0 2022-01-31T19:44:22.178 INFO:tasks.workunit.client.0.smithi167.stdout:2/833: write dd/d18/d62/f6e [213719,64870] 0 2022-01-31T19:44:22.179 INFO:tasks.workunit.client.0.smithi167.stdout:2/834: symlink dd/l116 0 2022-01-31T19:44:22.182 INFO:tasks.workunit.client.0.smithi167.stdout:2/835: dread dd/df/d3f/f41 [0,4194304] 0 2022-01-31T19:44:22.189 INFO:tasks.workunit.client.0.smithi167.stdout:2/836: dread dd/d18/f99 [0,4194304] 0 2022-01-31T19:44:22.189 INFO:tasks.workunit.client.0.smithi167.stdout:2/837: creat dd/d18/d8d/f117 x:0 0 0 2022-01-31T19:44:22.190 INFO:tasks.workunit.client.0.smithi167.stdout:2/838: fsync dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b 0 2022-01-31T19:44:22.190 INFO:tasks.workunit.client.0.smithi167.stdout:2/839: read dd/df/d1b/f56 [2482270,88022] 0 2022-01-31T19:44:22.191 INFO:tasks.workunit.client.0.smithi167.stdout:2/840: read - dd/df/d1b/d3d/d4a/da7/dcc/f107 zero size 2022-01-31T19:44:22.199 INFO:tasks.workunit.client.0.smithi167.stdout:2/841: dread dd/f29 [0,4194304] 0 2022-01-31T19:44:22.201 INFO:tasks.workunit.client.0.smithi167.stdout:2/842: unlink dd/d18/db7/lbc 0 2022-01-31T19:44:22.202 INFO:tasks.workunit.client.0.smithi167.stdout:2/843: symlink dd/df/d1b/d3d/d4a/da7/dcc/df5/l118 0 2022-01-31T19:44:22.203 INFO:tasks.workunit.client.0.smithi167.stdout:2/844: stat dd/df/d1b/l67 0 2022-01-31T19:44:22.204 INFO:tasks.workunit.client.0.smithi167.stdout:2/845: symlink dd/df/d22/d96/l119 0 2022-01-31T19:44:22.205 INFO:tasks.workunit.client.0.smithi167.stdout:2/846: write dd/d18/f3a [2258284,27685] 0 2022-01-31T19:44:22.205 INFO:tasks.workunit.client.0.smithi167.stdout:2/847: read dd/fbf [773560,82967] 0 2022-01-31T19:44:22.207 INFO:tasks.workunit.client.0.smithi167.stdout:2/848: unlink dd/df/d22/d96/d9e/dda/de6/l10c 0 2022-01-31T19:44:22.208 INFO:tasks.workunit.client.0.smithi167.stdout:2/849: rename dd/d18/d62/cc2 to dd/dcb/c11a 0 2022-01-31T19:44:22.217 INFO:tasks.workunit.client.0.smithi167.stdout:2/850: dread dd/fbf [0,4194304] 0 2022-01-31T19:44:22.218 INFO:tasks.workunit.client.0.smithi167.stdout:2/851: read dd/df/d1b/d3d/d4a/d4b/f7f [39203,15821] 0 2022-01-31T19:44:22.223 INFO:tasks.workunit.client.0.smithi167.stdout:2/852: rmdir dd/df/d22/d96/d9e/dda/de6 39 2022-01-31T19:44:22.224 INFO:tasks.workunit.client.0.smithi167.stdout:2/853: mkdir dd/d18/d62/d75/d11b 0 2022-01-31T19:44:22.245 INFO:tasks.workunit.client.0.smithi167.stdout:2/854: dwrite dd/df/d1b/d3d/d4a/d4b/f93 [0,4194304] 0 2022-01-31T19:44:22.262 INFO:tasks.workunit.client.0.smithi167.stdout:2/855: dwrite dd/df/d22/f77 [8388608,4194304] 0 2022-01-31T19:44:22.263 INFO:tasks.workunit.client.0.smithi167.stdout:2/856: mkdir dd/df/d1b/d3d/d4a/da7/dcc/d11c 0 2022-01-31T19:44:22.278 INFO:tasks.workunit.client.0.smithi167.stdout:2/857: dwrite dd/df/d1b/d2f/fb3 [0,4194304] 0 2022-01-31T19:44:22.302 INFO:tasks.workunit.client.0.smithi167.stdout:2/858: dwrite dd/df/d1b/d3d/d4a/da7/dcc/f10e [0,4194304] 0 2022-01-31T19:44:22.303 INFO:tasks.workunit.client.0.smithi167.stdout:2/859: stat dd/df/d22/d96/d9e/cc9 0 2022-01-31T19:44:22.305 INFO:tasks.workunit.client.0.smithi167.stdout:2/860: symlink dd/df/d22/d96/l11d 0 2022-01-31T19:44:22.305 INFO:tasks.workunit.client.0.smithi167.stdout:2/861: creat dd/d18/d63/f11e x:0 0 0 2022-01-31T19:44:22.306 INFO:tasks.workunit.client.0.smithi167.stdout:2/862: creat dd/df/d1b/d3d/d4a/d4b/d74/f11f x:0 0 0 2022-01-31T19:44:22.306 INFO:tasks.workunit.client.0.smithi167.stdout:2/863: rmdir dd/d18/d62/d75/d105 39 2022-01-31T19:44:22.306 INFO:tasks.workunit.client.0.smithi167.stdout:2/864: link dd/df/d1b/d3d/d4a/da7/dcc/f10e dd/d18/d63/f120 0 2022-01-31T19:44:22.308 INFO:tasks.workunit.client.0.smithi167.stdout:2/865: mkdir dd/df/d1b/d3d/d4a/da7/d121 0 2022-01-31T19:44:22.308 INFO:tasks.workunit.client.0.smithi167.stdout:2/866: readlink dd/df/d1b/d3d/d4a/d4b/d74/d85/dc4/ld7 0 2022-01-31T19:44:22.309 INFO:tasks.workunit.client.0.smithi167.stdout:2/867: read dd/f55 [11959183,70201] 0 2022-01-31T19:44:22.311 INFO:tasks.workunit.client.0.smithi167.stdout:8/873: sync 2022-01-31T19:44:22.311 INFO:tasks.workunit.client.0.smithi167.stdout:8/874: stat d6/db/de/c9b 0 2022-01-31T19:44:22.311 INFO:tasks.workunit.client.0.smithi167.stdout:8/875: creat d6/db/d28/d61/de3/f116 x:0 0 0 2022-01-31T19:44:22.312 INFO:tasks.workunit.client.0.smithi167.stdout:8/876: mkdir d6/db/d4e/d117 0 2022-01-31T19:44:22.313 INFO:tasks.workunit.client.0.smithi167.stdout:8/877: symlink d6/db/d4e/d73/l118 0 2022-01-31T19:44:22.314 INFO:tasks.workunit.client.0.smithi167.stdout:8/878: symlink d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/l119 0 2022-01-31T19:44:22.329 INFO:tasks.workunit.client.0.smithi167.stdout:8/879: dwrite d6/db/d4e/d73/d88/f96 [0,4194304] 0 2022-01-31T19:44:22.330 INFO:tasks.workunit.client.0.smithi167.stdout:8/880: creat d6/db/d28/d66/f11a x:0 0 0 2022-01-31T19:44:22.330 INFO:tasks.workunit.client.0.smithi167.stdout:8/881: write d6/f8 [1980691,97195] 0 2022-01-31T19:44:22.334 INFO:tasks.workunit.client.0.smithi167.stdout:8/882: unlink d6/d1e/d4c/f8b 0 2022-01-31T19:44:22.335 INFO:tasks.workunit.client.0.smithi167.stdout:8/883: mknod d6/d1e/d4c/dba/c11b 0 2022-01-31T19:44:22.339 INFO:tasks.workunit.client.0.smithi167.stdout:8/884: mkdir d6/db/d4e/dbd/d11c 0 2022-01-31T19:44:22.340 INFO:tasks.workunit.client.0.smithi167.stdout:8/885: truncate d6/d1e/d2d/f40 4134673 0 2022-01-31T19:44:22.349 INFO:tasks.workunit.client.0.smithi167.stdout:8/886: dread d6/d1e/f22 [4194304,4194304] 0 2022-01-31T19:44:22.362 INFO:tasks.workunit.client.0.smithi167.stdout:8/887: dwrite d6/db/de/d18/f21 [4194304,4194304] 0 2022-01-31T19:44:22.362 INFO:tasks.workunit.client.0.smithi167.stdout:8/888: readlink d6/db/d36/lfc 0 2022-01-31T19:44:22.363 INFO:tasks.workunit.client.0.smithi167.stdout:8/889: fdatasync d6/db/d28/d61/dd5/fdf 0 2022-01-31T19:44:22.380 INFO:tasks.workunit.client.0.smithi167.stdout:8/890: dwrite d6/db/d4e/fd0 [0,4194304] 0 2022-01-31T19:44:22.382 INFO:tasks.workunit.client.0.smithi167.stdout:8/891: getdents d6/db 0 2022-01-31T19:44:22.383 INFO:tasks.workunit.client.0.smithi167.stdout:8/892: symlink d6/db/de/d3d/dd6/l11d 0 2022-01-31T19:44:22.384 INFO:tasks.workunit.client.0.smithi167.stdout:8/893: creat d6/db/d4e/d117/f11e x:0 0 0 2022-01-31T19:44:22.385 INFO:tasks.workunit.client.0.smithi167.stdout:8/894: rmdir d6/db/d28 39 2022-01-31T19:44:22.385 INFO:tasks.workunit.client.0.smithi167.stdout:8/895: rmdir d6/db 39 2022-01-31T19:44:22.398 INFO:tasks.workunit.client.0.smithi167.stdout:8/896: dwrite d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/ff9 [0,4194304] 0 2022-01-31T19:44:22.403 INFO:tasks.workunit.client.0.smithi167.stdout:8/897: creat d6/db/de/d3d/dd6/f11f x:0 0 0 2022-01-31T19:44:22.403 INFO:tasks.workunit.client.0.smithi167.stdout:8/898: fsync d6/db/d28/d66/f11a 0 2022-01-31T19:44:22.403 INFO:tasks.workunit.client.0.smithi167.stdout:8/899: mknod d6/db/d28/d61/dd5/dbf/d105/d115/c120 0 2022-01-31T19:44:22.421 INFO:tasks.workunit.client.0.smithi167.stdout:8/900: dwrite d6/db/d4e/f58 [0,4194304] 0 2022-01-31T19:44:22.437 INFO:tasks.workunit.client.0.smithi167.stdout:8/901: dwrite d6/db/d36/d5d/fde [0,4194304] 0 2022-01-31T19:44:22.454 INFO:tasks.workunit.client.0.smithi167.stdout:8/902: dwrite d6/d1e/dcf/fe6 [0,4194304] 0 2022-01-31T19:44:22.455 INFO:tasks.workunit.client.0.smithi167.stdout:8/903: chown d6/l16 31 1 2022-01-31T19:44:22.462 INFO:tasks.workunit.client.0.smithi167.stdout:6/758: sync 2022-01-31T19:44:22.462 INFO:tasks.workunit.client.0.smithi167.stdout:3/957: sync 2022-01-31T19:44:22.465 INFO:tasks.workunit.client.0.smithi167.stdout:6/759: symlink d8/d33/d51/l10b 0 2022-01-31T19:44:22.467 INFO:tasks.workunit.client.0.smithi167.stdout:6/760: read d8/d20/f22 [3669814,5390] 0 2022-01-31T19:44:22.472 INFO:tasks.workunit.client.0.smithi167.stdout:8/904: dwrite d6/db/de/d18/f21 [4194304,4194304] 0 2022-01-31T19:44:22.473 INFO:tasks.workunit.client.0.smithi167.stdout:6/761: write d8/d33/d51/d5d/f6e [618818,85986] 0 2022-01-31T19:44:22.474 INFO:tasks.workunit.client.0.smithi167.stdout:6/762: rmdir d8/d33/d51/d6a 39 2022-01-31T19:44:22.474 INFO:tasks.workunit.client.0.smithi167.stdout:6/763: unlink d8/d20/f44 0 2022-01-31T19:44:22.479 INFO:tasks.workunit.client.0.smithi167.stdout:3/958: dwrite d9/d29/d3b/d41/d6b/d89/f97 [0,4194304] 0 2022-01-31T19:44:22.480 INFO:tasks.workunit.client.0.smithi167.stdout:3/959: chown d9/f79 686886 1 2022-01-31T19:44:22.490 INFO:tasks.workunit.client.0.smithi167.stdout:3/960: dread d9/d26/fb1 [0,4194304] 0 2022-01-31T19:44:22.491 INFO:tasks.workunit.client.0.smithi167.stdout:3/961: readlink d9/d29/d60/l64 0 2022-01-31T19:44:22.491 INFO:tasks.workunit.client.0.smithi167.stdout:3/962: dread - d9/d7f/f11d zero size 2022-01-31T19:44:22.492 INFO:tasks.workunit.client.0.smithi167.stdout:3/963: creat d9/d21/ddd/f123 x:0 0 0 2022-01-31T19:44:22.494 INFO:tasks.workunit.client.0.smithi167.stdout:3/964: unlink d9/d7f/dbb/ff0 0 2022-01-31T19:44:22.495 INFO:tasks.workunit.client.0.smithi167.stdout:8/905: dwrite d6/d1e/d2d/f10d [0,4194304] 0 2022-01-31T19:44:22.496 INFO:tasks.workunit.client.0.smithi167.stdout:6/764: dwrite d8/d33/d51/d5d/d86/fcc [0,4194304] 0 2022-01-31T19:44:22.497 INFO:tasks.workunit.client.0.smithi167.stdout:3/965: creat d9/d7f/dbe/d11b/f124 x:0 0 0 2022-01-31T19:44:22.499 INFO:tasks.workunit.client.0.smithi167.stdout:6/765: creat d8/d33/d51/d72/f10c x:0 0 0 2022-01-31T19:44:22.501 INFO:tasks.workunit.client.0.smithi167.stdout:6/766: mkdir d8/d33/d51/d6a/df1/d10d 0 2022-01-31T19:44:22.502 INFO:tasks.workunit.client.0.smithi167.stdout:6/767: write d8/d20/d54/f85 [5169111,65073] 0 2022-01-31T19:44:22.502 INFO:tasks.workunit.client.0.smithi167.stdout:6/768: write d8/d33/d51/d5d/d86/dbd/fd4 [370463,94483] 0 2022-01-31T19:44:22.513 INFO:tasks.workunit.client.0.smithi167.stdout:8/906: dread d6/db/d36/d5d/fde [0,4194304] 0 2022-01-31T19:44:22.513 INFO:tasks.workunit.client.0.smithi167.stdout:8/907: readlink d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/l119 0 2022-01-31T19:44:22.514 INFO:tasks.workunit.client.0.smithi167.stdout:8/908: truncate d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/fe5 2642665 0 2022-01-31T19:44:22.514 INFO:tasks.workunit.client.0.smithi167.stdout:8/909: rmdir d6/db 39 2022-01-31T19:44:22.515 INFO:tasks.workunit.client.0.smithi167.stdout:8/910: read d6/fab [1200747,96660] 0 2022-01-31T19:44:22.517 INFO:tasks.workunit.client.0.smithi167.stdout:8/911: creat d6/d1e/d4c/d109/f121 x:0 0 0 2022-01-31T19:44:22.518 INFO:tasks.workunit.client.0.smithi167.stdout:8/912: mknod d6/d1e/d4c/dba/c122 0 2022-01-31T19:44:22.520 INFO:tasks.workunit.client.0.smithi167.stdout:8/913: creat d6/db/d28/d61/dd5/dbf/d105/dc2/f123 x:0 0 0 2022-01-31T19:44:22.520 INFO:tasks.workunit.client.0.smithi167.stdout:8/914: chown d6/db/d4e/d117/f11e 723485 1 2022-01-31T19:44:22.520 INFO:tasks.workunit.client.0.smithi167.stdout:8/915: chown d6/d1e/d4c/dba/dec 1022 1 2022-01-31T19:44:22.520 INFO:tasks.workunit.client.0.smithi167.stdout:8/916: truncate d6/db/de/d18/fce 1463081 0 2022-01-31T19:44:22.521 INFO:tasks.workunit.client.0.smithi167.stdout:8/917: mkdir d6/db/d36/d124 0 2022-01-31T19:44:22.521 INFO:tasks.workunit.client.0.smithi167.stdout:8/918: dread - d6/db/d36/d5d/f63 zero size 2022-01-31T19:44:22.522 INFO:tasks.workunit.client.0.smithi167.stdout:8/919: creat d6/db/d28/d10c/f125 x:0 0 0 2022-01-31T19:44:22.523 INFO:tasks.workunit.client.0.smithi167.stdout:8/920: mkdir d6/d126 0 2022-01-31T19:44:22.523 INFO:tasks.workunit.client.0.smithi167.stdout:8/921: readlink d6/d1e/d2d/lae 0 2022-01-31T19:44:22.527 INFO:tasks.workunit.client.0.smithi167.stdout:3/966: dwrite d9/d29/d3b/d41/d6b/d100/f107 [0,4194304] 0 2022-01-31T19:44:22.527 INFO:tasks.workunit.client.0.smithi167.stdout:6/769: dwrite d8/d20/f42 [0,4194304] 0 2022-01-31T19:44:22.527 INFO:tasks.workunit.client.0.smithi167.stdout:6/770: chown d8/f9 1081 1 2022-01-31T19:44:22.534 INFO:tasks.workunit.client.0.smithi167.stdout:6/771: creat d8/d20/d54/f10e x:0 0 0 2022-01-31T19:44:22.535 INFO:tasks.workunit.client.0.smithi167.stdout:3/967: mknod d9/d29/d60/c125 0 2022-01-31T19:44:22.536 INFO:tasks.workunit.client.0.smithi167.stdout:3/968: fdatasync d9/d29/d3b/d41/d6b/f117 0 2022-01-31T19:44:22.536 INFO:tasks.workunit.client.0.smithi167.stdout:3/969: fdatasync d9/d21/f23 0 2022-01-31T19:44:22.536 INFO:tasks.workunit.client.0.smithi167.stdout:3/970: unlink d9/d29/d3b/d9b/da3/db0/cf1 0 2022-01-31T19:44:22.537 INFO:tasks.workunit.client.0.smithi167.stdout:3/971: truncate d9/d29/d3b/d9b/da3/dd9/fe3 1312735 0 2022-01-31T19:44:22.537 INFO:tasks.workunit.client.0.smithi167.stdout:3/972: mkdir d9/d29/d3b/d41/d6b/d100/d126 0 2022-01-31T19:44:22.539 INFO:tasks.workunit.client.0.smithi167.stdout:3/973: creat d9/d29/d3b/d41/deb/f127 x:0 0 0 2022-01-31T19:44:22.544 INFO:tasks.workunit.client.0.smithi167.stdout:6/772: write d8/d33/d51/d7e/dc4/fbc [2511044,130547] 0 2022-01-31T19:44:22.546 INFO:tasks.workunit.client.0.smithi167.stdout:8/922: dread d6/db/de/d3d/fe9 [4194304,4194304] 0 2022-01-31T19:44:22.549 INFO:tasks.workunit.client.0.smithi167.stdout:6/773: fsync d8/d33/d51/d5d/f6e 0 2022-01-31T19:44:22.550 INFO:tasks.workunit.client.0.smithi167.stdout:6/774: read - d8/d20/fdc zero size 2022-01-31T19:44:22.555 INFO:tasks.workunit.client.0.smithi167.stdout:3/974: rename d9/d21 to d9/d56/d128 0 2022-01-31T19:44:22.555 INFO:tasks.workunit.client.0.smithi167.stdout:3/975: write d9/d29/d3b/d9b/da3/f116 [1021587,118884] 0 2022-01-31T19:44:22.556 INFO:tasks.workunit.client.0.smithi167.stdout:6/775: symlink d8/d62/d92/l10f 0 2022-01-31T19:44:22.556 INFO:tasks.workunit.client.0.smithi167.stdout:6/776: write d8/d33/d51/d5d/f80 [1614581,44496] 0 2022-01-31T19:44:22.559 INFO:tasks.workunit.client.0.smithi167.stdout:9/911: sync 2022-01-31T19:44:22.559 INFO:tasks.workunit.client.0.smithi167.stdout:4/986: sync 2022-01-31T19:44:22.560 INFO:tasks.workunit.client.0.smithi167.stdout:4/987: chown d0/d5d/d12f/ddf/d75/dd6/d111 1882 1 2022-01-31T19:44:22.560 INFO:tasks.workunit.client.0.smithi167.stdout:3/976: dread d9/d56/d128/f77 [0,4194304] 0 2022-01-31T19:44:22.561 INFO:tasks.workunit.client.0.smithi167.stdout:9/912: mknod dd/d17/d7f/c134 0 2022-01-31T19:44:22.562 INFO:tasks.workunit.client.0.smithi167.stdout:0/684: sync 2022-01-31T19:44:22.562 INFO:tasks.workunit.client.0.smithi167.stdout:9/913: mknod dd/d19/d59/df9/d121/c135 0 2022-01-31T19:44:22.563 INFO:tasks.workunit.client.0.smithi167.stdout:0/685: mkdir d17/d19/d1c/d39/d7b/de5 0 2022-01-31T19:44:22.564 INFO:tasks.workunit.client.0.smithi167.stdout:0/686: symlink d17/d33/da4/le6 0 2022-01-31T19:44:22.565 INFO:tasks.workunit.client.0.smithi167.stdout:0/687: truncate d17/d2c/f3c 1267262 0 2022-01-31T19:44:22.566 INFO:tasks.workunit.client.0.smithi167.stdout:0/688: dread - d17/d2c/d87/fc4 zero size 2022-01-31T19:44:22.567 INFO:tasks.workunit.client.0.smithi167.stdout:0/689: creat d17/d19/fe7 x:0 0 0 2022-01-31T19:44:22.571 INFO:tasks.workunit.client.0.smithi167.stdout:0/690: write d17/d19/d1c/d39/d44/d88/fc9 [108385,14673] 0 2022-01-31T19:44:22.573 INFO:tasks.workunit.client.0.smithi167.stdout:8/923: dwrite d6/d1e/ffa [0,4194304] 0 2022-01-31T19:44:22.576 INFO:tasks.workunit.client.0.smithi167.stdout:9/914: dread dd/d35/f65 [0,4194304] 0 2022-01-31T19:44:22.576 INFO:tasks.workunit.client.0.smithi167.stdout:9/915: chown dd/d34/cdb 431608 1 2022-01-31T19:44:22.577 INFO:tasks.workunit.client.0.smithi167.stdout:9/916: write dd/f64 [4390213,90749] 0 2022-01-31T19:44:22.577 INFO:tasks.workunit.client.0.smithi167.stdout:4/988: dread d0/d5d/d12f/ddf/d1a/fca [0,4194304] 0 2022-01-31T19:44:22.578 INFO:tasks.workunit.client.0.smithi167.stdout:3/977: dread d9/d29/d3b/d9b/da3/f116 [0,4194304] 0 2022-01-31T19:44:22.579 INFO:tasks.workunit.client.0.smithi167.stdout:6/777: dwrite d8/d33/d4f/f76 [0,4194304] 0 2022-01-31T19:44:22.579 INFO:tasks.workunit.client.0.smithi167.stdout:6/778: chown d8/d33/d51/d6a/f9d 221 1 2022-01-31T19:44:22.583 INFO:tasks.workunit.client.0.smithi167.stdout:8/924: unlink d6/db/de/d18/c1c 0 2022-01-31T19:44:22.585 INFO:tasks.workunit.client.0.smithi167.stdout:3/978: write d9/d26/fb1 [3248607,75835] 0 2022-01-31T19:44:22.586 INFO:tasks.workunit.client.0.smithi167.stdout:8/925: read d6/db/d28/d61/f89 [904266,122392] 0 2022-01-31T19:44:22.587 INFO:tasks.workunit.client.0.smithi167.stdout:9/917: mknod dd/d17/d7f/c136 0 2022-01-31T19:44:22.587 INFO:tasks.workunit.client.0.smithi167.stdout:8/926: read d6/f32 [1799949,3537] 0 2022-01-31T19:44:22.588 INFO:tasks.workunit.client.0.smithi167.stdout:6/779: mkdir d8/d62/d92/d110 0 2022-01-31T19:44:22.589 INFO:tasks.workunit.client.0.smithi167.stdout:3/979: symlink d9/d29/d3b/d41/d6b/l129 0 2022-01-31T19:44:22.591 INFO:tasks.workunit.client.0.smithi167.stdout:8/927: symlink d6/db/d36/l127 0 2022-01-31T19:44:22.593 INFO:tasks.workunit.client.0.smithi167.stdout:6/780: mkdir d8/d33/d51/d7e/dc4/deb/d111 0 2022-01-31T19:44:22.596 INFO:tasks.workunit.client.0.smithi167.stdout:9/918: getdents dd/d19 0 2022-01-31T19:44:22.596 INFO:tasks.workunit.client.0.smithi167.stdout:9/919: chown dd/d19/d24/d56/d5b/da5/cd8 0 1 2022-01-31T19:44:22.596 INFO:tasks.workunit.client.0.smithi167.stdout:9/920: readlink dd/d17/d7f/def/l69 0 2022-01-31T19:44:22.596 INFO:tasks.workunit.client.0.smithi167.stdout:9/921: chown dd/d19/d24/laa 5 1 2022-01-31T19:44:22.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/922: write dd/d17/dc4/de2/f11e [1020757,115784] 0 2022-01-31T19:44:22.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/923: write dd/d19/d24/d56/dc9/d125/f132 [816031,82038] 0 2022-01-31T19:44:22.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/924: stat dd/d17/d7f/dfb/f120 0 2022-01-31T19:44:22.597 INFO:tasks.workunit.client.0.smithi167.stdout:9/925: chown dd/d17/dc4/de2/f11e 11117 1 2022-01-31T19:44:22.598 INFO:tasks.workunit.client.0.smithi167.stdout:9/926: write dd/d19/d24/d50/dd6/d11b/f12b [852431,68696] 0 2022-01-31T19:44:22.598 INFO:tasks.workunit.client.0.smithi167.stdout:2/868: sync 2022-01-31T19:44:22.598 INFO:tasks.workunit.client.0.smithi167.stdout:2/869: readlink dd/df/d1b/d3d/d4a/da7/lfa 0 2022-01-31T19:44:22.599 INFO:tasks.workunit.client.0.smithi167.stdout:8/928: rename d6/db/de/d3d/cf7 to d6/d1e/d4c/d109/c128 0 2022-01-31T19:44:22.599 INFO:tasks.workunit.client.0.smithi167.stdout:4/989: dwrite d0/d6/d7/d89/fa9 [0,4194304] 0 2022-01-31T19:44:22.600 INFO:tasks.workunit.client.0.smithi167.stdout:6/781: rmdir d8/d33/d4f 39 2022-01-31T19:44:22.601 INFO:tasks.workunit.client.0.smithi167.stdout:9/927: truncate dd/d17/d7f/fbb 1200179 0 2022-01-31T19:44:22.602 INFO:tasks.workunit.client.0.smithi167.stdout:8/929: mkdir d6/db/d36/d124/d129 0 2022-01-31T19:44:22.605 INFO:tasks.workunit.client.0.smithi167.stdout:6/782: mkdir d8/d33/d51/d7e/dc4/d112 0 2022-01-31T19:44:22.606 INFO:tasks.workunit.client.0.smithi167.stdout:6/783: write d8/d33/d51/d5d/d64/ff7 [975305,11704] 0 2022-01-31T19:44:22.608 INFO:tasks.workunit.client.0.smithi167.stdout:4/990: creat d0/d6/d7/d26/d40/d4c/db4/f146 x:0 0 0 2022-01-31T19:44:22.609 INFO:tasks.workunit.client.0.smithi167.stdout:4/991: write d0/d6/d7/f8 [5818671,97624] 0 2022-01-31T19:44:22.609 INFO:tasks.workunit.client.0.smithi167.stdout:4/992: dread - d0/d6/d7/d26/dd2/de5/f11f zero size 2022-01-31T19:44:22.609 INFO:tasks.workunit.client.0.smithi167.stdout:9/928: write dd/d19/d24/d56/d5b/da5/fed [1963050,100978] 0 2022-01-31T19:44:22.611 INFO:tasks.workunit.client.0.smithi167.stdout:2/870: dwrite dd/df/d1b/d3d/d4a/da7/f114 [0,4194304] 0 2022-01-31T19:44:22.611 INFO:tasks.workunit.client.0.smithi167.stdout:2/871: dread - dd/df/d1b/d3d/d4a/da7/fc6 zero size 2022-01-31T19:44:22.612 INFO:tasks.workunit.client.0.smithi167.stdout:6/784: link d8/d33/d51/d5d/d64/dab/db9/c104 d8/d33/d4f/dd1/c113 0 2022-01-31T19:44:22.614 INFO:tasks.workunit.client.0.smithi167.stdout:4/993: mknod d0/d5d/d12f/ddf/da2/dcf/db5/c147 0 2022-01-31T19:44:22.615 INFO:tasks.workunit.client.0.smithi167.stdout:4/994: dread - d0/d6/d7/d26/d40/d4c/db4/f146 zero size 2022-01-31T19:44:22.619 INFO:tasks.workunit.client.0.smithi167.stdout:4/995: dread d0/d6/dab/ff3 [0,4194304] 0 2022-01-31T19:44:22.625 INFO:tasks.workunit.client.0.smithi167.stdout:4/996: dread d0/d6/d7/d39/d4e/d61/f95 [4194304,4194304] 0 2022-01-31T19:44:22.627 INFO:tasks.workunit.client.0.smithi167.stdout:8/930: dwrite d6/db/de/d3d/fa7 [0,4194304] 0 2022-01-31T19:44:22.636 INFO:tasks.workunit.client.0.smithi167.stdout:6/785: dwrite d8/d33/d51/d5d/d64/dab/db9/fe6 [0,4194304] 0 2022-01-31T19:44:22.637 INFO:tasks.workunit.client.0.smithi167.stdout:2/872: dwrite dd/d18/f3a [0,4194304] 0 2022-01-31T19:44:22.640 INFO:tasks.workunit.client.0.smithi167.stdout:6/786: unlink d8/d33/d51/d72/dc2/ld5 0 2022-01-31T19:44:22.643 INFO:tasks.workunit.client.0.smithi167.stdout:2/873: rename dd/df/d1b/d3d/d4a/d4b/c73 to dd/d18/d62/d75/d105/c122 0 2022-01-31T19:44:22.644 INFO:tasks.workunit.client.0.smithi167.stdout:2/874: truncate dd/df/f1c 5702268 0 2022-01-31T19:44:22.645 INFO:tasks.workunit.client.0.smithi167.stdout:2/875: creat dd/df/d1b/d3d/f123 x:0 0 0 2022-01-31T19:44:22.680 INFO:tasks.workunit.client.0.smithi167.stdout:4/997: dwrite d0/d5d/d12f/ddf/d75/f98 [0,4194304] 0 2022-01-31T19:44:22.680 INFO:tasks.workunit.client.0.smithi167.stdout:4/998: stat d0/d6 0 2022-01-31T19:44:22.680 INFO:tasks.workunit.client.0.smithi167.stdout:4/999: dread - d0/d6/d7/d89/d11c/d130/f13f zero size 2022-01-31T19:44:22.681 INFO:tasks.workunit.client.0.smithi167.stdout:9/929: dwrite dd/d19/d24/d56/d5b/da5/fb4 [0,4194304] 0 2022-01-31T19:44:22.681 INFO:tasks.workunit.client.0.smithi167.stdout:8/931: dwrite d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/dc7/fc9 [0,4194304] 0 2022-01-31T19:44:22.681 INFO:tasks.workunit.client.0.smithi167.stdout:9/930: creat dd/d17/d7f/dfb/f137 x:0 0 0 2022-01-31T19:44:22.682 INFO:tasks.workunit.client.0.smithi167.stdout:6/787: dwrite d8/d33/d51/d7e/dc4/fbc [0,4194304] 0 2022-01-31T19:44:22.682 INFO:tasks.workunit.client.0.smithi167.stdout:2/876: dwrite dd/f106 [0,4194304] 0 2022-01-31T19:44:22.683 INFO:tasks.workunit.client.0.smithi167.stdout:2/877: dread - dd/df/d1b/d3d/d4a/d4b/d74/d85/fe2 zero size 2022-01-31T19:44:22.683 INFO:tasks.workunit.client.0.smithi167.stdout:2/878: write dd/d18/d8d/fa5 [4952034,92017] 0 2022-01-31T19:44:22.684 INFO:tasks.workunit.client.0.smithi167.stdout:8/932: unlink d6/db/d6a/f7c 0 2022-01-31T19:44:22.686 INFO:tasks.workunit.client.0.smithi167.stdout:9/931: chown dd/d6b/cf1 324 1 2022-01-31T19:44:22.687 INFO:tasks.workunit.client.0.smithi167.stdout:9/932: write dd/d35/f8d [5185671,72615] 0 2022-01-31T19:44:22.687 INFO:tasks.workunit.client.0.smithi167.stdout:6/788: mknod d8/d33/d51/d5d/d64/c114 0 2022-01-31T19:44:22.689 INFO:tasks.workunit.client.0.smithi167.stdout:2/879: link dd/d18/d8d/fa5 dd/df/d22/f124 0 2022-01-31T19:44:22.690 INFO:tasks.workunit.client.0.smithi167.stdout:8/933: mkdir d6/db/d12a 0 2022-01-31T19:44:22.691 INFO:tasks.workunit.client.0.smithi167.stdout:8/934: chown d6/d1e/f22 292757 1 2022-01-31T19:44:22.691 INFO:tasks.workunit.client.0.smithi167.stdout:9/933: symlink dd/d17/d7f/def/l138 0 2022-01-31T19:44:22.693 INFO:tasks.workunit.client.0.smithi167.stdout:6/789: mknod d8/d33/c115 0 2022-01-31T19:44:22.694 INFO:tasks.workunit.client.0.smithi167.stdout:2/880: getdents dd/df/d1b/d3d/d4a 0 2022-01-31T19:44:22.695 INFO:tasks.workunit.client.0.smithi167.stdout:8/935: truncate d6/db/de/d18/f21 5509671 0 2022-01-31T19:44:22.695 INFO:tasks.workunit.client.0.smithi167.stdout:8/936: chown d6/db/d28/d61/dd5/dbf/d105/dc2/ld9 306660 1 2022-01-31T19:44:22.696 INFO:tasks.workunit.client.0.smithi167.stdout:6/790: symlink d8/d33/d51/d6a/l116 0 2022-01-31T19:44:22.696 INFO:tasks.workunit.client.0.smithi167.stdout:8/937: write d6/db/d28/d61/dd5/f90 [725272,28711] 0 2022-01-31T19:44:22.697 INFO:tasks.workunit.client.0.smithi167.stdout:8/938: creat d6/d1e/d4c/d109/f12b x:0 0 0 2022-01-31T19:44:22.697 INFO:tasks.workunit.client.0.smithi167.stdout:8/939: chown d6/db/de/f26 172781182 1 2022-01-31T19:44:22.697 INFO:tasks.workunit.client.0.smithi167.stdout:2/881: symlink dd/df/d22/d96/d9e/dda/de6/l125 0 2022-01-31T19:44:22.697 INFO:tasks.workunit.client.0.smithi167.stdout:2/882: fsync dd/f23 0 2022-01-31T19:44:22.698 INFO:tasks.workunit.client.0.smithi167.stdout:2/883: truncate dd/df/d1b/d2f/f34 4573044 0 2022-01-31T19:44:22.701 INFO:tasks.workunit.client.0.smithi167.stdout:8/940: link d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/l119 d6/db/de/d3d/dd6/l12c 0 2022-01-31T19:44:22.702 INFO:tasks.workunit.client.0.smithi167.stdout:8/941: readlink d6/d1e/dcf/lfd 0 2022-01-31T19:44:22.702 INFO:tasks.workunit.client.0.smithi167.stdout:8/942: readlink d6/db/d28/d61/dd5/dbf/d105/dc2/ld9 0 2022-01-31T19:44:22.703 INFO:tasks.workunit.client.0.smithi167.stdout:8/943: read d6/db/fd [806213,28011] 0 2022-01-31T19:44:22.705 INFO:tasks.workunit.client.0.smithi167.stdout:6/791: mkdir d8/d33/d51/d72/d117 0 2022-01-31T19:44:22.708 INFO:tasks.workunit.client.0.smithi167.stdout:2/884: rmdir dd/df/d1b/d3d 39 2022-01-31T19:44:22.710 INFO:tasks.workunit.client.0.smithi167.stdout:2/885: symlink dd/df/d1b/d3d/d4a/da7/l126 0 2022-01-31T19:44:22.710 INFO:tasks.workunit.client.0.smithi167.stdout:2/886: truncate dd/df/d1b/d3d/d4a/fac 152153 0 2022-01-31T19:44:22.712 INFO:tasks.workunit.client.0.smithi167.stdout:2/887: fdatasync dd/d18/d8d/fa5 0 2022-01-31T19:44:22.712 INFO:tasks.workunit.client.0.smithi167.stdout:8/944: dread d6/db/de/d3d/fe9 [0,4194304] 0 2022-01-31T19:44:22.713 INFO:tasks.workunit.client.0.smithi167.stdout:9/934: dwrite dd/d19/d24/d50/dd1/f8a [0,4194304] 0 2022-01-31T19:44:22.713 INFO:tasks.workunit.client.0.smithi167.stdout:8/945: dread - d6/db/d4e/fe8 zero size 2022-01-31T19:44:22.714 INFO:tasks.workunit.client.0.smithi167.stdout:8/946: fsync d6/db/fd2 0 2022-01-31T19:44:22.714 INFO:tasks.workunit.client.0.smithi167.stdout:8/947: write d6/db/d28/d66/f11a [825612,78244] 0 2022-01-31T19:44:22.715 INFO:tasks.workunit.client.0.smithi167.stdout:9/935: write dd/d19/d24/d50/dd6/d11b/f12b [511141,90123] 0 2022-01-31T19:44:22.716 INFO:tasks.workunit.client.0.smithi167.stdout:6/792: rename d8/d20/fa6 to d8/d33/d51/f118 0 2022-01-31T19:44:22.716 INFO:tasks.workunit.client.0.smithi167.stdout:2/888: read dd/f23 [8198878,24506] 0 2022-01-31T19:44:22.717 INFO:tasks.workunit.client.0.smithi167.stdout:2/889: fsync dd/d18/d62/f64 0 2022-01-31T19:44:22.728 INFO:tasks.workunit.client.0.smithi167.stdout:8/948: symlink d6/d126/l12d 0 2022-01-31T19:44:22.728 INFO:tasks.workunit.client.0.smithi167.stdout:8/949: write d6/db/d6a/d8d/d9e/d60/f79 [384505,86531] 0 2022-01-31T19:44:22.730 INFO:tasks.workunit.client.0.smithi167.stdout:9/936: mknod dd/d19/d24/d50/c139 0 2022-01-31T19:44:22.730 INFO:tasks.workunit.client.0.smithi167.stdout:2/890: mknod dd/df/d1b/d3d/d4a/da7/dcc/c127 0 2022-01-31T19:44:22.731 INFO:tasks.workunit.client.0.smithi167.stdout:2/891: chown dd/d18/l87 32929441 1 2022-01-31T19:44:22.731 INFO:tasks.workunit.client.0.smithi167.stdout:2/892: rename dd to dd/df/d1b/d3d/d4a/da7/dcc/d11c/d128 22 2022-01-31T19:44:22.732 INFO:tasks.workunit.client.0.smithi167.stdout:2/893: dread dd/df/d1b/d2f/d36/fb9 [0,4194304] 0 2022-01-31T19:44:22.818 INFO:tasks.workunit.client.0.smithi167.stdout:8/950: creat d6/db/d36/d124/d129/f12e x:0 0 0 2022-01-31T19:44:22.821 INFO:tasks.workunit.client.0.smithi167.stdout:9/937: mkdir dd/de7/d13a 0 2022-01-31T19:44:22.821 INFO:tasks.workunit.client.0.smithi167.stdout:9/938: dread - dd/d17/d80/f106 zero size 2022-01-31T19:44:22.822 INFO:tasks.workunit.client.0.smithi167.stdout:2/894: getdents dd/df/d1b/d3d/d4a/d4b/d74/d85 0 2022-01-31T19:44:22.823 INFO:tasks.workunit.client.0.smithi167.stdout:9/939: mknod dd/d19/d24/d56/dc9/d125/d12e/c13b 0 2022-01-31T19:44:22.825 INFO:tasks.workunit.client.0.smithi167.stdout:9/940: read dd/d19/d24/d50/dd6/f103 [5774313,88652] 0 2022-01-31T19:44:22.826 INFO:tasks.workunit.client.0.smithi167.stdout:8/951: write d6/db/de/d18/fce [2177701,34358] 0 2022-01-31T19:44:22.827 INFO:tasks.workunit.client.0.smithi167.stdout:8/952: readlink d6/db/d4e/d73/l118 0 2022-01-31T19:44:22.827 INFO:tasks.workunit.client.0.smithi167.stdout:8/953: dread - d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/fe4 zero size 2022-01-31T19:44:22.828 INFO:tasks.workunit.client.0.smithi167.stdout:8/954: rename d6/db/de/c3a to d6/db/d28/daf/c12f 0 2022-01-31T19:44:22.829 INFO:tasks.workunit.client.0.smithi167.stdout:8/955: mknod d6/db/d4e/dbd/c130 0 2022-01-31T19:44:22.830 INFO:tasks.workunit.client.0.smithi167.stdout:8/956: write d6/db/de/f10 [754426,99278] 0 2022-01-31T19:44:22.830 INFO:tasks.workunit.client.0.smithi167.stdout:2/895: dread dd/d18/d62/d75/d105/f92 [0,4194304] 0 2022-01-31T19:44:22.833 INFO:tasks.workunit.client.0.smithi167.stdout:9/941: dread dd/d19/d24/d56/d5b/dcf/fe6 [0,4194304] 0 2022-01-31T19:44:22.833 INFO:tasks.workunit.client.0.smithi167.stdout:9/942: readlink l2 0 2022-01-31T19:44:22.834 INFO:tasks.workunit.client.0.smithi167.stdout:9/943: readlink dd/d19/d24/d50/d5d/l119 0 2022-01-31T19:44:22.834 INFO:tasks.workunit.client.0.smithi167.stdout:9/944: chown dd/d17/d7f/def/db3/cc0 59141 1 2022-01-31T19:44:22.841 INFO:tasks.workunit.client.0.smithi167.stdout:9/945: dread dd/d34/f51 [0,4194304] 0 2022-01-31T19:44:22.841 INFO:tasks.workunit.client.0.smithi167.stdout:9/946: write dd/d19/d24/d50/dd6/d11b/f12b [282325,116407] 0 2022-01-31T19:44:22.842 INFO:tasks.workunit.client.0.smithi167.stdout:9/947: chown dd/d34/f92 0 1 2022-01-31T19:44:22.842 INFO:tasks.workunit.client.0.smithi167.stdout:9/948: creat dd/de7/f13c x:0 0 0 2022-01-31T19:44:22.856 INFO:tasks.workunit.client.0.smithi167.stdout:9/949: dread dd/d19/d24/d50/dd1/fc2 [4194304,4194304] 0 2022-01-31T19:44:22.857 INFO:tasks.workunit.client.0.smithi167.stdout:9/950: chown dd/d17/d7f/def/lc3 12 1 2022-01-31T19:44:22.857 INFO:tasks.workunit.client.0.smithi167.stdout:8/957: dwrite d6/f57 [0,4194304] 0 2022-01-31T19:44:22.857 INFO:tasks.workunit.client.0.smithi167.stdout:2/896: dwrite dd/d18/d63/fd0 [0,4194304] 0 2022-01-31T19:44:22.858 INFO:tasks.workunit.client.0.smithi167.stdout:9/951: symlink dd/d19/d24/d50/l13d 0 2022-01-31T19:44:22.858 INFO:tasks.workunit.client.0.smithi167.stdout:9/952: chown dd/c29 1656292071 1 2022-01-31T19:44:22.859 INFO:tasks.workunit.client.0.smithi167.stdout:8/958: rmdir d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/dc7 39 2022-01-31T19:44:22.859 INFO:tasks.workunit.client.0.smithi167.stdout:8/959: readlink d6/d1e/d2d/l5f 0 2022-01-31T19:44:22.861 INFO:tasks.workunit.client.0.smithi167.stdout:2/897: symlink dd/df/d1b/d3d/d4a/d4b/l129 0 2022-01-31T19:44:22.861 INFO:tasks.workunit.client.0.smithi167.stdout:2/898: write dd/d18/db7/feb [5132816,4741] 0 2022-01-31T19:44:22.862 INFO:tasks.workunit.client.0.smithi167.stdout:9/953: mkdir dd/d19/d24/d56/d5b/dcf/d13e 0 2022-01-31T19:44:22.864 INFO:tasks.workunit.client.0.smithi167.stdout:2/899: creat dd/df/d1b/d3d/d4a/d4b/d74/f12a x:0 0 0 2022-01-31T19:44:22.864 INFO:tasks.workunit.client.0.smithi167.stdout:2/900: stat dd/df/d1b/d3d/d4a/da7/fb5 0 2022-01-31T19:44:22.865 INFO:tasks.workunit.client.0.smithi167.stdout:9/954: creat dd/de7/d13a/f13f x:0 0 0 2022-01-31T19:44:22.866 INFO:tasks.workunit.client.0.smithi167.stdout:2/901: link dd/df/d1b/d3d/d4a/c10a dd/df/c12b 0 2022-01-31T19:44:22.868 INFO:tasks.workunit.client.0.smithi167.stdout:2/902: link dd/df/d1b/d3d/d4a/da7/dcc/df5/l118 dd/df/d1b/d2f/l12c 0 2022-01-31T19:44:22.870 INFO:tasks.workunit.client.0.smithi167.stdout:2/903: creat dd/d18/d62/d75/dc8/f12d x:0 0 0 2022-01-31T19:44:22.870 INFO:tasks.workunit.client.0.smithi167.stdout:2/904: fdatasync dd/df/d1b/d3d/d4a/d4b/d54/fbd 0 2022-01-31T19:44:22.874 INFO:tasks.workunit.client.0.smithi167.stdout:2/905: dread dd/df/d22/f9d [0,4194304] 0 2022-01-31T19:44:22.875 INFO:tasks.workunit.client.0.smithi167.stdout:2/906: symlink dd/df/d1b/d3d/d4a/da7/dfe/l12e 0 2022-01-31T19:44:22.875 INFO:tasks.workunit.client.0.smithi167.stdout:2/907: chown dd/df/d1b/d3d/d4a/d4b/d74/f12a 428 1 2022-01-31T19:44:22.881 INFO:tasks.workunit.client.0.smithi167.stdout:8/960: dwrite d6/db/de/f86 [0,4194304] 0 2022-01-31T19:44:22.884 INFO:tasks.workunit.client.0.smithi167.stdout:8/961: link d6/db/d36/la6 d6/d1e/d4c/dba/l131 0 2022-01-31T19:44:22.885 INFO:tasks.workunit.client.0.smithi167.stdout:8/962: creat d6/db/d28/d61/dd5/f132 x:0 0 0 2022-01-31T19:44:22.901 INFO:tasks.workunit.client.0.smithi167.stdout:9/955: dwrite dd/d19/d24/d56/f12f [0,4194304] 0 2022-01-31T19:44:22.902 INFO:tasks.workunit.client.0.smithi167.stdout:2/908: dwrite dd/d18/db7/ffb [0,4194304] 0 2022-01-31T19:44:22.902 INFO:tasks.workunit.client.0.smithi167.stdout:2/909: truncate dd/d18/d63/fa3 4524032 0 2022-01-31T19:44:22.903 INFO:tasks.workunit.client.0.smithi167.stdout:8/963: dwrite d6/f1d [0,4194304] 0 2022-01-31T19:44:22.904 INFO:tasks.workunit.client.0.smithi167.stdout:9/956: rename dd/d17/d80/f54 to dd/d19/df6/f140 0 2022-01-31T19:44:22.905 INFO:tasks.workunit.client.0.smithi167.stdout:9/957: write dd/d19/d59/fce [2754534,118147] 0 2022-01-31T19:44:22.909 INFO:tasks.workunit.client.0.smithi167.stdout:8/964: link d6/d1e/d4c/l6d d6/db/d4e/d73/d88/l133 0 2022-01-31T19:44:22.909 INFO:tasks.workunit.client.0.smithi167.stdout:8/965: write d6/db/d28/d61/de3/fea [152387,19341] 0 2022-01-31T19:44:22.909 INFO:tasks.workunit.client.0.smithi167.stdout:8/966: read - d6/db/d28/d61/dd5/f103 zero size 2022-01-31T19:44:22.910 INFO:tasks.workunit.client.0.smithi167.stdout:8/967: mknod d6/d1e/d4c/dba/c134 0 2022-01-31T19:44:22.911 INFO:tasks.workunit.client.0.smithi167.stdout:9/958: dread dd/fee [0,4194304] 0 2022-01-31T19:44:22.911 INFO:tasks.workunit.client.0.smithi167.stdout:9/959: stat dd/d19/d24 0 2022-01-31T19:44:22.912 INFO:tasks.workunit.client.0.smithi167.stdout:8/968: link d6/d1e/d2d/fbe d6/db/d4e/d117/f135 0 2022-01-31T19:44:22.912 INFO:tasks.workunit.client.0.smithi167.stdout:8/969: readlink d6/db/d28/d61/dd5/dbf/d105/dc2/ld9 0 2022-01-31T19:44:22.916 INFO:tasks.workunit.client.0.smithi167.stdout:2/910: dread dd/df/d1b/d3d/d4a/f5b [0,4194304] 0 2022-01-31T19:44:22.917 INFO:tasks.workunit.client.0.smithi167.stdout:8/970: dread d6/db/d6a/d8d/d9e/fcc [0,4194304] 0 2022-01-31T19:44:22.917 INFO:tasks.workunit.client.0.smithi167.stdout:0/691: sync 2022-01-31T19:44:22.919 INFO:tasks.workunit.client.0.smithi167.stdout:2/911: mkdir dd/df/d1b/d3d/d4a/da7/dfe/d12f 0 2022-01-31T19:44:22.919 INFO:tasks.workunit.client.0.smithi167.stdout:2/912: dread - dd/df/d22/f108 zero size 2022-01-31T19:44:22.920 INFO:tasks.workunit.client.0.smithi167.stdout:8/971: creat d6/db/d28/daf/f136 x:0 0 0 2022-01-31T19:44:22.920 INFO:tasks.workunit.client.0.smithi167.stdout:8/972: readlink d6/d1e/d2d/d6b/ldc 0 2022-01-31T19:44:22.921 INFO:tasks.workunit.client.0.smithi167.stdout:2/913: write dd/d18/f4e [481421,34654] 0 2022-01-31T19:44:22.921 INFO:tasks.workunit.client.0.smithi167.stdout:0/692: mkdir d17/d19/d1c/d39/d44/d88/d98/de8 0 2022-01-31T19:44:22.923 INFO:tasks.workunit.client.0.smithi167.stdout:8/973: rename d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/ff9 to d6/db/d36/d124/f137 0 2022-01-31T19:44:22.925 INFO:tasks.workunit.client.0.smithi167.stdout:2/914: creat dd/df/d1b/d3d/d4a/da7/dfe/d12f/f130 x:0 0 0 2022-01-31T19:44:22.952 INFO:tasks.workunit.client.0.smithi167.stdout:8/974: dwrite d6/db/d28/f43 [0,4194304] 0 2022-01-31T19:44:22.954 INFO:tasks.workunit.client.0.smithi167.stdout:0/693: dwrite d17/d19/fe7 [0,4194304] 0 2022-01-31T19:44:22.958 INFO:tasks.workunit.client.0.smithi167.stdout:0/694: truncate d17/d19/d1c/f8f 424581 0 2022-01-31T19:44:22.958 INFO:tasks.workunit.client.0.smithi167.stdout:0/695: write d17/d2c/fb5 [33948,63792] 0 2022-01-31T19:44:22.959 INFO:tasks.workunit.client.0.smithi167.stdout:0/696: write d17/d2c/f76 [1125324,122804] 0 2022-01-31T19:44:22.959 INFO:tasks.workunit.client.0.smithi167.stdout:0/697: fdatasync d17/d33/f95 0 2022-01-31T19:44:22.959 INFO:tasks.workunit.client.0.smithi167.stdout:0/698: read - d17/d19/d1c/d39/d7b/da9/fb7 zero size 2022-01-31T19:44:22.959 INFO:tasks.workunit.client.0.smithi167.stdout:0/699: creat d17/d2c/dcd/fe9 x:0 0 0 2022-01-31T19:44:22.972 INFO:tasks.workunit.client.0.smithi167.stdout:8/975: dwrite d6/db/d28/d10c/f125 [0,4194304] 0 2022-01-31T19:44:22.980 INFO:tasks.workunit.client.0.smithi167.stdout:0/700: dwrite d17/d19/d1c/d39/d44/f56 [0,4194304] 0 2022-01-31T19:44:22.980 INFO:tasks.workunit.client.0.smithi167.stdout:0/701: fdatasync d17/d3e/f42 0 2022-01-31T19:44:22.998 INFO:tasks.workunit.client.0.smithi167.stdout:8/976: dwrite d6/d1e/dcf/ff5 [0,4194304] 0 2022-01-31T19:44:22.998 INFO:tasks.workunit.client.0.smithi167.stdout:8/977: truncate d6/db/d28/d61/de3/f116 831840 0 2022-01-31T19:44:23.000 INFO:tasks.workunit.client.0.smithi167.stdout:0/702: dwrite d17/d19/fb1 [0,4194304] 0 2022-01-31T19:44:23.000 INFO:tasks.workunit.client.0.smithi167.stdout:0/703: write d17/d19/d1c/d39/d44/d49/fa5 [360710,72536] 0 2022-01-31T19:44:23.007 INFO:tasks.workunit.client.0.smithi167.stdout:8/978: dread d6/db/d28/d61/dd5/dbf/d105/f108 [0,4194304] 0 2022-01-31T19:44:23.008 INFO:tasks.workunit.client.0.smithi167.stdout:0/704: symlink d17/d3e/da1/lea 0 2022-01-31T19:44:23.011 INFO:tasks.workunit.client.0.smithi167.stdout:8/979: creat d6/db/d36/d124/f138 x:0 0 0 2022-01-31T19:44:23.012 INFO:tasks.workunit.client.0.smithi167.stdout:0/705: rmdir d17 39 2022-01-31T19:44:23.012 INFO:tasks.workunit.client.0.smithi167.stdout:0/706: readlink d17/d19/d1c/d39/d7b/lb3 0 2022-01-31T19:44:23.013 INFO:tasks.workunit.client.0.smithi167.stdout:0/707: chown d17/d3e/f9b 100051 1 2022-01-31T19:44:23.013 INFO:tasks.workunit.client.0.smithi167.stdout:0/708: creat d17/d3e/da1/feb x:0 0 0 2022-01-31T19:44:23.015 INFO:tasks.workunit.client.0.smithi167.stdout:0/709: truncate d17/d3e/f61 56758 0 2022-01-31T19:44:23.016 INFO:tasks.workunit.client.0.smithi167.stdout:0/710: symlink d17/d3e/lec 0 2022-01-31T19:44:23.017 INFO:tasks.workunit.client.0.smithi167.stdout:0/711: truncate d17/d19/d1c/d39/d7b/f81 1657983 0 2022-01-31T19:44:23.017 INFO:tasks.workunit.client.0.smithi167.stdout:0/712: fdatasync d17/d19/d1c/d39/d44/d88/d98/fe4 0 2022-01-31T19:44:23.018 INFO:tasks.workunit.client.0.smithi167.stdout:0/713: truncate d17/d3e/f61 62990 0 2022-01-31T19:44:23.020 INFO:tasks.workunit.client.0.smithi167.stdout:0/714: rename d17/d19/d1c/d39/d7b/da9/fb7 to d17/d19/d1c/d39/d44/fed 0 2022-01-31T19:44:23.027 INFO:tasks.workunit.client.0.smithi167.stdout:8/980: dwrite d6/db/de/d3d/dd6/f11f [0,4194304] 0 2022-01-31T19:44:23.031 INFO:tasks.workunit.client.0.smithi167.stdout:0/715: rmdir d17/d19/d1c 39 2022-01-31T19:44:23.034 INFO:tasks.workunit.client.0.smithi167.stdout:8/981: unlink d6/d1e/d4c/dba/cee 0 2022-01-31T19:44:23.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/716: mkdir d17/d33/da4/dba/dee 0 2022-01-31T19:44:23.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/717: readlink d17/d33/le3 0 2022-01-31T19:44:23.037 INFO:tasks.workunit.client.0.smithi167.stdout:8/982: read d6/db/d28/d61/dd5/dbf/d105/dc2/dc3/fe5 [1208390,105131] 0 2022-01-31T19:44:23.038 INFO:tasks.workunit.client.0.smithi167.stdout:0/718: symlink d17/d19/d1c/d39/d7b/lef 0 2022-01-31T19:44:23.039 INFO:tasks.workunit.client.0.smithi167.stdout:0/719: mknod d17/d33/da4/cf0 0 2022-01-31T19:44:23.040 INFO:tasks.workunit.client.0.smithi167.stdout:0/720: mkdir d17/d3e/d7e/dc7/dde/df1 0 2022-01-31T19:44:23.041 INFO:tasks.workunit.client.0.smithi167.stdout:0/721: symlink d17/d3e/d7e/lf2 0 2022-01-31T19:44:23.042 INFO:tasks.workunit.client.0.smithi167.stdout:0/722: write d17/d3e/da1/fe0 [24538,96514] 0 2022-01-31T19:44:23.043 INFO:tasks.workunit.client.0.smithi167.stdout:0/723: mknod d17/d19/cf3 0 2022-01-31T19:44:23.043 INFO:tasks.workunit.client.0.smithi167.stdout:0/724: truncate d17/f7a 366993 0 2022-01-31T19:44:23.044 INFO:tasks.workunit.client.0.smithi167.stdout:0/725: dread d17/d19/d1c/d39/d44/d49/f6e [0,4194304] 0 2022-01-31T19:44:23.046 INFO:tasks.workunit.client.0.smithi167.stdout:0/726: mkdir d17/d2c/d87/df4 0 2022-01-31T19:44:23.063 INFO:tasks.workunit.client.0.smithi167.stdout:8/983: dwrite d6/db/d4e/f58 [0,4194304] 0 2022-01-31T19:44:23.064 INFO:tasks.workunit.client.0.smithi167.stdout:8/984: creat d6/db/d28/daf/dc4/f139 x:0 0 0 2022-01-31T19:44:23.064 INFO:tasks.workunit.client.0.smithi167.stdout:0/727: dwrite d17/d19/f4d [0,4194304] 0 2022-01-31T19:44:23.064 INFO:tasks.workunit.client.0.smithi167.stdout:0/728: fdatasync d17/f5e 0 2022-01-31T19:44:23.064 INFO:tasks.workunit.client.0.smithi167.stdout:0/729: dread - d17/d2c/d87/fc4 zero size 2022-01-31T19:44:23.065 INFO:tasks.workunit.client.0.smithi167.stdout:8/985: unlink d6/db/d6a/d8d/d9e/fcc 0 2022-01-31T19:44:23.066 INFO:tasks.workunit.client.0.smithi167.stdout:0/730: truncate d17/f2f 2975938 0 2022-01-31T19:44:23.067 INFO:tasks.workunit.client.0.smithi167.stdout:8/986: creat d6/db/d28/d61/dd5/dbf/d105/dc2/f13a x:0 0 0 2022-01-31T19:44:23.068 INFO:tasks.workunit.client.0.smithi167.stdout:0/731: write f10 [3274560,44645] 0 2022-01-31T19:44:23.068 INFO:tasks.workunit.client.0.smithi167.stdout:0/732: stat d17/d3e/d7e 0 2022-01-31T19:44:23.069 INFO:tasks.workunit.client.0.smithi167.stdout:0/733: read d17/d19/f1f [757199,61970] 0 2022-01-31T19:44:23.071 INFO:tasks.workunit.client.0.smithi167.stdout:3/980: sync 2022-01-31T19:44:23.073 INFO:tasks.workunit.client.0.smithi167.stdout:3/981: symlink d9/d29/d3b/d41/df5/l12a 0 2022-01-31T19:44:23.073 INFO:tasks.workunit.client.0.smithi167.stdout:3/982: fsync d9/d29/d3b/d41/d6b/f72 0 2022-01-31T19:44:23.074 INFO:tasks.workunit.client.0.smithi167.stdout:3/983: dread d9/d7f/da8/f112 [0,4194304] 0 2022-01-31T19:44:23.075 INFO:tasks.workunit.client.0.smithi167.stdout:3/984: mknod d9/d29/d9d/c12b 0 2022-01-31T19:44:23.080 INFO:tasks.workunit.client.0.smithi167.stdout:8/987: dread d6/d1e/dcf/fe6 [0,4194304] 0 2022-01-31T19:44:23.089 INFO:tasks.workunit.client.0.smithi167.stdout:0/734: dwrite d17/d2c/f64 [4194304,4194304] 0 2022-01-31T19:44:23.097 INFO:tasks.workunit.client.0.smithi167.stdout:0/735: write d17/d19/d1c/f8f [127784,106490] 0 2022-01-31T19:44:23.097 INFO:tasks.workunit.client.0.smithi167.stdout:0/736: write ff [4858704,42731] 0 2022-01-31T19:44:23.097 INFO:tasks.workunit.client.0.smithi167.stdout:0/737: write d17/d2c/d87/f97 [605310,13786] 0 2022-01-31T19:44:23.098 INFO:tasks.workunit.client.0.smithi167.stdout:0/738: mknod d17/d3e/d7e/dc7/dde/cf5 0 2022-01-31T19:44:23.099 INFO:tasks.workunit.client.0.smithi167.stdout:0/739: stat d17/d19/d1c/d39/d7b/dbd/dd9/ce2 0 2022-01-31T19:44:23.099 INFO:tasks.workunit.client.0.smithi167.stdout:0/740: readlink d17/d2c/l41 0 2022-01-31T19:44:23.100 INFO:tasks.workunit.client.0.smithi167.stdout:8/988: dwrite d6/db/d28/daf/dc4/f139 [0,4194304] 0 2022-01-31T19:44:23.100 INFO:tasks.workunit.client.0.smithi167.stdout:0/741: write d17/d19/d1c/d39/d44/d88/d98/fe4 [1295044,68723] 0 2022-01-31T19:44:23.102 INFO:tasks.workunit.client.0.smithi167.stdout:0/742: rmdir d17/d2c/dcd 39 2022-01-31T19:44:23.103 INFO:tasks.workunit.client.0.smithi167.stdout:8/989: dread d6/db/d28/d61/dd5/f90 [0,4194304] 0 2022-01-31T19:44:23.103 INFO:tasks.workunit.client.0.smithi167.stdout:8/990: stat d6/db/d28/daf/dc4/l10b 0 2022-01-31T19:44:23.105 INFO:tasks.workunit.client.0.smithi167.stdout:0/743: dread d17/f3d [0,4194304] 0 2022-01-31T19:44:23.106 INFO:tasks.workunit.client.0.smithi167.stdout:0/744: readlink d17/d19/l28 0 2022-01-31T19:44:23.106 INFO:tasks.workunit.client.0.smithi167.stdout:0/745: read d17/d19/d1c/d39/d44/d88/f8a [2762000,6758] 0 2022-01-31T19:44:23.109 INFO:tasks.workunit.client.0.smithi167.stdout:0/746: rename d17/d3e/lec to d17/d3e/d7e/dc7/dde/lf6 0 2022-01-31T19:44:23.117 INFO:tasks.workunit.client.0.smithi167.stdout:0/747: fdatasync d17/d19/d1c/f8f 0 2022-01-31T19:44:23.118 INFO:tasks.workunit.client.0.smithi167.stdout:8/991: dwrite d6/db/de/d18/f1b [4194304,4194304] 0 2022-01-31T19:44:23.118 INFO:tasks.workunit.client.0.smithi167.stdout:8/992: getdents d6/d1e/d4c/dba/dec 0 2022-01-31T19:44:23.118 INFO:tasks.workunit.client.0.smithi167.stdout:8/993: stat d6/d1e/d2d/d6b 0 2022-01-31T19:44:23.119 INFO:tasks.workunit.client.0.smithi167.stdout:0/748: dread d17/d2c/f3c [0,4194304] 0 2022-01-31T19:44:23.120 INFO:tasks.workunit.client.0.smithi167.stdout:0/749: chown d17/d33/fa3 26858 1 2022-01-31T19:44:23.120 INFO:tasks.workunit.client.0.smithi167.stdout:0/750: readlink d17/d33/le3 0 2022-01-31T19:44:23.121 INFO:tasks.workunit.client.0.smithi167.stdout:0/751: creat d17/d19/d1c/d39/d7b/dbd/ff7 x:0 0 0 2022-01-31T19:44:23.121 INFO:tasks.workunit.client.0.smithi167.stdout:0/752: write d17/d19/d1c/d39/d7b/dbd/ff7 [781891,73705] 0 2022-01-31T19:44:23.121 INFO:tasks.workunit.client.0.smithi167.stdout:0/753: truncate d17/d3e/d7e/dc7/fd7 1414480 0 2022-01-31T19:44:23.123 INFO:tasks.workunit.client.0.smithi167.stdout:8/994: dread d6/db/de/f10 [0,4194304] 0 2022-01-31T19:44:23.123 INFO:tasks.workunit.client.0.smithi167.stdout:8/995: fdatasync d6/d1e/dcf/ff5 0 2022-01-31T19:44:23.123 INFO:tasks.workunit.client.0.smithi167.stdout:8/996: readlink d6/db/d6a/d8d/l9d 0 2022-01-31T19:44:23.123 INFO:tasks.workunit.client.0.smithi167.stdout:8/997: chown d6/db/d36/d5d/f68 75084 1 2022-01-31T19:44:23.124 INFO:tasks.workunit.client.0.smithi167.stdout:8/998: write d6/db/d4e/d117/f11e [27416,127353] 0 2022-01-31T19:44:23.124 INFO:tasks.workunit.client.0.smithi167.stdout:8/999: fdatasync d6/db/fd 0 2022-01-31T19:44:23.131 INFO:tasks.workunit.client.0.smithi167.stdout:0/754: dread d17/d33/f95 [0,4194304] 0 2022-01-31T19:44:23.151 INFO:tasks.workunit.client.0.smithi167.stdout:0/755: dwrite d17/d19/d1c/d39/d44/d49/f80 [0,4194304] 0 2022-01-31T19:44:23.166 INFO:tasks.workunit.client.1.smithi171.stdout:2/351: sync 2022-01-31T19:44:23.166 INFO:tasks.workunit.client.1.smithi171.stdout:7/205: sync 2022-01-31T19:44:23.166 INFO:tasks.workunit.client.1.smithi171.stdout:0/285: sync 2022-01-31T19:44:23.167 INFO:tasks.workunit.client.1.smithi171.stdout:7/206: rename d1/d1e/d2a/d29/d31 to d1/d1e/d2a/d29/d31/d30/d3c/d46 22 2022-01-31T19:44:23.169 INFO:tasks.workunit.client.1.smithi171.stdout:5/274: sync 2022-01-31T19:44:23.169 INFO:tasks.workunit.client.1.smithi171.stdout:3/278: sync 2022-01-31T19:44:23.169 INFO:tasks.workunit.client.1.smithi171.stdout:9/362: sync 2022-01-31T19:44:23.169 INFO:tasks.workunit.client.1.smithi171.stdout:9/363: fdatasync dd/d1c/f29 0 2022-01-31T19:44:23.170 INFO:tasks.workunit.client.1.smithi171.stdout:9/364: truncate dd/df/f25 910439 0 2022-01-31T19:44:23.170 INFO:tasks.workunit.client.1.smithi171.stdout:1/271: sync 2022-01-31T19:44:23.170 INFO:tasks.workunit.client.1.smithi171.stdout:1/272: truncate dd/df/d19/d1b/f28 92087 0 2022-01-31T19:44:23.170 INFO:tasks.workunit.client.1.smithi171.stdout:1/273: readlink dd/df/d17/l3b 0 2022-01-31T19:44:23.171 INFO:tasks.workunit.client.1.smithi171.stdout:9/365: readlink dd/df/d15/d1a/d64/d6e/l71 0 2022-01-31T19:44:23.171 INFO:tasks.workunit.client.1.smithi171.stdout:1/274: truncate dd/d23/d34/d4b/f52 731867 0 2022-01-31T19:44:23.171 INFO:tasks.workunit.client.1.smithi171.stdout:4/276: sync 2022-01-31T19:44:23.171 INFO:tasks.workunit.client.1.smithi171.stdout:8/214: sync 2022-01-31T19:44:23.172 INFO:tasks.workunit.client.1.smithi171.stdout:8/215: fsync df/d18/d19/f1b 0 2022-01-31T19:44:23.172 INFO:tasks.workunit.client.1.smithi171.stdout:6/229: sync 2022-01-31T19:44:23.172 INFO:tasks.workunit.client.1.smithi171.stdout:8/216: write f0 [5411440,37388] 0 2022-01-31T19:44:23.172 INFO:tasks.workunit.client.1.smithi171.stdout:8/217: write df/d18/f4d [744300,1416] 0 2022-01-31T19:44:23.173 INFO:tasks.workunit.client.1.smithi171.stdout:1/275: read dd/d23/d34/d50/f58 [6565,124611] 0 2022-01-31T19:44:23.174 INFO:tasks.workunit.client.1.smithi171.stdout:8/218: dread df/f20 [0,4194304] 0 2022-01-31T19:44:23.178 INFO:tasks.workunit.client.0.smithi167.stdout:0/756: dwrite d17/d2c/dcd/fda [0,4194304] 0 2022-01-31T19:44:23.180 INFO:tasks.workunit.client.0.smithi167.stdout:0/757: mknod d17/d2c/cf8 0 2022-01-31T19:44:23.181 INFO:tasks.workunit.client.1.smithi171.stdout:7/207: dwrite d1/d24/f44 [0,4194304] 0 2022-01-31T19:44:23.181 INFO:tasks.workunit.client.1.smithi171.stdout:7/208: chown d1/d24/c13 11 1 2022-01-31T19:44:23.182 INFO:tasks.workunit.client.0.smithi167.stdout:0/758: mkdir d17/d3e/d7e/dc7/dde/df9 0 2022-01-31T19:44:23.182 INFO:tasks.workunit.client.0.smithi167.stdout:0/759: write d17/f3d [3984690,82845] 0 2022-01-31T19:44:23.184 INFO:tasks.workunit.client.0.smithi167.stdout:0/760: truncate d17/d19/d1c/d39/d44/f56 63893 0 2022-01-31T19:44:23.184 INFO:tasks.workunit.client.0.smithi167.stdout:0/761: getdents d17/d19/d1c/d39/d7b/de5 0 2022-01-31T19:44:23.188 INFO:tasks.workunit.client.0.smithi167.stdout:0/762: dread d17/d69/f78 [0,4194304] 0 2022-01-31T19:44:23.190 INFO:tasks.workunit.client.0.smithi167.stdout:0/763: link d17/d19/d1c/d39/d7b/cd4 d17/d33/da4/dba/dee/cfa 0 2022-01-31T19:44:23.195 INFO:tasks.workunit.client.0.smithi167.stdout:0/764: unlink d17/d2c/d87/fc4 0 2022-01-31T19:44:23.196 INFO:tasks.workunit.client.0.smithi167.stdout:0/765: getdents d17/d3e/d7e/dc7/dde 0 2022-01-31T19:44:23.197 INFO:tasks.workunit.client.0.smithi167.stdout:0/766: symlink d17/d33/da4/dba/dee/lfb 0 2022-01-31T19:44:23.199 INFO:tasks.workunit.client.0.smithi167.stdout:0/767: creat d17/d19/d1c/d39/ffc x:0 0 0 2022-01-31T19:44:23.199 INFO:tasks.workunit.client.0.smithi167.stdout:0/768: readlink d17/d69/l92 0 2022-01-31T19:44:23.200 INFO:tasks.workunit.client.0.smithi167.stdout:0/769: creat d17/d3e/d7e/dc7/dde/df1/ffd x:0 0 0 2022-01-31T19:44:23.201 INFO:tasks.workunit.client.0.smithi167.stdout:0/770: write d17/d19/d1c/d39/d44/fed [636934,115945] 0 2022-01-31T19:44:23.204 INFO:tasks.workunit.client.1.smithi171.stdout:1/276: dwrite dd/df/d19/d1b/f3e [0,4194304] 0 2022-01-31T19:44:23.210 INFO:tasks.workunit.client.1.smithi171.stdout:7/209: dwrite d1/d1e/d2a/d29/d31/f19 [0,4194304] 0 2022-01-31T19:44:23.210 INFO:tasks.workunit.client.1.smithi171.stdout:7/210: fsync d1/d1e/d2a/d29/d31/d30/f3f 0 2022-01-31T19:44:23.215 INFO:tasks.workunit.client.0.smithi167.stdout:0/771: dread d17/d2c/dcd/fd6 [0,4194304] 0 2022-01-31T19:44:23.216 INFO:tasks.workunit.client.0.smithi167.stdout:0/772: write d17/d19/d1c/d39/d44/d49/f80 [769165,95423] 0 2022-01-31T19:44:23.216 INFO:tasks.workunit.client.0.smithi167.stdout:0/773: truncate d17/d19/d1c/d39/d7b/f8c 787880 0 2022-01-31T19:44:23.217 INFO:tasks.workunit.client.0.smithi167.stdout:0/774: getdents d17/d19/d1c/d39/d44/d88 0 2022-01-31T19:44:23.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/211: dread d1/d1e/d2a/d29/d31/f19 [0,4194304] 0 2022-01-31T19:44:23.223 INFO:tasks.workunit.client.0.smithi167.stdout:6/793: sync 2022-01-31T19:44:23.224 INFO:tasks.workunit.client.0.smithi167.stdout:6/794: getdents d8/d33/d51/d5d 0 2022-01-31T19:44:23.224 INFO:tasks.workunit.client.0.smithi167.stdout:6/795: write d8/d33/d51/d5d/f9f [39715,80348] 0 2022-01-31T19:44:23.224 INFO:tasks.workunit.client.0.smithi167.stdout:6/796: readlink d8/d62/d92/lfc 0 2022-01-31T19:44:23.225 INFO:tasks.workunit.client.0.smithi167.stdout:0/775: write d17/f7a [84270,54532] 0 2022-01-31T19:44:23.226 INFO:tasks.workunit.client.0.smithi167.stdout:0/776: mknod d17/d2c/dcd/cfe 0 2022-01-31T19:44:23.226 INFO:tasks.workunit.client.0.smithi167.stdout:0/777: fdatasync d17/d2c/fb5 0 2022-01-31T19:44:23.228 INFO:tasks.workunit.client.0.smithi167.stdout:0/778: creat d17/d3e/d7e/dc7/dde/fff x:0 0 0 2022-01-31T19:44:23.229 INFO:tasks.workunit.client.0.smithi167.stdout:0/779: creat d17/d19/f100 x:0 0 0 2022-01-31T19:44:23.229 INFO:tasks.workunit.client.0.smithi167.stdout:0/780: chown d17/d33/c5d 0 1 2022-01-31T19:44:23.229 INFO:tasks.workunit.client.0.smithi167.stdout:0/781: fsync d17/d2c/dcd/fd6 0 2022-01-31T19:44:23.229 INFO:tasks.workunit.client.0.smithi167.stdout:0/782: chown d17/d3e/da1/fe0 82759618 1 2022-01-31T19:44:23.230 INFO:tasks.workunit.client.0.smithi167.stdout:0/783: fsync d17/d33/f95 0 2022-01-31T19:44:23.234 INFO:tasks.workunit.client.1.smithi171.stdout:2/352: rmdir d7/d55/d61 0 2022-01-31T19:44:23.235 INFO:tasks.workunit.client.0.smithi167.stdout:0/784: dread d17/d2c/dcd/fd6 [0,4194304] 0 2022-01-31T19:44:23.236 INFO:tasks.workunit.client.0.smithi167.stdout:0/785: chown d17/d19/d1c/d39/d44/f93 57009 1 2022-01-31T19:44:23.236 INFO:tasks.workunit.client.1.smithi171.stdout:0/286: creat d9/db/dc/d3f/f6d x:0 0 0 2022-01-31T19:44:23.237 INFO:tasks.workunit.client.0.smithi167.stdout:0/786: link d17/d3e/c52 d17/d19/d1c/d39/d44/d49/c101 0 2022-01-31T19:44:23.237 INFO:tasks.workunit.client.0.smithi167.stdout:0/787: chown d17/d3e/c52 31 1 2022-01-31T19:44:23.237 INFO:tasks.workunit.client.0.smithi167.stdout:0/788: stat d17/d33/da4/dba/dee/lfb 0 2022-01-31T19:44:23.238 INFO:tasks.workunit.client.0.smithi167.stdout:0/789: creat d17/d3e/d7e/dc7/dde/df1/f102 x:0 0 0 2022-01-31T19:44:23.239 INFO:tasks.workunit.client.0.smithi167.stdout:0/790: rename d17/d19/c2a to d17/d19/d1c/d39/d7b/da9/c103 0 2022-01-31T19:44:23.240 INFO:tasks.workunit.client.0.smithi167.stdout:0/791: chown d17/d19/d1c/d39/d7b/dbd/dd9 28404 1 2022-01-31T19:44:23.240 INFO:tasks.workunit.client.1.smithi171.stdout:3/279: unlink d4/d21/c34 0 2022-01-31T19:44:23.241 INFO:tasks.workunit.client.0.smithi167.stdout:0/792: mkdir d17/d19/d1c/d39/d7b/de5/d104 0 2022-01-31T19:44:23.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/366: mkdir dd/d1c/d7c 0 2022-01-31T19:44:23.242 INFO:tasks.workunit.client.0.smithi167.stdout:0/793: mkdir d17/d33/da4/dd0/d105 0 2022-01-31T19:44:23.242 INFO:tasks.workunit.client.1.smithi171.stdout:4/277: mkdir d2/d3/d5/d11/d12/d15/d57 0 2022-01-31T19:44:23.244 INFO:tasks.workunit.client.0.smithi167.stdout:0/794: rename d17/d33/c5d to d17/d19/d1c/d39/d44/d88/d98/c106 0 2022-01-31T19:44:23.245 INFO:tasks.workunit.client.0.smithi167.stdout:0/795: dread - d17/d19/d1c/d39/fc5 zero size 2022-01-31T19:44:23.247 INFO:tasks.workunit.client.1.smithi171.stdout:6/230: unlink dc/c11 0 2022-01-31T19:44:23.247 INFO:tasks.workunit.client.1.smithi171.stdout:8/219: unlink df/d2a/l48 0 2022-01-31T19:44:23.252 INFO:tasks.workunit.client.0.smithi167.stdout:6/797: dwrite d8/d33/d51/d6a/fa3 [0,4194304] 0 2022-01-31T19:44:23.252 INFO:tasks.workunit.client.0.smithi167.stdout:0/796: dread d17/d19/d1c/d39/d7b/dbd/ff7 [0,4194304] 0 2022-01-31T19:44:23.252 INFO:tasks.workunit.client.0.smithi167.stdout:0/797: dread - d17/d3e/d7e/dc7/dde/fff zero size 2022-01-31T19:44:23.253 INFO:tasks.workunit.client.1.smithi171.stdout:1/277: rename dd/df/d19/f21 to dd/d23/f5c 0 2022-01-31T19:44:23.254 INFO:tasks.workunit.client.1.smithi171.stdout:4/278: dread d2/f14 [0,4194304] 0 2022-01-31T19:44:23.254 INFO:tasks.workunit.client.1.smithi171.stdout:2/353: creat d7/d9/f6a x:0 0 0 2022-01-31T19:44:23.255 INFO:tasks.workunit.client.1.smithi171.stdout:5/275: mkdir d5/d6/d15/d5d 0 2022-01-31T19:44:23.256 INFO:tasks.workunit.client.1.smithi171.stdout:9/367: mkdir dd/d1c/d7c/d7d 0 2022-01-31T19:44:23.256 INFO:tasks.workunit.client.1.smithi171.stdout:9/368: truncate dd/df/d15/d38/d4d/f63 619147 0 2022-01-31T19:44:23.257 INFO:tasks.workunit.client.0.smithi167.stdout:0/798: symlink d17/d19/d1c/d39/d7b/dbd/l107 0 2022-01-31T19:44:23.257 INFO:tasks.workunit.client.0.smithi167.stdout:0/799: read - d17/d33/da4/faa zero size 2022-01-31T19:44:23.258 INFO:tasks.workunit.client.1.smithi171.stdout:8/220: creat df/d2a/d38/f50 x:0 0 0 2022-01-31T19:44:23.259 INFO:tasks.workunit.client.1.smithi171.stdout:3/280: dread d4/dd/f20 [0,4194304] 0 2022-01-31T19:44:23.259 INFO:tasks.workunit.client.1.smithi171.stdout:1/278: mknod dd/df/c5d 0 2022-01-31T19:44:23.260 INFO:tasks.workunit.client.1.smithi171.stdout:7/212: dwrite d1/d1e/d2a/d29/d31/fe [0,4194304] 0 2022-01-31T19:44:23.260 INFO:tasks.workunit.client.1.smithi171.stdout:7/213: fdatasync d1/d1e/d2a/d29/d31/d30/f39 0 2022-01-31T19:44:23.261 INFO:tasks.workunit.client.1.smithi171.stdout:3/281: dread d4/dd/d40/f42 [0,4194304] 0 2022-01-31T19:44:23.261 INFO:tasks.workunit.client.0.smithi167.stdout:0/800: link d17/l5b d17/d19/d1c/d39/d44/d88/d98/l108 0 2022-01-31T19:44:23.262 INFO:tasks.workunit.client.0.smithi167.stdout:0/801: creat d17/d2c/d87/f109 x:0 0 0 2022-01-31T19:44:23.263 INFO:tasks.workunit.client.1.smithi171.stdout:4/279: rename d2/d3/d5/l2b to d2/d3/d5/d11/l58 0 2022-01-31T19:44:23.264 INFO:tasks.workunit.client.0.smithi167.stdout:0/802: creat d17/d2c/d87/df4/f10a x:0 0 0 2022-01-31T19:44:23.264 INFO:tasks.workunit.client.0.smithi167.stdout:0/803: truncate d17/d2c/d87/f97 699533 0 2022-01-31T19:44:23.265 INFO:tasks.workunit.client.1.smithi171.stdout:2/354: mkdir d7/dd/d5e/d6b 0 2022-01-31T19:44:23.265 INFO:tasks.workunit.client.1.smithi171.stdout:2/355: truncate d7/d9/f46 1906067 0 2022-01-31T19:44:23.267 INFO:tasks.workunit.client.1.smithi171.stdout:5/276: rename d5/d6/d15/d29/f34 to d5/d6/f5e 0 2022-01-31T19:44:23.267 INFO:tasks.workunit.client.1.smithi171.stdout:5/277: creat d5/d6/dd/f5f x:0 0 0 2022-01-31T19:44:23.268 INFO:tasks.workunit.client.0.smithi167.stdout:0/804: creat d17/d19/d1c/d39/d7b/dbd/dd9/f10b x:0 0 0 2022-01-31T19:44:23.269 INFO:tasks.workunit.client.1.smithi171.stdout:2/356: creat d7/d26/d3d/d40/f6c x:0 0 0 2022-01-31T19:44:23.269 INFO:tasks.workunit.client.1.smithi171.stdout:2/357: write d7/d9/fe [891389,38719] 0 2022-01-31T19:44:23.269 INFO:tasks.workunit.client.0.smithi167.stdout:0/805: getdents d17/d3e/d7e 0 2022-01-31T19:44:23.277 INFO:tasks.workunit.client.1.smithi171.stdout:1/279: rename dd/df/d17 to dd/d23/d2e/d5e 0 2022-01-31T19:44:23.280 INFO:tasks.workunit.client.1.smithi171.stdout:5/278: write d5/d6/d15/f3f [619374,82704] 0 2022-01-31T19:44:23.282 INFO:tasks.workunit.client.1.smithi171.stdout:0/287: dwrite d9/db/dc/d3f/f6d [0,4194304] 0 2022-01-31T19:44:23.284 INFO:tasks.workunit.client.1.smithi171.stdout:9/369: dread f4 [4194304,4194304] 0 2022-01-31T19:44:23.285 INFO:tasks.workunit.client.1.smithi171.stdout:9/370: readlink dd/d1c/l39 0 2022-01-31T19:44:23.285 INFO:tasks.workunit.client.1.smithi171.stdout:9/371: fdatasync dd/df/d15/d1a/f2b 0 2022-01-31T19:44:23.286 INFO:tasks.workunit.client.1.smithi171.stdout:0/288: rename d9/db/dc/d3f/c27 to d9/db/d59/c6e 0 2022-01-31T19:44:23.286 INFO:tasks.workunit.client.1.smithi171.stdout:9/372: truncate f4 3287 0 2022-01-31T19:44:23.286 INFO:tasks.workunit.client.1.smithi171.stdout:9/373: readlink dd/df/d15/d1a/d64/d6e/l71 0 2022-01-31T19:44:23.287 INFO:tasks.workunit.client.1.smithi171.stdout:9/374: stat dd/d1c/l39 0 2022-01-31T19:44:23.287 INFO:tasks.workunit.client.1.smithi171.stdout:9/375: chown dd/df/d15/f3a 397611527 1 2022-01-31T19:44:23.287 INFO:tasks.workunit.client.1.smithi171.stdout:0/289: mkdir d9/db/dc/d3f/d6f 0 2022-01-31T19:44:23.288 INFO:tasks.workunit.client.1.smithi171.stdout:0/290: creat d9/db/dc/f70 x:0 0 0 2022-01-31T19:44:23.289 INFO:tasks.workunit.client.1.smithi171.stdout:9/376: dread dd/df/f25 [0,4194304] 0 2022-01-31T19:44:23.289 INFO:tasks.workunit.client.1.smithi171.stdout:9/377: chown dd/df/d15/d53 33 1 2022-01-31T19:44:23.289 INFO:tasks.workunit.client.1.smithi171.stdout:9/378: write dd/df/d15/f16 [268688,108177] 0 2022-01-31T19:44:23.290 INFO:tasks.workunit.client.1.smithi171.stdout:0/291: mkdir d9/db/d59/d71 0 2022-01-31T19:44:23.291 INFO:tasks.workunit.client.1.smithi171.stdout:0/292: symlink d9/d24/d4c/d35/l72 0 2022-01-31T19:44:23.291 INFO:tasks.workunit.client.1.smithi171.stdout:0/293: mkdir d9/db/d22/d73 0 2022-01-31T19:44:23.297 INFO:tasks.workunit.client.1.smithi171.stdout:6/231: dwrite dc/d19/d29/f24 [0,4194304] 0 2022-01-31T19:44:23.300 INFO:tasks.workunit.client.1.smithi171.stdout:6/232: rename dc/d19/d29/d2f to dc/d1e/d3f 0 2022-01-31T19:44:23.300 INFO:tasks.workunit.client.1.smithi171.stdout:6/233: stat dc/d1e 0 2022-01-31T19:44:23.301 INFO:tasks.workunit.client.1.smithi171.stdout:6/234: mknod dc/c40 0 2022-01-31T19:44:23.301 INFO:tasks.workunit.client.1.smithi171.stdout:6/235: readlink l7 0 2022-01-31T19:44:23.302 INFO:tasks.workunit.client.1.smithi171.stdout:6/236: truncate dc/d19/d30/f3d 234263 0 2022-01-31T19:44:23.303 INFO:tasks.workunit.client.1.smithi171.stdout:8/221: dwrite f5 [4194304,4194304] 0 2022-01-31T19:44:23.303 INFO:tasks.workunit.client.1.smithi171.stdout:8/222: write df/d18/f4e [920345,69008] 0 2022-01-31T19:44:23.304 INFO:tasks.workunit.client.1.smithi171.stdout:1/280: dwrite dd/d23/d2e/d5e/f45 [0,4194304] 0 2022-01-31T19:44:23.305 INFO:tasks.workunit.client.1.smithi171.stdout:8/223: truncate df/f16 138021 0 2022-01-31T19:44:23.306 INFO:tasks.workunit.client.1.smithi171.stdout:6/237: write dc/d19/d29/f24 [3058656,64766] 0 2022-01-31T19:44:23.308 INFO:tasks.workunit.client.1.smithi171.stdout:8/224: creat df/f51 x:0 0 0 2022-01-31T19:44:23.311 INFO:tasks.workunit.client.1.smithi171.stdout:8/225: read f3 [272664,16632] 0 2022-01-31T19:44:23.311 INFO:tasks.workunit.client.1.smithi171.stdout:8/226: creat df/d18/d19/d36/f52 x:0 0 0 2022-01-31T19:44:23.313 INFO:tasks.workunit.client.1.smithi171.stdout:8/227: creat df/d1a/f53 x:0 0 0 2022-01-31T19:44:23.313 INFO:tasks.workunit.client.1.smithi171.stdout:8/228: mknod df/c54 0 2022-01-31T19:44:23.326 INFO:tasks.workunit.client.1.smithi171.stdout:8/229: dread df/d18/f1e [0,4194304] 0 2022-01-31T19:44:23.330 INFO:tasks.workunit.client.1.smithi171.stdout:1/281: dwrite dd/df/d19/f27 [0,4194304] 0 2022-01-31T19:44:23.330 INFO:tasks.workunit.client.1.smithi171.stdout:1/282: write dd/d23/f41 [7031,7181] 0 2022-01-31T19:44:23.333 INFO:tasks.workunit.client.1.smithi171.stdout:1/283: rename dd/df/d19/d1b/d1e/f44 to dd/d23/d2e/d5e/f5f 0 2022-01-31T19:44:23.333 INFO:tasks.workunit.client.1.smithi171.stdout:1/284: chown dd/c37 82708 1 2022-01-31T19:44:23.333 INFO:tasks.workunit.client.1.smithi171.stdout:1/285: readlink dd/d23/d34/d4b/l4c 0 2022-01-31T19:44:23.333 INFO:tasks.workunit.client.1.smithi171.stdout:1/286: chown dd/df/d19/d36 0 1 2022-01-31T19:44:23.334 INFO:tasks.workunit.client.1.smithi171.stdout:1/287: truncate dd/d23/d2e/d5e/f20 4625970 0 2022-01-31T19:44:23.334 INFO:tasks.workunit.client.1.smithi171.stdout:1/288: creat dd/d23/f60 x:0 0 0 2022-01-31T19:44:23.334 INFO:tasks.workunit.client.1.smithi171.stdout:1/289: chown dd/df/d19/d36/f5b 812 1 2022-01-31T19:44:23.341 INFO:tasks.workunit.client.1.smithi171.stdout:1/290: dread fa [0,4194304] 0 2022-01-31T19:44:23.365 INFO:tasks.workunit.client.1.smithi171.stdout:2/358: write d7/d9/f4e [14575,44401] 0 2022-01-31T19:44:23.391 INFO:tasks.workunit.client.1.smithi171.stdout:3/282: dwrite d4/dd/d1a/f1b [0,4194304] 0 2022-01-31T19:44:23.391 INFO:tasks.workunit.client.1.smithi171.stdout:1/291: dwrite dd/df/d19/d36/f5b [0,4194304] 0 2022-01-31T19:44:23.392 INFO:tasks.workunit.client.1.smithi171.stdout:3/283: truncate d4/fc 2815721 0 2022-01-31T19:44:23.393 INFO:tasks.workunit.client.0.smithi167.stdout:6/798: dwrite d8/d20/f2f [0,4194304] 0 2022-01-31T19:44:23.394 INFO:tasks.workunit.client.1.smithi171.stdout:1/292: truncate dd/df/d19/f32 1343642 0 2022-01-31T19:44:23.395 INFO:tasks.workunit.client.1.smithi171.stdout:3/284: creat d4/d21/d3a/f4f x:0 0 0 2022-01-31T19:44:23.395 INFO:tasks.workunit.client.1.smithi171.stdout:3/285: chown d4/dd/d27/d2e 10 1 2022-01-31T19:44:23.396 INFO:tasks.workunit.client.1.smithi171.stdout:3/286: chown d4/dd/l15 55448177 1 2022-01-31T19:44:23.396 INFO:tasks.workunit.client.1.smithi171.stdout:2/359: dwrite d7/d9/fe [0,4194304] 0 2022-01-31T19:44:23.397 INFO:tasks.workunit.client.1.smithi171.stdout:3/287: creat d4/dd/d39/f50 x:0 0 0 2022-01-31T19:44:23.399 INFO:tasks.workunit.client.1.smithi171.stdout:5/279: dwrite d5/d6/d15/d29/f39 [0,4194304] 0 2022-01-31T19:44:23.400 INFO:tasks.workunit.client.1.smithi171.stdout:6/238: dwrite dc/d18/f39 [0,4194304] 0 2022-01-31T19:44:23.400 INFO:tasks.workunit.client.1.smithi171.stdout:5/280: fsync d5/ff 0 2022-01-31T19:44:23.401 INFO:tasks.workunit.client.0.smithi167.stdout:6/799: getdents d8/d33/d51/d7e 0 2022-01-31T19:44:23.401 INFO:tasks.workunit.client.1.smithi171.stdout:7/214: dwrite d1/d1b/fc [4194304,4194304] 0 2022-01-31T19:44:23.401 INFO:tasks.workunit.client.1.smithi171.stdout:7/215: dread - d1/d1e/f3d zero size 2022-01-31T19:44:23.414 INFO:tasks.workunit.client.1.smithi171.stdout:3/288: mkdir d4/dd/d39/d51 0 2022-01-31T19:44:23.420 INFO:tasks.workunit.client.0.smithi167.stdout:2/915: sync 2022-01-31T19:44:23.421 INFO:tasks.workunit.client.0.smithi167.stdout:2/916: creat dd/df/d22/d96/d9e/dda/de6/f131 x:0 0 0 2022-01-31T19:44:23.423 INFO:tasks.workunit.client.0.smithi167.stdout:2/917: mkdir dd/d18/d62/d75/d105/d5f/d132 0 2022-01-31T19:44:23.423 INFO:tasks.workunit.client.1.smithi171.stdout:0/294: rmdir d9/db/d22 39 2022-01-31T19:44:23.424 INFO:tasks.workunit.client.0.smithi167.stdout:2/918: getdents dd/df/d1b 0 2022-01-31T19:44:23.424 INFO:tasks.workunit.client.0.smithi167.stdout:2/919: write dd/d18/d8d/f117 [751946,68331] 0 2022-01-31T19:44:23.425 INFO:tasks.workunit.client.0.smithi167.stdout:9/960: sync 2022-01-31T19:44:23.425 INFO:tasks.workunit.client.0.smithi167.stdout:2/920: truncate dd/d18/f99 1109205 0 2022-01-31T19:44:23.426 INFO:tasks.workunit.client.0.smithi167.stdout:9/961: rename dd/d19/d24/d56/cac to dd/d19/d24/d56/d5b/c141 0 2022-01-31T19:44:23.428 INFO:tasks.workunit.client.0.smithi167.stdout:9/962: truncate dd/d19/d24/d50/dd6/f103 3941700 0 2022-01-31T19:44:23.429 INFO:tasks.workunit.client.0.smithi167.stdout:9/963: chown dd/d19/l90 112282 1 2022-01-31T19:44:23.429 INFO:tasks.workunit.client.1.smithi171.stdout:6/239: dwrite f9 [4194304,4194304] 0 2022-01-31T19:44:23.431 INFO:tasks.workunit.client.0.smithi167.stdout:9/964: rename dd/d17/d7f/def/c124 to dd/d19/d24/d56/d5b/c142 0 2022-01-31T19:44:23.432 INFO:tasks.workunit.client.1.smithi171.stdout:8/230: fsync df/d1a/f53 0 2022-01-31T19:44:23.445 INFO:tasks.workunit.client.1.smithi171.stdout:3/289: dwrite d4/dd/d27/f47 [0,4194304] 0 2022-01-31T19:44:23.445 INFO:tasks.workunit.client.1.smithi171.stdout:7/216: mkdir d1/d26/d47 0 2022-01-31T19:44:23.449 INFO:tasks.workunit.client.1.smithi171.stdout:0/295: truncate d9/f53 2253102 0 2022-01-31T19:44:23.450 INFO:tasks.workunit.client.1.smithi171.stdout:6/240: mknod dc/d21/c41 0 2022-01-31T19:44:23.453 INFO:tasks.workunit.client.1.smithi171.stdout:3/290: mkdir d4/dd/d1a/d52 0 2022-01-31T19:44:23.461 INFO:tasks.workunit.client.1.smithi171.stdout:8/231: dwrite df/d18/d19/d21/f32 [0,4194304] 0 2022-01-31T19:44:23.462 INFO:tasks.workunit.client.1.smithi171.stdout:7/217: rmdir d1/d1e 39 2022-01-31T19:44:23.466 INFO:tasks.workunit.client.1.smithi171.stdout:8/232: link df/d18/d19/d21/f32 df/d18/f55 0 2022-01-31T19:44:23.466 INFO:tasks.workunit.client.1.smithi171.stdout:8/233: write df/f4f [689657,52310] 0 2022-01-31T19:44:23.467 INFO:tasks.workunit.client.1.smithi171.stdout:8/234: creat df/d2a/d38/f56 x:0 0 0 2022-01-31T19:44:23.467 INFO:tasks.workunit.client.1.smithi171.stdout:7/218: rename d1/d1b/l2b to d1/d1b/l48 0 2022-01-31T19:44:23.472 INFO:tasks.workunit.client.1.smithi171.stdout:8/235: creat df/d27/f57 x:0 0 0 2022-01-31T19:44:23.473 INFO:tasks.workunit.client.1.smithi171.stdout:7/219: getdents d1/d1e/d2a/d29/d31/d30/d3c 0 2022-01-31T19:44:23.473 INFO:tasks.workunit.client.1.smithi171.stdout:7/220: chown d1/d1e/d2a/d29 2627 1 2022-01-31T19:44:23.473 INFO:tasks.workunit.client.1.smithi171.stdout:7/221: write d1/d1b/f22 [394121,11285] 0 2022-01-31T19:44:23.474 INFO:tasks.workunit.client.1.smithi171.stdout:8/236: truncate f3 712965 0 2022-01-31T19:44:23.474 INFO:tasks.workunit.client.1.smithi171.stdout:8/237: write df/d18/d19/f1b [1091976,33660] 0 2022-01-31T19:44:23.475 INFO:tasks.workunit.client.1.smithi171.stdout:8/238: write df/d2a/f49 [742468,124356] 0 2022-01-31T19:44:23.475 INFO:tasks.workunit.client.1.smithi171.stdout:8/239: chown df/f51 62 1 2022-01-31T19:44:23.476 INFO:tasks.workunit.client.1.smithi171.stdout:7/222: symlink d1/d1e/d2a/d29/d31/d30/d3c/l49 0 2022-01-31T19:44:23.476 INFO:tasks.workunit.client.1.smithi171.stdout:8/240: rename df/d2a/d38/f50 to df/d1a/f58 0 2022-01-31T19:44:23.477 INFO:tasks.workunit.client.1.smithi171.stdout:8/241: dread - df/f51 zero size 2022-01-31T19:44:23.477 INFO:tasks.workunit.client.1.smithi171.stdout:7/223: symlink d1/d1e/d2a/l4a 0 2022-01-31T19:44:23.478 INFO:tasks.workunit.client.1.smithi171.stdout:8/242: symlink df/l59 0 2022-01-31T19:44:23.480 INFO:tasks.workunit.client.1.smithi171.stdout:7/224: creat d1/d1b/f4b x:0 0 0 2022-01-31T19:44:23.481 INFO:tasks.workunit.client.1.smithi171.stdout:7/225: creat d1/d1b/f4c x:0 0 0 2022-01-31T19:44:23.525 INFO:tasks.workunit.client.1.smithi171.stdout:2/360: dwrite d7/f56 [0,4194304] 0 2022-01-31T19:44:23.526 INFO:tasks.workunit.client.1.smithi171.stdout:2/361: write d7/d26/d3d/d40/f51 [617768,22955] 0 2022-01-31T19:44:23.526 INFO:tasks.workunit.client.1.smithi171.stdout:2/362: fsync d7/d9/f4e 0 2022-01-31T19:44:23.526 INFO:tasks.workunit.client.1.smithi171.stdout:2/363: truncate d7/d26/d63/f67 776559 0 2022-01-31T19:44:23.528 INFO:tasks.workunit.client.0.smithi167.stdout:6/800: dwrite d8/d33/d51/d72/f95 [0,4194304] 0 2022-01-31T19:44:23.528 INFO:tasks.workunit.client.0.smithi167.stdout:2/921: dwrite dd/fbf [4194304,4194304] 0 2022-01-31T19:44:23.529 INFO:tasks.workunit.client.0.smithi167.stdout:2/922: truncate dd/df/d1b/d3d/f123 126560 0 2022-01-31T19:44:23.529 INFO:tasks.workunit.client.0.smithi167.stdout:9/965: dwrite dd/d17/d7f/f9d [0,4194304] 0 2022-01-31T19:44:23.529 INFO:tasks.workunit.client.0.smithi167.stdout:2/923: write dd/d18/f4e [4381746,106574] 0 2022-01-31T19:44:23.529 INFO:tasks.workunit.client.0.smithi167.stdout:2/924: chown dd/f15 2414574 1 2022-01-31T19:44:23.530 INFO:tasks.workunit.client.1.smithi171.stdout:2/364: read d7/f22 [4210567,119485] 0 2022-01-31T19:44:23.530 INFO:tasks.workunit.client.1.smithi171.stdout:7/226: dwrite d1/d1e/d2a/d29/f43 [0,4194304] 0 2022-01-31T19:44:23.530 INFO:tasks.workunit.client.1.smithi171.stdout:7/227: read d1/d1b/f22 [65959,58386] 0 2022-01-31T19:44:23.530 INFO:tasks.workunit.client.1.smithi171.stdout:7/228: write d1/d1e/d2a/d29/d31/d30/f39 [634908,73062] 0 2022-01-31T19:44:23.531 INFO:tasks.workunit.client.1.smithi171.stdout:3/291: dwrite d4/dd/f45 [0,4194304] 0 2022-01-31T19:44:23.531 INFO:tasks.workunit.client.1.smithi171.stdout:0/296: dwrite d9/db/f14 [0,4194304] 0 2022-01-31T19:44:23.531 INFO:tasks.workunit.client.1.smithi171.stdout:8/243: dwrite f0 [4194304,4194304] 0 2022-01-31T19:44:23.532 INFO:tasks.workunit.client.1.smithi171.stdout:0/297: write d9/f1d [3201028,55530] 0 2022-01-31T19:44:23.535 INFO:tasks.workunit.client.0.smithi167.stdout:6/801: dread d8/d62/f6f [0,4194304] 0 2022-01-31T19:44:23.535 INFO:tasks.workunit.client.1.smithi171.stdout:2/365: creat d7/d55/f6d x:0 0 0 2022-01-31T19:44:23.536 INFO:tasks.workunit.client.1.smithi171.stdout:2/366: stat d7/c5d 0 2022-01-31T19:44:23.536 INFO:tasks.workunit.client.1.smithi171.stdout:2/367: read d7/d26/d3d/d40/f5c [151530,129844] 0 2022-01-31T19:44:23.536 INFO:tasks.workunit.client.1.smithi171.stdout:2/368: creat d7/d26/d3d/d40/f6e x:0 0 0 2022-01-31T19:44:23.537 INFO:tasks.workunit.client.0.smithi167.stdout:3/985: sync 2022-01-31T19:44:23.537 INFO:tasks.workunit.client.1.smithi171.stdout:7/229: creat d1/d1e/d2a/d29/d31/d30/f4d x:0 0 0 2022-01-31T19:44:23.538 INFO:tasks.workunit.client.0.smithi167.stdout:9/966: write dd/d17/f2e [1131786,1877] 0 2022-01-31T19:44:23.538 INFO:tasks.workunit.client.0.smithi167.stdout:6/802: mknod d8/d33/d51/d5d/d64/c119 0 2022-01-31T19:44:23.539 INFO:tasks.workunit.client.1.smithi171.stdout:3/292: creat d4/dd/d39/d51/f53 x:0 0 0 2022-01-31T19:44:23.540 INFO:tasks.workunit.client.0.smithi167.stdout:3/986: creat d9/d29/d3b/d9b/da3/db0/d119/f12c x:0 0 0 2022-01-31T19:44:23.540 INFO:tasks.workunit.client.0.smithi167.stdout:3/987: creat d9/d7f/f12d x:0 0 0 2022-01-31T19:44:23.540 INFO:tasks.workunit.client.0.smithi167.stdout:2/925: rmdir dd/df/d22/d96/d9e/dda/de6 39 2022-01-31T19:44:23.541 INFO:tasks.workunit.client.0.smithi167.stdout:2/926: read dd/df/f5d [2982583,122907] 0 2022-01-31T19:44:23.542 INFO:tasks.workunit.client.0.smithi167.stdout:9/967: mknod dd/d17/d7f/c143 0 2022-01-31T19:44:23.542 INFO:tasks.workunit.client.0.smithi167.stdout:9/968: creat dd/f144 x:0 0 0 2022-01-31T19:44:23.542 INFO:tasks.workunit.client.0.smithi167.stdout:9/969: write dd/d19/f1b [412306,85887] 0 2022-01-31T19:44:23.542 INFO:tasks.workunit.client.0.smithi167.stdout:9/970: write dd/d17/d7f/dfb/f120 [996609,37078] 0 2022-01-31T19:44:23.544 INFO:tasks.workunit.client.0.smithi167.stdout:6/803: creat d8/d33/d51/d5d/d64/dab/f11a x:0 0 0 2022-01-31T19:44:23.544 INFO:tasks.workunit.client.0.smithi167.stdout:6/804: write d8/d62/d92/fa1 [1054052,8502] 0 2022-01-31T19:44:23.544 INFO:tasks.workunit.client.0.smithi167.stdout:6/805: read d8/d20/d54/fa7 [713883,11741] 0 2022-01-31T19:44:23.545 INFO:tasks.workunit.client.0.smithi167.stdout:6/806: creat d8/d33/d51/d5d/d64/dab/db9/f11b x:0 0 0 2022-01-31T19:44:23.545 INFO:tasks.workunit.client.0.smithi167.stdout:6/807: dread - d8/d20/d54/f10e zero size 2022-01-31T19:44:23.545 INFO:tasks.workunit.client.0.smithi167.stdout:6/808: truncate d8/d33/d51/d5d/d64/fcd 1030186 0 2022-01-31T19:44:23.545 INFO:tasks.workunit.client.0.smithi167.stdout:6/809: rename d8/d62 to d8/d62/d92/d11c 22 2022-01-31T19:44:23.546 INFO:tasks.workunit.client.1.smithi171.stdout:2/369: dread d7/d26/d3d/d40/f51 [0,4194304] 0 2022-01-31T19:44:23.546 INFO:tasks.workunit.client.1.smithi171.stdout:0/298: creat d9/d24/d4c/d35/f74 x:0 0 0 2022-01-31T19:44:23.547 INFO:tasks.workunit.client.0.smithi167.stdout:0/806: sync 2022-01-31T19:44:23.549 INFO:tasks.workunit.client.1.smithi171.stdout:7/230: symlink d1/d1e/d2a/d29/d31/l4e 0 2022-01-31T19:44:23.549 INFO:tasks.workunit.client.1.smithi171.stdout:7/231: dread - d1/d1e/d2a/d29/d31/d30/f3f zero size 2022-01-31T19:44:23.550 INFO:tasks.workunit.client.1.smithi171.stdout:3/293: creat d4/dd/d1a/d52/f54 x:0 0 0 2022-01-31T19:44:23.550 INFO:tasks.workunit.client.1.smithi171.stdout:3/294: dread - d4/dd/d39/d51/f53 zero size 2022-01-31T19:44:23.551 INFO:tasks.workunit.client.0.smithi167.stdout:3/988: truncate d9/d7f/f8c 3171372 0 2022-01-31T19:44:23.552 INFO:tasks.workunit.client.0.smithi167.stdout:2/927: mkdir dd/d18/d62/d75/d11b/d133 0 2022-01-31T19:44:23.554 INFO:tasks.workunit.client.1.smithi171.stdout:8/244: rmdir df/d2a 39 2022-01-31T19:44:23.554 INFO:tasks.workunit.client.1.smithi171.stdout:8/245: truncate df/d2a/d30/f35 1196235 0 2022-01-31T19:44:23.554 INFO:tasks.workunit.client.1.smithi171.stdout:8/246: readlink ld 0 2022-01-31T19:44:23.555 INFO:tasks.workunit.client.1.smithi171.stdout:8/247: stat df/d18/c2d 0 2022-01-31T19:44:23.555 INFO:tasks.workunit.client.0.smithi167.stdout:9/971: creat dd/d19/d24/d56/f145 x:0 0 0 2022-01-31T19:44:23.556 INFO:tasks.workunit.client.0.smithi167.stdout:2/928: read dd/f15 [914963,44972] 0 2022-01-31T19:44:23.556 INFO:tasks.workunit.client.1.smithi171.stdout:2/370: symlink d7/dd/d5e/l6f 0 2022-01-31T19:44:23.557 INFO:tasks.workunit.client.0.smithi167.stdout:6/810: unlink d8/d33/d51/d7e/dc4/fba 0 2022-01-31T19:44:23.558 INFO:tasks.workunit.client.1.smithi171.stdout:0/299: rename d9/db/d22/d3a/f6a to d9/d24/d4c/d35/f75 0 2022-01-31T19:44:23.558 INFO:tasks.workunit.client.1.smithi171.stdout:7/232: dread d1/d24/f1a [0,4194304] 0 2022-01-31T19:44:23.559 INFO:tasks.workunit.client.1.smithi171.stdout:3/295: rmdir d4/dd/d27/d2e 39 2022-01-31T19:44:23.559 INFO:tasks.workunit.client.1.smithi171.stdout:3/296: fdatasync d4/dd/d1a/f1b 0 2022-01-31T19:44:23.559 INFO:tasks.workunit.client.1.smithi171.stdout:3/297: write d4/dd/d1a/d52/f54 [130864,33726] 0 2022-01-31T19:44:23.560 INFO:tasks.workunit.client.1.smithi171.stdout:2/371: rename d7/dd/l30 to d7/d9/d1a/d45/l70 0 2022-01-31T19:44:23.561 INFO:tasks.workunit.client.1.smithi171.stdout:0/300: unlink d9/db/c23 0 2022-01-31T19:44:23.563 INFO:tasks.workunit.client.0.smithi167.stdout:9/972: mkdir dd/d19/d24/d56/d5b/dcf/d13e/d146 0 2022-01-31T19:44:23.563 INFO:tasks.workunit.client.0.smithi167.stdout:9/973: readlink dd/d17/d80/ldd 0 2022-01-31T19:44:23.564 INFO:tasks.workunit.client.1.smithi171.stdout:7/233: creat d1/d26/d47/f4f x:0 0 0 2022-01-31T19:44:23.564 INFO:tasks.workunit.client.1.smithi171.stdout:7/234: write d1/d1e/d2a/d29/d31/d30/f4d [601812,8262] 0 2022-01-31T19:44:23.564 INFO:tasks.workunit.client.1.smithi171.stdout:7/235: fdatasync d1/d24/f1a 0 2022-01-31T19:44:23.565 INFO:tasks.workunit.client.1.smithi171.stdout:3/298: truncate d4/fa 1862385 0 2022-01-31T19:44:23.565 INFO:tasks.workunit.client.1.smithi171.stdout:3/299: fdatasync d4/dd/f20 0 2022-01-31T19:44:23.569 INFO:tasks.workunit.client.1.smithi171.stdout:2/372: mkdir d7/d26/d63/d71 0 2022-01-31T19:44:23.570 INFO:tasks.workunit.client.1.smithi171.stdout:7/236: dread d1/d24/f1a [0,4194304] 0 2022-01-31T19:44:23.571 INFO:tasks.workunit.client.0.smithi167.stdout:2/929: rmdir dd/d18/d8d 39 2022-01-31T19:44:23.571 INFO:tasks.workunit.client.1.smithi171.stdout:3/300: mkdir d4/dd/d27/d28/d35/d55 0 2022-01-31T19:44:23.572 INFO:tasks.workunit.client.0.smithi167.stdout:9/974: dread dd/d17/d7f/def/fea [0,4194304] 0 2022-01-31T19:44:23.572 INFO:tasks.workunit.client.0.smithi167.stdout:9/975: chown dd/de7 3810 1 2022-01-31T19:44:23.572 INFO:tasks.workunit.client.0.smithi167.stdout:9/976: readlink dd/d17/d7f/lcc 0 2022-01-31T19:44:23.574 INFO:tasks.workunit.client.1.smithi171.stdout:2/373: rename d7/dd/l2b to d7/d9/d1a/l72 0 2022-01-31T19:44:23.575 INFO:tasks.workunit.client.0.smithi167.stdout:2/930: unlink dd/df/d1b/d3d/d4a/d4b/d54/f8b 0 2022-01-31T19:44:23.575 INFO:tasks.workunit.client.0.smithi167.stdout:2/931: truncate dd/d18/fa9 5323776 0 2022-01-31T19:44:23.575 INFO:tasks.workunit.client.0.smithi167.stdout:2/932: fsync dd/d18/d63/fa3 0 2022-01-31T19:44:23.575 INFO:tasks.workunit.client.0.smithi167.stdout:2/933: chown dd/d18/d62/d75/d105/f65 48 1 2022-01-31T19:44:23.576 INFO:tasks.workunit.client.0.smithi167.stdout:2/934: chown dd/df/d1b/d3d/d4a/da7/dcc/f107 24411648 1 2022-01-31T19:44:23.576 INFO:tasks.workunit.client.0.smithi167.stdout:9/977: creat dd/d19/d24/d50/f147 x:0 0 0 2022-01-31T19:44:23.577 INFO:tasks.workunit.client.1.smithi171.stdout:7/237: unlink d1/c3 0 2022-01-31T19:44:23.577 INFO:tasks.workunit.client.1.smithi171.stdout:2/374: symlink d7/dd/d5e/d6b/l73 0 2022-01-31T19:44:23.585 INFO:tasks.workunit.client.1.smithi171.stdout:8/248: dwrite f6 [0,4194304] 0 2022-01-31T19:44:23.589 INFO:tasks.workunit.client.1.smithi171.stdout:8/249: write df/d18/f1c [2359671,72358] 0 2022-01-31T19:44:23.610 INFO:tasks.workunit.client.1.smithi171.stdout:3/301: dwrite d4/fc [0,4194304] 0 2022-01-31T19:44:23.615 INFO:tasks.workunit.client.0.smithi167.stdout:6/811: dwrite d8/d33/d51/d5d/f80 [0,4194304] 0 2022-01-31T19:44:23.615 INFO:tasks.workunit.client.0.smithi167.stdout:6/812: creat d8/d33/d51/d7e/dc4/f11d x:0 0 0 2022-01-31T19:44:23.616 INFO:tasks.workunit.client.0.smithi167.stdout:6/813: write d8/d33/d51/d5d/d86/fcc [349071,1516] 0 2022-01-31T19:44:23.617 INFO:tasks.workunit.client.1.smithi171.stdout:3/302: link d4/d21/d3a/l4c d4/dd/d40/l56 0 2022-01-31T19:44:23.617 INFO:tasks.workunit.client.1.smithi171.stdout:3/303: readlink d4/d21/d3a/l4c 0 2022-01-31T19:44:23.617 INFO:tasks.workunit.client.1.smithi171.stdout:3/304: chown d4/dd/c14 155420685 1 2022-01-31T19:44:23.618 INFO:tasks.workunit.client.0.smithi167.stdout:9/978: dread dd/d17/f18 [4194304,4194304] 0 2022-01-31T19:44:23.620 INFO:tasks.workunit.client.0.smithi167.stdout:9/979: mkdir dd/d19/d24/d50/d5d/d148 0 2022-01-31T19:44:23.623 INFO:tasks.workunit.client.0.smithi167.stdout:9/980: creat dd/d19/d24/d50/d5d/d148/f149 x:0 0 0 2022-01-31T19:44:23.625 INFO:tasks.workunit.client.0.smithi167.stdout:9/981: symlink dd/de7/l14a 0 2022-01-31T19:44:23.625 INFO:tasks.workunit.client.0.smithi167.stdout:9/982: write dd/d34/f92 [1809295,130845] 0 2022-01-31T19:44:23.626 INFO:tasks.workunit.client.1.smithi171.stdout:0/301: dwrite d9/f1f [0,4194304] 0 2022-01-31T19:44:23.627 INFO:tasks.workunit.client.1.smithi171.stdout:7/238: dwrite d1/d1b/f3a [0,4194304] 0 2022-01-31T19:44:23.628 INFO:tasks.workunit.client.0.smithi167.stdout:9/983: rename dd/d17/f1f to dd/d19/d24/d50/d5d/f14b 0 2022-01-31T19:44:23.630 INFO:tasks.workunit.client.1.smithi171.stdout:3/305: dread d4/d21/f2a [0,4194304] 0 2022-01-31T19:44:23.631 INFO:tasks.workunit.client.1.smithi171.stdout:2/375: dwrite d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:23.632 INFO:tasks.workunit.client.1.smithi171.stdout:2/376: chown d7/d9/d1a/d45/l70 188538 1 2022-01-31T19:44:23.632 INFO:tasks.workunit.client.1.smithi171.stdout:2/377: chown d7/d9/f44 0 1 2022-01-31T19:44:23.632 INFO:tasks.workunit.client.1.smithi171.stdout:2/378: write d7/f3a [1467249,63053] 0 2022-01-31T19:44:23.633 INFO:tasks.workunit.client.1.smithi171.stdout:0/302: unlink d9/db/dc/d3f/d25/l65 0 2022-01-31T19:44:23.634 INFO:tasks.workunit.client.1.smithi171.stdout:7/239: creat d1/d26/f50 x:0 0 0 2022-01-31T19:44:23.635 INFO:tasks.workunit.client.1.smithi171.stdout:2/379: write d7/d9/f44 [276923,120047] 0 2022-01-31T19:44:23.636 INFO:tasks.workunit.client.1.smithi171.stdout:2/380: chown d7/d26/d3d/f66 99343 1 2022-01-31T19:44:23.636 INFO:tasks.workunit.client.1.smithi171.stdout:2/381: dread - d7/dd/f52 zero size 2022-01-31T19:44:23.638 INFO:tasks.workunit.client.1.smithi171.stdout:3/306: link d4/d21/d3a/f4f d4/dd/d27/d28/f57 0 2022-01-31T19:44:23.639 INFO:tasks.workunit.client.1.smithi171.stdout:0/303: symlink d9/db/dc/d3f/d25/l76 0 2022-01-31T19:44:23.639 INFO:tasks.workunit.client.0.smithi167.stdout:6/814: dwrite d8/d33/d51/d72/f75 [0,4194304] 0 2022-01-31T19:44:23.640 INFO:tasks.workunit.client.1.smithi171.stdout:8/250: dwrite df/d1a/f58 [0,4194304] 0 2022-01-31T19:44:23.640 INFO:tasks.workunit.client.0.smithi167.stdout:6/815: write d8/d62/f77 [3378818,37713] 0 2022-01-31T19:44:23.647 INFO:tasks.workunit.client.0.smithi167.stdout:9/984: rename dd/d19/d24 to dd/d19/d59/d14c 0 2022-01-31T19:44:23.647 INFO:tasks.workunit.client.0.smithi167.stdout:9/985: stat dd/d17/d80/ldd 0 2022-01-31T19:44:23.651 INFO:tasks.workunit.client.0.smithi167.stdout:9/986: write dd/d19/d59/d14c/d50/dd1/f8a [3782842,23506] 0 2022-01-31T19:44:23.652 INFO:tasks.workunit.client.0.smithi167.stdout:9/987: truncate dd/d19/d59/d14c/d50/d5d/f78 85139 0 2022-01-31T19:44:23.653 INFO:tasks.workunit.client.0.smithi167.stdout:9/988: rmdir dd/d19/d59/df9/d121 39 2022-01-31T19:44:23.654 INFO:tasks.workunit.client.0.smithi167.stdout:9/989: mkdir dd/d19/d59/d14c/d56/d14d 0 2022-01-31T19:44:23.654 INFO:tasks.workunit.client.1.smithi171.stdout:8/251: dread df/d2a/d38/f43 [0,4194304] 0 2022-01-31T19:44:23.655 INFO:tasks.workunit.client.0.smithi167.stdout:9/990: write dd/d35/f8d [2147443,82635] 0 2022-01-31T19:44:23.656 INFO:tasks.workunit.client.1.smithi171.stdout:7/240: rmdir d1/d1e 39 2022-01-31T19:44:23.656 INFO:tasks.workunit.client.1.smithi171.stdout:7/241: stat d1/d26/d47/f4f 0 2022-01-31T19:44:23.657 INFO:tasks.workunit.client.0.smithi167.stdout:9/991: link dd/d19/d59/d14c/d56/d5b/c142 dd/d19/d59/d14c/d50/c14e 0 2022-01-31T19:44:23.657 INFO:tasks.workunit.client.0.smithi167.stdout:9/992: creat dd/d35/d105/f14f x:0 0 0 2022-01-31T19:44:23.659 INFO:tasks.workunit.client.1.smithi171.stdout:7/242: write d1/d1e/d2a/d29/d31/fe [3843886,69266] 0 2022-01-31T19:44:23.659 INFO:tasks.workunit.client.1.smithi171.stdout:2/382: rename d7/d55/f6d to d7/d9/f74 0 2022-01-31T19:44:23.660 INFO:tasks.workunit.client.1.smithi171.stdout:3/307: unlink d4/f1d 0 2022-01-31T19:44:23.663 INFO:tasks.workunit.client.1.smithi171.stdout:0/304: rename d9/d24/d4c/d35/f75 to d9/d24/d4c/f77 0 2022-01-31T19:44:23.666 INFO:tasks.workunit.client.0.smithi167.stdout:6/816: dwrite d8/d33/f90 [0,4194304] 0 2022-01-31T19:44:23.669 INFO:tasks.workunit.client.0.smithi167.stdout:6/817: link d8/d33/d51/d5d/d64/dab/fd7 d8/d33/d4f/f11e 0 2022-01-31T19:44:23.670 INFO:tasks.workunit.client.0.smithi167.stdout:6/818: unlink d8/d33/d51/d5d/fc3 0 2022-01-31T19:44:23.671 INFO:tasks.workunit.client.0.smithi167.stdout:6/819: read d8/d33/d51/d5d/d64/dab/fd7 [427402,83984] 0 2022-01-31T19:44:23.672 INFO:tasks.workunit.client.1.smithi171.stdout:3/308: dread d4/dd/d1a/f1b [0,4194304] 0 2022-01-31T19:44:23.673 INFO:tasks.workunit.client.1.smithi171.stdout:0/305: write d9/f1e [4104545,123545] 0 2022-01-31T19:44:23.673 INFO:tasks.workunit.client.1.smithi171.stdout:0/306: truncate d9/db/d22/d5d/f5f 379334 0 2022-01-31T19:44:23.674 INFO:tasks.workunit.client.1.smithi171.stdout:0/307: creat d9/d24/d4c/f78 x:0 0 0 2022-01-31T19:44:23.674 INFO:tasks.workunit.client.1.smithi171.stdout:3/309: read d4/dd/d40/f42 [434445,124769] 0 2022-01-31T19:44:23.677 INFO:tasks.workunit.client.1.smithi171.stdout:2/383: dread d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:23.678 INFO:tasks.workunit.client.1.smithi171.stdout:0/308: creat d9/db/d22/f79 x:0 0 0 2022-01-31T19:44:23.680 INFO:tasks.workunit.client.0.smithi167.stdout:9/993: dwrite dd/d35/ff8 [0,4194304] 0 2022-01-31T19:44:23.680 INFO:tasks.workunit.client.1.smithi171.stdout:3/310: mkdir d4/dd/d27/d28/d35/d58 0 2022-01-31T19:44:23.682 INFO:tasks.workunit.client.1.smithi171.stdout:0/309: mkdir d9/db/d22/d3a/d7a 0 2022-01-31T19:44:23.682 INFO:tasks.workunit.client.1.smithi171.stdout:0/310: write d9/d24/f3c [4435777,34169] 0 2022-01-31T19:44:23.683 INFO:tasks.workunit.client.1.smithi171.stdout:0/311: write d9/db/d22/d3a/f48 [391078,98338] 0 2022-01-31T19:44:23.683 INFO:tasks.workunit.client.1.smithi171.stdout:0/312: dread - d9/db/dc/f70 zero size 2022-01-31T19:44:23.684 INFO:tasks.workunit.client.1.smithi171.stdout:3/311: mkdir d4/d21/d3a/d59 0 2022-01-31T19:44:23.686 INFO:tasks.workunit.client.1.smithi171.stdout:0/313: symlink d9/d24/d4c/d35/d67/l7b 0 2022-01-31T19:44:23.686 INFO:tasks.workunit.client.1.smithi171.stdout:0/314: chown d9/f53 4017981 1 2022-01-31T19:44:23.687 INFO:tasks.workunit.client.1.smithi171.stdout:3/312: mknod d4/dd/d1a/c5a 0 2022-01-31T19:44:23.688 INFO:tasks.workunit.client.0.smithi167.stdout:6/820: rmdir d8/d33/d51 39 2022-01-31T19:44:23.689 INFO:tasks.workunit.client.1.smithi171.stdout:0/315: link d9/db/dc/l44 d9/db/dc/d3f/l7c 0 2022-01-31T19:44:23.694 INFO:tasks.workunit.client.1.smithi171.stdout:0/316: unlink d9/db/d22/d3a/f48 0 2022-01-31T19:44:23.694 INFO:tasks.workunit.client.1.smithi171.stdout:8/252: dwrite df/f51 [0,4194304] 0 2022-01-31T19:44:23.695 INFO:tasks.workunit.client.1.smithi171.stdout:0/317: chown d9/db/d22/d3a/c5a 2015 1 2022-01-31T19:44:23.697 INFO:tasks.workunit.client.1.smithi171.stdout:8/253: getdents df/d18/d19/d21 0 2022-01-31T19:44:23.698 INFO:tasks.workunit.client.1.smithi171.stdout:8/254: mknod df/d1a/c5a 0 2022-01-31T19:44:23.698 INFO:tasks.workunit.client.1.smithi171.stdout:8/255: creat df/d27/f5b x:0 0 0 2022-01-31T19:44:23.700 INFO:tasks.workunit.client.1.smithi171.stdout:2/384: dwrite d7/d26/d3d/f66 [4194304,4194304] 0 2022-01-31T19:44:23.705 INFO:tasks.workunit.client.1.smithi171.stdout:2/385: read d7/f13 [1823776,89170] 0 2022-01-31T19:44:23.705 INFO:tasks.workunit.client.1.smithi171.stdout:2/386: fdatasync d7/d26/d3d/d40/f4f 0 2022-01-31T19:44:23.706 INFO:tasks.workunit.client.1.smithi171.stdout:2/387: creat d7/f75 x:0 0 0 2022-01-31T19:44:23.709 INFO:tasks.workunit.client.1.smithi171.stdout:2/388: unlink d7/d26/d3d/f66 0 2022-01-31T19:44:23.709 INFO:tasks.workunit.client.1.smithi171.stdout:2/389: creat d7/d26/d63/f76 x:0 0 0 2022-01-31T19:44:23.710 INFO:tasks.workunit.client.1.smithi171.stdout:2/390: symlink d7/d26/d3d/l77 0 2022-01-31T19:44:23.711 INFO:tasks.workunit.client.1.smithi171.stdout:2/391: rmdir d7/d9/d68 0 2022-01-31T19:44:23.724 INFO:tasks.workunit.client.0.smithi167.stdout:9/994: dwrite dd/d19/d59/d14c/d50/d5d/fbc [0,4194304] 0 2022-01-31T19:44:23.734 INFO:tasks.workunit.client.0.smithi167.stdout:6/821: dwrite d8/d33/d4f/dd1/fec [0,4194304] 0 2022-01-31T19:44:23.734 INFO:tasks.workunit.client.1.smithi171.stdout:8/256: dwrite df/f17 [0,4194304] 0 2022-01-31T19:44:23.735 INFO:tasks.workunit.client.1.smithi171.stdout:3/313: dwrite d4/dd/d27/f3f [0,4194304] 0 2022-01-31T19:44:23.742 INFO:tasks.workunit.client.1.smithi171.stdout:2/392: dwrite d7/f13 [0,4194304] 0 2022-01-31T19:44:23.743 INFO:tasks.workunit.client.1.smithi171.stdout:2/393: creat d7/d26/d3d/d40/f78 x:0 0 0 2022-01-31T19:44:23.743 INFO:tasks.workunit.client.1.smithi171.stdout:2/394: write d7/f11 [1980353,19977] 0 2022-01-31T19:44:23.743 INFO:tasks.workunit.client.1.smithi171.stdout:2/395: write d7/dd/f1e [1476523,39] 0 2022-01-31T19:44:23.743 INFO:tasks.workunit.client.1.smithi171.stdout:2/396: dread - d7/dd/f4c zero size 2022-01-31T19:44:23.746 INFO:tasks.workunit.client.1.smithi171.stdout:3/314: symlink d4/d21/d3a/l5b 0 2022-01-31T19:44:23.756 INFO:tasks.workunit.client.1.smithi171.stdout:2/397: dread d7/d26/f31 [0,4194304] 0 2022-01-31T19:44:23.763 INFO:tasks.workunit.client.1.smithi171.stdout:0/318: dwrite d9/d24/f3c [0,4194304] 0 2022-01-31T19:44:23.764 INFO:tasks.workunit.client.1.smithi171.stdout:0/319: mkdir d9/d24/d4c/d7d 0 2022-01-31T19:44:23.764 INFO:tasks.workunit.client.1.smithi171.stdout:0/320: chown d9/d24/d4c/d35/d67 28484072 1 2022-01-31T19:44:23.765 INFO:tasks.workunit.client.1.smithi171.stdout:2/398: dread d7/f13 [4194304,4194304] 0 2022-01-31T19:44:23.766 INFO:tasks.workunit.client.1.smithi171.stdout:0/321: rmdir d9/db/d22/d3a 39 2022-01-31T19:44:23.766 INFO:tasks.workunit.client.0.smithi167.stdout:9/995: dwrite dd/d17/d7f/def/fff [0,4194304] 0 2022-01-31T19:44:23.767 INFO:tasks.workunit.client.0.smithi167.stdout:6/822: dwrite d8/d33/d51/d72/fd6 [0,4194304] 0 2022-01-31T19:44:23.767 INFO:tasks.workunit.client.1.smithi171.stdout:8/257: dwrite df/d18/d19/d21/f32 [0,4194304] 0 2022-01-31T19:44:23.769 INFO:tasks.workunit.client.0.smithi167.stdout:9/996: creat dd/d34/d123/f150 x:0 0 0 2022-01-31T19:44:23.770 INFO:tasks.workunit.client.1.smithi171.stdout:3/315: dwrite d4/dd/d27/d28/d35/f38 [0,4194304] 0 2022-01-31T19:44:23.770 INFO:tasks.workunit.client.1.smithi171.stdout:3/316: write d4/dd/d27/d28/d35/f48 [356033,41233] 0 2022-01-31T19:44:23.772 INFO:tasks.workunit.client.0.smithi167.stdout:9/997: getdents dd/d17/d7f/dfb 0 2022-01-31T19:44:23.772 INFO:tasks.workunit.client.0.smithi167.stdout:9/998: dread - dd/d19/d59/d14c/d50/dd1/f7b zero size 2022-01-31T19:44:23.773 INFO:tasks.workunit.client.0.smithi167.stdout:9/999: mknod dd/d19/d59/d14c/d50/dd6/c151 0 2022-01-31T19:44:23.776 INFO:tasks.workunit.client.1.smithi171.stdout:8/258: mkdir df/d27/d5c 0 2022-01-31T19:44:23.776 INFO:tasks.workunit.client.1.smithi171.stdout:8/259: write df/f4f [1473662,65036] 0 2022-01-31T19:44:23.777 INFO:tasks.workunit.client.1.smithi171.stdout:8/260: fdatasync df/f51 0 2022-01-31T19:44:23.783 INFO:tasks.workunit.client.1.smithi171.stdout:8/261: fsync df/f4f 0 2022-01-31T19:44:23.786 INFO:tasks.workunit.client.0.smithi167.stdout:6/823: dread d8/d20/f22 [0,4194304] 0 2022-01-31T19:44:23.788 INFO:tasks.workunit.client.1.smithi171.stdout:8/262: mkdir df/d5d 0 2022-01-31T19:44:23.789 INFO:tasks.workunit.client.0.smithi167.stdout:6/824: rename d8/d20/d54/led to d8/d20/d103/d108/l11f 0 2022-01-31T19:44:23.789 INFO:tasks.workunit.client.0.smithi167.stdout:6/825: fdatasync d8/d62/f6f 0 2022-01-31T19:44:23.790 INFO:tasks.workunit.client.0.smithi167.stdout:6/826: read d8/f7a [2891524,13160] 0 2022-01-31T19:44:23.790 INFO:tasks.workunit.client.1.smithi171.stdout:8/263: rename f8 to df/d18/d19/d46/f5e 0 2022-01-31T19:44:23.791 INFO:tasks.workunit.client.0.smithi167.stdout:6/827: unlink d8/d33/d51/d6a/ffa 0 2022-01-31T19:44:23.792 INFO:tasks.workunit.client.0.smithi167.stdout:6/828: truncate d8/d62/f6f 4431577 0 2022-01-31T19:44:23.792 INFO:tasks.workunit.client.1.smithi171.stdout:8/264: creat df/d18/d19/f5f x:0 0 0 2022-01-31T19:44:23.793 INFO:tasks.workunit.client.0.smithi167.stdout:6/829: creat d8/d20/f120 x:0 0 0 2022-01-31T19:44:23.795 INFO:tasks.workunit.client.1.smithi171.stdout:8/265: read f6 [4072118,69718] 0 2022-01-31T19:44:23.796 INFO:tasks.workunit.client.0.smithi167.stdout:6/830: rename d8/d33/d51/d7e/dc4/f11d to d8/d62/d92/f121 0 2022-01-31T19:44:23.796 INFO:tasks.workunit.client.0.smithi167.stdout:6/831: symlink d8/d33/d51/d6a/df1/d106/l122 0 2022-01-31T19:44:23.800 INFO:tasks.workunit.client.1.smithi171.stdout:3/317: dwrite d4/dd/d1a/d52/f54 [0,4194304] 0 2022-01-31T19:44:23.802 INFO:tasks.workunit.client.1.smithi171.stdout:2/399: dwrite d7/f22 [0,4194304] 0 2022-01-31T19:44:23.803 INFO:tasks.workunit.client.1.smithi171.stdout:0/322: dwrite f4 [4194304,4194304] 0 2022-01-31T19:44:23.808 INFO:tasks.workunit.client.0.smithi167.stdout:3/989: sync 2022-01-31T19:44:23.808 INFO:tasks.workunit.client.0.smithi167.stdout:0/807: sync 2022-01-31T19:44:23.809 INFO:tasks.workunit.client.0.smithi167.stdout:2/935: sync 2022-01-31T19:44:23.811 INFO:tasks.workunit.client.0.smithi167.stdout:0/808: unlink d17/d19/d1c/d39/d7b/lbe 0 2022-01-31T19:44:23.813 INFO:tasks.workunit.client.1.smithi171.stdout:0/323: symlink d9/db/d22/d3a/l7e 0 2022-01-31T19:44:23.813 INFO:tasks.workunit.client.1.smithi171.stdout:0/324: write d9/db/dc/d3f/d25/f29 [417647,113462] 0 2022-01-31T19:44:23.814 INFO:tasks.workunit.client.1.smithi171.stdout:3/318: write d4/dd/d27/f3f [1066676,46490] 0 2022-01-31T19:44:23.814 INFO:tasks.workunit.client.1.smithi171.stdout:3/319: truncate d4/dd/d27/f3f 4216670 0 2022-01-31T19:44:23.815 INFO:tasks.workunit.client.0.smithi167.stdout:3/990: creat d9/d56/d128/f12e x:0 0 0 2022-01-31T19:44:23.815 INFO:tasks.workunit.client.0.smithi167.stdout:3/991: write d9/d29/d3b/d41/d6b/f117 [990145,51531] 0 2022-01-31T19:44:23.816 INFO:tasks.workunit.client.0.smithi167.stdout:0/809: rename d17/d2c/l41 to d17/d19/d1c/d39/d44/d88/ddf/l10c 0 2022-01-31T19:44:23.816 INFO:tasks.workunit.client.0.smithi167.stdout:0/810: read d17/d19/d1c/d39/d7b/f96 [256865,102449] 0 2022-01-31T19:44:23.816 INFO:tasks.workunit.client.0.smithi167.stdout:0/811: creat d17/d69/f10d x:0 0 0 2022-01-31T19:44:23.817 INFO:tasks.workunit.client.1.smithi171.stdout:0/325: symlink d9/d24/l7f 0 2022-01-31T19:44:23.817 INFO:tasks.workunit.client.1.smithi171.stdout:3/320: unlink d4/dd/d27/d28/f2d 0 2022-01-31T19:44:23.817 INFO:tasks.workunit.client.1.smithi171.stdout:3/321: creat d4/dd/d27/d28/d35/f5c x:0 0 0 2022-01-31T19:44:23.817 INFO:tasks.workunit.client.1.smithi171.stdout:3/322: fsync d4/dd/d39/f50 0 2022-01-31T19:44:23.819 INFO:tasks.workunit.client.1.smithi171.stdout:0/326: symlink d9/d24/d4c/d35/l80 0 2022-01-31T19:44:23.819 INFO:tasks.workunit.client.1.smithi171.stdout:0/327: write d9/db/dc/d3f/f2d [8462951,98252] 0 2022-01-31T19:44:23.820 INFO:tasks.workunit.client.0.smithi167.stdout:0/812: getdents d17/d19/d1c/d39/d7b/de5 0 2022-01-31T19:44:23.820 INFO:tasks.workunit.client.0.smithi167.stdout:0/813: chown d17/d19/d1c/d39/c3b 119637904 1 2022-01-31T19:44:23.820 INFO:tasks.workunit.client.0.smithi167.stdout:3/992: rename d9/d29/d3b/d8d/le5 to d9/d29/d3b/d41/l12f 0 2022-01-31T19:44:23.821 INFO:tasks.workunit.client.0.smithi167.stdout:0/814: creat d17/d2c/dcd/f10e x:0 0 0 2022-01-31T19:44:23.822 INFO:tasks.workunit.client.0.smithi167.stdout:0/815: readlink d17/l25 0 2022-01-31T19:44:23.822 INFO:tasks.workunit.client.0.smithi167.stdout:3/993: symlink d9/d29/d3b/d41/d6b/d11a/l130 0 2022-01-31T19:44:23.824 INFO:tasks.workunit.client.0.smithi167.stdout:3/994: unlink d9/d7f/d10a/c120 0 2022-01-31T19:44:23.825 INFO:tasks.workunit.client.0.smithi167.stdout:3/995: fdatasync d9/d29/d60/df8/f111 0 2022-01-31T19:44:23.826 INFO:tasks.workunit.client.0.smithi167.stdout:3/996: fsync d9/d29/d3b/fee 0 2022-01-31T19:44:23.827 INFO:tasks.workunit.client.1.smithi171.stdout:8/266: dwrite df/d18/d19/d21/f32 [0,4194304] 0 2022-01-31T19:44:23.830 INFO:tasks.workunit.client.1.smithi171.stdout:8/267: link df/d2a/c2f df/d18/d19/d46/c60 0 2022-01-31T19:44:23.831 INFO:tasks.workunit.client.1.smithi171.stdout:8/268: rmdir df/d2a 39 2022-01-31T19:44:23.831 INFO:tasks.workunit.client.1.smithi171.stdout:8/269: truncate df/d18/d19/f22 524685 0 2022-01-31T19:44:23.832 INFO:tasks.workunit.client.1.smithi171.stdout:8/270: creat df/d18/d19/f61 x:0 0 0 2022-01-31T19:44:23.836 INFO:tasks.workunit.client.1.smithi171.stdout:8/271: rename df/d2a/l4c to df/d2a/d38/l62 0 2022-01-31T19:44:23.837 INFO:tasks.workunit.client.1.smithi171.stdout:2/400: dwrite d7/d26/d3d/d40/f4f [0,4194304] 0 2022-01-31T19:44:23.838 INFO:tasks.workunit.client.1.smithi171.stdout:8/272: unlink df/d2a/l2b 0 2022-01-31T19:44:23.840 INFO:tasks.workunit.client.1.smithi171.stdout:2/401: getdents d7/d26/d3d 0 2022-01-31T19:44:23.844 INFO:tasks.workunit.client.1.smithi171.stdout:8/273: write fb [2372093,26248] 0 2022-01-31T19:44:23.844 INFO:tasks.workunit.client.1.smithi171.stdout:2/402: symlink d7/dd/l79 0 2022-01-31T19:44:23.849 INFO:tasks.workunit.client.1.smithi171.stdout:2/403: unlink d7/d26/d3d/l77 0 2022-01-31T19:44:23.854 INFO:tasks.workunit.client.0.smithi167.stdout:6/832: dwrite d8/d33/d51/d72/fd6 [4194304,4194304] 0 2022-01-31T19:44:23.855 INFO:tasks.workunit.client.0.smithi167.stdout:2/936: dwrite dd/df/d1b/d2f/d36/f2a [0,4194304] 0 2022-01-31T19:44:23.858 INFO:tasks.workunit.client.1.smithi171.stdout:0/328: dwrite d9/db/f21 [0,4194304] 0 2022-01-31T19:44:23.861 INFO:tasks.workunit.client.1.smithi171.stdout:0/329: unlink d9/db/d22/d3a/l56 0 2022-01-31T19:44:23.862 INFO:tasks.workunit.client.1.smithi171.stdout:0/330: truncate d9/db/dc/d3f/f6d 3609091 0 2022-01-31T19:44:23.862 INFO:tasks.workunit.client.1.smithi171.stdout:0/331: readlink d9/db/dc/d3f/d25/l36 0 2022-01-31T19:44:23.863 INFO:tasks.workunit.client.1.smithi171.stdout:0/332: write d9/f62 [184367,85394] 0 2022-01-31T19:44:23.863 INFO:tasks.workunit.client.0.smithi167.stdout:6/833: dread d8/d33/f32 [0,4194304] 0 2022-01-31T19:44:23.864 INFO:tasks.workunit.client.0.smithi167.stdout:3/997: dwrite d9/fd3 [0,4194304] 0 2022-01-31T19:44:23.864 INFO:tasks.workunit.client.1.smithi171.stdout:2/404: dread d7/f22 [0,4194304] 0 2022-01-31T19:44:23.865 INFO:tasks.workunit.client.0.smithi167.stdout:3/998: dread d9/d56/fab [0,4194304] 0 2022-01-31T19:44:23.865 INFO:tasks.workunit.client.0.smithi167.stdout:0/816: dwrite d17/d3e/fae [0,4194304] 0 2022-01-31T19:44:23.866 INFO:tasks.workunit.client.1.smithi171.stdout:2/405: creat d7/dd/d5e/d6b/f7a x:0 0 0 2022-01-31T19:44:23.866 INFO:tasks.workunit.client.1.smithi171.stdout:2/406: creat d7/f7b x:0 0 0 2022-01-31T19:44:23.866 INFO:tasks.workunit.client.1.smithi171.stdout:8/274: dwrite df/d18/d19/f5f [0,4194304] 0 2022-01-31T19:44:23.869 INFO:tasks.workunit.client.1.smithi171.stdout:2/407: unlink d7/d9/d1a/d45/l70 0 2022-01-31T19:44:23.870 INFO:tasks.workunit.client.1.smithi171.stdout:2/408: chown d7/d55/f65 0 1 2022-01-31T19:44:23.870 INFO:tasks.workunit.client.1.smithi171.stdout:3/323: rmdir d4/dd/d27/d28 39 2022-01-31T19:44:23.871 INFO:tasks.workunit.client.0.smithi167.stdout:0/817: creat d17/d19/d1c/d39/d44/d88/d98/de8/f10f x:0 0 0 2022-01-31T19:44:23.875 INFO:tasks.workunit.client.0.smithi167.stdout:6/834: dread d8/d33/d51/d72/fd6 [0,4194304] 0 2022-01-31T19:44:23.875 INFO:tasks.workunit.client.0.smithi167.stdout:6/835: truncate d8/d20/d54/f10e 478630 0 2022-01-31T19:44:23.875 INFO:tasks.workunit.client.0.smithi167.stdout:6/836: chown d8/d33/d51/d5d/d86/lc0 1448804 1 2022-01-31T19:44:23.876 INFO:tasks.workunit.client.0.smithi167.stdout:6/837: write d8/d20/d54/f63 [4350145,98727] 0 2022-01-31T19:44:23.876 INFO:tasks.workunit.client.0.smithi167.stdout:6/838: chown d8/d33/d4f/f11e 130971144 1 2022-01-31T19:44:23.877 INFO:tasks.workunit.client.1.smithi171.stdout:8/275: creat df/d2a/d30/f63 x:0 0 0 2022-01-31T19:44:23.877 INFO:tasks.workunit.client.1.smithi171.stdout:2/409: mkdir d7/d7c 0 2022-01-31T19:44:23.877 INFO:tasks.workunit.client.0.smithi167.stdout:0/818: creat d17/d19/d1c/de1/f110 x:0 0 0 2022-01-31T19:44:23.878 INFO:tasks.workunit.client.0.smithi167.stdout:0/819: write d17/d69/fd1 [875828,44127] 0 2022-01-31T19:44:23.880 INFO:tasks.workunit.client.0.smithi167.stdout:6/839: rename d8/d33/d51/d72/dc2/fce to d8/d20/d103/d108/f123 0 2022-01-31T19:44:23.880 INFO:tasks.workunit.client.0.smithi167.stdout:6/840: truncate d8/d62/fe1 1668673 0 2022-01-31T19:44:23.880 INFO:tasks.workunit.client.0.smithi167.stdout:6/841: readlink d8/d33/d4f/l70 0 2022-01-31T19:44:23.881 INFO:tasks.workunit.client.0.smithi167.stdout:0/820: link d17/d3e/c67 d17/d2c/dcd/c111 0 2022-01-31T19:44:23.882 INFO:tasks.workunit.client.0.smithi167.stdout:6/842: mknod d8/d33/d4f/dd1/c124 0 2022-01-31T19:44:23.883 INFO:tasks.workunit.client.1.smithi171.stdout:2/410: mknod d7/d7c/c7d 0 2022-01-31T19:44:23.884 INFO:tasks.workunit.client.0.smithi167.stdout:6/843: truncate d8/d33/d35/f8b 3398156 0 2022-01-31T19:44:23.884 INFO:tasks.workunit.client.1.smithi171.stdout:3/324: mknod d4/c5d 0 2022-01-31T19:44:23.886 INFO:tasks.workunit.client.1.smithi171.stdout:3/325: unlink d4/d21/d3a/l4c 0 2022-01-31T19:44:23.887 INFO:tasks.workunit.client.1.smithi171.stdout:3/326: creat d4/dd/d27/d28/d35/d55/f5e x:0 0 0 2022-01-31T19:44:23.889 INFO:tasks.workunit.client.1.smithi171.stdout:3/327: mknod d4/dd/d27/d2e/c5f 0 2022-01-31T19:44:23.906 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: cluster 2022-01-31T19:44:22.434590+0000 mgr.smithi171.asyxnc (mgr.24387) 23 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 2.2 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 121 MiB/s rd, 237 MiB/s wr, 344 op/s 2022-01-31T19:44:23.907 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.956801+0000 mon.smithi167 (mon.0) 667 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:23.907 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.963828+0000 mon.smithi171 (mon.1) 53 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.907 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.964374+0000 mon.smithi167 (mon.0) 668 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.908 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.965833+0000 mon.smithi171 (mon.1) 54 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.908 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.966205+0000 mon.smithi167 (mon.0) 669 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.908 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.967416+0000 mon.smithi171 (mon.1) 55 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.908 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.967648+0000 mon.smithi167 (mon.0) 670 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.909 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:22.975831+0000 mon.smithi167 (mon.0) 671 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:23.909 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.331606+0000 mon.smithi167 (mon.0) 672 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:23.909 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.335379+0000 mon.smithi171 (mon.1) 56 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.909 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.335607+0000 mon.smithi167 (mon.0) 673 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.909 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44 2022-01-31T19:44:23.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: :23.336455+0000 mon.smithi171 (mon.1) 57 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.336657+0000 mon.smithi167 (mon.0) 674 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.337455+0000 mon.smithi171 (mon.1) 58 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.337636+0000 mon.smithi167 (mon.0) 675 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:23.911 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.342308+0000 mon.smithi167 (mon.0) 676 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:23.911 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:23 smithi171 conmon[35325]: audit 2022-01-31T19:44:23.577842+0000 mon.smithi167 (mon.0) 677 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:23.912 INFO:tasks.workunit.client.1.smithi171.stdout:4/280: sync 2022-01-31T19:44:23.912 INFO:tasks.workunit.client.1.smithi171.stdout:4/281: fsync d2/d3/d5/d11/d12/d15/f24 0 2022-01-31T19:44:23.913 INFO:tasks.workunit.client.1.smithi171.stdout:4/282: creat d2/d3/d5/f59 x:0 0 0 2022-01-31T19:44:23.913 INFO:tasks.workunit.client.1.smithi171.stdout:4/283: chown d2/d3/d5/d11/d12/d15/l4e 25533963 1 2022-01-31T19:44:23.913 INFO:tasks.workunit.client.1.smithi171.stdout:4/284: creat d2/d3/d5/d11/d12/f5a x:0 0 0 2022-01-31T19:44:23.915 INFO:tasks.workunit.client.1.smithi171.stdout:8/276: dwrite df/f20 [0,4194304] 0 2022-01-31T19:44:23.915 INFO:tasks.workunit.client.1.smithi171.stdout:8/277: chown f7 88791520 1 2022-01-31T19:44:23.915 INFO:tasks.workunit.client.1.smithi171.stdout:8/278: write df/d18/f1e [2573791,69438] 0 2022-01-31T19:44:23.915 INFO:tasks.workunit.client.1.smithi171.stdout:8/279: write df/d1a/f29 [1026156,118640] 0 2022-01-31T19:44:23.916 INFO:tasks.workunit.client.1.smithi171.stdout:8/280: chown df/d2a/d30/c33 0 1 2022-01-31T19:44:23.916 INFO:tasks.workunit.client.1.smithi171.stdout:8/281: rename f5 to df/d27/f64 0 2022-01-31T19:44:23.919 INFO:tasks.workunit.client.0.smithi167.stdout:0/821: dwrite d17/d19/f100 [0,4194304] 0 2022-01-31T19:44:23.920 INFO:tasks.workunit.client.0.smithi167.stdout:0/822: dread d17/d19/d1c/d39/d7b/dbd/ff7 [0,4194304] 0 2022-01-31T19:44:23.921 INFO:tasks.workunit.client.0.smithi167.stdout:0/823: mkdir d17/d33/d112 0 2022-01-31T19:44:23.923 INFO:tasks.workunit.client.1.smithi171.stdout:2/411: dwrite d7/d26/d63/f76 [0,4194304] 0 2022-01-31T19:44:23.924 INFO:tasks.workunit.client.1.smithi171.stdout:3/328: dwrite d4/fb [0,4194304] 0 2022-01-31T19:44:23.924 INFO:tasks.workunit.client.1.smithi171.stdout:3/329: truncate d4/fa 2557889 0 2022-01-31T19:44:23.924 INFO:tasks.workunit.client.1.smithi171.stdout:3/330: write d4/dd/d39/f50 [378698,449] 0 2022-01-31T19:44:23.924 INFO:tasks.workunit.client.1.smithi171.stdout:8/282: fdatasync df/d18/f1e 0 2022-01-31T19:44:23.925 INFO:tasks.workunit.client.0.smithi167.stdout:6/844: dwrite d8/d33/d51/d5d/d86/f8d [0,4194304] 0 2022-01-31T19:44:23.928 INFO:tasks.workunit.client.0.smithi167.stdout:6/845: rename d8/d20/l29 to d8/d62/d92/d110/l125 0 2022-01-31T19:44:23.928 INFO:tasks.workunit.client.0.smithi167.stdout:6/846: chown d8/f1c 33892653 1 2022-01-31T19:44:23.934 INFO:tasks.workunit.client.1.smithi171.stdout:8/283: symlink df/d27/l65 0 2022-01-31T19:44:23.939 INFO:tasks.workunit.client.1.smithi171.stdout:2/412: link d7/f11 d7/dd/f7e 0 2022-01-31T19:44:23.940 INFO:tasks.workunit.client.1.smithi171.stdout:2/413: write d7/dd/f4c [824072,17058] 0 2022-01-31T19:44:23.940 INFO:tasks.workunit.client.1.smithi171.stdout:2/414: creat d7/d26/d63/f7f x:0 0 0 2022-01-31T19:44:23.940 INFO:tasks.workunit.client.1.smithi171.stdout:2/415: read - d7/d9/f6a zero size 2022-01-31T19:44:23.940 INFO:tasks.workunit.client.1.smithi171.stdout:2/416: fdatasync d7/d9/fe 0 2022-01-31T19:44:23.941 INFO:tasks.workunit.client.1.smithi171.stdout:8/284: symlink df/d27/d5c/l66 0 2022-01-31T19:44:23.943 INFO:tasks.workunit.client.1.smithi171.stdout:2/417: read d7/dd/f39 [1068906,58003] 0 2022-01-31T19:44:23.945 INFO:tasks.workunit.client.1.smithi171.stdout:2/418: link d7/d26/d3d/d40/f78 d7/dd/f80 0 2022-01-31T19:44:23.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/419: chown d7/dd/f7e 1892 1 2022-01-31T19:44:23.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/420: mkdir d7/dd/d5e/d6b/d81 0 2022-01-31T19:44:23.947 INFO:tasks.workunit.client.1.smithi171.stdout:2/421: chown d7/dd/l17 23766425 1 2022-01-31T19:44:23.947 INFO:tasks.workunit.client.1.smithi171.stdout:2/422: symlink d7/d55/l82 0 2022-01-31T19:44:23.948 INFO:tasks.workunit.client.1.smithi171.stdout:2/423: readlink d7/d26/l5a 0 2022-01-31T19:44:23.949 INFO:tasks.workunit.client.1.smithi171.stdout:2/424: mkdir d7/d7c/d83 0 2022-01-31T19:44:23.949 INFO:tasks.workunit.client.1.smithi171.stdout:2/425: dread - d7/d26/d3d/d40/f6c zero size 2022-01-31T19:44:23.949 INFO:tasks.workunit.client.1.smithi171.stdout:2/426: stat d7/d55/l82 0 2022-01-31T19:44:23.950 INFO:tasks.workunit.client.1.smithi171.stdout:2/427: truncate d7/f56 956263 0 2022-01-31T19:44:23.951 INFO:tasks.workunit.client.0.smithi167.stdout:0/824: dwrite d17/d19/d1c/fc3 [0,4194304] 0 2022-01-31T19:44:23.952 INFO:tasks.workunit.client.1.smithi171.stdout:2/428: mkdir d7/dd/d5e/d84 0 2022-01-31T19:44:23.953 INFO:tasks.workunit.client.0.smithi167.stdout:0/825: mkdir d17/d3e/d7e/dc7/dde/df1/d113 0 2022-01-31T19:44:23.953 INFO:tasks.workunit.client.0.smithi167.stdout:6/847: dwrite d8/d33/d51/d72/fc9 [0,4194304] 0 2022-01-31T19:44:23.954 INFO:tasks.workunit.client.0.smithi167.stdout:0/826: read - d17/d19/d1c/d39/fc5 zero size 2022-01-31T19:44:23.955 INFO:tasks.workunit.client.0.smithi167.stdout:0/827: mknod d17/d19/d1c/d39/d7b/dbd/dd9/c114 0 2022-01-31T19:44:23.956 INFO:tasks.workunit.client.0.smithi167.stdout:0/828: mknod d17/d19/d1c/d39/d7b/de5/c115 0 2022-01-31T19:44:23.956 INFO:tasks.workunit.client.0.smithi167.stdout:0/829: write d17/d2c/dcd/fe9 [850781,119568] 0 2022-01-31T19:44:23.956 INFO:tasks.workunit.client.0.smithi167.stdout:0/830: creat d17/d33/d112/f116 x:0 0 0 2022-01-31T19:44:23.957 INFO:tasks.workunit.client.1.smithi171.stdout:2/429: getdents d7/d9/d1a/d45 0 2022-01-31T19:44:23.958 INFO:tasks.workunit.client.0.smithi167.stdout:0/831: mkdir d17/d19/d1c/d117 0 2022-01-31T19:44:23.958 INFO:tasks.workunit.client.0.smithi167.stdout:0/832: mknod d17/d33/da4/c118 0 2022-01-31T19:44:23.960 INFO:tasks.workunit.client.0.smithi167.stdout:0/833: rename d17/d3e/l71 to d17/l119 0 2022-01-31T19:44:23.960 INFO:tasks.workunit.client.0.smithi167.stdout:0/834: dread - d17/d69/fca zero size 2022-01-31T19:44:23.960 INFO:tasks.workunit.client.0.smithi167.stdout:0/835: truncate d17/d3e/da1/feb 909197 0 2022-01-31T19:44:23.961 INFO:tasks.workunit.client.0.smithi167.stdout:0/836: stat d17/l25 0 2022-01-31T19:44:23.962 INFO:tasks.workunit.client.1.smithi171.stdout:2/430: unlink d7/f7b 0 2022-01-31T19:44:23.962 INFO:tasks.workunit.client.1.smithi171.stdout:2/431: chown d7/d9/fe 1520 1 2022-01-31T19:44:23.963 INFO:tasks.workunit.client.1.smithi171.stdout:2/432: mkdir d7/d7c/d83/d85 0 2022-01-31T19:44:23.963 INFO:tasks.workunit.client.1.smithi171.stdout:2/433: stat d7/d55/l60 0 2022-01-31T19:44:23.963 INFO:tasks.workunit.client.1.smithi171.stdout:2/434: readlink d7/d9/d1a/l72 0 2022-01-31T19:44:23.964 INFO:tasks.workunit.client.1.smithi171.stdout:2/435: creat d7/dd/d5e/f86 x:0 0 0 2022-01-31T19:44:23.964 INFO:tasks.workunit.client.1.smithi171.stdout:2/436: chown d7/d9/d1a/d45/c36 5549 1 2022-01-31T19:44:23.964 INFO:tasks.workunit.client.1.smithi171.stdout:2/437: truncate d7/d9/f6a 117427 0 2022-01-31T19:44:23.965 INFO:tasks.workunit.client.1.smithi171.stdout:8/285: dwrite df/d18/f4e [0,4194304] 0 2022-01-31T19:44:23.965 INFO:tasks.workunit.client.1.smithi171.stdout:8/286: fdatasync f0 0 2022-01-31T19:44:23.981 INFO:tasks.workunit.client.0.smithi167.stdout:0/837: dwrite d17/d2c/d87/df4/f10a [0,4194304] 0 2022-01-31T19:44:23.991 INFO:tasks.workunit.client.1.smithi171.stdout:8/287: dwrite df/d1a/f3b [4194304,4194304] 0 2022-01-31T19:44:23.995 INFO:tasks.workunit.client.0.smithi167.stdout:0/838: dwrite d17/d69/f10d [0,4194304] 0 2022-01-31T19:44:23.997 INFO:tasks.workunit.client.0.smithi167.stdout:0/839: mknod d17/d19/d1c/d39/d7b/de5/d104/c11a 0 2022-01-31T19:44:24.006 INFO:tasks.workunit.client.1.smithi171.stdout:9/379: sync 2022-01-31T19:44:24.007 INFO:tasks.workunit.client.1.smithi171.stdout:9/380: symlink dd/l7e 0 2022-01-31T19:44:24.009 INFO:tasks.workunit.client.1.smithi171.stdout:9/381: mknod dd/df/d3f/d73/c7f 0 2022-01-31T19:44:24.009 INFO:tasks.workunit.client.1.smithi171.stdout:9/382: unlink dd/d1c/f52 0 2022-01-31T19:44:24.012 INFO:tasks.workunit.client.0.smithi167.stdout:0/840: dwrite d17/d19/fb1 [4194304,4194304] 0 2022-01-31T19:44:24.014 INFO:tasks.workunit.client.0.smithi167.stdout:0/841: symlink d17/d2c/l11b 0 2022-01-31T19:44:24.015 INFO:tasks.workunit.client.0.smithi167.stdout:0/842: getdents d17/d19/d1c/d39/d7b/dbd 0 2022-01-31T19:44:24.016 INFO:tasks.workunit.client.0.smithi167.stdout:0/843: dread - d17/d33/da4/dba/fc1 zero size 2022-01-31T19:44:24.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/383: dwrite dd/df/d15/d1a/f56 [0,4194304] 0 2022-01-31T19:44:24.028 INFO:tasks.workunit.client.1.smithi171.stdout:9/384: write fc [1235497,104480] 0 2022-01-31T19:44:24.028 INFO:tasks.workunit.client.1.smithi171.stdout:9/385: truncate dd/f37 7483345 0 2022-01-31T19:44:24.037 INFO:tasks.workunit.client.0.smithi167.stdout:0/844: dwrite d17/d69/fa0 [0,4194304] 0 2022-01-31T19:44:24.047 INFO:tasks.workunit.client.1.smithi171.stdout:9/386: dread dd/f37 [0,4194304] 0 2022-01-31T19:44:24.050 INFO:tasks.workunit.client.1.smithi171.stdout:9/387: dread dd/df/d15/f69 [0,4194304] 0 2022-01-31T19:44:24.050 INFO:tasks.workunit.client.1.smithi171.stdout:9/388: mkdir dd/df/d15/d38/d4d/d80 0 2022-01-31T19:44:24.068 INFO:tasks.workunit.client.1.smithi171.stdout:9/389: dwrite dd/f5b [0,4194304] 0 2022-01-31T19:44:24.069 INFO:tasks.workunit.client.1.smithi171.stdout:9/390: rmdir dd/df/d15/d38/d4d 39 2022-01-31T19:44:24.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: cluster 2022-01-31T19:44:22.434590+0000 mgr.smithi171.asyxnc (mgr.24387) 23 : cluster 2022-01-31T19:44:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: [DBG] pgmap v20: 65 pgs: 65 active+clean; 2.2 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 121 MiB/s rd, 237 MiB/s wr, 344 op/s 2022-01-31T19:44:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:22.956801+0000 mon.smithi167 (mon.0) 667 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:22. 2022-01-31T19:44:24.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 963828+0000 mon.smithi171 (mon.1) 53 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T 2022-01-31T19:44:24.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 19:44:22.964374+0000 mon.smithi167 (mon.0) 2022-01-31T19:44:24.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 668 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01 2022-01-31T19:44:24.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: -31T19:44:22.965833+0000 mon.smithi171 (mon 2022-01-31T19:44:24.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: .1) 54 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:22. 2022-01-31T19:44:24.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 966205+0000 mon.smithi167 (mon.0) 669 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44 2022-01-31T19:44:24.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: :22.967416+0000 mon.smithi171 (mon.1) 55 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 2022-01-31T19:44:24.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:44:24.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 44:22.967648+0000 mon.smithi167 (mon.0) 670 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:22 2022-01-31T19:44:24.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: .975831+0000 mon.smithi167 (mon.0) 671 : audit 2022-01-31T19:44:24.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:24.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:23 2022-01-31T19:44:24.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: .331606+0000 mon.smithi167 (mon.0) 672 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:24.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19: 2022-01-31T19:44:24.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 44:23.335379+0000 mon.smithi171 (mon.1) 56 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:23. 2022-01-31T19:44:24.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 335607+0000 mon.smithi167 (mon.0) 673 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:23.336455+0000 mon.smithi171 2022-01-31T19:44:24.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: (mon.1) 57 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:23. 2022-01-31T19:44:24.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 336657+0000 mon.smithi167 (mon.0) 674 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:44:24.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: T19:44:23.337455+0000 mon.smithi171 (mon.1) 2022-01-31T19:44:24.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 58 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 2022-01-31T19: 2022-01-31T19:44:24.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: 44:23.337636+0000 mon.smithi167 (mon.0) 675 2022-01-31T19:44:24.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-01-31T19:44:24.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31T19:44:23 2022-01-31T19:44:24.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: .342308+0000 mon.smithi167 (mon.0) 676 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:24.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:44:24.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: T19:44:23.577842+0000 mon.smithi167 (mon.0) 677 : audit [INF] 2022-01-31T19:44:24.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:23 smithi167 conmon[32206]: from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:24.203 INFO:tasks.workunit.client.1.smithi171.stdout:9/391: link dd/df/d3f/f51 dd/f81 0 2022-01-31T19:44:24.204 INFO:tasks.workunit.client.1.smithi171.stdout:9/392: creat dd/df/d3f/f82 x:0 0 0 2022-01-31T19:44:24.205 INFO:tasks.workunit.client.1.smithi171.stdout:9/393: mknod dd/df/d15/c83 0 2022-01-31T19:44:24.207 INFO:tasks.workunit.client.1.smithi171.stdout:9/394: mkdir dd/df/d15/d1a/d84 0 2022-01-31T19:44:24.207 INFO:tasks.workunit.client.1.smithi171.stdout:9/395: dread - dd/df/d15/d38/d41/f50 zero size 2022-01-31T19:44:24.208 INFO:tasks.workunit.client.1.smithi171.stdout:9/396: write dd/df/d15/f69 [5069804,20828] 0 2022-01-31T19:44:24.208 INFO:tasks.workunit.client.1.smithi171.stdout:9/397: write dd/df/d15/d38/d41/f76 [344728,103540] 0 2022-01-31T19:44:24.209 INFO:tasks.workunit.client.1.smithi171.stdout:9/398: dread dd/d1c/f49 [0,4194304] 0 2022-01-31T19:44:24.212 INFO:tasks.workunit.client.1.smithi171.stdout:9/399: mknod dd/df/d15/c85 0 2022-01-31T19:44:24.213 INFO:tasks.workunit.client.1.smithi171.stdout:9/400: mkdir dd/df/d86 0 2022-01-31T19:44:24.213 INFO:tasks.workunit.client.1.smithi171.stdout:9/401: dread - dd/df/d3f/f82 zero size 2022-01-31T19:44:24.213 INFO:tasks.workunit.client.1.smithi171.stdout:9/402: fdatasync dd/f74 0 2022-01-31T19:44:24.243 INFO:tasks.workunit.client.1.smithi171.stdout:1/293: sync 2022-01-31T19:44:24.243 INFO:tasks.workunit.client.1.smithi171.stdout:1/294: readlink dd/df/d19/d1b/l2b 0 2022-01-31T19:44:24.396 INFO:tasks.workunit.client.1.smithi171.stdout:7/243: sync 2022-01-31T19:44:24.397 INFO:tasks.workunit.client.1.smithi171.stdout:0/333: sync 2022-01-31T19:44:24.397 INFO:tasks.workunit.client.1.smithi171.stdout:4/285: sync 2022-01-31T19:44:24.397 INFO:tasks.workunit.client.1.smithi171.stdout:4/286: fdatasync d2/d3/d5/f2e 0 2022-01-31T19:44:24.397 INFO:tasks.workunit.client.1.smithi171.stdout:6/241: sync 2022-01-31T19:44:24.398 INFO:tasks.workunit.client.1.smithi171.stdout:5/281: sync 2022-01-31T19:44:24.398 INFO:tasks.workunit.client.1.smithi171.stdout:0/334: creat d9/db/dc/d3f/f81 x:0 0 0 2022-01-31T19:44:24.398 INFO:tasks.workunit.client.1.smithi171.stdout:5/282: readlink l3 0 2022-01-31T19:44:24.399 INFO:tasks.workunit.client.1.smithi171.stdout:4/287: unlink d2/c9 0 2022-01-31T19:44:24.399 INFO:tasks.workunit.client.1.smithi171.stdout:0/335: mknod d9/db/d22/c82 0 2022-01-31T19:44:24.400 INFO:tasks.workunit.client.1.smithi171.stdout:0/336: readlink d9/db/dc/d3f/d25/l2a 0 2022-01-31T19:44:24.400 INFO:tasks.workunit.client.1.smithi171.stdout:4/288: creat d2/f5b x:0 0 0 2022-01-31T19:44:24.401 INFO:tasks.workunit.client.1.smithi171.stdout:4/289: link d2/f13 d2/f5c 0 2022-01-31T19:44:24.402 INFO:tasks.workunit.client.1.smithi171.stdout:7/244: write d1/d24/f44 [1796763,9982] 0 2022-01-31T19:44:24.404 INFO:tasks.workunit.client.1.smithi171.stdout:4/290: rename d2/d3/d5/dc/c33 to d2/d3/d5/d3b/c5d 0 2022-01-31T19:44:24.404 INFO:tasks.workunit.client.1.smithi171.stdout:4/291: write d2/d3/d5/dc/f39 [121440,124593] 0 2022-01-31T19:44:24.406 INFO:tasks.workunit.client.1.smithi171.stdout:7/245: creat d1/d1e/d2a/d29/d31/f51 x:0 0 0 2022-01-31T19:44:24.408 INFO:tasks.workunit.client.1.smithi171.stdout:4/292: link d2/c8 d2/d3/d5/d11/d36/c5e 0 2022-01-31T19:44:24.410 INFO:tasks.workunit.client.1.smithi171.stdout:5/283: dread d5/d6/d15/f1c [0,4194304] 0 2022-01-31T19:44:24.412 INFO:tasks.workunit.client.1.smithi171.stdout:5/284: mknod d5/d6/d15/d29/c60 0 2022-01-31T19:44:24.413 INFO:tasks.workunit.client.1.smithi171.stdout:7/246: rename d1/d24 to d1/d52 0 2022-01-31T19:44:24.413 INFO:tasks.workunit.client.1.smithi171.stdout:7/247: read - d1/d1e/d2a/f3b zero size 2022-01-31T19:44:24.414 INFO:tasks.workunit.client.1.smithi171.stdout:5/285: rename d5/d6/d15/d23/c50 to d5/d6/d15/d23/d58/c61 0 2022-01-31T19:44:24.415 INFO:tasks.workunit.client.1.smithi171.stdout:7/248: getdents d1/d1e/d2a/d29 0 2022-01-31T19:44:24.415 INFO:tasks.workunit.client.1.smithi171.stdout:4/293: write d2/d3/d5/dc/f26 [2242134,11993] 0 2022-01-31T19:44:24.416 INFO:tasks.workunit.client.1.smithi171.stdout:7/249: symlink d1/d26/d47/l53 0 2022-01-31T19:44:24.416 INFO:tasks.workunit.client.1.smithi171.stdout:7/250: stat d1/d52/f18 0 2022-01-31T19:44:24.417 INFO:tasks.workunit.client.1.smithi171.stdout:4/294: rename d2/ld to d2/d3/d5/dc/l5f 0 2022-01-31T19:44:24.418 INFO:tasks.workunit.client.1.smithi171.stdout:4/295: creat d2/d3/d5/d11/d12/f60 x:0 0 0 2022-01-31T19:44:24.419 INFO:tasks.workunit.client.1.smithi171.stdout:7/251: dread d1/d52/f28 [0,4194304] 0 2022-01-31T19:44:24.420 INFO:tasks.workunit.client.1.smithi171.stdout:4/296: mknod d2/d3/d5/d3b/c61 0 2022-01-31T19:44:24.421 INFO:tasks.workunit.client.1.smithi171.stdout:0/337: dwrite d9/db/d22/d3a/f3e [0,4194304] 0 2022-01-31T19:44:24.422 INFO:tasks.workunit.client.1.smithi171.stdout:4/297: creat d2/d3/d5/d11/f62 x:0 0 0 2022-01-31T19:44:24.422 INFO:tasks.workunit.client.1.smithi171.stdout:4/298: chown d2/d3/d5/d41/f47 342 1 2022-01-31T19:44:24.424 INFO:tasks.workunit.client.1.smithi171.stdout:6/242: dwrite dc/d18/f1b [0,4194304] 0 2022-01-31T19:44:24.426 INFO:tasks.workunit.client.1.smithi171.stdout:4/299: write d2/d3/f7 [3990088,86609] 0 2022-01-31T19:44:24.426 INFO:tasks.workunit.client.1.smithi171.stdout:4/300: truncate d2/d3/d5/f59 896067 0 2022-01-31T19:44:24.427 INFO:tasks.workunit.client.1.smithi171.stdout:0/338: creat d9/db/d22/d5d/f83 x:0 0 0 2022-01-31T19:44:24.431 INFO:tasks.workunit.client.1.smithi171.stdout:7/252: dread d1/d1e/d2a/d29/d31/ff [0,4194304] 0 2022-01-31T19:44:24.434 INFO:tasks.workunit.client.1.smithi171.stdout:4/301: mknod d2/d3/d5/d41/c63 0 2022-01-31T19:44:24.434 INFO:tasks.workunit.client.1.smithi171.stdout:5/286: dwrite d5/d6/d15/d29/f5c [0,4194304] 0 2022-01-31T19:44:24.435 INFO:tasks.workunit.client.1.smithi171.stdout:0/339: creat d9/d24/d4c/d35/f84 x:0 0 0 2022-01-31T19:44:24.441 INFO:tasks.workunit.client.1.smithi171.stdout:6/243: symlink dc/d18/l42 0 2022-01-31T19:44:24.442 INFO:tasks.workunit.client.1.smithi171.stdout:4/302: write d2/d3/d5/dc/f39 [868029,114938] 0 2022-01-31T19:44:24.443 INFO:tasks.workunit.client.1.smithi171.stdout:5/287: unlink d5/d59/l5b 0 2022-01-31T19:44:24.445 INFO:tasks.workunit.client.1.smithi171.stdout:0/340: link d9/f1f d9/db/d22/d3a/f85 0 2022-01-31T19:44:24.446 INFO:tasks.workunit.client.1.smithi171.stdout:0/341: chown f4 43 1 2022-01-31T19:44:24.446 INFO:tasks.workunit.client.1.smithi171.stdout:0/342: dread - d9/d24/d4c/f66 zero size 2022-01-31T19:44:24.447 INFO:tasks.workunit.client.1.smithi171.stdout:0/343: chown d9/db/dc/d3f/d25/f52 347209 1 2022-01-31T19:44:24.449 INFO:tasks.workunit.client.1.smithi171.stdout:6/244: mknod dc/d19/d29/d27/d37/c43 0 2022-01-31T19:44:24.449 INFO:tasks.workunit.client.1.smithi171.stdout:6/245: chown dc/d19/d29/d27/d37/d3b 1723515917 1 2022-01-31T19:44:24.449 INFO:tasks.workunit.client.1.smithi171.stdout:6/246: write dc/d19/f1d [777704,35723] 0 2022-01-31T19:44:24.450 INFO:tasks.workunit.client.1.smithi171.stdout:4/303: creat d2/d3/d5/d11/d12/d15/f64 x:0 0 0 2022-01-31T19:44:24.451 INFO:tasks.workunit.client.1.smithi171.stdout:0/344: symlink d9/d24/d4c/d7d/l86 0 2022-01-31T19:44:24.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/304: symlink d2/d3/d5/d11/d12/d15/l65 0 2022-01-31T19:44:24.451 INFO:tasks.workunit.client.1.smithi171.stdout:0/345: write d9/db/f21 [2730476,47652] 0 2022-01-31T19:44:24.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/305: truncate d2/d3/d5/dc/f19 949918 0 2022-01-31T19:44:24.452 INFO:tasks.workunit.client.1.smithi171.stdout:0/346: symlink d9/d24/d4c/d35/l87 0 2022-01-31T19:44:24.453 INFO:tasks.workunit.client.1.smithi171.stdout:4/306: mkdir d2/d3/d5/dc/d4b/d56/d66 0 2022-01-31T19:44:24.453 INFO:tasks.workunit.client.1.smithi171.stdout:0/347: mkdir d9/db/d22/d3a/d7a/d88 0 2022-01-31T19:44:24.454 INFO:tasks.workunit.client.1.smithi171.stdout:0/348: chown d9/db/d59 136145 1 2022-01-31T19:44:24.454 INFO:tasks.workunit.client.1.smithi171.stdout:4/307: mknod d2/d3/d5/d3b/c67 0 2022-01-31T19:44:24.455 INFO:tasks.workunit.client.1.smithi171.stdout:0/349: write d9/db/f41 [3706207,118842] 0 2022-01-31T19:44:24.455 INFO:tasks.workunit.client.1.smithi171.stdout:4/308: getdents d2/d3/d5/d11/d12/d15/d57 0 2022-01-31T19:44:24.458 INFO:tasks.workunit.client.1.smithi171.stdout:4/309: creat d2/d3/d5/dc/d4b/f68 x:0 0 0 2022-01-31T19:44:24.458 INFO:tasks.workunit.client.1.smithi171.stdout:4/310: readlink d2/d3/d5/lf 0 2022-01-31T19:44:24.458 INFO:tasks.workunit.client.1.smithi171.stdout:4/311: chown d2/d3/d5/d11/d12/d15/f64 882 1 2022-01-31T19:44:24.459 INFO:tasks.workunit.client.1.smithi171.stdout:4/312: chown d2/d3/d5/d3b/c5d 317 1 2022-01-31T19:44:24.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/313: write d2/d3/d5/d11/d12/d15/f52 [1772159,29718] 0 2022-01-31T19:44:24.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/314: getdents d2/d3/d5/dc/d4b/d56/d66 0 2022-01-31T19:44:24.466 INFO:tasks.workunit.client.1.smithi171.stdout:4/315: mknod d2/d3/d5/dc/d4b/d56/c69 0 2022-01-31T19:44:24.471 INFO:tasks.workunit.client.1.smithi171.stdout:5/288: dwrite d5/d6/d15/f33 [0,4194304] 0 2022-01-31T19:44:24.472 INFO:tasks.workunit.client.1.smithi171.stdout:6/247: dwrite dc/d21/f2e [0,4194304] 0 2022-01-31T19:44:24.472 INFO:tasks.workunit.client.1.smithi171.stdout:6/248: write dc/d19/d30/f3d [1065784,94656] 0 2022-01-31T19:44:24.475 INFO:tasks.workunit.client.1.smithi171.stdout:5/289: dread d5/d6/dd/f22 [0,4194304] 0 2022-01-31T19:44:24.477 INFO:tasks.workunit.client.1.smithi171.stdout:6/249: rename dc/d21/c41 to dc/d19/d29/c44 0 2022-01-31T19:44:24.479 INFO:tasks.workunit.client.1.smithi171.stdout:5/290: rename l3 to d5/d6/d15/d5d/l62 0 2022-01-31T19:44:24.479 INFO:tasks.workunit.client.1.smithi171.stdout:6/250: write fa [3912900,49207] 0 2022-01-31T19:44:24.481 INFO:tasks.workunit.client.1.smithi171.stdout:0/350: dread d9/db/f21 [0,4194304] 0 2022-01-31T19:44:24.481 INFO:tasks.workunit.client.1.smithi171.stdout:0/351: fsync d9/f62 0 2022-01-31T19:44:24.482 INFO:tasks.workunit.client.1.smithi171.stdout:0/352: write d9/d24/d4c/f66 [461013,1300] 0 2022-01-31T19:44:24.482 INFO:tasks.workunit.client.1.smithi171.stdout:6/251: creat dc/d1e/f45 x:0 0 0 2022-01-31T19:44:24.485 INFO:tasks.workunit.client.1.smithi171.stdout:0/353: dread d9/f62 [0,4194304] 0 2022-01-31T19:44:24.485 INFO:tasks.workunit.client.1.smithi171.stdout:0/354: fdatasync d9/d24/f4d 0 2022-01-31T19:44:24.485 INFO:tasks.workunit.client.1.smithi171.stdout:0/355: write d9/d24/d4c/d35/f74 [61633,70725] 0 2022-01-31T19:44:24.486 INFO:tasks.workunit.client.1.smithi171.stdout:6/252: link dc/d19/d30/l35 dc/l46 0 2022-01-31T19:44:24.487 INFO:tasks.workunit.client.1.smithi171.stdout:6/253: symlink dc/l47 0 2022-01-31T19:44:24.489 INFO:tasks.workunit.client.1.smithi171.stdout:6/254: symlink dc/d19/d29/d27/d37/l48 0 2022-01-31T19:44:24.489 INFO:tasks.workunit.client.1.smithi171.stdout:6/255: truncate dc/d19/d29/f34 597334 0 2022-01-31T19:44:24.489 INFO:tasks.workunit.client.1.smithi171.stdout:4/316: dwrite d2/d3/d5/d3b/f55 [0,4194304] 0 2022-01-31T19:44:24.492 INFO:tasks.workunit.client.1.smithi171.stdout:6/256: symlink dc/d19/d29/d27/l49 0 2022-01-31T19:44:24.506 INFO:tasks.workunit.client.1.smithi171.stdout:5/291: dwrite d5/d6/dd/f20 [0,4194304] 0 2022-01-31T19:44:24.506 INFO:tasks.workunit.client.1.smithi171.stdout:5/292: write d5/d6/d15/f1d [5080020,44072] 0 2022-01-31T19:44:24.508 INFO:tasks.workunit.client.1.smithi171.stdout:0/356: dwrite d9/f3b [0,4194304] 0 2022-01-31T19:44:24.508 INFO:tasks.workunit.client.1.smithi171.stdout:0/357: chown d9/db/dc/c40 18112 1 2022-01-31T19:44:24.513 INFO:tasks.workunit.client.1.smithi171.stdout:6/257: getdents dc 0 2022-01-31T19:44:24.517 INFO:tasks.workunit.client.1.smithi171.stdout:0/358: creat d9/d24/d4c/d35/f89 x:0 0 0 2022-01-31T19:44:24.518 INFO:tasks.workunit.client.1.smithi171.stdout:0/359: creat d9/f8a x:0 0 0 2022-01-31T19:44:24.526 INFO:tasks.workunit.client.1.smithi171.stdout:5/293: dwrite d5/d6/dd/f45 [0,4194304] 0 2022-01-31T19:44:24.527 INFO:tasks.workunit.client.1.smithi171.stdout:6/258: dread f9 [4194304,4194304] 0 2022-01-31T19:44:24.528 INFO:tasks.workunit.client.1.smithi171.stdout:5/294: symlink d5/d6/d15/d23/d3d/l63 0 2022-01-31T19:44:24.528 INFO:tasks.workunit.client.1.smithi171.stdout:5/295: chown d5/l47 26 1 2022-01-31T19:44:24.529 INFO:tasks.workunit.client.1.smithi171.stdout:6/259: creat dc/f4a x:0 0 0 2022-01-31T19:44:24.531 INFO:tasks.workunit.client.1.smithi171.stdout:5/296: getdents d5/d6/dd 0 2022-01-31T19:44:24.533 INFO:tasks.workunit.client.1.smithi171.stdout:5/297: creat d5/d6/d15/f64 x:0 0 0 2022-01-31T19:44:24.533 INFO:tasks.workunit.client.1.smithi171.stdout:0/360: dread f6 [0,4194304] 0 2022-01-31T19:44:24.533 INFO:tasks.workunit.client.1.smithi171.stdout:0/361: creat d9/db/dc/f8b x:0 0 0 2022-01-31T19:44:24.534 INFO:tasks.workunit.client.1.smithi171.stdout:4/317: dwrite d2/d3/d5/d3b/f55 [4194304,4194304] 0 2022-01-31T19:44:24.536 INFO:tasks.workunit.client.1.smithi171.stdout:5/298: rename d5/d6/dd/f5f to d5/d6/d15/d5d/f65 0 2022-01-31T19:44:24.540 INFO:tasks.workunit.client.1.smithi171.stdout:5/299: mkdir d5/d6/dd/d66 0 2022-01-31T19:44:24.543 INFO:tasks.workunit.client.1.smithi171.stdout:4/318: dread d2/d3/d5/d3b/f55 [0,4194304] 0 2022-01-31T19:44:24.544 INFO:tasks.workunit.client.1.smithi171.stdout:5/300: write d5/d6/d15/f3f [7949,90256] 0 2022-01-31T19:44:24.547 INFO:tasks.workunit.client.1.smithi171.stdout:4/319: rename d2/d3/d5/d11/d12/c23 to d2/d3/c6a 0 2022-01-31T19:44:24.547 INFO:tasks.workunit.client.1.smithi171.stdout:4/320: fsync d2/d3/d5/dc/d4b/f68 0 2022-01-31T19:44:24.547 INFO:tasks.workunit.client.1.smithi171.stdout:4/321: stat d2/d3/d5/d11/d12/f2a 0 2022-01-31T19:44:24.552 INFO:tasks.workunit.client.1.smithi171.stdout:5/301: rename d5/d6/d15/d29 to d5/d6/d15/d23/d67 0 2022-01-31T19:44:24.557 INFO:tasks.workunit.client.1.smithi171.stdout:6/260: dwrite dc/d18/f2a [0,4194304] 0 2022-01-31T19:44:24.558 INFO:tasks.workunit.client.1.smithi171.stdout:6/261: creat dc/d19/d30/d3c/f4b x:0 0 0 2022-01-31T19:44:24.560 INFO:tasks.workunit.client.1.smithi171.stdout:6/262: creat dc/d21/f4c x:0 0 0 2022-01-31T19:44:24.560 INFO:tasks.workunit.client.1.smithi171.stdout:6/263: write f2 [5211803,27436] 0 2022-01-31T19:44:24.560 INFO:tasks.workunit.client.1.smithi171.stdout:0/362: dwrite d9/f1d [4194304,4194304] 0 2022-01-31T19:44:24.562 INFO:tasks.workunit.client.1.smithi171.stdout:0/363: write d9/f5c [1160958,45568] 0 2022-01-31T19:44:24.562 INFO:tasks.workunit.client.1.smithi171.stdout:5/302: dread d5/d6/f16 [0,4194304] 0 2022-01-31T19:44:24.563 INFO:tasks.workunit.client.1.smithi171.stdout:5/303: stat d5/d6/d15/d23/d58 0 2022-01-31T19:44:24.564 INFO:tasks.workunit.client.1.smithi171.stdout:0/364: dread f3 [0,4194304] 0 2022-01-31T19:44:24.582 INFO:tasks.workunit.client.1.smithi171.stdout:6/264: dwrite dc/d19/d30/f3d [0,4194304] 0 2022-01-31T19:44:24.594 INFO:tasks.workunit.client.1.smithi171.stdout:5/304: getdents d5/d6/d15/d23 0 2022-01-31T19:44:24.594 INFO:tasks.workunit.client.1.smithi171.stdout:5/305: chown d5/d6/dd 17683014 1 2022-01-31T19:44:24.595 INFO:tasks.workunit.client.1.smithi171.stdout:5/306: read d5/d6/d15/d23/d67/f4d [23119,114335] 0 2022-01-31T19:44:24.595 INFO:tasks.workunit.client.1.smithi171.stdout:5/307: fsync d5/d6/d15/d5d/f65 0 2022-01-31T19:44:24.596 INFO:tasks.workunit.client.1.smithi171.stdout:0/365: rename d9/d24/d4c/d35/f84 to d9/db/d22/f8c 0 2022-01-31T19:44:24.597 INFO:tasks.workunit.client.1.smithi171.stdout:6/265: fsync dc/d18/f2c 0 2022-01-31T19:44:24.597 INFO:tasks.workunit.client.1.smithi171.stdout:6/266: chown dc/l47 173 1 2022-01-31T19:44:24.601 INFO:tasks.workunit.client.1.smithi171.stdout:3/331: sync 2022-01-31T19:44:24.601 INFO:tasks.workunit.client.1.smithi171.stdout:1/295: sync 2022-01-31T19:44:24.601 INFO:tasks.workunit.client.1.smithi171.stdout:9/403: sync 2022-01-31T19:44:24.602 INFO:tasks.workunit.client.1.smithi171.stdout:2/438: sync 2022-01-31T19:44:24.602 INFO:tasks.workunit.client.1.smithi171.stdout:8/288: sync 2022-01-31T19:44:24.602 INFO:tasks.workunit.client.1.smithi171.stdout:9/404: chown dd/df 1877775 1 2022-01-31T19:44:24.602 INFO:tasks.workunit.client.1.smithi171.stdout:2/439: write d7/dd/d5e/f69 [593633,38814] 0 2022-01-31T19:44:24.603 INFO:tasks.workunit.client.1.smithi171.stdout:9/405: write dd/df/d15/f4e [222750,30819] 0 2022-01-31T19:44:24.603 INFO:tasks.workunit.client.1.smithi171.stdout:1/296: write f7 [969813,123799] 0 2022-01-31T19:44:24.604 INFO:tasks.workunit.client.1.smithi171.stdout:5/308: rmdir d5/d6/d15/d23 39 2022-01-31T19:44:24.604 INFO:tasks.workunit.client.1.smithi171.stdout:0/366: creat d9/d24/d4c/d35/f8d x:0 0 0 2022-01-31T19:44:24.605 INFO:tasks.workunit.client.1.smithi171.stdout:6/267: mknod dc/d19/d29/d27/d37/c4d 0 2022-01-31T19:44:24.606 INFO:tasks.workunit.client.1.smithi171.stdout:5/309: dread d5/d6/f21 [0,4194304] 0 2022-01-31T19:44:24.607 INFO:tasks.workunit.client.1.smithi171.stdout:8/289: truncate f7 214577 0 2022-01-31T19:44:24.610 INFO:tasks.workunit.client.1.smithi171.stdout:3/332: mkdir d4/dd/d1a/d60 0 2022-01-31T19:44:24.610 INFO:tasks.workunit.client.1.smithi171.stdout:3/333: chown d4/fb 223781641 1 2022-01-31T19:44:24.610 INFO:tasks.workunit.client.1.smithi171.stdout:3/334: fdatasync d4/dd/f19 0 2022-01-31T19:44:24.611 INFO:tasks.workunit.client.1.smithi171.stdout:9/406: truncate dd/f37 6237138 0 2022-01-31T19:44:24.611 INFO:tasks.workunit.client.1.smithi171.stdout:9/407: fdatasync dd/df/d15/d1a/d64/f67 0 2022-01-31T19:44:24.611 INFO:tasks.workunit.client.1.smithi171.stdout:9/408: truncate dd/df/d3f/f82 337443 0 2022-01-31T19:44:24.612 INFO:tasks.workunit.client.1.smithi171.stdout:6/268: mknod dc/d19/d29/d27/d37/d3b/c4e 0 2022-01-31T19:44:24.612 INFO:tasks.workunit.client.1.smithi171.stdout:8/290: dread df/d18/d19/f1b [0,4194304] 0 2022-01-31T19:44:24.613 INFO:tasks.workunit.client.1.smithi171.stdout:5/310: creat d5/f68 x:0 0 0 2022-01-31T19:44:24.615 INFO:tasks.workunit.client.1.smithi171.stdout:3/335: write d4/dd/d40/f42 [203057,33580] 0 2022-01-31T19:44:24.615 INFO:tasks.workunit.client.1.smithi171.stdout:9/409: truncate dd/d1c/f1f 869831 0 2022-01-31T19:44:24.616 INFO:tasks.workunit.client.1.smithi171.stdout:8/291: truncate df/d18/f4e 2936776 0 2022-01-31T19:44:24.618 INFO:tasks.workunit.client.1.smithi171.stdout:3/336: symlink d4/d21/d3a/d59/l61 0 2022-01-31T19:44:24.619 INFO:tasks.workunit.client.1.smithi171.stdout:3/337: fdatasync d4/d21/d3a/f4f 0 2022-01-31T19:44:24.619 INFO:tasks.workunit.client.1.smithi171.stdout:3/338: fdatasync d4/d21/f2a 0 2022-01-31T19:44:24.619 INFO:tasks.workunit.client.1.smithi171.stdout:6/269: read dc/d19/d29/f34 [274679,130629] 0 2022-01-31T19:44:24.620 INFO:tasks.workunit.client.1.smithi171.stdout:8/292: mkdir df/d18/d19/d21/d67 0 2022-01-31T19:44:24.620 INFO:tasks.workunit.client.1.smithi171.stdout:6/270: mkdir dc/d21/d4f 0 2022-01-31T19:44:24.621 INFO:tasks.workunit.client.1.smithi171.stdout:6/271: symlink dc/d21/d4f/l50 0 2022-01-31T19:44:24.621 INFO:tasks.workunit.client.1.smithi171.stdout:3/339: write d4/fc [1171721,90898] 0 2022-01-31T19:44:24.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/272: symlink dc/d19/d30/l51 0 2022-01-31T19:44:24.625 INFO:tasks.workunit.client.1.smithi171.stdout:3/340: rename d4/dd/d27/d28/d35/d58 to d4/dd/d27/d4e/d62 0 2022-01-31T19:44:24.630 INFO:tasks.workunit.client.1.smithi171.stdout:5/311: dread d5/d6/d15/d23/d67/f4c [0,4194304] 0 2022-01-31T19:44:24.633 INFO:tasks.workunit.client.1.smithi171.stdout:5/312: mkdir d5/d6/d69 0 2022-01-31T19:44:24.635 INFO:tasks.workunit.client.1.smithi171.stdout:5/313: mkdir d5/d6/d15/d5d/d6a 0 2022-01-31T19:44:24.635 INFO:tasks.workunit.client.1.smithi171.stdout:5/314: dread - d5/d6/d15/f64 zero size 2022-01-31T19:44:24.635 INFO:tasks.workunit.client.1.smithi171.stdout:5/315: write d5/d6/d15/d23/f25 [203662,43235] 0 2022-01-31T19:44:24.637 INFO:tasks.workunit.client.1.smithi171.stdout:3/341: dread d4/dd/d27/f47 [0,4194304] 0 2022-01-31T19:44:24.637 INFO:tasks.workunit.client.1.smithi171.stdout:2/440: dwrite d7/dd/f39 [4194304,4194304] 0 2022-01-31T19:44:24.637 INFO:tasks.workunit.client.1.smithi171.stdout:0/367: dwrite d9/db/d22/d5d/f69 [0,4194304] 0 2022-01-31T19:44:24.642 INFO:tasks.workunit.client.1.smithi171.stdout:8/293: write df/f16 [916470,31663] 0 2022-01-31T19:44:24.643 INFO:tasks.workunit.client.1.smithi171.stdout:2/441: dread d7/d26/f31 [0,4194304] 0 2022-01-31T19:44:24.643 INFO:tasks.workunit.client.1.smithi171.stdout:8/294: dread - df/d2a/d30/f63 zero size 2022-01-31T19:44:24.646 INFO:tasks.workunit.client.1.smithi171.stdout:0/368: creat d9/db/d22/d3a/f8e x:0 0 0 2022-01-31T19:44:24.646 INFO:tasks.workunit.client.1.smithi171.stdout:0/369: fdatasync d9/db/d22/d3a/f42 0 2022-01-31T19:44:24.647 INFO:tasks.workunit.client.1.smithi171.stdout:2/442: creat d7/d7c/f87 x:0 0 0 2022-01-31T19:44:24.648 INFO:tasks.workunit.client.1.smithi171.stdout:8/295: link df/c54 df/c68 0 2022-01-31T19:44:24.649 INFO:tasks.workunit.client.1.smithi171.stdout:2/443: creat d7/d26/f88 x:0 0 0 2022-01-31T19:44:24.650 INFO:tasks.workunit.client.1.smithi171.stdout:6/273: dwrite dc/d19/d30/d3c/f4b [0,4194304] 0 2022-01-31T19:44:24.651 INFO:tasks.workunit.client.1.smithi171.stdout:6/274: write dc/d19/f1a [204523,17307] 0 2022-01-31T19:44:24.651 INFO:tasks.workunit.client.1.smithi171.stdout:6/275: read - dc/f4a zero size 2022-01-31T19:44:24.652 INFO:tasks.workunit.client.1.smithi171.stdout:6/276: chown dc/d19/d29/d27/d37/c4d 11250968 1 2022-01-31T19:44:24.652 INFO:tasks.workunit.client.1.smithi171.stdout:8/296: creat df/d18/d19/d46/f69 x:0 0 0 2022-01-31T19:44:24.653 INFO:tasks.workunit.client.1.smithi171.stdout:2/444: creat d7/d9/d1a/f89 x:0 0 0 2022-01-31T19:44:24.654 INFO:tasks.workunit.client.1.smithi171.stdout:9/410: dwrite dd/df/d15/d1a/f2b [0,4194304] 0 2022-01-31T19:44:24.654 INFO:tasks.workunit.client.1.smithi171.stdout:9/411: fsync dd/d1c/f70 0 2022-01-31T19:44:24.656 INFO:tasks.workunit.client.1.smithi171.stdout:6/277: mknod dc/d21/c52 0 2022-01-31T19:44:24.657 INFO:tasks.workunit.client.1.smithi171.stdout:2/445: symlink d7/dd/d5e/d6b/d81/l8a 0 2022-01-31T19:44:24.657 INFO:tasks.workunit.client.1.smithi171.stdout:2/446: chown d7/d55 0 1 2022-01-31T19:44:24.658 INFO:tasks.workunit.client.1.smithi171.stdout:2/447: chown d7/d26/d63/f7f 10844 1 2022-01-31T19:44:24.659 INFO:tasks.workunit.client.1.smithi171.stdout:9/412: symlink dd/df/d15/d53/l87 0 2022-01-31T19:44:24.659 INFO:tasks.workunit.client.1.smithi171.stdout:9/413: chown dd/df/f72 89360288 1 2022-01-31T19:44:24.663 INFO:tasks.workunit.client.1.smithi171.stdout:5/316: dwrite d5/f8 [0,4194304] 0 2022-01-31T19:44:24.663 INFO:tasks.workunit.client.1.smithi171.stdout:3/342: dwrite d4/dd/f1e [0,4194304] 0 2022-01-31T19:44:24.668 INFO:tasks.workunit.client.1.smithi171.stdout:9/414: creat dd/df/d86/f88 x:0 0 0 2022-01-31T19:44:24.669 INFO:tasks.workunit.client.1.smithi171.stdout:5/317: mkdir d5/d6/d15/d23/d6b 0 2022-01-31T19:44:24.670 INFO:tasks.workunit.client.1.smithi171.stdout:5/318: dread - d5/d6/f46 zero size 2022-01-31T19:44:24.670 INFO:tasks.workunit.client.1.smithi171.stdout:5/319: dread - d5/d6/f46 zero size 2022-01-31T19:44:24.671 INFO:tasks.workunit.client.1.smithi171.stdout:3/343: symlink d4/dd/l63 0 2022-01-31T19:44:24.672 INFO:tasks.workunit.client.1.smithi171.stdout:6/278: mkdir dc/d18/d53 0 2022-01-31T19:44:24.673 INFO:tasks.workunit.client.1.smithi171.stdout:6/279: chown dc/d19/f1d 1877 1 2022-01-31T19:44:24.673 INFO:tasks.workunit.client.1.smithi171.stdout:6/280: fsync dc/d1e/d3f/f38 0 2022-01-31T19:44:24.674 INFO:tasks.workunit.client.1.smithi171.stdout:2/448: getdents d7/d26/d3d 0 2022-01-31T19:44:24.675 INFO:tasks.workunit.client.1.smithi171.stdout:8/297: dwrite f6 [0,4194304] 0 2022-01-31T19:44:24.677 INFO:tasks.workunit.client.1.smithi171.stdout:9/415: rename dd/c13 to dd/df/d15/d1a/d84/c89 0 2022-01-31T19:44:24.677 INFO:tasks.workunit.client.1.smithi171.stdout:9/416: chown dd/df/c60 1566097 1 2022-01-31T19:44:24.679 INFO:tasks.workunit.client.1.smithi171.stdout:6/281: unlink dc/d19/d29/c44 0 2022-01-31T19:44:24.680 INFO:tasks.workunit.client.1.smithi171.stdout:2/449: write d7/dd/f7e [309612,48725] 0 2022-01-31T19:44:24.680 INFO:tasks.workunit.client.1.smithi171.stdout:2/450: truncate d7/d9/d1a/d45/f2f 388703 0 2022-01-31T19:44:24.683 INFO:tasks.workunit.client.1.smithi171.stdout:9/417: symlink dd/d1c/l8a 0 2022-01-31T19:44:24.685 INFO:tasks.workunit.client.1.smithi171.stdout:2/451: creat d7/dd/d5e/d84/f8b x:0 0 0 2022-01-31T19:44:24.687 INFO:tasks.workunit.client.1.smithi171.stdout:5/320: rename d5/d6/d15/l30 to d5/d6/d15/d5d/l6c 0 2022-01-31T19:44:24.690 INFO:tasks.workunit.client.1.smithi171.stdout:2/452: rename d7/d26/d3d/c58 to d7/dd/d5e/d6b/c8c 0 2022-01-31T19:44:24.691 INFO:tasks.workunit.client.1.smithi171.stdout:2/453: mkdir d7/dd/d5e/d6b/d81/d8d 0 2022-01-31T19:44:24.692 INFO:tasks.workunit.client.1.smithi171.stdout:5/321: dread d5/d6/dd/f22 [0,4194304] 0 2022-01-31T19:44:24.693 INFO:tasks.workunit.client.1.smithi171.stdout:5/322: truncate d5/f57 5008189 0 2022-01-31T19:44:24.693 INFO:tasks.workunit.client.1.smithi171.stdout:5/323: chown d5/d6/d15/d23/d3d 2941 1 2022-01-31T19:44:24.693 INFO:tasks.workunit.client.1.smithi171.stdout:2/454: rename d7/c5d to d7/d9/c8e 0 2022-01-31T19:44:24.694 INFO:tasks.workunit.client.1.smithi171.stdout:2/455: dread - d7/d9/f74 zero size 2022-01-31T19:44:24.696 INFO:tasks.workunit.client.1.smithi171.stdout:5/324: creat d5/d6/d15/f6d x:0 0 0 2022-01-31T19:44:24.696 INFO:tasks.workunit.client.1.smithi171.stdout:5/325: getdents d5/d6/d15/d23/d3d 0 2022-01-31T19:44:24.697 INFO:tasks.workunit.client.1.smithi171.stdout:5/326: chown d5/l40 0 1 2022-01-31T19:44:24.697 INFO:tasks.workunit.client.1.smithi171.stdout:6/282: dread dc/d1e/d3f/f38 [0,4194304] 0 2022-01-31T19:44:24.697 INFO:tasks.workunit.client.1.smithi171.stdout:5/327: mknod d5/d6/d15/d23/d67/c6e 0 2022-01-31T19:44:24.697 INFO:tasks.workunit.client.1.smithi171.stdout:5/328: readlink d5/d6/l10 0 2022-01-31T19:44:24.698 INFO:tasks.workunit.client.1.smithi171.stdout:5/329: dread - d5/d6/d15/f6d zero size 2022-01-31T19:44:24.698 INFO:tasks.workunit.client.1.smithi171.stdout:6/283: link dc/d19/d30/c3e dc/c54 0 2022-01-31T19:44:24.699 INFO:tasks.workunit.client.1.smithi171.stdout:5/330: rmdir d5/d6/d15/d5d 39 2022-01-31T19:44:24.699 INFO:tasks.workunit.client.1.smithi171.stdout:6/284: creat dc/d19/f55 x:0 0 0 2022-01-31T19:44:24.700 INFO:tasks.workunit.client.1.smithi171.stdout:8/298: dwrite f6 [4194304,4194304] 0 2022-01-31T19:44:24.700 INFO:tasks.workunit.client.1.smithi171.stdout:8/299: chown df/d27/d5c/l66 71989860 1 2022-01-31T19:44:24.700 INFO:tasks.workunit.client.1.smithi171.stdout:3/344: dwrite d4/dd/d39/f50 [0,4194304] 0 2022-01-31T19:44:24.701 INFO:tasks.workunit.client.1.smithi171.stdout:3/345: stat d4/dd/d1a/c5a 0 2022-01-31T19:44:24.704 INFO:tasks.workunit.client.1.smithi171.stdout:8/300: mknod df/d18/d19/d21/d67/c6a 0 2022-01-31T19:44:24.707 INFO:tasks.workunit.client.1.smithi171.stdout:2/456: fsync d7/dd/d5e/d84/f8b 0 2022-01-31T19:44:24.708 INFO:tasks.workunit.client.1.smithi171.stdout:6/285: dread dc/d21/f2e [0,4194304] 0 2022-01-31T19:44:24.711 INFO:tasks.workunit.client.1.smithi171.stdout:5/331: dwrite d5/f55 [0,4194304] 0 2022-01-31T19:44:24.712 INFO:tasks.workunit.client.1.smithi171.stdout:5/332: fsync d5/ff 0 2022-01-31T19:44:24.712 INFO:tasks.workunit.client.1.smithi171.stdout:3/346: mkdir d4/d21/d3a/d64 0 2022-01-31T19:44:24.713 INFO:tasks.workunit.client.1.smithi171.stdout:3/347: write d4/dd/d1a/f2b [3927562,29883] 0 2022-01-31T19:44:24.714 INFO:tasks.workunit.client.1.smithi171.stdout:8/301: unlink df/d27/l44 0 2022-01-31T19:44:24.715 INFO:tasks.workunit.client.1.smithi171.stdout:2/457: mknod d7/d7c/d83/d85/c8f 0 2022-01-31T19:44:24.716 INFO:tasks.workunit.client.1.smithi171.stdout:2/458: chown d7/d26/d3d/d40 1057316950 1 2022-01-31T19:44:24.716 INFO:tasks.workunit.client.1.smithi171.stdout:6/286: symlink dc/d1e/l56 0 2022-01-31T19:44:24.717 INFO:tasks.workunit.client.1.smithi171.stdout:6/287: write dc/f13 [1105576,97389] 0 2022-01-31T19:44:24.720 INFO:tasks.workunit.client.1.smithi171.stdout:5/333: mkdir d5/d6/d15/d5d/d6f 0 2022-01-31T19:44:24.721 INFO:tasks.workunit.client.1.smithi171.stdout:8/302: symlink df/d18/d19/d36/l6b 0 2022-01-31T19:44:24.723 INFO:tasks.workunit.client.1.smithi171.stdout:2/459: rename d7/dd/l29 to d7/dd/d5e/d84/l90 0 2022-01-31T19:44:24.724 INFO:tasks.workunit.client.1.smithi171.stdout:2/460: fdatasync d7/d9/d1a/f53 0 2022-01-31T19:44:24.726 INFO:tasks.workunit.client.1.smithi171.stdout:6/288: dread dc/d18/f2c [0,4194304] 0 2022-01-31T19:44:24.726 INFO:tasks.workunit.client.1.smithi171.stdout:6/289: readlink l7 0 2022-01-31T19:44:24.727 INFO:tasks.workunit.client.1.smithi171.stdout:8/303: mkdir df/d5d/d6c 0 2022-01-31T19:44:24.727 INFO:tasks.workunit.client.1.smithi171.stdout:8/304: creat df/d18/d19/d36/f6d x:0 0 0 2022-01-31T19:44:24.727 INFO:tasks.workunit.client.1.smithi171.stdout:2/461: mknod d7/d55/c91 0 2022-01-31T19:44:24.731 INFO:tasks.workunit.client.1.smithi171.stdout:8/305: unlink fa 0 2022-01-31T19:44:24.732 INFO:tasks.workunit.client.1.smithi171.stdout:2/462: link d7/d26/f5f d7/d26/f92 0 2022-01-31T19:44:24.733 INFO:tasks.workunit.client.1.smithi171.stdout:6/290: rename dc/d18/c1c to dc/d19/d29/c57 0 2022-01-31T19:44:24.734 INFO:tasks.workunit.client.1.smithi171.stdout:8/306: link df/d2a/d38/f56 df/d5d/d6c/f6e 0 2022-01-31T19:44:24.736 INFO:tasks.workunit.client.1.smithi171.stdout:6/291: unlink dc/d21/c2b 0 2022-01-31T19:44:24.738 INFO:tasks.workunit.client.1.smithi171.stdout:8/307: write fc [301389,28468] 0 2022-01-31T19:44:24.738 INFO:tasks.workunit.client.1.smithi171.stdout:6/292: rmdir dc/d19/d29/d27/d37 39 2022-01-31T19:44:24.740 INFO:tasks.workunit.client.1.smithi171.stdout:6/293: creat dc/d18/d53/f58 x:0 0 0 2022-01-31T19:44:24.741 INFO:tasks.workunit.client.1.smithi171.stdout:6/294: write dc/d18/f25 [1652217,107266] 0 2022-01-31T19:44:24.741 INFO:tasks.workunit.client.1.smithi171.stdout:6/295: dread - dc/d18/d53/f58 zero size 2022-01-31T19:44:24.742 INFO:tasks.workunit.client.1.smithi171.stdout:5/334: dread d5/d6/d15/d23/d67/f39 [0,4194304] 0 2022-01-31T19:44:24.743 INFO:tasks.workunit.client.1.smithi171.stdout:6/296: creat dc/d1e/d3f/f59 x:0 0 0 2022-01-31T19:44:24.745 INFO:tasks.workunit.client.1.smithi171.stdout:6/297: rmdir dc/d19/d30 39 2022-01-31T19:44:24.746 INFO:tasks.workunit.client.1.smithi171.stdout:6/298: creat dc/d21/f5a x:0 0 0 2022-01-31T19:44:24.746 INFO:tasks.workunit.client.1.smithi171.stdout:5/335: read d5/d6/dd/f45 [3665052,47219] 0 2022-01-31T19:44:24.747 INFO:tasks.workunit.client.1.smithi171.stdout:6/299: write dc/f14 [350880,88002] 0 2022-01-31T19:44:24.750 INFO:tasks.workunit.client.1.smithi171.stdout:5/336: write d5/f4f [3554685,12711] 0 2022-01-31T19:44:24.751 INFO:tasks.workunit.client.1.smithi171.stdout:6/300: dread dc/d19/d29/d27/f33 [0,4194304] 0 2022-01-31T19:44:24.751 INFO:tasks.workunit.client.1.smithi171.stdout:5/337: mknod d5/d6/dd/c70 0 2022-01-31T19:44:24.753 INFO:tasks.workunit.client.1.smithi171.stdout:6/301: rename dc/f14 to dc/d18/d53/f5b 0 2022-01-31T19:44:24.754 INFO:tasks.workunit.client.1.smithi171.stdout:2/463: dwrite d7/f14 [0,4194304] 0 2022-01-31T19:44:24.755 INFO:tasks.workunit.client.1.smithi171.stdout:2/464: chown d7/d9/d1a/d45 7 1 2022-01-31T19:44:24.758 INFO:tasks.workunit.client.1.smithi171.stdout:3/348: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:24.758 INFO:tasks.workunit.client.1.smithi171.stdout:8/308: dwrite df/d27/f57 [0,4194304] 0 2022-01-31T19:44:24.763 INFO:tasks.workunit.client.1.smithi171.stdout:6/302: link dc/f13 dc/d19/d29/f5c 0 2022-01-31T19:44:24.768 INFO:tasks.workunit.client.1.smithi171.stdout:6/303: dread dc/d21/f2e [0,4194304] 0 2022-01-31T19:44:24.775 INFO:tasks.workunit.client.1.smithi171.stdout:8/309: symlink df/d5d/d6c/l6f 0 2022-01-31T19:44:24.775 INFO:tasks.workunit.client.1.smithi171.stdout:8/310: fsync df/d18/d19/d36/f3e 0 2022-01-31T19:44:24.777 INFO:tasks.workunit.client.1.smithi171.stdout:2/465: mkdir d7/dd/d93 0 2022-01-31T19:44:24.778 INFO:tasks.workunit.client.1.smithi171.stdout:2/466: write d7/dd/d5e/d84/f8b [597846,121305] 0 2022-01-31T19:44:24.779 INFO:tasks.workunit.client.1.smithi171.stdout:3/349: truncate d4/dd/d1a/f2b 1354938 0 2022-01-31T19:44:24.780 INFO:tasks.workunit.client.1.smithi171.stdout:3/350: readlink d4/dd/l15 0 2022-01-31T19:44:24.780 INFO:tasks.workunit.client.1.smithi171.stdout:3/351: getdents d4/dd/d27/d4e/d62 0 2022-01-31T19:44:24.780 INFO:tasks.workunit.client.1.smithi171.stdout:6/304: write dc/d18/f39 [3317970,18806] 0 2022-01-31T19:44:24.781 INFO:tasks.workunit.client.1.smithi171.stdout:8/311: mkdir df/d18/d19/d36/d70 0 2022-01-31T19:44:24.781 INFO:tasks.workunit.client.1.smithi171.stdout:6/305: write dc/d18/d53/f5b [456618,103706] 0 2022-01-31T19:44:24.781 INFO:tasks.workunit.client.1.smithi171.stdout:6/306: write dc/f4a [809540,82007] 0 2022-01-31T19:44:24.784 INFO:tasks.workunit.client.1.smithi171.stdout:2/467: mknod d7/dd/d5e/d6b/d81/c94 0 2022-01-31T19:44:24.784 INFO:tasks.workunit.client.1.smithi171.stdout:2/468: chown d7/dd/f39 119780 1 2022-01-31T19:44:24.784 INFO:tasks.workunit.client.1.smithi171.stdout:2/469: creat d7/d26/d3d/d40/f95 x:0 0 0 2022-01-31T19:44:24.785 INFO:tasks.workunit.client.1.smithi171.stdout:2/470: truncate d7/d9/d1a/f89 394746 0 2022-01-31T19:44:24.785 INFO:tasks.workunit.client.1.smithi171.stdout:3/352: truncate d4/fa 536661 0 2022-01-31T19:44:24.791 INFO:tasks.workunit.client.1.smithi171.stdout:2/471: write d7/d55/f65 [3459754,55245] 0 2022-01-31T19:44:24.793 INFO:tasks.workunit.client.1.smithi171.stdout:3/353: dread d4/fa [0,4194304] 0 2022-01-31T19:44:24.821 INFO:tasks.workunit.client.1.smithi171.stdout:8/312: dwrite df/d18/d19/d21/f32 [0,4194304] 0 2022-01-31T19:44:24.822 INFO:tasks.workunit.client.1.smithi171.stdout:8/313: symlink df/d18/d19/d36/l71 0 2022-01-31T19:44:24.823 INFO:tasks.workunit.client.1.smithi171.stdout:8/314: write df/f16 [52901,9527] 0 2022-01-31T19:44:24.824 INFO:tasks.workunit.client.1.smithi171.stdout:2/472: dwrite d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:24.824 INFO:tasks.workunit.client.1.smithi171.stdout:6/307: dwrite dc/f13 [0,4194304] 0 2022-01-31T19:44:24.825 INFO:tasks.workunit.client.1.smithi171.stdout:3/354: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:24.825 INFO:tasks.workunit.client.1.smithi171.stdout:3/355: dread - d4/dd/d27/d28/d35/f5c zero size 2022-01-31T19:44:24.826 INFO:tasks.workunit.client.1.smithi171.stdout:6/308: mknod dc/c5d 0 2022-01-31T19:44:24.827 INFO:tasks.workunit.client.1.smithi171.stdout:3/356: rename d4/dd/d1a/c5a to d4/dd/d27/d2e/c65 0 2022-01-31T19:44:24.829 INFO:tasks.workunit.client.1.smithi171.stdout:6/309: mknod dc/d19/d29/d27/d37/c5e 0 2022-01-31T19:44:24.829 INFO:tasks.workunit.client.1.smithi171.stdout:6/310: chown c5 8739069 1 2022-01-31T19:44:24.829 INFO:tasks.workunit.client.1.smithi171.stdout:2/473: write d7/d9/f12 [1198898,41532] 0 2022-01-31T19:44:24.831 INFO:tasks.workunit.client.1.smithi171.stdout:3/357: creat d4/dd/f66 x:0 0 0 2022-01-31T19:44:24.832 INFO:tasks.workunit.client.1.smithi171.stdout:3/358: truncate d4/dd/f1e 4927998 0 2022-01-31T19:44:24.832 INFO:tasks.workunit.client.1.smithi171.stdout:3/359: fsync d4/fc 0 2022-01-31T19:44:24.832 INFO:tasks.workunit.client.1.smithi171.stdout:6/311: creat dc/d21/d4f/f5f x:0 0 0 2022-01-31T19:44:24.832 INFO:tasks.workunit.client.1.smithi171.stdout:2/474: mknod d7/c96 0 2022-01-31T19:44:24.833 INFO:tasks.workunit.client.1.smithi171.stdout:2/475: chown d7/dd/d5e/l6f 430503 1 2022-01-31T19:44:24.833 INFO:tasks.workunit.client.1.smithi171.stdout:3/360: mkdir d4/dd/d27/d28/d35/d67 0 2022-01-31T19:44:24.834 INFO:tasks.workunit.client.1.smithi171.stdout:3/361: fsync d4/dd/f19 0 2022-01-31T19:44:24.834 INFO:tasks.workunit.client.1.smithi171.stdout:2/476: link d7/d26/c2a d7/d55/c97 0 2022-01-31T19:44:24.835 INFO:tasks.workunit.client.1.smithi171.stdout:2/477: write d7/f3a [1229156,59795] 0 2022-01-31T19:44:24.835 INFO:tasks.workunit.client.1.smithi171.stdout:3/362: mknod d4/d21/d3a/d59/c68 0 2022-01-31T19:44:24.836 INFO:tasks.workunit.client.1.smithi171.stdout:6/312: getdents dc/d21 0 2022-01-31T19:44:24.836 INFO:tasks.workunit.client.1.smithi171.stdout:6/313: fsync dc/d18/f2a 0 2022-01-31T19:44:24.837 INFO:tasks.workunit.client.1.smithi171.stdout:2/478: truncate d7/f56 374744 0 2022-01-31T19:44:24.838 INFO:tasks.workunit.client.1.smithi171.stdout:2/479: fsync d7/f62 0 2022-01-31T19:44:24.838 INFO:tasks.workunit.client.1.smithi171.stdout:2/480: chown d7/dd/d5e/d6b/l73 2064252609 1 2022-01-31T19:44:24.839 INFO:tasks.workunit.client.1.smithi171.stdout:3/363: rename d4/d21/d3a/f3b to d4/d21/f69 0 2022-01-31T19:44:24.839 INFO:tasks.workunit.client.1.smithi171.stdout:6/314: getdents dc/d18 0 2022-01-31T19:44:24.840 INFO:tasks.workunit.client.1.smithi171.stdout:2/481: link d7/d9/d1a/d45/l28 d7/d7c/l98 0 2022-01-31T19:44:24.844 INFO:tasks.workunit.client.1.smithi171.stdout:8/315: dwrite df/d18/f1e [4194304,4194304] 0 2022-01-31T19:44:24.851 INFO:tasks.workunit.client.1.smithi171.stdout:2/482: dread d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:24.851 INFO:tasks.workunit.client.1.smithi171.stdout:3/364: truncate d4/dd/f19 4101428 0 2022-01-31T19:44:24.854 INFO:tasks.workunit.client.1.smithi171.stdout:2/483: mknod d7/c99 0 2022-01-31T19:44:24.854 INFO:tasks.workunit.client.1.smithi171.stdout:2/484: readlink d7/l33 0 2022-01-31T19:44:24.854 INFO:tasks.workunit.client.1.smithi171.stdout:2/485: chown d7/d26/f88 0 1 2022-01-31T19:44:24.868 INFO:tasks.workunit.client.1.smithi171.stdout:8/316: dwrite df/d18/f4d [0,4194304] 0 2022-01-31T19:44:24.868 INFO:tasks.workunit.client.1.smithi171.stdout:8/317: fsync df/d1a/f29 0 2022-01-31T19:44:24.877 INFO:tasks.workunit.client.1.smithi171.stdout:8/318: dread df/d1a/f3b [4194304,4194304] 0 2022-01-31T19:44:24.880 INFO:tasks.workunit.client.1.smithi171.stdout:3/365: dwrite d4/dd/d39/f50 [0,4194304] 0 2022-01-31T19:44:24.883 INFO:tasks.workunit.client.1.smithi171.stdout:2/486: rmdir d7/d9/d1a 39 2022-01-31T19:44:24.883 INFO:tasks.workunit.client.1.smithi171.stdout:2/487: chown d7/dd/f80 43033721 1 2022-01-31T19:44:24.889 INFO:tasks.workunit.client.1.smithi171.stdout:6/315: dwrite dc/d19/d30/d3c/f4b [0,4194304] 0 2022-01-31T19:44:24.889 INFO:tasks.workunit.client.1.smithi171.stdout:8/319: dread df/d18/f42 [0,4194304] 0 2022-01-31T19:44:24.890 INFO:tasks.workunit.client.1.smithi171.stdout:6/316: write dc/f4a [40554,37551] 0 2022-01-31T19:44:24.890 INFO:tasks.workunit.client.1.smithi171.stdout:6/317: creat dc/d21/d4f/f60 x:0 0 0 2022-01-31T19:44:24.891 INFO:tasks.workunit.client.1.smithi171.stdout:3/366: write d4/dd/f19 [3031290,95838] 0 2022-01-31T19:44:24.892 INFO:tasks.workunit.client.1.smithi171.stdout:3/367: stat d4/dd/d1a/d52 0 2022-01-31T19:44:24.898 INFO:tasks.workunit.client.1.smithi171.stdout:6/318: getdents dc/d19/d29/d27/d37 0 2022-01-31T19:44:24.899 INFO:tasks.workunit.client.1.smithi171.stdout:6/319: write dc/f4a [1286155,117401] 0 2022-01-31T19:44:24.900 INFO:tasks.workunit.client.1.smithi171.stdout:3/368: rename d4/dd/d27/d2e/c30 to d4/dd/c6a 0 2022-01-31T19:44:24.901 INFO:tasks.workunit.client.1.smithi171.stdout:7/253: sync 2022-01-31T19:44:24.901 INFO:tasks.workunit.client.1.smithi171.stdout:4/322: sync 2022-01-31T19:44:24.902 INFO:tasks.workunit.client.1.smithi171.stdout:4/323: stat d2/d3/d5/dc/f42 0 2022-01-31T19:44:24.902 INFO:tasks.workunit.client.1.smithi171.stdout:4/324: chown d2/d3/d5/f1a 97289398 1 2022-01-31T19:44:24.903 INFO:tasks.workunit.client.1.smithi171.stdout:8/320: truncate df/d18/d19/d21/f32 306475 0 2022-01-31T19:44:24.904 INFO:tasks.workunit.client.1.smithi171.stdout:8/321: write df/d27/f57 [1068983,7140] 0 2022-01-31T19:44:24.904 INFO:tasks.workunit.client.1.smithi171.stdout:8/322: unlink ld 0 2022-01-31T19:44:24.905 INFO:tasks.workunit.client.1.smithi171.stdout:6/320: rename dc/d18/c20 to dc/d19/d29/c61 0 2022-01-31T19:44:24.905 INFO:tasks.workunit.client.1.smithi171.stdout:6/321: chown dc/d19/d30/l35 126 1 2022-01-31T19:44:24.905 INFO:tasks.workunit.client.1.smithi171.stdout:6/322: fsync dc/d19/d29/f24 0 2022-01-31T19:44:24.905 INFO:tasks.workunit.client.1.smithi171.stdout:7/254: unlink d1/d1e/f3d 0 2022-01-31T19:44:24.906 INFO:tasks.workunit.client.1.smithi171.stdout:4/325: rmdir d2/d3/d5/d41 39 2022-01-31T19:44:24.911 INFO:tasks.workunit.client.1.smithi171.stdout:8/323: truncate df/d18/f42 3559749 0 2022-01-31T19:44:24.912 INFO:tasks.workunit.client.1.smithi171.stdout:6/323: unlink dc/l28 0 2022-01-31T19:44:24.913 INFO:tasks.workunit.client.1.smithi171.stdout:3/369: truncate d4/dd/d39/f50 35258 0 2022-01-31T19:44:24.915 INFO:tasks.workunit.client.1.smithi171.stdout:7/255: link d1/d52/l27 d1/d1e/d2a/l54 0 2022-01-31T19:44:24.916 INFO:tasks.workunit.client.1.smithi171.stdout:4/326: mkdir d2/d3/d5/d41/d6b 0 2022-01-31T19:44:24.917 INFO:tasks.workunit.client.1.smithi171.stdout:4/327: readlink d2/d3/d5/d11/d12/l20 0 2022-01-31T19:44:24.918 INFO:tasks.workunit.client.1.smithi171.stdout:4/328: write d2/f13 [1024750,62873] 0 2022-01-31T19:44:24.918 INFO:tasks.workunit.client.1.smithi171.stdout:6/324: mknod dc/d19/d29/c62 0 2022-01-31T19:44:24.921 INFO:tasks.workunit.client.1.smithi171.stdout:3/370: mkdir d4/dd/d1a/d6b 0 2022-01-31T19:44:24.922 INFO:tasks.workunit.client.1.smithi171.stdout:7/256: mknod d1/d1b/c55 0 2022-01-31T19:44:24.923 INFO:tasks.workunit.client.1.smithi171.stdout:7/257: chown d1/d1e/d2a/f3b 194 1 2022-01-31T19:44:24.923 INFO:tasks.workunit.client.1.smithi171.stdout:8/324: dread fb [0,4194304] 0 2022-01-31T19:44:24.926 INFO:tasks.workunit.client.1.smithi171.stdout:4/329: rename d2/d3/d5/d11/d36/c5e to d2/d3/d5/dc/d4b/c6c 0 2022-01-31T19:44:24.928 INFO:tasks.workunit.client.1.smithi171.stdout:7/258: creat d1/d1e/d2a/d29/d31/d30/d3c/f56 x:0 0 0 2022-01-31T19:44:24.928 INFO:tasks.workunit.client.1.smithi171.stdout:7/259: dread - d1/d52/f18 zero size 2022-01-31T19:44:24.929 INFO:tasks.workunit.client.1.smithi171.stdout:7/260: fsync d1/d1b/f4c 0 2022-01-31T19:44:24.929 INFO:tasks.workunit.client.1.smithi171.stdout:8/325: mknod df/d5d/d6c/c72 0 2022-01-31T19:44:24.931 INFO:tasks.workunit.client.1.smithi171.stdout:3/371: rename d4/d21/d3a/d64 to d4/dd/d27/d28/d35/d55/d6c 0 2022-01-31T19:44:24.932 INFO:tasks.workunit.client.1.smithi171.stdout:3/372: chown d4/d21/d3a/l44 623327586 1 2022-01-31T19:44:24.932 INFO:tasks.workunit.client.1.smithi171.stdout:3/373: readlink d4/dd/l33 0 2022-01-31T19:44:24.933 INFO:tasks.workunit.client.1.smithi171.stdout:8/326: rename df/d18/d19/d36/f52 to df/d18/d19/d36/f73 0 2022-01-31T19:44:24.936 INFO:tasks.workunit.client.1.smithi171.stdout:3/374: mknod d4/dd/d1a/c6d 0 2022-01-31T19:44:24.942 INFO:tasks.workunit.client.1.smithi171.stdout:6/325: dwrite fa [0,4194304] 0 2022-01-31T19:44:24.947 INFO:tasks.workunit.client.1.smithi171.stdout:6/326: creat dc/d19/d29/f63 x:0 0 0 2022-01-31T19:44:24.948 INFO:tasks.workunit.client.1.smithi171.stdout:6/327: dread - dc/d21/d4f/f60 zero size 2022-01-31T19:44:24.951 INFO:tasks.workunit.client.1.smithi171.stdout:4/330: rmdir d2/d3/d5/d11/d36 0 2022-01-31T19:44:24.952 INFO:tasks.workunit.client.1.smithi171.stdout:4/331: creat d2/f6d x:0 0 0 2022-01-31T19:44:24.952 INFO:tasks.workunit.client.1.smithi171.stdout:4/332: creat d2/d3/d5/dc/d4b/f6e x:0 0 0 2022-01-31T19:44:24.952 INFO:tasks.workunit.client.1.smithi171.stdout:4/333: dread - d2/d3/d5/d11/d12/f60 zero size 2022-01-31T19:44:24.952 INFO:tasks.workunit.client.1.smithi171.stdout:4/334: creat d2/d3/d5/dc/f6f x:0 0 0 2022-01-31T19:44:24.953 INFO:tasks.workunit.client.1.smithi171.stdout:6/328: symlink dc/d19/d30/l64 0 2022-01-31T19:44:24.955 INFO:tasks.workunit.client.1.smithi171.stdout:3/375: dwrite d4/dd/d1a/f1b [4194304,4194304] 0 2022-01-31T19:44:24.957 INFO:tasks.workunit.client.1.smithi171.stdout:4/335: mkdir d2/d3/d5/d70 0 2022-01-31T19:44:24.957 INFO:tasks.workunit.client.1.smithi171.stdout:7/261: dwrite d1/f36 [0,4194304] 0 2022-01-31T19:44:24.958 INFO:tasks.workunit.client.1.smithi171.stdout:8/327: dread df/d27/f57 [0,4194304] 0 2022-01-31T19:44:24.958 INFO:tasks.workunit.client.1.smithi171.stdout:8/328: write df/d18/d19/f5f [4360563,102213] 0 2022-01-31T19:44:24.958 INFO:tasks.workunit.client.1.smithi171.stdout:6/329: rmdir dc/d21 39 2022-01-31T19:44:24.960 INFO:tasks.workunit.client.1.smithi171.stdout:3/376: mkdir d4/dd/d27/d6e 0 2022-01-31T19:44:24.961 INFO:tasks.workunit.client.1.smithi171.stdout:4/336: mknod d2/d3/d5/c71 0 2022-01-31T19:44:24.964 INFO:tasks.workunit.client.1.smithi171.stdout:7/262: dread - d1/d1e/d2a/d29/d31/d30/d3c/f45 zero size 2022-01-31T19:44:24.964 INFO:tasks.workunit.client.1.smithi171.stdout:7/263: fsync d1/d52/f28 0 2022-01-31T19:44:24.964 INFO:tasks.workunit.client.1.smithi171.stdout:7/264: write d1/d1e/d2a/d29/d31/d30/d3c/f45 [108545,32591] 0 2022-01-31T19:44:24.965 INFO:tasks.workunit.client.1.smithi171.stdout:8/329: symlink df/l74 0 2022-01-31T19:44:24.965 INFO:tasks.workunit.client.1.smithi171.stdout:3/377: symlink d4/dd/d27/d2e/l6f 0 2022-01-31T19:44:24.967 INFO:tasks.workunit.client.1.smithi171.stdout:4/337: getdents d2/d3/d5/d41 0 2022-01-31T19:44:24.967 INFO:tasks.workunit.client.1.smithi171.stdout:4/338: truncate d2/f13 2401503 0 2022-01-31T19:44:24.971 INFO:tasks.workunit.client.1.smithi171.stdout:1/297: sync 2022-01-31T19:44:24.971 INFO:tasks.workunit.client.1.smithi171.stdout:1/298: truncate dd/df/f38 370868 0 2022-01-31T19:44:24.972 INFO:tasks.workunit.client.1.smithi171.stdout:7/265: mkdir d1/d26/d57 0 2022-01-31T19:44:24.972 INFO:tasks.workunit.client.1.smithi171.stdout:8/330: symlink df/d1a/l75 0 2022-01-31T19:44:24.974 INFO:tasks.workunit.client.1.smithi171.stdout:1/299: symlink dd/df/l61 0 2022-01-31T19:44:24.974 INFO:tasks.workunit.client.1.smithi171.stdout:1/300: write dd/df/d19/d1b/f28 [859293,87457] 0 2022-01-31T19:44:24.975 INFO:tasks.workunit.client.1.smithi171.stdout:9/418: sync 2022-01-31T19:44:24.975 INFO:tasks.workunit.client.1.smithi171.stdout:0/370: sync 2022-01-31T19:44:24.975 INFO:tasks.workunit.client.1.smithi171.stdout:0/371: stat d9/db/dc/d3f/f34 0 2022-01-31T19:44:24.976 INFO:tasks.workunit.client.1.smithi171.stdout:7/266: mkdir d1/d1e/d2a/d58 0 2022-01-31T19:44:24.977 INFO:tasks.workunit.client.1.smithi171.stdout:8/331: creat df/d5d/d6c/f76 x:0 0 0 2022-01-31T19:44:24.978 INFO:tasks.workunit.client.1.smithi171.stdout:9/419: getdents dd 0 2022-01-31T19:44:24.978 INFO:tasks.workunit.client.1.smithi171.stdout:7/267: symlink d1/d26/l59 0 2022-01-31T19:44:24.980 INFO:tasks.workunit.client.1.smithi171.stdout:8/332: symlink df/l77 0 2022-01-31T19:44:24.980 INFO:tasks.workunit.client.1.smithi171.stdout:8/333: stat df/d27/d5c/l66 0 2022-01-31T19:44:24.980 INFO:tasks.workunit.client.1.smithi171.stdout:8/334: write df/d18/d19/d36/f73 [557309,76510] 0 2022-01-31T19:44:24.981 INFO:tasks.workunit.client.1.smithi171.stdout:9/420: mknod dd/c8b 0 2022-01-31T19:44:24.982 INFO:tasks.workunit.client.1.smithi171.stdout:9/421: rename dd/df/d15/d38/d4d/c57 to dd/df/d15/c8c 0 2022-01-31T19:44:24.983 INFO:tasks.workunit.client.1.smithi171.stdout:8/335: dread df/f16 [0,4194304] 0 2022-01-31T19:44:24.984 INFO:tasks.workunit.client.1.smithi171.stdout:8/336: write df/d5d/d6c/f6e [622544,88303] 0 2022-01-31T19:44:24.986 INFO:tasks.workunit.client.1.smithi171.stdout:1/301: dread dd/df/d19/f27 [0,4194304] 0 2022-01-31T19:44:24.987 INFO:tasks.workunit.client.1.smithi171.stdout:8/337: dread f3 [0,4194304] 0 2022-01-31T19:44:24.988 INFO:tasks.workunit.client.1.smithi171.stdout:1/302: symlink dd/d23/d34/l62 0 2022-01-31T19:44:24.989 INFO:tasks.workunit.client.1.smithi171.stdout:1/303: creat dd/df/d19/d1b/f63 x:0 0 0 2022-01-31T19:44:24.989 INFO:tasks.workunit.client.1.smithi171.stdout:1/304: write dd/d23/d2e/d5e/f20 [4812602,123514] 0 2022-01-31T19:44:24.989 INFO:tasks.workunit.client.1.smithi171.stdout:1/305: chown dd/df/c33 1343 1 2022-01-31T19:44:24.989 INFO:tasks.workunit.client.1.smithi171.stdout:1/306: readlink dd/d23/l4f 0 2022-01-31T19:44:24.990 INFO:tasks.workunit.client.1.smithi171.stdout:3/378: dwrite d4/dd/d40/f42 [0,4194304] 0 2022-01-31T19:44:24.990 INFO:tasks.workunit.client.1.smithi171.stdout:3/379: readlink d4/d21/d3a/l5b 0 2022-01-31T19:44:24.990 INFO:tasks.workunit.client.1.smithi171.stdout:3/380: fsync d4/dd/d27/d28/d35/f5c 0 2022-01-31T19:44:24.991 INFO:tasks.workunit.client.1.smithi171.stdout:3/381: getdents d4/dd/d4b 0 2022-01-31T19:44:24.992 INFO:tasks.workunit.client.1.smithi171.stdout:3/382: rename d4/d21/f2a to d4/d21/d3a/d59/f70 0 2022-01-31T19:44:24.992 INFO:tasks.workunit.client.1.smithi171.stdout:3/383: creat d4/dd/d39/d51/f71 x:0 0 0 2022-01-31T19:44:24.993 INFO:tasks.workunit.client.1.smithi171.stdout:3/384: rmdir d4/dd/d27/d28/d35 39 2022-01-31T19:44:24.994 INFO:tasks.workunit.client.1.smithi171.stdout:3/385: symlink d4/d21/l72 0 2022-01-31T19:44:24.995 INFO:tasks.workunit.client.1.smithi171.stdout:3/386: chown d4/dd/c36 69 1 2022-01-31T19:44:24.995 INFO:tasks.workunit.client.1.smithi171.stdout:3/387: chown d4/dd/c6a 0 1 2022-01-31T19:44:24.995 INFO:tasks.workunit.client.1.smithi171.stdout:3/388: dread - d4/dd/f66 zero size 2022-01-31T19:44:24.996 INFO:tasks.workunit.client.1.smithi171.stdout:3/389: truncate d4/dd/d27/d28/d35/d55/f5e 917367 0 2022-01-31T19:44:24.996 INFO:tasks.workunit.client.1.smithi171.stdout:3/390: dread - d4/dd/d39/d51/f53 zero size 2022-01-31T19:44:24.997 INFO:tasks.workunit.client.1.smithi171.stdout:3/391: mkdir d4/dd/d73 0 2022-01-31T19:44:24.997 INFO:tasks.workunit.client.1.smithi171.stdout:3/392: fdatasync d4/dd/f66 0 2022-01-31T19:44:24.997 INFO:tasks.workunit.client.1.smithi171.stdout:3/393: stat d4/d21/d3a/f4f 0 2022-01-31T19:44:24.997 INFO:tasks.workunit.client.1.smithi171.stdout:3/394: creat d4/dd/d27/d28/d35/d55/f74 x:0 0 0 2022-01-31T19:44:24.998 INFO:tasks.workunit.client.1.smithi171.stdout:3/395: mknod d4/dd/d27/d6e/c75 0 2022-01-31T19:44:24.998 INFO:tasks.workunit.client.1.smithi171.stdout:3/396: dread - d4/d21/d3a/f4f zero size 2022-01-31T19:44:24.999 INFO:tasks.workunit.client.1.smithi171.stdout:8/338: read f7 [82734,103931] 0 2022-01-31T19:44:24.999 INFO:tasks.workunit.client.1.smithi171.stdout:8/339: chown df/d18/d19/d46/c60 58 1 2022-01-31T19:44:25.003 INFO:tasks.workunit.client.1.smithi171.stdout:3/397: rename d4/dd/d40/l56 to d4/dd/l76 0 2022-01-31T19:44:25.004 INFO:tasks.workunit.client.1.smithi171.stdout:3/398: stat d4/dd/d39/d51 0 2022-01-31T19:44:25.004 INFO:tasks.workunit.client.1.smithi171.stdout:1/307: dread f9 [0,4194304] 0 2022-01-31T19:44:25.006 INFO:tasks.workunit.client.1.smithi171.stdout:3/399: creat d4/dd/d73/f77 x:0 0 0 2022-01-31T19:44:25.006 INFO:tasks.workunit.client.1.smithi171.stdout:3/400: chown d4/dd/d1a/f1b 2135177 1 2022-01-31T19:44:25.006 INFO:tasks.workunit.client.1.smithi171.stdout:9/422: dwrite dd/d1c/f75 [0,4194304] 0 2022-01-31T19:44:25.008 INFO:tasks.workunit.client.1.smithi171.stdout:3/401: creat d4/dd/d27/d2e/f78 x:0 0 0 2022-01-31T19:44:25.009 INFO:tasks.workunit.client.1.smithi171.stdout:3/402: fdatasync d4/dd/d40/f42 0 2022-01-31T19:44:25.011 INFO:tasks.workunit.client.1.smithi171.stdout:9/423: symlink dd/df/d15/d1a/d64/d6e/l8d 0 2022-01-31T19:44:25.011 INFO:tasks.workunit.client.1.smithi171.stdout:9/424: chown dd/df/f72 6034 1 2022-01-31T19:44:25.014 INFO:tasks.workunit.client.1.smithi171.stdout:3/403: link d4/dd/d27/d2e/c5f d4/dd/d73/c79 0 2022-01-31T19:44:25.014 INFO:tasks.workunit.client.1.smithi171.stdout:3/404: stat d4/dd/d73 0 2022-01-31T19:44:25.014 INFO:tasks.workunit.client.1.smithi171.stdout:3/405: readlink d4/d21/d3a/l5b 0 2022-01-31T19:44:25.018 INFO:tasks.workunit.client.1.smithi171.stdout:9/425: unlink dd/df/d15/d53/l87 0 2022-01-31T19:44:25.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: cephadm 2022-01-31T 2022-01-31T19:44:25.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: 19:44:22.969005+0000 mgr.smithi171.asyxnc (mgr.24387) 24 : cephadm [INF] Adjusting osd_memory_target on smithi167 to 1584M 2022-01-31T19:44:25.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:22.981130+0000 mgr.smithi171.asyxnc (mgr.24387) 25 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:44:25.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:23.259160+0000 mgr.smithi171.asyxnc (mgr.24387) 26 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:25.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:23.338346+0000 mgr.smithi171.asyxnc (mgr.24387) 27 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 2608M 2022-01-31T19:44:25.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:23.343824+0000 mgr.smithi171.asyxnc (mgr.24387) 28 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2022-01-31T19:44:25.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:23.733692+0000 mgr.smithi171.asyxnc (mgr.24387) 29 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:25.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.237219+0000 mon.smithi167 (mon.0) 678 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.241624+0000 mon.smithi167 (mon.0) 679 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.246013+0000 mon.smithi167 (mon.0) 680 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.252614+0000 mon.smithi167 (mon.0) 681 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.258766+0000 mon.smithi167 (mon.0) 682 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.265912+0000 mon.smithi171 (mon.1) 59 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:25.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:24 smithi167 conmon[32206]: audit 2022-01-31T19:44:24.272588+0000 mon.smithi167 (mon.0) 683 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.024 INFO:tasks.workunit.client.1.smithi171.stdout:1/308: rename dd/d23/d34/d50 to dd/df/d19/d1b/d1e/d64 0 2022-01-31T19:44:25.024 INFO:tasks.workunit.client.1.smithi171.stdout:1/309: write dd/df/f3a [890237,113050] 0 2022-01-31T19:44:25.024 INFO:tasks.workunit.client.1.smithi171.stdout:1/310: link dd/df/f3a dd/d23/d34/f65 0 2022-01-31T19:44:25.025 INFO:tasks.workunit.client.1.smithi171.stdout:1/311: stat dd/f56 0 2022-01-31T19:44:25.025 INFO:tasks.workunit.client.1.smithi171.stdout:8/340: dwrite df/d2a/d38/f56 [0,4194304] 0 2022-01-31T19:44:25.026 INFO:tasks.workunit.client.1.smithi171.stdout:8/341: creat df/d18/d19/d46/f78 x:0 0 0 2022-01-31T19:44:25.026 INFO:tasks.workunit.client.1.smithi171.stdout:8/342: fsync f1 0 2022-01-31T19:44:25.029 INFO:tasks.workunit.client.1.smithi171.stdout:8/343: mkdir df/d18/d79 0 2022-01-31T19:44:25.030 INFO:tasks.workunit.client.1.smithi171.stdout:8/344: mkdir df/d18/d79/d7a 0 2022-01-31T19:44:25.030 INFO:tasks.workunit.client.1.smithi171.stdout:3/406: dread d4/fb [4194304,4194304] 0 2022-01-31T19:44:25.031 INFO:tasks.workunit.client.1.smithi171.stdout:8/345: truncate f6 5598644 0 2022-01-31T19:44:25.031 INFO:tasks.workunit.client.1.smithi171.stdout:8/346: chown df/d18/d19/d46/f78 0 1 2022-01-31T19:44:25.034 INFO:tasks.workunit.client.1.smithi171.stdout:8/347: write df/f17 [3910762,62066] 0 2022-01-31T19:44:25.038 INFO:tasks.workunit.client.1.smithi171.stdout:3/407: write d4/dd/f19 [3890897,12571] 0 2022-01-31T19:44:25.038 INFO:tasks.workunit.client.1.smithi171.stdout:8/348: read df/d27/f64 [7665458,6519] 0 2022-01-31T19:44:25.039 INFO:tasks.workunit.client.1.smithi171.stdout:8/349: truncate df/d2a/d38/f43 2546854 0 2022-01-31T19:44:25.040 INFO:tasks.workunit.client.1.smithi171.stdout:3/408: mkdir d4/dd/d1a/d52/d7a 0 2022-01-31T19:44:25.040 INFO:tasks.workunit.client.1.smithi171.stdout:9/426: dwrite dd/df/f32 [0,4194304] 0 2022-01-31T19:44:25.042 INFO:tasks.workunit.client.1.smithi171.stdout:8/350: write df/d18/d19/d21/f26 [542853,37345] 0 2022-01-31T19:44:25.044 INFO:tasks.workunit.client.1.smithi171.stdout:1/312: dwrite dd/df/d19/f47 [0,4194304] 0 2022-01-31T19:44:25.044 INFO:tasks.workunit.client.1.smithi171.stdout:1/313: chown dd/d23/c3d 1885 1 2022-01-31T19:44:25.044 INFO:tasks.workunit.client.1.smithi171.stdout:9/427: chown dd/f74 0 1 2022-01-31T19:44:25.045 INFO:tasks.workunit.client.1.smithi171.stdout:9/428: write dd/d1c/f29 [832097,21998] 0 2022-01-31T19:44:25.046 INFO:tasks.workunit.client.1.smithi171.stdout:9/429: chown dd/df/d15/f16 695947 1 2022-01-31T19:44:25.053 INFO:tasks.workunit.client.1.smithi171.stdout:1/314: rename dd/d23/d2e/d5e/f5f to dd/df/d19/d1b/f66 0 2022-01-31T19:44:25.056 INFO:tasks.workunit.client.1.smithi171.stdout:9/430: mknod dd/df/c8e 0 2022-01-31T19:44:25.056 INFO:tasks.workunit.client.1.smithi171.stdout:9/431: write dd/df/f25 [1339033,72022] 0 2022-01-31T19:44:25.057 INFO:tasks.workunit.client.1.smithi171.stdout:1/315: rename dd/d23/l3c to dd/df/d19/d1b/d1e/d64/l67 0 2022-01-31T19:44:25.058 INFO:tasks.workunit.client.1.smithi171.stdout:9/432: rename dd/df/d15/d38/c68 to dd/df/d15/d1a/c8f 0 2022-01-31T19:44:25.059 INFO:tasks.workunit.client.1.smithi171.stdout:1/316: creat dd/d23/d2e/f68 x:0 0 0 2022-01-31T19:44:25.061 INFO:tasks.workunit.client.1.smithi171.stdout:1/317: rename dd/d23/d34/d4b/c55 to dd/df/c69 0 2022-01-31T19:44:25.063 INFO:tasks.workunit.client.1.smithi171.stdout:8/351: dwrite df/d2a/f40 [0,4194304] 0 2022-01-31T19:44:25.063 INFO:tasks.workunit.client.1.smithi171.stdout:3/409: dwrite d4/dd/f1e [0,4194304] 0 2022-01-31T19:44:25.071 INFO:tasks.workunit.client.1.smithi171.stdout:1/318: creat dd/df/d19/d1b/d1e/f6a x:0 0 0 2022-01-31T19:44:25.071 INFO:tasks.workunit.client.1.smithi171.stdout:1/319: write dd/f42 [777664,87488] 0 2022-01-31T19:44:25.071 INFO:tasks.workunit.client.1.smithi171.stdout:9/433: rmdir dd/df 39 2022-01-31T19:44:25.074 INFO:tasks.workunit.client.1.smithi171.stdout:8/352: mkdir df/d27/d7b 0 2022-01-31T19:44:25.075 INFO:tasks.workunit.client.1.smithi171.stdout:3/410: truncate d4/d21/f69 3242786 0 2022-01-31T19:44:25.077 INFO:tasks.workunit.client.1.smithi171.stdout:1/320: mknod dd/df/d19/d36/c6b 0 2022-01-31T19:44:25.077 INFO:tasks.workunit.client.1.smithi171.stdout:3/411: write d4/dd/d27/d28/d35/f38 [325316,16137] 0 2022-01-31T19:44:25.091 INFO:tasks.workunit.client.1.smithi171.stdout:8/353: rename df/d18/d19/d46/f69 to df/d18/d79/f7c 0 2022-01-31T19:44:25.094 INFO:tasks.workunit.client.1.smithi171.stdout:1/321: symlink dd/df/l6c 0 2022-01-31T19:44:25.097 INFO:tasks.workunit.client.1.smithi171.stdout:3/412: dwrite d4/dd/d27/d28/d35/f48 [0,4194304] 0 2022-01-31T19:44:25.098 INFO:tasks.workunit.client.1.smithi171.stdout:3/413: creat d4/f7b x:0 0 0 2022-01-31T19:44:25.104 INFO:tasks.workunit.client.1.smithi171.stdout:9/434: dwrite dd/f81 [0,4194304] 0 2022-01-31T19:44:25.104 INFO:tasks.workunit.client.1.smithi171.stdout:1/322: dread dd/df/f3a [0,4194304] 0 2022-01-31T19:44:25.105 INFO:tasks.workunit.client.1.smithi171.stdout:9/435: dread - dd/f74 zero size 2022-01-31T19:44:25.105 INFO:tasks.workunit.client.1.smithi171.stdout:9/436: readlink dd/df/d15/d1a/d64/d6e/l71 0 2022-01-31T19:44:25.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:22.969005+0000 2022-01-31T19:44:25.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: mgr.smithi171.asyxnc (mgr.24387) 24 : cephadm [INF] Adjusting osd_memory_target on smithi167 to 1584M 2022-01-31T19:44:25.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: cephadm 2022-01-31 2022-01-31T19:44:25.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: T19:44:22.981130+0000 mgr.smithi171.asyxnc (mgr.24387) 25 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:44:25.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: cephadm 2022- 2022-01-31T19:44:25.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 01-31T19:44:23.259160+0000 mgr.smithi171.asyxnc ( 2022-01-31T19:44:25.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: mgr.24387) 26 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:25.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 2022-01-31T19:44:25.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:23. 2022-01-31T19:44:25.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 338346+0000 mgr.smithi171.asyxnc (mgr.24387) 27 : cephadm 2022-01-31T19:44:25.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: [INF] Adjusting osd_memory_target on smithi171 to 2608M 2022-01-31T19:44:25.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:25.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 2022-01-31T19:44: 2022-01-31T19:44:25.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 23.343824+0000 mgr.smithi171.asyxnc (mgr.24387) 28 : cephadm 2022-01-31T19:44:25.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: [INF] Updating smithi171:/etc/ceph/ceph.conf 2022-01-31T19:44:25.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: cephadm 2022-01-31T19: 2022-01-31T19:44:25.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 44:23.733692+0000 mgr.smithi171.asyxnc (mgr.24387) 29 : cephadm 2022-01-31T19:44:25.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:25.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022-01-31T19:44:24 2022-01-31T19:44:25.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: .237219+0000 mon.smithi167 (mon.0) 678 : audit 2022-01-31T19:44:25.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022-01-31T19:44:25.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 2022-01-31T19: 2022-01-31T19:44:25.113 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 44:24.241624+0000 mon.smithi167 2022-01-31T19:44:25.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: (mon.0) 679 : audit [INF] 2022-01-31T19:44:25.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:25.114 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: -01-31T19:44:24 2022-01-31T19:44:25.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: .246013+0000 mon.smithi167 (mon.0 2022-01-31T19:44:25.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: ) 680 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:25.115 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 01-31T19:44:24.252614+0000 2022-01-31T19:44:25.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: mon.smithi167 (mon.0) 681 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 2022-01-31T19:44:25.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:44:25.116 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: -31T19:44:24.258766+0000 mon.smithi167 2022-01-31T19:44:25.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: (mon.0) 682 : audit 2022-01-31T19:44:25.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022-01-31T19:44:25.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 2022-01-31T19:44 2022-01-31T19:44:25.117 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: :24.265912+0000 mon.smithi171 ( 2022-01-31T19:44:25.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: mon.1) 59 : audit [DBG] 2022-01-31T19:44:25.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:25.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: audit 2022 2022-01-31T19:44:25.118 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: -01-31T19:44: 2022-01-31T19:44:25.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 24.272588+0000 mon.smithi167 (mon.0) 2022-01-31T19:44:25.119 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:24 smithi171 conmon[35325]: 683 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.132 INFO:tasks.workunit.client.1.smithi171.stdout:5/338: sync 2022-01-31T19:44:25.133 INFO:tasks.workunit.client.1.smithi171.stdout:2/488: sync 2022-01-31T19:44:25.133 INFO:tasks.workunit.client.1.smithi171.stdout:2/489: dread - d7/dd/d5e/f86 zero size 2022-01-31T19:44:25.133 INFO:tasks.workunit.client.1.smithi171.stdout:2/490: fdatasync d7/f3a 0 2022-01-31T19:44:25.134 INFO:tasks.workunit.client.1.smithi171.stdout:5/339: dread d5/d6/f21 [0,4194304] 0 2022-01-31T19:44:25.136 INFO:tasks.workunit.client.1.smithi171.stdout:1/323: unlink dd/df/f3a 0 2022-01-31T19:44:25.137 INFO:tasks.workunit.client.1.smithi171.stdout:1/324: chown dd/df/d19/d1b/f63 3 1 2022-01-31T19:44:25.137 INFO:tasks.workunit.client.1.smithi171.stdout:1/325: readlink dd/d23/d34/l62 0 2022-01-31T19:44:25.138 INFO:tasks.workunit.client.1.smithi171.stdout:9/437: creat dd/df/d15/d38/d4d/f90 x:0 0 0 2022-01-31T19:44:25.139 INFO:tasks.workunit.client.1.smithi171.stdout:1/326: dread dd/df/f12 [0,4194304] 0 2022-01-31T19:44:25.140 INFO:tasks.workunit.client.1.smithi171.stdout:2/491: rename d7/d7c/l98 to d7/d9/l9a 0 2022-01-31T19:44:25.140 INFO:tasks.workunit.client.1.smithi171.stdout:5/340: mknod d5/d6/d15/d23/d58/c71 0 2022-01-31T19:44:25.143 INFO:tasks.workunit.client.1.smithi171.stdout:9/438: creat dd/df/d15/d1a/d84/f91 x:0 0 0 2022-01-31T19:44:25.143 INFO:tasks.workunit.client.1.smithi171.stdout:9/439: truncate dd/df/d15/d38/d41/f78 109665 0 2022-01-31T19:44:25.144 INFO:tasks.workunit.client.1.smithi171.stdout:1/327: creat dd/df/f6d x:0 0 0 2022-01-31T19:44:25.144 INFO:tasks.workunit.client.1.smithi171.stdout:2/492: creat d7/dd/d5e/d6b/d81/f9b x:0 0 0 2022-01-31T19:44:25.145 INFO:tasks.workunit.client.1.smithi171.stdout:5/341: mknod d5/d6/dd/d66/c72 0 2022-01-31T19:44:25.146 INFO:tasks.workunit.client.1.smithi171.stdout:9/440: chown dd/l18 3 1 2022-01-31T19:44:25.147 INFO:tasks.workunit.client.1.smithi171.stdout:2/493: rename d7/d26/d3d/d40/f51 to d7/dd/d5e/f9c 0 2022-01-31T19:44:25.149 INFO:tasks.workunit.client.1.smithi171.stdout:5/342: creat d5/d6/d15/d5d/d6a/f73 x:0 0 0 2022-01-31T19:44:25.149 INFO:tasks.workunit.client.1.smithi171.stdout:5/343: read d5/d6/d15/d23/d67/f39 [2133744,56310] 0 2022-01-31T19:44:25.150 INFO:tasks.workunit.client.1.smithi171.stdout:5/344: getdents d5/d6/d15/d5d/d6f 0 2022-01-31T19:44:25.151 INFO:tasks.workunit.client.1.smithi171.stdout:2/494: unlink d7/dd/f4c 0 2022-01-31T19:44:25.151 INFO:tasks.workunit.client.1.smithi171.stdout:5/345: dread d5/d6/dd/f35 [0,4194304] 0 2022-01-31T19:44:25.151 INFO:tasks.workunit.client.1.smithi171.stdout:9/441: mknod dd/c92 0 2022-01-31T19:44:25.152 INFO:tasks.workunit.client.1.smithi171.stdout:2/495: creat d7/d9/d1a/d45/f9d x:0 0 0 2022-01-31T19:44:25.154 INFO:tasks.workunit.client.1.smithi171.stdout:5/346: unlink d5/d6/d15/f6d 0 2022-01-31T19:44:25.156 INFO:tasks.workunit.client.1.smithi171.stdout:9/442: mknod dd/df/d15/d38/c93 0 2022-01-31T19:44:25.157 INFO:tasks.workunit.client.1.smithi171.stdout:9/443: mknod dd/d1c/d7c/d7d/c94 0 2022-01-31T19:44:25.160 INFO:tasks.workunit.client.1.smithi171.stdout:5/347: write d5/d6/dd/f19 [3698868,98172] 0 2022-01-31T19:44:25.169 INFO:tasks.workunit.client.1.smithi171.stdout:5/348: write d5/d6/d15/f1d [1268905,117670] 0 2022-01-31T19:44:25.170 INFO:tasks.workunit.client.1.smithi171.stdout:5/349: rmdir d5/d6/dd/d66 39 2022-01-31T19:44:25.171 INFO:tasks.workunit.client.1.smithi171.stdout:1/328: dwrite f7 [0,4194304] 0 2022-01-31T19:44:25.173 INFO:tasks.workunit.client.1.smithi171.stdout:5/350: mkdir d5/d6/d74 0 2022-01-31T19:44:25.173 INFO:tasks.workunit.client.1.smithi171.stdout:5/351: readlink d5/l43 0 2022-01-31T19:44:25.173 INFO:tasks.workunit.client.1.smithi171.stdout:5/352: stat d5/d6/d15 0 2022-01-31T19:44:25.174 INFO:tasks.workunit.client.1.smithi171.stdout:5/353: fsync d5/f4e 0 2022-01-31T19:44:25.175 INFO:tasks.workunit.client.1.smithi171.stdout:5/354: symlink d5/d6/d15/d23/d6b/l75 0 2022-01-31T19:44:25.175 INFO:tasks.workunit.client.1.smithi171.stdout:5/355: readlink d5/l4a 0 2022-01-31T19:44:25.175 INFO:tasks.workunit.client.1.smithi171.stdout:5/356: dread - d5/d6/d15/d23/d3d/f51 zero size 2022-01-31T19:44:25.177 INFO:tasks.workunit.client.1.smithi171.stdout:2/496: dwrite d7/f56 [0,4194304] 0 2022-01-31T19:44:25.177 INFO:tasks.workunit.client.1.smithi171.stdout:5/357: creat d5/d6/d15/d23/d67/f76 x:0 0 0 2022-01-31T19:44:25.177 INFO:tasks.workunit.client.1.smithi171.stdout:5/358: fdatasync d5/d6/d15/d23/d67/f39 0 2022-01-31T19:44:25.177 INFO:tasks.workunit.client.1.smithi171.stdout:5/359: creat d5/d6/d15/d5d/d6a/f77 x:0 0 0 2022-01-31T19:44:25.178 INFO:tasks.workunit.client.1.smithi171.stdout:9/444: dwrite dd/f79 [0,4194304] 0 2022-01-31T19:44:25.178 INFO:tasks.workunit.client.1.smithi171.stdout:9/445: creat dd/d1c/f95 x:0 0 0 2022-01-31T19:44:25.178 INFO:tasks.workunit.client.1.smithi171.stdout:5/360: creat d5/d6/d74/f78 x:0 0 0 2022-01-31T19:44:25.180 INFO:tasks.workunit.client.1.smithi171.stdout:5/361: read d5/d6/d15/d23/d67/f5c [682149,28004] 0 2022-01-31T19:44:25.181 INFO:tasks.workunit.client.1.smithi171.stdout:5/362: chown d5/d6/dd/f45 58 1 2022-01-31T19:44:25.183 INFO:tasks.workunit.client.1.smithi171.stdout:2/497: write d7/d9/d1a/f53 [4717126,11204] 0 2022-01-31T19:44:25.188 INFO:tasks.workunit.client.1.smithi171.stdout:9/446: dread dd/df/f21 [0,4194304] 0 2022-01-31T19:44:25.193 INFO:tasks.workunit.client.1.smithi171.stdout:2/498: creat d7/dd/d93/f9e x:0 0 0 2022-01-31T19:44:25.193 INFO:tasks.workunit.client.1.smithi171.stdout:2/499: chown d7/d7c/d83/d85/c8f 12 1 2022-01-31T19:44:25.194 INFO:tasks.workunit.client.1.smithi171.stdout:2/500: truncate d7/d26/d3d/d40/f6c 35759 0 2022-01-31T19:44:25.194 INFO:tasks.workunit.client.1.smithi171.stdout:2/501: write d7/d9/f74 [991885,127148] 0 2022-01-31T19:44:25.195 INFO:tasks.workunit.client.1.smithi171.stdout:9/447: mkdir dd/df/d3f/d73/d96 0 2022-01-31T19:44:25.198 INFO:tasks.workunit.client.1.smithi171.stdout:9/448: write dd/df/d15/f36 [1111432,114235] 0 2022-01-31T19:44:25.199 INFO:tasks.workunit.client.1.smithi171.stdout:1/329: dwrite dd/df/d19/d36/f5b [0,4194304] 0 2022-01-31T19:44:25.201 INFO:tasks.workunit.client.1.smithi171.stdout:9/449: link dd/f37 dd/df/d15/d38/d41/f97 0 2022-01-31T19:44:25.204 INFO:tasks.workunit.client.1.smithi171.stdout:9/450: chown dd/c48 12585902 1 2022-01-31T19:44:25.204 INFO:tasks.workunit.client.1.smithi171.stdout:9/451: chown dd/df/d15/d1a/d64/f67 4019 1 2022-01-31T19:44:25.204 INFO:tasks.workunit.client.1.smithi171.stdout:9/452: truncate dd/df/f72 103297 0 2022-01-31T19:44:25.205 INFO:tasks.workunit.client.1.smithi171.stdout:9/453: mknod dd/df/d15/d1a/d64/c98 0 2022-01-31T19:44:25.208 INFO:tasks.workunit.client.1.smithi171.stdout:2/502: dwrite d7/d9/d1a/f89 [0,4194304] 0 2022-01-31T19:44:25.211 INFO:tasks.workunit.client.1.smithi171.stdout:2/503: getdents d7/dd/d5e/d6b/d81 0 2022-01-31T19:44:25.213 INFO:tasks.workunit.client.1.smithi171.stdout:2/504: mkdir d7/d26/d63/d9f 0 2022-01-31T19:44:25.213 INFO:tasks.workunit.client.1.smithi171.stdout:2/505: symlink d7/d26/d63/d9f/la0 0 2022-01-31T19:44:25.214 INFO:tasks.workunit.client.1.smithi171.stdout:2/506: creat d7/dd/fa1 x:0 0 0 2022-01-31T19:44:25.215 INFO:tasks.workunit.client.1.smithi171.stdout:2/507: mkdir d7/dd/d5e/d6b/d81/d8d/da2 0 2022-01-31T19:44:25.223 INFO:tasks.workunit.client.1.smithi171.stdout:1/330: dwrite f9 [0,4194304] 0 2022-01-31T19:44:25.223 INFO:tasks.workunit.client.1.smithi171.stdout:9/454: dwrite fc [0,4194304] 0 2022-01-31T19:44:25.228 INFO:tasks.workunit.client.1.smithi171.stdout:2/508: write d7/d9/d1a/d45/f2f [1331299,81414] 0 2022-01-31T19:44:25.254 INFO:tasks.workunit.client.1.smithi171.stdout:9/455: dwrite dd/df/f72 [0,4194304] 0 2022-01-31T19:44:25.256 INFO:tasks.workunit.client.1.smithi171.stdout:2/509: dwrite d7/d26/f92 [0,4194304] 0 2022-01-31T19:44:25.256 INFO:tasks.workunit.client.1.smithi171.stdout:1/331: dwrite dd/d23/f29 [4194304,4194304] 0 2022-01-31T19:44:25.257 INFO:tasks.workunit.client.1.smithi171.stdout:2/510: write d7/dd/d5e/d84/f8b [18814,54268] 0 2022-01-31T19:44:25.257 INFO:tasks.workunit.client.1.smithi171.stdout:2/511: readlink d7/lc 0 2022-01-31T19:44:25.257 INFO:tasks.workunit.client.1.smithi171.stdout:1/332: truncate dd/df/d19/d1b/d1e/f4a 4300488 0 2022-01-31T19:44:25.259 INFO:tasks.workunit.client.1.smithi171.stdout:2/512: symlink d7/dd/d5e/la3 0 2022-01-31T19:44:25.259 INFO:tasks.workunit.client.1.smithi171.stdout:2/513: write d7/d26/d3d/d40/f4f [4736870,116785] 0 2022-01-31T19:44:25.260 INFO:tasks.workunit.client.1.smithi171.stdout:1/333: mknod dd/c6e 0 2022-01-31T19:44:25.260 INFO:tasks.workunit.client.1.smithi171.stdout:1/334: readlink dd/d23/d2e/d5e/l3b 0 2022-01-31T19:44:25.262 INFO:tasks.workunit.client.1.smithi171.stdout:2/514: mkdir d7/d9/d1a/d45/da4 0 2022-01-31T19:44:25.263 INFO:tasks.workunit.client.1.smithi171.stdout:1/335: unlink dd/d23/d2e/f68 0 2022-01-31T19:44:25.264 INFO:tasks.workunit.client.1.smithi171.stdout:2/515: rename d7/dd/d5e/f86 to d7/dd/d93/fa5 0 2022-01-31T19:44:25.265 INFO:tasks.workunit.client.1.smithi171.stdout:1/336: truncate dd/df/f12 1202001 0 2022-01-31T19:44:25.265 INFO:tasks.workunit.client.1.smithi171.stdout:1/337: read - dd/df/d19/d1b/f4e zero size 2022-01-31T19:44:25.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/516: stat d7/d26/l3b 0 2022-01-31T19:44:25.267 INFO:tasks.workunit.client.1.smithi171.stdout:1/338: unlink dd/d23/d2e/d5e/c2c 0 2022-01-31T19:44:25.267 INFO:tasks.workunit.client.1.smithi171.stdout:1/339: write fa [5654991,38228] 0 2022-01-31T19:44:25.270 INFO:tasks.workunit.client.1.smithi171.stdout:9/456: dwrite dd/f3c [0,4194304] 0 2022-01-31T19:44:25.270 INFO:tasks.workunit.client.1.smithi171.stdout:9/457: getdents dd/df/d15/d1a/d64/d66 0 2022-01-31T19:44:25.271 INFO:tasks.workunit.client.1.smithi171.stdout:9/458: truncate dd/d1c/f95 806705 0 2022-01-31T19:44:25.274 INFO:tasks.workunit.client.1.smithi171.stdout:9/459: unlink dd/c92 0 2022-01-31T19:44:25.275 INFO:tasks.workunit.client.1.smithi171.stdout:9/460: stat dd/f11 0 2022-01-31T19:44:25.275 INFO:tasks.workunit.client.1.smithi171.stdout:9/461: chown dd/df/d3f/f82 61849227 1 2022-01-31T19:44:25.276 INFO:tasks.workunit.client.1.smithi171.stdout:9/462: creat dd/df/d15/d1a/d84/f99 x:0 0 0 2022-01-31T19:44:25.276 INFO:tasks.workunit.client.1.smithi171.stdout:9/463: write dd/df/f62 [732358,61857] 0 2022-01-31T19:44:25.276 INFO:tasks.workunit.client.1.smithi171.stdout:9/464: chown dd/c8b 766 1 2022-01-31T19:44:25.283 INFO:tasks.workunit.client.1.smithi171.stdout:9/465: mknod dd/df/d15/d38/d41/c9a 0 2022-01-31T19:44:25.291 INFO:tasks.workunit.client.1.smithi171.stdout:2/517: dwrite d7/d26/f92 [0,4194304] 0 2022-01-31T19:44:25.291 INFO:tasks.workunit.client.1.smithi171.stdout:2/518: truncate d7/d26/f5f 4290532 0 2022-01-31T19:44:25.294 INFO:tasks.workunit.client.1.smithi171.stdout:9/466: dread dd/df/d15/d1a/f4b [0,4194304] 0 2022-01-31T19:44:25.296 INFO:tasks.workunit.client.1.smithi171.stdout:9/467: creat dd/df/f9b x:0 0 0 2022-01-31T19:44:25.298 INFO:tasks.workunit.client.1.smithi171.stdout:9/468: write fc [3838025,102732] 0 2022-01-31T19:44:25.300 INFO:tasks.workunit.client.1.smithi171.stdout:1/340: dwrite dd/df/d19/d1b/f63 [0,4194304] 0 2022-01-31T19:44:25.300 INFO:tasks.workunit.client.1.smithi171.stdout:9/469: getdents dd/df/d86 0 2022-01-31T19:44:25.301 INFO:tasks.workunit.client.1.smithi171.stdout:9/470: fdatasync dd/df/d15/f69 0 2022-01-31T19:44:25.306 INFO:tasks.workunit.client.1.smithi171.stdout:1/341: symlink dd/d23/d2e/l6f 0 2022-01-31T19:44:25.307 INFO:tasks.workunit.client.1.smithi171.stdout:1/342: readlink dd/d23/d2e/l51 0 2022-01-31T19:44:25.308 INFO:tasks.workunit.client.1.smithi171.stdout:1/343: creat dd/df/d19/d1b/d1e/d35/f70 x:0 0 0 2022-01-31T19:44:25.308 INFO:tasks.workunit.client.1.smithi171.stdout:1/344: stat dd/d23/d2e/d5e/f26 0 2022-01-31T19:44:25.309 INFO:tasks.workunit.client.1.smithi171.stdout:1/345: mknod dd/d23/d34/c71 0 2022-01-31T19:44:25.314 INFO:tasks.workunit.client.1.smithi171.stdout:2/519: dwrite d7/dd/f1e [0,4194304] 0 2022-01-31T19:44:25.318 INFO:tasks.workunit.client.1.smithi171.stdout:2/520: creat d7/dd/d5e/d6b/d81/fa6 x:0 0 0 2022-01-31T19:44:25.318 INFO:tasks.workunit.client.1.smithi171.stdout:2/521: creat d7/d26/fa7 x:0 0 0 2022-01-31T19:44:25.318 INFO:tasks.workunit.client.1.smithi171.stdout:2/522: readlink d7/dd/d5e/la3 0 2022-01-31T19:44:25.319 INFO:tasks.workunit.client.1.smithi171.stdout:2/523: fsync d7/d55/f59 0 2022-01-31T19:44:25.321 INFO:tasks.workunit.client.1.smithi171.stdout:2/524: truncate d7/d26/f5f 3812205 0 2022-01-31T19:44:25.331 INFO:tasks.workunit.client.1.smithi171.stdout:1/346: dwrite dd/df/d19/d1b/d1e/d35/f70 [0,4194304] 0 2022-01-31T19:44:25.334 INFO:tasks.workunit.client.1.smithi171.stdout:9/471: dwrite dd/df/f21 [0,4194304] 0 2022-01-31T19:44:25.335 INFO:tasks.workunit.client.1.smithi171.stdout:9/472: getdents dd/df/d15/d38/d41 0 2022-01-31T19:44:25.343 INFO:tasks.workunit.client.1.smithi171.stdout:2/525: dwrite d7/d55/f59 [0,4194304] 0 2022-01-31T19:44:25.343 INFO:tasks.workunit.client.1.smithi171.stdout:2/526: write d7/dd/f52 [569100,16182] 0 2022-01-31T19:44:25.343 INFO:tasks.workunit.client.1.smithi171.stdout:2/527: creat d7/dd/d5e/d6b/d81/fa8 x:0 0 0 2022-01-31T19:44:25.349 INFO:tasks.workunit.client.1.smithi171.stdout:9/473: dread f9 [0,4194304] 0 2022-01-31T19:44:25.350 INFO:tasks.workunit.client.1.smithi171.stdout:9/474: mkdir dd/df/d15/d38/d9c 0 2022-01-31T19:44:25.351 INFO:tasks.workunit.client.1.smithi171.stdout:9/475: symlink dd/df/d86/l9d 0 2022-01-31T19:44:25.355 INFO:tasks.workunit.client.1.smithi171.stdout:9/476: rename dd/df/d15/d38/d41/f78 to dd/df/d3f/f9e 0 2022-01-31T19:44:25.356 INFO:tasks.workunit.client.1.smithi171.stdout:9/477: mkdir dd/df/d3f/d73/d96/d9f 0 2022-01-31T19:44:25.358 INFO:tasks.workunit.client.1.smithi171.stdout:9/478: write dd/f37 [4470674,21888] 0 2022-01-31T19:44:25.358 INFO:tasks.workunit.client.1.smithi171.stdout:1/347: dwrite dd/df/d19/f27 [0,4194304] 0 2022-01-31T19:44:25.359 INFO:tasks.workunit.client.1.smithi171.stdout:9/479: rename dd/df/f25 to dd/df/fa0 0 2022-01-31T19:44:25.363 INFO:tasks.workunit.client.1.smithi171.stdout:1/348: write dd/d23/d34/f65 [785219,11613] 0 2022-01-31T19:44:25.364 INFO:tasks.workunit.client.1.smithi171.stdout:9/480: rename dd/df/d15/d38/d41/f97 to dd/df/d3f/d73/d96/d9f/fa1 0 2022-01-31T19:44:25.365 INFO:tasks.workunit.client.1.smithi171.stdout:9/481: read fc [675621,39675] 0 2022-01-31T19:44:25.366 INFO:tasks.workunit.client.1.smithi171.stdout:9/482: chown f9 189624864 1 2022-01-31T19:44:25.366 INFO:tasks.workunit.client.1.smithi171.stdout:9/483: stat dd/df/d15/c85 0 2022-01-31T19:44:25.366 INFO:tasks.workunit.client.1.smithi171.stdout:9/484: fsync dd/df/d15/f2c 0 2022-01-31T19:44:25.366 INFO:tasks.workunit.client.1.smithi171.stdout:9/485: write dd/d1c/f95 [1292521,113572] 0 2022-01-31T19:44:25.367 INFO:tasks.workunit.client.1.smithi171.stdout:1/349: rmdir dd/d23/d2e/d5e 39 2022-01-31T19:44:25.367 INFO:tasks.workunit.client.1.smithi171.stdout:1/350: creat dd/df/d19/d1b/f72 x:0 0 0 2022-01-31T19:44:25.368 INFO:tasks.workunit.client.1.smithi171.stdout:1/351: write fa [1912227,80789] 0 2022-01-31T19:44:25.369 INFO:tasks.workunit.client.1.smithi171.stdout:1/352: dread - dd/d23/f3f zero size 2022-01-31T19:44:25.369 INFO:tasks.workunit.client.1.smithi171.stdout:6/330: sync 2022-01-31T19:44:25.369 INFO:tasks.workunit.client.1.smithi171.stdout:4/339: sync 2022-01-31T19:44:25.369 INFO:tasks.workunit.client.1.smithi171.stdout:0/372: sync 2022-01-31T19:44:25.370 INFO:tasks.workunit.client.1.smithi171.stdout:3/414: sync 2022-01-31T19:44:25.370 INFO:tasks.workunit.client.1.smithi171.stdout:8/354: sync 2022-01-31T19:44:25.370 INFO:tasks.workunit.client.1.smithi171.stdout:5/363: sync 2022-01-31T19:44:25.370 INFO:tasks.workunit.client.1.smithi171.stdout:7/268: sync 2022-01-31T19:44:25.371 INFO:tasks.workunit.client.1.smithi171.stdout:8/355: write df/d1a/f58 [3448745,110564] 0 2022-01-31T19:44:25.371 INFO:tasks.workunit.client.1.smithi171.stdout:1/353: mkdir dd/d73 0 2022-01-31T19:44:25.372 INFO:tasks.workunit.client.1.smithi171.stdout:9/486: fsync dd/df/d3f/f9e 0 2022-01-31T19:44:25.382 INFO:tasks.workunit.client.1.smithi171.stdout:9/487: dread dd/df/d15/d1a/f56 [0,4194304] 0 2022-01-31T19:44:25.383 INFO:tasks.workunit.client.1.smithi171.stdout:0/373: creat d9/db/d59/d71/f8f x:0 0 0 2022-01-31T19:44:25.384 INFO:tasks.workunit.client.1.smithi171.stdout:4/340: symlink d2/d3/d5/dc/d4b/d56/d66/l72 0 2022-01-31T19:44:25.388 INFO:tasks.workunit.client.1.smithi171.stdout:8/356: mknod df/d18/d19/d36/c7d 0 2022-01-31T19:44:25.388 INFO:tasks.workunit.client.1.smithi171.stdout:4/341: dread d2/d3/d5/d11/d12/d15/f52 [8388608,4194304] 0 2022-01-31T19:44:25.388 INFO:tasks.workunit.client.1.smithi171.stdout:8/357: chown df/d18/d19/d21/f32 13 1 2022-01-31T19:44:25.389 INFO:tasks.workunit.client.1.smithi171.stdout:6/331: rmdir dc/d19 39 2022-01-31T19:44:25.391 INFO:tasks.workunit.client.1.smithi171.stdout:0/374: creat d9/db/f90 x:0 0 0 2022-01-31T19:44:25.392 INFO:tasks.workunit.client.1.smithi171.stdout:7/269: rmdir d1/d26/d47 39 2022-01-31T19:44:25.392 INFO:tasks.workunit.client.1.smithi171.stdout:7/270: chown d1/d26/l32 7174685 1 2022-01-31T19:44:25.393 INFO:tasks.workunit.client.1.smithi171.stdout:7/271: dread - d1/d1e/d2a/d29/d31/d30/f3f zero size 2022-01-31T19:44:25.393 INFO:tasks.workunit.client.1.smithi171.stdout:7/272: write d1/d1e/d2a/d29/d31/ff [5105429,112901] 0 2022-01-31T19:44:25.393 INFO:tasks.workunit.client.1.smithi171.stdout:7/273: stat d1/d52/f35 0 2022-01-31T19:44:25.393 INFO:tasks.workunit.client.1.smithi171.stdout:7/274: creat d1/d1e/d2a/d29/d31/d30/f5a x:0 0 0 2022-01-31T19:44:25.394 INFO:tasks.workunit.client.1.smithi171.stdout:9/488: rename dd/df/d15/d1a/d64/d66 to dd/df/d15/da2 0 2022-01-31T19:44:25.397 INFO:tasks.workunit.client.1.smithi171.stdout:7/275: symlink d1/d26/d57/l5b 0 2022-01-31T19:44:25.397 INFO:tasks.workunit.client.1.smithi171.stdout:7/276: write d1/d1b/f22 [4756621,50267] 0 2022-01-31T19:44:25.397 INFO:tasks.workunit.client.1.smithi171.stdout:7/277: stat d1/d52/f28 0 2022-01-31T19:44:25.398 INFO:tasks.workunit.client.1.smithi171.stdout:7/278: creat d1/d26/f5c x:0 0 0 2022-01-31T19:44:25.398 INFO:tasks.workunit.client.1.smithi171.stdout:5/364: dwrite d5/d6/d15/d5d/f65 [0,4194304] 0 2022-01-31T19:44:25.399 INFO:tasks.workunit.client.1.smithi171.stdout:5/365: fdatasync d5/d6/d15/d23/d3d/f51 0 2022-01-31T19:44:25.399 INFO:tasks.workunit.client.1.smithi171.stdout:1/354: dwrite dd/df/d19/d1b/f4e [0,4194304] 0 2022-01-31T19:44:25.400 INFO:tasks.workunit.client.1.smithi171.stdout:4/342: link d2/d3/d5/d11/d12/d15/f24 d2/d3/f73 0 2022-01-31T19:44:25.403 INFO:tasks.workunit.client.1.smithi171.stdout:3/415: dwrite d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:25.403 INFO:tasks.workunit.client.1.smithi171.stdout:8/358: dread df/d27/f64 [4194304,4194304] 0 2022-01-31T19:44:25.404 INFO:tasks.workunit.client.1.smithi171.stdout:3/416: chown d4/fa 2050070821 1 2022-01-31T19:44:25.404 INFO:tasks.workunit.client.1.smithi171.stdout:3/417: fsync d4/d21/f31 0 2022-01-31T19:44:25.404 INFO:tasks.workunit.client.1.smithi171.stdout:3/418: write d4/dd/d27/d28/d35/d55/f5e [1308205,29552] 0 2022-01-31T19:44:25.405 INFO:tasks.workunit.client.1.smithi171.stdout:6/332: write dc/d19/d29/f34 [755688,84902] 0 2022-01-31T19:44:25.406 INFO:tasks.workunit.client.1.smithi171.stdout:8/359: write df/d2a/d38/f56 [2153467,24104] 0 2022-01-31T19:44:25.406 INFO:tasks.workunit.client.1.smithi171.stdout:8/360: write f3 [1385994,115925] 0 2022-01-31T19:44:25.407 INFO:tasks.workunit.client.1.smithi171.stdout:7/279: mknod d1/d1e/d2a/d29/d31/d30/c5d 0 2022-01-31T19:44:25.410 INFO:tasks.workunit.client.1.smithi171.stdout:7/280: write d1/d52/f21 [941441,22604] 0 2022-01-31T19:44:25.410 INFO:tasks.workunit.client.1.smithi171.stdout:7/281: chown d1/d1e/d2a/d29/d31/c25 8 1 2022-01-31T19:44:25.410 INFO:tasks.workunit.client.1.smithi171.stdout:7/282: fdatasync d1/d1e/d2a/f3b 0 2022-01-31T19:44:25.411 INFO:tasks.workunit.client.1.smithi171.stdout:7/283: creat d1/d1b/f5e x:0 0 0 2022-01-31T19:44:25.414 INFO:tasks.workunit.client.1.smithi171.stdout:5/366: creat d5/d6/d15/d23/d58/f79 x:0 0 0 2022-01-31T19:44:25.422 INFO:tasks.workunit.client.1.smithi171.stdout:5/367: rename d5/l40 to d5/d6/d74/l7a 0 2022-01-31T19:44:25.425 INFO:tasks.workunit.client.1.smithi171.stdout:1/355: rmdir dd/df 39 2022-01-31T19:44:25.427 INFO:tasks.workunit.client.1.smithi171.stdout:0/375: dwrite d9/db/dc/d3f/f2d [0,4194304] 0 2022-01-31T19:44:25.428 INFO:tasks.workunit.client.1.smithi171.stdout:7/284: mkdir d1/d1e/d2a/d29/d31/d30/d3c/d5f 0 2022-01-31T19:44:25.428 INFO:tasks.workunit.client.1.smithi171.stdout:7/285: write d1/d26/f5c [900532,119340] 0 2022-01-31T19:44:25.428 INFO:tasks.workunit.client.1.smithi171.stdout:7/286: chown d1/d1e/d2a/d29/d31/d30 5237377 1 2022-01-31T19:44:25.430 INFO:tasks.workunit.client.1.smithi171.stdout:5/368: mknod d5/d6/d15/d23/d58/c7b 0 2022-01-31T19:44:25.430 INFO:tasks.workunit.client.1.smithi171.stdout:5/369: readlink d5/d6/d15/d23/d3d/l63 0 2022-01-31T19:44:25.432 INFO:tasks.workunit.client.1.smithi171.stdout:7/287: mknod d1/d1e/d2a/d29/d31/c60 0 2022-01-31T19:44:25.433 INFO:tasks.workunit.client.1.smithi171.stdout:7/288: chown d1/d52/c13 5003573 1 2022-01-31T19:44:25.433 INFO:tasks.workunit.client.1.smithi171.stdout:8/361: dwrite df/d18/f42 [0,4194304] 0 2022-01-31T19:44:25.434 INFO:tasks.workunit.client.1.smithi171.stdout:0/376: rmdir d9/db/d22/d3a 39 2022-01-31T19:44:25.434 INFO:tasks.workunit.client.1.smithi171.stdout:5/370: dread d5/d6/d15/d23/d67/f5c [4194304,4194304] 0 2022-01-31T19:44:25.435 INFO:tasks.workunit.client.1.smithi171.stdout:7/289: symlink d1/d26/l61 0 2022-01-31T19:44:25.435 INFO:tasks.workunit.client.1.smithi171.stdout:7/290: truncate d1/d26/f5c 1428140 0 2022-01-31T19:44:25.438 INFO:tasks.workunit.client.1.smithi171.stdout:8/362: mkdir df/d18/d19/d36/d70/d7e 0 2022-01-31T19:44:25.439 INFO:tasks.workunit.client.1.smithi171.stdout:5/371: symlink d5/d6/l7c 0 2022-01-31T19:44:25.439 INFO:tasks.workunit.client.1.smithi171.stdout:0/377: write d9/f1d [1181664,55637] 0 2022-01-31T19:44:25.440 INFO:tasks.workunit.client.1.smithi171.stdout:0/378: fdatasync d9/f1f 0 2022-01-31T19:44:25.440 INFO:tasks.workunit.client.1.smithi171.stdout:1/356: dread f3 [0,4194304] 0 2022-01-31T19:44:25.441 INFO:tasks.workunit.client.1.smithi171.stdout:5/372: creat d5/d6/dd/d66/f7d x:0 0 0 2022-01-31T19:44:25.442 INFO:tasks.workunit.client.1.smithi171.stdout:4/343: dwrite d2/d3/f73 [4194304,4194304] 0 2022-01-31T19:44:25.443 INFO:tasks.workunit.client.1.smithi171.stdout:5/373: creat d5/d6/d15/d23/d6b/f7e x:0 0 0 2022-01-31T19:44:25.443 INFO:tasks.workunit.client.1.smithi171.stdout:5/374: creat d5/d6/d74/f7f x:0 0 0 2022-01-31T19:44:25.445 INFO:tasks.workunit.client.1.smithi171.stdout:4/344: symlink d2/d3/d5/d11/d12/d15/l74 0 2022-01-31T19:44:25.447 INFO:tasks.workunit.client.1.smithi171.stdout:6/333: dwrite fa [4194304,4194304] 0 2022-01-31T19:44:25.448 INFO:tasks.workunit.client.1.smithi171.stdout:3/419: dwrite d4/dd/d1a/d52/f54 [0,4194304] 0 2022-01-31T19:44:25.448 INFO:tasks.workunit.client.1.smithi171.stdout:3/420: chown d4/dd/d1a 5 1 2022-01-31T19:44:25.450 INFO:tasks.workunit.client.1.smithi171.stdout:4/345: dread d2/d3/d5/dc/f19 [0,4194304] 0 2022-01-31T19:44:25.450 INFO:tasks.workunit.client.1.smithi171.stdout:4/346: dread - d2/d3/d5/d11/d12/f45 zero size 2022-01-31T19:44:25.452 INFO:tasks.workunit.client.1.smithi171.stdout:9/489: dwrite dd/fe [0,4194304] 0 2022-01-31T19:44:25.452 INFO:tasks.workunit.client.1.smithi171.stdout:9/490: chown dd/df/d15/d1a/d64 942601 1 2022-01-31T19:44:25.453 INFO:tasks.workunit.client.1.smithi171.stdout:1/357: dread fa [0,4194304] 0 2022-01-31T19:44:25.455 INFO:tasks.workunit.client.1.smithi171.stdout:6/334: link dc/d19/d29/d27/f3a dc/d1e/d3f/f65 0 2022-01-31T19:44:25.458 INFO:tasks.workunit.client.1.smithi171.stdout:0/379: dwrite f3 [0,4194304] 0 2022-01-31T19:44:25.458 INFO:tasks.workunit.client.1.smithi171.stdout:3/421: stat d4/l2c 0 2022-01-31T19:44:25.459 INFO:tasks.workunit.client.1.smithi171.stdout:3/422: creat d4/d21/d3a/f7c x:0 0 0 2022-01-31T19:44:25.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/347: mknod d2/d3/d5/d11/d12/d15/c75 0 2022-01-31T19:44:25.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/348: fsync d2/f5c 0 2022-01-31T19:44:25.465 INFO:tasks.workunit.client.1.smithi171.stdout:5/375: mkdir d5/d6/d15/d80 0 2022-01-31T19:44:25.466 INFO:tasks.workunit.client.1.smithi171.stdout:5/376: write d5/d6/dd/d66/f7d [460307,79043] 0 2022-01-31T19:44:25.468 INFO:tasks.workunit.client.1.smithi171.stdout:1/358: getdents dd/df/d19/d1b/d1e/d64 0 2022-01-31T19:44:25.468 INFO:tasks.workunit.client.1.smithi171.stdout:1/359: dread - dd/d23/d2e/d5e/f40 zero size 2022-01-31T19:44:25.470 INFO:tasks.workunit.client.1.smithi171.stdout:0/380: mknod d9/db/dc/c91 0 2022-01-31T19:44:25.470 INFO:tasks.workunit.client.1.smithi171.stdout:9/491: dwrite dd/df/d15/f2f [0,4194304] 0 2022-01-31T19:44:25.472 INFO:tasks.workunit.client.1.smithi171.stdout:3/423: symlink d4/dd/d27/l7d 0 2022-01-31T19:44:25.472 INFO:tasks.workunit.client.1.smithi171.stdout:3/424: dread - d4/dd/d73/f77 zero size 2022-01-31T19:44:25.472 INFO:tasks.workunit.client.1.smithi171.stdout:3/425: write d4/d21/d3a/f4f [841571,74173] 0 2022-01-31T19:44:25.473 INFO:tasks.workunit.client.1.smithi171.stdout:0/381: dread d9/db/dc/d3f/d25/f29 [0,4194304] 0 2022-01-31T19:44:25.475 INFO:tasks.workunit.client.1.smithi171.stdout:1/360: unlink dd/d23/c3d 0 2022-01-31T19:44:25.479 INFO:tasks.workunit.client.1.smithi171.stdout:9/492: creat dd/d1c/d7c/d7d/fa3 x:0 0 0 2022-01-31T19:44:25.481 INFO:tasks.workunit.client.1.smithi171.stdout:5/377: dwrite d5/d6/d15/d23/d3d/f49 [0,4194304] 0 2022-01-31T19:44:25.481 INFO:tasks.workunit.client.1.smithi171.stdout:5/378: chown d5/d6/f21 11 1 2022-01-31T19:44:25.485 INFO:tasks.workunit.client.1.smithi171.stdout:5/379: creat d5/d6/d15/d5d/f81 x:0 0 0 2022-01-31T19:44:25.485 INFO:tasks.workunit.client.1.smithi171.stdout:5/380: dread - d5/d6/d15/d5d/f81 zero size 2022-01-31T19:44:25.485 INFO:tasks.workunit.client.1.smithi171.stdout:5/381: creat d5/d6/dd/d66/f82 x:0 0 0 2022-01-31T19:44:25.486 INFO:tasks.workunit.client.1.smithi171.stdout:0/382: rename d9/db/d22/d5d to d9/d24/d92 0 2022-01-31T19:44:25.488 INFO:tasks.workunit.client.1.smithi171.stdout:0/383: mknod d9/d24/c93 0 2022-01-31T19:44:25.491 INFO:tasks.workunit.client.1.smithi171.stdout:0/384: rename d9/db/dc/d3f/f81 to d9/d24/d92/f94 0 2022-01-31T19:44:25.492 INFO:tasks.workunit.client.1.smithi171.stdout:0/385: truncate d9/db/d22/d3a/f8e 327596 0 2022-01-31T19:44:25.492 INFO:tasks.workunit.client.1.smithi171.stdout:0/386: write d9/d24/d92/f5f [251129,27940] 0 2022-01-31T19:44:25.492 INFO:tasks.workunit.client.1.smithi171.stdout:0/387: truncate d9/db/f21 38825 0 2022-01-31T19:44:25.493 INFO:tasks.workunit.client.1.smithi171.stdout:1/361: dread dd/d23/d34/d4b/f59 [0,4194304] 0 2022-01-31T19:44:25.494 INFO:tasks.workunit.client.1.smithi171.stdout:0/388: truncate d9/f3b 3320724 0 2022-01-31T19:44:25.495 INFO:tasks.workunit.client.1.smithi171.stdout:1/362: mkdir dd/d23/d74 0 2022-01-31T19:44:25.496 INFO:tasks.workunit.client.1.smithi171.stdout:1/363: creat dd/df/d19/f75 x:0 0 0 2022-01-31T19:44:25.498 INFO:tasks.workunit.client.1.smithi171.stdout:0/389: getdents d9/db 0 2022-01-31T19:44:25.498 INFO:tasks.workunit.client.1.smithi171.stdout:0/390: dread - d9/d24/f49 zero size 2022-01-31T19:44:25.499 INFO:tasks.workunit.client.1.smithi171.stdout:1/364: symlink dd/df/d19/l76 0 2022-01-31T19:44:25.501 INFO:tasks.workunit.client.1.smithi171.stdout:0/391: rename d9/d24/f49 to d9/db/d22/f95 0 2022-01-31T19:44:25.502 INFO:tasks.workunit.client.1.smithi171.stdout:0/392: write d9/f16 [589192,30371] 0 2022-01-31T19:44:25.502 INFO:tasks.workunit.client.1.smithi171.stdout:0/393: truncate d9/db/dc/d3f/d25/f52 472160 0 2022-01-31T19:44:25.503 INFO:tasks.workunit.client.1.smithi171.stdout:1/365: mkdir dd/df/d19/d1b/d77 0 2022-01-31T19:44:25.504 INFO:tasks.workunit.client.1.smithi171.stdout:0/394: unlink d9/d24/c93 0 2022-01-31T19:44:25.504 INFO:tasks.workunit.client.1.smithi171.stdout:0/395: stat d9/d24/d4c 0 2022-01-31T19:44:25.504 INFO:tasks.workunit.client.1.smithi171.stdout:1/366: stat dd/df/d19/d1b/d1e/d64/l67 0 2022-01-31T19:44:25.511 INFO:tasks.workunit.client.1.smithi171.stdout:9/493: dwrite dd/df/d15/f3a [0,4194304] 0 2022-01-31T19:44:25.512 INFO:tasks.workunit.client.1.smithi171.stdout:4/349: dwrite d2/d3/f25 [0,4194304] 0 2022-01-31T19:44:25.512 INFO:tasks.workunit.client.1.smithi171.stdout:3/426: dwrite d4/dd/d27/f47 [0,4194304] 0 2022-01-31T19:44:25.513 INFO:tasks.workunit.client.1.smithi171.stdout:5/382: dwrite d5/f8 [4194304,4194304] 0 2022-01-31T19:44:25.518 INFO:tasks.workunit.client.1.smithi171.stdout:4/350: dread d2/f5c [0,4194304] 0 2022-01-31T19:44:25.519 INFO:tasks.workunit.client.1.smithi171.stdout:3/427: mknod d4/dd/d27/d4e/c7e 0 2022-01-31T19:44:25.521 INFO:tasks.workunit.client.1.smithi171.stdout:5/383: mknod d5/d6/d15/d80/c83 0 2022-01-31T19:44:25.528 INFO:tasks.workunit.client.1.smithi171.stdout:3/428: mknod d4/dd/d27/d28/d35/c7f 0 2022-01-31T19:44:25.528 INFO:tasks.workunit.client.1.smithi171.stdout:3/429: fdatasync d4/dd/d27/d28/d35/f5c 0 2022-01-31T19:44:25.529 INFO:tasks.workunit.client.1.smithi171.stdout:3/430: chown d4/d21/d3a 19 1 2022-01-31T19:44:25.530 INFO:tasks.workunit.client.1.smithi171.stdout:4/351: dread d2/d3/d5/f59 [0,4194304] 0 2022-01-31T19:44:25.532 INFO:tasks.workunit.client.1.smithi171.stdout:3/431: symlink d4/dd/d27/d4e/d62/l80 0 2022-01-31T19:44:25.532 INFO:tasks.workunit.client.1.smithi171.stdout:3/432: getdents d4/dd/d27/d28/d35/d55/d6c 0 2022-01-31T19:44:25.532 INFO:tasks.workunit.client.1.smithi171.stdout:3/433: fsync d4/d21/d3a/d59/f70 0 2022-01-31T19:44:25.533 INFO:tasks.workunit.client.1.smithi171.stdout:3/434: dread - d4/d21/d3a/f7c zero size 2022-01-31T19:44:25.533 INFO:tasks.workunit.client.1.smithi171.stdout:5/384: dread d5/d6/d15/f3f [0,4194304] 0 2022-01-31T19:44:25.534 INFO:tasks.workunit.client.1.smithi171.stdout:3/435: creat d4/dd/d1a/f81 x:0 0 0 2022-01-31T19:44:25.534 INFO:tasks.workunit.client.1.smithi171.stdout:3/436: readlink d4/dd/l33 0 2022-01-31T19:44:25.536 INFO:tasks.workunit.client.1.smithi171.stdout:0/396: dwrite d9/db/dc/d3f/f2d [4194304,4194304] 0 2022-01-31T19:44:25.537 INFO:tasks.workunit.client.1.smithi171.stdout:0/397: stat d9/db/c1c 0 2022-01-31T19:44:25.539 INFO:tasks.workunit.client.1.smithi171.stdout:5/385: read d5/d6/dd/f22 [544555,80083] 0 2022-01-31T19:44:25.541 INFO:tasks.workunit.client.1.smithi171.stdout:9/494: dwrite dd/df/f62 [0,4194304] 0 2022-01-31T19:44:25.544 INFO:tasks.workunit.client.1.smithi171.stdout:9/495: truncate dd/d1c/f29 3575814 0 2022-01-31T19:44:25.544 INFO:tasks.workunit.client.1.smithi171.stdout:9/496: chown dd/d1c/l39 933 1 2022-01-31T19:44:25.546 INFO:tasks.workunit.client.1.smithi171.stdout:9/497: write dd/df/d15/d1a/f56 [3926026,7357] 0 2022-01-31T19:44:25.546 INFO:tasks.workunit.client.1.smithi171.stdout:5/386: getdents d5/d6/d15/d80 0 2022-01-31T19:44:25.550 INFO:tasks.workunit.client.1.smithi171.stdout:5/387: dread d5/f8 [0,4194304] 0 2022-01-31T19:44:25.550 INFO:tasks.workunit.client.1.smithi171.stdout:5/388: chown d5/d6/d15/d5d/f81 122235 1 2022-01-31T19:44:25.552 INFO:tasks.workunit.client.1.smithi171.stdout:5/389: symlink d5/d6/d15/d80/l84 0 2022-01-31T19:44:25.552 INFO:tasks.workunit.client.1.smithi171.stdout:9/498: write dd/df/d15/d1a/f2b [3528893,127589] 0 2022-01-31T19:44:25.553 INFO:tasks.workunit.client.1.smithi171.stdout:9/499: readlink dd/d1c/l59 0 2022-01-31T19:44:25.554 INFO:tasks.workunit.client.1.smithi171.stdout:5/390: symlink d5/d6/d15/d5d/d6a/l85 0 2022-01-31T19:44:25.554 INFO:tasks.workunit.client.1.smithi171.stdout:5/391: write d5/f68 [682005,128948] 0 2022-01-31T19:44:25.555 INFO:tasks.workunit.client.1.smithi171.stdout:9/500: symlink dd/df/d15/d38/d4d/d80/la4 0 2022-01-31T19:44:25.558 INFO:tasks.workunit.client.1.smithi171.stdout:5/392: mknod d5/d6/d15/d5d/d6a/c86 0 2022-01-31T19:44:25.558 INFO:tasks.workunit.client.1.smithi171.stdout:5/393: chown d5/d6/d15/d5d/d6a/c86 279152256 1 2022-01-31T19:44:25.560 INFO:tasks.workunit.client.1.smithi171.stdout:5/394: creat d5/d6/d15/d23/f87 x:0 0 0 2022-01-31T19:44:25.560 INFO:tasks.workunit.client.1.smithi171.stdout:5/395: stat d5/d6/d15/f1c 0 2022-01-31T19:44:25.561 INFO:tasks.workunit.client.1.smithi171.stdout:5/396: write d5/d6/dd/f20 [2637065,61724] 0 2022-01-31T19:44:25.561 INFO:tasks.workunit.client.1.smithi171.stdout:5/397: stat d5/d6/d74/l7a 0 2022-01-31T19:44:25.563 INFO:tasks.workunit.client.1.smithi171.stdout:4/352: dwrite d2/d3/d5/dc/f26 [0,4194304] 0 2022-01-31T19:44:25.564 INFO:tasks.workunit.client.1.smithi171.stdout:5/398: stat d5/d6/d15/d5d/l62 0 2022-01-31T19:44:25.564 INFO:tasks.workunit.client.1.smithi171.stdout:5/399: dread d5/d6/dd/f35 [0,4194304] 0 2022-01-31T19:44:25.567 INFO:tasks.workunit.client.1.smithi171.stdout:5/400: link d5/f37 d5/d6/d15/f88 0 2022-01-31T19:44:25.568 INFO:tasks.workunit.client.1.smithi171.stdout:5/401: readlink d5/d6/dd/l52 0 2022-01-31T19:44:25.568 INFO:tasks.workunit.client.1.smithi171.stdout:0/398: dwrite d9/db/d22/f5b [0,4194304] 0 2022-01-31T19:44:25.569 INFO:tasks.workunit.client.1.smithi171.stdout:0/399: readlink d9/d24/d4c/d35/l72 0 2022-01-31T19:44:25.569 INFO:tasks.workunit.client.1.smithi171.stdout:0/400: readlink d9/db/dc/d3f/d25/l76 0 2022-01-31T19:44:25.570 INFO:tasks.workunit.client.1.smithi171.stdout:0/401: readlink d9/db/lf 0 2022-01-31T19:44:25.570 INFO:tasks.workunit.client.1.smithi171.stdout:9/501: write dd/df/d15/d1a/f56 [441035,40704] 0 2022-01-31T19:44:25.571 INFO:tasks.workunit.client.1.smithi171.stdout:0/402: write d9/d24/d4c/f3d [4118916,7724] 0 2022-01-31T19:44:25.571 INFO:tasks.workunit.client.1.smithi171.stdout:3/437: dwrite d4/dd/f20 [0,4194304] 0 2022-01-31T19:44:25.572 INFO:tasks.workunit.client.1.smithi171.stdout:9/502: mkdir dd/df/d15/d1a/d64/d6e/da5 0 2022-01-31T19:44:25.573 INFO:tasks.workunit.client.1.smithi171.stdout:0/403: link d9/d24/d4c/d35/c64 d9/db/dc/d3f/d6f/c96 0 2022-01-31T19:44:25.587 INFO:tasks.workunit.client.1.smithi171.stdout:3/438: rename d4/dd/f1e to d4/dd/d27/d4e/d62/f82 0 2022-01-31T19:44:25.588 INFO:tasks.workunit.client.1.smithi171.stdout:3/439: write d4/dd/f20 [4294369,20165] 0 2022-01-31T19:44:25.588 INFO:tasks.workunit.client.1.smithi171.stdout:3/440: truncate d4/dd/d27/d28/d35/f38 5136139 0 2022-01-31T19:44:25.588 INFO:tasks.workunit.client.1.smithi171.stdout:3/441: chown d4/dd/d39/d51 35395 1 2022-01-31T19:44:25.589 INFO:tasks.workunit.client.1.smithi171.stdout:3/442: creat d4/dd/d27/f83 x:0 0 0 2022-01-31T19:44:25.589 INFO:tasks.workunit.client.1.smithi171.stdout:9/503: readlink dd/l30 0 2022-01-31T19:44:25.590 INFO:tasks.workunit.client.1.smithi171.stdout:0/404: dwrite d9/db/d22/f8c [0,4194304] 0 2022-01-31T19:44:25.591 INFO:tasks.workunit.client.1.smithi171.stdout:5/402: dwrite d5/f57 [0,4194304] 0 2022-01-31T19:44:25.591 INFO:tasks.workunit.client.1.smithi171.stdout:5/403: dread - d5/d6/d15/d5d/f81 zero size 2022-01-31T19:44:25.593 INFO:tasks.workunit.client.1.smithi171.stdout:0/405: symlink d9/db/l97 0 2022-01-31T19:44:25.594 INFO:tasks.workunit.client.1.smithi171.stdout:5/404: dread d5/d6/d15/f88 [0,4194304] 0 2022-01-31T19:44:25.594 INFO:tasks.workunit.client.1.smithi171.stdout:5/405: write d5/d6/d15/d23/f41 [789829,55320] 0 2022-01-31T19:44:25.595 INFO:tasks.workunit.client.1.smithi171.stdout:0/406: read d9/f1f [1814090,40732] 0 2022-01-31T19:44:25.600 INFO:tasks.workunit.client.1.smithi171.stdout:2/528: sync 2022-01-31T19:44:25.600 INFO:tasks.workunit.client.1.smithi171.stdout:2/529: dread - d7/d26/d63/f7f zero size 2022-01-31T19:44:25.602 INFO:tasks.workunit.client.1.smithi171.stdout:0/407: mkdir d9/db/d59/d71/d98 0 2022-01-31T19:44:25.602 INFO:tasks.workunit.client.1.smithi171.stdout:0/408: dread - d9/db/d22/f95 zero size 2022-01-31T19:44:25.602 INFO:tasks.workunit.client.1.smithi171.stdout:9/504: dread dd/df/f20 [0,4194304] 0 2022-01-31T19:44:25.603 INFO:tasks.workunit.client.1.smithi171.stdout:2/530: symlink d7/d55/la9 0 2022-01-31T19:44:25.604 INFO:tasks.workunit.client.1.smithi171.stdout:0/409: creat d9/d24/d92/f99 x:0 0 0 2022-01-31T19:44:25.606 INFO:tasks.workunit.client.1.smithi171.stdout:2/531: mknod d7/d9/d1a/d45/da4/caa 0 2022-01-31T19:44:25.606 INFO:tasks.workunit.client.1.smithi171.stdout:3/443: dwrite d4/dd/d1a/d52/f54 [0,4194304] 0 2022-01-31T19:44:25.607 INFO:tasks.workunit.client.1.smithi171.stdout:3/444: dread - d4/dd/d27/d28/d35/d55/f74 zero size 2022-01-31T19:44:25.607 INFO:tasks.workunit.client.1.smithi171.stdout:3/445: creat d4/dd/d1a/f84 x:0 0 0 2022-01-31T19:44:25.608 INFO:tasks.workunit.client.1.smithi171.stdout:9/505: getdents dd/d1c 0 2022-01-31T19:44:25.608 INFO:tasks.workunit.client.1.smithi171.stdout:9/506: chown dd/df/d15/d38/d4d/d80 145555 1 2022-01-31T19:44:25.608 INFO:tasks.workunit.client.1.smithi171.stdout:9/507: read dd/d1c/f3d [1014606,54265] 0 2022-01-31T19:44:25.609 INFO:tasks.workunit.client.1.smithi171.stdout:2/532: getdents d7/dd/d5e/d6b 0 2022-01-31T19:44:25.609 INFO:tasks.workunit.client.1.smithi171.stdout:2/533: chown d7/d26/c2e 805004 1 2022-01-31T19:44:25.610 INFO:tasks.workunit.client.1.smithi171.stdout:0/410: dread d9/db/dc/d3f/d25/f52 [0,4194304] 0 2022-01-31T19:44:25.611 INFO:tasks.workunit.client.1.smithi171.stdout:0/411: write d9/d24/d92/f94 [387230,117436] 0 2022-01-31T19:44:25.611 INFO:tasks.workunit.client.1.smithi171.stdout:0/412: chown d9/db/dc/d3f/l61 8871 1 2022-01-31T19:44:25.611 INFO:tasks.workunit.client.1.smithi171.stdout:0/413: creat d9/d24/d92/f9a x:0 0 0 2022-01-31T19:44:25.612 INFO:tasks.workunit.client.1.smithi171.stdout:0/414: truncate d9/f62 906086 0 2022-01-31T19:44:25.612 INFO:tasks.workunit.client.1.smithi171.stdout:2/534: creat d7/d9/d1a/d45/da4/fab x:0 0 0 2022-01-31T19:44:25.613 INFO:tasks.workunit.client.1.smithi171.stdout:0/415: dread d9/f62 [0,4194304] 0 2022-01-31T19:44:25.615 INFO:tasks.workunit.client.1.smithi171.stdout:2/535: creat d7/dd/d5e/d6b/d81/d8d/da2/fac x:0 0 0 2022-01-31T19:44:25.615 INFO:tasks.workunit.client.1.smithi171.stdout:2/536: creat d7/dd/d5e/d6b/d81/fad x:0 0 0 2022-01-31T19:44:25.616 INFO:tasks.workunit.client.1.smithi171.stdout:0/416: symlink d9/db/dc/l9b 0 2022-01-31T19:44:25.616 INFO:tasks.workunit.client.1.smithi171.stdout:0/417: fsync d9/d24/d4c/f66 0 2022-01-31T19:44:25.617 INFO:tasks.workunit.client.1.smithi171.stdout:2/537: mknod d7/d55/cae 0 2022-01-31T19:44:25.618 INFO:tasks.workunit.client.1.smithi171.stdout:0/418: rename d9/d24/c60 to d9/db/d22/d3a/d7a/c9c 0 2022-01-31T19:44:25.618 INFO:tasks.workunit.client.1.smithi171.stdout:0/419: write d9/db/d22/d3a/f8e [1362248,8578] 0 2022-01-31T19:44:25.622 INFO:tasks.workunit.client.1.smithi171.stdout:9/508: dwrite dd/df/f62 [4194304,4194304] 0 2022-01-31T19:44:25.623 INFO:tasks.workunit.client.1.smithi171.stdout:9/509: creat dd/df/d15/d53/fa6 x:0 0 0 2022-01-31T19:44:25.640 INFO:tasks.workunit.client.1.smithi171.stdout:3/446: dread d4/d21/f69 [0,4194304] 0 2022-01-31T19:44:25.641 INFO:tasks.workunit.client.1.smithi171.stdout:3/447: chown d4/dd/d27/d28/d35/d55/f5e 0 1 2022-01-31T19:44:25.641 INFO:tasks.workunit.client.1.smithi171.stdout:3/448: creat d4/dd/d27/d2e/f85 x:0 0 0 2022-01-31T19:44:25.643 INFO:tasks.workunit.client.1.smithi171.stdout:3/449: rename d4/d21/c4d to d4/dd/d1a/d52/d7a/c86 0 2022-01-31T19:44:25.644 INFO:tasks.workunit.client.1.smithi171.stdout:3/450: mkdir d4/dd/d87 0 2022-01-31T19:44:25.645 INFO:tasks.workunit.client.1.smithi171.stdout:9/510: dread dd/f81 [0,4194304] 0 2022-01-31T19:44:25.646 INFO:tasks.workunit.client.1.smithi171.stdout:9/511: readlink dd/l5d 0 2022-01-31T19:44:25.655 INFO:tasks.workunit.client.1.smithi171.stdout:0/420: dwrite d9/db/d22/d3a/f3e [0,4194304] 0 2022-01-31T19:44:25.655 INFO:tasks.workunit.client.1.smithi171.stdout:0/421: fdatasync d9/f1e 0 2022-01-31T19:44:25.657 INFO:tasks.workunit.client.1.smithi171.stdout:0/422: mkdir d9/d24/d92/d9d 0 2022-01-31T19:44:25.657 INFO:tasks.workunit.client.1.smithi171.stdout:3/451: dread d4/dd/d1a/f1b [4194304,4194304] 0 2022-01-31T19:44:25.658 INFO:tasks.workunit.client.1.smithi171.stdout:0/423: truncate d9/db/d22/d3a/f85 363697 0 2022-01-31T19:44:25.663 INFO:tasks.workunit.client.1.smithi171.stdout:9/512: dwrite dd/d1c/d7c/d7d/fa3 [0,4194304] 0 2022-01-31T19:44:25.663 INFO:tasks.workunit.client.1.smithi171.stdout:9/513: dread dd/df/d3f/f9e [0,4194304] 0 2022-01-31T19:44:25.666 INFO:tasks.workunit.client.1.smithi171.stdout:9/514: mkdir dd/df/d15/d38/d9c/da7 0 2022-01-31T19:44:25.670 INFO:tasks.workunit.client.1.smithi171.stdout:9/515: getdents dd/df/d15/d53 0 2022-01-31T19:44:25.670 INFO:tasks.workunit.client.1.smithi171.stdout:9/516: chown dd/df/d15/d1a/d84/f99 1 1 2022-01-31T19:44:25.671 INFO:tasks.workunit.client.1.smithi171.stdout:9/517: mkdir dd/d1c/da8 0 2022-01-31T19:44:25.672 INFO:tasks.workunit.client.1.smithi171.stdout:9/518: unlink dd/df/f72 0 2022-01-31T19:44:25.675 INFO:tasks.workunit.client.1.smithi171.stdout:9/519: rename dd/df/d86/f88 to dd/d1c/d7c/d7d/fa9 0 2022-01-31T19:44:25.677 INFO:tasks.workunit.client.1.smithi171.stdout:9/520: rename dd/df/d15/d38/d4d/l54 to dd/laa 0 2022-01-31T19:44:25.681 INFO:tasks.workunit.client.1.smithi171.stdout:9/521: write dd/df/d15/f36 [1314601,103640] 0 2022-01-31T19:44:25.682 INFO:tasks.workunit.client.1.smithi171.stdout:9/522: creat dd/df/d15/d1a/d64/fab x:0 0 0 2022-01-31T19:44:25.693 INFO:tasks.workunit.client.1.smithi171.stdout:3/452: dwrite d4/dd/d1a/d52/f54 [0,4194304] 0 2022-01-31T19:44:25.693 INFO:tasks.workunit.client.1.smithi171.stdout:3/453: write d4/f7b [344656,67166] 0 2022-01-31T19:44:25.694 INFO:tasks.workunit.client.1.smithi171.stdout:0/424: dwrite d9/d24/f4d [0,4194304] 0 2022-01-31T19:44:25.695 INFO:tasks.workunit.client.1.smithi171.stdout:9/523: dread dd/f5b [0,4194304] 0 2022-01-31T19:44:25.696 INFO:tasks.workunit.client.1.smithi171.stdout:9/524: creat dd/df/d15/d1a/d64/fac x:0 0 0 2022-01-31T19:44:25.699 INFO:tasks.workunit.client.1.smithi171.stdout:3/454: dread d4/d21/f69 [0,4194304] 0 2022-01-31T19:44:25.699 INFO:tasks.workunit.client.1.smithi171.stdout:3/455: write d4/dd/d27/d2e/f85 [272512,117865] 0 2022-01-31T19:44:25.703 INFO:tasks.workunit.client.1.smithi171.stdout:9/525: dread dd/d1c/f29 [0,4194304] 0 2022-01-31T19:44:25.703 INFO:tasks.workunit.client.1.smithi171.stdout:9/526: write dd/df/fa0 [1852481,12662] 0 2022-01-31T19:44:25.704 INFO:tasks.workunit.client.1.smithi171.stdout:9/527: creat dd/df/d15/d1a/d84/fad x:0 0 0 2022-01-31T19:44:25.704 INFO:tasks.workunit.client.1.smithi171.stdout:3/456: creat d4/dd/d40/f88 x:0 0 0 2022-01-31T19:44:25.705 INFO:tasks.workunit.client.1.smithi171.stdout:0/425: write d9/db/dc/d3f/d25/f63 [590327,3838] 0 2022-01-31T19:44:25.707 INFO:tasks.workunit.client.1.smithi171.stdout:3/457: mkdir d4/dd/d39/d51/d89 0 2022-01-31T19:44:25.708 INFO:tasks.workunit.client.1.smithi171.stdout:0/426: creat d9/db/d22/d3a/d7a/f9e x:0 0 0 2022-01-31T19:44:25.712 INFO:tasks.workunit.client.1.smithi171.stdout:0/427: rmdir d9/d24 39 2022-01-31T19:44:25.712 INFO:tasks.workunit.client.1.smithi171.stdout:0/428: rmdir d9/d24/d4c/d7d 39 2022-01-31T19:44:25.713 INFO:tasks.workunit.client.1.smithi171.stdout:0/429: fdatasync d9/db/f47 0 2022-01-31T19:44:25.713 INFO:tasks.workunit.client.1.smithi171.stdout:0/430: fsync d9/d24/d4c/d35/f74 0 2022-01-31T19:44:25.714 INFO:tasks.workunit.client.1.smithi171.stdout:0/431: mkdir d9/d24/d92/d9d/d9f 0 2022-01-31T19:44:25.714 INFO:tasks.workunit.client.1.smithi171.stdout:0/432: creat d9/fa0 x:0 0 0 2022-01-31T19:44:25.714 INFO:tasks.workunit.client.1.smithi171.stdout:0/433: dread - d9/d24/f26 zero size 2022-01-31T19:44:25.716 INFO:tasks.workunit.client.1.smithi171.stdout:0/434: link d9/d24/d4c/d35/l80 d9/db/d22/d3a/d7a/la1 0 2022-01-31T19:44:25.717 INFO:tasks.workunit.client.1.smithi171.stdout:0/435: read - d9/fa0 zero size 2022-01-31T19:44:25.717 INFO:tasks.workunit.client.1.smithi171.stdout:0/436: truncate d9/db/d22/d3a/f42 4137416 0 2022-01-31T19:44:25.718 INFO:tasks.workunit.client.1.smithi171.stdout:0/437: fdatasync d9/db/f90 0 2022-01-31T19:44:25.719 INFO:tasks.workunit.client.1.smithi171.stdout:0/438: link d9/db/c20 d9/d24/d92/d9d/ca2 0 2022-01-31T19:44:25.731 INFO:tasks.workunit.client.1.smithi171.stdout:9/528: dwrite dd/df/d3f/d73/d96/d9f/fa1 [4194304,4194304] 0 2022-01-31T19:44:25.733 INFO:tasks.workunit.client.1.smithi171.stdout:9/529: creat dd/df/d3f/d73/d96/fae x:0 0 0 2022-01-31T19:44:25.734 INFO:tasks.workunit.client.1.smithi171.stdout:3/458: dwrite d4/dd/f66 [0,4194304] 0 2022-01-31T19:44:25.737 INFO:tasks.workunit.client.1.smithi171.stdout:0/439: dwrite d9/db/dc/d3f/d25/f29 [0,4194304] 0 2022-01-31T19:44:25.741 INFO:tasks.workunit.client.1.smithi171.stdout:3/459: truncate d4/dd/f19 463999 0 2022-01-31T19:44:25.741 INFO:tasks.workunit.client.1.smithi171.stdout:3/460: write d4/dd/d1a/f81 [86759,94776] 0 2022-01-31T19:44:25.743 INFO:tasks.workunit.client.1.smithi171.stdout:0/440: creat d9/db/d22/d3a/d7a/d88/fa3 x:0 0 0 2022-01-31T19:44:25.744 INFO:tasks.workunit.client.1.smithi171.stdout:0/441: rmdir d9/d24/d4c/d7d 39 2022-01-31T19:44:25.744 INFO:tasks.workunit.client.1.smithi171.stdout:0/442: dread - d9/db/d22/d3a/d7a/f9e zero size 2022-01-31T19:44:25.744 INFO:tasks.workunit.client.1.smithi171.stdout:0/443: creat d9/db/d59/d71/fa4 x:0 0 0 2022-01-31T19:44:25.745 INFO:tasks.workunit.client.1.smithi171.stdout:3/461: rename d4/d21/d3a/d59/l61 to d4/dd/d40/l8a 0 2022-01-31T19:44:25.749 INFO:tasks.workunit.client.1.smithi171.stdout:3/462: read d4/dd/d27/d4e/d62/f82 [2101987,124652] 0 2022-01-31T19:44:25.750 INFO:tasks.workunit.client.1.smithi171.stdout:0/444: mknod d9/db/dc/ca5 0 2022-01-31T19:44:25.750 INFO:tasks.workunit.client.1.smithi171.stdout:0/445: fdatasync d9/db/f21 0 2022-01-31T19:44:25.751 INFO:tasks.workunit.client.1.smithi171.stdout:3/463: creat d4/dd/d1a/d52/d7a/f8b x:0 0 0 2022-01-31T19:44:25.751 INFO:tasks.workunit.client.1.smithi171.stdout:3/464: chown d4/d21/d3a/l44 64743145 1 2022-01-31T19:44:25.751 INFO:tasks.workunit.client.1.smithi171.stdout:3/465: fdatasync d4/d21/d3a/d59/f70 0 2022-01-31T19:44:25.752 INFO:tasks.workunit.client.1.smithi171.stdout:9/530: dwrite f4 [4194304,4194304] 0 2022-01-31T19:44:25.752 INFO:tasks.workunit.client.1.smithi171.stdout:9/531: truncate dd/df/d3f/f82 812715 0 2022-01-31T19:44:25.753 INFO:tasks.workunit.client.1.smithi171.stdout:0/446: creat d9/d24/d4c/fa6 x:0 0 0 2022-01-31T19:44:25.753 INFO:tasks.workunit.client.1.smithi171.stdout:0/447: fdatasync d9/db/d59/d71/fa4 0 2022-01-31T19:44:25.754 INFO:tasks.workunit.client.1.smithi171.stdout:3/466: mknod d4/dd/c8c 0 2022-01-31T19:44:25.754 INFO:tasks.workunit.client.1.smithi171.stdout:3/467: creat d4/dd/d1a/d52/f8d x:0 0 0 2022-01-31T19:44:25.756 INFO:tasks.workunit.client.1.smithi171.stdout:0/448: unlink d9/d24/d92/f83 0 2022-01-31T19:44:25.787 INFO:tasks.workunit.client.1.smithi171.stdout:9/532: dwrite dd/df/d15/d1a/d64/fab [0,4194304] 0 2022-01-31T19:44:25.787 INFO:tasks.workunit.client.1.smithi171.stdout:9/533: chown dd/df/d15/d53/c58 2126607 1 2022-01-31T19:44:25.788 INFO:tasks.workunit.client.1.smithi171.stdout:3/468: dwrite d4/dd/d27/d28/d35/f5c [0,4194304] 0 2022-01-31T19:44:25.788 INFO:tasks.workunit.client.1.smithi171.stdout:0/449: dwrite d9/db/d22/f5b [0,4194304] 0 2022-01-31T19:44:25.789 INFO:tasks.workunit.client.1.smithi171.stdout:0/450: dread d9/f62 [0,4194304] 0 2022-01-31T19:44:25.791 INFO:tasks.workunit.client.1.smithi171.stdout:9/534: unlink dd/d1c/l39 0 2022-01-31T19:44:25.791 INFO:tasks.workunit.client.1.smithi171.stdout:9/535: creat dd/df/d15/d1a/d84/faf x:0 0 0 2022-01-31T19:44:25.791 INFO:tasks.workunit.client.1.smithi171.stdout:9/536: truncate dd/d1c/f75 4734368 0 2022-01-31T19:44:25.796 INFO:tasks.workunit.client.1.smithi171.stdout:3/469: dread d4/dd/d27/d28/d35/f38 [0,4194304] 0 2022-01-31T19:44:25.797 INFO:tasks.workunit.client.1.smithi171.stdout:0/451: rename d9/db/dc/d3f/d25/l76 to d9/db/la7 0 2022-01-31T19:44:25.799 INFO:tasks.workunit.client.1.smithi171.stdout:9/537: symlink dd/df/lb0 0 2022-01-31T19:44:25.800 INFO:tasks.workunit.client.1.smithi171.stdout:0/452: truncate d9/f1f 151 0 2022-01-31T19:44:25.801 INFO:tasks.workunit.client.1.smithi171.stdout:0/453: stat d9/db/dc/d3f/d25/f29 0 2022-01-31T19:44:25.802 INFO:tasks.workunit.client.1.smithi171.stdout:9/538: mknod dd/df/d15/d38/d9c/cb1 0 2022-01-31T19:44:25.803 INFO:tasks.workunit.client.1.smithi171.stdout:9/539: dread dd/d1c/f49 [0,4194304] 0 2022-01-31T19:44:25.803 INFO:tasks.workunit.client.1.smithi171.stdout:0/454: dread d9/db/dc/d3f/d25/f52 [0,4194304] 0 2022-01-31T19:44:25.804 INFO:tasks.workunit.client.1.smithi171.stdout:3/470: rename d4/dd/d40 to d4/dd/d87/d8e 0 2022-01-31T19:44:25.805 INFO:tasks.workunit.client.1.smithi171.stdout:3/471: getdents d4/dd/d27/d4e/d62 0 2022-01-31T19:44:25.806 INFO:tasks.workunit.client.1.smithi171.stdout:3/472: read d4/dd/d27/d28/d35/d55/f5e [382939,123492] 0 2022-01-31T19:44:25.812 INFO:tasks.workunit.client.1.smithi171.stdout:0/455: dread d9/db/d22/d3a/f42 [0,4194304] 0 2022-01-31T19:44:25.812 INFO:tasks.workunit.client.1.smithi171.stdout:0/456: chown d9/f13 18630433 1 2022-01-31T19:44:25.815 INFO:tasks.workunit.client.1.smithi171.stdout:3/473: dread d4/fb [0,4194304] 0 2022-01-31T19:44:25.815 INFO:tasks.workunit.client.1.smithi171.stdout:3/474: chown d4/dd/d27/f47 1927 1 2022-01-31T19:44:25.817 INFO:tasks.workunit.client.1.smithi171.stdout:3/475: mkdir d4/dd/d27/d28/d35/d55/d8f 0 2022-01-31T19:44:25.818 INFO:tasks.workunit.client.1.smithi171.stdout:3/476: mkdir d4/d21/d3a/d90 0 2022-01-31T19:44:25.819 INFO:tasks.workunit.client.1.smithi171.stdout:3/477: dread d4/fa [0,4194304] 0 2022-01-31T19:44:25.832 INFO:tasks.workunit.client.1.smithi171.stdout:9/540: dwrite dd/df/d15/d1a/d64/fab [0,4194304] 0 2022-01-31T19:44:25.833 INFO:tasks.workunit.client.1.smithi171.stdout:9/541: rename dd/df/d15/d1a/d84/f99 to dd/df/d15/d1a/fb2 0 2022-01-31T19:44:25.836 INFO:tasks.workunit.client.1.smithi171.stdout:9/542: write dd/df/d15/f69 [55820,118331] 0 2022-01-31T19:44:25.837 INFO:tasks.workunit.client.1.smithi171.stdout:9/543: symlink dd/df/d15/d1a/d64/d6e/lb3 0 2022-01-31T19:44:25.838 INFO:tasks.workunit.client.1.smithi171.stdout:0/457: dwrite d9/f16 [4194304,4194304] 0 2022-01-31T19:44:25.839 INFO:tasks.workunit.client.1.smithi171.stdout:0/458: chown d9/db/f41 2027886 1 2022-01-31T19:44:25.843 INFO:tasks.workunit.client.1.smithi171.stdout:3/478: dwrite d4/dd/d27/d28/d35/f5c [0,4194304] 0 2022-01-31T19:44:25.843 INFO:tasks.workunit.client.1.smithi171.stdout:3/479: stat d4/dd/d39/d51/d89 0 2022-01-31T19:44:25.843 INFO:tasks.workunit.client.1.smithi171.stdout:3/480: fsync d4/dd/d87/d8e/f42 0 2022-01-31T19:44:25.843 INFO:tasks.workunit.client.1.smithi171.stdout:3/481: chown d4/dd/d27/d2e/l32 50297355 1 2022-01-31T19:44:25.848 INFO:tasks.workunit.client.1.smithi171.stdout:3/482: dread d4/dd/d27/d28/d35/f38 [0,4194304] 0 2022-01-31T19:44:25.850 INFO:tasks.workunit.client.1.smithi171.stdout:3/483: link d4/dd/f19 d4/dd/d27/d28/f91 0 2022-01-31T19:44:25.854 INFO:tasks.workunit.client.1.smithi171.stdout:7/291: sync 2022-01-31T19:44:25.854 INFO:tasks.workunit.client.1.smithi171.stdout:6/335: sync 2022-01-31T19:44:25.854 INFO:tasks.workunit.client.1.smithi171.stdout:1/367: sync 2022-01-31T19:44:25.854 INFO:tasks.workunit.client.1.smithi171.stdout:8/363: sync 2022-01-31T19:44:25.855 INFO:tasks.workunit.client.1.smithi171.stdout:3/484: rename d4/dd/d27/d2e to d4/dd/d27/d28/d35/d55/d8f/d92 0 2022-01-31T19:44:25.855 INFO:tasks.workunit.client.1.smithi171.stdout:1/368: rmdir dd/df/d19/d1b/d1e 39 2022-01-31T19:44:25.856 INFO:tasks.workunit.client.1.smithi171.stdout:8/364: truncate df/d27/f64 4249342 0 2022-01-31T19:44:25.857 INFO:tasks.workunit.client.1.smithi171.stdout:3/485: dread d4/dd/d1a/f81 [0,4194304] 0 2022-01-31T19:44:25.858 INFO:tasks.workunit.client.1.smithi171.stdout:3/486: write d4/dd/d27/f47 [4569855,71076] 0 2022-01-31T19:44:25.858 INFO:tasks.workunit.client.1.smithi171.stdout:6/336: rename dc/d19/d29/d27/d37/d3b/c4e to dc/d19/d29/d27/d37/d3b/c66 0 2022-01-31T19:44:25.859 INFO:tasks.workunit.client.1.smithi171.stdout:7/292: mknod d1/d1e/d2a/d29/d31/d30/d3c/c62 0 2022-01-31T19:44:25.860 INFO:tasks.workunit.client.1.smithi171.stdout:8/365: creat df/d18/d19/d21/f7f x:0 0 0 2022-01-31T19:44:25.860 INFO:tasks.workunit.client.1.smithi171.stdout:8/366: stat df/d18/d19/l1f 0 2022-01-31T19:44:25.861 INFO:tasks.workunit.client.1.smithi171.stdout:8/367: fsync df/f17 0 2022-01-31T19:44:25.861 INFO:tasks.workunit.client.1.smithi171.stdout:8/368: write f7 [1070934,100277] 0 2022-01-31T19:44:25.862 INFO:tasks.workunit.client.1.smithi171.stdout:7/293: read d1/f36 [4074387,116586] 0 2022-01-31T19:44:25.862 INFO:tasks.workunit.client.1.smithi171.stdout:9/544: dwrite dd/df/d3f/f4c [0,4194304] 0 2022-01-31T19:44:25.863 INFO:tasks.workunit.client.1.smithi171.stdout:6/337: mkdir dc/d19/d29/d27/d37/d3b/d67 0 2022-01-31T19:44:25.864 INFO:tasks.workunit.client.1.smithi171.stdout:0/459: dwrite d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:25.864 INFO:tasks.workunit.client.1.smithi171.stdout:3/487: link d4/c6 d4/d21/c93 0 2022-01-31T19:44:25.864 INFO:tasks.workunit.client.1.smithi171.stdout:7/294: read d1/d1b/fc [755962,90797] 0 2022-01-31T19:44:25.870 INFO:tasks.workunit.client.1.smithi171.stdout:6/338: rmdir dc/d19/d29/d27/d37 39 2022-01-31T19:44:25.874 INFO:tasks.workunit.client.1.smithi171.stdout:3/488: rename d4/dd/f20 to d4/d21/d3a/d90/f94 0 2022-01-31T19:44:25.874 INFO:tasks.workunit.client.1.smithi171.stdout:7/295: creat d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 x:0 0 0 2022-01-31T19:44:25.876 INFO:tasks.workunit.client.1.smithi171.stdout:6/339: mkdir dc/d19/d29/d27/d37/d68 0 2022-01-31T19:44:25.878 INFO:tasks.workunit.client.1.smithi171.stdout:7/296: mknod d1/d52/c64 0 2022-01-31T19:44:25.879 INFO:tasks.workunit.client.1.smithi171.stdout:7/297: chown d1/d52/f21 41 1 2022-01-31T19:44:25.879 INFO:tasks.workunit.client.1.smithi171.stdout:7/298: chown d1/d1e/d2a/d29 19475 1 2022-01-31T19:44:25.879 INFO:tasks.workunit.client.1.smithi171.stdout:1/369: dwrite dd/df/d19/d1b/f28 [0,4194304] 0 2022-01-31T19:44:25.879 INFO:tasks.workunit.client.1.smithi171.stdout:1/370: chown dd/df/d19/d1b/c43 1574490 1 2022-01-31T19:44:25.880 INFO:tasks.workunit.client.1.smithi171.stdout:6/340: rename dc/d19/d29/d27/d37/c5e to dc/c69 0 2022-01-31T19:44:25.881 INFO:tasks.workunit.client.1.smithi171.stdout:7/299: dread d1/d26/f5c [0,4194304] 0 2022-01-31T19:44:25.881 INFO:tasks.workunit.client.1.smithi171.stdout:7/300: fdatasync d1/d1e/d2a/d29/d31/f34 0 2022-01-31T19:44:25.881 INFO:tasks.workunit.client.1.smithi171.stdout:3/489: getdents d4/dd/d39/d51 0 2022-01-31T19:44:25.882 INFO:tasks.workunit.client.1.smithi171.stdout:3/490: chown d4/dd/l76 0 1 2022-01-31T19:44:25.883 INFO:tasks.workunit.client.1.smithi171.stdout:1/371: readlink dd/d23/d34/l62 0 2022-01-31T19:44:25.884 INFO:tasks.workunit.client.1.smithi171.stdout:6/341: mkdir dc/d19/d29/d6a 0 2022-01-31T19:44:25.884 INFO:tasks.workunit.client.1.smithi171.stdout:6/342: readlink dc/d1e/l56 0 2022-01-31T19:44:25.886 INFO:tasks.workunit.client.1.smithi171.stdout:7/301: unlink d1/d1b/l15 0 2022-01-31T19:44:25.887 INFO:tasks.workunit.client.1.smithi171.stdout:1/372: mknod dd/d23/d74/c78 0 2022-01-31T19:44:25.887 INFO:tasks.workunit.client.1.smithi171.stdout:7/302: write d1/d52/f1a [2204680,11305] 0 2022-01-31T19:44:25.887 INFO:tasks.workunit.client.1.smithi171.stdout:1/373: dread - dd/f56 zero size 2022-01-31T19:44:25.887 INFO:tasks.workunit.client.1.smithi171.stdout:1/374: chown dd/f42 440287052 1 2022-01-31T19:44:25.888 INFO:tasks.workunit.client.1.smithi171.stdout:1/375: chown dd/df/c5d 294754054 1 2022-01-31T19:44:25.888 INFO:tasks.workunit.client.1.smithi171.stdout:1/376: chown dd/d23/f5c 8168849 1 2022-01-31T19:44:25.888 INFO:tasks.workunit.client.1.smithi171.stdout:7/303: write d1/d26/f5c [1828698,38711] 0 2022-01-31T19:44:25.888 INFO:tasks.workunit.client.1.smithi171.stdout:7/304: chown d1/d1e/d2a/l4a 99 1 2022-01-31T19:44:25.890 INFO:tasks.workunit.client.1.smithi171.stdout:1/377: unlink dd/df/f6d 0 2022-01-31T19:44:25.890 INFO:tasks.workunit.client.1.smithi171.stdout:1/378: unlink dd/d23/f29 0 2022-01-31T19:44:25.891 INFO:tasks.workunit.client.1.smithi171.stdout:1/379: mknod dd/d23/d34/d4b/c79 0 2022-01-31T19:44:25.892 INFO:tasks.workunit.client.1.smithi171.stdout:1/380: truncate dd/df/d19/f75 292152 0 2022-01-31T19:44:25.892 INFO:tasks.workunit.client.1.smithi171.stdout:1/381: stat dd/d23/d34/l62 0 2022-01-31T19:44:25.893 INFO:tasks.workunit.client.1.smithi171.stdout:1/382: creat dd/f7a x:0 0 0 2022-01-31T19:44:25.895 INFO:tasks.workunit.client.1.smithi171.stdout:1/383: dread dd/d23/d34/f65 [0,4194304] 0 2022-01-31T19:44:25.896 INFO:tasks.workunit.client.1.smithi171.stdout:1/384: write dd/df/d19/d1b/d1e/d35/f70 [3954249,51025] 0 2022-01-31T19:44:25.904 INFO:tasks.workunit.client.1.smithi171.stdout:6/343: dread dc/d19/d30/f3d [0,4194304] 0 2022-01-31T19:44:25.904 INFO:tasks.workunit.client.1.smithi171.stdout:0/460: dwrite d9/db/d22/d3a/f85 [0,4194304] 0 2022-01-31T19:44:25.905 INFO:tasks.workunit.client.1.smithi171.stdout:0/461: creat d9/db/d22/d3a/d7a/fa8 x:0 0 0 2022-01-31T19:44:25.905 INFO:tasks.workunit.client.1.smithi171.stdout:0/462: creat d9/d24/d4c/fa9 x:0 0 0 2022-01-31T19:44:25.905 INFO:tasks.workunit.client.1.smithi171.stdout:0/463: chown d9/d24/d4c/f77 349686105 1 2022-01-31T19:44:25.906 INFO:tasks.workunit.client.1.smithi171.stdout:6/344: mknod dc/d19/d29/d6a/c6b 0 2022-01-31T19:44:25.907 INFO:tasks.workunit.client.1.smithi171.stdout:6/345: write dc/d18/f2c [2629905,99035] 0 2022-01-31T19:44:25.907 INFO:tasks.workunit.client.1.smithi171.stdout:3/491: dwrite d4/dd/d27/d28/f91 [0,4194304] 0 2022-01-31T19:44:25.907 INFO:tasks.workunit.client.1.smithi171.stdout:6/346: write dc/d21/d4f/f60 [683958,98733] 0 2022-01-31T19:44:25.912 INFO:tasks.workunit.client.1.smithi171.stdout:8/369: dwrite df/d18/d79/f7c [0,4194304] 0 2022-01-31T19:44:25.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:24.274055+0000 mgr.smithi171.asyxnc (mgr.24387) 30 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi171.asyxnc) 2022-01-31T19:44:25.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: cluster 2022-01-31T19:44:24.435198+0000 mgr.smithi171.asyxnc (mgr.24387) 31 : cluster [DBG] pgmap v21: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 173 MiB/s rd, 338 MiB/s wr, 507 op/s 2022-01-31T19:44:25.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: audit 2022-01-31T19:44:25.662445+0000 mon.smithi167 (mon.0) 684 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:25.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: audit 2022-01-31T19:44:25.669047+0000 mon.smithi171 (mon.1) 60 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi167.aciqpk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:44:25.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: audit 2022-01-31T19:44:25.669623+0000 mon.smithi167 (mon.0) 685 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi167.aciqpk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:44:25.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: audit 2022-01-31T19:44:25.670966+0000 mon.smithi171 (mon.1) 61 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:44:25.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:25 smithi167 conmon[32206]: audit 2022-01-31T19:44:25.671901+0000 mon.smithi171 (mon.1) 62 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:25.922 INFO:tasks.workunit.client.1.smithi171.stdout:4/353: sync 2022-01-31T19:44:25.923 INFO:tasks.workunit.client.1.smithi171.stdout:2/538: sync 2022-01-31T19:44:25.923 INFO:tasks.workunit.client.1.smithi171.stdout:5/406: sync 2022-01-31T19:44:25.923 INFO:tasks.workunit.client.1.smithi171.stdout:7/305: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [0,4194304] 0 2022-01-31T19:44:25.924 INFO:tasks.workunit.client.1.smithi171.stdout:6/347: dread dc/d19/d30/d3c/f4b [0,4194304] 0 2022-01-31T19:44:25.927 INFO:tasks.workunit.client.1.smithi171.stdout:6/348: symlink dc/d21/l6c 0 2022-01-31T19:44:25.929 INFO:tasks.workunit.client.1.smithi171.stdout:2/539: rmdir d7/dd/d5e/d6b/d81/d8d/da2 39 2022-01-31T19:44:25.929 INFO:tasks.workunit.client.1.smithi171.stdout:2/540: chown d7/dd/f39 308 1 2022-01-31T19:44:25.930 INFO:tasks.workunit.client.1.smithi171.stdout:6/349: rename dc/d1e/d3f/f38 to dc/d19/d29/d27/d37/d68/f6d 0 2022-01-31T19:44:25.930 INFO:tasks.workunit.client.1.smithi171.stdout:6/350: unlink dc/c5d 0 2022-01-31T19:44:25.932 INFO:tasks.workunit.client.1.smithi171.stdout:2/541: dread d7/d9/f46 [0,4194304] 0 2022-01-31T19:44:25.932 INFO:tasks.workunit.client.1.smithi171.stdout:2/542: readlink d7/dd/d5e/d6b/l73 0 2022-01-31T19:44:25.933 INFO:tasks.workunit.client.1.smithi171.stdout:6/351: mknod dc/c6e 0 2022-01-31T19:44:25.933 INFO:tasks.workunit.client.1.smithi171.stdout:2/543: rename d7/d26/d3d/d40/c49 to d7/d9/caf 0 2022-01-31T19:44:25.936 INFO:tasks.workunit.client.1.smithi171.stdout:6/352: unlink dc/d19/f1d 0 2022-01-31T19:44:25.936 INFO:tasks.workunit.client.1.smithi171.stdout:6/353: chown dc/d19/d29/d6a/c6b 427550208 1 2022-01-31T19:44:25.937 INFO:tasks.workunit.client.1.smithi171.stdout:6/354: mknod dc/c6f 0 2022-01-31T19:44:25.938 INFO:tasks.workunit.client.1.smithi171.stdout:6/355: mknod dc/d18/d53/c70 0 2022-01-31T19:44:25.939 INFO:tasks.workunit.client.1.smithi171.stdout:6/356: mkdir dc/d19/d29/d27/d71 0 2022-01-31T19:44:25.957 INFO:tasks.workunit.client.1.smithi171.stdout:2/544: dwrite d7/f22 [0,4194304] 0 2022-01-31T19:44:25.960 INFO:tasks.workunit.client.1.smithi171.stdout:0/464: dwrite d9/d24/d92/f94 [0,4194304] 0 2022-01-31T19:44:25.961 INFO:tasks.workunit.client.1.smithi171.stdout:2/545: write d7/d9/f2c [1116417,91266] 0 2022-01-31T19:44:25.961 INFO:tasks.workunit.client.1.smithi171.stdout:3/492: dwrite d4/dd/d27/d28/f57 [0,4194304] 0 2022-01-31T19:44:25.961 INFO:tasks.workunit.client.1.smithi171.stdout:3/493: fdatasync d4/d21/d3a/d90/f94 0 2022-01-31T19:44:25.962 INFO:tasks.workunit.client.1.smithi171.stdout:6/357: dwrite dc/d19/d29/f26 [0,4194304] 0 2022-01-31T19:44:25.962 INFO:tasks.workunit.client.1.smithi171.stdout:8/370: dwrite df/d18/d19/d36/f73 [0,4194304] 0 2022-01-31T19:44:25.963 INFO:tasks.workunit.client.1.smithi171.stdout:4/354: dwrite d2/d3/d5/dc/f19 [0,4194304] 0 2022-01-31T19:44:25.963 INFO:tasks.workunit.client.1.smithi171.stdout:0/465: mknod d9/db/d22/d3a/d7a/caa 0 2022-01-31T19:44:25.964 INFO:tasks.workunit.client.1.smithi171.stdout:0/466: creat d9/db/d22/fab x:0 0 0 2022-01-31T19:44:25.968 INFO:tasks.workunit.client.1.smithi171.stdout:3/494: creat d4/dd/d27/d28/d35/d55/d8f/f95 x:0 0 0 2022-01-31T19:44:25.970 INFO:tasks.workunit.client.1.smithi171.stdout:3/495: read d4/dd/d1a/d52/f54 [3006782,22638] 0 2022-01-31T19:44:25.973 INFO:tasks.workunit.client.1.smithi171.stdout:6/358: creat dc/d19/d30/f72 x:0 0 0 2022-01-31T19:44:25.990 INFO:tasks.workunit.client.1.smithi171.stdout:2/546: dwrite d7/d26/d3d/f42 [0,4194304] 0 2022-01-31T19:44:25.992 INFO:tasks.workunit.client.1.smithi171.stdout:2/547: symlink d7/d26/d63/d71/lb0 0 2022-01-31T19:44:25.992 INFO:tasks.workunit.client.1.smithi171.stdout:2/548: truncate d7/d9/d1a/f32 5179689 0 2022-01-31T19:44:25.994 INFO:tasks.workunit.client.1.smithi171.stdout:2/549: unlink d7/d9/f46 0 2022-01-31T19:44:25.996 INFO:tasks.workunit.client.1.smithi171.stdout:2/550: rename d7/f3a to d7/dd/d93/fb1 0 2022-01-31T19:44:25.998 INFO:tasks.workunit.client.1.smithi171.stdout:2/551: mknod d7/dd/d5e/d6b/d81/d8d/da2/cb2 0 2022-01-31T19:44:25.999 INFO:tasks.workunit.client.1.smithi171.stdout:2/552: readlink d7/d26/l5a 0 2022-01-31T19:44:25.999 INFO:tasks.workunit.client.1.smithi171.stdout:2/553: creat d7/d7c/fb3 x:0 0 0 2022-01-31T19:44:26.000 INFO:tasks.workunit.client.1.smithi171.stdout:2/554: write d7/dd/f35 [2458196,89025] 0 2022-01-31T19:44:26.000 INFO:tasks.workunit.client.1.smithi171.stdout:2/555: chown d7/dd/d5e/d6b/l73 1 1 2022-01-31T19:44:26.001 INFO:tasks.workunit.client.1.smithi171.stdout:2/556: read d7/d9/d1a/f89 [2360847,13872] 0 2022-01-31T19:44:26.001 INFO:tasks.workunit.client.1.smithi171.stdout:2/557: creat d7/d9/d1a/d45/da4/fb4 x:0 0 0 2022-01-31T19:44:26.002 INFO:tasks.workunit.client.1.smithi171.stdout:2/558: fdatasync d7/dd/d5e/d6b/d81/fa8 0 2022-01-31T19:44:26.002 INFO:tasks.workunit.client.1.smithi171.stdout:2/559: rmdir d7/d7c/d83 39 2022-01-31T19:44:26.022 INFO:tasks.workunit.client.1.smithi171.stdout:0/467: dwrite d9/f53 [0,4194304] 0 2022-01-31T19:44:26.022 INFO:tasks.workunit.client.1.smithi171.stdout:2/560: dread d7/d55/f59 [0,4194304] 0 2022-01-31T19:44:26.024 INFO:tasks.workunit.client.1.smithi171.stdout:0/468: truncate d9/db/d22/d3a/f85 3096994 0 2022-01-31T19:44:26.024 INFO:tasks.workunit.client.1.smithi171.stdout:0/469: write d9/db/d59/d71/f8f [508477,101632] 0 2022-01-31T19:44:26.026 INFO:tasks.workunit.client.1.smithi171.stdout:8/371: dwrite df/d18/d79/f7c [0,4194304] 0 2022-01-31T19:44:26.027 INFO:tasks.workunit.client.1.smithi171.stdout:8/372: stat df/d18/d19/l24 0 2022-01-31T19:44:26.027 INFO:tasks.workunit.client.1.smithi171.stdout:8/373: write df/d27/f5b [192982,108050] 0 2022-01-31T19:44:26.027 INFO:tasks.workunit.client.1.smithi171.stdout:8/374: unlink f7 0 2022-01-31T19:44:26.027 INFO:tasks.workunit.client.1.smithi171.stdout:8/375: write df/d2a/f49 [1198557,83769] 0 2022-01-31T19:44:26.028 INFO:tasks.workunit.client.1.smithi171.stdout:0/470: creat d9/d24/d92/d9d/d9f/fac x:0 0 0 2022-01-31T19:44:26.028 INFO:tasks.workunit.client.1.smithi171.stdout:0/471: creat d9/db/dc/d3f/d25/fad x:0 0 0 2022-01-31T19:44:26.029 INFO:tasks.workunit.client.1.smithi171.stdout:3/496: dwrite d4/dd/d27/f83 [0,4194304] 0 2022-01-31T19:44:26.029 INFO:tasks.workunit.client.1.smithi171.stdout:3/497: chown d4/dd/f66 67434 1 2022-01-31T19:44:26.030 INFO:tasks.workunit.client.1.smithi171.stdout:6/359: dwrite dc/d19/d29/d27/f3a [0,4194304] 0 2022-01-31T19:44:26.030 INFO:tasks.workunit.client.1.smithi171.stdout:6/360: unlink dc/ce 0 2022-01-31T19:44:26.030 INFO:tasks.workunit.client.1.smithi171.stdout:4/355: dwrite d2/d3/d5/f59 [0,4194304] 0 2022-01-31T19:44:26.031 INFO:tasks.workunit.client.1.smithi171.stdout:0/472: mknod d9/d24/d92/d9d/cae 0 2022-01-31T19:44:26.037 INFO:tasks.workunit.client.1.smithi171.stdout:6/361: link dc/c16 dc/d19/d30/d3c/c73 0 2022-01-31T19:44:26.038 INFO:tasks.workunit.client.1.smithi171.stdout:2/561: dwrite d7/d9/d1a/d45/da4/fab [0,4194304] 0 2022-01-31T19:44:26.044 INFO:tasks.workunit.client.1.smithi171.stdout:4/356: creat d2/d3/d5/dc/d4b/d56/d66/f76 x:0 0 0 2022-01-31T19:44:26.048 INFO:tasks.workunit.client.1.smithi171.stdout:0/473: dwrite d9/d24/d4c/d35/f74 [0,4194304] 0 2022-01-31T19:44:26.050 INFO:tasks.workunit.client.1.smithi171.stdout:6/362: symlink dc/d19/d29/l74 0 2022-01-31T19:44:26.053 INFO:tasks.workunit.client.1.smithi171.stdout:2/562: truncate d7/f13 3045582 0 2022-01-31T19:44:26.054 INFO:tasks.workunit.client.1.smithi171.stdout:4/357: mkdir d2/d3/d5/dc/d4b/d56/d66/d77 0 2022-01-31T19:44:26.058 INFO:tasks.workunit.client.1.smithi171.stdout:0/474: creat d9/db/d22/d3a/faf x:0 0 0 2022-01-31T19:44:26.058 INFO:tasks.workunit.client.1.smithi171.stdout:0/475: stat d9/db/lf 0 2022-01-31T19:44:26.060 INFO:tasks.workunit.client.1.smithi171.stdout:0/476: read d9/d24/d4c/d35/f74 [62668,52134] 0 2022-01-31T19:44:26.060 INFO:tasks.workunit.client.1.smithi171.stdout:6/363: dread f9 [4194304,4194304] 0 2022-01-31T19:44:26.066 INFO:tasks.workunit.client.1.smithi171.stdout:2/563: mkdir d7/d7c/d83/d85/db5 0 2022-01-31T19:44:26.066 INFO:tasks.workunit.client.1.smithi171.stdout:2/564: write d7/d26/f88 [53740,28130] 0 2022-01-31T19:44:26.066 INFO:tasks.workunit.client.1.smithi171.stdout:2/565: chown d7/l33 49837040 1 2022-01-31T19:44:26.067 INFO:tasks.workunit.client.1.smithi171.stdout:2/566: truncate d7/dd/fa1 401511 0 2022-01-31T19:44:26.067 INFO:tasks.workunit.client.1.smithi171.stdout:2/567: fdatasync d7/d9/d1a/f32 0 2022-01-31T19:44:26.069 INFO:tasks.workunit.client.1.smithi171.stdout:4/358: rename d2/d3/d5/dc/d4b/f68 to d2/d3/d5/d11/d12/d15/f78 0 2022-01-31T19:44:26.071 INFO:tasks.workunit.client.1.smithi171.stdout:0/477: creat d9/d24/d4c/d35/d67/fb0 x:0 0 0 2022-01-31T19:44:26.072 INFO:tasks.workunit.client.1.smithi171.stdout:6/364: symlink dc/d19/l75 0 2022-01-31T19:44:26.072 INFO:tasks.workunit.client.1.smithi171.stdout:2/568: symlink d7/d26/d3d/lb6 0 2022-01-31T19:44:26.073 INFO:tasks.workunit.client.1.smithi171.stdout:2/569: fdatasync d7/dd/f1b 0 2022-01-31T19:44:26.074 INFO:tasks.workunit.client.1.smithi171.stdout:4/359: mknod d2/d3/d5/dc/d4b/d56/d66/d77/c79 0 2022-01-31T19:44:26.074 INFO:tasks.workunit.client.1.smithi171.stdout:4/360: creat d2/d3/d5/dc/f7a x:0 0 0 2022-01-31T19:44:26.075 INFO:tasks.workunit.client.1.smithi171.stdout:0/478: symlink d9/d24/d4c/d35/lb1 0 2022-01-31T19:44:26.076 INFO:tasks.workunit.client.1.smithi171.stdout:6/365: creat dc/d19/d29/d6a/f76 x:0 0 0 2022-01-31T19:44:26.076 INFO:tasks.workunit.client.1.smithi171.stdout:2/570: rmdir d7/d7c/d83 39 2022-01-31T19:44:26.077 INFO:tasks.workunit.client.1.smithi171.stdout:6/366: write dc/d18/f25 [626551,58020] 0 2022-01-31T19:44:26.082 INFO:tasks.workunit.client.1.smithi171.stdout:4/361: mkdir d2/d3/d5/d70/d7b 0 2022-01-31T19:44:26.082 INFO:tasks.workunit.client.1.smithi171.stdout:2/571: creat d7/dd/d5e/d6b/fb7 x:0 0 0 2022-01-31T19:44:26.083 INFO:tasks.workunit.client.1.smithi171.stdout:2/572: creat d7/dd/d5e/d6b/fb8 x:0 0 0 2022-01-31T19:44:26.083 INFO:tasks.workunit.client.1.smithi171.stdout:4/362: mkdir d2/d3/d5/d70/d7c 0 2022-01-31T19:44:26.084 INFO:tasks.workunit.client.1.smithi171.stdout:2/573: mkdir d7/d7c/d83/d85/db9 0 2022-01-31T19:44:26.099 INFO:tasks.workunit.client.1.smithi171.stdout:0/479: dwrite d9/db/dc/f70 [0,4194304] 0 2022-01-31T19:44:26.099 INFO:tasks.workunit.client.1.smithi171.stdout:0/480: stat d9/d24/d4c/d35 0 2022-01-31T19:44:26.102 INFO:tasks.workunit.client.1.smithi171.stdout:0/481: symlink d9/d24/d4c/d7d/lb2 0 2022-01-31T19:44:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:24.274055+0000 2022-01-31T19:44:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: mgr.smithi171.asyxnc (mgr.24387) 30 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi171.asyxnc) 2022-01-31T19:44:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: cluster 2022-01-31T19:44:24.435198+0000 mgr.smithi171.asyxnc (mgr.24387) 31 : cluster [DBG] pgmap v21: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 173 MiB/s rd, 338 MiB/s wr, 507 op/s 2022-01-31T19:44:26.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: audit 2022-01-31T19:44 2022-01-31T19:44:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: :25.662445+0000 mon.smithi167 (mon.0) 2022-01-31T19:44:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: 684 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: audit 2022-01-31T19: 2022-01-31T19:44:26.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: 44:25.669047+0000 mon.smithi171 (mon.1) 60 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi167.aciqpk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:44:26.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: audit 2022-01-31T19:44:26.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: 2022-01-31T19:44:25.669623+0000 mon.smithi167 (mon.0) 685 : audit [INF] 2022-01-31T19:44:26.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi167.aciqpk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:44:26.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: audit 2022-01-31T19:44:25 2022-01-31T19:44:26.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: .670966+0000 mon.smithi171 (mon.1) 61 : audit [DBG] 2022-01-31T19:44:26.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:44:26.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: audit 2022-01-31T 2022-01-31T19:44:26.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: 19:44:25.671901+0000 mon.smithi171 (mon.1) 62 2022-01-31T19:44:26.112 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:25 smithi171 conmon[35325]: : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:26.113 INFO:tasks.workunit.client.1.smithi171.stdout:6/367: dwrite dc/d19/d30/f72 [0,4194304] 0 2022-01-31T19:44:26.113 INFO:tasks.workunit.client.1.smithi171.stdout:4/363: dwrite d2/d3/d5/dc/f26 [4194304,4194304] 0 2022-01-31T19:44:26.122 INFO:tasks.workunit.client.1.smithi171.stdout:0/482: dwrite d9/db/d22/d3a/f8e [0,4194304] 0 2022-01-31T19:44:26.125 INFO:tasks.workunit.client.1.smithi171.stdout:0/483: creat d9/d24/fb3 x:0 0 0 2022-01-31T19:44:26.126 INFO:tasks.workunit.client.1.smithi171.stdout:0/484: symlink d9/d24/d92/lb4 0 2022-01-31T19:44:26.126 INFO:tasks.workunit.client.1.smithi171.stdout:0/485: creat d9/db/d59/d71/fb5 x:0 0 0 2022-01-31T19:44:26.131 INFO:tasks.workunit.client.1.smithi171.stdout:6/368: dwrite dc/d19/d29/f34 [0,4194304] 0 2022-01-31T19:44:26.131 INFO:tasks.workunit.client.1.smithi171.stdout:6/369: write dc/d1e/d3f/f59 [1016231,88591] 0 2022-01-31T19:44:26.133 INFO:tasks.workunit.client.1.smithi171.stdout:0/486: rmdir d9/d24 39 2022-01-31T19:44:26.134 INFO:tasks.workunit.client.1.smithi171.stdout:0/487: write d9/db/dc/d3f/f6d [260918,12899] 0 2022-01-31T19:44:26.151 INFO:tasks.workunit.client.1.smithi171.stdout:6/370: dwrite dc/d19/d29/f5c [4194304,4194304] 0 2022-01-31T19:44:26.151 INFO:tasks.workunit.client.1.smithi171.stdout:6/371: stat dc/d19/d30 0 2022-01-31T19:44:26.152 INFO:tasks.workunit.client.1.smithi171.stdout:6/372: truncate dc/d19/d29/f26 2216726 0 2022-01-31T19:44:26.156 INFO:tasks.workunit.client.1.smithi171.stdout:0/488: dwrite d9/d24/d92/f9a [0,4194304] 0 2022-01-31T19:44:26.157 INFO:tasks.workunit.client.1.smithi171.stdout:6/373: dread dc/d19/d29/d27/d37/d68/f6d [0,4194304] 0 2022-01-31T19:44:26.157 INFO:tasks.workunit.client.1.smithi171.stdout:6/374: write dc/f17 [349050,68118] 0 2022-01-31T19:44:26.159 INFO:tasks.workunit.client.1.smithi171.stdout:0/489: symlink d9/db/d22/d3a/d7a/d88/lb6 0 2022-01-31T19:44:26.160 INFO:tasks.workunit.client.1.smithi171.stdout:6/375: truncate dc/d19/d29/f24 230973 0 2022-01-31T19:44:26.162 INFO:tasks.workunit.client.1.smithi171.stdout:0/490: rename d9/db/d22/f8c to d9/d24/d92/d9d/d9f/fb7 0 2022-01-31T19:44:26.164 INFO:tasks.workunit.client.1.smithi171.stdout:0/491: mkdir d9/db/dc/d3f/d6f/db8 0 2022-01-31T19:44:26.164 INFO:tasks.workunit.client.1.smithi171.stdout:0/492: getdents d9/d24/d92 0 2022-01-31T19:44:26.170 INFO:tasks.workunit.client.1.smithi171.stdout:6/376: dread dc/d19/d29/f5c [0,4194304] 0 2022-01-31T19:44:26.171 INFO:tasks.workunit.client.1.smithi171.stdout:0/493: rename d9/db/d22/d3a to d9/db/dc/d3f/db9 0 2022-01-31T19:44:26.172 INFO:tasks.workunit.client.1.smithi171.stdout:0/494: rmdir d9/db/dc/d3f/d6f/db8 0 2022-01-31T19:44:26.173 INFO:tasks.workunit.client.1.smithi171.stdout:0/495: mkdir d9/db/d59/d71/dba 0 2022-01-31T19:44:26.173 INFO:tasks.workunit.client.1.smithi171.stdout:0/496: chown d9/db/dc/d3f/d25/l54 247150 1 2022-01-31T19:44:26.183 INFO:tasks.workunit.client.1.smithi171.stdout:6/377: dread dc/d1e/d3f/f65 [0,4194304] 0 2022-01-31T19:44:26.193 INFO:tasks.workunit.client.1.smithi171.stdout:7/306: sync 2022-01-31T19:44:26.194 INFO:tasks.workunit.client.1.smithi171.stdout:1/385: sync 2022-01-31T19:44:26.194 INFO:tasks.workunit.client.1.smithi171.stdout:9/545: sync 2022-01-31T19:44:26.194 INFO:tasks.workunit.client.1.smithi171.stdout:9/546: fsync dd/df/fa0 0 2022-01-31T19:44:26.194 INFO:tasks.workunit.client.1.smithi171.stdout:7/307: fsync d1/d1b/f4c 0 2022-01-31T19:44:26.195 INFO:tasks.workunit.client.1.smithi171.stdout:7/308: unlink d1/d1e/d2a/d29/d31/f19 0 2022-01-31T19:44:26.196 INFO:tasks.workunit.client.1.smithi171.stdout:0/497: dwrite d9/d24/d92/f99 [0,4194304] 0 2022-01-31T19:44:26.196 INFO:tasks.workunit.client.1.smithi171.stdout:0/498: chown d9/d24/d92/lb4 1590 1 2022-01-31T19:44:26.196 INFO:tasks.workunit.client.1.smithi171.stdout:5/407: sync 2022-01-31T19:44:26.196 INFO:tasks.workunit.client.1.smithi171.stdout:5/408: creat d5/f89 x:0 0 0 2022-01-31T19:44:26.198 INFO:tasks.workunit.client.1.smithi171.stdout:9/547: creat dd/df/d15/d38/d41/fb4 x:0 0 0 2022-01-31T19:44:26.198 INFO:tasks.workunit.client.1.smithi171.stdout:9/548: readlink dd/df/d15/d38/d41/l42 0 2022-01-31T19:44:26.198 INFO:tasks.workunit.client.1.smithi171.stdout:1/386: dread dd/d23/d2e/f2f [0,4194304] 0 2022-01-31T19:44:26.199 INFO:tasks.workunit.client.1.smithi171.stdout:1/387: stat dd/d23/d2e/d5e/f20 0 2022-01-31T19:44:26.199 INFO:tasks.workunit.client.1.smithi171.stdout:1/388: readlink dd/d23/d2e/l6f 0 2022-01-31T19:44:26.199 INFO:tasks.workunit.client.1.smithi171.stdout:1/389: chown f7 668309 1 2022-01-31T19:44:26.199 INFO:tasks.workunit.client.1.smithi171.stdout:7/309: mknod d1/d1b/c65 0 2022-01-31T19:44:26.200 INFO:tasks.workunit.client.1.smithi171.stdout:0/499: mkdir d9/db/d59/d71/dba/dbb 0 2022-01-31T19:44:26.200 INFO:tasks.workunit.client.1.smithi171.stdout:0/500: chown d9/d24/d92/d9d/d9f 1 1 2022-01-31T19:44:26.201 INFO:tasks.workunit.client.1.smithi171.stdout:0/501: readlink d9/db/l97 0 2022-01-31T19:44:26.201 INFO:tasks.workunit.client.1.smithi171.stdout:0/502: chown d9/db/dc/d3f/db9/d7a/f9e 0 1 2022-01-31T19:44:26.201 INFO:tasks.workunit.client.1.smithi171.stdout:6/378: dwrite dc/d21/d4f/f60 [0,4194304] 0 2022-01-31T19:44:26.202 INFO:tasks.workunit.client.1.smithi171.stdout:5/409: link d5/d6/d15/d23/d6b/l75 d5/d6/d15/d23/d67/l8a 0 2022-01-31T19:44:26.203 INFO:tasks.workunit.client.1.smithi171.stdout:7/310: creat d1/d1e/d2a/d29/f66 x:0 0 0 2022-01-31T19:44:26.207 INFO:tasks.workunit.client.1.smithi171.stdout:0/503: unlink d9/db/dc/d3f/db9/c5a 0 2022-01-31T19:44:26.209 INFO:tasks.workunit.client.1.smithi171.stdout:6/379: link dc/d18/f2a dc/d19/d30/f77 0 2022-01-31T19:44:26.211 INFO:tasks.workunit.client.1.smithi171.stdout:5/410: rename d5/l47 to d5/d6/d15/d23/l8b 0 2022-01-31T19:44:26.213 INFO:tasks.workunit.client.1.smithi171.stdout:1/390: dwrite dd/df/d19/d1b/d1e/d35/f70 [0,4194304] 0 2022-01-31T19:44:26.214 INFO:tasks.workunit.client.1.smithi171.stdout:0/504: creat d9/d24/d4c/d35/fbc x:0 0 0 2022-01-31T19:44:26.216 INFO:tasks.workunit.client.1.smithi171.stdout:6/380: mknod dc/d18/c78 0 2022-01-31T19:44:26.219 INFO:tasks.workunit.client.1.smithi171.stdout:1/391: mkdir dd/d23/d74/d7b 0 2022-01-31T19:44:26.219 INFO:tasks.workunit.client.1.smithi171.stdout:1/392: write dd/d23/f57 [986236,57588] 0 2022-01-31T19:44:26.221 INFO:tasks.workunit.client.1.smithi171.stdout:6/381: rmdir dc/d18/d53 39 2022-01-31T19:44:26.226 INFO:tasks.workunit.client.1.smithi171.stdout:6/382: rename dc/d21/l6c to dc/d18/d53/l79 0 2022-01-31T19:44:26.227 INFO:tasks.workunit.client.1.smithi171.stdout:6/383: write dc/d21/d4f/f5f [15173,17396] 0 2022-01-31T19:44:26.227 INFO:tasks.workunit.client.1.smithi171.stdout:6/384: creat dc/d1e/d3f/f7a x:0 0 0 2022-01-31T19:44:26.229 INFO:tasks.workunit.client.1.smithi171.stdout:7/311: dwrite d1/f36 [0,4194304] 0 2022-01-31T19:44:26.231 INFO:tasks.workunit.client.1.smithi171.stdout:6/385: write dc/f13 [4311088,25850] 0 2022-01-31T19:44:26.231 INFO:tasks.workunit.client.1.smithi171.stdout:6/386: fdatasync dc/d19/d30/f77 0 2022-01-31T19:44:26.247 INFO:tasks.workunit.client.1.smithi171.stdout:0/505: dwrite d9/f8a [0,4194304] 0 2022-01-31T19:44:26.247 INFO:tasks.workunit.client.1.smithi171.stdout:1/393: dwrite dd/df/d19/f75 [0,4194304] 0 2022-01-31T19:44:26.248 INFO:tasks.workunit.client.1.smithi171.stdout:1/394: unlink dd/df/l61 0 2022-01-31T19:44:26.249 INFO:tasks.workunit.client.1.smithi171.stdout:0/506: truncate d9/f1d 2267213 0 2022-01-31T19:44:26.249 INFO:tasks.workunit.client.1.smithi171.stdout:5/411: dwrite d5/d6/d15/d5d/d6a/f73 [0,4194304] 0 2022-01-31T19:44:26.251 INFO:tasks.workunit.client.1.smithi171.stdout:7/312: dwrite d1/d1b/f4b [0,4194304] 0 2022-01-31T19:44:26.252 INFO:tasks.workunit.client.1.smithi171.stdout:1/395: mkdir dd/d23/d2e/d7c 0 2022-01-31T19:44:26.252 INFO:tasks.workunit.client.1.smithi171.stdout:6/387: dwrite dc/d18/f1b [4194304,4194304] 0 2022-01-31T19:44:26.254 INFO:tasks.workunit.client.1.smithi171.stdout:0/507: mknod d9/d24/d92/d9d/cbd 0 2022-01-31T19:44:26.256 INFO:tasks.workunit.client.1.smithi171.stdout:5/412: mknod d5/d6/c8c 0 2022-01-31T19:44:26.263 INFO:tasks.workunit.client.1.smithi171.stdout:7/313: link d1/d1e/d2a/d29/d31/d30/d3c/f45 d1/d1e/d2a/d29/d31/f67 0 2022-01-31T19:44:26.263 INFO:tasks.workunit.client.1.smithi171.stdout:1/396: getdents dd/df/d19/d36 0 2022-01-31T19:44:26.265 INFO:tasks.workunit.client.1.smithi171.stdout:0/508: mkdir d9/db/d59/d71/dbe 0 2022-01-31T19:44:26.265 INFO:tasks.workunit.client.1.smithi171.stdout:0/509: chown d9/db/dc/d3f/db9/d7a/caa 3 1 2022-01-31T19:44:26.266 INFO:tasks.workunit.client.1.smithi171.stdout:5/413: mkdir d5/d6/d15/d80/d8d 0 2022-01-31T19:44:26.267 INFO:tasks.workunit.client.1.smithi171.stdout:7/314: symlink d1/d26/d57/l68 0 2022-01-31T19:44:26.268 INFO:tasks.workunit.client.1.smithi171.stdout:7/315: chown d1/d26/l32 54405278 1 2022-01-31T19:44:26.269 INFO:tasks.workunit.client.1.smithi171.stdout:5/414: dread d5/f37 [0,4194304] 0 2022-01-31T19:44:26.270 INFO:tasks.workunit.client.1.smithi171.stdout:1/397: rmdir dd/d73 0 2022-01-31T19:44:26.270 INFO:tasks.workunit.client.1.smithi171.stdout:1/398: readlink dd/df/d19/l76 0 2022-01-31T19:44:26.271 INFO:tasks.workunit.client.1.smithi171.stdout:7/316: creat d1/d1e/d2a/d58/f69 x:0 0 0 2022-01-31T19:44:26.271 INFO:tasks.workunit.client.1.smithi171.stdout:5/415: symlink d5/d59/l8e 0 2022-01-31T19:44:26.272 INFO:tasks.workunit.client.1.smithi171.stdout:1/399: write dd/df/d19/f75 [1010964,104] 0 2022-01-31T19:44:26.272 INFO:tasks.workunit.client.1.smithi171.stdout:6/388: dwrite dc/d21/f4c [0,4194304] 0 2022-01-31T19:44:26.275 INFO:tasks.workunit.client.1.smithi171.stdout:7/317: rename d1/d1b/f3a to d1/d1e/d2a/d29/d31/f6a 0 2022-01-31T19:44:26.281 INFO:tasks.workunit.client.1.smithi171.stdout:5/416: rename d5/d6/d15/d5d/d6a/c86 to d5/d6/d15/d5d/d6a/c8f 0 2022-01-31T19:44:26.282 INFO:tasks.workunit.client.1.smithi171.stdout:5/417: truncate d5/d6/d15/d5d/d6a/f77 550024 0 2022-01-31T19:44:26.282 INFO:tasks.workunit.client.1.smithi171.stdout:6/389: dread dc/d19/d30/f72 [0,4194304] 0 2022-01-31T19:44:26.283 INFO:tasks.workunit.client.1.smithi171.stdout:1/400: creat dd/df/d19/d1b/d1e/d35/f7d x:0 0 0 2022-01-31T19:44:26.285 INFO:tasks.workunit.client.1.smithi171.stdout:7/318: mkdir d1/d1e/d2a/d29/d31/d30/d6b 0 2022-01-31T19:44:26.285 INFO:tasks.workunit.client.1.smithi171.stdout:7/319: truncate d1/d1e/d2a/d29/d31/f34 993684 0 2022-01-31T19:44:26.285 INFO:tasks.workunit.client.1.smithi171.stdout:0/510: dwrite d9/db/dc/d3f/db9/f85 [0,4194304] 0 2022-01-31T19:44:26.286 INFO:tasks.workunit.client.1.smithi171.stdout:6/390: write dc/d19/d29/f5c [5135529,129945] 0 2022-01-31T19:44:26.288 INFO:tasks.workunit.client.1.smithi171.stdout:1/401: mknod dd/df/d19/d1b/d1e/d35/c7e 0 2022-01-31T19:44:26.289 INFO:tasks.workunit.client.1.smithi171.stdout:7/320: symlink d1/d1e/d2a/d29/d31/l6c 0 2022-01-31T19:44:26.289 INFO:tasks.workunit.client.1.smithi171.stdout:7/321: creat d1/d1e/d2a/d29/f6d x:0 0 0 2022-01-31T19:44:26.290 INFO:tasks.workunit.client.1.smithi171.stdout:7/322: fsync d1/d1e/d2a/d29/d31/d30/f5a 0 2022-01-31T19:44:26.294 INFO:tasks.workunit.client.1.smithi171.stdout:5/418: rename d5/d6/d74 to d5/d90 0 2022-01-31T19:44:26.294 INFO:tasks.workunit.client.1.smithi171.stdout:5/419: creat d5/d6/d15/d23/d58/f91 x:0 0 0 2022-01-31T19:44:26.296 INFO:tasks.workunit.client.1.smithi171.stdout:0/511: rename d9/f53 to d9/d24/d92/fbf 0 2022-01-31T19:44:26.297 INFO:tasks.workunit.client.1.smithi171.stdout:0/512: symlink d9/lc0 0 2022-01-31T19:44:26.298 INFO:tasks.workunit.client.1.smithi171.stdout:0/513: mknod d9/db/dc/cc1 0 2022-01-31T19:44:26.301 INFO:tasks.workunit.client.1.smithi171.stdout:0/514: rename d9/db/dc/d3f/d25 to d9/db/dc2 0 2022-01-31T19:44:26.302 INFO:tasks.workunit.client.1.smithi171.stdout:0/515: creat d9/db/d59/fc3 x:0 0 0 2022-01-31T19:44:26.314 INFO:tasks.workunit.client.1.smithi171.stdout:1/402: dwrite dd/d23/d2e/d5e/f20 [0,4194304] 0 2022-01-31T19:44:26.315 INFO:tasks.workunit.client.1.smithi171.stdout:5/420: dwrite d5/d6/d15/d23/f87 [0,4194304] 0 2022-01-31T19:44:26.319 INFO:tasks.workunit.client.1.smithi171.stdout:0/516: rename d9/d24/d92/fbf to d9/db/fc4 0 2022-01-31T19:44:26.320 INFO:tasks.workunit.client.1.smithi171.stdout:0/517: write d9/f62 [203981,72768] 0 2022-01-31T19:44:26.323 INFO:tasks.workunit.client.1.smithi171.stdout:1/403: dread dd/df/d19/d1b/f4e [0,4194304] 0 2022-01-31T19:44:26.323 INFO:tasks.workunit.client.1.smithi171.stdout:1/404: creat dd/f7f x:0 0 0 2022-01-31T19:44:26.325 INFO:tasks.workunit.client.1.smithi171.stdout:0/518: truncate d9/db/dc/d3f/f2d 5900397 0 2022-01-31T19:44:26.325 INFO:tasks.workunit.client.1.smithi171.stdout:5/421: link d5/d6/d15/d23/d58/c61 d5/d6/d15/d5d/d6f/c92 0 2022-01-31T19:44:26.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/405: dread dd/df/d19/d1b/f49 [0,4194304] 0 2022-01-31T19:44:26.328 INFO:tasks.workunit.client.1.smithi171.stdout:1/406: unlink dd/d23/d34/d4b/c79 0 2022-01-31T19:44:26.329 INFO:tasks.workunit.client.1.smithi171.stdout:5/422: getdents d5/d6/d15 0 2022-01-31T19:44:26.330 INFO:tasks.workunit.client.1.smithi171.stdout:5/423: truncate d5/f89 604458 0 2022-01-31T19:44:26.344 INFO:tasks.workunit.client.1.smithi171.stdout:8/376: sync 2022-01-31T19:44:26.348 INFO:tasks.workunit.client.1.smithi171.stdout:0/519: dread f4 [0,4194304] 0 2022-01-31T19:44:26.354 INFO:tasks.workunit.client.1.smithi171.stdout:1/407: dwrite dd/d23/f5c [0,4194304] 0 2022-01-31T19:44:26.355 INFO:tasks.workunit.client.1.smithi171.stdout:1/408: mkdir dd/df/d19/d1b/d77/d80 0 2022-01-31T19:44:26.358 INFO:tasks.workunit.client.1.smithi171.stdout:1/409: dread dd/df/d19/d1b/f49 [0,4194304] 0 2022-01-31T19:44:26.361 INFO:tasks.workunit.client.1.smithi171.stdout:1/410: rmdir dd/d23/d2e/d7c 0 2022-01-31T19:44:26.364 INFO:tasks.workunit.client.1.smithi171.stdout:5/424: dwrite d5/f4e [4194304,4194304] 0 2022-01-31T19:44:26.366 INFO:tasks.workunit.client.1.smithi171.stdout:5/425: mknod d5/d59/c93 0 2022-01-31T19:44:26.366 INFO:tasks.workunit.client.1.smithi171.stdout:0/520: dread d9/db/dc/d3f/f34 [0,4194304] 0 2022-01-31T19:44:26.368 INFO:tasks.workunit.client.0.smithi167.stdout:2/937: sync 2022-01-31T19:44:26.368 INFO:tasks.workunit.client.0.smithi167.stdout:3/999: sync 2022-01-31T19:44:26.368 INFO:tasks.workunit.client.0.smithi167.stdout:6/848: sync 2022-01-31T19:44:26.369 INFO:tasks.workunit.client.0.smithi167.stdout:0/845: sync 2022-01-31T19:44:26.369 INFO:tasks.workunit.client.0.smithi167.stdout:0/846: chown d17/d2c/lac 1183701 1 2022-01-31T19:44:26.372 INFO:tasks.workunit.client.1.smithi171.stdout:8/377: dwrite df/d5d/d6c/f76 [0,4194304] 0 2022-01-31T19:44:26.378 INFO:tasks.workunit.client.1.smithi171.stdout:8/378: write fb [3962872,110633] 0 2022-01-31T19:44:26.380 INFO:tasks.workunit.client.1.smithi171.stdout:8/379: write df/d2a/d38/f56 [2894823,20441] 0 2022-01-31T19:44:26.382 INFO:tasks.workunit.client.1.smithi171.stdout:8/380: mknod df/d18/d79/c80 0 2022-01-31T19:44:26.384 INFO:tasks.workunit.client.1.smithi171.stdout:8/381: getdents df/d2a 0 2022-01-31T19:44:26.384 INFO:tasks.workunit.client.1.smithi171.stdout:8/382: creat df/d18/d19/d36/f81 x:0 0 0 2022-01-31T19:44:26.385 INFO:tasks.workunit.client.1.smithi171.stdout:8/383: mkdir df/d18/d19/d82 0 2022-01-31T19:44:26.387 INFO:tasks.workunit.client.1.smithi171.stdout:8/384: rename df/d5d/d6c/f6e to df/f83 0 2022-01-31T19:44:26.388 INFO:tasks.workunit.client.1.smithi171.stdout:8/385: creat df/f84 x:0 0 0 2022-01-31T19:44:26.393 INFO:tasks.workunit.client.1.smithi171.stdout:5/426: dwrite d5/d6/d15/f1d [4194304,4194304] 0 2022-01-31T19:44:26.393 INFO:tasks.workunit.client.1.smithi171.stdout:5/427: fsync d5/d6/d15/d23/d67/f76 0 2022-01-31T19:44:26.395 INFO:tasks.workunit.client.1.smithi171.stdout:5/428: creat d5/d59/f94 x:0 0 0 2022-01-31T19:44:26.399 INFO:tasks.workunit.client.1.smithi171.stdout:1/411: dwrite dd/d23/f57 [0,4194304] 0 2022-01-31T19:44:26.400 INFO:tasks.workunit.client.1.smithi171.stdout:0/521: dwrite d9/db/d59/fc3 [0,4194304] 0 2022-01-31T19:44:26.402 INFO:tasks.workunit.client.1.smithi171.stdout:1/412: mkdir dd/d23/d2e/d5e/d81 0 2022-01-31T19:44:26.402 INFO:tasks.workunit.client.1.smithi171.stdout:1/413: creat dd/d23/f82 x:0 0 0 2022-01-31T19:44:26.402 INFO:tasks.workunit.client.1.smithi171.stdout:1/414: chown dd/d23/f60 59518 1 2022-01-31T19:44:26.407 INFO:tasks.workunit.client.1.smithi171.stdout:0/522: rename f4 to d9/db/dc2/fc5 0 2022-01-31T19:44:26.407 INFO:tasks.workunit.client.1.smithi171.stdout:0/523: chown d9/d24/d92/l68 888016 1 2022-01-31T19:44:26.410 INFO:tasks.workunit.client.1.smithi171.stdout:8/386: dwrite df/d1a/f29 [0,4194304] 0 2022-01-31T19:44:26.414 INFO:tasks.workunit.client.1.smithi171.stdout:8/387: mkdir df/d27/d85 0 2022-01-31T19:44:26.418 INFO:tasks.workunit.client.1.smithi171.stdout:8/388: write df/d18/f42 [1475087,72497] 0 2022-01-31T19:44:26.420 INFO:tasks.workunit.client.1.smithi171.stdout:1/415: dwrite dd/d23/d2e/f2f [0,4194304] 0 2022-01-31T19:44:26.425 INFO:tasks.workunit.client.1.smithi171.stdout:1/416: mkdir dd/d23/d2e/d5e/d83 0 2022-01-31T19:44:26.426 INFO:tasks.workunit.client.1.smithi171.stdout:1/417: mkdir dd/df/d84 0 2022-01-31T19:44:26.426 INFO:tasks.workunit.client.1.smithi171.stdout:1/418: write dd/d23/f82 [182026,21008] 0 2022-01-31T19:44:26.426 INFO:tasks.workunit.client.1.smithi171.stdout:1/419: write dd/d23/d34/d4b/f59 [883459,1394] 0 2022-01-31T19:44:26.427 INFO:tasks.workunit.client.1.smithi171.stdout:1/420: write dd/df/d19/d1b/d1e/f6a [954559,125701] 0 2022-01-31T19:44:26.427 INFO:tasks.workunit.client.1.smithi171.stdout:1/421: write dd/df/d19/d1b/d1e/f6a [1532368,97034] 0 2022-01-31T19:44:26.429 INFO:tasks.workunit.client.1.smithi171.stdout:0/524: dwrite d9/db/dc/d3f/db9/f8e [0,4194304] 0 2022-01-31T19:44:26.433 INFO:tasks.workunit.client.1.smithi171.stdout:0/525: mkdir d9/d24/d92/d9d/dc6 0 2022-01-31T19:44:26.433 INFO:tasks.workunit.client.1.smithi171.stdout:0/526: readlink d9/db/d22/l43 0 2022-01-31T19:44:26.433 INFO:tasks.workunit.client.1.smithi171.stdout:0/527: chown d9/db/dc 7 1 2022-01-31T19:44:26.433 INFO:tasks.workunit.client.1.smithi171.stdout:0/528: fsync d9/db/dc/d3f/db9/d7a/f9e 0 2022-01-31T19:44:26.434 INFO:tasks.workunit.client.1.smithi171.stdout:0/529: creat d9/db/d22/fc7 x:0 0 0 2022-01-31T19:44:26.434 INFO:tasks.workunit.client.1.smithi171.stdout:0/530: fsync f8 0 2022-01-31T19:44:26.438 INFO:tasks.workunit.client.1.smithi171.stdout:8/389: dwrite df/d18/d19/d21/f32 [0,4194304] 0 2022-01-31T19:44:26.446 INFO:tasks.workunit.client.1.smithi171.stdout:8/390: dread df/d18/d19/d21/f26 [0,4194304] 0 2022-01-31T19:44:26.451 INFO:tasks.workunit.client.1.smithi171.stdout:8/391: symlink df/d18/d79/l86 0 2022-01-31T19:44:26.454 INFO:tasks.workunit.client.1.smithi171.stdout:8/392: mkdir df/d5d/d6c/d87 0 2022-01-31T19:44:26.454 INFO:tasks.workunit.client.1.smithi171.stdout:8/393: fdatasync df/f84 0 2022-01-31T19:44:26.454 INFO:tasks.workunit.client.1.smithi171.stdout:8/394: fsync df/d18/d19/d36/f6d 0 2022-01-31T19:44:26.455 INFO:tasks.workunit.client.1.smithi171.stdout:3/498: sync 2022-01-31T19:44:26.455 INFO:tasks.workunit.client.1.smithi171.stdout:3/499: write d4/dd/d1a/d52/d7a/f8b [650741,24406] 0 2022-01-31T19:44:26.455 INFO:tasks.workunit.client.1.smithi171.stdout:3/500: creat d4/dd/d1a/d52/f96 x:0 0 0 2022-01-31T19:44:26.456 INFO:tasks.workunit.client.1.smithi171.stdout:3/501: link d4/dd/d87/d8e/l8a d4/dd/d27/d6e/l97 0 2022-01-31T19:44:26.457 INFO:tasks.workunit.client.1.smithi171.stdout:8/395: truncate df/d18/d19/d21/f32 2032950 0 2022-01-31T19:44:26.458 INFO:tasks.workunit.client.1.smithi171.stdout:3/502: mknod d4/dd/d39/c98 0 2022-01-31T19:44:26.458 INFO:tasks.workunit.client.1.smithi171.stdout:8/396: mkdir df/d18/d19/d21/d88 0 2022-01-31T19:44:26.460 INFO:tasks.workunit.client.1.smithi171.stdout:3/503: getdents d4/dd/d1a 0 2022-01-31T19:44:26.460 INFO:tasks.workunit.client.1.smithi171.stdout:3/504: unlink d4/fc 0 2022-01-31T19:44:26.460 INFO:tasks.workunit.client.1.smithi171.stdout:3/505: chown d4/dd/d27/d28/d35/d55/d8f/d92/f78 5538591 1 2022-01-31T19:44:26.463 INFO:tasks.workunit.client.1.smithi171.stdout:3/506: creat d4/f99 x:0 0 0 2022-01-31T19:44:26.465 INFO:tasks.workunit.client.0.smithi167.stdout:6/849: symlink d8/d33/d51/d7e/dc4/d112/l126 0 2022-01-31T19:44:26.467 INFO:tasks.workunit.client.0.smithi167.stdout:0/847: mkdir d17/d33/d11c 0 2022-01-31T19:44:26.468 INFO:tasks.workunit.client.0.smithi167.stdout:0/848: write d17/d2c/d87/df4/f10a [782850,107905] 0 2022-01-31T19:44:26.469 INFO:tasks.workunit.client.0.smithi167.stdout:6/850: creat d8/d33/d51/d5d/d64/f127 x:0 0 0 2022-01-31T19:44:26.469 INFO:tasks.workunit.client.0.smithi167.stdout:0/849: write d17/d2c/f37 [40843,15386] 0 2022-01-31T19:44:26.471 INFO:tasks.workunit.client.0.smithi167.stdout:6/851: creat d8/d33/d51/d7e/dc4/f128 x:0 0 0 2022-01-31T19:44:26.473 INFO:tasks.workunit.client.0.smithi167.stdout:0/850: unlink d17/d19/d1c/d39/d44/c45 0 2022-01-31T19:44:26.474 INFO:tasks.workunit.client.0.smithi167.stdout:6/852: symlink d8/d33/d51/d7e/l129 0 2022-01-31T19:44:26.475 INFO:tasks.workunit.client.0.smithi167.stdout:0/851: write d17/d19/f1f [812362,61940] 0 2022-01-31T19:44:26.477 INFO:tasks.workunit.client.0.smithi167.stdout:6/853: mknod d8/c12a 0 2022-01-31T19:44:26.478 INFO:tasks.workunit.client.0.smithi167.stdout:6/854: creat d8/d33/d51/d5d/d86/f12b x:0 0 0 2022-01-31T19:44:26.478 INFO:tasks.workunit.client.0.smithi167.stdout:6/855: write d8/d20/fdc [961059,55875] 0 2022-01-31T19:44:26.482 INFO:tasks.workunit.client.1.smithi171.stdout:8/397: dwrite df/f17 [0,4194304] 0 2022-01-31T19:44:26.482 INFO:tasks.workunit.client.1.smithi171.stdout:8/398: write df/d2a/f40 [4711514,54959] 0 2022-01-31T19:44:26.482 INFO:tasks.workunit.client.1.smithi171.stdout:8/399: read df/d18/f4e [2270743,122218] 0 2022-01-31T19:44:26.484 INFO:tasks.workunit.client.1.smithi171.stdout:8/400: symlink df/d27/d7b/l89 0 2022-01-31T19:44:26.485 INFO:tasks.workunit.client.1.smithi171.stdout:8/401: readlink df/l74 0 2022-01-31T19:44:26.486 INFO:tasks.workunit.client.1.smithi171.stdout:8/402: creat df/d18/d19/d36/d70/d7e/f8a x:0 0 0 2022-01-31T19:44:26.486 INFO:tasks.workunit.client.1.smithi171.stdout:8/403: stat df/d5d/d6c/c72 0 2022-01-31T19:44:26.487 INFO:tasks.workunit.client.1.smithi171.stdout:3/507: dwrite d4/dd/d1a/d52/f54 [4194304,4194304] 0 2022-01-31T19:44:26.488 INFO:tasks.workunit.client.1.smithi171.stdout:8/404: symlink df/d18/d19/d36/d70/l8b 0 2022-01-31T19:44:26.490 INFO:tasks.workunit.client.1.smithi171.stdout:3/508: symlink d4/dd/d27/d28/d35/l9a 0 2022-01-31T19:44:26.490 INFO:tasks.workunit.client.1.smithi171.stdout:3/509: fdatasync d4/d21/d3a/d59/f70 0 2022-01-31T19:44:26.491 INFO:tasks.workunit.client.0.smithi167.stdout:6/856: dread d8/d33/d51/d7e/dc4/f97 [0,4194304] 0 2022-01-31T19:44:26.492 INFO:tasks.workunit.client.1.smithi171.stdout:8/405: rename df/d18/f1c to df/d18/d79/f8c 0 2022-01-31T19:44:26.492 INFO:tasks.workunit.client.1.smithi171.stdout:8/406: creat df/d18/d19/d46/f8d x:0 0 0 2022-01-31T19:44:26.492 INFO:tasks.workunit.client.1.smithi171.stdout:8/407: write df/d27/f57 [4623421,96451] 0 2022-01-31T19:44:26.493 INFO:tasks.workunit.client.0.smithi167.stdout:6/857: mknod d8/d33/d51/d5d/d64/dab/db9/c12c 0 2022-01-31T19:44:26.493 INFO:tasks.workunit.client.0.smithi167.stdout:0/852: dwrite d17/d19/fb1 [8388608,4194304] 0 2022-01-31T19:44:26.657 INFO:tasks.workunit.client.1.smithi171.stdout:3/510: rmdir d4/dd 39 2022-01-31T19:44:26.658 INFO:tasks.workunit.client.1.smithi171.stdout:3/511: dread d4/fa [0,4194304] 0 2022-01-31T19:44:26.659 INFO:tasks.workunit.client.1.smithi171.stdout:8/408: truncate df/d27/f57 1689778 0 2022-01-31T19:44:26.661 INFO:tasks.workunit.client.0.smithi167.stdout:6/858: dwrite d8/d62/d92/f121 [0,4194304] 0 2022-01-31T19:44:26.661 INFO:tasks.workunit.client.0.smithi167.stdout:6/859: truncate d8/d33/d51/d72/dc7/f10a 860328 0 2022-01-31T19:44:26.663 INFO:tasks.workunit.client.0.smithi167.stdout:0/853: dwrite d17/d3e/d7e/dc7/dde/fff [0,4194304] 0 2022-01-31T19:44:26.665 INFO:tasks.workunit.client.0.smithi167.stdout:6/860: link d8/d33/d51/d5d/f80 d8/d33/d51/d72/dc7/f12d 0 2022-01-31T19:44:26.665 INFO:tasks.workunit.client.0.smithi167.stdout:6/861: dread - d8/d33/d51/d5d/d64/dab/f11a zero size 2022-01-31T19:44:26.665 INFO:tasks.workunit.client.0.smithi167.stdout:6/862: truncate d8/d33/d51/ff2 217766 0 2022-01-31T19:44:26.668 INFO:tasks.workunit.client.0.smithi167.stdout:6/863: mkdir d8/d20/d54/d12e 0 2022-01-31T19:44:26.671 INFO:tasks.workunit.client.1.smithi171.stdout:4/364: sync 2022-01-31T19:44:26.671 INFO:tasks.workunit.client.1.smithi171.stdout:7/323: sync 2022-01-31T19:44:26.672 INFO:tasks.workunit.client.1.smithi171.stdout:9/549: sync 2022-01-31T19:44:26.672 INFO:tasks.workunit.client.1.smithi171.stdout:6/391: sync 2022-01-31T19:44:26.672 INFO:tasks.workunit.client.1.smithi171.stdout:2/574: sync 2022-01-31T19:44:26.672 INFO:tasks.workunit.client.1.smithi171.stdout:6/392: fsync dc/d19/d29/d6a/f76 0 2022-01-31T19:44:26.673 INFO:tasks.workunit.client.1.smithi171.stdout:4/365: read - d2/d3/d5/dc/d4b/f6e zero size 2022-01-31T19:44:26.673 INFO:tasks.workunit.client.1.smithi171.stdout:7/324: chown d1/d26/f5c 63524095 1 2022-01-31T19:44:26.673 INFO:tasks.workunit.client.1.smithi171.stdout:7/325: chown d1/d26/d57/l68 40 1 2022-01-31T19:44:26.673 INFO:tasks.workunit.client.1.smithi171.stdout:7/326: stat d1/d52/l17 0 2022-01-31T19:44:26.674 INFO:tasks.workunit.client.1.smithi171.stdout:7/327: truncate d1/d1e/d2a/d29/d31/d30/f4d 1023504 0 2022-01-31T19:44:26.674 INFO:tasks.workunit.client.1.smithi171.stdout:2/575: creat d7/d26/d63/fba x:0 0 0 2022-01-31T19:44:26.675 INFO:tasks.workunit.client.1.smithi171.stdout:6/393: symlink dc/d21/l7b 0 2022-01-31T19:44:26.675 INFO:tasks.workunit.client.1.smithi171.stdout:6/394: truncate dc/d19/d29/f24 336984 0 2022-01-31T19:44:26.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/366: symlink d2/d3/d5/d70/d7b/l7d 0 2022-01-31T19:44:26.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/367: chown d2/d3/d5/dc/f6f 8961 1 2022-01-31T19:44:26.676 INFO:tasks.workunit.client.1.smithi171.stdout:4/368: write d2/d3/d5/f38 [480641,81026] 0 2022-01-31T19:44:26.676 INFO:tasks.workunit.client.1.smithi171.stdout:7/328: mknod d1/d26/d57/c6e 0 2022-01-31T19:44:26.677 INFO:tasks.workunit.client.0.smithi167.stdout:6/864: rmdir d8/d20 39 2022-01-31T19:44:26.678 INFO:tasks.workunit.client.1.smithi171.stdout:6/395: mknod dc/d19/d29/c7c 0 2022-01-31T19:44:26.678 INFO:tasks.workunit.client.1.smithi171.stdout:6/396: readlink dc/l47 0 2022-01-31T19:44:26.678 INFO:tasks.workunit.client.1.smithi171.stdout:6/397: fdatasync dc/d1e/d3f/f59 0 2022-01-31T19:44:26.679 INFO:tasks.workunit.client.1.smithi171.stdout:8/409: dread df/d2a/d38/f43 [0,4194304] 0 2022-01-31T19:44:26.679 INFO:tasks.workunit.client.1.smithi171.stdout:7/329: symlink d1/d1e/l6f 0 2022-01-31T19:44:26.680 INFO:tasks.workunit.client.1.smithi171.stdout:2/576: dread d7/dd/f52 [0,4194304] 0 2022-01-31T19:44:26.681 INFO:tasks.workunit.client.1.smithi171.stdout:6/398: unlink dc/d18/c78 0 2022-01-31T19:44:26.681 INFO:tasks.workunit.client.1.smithi171.stdout:6/399: chown dc/c12 0 1 2022-01-31T19:44:26.681 INFO:tasks.workunit.client.1.smithi171.stdout:6/400: dread - dc/d21/f5a zero size 2022-01-31T19:44:26.682 INFO:tasks.workunit.client.1.smithi171.stdout:8/410: creat df/d27/d85/f8e x:0 0 0 2022-01-31T19:44:26.683 INFO:tasks.workunit.client.1.smithi171.stdout:8/411: creat df/d27/d85/f8f x:0 0 0 2022-01-31T19:44:26.684 INFO:tasks.workunit.client.1.smithi171.stdout:7/330: link d1/d1e/d2a/d29/d31/c25 d1/d1e/d2a/d29/c70 0 2022-01-31T19:44:26.685 INFO:tasks.workunit.client.1.smithi171.stdout:2/577: creat d7/d9/d1a/fbb x:0 0 0 2022-01-31T19:44:26.685 INFO:tasks.workunit.client.1.smithi171.stdout:2/578: truncate d7/d9/f44 3335966 0 2022-01-31T19:44:26.687 INFO:tasks.workunit.client.1.smithi171.stdout:6/401: symlink dc/d18/d53/l7d 0 2022-01-31T19:44:26.691 INFO:tasks.workunit.client.1.smithi171.stdout:3/512: dwrite d4/dd/f45 [0,4194304] 0 2022-01-31T19:44:26.693 INFO:tasks.workunit.client.1.smithi171.stdout:3/513: rmdir d4/dd/d39/d51/d89 0 2022-01-31T19:44:26.704 INFO:tasks.workunit.client.1.smithi171.stdout:7/331: dread d1/d1b/f22 [0,4194304] 0 2022-01-31T19:44:26.704 INFO:tasks.workunit.client.1.smithi171.stdout:7/332: creat d1/d26/f71 x:0 0 0 2022-01-31T19:44:26.707 INFO:tasks.workunit.client.1.smithi171.stdout:7/333: rename d1/d1e/l6f to d1/d1e/d2a/d29/d31/d30/d3c/l72 0 2022-01-31T19:44:26.707 INFO:tasks.workunit.client.1.smithi171.stdout:7/334: readlink d1/d26/l59 0 2022-01-31T19:44:26.711 INFO:tasks.workunit.client.1.smithi171.stdout:2/579: dread d7/d26/d3d/f42 [0,4194304] 0 2022-01-31T19:44:26.711 INFO:tasks.workunit.client.1.smithi171.stdout:2/580: stat d7/d26/d63/d9f/la0 0 2022-01-31T19:44:26.712 INFO:tasks.workunit.client.1.smithi171.stdout:7/335: write d1/d1e/d2a/d29/d31/d30/f4d [550086,69245] 0 2022-01-31T19:44:26.712 INFO:tasks.workunit.client.1.smithi171.stdout:7/336: creat d1/d1e/d2a/d29/d31/d30/f73 x:0 0 0 2022-01-31T19:44:26.712 INFO:tasks.workunit.client.1.smithi171.stdout:7/337: chown d1/d26/d47/f4f 52 1 2022-01-31T19:44:26.712 INFO:tasks.workunit.client.1.smithi171.stdout:7/338: chown d1/d1e/d2a/d29/d31/d30/d6b 3963 1 2022-01-31T19:44:26.715 INFO:tasks.workunit.client.0.smithi167.stdout:0/854: dwrite d17/d19/fe7 [0,4194304] 0 2022-01-31T19:44:26.715 INFO:tasks.workunit.client.0.smithi167.stdout:0/855: stat d17/f2f 0 2022-01-31T19:44:26.716 INFO:tasks.workunit.client.1.smithi171.stdout:2/581: getdents d7/d9 0 2022-01-31T19:44:26.716 INFO:tasks.workunit.client.1.smithi171.stdout:2/582: creat d7/d26/d63/fbc x:0 0 0 2022-01-31T19:44:26.716 INFO:tasks.workunit.client.1.smithi171.stdout:7/339: dread d1/d1e/d2a/d29/d31/f34 [0,4194304] 0 2022-01-31T19:44:26.718 INFO:tasks.workunit.client.0.smithi167.stdout:0/856: mkdir d17/d33/da4/dd0/d105/d11d 0 2022-01-31T19:44:26.718 INFO:tasks.workunit.client.0.smithi167.stdout:0/857: fdatasync d17/d3e/d7e/fce 0 2022-01-31T19:44:26.718 INFO:tasks.workunit.client.0.smithi167.stdout:0/858: creat d17/d33/d11c/f11e x:0 0 0 2022-01-31T19:44:26.719 INFO:tasks.workunit.client.0.smithi167.stdout:0/859: symlink d17/d3e/da1/l11f 0 2022-01-31T19:44:26.719 INFO:tasks.workunit.client.1.smithi171.stdout:2/583: symlink d7/dd/d5e/d6b/d81/lbd 0 2022-01-31T19:44:26.720 INFO:tasks.workunit.client.1.smithi171.stdout:2/584: chown d7/f11 245778 1 2022-01-31T19:44:26.720 INFO:tasks.workunit.client.1.smithi171.stdout:2/585: chown d7/d9/d1a/d45/c38 194 1 2022-01-31T19:44:26.720 INFO:tasks.workunit.client.1.smithi171.stdout:2/586: creat d7/d9/d1a/d45/da4/fbe x:0 0 0 2022-01-31T19:44:26.720 INFO:tasks.workunit.client.1.smithi171.stdout:7/340: mknod d1/d1e/d2a/d29/d31/c74 0 2022-01-31T19:44:26.721 INFO:tasks.workunit.client.1.smithi171.stdout:7/341: write d1/d1e/d2a/d29/d31/f38 [202857,1278] 0 2022-01-31T19:44:26.721 INFO:tasks.workunit.client.1.smithi171.stdout:7/342: mknod d1/d1e/d2a/c75 0 2022-01-31T19:44:26.721 INFO:tasks.workunit.client.1.smithi171.stdout:7/343: fdatasync d1/d1e/d2a/d29/d31/f6a 0 2022-01-31T19:44:26.721 INFO:tasks.workunit.client.1.smithi171.stdout:7/344: unlink d1/d1e/c2d 0 2022-01-31T19:44:26.722 INFO:tasks.workunit.client.1.smithi171.stdout:2/587: link d7/d26/d3d/d40/f4f d7/dd/fbf 0 2022-01-31T19:44:26.723 INFO:tasks.workunit.client.1.smithi171.stdout:2/588: chown d7/d7c/c7d 52 1 2022-01-31T19:44:26.723 INFO:tasks.workunit.client.0.smithi167.stdout:0/860: rmdir d17/d33/da4/dba/dee 39 2022-01-31T19:44:26.724 INFO:tasks.workunit.client.0.smithi167.stdout:0/861: truncate d17/d19/d1c/d39/fc5 958064 0 2022-01-31T19:44:26.724 INFO:tasks.workunit.client.0.smithi167.stdout:0/862: creat d17/d19/d1c/d39/d7b/de5/f120 x:0 0 0 2022-01-31T19:44:26.724 INFO:tasks.workunit.client.0.smithi167.stdout:0/863: write d17/d33/da4/dd0/fdb [911221,4306] 0 2022-01-31T19:44:26.725 INFO:tasks.workunit.client.0.smithi167.stdout:0/864: fdatasync d17/d19/d1c/de1/f110 0 2022-01-31T19:44:26.725 INFO:tasks.workunit.client.0.smithi167.stdout:0/865: mknod d17/d19/d1c/d39/d7b/de5/d104/c121 0 2022-01-31T19:44:26.725 INFO:tasks.workunit.client.0.smithi167.stdout:0/866: unlink d17/d19/d1c/d39/d7b/lb3 0 2022-01-31T19:44:26.725 INFO:tasks.workunit.client.0.smithi167.stdout:0/867: readlink d17/l66 0 2022-01-31T19:44:26.726 INFO:tasks.workunit.client.0.smithi167.stdout:6/865: dwrite d8/d33/d51/d7e/f7f [0,4194304] 0 2022-01-31T19:44:26.727 INFO:tasks.workunit.client.1.smithi171.stdout:7/345: unlink d1/d1e/d2a/d58/f69 0 2022-01-31T19:44:26.727 INFO:tasks.workunit.client.1.smithi171.stdout:7/346: readlink d1/d1e/d2a/d29/d31/l4e 0 2022-01-31T19:44:26.727 INFO:tasks.workunit.client.1.smithi171.stdout:2/589: mkdir d7/dd/d93/dc0 0 2022-01-31T19:44:26.728 INFO:tasks.workunit.client.1.smithi171.stdout:7/347: rmdir d1/d26/d47 39 2022-01-31T19:44:26.728 INFO:tasks.workunit.client.1.smithi171.stdout:2/590: rename d7/d55/f65 to d7/d7c/fc1 0 2022-01-31T19:44:26.729 INFO:tasks.workunit.client.0.smithi167.stdout:6/866: mknod d8/d33/d51/d72/dc7/c12f 0 2022-01-31T19:44:26.729 INFO:tasks.workunit.client.1.smithi171.stdout:8/412: dwrite df/f16 [0,4194304] 0 2022-01-31T19:44:26.730 INFO:tasks.workunit.client.1.smithi171.stdout:8/413: write df/d18/f1e [9346861,57337] 0 2022-01-31T19:44:26.730 INFO:tasks.workunit.client.1.smithi171.stdout:8/414: fdatasync df/d1a/f58 0 2022-01-31T19:44:26.730 INFO:tasks.workunit.client.1.smithi171.stdout:4/369: dwrite d2/d3/f1c [0,4194304] 0 2022-01-31T19:44:26.731 INFO:tasks.workunit.client.1.smithi171.stdout:4/370: dread - d2/d3/d5/d11/f62 zero size 2022-01-31T19:44:26.731 INFO:tasks.workunit.client.1.smithi171.stdout:3/514: dwrite d4/fa [0,4194304] 0 2022-01-31T19:44:26.732 INFO:tasks.workunit.client.1.smithi171.stdout:8/415: mkdir df/d27/d90 0 2022-01-31T19:44:26.738 INFO:tasks.workunit.client.1.smithi171.stdout:6/402: dwrite dc/d18/f1b [0,4194304] 0 2022-01-31T19:44:26.739 INFO:tasks.workunit.client.1.smithi171.stdout:6/403: fdatasync f2 0 2022-01-31T19:44:26.739 INFO:tasks.workunit.client.1.smithi171.stdout:6/404: chown dc/d18/l42 77 1 2022-01-31T19:44:26.740 INFO:tasks.workunit.client.1.smithi171.stdout:8/416: write df/d18/f1e [8326129,79068] 0 2022-01-31T19:44:26.741 INFO:tasks.workunit.client.1.smithi171.stdout:3/515: dread d4/dd/d1a/f2b [0,4194304] 0 2022-01-31T19:44:26.742 INFO:tasks.workunit.client.1.smithi171.stdout:6/405: write dc/f13 [570648,74768] 0 2022-01-31T19:44:26.752 INFO:tasks.workunit.client.1.smithi171.stdout:3/516: link d4/dd/d39/d51/f71 d4/dd/d1a/d52/f9b 0 2022-01-31T19:44:26.753 INFO:tasks.workunit.client.1.smithi171.stdout:6/406: creat dc/d18/d53/f7e x:0 0 0 2022-01-31T19:44:26.755 INFO:tasks.workunit.client.1.smithi171.stdout:8/417: rmdir df/d5d/d6c 39 2022-01-31T19:44:26.757 INFO:tasks.workunit.client.1.smithi171.stdout:6/407: dread dc/d21/d4f/f5f [0,4194304] 0 2022-01-31T19:44:26.758 INFO:tasks.workunit.client.1.smithi171.stdout:6/408: creat dc/f7f x:0 0 0 2022-01-31T19:44:26.761 INFO:tasks.workunit.client.1.smithi171.stdout:8/418: creat df/d18/d19/d21/d67/f91 x:0 0 0 2022-01-31T19:44:26.762 INFO:tasks.workunit.client.1.smithi171.stdout:7/348: getdents d1/d1e 0 2022-01-31T19:44:26.765 INFO:tasks.workunit.client.1.smithi171.stdout:6/409: mkdir dc/d19/d30/d3c/d80 0 2022-01-31T19:44:26.767 INFO:tasks.workunit.client.1.smithi171.stdout:8/419: getdents df/d18/d19/d21 0 2022-01-31T19:44:26.769 INFO:tasks.workunit.client.1.smithi171.stdout:8/420: creat df/d18/d19/d21/d88/f92 x:0 0 0 2022-01-31T19:44:26.769 INFO:tasks.workunit.client.1.smithi171.stdout:8/421: fdatasync fb 0 2022-01-31T19:44:26.776 INFO:tasks.workunit.client.1.smithi171.stdout:4/371: dwrite d2/f5c [0,4194304] 0 2022-01-31T19:44:26.779 INFO:tasks.workunit.client.0.smithi167.stdout:6/867: dwrite d8/d33/d51/d72/f75 [4194304,4194304] 0 2022-01-31T19:44:26.779 INFO:tasks.workunit.client.0.smithi167.stdout:6/868: dread - d8/d33/d51/d72/f10c zero size 2022-01-31T19:44:26.781 INFO:tasks.workunit.client.0.smithi167.stdout:0/868: dwrite d17/d2c/d87/df4/f10a [0,4194304] 0 2022-01-31T19:44:26.781 INFO:tasks.workunit.client.0.smithi167.stdout:0/869: chown d17/d19/d1c/d39/d44 44376038 1 2022-01-31T19:44:26.781 INFO:tasks.workunit.client.0.smithi167.stdout:0/870: stat d17/d3e/da1/l11f 0 2022-01-31T19:44:26.782 INFO:tasks.workunit.client.1.smithi171.stdout:2/591: dwrite d7/d26/d3d/d40/f6e [0,4194304] 0 2022-01-31T19:44:26.782 INFO:tasks.workunit.client.1.smithi171.stdout:2/592: chown d7/dd/d93 65386 1 2022-01-31T19:44:26.782 INFO:tasks.workunit.client.1.smithi171.stdout:2/593: write d7/d26/d3d/d40/f78 [657230,85669] 0 2022-01-31T19:44:26.783 INFO:tasks.workunit.client.1.smithi171.stdout:2/594: creat d7/dd/d93/fc2 x:0 0 0 2022-01-31T19:44:26.783 INFO:tasks.workunit.client.1.smithi171.stdout:2/595: write d7/d7c/f87 [79520,105459] 0 2022-01-31T19:44:26.783 INFO:tasks.workunit.client.1.smithi171.stdout:4/372: rename d2/d3/d5/d11 to d2/d3/d5/dc/d4b/d56/d66/d7e 0 2022-01-31T19:44:26.783 INFO:tasks.workunit.client.1.smithi171.stdout:8/422: dread df/d18/d19/d36/f73 [0,4194304] 0 2022-01-31T19:44:26.784 INFO:tasks.workunit.client.1.smithi171.stdout:3/517: dwrite d4/d21/f69 [0,4194304] 0 2022-01-31T19:44:26.786 INFO:tasks.workunit.client.1.smithi171.stdout:2/596: mkdir d7/d26/d63/d9f/dc3 0 2022-01-31T19:44:26.788 INFO:tasks.workunit.client.0.smithi167.stdout:6/869: symlink d8/d33/d51/l130 0 2022-01-31T19:44:26.789 INFO:tasks.workunit.client.0.smithi167.stdout:6/870: fsync d8/d62/d92/f121 0 2022-01-31T19:44:26.789 INFO:tasks.workunit.client.0.smithi167.stdout:6/871: getdents d8/d33/d51/d6a/df1 0 2022-01-31T19:44:26.790 INFO:tasks.workunit.client.1.smithi171.stdout:7/349: dwrite d1/d52/f18 [0,4194304] 0 2022-01-31T19:44:26.794 INFO:tasks.workunit.client.1.smithi171.stdout:6/410: getdents dc/d18/d53 0 2022-01-31T19:44:26.794 INFO:tasks.workunit.client.1.smithi171.stdout:6/411: chown dc/f13 6005 1 2022-01-31T19:44:26.795 INFO:tasks.workunit.client.1.smithi171.stdout:4/373: mkdir d2/d7f 0 2022-01-31T19:44:26.796 INFO:tasks.workunit.client.1.smithi171.stdout:4/374: creat d2/d3/d5/dc/f80 x:0 0 0 2022-01-31T19:44:26.796 INFO:tasks.workunit.client.1.smithi171.stdout:4/375: creat d2/d3/d5/dc/d4b/f81 x:0 0 0 2022-01-31T19:44:26.796 INFO:tasks.workunit.client.1.smithi171.stdout:4/376: truncate d2/d3/d5/dc/f80 323950 0 2022-01-31T19:44:26.796 INFO:tasks.workunit.client.1.smithi171.stdout:4/377: readlink d2/l4d 0 2022-01-31T19:44:26.797 INFO:tasks.workunit.client.1.smithi171.stdout:8/423: unlink df/d27/d7b/l89 0 2022-01-31T19:44:26.800 INFO:tasks.workunit.client.1.smithi171.stdout:3/518: mknod d4/d21/d3a/c9c 0 2022-01-31T19:44:26.803 INFO:tasks.workunit.client.1.smithi171.stdout:1/422: sync 2022-01-31T19:44:26.803 INFO:tasks.workunit.client.1.smithi171.stdout:5/429: sync 2022-01-31T19:44:26.804 INFO:tasks.workunit.client.1.smithi171.stdout:5/430: chown d5/f57 7 1 2022-01-31T19:44:26.804 INFO:tasks.workunit.client.1.smithi171.stdout:7/350: creat d1/d1e/d2a/d29/d31/d30/d3c/f76 x:0 0 0 2022-01-31T19:44:26.805 INFO:tasks.workunit.client.0.smithi167.stdout:2/938: sync 2022-01-31T19:44:26.806 INFO:tasks.workunit.client.1.smithi171.stdout:6/412: symlink dc/d19/d30/l81 0 2022-01-31T19:44:26.807 INFO:tasks.workunit.client.1.smithi171.stdout:8/424: creat df/d18/d79/d7a/f93 x:0 0 0 2022-01-31T19:44:26.807 INFO:tasks.workunit.client.1.smithi171.stdout:8/425: chown df/f17 33242703 1 2022-01-31T19:44:26.808 INFO:tasks.workunit.client.1.smithi171.stdout:2/597: dread d7/d9/f6a [0,4194304] 0 2022-01-31T19:44:26.809 INFO:tasks.workunit.client.1.smithi171.stdout:2/598: creat d7/d26/fc4 x:0 0 0 2022-01-31T19:44:26.809 INFO:tasks.workunit.client.1.smithi171.stdout:2/599: write d7/dd/f80 [1248443,126879] 0 2022-01-31T19:44:26.809 INFO:tasks.workunit.client.1.smithi171.stdout:2/600: readlink d7/lc 0 2022-01-31T19:44:26.809 INFO:tasks.workunit.client.1.smithi171.stdout:1/423: link dd/df/d19/d1b/d1e/f6a dd/df/d19/d1b/d77/f85 0 2022-01-31T19:44:26.810 INFO:tasks.workunit.client.1.smithi171.stdout:1/424: creat dd/d23/d2e/f86 x:0 0 0 2022-01-31T19:44:26.810 INFO:tasks.workunit.client.1.smithi171.stdout:1/425: creat dd/d23/d2e/f87 x:0 0 0 2022-01-31T19:44:26.811 INFO:tasks.workunit.client.1.smithi171.stdout:5/431: link d5/d6/d15/d23/d58/c61 d5/d6/c95 0 2022-01-31T19:44:26.812 INFO:tasks.workunit.client.1.smithi171.stdout:7/351: truncate d1/d1e/d2a/d29/f43 730092 0 2022-01-31T19:44:26.812 INFO:tasks.workunit.client.1.smithi171.stdout:5/432: write d5/f8 [1527159,63444] 0 2022-01-31T19:44:26.812 INFO:tasks.workunit.client.1.smithi171.stdout:5/433: dread - d5/d6/f46 zero size 2022-01-31T19:44:26.812 INFO:tasks.workunit.client.0.smithi167.stdout:2/939: mkdir dd/df/d1b/d2f/d36/d134 0 2022-01-31T19:44:26.813 INFO:tasks.workunit.client.0.smithi167.stdout:2/940: creat dd/d18/f135 x:0 0 0 2022-01-31T19:44:26.813 INFO:tasks.workunit.client.0.smithi167.stdout:0/871: dwrite d17/d19/d1c/fc3 [0,4194304] 0 2022-01-31T19:44:26.813 INFO:tasks.workunit.client.0.smithi167.stdout:0/872: fsync d17/d19/d1c/de1/f110 0 2022-01-31T19:44:26.813 INFO:tasks.workunit.client.0.smithi167.stdout:2/941: rmdir dd/df/d3f 39 2022-01-31T19:44:26.814 INFO:tasks.workunit.client.1.smithi171.stdout:6/413: unlink dc/d21/l7b 0 2022-01-31T19:44:26.815 INFO:tasks.workunit.client.1.smithi171.stdout:6/414: chown dc/d18/d53/f58 769211 1 2022-01-31T19:44:26.815 INFO:tasks.workunit.client.1.smithi171.stdout:3/519: dread d4/dd/d27/d28/d35/f5c [0,4194304] 0 2022-01-31T19:44:26.816 INFO:tasks.workunit.client.1.smithi171.stdout:8/426: mknod df/d18/d19/d82/c94 0 2022-01-31T19:44:26.816 INFO:tasks.workunit.client.0.smithi167.stdout:0/873: creat d17/d3e/d7e/dc7/dde/df1/d113/f122 x:0 0 0 2022-01-31T19:44:26.816 INFO:tasks.workunit.client.0.smithi167.stdout:2/942: creat dd/d18/d62/d75/f136 x:0 0 0 2022-01-31T19:44:26.817 INFO:tasks.workunit.client.0.smithi167.stdout:0/874: link d17/d69/f9e d17/d19/d1c/d39/d44/d88/f123 0 2022-01-31T19:44:26.819 INFO:tasks.workunit.client.1.smithi171.stdout:1/426: truncate f9 1656543 0 2022-01-31T19:44:26.820 INFO:tasks.workunit.client.0.smithi167.stdout:0/875: mknod d17/d19/d1c/d39/d7b/de5/d104/c124 0 2022-01-31T19:44:26.821 INFO:tasks.workunit.client.0.smithi167.stdout:0/876: chown d17/d2c/dcd/cfe 53812272 1 2022-01-31T19:44:26.821 INFO:tasks.workunit.client.1.smithi171.stdout:0/531: sync 2022-01-31T19:44:26.822 INFO:tasks.workunit.client.1.smithi171.stdout:0/532: fdatasync d9/db/d22/f79 0 2022-01-31T19:44:26.824 INFO:tasks.workunit.client.0.smithi167.stdout:0/877: truncate d17/d19/f100 2558515 0 2022-01-31T19:44:26.824 INFO:tasks.workunit.client.0.smithi167.stdout:0/878: truncate d17/d19/d1c/d39/d44/d88/f123 113768 0 2022-01-31T19:44:26.825 INFO:tasks.workunit.client.0.smithi167.stdout:6/872: dwrite d8/d33/d51/d5d/f9f [0,4194304] 0 2022-01-31T19:44:26.825 INFO:tasks.workunit.client.0.smithi167.stdout:6/873: chown d8/d33/d51/d5d/d86/fb6 191504119 1 2022-01-31T19:44:26.825 INFO:tasks.workunit.client.1.smithi171.stdout:7/352: mkdir d1/d1e/d2a/d29/d77 0 2022-01-31T19:44:26.826 INFO:tasks.workunit.client.1.smithi171.stdout:5/434: mknod d5/d6/d15/d23/d67/c96 0 2022-01-31T19:44:26.826 INFO:tasks.workunit.client.1.smithi171.stdout:4/378: dwrite d2/d3/d5/dc/d4b/f81 [0,4194304] 0 2022-01-31T19:44:26.826 INFO:tasks.workunit.client.1.smithi171.stdout:6/415: mknod dc/d19/d30/d3c/c82 0 2022-01-31T19:44:26.827 INFO:tasks.workunit.client.1.smithi171.stdout:8/427: mkdir df/d95 0 2022-01-31T19:44:26.827 INFO:tasks.workunit.client.1.smithi171.stdout:9/550: sync 2022-01-31T19:44:26.828 INFO:tasks.workunit.client.1.smithi171.stdout:2/601: rmdir d7/d26/d3d/d40 39 2022-01-31T19:44:26.828 INFO:tasks.workunit.client.0.smithi167.stdout:6/874: link d8/d33/d51/d5d/d64/dab/db9/cf9 d8/d33/d51/d72/dad/c131 0 2022-01-31T19:44:26.829 INFO:tasks.workunit.client.0.smithi167.stdout:2/943: dread dd/df/f5d [0,4194304] 0 2022-01-31T19:44:26.830 INFO:tasks.workunit.client.0.smithi167.stdout:6/875: mkdir d8/d33/d51/d5d/d86/d132 0 2022-01-31T19:44:26.830 INFO:tasks.workunit.client.1.smithi171.stdout:1/427: creat dd/d23/d74/d7b/f88 x:0 0 0 2022-01-31T19:44:26.831 INFO:tasks.workunit.client.0.smithi167.stdout:2/944: getdents dd/d18/d62 0 2022-01-31T19:44:26.833 INFO:tasks.workunit.client.1.smithi171.stdout:5/435: link d5/d6/f16 d5/f97 0 2022-01-31T19:44:26.834 INFO:tasks.workunit.client.0.smithi167.stdout:6/876: truncate d8/d33/d51/d7e/f7f 4191821 0 2022-01-31T19:44:26.834 INFO:tasks.workunit.client.0.smithi167.stdout:6/877: write d8/d33/d51/d7e/dc4/f9e [689998,4484] 0 2022-01-31T19:44:26.835 INFO:tasks.workunit.client.0.smithi167.stdout:2/945: mkdir dd/d18/db7/d137 0 2022-01-31T19:44:26.841 INFO:tasks.workunit.client.0.smithi167.stdout:2/946: write dd/d18/d62/d75/d105/f92 [1338329,130401] 0 2022-01-31T19:44:26.842 INFO:tasks.workunit.client.1.smithi171.stdout:4/379: mknod d2/d3/d5/d41/c82 0 2022-01-31T19:44:26.843 INFO:tasks.workunit.client.1.smithi171.stdout:5/436: write d5/d6/d15/f33 [496125,38122] 0 2022-01-31T19:44:26.843 INFO:tasks.workunit.client.1.smithi171.stdout:5/437: write d5/d6/d15/f33 [4442019,103936] 0 2022-01-31T19:44:26.843 INFO:tasks.workunit.client.1.smithi171.stdout:6/416: creat dc/d19/d30/d3c/d80/f83 x:0 0 0 2022-01-31T19:44:26.843 INFO:tasks.workunit.client.1.smithi171.stdout:6/417: read - dc/d1e/f45 zero size 2022-01-31T19:44:26.844 INFO:tasks.workunit.client.0.smithi167.stdout:2/947: rename dd/df/d1b/d3d/d4a/d4b/d74/d85/dc4/ld7 to dd/df/d1b/d3d/d4a/d4b/d54/l138 0 2022-01-31T19:44:26.845 INFO:tasks.workunit.client.1.smithi171.stdout:0/533: dwrite d9/db/d59/fc3 [4194304,4194304] 0 2022-01-31T19:44:26.845 INFO:tasks.workunit.client.1.smithi171.stdout:1/428: mknod dd/df/d19/d36/c89 0 2022-01-31T19:44:26.846 INFO:tasks.workunit.client.1.smithi171.stdout:1/429: chown dd/d23/d2e/l6f 29 1 2022-01-31T19:44:26.846 INFO:tasks.workunit.client.1.smithi171.stdout:1/430: write dd/d23/d2e/f87 [863440,33518] 0 2022-01-31T19:44:26.846 INFO:tasks.workunit.client.1.smithi171.stdout:1/431: stat dd/df 0 2022-01-31T19:44:26.846 INFO:tasks.workunit.client.1.smithi171.stdout:5/438: symlink d5/d90/l98 0 2022-01-31T19:44:26.847 INFO:tasks.workunit.client.0.smithi167.stdout:2/948: creat dd/d18/d63/f139 x:0 0 0 2022-01-31T19:44:26.847 INFO:tasks.workunit.client.0.smithi167.stdout:2/949: truncate dd/df/d1b/d3d/d4a/d4b/d74/d85/fe2 587125 0 2022-01-31T19:44:26.848 INFO:tasks.workunit.client.0.smithi167.stdout:2/950: creat dd/d18/d62/d75/f13a x:0 0 0 2022-01-31T19:44:26.848 INFO:tasks.workunit.client.1.smithi171.stdout:6/418: symlink dc/d19/d30/d3c/d80/l84 0 2022-01-31T19:44:26.849 INFO:tasks.workunit.client.1.smithi171.stdout:9/551: dread dd/df/d3f/f82 [0,4194304] 0 2022-01-31T19:44:26.849 INFO:tasks.workunit.client.1.smithi171.stdout:2/602: getdents d7/dd/d5e 0 2022-01-31T19:44:26.851 INFO:tasks.workunit.client.1.smithi171.stdout:1/432: creat dd/d23/d2e/d5e/f8a x:0 0 0 2022-01-31T19:44:26.852 INFO:tasks.workunit.client.1.smithi171.stdout:5/439: symlink d5/d59/l99 0 2022-01-31T19:44:26.853 INFO:tasks.workunit.client.0.smithi167.stdout:2/951: link dd/d18/d63/cf0 dd/df/d1b/d3d/d4a/c13b 0 2022-01-31T19:44:26.853 INFO:tasks.workunit.client.0.smithi167.stdout:6/878: dread d8/d33/d4f/d87/fb2 [0,4194304] 0 2022-01-31T19:44:26.854 INFO:tasks.workunit.client.1.smithi171.stdout:6/419: rename dc/d19/f55 to dc/d19/d29/f85 0 2022-01-31T19:44:26.854 INFO:tasks.workunit.client.1.smithi171.stdout:9/552: mkdir dd/d1c/db5 0 2022-01-31T19:44:26.855 INFO:tasks.workunit.client.0.smithi167.stdout:2/952: rename dd/f15 to dd/d18/f13c 0 2022-01-31T19:44:26.856 INFO:tasks.workunit.client.1.smithi171.stdout:2/603: creat d7/dd/d5e/fc5 x:0 0 0 2022-01-31T19:44:26.857 INFO:tasks.workunit.client.1.smithi171.stdout:0/534: getdents d9/db/d59 0 2022-01-31T19:44:26.857 INFO:tasks.workunit.client.1.smithi171.stdout:1/433: mknod dd/d23/c8b 0 2022-01-31T19:44:26.858 INFO:tasks.workunit.client.0.smithi167.stdout:2/953: mknod dd/d18/c13d 0 2022-01-31T19:44:26.859 INFO:tasks.workunit.client.1.smithi171.stdout:6/420: mkdir dc/d18/d86 0 2022-01-31T19:44:26.860 INFO:tasks.workunit.client.1.smithi171.stdout:5/440: dread d5/f8 [0,4194304] 0 2022-01-31T19:44:26.860 INFO:tasks.workunit.client.1.smithi171.stdout:9/553: mkdir dd/d1c/d7c/d7d/db6 0 2022-01-31T19:44:26.861 INFO:tasks.workunit.client.0.smithi167.stdout:6/879: rmdir d8/d33/d4f/dd1 39 2022-01-31T19:44:26.862 INFO:tasks.workunit.client.1.smithi171.stdout:6/421: write dc/d19/d29/d27/f3a [2692366,88810] 0 2022-01-31T19:44:26.862 INFO:tasks.workunit.client.1.smithi171.stdout:6/422: fdatasync dc/d18/d53/f7e 0 2022-01-31T19:44:26.862 INFO:tasks.workunit.client.1.smithi171.stdout:2/604: unlink d7/d9/d1a/d45/da4/caa 0 2022-01-31T19:44:26.863 INFO:tasks.workunit.client.0.smithi167.stdout:2/954: creat dd/df/d1b/d3d/d4a/d4b/d74/d85/f13e x:0 0 0 2022-01-31T19:44:26.864 INFO:tasks.workunit.client.0.smithi167.stdout:6/880: creat d8/d20/f133 x:0 0 0 2022-01-31T19:44:26.864 INFO:tasks.workunit.client.0.smithi167.stdout:2/955: symlink dd/df/d1b/d2f/l13f 0 2022-01-31T19:44:26.866 INFO:tasks.workunit.client.0.smithi167.stdout:6/881: rename d8/d62/d92/fa1 to d8/d20/d103/d108/f134 0 2022-01-31T19:44:26.870 INFO:tasks.workunit.client.1.smithi171.stdout:0/535: truncate d9/d24/d92/f9a 1169682 0 2022-01-31T19:44:26.870 INFO:tasks.workunit.client.1.smithi171.stdout:0/536: chown d9/d24/d92/f9a 440942259 1 2022-01-31T19:44:26.872 INFO:tasks.workunit.client.1.smithi171.stdout:1/434: symlink dd/df/d84/l8c 0 2022-01-31T19:44:26.872 INFO:tasks.workunit.client.1.smithi171.stdout:1/435: fsync dd/df/d19/f47 0 2022-01-31T19:44:26.873 INFO:tasks.workunit.client.1.smithi171.stdout:1/436: write dd/d23/d74/d7b/f88 [49480,93970] 0 2022-01-31T19:44:26.873 INFO:tasks.workunit.client.1.smithi171.stdout:1/437: fsync dd/d23/f5c 0 2022-01-31T19:44:26.874 INFO:tasks.workunit.client.0.smithi167.stdout:6/882: rename d8/d33 to d8/d20/d103/d108/d135 0 2022-01-31T19:44:26.874 INFO:tasks.workunit.client.0.smithi167.stdout:6/883: symlink d8/d20/d103/d108/d135/d51/d72/dc7/l136 0 2022-01-31T19:44:26.876 INFO:tasks.workunit.client.0.smithi167.stdout:6/884: rename d8/d20/d103/d108/d135/d51/d7e/dc4 to d8/d20/d103/d108/d135/d51/d137 0 2022-01-31T19:44:26.877 INFO:tasks.workunit.client.1.smithi171.stdout:9/554: creat dd/df/d15/d53/fb7 x:0 0 0 2022-01-31T19:44:26.877 INFO:tasks.workunit.client.1.smithi171.stdout:2/605: mknod d7/d26/d63/cc6 0 2022-01-31T19:44:26.877 INFO:tasks.workunit.client.1.smithi171.stdout:2/606: truncate d7/d26/fa7 1004143 0 2022-01-31T19:44:26.878 INFO:tasks.workunit.client.0.smithi167.stdout:6/885: mknod d8/d20/d103/d108/d135/d51/d6a/df1/d10d/c138 0 2022-01-31T19:44:26.880 INFO:tasks.workunit.client.1.smithi171.stdout:4/380: dwrite d2/d3/f1c [0,4194304] 0 2022-01-31T19:44:26.881 INFO:tasks.workunit.client.1.smithi171.stdout:4/381: readlink d2/d3/d5/dc/d4b/d56/d66/l72 0 2022-01-31T19:44:26.881 INFO:tasks.workunit.client.1.smithi171.stdout:4/382: chown d2/l37 11585618 1 2022-01-31T19:44:26.881 INFO:tasks.workunit.client.0.smithi167.stdout:0/879: dwrite d17/d69/f9e [0,4194304] 0 2022-01-31T19:44:26.882 INFO:tasks.workunit.client.1.smithi171.stdout:9/555: dread dd/df/d3f/d73/d96/d9f/fa1 [0,4194304] 0 2022-01-31T19:44:26.883 INFO:tasks.workunit.client.1.smithi171.stdout:4/383: link d2/d3/d5/dc/d4b/d56/d66/d7e/d12/d15/f52 d2/d3/d5/dc/d4b/d56/d66/d7e/f83 0 2022-01-31T19:44:26.885 INFO:tasks.workunit.client.1.smithi171.stdout:1/438: dread dd/df/d19/f32 [0,4194304] 0 2022-01-31T19:44:26.887 INFO:tasks.workunit.client.1.smithi171.stdout:9/556: dread dd/df/d15/f69 [4194304,4194304] 0 2022-01-31T19:44:26.888 INFO:tasks.workunit.client.0.smithi167.stdout:0/880: symlink d17/d33/l125 0 2022-01-31T19:44:26.889 INFO:tasks.workunit.client.1.smithi171.stdout:6/423: dwrite dc/d18/f25 [0,4194304] 0 2022-01-31T19:44:26.890 INFO:tasks.workunit.client.1.smithi171.stdout:6/424: fdatasync dc/d19/d29/d27/d37/d68/f6d 0 2022-01-31T19:44:26.892 INFO:tasks.workunit.client.0.smithi167.stdout:6/886: dread d8/d20/d103/d108/d135/d51/ff4 [0,4194304] 0 2022-01-31T19:44:26.893 INFO:tasks.workunit.client.1.smithi171.stdout:9/557: write dd/d1c/f3d [196978,74472] 0 2022-01-31T19:44:26.893 INFO:tasks.workunit.client.0.smithi167.stdout:6/887: fsync d8/f73 0 2022-01-31T19:44:26.894 INFO:tasks.workunit.client.0.smithi167.stdout:6/888: truncate d8/d20/d103/d108/d135/d51/d5d/d86/fcc 1923869 0 2022-01-31T19:44:26.896 INFO:tasks.workunit.client.0.smithi167.stdout:6/889: unlink d8/d20/d103/d108/d135/d51/d137/f9e 0 2022-01-31T19:44:26.898 INFO:tasks.workunit.client.0.smithi167.stdout:6/890: mkdir d8/d20/d103/d108/d135/d51/d6a/d139 0 2022-01-31T19:44:26.898 INFO:tasks.workunit.client.0.smithi167.stdout:6/891: chown d8/d20/d103/d108/d135/d51/d5d/d64 363 1 2022-01-31T19:44:26.899 INFO:tasks.workunit.client.1.smithi171.stdout:1/439: rename dd/df/d19/d36/c6b to dd/d23/d2e/d5e/d81/c8d 0 2022-01-31T19:44:26.901 INFO:tasks.workunit.client.0.smithi167.stdout:6/892: write d8/d20/d103/d108/d135/f40 [4599722,107340] 0 2022-01-31T19:44:26.901 INFO:tasks.workunit.client.1.smithi171.stdout:6/425: creat dc/d18/d86/f87 x:0 0 0 2022-01-31T19:44:26.902 INFO:tasks.workunit.client.1.smithi171.stdout:9/558: chown dd/f6a 3 1 2022-01-31T19:44:26.902 INFO:tasks.workunit.client.1.smithi171.stdout:9/559: symlink dd/d1c/d7c/lb8 0 2022-01-31T19:44:26.903 INFO:tasks.workunit.client.0.smithi167.stdout:6/893: rename d8/d20/d103/d108/d135/d51/d5d/d64/dab/db9 to d8/d20/d103/d108/d135/d51/d6a/d139/d13a 0 2022-01-31T19:44:26.904 INFO:tasks.workunit.client.1.smithi171.stdout:9/560: write dd/df/d15/f4e [666331,23056] 0 2022-01-31T19:44:26.904 INFO:tasks.workunit.client.1.smithi171.stdout:9/561: fsync dd/df/d15/d1a/d64/f67 0 2022-01-31T19:44:26.904 INFO:tasks.workunit.client.1.smithi171.stdout:6/426: write dc/d21/f2e [3582723,20895] 0 2022-01-31T19:44:26.905 INFO:tasks.workunit.client.1.smithi171.stdout:6/427: stat dc/d19/d29/d27/d37/d68/f6d 0 2022-01-31T19:44:26.905 INFO:tasks.workunit.client.1.smithi171.stdout:6/428: unlink dc/c16 0 2022-01-31T19:44:26.905 INFO:tasks.workunit.client.1.smithi171.stdout:6/429: write dc/d18/d53/f7e [339390,67831] 0 2022-01-31T19:44:26.906 INFO:tasks.workunit.client.0.smithi167.stdout:6/894: creat d8/d20/d103/d108/d135/f13b x:0 0 0 2022-01-31T19:44:26.906 INFO:tasks.workunit.client.0.smithi167.stdout:6/895: symlink d8/d20/d103/d108/d135/d51/d5d/d64/dab/l13c 0 2022-01-31T19:44:26.906 INFO:tasks.workunit.client.0.smithi167.stdout:6/896: write d8/d20/d103/d108/d135/d51/d72/fa5 [694261,19846] 0 2022-01-31T19:44:26.907 INFO:tasks.workunit.client.0.smithi167.stdout:6/897: rename d8/d20/d103/d108/d135/d51/d7e/f7f to d8/d20/d103/d108/d135/d4f/d87/f13d 0 2022-01-31T19:44:26.908 INFO:tasks.workunit.client.0.smithi167.stdout:2/956: dwrite dd/d18/d62/dea/f10f [0,4194304] 0 2022-01-31T19:44:26.909 INFO:tasks.workunit.client.0.smithi167.stdout:2/957: fsync dd/f106 0 2022-01-31T19:44:26.909 INFO:tasks.workunit.client.0.smithi167.stdout:2/958: fdatasync dd/d18/d62/f64 0 2022-01-31T19:44:26.909 INFO:tasks.workunit.client.0.smithi167.stdout:2/959: chown dd/df/c33 21 1 2022-01-31T19:44:26.910 INFO:tasks.workunit.client.1.smithi171.stdout:0/537: dwrite d9/d24/d92/f5f [0,4194304] 0 2022-01-31T19:44:26.910 INFO:tasks.workunit.client.1.smithi171.stdout:6/430: unlink dc/d19/d30/f72 0 2022-01-31T19:44:26.911 INFO:tasks.workunit.client.1.smithi171.stdout:6/431: fsync dc/d18/f25 0 2022-01-31T19:44:26.911 INFO:tasks.workunit.client.0.smithi167.stdout:6/898: unlink d8/d20/d103/d108/d135/fd2 0 2022-01-31T19:44:26.912 INFO:tasks.workunit.client.0.smithi167.stdout:6/899: truncate d8/d20/d103/d108/d135/d51/d137/f96 873884 0 2022-01-31T19:44:26.914 INFO:tasks.workunit.client.0.smithi167.stdout:6/900: mknod d8/d20/d103/d108/d135/d51/d6a/c13e 0 2022-01-31T19:44:26.916 INFO:tasks.workunit.client.1.smithi171.stdout:0/538: write d9/f16 [5985705,81570] 0 2022-01-31T19:44:26.916 INFO:tasks.workunit.client.1.smithi171.stdout:6/432: mknod dc/d19/d30/d3c/d80/c88 0 2022-01-31T19:44:26.917 INFO:tasks.workunit.client.1.smithi171.stdout:6/433: truncate dc/d19/d29/f24 701470 0 2022-01-31T19:44:26.917 INFO:tasks.workunit.client.1.smithi171.stdout:6/434: fsync dc/d21/d4f/f60 0 2022-01-31T19:44:26.917 INFO:tasks.workunit.client.1.smithi171.stdout:6/435: dread - dc/d1e/d3f/f7a zero size 2022-01-31T19:44:26.918 INFO:tasks.workunit.client.1.smithi171.stdout:6/436: fdatasync fa 0 2022-01-31T19:44:26.918 INFO:tasks.workunit.client.1.smithi171.stdout:6/437: truncate dc/d19/d29/f24 777093 0 2022-01-31T19:44:26.918 INFO:tasks.workunit.client.1.smithi171.stdout:0/539: symlink d9/lc8 0 2022-01-31T19:44:26.919 INFO:tasks.workunit.client.1.smithi171.stdout:5/441: dwrite d5/d6/d15/f88 [0,4194304] 0 2022-01-31T19:44:26.919 INFO:tasks.workunit.client.1.smithi171.stdout:5/442: creat d5/d6/d15/d23/f9a x:0 0 0 2022-01-31T19:44:26.920 INFO:tasks.workunit.client.0.smithi167.stdout:6/901: link d8/d20/f133 d8/d20/d103/d108/d135/d51/d6a/df1/d106/f13f 0 2022-01-31T19:44:26.920 INFO:tasks.workunit.client.0.smithi167.stdout:6/902: mkdir d8/d20/d103/d108/d135/d51/d5d/d86/d140 0 2022-01-31T19:44:26.920 INFO:tasks.workunit.client.0.smithi167.stdout:6/903: creat d8/d20/d103/d108/d135/d51/d7e/f141 x:0 0 0 2022-01-31T19:44:26.922 INFO:tasks.workunit.client.0.smithi167.stdout:6/904: mkdir d8/d20/d103/d142 0 2022-01-31T19:44:26.923 INFO:tasks.workunit.client.1.smithi171.stdout:4/384: dwrite d2/d3/d5/f2e [0,4194304] 0 2022-01-31T19:44:26.924 INFO:tasks.workunit.client.1.smithi171.stdout:5/443: link d5/f68 d5/d6/dd/d66/f9b 0 2022-01-31T19:44:26.928 INFO:tasks.workunit.client.0.smithi167.stdout:0/881: dwrite d17/d2c/dcd/fd6 [4194304,4194304] 0 2022-01-31T19:44:26.928 INFO:tasks.workunit.client.0.smithi167.stdout:6/905: symlink d8/d20/d103/d108/d135/d51/d72/l143 0 2022-01-31T19:44:26.935 INFO:tasks.workunit.client.1.smithi171.stdout:5/444: creat d5/d6/d15/d5d/d6f/f9c x:0 0 0 2022-01-31T19:44:26.939 INFO:tasks.workunit.client.0.smithi167.stdout:0/882: dread d17/d2c/d87/f8d [0,4194304] 0 2022-01-31T19:44:26.940 INFO:tasks.workunit.client.1.smithi171.stdout:5/445: rename d5/d6/d15/d23/d67/l54 to d5/d59/l9d 0 2022-01-31T19:44:26.940 INFO:tasks.workunit.client.1.smithi171.stdout:5/446: readlink d5/d6/d15/d5d/d6a/l85 0 2022-01-31T19:44:26.941 INFO:tasks.workunit.client.1.smithi171.stdout:5/447: dread - d5/d6/d15/f64 zero size 2022-01-31T19:44:26.943 INFO:tasks.workunit.client.0.smithi167.stdout:6/906: dread d8/d20/d103/d108/d135/f90 [0,4194304] 0 2022-01-31T19:44:26.944 INFO:tasks.workunit.client.1.smithi171.stdout:1/440: dwrite dd/df/f12 [0,4194304] 0 2022-01-31T19:44:26.945 INFO:tasks.workunit.client.1.smithi171.stdout:1/441: chown dd/df/d84 233 1 2022-01-31T19:44:26.947 INFO:tasks.workunit.client.0.smithi167.stdout:0/883: dread d17/d19/d1c/d39/d7b/f81 [0,4194304] 0 2022-01-31T19:44:26.948 INFO:tasks.workunit.client.1.smithi171.stdout:1/442: write dd/d23/f5c [1845615,31059] 0 2022-01-31T19:44:26.948 INFO:tasks.workunit.client.0.smithi167.stdout:6/907: write d8/d20/d103/d108/d135/d51/d72/fd6 [1372467,35401] 0 2022-01-31T19:44:26.949 INFO:tasks.workunit.client.1.smithi171.stdout:1/443: creat dd/df/d19/d1b/d77/d80/f8e x:0 0 0 2022-01-31T19:44:26.949 INFO:tasks.workunit.client.1.smithi171.stdout:1/444: stat dd/df/c33 0 2022-01-31T19:44:26.950 INFO:tasks.workunit.client.1.smithi171.stdout:1/445: symlink dd/d23/d34/d4b/l8f 0 2022-01-31T19:44:26.950 INFO:tasks.workunit.client.1.smithi171.stdout:1/446: dread - dd/d23/d2e/d5e/f8a zero size 2022-01-31T19:44:26.951 INFO:tasks.workunit.client.1.smithi171.stdout:1/447: mknod dd/d23/c90 0 2022-01-31T19:44:26.951 INFO:tasks.workunit.client.1.smithi171.stdout:1/448: write dd/df/d19/f27 [4690743,7186] 0 2022-01-31T19:44:26.958 INFO:tasks.workunit.client.0.smithi167.stdout:6/908: dread d8/d20/f2f [0,4194304] 0 2022-01-31T19:44:26.964 INFO:tasks.workunit.client.1.smithi171.stdout:0/540: dwrite d9/db/d59/d71/fa4 [0,4194304] 0 2022-01-31T19:44:26.965 INFO:tasks.workunit.client.0.smithi167.stdout:0/884: dread d17/d2c/dcd/fd6 [4194304,4194304] 0 2022-01-31T19:44:26.965 INFO:tasks.workunit.client.0.smithi167.stdout:0/885: write d17/d3e/d7e/dc7/dde/df1/ffd [360266,50476] 0 2022-01-31T19:44:26.966 INFO:tasks.workunit.client.0.smithi167.stdout:0/886: link d17/d2c/d87/df4/f10a d17/d2c/d87/f126 0 2022-01-31T19:44:26.967 INFO:tasks.workunit.client.1.smithi171.stdout:6/438: dwrite dc/d21/d4f/f5f [0,4194304] 0 2022-01-31T19:44:26.967 INFO:tasks.workunit.client.1.smithi171.stdout:6/439: creat dc/d1e/f89 x:0 0 0 2022-01-31T19:44:26.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/541: unlink d9/db/dc/d3f/db9/f85 0 2022-01-31T19:44:26.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/542: readlink d9/db/dc/d3f/db9/d7a/d88/lb6 0 2022-01-31T19:44:26.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/543: dread - d9/d24/d4c/d35/f89 zero size 2022-01-31T19:44:26.971 INFO:tasks.workunit.client.1.smithi171.stdout:4/385: dwrite d2/d3/d5/dc/f19 [0,4194304] 0 2022-01-31T19:44:26.971 INFO:tasks.workunit.client.1.smithi171.stdout:4/386: dread - d2/d3/d5/dc/d4b/d56/d66/d7e/d12/f5a zero size 2022-01-31T19:44:26.972 INFO:tasks.workunit.client.1.smithi171.stdout:4/387: creat d2/d3/d5/dc/d4b/d56/d66/f84 x:0 0 0 2022-01-31T19:44:26.972 INFO:tasks.workunit.client.1.smithi171.stdout:4/388: fdatasync d2/d3/f1c 0 2022-01-31T19:44:26.972 INFO:tasks.workunit.client.1.smithi171.stdout:4/389: write d2/d3/d5/dc/f6f [4945,101004] 0 2022-01-31T19:44:26.972 INFO:tasks.workunit.client.1.smithi171.stdout:4/390: chown d2/d3/d5/dc/d4b/f6e 608190054 1 2022-01-31T19:44:26.973 INFO:tasks.workunit.client.1.smithi171.stdout:9/562: dwrite dd/df/d15/f2c [0,4194304] 0 2022-01-31T19:44:26.973 INFO:tasks.workunit.client.1.smithi171.stdout:9/563: readlink dd/d1c/d7c/lb8 0 2022-01-31T19:44:26.974 INFO:tasks.workunit.client.0.smithi167.stdout:0/887: dread d17/d19/d1c/f8f [0,4194304] 0 2022-01-31T19:44:26.974 INFO:tasks.workunit.client.1.smithi171.stdout:4/391: read d2/d3/d5/f1a [2493957,51921] 0 2022-01-31T19:44:26.975 INFO:tasks.workunit.client.1.smithi171.stdout:4/392: chown d2/f14 7 1 2022-01-31T19:44:26.975 INFO:tasks.workunit.client.1.smithi171.stdout:0/544: creat d9/db/fc9 x:0 0 0 2022-01-31T19:44:26.975 INFO:tasks.workunit.client.0.smithi167.stdout:0/888: getdents d17/d19/d1c/d39/d7b 0 2022-01-31T19:44:26.976 INFO:tasks.workunit.client.1.smithi171.stdout:5/448: dwrite d5/d90/f7f [0,4194304] 0 2022-01-31T19:44:26.977 INFO:tasks.workunit.client.1.smithi171.stdout:5/449: write d5/d6/dd/f20 [4072395,116980] 0 2022-01-31T19:44:26.977 INFO:tasks.workunit.client.1.smithi171.stdout:5/450: write d5/d6/d15/d23/f87 [4492046,127112] 0 2022-01-31T19:44:26.978 INFO:tasks.workunit.client.1.smithi171.stdout:4/393: link d2/d3/d5/dc/d4b/d56/d66/d7e/l1b d2/d3/d5/d41/l85 0 2022-01-31T19:44:26.978 INFO:tasks.workunit.client.1.smithi171.stdout:4/394: dread - d2/d3/d5/dc/d4b/d56/d66/d7e/d12/d15/f64 zero size 2022-01-31T19:44:26.979 INFO:tasks.workunit.client.1.smithi171.stdout:0/545: stat d9/db/c20 0 2022-01-31T19:44:26.979 INFO:tasks.workunit.client.1.smithi171.stdout:0/546: fsync d9/d24/d4c/f3d 0 2022-01-31T19:44:26.979 INFO:tasks.workunit.client.1.smithi171.stdout:5/451: dread d5/f89 [0,4194304] 0 2022-01-31T19:44:26.980 INFO:tasks.workunit.client.1.smithi171.stdout:9/564: mkdir dd/df/d3f/db9 0 2022-01-31T19:44:26.980 INFO:tasks.workunit.client.1.smithi171.stdout:9/565: chown dd/df/d15/d1a/d64/d6e/lb3 214 1 2022-01-31T19:44:26.980 INFO:tasks.workunit.client.1.smithi171.stdout:5/452: readlink d5/d6/l7c 0 2022-01-31T19:44:26.980 INFO:tasks.workunit.client.1.smithi171.stdout:5/453: chown d5/d6/d15/d23/d67/f4c 3236894 1 2022-01-31T19:44:26.981 INFO:tasks.workunit.client.1.smithi171.stdout:5/454: truncate d5/d6/dd/f35 1881618 0 2022-01-31T19:44:26.986 INFO:tasks.workunit.client.1.smithi171.stdout:1/449: dwrite dd/df/d19/f47 [0,4194304] 0 2022-01-31T19:44:26.988 INFO:tasks.workunit.client.1.smithi171.stdout:4/395: creat d2/d3/f86 x:0 0 0 2022-01-31T19:44:26.988 INFO:tasks.workunit.client.1.smithi171.stdout:5/455: unlink d5/d6/d15/l2c 0 2022-01-31T19:44:26.988 INFO:tasks.workunit.client.1.smithi171.stdout:9/566: creat dd/df/d15/d38/d9c/fba x:0 0 0 2022-01-31T19:44:26.988 INFO:tasks.workunit.client.1.smithi171.stdout:4/396: creat d2/d3/d5/dc/f87 x:0 0 0 2022-01-31T19:44:26.989 INFO:tasks.workunit.client.1.smithi171.stdout:9/567: chown dd/df/f12 29544479 1 2022-01-31T19:44:26.990 INFO:tasks.workunit.client.0.smithi167.stdout:6/909: dwrite d8/d20/fdc [0,4194304] 0 2022-01-31T19:44:26.992 INFO:tasks.workunit.client.1.smithi171.stdout:1/450: write dd/df/d19/f75 [2663219,48721] 0 2022-01-31T19:44:26.992 INFO:tasks.workunit.client.1.smithi171.stdout:1/451: truncate dd/d23/f3f 678972 0 2022-01-31T19:44:26.997 INFO:tasks.workunit.client.1.smithi171.stdout:1/452: write dd/df/d19/d1b/f3e [4192360,103983] 0 2022-01-31T19:44:26.997 INFO:tasks.workunit.client.1.smithi171.stdout:1/453: readlink dd/df/l6c 0 2022-01-31T19:44:26.998 INFO:tasks.workunit.client.1.smithi171.stdout:5/456: mkdir d5/d6/d15/d23/d58/d9e 0 2022-01-31T19:44:26.999 INFO:tasks.workunit.client.1.smithi171.stdout:9/568: truncate dd/df/d3f/f51 2729409 0 2022-01-31T19:44:26.999 INFO:tasks.workunit.client.1.smithi171.stdout:9/569: fdatasync dd/f3c 0 2022-01-31T19:44:26.999 INFO:tasks.workunit.client.1.smithi171.stdout:9/570: fsync dd/df/d15/d1a/d64/f67 0 2022-01-31T19:44:27.000 INFO:tasks.workunit.client.1.smithi171.stdout:4/397: symlink d2/d3/d5/dc/d4b/d56/d66/d7e/d12/d15/d57/l88 0 2022-01-31T19:44:27.000 INFO:tasks.workunit.client.1.smithi171.stdout:4/398: dread - d2/d3/d5/dc/d4b/d56/d66/f76 zero size 2022-01-31T19:44:27.001 INFO:tasks.workunit.client.1.smithi171.stdout:4/399: write d2/d3/d5/dc/d4b/d56/d66/d7e/d12/f60 [567182,76477] 0 2022-01-31T19:44:27.001 INFO:tasks.workunit.client.1.smithi171.stdout:1/454: truncate dd/d23/f57 581208 0 2022-01-31T19:44:27.002 INFO:tasks.workunit.client.1.smithi171.stdout:1/455: chown dd/d23/d2e/f86 202776749 1 2022-01-31T19:44:27.002 INFO:tasks.workunit.client.1.smithi171.stdout:1/456: readlink dd/df/d84/l8c 0 2022-01-31T19:44:27.003 INFO:tasks.workunit.client.1.smithi171.stdout:6/440: dwrite dc/d19/d29/d6a/f76 [0,4194304] 0 2022-01-31T19:44:27.003 INFO:tasks.workunit.client.1.smithi171.stdout:5/457: rmdir d5/d6/d69 0 2022-01-31T19:44:27.005 INFO:tasks.workunit.client.1.smithi171.stdout:9/571: creat dd/df/d15/d38/fbb x:0 0 0 2022-01-31T19:44:27.005 INFO:tasks.workunit.client.1.smithi171.stdout:9/572: fsync dd/f37 0 2022-01-31T19:44:27.005 INFO:tasks.workunit.client.1.smithi171.stdout:9/573: write dd/df/f17 [4472833,101075] 0 2022-01-31T19:44:27.007 INFO:tasks.workunit.client.1.smithi171.stdout:4/400: rename d2/d3/d5/dc/d4b/d56/d66/d7e/d12/d15/f52 to d2/d3/d5/d41/f89 0 2022-01-31T19:44:27.007 INFO:tasks.workunit.client.1.smithi171.stdout:4/401: dread - d2/d3/d5/dc/d4b/d56/d66/f84 zero size 2022-01-31T19:44:27.011 INFO:tasks.workunit.client.1.smithi171.stdout:6/441: link dc/d19/d29/c62 dc/d19/d29/d27/d37/d3b/c8a 0 2022-01-31T19:44:27.013 INFO:tasks.workunit.client.1.smithi171.stdout:4/402: symlink d2/l8a 0 2022-01-31T19:44:27.013 INFO:tasks.workunit.client.1.smithi171.stdout:6/442: stat dc/d19/d30/d3c/d80/l84 0 2022-01-31T19:44:27.014 INFO:tasks.workunit.client.1.smithi171.stdout:6/443: getdents dc/d18/d53 0 2022-01-31T19:44:27.014 INFO:tasks.workunit.client.1.smithi171.stdout:9/574: rmdir dd/df/d15/d1a/d64 39 2022-01-31T19:44:27.015 INFO:tasks.workunit.client.1.smithi171.stdout:4/403: symlink d2/d3/d5/dc/d4b/d56/l8b 0 2022-01-31T19:44:27.015 INFO:tasks.workunit.client.1.smithi171.stdout:9/575: symlink dd/df/d86/lbc 0 2022-01-31T19:44:27.016 INFO:tasks.workunit.client.1.smithi171.stdout:4/404: truncate d2/d3/f1c 2341473 0 2022-01-31T19:44:27.016 INFO:tasks.workunit.client.1.smithi171.stdout:4/405: chown d2/d3/d5/dc/f42 9223991 1 2022-01-31T19:44:27.018 INFO:tasks.workunit.client.1.smithi171.stdout:9/576: symlink dd/d1c/da8/lbd 0 2022-01-31T19:44:27.021 INFO:tasks.workunit.client.1.smithi171.stdout:9/577: write dd/df/d15/f3a [547342,56792] 0 2022-01-31T19:44:27.023 INFO:tasks.workunit.client.1.smithi171.stdout:9/578: creat dd/d1c/db5/fbe x:0 0 0 2022-01-31T19:44:27.024 INFO:tasks.workunit.client.1.smithi171.stdout:9/579: creat dd/df/d15/d53/fbf x:0 0 0 2022-01-31T19:44:27.024 INFO:tasks.workunit.client.1.smithi171.stdout:9/580: chown dd/d1c/d7c/lb8 501323 1 2022-01-31T19:44:27.024 INFO:tasks.workunit.client.1.smithi171.stdout:9/581: dread - dd/d1c/d7c/d7d/fa9 zero size 2022-01-31T19:44:27.025 INFO:tasks.workunit.client.1.smithi171.stdout:9/582: symlink dd/df/d3f/d73/d96/lc0 0 2022-01-31T19:44:27.025 INFO:tasks.workunit.client.1.smithi171.stdout:9/583: stat dd/df/d15/d1a/f27 0 2022-01-31T19:44:27.025 INFO:tasks.workunit.client.1.smithi171.stdout:9/584: readlink dd/df/d3f/l5e 0 2022-01-31T19:44:27.026 INFO:tasks.workunit.client.1.smithi171.stdout:9/585: fdatasync dd/df/d15/f2c 0 2022-01-31T19:44:27.028 INFO:tasks.workunit.client.1.smithi171.stdout:9/586: dread dd/df/d15/f16 [0,4194304] 0 2022-01-31T19:44:27.028 INFO:tasks.workunit.client.1.smithi171.stdout:9/587: dread - dd/df/d15/d53/fa6 zero size 2022-01-31T19:44:27.028 INFO:tasks.workunit.client.1.smithi171.stdout:9/588: mknod dd/df/d3f/d73/d96/cc1 0 2022-01-31T19:44:27.029 INFO:tasks.workunit.client.1.smithi171.stdout:9/589: readlink dd/df/d3f/l5e 0 2022-01-31T19:44:27.029 INFO:tasks.workunit.client.1.smithi171.stdout:9/590: chown dd/d1c/l5f 282 1 2022-01-31T19:44:27.085 INFO:tasks.workunit.client.1.smithi171.stdout:6/444: read dc/d19/d29/d27/d37/d68/f6d [939411,40524] 0 2022-01-31T19:44:27.087 INFO:tasks.workunit.client.1.smithi171.stdout:6/445: mkdir dc/d8b 0 2022-01-31T19:44:27.088 INFO:tasks.workunit.client.1.smithi171.stdout:6/446: symlink dc/d21/d4f/l8c 0 2022-01-31T19:44:27.089 INFO:tasks.workunit.client.0.smithi167.stdout:6/910: link d8/d20/d54/f63 d8/d20/d103/d108/d135/d51/d72/dc7/f144 0 2022-01-31T19:44:27.090 INFO:tasks.workunit.client.1.smithi171.stdout:6/447: creat dc/d21/f8d x:0 0 0 2022-01-31T19:44:27.091 INFO:tasks.workunit.client.1.smithi171.stdout:6/448: creat dc/d19/d29/d27/d37/d3b/f8e x:0 0 0 2022-01-31T19:44:27.092 INFO:tasks.workunit.client.1.smithi171.stdout:6/449: write dc/d19/d29/f26 [1690048,69057] 0 2022-01-31T19:44:27.092 INFO:tasks.workunit.client.1.smithi171.stdout:6/450: write dc/f7f [673231,102016] 0 2022-01-31T19:44:27.093 INFO:tasks.workunit.client.1.smithi171.stdout:0/547: dwrite d9/d24/d92/f94 [0,4194304] 0 2022-01-31T19:44:27.094 INFO:tasks.workunit.client.1.smithi171.stdout:0/548: write d9/d24/d4c/d35/f8d [951715,123732] 0 2022-01-31T19:44:27.096 INFO:tasks.workunit.client.1.smithi171.stdout:0/549: creat d9/d24/d4c/d7d/fca x:0 0 0 2022-01-31T19:44:27.098 INFO:tasks.workunit.client.1.smithi171.stdout:0/550: mknod d9/db/d59/d71/dba/dbb/ccb 0 2022-01-31T19:44:27.104 INFO:tasks.workunit.client.1.smithi171.stdout:0/551: getdents d9 0 2022-01-31T19:44:27.104 INFO:tasks.workunit.client.1.smithi171.stdout:0/552: fsync d9/db/dc2/f52 0 2022-01-31T19:44:27.104 INFO:tasks.workunit.client.1.smithi171.stdout:0/553: write d9/d24/d4c/fa9 [24963,78785] 0 2022-01-31T19:44:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:26 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:25.651523+0000 mgr.smithi171.asyxnc (mgr.24387 2022-01-31T19:44:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:26 smithi171 conmon[35325]: ) 32 : cephadm [INF] Upgrade: Updating mgr.smithi167.aciqpk 2022-01-31T19:44:27.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:26 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:25.672839+0000 mgr.smithi171.asyxnc (mgr.24387) 33 : cephadm [INF] Deploying daemon mgr.smithi167.aciqpk on smithi167 2022-01-31T19:44:27.116 INFO:tasks.workunit.client.0.smithi167.stdout:0/889: dwrite d17/d2c/d87/df4/f10a [4194304,4194304] 0 2022-01-31T19:44:27.116 INFO:tasks.workunit.client.0.smithi167.stdout:0/890: dread - d17/d19/d1c/d39/d44/d49/fc2 zero size 2022-01-31T19:44:27.117 INFO:tasks.workunit.client.0.smithi167.stdout:0/891: write d17/f22 [4610430,31818] 0 2022-01-31T19:44:27.117 INFO:tasks.workunit.client.0.smithi167.stdout:0/892: stat d17/d3e/d7e/dc7/dde/lf6 0 2022-01-31T19:44:27.118 INFO:tasks.workunit.client.0.smithi167.stdout:0/893: dread - d17/d19/d1c/d39/d7b/de5/f120 zero size 2022-01-31T19:44:27.118 INFO:tasks.workunit.client.0.smithi167.stdout:0/894: read d17/d3e/da1/fe0 [82993,30016] 0 2022-01-31T19:44:27.129 INFO:tasks.workunit.client.1.smithi171.stdout:3/520: sync 2022-01-31T19:44:27.129 INFO:tasks.workunit.client.1.smithi171.stdout:3/521: chown d4/dd/f45 313412 1 2022-01-31T19:44:27.129 INFO:tasks.workunit.client.1.smithi171.stdout:7/353: sync 2022-01-31T19:44:27.130 INFO:tasks.workunit.client.1.smithi171.stdout:2/607: sync 2022-01-31T19:44:27.130 INFO:tasks.workunit.client.1.smithi171.stdout:8/428: sync 2022-01-31T19:44:27.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/354: creat d1/d1e/f78 x:0 0 0 2022-01-31T19:44:27.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/355: stat d1/f16 0 2022-01-31T19:44:27.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/356: fdatasync d1/d1e/d2a/d29/f6d 0 2022-01-31T19:44:27.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/357: write d1/d1e/d2a/d29/d31/f6a [4431517,81887] 0 2022-01-31T19:44:27.133 INFO:tasks.workunit.client.1.smithi171.stdout:5/458: dwrite d5/f57 [0,4194304] 0 2022-01-31T19:44:27.133 INFO:tasks.workunit.client.0.smithi167.stdout:6/911: dwrite d8/d20/d103/d108/d135/f40 [4194304,4194304] 0 2022-01-31T19:44:27.134 INFO:tasks.workunit.client.0.smithi167.stdout:6/912: truncate d8/d20/d103/d108/d135/d51/d5d/f80 4199552 0 2022-01-31T19:44:27.134 INFO:tasks.workunit.client.0.smithi167.stdout:6/913: chown d8/d20/d103/d108/d135/d51/d137/l100 33 1 2022-01-31T19:44:27.135 INFO:tasks.workunit.client.1.smithi171.stdout:5/459: write d5/d6/d15/d23/d67/f39 [4189481,121891] 0 2022-01-31T19:44:27.135 INFO:tasks.workunit.client.1.smithi171.stdout:7/358: mkdir d1/d1e/d2a/d29/d31/d30/d3c/d79 0 2022-01-31T19:44:27.135 INFO:tasks.workunit.client.1.smithi171.stdout:7/359: truncate d1/d26/d47/f4f 488793 0 2022-01-31T19:44:27.136 INFO:tasks.workunit.client.1.smithi171.stdout:7/360: truncate d1/d1e/d2a/d29/f6d 126351 0 2022-01-31T19:44:27.136 INFO:tasks.workunit.client.1.smithi171.stdout:7/361: chown d1/d1e/d2a/l54 13 1 2022-01-31T19:44:27.136 INFO:tasks.workunit.client.0.smithi167.stdout:6/914: symlink d8/d20/d103/d108/d135/d51/d5d/d86/d140/l145 0 2022-01-31T19:44:27.137 INFO:tasks.workunit.client.0.smithi167.stdout:6/915: creat d8/d20/d103/d142/f146 x:0 0 0 2022-01-31T19:44:27.138 INFO:tasks.workunit.client.1.smithi171.stdout:5/460: dread d5/d6/d15/d23/f41 [0,4194304] 0 2022-01-31T19:44:27.139 INFO:tasks.workunit.client.0.smithi167.stdout:6/916: mkdir d8/d20/d103/d108/d135/d51/d72/d147 0 2022-01-31T19:44:27.139 INFO:tasks.workunit.client.0.smithi167.stdout:6/917: chown d8/d20/d103/d108/d135/d51/d7e 13 1 2022-01-31T19:44:27.140 INFO:tasks.workunit.client.0.smithi167.stdout:6/918: readlink d8/d20/d103/d108/d135/d4f/l74 0 2022-01-31T19:44:27.140 INFO:tasks.workunit.client.0.smithi167.stdout:6/919: symlink d8/d20/d103/l148 0 2022-01-31T19:44:27.142 INFO:tasks.workunit.client.1.smithi171.stdout:3/522: dread d4/dd/d27/f83 [0,4194304] 0 2022-01-31T19:44:27.142 INFO:tasks.workunit.client.1.smithi171.stdout:3/523: fsync d4/dd/d27/f3f 0 2022-01-31T19:44:27.142 INFO:tasks.workunit.client.1.smithi171.stdout:3/524: dread - d4/dd/d27/d28/d35/d55/d8f/f95 zero size 2022-01-31T19:44:27.144 INFO:tasks.workunit.client.1.smithi171.stdout:2/608: dread d7/fa [0,4194304] 0 2022-01-31T19:44:27.144 INFO:tasks.workunit.client.1.smithi171.stdout:9/591: dwrite dd/df/f20 [0,4194304] 0 2022-01-31T19:44:27.145 INFO:tasks.workunit.client.1.smithi171.stdout:3/525: write d4/dd/d27/d28/d35/f48 [1037512,87453] 0 2022-01-31T19:44:27.145 INFO:tasks.workunit.client.1.smithi171.stdout:3/526: chown d4/dd/f19 5 1 2022-01-31T19:44:27.146 INFO:tasks.workunit.client.1.smithi171.stdout:6/451: dwrite dc/d19/d29/f5c [0,4194304] 0 2022-01-31T19:44:27.146 INFO:tasks.workunit.client.1.smithi171.stdout:6/452: readlink dc/l36 0 2022-01-31T19:44:27.147 INFO:tasks.workunit.client.1.smithi171.stdout:5/461: rename d5/d6/d15/d5d/f65 to d5/f9f 0 2022-01-31T19:44:27.148 INFO:tasks.workunit.client.0.smithi167.stdout:6/920: write d8/d20/d103/d108/d135/f40 [5586935,85527] 0 2022-01-31T19:44:27.148 INFO:tasks.workunit.client.1.smithi171.stdout:2/609: write d7/d9/d1a/f89 [2065948,18748] 0 2022-01-31T19:44:27.149 INFO:tasks.workunit.client.0.smithi167.stdout:6/921: symlink d8/d20/d103/d108/d135/d51/d72/l149 0 2022-01-31T19:44:27.152 INFO:tasks.workunit.client.1.smithi171.stdout:1/457: dread dd/d23/f57 [0,4194304] 0 2022-01-31T19:44:27.154 INFO:tasks.workunit.client.0.smithi167.stdout:6/922: read d8/f1c [4184246,32116] 0 2022-01-31T19:44:27.155 INFO:tasks.workunit.client.0.smithi167.stdout:6/923: chown d8/d20/d103/d108/d135/d51/d72/dc7/fdb 71 1 2022-01-31T19:44:27.155 INFO:tasks.workunit.client.1.smithi171.stdout:1/458: dread dd/d23/d2e/f87 [0,4194304] 0 2022-01-31T19:44:27.155 INFO:tasks.workunit.client.1.smithi171.stdout:1/459: truncate dd/df/d19/d1b/d1e/f4a 4988322 0 2022-01-31T19:44:27.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:26 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:25.651523+0000 mgr.smithi171.asyxnc (mgr.24387) 32 : cephadm [INF] Upgrade: Updating mgr.smithi167.aciqpk 2022-01-31T19:44:27.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:26 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:25.672839+0000 mgr.smithi171.asyxnc (mgr.24387) 33 : cephadm [INF] Deploying daemon mgr.smithi167.aciqpk on smithi167 2022-01-31T19:44:27.159 INFO:tasks.workunit.client.1.smithi171.stdout:5/462: dread d5/d6/dd/f19 [0,4194304] 0 2022-01-31T19:44:27.160 INFO:tasks.workunit.client.1.smithi171.stdout:3/527: rmdir d4/dd/d27/d28/d35/d55/d8f 39 2022-01-31T19:44:27.160 INFO:tasks.workunit.client.0.smithi167.stdout:0/895: dwrite d17/d19/fe7 [0,4194304] 0 2022-01-31T19:44:27.162 INFO:tasks.workunit.client.1.smithi171.stdout:8/429: dwrite df/f83 [4194304,4194304] 0 2022-01-31T19:44:27.162 INFO:tasks.workunit.client.1.smithi171.stdout:7/362: dwrite d1/d1b/f22 [4194304,4194304] 0 2022-01-31T19:44:27.162 INFO:tasks.workunit.client.1.smithi171.stdout:7/363: creat d1/d1e/d2a/d29/f7a x:0 0 0 2022-01-31T19:44:27.163 INFO:tasks.workunit.client.1.smithi171.stdout:7/364: chown d1/d1e/d2a/d29/d31/d30/d3c/f56 2 1 2022-01-31T19:44:27.163 INFO:tasks.workunit.client.1.smithi171.stdout:2/610: getdents d7/d26/d63/d71 0 2022-01-31T19:44:27.164 INFO:tasks.workunit.client.1.smithi171.stdout:6/453: creat dc/d8b/f8f x:0 0 0 2022-01-31T19:44:27.164 INFO:tasks.workunit.client.1.smithi171.stdout:1/460: getdents dd/df/d19/d1b/d1e 0 2022-01-31T19:44:27.165 INFO:tasks.workunit.client.1.smithi171.stdout:5/463: fsync d5/d6/d15/f3f 0 2022-01-31T19:44:27.165 INFO:tasks.workunit.client.0.smithi167.stdout:2/960: sync 2022-01-31T19:44:27.167 INFO:tasks.workunit.client.1.smithi171.stdout:1/461: dread dd/df/f38 [0,4194304] 0 2022-01-31T19:44:27.167 INFO:tasks.workunit.client.1.smithi171.stdout:3/528: link d4/dd/d27/d28/d35/d55/d8f/d92/l32 d4/d21/d3a/d90/l9d 0 2022-01-31T19:44:27.167 INFO:tasks.workunit.client.0.smithi167.stdout:2/961: dread dd/df/d1b/d3d/d4a/d4b/f7f [0,4194304] 0 2022-01-31T19:44:27.168 INFO:tasks.workunit.client.0.smithi167.stdout:2/962: truncate dd/d18/d62/f6e 1428114 0 2022-01-31T19:44:27.168 INFO:tasks.workunit.client.0.smithi167.stdout:2/963: mknod dd/df/d1b/d3d/d4a/d4b/d54/dca/c140 0 2022-01-31T19:44:27.169 INFO:tasks.workunit.client.1.smithi171.stdout:9/592: rename dd/df/d15/f16 to dd/fc2 0 2022-01-31T19:44:27.170 INFO:tasks.workunit.client.0.smithi167.stdout:2/964: creat dd/df/d1b/d3d/d4a/d4b/d54/dca/ddf/f141 x:0 0 0 2022-01-31T19:44:27.170 INFO:tasks.workunit.client.1.smithi171.stdout:6/454: dread dc/d1e/d3f/f59 [0,4194304] 0 2022-01-31T19:44:27.171 INFO:tasks.workunit.client.0.smithi167.stdout:2/965: getdents dd/df/d22/d96/d9e 0 2022-01-31T19:44:27.172 INFO:tasks.workunit.client.0.smithi167.stdout:2/966: write dd/d18/d62/d75/dc8/f12d [888678,56878] 0 2022-01-31T19:44:27.172 INFO:tasks.workunit.client.0.smithi167.stdout:0/896: dread d17/d2c/f79 [0,4194304] 0 2022-01-31T19:44:27.173 INFO:tasks.workunit.client.1.smithi171.stdout:7/365: link d1/d1e/d2a/d29/d31/d30/f3f d1/d26/d47/f7b 0 2022-01-31T19:44:27.173 INFO:tasks.workunit.client.1.smithi171.stdout:7/366: write d1/d1e/d2a/d29/d31/d30/d3c/f56 [643556,115656] 0 2022-01-31T19:44:27.173 INFO:tasks.workunit.client.1.smithi171.stdout:2/611: fdatasync d7/dd/d5e/f9c 0 2022-01-31T19:44:27.174 INFO:tasks.workunit.client.1.smithi171.stdout:2/612: stat d7/d9/d1a/c54 0 2022-01-31T19:44:27.174 INFO:tasks.workunit.client.1.smithi171.stdout:2/613: write d7/d26/d63/fbc [53106,53672] 0 2022-01-31T19:44:27.175 INFO:tasks.workunit.client.1.smithi171.stdout:5/464: mkdir d5/d59/da0 0 2022-01-31T19:44:27.175 INFO:tasks.workunit.client.0.smithi167.stdout:0/897: symlink d17/d3e/d7e/l127 0 2022-01-31T19:44:27.175 INFO:tasks.workunit.client.0.smithi167.stdout:0/898: mkdir d17/d19/d1c/d39/d7b/da9/d128 0 2022-01-31T19:44:27.176 INFO:tasks.workunit.client.0.smithi167.stdout:0/899: symlink d17/d19/d1c/d39/d7b/da9/d128/l129 0 2022-01-31T19:44:27.176 INFO:tasks.workunit.client.0.smithi167.stdout:0/900: creat d17/d33/d112/f12a x:0 0 0 2022-01-31T19:44:27.180 INFO:tasks.workunit.client.1.smithi171.stdout:1/462: truncate dd/df/d19/d1b/f28 3786186 0 2022-01-31T19:44:27.181 INFO:tasks.workunit.client.1.smithi171.stdout:7/367: creat d1/d1e/d2a/d29/d31/d30/d6b/f7c x:0 0 0 2022-01-31T19:44:27.181 INFO:tasks.workunit.client.1.smithi171.stdout:3/529: truncate d4/d21/f31 1175009 0 2022-01-31T19:44:27.181 INFO:tasks.workunit.client.1.smithi171.stdout:6/455: symlink dc/d19/d29/d27/d37/d3b/l90 0 2022-01-31T19:44:27.182 INFO:tasks.workunit.client.1.smithi171.stdout:9/593: creat dd/d1c/da8/fc3 x:0 0 0 2022-01-31T19:44:27.182 INFO:tasks.workunit.client.1.smithi171.stdout:9/594: fdatasync dd/df/d15/d1a/f27 0 2022-01-31T19:44:27.184 INFO:tasks.workunit.client.1.smithi171.stdout:8/430: dwrite df/d27/d85/f8f [0,4194304] 0 2022-01-31T19:44:27.184 INFO:tasks.workunit.client.1.smithi171.stdout:2/614: symlink d7/d9/d1a/lc7 0 2022-01-31T19:44:27.193 INFO:tasks.workunit.client.1.smithi171.stdout:3/530: rename d4/fb to d4/dd/d39/d51/f9e 0 2022-01-31T19:44:27.194 INFO:tasks.workunit.client.1.smithi171.stdout:3/531: stat d4/dd/d27/d4e/d62 0 2022-01-31T19:44:27.194 INFO:tasks.workunit.client.1.smithi171.stdout:3/532: write d4/dd/d27/d28/d35/d55/d8f/d92/f85 [890682,97209] 0 2022-01-31T19:44:27.195 INFO:tasks.workunit.client.1.smithi171.stdout:7/368: mknod d1/d1e/d2a/d29/d31/d30/d6b/c7d 0 2022-01-31T19:44:27.196 INFO:tasks.workunit.client.1.smithi171.stdout:7/369: fdatasync d1/d52/f28 0 2022-01-31T19:44:27.196 INFO:tasks.workunit.client.1.smithi171.stdout:6/456: mknod dc/d21/d4f/c91 0 2022-01-31T19:44:27.198 INFO:tasks.workunit.client.1.smithi171.stdout:7/370: dread d1/d1e/d2a/d29/d31/f67 [0,4194304] 0 2022-01-31T19:44:27.198 INFO:tasks.workunit.client.1.smithi171.stdout:7/371: readlink d1/d1b/l3e 0 2022-01-31T19:44:27.200 INFO:tasks.workunit.client.1.smithi171.stdout:2/615: dread d7/f14 [0,4194304] 0 2022-01-31T19:44:27.202 INFO:tasks.workunit.client.0.smithi167.stdout:2/967: dwrite dd/df/d1b/d3d/d4a/d4b/d74/d85/f9b [0,4194304] 0 2022-01-31T19:44:27.203 INFO:tasks.workunit.client.1.smithi171.stdout:5/465: dwrite d5/d6/dd/f45 [0,4194304] 0 2022-01-31T19:44:27.204 INFO:tasks.workunit.client.0.smithi167.stdout:2/968: creat dd/d18/d62/d75/d11b/f142 x:0 0 0 2022-01-31T19:44:27.205 INFO:tasks.workunit.client.0.smithi167.stdout:2/969: creat dd/d18/db7/f143 x:0 0 0 2022-01-31T19:44:27.207 INFO:tasks.workunit.client.1.smithi171.stdout:3/533: creat d4/dd/d87/f9f x:0 0 0 2022-01-31T19:44:27.208 INFO:tasks.workunit.client.1.smithi171.stdout:8/431: rename df/d2a/l2c to df/d2a/d30/l96 0 2022-01-31T19:44:27.208 INFO:tasks.workunit.client.1.smithi171.stdout:8/432: dread - df/d2a/d30/f63 zero size 2022-01-31T19:44:27.208 INFO:tasks.workunit.client.1.smithi171.stdout:8/433: stat df/d5d/d6c/d87 0 2022-01-31T19:44:27.209 INFO:tasks.workunit.client.1.smithi171.stdout:5/466: mkdir d5/d6/d15/d5d/d6f/da1 0 2022-01-31T19:44:27.209 INFO:tasks.workunit.client.1.smithi171.stdout:5/467: chown d5/d6/d15/d23/d67/l2f 30522 1 2022-01-31T19:44:27.209 INFO:tasks.workunit.client.1.smithi171.stdout:3/534: truncate d4/dd/f66 703732 0 2022-01-31T19:44:27.211 INFO:tasks.workunit.client.1.smithi171.stdout:3/535: write d4/dd/d1a/f1b [2884104,7560] 0 2022-01-31T19:44:27.212 INFO:tasks.workunit.client.1.smithi171.stdout:8/434: link df/d1a/f3b df/d18/d19/d82/f97 0 2022-01-31T19:44:27.212 INFO:tasks.workunit.client.1.smithi171.stdout:7/372: mkdir d1/d1e/d2a/d29/d31/d30/d3c/d7e 0 2022-01-31T19:44:27.212 INFO:tasks.workunit.client.1.smithi171.stdout:7/373: write d1/d1e/d2a/d29/f6d [738677,68881] 0 2022-01-31T19:44:27.214 INFO:tasks.workunit.client.1.smithi171.stdout:8/435: creat df/d95/f98 x:0 0 0 2022-01-31T19:44:27.215 INFO:tasks.workunit.client.1.smithi171.stdout:7/374: mknod d1/d1e/d2a/d29/d31/d30/d6b/c7f 0 2022-01-31T19:44:27.217 INFO:tasks.workunit.client.1.smithi171.stdout:7/375: dread d1/d1e/d2a/d29/d31/f34 [0,4194304] 0 2022-01-31T19:44:27.217 INFO:tasks.workunit.client.1.smithi171.stdout:7/376: unlink d1/d1e/d2a/d29/d31/fe 0 2022-01-31T19:44:27.217 INFO:tasks.workunit.client.1.smithi171.stdout:8/436: write df/d1a/f29 [318977,336] 0 2022-01-31T19:44:27.221 INFO:tasks.workunit.client.1.smithi171.stdout:8/437: rename df/d2a/d38/f43 to df/d18/d79/d7a/f99 0 2022-01-31T19:44:27.221 INFO:tasks.workunit.client.1.smithi171.stdout:8/438: write df/f51 [4377936,56521] 0 2022-01-31T19:44:27.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/377: mknod d1/d1e/d2a/d29/d31/d30/d3c/c80 0 2022-01-31T19:44:27.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/378: readlink d1/d1e/d2a/d29/d31/d30/d3c/l49 0 2022-01-31T19:44:27.223 INFO:tasks.workunit.client.1.smithi171.stdout:8/439: unlink df/d18/l2e 0 2022-01-31T19:44:27.224 INFO:tasks.workunit.client.1.smithi171.stdout:8/440: truncate df/d18/d19/d36/f81 253250 0 2022-01-31T19:44:27.224 INFO:tasks.workunit.client.1.smithi171.stdout:7/379: readlink d1/d1e/d2a/d29/d31/d30/d3c/l40 0 2022-01-31T19:44:27.224 INFO:tasks.workunit.client.1.smithi171.stdout:8/441: truncate df/d18/d19/d82/f97 1783132 0 2022-01-31T19:44:27.275 INFO:tasks.workunit.client.0.smithi167.stdout:2/970: getdents dd/df/d1b/d3d/d4a/da7/dcc/df5 0 2022-01-31T19:44:27.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/971: readlink dd/df/d1b/d3d/d4a/la2 0 2022-01-31T19:44:27.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/972: fdatasync dd/df/d1b/d3d/d4a/fac 0 2022-01-31T19:44:27.276 INFO:tasks.workunit.client.0.smithi167.stdout:2/973: dread - dd/d18/d62/d75/d11b/f142 zero size 2022-01-31T19:44:27.277 INFO:tasks.workunit.client.1.smithi171.stdout:5/468: read d5/d6/d15/d23/f87 [3803855,1870] 0 2022-01-31T19:44:27.279 INFO:tasks.workunit.client.1.smithi171.stdout:5/469: symlink d5/d6/d15/d80/d8d/la2 0 2022-01-31T19:44:27.279 INFO:tasks.workunit.client.1.smithi171.stdout:5/470: creat d5/d6/d15/d23/d58/fa3 x:0 0 0 2022-01-31T19:44:27.280 INFO:tasks.workunit.client.0.smithi167.stdout:2/974: symlink dd/df/d1b/d3d/d4a/d4b/l144 0 2022-01-31T19:44:27.280 INFO:tasks.workunit.client.0.smithi167.stdout:2/975: chown dd/d18/l7e 53 1 2022-01-31T19:44:27.295 INFO:tasks.workunit.client.1.smithi171.stdout:7/380: dread d1/d1e/d2a/d29/d31/ff [0,4194304] 0 2022-01-31T19:44:27.295 INFO:tasks.workunit.client.1.smithi171.stdout:7/381: fdatasync d1/d1e/d2a/f3b 0 2022-01-31T19:44:27.296 INFO:tasks.workunit.client.1.smithi171.stdout:7/382: chown d1/d1b/f4c 15 1 2022-01-31T19:44:27.321 INFO:tasks.workunit.client.1.smithi171.stdout:9/595: dwrite dd/f11 [0,4194304] 0 2022-01-31T19:44:27.321 INFO:tasks.workunit.client.1.smithi171.stdout:9/596: dread - dd/df/d15/d38/d41/f50 zero size 2022-01-31T19:44:27.321 INFO:tasks.workunit.client.1.smithi171.stdout:9/597: readlink dd/df/d15/d1a/l43 0 2022-01-31T19:44:27.323 INFO:tasks.workunit.client.1.smithi171.stdout:3/536: dwrite d4/d21/f69 [0,4194304] 0 2022-01-31T19:44:27.323 INFO:tasks.workunit.client.1.smithi171.stdout:6/457: dwrite dc/d1e/d3f/f65 [4194304,4194304] 0 2022-01-31T19:44:27.323 INFO:tasks.workunit.client.1.smithi171.stdout:6/458: creat dc/d19/d30/f92 x:0 0 0 2022-01-31T19:44:27.324 INFO:tasks.workunit.client.1.smithi171.stdout:2/616: dwrite d7/f13 [4194304,4194304] 0 2022-01-31T19:44:27.324 INFO:tasks.workunit.client.1.smithi171.stdout:5/471: dwrite d5/d6/dd/f19 [0,4194304] 0 2022-01-31T19:44:27.324 INFO:tasks.workunit.client.1.smithi171.stdout:8/442: dwrite df/f16 [4194304,4194304] 0 2022-01-31T19:44:27.324 INFO:tasks.workunit.client.1.smithi171.stdout:8/443: fdatasync df/d18/d19/d82/f97 0 2022-01-31T19:44:27.325 INFO:tasks.workunit.client.1.smithi171.stdout:8/444: stat df/f4f 0 2022-01-31T19:44:27.325 INFO:tasks.workunit.client.1.smithi171.stdout:5/472: write d5/d90/f7f [3411089,81747] 0 2022-01-31T19:44:27.325 INFO:tasks.workunit.client.1.smithi171.stdout:5/473: write d5/d6/d15/d23/d67/f4d [758487,63857] 0 2022-01-31T19:44:27.325 INFO:tasks.workunit.client.1.smithi171.stdout:9/598: stat dd/df/d15/c8c 0 2022-01-31T19:44:27.326 INFO:tasks.workunit.client.1.smithi171.stdout:9/599: fsync dd/d1c/da8/fc3 0 2022-01-31T19:44:27.326 INFO:tasks.workunit.client.0.smithi167.stdout:2/976: dwrite dd/d18/f4e [4194304,4194304] 0 2022-01-31T19:44:27.329 INFO:tasks.workunit.client.0.smithi167.stdout:2/977: mknod dd/df/d1b/d2f/d36/d134/c145 0 2022-01-31T19:44:27.330 INFO:tasks.workunit.client.0.smithi167.stdout:2/978: dread - dd/df/d1b/d3d/d4a/da7/dfe/d12f/f130 zero size 2022-01-31T19:44:27.330 INFO:tasks.workunit.client.1.smithi171.stdout:1/463: truncate f3 2586056 0 2022-01-31T19:44:27.331 INFO:tasks.workunit.client.1.smithi171.stdout:1/464: chown c2 471997073 1 2022-01-31T19:44:27.340 INFO:tasks.workunit.client.1.smithi171.stdout:7/383: dwrite d1/d1e/d2a/d29/d31/d30/f73 [0,4194304] 0 2022-01-31T19:44:27.341 INFO:tasks.workunit.client.1.smithi171.stdout:7/384: truncate d1/d1e/d2a/d29/f7a 7113 0 2022-01-31T19:44:27.341 INFO:tasks.workunit.client.1.smithi171.stdout:7/385: dread - d1/d1e/d2a/d29/d31/d30/d6b/f7c zero size 2022-01-31T19:44:27.341 INFO:tasks.workunit.client.1.smithi171.stdout:3/537: mknod d4/dd/d27/d28/d35/d55/d6c/ca0 0 2022-01-31T19:44:27.341 INFO:tasks.workunit.client.1.smithi171.stdout:7/386: write d1/f5 [845516,102843] 0 2022-01-31T19:44:27.342 INFO:tasks.workunit.client.1.smithi171.stdout:6/459: creat dc/d19/d29/d27/f93 x:0 0 0 2022-01-31T19:44:27.358 INFO:tasks.workunit.client.1.smithi171.stdout:2/617: rename d7/d9/d1a/f32 to d7/fc8 0 2022-01-31T19:44:27.358 INFO:tasks.workunit.client.1.smithi171.stdout:2/618: getdents d7/d7c/d83/d85/db9 0 2022-01-31T19:44:27.359 INFO:tasks.workunit.client.1.smithi171.stdout:5/474: dwrite d5/f55 [0,4194304] 0 2022-01-31T19:44:27.361 INFO:tasks.workunit.client.1.smithi171.stdout:1/465: link dd/df/d19/d1b/f72 dd/f91 0 2022-01-31T19:44:27.361 INFO:tasks.workunit.client.1.smithi171.stdout:5/475: dread d5/d6/d15/d23/f25 [0,4194304] 0 2022-01-31T19:44:27.362 INFO:tasks.workunit.client.1.smithi171.stdout:5/476: read d5/ff [2310762,13154] 0 2022-01-31T19:44:27.363 INFO:tasks.workunit.client.1.smithi171.stdout:3/538: creat d4/dd/d39/fa1 x:0 0 0 2022-01-31T19:44:27.365 INFO:tasks.workunit.client.1.smithi171.stdout:7/387: mkdir d1/d26/d81 0 2022-01-31T19:44:27.367 INFO:tasks.workunit.client.1.smithi171.stdout:2/619: creat d7/dd/d5e/d84/fc9 x:0 0 0 2022-01-31T19:44:27.367 INFO:tasks.workunit.client.1.smithi171.stdout:8/445: dwrite df/d18/d19/d36/d70/d7e/f8a [0,4194304] 0 2022-01-31T19:44:27.367 INFO:tasks.workunit.client.1.smithi171.stdout:8/446: fsync df/d18/d19/d21/d67/f91 0 2022-01-31T19:44:27.367 INFO:tasks.workunit.client.1.smithi171.stdout:8/447: dread - df/d18/d19/f61 zero size 2022-01-31T19:44:27.368 INFO:tasks.workunit.client.1.smithi171.stdout:6/460: truncate fa 7895492 0 2022-01-31T19:44:27.369 INFO:tasks.workunit.client.1.smithi171.stdout:5/477: write d5/d6/dd/d66/f9b [81943,103946] 0 2022-01-31T19:44:27.371 INFO:tasks.workunit.client.1.smithi171.stdout:2/620: creat d7/d26/d63/d9f/dc3/fca x:0 0 0 2022-01-31T19:44:27.373 INFO:tasks.workunit.client.0.smithi167.stdout:2/979: dwrite dd/df/d1b/f56 [0,4194304] 0 2022-01-31T19:44:27.373 INFO:tasks.workunit.client.0.smithi167.stdout:2/980: chown dd/d18/d62/d75/d11b/f142 3298079 1 2022-01-31T19:44:27.374 INFO:tasks.workunit.client.1.smithi171.stdout:8/448: stat df/d2a/d30/l96 0 2022-01-31T19:44:27.375 INFO:tasks.workunit.client.0.smithi167.stdout:2/981: truncate dd/d18/db7/feb 4947254 0 2022-01-31T19:44:27.383 INFO:tasks.workunit.client.1.smithi171.stdout:2/621: mkdir d7/dd/d93/dcb 0 2022-01-31T19:44:27.383 INFO:tasks.workunit.client.1.smithi171.stdout:2/622: chown d7/dd/d5e/d84 20 1 2022-01-31T19:44:27.384 INFO:tasks.workunit.client.1.smithi171.stdout:8/449: symlink df/d18/d19/d21/d88/l9a 0 2022-01-31T19:44:27.384 INFO:tasks.workunit.client.1.smithi171.stdout:4/406: sync 2022-01-31T19:44:27.384 INFO:tasks.workunit.client.1.smithi171.stdout:8/450: readlink df/l15 0 2022-01-31T19:44:27.384 INFO:tasks.workunit.client.1.smithi171.stdout:8/451: chown df/c13 33348354 1 2022-01-31T19:44:27.385 INFO:tasks.workunit.client.1.smithi171.stdout:6/461: rmdir dc/d1e 39 2022-01-31T19:44:27.385 INFO:tasks.workunit.client.1.smithi171.stdout:6/462: chown dc/d19/d30/l51 14081 1 2022-01-31T19:44:27.386 INFO:tasks.workunit.client.1.smithi171.stdout:6/463: fdatasync dc/f17 0 2022-01-31T19:44:27.386 INFO:tasks.workunit.client.1.smithi171.stdout:6/464: write dc/d19/d29/f24 [1268461,35908] 0 2022-01-31T19:44:27.386 INFO:tasks.workunit.client.1.smithi171.stdout:0/554: sync 2022-01-31T19:44:27.387 INFO:tasks.workunit.client.1.smithi171.stdout:1/466: dwrite dd/df/d19/d1b/d1e/f6a [0,4194304] 0 2022-01-31T19:44:27.387 INFO:tasks.workunit.client.1.smithi171.stdout:2/623: write d7/dd/d5e/d6b/d81/fa8 [822714,79517] 0 2022-01-31T19:44:27.390 INFO:tasks.workunit.client.1.smithi171.stdout:2/624: write d7/dd/f1e [2009709,130093] 0 2022-01-31T19:44:27.390 INFO:tasks.workunit.client.1.smithi171.stdout:3/539: dwrite d4/dd/d1a/d52/f9b [0,4194304] 0 2022-01-31T19:44:27.391 INFO:tasks.workunit.client.1.smithi171.stdout:2/625: creat d7/d55/fcc x:0 0 0 2022-01-31T19:44:27.391 INFO:tasks.workunit.client.1.smithi171.stdout:2/626: dread - d7/dd/d5e/d6b/f7a zero size 2022-01-31T19:44:27.391 INFO:tasks.workunit.client.1.smithi171.stdout:2/627: write d7/dd/fa1 [745114,126122] 0 2022-01-31T19:44:27.392 INFO:tasks.workunit.client.1.smithi171.stdout:8/452: mknod df/d18/d19/d36/d70/c9b 0 2022-01-31T19:44:27.392 INFO:tasks.workunit.client.1.smithi171.stdout:8/453: chown df/d2a/f40 248 1 2022-01-31T19:44:27.392 INFO:tasks.workunit.client.1.smithi171.stdout:8/454: creat df/d95/f9c x:0 0 0 2022-01-31T19:44:27.393 INFO:tasks.workunit.client.1.smithi171.stdout:6/465: symlink dc/d19/d29/d27/d37/l94 0 2022-01-31T19:44:27.394 INFO:tasks.workunit.client.1.smithi171.stdout:8/455: dread fc [0,4194304] 0 2022-01-31T19:44:27.394 INFO:tasks.workunit.client.1.smithi171.stdout:8/456: chown df/d2a/d30/f63 1077 1 2022-01-31T19:44:27.399 INFO:tasks.workunit.client.1.smithi171.stdout:2/628: write d7/d9/f12 [2712033,128388] 0 2022-01-31T19:44:27.400 INFO:tasks.workunit.client.1.smithi171.stdout:1/467: mknod dd/df/d19/d1b/d77/c92 0 2022-01-31T19:44:27.401 INFO:tasks.workunit.client.1.smithi171.stdout:4/407: getdents d2/d3/d5 0 2022-01-31T19:44:27.401 INFO:tasks.workunit.client.1.smithi171.stdout:4/408: write d2/d3/d5/dc/d4b/d56/d66/d7e/f62 [265464,109219] 0 2022-01-31T19:44:27.405 INFO:tasks.workunit.client.1.smithi171.stdout:3/540: rename d4/dd/d27/d6e/l97 to d4/dd/d27/d28/d35/la2 0 2022-01-31T19:44:27.407 INFO:tasks.workunit.client.1.smithi171.stdout:0/555: getdents d9/db/dc/d3f 0 2022-01-31T19:44:27.408 INFO:tasks.workunit.client.1.smithi171.stdout:0/556: fsync d9/db/dc/d3f/db9/d7a/f9e 0 2022-01-31T19:44:27.408 INFO:tasks.workunit.client.1.smithi171.stdout:0/557: write d9/db/dc/d3f/f2d [287084,78498] 0 2022-01-31T19:44:27.417 INFO:tasks.workunit.client.1.smithi171.stdout:1/468: unlink dd/d23/d2e/d5e/f26 0 2022-01-31T19:44:27.417 INFO:tasks.workunit.client.1.smithi171.stdout:1/469: fsync dd/df/d19/d1b/f49 0 2022-01-31T19:44:27.418 INFO:tasks.workunit.client.1.smithi171.stdout:3/541: mkdir d4/dd/d27/d28/da3 0 2022-01-31T19:44:27.419 INFO:tasks.workunit.client.1.smithi171.stdout:0/558: mkdir d9/db/dc/d3f/d6f/dcc 0 2022-01-31T19:44:27.421 INFO:tasks.workunit.client.1.smithi171.stdout:3/542: rename d4/d21/d3a/c9c to d4/dd/d4b/ca4 0 2022-01-31T19:44:27.423 INFO:tasks.workunit.client.1.smithi171.stdout:3/543: mknod d4/dd/d1a/d6b/ca5 0 2022-01-31T19:44:27.432 INFO:tasks.workunit.client.1.smithi171.stdout:7/388: dwrite d1/d1e/d2a/d29/d31/f51 [0,4194304] 0 2022-01-31T19:44:27.441 INFO:tasks.workunit.client.0.smithi167.stdout:2/982: dwrite dd/d18/d62/f91 [0,4194304] 0 2022-01-31T19:44:27.441 INFO:tasks.workunit.client.0.smithi167.stdout:2/983: fdatasync dd/d18/f13c 0 2022-01-31T19:44:27.441 INFO:tasks.workunit.client.0.smithi167.stdout:2/984: creat dd/d18/d62/d75/dc8/f146 x:0 0 0 2022-01-31T19:44:27.446 INFO:tasks.workunit.client.1.smithi171.stdout:5/478: dwrite d5/d6/d15/f88 [0,4194304] 0 2022-01-31T19:44:27.447 INFO:tasks.workunit.client.1.smithi171.stdout:6/466: dwrite dc/d19/d29/f63 [0,4194304] 0 2022-01-31T19:44:27.447 INFO:tasks.workunit.client.1.smithi171.stdout:4/409: dwrite d2/d3/d5/dc/f6f [0,4194304] 0 2022-01-31T19:44:27.448 INFO:tasks.workunit.client.1.smithi171.stdout:3/544: dwrite d4/dd/d1a/d52/f54 [0,4194304] 0 2022-01-31T19:44:27.448 INFO:tasks.workunit.client.1.smithi171.stdout:3/545: truncate d4/dd/d1a/d52/f8d 962842 0 2022-01-31T19:44:27.449 INFO:tasks.workunit.client.1.smithi171.stdout:3/546: creat d4/dd/d87/fa6 x:0 0 0 2022-01-31T19:44:27.449 INFO:tasks.workunit.client.1.smithi171.stdout:0/559: dwrite d9/db/f41 [0,4194304] 0 2022-01-31T19:44:27.451 INFO:tasks.workunit.client.0.smithi167.stdout:2/985: dread dd/d18/d8d/fa5 [0,4194304] 0 2022-01-31T19:44:27.451 INFO:tasks.workunit.client.0.smithi167.stdout:2/986: chown dd/d18/db7/ffb 182634 1 2022-01-31T19:44:27.452 INFO:tasks.workunit.client.1.smithi171.stdout:0/560: write d9/f8a [1357935,113464] 0 2022-01-31T19:44:27.455 INFO:tasks.workunit.client.1.smithi171.stdout:4/410: dread d2/d3/d5/f59 [0,4194304] 0 2022-01-31T19:44:27.455 INFO:tasks.workunit.client.1.smithi171.stdout:4/411: chown d2/d3/d5/dc/d4b 52531805 1 2022-01-31T19:44:27.456 INFO:tasks.workunit.client.1.smithi171.stdout:4/412: write d2/f3c [4608179,76940] 0 2022-01-31T19:44:27.457 INFO:tasks.workunit.client.0.smithi167.stdout:6/924: sync 2022-01-31T19:44:27.457 INFO:tasks.workunit.client.0.smithi167.stdout:0/901: sync 2022-01-31T19:44:27.457 INFO:tasks.workunit.client.0.smithi167.stdout:0/902: chown d17/d3e/da1 2 1 2022-01-31T19:44:27.460 INFO:tasks.workunit.client.0.smithi167.stdout:0/903: write d17/d19/d1c/d39/d7b/f81 [327672,48295] 0 2022-01-31T19:44:27.462 INFO:tasks.workunit.client.0.smithi167.stdout:0/904: unlink d17/d2c/cb2 0 2022-01-31T19:44:27.462 INFO:tasks.workunit.client.0.smithi167.stdout:0/905: readlink d17/d19/d1c/d39/d7b/da9/d128/l129 0 2022-01-31T19:44:27.463 INFO:tasks.workunit.client.0.smithi167.stdout:0/906: getdents d17/d19/d1c/de1 0 2022-01-31T19:44:27.463 INFO:tasks.workunit.client.0.smithi167.stdout:0/907: stat d17/d3e/d7e/c9f 0 2022-01-31T19:44:27.465 INFO:tasks.workunit.client.0.smithi167.stdout:2/987: dread dd/df/d1b/f56 [0,4194304] 0 2022-01-31T19:44:27.465 INFO:tasks.workunit.client.0.smithi167.stdout:2/988: chown dd/df/d1b/d3d/d4a/d4b/d54 289957 1 2022-01-31T19:44:27.466 INFO:tasks.workunit.client.0.smithi167.stdout:2/989: fdatasync dd/d18/d62/d75/d105/fc5 0 2022-01-31T19:44:27.493 INFO:tasks.workunit.client.1.smithi171.stdout:8/457: rmdir df/d18/d19/d21/d88 39 2022-01-31T19:44:27.497 INFO:tasks.workunit.client.1.smithi171.stdout:7/389: mknod d1/d1e/d2a/d29/d77/c82 0 2022-01-31T19:44:27.499 INFO:tasks.workunit.client.1.smithi171.stdout:2/629: getdents d7/d55 0 2022-01-31T19:44:27.503 INFO:tasks.workunit.client.1.smithi171.stdout:5/479: creat d5/d6/d15/d23/d3d/fa4 x:0 0 0 2022-01-31T19:44:27.508 INFO:tasks.workunit.client.1.smithi171.stdout:6/467: mknod dc/d19/d29/d27/d37/c95 0 2022-01-31T19:44:27.512 INFO:tasks.workunit.client.1.smithi171.stdout:3/547: mkdir d4/dd/d27/d28/d35/d55/d8f/da7 0 2022-01-31T19:44:27.519 INFO:tasks.workunit.client.1.smithi171.stdout:0/561: rename d9/d24/d4c/d35/l72 to d9/db/dc/d3f/db9/d7a/d88/lcd 0 2022-01-31T19:44:27.520 INFO:tasks.workunit.client.1.smithi171.stdout:4/413: creat d2/d3/d5/dc/d4b/d56/d66/d7e/f8c x:0 0 0 2022-01-31T19:44:27.521 INFO:tasks.workunit.client.1.smithi171.stdout:8/458: creat df/d18/d79/f9d x:0 0 0 2022-01-31T19:44:27.521 INFO:tasks.workunit.client.1.smithi171.stdout:7/390: creat d1/d1e/d2a/f83 x:0 0 0 2022-01-31T19:44:27.523 INFO:tasks.workunit.client.1.smithi171.stdout:0/562: mknod d9/db/cce 0 2022-01-31T19:44:27.528 INFO:tasks.workunit.client.1.smithi171.stdout:4/414: write d2/d3/f25 [3371908,41927] 0 2022-01-31T19:44:27.530 INFO:tasks.workunit.client.1.smithi171.stdout:4/415: write d2/d3/d5/d41/f89 [2237046,18168] 0 2022-01-31T19:44:27.530 INFO:tasks.workunit.client.1.smithi171.stdout:2/630: dwrite d7/d9/f6a [0,4194304] 0 2022-01-31T19:44:27.531 INFO:tasks.workunit.client.1.smithi171.stdout:3/548: dread d4/dd/d27/d28/d35/f48 [0,4194304] 0 2022-01-31T19:44:27.532 INFO:tasks.workunit.client.1.smithi171.stdout:3/549: readlink d4/dd/d27/l7d 0 2022-01-31T19:44:27.533 INFO:tasks.workunit.client.1.smithi171.stdout:4/416: creat d2/d3/d5/dc/d4b/d56/d66/d7e/d12/f8d x:0 0 0 2022-01-31T19:44:27.533 INFO:tasks.workunit.client.1.smithi171.stdout:2/631: write d7/f56 [913308,90205] 0 2022-01-31T19:44:27.536 INFO:tasks.workunit.client.1.smithi171.stdout:4/417: mknod d2/d3/d5/dc/d4b/c8e 0 2022-01-31T19:44:27.537 INFO:tasks.workunit.client.1.smithi171.stdout:2/632: symlink d7/d26/d3d/lcd 0 2022-01-31T19:44:27.539 INFO:tasks.workunit.client.1.smithi171.stdout:4/418: creat d2/d3/d5/dc/d4b/d56/d66/d77/f8f x:0 0 0 2022-01-31T19:44:27.539 INFO:tasks.workunit.client.1.smithi171.stdout:4/419: chown d2/d3/d5/dc/d4b/d56/l8b 14714 1 2022-01-31T19:44:27.540 INFO:tasks.workunit.client.1.smithi171.stdout:4/420: creat d2/f90 x:0 0 0 2022-01-31T19:44:27.540 INFO:tasks.workunit.client.1.smithi171.stdout:4/421: write d2/d3/d5/dc/d4b/d56/d66/d7e/f62 [750413,65704] 0 2022-01-31T19:44:27.545 INFO:tasks.workunit.client.0.smithi167.stdout:0/908: unlink d17/d19/fb1 0 2022-01-31T19:44:27.548 INFO:tasks.workunit.client.1.smithi171.stdout:3/550: dread d4/dd/d27/f3f [0,4194304] 0 2022-01-31T19:44:27.548 INFO:tasks.workunit.client.0.smithi167.stdout:0/909: rename d17/d69/f10d to d17/d19/d1c/d39/d44/d88/d98/de8/f12b 0 2022-01-31T19:44:27.548 INFO:tasks.workunit.client.0.smithi167.stdout:2/990: dread dd/df/d1b/d3d/f123 [0,4194304] 0 2022-01-31T19:44:27.549 INFO:tasks.workunit.client.1.smithi171.stdout:3/551: write d4/dd/d27/d28/d35/f38 [3386323,39838] 0 2022-01-31T19:44:27.550 INFO:tasks.workunit.client.1.smithi171.stdout:3/552: chown d4/dd/d1a/f1b 3569787 1 2022-01-31T19:44:27.550 INFO:tasks.workunit.client.1.smithi171.stdout:3/553: chown d4/l2c 652424030 1 2022-01-31T19:44:27.550 INFO:tasks.workunit.client.1.smithi171.stdout:3/554: chown d4/dd/d1a/d52/f96 10275593 1 2022-01-31T19:44:27.550 INFO:tasks.workunit.client.1.smithi171.stdout:2/633: dread d7/dd/f80 [0,4194304] 0 2022-01-31T19:44:27.551 INFO:tasks.workunit.client.1.smithi171.stdout:3/555: mknod d4/dd/d27/d28/d35/d55/d8f/da7/ca8 0 2022-01-31T19:44:27.551 INFO:tasks.workunit.client.1.smithi171.stdout:2/634: rename d7/d55/fcc to d7/d9/d1a/d45/da4/fce 0 2022-01-31T19:44:27.552 INFO:tasks.workunit.client.1.smithi171.stdout:2/635: fsync d7/d9/d1a/d45/da4/fb4 0 2022-01-31T19:44:27.552 INFO:tasks.workunit.client.1.smithi171.stdout:3/556: creat d4/dd/d27/d28/d35/fa9 x:0 0 0 2022-01-31T19:44:27.553 INFO:tasks.workunit.client.1.smithi171.stdout:3/557: creat d4/dd/d39/faa x:0 0 0 2022-01-31T19:44:27.554 INFO:tasks.workunit.client.0.smithi167.stdout:2/991: dread dd/df/d1b/d2f/d36/fb9 [0,4194304] 0 2022-01-31T19:44:27.554 INFO:tasks.workunit.client.0.smithi167.stdout:0/910: dread d17/d3e/d7e/f73 [0,4194304] 0 2022-01-31T19:44:27.555 INFO:tasks.workunit.client.0.smithi167.stdout:2/992: mkdir dd/df/d1b/d3d/d4a/d4b/d147 0 2022-01-31T19:44:27.555 INFO:tasks.workunit.client.1.smithi171.stdout:2/636: mkdir d7/dd/d5e/d6b/dcf 0 2022-01-31T19:44:27.555 INFO:tasks.workunit.client.1.smithi171.stdout:2/637: chown d7/d26/d3d/d40/f78 17 1 2022-01-31T19:44:27.555 INFO:tasks.workunit.client.1.smithi171.stdout:2/638: creat d7/dd/d93/fd0 x:0 0 0 2022-01-31T19:44:27.556 INFO:tasks.workunit.client.0.smithi167.stdout:0/911: symlink d17/d19/d1c/d39/d44/d49/l12c 0 2022-01-31T19:44:27.557 INFO:tasks.workunit.client.1.smithi171.stdout:5/480: dwrite d5/f57 [4194304,4194304] 0 2022-01-31T19:44:27.557 INFO:tasks.workunit.client.1.smithi171.stdout:3/558: rename d4/dd/d1a to d4/dd/d27/d28/d35/d55/d8f/d92/dab 0 2022-01-31T19:44:27.557 INFO:tasks.workunit.client.1.smithi171.stdout:2/639: mknod d7/d7c/d83/d85/db9/cd1 0 2022-01-31T19:44:27.559 INFO:tasks.workunit.client.1.smithi171.stdout:5/481: truncate d5/d6/dd/f19 1252652 0 2022-01-31T19:44:27.560 INFO:tasks.workunit.client.1.smithi171.stdout:5/482: write d5/d6/dd/f35 [1872700,6936] 0 2022-01-31T19:44:27.560 INFO:tasks.workunit.client.1.smithi171.stdout:5/483: fdatasync d5/d6/dd/f19 0 2022-01-31T19:44:27.561 INFO:tasks.workunit.client.1.smithi171.stdout:3/559: mknod d4/dd/d27/d4e/cac 0 2022-01-31T19:44:27.562 INFO:tasks.workunit.client.1.smithi171.stdout:2/640: mkdir d7/d26/d3d/d40/dd2 0 2022-01-31T19:44:27.562 INFO:tasks.workunit.client.1.smithi171.stdout:2/641: creat d7/d7c/fd3 x:0 0 0 2022-01-31T19:44:27.563 INFO:tasks.workunit.client.1.smithi171.stdout:5/484: mknod d5/d59/da0/ca5 0 2022-01-31T19:44:27.565 INFO:tasks.workunit.client.1.smithi171.stdout:2/642: read d7/dd/f39 [1075517,114455] 0 2022-01-31T19:44:27.565 INFO:tasks.workunit.client.1.smithi171.stdout:2/643: creat d7/d9/d1a/d45/da4/fd4 x:0 0 0 2022-01-31T19:44:27.565 INFO:tasks.workunit.client.1.smithi171.stdout:2/644: write d7/dd/f52 [1505460,49660] 0 2022-01-31T19:44:27.565 INFO:tasks.workunit.client.1.smithi171.stdout:2/645: dread - d7/d7c/fd3 zero size 2022-01-31T19:44:27.566 INFO:tasks.workunit.client.1.smithi171.stdout:2/646: fsync d7/d26/d3d/d40/f5c 0 2022-01-31T19:44:27.566 INFO:tasks.workunit.client.1.smithi171.stdout:2/647: creat d7/fd5 x:0 0 0 2022-01-31T19:44:27.566 INFO:tasks.workunit.client.1.smithi171.stdout:3/560: creat d4/dd/d27/d28/d35/d55/d6c/fad x:0 0 0 2022-01-31T19:44:27.567 INFO:tasks.workunit.client.0.smithi167.stdout:0/912: dread d17/d19/d1c/d39/d44/d88/d98/de8/f12b [0,4194304] 0 2022-01-31T19:44:27.567 INFO:tasks.workunit.client.0.smithi167.stdout:0/913: write d17/d2c/d87/f97 [26827,39896] 0 2022-01-31T19:44:27.568 INFO:tasks.workunit.client.0.smithi167.stdout:0/914: mkdir d17/d33/da4/dd0/d105/d12d 0 2022-01-31T19:44:27.568 INFO:tasks.workunit.client.0.smithi167.stdout:0/915: truncate d17/d19/f51 31693 0 2022-01-31T19:44:27.569 INFO:tasks.workunit.client.1.smithi171.stdout:2/648: creat d7/d7c/d83/d85/fd6 x:0 0 0 2022-01-31T19:44:27.569 INFO:tasks.workunit.client.0.smithi167.stdout:0/916: creat d17/d2c/f12e x:0 0 0 2022-01-31T19:44:27.570 INFO:tasks.workunit.client.0.smithi167.stdout:0/917: dread d17/d19/d1c/d39/d44/f56 [0,4194304] 0 2022-01-31T19:44:27.570 INFO:tasks.workunit.client.0.smithi167.stdout:0/918: write d17/d19/d1c/d39/d44/d88/d98/de8/f10f [10113,83729] 0 2022-01-31T19:44:27.576 INFO:tasks.workunit.client.1.smithi171.stdout:2/649: dread d7/dd/f7e [0,4194304] 0 2022-01-31T19:44:27.576 INFO:tasks.workunit.client.1.smithi171.stdout:2/650: chown d7/dd/d5e/d6b/dcf 7217 1 2022-01-31T19:44:27.577 INFO:tasks.workunit.client.0.smithi167.stdout:0/919: write fc [1483383,122722] 0 2022-01-31T19:44:27.577 INFO:tasks.workunit.client.1.smithi171.stdout:2/651: link d7/dd/d5e/d6b/d81/d8d/da2/cb2 d7/d26/d3d/d40/cd7 0 2022-01-31T19:44:27.578 INFO:tasks.workunit.client.0.smithi167.stdout:0/920: mknod d17/d33/da4/c12f 0 2022-01-31T19:44:27.579 INFO:tasks.workunit.client.0.smithi167.stdout:0/921: mknod d17/d3e/d7e/c130 0 2022-01-31T19:44:27.579 INFO:tasks.workunit.client.0.smithi167.stdout:0/922: fsync d17/d69/f78 0 2022-01-31T19:44:27.580 INFO:tasks.workunit.client.1.smithi171.stdout:5/485: dread d5/f9f [0,4194304] 0 2022-01-31T19:44:27.580 INFO:tasks.workunit.client.1.smithi171.stdout:5/486: write d5/d6/d15/d5d/d6f/f9c [873981,116096] 0 2022-01-31T19:44:27.581 INFO:tasks.workunit.client.1.smithi171.stdout:5/487: creat d5/d6/d15/d5d/d6f/fa6 x:0 0 0 2022-01-31T19:44:27.581 INFO:tasks.workunit.client.1.smithi171.stdout:5/488: write d5/d6/d15/d23/f25 [597530,47978] 0 2022-01-31T19:44:27.581 INFO:tasks.workunit.client.1.smithi171.stdout:8/459: dwrite df/d18/d19/d21/f26 [0,4194304] 0 2022-01-31T19:44:27.582 INFO:tasks.workunit.client.0.smithi167.stdout:0/923: mkdir d17/d2c/d87/df4/d131 0 2022-01-31T19:44:27.582 INFO:tasks.workunit.client.1.smithi171.stdout:5/489: creat d5/d59/da0/fa7 x:0 0 0 2022-01-31T19:44:27.583 INFO:tasks.workunit.client.0.smithi167.stdout:2/993: dwrite dd/d18/d8d/fa5 [0,4194304] 0 2022-01-31T19:44:27.584 INFO:tasks.workunit.client.0.smithi167.stdout:2/994: chown dd/f55 1017840 1 2022-01-31T19:44:27.584 INFO:tasks.workunit.client.1.smithi171.stdout:8/460: link df/d2a/f49 df/d18/d19/d36/d70/f9e 0 2022-01-31T19:44:27.584 INFO:tasks.workunit.client.1.smithi171.stdout:8/461: symlink df/d18/d79/d7a/l9f 0 2022-01-31T19:44:27.585 INFO:tasks.workunit.client.1.smithi171.stdout:8/462: creat df/d5d/d6c/d87/fa0 x:0 0 0 2022-01-31T19:44:27.586 INFO:tasks.workunit.client.1.smithi171.stdout:8/463: chown df/d18/d79/c80 0 1 2022-01-31T19:44:27.586 INFO:tasks.workunit.client.1.smithi171.stdout:8/464: creat df/d18/d79/fa1 x:0 0 0 2022-01-31T19:44:27.586 INFO:tasks.workunit.client.1.smithi171.stdout:8/465: fdatasync df/d18/d19/d21/f7f 0 2022-01-31T19:44:27.586 INFO:tasks.workunit.client.1.smithi171.stdout:8/466: chown df/d18/d19/f61 179891449 1 2022-01-31T19:44:27.587 INFO:tasks.workunit.client.1.smithi171.stdout:5/490: dread d5/d6/dd/f22 [0,4194304] 0 2022-01-31T19:44:27.587 INFO:tasks.workunit.client.1.smithi171.stdout:5/491: stat d5/d6/d15/d23/d58 0 2022-01-31T19:44:27.587 INFO:tasks.workunit.client.1.smithi171.stdout:8/467: mkdir df/d18/d79/da2 0 2022-01-31T19:44:27.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/492: creat d5/d90/fa8 x:0 0 0 2022-01-31T19:44:27.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/493: mknod d5/d6/d15/d23/d3d/ca9 0 2022-01-31T19:44:27.590 INFO:tasks.workunit.client.1.smithi171.stdout:5/494: symlink d5/d6/d15/d5d/laa 0 2022-01-31T19:44:27.595 INFO:tasks.workunit.client.1.smithi171.stdout:5/495: write d5/d6/d15/d23/d3d/f49 [201611,60099] 0 2022-01-31T19:44:27.596 INFO:tasks.workunit.client.1.smithi171.stdout:0/563: dwrite d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:27.596 INFO:tasks.workunit.client.1.smithi171.stdout:4/422: fdatasync d2/d3/d5/d41/f89 0 2022-01-31T19:44:27.597 INFO:tasks.workunit.client.1.smithi171.stdout:6/468: dwrite dc/d19/d30/f92 [0,4194304] 0 2022-01-31T19:44:27.597 INFO:tasks.workunit.client.1.smithi171.stdout:6/469: dread - dc/d18/d53/f58 zero size 2022-01-31T19:44:27.598 INFO:tasks.workunit.client.1.smithi171.stdout:6/470: fsync dc/d19/d29/f63 0 2022-01-31T19:44:27.598 INFO:tasks.workunit.client.1.smithi171.stdout:6/471: write dc/d19/d29/d27/d37/d3b/f8e [233298,29577] 0 2022-01-31T19:44:27.599 INFO:tasks.workunit.client.1.smithi171.stdout:7/391: dwrite d1/d26/d47/f7b [0,4194304] 0 2022-01-31T19:44:27.599 INFO:tasks.workunit.client.1.smithi171.stdout:7/392: readlink d1/d1b/l3e 0 2022-01-31T19:44:27.601 INFO:tasks.workunit.client.1.smithi171.stdout:0/564: symlink d9/db/lcf 0 2022-01-31T19:44:27.605 INFO:tasks.workunit.client.1.smithi171.stdout:4/423: unlink d2/d3/d5/dc/d4b/d56/d66/d7e/d12/d15/f64 0 2022-01-31T19:44:27.605 INFO:tasks.workunit.client.1.smithi171.stdout:4/424: dread - d2/d3/d5/f46 zero size 2022-01-31T19:44:27.610 INFO:tasks.workunit.client.1.smithi171.stdout:6/472: dread dc/d18/f2a [0,4194304] 0 2022-01-31T19:44:27.610 INFO:tasks.workunit.client.1.smithi171.stdout:6/473: creat dc/d8b/f96 x:0 0 0 2022-01-31T19:44:27.610 INFO:tasks.workunit.client.1.smithi171.stdout:6/474: write dc/d21/f5a [844371,59690] 0 2022-01-31T19:44:27.611 INFO:tasks.workunit.client.1.smithi171.stdout:6/475: chown dc/d18/d53/l79 9 1 2022-01-31T19:44:27.613 INFO:tasks.workunit.client.0.smithi167.stdout:2/995: dwrite dd/df/d1b/d2f/d36/fb9 [0,4194304] 0 2022-01-31T19:44:27.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/393: symlink d1/d1e/d2a/l84 0 2022-01-31T19:44:27.615 INFO:tasks.workunit.client.1.smithi171.stdout:7/394: creat d1/d1e/f85 x:0 0 0 2022-01-31T19:44:27.618 INFO:tasks.workunit.client.1.smithi171.stdout:0/565: symlink d9/db/dc/ld0 0 2022-01-31T19:44:27.618 INFO:tasks.workunit.client.1.smithi171.stdout:0/566: creat d9/db/d22/fd1 x:0 0 0 2022-01-31T19:44:27.619 INFO:tasks.workunit.client.1.smithi171.stdout:0/567: readlink d9/d24/d92/l68 0 2022-01-31T19:44:27.619 INFO:tasks.workunit.client.1.smithi171.stdout:0/568: dread - d9/d24/d4c/f77 zero size 2022-01-31T19:44:27.619 INFO:tasks.workunit.client.1.smithi171.stdout:0/569: fsync d9/db/dc/d3f/f2d 0 2022-01-31T19:44:27.619 INFO:tasks.workunit.client.1.smithi171.stdout:3/561: dwrite d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f9b [0,4194304] 0 2022-01-31T19:44:27.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/476: dread dc/f4a [0,4194304] 0 2022-01-31T19:44:27.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/477: chown dc/d18 20 1 2022-01-31T19:44:27.622 INFO:tasks.workunit.client.1.smithi171.stdout:6/478: chown dc/d21/f4c 1524955677 1 2022-01-31T19:44:27.623 INFO:tasks.workunit.client.1.smithi171.stdout:6/479: write dc/d19/d29/f34 [5207836,41019] 0 2022-01-31T19:44:27.632 INFO:tasks.workunit.client.1.smithi171.stdout:0/570: creat d9/d24/d92/d9d/dc6/fd2 x:0 0 0 2022-01-31T19:44:27.634 INFO:tasks.workunit.client.1.smithi171.stdout:3/562: link d4/dd/l33 d4/dd/d27/d4e/d62/lae 0 2022-01-31T19:44:27.634 INFO:tasks.workunit.client.1.smithi171.stdout:3/563: stat d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f96 0 2022-01-31T19:44:27.635 INFO:tasks.workunit.client.1.smithi171.stdout:6/480: rmdir dc/d19/d29/d27/d71 0 2022-01-31T19:44:27.635 INFO:tasks.workunit.client.1.smithi171.stdout:6/481: chown dc/d8b 116 1 2022-01-31T19:44:27.636 INFO:tasks.workunit.client.1.smithi171.stdout:0/571: rmdir d9/d24/d92/d9d 39 2022-01-31T19:44:27.637 INFO:tasks.workunit.client.1.smithi171.stdout:0/572: stat d9/d24/c51 0 2022-01-31T19:44:27.637 INFO:tasks.workunit.client.1.smithi171.stdout:0/573: dread - d9/db/fc9 zero size 2022-01-31T19:44:27.637 INFO:tasks.workunit.client.1.smithi171.stdout:0/574: write d9/db/dc2/fc5 [261320,50149] 0 2022-01-31T19:44:27.637 INFO:tasks.workunit.client.1.smithi171.stdout:0/575: fdatasync d9/d24/f4d 0 2022-01-31T19:44:27.638 INFO:tasks.workunit.client.1.smithi171.stdout:0/576: chown d9/db/dc/d3f/l7c 5757 1 2022-01-31T19:44:27.638 INFO:tasks.workunit.client.1.smithi171.stdout:6/482: creat dc/d1e/d3f/f97 x:0 0 0 2022-01-31T19:44:27.639 INFO:tasks.workunit.client.1.smithi171.stdout:0/577: creat d9/db/dc2/fd3 x:0 0 0 2022-01-31T19:44:27.639 INFO:tasks.workunit.client.1.smithi171.stdout:6/483: write dc/d19/d29/f5c [3468121,98280] 0 2022-01-31T19:44:27.640 INFO:tasks.workunit.client.1.smithi171.stdout:6/484: write dc/d1e/d3f/f65 [9072410,81292] 0 2022-01-31T19:44:27.640 INFO:tasks.workunit.client.1.smithi171.stdout:6/485: creat dc/d21/f98 x:0 0 0 2022-01-31T19:44:27.640 INFO:tasks.workunit.client.1.smithi171.stdout:8/468: dwrite df/f20 [0,4194304] 0 2022-01-31T19:44:27.640 INFO:tasks.workunit.client.1.smithi171.stdout:8/469: creat df/d27/fa3 x:0 0 0 2022-01-31T19:44:27.641 INFO:tasks.workunit.client.1.smithi171.stdout:6/486: write dc/d19/d30/f92 [3652165,12119] 0 2022-01-31T19:44:27.641 INFO:tasks.workunit.client.1.smithi171.stdout:5/496: dwrite d5/d6/dd/f45 [0,4194304] 0 2022-01-31T19:44:27.642 INFO:tasks.workunit.client.1.smithi171.stdout:5/497: creat d5/d6/d15/d23/d6b/fab x:0 0 0 2022-01-31T19:44:27.642 INFO:tasks.workunit.client.1.smithi171.stdout:7/395: dwrite d1/d26/f50 [0,4194304] 0 2022-01-31T19:44:27.643 INFO:tasks.workunit.client.1.smithi171.stdout:7/396: creat d1/d1e/d2a/d29/d31/d30/d6b/f86 x:0 0 0 2022-01-31T19:44:27.643 INFO:tasks.workunit.client.0.smithi167.stdout:2/996: dwrite dd/d18/d62/f64 [0,4194304] 0 2022-01-31T19:44:27.644 INFO:tasks.workunit.client.0.smithi167.stdout:0/924: getdents d17/d19/d1c/d39/d44/d88/d98/de8 0 2022-01-31T19:44:27.644 INFO:tasks.workunit.client.1.smithi171.stdout:8/470: mkdir df/d18/da4 0 2022-01-31T19:44:27.645 INFO:tasks.workunit.client.1.smithi171.stdout:8/471: fdatasync df/d18/f4e 0 2022-01-31T19:44:27.646 INFO:tasks.workunit.client.1.smithi171.stdout:7/397: dread d1/d26/f5c [0,4194304] 0 2022-01-31T19:44:27.648 INFO:tasks.workunit.client.1.smithi171.stdout:2/652: fsync d7/fd5 0 2022-01-31T19:44:27.648 INFO:tasks.workunit.client.1.smithi171.stdout:2/653: getdents d7/dd/d5e/d6b/dcf 0 2022-01-31T19:44:27.648 INFO:tasks.workunit.client.1.smithi171.stdout:2/654: chown d7/d7c 30 1 2022-01-31T19:44:27.648 INFO:tasks.workunit.client.1.smithi171.stdout:2/655: fsync d7/d9/d1a/d45/da4/fce 0 2022-01-31T19:44:27.649 INFO:tasks.workunit.client.0.smithi167.stdout:0/925: link d17/d33/fa3 d17/d3e/d7e/dc7/dde/df1/d113/f132 0 2022-01-31T19:44:27.649 INFO:tasks.workunit.client.0.smithi167.stdout:0/926: truncate d17/d19/d1c/d39/d7b/f81 2351442 0 2022-01-31T19:44:27.655 INFO:tasks.workunit.client.1.smithi171.stdout:0/578: dread d9/db/dc/d3f/db9/f3e [0,4194304] 0 2022-01-31T19:44:27.655 INFO:tasks.workunit.client.1.smithi171.stdout:0/579: write d9/db/f21 [647448,4981] 0 2022-01-31T19:44:27.657 INFO:tasks.workunit.client.1.smithi171.stdout:0/580: dread d9/d24/d4c/fa9 [0,4194304] 0 2022-01-31T19:44:27.657 INFO:tasks.workunit.client.1.smithi171.stdout:0/581: chown d9/d24/d4c/d35/f74 294257685 1 2022-01-31T19:44:27.657 INFO:tasks.workunit.client.1.smithi171.stdout:0/582: dread - d9/db/d22/fc7 zero size 2022-01-31T19:44:27.659 INFO:tasks.workunit.client.1.smithi171.stdout:5/498: rename d5/d59/da0/ca5 to d5/d59/da0/cac 0 2022-01-31T19:44:27.667 INFO:tasks.workunit.client.1.smithi171.stdout:6/487: truncate dc/d19/d30/d3c/f4b 610214 0 2022-01-31T19:44:27.668 INFO:tasks.workunit.client.1.smithi171.stdout:2/656: symlink d7/dd/d5e/d6b/d81/d8d/da2/ld8 0 2022-01-31T19:44:27.669 INFO:tasks.workunit.client.1.smithi171.stdout:0/583: dread d9/f13 [4194304,4194304] 0 2022-01-31T19:44:27.670 INFO:tasks.workunit.client.1.smithi171.stdout:0/584: read - d9/db/f47 zero size 2022-01-31T19:44:27.673 INFO:tasks.workunit.client.1.smithi171.stdout:9/600: sync 2022-01-31T19:44:27.673 INFO:tasks.workunit.client.1.smithi171.stdout:9/601: creat dd/df/d15/d53/fc4 x:0 0 0 2022-01-31T19:44:27.675 INFO:tasks.workunit.client.1.smithi171.stdout:2/657: unlink d7/dd/d5e/d6b/c8c 0 2022-01-31T19:44:27.676 INFO:tasks.workunit.client.1.smithi171.stdout:0/585: creat d9/fd4 x:0 0 0 2022-01-31T19:44:27.677 INFO:tasks.workunit.client.1.smithi171.stdout:4/425: rename d2/d3/d5/dc/d4b/d56/d66 to d2/d3/d5/d91 0 2022-01-31T19:44:27.678 INFO:tasks.workunit.client.1.smithi171.stdout:2/658: write d7/dd/f7e [1112896,37662] 0 2022-01-31T19:44:27.680 INFO:tasks.workunit.client.1.smithi171.stdout:8/472: dwrite f0 [0,4194304] 0 2022-01-31T19:44:27.682 INFO:tasks.workunit.client.1.smithi171.stdout:9/602: link dd/df/c28 dd/d1c/cc5 0 2022-01-31T19:44:27.682 INFO:tasks.workunit.client.1.smithi171.stdout:9/603: write dd/df/d15/d38/d4d/f63 [646244,38139] 0 2022-01-31T19:44:27.682 INFO:tasks.workunit.client.0.smithi167.stdout:2/997: dwrite dd/df/d22/d96/d9e/fb2 [0,4194304] 0 2022-01-31T19:44:27.685 INFO:tasks.workunit.client.0.smithi167.stdout:2/998: mknod dd/df/d1b/d3d/d4a/d4b/d54/dca/ddf/c148 0 2022-01-31T19:44:27.693 INFO:tasks.workunit.client.1.smithi171.stdout:8/473: write df/d18/d79/f8c [4475763,92363] 0 2022-01-31T19:44:27.695 INFO:tasks.workunit.client.0.smithi167.stdout:2/999: truncate dd/fb6 1096035 0 2022-01-31T19:44:27.702 INFO:tasks.workunit.client.1.smithi171.stdout:6/488: getdents dc/d8b 0 2022-01-31T19:44:27.702 INFO:tasks.workunit.client.1.smithi171.stdout:6/489: chown dc/d19/d29/d27/d37/c4d 1168888072 1 2022-01-31T19:44:27.703 INFO:tasks.workunit.client.1.smithi171.stdout:9/604: symlink dd/df/d15/d38/d4d/d80/lc6 0 2022-01-31T19:44:27.709 INFO:tasks.workunit.client.1.smithi171.stdout:3/564: rename d4/dd/d39/faa to d4/dd/d87/d8e/faf 0 2022-01-31T19:44:27.710 INFO:tasks.workunit.client.1.smithi171.stdout:0/586: dwrite d9/d24/d92/f5f [0,4194304] 0 2022-01-31T19:44:27.710 INFO:tasks.workunit.client.1.smithi171.stdout:8/474: rename df/l3a to df/la5 0 2022-01-31T19:44:27.713 INFO:tasks.workunit.client.1.smithi171.stdout:2/659: dwrite d7/d26/d3d/d40/f57 [0,4194304] 0 2022-01-31T19:44:27.714 INFO:tasks.workunit.client.1.smithi171.stdout:6/490: mkdir dc/d19/d29/d27/d37/d68/d99 0 2022-01-31T19:44:27.715 INFO:tasks.workunit.client.1.smithi171.stdout:6/491: readlink dc/d19/l2d 0 2022-01-31T19:44:27.716 INFO:tasks.workunit.client.1.smithi171.stdout:9/605: link fc dd/df/d15/d1a/d64/fc7 0 2022-01-31T19:44:27.717 INFO:tasks.workunit.client.1.smithi171.stdout:3/565: rmdir d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52 39 2022-01-31T19:44:27.718 INFO:tasks.workunit.client.1.smithi171.stdout:0/587: mknod d9/db/dc/d3f/db9/d7a/cd5 0 2022-01-31T19:44:27.719 INFO:tasks.workunit.client.1.smithi171.stdout:8/475: unlink df/d18/d19/d21/f7f 0 2022-01-31T19:44:27.722 INFO:tasks.workunit.client.1.smithi171.stdout:2/660: unlink d7/d9/d1a/d45/c38 0 2022-01-31T19:44:27.723 INFO:tasks.workunit.client.1.smithi171.stdout:2/661: dread - d7/d9/d1a/d45/da4/fb4 zero size 2022-01-31T19:44:27.724 INFO:tasks.workunit.client.1.smithi171.stdout:0/588: creat d9/db/d59/d71/dbe/fd6 x:0 0 0 2022-01-31T19:44:27.724 INFO:tasks.workunit.client.1.smithi171.stdout:8/476: truncate df/d18/f4d 346476 0 2022-01-31T19:44:27.726 INFO:tasks.workunit.client.1.smithi171.stdout:3/566: symlink d4/dd/d87/d8e/lb0 0 2022-01-31T19:44:27.727 INFO:tasks.workunit.client.1.smithi171.stdout:3/567: chown d4/dd/d27/d28/d35/d55 1 1 2022-01-31T19:44:27.727 INFO:tasks.workunit.client.1.smithi171.stdout:3/568: readlink d4/d21/d3a/d90/l9d 0 2022-01-31T19:44:27.728 INFO:tasks.workunit.client.1.smithi171.stdout:0/589: rename d9/d24/d4c/f5e to d9/db/dc2/fd7 0 2022-01-31T19:44:27.731 INFO:tasks.workunit.client.1.smithi171.stdout:8/477: mknod df/d18/da4/ca6 0 2022-01-31T19:44:27.734 INFO:tasks.workunit.client.1.smithi171.stdout:4/426: dwrite d2/d3/d5/dc/f6f [0,4194304] 0 2022-01-31T19:44:27.735 INFO:tasks.workunit.client.1.smithi171.stdout:8/478: write df/d18/d19/f1b [724037,20323] 0 2022-01-31T19:44:27.735 INFO:tasks.workunit.client.1.smithi171.stdout:8/479: stat df/l59 0 2022-01-31T19:44:27.736 INFO:tasks.workunit.client.1.smithi171.stdout:2/662: dread d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:27.736 INFO:tasks.workunit.client.1.smithi171.stdout:4/427: creat d2/d3/d5/d3b/f92 x:0 0 0 2022-01-31T19:44:27.737 INFO:tasks.workunit.client.1.smithi171.stdout:6/492: dwrite dc/d18/d86/f87 [0,4194304] 0 2022-01-31T19:44:27.738 INFO:tasks.workunit.client.1.smithi171.stdout:1/470: sync 2022-01-31T19:44:27.740 INFO:tasks.workunit.client.1.smithi171.stdout:2/663: mknod d7/dd/d93/dc0/cd9 0 2022-01-31T19:44:27.741 INFO:tasks.workunit.client.1.smithi171.stdout:8/480: truncate df/d18/f4d 422456 0 2022-01-31T19:44:27.741 INFO:tasks.workunit.client.1.smithi171.stdout:8/481: chown df/d2a/f49 2 1 2022-01-31T19:44:27.743 INFO:tasks.workunit.client.1.smithi171.stdout:0/590: dread d9/d24/d92/d9d/d9f/fb7 [0,4194304] 0 2022-01-31T19:44:27.743 INFO:tasks.workunit.client.1.smithi171.stdout:0/591: creat d9/db/d59/d71/dbe/fd8 x:0 0 0 2022-01-31T19:44:27.743 INFO:tasks.workunit.client.1.smithi171.stdout:4/428: creat d2/d3/d5/d70/d7b/f93 x:0 0 0 2022-01-31T19:44:27.751 INFO:tasks.workunit.client.1.smithi171.stdout:3/569: dwrite d4/dd/d27/d28/d35/f48 [0,4194304] 0 2022-01-31T19:44:27.751 INFO:tasks.workunit.client.1.smithi171.stdout:3/570: chown d4/d21/d3a/d59/c68 8230008 1 2022-01-31T19:44:27.751 INFO:tasks.workunit.client.1.smithi171.stdout:3/571: chown d4/dd/d27/d4e/d62/lae 100828 1 2022-01-31T19:44:27.754 INFO:tasks.workunit.client.1.smithi171.stdout:2/664: rmdir d7/d7c/d83 39 2022-01-31T19:44:27.756 INFO:tasks.workunit.client.1.smithi171.stdout:6/493: dread f2 [0,4194304] 0 2022-01-31T19:44:27.758 INFO:tasks.workunit.client.1.smithi171.stdout:0/592: link d9/db/dc/d3f/f2d d9/d24/d92/d9d/dc6/fd9 0 2022-01-31T19:44:27.759 INFO:tasks.workunit.client.1.smithi171.stdout:4/429: getdents d2/d3/d5/d91/d7e/d12/d15/d57 0 2022-01-31T19:44:27.760 INFO:tasks.workunit.client.1.smithi171.stdout:4/430: write d2/d3/d5/d91/d7e/f62 [1412068,127534] 0 2022-01-31T19:44:27.760 INFO:tasks.workunit.client.1.smithi171.stdout:4/431: fdatasync d2/d3/f25 0 2022-01-31T19:44:27.762 INFO:tasks.workunit.client.1.smithi171.stdout:1/471: dwrite f9 [0,4194304] 0 2022-01-31T19:44:27.762 INFO:tasks.workunit.client.1.smithi171.stdout:3/572: dread d4/d21/d3a/f4f [0,4194304] 0 2022-01-31T19:44:27.762 INFO:tasks.workunit.client.1.smithi171.stdout:3/573: write d4/dd/d27/d28/d35/d55/d8f/d92/dab/f84 [765097,20861] 0 2022-01-31T19:44:27.762 INFO:tasks.workunit.client.1.smithi171.stdout:3/574: dread - d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f96 zero size 2022-01-31T19:44:27.763 INFO:tasks.workunit.client.1.smithi171.stdout:2/665: rename d7/dd/f7e to d7/dd/d93/dcb/fda 0 2022-01-31T19:44:27.764 INFO:tasks.workunit.client.1.smithi171.stdout:6/494: symlink dc/d19/d29/d27/l9a 0 2022-01-31T19:44:27.764 INFO:tasks.workunit.client.1.smithi171.stdout:6/495: chown dc/d18/d53/f5b 506 1 2022-01-31T19:44:27.764 INFO:tasks.workunit.client.1.smithi171.stdout:6/496: write dc/d8b/f8f [226031,80208] 0 2022-01-31T19:44:27.765 INFO:tasks.workunit.client.1.smithi171.stdout:6/497: creat dc/d19/d30/f9b x:0 0 0 2022-01-31T19:44:27.765 INFO:tasks.workunit.client.1.smithi171.stdout:6/498: stat dc/d21/d4f/f60 0 2022-01-31T19:44:27.765 INFO:tasks.workunit.client.1.smithi171.stdout:6/499: truncate dc/d1e/f45 944177 0 2022-01-31T19:44:27.767 INFO:tasks.workunit.client.1.smithi171.stdout:0/593: symlink d9/d24/d4c/d35/d67/lda 0 2022-01-31T19:44:27.767 INFO:tasks.workunit.client.1.smithi171.stdout:0/594: readlink d9/d24/d92/lb4 0 2022-01-31T19:44:27.770 INFO:tasks.workunit.client.1.smithi171.stdout:1/472: creat dd/d23/d2e/d5e/d81/f93 x:0 0 0 2022-01-31T19:44:27.770 INFO:tasks.workunit.client.1.smithi171.stdout:1/473: readlink dd/df/d84/l8c 0 2022-01-31T19:44:27.771 INFO:tasks.workunit.client.1.smithi171.stdout:2/666: creat d7/d26/d3d/fdb x:0 0 0 2022-01-31T19:44:27.772 INFO:tasks.workunit.client.1.smithi171.stdout:2/667: write d7/d7c/fd3 [744143,11039] 0 2022-01-31T19:44:27.772 INFO:tasks.workunit.client.1.smithi171.stdout:2/668: write d7/d26/fa7 [1485198,86529] 0 2022-01-31T19:44:27.772 INFO:tasks.workunit.client.1.smithi171.stdout:2/669: write d7/d9/f74 [1923119,94700] 0 2022-01-31T19:44:27.772 INFO:tasks.workunit.client.1.smithi171.stdout:2/670: getdents d7/d9 0 2022-01-31T19:44:27.773 INFO:tasks.workunit.client.1.smithi171.stdout:0/595: truncate f8 3751652 0 2022-01-31T19:44:27.773 INFO:tasks.workunit.client.1.smithi171.stdout:4/432: truncate d2/d3/f7 2535895 0 2022-01-31T19:44:27.774 INFO:tasks.workunit.client.1.smithi171.stdout:2/671: write d7/fc8 [2162266,31446] 0 2022-01-31T19:44:27.774 INFO:tasks.workunit.client.1.smithi171.stdout:2/672: fsync d7/dd/d5e/d6b/d81/fad 0 2022-01-31T19:44:27.774 INFO:tasks.workunit.client.1.smithi171.stdout:2/673: chown d7/d26/d63/f7f 28780038 1 2022-01-31T19:44:27.775 INFO:tasks.workunit.client.1.smithi171.stdout:0/596: creat d9/db/dc/d3f/db9/d7a/fdb x:0 0 0 2022-01-31T19:44:27.776 INFO:tasks.workunit.client.1.smithi171.stdout:6/500: dread dc/d19/d29/f5c [0,4194304] 0 2022-01-31T19:44:27.777 INFO:tasks.workunit.client.1.smithi171.stdout:4/433: link d2/d3/d5/d91/d7e/d12/d15/l4e d2/d3/d5/dc/d4b/l94 0 2022-01-31T19:44:27.777 INFO:tasks.workunit.client.1.smithi171.stdout:4/434: fsync d2/d3/d5/d41/f89 0 2022-01-31T19:44:27.777 INFO:tasks.workunit.client.1.smithi171.stdout:4/435: chown d2/d3/d5/d41/f89 479 1 2022-01-31T19:44:27.778 INFO:tasks.workunit.client.1.smithi171.stdout:1/474: getdents dd/df/d19/d1b/d77/d80 0 2022-01-31T19:44:27.778 INFO:tasks.workunit.client.1.smithi171.stdout:1/475: write dd/d23/d34/f65 [2002039,104369] 0 2022-01-31T19:44:27.780 INFO:tasks.workunit.client.1.smithi171.stdout:0/597: unlink d9/db/f14 0 2022-01-31T19:44:27.781 INFO:tasks.workunit.client.1.smithi171.stdout:2/674: dread d7/fa [0,4194304] 0 2022-01-31T19:44:27.784 INFO:tasks.workunit.client.1.smithi171.stdout:6/501: mknod dc/d19/d29/d27/d37/d3b/c9c 0 2022-01-31T19:44:27.789 INFO:tasks.workunit.client.1.smithi171.stdout:3/575: dwrite d4/dd/d87/d8e/f42 [0,4194304] 0 2022-01-31T19:44:27.790 INFO:tasks.workunit.client.1.smithi171.stdout:6/502: write dc/d18/f39 [3730507,81247] 0 2022-01-31T19:44:27.793 INFO:tasks.workunit.client.1.smithi171.stdout:6/503: rename dc/d19/d29/d6a/f76 to dc/d18/f9d 0 2022-01-31T19:44:27.793 INFO:tasks.workunit.client.1.smithi171.stdout:6/504: chown c3 10522 1 2022-01-31T19:44:27.796 INFO:tasks.workunit.client.1.smithi171.stdout:3/576: write d4/dd/d27/d28/d35/f5c [1743779,106501] 0 2022-01-31T19:44:27.798 INFO:tasks.workunit.client.1.smithi171.stdout:3/577: rmdir d4/dd/d27/d28/d35/d55 39 2022-01-31T19:44:27.799 INFO:tasks.workunit.client.1.smithi171.stdout:3/578: symlink d4/dd/d27/lb1 0 2022-01-31T19:44:27.800 INFO:tasks.workunit.client.1.smithi171.stdout:3/579: mkdir d4/dd/d39/d51/db2 0 2022-01-31T19:44:27.801 INFO:tasks.workunit.client.1.smithi171.stdout:3/580: symlink d4/dd/d39/d51/lb3 0 2022-01-31T19:44:27.806 INFO:tasks.workunit.client.1.smithi171.stdout:6/505: write dc/d19/d29/f63 [961888,47961] 0 2022-01-31T19:44:27.806 INFO:tasks.workunit.client.1.smithi171.stdout:6/506: creat dc/d18/d86/f9e x:0 0 0 2022-01-31T19:44:27.810 INFO:tasks.workunit.client.1.smithi171.stdout:6/507: unlink dc/d18/d53/c70 0 2022-01-31T19:44:27.836 INFO:tasks.workunit.client.1.smithi171.stdout:1/476: dwrite dd/df/d19/d1b/d1e/d64/f58 [4194304,4194304] 0 2022-01-31T19:44:27.837 INFO:tasks.workunit.client.1.smithi171.stdout:1/477: creat dd/d23/d74/d7b/f94 x:0 0 0 2022-01-31T19:44:27.837 INFO:tasks.workunit.client.1.smithi171.stdout:1/478: write dd/f7a [614433,41931] 0 2022-01-31T19:44:27.837 INFO:tasks.workunit.client.1.smithi171.stdout:1/479: getdents dd/df/d19/d36 0 2022-01-31T19:44:27.839 INFO:tasks.workunit.client.1.smithi171.stdout:1/480: unlink dd/df/d19/d1b/f66 0 2022-01-31T19:44:27.840 INFO:tasks.workunit.client.1.smithi171.stdout:1/481: creat dd/df/d19/d36/f95 x:0 0 0 2022-01-31T19:44:27.840 INFO:tasks.workunit.client.1.smithi171.stdout:4/436: dwrite d2/f90 [0,4194304] 0 2022-01-31T19:44:27.843 INFO:tasks.workunit.client.1.smithi171.stdout:2/675: dwrite d7/d7c/fc1 [0,4194304] 0 2022-01-31T19:44:27.844 INFO:tasks.workunit.client.1.smithi171.stdout:3/581: dwrite d4/dd/d27/d28/d35/d55/f74 [0,4194304] 0 2022-01-31T19:44:27.845 INFO:tasks.workunit.client.1.smithi171.stdout:6/508: dwrite dc/d19/d29/d27/f33 [0,4194304] 0 2022-01-31T19:44:27.845 INFO:tasks.workunit.client.1.smithi171.stdout:0/598: dwrite d9/db/dc/d3f/f2d [0,4194304] 0 2022-01-31T19:44:27.850 INFO:tasks.workunit.client.1.smithi171.stdout:3/582: creat d4/dd/d39/d51/fb4 x:0 0 0 2022-01-31T19:44:27.850 INFO:tasks.workunit.client.1.smithi171.stdout:3/583: truncate d4/dd/d39/d51/fb4 405645 0 2022-01-31T19:44:27.855 INFO:tasks.workunit.client.1.smithi171.stdout:6/509: creat dc/d19/d29/d27/d37/d3b/d67/f9f x:0 0 0 2022-01-31T19:44:27.855 INFO:tasks.workunit.client.1.smithi171.stdout:6/510: chown dc/f7f 0 1 2022-01-31T19:44:27.857 INFO:tasks.workunit.client.1.smithi171.stdout:2/676: rmdir d7/d7c 39 2022-01-31T19:44:27.858 INFO:tasks.workunit.client.1.smithi171.stdout:2/677: dread d7/d26/d63/f67 [0,4194304] 0 2022-01-31T19:44:27.859 INFO:tasks.workunit.client.1.smithi171.stdout:3/584: mkdir d4/d21/d3a/db5 0 2022-01-31T19:44:27.860 INFO:tasks.workunit.client.1.smithi171.stdout:6/511: rename dc/d18/d53/f5b to dc/d19/d29/d6a/fa0 0 2022-01-31T19:44:27.861 INFO:tasks.workunit.client.1.smithi171.stdout:6/512: chown dc/d19/d29/l74 90385803 1 2022-01-31T19:44:27.863 INFO:tasks.workunit.client.1.smithi171.stdout:2/678: mkdir d7/dd/d93/ddc 0 2022-01-31T19:44:27.863 INFO:tasks.workunit.client.1.smithi171.stdout:6/513: symlink dc/d19/d29/d27/d37/d3b/d67/la1 0 2022-01-31T19:44:27.864 INFO:tasks.workunit.client.1.smithi171.stdout:6/514: fdatasync dc/d19/d29/d27/d37/d68/f6d 0 2022-01-31T19:44:27.864 INFO:tasks.workunit.client.1.smithi171.stdout:6/515: creat dc/d21/fa2 x:0 0 0 2022-01-31T19:44:27.865 INFO:tasks.workunit.client.1.smithi171.stdout:6/516: creat dc/d19/d30/fa3 x:0 0 0 2022-01-31T19:44:27.867 INFO:tasks.workunit.client.1.smithi171.stdout:6/517: creat dc/d1e/fa4 x:0 0 0 2022-01-31T19:44:27.868 INFO:tasks.workunit.client.1.smithi171.stdout:6/518: symlink dc/d18/la5 0 2022-01-31T19:44:27.868 INFO:tasks.workunit.client.1.smithi171.stdout:6/519: read - dc/d1e/d3f/f97 zero size 2022-01-31T19:44:27.868 INFO:tasks.workunit.client.1.smithi171.stdout:6/520: symlink dc/d18/d53/la6 0 2022-01-31T19:44:27.869 INFO:tasks.workunit.client.1.smithi171.stdout:6/521: fdatasync dc/d19/d30/f3d 0 2022-01-31T19:44:27.869 INFO:tasks.workunit.client.1.smithi171.stdout:2/679: write d7/dd/f1e [3664503,12910] 0 2022-01-31T19:44:27.869 INFO:tasks.workunit.client.1.smithi171.stdout:2/680: chown d7/l33 22204712 1 2022-01-31T19:44:27.870 INFO:tasks.workunit.client.1.smithi171.stdout:2/681: chown d7/f75 339 1 2022-01-31T19:44:27.870 INFO:tasks.workunit.client.1.smithi171.stdout:2/682: readlink d7/d55/l60 0 2022-01-31T19:44:27.870 INFO:tasks.workunit.client.1.smithi171.stdout:2/683: write d7/dd/d5e/d6b/d81/fa6 [260839,57561] 0 2022-01-31T19:44:27.885 INFO:tasks.workunit.client.1.smithi171.stdout:4/437: dwrite d2/d3/d5/d91/d7e/f83 [8388608,4194304] 0 2022-01-31T19:44:27.886 INFO:tasks.workunit.client.1.smithi171.stdout:4/438: chown d2/d3/d5/dc/d4b/d56/c69 133837 1 2022-01-31T19:44:27.889 INFO:tasks.workunit.client.1.smithi171.stdout:4/439: write d2/d3/f73 [5476036,71606] 0 2022-01-31T19:44:27.890 INFO:tasks.workunit.client.1.smithi171.stdout:4/440: rename d2/d3/d5/dc/f80 to d2/d3/d5/dc/f95 0 2022-01-31T19:44:27.891 INFO:tasks.workunit.client.1.smithi171.stdout:4/441: creat d2/d3/d5/d70/d7c/f96 x:0 0 0 2022-01-31T19:44:27.892 INFO:tasks.workunit.client.1.smithi171.stdout:4/442: mknod d2/d3/d5/d70/c97 0 2022-01-31T19:44:27.894 INFO:tasks.workunit.client.1.smithi171.stdout:4/443: rename d2/c29 to d2/d3/d5/d70/d7b/c98 0 2022-01-31T19:44:27.894 INFO:tasks.workunit.client.1.smithi171.stdout:4/444: stat d2/d3/d5/d91/d7e/d12/f1d 0 2022-01-31T19:44:27.895 INFO:tasks.workunit.client.1.smithi171.stdout:4/445: link d2/f14 d2/d3/d5/d41/d6b/f99 0 2022-01-31T19:44:27.896 INFO:tasks.workunit.client.1.smithi171.stdout:4/446: dread - d2/d3/d5/d41/f47 zero size 2022-01-31T19:44:27.898 INFO:tasks.workunit.client.1.smithi171.stdout:1/482: dwrite dd/df/d19/d1b/f3e [4194304,4194304] 0 2022-01-31T19:44:27.898 INFO:tasks.workunit.client.1.smithi171.stdout:0/599: dwrite d9/db/dc/d3f/f2d [0,4194304] 0 2022-01-31T19:44:27.899 INFO:tasks.workunit.client.1.smithi171.stdout:1/483: creat dd/df/d19/f96 x:0 0 0 2022-01-31T19:44:27.899 INFO:tasks.workunit.client.1.smithi171.stdout:4/447: mknod d2/d3/d5/dc/d4b/d56/c9a 0 2022-01-31T19:44:27.899 INFO:tasks.workunit.client.1.smithi171.stdout:2/684: dwrite d7/dd/f35 [0,4194304] 0 2022-01-31T19:44:27.900 INFO:tasks.workunit.client.1.smithi171.stdout:2/685: write d7/dd/f3c [1286268,63991] 0 2022-01-31T19:44:27.904 INFO:tasks.workunit.client.1.smithi171.stdout:1/484: mkdir dd/df/d19/d1b/d1e/d97 0 2022-01-31T19:44:27.909 INFO:tasks.workunit.client.1.smithi171.stdout:6/522: rmdir dc/d19 39 2022-01-31T19:44:27.911 INFO:tasks.workunit.client.1.smithi171.stdout:4/448: symlink d2/d3/d5/dc/l9b 0 2022-01-31T19:44:27.918 INFO:tasks.workunit.client.1.smithi171.stdout:0/600: dwrite d9/db/dc2/f29 [0,4194304] 0 2022-01-31T19:44:27.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:27 smithi167 conmon[32206]: cluster 2022-01-31T19:44:26.435628+0000 mgr.smithi171.asyxnc (mgr.24387) 34 : cluster 2022-01-31T19:44:27.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:27 smithi167 conmon[32206]: [DBG] pgmap v22: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 122 MiB/s rd, 241 MiB/s wr, 360 op/s 2022-01-31T19:44:27.924 INFO:tasks.workunit.client.1.smithi171.stdout:2/686: symlink d7/dd/d93/ddc/ldd 0 2022-01-31T19:44:27.925 INFO:tasks.workunit.client.1.smithi171.stdout:1/485: getdents dd/df/d19/d1b/d1e 0 2022-01-31T19:44:27.927 INFO:tasks.workunit.client.1.smithi171.stdout:2/687: creat d7/d7c/d83/d85/db9/fde x:0 0 0 2022-01-31T19:44:27.928 INFO:tasks.workunit.client.1.smithi171.stdout:0/601: getdents d9/db/dc2 0 2022-01-31T19:44:27.932 INFO:tasks.workunit.client.1.smithi171.stdout:2/688: symlink d7/d26/d63/d71/ldf 0 2022-01-31T19:44:27.933 INFO:tasks.workunit.client.1.smithi171.stdout:6/523: rename dc/d18/d53 to dc/d19/d29/da7 0 2022-01-31T19:44:27.933 INFO:tasks.workunit.client.1.smithi171.stdout:2/689: creat d7/d7c/d83/fe0 x:0 0 0 2022-01-31T19:44:27.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/486: rename dd/df/d19/d1b/d77 to dd/d23/d2e/d5e/d81/d98 0 2022-01-31T19:44:27.936 INFO:tasks.workunit.client.1.smithi171.stdout:6/524: truncate f9 5176724 0 2022-01-31T19:44:27.937 INFO:tasks.workunit.client.1.smithi171.stdout:6/525: mknod dc/d1e/d3f/ca8 0 2022-01-31T19:44:27.941 INFO:tasks.workunit.client.1.smithi171.stdout:6/526: mknod dc/d19/d29/d27/d37/ca9 0 2022-01-31T19:44:27.942 INFO:tasks.workunit.client.1.smithi171.stdout:6/527: chown dc 939828 1 2022-01-31T19:44:27.944 INFO:tasks.workunit.client.1.smithi171.stdout:6/528: rename dc/f13 to dc/d21/faa 0 2022-01-31T19:44:27.944 INFO:tasks.workunit.client.1.smithi171.stdout:6/529: write dc/d8b/f96 [1017956,13500] 0 2022-01-31T19:44:27.945 INFO:tasks.workunit.client.1.smithi171.stdout:6/530: write dc/f17 [1264123,51132] 0 2022-01-31T19:44:27.946 INFO:tasks.workunit.client.1.smithi171.stdout:6/531: rename dc/d18/d86/f9e to dc/d21/fab 0 2022-01-31T19:44:27.947 INFO:tasks.workunit.client.1.smithi171.stdout:6/532: chown dc/d19/d29/d27/d37/l48 213314 1 2022-01-31T19:44:27.949 INFO:tasks.workunit.client.1.smithi171.stdout:6/533: mknod dc/d19/cac 0 2022-01-31T19:44:27.949 INFO:tasks.workunit.client.1.smithi171.stdout:6/534: write dc/d21/f8d [891776,37692] 0 2022-01-31T19:44:27.949 INFO:tasks.workunit.client.1.smithi171.stdout:6/535: mknod dc/d19/d29/d27/d37/d3b/cad 0 2022-01-31T19:44:27.972 INFO:tasks.workunit.client.1.smithi171.stdout:2/690: dwrite d7/dd/d93/fb1 [0,4194304] 0 2022-01-31T19:44:27.973 INFO:tasks.workunit.client.1.smithi171.stdout:2/691: link d7/dd/d5e/d6b/d81/d8d/da2/ld8 d7/d9/d1a/d45/da4/le1 0 2022-01-31T19:44:27.977 INFO:tasks.workunit.client.1.smithi171.stdout:1/487: dwrite dd/d23/d2e/d5e/d81/d98/d80/f8e [0,4194304] 0 2022-01-31T19:44:27.977 INFO:tasks.workunit.client.1.smithi171.stdout:1/488: dread - dd/df/d19/d1b/f72 zero size 2022-01-31T19:44:27.978 INFO:tasks.workunit.client.1.smithi171.stdout:6/536: dwrite dc/f17 [0,4194304] 0 2022-01-31T19:44:27.983 INFO:tasks.workunit.client.1.smithi171.stdout:2/692: dread d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:28.012 INFO:tasks.workunit.client.1.smithi171.stdout:2/693: dwrite d7/f56 [0,4194304] 0 2022-01-31T19:44:28.013 INFO:tasks.workunit.client.1.smithi171.stdout:6/537: dwrite dc/d19/d29/f85 [0,4194304] 0 2022-01-31T19:44:28.023 INFO:tasks.workunit.client.1.smithi171.stdout:5/499: sync 2022-01-31T19:44:28.024 INFO:tasks.workunit.client.1.smithi171.stdout:7/398: sync 2022-01-31T19:44:28.024 INFO:tasks.workunit.client.1.smithi171.stdout:9/606: sync 2022-01-31T19:44:28.026 INFO:tasks.workunit.client.1.smithi171.stdout:1/489: link dd/df/d19/d1b/f3e dd/d23/d74/d7b/f99 0 2022-01-31T19:44:28.027 INFO:tasks.workunit.client.1.smithi171.stdout:1/490: fdatasync dd/d23/f60 0 2022-01-31T19:44:28.027 INFO:tasks.workunit.client.1.smithi171.stdout:7/399: dread d1/d1e/d2a/d29/f6d [0,4194304] 0 2022-01-31T19:44:28.027 INFO:tasks.workunit.client.1.smithi171.stdout:7/400: read - d1/d1e/d2a/d29/f66 zero size 2022-01-31T19:44:28.028 INFO:tasks.workunit.client.1.smithi171.stdout:7/401: chown d1/d26/f5c 1 1 2022-01-31T19:44:28.029 INFO:tasks.workunit.client.1.smithi171.stdout:2/694: rmdir d7/d26/d63/d71 39 2022-01-31T19:44:28.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/538: stat l7 0 2022-01-31T19:44:28.031 INFO:tasks.workunit.client.1.smithi171.stdout:5/500: truncate d5/d6/d15/d23/f87 2222554 0 2022-01-31T19:44:28.031 INFO:tasks.workunit.client.1.smithi171.stdout:5/501: chown d5/d6/dd/f35 1225498409 1 2022-01-31T19:44:28.032 INFO:tasks.workunit.client.1.smithi171.stdout:6/539: write fa [3569146,90121] 0 2022-01-31T19:44:28.032 INFO:tasks.workunit.client.1.smithi171.stdout:9/607: link dd/df/d15/d53/c58 dd/df/d15/d1a/cc8 0 2022-01-31T19:44:28.033 INFO:tasks.workunit.client.1.smithi171.stdout:1/491: rmdir dd/d23/d34 39 2022-01-31T19:44:28.034 INFO:tasks.workunit.client.1.smithi171.stdout:1/492: dread dd/df/d19/f32 [0,4194304] 0 2022-01-31T19:44:28.035 INFO:tasks.workunit.client.1.smithi171.stdout:1/493: fdatasync dd/df/d19/d1b/d1e/d64/f58 0 2022-01-31T19:44:28.035 INFO:tasks.workunit.client.1.smithi171.stdout:1/494: readlink dd/df/l6c 0 2022-01-31T19:44:28.036 INFO:tasks.workunit.client.1.smithi171.stdout:7/402: link d1/d26/d47/f4f d1/f87 0 2022-01-31T19:44:28.036 INFO:tasks.workunit.client.1.smithi171.stdout:7/403: dread - d1/d1e/d2a/d29/d31/d30/d3c/f76 zero size 2022-01-31T19:44:28.036 INFO:tasks.workunit.client.1.smithi171.stdout:7/404: creat d1/d1e/d2a/d29/d31/d30/f88 x:0 0 0 2022-01-31T19:44:28.038 INFO:tasks.workunit.client.1.smithi171.stdout:2/695: mkdir d7/d9/d1a/de2 0 2022-01-31T19:44:28.038 INFO:tasks.workunit.client.1.smithi171.stdout:2/696: chown d7/dd/d5e/d84/l90 2006314 1 2022-01-31T19:44:28.039 INFO:tasks.workunit.client.1.smithi171.stdout:5/502: link d5/d6/d15/d80/l84 d5/d6/d15/d23/d6b/lad 0 2022-01-31T19:44:28.040 INFO:tasks.workunit.client.1.smithi171.stdout:5/503: write d5/d6/d15/d23/d67/f5c [3449396,97963] 0 2022-01-31T19:44:28.040 INFO:tasks.workunit.client.1.smithi171.stdout:6/540: mkdir dc/d19/dae 0 2022-01-31T19:44:28.041 INFO:tasks.workunit.client.1.smithi171.stdout:9/608: rename dd/df/d3f/d73/c7f to dd/df/cc9 0 2022-01-31T19:44:28.043 INFO:tasks.workunit.client.1.smithi171.stdout:2/697: creat d7/dd/d93/dc0/fe3 x:0 0 0 2022-01-31T19:44:28.044 INFO:tasks.workunit.client.1.smithi171.stdout:5/504: mkdir d5/d6/d15/d23/d6b/dae 0 2022-01-31T19:44:28.045 INFO:tasks.workunit.client.1.smithi171.stdout:5/505: write d5/d6/d15/d23/f87 [2621750,52682] 0 2022-01-31T19:44:28.046 INFO:tasks.workunit.client.1.smithi171.stdout:6/541: mkdir dc/d19/d29/daf 0 2022-01-31T19:44:28.048 INFO:tasks.workunit.client.1.smithi171.stdout:1/495: rename dd/d23/d2e/d5e/c18 to dd/df/d19/c9a 0 2022-01-31T19:44:28.050 INFO:tasks.workunit.client.1.smithi171.stdout:9/609: mkdir dd/df/d15/d1a/d84/dca 0 2022-01-31T19:44:28.052 INFO:tasks.workunit.client.1.smithi171.stdout:6/542: dread dc/d21/f5a [0,4194304] 0 2022-01-31T19:44:28.052 INFO:tasks.workunit.client.1.smithi171.stdout:6/543: fsync dc/d21/f2e 0 2022-01-31T19:44:28.053 INFO:tasks.workunit.client.1.smithi171.stdout:6/544: read - dc/d19/d30/fa3 zero size 2022-01-31T19:44:28.054 INFO:tasks.workunit.client.1.smithi171.stdout:5/506: unlink d5/d6/d15/d23/d67/f48 0 2022-01-31T19:44:28.054 INFO:tasks.workunit.client.1.smithi171.stdout:9/610: mkdir dd/df/d15/d1a/dcb 0 2022-01-31T19:44:28.054 INFO:tasks.workunit.client.1.smithi171.stdout:7/405: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [4194304,4194304] 0 2022-01-31T19:44:28.055 INFO:tasks.workunit.client.1.smithi171.stdout:9/611: write dd/df/d3f/f51 [908624,3048] 0 2022-01-31T19:44:28.061 INFO:tasks.workunit.client.1.smithi171.stdout:7/406: creat d1/d26/d57/f89 x:0 0 0 2022-01-31T19:44:28.061 INFO:tasks.workunit.client.1.smithi171.stdout:7/407: readlink d1/d26/d47/l53 0 2022-01-31T19:44:28.069 INFO:tasks.workunit.client.1.smithi171.stdout:9/612: dread dd/df/f62 [0,4194304] 0 2022-01-31T19:44:28.069 INFO:tasks.workunit.client.1.smithi171.stdout:7/408: rmdir d1/d1e/d2a/d29/d31/d30 39 2022-01-31T19:44:28.070 INFO:tasks.workunit.client.1.smithi171.stdout:7/409: mknod d1/d1e/d2a/d29/d31/d30/c8a 0 2022-01-31T19:44:28.071 INFO:tasks.workunit.client.1.smithi171.stdout:2/698: dwrite d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:28.071 INFO:tasks.workunit.client.1.smithi171.stdout:2/699: dread - d7/dd/d93/f9e zero size 2022-01-31T19:44:28.072 INFO:tasks.workunit.client.1.smithi171.stdout:7/410: creat d1/d1e/d2a/d29/d31/d30/d3c/d7e/f8b x:0 0 0 2022-01-31T19:44:28.072 INFO:tasks.workunit.client.1.smithi171.stdout:7/411: stat d1/d1b/f22 0 2022-01-31T19:44:28.073 INFO:tasks.workunit.client.1.smithi171.stdout:6/545: dwrite dc/d18/f9d [0,4194304] 0 2022-01-31T19:44:28.073 INFO:tasks.workunit.client.1.smithi171.stdout:6/546: readlink dc/d19/d29/d27/d37/d3b/d67/la1 0 2022-01-31T19:44:28.074 INFO:tasks.workunit.client.1.smithi171.stdout:2/700: dread d7/d7c/f87 [0,4194304] 0 2022-01-31T19:44:28.074 INFO:tasks.workunit.client.1.smithi171.stdout:2/701: creat d7/dd/d5e/d6b/d81/fe4 x:0 0 0 2022-01-31T19:44:28.074 INFO:tasks.workunit.client.1.smithi171.stdout:7/412: creat d1/d1e/d2a/d29/d77/f8c x:0 0 0 2022-01-31T19:44:28.081 INFO:tasks.workunit.client.1.smithi171.stdout:6/547: dread dc/d18/f22 [0,4194304] 0 2022-01-31T19:44:28.081 INFO:tasks.workunit.client.1.smithi171.stdout:7/413: creat d1/d1e/d2a/d29/d31/d30/d3c/d79/f8d x:0 0 0 2022-01-31T19:44:28.083 INFO:tasks.workunit.client.1.smithi171.stdout:6/548: getdents dc/d19/d29/d27/d37 0 2022-01-31T19:44:28.084 INFO:tasks.workunit.client.1.smithi171.stdout:2/702: rename d7/dd/d93 to d7/d26/de5 0 2022-01-31T19:44:28.085 INFO:tasks.workunit.client.1.smithi171.stdout:5/507: dwrite d5/d6/d15/d23/f87 [0,4194304] 0 2022-01-31T19:44:28.086 INFO:tasks.workunit.client.1.smithi171.stdout:6/549: symlink dc/d19/dae/lb0 0 2022-01-31T19:44:28.086 INFO:tasks.workunit.client.1.smithi171.stdout:2/703: creat d7/dd/fe6 x:0 0 0 2022-01-31T19:44:28.086 INFO:tasks.workunit.client.1.smithi171.stdout:7/414: dread d1/d52/f21 [0,4194304] 0 2022-01-31T19:44:28.088 INFO:tasks.workunit.client.1.smithi171.stdout:8/482: sync 2022-01-31T19:44:28.089 INFO:tasks.workunit.client.1.smithi171.stdout:8/483: read df/f83 [2490786,34371] 0 2022-01-31T19:44:28.091 INFO:tasks.workunit.client.1.smithi171.stdout:7/415: mkdir d1/d52/d8e 0 2022-01-31T19:44:28.092 INFO:tasks.workunit.client.1.smithi171.stdout:7/416: write d1/d1e/d2a/d29/d31/d30/d3c/f56 [220594,116828] 0 2022-01-31T19:44:28.093 INFO:tasks.workunit.client.1.smithi171.stdout:6/550: symlink dc/d19/d30/d3c/lb1 0 2022-01-31T19:44:28.094 INFO:tasks.workunit.client.1.smithi171.stdout:8/484: link df/d2a/d30/l96 df/d18/la7 0 2022-01-31T19:44:28.095 INFO:tasks.workunit.client.1.smithi171.stdout:2/704: rmdir d7/d7c/d83 39 2022-01-31T19:44:28.096 INFO:tasks.workunit.client.1.smithi171.stdout:2/705: dread - d7/d7c/d83/d85/db9/fde zero size 2022-01-31T19:44:28.096 INFO:tasks.workunit.client.1.smithi171.stdout:7/417: write d1/d1e/d2a/d29/d31/d30/f3f [2340958,65119] 0 2022-01-31T19:44:28.097 INFO:tasks.workunit.client.1.smithi171.stdout:8/485: chown l4 0 1 2022-01-31T19:44:28.097 INFO:tasks.workunit.client.1.smithi171.stdout:2/706: read d7/f62 [4163644,123860] 0 2022-01-31T19:44:28.099 INFO:tasks.workunit.client.1.smithi171.stdout:8/486: mkdir df/d18/d79/da2/da8 0 2022-01-31T19:44:28.100 INFO:tasks.workunit.client.1.smithi171.stdout:8/487: stat f0 0 2022-01-31T19:44:28.100 INFO:tasks.workunit.client.1.smithi171.stdout:8/488: chown df/d18/d79/d7a/l9f 49449927 1 2022-01-31T19:44:28.100 INFO:tasks.workunit.client.1.smithi171.stdout:2/707: symlink d7/d9/d1a/de2/le7 0 2022-01-31T19:44:28.103 INFO:tasks.workunit.client.1.smithi171.stdout:5/508: dwrite d5/d6/d15/d5d/d6f/f9c [0,4194304] 0 2022-01-31T19:44:28.103 INFO:tasks.workunit.client.1.smithi171.stdout:5/509: chown d5/d6/d15/d23/d67/l2f 1847 1 2022-01-31T19:44:28.103 INFO:tasks.workunit.client.1.smithi171.stdout:8/489: symlink df/d18/d19/d21/d67/la9 0 2022-01-31T19:44:28.103 INFO:tasks.workunit.client.1.smithi171.stdout:2/708: creat d7/d7c/fe8 x:0 0 0 2022-01-31T19:44:28.104 INFO:tasks.workunit.client.1.smithi171.stdout:2/709: chown d7/d26/fc4 4023 1 2022-01-31T19:44:28.104 INFO:tasks.workunit.client.1.smithi171.stdout:8/490: write df/d18/d19/d36/f6d [219704,105707] 0 2022-01-31T19:44:28.104 INFO:tasks.workunit.client.1.smithi171.stdout:8/491: fdatasync fc 0 2022-01-31T19:44:28.105 INFO:tasks.workunit.client.1.smithi171.stdout:6/551: dread dc/d18/f39 [0,4194304] 0 2022-01-31T19:44:28.105 INFO:tasks.workunit.client.1.smithi171.stdout:6/552: chown dc/d21/d4f/l8c 1 1 2022-01-31T19:44:28.105 INFO:tasks.workunit.client.1.smithi171.stdout:6/553: write dc/f4a [1554938,97780] 0 2022-01-31T19:44:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:27 smithi171 conmon[35325]: cluster 2022-01-31T19:44:26.435628+0000 mgr.smithi171.asyxnc (mgr.24387) 2022-01-31T19:44:28.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:27 smithi171 conmon[35325]: 34 : cluster [DBG] pgmap v22: 65 pgs: 65 active+clean; 2.7 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 122 MiB/s rd, 241 MiB/s wr, 360 op/s 2022-01-31T19:44:28.109 INFO:tasks.workunit.client.1.smithi171.stdout:5/510: rmdir d5/d6/d15/d80 39 2022-01-31T19:44:28.110 INFO:tasks.workunit.client.1.smithi171.stdout:6/554: symlink dc/d19/d29/d27/d37/d68/lb2 0 2022-01-31T19:44:28.112 INFO:tasks.workunit.client.1.smithi171.stdout:5/511: symlink d5/d6/d15/d23/d6b/laf 0 2022-01-31T19:44:28.112 INFO:tasks.workunit.client.1.smithi171.stdout:5/512: chown d5/d6/d15/d23/l8b 7238405 1 2022-01-31T19:44:28.113 INFO:tasks.workunit.client.1.smithi171.stdout:6/555: rmdir dc/d19/d29/d27 39 2022-01-31T19:44:28.113 INFO:tasks.workunit.client.1.smithi171.stdout:2/710: dread d7/fc8 [0,4194304] 0 2022-01-31T19:44:28.114 INFO:tasks.workunit.client.1.smithi171.stdout:6/556: getdents dc/d19/d30 0 2022-01-31T19:44:28.114 INFO:tasks.workunit.client.1.smithi171.stdout:6/557: creat dc/d21/d4f/fb3 x:0 0 0 2022-01-31T19:44:28.115 INFO:tasks.workunit.client.1.smithi171.stdout:2/711: symlink d7/d7c/d83/d85/le9 0 2022-01-31T19:44:28.116 INFO:tasks.workunit.client.1.smithi171.stdout:2/712: getdents d7/d26/d3d/d40/dd2 0 2022-01-31T19:44:28.116 INFO:tasks.workunit.client.1.smithi171.stdout:2/713: chown d7/d7c/fb3 7317824 1 2022-01-31T19:44:28.117 INFO:tasks.workunit.client.1.smithi171.stdout:6/558: creat dc/d19/d29/d6a/fb4 x:0 0 0 2022-01-31T19:44:28.117 INFO:tasks.workunit.client.1.smithi171.stdout:6/559: write dc/d19/f1a [1278203,104071] 0 2022-01-31T19:44:28.127 INFO:tasks.workunit.client.1.smithi171.stdout:7/418: dwrite d1/d26/f71 [0,4194304] 0 2022-01-31T19:44:28.128 INFO:tasks.workunit.client.1.smithi171.stdout:5/513: dwrite d5/f4f [0,4194304] 0 2022-01-31T19:44:28.128 INFO:tasks.workunit.client.1.smithi171.stdout:8/492: dwrite df/d27/d85/f8e [0,4194304] 0 2022-01-31T19:44:28.129 INFO:tasks.workunit.client.1.smithi171.stdout:6/560: dread dc/d19/d29/f63 [0,4194304] 0 2022-01-31T19:44:28.131 INFO:tasks.workunit.client.1.smithi171.stdout:7/419: write d1/d1e/d2a/d29/d31/f6a [2584000,27268] 0 2022-01-31T19:44:28.138 INFO:tasks.workunit.client.1.smithi171.stdout:2/714: rename d7/d9/d1a/de2 to d7/dd/dea 0 2022-01-31T19:44:28.138 INFO:tasks.workunit.client.1.smithi171.stdout:2/715: dread - d7/dd/d5e/d6b/d81/fad zero size 2022-01-31T19:44:28.148 INFO:tasks.workunit.client.0.smithi167.stdout:0/927: sync 2022-01-31T19:44:28.149 INFO:tasks.workunit.client.0.smithi167.stdout:6/925: sync 2022-01-31T19:44:28.163 INFO:tasks.workunit.client.1.smithi171.stdout:5/514: mkdir d5/db0 0 2022-01-31T19:44:28.164 INFO:tasks.workunit.client.1.smithi171.stdout:5/515: write d5/d6/d15/d5d/d6f/fa6 [971718,79416] 0 2022-01-31T19:44:28.164 INFO:tasks.workunit.client.1.smithi171.stdout:8/493: symlink df/d18/laa 0 2022-01-31T19:44:28.165 INFO:tasks.workunit.client.1.smithi171.stdout:6/561: mkdir dc/d21/d4f/db5 0 2022-01-31T19:44:28.174 INFO:tasks.workunit.client.1.smithi171.stdout:7/420: mkdir d1/d26/d47/d8f 0 2022-01-31T19:44:28.174 INFO:tasks.workunit.client.1.smithi171.stdout:7/421: chown d1/d1e/d2a/d29/d31/d30/f5a 3384 1 2022-01-31T19:44:28.178 INFO:tasks.workunit.client.0.smithi167.stdout:6/926: dwrite d8/d20/d103/d108/d135/d51/d137/f128 [0,4194304] 0 2022-01-31T19:44:28.180 INFO:tasks.workunit.client.1.smithi171.stdout:5/516: dread d5/d6/d15/d23/d67/f4c [0,4194304] 0 2022-01-31T19:44:28.183 INFO:tasks.workunit.client.0.smithi167.stdout:0/928: dwrite d17/d19/d1c/d39/d7b/dbd/ff7 [0,4194304] 0 2022-01-31T19:44:28.183 INFO:tasks.workunit.client.1.smithi171.stdout:2/716: rmdir d7/dd/dea 39 2022-01-31T19:44:28.184 INFO:tasks.workunit.client.1.smithi171.stdout:2/717: chown d7/d9/d1a/d45/da4/fce 7981075 1 2022-01-31T19:44:28.184 INFO:tasks.workunit.client.1.smithi171.stdout:6/562: dread dc/d18/f9d [0,4194304] 0 2022-01-31T19:44:28.185 INFO:tasks.workunit.client.1.smithi171.stdout:8/494: rename df/d2a/d38 to df/d18/d19/d21/dab 0 2022-01-31T19:44:28.185 INFO:tasks.workunit.client.1.smithi171.stdout:8/495: truncate df/d18/d19/f1b 1932618 0 2022-01-31T19:44:28.186 INFO:tasks.workunit.client.0.smithi167.stdout:0/929: dread d17/d19/d1c/d39/d44/d88/d98/de8/f10f [0,4194304] 0 2022-01-31T19:44:28.186 INFO:tasks.workunit.client.0.smithi167.stdout:6/927: write d8/d20/d103/d108/d135/f5f [688697,56285] 0 2022-01-31T19:44:28.187 INFO:tasks.workunit.client.1.smithi171.stdout:5/517: symlink d5/d6/d15/d23/d6b/dae/lb1 0 2022-01-31T19:44:28.188 INFO:tasks.workunit.client.0.smithi167.stdout:6/928: write d8/d20/d103/d108/d135/f32 [3972447,115438] 0 2022-01-31T19:44:28.189 INFO:tasks.workunit.client.0.smithi167.stdout:6/929: creat d8/d20/d103/d108/f14a x:0 0 0 2022-01-31T19:44:28.190 INFO:tasks.workunit.client.1.smithi171.stdout:6/563: symlink dc/d18/d86/lb6 0 2022-01-31T19:44:28.191 INFO:tasks.workunit.client.0.smithi167.stdout:6/930: mkdir d8/d20/d103/d108/d135/d51/d137/d112/d14b 0 2022-01-31T19:44:28.192 INFO:tasks.workunit.client.1.smithi171.stdout:2/718: dread d7/f56 [0,4194304] 0 2022-01-31T19:44:28.193 INFO:tasks.workunit.client.0.smithi167.stdout:6/931: mkdir d8/d20/d54/d12e/d14c 0 2022-01-31T19:44:28.194 INFO:tasks.workunit.client.0.smithi167.stdout:6/932: mknod d8/d20/d103/d108/d135/d51/d6a/df1/c14d 0 2022-01-31T19:44:28.195 INFO:tasks.workunit.client.0.smithi167.stdout:6/933: chown d8/d20/d103/d108/d135/d51/d7e 60 1 2022-01-31T19:44:28.195 INFO:tasks.workunit.client.0.smithi167.stdout:6/934: write d8/d20/d103/d108/d135/d51/d72/f10c [716654,48998] 0 2022-01-31T19:44:28.196 INFO:tasks.workunit.client.0.smithi167.stdout:6/935: mkdir d8/d62/d92/d110/d14e 0 2022-01-31T19:44:28.197 INFO:tasks.workunit.client.1.smithi171.stdout:7/422: dwrite d1/d1e/f85 [0,4194304] 0 2022-01-31T19:44:28.199 INFO:tasks.workunit.client.0.smithi167.stdout:6/936: write d8/d20/d103/d108/d135/f40 [4809429,76896] 0 2022-01-31T19:44:28.199 INFO:tasks.workunit.client.0.smithi167.stdout:6/937: readlink d8/d20/d103/d108/d135/d4f/l74 0 2022-01-31T19:44:28.199 INFO:tasks.workunit.client.0.smithi167.stdout:6/938: write d8/d20/d103/d108/d135/d51/d7e/f9b [246725,33083] 0 2022-01-31T19:44:28.201 INFO:tasks.workunit.client.0.smithi167.stdout:6/939: unlink d8/d20/d103/d108/d135/d4f/f11e 0 2022-01-31T19:44:28.202 INFO:tasks.workunit.client.0.smithi167.stdout:6/940: write d8/d20/d103/d108/d135/d51/d5d/d86/fb6 [143524,61066] 0 2022-01-31T19:44:28.202 INFO:tasks.workunit.client.1.smithi171.stdout:6/564: mknod dc/d19/d30/cb7 0 2022-01-31T19:44:28.203 INFO:tasks.workunit.client.1.smithi171.stdout:6/565: chown dc/d19/d29/c7c 6504 1 2022-01-31T19:44:28.203 INFO:tasks.workunit.client.0.smithi167.stdout:6/941: mkdir d8/d20/d103/d108/d135/d51/d5d/d14f 0 2022-01-31T19:44:28.204 INFO:tasks.workunit.client.0.smithi167.stdout:6/942: rmdir d8/d62 39 2022-01-31T19:44:28.206 INFO:tasks.workunit.client.1.smithi171.stdout:7/423: creat d1/d1e/d2a/d29/d31/d30/d3c/d5f/f90 x:0 0 0 2022-01-31T19:44:28.207 INFO:tasks.workunit.client.1.smithi171.stdout:6/566: dread dc/d19/d29/f63 [0,4194304] 0 2022-01-31T19:44:28.208 INFO:tasks.workunit.client.1.smithi171.stdout:7/424: unlink d1/d1e/d2a/d29/d31/l4e 0 2022-01-31T19:44:28.208 INFO:tasks.workunit.client.1.smithi171.stdout:6/567: creat dc/d19/d29/da7/fb8 x:0 0 0 2022-01-31T19:44:28.214 INFO:tasks.workunit.client.1.smithi171.stdout:2/719: dwrite d7/dd/d5e/d6b/d81/fe4 [0,4194304] 0 2022-01-31T19:44:28.215 INFO:tasks.workunit.client.1.smithi171.stdout:2/720: write d7/f62 [4833518,54998] 0 2022-01-31T19:44:28.217 INFO:tasks.workunit.client.0.smithi167.stdout:6/943: dread d8/d20/d103/d108/d135/d4f/d87/f13d [0,4194304] 0 2022-01-31T19:44:28.217 INFO:tasks.workunit.client.0.smithi167.stdout:6/944: fsync d8/d20/d103/d108/d135/d51/d5d/d64/dab/fd8 0 2022-01-31T19:44:28.228 INFO:tasks.workunit.client.1.smithi171.stdout:2/721: truncate d7/f62 2454814 0 2022-01-31T19:44:28.229 INFO:tasks.workunit.client.1.smithi171.stdout:2/722: getdents d7/dd/d5e/d6b 0 2022-01-31T19:44:28.230 INFO:tasks.workunit.client.1.smithi171.stdout:2/723: symlink d7/d26/d3d/d40/leb 0 2022-01-31T19:44:28.230 INFO:tasks.workunit.client.1.smithi171.stdout:2/724: symlink d7/dd/d5e/d6b/lec 0 2022-01-31T19:44:28.231 INFO:tasks.workunit.client.1.smithi171.stdout:2/725: mkdir d7/dd/d5e/d6b/ded 0 2022-01-31T19:44:28.238 INFO:tasks.workunit.client.1.smithi171.stdout:2/726: dread d7/dd/f39 [4194304,4194304] 0 2022-01-31T19:44:28.239 INFO:tasks.workunit.client.1.smithi171.stdout:2/727: fsync d7/d26/de5/fd0 0 2022-01-31T19:44:28.239 INFO:tasks.workunit.client.1.smithi171.stdout:2/728: dread - d7/d7c/fe8 zero size 2022-01-31T19:44:28.239 INFO:tasks.workunit.client.1.smithi171.stdout:2/729: chown d7 14599557 1 2022-01-31T19:44:28.239 INFO:tasks.workunit.client.1.smithi171.stdout:2/730: readlink d7/dd/d5e/d84/l90 0 2022-01-31T19:44:28.240 INFO:tasks.workunit.client.1.smithi171.stdout:2/731: write d7/d9/fe [694067,124954] 0 2022-01-31T19:44:28.240 INFO:tasks.workunit.client.1.smithi171.stdout:2/732: chown d7/f11 1 1 2022-01-31T19:44:28.241 INFO:tasks.workunit.client.1.smithi171.stdout:2/733: dread - d7/dd/fe6 zero size 2022-01-31T19:44:28.241 INFO:tasks.workunit.client.1.smithi171.stdout:2/734: chown d7/dd/f1b 402980 1 2022-01-31T19:44:28.241 INFO:tasks.workunit.client.1.smithi171.stdout:2/735: truncate d7/fc8 5742875 0 2022-01-31T19:44:28.242 INFO:tasks.workunit.client.1.smithi171.stdout:7/425: dwrite d1/d1e/d2a/d29/d31/d30/d6b/f86 [0,4194304] 0 2022-01-31T19:44:28.242 INFO:tasks.workunit.client.1.smithi171.stdout:6/568: dwrite dc/d21/fa2 [0,4194304] 0 2022-01-31T19:44:28.244 INFO:tasks.workunit.client.1.smithi171.stdout:7/426: truncate d1/d52/f21 182012 0 2022-01-31T19:44:28.244 INFO:tasks.workunit.client.1.smithi171.stdout:7/427: read - d1/d1e/d2a/d29/d31/d30/d6b/f7c zero size 2022-01-31T19:44:28.245 INFO:tasks.workunit.client.1.smithi171.stdout:7/428: stat d1/d52/f1a 0 2022-01-31T19:44:28.247 INFO:tasks.workunit.client.1.smithi171.stdout:6/569: rename dc/d19/d29/d27/d37/c43 to dc/d19/d29/d27/d37/d68/cb9 0 2022-01-31T19:44:28.248 INFO:tasks.workunit.client.1.smithi171.stdout:7/429: mknod d1/d1e/d2a/d29/d31/d30/d3c/d5f/c91 0 2022-01-31T19:44:28.249 INFO:tasks.workunit.client.1.smithi171.stdout:3/585: sync 2022-01-31T19:44:28.249 INFO:tasks.workunit.client.1.smithi171.stdout:4/449: sync 2022-01-31T19:44:28.249 INFO:tasks.workunit.client.1.smithi171.stdout:4/450: chown d2/d3/d5/d91/d7e/l58 51 1 2022-01-31T19:44:28.250 INFO:tasks.workunit.client.1.smithi171.stdout:2/736: dread d7/dd/f1b [0,4194304] 0 2022-01-31T19:44:28.250 INFO:tasks.workunit.client.1.smithi171.stdout:4/451: read - d2/d3/d5/d70/d7b/f93 zero size 2022-01-31T19:44:28.250 INFO:tasks.workunit.client.1.smithi171.stdout:2/737: stat d7/d26/d63/d9f/la0 0 2022-01-31T19:44:28.250 INFO:tasks.workunit.client.1.smithi171.stdout:7/430: symlink d1/d1e/d2a/d29/d31/d30/d3c/d79/l92 0 2022-01-31T19:44:28.251 INFO:tasks.workunit.client.1.smithi171.stdout:2/738: creat d7/d26/d63/d71/fee x:0 0 0 2022-01-31T19:44:28.252 INFO:tasks.workunit.client.1.smithi171.stdout:2/739: write d7/d26/d63/f67 [191213,61522] 0 2022-01-31T19:44:28.252 INFO:tasks.workunit.client.1.smithi171.stdout:2/740: chown d7/d9/d1a/d45 186474 1 2022-01-31T19:44:28.252 INFO:tasks.workunit.client.1.smithi171.stdout:4/452: link d2/d3/d5/d91/d7e/l1f d2/d3/d5/d70/l9c 0 2022-01-31T19:44:28.254 INFO:tasks.workunit.client.1.smithi171.stdout:7/431: truncate d1/d1e/d2a/d29/d31/ff 1026890 0 2022-01-31T19:44:28.254 INFO:tasks.workunit.client.1.smithi171.stdout:6/570: dread dc/d18/f9d [0,4194304] 0 2022-01-31T19:44:28.254 INFO:tasks.workunit.client.1.smithi171.stdout:7/432: dread - d1/d1b/f4c zero size 2022-01-31T19:44:28.255 INFO:tasks.workunit.client.1.smithi171.stdout:3/586: write d4/dd/d27/d28/f91 [2253893,36566] 0 2022-01-31T19:44:28.255 INFO:tasks.workunit.client.1.smithi171.stdout:3/587: truncate d4/dd/f66 1188518 0 2022-01-31T19:44:28.256 INFO:tasks.workunit.client.1.smithi171.stdout:2/741: truncate d7/f62 955541 0 2022-01-31T19:44:28.256 INFO:tasks.workunit.client.1.smithi171.stdout:4/453: truncate d2/f90 989689 0 2022-01-31T19:44:28.257 INFO:tasks.workunit.client.1.smithi171.stdout:6/571: unlink dc/d1e/d3f/f97 0 2022-01-31T19:44:28.257 INFO:tasks.workunit.client.1.smithi171.stdout:6/572: chown f2 3848919 1 2022-01-31T19:44:28.257 INFO:tasks.workunit.client.1.smithi171.stdout:6/573: fdatasync dc/d21/d4f/fb3 0 2022-01-31T19:44:28.258 INFO:tasks.workunit.client.1.smithi171.stdout:3/588: mkdir d4/dd/d27/d28/db6 0 2022-01-31T19:44:28.258 INFO:tasks.workunit.client.1.smithi171.stdout:2/742: rmdir d7/d7c/d83/d85/db5 0 2022-01-31T19:44:28.259 INFO:tasks.workunit.client.1.smithi171.stdout:4/454: creat d2/d3/d5/d91/f9d x:0 0 0 2022-01-31T19:44:28.259 INFO:tasks.workunit.client.1.smithi171.stdout:4/455: write d2/d3/d5/d91/d7e/d12/d15/f78 [991424,53445] 0 2022-01-31T19:44:28.260 INFO:tasks.workunit.client.1.smithi171.stdout:4/456: creat d2/f9e x:0 0 0 2022-01-31T19:44:28.261 INFO:tasks.workunit.client.1.smithi171.stdout:6/574: mknod dc/d21/d4f/db5/cba 0 2022-01-31T19:44:28.261 INFO:tasks.workunit.client.1.smithi171.stdout:6/575: write dc/d19/d29/d27/d37/d3b/d67/f9f [817679,88701] 0 2022-01-31T19:44:28.261 INFO:tasks.workunit.client.1.smithi171.stdout:6/576: read - dc/d21/fab zero size 2022-01-31T19:44:28.261 INFO:tasks.workunit.client.1.smithi171.stdout:6/577: creat dc/d19/d29/d6a/fbb x:0 0 0 2022-01-31T19:44:28.261 INFO:tasks.workunit.client.1.smithi171.stdout:3/589: symlink d4/dd/d27/d28/d35/d55/d8f/d92/dab/d60/lb7 0 2022-01-31T19:44:28.262 INFO:tasks.workunit.client.1.smithi171.stdout:4/457: rename d2/d3/d5/f59 to d2/d3/d5/d91/d7e/f9f 0 2022-01-31T19:44:28.264 INFO:tasks.workunit.client.1.smithi171.stdout:6/578: mknod dc/d1e/cbc 0 2022-01-31T19:44:28.264 INFO:tasks.workunit.client.1.smithi171.stdout:3/590: dread d4/dd/d27/d28/f57 [0,4194304] 0 2022-01-31T19:44:28.265 INFO:tasks.workunit.client.1.smithi171.stdout:2/743: mknod d7/dd/d5e/cef 0 2022-01-31T19:44:28.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/744: truncate d7/d9/d1a/d45/f9d 127078 0 2022-01-31T19:44:28.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/745: dread - d7/dd/d5e/d6b/d81/fad zero size 2022-01-31T19:44:28.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/746: fsync d7/dd/f3c 0 2022-01-31T19:44:28.266 INFO:tasks.workunit.client.1.smithi171.stdout:2/747: write d7/dd/d5e/d84/f8b [1731193,87829] 0 2022-01-31T19:44:28.267 INFO:tasks.workunit.client.1.smithi171.stdout:6/579: mknod dc/d1e/d3f/cbd 0 2022-01-31T19:44:28.268 INFO:tasks.workunit.client.1.smithi171.stdout:3/591: rename d4/dd/d27/d28/d35/d55/d8f/d92/dab/c6d to d4/dd/d87/cb8 0 2022-01-31T19:44:28.270 INFO:tasks.workunit.client.1.smithi171.stdout:2/748: unlink d7/dd/d5e/d6b/d81/d8d/da2/ld8 0 2022-01-31T19:44:28.272 INFO:tasks.workunit.client.1.smithi171.stdout:4/458: link d2/d3/d5/d91/d7e/d12/l40 d2/d3/la0 0 2022-01-31T19:44:28.273 INFO:tasks.workunit.client.1.smithi171.stdout:6/580: mknod dc/d19/d29/d27/d37/cbe 0 2022-01-31T19:44:28.277 INFO:tasks.workunit.client.1.smithi171.stdout:2/749: truncate d7/d9/d1a/f89 2591111 0 2022-01-31T19:44:28.277 INFO:tasks.workunit.client.1.smithi171.stdout:6/581: symlink dc/d1e/lbf 0 2022-01-31T19:44:28.279 INFO:tasks.workunit.client.1.smithi171.stdout:6/582: link dc/d19/d29/f24 dc/d19/d29/d27/d37/d3b/fc0 0 2022-01-31T19:44:28.286 INFO:tasks.workunit.client.1.smithi171.stdout:4/459: dwrite d2/d3/d5/d91/d7e/f8c [0,4194304] 0 2022-01-31T19:44:28.286 INFO:tasks.workunit.client.1.smithi171.stdout:4/460: fsync d2/d3/d5/d41/d6b/f99 0 2022-01-31T19:44:28.287 INFO:tasks.workunit.client.1.smithi171.stdout:4/461: write d2/d3/d5/d91/d7e/d12/f60 [635700,64697] 0 2022-01-31T19:44:28.287 INFO:tasks.workunit.client.1.smithi171.stdout:4/462: chown d2/f6d 1239 1 2022-01-31T19:44:28.313 INFO:tasks.workunit.client.1.smithi171.stdout:4/463: write d2/d3/d5/dc/f6f [4069830,2191] 0 2022-01-31T19:44:28.314 INFO:tasks.workunit.client.1.smithi171.stdout:4/464: write d2/d3/d5/d91/d7e/d12/d15/f24 [4022023,18996] 0 2022-01-31T19:44:28.316 INFO:tasks.workunit.client.1.smithi171.stdout:3/592: dread d4/dd/d27/d28/d35/d55/d8f/d92/dab/f2b [0,4194304] 0 2022-01-31T19:44:28.322 INFO:tasks.workunit.client.1.smithi171.stdout:6/583: dwrite dc/d18/f1b [8388608,4194304] 0 2022-01-31T19:44:28.322 INFO:tasks.workunit.client.1.smithi171.stdout:6/584: fdatasync dc/d21/d4f/fb3 0 2022-01-31T19:44:28.324 INFO:tasks.workunit.client.1.smithi171.stdout:6/585: write dc/d19/d29/d6a/fa0 [67569,50188] 0 2022-01-31T19:44:28.329 INFO:tasks.workunit.client.1.smithi171.stdout:4/465: truncate d2/d3/f25 3677644 0 2022-01-31T19:44:28.333 INFO:tasks.workunit.client.1.smithi171.stdout:3/593: dwrite d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/d7a/f8b [0,4194304] 0 2022-01-31T19:44:28.335 INFO:tasks.workunit.client.1.smithi171.stdout:7/433: dread d1/d1e/d2a/d29/d31/ff [0,4194304] 0 2022-01-31T19:44:28.335 INFO:tasks.workunit.client.1.smithi171.stdout:7/434: dread - d1/d1e/d2a/d29/d31/d30/d3c/f76 zero size 2022-01-31T19:44:28.338 INFO:tasks.workunit.client.1.smithi171.stdout:6/586: truncate dc/d18/f39 2313695 0 2022-01-31T19:44:28.339 INFO:tasks.workunit.client.1.smithi171.stdout:4/466: mknod d2/d3/d5/d91/d77/ca1 0 2022-01-31T19:44:28.339 INFO:tasks.workunit.client.1.smithi171.stdout:4/467: truncate d2/d3/d5/dc/f42 4973719 0 2022-01-31T19:44:28.339 INFO:tasks.workunit.client.1.smithi171.stdout:4/468: creat d2/fa2 x:0 0 0 2022-01-31T19:44:28.340 INFO:tasks.workunit.client.1.smithi171.stdout:4/469: dread - d2/f5b zero size 2022-01-31T19:44:28.350 INFO:tasks.workunit.client.1.smithi171.stdout:3/594: unlink d4/dd/l24 0 2022-01-31T19:44:28.353 INFO:tasks.workunit.client.1.smithi171.stdout:7/435: rename d1/d1e/d2a/d29/c70 to d1/c93 0 2022-01-31T19:44:28.368 INFO:tasks.workunit.client.1.smithi171.stdout:6/587: dwrite dc/d19/d30/f77 [4194304,4194304] 0 2022-01-31T19:44:28.369 INFO:tasks.workunit.client.1.smithi171.stdout:6/588: fdatasync dc/f7f 0 2022-01-31T19:44:28.371 INFO:tasks.workunit.client.1.smithi171.stdout:6/589: rename dc/d19/d30/d3c/lb1 to dc/d19/d29/daf/lc1 0 2022-01-31T19:44:28.371 INFO:tasks.workunit.client.1.smithi171.stdout:6/590: truncate dc/d1e/d3f/f59 1798820 0 2022-01-31T19:44:28.371 INFO:tasks.workunit.client.1.smithi171.stdout:6/591: mknod dc/d19/d29/d27/d37/cc2 0 2022-01-31T19:44:28.372 INFO:tasks.workunit.client.1.smithi171.stdout:6/592: rename dc/d19/d29/d27/d37/ca9 to dc/d19/d29/d27/d37/d3b/d67/cc3 0 2022-01-31T19:44:28.374 INFO:tasks.workunit.client.1.smithi171.stdout:6/593: mknod dc/d19/d29/d6a/cc4 0 2022-01-31T19:44:28.374 INFO:tasks.workunit.client.1.smithi171.stdout:6/594: write dc/d19/d29/d27/f93 [993703,73075] 0 2022-01-31T19:44:28.375 INFO:tasks.workunit.client.1.smithi171.stdout:6/595: creat dc/d8b/fc5 x:0 0 0 2022-01-31T19:44:28.376 INFO:tasks.workunit.client.0.smithi167.stdout:6/945: sync 2022-01-31T19:44:28.376 INFO:tasks.workunit.client.0.smithi167.stdout:0/930: sync 2022-01-31T19:44:28.376 INFO:tasks.workunit.client.0.smithi167.stdout:6/946: fsync d8/d20/f22 0 2022-01-31T19:44:28.376 INFO:tasks.workunit.client.0.smithi167.stdout:6/947: readlink d8/d20/d103/d108/d135/d51/d5d/l99 0 2022-01-31T19:44:28.377 INFO:tasks.workunit.client.0.smithi167.stdout:0/931: readlink d17/d2c/lac 0 2022-01-31T19:44:28.377 INFO:tasks.workunit.client.0.smithi167.stdout:0/932: rename d17/l119 to d17/d3e/d7e/dc7/l133 0 2022-01-31T19:44:28.378 INFO:tasks.workunit.client.0.smithi167.stdout:0/933: truncate d17/d2c/f3c 199027 0 2022-01-31T19:44:28.378 INFO:tasks.workunit.client.1.smithi171.stdout:6/596: mknod dc/d19/d30/d3c/cc6 0 2022-01-31T19:44:28.379 INFO:tasks.workunit.client.1.smithi171.stdout:6/597: dread - dc/d8b/fc5 zero size 2022-01-31T19:44:28.379 INFO:tasks.workunit.client.1.smithi171.stdout:6/598: creat dc/d19/d30/d3c/fc7 x:0 0 0 2022-01-31T19:44:28.379 INFO:tasks.workunit.client.1.smithi171.stdout:6/599: creat dc/d1e/d3f/fc8 x:0 0 0 2022-01-31T19:44:28.380 INFO:tasks.workunit.client.0.smithi167.stdout:0/934: truncate d17/d19/d1c/fc3 2419018 0 2022-01-31T19:44:28.380 INFO:tasks.workunit.client.0.smithi167.stdout:0/935: chown d17/d19/d1c/d39/d44/d49 4109 1 2022-01-31T19:44:28.381 INFO:tasks.workunit.client.1.smithi171.stdout:4/470: dwrite d2/d3/d5/f2e [0,4194304] 0 2022-01-31T19:44:28.381 INFO:tasks.workunit.client.1.smithi171.stdout:4/471: chown d2/d3/d5/d41/d6b 1582135682 1 2022-01-31T19:44:28.381 INFO:tasks.workunit.client.1.smithi171.stdout:7/436: dwrite d1/d1e/d2a/d29/d77/f8c [0,4194304] 0 2022-01-31T19:44:28.382 INFO:tasks.workunit.client.1.smithi171.stdout:3/595: dwrite d4/dd/d27/d28/d35/d55/d8f/d92/dab/f84 [0,4194304] 0 2022-01-31T19:44:28.382 INFO:tasks.workunit.client.1.smithi171.stdout:7/437: unlink d1/d1b/l3e 0 2022-01-31T19:44:28.382 INFO:tasks.workunit.client.1.smithi171.stdout:4/472: write d2/f3c [1347749,32212] 0 2022-01-31T19:44:28.384 INFO:tasks.workunit.client.0.smithi167.stdout:6/948: dread d8/d20/d103/d108/d135/d4f/f8e [0,4194304] 0 2022-01-31T19:44:28.386 INFO:tasks.workunit.client.0.smithi167.stdout:6/949: fdatasync d8/d20/d103/d108/d135/d51/d72/f83 0 2022-01-31T19:44:28.387 INFO:tasks.workunit.client.0.smithi167.stdout:6/950: creat d8/d20/d103/d108/d135/d35/f150 x:0 0 0 2022-01-31T19:44:28.388 INFO:tasks.workunit.client.1.smithi171.stdout:3/596: getdents d4/dd/d27/d6e 0 2022-01-31T19:44:28.389 INFO:tasks.workunit.client.0.smithi167.stdout:6/951: rename d8/d20/d103/d108/d135/d51/d5d to d8/d20/d103/d108/d151 0 2022-01-31T19:44:28.391 INFO:tasks.workunit.client.1.smithi171.stdout:7/438: link d1/d52/f44 d1/d1e/d2a/d29/d31/d30/d6b/f94 0 2022-01-31T19:44:28.391 INFO:tasks.workunit.client.1.smithi171.stdout:7/439: stat d1/d1e/d2a/d29/d31/d30/f88 0 2022-01-31T19:44:28.393 INFO:tasks.workunit.client.1.smithi171.stdout:4/473: symlink d2/d3/d5/d70/d7c/la3 0 2022-01-31T19:44:28.393 INFO:tasks.workunit.client.1.smithi171.stdout:4/474: readlink d2/d3/d5/d91/d7e/d12/d15/l4e 0 2022-01-31T19:44:28.394 INFO:tasks.workunit.client.1.smithi171.stdout:4/475: fsync d2/d3/d5/d91/d7e/f62 0 2022-01-31T19:44:28.394 INFO:tasks.workunit.client.0.smithi167.stdout:6/952: rename d8/d62/d92/f121 to d8/dca/f152 0 2022-01-31T19:44:28.395 INFO:tasks.workunit.client.0.smithi167.stdout:6/953: fdatasync d8/d20/d103/d108/d135/d51/d7e/f141 0 2022-01-31T19:44:28.395 INFO:tasks.workunit.client.1.smithi171.stdout:6/600: fsync dc/d1e/d3f/fc8 0 2022-01-31T19:44:28.396 INFO:tasks.workunit.client.1.smithi171.stdout:6/601: truncate dc/d19/d29/f24 1417912 0 2022-01-31T19:44:28.396 INFO:tasks.workunit.client.1.smithi171.stdout:6/602: fsync dc/d21/f8d 0 2022-01-31T19:44:28.396 INFO:tasks.workunit.client.1.smithi171.stdout:7/440: symlink d1/d1e/d2a/d29/d31/d30/d6b/l95 0 2022-01-31T19:44:28.397 INFO:tasks.workunit.client.1.smithi171.stdout:6/603: write dc/d19/d29/d27/f93 [988832,8079] 0 2022-01-31T19:44:28.397 INFO:tasks.workunit.client.1.smithi171.stdout:4/476: unlink d2/d3/d5/d91/d7e/d12/d15/f3a 0 2022-01-31T19:44:28.399 INFO:tasks.workunit.client.0.smithi167.stdout:6/954: write d8/d20/d103/d108/d135/d51/d6a/fa3 [2137120,65699] 0 2022-01-31T19:44:28.400 INFO:tasks.workunit.client.1.smithi171.stdout:4/477: mknod d2/d3/d5/d91/ca4 0 2022-01-31T19:44:28.401 INFO:tasks.workunit.client.0.smithi167.stdout:6/955: rmdir d8/d20/d54 39 2022-01-31T19:44:28.403 INFO:tasks.workunit.client.1.smithi171.stdout:6/604: getdents dc/d1e/d3f 0 2022-01-31T19:44:28.403 INFO:tasks.workunit.client.1.smithi171.stdout:6/605: stat dc/d18/f1b 0 2022-01-31T19:44:28.404 INFO:tasks.workunit.client.1.smithi171.stdout:4/478: mkdir d2/d3/d5/d91/d77/da5 0 2022-01-31T19:44:28.405 INFO:tasks.workunit.client.1.smithi171.stdout:4/479: fdatasync d2/d3/f25 0 2022-01-31T19:44:28.405 INFO:tasks.workunit.client.1.smithi171.stdout:4/480: stat d2/d3/d5/d70/d7b/f93 0 2022-01-31T19:44:28.406 INFO:tasks.workunit.client.0.smithi167.stdout:0/936: dwrite d17/d19/d1c/d39/d44/d88/f123 [0,4194304] 0 2022-01-31T19:44:28.406 INFO:tasks.workunit.client.0.smithi167.stdout:0/937: chown d17/d2c/dcd/c111 708 1 2022-01-31T19:44:28.409 INFO:tasks.workunit.client.0.smithi167.stdout:6/956: write d8/d20/d103/d108/d135/f3c [6962303,90301] 0 2022-01-31T19:44:28.414 INFO:tasks.workunit.client.1.smithi171.stdout:3/597: dwrite d4/dd/d39/d51/f53 [0,4194304] 0 2022-01-31T19:44:28.418 INFO:tasks.workunit.client.1.smithi171.stdout:7/441: dwrite d1/d1e/d2a/d29/d31/ff [0,4194304] 0 2022-01-31T19:44:28.420 INFO:tasks.workunit.client.1.smithi171.stdout:4/481: write d2/d3/d5/dc/f6f [1475732,73340] 0 2022-01-31T19:44:28.421 INFO:tasks.workunit.client.1.smithi171.stdout:3/598: creat d4/dd/d27/d28/d35/d55/fb9 x:0 0 0 2022-01-31T19:44:28.423 INFO:tasks.workunit.client.1.smithi171.stdout:3/599: creat d4/dd/d27/d28/d35/d55/d8f/fba x:0 0 0 2022-01-31T19:44:28.423 INFO:tasks.workunit.client.1.smithi171.stdout:3/600: dread - d4/dd/d27/d28/d35/d55/d6c/fad zero size 2022-01-31T19:44:28.424 INFO:tasks.workunit.client.1.smithi171.stdout:3/601: write d4/dd/d27/f3f [4653403,92888] 0 2022-01-31T19:44:28.424 INFO:tasks.workunit.client.1.smithi171.stdout:3/602: creat d4/dd/d39/fbb x:0 0 0 2022-01-31T19:44:28.424 INFO:tasks.workunit.client.1.smithi171.stdout:4/482: read d2/d3/d5/d41/d6b/f99 [3347625,129301] 0 2022-01-31T19:44:28.424 INFO:tasks.workunit.client.1.smithi171.stdout:4/483: chown d2/d3/d5/d91/d7e/d12/f5a 11154289 1 2022-01-31T19:44:28.425 INFO:tasks.workunit.client.1.smithi171.stdout:4/484: chown d2/d3/d5/dc/f4f 29454622 1 2022-01-31T19:44:28.425 INFO:tasks.workunit.client.1.smithi171.stdout:3/603: write d4/dd/d27/d28/f57 [3351584,52036] 0 2022-01-31T19:44:28.427 INFO:tasks.workunit.client.1.smithi171.stdout:7/442: dread d1/f36 [4194304,4194304] 0 2022-01-31T19:44:28.428 INFO:tasks.workunit.client.1.smithi171.stdout:7/443: symlink d1/d1b/l96 0 2022-01-31T19:44:28.429 INFO:tasks.workunit.client.1.smithi171.stdout:7/444: getdents d1/d1e/d2a/d29/d31/d30 0 2022-01-31T19:44:28.430 INFO:tasks.workunit.client.0.smithi167.stdout:0/938: dwrite d17/d19/d1c/d39/d7b/dbd/ff7 [0,4194304] 0 2022-01-31T19:44:28.430 INFO:tasks.workunit.client.0.smithi167.stdout:0/939: creat d17/d3e/d7e/dc7/dde/df1/f134 x:0 0 0 2022-01-31T19:44:28.432 INFO:tasks.workunit.client.0.smithi167.stdout:6/957: dwrite d8/d62/f6f [4194304,4194304] 0 2022-01-31T19:44:28.435 INFO:tasks.workunit.client.1.smithi171.stdout:0/602: sync 2022-01-31T19:44:28.435 INFO:tasks.workunit.client.1.smithi171.stdout:9/613: sync 2022-01-31T19:44:28.435 INFO:tasks.workunit.client.1.smithi171.stdout:1/496: sync 2022-01-31T19:44:28.436 INFO:tasks.workunit.client.0.smithi167.stdout:6/958: rename d8/d20/c7c to d8/d20/d103/d108/d135/d51/d6a/df1/d106/c153 0 2022-01-31T19:44:28.436 INFO:tasks.workunit.client.0.smithi167.stdout:0/940: rmdir d17/d19/d1c/d39/d44 39 2022-01-31T19:44:28.437 INFO:tasks.workunit.client.1.smithi171.stdout:0/603: fdatasync f6 0 2022-01-31T19:44:28.437 INFO:tasks.workunit.client.1.smithi171.stdout:1/497: write dd/d23/d34/d4b/f52 [1205713,74770] 0 2022-01-31T19:44:28.438 INFO:tasks.workunit.client.1.smithi171.stdout:1/498: chown dd/d23/d2e/d5e/d81/d98/c92 776371858 1 2022-01-31T19:44:28.438 INFO:tasks.workunit.client.1.smithi171.stdout:9/614: mknod dd/d1c/da8/ccc 0 2022-01-31T19:44:28.438 INFO:tasks.workunit.client.1.smithi171.stdout:4/485: dread d2/d3/f7 [0,4194304] 0 2022-01-31T19:44:28.439 INFO:tasks.workunit.client.1.smithi171.stdout:4/486: chown d2/d3/d5/d41/c82 73 1 2022-01-31T19:44:28.439 INFO:tasks.workunit.client.1.smithi171.stdout:4/487: stat d2/f90 0 2022-01-31T19:44:28.440 INFO:tasks.workunit.client.1.smithi171.stdout:0/604: mkdir d9/db/d22/d73/ddc 0 2022-01-31T19:44:28.440 INFO:tasks.workunit.client.1.smithi171.stdout:1/499: symlink dd/df/l9b 0 2022-01-31T19:44:28.443 INFO:tasks.workunit.client.1.smithi171.stdout:0/605: rename d9/db/dc/d3f/l61 to d9/db/dc/d3f/ldd 0 2022-01-31T19:44:28.445 INFO:tasks.workunit.client.1.smithi171.stdout:1/500: creat dd/df/d19/d1b/d1e/d97/f9c x:0 0 0 2022-01-31T19:44:28.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/501: chown dd/c6e 46571 1 2022-01-31T19:44:28.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/502: creat dd/df/d19/d36/f9d x:0 0 0 2022-01-31T19:44:28.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/503: readlink dd/df/l9b 0 2022-01-31T19:44:28.447 INFO:tasks.workunit.client.1.smithi171.stdout:1/504: write dd/df/d19/f75 [4039945,57430] 0 2022-01-31T19:44:28.448 INFO:tasks.workunit.client.1.smithi171.stdout:0/606: rename d9/db/d59/d71/dba/dbb/ccb to d9/d24/d4c/d35/cde 0 2022-01-31T19:44:28.452 INFO:tasks.workunit.client.1.smithi171.stdout:1/505: rename dd/d23/d74/d7b/f94 to dd/d23/f9e 0 2022-01-31T19:44:28.453 INFO:tasks.workunit.client.1.smithi171.stdout:0/607: mknod d9/db/cdf 0 2022-01-31T19:44:28.454 INFO:tasks.workunit.client.1.smithi171.stdout:0/608: creat d9/fe0 x:0 0 0 2022-01-31T19:44:28.454 INFO:tasks.workunit.client.1.smithi171.stdout:0/609: dread - d9/db/dc/d3f/db9/d7a/d88/fa3 zero size 2022-01-31T19:44:28.454 INFO:tasks.workunit.client.1.smithi171.stdout:1/506: write dd/d23/d74/d7b/f99 [4853697,118000] 0 2022-01-31T19:44:28.455 INFO:tasks.workunit.client.1.smithi171.stdout:0/610: symlink d9/db/dc/d3f/db9/d7a/d88/le1 0 2022-01-31T19:44:28.459 INFO:tasks.workunit.client.1.smithi171.stdout:0/611: rename d9/d24/d92/d9d/dc6/fd2 to d9/db/d59/d71/fe2 0 2022-01-31T19:44:28.459 INFO:tasks.workunit.client.1.smithi171.stdout:0/612: write d9/d24/d4c/d35/f8d [1010966,33498] 0 2022-01-31T19:44:28.459 INFO:tasks.workunit.client.1.smithi171.stdout:0/613: stat d9/d24/d92/f94 0 2022-01-31T19:44:28.459 INFO:tasks.workunit.client.1.smithi171.stdout:0/614: fdatasync d9/db/f41 0 2022-01-31T19:44:28.462 INFO:tasks.workunit.client.1.smithi171.stdout:1/507: write dd/df/d19/f27 [1368221,129862] 0 2022-01-31T19:44:28.462 INFO:tasks.workunit.client.1.smithi171.stdout:1/508: write dd/d23/d34/d4b/f59 [934373,65277] 0 2022-01-31T19:44:28.462 INFO:tasks.workunit.client.1.smithi171.stdout:1/509: truncate dd/df/d19/d1b/d1e/d97/f9c 800739 0 2022-01-31T19:44:28.464 INFO:tasks.workunit.client.1.smithi171.stdout:1/510: creat dd/df/d19/d1b/d1e/f9f x:0 0 0 2022-01-31T19:44:28.466 INFO:tasks.workunit.client.1.smithi171.stdout:1/511: link dd/d23/d2e/d5e/f45 dd/d23/d2e/d5e/d83/fa0 0 2022-01-31T19:44:28.468 INFO:tasks.workunit.client.1.smithi171.stdout:4/488: dread d2/d3/d5/dc/f42 [0,4194304] 0 2022-01-31T19:44:28.469 INFO:tasks.workunit.client.1.smithi171.stdout:9/615: dwrite dd/df/d15/d38/d4d/f90 [0,4194304] 0 2022-01-31T19:44:28.471 INFO:tasks.workunit.client.1.smithi171.stdout:4/489: getdents d2/d3/d5 0 2022-01-31T19:44:28.492 INFO:tasks.workunit.client.0.smithi167.stdout:6/959: dwrite d8/d20/d103/d108/d151/d64/dab/fd7 [0,4194304] 0 2022-01-31T19:44:28.493 INFO:tasks.workunit.client.0.smithi167.stdout:6/960: chown d8/l17 1736 1 2022-01-31T19:44:28.493 INFO:tasks.workunit.client.0.smithi167.stdout:6/961: stat d8/f73 0 2022-01-31T19:44:28.494 INFO:tasks.workunit.client.0.smithi167.stdout:0/941: dwrite d17/d2c/f3c [0,4194304] 0 2022-01-31T19:44:28.494 INFO:tasks.workunit.client.1.smithi171.stdout:7/445: dwrite d1/d1e/d2a/d29/d31/f6a [4194304,4194304] 0 2022-01-31T19:44:28.495 INFO:tasks.workunit.client.1.smithi171.stdout:7/446: write d1/d1e/d2a/d29/d31/d30/d6b/f7c [738431,4968] 0 2022-01-31T19:44:28.495 INFO:tasks.workunit.client.1.smithi171.stdout:3/604: dwrite d4/dd/d27/f3f [0,4194304] 0 2022-01-31T19:44:28.495 INFO:tasks.workunit.client.1.smithi171.stdout:3/605: chown c2 1836000580 1 2022-01-31T19:44:28.496 INFO:tasks.workunit.client.1.smithi171.stdout:3/606: readlink d4/dd/d27/d28/d35/d55/d8f/d92/l32 0 2022-01-31T19:44:28.496 INFO:tasks.workunit.client.0.smithi167.stdout:0/942: mkdir d17/d19/d1c/de1/d135 0 2022-01-31T19:44:28.498 INFO:tasks.workunit.client.1.smithi171.stdout:4/490: dwrite d2/d3/d5/d91/d7e/f8c [0,4194304] 0 2022-01-31T19:44:28.499 INFO:tasks.workunit.client.1.smithi171.stdout:0/615: dwrite d9/d24/d92/d9d/d9f/fac [0,4194304] 0 2022-01-31T19:44:28.500 INFO:tasks.workunit.client.1.smithi171.stdout:7/447: creat d1/d26/d81/f97 x:0 0 0 2022-01-31T19:44:28.507 INFO:tasks.workunit.client.1.smithi171.stdout:4/491: unlink d2/d3/d5/l34 0 2022-01-31T19:44:28.512 INFO:tasks.workunit.client.1.smithi171.stdout:0/616: dread f3 [0,4194304] 0 2022-01-31T19:44:28.512 INFO:tasks.workunit.client.1.smithi171.stdout:0/617: chown d9/d24/f26 6 1 2022-01-31T19:44:28.515 INFO:tasks.workunit.client.0.smithi167.stdout:6/962: dread d8/d20/d54/f85 [0,4194304] 0 2022-01-31T19:44:28.515 INFO:tasks.workunit.client.1.smithi171.stdout:4/492: dread d2/d3/d5/dc/f42 [0,4194304] 0 2022-01-31T19:44:28.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/493: dread - d2/d3/d5/d91/f76 zero size 2022-01-31T19:44:28.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/494: fsync d2/d3/d5/d91/d7e/d12/f8d 0 2022-01-31T19:44:28.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/495: chown d2/d3/d5/d91/d7e/d12/d15/l74 1833930 1 2022-01-31T19:44:28.517 INFO:tasks.workunit.client.0.smithi167.stdout:6/963: write d8/d20/d103/d108/d135/d51/ff4 [46269,41546] 0 2022-01-31T19:44:28.517 INFO:tasks.workunit.client.0.smithi167.stdout:6/964: mknod d8/d20/d54/c154 0 2022-01-31T19:44:28.518 INFO:tasks.workunit.client.1.smithi171.stdout:1/512: dwrite dd/d23/f5c [8388608,4194304] 0 2022-01-31T19:44:28.519 INFO:tasks.workunit.client.0.smithi167.stdout:6/965: symlink d8/d62/l155 0 2022-01-31T19:44:28.525 INFO:tasks.workunit.client.1.smithi171.stdout:7/448: link d1/d1e/d2a/d29/d31/d30/d3c/d5f/f90 d1/d1e/d2a/d58/f98 0 2022-01-31T19:44:28.526 INFO:tasks.workunit.client.1.smithi171.stdout:7/449: fsync d1/d52/f1a 0 2022-01-31T19:44:28.526 INFO:tasks.workunit.client.1.smithi171.stdout:3/607: rename d4/c5d to d4/dd/d87/d8e/cbc 0 2022-01-31T19:44:28.528 INFO:tasks.workunit.client.1.smithi171.stdout:0/618: getdents d9/d24/d4c 0 2022-01-31T19:44:28.529 INFO:tasks.workunit.client.1.smithi171.stdout:4/496: mkdir d2/d3/d5/d91/da6 0 2022-01-31T19:44:28.534 INFO:tasks.workunit.client.1.smithi171.stdout:1/513: creat dd/df/d84/fa1 x:0 0 0 2022-01-31T19:44:28.534 INFO:tasks.workunit.client.1.smithi171.stdout:1/514: dread - dd/df/d19/d1b/d1e/f9f zero size 2022-01-31T19:44:28.534 INFO:tasks.workunit.client.1.smithi171.stdout:1/515: write dd/df/f38 [517790,5064] 0 2022-01-31T19:44:28.534 INFO:tasks.workunit.client.1.smithi171.stdout:1/516: write dd/d23/d2e/d5e/d81/d98/d80/f8e [4693906,117098] 0 2022-01-31T19:44:28.535 INFO:tasks.workunit.client.1.smithi171.stdout:3/608: creat d4/dd/d27/d28/d35/d55/d8f/d92/dab/d60/fbd x:0 0 0 2022-01-31T19:44:28.535 INFO:tasks.workunit.client.1.smithi171.stdout:0/619: link d9/db/dc2/l2a d9/d24/d4c/d35/le3 0 2022-01-31T19:44:28.536 INFO:tasks.workunit.client.1.smithi171.stdout:0/620: truncate d9/d24/d4c/d35/f89 390625 0 2022-01-31T19:44:28.536 INFO:tasks.workunit.client.0.smithi167.stdout:6/966: dwrite d8/f7a [0,4194304] 0 2022-01-31T19:44:28.537 INFO:tasks.workunit.client.0.smithi167.stdout:6/967: fsync d8/d20/d103/d108/d135/d51/ff2 0 2022-01-31T19:44:28.537 INFO:tasks.workunit.client.1.smithi171.stdout:4/497: creat d2/d3/d5/d70/fa7 x:0 0 0 2022-01-31T19:44:28.537 INFO:tasks.workunit.client.1.smithi171.stdout:4/498: truncate d2/d3/d5/d91/f76 751058 0 2022-01-31T19:44:28.539 INFO:tasks.workunit.client.1.smithi171.stdout:4/499: write d2/d3/d5/d91/d7e/d12/d15/f24 [6240948,77030] 0 2022-01-31T19:44:28.540 INFO:tasks.workunit.client.0.smithi167.stdout:6/968: symlink d8/d20/d103/d108/d135/d51/d72/dc7/l156 0 2022-01-31T19:44:28.559 INFO:tasks.workunit.client.1.smithi171.stdout:3/609: mkdir d4/d21/d3a/db5/dbe 0 2022-01-31T19:44:28.563 INFO:tasks.workunit.client.1.smithi171.stdout:1/517: dwrite dd/df/d19/d1b/d1e/d35/f70 [4194304,4194304] 0 2022-01-31T19:44:28.565 INFO:tasks.workunit.client.1.smithi171.stdout:1/518: rmdir dd/d23/d2e/d5e 39 2022-01-31T19:44:28.570 INFO:tasks.workunit.client.1.smithi171.stdout:5/518: sync 2022-01-31T19:44:28.571 INFO:tasks.workunit.client.1.smithi171.stdout:8/496: sync 2022-01-31T19:44:28.573 INFO:tasks.workunit.client.1.smithi171.stdout:3/610: dread d4/d21/d3a/d90/f94 [0,4194304] 0 2022-01-31T19:44:28.574 INFO:tasks.workunit.client.1.smithi171.stdout:1/519: mkdir dd/da2 0 2022-01-31T19:44:28.574 INFO:tasks.workunit.client.1.smithi171.stdout:1/520: write dd/df/d19/d36/f5b [4251666,96541] 0 2022-01-31T19:44:28.574 INFO:tasks.workunit.client.1.smithi171.stdout:3/611: mkdir d4/dd/d27/d28/d35/d55/dbf 0 2022-01-31T19:44:28.576 INFO:tasks.workunit.client.1.smithi171.stdout:3/612: mkdir d4/dd/d27/d28/db6/dc0 0 2022-01-31T19:44:28.576 INFO:tasks.workunit.client.1.smithi171.stdout:3/613: write d4/dd/d39/f50 [148511,102802] 0 2022-01-31T19:44:28.577 INFO:tasks.workunit.client.1.smithi171.stdout:1/521: dread dd/df/f38 [0,4194304] 0 2022-01-31T19:44:28.577 INFO:tasks.workunit.client.1.smithi171.stdout:3/614: mkdir d4/d21/d3a/d59/dc1 0 2022-01-31T19:44:28.578 INFO:tasks.workunit.client.1.smithi171.stdout:1/522: creat dd/d23/d2e/d5e/d81/fa3 x:0 0 0 2022-01-31T19:44:28.579 INFO:tasks.workunit.client.1.smithi171.stdout:3/615: creat d4/d21/d3a/d59/dc1/fc2 x:0 0 0 2022-01-31T19:44:28.579 INFO:tasks.workunit.client.1.smithi171.stdout:3/616: truncate d4/d21/d3a/f7c 570324 0 2022-01-31T19:44:28.580 INFO:tasks.workunit.client.1.smithi171.stdout:3/617: fdatasync d4/dd/d87/fa6 0 2022-01-31T19:44:28.581 INFO:tasks.workunit.client.1.smithi171.stdout:1/523: symlink dd/df/d19/d1b/d1e/d97/la4 0 2022-01-31T19:44:28.584 INFO:tasks.workunit.client.1.smithi171.stdout:3/618: read d4/dd/f19 [942822,123563] 0 2022-01-31T19:44:28.584 INFO:tasks.workunit.client.1.smithi171.stdout:3/619: chown d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f96 25020 1 2022-01-31T19:44:28.598 INFO:tasks.workunit.client.1.smithi171.stdout:1/524: dwrite dd/f56 [0,4194304] 0 2022-01-31T19:44:28.619 INFO:tasks.workunit.client.1.smithi171.stdout:1/525: dread f9 [0,4194304] 0 2022-01-31T19:44:28.620 INFO:tasks.workunit.client.1.smithi171.stdout:3/620: dwrite d4/d21/d3a/d59/f70 [0,4194304] 0 2022-01-31T19:44:28.621 INFO:tasks.workunit.client.1.smithi171.stdout:1/526: rename dd/df/d19/d36/f95 to dd/d23/d2e/fa5 0 2022-01-31T19:44:28.622 INFO:tasks.workunit.client.1.smithi171.stdout:1/527: fsync dd/df/d19/f27 0 2022-01-31T19:44:28.625 INFO:tasks.workunit.client.1.smithi171.stdout:3/621: mkdir d4/d21/d3a/db5/dbe/dc3 0 2022-01-31T19:44:28.627 INFO:tasks.workunit.client.1.smithi171.stdout:1/528: creat dd/da2/fa6 x:0 0 0 2022-01-31T19:44:28.631 INFO:tasks.workunit.client.1.smithi171.stdout:3/622: creat d4/dd/d27/d28/d35/d67/fc4 x:0 0 0 2022-01-31T19:44:28.631 INFO:tasks.workunit.client.1.smithi171.stdout:3/623: fsync d4/dd/d87/f9f 0 2022-01-31T19:44:28.631 INFO:tasks.workunit.client.1.smithi171.stdout:3/624: read - d4/dd/d27/d28/d35/d67/fc4 zero size 2022-01-31T19:44:28.631 INFO:tasks.workunit.client.1.smithi171.stdout:3/625: chown d4/dd/d27/d28/d35/d55/d8f/d92/dab/c1f 24610951 1 2022-01-31T19:44:28.632 INFO:tasks.workunit.client.1.smithi171.stdout:3/626: dread - d4/dd/d87/f9f zero size 2022-01-31T19:44:28.632 INFO:tasks.workunit.client.1.smithi171.stdout:1/529: dread dd/df/d19/d1b/d1e/f4a [0,4194304] 0 2022-01-31T19:44:28.633 INFO:tasks.workunit.client.1.smithi171.stdout:3/627: truncate d4/dd/d27/d28/d35/d55/d8f/d92/dab/f2b 188851 0 2022-01-31T19:44:28.633 INFO:tasks.workunit.client.1.smithi171.stdout:3/628: truncate d4/dd/d39/fbb 426111 0 2022-01-31T19:44:28.634 INFO:tasks.workunit.client.1.smithi171.stdout:1/530: rmdir dd/d23/d2e/d5e/d81/d98 39 2022-01-31T19:44:28.634 INFO:tasks.workunit.client.1.smithi171.stdout:1/531: fsync dd/d23/d2e/f87 0 2022-01-31T19:44:28.635 INFO:tasks.workunit.client.1.smithi171.stdout:3/629: link d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/d7a/f8b d4/dd/d27/d28/d35/d55/dbf/fc5 0 2022-01-31T19:44:28.635 INFO:tasks.workunit.client.1.smithi171.stdout:3/630: stat d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f8d 0 2022-01-31T19:44:28.636 INFO:tasks.workunit.client.1.smithi171.stdout:3/631: creat d4/fc6 x:0 0 0 2022-01-31T19:44:28.638 INFO:tasks.workunit.client.1.smithi171.stdout:3/632: truncate d4/d21/f31 218654 0 2022-01-31T19:44:28.638 INFO:tasks.workunit.client.1.smithi171.stdout:3/633: dread - d4/dd/d87/d8e/faf zero size 2022-01-31T19:44:28.638 INFO:tasks.workunit.client.1.smithi171.stdout:3/634: unlink d4/dd/f66 0 2022-01-31T19:44:28.654 INFO:tasks.workunit.client.1.smithi171.stdout:1/532: dwrite f7 [0,4194304] 0 2022-01-31T19:44:28.663 INFO:tasks.workunit.client.1.smithi171.stdout:3/635: dwrite d4/dd/d27/d28/d35/d55/d8f/fba [0,4194304] 0 2022-01-31T19:44:28.671 INFO:tasks.workunit.client.1.smithi171.stdout:1/533: dwrite dd/df/d84/fa1 [0,4194304] 0 2022-01-31T19:44:28.673 INFO:tasks.workunit.client.1.smithi171.stdout:3/636: truncate d4/d21/d3a/d59/f70 1101770 0 2022-01-31T19:44:28.674 INFO:tasks.workunit.client.1.smithi171.stdout:3/637: write d4/dd/d39/d51/fb4 [771465,22078] 0 2022-01-31T19:44:28.675 INFO:tasks.workunit.client.1.smithi171.stdout:1/534: mkdir dd/df/d19/d1b/d1e/d64/da7 0 2022-01-31T19:44:28.676 INFO:tasks.workunit.client.1.smithi171.stdout:1/535: creat dd/d23/d2e/d5e/d81/fa8 x:0 0 0 2022-01-31T19:44:28.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/638: mkdir d4/dd/dc7 0 2022-01-31T19:44:28.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/639: chown d4/dd/d27/d28/d35/d55/d8f/d92/dab/d6b 1589147 1 2022-01-31T19:44:28.677 INFO:tasks.workunit.client.0.smithi167.stdout:0/943: sync 2022-01-31T19:44:28.678 INFO:tasks.workunit.client.0.smithi167.stdout:0/944: dread - d17/d33/d112/f116 zero size 2022-01-31T19:44:28.678 INFO:tasks.workunit.client.0.smithi167.stdout:0/945: creat d17/d33/d11c/f136 x:0 0 0 2022-01-31T19:44:28.678 INFO:tasks.workunit.client.0.smithi167.stdout:0/946: creat d17/d33/da4/dd0/d105/d11d/f137 x:0 0 0 2022-01-31T19:44:28.679 INFO:tasks.workunit.client.0.smithi167.stdout:0/947: rename d17/d33/da4/c118 to d17/d33/da4/dba/c138 0 2022-01-31T19:44:28.680 INFO:tasks.workunit.client.0.smithi167.stdout:0/948: rmdir d17/d19/d1c 39 2022-01-31T19:44:28.680 INFO:tasks.workunit.client.0.smithi167.stdout:0/949: write d17/d3e/d7e/fce [648838,83986] 0 2022-01-31T19:44:28.682 INFO:tasks.workunit.client.0.smithi167.stdout:0/950: mkdir d17/d2c/dcd/d139 0 2022-01-31T19:44:28.682 INFO:tasks.workunit.client.0.smithi167.stdout:0/951: truncate d17/d19/f4d 4921934 0 2022-01-31T19:44:28.684 INFO:tasks.workunit.client.1.smithi171.stdout:1/536: dread f7 [0,4194304] 0 2022-01-31T19:44:28.685 INFO:tasks.workunit.client.1.smithi171.stdout:1/537: mkdir dd/df/d84/da9 0 2022-01-31T19:44:28.685 INFO:tasks.workunit.client.1.smithi171.stdout:1/538: write dd/d23/d74/d7b/f88 [747336,93721] 0 2022-01-31T19:44:28.686 INFO:tasks.workunit.client.1.smithi171.stdout:1/539: unlink dd/df/d19/l76 0 2022-01-31T19:44:28.687 INFO:tasks.workunit.client.1.smithi171.stdout:1/540: dread - dd/df/d19/d1b/d1e/d35/f7d zero size 2022-01-31T19:44:28.687 INFO:tasks.workunit.client.1.smithi171.stdout:1/541: write dd/d23/d74/d7b/f88 [1807923,87607] 0 2022-01-31T19:44:28.687 INFO:tasks.workunit.client.1.smithi171.stdout:1/542: dread - dd/d23/d2e/d5e/d81/f93 zero size 2022-01-31T19:44:28.688 INFO:tasks.workunit.client.1.smithi171.stdout:1/543: symlink dd/d23/d34/laa 0 2022-01-31T19:44:28.688 INFO:tasks.workunit.client.1.smithi171.stdout:1/544: creat dd/df/d19/d36/fab x:0 0 0 2022-01-31T19:44:28.717 INFO:tasks.workunit.client.0.smithi167.stdout:0/952: dwrite fa [4194304,4194304] 0 2022-01-31T19:44:28.718 INFO:tasks.workunit.client.1.smithi171.stdout:3/640: dwrite d4/dd/d87/d8e/f42 [0,4194304] 0 2022-01-31T19:44:28.719 INFO:tasks.workunit.client.1.smithi171.stdout:3/641: dread d4/dd/d39/fbb [0,4194304] 0 2022-01-31T19:44:28.719 INFO:tasks.workunit.client.1.smithi171.stdout:1/545: dwrite fa [0,4194304] 0 2022-01-31T19:44:28.721 INFO:tasks.workunit.client.0.smithi167.stdout:0/953: rename d17/d3e/d7e/dc7/dde/df9 to d17/d33/da4/dba/dee/d13a 0 2022-01-31T19:44:28.723 INFO:tasks.workunit.client.1.smithi171.stdout:3/642: rename d4/dd/d27/d28/d35/d55/d8f/d92/c3d to d4/dd/d27/d4e/d62/cc8 0 2022-01-31T19:44:28.724 INFO:tasks.workunit.client.0.smithi167.stdout:0/954: rename d17/d33/da4/dd0/d105/d11d/f137 to d17/d33/da4/dd0/d105/d11d/f13b 0 2022-01-31T19:44:28.725 INFO:tasks.workunit.client.1.smithi171.stdout:3/643: read d4/dd/d27/d4e/d62/f82 [2023205,21816] 0 2022-01-31T19:44:28.725 INFO:tasks.workunit.client.1.smithi171.stdout:3/644: truncate d4/dd/d27/f47 2607932 0 2022-01-31T19:44:28.726 INFO:tasks.workunit.client.0.smithi167.stdout:0/955: getdents d17/d3e/d7e/dc7/dde/df1/d113 0 2022-01-31T19:44:28.727 INFO:tasks.workunit.client.1.smithi171.stdout:3/645: creat d4/d21/d3a/d90/fc9 x:0 0 0 2022-01-31T19:44:28.728 INFO:tasks.workunit.client.0.smithi167.stdout:0/956: creat d17/d19/d1c/d39/d7b/de5/d104/f13c x:0 0 0 2022-01-31T19:44:28.728 INFO:tasks.workunit.client.0.smithi167.stdout:0/957: getdents d17/d3e/d7e/dc7/dde/df1/d113 0 2022-01-31T19:44:28.728 INFO:tasks.workunit.client.1.smithi171.stdout:2/750: sync 2022-01-31T19:44:28.729 INFO:tasks.workunit.client.0.smithi167.stdout:0/958: write d17/d19/d1c/d39/d7b/dbd/ff7 [1692186,124727] 0 2022-01-31T19:44:28.732 INFO:tasks.workunit.client.1.smithi171.stdout:3/646: symlink d4/dd/d87/d8e/lca 0 2022-01-31T19:44:28.732 INFO:tasks.workunit.client.0.smithi167.stdout:0/959: getdents d17/d19/d1c/d39/d7b/dbd/dd9 0 2022-01-31T19:44:28.733 INFO:tasks.workunit.client.0.smithi167.stdout:0/960: fsync d17/d69/f9e 0 2022-01-31T19:44:28.733 INFO:tasks.workunit.client.0.smithi167.stdout:0/961: creat d17/d33/da4/dd0/d105/d11d/f13d x:0 0 0 2022-01-31T19:44:28.734 INFO:tasks.workunit.client.1.smithi171.stdout:3/647: mkdir d4/d21/d3a/db5/dbe/dc3/dcb 0 2022-01-31T19:44:28.734 INFO:tasks.workunit.client.1.smithi171.stdout:3/648: write d4/d21/d3a/d59/f70 [2026305,80431] 0 2022-01-31T19:44:28.735 INFO:tasks.workunit.client.0.smithi167.stdout:0/962: read d17/d2c/d87/df4/f10a [1457039,73950] 0 2022-01-31T19:44:28.736 INFO:tasks.workunit.client.1.smithi171.stdout:3/649: creat d4/dd/d73/fcc x:0 0 0 2022-01-31T19:44:28.736 INFO:tasks.workunit.client.1.smithi171.stdout:3/650: creat d4/dd/d27/fcd x:0 0 0 2022-01-31T19:44:28.737 INFO:tasks.workunit.client.1.smithi171.stdout:3/651: mknod d4/dd/d27/d28/d35/d55/d8f/d92/dab/d60/cce 0 2022-01-31T19:44:28.738 INFO:tasks.workunit.client.1.smithi171.stdout:3/652: creat d4/dd/d4b/fcf x:0 0 0 2022-01-31T19:44:28.740 INFO:tasks.workunit.client.1.smithi171.stdout:3/653: mknod d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/cd0 0 2022-01-31T19:44:28.740 INFO:tasks.workunit.client.1.smithi171.stdout:3/654: truncate d4/dd/d27/d28/d35/d55/d8f/d92/dab/f2b 395915 0 2022-01-31T19:44:28.740 INFO:tasks.workunit.client.1.smithi171.stdout:3/655: chown d4/d21/l72 53684592 1 2022-01-31T19:44:28.741 INFO:tasks.workunit.client.1.smithi171.stdout:3/656: unlink d4/dd/d39/fbb 0 2022-01-31T19:44:28.742 INFO:tasks.workunit.client.1.smithi171.stdout:3/657: readlink d4/dd/d27/d28/d35/la2 0 2022-01-31T19:44:28.743 INFO:tasks.workunit.client.1.smithi171.stdout:3/658: truncate d4/dd/d39/d51/f9e 2879290 0 2022-01-31T19:44:28.743 INFO:tasks.workunit.client.0.smithi167.stdout:0/963: truncate d17/d3e/f61 854812 0 2022-01-31T19:44:28.743 INFO:tasks.workunit.client.0.smithi167.stdout:0/964: stat d17/d19/la6 0 2022-01-31T19:44:28.744 INFO:tasks.workunit.client.0.smithi167.stdout:0/965: truncate f10 2113046 0 2022-01-31T19:44:28.745 INFO:tasks.workunit.client.0.smithi167.stdout:0/966: creat d17/d19/d1c/d39/d7b/dbd/f13e x:0 0 0 2022-01-31T19:44:28.745 INFO:tasks.workunit.client.0.smithi167.stdout:0/967: truncate d17/d69/fd1 1084451 0 2022-01-31T19:44:28.745 INFO:tasks.workunit.client.0.smithi167.stdout:0/968: fsync d17/d3e/d7e/dc7/dde/df1/f102 0 2022-01-31T19:44:28.745 INFO:tasks.workunit.client.0.smithi167.stdout:0/969: chown d17/d33/da4/dd0/d105/d11d 1174 1 2022-01-31T19:44:28.746 INFO:tasks.workunit.client.0.smithi167.stdout:0/970: write d17/d3e/d7e/dc7/fd7 [1613477,7225] 0 2022-01-31T19:44:28.753 INFO:tasks.workunit.client.1.smithi171.stdout:2/751: dwrite d7/f11 [0,4194304] 0 2022-01-31T19:44:28.756 INFO:tasks.workunit.client.0.smithi167.stdout:6/969: sync 2022-01-31T19:44:28.757 INFO:tasks.workunit.client.1.smithi171.stdout:2/752: unlink d7/d9/f44 0 2022-01-31T19:44:28.758 INFO:tasks.workunit.client.1.smithi171.stdout:2/753: mkdir d7/d26/de5/ddc/df0 0 2022-01-31T19:44:28.763 INFO:tasks.workunit.client.1.smithi171.stdout:3/659: dwrite d4/dd/d87/d8e/faf [0,4194304] 0 2022-01-31T19:44:28.767 INFO:tasks.workunit.client.1.smithi171.stdout:3/660: rename d4/dd/d27/d28/da3 to d4/d21/d3a/db5/dbe/dd1 0 2022-01-31T19:44:28.774 INFO:tasks.workunit.client.1.smithi171.stdout:3/661: write d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f9b [3705490,89268] 0 2022-01-31T19:44:28.782 INFO:tasks.workunit.client.1.smithi171.stdout:2/754: dwrite d7/dd/d5e/d6b/d81/f9b [0,4194304] 0 2022-01-31T19:44:28.783 INFO:tasks.workunit.client.0.smithi167.stdout:6/970: dwrite d8/d20/d103/d108/d135/d51/d6a/f9d [0,4194304] 0 2022-01-31T19:44:28.784 INFO:tasks.workunit.client.1.smithi171.stdout:2/755: mkdir d7/d7c/d83/d85/db9/df1 0 2022-01-31T19:44:28.785 INFO:tasks.workunit.client.1.smithi171.stdout:2/756: readlink d7/d9/d1a/lc7 0 2022-01-31T19:44:28.788 INFO:tasks.workunit.client.0.smithi167.stdout:6/971: write d8/d20/d103/d108/d135/d51/d72/fc9 [890256,7233] 0 2022-01-31T19:44:28.790 INFO:tasks.workunit.client.0.smithi167.stdout:6/972: truncate d8/f9 249021 0 2022-01-31T19:44:28.791 INFO:tasks.workunit.client.1.smithi171.stdout:2/757: dread d7/d9/f12 [0,4194304] 0 2022-01-31T19:44:28.791 INFO:tasks.workunit.client.1.smithi171.stdout:2/758: dread - d7/dd/d5e/d6b/d81/d8d/da2/fac zero size 2022-01-31T19:44:28.791 INFO:tasks.workunit.client.1.smithi171.stdout:3/662: dwrite d4/dd/d27/d28/d35/d55/d8f/f95 [0,4194304] 0 2022-01-31T19:44:28.792 INFO:tasks.workunit.client.1.smithi171.stdout:2/759: symlink d7/d26/d3d/d40/dd2/lf2 0 2022-01-31T19:44:28.795 INFO:tasks.workunit.client.1.smithi171.stdout:3/663: link d4/dd/d27/f47 d4/dd/d27/d28/d35/fd2 0 2022-01-31T19:44:28.796 INFO:tasks.workunit.client.1.smithi171.stdout:2/760: mkdir d7/d55/df3 0 2022-01-31T19:44:28.796 INFO:tasks.workunit.client.1.smithi171.stdout:3/664: symlink d4/dd/d27/d6e/ld3 0 2022-01-31T19:44:28.798 INFO:tasks.workunit.client.1.smithi171.stdout:3/665: write d4/d21/d3a/f7c [76232,55209] 0 2022-01-31T19:44:28.798 INFO:tasks.workunit.client.1.smithi171.stdout:3/666: write d4/dd/d27/d28/d35/d67/fc4 [182601,52123] 0 2022-01-31T19:44:28.798 INFO:tasks.workunit.client.1.smithi171.stdout:2/761: link d7/d26/de5/fc2 d7/d26/d3d/d40/ff4 0 2022-01-31T19:44:28.799 INFO:tasks.workunit.client.1.smithi171.stdout:2/762: dread - d7/d7c/fe8 zero size 2022-01-31T19:44:28.799 INFO:tasks.workunit.client.1.smithi171.stdout:3/667: unlink d4/dd/d87/d8e/lb0 0 2022-01-31T19:44:28.802 INFO:tasks.workunit.client.1.smithi171.stdout:3/668: mkdir d4/dd/d27/d28/d35/d55/d8f/d92/dab/dd4 0 2022-01-31T19:44:28.807 INFO:tasks.workunit.client.1.smithi171.stdout:6/606: sync 2022-01-31T19:44:28.811 INFO:tasks.workunit.client.0.smithi167.stdout:6/973: dwrite d8/d20/d103/d108/d151/d64/fcd [0,4194304] 0 2022-01-31T19:44:28.815 INFO:tasks.workunit.client.0.smithi167.stdout:6/974: mknod d8/d20/d103/d108/c157 0 2022-01-31T19:44:28.818 INFO:tasks.workunit.client.0.smithi167.stdout:6/975: link d8/d20/d103/d108/d135/d4f/dd1/fec d8/d20/d54/d12e/f158 0 2022-01-31T19:44:28.819 INFO:tasks.workunit.client.1.smithi171.stdout:2/763: dwrite d7/d9/fe [0,4194304] 0 2022-01-31T19:44:28.820 INFO:tasks.workunit.client.1.smithi171.stdout:2/764: dread d7/d26/d3d/d40/f6c [0,4194304] 0 2022-01-31T19:44:28.821 INFO:tasks.workunit.client.1.smithi171.stdout:2/765: symlink d7/d26/de5/dc0/lf5 0 2022-01-31T19:44:28.822 INFO:tasks.workunit.client.1.smithi171.stdout:7/450: sync 2022-01-31T19:44:28.822 INFO:tasks.workunit.client.1.smithi171.stdout:7/451: write d1/d1e/d2a/d29/d31/f38 [398785,76552] 0 2022-01-31T19:44:28.823 INFO:tasks.workunit.client.0.smithi167.stdout:6/976: read d8/d20/d103/d108/d151/d64/dab/f102 [1178248,122481] 0 2022-01-31T19:44:28.824 INFO:tasks.workunit.client.1.smithi171.stdout:2/766: mknod d7/dd/dea/cf6 0 2022-01-31T19:44:28.824 INFO:tasks.workunit.client.1.smithi171.stdout:2/767: read d7/d26/f5f [1914426,15751] 0 2022-01-31T19:44:28.825 INFO:tasks.workunit.client.1.smithi171.stdout:2/768: readlink d7/d55/l60 0 2022-01-31T19:44:28.826 INFO:tasks.workunit.client.1.smithi171.stdout:2/769: write d7/d26/f92 [467355,36410] 0 2022-01-31T19:44:28.826 INFO:tasks.workunit.client.0.smithi167.stdout:6/977: mknod d8/d20/d103/d108/d151/d64/c159 0 2022-01-31T19:44:28.826 INFO:tasks.workunit.client.0.smithi167.stdout:6/978: read d8/d20/d103/d108/d151/d64/dab/fe3 [6301170,1318] 0 2022-01-31T19:44:28.827 INFO:tasks.workunit.client.0.smithi167.stdout:6/979: fdatasync d8/d20/d103/d108/d151/d64/dab/f102 0 2022-01-31T19:44:28.827 INFO:tasks.workunit.client.1.smithi171.stdout:2/770: write d7/d26/f92 [710526,130455] 0 2022-01-31T19:44:28.828 INFO:tasks.workunit.client.1.smithi171.stdout:7/452: creat d1/d1e/d2a/d29/d31/f99 x:0 0 0 2022-01-31T19:44:28.829 INFO:tasks.workunit.client.1.smithi171.stdout:2/771: getdents d7/d26/d3d 0 2022-01-31T19:44:28.832 INFO:tasks.workunit.client.1.smithi171.stdout:6/607: dwrite dc/d19/d29/d6a/fb4 [0,4194304] 0 2022-01-31T19:44:28.832 INFO:tasks.workunit.client.1.smithi171.stdout:6/608: fsync dc/d21/f8d 0 2022-01-31T19:44:28.832 INFO:tasks.workunit.client.1.smithi171.stdout:6/609: truncate dc/d19/d29/d6a/fbb 919717 0 2022-01-31T19:44:28.833 INFO:tasks.workunit.client.1.smithi171.stdout:8/497: sync 2022-01-31T19:44:28.834 INFO:tasks.workunit.client.1.smithi171.stdout:1/546: sync 2022-01-31T19:44:28.834 INFO:tasks.workunit.client.1.smithi171.stdout:8/498: readlink df/l74 0 2022-01-31T19:44:28.834 INFO:tasks.workunit.client.1.smithi171.stdout:5/519: sync 2022-01-31T19:44:28.834 INFO:tasks.workunit.client.1.smithi171.stdout:5/520: chown d5/d6/dd/l52 21313 1 2022-01-31T19:44:28.835 INFO:tasks.workunit.client.1.smithi171.stdout:5/521: dread - d5/d6/d15/d23/d6b/f7e zero size 2022-01-31T19:44:28.835 INFO:tasks.workunit.client.1.smithi171.stdout:5/522: chown d5/d6/f21 238264 1 2022-01-31T19:44:28.835 INFO:tasks.workunit.client.1.smithi171.stdout:0/621: sync 2022-01-31T19:44:28.836 INFO:tasks.workunit.client.0.smithi167.stdout:6/980: dread d8/f1a [0,4194304] 0 2022-01-31T19:44:28.837 INFO:tasks.workunit.client.1.smithi171.stdout:9/616: sync 2022-01-31T19:44:28.837 INFO:tasks.workunit.client.1.smithi171.stdout:9/617: creat dd/df/d15/d1a/d84/fcd x:0 0 0 2022-01-31T19:44:28.837 INFO:tasks.workunit.client.1.smithi171.stdout:8/499: symlink df/lac 0 2022-01-31T19:44:28.837 INFO:tasks.workunit.client.1.smithi171.stdout:8/500: fdatasync df/d18/d19/d36/d70/f9e 0 2022-01-31T19:44:28.838 INFO:tasks.workunit.client.1.smithi171.stdout:5/523: getdents d5/d6/dd 0 2022-01-31T19:44:28.839 INFO:tasks.workunit.client.1.smithi171.stdout:5/524: chown d5/d6/d15/d23/d3d/l63 1034 1 2022-01-31T19:44:28.839 INFO:tasks.workunit.client.1.smithi171.stdout:5/525: creat d5/d6/d15/d23/d3d/fb2 x:0 0 0 2022-01-31T19:44:28.839 INFO:tasks.workunit.client.1.smithi171.stdout:4/500: sync 2022-01-31T19:44:28.839 INFO:tasks.workunit.client.1.smithi171.stdout:4/501: chown d2/d3/d5/dc/d4b/d56/c9a 15 1 2022-01-31T19:44:28.840 INFO:tasks.workunit.client.1.smithi171.stdout:4/502: readlink d2/d3/d5/d70/d7c/la3 0 2022-01-31T19:44:28.840 INFO:tasks.workunit.client.1.smithi171.stdout:0/622: rename d9/db/dc2/l54 to d9/db/dc/d3f/le4 0 2022-01-31T19:44:28.841 INFO:tasks.workunit.client.1.smithi171.stdout:0/623: read - d9/db/d59/d71/dbe/fd8 zero size 2022-01-31T19:44:28.841 INFO:tasks.workunit.client.1.smithi171.stdout:2/772: dread d7/d26/de5/fb1 [0,4194304] 0 2022-01-31T19:44:28.841 INFO:tasks.workunit.client.1.smithi171.stdout:2/773: fsync d7/dd/d5e/d6b/d81/fad 0 2022-01-31T19:44:28.842 INFO:tasks.workunit.client.1.smithi171.stdout:9/618: creat dd/df/d15/d1a/dcb/fce x:0 0 0 2022-01-31T19:44:28.842 INFO:tasks.workunit.client.1.smithi171.stdout:9/619: fdatasync dd/df/d3f/f9e 0 2022-01-31T19:44:28.845 INFO:tasks.workunit.client.1.smithi171.stdout:4/503: creat d2/d7f/fa8 x:0 0 0 2022-01-31T19:44:28.846 INFO:tasks.workunit.client.0.smithi167.stdout:6/981: dread d8/d20/d103/d108/d151/d64/dab/f102 [0,4194304] 0 2022-01-31T19:44:28.848 INFO:tasks.workunit.client.1.smithi171.stdout:8/501: dread df/d18/d19/d21/f32 [0,4194304] 0 2022-01-31T19:44:28.849 INFO:tasks.workunit.client.1.smithi171.stdout:8/502: truncate df/d18/d19/d21/dab/f56 8883972 0 2022-01-31T19:44:28.849 INFO:tasks.workunit.client.1.smithi171.stdout:8/503: fsync df/d1a/f53 0 2022-01-31T19:44:28.851 INFO:tasks.workunit.client.1.smithi171.stdout:2/774: getdents d7 0 2022-01-31T19:44:28.852 INFO:tasks.workunit.client.1.smithi171.stdout:2/775: dread - d7/d7c/d83/d85/fd6 zero size 2022-01-31T19:44:28.852 INFO:tasks.workunit.client.1.smithi171.stdout:2/776: chown d7/d9/l9a 1863532786 1 2022-01-31T19:44:28.853 INFO:tasks.workunit.client.1.smithi171.stdout:0/624: dread d9/d24/f4d [0,4194304] 0 2022-01-31T19:44:28.853 INFO:tasks.workunit.client.1.smithi171.stdout:8/504: mkdir df/d18/dad 0 2022-01-31T19:44:28.853 INFO:tasks.workunit.client.1.smithi171.stdout:8/505: chown df/d18/f42 238 1 2022-01-31T19:44:28.854 INFO:tasks.workunit.client.1.smithi171.stdout:5/526: rmdir d5/d59 39 2022-01-31T19:44:28.855 INFO:tasks.workunit.client.1.smithi171.stdout:5/527: write d5/d6/dd/f35 [2673806,48783] 0 2022-01-31T19:44:28.855 INFO:tasks.workunit.client.1.smithi171.stdout:0/625: symlink d9/db/d59/le5 0 2022-01-31T19:44:28.855 INFO:tasks.workunit.client.1.smithi171.stdout:0/626: write d9/db/f47 [452235,70819] 0 2022-01-31T19:44:28.856 INFO:tasks.workunit.client.1.smithi171.stdout:0/627: write d9/db/dc2/fad [577461,126656] 0 2022-01-31T19:44:28.857 INFO:tasks.workunit.client.1.smithi171.stdout:0/628: creat d9/db/d59/fe6 x:0 0 0 2022-01-31T19:44:28.862 INFO:tasks.workunit.client.1.smithi171.stdout:0/629: link d9/d24/d92/f69 d9/db/dc/d3f/db9/d7a/fe7 0 2022-01-31T19:44:28.862 INFO:tasks.workunit.client.1.smithi171.stdout:0/630: write d9/db/dc/d3f/db9/d7a/d88/fa3 [992656,26992] 0 2022-01-31T19:44:28.862 INFO:tasks.workunit.client.1.smithi171.stdout:0/631: chown d9/d24/d4c/d35/f8d 29066 1 2022-01-31T19:44:28.863 INFO:tasks.workunit.client.1.smithi171.stdout:0/632: readlink d9/db/l4e 0 2022-01-31T19:44:28.869 INFO:tasks.workunit.client.1.smithi171.stdout:7/453: dwrite d1/d1e/d2a/d29/d31/d30/f73 [4194304,4194304] 0 2022-01-31T19:44:28.874 INFO:tasks.workunit.client.1.smithi171.stdout:1/547: dwrite dd/d23/d2e/d5e/d81/f93 [0,4194304] 0 2022-01-31T19:44:28.874 INFO:tasks.workunit.client.1.smithi171.stdout:7/454: mknod d1/d1e/d2a/d29/d77/c9a 0 2022-01-31T19:44:28.875 INFO:tasks.workunit.client.0.smithi167.stdout:6/982: rename d8/d20/d103/d108/d135/d4f to d8/d20/d54/d12e/d15a 0 2022-01-31T19:44:28.876 INFO:tasks.workunit.client.1.smithi171.stdout:1/548: getdents dd/d23/d2e/d5e/d83 0 2022-01-31T19:44:28.876 INFO:tasks.workunit.client.1.smithi171.stdout:1/549: chown dd/df/d19 3 1 2022-01-31T19:44:28.877 INFO:tasks.workunit.client.1.smithi171.stdout:1/550: chown dd/d23/d2e/d5e/d81/f93 5300 1 2022-01-31T19:44:28.877 INFO:tasks.workunit.client.1.smithi171.stdout:7/455: creat d1/d1e/d2a/f9b x:0 0 0 2022-01-31T19:44:28.878 INFO:tasks.workunit.client.1.smithi171.stdout:7/456: chown d1/d52/l27 238620 1 2022-01-31T19:44:28.880 INFO:tasks.workunit.client.1.smithi171.stdout:7/457: dread d1/d1e/d2a/d29/f6d [0,4194304] 0 2022-01-31T19:44:28.880 INFO:tasks.workunit.client.0.smithi167.stdout:6/983: write d8/d20/d103/d108/d151/f6e [2727428,112011] 0 2022-01-31T19:44:28.885 INFO:tasks.workunit.client.1.smithi171.stdout:7/458: write d1/d1e/d2a/d29/d31/d30/f3f [4007008,104059] 0 2022-01-31T19:44:28.886 INFO:tasks.workunit.client.1.smithi171.stdout:7/459: chown d1/c93 83 1 2022-01-31T19:44:28.887 INFO:tasks.workunit.client.0.smithi167.stdout:6/984: dread d8/d20/d103/d108/d135/d51/d137/f96 [0,4194304] 0 2022-01-31T19:44:28.889 INFO:tasks.workunit.client.0.smithi167.stdout:6/985: symlink d8/d20/d103/d108/d135/d51/d7e/l15b 0 2022-01-31T19:44:28.890 INFO:tasks.workunit.client.1.smithi171.stdout:4/504: dwrite d2/f9e [0,4194304] 0 2022-01-31T19:44:28.891 INFO:tasks.workunit.client.1.smithi171.stdout:9/620: dwrite dd/df/d15/d38/d9c/fba [0,4194304] 0 2022-01-31T19:44:28.891 INFO:tasks.workunit.client.1.smithi171.stdout:9/621: chown dd/d1c/l4f 58619 1 2022-01-31T19:44:28.892 INFO:tasks.workunit.client.1.smithi171.stdout:4/505: creat d2/d3/d5/d41/d6b/fa9 x:0 0 0 2022-01-31T19:44:28.894 INFO:tasks.workunit.client.1.smithi171.stdout:9/622: truncate f9 1496240 0 2022-01-31T19:44:28.895 INFO:tasks.workunit.client.1.smithi171.stdout:4/506: mknod d2/d3/d5/d91/da6/caa 0 2022-01-31T19:44:28.896 INFO:tasks.workunit.client.1.smithi171.stdout:4/507: fsync d2/f9e 0 2022-01-31T19:44:28.896 INFO:tasks.workunit.client.1.smithi171.stdout:4/508: read - d2/f6d zero size 2022-01-31T19:44:28.896 INFO:tasks.workunit.client.0.smithi167.stdout:6/986: dread d8/d20/d54/fa7 [0,4194304] 0 2022-01-31T19:44:28.901 INFO:tasks.workunit.client.1.smithi171.stdout:2/777: dwrite d7/dd/d5e/d6b/d81/f9b [0,4194304] 0 2022-01-31T19:44:28.902 INFO:tasks.workunit.client.1.smithi171.stdout:2/778: dread - d7/d26/de5/dc0/fe3 zero size 2022-01-31T19:44:28.902 INFO:tasks.workunit.client.1.smithi171.stdout:2/779: truncate d7/d26/d3d/d40/f57 682232 0 2022-01-31T19:44:28.903 INFO:tasks.workunit.client.1.smithi171.stdout:5/528: dwrite d5/d6/f21 [0,4194304] 0 2022-01-31T19:44:28.904 INFO:tasks.workunit.client.1.smithi171.stdout:2/780: mknod d7/d7c/d83/d85/db9/cf7 0 2022-01-31T19:44:28.904 INFO:tasks.workunit.client.1.smithi171.stdout:2/781: truncate d7/d9/f6a 4229188 0 2022-01-31T19:44:28.905 INFO:tasks.workunit.client.0.smithi167.stdout:6/987: write d8/f1c [3972733,38818] 0 2022-01-31T19:44:28.907 INFO:tasks.workunit.client.1.smithi171.stdout:5/529: link d5/f55 d5/d6/fb3 0 2022-01-31T19:44:28.907 INFO:tasks.workunit.client.0.smithi167.stdout:6/988: rename d8/c10 to d8/d20/d103/d108/d135/d51/d137/c15c 0 2022-01-31T19:44:28.909 INFO:tasks.workunit.client.1.smithi171.stdout:0/633: dwrite d9/db/dc/f8b [0,4194304] 0 2022-01-31T19:44:28.909 INFO:tasks.workunit.client.1.smithi171.stdout:2/782: getdents d7/dd/d5e/d6b 0 2022-01-31T19:44:28.910 INFO:tasks.workunit.client.1.smithi171.stdout:5/530: chown d5/d6/d15/l31 1632049460 1 2022-01-31T19:44:28.910 INFO:tasks.workunit.client.0.smithi167.stdout:6/989: link d8/d20/d54/fa7 d8/d20/d103/d108/d135/d51/d137/deb/d111/f15d 0 2022-01-31T19:44:28.911 INFO:tasks.workunit.client.1.smithi171.stdout:0/634: mknod d9/d24/d92/d9d/ce8 0 2022-01-31T19:44:28.912 INFO:tasks.workunit.client.1.smithi171.stdout:0/635: write d9/db/d59/d71/dbe/fd8 [902581,89992] 0 2022-01-31T19:44:28.912 INFO:tasks.workunit.client.1.smithi171.stdout:0/636: write d9/f3b [3334379,103501] 0 2022-01-31T19:44:28.913 INFO:tasks.workunit.client.1.smithi171.stdout:2/783: symlink d7/d26/de5/ddc/df0/lf8 0 2022-01-31T19:44:28.913 INFO:tasks.workunit.client.0.smithi167.stdout:6/990: rename d8/d20/d103/d108/d135/d51/d6a/fa3 to d8/d20/d103/d108/d135/d51/d6a/d139/f15e 0 2022-01-31T19:44:28.913 INFO:tasks.workunit.client.0.smithi167.stdout:6/991: read d8/d20/d103/d108/d135/d51/d72/fc9 [2987653,15715] 0 2022-01-31T19:44:28.914 INFO:tasks.workunit.client.0.smithi167.stdout:6/992: write d8/d20/f26 [5002997,68697] 0 2022-01-31T19:44:28.914 INFO:tasks.workunit.client.1.smithi171.stdout:1/551: dwrite dd/d23/d2e/d5e/f20 [0,4194304] 0 2022-01-31T19:44:28.915 INFO:tasks.workunit.client.1.smithi171.stdout:5/531: symlink d5/d6/d15/d23/d58/d9e/lb4 0 2022-01-31T19:44:28.915 INFO:tasks.workunit.client.1.smithi171.stdout:7/460: dwrite d1/d1e/d2a/d29/d31/d30/f73 [4194304,4194304] 0 2022-01-31T19:44:28.916 INFO:tasks.workunit.client.1.smithi171.stdout:0/637: mkdir d9/db/d59/de9 0 2022-01-31T19:44:28.916 INFO:tasks.workunit.client.1.smithi171.stdout:0/638: truncate d9/fd4 217626 0 2022-01-31T19:44:28.918 INFO:tasks.workunit.client.1.smithi171.stdout:1/552: symlink dd/d23/d2e/d5e/d83/lac 0 2022-01-31T19:44:28.918 INFO:tasks.workunit.client.1.smithi171.stdout:1/553: dread - dd/d23/d2e/d5e/d81/fa3 zero size 2022-01-31T19:44:28.922 INFO:tasks.workunit.client.1.smithi171.stdout:7/461: unlink d1/d52/f1a 0 2022-01-31T19:44:28.922 INFO:tasks.workunit.client.1.smithi171.stdout:9/623: dwrite dd/df/d15/d38/d41/fb4 [0,4194304] 0 2022-01-31T19:44:28.923 INFO:tasks.workunit.client.1.smithi171.stdout:0/639: mknod d9/cea 0 2022-01-31T19:44:28.924 INFO:tasks.workunit.client.1.smithi171.stdout:1/554: mkdir dd/d23/d2e/d5e/d81/d98/dad 0 2022-01-31T19:44:28.925 INFO:tasks.workunit.client.1.smithi171.stdout:4/509: dwrite d2/d3/d5/f46 [0,4194304] 0 2022-01-31T19:44:28.925 INFO:tasks.workunit.client.1.smithi171.stdout:4/510: chown d2/d3/d5/d91/d7e/f27 4456095 1 2022-01-31T19:44:28.926 INFO:tasks.workunit.client.1.smithi171.stdout:7/462: truncate d1/d1e/d2a/d29/d31/f38 166382 0 2022-01-31T19:44:28.932 INFO:tasks.workunit.client.1.smithi171.stdout:9/624: mkdir dd/d1c/d7c/dcf 0 2022-01-31T19:44:28.936 INFO:tasks.workunit.client.1.smithi171.stdout:2/784: dread d7/d26/d3d/d40/f6e [0,4194304] 0 2022-01-31T19:44:28.936 INFO:tasks.workunit.client.1.smithi171.stdout:2/785: chown d7/d55 11587463 1 2022-01-31T19:44:28.936 INFO:tasks.workunit.client.1.smithi171.stdout:2/786: write d7/dd/d5e/d6b/d81/fad [137721,118922] 0 2022-01-31T19:44:28.941 INFO:tasks.workunit.client.1.smithi171.stdout:4/511: rename d2/d3/d5/d91/d7e/d12/d15/f24 to d2/d3/d5/d91/fab 0 2022-01-31T19:44:28.942 INFO:tasks.workunit.client.1.smithi171.stdout:4/512: write d2/d3/d5/d91/d77/f8f [1016659,90439] 0 2022-01-31T19:44:28.943 INFO:tasks.workunit.client.1.smithi171.stdout:9/625: creat dd/df/d15/d1a/d64/fd0 x:0 0 0 2022-01-31T19:44:28.944 INFO:tasks.workunit.client.1.smithi171.stdout:0/640: rmdir d9/db/d59/d71 39 2022-01-31T19:44:28.945 INFO:tasks.workunit.client.1.smithi171.stdout:2/787: symlink d7/d26/de5/ddc/df0/lf9 0 2022-01-31T19:44:28.946 INFO:tasks.workunit.client.1.smithi171.stdout:4/513: truncate d2/d3/d5/d91/d7e/f27 3490850 0 2022-01-31T19:44:28.947 INFO:tasks.workunit.client.1.smithi171.stdout:7/463: symlink d1/d1e/d2a/d29/d31/d30/d3c/l9c 0 2022-01-31T19:44:28.947 INFO:tasks.workunit.client.1.smithi171.stdout:9/626: creat dd/df/d3f/fd1 x:0 0 0 2022-01-31T19:44:28.948 INFO:tasks.workunit.client.1.smithi171.stdout:9/627: fdatasync dd/d1c/da8/fc3 0 2022-01-31T19:44:28.948 INFO:tasks.workunit.client.1.smithi171.stdout:0/641: symlink d9/db/d22/leb 0 2022-01-31T19:44:28.949 INFO:tasks.workunit.client.0.smithi167.stdout:6/993: dwrite d8/d20/d54/d12e/d15a/dd1/fec [0,4194304] 0 2022-01-31T19:44:28.949 INFO:tasks.workunit.client.1.smithi171.stdout:2/788: symlink d7/dd/d5e/d84/lfa 0 2022-01-31T19:44:28.950 INFO:tasks.workunit.client.0.smithi167.stdout:6/994: truncate d8/d20/d103/d108/d135/d51/d72/fd6 1596593 0 2022-01-31T19:44:28.951 INFO:tasks.workunit.client.1.smithi171.stdout:1/555: dwrite dd/f91 [0,4194304] 0 2022-01-31T19:44:28.953 INFO:tasks.workunit.client.1.smithi171.stdout:4/514: truncate d2/f13 1993869 0 2022-01-31T19:44:28.953 INFO:tasks.workunit.client.1.smithi171.stdout:4/515: chown d2/d3/d5/d91/d7e/d12/f2a 2575 1 2022-01-31T19:44:28.953 INFO:tasks.workunit.client.1.smithi171.stdout:9/628: read dd/df/f17 [1913876,129645] 0 2022-01-31T19:44:28.954 INFO:tasks.workunit.client.1.smithi171.stdout:1/556: dread dd/df/f38 [0,4194304] 0 2022-01-31T19:44:28.954 INFO:tasks.workunit.client.1.smithi171.stdout:1/557: fsync dd/d23/f57 0 2022-01-31T19:44:28.954 INFO:tasks.workunit.client.1.smithi171.stdout:7/464: mknod d1/d1e/d2a/d29/d77/c9d 0 2022-01-31T19:44:28.955 INFO:tasks.workunit.client.1.smithi171.stdout:0/642: unlink d9/db/dc2/fc5 0 2022-01-31T19:44:28.958 INFO:tasks.workunit.client.1.smithi171.stdout:5/532: dwrite d5/d6/d15/f64 [0,4194304] 0 2022-01-31T19:44:28.958 INFO:tasks.workunit.client.1.smithi171.stdout:5/533: creat d5/fb5 x:0 0 0 2022-01-31T19:44:28.960 INFO:tasks.workunit.client.0.smithi167.stdout:0/971: sync 2022-01-31T19:44:28.961 INFO:tasks.workunit.client.0.smithi167.stdout:6/995: dread d8/d20/d103/d108/d135/d51/d6a/d139/f15e [0,4194304] 0 2022-01-31T19:44:28.961 INFO:tasks.workunit.client.0.smithi167.stdout:6/996: chown d8/c45 12933598 1 2022-01-31T19:44:28.962 INFO:tasks.workunit.client.1.smithi171.stdout:9/629: creat dd/df/d15/d1a/d64/d6e/fd2 x:0 0 0 2022-01-31T19:44:28.962 INFO:tasks.workunit.client.1.smithi171.stdout:9/630: creat dd/df/d15/d1a/d64/fd3 x:0 0 0 2022-01-31T19:44:28.963 INFO:tasks.workunit.client.1.smithi171.stdout:9/631: read - dd/df/d15/d1a/d64/fac zero size 2022-01-31T19:44:28.963 INFO:tasks.workunit.client.1.smithi171.stdout:4/516: dread d2/d3/d5/d41/f89 [0,4194304] 0 2022-01-31T19:44:28.964 INFO:tasks.workunit.client.0.smithi167.stdout:0/972: symlink d17/d3e/d7e/dc7/dde/l13f 0 2022-01-31T19:44:28.965 INFO:tasks.workunit.client.0.smithi167.stdout:0/973: creat d17/d19/d1c/d39/d7b/de5/d104/f140 x:0 0 0 2022-01-31T19:44:28.965 INFO:tasks.workunit.client.0.smithi167.stdout:0/974: write d17/d19/d1c/d39/ffc [730010,126815] 0 2022-01-31T19:44:28.965 INFO:tasks.workunit.client.0.smithi167.stdout:6/997: rmdir d8/d20/d103/d108/d135/d51/d72 39 2022-01-31T19:44:28.966 INFO:tasks.workunit.client.0.smithi167.stdout:6/998: chown d8/d20/d103/d108/d135/d35/f150 4 1 2022-01-31T19:44:28.967 INFO:tasks.workunit.client.0.smithi167.stdout:6/999: read d8/d20/d103/d108/d151/d86/fb6 [48757,34589] 0 2022-01-31T19:44:28.967 INFO:tasks.workunit.client.1.smithi171.stdout:0/643: mkdir d9/db/dc/d3f/db9/dec 0 2022-01-31T19:44:28.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/644: creat d9/db/d59/fed x:0 0 0 2022-01-31T19:44:28.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/645: chown d9/db/l15 1845 1 2022-01-31T19:44:28.968 INFO:tasks.workunit.client.1.smithi171.stdout:1/558: dread f9 [0,4194304] 0 2022-01-31T19:44:28.969 INFO:tasks.workunit.client.1.smithi171.stdout:1/559: chown dd/d23/f3f 8101725 1 2022-01-31T19:44:28.969 INFO:tasks.workunit.client.1.smithi171.stdout:7/465: mknod d1/d1e/d2a/d29/d31/d30/d6b/c9e 0 2022-01-31T19:44:28.969 INFO:tasks.workunit.client.1.smithi171.stdout:7/466: chown d1/d1b/f4c 208 1 2022-01-31T19:44:28.970 INFO:tasks.workunit.client.1.smithi171.stdout:2/789: dwrite d7/d26/f92 [0,4194304] 0 2022-01-31T19:44:28.970 INFO:tasks.workunit.client.0.smithi167.stdout:0/975: rename d17/d19/d1c/d39/d7b/da9 to d17/d19/d1c/de1/d141 0 2022-01-31T19:44:28.971 INFO:tasks.workunit.client.0.smithi167.stdout:0/976: truncate d17/d3e/f68 207490 0 2022-01-31T19:44:28.971 INFO:tasks.workunit.client.0.smithi167.stdout:0/977: truncate d17/d2c/d87/f8d 1367944 0 2022-01-31T19:44:28.972 INFO:tasks.workunit.client.0.smithi167.stdout:0/978: unlink d17/d69/f94 0 2022-01-31T19:44:28.973 INFO:tasks.workunit.client.0.smithi167.stdout:0/979: creat d17/d19/d1c/d39/d7b/de5/f142 x:0 0 0 2022-01-31T19:44:28.974 INFO:tasks.workunit.client.1.smithi171.stdout:2/790: dread d7/d26/d3d/d40/f57 [0,4194304] 0 2022-01-31T19:44:28.974 INFO:tasks.workunit.client.1.smithi171.stdout:2/791: chown d7/d55/c91 0 1 2022-01-31T19:44:28.977 INFO:tasks.workunit.client.1.smithi171.stdout:9/632: creat dd/d1c/d7c/dcf/fd4 x:0 0 0 2022-01-31T19:44:28.977 INFO:tasks.workunit.client.1.smithi171.stdout:4/517: chown d2/d3/c3d 28829 1 2022-01-31T19:44:28.978 INFO:tasks.workunit.client.1.smithi171.stdout:2/792: dread d7/dd/fa1 [0,4194304] 0 2022-01-31T19:44:28.978 INFO:tasks.workunit.client.1.smithi171.stdout:2/793: chown d7/d9/d1a/d45 1454533623 1 2022-01-31T19:44:28.978 INFO:tasks.workunit.client.1.smithi171.stdout:2/794: stat d7/d55/f59 0 2022-01-31T19:44:28.982 INFO:tasks.workunit.client.1.smithi171.stdout:0/646: link d9/db/d22/f5b d9/db/fee 0 2022-01-31T19:44:28.986 INFO:tasks.workunit.client.1.smithi171.stdout:7/467: mknod d1/d52/c9f 0 2022-01-31T19:44:28.986 INFO:tasks.workunit.client.1.smithi171.stdout:7/468: truncate d1/d1e/d2a/f9b 89424 0 2022-01-31T19:44:28.989 INFO:tasks.workunit.client.1.smithi171.stdout:9/633: creat dd/df/d15/d38/d4d/d80/fd5 x:0 0 0 2022-01-31T19:44:28.991 INFO:tasks.workunit.client.1.smithi171.stdout:3/669: sync 2022-01-31T19:44:28.992 INFO:tasks.workunit.client.1.smithi171.stdout:6/610: sync 2022-01-31T19:44:28.993 INFO:tasks.workunit.client.0.smithi167.stdout:0/980: dwrite d17/d19/d1c/d39/d44/f65 [0,4194304] 0 2022-01-31T19:44:28.993 INFO:tasks.workunit.client.1.smithi171.stdout:4/518: symlink d2/d3/d5/d3b/lac 0 2022-01-31T19:44:28.994 INFO:tasks.workunit.client.1.smithi171.stdout:8/506: sync 2022-01-31T19:44:28.994 INFO:tasks.workunit.client.1.smithi171.stdout:2/795: truncate d7/d7c/fd3 65852 0 2022-01-31T19:44:28.994 INFO:tasks.workunit.client.1.smithi171.stdout:8/507: read df/d18/d79/f8c [3314022,26384] 0 2022-01-31T19:44:28.995 INFO:tasks.workunit.client.1.smithi171.stdout:2/796: write d7/d26/d3d/d40/f57 [1645150,112815] 0 2022-01-31T19:44:28.995 INFO:tasks.workunit.client.1.smithi171.stdout:8/508: fsync df/d18/d19/d46/f8d 0 2022-01-31T19:44:28.995 INFO:tasks.workunit.client.1.smithi171.stdout:1/560: getdents dd/d23/d34/d4b 0 2022-01-31T19:44:28.998 INFO:tasks.workunit.client.1.smithi171.stdout:0/647: creat d9/db/d59/d71/dbe/fef x:0 0 0 2022-01-31T19:44:28.998 INFO:tasks.workunit.client.1.smithi171.stdout:0/648: chown d9/fe0 673 1 2022-01-31T19:44:28.999 INFO:tasks.workunit.client.0.smithi167.stdout:0/981: rename d17/d2c/d87/f126 to d17/d69/f143 0 2022-01-31T19:44:28.999 INFO:tasks.workunit.client.0.smithi167.stdout:0/982: truncate d17/d19/f51 405578 0 2022-01-31T19:44:29.001 INFO:tasks.workunit.client.1.smithi171.stdout:7/469: creat d1/d26/d47/d8f/fa0 x:0 0 0 2022-01-31T19:44:29.001 INFO:tasks.workunit.client.1.smithi171.stdout:7/470: readlink d1/d52/l27 0 2022-01-31T19:44:29.001 INFO:tasks.workunit.client.1.smithi171.stdout:7/471: fdatasync d1/f36 0 2022-01-31T19:44:29.002 INFO:tasks.workunit.client.1.smithi171.stdout:7/472: chown d1/d1e/d2a/d29/d31/d30/d6b/c7d 69868652 1 2022-01-31T19:44:29.002 INFO:tasks.workunit.client.1.smithi171.stdout:7/473: read d1/d1e/d2a/d29/d31/d30/d3c/f56 [754246,771] 0 2022-01-31T19:44:29.004 INFO:tasks.workunit.client.1.smithi171.stdout:9/634: symlink dd/df/d3f/d73/d96/d9f/ld6 0 2022-01-31T19:44:29.004 INFO:tasks.workunit.client.1.smithi171.stdout:9/635: creat dd/df/d15/d38/d41/fd7 x:0 0 0 2022-01-31T19:44:29.005 INFO:tasks.workunit.client.0.smithi167.stdout:0/983: rmdir d17/d19 39 2022-01-31T19:44:29.005 INFO:tasks.workunit.client.1.smithi171.stdout:9/636: write dd/f3c [3974123,11767] 0 2022-01-31T19:44:29.007 INFO:tasks.workunit.client.1.smithi171.stdout:3/670: creat d4/d21/d3a/db5/fd5 x:0 0 0 2022-01-31T19:44:29.008 INFO:tasks.workunit.client.1.smithi171.stdout:6/611: symlink dc/d1e/d3f/lc9 0 2022-01-31T19:44:29.009 INFO:tasks.workunit.client.1.smithi171.stdout:6/612: write dc/d19/d29/da7/fb8 [372992,13573] 0 2022-01-31T19:44:29.009 INFO:tasks.workunit.client.1.smithi171.stdout:6/613: write dc/d19/d30/f92 [69788,76932] 0 2022-01-31T19:44:29.010 INFO:tasks.workunit.client.1.smithi171.stdout:4/519: unlink d2/d3/d5/d41/f89 0 2022-01-31T19:44:29.014 INFO:tasks.workunit.client.1.smithi171.stdout:2/797: rename d7/d7c/c7d to d7/d26/d63/d71/cfb 0 2022-01-31T19:44:29.015 INFO:tasks.workunit.client.1.smithi171.stdout:1/561: rmdir dd/df/d19/d1b/d1e/d64 39 2022-01-31T19:44:29.017 INFO:tasks.workunit.client.1.smithi171.stdout:2/798: read d7/d26/d3d/d40/f4f [4183134,35138] 0 2022-01-31T19:44:29.019 INFO:tasks.workunit.client.1.smithi171.stdout:0/649: symlink d9/lf0 0 2022-01-31T19:44:29.019 INFO:tasks.workunit.client.1.smithi171.stdout:7/474: truncate d1/d26/d47/f4f 166086 0 2022-01-31T19:44:29.022 INFO:tasks.workunit.client.1.smithi171.stdout:1/562: write fa [2565600,119931] 0 2022-01-31T19:44:29.023 INFO:tasks.workunit.client.1.smithi171.stdout:3/671: unlink d4/c12 0 2022-01-31T19:44:29.025 INFO:tasks.workunit.client.1.smithi171.stdout:0/650: creat d9/d24/d92/d9d/ff1 x:0 0 0 2022-01-31T19:44:29.025 INFO:tasks.workunit.client.1.smithi171.stdout:0/651: stat d9/d24/f4d 0 2022-01-31T19:44:29.025 INFO:tasks.workunit.client.1.smithi171.stdout:0/652: stat d9/d24/d4c/d7d/lb2 0 2022-01-31T19:44:29.026 INFO:tasks.workunit.client.1.smithi171.stdout:7/475: symlink d1/d1e/d2a/d29/d77/la1 0 2022-01-31T19:44:29.026 INFO:tasks.workunit.client.1.smithi171.stdout:7/476: fsync d1/d52/f28 0 2022-01-31T19:44:29.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/637: symlink dd/ld8 0 2022-01-31T19:44:29.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/638: fdatasync dd/df/d15/d1a/d64/d6e/fd2 0 2022-01-31T19:44:29.029 INFO:tasks.workunit.client.1.smithi171.stdout:3/672: mknod d4/d21/d3a/db5/dbe/dd1/cd6 0 2022-01-31T19:44:29.031 INFO:tasks.workunit.client.1.smithi171.stdout:2/799: getdents d7/dd/d5e/d6b/d81 0 2022-01-31T19:44:29.031 INFO:tasks.workunit.client.1.smithi171.stdout:2/800: fdatasync d7/d9/f6a 0 2022-01-31T19:44:29.034 INFO:tasks.workunit.client.1.smithi171.stdout:9/639: creat dd/df/d15/da2/fd9 x:0 0 0 2022-01-31T19:44:29.035 INFO:tasks.workunit.client.1.smithi171.stdout:0/653: rename d9/l18 to d9/db/dc/d3f/d6f/dcc/lf2 0 2022-01-31T19:44:29.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/984: dwrite d17/d19/f51 [0,4194304] 0 2022-01-31T19:44:29.036 INFO:tasks.workunit.client.0.smithi167.stdout:0/985: write d17/d3e/fae [4601274,92987] 0 2022-01-31T19:44:29.037 INFO:tasks.workunit.client.1.smithi171.stdout:6/614: dwrite dc/d19/d30/d3c/fc7 [0,4194304] 0 2022-01-31T19:44:29.038 INFO:tasks.workunit.client.1.smithi171.stdout:4/520: dwrite d2/d3/d5/dc/f6f [4194304,4194304] 0 2022-01-31T19:44:29.038 INFO:tasks.workunit.client.1.smithi171.stdout:4/521: chown d2/d3/d5/d91/d7e/f8c 1980165 1 2022-01-31T19:44:29.038 INFO:tasks.workunit.client.1.smithi171.stdout:9/640: link l8 dd/df/d15/d1a/d64/lda 0 2022-01-31T19:44:29.038 INFO:tasks.workunit.client.1.smithi171.stdout:9/641: write dd/df/d15/d1a/fb2 [282142,11402] 0 2022-01-31T19:44:29.039 INFO:tasks.workunit.client.1.smithi171.stdout:9/642: creat dd/d1c/d7c/d7d/fdb x:0 0 0 2022-01-31T19:44:29.040 INFO:tasks.workunit.client.1.smithi171.stdout:0/654: mknod d9/d24/d92/d9d/d9f/cf3 0 2022-01-31T19:44:29.040 INFO:tasks.workunit.client.1.smithi171.stdout:0/655: chown d9/f62 1330 1 2022-01-31T19:44:29.043 INFO:tasks.workunit.client.0.smithi167.stdout:0/986: dread d17/d2c/f37 [0,4194304] 0 2022-01-31T19:44:29.043 INFO:tasks.workunit.client.0.smithi167.stdout:0/987: dread - d17/d19/d1c/d39/d7b/dbd/f13e zero size 2022-01-31T19:44:29.043 INFO:tasks.workunit.client.0.smithi167.stdout:0/988: stat d17/d19/d1c/d39/d44/d88/d98 0 2022-01-31T19:44:29.045 INFO:tasks.workunit.client.1.smithi171.stdout:6/615: rename dc/c69 to dc/d19/d29/d27/d37/d68/cca 0 2022-01-31T19:44:29.046 INFO:tasks.workunit.client.1.smithi171.stdout:6/616: stat dc/d19/d30/f92 0 2022-01-31T19:44:29.047 INFO:tasks.workunit.client.0.smithi167.stdout:0/989: dread d17/d3e/d7e/dc7/dde/df1/ffd [0,4194304] 0 2022-01-31T19:44:29.048 INFO:tasks.workunit.client.1.smithi171.stdout:1/563: dread dd/df/d19/d1b/f28 [0,4194304] 0 2022-01-31T19:44:29.048 INFO:tasks.workunit.client.1.smithi171.stdout:1/564: creat dd/d23/d2e/d5e/d81/fae x:0 0 0 2022-01-31T19:44:29.049 INFO:tasks.workunit.client.1.smithi171.stdout:9/643: symlink dd/d1c/db5/ldc 0 2022-01-31T19:44:29.051 INFO:tasks.workunit.client.1.smithi171.stdout:0/656: mkdir d9/d24/d4c/d35/d67/df4 0 2022-01-31T19:44:29.052 INFO:tasks.workunit.client.1.smithi171.stdout:0/657: write d9/db/d59/d71/fe2 [370602,52954] 0 2022-01-31T19:44:29.054 INFO:tasks.workunit.client.1.smithi171.stdout:6/617: mknod dc/d21/ccb 0 2022-01-31T19:44:29.055 INFO:tasks.workunit.client.0.smithi167.stdout:0/990: write d17/d33/f95 [3393464,26425] 0 2022-01-31T19:44:29.056 INFO:tasks.workunit.client.1.smithi171.stdout:0/658: dread d9/db/d59/d71/f8f [0,4194304] 0 2022-01-31T19:44:29.057 INFO:tasks.workunit.client.0.smithi167.stdout:0/991: getdents d17/d19 0 2022-01-31T19:44:29.063 INFO:tasks.workunit.client.1.smithi171.stdout:7/477: dwrite d1/d1e/d2a/d29/d31/f6a [4194304,4194304] 0 2022-01-31T19:44:29.063 INFO:tasks.workunit.client.1.smithi171.stdout:7/478: unlink d1/d1b/c55 0 2022-01-31T19:44:29.064 INFO:tasks.workunit.client.1.smithi171.stdout:2/801: dwrite d7/d26/de5/f9e [0,4194304] 0 2022-01-31T19:44:29.064 INFO:tasks.workunit.client.1.smithi171.stdout:4/522: dwrite d2/fa2 [0,4194304] 0 2022-01-31T19:44:29.066 INFO:tasks.workunit.client.1.smithi171.stdout:0/659: dread d9/db/d59/d71/fa4 [0,4194304] 0 2022-01-31T19:44:29.067 INFO:tasks.workunit.client.0.smithi167.stdout:0/992: dread d17/d19/f51 [0,4194304] 0 2022-01-31T19:44:29.067 INFO:tasks.workunit.client.0.smithi167.stdout:0/993: write d17/d3e/d7e/dc7/dde/df1/f134 [996250,14661] 0 2022-01-31T19:44:29.067 INFO:tasks.workunit.client.0.smithi167.stdout:0/994: dread - d17/d33/da4/fb4 zero size 2022-01-31T19:44:29.068 INFO:tasks.workunit.client.0.smithi167.stdout:0/995: unlink d17/d3e/f61 0 2022-01-31T19:44:29.068 INFO:tasks.workunit.client.0.smithi167.stdout:0/996: chown d17/d19/d1c/d39/d44/f93 619352599 1 2022-01-31T19:44:29.069 INFO:tasks.workunit.client.0.smithi167.stdout:0/997: rename d17/d2c/f3c to d17/d3e/d7e/f144 0 2022-01-31T19:44:29.070 INFO:tasks.workunit.client.0.smithi167.stdout:0/998: rename d17/d19/d1c/d39/d7b/dbd/ff7 to d17/d3e/da1/f145 0 2022-01-31T19:44:29.071 INFO:tasks.workunit.client.0.smithi167.stdout:0/999: readlink d17/d33/da4/lcc 0 2022-01-31T19:44:29.073 INFO:tasks.workunit.client.0.smithi167.stderr:+ rm -rf -- ./tmp.orDGTlWuTY 2022-01-31T19:44:29.107 INFO:tasks.workunit.client.1.smithi171.stdout:1/565: rename dd/d23/d2e/f87 to dd/df/d19/d1b/d1e/d64/da7/faf 0 2022-01-31T19:44:29.108 INFO:tasks.workunit.client.1.smithi171.stdout:9/644: mkdir dd/df/d15/d1a/d64/d6e/ddd 0 2022-01-31T19:44:29.108 INFO:tasks.workunit.client.1.smithi171.stdout:9/645: fdatasync dd/df/d15/d1a/d84/fcd 0 2022-01-31T19:44:29.109 INFO:tasks.workunit.client.1.smithi171.stdout:6/618: creat dc/d19/d30/d3c/d80/fcc x:0 0 0 2022-01-31T19:44:29.111 INFO:tasks.workunit.client.1.smithi171.stdout:7/479: rmdir d1/d52/d8e 0 2022-01-31T19:44:29.112 INFO:tasks.workunit.client.1.smithi171.stdout:2/802: rmdir d7/d9 39 2022-01-31T19:44:29.115 INFO:tasks.workunit.client.1.smithi171.stdout:4/523: write d2/d3/d5/d91/d7e/d12/d15/f78 [293280,70249] 0 2022-01-31T19:44:29.115 INFO:tasks.workunit.client.1.smithi171.stdout:0/660: getdents d9/d24/d92/d9d/d9f 0 2022-01-31T19:44:29.123 INFO:tasks.workunit.client.1.smithi171.stdout:1/566: truncate dd/df/d19/f47 676540 0 2022-01-31T19:44:29.124 INFO:tasks.workunit.client.1.smithi171.stdout:9/646: symlink dd/d1c/d7c/d7d/db6/lde 0 2022-01-31T19:44:29.125 INFO:tasks.workunit.client.1.smithi171.stdout:6/619: link dc/d19/d29/f85 dc/d21/d4f/db5/fcd 0 2022-01-31T19:44:29.126 INFO:tasks.workunit.client.1.smithi171.stdout:7/480: symlink d1/d26/d57/la2 0 2022-01-31T19:44:29.126 INFO:tasks.workunit.client.1.smithi171.stdout:7/481: dread - d1/d1e/d2a/d29/d31/d30/d3c/d5f/f90 zero size 2022-01-31T19:44:29.131 INFO:tasks.workunit.client.1.smithi171.stdout:0/661: rename d9/db/dc2/c38 to d9/d24/d92/d9d/dc6/cf5 0 2022-01-31T19:44:29.132 INFO:tasks.workunit.client.1.smithi171.stdout:1/567: mkdir dd/df/d19/d1b/d1e/d35/db0 0 2022-01-31T19:44:29.133 INFO:tasks.workunit.client.1.smithi171.stdout:9/647: mkdir dd/d1c/d7c/ddf 0 2022-01-31T19:44:29.134 INFO:tasks.workunit.client.1.smithi171.stdout:9/648: chown dd/df/d3f/d73/d96/cc1 2370 1 2022-01-31T19:44:29.137 INFO:tasks.workunit.client.1.smithi171.stdout:7/482: mknod d1/d1e/d2a/d58/ca3 0 2022-01-31T19:44:29.137 INFO:tasks.workunit.client.1.smithi171.stdout:7/483: chown d1/d26/l32 8002844 1 2022-01-31T19:44:29.137 INFO:tasks.workunit.client.1.smithi171.stdout:7/484: chown d1/d1e/d2a/d58/ca3 125316668 1 2022-01-31T19:44:29.137 INFO:tasks.workunit.client.1.smithi171.stdout:7/485: creat d1/d1e/d2a/d29/d31/d30/d3c/d5f/fa4 x:0 0 0 2022-01-31T19:44:29.138 INFO:tasks.workunit.client.1.smithi171.stdout:7/486: chown d1/d1e/d2a/d29/d31/d30/d3c/f76 24952 1 2022-01-31T19:44:29.138 INFO:tasks.workunit.client.1.smithi171.stdout:2/803: mkdir d7/dd/dfc 0 2022-01-31T19:44:29.138 INFO:tasks.workunit.client.1.smithi171.stdout:7/487: rename d1/d52/f44 to d1/d26/d57/fa5 0 2022-01-31T19:44:29.139 INFO:tasks.workunit.client.1.smithi171.stdout:2/804: unlink d7/c50 0 2022-01-31T19:44:29.140 INFO:tasks.workunit.client.1.smithi171.stdout:2/805: dread - d7/d7c/d83/d85/db9/fde zero size 2022-01-31T19:44:29.140 INFO:tasks.workunit.client.1.smithi171.stdout:7/488: truncate d1/d1e/d2a/d29/d31/f51 1484005 0 2022-01-31T19:44:29.141 INFO:tasks.workunit.client.1.smithi171.stdout:7/489: creat d1/d1e/d2a/fa6 x:0 0 0 2022-01-31T19:44:29.143 INFO:tasks.workunit.client.1.smithi171.stdout:2/806: rmdir d7/d7c 39 2022-01-31T19:44:29.144 INFO:tasks.workunit.client.1.smithi171.stdout:5/534: sync 2022-01-31T19:44:29.147 INFO:tasks.workunit.client.1.smithi171.stdout:2/807: write d7/f11 [379292,44734] 0 2022-01-31T19:44:29.149 INFO:tasks.workunit.client.1.smithi171.stdout:9/649: dread dd/d1c/f3d [0,4194304] 0 2022-01-31T19:44:29.150 INFO:tasks.workunit.client.1.smithi171.stdout:9/650: creat dd/d1c/da8/fe0 x:0 0 0 2022-01-31T19:44:29.150 INFO:tasks.workunit.client.1.smithi171.stdout:9/651: fsync dd/df/d15/d1a/d64/fac 0 2022-01-31T19:44:29.152 INFO:tasks.workunit.client.1.smithi171.stdout:6/620: dwrite dc/d19/d30/f9b [0,4194304] 0 2022-01-31T19:44:29.155 INFO:tasks.workunit.client.1.smithi171.stdout:6/621: mknod dc/d19/d30/cce 0 2022-01-31T19:44:29.156 INFO:tasks.workunit.client.1.smithi171.stdout:6/622: rename dc/d19/d29/d27/d37/d68 to dc/d19/d29/d27/d37/d68/d99/dcf 22 2022-01-31T19:44:29.156 INFO:tasks.workunit.client.1.smithi171.stdout:6/623: truncate dc/d19/d29/f34 3387771 0 2022-01-31T19:44:29.158 INFO:tasks.workunit.client.1.smithi171.stdout:6/624: mknod dc/d21/d4f/cd0 0 2022-01-31T19:44:29.160 INFO:tasks.workunit.client.1.smithi171.stdout:6/625: mkdir dc/d19/d29/d27/d37/d3b/d67/dd1 0 2022-01-31T19:44:29.161 INFO:tasks.workunit.client.1.smithi171.stdout:1/568: dread dd/df/d19/d36/f5b [0,4194304] 0 2022-01-31T19:44:29.161 INFO:tasks.workunit.client.1.smithi171.stdout:1/569: fsync dd/df/d19/d1b/f72 0 2022-01-31T19:44:29.163 INFO:tasks.workunit.client.1.smithi171.stdout:6/626: creat dc/d1e/d3f/fd2 x:0 0 0 2022-01-31T19:44:29.165 INFO:tasks.workunit.client.1.smithi171.stdout:1/570: creat dd/d23/d2e/d5e/d81/d98/fb1 x:0 0 0 2022-01-31T19:44:29.166 INFO:tasks.workunit.client.1.smithi171.stdout:6/627: creat dc/d19/d29/d27/d37/d3b/d67/dd1/fd3 x:0 0 0 2022-01-31T19:44:29.169 INFO:tasks.workunit.client.1.smithi171.stdout:6/628: read dc/d19/d29/f26 [1908137,30606] 0 2022-01-31T19:44:29.170 INFO:tasks.workunit.client.1.smithi171.stdout:1/571: dread dd/d23/d34/d4b/f52 [0,4194304] 0 2022-01-31T19:44:29.171 INFO:tasks.workunit.client.1.smithi171.stdout:6/629: rename dc/d21/f2e to dc/d19/d30/d3c/d80/fd4 0 2022-01-31T19:44:29.172 INFO:tasks.workunit.client.1.smithi171.stdout:6/630: chown dc/d21/d4f/l50 940074 1 2022-01-31T19:44:29.172 INFO:tasks.workunit.client.1.smithi171.stdout:6/631: mkdir dc/d19/d29/d27/d37/d3b/dd5 0 2022-01-31T19:44:29.173 INFO:tasks.workunit.client.1.smithi171.stdout:0/662: dwrite d9/db/dc2/fd3 [0,4194304] 0 2022-01-31T19:44:29.177 INFO:tasks.workunit.client.1.smithi171.stdout:7/490: dwrite d1/d52/f21 [0,4194304] 0 2022-01-31T19:44:29.180 INFO:tasks.workunit.client.1.smithi171.stdout:7/491: creat d1/d1e/d2a/d29/fa7 x:0 0 0 2022-01-31T19:44:29.182 INFO:tasks.workunit.client.1.smithi171.stdout:7/492: mknod d1/d52/ca8 0 2022-01-31T19:44:29.183 INFO:tasks.workunit.client.1.smithi171.stdout:9/652: dwrite dd/d1c/f95 [0,4194304] 0 2022-01-31T19:44:29.183 INFO:tasks.workunit.client.1.smithi171.stdout:7/493: mknod d1/d1b/ca9 0 2022-01-31T19:44:29.184 INFO:tasks.workunit.client.1.smithi171.stdout:7/494: dread d1/d52/f28 [0,4194304] 0 2022-01-31T19:44:29.185 INFO:tasks.workunit.client.1.smithi171.stdout:7/495: unlink d1/d52/l17 0 2022-01-31T19:44:29.186 INFO:tasks.workunit.client.1.smithi171.stdout:7/496: dread d1/d1e/d2a/d29/d31/d30/d3c/f45 [0,4194304] 0 2022-01-31T19:44:29.186 INFO:tasks.workunit.client.1.smithi171.stdout:7/497: chown d1/d26/d47/d8f/fa0 379320897 1 2022-01-31T19:44:29.187 INFO:tasks.workunit.client.1.smithi171.stdout:5/535: dwrite d5/d90/f78 [0,4194304] 0 2022-01-31T19:44:29.187 INFO:tasks.workunit.client.1.smithi171.stdout:9/653: rename dd/df/d15/d1a/d84/fad to dd/d1c/da8/fe1 0 2022-01-31T19:44:29.187 INFO:tasks.workunit.client.1.smithi171.stdout:9/654: write dd/df/d15/d53/fb7 [958689,32470] 0 2022-01-31T19:44:29.192 INFO:tasks.workunit.client.1.smithi171.stdout:7/498: truncate d1/d1e/d2a/d29/d77/f8c 2200831 0 2022-01-31T19:44:29.194 INFO:tasks.workunit.client.1.smithi171.stdout:5/536: rmdir d5/d6/d15/d80/d8d 39 2022-01-31T19:44:29.197 INFO:tasks.workunit.client.1.smithi171.stdout:9/655: creat dd/df/d15/d1a/d84/dca/fe2 x:0 0 0 2022-01-31T19:44:29.197 INFO:tasks.workunit.client.1.smithi171.stdout:9/656: dread - dd/df/d15/d38/d41/f50 zero size 2022-01-31T19:44:29.198 INFO:tasks.workunit.client.1.smithi171.stdout:2/808: dwrite d7/d9/d1a/d45/da4/fb4 [0,4194304] 0 2022-01-31T19:44:29.200 INFO:tasks.workunit.client.1.smithi171.stdout:7/499: rename d1/d1e/d2a/d29/d31/d30/f3f to d1/d1e/d2a/d29/d31/d30/d3c/d5f/faa 0 2022-01-31T19:44:29.202 INFO:tasks.workunit.client.1.smithi171.stdout:5/537: link d5/d6/f16 d5/d6/d15/d23/d58/d9e/fb6 0 2022-01-31T19:44:29.205 INFO:tasks.workunit.client.1.smithi171.stdout:1/572: dwrite dd/d23/d2e/f2f [8388608,4194304] 0 2022-01-31T19:44:29.205 INFO:tasks.workunit.client.1.smithi171.stdout:0/663: dwrite d9/d24/d4c/f66 [0,4194304] 0 2022-01-31T19:44:29.205 INFO:tasks.workunit.client.1.smithi171.stdout:1/573: read dd/d23/d2e/d5e/d81/d98/f85 [330595,15472] 0 2022-01-31T19:44:29.213 INFO:tasks.workunit.client.1.smithi171.stdout:2/809: write d7/d9/d1a/d45/da4/fd4 [505411,118852] 0 2022-01-31T19:44:29.218 INFO:tasks.workunit.client.1.smithi171.stdout:7/500: mkdir d1/d1e/d2a/d29/d77/dab 0 2022-01-31T19:44:29.218 INFO:tasks.workunit.client.1.smithi171.stdout:7/501: chown d1/d1b/f4b 126 1 2022-01-31T19:44:29.218 INFO:tasks.workunit.client.1.smithi171.stdout:5/538: rmdir d5/d59/da0 39 2022-01-31T19:44:29.222 INFO:tasks.workunit.client.1.smithi171.stdout:9/657: mkdir dd/df/d3f/de3 0 2022-01-31T19:44:29.225 INFO:tasks.workunit.client.1.smithi171.stdout:7/502: dread d1/d1e/d2a/d29/d31/f51 [0,4194304] 0 2022-01-31T19:44:29.226 INFO:tasks.workunit.client.1.smithi171.stdout:2/810: dread d7/d9/f2c [4194304,4194304] 0 2022-01-31T19:44:29.227 INFO:tasks.workunit.client.1.smithi171.stdout:2/811: stat d7/d9/d1a/l4b 0 2022-01-31T19:44:29.227 INFO:tasks.workunit.client.1.smithi171.stdout:2/812: readlink d7/dd/d5e/d84/lfa 0 2022-01-31T19:44:29.227 INFO:tasks.workunit.client.1.smithi171.stdout:2/813: write d7/fd5 [583045,108345] 0 2022-01-31T19:44:29.229 INFO:tasks.workunit.client.1.smithi171.stdout:5/539: rename d5/d6 to d5/d90/db7 0 2022-01-31T19:44:29.230 INFO:tasks.workunit.client.1.smithi171.stdout:1/574: dwrite dd/df/d19/d1b/d1e/d35/f7d [0,4194304] 0 2022-01-31T19:44:29.232 INFO:tasks.workunit.client.1.smithi171.stdout:0/664: dwrite d9/d24/f3c [0,4194304] 0 2022-01-31T19:44:29.234 INFO:tasks.workunit.client.1.smithi171.stdout:9/658: mkdir dd/df/d15/de4 0 2022-01-31T19:44:29.234 INFO:tasks.workunit.client.1.smithi171.stdout:9/659: chown dd/df/d15/d1a/f4b 79190 1 2022-01-31T19:44:29.234 INFO:tasks.workunit.client.1.smithi171.stdout:9/660: chown dd/df/d15/f36 41 1 2022-01-31T19:44:29.235 INFO:tasks.workunit.client.1.smithi171.stdout:2/814: link d7/d26/d3d/d40/f6e d7/d26/d3d/d40/ffd 0 2022-01-31T19:44:29.240 INFO:tasks.workunit.client.1.smithi171.stdout:0/665: mknod d9/d24/d4c/d35/cf6 0 2022-01-31T19:44:29.240 INFO:tasks.workunit.client.1.smithi171.stdout:0/666: chown d9/d24/d4c/d35/fbc 916469 1 2022-01-31T19:44:29.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/661: symlink dd/df/d15/d53/le5 0 2022-01-31T19:44:29.243 INFO:tasks.workunit.client.1.smithi171.stdout:5/540: link d5/d90/db7/d15/d5d/d6f/fa6 d5/d90/db7/d15/fb8 0 2022-01-31T19:44:29.243 INFO:tasks.workunit.client.1.smithi171.stdout:5/541: chown d5/d59/l9d 353190 1 2022-01-31T19:44:29.245 INFO:tasks.workunit.client.1.smithi171.stdout:9/662: rmdir dd/df/d3f/d73 39 2022-01-31T19:44:29.245 INFO:tasks.workunit.client.1.smithi171.stdout:9/663: chown dd/df/d15/d1a/d64/d6e/ddd 48137585 1 2022-01-31T19:44:29.246 INFO:tasks.workunit.client.1.smithi171.stdout:5/542: creat d5/d90/db7/d15/d5d/fb9 x:0 0 0 2022-01-31T19:44:29.247 INFO:tasks.workunit.client.1.smithi171.stdout:0/667: link d9/db/dc/d3f/db9/d7a/la1 d9/db/d59/d71/lf7 0 2022-01-31T19:44:29.248 INFO:tasks.workunit.client.1.smithi171.stdout:5/543: mknod d5/d90/db7/d15/d80/cba 0 2022-01-31T19:44:29.248 INFO:tasks.workunit.client.1.smithi171.stdout:9/664: dread dd/df/fa0 [0,4194304] 0 2022-01-31T19:44:29.249 INFO:tasks.workunit.client.1.smithi171.stdout:5/544: write d5/d90/db7/d15/d23/d67/f5c [839826,41637] 0 2022-01-31T19:44:29.249 INFO:tasks.workunit.client.1.smithi171.stdout:5/545: dread - d5/d90/db7/d15/d23/d58/fa3 zero size 2022-01-31T19:44:29.250 INFO:tasks.workunit.client.1.smithi171.stdout:0/668: unlink d9/db/d22/c58 0 2022-01-31T19:44:29.251 INFO:tasks.workunit.client.1.smithi171.stdout:1/575: dwrite dd/f14 [0,4194304] 0 2022-01-31T19:44:29.251 INFO:tasks.workunit.client.1.smithi171.stdout:1/576: truncate dd/d23/f9e 480820 0 2022-01-31T19:44:29.251 INFO:tasks.workunit.client.1.smithi171.stdout:1/577: creat dd/da2/fb2 x:0 0 0 2022-01-31T19:44:29.251 INFO:tasks.workunit.client.1.smithi171.stdout:1/578: fdatasync dd/df/d19/d36/fab 0 2022-01-31T19:44:29.252 INFO:tasks.workunit.client.1.smithi171.stdout:0/669: mknod d9/d24/d4c/d7d/cf8 0 2022-01-31T19:44:29.252 INFO:tasks.workunit.client.1.smithi171.stdout:0/670: getdents d9/db/dc/d3f/db9/dec 0 2022-01-31T19:44:29.254 INFO:tasks.workunit.client.1.smithi171.stdout:1/579: symlink dd/df/d19/lb3 0 2022-01-31T19:44:29.257 INFO:tasks.workunit.client.1.smithi171.stdout:8/509: sync 2022-01-31T19:44:29.257 INFO:tasks.workunit.client.1.smithi171.stdout:5/546: dread d5/ff [0,4194304] 0 2022-01-31T19:44:29.263 INFO:tasks.workunit.client.1.smithi171.stdout:0/671: creat d9/db/dc/d3f/d6f/ff9 x:0 0 0 2022-01-31T19:44:29.264 INFO:tasks.workunit.client.1.smithi171.stdout:8/510: dread df/d1a/f29 [0,4194304] 0 2022-01-31T19:44:29.265 INFO:tasks.workunit.client.1.smithi171.stdout:1/580: creat dd/d23/d74/fb4 x:0 0 0 2022-01-31T19:44:29.266 INFO:tasks.workunit.client.1.smithi171.stdout:5/547: rmdir d5/d59 39 2022-01-31T19:44:29.266 INFO:tasks.workunit.client.1.smithi171.stdout:5/548: fsync d5/d90/db7/d15/d23/d67/f76 0 2022-01-31T19:44:29.267 INFO:tasks.workunit.client.1.smithi171.stdout:1/581: write dd/df/d19/d1b/f63 [3841956,2959] 0 2022-01-31T19:44:29.268 INFO:tasks.workunit.client.1.smithi171.stdout:5/549: write d5/d90/db7/fb3 [878063,72333] 0 2022-01-31T19:44:29.268 INFO:tasks.workunit.client.1.smithi171.stdout:9/665: rename dd/d1c to dd/df/d3f/d73/d96/d9f/de6 0 2022-01-31T19:44:29.269 INFO:tasks.workunit.client.1.smithi171.stdout:5/550: write d5/f8 [8249214,16038] 0 2022-01-31T19:44:29.269 INFO:tasks.workunit.client.1.smithi171.stdout:1/582: truncate dd/d23/f57 517888 0 2022-01-31T19:44:29.271 INFO:tasks.workunit.client.1.smithi171.stdout:9/666: mknod dd/df/d15/d1a/d64/ce7 0 2022-01-31T19:44:29.273 INFO:tasks.workunit.client.1.smithi171.stdout:0/672: getdents d9/db/dc/d3f/db9/d7a 0 2022-01-31T19:44:29.273 INFO:tasks.workunit.client.1.smithi171.stdout:0/673: creat d9/db/dc/d3f/d6f/ffa x:0 0 0 2022-01-31T19:44:29.275 INFO:tasks.workunit.client.1.smithi171.stdout:0/674: mkdir d9/d24/d92/d9d/dc6/dfb 0 2022-01-31T19:44:29.276 INFO:tasks.workunit.client.1.smithi171.stdout:9/667: write dd/f79 [2790480,99158] 0 2022-01-31T19:44:29.277 INFO:tasks.workunit.client.1.smithi171.stdout:0/675: creat d9/d24/d92/d9d/dc6/dfb/ffc x:0 0 0 2022-01-31T19:44:29.279 INFO:tasks.workunit.client.1.smithi171.stdout:5/551: dread d5/d90/db7/d15/fb8 [0,4194304] 0 2022-01-31T19:44:29.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/668: symlink dd/df/d3f/d73/d96/d9f/de6/le8 0 2022-01-31T19:44:29.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/669: truncate dd/df/d15/d53/fb7 1015471 0 2022-01-31T19:44:29.282 INFO:tasks.workunit.client.1.smithi171.stdout:9/670: chown dd/df/d3f/d73/d96/d9f/de6/d7c/d7d/db6/lde 2 1 2022-01-31T19:44:29.284 INFO:tasks.workunit.client.1.smithi171.stdout:0/676: rename d9/db/dc/f8b to d9/db/dc/d3f/db9/d7a/d88/ffd 0 2022-01-31T19:44:29.286 INFO:tasks.workunit.client.1.smithi171.stdout:5/552: unlink d5/f68 0 2022-01-31T19:44:29.288 INFO:tasks.workunit.client.1.smithi171.stdout:0/677: rmdir d9/db/d59/d71/dbe 39 2022-01-31T19:44:29.289 INFO:tasks.workunit.client.1.smithi171.stdout:8/511: dwrite df/d2a/f49 [0,4194304] 0 2022-01-31T19:44:29.290 INFO:tasks.workunit.client.1.smithi171.stdout:1/583: dwrite dd/f7f [0,4194304] 0 2022-01-31T19:44:29.293 INFO:tasks.workunit.client.1.smithi171.stdout:8/512: mkdir df/d1a/dae 0 2022-01-31T19:44:29.296 INFO:tasks.workunit.client.1.smithi171.stdout:1/584: mknod dd/df/d19/d1b/d1e/cb5 0 2022-01-31T19:44:29.297 INFO:tasks.workunit.client.1.smithi171.stdout:8/513: write df/d18/d19/d36/f73 [1743713,63777] 0 2022-01-31T19:44:29.300 INFO:tasks.workunit.client.1.smithi171.stdout:8/514: creat df/d2a/faf x:0 0 0 2022-01-31T19:44:29.300 INFO:tasks.workunit.client.1.smithi171.stdout:8/515: fsync df/f83 0 2022-01-31T19:44:29.300 INFO:tasks.workunit.client.1.smithi171.stdout:8/516: symlink df/d5d/d6c/lb0 0 2022-01-31T19:44:29.303 INFO:tasks.workunit.client.1.smithi171.stdout:8/517: mknod df/d18/d19/d82/cb1 0 2022-01-31T19:44:29.304 INFO:tasks.workunit.client.1.smithi171.stdout:8/518: stat df/d18/d79/da2/da8 0 2022-01-31T19:44:29.306 INFO:tasks.workunit.client.1.smithi171.stdout:0/678: dread d9/db/dc/d3f/f6d [0,4194304] 0 2022-01-31T19:44:29.307 INFO:tasks.workunit.client.1.smithi171.stdout:0/679: creat d9/db/d59/ffe x:0 0 0 2022-01-31T19:44:29.307 INFO:tasks.workunit.client.1.smithi171.stdout:8/519: mkdir df/d18/d19/db2 0 2022-01-31T19:44:29.307 INFO:tasks.workunit.client.1.smithi171.stdout:8/520: unlink fc 0 2022-01-31T19:44:29.308 INFO:tasks.workunit.client.1.smithi171.stdout:9/671: dwrite dd/df/d3f/d73/d96/d9f/de6/d7c/d7d/fa3 [0,4194304] 0 2022-01-31T19:44:29.309 INFO:tasks.workunit.client.1.smithi171.stdout:9/672: readlink dd/laa 0 2022-01-31T19:44:29.310 INFO:tasks.workunit.client.1.smithi171.stdout:5/553: dwrite d5/d90/db7/d15/d23/f87 [0,4194304] 0 2022-01-31T19:44:29.310 INFO:tasks.workunit.client.1.smithi171.stdout:1/585: dread dd/df/d19/d1b/d1e/f4a [0,4194304] 0 2022-01-31T19:44:29.311 INFO:tasks.workunit.client.1.smithi171.stdout:1/586: truncate dd/df/d19/d36/f5b 5214508 0 2022-01-31T19:44:29.311 INFO:tasks.workunit.client.1.smithi171.stdout:0/680: rename d9/d24/d92/l68 to d9/db/dc/d3f/db9/lff 0 2022-01-31T19:44:29.313 INFO:tasks.workunit.client.1.smithi171.stdout:8/521: creat df/d5d/d6c/fb3 x:0 0 0 2022-01-31T19:44:29.325 INFO:tasks.workunit.client.1.smithi171.stdout:1/587: dwrite dd/d23/d2e/d5e/d81/d98/f85 [0,4194304] 0 2022-01-31T19:44:29.329 INFO:tasks.workunit.client.1.smithi171.stdout:8/522: dread fb [0,4194304] 0 2022-01-31T19:44:29.329 INFO:tasks.workunit.client.1.smithi171.stdout:8/523: truncate df/d95/f98 712607 0 2022-01-31T19:44:29.341 INFO:tasks.workunit.client.1.smithi171.stdout:1/588: dwrite dd/d23/d74/d7b/f99 [4194304,4194304] 0 2022-01-31T19:44:29.384 INFO:tasks.workunit.client.1.smithi171.stdout:9/673: mknod dd/df/d15/d1a/d84/ce9 0 2022-01-31T19:44:29.385 INFO:tasks.workunit.client.1.smithi171.stdout:5/554: link d5/d90/db7/d15/d5d/d6a/l85 d5/d59/da0/lbb 0 2022-01-31T19:44:29.389 INFO:tasks.workunit.client.1.smithi171.stdout:0/681: mkdir d9/d24/d100 0 2022-01-31T19:44:29.392 INFO:tasks.workunit.client.1.smithi171.stdout:8/524: creat df/d27/d90/fb4 x:0 0 0 2022-01-31T19:44:29.393 INFO:tasks.workunit.client.1.smithi171.stdout:8/525: fdatasync df/d2a/d30/f63 0 2022-01-31T19:44:29.393 INFO:tasks.workunit.client.1.smithi171.stdout:8/526: write df/d18/f42 [4804067,81256] 0 2022-01-31T19:44:29.395 INFO:tasks.workunit.client.1.smithi171.stdout:1/589: stat dd/d23/d2e/d5e/d81/c8d 0 2022-01-31T19:44:29.396 INFO:tasks.workunit.client.1.smithi171.stdout:1/590: chown dd/df/d19/f96 107 1 2022-01-31T19:44:29.396 INFO:tasks.workunit.client.1.smithi171.stdout:1/591: truncate dd/d23/f9e 1015161 0 2022-01-31T19:44:29.397 INFO:tasks.workunit.client.1.smithi171.stdout:9/674: creat dd/df/d3f/d73/fea x:0 0 0 2022-01-31T19:44:29.398 INFO:tasks.workunit.client.1.smithi171.stdout:9/675: chown dd/df/d15/d1a/d84/f91 109519567 1 2022-01-31T19:44:29.398 INFO:tasks.workunit.client.1.smithi171.stdout:9/676: write dd/df/d15/d53/fbf [713580,23585] 0 2022-01-31T19:44:29.398 INFO:tasks.workunit.client.1.smithi171.stdout:9/677: fdatasync dd/f81 0 2022-01-31T19:44:29.399 INFO:tasks.workunit.client.1.smithi171.stdout:5/555: creat d5/db0/fbc x:0 0 0 2022-01-31T19:44:29.399 INFO:tasks.workunit.client.1.smithi171.stdout:5/556: chown d5/fb5 2832 1 2022-01-31T19:44:29.399 INFO:tasks.workunit.client.1.smithi171.stdout:5/557: fsync d5/d90/db7/f46 0 2022-01-31T19:44:29.405 INFO:tasks.workunit.client.1.smithi171.stdout:5/558: write d5/d90/db7/f16 [1111630,128435] 0 2022-01-31T19:44:29.407 INFO:tasks.workunit.client.1.smithi171.stdout:0/682: mkdir d9/d24/d4c/d101 0 2022-01-31T19:44:29.408 INFO:tasks.workunit.client.1.smithi171.stdout:0/683: creat d9/f102 x:0 0 0 2022-01-31T19:44:29.410 INFO:tasks.workunit.client.1.smithi171.stdout:1/592: mkdir dd/d23/d2e/d5e/d81/d98/dad/db6 0 2022-01-31T19:44:29.410 INFO:tasks.workunit.client.1.smithi171.stdout:8/527: dwrite df/d27/d85/f8f [0,4194304] 0 2022-01-31T19:44:29.410 INFO:tasks.workunit.client.1.smithi171.stdout:9/678: truncate dd/df/d3f/d73/d96/d9f/de6/f1f 221166 0 2022-01-31T19:44:29.415 INFO:tasks.workunit.client.1.smithi171.stdout:8/528: mknod df/d27/d5c/cb5 0 2022-01-31T19:44:29.416 INFO:tasks.workunit.client.1.smithi171.stdout:8/529: fsync df/d27/d90/fb4 0 2022-01-31T19:44:29.416 INFO:tasks.workunit.client.1.smithi171.stdout:9/679: mkdir dd/df/d86/deb 0 2022-01-31T19:44:29.419 INFO:tasks.workunit.client.1.smithi171.stdout:9/680: symlink dd/df/d15/d38/lec 0 2022-01-31T19:44:29.421 INFO:tasks.workunit.client.1.smithi171.stdout:9/681: symlink dd/df/d3f/led 0 2022-01-31T19:44:29.424 INFO:tasks.workunit.client.1.smithi171.stdout:8/530: dread df/d18/d19/d46/f5e [0,4194304] 0 2022-01-31T19:44:29.424 INFO:tasks.workunit.client.1.smithi171.stdout:8/531: write df/d18/d19/d36/f47 [410941,52199] 0 2022-01-31T19:44:29.428 INFO:tasks.workunit.client.1.smithi171.stdout:0/684: dwrite d9/db/d22/fd1 [0,4194304] 0 2022-01-31T19:44:29.428 INFO:tasks.workunit.client.1.smithi171.stdout:0/685: fdatasync d9/f1f 0 2022-01-31T19:44:29.433 INFO:tasks.workunit.client.1.smithi171.stdout:0/686: mknod d9/d24/d92/d9d/d9f/c103 0 2022-01-31T19:44:29.434 INFO:tasks.workunit.client.1.smithi171.stdout:1/593: truncate dd/d23/f9e 825421 0 2022-01-31T19:44:29.434 INFO:tasks.workunit.client.1.smithi171.stdout:1/594: dread - dd/df/d19/f96 zero size 2022-01-31T19:44:29.437 INFO:tasks.workunit.client.1.smithi171.stdout:0/687: rename d9/db/dc/d3f/db9/f42 to d9/db/dc/f104 0 2022-01-31T19:44:29.439 INFO:tasks.workunit.client.1.smithi171.stdout:0/688: write d9/db/d22/fd1 [5156006,109169] 0 2022-01-31T19:44:29.444 INFO:tasks.workunit.client.1.smithi171.stdout:0/689: write d9/db/dc/d3f/db9/d7a/fe7 [4108678,51591] 0 2022-01-31T19:44:29.445 INFO:tasks.workunit.client.1.smithi171.stdout:1/595: dread fa [0,4194304] 0 2022-01-31T19:44:29.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/596: chown f9 2123 1 2022-01-31T19:44:29.446 INFO:tasks.workunit.client.1.smithi171.stdout:0/690: unlink d9/d24/d92/f99 0 2022-01-31T19:44:29.446 INFO:tasks.workunit.client.1.smithi171.stdout:0/691: stat d9/d24/d4c/d35/d67/l7b 0 2022-01-31T19:44:29.447 INFO:tasks.workunit.client.1.smithi171.stdout:0/692: write d9/db/f47 [557404,1654] 0 2022-01-31T19:44:29.448 INFO:tasks.workunit.client.1.smithi171.stdout:0/693: symlink d9/db/d22/l105 0 2022-01-31T19:44:29.448 INFO:tasks.workunit.client.1.smithi171.stdout:0/694: write d9/d24/d4c/d35/f89 [48998,71664] 0 2022-01-31T19:44:29.448 INFO:tasks.workunit.client.1.smithi171.stdout:0/695: readlink d9/db/dc2/l36 0 2022-01-31T19:44:29.449 INFO:tasks.workunit.client.1.smithi171.stdout:0/696: creat d9/d24/d4c/d7d/f106 x:0 0 0 2022-01-31T19:44:29.449 INFO:tasks.workunit.client.1.smithi171.stdout:0/697: chown d9/db/dc2/l36 518934 1 2022-01-31T19:44:29.450 INFO:tasks.workunit.client.1.smithi171.stdout:0/698: mkdir d9/d24/d92/d9d/d9f/d107 0 2022-01-31T19:44:29.451 INFO:tasks.workunit.client.1.smithi171.stdout:9/682: dwrite dd/df/d15/f3a [0,4194304] 0 2022-01-31T19:44:29.455 INFO:tasks.workunit.client.1.smithi171.stdout:8/532: dwrite df/f84 [0,4194304] 0 2022-01-31T19:44:29.456 INFO:tasks.workunit.client.1.smithi171.stdout:8/533: write df/d18/d79/d7a/f99 [3192765,125615] 0 2022-01-31T19:44:29.456 INFO:tasks.workunit.client.1.smithi171.stdout:8/534: fdatasync f0 0 2022-01-31T19:44:29.460 INFO:tasks.workunit.client.1.smithi171.stdout:9/683: dread dd/df/d3f/d73/d96/d9f/de6/f29 [0,4194304] 0 2022-01-31T19:44:29.460 INFO:tasks.workunit.client.1.smithi171.stdout:9/684: readlink dd/df/d15/l1e 0 2022-01-31T19:44:29.460 INFO:tasks.workunit.client.1.smithi171.stdout:8/535: creat df/d18/d19/d21/fb6 x:0 0 0 2022-01-31T19:44:29.461 INFO:tasks.workunit.client.1.smithi171.stdout:8/536: creat df/d27/fb7 x:0 0 0 2022-01-31T19:44:29.461 INFO:tasks.workunit.client.1.smithi171.stdout:0/699: dread d9/d24/d92/d9d/dc6/fd9 [0,4194304] 0 2022-01-31T19:44:29.462 INFO:tasks.workunit.client.1.smithi171.stdout:8/537: creat df/d18/da4/fb8 x:0 0 0 2022-01-31T19:44:29.462 INFO:tasks.workunit.client.1.smithi171.stdout:8/538: chown df/f84 24377752 1 2022-01-31T19:44:29.463 INFO:tasks.workunit.client.1.smithi171.stdout:8/539: fsync df/d18/d19/f1b 0 2022-01-31T19:44:29.464 INFO:tasks.workunit.client.1.smithi171.stdout:0/700: symlink d9/d24/d4c/d35/d67/df4/l108 0 2022-01-31T19:44:29.465 INFO:tasks.workunit.client.1.smithi171.stdout:0/701: stat d9/d24/d4c/d35/d67 0 2022-01-31T19:44:29.465 INFO:tasks.workunit.client.1.smithi171.stdout:0/702: fdatasync d9/db/d59/d71/fe2 0 2022-01-31T19:44:29.465 INFO:tasks.workunit.client.1.smithi171.stdout:0/703: dread - d9/d24/d4c/f78 zero size 2022-01-31T19:44:29.465 INFO:tasks.workunit.client.1.smithi171.stdout:0/704: chown d9/db/dc/d3f/db9/d7a/fe7 2336762 1 2022-01-31T19:44:29.466 INFO:tasks.workunit.client.1.smithi171.stdout:0/705: creat d9/db/d59/d71/dba/f109 x:0 0 0 2022-01-31T19:44:29.466 INFO:tasks.workunit.client.1.smithi171.stdout:9/685: write dd/df/d15/d1a/f56 [2840763,72401] 0 2022-01-31T19:44:29.466 INFO:tasks.workunit.client.1.smithi171.stdout:9/686: dread - dd/df/d15/d1a/d64/d6e/fd2 zero size 2022-01-31T19:44:29.466 INFO:tasks.workunit.client.1.smithi171.stdout:9/687: chown dd/f37 7277 1 2022-01-31T19:44:29.467 INFO:tasks.workunit.client.1.smithi171.stdout:9/688: fsync dd/df/d3f/d73/d96/d9f/de6/db5/fbe 0 2022-01-31T19:44:29.467 INFO:tasks.workunit.client.1.smithi171.stdout:9/689: truncate dd/df/d3f/d73/d96/d9f/de6/f3d 5240484 0 2022-01-31T19:44:29.467 INFO:tasks.workunit.client.1.smithi171.stdout:9/690: chown dd/df/d15/f4a 243608103 1 2022-01-31T19:44:29.467 INFO:tasks.workunit.client.1.smithi171.stdout:9/691: write dd/df/d3f/d73/d96/d9f/de6/d7c/d7d/fdb [209543,102374] 0 2022-01-31T19:44:29.468 INFO:tasks.workunit.client.1.smithi171.stdout:8/540: write df/d27/d85/f8f [1899175,5195] 0 2022-01-31T19:44:29.469 INFO:tasks.workunit.client.1.smithi171.stdout:8/541: dread - df/d5d/d6c/d87/fa0 zero size 2022-01-31T19:44:29.469 INFO:tasks.workunit.client.1.smithi171.stdout:0/706: symlink d9/db/dc/d3f/d6f/dcc/l10a 0 2022-01-31T19:44:29.469 INFO:tasks.workunit.client.1.smithi171.stdout:0/707: read - d9/d24/d4c/d7d/f106 zero size 2022-01-31T19:44:29.469 INFO:tasks.workunit.client.1.smithi171.stdout:1/597: dwrite dd/da2/fa6 [0,4194304] 0 2022-01-31T19:44:29.470 INFO:tasks.workunit.client.1.smithi171.stdout:1/598: getdents dd/d23/d2e/d5e/d81/d98/d80 0 2022-01-31T19:44:29.470 INFO:tasks.workunit.client.1.smithi171.stdout:8/542: mkdir df/db9 0 2022-01-31T19:44:29.474 INFO:tasks.workunit.client.1.smithi171.stdout:1/599: mknod dd/df/d19/d36/cb7 0 2022-01-31T19:44:29.475 INFO:tasks.workunit.client.1.smithi171.stdout:8/543: truncate df/d18/d79/f8c 6147554 0 2022-01-31T19:44:29.476 INFO:tasks.workunit.client.1.smithi171.stdout:1/600: link dd/d23/d34/c71 dd/df/d84/cb8 0 2022-01-31T19:44:29.476 INFO:tasks.workunit.client.1.smithi171.stdout:1/601: readlink l6 0 2022-01-31T19:44:29.476 INFO:tasks.workunit.client.1.smithi171.stdout:1/602: write dd/d23/d2e/d5e/f8a [931556,15570] 0 2022-01-31T19:44:29.478 INFO:tasks.workunit.client.1.smithi171.stdout:1/603: write dd/f91 [165733,71370] 0 2022-01-31T19:44:29.491 INFO:tasks.workunit.client.1.smithi171.stdout:9/692: dwrite dd/df/d15/d1a/fb2 [0,4194304] 0 2022-01-31T19:44:29.497 INFO:tasks.workunit.client.1.smithi171.stdout:9/693: link dd/df/d15/d1a/d64/f67 dd/df/d15/d38/d4d/d80/fee 0 2022-01-31T19:44:29.510 INFO:tasks.workunit.client.1.smithi171.stdout:0/708: dwrite d9/db/dc/d3f/f6d [0,4194304] 0 2022-01-31T19:44:29.510 INFO:tasks.workunit.client.1.smithi171.stdout:9/694: dread f9 [0,4194304] 0 2022-01-31T19:44:29.511 INFO:tasks.workunit.client.1.smithi171.stdout:9/695: truncate dd/df/f17 1968661 0 2022-01-31T19:44:29.513 INFO:tasks.workunit.client.1.smithi171.stdout:0/709: write d9/d24/f3c [995869,54270] 0 2022-01-31T19:44:29.514 INFO:tasks.workunit.client.1.smithi171.stdout:9/696: rename dd/df/d15/f22 to dd/df/d3f/de3/fef 0 2022-01-31T19:44:29.515 INFO:tasks.workunit.client.1.smithi171.stdout:0/710: mknod d9/d24/d92/d9d/d9f/c10b 0 2022-01-31T19:44:29.516 INFO:tasks.workunit.client.1.smithi171.stdout:0/711: link d9/db/dc2/l36 d9/d24/d4c/d35/d67/l10c 0 2022-01-31T19:44:29.517 INFO:tasks.workunit.client.1.smithi171.stdout:0/712: creat d9/db/dc/d3f/db9/f10d x:0 0 0 2022-01-31T19:44:29.518 INFO:tasks.workunit.client.1.smithi171.stdout:0/713: mkdir d9/d24/d4c/d10e 0 2022-01-31T19:44:29.519 INFO:tasks.workunit.client.1.smithi171.stdout:8/544: dwrite df/d18/d79/d7a/f99 [0,4194304] 0 2022-01-31T19:44:29.520 INFO:tasks.workunit.client.1.smithi171.stdout:8/545: fsync df/d27/f5b 0 2022-01-31T19:44:29.521 INFO:tasks.workunit.client.1.smithi171.stdout:0/714: dread d9/db/dc/d3f/db9/f3e [0,4194304] 0 2022-01-31T19:44:29.525 INFO:tasks.workunit.client.1.smithi171.stdout:8/546: getdents df/d5d/d6c 0 2022-01-31T19:44:29.525 INFO:tasks.workunit.client.1.smithi171.stdout:8/547: dread - df/d1a/f28 zero size 2022-01-31T19:44:29.527 INFO:tasks.workunit.client.1.smithi171.stdout:8/548: mknod df/d18/d19/d21/d67/cba 0 2022-01-31T19:44:29.528 INFO:tasks.workunit.client.1.smithi171.stdout:8/549: stat f6 0 2022-01-31T19:44:29.528 INFO:tasks.workunit.client.1.smithi171.stdout:8/550: chown f1 59592 1 2022-01-31T19:44:29.528 INFO:tasks.workunit.client.1.smithi171.stdout:8/551: creat df/d27/d90/fbb x:0 0 0 2022-01-31T19:44:29.529 INFO:tasks.workunit.client.1.smithi171.stdout:8/552: mkdir df/d18/d19/d46/dbc 0 2022-01-31T19:44:29.530 INFO:tasks.workunit.client.1.smithi171.stdout:8/553: creat df/d18/d19/d82/fbd x:0 0 0 2022-01-31T19:44:29.531 INFO:tasks.workunit.client.1.smithi171.stdout:8/554: link df/c68 df/d18/d79/d7a/cbe 0 2022-01-31T19:44:29.533 INFO:tasks.workunit.client.1.smithi171.stdout:8/555: creat df/d5d/d6c/fbf x:0 0 0 2022-01-31T19:44:29.535 INFO:tasks.workunit.client.1.smithi171.stdout:9/697: dwrite dd/fc2 [0,4194304] 0 2022-01-31T19:44:29.535 INFO:tasks.workunit.client.1.smithi171.stdout:8/556: mkdir df/d18/d79/d7a/dc0 0 2022-01-31T19:44:29.536 INFO:tasks.workunit.client.1.smithi171.stdout:8/557: readlink df/d1a/l75 0 2022-01-31T19:44:29.536 INFO:tasks.workunit.client.1.smithi171.stdout:9/698: write dd/df/d15/d1a/d64/fc7 [1435523,116570] 0 2022-01-31T19:44:29.537 INFO:tasks.workunit.client.1.smithi171.stdout:0/715: dread d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:29.539 INFO:tasks.workunit.client.1.smithi171.stdout:0/716: write d9/d24/d92/d9d/d9f/fac [1382151,101449] 0 2022-01-31T19:44:29.539 INFO:tasks.workunit.client.1.smithi171.stdout:8/558: write df/f83 [109976,66340] 0 2022-01-31T19:44:29.540 INFO:tasks.workunit.client.1.smithi171.stdout:0/717: write d9/db/dc/d3f/f6d [866285,60780] 0 2022-01-31T19:44:29.544 INFO:tasks.workunit.client.1.smithi171.stdout:9/699: link dd/df/d15/d1a/d64/c98 dd/df/cf0 0 2022-01-31T19:44:29.548 INFO:tasks.workunit.client.1.smithi171.stdout:8/559: rmdir df/db9 0 2022-01-31T19:44:29.550 INFO:tasks.workunit.client.1.smithi171.stdout:4/524: sync 2022-01-31T19:44:29.551 INFO:tasks.workunit.client.1.smithi171.stdout:3/673: sync 2022-01-31T19:44:29.551 INFO:tasks.workunit.client.1.smithi171.stdout:6/632: sync 2022-01-31T19:44:29.551 INFO:tasks.workunit.client.1.smithi171.stdout:4/525: chown d2/d3/d5/d70/d7c/la3 3820 1 2022-01-31T19:44:29.551 INFO:tasks.workunit.client.1.smithi171.stdout:3/674: creat d4/d21/d3a/d59/dc1/fd7 x:0 0 0 2022-01-31T19:44:29.552 INFO:tasks.workunit.client.1.smithi171.stdout:8/560: mknod df/d18/d19/d46/cc1 0 2022-01-31T19:44:29.554 INFO:tasks.workunit.client.1.smithi171.stdout:6/633: rename c5 to dc/d19/d29/d27/d37/d68/d99/cd6 0 2022-01-31T19:44:29.555 INFO:tasks.workunit.client.1.smithi171.stdout:4/526: mkdir d2/d3/d5/d91/d7e/d12/d15/d57/dad 0 2022-01-31T19:44:29.556 INFO:tasks.workunit.client.1.smithi171.stdout:8/561: truncate df/d18/d79/f8c 6034744 0 2022-01-31T19:44:29.556 INFO:tasks.workunit.client.1.smithi171.stdout:8/562: unlink df/d18/l37 0 2022-01-31T19:44:29.557 INFO:tasks.workunit.client.1.smithi171.stdout:4/527: stat d2/d3/c51 0 2022-01-31T19:44:29.558 INFO:tasks.workunit.client.1.smithi171.stdout:8/563: mknod df/d5d/d6c/d87/cc2 0 2022-01-31T19:44:29.559 INFO:tasks.workunit.client.1.smithi171.stdout:4/528: mknod d2/d3/d5/d91/d7e/d12/cae 0 2022-01-31T19:44:29.559 INFO:tasks.workunit.client.1.smithi171.stdout:4/529: creat d2/d3/d5/d3b/faf x:0 0 0 2022-01-31T19:44:29.561 INFO:tasks.workunit.client.1.smithi171.stdout:9/700: dread dd/df/d3f/d73/d96/d9f/de6/f95 [0,4194304] 0 2022-01-31T19:44:29.564 INFO:tasks.workunit.client.1.smithi171.stdout:6/634: dread dc/d19/d30/f92 [0,4194304] 0 2022-01-31T19:44:29.564 INFO:tasks.workunit.client.1.smithi171.stdout:6/635: write dc/d8b/f8f [932757,56148] 0 2022-01-31T19:44:29.564 INFO:tasks.workunit.client.1.smithi171.stdout:6/636: stat f9 0 2022-01-31T19:44:29.564 INFO:tasks.workunit.client.1.smithi171.stdout:4/530: fdatasync d2/d3/d5/d91/d7e/f83 0 2022-01-31T19:44:29.565 INFO:tasks.workunit.client.1.smithi171.stdout:4/531: write d2/d3/d5/d70/fa7 [960432,74660] 0 2022-01-31T19:44:29.565 INFO:tasks.workunit.client.1.smithi171.stdout:3/675: write d4/dd/d87/d8e/faf [4054052,78124] 0 2022-01-31T19:44:29.565 INFO:tasks.workunit.client.1.smithi171.stdout:4/532: write d2/f5c [1922437,71701] 0 2022-01-31T19:44:29.571 INFO:tasks.workunit.client.1.smithi171.stdout:6/637: creat dc/d18/fd7 x:0 0 0 2022-01-31T19:44:29.571 INFO:tasks.workunit.client.1.smithi171.stdout:3/676: symlink d4/dd/d27/d28/d35/d55/dbf/ld8 0 2022-01-31T19:44:29.573 INFO:tasks.workunit.client.1.smithi171.stdout:3/677: mknod d4/d21/d3a/db5/dbe/dd1/cd9 0 2022-01-31T19:44:29.573 INFO:tasks.workunit.client.1.smithi171.stdout:3/678: stat d4/dd/d27/lb1 0 2022-01-31T19:44:29.574 INFO:tasks.workunit.client.1.smithi171.stdout:4/533: mknod d2/d3/d5/d91/d7e/cb0 0 2022-01-31T19:44:29.574 INFO:tasks.workunit.client.1.smithi171.stdout:4/534: fdatasync d2/d3/d5/d91/d7e/d12/f1d 0 2022-01-31T19:44:29.575 INFO:tasks.workunit.client.1.smithi171.stdout:3/679: symlink d4/dd/d27/d28/d35/d55/d6c/lda 0 2022-01-31T19:44:29.579 INFO:tasks.workunit.client.1.smithi171.stdout:4/535: write d2/f9e [3288474,119873] 0 2022-01-31T19:44:29.579 INFO:tasks.workunit.client.1.smithi171.stdout:4/536: read - d2/d3/d5/d91/d7e/d12/f2a zero size 2022-01-31T19:44:29.579 INFO:tasks.workunit.client.1.smithi171.stdout:4/537: chown d2/d3/d5/d91/f9d 50733 1 2022-01-31T19:44:29.581 INFO:tasks.workunit.client.1.smithi171.stdout:4/538: read d2/d3/d5/d91/d7e/d12/f1d [1354134,66120] 0 2022-01-31T19:44:29.583 INFO:tasks.workunit.client.1.smithi171.stdout:4/539: rename d2/d3/d5/d70/d7b/l7d to d2/d3/d5/d91/d7e/d12/d15/lb1 0 2022-01-31T19:44:29.586 INFO:tasks.workunit.client.1.smithi171.stdout:6/638: dread dc/d1e/d3f/f65 [4194304,4194304] 0 2022-01-31T19:44:29.597 INFO:tasks.workunit.client.1.smithi171.stdout:3/680: dwrite d4/dd/d39/d51/f71 [0,4194304] 0 2022-01-31T19:44:29.627 INFO:tasks.workunit.client.1.smithi171.stdout:8/564: rmdir df 39 2022-01-31T19:44:29.627 INFO:tasks.workunit.client.1.smithi171.stdout:8/565: dread - df/d18/d19/f61 zero size 2022-01-31T19:44:29.628 INFO:tasks.workunit.client.1.smithi171.stdout:8/566: write df/d18/f4d [1180014,20878] 0 2022-01-31T19:44:29.628 INFO:tasks.workunit.client.1.smithi171.stdout:8/567: chown df/d18/f41 0 1 2022-01-31T19:44:29.628 INFO:tasks.workunit.client.1.smithi171.stdout:8/568: dread - df/d18/d19/d46/f78 zero size 2022-01-31T19:44:29.629 INFO:tasks.workunit.client.1.smithi171.stdout:4/540: dwrite d2/d3/d5/d91/d7e/d12/f8d [0,4194304] 0 2022-01-31T19:44:29.630 INFO:tasks.workunit.client.1.smithi171.stdout:4/541: creat d2/d3/d5/d41/d6b/fb2 x:0 0 0 2022-01-31T19:44:29.630 INFO:tasks.workunit.client.1.smithi171.stdout:4/542: fdatasync d2/d3/d5/d91/d7e/d12/f1d 0 2022-01-31T19:44:29.631 INFO:tasks.workunit.client.1.smithi171.stdout:3/681: dwrite d4/dd/d27/d28/d35/d55/d8f/d92/dab/d52/f96 [0,4194304] 0 2022-01-31T19:44:29.632 INFO:tasks.workunit.client.1.smithi171.stdout:3/682: dread - d4/d21/d3a/d59/dc1/fc2 zero size 2022-01-31T19:44:29.632 INFO:tasks.workunit.client.1.smithi171.stdout:3/683: readlink d4/l26 0 2022-01-31T19:44:29.632 INFO:tasks.workunit.client.1.smithi171.stdout:6/639: dwrite dc/d19/d29/d27/f3a [0,4194304] 0 2022-01-31T19:44:29.633 INFO:tasks.workunit.client.1.smithi171.stdout:4/543: mknod d2/d3/d5/cb3 0 2022-01-31T19:44:29.635 INFO:tasks.workunit.client.1.smithi171.stdout:3/684: creat d4/dd/d27/d28/d35/d55/d8f/d92/dab/dd4/fdb x:0 0 0 2022-01-31T19:44:29.635 INFO:tasks.workunit.client.1.smithi171.stdout:3/685: write d4/dd/d27/d28/d35/d55/d8f/d92/dab/d60/fbd [423915,46697] 0 2022-01-31T19:44:29.635 INFO:tasks.workunit.client.1.smithi171.stdout:8/569: dread df/f83 [4194304,4194304] 0 2022-01-31T19:44:29.635 INFO:tasks.workunit.client.1.smithi171.stdout:8/570: chown f2 2 1 2022-01-31T19:44:29.635 INFO:tasks.workunit.client.1.smithi171.stdout:8/571: dread - df/d18/d79/fa1 zero size 2022-01-31T19:44:29.636 INFO:tasks.workunit.client.1.smithi171.stdout:8/572: fsync f2 0 2022-01-31T19:44:29.636 INFO:tasks.workunit.client.1.smithi171.stdout:6/640: unlink dc/d19/l2d 0 2022-01-31T19:44:29.638 INFO:tasks.workunit.client.1.smithi171.stdout:4/544: readlink d2/d3/d5/d91/d7e/l1b 0 2022-01-31T19:44:29.638 INFO:tasks.workunit.client.1.smithi171.stdout:4/545: fsync d2/d3/d5/dc/f19 0 2022-01-31T19:44:29.638 INFO:tasks.workunit.client.1.smithi171.stdout:4/546: write d2/d3/d5/d41/f47 [73365,88335] 0 2022-01-31T19:44:29.638 INFO:tasks.workunit.client.1.smithi171.stdout:4/547: write d2/f5b [432710,16049] 0 2022-01-31T19:44:29.642 INFO:tasks.workunit.client.1.smithi171.stdout:3/686: creat d4/dd/d27/d28/d35/d55/d6c/fdc x:0 0 0 2022-01-31T19:44:29.643 INFO:tasks.workunit.client.1.smithi171.stdout:8/573: rename df/d18/d19/d36/d70/c9b to df/d18/d79/da2/cc3 0 2022-01-31T19:44:29.644 INFO:tasks.workunit.client.1.smithi171.stdout:6/641: symlink dc/d19/d29/da7/ld8 0 2022-01-31T19:44:29.645 INFO:tasks.workunit.client.1.smithi171.stdout:4/548: rename d2/d3/d5/d70/c97 to d2/d3/d5/d41/cb4 0 2022-01-31T19:44:29.646 INFO:tasks.workunit.client.1.smithi171.stdout:8/574: creat df/d18/d79/d7a/dc0/fc4 x:0 0 0 2022-01-31T19:44:29.648 INFO:tasks.workunit.client.1.smithi171.stdout:8/575: symlink df/d18/d19/d36/lc5 0 2022-01-31T19:44:29.652 INFO:tasks.workunit.client.1.smithi171.stdout:6/642: dread dc/d19/d29/d27/d37/d3b/d67/f9f [0,4194304] 0 2022-01-31T19:44:29.654 INFO:tasks.workunit.client.1.smithi171.stdout:6/643: mknod dc/d19/d29/d27/d37/d68/d99/cd9 0 2022-01-31T19:44:29.654 INFO:tasks.workunit.client.1.smithi171.stdout:6/644: dread - dc/d18/fd7 zero size 2022-01-31T19:44:29.654 INFO:tasks.workunit.client.1.smithi171.stdout:6/645: chown dc/d21/d4f/db5/cba 662109238 1 2022-01-31T19:44:29.656 INFO:tasks.workunit.client.1.smithi171.stdout:6/646: rename dc/d18/fd7 to dc/d19/d30/fda 0 2022-01-31T19:44:29.656 INFO:tasks.workunit.client.1.smithi171.stdout:6/647: readlink dc/d19/d29/d27/d37/l48 0 2022-01-31T19:44:29.657 INFO:tasks.workunit.client.1.smithi171.stdout:6/648: rmdir dc/d19/d29/d27/d37 39 2022-01-31T19:44:29.657 INFO:tasks.workunit.client.1.smithi171.stdout:6/649: stat dc/d19/d29/d27/d37/d3b/d67/dd1/fd3 0 2022-01-31T19:44:29.658 INFO:tasks.workunit.client.1.smithi171.stdout:6/650: rename dc/d21/d4f/db5 to dc/d19/d29/daf/ddb 0 2022-01-31T19:44:29.659 INFO:tasks.workunit.client.1.smithi171.stdout:6/651: read dc/d19/d29/da7/fb8 [295687,46781] 0 2022-01-31T19:44:29.659 INFO:tasks.workunit.client.1.smithi171.stdout:6/652: write dc/d21/fab [443785,115264] 0 2022-01-31T19:44:29.659 INFO:tasks.workunit.client.1.smithi171.stdout:6/653: creat dc/d1e/d3f/fdc x:0 0 0 2022-01-31T19:44:29.660 INFO:tasks.workunit.client.1.smithi171.stdout:6/654: creat dc/d19/d29/d27/d37/d3b/d67/dd1/fdd x:0 0 0 2022-01-31T19:44:29.661 INFO:tasks.workunit.client.1.smithi171.stdout:6/655: creat dc/d19/d30/fde x:0 0 0 2022-01-31T19:44:29.667 INFO:tasks.workunit.client.1.smithi171.stdout:8/576: dread df/f84 [0,4194304] 0 2022-01-31T19:44:29.667 INFO:tasks.workunit.client.1.smithi171.stdout:6/656: write dc/d19/d30/f77 [5512241,58053] 0 2022-01-31T19:44:29.668 INFO:tasks.workunit.client.1.smithi171.stdout:8/577: rename df/d27/d5c/l66 to df/d18/d79/d7a/dc0/lc6 0 2022-01-31T19:44:29.670 INFO:tasks.workunit.client.1.smithi171.stdout:6/657: write dc/d19/d29/d27/f3a [6449195,127394] 0 2022-01-31T19:44:29.671 INFO:tasks.workunit.client.1.smithi171.stdout:8/578: unlink df/d18/d19/d46/cc1 0 2022-01-31T19:44:29.672 INFO:tasks.workunit.client.1.smithi171.stdout:4/549: dwrite d2/d3/d5/d3b/f55 [8388608,4194304] 0 2022-01-31T19:44:29.672 INFO:tasks.workunit.client.1.smithi171.stdout:4/550: getdents d2/d3/d5/d91/d7e/d12/d15/d57/dad 0 2022-01-31T19:44:29.673 INFO:tasks.workunit.client.1.smithi171.stdout:4/551: dread - d2/d3/d5/d3b/f92 zero size 2022-01-31T19:44:29.673 INFO:tasks.workunit.client.1.smithi171.stdout:4/552: readlink d2/d3/d5/dc/d4b/l94 0 2022-01-31T19:44:29.673 INFO:tasks.workunit.client.1.smithi171.stdout:4/553: chown d2/d3/d5/d91/d7e/d12/d15 32265631 1 2022-01-31T19:44:29.674 INFO:tasks.workunit.client.1.smithi171.stdout:4/554: creat d2/d3/d5/d70/d7b/fb5 x:0 0 0 2022-01-31T19:44:29.674 INFO:tasks.workunit.client.1.smithi171.stdout:3/687: dwrite d4/dd/d27/d28/d35/d55/d8f/d92/dab/f1b [0,4194304] 0 2022-01-31T19:44:29.674 INFO:tasks.workunit.client.1.smithi171.stdout:8/579: creat df/d18/d79/fc7 x:0 0 0 2022-01-31T19:44:29.688 INFO:tasks.workunit.client.1.smithi171.stdout:6/658: dread dc/d19/d29/f63 [0,4194304] 0 2022-01-31T19:44:29.688 INFO:tasks.workunit.client.1.smithi171.stdout:6/659: dread - dc/d1e/fa4 zero size 2022-01-31T19:44:29.689 INFO:tasks.workunit.client.1.smithi171.stdout:6/660: creat dc/d19/dae/fdf x:0 0 0 2022-01-31T19:44:29.690 INFO:tasks.workunit.client.1.smithi171.stdout:6/661: write dc/d19/d29/d6a/fbb [1677447,63548] 0 2022-01-31T19:44:29.693 INFO:tasks.workunit.client.1.smithi171.stdout:6/662: read dc/d18/f2c [3851727,118011] 0 2022-01-31T19:44:29.693 INFO:tasks.workunit.client.1.smithi171.stdout:6/663: write dc/d21/fa2 [5219498,105082] 0 2022-01-31T19:44:29.696 INFO:tasks.workunit.client.1.smithi171.stdout:6/664: rmdir dc/d19/d29/daf 39 2022-01-31T19:44:29.696 INFO:tasks.workunit.client.1.smithi171.stdout:6/665: chown dc/d1e/d3f/cbd 0 1 2022-01-31T19:44:29.696 INFO:tasks.workunit.client.1.smithi171.stdout:6/666: read - dc/d21/f98 zero size 2022-01-31T19:44:29.698 INFO:tasks.workunit.client.1.smithi171.stdout:6/667: dread dc/d18/f39 [0,4194304] 0 2022-01-31T19:44:29.699 INFO:tasks.workunit.client.1.smithi171.stdout:6/668: stat dc/d19/d29 0 2022-01-31T19:44:29.699 INFO:tasks.workunit.client.1.smithi171.stdout:3/688: dwrite d4/fa [0,4194304] 0 2022-01-31T19:44:29.699 INFO:tasks.workunit.client.1.smithi171.stdout:8/580: dwrite df/d1a/f58 [4194304,4194304] 0 2022-01-31T19:44:29.700 INFO:tasks.workunit.client.1.smithi171.stdout:3/689: write d4/dd/d27/d28/d35/fd2 [2148806,78991] 0 2022-01-31T19:44:29.701 INFO:tasks.workunit.client.1.smithi171.stdout:6/669: rename dc/d19/f1a to dc/fe0 0 2022-01-31T19:44:29.707 INFO:tasks.workunit.client.1.smithi171.stdout:4/555: rmdir d2/d3/d5/d70 39 2022-01-31T19:44:29.707 INFO:tasks.workunit.client.1.smithi171.stdout:4/556: write d2/d3/d5/d41/d6b/fa9 [592351,114894] 0 2022-01-31T19:44:29.708 INFO:tasks.workunit.client.1.smithi171.stdout:8/581: mkdir df/d18/d19/d36/dc8 0 2022-01-31T19:44:29.710 INFO:tasks.workunit.client.1.smithi171.stdout:3/690: mkdir d4/d21/d3a/ddd 0 2022-01-31T19:44:29.720 INFO:tasks.workunit.client.1.smithi171.stdout:6/670: symlink dc/d19/d29/daf/le1 0 2022-01-31T19:44:29.720 INFO:tasks.workunit.client.1.smithi171.stdout:4/557: mkdir d2/d3/d5/d91/d7e/d12/db6 0 2022-01-31T19:44:29.722 INFO:tasks.workunit.client.1.smithi171.stdout:8/582: link df/d18/d79/d7a/l9f df/d18/d19/d21/d67/lc9 0 2022-01-31T19:44:29.722 INFO:tasks.workunit.client.1.smithi171.stdout:8/583: dread - df/d18/f41 zero size 2022-01-31T19:44:29.722 INFO:tasks.workunit.client.1.smithi171.stdout:8/584: stat df/d18/d19/d36/d70/d7e 0 2022-01-31T19:44:29.723 INFO:tasks.workunit.client.1.smithi171.stdout:3/691: mkdir d4/dd/d27/d28/d35/d55/d8f/dde 0 2022-01-31T19:44:29.725 INFO:tasks.workunit.client.1.smithi171.stdout:4/558: creat d2/d3/d5/d91/d7e/d12/db6/fb7 x:0 0 0 2022-01-31T19:44:29.726 INFO:tasks.workunit.client.1.smithi171.stdout:4/559: fdatasync d2/d3/d5/d91/d7e/d12/f5a 0 2022-01-31T19:44:29.727 INFO:tasks.workunit.client.1.smithi171.stdout:8/585: mkdir df/d27/d7b/dca 0 2022-01-31T19:44:29.731 INFO:tasks.workunit.client.1.smithi171.stdout:4/560: symlink d2/d3/d5/d91/d7e/d12/lb8 0 2022-01-31T19:44:29.734 INFO:tasks.workunit.client.1.smithi171.stdout:8/586: link df/d27/d90/fb4 df/d18/d19/d21/d67/fcb 0 2022-01-31T19:44:29.737 INFO:tasks.workunit.client.1.smithi171.stdout:8/587: rename df/d18/d19/d36/dc8 to df/dcc 0 2022-01-31T19:44:29.737 INFO:tasks.workunit.client.1.smithi171.stdout:8/588: truncate df/d18/d79/d7a/dc0/fc4 35555 0 2022-01-31T19:44:29.738 INFO:tasks.workunit.client.1.smithi171.stdout:4/561: rmdir d2/d3/d5/dc/d4b 39 2022-01-31T19:44:29.745 INFO:tasks.workunit.client.1.smithi171.stdout:4/562: creat d2/d3/d5/dc/d4b/d56/fb9 x:0 0 0 2022-01-31T19:44:29.745 INFO:tasks.workunit.client.1.smithi171.stdout:4/563: creat d2/fba x:0 0 0 2022-01-31T19:44:29.747 INFO:tasks.workunit.client.1.smithi171.stdout:8/589: dread df/d27/f64 [0,4194304] 0 2022-01-31T19:44:29.747 INFO:tasks.workunit.client.1.smithi171.stdout:8/590: chown df/d18/f4e 6 1 2022-01-31T19:44:29.747 INFO:tasks.workunit.client.1.smithi171.stdout:8/591: stat df/d18/d19/f61 0 2022-01-31T19:44:29.747 INFO:tasks.workunit.client.1.smithi171.stdout:4/564: symlink d2/d3/d5/d41/d6b/lbb 0 2022-01-31T19:44:29.748 INFO:tasks.workunit.client.1.smithi171.stdout:8/592: mkdir df/d18/d19/d21/dcd 0 2022-01-31T19:44:29.748 INFO:tasks.workunit.client.1.smithi171.stdout:8/593: fsync df/d18/d19/d21/fb6 0 2022-01-31T19:44:29.749 INFO:tasks.workunit.client.1.smithi171.stdout:4/565: mkdir d2/d3/d5/dc/d4b/dbc 0 2022-01-31T19:44:29.749 INFO:tasks.workunit.client.1.smithi171.stdout:4/566: chown d2/d3/d5/d3b/c67 18 1 2022-01-31T19:44:29.750 INFO:tasks.workunit.client.1.smithi171.stdout:6/671: dwrite dc/d19/d30/f92 [0,4194304] 0 2022-01-31T19:44:29.751 INFO:tasks.workunit.client.1.smithi171.stdout:8/594: creat df/d18/fce x:0 0 0 2022-01-31T19:44:29.751 INFO:tasks.workunit.client.1.smithi171.stdout:8/595: dread - df/d27/fa3 zero size 2022-01-31T19:44:29.752 INFO:tasks.workunit.client.1.smithi171.stdout:8/596: readlink df/d27/l65 0 2022-01-31T19:44:29.752 INFO:tasks.workunit.client.1.smithi171.stdout:3/692: dwrite d4/fc6 [0,4194304] 0 2022-01-31T19:44:29.761 INFO:tasks.workunit.client.1.smithi171.stdout:3/693: symlink d4/dd/d27/d28/d35/d55/d8f/d92/dab/d6b/ldf 0 2022-01-31T19:44:29.770 INFO:tasks.workunit.client.1.smithi171.stdout:7/503: sync 2022-01-31T19:44:29.771 INFO:tasks.workunit.client.1.smithi171.stdout:7/504: symlink d1/lac 0 2022-01-31T19:44:29.772 INFO:tasks.workunit.client.1.smithi171.stdout:7/505: fsync d1/d26/f71 0 2022-01-31T19:44:29.772 INFO:tasks.workunit.client.1.smithi171.stdout:7/506: mknod d1/d1e/d2a/d29/d31/d30/d3c/d7e/cad 0 2022-01-31T19:44:29.773 INFO:tasks.workunit.client.1.smithi171.stdout:7/507: getdents d1/d1e 0 2022-01-31T19:44:29.800 INFO:tasks.workunit.client.1.smithi171.stdout:6/672: dwrite dc/d19/d29/d27/d37/d3b/f8e [0,4194304] 0 2022-01-31T19:44:29.804 INFO:tasks.workunit.client.1.smithi171.stdout:4/567: dwrite d2/f14 [8388608,4194304] 0 2022-01-31T19:44:29.804 INFO:tasks.workunit.client.1.smithi171.stdout:4/568: creat d2/d3/d5/d41/d6b/fbd x:0 0 0 2022-01-31T19:44:29.805 INFO:tasks.workunit.client.1.smithi171.stdout:4/569: chown d2/d3/d5/d91/da6 2 1 2022-01-31T19:44:29.805 INFO:tasks.workunit.client.1.smithi171.stdout:4/570: write d2/d3/d5/d91/d7e/f83 [5703262,43575] 0 2022-01-31T19:44:29.806 INFO:tasks.workunit.client.1.smithi171.stdout:8/597: dwrite df/d18/d19/d21/d88/f92 [0,4194304] 0 2022-01-31T19:44:29.806 INFO:tasks.workunit.client.1.smithi171.stdout:3/694: dwrite d4/dd/d27/d28/d35/fa9 [0,4194304] 0 2022-01-31T19:44:29.808 INFO:tasks.workunit.client.1.smithi171.stdout:7/508: dwrite d1/d1e/d2a/f3b [0,4194304] 0 2022-01-31T19:44:29.808 INFO:tasks.workunit.client.1.smithi171.stdout:7/509: write d1/d1e/d2a/d29/d31/f51 [2187482,10554] 0 2022-01-31T19:44:29.808 INFO:tasks.workunit.client.1.smithi171.stdout:6/673: dread dc/d19/d30/f77 [4194304,4194304] 0 2022-01-31T19:44:29.809 INFO:tasks.workunit.client.1.smithi171.stdout:4/571: dread d2/d3/d5/dc/f39 [0,4194304] 0 2022-01-31T19:44:29.815 INFO:tasks.workunit.client.1.smithi171.stdout:3/695: creat d4/dd/d27/d28/db6/dc0/fe0 x:0 0 0 2022-01-31T19:44:29.819 INFO:tasks.workunit.client.1.smithi171.stdout:6/674: link dc/d21/fa2 dc/d18/fe2 0 2022-01-31T19:44:29.820 INFO:tasks.workunit.client.1.smithi171.stdout:8/598: rmdir df/d18/d79/d7a/dc0 39 2022-01-31T19:44:29.820 INFO:tasks.workunit.client.1.smithi171.stdout:8/599: chown df/d27/d90 7660 1 2022-01-31T19:44:29.820 INFO:tasks.workunit.client.1.smithi171.stdout:8/600: readlink df/d18/d79/d7a/dc0/lc6 0 2022-01-31T19:44:29.821 INFO:tasks.workunit.client.1.smithi171.stdout:8/601: write df/d18/d19/f1b [1432950,20965] 0 2022-01-31T19:44:29.822 INFO:tasks.workunit.client.1.smithi171.stdout:3/696: creat d4/dd/d27/d28/d35/d55/d8f/d92/fe1 x:0 0 0 2022-01-31T19:44:29.824 INFO:tasks.workunit.client.1.smithi171.stdout:6/675: link dc/d19/d29/d27/d37/d3b/d67/dd1/fdd dc/fe3 0 2022-01-31T19:44:29.824 INFO:tasks.workunit.client.1.smithi171.stdout:6/676: creat dc/d1e/d3f/fe4 x:0 0 0 2022-01-31T19:44:29.824 INFO:tasks.workunit.client.1.smithi171.stdout:6/677: fdatasync dc/d18/f25 0 2022-01-31T19:44:29.833 INFO:tasks.workunit.client.1.smithi171.stdout:6/678: read dc/d19/d29/daf/ddb/fcd [2502340,96501] 0 2022-01-31T19:44:29.834 INFO:tasks.workunit.client.1.smithi171.stdout:4/572: dwrite d2/d3/d5/d3b/faf [0,4194304] 0 2022-01-31T19:44:29.835 INFO:tasks.workunit.client.1.smithi171.stdout:7/510: dwrite d1/f36 [4194304,4194304] 0 2022-01-31T19:44:29.836 INFO:tasks.workunit.client.1.smithi171.stdout:8/602: dread df/f20 [0,4194304] 0 2022-01-31T19:44:29.836 INFO:tasks.workunit.client.1.smithi171.stdout:8/603: fsync df/d18/d79/d7a/dc0/fc4 0 2022-01-31T19:44:29.836 INFO:tasks.workunit.client.1.smithi171.stdout:8/604: read - df/d2a/faf zero size 2022-01-31T19:44:29.838 INFO:tasks.workunit.client.1.smithi171.stdout:4/573: link d2/d3/d5/d41/d6b/f99 d2/d3/d5/d91/fbe 0 2022-01-31T19:44:29.844 INFO:tasks.workunit.client.1.smithi171.stdout:8/605: write df/f83 [4158022,73425] 0 2022-01-31T19:44:29.849 INFO:tasks.workunit.client.1.smithi171.stdout:7/511: dread d1/d26/d57/fa5 [0,4194304] 0 2022-01-31T19:44:29.849 INFO:tasks.workunit.client.1.smithi171.stdout:7/512: readlink d1/d1e/d2a/d29/d31/d30/d3c/l72 0 2022-01-31T19:44:29.850 INFO:tasks.workunit.client.1.smithi171.stdout:7/513: mknod d1/d1e/d2a/d29/d31/d30/d3c/d79/cae 0 2022-01-31T19:44:29.850 INFO:tasks.workunit.client.1.smithi171.stdout:7/514: stat d1/d1e/d2a/d29/d31/f6a 0 2022-01-31T19:44:29.852 INFO:tasks.workunit.client.1.smithi171.stdout:3/697: dwrite d4/dd/d27/d28/d35/d55/d8f/d92/dab/f2b [0,4194304] 0 2022-01-31T19:44:29.853 INFO:tasks.workunit.client.1.smithi171.stdout:3/698: chown d4/dd/d87/d8e/cbc 275 1 2022-01-31T19:44:29.854 INFO:tasks.workunit.client.1.smithi171.stdout:3/699: mknod d4/dd/d27/d28/ce2 0 2022-01-31T19:44:29.858 INFO:tasks.workunit.client.1.smithi171.stdout:6/679: dwrite dc/d19/d29/da7/f7e [0,4194304] 0 2022-01-31T19:44:29.859 INFO:tasks.workunit.client.1.smithi171.stdout:3/700: dread d4/dd/d39/d51/fb4 [0,4194304] 0 2022-01-31T19:44:29.860 INFO:tasks.workunit.client.1.smithi171.stdout:8/606: dwrite f3 [0,4194304] 0 2022-01-31T19:44:29.861 INFO:tasks.workunit.client.1.smithi171.stdout:6/680: mknod dc/d18/d86/ce5 0 2022-01-31T19:44:29.861 INFO:tasks.workunit.client.1.smithi171.stdout:4/574: write d2/d3/d5/d91/d7e/f27 [3056079,112914] 0 2022-01-31T19:44:29.862 INFO:tasks.workunit.client.1.smithi171.stdout:3/701: rename d4/dd/d39/d51/f53 to d4/dd/d27/d28/db6/fe3 0 2022-01-31T19:44:29.863 INFO:tasks.workunit.client.1.smithi171.stdout:8/607: getdents df/d18/d19/d21 0 2022-01-31T19:44:29.864 INFO:tasks.workunit.client.1.smithi171.stdout:6/681: truncate dc/d19/d30/f92 2594514 0 2022-01-31T19:44:29.865 INFO:tasks.workunit.client.1.smithi171.stdout:4/575: creat d2/d3/d5/d41/fbf x:0 0 0 2022-01-31T19:44:29.868 INFO:tasks.workunit.client.1.smithi171.stdout:8/608: unlink df/c13 0 2022-01-31T19:44:29.870 INFO:tasks.workunit.client.1.smithi171.stdout:4/576: rename d2/d3/d5/d91/d77/f8f to d2/d3/d5/d91/d7e/d12/fc0 0 2022-01-31T19:44:29.871 INFO:tasks.workunit.client.1.smithi171.stdout:7/515: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [4194304,4194304] 0 2022-01-31T19:44:29.871 INFO:tasks.workunit.client.1.smithi171.stdout:7/516: fdatasync d1/d1e/d2a/d29/d77/f8c 0 2022-01-31T19:44:29.872 INFO:tasks.workunit.client.1.smithi171.stdout:3/702: getdents d4/d21/d3a/db5 0 2022-01-31T19:44:29.872 INFO:tasks.workunit.client.1.smithi171.stdout:3/703: dread - d4/dd/d27/d28/d35/d55/fb9 zero size 2022-01-31T19:44:29.873 INFO:tasks.workunit.client.1.smithi171.stdout:3/704: write d4/dd/d87/d8e/f88 [381572,119787] 0 2022-01-31T19:44:29.874 INFO:tasks.workunit.client.1.smithi171.stdout:3/705: write d4/d21/f69 [2446164,122676] 0 2022-01-31T19:44:29.881 INFO:tasks.workunit.client.1.smithi171.stdout:7/517: symlink d1/d1e/d2a/d29/d77/laf 0 2022-01-31T19:44:29.883 INFO:tasks.workunit.client.1.smithi171.stdout:7/518: rename d1/d26/d57/f89 to d1/d26/d81/fb0 0 2022-01-31T19:44:29.887 INFO:tasks.workunit.client.1.smithi171.stdout:4/577: dread d2/d3/d5/f2e [0,4194304] 0 2022-01-31T19:44:29.887 INFO:tasks.workunit.client.1.smithi171.stdout:4/578: dread - d2/d3/d5/d41/fbf zero size 2022-01-31T19:44:29.889 INFO:tasks.workunit.client.1.smithi171.stdout:4/579: creat d2/d3/d5/fc1 x:0 0 0 2022-01-31T19:44:29.890 INFO:tasks.workunit.client.1.smithi171.stdout:4/580: symlink d2/d3/d5/d91/d7e/lc2 0 2022-01-31T19:44:29.894 INFO:tasks.workunit.client.1.smithi171.stdout:4/581: creat d2/d3/d5/d70/d7c/fc3 x:0 0 0 2022-01-31T19:44:29.896 INFO:tasks.workunit.client.1.smithi171.stdout:6/682: dwrite dc/d19/d30/f3d [0,4194304] 0 2022-01-31T19:44:29.898 INFO:tasks.workunit.client.1.smithi171.stdout:3/706: dread d4/dd/d27/d28/f91 [0,4194304] 0 2022-01-31T19:44:29.898 INFO:tasks.workunit.client.1.smithi171.stdout:7/519: dread d1/d1b/f22 [4194304,4194304] 0 2022-01-31T19:44:29.898 INFO:tasks.workunit.client.1.smithi171.stdout:7/520: readlink d1/d1e/d2a/d29/l33 0 2022-01-31T19:44:29.898 INFO:tasks.workunit.client.1.smithi171.stdout:3/707: truncate d4/dd/d87/d8e/faf 4861878 0 2022-01-31T19:44:29.899 INFO:tasks.workunit.client.1.smithi171.stdout:7/521: chown d1/d1e/d2a/d29/d31/d30/d6b/f86 1918630 1 2022-01-31T19:44:29.899 INFO:tasks.workunit.client.1.smithi171.stdout:3/708: dread - d4/dd/d4b/fcf zero size 2022-01-31T19:44:29.899 INFO:tasks.workunit.client.1.smithi171.stdout:7/522: chown d1/d1e/d2a/d29/d31/d30/c8a 47 1 2022-01-31T19:44:29.900 INFO:tasks.workunit.client.1.smithi171.stdout:7/523: chown d1/d26/d57/l68 77887 1 2022-01-31T19:44:29.900 INFO:tasks.workunit.client.1.smithi171.stdout:7/524: chown d1/d1b/ca9 992 1 2022-01-31T19:44:29.900 INFO:tasks.workunit.client.1.smithi171.stdout:6/683: creat dc/d19/d29/daf/ddb/fe6 x:0 0 0 2022-01-31T19:44:29.901 INFO:tasks.workunit.client.1.smithi171.stdout:3/709: symlink d4/d21/le4 0 2022-01-31T19:44:29.901 INFO:tasks.workunit.client.1.smithi171.stdout:4/582: rmdir d2/d3/d5/d91/d7e/d12/d15 39 2022-01-31T19:44:29.902 INFO:tasks.workunit.client.1.smithi171.stdout:7/525: link d1/d1e/d2a/d29/d31/d30/f39 d1/d1e/d2a/d58/fb1 0 2022-01-31T19:44:29.902 INFO:tasks.workunit.client.1.smithi171.stdout:7/526: creat d1/d1e/d2a/d29/fb2 x:0 0 0 2022-01-31T19:44:29.903 INFO:tasks.workunit.client.1.smithi171.stdout:7/527: write d1/d1e/d2a/d29/d31/d30/d3c/f56 [531935,125900] 0 2022-01-31T19:44:29.904 INFO:tasks.workunit.client.1.smithi171.stdout:4/583: mkdir d2/d3/d5/d91/d7e/d12/d15/dc4 0 2022-01-31T19:44:29.904 INFO:tasks.workunit.client.1.smithi171.stdout:4/584: fsync d2/d3/d5/d91/f76 0 2022-01-31T19:44:29.906 INFO:tasks.workunit.client.1.smithi171.stdout:7/528: symlink d1/d1e/d2a/d29/lb3 0 2022-01-31T19:44:29.906 INFO:tasks.workunit.client.1.smithi171.stdout:7/529: creat d1/d1e/d2a/d58/fb4 x:0 0 0 2022-01-31T19:44:29.906 INFO:tasks.workunit.client.1.smithi171.stdout:7/530: readlink d1/d1e/d2a/l84 0 2022-01-31T19:44:29.906 INFO:tasks.workunit.client.1.smithi171.stdout:7/531: chown d1/d1b/l96 723111 1 2022-01-31T19:44:29.907 INFO:tasks.workunit.client.1.smithi171.stdout:3/710: getdents d4/dd/d27/d28/d35/d55 0 2022-01-31T19:44:29.907 INFO:tasks.workunit.client.1.smithi171.stdout:8/609: dwrite df/d18/d79/f8c [0,4194304] 0 2022-01-31T19:44:29.908 INFO:tasks.workunit.client.1.smithi171.stdout:4/585: unlink d2/d3/d5/d41/d6b/lbb 0 2022-01-31T19:44:29.908 INFO:tasks.workunit.client.1.smithi171.stdout:4/586: write d2/fba [353807,4699] 0 2022-01-31T19:44:29.909 INFO:tasks.workunit.client.1.smithi171.stdout:7/532: symlink d1/d1e/d2a/d58/lb5 0 2022-01-31T19:44:29.913 INFO:tasks.workunit.client.1.smithi171.stdout:2/815: sync 2022-01-31T19:44:29.914 INFO:tasks.workunit.client.1.smithi171.stdout:2/816: write d7/d26/d63/f64 [5193,35107] 0 2022-01-31T19:44:29.924 INFO:tasks.workunit.client.1.smithi171.stdout:4/587: mknod d2/d3/d5/d41/cc5 0 2022-01-31T19:44:29.930 INFO:tasks.workunit.client.1.smithi171.stdout:2/817: creat d7/d55/df3/ffe x:0 0 0 2022-01-31T19:44:29.930 INFO:tasks.workunit.client.1.smithi171.stdout:8/610: truncate df/d18/d79/f8c 5781518 0 2022-01-31T19:44:29.932 INFO:tasks.workunit.client.1.smithi171.stdout:4/588: symlink d2/d3/d5/dc/d4b/lc6 0 2022-01-31T19:44:29.932 INFO:tasks.workunit.client.1.smithi171.stdout:4/589: write d2/d3/d5/d70/d7b/f93 [927246,65929] 0 2022-01-31T19:44:29.932 INFO:tasks.workunit.client.1.smithi171.stdout:3/711: rename d4/dd/d27/d28/d35/d55/d8f to d4/d21/de5 0 2022-01-31T19:44:29.933 INFO:tasks.workunit.client.1.smithi171.stdout:8/611: rmdir df/d5d 39 2022-01-31T19:44:29.936 INFO:tasks.workunit.client.1.smithi171.stdout:4/590: mkdir d2/d3/d5/d91/d7e/dc7 0 2022-01-31T19:44:29.937 INFO:tasks.workunit.client.1.smithi171.stdout:3/712: mknod d4/dd/d27/d28/d35/d55/ce6 0 2022-01-31T19:44:29.938 INFO:tasks.workunit.client.1.smithi171.stdout:4/591: truncate d2/d3/d5/f1a 4982363 0 2022-01-31T19:44:29.939 INFO:tasks.workunit.client.1.smithi171.stdout:4/592: creat d2/d3/d5/d41/d6b/fc8 x:0 0 0 2022-01-31T19:44:29.941 INFO:tasks.workunit.client.1.smithi171.stdout:2/818: dread d7/d26/de5/f9e [0,4194304] 0 2022-01-31T19:44:29.943 INFO:tasks.workunit.client.1.smithi171.stdout:4/593: dread d2/f3c [4194304,4194304] 0 2022-01-31T19:44:29.943 INFO:tasks.workunit.client.1.smithi171.stdout:2/819: unlink d7/d9/d1a/d45/c36 0 2022-01-31T19:44:29.944 INFO:tasks.workunit.client.1.smithi171.stdout:4/594: symlink d2/d3/d5/dc/d4b/dbc/lc9 0 2022-01-31T19:44:29.944 INFO:tasks.workunit.client.1.smithi171.stdout:4/595: fsync d2/d3/d5/fc1 0 2022-01-31T19:44:29.945 INFO:tasks.workunit.client.1.smithi171.stdout:2/820: creat d7/dd/d5e/d6b/d81/d8d/da2/fff x:0 0 0 2022-01-31T19:44:29.947 INFO:tasks.workunit.client.1.smithi171.stdout:9/701: sync 2022-01-31T19:44:29.947 INFO:tasks.workunit.client.1.smithi171.stdout:5/559: sync 2022-01-31T19:44:29.948 INFO:tasks.workunit.client.1.smithi171.stdout:0/718: sync 2022-01-31T19:44:29.948 INFO:tasks.workunit.client.1.smithi171.stdout:1/604: sync 2022-01-31T19:44:29.949 INFO:tasks.workunit.client.1.smithi171.stdout:4/596: symlink d2/d3/d5/d91/d7e/d12/db6/lca 0 2022-01-31T19:44:29.951 INFO:tasks.workunit.client.1.smithi171.stdout:9/702: mkdir dd/df/d15/d1a/d84/dca/df1 0 2022-01-31T19:44:29.951 INFO:tasks.workunit.client.1.smithi171.stdout:5/560: write d5/d90/db7/d15/d23/d67/f4c [1053253,89679] 0 2022-01-31T19:44:29.954 INFO:tasks.workunit.client.1.smithi171.stdout:0/719: mkdir d9/db/dc/d3f/db9/d10f 0 2022-01-31T19:44:29.957 INFO:tasks.workunit.client.1.smithi171.stdout:5/561: creat d5/d90/db7/d15/d23/d58/d9e/fbd x:0 0 0 2022-01-31T19:44:29.960 INFO:tasks.workunit.client.1.smithi171.stdout:0/720: getdents d9/db/dc2 0 2022-01-31T19:44:29.960 INFO:tasks.workunit.client.1.smithi171.stdout:5/562: getdents d5/db0 0 2022-01-31T19:44:29.961 INFO:tasks.workunit.client.1.smithi171.stdout:0/721: write d9/db/dc/d3f/db9/d7a/fdb [55509,100137] 0 2022-01-31T19:44:29.961 INFO:tasks.workunit.client.1.smithi171.stdout:0/722: write d9/db/f47 [748531,38950] 0 2022-01-31T19:44:29.962 INFO:tasks.workunit.client.1.smithi171.stdout:0/723: creat d9/db/d59/d71/f110 x:0 0 0 2022-01-31T19:44:29.963 INFO:tasks.workunit.client.1.smithi171.stdout:9/703: dread dd/df/d15/d38/d41/fb4 [0,4194304] 0 2022-01-31T19:44:29.964 INFO:tasks.workunit.client.1.smithi171.stdout:5/563: read d5/d90/db7/d15/d5d/d6a/f77 [418674,11119] 0 2022-01-31T19:44:29.964 INFO:tasks.workunit.client.1.smithi171.stdout:5/564: chown d5/d90/db7/d15/d23/d58/f79 202862 1 2022-01-31T19:44:29.965 INFO:tasks.workunit.client.1.smithi171.stdout:5/565: fdatasync d5/d90/db7/d15/f64 0 2022-01-31T19:44:29.965 INFO:tasks.workunit.client.1.smithi171.stdout:9/704: mknod dd/df/d3f/d73/cf2 0 2022-01-31T19:44:29.966 INFO:tasks.workunit.client.1.smithi171.stdout:1/605: dread dd/df/d19/d1b/d1e/d35/f70 [4194304,4194304] 0 2022-01-31T19:44:29.967 INFO:tasks.workunit.client.1.smithi171.stdout:5/566: unlink d5/d90/db7/d15/d23/d6b/lad 0 2022-01-31T19:44:29.967 INFO:tasks.workunit.client.1.smithi171.stdout:5/567: unlink d5/f9f 0 2022-01-31T19:44:29.967 INFO:tasks.workunit.client.1.smithi171.stdout:9/705: mknod dd/df/d15/d1a/d64/cf3 0 2022-01-31T19:44:29.968 INFO:tasks.workunit.client.1.smithi171.stdout:9/706: truncate dd/df/fa0 2846219 0 2022-01-31T19:44:29.968 INFO:tasks.workunit.client.1.smithi171.stdout:9/707: truncate dd/df/d15/d1a/d64/fac 23324 0 2022-01-31T19:44:29.968 INFO:tasks.workunit.client.1.smithi171.stdout:9/708: chown dd/df/d15/d1a/l43 7723908 1 2022-01-31T19:44:29.968 INFO:tasks.workunit.client.1.smithi171.stdout:9/709: readlink dd/df/d3f/d73/d96/d9f/de6/da8/lbd 0 2022-01-31T19:44:29.969 INFO:tasks.workunit.client.1.smithi171.stdout:1/606: mkdir dd/df/d19/d1b/d1e/d64/db9 0 2022-01-31T19:44:29.969 INFO:tasks.workunit.client.1.smithi171.stdout:1/607: readlink dd/d23/l4f 0 2022-01-31T19:44:29.970 INFO:tasks.workunit.client.1.smithi171.stdout:9/710: read dd/df/f17 [640953,28458] 0 2022-01-31T19:44:29.970 INFO:tasks.workunit.client.1.smithi171.stdout:5/568: getdents d5/d90/db7/d15/d23/d3d 0 2022-01-31T19:44:29.971 INFO:tasks.workunit.client.1.smithi171.stdout:9/711: mknod dd/df/d3f/de3/cf4 0 2022-01-31T19:44:29.971 INFO:tasks.workunit.client.1.smithi171.stdout:9/712: chown dd/df/d15/d38/d4d/d80/fee 1889651 1 2022-01-31T19:44:29.971 INFO:tasks.workunit.client.1.smithi171.stdout:9/713: chown dd/df/d15/da2/fd9 5030858 1 2022-01-31T19:44:29.972 INFO:tasks.workunit.client.1.smithi171.stdout:5/569: symlink d5/d90/db7/d15/d23/lbe 0 2022-01-31T19:44:29.973 INFO:tasks.workunit.client.1.smithi171.stdout:5/570: mknod d5/d90/db7/d15/d23/d58/cbf 0 2022-01-31T19:44:29.973 INFO:tasks.workunit.client.1.smithi171.stdout:5/571: rmdir d5/d59 39 2022-01-31T19:44:29.974 INFO:tasks.workunit.client.1.smithi171.stdout:5/572: chown d5/d90/db7/d15/d5d/l62 2018 1 2022-01-31T19:44:29.974 INFO:tasks.workunit.client.1.smithi171.stdout:5/573: dread - d5/fb5 zero size 2022-01-31T19:44:29.974 INFO:tasks.workunit.client.1.smithi171.stdout:5/574: fdatasync d5/f4e 0 2022-01-31T19:44:29.974 INFO:tasks.workunit.client.1.smithi171.stdout:5/575: fdatasync d5/d90/db7/d15/d5d/d6a/f73 0 2022-01-31T19:44:29.978 INFO:tasks.workunit.client.1.smithi171.stdout:5/576: write d5/d90/db7/dd/f20 [2744975,42889] 0 2022-01-31T19:44:29.980 INFO:tasks.workunit.client.1.smithi171.stdout:5/577: creat d5/d90/db7/d15/d23/d6b/fc0 x:0 0 0 2022-01-31T19:44:29.980 INFO:tasks.workunit.client.1.smithi171.stdout:5/578: chown d5/d90/db7/d15/d23/f41 357 1 2022-01-31T19:44:29.980 INFO:tasks.workunit.client.1.smithi171.stdout:5/579: fsync d5/d90/db7/d15/f3f 0 2022-01-31T19:44:29.995 INFO:tasks.workunit.client.1.smithi171.stdout:5/580: dread d5/f57 [0,4194304] 0 2022-01-31T19:44:29.996 INFO:tasks.workunit.client.1.smithi171.stdout:5/581: creat d5/d90/db7/fc1 x:0 0 0 2022-01-31T19:44:29.996 INFO:tasks.workunit.client.1.smithi171.stdout:5/582: fdatasync d5/d90/db7/d15/d23/d3d/fb2 0 2022-01-31T19:44:29.997 INFO:tasks.workunit.client.1.smithi171.stdout:5/583: rmdir d5/d90/db7/d15/d5d/d6f/da1 0 2022-01-31T19:44:29.998 INFO:tasks.workunit.client.1.smithi171.stdout:5/584: chown d5/d90/db7/d15/f33 10962724 1 2022-01-31T19:44:29.999 INFO:tasks.workunit.client.1.smithi171.stdout:5/585: symlink d5/d90/db7/d15/d5d/d6a/lc2 0 2022-01-31T19:44:30.000 INFO:tasks.workunit.client.1.smithi171.stdout:5/586: fdatasync d5/d90/db7/f21 0 2022-01-31T19:44:30.001 INFO:tasks.workunit.client.1.smithi171.stdout:5/587: creat d5/d90/db7/d15/d80/fc3 x:0 0 0 2022-01-31T19:44:30.008 INFO:tasks.workunit.client.1.smithi171.stdout:5/588: dread d5/d90/f78 [0,4194304] 0 2022-01-31T19:44:30.008 INFO:tasks.workunit.client.1.smithi171.stdout:5/589: chown d5/d90/db7/d15/d23/d3d 359659 1 2022-01-31T19:44:30.010 INFO:tasks.workunit.client.1.smithi171.stdout:5/590: dread d5/d90/db7/d15/d23/f25 [0,4194304] 0 2022-01-31T19:44:30.011 INFO:tasks.workunit.client.1.smithi171.stdout:5/591: chown d5/d90/db7/d15/d23/lbe 392 1 2022-01-31T19:44:30.022 INFO:tasks.workunit.client.1.smithi171.stdout:4/597: dwrite d2/d3/f25 [0,4194304] 0 2022-01-31T19:44:30.022 INFO:tasks.workunit.client.1.smithi171.stdout:8/612: dwrite df/d18/d19/d82/f97 [0,4194304] 0 2022-01-31T19:44:30.023 INFO:tasks.workunit.client.1.smithi171.stdout:3/713: dwrite d4/d21/de5/d92/dab/d52/f54 [4194304,4194304] 0 2022-01-31T19:44:30.025 INFO:tasks.workunit.client.1.smithi171.stdout:9/714: dwrite dd/f3c [0,4194304] 0 2022-01-31T19:44:30.026 INFO:tasks.workunit.client.1.smithi171.stdout:9/715: fdatasync dd/df/d3f/d73/d96/d9f/de6/f95 0 2022-01-31T19:44:30.026 INFO:tasks.workunit.client.1.smithi171.stdout:9/716: chown l8 1 1 2022-01-31T19:44:30.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/717: stat dd/df/d3f/l5e 0 2022-01-31T19:44:30.027 INFO:tasks.workunit.client.1.smithi171.stdout:9/718: fdatasync dd/df/d3f/fd1 0 2022-01-31T19:44:30.028 INFO:tasks.workunit.client.1.smithi171.stdout:1/608: dwrite dd/df/d19/d1b/d1e/d64/f58 [12582912,4194304] 0 2022-01-31T19:44:30.028 INFO:tasks.workunit.client.1.smithi171.stdout:4/598: link d2/d3/d5/dc/f7a d2/d3/d5/d91/d77/da5/fcb 0 2022-01-31T19:44:30.029 INFO:tasks.workunit.client.1.smithi171.stdout:4/599: truncate d2/d7f/fa8 979030 0 2022-01-31T19:44:30.030 INFO:tasks.workunit.client.1.smithi171.stdout:8/613: unlink df/d18/d19/f1b 0 2022-01-31T19:44:30.030 INFO:tasks.workunit.client.1.smithi171.stdout:5/592: dread d5/d90/db7/d15/f33 [0,4194304] 0 2022-01-31T19:44:30.031 INFO:tasks.workunit.client.1.smithi171.stdout:3/714: rename d4/dd/cf to d4/d21/de5/dde/ce7 0 2022-01-31T19:44:30.038 INFO:tasks.workunit.client.1.smithi171.stdout:8/614: dread df/f17 [0,4194304] 0 2022-01-31T19:44:30.045 INFO:tasks.workunit.client.1.smithi171.stdout:1/609: symlink dd/df/d19/d1b/lba 0 2022-01-31T19:44:30.047 INFO:tasks.workunit.client.1.smithi171.stdout:4/600: link d2/d3/d5/d91/fbe d2/d3/d5/dc/fcc 0 2022-01-31T19:44:30.048 INFO:tasks.workunit.client.1.smithi171.stdout:3/715: mknod d4/dd/d27/d28/ce8 0 2022-01-31T19:44:30.061 INFO:tasks.workunit.client.1.smithi171.stdout:4/601: dread d2/d3/d5/d91/d7e/f48 [0,4194304] 0 2022-01-31T19:44:30.061 INFO:tasks.workunit.client.1.smithi171.stdout:4/602: chown d2/l4d 52600695 1 2022-01-31T19:44:30.062 INFO:tasks.workunit.client.1.smithi171.stdout:4/603: link d2/d3/d5/d70/d7b/fb5 d2/d3/d5/d91/d7e/dc7/fcd 0 2022-01-31T19:44:30.063 INFO:tasks.workunit.client.1.smithi171.stdout:4/604: chown d2/d3/d5/d91/d7e/f8c 2963 1 2022-01-31T19:44:30.063 INFO:tasks.workunit.client.1.smithi171.stdout:4/605: mkdir d2/d7f/dce 0 2022-01-31T19:44:30.066 INFO:tasks.workunit.client.1.smithi171.stdout:9/719: dwrite dd/df/f20 [0,4194304] 0 2022-01-31T19:44:30.067 INFO:tasks.workunit.client.1.smithi171.stdout:9/720: unlink dd/df/d3f/de3/fef 0 2022-01-31T19:44:30.070 INFO:tasks.workunit.client.1.smithi171.stdout:9/721: dread dd/df/fa0 [0,4194304] 0 2022-01-31T19:44:30.070 INFO:tasks.workunit.client.1.smithi171.stdout:9/722: fdatasync dd/df/d3f/d73/d96/d9f/fa1 0 2022-01-31T19:44:30.097 INFO:tasks.workunit.client.1.smithi171.stdout:5/593: dwrite d5/d90/db7/f46 [0,4194304] 0 2022-01-31T19:44:30.097 INFO:tasks.workunit.client.1.smithi171.stdout:8/615: dwrite df/d2a/f49 [0,4194304] 0 2022-01-31T19:44:30.098 INFO:tasks.workunit.client.1.smithi171.stdout:9/723: dwrite dd/df/d15/d1a/d84/dca/fe2 [0,4194304] 0 2022-01-31T19:44:30.098 INFO:tasks.workunit.client.1.smithi171.stdout:1/610: dwrite dd/df/d19/d1b/d1e/d35/f70 [4194304,4194304] 0 2022-01-31T19:44:30.099 INFO:tasks.workunit.client.1.smithi171.stdout:3/716: dwrite d4/dd/d73/f77 [0,4194304] 0 2022-01-31T19:44:30.105 INFO:tasks.workunit.client.1.smithi171.stdout:9/724: symlink dd/df/d15/d1a/d64/d6e/da5/lf5 0 2022-01-31T19:44:30.105 INFO:tasks.workunit.client.1.smithi171.stdout:9/725: chown dd/df/d3f/de3/cf4 86259053 1 2022-01-31T19:44:30.106 INFO:tasks.workunit.client.1.smithi171.stdout:1/611: mknod dd/df/d19/cbb 0 2022-01-31T19:44:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: cluster 2022-01-31T19:44:28.436152+0000 mgr.smithi171.asyxnc (mgr.24387) 35 : cluster [DBG] pgmap v23: 65 pgs: 65 active+clean; 3.0 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 137 MiB/s rd, 285 MiB/s wr, 398 op/s 2022-01-31T19:44:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: audit 2022-01-31T19:44:29.503823+0000 mon.smithi167 (mon.0) 686 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: audit 2022-01-31T19:44:29.509714+0000 mon.smithi171 (mon.1) 63 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:30.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: audit 2022-01-31T19:44:29.511121+0000 mon.smithi171 (mon.1) 64 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:30.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: audit 2022-01-31T19:44:30.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: 2022-01-31T19:44:29.511856+0000 mon.smithi171 (mon. 2022-01-31T19:44:30.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:29 smithi171 conmon[35325]: 1) 65 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:44:30.112 INFO:tasks.workunit.client.1.smithi171.stdout:4/606: rename d2/d3/d5/d91/d7e to d2/d3/d5/d41/dcf 0 2022-01-31T19:44:30.116 INFO:tasks.workunit.client.1.smithi171.stdout:3/717: mkdir d4/d21/de5/d92/dab/d52/de9 0 2022-01-31T19:44:30.118 INFO:tasks.workunit.client.1.smithi171.stdout:5/594: rename d5/d90/db7/d15/d23/d6b/fab to d5/d90/db7/d15/d5d/fc4 0 2022-01-31T19:44:30.119 INFO:tasks.workunit.client.1.smithi171.stdout:5/595: readlink d5/d90/db7/d15/d23/d6b/laf 0 2022-01-31T19:44:30.119 INFO:tasks.workunit.client.1.smithi171.stdout:4/607: creat d2/d3/d5/dc/d4b/dbc/fd0 x:0 0 0 2022-01-31T19:44:30.120 INFO:tasks.workunit.client.1.smithi171.stdout:4/608: chown d2/d3/d5/d41/dcf/d12/f1d 175648004 1 2022-01-31T19:44:30.120 INFO:tasks.workunit.client.1.smithi171.stdout:3/718: truncate d4/dd/d27/d28/f91 2042450 0 2022-01-31T19:44:30.121 INFO:tasks.workunit.client.1.smithi171.stdout:1/612: rmdir dd/d23/d2e/d5e/d81 39 2022-01-31T19:44:30.121 INFO:tasks.workunit.client.1.smithi171.stdout:1/613: stat dd/df/d84 0 2022-01-31T19:44:30.123 INFO:tasks.workunit.client.1.smithi171.stdout:8/616: rename df/c54 to df/d18/d19/d21/dcd/ccf 0 2022-01-31T19:44:30.123 INFO:tasks.workunit.client.1.smithi171.stdout:8/617: chown df/f51 0 1 2022-01-31T19:44:30.124 INFO:tasks.workunit.client.1.smithi171.stdout:1/614: symlink dd/d23/lbc 0 2022-01-31T19:44:30.124 INFO:tasks.workunit.client.1.smithi171.stdout:3/719: dread d4/dd/d39/f50 [0,4194304] 0 2022-01-31T19:44:30.125 INFO:tasks.workunit.client.1.smithi171.stdout:8/618: creat df/d18/da4/fd0 x:0 0 0 2022-01-31T19:44:30.125 INFO:tasks.workunit.client.1.smithi171.stdout:8/619: write df/d27/fb7 [593190,40150] 0 2022-01-31T19:44:30.126 INFO:tasks.workunit.client.1.smithi171.stdout:8/620: write df/d18/d19/d82/f97 [2949069,12792] 0 2022-01-31T19:44:30.126 INFO:tasks.workunit.client.1.smithi171.stdout:1/615: rename dd/d23/d34/d4b/l8f to dd/da2/lbd 0 2022-01-31T19:44:30.128 INFO:tasks.workunit.client.1.smithi171.stdout:8/621: dread df/d2a/f40 [4194304,4194304] 0 2022-01-31T19:44:30.128 INFO:tasks.workunit.client.1.smithi171.stdout:1/616: creat dd/df/d19/d1b/fbe x:0 0 0 2022-01-31T19:44:30.128 INFO:tasks.workunit.client.1.smithi171.stdout:1/617: dread - dd/da2/fb2 zero size 2022-01-31T19:44:30.129 INFO:tasks.workunit.client.1.smithi171.stdout:9/726: dwrite dd/df/d15/f4e [0,4194304] 0 2022-01-31T19:44:30.130 INFO:tasks.workunit.client.1.smithi171.stdout:9/727: chown dd/df/d3f/d73/d96/d9f/de6 11429624 1 2022-01-31T19:44:30.130 INFO:tasks.workunit.client.1.smithi171.stdout:9/728: creat dd/df/d15/d1a/d84/ff6 x:0 0 0 2022-01-31T19:44:30.130 INFO:tasks.workunit.client.1.smithi171.stdout:9/729: fsync dd/df/d15/da2/fd9 0 2022-01-31T19:44:30.131 INFO:tasks.workunit.client.1.smithi171.stdout:8/622: write df/d27/f64 [865897,55666] 0 2022-01-31T19:44:30.132 INFO:tasks.workunit.client.1.smithi171.stdout:4/609: dread d2/d3/d5/d41/dcf/f27 [0,4194304] 0 2022-01-31T19:44:30.136 INFO:tasks.workunit.client.1.smithi171.stdout:8/623: rename df/d18/d19/d36/f73 to df/d18/d19/d82/fd1 0 2022-01-31T19:44:30.138 INFO:tasks.workunit.client.1.smithi171.stdout:6/684: sync 2022-01-31T19:44:30.138 INFO:tasks.workunit.client.1.smithi171.stdout:2/821: sync 2022-01-31T19:44:30.139 INFO:tasks.workunit.client.1.smithi171.stdout:0/724: sync 2022-01-31T19:44:30.139 INFO:tasks.workunit.client.1.smithi171.stdout:7/533: sync 2022-01-31T19:44:30.140 INFO:tasks.workunit.client.1.smithi171.stdout:3/720: dread d4/dd/d27/d28/d35/d55/dbf/fc5 [0,4194304] 0 2022-01-31T19:44:30.141 INFO:tasks.workunit.client.1.smithi171.stdout:5/596: dread d5/f97 [0,4194304] 0 2022-01-31T19:44:30.143 INFO:tasks.workunit.client.1.smithi171.stdout:2/822: mknod d7/d26/d3d/d40/c100 0 2022-01-31T19:44:30.144 INFO:tasks.workunit.client.1.smithi171.stdout:2/823: dread - d7/d9/d1a/fbb zero size 2022-01-31T19:44:30.152 INFO:tasks.workunit.client.1.smithi171.stdout:7/534: dread d1/d1e/d2a/d29/d31/f51 [0,4194304] 0 2022-01-31T19:44:30.153 INFO:tasks.workunit.client.1.smithi171.stdout:7/535: chown d1/d1b/f4c 6003998 1 2022-01-31T19:44:30.153 INFO:tasks.workunit.client.1.smithi171.stdout:7/536: unlink d1/d52/c13 0 2022-01-31T19:44:30.156 INFO:tasks.workunit.client.1.smithi171.stdout:6/685: dread dc/d21/d4f/f5f [0,4194304] 0 2022-01-31T19:44:30.156 INFO:tasks.workunit.client.1.smithi171.stdout:6/686: fsync dc/d1e/d3f/fc8 0 2022-01-31T19:44:30.158 INFO:tasks.workunit.client.1.smithi171.stdout:5/597: dread d5/d90/db7/dd/f45 [0,4194304] 0 2022-01-31T19:44:30.163 INFO:tasks.workunit.client.1.smithi171.stdout:6/687: truncate dc/d19/d29/f85 3850341 0 2022-01-31T19:44:30.167 INFO:tasks.workunit.client.1.smithi171.stdout:6/688: creat dc/d19/fe7 x:0 0 0 2022-01-31T19:44:30.168 INFO:tasks.workunit.client.1.smithi171.stdout:6/689: write dc/d19/d30/fa3 [627441,24875] 0 2022-01-31T19:44:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: cluster 2022-01-31T19:44:28.436152+0000 mgr.smithi171.asyxnc 2022-01-31T19:44:30.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: (mgr.24387) 35 : cluster [DBG] pgmap v23: 65 pgs: 65 active+clean; 3.0 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 137 MiB/s rd, 285 MiB/s wr, 398 op/s 2022-01-31T19:44:30.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: audit 2022-01-31T19:44:29.503823+0000 mon.smithi167 (mon.0) 686 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:30.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: audit 2022-01-31T19:44:29.509714+0000 mon.smithi171 (mon.1) 63 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:30.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: audit 2022-01-31T19:44:29.511121+0000 mon.smithi171 (mon.1) 64 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:30.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: audit 2022-01-31T19:44:29.511856+0000 mon.smithi171 ( 2022-01-31T19:44:30.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:29 smithi167 conmon[32206]: mon.1) 65 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:44:30.198 INFO:tasks.workunit.client.1.smithi171.stdout:6/690: dread dc/d1e/d3f/f65 [0,4194304] 0 2022-01-31T19:44:30.198 INFO:tasks.workunit.client.1.smithi171.stdout:6/691: chown dc/d1e/d3f/cbd 7335188 1 2022-01-31T19:44:30.199 INFO:tasks.workunit.client.1.smithi171.stdout:6/692: truncate dc/d19/d29/da7/f7e 3413673 0 2022-01-31T19:44:30.201 INFO:tasks.workunit.client.1.smithi171.stdout:0/725: dwrite d9/db/dc/d3f/d6f/ffa [0,4194304] 0 2022-01-31T19:44:30.203 INFO:tasks.workunit.client.1.smithi171.stdout:0/726: mknod d9/db/d59/d71/c111 0 2022-01-31T19:44:30.204 INFO:tasks.workunit.client.1.smithi171.stdout:0/727: creat d9/d24/d4c/d35/d67/f112 x:0 0 0 2022-01-31T19:44:30.205 INFO:tasks.workunit.client.1.smithi171.stdout:0/728: symlink d9/db/d59/d71/l113 0 2022-01-31T19:44:30.205 INFO:tasks.workunit.client.1.smithi171.stdout:0/729: chown d9/f102 4 1 2022-01-31T19:44:30.206 INFO:tasks.workunit.client.1.smithi171.stdout:1/618: dwrite dd/df/d19/d1b/d1e/f9f [0,4194304] 0 2022-01-31T19:44:30.206 INFO:tasks.workunit.client.1.smithi171.stdout:9/730: dwrite f4 [0,4194304] 0 2022-01-31T19:44:30.207 INFO:tasks.workunit.client.1.smithi171.stdout:4/610: dwrite d2/d3/d5/d41/dcf/f62 [0,4194304] 0 2022-01-31T19:44:30.208 INFO:tasks.workunit.client.1.smithi171.stdout:3/721: dwrite d4/d21/f31 [0,4194304] 0 2022-01-31T19:44:30.208 INFO:tasks.workunit.client.1.smithi171.stdout:1/619: mknod dd/df/d19/d36/cbf 0 2022-01-31T19:44:30.212 INFO:tasks.workunit.client.1.smithi171.stdout:9/731: rename dd/df/d15/d38/d41/fb4 to dd/df/d15/de4/ff7 0 2022-01-31T19:44:30.212 INFO:tasks.workunit.client.1.smithi171.stdout:9/732: chown dd/df/d15/d1a/d64/fab 106436394 1 2022-01-31T19:44:30.213 INFO:tasks.workunit.client.1.smithi171.stdout:9/733: write dd/df/d3f/d73/d96/d9f/de6/da8/fe0 [925735,48191] 0 2022-01-31T19:44:30.213 INFO:tasks.workunit.client.1.smithi171.stdout:3/722: creat d4/fea x:0 0 0 2022-01-31T19:44:30.214 INFO:tasks.workunit.client.1.smithi171.stdout:7/537: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [0,4194304] 0 2022-01-31T19:44:30.214 INFO:tasks.workunit.client.1.smithi171.stdout:7/538: chown d1/d1e/d2a/d29/d31/d30/d3c/l9c 82235 1 2022-01-31T19:44:30.214 INFO:tasks.workunit.client.1.smithi171.stdout:7/539: readlink d1/d1e/d2a/d58/lb5 0 2022-01-31T19:44:30.214 INFO:tasks.workunit.client.1.smithi171.stdout:5/598: dwrite d5/f55 [0,4194304] 0 2022-01-31T19:44:30.215 INFO:tasks.workunit.client.1.smithi171.stdout:5/599: readlink d5/d90/db7/d15/d5d/d6a/lc2 0 2022-01-31T19:44:30.215 INFO:tasks.workunit.client.1.smithi171.stdout:7/540: write d1/d1e/d2a/d58/fb1 [1389425,115382] 0 2022-01-31T19:44:30.215 INFO:tasks.workunit.client.1.smithi171.stdout:7/541: creat d1/d1b/fb6 x:0 0 0 2022-01-31T19:44:30.216 INFO:tasks.workunit.client.1.smithi171.stdout:1/620: creat dd/df/d19/d1b/d1e/d64/da7/fc0 x:0 0 0 2022-01-31T19:44:30.216 INFO:tasks.workunit.client.1.smithi171.stdout:2/824: dwrite d7/d26/d3d/d40/ffd [0,4194304] 0 2022-01-31T19:44:30.217 INFO:tasks.workunit.client.1.smithi171.stdout:8/624: dwrite df/d18/d19/d21/f26 [0,4194304] 0 2022-01-31T19:44:30.217 INFO:tasks.workunit.client.1.smithi171.stdout:6/693: dread dc/f17 [0,4194304] 0 2022-01-31T19:44:30.217 INFO:tasks.workunit.client.1.smithi171.stdout:6/694: dread - dc/d19/d29/d27/d37/d3b/d67/dd1/fdd zero size 2022-01-31T19:44:30.218 INFO:tasks.workunit.client.1.smithi171.stdout:8/625: read df/f83 [8497025,70979] 0 2022-01-31T19:44:30.222 INFO:tasks.workunit.client.1.smithi171.stdout:9/734: rename dd/df/d15/d1a/f27 to dd/df/d3f/d73/d96/d9f/ff8 0 2022-01-31T19:44:30.223 INFO:tasks.workunit.client.1.smithi171.stdout:3/723: symlink d4/d21/de5/d92/leb 0 2022-01-31T19:44:30.224 INFO:tasks.workunit.client.1.smithi171.stdout:4/611: dwrite d2/d3/d5/dc/f95 [0,4194304] 0 2022-01-31T19:44:30.224 INFO:tasks.workunit.client.1.smithi171.stdout:4/612: write d2/d3/d5/fc1 [344372,56673] 0 2022-01-31T19:44:30.229 INFO:tasks.workunit.client.1.smithi171.stdout:5/600: symlink d5/lc5 0 2022-01-31T19:44:30.237 INFO:tasks.workunit.client.1.smithi171.stdout:6/695: symlink dc/d8b/le8 0 2022-01-31T19:44:30.237 INFO:tasks.workunit.client.1.smithi171.stdout:6/696: chown dc/d19/l75 960521 1 2022-01-31T19:44:30.238 INFO:tasks.workunit.client.1.smithi171.stdout:8/626: mknod df/dcc/cd2 0 2022-01-31T19:44:30.240 INFO:tasks.workunit.client.1.smithi171.stdout:7/542: rename d1/d52/l27 to d1/d1e/d2a/lb7 0 2022-01-31T19:44:30.240 INFO:tasks.workunit.client.1.smithi171.stdout:6/697: dread dc/d18/f39 [0,4194304] 0 2022-01-31T19:44:30.241 INFO:tasks.workunit.client.1.smithi171.stdout:3/724: creat d4/dd/d27/d28/d35/d55/dbf/fec x:0 0 0 2022-01-31T19:44:30.243 INFO:tasks.workunit.client.1.smithi171.stdout:2/825: dwrite d7/dd/d5e/d6b/f7a [0,4194304] 0 2022-01-31T19:44:30.243 INFO:tasks.workunit.client.1.smithi171.stdout:2/826: dread - d7/d7c/fe8 zero size 2022-01-31T19:44:30.244 INFO:tasks.workunit.client.1.smithi171.stdout:3/725: write d4/dd/d27/f47 [805647,109804] 0 2022-01-31T19:44:30.244 INFO:tasks.workunit.client.1.smithi171.stdout:4/613: symlink d2/d3/d5/d91/d77/da5/ld1 0 2022-01-31T19:44:30.245 INFO:tasks.workunit.client.1.smithi171.stdout:4/614: readlink d2/d3/d5/d41/dcf/d12/l21 0 2022-01-31T19:44:30.245 INFO:tasks.workunit.client.1.smithi171.stdout:4/615: creat d2/d3/d5/fd2 x:0 0 0 2022-01-31T19:44:30.251 INFO:tasks.workunit.client.1.smithi171.stdout:5/601: creat d5/d90/db7/dd/d66/fc6 x:0 0 0 2022-01-31T19:44:30.251 INFO:tasks.workunit.client.1.smithi171.stdout:5/602: dread - d5/d90/db7/d15/d23/d6b/f7e zero size 2022-01-31T19:44:30.253 INFO:tasks.workunit.client.1.smithi171.stdout:9/735: rename dd/df/f21 to dd/df/d3f/d73/d96/d9f/de6/d7c/d7d/ff9 0 2022-01-31T19:44:30.256 INFO:tasks.workunit.client.1.smithi171.stdout:1/621: dwrite dd/f7f [0,4194304] 0 2022-01-31T19:44:30.257 INFO:tasks.workunit.client.1.smithi171.stdout:6/698: mkdir dc/d21/de9 0 2022-01-31T19:44:30.257 INFO:tasks.workunit.client.1.smithi171.stdout:6/699: chown dc/d19/d29/d27/d37/cc2 39827 1 2022-01-31T19:44:30.261 INFO:tasks.workunit.client.1.smithi171.stdout:3/726: truncate d4/d21/de5/d92/dab/d52/f96 174515 0 2022-01-31T19:44:30.261 INFO:tasks.workunit.client.1.smithi171.stdout:3/727: read - d4/d21/de5/d92/dab/dd4/fdb zero size 2022-01-31T19:44:30.266 INFO:tasks.workunit.client.1.smithi171.stdout:7/543: dread d1/d1e/d2a/d29/d31/d30/d6b/f86 [0,4194304] 0 2022-01-31T19:44:30.267 INFO:tasks.workunit.client.1.smithi171.stdout:4/616: symlink d2/d3/d5/dc/d4b/dbc/ld3 0 2022-01-31T19:44:30.269 INFO:tasks.workunit.client.1.smithi171.stdout:5/603: rename d5/d90/l98 to d5/d90/db7/d15/d5d/lc7 0 2022-01-31T19:44:30.271 INFO:tasks.workunit.client.1.smithi171.stdout:8/627: dwrite df/d27/fa3 [0,4194304] 0 2022-01-31T19:44:30.274 INFO:tasks.workunit.client.1.smithi171.stdout:3/728: creat d4/d21/d3a/db5/fed x:0 0 0 2022-01-31T19:44:30.280 INFO:tasks.workunit.client.1.smithi171.stdout:7/544: dread d1/d1b/f4b [0,4194304] 0 2022-01-31T19:44:30.280 INFO:tasks.workunit.client.1.smithi171.stdout:7/545: truncate d1/d1e/d2a/d29/fa7 880364 0 2022-01-31T19:44:30.281 INFO:tasks.workunit.client.1.smithi171.stdout:7/546: readlink d1/d1b/l4 0 2022-01-31T19:44:30.281 INFO:tasks.workunit.client.1.smithi171.stdout:5/604: mkdir d5/db0/dc8 0 2022-01-31T19:44:30.282 INFO:tasks.workunit.client.1.smithi171.stdout:4/617: mkdir d2/d7f/dce/dd4 0 2022-01-31T19:44:30.282 INFO:tasks.workunit.client.1.smithi171.stdout:2/827: dwrite d7/d26/de5/dc0/fe3 [0,4194304] 0 2022-01-31T19:44:30.282 INFO:tasks.workunit.client.1.smithi171.stdout:2/828: write d7/d7c/f87 [423565,29145] 0 2022-01-31T19:44:30.284 INFO:tasks.workunit.client.1.smithi171.stdout:9/736: rmdir dd/df/d15/d1a/d64 39 2022-01-31T19:44:30.294 INFO:tasks.workunit.client.1.smithi171.stdout:7/547: write d1/d1e/d2a/d29/d31/f6a [4118355,4675] 0 2022-01-31T19:44:30.294 INFO:tasks.workunit.client.1.smithi171.stdout:7/548: creat d1/d1e/d2a/d58/fb8 x:0 0 0 2022-01-31T19:44:30.296 INFO:tasks.workunit.client.1.smithi171.stdout:5/605: symlink d5/db0/dc8/lc9 0 2022-01-31T19:44:30.297 INFO:tasks.workunit.client.1.smithi171.stdout:5/606: fsync d5/f37 0 2022-01-31T19:44:30.297 INFO:tasks.workunit.client.1.smithi171.stdout:4/618: mkdir d2/d3/d5/d70/d7c/dd5 0 2022-01-31T19:44:30.299 INFO:tasks.workunit.client.1.smithi171.stdout:2/829: truncate d7/f14 613031 0 2022-01-31T19:44:30.302 INFO:tasks.workunit.client.1.smithi171.stdout:5/607: unlink d5/d90/db7/d15/f3f 0 2022-01-31T19:44:30.303 INFO:tasks.workunit.client.1.smithi171.stdout:4/619: truncate d2/f5c 1051608 0 2022-01-31T19:44:30.306 INFO:tasks.workunit.client.1.smithi171.stdout:5/608: rename d5/d90/db7/f16 to d5/d90/db7/d15/d23/d3d/fca 0 2022-01-31T19:44:30.308 INFO:tasks.workunit.client.1.smithi171.stdout:4/620: creat d2/d3/d5/dc/fd6 x:0 0 0 2022-01-31T19:44:30.308 INFO:tasks.workunit.client.1.smithi171.stdout:4/621: truncate d2/d3/d5/d41/d6b/fa9 1137668 0 2022-01-31T19:44:30.308 INFO:tasks.workunit.client.1.smithi171.stdout:4/622: write d2/d3/d5/d91/f76 [1745991,129631] 0 2022-01-31T19:44:30.310 INFO:tasks.workunit.client.1.smithi171.stdout:4/623: rename d2/d3/d5/d41/dcf/d12/l20 to d2/d3/d5/d91/ld7 0 2022-01-31T19:44:30.311 INFO:tasks.workunit.client.1.smithi171.stdout:4/624: mknod d2/d7f/dce/cd8 0 2022-01-31T19:44:30.311 INFO:tasks.workunit.client.1.smithi171.stdout:4/625: chown d2/d3/d5/d3b/c61 2 1 2022-01-31T19:44:30.312 INFO:tasks.workunit.client.1.smithi171.stdout:4/626: creat d2/d3/d5/d3b/fd9 x:0 0 0 2022-01-31T19:44:30.313 INFO:tasks.workunit.client.1.smithi171.stdout:8/628: dwrite df/d18/d79/fc7 [0,4194304] 0 2022-01-31T19:44:30.313 INFO:tasks.workunit.client.1.smithi171.stdout:4/627: truncate d2/d3/d5/dc/f26 4368475 0 2022-01-31T19:44:30.314 INFO:tasks.workunit.client.1.smithi171.stdout:1/622: dwrite dd/df/d19/d1b/f63 [0,4194304] 0 2022-01-31T19:44:30.314 INFO:tasks.workunit.client.1.smithi171.stdout:1/623: readlink dd/d23/l54 0 2022-01-31T19:44:30.318 INFO:tasks.workunit.client.1.smithi171.stdout:8/629: mkdir df/d18/d19/d82/dd3 0 2022-01-31T19:44:30.318 INFO:tasks.workunit.client.1.smithi171.stdout:8/630: fdatasync df/d27/fb7 0 2022-01-31T19:44:30.319 INFO:tasks.workunit.client.1.smithi171.stdout:3/729: dwrite d4/dd/d39/d51/f71 [0,4194304] 0 2022-01-31T19:44:30.319 INFO:tasks.workunit.client.1.smithi171.stdout:3/730: fsync d4/dd/d27/d28/d35/f48 0 2022-01-31T19:44:30.319 INFO:tasks.workunit.client.1.smithi171.stdout:3/731: fdatasync d4/dd/d4b/fcf 0 2022-01-31T19:44:30.320 INFO:tasks.workunit.client.1.smithi171.stdout:1/624: creat dd/df/d19/d1b/d1e/d64/fc1 x:0 0 0 2022-01-31T19:44:30.320 INFO:tasks.workunit.client.1.smithi171.stdout:3/732: mknod d4/d21/de5/cee 0 2022-01-31T19:44:30.321 INFO:tasks.workunit.client.1.smithi171.stdout:1/625: mknod dd/d23/d2e/d5e/d81/d98/d80/cc2 0 2022-01-31T19:44:30.322 INFO:tasks.workunit.client.1.smithi171.stdout:3/733: creat d4/dd/d27/d6e/fef x:0 0 0 2022-01-31T19:44:30.323 INFO:tasks.workunit.client.1.smithi171.stdout:3/734: truncate d4/d21/d3a/db5/fd5 747308 0 2022-01-31T19:44:30.323 INFO:tasks.workunit.client.1.smithi171.stdout:3/735: chown d4/d21/d3a/d59/dc1/fc2 669 1 2022-01-31T19:44:30.325 INFO:tasks.workunit.client.1.smithi171.stdout:1/626: rename dd/f14 to dd/d23/d2e/fc3 0 2022-01-31T19:44:30.325 INFO:tasks.workunit.client.1.smithi171.stdout:1/627: fdatasync fa 0 2022-01-31T19:44:30.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/628: mknod dd/d23/d2e/d5e/d81/d98/d80/cc4 0 2022-01-31T19:44:30.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/629: write dd/d23/f57 [549377,93834] 0 2022-01-31T19:44:30.328 INFO:tasks.workunit.client.1.smithi171.stdout:7/549: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d5f/f90 [0,4194304] 0 2022-01-31T19:44:30.328 INFO:tasks.workunit.client.1.smithi171.stdout:1/630: creat dd/d23/d2e/d5e/d81/d98/fc5 x:0 0 0 2022-01-31T19:44:30.328 INFO:tasks.workunit.client.1.smithi171.stdout:7/550: dread - d1/d1e/d2a/f83 zero size 2022-01-31T19:44:30.330 INFO:tasks.workunit.client.1.smithi171.stdout:7/551: readlink d1/d1e/d2a/d29/d31/l6c 0 2022-01-31T19:44:30.333 INFO:tasks.workunit.client.1.smithi171.stdout:9/737: dwrite dd/df/d3f/d73/d96/d9f/de6/da8/fe1 [0,4194304] 0 2022-01-31T19:44:30.336 INFO:tasks.workunit.client.1.smithi171.stdout:9/738: read dd/df/d3f/d73/d96/d9f/ff8 [2027525,35915] 0 2022-01-31T19:44:30.339 INFO:tasks.workunit.client.1.smithi171.stdout:5/609: dwrite d5/d90/db7/fc1 [0,4194304] 0 2022-01-31T19:44:30.339 INFO:tasks.workunit.client.1.smithi171.stdout:5/610: creat d5/d90/db7/d15/d80/fcb x:0 0 0 2022-01-31T19:44:30.339 INFO:tasks.workunit.client.1.smithi171.stdout:5/611: read - d5/d90/db7/d15/d23/d58/fa3 zero size 2022-01-31T19:44:30.340 INFO:tasks.workunit.client.1.smithi171.stdout:5/612: chown d5/f37 3725276 1 2022-01-31T19:44:30.340 INFO:tasks.workunit.client.1.smithi171.stdout:5/613: chown d5/d90/db7/d15/d23/d58/d9e/lb4 38470 1 2022-01-31T19:44:30.341 INFO:tasks.workunit.client.1.smithi171.stdout:2/830: dwrite d7/d9/d1a/d45/da4/fab [4194304,4194304] 0 2022-01-31T19:44:30.346 INFO:tasks.workunit.client.1.smithi171.stdout:4/628: dwrite d2/d3/d5/d41/dcf/d12/d15/f78 [0,4194304] 0 2022-01-31T19:44:30.346 INFO:tasks.workunit.client.1.smithi171.stdout:4/629: stat d2/d7f/dce 0 2022-01-31T19:44:30.354 INFO:tasks.workunit.client.1.smithi171.stdout:2/831: truncate d7/d9/f2c 997156 0 2022-01-31T19:44:30.354 INFO:tasks.workunit.client.1.smithi171.stdout:2/832: stat d7/dd/f52 0 2022-01-31T19:44:30.356 INFO:tasks.workunit.client.1.smithi171.stdout:3/736: dwrite d4/f99 [0,4194304] 0 2022-01-31T19:44:30.358 INFO:tasks.workunit.client.1.smithi171.stdout:5/614: dread d5/d90/db7/d15/d5d/d6f/f9c [0,4194304] 0 2022-01-31T19:44:30.358 INFO:tasks.workunit.client.1.smithi171.stdout:4/630: symlink d2/d7f/dce/dd4/lda 0 2022-01-31T19:44:30.358 INFO:tasks.workunit.client.1.smithi171.stdout:4/631: dread - d2/d3/d5/d41/dcf/d12/f45 zero size 2022-01-31T19:44:30.359 INFO:tasks.workunit.client.1.smithi171.stdout:1/631: dwrite dd/f91 [4194304,4194304] 0 2022-01-31T19:44:30.362 INFO:tasks.workunit.client.1.smithi171.stdout:3/737: creat d4/dd/ff0 x:0 0 0 2022-01-31T19:44:30.362 INFO:tasks.workunit.client.1.smithi171.stdout:7/552: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [0,4194304] 0 2022-01-31T19:44:30.363 INFO:tasks.workunit.client.1.smithi171.stdout:5/615: symlink d5/d90/db7/dd/lcc 0 2022-01-31T19:44:30.363 INFO:tasks.workunit.client.1.smithi171.stdout:5/616: write d5/d90/db7/dd/f22 [1177579,28108] 0 2022-01-31T19:44:30.364 INFO:tasks.workunit.client.1.smithi171.stdout:4/632: truncate d2/d3/d5/d91/fbe 1337353 0 2022-01-31T19:44:30.365 INFO:tasks.workunit.client.1.smithi171.stdout:2/833: dread d7/d9/d1a/d45/f9d [0,4194304] 0 2022-01-31T19:44:30.370 INFO:tasks.workunit.client.1.smithi171.stdout:1/632: creat dd/df/d19/fc6 x:0 0 0 2022-01-31T19:44:30.370 INFO:tasks.workunit.client.1.smithi171.stdout:1/633: dread - dd/d23/d2e/d5e/d81/fae zero size 2022-01-31T19:44:30.371 INFO:tasks.workunit.client.1.smithi171.stdout:1/634: write dd/d23/f41 [10054,89799] 0 2022-01-31T19:44:30.371 INFO:tasks.workunit.client.1.smithi171.stdout:1/635: dread - dd/df/d19/f96 zero size 2022-01-31T19:44:30.372 INFO:tasks.workunit.client.1.smithi171.stdout:3/738: link d4/d21/le4 d4/dd/d27/d28/db6/lf1 0 2022-01-31T19:44:30.375 INFO:tasks.workunit.client.1.smithi171.stdout:7/553: truncate d1/d26/d47/f7b 3235084 0 2022-01-31T19:44:30.378 INFO:tasks.workunit.client.1.smithi171.stdout:2/834: mkdir d7/dd/dfc/d101 0 2022-01-31T19:44:30.379 INFO:tasks.workunit.client.1.smithi171.stdout:2/835: stat d7/d9 0 2022-01-31T19:44:30.379 INFO:tasks.workunit.client.1.smithi171.stdout:2/836: chown d7/c99 0 1 2022-01-31T19:44:30.383 INFO:tasks.workunit.client.1.smithi171.stdout:3/739: mknod d4/dd/dc7/cf2 0 2022-01-31T19:44:30.387 INFO:tasks.workunit.client.1.smithi171.stdout:2/837: mknod d7/dd/dfc/d101/c102 0 2022-01-31T19:44:30.391 INFO:tasks.workunit.client.1.smithi171.stdout:2/838: symlink d7/d26/de5/dc0/l103 0 2022-01-31T19:44:30.392 INFO:tasks.workunit.client.1.smithi171.stdout:2/839: symlink d7/dd/d5e/d84/l104 0 2022-01-31T19:44:30.399 INFO:tasks.workunit.client.1.smithi171.stdout:1/636: dwrite dd/d23/d2e/fa5 [0,4194304] 0 2022-01-31T19:44:30.401 INFO:tasks.workunit.client.1.smithi171.stdout:4/633: dwrite d2/f90 [0,4194304] 0 2022-01-31T19:44:30.402 INFO:tasks.workunit.client.1.smithi171.stdout:4/634: stat d2/d3/d5/d91/d77/da5/fcb 0 2022-01-31T19:44:30.402 INFO:tasks.workunit.client.1.smithi171.stdout:1/637: link dd/d23/d2e/d5e/d81/d98/fb1 dd/df/d19/d1b/d1e/d64/fc7 0 2022-01-31T19:44:30.404 INFO:tasks.workunit.client.1.smithi171.stdout:4/635: creat d2/d3/d5/d41/dcf/dc7/fdb x:0 0 0 2022-01-31T19:44:30.408 INFO:tasks.workunit.client.1.smithi171.stdout:5/617: dwrite d5/f8 [4194304,4194304] 0 2022-01-31T19:44:30.408 INFO:tasks.workunit.client.1.smithi171.stdout:7/554: dwrite d1/f16 [8388608,4194304] 0 2022-01-31T19:44:30.412 INFO:tasks.workunit.client.1.smithi171.stdout:5/618: write d5/d90/db7/d15/d5d/d6a/f77 [187357,107114] 0 2022-01-31T19:44:30.416 INFO:tasks.workunit.client.1.smithi171.stdout:5/619: getdents d5/d90/db7/dd/d66 0 2022-01-31T19:44:30.417 INFO:tasks.workunit.client.1.smithi171.stdout:4/636: dread d2/d3/d5/f46 [0,4194304] 0 2022-01-31T19:44:30.418 INFO:tasks.workunit.client.1.smithi171.stdout:1/638: dread dd/da2/fa6 [0,4194304] 0 2022-01-31T19:44:30.418 INFO:tasks.workunit.client.1.smithi171.stdout:1/639: creat dd/df/d19/fc8 x:0 0 0 2022-01-31T19:44:30.419 INFO:tasks.workunit.client.1.smithi171.stdout:1/640: dread - dd/d23/d2e/d5e/d81/fa3 zero size 2022-01-31T19:44:30.419 INFO:tasks.workunit.client.1.smithi171.stdout:1/641: readlink dd/df/d19/d1b/l2b 0 2022-01-31T19:44:30.419 INFO:tasks.workunit.client.1.smithi171.stdout:1/642: readlink dd/df/d19/d1b/d1e/d97/la4 0 2022-01-31T19:44:30.419 INFO:tasks.workunit.client.1.smithi171.stdout:1/643: write dd/df/d19/d1b/d1e/d97/f9c [1451686,35009] 0 2022-01-31T19:44:30.420 INFO:tasks.workunit.client.1.smithi171.stdout:1/644: creat dd/df/d19/d1b/d1e/fc9 x:0 0 0 2022-01-31T19:44:30.420 INFO:tasks.workunit.client.1.smithi171.stdout:3/740: dwrite d4/dd/d27/d28/d35/d55/f74 [0,4194304] 0 2022-01-31T19:44:30.420 INFO:tasks.workunit.client.1.smithi171.stdout:3/741: creat d4/dd/d39/ff3 x:0 0 0 2022-01-31T19:44:30.420 INFO:tasks.workunit.client.1.smithi171.stdout:5/620: mknod d5/db0/dc8/ccd 0 2022-01-31T19:44:30.421 INFO:tasks.workunit.client.1.smithi171.stdout:4/637: link d2/d3/d5/dc/fd6 d2/fdc 0 2022-01-31T19:44:30.422 INFO:tasks.workunit.client.1.smithi171.stdout:1/645: write dd/df/d19/d1b/d1e/f6a [2844178,73812] 0 2022-01-31T19:44:30.422 INFO:tasks.workunit.client.1.smithi171.stdout:1/646: chown dd/d23/d2e/d5e/d81/fae 22 1 2022-01-31T19:44:30.422 INFO:tasks.workunit.client.1.smithi171.stdout:2/840: dwrite d7/d26/de5/fc2 [0,4194304] 0 2022-01-31T19:44:30.432 INFO:tasks.workunit.client.1.smithi171.stdout:5/621: getdents d5/d90/db7/d15/d80/d8d 0 2022-01-31T19:44:30.433 INFO:tasks.workunit.client.1.smithi171.stdout:4/638: symlink d2/d3/d5/ldd 0 2022-01-31T19:44:30.441 INFO:tasks.workunit.client.1.smithi171.stdout:2/841: chown d7/dd/d5e/la3 701432744 1 2022-01-31T19:44:30.441 INFO:tasks.workunit.client.1.smithi171.stdout:2/842: readlink d7/d26/de5/ddc/df0/lf9 0 2022-01-31T19:44:30.442 INFO:tasks.workunit.client.1.smithi171.stdout:2/843: write d7/d26/d63/d9f/dc3/fca [666067,84814] 0 2022-01-31T19:44:30.444 INFO:tasks.workunit.client.1.smithi171.stdout:1/647: mkdir dd/df/d19/d1b/d1e/d64/da7/dca 0 2022-01-31T19:44:30.444 INFO:tasks.workunit.client.1.smithi171.stdout:1/648: fsync dd/df/d19/fc8 0 2022-01-31T19:44:30.444 INFO:tasks.workunit.client.1.smithi171.stdout:4/639: dread d2/d3/d5/d41/dcf/d12/f8d [0,4194304] 0 2022-01-31T19:44:30.444 INFO:tasks.workunit.client.1.smithi171.stdout:5/622: rename d5/d90/db7/d15/d5d/l6c to d5/d90/db7/d15/d23/d3d/lce 0 2022-01-31T19:44:30.445 INFO:tasks.workunit.client.1.smithi171.stdout:4/640: chown d2/d3/d5/d41/f47 182 1 2022-01-31T19:44:30.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/649: dread dd/df/d19/d1b/d1e/d64/da7/faf [0,4194304] 0 2022-01-31T19:44:30.446 INFO:tasks.workunit.client.1.smithi171.stdout:1/650: chown dd/df/d84/cb8 0 1 2022-01-31T19:44:30.447 INFO:tasks.workunit.client.1.smithi171.stdout:2/844: dread d7/d26/de5/fb1 [0,4194304] 0 2022-01-31T19:44:30.447 INFO:tasks.workunit.client.1.smithi171.stdout:2/845: write d7/d26/f88 [94747,84332] 0 2022-01-31T19:44:30.447 INFO:tasks.workunit.client.1.smithi171.stdout:5/623: write d5/f8 [7936686,37802] 0 2022-01-31T19:44:30.450 INFO:tasks.workunit.client.1.smithi171.stdout:2/846: rmdir d7/d26/d63/d71 39 2022-01-31T19:44:30.450 INFO:tasks.workunit.client.1.smithi171.stdout:2/847: readlink d7/d26/d3d/lb6 0 2022-01-31T19:44:30.450 INFO:tasks.workunit.client.1.smithi171.stdout:2/848: chown d7/d26/d63/d71/fee 4763824 1 2022-01-31T19:44:30.451 INFO:tasks.workunit.client.1.smithi171.stdout:2/849: write d7/f14 [667536,98525] 0 2022-01-31T19:44:30.451 INFO:tasks.workunit.client.1.smithi171.stdout:2/850: fsync d7/d26/d3d/d40/f4f 0 2022-01-31T19:44:30.451 INFO:tasks.workunit.client.1.smithi171.stdout:2/851: stat d7/d55/l60 0 2022-01-31T19:44:30.476 INFO:tasks.workunit.client.1.smithi171.stdout:1/651: dwrite dd/df/d19/f32 [0,4194304] 0 2022-01-31T19:44:30.478 INFO:tasks.workunit.client.1.smithi171.stdout:4/641: dwrite d2/d3/d5/d41/dcf/d12/d15/f78 [0,4194304] 0 2022-01-31T19:44:30.479 INFO:tasks.workunit.client.1.smithi171.stdout:4/642: stat d2/f3c 0 2022-01-31T19:44:30.479 INFO:tasks.workunit.client.1.smithi171.stdout:5/624: dwrite d5/d59/da0/fa7 [0,4194304] 0 2022-01-31T19:44:30.488 INFO:tasks.workunit.client.1.smithi171.stdout:1/652: dread dd/d23/f5c [4194304,4194304] 0 2022-01-31T19:44:30.492 INFO:tasks.workunit.client.1.smithi171.stdout:5/625: dread d5/d90/db7/fc1 [0,4194304] 0 2022-01-31T19:44:30.493 INFO:tasks.workunit.client.1.smithi171.stdout:5/626: creat d5/d90/db7/d15/d5d/d6f/fcf x:0 0 0 2022-01-31T19:44:30.495 INFO:tasks.workunit.client.1.smithi171.stdout:5/627: creat d5/d90/db7/d15/d5d/fd0 x:0 0 0 2022-01-31T19:44:30.495 INFO:tasks.workunit.client.1.smithi171.stdout:5/628: dread - d5/d90/db7/d15/d5d/d6f/fcf zero size 2022-01-31T19:44:30.497 INFO:tasks.workunit.client.1.smithi171.stdout:5/629: read d5/f55 [2222073,74975] 0 2022-01-31T19:44:30.499 INFO:tasks.workunit.client.1.smithi171.stdout:5/630: mkdir d5/d90/dd1 0 2022-01-31T19:44:30.500 INFO:tasks.workunit.client.1.smithi171.stdout:1/653: dread dd/df/d19/d1b/f72 [4194304,4194304] 0 2022-01-31T19:44:30.500 INFO:tasks.workunit.client.1.smithi171.stdout:1/654: truncate dd/da2/fb2 792997 0 2022-01-31T19:44:30.501 INFO:tasks.workunit.client.1.smithi171.stdout:5/631: creat d5/db0/fd2 x:0 0 0 2022-01-31T19:44:30.501 INFO:tasks.workunit.client.1.smithi171.stdout:5/632: dread - d5/d90/db7/d15/d23/f9a zero size 2022-01-31T19:44:30.501 INFO:tasks.workunit.client.1.smithi171.stdout:4/643: dwrite d2/d3/d5/d41/dcf/d12/f1d [0,4194304] 0 2022-01-31T19:44:30.504 INFO:tasks.workunit.client.1.smithi171.stdout:1/655: read dd/f56 [3371191,18389] 0 2022-01-31T19:44:30.505 INFO:tasks.workunit.client.1.smithi171.stdout:5/633: dread d5/d90/db7/fc1 [0,4194304] 0 2022-01-31T19:44:30.507 INFO:tasks.workunit.client.1.smithi171.stdout:1/656: read dd/df/d19/d1b/f63 [2206662,15841] 0 2022-01-31T19:44:30.507 INFO:tasks.workunit.client.1.smithi171.stdout:5/634: write d5/d90/db7/dd/d66/f9b [654199,2647] 0 2022-01-31T19:44:30.508 INFO:tasks.workunit.client.1.smithi171.stdout:1/657: mkdir dd/d23/d74/dcb 0 2022-01-31T19:44:30.511 INFO:tasks.workunit.client.1.smithi171.stdout:5/635: mkdir d5/d90/dd3 0 2022-01-31T19:44:30.511 INFO:tasks.workunit.client.1.smithi171.stdout:5/636: write d5/d90/db7/d15/d5d/fc4 [607944,1512] 0 2022-01-31T19:44:30.513 INFO:tasks.workunit.client.1.smithi171.stdout:5/637: unlink d5/d90/db7/dd/l52 0 2022-01-31T19:44:30.513 INFO:tasks.workunit.client.1.smithi171.stdout:5/638: truncate d5/d90/db7/d15/d23/d67/f4d 1730630 0 2022-01-31T19:44:30.513 INFO:tasks.workunit.client.1.smithi171.stdout:5/639: creat d5/db0/fd4 x:0 0 0 2022-01-31T19:44:30.514 INFO:tasks.workunit.client.1.smithi171.stdout:1/658: write dd/f91 [4115621,53887] 0 2022-01-31T19:44:30.515 INFO:tasks.workunit.client.1.smithi171.stdout:1/659: chown dd/d23/d74/fb4 5627 1 2022-01-31T19:44:30.515 INFO:tasks.workunit.client.1.smithi171.stdout:5/640: fdatasync d5/d90/db7/d15/f42 0 2022-01-31T19:44:30.515 INFO:tasks.workunit.client.1.smithi171.stdout:1/660: rename dd/df/d19/d1b/c43 to dd/df/d19/d1b/d1e/d35/db0/ccc 0 2022-01-31T19:44:30.518 INFO:tasks.workunit.client.1.smithi171.stdout:5/641: write d5/f8 [708093,60302] 0 2022-01-31T19:44:30.520 INFO:tasks.workunit.client.1.smithi171.stdout:5/642: rename d5/d90/db7/d15/d5d/d6f/c92 to d5/d90/dd1/cd5 0 2022-01-31T19:44:30.520 INFO:tasks.workunit.client.1.smithi171.stdout:5/643: readlink d5/d90/db7/d15/d80/l84 0 2022-01-31T19:44:30.520 INFO:tasks.workunit.client.1.smithi171.stdout:5/644: fdatasync d5/d90/db7/d15/f64 0 2022-01-31T19:44:30.520 INFO:tasks.workunit.client.1.smithi171.stdout:5/645: creat d5/fd6 x:0 0 0 2022-01-31T19:44:30.523 INFO:tasks.workunit.client.1.smithi171.stdout:5/646: mkdir d5/d90/db7/d15/d23/d58/d9e/dd7 0 2022-01-31T19:44:30.523 INFO:tasks.workunit.client.1.smithi171.stdout:5/647: chown d5/d59/c93 255 1 2022-01-31T19:44:30.524 INFO:tasks.workunit.client.1.smithi171.stdout:4/644: dwrite d2/d3/d5/d41/dcf/f83 [0,4194304] 0 2022-01-31T19:44:30.525 INFO:tasks.workunit.client.1.smithi171.stdout:1/661: dread dd/df/d19/d1b/d1e/f4a [0,4194304] 0 2022-01-31T19:44:30.525 INFO:tasks.workunit.client.1.smithi171.stdout:1/662: dread - dd/df/d19/d1b/d1e/d64/f53 zero size 2022-01-31T19:44:30.529 INFO:tasks.workunit.client.1.smithi171.stdout:4/645: dread d2/d3/d5/d91/fab [0,4194304] 0 2022-01-31T19:44:30.529 INFO:tasks.workunit.client.1.smithi171.stdout:1/663: rename dd/d23/f5c to dd/d23/d2e/d5e/d81/fcd 0 2022-01-31T19:44:30.530 INFO:tasks.workunit.client.1.smithi171.stdout:1/664: mkdir dd/df/d19/dce 0 2022-01-31T19:44:30.531 INFO:tasks.workunit.client.1.smithi171.stdout:1/665: creat dd/df/d19/d1b/d1e/d97/fcf x:0 0 0 2022-01-31T19:44:30.532 INFO:tasks.workunit.client.1.smithi171.stdout:4/646: dread d2/d3/d5/d41/dcf/d12/fc0 [0,4194304] 0 2022-01-31T19:44:30.533 INFO:tasks.workunit.client.1.smithi171.stdout:1/666: link dd/d23/d34/d4b/f52 dd/df/d19/d1b/d1e/d35/db0/fd0 0 2022-01-31T19:44:30.534 INFO:tasks.workunit.client.1.smithi171.stdout:1/667: write dd/d23/f9e [444517,76992] 0 2022-01-31T19:44:30.534 INFO:tasks.workunit.client.1.smithi171.stdout:4/647: write d2/d3/d5/d41/dcf/f27 [2641707,72447] 0 2022-01-31T19:44:30.548 INFO:tasks.workunit.client.1.smithi171.stdout:5/648: dwrite d5/d90/db7/d15/f1c [4194304,4194304] 0 2022-01-31T19:44:30.549 INFO:tasks.workunit.client.1.smithi171.stdout:1/668: getdents dd/df/d19/d1b 0 2022-01-31T19:44:30.558 INFO:tasks.workunit.client.1.smithi171.stdout:5/649: mkdir d5/d90/db7/d15/d23/d67/dd8 0 2022-01-31T19:44:30.559 INFO:tasks.workunit.client.1.smithi171.stdout:5/650: chown d5/d90/db7/d15/f64 5298161 1 2022-01-31T19:44:30.584 INFO:tasks.workunit.client.1.smithi171.stdout:1/669: dwrite dd/df/d19/d36/f5b [4194304,4194304] 0 2022-01-31T19:44:30.584 INFO:tasks.workunit.client.1.smithi171.stdout:1/670: stat dd/d23 0 2022-01-31T19:44:30.584 INFO:tasks.workunit.client.1.smithi171.stdout:1/671: write dd/f42 [1285091,77944] 0 2022-01-31T19:44:30.584 INFO:tasks.workunit.client.1.smithi171.stdout:1/672: dread - dd/df/d19/fc8 zero size 2022-01-31T19:44:30.585 INFO:tasks.workunit.client.1.smithi171.stdout:1/673: symlink dd/df/d19/d1b/d1e/d35/ld1 0 2022-01-31T19:44:30.586 INFO:tasks.workunit.client.1.smithi171.stdout:1/674: write dd/df/d19/d1b/f3e [7988631,37441] 0 2022-01-31T19:44:30.586 INFO:tasks.workunit.client.1.smithi171.stdout:1/675: write dd/d23/d34/d4b/f52 [1506739,89158] 0 2022-01-31T19:44:30.586 INFO:tasks.workunit.client.1.smithi171.stdout:1/676: write dd/df/d19/d1b/d1e/d64/fc7 [17629,125924] 0 2022-01-31T19:44:30.587 INFO:tasks.workunit.client.1.smithi171.stdout:1/677: readlink dd/d23/d2e/d5e/d83/lac 0 2022-01-31T19:44:30.587 INFO:tasks.workunit.client.1.smithi171.stdout:1/678: readlink dd/d23/d2e/l6f 0 2022-01-31T19:44:30.588 INFO:tasks.workunit.client.1.smithi171.stdout:6/700: sync 2022-01-31T19:44:30.588 INFO:tasks.workunit.client.1.smithi171.stdout:6/701: readlink dc/d19/d29/l74 0 2022-01-31T19:44:30.589 INFO:tasks.workunit.client.1.smithi171.stdout:0/730: sync 2022-01-31T19:44:30.589 INFO:tasks.workunit.client.1.smithi171.stdout:5/651: dwrite d5/d90/db7/d15/d5d/fb9 [0,4194304] 0 2022-01-31T19:44:30.593 INFO:tasks.workunit.client.1.smithi171.stdout:6/702: rename dc/d19/d29/d6a/cc4 to dc/d19/d30/d3c/d80/cea 0 2022-01-31T19:44:30.594 INFO:tasks.workunit.client.1.smithi171.stdout:0/731: symlink d9/d24/d4c/d35/d67/l114 0 2022-01-31T19:44:30.598 INFO:tasks.workunit.client.1.smithi171.stdout:6/703: dread dc/d19/d29/d27/d37/d3b/d67/f9f [0,4194304] 0 2022-01-31T19:44:30.599 INFO:tasks.workunit.client.1.smithi171.stdout:6/704: fsync dc/d19/d30/d3c/d80/fcc 0 2022-01-31T19:44:30.601 INFO:tasks.workunit.client.1.smithi171.stdout:6/705: rename dc/d1e/d3f/fdc to dc/d19/d29/d27/d37/d3b/d67/feb 0 2022-01-31T19:44:30.604 INFO:tasks.workunit.client.1.smithi171.stdout:6/706: rename dc/d18/f22 to dc/d19/d29/fec 0 2022-01-31T19:44:30.604 INFO:tasks.workunit.client.1.smithi171.stdout:6/707: write dc/fe3 [68071,89811] 0 2022-01-31T19:44:30.605 INFO:tasks.workunit.client.1.smithi171.stdout:6/708: creat dc/d19/d29/daf/fed x:0 0 0 2022-01-31T19:44:30.606 INFO:tasks.workunit.client.1.smithi171.stdout:6/709: mkdir dc/d8b/dee 0 2022-01-31T19:44:30.623 INFO:tasks.workunit.client.1.smithi171.stdout:5/652: dwrite d5/d90/db7/d15/d5d/fd0 [0,4194304] 0 2022-01-31T19:44:30.623 INFO:tasks.workunit.client.1.smithi171.stdout:5/653: creat d5/d90/db7/d15/d5d/d6f/fd9 x:0 0 0 2022-01-31T19:44:30.623 INFO:tasks.workunit.client.1.smithi171.stdout:5/654: fsync d5/d90/db7/d15/d23/d3d/fb2 0 2022-01-31T19:44:30.626 INFO:tasks.workunit.client.1.smithi171.stdout:5/655: write d5/d90/db7/d15/f1c [439982,58474] 0 2022-01-31T19:44:30.627 INFO:tasks.workunit.client.1.smithi171.stdout:5/656: write d5/d90/db7/dd/f45 [2946102,108328] 0 2022-01-31T19:44:30.627 INFO:tasks.workunit.client.1.smithi171.stdout:5/657: chown d5/d90/db7/d15/d23/d67/c60 662 1 2022-01-31T19:44:30.630 INFO:tasks.workunit.client.1.smithi171.stdout:1/679: dwrite dd/df/d19/d1b/d1e/d64/f58 [12582912,4194304] 0 2022-01-31T19:44:30.631 INFO:tasks.workunit.client.1.smithi171.stdout:0/732: dwrite d9/d24/d4c/d35/f8d [0,4194304] 0 2022-01-31T19:44:30.631 INFO:tasks.workunit.client.1.smithi171.stdout:0/733: unlink d9/db/dc/d3f/db9/f10d 0 2022-01-31T19:44:30.631 INFO:tasks.workunit.client.1.smithi171.stdout:0/734: chown d9/db/lf 481945843 1 2022-01-31T19:44:30.631 INFO:tasks.workunit.client.1.smithi171.stdout:0/735: getdents d9/db/dc2 0 2022-01-31T19:44:30.632 INFO:tasks.workunit.client.1.smithi171.stdout:0/736: fdatasync d9/f16 0 2022-01-31T19:44:30.632 INFO:tasks.workunit.client.1.smithi171.stdout:1/680: truncate dd/f56 4120630 0 2022-01-31T19:44:30.633 INFO:tasks.workunit.client.1.smithi171.stdout:1/681: chown dd/d23/d2e/f2f 70 1 2022-01-31T19:44:30.633 INFO:tasks.workunit.client.1.smithi171.stdout:6/710: dwrite f9 [4194304,4194304] 0 2022-01-31T19:44:30.633 INFO:tasks.workunit.client.1.smithi171.stdout:6/711: chown dc/d8b/le8 1210344841 1 2022-01-31T19:44:30.636 INFO:tasks.workunit.client.1.smithi171.stdout:0/737: mkdir d9/d24/d92/d9d/dc6/d115 0 2022-01-31T19:44:30.637 INFO:tasks.workunit.client.1.smithi171.stdout:1/682: creat dd/df/d19/d1b/d1e/d64/db9/fd2 x:0 0 0 2022-01-31T19:44:30.637 INFO:tasks.workunit.client.1.smithi171.stdout:1/683: dread - dd/d23/d2e/d5e/d81/d98/fc5 zero size 2022-01-31T19:44:30.638 INFO:tasks.workunit.client.1.smithi171.stdout:1/684: chown dd/df/d19/d1b/f3e 24 1 2022-01-31T19:44:30.639 INFO:tasks.workunit.client.1.smithi171.stdout:0/738: mknod d9/db/d59/de9/c116 0 2022-01-31T19:44:30.643 INFO:tasks.workunit.client.1.smithi171.stdout:1/685: rename dd/df/l9b to dd/df/d19/d1b/d1e/d64/db9/ld3 0 2022-01-31T19:44:30.643 INFO:tasks.workunit.client.1.smithi171.stdout:0/739: getdents d9/db/d59/d71/dba/dbb 0 2022-01-31T19:44:30.644 INFO:tasks.workunit.client.1.smithi171.stdout:0/740: readlink d9/db/d59/d71/lf7 0 2022-01-31T19:44:30.649 INFO:tasks.workunit.client.1.smithi171.stdout:6/712: dread dc/d19/d30/d3c/d80/fd4 [0,4194304] 0 2022-01-31T19:44:30.652 INFO:tasks.workunit.client.1.smithi171.stdout:1/686: write dd/df/d19/d36/f5b [322365,105195] 0 2022-01-31T19:44:30.652 INFO:tasks.workunit.client.1.smithi171.stdout:1/687: write dd/d23/d2e/f86 [625680,57724] 0 2022-01-31T19:44:30.657 INFO:tasks.workunit.client.1.smithi171.stdout:1/688: dread dd/f42 [0,4194304] 0 2022-01-31T19:44:30.657 INFO:tasks.workunit.client.1.smithi171.stdout:1/689: chown dd/df/d19/d1b/d1e/d97 37525 1 2022-01-31T19:44:30.658 INFO:tasks.workunit.client.1.smithi171.stdout:1/690: mkdir dd/d23/dd4 0 2022-01-31T19:44:30.668 INFO:tasks.workunit.client.1.smithi171.stdout:0/741: dwrite d9/f13 [8388608,4194304] 0 2022-01-31T19:44:30.669 INFO:tasks.workunit.client.1.smithi171.stdout:0/742: creat d9/db/d59/de9/f117 x:0 0 0 2022-01-31T19:44:30.677 INFO:tasks.workunit.client.1.smithi171.stdout:6/713: dwrite dc/d1e/d3f/f59 [0,4194304] 0 2022-01-31T19:44:30.679 INFO:tasks.workunit.client.1.smithi171.stdout:0/743: dread d9/db/dc/f104 [0,4194304] 0 2022-01-31T19:44:30.679 INFO:tasks.workunit.client.1.smithi171.stdout:1/691: dwrite dd/df/d19/fc6 [0,4194304] 0 2022-01-31T19:44:30.683 INFO:tasks.workunit.client.1.smithi171.stdout:0/744: rename d9/db/f90 to d9/db/d59/d71/dba/f118 0 2022-01-31T19:44:30.684 INFO:tasks.workunit.client.1.smithi171.stdout:0/745: fdatasync d9/f1f 0 2022-01-31T19:44:30.684 INFO:tasks.workunit.client.1.smithi171.stdout:6/714: dread dc/d19/d30/f92 [0,4194304] 0 2022-01-31T19:44:30.685 INFO:tasks.workunit.client.1.smithi171.stdout:6/715: truncate dc/d8b/fc5 291373 0 2022-01-31T19:44:30.685 INFO:tasks.workunit.client.1.smithi171.stdout:1/692: mkdir dd/d23/d74/d7b/dd5 0 2022-01-31T19:44:30.693 INFO:tasks.workunit.client.1.smithi171.stdout:1/693: dread dd/d23/d2e/fa5 [0,4194304] 0 2022-01-31T19:44:30.696 INFO:tasks.workunit.client.1.smithi171.stdout:0/746: rmdir d9 39 2022-01-31T19:44:30.697 INFO:tasks.workunit.client.1.smithi171.stdout:0/747: fsync d9/fa0 0 2022-01-31T19:44:30.698 INFO:tasks.workunit.client.1.smithi171.stdout:1/694: mkdir dd/df/d19/d1b/d1e/d35/dd6 0 2022-01-31T19:44:30.698 INFO:tasks.workunit.client.1.smithi171.stdout:1/695: dread - dd/df/d19/d1b/fbe zero size 2022-01-31T19:44:30.699 INFO:tasks.workunit.client.1.smithi171.stdout:6/716: dwrite dc/d19/d29/f26 [0,4194304] 0 2022-01-31T19:44:30.700 INFO:tasks.workunit.client.1.smithi171.stdout:1/696: creat dd/d23/d34/d4b/fd7 x:0 0 0 2022-01-31T19:44:30.700 INFO:tasks.workunit.client.1.smithi171.stdout:1/697: getdents dd/d23/d74/dcb 0 2022-01-31T19:44:30.701 INFO:tasks.workunit.client.1.smithi171.stdout:0/748: truncate d9/db/dc/d3f/f34 4364474 0 2022-01-31T19:44:30.701 INFO:tasks.workunit.client.1.smithi171.stdout:0/749: chown d9/d24/d4c/d35/d67/l10c 2034938957 1 2022-01-31T19:44:30.702 INFO:tasks.workunit.client.1.smithi171.stdout:6/717: link dc/f17 dc/d19/d29/daf/fef 0 2022-01-31T19:44:30.703 INFO:tasks.workunit.client.1.smithi171.stdout:1/698: unlink dd/df/f12 0 2022-01-31T19:44:30.705 INFO:tasks.workunit.client.1.smithi171.stdout:6/718: mkdir dc/d19/d29/da7/df0 0 2022-01-31T19:44:30.707 INFO:tasks.workunit.client.1.smithi171.stdout:1/699: getdents dd/d23 0 2022-01-31T19:44:30.707 INFO:tasks.workunit.client.1.smithi171.stdout:1/700: chown dd/d23/d2e/d5e/d83 503955 1 2022-01-31T19:44:30.709 INFO:tasks.workunit.client.1.smithi171.stdout:6/719: mknod dc/d1e/d3f/cf1 0 2022-01-31T19:44:30.709 INFO:tasks.workunit.client.1.smithi171.stdout:6/720: readlink dc/d19/d29/daf/le1 0 2022-01-31T19:44:30.709 INFO:tasks.workunit.client.1.smithi171.stdout:6/721: unlink dc/d21/f4c 0 2022-01-31T19:44:30.709 INFO:tasks.workunit.client.1.smithi171.stdout:6/722: write dc/d19/d29/d27/d37/d3b/d67/feb [382693,113195] 0 2022-01-31T19:44:30.711 INFO:tasks.workunit.client.1.smithi171.stdout:1/701: mkdir dd/d23/d2e/d5e/dd8 0 2022-01-31T19:44:30.711 INFO:tasks.workunit.client.1.smithi171.stdout:1/702: write dd/df/d19/d1b/f4e [4442111,71359] 0 2022-01-31T19:44:30.711 INFO:tasks.workunit.client.1.smithi171.stdout:1/703: unlink dd/df/d19/fc8 0 2022-01-31T19:44:30.718 INFO:tasks.workunit.client.1.smithi171.stdout:1/704: creat dd/df/d19/dce/fd9 x:0 0 0 2022-01-31T19:44:30.719 INFO:tasks.workunit.client.1.smithi171.stdout:1/705: truncate dd/df/d19/d1b/f3e 656942 0 2022-01-31T19:44:30.721 INFO:tasks.workunit.client.1.smithi171.stdout:1/706: mkdir dd/df/d19/d1b/d1e/d64/dda 0 2022-01-31T19:44:30.724 INFO:tasks.workunit.client.1.smithi171.stdout:1/707: rename dd/df/d19/d1b/d1e/d97 to dd/d23/d74/dcb/ddb 0 2022-01-31T19:44:30.731 INFO:tasks.workunit.client.1.smithi171.stdout:0/750: dwrite d9/db/dc/d3f/db9/d7a/d88/fa3 [0,4194304] 0 2022-01-31T19:44:30.731 INFO:tasks.workunit.client.1.smithi171.stdout:0/751: chown d9/db/d22/l105 390 1 2022-01-31T19:44:30.731 INFO:tasks.workunit.client.1.smithi171.stdout:0/752: readlink d9/db/la7 0 2022-01-31T19:44:30.738 INFO:tasks.workunit.client.1.smithi171.stdout:0/753: rename d9/d24/d4c/d7d/fca to d9/db/dc2/f119 0 2022-01-31T19:44:30.743 INFO:tasks.workunit.client.1.smithi171.stdout:1/708: dwrite dd/d23/d2e/d5e/d81/d98/f85 [0,4194304] 0 2022-01-31T19:44:30.744 INFO:tasks.workunit.client.1.smithi171.stdout:6/723: dwrite dc/d19/d29/d27/d37/d3b/d67/dd1/fdd [0,4194304] 0 2022-01-31T19:44:30.751 INFO:tasks.workunit.client.1.smithi171.stdout:6/724: link dc/d19/d29/c62 dc/d19/d29/daf/cf2 0 2022-01-31T19:44:30.752 INFO:tasks.workunit.client.1.smithi171.stdout:1/709: rmdir dd/df/d19/d1b/d1e 39 2022-01-31T19:44:30.754 INFO:tasks.workunit.client.1.smithi171.stdout:1/710: mknod dd/df/d84/cdc 0 2022-01-31T19:44:30.757 INFO:tasks.workunit.client.1.smithi171.stdout:1/711: unlink dd/c6e 0 2022-01-31T19:44:30.757 INFO:tasks.workunit.client.1.smithi171.stdout:1/712: chown dd/df/d19/d1b/d1e/fc9 16380 1 2022-01-31T19:44:30.757 INFO:tasks.workunit.client.1.smithi171.stdout:1/713: write dd/df/d19/dce/fd9 [121574,55998] 0 2022-01-31T19:44:30.758 INFO:tasks.workunit.client.1.smithi171.stdout:0/754: dwrite d9/db/d22/fab [0,4194304] 0 2022-01-31T19:44:30.770 INFO:tasks.workunit.client.1.smithi171.stdout:7/555: sync 2022-01-31T19:44:30.770 INFO:tasks.workunit.client.1.smithi171.stdout:8/631: sync 2022-01-31T19:44:30.772 INFO:tasks.workunit.client.1.smithi171.stdout:7/556: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9 0 2022-01-31T19:44:30.772 INFO:tasks.workunit.client.1.smithi171.stdout:7/557: write d1/d1e/d2a/d29/f43 [1326439,5274] 0 2022-01-31T19:44:30.773 INFO:tasks.workunit.client.1.smithi171.stdout:9/739: sync 2022-01-31T19:44:30.773 INFO:tasks.workunit.client.1.smithi171.stdout:9/740: chown dd/df/d15/d1a/f4b 166 1 2022-01-31T19:44:30.776 INFO:tasks.workunit.client.1.smithi171.stdout:9/741: rename dd/df/d3f/de3 to dd/df/d15/d38/d4d/d80/dfa 0 2022-01-31T19:44:30.779 INFO:tasks.workunit.client.1.smithi171.stdout:9/742: creat dd/df/d15/d38/d9c/da7/ffb x:0 0 0 2022-01-31T19:44:30.779 INFO:tasks.workunit.client.1.smithi171.stdout:9/743: write dd/df/d15/de4/ff7 [1387296,24106] 0 2022-01-31T19:44:30.782 INFO:tasks.workunit.client.1.smithi171.stdout:9/744: getdents dd/df/d15/d53 0 2022-01-31T19:44:30.782 INFO:tasks.workunit.client.1.smithi171.stdout:9/745: chown dd/df/d15/d1a/d64 332578 1 2022-01-31T19:44:30.782 INFO:tasks.workunit.client.1.smithi171.stdout:9/746: read - dd/df/d15/d38/d41/fd7 zero size 2022-01-31T19:44:30.788 INFO:tasks.workunit.client.1.smithi171.stdout:6/725: dwrite dc/d21/f8d [0,4194304] 0 2022-01-31T19:44:30.789 INFO:tasks.workunit.client.1.smithi171.stdout:6/726: chown dc/d19/d29/d27/d37/d3b/cad 1589461 1 2022-01-31T19:44:30.789 INFO:tasks.workunit.client.1.smithi171.stdout:1/714: dwrite dd/d23/d34/d4b/f52 [0,4194304] 0 2022-01-31T19:44:30.791 INFO:tasks.workunit.client.1.smithi171.stdout:0/755: dwrite d9/db/dc/d3f/db9/d7a/fe7 [0,4194304] 0 2022-01-31T19:44:30.792 INFO:tasks.workunit.client.1.smithi171.stdout:7/558: dwrite d1/d26/d47/d8f/fa0 [0,4194304] 0 2022-01-31T19:44:30.792 INFO:tasks.workunit.client.1.smithi171.stdout:6/727: read dc/f17 [359915,101531] 0 2022-01-31T19:44:30.792 INFO:tasks.workunit.client.1.smithi171.stdout:6/728: fsync dc/d19/d29/f63 0 2022-01-31T19:44:30.792 INFO:tasks.workunit.client.1.smithi171.stdout:7/559: truncate d1/d1e/f78 847194 0 2022-01-31T19:44:30.792 INFO:tasks.workunit.client.1.smithi171.stdout:7/560: fdatasync d1/d26/d47/d8f/fa0 0 2022-01-31T19:44:30.796 INFO:tasks.workunit.client.1.smithi171.stdout:6/729: rename dc/d21/d4f/c91 to dc/d19/d29/d27/d37/cf3 0 2022-01-31T19:44:30.796 INFO:tasks.workunit.client.1.smithi171.stdout:9/747: dread dd/df/d15/d1a/f2b [0,4194304] 0 2022-01-31T19:44:30.797 INFO:tasks.workunit.client.1.smithi171.stdout:7/561: truncate d1/d1e/d2a/d29/d31/f6a 7879318 0 2022-01-31T19:44:30.798 INFO:tasks.workunit.client.1.smithi171.stdout:6/730: write fa [2932588,96647] 0 2022-01-31T19:44:30.799 INFO:tasks.workunit.client.1.smithi171.stdout:6/731: mkdir dc/d19/df4 0 2022-01-31T19:44:30.800 INFO:tasks.workunit.client.1.smithi171.stdout:6/732: readlink dc/d19/d29/d27/d37/l94 0 2022-01-31T19:44:30.800 INFO:tasks.workunit.client.1.smithi171.stdout:6/733: creat dc/d19/d29/daf/ff5 x:0 0 0 2022-01-31T19:44:30.808 INFO:tasks.workunit.client.1.smithi171.stdout:6/734: dread dc/d19/d30/f77 [4194304,4194304] 0 2022-01-31T19:44:30.810 INFO:tasks.workunit.client.1.smithi171.stdout:6/735: mknod dc/d19/d29/d27/d37/d3b/d67/dd1/cf6 0 2022-01-31T19:44:30.810 INFO:tasks.workunit.client.1.smithi171.stdout:6/736: fdatasync dc/d19/d29/da7/fb8 0 2022-01-31T19:44:30.811 INFO:tasks.workunit.client.1.smithi171.stdout:6/737: rename dc/d19/d29/da7/l79 to dc/d19/df4/lf7 0 2022-01-31T19:44:30.814 INFO:tasks.workunit.client.1.smithi171.stdout:6/738: link dc/d21/fa2 dc/d19/d29/d27/d37/ff8 0 2022-01-31T19:44:30.816 INFO:tasks.workunit.client.1.smithi171.stdout:1/715: dwrite f3 [0,4194304] 0 2022-01-31T19:44:30.816 INFO:tasks.workunit.client.1.smithi171.stdout:0/756: dread d9/d24/d4c/d35/f8d [0,4194304] 0 2022-01-31T19:44:30.816 INFO:tasks.workunit.client.1.smithi171.stdout:1/716: chown dd/d23/d34/l62 339 1 2022-01-31T19:44:30.817 INFO:tasks.workunit.client.1.smithi171.stdout:1/717: write dd/df/d19/d1b/d1e/d64/fc7 [1182881,83225] 0 2022-01-31T19:44:30.817 INFO:tasks.workunit.client.1.smithi171.stdout:6/739: unlink dc/d19/d29/d27/d37/d3b/c8a 0 2022-01-31T19:44:30.817 INFO:tasks.workunit.client.1.smithi171.stdout:6/740: creat dc/d19/d30/d3c/d80/ff9 x:0 0 0 2022-01-31T19:44:30.817 INFO:tasks.workunit.client.1.smithi171.stdout:6/741: write dc/d18/d86/f87 [4548530,36423] 0 2022-01-31T19:44:30.818 INFO:tasks.workunit.client.1.smithi171.stdout:1/718: write dd/df/d19/d1b/d1e/d64/da7/faf [687964,5070] 0 2022-01-31T19:44:30.818 INFO:tasks.workunit.client.1.smithi171.stdout:1/719: chown dd/df/d19/d1b/d1e/f4a 1330092 1 2022-01-31T19:44:30.819 INFO:tasks.workunit.client.1.smithi171.stdout:1/720: read - dd/df/d19/d36/fab zero size 2022-01-31T19:44:30.819 INFO:tasks.workunit.client.1.smithi171.stdout:0/757: getdents d9/d24/d92 0 2022-01-31T19:44:30.822 INFO:tasks.workunit.client.1.smithi171.stdout:0/758: mkdir d9/d24/d92/d9d/d11a 0 2022-01-31T19:44:30.823 INFO:tasks.workunit.client.1.smithi171.stdout:0/759: write d9/db/fc9 [2706,92646] 0 2022-01-31T19:44:30.823 INFO:tasks.workunit.client.1.smithi171.stdout:0/760: truncate d9/f102 695927 0 2022-01-31T19:44:30.823 INFO:tasks.workunit.client.1.smithi171.stdout:0/761: stat f8 0 2022-01-31T19:44:30.824 INFO:tasks.workunit.client.1.smithi171.stdout:0/762: symlink d9/db/dc/l11b 0 2022-01-31T19:44:30.825 INFO:tasks.workunit.client.1.smithi171.stdout:0/763: rename f8 to d9/db/dc/d3f/d6f/f11c 0 2022-01-31T19:44:30.826 INFO:tasks.workunit.client.1.smithi171.stdout:0/764: write d9/d24/d4c/d35/f89 [988541,50240] 0 2022-01-31T19:44:30.831 INFO:tasks.workunit.client.1.smithi171.stdout:0/765: write d9/db/dc/f70 [2743601,28614] 0 2022-01-31T19:44:30.833 INFO:tasks.workunit.client.1.smithi171.stdout:0/766: symlink d9/db/dc2/l11d 0 2022-01-31T19:44:30.833 INFO:tasks.workunit.client.1.smithi171.stdout:3/742: sync 2022-01-31T19:44:30.834 INFO:tasks.workunit.client.1.smithi171.stdout:0/767: creat d9/d24/d92/d9d/d9f/f11e x:0 0 0 2022-01-31T19:44:30.835 INFO:tasks.workunit.client.1.smithi171.stdout:0/768: creat d9/db/dc/d3f/db9/dec/f11f x:0 0 0 2022-01-31T19:44:30.841 INFO:tasks.workunit.client.1.smithi171.stdout:3/743: dread d4/dd/d73/f77 [0,4194304] 0 2022-01-31T19:44:30.841 INFO:tasks.workunit.client.1.smithi171.stdout:3/744: fdatasync d4/dd/d39/f50 0 2022-01-31T19:44:30.842 INFO:tasks.workunit.client.1.smithi171.stdout:3/745: mkdir d4/d21/de5/d92/dab/d60/df4 0 2022-01-31T19:44:30.843 INFO:tasks.workunit.client.1.smithi171.stdout:6/742: dwrite dc/d19/d29/d6a/fbb [0,4194304] 0 2022-01-31T19:44:30.843 INFO:tasks.workunit.client.1.smithi171.stdout:3/746: read d4/d21/f31 [604945,110734] 0 2022-01-31T19:44:30.843 INFO:tasks.workunit.client.1.smithi171.stdout:3/747: dread - d4/dd/ff0 zero size 2022-01-31T19:44:30.844 INFO:tasks.workunit.client.1.smithi171.stdout:3/748: fsync d4/d21/de5/d92/dab/d52/f9b 0 2022-01-31T19:44:30.844 INFO:tasks.workunit.client.1.smithi171.stdout:1/721: dwrite dd/df/d19/d1b/d1e/d64/fc1 [0,4194304] 0 2022-01-31T19:44:30.845 INFO:tasks.workunit.client.1.smithi171.stdout:6/743: mknod dc/d19/d29/d27/d37/d3b/d67/dd1/cfa 0 2022-01-31T19:44:30.846 INFO:tasks.workunit.client.1.smithi171.stdout:3/749: creat d4/d21/d3a/db5/dbe/ff5 x:0 0 0 2022-01-31T19:44:30.847 INFO:tasks.workunit.client.1.smithi171.stdout:3/750: write d4/d21/d3a/f4f [3047317,15624] 0 2022-01-31T19:44:30.847 INFO:tasks.workunit.client.1.smithi171.stdout:3/751: stat d4/l2c 0 2022-01-31T19:44:30.848 INFO:tasks.workunit.client.1.smithi171.stdout:1/722: creat dd/d23/d2e/d5e/fdd x:0 0 0 2022-01-31T19:44:30.850 INFO:tasks.workunit.client.1.smithi171.stdout:3/752: link d4/d21/de5/cee d4/dd/dc7/cf6 0 2022-01-31T19:44:30.850 INFO:tasks.workunit.client.1.smithi171.stdout:3/753: fdatasync d4/dd/d27/d28/d35/d55/dbf/fc5 0 2022-01-31T19:44:30.851 INFO:tasks.workunit.client.1.smithi171.stdout:3/754: chown d4/dd/d27/d28/d35/c7f 1569578 1 2022-01-31T19:44:30.851 INFO:tasks.workunit.client.1.smithi171.stdout:3/755: write d4/dd/d27/d28/d35/f48 [2475499,117166] 0 2022-01-31T19:44:30.852 INFO:tasks.workunit.client.1.smithi171.stdout:3/756: dread - d4/dd/d87/f9f zero size 2022-01-31T19:44:30.853 INFO:tasks.workunit.client.1.smithi171.stdout:1/723: rmdir dd/df/d19/d1b/d1e/d64/dda 0 2022-01-31T19:44:30.853 INFO:tasks.workunit.client.1.smithi171.stdout:0/769: dread d9/db/dc/d3f/db9/d7a/d88/fa3 [0,4194304] 0 2022-01-31T19:44:30.854 INFO:tasks.workunit.client.1.smithi171.stdout:3/757: fsync d4/dd/d27/d28/d35/d55/d6c/fdc 0 2022-01-31T19:44:30.860 INFO:tasks.workunit.client.1.smithi171.stdout:0/770: unlink d9/d24/l6b 0 2022-01-31T19:44:30.861 INFO:tasks.workunit.client.1.smithi171.stdout:3/758: rmdir d4/dd/d27/d4e 39 2022-01-31T19:44:30.861 INFO:tasks.workunit.client.1.smithi171.stdout:3/759: stat d4/dd/d27/d28/d35/fd2 0 2022-01-31T19:44:30.861 INFO:tasks.workunit.client.1.smithi171.stdout:3/760: chown d4/dd/dc7/cf6 3987035 1 2022-01-31T19:44:30.862 INFO:tasks.workunit.client.1.smithi171.stdout:3/761: truncate d4/dd/d87/f9f 95673 0 2022-01-31T19:44:30.862 INFO:tasks.workunit.client.1.smithi171.stdout:3/762: fsync d4/dd/d27/d28/d35/f48 0 2022-01-31T19:44:30.862 INFO:tasks.workunit.client.1.smithi171.stdout:3/763: creat d4/dd/d27/d28/d35/ff7 x:0 0 0 2022-01-31T19:44:30.862 INFO:tasks.workunit.client.1.smithi171.stdout:3/764: truncate d4/dd/d27/d28/db6/dc0/fe0 934568 0 2022-01-31T19:44:30.863 INFO:tasks.workunit.client.1.smithi171.stdout:6/744: dread f9 [0,4194304] 0 2022-01-31T19:44:30.863 INFO:tasks.workunit.client.1.smithi171.stdout:0/771: creat d9/d24/d92/d9d/dc6/f120 x:0 0 0 2022-01-31T19:44:30.864 INFO:tasks.workunit.client.1.smithi171.stdout:3/765: mknod d4/d21/d3a/db5/dbe/dc3/dcb/cf8 0 2022-01-31T19:44:30.875 INFO:tasks.workunit.client.1.smithi171.stdout:1/724: dwrite dd/d23/d2e/fa5 [4194304,4194304] 0 2022-01-31T19:44:30.890 INFO:tasks.workunit.client.1.smithi171.stdout:3/766: dwrite d4/d21/de5/fba [0,4194304] 0 2022-01-31T19:44:30.896 INFO:tasks.workunit.client.1.smithi171.stdout:3/767: write d4/dd/d27/f3f [3062034,10556] 0 2022-01-31T19:44:30.896 INFO:tasks.workunit.client.1.smithi171.stdout:3/768: readlink d4/d21/d3a/l44 0 2022-01-31T19:44:30.898 INFO:tasks.workunit.client.1.smithi171.stdout:1/725: dwrite dd/d23/d2e/d5e/d81/d98/fc5 [0,4194304] 0 2022-01-31T19:44:30.898 INFO:tasks.workunit.client.1.smithi171.stdout:1/726: stat dd/d23 0 2022-01-31T19:44:30.898 INFO:tasks.workunit.client.1.smithi171.stdout:1/727: fdatasync dd/df/d19/d1b/d1e/f4a 0 2022-01-31T19:44:30.899 INFO:tasks.workunit.client.1.smithi171.stdout:1/728: fsync dd/df/d19/d1b/f4e 0 2022-01-31T19:44:30.900 INFO:tasks.workunit.client.1.smithi171.stdout:1/729: rename dd/df/d19/f32 to dd/d23/dd4/fde 0 2022-01-31T19:44:30.901 INFO:tasks.workunit.client.1.smithi171.stdout:1/730: mknod dd/d23/d2e/d5e/d81/d98/dad/cdf 0 2022-01-31T19:44:30.907 INFO:tasks.workunit.client.1.smithi171.stdout:3/769: dread d4/dd/d27/d28/d35/d55/f74 [0,4194304] 0 2022-01-31T19:44:30.907 INFO:tasks.workunit.client.1.smithi171.stdout:3/770: truncate d4/dd/d27/d6e/fef 886901 0 2022-01-31T19:44:30.920 INFO:tasks.workunit.client.1.smithi171.stdout:3/771: dread d4/d21/d3a/f4f [0,4194304] 0 2022-01-31T19:44:30.920 INFO:tasks.workunit.client.1.smithi171.stdout:1/731: dwrite dd/d23/d34/d4b/f59 [0,4194304] 0 2022-01-31T19:44:30.920 INFO:tasks.workunit.client.1.smithi171.stdout:1/732: truncate dd/d23/d2e/d5e/d81/d98/fc5 4944094 0 2022-01-31T19:44:30.921 INFO:tasks.workunit.client.1.smithi171.stdout:1/733: creat dd/d23/d2e/d5e/fe0 x:0 0 0 2022-01-31T19:44:30.921 INFO:tasks.workunit.client.1.smithi171.stdout:1/734: stat dd/c4d 0 2022-01-31T19:44:30.921 INFO:tasks.workunit.client.1.smithi171.stdout:1/735: dread dd/df/f38 [0,4194304] 0 2022-01-31T19:44:30.922 INFO:tasks.workunit.client.1.smithi171.stdout:3/772: write d4/dd/d27/d28/d35/d55/f5e [1213780,36213] 0 2022-01-31T19:44:30.924 INFO:tasks.workunit.client.1.smithi171.stdout:3/773: rename d4/d21/c22 to d4/d21/cf9 0 2022-01-31T19:44:30.926 INFO:tasks.workunit.client.1.smithi171.stdout:3/774: truncate d4/dd/d39/d51/f71 3812345 0 2022-01-31T19:44:30.929 INFO:tasks.workunit.client.1.smithi171.stdout:3/775: unlink d4/d21/de5/d92/dab/d6b/ca5 0 2022-01-31T19:44:30.930 INFO:tasks.workunit.client.1.smithi171.stdout:1/736: dread dd/df/d19/d1b/d1e/f4a [0,4194304] 0 2022-01-31T19:44:30.931 INFO:tasks.workunit.client.1.smithi171.stdout:1/737: fsync dd/df/d19/d1b/f4e 0 2022-01-31T19:44:30.932 INFO:tasks.workunit.client.1.smithi171.stdout:1/738: mknod dd/df/d19/d1b/d1e/d64/da7/ce1 0 2022-01-31T19:44:30.934 INFO:tasks.workunit.client.1.smithi171.stdout:1/739: symlink dd/df/d19/d1b/d1e/le2 0 2022-01-31T19:44:30.934 INFO:tasks.workunit.client.1.smithi171.stdout:1/740: truncate dd/df/d19/d1b/d1e/f6a 4892380 0 2022-01-31T19:44:30.934 INFO:tasks.workunit.client.1.smithi171.stdout:1/741: dread - dd/d23/d74/fb4 zero size 2022-01-31T19:44:30.934 INFO:tasks.workunit.client.1.smithi171.stdout:1/742: getdents dd/d23/d34 0 2022-01-31T19:44:30.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/743: chown dd/f56 0 1 2022-01-31T19:44:30.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/744: creat dd/df/d19/d1b/d1e/d64/fe3 x:0 0 0 2022-01-31T19:44:30.940 INFO:tasks.workunit.client.1.smithi171.stdout:3/776: dread d4/f99 [0,4194304] 0 2022-01-31T19:44:30.942 INFO:tasks.workunit.client.1.smithi171.stdout:3/777: mkdir d4/dd/d4b/dfa 0 2022-01-31T19:44:30.944 INFO:tasks.workunit.client.1.smithi171.stdout:3/778: mkdir d4/d21/de5/da7/dfb 0 2022-01-31T19:44:30.956 INFO:tasks.workunit.client.1.smithi171.stdout:1/745: dwrite dd/d23/d2e/d5e/f8a [0,4194304] 0 2022-01-31T19:44:30.956 INFO:tasks.workunit.client.1.smithi171.stdout:1/746: chown dd/df/d19/d36/c89 2 1 2022-01-31T19:44:30.960 INFO:tasks.workunit.client.1.smithi171.stdout:1/747: write dd/df/d84/fa1 [2001183,107110] 0 2022-01-31T19:44:30.960 INFO:tasks.workunit.client.1.smithi171.stdout:1/748: stat dd/d23/d2e/d5e/d81/fa8 0 2022-01-31T19:44:30.961 INFO:tasks.workunit.client.1.smithi171.stdout:1/749: chown dd/df/d19/d1b/d1e/d64/fc1 1063334642 1 2022-01-31T19:44:30.961 INFO:tasks.workunit.client.1.smithi171.stdout:1/750: dread dd/da2/fb2 [0,4194304] 0 2022-01-31T19:44:30.962 INFO:tasks.workunit.client.1.smithi171.stdout:1/751: stat dd/d23/d2e/d5e/d81/d98/d80/f8e 0 2022-01-31T19:44:30.963 INFO:tasks.workunit.client.1.smithi171.stdout:3/779: dwrite d4/d21/de5/fba [0,4194304] 0 2022-01-31T19:44:30.964 INFO:tasks.workunit.client.1.smithi171.stdout:3/780: creat d4/dd/d27/d28/d35/d55/dbf/ffc x:0 0 0 2022-01-31T19:44:30.965 INFO:tasks.workunit.client.1.smithi171.stdout:3/781: mknod d4/d21/de5/dde/cfd 0 2022-01-31T19:44:30.966 INFO:tasks.workunit.client.1.smithi171.stdout:3/782: symlink d4/dd/dc7/lfe 0 2022-01-31T19:44:30.968 INFO:tasks.workunit.client.1.smithi171.stdout:3/783: creat d4/dd/d39/d51/db2/fff x:0 0 0 2022-01-31T19:44:30.969 INFO:tasks.workunit.client.1.smithi171.stdout:3/784: truncate d4/dd/d87/fa6 523291 0 2022-01-31T19:44:30.969 INFO:tasks.workunit.client.1.smithi171.stdout:3/785: chown d4/d21/de5/d92/dab 181860413 1 2022-01-31T19:44:30.970 INFO:tasks.workunit.client.1.smithi171.stdout:3/786: dread d4/dd/d39/d51/fb4 [0,4194304] 0 2022-01-31T19:44:30.971 INFO:tasks.workunit.client.1.smithi171.stdout:3/787: symlink d4/d21/de5/d92/dab/dd4/l100 0 2022-01-31T19:44:30.971 INFO:tasks.workunit.client.1.smithi171.stdout:3/788: creat d4/dd/d39/d51/f101 x:0 0 0 2022-01-31T19:44:30.973 INFO:tasks.workunit.client.1.smithi171.stdout:3/789: unlink d4/dd/c36 0 2022-01-31T19:44:30.973 INFO:tasks.workunit.client.1.smithi171.stdout:3/790: creat d4/dd/d27/d28/d35/d55/d6c/f102 x:0 0 0 2022-01-31T19:44:30.978 INFO:tasks.workunit.client.1.smithi171.stdout:1/752: dwrite dd/df/d19/d1b/d1e/d64/fc1 [0,4194304] 0 2022-01-31T19:44:30.978 INFO:tasks.workunit.client.1.smithi171.stdout:1/753: readlink dd/da2/lbd 0 2022-01-31T19:44:30.978 INFO:tasks.workunit.client.1.smithi171.stdout:1/754: readlink dd/df/d19/d1b/l2b 0 2022-01-31T19:44:30.980 INFO:tasks.workunit.client.1.smithi171.stdout:3/791: dread d4/dd/f19 [0,4194304] 0 2022-01-31T19:44:30.981 INFO:tasks.workunit.client.1.smithi171.stdout:3/792: write d4/d21/de5/d92/fe1 [686557,82950] 0 2022-01-31T19:44:30.981 INFO:tasks.workunit.client.1.smithi171.stdout:3/793: chown d4/d21/de5/d92/dab/d6b 1130528 1 2022-01-31T19:44:30.981 INFO:tasks.workunit.client.1.smithi171.stdout:3/794: fdatasync d4/dd/d87/f9f 0 2022-01-31T19:44:30.981 INFO:tasks.workunit.client.1.smithi171.stdout:3/795: write d4/d21/de5/d92/dab/dd4/fdb [965251,129052] 0 2022-01-31T19:44:30.990 INFO:tasks.workunit.client.1.smithi171.stdout:1/755: rmdir dd 39 2022-01-31T19:44:30.990 INFO:tasks.workunit.client.1.smithi171.stdout:1/756: write dd/d23/d2e/d5e/f40 [484893,42975] 0 2022-01-31T19:44:30.991 INFO:tasks.workunit.client.1.smithi171.stdout:3/796: mknod d4/dd/d27/d6e/c103 0 2022-01-31T19:44:30.992 INFO:tasks.workunit.client.1.smithi171.stdout:3/797: mknod d4/dd/d27/d28/db6/dc0/c104 0 2022-01-31T19:44:30.994 INFO:tasks.workunit.client.1.smithi171.stdout:3/798: creat d4/f105 x:0 0 0 2022-01-31T19:44:30.994 INFO:tasks.workunit.client.1.smithi171.stdout:3/799: write d4/d21/d3a/d59/dc1/fc2 [675111,82505] 0 2022-01-31T19:44:31.011 INFO:tasks.workunit.client.1.smithi171.stdout:1/757: dwrite dd/df/d19/d1b/d1e/f6a [4194304,4194304] 0 2022-01-31T19:44:31.012 INFO:tasks.workunit.client.1.smithi171.stdout:1/758: chown dd/df/d19/d1b/f4e 493 1 2022-01-31T19:44:31.012 INFO:tasks.workunit.client.1.smithi171.stdout:3/800: dwrite d4/dd/d27/d28/d35/f38 [4194304,4194304] 0 2022-01-31T19:44:31.013 INFO:tasks.workunit.client.1.smithi171.stdout:1/759: write dd/df/d19/d1b/f63 [1249531,110646] 0 2022-01-31T19:44:31.017 INFO:tasks.workunit.client.1.smithi171.stdout:3/801: unlink d4/dd/d27/d28/d35/d55/f74 0 2022-01-31T19:44:31.018 INFO:tasks.workunit.client.1.smithi171.stdout:3/802: dread - d4/dd/d39/d51/f101 zero size 2022-01-31T19:44:31.018 INFO:tasks.workunit.client.1.smithi171.stdout:1/760: creat dd/d23/d74/dcb/ddb/fe4 x:0 0 0 2022-01-31T19:44:31.019 INFO:tasks.workunit.client.1.smithi171.stdout:3/803: getdents d4/d21/de5/d92 0 2022-01-31T19:44:31.019 INFO:tasks.workunit.client.1.smithi171.stdout:3/804: truncate d4/d21/d3a/db5/fd5 1286125 0 2022-01-31T19:44:31.029 INFO:tasks.workunit.client.1.smithi171.stdout:3/805: dread d4/fa [0,4194304] 0 2022-01-31T19:44:31.029 INFO:tasks.workunit.client.1.smithi171.stdout:3/806: fdatasync d4/dd/f45 0 2022-01-31T19:44:31.036 INFO:tasks.workunit.client.1.smithi171.stdout:1/761: dwrite dd/d23/d2e/d5e/d81/d98/d80/f8e [0,4194304] 0 2022-01-31T19:44:31.038 INFO:tasks.workunit.client.1.smithi171.stdout:1/762: link fa dd/d23/d34/d4b/fe5 0 2022-01-31T19:44:31.039 INFO:tasks.workunit.client.1.smithi171.stdout:1/763: unlink dd/df/l15 0 2022-01-31T19:44:31.048 INFO:tasks.workunit.client.1.smithi171.stdout:1/764: fdatasync dd/df/d19/d1b/f63 0 2022-01-31T19:44:31.048 INFO:tasks.workunit.client.1.smithi171.stdout:1/765: write dd/d23/d2e/d5e/d81/fae [583656,4383] 0 2022-01-31T19:44:31.049 INFO:tasks.workunit.client.1.smithi171.stdout:1/766: rmdir dd/d23/d74/d7b/dd5 0 2022-01-31T19:44:31.050 INFO:tasks.workunit.client.1.smithi171.stdout:1/767: rmdir dd/df/d19/d1b 39 2022-01-31T19:44:31.052 INFO:tasks.workunit.client.1.smithi171.stdout:3/807: dwrite d4/dd/d27/d28/d35/d55/d6c/fad [0,4194304] 0 2022-01-31T19:44:31.052 INFO:tasks.workunit.client.1.smithi171.stdout:3/808: creat d4/dd/d27/f106 x:0 0 0 2022-01-31T19:44:31.053 INFO:tasks.workunit.client.1.smithi171.stdout:3/809: fdatasync d4/dd/d27/d28/d35/f38 0 2022-01-31T19:44:31.056 INFO:tasks.workunit.client.1.smithi171.stdout:3/810: unlink d4/dd/d27/d28/d35/d55/d6c/fad 0 2022-01-31T19:44:31.058 INFO:tasks.workunit.client.1.smithi171.stdout:2/852: sync 2022-01-31T19:44:31.058 INFO:tasks.workunit.client.1.smithi171.stdout:4/648: sync 2022-01-31T19:44:31.058 INFO:tasks.workunit.client.1.smithi171.stdout:5/658: sync 2022-01-31T19:44:31.059 INFO:tasks.workunit.client.1.smithi171.stdout:4/649: getdents d2/d3/d5/d41/dcf/d12/d15/dc4 0 2022-01-31T19:44:31.059 INFO:tasks.workunit.client.1.smithi171.stdout:4/650: fdatasync d2/f5b 0 2022-01-31T19:44:31.061 INFO:tasks.workunit.client.1.smithi171.stdout:2/853: creat d7/dd/dfc/f105 x:0 0 0 2022-01-31T19:44:31.062 INFO:tasks.workunit.client.1.smithi171.stdout:4/651: link d2/d3/d5/ldd d2/d3/d5/d41/dcf/d12/db6/lde 0 2022-01-31T19:44:31.063 INFO:tasks.workunit.client.1.smithi171.stdout:2/854: mknod d7/dd/dfc/d101/c106 0 2022-01-31T19:44:31.070 INFO:tasks.workunit.client.1.smithi171.stdout:3/811: getdents d4/dd/d27/d28/d35/d55/d6c 0 2022-01-31T19:44:31.070 INFO:tasks.workunit.client.1.smithi171.stdout:3/812: write d4/fc6 [4844447,125617] 0 2022-01-31T19:44:31.070 INFO:tasks.workunit.client.1.smithi171.stdout:3/813: read d4/d21/de5/d92/dab/f81 [164905,122319] 0 2022-01-31T19:44:31.070 INFO:tasks.workunit.client.1.smithi171.stdout:3/814: fsync d4/fa 0 2022-01-31T19:44:31.073 INFO:tasks.workunit.client.1.smithi171.stdout:3/815: symlink d4/dd/d39/d51/l107 0 2022-01-31T19:44:31.073 INFO:tasks.workunit.client.1.smithi171.stdout:3/816: mkdir d4/d21/de5/d92/dab/d52/d7a/d108 0 2022-01-31T19:44:31.074 INFO:tasks.workunit.client.1.smithi171.stdout:3/817: readlink d4/d21/l72 0 2022-01-31T19:44:31.074 INFO:tasks.workunit.client.1.smithi171.stdout:3/818: mknod d4/dd/d27/d4e/c109 0 2022-01-31T19:44:31.081 INFO:tasks.workunit.client.1.smithi171.stdout:5/659: dwrite d5/d90/db7/d15/d23/f87 [0,4194304] 0 2022-01-31T19:44:31.081 INFO:tasks.workunit.client.1.smithi171.stdout:5/660: fdatasync d5/d90/db7/d15/d23/d67/f76 0 2022-01-31T19:44:31.082 INFO:tasks.workunit.client.1.smithi171.stdout:3/819: dread d4/dd/d27/f47 [0,4194304] 0 2022-01-31T19:44:31.082 INFO:tasks.workunit.client.1.smithi171.stdout:3/820: symlink d4/d21/d3a/db5/dbe/dc3/dcb/l10a 0 2022-01-31T19:44:31.084 INFO:tasks.workunit.client.1.smithi171.stdout:3/821: dread d4/dd/d39/f50 [0,4194304] 0 2022-01-31T19:44:31.085 INFO:tasks.workunit.client.1.smithi171.stdout:3/822: mkdir d4/d21/de5/d92/dab/d52/d10b 0 2022-01-31T19:44:31.085 INFO:tasks.workunit.client.1.smithi171.stdout:3/823: chown d4/d21/de5/d92/l32 80974698 1 2022-01-31T19:44:31.085 INFO:tasks.workunit.client.1.smithi171.stdout:3/824: creat d4/d21/d3a/db5/dbe/f10c x:0 0 0 2022-01-31T19:44:31.086 INFO:tasks.workunit.client.1.smithi171.stdout:5/661: write d5/d90/db7/d15/d23/f87 [177415,27327] 0 2022-01-31T19:44:31.087 INFO:tasks.workunit.client.1.smithi171.stdout:3/825: rename d4/dd/d4b/fcf to d4/d21/d3a/d90/f10d 0 2022-01-31T19:44:31.087 INFO:tasks.workunit.client.1.smithi171.stdout:3/826: unlink d4/d21/d3a/f7c 0 2022-01-31T19:44:31.102 INFO:tasks.workunit.client.1.smithi171.stdout:5/662: dread d5/d90/db7/d15/d5d/fb9 [0,4194304] 0 2022-01-31T19:44:31.103 INFO:tasks.workunit.client.1.smithi171.stdout:2/855: dwrite d7/dd/f3c [0,4194304] 0 2022-01-31T19:44:31.103 INFO:tasks.workunit.client.1.smithi171.stdout:2/856: stat d7/d26/d63/d9f/la0 0 2022-01-31T19:44:31.104 INFO:tasks.workunit.client.1.smithi171.stdout:5/663: unlink d5/d90/db7/d15/d23/d67/f39 0 2022-01-31T19:44:31.104 INFO:tasks.workunit.client.1.smithi171.stdout:4/652: dwrite d2/d7f/fa8 [0,4194304] 0 2022-01-31T19:44:31.104 INFO:tasks.workunit.client.1.smithi171.stdout:3/827: dread d4/d21/de5/d92/dab/f2b [0,4194304] 0 2022-01-31T19:44:31.105 INFO:tasks.workunit.client.1.smithi171.stdout:4/653: creat d2/d3/d5/dc/fdf x:0 0 0 2022-01-31T19:44:31.105 INFO:tasks.workunit.client.1.smithi171.stdout:3/828: write d4/d21/de5/d92/dab/d52/f9b [1274964,13532] 0 2022-01-31T19:44:31.106 INFO:tasks.workunit.client.1.smithi171.stdout:2/857: mkdir d7/d26/de5/dcb/d107 0 2022-01-31T19:44:31.107 INFO:tasks.workunit.client.1.smithi171.stdout:5/664: creat d5/db0/dc8/fda x:0 0 0 2022-01-31T19:44:31.108 INFO:tasks.workunit.client.1.smithi171.stdout:5/665: write d5/d90/db7/d15/d5d/fc4 [1117511,118022] 0 2022-01-31T19:44:31.117 INFO:tasks.workunit.client.1.smithi171.stdout:4/654: creat d2/d3/d5/d41/dcf/d12/d15/dc4/fe0 x:0 0 0 2022-01-31T19:44:31.117 INFO:tasks.workunit.client.1.smithi171.stdout:4/655: truncate d2/d3/d5/dc/d4b/d56/fb9 388234 0 2022-01-31T19:44:31.118 INFO:tasks.workunit.client.1.smithi171.stdout:3/829: mkdir d4/d21/de5/da7/dfb/d10e 0 2022-01-31T19:44:31.119 INFO:tasks.workunit.client.1.smithi171.stdout:3/830: chown d4/dd/d27/f83 50 1 2022-01-31T19:44:31.120 INFO:tasks.workunit.client.1.smithi171.stdout:5/666: rename d5/d59/c93 to d5/d90/db7/cdb 0 2022-01-31T19:44:31.120 INFO:tasks.workunit.client.1.smithi171.stdout:5/667: creat d5/d90/db7/d15/d80/fdc x:0 0 0 2022-01-31T19:44:31.120 INFO:tasks.workunit.client.1.smithi171.stdout:5/668: creat d5/d90/db7/d15/d5d/d6f/fdd x:0 0 0 2022-01-31T19:44:31.121 INFO:tasks.workunit.client.1.smithi171.stdout:5/669: fdatasync d5/d90/db7/d15/d80/fcb 0 2022-01-31T19:44:31.121 INFO:tasks.workunit.client.1.smithi171.stdout:5/670: chown d5/d90/db7/d15/d23/d67/c6e 248238 1 2022-01-31T19:44:31.123 INFO:tasks.workunit.client.1.smithi171.stdout:5/671: unlink d5/d90/db7/dd/d66/c72 0 2022-01-31T19:44:31.124 INFO:tasks.workunit.client.1.smithi171.stdout:2/858: dwrite d7/d9/d1a/d45/da4/fab [0,4194304] 0 2022-01-31T19:44:31.124 INFO:tasks.workunit.client.1.smithi171.stdout:2/859: chown d7/d26/de5/f9e 52879408 1 2022-01-31T19:44:31.127 INFO:tasks.workunit.client.1.smithi171.stdout:4/656: rename d2/d7f/dce to d2/d3/de1 0 2022-01-31T19:44:31.127 INFO:tasks.workunit.client.1.smithi171.stdout:4/657: chown d2/d3/d5/d3b/f55 201 1 2022-01-31T19:44:31.127 INFO:tasks.workunit.client.1.smithi171.stdout:4/658: write d2/d3/d5/d41/d6b/fbd [200970,57816] 0 2022-01-31T19:44:31.127 INFO:tasks.workunit.client.1.smithi171.stdout:4/659: read - d2/d3/d5/d41/fbf zero size 2022-01-31T19:44:31.128 INFO:tasks.workunit.client.1.smithi171.stdout:4/660: creat d2/d3/d5/d41/dcf/d12/d15/dc4/fe2 x:0 0 0 2022-01-31T19:44:31.128 INFO:tasks.workunit.client.1.smithi171.stdout:4/661: chown d2/d3/d5/f38 149781 1 2022-01-31T19:44:31.128 INFO:tasks.workunit.client.1.smithi171.stdout:4/662: truncate d2/d3/d5/d91/fbe 1463260 0 2022-01-31T19:44:31.134 INFO:tasks.workunit.client.1.smithi171.stdout:2/860: mknod d7/dd/dea/c108 0 2022-01-31T19:44:31.137 INFO:tasks.workunit.client.1.smithi171.stdout:3/831: rename d4/dd/d27/d28/d35/fa9 to d4/d21/d3a/d59/dc1/f10f 0 2022-01-31T19:44:31.140 INFO:tasks.workunit.client.1.smithi171.stdout:4/663: creat d2/d3/de1/dd4/fe3 x:0 0 0 2022-01-31T19:44:31.141 INFO:tasks.workunit.client.1.smithi171.stdout:5/672: rmdir d5/d90/db7/dd/d66 39 2022-01-31T19:44:31.143 INFO:tasks.workunit.client.1.smithi171.stdout:3/832: unlink d4/dd/d27/d4e/d62/cc8 0 2022-01-31T19:44:31.144 INFO:tasks.workunit.client.1.smithi171.stdout:3/833: fsync d4/d21/de5/d92/dab/d60/fbd 0 2022-01-31T19:44:31.144 INFO:tasks.workunit.client.1.smithi171.stdout:3/834: fsync d4/dd/d39/d51/fb4 0 2022-01-31T19:44:31.144 INFO:tasks.workunit.client.1.smithi171.stdout:4/664: mkdir d2/d3/d5/d91/de4 0 2022-01-31T19:44:31.145 INFO:tasks.workunit.client.1.smithi171.stdout:3/835: dread d4/dd/d87/f9f [0,4194304] 0 2022-01-31T19:44:31.146 INFO:tasks.workunit.client.1.smithi171.stdout:9/748: sync 2022-01-31T19:44:31.147 INFO:tasks.workunit.client.1.smithi171.stdout:9/749: dread dd/f6a [0,4194304] 0 2022-01-31T19:44:31.148 INFO:tasks.workunit.client.1.smithi171.stdout:9/750: mkdir dd/df/d15/d1a/d64/dfc 0 2022-01-31T19:44:31.149 INFO:tasks.workunit.client.1.smithi171.stdout:2/861: dwrite d7/dd/d5e/d84/f8b [0,4194304] 0 2022-01-31T19:44:31.151 INFO:tasks.workunit.client.1.smithi171.stdout:3/836: read d4/dd/d87/d8e/f42 [3513728,94389] 0 2022-01-31T19:44:31.151 INFO:tasks.workunit.client.1.smithi171.stdout:3/837: stat d4/dd/d27/d28/d35/d55/dbf/fc5 0 2022-01-31T19:44:31.152 INFO:tasks.workunit.client.1.smithi171.stdout:3/838: readlink d4/dd/d27/d28/d35/d55/dbf/ld8 0 2022-01-31T19:44:31.153 INFO:tasks.workunit.client.1.smithi171.stdout:3/839: write d4/dd/d87/d8e/f88 [109814,55517] 0 2022-01-31T19:44:31.153 INFO:tasks.workunit.client.1.smithi171.stdout:3/840: truncate d4/d21/de5/d92/dab/f1b 8923926 0 2022-01-31T19:44:31.153 INFO:tasks.workunit.client.1.smithi171.stdout:3/841: dread - d4/dd/ff0 zero size 2022-01-31T19:44:31.154 INFO:tasks.workunit.client.1.smithi171.stdout:3/842: fsync d4/dd/ff0 0 2022-01-31T19:44:31.154 INFO:tasks.workunit.client.1.smithi171.stdout:3/843: write d4/dd/d73/fcc [991137,28101] 0 2022-01-31T19:44:31.154 INFO:tasks.workunit.client.1.smithi171.stdout:3/844: chown d4/d21/de5/dde/ce7 505 1 2022-01-31T19:44:31.155 INFO:tasks.workunit.client.1.smithi171.stdout:2/862: link d7/d7c/d83/d85/fd6 d7/d9/f109 0 2022-01-31T19:44:31.155 INFO:tasks.workunit.client.1.smithi171.stdout:2/863: mknod d7/d55/df3/c10a 0 2022-01-31T19:44:31.158 INFO:tasks.workunit.client.1.smithi171.stdout:2/864: symlink d7/d9/d1a/d45/da4/l10b 0 2022-01-31T19:44:31.161 INFO:tasks.workunit.client.1.smithi171.stdout:2/865: write d7/dd/d5e/d84/f8b [2923214,98231] 0 2022-01-31T19:44:31.162 INFO:tasks.workunit.client.1.smithi171.stdout:2/866: getdents d7/d26/de5 0 2022-01-31T19:44:31.163 INFO:tasks.workunit.client.1.smithi171.stdout:2/867: readlink d7/d9/d1a/l72 0 2022-01-31T19:44:31.163 INFO:tasks.workunit.client.1.smithi171.stdout:2/868: creat d7/d9/d1a/d45/f10c x:0 0 0 2022-01-31T19:44:31.163 INFO:tasks.workunit.client.1.smithi171.stdout:2/869: fdatasync d7/d26/d63/d71/fee 0 2022-01-31T19:44:31.164 INFO:tasks.workunit.client.1.smithi171.stdout:2/870: chown d7/d26/d63/f64 1789298 1 2022-01-31T19:44:31.179 INFO:tasks.workunit.client.1.smithi171.stdout:5/673: dwrite d5/d90/db7/d15/d23/d58/f91 [0,4194304] 0 2022-01-31T19:44:31.180 INFO:tasks.workunit.client.1.smithi171.stdout:5/674: dread - d5/d90/db7/d15/d5d/d6f/fdd zero size 2022-01-31T19:44:31.180 INFO:tasks.workunit.client.1.smithi171.stdout:9/751: dwrite dd/df/d15/d53/fc4 [0,4194304] 0 2022-01-31T19:44:31.180 INFO:tasks.workunit.client.1.smithi171.stdout:9/752: creat dd/df/d15/d38/d4d/ffd x:0 0 0 2022-01-31T19:44:31.181 INFO:tasks.workunit.client.1.smithi171.stdout:3/845: dwrite d4/dd/d27/d28/d35/ff7 [0,4194304] 0 2022-01-31T19:44:31.182 INFO:tasks.workunit.client.1.smithi171.stdout:8/632: sync 2022-01-31T19:44:31.182 INFO:tasks.workunit.client.1.smithi171.stdout:7/562: sync 2022-01-31T19:44:31.182 INFO:tasks.workunit.client.1.smithi171.stdout:8/633: chown df/d18/d19/d82/f97 686764443 1 2022-01-31T19:44:31.183 INFO:tasks.workunit.client.1.smithi171.stdout:7/563: chown d1/d1e/d2a/d29/d31/d30/f4d 136 1 2022-01-31T19:44:31.183 INFO:tasks.workunit.client.1.smithi171.stdout:7/564: chown d1/f36 67 1 2022-01-31T19:44:31.183 INFO:tasks.workunit.client.1.smithi171.stdout:5/675: creat d5/d90/db7/dd/d66/fde x:0 0 0 2022-01-31T19:44:31.187 INFO:tasks.workunit.client.1.smithi171.stdout:9/753: mkdir dd/df/d15/d1a/d84/dfe 0 2022-01-31T19:44:31.188 INFO:tasks.workunit.client.1.smithi171.stdout:2/871: dwrite d7/d9/d1a/d45/da4/fce [0,4194304] 0 2022-01-31T19:44:31.188 INFO:tasks.workunit.client.1.smithi171.stdout:3/846: creat d4/dd/d87/d8e/f110 x:0 0 0 2022-01-31T19:44:31.191 INFO:tasks.workunit.client.1.smithi171.stdout:8/634: rename df/l59 to df/d18/d79/d7a/ld4 0 2022-01-31T19:44:31.192 INFO:tasks.workunit.client.1.smithi171.stdout:9/754: symlink dd/df/d3f/d73/d96/d9f/de6/lff 0 2022-01-31T19:44:31.196 INFO:tasks.workunit.client.1.smithi171.stdout:2/872: mknod d7/d26/d3d/d40/c10d 0 2022-01-31T19:44:31.199 INFO:tasks.workunit.client.1.smithi171.stdout:3/847: mknod d4/dd/d27/d4e/c111 0 2022-01-31T19:44:31.200 INFO:tasks.workunit.client.1.smithi171.stdout:8/635: rename df/d27/f5b to df/d27/fd5 0 2022-01-31T19:44:31.200 INFO:tasks.workunit.client.1.smithi171.stdout:2/873: symlink d7/d9/d1a/d45/l10e 0 2022-01-31T19:44:31.203 INFO:tasks.workunit.client.1.smithi171.stdout:7/565: rename d1/d1e/d2a/d29/d77/laf to d1/d1e/d2a/d29/d31/lba 0 2022-01-31T19:44:31.205 INFO:tasks.workunit.client.1.smithi171.stdout:8/636: rmdir df/d18/d79/da2/da8 0 2022-01-31T19:44:31.205 INFO:tasks.workunit.client.1.smithi171.stdout:8/637: chown df/d18/d19/d82/dd3 1 1 2022-01-31T19:44:31.206 INFO:tasks.workunit.client.1.smithi171.stdout:2/874: creat d7/d26/d63/d9f/f10f x:0 0 0 2022-01-31T19:44:31.206 INFO:tasks.workunit.client.1.smithi171.stdout:2/875: readlink d7/d55/la9 0 2022-01-31T19:44:31.208 INFO:tasks.workunit.client.1.smithi171.stdout:5/676: rename d5/d90/db7/d15/d5d/d6a/lc2 to d5/ldf 0 2022-01-31T19:44:31.209 INFO:tasks.workunit.client.1.smithi171.stdout:7/566: creat d1/d1e/d2a/d29/d31/d30/d3c/fbb x:0 0 0 2022-01-31T19:44:31.211 INFO:tasks.workunit.client.1.smithi171.stdout:8/638: unlink df/d5d/d6c/d87/fa0 0 2022-01-31T19:44:31.212 INFO:tasks.workunit.client.1.smithi171.stdout:8/639: fdatasync df/f4f 0 2022-01-31T19:44:31.213 INFO:tasks.workunit.client.1.smithi171.stdout:5/677: truncate d5/d90/db7/fc1 887073 0 2022-01-31T19:44:31.214 INFO:tasks.workunit.client.1.smithi171.stdout:5/678: fdatasync d5/d90/db7/dd/d66/f7d 0 2022-01-31T19:44:31.214 INFO:tasks.workunit.client.1.smithi171.stdout:7/567: write d1/d1e/d2a/d29/d31/d30/f73 [7876554,52909] 0 2022-01-31T19:44:31.214 INFO:tasks.workunit.client.1.smithi171.stdout:9/755: dwrite dd/df/d15/d1a/d64/d6e/fd2 [0,4194304] 0 2022-01-31T19:44:31.215 INFO:tasks.workunit.client.1.smithi171.stdout:9/756: chown dd/l5d 86279 1 2022-01-31T19:44:31.215 INFO:tasks.workunit.client.1.smithi171.stdout:5/679: symlink d5/d90/dd3/le0 0 2022-01-31T19:44:31.216 INFO:tasks.workunit.client.1.smithi171.stdout:5/680: creat d5/d90/db7/dd/d66/fe1 x:0 0 0 2022-01-31T19:44:31.216 INFO:tasks.workunit.client.1.smithi171.stdout:7/568: mknod d1/d1e/cbc 0 2022-01-31T19:44:31.217 INFO:tasks.workunit.client.1.smithi171.stdout:3/848: dwrite d4/d21/de5/d92/dab/f1b [0,4194304] 0 2022-01-31T19:44:31.218 INFO:tasks.workunit.client.1.smithi171.stdout:9/757: symlink dd/df/d15/d1a/d64/d6e/ddd/l100 0 2022-01-31T19:44:31.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/569: unlink d1/d26/d57/fa5 0 2022-01-31T19:44:31.231 INFO:tasks.workunit.client.1.smithi171.stdout:8/640: dread f2 [0,4194304] 0 2022-01-31T19:44:31.232 INFO:tasks.workunit.client.1.smithi171.stdout:9/758: rmdir dd/df/d3f/d73/d96/d9f/de6/db5 39 2022-01-31T19:44:31.236 INFO:tasks.workunit.client.1.smithi171.stdout:9/759: link dd/f11 dd/df/d15/de4/f101 0 2022-01-31T19:44:31.237 INFO:tasks.workunit.client.1.smithi171.stdout:9/760: rmdir dd/df/d15/d1a/d64/d6e 39 2022-01-31T19:44:31.240 INFO:tasks.workunit.client.1.smithi171.stdout:9/761: getdents dd/df/d15 0 2022-01-31T19:44:31.240 INFO:tasks.workunit.client.1.smithi171.stdout:9/762: truncate dd/df/d15/d1a/d64/f67 94973 0 2022-01-31T19:44:31.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/763: dread - dd/f74 zero size 2022-01-31T19:44:31.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/764: chown dd/df/d15/d38/d41/l47 11206 1 2022-01-31T19:44:31.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/765: chown dd/f37 189675788 1 2022-01-31T19:44:31.241 INFO:tasks.workunit.client.1.smithi171.stdout:9/766: dread - dd/df/d15/d38/d4d/ffd zero size 2022-01-31T19:44:31.242 INFO:tasks.workunit.client.1.smithi171.stdout:9/767: creat dd/df/d15/d1a/d64/f102 x:0 0 0 2022-01-31T19:44:31.243 INFO:tasks.workunit.client.1.smithi171.stdout:8/641: dread df/d27/d85/f8e [0,4194304] 0 2022-01-31T19:44:31.243 INFO:tasks.workunit.client.1.smithi171.stdout:8/642: write df/d18/d19/d21/d67/fcb [158853,125055] 0 2022-01-31T19:44:31.243 INFO:tasks.workunit.client.1.smithi171.stdout:9/768: unlink dd/df/d15/de4/ff7 0 2022-01-31T19:44:31.244 INFO:tasks.workunit.client.1.smithi171.stdout:3/849: dwrite d4/d21/f69 [0,4194304] 0 2022-01-31T19:44:31.244 INFO:tasks.workunit.client.1.smithi171.stdout:8/643: mkdir df/d18/d19/d46/dd6 0 2022-01-31T19:44:31.245 INFO:tasks.workunit.client.1.smithi171.stdout:8/644: fdatasync df/d18/d19/d82/f97 0 2022-01-31T19:44:31.245 INFO:tasks.workunit.client.1.smithi171.stdout:9/769: symlink dd/df/d15/l103 0 2022-01-31T19:44:31.245 INFO:tasks.workunit.client.1.smithi171.stdout:9/770: fsync dd/df/d15/d38/d4d/ffd 0 2022-01-31T19:44:31.247 INFO:tasks.workunit.client.1.smithi171.stdout:8/645: truncate df/d27/d85/f8f 3936644 0 2022-01-31T19:44:31.247 INFO:tasks.workunit.client.1.smithi171.stdout:7/570: dwrite d1/d1b/fc [4194304,4194304] 0 2022-01-31T19:44:31.249 INFO:tasks.workunit.client.1.smithi171.stdout:5/681: dwrite d5/d90/db7/d15/f88 [0,4194304] 0 2022-01-31T19:44:31.249 INFO:tasks.workunit.client.1.smithi171.stdout:5/682: dread - d5/d90/db7/dd/d66/fc6 zero size 2022-01-31T19:44:31.249 INFO:tasks.workunit.client.1.smithi171.stdout:5/683: chown d5/d90/db7/d15/d23/d67/f4d 118 1 2022-01-31T19:44:31.250 INFO:tasks.workunit.client.1.smithi171.stdout:9/771: rename dd/df/d15/d1a/d64/d6e/da5/lf5 to dd/df/d15/d38/d4d/d80/dfa/l104 0 2022-01-31T19:44:31.250 INFO:tasks.workunit.client.1.smithi171.stdout:3/850: mkdir d4/d112 0 2022-01-31T19:44:31.252 INFO:tasks.workunit.client.1.smithi171.stdout:8/646: mkdir df/d27/d7b/dca/dd7 0 2022-01-31T19:44:31.255 INFO:tasks.workunit.client.1.smithi171.stdout:9/772: write fc [849257,64926] 0 2022-01-31T19:44:31.267 INFO:tasks.workunit.client.1.smithi171.stdout:8/647: rename df/d27/d7b/dca/dd7 to df/d2a/d30/dd8 0 2022-01-31T19:44:31.270 INFO:tasks.workunit.client.1.smithi171.stdout:3/851: dwrite d4/d21/d3a/db5/fed [0,4194304] 0 2022-01-31T19:44:31.270 INFO:tasks.workunit.client.1.smithi171.stdout:3/852: creat d4/d21/d3a/d59/dc1/f113 x:0 0 0 2022-01-31T19:44:31.270 INFO:tasks.workunit.client.1.smithi171.stdout:3/853: chown d4/dd/d27/d28/ce2 861507909 1 2022-01-31T19:44:31.271 INFO:tasks.workunit.client.1.smithi171.stdout:5/684: dwrite d5/d90/db7/d15/d23/f87 [0,4194304] 0 2022-01-31T19:44:31.273 INFO:tasks.workunit.client.1.smithi171.stdout:9/773: creat dd/df/d15/d38/d4d/d80/dfa/f105 x:0 0 0 2022-01-31T19:44:31.274 INFO:tasks.workunit.client.1.smithi171.stdout:6/745: sync 2022-01-31T19:44:31.274 INFO:tasks.workunit.client.1.smithi171.stdout:0/772: sync 2022-01-31T19:44:31.274 INFO:tasks.workunit.client.1.smithi171.stdout:0/773: chown d9/d24/d92/d9d/d11a 2197569 1 2022-01-31T19:44:31.274 INFO:tasks.workunit.client.1.smithi171.stdout:0/774: fdatasync d9/d24/d4c/fa9 0 2022-01-31T19:44:31.275 INFO:tasks.workunit.client.1.smithi171.stdout:0/775: write d9/db/d59/d71/fb5 [557565,128291] 0 2022-01-31T19:44:31.279 INFO:tasks.workunit.client.1.smithi171.stdout:8/648: unlink df/d5d/d6c/f76 0 2022-01-31T19:44:31.281 INFO:tasks.workunit.client.1.smithi171.stdout:7/571: dread d1/f87 [0,4194304] 0 2022-01-31T19:44:31.281 INFO:tasks.workunit.client.1.smithi171.stdout:7/572: fsync d1/f5 0 2022-01-31T19:44:31.282 INFO:tasks.workunit.client.1.smithi171.stdout:5/685: mkdir d5/d90/db7/d15/d80/d8d/de2 0 2022-01-31T19:44:31.282 INFO:tasks.workunit.client.1.smithi171.stdout:3/854: dread d4/dd/d27/d28/d35/ff7 [0,4194304] 0 2022-01-31T19:44:31.283 INFO:tasks.workunit.client.1.smithi171.stdout:9/774: read dd/fc2 [3712901,98804] 0 2022-01-31T19:44:31.283 INFO:tasks.workunit.client.1.smithi171.stdout:9/775: chown dd/df/d3f/fd1 8 1 2022-01-31T19:44:31.283 INFO:tasks.workunit.client.1.smithi171.stdout:9/776: fsync dd/df/d15/f36 0 2022-01-31T19:44:31.283 INFO:tasks.workunit.client.1.smithi171.stdout:9/777: write dd/df/d15/d38/d4d/f63 [825264,34229] 0 2022-01-31T19:44:31.284 INFO:tasks.workunit.client.1.smithi171.stdout:0/776: link d9/d24/d92/d9d/dc6/fd9 d9/db/f121 0 2022-01-31T19:44:31.284 INFO:tasks.workunit.client.1.smithi171.stdout:8/649: mknod df/d18/d19/cd9 0 2022-01-31T19:44:31.286 INFO:tasks.workunit.client.1.smithi171.stdout:5/686: symlink d5/d90/db7/d15/d23/d58/d9e/dd7/le3 0 2022-01-31T19:44:31.287 INFO:tasks.workunit.client.1.smithi171.stdout:3/855: rmdir d4/d21/d3a/d59 39 2022-01-31T19:44:31.288 INFO:tasks.workunit.client.1.smithi171.stdout:9/778: truncate dd/df/d3f/f4c 1029588 0 2022-01-31T19:44:31.289 INFO:tasks.workunit.client.1.smithi171.stdout:8/650: mknod df/d5d/d6c/cda 0 2022-01-31T19:44:31.292 INFO:tasks.workunit.client.1.smithi171.stdout:7/573: rename d1/d1e/d2a/d29/d31/c74 to d1/d26/d47/cbd 0 2022-01-31T19:44:31.293 INFO:tasks.workunit.client.1.smithi171.stdout:3/856: write d4/dd/d27/d28/db6/fe3 [2229637,7323] 0 2022-01-31T19:44:31.294 INFO:tasks.workunit.client.1.smithi171.stdout:8/651: mkdir df/d18/d19/d21/d67/ddb 0 2022-01-31T19:44:31.296 INFO:tasks.workunit.client.1.smithi171.stdout:7/574: symlink d1/d26/lbe 0 2022-01-31T19:44:31.297 INFO:tasks.workunit.client.1.smithi171.stdout:6/746: dwrite dc/d19/dae/fdf [0,4194304] 0 2022-01-31T19:44:31.298 INFO:tasks.workunit.client.1.smithi171.stdout:3/857: truncate d4/d21/d3a/d90/f94 4240316 0 2022-01-31T19:44:31.300 INFO:tasks.workunit.client.1.smithi171.stdout:7/575: creat d1/d1e/d2a/d29/d31/d30/d6b/fbf x:0 0 0 2022-01-31T19:44:31.300 INFO:tasks.workunit.client.1.smithi171.stdout:7/576: dread - d1/d1e/d2a/d29/d31/d30/d3c/d5f/fa4 zero size 2022-01-31T19:44:31.301 INFO:tasks.workunit.client.1.smithi171.stdout:6/747: mknod dc/d18/cfb 0 2022-01-31T19:44:31.302 INFO:tasks.workunit.client.1.smithi171.stdout:6/748: write dc/fe0 [1339042,24898] 0 2022-01-31T19:44:31.303 INFO:tasks.workunit.client.1.smithi171.stdout:6/749: write dc/d19/d29/f34 [515174,115696] 0 2022-01-31T19:44:31.303 INFO:tasks.workunit.client.1.smithi171.stdout:3/858: creat d4/d21/d3a/db5/dbe/dc3/f114 x:0 0 0 2022-01-31T19:44:31.303 INFO:tasks.workunit.client.1.smithi171.stdout:3/859: chown d4/d21/d3a/f4f 1 1 2022-01-31T19:44:31.305 INFO:tasks.workunit.client.1.smithi171.stdout:6/750: link dc/d19/d29/l74 dc/d19/d29/daf/ddb/lfc 0 2022-01-31T19:44:31.305 INFO:tasks.workunit.client.1.smithi171.stdout:8/652: dread df/d18/d19/d36/d70/d7e/f8a [0,4194304] 0 2022-01-31T19:44:31.306 INFO:tasks.workunit.client.1.smithi171.stdout:8/653: readlink df/d5d/d6c/lb0 0 2022-01-31T19:44:31.306 INFO:tasks.workunit.client.1.smithi171.stdout:5/687: dwrite d5/d90/db7/d15/d23/d3d/fb2 [0,4194304] 0 2022-01-31T19:44:31.306 INFO:tasks.workunit.client.1.smithi171.stdout:3/860: mkdir d4/d21/d3a/db5/dbe/dd1/d115 0 2022-01-31T19:44:31.316 INFO:tasks.workunit.client.1.smithi171.stdout:6/751: creat dc/d19/d29/ffd x:0 0 0 2022-01-31T19:44:31.319 INFO:tasks.workunit.client.1.smithi171.stdout:6/752: dread dc/d19/d29/d27/d37/d3b/d67/feb [0,4194304] 0 2022-01-31T19:44:31.319 INFO:tasks.workunit.client.1.smithi171.stdout:6/753: readlink dc/d19/dae/lb0 0 2022-01-31T19:44:31.319 INFO:tasks.workunit.client.1.smithi171.stdout:5/688: creat d5/d90/db7/d15/d23/d58/fe4 x:0 0 0 2022-01-31T19:44:31.319 INFO:tasks.workunit.client.1.smithi171.stdout:5/689: fsync d5/d90/db7/dd/d66/fde 0 2022-01-31T19:44:31.320 INFO:tasks.workunit.client.1.smithi171.stdout:8/654: dwrite df/d18/d19/d36/f3e [0,4194304] 0 2022-01-31T19:44:31.320 INFO:tasks.workunit.client.1.smithi171.stdout:8/655: chown df/d18/d19/d21/dcd 0 1 2022-01-31T19:44:31.321 INFO:tasks.workunit.client.1.smithi171.stdout:3/861: truncate d4/dd/f45 3304469 0 2022-01-31T19:44:31.322 INFO:tasks.workunit.client.1.smithi171.stdout:6/754: unlink dc/d18/d86/ce5 0 2022-01-31T19:44:31.324 INFO:tasks.workunit.client.1.smithi171.stdout:5/690: mknod d5/d90/db7/d15/d23/d67/dd8/ce5 0 2022-01-31T19:44:31.336 INFO:tasks.workunit.client.1.smithi171.stdout:6/755: rename dc/d1e/d3f/lc9 to dc/d19/d29/d27/d37/d68/lfe 0 2022-01-31T19:44:31.337 INFO:tasks.workunit.client.1.smithi171.stdout:6/756: fsync dc/d19/d30/d3c/d80/ff9 0 2022-01-31T19:44:31.337 INFO:tasks.workunit.client.1.smithi171.stdout:5/691: mknod d5/d90/db7/d15/d80/d8d/ce6 0 2022-01-31T19:44:31.337 INFO:tasks.workunit.client.1.smithi171.stdout:3/862: rmdir d4/d21/d3a/db5/dbe 39 2022-01-31T19:44:31.338 INFO:tasks.workunit.client.1.smithi171.stdout:3/863: read - d4/dd/d27/d28/d35/d55/dbf/fec zero size 2022-01-31T19:44:31.339 INFO:tasks.workunit.client.1.smithi171.stdout:5/692: link d5/d90/db7/l7c d5/d90/db7/d15/d23/d58/le7 0 2022-01-31T19:44:31.341 INFO:tasks.workunit.client.1.smithi171.stdout:6/757: dread dc/d8b/fc5 [0,4194304] 0 2022-01-31T19:44:31.343 INFO:tasks.workunit.client.1.smithi171.stdout:3/864: rename d4/dd/d39/c98 to d4/d21/de5/d92/dab/d60/df4/c116 0 2022-01-31T19:44:31.343 INFO:tasks.workunit.client.1.smithi171.stdout:3/865: read - d4/dd/d39/fa1 zero size 2022-01-31T19:44:31.344 INFO:tasks.workunit.client.1.smithi171.stdout:3/866: write d4/dd/d87/fa6 [10749,105693] 0 2022-01-31T19:44:31.344 INFO:tasks.workunit.client.1.smithi171.stdout:3/867: write d4/d21/de5/d92/dab/d52/f8d [807864,28875] 0 2022-01-31T19:44:31.345 INFO:tasks.workunit.client.1.smithi171.stdout:1/768: sync 2022-01-31T19:44:31.345 INFO:tasks.workunit.client.1.smithi171.stdout:1/769: stat dd/f42 0 2022-01-31T19:44:31.346 INFO:tasks.workunit.client.1.smithi171.stdout:5/693: write d5/d90/db7/d15/d23/f41 [346329,47131] 0 2022-01-31T19:44:31.348 INFO:tasks.workunit.client.1.smithi171.stdout:3/868: creat d4/dd/d27/d4e/f117 x:0 0 0 2022-01-31T19:44:31.349 INFO:tasks.workunit.client.1.smithi171.stdout:8/656: dwrite df/d5d/d6c/fb3 [0,4194304] 0 2022-01-31T19:44:31.350 INFO:tasks.workunit.client.1.smithi171.stdout:1/770: truncate dd/d23/d2e/d5e/f8a 4109505 0 2022-01-31T19:44:31.351 INFO:tasks.workunit.client.1.smithi171.stdout:1/771: truncate dd/d23/d34/d4b/fd7 519701 0 2022-01-31T19:44:31.353 INFO:tasks.workunit.client.1.smithi171.stdout:8/657: creat df/fdc x:0 0 0 2022-01-31T19:44:31.355 INFO:tasks.workunit.client.1.smithi171.stdout:8/658: mkdir df/d18/d79/da2/ddd 0 2022-01-31T19:44:31.358 INFO:tasks.workunit.client.1.smithi171.stdout:8/659: creat df/d18/d19/d82/dd3/fde x:0 0 0 2022-01-31T19:44:31.385 INFO:tasks.workunit.client.1.smithi171.stdout:5/694: dwrite d5/d90/db7/d15/d23/f25 [0,4194304] 0 2022-01-31T19:44:31.385 INFO:tasks.workunit.client.1.smithi171.stdout:5/695: write d5/d90/db7/fb3 [4677093,128926] 0 2022-01-31T19:44:31.386 INFO:tasks.workunit.client.1.smithi171.stdout:5/696: creat d5/d90/fe8 x:0 0 0 2022-01-31T19:44:31.389 INFO:tasks.workunit.client.1.smithi171.stdout:5/697: symlink d5/d90/db7/d15/d23/d6b/dae/le9 0 2022-01-31T19:44:31.389 INFO:tasks.workunit.client.1.smithi171.stdout:5/698: truncate d5/fb5 84133 0 2022-01-31T19:44:31.390 INFO:tasks.workunit.client.1.smithi171.stdout:5/699: mknod d5/d90/dd3/cea 0 2022-01-31T19:44:31.399 INFO:tasks.workunit.client.1.smithi171.stdout:3/869: dwrite d4/dd/d27/d28/db6/fe3 [0,4194304] 0 2022-01-31T19:44:31.400 INFO:tasks.workunit.client.1.smithi171.stdout:1/772: dwrite dd/d23/d2e/d5e/f40 [0,4194304] 0 2022-01-31T19:44:31.400 INFO:tasks.workunit.client.1.smithi171.stdout:1/773: dread - dd/df/d19/f96 zero size 2022-01-31T19:44:31.401 INFO:tasks.workunit.client.1.smithi171.stdout:8/660: dwrite df/d18/d79/f9d [0,4194304] 0 2022-01-31T19:44:31.401 INFO:tasks.workunit.client.1.smithi171.stdout:8/661: truncate df/d2a/f49 4671124 0 2022-01-31T19:44:31.403 INFO:tasks.workunit.client.1.smithi171.stdout:1/774: read dd/d23/f57 [174134,20497] 0 2022-01-31T19:44:31.404 INFO:tasks.workunit.client.1.smithi171.stdout:8/662: rmdir df/d18/d19/d36/d70 39 2022-01-31T19:44:31.417 INFO:tasks.workunit.client.1.smithi171.stdout:8/663: dread df/d1a/f58 [0,4194304] 0 2022-01-31T19:44:31.420 INFO:tasks.workunit.client.1.smithi171.stdout:8/664: mknod df/d18/d19/d46/dd6/cdf 0 2022-01-31T19:44:31.421 INFO:tasks.workunit.client.1.smithi171.stdout:8/665: getdents df/d18/d19/d21/d88 0 2022-01-31T19:44:31.421 INFO:tasks.workunit.client.1.smithi171.stdout:8/666: mkdir df/d1a/dae/de0 0 2022-01-31T19:44:31.425 INFO:tasks.workunit.client.1.smithi171.stdout:3/870: dwrite d4/dd/f45 [4194304,4194304] 0 2022-01-31T19:44:31.426 INFO:tasks.workunit.client.1.smithi171.stdout:1/775: dwrite dd/df/d19/d1b/d1e/d64/db9/fd2 [0,4194304] 0 2022-01-31T19:44:31.429 INFO:tasks.workunit.client.1.smithi171.stdout:3/871: creat d4/d21/de5/d92/dab/d52/f118 x:0 0 0 2022-01-31T19:44:31.432 INFO:tasks.workunit.client.1.smithi171.stdout:1/776: rmdir dd/d23/d2e/d5e/d81/d98/dad/db6 0 2022-01-31T19:44:31.434 INFO:tasks.workunit.client.1.smithi171.stdout:3/872: mknod d4/dd/d27/d4e/d62/c119 0 2022-01-31T19:44:31.435 INFO:tasks.workunit.client.1.smithi171.stdout:1/777: dread dd/da2/fb2 [0,4194304] 0 2022-01-31T19:44:31.435 INFO:tasks.workunit.client.1.smithi171.stdout:1/778: fsync dd/f7a 0 2022-01-31T19:44:31.437 INFO:tasks.workunit.client.1.smithi171.stdout:1/779: rename dd/d23/d74/dcb/ddb/fcf to dd/d23/d2e/fe6 0 2022-01-31T19:44:31.459 INFO:tasks.workunit.client.1.smithi171.stdout:3/873: dwrite d4/dd/d27/d28/db6/dc0/fe0 [0,4194304] 0 2022-01-31T19:44:31.459 INFO:tasks.workunit.client.1.smithi171.stdout:3/874: dread - d4/d21/de5/d92/f78 zero size 2022-01-31T19:44:31.460 INFO:tasks.workunit.client.1.smithi171.stdout:8/667: dwrite df/d2a/f40 [4194304,4194304] 0 2022-01-31T19:44:31.460 INFO:tasks.workunit.client.1.smithi171.stdout:8/668: readlink df/d1a/l75 0 2022-01-31T19:44:31.464 INFO:tasks.workunit.client.1.smithi171.stdout:3/875: creat d4/d21/de5/d92/dab/d60/df4/f11a x:0 0 0 2022-01-31T19:44:31.465 INFO:tasks.workunit.client.1.smithi171.stdout:8/669: mkdir df/d5d/d6c/d87/de1 0 2022-01-31T19:44:31.467 INFO:tasks.workunit.client.1.smithi171.stdout:3/876: mkdir d4/dd/d27/d4e/d62/d11b 0 2022-01-31T19:44:31.468 INFO:tasks.workunit.client.1.smithi171.stdout:8/670: mknod df/d18/da4/ce2 0 2022-01-31T19:44:31.469 INFO:tasks.workunit.client.1.smithi171.stdout:1/780: dwrite dd/d23/d2e/d5e/d81/fa8 [0,4194304] 0 2022-01-31T19:44:31.469 INFO:tasks.workunit.client.1.smithi171.stdout:3/877: mknod d4/d112/c11c 0 2022-01-31T19:44:31.470 INFO:tasks.workunit.client.1.smithi171.stdout:8/671: creat df/d18/d19/d46/dbc/fe3 x:0 0 0 2022-01-31T19:44:31.475 INFO:tasks.workunit.client.1.smithi171.stdout:3/878: rename d4/d21/de5/d92/dab/d52/f9b to d4/d21/de5/d92/dab/dd4/f11d 0 2022-01-31T19:44:31.475 INFO:tasks.workunit.client.1.smithi171.stdout:3/879: chown d4/dd/d39/d51/l107 0 1 2022-01-31T19:44:31.476 INFO:tasks.workunit.client.1.smithi171.stdout:3/880: getdents d4/dd/d4b 0 2022-01-31T19:44:31.476 INFO:tasks.workunit.client.1.smithi171.stdout:3/881: write d4/d21/de5/d92/f78 [921681,27945] 0 2022-01-31T19:44:31.476 INFO:tasks.workunit.client.1.smithi171.stdout:3/882: unlink d4/dd/d87/d8e/faf 0 2022-01-31T19:44:31.477 INFO:tasks.workunit.client.1.smithi171.stdout:4/665: sync 2022-01-31T19:44:31.477 INFO:tasks.workunit.client.1.smithi171.stdout:4/666: fsync d2/d3/d5/d41/dcf/dc7/fdb 0 2022-01-31T19:44:31.479 INFO:tasks.workunit.client.1.smithi171.stdout:3/883: truncate d4/d21/d3a/db5/fed 1101002 0 2022-01-31T19:44:31.480 INFO:tasks.workunit.client.1.smithi171.stdout:4/667: mknod d2/d3/d5/d41/dcf/d12/d15/d57/dad/ce5 0 2022-01-31T19:44:31.483 INFO:tasks.workunit.client.1.smithi171.stdout:1/781: truncate dd/d23/d2e/d5e/d81/fa8 3929823 0 2022-01-31T19:44:31.485 INFO:tasks.workunit.client.1.smithi171.stdout:3/884: symlink d4/l11e 0 2022-01-31T19:44:31.485 INFO:tasks.workunit.client.1.smithi171.stdout:3/885: chown d4/dd/d27/d28/d35/d55/ce6 228968 1 2022-01-31T19:44:31.486 INFO:tasks.workunit.client.1.smithi171.stdout:3/886: rename d4/dd/d27/d28/c2f to d4/d21/d3a/db5/dbe/dc3/dcb/c11f 0 2022-01-31T19:44:31.486 INFO:tasks.workunit.client.1.smithi171.stdout:3/887: truncate d4/d21/d3a/db5/dbe/ff5 159239 0 2022-01-31T19:44:31.487 INFO:tasks.workunit.client.1.smithi171.stdout:3/888: read d4/dd/d39/d51/f9e [2811025,121180] 0 2022-01-31T19:44:31.488 INFO:tasks.workunit.client.1.smithi171.stdout:3/889: creat d4/d21/de5/d92/dab/d60/f120 x:0 0 0 2022-01-31T19:44:31.494 INFO:tasks.workunit.client.1.smithi171.stdout:4/668: dread d2/d3/d5/d41/dcf/d12/f8d [0,4194304] 0 2022-01-31T19:44:31.495 INFO:tasks.workunit.client.1.smithi171.stdout:1/782: dread dd/df/d19/d1b/f3e [0,4194304] 0 2022-01-31T19:44:31.497 INFO:tasks.workunit.client.1.smithi171.stdout:4/669: write d2/d3/d5/d41/dcf/f83 [2515210,108535] 0 2022-01-31T19:44:31.501 INFO:tasks.workunit.client.1.smithi171.stdout:1/783: dread dd/d23/f82 [0,4194304] 0 2022-01-31T19:44:31.501 INFO:tasks.workunit.client.1.smithi171.stdout:4/670: symlink d2/d3/d5/dc/d4b/le6 0 2022-01-31T19:44:31.521 INFO:tasks.workunit.client.1.smithi171.stdout:1/784: dwrite dd/df/f38 [0,4194304] 0 2022-01-31T19:44:31.524 INFO:tasks.workunit.client.1.smithi171.stdout:1/785: creat dd/d23/d2e/d5e/dd8/fe7 x:0 0 0 2022-01-31T19:44:31.524 INFO:tasks.workunit.client.1.smithi171.stdout:1/786: readlink dd/d23/l4f 0 2022-01-31T19:44:31.527 INFO:tasks.workunit.client.1.smithi171.stdout:3/890: dwrite d4/dd/d27/d28/d35/f5c [4194304,4194304] 0 2022-01-31T19:44:31.529 INFO:tasks.workunit.client.1.smithi171.stdout:3/891: dread d4/dd/d27/d28/d35/d55/f5e [0,4194304] 0 2022-01-31T19:44:31.532 INFO:tasks.workunit.client.1.smithi171.stdout:3/892: unlink d4/dd/d27/f47 0 2022-01-31T19:44:31.532 INFO:tasks.workunit.client.1.smithi171.stdout:3/893: readlink d4/dd/d27/l7d 0 2022-01-31T19:44:31.533 INFO:tasks.workunit.client.1.smithi171.stdout:3/894: truncate d4/dd/d39/d51/f101 605589 0 2022-01-31T19:44:31.536 INFO:tasks.workunit.client.1.smithi171.stdout:3/895: creat d4/d21/d3a/db5/dbe/f121 x:0 0 0 2022-01-31T19:44:31.537 INFO:tasks.workunit.client.1.smithi171.stdout:3/896: fdatasync d4/dd/d27/f106 0 2022-01-31T19:44:31.538 INFO:tasks.workunit.client.1.smithi171.stdout:3/897: mkdir d4/d21/de5/d92/dab/d52/de9/d122 0 2022-01-31T19:44:31.539 INFO:tasks.workunit.client.1.smithi171.stdout:3/898: mknod d4/dd/d27/d28/d35/d55/d6c/c123 0 2022-01-31T19:44:31.539 INFO:tasks.workunit.client.1.smithi171.stdout:3/899: truncate d4/d21/d3a/db5/dbe/f121 467420 0 2022-01-31T19:44:31.540 INFO:tasks.workunit.client.1.smithi171.stdout:3/900: mkdir d4/d21/d3a/db5/dbe/dd1/d124 0 2022-01-31T19:44:31.543 INFO:tasks.workunit.client.1.smithi171.stdout:3/901: mknod d4/d21/de5/d92/dab/d60/df4/c125 0 2022-01-31T19:44:31.543 INFO:tasks.workunit.client.1.smithi171.stdout:3/902: getdents d4/dd/dc7 0 2022-01-31T19:44:31.544 INFO:tasks.workunit.client.1.smithi171.stdout:3/903: write d4/dd/d27/f83 [992921,76247] 0 2022-01-31T19:44:31.544 INFO:tasks.workunit.client.1.smithi171.stdout:3/904: stat d4/d21/de5/d92/dab/d52/f96 0 2022-01-31T19:44:31.548 INFO:tasks.workunit.client.1.smithi171.stdout:3/905: rename d4/dd/d27/d28/d35/d55/dbf to d4/dd/d27/d28/d126 0 2022-01-31T19:44:31.569 INFO:tasks.workunit.client.1.smithi171.stdout:2/876: sync 2022-01-31T19:44:31.569 INFO:tasks.workunit.client.1.smithi171.stdout:2/877: readlink d7/dd/d5e/d84/l104 0 2022-01-31T19:44:31.569 INFO:tasks.workunit.client.1.smithi171.stdout:2/878: dread - d7/d26/d63/f7f zero size 2022-01-31T19:44:31.570 INFO:tasks.workunit.client.1.smithi171.stdout:2/879: write d7/d26/f5f [154218,55027] 0 2022-01-31T19:44:31.572 INFO:tasks.workunit.client.1.smithi171.stdout:3/906: dwrite d4/dd/d87/d8e/f110 [0,4194304] 0 2022-01-31T19:44:31.572 INFO:tasks.workunit.client.1.smithi171.stdout:2/880: symlink d7/dd/dea/l110 0 2022-01-31T19:44:31.572 INFO:tasks.workunit.client.1.smithi171.stdout:2/881: read d7/d26/d3d/d40/f78 [406974,115614] 0 2022-01-31T19:44:31.576 INFO:tasks.workunit.client.1.smithi171.stdout:2/882: creat d7/dd/d5e/d6b/dcf/f111 x:0 0 0 2022-01-31T19:44:31.576 INFO:tasks.workunit.client.1.smithi171.stdout:2/883: chown d7/d9/d1a/d45/l28 12258996 1 2022-01-31T19:44:31.577 INFO:tasks.workunit.client.1.smithi171.stdout:2/884: read d7/fd5 [140480,5725] 0 2022-01-31T19:44:31.577 INFO:tasks.workunit.client.1.smithi171.stdout:2/885: chown d7/d26/d3d/lcd 353490194 1 2022-01-31T19:44:31.578 INFO:tasks.workunit.client.1.smithi171.stdout:2/886: dread - d7/f75 zero size 2022-01-31T19:44:31.583 INFO:tasks.workunit.client.1.smithi171.stdout:2/887: dread d7/dd/fa1 [0,4194304] 0 2022-01-31T19:44:31.590 INFO:tasks.workunit.client.1.smithi171.stdout:3/907: rmdir d4 39 2022-01-31T19:44:31.592 INFO:tasks.workunit.client.1.smithi171.stdout:2/888: mkdir d7/dd/d5e/d6b/ded/d112 0 2022-01-31T19:44:31.592 INFO:tasks.workunit.client.1.smithi171.stdout:2/889: write d7/d26/de5/fd0 [897478,60921] 0 2022-01-31T19:44:31.594 INFO:tasks.workunit.client.1.smithi171.stdout:3/908: mknod d4/d21/d3a/c127 0 2022-01-31T19:44:31.594 INFO:tasks.workunit.client.1.smithi171.stdout:3/909: fdatasync d4/d21/d3a/db5/dbe/dc3/f114 0 2022-01-31T19:44:31.597 INFO:tasks.workunit.client.1.smithi171.stdout:2/890: mknod d7/d9/d1a/d45/c113 0 2022-01-31T19:44:31.598 INFO:tasks.workunit.client.1.smithi171.stdout:2/891: write d7/dd/d5e/d6b/d81/fad [63111,9724] 0 2022-01-31T19:44:31.599 INFO:tasks.workunit.client.1.smithi171.stdout:2/892: stat d7/dd/d5e/d84/fc9 0 2022-01-31T19:44:31.599 INFO:tasks.workunit.client.1.smithi171.stdout:3/910: symlink d4/dd/d27/d28/d126/l128 0 2022-01-31T19:44:31.600 INFO:tasks.workunit.client.1.smithi171.stdout:3/911: getdents d4/d21/d3a/db5/dbe/dd1/d115 0 2022-01-31T19:44:31.602 INFO:tasks.workunit.client.1.smithi171.stdout:3/912: symlink d4/d21/de5/d92/dab/l129 0 2022-01-31T19:44:31.607 INFO:tasks.workunit.client.1.smithi171.stdout:2/893: dread d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:31.614 INFO:tasks.workunit.client.1.smithi171.stdout:3/913: dread d4/dd/d27/d28/d35/f48 [0,4194304] 0 2022-01-31T19:44:31.617 INFO:tasks.workunit.client.1.smithi171.stdout:3/914: mknod d4/dd/d27/d6e/c12a 0 2022-01-31T19:44:31.618 INFO:tasks.workunit.client.1.smithi171.stdout:3/915: read - d4/d21/de5/d92/dab/d52/f118 zero size 2022-01-31T19:44:31.618 INFO:tasks.workunit.client.1.smithi171.stdout:2/894: dread d7/dd/f1e [0,4194304] 0 2022-01-31T19:44:31.621 INFO:tasks.workunit.client.1.smithi171.stdout:2/895: symlink d7/d26/d63/d71/l114 0 2022-01-31T19:44:31.624 INFO:tasks.workunit.client.1.smithi171.stdout:3/916: write d4/d21/de5/d92/dab/f84 [202601,114709] 0 2022-01-31T19:44:31.626 INFO:tasks.workunit.client.1.smithi171.stdout:3/917: symlink d4/d21/l12b 0 2022-01-31T19:44:31.628 INFO:tasks.workunit.client.1.smithi171.stdout:3/918: link d4/dd/d27/d4e/c7e d4/d21/de5/d92/c12c 0 2022-01-31T19:44:31.629 INFO:tasks.workunit.client.1.smithi171.stdout:3/919: creat d4/d21/de5/da7/dfb/d10e/f12d x:0 0 0 2022-01-31T19:44:31.648 INFO:tasks.workunit.client.1.smithi171.stdout:2/896: dwrite d7/dd/d5e/f9c [0,4194304] 0 2022-01-31T19:44:31.648 INFO:tasks.workunit.client.1.smithi171.stdout:2/897: write d7/d55/df3/ffe [350733,11741] 0 2022-01-31T19:44:31.652 INFO:tasks.workunit.client.1.smithi171.stdout:2/898: rename d7/d26/de5/dcb/d107 to d7/d26/d3d/d40/dd2/d115 0 2022-01-31T19:44:31.653 INFO:tasks.workunit.client.1.smithi171.stdout:2/899: truncate d7/dd/f35 1527779 0 2022-01-31T19:44:31.656 INFO:tasks.workunit.client.1.smithi171.stdout:2/900: unlink d7/d7c/d83/d85/c8f 0 2022-01-31T19:44:31.661 INFO:tasks.workunit.client.1.smithi171.stdout:2/901: rename d7/d9/d1a/d45 to d7/d26/d3d/d40/dd2/d115/d116 0 2022-01-31T19:44:31.662 INFO:tasks.workunit.client.1.smithi171.stdout:2/902: truncate d7/d7c/d83/fe0 777476 0 2022-01-31T19:44:31.667 INFO:tasks.workunit.client.1.smithi171.stdout:2/903: write d7/dd/f3c [2344887,27824] 0 2022-01-31T19:44:31.682 INFO:tasks.workunit.client.1.smithi171.stdout:9/779: sync 2022-01-31T19:44:31.683 INFO:tasks.workunit.client.1.smithi171.stdout:7/577: sync 2022-01-31T19:44:31.683 INFO:tasks.workunit.client.1.smithi171.stdout:0/777: sync 2022-01-31T19:44:31.683 INFO:tasks.workunit.client.1.smithi171.stdout:0/778: truncate d9/db/d22/fd1 5535583 0 2022-01-31T19:44:31.684 INFO:tasks.workunit.client.1.smithi171.stdout:0/779: readlink d9/db/dc/d3f/db9/d7a/d88/lcd 0 2022-01-31T19:44:31.684 INFO:tasks.workunit.client.1.smithi171.stdout:6/758: sync 2022-01-31T19:44:31.685 INFO:tasks.workunit.client.1.smithi171.stdout:2/904: dread d7/d26/d3d/d40/dd2/d115/d116/da4/fab [4194304,4194304] 0 2022-01-31T19:44:31.685 INFO:tasks.workunit.client.1.smithi171.stdout:0/780: write d9/db/dc/d3f/db9/f8e [2987390,45197] 0 2022-01-31T19:44:31.686 INFO:tasks.workunit.client.1.smithi171.stdout:7/578: dread d1/f87 [0,4194304] 0 2022-01-31T19:44:31.686 INFO:tasks.workunit.client.1.smithi171.stdout:2/905: mkdir d7/dd/d5e/d6b/d81/d8d/da2/d117 0 2022-01-31T19:44:31.687 INFO:tasks.workunit.client.1.smithi171.stdout:0/781: getdents d9/db/d59/d71/dbe 0 2022-01-31T19:44:31.688 INFO:tasks.workunit.client.1.smithi171.stdout:2/906: creat d7/d26/de5/f118 x:0 0 0 2022-01-31T19:44:31.689 INFO:tasks.workunit.client.1.smithi171.stdout:2/907: creat d7/d9/f119 x:0 0 0 2022-01-31T19:44:31.691 INFO:tasks.workunit.client.1.smithi171.stdout:2/908: creat d7/d26/d63/d71/f11a x:0 0 0 2022-01-31T19:44:31.692 INFO:tasks.workunit.client.1.smithi171.stdout:7/579: dread d1/d1e/d2a/d29/d31/d30/f39 [0,4194304] 0 2022-01-31T19:44:31.696 INFO:tasks.workunit.client.1.smithi171.stdout:7/580: mkdir d1/d1e/d2a/d29/d31/d30/d3c/dc0 0 2022-01-31T19:44:31.697 INFO:tasks.workunit.client.1.smithi171.stdout:6/759: dread dc/d19/d29/f5c [4194304,4194304] 0 2022-01-31T19:44:31.698 INFO:tasks.workunit.client.1.smithi171.stdout:7/581: mknod d1/d1e/d2a/d29/d31/d30/d3c/d7e/cc1 0 2022-01-31T19:44:31.698 INFO:tasks.workunit.client.1.smithi171.stdout:9/780: dread dd/df/d15/d1a/f56 [0,4194304] 0 2022-01-31T19:44:31.699 INFO:tasks.workunit.client.1.smithi171.stdout:6/760: creat dc/d19/d29/d27/d37/fff x:0 0 0 2022-01-31T19:44:31.699 INFO:tasks.workunit.client.1.smithi171.stdout:6/761: chown dc/d19/d29/daf/ddb/fcd 0 1 2022-01-31T19:44:31.703 INFO:tasks.workunit.client.1.smithi171.stdout:6/762: mkdir dc/d19/d30/d3c/d100 0 2022-01-31T19:44:31.706 INFO:tasks.workunit.client.1.smithi171.stdout:0/782: dread d9/f8a [0,4194304] 0 2022-01-31T19:44:31.706 INFO:tasks.workunit.client.1.smithi171.stdout:0/783: dread - d9/fa0 zero size 2022-01-31T19:44:31.707 INFO:tasks.workunit.client.1.smithi171.stdout:0/784: mknod d9/d24/d4c/d10e/c122 0 2022-01-31T19:44:31.710 INFO:tasks.workunit.client.1.smithi171.stdout:0/785: read d9/d24/d92/d9d/d9f/fac [2438171,15154] 0 2022-01-31T19:44:31.712 INFO:tasks.workunit.client.1.smithi171.stdout:0/786: symlink d9/db/dc/d3f/d6f/l123 0 2022-01-31T19:44:31.713 INFO:tasks.workunit.client.1.smithi171.stdout:0/787: chown d9/d24/d92/d9d/d9f 168701708 1 2022-01-31T19:44:31.714 INFO:tasks.workunit.client.1.smithi171.stdout:0/788: link d9/db/dc/c91 d9/c124 0 2022-01-31T19:44:31.715 INFO:tasks.workunit.client.1.smithi171.stdout:0/789: creat d9/d24/d4c/d35/f125 x:0 0 0 2022-01-31T19:44:31.738 INFO:tasks.workunit.client.1.smithi171.stdout:9/781: dwrite dd/df/f62 [4194304,4194304] 0 2022-01-31T19:44:31.739 INFO:tasks.workunit.client.1.smithi171.stdout:9/782: chown dd/df/d3f/led 40260 1 2022-01-31T19:44:31.740 INFO:tasks.workunit.client.1.smithi171.stdout:7/582: dwrite d1/d1e/f85 [0,4194304] 0 2022-01-31T19:44:31.740 INFO:tasks.workunit.client.1.smithi171.stdout:9/783: write dd/f3c [4139758,15390] 0 2022-01-31T19:44:31.741 INFO:tasks.workunit.client.1.smithi171.stdout:2/909: dwrite d7/f22 [0,4194304] 0 2022-01-31T19:44:31.741 INFO:tasks.workunit.client.1.smithi171.stdout:7/583: rmdir d1/d1e 39 2022-01-31T19:44:31.742 INFO:tasks.workunit.client.1.smithi171.stdout:7/584: write d1/d1b/f5e [471531,29071] 0 2022-01-31T19:44:31.742 INFO:tasks.workunit.client.1.smithi171.stdout:6/763: dwrite dc/d19/d29/f24 [0,4194304] 0 2022-01-31T19:44:31.743 INFO:tasks.workunit.client.1.smithi171.stdout:0/790: dwrite d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:31.743 INFO:tasks.workunit.client.1.smithi171.stdout:9/784: unlink dd/df/d15/d1a/l43 0 2022-01-31T19:44:31.743 INFO:tasks.workunit.client.1.smithi171.stdout:9/785: getdents dd/df/d15/d53 0 2022-01-31T19:44:31.744 INFO:tasks.workunit.client.1.smithi171.stdout:2/910: dread d7/dd/d5e/d6b/d81/fa8 [0,4194304] 0 2022-01-31T19:44:31.744 INFO:tasks.workunit.client.1.smithi171.stdout:7/585: mknod d1/d52/cc2 0 2022-01-31T19:44:31.745 INFO:tasks.workunit.client.1.smithi171.stdout:7/586: write d1/d1e/d2a/d29/d31/d30/f5a [72208,41311] 0 2022-01-31T19:44:31.745 INFO:tasks.workunit.client.1.smithi171.stdout:6/764: creat dc/d8b/dee/f101 x:0 0 0 2022-01-31T19:44:31.746 INFO:tasks.workunit.client.1.smithi171.stdout:0/791: mkdir d9/db/d59/d71/dba/d126 0 2022-01-31T19:44:31.747 INFO:tasks.workunit.client.1.smithi171.stdout:0/792: fsync d9/f62 0 2022-01-31T19:44:31.747 INFO:tasks.workunit.client.1.smithi171.stdout:9/786: creat dd/df/d15/d1a/d64/dfc/f106 x:0 0 0 2022-01-31T19:44:31.748 INFO:tasks.workunit.client.1.smithi171.stdout:2/911: unlink d7/d26/d63/d71/lb0 0 2022-01-31T19:44:31.749 INFO:tasks.workunit.client.1.smithi171.stdout:7/587: mknod d1/d1e/d2a/cc3 0 2022-01-31T19:44:31.751 INFO:tasks.workunit.client.1.smithi171.stdout:2/912: rmdir d7/d26 39 2022-01-31T19:44:31.752 INFO:tasks.workunit.client.1.smithi171.stdout:7/588: mkdir d1/d1e/d2a/d29/d31/d30/dc4 0 2022-01-31T19:44:31.754 INFO:tasks.workunit.client.1.smithi171.stdout:2/913: write d7/dd/f3c [2737102,40772] 0 2022-01-31T19:44:31.759 INFO:tasks.workunit.client.1.smithi171.stdout:7/589: truncate d1/d1e/d2a/f3b 3197896 0 2022-01-31T19:44:31.759 INFO:tasks.workunit.client.1.smithi171.stdout:7/590: chown d1/d1e/d2a/d29/d31/d30/f4d 12122987 1 2022-01-31T19:44:31.759 INFO:tasks.workunit.client.1.smithi171.stdout:7/591: read - d1/d1e/d2a/d29/f66 zero size 2022-01-31T19:44:31.762 INFO:tasks.workunit.client.1.smithi171.stdout:2/914: unlink d7/d26/d3d/d40/f6e 0 2022-01-31T19:44:31.762 INFO:tasks.workunit.client.1.smithi171.stdout:2/915: stat d7/d9/d1a/f53 0 2022-01-31T19:44:31.763 INFO:tasks.workunit.client.1.smithi171.stdout:2/916: creat d7/d26/de5/ddc/df0/f11b x:0 0 0 2022-01-31T19:44:31.764 INFO:tasks.workunit.client.1.smithi171.stdout:2/917: chown d7/c34 8876703 1 2022-01-31T19:44:31.764 INFO:tasks.workunit.client.1.smithi171.stdout:2/918: write d7/dd/f80 [1710521,90196] 0 2022-01-31T19:44:31.767 INFO:tasks.workunit.client.1.smithi171.stdout:2/919: unlink d7/dd/d5e/d6b/d81/l8a 0 2022-01-31T19:44:31.767 INFO:tasks.workunit.client.1.smithi171.stdout:2/920: unlink d7/dd/fa1 0 2022-01-31T19:44:31.771 INFO:tasks.workunit.client.1.smithi171.stdout:7/592: dread d1/f36 [4194304,4194304] 0 2022-01-31T19:44:31.782 INFO:tasks.workunit.client.1.smithi171.stdout:0/793: truncate d9/db/dc/d3f/db9/f8e 1398139 0 2022-01-31T19:44:31.794 INFO:tasks.workunit.client.1.smithi171.stdout:2/921: dwrite d7/d26/d3d/d40/ff4 [4194304,4194304] 0 2022-01-31T19:44:31.794 INFO:tasks.workunit.client.1.smithi171.stdout:2/922: truncate d7/d7c/d83/d85/db9/fde 431266 0 2022-01-31T19:44:31.794 INFO:tasks.workunit.client.1.smithi171.stdout:2/923: truncate d7/dd/d5e/d84/fc9 926184 0 2022-01-31T19:44:31.806 INFO:tasks.workunit.client.1.smithi171.stdout:7/593: dwrite d1/d1e/d2a/d29/d31/ff [0,4194304] 0 2022-01-31T19:44:31.806 INFO:tasks.workunit.client.1.smithi171.stdout:7/594: readlink d1/d52/l1f 0 2022-01-31T19:44:31.809 INFO:tasks.workunit.client.1.smithi171.stdout:0/794: dwrite d9/db/f47 [0,4194304] 0 2022-01-31T19:44:31.813 INFO:tasks.workunit.client.1.smithi171.stdout:2/924: dwrite d7/dd/d5e/d6b/d81/f9b [4194304,4194304] 0 2022-01-31T19:44:31.819 INFO:tasks.workunit.client.1.smithi171.stdout:2/925: link d7/d7c/d83/d85/db9/cf7 d7/d7c/d83/d85/db9/df1/c11c 0 2022-01-31T19:44:31.822 INFO:tasks.workunit.client.1.smithi171.stdout:2/926: rmdir d7/d26/de5/ddc/df0 39 2022-01-31T19:44:31.834 INFO:tasks.workunit.client.1.smithi171.stdout:8/672: sync 2022-01-31T19:44:31.835 INFO:tasks.workunit.client.1.smithi171.stdout:5/700: sync 2022-01-31T19:44:31.836 INFO:tasks.workunit.client.1.smithi171.stdout:8/673: chown df/d27/l3f 31849889 1 2022-01-31T19:44:31.836 INFO:tasks.workunit.client.1.smithi171.stdout:5/701: dread - d5/d90/db7/d15/d80/fcb zero size 2022-01-31T19:44:31.837 INFO:tasks.workunit.client.1.smithi171.stdout:8/674: creat df/d2a/d30/fe4 x:0 0 0 2022-01-31T19:44:31.840 INFO:tasks.workunit.client.1.smithi171.stdout:5/702: write d5/d90/db7/d15/f1c [107300,88750] 0 2022-01-31T19:44:31.841 INFO:tasks.workunit.client.1.smithi171.stdout:5/703: dread - d5/d90/db7/d15/d23/f9a zero size 2022-01-31T19:44:31.841 INFO:tasks.workunit.client.1.smithi171.stdout:5/704: readlink d5/d90/db7/d15/d23/d67/l2f 0 2022-01-31T19:44:31.842 INFO:tasks.workunit.client.1.smithi171.stdout:5/705: getdents d5/d90/db7/d15/d23/d67/dd8 0 2022-01-31T19:44:31.842 INFO:tasks.workunit.client.1.smithi171.stdout:5/706: mkdir d5/d90/db7/deb 0 2022-01-31T19:44:31.843 INFO:tasks.workunit.client.1.smithi171.stdout:5/707: chown d5/d90/db7/dd/f19 704577 1 2022-01-31T19:44:31.843 INFO:tasks.workunit.client.1.smithi171.stdout:5/708: creat d5/d90/db7/d15/d5d/d6a/fec x:0 0 0 2022-01-31T19:44:31.844 INFO:tasks.workunit.client.1.smithi171.stdout:5/709: mkdir d5/ded 0 2022-01-31T19:44:31.846 INFO:tasks.workunit.client.1.smithi171.stdout:5/710: link d5/d90/db7/d15/d23/d67/c4b d5/d59/da0/cee 0 2022-01-31T19:44:31.847 INFO:tasks.workunit.client.1.smithi171.stdout:7/595: dwrite d1/d1e/d2a/d29/d31/f38 [0,4194304] 0 2022-01-31T19:44:31.847 INFO:tasks.workunit.client.1.smithi171.stdout:5/711: write d5/d90/db7/d15/d23/d3d/fb2 [3785276,87234] 0 2022-01-31T19:44:31.847 INFO:tasks.workunit.client.1.smithi171.stdout:5/712: stat d5/d90/dd1 0 2022-01-31T19:44:31.848 INFO:tasks.workunit.client.1.smithi171.stdout:5/713: dread - d5/d90/db7/d15/d23/d58/d9e/fbd zero size 2022-01-31T19:44:31.848 INFO:tasks.workunit.client.1.smithi171.stdout:0/795: dwrite d9/db/f47 [0,4194304] 0 2022-01-31T19:44:31.849 INFO:tasks.workunit.client.1.smithi171.stdout:2/927: dwrite d7/dd/dfc/f105 [0,4194304] 0 2022-01-31T19:44:31.849 INFO:tasks.workunit.client.1.smithi171.stdout:2/928: write d7/d26/f92 [2229197,54479] 0 2022-01-31T19:44:31.851 INFO:tasks.workunit.client.1.smithi171.stdout:5/714: symlink d5/d90/db7/d15/d23/d58/lef 0 2022-01-31T19:44:31.851 INFO:tasks.workunit.client.1.smithi171.stdout:8/675: dread df/d18/f4e [0,4194304] 0 2022-01-31T19:44:31.855 INFO:tasks.workunit.client.1.smithi171.stdout:7/596: dread d1/d1e/d2a/d29/f6d [0,4194304] 0 2022-01-31T19:44:31.855 INFO:tasks.workunit.client.1.smithi171.stdout:0/796: link d9/db/dc/l45 d9/db/dc/d3f/db9/l127 0 2022-01-31T19:44:31.855 INFO:tasks.workunit.client.1.smithi171.stdout:8/676: write df/d18/f55 [1649334,21558] 0 2022-01-31T19:44:31.862 INFO:tasks.workunit.client.1.smithi171.stdout:7/597: write d1/d1e/d2a/d29/d31/d30/d6b/f86 [4948794,5665] 0 2022-01-31T19:44:31.863 INFO:tasks.workunit.client.1.smithi171.stdout:7/598: creat d1/d26/d47/d8f/fc5 x:0 0 0 2022-01-31T19:44:31.864 INFO:tasks.workunit.client.1.smithi171.stdout:7/599: write d1/d1e/d2a/f3b [1183079,39548] 0 2022-01-31T19:44:31.865 INFO:tasks.workunit.client.1.smithi171.stdout:0/797: rename d9/d24/d92/d9d/d9f/d107 to d9/db/d22/d73/ddc/d128 0 2022-01-31T19:44:31.868 INFO:tasks.workunit.client.1.smithi171.stdout:2/929: dwrite d7/d26/f5f [0,4194304] 0 2022-01-31T19:44:31.868 INFO:tasks.workunit.client.1.smithi171.stdout:2/930: fdatasync d7/d26/d63/f7f 0 2022-01-31T19:44:31.868 INFO:tasks.workunit.client.1.smithi171.stdout:2/931: truncate d7/dd/d5e/d6b/d81/fa6 790141 0 2022-01-31T19:44:31.869 INFO:tasks.workunit.client.1.smithi171.stdout:8/677: link df/d18/f1e df/d18/d19/fe5 0 2022-01-31T19:44:31.873 INFO:tasks.workunit.client.1.smithi171.stdout:5/715: rmdir d5 39 2022-01-31T19:44:31.876 INFO:tasks.workunit.client.1.smithi171.stdout:7/600: rename d1/f36 to d1/d1e/d2a/d29/d31/d30/fc6 0 2022-01-31T19:44:31.877 INFO:tasks.workunit.client.1.smithi171.stdout:0/798: symlink d9/db/d59/d71/dba/d126/l129 0 2022-01-31T19:44:31.877 INFO:tasks.workunit.client.1.smithi171.stdout:0/799: fsync d9/d24/d92/f94 0 2022-01-31T19:44:31.878 INFO:tasks.workunit.client.1.smithi171.stdout:2/932: creat d7/d26/de5/dc0/f11d x:0 0 0 2022-01-31T19:44:31.879 INFO:tasks.workunit.client.1.smithi171.stdout:8/678: mknod df/d18/d19/d46/ce6 0 2022-01-31T19:44:31.880 INFO:tasks.workunit.client.1.smithi171.stdout:2/933: write d7/dd/d5e/d6b/d81/f9b [3559115,53651] 0 2022-01-31T19:44:31.880 INFO:tasks.workunit.client.1.smithi171.stdout:7/601: getdents d1/d26/d47 0 2022-01-31T19:44:31.880 INFO:tasks.workunit.client.1.smithi171.stdout:7/602: unlink d1/d1b/ca9 0 2022-01-31T19:44:31.881 INFO:tasks.workunit.client.1.smithi171.stdout:7/603: creat d1/d1e/d2a/d29/d31/d30/d3c/d7e/fc7 x:0 0 0 2022-01-31T19:44:31.882 INFO:tasks.workunit.client.1.smithi171.stdout:5/716: dread d5/d90/db7/f46 [0,4194304] 0 2022-01-31T19:44:31.882 INFO:tasks.workunit.client.1.smithi171.stdout:5/717: chown d5/d90/db7/dd/f20 356 1 2022-01-31T19:44:31.883 INFO:tasks.workunit.client.1.smithi171.stdout:0/800: unlink d9/db/dc2/f29 0 2022-01-31T19:44:31.883 INFO:tasks.workunit.client.1.smithi171.stdout:0/801: chown d9/db/dc/d3f/d6f/l123 12 1 2022-01-31T19:44:31.883 INFO:tasks.workunit.client.1.smithi171.stdout:0/802: chown d9/db/d22/fd1 0 1 2022-01-31T19:44:31.884 INFO:tasks.workunit.client.1.smithi171.stdout:8/679: creat df/d18/d19/d21/d88/fe7 x:0 0 0 2022-01-31T19:44:31.884 INFO:tasks.workunit.client.1.smithi171.stdout:8/680: readlink df/d18/la7 0 2022-01-31T19:44:31.885 INFO:tasks.workunit.client.1.smithi171.stdout:8/681: chown df/d18/d79/d7a/ld4 86094 1 2022-01-31T19:44:31.885 INFO:tasks.workunit.client.1.smithi171.stdout:2/934: write d7/d26/d3d/d40/dd2/d115/d116/da4/fd4 [570209,90617] 0 2022-01-31T19:44:31.886 INFO:tasks.workunit.client.1.smithi171.stdout:2/935: fsync d7/dd/d5e/d6b/d81/d8d/da2/fff 0 2022-01-31T19:44:31.887 INFO:tasks.workunit.client.1.smithi171.stdout:7/604: creat d1/d52/fc8 x:0 0 0 2022-01-31T19:44:31.892 INFO:tasks.workunit.client.1.smithi171.stdout:5/718: mkdir d5/d90/db7/d15/d23/d58/d9e/dd7/df0 0 2022-01-31T19:44:31.893 INFO:tasks.workunit.client.1.smithi171.stdout:0/803: symlink d9/d24/d4c/d101/l12a 0 2022-01-31T19:44:31.894 INFO:tasks.workunit.client.1.smithi171.stdout:8/682: mkdir df/d1a/de8 0 2022-01-31T19:44:31.895 INFO:tasks.workunit.client.1.smithi171.stdout:7/605: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9 0 2022-01-31T19:44:31.899 INFO:tasks.workunit.client.1.smithi171.stdout:0/804: creat d9/d24/d92/d9d/dc6/dfb/f12b x:0 0 0 2022-01-31T19:44:31.901 INFO:tasks.workunit.client.1.smithi171.stdout:8/683: symlink df/d18/d79/d7a/le9 0 2022-01-31T19:44:31.903 INFO:tasks.workunit.client.1.smithi171.stdout:1/787: sync 2022-01-31T19:44:31.903 INFO:tasks.workunit.client.1.smithi171.stdout:3/920: sync 2022-01-31T19:44:31.903 INFO:tasks.workunit.client.1.smithi171.stdout:4/671: sync 2022-01-31T19:44:31.904 INFO:tasks.workunit.client.1.smithi171.stdout:2/936: dwrite d7/f22 [0,4194304] 0 2022-01-31T19:44:31.904 INFO:tasks.workunit.client.1.smithi171.stdout:7/606: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/dca 0 2022-01-31T19:44:31.905 INFO:tasks.workunit.client.1.smithi171.stdout:7/607: write d1/d1e/d2a/d29/d31/d30/d3c/d5f/faa [3809131,118334] 0 2022-01-31T19:44:31.905 INFO:tasks.workunit.client.1.smithi171.stdout:5/719: symlink d5/d90/db7/d15/lf1 0 2022-01-31T19:44:31.906 INFO:tasks.workunit.client.1.smithi171.stdout:5/720: dread - d5/d90/db7/d15/d23/d6b/fc0 zero size 2022-01-31T19:44:31.908 INFO:tasks.workunit.client.1.smithi171.stdout:0/805: link d9/db/dc2/l2a d9/d24/d4c/d35/d67/df4/l12c 0 2022-01-31T19:44:31.908 INFO:tasks.workunit.client.1.smithi171.stdout:0/806: write d9/db/d59/ffe [323987,100312] 0 2022-01-31T19:44:31.910 INFO:tasks.workunit.client.1.smithi171.stdout:8/684: mknod df/d1a/dae/de0/cea 0 2022-01-31T19:44:31.910 INFO:tasks.workunit.client.1.smithi171.stdout:8/685: stat df/d95 0 2022-01-31T19:44:31.910 INFO:tasks.workunit.client.1.smithi171.stdout:8/686: creat df/d18/d19/d82/dd3/feb x:0 0 0 2022-01-31T19:44:31.911 INFO:tasks.workunit.client.1.smithi171.stdout:3/921: mknod d4/d21/c12e 0 2022-01-31T19:44:31.912 INFO:tasks.workunit.client.1.smithi171.stdout:4/672: truncate d2/d3/d5/d41/dcf/d12/f8d 268353 0 2022-01-31T19:44:31.913 INFO:tasks.workunit.client.1.smithi171.stdout:4/673: chown d2/l37 15 1 2022-01-31T19:44:31.913 INFO:tasks.workunit.client.1.smithi171.stdout:4/674: chown d2/d3/d5/d3b/fd9 2353 1 2022-01-31T19:44:31.913 INFO:tasks.workunit.client.1.smithi171.stdout:4/675: stat d2/d3/d5/d91/d77/da5 0 2022-01-31T19:44:31.915 INFO:tasks.workunit.client.1.smithi171.stdout:2/937: unlink d7/d55/cae 0 2022-01-31T19:44:31.915 INFO:tasks.workunit.client.1.smithi171.stdout:2/938: fdatasync d7/d26/d63/d9f/dc3/fca 0 2022-01-31T19:44:31.916 INFO:tasks.workunit.client.1.smithi171.stdout:2/939: fsync d7/f11 0 2022-01-31T19:44:31.916 INFO:tasks.workunit.client.1.smithi171.stdout:5/721: creat d5/d90/dd3/ff2 x:0 0 0 2022-01-31T19:44:31.917 INFO:tasks.workunit.client.1.smithi171.stdout:5/722: getdents d5/ded 0 2022-01-31T19:44:31.917 INFO:tasks.workunit.client.1.smithi171.stdout:5/723: fdatasync d5/db0/fd4 0 2022-01-31T19:44:31.920 INFO:tasks.workunit.client.1.smithi171.stdout:2/940: chown d7/d26/d3d/d40/f57 762774 1 2022-01-31T19:44:31.921 INFO:tasks.workunit.client.1.smithi171.stdout:4/676: symlink d2/d3/d5/d70/d7c/le7 0 2022-01-31T19:44:31.923 INFO:tasks.workunit.client.1.smithi171.stdout:7/608: rename d1/d1e/d2a/d29/d31/c25 to d1/d1e/ccb 0 2022-01-31T19:44:31.924 INFO:tasks.workunit.client.1.smithi171.stdout:7/609: creat d1/d1e/d2a/d58/fcc x:0 0 0 2022-01-31T19:44:31.924 INFO:tasks.workunit.client.1.smithi171.stdout:7/610: chown d1/d1e/d2a/d29/d31/d30/d3c/d5f/c91 3 1 2022-01-31T19:44:31.925 INFO:tasks.workunit.client.1.smithi171.stdout:1/788: dwrite dd/d23/d2e/d5e/d81/d98/f85 [0,4194304] 0 2022-01-31T19:44:31.925 INFO:tasks.workunit.client.1.smithi171.stdout:4/677: truncate d2/d3/d5/dc/f95 3281767 0 2022-01-31T19:44:31.925 INFO:tasks.workunit.client.1.smithi171.stdout:4/678: dread - d2/d3/d5/d41/dcf/d12/f2a zero size 2022-01-31T19:44:31.927 INFO:tasks.workunit.client.1.smithi171.stdout:0/807: rename d9/db/d59/d71/dba to d9/d24/d92/d12d 0 2022-01-31T19:44:31.929 INFO:tasks.workunit.client.1.smithi171.stdout:7/611: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd 0 2022-01-31T19:44:31.930 INFO:tasks.workunit.client.1.smithi171.stdout:2/941: getdents d7/d9/d1a 0 2022-01-31T19:44:31.930 INFO:tasks.workunit.client.1.smithi171.stdout:2/942: write d7/d26/d63/fbc [869842,130648] 0 2022-01-31T19:44:31.931 INFO:tasks.workunit.client.1.smithi171.stdout:2/943: write d7/d9/f12 [134392,127840] 0 2022-01-31T19:44:31.931 INFO:tasks.workunit.client.1.smithi171.stdout:1/789: creat dd/df/fe8 x:0 0 0 2022-01-31T19:44:31.932 INFO:tasks.workunit.client.1.smithi171.stdout:1/790: read dd/f7a [652232,91642] 0 2022-01-31T19:44:31.932 INFO:tasks.workunit.client.1.smithi171.stdout:7/612: creat d1/d1e/d2a/d29/d77/dab/fce x:0 0 0 2022-01-31T19:44:31.933 INFO:tasks.workunit.client.1.smithi171.stdout:7/613: write d1/d1e/d2a/d29/fb2 [370062,17843] 0 2022-01-31T19:44:31.933 INFO:tasks.workunit.client.1.smithi171.stdout:7/614: chown d1/d1e/d2a/d29/d31/d30/f5a 13 1 2022-01-31T19:44:31.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/791: mkdir dd/d23/dd4/de9 0 2022-01-31T19:44:31.936 INFO:tasks.workunit.client.1.smithi171.stdout:2/944: mkdir d7/d26/d3d/d40/dd2/d115/d116/da4/d11e 0 2022-01-31T19:44:31.937 INFO:tasks.workunit.client.1.smithi171.stdout:8/687: dwrite df/d1a/f53 [0,4194304] 0 2022-01-31T19:44:31.938 INFO:tasks.workunit.client.1.smithi171.stdout:1/792: rename dd/df/d19/d1b/d1e to dd/df/d84/dea 0 2022-01-31T19:44:31.939 INFO:tasks.workunit.client.1.smithi171.stdout:2/945: creat d7/d7c/d83/d85/f11f x:0 0 0 2022-01-31T19:44:31.940 INFO:tasks.workunit.client.1.smithi171.stdout:2/946: readlink d7/d9/d1a/lc7 0 2022-01-31T19:44:31.942 INFO:tasks.workunit.client.1.smithi171.stdout:0/808: dread d9/f13 [8388608,4194304] 0 2022-01-31T19:44:31.942 INFO:tasks.workunit.client.1.smithi171.stdout:0/809: dread - d9/db/d59/fe6 zero size 2022-01-31T19:44:31.943 INFO:tasks.workunit.client.1.smithi171.stdout:2/947: symlink d7/d26/d3d/d40/dd2/d115/l120 0 2022-01-31T19:44:31.943 INFO:tasks.workunit.client.1.smithi171.stdout:2/948: readlink d7/d26/de5/ddc/df0/lf8 0 2022-01-31T19:44:31.944 INFO:tasks.workunit.client.1.smithi171.stdout:0/810: unlink d9/d24/d92/d9d/d9f/f11e 0 2022-01-31T19:44:31.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/949: truncate d7/d26/d3d/d40/dd2/d115/d116/da4/fb4 2772803 0 2022-01-31T19:44:31.946 INFO:tasks.workunit.client.1.smithi171.stdout:2/950: dread - d7/dd/d5e/d6b/d81/d8d/da2/fff zero size 2022-01-31T19:44:31.947 INFO:tasks.workunit.client.1.smithi171.stdout:7/615: fsync d1/d1e/d2a/d29/d31/d30/d6b/f86 0 2022-01-31T19:44:31.947 INFO:tasks.workunit.client.1.smithi171.stdout:7/616: truncate d1/d1e/d2a/d29/d31/d30/d3c/fbb 492451 0 2022-01-31T19:44:31.947 INFO:tasks.workunit.client.1.smithi171.stdout:7/617: fsync d1/d1e/d2a/f3b 0 2022-01-31T19:44:31.947 INFO:tasks.workunit.client.1.smithi171.stdout:0/811: readlink d9/db/dc/d3f/l57 0 2022-01-31T19:44:31.948 INFO:tasks.workunit.client.1.smithi171.stdout:0/812: fsync d9/fd4 0 2022-01-31T19:44:31.948 INFO:tasks.workunit.client.1.smithi171.stdout:3/922: dwrite d4/d21/d3a/d59/dc1/fc2 [0,4194304] 0 2022-01-31T19:44:31.948 INFO:tasks.workunit.client.1.smithi171.stdout:2/951: truncate d7/dd/dfc/f105 2301229 0 2022-01-31T19:44:31.949 INFO:tasks.workunit.client.1.smithi171.stdout:2/952: chown d7/dd/f80 283305 1 2022-01-31T19:44:31.949 INFO:tasks.workunit.client.1.smithi171.stdout:2/953: fdatasync d7/d26/f5f 0 2022-01-31T19:44:31.949 INFO:tasks.workunit.client.1.smithi171.stdout:4/679: dwrite d2/d3/d5/d41/dcf/f62 [0,4194304] 0 2022-01-31T19:44:31.950 INFO:tasks.workunit.client.1.smithi171.stdout:8/688: dread df/d18/d19/f5f [0,4194304] 0 2022-01-31T19:44:31.951 INFO:tasks.workunit.client.1.smithi171.stdout:7/618: rename d1/d1e/d2a/d29/d31/d30/d3c/d5f/faa to d1/d1e/d2a/d29/d31/d30/d3c/d5f/fcf 0 2022-01-31T19:44:31.951 INFO:tasks.workunit.client.1.smithi171.stdout:0/813: mknod d9/db/c12e 0 2022-01-31T19:44:31.951 INFO:tasks.workunit.client.1.smithi171.stdout:0/814: truncate d9/db/dc/d3f/db9/d7a/d88/fa3 4377373 0 2022-01-31T19:44:31.954 INFO:tasks.workunit.client.1.smithi171.stdout:1/793: dread dd/d23/d34/d4b/f52 [0,4194304] 0 2022-01-31T19:44:31.957 INFO:tasks.workunit.client.1.smithi171.stdout:2/954: dread d7/fa [0,4194304] 0 2022-01-31T19:44:31.959 INFO:tasks.workunit.client.1.smithi171.stdout:2/955: write d7/dd/f1e [4149003,46918] 0 2022-01-31T19:44:31.959 INFO:tasks.workunit.client.1.smithi171.stdout:8/689: dread df/d27/f57 [0,4194304] 0 2022-01-31T19:44:31.964 INFO:tasks.workunit.client.1.smithi171.stdout:4/680: rename d2/d3/f7 to d2/d3/d5/d41/dcf/dc7/fe8 0 2022-01-31T19:44:31.974 INFO:tasks.workunit.client.1.smithi171.stdout:7/619: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd0 x:0 0 0 2022-01-31T19:44:31.979 INFO:tasks.workunit.client.1.smithi171.stdout:0/815: dwrite d9/f3b [0,4194304] 0 2022-01-31T19:44:31.980 INFO:tasks.workunit.client.1.smithi171.stdout:3/923: dwrite d4/dd/d27/d28/d35/d55/f5e [0,4194304] 0 2022-01-31T19:44:31.980 INFO:tasks.workunit.client.1.smithi171.stdout:3/924: chown d4/d21/d3a/l5b 2829000 1 2022-01-31T19:44:31.980 INFO:tasks.workunit.client.1.smithi171.stdout:3/925: getdents d4/dd/dc7 0 2022-01-31T19:44:31.983 INFO:tasks.workunit.client.1.smithi171.stdout:3/926: write d4/d21/d3a/d59/dc1/fc2 [967192,8094] 0 2022-01-31T19:44:31.985 INFO:tasks.workunit.client.1.smithi171.stdout:1/794: link dd/df/d19/f96 dd/d23/dd4/de9/feb 0 2022-01-31T19:44:31.987 INFO:tasks.workunit.client.1.smithi171.stdout:2/956: mkdir d7/d7c/d83/d85/d121 0 2022-01-31T19:44:31.987 INFO:tasks.workunit.client.1.smithi171.stdout:2/957: write d7/d26/de5/fd0 [1644271,120399] 0 2022-01-31T19:44:31.988 INFO:tasks.workunit.client.1.smithi171.stdout:2/958: readlink d7/d26/d3d/d40/dd2/lf2 0 2022-01-31T19:44:31.988 INFO:tasks.workunit.client.1.smithi171.stdout:8/690: creat df/d2a/fec x:0 0 0 2022-01-31T19:44:31.992 INFO:tasks.workunit.client.1.smithi171.stdout:4/681: link d2/d3/d5/d3b/faf d2/d3/d5/dc/d4b/dbc/fe9 0 2022-01-31T19:44:31.995 INFO:tasks.workunit.client.1.smithi171.stdout:7/620: mkdir d1/d1e/d2a/d29/d31/d30/d3c/d7e/dd1 0 2022-01-31T19:44:31.998 INFO:tasks.workunit.client.1.smithi171.stdout:0/816: creat d9/db/dc/d3f/db9/d7a/f12f x:0 0 0 2022-01-31T19:44:32.004 INFO:tasks.workunit.client.1.smithi171.stdout:0/817: write d9/d24/f4d [974938,22207] 0 2022-01-31T19:44:32.004 INFO:tasks.workunit.client.1.smithi171.stdout:0/818: stat d9/db/dc/d3f/f6d 0 2022-01-31T19:44:32.005 INFO:tasks.workunit.client.1.smithi171.stdout:0/819: fsync d9/db/dc/f104 0 2022-01-31T19:44:32.005 INFO:tasks.workunit.client.1.smithi171.stdout:3/927: mkdir d4/dd/d4b/dfa/d12f 0 2022-01-31T19:44:32.007 INFO:tasks.workunit.client.1.smithi171.stdout:7/621: rename d1/d1b/f5e to d1/d52/fd2 0 2022-01-31T19:44:32.009 INFO:tasks.workunit.client.1.smithi171.stdout:0/820: creat d9/d24/d92/d9d/dc6/d115/f130 x:0 0 0 2022-01-31T19:44:32.010 INFO:tasks.workunit.client.1.smithi171.stdout:7/622: dread d1/d1e/d2a/d29/f6d [0,4194304] 0 2022-01-31T19:44:32.011 INFO:tasks.workunit.client.1.smithi171.stdout:0/821: read d9/d24/d92/f69 [825185,86430] 0 2022-01-31T19:44:32.013 INFO:tasks.workunit.client.1.smithi171.stdout:3/928: mknod d4/dd/d39/d51/c130 0 2022-01-31T19:44:32.014 INFO:tasks.workunit.client.1.smithi171.stdout:4/682: mknod d2/d3/cea 0 2022-01-31T19:44:32.014 INFO:tasks.workunit.client.1.smithi171.stdout:7/623: truncate d1/d1e/d2a/d29/d31/ff 1896871 0 2022-01-31T19:44:32.016 INFO:tasks.workunit.client.1.smithi171.stdout:0/822: link d9/d24/d4c/d35/f8d d9/db/f131 0 2022-01-31T19:44:32.016 INFO:tasks.workunit.client.1.smithi171.stdout:0/823: chown d9/d24/d92/d9d/ff1 0 1 2022-01-31T19:44:32.016 INFO:tasks.workunit.client.1.smithi171.stdout:3/929: write d4/dd/d39/d51/f71 [2899606,11165] 0 2022-01-31T19:44:32.016 INFO:tasks.workunit.client.1.smithi171.stdout:3/930: truncate d4/dd/d27/d28/d126/fc5 4884649 0 2022-01-31T19:44:32.017 INFO:tasks.workunit.client.1.smithi171.stdout:4/683: mknod d2/d3/d5/d91/d77/da5/ceb 0 2022-01-31T19:44:32.017 INFO:tasks.workunit.client.1.smithi171.stdout:4/684: readlink d2/d3/d5/d41/dcf/d12/db6/lca 0 2022-01-31T19:44:32.018 INFO:tasks.workunit.client.1.smithi171.stdout:7/624: symlink d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/ld3 0 2022-01-31T19:44:32.020 INFO:tasks.workunit.client.1.smithi171.stdout:0/824: mkdir d9/d24/d4c/d101/d132 0 2022-01-31T19:44:32.020 INFO:tasks.workunit.client.1.smithi171.stdout:1/795: dwrite dd/d23/d74/dcb/ddb/f9c [0,4194304] 0 2022-01-31T19:44:32.021 INFO:tasks.workunit.client.1.smithi171.stdout:2/959: dwrite d7/d9/fe [0,4194304] 0 2022-01-31T19:44:32.022 INFO:tasks.workunit.client.1.smithi171.stdout:8/691: dwrite df/d18/d19/d46/f78 [0,4194304] 0 2022-01-31T19:44:32.022 INFO:tasks.workunit.client.1.smithi171.stdout:8/692: read - df/d18/d19/d82/dd3/fde zero size 2022-01-31T19:44:32.022 INFO:tasks.workunit.client.1.smithi171.stdout:8/693: write df/d27/fd5 [923529,16801] 0 2022-01-31T19:44:32.022 INFO:tasks.workunit.client.1.smithi171.stdout:8/694: chown df/d18/d19/d82/fbd 7 1 2022-01-31T19:44:32.023 INFO:tasks.workunit.client.1.smithi171.stdout:8/695: readlink df/d27/l65 0 2022-01-31T19:44:32.024 INFO:tasks.workunit.client.1.smithi171.stdout:3/931: creat d4/d21/d3a/db5/dbe/f131 x:0 0 0 2022-01-31T19:44:32.025 INFO:tasks.workunit.client.1.smithi171.stdout:3/932: fdatasync d4/d21/d3a/d90/fc9 0 2022-01-31T19:44:32.025 INFO:tasks.workunit.client.1.smithi171.stdout:0/825: read d9/f62 [7991,30654] 0 2022-01-31T19:44:32.026 INFO:tasks.workunit.client.1.smithi171.stdout:2/960: write d7/dd/d5e/d6b/f7a [2625699,93924] 0 2022-01-31T19:44:32.026 INFO:tasks.workunit.client.1.smithi171.stdout:2/961: readlink d7/d26/d3d/d40/dd2/d115/d116/l28 0 2022-01-31T19:44:32.026 INFO:tasks.workunit.client.1.smithi171.stdout:2/962: dread - d7/d26/d63/f7f zero size 2022-01-31T19:44:32.027 INFO:tasks.workunit.client.1.smithi171.stdout:2/963: chown d7/dd 17967955 1 2022-01-31T19:44:32.027 INFO:tasks.workunit.client.1.smithi171.stdout:1/796: rename dd/d23/d34/c71 to dd/d23/d34/cec 0 2022-01-31T19:44:32.034 INFO:tasks.workunit.client.1.smithi171.stdout:8/696: dread df/d18/d19/d21/f26 [0,4194304] 0 2022-01-31T19:44:32.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/933: mknod d4/d21/de5/d92/dab/d52/d7a/c132 0 2022-01-31T19:44:32.036 INFO:tasks.workunit.client.1.smithi171.stdout:3/934: chown d4/d21/de5/d92/dab/d52/d7a/c132 436 1 2022-01-31T19:44:32.037 INFO:tasks.workunit.client.1.smithi171.stdout:3/935: write d4/dd/d87/f9f [1132716,109310] 0 2022-01-31T19:44:32.037 INFO:tasks.workunit.client.1.smithi171.stdout:0/826: truncate d9/db/dc/f104 1162888 0 2022-01-31T19:44:32.038 INFO:tasks.workunit.client.1.smithi171.stdout:3/936: write d4/dd/d87/fa6 [330984,124448] 0 2022-01-31T19:44:32.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:31 smithi167 conmon[32206]: cluster 2022-01-31T19:44:30.436623+0000 mgr.smithi171.asyxnc (mgr.24387) 36 : cluster [DBG] pgmap v24: 65 pgs: 65 active+clean; 3.4 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 142 MiB/s rd, 305 MiB/s wr, 467 op/s 2022-01-31T19:44:32.040 INFO:tasks.workunit.client.1.smithi171.stdout:7/625: dwrite d1/d1e/d2a/d29/f7a [0,4194304] 0 2022-01-31T19:44:32.040 INFO:tasks.workunit.client.1.smithi171.stdout:1/797: mkdir dd/df/d84/dea/d64/da7/dca/ded 0 2022-01-31T19:44:32.041 INFO:tasks.workunit.client.1.smithi171.stdout:1/798: readlink l6 0 2022-01-31T19:44:32.041 INFO:tasks.workunit.client.1.smithi171.stdout:2/964: symlink d7/d26/d3d/d40/dd2/d115/d116/l122 0 2022-01-31T19:44:32.041 INFO:tasks.workunit.client.1.smithi171.stdout:2/965: chown d7/d26/de5/ddc 2622587 1 2022-01-31T19:44:32.041 INFO:tasks.workunit.client.1.smithi171.stdout:2/966: readlink d7/d55/l60 0 2022-01-31T19:44:32.042 INFO:tasks.workunit.client.1.smithi171.stdout:0/827: creat d9/d24/f133 x:0 0 0 2022-01-31T19:44:32.043 INFO:tasks.workunit.client.1.smithi171.stdout:0/828: truncate d9/db/dc2/fad 1224996 0 2022-01-31T19:44:32.043 INFO:tasks.workunit.client.1.smithi171.stdout:3/937: mknod d4/d21/de5/d92/dab/c133 0 2022-01-31T19:44:32.044 INFO:tasks.workunit.client.1.smithi171.stdout:3/938: fsync d4/dd/d27/d28/f57 0 2022-01-31T19:44:32.050 INFO:tasks.workunit.client.1.smithi171.stdout:7/626: mknod d1/d1e/d2a/d29/d77/dab/cd4 0 2022-01-31T19:44:32.052 INFO:tasks.workunit.client.1.smithi171.stdout:6/765: sync 2022-01-31T19:44:32.052 INFO:tasks.workunit.client.1.smithi171.stdout:6/766: fsync dc/d19/d30/d3c/d80/f83 0 2022-01-31T19:44:32.055 INFO:tasks.workunit.client.1.smithi171.stdout:3/939: mkdir d4/d21/d3a/db5/dbe/dd1/d115/d134 0 2022-01-31T19:44:32.056 INFO:tasks.workunit.client.1.smithi171.stdout:6/767: mknod dc/d19/d29/d27/d37/d3b/d67/dd1/c102 0 2022-01-31T19:44:32.056 INFO:tasks.workunit.client.1.smithi171.stdout:6/768: write dc/d19/d29/d27/d37/d3b/d67/dd1/fd3 [769939,102568] 0 2022-01-31T19:44:32.057 INFO:tasks.workunit.client.1.smithi171.stdout:2/967: mkdir d7/d26/d3d/d40/dd2/d115/d116/da4/d123 0 2022-01-31T19:44:32.058 INFO:tasks.workunit.client.1.smithi171.stdout:2/968: mkdir d7/dd/d5e/d6b/d81/d8d/da2/d124 0 2022-01-31T19:44:32.059 INFO:tasks.workunit.client.1.smithi171.stdout:2/969: symlink d7/dd/d5e/d6b/d81/d8d/l125 0 2022-01-31T19:44:32.059 INFO:tasks.workunit.client.1.smithi171.stdout:2/970: write d7/dd/d5e/d6b/d81/fad [249973,36678] 0 2022-01-31T19:44:32.060 INFO:tasks.workunit.client.1.smithi171.stdout:6/769: write dc/d19/d30/d3c/fc7 [267613,88175] 0 2022-01-31T19:44:32.060 INFO:tasks.workunit.client.1.smithi171.stdout:2/971: truncate d7/dd/dfc/f105 179188 0 2022-01-31T19:44:32.063 INFO:tasks.workunit.client.1.smithi171.stdout:6/770: getdents dc/d19/d29/d27/d37/d68 0 2022-01-31T19:44:32.063 INFO:tasks.workunit.client.1.smithi171.stdout:2/972: read d7/d26/de5/fc2 [3901568,46763] 0 2022-01-31T19:44:32.066 INFO:tasks.workunit.client.1.smithi171.stdout:6/771: symlink dc/d19/d30/d3c/d80/l103 0 2022-01-31T19:44:32.066 INFO:tasks.workunit.client.1.smithi171.stdout:6/772: stat dc/d19/d29/d27/d37/d3b/d67/dd1 0 2022-01-31T19:44:32.067 INFO:tasks.workunit.client.1.smithi171.stdout:8/697: dwrite df/d18/d79/f9d [0,4194304] 0 2022-01-31T19:44:32.068 INFO:tasks.workunit.client.1.smithi171.stdout:1/799: dwrite dd/df/d84/dea/d64/f58 [0,4194304] 0 2022-01-31T19:44:32.076 INFO:tasks.workunit.client.1.smithi171.stdout:2/973: rename d7/d26 to d7/dd/dfc/d101/d126 0 2022-01-31T19:44:32.077 INFO:tasks.workunit.client.1.smithi171.stdout:1/800: mkdir dd/d23/d2e/d5e/d81/d98/d80/dee 0 2022-01-31T19:44:32.077 INFO:tasks.workunit.client.1.smithi171.stdout:1/801: fdatasync dd/d23/d34/d4b/f52 0 2022-01-31T19:44:32.077 INFO:tasks.workunit.client.1.smithi171.stdout:2/974: read d7/fa [732338,14756] 0 2022-01-31T19:44:32.080 INFO:tasks.workunit.client.1.smithi171.stdout:1/802: read dd/d23/d2e/d5e/f40 [2348402,30543] 0 2022-01-31T19:44:32.081 INFO:tasks.workunit.client.1.smithi171.stdout:8/698: dread df/f17 [0,4194304] 0 2022-01-31T19:44:32.081 INFO:tasks.workunit.client.1.smithi171.stdout:8/699: chown df/d18/f4d 157 1 2022-01-31T19:44:32.081 INFO:tasks.workunit.client.1.smithi171.stdout:8/700: fsync df/d18/d19/d82/fd1 0 2022-01-31T19:44:32.081 INFO:tasks.workunit.client.1.smithi171.stdout:0/829: dwrite d9/db/fc4 [0,4194304] 0 2022-01-31T19:44:32.082 INFO:tasks.workunit.client.1.smithi171.stdout:3/940: dwrite d4/d21/de5/d92/dab/d52/f54 [0,4194304] 0 2022-01-31T19:44:32.082 INFO:tasks.workunit.client.1.smithi171.stdout:6/773: rename dc/d19/d29/d27/f93 to dc/d19/d29/d27/d37/d3b/d67/dd1/f104 0 2022-01-31T19:44:32.083 INFO:tasks.workunit.client.1.smithi171.stdout:2/975: creat d7/dd/dfc/d101/d126/d3d/d40/dd2/f127 x:0 0 0 2022-01-31T19:44:32.083 INFO:tasks.workunit.client.1.smithi171.stdout:2/976: fsync d7/dd/d5e/d6b/d81/f9b 0 2022-01-31T19:44:32.083 INFO:tasks.workunit.client.1.smithi171.stdout:3/941: stat d4/dd/d87/d8e/l8a 0 2022-01-31T19:44:32.087 INFO:tasks.workunit.client.1.smithi171.stdout:0/830: dread d9/db/f121 [0,4194304] 0 2022-01-31T19:44:32.089 INFO:tasks.workunit.client.1.smithi171.stdout:2/977: symlink d7/dd/dfc/d101/d126/d3d/d40/dd2/d115/d116/da4/d123/l128 0 2022-01-31T19:44:32.089 INFO:tasks.workunit.client.1.smithi171.stdout:8/701: dread f2 [0,4194304] 0 2022-01-31T19:44:32.091 INFO:tasks.workunit.client.1.smithi171.stdout:6/774: mkdir dc/d19/d30/d3c/d105 0 2022-01-31T19:44:32.091 INFO:tasks.workunit.client.1.smithi171.stdout:6/775: creat dc/d19/d29/d6a/f106 x:0 0 0 2022-01-31T19:44:32.091 INFO:tasks.workunit.client.1.smithi171.stdout:6/776: fdatasync dc/d18/fe2 0 2022-01-31T19:44:32.092 INFO:tasks.workunit.client.1.smithi171.stdout:0/831: mknod d9/d24/d92/c134 0 2022-01-31T19:44:32.092 INFO:tasks.workunit.client.1.smithi171.stdout:2/978: mknod d7/d9/c129 0 2022-01-31T19:44:32.093 INFO:tasks.workunit.client.1.smithi171.stdout:8/702: symlink df/d27/d90/led 0 2022-01-31T19:44:32.094 INFO:tasks.workunit.client.1.smithi171.stdout:8/703: fdatasync df/d18/d19/d36/d70/f9e 0 2022-01-31T19:44:32.094 INFO:tasks.workunit.client.1.smithi171.stdout:8/704: chown df/d18/f41 0 1 2022-01-31T19:44:32.095 INFO:tasks.workunit.client.1.smithi171.stdout:6/777: write dc/d19/d29/daf/fef [1441449,82867] 0 2022-01-31T19:44:32.095 INFO:tasks.workunit.client.1.smithi171.stdout:3/942: link d4/dd/d27/d28/d35/f38 d4/d21/de5/d92/dab/dd4/f135 0 2022-01-31T19:44:32.095 INFO:tasks.workunit.client.1.smithi171.stdout:6/778: creat dc/d19/dae/f107 x:0 0 0 2022-01-31T19:44:32.095 INFO:tasks.workunit.client.1.smithi171.stdout:6/779: write dc/d19/d29/daf/ff5 [29462,8124] 0 2022-01-31T19:44:32.096 INFO:tasks.workunit.client.1.smithi171.stdout:6/780: creat dc/d19/d29/da7/f108 x:0 0 0 2022-01-31T19:44:32.096 INFO:tasks.workunit.client.1.smithi171.stdout:6/781: truncate dc/d19/d29/da7/f108 784118 0 2022-01-31T19:44:32.097 INFO:tasks.workunit.client.1.smithi171.stdout:6/782: write dc/d19/d29/daf/ddb/fcd [4376828,37037] 0 2022-01-31T19:44:32.097 INFO:tasks.workunit.client.1.smithi171.stdout:3/943: symlink d4/d21/d3a/db5/l136 0 2022-01-31T19:44:32.097 INFO:tasks.workunit.client.1.smithi171.stdout:3/944: dread - d4/d21/d3a/d59/dc1/fd7 zero size 2022-01-31T19:44:32.098 INFO:tasks.workunit.client.1.smithi171.stdout:1/803: dwrite dd/d23/d2e/d5e/fdd [0,4194304] 0 2022-01-31T19:44:32.099 INFO:tasks.workunit.client.1.smithi171.stdout:6/783: mkdir dc/d19/d30/d3c/d105/d109 0 2022-01-31T19:44:32.101 INFO:tasks.workunit.client.1.smithi171.stdout:3/945: rename d4/d21/de5/d92/dab/d52/d7a/d108 to d4/dd/d73/d137 0 2022-01-31T19:44:32.102 INFO:tasks.workunit.client.1.smithi171.stdout:8/705: dread df/d18/d19/d36/d70/d7e/f8a [0,4194304] 0 2022-01-31T19:44:32.102 INFO:tasks.workunit.client.1.smithi171.stdout:1/804: getdents dd/d23/d2e/d5e/d83 0 2022-01-31T19:44:32.104 INFO:tasks.workunit.client.1.smithi171.stdout:6/784: creat dc/d19/d29/d27/d37/d68/f10a x:0 0 0 2022-01-31T19:44:32.104 INFO:tasks.workunit.client.1.smithi171.stdout:6/785: write dc/d19/d29/daf/ff5 [336563,62606] 0 2022-01-31T19:44:32.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:31 smithi171 conmon[35325]: cluster 2022-01-31T19:44:30.436623+0000 mgr.smithi171.asyxnc (mgr.24387 2022-01-31T19:44:32.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:31 smithi171 conmon[35325]: ) 36 : cluster [DBG] pgmap v24: 65 pgs: 65 active+clean; 3.4 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 142 MiB/s rd, 305 MiB/s wr, 467 op/s 2022-01-31T19:44:32.108 INFO:tasks.workunit.client.1.smithi171.stdout:8/706: mknod df/d5d/d6c/d87/de1/cee 0 2022-01-31T19:44:32.109 INFO:tasks.workunit.client.1.smithi171.stdout:6/786: truncate dc/fe3 454303 0 2022-01-31T19:44:32.109 INFO:tasks.workunit.client.1.smithi171.stdout:8/707: mkdir df/d18/d19/d21/def 0 2022-01-31T19:44:32.111 INFO:tasks.workunit.client.1.smithi171.stdout:8/708: creat df/d18/d19/db2/ff0 x:0 0 0 2022-01-31T19:44:32.112 INFO:tasks.workunit.client.1.smithi171.stdout:3/946: dread d4/d21/f31 [0,4194304] 0 2022-01-31T19:44:32.112 INFO:tasks.workunit.client.1.smithi171.stdout:3/947: readlink d4/d21/d3a/db5/l136 0 2022-01-31T19:44:32.112 INFO:tasks.workunit.client.1.smithi171.stdout:3/948: dread - d4/d21/de5/d92/dab/d60/df4/f11a zero size 2022-01-31T19:44:32.113 INFO:tasks.workunit.client.1.smithi171.stdout:6/787: write dc/d19/d29/fec [1951107,54346] 0 2022-01-31T19:44:32.113 INFO:tasks.workunit.client.1.smithi171.stdout:0/832: dwrite d9/db/dc2/f119 [0,4194304] 0 2022-01-31T19:44:32.113 INFO:tasks.workunit.client.1.smithi171.stdout:8/709: mkdir df/d5d/d6c/d87/df1 0 2022-01-31T19:44:32.113 INFO:tasks.workunit.client.1.smithi171.stdout:8/710: truncate df/d2a/d30/f63 89777 0 2022-01-31T19:44:32.114 INFO:tasks.workunit.client.1.smithi171.stdout:3/949: creat d4/d21/de5/d92/dab/dd4/f138 x:0 0 0 2022-01-31T19:44:32.115 INFO:tasks.workunit.client.1.smithi171.stdout:3/950: read d4/d21/de5/d92/dab/d60/fbd [301832,15317] 0 2022-01-31T19:44:32.122 INFO:tasks.workunit.client.1.smithi171.stdout:0/833: mknod d9/db/dc2/c135 0 2022-01-31T19:44:32.125 INFO:tasks.workunit.client.1.smithi171.stdout:6/788: rename dc/d21/f8d to dc/d8b/dee/f10b 0 2022-01-31T19:44:32.125 INFO:tasks.workunit.client.1.smithi171.stdout:6/789: chown dc/d19/d29/da7/df0 31428 1 2022-01-31T19:44:32.125 INFO:tasks.workunit.client.1.smithi171.stdout:6/790: readlink dc/d19/l75 0 2022-01-31T19:44:32.125 INFO:tasks.workunit.client.1.smithi171.stdout:6/791: creat dc/f10c x:0 0 0 2022-01-31T19:44:32.126 INFO:tasks.workunit.client.1.smithi171.stdout:6/792: fsync dc/d19/d30/d3c/d80/fd4 0 2022-01-31T19:44:32.126 INFO:tasks.workunit.client.1.smithi171.stdout:6/793: write dc/d8b/dee/f101 [789398,15389] 0 2022-01-31T19:44:32.126 INFO:tasks.workunit.client.1.smithi171.stdout:1/805: dwrite dd/df/d84/dea/d64/db9/fd2 [4194304,4194304] 0 2022-01-31T19:44:32.127 INFO:tasks.workunit.client.1.smithi171.stdout:2/979: dwrite d7/dd/d5e/f9c [0,4194304] 0 2022-01-31T19:44:32.127 INFO:tasks.workunit.client.1.smithi171.stdout:8/711: creat df/d27/d90/ff2 x:0 0 0 2022-01-31T19:44:32.128 INFO:tasks.workunit.client.1.smithi171.stdout:3/951: mkdir d4/d21/de5/d92/dab/dd4/d139 0 2022-01-31T19:44:32.129 INFO:tasks.workunit.client.1.smithi171.stdout:0/834: link d9/d24/d4c/d35/d67/df4/l12c d9/db/dc/d3f/db9/d7a/l136 0 2022-01-31T19:44:32.130 INFO:tasks.workunit.client.1.smithi171.stdout:0/835: dread - d9/db/d59/de9/f117 zero size 2022-01-31T19:44:32.130 INFO:tasks.workunit.client.1.smithi171.stdout:0/836: creat d9/d24/d92/d9d/dc6/f137 x:0 0 0 2022-01-31T19:44:32.132 INFO:tasks.workunit.client.1.smithi171.stdout:6/794: symlink dc/d18/l10d 0 2022-01-31T19:44:32.136 INFO:tasks.workunit.client.1.smithi171.stdout:1/806: rename dd/d23/d74/fb4 to dd/df/d84/dea/d64/da7/fef 0 2022-01-31T19:44:32.137 INFO:tasks.workunit.client.1.smithi171.stdout:1/807: dread - dd/df/d19/d36/fab zero size 2022-01-31T19:44:32.137 INFO:tasks.workunit.client.1.smithi171.stdout:1/808: chown f3 9 1 2022-01-31T19:44:32.137 INFO:tasks.workunit.client.1.smithi171.stdout:1/809: dread dd/d23/f41 [0,4194304] 0 2022-01-31T19:44:32.138 INFO:tasks.workunit.client.1.smithi171.stdout:1/810: chown dd/d23/d2e/d5e/f8a 18844900 1 2022-01-31T19:44:32.150 INFO:tasks.workunit.client.1.smithi171.stdout:2/980: symlink d7/dd/d5e/d6b/d81/d8d/da2/d117/l12a 0 2022-01-31T19:44:32.152 INFO:tasks.workunit.client.1.smithi171.stdout:0/837: creat d9/db/dc2/f138 x:0 0 0 2022-01-31T19:44:32.153 INFO:tasks.workunit.client.1.smithi171.stdout:0/838: dread - d9/d24/d92/d9d/dc6/d115/f130 zero size 2022-01-31T19:44:32.154 INFO:tasks.workunit.client.1.smithi171.stdout:8/712: rename df/d27/d7b/dca to df/d18/d19/d21/d67/df3 0 2022-01-31T19:44:32.155 INFO:tasks.workunit.client.1.smithi171.stdout:1/811: mkdir dd/df/d84/dea/d35/df0 0 2022-01-31T19:44:32.155 INFO:tasks.workunit.client.1.smithi171.stdout:1/812: truncate dd/d23/d34/d4b/fe5 6099642 0 2022-01-31T19:44:32.158 INFO:tasks.workunit.client.1.smithi171.stdout:2/981: rename d7/dd/d5e/d6b/d81/fad to d7/dd/d5e/d6b/d81/f12b 0 2022-01-31T19:44:32.159 INFO:tasks.workunit.client.1.smithi171.stdout:6/795: truncate dc/d8b/dee/f101 792103 0 2022-01-31T19:44:32.161 INFO:tasks.workunit.client.1.smithi171.stdout:3/952: rename d4/d21/d3a/db5/l136 to d4/d21/d3a/ddd/l13a 0 2022-01-31T19:44:32.162 INFO:tasks.workunit.client.1.smithi171.stdout:8/713: creat df/d5d/d6c/d87/ff4 x:0 0 0 2022-01-31T19:44:32.164 INFO:tasks.workunit.client.1.smithi171.stdout:1/813: creat dd/d23/d2e/d5e/d81/d98/d80/dee/ff1 x:0 0 0 2022-01-31T19:44:32.164 INFO:tasks.workunit.client.1.smithi171.stdout:1/814: write dd/df/d84/dea/d64/da7/fef [920277,117927] 0 2022-01-31T19:44:32.167 INFO:tasks.workunit.client.1.smithi171.stdout:8/714: mkdir df/d27/d85/df5 0 2022-01-31T19:44:32.169 INFO:tasks.workunit.client.1.smithi171.stdout:1/815: mkdir dd/df/d84/dea/d35/dd6/df2 0 2022-01-31T19:44:32.169 INFO:tasks.workunit.client.1.smithi171.stdout:1/816: fdatasync dd/d23/d2e/d5e/fdd 0 2022-01-31T19:44:32.172 INFO:tasks.workunit.client.1.smithi171.stdout:0/839: rename d9/d24/d4c/d35/d67/l7b to d9/db/dc/d3f/l139 0 2022-01-31T19:44:32.173 INFO:tasks.workunit.client.1.smithi171.stdout:8/715: unlink df/d2a/c2f 0 2022-01-31T19:44:32.175 INFO:tasks.workunit.client.1.smithi171.stdout:0/840: unlink d9/db/d59/fc3 0 2022-01-31T19:44:32.175 INFO:tasks.workunit.client.1.smithi171.stdout:0/841: truncate d9/d24/d4c/d35/d67/f112 1019560 0 2022-01-31T19:44:32.176 INFO:tasks.workunit.client.1.smithi171.stdout:8/716: mkdir df/d18/d19/d36/d70/df6 0 2022-01-31T19:44:32.177 INFO:tasks.workunit.client.1.smithi171.stdout:1/817: read dd/d23/d2e/d5e/f8a [1632782,107848] 0 2022-01-31T19:44:32.178 INFO:tasks.workunit.client.1.smithi171.stdout:0/842: rmdir d9/db/dc/d3f/db9/d7a 39 2022-01-31T19:44:32.179 INFO:tasks.workunit.client.1.smithi171.stdout:5/724: sync 2022-01-31T19:44:32.179 INFO:tasks.workunit.client.1.smithi171.stdout:9/787: sync 2022-01-31T19:44:32.179 INFO:tasks.workunit.client.1.smithi171.stdout:9/788: stat dd/df/d15/d38/d41/f76 0 2022-01-31T19:44:32.180 INFO:tasks.workunit.client.1.smithi171.stdout:9/789: write f9 [2150665,19615] 0 2022-01-31T19:44:32.180 INFO:tasks.workunit.client.1.smithi171.stdout:9/790: creat dd/df/d3f/d73/d96/f107 x:0 0 0 2022-01-31T19:44:32.180 INFO:tasks.workunit.client.1.smithi171.stdout:9/791: chown dd/df/d3f/d73/d96/d9f/de6/db5/ldc 1 1 2022-01-31T19:44:32.181 INFO:tasks.workunit.client.1.smithi171.stdout:1/818: creat dd/ff3 x:0 0 0 2022-01-31T19:44:32.181 INFO:tasks.workunit.client.1.smithi171.stdout:5/725: creat d5/d90/ff3 x:0 0 0 2022-01-31T19:44:32.181 INFO:tasks.workunit.client.1.smithi171.stdout:5/726: stat d5/d90/db7/deb 0 2022-01-31T19:44:32.182 INFO:tasks.workunit.client.1.smithi171.stdout:8/717: read df/d18/f55 [1205155,99064] 0 2022-01-31T19:44:32.183 INFO:tasks.workunit.client.1.smithi171.stdout:0/843: dread d9/db/d59/d71/fe2 [0,4194304] 0 2022-01-31T19:44:32.183 INFO:tasks.workunit.client.1.smithi171.stdout:9/792: mkdir dd/df/d3f/d73/d96/d9f/de6/d7c/dcf/d108 0 2022-01-31T19:44:32.183 INFO:tasks.workunit.client.1.smithi171.stdout:9/793: write dd/df/d15/d1a/f2b [5048412,48204] 0 2022-01-31T19:44:32.184 INFO:tasks.workunit.client.1.smithi171.stdout:1/819: link dd/d23/d2e/d5e/fe0 dd/d23/d2e/d5e/d81/d98/d80/dee/ff4 0 2022-01-31T19:44:32.185 INFO:tasks.workunit.client.1.smithi171.stdout:5/727: rename d5/d90/db7/d15/d23/d58/d9e/lb4 to d5/d90/db7/deb/lf4 0 2022-01-31T19:44:32.186 INFO:tasks.workunit.client.1.smithi171.stdout:0/844: truncate d9/db/dc2/f52 307361 0 2022-01-31T19:44:32.187 INFO:tasks.workunit.client.1.smithi171.stdout:0/845: fsync d9/db/d22/fd1 0 2022-01-31T19:44:32.187 INFO:tasks.workunit.client.1.smithi171.stdout:1/820: mkdir dd/d23/d2e/d5e/d81/d98/dad/df5 0 2022-01-31T19:44:32.188 INFO:tasks.workunit.client.1.smithi171.stdout:3/953: dwrite d4/dd/d27/f106 [0,4194304] 0 2022-01-31T19:44:32.188 INFO:tasks.workunit.client.1.smithi171.stdout:3/954: write d4/dd/d27/d6e/fef [862525,101276] 0 2022-01-31T19:44:32.189 INFO:tasks.workunit.client.1.smithi171.stdout:2/982: dwrite d7/dd/dfc/d101/d126/de5/fc2 [4194304,4194304] 0 2022-01-31T19:44:32.195 INFO:tasks.workunit.client.1.smithi171.stdout:5/728: mknod d5/d90/db7/d15/d23/d58/d9e/cf5 0 2022-01-31T19:44:32.195 INFO:tasks.workunit.client.1.smithi171.stdout:0/846: symlink d9/d24/d4c/d7d/l13a 0 2022-01-31T19:44:32.196 INFO:tasks.workunit.client.1.smithi171.stdout:1/821: creat dd/d23/d2e/d5e/ff6 x:0 0 0 2022-01-31T19:44:32.203 INFO:tasks.workunit.client.1.smithi171.stdout:0/847: write d9/d24/d92/d9d/d9f/fac [653724,17912] 0 2022-01-31T19:44:32.203 INFO:tasks.workunit.client.1.smithi171.stdout:3/955: unlink d4/d21/d3a/d59/c68 0 2022-01-31T19:44:32.204 INFO:tasks.workunit.client.1.smithi171.stdout:5/729: creat d5/db0/ff6 x:0 0 0 2022-01-31T19:44:32.204 INFO:tasks.workunit.client.1.smithi171.stdout:5/730: chown d5/d90/db7/dd/f20 50 1 2022-01-31T19:44:32.205 INFO:tasks.workunit.client.1.smithi171.stdout:8/718: dwrite df/d18/d19/d46/f5e [0,4194304] 0 2022-01-31T19:44:32.208 INFO:tasks.workunit.client.1.smithi171.stdout:1/822: dread dd/f7f [0,4194304] 0 2022-01-31T19:44:32.208 INFO:tasks.workunit.client.1.smithi171.stdout:0/848: mknod d9/d24/d4c/d7d/c13b 0 2022-01-31T19:44:32.209 INFO:tasks.workunit.client.1.smithi171.stdout:0/849: truncate d9/db/d59/fed 477852 0 2022-01-31T19:44:32.209 INFO:tasks.workunit.client.1.smithi171.stdout:3/956: mknod d4/dd/d87/c13b 0 2022-01-31T19:44:32.210 INFO:tasks.workunit.client.1.smithi171.stdout:5/731: symlink d5/d90/db7/d15/d23/d67/dd8/lf7 0 2022-01-31T19:44:32.213 INFO:tasks.workunit.client.1.smithi171.stdout:1/823: symlink dd/df/d84/dea/d35/df0/lf7 0 2022-01-31T19:44:32.214 INFO:tasks.workunit.client.1.smithi171.stdout:0/850: rename d9/db/dc/d3f/db9/d7a/c9c to d9/d24/d92/d9d/dc6/dfb/c13c 0 2022-01-31T19:44:32.215 INFO:tasks.workunit.client.1.smithi171.stdout:5/732: mkdir d5/d90/db7/d15/d23/d6b/dae/df8 0 2022-01-31T19:44:32.217 INFO:tasks.workunit.client.1.smithi171.stdout:1/824: rmdir dd/d23 39 2022-01-31T19:44:32.219 INFO:tasks.workunit.client.1.smithi171.stdout:5/733: rename d5/d90/db7/d15/d23/d67/f4d to d5/d90/db7/d15/d80/d8d/ff9 0 2022-01-31T19:44:32.220 INFO:tasks.workunit.client.1.smithi171.stdout:8/719: getdents df/d18/d19/d46/dbc 0 2022-01-31T19:44:32.220 INFO:tasks.workunit.client.1.smithi171.stdout:9/794: dwrite dd/df/d15/d38/d41/fd7 [0,4194304] 0 2022-01-31T19:44:32.221 INFO:tasks.workunit.client.1.smithi171.stdout:9/795: truncate dd/df/d15/f3a 4720580 0 2022-01-31T19:44:32.221 INFO:tasks.workunit.client.1.smithi171.stdout:1/825: dread dd/df/d19/dce/fd9 [0,4194304] 0 2022-01-31T19:44:32.222 INFO:tasks.workunit.client.1.smithi171.stdout:2/983: dwrite d7/dd/dfc/d101/d126/de5/fb1 [0,4194304] 0 2022-01-31T19:44:32.222 INFO:tasks.workunit.client.1.smithi171.stdout:5/734: link d5/d90/db7/d15/d23/d58/lef d5/lfa 0 2022-01-31T19:44:32.222 INFO:tasks.workunit.client.1.smithi171.stdout:5/735: truncate d5/d90/db7/d15/fb8 1221337 0 2022-01-31T19:44:32.222 INFO:tasks.workunit.client.1.smithi171.stdout:5/736: fdatasync d5/d59/da0/fa7 0 2022-01-31T19:44:32.223 INFO:tasks.workunit.client.1.smithi171.stdout:8/720: rename df/d18/d19/d82/c94 to df/d1a/dae/cf7 0 2022-01-31T19:44:32.224 INFO:tasks.workunit.client.1.smithi171.stdout:8/721: dread - df/d18/fce zero size 2022-01-31T19:44:32.228 INFO:tasks.workunit.client.1.smithi171.stdout:1/826: write dd/d23/d2e/f2f [11264066,69079] 0 2022-01-31T19:44:32.232 INFO:tasks.workunit.client.1.smithi171.stdout:2/984: creat d7/dd/dfc/d101/d126/d3d/d40/dd2/d115/d116/da4/f12c x:0 0 0 2022-01-31T19:44:32.233 INFO:tasks.workunit.client.1.smithi171.stdout:5/737: rmdir d5 39 2022-01-31T19:44:32.233 INFO:tasks.workunit.client.1.smithi171.stdout:5/738: write d5/d90/db7/d15/d23/f9a [661378,19441] 0 2022-01-31T19:44:32.234 INFO:tasks.workunit.client.1.smithi171.stdout:2/985: write d7/dd/f52 [1270540,88565] 0 2022-01-31T19:44:32.238 INFO:tasks.workunit.client.1.smithi171.stdout:3/957: dwrite d4/dd/d27/d4e/d62/f82 [0,4194304] 0 2022-01-31T19:44:32.242 INFO:tasks.workunit.client.1.smithi171.stdout:5/739: rmdir d5/d90/db7/d15/d5d/d6f 39 2022-01-31T19:44:32.242 INFO:tasks.workunit.client.1.smithi171.stdout:5/740: chown d5/d90/db7/dd/f35 3333786 1 2022-01-31T19:44:32.244 INFO:tasks.workunit.client.1.smithi171.stdout:0/851: dwrite d9/f102 [0,4194304] 0 2022-01-31T19:44:32.246 INFO:tasks.workunit.client.1.smithi171.stdout:1/827: dwrite dd/d23/d2e/d5e/d83/fa0 [0,4194304] 0 2022-01-31T19:44:32.247 INFO:tasks.workunit.client.1.smithi171.stdout:9/796: rename dd/df/d3f/d73/d96/d9f to dd/d109 0 2022-01-31T19:44:32.248 INFO:tasks.workunit.client.1.smithi171.stdout:9/797: creat dd/df/d15/d38/d4d/f10a x:0 0 0 2022-01-31T19:44:32.248 INFO:tasks.workunit.client.1.smithi171.stdout:9/798: truncate dd/df/d3f/d73/fea 985413 0 2022-01-31T19:44:32.248 INFO:tasks.workunit.client.1.smithi171.stdout:8/722: symlink df/d18/d19/d46/dd6/lf8 0 2022-01-31T19:44:32.249 INFO:tasks.workunit.client.1.smithi171.stdout:3/958: mkdir d4/dd/d27/d13c 0 2022-01-31T19:44:32.250 INFO:tasks.workunit.client.1.smithi171.stdout:3/959: write d4/d21/de5/d92/dab/d52/f54 [7833356,21207] 0 2022-01-31T19:44:32.251 INFO:tasks.workunit.client.1.smithi171.stdout:0/852: mknod d9/db/d59/d71/d98/c13d 0 2022-01-31T19:44:32.252 INFO:tasks.workunit.client.1.smithi171.stdout:1/828: creat dd/d23/d2e/d5e/d81/d98/ff8 x:0 0 0 2022-01-31T19:44:32.253 INFO:tasks.workunit.client.1.smithi171.stdout:2/986: rename d7/dd/f39 to d7/d7c/d83/d85/f12d 0 2022-01-31T19:44:32.259 INFO:tasks.workunit.client.1.smithi171.stdout:3/960: dread d4/dd/d27/d6e/fef [0,4194304] 0 2022-01-31T19:44:32.261 INFO:tasks.workunit.client.1.smithi171.stdout:3/961: write d4/d21/de5/d92/f85 [528557,111253] 0 2022-01-31T19:44:32.261 INFO:tasks.workunit.client.1.smithi171.stdout:9/799: link dd/df/f17 dd/df/d3f/d73/d96/f10b 0 2022-01-31T19:44:32.261 INFO:tasks.workunit.client.1.smithi171.stdout:9/800: readlink dd/df/d15/l45 0 2022-01-31T19:44:32.262 INFO:tasks.workunit.client.1.smithi171.stdout:8/723: creat df/d1a/ff9 x:0 0 0 2022-01-31T19:44:32.263 INFO:tasks.workunit.client.1.smithi171.stdout:1/829: creat dd/df/d19/dce/ff9 x:0 0 0 2022-01-31T19:44:32.264 INFO:tasks.workunit.client.1.smithi171.stdout:1/830: fsync dd/d23/d2e/f86 0 2022-01-31T19:44:32.266 INFO:tasks.workunit.client.1.smithi171.stdout:8/724: dread df/d27/fd5 [0,4194304] 0 2022-01-31T19:44:32.266 INFO:tasks.workunit.client.1.smithi171.stdout:5/741: rename d5/d90/db7/d15/d80/d8d/ce6 to d5/d59/da0/cfb 0 2022-01-31T19:44:32.274 INFO:tasks.workunit.client.1.smithi171.stdout:8/725: link df/d5d/d6c/fb3 df/d27/d7b/ffa 0 2022-01-31T19:44:32.275 INFO:tasks.workunit.client.1.smithi171.stdout:0/853: dwrite d9/db/d22/f5b [0,4194304] 0 2022-01-31T19:44:32.276 INFO:tasks.workunit.client.1.smithi171.stdout:2/987: rename d7/dd/dfc/d101/d126/d3d/d40/c10d to d7/dd/dfc/d101/d126/d3d/d40/dd2/d115/d116/c12e 0 2022-01-31T19:44:32.278 INFO:tasks.workunit.client.1.smithi171.stdout:9/801: rename dd/l18 to dd/df/d3f/d73/d96/l10c 0 2022-01-31T19:44:32.284 INFO:tasks.workunit.client.1.smithi171.stdout:5/742: dread d5/d90/db7/d15/d23/d58/f91 [0,4194304] 0 2022-01-31T19:44:32.285 INFO:tasks.workunit.client.1.smithi171.stdout:2/988: mkdir d7/dd/dfc/d101/d126/d3d/d40/dd2/d12f 0 2022-01-31T19:44:32.289 INFO:tasks.workunit.client.1.smithi171.stdout:0/854: dread d9/db/fee [0,4194304] 0 2022-01-31T19:44:32.292 INFO:tasks.workunit.client.1.smithi171.stdout:1/831: rename dd/f7a to dd/df/ffa 0 2022-01-31T19:44:32.293 INFO:tasks.workunit.client.1.smithi171.stdout:1/832: mkdir dd/df/d84/dea/d35/dfb 0 2022-01-31T19:44:32.294 INFO:tasks.workunit.client.1.smithi171.stdout:0/855: mknod d9/db/dc/d3f/c13e 0 2022-01-31T19:44:32.295 INFO:tasks.workunit.client.1.smithi171.stdout:1/833: symlink dd/df/d84/dea/d64/da7/lfc 0 2022-01-31T19:44:32.297 INFO:tasks.workunit.client.1.smithi171.stdout:0/856: write d9/f3b [2350164,112611] 0 2022-01-31T19:44:32.298 INFO:tasks.workunit.client.1.smithi171.stdout:0/857: unlink d9/d24/f133 0 2022-01-31T19:44:32.298 INFO:tasks.workunit.client.1.smithi171.stdout:0/858: write d9/d24/fb3 [805295,4231] 0 2022-01-31T19:44:32.299 INFO:tasks.workunit.client.1.smithi171.stdout:0/859: mkdir d9/d24/d92/d9d/dc6/dfb/d13f 0 2022-01-31T19:44:32.300 INFO:tasks.workunit.client.1.smithi171.stdout:0/860: fdatasync d9/db/d59/de9/f117 0 2022-01-31T19:44:32.300 INFO:tasks.workunit.client.1.smithi171.stdout:0/861: chown d9/db/d59/fe6 3 1 2022-01-31T19:44:32.301 INFO:tasks.workunit.client.1.smithi171.stdout:0/862: symlink d9/d24/d92/d9d/d11a/l140 0 2022-01-31T19:44:32.301 INFO:tasks.workunit.client.1.smithi171.stdout:0/863: readlink d9/lf0 0 2022-01-31T19:44:32.302 INFO:tasks.workunit.client.1.smithi171.stdout:0/864: mknod d9/db/dc2/c141 0 2022-01-31T19:44:32.302 INFO:tasks.workunit.client.1.smithi171.stdout:0/865: mknod d9/c142 0 2022-01-31T19:44:32.305 INFO:tasks.workunit.client.1.smithi171.stdout:1/834: dread dd/df/d19/d1b/f4e [0,4194304] 0 2022-01-31T19:44:32.306 INFO:tasks.workunit.client.1.smithi171.stdout:1/835: stat dd/d23/d2e/d5e/fe0 0 2022-01-31T19:44:32.318 INFO:tasks.workunit.client.1.smithi171.stdout:8/726: dwrite df/d18/da4/fb8 [0,4194304] 0 2022-01-31T19:44:32.319 INFO:tasks.workunit.client.1.smithi171.stdout:9/802: dwrite dd/df/d15/d38/d41/f76 [0,4194304] 0 2022-01-31T19:44:32.323 INFO:tasks.workunit.client.1.smithi171.stdout:9/803: creat dd/df/d15/d1a/d64/d6e/ddd/f10d x:0 0 0 2022-01-31T19:44:32.324 INFO:tasks.workunit.client.1.smithi171.stdout:9/804: rename dd/df/f12 to dd/df/d15/de4/f10e 0 2022-01-31T19:44:32.327 INFO:tasks.workunit.client.1.smithi171.stdout:5/743: dwrite d5/d90/db7/d15/d23/d6b/fc0 [0,4194304] 0 2022-01-31T19:44:32.329 INFO:tasks.workunit.client.1.smithi171.stdout:2/989: dwrite d7/dd/d5e/d6b/d81/f9b [0,4194304] 0 2022-01-31T19:44:32.329 INFO:tasks.workunit.client.1.smithi171.stdout:5/744: mkdir d5/d90/dd1/dfc 0 2022-01-31T19:44:32.331 INFO:tasks.workunit.client.1.smithi171.stdout:5/745: write d5/d90/db7/d15/d23/f87 [3423418,784] 0 2022-01-31T19:44:32.331 INFO:tasks.workunit.client.1.smithi171.stdout:5/746: chown d5/d59/da0/cfb 0 1 2022-01-31T19:44:32.331 INFO:tasks.workunit.client.1.smithi171.stdout:2/990: read d7/f56 [1031255,125161] 0 2022-01-31T19:44:32.333 INFO:tasks.workunit.client.1.smithi171.stdout:1/836: dwrite dd/d23/d2e/d5e/d81/d98/d80/dee/ff4 [0,4194304] 0 2022-01-31T19:44:32.334 INFO:tasks.workunit.client.1.smithi171.stdout:2/991: rename d7/dd/d5e/d6b/d81/fa8 to d7/dd/dfc/d101/d126/de5/ddc/f130 0 2022-01-31T19:44:32.335 INFO:tasks.workunit.client.1.smithi171.stdout:2/992: write d7/d7c/d83/fe0 [165846,113943] 0 2022-01-31T19:44:32.339 INFO:tasks.workunit.client.1.smithi171.stdout:2/993: creat d7/dd/dfc/d101/d126/d3d/d40/dd2/d115/d116/da4/f131 x:0 0 0 2022-01-31T19:44:32.343 INFO:tasks.workunit.client.1.smithi171.stdout:9/805: dread dd/d109/de6/f29 [0,4194304] 0 2022-01-31T19:44:32.346 INFO:tasks.workunit.client.1.smithi171.stdout:9/806: chown dd/df/cc9 11022 1 2022-01-31T19:44:32.347 INFO:tasks.workunit.client.1.smithi171.stdout:9/807: dread - dd/df/d15/da2/fd9 zero size 2022-01-31T19:44:32.347 INFO:tasks.workunit.client.1.smithi171.stdout:9/808: rename dd/df/d15/c8c to dd/df/d15/da2/c10f 0 2022-01-31T19:44:32.348 INFO:tasks.workunit.client.1.smithi171.stdout:9/809: chown dd/df/d15/de4 1145779 1 2022-01-31T19:44:32.348 INFO:tasks.workunit.client.1.smithi171.stdout:9/810: dread - dd/df/d15/d38/d4d/ffd zero size 2022-01-31T19:44:32.350 INFO:tasks.workunit.client.1.smithi171.stdout:9/811: mkdir dd/df/d15/d1a/d64/d6e/ddd/d110 0 2022-01-31T19:44:32.351 INFO:tasks.workunit.client.1.smithi171.stdout:8/727: dwrite df/f84 [4194304,4194304] 0 2022-01-31T19:44:32.351 INFO:tasks.workunit.client.1.smithi171.stdout:5/747: dread d5/d90/db7/d15/d5d/fd0 [0,4194304] 0 2022-01-31T19:44:32.353 INFO:tasks.workunit.client.1.smithi171.stdout:9/812: link dd/df/d15/d1a/d64/cf3 dd/df/d15/d1a/d64/c111 0 2022-01-31T19:44:32.353 INFO:tasks.workunit.client.1.smithi171.stdout:9/813: stat dd/df/d15/d1a/d84/ce9 0 2022-01-31T19:44:32.353 INFO:tasks.workunit.client.1.smithi171.stdout:9/814: write dd/d109/de6/d7c/d7d/fdb [713523,110733] 0 2022-01-31T19:44:32.354 INFO:tasks.workunit.client.1.smithi171.stdout:1/837: dwrite dd/d23/d2e/fa5 [0,4194304] 0 2022-01-31T19:44:32.355 INFO:tasks.workunit.client.1.smithi171.stdout:8/728: mkdir df/d2a/dfb 0 2022-01-31T19:44:32.357 INFO:tasks.workunit.client.1.smithi171.stdout:5/748: write d5/d90/db7/d15/f1d [2203514,47534] 0 2022-01-31T19:44:32.357 INFO:tasks.workunit.client.1.smithi171.stdout:5/749: fdatasync d5/d90/db7/d15/d23/f41 0 2022-01-31T19:44:32.357 INFO:tasks.workunit.client.1.smithi171.stdout:9/815: truncate dd/f11 3674803 0 2022-01-31T19:44:32.359 INFO:tasks.workunit.client.1.smithi171.stdout:5/750: write d5/d90/db7/d15/d23/d6b/fc0 [3368644,37514] 0 2022-01-31T19:44:32.360 INFO:tasks.workunit.client.1.smithi171.stdout:1/838: mkdir dd/df/d84/da9/dfd 0 2022-01-31T19:44:32.360 INFO:tasks.workunit.client.1.smithi171.stdout:1/839: truncate f3 4938830 0 2022-01-31T19:44:32.360 INFO:tasks.workunit.client.1.smithi171.stdout:4/685: sync 2022-01-31T19:44:32.361 INFO:tasks.workunit.client.1.smithi171.stdout:4/686: truncate d2/d3/d5/dc/f26 5299947 0 2022-01-31T19:44:32.361 INFO:tasks.workunit.client.1.smithi171.stdout:8/729: getdents df/d18/d19/d21 0 2022-01-31T19:44:32.367 INFO:tasks.workunit.client.1.smithi171.stdout:9/816: mkdir dd/df/d112 0 2022-01-31T19:44:32.369 INFO:tasks.workunit.client.1.smithi171.stdout:5/751: rename d5/d90/db7/d15/d5d/laa to d5/d90/db7/d15/d23/d67/lfd 0 2022-01-31T19:44:32.371 INFO:tasks.workunit.client.1.smithi171.stdout:4/687: mknod d2/d3/d5/d41/dcf/d12/d15/dc4/cec 0 2022-01-31T19:44:32.373 INFO:tasks.workunit.client.1.smithi171.stdout:5/752: dread d5/d90/db7/f5e [0,4194304] 0 2022-01-31T19:44:32.379 INFO:tasks.workunit.client.1.smithi171.stdout:1/840: rmdir dd 39 2022-01-31T19:44:32.380 INFO:tasks.workunit.client.1.smithi171.stdout:4/688: unlink d2/d3/d5/d41/dcf/d12/f45 0 2022-01-31T19:44:32.381 INFO:tasks.workunit.client.1.smithi171.stdout:5/753: mkdir d5/d90/db7/d15/d23/d58/d9e/dd7/dfe 0 2022-01-31T19:44:32.383 INFO:tasks.workunit.client.1.smithi171.stdout:1/841: getdents dd/df/d19/d1b 0 2022-01-31T19:44:32.384 INFO:tasks.workunit.client.1.smithi171.stdout:4/689: mkdir d2/d3/d5/ded 0 2022-01-31T19:44:32.386 INFO:tasks.workunit.client.1.smithi171.stdout:5/754: rename d5/d90/db7/d15/d23/lbe to d5/d90/dd3/lff 0 2022-01-31T19:44:32.387 INFO:tasks.workunit.client.1.smithi171.stdout:4/690: truncate d2/d3/d5/d41/dcf/f62 3950303 0 2022-01-31T19:44:32.389 INFO:tasks.workunit.client.1.smithi171.stdout:5/755: creat d5/d90/db7/d15/d80/d8d/f100 x:0 0 0 2022-01-31T19:44:32.389 INFO:tasks.workunit.client.1.smithi171.stdout:5/756: rename d5/d90 to d5/d90/db7/dd/d101 22 2022-01-31T19:44:32.390 INFO:tasks.workunit.client.1.smithi171.stdout:4/691: mkdir d2/d3/d5/d70/dee 0 2022-01-31T19:44:32.390 INFO:tasks.workunit.client.1.smithi171.stdout:5/757: link d5/d90/db7/deb/lf4 d5/d90/db7/d15/d23/d6b/l102 0 2022-01-31T19:44:32.392 INFO:tasks.workunit.client.1.smithi171.stdout:8/730: dwrite df/d18/d79/fc7 [0,4194304] 0 2022-01-31T19:44:32.393 INFO:tasks.workunit.client.1.smithi171.stdout:5/758: rename d5/d90/db7/d15/d23/d67/lfd to d5/db0/dc8/l103 0 2022-01-31T19:44:32.393 INFO:tasks.workunit.client.1.smithi171.stdout:5/759: chown d5/c5a 111 1 2022-01-31T19:44:32.394 INFO:tasks.workunit.client.1.smithi171.stdout:5/760: chown d5/d90/db7/d15/d80 77020624 1 2022-01-31T19:44:32.398 INFO:tasks.workunit.client.1.smithi171.stdout:9/817: dwrite dd/d109/de6/d7c/d7d/fa3 [4194304,4194304] 0 2022-01-31T19:44:32.399 INFO:tasks.workunit.client.1.smithi171.stdout:1/842: dread dd/d23/d2e/d5e/f45 [0,4194304] 0 2022-01-31T19:44:32.400 INFO:tasks.workunit.client.1.smithi171.stdout:1/843: truncate dd/d23/f60 1003691 0 2022-01-31T19:44:32.410 INFO:tasks.workunit.client.1.smithi171.stdout:4/692: dwrite d2/d3/d5/d41/dcf/d12/d15/dc4/fe0 [0,4194304] 0 2022-01-31T19:44:32.415 INFO:tasks.workunit.client.1.smithi171.stdout:9/818: dread dd/d109/de6/f3d [0,4194304] 0 2022-01-31T19:44:32.416 INFO:tasks.workunit.client.1.smithi171.stdout:8/731: dread df/d18/d79/fc7 [0,4194304] 0 2022-01-31T19:44:32.417 INFO:tasks.workunit.client.1.smithi171.stdout:8/732: chown df/d18/d19/d21/dcd/ccf 14546 1 2022-01-31T19:44:32.417 INFO:tasks.workunit.client.1.smithi171.stdout:1/844: read fa [5626734,59101] 0 2022-01-31T19:44:32.418 INFO:tasks.workunit.client.1.smithi171.stdout:9/819: rename dd/df/cf0 to dd/df/d86/c113 0 2022-01-31T19:44:32.418 INFO:tasks.workunit.client.1.smithi171.stdout:9/820: creat dd/df/d15/d38/d9c/da7/f114 x:0 0 0 2022-01-31T19:44:32.418 INFO:tasks.workunit.client.1.smithi171.stdout:9/821: chown dd/df/d15/d1a/d64/c98 1644254 1 2022-01-31T19:44:32.419 INFO:tasks.workunit.client.1.smithi171.stdout:9/822: readlink dd/df/d15/l1e 0 2022-01-31T19:44:32.422 INFO:tasks.workunit.client.1.smithi171.stdout:1/845: mkdir dd/df/dfe 0 2022-01-31T19:44:32.424 INFO:tasks.workunit.client.1.smithi171.stdout:9/823: mknod dd/d109/de6/da8/c115 0 2022-01-31T19:44:32.424 INFO:tasks.workunit.client.1.smithi171.stdout:9/824: write dd/fc2 [4459174,113344] 0 2022-01-31T19:44:32.425 INFO:tasks.workunit.client.1.smithi171.stdout:8/733: getdents df/d18 0 2022-01-31T19:44:32.428 INFO:tasks.workunit.client.1.smithi171.stdout:1/846: creat dd/df/d19/dce/fff x:0 0 0 2022-01-31T19:44:32.429 INFO:tasks.workunit.client.1.smithi171.stdout:1/847: chown dd/d23/d2e/d5e/d81/d98/dad 249482434 1 2022-01-31T19:44:32.430 INFO:tasks.workunit.client.1.smithi171.stdout:9/825: mknod dd/d109/de6/db5/c116 0 2022-01-31T19:44:32.431 INFO:tasks.workunit.client.1.smithi171.stdout:1/848: rmdir dd/d23/d34 39 2022-01-31T19:44:32.431 INFO:tasks.workunit.client.1.smithi171.stdout:8/734: write df/d18/d19/d36/d70/f9e [1409480,45911] 0 2022-01-31T19:44:32.433 INFO:tasks.workunit.client.1.smithi171.stdout:9/826: mkdir dd/df/d117 0 2022-01-31T19:44:32.433 INFO:tasks.workunit.client.1.smithi171.stdout:8/735: mknod df/d18/d19/d21/dab/cfc 0 2022-01-31T19:44:32.434 INFO:tasks.workunit.client.1.smithi171.stdout:9/827: creat dd/df/d3f/d73/f118 x:0 0 0 2022-01-31T19:44:32.436 INFO:tasks.workunit.client.1.smithi171.stdout:5/761: dwrite d5/d90/db7/d15/d80/fcb [0,4194304] 0 2022-01-31T19:44:32.437 INFO:tasks.workunit.client.1.smithi171.stdout:8/736: dread df/d18/d19/d21/dab/f56 [8388608,4194304] 0 2022-01-31T19:44:32.437 INFO:tasks.workunit.client.1.smithi171.stdout:8/737: write df/d2a/d30/fe4 [912595,45665] 0 2022-01-31T19:44:32.437 INFO:tasks.workunit.client.1.smithi171.stdout:8/738: truncate df/d2a/fec 636837 0 2022-01-31T19:44:32.437 INFO:tasks.workunit.client.1.smithi171.stdout:8/739: readlink df/d18/d19/d36/l6b 0 2022-01-31T19:44:32.438 INFO:tasks.workunit.client.1.smithi171.stdout:4/693: dwrite d2/d3/d5/dc/f39 [0,4194304] 0 2022-01-31T19:44:32.440 INFO:tasks.workunit.client.1.smithi171.stdout:5/762: creat d5/d90/db7/d15/d23/f104 x:0 0 0 2022-01-31T19:44:32.443 INFO:tasks.workunit.client.1.smithi171.stdout:8/740: rename fb to df/d18/d19/d21/d88/ffd 0 2022-01-31T19:44:32.444 INFO:tasks.workunit.client.1.smithi171.stdout:8/741: readlink df/d18/d19/l1f 0 2022-01-31T19:44:32.446 INFO:tasks.workunit.client.1.smithi171.stdout:4/694: truncate d2/d3/d5/d41/dcf/f83 1706664 0 2022-01-31T19:44:32.448 INFO:tasks.workunit.client.1.smithi171.stdout:5/763: rename d5/d90/db7/d15/d23/d67/c6e to d5/d90/db7/d15/d5d/d6a/c105 0 2022-01-31T19:44:32.449 INFO:tasks.workunit.client.1.smithi171.stdout:8/742: mknod df/d27/d85/df5/cfe 0 2022-01-31T19:44:32.450 INFO:tasks.workunit.client.1.smithi171.stdout:4/695: creat d2/d3/d5/d41/dcf/d12/d15/d57/fef x:0 0 0 2022-01-31T19:44:32.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/696: truncate d2/f6d 760674 0 2022-01-31T19:44:32.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/697: write d2/d3/d5/dc/d4b/dbc/fd0 [224229,13783] 0 2022-01-31T19:44:32.451 INFO:tasks.workunit.client.1.smithi171.stdout:5/764: mknod d5/d90/db7/d15/d5d/d6f/c106 0 2022-01-31T19:44:32.452 INFO:tasks.workunit.client.1.smithi171.stdout:5/765: truncate d5/d90/db7/d15/d80/fcb 5110139 0 2022-01-31T19:44:32.452 INFO:tasks.workunit.client.1.smithi171.stdout:5/766: truncate d5/d90/fa8 298703 0 2022-01-31T19:44:32.456 INFO:tasks.workunit.client.1.smithi171.stdout:7/627: sync 2022-01-31T19:44:32.457 INFO:tasks.workunit.client.1.smithi171.stdout:4/698: read d2/d3/f1c [622737,122086] 0 2022-01-31T19:44:32.457 INFO:tasks.workunit.client.1.smithi171.stdout:4/699: chown d2/d3/d5/dc/f26 4566 1 2022-01-31T19:44:32.459 INFO:tasks.workunit.client.1.smithi171.stdout:4/700: mknod d2/d3/d5/d41/dcf/d12/cf0 0 2022-01-31T19:44:32.459 INFO:tasks.workunit.client.1.smithi171.stdout:4/701: chown d2/d3/d5/d41/dcf/lc2 213 1 2022-01-31T19:44:32.460 INFO:tasks.workunit.client.1.smithi171.stdout:4/702: chown d2/d3/d5/dc/d4b/d56 12118193 1 2022-01-31T19:44:32.460 INFO:tasks.workunit.client.1.smithi171.stdout:4/703: write d2/d3/d5/f38 [976257,73978] 0 2022-01-31T19:44:32.461 INFO:tasks.workunit.client.1.smithi171.stdout:4/704: mknod d2/d3/d5/d91/cf1 0 2022-01-31T19:44:32.462 INFO:tasks.workunit.client.1.smithi171.stdout:4/705: link d2/d3/d5/d41/dcf/d12/lb8 d2/d3/d5/dc/lf2 0 2022-01-31T19:44:32.462 INFO:tasks.workunit.client.1.smithi171.stdout:4/706: write d2/d3/de1/dd4/fe3 [143584,55810] 0 2022-01-31T19:44:32.464 INFO:tasks.workunit.client.1.smithi171.stdout:8/743: dread df/d18/d79/f8c [0,4194304] 0 2022-01-31T19:44:32.464 INFO:tasks.workunit.client.1.smithi171.stdout:8/744: stat df/d27/d90/fbb 0 2022-01-31T19:44:32.465 INFO:tasks.workunit.client.1.smithi171.stdout:8/745: fsync df/d18/d19/d21/d88/ffd 0 2022-01-31T19:44:32.465 INFO:tasks.workunit.client.1.smithi171.stdout:8/746: dread - df/d27/d90/ff2 zero size 2022-01-31T19:44:32.465 INFO:tasks.workunit.client.1.smithi171.stdout:8/747: fdatasync df/d18/f4e 0 2022-01-31T19:44:32.466 INFO:tasks.workunit.client.1.smithi171.stdout:4/707: creat d2/d3/d5/d41/dcf/d12/ff3 x:0 0 0 2022-01-31T19:44:32.467 INFO:tasks.workunit.client.1.smithi171.stdout:1/849: dwrite dd/d23/f9e [0,4194304] 0 2022-01-31T19:44:32.467 INFO:tasks.workunit.client.1.smithi171.stdout:9/828: dwrite dd/df/d15/d1a/d64/fab [0,4194304] 0 2022-01-31T19:44:32.471 INFO:tasks.workunit.client.1.smithi171.stdout:1/850: dread dd/df/d84/dea/f4a [4194304,4194304] 0 2022-01-31T19:44:32.473 INFO:tasks.workunit.client.1.smithi171.stdout:8/748: write df/d18/d19/d21/d88/f92 [2508816,111106] 0 2022-01-31T19:44:32.474 INFO:tasks.workunit.client.1.smithi171.stdout:1/851: dread dd/d23/d2e/d5e/d83/fa0 [0,4194304] 0 2022-01-31T19:44:32.476 INFO:tasks.workunit.client.1.smithi171.stdout:1/852: creat dd/df/d84/dea/d35/dd6/df2/f100 x:0 0 0 2022-01-31T19:44:32.477 INFO:tasks.workunit.client.1.smithi171.stdout:1/853: mknod dd/d23/d74/d7b/c101 0 2022-01-31T19:44:32.477 INFO:tasks.workunit.client.1.smithi171.stdout:1/854: chown dd/df/d19/dce/fff 16966378 1 2022-01-31T19:44:32.481 INFO:tasks.workunit.client.1.smithi171.stdout:8/749: dread df/d1a/f3b [0,4194304] 0 2022-01-31T19:44:32.481 INFO:tasks.workunit.client.1.smithi171.stdout:8/750: truncate df/d2a/fec 1514397 0 2022-01-31T19:44:32.482 INFO:tasks.workunit.client.1.smithi171.stdout:9/829: dread dd/df/d15/d38/d9c/fba [0,4194304] 0 2022-01-31T19:44:32.483 INFO:tasks.workunit.client.1.smithi171.stdout:9/830: creat dd/d109/de6/da8/f119 x:0 0 0 2022-01-31T19:44:32.485 INFO:tasks.workunit.client.1.smithi171.stdout:5/767: dwrite d5/d90/fe8 [0,4194304] 0 2022-01-31T19:44:32.486 INFO:tasks.workunit.client.1.smithi171.stdout:9/831: truncate dd/df/d15/f4a 498330 0 2022-01-31T19:44:32.486 INFO:tasks.workunit.client.1.smithi171.stdout:9/832: creat dd/df/d3f/d73/f11a x:0 0 0 2022-01-31T19:44:32.489 INFO:tasks.workunit.client.1.smithi171.stdout:1/855: dread dd/d23/d2e/d5e/d81/d98/f85 [0,4194304] 0 2022-01-31T19:44:32.490 INFO:tasks.workunit.client.1.smithi171.stdout:9/833: mknod dd/d109/de6/d7c/dcf/c11b 0 2022-01-31T19:44:32.490 INFO:tasks.workunit.client.1.smithi171.stdout:1/856: chown dd/df/ffa 4042 1 2022-01-31T19:44:32.491 INFO:tasks.workunit.client.1.smithi171.stdout:8/751: dread df/d27/d85/f8f [0,4194304] 0 2022-01-31T19:44:32.491 INFO:tasks.workunit.client.1.smithi171.stdout:1/857: creat dd/d23/d74/dcb/ddb/f102 x:0 0 0 2022-01-31T19:44:32.492 INFO:tasks.workunit.client.1.smithi171.stdout:4/708: dwrite d2/f14 [0,4194304] 0 2022-01-31T19:44:32.492 INFO:tasks.workunit.client.1.smithi171.stdout:1/858: creat dd/d23/d74/dcb/ddb/f103 x:0 0 0 2022-01-31T19:44:32.494 INFO:tasks.workunit.client.1.smithi171.stdout:8/752: getdents df/d18/d19/d46 0 2022-01-31T19:44:32.497 INFO:tasks.workunit.client.1.smithi171.stdout:5/768: dread d5/d90/db7/dd/f45 [0,4194304] 0 2022-01-31T19:44:32.497 INFO:tasks.workunit.client.1.smithi171.stdout:5/769: fsync d5/d90/db7/dd/d66/f7d 0 2022-01-31T19:44:32.497 INFO:tasks.workunit.client.1.smithi171.stdout:4/709: truncate d2/d3/f28 876757 0 2022-01-31T19:44:32.498 INFO:tasks.workunit.client.1.smithi171.stdout:8/753: symlink df/d18/dad/lff 0 2022-01-31T19:44:32.498 INFO:tasks.workunit.client.1.smithi171.stdout:5/770: mkdir d5/d90/db7/d15/d5d/d107 0 2022-01-31T19:44:32.508 INFO:tasks.workunit.client.1.smithi171.stdout:5/771: write d5/d90/db7/d15/d23/d6b/fc0 [2218418,52172] 0 2022-01-31T19:44:32.510 INFO:tasks.workunit.client.1.smithi171.stdout:5/772: creat d5/d90/db7/d15/d23/d58/d9e/dd7/df0/f108 x:0 0 0 2022-01-31T19:44:32.512 INFO:tasks.workunit.client.1.smithi171.stdout:5/773: mkdir d5/d90/db7/d15/d80/d109 0 2022-01-31T19:44:32.514 INFO:tasks.workunit.client.1.smithi171.stdout:5/774: rmdir d5/d59 39 2022-01-31T19:44:32.514 INFO:tasks.workunit.client.1.smithi171.stdout:5/775: truncate d5/d90/db7/d15/d80/fc3 516002 0 2022-01-31T19:44:32.520 INFO:tasks.workunit.client.1.smithi171.stdout:9/834: dwrite dd/fc2 [0,4194304] 0 2022-01-31T19:44:32.522 INFO:tasks.workunit.client.1.smithi171.stdout:8/754: dwrite df/d2a/d30/fe4 [0,4194304] 0 2022-01-31T19:44:32.522 INFO:tasks.workunit.client.1.smithi171.stdout:8/755: fsync df/d18/d19/d21/f26 0 2022-01-31T19:44:32.522 INFO:tasks.workunit.client.1.smithi171.stdout:8/756: chown df/d1a/dae/de0 2348642 1 2022-01-31T19:44:32.523 INFO:tasks.workunit.client.1.smithi171.stdout:9/835: link fc dd/df/d3f/d73/d96/f11c 0 2022-01-31T19:44:32.523 INFO:tasks.workunit.client.1.smithi171.stdout:9/836: chown dd/df/d15/d53/c58 2028156651 1 2022-01-31T19:44:32.526 INFO:tasks.workunit.client.1.smithi171.stdout:9/837: mkdir dd/df/d86/d11d 0 2022-01-31T19:44:32.526 INFO:tasks.workunit.client.1.smithi171.stdout:9/838: write dd/df/d15/d38/d4d/f63 [1212861,8452] 0 2022-01-31T19:44:32.528 INFO:tasks.workunit.client.1.smithi171.stdout:9/839: unlink dd/df/d15/d1a/d64/d6e/l6f 0 2022-01-31T19:44:32.528 INFO:tasks.workunit.client.1.smithi171.stdout:9/840: truncate dd/df/d15/d1a/d84/f91 915949 0 2022-01-31T19:44:32.529 INFO:tasks.workunit.client.1.smithi171.stdout:8/757: dread df/d18/d19/d21/f26 [0,4194304] 0 2022-01-31T19:44:32.529 INFO:tasks.workunit.client.1.smithi171.stdout:8/758: write df/d18/d19/d46/f8d [455518,87479] 0 2022-01-31T19:44:32.530 INFO:tasks.workunit.client.1.smithi171.stdout:9/841: creat dd/d109/de6/d7c/d7d/db6/f11e x:0 0 0 2022-01-31T19:44:32.530 INFO:tasks.workunit.client.1.smithi171.stdout:9/842: dread - dd/df/d15/d53/fa6 zero size 2022-01-31T19:44:32.530 INFO:tasks.workunit.client.1.smithi171.stdout:9/843: fsync dd/df/d3f/f9e 0 2022-01-31T19:44:32.531 INFO:tasks.workunit.client.1.smithi171.stdout:8/759: creat df/d18/d19/d46/f100 x:0 0 0 2022-01-31T19:44:32.531 INFO:tasks.workunit.client.1.smithi171.stdout:8/760: stat df/d5d/d6c/d87/de1 0 2022-01-31T19:44:32.533 INFO:tasks.workunit.client.1.smithi171.stdout:9/844: symlink dd/df/d15/d53/l11f 0 2022-01-31T19:44:32.533 INFO:tasks.workunit.client.1.smithi171.stdout:9/845: read - dd/df/d15/d38/d9c/da7/ffb zero size 2022-01-31T19:44:32.533 INFO:tasks.workunit.client.1.smithi171.stdout:9/846: chown dd/df/d15/d38/d4d/d80/dfa/l104 700642 1 2022-01-31T19:44:32.533 INFO:tasks.workunit.client.1.smithi171.stdout:8/761: getdents df/d18/d19/d82 0 2022-01-31T19:44:32.534 INFO:tasks.workunit.client.1.smithi171.stdout:5/776: dwrite d5/d90/db7/d15/d23/d58/d9e/dd7/df0/f108 [0,4194304] 0 2022-01-31T19:44:32.534 INFO:tasks.workunit.client.1.smithi171.stdout:9/847: dread dd/f6a [0,4194304] 0 2022-01-31T19:44:32.535 INFO:tasks.workunit.client.1.smithi171.stdout:9/848: write dd/df/d15/d1a/d64/fd0 [353544,28492] 0 2022-01-31T19:44:32.535 INFO:tasks.workunit.client.1.smithi171.stdout:8/762: symlink df/dcc/l101 0 2022-01-31T19:44:32.540 INFO:tasks.workunit.client.1.smithi171.stdout:5/777: write d5/d90/db7/dd/f20 [1944141,90134] 0 2022-01-31T19:44:32.541 INFO:tasks.workunit.client.1.smithi171.stdout:5/778: dread - d5/d90/db7/d15/d23/d58/f79 zero size 2022-01-31T19:44:32.543 INFO:tasks.workunit.client.1.smithi171.stdout:8/763: creat df/d18/d19/d21/dab/f102 x:0 0 0 2022-01-31T19:44:32.545 INFO:tasks.workunit.client.1.smithi171.stdout:8/764: link df/d18/d19/d21/c23 df/d5d/d6c/d87/de1/c103 0 2022-01-31T19:44:32.546 INFO:tasks.workunit.client.1.smithi171.stdout:5/779: getdents d5/d90/db7/d15/d23/d58/d9e/dd7/df0 0 2022-01-31T19:44:32.548 INFO:tasks.workunit.client.1.smithi171.stdout:8/765: mknod df/d18/da4/c104 0 2022-01-31T19:44:32.549 INFO:tasks.workunit.client.1.smithi171.stdout:8/766: creat df/d18/dad/f105 x:0 0 0 2022-01-31T19:44:32.549 INFO:tasks.workunit.client.1.smithi171.stdout:8/767: readlink df/d18/d19/d36/l71 0 2022-01-31T19:44:32.549 INFO:tasks.workunit.client.1.smithi171.stdout:8/768: dread - df/d18/d79/d7a/f93 zero size 2022-01-31T19:44:32.550 INFO:tasks.workunit.client.1.smithi171.stdout:8/769: creat df/d5d/d6c/f106 x:0 0 0 2022-01-31T19:44:32.551 INFO:tasks.workunit.client.1.smithi171.stdout:8/770: mkdir df/d18/d79/da2/d107 0 2022-01-31T19:44:32.553 INFO:tasks.workunit.client.1.smithi171.stdout:8/771: symlink df/d18/d19/d46/dbc/l108 0 2022-01-31T19:44:32.557 INFO:tasks.workunit.client.1.smithi171.stdout:9/849: dwrite fc [0,4194304] 0 2022-01-31T19:44:32.560 INFO:tasks.workunit.client.1.smithi171.stdout:8/772: dread df/d27/f64 [0,4194304] 0 2022-01-31T19:44:32.561 INFO:tasks.workunit.client.1.smithi171.stdout:8/773: creat df/d18/d19/db2/f109 x:0 0 0 2022-01-31T19:44:32.563 INFO:tasks.workunit.client.1.smithi171.stdout:8/774: link df/la5 df/d27/d5c/l10a 0 2022-01-31T19:44:32.563 INFO:tasks.workunit.client.1.smithi171.stdout:9/850: dread dd/df/d15/d53/fbf [0,4194304] 0 2022-01-31T19:44:32.565 INFO:tasks.workunit.client.1.smithi171.stdout:8/775: creat df/d18/d79/d7a/f10b x:0 0 0 2022-01-31T19:44:32.566 INFO:tasks.workunit.client.1.smithi171.stdout:9/851: creat dd/df/d117/f120 x:0 0 0 2022-01-31T19:44:32.567 INFO:tasks.workunit.client.1.smithi171.stdout:9/852: chown dd/d109/de6/d7c/d7d/db6/lde 59 1 2022-01-31T19:44:32.567 INFO:tasks.workunit.client.1.smithi171.stdout:9/853: mkdir dd/df/d15/d1a/d64/d121 0 2022-01-31T19:44:32.578 INFO:tasks.workunit.client.1.smithi171.stdout:5/780: dwrite d5/d90/db7/d15/f64 [0,4194304] 0 2022-01-31T19:44:32.584 INFO:tasks.workunit.client.1.smithi171.stdout:8/776: dwrite df/fdc [0,4194304] 0 2022-01-31T19:44:32.595 INFO:tasks.workunit.client.1.smithi171.stdout:8/777: dread df/fdc [0,4194304] 0 2022-01-31T19:44:32.601 INFO:tasks.workunit.client.1.smithi171.stdout:5/781: dwrite d5/d90/fa8 [0,4194304] 0 2022-01-31T19:44:32.602 INFO:tasks.workunit.client.1.smithi171.stdout:5/782: truncate d5/db0/fbc 536472 0 2022-01-31T19:44:32.606 INFO:tasks.workunit.client.1.smithi171.stdout:5/783: write d5/d90/f78 [2773120,39332] 0 2022-01-31T19:44:32.611 INFO:tasks.workunit.client.1.smithi171.stdout:6/796: sync 2022-01-31T19:44:32.611 INFO:tasks.workunit.client.1.smithi171.stdout:6/797: dread - dc/d19/d29/d27/d37/fff zero size 2022-01-31T19:44:32.612 INFO:tasks.workunit.client.1.smithi171.stdout:6/798: fdatasync dc/d18/f2a 0 2022-01-31T19:44:32.612 INFO:tasks.workunit.client.1.smithi171.stdout:6/799: dread - dc/d19/dae/f107 zero size 2022-01-31T19:44:32.612 INFO:tasks.workunit.client.1.smithi171.stdout:6/800: readlink dc/d19/d30/d3c/d80/l103 0 2022-01-31T19:44:32.613 INFO:tasks.workunit.client.1.smithi171.stdout:3/962: sync 2022-01-31T19:44:32.613 INFO:tasks.workunit.client.1.smithi171.stdout:5/784: mkdir d5/d90/dd1/d10a 0 2022-01-31T19:44:32.614 INFO:tasks.workunit.client.1.smithi171.stdout:6/801: link dc/d19/d29/d27/d37/cbe dc/d21/d4f/c10e 0 2022-01-31T19:44:32.614 INFO:tasks.workunit.client.1.smithi171.stdout:6/802: creat dc/d19/d29/da7/f10f x:0 0 0 2022-01-31T19:44:32.616 INFO:tasks.workunit.client.1.smithi171.stdout:3/963: creat d4/dd/d27/d13c/f13d x:0 0 0 2022-01-31T19:44:32.616 INFO:tasks.workunit.client.1.smithi171.stdout:3/964: write d4/d21/d3a/db5/fd5 [2084449,119970] 0 2022-01-31T19:44:32.617 INFO:tasks.workunit.client.1.smithi171.stdout:5/785: creat d5/db0/dc8/f10b x:0 0 0 2022-01-31T19:44:32.617 INFO:tasks.workunit.client.1.smithi171.stdout:5/786: write d5/f55 [5250149,99139] 0 2022-01-31T19:44:32.618 INFO:tasks.workunit.client.1.smithi171.stdout:5/787: truncate d5/d90/db7/f21 4867692 0 2022-01-31T19:44:32.619 INFO:tasks.workunit.client.1.smithi171.stdout:6/803: mkdir dc/d8b/d110 0 2022-01-31T19:44:32.620 INFO:tasks.workunit.client.1.smithi171.stdout:3/965: unlink d4/d21/d3a/d90/fc9 0 2022-01-31T19:44:32.622 INFO:tasks.workunit.client.1.smithi171.stdout:3/966: read d4/dd/d87/f9f [94704,105887] 0 2022-01-31T19:44:32.622 INFO:tasks.workunit.client.1.smithi171.stdout:3/967: chown d4/dd/dc7/cf2 55537218 1 2022-01-31T19:44:32.623 INFO:tasks.workunit.client.1.smithi171.stdout:3/968: stat d4/dd/d4b 0 2022-01-31T19:44:32.623 INFO:tasks.workunit.client.1.smithi171.stdout:5/788: rmdir d5/d90/db7/d15/d23/d6b/dae/df8 0 2022-01-31T19:44:32.627 INFO:tasks.workunit.client.1.smithi171.stdout:8/778: dwrite df/d18/d19/d21/d88/ffd [0,4194304] 0 2022-01-31T19:44:32.627 INFO:tasks.workunit.client.1.smithi171.stdout:8/779: write df/d18/d19/db2/f109 [1015572,56539] 0 2022-01-31T19:44:32.630 INFO:tasks.workunit.client.1.smithi171.stdout:5/789: rename d5/d90/f78 to d5/d90/db7/d15/d5d/d107/f10c 0 2022-01-31T19:44:32.630 INFO:tasks.workunit.client.1.smithi171.stdout:8/780: symlink df/l10c 0 2022-01-31T19:44:32.630 INFO:tasks.workunit.client.1.smithi171.stdout:8/781: chown df/d18/d79 12263628 1 2022-01-31T19:44:32.642 INFO:tasks.workunit.client.1.smithi171.stdout:8/782: dread df/d27/fb7 [0,4194304] 0 2022-01-31T19:44:32.660 INFO:tasks.workunit.client.1.smithi171.stdout:5/790: dwrite d5/d90/db7/d15/d23/d3d/f49 [0,4194304] 0 2022-01-31T19:44:32.661 INFO:tasks.workunit.client.1.smithi171.stdout:6/804: dwrite dc/d19/d29/d6a/fb4 [0,4194304] 0 2022-01-31T19:44:32.662 INFO:tasks.workunit.client.1.smithi171.stdout:3/969: dwrite d4/dd/d39/ff3 [0,4194304] 0 2022-01-31T19:44:32.662 INFO:tasks.workunit.client.1.smithi171.stdout:5/791: truncate d5/f4f 1603786 0 2022-01-31T19:44:32.666 INFO:tasks.workunit.client.1.smithi171.stdout:3/970: creat d4/d21/de5/d92/dab/d52/d7a/f13e x:0 0 0 2022-01-31T19:44:32.668 INFO:tasks.workunit.client.1.smithi171.stdout:5/792: mknod d5/db0/c10d 0 2022-01-31T19:44:32.669 INFO:tasks.workunit.client.1.smithi171.stdout:5/793: symlink d5/d90/l10e 0 2022-01-31T19:44:32.670 INFO:tasks.workunit.client.1.smithi171.stdout:5/794: fdatasync d5/d90/db7/d15/d23/f87 0 2022-01-31T19:44:32.670 INFO:tasks.workunit.client.1.smithi171.stdout:6/805: rename dc/d19/d29/da7 to dc/d19/d29/d27/d37/d111 0 2022-01-31T19:44:32.671 INFO:tasks.workunit.client.1.smithi171.stdout:8/783: dwrite df/d27/d90/fb4 [0,4194304] 0 2022-01-31T19:44:32.671 INFO:tasks.workunit.client.1.smithi171.stdout:8/784: chown df/d18/d19/d36/d70/d7e/f8a 0 1 2022-01-31T19:44:32.673 INFO:tasks.workunit.client.1.smithi171.stdout:6/806: mknod dc/d18/c112 0 2022-01-31T19:44:32.674 INFO:tasks.workunit.client.1.smithi171.stdout:3/971: rename d4/l11e to d4/dd/d27/d28/l13f 0 2022-01-31T19:44:32.675 INFO:tasks.workunit.client.1.smithi171.stdout:8/785: mknod df/d18/d19/d21/d88/c10d 0 2022-01-31T19:44:32.676 INFO:tasks.workunit.client.1.smithi171.stdout:8/786: chown df/d27/d90/fb4 600005 1 2022-01-31T19:44:32.676 INFO:tasks.workunit.client.1.smithi171.stdout:8/787: chown df/dcc 101513 1 2022-01-31T19:44:32.677 INFO:tasks.workunit.client.1.smithi171.stdout:6/807: write dc/d18/f39 [208134,54502] 0 2022-01-31T19:44:32.677 INFO:tasks.workunit.client.1.smithi171.stdout:6/808: chown dc/d19/d29/d27/f3a 1 1 2022-01-31T19:44:32.677 INFO:tasks.workunit.client.1.smithi171.stdout:3/972: mkdir d4/d21/de5/d92/dab/d52/d7a/d140 0 2022-01-31T19:44:32.681 INFO:tasks.workunit.client.1.smithi171.stdout:5/795: dread d5/d90/db7/d15/f33 [0,4194304] 0 2022-01-31T19:44:32.682 INFO:tasks.workunit.client.1.smithi171.stdout:3/973: write d4/d21/de5/d92/dab/f2b [2627965,97517] 0 2022-01-31T19:44:32.684 INFO:tasks.workunit.client.1.smithi171.stdout:5/796: read d5/d90/f7f [2069403,66250] 0 2022-01-31T19:44:32.684 INFO:tasks.workunit.client.1.smithi171.stdout:5/797: write d5/db0/ff6 [23714,51009] 0 2022-01-31T19:44:32.685 INFO:tasks.workunit.client.1.smithi171.stdout:8/788: creat df/d95/f10e x:0 0 0 2022-01-31T19:44:32.688 INFO:tasks.workunit.client.1.smithi171.stdout:3/974: mkdir d4/d21/d3a/db5/dbe/dd1/d115/d141 0 2022-01-31T19:44:32.693 INFO:tasks.workunit.client.1.smithi171.stdout:5/798: link d5/d90/db7/d15/d5d/d6a/f77 d5/d90/db7/d15/d23/d3d/f10f 0 2022-01-31T19:44:32.693 INFO:tasks.workunit.client.1.smithi171.stdout:3/975: link d4/d21/de5/d92/dab/d6b/ldf d4/dd/d27/l142 0 2022-01-31T19:44:32.694 INFO:tasks.workunit.client.1.smithi171.stdout:3/976: readlink d4/dd/d27/d6e/ld3 0 2022-01-31T19:44:32.695 INFO:tasks.workunit.client.1.smithi171.stdout:5/799: symlink d5/db0/dc8/l110 0 2022-01-31T19:44:32.696 INFO:tasks.workunit.client.1.smithi171.stdout:3/977: mkdir d4/d21/d3a/d90/d143 0 2022-01-31T19:44:32.697 INFO:tasks.workunit.client.1.smithi171.stdout:5/800: write d5/d90/db7/f21 [1571322,42656] 0 2022-01-31T19:44:32.697 INFO:tasks.workunit.client.1.smithi171.stdout:5/801: chown d5/d59/da0/cfb 8 1 2022-01-31T19:44:32.700 INFO:tasks.workunit.client.1.smithi171.stdout:8/789: rmdir df/d95 39 2022-01-31T19:44:32.700 INFO:tasks.workunit.client.1.smithi171.stdout:8/790: readlink df/d18/laa 0 2022-01-31T19:44:32.701 INFO:tasks.workunit.client.1.smithi171.stdout:5/802: stat d5/db0/dc8/l103 0 2022-01-31T19:44:32.701 INFO:tasks.workunit.client.1.smithi171.stdout:5/803: creat d5/d90/f111 x:0 0 0 2022-01-31T19:44:32.704 INFO:tasks.workunit.client.1.smithi171.stdout:5/804: creat d5/d90/db7/d15/d5d/d6f/f112 x:0 0 0 2022-01-31T19:44:32.705 INFO:tasks.workunit.client.1.smithi171.stdout:5/805: getdents d5/d90/db7/d15 0 2022-01-31T19:44:32.707 INFO:tasks.workunit.client.1.smithi171.stdout:5/806: stat d5/d90/db7/d15/d23/d67/c4b 0 2022-01-31T19:44:32.708 INFO:tasks.workunit.client.1.smithi171.stdout:5/807: creat d5/d90/dd1/dfc/f113 x:0 0 0 2022-01-31T19:44:32.708 INFO:tasks.workunit.client.1.smithi171.stdout:5/808: fdatasync d5/d90/db7/dd/d66/fc6 0 2022-01-31T19:44:32.750 INFO:tasks.workunit.client.1.smithi171.stdout:5/809: dwrite d5/d90/db7/d15/d23/d67/f5c [4194304,4194304] 0 2022-01-31T19:44:32.751 INFO:tasks.workunit.client.1.smithi171.stdout:5/810: creat d5/db0/f114 x:0 0 0 2022-01-31T19:44:32.751 INFO:tasks.workunit.client.1.smithi171.stdout:5/811: truncate d5/d90/db7/d15/d80/d8d/f100 64091 0 2022-01-31T19:44:32.753 INFO:tasks.workunit.client.1.smithi171.stdout:5/812: creat d5/d59/da0/f115 x:0 0 0 2022-01-31T19:44:32.755 INFO:tasks.workunit.client.1.smithi171.stdout:5/813: creat d5/d90/db7/d15/d80/d8d/de2/f116 x:0 0 0 2022-01-31T19:44:32.768 INFO:tasks.workunit.client.1.smithi171.stdout:5/814: rmdir d5/d90/db7/d15/d80/d8d/de2 39 2022-01-31T19:44:32.776 INFO:tasks.workunit.client.1.smithi171.stdout:2/994: sync 2022-01-31T19:44:32.776 INFO:tasks.workunit.client.1.smithi171.stdout:0/866: sync 2022-01-31T19:44:32.776 INFO:tasks.workunit.client.1.smithi171.stdout:0/867: readlink d9/db/dc/l4f 0 2022-01-31T19:44:32.784 INFO:tasks.workunit.client.1.smithi171.stdout:2/995: dread d7/d9/d1a/f53 [4194304,4194304] 0 2022-01-31T19:44:32.787 INFO:tasks.workunit.client.1.smithi171.stdout:2/996: rmdir d7/dd/dfc/d101/d126/d3d/d40/dd2/d115/d116/da4/d11e 0 2022-01-31T19:44:32.808 INFO:tasks.workunit.client.1.smithi171.stdout:5/815: dwrite d5/d90/db7/dd/d66/f82 [0,4194304] 0 2022-01-31T19:44:32.811 INFO:tasks.workunit.client.1.smithi171.stdout:0/868: dwrite d9/db/dc2/f119 [0,4194304] 0 2022-01-31T19:44:32.811 INFO:tasks.workunit.client.1.smithi171.stdout:5/816: rename d5/d90/db7/d15/d23/l8b to d5/d90/db7/d15/d80/d109/l117 0 2022-01-31T19:44:32.812 INFO:tasks.workunit.client.1.smithi171.stdout:2/997: dwrite d7/dd/dfc/d101/d126/d3d/d40/dd2/d115/d116/f10c [0,4194304] 0 2022-01-31T19:44:32.812 INFO:tasks.workunit.client.1.smithi171.stdout:2/998: fsync d7/dd/dfc/d101/d126/d63/d9f/f10f 0 2022-01-31T19:44:32.813 INFO:tasks.workunit.client.1.smithi171.stdout:0/869: getdents d9/d24 0 2022-01-31T19:44:32.813 INFO:tasks.workunit.client.1.smithi171.stdout:0/870: read d9/f8a [2502489,74910] 0 2022-01-31T19:44:32.814 INFO:tasks.workunit.client.1.smithi171.stdout:5/817: creat d5/f118 x:0 0 0 2022-01-31T19:44:32.815 INFO:tasks.workunit.client.1.smithi171.stdout:5/818: truncate d5/d90/db7/d15/fb8 2244296 0 2022-01-31T19:44:32.816 INFO:tasks.workunit.client.1.smithi171.stdout:2/999: creat d7/dd/dea/f132 x:0 0 0 2022-01-31T19:44:32.817 INFO:tasks.workunit.client.1.smithi171.stdout:0/871: getdents d9/d24/d4c/d35/d67/df4 0 2022-01-31T19:44:32.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/872: stat d9/db/f41 0 2022-01-31T19:44:32.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/873: truncate d9/d24/d92/f5f 4622867 0 2022-01-31T19:44:32.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/874: dread - d9/fe0 zero size 2022-01-31T19:44:32.818 INFO:tasks.workunit.client.1.smithi171.stdout:0/875: write d9/db/dc2/f138 [1030297,128401] 0 2022-01-31T19:44:32.819 INFO:tasks.workunit.client.1.smithi171.stdout:0/876: write d9/d24/d92/d9d/dc6/dfb/f12b [338993,28300] 0 2022-01-31T19:44:32.820 INFO:tasks.workunit.client.1.smithi171.stdout:5/819: symlink d5/d90/dd1/d10a/l119 0 2022-01-31T19:44:32.820 INFO:tasks.workunit.client.1.smithi171.stdout:5/820: fsync d5/fd6 0 2022-01-31T19:44:32.842 INFO:tasks.workunit.client.1.smithi171.stdout:0/877: dwrite d9/db/dc/f104 [0,4194304] 0 2022-01-31T19:44:32.842 INFO:tasks.workunit.client.1.smithi171.stdout:0/878: read - d9/d24/d4c/d35/f125 zero size 2022-01-31T19:44:32.842 INFO:tasks.workunit.client.1.smithi171.stdout:0/879: write d9/db/dc2/fd7 [116037,19605] 0 2022-01-31T19:44:32.843 INFO:tasks.workunit.client.1.smithi171.stdout:0/880: stat d9/db/dc/d3f/db9/l127 0 2022-01-31T19:44:32.843 INFO:tasks.workunit.client.1.smithi171.stdout:0/881: read - d9/db/d22/fc7 zero size 2022-01-31T19:44:32.844 INFO:tasks.workunit.client.1.smithi171.stdout:5/821: dwrite d5/d90/db7/d15/d5d/d6a/f77 [0,4194304] 0 2022-01-31T19:44:32.850 INFO:tasks.workunit.client.1.smithi171.stdout:5/822: symlink d5/d90/db7/dd/d66/l11a 0 2022-01-31T19:44:32.850 INFO:tasks.workunit.client.1.smithi171.stdout:5/823: write d5/d90/db7/d15/d5d/d6f/f9c [5021967,92272] 0 2022-01-31T19:44:32.851 INFO:tasks.workunit.client.1.smithi171.stdout:5/824: write d5/d90/db7/d15/d23/d6b/fc0 [5125411,100665] 0 2022-01-31T19:44:32.851 INFO:tasks.workunit.client.1.smithi171.stdout:5/825: mknod d5/d90/db7/d15/d23/d58/d9e/dd7/c11b 0 2022-01-31T19:44:32.852 INFO:tasks.workunit.client.1.smithi171.stdout:5/826: getdents d5/d90/db7/d15/d5d 0 2022-01-31T19:44:32.859 INFO:tasks.workunit.client.1.smithi171.stdout:4/710: sync 2022-01-31T19:44:32.859 INFO:tasks.workunit.client.1.smithi171.stdout:9/854: sync 2022-01-31T19:44:32.859 INFO:tasks.workunit.client.1.smithi171.stdout:7/628: sync 2022-01-31T19:44:32.860 INFO:tasks.workunit.client.1.smithi171.stdout:9/855: chown dd/df/d15/d1a/d64/lda 3 1 2022-01-31T19:44:32.860 INFO:tasks.workunit.client.1.smithi171.stdout:4/711: chown d2/d3/d5/d3b/c61 20 1 2022-01-31T19:44:32.860 INFO:tasks.workunit.client.1.smithi171.stdout:9/856: fdatasync dd/df/d15/d1a/dcb/fce 0 2022-01-31T19:44:32.861 INFO:tasks.workunit.client.1.smithi171.stdout:4/712: creat d2/d3/d5/d41/dcf/d12/d15/dc4/ff4 x:0 0 0 2022-01-31T19:44:32.862 INFO:tasks.workunit.client.1.smithi171.stdout:4/713: write d2/d3/d5/d70/d7b/f93 [544705,89180] 0 2022-01-31T19:44:32.862 INFO:tasks.workunit.client.1.smithi171.stdout:4/714: read d2/d3/d5/d41/dcf/d12/d15/f3e [2072032,114290] 0 2022-01-31T19:44:32.862 INFO:tasks.workunit.client.1.smithi171.stdout:9/857: unlink dd/df/d15/d38/d4d/c6c 0 2022-01-31T19:44:32.863 INFO:tasks.workunit.client.1.smithi171.stdout:9/858: mknod dd/df/d3f/d73/c122 0 2022-01-31T19:44:32.863 INFO:tasks.workunit.client.1.smithi171.stdout:9/859: readlink dd/df/d15/d1a/d64/d6e/lb3 0 2022-01-31T19:44:32.864 INFO:tasks.workunit.client.1.smithi171.stdout:7/629: rmdir d1/d1e/d2a/d29/d31/d30/d6b 39 2022-01-31T19:44:32.865 INFO:tasks.workunit.client.1.smithi171.stdout:9/860: rename dd/df/d15/d38/d4d/d80/dfa/cf4 to dd/df/d3f/d73/d96/c123 0 2022-01-31T19:44:32.867 INFO:tasks.workunit.client.1.smithi171.stdout:7/630: mkdir d1/d1e/d2a/d29/d77/dab/dd5 0 2022-01-31T19:44:32.867 INFO:tasks.workunit.client.1.smithi171.stdout:7/631: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd6 x:0 0 0 2022-01-31T19:44:32.867 INFO:tasks.workunit.client.1.smithi171.stdout:9/861: getdents dd/d109/de6/d7c/d7d 0 2022-01-31T19:44:32.871 INFO:tasks.workunit.client.1.smithi171.stdout:9/862: unlink dd/df/d15/d1a/d64/d6e/ddd/l100 0 2022-01-31T19:44:32.872 INFO:tasks.workunit.client.1.smithi171.stdout:0/882: dwrite d9/db/d22/fc7 [0,4194304] 0 2022-01-31T19:44:32.872 INFO:tasks.workunit.client.1.smithi171.stdout:9/863: read dd/d109/de6/d7c/d7d/fdb [488657,90489] 0 2022-01-31T19:44:32.874 INFO:tasks.workunit.client.1.smithi171.stdout:0/883: symlink d9/db/d22/d73/ddc/l143 0 2022-01-31T19:44:32.875 INFO:tasks.workunit.client.1.smithi171.stdout:0/884: dread - d9/d24/d92/d9d/dc6/f137 zero size 2022-01-31T19:44:32.875 INFO:tasks.workunit.client.1.smithi171.stdout:0/885: creat d9/d24/d92/d9d/dc6/f144 x:0 0 0 2022-01-31T19:44:32.876 INFO:tasks.workunit.client.1.smithi171.stdout:7/632: rename d1/d52 to d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7 0 2022-01-31T19:44:32.876 INFO:tasks.workunit.client.1.smithi171.stdout:7/633: fdatasync d1/d26/d47/d8f/fc5 0 2022-01-31T19:44:32.879 INFO:tasks.workunit.client.1.smithi171.stdout:0/886: link d9/d24/f4d d9/d24/d92/d9d/d9f/f145 0 2022-01-31T19:44:32.879 INFO:tasks.workunit.client.1.smithi171.stdout:5/827: dwrite d5/d90/db7/d15/d23/d67/f4c [0,4194304] 0 2022-01-31T19:44:32.879 INFO:tasks.workunit.client.1.smithi171.stdout:0/887: truncate d9/db/d59/ffe 427473 0 2022-01-31T19:44:32.879 INFO:tasks.workunit.client.1.smithi171.stdout:0/888: chown d9/db/d59/d71/lf7 19539225 1 2022-01-31T19:44:32.880 INFO:tasks.workunit.client.1.smithi171.stdout:0/889: truncate d9/db/f121 6502698 0 2022-01-31T19:44:32.880 INFO:tasks.workunit.client.1.smithi171.stdout:1/859: sync 2022-01-31T19:44:32.882 INFO:tasks.workunit.client.1.smithi171.stdout:9/864: mknod dd/c124 0 2022-01-31T19:44:32.882 INFO:tasks.workunit.client.1.smithi171.stdout:9/865: getdents dd/df/d15/d1a/d84/dca/df1 0 2022-01-31T19:44:32.882 INFO:tasks.workunit.client.1.smithi171.stdout:9/866: truncate dd/df/d15/d38/d4d/f10a 240949 0 2022-01-31T19:44:32.882 INFO:tasks.workunit.client.1.smithi171.stdout:4/715: dwrite d2/d7f/fa8 [0,4194304] 0 2022-01-31T19:44:32.891 INFO:tasks.workunit.client.1.smithi171.stdout:4/716: dread d2/d3/d5/d41/dcf/d12/d15/f3e [0,4194304] 0 2022-01-31T19:44:32.891 INFO:tasks.workunit.client.1.smithi171.stdout:4/717: write d2/d3/d5/dc/f7a [905546,78573] 0 2022-01-31T19:44:32.892 INFO:tasks.workunit.client.1.smithi171.stdout:1/860: unlink dd/df/d19/f27 0 2022-01-31T19:44:32.892 INFO:tasks.workunit.client.1.smithi171.stdout:9/867: rename f4 to dd/df/d15/f125 0 2022-01-31T19:44:32.892 INFO:tasks.workunit.client.1.smithi171.stdout:4/718: read d2/d3/d5/d3b/faf [4103968,83802] 0 2022-01-31T19:44:32.894 INFO:tasks.workunit.client.1.smithi171.stdout:1/861: unlink dd/df/d19/d1b/f63 0 2022-01-31T19:44:32.896 INFO:tasks.workunit.client.1.smithi171.stdout:6/809: sync 2022-01-31T19:44:32.896 INFO:tasks.workunit.client.1.smithi171.stdout:8/791: sync 2022-01-31T19:44:32.897 INFO:tasks.workunit.client.1.smithi171.stdout:3/978: sync 2022-01-31T19:44:32.898 INFO:tasks.workunit.client.1.smithi171.stdout:9/868: symlink dd/df/d3f/d73/l126 0 2022-01-31T19:44:32.900 INFO:tasks.workunit.client.1.smithi171.stdout:0/890: dread d9/d24/d4c/d35/f74 [0,4194304] 0 2022-01-31T19:44:32.904 INFO:tasks.workunit.client.1.smithi171.stdout:7/634: dwrite d1/d1e/f85 [0,4194304] 0 2022-01-31T19:44:32.904 INFO:tasks.workunit.client.1.smithi171.stdout:7/635: truncate d1/d1e/d2a/d29/d31/d30/f5a 917844 0 2022-01-31T19:44:32.905 INFO:tasks.workunit.client.1.smithi171.stdout:7/636: dread - d1/d1e/d2a/d29/d31/d30/d3c/d79/f8d zero size 2022-01-31T19:44:32.905 INFO:tasks.workunit.client.1.smithi171.stdout:7/637: fsync d1/d1b/f22 0 2022-01-31T19:44:32.905 INFO:tasks.workunit.client.1.smithi171.stdout:7/638: chown d1/d26/d57/l68 1574 1 2022-01-31T19:44:32.905 INFO:tasks.workunit.client.1.smithi171.stdout:4/719: getdents d2/d3/d5/d41/dcf/d12/d15/d57/dad 0 2022-01-31T19:44:32.906 INFO:tasks.workunit.client.1.smithi171.stdout:3/979: link d4/d21/de5/da7/ca8 d4/d21/d3a/db5/dbe/dd1/d115/d134/c144 0 2022-01-31T19:44:32.906 INFO:tasks.workunit.client.1.smithi171.stdout:1/862: getdents dd/d23/d2e/d5e/d81/d98 0 2022-01-31T19:44:32.906 INFO:tasks.workunit.client.1.smithi171.stdout:3/980: truncate d4/d21/de5/d92/dab/d60/df4/f11a 827247 0 2022-01-31T19:44:32.906 INFO:tasks.workunit.client.1.smithi171.stdout:1/863: readlink dd/df/d84/dea/d64/db9/ld3 0 2022-01-31T19:44:32.906 INFO:tasks.workunit.client.1.smithi171.stdout:9/869: creat dd/f127 x:0 0 0 2022-01-31T19:44:32.907 INFO:tasks.workunit.client.1.smithi171.stdout:0/891: creat d9/d24/d4c/d101/d132/f146 x:0 0 0 2022-01-31T19:44:32.908 INFO:tasks.workunit.client.1.smithi171.stdout:0/892: stat d9/f1d 0 2022-01-31T19:44:32.908 INFO:tasks.workunit.client.1.smithi171.stdout:1/864: write dd/d23/d34/f65 [1682710,4309] 0 2022-01-31T19:44:32.908 INFO:tasks.workunit.client.1.smithi171.stdout:0/893: read d9/db/dc/d3f/db9/d7a/d88/fa3 [4369642,68082] 0 2022-01-31T19:44:32.908 INFO:tasks.workunit.client.1.smithi171.stdout:7/639: unlink d1/c93 0 2022-01-31T19:44:32.909 INFO:tasks.workunit.client.1.smithi171.stdout:7/640: fsync d1/d1b/f22 0 2022-01-31T19:44:32.909 INFO:tasks.workunit.client.1.smithi171.stdout:4/720: creat d2/d3/d5/dc/d4b/ff5 x:0 0 0 2022-01-31T19:44:32.910 INFO:tasks.workunit.client.1.smithi171.stdout:9/870: truncate dd/df/d15/f69 2083624 0 2022-01-31T19:44:32.912 INFO:tasks.workunit.client.1.smithi171.stdout:7/641: unlink d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/c64 0 2022-01-31T19:44:32.914 INFO:tasks.workunit.client.1.smithi171.stdout:7/642: symlink d1/d1e/d2a/d29/d31/d30/dc4/ld8 0 2022-01-31T19:44:32.916 INFO:tasks.workunit.client.1.smithi171.stdout:7/643: creat d1/fd9 x:0 0 0 2022-01-31T19:44:32.917 INFO:tasks.workunit.client.1.smithi171.stdout:7/644: dread - d1/d1b/f4c zero size 2022-01-31T19:44:32.917 INFO:tasks.workunit.client.1.smithi171.stdout:7/645: dread - d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd6 zero size 2022-01-31T19:44:32.917 INFO:tasks.workunit.client.1.smithi171.stdout:7/646: creat d1/d1e/d2a/d29/d31/d30/d3c/fda x:0 0 0 2022-01-31T19:44:32.919 INFO:tasks.workunit.client.1.smithi171.stdout:1/865: dread dd/df/d84/dea/d35/f70 [0,4194304] 0 2022-01-31T19:44:32.921 INFO:tasks.workunit.client.1.smithi171.stdout:1/866: rename dd/d23/d2e/d5e/ff6 to dd/d23/d2e/d5e/d81/d98/d80/f104 0 2022-01-31T19:44:32.922 INFO:tasks.workunit.client.1.smithi171.stdout:1/867: stat dd/df/d84/dea/d64/da7/ce1 0 2022-01-31T19:44:32.925 INFO:tasks.workunit.client.1.smithi171.stdout:9/871: dread dd/df/d15/d38/d41/fd7 [0,4194304] 0 2022-01-31T19:44:32.926 INFO:tasks.workunit.client.1.smithi171.stdout:9/872: symlink dd/d109/l128 0 2022-01-31T19:44:32.926 INFO:tasks.workunit.client.1.smithi171.stdout:9/873: write dd/df/d15/d38/d4d/f10a [520397,105282] 0 2022-01-31T19:44:32.927 INFO:tasks.workunit.client.1.smithi171.stdout:5/828: dwrite d5/f89 [0,4194304] 0 2022-01-31T19:44:32.927 INFO:tasks.workunit.client.1.smithi171.stdout:1/868: write dd/df/d84/dea/f4a [2271842,45923] 0 2022-01-31T19:44:32.928 INFO:tasks.workunit.client.1.smithi171.stdout:5/829: symlink d5/d90/db7/d15/l11c 0 2022-01-31T19:44:32.930 INFO:tasks.workunit.client.1.smithi171.stdout:1/869: unlink dd/df/c69 0 2022-01-31T19:44:32.930 INFO:tasks.workunit.client.1.smithi171.stdout:1/870: chown dd/d23/f9e 53809340 1 2022-01-31T19:44:32.930 INFO:tasks.workunit.client.1.smithi171.stdout:1/871: read - dd/d23/d2e/d5e/d81/d98/d80/dee/ff1 zero size 2022-01-31T19:44:32.931 INFO:tasks.workunit.client.1.smithi171.stdout:1/872: symlink dd/d23/d74/l105 0 2022-01-31T19:44:32.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/873: rmdir dd/df/dfe 0 2022-01-31T19:44:32.936 INFO:tasks.workunit.client.1.smithi171.stdout:1/874: chown dd/d23/f3f 150222426 1 2022-01-31T19:44:32.936 INFO:tasks.workunit.client.1.smithi171.stdout:1/875: readlink dd/d23/d34/d4b/l4c 0 2022-01-31T19:44:32.940 INFO:tasks.workunit.client.1.smithi171.stdout:9/874: write dd/df/d15/d1a/f2b [2268501,128489] 0 2022-01-31T19:44:32.943 INFO:tasks.workunit.client.1.smithi171.stdout:8/792: dwrite df/d18/d19/d36/f81 [0,4194304] 0 2022-01-31T19:44:32.946 INFO:tasks.workunit.client.1.smithi171.stdout:9/875: creat dd/df/d15/d1a/d64/d121/f129 x:0 0 0 2022-01-31T19:44:32.949 INFO:tasks.workunit.client.1.smithi171.stdout:9/876: truncate dd/df/d15/f2f 1387388 0 2022-01-31T19:44:32.950 INFO:tasks.workunit.client.1.smithi171.stdout:3/981: dwrite d4/d21/d3a/db5/dbe/f131 [0,4194304] 0 2022-01-31T19:44:32.951 INFO:tasks.workunit.client.1.smithi171.stdout:9/877: mkdir dd/df/d15/d38/d9c/da7/d12a 0 2022-01-31T19:44:32.951 INFO:tasks.workunit.client.1.smithi171.stdout:9/878: dread - dd/df/d15/d53/fa6 zero size 2022-01-31T19:44:32.951 INFO:tasks.workunit.client.1.smithi171.stdout:6/810: dwrite dc/d19/d29/d27/d37/d3b/d67/dd1/fd3 [0,4194304] 0 2022-01-31T19:44:32.951 INFO:tasks.workunit.client.1.smithi171.stdout:5/830: dread d5/d90/fa8 [0,4194304] 0 2022-01-31T19:44:32.952 INFO:tasks.workunit.client.1.smithi171.stdout:5/831: chown d5/d90/db7/d15/d23/f87 44817 1 2022-01-31T19:44:32.952 INFO:tasks.workunit.client.1.smithi171.stdout:6/811: read dc/f4a [1352292,701] 0 2022-01-31T19:44:32.952 INFO:tasks.workunit.client.1.smithi171.stdout:6/812: fsync dc/d19/d30/d3c/fc7 0 2022-01-31T19:44:32.953 INFO:tasks.workunit.client.1.smithi171.stdout:5/832: write d5/d90/db7/d15/d5d/fd0 [2343275,115445] 0 2022-01-31T19:44:32.953 INFO:tasks.workunit.client.1.smithi171.stdout:3/982: link d4/dd/d27/d28/d35/d55/ce6 d4/d21/d3a/d90/c145 0 2022-01-31T19:44:32.954 INFO:tasks.workunit.client.1.smithi171.stdout:7/647: dwrite d1/d1e/d2a/d58/fb1 [0,4194304] 0 2022-01-31T19:44:32.955 INFO:tasks.workunit.client.1.smithi171.stdout:9/879: rename dd/df/d15/d1a/d64/d6e/l71 to dd/d109/de6/db5/l12b 0 2022-01-31T19:44:32.955 INFO:tasks.workunit.client.1.smithi171.stdout:9/880: readlink dd/df/d15/l103 0 2022-01-31T19:44:32.957 INFO:tasks.workunit.client.1.smithi171.stdout:9/881: write dd/d109/de6/da8/fe1 [1292659,96559] 0 2022-01-31T19:44:32.961 INFO:tasks.workunit.client.1.smithi171.stdout:0/894: dwrite d9/db/dc/d3f/f6d [0,4194304] 0 2022-01-31T19:44:32.961 INFO:tasks.workunit.client.1.smithi171.stdout:0/895: write d9/db/dc/d3f/db9/f8e [2335980,46318] 0 2022-01-31T19:44:32.963 INFO:tasks.workunit.client.1.smithi171.stdout:5/833: creat d5/d90/db7/d15/d23/d58/f11d x:0 0 0 2022-01-31T19:44:32.965 INFO:tasks.workunit.client.1.smithi171.stdout:3/983: creat d4/d21/de5/d92/dab/f146 x:0 0 0 2022-01-31T19:44:32.971 INFO:tasks.workunit.client.1.smithi171.stdout:7/648: truncate d1/d1e/d2a/d29/d31/f38 3433698 0 2022-01-31T19:44:32.972 INFO:tasks.workunit.client.1.smithi171.stdout:7/649: chown d1/d1e/ccb 0 1 2022-01-31T19:44:32.974 INFO:tasks.workunit.client.1.smithi171.stdout:0/896: dread d9/d24/d92/d9d/d9f/f145 [0,4194304] 0 2022-01-31T19:44:32.974 INFO:tasks.workunit.client.1.smithi171.stdout:9/882: mknod dd/df/d15/d38/d41/c12c 0 2022-01-31T19:44:32.974 INFO:tasks.workunit.client.1.smithi171.stdout:9/883: readlink dd/df/d15/l45 0 2022-01-31T19:44:32.975 INFO:tasks.workunit.client.1.smithi171.stdout:9/884: readlink dd/d109/de6/l5f 0 2022-01-31T19:44:32.979 INFO:tasks.workunit.client.1.smithi171.stdout:9/885: rmdir dd/df/d117 39 2022-01-31T19:44:32.980 INFO:tasks.workunit.client.1.smithi171.stdout:9/886: write dd/d109/de6/da8/fe0 [966177,79288] 0 2022-01-31T19:44:32.980 INFO:tasks.workunit.client.1.smithi171.stdout:9/887: creat dd/df/d15/d38/d4d/d80/f12d x:0 0 0 2022-01-31T19:44:32.980 INFO:tasks.workunit.client.1.smithi171.stdout:9/888: fdatasync dd/df/d117/f120 0 2022-01-31T19:44:32.982 INFO:tasks.workunit.client.1.smithi171.stdout:9/889: symlink dd/df/d15/d1a/d64/d6e/ddd/l12e 0 2022-01-31T19:44:32.982 INFO:tasks.workunit.client.1.smithi171.stdout:9/890: truncate dd/df/d117/f120 259593 0 2022-01-31T19:44:32.983 INFO:tasks.workunit.client.1.smithi171.stdout:9/891: chown dd/df/d3f/d73/d96/f10b 5 1 2022-01-31T19:44:32.983 INFO:tasks.workunit.client.1.smithi171.stdout:3/984: dread d4/d21/de5/d92/dab/f2b [0,4194304] 0 2022-01-31T19:44:32.984 INFO:tasks.workunit.client.1.smithi171.stdout:9/892: creat dd/d109/de6/d7c/dcf/f12f x:0 0 0 2022-01-31T19:44:32.984 INFO:tasks.workunit.client.1.smithi171.stdout:9/893: dread - dd/df/d3f/d73/f118 zero size 2022-01-31T19:44:32.985 INFO:tasks.workunit.client.1.smithi171.stdout:3/985: mkdir d4/dd/d27/d28/d126/d147 0 2022-01-31T19:44:32.985 INFO:tasks.workunit.client.1.smithi171.stdout:6/813: dwrite dc/d19/d29/fec [0,4194304] 0 2022-01-31T19:44:32.987 INFO:tasks.workunit.client.1.smithi171.stdout:3/986: mkdir d4/dd/d4b/dfa/d12f/d148 0 2022-01-31T19:44:32.988 INFO:tasks.workunit.client.1.smithi171.stdout:8/793: dwrite df/d18/f55 [0,4194304] 0 2022-01-31T19:44:32.989 INFO:tasks.workunit.client.1.smithi171.stdout:4/721: dwrite d2/d3/f28 [0,4194304] 0 2022-01-31T19:44:32.989 INFO:tasks.workunit.client.1.smithi171.stdout:4/722: stat d2/d3/d5/lf 0 2022-01-31T19:44:32.990 INFO:tasks.workunit.client.1.smithi171.stdout:4/723: chown d2/d3/d5/d41/dcf/d12/cf0 7793 1 2022-01-31T19:44:32.990 INFO:tasks.workunit.client.1.smithi171.stdout:4/724: write d2/d3/d5/d41/dcf/d12/db6/fb7 [358476,43775] 0 2022-01-31T19:44:32.990 INFO:tasks.workunit.client.1.smithi171.stdout:4/725: read - d2/d3/d5/dc/f4f zero size 2022-01-31T19:44:32.990 INFO:tasks.workunit.client.1.smithi171.stdout:4/726: creat d2/d3/d5/dc/d4b/d56/ff6 x:0 0 0 2022-01-31T19:44:32.991 INFO:tasks.workunit.client.1.smithi171.stdout:4/727: creat d2/d3/de1/dd4/ff7 x:0 0 0 2022-01-31T19:44:32.991 INFO:tasks.workunit.client.1.smithi171.stdout:8/794: write df/d18/d19/d21/f26 [3360488,121629] 0 2022-01-31T19:44:32.991 INFO:tasks.workunit.client.1.smithi171.stdout:5/834: dwrite d5/d90/db7/d15/d5d/d6f/fdd [0,4194304] 0 2022-01-31T19:44:32.992 INFO:tasks.workunit.client.1.smithi171.stdout:6/814: symlink dc/d19/d29/d27/d37/d111/df0/l113 0 2022-01-31T19:44:32.993 INFO:tasks.workunit.client.1.smithi171.stdout:3/987: rename d4/d21/d3a/l5b to d4/dd/d4b/dfa/d12f/l149 0 2022-01-31T19:44:32.995 INFO:tasks.workunit.client.1.smithi171.stdout:7/650: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f35 [0,4194304] 0 2022-01-31T19:44:32.996 INFO:tasks.workunit.client.1.smithi171.stdout:9/894: dread dd/df/d15/d1a/d64/d6e/fd2 [0,4194304] 0 2022-01-31T19:44:32.996 INFO:tasks.workunit.client.1.smithi171.stdout:9/895: truncate dd/df/d15/d38/d41/f7a 794534 0 2022-01-31T19:44:32.998 INFO:tasks.workunit.client.1.smithi171.stdout:4/728: mkdir d2/d3/d5/d91/de4/df8 0 2022-01-31T19:44:33.000 INFO:tasks.workunit.client.1.smithi171.stdout:8/795: link df/d18/d79/fc7 df/d1a/dae/f10f 0 2022-01-31T19:44:33.001 INFO:tasks.workunit.client.1.smithi171.stdout:8/796: fsync f1 0 2022-01-31T19:44:33.002 INFO:tasks.workunit.client.1.smithi171.stdout:0/897: dwrite d9/db/d22/fd1 [0,4194304] 0 2022-01-31T19:44:33.002 INFO:tasks.workunit.client.1.smithi171.stdout:0/898: fdatasync d9/db/dc/d3f/db9/d7a/fe7 0 2022-01-31T19:44:33.005 INFO:tasks.workunit.client.1.smithi171.stdout:8/797: dread df/d18/f1e [0,4194304] 0 2022-01-31T19:44:33.009 INFO:tasks.workunit.client.1.smithi171.stdout:5/835: unlink d5/lc5 0 2022-01-31T19:44:33.011 INFO:tasks.workunit.client.1.smithi171.stdout:3/988: link d4/d21/c93 d4/d21/de5/d92/dab/dd4/c14a 0 2022-01-31T19:44:33.017 INFO:tasks.workunit.client.1.smithi171.stdout:7/651: rename d1/d1b/ld to d1/d1e/d2a/d58/ldb 0 2022-01-31T19:44:33.017 INFO:tasks.workunit.client.1.smithi171.stdout:7/652: fdatasync d1/d26/d81/f97 0 2022-01-31T19:44:33.017 INFO:tasks.workunit.client.1.smithi171.stdout:7/653: stat d1/d1e/d2a/d29/d31/d30/d3c/d79 0 2022-01-31T19:44:33.019 INFO:tasks.workunit.client.1.smithi171.stdout:4/729: rmdir d2/d3/d5/d91/de4 39 2022-01-31T19:44:33.021 INFO:tasks.workunit.client.1.smithi171.stdout:4/730: write d2/d7f/fa8 [2057536,12863] 0 2022-01-31T19:44:33.028 INFO:tasks.workunit.client.1.smithi171.stdout:0/899: mknod d9/d24/d4c/d35/d67/df4/c147 0 2022-01-31T19:44:33.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/815: getdents dc/d18/d86 0 2022-01-31T19:44:33.030 INFO:tasks.workunit.client.1.smithi171.stdout:6/816: truncate dc/d8b/f96 1702168 0 2022-01-31T19:44:33.030 INFO:tasks.workunit.client.1.smithi171.stdout:6/817: dread - dc/d1e/d3f/f7a zero size 2022-01-31T19:44:33.030 INFO:tasks.workunit.client.1.smithi171.stdout:5/836: dwrite d5/d90/db7/d15/d80/d8d/ff9 [0,4194304] 0 2022-01-31T19:44:33.031 INFO:tasks.workunit.client.1.smithi171.stdout:3/989: rmdir d4/dd/d27/d28/d35/d55 39 2022-01-31T19:44:33.031 INFO:tasks.workunit.client.1.smithi171.stdout:8/798: dwrite df/d27/fa3 [0,4194304] 0 2022-01-31T19:44:33.032 INFO:tasks.workunit.client.1.smithi171.stdout:8/799: stat df/f16 0 2022-01-31T19:44:33.032 INFO:tasks.workunit.client.1.smithi171.stdout:7/654: truncate d1/f16 584196 0 2022-01-31T19:44:33.033 INFO:tasks.workunit.client.1.smithi171.stdout:8/800: write df/d1a/dae/f10f [7277,108046] 0 2022-01-31T19:44:33.033 INFO:tasks.workunit.client.1.smithi171.stdout:0/900: mknod d9/d24/d4c/d101/c148 0 2022-01-31T19:44:33.034 INFO:tasks.workunit.client.1.smithi171.stdout:6/818: rmdir dc/d19/d30/d3c/d105 39 2022-01-31T19:44:33.035 INFO:tasks.workunit.client.1.smithi171.stdout:3/990: dread d4/d21/de5/d92/dab/d52/f96 [0,4194304] 0 2022-01-31T19:44:33.038 INFO:tasks.workunit.client.1.smithi171.stdout:9/896: write dd/df/d15/f2f [1153303,18414] 0 2022-01-31T19:44:33.043 INFO:tasks.workunit.client.1.smithi171.stdout:5/837: getdents d5/d90/db7/d15/d80 0 2022-01-31T19:44:33.046 INFO:tasks.workunit.client.1.smithi171.stdout:7/655: symlink d1/d1e/d2a/d29/ldc 0 2022-01-31T19:44:33.047 INFO:tasks.workunit.client.1.smithi171.stdout:7/656: write d1/d1e/d2a/d29/d31/d30/d3c/d7e/fc7 [764355,57721] 0 2022-01-31T19:44:33.047 INFO:tasks.workunit.client.1.smithi171.stdout:7/657: fsync d1/d1b/f4b 0 2022-01-31T19:44:33.047 INFO:tasks.workunit.client.1.smithi171.stdout:7/658: chown d1/d1e/d2a 89 1 2022-01-31T19:44:33.048 INFO:tasks.workunit.client.1.smithi171.stdout:5/838: write d5/fb5 [63333,99724] 0 2022-01-31T19:44:33.049 INFO:tasks.workunit.client.1.smithi171.stdout:4/731: mknod d2/d3/cf9 0 2022-01-31T19:44:33.049 INFO:tasks.workunit.client.1.smithi171.stdout:5/839: write d5/d90/db7/d15/f42 [26971,77139] 0 2022-01-31T19:44:33.049 INFO:tasks.workunit.client.1.smithi171.stdout:5/840: creat d5/d59/da0/f11e x:0 0 0 2022-01-31T19:44:33.049 INFO:tasks.workunit.client.1.smithi171.stdout:5/841: truncate d5/d90/db7/d15/d23/d6b/f7e 502134 0 2022-01-31T19:44:33.050 INFO:tasks.workunit.client.1.smithi171.stdout:5/842: dread - d5/d90/db7/d15/d5d/f81 zero size 2022-01-31T19:44:33.050 INFO:tasks.workunit.client.1.smithi171.stdout:8/801: creat df/d18/d19/d21/d67/f110 x:0 0 0 2022-01-31T19:44:33.050 INFO:tasks.workunit.client.1.smithi171.stdout:8/802: dread - df/d5d/d6c/fbf zero size 2022-01-31T19:44:33.051 INFO:tasks.workunit.client.1.smithi171.stdout:0/901: mknod d9/d24/d92/d9d/dc6/d115/c149 0 2022-01-31T19:44:33.051 INFO:tasks.workunit.client.1.smithi171.stdout:6/819: mknod dc/d18/c114 0 2022-01-31T19:44:33.052 INFO:tasks.workunit.client.1.smithi171.stdout:6/820: creat dc/d19/d29/f115 x:0 0 0 2022-01-31T19:44:33.052 INFO:tasks.workunit.client.1.smithi171.stdout:3/991: symlink d4/dd/d27/d4e/d62/d11b/l14b 0 2022-01-31T19:44:33.053 INFO:tasks.workunit.client.1.smithi171.stdout:9/897: mknod dd/df/d15/d1a/d64/d6e/ddd/d110/c130 0 2022-01-31T19:44:33.053 INFO:tasks.workunit.client.1.smithi171.stdout:9/898: fdatasync dd/df/d3f/d73/d96/fae 0 2022-01-31T19:44:33.054 INFO:tasks.workunit.client.1.smithi171.stdout:8/803: dread df/d27/d85/f8e [0,4194304] 0 2022-01-31T19:44:33.055 INFO:tasks.workunit.client.1.smithi171.stdout:7/659: unlink d1/d1e/d2a/d29/ldc 0 2022-01-31T19:44:33.080 INFO:tasks.workunit.client.1.smithi171.stdout:8/804: dwrite df/d27/d7b/ffa [0,4194304] 0 2022-01-31T19:44:33.080 INFO:tasks.workunit.client.1.smithi171.stdout:8/805: chown df/d1a/de8 27 1 2022-01-31T19:44:33.099 INFO:tasks.workunit.client.1.smithi171.stdout:4/732: rename d2/d3/d5/dc/d4b/d56/ff6 to d2/d3/d5/d41/dcf/d12/d15/d57/dad/ffa 0 2022-01-31T19:44:33.104 INFO:tasks.workunit.client.1.smithi171.stdout:6/821: unlink c8 0 2022-01-31T19:44:33.105 INFO:tasks.workunit.client.1.smithi171.stdout:3/992: creat d4/dd/d87/f14c x:0 0 0 2022-01-31T19:44:33.109 INFO:tasks.workunit.client.1.smithi171.stdout:9/899: link dd/df/d15/d38/c7b dd/df/d15/d1a/d84/dfe/c131 0 2022-01-31T19:44:33.109 INFO:tasks.workunit.client.1.smithi171.stdout:9/900: dread - dd/df/d15/d38/d4d/ffd zero size 2022-01-31T19:44:33.109 INFO:tasks.workunit.client.1.smithi171.stdout:9/901: chown dd/df/d15/d53/fbf 86792 1 2022-01-31T19:44:33.110 INFO:tasks.workunit.client.1.smithi171.stdout:7/660: dread - d1/d1e/d2a/d29/d31/f99 zero size 2022-01-31T19:44:33.114 INFO:tasks.workunit.client.1.smithi171.stdout:8/806: creat df/d2a/dfb/f111 x:0 0 0 2022-01-31T19:44:33.120 INFO:tasks.workunit.client.1.smithi171.stdout:4/733: fsync d2/d3/f44 0 2022-01-31T19:44:33.120 INFO:tasks.workunit.client.1.smithi171.stdout:3/993: getdents d4/d21/de5/da7/dfb 0 2022-01-31T19:44:33.120 INFO:tasks.workunit.client.1.smithi171.stdout:4/734: fsync d2/d3/d5/d41/f47 0 2022-01-31T19:44:33.121 INFO:tasks.workunit.client.1.smithi171.stdout:0/902: rmdir d9/d24/d92/d9d/dc6/dfb 39 2022-01-31T19:44:33.121 INFO:tasks.workunit.client.1.smithi171.stdout:0/903: write d9/db/dc2/f138 [1563133,34663] 0 2022-01-31T19:44:33.121 INFO:tasks.workunit.client.1.smithi171.stdout:0/904: dread - d9/d24/f26 zero size 2022-01-31T19:44:33.122 INFO:tasks.workunit.client.1.smithi171.stdout:4/735: write d2/d3/d5/d41/dcf/d12/d15/f78 [1839644,125857] 0 2022-01-31T19:44:33.122 INFO:tasks.workunit.client.1.smithi171.stdout:4/736: truncate d2/d3/d5/d41/d6b/fc8 567673 0 2022-01-31T19:44:33.123 INFO:tasks.workunit.client.1.smithi171.stdout:3/994: link d4/dd/d27/lb1 d4/dd/d87/l14d 0 2022-01-31T19:44:33.123 INFO:tasks.workunit.client.1.smithi171.stdout:3/995: truncate d4/dd/d27/d28/d35/d55/d6c/f102 527307 0 2022-01-31T19:44:33.126 INFO:tasks.workunit.client.1.smithi171.stdout:9/902: dread dd/df/d15/f4a [0,4194304] 0 2022-01-31T19:44:33.131 INFO:tasks.workunit.client.1.smithi171.stdout:6/822: dwrite dc/d1e/f45 [0,4194304] 0 2022-01-31T19:44:33.132 INFO:tasks.workunit.client.1.smithi171.stdout:6/823: read - dc/d19/d29/daf/fed zero size 2022-01-31T19:44:33.132 INFO:tasks.workunit.client.1.smithi171.stdout:7/661: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f21 [0,4194304] 0 2022-01-31T19:44:33.144 INFO:tasks.workunit.client.1.smithi171.stdout:3/996: dwrite d4/dd/d27/d28/d35/f48 [0,4194304] 0 2022-01-31T19:44:33.164 INFO:tasks.workunit.client.1.smithi171.stdout:3/997: dwrite d4/d21/d3a/db5/dbe/dc3/f114 [0,4194304] 0 2022-01-31T19:44:33.180 INFO:tasks.workunit.client.1.smithi171.stdout:8/807: getdents df/d18/d19 0 2022-01-31T19:44:33.182 INFO:tasks.workunit.client.1.smithi171.stdout:4/737: creat d2/d3/d5/d70/dee/ffb x:0 0 0 2022-01-31T19:44:33.183 INFO:tasks.workunit.client.1.smithi171.stdout:8/808: write df/d18/d19/d21/dab/f56 [1120483,112405] 0 2022-01-31T19:44:33.184 INFO:tasks.workunit.client.1.smithi171.stdout:9/903: creat dd/df/d117/f132 x:0 0 0 2022-01-31T19:44:33.205 INFO:tasks.workunit.client.1.smithi171.stdout:4/738: dwrite d2/fa2 [0,4194304] 0 2022-01-31T19:44:33.205 INFO:tasks.workunit.client.1.smithi171.stdout:4/739: chown d2/l4d 0 1 2022-01-31T19:44:33.211 INFO:tasks.workunit.client.1.smithi171.stdout:0/905: symlink d9/db/dc/d3f/db9/d7a/l14a 0 2022-01-31T19:44:33.221 INFO:tasks.workunit.client.1.smithi171.stdout:7/662: rename d1/d1e/d2a/d29/f43 to d1/d1e/fdd 0 2022-01-31T19:44:33.221 INFO:tasks.workunit.client.1.smithi171.stdout:7/663: write d1/d1b/f4c [90485,104700] 0 2022-01-31T19:44:33.221 INFO:tasks.workunit.client.1.smithi171.stdout:7/664: write d1/d1e/d2a/d29/d31/d30/d3c/fda [798315,119184] 0 2022-01-31T19:44:33.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/665: fdatasync d1/d1e/d2a/d29/d31/f6a 0 2022-01-31T19:44:33.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/666: creat d1/d26/d57/fde x:0 0 0 2022-01-31T19:44:33.228 INFO:tasks.workunit.client.1.smithi171.stdout:4/740: dwrite d2/d3/d5/d41/dcf/f27 [0,4194304] 0 2022-01-31T19:44:33.229 INFO:tasks.workunit.client.1.smithi171.stdout:4/741: write d2/d3/d5/f2e [2764041,15035] 0 2022-01-31T19:44:33.229 INFO:tasks.workunit.client.1.smithi171.stdout:4/742: chown d2/f14 1288063 1 2022-01-31T19:44:33.230 INFO:tasks.workunit.client.1.smithi171.stdout:4/743: creat d2/d3/d5/d70/dee/ffc x:0 0 0 2022-01-31T19:44:33.303 INFO:tasks.workunit.client.1.smithi171.stdout:9/904: mknod dd/df/d15/d38/d4d/c133 0 2022-01-31T19:44:33.303 INFO:tasks.workunit.client.1.smithi171.stdout:9/905: dread - dd/df/d3f/d73/d96/f107 zero size 2022-01-31T19:44:33.304 INFO:tasks.workunit.client.1.smithi171.stdout:9/906: chown dd/d109/de6/db5/l12b 60 1 2022-01-31T19:44:33.307 INFO:tasks.workunit.client.1.smithi171.stdout:0/906: getdents d9/db/dc/d3f/d6f 0 2022-01-31T19:44:33.308 INFO:tasks.workunit.client.1.smithi171.stdout:6/824: getdents dc 0 2022-01-31T19:44:33.308 INFO:tasks.workunit.client.1.smithi171.stdout:6/825: getdents dc/d21/de9 0 2022-01-31T19:44:33.311 INFO:tasks.workunit.client.1.smithi171.stdout:3/998: rename d4/d21/d3a/db5/dbe/dd1/cd9 to d4/dd/d39/c14e 0 2022-01-31T19:44:33.311 INFO:tasks.workunit.client.1.smithi171.stdout:3/999: stat d4/d21/d3a/db5/dbe/dd1/d115/d141 0 2022-01-31T19:44:33.311 INFO:tasks.workunit.client.1.smithi171.stdout:1/876: sync 2022-01-31T19:44:33.311 INFO:tasks.workunit.client.1.smithi171.stdout:5/843: sync 2022-01-31T19:44:33.312 INFO:tasks.workunit.client.1.smithi171.stdout:5/844: write d5/d59/f94 [577668,49033] 0 2022-01-31T19:44:33.317 INFO:tasks.workunit.client.1.smithi171.stdout:4/744: creat d2/d3/d5/d91/da6/ffd x:0 0 0 2022-01-31T19:44:33.317 INFO:tasks.workunit.client.1.smithi171.stdout:4/745: write d2/d3/d5/d70/fa7 [1813378,36748] 0 2022-01-31T19:44:33.319 INFO:tasks.workunit.client.1.smithi171.stdout:1/877: link dd/df/d19/lb3 dd/d23/l106 0 2022-01-31T19:44:33.321 INFO:tasks.workunit.client.1.smithi171.stdout:5/845: symlink d5/d90/db7/d15/d23/d3d/l11f 0 2022-01-31T19:44:33.322 INFO:tasks.workunit.client.1.smithi171.stdout:7/667: rename d1/d1e/d2a/d29/d31/f51 to d1/d1e/d2a/d29/d31/d30/d3c/d7e/dd1/fdf 0 2022-01-31T19:44:33.325 INFO:tasks.workunit.client.1.smithi171.stdout:4/746: truncate d2/f9e 1264656 0 2022-01-31T19:44:33.325 INFO:tasks.workunit.client.1.smithi171.stdout:1/878: write dd/df/d84/dea/d35/f7d [1819604,83875] 0 2022-01-31T19:44:33.325 INFO:tasks.workunit.client.1.smithi171.stdout:1/879: dread - dd/df/d19/f96 zero size 2022-01-31T19:44:33.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/880: fsync dd/d23/d2e/d5e/fe0 0 2022-01-31T19:44:33.326 INFO:tasks.workunit.client.1.smithi171.stdout:1/881: chown dd/d23/d74/dcb/ddb/f102 2188 1 2022-01-31T19:44:33.326 INFO:tasks.workunit.client.1.smithi171.stdout:5/846: read d5/d90/db7/d15/d80/fc3 [305871,61956] 0 2022-01-31T19:44:33.326 INFO:tasks.workunit.client.1.smithi171.stdout:5/847: write d5/d90/fe8 [4251325,3403] 0 2022-01-31T19:44:33.327 INFO:tasks.workunit.client.1.smithi171.stdout:1/882: symlink dd/df/d84/dea/l107 0 2022-01-31T19:44:33.327 INFO:tasks.workunit.client.1.smithi171.stdout:5/848: creat d5/d90/db7/d15/d80/f120 x:0 0 0 2022-01-31T19:44:33.328 INFO:tasks.workunit.client.1.smithi171.stdout:1/883: rmdir dd 39 2022-01-31T19:44:33.329 INFO:tasks.workunit.client.1.smithi171.stdout:5/849: link d5/d90/db7/dd/lcc d5/d90/db7/d15/d23/d58/d9e/l121 0 2022-01-31T19:44:33.330 INFO:tasks.workunit.client.1.smithi171.stdout:1/884: truncate dd/d23/f3f 470593 0 2022-01-31T19:44:33.330 INFO:tasks.workunit.client.1.smithi171.stdout:1/885: stat dd/df/d84/dea/d64/fc7 0 2022-01-31T19:44:33.338 INFO:tasks.workunit.client.1.smithi171.stdout:9/907: dwrite dd/df/d15/d1a/d64/fc7 [0,4194304] 0 2022-01-31T19:44:33.339 INFO:tasks.workunit.client.1.smithi171.stdout:7/668: dread d1/d1b/fc [4194304,4194304] 0 2022-01-31T19:44:33.340 INFO:tasks.workunit.client.1.smithi171.stdout:0/907: dwrite d9/db/fc9 [0,4194304] 0 2022-01-31T19:44:33.341 INFO:tasks.workunit.client.1.smithi171.stdout:7/669: getdents d1/d1e/d2a/d29/d31/d30/d3c/d5f 0 2022-01-31T19:44:33.342 INFO:tasks.workunit.client.1.smithi171.stdout:7/670: dread - d1/d1e/d2a/d29/d31/d30/d6b/fbf zero size 2022-01-31T19:44:33.342 INFO:tasks.workunit.client.1.smithi171.stdout:7/671: write d1/f16 [990204,62146] 0 2022-01-31T19:44:33.344 INFO:tasks.workunit.client.1.smithi171.stdout:6/826: dwrite dc/d8b/dee/f10b [0,4194304] 0 2022-01-31T19:44:33.347 INFO:tasks.workunit.client.1.smithi171.stdout:9/908: dread f9 [0,4194304] 0 2022-01-31T19:44:33.348 INFO:tasks.workunit.client.1.smithi171.stdout:9/909: write dd/df/d3f/d73/f11a [828129,15106] 0 2022-01-31T19:44:33.351 INFO:tasks.workunit.client.1.smithi171.stdout:0/908: rename d9/db/c1c to d9/d24/d100/c14b 0 2022-01-31T19:44:33.352 INFO:tasks.workunit.client.1.smithi171.stdout:4/747: dwrite d2/d3/d5/d70/dee/ffc [0,4194304] 0 2022-01-31T19:44:33.353 INFO:tasks.workunit.client.1.smithi171.stdout:7/672: creat d1/d1e/d2a/d29/d31/d30/d6b/fe0 x:0 0 0 2022-01-31T19:44:33.358 INFO:tasks.workunit.client.1.smithi171.stdout:9/910: unlink dd/df/cc9 0 2022-01-31T19:44:33.359 INFO:tasks.workunit.client.1.smithi171.stdout:9/911: creat dd/df/d15/d1a/d64/f134 x:0 0 0 2022-01-31T19:44:33.359 INFO:tasks.workunit.client.1.smithi171.stdout:0/909: truncate d9/db/f131 1889201 0 2022-01-31T19:44:33.363 INFO:tasks.workunit.client.1.smithi171.stdout:9/912: truncate dd/df/d15/d53/fc4 2717214 0 2022-01-31T19:44:33.364 INFO:tasks.workunit.client.1.smithi171.stdout:7/673: dread d1/d1b/fc [0,4194304] 0 2022-01-31T19:44:33.364 INFO:tasks.workunit.client.1.smithi171.stdout:0/910: getdents d9/db/d22/d73/ddc 0 2022-01-31T19:44:33.365 INFO:tasks.workunit.client.1.smithi171.stdout:7/674: rmdir d1/d1e/d2a/d29/d31/d30/d3c 39 2022-01-31T19:44:33.366 INFO:tasks.workunit.client.1.smithi171.stdout:9/913: getdents dd/d109/de6/d7c/dcf 0 2022-01-31T19:44:33.369 INFO:tasks.workunit.client.1.smithi171.stdout:9/914: mkdir dd/df/d15/d38/d4d/d80/d135 0 2022-01-31T19:44:33.369 INFO:tasks.workunit.client.1.smithi171.stdout:7/675: truncate d1/d1e/d2a/d29/d31/d30/d3c/f56 96126 0 2022-01-31T19:44:33.371 INFO:tasks.workunit.client.1.smithi171.stdout:7/676: read d1/d1e/d2a/d29/d31/d30/d6b/f86 [2087308,19388] 0 2022-01-31T19:44:33.371 INFO:tasks.workunit.client.1.smithi171.stdout:7/677: unlink d1/d1e/d2a/d29/f7a 0 2022-01-31T19:44:33.371 INFO:tasks.workunit.client.1.smithi171.stdout:7/678: creat d1/d1e/d2a/d29/d31/d30/d3c/d7e/fe1 x:0 0 0 2022-01-31T19:44:33.371 INFO:tasks.workunit.client.1.smithi171.stdout:7/679: write d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/fd2 [868157,127327] 0 2022-01-31T19:44:33.376 INFO:tasks.workunit.client.1.smithi171.stdout:9/915: creat dd/f136 x:0 0 0 2022-01-31T19:44:33.382 INFO:tasks.workunit.client.1.smithi171.stdout:9/916: truncate dd/f37 7115685 0 2022-01-31T19:44:33.382 INFO:tasks.workunit.client.1.smithi171.stdout:9/917: creat dd/df/d15/d1a/d64/f137 x:0 0 0 2022-01-31T19:44:33.383 INFO:tasks.workunit.client.1.smithi171.stdout:5/850: dwrite d5/d59/f94 [0,4194304] 0 2022-01-31T19:44:33.383 INFO:tasks.workunit.client.1.smithi171.stdout:8/809: sync 2022-01-31T19:44:33.384 INFO:tasks.workunit.client.1.smithi171.stdout:4/748: dwrite d2/d3/d5/d41/d6b/fa9 [0,4194304] 0 2022-01-31T19:44:33.386 INFO:tasks.workunit.client.1.smithi171.stdout:6/827: dwrite dc/d19/dae/fdf [0,4194304] 0 2022-01-31T19:44:33.389 INFO:tasks.workunit.client.1.smithi171.stdout:7/680: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f28 [0,4194304] 0 2022-01-31T19:44:33.392 INFO:tasks.workunit.client.1.smithi171.stdout:5/851: creat d5/d59/f122 x:0 0 0 2022-01-31T19:44:33.394 INFO:tasks.workunit.client.1.smithi171.stdout:8/810: creat df/d27/d5c/f112 x:0 0 0 2022-01-31T19:44:33.399 INFO:tasks.workunit.client.1.smithi171.stdout:6/828: mkdir dc/d8b/dee/d116 0 2022-01-31T19:44:33.401 INFO:tasks.workunit.client.1.smithi171.stdout:7/681: dread d1/d1b/f22 [0,4194304] 0 2022-01-31T19:44:33.402 INFO:tasks.workunit.client.1.smithi171.stdout:9/918: getdents dd/df/d15/d38/d41 0 2022-01-31T19:44:33.402 INFO:tasks.workunit.client.1.smithi171.stdout:9/919: getdents dd/df/d86/d11d 0 2022-01-31T19:44:33.402 INFO:tasks.workunit.client.1.smithi171.stdout:9/920: dread - dd/df/d117/f132 zero size 2022-01-31T19:44:33.407 INFO:tasks.workunit.client.1.smithi171.stdout:4/749: dwrite d2/f90 [0,4194304] 0 2022-01-31T19:44:33.407 INFO:tasks.workunit.client.1.smithi171.stdout:5/852: creat d5/d90/db7/d15/d80/d109/f123 x:0 0 0 2022-01-31T19:44:33.408 INFO:tasks.workunit.client.1.smithi171.stdout:5/853: readlink d5/d90/db7/d15/d23/d58/lef 0 2022-01-31T19:44:33.408 INFO:tasks.workunit.client.1.smithi171.stdout:5/854: write d5/d90/db7/d15/f42 [1022985,83037] 0 2022-01-31T19:44:33.408 INFO:tasks.workunit.client.1.smithi171.stdout:8/811: rmdir df/d18/d79 39 2022-01-31T19:44:33.408 INFO:tasks.workunit.client.1.smithi171.stdout:8/812: fdatasync df/d18/d79/d7a/f10b 0 2022-01-31T19:44:33.409 INFO:tasks.workunit.client.1.smithi171.stdout:6/829: truncate dc/d19/d29/d27/f3a 361849 0 2022-01-31T19:44:33.410 INFO:tasks.workunit.client.1.smithi171.stdout:6/830: chown dc/d19/df4/lf7 323059 1 2022-01-31T19:44:33.410 INFO:tasks.workunit.client.1.smithi171.stdout:6/831: dread - dc/d19/d30/fda zero size 2022-01-31T19:44:33.410 INFO:tasks.workunit.client.1.smithi171.stdout:6/832: write dc/d19/d29/f34 [1878971,7405] 0 2022-01-31T19:44:33.411 INFO:tasks.workunit.client.1.smithi171.stdout:6/833: chown dc/d19/d29/d27/f3a 4 1 2022-01-31T19:44:33.412 INFO:tasks.workunit.client.1.smithi171.stdout:7/682: creat d1/d1e/d2a/d29/d31/d30/d3c/dc0/fe2 x:0 0 0 2022-01-31T19:44:33.413 INFO:tasks.workunit.client.1.smithi171.stdout:7/683: write d1/d26/d81/f97 [578432,62450] 0 2022-01-31T19:44:33.413 INFO:tasks.workunit.client.1.smithi171.stdout:6/834: read dc/d18/d86/f87 [1411376,118963] 0 2022-01-31T19:44:33.413 INFO:tasks.workunit.client.1.smithi171.stdout:6/835: fsync dc/d19/d29/d27/d37/d68/f10a 0 2022-01-31T19:44:33.417 INFO:tasks.workunit.client.1.smithi171.stdout:4/750: readlink d2/d3/d5/d41/dcf/l58 0 2022-01-31T19:44:33.418 INFO:tasks.workunit.client.1.smithi171.stdout:5/855: mkdir d5/d90/db7/d15/d124 0 2022-01-31T19:44:33.420 INFO:tasks.workunit.client.1.smithi171.stdout:5/856: write d5/f8 [1750868,99907] 0 2022-01-31T19:44:33.420 INFO:tasks.workunit.client.1.smithi171.stdout:5/857: chown d5/d90/db7/d15/d80/f120 108106636 1 2022-01-31T19:44:33.420 INFO:tasks.workunit.client.1.smithi171.stdout:8/813: mkdir df/d5d/d6c/d113 0 2022-01-31T19:44:33.425 INFO:tasks.workunit.client.1.smithi171.stdout:6/836: mknod dc/d19/d29/d27/d37/d3b/d67/c117 0 2022-01-31T19:44:33.425 INFO:tasks.workunit.client.1.smithi171.stdout:9/921: getdents dd/df/d15/d1a/d64/dfc 0 2022-01-31T19:44:33.426 INFO:tasks.workunit.client.1.smithi171.stdout:9/922: dread - dd/d109/de6/da8/f119 zero size 2022-01-31T19:44:33.426 INFO:tasks.workunit.client.1.smithi171.stdout:9/923: unlink dd/df/f20 0 2022-01-31T19:44:33.427 INFO:tasks.workunit.client.1.smithi171.stdout:4/751: mkdir d2/d3/d5/d41/dcf/d12/d15/dc4/dfe 0 2022-01-31T19:44:33.427 INFO:tasks.workunit.client.1.smithi171.stdout:4/752: creat d2/d3/d5/dc/d4b/dbc/fff x:0 0 0 2022-01-31T19:44:33.428 INFO:tasks.workunit.client.1.smithi171.stdout:4/753: write d2/d3/d5/d41/dcf/d12/f60 [556831,47075] 0 2022-01-31T19:44:33.428 INFO:tasks.workunit.client.1.smithi171.stdout:5/858: creat d5/d90/db7/d15/d5d/d107/f125 x:0 0 0 2022-01-31T19:44:33.429 INFO:tasks.workunit.client.1.smithi171.stdout:8/814: link df/d18/d79/f8c df/d18/d19/d21/d67/df3/f114 0 2022-01-31T19:44:33.431 INFO:tasks.workunit.client.1.smithi171.stdout:7/684: rmdir d1/d1b 39 2022-01-31T19:44:33.433 INFO:tasks.workunit.client.1.smithi171.stdout:6/837: unlink dc/d19/d29/d27/d37/d111/f7e 0 2022-01-31T19:44:33.434 INFO:tasks.workunit.client.1.smithi171.stdout:6/838: readlink dc/d19/d29/d27/d37/d111/ld8 0 2022-01-31T19:44:33.434 INFO:tasks.workunit.client.1.smithi171.stdout:6/839: write dc/d21/fa2 [1063952,38585] 0 2022-01-31T19:44:33.436 INFO:tasks.workunit.client.1.smithi171.stdout:9/924: creat dd/df/f138 x:0 0 0 2022-01-31T19:44:33.439 INFO:tasks.workunit.client.1.smithi171.stdout:4/754: link d2/d3/d5/d91/ca4 d2/d3/d5/d41/dcf/d12/d15/dc4/c100 0 2022-01-31T19:44:33.440 INFO:tasks.workunit.client.1.smithi171.stdout:5/859: creat d5/d90/db7/dd/d66/f126 x:0 0 0 2022-01-31T19:44:33.441 INFO:tasks.workunit.client.1.smithi171.stdout:5/860: dread - d5/d90/db7/d15/d23/d58/fa3 zero size 2022-01-31T19:44:33.441 INFO:tasks.workunit.client.1.smithi171.stdout:8/815: symlink df/d18/d19/d82/dd3/l115 0 2022-01-31T19:44:33.443 INFO:tasks.workunit.client.1.smithi171.stdout:7/685: symlink d1/d1e/d2a/d29/d77/dab/dd5/le3 0 2022-01-31T19:44:33.443 INFO:tasks.workunit.client.1.smithi171.stdout:7/686: readlink d1/d1e/d2a/l54 0 2022-01-31T19:44:33.443 INFO:tasks.workunit.client.1.smithi171.stdout:7/687: fdatasync d1/d1e/d2a/d29/d31/d30/d3c/d79/f8d 0 2022-01-31T19:44:33.445 INFO:tasks.workunit.client.1.smithi171.stdout:6/840: link dc/d19/d30/d3c/fc7 dc/d19/d29/daf/ddb/f118 0 2022-01-31T19:44:33.446 INFO:tasks.workunit.client.1.smithi171.stdout:6/841: dread - dc/d21/d4f/fb3 zero size 2022-01-31T19:44:33.448 INFO:tasks.workunit.client.1.smithi171.stdout:4/755: rename d2/d3/d5/d41/dcf/d12/f2a to d2/d3/d5/d91/d77/f101 0 2022-01-31T19:44:33.449 INFO:tasks.workunit.client.1.smithi171.stdout:8/816: dread df/d18/f1e [4194304,4194304] 0 2022-01-31T19:44:33.450 INFO:tasks.workunit.client.1.smithi171.stdout:5/861: symlink d5/d90/dd1/d10a/l127 0 2022-01-31T19:44:33.450 INFO:tasks.workunit.client.1.smithi171.stdout:5/862: write d5/d90/db7/d15/d80/fdc [39672,31065] 0 2022-01-31T19:44:33.450 INFO:tasks.workunit.client.1.smithi171.stdout:5/863: readlink d5/d90/db7/d15/lf1 0 2022-01-31T19:44:33.450 INFO:tasks.workunit.client.1.smithi171.stdout:5/864: write d5/d90/db7/dd/d66/f126 [93466,89712] 0 2022-01-31T19:44:33.451 INFO:tasks.workunit.client.1.smithi171.stdout:5/865: fdatasync d5/d90/db7/f46 0 2022-01-31T19:44:33.452 INFO:tasks.workunit.client.1.smithi171.stdout:6/842: rmdir dc/d19/d30/d3c/d105/d109 0 2022-01-31T19:44:33.453 INFO:tasks.workunit.client.1.smithi171.stdout:5/866: mknod d5/d59/da0/c128 0 2022-01-31T19:44:33.453 INFO:tasks.workunit.client.1.smithi171.stdout:5/867: chown d5/d90/db7/d15/d23/d67/c28 2509 1 2022-01-31T19:44:33.458 INFO:tasks.workunit.client.1.smithi171.stdout:5/868: write d5/d90/db7/d15/d5d/fb9 [3461568,54314] 0 2022-01-31T19:44:33.458 INFO:tasks.workunit.client.1.smithi171.stdout:5/869: chown d5/d90/db7/d15/d23 863896419 1 2022-01-31T19:44:33.472 INFO:tasks.workunit.client.1.smithi171.stdout:9/925: dwrite dd/df/d15/d53/fa6 [0,4194304] 0 2022-01-31T19:44:33.491 INFO:tasks.workunit.client.1.smithi171.stdout:8/817: dwrite df/d18/d19/d46/f100 [0,4194304] 0 2022-01-31T19:44:33.493 INFO:tasks.workunit.client.1.smithi171.stdout:7/688: dwrite d1/d1e/d2a/d29/d31/d30/f73 [0,4194304] 0 2022-01-31T19:44:33.494 INFO:tasks.workunit.client.1.smithi171.stdout:6/843: dwrite dc/d8b/dee/f10b [0,4194304] 0 2022-01-31T19:44:33.494 INFO:tasks.workunit.client.1.smithi171.stdout:4/756: dwrite d2/d3/d5/d41/d6b/fa9 [0,4194304] 0 2022-01-31T19:44:33.495 INFO:tasks.workunit.client.1.smithi171.stdout:8/818: rename df/d27/d85 to df/d18/dad/d116 0 2022-01-31T19:44:33.495 INFO:tasks.workunit.client.1.smithi171.stdout:9/926: dwrite dd/df/d15/d1a/fb2 [0,4194304] 0 2022-01-31T19:44:33.496 INFO:tasks.workunit.client.1.smithi171.stdout:9/927: chown dd/df/d15/d1a/d84/faf 988735 1 2022-01-31T19:44:33.496 INFO:tasks.workunit.client.1.smithi171.stdout:7/689: mkdir d1/d1e/d2a/d29/d77/de4 0 2022-01-31T19:44:33.507 INFO:tasks.workunit.client.1.smithi171.stdout:4/757: creat d2/d3/d5/d91/de4/f102 x:0 0 0 2022-01-31T19:44:33.507 INFO:tasks.workunit.client.1.smithi171.stdout:4/758: readlink d2/d3/d5/d70/l9c 0 2022-01-31T19:44:33.515 INFO:tasks.workunit.client.1.smithi171.stdout:8/819: mknod df/d18/d19/db2/c117 0 2022-01-31T19:44:33.517 INFO:tasks.workunit.client.1.smithi171.stdout:9/928: creat dd/df/d15/de4/f139 x:0 0 0 2022-01-31T19:44:33.517 INFO:tasks.workunit.client.1.smithi171.stdout:9/929: dread - dd/d109/de6/db5/fbe zero size 2022-01-31T19:44:33.518 INFO:tasks.workunit.client.1.smithi171.stdout:8/820: mkdir df/d18/d19/d46/dd6/d118 0 2022-01-31T19:44:33.519 INFO:tasks.workunit.client.1.smithi171.stdout:8/821: fsync df/d27/fd5 0 2022-01-31T19:44:33.519 INFO:tasks.workunit.client.1.smithi171.stdout:8/822: unlink df/d5d/d6c/c72 0 2022-01-31T19:44:33.520 INFO:tasks.workunit.client.1.smithi171.stdout:6/844: rename dc/d19/d29/d27/d37 to dc/d18/d86/d119 0 2022-01-31T19:44:33.521 INFO:tasks.workunit.client.1.smithi171.stdout:6/845: chown dc/d21/de9 169864587 1 2022-01-31T19:44:33.521 INFO:tasks.workunit.client.1.smithi171.stdout:9/930: truncate dd/f6a 593531 0 2022-01-31T19:44:33.522 INFO:tasks.workunit.client.1.smithi171.stdout:9/931: write dd/d109/ff8 [3484731,829] 0 2022-01-31T19:44:33.524 INFO:tasks.workunit.client.1.smithi171.stdout:8/823: symlink df/d18/dad/d116/df5/l119 0 2022-01-31T19:44:33.524 INFO:tasks.workunit.client.1.smithi171.stdout:8/824: write df/d2a/d30/f63 [862444,22906] 0 2022-01-31T19:44:33.524 INFO:tasks.workunit.client.1.smithi171.stdout:8/825: truncate df/d18/d19/d36/f6d 822119 0 2022-01-31T19:44:33.525 INFO:tasks.workunit.client.1.smithi171.stdout:8/826: write df/f4f [1211127,84762] 0 2022-01-31T19:44:33.526 INFO:tasks.workunit.client.1.smithi171.stdout:6/846: mknod dc/d1e/c11a 0 2022-01-31T19:44:33.526 INFO:tasks.workunit.client.1.smithi171.stdout:6/847: readlink dc/d19/d29/d27/l49 0 2022-01-31T19:44:33.528 INFO:tasks.workunit.client.1.smithi171.stdout:9/932: symlink dd/df/d3f/d73/d96/l13a 0 2022-01-31T19:44:33.531 INFO:tasks.workunit.client.1.smithi171.stdout:8/827: symlink df/d18/d19/d46/dbc/l11a 0 2022-01-31T19:44:33.532 INFO:tasks.workunit.client.1.smithi171.stdout:6/848: symlink dc/d1e/d3f/l11b 0 2022-01-31T19:44:33.534 INFO:tasks.workunit.client.1.smithi171.stdout:9/933: mkdir dd/df/d15/d38/d13b 0 2022-01-31T19:44:33.536 INFO:tasks.workunit.client.1.smithi171.stdout:8/828: creat df/d18/d19/d46/f11b x:0 0 0 2022-01-31T19:44:33.538 INFO:tasks.workunit.client.1.smithi171.stdout:6/849: link dc/d1e/d3f/cf1 dc/d19/d30/d3c/d100/c11c 0 2022-01-31T19:44:33.538 INFO:tasks.workunit.client.1.smithi171.stdout:6/850: dread - dc/d18/d86/d119/d111/f58 zero size 2022-01-31T19:44:33.540 INFO:tasks.workunit.client.1.smithi171.stdout:9/934: rename dd/f81 to dd/df/d15/d38/d4d/f13c 0 2022-01-31T19:44:33.541 INFO:tasks.workunit.client.1.smithi171.stdout:4/759: dwrite d2/d3/d5/d41/dcf/dc7/fcd [0,4194304] 0 2022-01-31T19:44:33.541 INFO:tasks.workunit.client.1.smithi171.stdout:4/760: fsync d2/d3/d5/d41/d6b/f99 0 2022-01-31T19:44:33.541 INFO:tasks.workunit.client.1.smithi171.stdout:4/761: write d2/d3/d5/d41/dcf/d12/f60 [1106406,77650] 0 2022-01-31T19:44:33.542 INFO:tasks.workunit.client.1.smithi171.stdout:8/829: symlink df/d1a/dae/de0/l11c 0 2022-01-31T19:44:33.543 INFO:tasks.workunit.client.1.smithi171.stdout:8/830: write df/d1a/f58 [5761386,78512] 0 2022-01-31T19:44:33.544 INFO:tasks.workunit.client.1.smithi171.stdout:6/851: rename dc/d19/d29/d27/l9a to dc/d19/l11d 0 2022-01-31T19:44:33.546 INFO:tasks.workunit.client.1.smithi171.stdout:9/935: mknod dd/df/d15/c13d 0 2022-01-31T19:44:33.547 INFO:tasks.workunit.client.1.smithi171.stdout:9/936: truncate dd/df/d3f/d73/fea 1822407 0 2022-01-31T19:44:33.547 INFO:tasks.workunit.client.1.smithi171.stdout:9/937: chown dd/df/d15/c13d 1 1 2022-01-31T19:44:33.547 INFO:tasks.workunit.client.1.smithi171.stdout:6/852: read dc/d19/d30/f3d [3898091,124747] 0 2022-01-31T19:44:33.549 INFO:tasks.workunit.client.1.smithi171.stdout:9/938: write dd/d109/de6/d7c/d7d/fa3 [2981178,119765] 0 2022-01-31T19:44:33.550 INFO:tasks.workunit.client.1.smithi171.stdout:9/939: write dd/df/d15/d38/d4d/ffd [57094,33737] 0 2022-01-31T19:44:33.553 INFO:tasks.workunit.client.1.smithi171.stdout:4/762: rename d2/d3/d5/dc/d4b/lc6 to d2/d3/d5/d41/dcf/l103 0 2022-01-31T19:44:33.556 INFO:tasks.workunit.client.1.smithi171.stdout:4/763: mkdir d2/d3/d5/d41/dcf/d12/d15/d104 0 2022-01-31T19:44:33.556 INFO:tasks.workunit.client.1.smithi171.stdout:4/764: dread - d2/d3/d5/d41/dcf/d12/d15/d57/fef zero size 2022-01-31T19:44:33.562 INFO:tasks.workunit.client.1.smithi171.stdout:6/853: write dc/d18/f1b [2362439,78990] 0 2022-01-31T19:44:33.562 INFO:tasks.workunit.client.1.smithi171.stdout:9/940: write dd/fe [16897,104704] 0 2022-01-31T19:44:33.563 INFO:tasks.workunit.client.1.smithi171.stdout:4/765: dread d2/d3/d5/d91/f76 [0,4194304] 0 2022-01-31T19:44:33.564 INFO:tasks.workunit.client.1.smithi171.stdout:6/854: creat dc/f11e x:0 0 0 2022-01-31T19:44:33.565 INFO:tasks.workunit.client.1.smithi171.stdout:4/766: unlink d2/d3/cf9 0 2022-01-31T19:44:33.565 INFO:tasks.workunit.client.1.smithi171.stdout:4/767: chown d2/d3/d5/d41/dcf/d12/d15/l4e 141 1 2022-01-31T19:44:33.566 INFO:tasks.workunit.client.1.smithi171.stdout:8/831: dread df/d18/d19/f5f [0,4194304] 0 2022-01-31T19:44:33.567 INFO:tasks.workunit.client.1.smithi171.stdout:8/832: fdatasync df/d2a/fec 0 2022-01-31T19:44:33.568 INFO:tasks.workunit.client.1.smithi171.stdout:6/855: dread dc/f7f [0,4194304] 0 2022-01-31T19:44:33.580 INFO:tasks.workunit.client.1.smithi171.stdout:4/768: unlink d2/d3/d5/d3b/f55 0 2022-01-31T19:44:33.593 INFO:tasks.workunit.client.1.smithi171.stdout:4/769: dread d2/d3/d5/d41/d6b/f99 [0,4194304] 0 2022-01-31T19:44:33.593 INFO:tasks.workunit.client.1.smithi171.stdout:4/770: read - d2/d3/d5/d70/dee/ffb zero size 2022-01-31T19:44:33.596 INFO:tasks.workunit.client.1.smithi171.stdout:6/856: dwrite dc/d18/f25 [0,4194304] 0 2022-01-31T19:44:33.599 INFO:tasks.workunit.client.1.smithi171.stdout:6/857: write dc/d19/d29/daf/ddb/fcd [1326500,61653] 0 2022-01-31T19:44:33.607 INFO:tasks.workunit.client.1.smithi171.stdout:4/771: chown d2/d3/d5/d41/dcf/l1f 261944921 1 2022-01-31T19:44:33.607 INFO:tasks.workunit.client.1.smithi171.stdout:4/772: write d2/d3/d5/d41/fbf [681959,131069] 0 2022-01-31T19:44:33.607 INFO:tasks.workunit.client.1.smithi171.stdout:4/773: read - d2/d3/d5/d91/d77/f101 zero size 2022-01-31T19:44:33.607 INFO:tasks.workunit.client.1.smithi171.stdout:4/774: creat d2/d3/d5/dc/f105 x:0 0 0 2022-01-31T19:44:33.608 INFO:tasks.workunit.client.1.smithi171.stdout:6/858: getdents dc/d19/d29/d6a 0 2022-01-31T19:44:33.610 INFO:tasks.workunit.client.1.smithi171.stdout:6/859: rename dc/d19/d29/d6a/c6b to dc/d21/de9/c11f 0 2022-01-31T19:44:33.613 INFO:tasks.workunit.client.1.smithi171.stdout:4/775: symlink d2/d3/d5/d41/dcf/l106 0 2022-01-31T19:44:33.614 INFO:tasks.workunit.client.1.smithi171.stdout:4/776: fdatasync d2/d3/d5/d41/f47 0 2022-01-31T19:44:33.616 INFO:tasks.workunit.client.1.smithi171.stdout:6/860: mknod dc/d18/d86/d119/d3b/d67/dd1/c120 0 2022-01-31T19:44:33.616 INFO:tasks.workunit.client.1.smithi171.stdout:6/861: write dc/d19/d30/fa3 [681545,15439] 0 2022-01-31T19:44:33.642 INFO:tasks.workunit.client.1.smithi171.stdout:8/833: dwrite df/d18/d19/d46/f5e [0,4194304] 0 2022-01-31T19:44:33.643 INFO:tasks.workunit.client.1.smithi171.stdout:8/834: creat df/f11d x:0 0 0 2022-01-31T19:44:33.643 INFO:tasks.workunit.client.1.smithi171.stdout:4/777: dwrite d2/d3/f44 [0,4194304] 0 2022-01-31T19:44:33.645 INFO:tasks.workunit.client.1.smithi171.stdout:8/835: mknod df/d5d/d6c/d87/c11e 0 2022-01-31T19:44:33.645 INFO:tasks.workunit.client.1.smithi171.stdout:8/836: fdatasync df/d2a/faf 0 2022-01-31T19:44:33.646 INFO:tasks.workunit.client.1.smithi171.stdout:8/837: chown df/d18/d79/da2/cc3 429295398 1 2022-01-31T19:44:33.649 INFO:tasks.workunit.client.1.smithi171.stdout:6/862: dwrite dc/d1e/f45 [0,4194304] 0 2022-01-31T19:44:33.649 INFO:tasks.workunit.client.1.smithi171.stdout:4/778: mknod d2/d3/d5/d41/dcf/d12/db6/c107 0 2022-01-31T19:44:33.652 INFO:tasks.workunit.client.1.smithi171.stdout:6/863: getdents dc/d19/dae 0 2022-01-31T19:44:33.653 INFO:tasks.workunit.client.1.smithi171.stdout:4/779: creat d2/d3/d5/d70/f108 x:0 0 0 2022-01-31T19:44:33.654 INFO:tasks.workunit.client.1.smithi171.stdout:6/864: creat dc/d19/d29/d27/f121 x:0 0 0 2022-01-31T19:44:33.656 INFO:tasks.workunit.client.1.smithi171.stdout:4/780: chown d2/d3/c2f 155 1 2022-01-31T19:44:33.656 INFO:tasks.workunit.client.1.smithi171.stdout:4/781: chown d2/d3/d5/d41/dcf/l103 0 1 2022-01-31T19:44:33.656 INFO:tasks.workunit.client.1.smithi171.stdout:4/782: chown d2/d3/d5/d41/c63 50 1 2022-01-31T19:44:33.656 INFO:tasks.workunit.client.1.smithi171.stdout:4/783: chown d2/d3/d5/d41/dcf/d12/c54 102989 1 2022-01-31T19:44:33.657 INFO:tasks.workunit.client.1.smithi171.stdout:4/784: dread - d2/d3/f86 zero size 2022-01-31T19:44:33.658 INFO:tasks.workunit.client.1.smithi171.stdout:6/865: symlink dc/d21/de9/l122 0 2022-01-31T19:44:33.658 INFO:tasks.workunit.client.1.smithi171.stdout:8/838: dread df/d18/d79/f8c [0,4194304] 0 2022-01-31T19:44:33.659 INFO:tasks.workunit.client.1.smithi171.stdout:8/839: truncate df/d18/fce 902099 0 2022-01-31T19:44:33.664 INFO:tasks.workunit.client.1.smithi171.stdout:6/866: rename dc/d8b to dc/d19/d29/d123 0 2022-01-31T19:44:33.665 INFO:tasks.workunit.client.1.smithi171.stdout:6/867: write dc/d21/fab [1182195,15143] 0 2022-01-31T19:44:33.665 INFO:tasks.workunit.client.1.smithi171.stdout:6/868: fdatasync dc/d1e/d3f/fd2 0 2022-01-31T19:44:33.669 INFO:tasks.workunit.client.1.smithi171.stdout:6/869: rename dc/d18/d86/d119/d68/f6d to dc/d18/d86/d119/f124 0 2022-01-31T19:44:33.673 INFO:tasks.workunit.client.1.smithi171.stdout:8/840: rename df/d18/d19/l24 to df/d18/d79/da2/ddd/l11f 0 2022-01-31T19:44:33.674 INFO:tasks.workunit.client.1.smithi171.stdout:8/841: readlink df/d18/d19/d46/dd6/lf8 0 2022-01-31T19:44:33.674 INFO:tasks.workunit.client.1.smithi171.stdout:8/842: creat df/d18/d19/db2/f120 x:0 0 0 2022-01-31T19:44:33.674 INFO:tasks.workunit.client.1.smithi171.stdout:8/843: fdatasync df/d18/d79/f8c 0 2022-01-31T19:44:33.678 INFO:tasks.workunit.client.1.smithi171.stdout:8/844: creat df/d18/d19/d21/d67/df3/f121 x:0 0 0 2022-01-31T19:44:33.678 INFO:tasks.workunit.client.1.smithi171.stdout:6/870: dread dc/fe0 [0,4194304] 0 2022-01-31T19:44:33.683 INFO:tasks.workunit.client.1.smithi171.stdout:4/785: dwrite d2/d3/d5/d41/dcf/dc7/fdb [0,4194304] 0 2022-01-31T19:44:33.683 INFO:tasks.workunit.client.1.smithi171.stdout:4/786: write d2/d3/d5/d41/dcf/d12/d15/dc4/fe2 [934383,23116] 0 2022-01-31T19:44:33.688 INFO:tasks.workunit.client.1.smithi171.stdout:6/871: mkdir dc/d1e/d3f/d125 0 2022-01-31T19:44:33.694 INFO:tasks.workunit.client.1.smithi171.stdout:6/872: unlink dc/d19/d29/d6a/fb4 0 2022-01-31T19:44:33.705 INFO:tasks.workunit.client.1.smithi171.stdout:8/845: dwrite df/d18/f41 [0,4194304] 0 2022-01-31T19:44:33.708 INFO:tasks.workunit.client.1.smithi171.stdout:8/846: read df/d1a/dae/f10f [1176960,109446] 0 2022-01-31T19:44:33.709 INFO:tasks.workunit.client.1.smithi171.stdout:8/847: truncate df/d18/f1e 5007592 0 2022-01-31T19:44:33.709 INFO:tasks.workunit.client.1.smithi171.stdout:8/848: chown df/d18/d19/d21/def 56 1 2022-01-31T19:44:33.710 INFO:tasks.workunit.client.1.smithi171.stdout:8/849: creat df/dcc/f122 x:0 0 0 2022-01-31T19:44:33.713 INFO:tasks.workunit.client.1.smithi171.stdout:8/850: creat df/d1a/dae/f123 x:0 0 0 2022-01-31T19:44:33.714 INFO:tasks.workunit.client.1.smithi171.stdout:6/873: dread dc/d19/d29/d6a/fbb [0,4194304] 0 2022-01-31T19:44:33.714 INFO:tasks.workunit.client.1.smithi171.stdout:8/851: read df/d18/d19/d82/f97 [808688,116330] 0 2022-01-31T19:44:33.715 INFO:tasks.workunit.client.1.smithi171.stdout:8/852: readlink df/d18/d79/da2/ddd/l11f 0 2022-01-31T19:44:33.715 INFO:tasks.workunit.client.1.smithi171.stdout:6/874: mknod dc/d19/d30/c126 0 2022-01-31T19:44:33.715 INFO:tasks.workunit.client.1.smithi171.stdout:6/875: stat dc/d1e/l56 0 2022-01-31T19:44:33.716 INFO:tasks.workunit.client.1.smithi171.stdout:8/853: symlink df/d18/d79/d7a/l124 0 2022-01-31T19:44:33.720 INFO:tasks.workunit.client.1.smithi171.stdout:6/876: stat dc/d19/d29/c57 0 2022-01-31T19:44:33.731 INFO:tasks.workunit.client.1.smithi171.stdout:4/787: dwrite d2/d3/d5/d41/dcf/d12/d15/dc4/fe2 [0,4194304] 0 2022-01-31T19:44:33.737 INFO:tasks.workunit.client.1.smithi171.stdout:4/788: creat d2/d3/d5/d41/dcf/f109 x:0 0 0 2022-01-31T19:44:33.737 INFO:tasks.workunit.client.1.smithi171.stdout:1/886: sync 2022-01-31T19:44:33.741 INFO:tasks.workunit.client.1.smithi171.stdout:1/887: mkdir dd/d23/d2e/d5e/d108 0 2022-01-31T19:44:33.741 INFO:tasks.workunit.client.1.smithi171.stdout:1/888: chown dd/d23/d2e/d5e/fe0 1 1 2022-01-31T19:44:33.741 INFO:tasks.workunit.client.1.smithi171.stdout:1/889: truncate dd/df/d84/dea/fc9 443919 0 2022-01-31T19:44:33.742 INFO:tasks.workunit.client.1.smithi171.stdout:4/789: dread d2/d3/d5/d70/fa7 [0,4194304] 0 2022-01-31T19:44:33.743 INFO:tasks.workunit.client.1.smithi171.stdout:4/790: fdatasync d2/d3/f86 0 2022-01-31T19:44:33.743 INFO:tasks.workunit.client.1.smithi171.stdout:1/890: rename dd/df/d84/dea/d35/f70 to dd/df/d84/dea/f109 0 2022-01-31T19:44:33.744 INFO:tasks.workunit.client.1.smithi171.stdout:1/891: write dd/d23/d2e/f86 [871844,33362] 0 2022-01-31T19:44:33.745 INFO:tasks.workunit.client.1.smithi171.stdout:6/877: dwrite dc/f4a [0,4194304] 0 2022-01-31T19:44:33.746 INFO:tasks.workunit.client.1.smithi171.stdout:4/791: mkdir d2/d3/d5/dc/d4b/dbc/d10a 0 2022-01-31T19:44:33.748 INFO:tasks.workunit.client.1.smithi171.stdout:1/892: creat dd/d23/d2e/f10a x:0 0 0 2022-01-31T19:44:33.751 INFO:tasks.workunit.client.1.smithi171.stdout:6/878: mknod dc/d19/d30/d3c/c127 0 2022-01-31T19:44:33.755 INFO:tasks.workunit.client.1.smithi171.stdout:6/879: unlink dc/d18/f1b 0 2022-01-31T19:44:33.756 INFO:tasks.workunit.client.1.smithi171.stdout:6/880: dread - dc/d18/d86/d119/d68/f10a zero size 2022-01-31T19:44:33.757 INFO:tasks.workunit.client.1.smithi171.stdout:6/881: creat dc/d19/d29/d123/dee/f128 x:0 0 0 2022-01-31T19:44:33.771 INFO:tasks.workunit.client.1.smithi171.stdout:1/893: dwrite dd/df/d84/fa1 [0,4194304] 0 2022-01-31T19:44:33.771 INFO:tasks.workunit.client.1.smithi171.stdout:1/894: chown f3 3155 1 2022-01-31T19:44:33.772 INFO:tasks.workunit.client.1.smithi171.stdout:6/882: truncate dc/f4a 984212 0 2022-01-31T19:44:33.777 INFO:tasks.workunit.client.1.smithi171.stdout:1/895: truncate dd/d23/d2e/fa5 6444839 0 2022-01-31T19:44:33.777 INFO:tasks.workunit.client.1.smithi171.stdout:6/883: creat dc/d19/d30/d3c/d105/f129 x:0 0 0 2022-01-31T19:44:33.778 INFO:tasks.workunit.client.1.smithi171.stdout:6/884: fdatasync dc/d18/d86/d119/d3b/d67/dd1/fd3 0 2022-01-31T19:44:33.778 INFO:tasks.workunit.client.1.smithi171.stdout:6/885: dread - dc/d18/d86/d119/d68/f10a zero size 2022-01-31T19:44:33.778 INFO:tasks.workunit.client.1.smithi171.stdout:6/886: readlink dc/d19/d29/daf/ddb/lfc 0 2022-01-31T19:44:33.779 INFO:tasks.workunit.client.1.smithi171.stdout:6/887: unlink dc/d18/d86/d119/c4d 0 2022-01-31T19:44:33.780 INFO:tasks.workunit.client.1.smithi171.stdout:6/888: mkdir dc/d19/d29/d123/dee/d116/d12a 0 2022-01-31T19:44:33.782 INFO:tasks.workunit.client.1.smithi171.stdout:6/889: creat dc/d18/d86/d119/d68/f12b x:0 0 0 2022-01-31T19:44:33.784 INFO:tasks.workunit.client.1.smithi171.stdout:6/890: write f2 [5175559,44370] 0 2022-01-31T19:44:33.787 INFO:tasks.workunit.client.1.smithi171.stdout:6/891: creat dc/d19/d29/d6a/f12c x:0 0 0 2022-01-31T19:44:33.787 INFO:tasks.workunit.client.1.smithi171.stdout:6/892: creat dc/d19/d30/d3c/d105/f12d x:0 0 0 2022-01-31T19:44:33.789 INFO:tasks.workunit.client.1.smithi171.stdout:6/893: rmdir dc/d21/de9 39 2022-01-31T19:44:33.790 INFO:tasks.workunit.client.1.smithi171.stdout:6/894: write dc/f10c [462695,77748] 0 2022-01-31T19:44:33.790 INFO:tasks.workunit.client.1.smithi171.stdout:6/895: truncate f2 5771129 0 2022-01-31T19:44:33.791 INFO:tasks.workunit.client.1.smithi171.stdout:6/896: chown dc/d19/d29/f24 1003 1 2022-01-31T19:44:33.796 INFO:tasks.workunit.client.1.smithi171.stdout:6/897: write dc/d1e/f45 [3754620,19845] 0 2022-01-31T19:44:33.799 INFO:tasks.workunit.client.1.smithi171.stdout:6/898: rename dc/d19/d30/fda to dc/d21/f12e 0 2022-01-31T19:44:33.799 INFO:tasks.workunit.client.1.smithi171.stdout:6/899: dread - dc/d19/d29/ffd zero size 2022-01-31T19:44:33.799 INFO:tasks.workunit.client.1.smithi171.stdout:6/900: fdatasync dc/d19/d29/daf/ddb/f118 0 2022-01-31T19:44:33.800 INFO:tasks.workunit.client.1.smithi171.stdout:1/896: dwrite dd/d23/f9e [0,4194304] 0 2022-01-31T19:44:33.801 INFO:tasks.workunit.client.1.smithi171.stdout:6/901: rename dc/d18/d86/d119/d3b/d67/dd1/cf6 to dc/d18/d86/d119/c12f 0 2022-01-31T19:44:33.803 INFO:tasks.workunit.client.1.smithi171.stdout:1/897: rmdir dd/d23/d2e/d5e/d81/d98 39 2022-01-31T19:44:33.806 INFO:tasks.workunit.client.1.smithi171.stdout:1/898: mkdir dd/df/d19/d10b 0 2022-01-31T19:44:33.807 INFO:tasks.workunit.client.1.smithi171.stdout:6/902: getdents dc/d19/d29/d27 0 2022-01-31T19:44:33.810 INFO:tasks.workunit.client.1.smithi171.stdout:1/899: truncate dd/d23/dd4/fde 2625225 0 2022-01-31T19:44:33.813 INFO:tasks.workunit.client.1.smithi171.stdout:6/903: mkdir dc/d18/d86/d119/d111/df0/d130 0 2022-01-31T19:44:33.816 INFO:tasks.workunit.client.1.smithi171.stdout:1/900: rename dd/df/d84/dea/d64/l67 to dd/d23/d2e/d5e/l10c 0 2022-01-31T19:44:33.818 INFO:tasks.workunit.client.1.smithi171.stdout:6/904: mkdir dc/d19/d29/d123/d131 0 2022-01-31T19:44:33.818 INFO:tasks.workunit.client.1.smithi171.stdout:6/905: dread - dc/d19/d30/d3c/d105/f129 zero size 2022-01-31T19:44:33.822 INFO:tasks.workunit.client.1.smithi171.stdout:1/901: truncate dd/d23/d34/d4b/f52 2628249 0 2022-01-31T19:44:33.822 INFO:tasks.workunit.client.1.smithi171.stdout:1/902: truncate dd/f56 4227269 0 2022-01-31T19:44:33.823 INFO:tasks.workunit.client.1.smithi171.stdout:1/903: getdents dd/d23/d2e/d5e/d81/d98/dad 0 2022-01-31T19:44:33.843 INFO:tasks.workunit.client.1.smithi171.stdout:1/904: dwrite dd/df/d19/f75 [0,4194304] 0 2022-01-31T19:44:33.849 INFO:tasks.workunit.client.1.smithi171.stdout:5/870: sync 2022-01-31T19:44:33.849 INFO:tasks.workunit.client.1.smithi171.stdout:5/871: getdents d5/d90/db7/d15/d5d 0 2022-01-31T19:44:33.850 INFO:tasks.workunit.client.1.smithi171.stdout:5/872: truncate d5/d90/dd1/dfc/f113 713403 0 2022-01-31T19:44:33.850 INFO:tasks.workunit.client.1.smithi171.stdout:0/911: sync 2022-01-31T19:44:33.850 INFO:tasks.workunit.client.1.smithi171.stdout:0/912: truncate d9/db/dc/d3f/db9/d7a/f12f 102091 0 2022-01-31T19:44:33.851 INFO:tasks.workunit.client.1.smithi171.stdout:0/913: dread d9/db/d59/ffe [0,4194304] 0 2022-01-31T19:44:33.852 INFO:tasks.workunit.client.1.smithi171.stdout:5/873: creat d5/d90/db7/d15/f129 x:0 0 0 2022-01-31T19:44:33.853 INFO:tasks.workunit.client.1.smithi171.stdout:5/874: symlink d5/d90/db7/d15/d80/d109/l12a 0 2022-01-31T19:44:33.864 INFO:tasks.workunit.client.1.smithi171.stdout:1/905: dwrite dd/d23/d2e/f10a [0,4194304] 0 2022-01-31T19:44:33.866 INFO:tasks.workunit.client.1.smithi171.stdout:1/906: mknod dd/da2/c10d 0 2022-01-31T19:44:33.869 INFO:tasks.workunit.client.1.smithi171.stdout:1/907: dread dd/df/ffa [0,4194304] 0 2022-01-31T19:44:33.870 INFO:tasks.workunit.client.1.smithi171.stdout:1/908: rmdir dd/da2 39 2022-01-31T19:44:33.878 INFO:tasks.workunit.client.1.smithi171.stdout:1/909: rmdir dd/df/d19/dce 39 2022-01-31T19:44:33.878 INFO:tasks.workunit.client.1.smithi171.stdout:1/910: getdents dd/d23/dd4/de9 0 2022-01-31T19:44:33.878 INFO:tasks.workunit.client.1.smithi171.stdout:1/911: dread - dd/d23/d2e/fe6 zero size 2022-01-31T19:44:33.878 INFO:tasks.workunit.client.1.smithi171.stdout:1/912: link dd/df/d19/d36/c89 dd/df/d84/dea/d64/da7/dca/ded/c10e 0 2022-01-31T19:44:33.879 INFO:tasks.workunit.client.1.smithi171.stdout:1/913: chown dd/df/fe8 2709750 1 2022-01-31T19:44:33.879 INFO:tasks.workunit.client.1.smithi171.stdout:0/914: dwrite d9/d24/f4d [0,4194304] 0 2022-01-31T19:44:33.879 INFO:tasks.workunit.client.1.smithi171.stdout:0/915: chown d9/db/dc/d3f/db9/f8e 242297579 1 2022-01-31T19:44:33.879 INFO:tasks.workunit.client.1.smithi171.stdout:1/914: creat dd/d23/dd4/de9/f10f x:0 0 0 2022-01-31T19:44:33.880 INFO:tasks.workunit.client.1.smithi171.stdout:1/915: chown dd/df/d84/dea/f6a 2 1 2022-01-31T19:44:33.884 INFO:tasks.workunit.client.1.smithi171.stdout:1/916: dread dd/d23/d2e/d5e/d81/d98/d80/dee/ff4 [0,4194304] 0 2022-01-31T19:44:33.885 INFO:tasks.workunit.client.1.smithi171.stdout:1/917: read dd/d23/d2e/d5e/d81/d98/fc5 [3598877,104377] 0 2022-01-31T19:44:33.885 INFO:tasks.workunit.client.1.smithi171.stdout:1/918: fdatasync dd/d23/d74/dcb/ddb/fe4 0 2022-01-31T19:44:33.885 INFO:tasks.workunit.client.1.smithi171.stdout:1/919: write dd/df/d84/dea/d64/da7/fc0 [106344,120913] 0 2022-01-31T19:44:33.886 INFO:tasks.workunit.client.1.smithi171.stdout:1/920: write dd/df/d84/dea/d64/fc1 [4489556,3247] 0 2022-01-31T19:44:33.886 INFO:tasks.workunit.client.1.smithi171.stdout:1/921: write dd/df/d84/dea/d64/f53 [124658,37981] 0 2022-01-31T19:44:33.894 INFO:tasks.workunit.client.1.smithi171.stdout:0/916: dwrite d9/db/d59/d71/dbe/fef [0,4194304] 0 2022-01-31T19:44:33.905 INFO:tasks.workunit.client.1.smithi171.stdout:1/922: dwrite dd/df/d19/d1b/f4e [4194304,4194304] 0 2022-01-31T19:44:33.907 INFO:tasks.workunit.client.1.smithi171.stdout:1/923: mknod dd/da2/c110 0 2022-01-31T19:44:33.908 INFO:tasks.workunit.client.1.smithi171.stdout:1/924: unlink dd/df/d84/dea/d35/dd6/df2/f100 0 2022-01-31T19:44:33.911 INFO:tasks.workunit.client.1.smithi171.stdout:1/925: write dd/df/d19/d1b/f72 [4730521,111231] 0 2022-01-31T19:44:33.911 INFO:tasks.workunit.client.1.smithi171.stdout:1/926: write dd/da2/fb2 [1126836,119117] 0 2022-01-31T19:44:33.911 INFO:tasks.workunit.client.1.smithi171.stdout:0/917: dwrite d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:33.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:33 smithi167 conmon[32206]: cluster 2022-01-31T19:44:32.437040+0000 mgr.smithi171.asyxnc (mgr.24387) 37 : cluster [DBG] pgmap v25: 65 pgs: 65 active+clean; 3.4 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 98 MiB/s rd, 219 MiB/s wr, 341 op/s 2022-01-31T19:44:33.921 INFO:tasks.workunit.client.1.smithi171.stdout:0/918: dread d9/f1d [0,4194304] 0 2022-01-31T19:44:33.926 INFO:tasks.workunit.client.1.smithi171.stdout:1/927: dwrite dd/d23/d34/f65 [0,4194304] 0 2022-01-31T19:44:33.927 INFO:tasks.workunit.client.1.smithi171.stdout:1/928: rmdir dd/d23 39 2022-01-31T19:44:33.927 INFO:tasks.workunit.client.1.smithi171.stdout:1/929: symlink dd/d23/d74/l111 0 2022-01-31T19:44:33.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/930: dread dd/d23/d2e/d5e/f20 [4194304,4194304] 0 2022-01-31T19:44:33.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/931: write dd/df/d84/dea/f9f [5170004,120735] 0 2022-01-31T19:44:33.935 INFO:tasks.workunit.client.1.smithi171.stdout:1/932: chown dd/d23/d2e/d5e/f20 146797157 1 2022-01-31T19:44:33.936 INFO:tasks.workunit.client.1.smithi171.stdout:1/933: creat dd/df/d84/dea/d35/f112 x:0 0 0 2022-01-31T19:44:33.937 INFO:tasks.workunit.client.1.smithi171.stdout:1/934: mkdir dd/df/d84/dea/d35/dd6/d113 0 2022-01-31T19:44:33.937 INFO:tasks.workunit.client.1.smithi171.stdout:1/935: stat dd/df/d19/cbb 0 2022-01-31T19:44:33.938 INFO:tasks.workunit.client.1.smithi171.stdout:1/936: truncate dd/d23/f3f 565974 0 2022-01-31T19:44:33.938 INFO:tasks.workunit.client.1.smithi171.stdout:1/937: truncate dd/df/d84/dea/d64/fc7 487703 0 2022-01-31T19:44:33.942 INFO:tasks.workunit.client.1.smithi171.stdout:1/938: write dd/d23/d2e/d5e/f20 [2636825,95203] 0 2022-01-31T19:44:33.945 INFO:tasks.workunit.client.1.smithi171.stdout:1/939: rename dd/df/d84/dea/l107 to dd/df/d84/dea/d35/dd6/df2/l114 0 2022-01-31T19:44:33.946 INFO:tasks.workunit.client.1.smithi171.stdout:0/919: dwrite d9/db/f131 [0,4194304] 0 2022-01-31T19:44:33.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/920: dwrite d9/db/dc/d3f/db9/d7a/fa8 [0,4194304] 0 2022-01-31T19:44:33.968 INFO:tasks.workunit.client.1.smithi171.stdout:0/921: read - d9/db/dc/d3f/d6f/ff9 zero size 2022-01-31T19:44:33.970 INFO:tasks.workunit.client.1.smithi171.stdout:1/940: dwrite dd/df/d84/dea/f109 [0,4194304] 0 2022-01-31T19:44:33.971 INFO:tasks.workunit.client.1.smithi171.stdout:1/941: chown dd/df/d84/dea/d35/c7e 117547 1 2022-01-31T19:44:33.971 INFO:tasks.workunit.client.1.smithi171.stdout:1/942: stat dd/df/d19/cbb 0 2022-01-31T19:44:33.974 INFO:tasks.workunit.client.1.smithi171.stdout:0/922: link d9/d24/d4c/d35/d67/l114 d9/d24/d92/d9d/dc6/d115/l14c 0 2022-01-31T19:44:33.986 INFO:tasks.workunit.client.1.smithi171.stdout:4/792: sync 2022-01-31T19:44:33.986 INFO:tasks.workunit.client.1.smithi171.stdout:6/906: sync 2022-01-31T19:44:33.986 INFO:tasks.workunit.client.1.smithi171.stdout:5/875: sync 2022-01-31T19:44:33.986 INFO:tasks.workunit.client.1.smithi171.stdout:4/793: chown d2/d7f/fa8 55131005 1 2022-01-31T19:44:33.987 INFO:tasks.workunit.client.1.smithi171.stdout:5/876: write d5/d90/db7/dd/d66/fe1 [773179,67869] 0 2022-01-31T19:44:33.987 INFO:tasks.workunit.client.1.smithi171.stdout:5/877: dread - d5/d90/db7/d15/d80/d8d/de2/f116 zero size 2022-01-31T19:44:33.987 INFO:tasks.workunit.client.1.smithi171.stdout:7/690: sync 2022-01-31T19:44:33.988 INFO:tasks.workunit.client.1.smithi171.stdout:8/854: sync 2022-01-31T19:44:33.988 INFO:tasks.workunit.client.1.smithi171.stdout:4/794: write d2/d3/d5/f1a [1113010,100522] 0 2022-01-31T19:44:33.989 INFO:tasks.workunit.client.1.smithi171.stdout:9/941: sync 2022-01-31T19:44:33.989 INFO:tasks.workunit.client.1.smithi171.stdout:6/907: rename dc/d21/faa to dc/d21/de9/f132 0 2022-01-31T19:44:33.989 INFO:tasks.workunit.client.1.smithi171.stdout:6/908: truncate dc/d19/d29/daf/ddb/fcd 5295331 0 2022-01-31T19:44:33.990 INFO:tasks.workunit.client.1.smithi171.stdout:5/878: getdents d5/d90/db7/d15/d23/d6b 0 2022-01-31T19:44:33.991 INFO:tasks.workunit.client.1.smithi171.stdout:5/879: write d5/d90/db7/d15/f129 [261744,9450] 0 2022-01-31T19:44:33.992 INFO:tasks.workunit.client.1.smithi171.stdout:6/909: symlink dc/d21/l133 0 2022-01-31T19:44:33.993 INFO:tasks.workunit.client.1.smithi171.stdout:5/880: creat d5/d90/db7/d15/d80/d8d/de2/f12b x:0 0 0 2022-01-31T19:44:33.999 INFO:tasks.workunit.client.1.smithi171.stdout:6/910: getdents dc/d18 0 2022-01-31T19:44:33.999 INFO:tasks.workunit.client.1.smithi171.stdout:6/911: creat dc/d18/d86/d119/d68/f134 x:0 0 0 2022-01-31T19:44:34.000 INFO:tasks.workunit.client.1.smithi171.stdout:6/912: mkdir dc/d135 0 2022-01-31T19:44:34.000 INFO:tasks.workunit.client.1.smithi171.stdout:5/881: read d5/d90/db7/d15/d23/d67/f5c [2344122,98758] 0 2022-01-31T19:44:34.001 INFO:tasks.workunit.client.1.smithi171.stdout:6/913: creat dc/d135/f136 x:0 0 0 2022-01-31T19:44:34.002 INFO:tasks.workunit.client.1.smithi171.stdout:5/882: mknod d5/d90/dd1/dfc/c12c 0 2022-01-31T19:44:34.003 INFO:tasks.workunit.client.1.smithi171.stdout:6/914: creat dc/d19/d29/d123/d131/f137 x:0 0 0 2022-01-31T19:44:34.005 INFO:tasks.workunit.client.1.smithi171.stdout:5/883: mkdir d5/d90/d12d 0 2022-01-31T19:44:34.006 INFO:tasks.workunit.client.1.smithi171.stdout:6/915: mknod dc/d19/dae/c138 0 2022-01-31T19:44:34.008 INFO:tasks.workunit.client.1.smithi171.stdout:5/884: rename d5/d90/db7/d15/d5d/d6f/fa6 to d5/d90/db7/d15/d80/d109/f12e 0 2022-01-31T19:44:34.009 INFO:tasks.workunit.client.1.smithi171.stdout:5/885: creat d5/d59/da0/f12f x:0 0 0 2022-01-31T19:44:34.010 INFO:tasks.workunit.client.1.smithi171.stdout:6/916: dread dc/d19/d30/d3c/f4b [0,4194304] 0 2022-01-31T19:44:34.010 INFO:tasks.workunit.client.1.smithi171.stdout:0/923: dwrite d9/db/d59/d71/dbe/fef [0,4194304] 0 2022-01-31T19:44:34.014 INFO:tasks.workunit.client.1.smithi171.stdout:6/917: truncate dc/d19/d29/d27/f33 2093714 0 2022-01-31T19:44:34.014 INFO:tasks.workunit.client.1.smithi171.stdout:6/918: truncate dc/d18/f25 4416092 0 2022-01-31T19:44:34.031 INFO:tasks.workunit.client.1.smithi171.stdout:7/691: dwrite d1/d1e/d2a/d29/d31/d30/f73 [4194304,4194304] 0 2022-01-31T19:44:34.031 INFO:tasks.workunit.client.1.smithi171.stdout:7/692: write d1/d1e/d2a/d58/fcc [1042055,129395] 0 2022-01-31T19:44:34.031 INFO:tasks.workunit.client.1.smithi171.stdout:7/693: readlink d1/d26/l61 0 2022-01-31T19:44:34.050 INFO:tasks.workunit.client.1.smithi171.stdout:8/855: dwrite df/d18/d19/d21/f26 [0,4194304] 0 2022-01-31T19:44:34.050 INFO:tasks.workunit.client.1.smithi171.stdout:8/856: dread - df/d18/d19/d21/dab/f102 zero size 2022-01-31T19:44:34.051 INFO:tasks.workunit.client.1.smithi171.stdout:8/857: readlink df/l74 0 2022-01-31T19:44:34.052 INFO:tasks.workunit.client.1.smithi171.stdout:8/858: mkdir df/d18/d79/da2/ddd/d125 0 2022-01-31T19:44:34.052 INFO:tasks.workunit.client.1.smithi171.stdout:8/859: fdatasync df/d18/d19/d21/dab/f56 0 2022-01-31T19:44:34.052 INFO:tasks.workunit.client.1.smithi171.stdout:9/942: dwrite dd/f79 [0,4194304] 0 2022-01-31T19:44:34.053 INFO:tasks.workunit.client.1.smithi171.stdout:9/943: chown dd/df/c60 10 1 2022-01-31T19:44:34.054 INFO:tasks.workunit.client.1.smithi171.stdout:5/886: dwrite d5/d90/db7/d15/d23/f25 [0,4194304] 0 2022-01-31T19:44:34.054 INFO:tasks.workunit.client.1.smithi171.stdout:7/694: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d79/f8d [0,4194304] 0 2022-01-31T19:44:34.054 INFO:tasks.workunit.client.1.smithi171.stdout:6/919: dwrite dc/d19/d29/d123/f96 [0,4194304] 0 2022-01-31T19:44:34.054 INFO:tasks.workunit.client.1.smithi171.stdout:6/920: chown dc/c40 22243546 1 2022-01-31T19:44:34.063 INFO:tasks.workunit.client.1.smithi171.stdout:7/695: mknod d1/d1e/d2a/d29/d31/d30/d6b/ce5 0 2022-01-31T19:44:34.063 INFO:tasks.workunit.client.1.smithi171.stdout:7/696: fsync d1/d1e/d2a/d29/d31/d30/f4d 0 2022-01-31T19:44:34.064 INFO:tasks.workunit.client.1.smithi171.stdout:7/697: unlink d1/d26/l32 0 2022-01-31T19:44:34.065 INFO:tasks.workunit.client.1.smithi171.stdout:7/698: creat d1/d1e/d2a/d29/d31/d30/dc4/fe6 x:0 0 0 2022-01-31T19:44:34.066 INFO:tasks.workunit.client.1.smithi171.stdout:7/699: write d1/d1e/d2a/f9b [1073040,126865] 0 2022-01-31T19:44:34.066 INFO:tasks.workunit.client.1.smithi171.stdout:7/700: readlink d1/d1b/l48 0 2022-01-31T19:44:34.082 INFO:tasks.workunit.client.1.smithi171.stdout:8/860: dwrite df/d18/d79/d7a/dc0/fc4 [0,4194304] 0 2022-01-31T19:44:34.084 INFO:tasks.workunit.client.1.smithi171.stdout:5/887: dwrite d5/d90/db7/d15/d23/d58/f11d [0,4194304] 0 2022-01-31T19:44:34.084 INFO:tasks.workunit.client.1.smithi171.stdout:6/921: dwrite dc/d21/de9/f132 [4194304,4194304] 0 2022-01-31T19:44:34.086 INFO:tasks.workunit.client.1.smithi171.stdout:5/888: truncate d5/d90/db7/d15/d23/d3d/f10f 2299211 0 2022-01-31T19:44:34.087 INFO:tasks.workunit.client.1.smithi171.stdout:6/922: getdents dc/d19/d29 0 2022-01-31T19:44:34.088 INFO:tasks.workunit.client.1.smithi171.stdout:5/889: mknod d5/d90/db7/d15/d23/c130 0 2022-01-31T19:44:34.089 INFO:tasks.workunit.client.1.smithi171.stdout:5/890: write d5/d90/db7/d15/d23/d58/f91 [2835023,68530] 0 2022-01-31T19:44:34.090 INFO:tasks.workunit.client.1.smithi171.stdout:5/891: readlink d5/d59/l8e 0 2022-01-31T19:44:34.090 INFO:tasks.workunit.client.1.smithi171.stdout:6/923: unlink dc/d18/f2a 0 2022-01-31T19:44:34.101 INFO:tasks.workunit.client.1.smithi171.stdout:8/861: dread df/d18/f4e [0,4194304] 0 2022-01-31T19:44:34.102 INFO:tasks.workunit.client.1.smithi171.stdout:8/862: chown df/d18/d79/d7a/l124 3543225 1 2022-01-31T19:44:34.105 INFO:tasks.workunit.client.1.smithi171.stdout:8/863: dread df/d95/f98 [0,4194304] 0 2022-01-31T19:44:34.106 INFO:tasks.workunit.client.1.smithi171.stdout:8/864: creat df/d18/d19/d21/d67/df3/f126 x:0 0 0 2022-01-31T19:44:34.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:33 smithi171 conmon[35325]: cluster 2022-01-31T19:44:32.437040+0000 mgr.smithi171.asyxnc (mgr.24387) 37 : cluster [DBG] 2022-01-31T19:44:34.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:33 smithi171 conmon[35325]: pgmap v25: 65 pgs: 65 active+clean; 3.4 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 98 MiB/s rd, 219 MiB/s wr, 341 op/s 2022-01-31T19:44:34.122 INFO:tasks.workunit.client.1.smithi171.stdout:5/892: dwrite d5/d90/db7/dd/f45 [4194304,4194304] 0 2022-01-31T19:44:34.126 INFO:tasks.workunit.client.1.smithi171.stdout:6/924: dwrite dc/d19/d29/f34 [0,4194304] 0 2022-01-31T19:44:34.126 INFO:tasks.workunit.client.1.smithi171.stdout:6/925: truncate dc/d1e/d3f/fe4 918149 0 2022-01-31T19:44:34.127 INFO:tasks.workunit.client.1.smithi171.stdout:6/926: getdents dc/d19/dae 0 2022-01-31T19:44:34.128 INFO:tasks.workunit.client.1.smithi171.stdout:6/927: symlink dc/d19/dae/l139 0 2022-01-31T19:44:34.128 INFO:tasks.workunit.client.1.smithi171.stdout:6/928: chown dc/d19/d29/d27/f33 623986419 1 2022-01-31T19:44:34.129 INFO:tasks.workunit.client.1.smithi171.stdout:8/865: dwrite df/d27/f57 [0,4194304] 0 2022-01-31T19:44:34.129 INFO:tasks.workunit.client.1.smithi171.stdout:6/929: symlink dc/d1e/l13a 0 2022-01-31T19:44:34.134 INFO:tasks.workunit.client.1.smithi171.stdout:8/866: creat df/d27/d7b/f127 x:0 0 0 2022-01-31T19:44:34.135 INFO:tasks.workunit.client.1.smithi171.stdout:8/867: write df/d18/fce [1745928,129701] 0 2022-01-31T19:44:34.135 INFO:tasks.workunit.client.1.smithi171.stdout:8/868: chown df/d18/d19/d82/dd3/l115 1014 1 2022-01-31T19:44:34.136 INFO:tasks.workunit.client.1.smithi171.stdout:6/930: rename dc/d19/d29/d123/le8 to dc/d1e/d3f/d125/l13b 0 2022-01-31T19:44:34.136 INFO:tasks.workunit.client.1.smithi171.stdout:8/869: rmdir df/d18/d79/da2 39 2022-01-31T19:44:34.138 INFO:tasks.workunit.client.1.smithi171.stdout:6/931: chown dc/d19/d30/d3c/c82 0 1 2022-01-31T19:44:34.139 INFO:tasks.workunit.client.1.smithi171.stdout:6/932: link dc/d19/d29/d123/f96 dc/d1e/d3f/d125/f13c 0 2022-01-31T19:44:34.143 INFO:tasks.workunit.client.1.smithi171.stdout:6/933: mknod dc/d19/d29/d6a/c13d 0 2022-01-31T19:44:34.145 INFO:tasks.workunit.client.1.smithi171.stdout:5/893: dwrite d5/d90/db7/d15/f1d [0,4194304] 0 2022-01-31T19:44:34.145 INFO:tasks.workunit.client.1.smithi171.stdout:5/894: fdatasync d5/d90/db7/dd/d66/f82 0 2022-01-31T19:44:34.145 INFO:tasks.workunit.client.1.smithi171.stdout:5/895: chown d5/d90/db7/d15/d80/d8d/de2/f12b 17249215 1 2022-01-31T19:44:34.145 INFO:tasks.workunit.client.1.smithi171.stdout:6/934: dread dc/f7f [0,4194304] 0 2022-01-31T19:44:34.146 INFO:tasks.workunit.client.1.smithi171.stdout:6/935: creat dc/d18/d86/d119/d3b/d67/f13e x:0 0 0 2022-01-31T19:44:34.147 INFO:tasks.workunit.client.1.smithi171.stdout:6/936: symlink dc/d19/d29/daf/ddb/l13f 0 2022-01-31T19:44:34.155 INFO:tasks.workunit.client.1.smithi171.stdout:5/896: dread d5/d90/db7/d15/d23/d58/d9e/dd7/df0/f108 [0,4194304] 0 2022-01-31T19:44:34.163 INFO:tasks.workunit.client.1.smithi171.stdout:8/870: dwrite df/d27/fb7 [0,4194304] 0 2022-01-31T19:44:34.164 INFO:tasks.workunit.client.1.smithi171.stdout:6/937: dwrite dc/d1e/d3f/fc8 [0,4194304] 0 2022-01-31T19:44:34.166 INFO:tasks.workunit.client.1.smithi171.stdout:5/897: dread d5/ff [0,4194304] 0 2022-01-31T19:44:34.169 INFO:tasks.workunit.client.1.smithi171.stdout:8/871: dread df/d27/f64 [0,4194304] 0 2022-01-31T19:44:34.170 INFO:tasks.workunit.client.1.smithi171.stdout:5/898: symlink d5/d90/db7/d15/d23/d67/dd8/l131 0 2022-01-31T19:44:34.171 INFO:tasks.workunit.client.1.smithi171.stdout:6/938: getdents dc/d19/d30/d3c/d80 0 2022-01-31T19:44:34.171 INFO:tasks.workunit.client.1.smithi171.stdout:6/939: truncate dc/d18/d86/d119/d3b/d67/f13e 673415 0 2022-01-31T19:44:34.172 INFO:tasks.workunit.client.1.smithi171.stdout:8/872: truncate df/d18/da4/fb8 2766015 0 2022-01-31T19:44:34.176 INFO:tasks.workunit.client.1.smithi171.stdout:5/899: creat d5/d90/db7/dd/f132 x:0 0 0 2022-01-31T19:44:34.178 INFO:tasks.workunit.client.1.smithi171.stdout:1/943: sync 2022-01-31T19:44:34.178 INFO:tasks.workunit.client.1.smithi171.stdout:8/873: creat df/d18/d79/da2/ddd/d125/f128 x:0 0 0 2022-01-31T19:44:34.179 INFO:tasks.workunit.client.1.smithi171.stdout:8/874: stat df/d1a/de8 0 2022-01-31T19:44:34.180 INFO:tasks.workunit.client.1.smithi171.stdout:5/900: symlink d5/d90/db7/d15/d23/d3d/l133 0 2022-01-31T19:44:34.180 INFO:tasks.workunit.client.1.smithi171.stdout:5/901: fsync d5/d90/db7/d15/d23/d3d/f49 0 2022-01-31T19:44:34.180 INFO:tasks.workunit.client.1.smithi171.stdout:5/902: chown d5/d90/db7/d15/d23/d67/dd8/l131 38001368 1 2022-01-31T19:44:34.180 INFO:tasks.workunit.client.1.smithi171.stdout:1/944: write dd/d23/d34/d4b/f59 [1733080,112616] 0 2022-01-31T19:44:34.181 INFO:tasks.workunit.client.1.smithi171.stdout:4/795: sync 2022-01-31T19:44:34.184 INFO:tasks.workunit.client.1.smithi171.stdout:8/875: link df/d18/d19/d82/dd3/l115 df/d5d/d6c/d87/df1/l129 0 2022-01-31T19:44:34.185 INFO:tasks.workunit.client.1.smithi171.stdout:8/876: stat df/d18/d79/d7a/le9 0 2022-01-31T19:44:34.185 INFO:tasks.workunit.client.1.smithi171.stdout:4/796: creat d2/d3/d5/ded/f10b x:0 0 0 2022-01-31T19:44:34.185 INFO:tasks.workunit.client.1.smithi171.stdout:4/797: chown d2/d3/d5/d41/dcf/d12/cae 1 1 2022-01-31T19:44:34.185 INFO:tasks.workunit.client.1.smithi171.stdout:4/798: chown d2/d3/d5/dc/fdf 7997931 1 2022-01-31T19:44:34.186 INFO:tasks.workunit.client.1.smithi171.stdout:6/940: dwrite dc/d19/d29/daf/fed [0,4194304] 0 2022-01-31T19:44:34.187 INFO:tasks.workunit.client.1.smithi171.stdout:8/877: mknod df/d2a/d30/dd8/c12a 0 2022-01-31T19:44:34.189 INFO:tasks.workunit.client.1.smithi171.stdout:4/799: creat d2/d3/d5/d70/d7b/f10c x:0 0 0 2022-01-31T19:44:34.193 INFO:tasks.workunit.client.1.smithi171.stdout:6/941: link dc/d19/d29/c57 dc/d18/d86/d119/d111/df0/d130/c140 0 2022-01-31T19:44:34.193 INFO:tasks.workunit.client.1.smithi171.stdout:6/942: stat dc/d18/d86/d119/cc2 0 2022-01-31T19:44:34.193 INFO:tasks.workunit.client.1.smithi171.stdout:6/943: truncate dc/d18/d86/d119/f124 4633497 0 2022-01-31T19:44:34.194 INFO:tasks.workunit.client.1.smithi171.stdout:8/878: unlink df/d18/d79/d7a/dc0/lc6 0 2022-01-31T19:44:34.196 INFO:tasks.workunit.client.1.smithi171.stdout:4/800: rename d2/d3/f1c to d2/d3/d5/d91/de4/df8/f10d 0 2022-01-31T19:44:34.197 INFO:tasks.workunit.client.1.smithi171.stdout:4/801: fsync d2/d3/f73 0 2022-01-31T19:44:34.204 INFO:tasks.workunit.client.1.smithi171.stdout:4/802: getdents d2/d3/d5/ded 0 2022-01-31T19:44:34.214 INFO:tasks.workunit.client.1.smithi171.stdout:9/944: sync 2022-01-31T19:44:34.215 INFO:tasks.workunit.client.1.smithi171.stdout:9/945: chown dd/df/d3f/f9e 131281176 1 2022-01-31T19:44:34.217 INFO:tasks.workunit.client.1.smithi171.stdout:0/924: sync 2022-01-31T19:44:34.217 INFO:tasks.workunit.client.1.smithi171.stdout:7/701: sync 2022-01-31T19:44:34.218 INFO:tasks.workunit.client.1.smithi171.stdout:0/925: mknod d9/c14d 0 2022-01-31T19:44:34.220 INFO:tasks.workunit.client.1.smithi171.stdout:0/926: creat d9/db/dc/d3f/db9/f14e x:0 0 0 2022-01-31T19:44:34.221 INFO:tasks.workunit.client.1.smithi171.stdout:4/803: dread d2/d3/f44 [0,4194304] 0 2022-01-31T19:44:34.222 INFO:tasks.workunit.client.1.smithi171.stdout:7/702: write d1/d1e/d2a/d29/d31/d30/f39 [3601100,57857] 0 2022-01-31T19:44:34.223 INFO:tasks.workunit.client.1.smithi171.stdout:0/927: creat d9/d24/d4c/d7d/f14f x:0 0 0 2022-01-31T19:44:34.225 INFO:tasks.workunit.client.1.smithi171.stdout:1/945: dwrite dd/df/d84/dea/f9f [0,4194304] 0 2022-01-31T19:44:34.225 INFO:tasks.workunit.client.1.smithi171.stdout:8/879: dwrite df/d95/f9c [0,4194304] 0 2022-01-31T19:44:34.226 INFO:tasks.workunit.client.1.smithi171.stdout:6/944: dwrite dc/d18/d86/d119/d68/f12b [0,4194304] 0 2022-01-31T19:44:34.227 INFO:tasks.workunit.client.1.smithi171.stdout:5/903: dwrite d5/d90/db7/d15/d80/fcb [0,4194304] 0 2022-01-31T19:44:34.227 INFO:tasks.workunit.client.1.smithi171.stdout:5/904: chown d5/d90/db7/dd/d66/l11a 79 1 2022-01-31T19:44:34.227 INFO:tasks.workunit.client.1.smithi171.stdout:4/804: mknod d2/d3/d5/d3b/c10e 0 2022-01-31T19:44:34.228 INFO:tasks.workunit.client.1.smithi171.stdout:4/805: readlink d2/d3/d5/d41/dcf/d12/l21 0 2022-01-31T19:44:34.230 INFO:tasks.workunit.client.1.smithi171.stdout:1/946: mknod dd/d23/dd4/de9/c115 0 2022-01-31T19:44:34.231 INFO:tasks.workunit.client.1.smithi171.stdout:8/880: symlink df/d5d/d6c/l12b 0 2022-01-31T19:44:34.232 INFO:tasks.workunit.client.1.smithi171.stdout:0/928: read f3 [3639576,116709] 0 2022-01-31T19:44:34.233 INFO:tasks.workunit.client.1.smithi171.stdout:6/945: creat dc/d19/d30/d3c/d80/f141 x:0 0 0 2022-01-31T19:44:34.235 INFO:tasks.workunit.client.1.smithi171.stdout:5/905: mknod d5/d90/db7/d15/d80/d8d/c134 0 2022-01-31T19:44:34.236 INFO:tasks.workunit.client.1.smithi171.stdout:1/947: mkdir dd/d23/d2e/d5e/d83/d116 0 2022-01-31T19:44:34.236 INFO:tasks.workunit.client.1.smithi171.stdout:8/881: rmdir df/d18/d79/da2 39 2022-01-31T19:44:34.237 INFO:tasks.workunit.client.1.smithi171.stdout:8/882: dread - df/d18/d19/d21/d67/df3/f121 zero size 2022-01-31T19:44:34.237 INFO:tasks.workunit.client.1.smithi171.stdout:8/883: dread - df/d18/d19/d82/dd3/fde zero size 2022-01-31T19:44:34.239 INFO:tasks.workunit.client.1.smithi171.stdout:1/948: symlink dd/d23/d2e/d5e/l117 0 2022-01-31T19:44:34.239 INFO:tasks.workunit.client.1.smithi171.stdout:0/929: truncate f6 976364 0 2022-01-31T19:44:34.240 INFO:tasks.workunit.client.1.smithi171.stdout:8/884: read df/d18/d79/f9d [1888082,6440] 0 2022-01-31T19:44:34.244 INFO:tasks.workunit.client.1.smithi171.stdout:0/930: rename d9/db/cce to d9/d24/d4c/d10e/c150 0 2022-01-31T19:44:34.244 INFO:tasks.workunit.client.1.smithi171.stdout:0/931: creat d9/db/dc2/f151 x:0 0 0 2022-01-31T19:44:34.244 INFO:tasks.workunit.client.1.smithi171.stdout:0/932: fdatasync d9/db/dc2/fad 0 2022-01-31T19:44:34.245 INFO:tasks.workunit.client.1.smithi171.stdout:1/949: write dd/df/d84/dea/d35/f7d [1112880,116564] 0 2022-01-31T19:44:34.245 INFO:tasks.workunit.client.1.smithi171.stdout:9/946: dwrite dd/df/d15/f4e [0,4194304] 0 2022-01-31T19:44:34.246 INFO:tasks.workunit.client.1.smithi171.stdout:9/947: truncate dd/df/d15/d1a/d84/ff6 550361 0 2022-01-31T19:44:34.246 INFO:tasks.workunit.client.1.smithi171.stdout:9/948: creat dd/df/d15/d38/d9c/da7/f13e x:0 0 0 2022-01-31T19:44:34.249 INFO:tasks.workunit.client.1.smithi171.stdout:1/950: creat dd/d23/d2e/d5e/f118 x:0 0 0 2022-01-31T19:44:34.256 INFO:tasks.workunit.client.1.smithi171.stdout:9/949: mkdir dd/d109/de6/d7c/dcf/d108/d13f 0 2022-01-31T19:44:34.409 INFO:tasks.workunit.client.1.smithi171.stdout:1/951: creat dd/d23/d2e/d5e/d81/d98/d80/f119 x:0 0 0 2022-01-31T19:44:34.409 INFO:tasks.workunit.client.1.smithi171.stdout:7/703: dwrite d1/d26/d81/f97 [0,4194304] 0 2022-01-31T19:44:34.409 INFO:tasks.workunit.client.1.smithi171.stdout:9/950: dread dd/df/d3f/f4c [0,4194304] 0 2022-01-31T19:44:34.410 INFO:tasks.workunit.client.1.smithi171.stdout:9/951: chown dd/df/d15/d38/d9c/cb1 56 1 2022-01-31T19:44:34.410 INFO:tasks.workunit.client.1.smithi171.stdout:9/952: rmdir dd/d109 39 2022-01-31T19:44:34.410 INFO:tasks.workunit.client.1.smithi171.stdout:9/953: creat dd/df/d15/d1a/d84/dfe/f140 x:0 0 0 2022-01-31T19:44:34.410 INFO:tasks.workunit.client.1.smithi171.stdout:1/952: dread f9 [0,4194304] 0 2022-01-31T19:44:34.411 INFO:tasks.workunit.client.1.smithi171.stdout:1/953: rename dd/f91 to dd/d23/d74/dcb/f11a 0 2022-01-31T19:44:34.411 INFO:tasks.workunit.client.1.smithi171.stdout:9/954: rmdir dd/df/d15/d1a/d84/dca 39 2022-01-31T19:44:34.411 INFO:tasks.workunit.client.1.smithi171.stdout:4/806: dwrite d2/d3/d5/dc/d4b/f81 [0,4194304] 0 2022-01-31T19:44:34.411 INFO:tasks.workunit.client.1.smithi171.stdout:1/954: getdents dd/df/d84/dea/d64/db9 0 2022-01-31T19:44:34.412 INFO:tasks.workunit.client.1.smithi171.stdout:5/906: dwrite d5/d90/db7/d15/f64 [0,4194304] 0 2022-01-31T19:44:34.412 INFO:tasks.workunit.client.1.smithi171.stdout:5/907: chown d5/d90/db7/d15/d23 628604 1 2022-01-31T19:44:34.412 INFO:tasks.workunit.client.1.smithi171.stdout:5/908: fsync d5/d90/db7/d15/d23/d3d/fb2 0 2022-01-31T19:44:34.412 INFO:tasks.workunit.client.1.smithi171.stdout:6/946: dwrite dc/f4a [0,4194304] 0 2022-01-31T19:44:34.413 INFO:tasks.workunit.client.1.smithi171.stdout:4/807: creat d2/d3/d5/d41/dcf/d12/d15/d57/dad/f10f x:0 0 0 2022-01-31T19:44:34.413 INFO:tasks.workunit.client.1.smithi171.stdout:1/955: rename dd/df/d19/dce to dd/df/d84/dea/d64/d11b 0 2022-01-31T19:44:34.413 INFO:tasks.workunit.client.1.smithi171.stdout:4/808: mknod d2/d3/d5/d70/dee/c110 0 2022-01-31T19:44:34.413 INFO:tasks.workunit.client.1.smithi171.stdout:4/809: rmdir d2/d3/d5/dc 39 2022-01-31T19:44:34.413 INFO:tasks.workunit.client.1.smithi171.stdout:6/947: rename dc/d18 to dc/d19/d29/d123/d110/d142 0 2022-01-31T19:44:34.414 INFO:tasks.workunit.client.1.smithi171.stdout:4/810: mknod d2/d3/d5/d70/d7c/c111 0 2022-01-31T19:44:34.414 INFO:tasks.workunit.client.1.smithi171.stdout:0/933: dwrite d9/db/d59/d71/fe2 [0,4194304] 0 2022-01-31T19:44:34.414 INFO:tasks.workunit.client.1.smithi171.stdout:6/948: mknod dc/d19/d29/daf/ddb/c143 0 2022-01-31T19:44:34.414 INFO:tasks.workunit.client.1.smithi171.stdout:0/934: mkdir d9/d24/d4c/d152 0 2022-01-31T19:44:34.415 INFO:tasks.workunit.client.1.smithi171.stdout:9/955: dread dd/df/d15/d1a/d84/dca/fe2 [0,4194304] 0 2022-01-31T19:44:34.415 INFO:tasks.workunit.client.1.smithi171.stdout:7/704: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d7e/fe1 [0,4194304] 0 2022-01-31T19:44:34.415 INFO:tasks.workunit.client.1.smithi171.stdout:7/705: write d1/d1e/d2a/fa6 [528740,18623] 0 2022-01-31T19:44:34.415 INFO:tasks.workunit.client.1.smithi171.stdout:7/706: dread - d1/d1e/d2a/d58/fb4 zero size 2022-01-31T19:44:34.416 INFO:tasks.workunit.client.1.smithi171.stdout:6/949: mkdir dc/d19/d29/d27/d144 0 2022-01-31T19:44:34.416 INFO:tasks.workunit.client.1.smithi171.stdout:6/950: write dc/d21/fab [1613303,47653] 0 2022-01-31T19:44:34.416 INFO:tasks.workunit.client.1.smithi171.stdout:8/885: dwrite df/d27/fa3 [0,4194304] 0 2022-01-31T19:44:34.416 INFO:tasks.workunit.client.1.smithi171.stdout:8/886: dread - df/d1a/dae/f123 zero size 2022-01-31T19:44:34.416 INFO:tasks.workunit.client.1.smithi171.stdout:9/956: write dd/df/d15/d38/d9c/fba [1281883,37514] 0 2022-01-31T19:44:34.417 INFO:tasks.workunit.client.1.smithi171.stdout:1/956: dwrite dd/d23/d2e/d5e/d81/d98/d80/f8e [0,4194304] 0 2022-01-31T19:44:34.417 INFO:tasks.workunit.client.1.smithi171.stdout:7/707: rename d1/lac to d1/d1e/d2a/d29/le7 0 2022-01-31T19:44:34.417 INFO:tasks.workunit.client.1.smithi171.stdout:6/951: truncate dc/d19/d29/d123/d110/d142/d86/d119/d3b/d67/dd1/fd3 984778 0 2022-01-31T19:44:34.418 INFO:tasks.workunit.client.1.smithi171.stdout:5/909: dwrite d5/d59/f122 [0,4194304] 0 2022-01-31T19:44:34.418 INFO:tasks.workunit.client.1.smithi171.stdout:8/887: mkdir df/d18/d19/d21/dcd/d12c 0 2022-01-31T19:44:34.418 INFO:tasks.workunit.client.1.smithi171.stdout:9/957: mkdir dd/df/d3f/db9/d141 0 2022-01-31T19:44:34.418 INFO:tasks.workunit.client.1.smithi171.stdout:9/958: dread - dd/d109/de6/d7c/dcf/fd4 zero size 2022-01-31T19:44:34.418 INFO:tasks.workunit.client.1.smithi171.stdout:9/959: chown dd/d109/de6/d7c/d7d/db6/f11e 7075 1 2022-01-31T19:44:34.419 INFO:tasks.workunit.client.1.smithi171.stdout:0/935: dwrite d9/d24/d92/d9d/dc6/f144 [0,4194304] 0 2022-01-31T19:44:34.419 INFO:tasks.workunit.client.1.smithi171.stdout:0/936: fdatasync d9/db/fc4 0 2022-01-31T19:44:34.419 INFO:tasks.workunit.client.1.smithi171.stdout:0/937: fsync d9/d24/d92/d9d/d9f/fb7 0 2022-01-31T19:44:34.419 INFO:tasks.workunit.client.1.smithi171.stdout:0/938: write d9/d24/d4c/f78 [400592,110251] 0 2022-01-31T19:44:34.420 INFO:tasks.workunit.client.1.smithi171.stdout:1/957: mkdir dd/df/d84/dea/d35/d11c 0 2022-01-31T19:44:34.420 INFO:tasks.workunit.client.1.smithi171.stdout:1/958: write dd/da2/fb2 [2156615,42725] 0 2022-01-31T19:44:34.420 INFO:tasks.workunit.client.1.smithi171.stdout:4/811: dwrite d2/f90 [0,4194304] 0 2022-01-31T19:44:34.420 INFO:tasks.workunit.client.1.smithi171.stdout:4/812: dread - d2/d3/d5/fd2 zero size 2022-01-31T19:44:34.420 INFO:tasks.workunit.client.1.smithi171.stdout:4/813: creat d2/d3/d5/d41/dcf/d12/d15/dc4/f112 x:0 0 0 2022-01-31T19:44:34.421 INFO:tasks.workunit.client.1.smithi171.stdout:8/888: symlink df/d18/d79/da2/ddd/d125/l12d 0 2022-01-31T19:44:34.421 INFO:tasks.workunit.client.1.smithi171.stdout:0/939: symlink d9/db/d59/l153 0 2022-01-31T19:44:34.421 INFO:tasks.workunit.client.1.smithi171.stdout:9/960: dread dd/d109/de6/f1f [0,4194304] 0 2022-01-31T19:44:34.421 INFO:tasks.workunit.client.1.smithi171.stdout:7/708: dwrite d1/d26/d81/fb0 [0,4194304] 0 2022-01-31T19:44:34.422 INFO:tasks.workunit.client.1.smithi171.stdout:7/709: unlink d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f35 0 2022-01-31T19:44:34.422 INFO:tasks.workunit.client.1.smithi171.stdout:7/710: creat d1/d1e/d2a/d29/d31/d30/d3c/dc0/fe8 x:0 0 0 2022-01-31T19:44:34.422 INFO:tasks.workunit.client.1.smithi171.stdout:1/959: link dd/d23/d2e/d5e/d81/d98/d80/f119 dd/df/d84/dea/d35/dd6/d113/f11d 0 2022-01-31T19:44:34.422 INFO:tasks.workunit.client.1.smithi171.stdout:1/960: creat dd/d23/dd4/de9/f11e x:0 0 0 2022-01-31T19:44:34.422 INFO:tasks.workunit.client.1.smithi171.stdout:1/961: write dd/d23/d2e/fe6 [879631,104383] 0 2022-01-31T19:44:34.423 INFO:tasks.workunit.client.1.smithi171.stdout:1/962: chown dd/df/d84/dea/d64/db9/fd2 26332 1 2022-01-31T19:44:34.423 INFO:tasks.workunit.client.1.smithi171.stdout:6/952: dwrite dc/d19/d29/f63 [0,4194304] 0 2022-01-31T19:44:34.423 INFO:tasks.workunit.client.1.smithi171.stdout:8/889: symlink df/d18/d19/d21/dcd/l12e 0 2022-01-31T19:44:34.423 INFO:tasks.workunit.client.1.smithi171.stdout:0/940: creat d9/d24/d92/d12d/dbb/f154 x:0 0 0 2022-01-31T19:44:34.424 INFO:tasks.workunit.client.1.smithi171.stdout:6/953: dread dc/d21/d4f/f60 [0,4194304] 0 2022-01-31T19:44:34.424 INFO:tasks.workunit.client.1.smithi171.stdout:8/890: symlink df/d18/d19/d21/l12f 0 2022-01-31T19:44:34.424 INFO:tasks.workunit.client.1.smithi171.stdout:8/891: getdents df/d18/d19/d21/dcd/d12c 0 2022-01-31T19:44:34.424 INFO:tasks.workunit.client.1.smithi171.stdout:0/941: rmdir d9/d24/d4c/d35 39 2022-01-31T19:44:34.424 INFO:tasks.workunit.client.1.smithi171.stdout:0/942: write d9/d24/d92/f9a [1326331,45579] 0 2022-01-31T19:44:34.425 INFO:tasks.workunit.client.1.smithi171.stdout:6/954: getdents dc/d1e 0 2022-01-31T19:44:34.425 INFO:tasks.workunit.client.1.smithi171.stdout:8/892: mknod df/d5d/d6c/d113/c130 0 2022-01-31T19:44:34.425 INFO:tasks.workunit.client.1.smithi171.stdout:8/893: readlink df/d18/d19/d21/d88/l9a 0 2022-01-31T19:44:34.425 INFO:tasks.workunit.client.1.smithi171.stdout:0/943: mkdir d9/d24/d4c/d152/d155 0 2022-01-31T19:44:34.426 INFO:tasks.workunit.client.1.smithi171.stdout:0/944: truncate d9/db/d59/d71/dbe/fd8 1579970 0 2022-01-31T19:44:34.426 INFO:tasks.workunit.client.1.smithi171.stdout:6/955: rename dc/d19/d29/f26 to dc/d19/d29/d123/d110/d142/d86/d119/f145 0 2022-01-31T19:44:34.426 INFO:tasks.workunit.client.1.smithi171.stdout:6/956: chown dc/d19/d30/d3c/d100 0 1 2022-01-31T19:44:34.426 INFO:tasks.workunit.client.1.smithi171.stdout:6/957: read dc/d19/d29/daf/ff5 [99337,115520] 0 2022-01-31T19:44:34.426 INFO:tasks.workunit.client.1.smithi171.stdout:7/711: dwrite d1/d1e/d2a/d58/fb4 [0,4194304] 0 2022-01-31T19:44:34.427 INFO:tasks.workunit.client.1.smithi171.stdout:7/712: write d1/d1e/d2a/d29/d31/d30/d6b/f86 [3506167,10648] 0 2022-01-31T19:44:34.427 INFO:tasks.workunit.client.1.smithi171.stdout:4/814: write d2/d3/d5/dc/f42 [6021734,27149] 0 2022-01-31T19:44:34.427 INFO:tasks.workunit.client.1.smithi171.stdout:4/815: fsync d2/d3/d5/dc/d4b/f6e 0 2022-01-31T19:44:34.427 INFO:tasks.workunit.client.1.smithi171.stdout:9/961: dwrite dd/df/d15/d1a/d64/fab [0,4194304] 0 2022-01-31T19:44:34.428 INFO:tasks.workunit.client.1.smithi171.stdout:9/962: creat dd/df/d15/da2/f142 x:0 0 0 2022-01-31T19:44:34.428 INFO:tasks.workunit.client.1.smithi171.stdout:9/963: creat dd/df/d15/d38/d4d/d80/f143 x:0 0 0 2022-01-31T19:44:34.428 INFO:tasks.workunit.client.1.smithi171.stdout:9/964: chown dd/df/d15/d38/d41/f76 14147 1 2022-01-31T19:44:34.428 INFO:tasks.workunit.client.1.smithi171.stdout:8/894: rename df/d18/d19/d21/d67/f110 to df/f131 0 2022-01-31T19:44:34.429 INFO:tasks.workunit.client.1.smithi171.stdout:8/895: creat df/d2a/f132 x:0 0 0 2022-01-31T19:44:34.429 INFO:tasks.workunit.client.1.smithi171.stdout:7/713: write d1/d1e/d2a/d29/d31/d30/d6b/f7c [623141,63748] 0 2022-01-31T19:44:34.429 INFO:tasks.workunit.client.1.smithi171.stdout:0/945: dread d9/f5c [0,4194304] 0 2022-01-31T19:44:34.429 INFO:tasks.workunit.client.1.smithi171.stdout:9/965: rename dd/df/d15/d1a/d64/fab to dd/d109/de6/d7c/d7d/db6/f144 0 2022-01-31T19:44:34.429 INFO:tasks.workunit.client.1.smithi171.stdout:8/896: symlink df/d18/d19/d21/d67/df3/l133 0 2022-01-31T19:44:34.430 INFO:tasks.workunit.client.1.smithi171.stdout:7/714: symlink d1/d1e/d2a/d29/d31/d30/d3c/d7e/le9 0 2022-01-31T19:44:34.430 INFO:tasks.workunit.client.1.smithi171.stdout:8/897: read f6 [2184613,82069] 0 2022-01-31T19:44:34.430 INFO:tasks.workunit.client.1.smithi171.stdout:0/946: link d9/d24/d4c/d7d/l13a d9/db/dc/d3f/d6f/l156 0 2022-01-31T19:44:34.430 INFO:tasks.workunit.client.1.smithi171.stdout:9/966: link dd/d109/de6/da8/fe1 dd/df/d15/d38/f145 0 2022-01-31T19:44:34.431 INFO:tasks.workunit.client.1.smithi171.stdout:9/967: stat dd/f6a 0 2022-01-31T19:44:34.431 INFO:tasks.workunit.client.1.smithi171.stdout:9/968: fsync dd/df/d15/d53/fc4 0 2022-01-31T19:44:34.431 INFO:tasks.workunit.client.1.smithi171.stdout:6/958: dwrite dc/d19/d30/d3c/d80/ff9 [0,4194304] 0 2022-01-31T19:44:34.431 INFO:tasks.workunit.client.1.smithi171.stdout:8/898: rename df/d18/d19/d21/d88/f92 to df/d18/da4/f134 0 2022-01-31T19:44:34.431 INFO:tasks.workunit.client.1.smithi171.stdout:8/899: creat df/d18/dad/f135 x:0 0 0 2022-01-31T19:44:34.432 INFO:tasks.workunit.client.1.smithi171.stdout:8/900: write df/f4f [1900313,120423] 0 2022-01-31T19:44:34.432 INFO:tasks.workunit.client.1.smithi171.stdout:0/947: creat d9/db/dc/d3f/d6f/f157 x:0 0 0 2022-01-31T19:44:34.432 INFO:tasks.workunit.client.1.smithi171.stdout:0/948: creat d9/db/dc/d3f/db9/f158 x:0 0 0 2022-01-31T19:44:34.432 INFO:tasks.workunit.client.1.smithi171.stdout:4/816: dwrite d2/d3/d5/d41/dcf/d12/f5a [0,4194304] 0 2022-01-31T19:44:34.432 INFO:tasks.workunit.client.1.smithi171.stdout:4/817: fdatasync d2/d3/d5/f46 0 2022-01-31T19:44:34.433 INFO:tasks.workunit.client.1.smithi171.stdout:4/818: chown d2/d3/de1/dd4/lda 445 1 2022-01-31T19:44:34.433 INFO:tasks.workunit.client.1.smithi171.stdout:9/969: truncate dd/df/d15/d1a/d84/f91 1907775 0 2022-01-31T19:44:34.433 INFO:tasks.workunit.client.1.smithi171.stdout:9/970: creat dd/df/d15/de4/f146 x:0 0 0 2022-01-31T19:44:34.433 INFO:tasks.workunit.client.1.smithi171.stdout:6/959: readlink dc/d19/d30/l81 0 2022-01-31T19:44:34.434 INFO:tasks.workunit.client.1.smithi171.stdout:6/960: write dc/d1e/f45 [4612298,122270] 0 2022-01-31T19:44:34.434 INFO:tasks.workunit.client.1.smithi171.stdout:0/949: rename f6 to d9/d24/d92/d9d/f159 0 2022-01-31T19:44:34.434 INFO:tasks.workunit.client.1.smithi171.stdout:4/819: mkdir d2/d3/d5/d91/da6/d113 0 2022-01-31T19:44:34.434 INFO:tasks.workunit.client.1.smithi171.stdout:9/971: rmdir dd/df/d15/d38/d9c/da7/d12a 0 2022-01-31T19:44:34.437 INFO:tasks.workunit.client.1.smithi171.stdout:7/715: dwrite d1/d26/d81/fb0 [0,4194304] 0 2022-01-31T19:44:34.438 INFO:tasks.workunit.client.1.smithi171.stdout:7/716: creat d1/d1e/d2a/d29/fea x:0 0 0 2022-01-31T19:44:34.438 INFO:tasks.workunit.client.1.smithi171.stdout:8/901: dwrite df/d18/d79/d7a/f10b [0,4194304] 0 2022-01-31T19:44:34.438 INFO:tasks.workunit.client.1.smithi171.stdout:7/717: write d1/d1e/d2a/d29/d31/d30/f73 [1473694,41882] 0 2022-01-31T19:44:34.438 INFO:tasks.workunit.client.1.smithi171.stdout:8/902: write df/f20 [2891563,507] 0 2022-01-31T19:44:34.439 INFO:tasks.workunit.client.1.smithi171.stdout:0/950: truncate d9/d24/d4c/f3d 6834245 0 2022-01-31T19:44:34.439 INFO:tasks.workunit.client.1.smithi171.stdout:4/820: creat d2/d3/d5/d41/dcf/f114 x:0 0 0 2022-01-31T19:44:34.439 INFO:tasks.workunit.client.1.smithi171.stdout:4/821: chown d2/d3/d5/d3b/faf 154719476 1 2022-01-31T19:44:34.439 INFO:tasks.workunit.client.1.smithi171.stdout:8/903: getdents df/d18/dad/d116 0 2022-01-31T19:44:34.440 INFO:tasks.workunit.client.1.smithi171.stdout:8/904: chown df/fdc 105 1 2022-01-31T19:44:34.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/972: dwrite dd/df/d15/f3a [4194304,4194304] 0 2022-01-31T19:44:34.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/973: fsync dd/df/d15/d1a/d64/d121/f129 0 2022-01-31T19:44:34.440 INFO:tasks.workunit.client.1.smithi171.stdout:9/974: write dd/d109/de6/d7c/d7d/db6/f11e [348529,3847] 0 2022-01-31T19:44:34.440 INFO:tasks.workunit.client.1.smithi171.stdout:0/951: creat d9/db/d59/d71/d98/f15a x:0 0 0 2022-01-31T19:44:34.441 INFO:tasks.workunit.client.1.smithi171.stdout:8/905: mkdir df/d18/d19/d21/dcd/d12c/d136 0 2022-01-31T19:44:34.441 INFO:tasks.workunit.client.1.smithi171.stdout:7/718: dread d1/d1e/d2a/d29/fa7 [0,4194304] 0 2022-01-31T19:44:34.441 INFO:tasks.workunit.client.1.smithi171.stdout:8/906: fdatasync df/d18/d19/d46/f8d 0 2022-01-31T19:44:34.441 INFO:tasks.workunit.client.1.smithi171.stdout:7/719: dread - d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/fc8 zero size 2022-01-31T19:44:34.441 INFO:tasks.workunit.client.1.smithi171.stdout:7/720: dread - d1/d1e/d2a/d29/f66 zero size 2022-01-31T19:44:34.442 INFO:tasks.workunit.client.1.smithi171.stdout:0/952: creat d9/db/d59/de9/f15b x:0 0 0 2022-01-31T19:44:34.442 INFO:tasks.workunit.client.1.smithi171.stdout:0/953: read - d9/db/dc/d3f/d6f/ff9 zero size 2022-01-31T19:44:34.442 INFO:tasks.workunit.client.1.smithi171.stdout:7/721: symlink d1/d1e/d2a/d29/d31/d30/dc4/leb 0 2022-01-31T19:44:34.442 INFO:tasks.workunit.client.1.smithi171.stdout:7/722: truncate d1/d1e/d2a/d29/d77/dab/fce 944912 0 2022-01-31T19:44:34.443 INFO:tasks.workunit.client.1.smithi171.stdout:0/954: dread d9/db/d59/d71/dbe/fd8 [0,4194304] 0 2022-01-31T19:44:34.443 INFO:tasks.workunit.client.1.smithi171.stdout:6/961: truncate dc/d19/d29/d123/d110/d142/d86/d119/f124 3658222 0 2022-01-31T19:44:34.443 INFO:tasks.workunit.client.1.smithi171.stdout:6/962: creat dc/d21/f146 x:0 0 0 2022-01-31T19:44:34.443 INFO:tasks.workunit.client.1.smithi171.stdout:6/963: read dc/d19/d29/d123/d110/d142/d86/d119/d111/fb8 [258382,7636] 0 2022-01-31T19:44:34.444 INFO:tasks.workunit.client.1.smithi171.stdout:6/964: rmdir dc/d19/d29/d123/d110/d142/d86/d119/d68 39 2022-01-31T19:44:34.444 INFO:tasks.workunit.client.1.smithi171.stdout:6/965: truncate dc/d1e/fa4 868210 0 2022-01-31T19:44:34.445 INFO:tasks.workunit.client.1.smithi171.stdout:0/955: dread d9/db/dc/d3f/f4a [0,4194304] 0 2022-01-31T19:44:34.447 INFO:tasks.workunit.client.1.smithi171.stdout:7/723: dread d1/d1e/d2a/d29/d31/d30/fc6 [0,4194304] 0 2022-01-31T19:44:34.448 INFO:tasks.workunit.client.1.smithi171.stdout:7/724: dread - d1/d26/d47/d8f/fc5 zero size 2022-01-31T19:44:34.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/822: dwrite d2/d3/d5/d41/dcf/f27 [0,4194304] 0 2022-01-31T19:44:34.451 INFO:tasks.workunit.client.1.smithi171.stdout:4/823: readlink d2/d3/d5/d41/l85 0 2022-01-31T19:44:34.452 INFO:tasks.workunit.client.1.smithi171.stdout:4/824: readlink d2/d3/d5/d41/dcf/l106 0 2022-01-31T19:44:34.452 INFO:tasks.workunit.client.1.smithi171.stdout:4/825: dread - d2/d3/d5/d41/dcf/f114 zero size 2022-01-31T19:44:34.452 INFO:tasks.workunit.client.1.smithi171.stdout:4/826: readlink d2/d3/d5/d41/dcf/d12/d15/l31 0 2022-01-31T19:44:34.453 INFO:tasks.workunit.client.1.smithi171.stdout:4/827: mknod d2/d3/d5/d70/d7c/c115 0 2022-01-31T19:44:34.453 INFO:tasks.workunit.client.1.smithi171.stdout:4/828: write d2/fa2 [4431985,52817] 0 2022-01-31T19:44:34.453 INFO:tasks.workunit.client.1.smithi171.stdout:7/725: write d1/d1e/d2a/d58/fb4 [3667157,75423] 0 2022-01-31T19:44:34.456 INFO:tasks.workunit.client.1.smithi171.stdout:6/966: dread f9 [4194304,4194304] 0 2022-01-31T19:44:34.456 INFO:tasks.workunit.client.1.smithi171.stdout:6/967: write dc/d21/f98 [713842,29662] 0 2022-01-31T19:44:34.456 INFO:tasks.workunit.client.1.smithi171.stdout:6/968: creat dc/d1e/d3f/f147 x:0 0 0 2022-01-31T19:44:34.456 INFO:tasks.workunit.client.1.smithi171.stdout:6/969: chown dc/d19/d30/d3c/d80/f141 6 1 2022-01-31T19:44:34.457 INFO:tasks.workunit.client.1.smithi171.stdout:7/726: dread d1/d1b/f4c [0,4194304] 0 2022-01-31T19:44:34.458 INFO:tasks.workunit.client.1.smithi171.stdout:7/727: fdatasync d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f18 0 2022-01-31T19:44:34.458 INFO:tasks.workunit.client.1.smithi171.stdout:6/970: fdatasync dc/d19/d29/d6a/fa0 0 2022-01-31T19:44:34.458 INFO:tasks.workunit.client.1.smithi171.stdout:6/971: creat dc/d1e/d3f/f148 x:0 0 0 2022-01-31T19:44:34.461 INFO:tasks.workunit.client.1.smithi171.stdout:9/975: dwrite dd/df/f138 [0,4194304] 0 2022-01-31T19:44:34.461 INFO:tasks.workunit.client.1.smithi171.stdout:8/907: dwrite df/d18/d19/d21/d67/df3/f121 [0,4194304] 0 2022-01-31T19:44:34.462 INFO:tasks.workunit.client.1.smithi171.stdout:9/976: creat dd/df/f147 x:0 0 0 2022-01-31T19:44:34.462 INFO:tasks.workunit.client.1.smithi171.stdout:0/956: rmdir d9/db/d59/de9 39 2022-01-31T19:44:34.462 INFO:tasks.workunit.client.1.smithi171.stdout:0/957: dread d9/db/d59/d71/dbe/fd8 [0,4194304] 0 2022-01-31T19:44:34.463 INFO:tasks.workunit.client.1.smithi171.stdout:7/728: creat d1/d1e/d2a/d29/d77/dab/dd5/fec x:0 0 0 2022-01-31T19:44:34.463 INFO:tasks.workunit.client.1.smithi171.stdout:7/729: fsync d1/d1e/d2a/d29/d31/d30/d3c/d7e/fc7 0 2022-01-31T19:44:34.464 INFO:tasks.workunit.client.1.smithi171.stdout:7/730: creat d1/d1e/d2a/d29/fed x:0 0 0 2022-01-31T19:44:34.466 INFO:tasks.workunit.client.1.smithi171.stdout:6/972: creat dc/d19/d29/d123/dee/d116/f149 x:0 0 0 2022-01-31T19:44:34.470 INFO:tasks.workunit.client.1.smithi171.stdout:8/908: mknod df/d18/d19/d21/d88/c137 0 2022-01-31T19:44:34.471 INFO:tasks.workunit.client.1.smithi171.stdout:4/829: dread d2/d7f/fa8 [0,4194304] 0 2022-01-31T19:44:34.475 INFO:tasks.workunit.client.1.smithi171.stdout:9/977: rename dd/df/d15/f2f to dd/df/d86/d11d/f148 0 2022-01-31T19:44:34.475 INFO:tasks.workunit.client.1.smithi171.stdout:9/978: dread - dd/df/d15/da2/fd9 zero size 2022-01-31T19:44:34.476 INFO:tasks.workunit.client.1.smithi171.stdout:8/909: mkdir df/d18/d19/d21/dab/d138 0 2022-01-31T19:44:34.478 INFO:tasks.workunit.client.1.smithi171.stdout:4/830: creat d2/d3/d5/d41/f116 x:0 0 0 2022-01-31T19:44:34.478 INFO:tasks.workunit.client.1.smithi171.stdout:4/831: write d2/fba [1265657,32192] 0 2022-01-31T19:44:34.479 INFO:tasks.workunit.client.1.smithi171.stdout:9/979: rename dd/l5d to dd/df/d15/d38/d13b/l149 0 2022-01-31T19:44:34.480 INFO:tasks.workunit.client.1.smithi171.stdout:8/910: creat df/dcc/f139 x:0 0 0 2022-01-31T19:44:34.481 INFO:tasks.workunit.client.1.smithi171.stdout:8/911: chown df/d18/dad/d116/df5 105960114 1 2022-01-31T19:44:34.484 INFO:tasks.workunit.client.1.smithi171.stdout:4/832: getdents d2/d3/d5/dc 0 2022-01-31T19:44:34.484 INFO:tasks.workunit.client.1.smithi171.stdout:4/833: stat d2/f90 0 2022-01-31T19:44:34.486 INFO:tasks.workunit.client.1.smithi171.stdout:9/980: symlink dd/df/d15/d38/d9c/l14a 0 2022-01-31T19:44:34.488 INFO:tasks.workunit.client.1.smithi171.stdout:8/912: unlink df/d18/d79/da2/ddd/d125/l12d 0 2022-01-31T19:44:34.490 INFO:tasks.workunit.client.1.smithi171.stdout:9/981: truncate dd/f79 717753 0 2022-01-31T19:44:34.491 INFO:tasks.workunit.client.1.smithi171.stdout:8/913: creat df/d18/d19/f13a x:0 0 0 2022-01-31T19:44:34.492 INFO:tasks.workunit.client.1.smithi171.stdout:9/982: mkdir dd/d109/de6/d14b 0 2022-01-31T19:44:34.493 INFO:tasks.workunit.client.1.smithi171.stdout:8/914: stat df/d5d/d6c/d87/de1/c103 0 2022-01-31T19:44:34.493 INFO:tasks.workunit.client.1.smithi171.stdout:0/958: dwrite d9/d24/d4c/d35/fbc [0,4194304] 0 2022-01-31T19:44:34.494 INFO:tasks.workunit.client.1.smithi171.stdout:8/915: read df/d18/d19/d36/f81 [1911598,31803] 0 2022-01-31T19:44:34.495 INFO:tasks.workunit.client.1.smithi171.stdout:8/916: stat df/d18/d19/db2/c117 0 2022-01-31T19:44:34.497 INFO:tasks.workunit.client.1.smithi171.stdout:0/959: unlink d9/db/d59/d71/dbe/fd6 0 2022-01-31T19:44:34.497 INFO:tasks.workunit.client.1.smithi171.stdout:8/917: creat df/d18/d79/da2/d107/f13b x:0 0 0 2022-01-31T19:44:34.498 INFO:tasks.workunit.client.1.smithi171.stdout:0/960: symlink d9/d24/d4c/d101/d132/l15c 0 2022-01-31T19:44:34.499 INFO:tasks.workunit.client.1.smithi171.stdout:8/918: mknod df/d18/d19/d82/c13c 0 2022-01-31T19:44:34.500 INFO:tasks.workunit.client.1.smithi171.stdout:0/961: truncate d9/db/d59/d71/dbe/fef 2096509 0 2022-01-31T19:44:34.501 INFO:tasks.workunit.client.1.smithi171.stdout:8/919: creat df/d18/d19/d21/dcd/d12c/f13d x:0 0 0 2022-01-31T19:44:34.501 INFO:tasks.workunit.client.1.smithi171.stdout:8/920: chown df/d18/da4/ca6 182144929 1 2022-01-31T19:44:34.501 INFO:tasks.workunit.client.1.smithi171.stdout:8/921: dread - df/d18/d19/db2/ff0 zero size 2022-01-31T19:44:34.502 INFO:tasks.workunit.client.1.smithi171.stdout:8/922: write df/d1a/f58 [6419115,9895] 0 2022-01-31T19:44:34.502 INFO:tasks.workunit.client.1.smithi171.stdout:8/923: write df/d18/d19/f61 [777000,126523] 0 2022-01-31T19:44:34.504 INFO:tasks.workunit.client.1.smithi171.stdout:6/973: dwrite dc/d19/d29/f34 [4194304,4194304] 0 2022-01-31T19:44:34.504 INFO:tasks.workunit.client.1.smithi171.stdout:6/974: fdatasync dc/d21/f98 0 2022-01-31T19:44:34.505 INFO:tasks.workunit.client.1.smithi171.stdout:4/834: dwrite d2/f14 [0,4194304] 0 2022-01-31T19:44:34.507 INFO:tasks.workunit.client.1.smithi171.stdout:6/975: symlink dc/d21/de9/l14a 0 2022-01-31T19:44:34.507 INFO:tasks.workunit.client.1.smithi171.stdout:0/962: write d9/db/dc/d3f/db9/d7a/fa8 [3512705,72546] 0 2022-01-31T19:44:34.516 INFO:tasks.workunit.client.1.smithi171.stdout:9/983: dwrite dd/df/d3f/fd1 [0,4194304] 0 2022-01-31T19:44:34.517 INFO:tasks.workunit.client.1.smithi171.stdout:8/924: getdents df/d18/d79/da2/ddd/d125 0 2022-01-31T19:44:34.518 INFO:tasks.workunit.client.1.smithi171.stdout:8/925: fdatasync df/d18/d79/d7a/f93 0 2022-01-31T19:44:34.520 INFO:tasks.workunit.client.1.smithi171.stdout:0/963: unlink d9/d24/d4c/d101/d132/l15c 0 2022-01-31T19:44:34.521 INFO:tasks.workunit.client.1.smithi171.stdout:0/964: truncate d9/d24/d4c/d101/d132/f146 785611 0 2022-01-31T19:44:34.521 INFO:tasks.workunit.client.1.smithi171.stdout:0/965: fdatasync d9/db/f131 0 2022-01-31T19:44:34.526 INFO:tasks.workunit.client.1.smithi171.stdout:9/984: unlink dd/df/d15/l103 0 2022-01-31T19:44:34.529 INFO:tasks.workunit.client.1.smithi171.stdout:9/985: write dd/df/d15/f36 [3110094,165] 0 2022-01-31T19:44:34.530 INFO:tasks.workunit.client.1.smithi171.stdout:0/966: mkdir d9/d24/d4c/d101/d15d 0 2022-01-31T19:44:34.536 INFO:tasks.workunit.client.1.smithi171.stdout:0/967: rename d9/db/fc9 to d9/d24/d4c/d35/d67/f15e 0 2022-01-31T19:44:34.542 INFO:tasks.workunit.client.1.smithi171.stdout:6/976: dwrite dc/d19/d29/d123/dee/f128 [0,4194304] 0 2022-01-31T19:44:34.545 INFO:tasks.workunit.client.1.smithi171.stdout:6/977: rename dc/d19/d29/d123/d110/d142/d86/d119/d3b/d67/f13e to dc/d19/d29/f14b 0 2022-01-31T19:44:34.552 INFO:tasks.workunit.client.1.smithi171.stdout:4/835: dwrite d2/fba [0,4194304] 0 2022-01-31T19:44:34.553 INFO:tasks.workunit.client.1.smithi171.stdout:4/836: truncate d2/d3/d5/dc/f6f 9387875 0 2022-01-31T19:44:34.554 INFO:tasks.workunit.client.1.smithi171.stdout:0/968: dwrite d9/db/dc/d3f/d6f/ff9 [0,4194304] 0 2022-01-31T19:44:34.554 INFO:tasks.workunit.client.1.smithi171.stdout:0/969: chown d9/d24/d92/d9d/d9f/c10b 7 1 2022-01-31T19:44:34.555 INFO:tasks.workunit.client.1.smithi171.stdout:4/837: rmdir d2/d3/de1 39 2022-01-31T19:44:34.556 INFO:tasks.workunit.client.1.smithi171.stdout:4/838: write d2/d3/d5/dc/fd6 [525037,18763] 0 2022-01-31T19:44:34.556 INFO:tasks.workunit.client.1.smithi171.stdout:4/839: write d2/d3/d5/d91/de4/f102 [133437,120714] 0 2022-01-31T19:44:34.556 INFO:tasks.workunit.client.1.smithi171.stdout:9/986: dwrite dd/df/d15/f4e [0,4194304] 0 2022-01-31T19:44:34.556 INFO:tasks.workunit.client.1.smithi171.stdout:9/987: fsync dd/df/d15/d38/d4d/ffd 0 2022-01-31T19:44:34.558 INFO:tasks.workunit.client.1.smithi171.stdout:0/970: creat d9/d24/d4c/d35/d67/f15f x:0 0 0 2022-01-31T19:44:34.561 INFO:tasks.workunit.client.1.smithi171.stdout:9/988: mkdir dd/d109/de6/d14c 0 2022-01-31T19:44:34.561 INFO:tasks.workunit.client.1.smithi171.stdout:9/989: stat dd/d109/de6/d7c/dcf/c11b 0 2022-01-31T19:44:34.561 INFO:tasks.workunit.client.1.smithi171.stdout:9/990: chown dd/d109/de6/c3e 1695564 1 2022-01-31T19:44:34.562 INFO:tasks.workunit.client.1.smithi171.stdout:9/991: chown dd/d109/de6/db5/ldc 0 1 2022-01-31T19:44:34.563 INFO:tasks.workunit.client.1.smithi171.stdout:4/840: unlink d2/d3/d5/d41/dcf/f9f 0 2022-01-31T19:44:34.566 INFO:tasks.workunit.client.1.smithi171.stdout:9/992: symlink dd/df/d15/d1a/d64/d6e/ddd/l14d 0 2022-01-31T19:44:34.568 INFO:tasks.workunit.client.1.smithi171.stdout:4/841: rmdir d2/d3/de1/dd4 39 2022-01-31T19:44:34.568 INFO:tasks.workunit.client.1.smithi171.stdout:4/842: creat d2/d7f/f117 x:0 0 0 2022-01-31T19:44:34.579 INFO:tasks.workunit.client.1.smithi171.stdout:6/978: dwrite dc/d19/dae/f107 [0,4194304] 0 2022-01-31T19:44:34.579 INFO:tasks.workunit.client.1.smithi171.stdout:9/993: dread dd/fc2 [0,4194304] 0 2022-01-31T19:44:34.582 INFO:tasks.workunit.client.1.smithi171.stdout:9/994: rename dd/df/d15/d38/d41/f50 to dd/df/d3f/db9/d141/f14e 0 2022-01-31T19:44:34.592 INFO:tasks.workunit.client.1.smithi171.stdout:9/995: dread dd/df/d15/d1a/f4b [0,4194304] 0 2022-01-31T19:44:34.592 INFO:tasks.workunit.client.1.smithi171.stdout:9/996: chown dd/df/d15/d1a/d64/dfc/f106 159513 1 2022-01-31T19:44:34.592 INFO:tasks.workunit.client.1.smithi171.stdout:4/843: dwrite d2/d3/d5/d41/dcf/d12/d15/dc4/fe0 [0,4194304] 0 2022-01-31T19:44:34.593 INFO:tasks.workunit.client.1.smithi171.stdout:6/979: dread dc/d19/d29/daf/ddb/f118 [0,4194304] 0 2022-01-31T19:44:34.593 INFO:tasks.workunit.client.1.smithi171.stdout:6/980: write dc/d19/d30/d3c/f4b [1499245,66610] 0 2022-01-31T19:44:34.593 INFO:tasks.workunit.client.1.smithi171.stdout:6/981: readlink dc/d1e/lbf 0 2022-01-31T19:44:34.594 INFO:tasks.workunit.client.1.smithi171.stdout:4/844: dread d2/d3/d5/dc/fd6 [0,4194304] 0 2022-01-31T19:44:34.597 INFO:tasks.workunit.client.1.smithi171.stdout:6/982: dread dc/d19/d29/d123/d110/d142/fe2 [4194304,4194304] 0 2022-01-31T19:44:34.599 INFO:tasks.workunit.client.1.smithi171.stdout:9/997: mknod dd/df/d15/d1a/d64/d6e/c14f 0 2022-01-31T19:44:34.600 INFO:tasks.workunit.client.1.smithi171.stdout:4/845: mkdir d2/d3/d5/d91/d118 0 2022-01-31T19:44:34.601 INFO:tasks.workunit.client.1.smithi171.stdout:9/998: mknod dd/d109/de6/c150 0 2022-01-31T19:44:34.602 INFO:tasks.workunit.client.1.smithi171.stdout:9/999: dread - dd/df/d15/d1a/d64/d6e/ddd/f10d zero size 2022-01-31T19:44:34.603 INFO:tasks.workunit.client.1.smithi171.stdout:4/846: link d2/d3/c49 d2/d3/d5/dc/c119 0 2022-01-31T19:44:34.604 INFO:tasks.workunit.client.1.smithi171.stdout:4/847: creat d2/d3/d5/d91/d77/f11a x:0 0 0 2022-01-31T19:44:34.605 INFO:tasks.workunit.client.1.smithi171.stdout:4/848: write d2/d3/d5/d41/dcf/d12/f8d [1003299,82590] 0 2022-01-31T19:44:34.605 INFO:tasks.workunit.client.1.smithi171.stdout:4/849: write d2/d3/d5/fd2 [283351,38634] 0 2022-01-31T19:44:34.625 INFO:tasks.workunit.client.1.smithi171.stdout:4/850: dwrite d2/f3c [4194304,4194304] 0 2022-01-31T19:44:34.627 INFO:tasks.workunit.client.1.smithi171.stdout:4/851: write d2/d3/d5/dc/fcc [742394,92055] 0 2022-01-31T19:44:34.632 INFO:tasks.workunit.client.1.smithi171.stdout:4/852: dread d2/d3/d5/d41/dcf/d12/f60 [0,4194304] 0 2022-01-31T19:44:34.633 INFO:tasks.workunit.client.1.smithi171.stdout:4/853: symlink d2/d3/d5/d41/dcf/d12/d15/dc4/l11b 0 2022-01-31T19:44:34.635 INFO:tasks.workunit.client.1.smithi171.stdout:4/854: creat d2/d3/d5/dc/d4b/d56/f11c x:0 0 0 2022-01-31T19:44:34.635 INFO:tasks.workunit.client.1.smithi171.stdout:4/855: creat d2/d7f/f11d x:0 0 0 2022-01-31T19:44:34.655 INFO:tasks.workunit.client.1.smithi171.stdout:1/963: sync 2022-01-31T19:44:34.655 INFO:tasks.workunit.client.1.smithi171.stdout:5/910: sync 2022-01-31T19:44:34.655 INFO:tasks.workunit.client.1.smithi171.stdout:1/964: chown dd/d23/d34/d4b/l4c 132287600 1 2022-01-31T19:44:34.655 INFO:tasks.workunit.client.1.smithi171.stdout:1/965: fdatasync dd/da2/fb2 0 2022-01-31T19:44:34.666 INFO:tasks.workunit.client.1.smithi171.stdout:4/856: mkdir d2/d3/d5/d41/dcf/d12/d15/dc4/dfe/d11e 0 2022-01-31T19:44:34.666 INFO:tasks.workunit.client.1.smithi171.stdout:4/857: write d2/d3/d5/dc/f87 [485029,101468] 0 2022-01-31T19:44:34.667 INFO:tasks.workunit.client.1.smithi171.stdout:4/858: fdatasync d2/d3/d5/fd2 0 2022-01-31T19:44:34.667 INFO:tasks.workunit.client.1.smithi171.stdout:4/859: dread - d2/d3/d5/d41/dcf/f109 zero size 2022-01-31T19:44:34.667 INFO:tasks.workunit.client.1.smithi171.stdout:4/860: stat d2/d3/d5/d41/dcf/dc7/fe8 0 2022-01-31T19:44:34.668 INFO:tasks.workunit.client.1.smithi171.stdout:1/966: link dd/d23/f82 dd/d23/d2e/d5e/dd8/f11f 0 2022-01-31T19:44:34.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/861: rename d2/d3/d5/d41/dcf/d12/d15/d57/dad/ffa to d2/d3/d5/d91/d77/da5/f11f 0 2022-01-31T19:44:34.675 INFO:tasks.workunit.client.1.smithi171.stdout:4/862: fsync d2/d3/d5/dc/d4b/dbc/fd0 0 2022-01-31T19:44:34.678 INFO:tasks.workunit.client.1.smithi171.stdout:4/863: rmdir d2/d3/d5/d70 39 2022-01-31T19:44:34.679 INFO:tasks.workunit.client.1.smithi171.stdout:4/864: mknod d2/d3/d5/d41/dcf/d12/c120 0 2022-01-31T19:44:34.681 INFO:tasks.workunit.client.1.smithi171.stdout:4/865: rename d2/d3/d5/d41/dcf/d12/f8d to d2/d3/d5/d41/dcf/d12/d15/d104/f121 0 2022-01-31T19:44:34.681 INFO:tasks.workunit.client.1.smithi171.stdout:4/866: fsync d2/d3/d5/d91/f76 0 2022-01-31T19:44:34.681 INFO:tasks.workunit.client.1.smithi171.stdout:4/867: chown d2/d3/d5/d41/dcf/d12/l4c 15 1 2022-01-31T19:44:34.683 INFO:tasks.workunit.client.1.smithi171.stdout:4/868: mkdir d2/d3/d5/d41/dcf/d122 0 2022-01-31T19:44:34.684 INFO:tasks.workunit.client.1.smithi171.stdout:4/869: symlink d2/d3/de1/l123 0 2022-01-31T19:44:34.685 INFO:tasks.workunit.client.1.smithi171.stdout:4/870: rmdir d2/d3/d5/d70/dee 39 2022-01-31T19:44:34.686 INFO:tasks.workunit.client.1.smithi171.stdout:4/871: rename d2/d3/d5/d41/cc5 to d2/d3/d5/d41/dcf/d12/d15/c124 0 2022-01-31T19:44:34.688 INFO:tasks.workunit.client.1.smithi171.stdout:4/872: write d2/d3/d5/d3b/faf [294752,122144] 0 2022-01-31T19:44:34.688 INFO:tasks.workunit.client.1.smithi171.stdout:4/873: dread - d2/d3/d5/d3b/fd9 zero size 2022-01-31T19:44:34.689 INFO:tasks.workunit.client.1.smithi171.stdout:4/874: stat d2/d3/d5/d91/fbe 0 2022-01-31T19:44:34.694 INFO:tasks.workunit.client.1.smithi171.stdout:4/875: creat d2/d3/d5/d91/d118/f125 x:0 0 0 2022-01-31T19:44:34.695 INFO:tasks.workunit.client.1.smithi171.stdout:1/967: dwrite dd/d23/d2e/d5e/dd8/fe7 [0,4194304] 0 2022-01-31T19:44:34.700 INFO:tasks.workunit.client.1.smithi171.stdout:1/968: rename dd/df/d84/dea/d64/da7/faf to dd/df/d84/dea/f120 0 2022-01-31T19:44:34.706 INFO:tasks.workunit.client.1.smithi171.stdout:1/969: rename dd/df/d84/dea/d35/ld1 to dd/d23/d2e/d5e/dd8/l121 0 2022-01-31T19:44:34.706 INFO:tasks.workunit.client.1.smithi171.stdout:1/970: readlink dd/df/d19/d1b/lba 0 2022-01-31T19:44:34.710 INFO:tasks.workunit.client.1.smithi171.stdout:1/971: unlink dd/d23/d34/d4b/f52 0 2022-01-31T19:44:34.710 INFO:tasks.workunit.client.1.smithi171.stdout:1/972: truncate dd/d23/d2e/d5e/d81/f93 5065084 0 2022-01-31T19:44:34.710 INFO:tasks.workunit.client.1.smithi171.stdout:1/973: stat dd/df/d19/d1b/fbe 0 2022-01-31T19:44:34.713 INFO:tasks.workunit.client.1.smithi171.stdout:1/974: rename dd/d23/d2e/d5e/d83/d116 to dd/df/d84/dea/d122 0 2022-01-31T19:44:34.714 INFO:tasks.workunit.client.1.smithi171.stdout:1/975: mknod dd/df/c123 0 2022-01-31T19:44:34.714 INFO:tasks.workunit.client.1.smithi171.stdout:1/976: write dd/d23/d2e/d5e/d81/fa8 [4379475,9740] 0 2022-01-31T19:44:34.733 INFO:tasks.workunit.client.1.smithi171.stdout:1/977: dwrite dd/d23/d2e/f10a [4194304,4194304] 0 2022-01-31T19:44:34.737 INFO:tasks.workunit.client.1.smithi171.stdout:1/978: mkdir dd/d23/d2e/d5e/d81/d98/dad/df5/d124 0 2022-01-31T19:44:34.746 INFO:tasks.workunit.client.1.smithi171.stdout:7/731: sync 2022-01-31T19:44:34.746 INFO:tasks.workunit.client.1.smithi171.stdout:7/732: chown d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/l1f 19 1 2022-01-31T19:44:34.748 INFO:tasks.workunit.client.1.smithi171.stdout:7/733: link d1/d1e/d2a/d29/d77/c9d d1/d26/d47/cee 0 2022-01-31T19:44:34.752 INFO:tasks.workunit.client.1.smithi171.stdout:7/734: rmdir d1/d1e/d2a/d58 39 2022-01-31T19:44:34.777 INFO:tasks.workunit.client.1.smithi171.stdout:7/735: dwrite d1/d1b/f4c [0,4194304] 0 2022-01-31T19:44:34.779 INFO:tasks.workunit.client.1.smithi171.stdout:7/736: write d1/d1e/d2a/d29/d31/d30/d3c/d7e/fe1 [408366,10163] 0 2022-01-31T19:44:34.783 INFO:tasks.workunit.client.1.smithi171.stdout:7/737: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/fef x:0 0 0 2022-01-31T19:44:34.799 INFO:tasks.workunit.client.1.smithi171.stdout:7/738: dwrite d1/d1e/d2a/f83 [0,4194304] 0 2022-01-31T19:44:34.817 INFO:tasks.workunit.client.1.smithi171.stdout:7/739: dread d1/d26/f71 [0,4194304] 0 2022-01-31T19:44:34.818 INFO:tasks.workunit.client.1.smithi171.stdout:7/740: write d1/d1e/d2a/d58/fb8 [96388,85158] 0 2022-01-31T19:44:34.819 INFO:tasks.workunit.client.1.smithi171.stdout:7/741: symlink d1/d1e/d2a/d29/d77/lf0 0 2022-01-31T19:44:34.819 INFO:tasks.workunit.client.1.smithi171.stdout:7/742: write d1/d1e/d2a/d29/fa7 [1068446,108429] 0 2022-01-31T19:44:34.820 INFO:tasks.workunit.client.1.smithi171.stdout:7/743: truncate d1/d1e/d2a/d29/d31/d30/f39 4109183 0 2022-01-31T19:44:34.821 INFO:tasks.workunit.client.1.smithi171.stdout:7/744: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/df1 0 2022-01-31T19:44:34.823 INFO:tasks.workunit.client.1.smithi171.stdout:7/745: symlink d1/d1e/d2a/d29/d31/d30/d3c/db9/lf2 0 2022-01-31T19:44:34.824 INFO:tasks.workunit.client.1.smithi171.stdout:7/746: symlink d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/lf3 0 2022-01-31T19:44:34.827 INFO:tasks.workunit.client.1.smithi171.stdout:7/747: link d1/d1e/d2a/d29/d77/dab/fce d1/d1e/d2a/d29/ff4 0 2022-01-31T19:44:34.842 INFO:tasks.workunit.client.1.smithi171.stdout:7/748: dwrite d1/d1e/d2a/d29/d31/d30/d3c/dc0/fe2 [0,4194304] 0 2022-01-31T19:44:34.845 INFO:tasks.workunit.client.1.smithi171.stdout:0/971: sync 2022-01-31T19:44:34.845 INFO:tasks.workunit.client.1.smithi171.stdout:8/926: sync 2022-01-31T19:44:34.845 INFO:tasks.workunit.client.1.smithi171.stdout:0/972: write d9/db/d59/fed [1375342,771] 0 2022-01-31T19:44:34.846 INFO:tasks.workunit.client.1.smithi171.stdout:0/973: fdatasync d9/d24/d92/d9d/f159 0 2022-01-31T19:44:34.849 INFO:tasks.workunit.client.1.smithi171.stdout:7/749: link d1/d1e/d2a/d29/d31/d30/d3c/l49 d1/d1e/d2a/d29/d31/d30/dc4/lf5 0 2022-01-31T19:44:34.860 INFO:tasks.workunit.client.1.smithi171.stdout:7/750: dread d1/d1e/d2a/d29/d31/d30/f73 [0,4194304] 0 2022-01-31T19:44:34.863 INFO:tasks.workunit.client.1.smithi171.stdout:8/927: dwrite df/d2a/d30/f63 [0,4194304] 0 2022-01-31T19:44:34.864 INFO:tasks.workunit.client.1.smithi171.stdout:8/928: write df/d18/d79/f9d [2111687,72538] 0 2022-01-31T19:44:34.875 INFO:tasks.workunit.client.1.smithi171.stdout:8/929: dread f0 [4194304,4194304] 0 2022-01-31T19:44:34.875 INFO:tasks.workunit.client.1.smithi171.stdout:8/930: write df/d18/da4/fd0 [273813,67032] 0 2022-01-31T19:44:34.877 INFO:tasks.workunit.client.1.smithi171.stdout:7/751: dwrite d1/d1e/d2a/d29/d31/d30/dc4/fe6 [0,4194304] 0 2022-01-31T19:44:34.884 INFO:tasks.workunit.client.1.smithi171.stdout:8/931: link f0 df/d18/d19/d21/d67/f13e 0 2022-01-31T19:44:34.886 INFO:tasks.workunit.client.1.smithi171.stdout:7/752: creat d1/ff6 x:0 0 0 2022-01-31T19:44:34.888 INFO:tasks.workunit.client.1.smithi171.stdout:8/932: mknod df/d2a/dfb/c13f 0 2022-01-31T19:44:34.888 INFO:tasks.workunit.client.1.smithi171.stdout:8/933: chown df/d2a/dfb 1305183879 1 2022-01-31T19:44:34.888 INFO:tasks.workunit.client.1.smithi171.stdout:8/934: creat df/d18/dad/f140 x:0 0 0 2022-01-31T19:44:34.889 INFO:tasks.workunit.client.1.smithi171.stdout:8/935: rmdir df/d18/d79 39 2022-01-31T19:44:34.892 INFO:tasks.workunit.client.1.smithi171.stdout:7/753: dread d1/d1e/d2a/d29/d77/f8c [0,4194304] 0 2022-01-31T19:44:34.894 INFO:tasks.workunit.client.1.smithi171.stdout:7/754: symlink d1/d1e/d2a/d29/lf7 0 2022-01-31T19:44:34.894 INFO:tasks.workunit.client.1.smithi171.stdout:7/755: write d1/d1b/fb6 [461165,73553] 0 2022-01-31T19:44:34.895 INFO:tasks.workunit.client.1.smithi171.stdout:7/756: write d1/d1e/d2a/d58/f98 [5161209,43362] 0 2022-01-31T19:44:34.895 INFO:tasks.workunit.client.1.smithi171.stdout:7/757: write d1/f16 [1983220,5051] 0 2022-01-31T19:44:34.896 INFO:tasks.workunit.client.1.smithi171.stdout:7/758: mkdir d1/d1e/d2a/d58/df8 0 2022-01-31T19:44:34.896 INFO:tasks.workunit.client.1.smithi171.stdout:7/759: unlink d1/d26/f71 0 2022-01-31T19:44:34.898 INFO:tasks.workunit.client.1.smithi171.stdout:7/760: unlink d1/d1b/fb6 0 2022-01-31T19:44:34.898 INFO:tasks.workunit.client.1.smithi171.stdout:7/761: chown d1/d1e/d2a/d29/d31/d30/d3c/d7e/f8b 65 1 2022-01-31T19:44:34.899 INFO:tasks.workunit.client.1.smithi171.stdout:7/762: creat d1/d1e/d2a/d29/ff9 x:0 0 0 2022-01-31T19:44:34.900 INFO:tasks.workunit.client.1.smithi171.stdout:7/763: write d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [468132,48361] 0 2022-01-31T19:44:34.918 INFO:tasks.workunit.client.1.smithi171.stdout:8/936: dwrite df/d18/d19/d21/d67/df3/f121 [4194304,4194304] 0 2022-01-31T19:44:34.919 INFO:tasks.workunit.client.1.smithi171.stdout:8/937: dread - df/dcc/f139 zero size 2022-01-31T19:44:34.920 INFO:tasks.workunit.client.1.smithi171.stdout:7/764: dwrite d1/d26/d57/fde [0,4194304] 0 2022-01-31T19:44:34.920 INFO:tasks.workunit.client.1.smithi171.stdout:8/938: write df/d27/f64 [2837858,58730] 0 2022-01-31T19:44:34.923 INFO:tasks.workunit.client.1.smithi171.stdout:7/765: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/df1/ffa x:0 0 0 2022-01-31T19:44:34.924 INFO:tasks.workunit.client.1.smithi171.stdout:7/766: chown d1/d26/l59 2209 1 2022-01-31T19:44:34.924 INFO:tasks.workunit.client.1.smithi171.stdout:7/767: fdatasync d1/d1e/d2a/d29/fa7 0 2022-01-31T19:44:34.924 INFO:tasks.workunit.client.1.smithi171.stdout:8/939: symlink df/d18/d19/d21/dcd/l141 0 2022-01-31T19:44:34.925 INFO:tasks.workunit.client.1.smithi171.stdout:7/768: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/ffb x:0 0 0 2022-01-31T19:44:34.926 INFO:tasks.workunit.client.1.smithi171.stdout:7/769: rmdir d1/d1e/d2a/d29/d31/d30/d3c 39 2022-01-31T19:44:34.926 INFO:tasks.workunit.client.1.smithi171.stdout:7/770: symlink d1/d26/lfc 0 2022-01-31T19:44:34.934 INFO:tasks.workunit.client.1.smithi171.stdout:8/940: dread f3 [0,4194304] 0 2022-01-31T19:44:34.936 INFO:tasks.workunit.client.1.smithi171.stdout:8/941: rename df/d18/dad/lff to df/d18/d19/d21/dab/d138/l142 0 2022-01-31T19:44:34.951 INFO:tasks.workunit.client.1.smithi171.stdout:8/942: dwrite df/d18/d19/d21/dab/f56 [4194304,4194304] 0 2022-01-31T19:44:34.951 INFO:tasks.workunit.client.1.smithi171.stdout:8/943: creat df/d2a/dfb/f143 x:0 0 0 2022-01-31T19:44:34.960 INFO:tasks.workunit.client.1.smithi171.stdout:8/944: dread df/d1a/f53 [0,4194304] 0 2022-01-31T19:44:34.982 INFO:tasks.workunit.client.1.smithi171.stdout:8/945: dwrite df/d5d/d6c/fbf [0,4194304] 0 2022-01-31T19:44:34.984 INFO:tasks.workunit.client.1.smithi171.stdout:8/946: mknod df/dcc/c144 0 2022-01-31T19:44:35.004 INFO:tasks.workunit.client.1.smithi171.stdout:8/947: dwrite df/d18/d79/f9d [0,4194304] 0 2022-01-31T19:44:35.005 INFO:tasks.workunit.client.1.smithi171.stdout:8/948: creat df/d18/d79/da2/ddd/d125/f145 x:0 0 0 2022-01-31T19:44:35.006 INFO:tasks.workunit.client.1.smithi171.stdout:8/949: link df/d18/da4/f134 df/d18/d79/da2/ddd/d125/f146 0 2022-01-31T19:44:35.011 INFO:tasks.workunit.client.1.smithi171.stdout:4/876: sync 2022-01-31T19:44:35.011 INFO:tasks.workunit.client.1.smithi171.stdout:6/983: sync 2022-01-31T19:44:35.011 INFO:tasks.workunit.client.1.smithi171.stdout:5/911: sync 2022-01-31T19:44:35.012 INFO:tasks.workunit.client.1.smithi171.stdout:5/912: symlink d5/d90/db7/d15/d80/l135 0 2022-01-31T19:44:35.013 INFO:tasks.workunit.client.1.smithi171.stdout:8/950: dread df/d1a/f58 [0,4194304] 0 2022-01-31T19:44:35.014 INFO:tasks.workunit.client.1.smithi171.stdout:4/877: read d2/d3/d5/dc/fd6 [403139,21826] 0 2022-01-31T19:44:35.014 INFO:tasks.workunit.client.1.smithi171.stdout:5/913: unlink d5/db0/fd2 0 2022-01-31T19:44:35.015 INFO:tasks.workunit.client.1.smithi171.stdout:8/951: getdents df/d5d/d6c/d87/df1 0 2022-01-31T19:44:35.018 INFO:tasks.workunit.client.1.smithi171.stdout:4/878: unlink d2/f5c 0 2022-01-31T19:44:35.019 INFO:tasks.workunit.client.1.smithi171.stdout:5/914: truncate d5/d90/db7/fc1 657970 0 2022-01-31T19:44:35.020 INFO:tasks.workunit.client.1.smithi171.stdout:4/879: dread d2/d3/d5/d41/f47 [0,4194304] 0 2022-01-31T19:44:35.020 INFO:tasks.workunit.client.1.smithi171.stdout:4/880: fdatasync d2/d3/d5/d41/dcf/f8c 0 2022-01-31T19:44:35.021 INFO:tasks.workunit.client.1.smithi171.stdout:8/952: creat df/d18/d19/d21/d67/ddb/f147 x:0 0 0 2022-01-31T19:44:35.021 INFO:tasks.workunit.client.1.smithi171.stdout:5/915: dread d5/d90/db7/d15/d80/fdc [0,4194304] 0 2022-01-31T19:44:35.023 INFO:tasks.workunit.client.1.smithi171.stdout:4/881: truncate d2/d3/d5/d41/dcf/d12/d15/dc4/fe2 1050500 0 2022-01-31T19:44:35.024 INFO:tasks.workunit.client.1.smithi171.stdout:8/953: write df/d27/fa3 [3306045,82539] 0 2022-01-31T19:44:35.024 INFO:tasks.workunit.client.1.smithi171.stdout:8/954: creat df/d18/d19/d21/d88/f148 x:0 0 0 2022-01-31T19:44:35.024 INFO:tasks.workunit.client.1.smithi171.stdout:8/955: creat df/d18/d19/d21/d67/f149 x:0 0 0 2022-01-31T19:44:35.025 INFO:tasks.workunit.client.1.smithi171.stdout:4/882: rmdir d2/d3/d5/d70/d7b 39 2022-01-31T19:44:35.028 INFO:tasks.workunit.client.1.smithi171.stdout:5/916: rename d5/d90/db7/d15/d80/d109 to d5/d90/db7/d136 0 2022-01-31T19:44:35.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/984: dwrite dc/f7f [0,4194304] 0 2022-01-31T19:44:35.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/985: creat dc/f14c x:0 0 0 2022-01-31T19:44:35.029 INFO:tasks.workunit.client.1.smithi171.stdout:6/986: creat dc/d19/d30/d3c/d105/f14d x:0 0 0 2022-01-31T19:44:35.030 INFO:tasks.workunit.client.1.smithi171.stdout:6/987: dread - dc/d19/d30/d3c/d80/f83 zero size 2022-01-31T19:44:35.031 INFO:tasks.workunit.client.1.smithi171.stdout:8/956: rename df/dcc to df/d18/d79/d7a/dc0/d14a 0 2022-01-31T19:44:35.032 INFO:tasks.workunit.client.1.smithi171.stdout:8/957: write df/d18/fce [2866582,59402] 0 2022-01-31T19:44:35.032 INFO:tasks.workunit.client.1.smithi171.stdout:8/958: truncate df/d27/d5c/f112 46362 0 2022-01-31T19:44:35.042 INFO:tasks.workunit.client.1.smithi171.stdout:6/988: truncate dc/d19/d29/d123/d110/d142/d86/d119/d3b/d67/dd1/f104 864508 0 2022-01-31T19:44:35.043 INFO:tasks.workunit.client.1.smithi171.stdout:4/883: rename d2/d3/d5/d70/l9c to d2/d3/d5/d91/d77/l126 0 2022-01-31T19:44:35.045 INFO:tasks.workunit.client.1.smithi171.stdout:6/989: stat dc/d19/d29/d123/d110/d142/c1f 0 2022-01-31T19:44:35.046 INFO:tasks.workunit.client.1.smithi171.stdout:6/990: write dc/f7f [3203031,50699] 0 2022-01-31T19:44:35.046 INFO:tasks.workunit.client.1.smithi171.stdout:6/991: readlink dc/d19/d29/d123/d110/d142/d86/d119/d111/ld8 0 2022-01-31T19:44:35.047 INFO:tasks.workunit.client.1.smithi171.stdout:8/959: dwrite df/d18/d19/d46/f8d [0,4194304] 0 2022-01-31T19:44:35.048 INFO:tasks.workunit.client.1.smithi171.stdout:4/884: rename d2/d3/d5/d41/dcf/d12/f60 to d2/d3/d5/d41/dcf/d12/d15/dc4/dfe/f127 0 2022-01-31T19:44:35.048 INFO:tasks.workunit.client.1.smithi171.stdout:8/960: write df/d2a/d30/f35 [55813,29872] 0 2022-01-31T19:44:35.049 INFO:tasks.workunit.client.1.smithi171.stdout:6/992: unlink dc/d1e/d3f/f65 0 2022-01-31T19:44:35.050 INFO:tasks.workunit.client.1.smithi171.stdout:6/993: stat dc/d19/d30/d3c/d105 0 2022-01-31T19:44:35.052 INFO:tasks.workunit.client.1.smithi171.stdout:6/994: dread dc/d1e/f45 [4194304,4194304] 0 2022-01-31T19:44:35.056 INFO:tasks.workunit.client.1.smithi171.stdout:6/995: write f9 [6385850,82222] 0 2022-01-31T19:44:35.056 INFO:tasks.workunit.client.1.smithi171.stdout:6/996: chown f9 6358 1 2022-01-31T19:44:35.056 INFO:tasks.workunit.client.1.smithi171.stdout:6/997: getdents dc/d19/d29/d123/d110/d142/d86/d119/d3b/dd5 0 2022-01-31T19:44:35.057 INFO:tasks.workunit.client.1.smithi171.stdout:8/961: dread df/d2a/d30/f63 [0,4194304] 0 2022-01-31T19:44:35.057 INFO:tasks.workunit.client.1.smithi171.stdout:8/962: stat df/d18/d19/d21/d67 0 2022-01-31T19:44:35.061 INFO:tasks.workunit.client.1.smithi171.stdout:5/917: rename d5/d90/db7/dd/l2a to d5/d90/db7/d15/d23/d58/d9e/dd7/l137 0 2022-01-31T19:44:35.069 INFO:tasks.workunit.client.1.smithi171.stdout:8/963: rename df/d27/d5c/l10a to df/d18/d19/d46/l14b 0 2022-01-31T19:44:35.069 INFO:tasks.workunit.client.1.smithi171.stdout:8/964: creat df/d1a/f14c x:0 0 0 2022-01-31T19:44:35.072 INFO:tasks.workunit.client.1.smithi171.stdout:8/965: mknod df/d2a/dfb/c14d 0 2022-01-31T19:44:35.077 INFO:tasks.workunit.client.1.smithi171.stdout:8/966: symlink df/l14e 0 2022-01-31T19:44:35.078 INFO:tasks.workunit.client.1.smithi171.stdout:1/979: sync 2022-01-31T19:44:35.079 INFO:tasks.workunit.client.1.smithi171.stdout:4/885: dwrite d2/d3/d5/dc/d4b/d56/f11c [0,4194304] 0 2022-01-31T19:44:35.080 INFO:tasks.workunit.client.1.smithi171.stdout:6/998: dwrite dc/d19/d30/d3c/d105/f14d [0,4194304] 0 2022-01-31T19:44:35.080 INFO:tasks.workunit.client.1.smithi171.stdout:1/980: mkdir dd/d23/d74/dcb/d125 0 2022-01-31T19:44:35.084 INFO:tasks.workunit.client.1.smithi171.stdout:6/999: mkdir dc/d19/d29/daf/d14e 0 2022-01-31T19:44:35.085 INFO:tasks.workunit.client.1.smithi171.stdout:4/886: mknod d2/d3/d5/dc/d4b/dbc/c128 0 2022-01-31T19:44:35.087 INFO:tasks.workunit.client.1.smithi171.stdout:4/887: link d2/d3/d5/d41/d6b/f99 d2/d3/d5/d91/de4/df8/f129 0 2022-01-31T19:44:35.087 INFO:tasks.workunit.client.1.smithi171.stdout:4/888: creat d2/d3/d5/d41/dcf/d12/f12a x:0 0 0 2022-01-31T19:44:35.087 INFO:tasks.workunit.client.1.smithi171.stdout:1/981: mkdir dd/df/d84/dea/d35/db0/d126 0 2022-01-31T19:44:35.088 INFO:tasks.workunit.client.1.smithi171.stdout:1/982: chown dd/df/d84/dea/d35/dd6 220853401 1 2022-01-31T19:44:35.092 INFO:tasks.workunit.client.1.smithi171.stdout:0/974: sync 2022-01-31T19:44:35.092 INFO:tasks.workunit.client.1.smithi171.stdout:7/771: sync 2022-01-31T19:44:35.093 INFO:tasks.workunit.client.1.smithi171.stdout:7/772: chown d1/d1e/d2a/d29/d31/d30/dc4 1986056 1 2022-01-31T19:44:35.095 INFO:tasks.workunit.client.1.smithi171.stdout:4/889: mkdir d2/d3/d5/d91/de4/df8/d12b 0 2022-01-31T19:44:35.096 INFO:tasks.workunit.client.1.smithi171.stdout:8/967: dwrite df/d18/d19/d46/f11b [0,4194304] 0 2022-01-31T19:44:35.097 INFO:tasks.workunit.client.1.smithi171.stdout:1/983: rename dd/d23/l106 to dd/d23/d34/d4b/l127 0 2022-01-31T19:44:35.097 INFO:tasks.workunit.client.1.smithi171.stdout:1/984: read - dd/d23/d2e/d5e/d81/d98/d80/dee/ff1 zero size 2022-01-31T19:44:35.098 INFO:tasks.workunit.client.1.smithi171.stdout:1/985: read dd/df/d19/d1b/f28 [430462,107657] 0 2022-01-31T19:44:35.098 INFO:tasks.workunit.client.1.smithi171.stdout:1/986: readlink dd/d23/d2e/d5e/dd8/l121 0 2022-01-31T19:44:35.099 INFO:tasks.workunit.client.1.smithi171.stdout:4/890: mkdir d2/d3/d5/d41/d12c 0 2022-01-31T19:44:35.100 INFO:tasks.workunit.client.1.smithi171.stdout:8/968: getdents df/d18/d79/da2 0 2022-01-31T19:44:35.100 INFO:tasks.workunit.client.1.smithi171.stdout:8/969: stat df/d18/d19/db2 0 2022-01-31T19:44:35.101 INFO:tasks.workunit.client.1.smithi171.stdout:7/773: getdents d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7 0 2022-01-31T19:44:35.102 INFO:tasks.workunit.client.1.smithi171.stdout:8/970: mknod df/d18/d19/d36/d70/c14f 0 2022-01-31T19:44:35.103 INFO:tasks.workunit.client.1.smithi171.stdout:8/971: write df/d18/d19/d21/d67/f149 [158125,82096] 0 2022-01-31T19:44:35.103 INFO:tasks.workunit.client.1.smithi171.stdout:8/972: write df/d18/d19/d21/fb6 [7796,71857] 0 2022-01-31T19:44:35.103 INFO:tasks.workunit.client.1.smithi171.stdout:8/973: chown df/d18/d19/d21/d67/df3/f126 10 1 2022-01-31T19:44:35.103 INFO:tasks.workunit.client.1.smithi171.stdout:8/974: creat df/d18/d19/f150 x:0 0 0 2022-01-31T19:44:35.105 INFO:tasks.workunit.client.1.smithi171.stdout:8/975: creat df/d18/d19/d21/f151 x:0 0 0 2022-01-31T19:44:35.105 INFO:tasks.workunit.client.1.smithi171.stdout:8/976: read - df/d18/dad/f135 zero size 2022-01-31T19:44:35.105 INFO:tasks.workunit.client.1.smithi171.stdout:8/977: chown df/d18/d19/d46/f5e 2954696 1 2022-01-31T19:44:35.107 INFO:tasks.workunit.client.1.smithi171.stdout:0/975: dread d9/db/fc4 [0,4194304] 0 2022-01-31T19:44:35.107 INFO:tasks.workunit.client.1.smithi171.stdout:0/976: fsync d9/f1d 0 2022-01-31T19:44:35.107 INFO:tasks.workunit.client.1.smithi171.stdout:0/977: dread - d9/d24/d4c/d7d/f106 zero size 2022-01-31T19:44:35.157 INFO:tasks.workunit.client.1.smithi171.stdout:4/891: dread d2/d3/d5/d41/dcf/dc7/fdb [0,4194304] 0 2022-01-31T19:44:35.157 INFO:tasks.workunit.client.1.smithi171.stdout:4/892: write d2/d3/d5/dc/f19 [4738652,68349] 0 2022-01-31T19:44:35.159 INFO:tasks.workunit.client.1.smithi171.stdout:4/893: creat d2/d3/d5/dc/d4b/dbc/d10a/f12d x:0 0 0 2022-01-31T19:44:35.160 INFO:tasks.workunit.client.1.smithi171.stdout:4/894: symlink d2/d3/d5/d91/d118/l12e 0 2022-01-31T19:44:35.161 INFO:tasks.workunit.client.1.smithi171.stdout:7/774: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d79/f8d [0,4194304] 0 2022-01-31T19:44:35.162 INFO:tasks.workunit.client.1.smithi171.stdout:7/775: readlink d1/d1e/d2a/d29/d31/d30/d3c/l9c 0 2022-01-31T19:44:35.162 INFO:tasks.workunit.client.1.smithi171.stdout:4/895: symlink d2/d3/d5/d91/de4/df8/d12b/l12f 0 2022-01-31T19:44:35.164 INFO:tasks.workunit.client.1.smithi171.stdout:7/776: rmdir d1/d26/d47/d8f 39 2022-01-31T19:44:35.167 INFO:tasks.workunit.client.1.smithi171.stdout:0/978: dwrite d9/db/dc/d3f/db9/f8e [0,4194304] 0 2022-01-31T19:44:35.167 INFO:tasks.workunit.client.1.smithi171.stdout:7/777: unlink d1/d1e/d2a/f83 0 2022-01-31T19:44:35.167 INFO:tasks.workunit.client.1.smithi171.stdout:7/778: stat d1/d1e/d2a/d29/d31/d30/c5d 0 2022-01-31T19:44:35.167 INFO:tasks.workunit.client.1.smithi171.stdout:7/779: readlink d1/d1e/d2a/d29/d31/d30/dc4/ld8 0 2022-01-31T19:44:35.168 INFO:tasks.workunit.client.1.smithi171.stdout:7/780: fdatasync d1/d1b/f4c 0 2022-01-31T19:44:35.168 INFO:tasks.workunit.client.1.smithi171.stdout:0/979: rename d9/d24/d4c/d35/d67/df4/l108 to d9/d24/l160 0 2022-01-31T19:44:35.170 INFO:tasks.workunit.client.1.smithi171.stdout:0/980: rename d9/db/dc/d3f/db9/d7a/cd5 to d9/db/dc2/c161 0 2022-01-31T19:44:35.171 INFO:tasks.workunit.client.1.smithi171.stdout:8/978: dwrite df/d18/d19/d21/d67/fcb [0,4194304] 0 2022-01-31T19:44:35.172 INFO:tasks.workunit.client.1.smithi171.stdout:7/781: getdents d1/d1e/d2a/d29 0 2022-01-31T19:44:35.175 INFO:tasks.workunit.client.1.smithi171.stdout:8/979: mknod df/d18/dad/d116/c152 0 2022-01-31T19:44:35.177 INFO:tasks.workunit.client.1.smithi171.stdout:7/782: rename d1/d1b/f4c to d1/d1e/d2a/ffd 0 2022-01-31T19:44:35.177 INFO:tasks.workunit.client.1.smithi171.stdout:7/783: write d1/d1e/d2a/d58/fb8 [84587,6651] 0 2022-01-31T19:44:35.178 INFO:tasks.workunit.client.1.smithi171.stdout:7/784: chown d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/fd2 2 1 2022-01-31T19:44:35.178 INFO:tasks.workunit.client.1.smithi171.stdout:7/785: fsync d1/d1e/d2a/d29/d31/d30/d3c/fda 0 2022-01-31T19:44:35.179 INFO:tasks.workunit.client.1.smithi171.stdout:4/896: dwrite d2/d3/d5/d91/d77/da5/fcb [0,4194304] 0 2022-01-31T19:44:35.179 INFO:tasks.workunit.client.1.smithi171.stdout:4/897: write d2/d3/d5/fc1 [508682,49886] 0 2022-01-31T19:44:35.180 INFO:tasks.workunit.client.1.smithi171.stdout:4/898: stat d2/d3/d5/dc/f4f 0 2022-01-31T19:44:35.180 INFO:tasks.workunit.client.1.smithi171.stdout:0/981: link d9/db/dc/d3f/ldd d9/d24/l162 0 2022-01-31T19:44:35.182 INFO:tasks.workunit.client.1.smithi171.stdout:4/899: write d2/d3/d5/dc/f95 [672920,91589] 0 2022-01-31T19:44:35.184 INFO:tasks.workunit.client.1.smithi171.stdout:8/980: link df/d18/d19/d21/d67/c6a df/d18/d79/da2/d107/c153 0 2022-01-31T19:44:35.187 INFO:tasks.workunit.client.1.smithi171.stdout:7/786: dread d1/d1e/f78 [0,4194304] 0 2022-01-31T19:44:35.188 INFO:tasks.workunit.client.1.smithi171.stdout:0/982: getdents d9/db/d59/d71/dbe 0 2022-01-31T19:44:35.189 INFO:tasks.workunit.client.1.smithi171.stdout:0/983: mkdir d9/db/dc/d3f/d6f/d163 0 2022-01-31T19:44:35.194 INFO:tasks.workunit.client.1.smithi171.stdout:0/984: dread d9/db/dc2/f138 [0,4194304] 0 2022-01-31T19:44:35.194 INFO:tasks.workunit.client.1.smithi171.stdout:0/985: chown d9/db/dc/d3f/d6f/ffa 97198094 1 2022-01-31T19:44:35.195 INFO:tasks.workunit.client.1.smithi171.stdout:0/986: chown d9/d24/d4c/d101/c148 7878602 1 2022-01-31T19:44:35.195 INFO:tasks.workunit.client.1.smithi171.stdout:0/987: write d9/d24/d4c/fa9 [1115006,88256] 0 2022-01-31T19:44:35.213 INFO:tasks.workunit.client.1.smithi171.stdout:4/900: dwrite d2/d3/d5/dc/d4b/dbc/fd0 [0,4194304] 0 2022-01-31T19:44:35.214 INFO:tasks.workunit.client.1.smithi171.stdout:7/787: dwrite d1/f5 [0,4194304] 0 2022-01-31T19:44:35.215 INFO:tasks.workunit.client.1.smithi171.stdout:4/901: dread d2/d3/d5/dc/f26 [4194304,4194304] 0 2022-01-31T19:44:35.219 INFO:tasks.workunit.client.1.smithi171.stdout:8/981: dwrite df/d18/f55 [0,4194304] 0 2022-01-31T19:44:35.219 INFO:tasks.workunit.client.1.smithi171.stdout:7/788: mknod d1/d26/cfe 0 2022-01-31T19:44:35.219 INFO:tasks.workunit.client.1.smithi171.stdout:4/902: getdents d2/d3/de1 0 2022-01-31T19:44:35.220 INFO:tasks.workunit.client.1.smithi171.stdout:7/789: read - d1/d1e/d2a/d29/d31/d30/d3c/d7e/f8b zero size 2022-01-31T19:44:35.220 INFO:tasks.workunit.client.1.smithi171.stdout:4/903: chown d2/d3/d5/d41/dcf/d12/db6/fb7 1281232 1 2022-01-31T19:44:35.220 INFO:tasks.workunit.client.1.smithi171.stdout:7/790: creat d1/d1e/d2a/fff x:0 0 0 2022-01-31T19:44:35.223 INFO:tasks.workunit.client.1.smithi171.stdout:8/982: truncate df/d18/f41 1466067 0 2022-01-31T19:44:35.226 INFO:tasks.workunit.client.1.smithi171.stdout:4/904: link d2/d3/d5/d41/dcf/f48 d2/d3/d5/d41/dcf/d12/d15/d57/dad/f130 0 2022-01-31T19:44:35.228 INFO:tasks.workunit.client.1.smithi171.stdout:7/791: link d1/d1e/d2a/d29/d31/f38 d1/d26/d47/d8f/f100 0 2022-01-31T19:44:35.228 INFO:tasks.workunit.client.1.smithi171.stdout:0/988: dwrite d9/db/dc2/f63 [0,4194304] 0 2022-01-31T19:44:35.229 INFO:tasks.workunit.client.1.smithi171.stdout:8/983: dread df/d18/d19/d21/d67/f13e [4194304,4194304] 0 2022-01-31T19:44:35.229 INFO:tasks.workunit.client.1.smithi171.stdout:0/989: chown d9/d24/d4c/d101/d132/f146 50399 1 2022-01-31T19:44:35.229 INFO:tasks.workunit.client.1.smithi171.stdout:0/990: read d9/f16 [6786080,639] 0 2022-01-31T19:44:35.232 INFO:tasks.workunit.client.1.smithi171.stdout:4/905: creat d2/d3/d5/d91/de4/df8/d12b/f131 x:0 0 0 2022-01-31T19:44:35.232 INFO:tasks.workunit.client.1.smithi171.stdout:4/906: chown d2/f14 3471 1 2022-01-31T19:44:35.235 INFO:tasks.workunit.client.1.smithi171.stdout:7/792: mknod d1/d1e/d2a/d29/d77/de4/c101 0 2022-01-31T19:44:35.235 INFO:tasks.workunit.client.1.smithi171.stdout:7/793: chown d1/d1e/d2a/d29/d31/d30/d3c/d79/l92 452733598 1 2022-01-31T19:44:35.235 INFO:tasks.workunit.client.1.smithi171.stdout:7/794: creat d1/d1e/d2a/d29/d77/dab/f102 x:0 0 0 2022-01-31T19:44:35.236 INFO:tasks.workunit.client.1.smithi171.stdout:7/795: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/df1/f103 x:0 0 0 2022-01-31T19:44:35.237 INFO:tasks.workunit.client.1.smithi171.stdout:8/984: truncate df/d18/d19/d46/f5e 432666 0 2022-01-31T19:44:35.237 INFO:tasks.workunit.client.1.smithi171.stdout:8/985: dread - df/d27/d90/fbb zero size 2022-01-31T19:44:35.237 INFO:tasks.workunit.client.1.smithi171.stdout:8/986: dread - df/d2a/f132 zero size 2022-01-31T19:44:35.237 INFO:tasks.workunit.client.1.smithi171.stdout:4/907: mkdir d2/d3/d5/dc/d4b/dbc/d10a/d132 0 2022-01-31T19:44:35.238 INFO:tasks.workunit.client.1.smithi171.stdout:4/908: fsync d2/d3/d5/fd2 0 2022-01-31T19:44:35.241 INFO:tasks.workunit.client.1.smithi171.stdout:8/987: unlink df/d27/d90/ff2 0 2022-01-31T19:44:35.242 INFO:tasks.workunit.client.1.smithi171.stdout:4/909: symlink d2/d3/d5/d41/dcf/d12/d15/dc4/l133 0 2022-01-31T19:44:35.242 INFO:tasks.workunit.client.1.smithi171.stdout:4/910: creat d2/d3/d5/d91/de4/f134 x:0 0 0 2022-01-31T19:44:35.242 INFO:tasks.workunit.client.1.smithi171.stdout:4/911: chown d2/d3/d5/dc/c119 1686 1 2022-01-31T19:44:35.243 INFO:tasks.workunit.client.1.smithi171.stdout:7/796: mkdir d1/d1e/d2a/d29/d31/d30/d6b/d104 0 2022-01-31T19:44:35.243 INFO:tasks.workunit.client.1.smithi171.stdout:7/797: chown d1/d1b/f4b 7753 1 2022-01-31T19:44:35.246 INFO:tasks.workunit.client.1.smithi171.stdout:8/988: mknod df/d18/d19/d21/dcd/d12c/c154 0 2022-01-31T19:44:35.246 INFO:tasks.workunit.client.1.smithi171.stdout:8/989: stat df/f17 0 2022-01-31T19:44:35.246 INFO:tasks.workunit.client.1.smithi171.stdout:8/990: readlink df/d1a/l45 0 2022-01-31T19:44:35.247 INFO:tasks.workunit.client.1.smithi171.stdout:4/912: symlink d2/d3/d5/d70/d7c/l135 0 2022-01-31T19:44:35.248 INFO:tasks.workunit.client.1.smithi171.stdout:4/913: write d2/d3/d5/d70/dee/ffb [105406,100491] 0 2022-01-31T19:44:35.248 INFO:tasks.workunit.client.1.smithi171.stdout:7/798: write d1/d1e/d2a/ffd [2589950,113354] 0 2022-01-31T19:44:35.249 INFO:tasks.workunit.client.1.smithi171.stdout:8/991: mknod df/d18/d19/d21/dcd/d12c/d136/c155 0 2022-01-31T19:44:35.249 INFO:tasks.workunit.client.1.smithi171.stdout:8/992: dread - df/d95/f10e zero size 2022-01-31T19:44:35.250 INFO:tasks.workunit.client.1.smithi171.stdout:8/993: write df/d18/d79/f9d [4880898,79172] 0 2022-01-31T19:44:35.250 INFO:tasks.workunit.client.1.smithi171.stdout:8/994: creat df/d18/da4/f156 x:0 0 0 2022-01-31T19:44:35.250 INFO:tasks.workunit.client.1.smithi171.stdout:4/914: dread d2/f6d [0,4194304] 0 2022-01-31T19:44:35.251 INFO:tasks.workunit.client.1.smithi171.stdout:7/799: write d1/d1e/d2a/d29/d31/f6a [5309518,17478] 0 2022-01-31T19:44:35.251 INFO:tasks.workunit.client.1.smithi171.stdout:8/995: mknod df/d5d/c157 0 2022-01-31T19:44:35.253 INFO:tasks.workunit.client.1.smithi171.stdout:7/800: symlink d1/d1e/d2a/d29/d31/d30/d6b/d104/l105 0 2022-01-31T19:44:35.254 INFO:tasks.workunit.client.1.smithi171.stdout:7/801: chown d1/d1e/d2a 97558 1 2022-01-31T19:44:35.254 INFO:tasks.workunit.client.1.smithi171.stdout:0/991: dwrite d9/db/dc/d3f/f6d [0,4194304] 0 2022-01-31T19:44:35.255 INFO:tasks.workunit.client.1.smithi171.stdout:0/992: chown d9/db/dc/d3f/c13e 3 1 2022-01-31T19:44:35.256 INFO:tasks.workunit.client.1.smithi171.stdout:7/802: write d1/d1e/d2a/d29/d31/d30/d6b/f94 [536753,12572] 0 2022-01-31T19:44:35.256 INFO:tasks.workunit.client.1.smithi171.stdout:7/803: readlink d1/d1e/d2a/lb7 0 2022-01-31T19:44:35.256 INFO:tasks.workunit.client.1.smithi171.stdout:7/804: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/f106 x:0 0 0 2022-01-31T19:44:35.263 INFO:tasks.workunit.client.1.smithi171.stdout:0/993: dread d9/d24/d92/f9a [0,4194304] 0 2022-01-31T19:44:35.263 INFO:tasks.workunit.client.1.smithi171.stdout:0/994: fsync d9/db/d59/d71/fa4 0 2022-01-31T19:44:35.278 INFO:tasks.workunit.client.1.smithi171.stdout:0/995: dread d9/db/dc2/f119 [0,4194304] 0 2022-01-31T19:44:35.283 INFO:tasks.workunit.client.1.smithi171.stdout:4/915: dwrite d2/d3/d5/d91/d118/f125 [0,4194304] 0 2022-01-31T19:44:35.283 INFO:tasks.workunit.client.1.smithi171.stdout:4/916: read d2/d3/d5/fc1 [235574,4703] 0 2022-01-31T19:44:35.284 INFO:tasks.workunit.client.1.smithi171.stdout:7/805: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f28 [0,4194304] 0 2022-01-31T19:44:35.284 INFO:tasks.workunit.client.1.smithi171.stdout:4/917: creat d2/d3/d5/d41/d6b/f136 x:0 0 0 2022-01-31T19:44:35.284 INFO:tasks.workunit.client.1.smithi171.stdout:8/996: dwrite df/d18/d19/d21/d67/f149 [0,4194304] 0 2022-01-31T19:44:35.290 INFO:tasks.workunit.client.1.smithi171.stdout:4/918: rmdir d2/d3/d5/dc/d4b/dbc/d10a/d132 0 2022-01-31T19:44:35.291 INFO:tasks.workunit.client.1.smithi171.stdout:4/919: chown d2/l8a 96103419 1 2022-01-31T19:44:35.291 INFO:tasks.workunit.client.1.smithi171.stdout:4/920: readlink d2/d3/d5/d41/dcf/d12/d15/d57/l88 0 2022-01-31T19:44:35.291 INFO:tasks.workunit.client.1.smithi171.stdout:4/921: fsync d2/d3/d5/d91/d77/f11a 0 2022-01-31T19:44:35.292 INFO:tasks.workunit.client.1.smithi171.stdout:4/922: write d2/d3/d5/d41/dcf/d12/d15/d104/f121 [628130,28973] 0 2022-01-31T19:44:35.292 INFO:tasks.workunit.client.1.smithi171.stdout:4/923: chown d2/d3/f44 1093176416 1 2022-01-31T19:44:35.293 INFO:tasks.workunit.client.1.smithi171.stdout:7/806: symlink d1/d1e/d2a/d29/d31/d30/d3c/db9/df1/l107 0 2022-01-31T19:44:35.294 INFO:tasks.workunit.client.1.smithi171.stdout:0/996: dwrite d9/f13 [8388608,4194304] 0 2022-01-31T19:44:35.296 INFO:tasks.workunit.client.1.smithi171.stdout:5/918: sync 2022-01-31T19:44:35.297 INFO:tasks.workunit.client.1.smithi171.stdout:5/919: truncate d5/d90/db7/dd/d66/f82 4785338 0 2022-01-31T19:44:35.299 INFO:tasks.workunit.client.1.smithi171.stdout:8/997: creat df/d18/d79/f158 x:0 0 0 2022-01-31T19:44:35.302 INFO:tasks.workunit.client.1.smithi171.stdout:0/997: mknod d9/db/dc/d3f/db9/d10f/c164 0 2022-01-31T19:44:35.302 INFO:tasks.workunit.client.1.smithi171.stdout:0/998: dread - d9/db/dc/d3f/db9/d7a/f9e zero size 2022-01-31T19:44:35.303 INFO:tasks.workunit.client.1.smithi171.stdout:8/998: creat df/d1a/dae/de0/f159 x:0 0 0 2022-01-31T19:44:35.304 INFO:tasks.workunit.client.1.smithi171.stdout:7/807: rmdir d1/d1e/d2a/d29/d77/dab/dd5 39 2022-01-31T19:44:35.306 INFO:tasks.workunit.client.1.smithi171.stdout:0/999: creat d9/db/d59/d71/dbe/f165 x:0 0 0 2022-01-31T19:44:35.309 INFO:tasks.workunit.client.1.smithi171.stdout:7/808: link d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/fef d1/d1e/d2a/d29/d31/d30/d3c/d5f/f108 0 2022-01-31T19:44:35.309 INFO:tasks.workunit.client.1.smithi171.stdout:7/809: fdatasync d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd0 0 2022-01-31T19:44:35.309 INFO:tasks.workunit.client.1.smithi171.stdout:7/810: chown d1/d1e/d2a/d29/fea 403 1 2022-01-31T19:44:35.317 INFO:tasks.workunit.client.1.smithi171.stdout:8/999: dread df/d18/f55 [0,4194304] 0 2022-01-31T19:44:35.319 INFO:tasks.workunit.client.1.smithi171.stdout:4/924: dwrite d2/d3/f25 [0,4194304] 0 2022-01-31T19:44:35.320 INFO:tasks.workunit.client.1.smithi171.stdout:4/925: truncate d2/d3/d5/d41/dcf/f27 360884 0 2022-01-31T19:44:35.320 INFO:tasks.workunit.client.1.smithi171.stdout:4/926: chown d2/d3/d5/d3b/c10e 38526 1 2022-01-31T19:44:35.321 INFO:tasks.workunit.client.1.smithi171.stdout:1/987: sync 2022-01-31T19:44:35.322 INFO:tasks.workunit.client.1.smithi171.stdout:4/927: creat d2/d3/d5/d70/d7b/f137 x:0 0 0 2022-01-31T19:44:35.325 INFO:tasks.workunit.client.1.smithi171.stdout:5/920: dwrite d5/d90/ff3 [0,4194304] 0 2022-01-31T19:44:35.328 INFO:tasks.workunit.client.1.smithi171.stdout:1/988: dread f9 [0,4194304] 0 2022-01-31T19:44:35.330 INFO:tasks.workunit.client.1.smithi171.stdout:5/921: creat d5/d90/db7/d15/d23/d6b/dae/f138 x:0 0 0 2022-01-31T19:44:35.330 INFO:tasks.workunit.client.1.smithi171.stdout:5/922: chown d5/f8 117 1 2022-01-31T19:44:35.331 INFO:tasks.workunit.client.1.smithi171.stdout:1/989: mknod dd/d23/d2e/d5e/dd8/c128 0 2022-01-31T19:44:35.331 INFO:tasks.workunit.client.1.smithi171.stdout:1/990: readlink dd/d23/d2e/d5e/l117 0 2022-01-31T19:44:35.332 INFO:tasks.workunit.client.1.smithi171.stdout:1/991: chown dd/d23/d2e/d5e/d81/fa8 61488455 1 2022-01-31T19:44:35.332 INFO:tasks.workunit.client.1.smithi171.stdout:1/992: chown dd/df/d19/d1b/f72 392020 1 2022-01-31T19:44:35.334 INFO:tasks.workunit.client.1.smithi171.stdout:5/923: rename d5/d90/db7/d15/d5d/d6a/l85 to d5/d90/dd1/dfc/l139 0 2022-01-31T19:44:35.334 INFO:tasks.workunit.client.1.smithi171.stdout:5/924: readlink d5/d90/db7/l7c 0 2022-01-31T19:44:35.335 INFO:tasks.workunit.client.1.smithi171.stdout:5/925: fsync d5/d90/db7/d15/d23/d58/fe4 0 2022-01-31T19:44:35.335 INFO:tasks.workunit.client.1.smithi171.stdout:5/926: readlink d5/d90/db7/d15/d23/d3d/l11f 0 2022-01-31T19:44:35.335 INFO:tasks.workunit.client.1.smithi171.stdout:5/927: stat d5/d59/da0/f11e 0 2022-01-31T19:44:35.335 INFO:tasks.workunit.client.1.smithi171.stdout:5/928: stat d5/d90/db7/d15/d23/f104 0 2022-01-31T19:44:35.336 INFO:tasks.workunit.client.1.smithi171.stdout:5/929: link d5/d90/db7/dd/d66/fde d5/f13a 0 2022-01-31T19:44:35.336 INFO:tasks.workunit.client.1.smithi171.stdout:5/930: chown d5/d90/dd3 538 1 2022-01-31T19:44:35.338 INFO:tasks.workunit.client.1.smithi171.stdout:5/931: link d5/d90/db7/d15/d23/c3a d5/d90/d12d/c13b 0 2022-01-31T19:44:35.340 INFO:tasks.workunit.client.1.smithi171.stdout:7/811: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd0 [0,4194304] 0 2022-01-31T19:44:35.343 INFO:tasks.workunit.client.1.smithi171.stdout:7/812: read d1/d1e/d2a/d29/d31/ff [1443427,111516] 0 2022-01-31T19:44:35.344 INFO:tasks.workunit.client.1.smithi171.stdout:4/928: dwrite d2/d3/d5/fd2 [0,4194304] 0 2022-01-31T19:44:35.345 INFO:tasks.workunit.client.1.smithi171.stdout:4/929: fdatasync d2/d3/f25 0 2022-01-31T19:44:35.345 INFO:tasks.workunit.client.1.smithi171.stdout:7/813: link d1/d1b/fc d1/d1e/d2a/d29/d77/f109 0 2022-01-31T19:44:35.346 INFO:tasks.workunit.client.1.smithi171.stdout:7/814: readlink d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/lf3 0 2022-01-31T19:44:35.346 INFO:tasks.workunit.client.1.smithi171.stdout:7/815: write d1/d1e/d2a/d58/f98 [4457291,72488] 0 2022-01-31T19:44:35.350 INFO:tasks.workunit.client.1.smithi171.stdout:7/816: mknod d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/c10a 0 2022-01-31T19:44:35.350 INFO:tasks.workunit.client.1.smithi171.stdout:7/817: chown d1/d1e/d2a/d29/l33 0 1 2022-01-31T19:44:35.352 INFO:tasks.workunit.client.1.smithi171.stdout:7/818: mkdir d1/d1e/d2a/d58/d10b 0 2022-01-31T19:44:35.354 INFO:tasks.workunit.client.1.smithi171.stdout:7/819: mknod d1/d1e/d2a/d29/d31/d30/d3c/dc0/c10c 0 2022-01-31T19:44:35.366 INFO:tasks.workunit.client.1.smithi171.stdout:7/820: dread d1/d1e/d2a/d29/d31/d30/d3c/d7e/fe1 [0,4194304] 0 2022-01-31T19:44:35.371 INFO:tasks.workunit.client.1.smithi171.stdout:5/932: dwrite d5/d90/db7/d15/d80/d8d/de2/f116 [0,4194304] 0 2022-01-31T19:44:35.371 INFO:tasks.workunit.client.1.smithi171.stdout:5/933: fsync d5/d90/db7/d15/d23/f9a 0 2022-01-31T19:44:35.372 INFO:tasks.workunit.client.1.smithi171.stdout:4/930: dwrite d2/d3/de1/dd4/ff7 [0,4194304] 0 2022-01-31T19:44:35.377 INFO:tasks.workunit.client.1.smithi171.stdout:5/934: unlink d5/d90/db7/dd/d66/l11a 0 2022-01-31T19:44:35.391 INFO:tasks.workunit.client.1.smithi171.stdout:7/821: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d7e/fe1 [0,4194304] 0 2022-01-31T19:44:35.396 INFO:tasks.workunit.client.1.smithi171.stdout:7/822: link d1/d1e/d2a/l4a d1/d1e/d2a/d29/d31/d30/dc4/l10d 0 2022-01-31T19:44:35.397 INFO:tasks.workunit.client.1.smithi171.stdout:7/823: creat d1/d1e/d2a/d29/d31/d30/d6b/f10e x:0 0 0 2022-01-31T19:44:35.397 INFO:tasks.workunit.client.1.smithi171.stdout:7/824: fsync d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f21 0 2022-01-31T19:44:35.407 INFO:tasks.workunit.client.1.smithi171.stdout:4/931: dwrite d2/d3/d5/d91/f76 [0,4194304] 0 2022-01-31T19:44:35.411 INFO:tasks.workunit.client.1.smithi171.stdout:4/932: symlink d2/d3/d5/d3b/l138 0 2022-01-31T19:44:35.411 INFO:tasks.workunit.client.1.smithi171.stdout:5/935: dwrite d5/d90/db7/d15/d23/d3d/f51 [0,4194304] 0 2022-01-31T19:44:35.411 INFO:tasks.workunit.client.1.smithi171.stdout:7/825: dread d1/d1e/d2a/d29/d31/d30/d3c/d5f/f90 [0,4194304] 0 2022-01-31T19:44:35.412 INFO:tasks.workunit.client.1.smithi171.stdout:4/933: unlink d2/d3/d5/dc/d4b/le6 0 2022-01-31T19:44:35.416 INFO:tasks.workunit.client.1.smithi171.stdout:5/936: mknod d5/d90/c13c 0 2022-01-31T19:44:35.416 INFO:tasks.workunit.client.1.smithi171.stdout:7/826: stat d1/d1e/d2a/d58/ldb 0 2022-01-31T19:44:35.416 INFO:tasks.workunit.client.1.smithi171.stdout:7/827: chown d1/d1e/d2a/d29/fea 6337 1 2022-01-31T19:44:35.417 INFO:tasks.workunit.client.1.smithi171.stdout:5/937: rename d5/d90/db7/d15/d80/l135 to d5/d90/db7/d15/d80/d8d/l13d 0 2022-01-31T19:44:35.419 INFO:tasks.workunit.client.1.smithi171.stdout:7/828: rename d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/fc8 to d1/d1e/d2a/d29/d31/d30/d3c/d7e/f10f 0 2022-01-31T19:44:35.419 INFO:tasks.workunit.client.1.smithi171.stdout:7/829: readlink d1/d1e/d2a/l84 0 2022-01-31T19:44:35.421 INFO:tasks.workunit.client.1.smithi171.stdout:5/938: creat d5/d90/db7/d15/d23/d58/d9e/f13e x:0 0 0 2022-01-31T19:44:35.422 INFO:tasks.workunit.client.1.smithi171.stdout:5/939: mknod d5/d90/db7/d15/d23/d67/c13f 0 2022-01-31T19:44:35.427 INFO:tasks.workunit.client.1.smithi171.stdout:7/830: dread d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f21 [0,4194304] 0 2022-01-31T19:44:35.438 INFO:tasks.workunit.client.1.smithi171.stdout:7/831: dread d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [0,4194304] 0 2022-01-31T19:44:35.439 INFO:tasks.workunit.client.1.smithi171.stdout:7/832: mknod d1/d1e/d2a/d29/d31/d30/d3c/db9/c110 0 2022-01-31T19:44:35.440 INFO:tasks.workunit.client.1.smithi171.stdout:4/934: dwrite d2/f90 [4194304,4194304] 0 2022-01-31T19:44:35.440 INFO:tasks.workunit.client.1.smithi171.stdout:4/935: truncate d2/d3/d5/d91/d77/f101 596424 0 2022-01-31T19:44:35.441 INFO:tasks.workunit.client.1.smithi171.stdout:5/940: dwrite d5/d90/fa8 [0,4194304] 0 2022-01-31T19:44:35.442 INFO:tasks.workunit.client.1.smithi171.stdout:4/936: write d2/d3/d5/d41/dcf/d12/d15/d104/f121 [11417,82971] 0 2022-01-31T19:44:35.442 INFO:tasks.workunit.client.1.smithi171.stdout:4/937: readlink d2/d3/d5/d41/dcf/l106 0 2022-01-31T19:44:35.442 INFO:tasks.workunit.client.1.smithi171.stdout:4/938: write d2/d3/d5/d91/d77/da5/f11f [1010827,26650] 0 2022-01-31T19:44:35.442 INFO:tasks.workunit.client.1.smithi171.stdout:7/833: symlink d1/d1e/d2a/d29/d31/l111 0 2022-01-31T19:44:35.443 INFO:tasks.workunit.client.1.smithi171.stdout:4/939: getdents d2/d3/d5/d41/dcf/d12/d15/dc4/dfe/d11e 0 2022-01-31T19:44:35.443 INFO:tasks.workunit.client.1.smithi171.stdout:7/834: chown d1/d26/d47/d8f 1563600224 1 2022-01-31T19:44:35.443 INFO:tasks.workunit.client.1.smithi171.stdout:5/941: mknod d5/d90/dd1/c140 0 2022-01-31T19:44:35.444 INFO:tasks.workunit.client.1.smithi171.stdout:5/942: mknod d5/d90/dd1/dfc/c141 0 2022-01-31T19:44:35.454 INFO:tasks.workunit.client.1.smithi171.stdout:7/835: dread d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f28 [0,4194304] 0 2022-01-31T19:44:35.456 INFO:tasks.workunit.client.1.smithi171.stdout:7/836: rmdir d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9 39 2022-01-31T19:44:35.458 INFO:tasks.workunit.client.1.smithi171.stdout:7/837: link d1/d1e/d2a/d29/d31/c60 d1/d1e/d2a/c112 0 2022-01-31T19:44:35.459 INFO:tasks.workunit.client.1.smithi171.stdout:7/838: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/f113 x:0 0 0 2022-01-31T19:44:35.461 INFO:tasks.workunit.client.1.smithi171.stdout:7/839: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/d114 0 2022-01-31T19:44:35.462 INFO:tasks.workunit.client.1.smithi171.stdout:7/840: symlink d1/d1e/d2a/d29/d31/d30/d3c/db9/l115 0 2022-01-31T19:44:35.465 INFO:tasks.workunit.client.1.smithi171.stdout:7/841: rename d1/d1e/d2a/d29/d31/d30/d3c/l49 to d1/d1e/d2a/d58/df8/l116 0 2022-01-31T19:44:35.465 INFO:tasks.workunit.client.1.smithi171.stdout:7/842: chown d1/d1e/d2a/d29/d31/d30/d3c/c80 1116 1 2022-01-31T19:44:35.465 INFO:tasks.workunit.client.1.smithi171.stdout:7/843: fdatasync d1/d1e/d2a/d29/d31/d30/f88 0 2022-01-31T19:44:35.465 INFO:tasks.workunit.client.1.smithi171.stdout:4/940: dwrite d2/d3/d5/dc/fd6 [0,4194304] 0 2022-01-31T19:44:35.466 INFO:tasks.workunit.client.1.smithi171.stdout:7/844: dread d1/d26/d47/f4f [0,4194304] 0 2022-01-31T19:44:35.476 INFO:tasks.workunit.client.1.smithi171.stdout:4/941: dread d2/d3/d5/d41/d6b/f99 [0,4194304] 0 2022-01-31T19:44:35.476 INFO:tasks.workunit.client.1.smithi171.stdout:4/942: dread - d2/d3/d5/d41/d6b/fb2 zero size 2022-01-31T19:44:35.476 INFO:tasks.workunit.client.1.smithi171.stdout:4/943: chown d2/d3/d5/dc/d4b/dbc/c128 255 1 2022-01-31T19:44:35.478 INFO:tasks.workunit.client.1.smithi171.stdout:5/943: dwrite d5/d90/db7/d15/d80/d8d/de2/f12b [0,4194304] 0 2022-01-31T19:44:35.485 INFO:tasks.workunit.client.1.smithi171.stdout:5/944: truncate d5/d90/db7/d15/d23/f25 239832 0 2022-01-31T19:44:35.495 INFO:tasks.workunit.client.1.smithi171.stdout:7/845: dwrite d1/d1e/d2a/d58/fb8 [0,4194304] 0 2022-01-31T19:44:35.498 INFO:tasks.workunit.client.1.smithi171.stdout:4/944: dwrite d2/f6d [0,4194304] 0 2022-01-31T19:44:35.498 INFO:tasks.workunit.client.1.smithi171.stdout:4/945: fdatasync d2/d3/d5/d41/dcf/dc7/fcd 0 2022-01-31T19:44:35.501 INFO:tasks.workunit.client.1.smithi171.stdout:4/946: mkdir d2/d3/d5/d41/dcf/d12/d15/d57/d139 0 2022-01-31T19:44:35.501 INFO:tasks.workunit.client.1.smithi171.stdout:4/947: dread - d2/d3/d5/d70/f108 zero size 2022-01-31T19:44:35.505 INFO:tasks.workunit.client.1.smithi171.stdout:4/948: rmdir d2/d3/d5/d91/de4 39 2022-01-31T19:44:35.506 INFO:tasks.workunit.client.1.smithi171.stdout:1/993: sync 2022-01-31T19:44:35.507 INFO:tasks.workunit.client.1.smithi171.stdout:4/949: truncate d2/d3/d5/d41/dcf/d12/d15/dc4/fe2 928227 0 2022-01-31T19:44:35.507 INFO:tasks.workunit.client.1.smithi171.stdout:4/950: fsync d2/d3/d5/dc/fdf 0 2022-01-31T19:44:35.508 INFO:tasks.workunit.client.1.smithi171.stdout:1/994: mkdir dd/df/d84/dea/d64/da7/d129 0 2022-01-31T19:44:35.508 INFO:tasks.workunit.client.1.smithi171.stdout:1/995: stat dd/d23/d34/l62 0 2022-01-31T19:44:35.510 INFO:tasks.workunit.client.1.smithi171.stdout:1/996: mkdir dd/da2/d12a 0 2022-01-31T19:44:35.510 INFO:tasks.workunit.client.1.smithi171.stdout:1/997: write dd/df/d84/dea/d64/fc1 [4962148,76012] 0 2022-01-31T19:44:35.514 INFO:tasks.workunit.client.1.smithi171.stdout:4/951: write d2/d3/d5/d91/f76 [2732160,42007] 0 2022-01-31T19:44:35.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/952: mkdir d2/d3/d5/d70/d7c/dd5/d13a 0 2022-01-31T19:44:35.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/953: chown d2/f14 0 1 2022-01-31T19:44:35.516 INFO:tasks.workunit.client.1.smithi171.stdout:4/954: dread - d2/d3/d5/d41/f116 zero size 2022-01-31T19:44:35.519 INFO:tasks.workunit.client.1.smithi171.stdout:4/955: dread d2/d3/f73 [0,4194304] 0 2022-01-31T19:44:35.521 INFO:tasks.workunit.client.1.smithi171.stdout:4/956: mkdir d2/d3/de1/dd4/d13b 0 2022-01-31T19:44:35.522 INFO:tasks.workunit.client.1.smithi171.stdout:4/957: dread - d2/d3/d5/d70/d7c/fc3 zero size 2022-01-31T19:44:35.523 INFO:tasks.workunit.client.1.smithi171.stdout:4/958: mknod d2/d3/de1/dd4/d13b/c13c 0 2022-01-31T19:44:35.523 INFO:tasks.workunit.client.1.smithi171.stdout:4/959: chown d2/d3/d5/d91/f9d 890 1 2022-01-31T19:44:35.524 INFO:tasks.workunit.client.1.smithi171.stdout:5/945: dwrite d5/d90/db7/d15/d23/d58/f91 [0,4194304] 0 2022-01-31T19:44:35.525 INFO:tasks.workunit.client.1.smithi171.stdout:7/846: dwrite d1/d1e/d2a/d29/f66 [0,4194304] 0 2022-01-31T19:44:35.529 INFO:tasks.workunit.client.1.smithi171.stdout:7/847: read d1/d1e/d2a/d29/f66 [857354,45985] 0 2022-01-31T19:44:35.532 INFO:tasks.workunit.client.1.smithi171.stdout:7/848: mknod d1/d1e/d2a/d29/d31/d30/d3c/d7e/c117 0 2022-01-31T19:44:35.532 INFO:tasks.workunit.client.1.smithi171.stdout:7/849: write d1/ff6 [99881,19990] 0 2022-01-31T19:44:35.533 INFO:tasks.workunit.client.1.smithi171.stdout:1/998: dwrite dd/df/d84/dea/d64/da7/fc0 [0,4194304] 0 2022-01-31T19:44:35.555 INFO:tasks.workunit.client.1.smithi171.stdout:4/960: dwrite d2/d3/d5/dc/fdf [0,4194304] 0 2022-01-31T19:44:35.556 INFO:tasks.workunit.client.1.smithi171.stdout:5/946: dwrite d5/d90/db7/d15/d23/d3d/f51 [0,4194304] 0 2022-01-31T19:44:35.557 INFO:tasks.workunit.client.1.smithi171.stdout:5/947: chown d5/d90/db7/d15/d23/d6b/dae/lb1 13 1 2022-01-31T19:44:35.557 INFO:tasks.workunit.client.1.smithi171.stdout:5/948: dread - d5/d90/db7/d15/d5d/d6a/fec zero size 2022-01-31T19:44:35.561 INFO:tasks.workunit.client.1.smithi171.stdout:5/949: write d5/d59/f122 [3346753,68602] 0 2022-01-31T19:44:35.561 INFO:tasks.workunit.client.1.smithi171.stdout:5/950: chown d5/d59/l99 108889319 1 2022-01-31T19:44:35.561 INFO:tasks.workunit.client.1.smithi171.stdout:5/951: write d5/db0/fbc [959277,46377] 0 2022-01-31T19:44:35.561 INFO:tasks.workunit.client.1.smithi171.stdout:4/961: mknod d2/d3/d5/d70/dee/c13d 0 2022-01-31T19:44:35.562 INFO:tasks.workunit.client.1.smithi171.stdout:4/962: stat d2/d3/d5/dc/f95 0 2022-01-31T19:44:35.562 INFO:tasks.workunit.client.1.smithi171.stdout:1/999: dwrite dd/d23/d74/dcb/ddb/f103 [0,4194304] 0 2022-01-31T19:44:35.563 INFO:tasks.workunit.client.1.smithi171.stdout:4/963: mkdir d2/d3/d5/d41/dcf/d12/d15/d57/d139/d13e 0 2022-01-31T19:44:35.570 INFO:tasks.workunit.client.1.smithi171.stdout:5/952: dread d5/d90/db7/d15/d5d/d6f/fdd [0,4194304] 0 2022-01-31T19:44:35.573 INFO:tasks.workunit.client.1.smithi171.stdout:5/953: dread d5/d90/db7/d15/f42 [0,4194304] 0 2022-01-31T19:44:35.586 INFO:tasks.workunit.client.1.smithi171.stdout:5/954: dread d5/d90/fe8 [0,4194304] 0 2022-01-31T19:44:35.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/955: truncate d5/d90/db7/d136/f12e 318539 0 2022-01-31T19:44:35.588 INFO:tasks.workunit.client.1.smithi171.stdout:5/956: truncate d5/d90/f111 535753 0 2022-01-31T19:44:35.591 INFO:tasks.workunit.client.1.smithi171.stdout:5/957: truncate d5/d90/db7/d15/f33 4373659 0 2022-01-31T19:44:35.594 INFO:tasks.workunit.client.1.smithi171.stdout:5/958: read d5/d90/db7/dd/d66/fe1 [750265,77893] 0 2022-01-31T19:44:35.596 INFO:tasks.workunit.client.1.smithi171.stdout:5/959: dread d5/d90/db7/dd/d66/fe1 [0,4194304] 0 2022-01-31T19:44:35.611 INFO:tasks.workunit.client.1.smithi171.stdout:4/964: dwrite d2/f13 [0,4194304] 0 2022-01-31T19:44:35.615 INFO:tasks.workunit.client.1.smithi171.stdout:4/965: write d2/d3/d5/d41/dcf/d12/d15/d104/f121 [107743,6685] 0 2022-01-31T19:44:35.616 INFO:tasks.workunit.client.1.smithi171.stdout:5/960: dwrite d5/d90/f111 [0,4194304] 0 2022-01-31T19:44:35.616 INFO:tasks.workunit.client.1.smithi171.stdout:4/966: truncate d2/d3/d5/dc/d4b/dbc/fd0 2828073 0 2022-01-31T19:44:35.644 INFO:tasks.workunit.client.1.smithi171.stdout:4/967: dwrite d2/d3/d5/dc/d4b/f6e [0,4194304] 0 2022-01-31T19:44:35.645 INFO:tasks.workunit.client.1.smithi171.stdout:5/961: dwrite d5/d90/db7/d15/d80/d8d/f100 [0,4194304] 0 2022-01-31T19:44:35.647 INFO:tasks.workunit.client.1.smithi171.stdout:4/968: rmdir d2/d3/de1/dd4/d13b 39 2022-01-31T19:44:35.647 INFO:tasks.workunit.client.1.smithi171.stdout:4/969: chown d2/d3/d5/d41/dcf/d12/c2c 371412530 1 2022-01-31T19:44:35.649 INFO:tasks.workunit.client.1.smithi171.stdout:5/962: symlink d5/db0/l142 0 2022-01-31T19:44:35.656 INFO:tasks.workunit.client.1.smithi171.stdout:7/850: sync 2022-01-31T19:44:35.657 INFO:tasks.workunit.client.1.smithi171.stdout:7/851: mknod d1/d26/d47/c118 0 2022-01-31T19:44:35.658 INFO:tasks.workunit.client.1.smithi171.stdout:7/852: chown d1/f16 149551 1 2022-01-31T19:44:35.658 INFO:tasks.workunit.client.1.smithi171.stdout:7/853: creat d1/d26/d47/d8f/f119 x:0 0 0 2022-01-31T19:44:35.661 INFO:tasks.workunit.client.1.smithi171.stdout:7/854: rename d1/d1e/d2a/d29/d31/d30/f5a to d1/d26/d47/f11a 0 2022-01-31T19:44:35.661 INFO:tasks.workunit.client.1.smithi171.stdout:7/855: write d1/d1e/d2a/d29/d31/d30/d6b/f86 [5868823,14742] 0 2022-01-31T19:44:35.661 INFO:tasks.workunit.client.1.smithi171.stdout:7/856: chown d1/d1e/d2a/d29/d31/d30/d3c/f56 7 1 2022-01-31T19:44:35.662 INFO:tasks.workunit.client.1.smithi171.stdout:7/857: truncate d1/d1e/d2a/f3b 1406216 0 2022-01-31T19:44:35.662 INFO:tasks.workunit.client.1.smithi171.stdout:7/858: write d1/d1e/d2a/d29/fea [889989,47452] 0 2022-01-31T19:44:35.663 INFO:tasks.workunit.client.1.smithi171.stdout:7/859: fdatasync d1/d1e/d2a/d29/d31/d30/d3c/d5f/f108 0 2022-01-31T19:44:35.663 INFO:tasks.workunit.client.1.smithi171.stdout:5/963: read d5/f37 [2906029,66158] 0 2022-01-31T19:44:35.663 INFO:tasks.workunit.client.1.smithi171.stdout:5/964: dread - d5/d90/db7/d15/d5d/f81 zero size 2022-01-31T19:44:35.664 INFO:tasks.workunit.client.1.smithi171.stdout:7/860: mknod d1/d1e/d2a/d29/d77/c11b 0 2022-01-31T19:44:35.664 INFO:tasks.workunit.client.1.smithi171.stdout:5/965: mknod d5/d90/db7/d15/d23/d58/d9e/c143 0 2022-01-31T19:44:35.665 INFO:tasks.workunit.client.1.smithi171.stdout:5/966: fdatasync d5/d90/db7/d15/d5d/d6f/f9c 0 2022-01-31T19:44:35.665 INFO:tasks.workunit.client.1.smithi171.stdout:5/967: creat d5/d90/db7/d15/d80/d8d/f144 x:0 0 0 2022-01-31T19:44:35.666 INFO:tasks.workunit.client.1.smithi171.stdout:7/861: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/d11c 0 2022-01-31T19:44:35.666 INFO:tasks.workunit.client.1.smithi171.stdout:7/862: dread - d1/d26/d47/d8f/f119 zero size 2022-01-31T19:44:35.666 INFO:tasks.workunit.client.1.smithi171.stdout:7/863: truncate d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/fd2 1977080 0 2022-01-31T19:44:35.679 INFO:tasks.workunit.client.1.smithi171.stdout:4/970: dwrite d2/d3/d5/d41/dcf/d12/d15/d57/dad/f130 [0,4194304] 0 2022-01-31T19:44:35.679 INFO:tasks.workunit.client.1.smithi171.stdout:4/971: stat d2/d3/d5/d70/fa7 0 2022-01-31T19:44:35.683 INFO:tasks.workunit.client.1.smithi171.stdout:4/972: rename d2/d3/de1/dd4/d13b to d2/d3/d5/d41/dcf/d12/d15/dc4/d13f 0 2022-01-31T19:44:35.683 INFO:tasks.workunit.client.1.smithi171.stdout:4/973: fdatasync d2/d3/d5/d41/dcf/d12/f12a 0 2022-01-31T19:44:35.684 INFO:tasks.workunit.client.1.smithi171.stdout:4/974: write d2/d3/d5/d70/d7b/fb5 [4489029,43829] 0 2022-01-31T19:44:35.684 INFO:tasks.workunit.client.1.smithi171.stdout:7/864: rmdir d1/d26 39 2022-01-31T19:44:35.689 INFO:tasks.workunit.client.1.smithi171.stdout:4/975: rename d2/d3/d5/d3b/c10e to d2/d3/d5/dc/c140 0 2022-01-31T19:44:35.689 INFO:tasks.workunit.client.1.smithi171.stdout:4/976: write d2/d3/d5/d70/d7c/f96 [870860,84641] 0 2022-01-31T19:44:35.690 INFO:tasks.workunit.client.1.smithi171.stdout:4/977: creat d2/d3/d5/d91/da6/f141 x:0 0 0 2022-01-31T19:44:35.693 INFO:tasks.workunit.client.1.smithi171.stdout:7/865: rename d1/d1e/d2a/d29/d31/d30/f88 to d1/d1e/d2a/d29/d31/d30/d3c/dc0/f11d 0 2022-01-31T19:44:35.693 INFO:tasks.workunit.client.1.smithi171.stdout:7/866: truncate d1/d26/d47/d8f/fc5 117849 0 2022-01-31T19:44:35.694 INFO:tasks.workunit.client.1.smithi171.stdout:4/978: mkdir d2/d3/d5/d41/dcf/d122/d142 0 2022-01-31T19:44:35.694 INFO:tasks.workunit.client.1.smithi171.stdout:4/979: write d2/d3/d5/d91/f84 [58596,52462] 0 2022-01-31T19:44:35.695 INFO:tasks.workunit.client.1.smithi171.stdout:5/968: dwrite d5/d90/db7/d15/d23/f25 [0,4194304] 0 2022-01-31T19:44:35.695 INFO:tasks.workunit.client.1.smithi171.stdout:7/867: creat d1/d1e/d2a/d29/d31/f11e x:0 0 0 2022-01-31T19:44:35.696 INFO:tasks.workunit.client.1.smithi171.stdout:7/868: read - d1/d1e/d2a/d29/ff9 zero size 2022-01-31T19:44:35.697 INFO:tasks.workunit.client.1.smithi171.stdout:4/980: link d2/d3/d5/d91/d77/l126 d2/d3/d5/d3b/l143 0 2022-01-31T19:44:35.699 INFO:tasks.workunit.client.1.smithi171.stdout:7/869: unlink d1/d1e/d2a/d29/fa7 0 2022-01-31T19:44:35.700 INFO:tasks.workunit.client.1.smithi171.stdout:7/870: dread - d1/d1e/d2a/d29/fed zero size 2022-01-31T19:44:35.703 INFO:tasks.workunit.client.1.smithi171.stdout:7/871: creat d1/d1e/d2a/d29/d77/de4/f11f x:0 0 0 2022-01-31T19:44:35.703 INFO:tasks.workunit.client.1.smithi171.stdout:7/872: read - d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/f106 zero size 2022-01-31T19:44:35.706 INFO:tasks.workunit.client.1.smithi171.stdout:7/873: symlink d1/d1e/d2a/d29/d31/d30/l120 0 2022-01-31T19:44:35.708 INFO:tasks.workunit.client.1.smithi171.stdout:7/874: link d1/d1e/d2a/d29/d77/dab/dd5/le3 d1/d26/d47/l121 0 2022-01-31T19:44:35.709 INFO:tasks.workunit.client.1.smithi171.stdout:5/969: dread d5/d90/db7/d15/d23/d58/f11d [0,4194304] 0 2022-01-31T19:44:35.712 INFO:tasks.workunit.client.1.smithi171.stdout:5/970: creat d5/d90/db7/d15/d23/d67/dd8/f145 x:0 0 0 2022-01-31T19:44:35.732 INFO:tasks.workunit.client.1.smithi171.stdout:7/875: dwrite d1/d1e/f85 [0,4194304] 0 2022-01-31T19:44:35.735 INFO:tasks.workunit.client.1.smithi171.stdout:4/981: dwrite d2/d3/d5/dc/fdf [0,4194304] 0 2022-01-31T19:44:35.737 INFO:tasks.workunit.client.1.smithi171.stdout:4/982: symlink d2/d3/d5/d41/dcf/d12/d15/d104/l144 0 2022-01-31T19:44:35.737 INFO:tasks.workunit.client.1.smithi171.stdout:4/983: readlink d2/d3/d5/d41/dcf/d12/d15/lb1 0 2022-01-31T19:44:35.739 INFO:tasks.workunit.client.1.smithi171.stdout:4/984: dread d2/d3/d5/d41/d6b/fbd [0,4194304] 0 2022-01-31T19:44:35.740 INFO:tasks.workunit.client.1.smithi171.stdout:4/985: symlink d2/d3/d5/d91/de4/df8/d12b/l145 0 2022-01-31T19:44:35.740 INFO:tasks.workunit.client.1.smithi171.stdout:4/986: dread - d2/d3/d5/d91/da6/f141 zero size 2022-01-31T19:44:35.740 INFO:tasks.workunit.client.1.smithi171.stdout:4/987: fdatasync d2/d3/d5/d41/dcf/f8c 0 2022-01-31T19:44:35.741 INFO:tasks.workunit.client.1.smithi171.stdout:4/988: link d2/d3/d5/d41/dcf/lc2 d2/d7f/l146 0 2022-01-31T19:44:35.742 INFO:tasks.workunit.client.1.smithi171.stdout:4/989: symlink d2/d3/d5/d70/d7c/l147 0 2022-01-31T19:44:35.743 INFO:tasks.workunit.client.1.smithi171.stdout:4/990: write d2/d3/d5/d41/dcf/d12/d15/d57/dad/f130 [774702,21451] 0 2022-01-31T19:44:35.743 INFO:tasks.workunit.client.1.smithi171.stdout:4/991: truncate d2/d3/d5/d91/f76 1256912 0 2022-01-31T19:44:35.744 INFO:tasks.workunit.client.1.smithi171.stdout:4/992: chown d2/c8 92713088 1 2022-01-31T19:44:35.745 INFO:tasks.workunit.client.1.smithi171.stdout:4/993: symlink d2/d3/d5/l148 0 2022-01-31T19:44:35.746 INFO:tasks.workunit.client.1.smithi171.stdout:4/994: creat d2/d3/d5/d41/dcf/d12/d15/dc4/dfe/d11e/f149 x:0 0 0 2022-01-31T19:44:35.747 INFO:tasks.workunit.client.1.smithi171.stdout:4/995: symlink d2/d3/d5/d41/dcf/d12/d15/dc4/dfe/d11e/l14a 0 2022-01-31T19:44:35.748 INFO:tasks.workunit.client.1.smithi171.stdout:4/996: write d2/d3/d5/dc/d4b/d56/fb9 [21616,93627] 0 2022-01-31T19:44:35.748 INFO:tasks.workunit.client.1.smithi171.stdout:4/997: truncate d2/d3/d5/dc/d4b/ff5 910840 0 2022-01-31T19:44:35.767 INFO:tasks.workunit.client.1.smithi171.stdout:4/998: dwrite d2/d3/d5/d41/dcf/f62 [0,4194304] 0 2022-01-31T19:44:35.770 INFO:tasks.workunit.client.1.smithi171.stdout:4/999: link d2/fa2 d2/d3/d5/d70/f14b 0 2022-01-31T19:44:36.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:35 smithi171 conmon[35325]: cluster 2022-01-31T19:44:34.438307+0000 mgr.smithi171.asyxnc (mgr.24387) 2022-01-31T19:44:36.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:35 smithi171 conmon[35325]: 38 : cluster [DBG] pgmap v26: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 153 MiB/s rd, 328 MiB/s wr, 544 op/s 2022-01-31T19:44:36.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:35 smithi167 conmon[32206]: cluster 2022-01-31T19:44:34.438307+0000 mgr.smithi171.asyxnc (mgr.24387 2022-01-31T19:44:36.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:35 smithi167 conmon[32206]: ) 38 : cluster [DBG] pgmap v26: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 153 MiB/s rd, 328 MiB/s wr, 544 op/s 2022-01-31T19:44:37.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:37 smithi167 conmon[32206]: cluster 2022-01-31T19:44:36.438679+0000 mgr.smithi171.asyxnc 2022-01-31T19:44:37.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:37 smithi167 conmon[32206]: (mgr.24387) 39 : cluster [DBG] pgmap v27: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 101 MiB/s rd, 227 MiB/s wr, 381 op/s 2022-01-31T19:44:37.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:37 smithi167 conmon[32206]: audit 2022-01-31T19:44:36.993039+0000 mon.smithi167 (mon.0 2022-01-31T19:44:37.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:37 smithi167 conmon[32206]: ) 687 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:37 smithi171 conmon[35325]: cluster 2022-01-31T19:44:36.438679+0000 mgr.smithi171.asyxnc (mgr 2022-01-31T19:44:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:37 smithi171 conmon[35325]: .24387) 39 : cluster [DBG] pgmap v27: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 101 MiB/s rd, 227 MiB/s wr, 381 op/s 2022-01-31T19:44:38.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:37 smithi171 conmon[35325]: audit 2022-01-31T19:44:36.993039+0000 mon.smithi167 (mon.0) 687 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:38.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:37 smithi171 conmon[35325]: 2022-01-31T19:44:38.215 INFO:tasks.workunit.client.1.smithi171.stdout:7/876: sync 2022-01-31T19:44:38.216 INFO:tasks.workunit.client.1.smithi171.stdout:5/971: sync 2022-01-31T19:44:38.296 INFO:tasks.workunit.client.1.smithi171.stdout:7/877: creat d1/d1e/f122 x:0 0 0 2022-01-31T19:44:38.299 INFO:tasks.workunit.client.1.smithi171.stdout:5/972: rename d5/d90/db7/d15/d23/d58/d9e to d5/d90/db7/d146 0 2022-01-31T19:44:38.300 INFO:tasks.workunit.client.1.smithi171.stdout:7/878: mkdir d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/d123 0 2022-01-31T19:44:38.314 INFO:tasks.workunit.client.1.smithi171.stdout:7/879: dread d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f18 [0,4194304] 0 2022-01-31T19:44:38.314 INFO:tasks.workunit.client.1.smithi171.stdout:7/880: truncate d1/d1e/d2a/d29/d77/dab/fce 954711 0 2022-01-31T19:44:38.315 INFO:tasks.workunit.client.1.smithi171.stdout:7/881: symlink d1/l124 0 2022-01-31T19:44:38.316 INFO:tasks.workunit.client.1.smithi171.stdout:7/882: fdatasync d1/f5 0 2022-01-31T19:44:38.316 INFO:tasks.workunit.client.1.smithi171.stdout:5/973: dwrite d5/d90/db7/d15/d23/d67/f5c [0,4194304] 0 2022-01-31T19:44:38.317 INFO:tasks.workunit.client.1.smithi171.stdout:7/883: symlink d1/d1e/d2a/d58/d10b/l125 0 2022-01-31T19:44:38.326 INFO:tasks.workunit.client.1.smithi171.stdout:7/884: dread d1/d26/d47/d8f/fa0 [0,4194304] 0 2022-01-31T19:44:38.326 INFO:tasks.workunit.client.1.smithi171.stdout:7/885: truncate d1/d26/d47/f11a 1076495 0 2022-01-31T19:44:38.326 INFO:tasks.workunit.client.1.smithi171.stdout:7/886: fsync d1/d26/d47/f4f 0 2022-01-31T19:44:38.328 INFO:tasks.workunit.client.1.smithi171.stdout:5/974: dread d5/f4e [4194304,4194304] 0 2022-01-31T19:44:38.328 INFO:tasks.workunit.client.1.smithi171.stdout:5/975: fdatasync d5/d90/db7/dd/d66/f9b 0 2022-01-31T19:44:38.330 INFO:tasks.workunit.client.1.smithi171.stdout:5/976: mknod d5/d90/d12d/c147 0 2022-01-31T19:44:38.332 INFO:tasks.workunit.client.1.smithi171.stdout:5/977: symlink d5/d90/db7/d15/d80/d8d/l148 0 2022-01-31T19:44:38.341 INFO:tasks.workunit.client.1.smithi171.stdout:5/978: dread d5/d90/db7/d15/d80/d8d/de2/f12b [0,4194304] 0 2022-01-31T19:44:38.343 INFO:tasks.workunit.client.1.smithi171.stdout:5/979: symlink d5/d90/db7/d15/d5d/d6f/l149 0 2022-01-31T19:44:38.345 INFO:tasks.workunit.client.1.smithi171.stdout:7/887: dwrite d1/d26/d47/d8f/fa0 [0,4194304] 0 2022-01-31T19:44:38.347 INFO:tasks.workunit.client.1.smithi171.stdout:7/888: creat d1/d1e/d2a/d29/f126 x:0 0 0 2022-01-31T19:44:38.348 INFO:tasks.workunit.client.1.smithi171.stdout:7/889: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/d123/f127 x:0 0 0 2022-01-31T19:44:38.350 INFO:tasks.workunit.client.1.smithi171.stdout:7/890: truncate d1/d1e/d2a/d29/d31/d30/dc4/fe6 162406 0 2022-01-31T19:44:38.350 INFO:tasks.workunit.client.1.smithi171.stdout:7/891: read d1/d1e/d2a/d29/d31/d30/d3c/f45 [41612,94059] 0 2022-01-31T19:44:38.359 INFO:tasks.workunit.client.1.smithi171.stdout:5/980: dwrite d5/d90/db7/d15/d80/d8d/f144 [0,4194304] 0 2022-01-31T19:44:38.360 INFO:tasks.workunit.client.1.smithi171.stdout:5/981: write d5/d90/db7/dd/d66/f9b [1331162,60686] 0 2022-01-31T19:44:38.360 INFO:tasks.workunit.client.1.smithi171.stdout:5/982: readlink d5/ldf 0 2022-01-31T19:44:38.365 INFO:tasks.workunit.client.1.smithi171.stdout:5/983: mkdir d5/ded/d14a 0 2022-01-31T19:44:38.366 INFO:tasks.workunit.client.1.smithi171.stdout:5/984: creat d5/d90/db7/d15/d124/f14b x:0 0 0 2022-01-31T19:44:38.366 INFO:tasks.workunit.client.1.smithi171.stdout:5/985: stat d5/d59/da0/f12f 0 2022-01-31T19:44:38.372 INFO:tasks.workunit.client.1.smithi171.stdout:5/986: dread d5/d90/db7/d15/d23/f41 [0,4194304] 0 2022-01-31T19:44:38.374 INFO:tasks.workunit.client.1.smithi171.stdout:5/987: mkdir d5/d90/db7/deb/d14c 0 2022-01-31T19:44:38.375 INFO:tasks.workunit.client.1.smithi171.stdout:5/988: symlink d5/d90/db7/d15/d80/d8d/l14d 0 2022-01-31T19:44:38.376 INFO:tasks.workunit.client.1.smithi171.stdout:7/892: dwrite d1/d1e/d2a/d29/d31/d30/d3c/d7e/dd1/fdf [0,4194304] 0 2022-01-31T19:44:38.376 INFO:tasks.workunit.client.1.smithi171.stdout:7/893: stat d1/fd9 0 2022-01-31T19:44:38.376 INFO:tasks.workunit.client.1.smithi171.stdout:7/894: dread - d1/d1e/d2a/d29/d31/f11e zero size 2022-01-31T19:44:38.377 INFO:tasks.workunit.client.1.smithi171.stdout:5/989: mkdir d5/d90/dd1/dfc/d14e 0 2022-01-31T19:44:38.377 INFO:tasks.workunit.client.1.smithi171.stdout:5/990: readlink d5/d90/dd3/le0 0 2022-01-31T19:44:38.377 INFO:tasks.workunit.client.1.smithi171.stdout:5/991: stat d5/d59/da0/f11e 0 2022-01-31T19:44:38.378 INFO:tasks.workunit.client.1.smithi171.stdout:7/895: creat d1/d26/f128 x:0 0 0 2022-01-31T19:44:38.380 INFO:tasks.workunit.client.1.smithi171.stdout:5/992: mkdir d5/d90/db7/d15/d80/d8d/de2/d14f 0 2022-01-31T19:44:38.381 INFO:tasks.workunit.client.1.smithi171.stdout:5/993: mkdir d5/d90/db7/d15/d80/d8d/de2/d14f/d150 0 2022-01-31T19:44:38.382 INFO:tasks.workunit.client.1.smithi171.stdout:5/994: stat d5/d90/db7/d15/d80/d8d/f144 0 2022-01-31T19:44:38.382 INFO:tasks.workunit.client.1.smithi171.stdout:5/995: truncate d5/d90/db7/dd/f19 1473431 0 2022-01-31T19:44:38.382 INFO:tasks.workunit.client.1.smithi171.stdout:5/996: rmdir d5/d90/dd1/d10a 39 2022-01-31T19:44:38.385 INFO:tasks.workunit.client.1.smithi171.stdout:5/997: rmdir d5/d59 39 2022-01-31T19:44:38.401 INFO:tasks.workunit.client.1.smithi171.stdout:7/896: dwrite d1/d1e/d2a/d58/fb4 [0,4194304] 0 2022-01-31T19:44:38.402 INFO:tasks.workunit.client.1.smithi171.stdout:7/897: write d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd6 [451414,67047] 0 2022-01-31T19:44:38.402 INFO:tasks.workunit.client.1.smithi171.stdout:5/998: dwrite d5/d90/db7/d15/d5d/d6f/fdd [0,4194304] 0 2022-01-31T19:44:38.402 INFO:tasks.workunit.client.1.smithi171.stdout:5/999: dread - d5/d90/db7/d15/d23/d3d/fa4 zero size 2022-01-31T19:44:38.403 INFO:tasks.workunit.client.1.smithi171.stdout:7/898: unlink d1/l124 0 2022-01-31T19:44:38.404 INFO:tasks.workunit.client.1.smithi171.stdout:7/899: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/f129 x:0 0 0 2022-01-31T19:44:38.416 INFO:tasks.workunit.client.1.smithi171.stdout:7/900: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/ffb [0,4194304] 0 2022-01-31T19:44:38.417 INFO:tasks.workunit.client.1.smithi171.stdout:7/901: creat d1/d1e/d2a/d29/f12a x:0 0 0 2022-01-31T19:44:38.417 INFO:tasks.workunit.client.1.smithi171.stdout:7/902: truncate d1/fd9 618256 0 2022-01-31T19:44:38.417 INFO:tasks.workunit.client.1.smithi171.stdout:7/903: chown d1/d1b/l48 7296 1 2022-01-31T19:44:38.419 INFO:tasks.workunit.client.1.smithi171.stdout:7/904: symlink d1/d1e/d2a/d29/d31/d30/d6b/d104/l12b 0 2022-01-31T19:44:38.445 INFO:tasks.workunit.client.1.smithi171.stdout:7/905: dwrite d1/d1e/d2a/d29/f126 [0,4194304] 0 2022-01-31T19:44:38.447 INFO:tasks.workunit.client.1.smithi171.stdout:7/906: symlink d1/d1e/d2a/d58/d10b/l12c 0 2022-01-31T19:44:38.448 INFO:tasks.workunit.client.1.smithi171.stdout:7/907: creat d1/d1e/d2a/d58/df8/f12d x:0 0 0 2022-01-31T19:44:38.450 INFO:tasks.workunit.client.1.smithi171.stdout:7/908: mknod d1/d1b/c12e 0 2022-01-31T19:44:38.455 INFO:tasks.workunit.client.1.smithi171.stdout:7/909: dread d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd6 [0,4194304] 0 2022-01-31T19:44:38.456 INFO:tasks.workunit.client.1.smithi171.stdout:7/910: write d1/d1e/d2a/d29/d31/d30/d3c/d5f/f63 [1565745,121036] 0 2022-01-31T19:44:38.462 INFO:tasks.workunit.client.1.smithi171.stdout:7/911: dread d1/d1e/d2a/d58/f98 [0,4194304] 0 2022-01-31T19:44:38.462 INFO:tasks.workunit.client.1.smithi171.stdout:7/912: write d1/d1e/d2a/d29/d31/d30/d3c/fbb [1201660,79262] 0 2022-01-31T19:44:38.464 INFO:tasks.workunit.client.1.smithi171.stdout:7/913: mknod d1/d1e/d2a/d29/d31/c12f 0 2022-01-31T19:44:38.464 INFO:tasks.workunit.client.1.smithi171.stdout:7/914: truncate d1/d1e/d2a/d29/d31/d30/d3c/db9/dca/dcd/fd0 806177 0 2022-01-31T19:44:38.465 INFO:tasks.workunit.client.1.smithi171.stdout:7/915: stat d1/d1e/d2a/d29/d31/d30/d6b/fbf 0 2022-01-31T19:44:38.467 INFO:tasks.workunit.client.1.smithi171.stdout:7/916: rename d1/d1b/fc to d1/d1e/d2a/d29/d31/d30/d6b/f130 0 2022-01-31T19:44:38.467 INFO:tasks.workunit.client.1.smithi171.stdout:7/917: readlink d1/d1e/d2a/d29/d31/d30/l120 0 2022-01-31T19:44:38.468 INFO:tasks.workunit.client.1.smithi171.stdout:7/918: truncate d1/d1e/d2a/d29/d31/d30/d3c/db9/f129 586917 0 2022-01-31T19:44:38.468 INFO:tasks.workunit.client.1.smithi171.stdout:7/919: fsync d1/d1e/d2a/d29/d31/ff 0 2022-01-31T19:44:38.475 INFO:tasks.workunit.client.1.smithi171.stdout:7/920: dread d1/d26/f50 [0,4194304] 0 2022-01-31T19:44:38.475 INFO:tasks.workunit.client.1.smithi171.stdout:7/921: dread - d1/d1e/d2a/d29/ff9 zero size 2022-01-31T19:44:38.487 INFO:tasks.workunit.client.1.smithi171.stdout:7/922: dwrite d1/d1e/d2a/d29/fb2 [0,4194304] 0 2022-01-31T19:44:38.489 INFO:tasks.workunit.client.1.smithi171.stdout:7/923: dread d1/d1e/d2a/d29/d31/d30/dc4/fe6 [0,4194304] 0 2022-01-31T19:44:38.490 INFO:tasks.workunit.client.1.smithi171.stdout:7/924: write d1/d1e/d2a/d58/fb1 [1353224,45966] 0 2022-01-31T19:44:38.509 INFO:tasks.workunit.client.1.smithi171.stdout:7/925: dwrite d1/d1e/d2a/d29/d31/d30/d6b/fbf [0,4194304] 0 2022-01-31T19:44:38.515 INFO:tasks.workunit.client.1.smithi171.stdout:7/926: rename d1/d1e/d2a/d29/d31/d30/d3c/db9/dca to d1/d1e/d2a/d29/d31/d30/d3c/db9/d131 0 2022-01-31T19:44:38.515 INFO:tasks.workunit.client.1.smithi171.stdout:7/927: chown d1/d26/d47/cbd 20082392 1 2022-01-31T19:44:38.516 INFO:tasks.workunit.client.1.smithi171.stdout:7/928: write d1/d1e/d2a/d29/d31/d30/d6b/fbf [3380040,23604] 0 2022-01-31T19:44:38.544 INFO:tasks.workunit.client.1.smithi171.stdout:7/929: dwrite d1/d1e/d2a/d29/d77/dab/f102 [0,4194304] 0 2022-01-31T19:44:38.546 INFO:tasks.workunit.client.1.smithi171.stdout:7/930: truncate d1/d1e/d2a/d29/d31/d30/f4d 477546 0 2022-01-31T19:44:38.547 INFO:tasks.workunit.client.1.smithi171.stdout:7/931: creat d1/d1e/d2a/d29/d31/d30/d3c/f132 x:0 0 0 2022-01-31T19:44:38.548 INFO:tasks.workunit.client.1.smithi171.stdout:7/932: creat d1/d1e/d2a/d29/d31/d30/d3c/f133 x:0 0 0 2022-01-31T19:44:38.549 INFO:tasks.workunit.client.1.smithi171.stdout:7/933: getdents d1/d1b 0 2022-01-31T19:44:38.550 INFO:tasks.workunit.client.1.smithi171.stdout:7/934: dread - d1/d1e/d2a/d29/d31/d30/d3c/d7e/f8b zero size 2022-01-31T19:44:38.550 INFO:tasks.workunit.client.1.smithi171.stdout:7/935: chown d1/d1e/d2a/d58/fb4 403039 1 2022-01-31T19:44:38.551 INFO:tasks.workunit.client.1.smithi171.stdout:7/936: read d1/d1e/d2a/f9b [560515,106131] 0 2022-01-31T19:44:38.551 INFO:tasks.workunit.client.1.smithi171.stdout:7/937: chown d1/d1e/d2a/d29/d31/d30/d3c/l72 30 1 2022-01-31T19:44:38.552 INFO:tasks.workunit.client.1.smithi171.stdout:7/938: creat d1/d1e/d2a/d29/d31/f134 x:0 0 0 2022-01-31T19:44:38.552 INFO:tasks.workunit.client.1.smithi171.stdout:7/939: dread - d1/d1e/d2a/d29/d31/d30/d3c/f133 zero size 2022-01-31T19:44:38.553 INFO:tasks.workunit.client.1.smithi171.stdout:7/940: creat d1/d1e/d2a/d29/d31/d30/d3c/f135 x:0 0 0 2022-01-31T19:44:38.555 INFO:tasks.workunit.client.1.smithi171.stdout:7/941: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/d131/f136 x:0 0 0 2022-01-31T19:44:38.556 INFO:tasks.workunit.client.1.smithi171.stdout:7/942: symlink d1/d26/d47/d8f/l137 0 2022-01-31T19:44:38.559 INFO:tasks.workunit.client.1.smithi171.stdout:7/943: rename d1/d1e/d2a/d29/d31/ff to d1/f138 0 2022-01-31T19:44:38.560 INFO:tasks.workunit.client.1.smithi171.stdout:7/944: creat d1/d1e/d2a/d29/d31/d30/d3c/f139 x:0 0 0 2022-01-31T19:44:38.560 INFO:tasks.workunit.client.1.smithi171.stdout:7/945: write d1/d1e/d2a/d58/fcc [163140,98292] 0 2022-01-31T19:44:38.566 INFO:tasks.workunit.client.1.smithi171.stdout:7/946: dread d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/f18 [0,4194304] 0 2022-01-31T19:44:38.567 INFO:tasks.workunit.client.1.smithi171.stdout:7/947: symlink d1/d1e/d2a/d29/d31/d30/d3c/d7e/l13a 0 2022-01-31T19:44:38.568 INFO:tasks.workunit.client.1.smithi171.stdout:7/948: mkdir d1/d1e/d13b 0 2022-01-31T19:44:38.595 INFO:tasks.workunit.client.1.smithi171.stdout:7/949: dwrite d1/d1e/d2a/d29/d31/d30/d6b/f130 [0,4194304] 0 2022-01-31T19:44:38.596 INFO:tasks.workunit.client.1.smithi171.stdout:7/950: symlink d1/d1e/l13c 0 2022-01-31T19:44:38.598 INFO:tasks.workunit.client.1.smithi171.stdout:7/951: mkdir d1/d1e/d2a/d29/d31/d30/d13d 0 2022-01-31T19:44:38.616 INFO:tasks.workunit.client.1.smithi171.stdout:7/952: dwrite d1/d1e/d2a/d29/d77/f109 [4194304,4194304] 0 2022-01-31T19:44:38.639 INFO:tasks.workunit.client.1.smithi171.stdout:7/953: dwrite d1/d26/d47/d8f/fa0 [4194304,4194304] 0 2022-01-31T19:44:38.642 INFO:tasks.workunit.client.1.smithi171.stdout:7/954: rename d1/d26/d47/d8f/f100 to d1/d1e/d2a/f13e 0 2022-01-31T19:44:38.643 INFO:tasks.workunit.client.1.smithi171.stdout:7/955: write d1/d1e/d2a/d29/d31/d30/dc4/fe6 [603482,46905] 0 2022-01-31T19:44:38.644 INFO:tasks.workunit.client.1.smithi171.stdout:7/956: write d1/d1e/d2a/f3b [668741,109018] 0 2022-01-31T19:44:38.644 INFO:tasks.workunit.client.1.smithi171.stdout:7/957: stat d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/c23 0 2022-01-31T19:44:38.645 INFO:tasks.workunit.client.1.smithi171.stdout:7/958: rmdir d1/d26/d81 39 2022-01-31T19:44:38.645 INFO:tasks.workunit.client.1.smithi171.stdout:7/959: stat d1/d1e/d2a/d29/d31/d30/d3c/d7e/fe1 0 2022-01-31T19:44:38.647 INFO:tasks.workunit.client.1.smithi171.stdout:7/960: rename d1/d1e/d2a/d29/d31/d30/d3c/d7e/f10f to d1/d26/d81/f13f 0 2022-01-31T19:44:38.648 INFO:tasks.workunit.client.1.smithi171.stdout:7/961: truncate d1/d1e/d2a/d29/d31/d30/f4d 1202663 0 2022-01-31T19:44:39.231 INFO:tasks.workunit.client.1.smithi171.stdout:7/962: sync 2022-01-31T19:44:39.233 INFO:tasks.workunit.client.1.smithi171.stdout:7/963: rename d1/d1e/d2a/d29/d31/d30/d3c/f45 to d1/d1e/d2a/d29/d31/d30/d6b/d104/f140 0 2022-01-31T19:44:39.233 INFO:tasks.workunit.client.1.smithi171.stdout:7/964: chown d1/d1e/d2a/d29/d31/d30/d6b/f86 2 1 2022-01-31T19:44:39.234 INFO:tasks.workunit.client.1.smithi171.stdout:7/965: creat d1/d26/f141 x:0 0 0 2022-01-31T19:44:39.249 INFO:tasks.workunit.client.1.smithi171.stdout:7/966: dwrite d1/d1e/d2a/d29/fed [0,4194304] 0 2022-01-31T19:44:39.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: audit 2022-01-31T19:44:37.963475+0000 mon.smithi167 2022-01-31T19:44:39.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: (mon.0) 688 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:39.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: cluster 2022-01-31T19:44:38.439103+0000 mgr.smithi171.asyxnc (mgr.24387) 40 : cluster [DBG] pgmap v28: 65 pgs: 65 active+clean; 3.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 110 MiB/s rd, 243 MiB/s wr, 406 op/s 2022-01-31T19:44:39.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: cluster 2022-01-31T19:44:38 2022-01-31T19:44:39.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: .541753+0000 mon.smithi167 (mon.0) 689 : cluster 2022-01-31T19:44:39.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: [DBG] Standby manager daemon smithi167.aciqpk restarted 2022-01-31T19:44:39.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: cluster 2022-01-31T19: 2022-01-31T19:44:39.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: 44:38.541883+0000 mon.smithi167 (mon.0) 690 2022-01-31T19:44:39.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: : cluster [DBG] Standby manager daemon smithi167.aciqpk started 2022-01-31T19:44:39.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:44:39.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: 31T19:44:38.544650+0000 mon.smithi167 (mon 2022-01-31T19:44:39.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: .0) 691 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/crt"}]: dispatch 2022-01-31T19:44:39.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: audit 2022-01-31T19:44:39.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: 2022-01-31T19 2022-01-31T19:44:39.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: :44:38.545057+0000 mon.smithi167 (mon.0 2022-01-31T19:44:39.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: ) 692 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:44:39.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: audit 2022-01-31T19:44:39.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: 2022-01-31T19:44:38 2022-01-31T19:44:39.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: .547224+0000 mon.smithi167 (mon.0) 693 : audit 2022-01-31T19:44:39.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/key"}]: dispatch 2022-01-31T19:44:39.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: audit 2022-01-31T19:44:39.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: 2022-01-31T19:44:38.548114+0000 2022-01-31T19:44:39.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: mon.smithi167 (mon.0) 694 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:44:39.363 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:38 smithi171 conmon[35325]: 2022-01-31T19:44:39.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: audit 2022-01-31T19:44:37 2022-01-31T19:44:39.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: .963475+0000 mon.smithi167 (mon.0) 688 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:39.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: cluster 2022-01-31T19:44:38.439103+0000 mgr.smithi171.asyxnc (mgr.24387) 40 : cluster [DBG] pgmap v28: 65 pgs: 65 active+clean; 3.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 110 MiB/s rd, 243 MiB/s wr, 406 op/s 2022-01-31T19:44:39.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: cluster 2022-01-31T19:44:38.541753+0000 mon.smithi167 (mon.0) 689 : cluster [DBG] Standby manager daemon smithi167.aciqpk restarted 2022-01-31T19:44:39.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: cluster 2022-01-31T19:44:38.541883+0000 mon.smithi167 (mon.0) 690 : cluster [DBG] Standby manager daemon smithi167.aciqpk started 2022-01-31T19:44:39.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: audit 2022-01-31T19:44:38.544650+0000 mon.smithi167 (mon.0) 691 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/crt"}]: dispatch 2022-01-31T19:44:39.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: audit 2022-01-31T19:44:38.545057+0000 mon.smithi167 (mon.0) 692 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:44:39.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: audit 2022-01-31T19:44:38.547224+0000 mon.smithi167 (mon.0) 693 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi167.aciqpk/key"}]: dispatch 2022-01-31T19:44:39.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: audit 2022 2022-01-31T19:44:39.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:38 smithi167 conmon[32206]: -01-31T19:44:38.548114+0000 mon.smithi167 (mon.0) 694 : audit [DBG] from='mgr.? 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:44:40.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:39 smithi171 conmon[35325]: cluster 2022-01-31T19:44:38.972973+0000 mon.smithi167 2022-01-31T19:44:40.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:39 smithi171 conmon[35325]: (mon.0) 695 : cluster [DBG] mgrmap e20: smithi171.asyxnc(active, since 46s), standbys: smithi167.aciqpk 2022-01-31T19:44:40.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:39 smithi167 conmon[32206]: cluster 2022-01-31T19:44:38.972973+0000 mon.smithi167 (mon.0) 695 : cluster [DBG] mgrmap e20: smithi171.asyxnc(active, since 46s), standbys: smithi167.aciqpk 2022-01-31T19:44:41.075 INFO:tasks.workunit.client.1.smithi171.stdout:7/967: sync 2022-01-31T19:44:41.077 INFO:tasks.workunit.client.1.smithi171.stdout:7/968: symlink d1/d1e/l142 0 2022-01-31T19:44:41.078 INFO:tasks.workunit.client.1.smithi171.stdout:7/969: dread - d1/d1e/d2a/d58/df8/f12d zero size 2022-01-31T19:44:41.096 INFO:tasks.workunit.client.1.smithi171.stdout:7/970: dwrite d1/d1e/d2a/d29/d31/d30/d3c/db9/f129 [0,4194304] 0 2022-01-31T19:44:41.112 INFO:tasks.workunit.client.1.smithi171.stdout:7/971: dwrite d1/d1e/d2a/d29/d31/d30/d3c/fbb [0,4194304] 0 2022-01-31T19:44:41.112 INFO:tasks.workunit.client.1.smithi171.stdout:7/972: fdatasync d1/d1e/d2a/d58/fb8 0 2022-01-31T19:44:41.119 INFO:tasks.workunit.client.1.smithi171.stdout:7/973: dread d1/f5 [0,4194304] 0 2022-01-31T19:44:41.120 INFO:tasks.workunit.client.1.smithi171.stdout:7/974: mkdir d1/d1e/d143 0 2022-01-31T19:44:41.121 INFO:tasks.workunit.client.1.smithi171.stdout:7/975: creat d1/d1e/d2a/d29/d31/d30/d3c/db9/d114/f144 x:0 0 0 2022-01-31T19:44:41.122 INFO:tasks.workunit.client.1.smithi171.stdout:7/976: rename d1/d1e/d2a/d58/ldb to d1/d1e/d2a/d29/d31/d30/l145 0 2022-01-31T19:44:41.131 INFO:tasks.workunit.client.1.smithi171.stdout:7/977: dread d1/d1e/d2a/d29/d31/d30/f39 [0,4194304] 0 2022-01-31T19:44:41.133 INFO:tasks.workunit.client.1.smithi171.stdout:7/978: link d1/d1e/d2a/d29/d31/f99 d1/d1e/d2a/d29/d31/f146 0 2022-01-31T19:44:41.216 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-01-31T19:44:41.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-01-31T19:44:42.054 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: cluster 2022-01-31T19:44:40.439627+0000 mgr.smithi171.asyxnc (mgr 2022-01-31T19:44:42.054 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: .24387) 41 : cluster [DBG] pgmap v29: 65 pgs: 65 active+clean; 3.2 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 105 MiB/s rd, 235 MiB/s wr, 471 op/s 2022-01-31T19:44:42.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: audit 2022-01-31T19:44: 2022-01-31T19:44:42.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: 41.046037+0000 mon.smithi167 (mon.0) 696 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:42.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: audit 2022-01-31 2022-01-31T19:44:42.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: T19:44:41.051981+0000 mon.smithi167 (mon.0) 697 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:42.056 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:42.056 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: 01-31T19:44:41.056815+0000 2022-01-31T19:44:42.056 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: mon.smithi171 (mon.1) 66 : audit [DBG] 2022-01-31T19:44:42.056 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:42.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: audit 2022-01-31T19 2022-01-31T19:44:42.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: :44:41.062196+0000 mon.smithi171 (mon. 2022-01-31T19:44:42.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: 1) 67 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr fail", "who": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:44:42.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:42.058 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: 01-31T19:44:41.062383+0000 mon.smithi167 ( 2022-01-31T19:44:42.058 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: mon.0) 698 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr fail", "who": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:44:42.058 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: cluster 2022-01-31T19:44:41. 2022-01-31T19:44:42.058 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:41 smithi171 conmon[35325]: 065791+0000 mon.smithi167 (mon.0) 699 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2022-01-31T19:44:42.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: cluster 2022-01-31T19:44:40.439627+0000 mgr.smithi171.asyxnc (mgr.24387) 41 : cluster [DBG] pgmap v29: 65 pgs: 65 active+clean; 3.2 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 105 MiB/s rd, 235 MiB/s wr, 471 op/s 2022-01-31T19:44:42.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: audit 2022-01-31T19:44:41.046037+0000 mon.smithi167 (mon.0) 696 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:42.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: audit 2022-01-31T19:44:41.051981+0000 mon.smithi167 (mon.0) 697 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' 2022-01-31T19:44:42.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: audit 2022-01-31T19:44:41.056815+0000 mon.smithi171 (mon.1) 66 : audit [DBG] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:42.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: audit 2022-01-31T19:44:41.062196+0000 mon.smithi171 (mon.1) 67 : audit [INF] from='mgr.24387 172.21.15.171:0/1584952245' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr fail", "who": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:44:42.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: audit 2022-01-31T19:44:41.062383+0000 mon.smithi167 (mon.0) 698 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "mgr fail", "who": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:44:42.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:41 smithi167 conmon[32206]: cluster 2022-01-31T19:44:41.065791+0000 mon.smithi167 (mon.0) 699 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2022-01-31T19:44:42.214 INFO:tasks.workunit.client.1.smithi171.stdout:7/979: sync 2022-01-31T19:44:42.233 INFO:tasks.workunit.client.1.smithi171.stdout:7/980: dwrite d1/d1e/d2a/d29/d31/d30/d3c/fbb [0,4194304] 0 2022-01-31T19:44:42.233 INFO:tasks.workunit.client.1.smithi171.stdout:7/981: readlink d1/d26/l61 0 2022-01-31T19:44:42.236 INFO:tasks.workunit.client.1.smithi171.stdout:7/982: truncate d1/d26/d47/f11a 129122 0 2022-01-31T19:44:42.238 INFO:tasks.workunit.client.1.smithi171.stdout:7/983: symlink d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/d123/l147 0 2022-01-31T19:44:42.238 INFO:tasks.workunit.client.1.smithi171.stdout:7/984: chown d1/d1e/d2a/d29/lf7 15 1 2022-01-31T19:44:42.238 INFO:tasks.workunit.client.1.smithi171.stdout:7/985: symlink d1/d1e/d2a/d29/d31/d30/l148 0 2022-01-31T19:44:42.239 INFO:tasks.workunit.client.1.smithi171.stdout:7/986: write d1/d1e/d2a/d29/d77/dab/dd5/fec [235199,64302] 0 2022-01-31T19:44:42.241 INFO:tasks.workunit.client.1.smithi171.stdout:7/987: link d1/d1e/d2a/l84 d1/d26/l149 0 2022-01-31T19:44:42.241 INFO:tasks.workunit.client.1.smithi171.stdout:7/988: dread - d1/d1e/d2a/d29/d31/d30/d3c/f135 zero size 2022-01-31T19:44:42.241 INFO:tasks.workunit.client.1.smithi171.stdout:7/989: readlink d1/d26/d47/l121 0 2022-01-31T19:44:42.241 INFO:tasks.workunit.client.1.smithi171.stdout:7/990: chown d1/d1e/d2a/d29/d31/d30/d3c/db9/d131/dcd/fd0 3 1 2022-01-31T19:44:42.242 INFO:tasks.workunit.client.1.smithi171.stdout:7/991: mkdir d1/d1e/d2a/d29/d31/d30/d3c/d5f/d14a 0 2022-01-31T19:44:42.242 INFO:tasks.workunit.client.1.smithi171.stdout:7/992: symlink d1/d1e/d2a/d29/d31/d30/d3c/d5f/l14b 0 2022-01-31T19:44:42.243 INFO:tasks.workunit.client.1.smithi171.stdout:7/993: dread - d1/d1e/d2a/d29/d31/d30/d3c/f133 zero size 2022-01-31T19:44:42.244 INFO:tasks.workunit.client.1.smithi171.stdout:7/994: rename d1/d26/d47/cee to d1/d1e/d2a/d29/d31/d30/d6b/d104/c14c 0 2022-01-31T19:44:42.244 INFO:tasks.workunit.client.1.smithi171.stdout:7/995: truncate d1/d1e/d2a/ffd 512962 0 2022-01-31T19:44:42.246 INFO:tasks.workunit.client.1.smithi171.stdout:7/996: link d1/d1e/d2a/d29/d31/d30/d3c/d7e/l13a d1/d1e/d2a/d29/d31/d30/d3c/d5f/l14d 0 2022-01-31T19:44:42.246 INFO:tasks.workunit.client.1.smithi171.stdout:7/997: stat d1/d1e/d2a/f3b 0 2022-01-31T19:44:42.248 INFO:tasks.workunit.client.1.smithi171.stdout:7/998: truncate d1/d1e/d2a/d29/d31/d30/d3c/db9/dc9/dd7/fd2 539 0 2022-01-31T19:44:42.250 INFO:tasks.workunit.client.1.smithi171.stdout:7/999: dread d1/d1e/fdd [0,4194304] 0 2022-01-31T19:44:42.254 INFO:tasks.workunit.client.1.smithi171.stderr:+ rm -rf -- ./tmp.TTnjqkluUa 2022-01-31T19:44:42.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:41.058794+0000 mgr.smithi171.asyxnc (mgr.24387) 42 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi171.asyxnc) 2022-01-31T19:44:42.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:41.061966+0000 mgr.smithi171.asyxnc (mgr.24387) 43 : cephadm [INF] Failing over to other MGR 2022-01-31T19:44:42.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.053889+0000 mon.smithi167 (mon.0) 700 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd='[{"prefix": "mgr fail", "who": "smithi171.asyxnc"}]': finished 2022-01-31T19:44:42.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: cluster 2022-01-31T19:44:42.053962+0000 mon.smithi167 (mon.0) 701 : cluster [DBG] mgrmap e21: smithi167.aciqpk(active, starting, since 0.990935s) 2022-01-31T19:44:42.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.056087+0000 mon.smithi167 (mon.0) 702 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:44:42.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.056274+0000 mon.smithi167 (mon.0) 703 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:44:42.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.056635+0000 mon.smithi167 (mon.0) 704 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.sgnfie"}]: dispatch 2022-01-31T19:44:42.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.056950+0000 mon.smithi167 (mon.0) 705 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.rdkazj"}]: dispatch 2022-01-31T19:44:42.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.057212+0000 mon.smithi167 (mon.0) 706 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.dyqqtj"}]: dispatch 2022-01-31T19:44:42.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.057422+0000 mon.smithi167 (mon.0) 707 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.erzfea"}]: dispatch 2022-01-31T19:44:42.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.057697+0000 mon.smithi167 (mon.0) 708 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:44:42.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.057889+0000 mon.smithi167 (mon.0) 709 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:44:42.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.058109+0000 mon.smithi167 (mon.0) 710 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:44:42.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.058317+0000 mon.smithi167 (mon.0) 711 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:44:42.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.058528+0000 mon.smithi167 (mon.0) 712 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:44:42.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.058759+0000 mon.smithi167 (mon.0) 713 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:44:42.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.059030+0000 mon.smithi167 (mon.0) 714 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:44:42.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.059354+0000 mon.smithi167 (mon.0) 715 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:44:42.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.059666+0000 mon.smithi167 (mon.0) 716 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:44:42.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.059989+0000 mon.smithi167 (mon.0) 717 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:44:42.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: cluster 2022-01-31T19:44:42.091294+0000 mon.smithi167 (mon.0) 718 : cluster [INF] Manager daemon smithi167.aciqpk is now available 2022-01-31T19:44:42.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.112051+0000 mon.smithi167 (mon.0) 719 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:42.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.114172+0000 mon.smithi167 (mon.0) 720 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:42.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.115600+0000 mon.smithi167 (mon.0) 721 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:42.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.116954+0000 mon.smithi167 (mon.0) 722 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:44:42.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.146190+0000 mon.smithi167 (mon.0) 723 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:44:42.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.149719+0000 mon.smithi167 (mon.0) 724 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:44:42.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:42 smithi167 conmon[32206]: audit 2022-01-31T19:44:42.622504+0000 mon.smithi167 (mon.0) 725 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:42.721 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.718+0000 7efd797f6700 1 -- 172.21.15.167:0/2045373825 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd7405e650 msgr2=0x7efd740f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:42.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.718+0000 7efd797f6700 1 --2- 172.21.15.167:0/2045373825 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd7405e650 0x7efd740f5f60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7efd6c0099e0 tx=0x7efd6c004d10).stop 2022-01-31T19:44:42.722 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.719+0000 7efd797f6700 1 -- 172.21.15.167:0/2045373825 shutdown_connections 2022-01-31T19:44:42.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.719+0000 7efd797f6700 1 --2- 172.21.15.167:0/2045373825 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd7405e650 0x7efd740f5f60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:42.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.719+0000 7efd797f6700 1 --2- 172.21.15.167:0/2045373825 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd740f4d60 0x7efd740f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:44:42.723 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.719+0000 7efd797f6700 1 -- 172.21.15.167:0/2045373825 >> 172.21.15.167:0/2045373825 conn(0x7efd740f04e0 msgr2=0x7efd740f2900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:42.724 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.721+0000 7efd797f6700 1 -- 172.21.15.167:0/2045373825 shutdown_connections 2022-01-31T19:44:42.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.722+0000 7efd797f6700 1 -- 172.21.15.167:0/2045373825 wait complete. 2022-01-31T19:44:42.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.722+0000 7efd797f6700 1 Processor -- start 2022-01-31T19:44:42.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.723+0000 7efd797f6700 1 -- start start 2022-01-31T19:44:42.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.723+0000 7efd797f6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd7405e650 0x7efd74061980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:42.725 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.723+0000 7efd797f6700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 0x7efd74061e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:42.726 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.723+0000 7efd797f6700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74062500 con 0x7efd740f4d60 2022-01-31T19:44:42.726 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.723+0000 7efd797f6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74062db0 con 0x7efd7405e650 2022-01-31T19:44:42.726 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd737fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 0x7efd74061e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:42.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd737fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 0x7efd74061e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34338/0 (socket says 172.21.15.167:34338) 2022-01-31T19:44:42.727 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd737fe700 1 -- 172.21.15.167:0/773034101 learned_addr learned my addr 172.21.15.167:0/773034101 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:42.728 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd73fff700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd7405e650 0x7efd74061980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:42.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd737fe700 1 -- 172.21.15.167:0/773034101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd7405e650 msgr2=0x7efd74061980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:42.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd737fe700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd7405e650 0x7efd74061980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:42.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd737fe700 1 -- 172.21.15.167:0/773034101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c009640 con 0x7efd740f4d60 2022-01-31T19:44:42.729 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.725+0000 7efd73fff700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd7405e650 0x7efd74061980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-01-31T19:44:42.730 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.726+0000 7efd737fe700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 0x7efd74061e70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7efd6c00b3b0 tx=0x7efd6c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:42.730 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.726+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6c00e070 con 0x7efd740f4d60 2022-01-31T19:44:42.730 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.726+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd74063000 con 0x7efd740f4d60 2022-01-31T19:44:42.730 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.727+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd740634e0 con 0x7efd740f4d60 2022-01-31T19:44:42.731 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.727+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7efd6c0049b0 con 0x7efd740f4d60 2022-01-31T19:44:42.731 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.727+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6c018070 con 0x7efd740f4d60 2022-01-31T19:44:42.731 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.728+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd54004fa0 con 0x7efd740f4d60 2022-01-31T19:44:42.738 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.737+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 21) v1 ==== 41212+0+0 (secure 0 0 0) 0x7efd6c0093b0 con 0x7efd740f4d60 2022-01-31T19:44:42.739 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.737+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7efd6c047a50 con 0x7efd740f4d60 2022-01-31T19:44:42.739 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:42.737+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7efd6c06fcb0 con 0x7efd740f4d60 2022-01-31T19:44:42.980 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:41.058794+0000 mgr.smithi171.asyxnc (mgr.24387) 42 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi171.asyxnc) 2022-01-31T19:44:42.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:41.061966+0000 mgr.smithi171.asyxnc (mgr.24387) 43 : cephadm [INF] Failing over to other MGR 2022-01-31T19:44:42.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.053889+0000 mon.smithi167 (mon.0) 700 : audit [INF] from='mgr.24387 ' entity='mgr.smithi171.asyxnc' cmd='[{"prefix": "mgr fail", "who": "smithi171.asyxnc"}]': finished 2022-01-31T19:44:42.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: cluster 2022-01-31T19:44:42.053962+0000 mon.smithi167 (mon.0) 701 : cluster [DBG] mgrmap e21: smithi167.aciqpk(active, starting, since 0.990935s) 2022-01-31T19:44:42.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.056087+0000 mon.smithi167 (mon.0) 702 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi167"}]: dispatch 2022-01-31T19:44:42.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.056274+0000 mon.smithi167 (mon.0) 703 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata", "id": "smithi171"}]: dispatch 2022-01-31T19:44:42.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.056635+0000 mon.smithi167 (mon.0) 704 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.sgnfie"}]: dispatch 2022-01-31T19:44:42.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.056950+0000 mon.smithi167 (mon.0) 705 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi167.rdkazj"}]: dispatch 2022-01-31T19:44:42.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.057212+0000 mon.smithi167 (mon.0) 706 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.dyqqtj"}]: dispatch 2022-01-31T19:44:42.983 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.057422+0000 mon.smithi167 (mon.0) 707 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi171.erzfea"}]: dispatch 2022-01-31T19:44:42.983 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.057697+0000 mon.smithi167 (mon.0) 708 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi167.aciqpk", "id": "smithi167.aciqpk"}]: dispatch 2022-01-31T19:44:42.983 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.057889+0000 mon.smithi167 (mon.0) 709 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-01-31T19:44:42.983 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.058109+0000 mon.smithi167 (mon.0) 710 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-01-31T19:44:42.984 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.058317+0000 mon.smithi167 (mon.0) 711 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-01-31T19:44:42.984 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.058528+0000 mon.smithi167 (mon.0) 712 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-01-31T19:44:42.984 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.058759+0000 mon.smithi167 (mon.0) 713 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-01-31T19:44:42.984 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.059030+0000 mon.smithi167 (mon.0) 714 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-01-31T19:44:42.985 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.059354+0000 mon.smithi167 (mon.0) 715 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-01-31T19:44:42.985 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.059666+0000 mon.smithi167 (mon.0) 716 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-01-31T19:44:42.985 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.059989+0000 mon.smithi167 (mon.0) 717 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-01-31T19:44:42.985 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: cluster 2022-01-31T19:44:42.091294+0000 mon.smithi167 (mon.0) 718 : cluster [INF] Manager daemon smithi167.aciqpk is now available 2022-01-31T19:44:42.985 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.112051+0000 mon.smithi167 (mon.0) 719 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:42.986 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.114172+0000 mon.smithi167 (mon.0) 720 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:42.986 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.115600+0000 mon.smithi167 (mon.0) 721 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:42.986 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.116954+0000 mon.smithi167 (mon.0) 722 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:44:42.986 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.146190+0000 mon.smithi167 (mon.0) 723 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/mirror_snapshot_schedule"}]: dispatch 2022-01-31T19:44:42.987 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.149719+0000 mon.smithi167 (mon.0) 724 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi167.aciqpk/trash_purge_schedule"}]: dispatch 2022-01-31T19:44:42.987 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:42 smithi171 conmon[35325]: audit 2022-01-31T19:44:42.622504+0000 mon.smithi167 (mon.0) 725 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:43.060 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.059+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 22) v1 ==== 41566+0+0 (secure 0 0 0) 0x7efd6c01bd70 con 0x7efd740f4d60 2022-01-31T19:44:43.060 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.059+0000 7efd70ff9700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efd5c039600 0x7efd5c03b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.061 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.059+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 --> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd5c03bee0 con 0x7efd5c039600 2022-01-31T19:44:43.072 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.070+0000 7efd73fff700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efd5c039600 0x7efd5c03b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:43.072 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.071+0000 7efd73fff700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efd5c039600 0x7efd5c03b9f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7efd74061710 tx=0x7efd68008040).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:43.074 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.073+0000 7efd70ff9700 1 -- 172.21.15.167:0/773034101 <== mgr.14650 v2:172.21.15.167:6800/2258949773 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+257 (secure 0 0 0) 0x7efd5c03bee0 con 0x7efd5c039600 2022-01-31T19:44:43.077 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efd5c039600 msgr2=0x7efd5c03b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.077 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efd5c039600 0x7efd5c03b9f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7efd74061710 tx=0x7efd68008040).stop 2022-01-31T19:44:43.077 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 msgr2=0x7efd74061e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.077 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 0x7efd74061e70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7efd6c00b3b0 tx=0x7efd6c004d10).stop 2022-01-31T19:44:43.078 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 shutdown_connections 2022-01-31T19:44:43.078 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efd5c039600 0x7efd5c03b9f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.078 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efd7405e650 0x7efd74061980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.078 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 --2- 172.21.15.167:0/773034101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd740f4d60 0x7efd74061e70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.079 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 >> 172.21.15.167:0/773034101 conn(0x7efd740f04e0 msgr2=0x7efd740f7180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:43.079 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 shutdown_connections 2022-01-31T19:44:43.079 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.076+0000 7efd797f6700 1 -- 172.21.15.167:0/773034101 wait complete. 2022-01-31T19:44:43.088 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:44:43.215 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1462113308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 msgr2=0x7fbbac0f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.216 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1462113308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac0f6a70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fbb9c004660 tx=0x7fbb9c009b30).stop 2022-01-31T19:44:43.216 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1462113308 shutdown_connections 2022-01-31T19:44:43.216 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1462113308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbac0f78f0 0x7fbbac0f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:44:43.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1462113308 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac0f6a70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1462113308 >> 172.21.15.167:0/1462113308 conn(0x7fbbac0f1db0 msgr2=0x7fbbac0f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:43.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1462113308 shutdown_connections 2022-01-31T19:44:43.217 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.214+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1462113308 wait complete. 2022-01-31T19:44:43.219 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbb11a1700 1 Processor -- start 2022-01-31T19:44:43.219 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbb11a1700 1 -- start start 2022-01-31T19:44:43.220 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbb11a1700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.220 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbb11a1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbac0f78f0 0x7fbbac1051c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.220 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbb11a1700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac105850 con 0x7fbbac0f6650 2022-01-31T19:44:43.220 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbb11a1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac10b360 con 0x7fbbac0f78f0 2022-01-31T19:44:43.221 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbabfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:43.221 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbabfff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34370/0 (socket says 172.21.15.167:34370) 2022-01-31T19:44:43.221 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.215+0000 7fbbabfff700 1 -- 172.21.15.167:0/1200563037 learned_addr learned my addr 172.21.15.167:0/1200563037 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:43.221 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbbabfff700 1 -- 172.21.15.167:0/1200563037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbac0f78f0 msgr2=0x7fbbac1051c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-01-31T19:44:43.221 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbbabfff700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbac0f78f0 0x7fbbac1051c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:44:43.222 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbbabfff700 1 -- 172.21.15.167:0/1200563037 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c005040 con 0x7fbbac0f6650 2022-01-31T19:44:43.222 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbbabfff700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac104cd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fbb9c000c00 tx=0x7fbb9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:43.222 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb9c018070 con 0x7fbbac0f6650 2022-01-31T19:44:43.222 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac10b5b0 con 0x7fbbac0f6650 2022-01-31T19:44:43.223 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac10bb10 con 0x7fbbac0f6650 2022-01-31T19:44:43.223 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.216+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fbb9c007d70 con 0x7fbbac0f6650 2022-01-31T19:44:43.223 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.217+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb9c01c620 con 0x7fbbac0f6650 2022-01-31T19:44:43.223 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.218+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8c004fa0 con 0x7fbbac0f6650 2022-01-31T19:44:43.225 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.219+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 22) v1 ==== 41566+0+0 (secure 0 0 0) 0x7fbb9c01d960 con 0x7fbbac0f6650 2022-01-31T19:44:43.225 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.219+0000 7fbba8ff9700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fbb940344e0 0x7fbb940369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.219+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7fbb9c048390 con 0x7fbbac0f6650 2022-01-31T19:44:43.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.219+0000 7fbbab7fe700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fbb940344e0 0x7fbb940369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:43.226 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.220+0000 7fbbab7fe700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fbb940344e0 0x7fbb940369a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fbba0009500 tx=0x7fbba0006a30).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:43.227 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.226+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7fbb9c01b780 con 0x7fbbac0f6650 2022-01-31T19:44:43.477 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.476+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 --> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb8c000bc0 con 0x7fbb940344e0 2022-01-31T19:44:43.487 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.485+0000 7fbba8ff9700 1 -- 172.21.15.167:0/1200563037 <== mgr.14650 v2:172.21.15.167:6800/2258949773 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7fbb8c000bc0 con 0x7fbb940344e0 2022-01-31T19:44:43.487 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:44:43.487 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (5m) 6s ago 6m 22.1M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:44:43.488 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (6m) 6s ago 6m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:44:43.488 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (6m) 25s ago 5m 7151k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:44:43.488 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (5m) 6s ago 6m 38.3M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:44:43.488 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.rdkazj smithi167 running (2m) 6s ago 2m 334M - 16.2.5 46cf6318c64e 29f596ef1666 2022-01-31T19:44:43.489 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.sgnfie smithi167 running (2m) 6s ago 2m 2504M - 16.2.5 46cf6318c64e 8c462c58dec4 2022-01-31T19:44:43.489 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.dyqqtj smithi171 running (2m) 25s ago 2m 794M - 16.2.5 46cf6318c64e 5e84b5044af1 2022-01-31T19:44:43.489 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.erzfea smithi171 running (2m) 25s ago 2m 151M - 16.2.5 46cf6318c64e 802065eafb38 2022-01-31T19:44:43.489 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:8443 running (14s) 6s ago 7m 47.6M - 17.0.0-10469-g29e1fc17 e02b80923a35 493415fe7392 2022-01-31T19:44:43.489 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (59s) 25s ago 5m 442M - 17.0.0-10469-g29e1fc17 e02b80923a35 56429afd72ba 2022-01-31T19:44:43.490 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running (7m) 6s ago 7m 96.4M 2048M 16.2.5 46cf6318c64e 0ebba348055e 2022-01-31T19:44:43.490 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (5m) 25s ago 5m 68.7M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:44:43.490 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (6m) 6s ago 6m 18.2M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:44:43.491 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (5m) 25s ago 5m 18.2M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:44:43.491 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (5m) 6s ago 5m 581M 1584M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:44:43.491 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (5m) 6s ago 5m 643M 1584M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:44:43.492 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (4m) 6s ago 4m 528M 1584M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:44:43.492 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (4m) 25s ago 4m 307M 2608M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:44:43.492 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (4m) 25s ago 4m 242M 2608M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:44:43.492 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (3m) 25s ago 3m 287M 2608M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:44:43.492 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (5m) 6s ago 6m 48.1M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:44:43.493 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.488+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fbb940344e0 msgr2=0x7fbb940369a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.488+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fbb940344e0 0x7fbb940369a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fbba0009500 tx=0x7fbba0006a30).stop 2022-01-31T19:44:43.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.488+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 msgr2=0x7fbbac104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.488+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac104cd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fbb9c000c00 tx=0x7fbb9c009b30).stop 2022-01-31T19:44:43.494 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 shutdown_connections 2022-01-31T19:44:43.495 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fbb940344e0 0x7fbb940369a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.495 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbbac0f6650 0x7fbbac104cd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.495 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 --2- 172.21.15.167:0/1200563037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbac0f78f0 0x7fbbac1051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-01-31T19:44:43.495 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 >> 172.21.15.167:0/1200563037 conn(0x7fbbac0f1db0 msgr2=0x7fbbac06a600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:43.496 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 shutdown_connections 2022-01-31T19:44:43.496 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.489+0000 7fbbb11a1700 1 -- 172.21.15.167:0/1200563037 wait complete. 2022-01-31T19:44:43.652 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.644+0000 7efc99b07700 1 -- 172.21.15.167:0/3929683387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc94068370 msgr2=0x7efc940687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.652 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.644+0000 7efc99b07700 1 --2- 172.21.15.167:0/3929683387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc94068370 0x7efc940687f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7efc880045a0 tx=0x7efc88009b30).stop 2022-01-31T19:44:43.652 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.644+0000 7efc99b07700 1 -- 172.21.15.167:0/3929683387 shutdown_connections 2022-01-31T19:44:43.653 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.644+0000 7efc99b07700 1 --2- 172.21.15.167:0/3929683387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc94068370 0x7efc940687f0 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0x7efc880045a0 tx=0x7efc88009b30).stop 2022-01-31T19:44:43.653 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.644+0000 7efc99b07700 1 --2- 172.21.15.167:0/3929683387 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc940697f0 0x7efc94067e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.653 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.644+0000 7efc99b07700 1 -- 172.21.15.167:0/3929683387 >> 172.21.15.167:0/3929683387 conn(0x7efc940637e0 msgr2=0x7efc94065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:43.654 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.646+0000 7efc99b07700 1 -- 172.21.15.167:0/3929683387 shutdown_connections 2022-01-31T19:44:43.654 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.646+0000 7efc99b07700 1 -- 172.21.15.167:0/3929683387 wait complete. 2022-01-31T19:44:43.654 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.647+0000 7efc99b07700 1 Processor -- start 2022-01-31T19:44:43.654 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.647+0000 7efc99b07700 1 -- start start 2022-01-31T19:44:43.655 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.647+0000 7efc99b07700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 0x7efc94115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.655 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.648+0000 7efc99b07700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc941164c0 0x7efc941173b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.655 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.648+0000 7efc99b07700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94116990 con 0x7efc940697f0 2022-01-31T19:44:43.656 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.648+0000 7efc99b07700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94116ad0 con 0x7efc941164c0 2022-01-31T19:44:43.656 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.648+0000 7efc937fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 0x7efc94115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:43.656 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.648+0000 7efc937fe700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 0x7efc94115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34388/0 (socket says 172.21.15.167:34388) 2022-01-31T19:44:43.656 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.648+0000 7efc937fe700 1 -- 172.21.15.167:0/1056107724 learned_addr learned my addr 172.21.15.167:0/1056107724 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:43.657 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.649+0000 7efc92ffd700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc941164c0 0x7efc941173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:43.658 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.652+0000 7efc937fe700 1 -- 172.21.15.167:0/1056107724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc941164c0 msgr2=0x7efc941173b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:43.658 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.652+0000 7efc937fe700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc941164c0 0x7efc941173b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:43.658 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.652+0000 7efc937fe700 1 -- 172.21.15.167:0/1056107724 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc88005040 con 0x7efc940697f0 2022-01-31T19:44:43.658 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.653+0000 7efc937fe700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 0x7efc94115fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7efc84002700 tx=0x7efc84009cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:43.659 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.654+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc840079c0 con 0x7efc940697f0 2022-01-31T19:44:43.659 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.654+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7efc8400f040 con 0x7efc940697f0 2022-01-31T19:44:43.659 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.654+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc840147a0 con 0x7efc940697f0 2022-01-31T19:44:43.659 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.654+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc94117960 con 0x7efc940697f0 2022-01-31T19:44:43.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.654+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc94117f20 con 0x7efc940697f0 2022-01-31T19:44:43.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.656+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc74004fa0 con 0x7efc940697f0 2022-01-31T19:44:43.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.657+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 22) v1 ==== 41566+0+0 (secure 0 0 0) 0x7efc8401b060 con 0x7efc940697f0 2022-01-31T19:44:43.660 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.657+0000 7efc7bfff700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc7c034550 0x7efc7c036a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:43.661 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.657+0000 7efc92ffd700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc7c034550 0x7efc7c036a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:43.662 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.657+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7efc840461e0 con 0x7efc940697f0 2022-01-31T19:44:43.662 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.658+0000 7efc92ffd700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc7c034550 0x7efc7c036a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7efc88013590 tx=0x7efc88009b30).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:43.667 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:43.665+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7efc84014c80 con 0x7efc940697f0 2022-01-31T19:44:44.093 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.091+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efc74005e80 con 0x7efc940697f0 2022-01-31T19:44:44.094 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.091+0000 7efc7bfff700 1 -- 172.21.15.167:0/1056107724 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+691 (secure 0 0 0) 0x7efc84012b30 con 0x7efc940697f0 2022-01-31T19:44:44.094 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:44:44.094 INFO:teuthology.orchestra.run.smithi167.stdout: "mon": { 2022-01-31T19:44:44.095 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:44:44.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:44.095 INFO:teuthology.orchestra.run.smithi167.stdout: "mgr": { 2022-01-31T19:44:44.095 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev)": 1 2022-01-31T19:44:44.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:44.096 INFO:teuthology.orchestra.run.smithi167.stdout: "osd": { 2022-01-31T19:44:44.096 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6 2022-01-31T19:44:44.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:44.097 INFO:teuthology.orchestra.run.smithi167.stdout: "mds": { 2022-01-31T19:44:44.097 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4 2022-01-31T19:44:44.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:44:44.097 INFO:teuthology.orchestra.run.smithi167.stdout: "overall": { 2022-01-31T19:44:44.097 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2022-01-31T19:44:44.098 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev)": 1 2022-01-31T19:44:44.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2022-01-31T19:44:44.098 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:44:44.099 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc7c034550 msgr2=0x7efc7c036a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:44.099 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc7c034550 0x7efc7c036a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7efc88013590 tx=0x7efc88009b30).stop 2022-01-31T19:44:44.100 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 msgr2=0x7efc94115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:44.100 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 0x7efc94115fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7efc84002700 tx=0x7efc84009cc0).stop 2022-01-31T19:44:44.100 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 shutdown_connections 2022-01-31T19:44:44.101 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc7c034550 0x7efc7c036a10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.101 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc940697f0 0x7efc94115fd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.101 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 --2- 172.21.15.167:0/1056107724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc941164c0 0x7efc941173b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.101 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 >> 172.21.15.167:0/1056107724 conn(0x7efc940637e0 msgr2=0x7efc941013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:44.102 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 shutdown_connections 2022-01-31T19:44:44.102 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.094+0000 7efc99b07700 1 -- 172.21.15.167:0/1056107724 wait complete. 2022-01-31T19:44:44.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 -- 172.21.15.167:0/2763666027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f5560 msgr2=0x7f83540f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:44.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 --2- 172.21.15.167:0/2763666027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f5560 0x7f83540f5980 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8348004660 tx=0x7f8348009b30).stop 2022-01-31T19:44:44.231 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 -- 172.21.15.167:0/2763666027 shutdown_connections 2022-01-31T19:44:44.232 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 --2- 172.21.15.167:0/2763666027 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83540f6800 0x7f83540f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.232 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 --2- 172.21.15.167:0/2763666027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f5560 0x7f83540f5980 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.232 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 -- 172.21.15.167:0/2763666027 >> 172.21.15.167:0/2763666027 conn(0x7f83540f0ce0 msgr2=0x7f83540f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:44.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 -- 172.21.15.167:0/2763666027 shutdown_connections 2022-01-31T19:44:44.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.228+0000 7f8359b06700 1 -- 172.21.15.167:0/2763666027 wait complete. 2022-01-31T19:44:44.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.229+0000 7f8359b06700 1 Processor -- start 2022-01-31T19:44:44.233 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.229+0000 7f8359b06700 1 -- start start 2022-01-31T19:44:44.235 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.229+0000 7f8359b06700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83540f5560 0x7f8354109bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:44.235 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.229+0000 7f8359b06700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 0x7f835410c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:44.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.229+0000 7f8359b06700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83540604a0 con 0x7f83540f6800 2022-01-31T19:44:44.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.229+0000 7f8359b06700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83540605e0 con 0x7f83540f5560 2022-01-31T19:44:44.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 0x7f835410c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:44.236 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 0x7f835410c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34408/0 (socket says 172.21.15.167:34408) 2022-01-31T19:44:44.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 -- 172.21.15.167:0/2560504643 learned_addr learned my addr 172.21.15.167:0/2560504643 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:44:44.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8358b04700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83540f5560 0x7f8354109bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:44:44.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 -- 172.21.15.167:0/2560504643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83540f5560 msgr2=0x7f8354109bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:44.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83540f5560 0x7f8354109bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.237 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 -- 172.21.15.167:0/2560504643 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348005040 con 0x7f83540f6800 2022-01-31T19:44:44.238 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8353fff700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 0x7f835410c0b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f834000e7f0 tx=0x7f834000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:44.238 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83400092e0 con 0x7f83540f6800 2022-01-31T19:44:44.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835410c660 con 0x7f83540f6800 2022-01-31T19:44:44.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f834000f040 con 0x7f83540f6800 2022-01-31T19:44:44.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.230+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8340014750 con 0x7f83540f6800 2022-01-31T19:44:44.239 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.231+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835410cbf0 con 0x7f83540f6800 2022-01-31T19:44:44.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.232+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 22) v1 ==== 41566+0+0 (secure 0 0 0) 0x7f834001b030 con 0x7f83540f6800 2022-01-31T19:44:44.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.232+0000 7f83517fa700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f83440344e0 0x7f83440369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:44:44.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.232+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7f8340046220 con 0x7f83540f6800 2022-01-31T19:44:44.240 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.233+0000 7f8358b04700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f83440344e0 0x7f83440369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:44:44.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.232+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83540591e0 con 0x7f83540f6800 2022-01-31T19:44:44.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.233+0000 7f8358b04700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f83440344e0 0x7f83440369a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8348000c00 tx=0x7f8348016040).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:44:44.241 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.237+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7f834001b2c0 con 0x7f83540f6800 2022-01-31T19:44:44.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:44 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:42.498888+0000 mgr.smithi167.aciqpk (mgr.14650) 1 : cephadm [INF] [31/Jan/2022:19:44:42] ENGINE Bus STARTING 2022-01-31T19:44:44.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:44 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:42.616970+0000 mgr.smithi167.aciqpk (mgr.14650) 2 : cephadm [INF] [31/Jan/2022:19:44:42] ENGINE Serving on https://172.21.15.167:7150 2022-01-31T19:44:44.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:44 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:42.617106+0000 mgr.smithi167.aciqpk (mgr.14650) 3 : cephadm [INF] [31/Jan/2022:19:44:42] ENGINE Bus STARTED 2022-01-31T19:44:44.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:44 smithi171 conmon[35325]: cluster 2022-01-31T19:44:43.060781+0000 mon.smithi167 (mon.0) 726 : cluster [DBG] mgrmap e22: smithi167.aciqpk(active, since 1.99775s) 2022-01-31T19:44:44.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:44 smithi171 conmon[35325]: audit 2022-01-31T19:44:43.073220+0000 mgr.smithi167.aciqpk (mgr.14650) 4 : audit [DBG] from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:44.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:44 smithi171 conmon[35325]: cluster 2022-01-31T19:44:43.076199+0000 mgr.smithi167.aciqpk (mgr.14650) 5 : cluster [DBG] pgmap v3: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:44.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:42.498888+0000 mgr.smithi167.aciqpk (mgr.14650) 2022-01-31T19:44:44.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: 1 : cephadm [INF] [31/Jan/2022:19:44:42] ENGINE Bus STARTING 2022-01-31T19:44:44.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:42.616970+0000 mgr.smithi167.aciqpk (mgr.14650) 2 : cephadm [INF] [31/Jan/2022:19:44:42] ENGINE Serving on https://172.21.15.167:7150 2022-01-31T19:44:44.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:42.617106+0000 mgr.smithi167.aciqpk (mgr.14650) 3 : cephadm [INF] [31/Jan/2022:19:44:42] ENGINE Bus STARTED 2022-01-31T19:44:44.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: cluster 2022-01-31T19:44:43.060781+0000 mon.smithi167 (mon.0) 726 : cluster [DBG] mgrmap e22: smithi167.aciqpk(active, since 1.99775s) 2022-01-31T19:44:44.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: audit 2022-01-31T19:44:43.073220+0000 mgr.smithi167.aciqpk (mgr.14650) 4 : audit [DBG] from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:44.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:44 smithi167 conmon[32206]: cluster 2022-01-31T19:44:43.076199+0000 mgr.smithi167.aciqpk (mgr.14650) 5 : cluster [DBG] pgmap v3: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:44.555 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.553+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8354045bb0 con 0x7f83540f6800 2022-01-31T19:44:44.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.553+0000 7f83517fa700 1 -- 172.21.15.167:0/2560504643 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1706 (secure 0 0 0) 0x7f8340010ae0 con 0x7f83540f6800 2022-01-31T19:44:44.556 INFO:teuthology.orchestra.run.smithi167.stdout:e17 2022-01-31T19:44:44.556 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:44:44.557 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:44:44.557 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:44:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:44:44.558 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:44:44.558 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:44:44.558 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 17 2022-01-31T19:44:44.558 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:44:44.559 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:44:44.559 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:44:44.559 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:44:44.559 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:44:44.559 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:44:44.560 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:44:44.560 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:44:44.560 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:44:44.560 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:44:44.561 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:44:44.561 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:44:44.561 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:44:44.561 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:44:44.561 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:44:44.562 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:44:44.562 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:44:44.562 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:44:44.563 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:44:44.563 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:44:44.563 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:44:44.564 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:44:44.564 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:44:44.564 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:44:44.564 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:44:44.565 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:44:44.565 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:44:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:44:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:44:44.567 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f83440344e0 msgr2=0x7f83440369a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:44.567 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f83440344e0 0x7f83440369a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8348000c00 tx=0x7f8348016040).stop 2022-01-31T19:44:44.567 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 msgr2=0x7f835410c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:44:44.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 0x7f835410c0b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f834000e7f0 tx=0x7f834000b040).stop 2022-01-31T19:44:44.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 shutdown_connections 2022-01-31T19:44:44.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f83440344e0 0x7f83440369a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83540f5560 0x7f8354109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.568 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 --2- 172.21.15.167:0/2560504643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83540f6800 0x7f835410c0b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:44:44.569 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 >> 172.21.15.167:0/2560504643 conn(0x7f83540f0ce0 msgr2=0x7f83540fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:44:44.569 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 shutdown_connections 2022-01-31T19:44:44.569 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:44:44.556+0000 7f8359b06700 1 -- 172.21.15.167:0/2560504643 wait complete. 2022-01-31T19:44:44.570 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 17 2022-01-31T19:44:45.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: audit 2022-01-31T19:44:43.477477+0000 2022-01-31T19:44:45.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14650) 6 : audit [DBG] from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:45.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: cluster 2022-01-31T19:44:44.059179+0000 2022-01-31T19:44:45.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: mgr.smithi167.aciqpk (mgr.14650) 7 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:45.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: cluster 2022-01-31T19:44: 2022-01-31T19:44:45.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: 44.081476+0000 mon.smithi167 (mon.0) 727 : 2022-01-31T19:44:45.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:44:45.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: audit 2022-01- 2022-01-31T19:44:45.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: 31T19:44:44.093081+0000 mon.smithi167 (mon 2022-01-31T19:44:45.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: .0) 728 : audit [DBG] from='client.? 172.21.15.167:0/1056107724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:44:45.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: audit 2022-01-31T19:44:45.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: 2022-01-31T19:44:44.554807+0000 2022-01-31T19:44:45.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:45 smithi171 conmon[35325]: mon.smithi167 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.167:0/2560504643' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:44:45.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: audit 2022-01-31T19:44:43. 2022-01-31T19:44:45.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: 477477+0000 mgr.smithi167.aciqpk (mgr.14650) 6 : audit [DBG] from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:44:45.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: cluster 2022-01-31T19:44:44.059179+0000 mgr.smithi167.aciqpk (mgr.14650) 7 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:45.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: cluster 2022-01-31T19:44:44.081476+0000 mon.smithi167 (mon.0) 727 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi167.sgnfie=up:active,1=cephfs.smithi171.dyqqtj=up:active} 2 up:standby-replay 2022-01-31T19:44:45.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: audit 2022 2022-01-31T19:44:45.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: -01-31T19:44:44.093081+0000 mon.smithi167 (mon.0) 728 : audit [DBG] from='client.? 172.21.15.167:0/1056107724' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:44:45.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: 2022-01-31T19:44:45.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: audit 2022- 2022-01-31T19:44:45.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: 01-31T19:44:44 2022-01-31T19:44:45.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: .554807+0000 mon.smithi167 (mon.0 2022-01-31T19:44:45.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:45 smithi167 conmon[32206]: ) 729 : audit [DBG] from='client.? 172.21.15.167:0/2560504643' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:44:47.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:46 smithi171 conmon[35325]: cluster 2022-01-31T19:44:46.059508 2022-01-31T19:44:47.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:46 smithi171 conmon[35325]: +0000 mgr.smithi167.aciqpk (mgr.14650) 8 : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:47.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:46 smithi167 conmon[32206]: cluster 2022-01-31T19:44:46.059508+0000 mgr.smithi167.aciqpk 2022-01-31T19:44:47.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:46 smithi167 conmon[32206]: (mgr.14650) 8 : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:48.538 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:48 smithi171 conmon[35325]: audit 2022-01-31T19:44:47.260751+0000 mon.smithi167 (mon.0) 730 : audit 2022-01-31T19:44:48.539 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:48 smithi171 conmon[35325]: [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:48.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:48 smithi167 conmon[32206]: audit 2022-01-31T19:44:47.260751+0000 mon.smithi167 (mon.0) 730 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:49.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: cluster 2022-01-31T19:44:48 2022-01-31T19:44:49.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: .059896+0000 mgr.smithi167.aciqpk (mgr.14650) 9 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:49.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: audit 2022-01-31T19:44:49.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: 2022-01-31T19:44:48.431583+0000 mon.smithi167 (mon.0) 731 : audit 2022-01-31T19:44:49.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:49.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:49.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: 01-31T19:44:48.984250+0000 2022-01-31T19:44:49.522 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:49 smithi171 conmon[35325]: mon.smithi167 (mon.0) 732 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:49.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:49 smithi167 conmon[32206]: cluster 2022-01-31T19:44:48.059896+0000 2022-01-31T19:44:49.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:49 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14650) 9 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 2.3 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail 2022-01-31T19:44:49.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:49 smithi167 conmon[32206]: audit 2022-01-31T19:44:48.431583+0000 mon.smithi167 (mon.0) 731 : audit [INF] 2022-01-31T19:44:49.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:49 smithi167 conmon[32206]: from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:49.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:49 smithi167 conmon[32206]: audit 2022-01-31T19:44 2022-01-31T19:44:49.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:49 smithi167 conmon[32206]: :48.984250+0000 mon.smithi167 (mon.0) 732 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:50.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:50 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:48.432591+0000 mgr.smithi167.aciqpk (mgr. 2022-01-31T19:44:50.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:50 smithi171 conmon[35325]: 14650) 10 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2022-01-31T19:44:50.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:50 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:48.638450+0000 mgr.smithi167.aciqpk (mgr.14650) 11 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:50.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:50 smithi171 conmon[35325]: audit 2022- 2022-01-31T19:44:50.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:50 smithi171 conmon[35325]: 01-31T19:44:49.768647+0000 mon.smithi167 (mon.0) 733 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:50.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:50 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:48.432591+0000 mgr.smithi167.aciqpk (mgr.14650) 10 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2022-01-31T19:44:50.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:50 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:48.638450+0000 mgr.smithi167.aciqpk (mgr.14650) 11 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:50.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:50 smithi167 conmon[32206]: audit 2022-01-31T19:44:49.768647+0000 mon.smithi167 (mon.0) 733 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:51.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: cluster 2022-01-31T19:44:50. 2022-01-31T19:44:51.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: 060356+0000 mgr.smithi167.aciqpk (mgr.14650) 12 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 1.7 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2022-01-31T19:44:51.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: cluster 2022-01-31T19:44:50. 2022-01-31T19:44:51.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: 503560+0000 mon.smithi167 (mon.0) 734 : cluster [DBG] Standby manager daemon smithi171.asyxnc started 2022-01-31T19:44:51.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: audit 2022-01-31T19:44:51.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: 2022-01-31T19:44:50 2022-01-31T19:44:51.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: .504461+0000 mon.smithi171 (mon. 2022-01-31T19:44:51.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: 1) 68 : audit [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:44:51.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: audit 2022-01-31T 2022-01-31T19:44:51.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: 19:44:50.504891+0000 mon.smithi171 (mon.1) 69 : audit [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:44:51.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:44:51.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: -31T19:44:50.505870+0000 mon.smithi171 ( 2022-01-31T19:44:51.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: mon.1) 70 : audit [DBG] 2022-01-31T19:44:51.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:44:51.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: audit 2022-01-31T19:44:51.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: 2022-01-31T19 2022-01-31T19:44:51.612 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: :44:50.506206+0000 mon.smithi171 2022-01-31T19:44:51.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: (mon.1) 71 : audit 2022-01-31T19:44:51.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:44:51.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: audit 2022-01 2022-01-31T19:44:51.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: -31T19:44:50.743035 2022-01-31T19:44:51.613 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 735 2022-01-31T19:44:51.614 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:51 smithi171 conmon[35325]: : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:51.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: cluster 2022-01-31T19:44:50.060356+0000 mgr.smithi167.aciqpk (mgr.14650) 2022-01-31T19:44:51.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: 12 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 1.7 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2022-01-31T19:44:51.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: cluster 2022-01-31T19:44:50.503560+0000 mon.smithi167 (mon.0) 734 : cluster [DBG] Standby manager daemon smithi171.asyxnc started 2022-01-31T19:44:51.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: audit 2022-01-31T19:44:50.504461+0000 mon.smithi171 (mon.1) 68 : audit [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:44:51.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: audit 2022-01-31T19:44:50.504891+0000 mon.smithi171 (mon.1) 69 : audit [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:44:51.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: audit 2022-01-31T19:44:50.505870+0000 2022-01-31T19:44:51.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: mon.smithi171 (mon.1) 70 : audit [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:44:51.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: audit 2022-01-31T19:44:50.506206+0000 mon.smithi171 (mon.1) 71 : audit [DBG] from='mgr.? 172.21.15.171:0/2809426361' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:44:51.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: audit 2022-01-31T19:44:50.743035+0000 mon.smithi167 ( 2022-01-31T19:44:51.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:51 smithi167 conmon[32206]: mon.0) 735 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:52.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:52 smithi171 conmon[35325]: cluster 2022-01-31T19:44:51.285306+0000 mon.smithi167 (mon.0) 736 : cluster 2022-01-31T19:44:52.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:52 smithi171 conmon[35325]: [DBG] mgrmap e23: smithi167.aciqpk(active, since 10s), standbys: smithi171.asyxnc 2022-01-31T19:44:52.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:52 smithi171 conmon[35325]: audit 2022-01-31T19:44:51.288071+0000 mon.smithi167 (mon.0) 737 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi171.asyxnc", "id": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:44:52.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:52 smithi167 conmon[32206]: cluster 2022-01-31T19:44:51.285306+0000 mon.smithi167 (mon.0) 736 : cluster 2022-01-31T19:44:52.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:52 smithi167 conmon[32206]: [DBG] mgrmap e23: smithi167.aciqpk(active, since 10s), standbys: smithi171.asyxnc 2022-01-31T19:44:52.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:52 smithi167 conmon[32206]: audit 2022-01-31T19:44:51.288071+0000 mon.smithi167 (mon.0) 737 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr metadata", "who": "smithi171.asyxnc", "id": "smithi171.asyxnc"}]: dispatch 2022-01-31T19:44:53.406 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:53 smithi167 conmon[32206]: cluster 2022-01-31T19:44:52.060676+0000 mgr.smithi167.aciqpk (mgr.14650) 13 : cluster [DBG] 2022-01-31T19:44:53.407 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:53 smithi167 conmon[32206]: pgmap v8: 65 pgs: 65 active+clean; 1.7 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2022-01-31T19:44:53.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:53 smithi171 conmon[35325]: cluster 2022-01-31T19:44:52.060676+0000 mgr.smithi167.aciqpk 2022-01-31T19:44:53.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:53 smithi171 conmon[35325]: (mgr.14650) 13 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 1.7 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2022-01-31T19:44:55.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:55 smithi167 conmon[32206]: cluster 2022-01-31T19:44:54.061127+0000 mgr.smithi167.aciqpk (mgr.14650) 14 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 792 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 335 op/s 2022-01-31T19:44:55.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:55 smithi171 conmon[35325]: cluster 2022-01-31T19:44:54.061127+0000 mgr.smithi167.aciqpk 2022-01-31T19:44:55.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:55 smithi171 conmon[35325]: (mgr.14650) 14 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 792 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 335 op/s 2022-01-31T19:44:56.180 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-01-31T19:44:56.181 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2022-01-31T19:44:56.519 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:56 smithi171 conmon[35325]: audit 2022-01-31T19:44:55.339052+0000 mon.smithi167 (mon.0) 738 : audit 2022-01-31T19:44:56.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:56 smithi171 conmon[35325]: [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:56.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:56 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:55.340182+0000 mgr.smithi167.aciqpk (mgr.14650) 15 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:44:56.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:56 smithi171 conmon[35325]: audit 2022-01-31T19:44:55.732831+0000 mon.smithi167 (mon.0) 739 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:56.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:56 smithi171 conmon[35325]: audit 2022-01-31T19:44:55.736264+0000 mon.smithi167 (mon.0) 740 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:56.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:56 smithi171 conmon[35325]: audit 2022-01-31T19:44:55.740969+0000 mon.smithi167 (mon.0) 741 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:56.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:56 smithi167 conmon[32206]: audit 2022-01-31T19:44:55.339052+0000 mon.smithi167 (mon. 2022-01-31T19:44:56.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:56 smithi167 conmon[32206]: 0) 738 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:56.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:56 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:55.340182+0000 mgr.smithi167.aciqpk (mgr.14650) 15 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.conf 2022-01-31T19:44:56.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:56 smithi167 conmon[32206]: audit 2022-01-31T19:44:55.732831+0000 mon.smithi167 (mon.0) 739 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:56.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:56 smithi167 conmon[32206]: audit 2022-01-31T19:44:55.736264+0000 mon.smithi167 (mon.0) 740 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:44:56.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:56 smithi167 conmon[32206]: audit 2022-01-31T19:44:55.740969+0000 mon.smithi167 (mon.0) 741 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:44:57.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:57 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:55.529346+0000 mgr.smithi167.aciqpk (mgr.14650 2022-01-31T19:44:57.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:57 smithi171 conmon[35325]: ) 16 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:57.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:57 smithi171 conmon[35325]: cluster 2022-01-31T19:44:56.061548+0000 mgr.smithi167.aciqpk (mgr.14650) 17 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 792 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 306 op/s 2022-01-31T19:44:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:57 smithi167 conmon[32206]: cephadm 2022-01-31T19:44:55.529346+0000 2022-01-31T19:44:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:57 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14650) 16 : cephadm [INF] Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2022-01-31T19:44:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:57 smithi167 conmon[32206]: cluster 2022-01-31T19:44:56.061548+0000 mgr.smithi167.aciqpk (mgr.14650) 17 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 792 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 306 op/s 2022-01-31T19:44:58.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:58 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:57.401697+0000 mgr.smithi167.aciqpk (mgr.14650) 18 : cephadm [INF] Upgrade: Updating mgr.smithi171.asyxnc 2022-01-31T19:44:58.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:58 smithi171 conmon[35325]: audit 2022-01-31T19:44:57.402186+0000 mon.smithi167 (mon.0) 742 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:44:58.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:58 smithi171 conmon[35325]: audit 2022-01-31T19:44:57.403106+0000 mon.smithi167 (mon.0) 743 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:44:58.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:58 smithi171 conmon[35325]: audit 2022-01-31T19:44:57.403912+0000 mon.smithi167 (mon.0) 744 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:58.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:58 smithi171 conmon[35325]: cephadm 2022-01-31T19:44:57.404645+0000 mgr.smithi167.aciqpk (mgr.14650) 19 : cephadm [INF] Deploying daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:44:58.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: cephadm 2022-01-31T19:44 2022-01-31T19:44:58.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: :57.401697+0000 mgr.smithi167.aciqpk (mgr.14650) 18 : cephadm [INF] Upgrade: Updating mgr.smithi171.asyxnc 2022-01-31T19:44:58.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: audit 2022-01-31T19:44:57.402186+0000 mon.smithi167 (mon. 2022-01-31T19:44:58.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: 0) 742 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi171.asyxnc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-01-31T19:44:58.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: audit 2022-01-31T19 2022-01-31T19:44:58.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: :44:57.403106+0000 mon.smithi167 (mon.0) 743 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "mgr services"}]: dispatch 2022-01-31T19:44:58.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: audit 2022-01-31 2022-01-31T19:44:58.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: T19:44:57.403912+0000 mon.smithi167 (mon. 2022-01-31T19:44:58.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: 0) 744 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:44:58.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: cephadm 2022-01-31T19:44: 2022-01-31T19:44:58.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: 57.404645+0000 mgr.smithi167.aciqpk (mgr.14650) 19 : cephadm 2022-01-31T19:44:58.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:58 smithi167 conmon[32206]: [INF] Deploying daemon mgr.smithi171.asyxnc on smithi171 2022-01-31T19:44:59.395 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:59 smithi171 conmon[35325]: cluster 2022-01-31T19:44:58. 2022-01-31T19:44:59.395 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:44:59 smithi171 conmon[35325]: 061876+0000 mgr.smithi167.aciqpk (mgr.14650) 20 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 792 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 306 op/s 2022-01-31T19:44:59.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:59 smithi167 conmon[32206]: cluster 2022-01-31T19:44:58.061876+0000 mgr.smithi167.aciqpk (mgr.14650) 20 : cluster 2022-01-31T19:44:59.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:44:59 smithi167 conmon[32206]: [DBG] pgmap v11: 65 pgs: 65 active+clean; 792 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 306 op/s 2022-01-31T19:45:01.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: cluster 2022-01-31T19:45:00. 2022-01-31T19:45:01.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: 062830+0000 mgr.smithi167.aciqpk (mgr.14650) 21 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 415 op/s 2022-01-31T19:45:01.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: audit 2022-01-31T19:45:00.269539+0000 mon.smithi167 (mon.0) 745 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:01.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: audit 2022-01-31T19:45:00.270438+0000 mon.smithi167 (mon.0) 746 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:45:01.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: audit 2022-01-31T19:45:00.271711 2022-01-31T19:45:01.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: +0000 mon.smithi167 (mon.0) 747 : audit 2022-01-31T19:45:01.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:45:01.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: audit 2022-01-31T19:45:01.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: 2022-01-31T 2022-01-31T19:45:01.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: 19:45:00.272369+0000 2022-01-31T19:45:01.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: mon.smithi167 (mon.0) 748 : audit 2022-01-31T19:45:01.362 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:01 smithi171 conmon[35325]: [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:45:01.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: cluster 2022-01-31T19:45:00.062830+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:45:01.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: .14650) 21 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 415 op/s 2022-01-31T19:45:01.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: audit 2022-01-31T19:45:00.269539+0000 mon.smithi167 (mon.0) 745 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:01.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: audit 2022-01-31T19:45:00.270438+0000 mon.smithi167 (mon.0) 746 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:45:01.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: audit 2022-01-31T19:45:00.271711+0000 mon.smithi167 (mon.0) 747 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:45:01.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: audit 2022-01-31T19:45:00.272369+0000 mon.smithi167 ( 2022-01-31T19:45:01.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:01 smithi167 conmon[32206]: mon.0) 748 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-01-31T19:45:03.277 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:03 smithi171 conmon[35325]: cluster 2022-01-31T19:45:02.063213+0000 mgr.smithi167.aciqpk 2022-01-31T19:45:03.277 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:03 smithi171 conmon[35325]: (mgr.14650) 22 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2022-01-31T19:45:03.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:03 smithi167 conmon[32206]: cluster 2022-01-31T19:45:02.063213+0000 2022-01-31T19:45:03.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:03 smithi167 conmon[32206]: mgr.smithi167.aciqpk (mgr.14650) 22 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 295 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 268 op/s 2022-01-31T19:45:05.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:05 smithi171 conmon[35325]: cluster 2022-01-31T19:45:04. 2022-01-31T19:45:05.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:05 smithi171 conmon[35325]: 063815+0000 mgr.smithi167.aciqpk (mgr.14650) 23 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 299 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 486 op/s 2022-01-31T19:45:05.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:05 smithi167 conmon[32206]: cluster 2022-01-31T19:45:04.063815+0000 mgr.smithi167.aciqpk ( 2022-01-31T19:45:05.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:05 smithi167 conmon[32206]: mgr.14650) 23 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 299 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 486 op/s 2022-01-31T19:45:07.001 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:06 smithi171 conmon[35325]: cluster 2022-01-31T19:45:06.064158+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:45:07.001 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:06 smithi171 conmon[35325]: .14650) 24 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 299 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 326 op/s 2022-01-31T19:45:07.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:06 smithi167 conmon[32206]: cluster 2022-01-31T19:45:06.064158+0000 mgr.smithi167.aciqpk (mgr.14650) 24 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 299 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 326 op/s 2022-01-31T19:45:09.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: audit 2022-01-31T19:45:08.035815+0000 mon.smithi167 (mon.0 2022-01-31T19:45:09.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: ) 749 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:09.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: cluster 2022-01-31T19:45:08.064598+0000 mgr.smithi167.aciqpk (mgr.14650) 25 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 299 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 326 op/s 2022-01-31T19:45:09.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: cluster 2022-01-31T19:45:08.537046+0000 mon.smithi167 (mon.0) 750 : cluster [DBG] Standby manager daemon smithi171.asyxnc restarted 2022-01-31T19:45:09.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: cluster 2022-01-31T19:45:08.537169+0000 mon.smithi167 (mon.0) 751 : cluster [DBG] Standby manager daemon smithi171.asyxnc started 2022-01-31T19:45:09.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: audit 2022-01-31T19:45:08.539588+0000 mon.smithi171 (mon.1) 72 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:45:09.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: audit 2022-01-31T19:45:08.540283+0000 mon.smithi171 (mon.1 2022-01-31T19:45:09.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: ) 73 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:45:09.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: audit 2022-01-31T19:45:08.541593+0000 mon.smithi171 (mon.1) 74 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:45:09.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: audit 2022-01-31T19:45:08.542068+0000 mon.smithi171 (mon. 2022-01-31T19:45:09.361 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:09 smithi171 conmon[35325]: 1) 75 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:45:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: audit 2022-01-31T19:45:08.035815+0000 mon.smithi167 (mon.0) 749 : audit [INF] 2022-01-31T19:45:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: cluster 2022-01-31T19:45:08.064598+0000 mgr.smithi167.aciqpk (mgr.14650) 25 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 299 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 326 op/s 2022-01-31T19:45:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: cluster 2022-01-31T19:45:08.537046+0000 mon.smithi167 (mon.0) 750 : cluster [DBG] Standby manager daemon smithi171.asyxnc restarted 2022-01-31T19:45:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: cluster 2022-01-31T19:45:08.537169+0000 mon.smithi167 (mon.0) 751 : cluster [DBG] Standby manager daemon smithi171.asyxnc started 2022-01-31T19:45:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: audit 2022-01-31T19:45:08.539588+0000 mon.smithi171 (mon.1) 72 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/crt"}]: dispatch 2022-01-31T19:45:09.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: audit 2022-01-31T19:45:08.540283+0000 mon.smithi171 (mon.1) 73 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-01-31T19:45:09.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: audit 2022 2022-01-31T19:45:09.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: -01-31T19:45:08.541593+0000 mon.smithi171 (mon.1) 74 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi171.asyxnc/key"}]: dispatch 2022-01-31T19:45:09.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: audit 2022-01-31T19:45:08.542068+0000 mon.smithi171 (mon.1 2022-01-31T19:45:09.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:09 smithi167 conmon[32206]: ) 75 : audit [DBG] from='mgr.? 172.21.15.171:0/422623214' entity='mgr.smithi171.asyxnc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-01-31T19:45:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:10 smithi171 conmon[35325]: cluster 2022-01-31T19:45:09.044188+0000 mon.smithi167 (mon 2022-01-31T19:45:10.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:10 smithi171 conmon[35325]: .0) 752 : cluster [DBG] mgrmap e24: smithi167.aciqpk(active, since 27s), standbys: smithi171.asyxnc 2022-01-31T19:45:10.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:10 smithi171 conmon[35325]: audit 2022-01-31T19:45:09.096898+0000 mon.smithi167 (mon.0) 753 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:10.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:10 smithi167 conmon[32206]: cluster 2022-01-31T19:45:09.044188+0000 mon.smithi167 (mon.0) 752 : cluster [DBG] mgrmap e24: smithi167.aciqpk(active, since 27s), standbys: smithi171.asyxnc 2022-01-31T19:45:10.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:10 smithi167 conmon[32206]: audit 2022-01-31T19:45:09.096898+0000 mon.smithi167 (mon.0) 753 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:11.053 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:11 smithi171 conmon[35325]: cluster 2022-01-31T19:45:10.065034+0000 mgr.smithi167.aciqpk 2022-01-31T19:45:11.053 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:11 smithi171 conmon[35325]: (mgr.14650) 26 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 303 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 512 op/s 2022-01-31T19:45:11.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:11 smithi167 conmon[32206]: cluster 2022-01-31T19:45:10.065034+0000 mgr.smithi167.aciqpk (mgr.14650) 2022-01-31T19:45:11.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:11 smithi167 conmon[32206]: 26 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 303 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 512 op/s 2022-01-31T19:45:13.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: cluster 2022-01-31T19:45:12.065419+0000 mgr.smithi167.aciqpk (mgr.14650) 27 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 303 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 403 op/s 2022-01-31T19:45:13.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.114467 2022-01-31T19:45:13.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: +0000 mon.smithi167 (mon.0) 754 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:13.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.119338+0000 mon.smithi167 (mon.0) 755 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:13.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.125621+0000 mon.smithi167 (mon.0) 756 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:45:13.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.127940+0000 mon.smithi167 (mon.0) 757 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:45:13.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: cephadm 2022-01-31T19:45:12.128638+0000 mgr.smithi167.aciqpk (mgr.14650) 28 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-01-31T19:45:13.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.132258+0000 mon.smithi167 (mon.0) 758 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:13.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.133079+0000 mon.smithi167 (mon.0) 759 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi167.aciqpk"}]: dispatch 2022-01-31T19:45:13.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.139678+0000 mon.smithi167 (mon.0) 760 : audit [INF] 2022-01-31T19:45:13.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi167.aciqpk"}]': finished 2022-01-31T19:45:13.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:12.140426+0000 mon.smithi167 (mon.0) 761 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi171.asyxnc"}]: dispatch 2022-01-31T19:45:13.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:13.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: 2022-01-31T19:45:12.144783+0000 mon.smithi167 (mon.0) 762 : audit [INF] 2022-01-31T19:45:13.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi171.asyxnc"}]': finished 2022-01-31T19:45:13.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: audit 2022-01-31T19:45:13.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: 2022-01-31T19:45:12.145624+0000 mon.smithi167 2022-01-31T19:45:13.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:13 smithi167 conmon[32206]: (mon.0) 763 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-01-31T19:45:13.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: cluster 2022-01-31T19:45:12.065419+0000 mgr.smithi167.aciqpk (mgr.14650) 27 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 303 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 403 op/s 2022-01-31T19:45:13.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.114467+0000 mon.smithi167 (mon.0) 754 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:13.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.119338+0000 mon.smithi167 (mon.0) 755 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:13.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.125621+0000 mon.smithi167 (mon.0) 756 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-01-31T19:45:13.609 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.127940+0000 mon.smithi167 (mon.0) 757 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:45:13.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: cephadm 2022-01-31T19:45:12.128638+0000 mgr.smithi167.aciqpk (mgr.14650) 28 : cephadm [INF] Upgrade: Setting container_image for all mgr 2022-01-31T19:45:13.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.132258+0000 mon.smithi167 (mon.0) 758 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:13.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.133079+0000 mon.smithi167 (mon.0) 759 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi167.aciqpk"}]: dispatch 2022-01-31T19:45:13.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.139678+0000 mon.smithi167 (mon.0) 760 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi167.aciqpk"}]': finished 2022-01-31T19:45:13.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.140426+0000 mon.smithi167 (mon.0) 761 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi171.asyxnc"}]: dispatch 2022-01-31T19:45:13.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.144783+0000 mon.smithi167 (mon.0) 762 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi171.asyxnc"}]': finished 2022-01-31T19:45:13.611 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:13 smithi171 conmon[35325]: audit 2022-01-31T19:45:12.145624+0000 mon.smithi167 (mon.0) 763 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "quorum_status"}]: dispatch 2022-01-31T19:45:14.692 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 -- 172.21.15.167:0/1044980489 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 msgr2=0x7f75e80f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:14.693 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/1044980489 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 0x7f75e80f82e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f75e4004660 tx=0x7f75e4009b30).stop 2022-01-31T19:45:14.693 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 -- 172.21.15.167:0/1044980489 shutdown_connections 2022-01-31T19:45:14.693 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/1044980489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:14.693 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/1044980489 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 0x7f75e80f82e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:14.694 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 -- 172.21.15.167:0/1044980489 >> 172.21.15.167:0/1044980489 conn(0x7f75e80f1f70 msgr2=0x7f75e80f4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:14.694 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 -- 172.21.15.167:0/1044980489 shutdown_connections 2022-01-31T19:45:14.694 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 -- 172.21.15.167:0/1044980489 wait complete. 2022-01-31T19:45:14.694 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.690+0000 7f75ef4ab700 1 Processor -- start 2022-01-31T19:45:14.695 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ef4ab700 1 -- start start 2022-01-31T19:45:14.695 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ef4ab700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:14.695 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ef4ab700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 0x7f75e8069f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:14.695 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ef4ab700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e806a460 con 0x7f75e80f7ec0 2022-01-31T19:45:14.696 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ef4ab700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e806a5a0 con 0x7f75e80f64d0 2022-01-31T19:45:14.696 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ed247700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:14.696 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ed247700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40890/0 (socket says 172.21.15.167:40890) 2022-01-31T19:45:14.696 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75ed247700 1 -- 172.21.15.167:0/693128384 learned_addr learned my addr 172.21.15.167:0/693128384 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:45:14.696 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.691+0000 7f75eca46700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 0x7f75e8069f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:14.697 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ed247700 1 -- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 msgr2=0x7f75e8069f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:14.697 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ed247700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 0x7f75e8069f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:14.697 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ed247700 1 -- 172.21.15.167:0/693128384 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4005040 con 0x7f75e80f64d0 2022-01-31T19:45:14.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ed247700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e806ba80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f75e4014040 tx=0x7f75e400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:14.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e4019070 con 0x7f75e80f64d0 2022-01-31T19:45:14.699 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e806a820 con 0x7f75e80f64d0 2022-01-31T19:45:14.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f75e4007e70 con 0x7f75e80f64d0 2022-01-31T19:45:14.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e4014d10 con 0x7f75e80f64d0 2022-01-31T19:45:14.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.692+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e806ad80 con 0x7f75e80f64d0 2022-01-31T19:45:14.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.693+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e8045bb0 con 0x7f75e80f64d0 2022-01-31T19:45:14.700 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.694+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 24) v1 ==== 82737+0+0 (secure 0 0 0) 0x7f75e4025070 con 0x7f75e80f64d0 2022-01-31T19:45:14.701 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.694+0000 7f75ddffb700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f75d4064d20 0x7f75d40671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:14.701 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.694+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7f75e4082630 con 0x7f75e80f64d0 2022-01-31T19:45:14.701 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.694+0000 7f75eca46700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f75d4064d20 0x7f75d40671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:45:14.701 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.694+0000 7f75eca46700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f75d4064d20 0x7f75d40671e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f75d8009fd0 tx=0x7f75d800b040).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:14.702 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.698+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7f75e4050a90 con 0x7f75e80f64d0 2022-01-31T19:45:14.952 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.951+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 --> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75e80ff1f0 con 0x7f75d4064d20 2022-01-31T19:45:14.952 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: cephadm 2022-01-31T19:45:13.665721+0000 mgr.smithi167.aciqpk (mgr.14650) 29 : cephadm [INF] Upgrade: Updating mon.smithi167 2022-01-31T19:45:14.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: audit 2022-01-31T19:45:13.670511+0000 mon.smithi167 (mon.0) 764 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:14.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: audit 2022-01-31T19:45:13.671583+0000 mon.smithi167 (mon.0) 765 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:45:14.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: audit 2022-01-31T19:45:13.672170+0000 mon.smithi167 (mon.0) 766 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-01-31T19:45:14.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: audit 2022-01-31T19:45:13.672812+0000 mon.smithi167 (mon.0) 767 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:45:14.954 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: cephadm 2022-01-31T19:45:13.673572+0000 mgr.smithi167.aciqpk (mgr.14650) 30 : cephadm [INF] Deploying daemon mon.smithi167 on smithi167 2022-01-31T19:45:14.954 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:14 smithi167 conmon[32206]: cluster 2022-01-31T19:45:14.065914+0000 mgr.smithi167.aciqpk (mgr.14650) 31 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 642 op/s 2022-01-31T19:45:14.956 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.953+0000 7f75ddffb700 1 -- 172.21.15.167:0/693128384 <== mgr.14650 v2:172.21.15.167:6800/2258949773 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7f75e80ff1f0 con 0x7f75d4064d20 2022-01-31T19:45:14.957 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.956+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f75d4064d20 msgr2=0x7f75d40671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:14.957 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.956+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f75d4064d20 0x7f75d40671e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f75d8009fd0 tx=0x7f75d800b040).stop 2022-01-31T19:45:14.958 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.956+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 msgr2=0x7f75e806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:14.958 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.956+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e806ba80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f75e4014040 tx=0x7f75e400b040).stop 2022-01-31T19:45:14.958 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 shutdown_connections 2022-01-31T19:45:14.959 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f75d4064d20 0x7f75d40671e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:14.959 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f75e80f64d0 0x7f75e806ba80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:14.959 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 --2- 172.21.15.167:0/693128384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75e80f7ec0 0x7f75e8069f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:14.959 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 >> 172.21.15.167:0/693128384 conn(0x7f75e80f1f70 msgr2=0x7f75e80fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:14.960 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 shutdown_connections 2022-01-31T19:45:14.960 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:14.958+0000 7f75ef4ab700 1 -- 172.21.15.167:0/693128384 wait complete. 2022-01-31T19:45:14.978 INFO:teuthology.orchestra.run.smithi167.stdout:true 2022-01-31T19:45:15.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: cephadm 2022-01-31T19:45:13.665721+0000 mgr.smithi167.aciqpk (mgr 2022-01-31T19:45:15.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: .14650) 29 : cephadm [INF] Upgrade: Updating mon.smithi167 2022-01-31T19:45:15.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: audit 2022-01-31T19:45:13.670511+0000 mon.smithi167 (mon.0) 764 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' 2022-01-31T19:45:15.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: audit 2022-01-31T19:45:13.671583+0000 mon.smithi167 (mon.0) 765 : audit [INF] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-01-31T19:45:15.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: audit 2022-01-31T19:45:13.672170+0000 mon.smithi167 (mon.0) 766 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-01-31T19:45:15.110 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: audit 2022-01-31T19:45:13.672812+0000 mon.smithi167 (mon.0) 767 : audit [DBG] from='mgr.14650 172.21.15.167:0/972846124' entity='mgr.smithi167.aciqpk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-01-31T19:45:15.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: cephadm 2022-01-31T19:45:13.673572+0000 mgr.smithi167.aciqpk (mgr.14650) 30 : cephadm [INF] Deploying daemon mon.smithi167 on smithi167 2022-01-31T19:45:15.111 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:14 smithi171 conmon[35325]: cluster 2022-01-31T19:45:14.065914+0000 mgr.smithi167.aciqpk (mgr.14650) 31 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 642 op/s 2022-01-31T19:45:15.117 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.108+0000 7efc6d17b700 1 -- 172.21.15.167:0/298638840 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f5560 msgr2=0x7efc680f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.118 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.108+0000 7efc6d17b700 1 --2- 172.21.15.167:0/298638840 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f5560 0x7efc680f5980 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7efc58004660 tx=0x7efc58009b30).stop 2022-01-31T19:45:15.118 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.108+0000 7efc6d17b700 1 -- 172.21.15.167:0/298638840 shutdown_connections 2022-01-31T19:45:15.119 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.108+0000 7efc6d17b700 1 --2- 172.21.15.167:0/298638840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f6800 0x7efc680f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.119 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.108+0000 7efc6d17b700 1 --2- 172.21.15.167:0/298638840 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f5560 0x7efc680f5980 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.119 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.108+0000 7efc6d17b700 1 -- 172.21.15.167:0/298638840 >> 172.21.15.167:0/298638840 conn(0x7efc680f0ce0 msgr2=0x7efc680f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:15.120 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.109+0000 7efc6d17b700 1 -- 172.21.15.167:0/298638840 shutdown_connections 2022-01-31T19:45:15.120 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.109+0000 7efc6d17b700 1 -- 172.21.15.167:0/298638840 wait complete. 2022-01-31T19:45:15.121 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc6d17b700 1 Processor -- start 2022-01-31T19:45:15.121 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc6d17b700 1 -- start start 2022-01-31T19:45:15.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc6d17b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 0x7efc68109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:15.122 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc6d17b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f6800 0x7efc6810c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:15.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc6d17b700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680604c0 con 0x7efc680f6800 2022-01-31T19:45:15.123 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc6d17b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc68060600 con 0x7efc680f5560 2022-01-31T19:45:15.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc67fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 0x7efc68109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:15.124 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc67fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 0x7efc68109b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40904/0 (socket says 172.21.15.167:40904) 2022-01-31T19:45:15.125 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc67fff700 1 -- 172.21.15.167:0/2973232021 learned_addr learned my addr 172.21.15.167:0/2973232021 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:45:15.125 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.110+0000 7efc677fe700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f6800 0x7efc6810c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:15.125 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc67fff700 1 -- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f6800 msgr2=0x7efc6810c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc67fff700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f6800 0x7efc6810c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.126 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc67fff700 1 -- 172.21.15.167:0/2973232021 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58005040 con 0x7efc680f5560 2022-01-31T19:45:15.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc677fe700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f6800 0x7efc6810c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-01-31T19:45:15.127 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc67fff700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 0x7efc68109b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7efc58014040 tx=0x7efc5800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:15.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc58019070 con 0x7efc680f5560 2022-01-31T19:45:15.128 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7efc58007e70 con 0x7efc680f5560 2022-01-31T19:45:15.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc58014d10 con 0x7efc680f5560 2022-01-31T19:45:15.129 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6810c5e0 con 0x7efc680f5560 2022-01-31T19:45:15.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.111+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6810cb40 con 0x7efc680f5560 2022-01-31T19:45:15.130 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.112+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc68045bb0 con 0x7efc680f5560 2022-01-31T19:45:15.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.113+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 24) v1 ==== 82737+0+0 (secure 0 0 0) 0x7efc58025070 con 0x7efc680f5560 2022-01-31T19:45:15.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.113+0000 7efc64ff9700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc50064d20 0x7efc500671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:15.131 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.113+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7efc580824d0 con 0x7efc680f5560 2022-01-31T19:45:15.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.113+0000 7efc677fe700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc50064d20 0x7efc500671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:45:15.132 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.114+0000 7efc677fe700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc50064d20 0x7efc500671e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7efc5c009fd0 tx=0x7efc5c00b040).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:15.133 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.118+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7efc580509b0 con 0x7efc680f5560 2022-01-31T19:45:15.361 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.360+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 --> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efc680fde60 con 0x7efc50064d20 2022-01-31T19:45:15.378 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.377+0000 7efc64ff9700 1 -- 172.21.15.167:0/2973232021 <== mgr.14650 v2:172.21.15.167:6800/2258949773 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7efc680fde60 con 0x7efc50064d20 2022-01-31T19:45:15.380 INFO:teuthology.orchestra.run.smithi167.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-01-31T19:45:15.381 INFO:teuthology.orchestra.run.smithi167.stdout:alertmanager.smithi167 smithi167 *:9093,9094 running (5m) 25s ago 7m 22.1M - 0.20.0 0881eb8f169f 3220d7bd944c 2022-01-31T19:45:15.381 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi167 smithi167 running (7m) 25s ago 6m 7184k - 16.2.5 46cf6318c64e 04c49164eb24 2022-01-31T19:45:15.381 INFO:teuthology.orchestra.run.smithi167.stdout:crash.smithi171 smithi171 running (6m) 7s ago 6m 7151k - 16.2.5 46cf6318c64e 9eec51245322 2022-01-31T19:45:15.381 INFO:teuthology.orchestra.run.smithi167.stdout:grafana.smithi167 smithi167 *:3000 running (5m) 25s ago 6m 38.3M - 6.7.4 557c83e11646 a102a3bc6222 2022-01-31T19:45:15.382 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.rdkazj smithi167 running (3m) 25s ago 3m 397M - 16.2.5 46cf6318c64e 29f596ef1666 2022-01-31T19:45:15.382 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi167.sgnfie smithi167 running (3m) 25s ago 3m 3126M - 16.2.5 46cf6318c64e 8c462c58dec4 2022-01-31T19:45:15.382 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.dyqqtj smithi171 running (3m) 7s ago 3m 2652M - 16.2.5 46cf6318c64e 5e84b5044af1 2022-01-31T19:45:15.382 INFO:teuthology.orchestra.run.smithi167.stdout:mds.cephfs.smithi171.erzfea smithi171 running (3m) 7s ago 3m 358M - 16.2.5 46cf6318c64e 802065eafb38 2022-01-31T19:45:15.383 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi167.aciqpk smithi167 *:8443 running (46s) 25s ago 8m 440M - 17.0.0-10469-g29e1fc17 e02b80923a35 493415fe7392 2022-01-31T19:45:15.383 INFO:teuthology.orchestra.run.smithi167.stdout:mgr.smithi171.asyxnc smithi171 *:8443 running (15s) 7s ago 6m 49.9M - 17.0.0-10469-g29e1fc17 e02b80923a35 e477299653f6 2022-01-31T19:45:15.383 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi167 smithi167 running (8m) 25s ago 8m 100M 2048M 16.2.5 46cf6318c64e 0ebba348055e 2022-01-31T19:45:15.383 INFO:teuthology.orchestra.run.smithi167.stdout:mon.smithi171 smithi171 running (6m) 7s ago 6m 87.9M 2048M 16.2.5 46cf6318c64e 8f865af70de6 2022-01-31T19:45:15.384 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi167 smithi167 *:9100 running (6m) 25s ago 6m 18.2M - 0.18.1 e5a616e4b9cf fe3bfffb6eff 2022-01-31T19:45:15.384 INFO:teuthology.orchestra.run.smithi167.stdout:node-exporter.smithi171 smithi171 *:9100 running (6m) 7s ago 6m 18.2M - 0.18.1 e5a616e4b9cf b56eace9928e 2022-01-31T19:45:15.384 INFO:teuthology.orchestra.run.smithi167.stdout:osd.0 smithi167 running (5m) 25s ago 5m 664M 1584M 16.2.5 46cf6318c64e e1b24b01735f 2022-01-31T19:45:15.384 INFO:teuthology.orchestra.run.smithi167.stdout:osd.1 smithi167 running (5m) 25s ago 5m 736M 1584M 16.2.5 46cf6318c64e 760db78b0452 2022-01-31T19:45:15.385 INFO:teuthology.orchestra.run.smithi167.stdout:osd.2 smithi167 running (5m) 25s ago 5m 596M 1584M 16.2.5 46cf6318c64e cbde33c1a75b 2022-01-31T19:45:15.385 INFO:teuthology.orchestra.run.smithi167.stdout:osd.3 smithi171 running (5m) 7s ago 5m 910M 2608M 16.2.5 46cf6318c64e d71e404ff269 2022-01-31T19:45:15.386 INFO:teuthology.orchestra.run.smithi167.stdout:osd.4 smithi171 running (4m) 7s ago 4m 715M 2608M 16.2.5 46cf6318c64e 76a3ded6b06a 2022-01-31T19:45:15.386 INFO:teuthology.orchestra.run.smithi167.stdout:osd.5 smithi171 running (4m) 7s ago 4m 844M 2608M 16.2.5 46cf6318c64e 02be289e7e50 2022-01-31T19:45:15.386 INFO:teuthology.orchestra.run.smithi167.stdout:prometheus.smithi167 smithi167 *:9095 running (5m) 25s ago 6m 48.1M - 2.18.1 de242295e225 cd114efaf2d6 2022-01-31T19:45:15.387 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc50064d20 msgr2=0x7efc500671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.387 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc50064d20 0x7efc500671e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7efc5c009fd0 tx=0x7efc5c00b040).stop 2022-01-31T19:45:15.387 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 msgr2=0x7efc68109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.387 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 0x7efc68109b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7efc58014040 tx=0x7efc5800b040).stop 2022-01-31T19:45:15.388 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 shutdown_connections 2022-01-31T19:45:15.388 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7efc50064d20 0x7efc500671e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.388 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc680f5560 0x7efc68109b90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.388 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 --2- 172.21.15.167:0/2973232021 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc680f6800 0x7efc6810c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.389 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 >> 172.21.15.167:0/2973232021 conn(0x7efc680f0ce0 msgr2=0x7efc680fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:15.389 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 shutdown_connections 2022-01-31T19:45:15.389 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.381+0000 7efc6d17b700 1 -- 172.21.15.167:0/2973232021 wait complete. 2022-01-31T19:45:15.542 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 -- 172.21.15.167:0/3621913088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f4d60 msgr2=0x7fb2740f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.542 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 --2- 172.21.15.167:0/3621913088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f4d60 0x7fb2740f5180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fb26c0099e0 tx=0x7fb26c004d10).stop 2022-01-31T19:45:15.543 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 -- 172.21.15.167:0/3621913088 shutdown_connections 2022-01-31T19:45:15.543 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 --2- 172.21.15.167:0/3621913088 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f5f60 0x7fb2740f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-01-31T19:45:15.543 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 --2- 172.21.15.167:0/3621913088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f4d60 0x7fb2740f5180 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.543 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 -- 172.21.15.167:0/3621913088 >> 172.21.15.167:0/3621913088 conn(0x7fb2740f04e0 msgr2=0x7fb2740f2900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:15.544 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 -- 172.21.15.167:0/3621913088 shutdown_connections 2022-01-31T19:45:15.544 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.541+0000 7fb27a912700 1 -- 172.21.15.167:0/3621913088 wait complete. 2022-01-31T19:45:15.545 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb27a912700 1 Processor -- start 2022-01-31T19:45:15.545 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb27a912700 1 -- start start 2022-01-31T19:45:15.545 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb27a912700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f4d60 0x7fb27410d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:15.545 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb27a912700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f5f60 0x7fb27410db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:15.546 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb27a912700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27410e1a0 con 0x7fb2740f5f60 2022-01-31T19:45:15.546 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb27a912700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27410ea50 con 0x7fb2740f4d60 2022-01-31T19:45:15.546 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb279910700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f4d60 0x7fb27410d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:15.546 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb279910700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f4d60 0x7fb27410d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.167:40916/0 (socket says 172.21.15.167:40916) 2022-01-31T19:45:15.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.542+0000 7fb279910700 1 -- 172.21.15.167:0/98892636 learned_addr learned my addr 172.21.15.167:0/98892636 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:45:15.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.543+0000 7fb27910f700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f5f60 0x7fb27410db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:15.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.543+0000 7fb27910f700 1 -- 172.21.15.167:0/98892636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f4d60 msgr2=0x7fb27410d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.547 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.543+0000 7fb27910f700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f4d60 0x7fb27410d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.543+0000 7fb27910f700 1 -- 172.21.15.167:0/98892636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c009640 con 0x7fb2740f5f60 2022-01-31T19:45:15.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.544+0000 7fb27910f700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f5f60 0x7fb27410db60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fb270009390 tx=0x7fb270006d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:15.548 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.545+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2700125f0 con 0x7fb2740f5f60 2022-01-31T19:45:15.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.545+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7fb270012c00 con 0x7fb2740f5f60 2022-01-31T19:45:15.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.545+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb270011a90 con 0x7fb2740f5f60 2022-01-31T19:45:15.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.545+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27410ec70 con 0x7fb2740f5f60 2022-01-31T19:45:15.549 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.545+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27410f1b0 con 0x7fb2740f5f60 2022-01-31T19:45:15.554 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.548+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c004fa0 con 0x7fb2740f5f60 2022-01-31T19:45:15.556 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.554+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 24) v1 ==== 82737+0+0 (secure 0 0 0) 0x7fb27000f030 con 0x7fb2740f5f60 2022-01-31T19:45:15.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.554+0000 7fb26a7fc700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fb260064ec0 0x7fb260067380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:15.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.554+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7fb270081550 con 0x7fb2740f5f60 2022-01-31T19:45:15.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.554+0000 7fb279910700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fb260064ec0 0x7fb260067380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:45:15.557 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.555+0000 7fb279910700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fb260064ec0 0x7fb260067380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb26c003380 tx=0x7fb26c004d10).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:15.558 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.557+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7fb27004f9f0 con 0x7fb2740f5f60 2022-01-31T19:45:15.893 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.892+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb25c005e80 con 0x7fb2740f5f60 2022-01-31T19:45:15.893 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.892+0000 7fb26a7fc700 1 -- 172.21.15.167:0/98892636 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+691 (secure 0 0 0) 0x7fb27007fb70 con 0x7fb2740f5f60 2022-01-31T19:45:15.894 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2022-01-31T19:45:15.894 INFO:teuthology.orchestra.run.smithi167.stdout: "mon": { 2022-01-31T19:45:15.894 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 2 2022-01-31T19:45:15.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:45:15.895 INFO:teuthology.orchestra.run.smithi167.stdout: "mgr": { 2022-01-31T19:45:15.895 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev)": 2 2022-01-31T19:45:15.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:45:15.895 INFO:teuthology.orchestra.run.smithi167.stdout: "osd": { 2022-01-31T19:45:15.896 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 6 2022-01-31T19:45:15.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:45:15.896 INFO:teuthology.orchestra.run.smithi167.stdout: "mds": { 2022-01-31T19:45:15.896 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 4 2022-01-31T19:45:15.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2022-01-31T19:45:15.897 INFO:teuthology.orchestra.run.smithi167.stdout: "overall": { 2022-01-31T19:45:15.897 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 12, 2022-01-31T19:45:15.897 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev)": 2 2022-01-31T19:45:15.898 INFO:teuthology.orchestra.run.smithi167.stdout: } 2022-01-31T19:45:15.898 INFO:teuthology.orchestra.run.smithi167.stdout:} 2022-01-31T19:45:15.899 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fb260064ec0 msgr2=0x7fb260067380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.899 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fb260064ec0 0x7fb260067380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb26c003380 tx=0x7fb26c004d10).stop 2022-01-31T19:45:15.899 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f5f60 msgr2=0x7fb27410db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:15.900 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f5f60 0x7fb27410db60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fb270009390 tx=0x7fb270006d60).stop 2022-01-31T19:45:15.900 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 shutdown_connections 2022-01-31T19:45:15.900 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7fb260064ec0 0x7fb260067380 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.900 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740f4d60 0x7fb27410d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.900 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 --2- 172.21.15.167:0/98892636 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2740f5f60 0x7fb27410db60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:15.901 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 >> 172.21.15.167:0/98892636 conn(0x7fb2740f04e0 msgr2=0x7fb2740f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:15.901 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 shutdown_connections 2022-01-31T19:45:15.901 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:15.894+0000 7fb27a912700 1 -- 172.21.15.167:0/98892636 wait complete. 2022-01-31T19:45:16.013 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:15 smithi167 conmon[32206]: audit 2022-01-31T19:45:14.952554+0000 mgr.smithi167.aciqpk 2022-01-31T19:45:16.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:15 smithi167 conmon[32206]: (mgr.14650) 32 : audit [DBG] from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:45:16.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:15 smithi167 conmon[32206]: audit 2022-01-31T19:45:15.361966+0000 mgr.smithi167.aciqpk (mgr.14650) 33 : 2022-01-31T19:45:16.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:15 smithi167 conmon[32206]: audit [DBG] from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:45:16.020 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 -- 172.21.15.167:0/4114263180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f6d60 msgr2=0x7f4f900f7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:16.020 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/4114263180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f6d60 0x7f4f900f7180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f4f80004660 tx=0x7f4f80009b30).stop 2022-01-31T19:45:16.021 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 -- 172.21.15.167:0/4114263180 shutdown_connections 2022-01-31T19:45:16.021 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/4114263180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f900f5ab0 0x7f4f900f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:16.021 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/4114263180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f6d60 0x7f4f900f7180 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:16.022 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 -- 172.21.15.167:0/4114263180 >> 172.21.15.167:0/4114263180 conn(0x7f4f900f0640 msgr2=0x7f4f900f2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:16.022 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 -- 172.21.15.167:0/4114263180 shutdown_connections 2022-01-31T19:45:16.023 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.012+0000 7f4f94d9b700 1 -- 172.21.15.167:0/4114263180 wait complete. 2022-01-31T19:45:16.023 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f94d9b700 1 Processor -- start 2022-01-31T19:45:16.024 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f94d9b700 1 -- start start 2022-01-31T19:45:16.024 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f94d9b700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 0x7f4f90109d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:16.025 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f94d9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f900f6d60 0x7f4f9010c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:16.025 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f94d9b700 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f900f5740 con 0x7f4f900f5ab0 2022-01-31T19:45:16.026 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f94d9b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f900f5880 con 0x7f4f900f6d60 2022-01-31T19:45:16.026 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f8e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 0x7f4f90109d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:16.027 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f8e59c700 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 0x7f4f90109d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:34612/0 (socket says 172.21.15.167:34612) 2022-01-31T19:45:16.027 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f8e59c700 1 -- 172.21.15.167:0/1351576453 learned_addr learned my addr 172.21.15.167:0/1351576453 (peer_addr_for_me v2:172.21.15.167:0/0) 2022-01-31T19:45:16.028 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.013+0000 7f4f8dd9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f900f6d60 0x7f4f9010c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-01-31T19:45:16.028 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f8e59c700 1 -- 172.21.15.167:0/1351576453 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f900f6d60 msgr2=0x7f4f9010c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:16.029 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f8e59c700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f900f6d60 0x7f4f9010c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:16.029 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f8e59c700 1 -- 172.21.15.167:0/1351576453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84005040 con 0x7f4f900f5ab0 2022-01-31T19:45:16.029 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f8e59c700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 0x7f4f90109d50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f4f80014040 tx=0x7f4f8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:16.030 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f80019070 con 0x7f4f900f5ab0 2022-01-31T19:45:16.030 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 2 ==== config(26 keys) v1 ==== 1082+0+0 (secure 0 0 0) 0x7f4f80007e70 con 0x7f4f900f5ab0 2022-01-31T19:45:16.031 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f80005040 con 0x7f4f900f5ab0 2022-01-31T19:45:16.031 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.014+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f80014dc0 con 0x7f4f900f5ab0 2022-01-31T19:45:16.032 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.015+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9010cb40 con 0x7f4f900f5ab0 2022-01-31T19:45:16.032 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.015+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f90045bb0 con 0x7f4f900f5ab0 2022-01-31T19:45:16.033 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.016+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 24) v1 ==== 82737+0+0 (secure 0 0 0) 0x7f4f80025070 con 0x7f4f900f5ab0 2022-01-31T19:45:16.033 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.016+0000 7f4f7effd700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f4f78064d20 0x7f4f780671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-01-31T19:45:16.034 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.016+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5687+0+0 (secure 0 0 0) 0x7f4f80082570 con 0x7f4f900f5ab0 2022-01-31T19:45:16.034 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.017+0000 7f4f8dd9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f4f78064d20 0x7f4f780671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-01-31T19:45:16.034 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.017+0000 7f4f8dd9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f4f78064d20 0x7f4f780671e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f4f900f6b30 tx=0x7f4f84009b30).ready entity=mgr.14650 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-01-31T19:45:16.035 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.022+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162801 (secure 0 0 0) 0x7f4f80050a50 con 0x7f4f900f5ab0 2022-01-31T19:45:16.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:15 smithi171 conmon[35325]: audit 2022-01-31T19:45:14.952554+0000 mgr.smithi167.aciqpk (mgr.14650) 32 : audit [DBG] from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:45:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:15 smithi171 conmon[35325]: audit 2022-01-31T19:45 2022-01-31T19:45:16.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:15 smithi171 conmon[35325]: :15.361966+0000 mgr.smithi167.aciqpk (mgr.14650) 33 : audit [DBG] from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-01-31T19:45:16.315 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.315+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4f900587c0 con 0x7f4f900f5ab0 2022-01-31T19:45:16.316 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.316+0000 7f4f7effd700 1 -- 172.21.15.167:0/1351576453 <== mon.0 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1706 (secure 0 0 0) 0x7f4f80023070 con 0x7f4f900f5ab0 2022-01-31T19:45:16.317 INFO:teuthology.orchestra.run.smithi167.stdout:e17 2022-01-31T19:45:16.318 INFO:teuthology.orchestra.run.smithi167.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:16.318 INFO:teuthology.orchestra.run.smithi167.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:16.318 INFO:teuthology.orchestra.run.smithi167.stdout:legacy client fscid: 1 2022-01-31T19:45:16.319 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:45:16.319 INFO:teuthology.orchestra.run.smithi167.stdout:Filesystem 'cephfs' (1) 2022-01-31T19:45:16.319 INFO:teuthology.orchestra.run.smithi167.stdout:fs_name cephfs 2022-01-31T19:45:16.319 INFO:teuthology.orchestra.run.smithi167.stdout:epoch 17 2022-01-31T19:45:16.319 INFO:teuthology.orchestra.run.smithi167.stdout:flags 32 2022-01-31T19:45:16.320 INFO:teuthology.orchestra.run.smithi167.stdout:created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:16.320 INFO:teuthology.orchestra.run.smithi167.stdout:modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:16.320 INFO:teuthology.orchestra.run.smithi167.stdout:tableserver 0 2022-01-31T19:45:16.320 INFO:teuthology.orchestra.run.smithi167.stdout:root 0 2022-01-31T19:45:16.321 INFO:teuthology.orchestra.run.smithi167.stdout:session_timeout 60 2022-01-31T19:45:16.321 INFO:teuthology.orchestra.run.smithi167.stdout:session_autoclose 300 2022-01-31T19:45:16.321 INFO:teuthology.orchestra.run.smithi167.stdout:max_file_size 1099511627776 2022-01-31T19:45:16.321 INFO:teuthology.orchestra.run.smithi167.stdout:required_client_features {} 2022-01-31T19:45:16.321 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure 0 2022-01-31T19:45:16.322 INFO:teuthology.orchestra.run.smithi167.stdout:last_failure_osd_epoch 0 2022-01-31T19:45:16.322 INFO:teuthology.orchestra.run.smithi167.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:16.322 INFO:teuthology.orchestra.run.smithi167.stdout:max_mds 2 2022-01-31T19:45:16.322 INFO:teuthology.orchestra.run.smithi167.stdout:in 0,1 2022-01-31T19:45:16.323 INFO:teuthology.orchestra.run.smithi167.stdout:up {0=14436,1=24267} 2022-01-31T19:45:16.323 INFO:teuthology.orchestra.run.smithi167.stdout:failed 2022-01-31T19:45:16.323 INFO:teuthology.orchestra.run.smithi167.stdout:damaged 2022-01-31T19:45:16.323 INFO:teuthology.orchestra.run.smithi167.stdout:stopped 2022-01-31T19:45:16.324 INFO:teuthology.orchestra.run.smithi167.stdout:data_pools [3] 2022-01-31T19:45:16.324 INFO:teuthology.orchestra.run.smithi167.stdout:metadata_pool 2 2022-01-31T19:45:16.324 INFO:teuthology.orchestra.run.smithi167.stdout:inline_data enabled 2022-01-31T19:45:16.324 INFO:teuthology.orchestra.run.smithi167.stdout:balancer 2022-01-31T19:45:16.324 INFO:teuthology.orchestra.run.smithi167.stdout:standby_count_wanted 1 2022-01-31T19:45:16.325 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577]] 2022-01-31T19:45:16.325 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983]] 2022-01-31T19:45:16.325 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981]] 2022-01-31T19:45:16.325 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319]] 2022-01-31T19:45:16.326 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:45:16.326 INFO:teuthology.orchestra.run.smithi167.stdout: 2022-01-31T19:45:16.327 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.320+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f4f78064d20 msgr2=0x7f4f780671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:16.327 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.320+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f4f78064d20 0x7f4f780671e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f4f900f6b30 tx=0x7f4f84009b30).stop 2022-01-31T19:45:16.327 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.320+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 msgr2=0x7f4f90109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-01-31T19:45:16.327 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.320+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 0x7f4f90109d50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f4f80014040 tx=0x7f4f8000b040).stop 2022-01-31T19:45:16.328 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.322+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 shutdown_connections 2022-01-31T19:45:16.328 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.322+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:6800/2258949773,v1:172.21.15.167:6801/2258949773] conn(0x7f4f78064d20 0x7f4f780671e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:16.328 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.322+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900f5ab0 0x7f4f90109d50 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0x7f4f80014040 tx=0x7f4f8000b040).stop 2022-01-31T19:45:16.329 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.322+0000 7f4f94d9b700 1 --2- 172.21.15.167:0/1351576453 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f900f6d60 0x7f4f9010c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-01-31T19:45:16.329 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.322+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 >> 172.21.15.167:0/1351576453 conn(0x7f4f900f0640 msgr2=0x7f4f900fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-01-31T19:45:16.329 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.323+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 shutdown_connections 2022-01-31T19:45:16.329 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:45:16.326+0000 7f4f94d9b700 1 -- 172.21.15.167:0/1351576453 wait complete. 2022-01-31T19:45:16.330 INFO:teuthology.orchestra.run.smithi167.stderr:dumped fsmap epoch 17 2022-01-31T19:45:17.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:16 smithi167 conmon[32206]: audit 2022-01-31T19:45:15.892835+0000 mon.smithi167 (mon.0) 768 : audit [DBG] 2022-01-31T19:45:17.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:16 smithi167 conmon[32206]: from='client.? 172.21.15.167:0/98892636' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:45:17.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:16 smithi167 conmon[32206]: cluster 2022-01-31T19:45:16.066371+0000 mgr.smithi167.aciqpk (mgr.14650) 34 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2022-01-31T19:45:17.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:16 smithi167 conmon[32206]: audit 2022-01-31T19:45:16.316555+0000 mon.smithi167 ( 2022-01-31T19:45:17.016 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:16 smithi167 conmon[32206]: mon.0) 769 : audit [DBG] from='client.? 172.21.15.167:0/1351576453' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:45:17.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:16 smithi171 conmon[35325]: audit 2022-01-31T19:45:15.892835+0000 mon.smithi167 (mon. 2022-01-31T19:45:17.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:16 smithi171 conmon[35325]: 0) 768 : audit [DBG] from='client.? 172.21.15.167:0/98892636' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2022-01-31T19:45:17.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:16 smithi171 conmon[35325]: cluster 2022-01-31T19:45:16.066371+0000 mgr.smithi167.aciqpk (mgr.14650) 34 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 308 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2022-01-31T19:45:17.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:45:16 smithi171 conmon[35325]: audit 2022-01-31T19:45:16.316555+0000 mon.smithi167 (mon.0) 769 : audit [DBG] from='client.? 172.21.15.167:0/1351576453' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-01-31T19:45:17.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:17 smithi167 systemd[1]: Stopping Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:45:17.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:17 smithi167 bash[101581]: Error: no container with name or ID "ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c-mon.smithi167" found: no such container 2022-01-31T19:45:17.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:17 smithi167 conmon[32206]: debug 2022-01-31T19:45:17.660+0000 7f07d86f3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi167 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2022-01-31T19:45:17.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:17 smithi167 conmon[32206]: debug 2022-01-31T19:45:17.660+0000 7f07d86f3700 -1 mon.smithi167@0(leader) e2 *** Got Signal Terminated *** 2022-01-31T19:45:19.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:18 smithi167 bash[101581]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c-mon-smithi167 2022-01-31T19:45:20.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:20 smithi167 bash[101581]: Error: no container with name or ID "ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c-mon.smithi167" found: no such container 2022-01-31T19:45:20.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:20 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Succeeded. 2022-01-31T19:45:20.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:20 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:20.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:20 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:45:22.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:21 smithi167 podman[101922]: 2022-01-31T19:45:22.861 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 bash[101772]: ba7f9ea65544c363703fd73d09a1cf5620a623ace8fa3391bfa04e156846d9b4 2022-01-31T19:45:22.861 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:22.862 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:22.862 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:22.862 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: load: jerasure load: lrc 2022-01-31T19:45:22.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:22.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:22.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:22.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB SUMMARY 2022-01-31T19:45:22.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB Session ID: 0RWQ7EVOGMZS8DMWV9CK 2022-01-31T19:45:22.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:22.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:22.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: MANIFEST file: MANIFEST-000008 size: 392 Bytes 2022-01-31T19:45:22.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 1, files: 000014.sst 2022-01-31T19:45:22.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000012.log size: 7834206 ; 2022-01-31T19:45:22.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:22.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:22.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:22.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:22.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.env: 0x55fe400dec00 2022-01-31T19:45:22.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:22.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.info_log: 0x55fe411fa700 2022-01-31T19:45:22.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:22.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:22.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:22.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:22.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:22.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:22.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:22.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:22.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:22.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:22.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:22.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:22.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:22.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:22.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:22.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:22.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:22.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:22.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:22.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:22.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:22.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:22.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:22.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:22.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_buffer_manager: 0x55fe412e2330 2022-01-31T19:45:22.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:22.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:22.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:22.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:22.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:22.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:22.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:22.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:22.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:22.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:22.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:22.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:22.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:22.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:22.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.row_cache: None 2022-01-31T19:45:22.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:22.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:22.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:22.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:22.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:22.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:22.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:22.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:22.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:22.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:22.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:22.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:22.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:22.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:22.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:22.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:22.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:22.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:22.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:22.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:22.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:22.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:22.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:22.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:22.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:22.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:22.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:22.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:22.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:22.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:22.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:22.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:22.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:22.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:22.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:22.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:22.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:22.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:22.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:22.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:22.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:22.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:22.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:22.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:22.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000008 2022-01-31T19:45:22.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:22.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:22.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.merge_operator: 2022-01-31T19:45:22.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:22.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:22.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:22.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:22.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:22.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fe411c5c20) 2022-01-31T19:45:22.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:22.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:22.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:22.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:22.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:22.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:22.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:22.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:22.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:22.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:22.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:22.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55fe4122a170 2022-01-31T19:45:22.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:22.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:22.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:22.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:22.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:22.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:22.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:22.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:22.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:22.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:22.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:22.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:22.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:22.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:22.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:22.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:22.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:22.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:22.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:22.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:22.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:22.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:22.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:22.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:22.897 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:22.897 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:22.897 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:22.897 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:22.898 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:22.898 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:22.898 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:22.898 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:22.899 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:22.899 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:22.899 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:22.899 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:22.899 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:22.900 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:22.900 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:22.900 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:22.900 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:22.901 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:22.901 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:22.901 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:22.901 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:22.901 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:22.902 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:22.902 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:22.902 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:22.902 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:22.903 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:22.903 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:22.903 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:22.903 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:22.903 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:22.904 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:22.904 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:22.904 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:22.905 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:22.905 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:22.905 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:22.905 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:22.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:22.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:22.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:22.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:22.906 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:22.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:22.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:22.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:22.907 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:22.908 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:22.908 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:22.908 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:22.908 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:22.908 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:22.914 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:22.914 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:22.914 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:22.915 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:22.915 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:22.915 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:22.915 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:22.916 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:22.916 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:22.916 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:22.916 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:22.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:22.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:22.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:22.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:22.917 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:22.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:22.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:22.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:22.918 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:22.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:22.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:22.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:22.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 16, last_sequence is 6553, log_number is 12,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:22.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 12 2022-01-31T19:45:22.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4083] Creating manifest 16 2022-01-31T19:45:22.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:22.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322742165, "job": 1, "event": "recovery_started", "wal_files": [12]} 2022-01-31T19:45:22.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #12 mode 2 2022-01-31T19:45:22.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-01-31T19:45:22.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322757904, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 17, "file_size": 7465891, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7456782, "index_size": 6824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 10631, "raw_average_key_size": 21, "raw_value_size": 7442143, "raw_average_value_size": 15095, "num_data_blocks": 308, "num_entries": 493, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1643658322, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a46c0b0b-fed8-4bd0-bfc5-9623952dfff7", "db_session_id": "0RWQ7EVOGMZS8DMWV9CK"}} 2022-01-31T19:45:22.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4083] Creating manifest 18 2022-01-31T19:45:22.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:22.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322760176, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:22.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000012.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:22.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55fe41210a80 2022-01-31T19:45:22.922 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB pointer 0x55fe41220000 2022-01-31T19:45:22.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:22.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:22.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:22.923 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:22.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:22.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:22.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:22.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:22.924 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:22.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:22.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:22.925 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:22.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:22.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:22.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.926 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:22.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:22.927 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:22.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:22.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.007, interval 0.007 2022-01-31T19:45:22.928 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:22.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:22.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:22.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:22.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.01 GB write, 336.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:22.929 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.01 GB write, 336.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:22.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:22.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:22.930 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:22.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:22.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:22.931 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:22.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:22.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.932 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:22.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:22.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:22.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:22.933 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:22.934 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.007, interval 0.000 2022-01-31T19:45:22.934 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:22.934 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:22.934 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:22.934 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:22.935 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.01 GB write, 336.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:22.935 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:22.935 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:22.935 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.936 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:22.936 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:22.936 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:22.936 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:22.936 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:22.937 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:22.937 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:22.937 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:22.937 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:22.938 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.938 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:22.938 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:22.938 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:22.939 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:22.939 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:22.939 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:22.939 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:22.940 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:22.940 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:22.940 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:22.940 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:22.940 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:22.941 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:22.941 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:22.941 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:22.941 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:22.942 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:22.942 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:22.942 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:22.942 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:22.943 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:22.943 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:22.943 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:22.943 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:22.943 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:22.944 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:22.944 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:22.944 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:22.944 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:22.945 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:22.945 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.945 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.945 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:22.946 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:22.946 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:22.946 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:22.946 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:22.946 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:22.947 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:22.947 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:22.947 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:22.947 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:22.948 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:22.948 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.948 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:22.948 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:22.948 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:22.949 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:22.949 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:22.949 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:22.949 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:22.949 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:22.950 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:22.950 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:22.950 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:22.950 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:22.950 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:22.951 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:22.951 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: main() 2022-01-31T19:45:22.951 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: __libc_start_main() 2022-01-31T19:45:22.951 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: _start() 2022-01-31T19:45:22.952 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: *** Caught signal (Aborted) ** 2022-01-31T19:45:22.952 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:22.952 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:22.764+0000 7f6fb3912880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:22.952 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:22.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:22.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:22.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:22.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:22.953 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:22.954 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:22.954 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:22.954 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:22.954 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:22.955 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:22.955 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: main() 2022-01-31T19:45:22.955 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: __libc_start_main() 2022-01-31T19:45:22.955 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: _start() 2022-01-31T19:45:22.956 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:22.956 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:22.956 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:22.956 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: gsignal() 2022-01-31T19:45:22.957 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: abort() 2022-01-31T19:45:22.957 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:22.957 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:22.957 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:22.958 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:22.958 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:22.958 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:22.958 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:22.958 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:22.959 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 12: main() 2022-01-31T19:45:22.959 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 13: __libc_start_main() 2022-01-31T19:45:22.959 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 14: _start() 2022-01-31T19:45:22.959 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: *** Caught signal (Aborted) ** 2022-01-31T19:45:22.960 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:22.960 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:22.960 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:22.960 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:22.960 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:22.961 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:22.961 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:22.961 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:22.961 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:22.962 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:22.962 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:22.962 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:22.962 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:22.963 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:22.963 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:22.963 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:22.963 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:22.963 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:22.964 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:22.765+0000 7f6fb3912880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:22.964 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:22.964 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:22.964 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:22.964 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:22.965 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: gsignal() 2022-01-31T19:45:22.965 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: abort() 2022-01-31T19:45:22.965 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:22.965 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:22.965 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:22.966 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:22.966 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:22.966 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:22.966 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:22.966 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:22.967 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 12: main() 2022-01-31T19:45:22.967 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 13: __libc_start_main() 2022-01-31T19:45:22.967 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 14: _start() 2022-01-31T19:45:22.967 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:22.968 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:22.968 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command assert hook 0x55fe411c0a50 2022-01-31T19:45:22.968 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command abort hook 0x55fe411c0a50 2022-01-31T19:45:22.968 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command leak_some_memory hook 0x55fe411c0a50 2022-01-31T19:45:22.968 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perfcounters_dump hook 0x55fe411c0a50 2022-01-31T19:45:22.968 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command 1 hook 0x55fe411c0a50 2022-01-31T19:45:22.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf dump hook 0x55fe411c0a50 2022-01-31T19:45:22.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perfcounters_schema hook 0x55fe411c0a50 2022-01-31T19:45:22.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf histogram dump hook 0x55fe411c0a50 2022-01-31T19:45:22.969 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command 2 hook 0x55fe411c0a50 2022-01-31T19:45:22.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf schema hook 0x55fe411c0a50 2022-01-31T19:45:22.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf histogram schema hook 0x55fe411c0a50 2022-01-31T19:45:22.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf reset hook 0x55fe411c0a50 2022-01-31T19:45:22.970 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config show hook 0x55fe411c0a50 2022-01-31T19:45:22.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config help hook 0x55fe411c0a50 2022-01-31T19:45:22.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config set hook 0x55fe411c0a50 2022-01-31T19:45:22.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config unset hook 0x55fe411c0a50 2022-01-31T19:45:22.971 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config get hook 0x55fe411c0a50 2022-01-31T19:45:22.972 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config diff hook 0x55fe411c0a50 2022-01-31T19:45:22.972 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config diff get hook 0x55fe411c0a50 2022-01-31T19:45:22.972 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command injectargs hook 0x55fe411c0a50 2022-01-31T19:45:22.972 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command log flush hook 0x55fe411c0a50 2022-01-31T19:45:22.972 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command log dump hook 0x55fe411c0a50 2022-01-31T19:45:22.973 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command log reopen hook 0x55fe411c0a50 2022-01-31T19:45:22.973 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command dump_mempools hook 0x55fe412ae068 2022-01-31T19:45:22.973 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:22.973 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:22.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:22.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:22.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:22.974 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command 0 hook 0x55fe411c50a0 2022-01-31T19:45:22.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command version hook 0x55fe411c50a0 2022-01-31T19:45:22.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command git_version hook 0x55fe411c50a0 2022-01-31T19:45:22.975 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command help hook 0x55fe411c09f0 2022-01-31T19:45:22.976 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command get_command_descriptions hook 0x55fe411c09e0 2022-01-31T19:45:22.976 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) entry start 2022-01-31T19:45:22.976 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: load: jerasure load: lrc 2022-01-31T19:45:22.976 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:22.976 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:22.977 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:22.977 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:22.977 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:22.977 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:22.978 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:22.978 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:22.978 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:22.978 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:22.978 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB SUMMARY 2022-01-31T19:45:22.979 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB Session ID: 0RWQ7EVOGMZS8DMWV9CK 2022-01-31T19:45:22.979 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:22.979 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:22.979 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: MANIFEST file: MANIFEST-000008 size: 392 Bytes 2022-01-31T19:45:22.980 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 1, files: 000014.sst 2022-01-31T19:45:22.980 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000012.log size: 7834206 ; 2022-01-31T19:45:22.980 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:22.980 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:22.980 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:22.981 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:22.981 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.env: 0x55fe400dec00 2022-01-31T19:45:22.981 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:22.981 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.info_log: 0x55fe411fa700 2022-01-31T19:45:22.982 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:22.982 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:22.982 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:22.982 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:22.983 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:22.983 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:22.983 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:22.983 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:22.984 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:22.984 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:22.984 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:22.984 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:22.984 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:22.985 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:22.985 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:22.985 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:22.985 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:22.986 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:22.986 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:22.986 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:22.986 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:22.987 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:22.987 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:22.987 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:22.987 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_buffer_manager: 0x55fe412e2330 2022-01-31T19:45:22.988 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:22.988 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:22.988 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:22.988 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:22.989 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:22.989 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:22.989 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:22.989 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:22.990 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:22.990 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:22.990 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:22.990 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:22.991 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:22.991 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:22.991 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.row_cache: None 2022-01-31T19:45:22.991 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:22.991 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:22.992 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:22.992 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:22.992 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:22.992 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:22.993 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:22.993 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:22.993 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:22.993 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:22.993 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:22.994 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:22.994 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:22.994 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:22.994 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:22.995 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:22.995 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:22.995 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:22.995 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:22.996 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:22.996 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:22.996 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:22.996 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:22.996 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:22.997 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:22.997 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:22.997 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:22.997 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:22.997 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:22.998 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:22.998 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:22.998 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:22.998 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:22.999 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:22.999 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:22.999 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:22.999 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:22.999 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:23.000 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:23.000 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:23.000 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:23.000 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:23.000 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:23.001 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:23.001 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000008 2022-01-31T19:45:23.001 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:23.001 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:23.001 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.merge_operator: 2022-01-31T19:45:23.002 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:23.002 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:23.002 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:23.002 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:23.002 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:23.003 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fe411c5c20) 2022-01-31T19:45:23.003 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:23.003 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:23.003 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:23.004 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:23.004 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:23.004 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:23.004 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:23.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:23.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:23.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:23.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:23.005 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55fe4122a170 2022-01-31T19:45:23.006 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:23.006 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:23.006 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:23.006 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:23.007 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:23.007 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:23.007 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:23.007 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:23.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:23.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:23.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:23.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:23.008 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:23.009 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:23.009 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:23.009 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:23.009 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:23.009 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:23.010 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:23.010 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:23.010 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:23.010 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:23.011 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:23.011 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:23.011 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:23.011 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:23.011 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:23.012 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:23.012 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:23.012 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:23.012 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:23.013 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:23.013 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:23.013 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:23.013 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:23.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:23.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:23.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:23.014 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:23.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:23.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:23.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:23.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:23.015 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:23.016 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:23.016 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:23.016 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:23.016 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:23.017 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:23.017 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:23.017 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:23.017 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:23.018 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:23.018 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:23.018 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:23.018 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:23.018 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:23.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:23.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:23.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:23.019 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:23.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:23.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:23.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:23.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:23.020 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:23.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:23.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:23.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:23.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:23.021 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:23.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:23.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:23.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:23.022 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:23.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:23.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:23.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:23.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:23.023 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:23.024 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:23.024 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:23.024 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:23.024 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:23.025 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:23.025 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:23.025 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:23.025 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:23.025 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:23.026 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:23.026 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:23.026 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:23.026 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:23.027 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:23.027 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:23.027 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:23.027 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:23.027 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:23.028 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:23.028 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 16, last_sequence is 6553, log_number is 12,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:23.028 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 12 2022-01-31T19:45:23.028 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4083] Creating manifest 16 2022-01-31T19:45:23.029 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:23.029 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322742165, "job": 1, "event": "recovery_started", "wal_files": [12]} 2022-01-31T19:45:23.029 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #12 mode 2 2022-01-31T19:45:23.029 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-01-31T19:45:23.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322757904, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 17, "file_size": 7465891, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7456782, "index_size": 6824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 10631, "raw_average_key_size": 21, "raw_value_size": 7442143, "raw_average_value_size": 15095, "num_data_blocks": 308, "num_entries": 493, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1643658322, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a46c0b0b-fed8-4bd0-bfc5-9623952dfff7", "db_session_id": "0RWQ7EVOGMZS8DMWV9CK"}} 2022-01-31T19:45:23.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4083] Creating manifest 18 2022-01-31T19:45:23.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:23.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322760176, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:23.030 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000012.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:23.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55fe41210a80 2022-01-31T19:45:23.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB pointer 0x55fe41220000 2022-01-31T19:45:23.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:23.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:23.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:23.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:23.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:23.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:23.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:23.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:23.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:23.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:23.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:23.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:23.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.007, interval 0.007 2022-01-31T19:45:23.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:23.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:23.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:23.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:23.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.01 GB write, 336.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.01 GB write, 336.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:23.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:23.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:23.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:23.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:23.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.007, interval 0.000 2022-01-31T19:45:23.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:23.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:23.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:23.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:23.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.01 GB write, 336.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:23.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:23.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: cephx 2022-01-31T19:45:23.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: cephx 2022-01-31T19:45:23.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: cephx 2022-01-31T19:45:23.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: none 2022-01-31T19:45:23.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:23.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:23.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: cephx 2022-01-31T19:45:23.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: cephx 2022-01-31T19:45:23.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: cephx 2022-01-31T19:45:23.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: none 2022-01-31T19:45:23.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:23.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: adding auth protocol: cephx 2022-01-31T19:45:23.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: adding auth protocol: cephx 2022-01-31T19:45:23.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:23.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:23.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:23.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:23.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:23.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:23.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:23.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:23.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:23.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:23.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:23.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:23.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:23.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:23.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:23.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:23.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:23.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:23.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:23.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:23.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:23.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:23.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:23.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:23.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:23.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:23.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:23.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:23.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:23.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:23.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:23.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:23.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:23.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:23.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:23.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:23.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:23.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:23.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:23.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:23.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:23.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: -290> 2022-01-31T19:45:22.764+0000 7f6fb3912880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:23.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: *** Caught signal (Aborted) ** 2022-01-31T19:45:23.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:23.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:23.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:23.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:23.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:23.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:23.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:23.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:23.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:23.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:23.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:23.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: main() 2022-01-31T19:45:23.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: __libc_start_main() 2022-01-31T19:45:23.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: _start() 2022-01-31T19:45:23.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: -289> 2022-01-31T19:45:22.765+0000 7f6fb3912880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:23.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:23.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:23.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: gsignal() 2022-01-31T19:45:23.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: abort() 2022-01-31T19:45:23.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:23.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 12: main() 2022-01-31T19:45:23.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 13: __libc_start_main() 2022-01-31T19:45:23.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 14: _start() 2022-01-31T19:45:23.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:23.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command assert hook 0x55fe411c0a50 2022-01-31T19:45:23.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command abort hook 0x55fe411c0a50 2022-01-31T19:45:23.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command leak_some_memory hook 0x55fe411c0a50 2022-01-31T19:45:23.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perfcounters_dump hook 0x55fe411c0a50 2022-01-31T19:45:23.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command 1 hook 0x55fe411c0a50 2022-01-31T19:45:23.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf dump hook 0x55fe411c0a50 2022-01-31T19:45:23.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perfcounters_schema hook 0x55fe411c0a50 2022-01-31T19:45:23.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf histogram dump hook 0x55fe411c0a50 2022-01-31T19:45:23.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command 2 hook 0x55fe411c0a50 2022-01-31T19:45:23.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf schema hook 0x55fe411c0a50 2022-01-31T19:45:23.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf histogram schema hook 0x55fe411c0a50 2022-01-31T19:45:23.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command perf reset hook 0x55fe411c0a50 2022-01-31T19:45:23.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config show hook 0x55fe411c0a50 2022-01-31T19:45:23.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config help hook 0x55fe411c0a50 2022-01-31T19:45:23.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config set hook 0x55fe411c0a50 2022-01-31T19:45:23.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config unset hook 0x55fe411c0a50 2022-01-31T19:45:23.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config get hook 0x55fe411c0a50 2022-01-31T19:45:23.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config diff hook 0x55fe411c0a50 2022-01-31T19:45:23.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command config diff get hook 0x55fe411c0a50 2022-01-31T19:45:23.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command injectargs hook 0x55fe411c0a50 2022-01-31T19:45:23.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command log flush hook 0x55fe411c0a50 2022-01-31T19:45:23.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command log dump hook 0x55fe411c0a50 2022-01-31T19:45:23.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command log reopen hook 0x55fe411c0a50 2022-01-31T19:45:23.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command dump_mempools hook 0x55fe412ae068 2022-01-31T19:45:23.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:23.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:23.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:23.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:23.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:23.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command 0 hook 0x55fe411c50a0 2022-01-31T19:45:23.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command version hook 0x55fe411c50a0 2022-01-31T19:45:23.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command git_version hook 0x55fe411c50a0 2022-01-31T19:45:23.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command help hook 0x55fe411c09f0 2022-01-31T19:45:23.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) register_command get_command_descriptions hook 0x55fe411c09e0 2022-01-31T19:45:23.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: asok(0x55fe414a8000) entry start 2022-01-31T19:45:23.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: load: jerasure load: lrc 2022-01-31T19:45:23.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:23.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:23.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:23.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:23.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:23.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:23.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:23.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:23.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:23.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:23.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB SUMMARY 2022-01-31T19:45:23.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB Session ID: 0RWQ7EVOGMZS8DMWV9CK 2022-01-31T19:45:23.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:23.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:23.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: MANIFEST file: MANIFEST-000008 size: 392 Bytes 2022-01-31T19:45:23.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 1, files: 000014.sst 2022-01-31T19:45:23.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000012.log size: 7834206 ; 2022-01-31T19:45:23.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:23.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:23.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:23.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:23.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.env: 0x55fe400dec00 2022-01-31T19:45:23.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:23.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.info_log: 0x55fe411fa700 2022-01-31T19:45:23.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:23.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:23.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:23.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:23.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:23.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:23.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:23.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:23.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:23.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:23.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:23.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:23.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:23.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:23.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:23.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:23.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:23.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:23.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:23.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:23.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:23.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:23.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:23.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:23.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_buffer_manager: 0x55fe412e2330 2022-01-31T19:45:23.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:23.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:23.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:23.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:23.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:23.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:23.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:23.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:23.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:23.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:23.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:23.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:23.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:23.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:23.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.row_cache: None 2022-01-31T19:45:23.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:23.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:23.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:23.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:23.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:23.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:23.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:23.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:23.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:23.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:23.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:23.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:23.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:23.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:23.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:23.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:23.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:23.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:23.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:23.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:23.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:23.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:23.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:23.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:23.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:23.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:23.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:23.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:23.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:23.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:23.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:23.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:23.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:23.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:23.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:23.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:23.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:23.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:23.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:23.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:23.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:23.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:23.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:23.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:23.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000008 2022-01-31T19:45:23.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:23.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:23.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.merge_operator: 2022-01-31T19:45:23.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:23.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:23.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:23.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:23.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:23.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fe411c5c20) 2022-01-31T19:45:23.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:23.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:23.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:23.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:23.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:23.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:23.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:23.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:23.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:23.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:23.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:23.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55fe4122a170 2022-01-31T19:45:23.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:23.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:23.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:23.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:23.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:23.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:23.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:23.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:23.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:23.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:23.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:23.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:23.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:23.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:23.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:23.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:23.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:23.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:23.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:23.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:23.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:23.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:23.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:23.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:23.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:23.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:23.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:23.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:23.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:23.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:23.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:23.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:23.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:23.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:23.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:23.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:23.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:23.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:23.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:23.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:23.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:23.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:23.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:23.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:23.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:23.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:23.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:23.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:23.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:23.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:23.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:23.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:23.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:23.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:23.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:23.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:23.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:23.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:23.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:23.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:23.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:23.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:23.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:23.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:23.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:23.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:23.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:23.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:23.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:23.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:23.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:23.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:23.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:23.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:23.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:23.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:23.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:23.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:23.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:23.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:23.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:23.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:23.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:23.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:23.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:23.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:23.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:23.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:23.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:23.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:23.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:23.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:23.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:23.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:23.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:23.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:23.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:23.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:23.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:23.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 16, last_sequence is 6553, log_number is 12,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:23.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 12 2022-01-31T19:45:23.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4083] Creating manifest 16 2022-01-31T19:45:23.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:23.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322742165, "job": 1, "event": "recovery_started", "wal_files": [12]} 2022-01-31T19:45:23.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #12 mode 2 2022-01-31T19:45:23.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-01-31T19:45:23.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322757904, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 17, "file_size": 7465891, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 7456782, "index_size": 6824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 10631, "raw_average_key_size": 21, "raw_value_size": 7442143, "raw_average_value_size": 15095, "num_data_blocks": 308, "num_entries": 493, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1643658322, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a46c0b0b-fed8-4bd0-bfc5-9623952dfff7", "db_session_id": "0RWQ7EVOGMZS8DMWV9CK"}} 2022-01-31T19:45:23.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:4083] Creating manifest 18 2022-01-31T19:45:23.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:23.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658322760176, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:23.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000012.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:23.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55fe41210a80 2022-01-31T19:45:23.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: DB pointer 0x55fe41220000 2022-01-31T19:45:23.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:23.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:23.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:23.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:23.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:23.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:23.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:23.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:23.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:23.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:23.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:23.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:23.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.007, interval 0.007 2022-01-31T19:45:23.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:23.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:23.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:23.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:23.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.01 GB write, 336.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.01 GB write, 336.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:23.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:23.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:23.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:23.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:23.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:23.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:23.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 749.7 0.01 0.00 1 0.009 0 0 2022-01-31T19:45:23.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:23.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.007, interval 0.000 2022-01-31T19:45:23.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:23.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:23.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:23.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:23.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.01 GB write, 336.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:23.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:23.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:23.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: cephx 2022-01-31T19:45:23.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: cephx 2022-01-31T19:45:23.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: cephx 2022-01-31T19:45:23.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding auth protocol: none 2022-01-31T19:45:23.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: crc 2022-01-31T19:45:23.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8140) adding con mode: secure 2022-01-31T19:45:23.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:23.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:23.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: cephx 2022-01-31T19:45:23.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: cephx 2022-01-31T19:45:23.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: cephx 2022-01-31T19:45:23.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding auth protocol: none 2022-01-31T19:45:23.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: crc 2022-01-31T19:45:23.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: AuthRegistry(0x55fe412e8a40) adding con mode: secure 2022-01-31T19:45:23.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:23.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: adding auth protocol: cephx 2022-01-31T19:45:23.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: adding auth protocol: cephx 2022-01-31T19:45:23.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:23.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:23.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:23.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:23.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:23.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:23.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:23.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:23.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:23.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:23.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:23.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:23.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:23.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:23.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:23.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:23.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:23.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:23.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:23.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:23.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:23.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:23.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:23.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:23.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:23.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:23.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:23.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:23.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:23.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:23.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:23.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:23.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:23.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:23.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:23.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:23.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:23.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:23.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:23.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:23.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:23.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:23.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 ceph-mon[101987]: *** Caught signal (Aborted) ** 2022-01-31T19:45:23.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:23.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:23.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:23.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:23.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:23.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:23.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:23.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:23.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:23.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:23.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: -1> 2022-01-31T19:45:22.764+0000 7f6fb3912880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f6fb3912880 time 2022-01-31T19:45:22.763349+0000 2022-01-31T19:45:23.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:23.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6fb2b6c1d4] 2022-01-31T19:45:23.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: main() 2022-01-31T19:45:23.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: __libc_start_main() 2022-01-31T19:45:23.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: _start() 2022-01-31T19:45:23.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 0> 2022-01-31T19:45:22.765+0000 7f6fb3912880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:23.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: in thread 7f6fb3912880 thread_name:ceph-mon 2022-01-31T19:45:23.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:23.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f6fb0affc20] 2022-01-31T19:45:23.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2: gsignal() 2022-01-31T19:45:23.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 3: abort() 2022-01-31T19:45:23.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f6fb2b6c232] 2022-01-31T19:45:23.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f6fb2b6c3f5] 2022-01-31T19:45:23.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f6fb3136838] 2022-01-31T19:45:23.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55fe3f94bfa3] 2022-01-31T19:45:23.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55fe3f86fb0e] 2022-01-31T19:45:23.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fe3f70151c] 2022-01-31T19:45:23.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 10: (Monitor::init_paxos()+0x10c) [0x55fe3f70182c] 2022-01-31T19:45:23.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 11: (Monitor::preinit()+0xd30) [0x55fe3f72f2a0] 2022-01-31T19:45:23.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 12: main() 2022-01-31T19:45:23.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 13: __libc_start_main() 2022-01-31T19:45:23.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 14: _start() 2022-01-31T19:45:23.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:23.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 conmon[101962]: 2022-01-31T19:45:23.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:22 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:23.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:23 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Main process exited, code=exited, status=134/n/a 2022-01-31T19:45:23.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:23 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Failed with result 'exit-code'. 2022-01-31T19:45:33.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Service RestartSec=10s expired, scheduling restart. 2022-01-31T19:45:33.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Scheduled restart job, restart counter is at 1. 2022-01-31T19:45:33.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:33.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:45:34.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 podman[102234]: 2022-01-31T19:45:34.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:34.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:34.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:34.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: load: jerasure load: lrc 2022-01-31T19:45:34.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 bash[102078]: eadd553cbe4faa7556f62c88a64ff674ebc2b256b17c633171cb4db5fd20c6aa 2022-01-31T19:45:34.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:34.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:34.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:34.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB SUMMARY 2022-01-31T19:45:34.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB Session ID: 71CIRQWJIUABD8LIDS9M 2022-01-31T19:45:34.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:34.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:34.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: MANIFEST file: MANIFEST-000018 size: 207 Bytes 2022-01-31T19:45:34.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:34.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000019.log size: 0 ; 2022-01-31T19:45:34.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:34.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:34.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:34.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:34.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.env: 0x5628ae1bec00 2022-01-31T19:45:34.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:34.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.info_log: 0x5628b07e6700 2022-01-31T19:45:34.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:34.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:34.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:34.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:34.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:34.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:34.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:34.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:34.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:34.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:34.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:34.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:34.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:34.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:34.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:34.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:34.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:34.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:34.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:34.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:34.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:34.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:34.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:34.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:34.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_buffer_manager: 0x5628b08ce330 2022-01-31T19:45:34.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:34.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:34.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:34.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:34.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:34.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:34.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:34.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:34.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:34.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:34.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:34.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:34.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:34.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:34.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.row_cache: None 2022-01-31T19:45:34.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:34.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:34.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:34.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:34.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:34.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:34.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:34.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:34.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:34.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:34.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:34.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:34.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:34.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:34.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:34.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:34.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:34.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:34.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:34.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:34.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:34.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:34.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:34.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:34.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:34.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:34.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:34.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:34.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:34.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:34.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:34.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:34.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:34.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:34.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:34.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:34.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:34.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:34.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:34.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:34.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:34.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:34.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:34.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:34.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000018 2022-01-31T19:45:34.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:34.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:34.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.merge_operator: 2022-01-31T19:45:34.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:34.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:34.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:34.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:34.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:34.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5628b07b1c20) 2022-01-31T19:45:34.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:34.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:34.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:34.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:34.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:34.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:34.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:34.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:34.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:34.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:34.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:34.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x5628b0816170 2022-01-31T19:45:34.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:34.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:34.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:34.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:34.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:34.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:34.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:34.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:34.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:34.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:34.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:34.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:34.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:34.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:34.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:34.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:34.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:34.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:34.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:34.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:34.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:34.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:34.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:34.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:34.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:34.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:34.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:34.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:34.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:34.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:34.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:34.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:34.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:34.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:34.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:34.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:34.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:34.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:34.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:34.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:34.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:34.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:34.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:34.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:34.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:34.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:34.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:34.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:34.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:34.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:34.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:34.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:34.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:34.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:34.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:34.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:34.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:34.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:34.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:34.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:34.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:34.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:34.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:34.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:34.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:34.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:34.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:34.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:34.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:34.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:34.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:34.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:34.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:34.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:34.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:34.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:34.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:34.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:34.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:34.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:34.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:34.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:34.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:34.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:34.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:34.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:34.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:34.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:34.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:34.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:34.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:34.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:34.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:34.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:34.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:34.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:34.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:34.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:34.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000018 succeeded,manifest_file_number is 18, next_file_number is 20, last_sequence is 8022, log_number is 13,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:34.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 13 2022-01-31T19:45:34.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4083] Creating manifest 22 2022-01-31T19:45:34.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658333920397, "job": 1, "event": "recovery_started", "wal_files": [19]} 2022-01-31T19:45:34.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #19 mode 2 2022-01-31T19:45:34.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4083] Creating manifest 23 2022-01-31T19:45:34.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658333920741, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:34.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000019.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:34.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5628b07fca80 2022-01-31T19:45:34.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB pointer 0x5628b080c000 2022-01-31T19:45:34.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:34.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:34.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:34.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:34.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:34.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:34.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:34.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:34.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:34.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:34.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:34.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:34.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:34.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:34.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:34.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:34.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:34.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:34.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:34.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:34.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:34.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:34.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:34.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:34.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:34.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:34.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:34.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:34.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:34.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:34.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:34.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:34.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:34.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:34.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:34.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:34.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:34.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:34.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:34.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:34.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:34.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:34.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:34.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:34.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:34.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:34.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:34.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:34.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:34.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:34.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:34.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:34.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:34.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:34.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:34.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:34.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: main() 2022-01-31T19:45:34.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: __libc_start_main() 2022-01-31T19:45:34.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: _start() 2022-01-31T19:45:34.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: *** Caught signal (Aborted) ** 2022-01-31T19:45:34.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:33.924+0000 7f2528f74880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: main() 2022-01-31T19:45:34.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: __libc_start_main() 2022-01-31T19:45:34.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: _start() 2022-01-31T19:45:34.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: gsignal() 2022-01-31T19:45:34.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: abort() 2022-01-31T19:45:34.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 12: main() 2022-01-31T19:45:34.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 13: __libc_start_main() 2022-01-31T19:45:34.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 14: _start() 2022-01-31T19:45:34.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: *** Caught signal (Aborted) ** 2022-01-31T19:45:34.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:34.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:34.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:34.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:34.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:34.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:34.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:33.925+0000 7f2528f74880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:34.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: gsignal() 2022-01-31T19:45:34.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: abort() 2022-01-31T19:45:34.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 12: main() 2022-01-31T19:45:34.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 13: __libc_start_main() 2022-01-31T19:45:34.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 14: _start() 2022-01-31T19:45:34.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:34.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command assert hook 0x5628b07aca50 2022-01-31T19:45:34.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command abort hook 0x5628b07aca50 2022-01-31T19:45:34.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command leak_some_memory hook 0x5628b07aca50 2022-01-31T19:45:34.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perfcounters_dump hook 0x5628b07aca50 2022-01-31T19:45:34.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command 1 hook 0x5628b07aca50 2022-01-31T19:45:34.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf dump hook 0x5628b07aca50 2022-01-31T19:45:34.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perfcounters_schema hook 0x5628b07aca50 2022-01-31T19:45:34.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf histogram dump hook 0x5628b07aca50 2022-01-31T19:45:34.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command 2 hook 0x5628b07aca50 2022-01-31T19:45:34.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf schema hook 0x5628b07aca50 2022-01-31T19:45:34.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf histogram schema hook 0x5628b07aca50 2022-01-31T19:45:34.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf reset hook 0x5628b07aca50 2022-01-31T19:45:34.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config show hook 0x5628b07aca50 2022-01-31T19:45:34.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config help hook 0x5628b07aca50 2022-01-31T19:45:34.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config set hook 0x5628b07aca50 2022-01-31T19:45:34.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config unset hook 0x5628b07aca50 2022-01-31T19:45:34.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config get hook 0x5628b07aca50 2022-01-31T19:45:34.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config diff hook 0x5628b07aca50 2022-01-31T19:45:34.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config diff get hook 0x5628b07aca50 2022-01-31T19:45:34.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command injectargs hook 0x5628b07aca50 2022-01-31T19:45:34.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command log flush hook 0x5628b07aca50 2022-01-31T19:45:34.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command log dump hook 0x5628b07aca50 2022-01-31T19:45:34.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command log reopen hook 0x5628b07aca50 2022-01-31T19:45:34.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command dump_mempools hook 0x5628b089a068 2022-01-31T19:45:34.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:34.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:34.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:34.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:34.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:34.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command 0 hook 0x5628b07b10a0 2022-01-31T19:45:34.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command version hook 0x5628b07b10a0 2022-01-31T19:45:34.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command git_version hook 0x5628b07b10a0 2022-01-31T19:45:34.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command help hook 0x5628b07ac9f0 2022-01-31T19:45:34.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command get_command_descriptions hook 0x5628b07ac9e0 2022-01-31T19:45:34.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) entry start 2022-01-31T19:45:34.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: load: jerasure load: lrc 2022-01-31T19:45:34.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:34.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:34.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:34.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:34.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:34.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:34.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:34.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:34.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:34.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:34.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB SUMMARY 2022-01-31T19:45:34.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB Session ID: 71CIRQWJIUABD8LIDS9M 2022-01-31T19:45:34.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:34.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:34.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: MANIFEST file: MANIFEST-000018 size: 207 Bytes 2022-01-31T19:45:34.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:34.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000019.log size: 0 ; 2022-01-31T19:45:34.208 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:34.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:34.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:34.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:34.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.env: 0x5628ae1bec00 2022-01-31T19:45:34.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:34.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.info_log: 0x5628b07e6700 2022-01-31T19:45:34.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:34.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:34.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:34.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:34.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:34.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:34.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:34.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:34.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:34.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:34.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:34.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:34.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:34.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:34.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:34.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:34.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:34.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:34.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:34.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:34.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:34.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:34.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:34.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:34.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_buffer_manager: 0x5628b08ce330 2022-01-31T19:45:34.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:34.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:34.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:34.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:34.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:34.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:34.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:34.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:34.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:34.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:34.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:34.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:34.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:34.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:34.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.row_cache: None 2022-01-31T19:45:34.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:34.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:34.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:34.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:34.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:34.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:34.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:34.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:34.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:34.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:34.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:34.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:34.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:34.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:34.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:34.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:34.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:34.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:34.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:34.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:34.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:34.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:34.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:34.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:34.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:34.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:34.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:34.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:34.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:34.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:34.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:34.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:34.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:34.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:34.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:34.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:34.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:34.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:34.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:34.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:34.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:34.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:34.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:34.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:34.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000018 2022-01-31T19:45:34.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:34.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:34.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.merge_operator: 2022-01-31T19:45:34.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:34.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:34.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:34.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:34.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:34.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5628b07b1c20) 2022-01-31T19:45:34.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:34.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:34.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:34.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:34.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:34.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:34.234 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:34.234 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:34.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:34.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:34.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:34.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x5628b0816170 2022-01-31T19:45:34.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:34.237 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:34.237 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:34.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:34.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:34.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:34.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:34.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:34.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:34.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:34.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:34.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:34.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:34.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:34.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:34.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:34.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:34.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:34.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:34.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:34.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:34.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:34.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:34.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:34.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:34.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:34.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:34.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:34.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:34.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:34.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:34.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:34.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:34.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:34.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:34.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:34.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:34.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:34.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:34.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:34.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:34.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:34.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:34.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:34.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:34.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:34.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:34.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:34.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:34.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:34.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:34.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:34.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:34.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:34.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:34.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:34.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:34.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:34.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:34.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:34.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:34.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:34.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:34.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:34.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:34.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:34.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:34.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:34.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:34.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:34.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:34.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:34.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:34.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:34.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:34.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:34.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:34.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:34.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:34.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:34.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:34.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:34.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:34.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:34.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:34.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:34.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:34.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:34.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:34.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:34.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:34.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:34.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:34.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:34.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:34.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:34.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:34.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:34.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000018 succeeded,manifest_file_number is 18, next_file_number is 20, last_sequence is 8022, log_number is 13,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:34.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 13 2022-01-31T19:45:34.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4083] Creating manifest 22 2022-01-31T19:45:34.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658333920397, "job": 1, "event": "recovery_started", "wal_files": [19]} 2022-01-31T19:45:34.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #19 mode 2 2022-01-31T19:45:34.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4083] Creating manifest 23 2022-01-31T19:45:34.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658333920741, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:34.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000019.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:34.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5628b07fca80 2022-01-31T19:45:34.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB pointer 0x5628b080c000 2022-01-31T19:45:34.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:34.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:34.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:34.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:34.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:34.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:34.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:34.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:34.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:34.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:34.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:34.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:34.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:34.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:34.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:34.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:34.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:34.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:34.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:34.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: cephx 2022-01-31T19:45:34.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: cephx 2022-01-31T19:45:34.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: cephx 2022-01-31T19:45:34.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: none 2022-01-31T19:45:34.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:34.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:34.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: cephx 2022-01-31T19:45:34.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: cephx 2022-01-31T19:45:34.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: cephx 2022-01-31T19:45:34.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: none 2022-01-31T19:45:34.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:34.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: adding auth protocol: cephx 2022-01-31T19:45:34.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: adding auth protocol: cephx 2022-01-31T19:45:34.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:34.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:34.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:34.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:34.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:34.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:34.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:34.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:34.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:34.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:34.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:34.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:34.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:34.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:34.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:34.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:34.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:34.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:34.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:34.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:34.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:34.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:34.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:34.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:34.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:34.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:34.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:34.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:34.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:34.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:34.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:34.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:34.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:34.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:34.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:34.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:34.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:34.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:34.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: -288> 2022-01-31T19:45:33.924+0000 7f2528f74880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: *** Caught signal (Aborted) ** 2022-01-31T19:45:34.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:34.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:34.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:34.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:34.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:34.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:34.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: main() 2022-01-31T19:45:34.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: __libc_start_main() 2022-01-31T19:45:34.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: _start() 2022-01-31T19:45:34.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: -287> 2022-01-31T19:45:33.925+0000 7f2528f74880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:34.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: gsignal() 2022-01-31T19:45:34.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: abort() 2022-01-31T19:45:34.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command assert hook 0x5628b07aca50 2022-01-31T19:45:34.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 12: main() 2022-01-31T19:45:34.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command abort hook 0x5628b07aca50 2022-01-31T19:45:34.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 13: __libc_start_main() 2022-01-31T19:45:34.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 14: _start() 2022-01-31T19:45:34.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command leak_some_memory hook 0x5628b07aca50 2022-01-31T19:45:34.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:34.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perfcounters_dump hook 0x5628b07aca50 2022-01-31T19:45:34.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command 1 hook 0x5628b07aca50 2022-01-31T19:45:34.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf dump hook 0x5628b07aca50 2022-01-31T19:45:34.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perfcounters_schema hook 0x5628b07aca50 2022-01-31T19:45:34.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf histogram dump hook 0x5628b07aca50 2022-01-31T19:45:34.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command 2 hook 0x5628b07aca50 2022-01-31T19:45:34.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf schema hook 0x5628b07aca50 2022-01-31T19:45:34.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf histogram schema hook 0x5628b07aca50 2022-01-31T19:45:34.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command perf reset hook 0x5628b07aca50 2022-01-31T19:45:34.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config show hook 0x5628b07aca50 2022-01-31T19:45:34.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config help hook 0x5628b07aca50 2022-01-31T19:45:34.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config set hook 0x5628b07aca50 2022-01-31T19:45:34.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config unset hook 0x5628b07aca50 2022-01-31T19:45:34.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config get hook 0x5628b07aca50 2022-01-31T19:45:34.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config diff hook 0x5628b07aca50 2022-01-31T19:45:34.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command config diff get hook 0x5628b07aca50 2022-01-31T19:45:34.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command injectargs hook 0x5628b07aca50 2022-01-31T19:45:34.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command log flush hook 0x5628b07aca50 2022-01-31T19:45:34.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command log dump hook 0x5628b07aca50 2022-01-31T19:45:34.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command log reopen hook 0x5628b07aca50 2022-01-31T19:45:34.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command dump_mempools hook 0x5628b089a068 2022-01-31T19:45:34.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:34.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:34.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:34.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:34.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:34.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command 0 hook 0x5628b07b10a0 2022-01-31T19:45:34.323 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command version hook 0x5628b07b10a0 2022-01-31T19:45:34.323 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command git_version hook 0x5628b07b10a0 2022-01-31T19:45:34.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command help hook 0x5628b07ac9f0 2022-01-31T19:45:34.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) register_command get_command_descriptions hook 0x5628b07ac9e0 2022-01-31T19:45:34.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: asok(0x5628b0a94000) entry start 2022-01-31T19:45:34.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: load: jerasure load: lrc 2022-01-31T19:45:34.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:34.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:34.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:34.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:34.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:34.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:34.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:34.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:34.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:34.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:34.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB SUMMARY 2022-01-31T19:45:34.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB Session ID: 71CIRQWJIUABD8LIDS9M 2022-01-31T19:45:34.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:34.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:34.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: MANIFEST file: MANIFEST-000018 size: 207 Bytes 2022-01-31T19:45:34.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:34.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000019.log size: 0 ; 2022-01-31T19:45:34.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:34.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:34.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:34.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:34.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.env: 0x5628ae1bec00 2022-01-31T19:45:34.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:34.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.info_log: 0x5628b07e6700 2022-01-31T19:45:34.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:34.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:34.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:34.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:34.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:34.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:34.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:34.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:34.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:34.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:34.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:34.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:34.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:34.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:34.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:34.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:34.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:34.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:34.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:34.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:34.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:34.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:34.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:34.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:34.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:34.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_buffer_manager: 0x5628b08ce330 2022-01-31T19:45:34.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:34.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:34.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:34.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:34.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:34.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:34.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:34.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:34.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:34.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:34.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:34.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:34.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:34.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:34.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.row_cache: None 2022-01-31T19:45:34.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:34.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:34.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:34.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:34.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:34.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:34.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:34.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:34.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:34.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:34.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:34.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:34.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:34.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:34.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:34.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:34.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:34.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:34.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:34.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:34.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:34.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:34.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:34.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:34.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:34.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:34.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:34.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:34.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:34.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:34.352 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:34.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:34.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:34.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:34.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:34.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:34.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:34.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:34.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:34.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:34.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:34.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:34.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:34.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:34.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000018 2022-01-31T19:45:34.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:34.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:34.357 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.merge_operator: 2022-01-31T19:45:34.357 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:34.357 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:34.357 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:34.358 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:34.358 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:34.358 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5628b07b1c20) 2022-01-31T19:45:34.358 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:34.359 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:34.359 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:34.359 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:34.359 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:34.360 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:34.360 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:34.360 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:34.360 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:34.360 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:34.361 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:34.361 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x5628b0816170 2022-01-31T19:45:34.361 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:34.361 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:34.361 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:34.362 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:34.362 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:34.362 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:34.362 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:34.363 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:34.363 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:34.363 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:34.363 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:34.363 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:34.364 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:34.364 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:34.364 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:34.364 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:34.365 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:34.365 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:34.366 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:34.366 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:34.366 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:34.366 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:34.367 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:34.367 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:34.367 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:34.367 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:34.368 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:34.368 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:34.368 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:34.368 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:34.369 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:34.369 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:34.369 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:34.370 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:34.370 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:34.370 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:34.370 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:34.371 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:34.371 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:34.371 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:34.371 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:34.372 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:34.372 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:34.372 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:34.372 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:34.373 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:34.373 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:34.373 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:34.374 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:34.374 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:34.374 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:34.375 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:34.375 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:34.375 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:34.375 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:34.375 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:34.376 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:34.376 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:34.376 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:34.376 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:34.377 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:34.377 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:34.377 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:34.377 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:34.378 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:34.378 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:34.378 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:34.378 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:34.379 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:34.379 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:34.379 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:34.379 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:34.379 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:34.380 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:34.380 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:34.380 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:34.380 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:34.381 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:34.381 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:34.381 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:34.381 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:34.381 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:34.382 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:34.382 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:34.383 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:34.383 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:34.383 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:34.383 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:34.383 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:34.384 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:34.384 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:34.384 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:34.384 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:34.385 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:34.385 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:34.385 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:34.385 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:34.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:34.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000018 succeeded,manifest_file_number is 18, next_file_number is 20, last_sequence is 8022, log_number is 13,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:34.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 13 2022-01-31T19:45:34.386 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4083] Creating manifest 22 2022-01-31T19:45:34.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658333920397, "job": 1, "event": "recovery_started", "wal_files": [19]} 2022-01-31T19:45:34.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #19 mode 2 2022-01-31T19:45:34.387 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:4083] Creating manifest 23 2022-01-31T19:45:34.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:34.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658333920741, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:34.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000019.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:34.388 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5628b07fca80 2022-01-31T19:45:34.389 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: DB pointer 0x5628b080c000 2022-01-31T19:45:34.389 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:34.389 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:34.389 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:34.389 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.390 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:34.390 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:34.390 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:34.391 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:34.391 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:34.391 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:34.391 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.392 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.392 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.392 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.392 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.392 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.393 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.393 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.393 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.393 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.394 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.394 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.394 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.394 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.395 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.395 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:34.395 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:34.395 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:34.396 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.396 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.396 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:34.396 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:34.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.397 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.398 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.398 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.398 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.398 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.399 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:34.399 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.399 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:34.400 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:34.400 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:34.400 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:34.400 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.401 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:34.401 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:34.401 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:34.401 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:34.401 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.402 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:34.402 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:34.402 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.402 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:34.403 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: cephx 2022-01-31T19:45:34.403 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: cephx 2022-01-31T19:45:34.403 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: cephx 2022-01-31T19:45:34.403 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding auth protocol: none 2022-01-31T19:45:34.403 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.404 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.404 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.404 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.404 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.405 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.405 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.405 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.405 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.406 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.406 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: crc 2022-01-31T19:45:34.406 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154e140) adding con mode: secure 2022-01-31T19:45:34.406 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:34.407 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:34.407 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: cephx 2022-01-31T19:45:34.407 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: cephx 2022-01-31T19:45:34.408 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: cephx 2022-01-31T19:45:34.408 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding auth protocol: none 2022-01-31T19:45:34.408 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.408 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.408 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.409 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.410 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.410 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.410 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.410 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: crc 2022-01-31T19:45:34.411 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: AuthRegistry(0x5628b154ea40) adding con mode: secure 2022-01-31T19:45:34.411 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:34.411 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: adding auth protocol: cephx 2022-01-31T19:45:34.411 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: adding auth protocol: cephx 2022-01-31T19:45:34.412 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:34.412 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:34.412 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:34.412 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:34.413 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:34.413 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:34.413 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:34.413 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:34.414 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:34.414 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.414 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:34.414 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:34.414 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:34.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:34.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:34.415 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:34.416 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:34.417 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:34.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:34.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:34.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:34.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:34.418 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:34.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:34.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:34.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:34.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:34.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:34.421 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.422 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.423 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.424 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:34.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:34.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:34.425 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 ceph-mon[102298]: *** Caught signal (Aborted) ** 2022-01-31T19:45:34.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:34.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.426 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:34.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:34.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.427 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.428 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:34.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:34.429 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:34.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:34.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: -1> 2022-01-31T19:45:33.924+0000 7f2528f74880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f2528f74880 time 2022-01-31T19:45:33.923539+0000 2022-01-31T19:45:34.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:34.430 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f25281ce1d4] 2022-01-31T19:45:34.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.431 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.432 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: main() 2022-01-31T19:45:34.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: __libc_start_main() 2022-01-31T19:45:34.433 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: _start() 2022-01-31T19:45:34.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 0> 2022-01-31T19:45:33.925+0000 7f2528f74880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:34.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: in thread 7f2528f74880 thread_name:ceph-mon 2022-01-31T19:45:34.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.434 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:34.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2526161c20] 2022-01-31T19:45:34.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2: gsignal() 2022-01-31T19:45:34.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 3: abort() 2022-01-31T19:45:34.435 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f25281ce232] 2022-01-31T19:45:34.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f25281ce3f5] 2022-01-31T19:45:34.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f2528798838] 2022-01-31T19:45:34.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5628ada2bfa3] 2022-01-31T19:45:34.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5628ad94fb0e] 2022-01-31T19:45:34.436 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5628ad7e151c] 2022-01-31T19:45:34.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 10: (Monitor::init_paxos()+0x10c) [0x5628ad7e182c] 2022-01-31T19:45:34.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 11: (Monitor::preinit()+0xd30) [0x5628ad80f2a0] 2022-01-31T19:45:34.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 12: main() 2022-01-31T19:45:34.437 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 13: __libc_start_main() 2022-01-31T19:45:34.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 14: _start() 2022-01-31T19:45:34.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:34.438 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:33 smithi167 conmon[102274]: 2022-01-31T19:45:34.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:34 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Main process exited, code=exited, status=134/n/a 2022-01-31T19:45:34.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:34 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Failed with result 'exit-code'. 2022-01-31T19:45:44.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Service RestartSec=10s expired, scheduling restart. 2022-01-31T19:45:44.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Scheduled restart job, restart counter is at 2. 2022-01-31T19:45:44.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:44.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:45:45.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 podman[102530]: 2022-01-31T19:45:45.031 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 bash[102377]: 99ce00f783e89455db01ae45921e078b6dae5515742384842715df40c488ef38 2022-01-31T19:45:45.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:45.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:45.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:45.032 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: load: jerasure load: lrc 2022-01-31T19:45:45.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:45.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:45.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:45.033 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB SUMMARY 2022-01-31T19:45:45.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB Session ID: N5QBT075AJK06BCR8EH8 2022-01-31T19:45:45.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:45.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:45.034 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: MANIFEST file: MANIFEST-000023 size: 207 Bytes 2022-01-31T19:45:45.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:45.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000024.log size: 0 ; 2022-01-31T19:45:45.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:45.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:45.035 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:45.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:45.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.env: 0x5604f1b60c00 2022-01-31T19:45:45.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:45.036 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.info_log: 0x5604f24e2700 2022-01-31T19:45:45.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:45.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:45.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:45.037 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:45.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:45.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:45.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:45.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:45.038 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:45.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:45.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:45.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:45.039 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:45.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:45.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:45.040 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:45.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:45.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:45.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:45.041 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:45.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:45.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:45.042 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:45.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:45.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_buffer_manager: 0x5604f25ca330 2022-01-31T19:45:45.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:45.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:45.043 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:45.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:45.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:45.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:45.044 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:45.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:45.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:45.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:45.045 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:45.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:45.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:45.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:45.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.row_cache: None 2022-01-31T19:45:45.046 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:45.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:45.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:45.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:45.047 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:45.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:45.048 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:45.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:45.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:45.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:45.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:45.049 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:45.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:45.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:45.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:45.050 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:45.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:45.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:45.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:45.051 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:45.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:45.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:45.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:45.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:45.052 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:45.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:45.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:45.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:45.053 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:45.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:45.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:45.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:45.054 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:45.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:45.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:45.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:45.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:45.055 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:45.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:45.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:45.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:45.056 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:45.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:45.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:45.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000023 2022-01-31T19:45:45.057 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:45.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:45.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.merge_operator: 2022-01-31T19:45:45.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:45.058 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:45.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:45.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:45.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:45.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5604f24adc20) 2022-01-31T19:45:45.059 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:45.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:45.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:45.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:45.060 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:45.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:45.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:45.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:45.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:45.061 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:45.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:45.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x5604f2512170 2022-01-31T19:45:45.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:45.062 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:45.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:45.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:45.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:45.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:45.063 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:45.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:45.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:45.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:45.064 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:45.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:45.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:45.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:45.065 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:45.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:45.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:45.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:45.066 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:45.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:45.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:45.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:45.067 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:45.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:45.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:45.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:45.068 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:45.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:45.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:45.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:45.069 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:45.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:45.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:45.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:45.070 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:45.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:45.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:45.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:45.071 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:45.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:45.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:45.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:45.072 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:45.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:45.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:45.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:45.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:45.073 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:45.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:45.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:45.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:45.074 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:45.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:45.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:45.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:45.075 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:45.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:45.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:45.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:45.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:45.076 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:45.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:45.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:45.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:45.077 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:45.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:45.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:45.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:45.078 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:45.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:45.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:45.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:45.079 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:45.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:45.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:45.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:45.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:45.080 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:45.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:45.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:45.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:45.081 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:45.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:45.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:45.082 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:45.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:45.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:45.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:45.083 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:45.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:45.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:45.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:45.084 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:45.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:45.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:45.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:45.085 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:45.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:45.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000023 succeeded,manifest_file_number is 23, next_file_number is 25, last_sequence is 8022, log_number is 20,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:45.086 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 20 2022-01-31T19:45:45.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4083] Creating manifest 27 2022-01-31T19:45:45.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658344932593, "job": 1, "event": "recovery_started", "wal_files": [24]} 2022-01-31T19:45:45.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #24 mode 2 2022-01-31T19:45:45.087 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4083] Creating manifest 28 2022-01-31T19:45:45.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658344932933, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:45.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000024.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:45.088 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5604f24f8a80 2022-01-31T19:45:45.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB pointer 0x5604f2508000 2022-01-31T19:45:45.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:45.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:45.089 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:45.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:45.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:45.090 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:45.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:45.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:45.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:45.091 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.092 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.093 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.094 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:45.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:45.095 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:45.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:45.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.096 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:45.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.097 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.098 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.099 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:45.100 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:45.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:45.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.101 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:45.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:45.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:45.102 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:45.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:45.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:45.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:45.103 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:45.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:45.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:45.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.104 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:45.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:45.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:45.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:45.105 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:45.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:45.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:45.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:45.106 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:45.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:45.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:45.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:45.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:45.107 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:45.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:45.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:45.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:45.108 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:45.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:45.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:45.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:45.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:45.109 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:45.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:45.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:45.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:45.110 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.111 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.112 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.113 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.114 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:45.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:45.115 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:45.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.116 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.117 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: main() 2022-01-31T19:45:45.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: __libc_start_main() 2022-01-31T19:45:45.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: _start() 2022-01-31T19:45:45.118 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: *** Caught signal (Aborted) ** 2022-01-31T19:45:45.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:44.935+0000 7fd9353ba880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.119 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.120 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.121 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: main() 2022-01-31T19:45:45.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: __libc_start_main() 2022-01-31T19:45:45.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: _start() 2022-01-31T19:45:45.122 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: gsignal() 2022-01-31T19:45:45.123 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: abort() 2022-01-31T19:45:45.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.124 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.125 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 12: main() 2022-01-31T19:45:45.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 13: __libc_start_main() 2022-01-31T19:45:45.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 14: _start() 2022-01-31T19:45:45.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:44.936+0000 7fd9353ba880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:45.126 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: *** Caught signal (Aborted) ** 2022-01-31T19:45:45.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.127 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:45.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:45.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.128 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.129 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:45.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:45.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:45.130 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:45.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.131 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: gsignal() 2022-01-31T19:45:45.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: abort() 2022-01-31T19:45:45.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.132 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.133 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 12: main() 2022-01-31T19:45:45.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 13: __libc_start_main() 2022-01-31T19:45:45.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 14: _start() 2022-01-31T19:45:45.134 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:45.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command assert hook 0x5604f24a8a50 2022-01-31T19:45:45.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command abort hook 0x5604f24a8a50 2022-01-31T19:45:45.135 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command leak_some_memory hook 0x5604f24a8a50 2022-01-31T19:45:45.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perfcounters_dump hook 0x5604f24a8a50 2022-01-31T19:45:45.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command 1 hook 0x5604f24a8a50 2022-01-31T19:45:45.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf dump hook 0x5604f24a8a50 2022-01-31T19:45:45.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perfcounters_schema hook 0x5604f24a8a50 2022-01-31T19:45:45.136 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf histogram dump hook 0x5604f24a8a50 2022-01-31T19:45:45.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command 2 hook 0x5604f24a8a50 2022-01-31T19:45:45.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf schema hook 0x5604f24a8a50 2022-01-31T19:45:45.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf histogram schema hook 0x5604f24a8a50 2022-01-31T19:45:45.137 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf reset hook 0x5604f24a8a50 2022-01-31T19:45:45.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config show hook 0x5604f24a8a50 2022-01-31T19:45:45.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config help hook 0x5604f24a8a50 2022-01-31T19:45:45.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config set hook 0x5604f24a8a50 2022-01-31T19:45:45.138 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config unset hook 0x5604f24a8a50 2022-01-31T19:45:45.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config get hook 0x5604f24a8a50 2022-01-31T19:45:45.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config diff hook 0x5604f24a8a50 2022-01-31T19:45:45.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config diff get hook 0x5604f24a8a50 2022-01-31T19:45:45.139 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command injectargs hook 0x5604f24a8a50 2022-01-31T19:45:45.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command log flush hook 0x5604f24a8a50 2022-01-31T19:45:45.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command log dump hook 0x5604f24a8a50 2022-01-31T19:45:45.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command log reopen hook 0x5604f24a8a50 2022-01-31T19:45:45.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command dump_mempools hook 0x5604f2596068 2022-01-31T19:45:45.140 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:45.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:45.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:45.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:45.141 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:45.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command 0 hook 0x5604f24ad0a0 2022-01-31T19:45:45.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command version hook 0x5604f24ad0a0 2022-01-31T19:45:45.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command git_version hook 0x5604f24ad0a0 2022-01-31T19:45:45.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command help hook 0x5604f24a89f0 2022-01-31T19:45:45.142 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command get_command_descriptions hook 0x5604f24a89e0 2022-01-31T19:45:45.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) entry start 2022-01-31T19:45:45.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: load: jerasure load: lrc 2022-01-31T19:45:45.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:45.143 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:45.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:45.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:45.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:45.144 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:45.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:45.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:45.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:45.145 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:45.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB SUMMARY 2022-01-31T19:45:45.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB Session ID: N5QBT075AJK06BCR8EH8 2022-01-31T19:45:45.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:45.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:45.146 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: MANIFEST file: MANIFEST-000023 size: 207 Bytes 2022-01-31T19:45:45.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:45.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000024.log size: 0 ; 2022-01-31T19:45:45.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:45.147 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:45.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:45.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:45.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.env: 0x5604f1b60c00 2022-01-31T19:45:45.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:45.148 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.info_log: 0x5604f24e2700 2022-01-31T19:45:45.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:45.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:45.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:45.149 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:45.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:45.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:45.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:45.150 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:45.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:45.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:45.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:45.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:45.151 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:45.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:45.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:45.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:45.152 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:45.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:45.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:45.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:45.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:45.153 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:45.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:45.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:45.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_buffer_manager: 0x5604f25ca330 2022-01-31T19:45:45.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:45.154 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:45.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:45.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:45.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:45.155 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:45.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:45.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:45.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:45.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:45.156 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:45.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:45.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:45.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:45.157 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.row_cache: None 2022-01-31T19:45:45.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:45.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:45.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:45.158 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:45.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:45.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:45.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:45.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:45.159 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:45.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:45.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:45.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:45.160 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:45.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:45.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:45.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:45.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:45.161 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:45.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:45.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:45.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:45.162 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:45.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:45.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:45.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:45.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:45.163 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:45.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:45.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:45.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:45.164 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:45.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:45.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:45.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:45.165 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:45.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:45.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:45.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:45.166 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:45.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:45.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:45.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:45.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:45.167 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:45.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000023 2022-01-31T19:45:45.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:45.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:45.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.merge_operator: 2022-01-31T19:45:45.168 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:45.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:45.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:45.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:45.169 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5604f24adc20) 2022-01-31T19:45:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:45.170 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:45.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:45.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:45.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:45.171 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:45.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:45.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:45.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:45.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x5604f2512170 2022-01-31T19:45:45.172 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:45.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:45.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:45.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:45.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:45.173 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:45.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:45.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:45.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:45.174 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:45.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:45.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:45.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:45.175 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:45.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:45.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:45.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:45.176 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:45.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:45.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:45.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:45.177 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:45.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:45.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:45.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:45.178 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:45.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:45.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:45.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:45.179 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:45.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:45.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:45.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:45.180 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:45.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:45.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:45.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:45.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:45.181 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:45.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:45.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:45.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:45.182 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:45.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:45.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:45.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:45.183 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:45.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:45.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:45.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:45.184 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:45.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:45.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:45.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:45.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:45.185 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:45.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:45.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:45.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:45.186 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:45.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:45.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:45.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:45.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:45.187 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:45.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:45.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:45.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:45.188 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:45.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:45.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:45.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:45.189 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:45.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:45.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:45.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:45.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:45.190 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:45.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:45.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:45.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:45.191 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:45.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:45.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:45.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:45.192 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:45.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:45.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:45.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:45.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:45.193 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:45.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:45.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:45.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:45.194 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:45.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:45.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:45.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:45.195 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000023 succeeded,manifest_file_number is 23, next_file_number is 25, last_sequence is 8022, log_number is 20,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:45.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 20 2022-01-31T19:45:45.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4083] Creating manifest 27 2022-01-31T19:45:45.196 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658344932593, "job": 1, "event": "recovery_started", "wal_files": [24]} 2022-01-31T19:45:45.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #24 mode 2 2022-01-31T19:45:45.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4083] Creating manifest 28 2022-01-31T19:45:45.197 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658344932933, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:45.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000024.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:45.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5604f24f8a80 2022-01-31T19:45:45.198 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB pointer 0x5604f2508000 2022-01-31T19:45:45.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:45.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:45.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:45.199 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:45.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:45.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:45.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:45.200 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:45.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:45.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.201 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.202 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.203 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.204 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.205 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:45.205 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:45.205 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:45.205 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.206 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.206 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:45.206 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.206 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:45.207 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.207 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.207 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.207 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.207 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.208 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.208 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.208 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.209 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:45.210 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:45.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:45.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.211 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:45.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:45.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: cephx 2022-01-31T19:45:45.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: cephx 2022-01-31T19:45:45.212 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: cephx 2022-01-31T19:45:45.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: none 2022-01-31T19:45:45.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.213 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.214 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.215 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:45.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:45.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: cephx 2022-01-31T19:45:45.216 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: cephx 2022-01-31T19:45:45.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: cephx 2022-01-31T19:45:45.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: none 2022-01-31T19:45:45.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.217 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.218 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.219 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:45.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: adding auth protocol: cephx 2022-01-31T19:45:45.220 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: adding auth protocol: cephx 2022-01-31T19:45:45.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:45.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:45.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:45.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:45.221 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:45.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:45.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:45.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:45.222 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:45.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:45.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:45.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:45.223 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:45.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:45.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:45.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:45.224 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:45.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:45.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:45.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:45.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:45.225 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:45.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:45.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:45.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:45.226 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:45.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:45.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:45.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:45.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:45.227 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:45.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:45.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:45.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:45.228 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:45.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.229 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.230 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.231 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.232 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:45.233 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:45.234 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:45.234 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: *** Caught signal (Aborted) ** 2022-01-31T19:45:45.234 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.234 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:45.235 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:45.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.236 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.237 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.237 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.237 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.237 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:45.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:45.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:45.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:45.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: -288> 2022-01-31T19:45:44.935+0000 7fd9353ba880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.238 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.239 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.240 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: main() 2022-01-31T19:45:45.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: __libc_start_main() 2022-01-31T19:45:45.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: _start() 2022-01-31T19:45:45.241 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: -287> 2022-01-31T19:45:44.936+0000 7fd9353ba880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:45.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.242 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: gsignal() 2022-01-31T19:45:45.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: abort() 2022-01-31T19:45:45.243 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.244 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command assert hook 0x5604f24a8a50 2022-01-31T19:45:45.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command abort hook 0x5604f24a8a50 2022-01-31T19:45:45.245 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 12: main() 2022-01-31T19:45:45.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 13: __libc_start_main() 2022-01-31T19:45:45.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command leak_some_memory hook 0x5604f24a8a50 2022-01-31T19:45:45.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 14: _start() 2022-01-31T19:45:45.246 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:45.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perfcounters_dump hook 0x5604f24a8a50 2022-01-31T19:45:45.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command 1 hook 0x5604f24a8a50 2022-01-31T19:45:45.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf dump hook 0x5604f24a8a50 2022-01-31T19:45:45.247 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perfcounters_schema hook 0x5604f24a8a50 2022-01-31T19:45:45.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf histogram dump hook 0x5604f24a8a50 2022-01-31T19:45:45.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command 2 hook 0x5604f24a8a50 2022-01-31T19:45:45.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf schema hook 0x5604f24a8a50 2022-01-31T19:45:45.248 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf histogram schema hook 0x5604f24a8a50 2022-01-31T19:45:45.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command perf reset hook 0x5604f24a8a50 2022-01-31T19:45:45.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config show hook 0x5604f24a8a50 2022-01-31T19:45:45.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config help hook 0x5604f24a8a50 2022-01-31T19:45:45.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config set hook 0x5604f24a8a50 2022-01-31T19:45:45.249 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config unset hook 0x5604f24a8a50 2022-01-31T19:45:45.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config get hook 0x5604f24a8a50 2022-01-31T19:45:45.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config diff hook 0x5604f24a8a50 2022-01-31T19:45:45.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command config diff get hook 0x5604f24a8a50 2022-01-31T19:45:45.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command injectargs hook 0x5604f24a8a50 2022-01-31T19:45:45.250 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command log flush hook 0x5604f24a8a50 2022-01-31T19:45:45.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command log dump hook 0x5604f24a8a50 2022-01-31T19:45:45.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command log reopen hook 0x5604f24a8a50 2022-01-31T19:45:45.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command dump_mempools hook 0x5604f2596068 2022-01-31T19:45:45.251 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:45.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:45.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:45.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:45.252 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:45.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command 0 hook 0x5604f24ad0a0 2022-01-31T19:45:45.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command version hook 0x5604f24ad0a0 2022-01-31T19:45:45.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command git_version hook 0x5604f24ad0a0 2022-01-31T19:45:45.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command help hook 0x5604f24a89f0 2022-01-31T19:45:45.253 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) register_command get_command_descriptions hook 0x5604f24a89e0 2022-01-31T19:45:45.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: asok(0x5604f2790000) entry start 2022-01-31T19:45:45.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: load: jerasure load: lrc 2022-01-31T19:45:45.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:45.254 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:45.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:45.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:45.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:45.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:45.255 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:45.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:45.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:45.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:45.256 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB SUMMARY 2022-01-31T19:45:45.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB Session ID: N5QBT075AJK06BCR8EH8 2022-01-31T19:45:45.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:45.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:45.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: MANIFEST file: MANIFEST-000023 size: 207 Bytes 2022-01-31T19:45:45.257 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:45.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000024.log size: 0 ; 2022-01-31T19:45:45.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:45.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:45.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:45.258 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:45.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.env: 0x5604f1b60c00 2022-01-31T19:45:45.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:45.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.info_log: 0x5604f24e2700 2022-01-31T19:45:45.259 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:45.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:45.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:45.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:45.260 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:45.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:45.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:45.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:45.261 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:45.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:45.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:45.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:45.262 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:45.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:45.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:45.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:45.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:45.263 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:45.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:45.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:45.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:45.264 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:45.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:45.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:45.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_buffer_manager: 0x5604f25ca330 2022-01-31T19:45:45.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:45.265 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:45.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:45.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:45.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:45.266 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:45.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:45.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:45.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:45.267 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:45.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:45.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:45.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:45.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:45.268 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.row_cache: None 2022-01-31T19:45:45.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:45.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:45.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:45.269 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:45.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:45.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:45.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:45.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:45.270 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:45.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:45.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:45.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:45.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:45.271 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:45.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:45.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:45.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:45.272 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:45.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:45.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:45.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:45.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:45.273 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:45.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:45.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:45.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:45.274 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:45.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:45.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:45.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:45.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:45.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:45.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:45.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:45.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:45.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:45.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:45.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:45.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:45.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:45.277 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:45.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:45.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:45.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:45.278 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000023 2022-01-31T19:45:45.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:45.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:45.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.merge_operator: 2022-01-31T19:45:45.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:45.279 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:45.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:45.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:45.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:45.280 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5604f24adc20) 2022-01-31T19:45:45.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:45.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:45.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:45.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:45.281 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:45.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:45.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:45.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:45.282 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:45.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:45.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:45.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x5604f2512170 2022-01-31T19:45:45.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:45.283 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:45.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:45.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:45.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:45.284 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:45.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:45.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:45.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:45.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:45.285 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:45.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:45.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:45.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:45.286 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:45.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:45.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:45.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:45.287 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:45.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:45.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:45.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:45.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:45.288 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:45.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:45.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:45.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:45.289 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:45.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:45.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:45.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:45.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:45.290 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:45.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:45.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:45.291 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:45.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:45.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:45.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:45.292 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:45.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:45.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:45.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:45.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:45.293 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:45.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:45.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:45.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:45.294 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:45.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:45.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:45.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:45.295 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:45.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:45.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:45.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:45.296 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:45.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:45.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:45.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:45.297 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:45.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:45.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:45.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:45.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:45.298 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:45.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:45.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:45.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:45.299 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:45.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:45.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:45.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:45.300 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:45.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:45.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:45.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:45.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:45.301 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:45.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:45.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:45.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:45.302 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:45.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:45.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:45.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:45.303 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:45.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:45.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:45.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:45.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:45.304 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:45.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:45.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:45.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:45.305 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:45.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:45.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:45.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.306 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000023 succeeded,manifest_file_number is 23, next_file_number is 25, last_sequence is 8022, log_number is 20,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:45.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 20 2022-01-31T19:45:45.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4083] Creating manifest 27 2022-01-31T19:45:45.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658344932593, "job": 1, "event": "recovery_started", "wal_files": [24]} 2022-01-31T19:45:45.307 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #24 mode 2 2022-01-31T19:45:45.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:4083] Creating manifest 28 2022-01-31T19:45:45.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:45.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658344932933, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:45.308 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000024.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:45.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5604f24f8a80 2022-01-31T19:45:45.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: DB pointer 0x5604f2508000 2022-01-31T19:45:45.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:45.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:45.309 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:45.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:45.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:45.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:45.310 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:45.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:45.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:45.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.311 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.312 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.313 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.314 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:45.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:45.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:45.315 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:45.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:45.316 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.317 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:45.318 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:45.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:45.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:45.319 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:45.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:45.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:45.320 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:45.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:45.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:45.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:45.321 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:45.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: cephx 2022-01-31T19:45:45.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: cephx 2022-01-31T19:45:45.322 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: cephx 2022-01-31T19:45:45.323 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding auth protocol: none 2022-01-31T19:45:45.323 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.323 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.323 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.324 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: crc 2022-01-31T19:45:45.325 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324a140) adding con mode: secure 2022-01-31T19:45:45.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:45.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:45.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: cephx 2022-01-31T19:45:45.326 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: cephx 2022-01-31T19:45:45.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: cephx 2022-01-31T19:45:45.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding auth protocol: none 2022-01-31T19:45:45.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.327 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.328 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.329 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: crc 2022-01-31T19:45:45.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: AuthRegistry(0x5604f324aa40) adding con mode: secure 2022-01-31T19:45:45.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:45.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: adding auth protocol: cephx 2022-01-31T19:45:45.330 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: adding auth protocol: cephx 2022-01-31T19:45:45.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:45.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:45.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:45.331 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:45.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:45.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:45.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:45.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:45.332 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:45.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:45.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:45.333 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:45.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:45.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:45.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:45.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:45.334 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:45.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:45.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:45.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:45.335 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:45.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:45.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:45.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:45.336 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:45.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:45.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:45.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:45.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:45.337 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:45.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:45.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:45.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:45.338 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:45.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:45.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.339 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:45.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.340 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.341 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.342 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:45.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:45.343 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:45.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 ceph-mon[102593]: *** Caught signal (Aborted) ** 2022-01-31T19:45:45.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:45.344 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:45.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:45.345 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.346 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.347 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:45.348 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:45.348 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:45.348 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:45.348 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: -1> 2022-01-31T19:45:44.935+0000 7fd9353ba880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fd9353ba880 time 2022-01-31T19:45:44.935743+0000 2022-01-31T19:45:45.348 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:45.349 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.349 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.349 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd9346141d4] 2022-01-31T19:45:45.349 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.350 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.350 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.350 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.350 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.350 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.351 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.351 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: main() 2022-01-31T19:45:45.351 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: __libc_start_main() 2022-01-31T19:45:45.351 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: _start() 2022-01-31T19:45:45.352 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.352 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 0> 2022-01-31T19:45:44.936+0000 7fd9353ba880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:45.352 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: in thread 7fd9353ba880 thread_name:ceph-mon 2022-01-31T19:45:45.352 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.352 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:45.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fd9325a7c20] 2022-01-31T19:45:45.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2: gsignal() 2022-01-31T19:45:45.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 3: abort() 2022-01-31T19:45:45.353 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fd934614232] 2022-01-31T19:45:45.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fd9346143f5] 2022-01-31T19:45:45.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fd934bde838] 2022-01-31T19:45:45.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x5604f13cdfa3] 2022-01-31T19:45:45.354 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 8: (PaxosService::refresh(bool*)+0x10e) [0x5604f12f1b0e] 2022-01-31T19:45:45.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x5604f118351c] 2022-01-31T19:45:45.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 10: (Monitor::init_paxos()+0x10c) [0x5604f118382c] 2022-01-31T19:45:45.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 11: (Monitor::preinit()+0xd30) [0x5604f11b12a0] 2022-01-31T19:45:45.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 12: main() 2022-01-31T19:45:45.355 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 13: __libc_start_main() 2022-01-31T19:45:45.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 14: _start() 2022-01-31T19:45:45.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:45.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:44 smithi167 conmon[102569]: 2022-01-31T19:45:45.356 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:45 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:45.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:45 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Main process exited, code=exited, status=134/n/a 2022-01-31T19:45:45.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:45 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Failed with result 'exit-code'. 2022-01-31T19:45:55.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:55 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Service RestartSec=10s expired, scheduling restart. 2022-01-31T19:45:55.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:55 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Scheduled restart job, restart counter is at 3. 2022-01-31T19:45:55.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:55 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:55.921 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:55 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:45:56.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:56 smithi167 podman[102837]: 2022-01-31T19:45:57.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:57.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:57.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:57.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 bash[102686]: 37e41fa11b0d885199711a09da2dbf1a224b325e23b2f33d53f36b80458273f9 2022-01-31T19:45:57.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: load: jerasure load: lrc 2022-01-31T19:45:57.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:57.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:57.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:57.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB SUMMARY 2022-01-31T19:45:57.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB Session ID: VK69Q23KAXCBD0WYNTKX 2022-01-31T19:45:57.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:57.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:57.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: MANIFEST file: MANIFEST-000028 size: 207 Bytes 2022-01-31T19:45:57.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:57.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000029.log size: 0 ; 2022-01-31T19:45:57.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:57.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:57.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:57.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:57.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.env: 0x559359793c00 2022-01-31T19:45:57.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:57.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.info_log: 0x55935b308700 2022-01-31T19:45:57.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:57.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:57.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:57.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:57.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:57.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:57.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:57.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:57.554 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:57.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:57.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:57.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:57.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:57.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:57.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:57.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:57.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:57.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:57.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:57.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:57.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:57.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:57.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:57.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:57.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_buffer_manager: 0x55935b3f0330 2022-01-31T19:45:57.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:57.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:57.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:57.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:57.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:57.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:57.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:57.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:57.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:57.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:57.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:57.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:57.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:57.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:57.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.row_cache: None 2022-01-31T19:45:57.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:57.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:57.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:57.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:57.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:57.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:57.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:57.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:57.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:57.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:57.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:57.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:57.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:57.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:57.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:57.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:57.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:57.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:57.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:57.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:57.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:57.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:57.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:57.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:57.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:57.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:57.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:57.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:57.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:57.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:57.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:57.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:57.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:57.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:57.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:57.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:57.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:57.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:57.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:57.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:57.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:57.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:57.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:57.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:57.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000028 2022-01-31T19:45:57.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:57.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:57.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.merge_operator: 2022-01-31T19:45:57.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:57.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:57.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:57.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:57.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:57.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55935b2d3c20) 2022-01-31T19:45:57.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:57.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:57.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:57.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:57.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:57.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:57.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:57.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:57.578 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:57.578 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:57.578 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:57.578 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55935b338170 2022-01-31T19:45:57.579 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:57.579 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:57.579 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:57.579 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:57.580 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:57.580 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:57.580 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:57.580 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:57.581 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:57.581 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:57.581 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:57.581 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:57.581 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:57.582 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:57.582 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:57.582 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:57.582 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:57.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:57.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:57.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:57.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:57.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:57.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:57.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:57.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:57.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:57.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:57.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:57.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:57.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:57.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:57.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:57.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:57.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:57.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:57.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:57.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:57.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:57.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:57.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:57.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:57.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:57.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:57.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:57.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:57.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:57.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:57.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:57.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:57.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:57.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:57.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:57.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:57.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:57.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:57.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:57.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:57.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:57.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:57.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:57.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:57.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:57.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:57.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:57.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:57.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:57.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:57.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:57.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:57.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:57.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:57.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:57.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:57.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:57.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:57.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:57.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:57.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:57.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:57.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:57.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:57.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:57.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:57.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:57.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:57.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:57.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:57.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:57.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:57.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:57.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:57.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:57.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:57.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:57.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:57.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:57.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:57.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:57.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000028 succeeded,manifest_file_number is 28, next_file_number is 30, last_sequence is 8022, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:57.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 25 2022-01-31T19:45:57.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4083] Creating manifest 32 2022-01-31T19:45:57.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658357369332, "job": 1, "event": "recovery_started", "wal_files": [29]} 2022-01-31T19:45:57.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #29 mode 2 2022-01-31T19:45:57.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4083] Creating manifest 33 2022-01-31T19:45:57.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658357369717, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:57.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000029.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:57.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55935b31ea80 2022-01-31T19:45:57.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB pointer 0x55935b32e000 2022-01-31T19:45:57.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:57.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:57.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:57.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:57.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:57.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:57.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:57.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:57.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:57.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:57.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:57.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:57.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:57.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:57.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:57.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:57.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:57.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:57.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:57.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:57.620 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:57.620 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:57.620 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:57.620 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:57.621 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:57.621 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:57.621 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:57.621 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.622 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:57.622 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:57.622 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:57.622 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:57.622 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:57.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:57.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:57.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:57.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:57.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:57.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:57.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:57.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:57.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:57.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:57.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:57.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:57.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:57.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:57.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:57.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:57.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:57.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:57.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:57.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:57.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:57.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:57.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:57.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:57.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: main() 2022-01-31T19:45:57.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: __libc_start_main() 2022-01-31T19:45:57.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: _start() 2022-01-31T19:45:57.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: *** Caught signal (Aborted) ** 2022-01-31T19:45:57.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.372+0000 7fce726dc880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: main() 2022-01-31T19:45:57.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: __libc_start_main() 2022-01-31T19:45:57.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: _start() 2022-01-31T19:45:57.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: gsignal() 2022-01-31T19:45:57.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: abort() 2022-01-31T19:45:57.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 12: main() 2022-01-31T19:45:57.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 13: __libc_start_main() 2022-01-31T19:45:57.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 14: _start() 2022-01-31T19:45:57.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: *** Caught signal (Aborted) ** 2022-01-31T19:45:57.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:57.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:57.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:57.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:57.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:57.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:57.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.373+0000 7fce726dc880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:57.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: gsignal() 2022-01-31T19:45:57.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: abort() 2022-01-31T19:45:57.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 12: main() 2022-01-31T19:45:57.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 13: __libc_start_main() 2022-01-31T19:45:57.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 14: _start() 2022-01-31T19:45:57.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:57.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command assert hook 0x55935b2cea50 2022-01-31T19:45:57.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command abort hook 0x55935b2cea50 2022-01-31T19:45:57.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command leak_some_memory hook 0x55935b2cea50 2022-01-31T19:45:57.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perfcounters_dump hook 0x55935b2cea50 2022-01-31T19:45:57.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command 1 hook 0x55935b2cea50 2022-01-31T19:45:57.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf dump hook 0x55935b2cea50 2022-01-31T19:45:57.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perfcounters_schema hook 0x55935b2cea50 2022-01-31T19:45:57.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf histogram dump hook 0x55935b2cea50 2022-01-31T19:45:57.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command 2 hook 0x55935b2cea50 2022-01-31T19:45:57.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf schema hook 0x55935b2cea50 2022-01-31T19:45:57.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf histogram schema hook 0x55935b2cea50 2022-01-31T19:45:57.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf reset hook 0x55935b2cea50 2022-01-31T19:45:57.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config show hook 0x55935b2cea50 2022-01-31T19:45:57.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config help hook 0x55935b2cea50 2022-01-31T19:45:57.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config set hook 0x55935b2cea50 2022-01-31T19:45:57.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config unset hook 0x55935b2cea50 2022-01-31T19:45:57.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config get hook 0x55935b2cea50 2022-01-31T19:45:57.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config diff hook 0x55935b2cea50 2022-01-31T19:45:57.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config diff get hook 0x55935b2cea50 2022-01-31T19:45:57.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command injectargs hook 0x55935b2cea50 2022-01-31T19:45:57.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command log flush hook 0x55935b2cea50 2022-01-31T19:45:57.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command log dump hook 0x55935b2cea50 2022-01-31T19:45:57.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command log reopen hook 0x55935b2cea50 2022-01-31T19:45:57.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command dump_mempools hook 0x55935b3bc068 2022-01-31T19:45:57.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:57.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:57.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:57.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:57.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:57.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command 0 hook 0x55935b2d30a0 2022-01-31T19:45:57.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command version hook 0x55935b2d30a0 2022-01-31T19:45:57.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command git_version hook 0x55935b2d30a0 2022-01-31T19:45:57.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command help hook 0x55935b2ce9f0 2022-01-31T19:45:57.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command get_command_descriptions hook 0x55935b2ce9e0 2022-01-31T19:45:57.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) entry start 2022-01-31T19:45:57.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: load: jerasure load: lrc 2022-01-31T19:45:57.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:57.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:57.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:57.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:57.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:57.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:57.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:57.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:57.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:57.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:57.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB SUMMARY 2022-01-31T19:45:57.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB Session ID: VK69Q23KAXCBD0WYNTKX 2022-01-31T19:45:57.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:57.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:57.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: MANIFEST file: MANIFEST-000028 size: 207 Bytes 2022-01-31T19:45:57.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:57.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000029.log size: 0 ; 2022-01-31T19:45:57.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:57.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:57.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:57.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:57.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.env: 0x559359793c00 2022-01-31T19:45:57.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:57.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.info_log: 0x55935b308700 2022-01-31T19:45:57.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:57.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:57.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:57.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:57.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:57.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:57.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:57.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:57.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:57.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:57.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:57.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:57.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:57.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:57.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:57.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:57.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:57.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:57.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:57.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:57.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_buffer_manager: 0x55935b3f0330 2022-01-31T19:45:57.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:57.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:57.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:57.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:57.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:57.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:57.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:57.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:57.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:57.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:57.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:57.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:57.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:57.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:57.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.row_cache: None 2022-01-31T19:45:57.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:57.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:57.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:57.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:57.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:57.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:57.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:57.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:57.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:57.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:57.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:57.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:57.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:57.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:57.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:57.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:57.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:57.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:57.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:57.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:57.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:57.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:57.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:57.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:57.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:57.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:57.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:57.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:57.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:57.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:57.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:57.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:57.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:57.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:57.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:57.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:57.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:57.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:57.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:57.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:57.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:57.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:57.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:57.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:57.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000028 2022-01-31T19:45:57.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:57.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:57.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.merge_operator: 2022-01-31T19:45:57.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:57.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:57.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:57.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:57.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:57.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55935b2d3c20) 2022-01-31T19:45:57.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:57.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:57.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:57.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:57.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:57.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:57.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:57.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:57.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:57.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:57.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:57.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55935b338170 2022-01-31T19:45:57.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:57.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:57.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:57.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:57.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:57.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:57.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:57.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:57.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:57.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:57.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:57.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:57.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:57.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:57.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:57.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:57.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:57.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:57.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:57.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:57.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:57.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:57.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:57.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:57.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:57.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:57.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:57.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:57.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:57.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:57.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:57.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:57.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:57.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:57.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:57.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:57.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:57.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:57.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:57.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:57.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:57.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:57.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:57.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:57.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:57.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:57.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:57.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:57.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:57.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:57.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:57.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:57.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:57.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:57.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:57.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:57.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:57.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:57.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:57.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:57.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:57.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:57.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:57.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:57.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:57.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:57.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:57.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:57.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:57.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:57.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:57.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:57.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:57.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:57.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:57.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:57.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:57.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:57.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:57.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:57.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:57.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:57.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:57.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:57.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:57.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:57.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:57.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:57.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:57.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:57.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:57.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:57.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:57.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:57.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:57.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:57.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:57.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:57.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000028 succeeded,manifest_file_number is 28, next_file_number is 30, last_sequence is 8022, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:57.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 25 2022-01-31T19:45:57.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4083] Creating manifest 32 2022-01-31T19:45:57.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658357369332, "job": 1, "event": "recovery_started", "wal_files": [29]} 2022-01-31T19:45:57.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #29 mode 2 2022-01-31T19:45:57.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4083] Creating manifest 33 2022-01-31T19:45:57.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658357369717, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:57.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000029.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:57.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55935b31ea80 2022-01-31T19:45:57.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB pointer 0x55935b32e000 2022-01-31T19:45:57.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:57.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:57.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:57.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:57.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:57.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:57.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:57.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:57.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:57.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:57.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:57.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:57.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:57.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:57.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:57.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:57.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:57.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:57.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:57.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: cephx 2022-01-31T19:45:57.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: cephx 2022-01-31T19:45:57.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: cephx 2022-01-31T19:45:57.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: none 2022-01-31T19:45:57.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:57.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:57.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: cephx 2022-01-31T19:45:57.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: cephx 2022-01-31T19:45:57.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: cephx 2022-01-31T19:45:57.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: none 2022-01-31T19:45:57.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:57.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: adding auth protocol: cephx 2022-01-31T19:45:57.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: adding auth protocol: cephx 2022-01-31T19:45:57.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:57.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:57.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:57.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:57.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:57.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:57.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:57.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:57.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:57.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:57.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:57.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:57.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:57.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:57.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:57.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:57.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:57.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:57.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:57.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:57.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:57.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:57.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:57.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:57.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:57.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:57.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:57.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:57.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:57.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:57.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:57.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:57.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:57.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:57.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:57.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:57.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:57.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:57.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: *** Caught signal (Aborted) ** 2022-01-31T19:45:57.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:57.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:57.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:57.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:57.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:57.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:57.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: -288> 2022-01-31T19:45:57.372+0000 7fce726dc880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: main() 2022-01-31T19:45:57.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: __libc_start_main() 2022-01-31T19:45:57.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: _start() 2022-01-31T19:45:57.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: -287> 2022-01-31T19:45:57.373+0000 7fce726dc880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:57.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: gsignal() 2022-01-31T19:45:57.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: abort() 2022-01-31T19:45:57.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 12: main() 2022-01-31T19:45:57.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 13: __libc_start_main() 2022-01-31T19:45:57.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 14: _start() 2022-01-31T19:45:57.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:57.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command assert hook 0x55935b2cea50 2022-01-31T19:45:57.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command abort hook 0x55935b2cea50 2022-01-31T19:45:57.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command leak_some_memory hook 0x55935b2cea50 2022-01-31T19:45:57.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perfcounters_dump hook 0x55935b2cea50 2022-01-31T19:45:57.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command 1 hook 0x55935b2cea50 2022-01-31T19:45:57.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf dump hook 0x55935b2cea50 2022-01-31T19:45:57.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perfcounters_schema hook 0x55935b2cea50 2022-01-31T19:45:57.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf histogram dump hook 0x55935b2cea50 2022-01-31T19:45:57.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command 2 hook 0x55935b2cea50 2022-01-31T19:45:57.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf schema hook 0x55935b2cea50 2022-01-31T19:45:57.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf histogram schema hook 0x55935b2cea50 2022-01-31T19:45:57.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command perf reset hook 0x55935b2cea50 2022-01-31T19:45:57.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config show hook 0x55935b2cea50 2022-01-31T19:45:57.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config help hook 0x55935b2cea50 2022-01-31T19:45:57.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config set hook 0x55935b2cea50 2022-01-31T19:45:57.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config unset hook 0x55935b2cea50 2022-01-31T19:45:57.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config get hook 0x55935b2cea50 2022-01-31T19:45:57.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config diff hook 0x55935b2cea50 2022-01-31T19:45:57.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command config diff get hook 0x55935b2cea50 2022-01-31T19:45:57.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command injectargs hook 0x55935b2cea50 2022-01-31T19:45:57.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command log flush hook 0x55935b2cea50 2022-01-31T19:45:57.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command log dump hook 0x55935b2cea50 2022-01-31T19:45:57.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command log reopen hook 0x55935b2cea50 2022-01-31T19:45:57.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command dump_mempools hook 0x55935b3bc068 2022-01-31T19:45:57.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:45:57.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:45:57.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: pidfile_write: ignore empty --pid-file 2022-01-31T19:45:57.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:57.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:45:57.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command 0 hook 0x55935b2d30a0 2022-01-31T19:45:57.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command version hook 0x55935b2d30a0 2022-01-31T19:45:57.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command git_version hook 0x55935b2d30a0 2022-01-31T19:45:57.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command help hook 0x55935b2ce9f0 2022-01-31T19:45:57.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) register_command get_command_descriptions hook 0x55935b2ce9e0 2022-01-31T19:45:57.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: asok(0x55935b5b6000) entry start 2022-01-31T19:45:57.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: load: jerasure load: lrc 2022-01-31T19:45:57.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:57.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:57.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:57.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:45:57.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:45:57.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: set rocksdb option compression = kNoCompression 2022-01-31T19:45:57.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:45:57.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:45:57.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:45:57.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:45:57.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB SUMMARY 2022-01-31T19:45:57.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB Session ID: VK69Q23KAXCBD0WYNTKX 2022-01-31T19:45:57.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:45:57.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:45:57.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: MANIFEST file: MANIFEST-000028 size: 207 Bytes 2022-01-31T19:45:57.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:45:57.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000029.log size: 0 ; 2022-01-31T19:45:57.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:45:57.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:45:57.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:45:57.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:45:57.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.env: 0x559359793c00 2022-01-31T19:45:57.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.fs: Posix File System 2022-01-31T19:45:57.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.info_log: 0x55935b308700 2022-01-31T19:45:57.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:45:57.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.statistics: (nil) 2022-01-31T19:45:57.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_fsync: 0 2022-01-31T19:45:57.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:45:57.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:45:57.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:45:57.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:45:57.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:45:57.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:45:57.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:45:57.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:45:57.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:45:57.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:45:57.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:45:57.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_log_dir: 2022-01-31T19:45:57.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:45:57.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:45:57.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:45:57.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:45:57.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:45:57.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:45:57.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:45:57.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:45:57.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:45:57.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_buffer_manager: 0x55935b3f0330 2022-01-31T19:45:57.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:45:57.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:45:57.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:45:57.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:45:57.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:45:57.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:45:57.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:45:57.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:45:57.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:45:57.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.unordered_write: 0 2022-01-31T19:45:57.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:45:57.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:45:57.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:45:57.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:45:57.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.row_cache: None 2022-01-31T19:45:57.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_filter: None 2022-01-31T19:45:57.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:45:57.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:45:57.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:45:57.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:45:57.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:45:57.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:45:57.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:45:57.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:45:57.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:45:57.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:45:57.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:45:57.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:45:57.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:45:57.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:45:57.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:45:57.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:45:57.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:45:57.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:45:57.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:45:57.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:45:57.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:45:57.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:45:57.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:45:57.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:45:57.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:45:57.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:45:57.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:45:57.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_open_files: -1 2022-01-31T19:45:57.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:45:57.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:45:57.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:45:57.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:45:57.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:45:57.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Compression algorithms supported: 2022-01-31T19:45:57.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:45:57.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZSTD supported: 0 2022-01-31T19:45:57.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:45:57.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:45:57.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:45:57.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:45:57.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:45:57.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:45:57.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:45:57.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000028 2022-01-31T19:45:57.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:45:57.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:45:57.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.merge_operator: 2022-01-31T19:45:57.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_filter: None 2022-01-31T19:45:57.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:45:57.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:45:57.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:45:57.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:45:57.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55935b2d3c20) 2022-01-31T19:45:57.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:45:57.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:45:57.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:45:57.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:45:57.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:45:57.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:45:57.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:45:57.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:45:57.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:45:57.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:45:57.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:45:57.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55935b338170 2022-01-31T19:45:57.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:45:57.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:45:57.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:45:57.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:45:57.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:45:57.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:45:57.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:45:57.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:45:57.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:45:57.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:45:57.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:45:57.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:45:57.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:45:57.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:45:57.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:45:57.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:45:57.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:45:57.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:45:57.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:45:57.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:45:57.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:45:57.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:45:57.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:45:57.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:45:57.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression: NoCompression 2022-01-31T19:45:57.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:45:57.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:45:57.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:45:57.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.num_levels: 7 2022-01-31T19:45:57.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:45:57.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:45:57.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:45:57.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:45:57.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:45:57.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:45:57.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:45:57.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:57.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:45:57.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:45:57.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:45:57.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:45:57.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:45:57.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:45:57.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:45:57.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:45:57.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:45:57.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:45:57.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:45:57.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:45:57.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:45:57.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:45:57.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:45:57.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:45:57.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:45:57.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:45:57.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:45:57.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:45:57.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:45:57.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:45:57.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:45:57.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:45:57.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:45:57.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:45:57.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:45:57.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:45:57.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:45:57.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:45:57.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:45:57.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:45:57.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:45:57.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:45:57.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:45:57.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:45:57.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:45:57.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:45:57.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:45:57.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:45:57.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:45:57.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:45:57.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:45:57.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:45:57.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:45:57.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:45:57.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:45:57.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:45:57.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:45:57.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:45:57.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:45:57.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:45:57.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:45:57.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.ttl: 2592000 2022-01-31T19:45:57.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:45:57.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:45:57.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:45:57.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:45:57.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:45:57.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:45:57.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:45:57.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000028 succeeded,manifest_file_number is 28, next_file_number is 30, last_sequence is 8022, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:45:57.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 25 2022-01-31T19:45:57.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4083] Creating manifest 32 2022-01-31T19:45:57.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658357369332, "job": 1, "event": "recovery_started", "wal_files": [29]} 2022-01-31T19:45:57.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #29 mode 2 2022-01-31T19:45:57.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:4083] Creating manifest 33 2022-01-31T19:45:57.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:45:57.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658357369717, "job": 1, "event": "recovery_finished"} 2022-01-31T19:45:57.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000029.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:45:57.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55935b31ea80 2022-01-31T19:45:57.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: DB pointer 0x55935b32e000 2022-01-31T19:45:57.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:45:57.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:45:57.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:45:57.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:57.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:45:57.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:57.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:57.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:45:57.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:45:57.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:57.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:57.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:57.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:57.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:57.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:45:57.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:45:57.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:45:57.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:45:57.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:45:57.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:45:57.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:45:57.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:45:57.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:45:57.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:45:57.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:45:57.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:45:57.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: cephx 2022-01-31T19:45:57.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: cephx 2022-01-31T19:45:57.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: cephx 2022-01-31T19:45:57.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding auth protocol: none 2022-01-31T19:45:57.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.861 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.861 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.861 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.862 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.862 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: crc 2022-01-31T19:45:57.862 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070140) adding con mode: secure 2022-01-31T19:45:57.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:57.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:57.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: cephx 2022-01-31T19:45:57.863 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: cephx 2022-01-31T19:45:57.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: cephx 2022-01-31T19:45:57.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding auth protocol: none 2022-01-31T19:45:57.864 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.865 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.866 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: crc 2022-01-31T19:45:57.867 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: AuthRegistry(0x55935c070a40) adding con mode: secure 2022-01-31T19:45:57.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:45:57.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: adding auth protocol: cephx 2022-01-31T19:45:57.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: adding auth protocol: cephx 2022-01-31T19:45:57.868 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:57.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:45:57.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:45:57.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:45:57.869 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:45:57.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:45:57.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:45:57.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:57.870 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:45:57.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:45:57.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:45:57.871 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:45:57.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:45:57.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:45:57.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:45:57.872 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:45:57.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:45:57.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:45:57.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:45:57.873 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:45:57.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:45:57.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:45:57.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:45:57.874 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:45:57.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:45:57.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:45:57.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:45:57.875 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:45:57.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:45:57.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:45:57.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:45:57.876 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:45:57.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:45:57.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:45:57.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:45:57.877 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.878 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:45:57.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.879 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.880 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.881 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:45:57.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:45:57.882 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:45:57.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 ceph-mon[102910]: *** Caught signal (Aborted) ** 2022-01-31T19:45:57.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:45:57.883 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:45:57.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:45:57.884 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.885 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.886 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:45:57.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:45:57.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:45:57.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:57.887 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: -1> 2022-01-31T19:45:57.372+0000 7fce726dc880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7fce726dc880 time 2022-01-31T19:45:57.372768+0000 2022-01-31T19:45:57.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:45:57.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fce719361d4] 2022-01-31T19:45:57.888 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.889 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: main() 2022-01-31T19:45:57.890 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: __libc_start_main() 2022-01-31T19:45:57.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: _start() 2022-01-31T19:45:57.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 0> 2022-01-31T19:45:57.373+0000 7fce726dc880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:45:57.891 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: in thread 7fce726dc880 thread_name:ceph-mon 2022-01-31T19:45:57.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:45:57.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7fce6f8c9c20] 2022-01-31T19:45:57.892 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2: gsignal() 2022-01-31T19:45:57.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 3: abort() 2022-01-31T19:45:57.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fce71936232] 2022-01-31T19:45:57.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7fce719363f5] 2022-01-31T19:45:57.893 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7fce71f00838] 2022-01-31T19:45:57.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x559359000fa3] 2022-01-31T19:45:57.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 8: (PaxosService::refresh(bool*)+0x10e) [0x559358f24b0e] 2022-01-31T19:45:57.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x559358db651c] 2022-01-31T19:45:57.894 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 10: (Monitor::init_paxos()+0x10c) [0x559358db682c] 2022-01-31T19:45:57.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 11: (Monitor::preinit()+0xd30) [0x559358de42a0] 2022-01-31T19:45:57.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 12: main() 2022-01-31T19:45:57.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 13: __libc_start_main() 2022-01-31T19:45:57.895 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 14: _start() 2022-01-31T19:45:57.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:45:57.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 conmon[102885]: 2022-01-31T19:45:57.896 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:45:57.919 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Main process exited, code=exited, status=134/n/a 2022-01-31T19:45:57.920 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:45:57 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Failed with result 'exit-code'. 2022-01-31T19:46:08.275 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:07 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Service RestartSec=10s expired, scheduling restart. 2022-01-31T19:46:08.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:07 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Scheduled restart job, restart counter is at 4. 2022-01-31T19:46:08.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:07 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:46:08.276 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:07 smithi167 systemd[1]: Starting Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c... 2022-01-31T19:46:08.529 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 podman[103143]: 2022-01-31T19:46:08.529 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 bash[102990]: 1aed899c65a69a16cd44ca15b59aa0e5eba82345cba74c584ab231d664e60946 2022-01-31T19:46:08.529 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:46:08.530 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:46:08.530 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: pidfile_write: ignore empty --pid-file 2022-01-31T19:46:08.530 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: load: jerasure load: lrc 2022-01-31T19:46:08.530 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:46:08.532 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:46:08.532 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:46:08.532 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB SUMMARY 2022-01-31T19:46:08.533 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB Session ID: DWRKQH202539FA7Q9CYG 2022-01-31T19:46:08.533 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:46:08.533 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:46:08.533 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: MANIFEST file: MANIFEST-000033 size: 207 Bytes 2022-01-31T19:46:08.534 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:46:08.534 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000034.log size: 0 ; 2022-01-31T19:46:08.534 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:46:08.534 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:46:08.535 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:46:08.535 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:46:08.535 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.env: 0x55d01e8acc00 2022-01-31T19:46:08.535 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.fs: Posix File System 2022-01-31T19:46:08.536 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.info_log: 0x55d02061c700 2022-01-31T19:46:08.536 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:46:08.536 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.statistics: (nil) 2022-01-31T19:46:08.536 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_fsync: 0 2022-01-31T19:46:08.537 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:46:08.537 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:46:08.537 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:46:08.537 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:46:08.538 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:46:08.538 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:46:08.538 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:46:08.538 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:46:08.539 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:46:08.539 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:46:08.539 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:46:08.539 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_log_dir: 2022-01-31T19:46:08.539 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:46:08.540 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:46:08.540 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:46:08.540 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:46:08.540 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:46:08.541 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:46:08.541 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:46:08.541 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:46:08.541 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:46:08.541 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_buffer_manager: 0x55d020704330 2022-01-31T19:46:08.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:46:08.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:46:08.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:46:08.542 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:46:08.543 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:46:08.543 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:46:08.543 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:46:08.543 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:46:08.544 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:46:08.544 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.unordered_write: 0 2022-01-31T19:46:08.544 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:46:08.544 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:46:08.544 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:46:08.545 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:46:08.545 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.row_cache: None 2022-01-31T19:46:08.545 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_filter: None 2022-01-31T19:46:08.545 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:46:08.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:46:08.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:46:08.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:46:08.546 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:46:08.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:46:08.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:46:08.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:46:08.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:46:08.547 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:46:08.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:46:08.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:46:08.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:46:08.548 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:46:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:46:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:46:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:46:08.549 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:46:08.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:46:08.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:46:08.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:46:08.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:46:08.550 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:46:08.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:46:08.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:46:08.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:46:08.551 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:46:08.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_open_files: -1 2022-01-31T19:46:08.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:46:08.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:46:08.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:46:08.552 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:46:08.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:46:08.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Compression algorithms supported: 2022-01-31T19:46:08.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:46:08.553 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZSTD supported: 0 2022-01-31T19:46:08.554 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:46:08.554 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:46:08.554 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:46:08.554 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:46:08.554 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:46:08.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:46:08.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:46:08.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000033 2022-01-31T19:46:08.555 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:46:08.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:46:08.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.merge_operator: 2022-01-31T19:46:08.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_filter: None 2022-01-31T19:46:08.556 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:46:08.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:46:08.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:46:08.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:46:08.557 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d0205e7c20) 2022-01-31T19:46:08.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:46:08.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:46:08.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:46:08.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:46:08.558 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:46:08.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:46:08.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:46:08.559 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:46:08.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:46:08.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:46:08.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:46:08.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55d02064c170 2022-01-31T19:46:08.560 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:46:08.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:46:08.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:46:08.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:46:08.561 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:46:08.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:46:08.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:46:08.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:46:08.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:46:08.562 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:46:08.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:46:08.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:46:08.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:46:08.563 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:46:08.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:46:08.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:46:08.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:46:08.564 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:46:08.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:46:08.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:46:08.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:46:08.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:46:08.565 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:46:08.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:46:08.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression: NoCompression 2022-01-31T19:46:08.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:46:08.566 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:46:08.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:46:08.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.num_levels: 7 2022-01-31T19:46:08.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:46:08.567 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:46:08.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:46:08.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:46:08.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:46:08.568 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:46:08.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:46:08.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:46:08.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:46:08.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:46:08.569 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:46:08.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:46:08.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:46:08.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:46:08.570 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:46:08.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:46:08.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:46:08.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:46:08.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:46:08.571 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:46:08.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:46:08.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:46:08.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:46:08.572 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:46:08.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:46:08.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:46:08.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:46:08.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:46:08.573 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:46:08.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:46:08.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:46:08.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:46:08.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:46:08.574 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:46:08.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:46:08.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:46:08.575 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:46:08.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:46:08.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:46:08.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:46:08.576 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:46:08.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:46:08.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:46:08.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:46:08.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:46:08.577 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:46:08.578 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:46:08.582 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:46:08.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:46:08.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:46:08.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:46:08.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:46:08.583 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:46:08.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:46:08.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:46:08.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:46:08.584 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:46:08.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:46:08.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:46:08.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:46:08.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:46:08.585 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.ttl: 2592000 2022-01-31T19:46:08.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:46:08.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:46:08.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:46:08.586 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:46:08.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:46:08.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:46:08.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:46:08.587 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000033 succeeded,manifest_file_number is 33, next_file_number is 35, last_sequence is 8022, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:46:08.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 30 2022-01-31T19:46:08.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4083] Creating manifest 37 2022-01-31T19:46:08.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.588 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658368457319, "job": 1, "event": "recovery_started", "wal_files": [34]} 2022-01-31T19:46:08.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #34 mode 2 2022-01-31T19:46:08.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4083] Creating manifest 38 2022-01-31T19:46:08.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658368457664, "job": 1, "event": "recovery_finished"} 2022-01-31T19:46:08.589 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000034.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:46:08.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d020632a80 2022-01-31T19:46:08.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB pointer 0x55d020642000 2022-01-31T19:46:08.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:46:08.590 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:46:08.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:46:08.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:46:08.591 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:46:08.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:46:08.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:46:08.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:46:08.592 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:46:08.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.593 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.594 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.595 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:46:08.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:46:08.596 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:46:08.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:46:08.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.597 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:46:08.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.598 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.599 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.600 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:46:08.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:46:08.601 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:46:08.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:46:08.602 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:46:08.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:46:08.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:46:08.603 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:46:08.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:46:08.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:46:08.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:46:08.604 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:46:08.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:46:08.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:46:08.605 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:46:08.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:46:08.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:46:08.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:46:08.606 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:46:08.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:46:08.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:46:08.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:46:08.607 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:46:08.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:46:08.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:46:08.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:46:08.608 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:46:08.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:46:08.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:46:08.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:46:08.609 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:46:08.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:46:08.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:46:08.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:46:08.610 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:46:08.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:46:08.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:46:08.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:46:08.611 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:46:08.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.612 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.613 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.614 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.615 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.616 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.617 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:46:08.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:46:08.618 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:46:08.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.619 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: main() 2022-01-31T19:46:08.620 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: __libc_start_main() 2022-01-31T19:46:08.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: _start() 2022-01-31T19:46:08.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: *** Caught signal (Aborted) ** 2022-01-31T19:46:08.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.623 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.461+0000 7f24a117e880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.624 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.625 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: main() 2022-01-31T19:46:08.626 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: __libc_start_main() 2022-01-31T19:46:08.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: _start() 2022-01-31T19:46:08.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.627 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: gsignal() 2022-01-31T19:46:08.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: abort() 2022-01-31T19:46:08.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.628 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.629 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 12: main() 2022-01-31T19:46:08.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 13: __libc_start_main() 2022-01-31T19:46:08.630 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 14: _start() 2022-01-31T19:46:08.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.462+0000 7f24a117e880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:46:08.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: *** Caught signal (Aborted) ** 2022-01-31T19:46:08.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.631 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:46:08.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:46:08.632 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.633 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:46:08.634 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:46:08.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:46:08.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:46:08.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.635 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: gsignal() 2022-01-31T19:46:08.636 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: abort() 2022-01-31T19:46:08.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.637 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.638 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 12: main() 2022-01-31T19:46:08.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 13: __libc_start_main() 2022-01-31T19:46:08.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 14: _start() 2022-01-31T19:46:08.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:46:08.639 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command assert hook 0x55d0205e2a50 2022-01-31T19:46:08.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command abort hook 0x55d0205e2a50 2022-01-31T19:46:08.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command leak_some_memory hook 0x55d0205e2a50 2022-01-31T19:46:08.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perfcounters_dump hook 0x55d0205e2a50 2022-01-31T19:46:08.640 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command 1 hook 0x55d0205e2a50 2022-01-31T19:46:08.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf dump hook 0x55d0205e2a50 2022-01-31T19:46:08.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perfcounters_schema hook 0x55d0205e2a50 2022-01-31T19:46:08.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf histogram dump hook 0x55d0205e2a50 2022-01-31T19:46:08.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command 2 hook 0x55d0205e2a50 2022-01-31T19:46:08.641 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf schema hook 0x55d0205e2a50 2022-01-31T19:46:08.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf histogram schema hook 0x55d0205e2a50 2022-01-31T19:46:08.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf reset hook 0x55d0205e2a50 2022-01-31T19:46:08.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config show hook 0x55d0205e2a50 2022-01-31T19:46:08.642 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config help hook 0x55d0205e2a50 2022-01-31T19:46:08.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config set hook 0x55d0205e2a50 2022-01-31T19:46:08.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config unset hook 0x55d0205e2a50 2022-01-31T19:46:08.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config get hook 0x55d0205e2a50 2022-01-31T19:46:08.643 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config diff hook 0x55d0205e2a50 2022-01-31T19:46:08.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config diff get hook 0x55d0205e2a50 2022-01-31T19:46:08.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command injectargs hook 0x55d0205e2a50 2022-01-31T19:46:08.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command log flush hook 0x55d0205e2a50 2022-01-31T19:46:08.644 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command log dump hook 0x55d0205e2a50 2022-01-31T19:46:08.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command log reopen hook 0x55d0205e2a50 2022-01-31T19:46:08.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command dump_mempools hook 0x55d0206d0068 2022-01-31T19:46:08.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:46:08.645 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:46:08.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: pidfile_write: ignore empty --pid-file 2022-01-31T19:46:08.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:46:08.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:46:08.646 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command 0 hook 0x55d0205e70a0 2022-01-31T19:46:08.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command version hook 0x55d0205e70a0 2022-01-31T19:46:08.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command git_version hook 0x55d0205e70a0 2022-01-31T19:46:08.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command help hook 0x55d0205e29f0 2022-01-31T19:46:08.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command get_command_descriptions hook 0x55d0205e29e0 2022-01-31T19:46:08.647 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) entry start 2022-01-31T19:46:08.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: load: jerasure load: lrc 2022-01-31T19:46:08.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:46:08.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:46:08.648 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option compression = kNoCompression 2022-01-31T19:46:08.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:46:08.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:46:08.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option compression = kNoCompression 2022-01-31T19:46:08.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:46:08.649 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:46:08.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:46:08.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:46:08.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB SUMMARY 2022-01-31T19:46:08.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB Session ID: DWRKQH202539FA7Q9CYG 2022-01-31T19:46:08.650 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:46:08.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:46:08.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: MANIFEST file: MANIFEST-000033 size: 207 Bytes 2022-01-31T19:46:08.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:46:08.651 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000034.log size: 0 ; 2022-01-31T19:46:08.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:46:08.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:46:08.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:46:08.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:46:08.652 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.env: 0x55d01e8acc00 2022-01-31T19:46:08.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.fs: Posix File System 2022-01-31T19:46:08.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.info_log: 0x55d02061c700 2022-01-31T19:46:08.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:46:08.653 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.statistics: (nil) 2022-01-31T19:46:08.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_fsync: 0 2022-01-31T19:46:08.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:46:08.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:46:08.654 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:46:08.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:46:08.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:46:08.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:46:08.655 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:46:08.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:46:08.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:46:08.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:46:08.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:46:08.656 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_log_dir: 2022-01-31T19:46:08.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:46:08.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:46:08.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:46:08.657 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:46:08.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:46:08.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:46:08.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:46:08.658 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:46:08.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:46:08.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_buffer_manager: 0x55d020704330 2022-01-31T19:46:08.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:46:08.659 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:46:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:46:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:46:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:46:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:46:08.660 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:46:08.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:46:08.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:46:08.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.unordered_write: 0 2022-01-31T19:46:08.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:46:08.661 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:46:08.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:46:08.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:46:08.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.row_cache: None 2022-01-31T19:46:08.662 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_filter: None 2022-01-31T19:46:08.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:46:08.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:46:08.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:46:08.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:46:08.663 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:46:08.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:46:08.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:46:08.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:46:08.664 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:46:08.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:46:08.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:46:08.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:46:08.665 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:46:08.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:46:08.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:46:08.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:46:08.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:46:08.666 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:46:08.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:46:08.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:46:08.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:46:08.667 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:46:08.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:46:08.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:46:08.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:46:08.668 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:46:08.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:46:08.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_open_files: -1 2022-01-31T19:46:08.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:46:08.669 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:46:08.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:46:08.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:46:08.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:46:08.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Compression algorithms supported: 2022-01-31T19:46:08.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:46:08.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZSTD supported: 0 2022-01-31T19:46:08.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:46:08.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:46:08.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:46:08.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:46:08.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:46:08.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:46:08.672 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:46:08.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000033 2022-01-31T19:46:08.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:46:08.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:46:08.673 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.merge_operator: 2022-01-31T19:46:08.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_filter: None 2022-01-31T19:46:08.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:46:08.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:46:08.674 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:46:08.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:46:08.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d0205e7c20) 2022-01-31T19:46:08.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:46:08.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:46:08.675 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:46:08.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:46:08.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:46:08.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:46:08.676 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:46:08.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:46:08.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:46:08.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:46:08.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:46:08.677 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55d02064c170 2022-01-31T19:46:08.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:46:08.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:46:08.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:46:08.678 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:46:08.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:46:08.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:46:08.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:46:08.679 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:46:08.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:46:08.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:46:08.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:46:08.680 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:46:08.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:46:08.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:46:08.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:46:08.681 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:46:08.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:46:08.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:46:08.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:46:08.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:46:08.682 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:46:08.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:46:08.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:46:08.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:46:08.683 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression: NoCompression 2022-01-31T19:46:08.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:46:08.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:46:08.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:46:08.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.num_levels: 7 2022-01-31T19:46:08.684 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:46:08.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:46:08.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:46:08.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:46:08.685 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:46:08.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:46:08.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:46:08.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:46:08.686 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:46:08.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:46:08.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:46:08.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:46:08.687 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:46:08.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:46:08.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:46:08.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:46:08.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:46:08.688 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:46:08.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:46:08.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:46:08.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:46:08.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:46:08.689 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:46:08.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:46:08.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:46:08.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:46:08.690 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:46:08.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:46:08.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:46:08.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:46:08.691 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:46:08.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:46:08.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:46:08.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:46:08.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:46:08.692 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:46:08.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:46:08.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:46:08.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:46:08.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:46:08.693 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:46:08.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:46:08.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:46:08.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:46:08.694 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:46:08.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:46:08.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:46:08.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:46:08.695 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:46:08.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:46:08.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:46:08.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:46:08.696 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:46:08.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:46:08.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:46:08.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:46:08.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:46:08.697 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:46:08.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:46:08.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:46:08.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:46:08.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.ttl: 2592000 2022-01-31T19:46:08.698 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:46:08.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:46:08.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:46:08.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:46:08.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:46:08.699 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:46:08.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:46:08.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000033 succeeded,manifest_file_number is 33, next_file_number is 35, last_sequence is 8022, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:46:08.700 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 30 2022-01-31T19:46:08.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4083] Creating manifest 37 2022-01-31T19:46:08.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658368457319, "job": 1, "event": "recovery_started", "wal_files": [34]} 2022-01-31T19:46:08.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #34 mode 2 2022-01-31T19:46:08.701 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4083] Creating manifest 38 2022-01-31T19:46:08.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658368457664, "job": 1, "event": "recovery_finished"} 2022-01-31T19:46:08.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000034.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:46:08.702 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d020632a80 2022-01-31T19:46:08.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB pointer 0x55d020642000 2022-01-31T19:46:08.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:46:08.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:46:08.703 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:46:08.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:46:08.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:46:08.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:46:08.704 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:46:08.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:46:08.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:46:08.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.705 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.706 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.707 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.708 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:46:08.709 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:46:08.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:46:08.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.710 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:46:08.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:46:08.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.711 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.712 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.713 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.714 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:46:08.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:46:08.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:46:08.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.715 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:46:08.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:46:08.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: cephx 2022-01-31T19:46:08.716 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: cephx 2022-01-31T19:46:08.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: cephx 2022-01-31T19:46:08.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: none 2022-01-31T19:46:08.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.717 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.718 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.719 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:46:08.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:46:08.720 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: cephx 2022-01-31T19:46:08.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: cephx 2022-01-31T19:46:08.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: cephx 2022-01-31T19:46:08.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: none 2022-01-31T19:46:08.721 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.722 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.723 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:46:08.724 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: adding auth protocol: cephx 2022-01-31T19:46:08.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: adding auth protocol: cephx 2022-01-31T19:46:08.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:46:08.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:46:08.725 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:46:08.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:46:08.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:46:08.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:46:08.726 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:46:08.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:46:08.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:46:08.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.727 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:46:08.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:46:08.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:46:08.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:46:08.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:46:08.728 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:46:08.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:46:08.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:46:08.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:46:08.729 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:46:08.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:46:08.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:46:08.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:46:08.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:46:08.730 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:46:08.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:46:08.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:46:08.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:46:08.731 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:46:08.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:46:08.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:46:08.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:46:08.732 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:46:08.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:46:08.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:46:08.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:46:08.733 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.734 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.735 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.736 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.737 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:46:08.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:46:08.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:46:08.738 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: *** Caught signal (Aborted) ** 2022-01-31T19:46:08.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.739 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:46:08.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:46:08.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.740 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.741 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:46:08.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:46:08.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:46:08.742 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:46:08.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: -288> 2022-01-31T19:46:08.461+0000 7f24a117e880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.743 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.744 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.745 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: main() 2022-01-31T19:46:08.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: __libc_start_main() 2022-01-31T19:46:08.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: _start() 2022-01-31T19:46:08.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.746 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: -287> 2022-01-31T19:46:08.462+0000 7f24a117e880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:46:08.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.747 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: gsignal() 2022-01-31T19:46:08.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: abort() 2022-01-31T19:46:08.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.748 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command assert hook 0x55d0205e2a50 2022-01-31T19:46:08.749 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command abort hook 0x55d0205e2a50 2022-01-31T19:46:08.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.750 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command leak_some_memory hook 0x55d0205e2a50 2022-01-31T19:46:08.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 12: main() 2022-01-31T19:46:08.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 13: __libc_start_main() 2022-01-31T19:46:08.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perfcounters_dump hook 0x55d0205e2a50 2022-01-31T19:46:08.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 14: _start() 2022-01-31T19:46:08.751 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:46:08.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command 1 hook 0x55d0205e2a50 2022-01-31T19:46:08.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf dump hook 0x55d0205e2a50 2022-01-31T19:46:08.752 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perfcounters_schema hook 0x55d0205e2a50 2022-01-31T19:46:08.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf histogram dump hook 0x55d0205e2a50 2022-01-31T19:46:08.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command 2 hook 0x55d0205e2a50 2022-01-31T19:46:08.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf schema hook 0x55d0205e2a50 2022-01-31T19:46:08.753 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf histogram schema hook 0x55d0205e2a50 2022-01-31T19:46:08.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command perf reset hook 0x55d0205e2a50 2022-01-31T19:46:08.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config show hook 0x55d0205e2a50 2022-01-31T19:46:08.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config help hook 0x55d0205e2a50 2022-01-31T19:46:08.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config set hook 0x55d0205e2a50 2022-01-31T19:46:08.754 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config unset hook 0x55d0205e2a50 2022-01-31T19:46:08.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config get hook 0x55d0205e2a50 2022-01-31T19:46:08.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config diff hook 0x55d0205e2a50 2022-01-31T19:46:08.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command config diff get hook 0x55d0205e2a50 2022-01-31T19:46:08.755 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command injectargs hook 0x55d0205e2a50 2022-01-31T19:46:08.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command log flush hook 0x55d0205e2a50 2022-01-31T19:46:08.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command log dump hook 0x55d0205e2a50 2022-01-31T19:46:08.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command log reopen hook 0x55d0205e2a50 2022-01-31T19:46:08.756 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command dump_mempools hook 0x55d0206d0068 2022-01-31T19:46:08.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set uid:gid to 167:167 (ceph:ceph) 2022-01-31T19:46:08.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev), process ceph-mon, pid 7 2022-01-31T19:46:08.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: pidfile_write: ignore empty --pid-file 2022-01-31T19:46:08.757 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) init /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:46:08.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) bind_and_listen /var/run/ceph/ceph-mon.smithi167.asok 2022-01-31T19:46:08.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command 0 hook 0x55d0205e70a0 2022-01-31T19:46:08.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command version hook 0x55d0205e70a0 2022-01-31T19:46:08.758 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command git_version hook 0x55d0205e70a0 2022-01-31T19:46:08.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command help hook 0x55d0205e29f0 2022-01-31T19:46:08.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) register_command get_command_descriptions hook 0x55d0205e29e0 2022-01-31T19:46:08.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: asok(0x55d0208ca000) entry start 2022-01-31T19:46:08.759 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: load: jerasure load: lrc 2022-01-31T19:46:08.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:46:08.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:46:08.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option compression = kNoCompression 2022-01-31T19:46:08.760 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option level_compaction_dynamic_level_bytes = true 2022-01-31T19:46:08.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option write_buffer_size = 33554432 2022-01-31T19:46:08.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: set rocksdb option compression = kNoCompression 2022-01-31T19:46:08.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: verify_sharding column families from rocksdb: [default] 2022-01-31T19:46:08.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: RocksDB version: 6.15.5 2022-01-31T19:46:08.761 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-01-31T19:46:08.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Compile date Jan 30 2022 2022-01-31T19:46:08.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB SUMMARY 2022-01-31T19:46:08.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB Session ID: DWRKQH202539FA7Q9CYG 2022-01-31T19:46:08.762 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: CURRENT file: CURRENT 2022-01-31T19:46:08.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: IDENTITY file: IDENTITY 2022-01-31T19:46:08.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: MANIFEST file: MANIFEST-000033 size: 207 Bytes 2022-01-31T19:46:08.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi167/store.db dir, Total Num: 2, files: 000014.sst 000017.sst 2022-01-31T19:46:08.763 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi167/store.db: 000034.log size: 0 ; 2022-01-31T19:46:08.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.error_if_exists: 0 2022-01-31T19:46:08.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.create_if_missing: 0 2022-01-31T19:46:08.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.paranoid_checks: 1 2022-01-31T19:46:08.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-01-31T19:46:08.764 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.env: 0x55d01e8acc00 2022-01-31T19:46:08.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.fs: Posix File System 2022-01-31T19:46:08.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.info_log: 0x55d02061c700 2022-01-31T19:46:08.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_file_opening_threads: 16 2022-01-31T19:46:08.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.statistics: (nil) 2022-01-31T19:46:08.765 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_fsync: 0 2022-01-31T19:46:08.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_log_file_size: 0 2022-01-31T19:46:08.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-01-31T19:46:08.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.log_file_time_to_roll: 0 2022-01-31T19:46:08.766 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.keep_log_file_num: 1000 2022-01-31T19:46:08.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.recycle_log_file_num: 0 2022-01-31T19:46:08.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_fallocate: 1 2022-01-31T19:46:08.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_mmap_reads: 0 2022-01-31T19:46:08.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_mmap_writes: 0 2022-01-31T19:46:08.767 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_direct_reads: 0 2022-01-31T19:46:08.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-01-31T19:46:08.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.create_missing_column_families: 0 2022-01-31T19:46:08.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_log_dir: 2022-01-31T19:46:08.768 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi167/store.db 2022-01-31T19:46:08.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_cache_numshardbits: 6 2022-01-31T19:46:08.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.WAL_ttl_seconds: 0 2022-01-31T19:46:08.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.WAL_size_limit_MB: 0 2022-01-31T19:46:08.769 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-01-31T19:46:08.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-01-31T19:46:08.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.is_fd_close_on_exec: 1 2022-01-31T19:46:08.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.advise_random_on_open: 1 2022-01-31T19:46:08.770 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_write_buffer_size: 0 2022-01-31T19:46:08.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_buffer_manager: 0x55d020704330 2022-01-31T19:46:08.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-01-31T19:46:08.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-01-31T19:46:08.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-01-31T19:46:08.771 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.use_adaptive_mutex: 0 2022-01-31T19:46:08.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.rate_limiter: (nil) 2022-01-31T19:46:08.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-01-31T19:46:08.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_recovery_mode: 2 2022-01-31T19:46:08.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_thread_tracking: 0 2022-01-31T19:46:08.772 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_pipelined_write: 0 2022-01-31T19:46:08.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.unordered_write: 0 2022-01-31T19:46:08.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-01-31T19:46:08.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-01-31T19:46:08.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-01-31T19:46:08.773 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-01-31T19:46:08.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.row_cache: None 2022-01-31T19:46:08.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_filter: None 2022-01-31T19:46:08.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-01-31T19:46:08.774 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_ingest_behind: 0 2022-01-31T19:46:08.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.preserve_deletes: 0 2022-01-31T19:46:08.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.two_write_queues: 0 2022-01-31T19:46:08.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.manual_wal_flush: 0 2022-01-31T19:46:08.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.atomic_flush: 0 2022-01-31T19:46:08.775 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-01-31T19:46:08.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.persist_stats_to_disk: 0 2022-01-31T19:46:08.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_dbid_to_manifest: 0 2022-01-31T19:46:08.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.log_readahead_size: 0 2022-01-31T19:46:08.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-01-31T19:46:08.776 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.best_efforts_recovery: 0 2022-01-31T19:46:08.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-01-31T19:46:08.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-01-31T19:46:08.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.allow_data_in_errors: 0 2022-01-31T19:46:08.777 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.db_host_id: __hostname__ 2022-01-31T19:46:08.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_jobs: 2 2022-01-31T19:46:08.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_compactions: -1 2022-01-31T19:46:08.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_subcompactions: 1 2022-01-31T19:46:08.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-01-31T19:46:08.778 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-01-31T19:46:08.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.delayed_write_rate : 16777216 2022-01-31T19:46:08.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_total_wal_size: 0 2022-01-31T19:46:08.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-01-31T19:46:08.779 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_dump_period_sec: 600 2022-01-31T19:46:08.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_persist_period_sec: 600 2022-01-31T19:46:08.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-01-31T19:46:08.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_open_files: -1 2022-01-31T19:46:08.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bytes_per_sync: 0 2022-01-31T19:46:08.780 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.wal_bytes_per_sync: 0 2022-01-31T19:46:08.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.strict_bytes_per_sync: 0 2022-01-31T19:46:08.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_readahead_size: 0 2022-01-31T19:46:08.781 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_background_flushes: -1 2022-01-31T19:46:08.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Compression algorithms supported: 2022-01-31T19:46:08.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-01-31T19:46:08.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZSTD supported: 0 2022-01-31T19:46:08.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kXpressCompression supported: 0 2022-01-31T19:46:08.782 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kLZ4HCCompression supported: 1 2022-01-31T19:46:08.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kLZ4Compression supported: 1 2022-01-31T19:46:08.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kBZip2Compression supported: 0 2022-01-31T19:46:08.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kZlibCompression supported: 1 2022-01-31T19:46:08.783 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: kSnappyCompression supported: 1 2022-01-31T19:46:08.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Fast CRC32 supported: Supported on x86 2022-01-31T19:46:08.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000033 2022-01-31T19:46:08.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-01-31T19:46:08.784 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-01-31T19:46:08.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.merge_operator: 2022-01-31T19:46:08.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_filter: None 2022-01-31T19:46:08.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_filter_factory: None 2022-01-31T19:46:08.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.sst_partitioner_factory: None 2022-01-31T19:46:08.785 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_factory: SkipListFactory 2022-01-31T19:46:08.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_factory: BlockBasedTable 2022-01-31T19:46:08.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d0205e7c20) 2022-01-31T19:46:08.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks: 1 2022-01-31T19:46:08.786 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-01-31T19:46:08.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-01-31T19:46:08.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: pin_top_level_index_and_filter: 1 2022-01-31T19:46:08.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_type: 0 2022-01-31T19:46:08.787 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_index_type: 0 2022-01-31T19:46:08.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_shortening: 1 2022-01-31T19:46:08.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2022-01-31T19:46:08.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: hash_index_allow_collision: 1 2022-01-31T19:46:08.788 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: checksum: 1 2022-01-31T19:46:08.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: no_block_cache: 0 2022-01-31T19:46:08.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache: 0x55d02064c170 2022-01-31T19:46:08.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_name: BinnedLRUCache 2022-01-31T19:46:08.789 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_options: 2022-01-31T19:46:08.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: capacity : 536870912 2022-01-31T19:46:08.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: num_shard_bits : 4 2022-01-31T19:46:08.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: strict_capacity_limit : 0 2022-01-31T19:46:08.790 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: high_pri_pool_ratio: 0.000 2022-01-31T19:46:08.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_cache_compressed: (nil) 2022-01-31T19:46:08.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: persistent_cache: (nil) 2022-01-31T19:46:08.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size: 4096 2022-01-31T19:46:08.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_size_deviation: 10 2022-01-31T19:46:08.791 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_restart_interval: 16 2022-01-31T19:46:08.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: index_block_restart_interval: 1 2022-01-31T19:46:08.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_block_size: 4096 2022-01-31T19:46:08.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: partition_filters: 0 2022-01-31T19:46:08.792 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: use_delta_encoding: 1 2022-01-31T19:46:08.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-01-31T19:46:08.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: whole_key_filtering: 1 2022-01-31T19:46:08.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: verify_compression: 0 2022-01-31T19:46:08.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: read_amp_bytes_per_bit: 0 2022-01-31T19:46:08.793 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: format_version: 4 2022-01-31T19:46:08.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_index_compression: 1 2022-01-31T19:46:08.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: block_align: 0 2022-01-31T19:46:08.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.write_buffer_size: 33554432 2022-01-31T19:46:08.794 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_number: 2 2022-01-31T19:46:08.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression: NoCompression 2022-01-31T19:46:08.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression: Disabled 2022-01-31T19:46:08.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.prefix_extractor: nullptr 2022-01-31T19:46:08.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-01-31T19:46:08.795 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.num_levels: 7 2022-01-31T19:46:08.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-01-31T19:46:08.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-01-31T19:46:08.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-01-31T19:46:08.796 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-01-31T19:46:08.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-01-31T19:46:08.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-01-31T19:46:08.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-01-31T19:46:08.797 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:46:08.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-01-31T19:46:08.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-01-31T19:46:08.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.window_bits: -14 2022-01-31T19:46:08.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.level: 32767 2022-01-31T19:46:08.798 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.strategy: 0 2022-01-31T19:46:08.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-01-31T19:46:08.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-01-31T19:46:08.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-01-31T19:46:08.799 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compression_opts.enabled: false 2022-01-31T19:46:08.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-01-31T19:46:08.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-01-31T19:46:08.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-01-31T19:46:08.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.target_file_size_base: 67108864 2022-01-31T19:46:08.800 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.target_file_size_multiplier: 1 2022-01-31T19:46:08.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-01-31T19:46:08.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-01-31T19:46:08.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-01-31T19:46:08.801 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-01-31T19:46:08.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-01-31T19:46:08.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-01-31T19:46:08.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-01-31T19:46:08.802 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-01-31T19:46:08.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-01-31T19:46:08.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-01-31T19:46:08.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-01-31T19:46:08.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-01-31T19:46:08.803 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.arena_block_size: 4194304 2022-01-31T19:46:08.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-01-31T19:46:08.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-01-31T19:46:08.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-01-31T19:46:08.804 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.disable_auto_compactions: 0 2022-01-31T19:46:08.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-01-31T19:46:08.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-01-31T19:46:08.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-01-31T19:46:08.805 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-01-31T19:46:08.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-01-31T19:46:08.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-01-31T19:46:08.806 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-01-31T19:46:08.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-01-31T19:46:08.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-01-31T19:46:08.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-01-31T19:46:08.807 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.table_properties_collectors: 2022-01-31T19:46:08.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.inplace_update_support: 0 2022-01-31T19:46:08.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.inplace_update_num_locks: 10000 2022-01-31T19:46:08.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-01-31T19:46:08.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-01-31T19:46:08.808 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.memtable_huge_page_size: 0 2022-01-31T19:46:08.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.bloom_locality: 0 2022-01-31T19:46:08.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.max_successive_merges: 0 2022-01-31T19:46:08.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.optimize_filters_for_hits: 0 2022-01-31T19:46:08.809 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.paranoid_file_checks: 0 2022-01-31T19:46:08.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.force_consistency_checks: 1 2022-01-31T19:46:08.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.report_bg_io_stats: 0 2022-01-31T19:46:08.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.ttl: 2592000 2022-01-31T19:46:08.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.periodic_compaction_seconds: 0 2022-01-31T19:46:08.810 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_blob_files: false 2022-01-31T19:46:08.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.min_blob_size: 0 2022-01-31T19:46:08.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_file_size: 268435456 2022-01-31T19:46:08.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_compression_type: NoCompression 2022-01-31T19:46:08.811 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.enable_blob_garbage_collection: false 2022-01-31T19:46:08.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-01-31T19:46:08.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi167/store.db/MANIFEST-000033 succeeded,manifest_file_number is 33, next_file_number is 35, last_sequence is 8022, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-01-31T19:46:08.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 30 2022-01-31T19:46:08.812 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4083] Creating manifest 37 2022-01-31T19:46:08.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658368457319, "job": 1, "event": "recovery_started", "wal_files": [34]} 2022-01-31T19:46:08.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #34 mode 2 2022-01-31T19:46:08.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:4083] Creating manifest 38 2022-01-31T19:46:08.813 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/version_set.cc:3459] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2022-01-31T19:46:08.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1643658368457664, "job": 1, "event": "recovery_finished"} 2022-01-31T19:46:08.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi167/store.db/000034.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-01-31T19:46:08.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d020632a80 2022-01-31T19:46:08.814 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: DB pointer 0x55d020642000 2022-01-31T19:46:08.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-01-31T19:46:08.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-01-31T19:46:08.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** DB Stats ** 2022-01-31T19:46:08.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.815 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-01-31T19:46:08.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-01-31T19:46:08.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:46:08.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-01-31T19:46:08.816 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-01-31T19:46:08.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-01-31T19:46:08.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.817 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.818 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.819 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:46:08.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:46:08.820 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:46:08.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:46:08.821 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:46:08.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.822 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L0 1/0 7.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: L6 1/0 17.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.823 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Sum 2/0 24.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-01-31T19:46:08.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** Compaction Stats [default] ** 2022-01-31T19:46:08.824 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-01-31T19:46:08.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-01-31T19:46:08.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-01-31T19:46:08.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-01-31T19:46:08.825 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-01-31T19:46:08.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-01-31T19:46:08.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2022-01-31T19:46:08.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.826 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-01-31T19:46:08.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-01-31T19:46:08.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2022-01-31T19:46:08.827 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: cephx 2022-01-31T19:46:08.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: cephx 2022-01-31T19:46:08.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: cephx 2022-01-31T19:46:08.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding auth protocol: none 2022-01-31T19:46:08.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.828 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.829 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.830 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: crc 2022-01-31T19:46:08.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384140) adding con mode: secure 2022-01-31T19:46:08.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:46:08.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: starting mon.smithi167 rank 0 at public addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] at bind addrs [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi167 fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:46:08.831 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: cephx 2022-01-31T19:46:08.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: cephx 2022-01-31T19:46:08.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: cephx 2022-01-31T19:46:08.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding auth protocol: none 2022-01-31T19:46:08.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.832 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.833 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: crc 2022-01-31T19:46:08.834 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: AuthRegistry(0x55d021384a40) adding con mode: secure 2022-01-31T19:46:08.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-smithi167/keyring 2022-01-31T19:46:08.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: adding auth protocol: cephx 2022-01-31T19:46:08.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: adding auth protocol: cephx 2022-01-31T19:46:08.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:46:08.835 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2022-01-31T19:46:08.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???) e2 preinit fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c 2022-01-31T19:46:08.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???).mds e17 new map 2022-01-31T19:46:08.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: mon.smithi167@-1(???).mds e17 print_map 2022-01-31T19:46:08.836 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: e17 2022-01-31T19:46:08.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-01-31T19:46:08.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:46:08.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: legacy client fscid: 1 2022-01-31T19:46:08.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.837 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: Filesystem 'cephfs' (1) 2022-01-31T19:46:08.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: fs_name cephfs 2022-01-31T19:46:08.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: epoch 17 2022-01-31T19:46:08.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-01-31T19:46:08.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: created 2022-01-31T19:41:48.803120+0000 2022-01-31T19:46:08.838 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: modified 2022-01-31T19:44:44.056897+0000 2022-01-31T19:46:08.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: tableserver 0 2022-01-31T19:46:08.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: root 0 2022-01-31T19:46:08.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_timeout 60 2022-01-31T19:46:08.839 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: session_autoclose 300 2022-01-31T19:46:08.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_file_size 1099511627776 2022-01-31T19:46:08.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: required_client_features {} 2022-01-31T19:46:08.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure 0 2022-01-31T19:46:08.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: last_failure_osd_epoch 0 2022-01-31T19:46:08.840 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-01-31T19:46:08.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: max_mds 2 2022-01-31T19:46:08.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in 0,1 2022-01-31T19:46:08.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: up {0=14436,1=24267} 2022-01-31T19:46:08.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: failed 2022-01-31T19:46:08.841 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: damaged 2022-01-31T19:46:08.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: stopped 2022-01-31T19:46:08.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: data_pools [3] 2022-01-31T19:46:08.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: metadata_pool 2 2022-01-31T19:46:08.842 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: inline_data enabled 2022-01-31T19:46:08.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: balancer 2022-01-31T19:46:08.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: standby_count_wanted 1 2022-01-31T19:46:08.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.sgnfie{0:14436} state up:active seq 2 export targets 1 join_fscid=1 addr [v2:172.21.15.167:6826/1333633577,v1:172.21.15.167:6827/1333633577] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.843 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi167.rdkazj{0:14452} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.167:6828/2852537983,v1:172.21.15.167:6829/2852537983] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.dyqqtj{1:24267} state up:active seq 2 export targets 0 join_fscid=1 addr [v2:172.21.15.171:6824/4256455981,v1:172.21.15.171:6825/4256455981] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: [mds.cephfs.smithi171.erzfea{1:24279} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.171:6826/1322376319,v1:172.21.15.171:6827/1322376319] compat {c=[1],r=[1],i=[77f]}] 2022-01-31T19:46:08.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.844 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.845 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.846 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.847 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: main() 2022-01-31T19:46:08.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: __libc_start_main() 2022-01-31T19:46:08.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: _start() 2022-01-31T19:46:08.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 ceph-mon[103207]: *** Caught signal (Aborted) ** 2022-01-31T19:46:08.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.848 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2022-01-31T19:46:08.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 2: gsignal() 2022-01-31T19:46:08.849 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 3: abort() 2022-01-31T19:46:08.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.850 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.851 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 12: main() 2022-01-31T19:46:08.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 13: __libc_start_main() 2022-01-31T19:46:08.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: 14: _start() 2022-01-31T19:46:08.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:46:08.852 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: -1> 2022-01-31T19:46:08.461+0000 7f24a117e880 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: In function 'void FSMap::sanity(bool) const' thread 7f24a117e880 time 2022-01-31T19:46:08.460430+0000 2022-01-31T19:46:08.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10469-g29e1fc17/rpm/el8/BUILD/ceph-17.0.0-10469-g29e1fc17/src/mds/FSMap.cc: 878: FAILED ceph_assert(info.compat.writeable(fs->mds_map.compat)) 2022-01-31T19:46:08.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f24a03d81d4] 2022-01-31T19:46:08.853 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.854 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: main() 2022-01-31T19:46:08.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: __libc_start_main() 2022-01-31T19:46:08.855 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: _start() 2022-01-31T19:46:08.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 0> 2022-01-31T19:46:08.462+0000 7f24a117e880 -1 *** Caught signal (Aborted) ** 2022-01-31T19:46:08.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: in thread 7f24a117e880 thread_name:ceph-mon 2022-01-31T19:46:08.856 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: ceph version 17.0.0-10469-g29e1fc17 (29e1fc1722aa5915b44828a5ad02ec45ce760aa3) quincy (dev) 2022-01-31T19:46:08.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 1: /lib64/libpthread.so.0(+0x12c20) [0x7f249e36bc20] 2022-01-31T19:46:08.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2: gsignal() 2022-01-31T19:46:08.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 3: abort() 2022-01-31T19:46:08.857 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f24a03d8232] 2022-01-31T19:46:08.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2833f5) [0x7f24a03d83f5] 2022-01-31T19:46:08.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 6: (FSMap::sanity(bool) const+0x2a8) [0x7f24a09a2838] 2022-01-31T19:46:08.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 7: (MDSMonitor::update_from_paxos(bool*)+0x3a3) [0x55d01e119fa3] 2022-01-31T19:46:08.858 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 8: (PaxosService::refresh(bool*)+0x10e) [0x55d01e03db0e] 2022-01-31T19:46:08.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 9: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55d01decf51c] 2022-01-31T19:46:08.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 10: (Monitor::init_paxos()+0x10c) [0x55d01decf82c] 2022-01-31T19:46:08.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 11: (Monitor::preinit()+0xd30) [0x55d01defd2a0] 2022-01-31T19:46:08.859 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 12: main() 2022-01-31T19:46:08.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 13: __libc_start_main() 2022-01-31T19:46:08.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 14: _start() 2022-01-31T19:46:08.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-31T19:46:08.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 conmon[103183]: 2022-01-31T19:46:08.860 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 systemd[1]: Started Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:46:09.419 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Main process exited, code=exited, status=134/n/a 2022-01-31T19:46:09.420 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:08 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Failed with result 'exit-code'. 2022-01-31T19:46:19.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:19 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Service RestartSec=10s expired, scheduling restart. 2022-01-31T19:46:19.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:19 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Scheduled restart job, restart counter is at 5. 2022-01-31T19:46:19.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:19 smithi167 systemd[1]: Stopped Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:46:19.670 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:19 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Start request repeated too quickly. 2022-01-31T19:46:19.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:19 smithi167 systemd[1]: ceph-1c5a1df6-82cd-11ec-8c35-001a4aab830c@mon.smithi167.service: Failed with result 'exit-code'. 2022-01-31T19:46:19.671 INFO:journalctl@ceph.mon.smithi167.smithi167.stdout:Jan 31 19:46:19 smithi167 systemd[1]: Failed to start Ceph mon.smithi167 for 1c5a1df6-82cd-11ec-8c35-001a4aab830c. 2022-01-31T19:47:19.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:19 smithi171 conmon[35325]: debug 2022-01-31T19:47:19.105+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:24.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:24 smithi171 conmon[35325]: debug 2022-01-31T19:47:24.106+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:29 smithi171 conmon[35325]: debug 2022-01-31T19:47:29.107+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:34.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:34 smithi171 conmon[35325]: debug 2022-01-31T19:47:34.107+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:39.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:39 smithi171 conmon[35325]: debug 2022-01-31T19:47:39.108+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:44 smithi171 conmon[35325]: debug 2022-01-31T19:47:44.110+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:49.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:49 smithi171 conmon[35325]: debug 2022-01-31T19:47:49.111+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 380 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:54.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:54 smithi171 conmon[35325]: debug 2022-01-31T19:47:54.112+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 500 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:47:59.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:47:59 smithi171 conmon[35325]: debug 2022-01-31T19:47:59.113+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 618 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:04.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:04 smithi171 conmon[35325]: debug 2022-01-31T19:48:04.114+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 739 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:09.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:09 smithi171 conmon[35325]: debug 2022-01-31T19:48:09.115+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 857 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:14.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:14 smithi171 conmon[35325]: debug 2022-01-31T19:48:14.116+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 976 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:19 smithi171 conmon[35325]: debug 2022-01-31T19:48:19.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1096 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:24.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:24 smithi171 conmon[35325]: debug 2022-01-31T19:48:24.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1215 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:29 smithi171 conmon[35325]: debug 2022-01-31T19:48:29.119+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1333 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:34.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:34 smithi171 conmon[35325]: debug 2022-01-31T19:48:34.121+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1454 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:39.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:39 smithi171 conmon[35325]: debug 2022-01-31T19:48:39.122+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:44 smithi171 conmon[35325]: debug 2022-01-31T19:48:44.123+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:49.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:49 smithi171 conmon[35325]: debug 2022-01-31T19:48:49.124+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1791 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:54.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:54 smithi171 conmon[35325]: debug 2022-01-31T19:48:54.125+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 1907 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:48:59.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:48:59 smithi171 conmon[35325]: debug 2022-01-31T19:48:59.126+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2013 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:04.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:04 smithi171 conmon[35325]: debug 2022-01-31T19:49:04.127+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2124 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:09.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:09 smithi171 conmon[35325]: debug 2022-01-31T19:49:09.127+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2230 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:14.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:14 smithi171 conmon[35325]: debug 2022-01-31T19:49:14.128+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2341 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:19 smithi171 conmon[35325]: debug 2022-01-31T19:49:19.129+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2447 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:24.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:24 smithi171 conmon[35325]: debug 2022-01-31T19:49:24.132+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2558 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:29 smithi171 conmon[35325]: debug 2022-01-31T19:49:29.133+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2664 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:34.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:34 smithi171 conmon[35325]: debug 2022-01-31T19:49:34.134+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2780 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:39.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:39 smithi171 conmon[35325]: debug 2022-01-31T19:49:39.136+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 2886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:44 smithi171 conmon[35325]: debug 2022-01-31T19:49:44.137+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3006 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:49.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:49 smithi171 conmon[35325]: debug 2022-01-31T19:49:49.138+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:54.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:54 smithi171 conmon[35325]: debug 2022-01-31T19:49:54.140+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3227 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:49:59.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:49:59 smithi171 conmon[35325]: debug 2022-01-31T19:49:59.140+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3339 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:04.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:04 smithi171 conmon[35325]: debug 2022-01-31T19:50:04.141+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3454 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:09.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:09 smithi171 conmon[35325]: debug 2022-01-31T19:50:09.143+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3560 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:14.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:14 smithi171 conmon[35325]: debug 2022-01-31T19:50:14.145+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3671 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:19 smithi171 conmon[35325]: debug 2022-01-31T19:50:19.146+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3777 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:24.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:24 smithi171 conmon[35325]: debug 2022-01-31T19:50:24.147+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:29 smithi171 conmon[35325]: debug 2022-01-31T19:50:29.149+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 3994 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:34.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:34 smithi171 conmon[35325]: debug 2022-01-31T19:50:34.150+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4105 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:39.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:39 smithi171 conmon[35325]: debug 2022-01-31T19:50:39.152+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4211 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:44 smithi171 conmon[35325]: debug 2022-01-31T19:50:44.153+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4322 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:46.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:50:46.447+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T19:50:49.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:49 smithi171 conmon[35325]: debug 2022-01-31T19:50:49.154+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4428 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:54.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:54 smithi171 conmon[35325]: debug 2022-01-31T19:50:54.155+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4539 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:50:59.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:50:59 smithi171 conmon[35325]: debug 2022-01-31T19:50:59.157+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4645 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:04.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:04 smithi171 conmon[35325]: debug 2022-01-31T19:51:04.159+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4755 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:09.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:09 smithi171 conmon[35325]: debug 2022-01-31T19:51:09.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4861 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:14.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:14 smithi171 conmon[35325]: debug 2022-01-31T19:51:14.163+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 4971 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:19 smithi171 conmon[35325]: debug 2022-01-31T19:51:19.164+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5077 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:24.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:24 smithi171 conmon[35325]: debug 2022-01-31T19:51:24.166+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5187 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:29 smithi171 conmon[35325]: debug 2022-01-31T19:51:29.167+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5293 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:34.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:34 smithi171 conmon[35325]: debug 2022-01-31T19:51:34.169+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5403 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:39.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:39 smithi171 conmon[35325]: debug 2022-01-31T19:51:39.170+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5509 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:44 smithi171 conmon[35325]: debug 2022-01-31T19:51:44.171+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5618 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:49.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:49 smithi171 conmon[35325]: debug 2022-01-31T19:51:49.173+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5725 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:54.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:54 smithi171 conmon[35325]: debug 2022-01-31T19:51:54.176+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5834 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:51:59.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:51:59 smithi171 conmon[35325]: debug 2022-01-31T19:51:59.177+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 5941 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:04.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:04 smithi171 conmon[35325]: debug 2022-01-31T19:52:04.179+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6051 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:09.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:09 smithi171 conmon[35325]: debug 2022-01-31T19:52:09.181+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6158 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:14.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:14 smithi171 conmon[35325]: debug 2022-01-31T19:52:14.183+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6268 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:19 smithi171 conmon[35325]: debug 2022-01-31T19:52:19.184+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6375 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:24.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:24 smithi171 conmon[35325]: debug 2022-01-31T19:52:24.186+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6484 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:29 smithi171 conmon[35325]: debug 2022-01-31T19:52:29.187+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6592 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:34 smithi171 conmon[35325]: debug 2022-01-31T19:52:34.188+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6701 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:39 smithi171 conmon[35325]: debug 2022-01-31T19:52:39.190+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:44 smithi171 conmon[35325]: debug 2022-01-31T19:52:44.193+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 6919 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:49 smithi171 conmon[35325]: debug 2022-01-31T19:52:49.194+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7027 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:54 smithi171 conmon[35325]: debug 2022-01-31T19:52:54.196+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7136 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:52:59.457 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:52:59 smithi171 conmon[35325]: debug 2022-01-31T19:52:59.198+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7244 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:04.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:04 smithi171 conmon[35325]: debug 2022-01-31T19:53:04.200+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7353 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:09 smithi171 conmon[35325]: debug 2022-01-31T19:53:09.202+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:14.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:14 smithi171 conmon[35325]: debug 2022-01-31T19:53:14.204+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:19.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:19 smithi171 conmon[35325]: debug 2022-01-31T19:53:19.205+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7677 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:24.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:24 smithi171 conmon[35325]: debug 2022-01-31T19:53:24.207+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7785 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:29 smithi171 conmon[35325]: debug 2022-01-31T19:53:29.210+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 7893 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:34 smithi171 conmon[35325]: debug 2022-01-31T19:53:34.211+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8001 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:39 smithi171 conmon[35325]: debug 2022-01-31T19:53:39.213+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8109 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:44 smithi171 conmon[35325]: debug 2022-01-31T19:53:44.215+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8218 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:49 smithi171 conmon[35325]: debug 2022-01-31T19:53:49.217+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8326 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:54 smithi171 conmon[35325]: debug 2022-01-31T19:53:54.219+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8435 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:53:59.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:53:59 smithi171 conmon[35325]: debug 2022-01-31T19:53:59.221+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8543 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:04.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:04 smithi171 conmon[35325]: debug 2022-01-31T19:54:04.223+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8652 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:09 smithi171 conmon[35325]: debug 2022-01-31T19:54:09.224+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8752 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:09.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:09 smithi171 conmon[35325]: 2022-01-31T19:54:14.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:14 smithi171 conmon[35325]: debug 2022-01-31T19:54:14.226+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8862 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:19 smithi171 conmon[35325]: debug 2022-01-31T19:54:19.229+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 8973 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:24.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:24 smithi171 conmon[35325]: debug 2022-01-31T19:54:24.231+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9084 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:29.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:29 smithi171 conmon[35325]: debug 2022-01-31T19:54:29.233+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:34 smithi171 conmon[35325]: debug 2022-01-31T19:54:34.234+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9304 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:39 smithi171 conmon[35325]: debug 2022-01-31T19:54:39.236+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9437 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:44 smithi171 conmon[35325]: debug 2022-01-31T19:54:44.238+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9551 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:49 smithi171 conmon[35325]: debug 2022-01-31T19:54:49.240+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9661 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:54.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:54 smithi171 conmon[35325]: debug 2022-01-31T19:54:54.241+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9775 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:54:59.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:54:59 smithi171 conmon[35325]: debug 2022-01-31T19:54:59.243+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 9888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:04.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:04 smithi171 conmon[35325]: debug 2022-01-31T19:55:04.245+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10008 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:09 smithi171 conmon[35325]: debug 2022-01-31T19:55:09.248+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10129 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:14.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:14 smithi171 conmon[35325]: debug 2022-01-31T19:55:14.250+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10294 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:19.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:19 smithi171 conmon[35325]: debug 2022-01-31T19:55:19.252+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10401 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:24.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:24 smithi171 conmon[35325]: debug 2022-01-31T19:55:24.254+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10511 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:29.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:29 smithi171 conmon[35325]: debug 2022-01-31T19:55:29.256+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10617 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:34 smithi171 conmon[35325]: debug 2022-01-31T19:55:34.258+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10724 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:39 smithi171 conmon[35325]: debug 2022-01-31T19:55:39.261+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10830 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:44 smithi171 conmon[35325]: debug 2022-01-31T19:55:44.262+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 10941 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:46.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T19:55:46.448+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T19:55:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:49 smithi171 conmon[35325]: debug 2022-01-31T19:55:49.264+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11047 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:54 smithi171 conmon[35325]: debug 2022-01-31T19:55:54.267+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11158 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:55:59.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:55:59 smithi171 conmon[35325]: debug 2022-01-31T19:55:59.270+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11264 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:04.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:04 smithi171 conmon[35325]: debug 2022-01-31T19:56:04.272+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11375 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:09 smithi171 conmon[35325]: debug 2022-01-31T19:56:09.273+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11481 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:14.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:14 smithi171 conmon[35325]: debug 2022-01-31T19:56:14.276+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11592 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:19.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:19 smithi171 conmon[35325]: debug 2022-01-31T19:56:19.278+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11698 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:24.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:24 smithi171 conmon[35325]: debug 2022-01-31T19:56:24.280+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:29.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:29 smithi171 conmon[35325]: debug 2022-01-31T19:56:29.281+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 11915 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:34 smithi171 conmon[35325]: debug 2022-01-31T19:56:34.284+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12026 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:39 smithi171 conmon[35325]: debug 2022-01-31T19:56:39.286+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12128 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:44 smithi171 conmon[35325]: debug 2022-01-31T19:56:44.289+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12239 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:49 smithi171 conmon[35325]: debug 2022-01-31T19:56:49.291+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12345 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:54 smithi171 conmon[35325]: debug 2022-01-31T19:56:54.294+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12456 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:56:59.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:56:59 smithi171 conmon[35325]: debug 2022-01-31T19:56:59.296+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12562 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:04.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:04 smithi171 conmon[35325]: debug 2022-01-31T19:57:04.298+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12673 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:09 smithi171 conmon[35325]: debug 2022-01-31T19:57:09.301+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12779 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:14.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:14 smithi171 conmon[35325]: debug 2022-01-31T19:57:14.303+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12890 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:19.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:19 smithi171 conmon[35325]: debug 2022-01-31T19:57:19.305+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 12996 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:24.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:24 smithi171 conmon[35325]: debug 2022-01-31T19:57:24.307+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13107 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:29.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:29 smithi171 conmon[35325]: debug 2022-01-31T19:57:29.311+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13213 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:34 smithi171 conmon[35325]: debug 2022-01-31T19:57:34.313+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13324 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:39 smithi171 conmon[35325]: debug 2022-01-31T19:57:39.316+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13430 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:44 smithi171 conmon[35325]: debug 2022-01-31T19:57:44.318+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13541 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:49 smithi171 conmon[35325]: debug 2022-01-31T19:57:49.320+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13647 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:54 smithi171 conmon[35325]: debug 2022-01-31T19:57:54.323+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13758 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:57:59.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:57:59 smithi171 conmon[35325]: debug 2022-01-31T19:57:59.325+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13864 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:04.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:04 smithi171 conmon[35325]: debug 2022-01-31T19:58:04.327+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 13975 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:09 smithi171 conmon[35325]: debug 2022-01-31T19:58:09.330+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14081 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:14.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:14 smithi171 conmon[35325]: debug 2022-01-31T19:58:14.333+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:19.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:19 smithi171 conmon[35325]: debug 2022-01-31T19:58:19.336+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14298 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:24.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:24 smithi171 conmon[35325]: debug 2022-01-31T19:58:24.338+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14409 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:29.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:29 smithi171 conmon[35325]: debug 2022-01-31T19:58:29.341+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14515 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:34.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:34 smithi171 conmon[35325]: debug 2022-01-31T19:58:34.344+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14625 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:39.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:39 smithi171 conmon[35325]: debug 2022-01-31T19:58:39.346+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14732 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:44 smithi171 conmon[35325]: debug 2022-01-31T19:58:44.349+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14838 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:49 smithi171 conmon[35325]: debug 2022-01-31T19:58:49.350+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 14945 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:54 smithi171 conmon[35325]: debug 2022-01-31T19:58:54.353+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15055 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:58:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:58:59 smithi171 conmon[35325]: debug 2022-01-31T19:58:59.357+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:04.793 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:04 smithi171 conmon[35325]: debug 2022-01-31T19:59:04.360+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15271 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:09 smithi171 conmon[35325]: debug 2022-01-31T19:59:09.363+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15379 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:14 smithi171 conmon[35325]: debug 2022-01-31T19:59:14.365+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15488 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:19 smithi171 conmon[35325]: debug 2022-01-31T19:59:19.368+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15596 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:24.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:24 smithi171 conmon[35325]: debug 2022-01-31T19:59:24.371+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15705 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:29 smithi171 conmon[35325]: debug 2022-01-31T19:59:29.374+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15813 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:34 smithi171 conmon[35325]: debug 2022-01-31T19:59:34.376+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 15922 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:39 smithi171 conmon[35325]: debug 2022-01-31T19:59:39.378+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16030 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:44.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:44 smithi171 conmon[35325]: debug 2022-01-31T19:59:44.382+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16139 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:49 smithi171 conmon[35325]: debug 2022-01-31T19:59:49.385+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16243 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:54 smithi171 conmon[35325]: debug 2022-01-31T19:59:54.388+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16352 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T19:59:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 19:59:59 smithi171 conmon[35325]: debug 2022-01-31T19:59:59.391+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16460 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:04 smithi171 conmon[35325]: debug 2022-01-31T20:00:04.394+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16565 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:09 smithi171 conmon[35325]: debug 2022-01-31T20:00:09.396+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16673 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:14 smithi171 conmon[35325]: debug 2022-01-31T20:00:14.400+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16782 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:19 smithi171 conmon[35325]: debug 2022-01-31T20:00:19.401+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16890 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:24.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:24 smithi171 conmon[35325]: debug 2022-01-31T20:00:24.405+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 16999 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:29 smithi171 conmon[35325]: debug 2022-01-31T20:00:29.408+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17107 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:34 smithi171 conmon[35325]: debug 2022-01-31T20:00:34.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17216 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:39 smithi171 conmon[35325]: debug 2022-01-31T20:00:39.414+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17324 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:44.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:44 smithi171 conmon[35325]: debug 2022-01-31T20:00:44.418+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17433 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:46.448 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:00:46.449+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:00:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:49 smithi171 conmon[35325]: debug 2022-01-31T20:00:49.421+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17541 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:54 smithi171 conmon[35325]: debug 2022-01-31T20:00:54.424+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17650 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:00:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:00:59 smithi171 conmon[35325]: debug 2022-01-31T20:00:59.427+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17758 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:04 smithi171 conmon[35325]: debug 2022-01-31T20:01:04.429+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17867 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:09 smithi171 conmon[35325]: debug 2022-01-31T20:01:09.432+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 17975 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:14 smithi171 conmon[35325]: debug 2022-01-31T20:01:14.435+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18084 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:19.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:19 smithi171 conmon[35325]: debug 2022-01-31T20:01:19.439+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18184 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:24.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:24 smithi171 conmon[35325]: debug 2022-01-31T20:01:24.443+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18293 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:29 smithi171 conmon[35325]: debug 2022-01-31T20:01:29.446+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18401 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:34 smithi171 conmon[35325]: debug 2022-01-31T20:01:34.449+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18510 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:39 smithi171 conmon[35325]: debug 2022-01-31T20:01:39.453+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18618 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:44.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:44 smithi171 conmon[35325]: debug 2022-01-31T20:01:44.456+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18727 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:49 smithi171 conmon[35325]: debug 2022-01-31T20:01:49.458+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18835 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:54 smithi171 conmon[35325]: debug 2022-01-31T20:01:54.461+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 18944 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:01:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:01:59 smithi171 conmon[35325]: debug 2022-01-31T20:01:59.465+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19051 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:04 smithi171 conmon[35325]: debug 2022-01-31T20:02:04.469+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19161 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:09 smithi171 conmon[35325]: debug 2022-01-31T20:02:09.472+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19268 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:14 smithi171 conmon[35325]: debug 2022-01-31T20:02:14.475+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19378 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:19 smithi171 conmon[35325]: debug 2022-01-31T20:02:19.479+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19485 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:24.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:24 smithi171 conmon[35325]: debug 2022-01-31T20:02:24.482+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19595 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:29 smithi171 conmon[35325]: debug 2022-01-31T20:02:29.485+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19701 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:34 smithi171 conmon[35325]: debug 2022-01-31T20:02:34.489+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19812 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:39 smithi171 conmon[35325]: debug 2022-01-31T20:02:39.491+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 19918 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:44.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:44 smithi171 conmon[35325]: debug 2022-01-31T20:02:44.494+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20029 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:49 smithi171 conmon[35325]: debug 2022-01-31T20:02:49.499+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20135 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:54 smithi171 conmon[35325]: debug 2022-01-31T20:02:54.502+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20246 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:02:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:02:59 smithi171 conmon[35325]: debug 2022-01-31T20:02:59.506+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20352 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:04 smithi171 conmon[35325]: debug 2022-01-31T20:03:04.509+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20463 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:09 smithi171 conmon[35325]: debug 2022-01-31T20:03:09.512+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:14 smithi171 conmon[35325]: debug 2022-01-31T20:03:14.516+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20680 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:19 smithi171 conmon[35325]: debug 2022-01-31T20:03:19.520+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20786 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:24.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:24 smithi171 conmon[35325]: debug 2022-01-31T20:03:24.523+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 20897 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:29 smithi171 conmon[35325]: debug 2022-01-31T20:03:29.526+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21003 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:34 smithi171 conmon[35325]: debug 2022-01-31T20:03:34.529+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21114 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:39 smithi171 conmon[35325]: debug 2022-01-31T20:03:39.534+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21220 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:44.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:44 smithi171 conmon[35325]: debug 2022-01-31T20:03:44.537+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21331 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:49 smithi171 conmon[35325]: debug 2022-01-31T20:03:49.541+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21437 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:54 smithi171 conmon[35325]: debug 2022-01-31T20:03:54.545+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21548 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:03:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:03:59 smithi171 conmon[35325]: debug 2022-01-31T20:03:59.549+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21654 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:04 smithi171 conmon[35325]: debug 2022-01-31T20:04:04.553+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21765 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:09 smithi171 conmon[35325]: debug 2022-01-31T20:04:09.556+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21871 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:14 smithi171 conmon[35325]: debug 2022-01-31T20:04:14.559+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 21982 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:19 smithi171 conmon[35325]: debug 2022-01-31T20:04:19.562+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22088 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:24.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:24 smithi171 conmon[35325]: debug 2022-01-31T20:04:24.567+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22199 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:29 smithi171 conmon[35325]: debug 2022-01-31T20:04:29.571+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22305 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:34 smithi171 conmon[35325]: debug 2022-01-31T20:04:34.575+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22416 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:39 smithi171 conmon[35325]: debug 2022-01-31T20:04:39.578+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22522 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:44.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:44 smithi171 conmon[35325]: debug 2022-01-31T20:04:44.582+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22633 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:49 smithi171 conmon[35325]: debug 2022-01-31T20:04:49.585+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22739 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:54 smithi171 conmon[35325]: debug 2022-01-31T20:04:54.589+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22850 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:04:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:04:59 smithi171 conmon[35325]: debug 2022-01-31T20:04:59.592+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 22956 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:04 smithi171 conmon[35325]: debug 2022-01-31T20:05:04.596+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23067 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:04.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:04 smithi171 conmon[35325]: 2022-01-31T20:05:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:09 smithi171 conmon[35325]: debug 2022-01-31T20:05:09.599+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23173 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:14 smithi171 conmon[35325]: debug 2022-01-31T20:05:14.604+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23283 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:20.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:19 smithi171 conmon[35325]: debug 2022-01-31T20:05:19.608+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23389 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:25.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:24 smithi171 conmon[35325]: debug 2022-01-31T20:05:24.612+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23500 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:30.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:29 smithi171 conmon[35325]: debug 2022-01-31T20:05:29.616+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23606 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:35.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:34 smithi171 conmon[35325]: debug 2022-01-31T20:05:34.619+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23717 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:40.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:39 smithi171 conmon[35325]: debug 2022-01-31T20:05:39.623+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23823 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:45.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:44 smithi171 conmon[35325]: debug 2022-01-31T20:05:44.627+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 23933 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:46.450 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:05:46.449+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:05:50.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:49 smithi171 conmon[35325]: debug 2022-01-31T20:05:49.631+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24041 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:05:55.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:54 smithi171 conmon[35325]: debug 2022-01-31T20:05:54.635+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24150 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:00.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:05:59 smithi171 conmon[35325]: debug 2022-01-31T20:05:59.640+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24257 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:04 smithi171 conmon[35325]: debug 2022-01-31T20:06:04.644+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24366 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:10.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:09 smithi171 conmon[35325]: debug 2022-01-31T20:06:09.648+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24475 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:15.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:14 smithi171 conmon[35325]: debug 2022-01-31T20:06:14.652+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24583 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:20.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:19 smithi171 conmon[35325]: debug 2022-01-31T20:06:19.656+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24691 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:25.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:24 smithi171 conmon[35325]: debug 2022-01-31T20:06:24.660+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24800 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:30.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:29 smithi171 conmon[35325]: debug 2022-01-31T20:06:29.664+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 24909 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:35.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:34 smithi171 conmon[35325]: debug 2022-01-31T20:06:34.667+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25017 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:40.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:39 smithi171 conmon[35325]: debug 2022-01-31T20:06:39.671+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25126 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:45.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:44 smithi171 conmon[35325]: debug 2022-01-31T20:06:44.673+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25234 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:50.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:49 smithi171 conmon[35325]: debug 2022-01-31T20:06:49.680+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25343 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:06:55.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:54 smithi171 conmon[35325]: debug 2022-01-31T20:06:54.684+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25451 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:00.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:06:59 smithi171 conmon[35325]: debug 2022-01-31T20:06:59.688+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25560 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:04 smithi171 conmon[35325]: debug 2022-01-31T20:07:04.693+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25668 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:10.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:09 smithi171 conmon[35325]: debug 2022-01-31T20:07:09.696+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25777 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:15.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:14 smithi171 conmon[35325]: debug 2022-01-31T20:07:14.701+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:20.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:19 smithi171 conmon[35325]: debug 2022-01-31T20:07:19.705+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 25995 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:25.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:24 smithi171 conmon[35325]: debug 2022-01-31T20:07:24.709+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26103 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:30.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:29 smithi171 conmon[35325]: debug 2022-01-31T20:07:29.713+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26212 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:35.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:34 smithi171 conmon[35325]: debug 2022-01-31T20:07:34.718+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26320 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:40.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:39 smithi171 conmon[35325]: debug 2022-01-31T20:07:39.722+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26429 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:45.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:44 smithi171 conmon[35325]: debug 2022-01-31T20:07:44.726+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26537 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:50.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:49 smithi171 conmon[35325]: debug 2022-01-31T20:07:49.731+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26646 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:07:55.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:54 smithi171 conmon[35325]: debug 2022-01-31T20:07:54.735+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26754 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:00.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:07:59 smithi171 conmon[35325]: debug 2022-01-31T20:07:59.739+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26863 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:04 smithi171 conmon[35325]: debug 2022-01-31T20:08:04.743+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 26971 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:10.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:09 smithi171 conmon[35325]: debug 2022-01-31T20:08:09.747+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27080 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:15.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:14 smithi171 conmon[35325]: debug 2022-01-31T20:08:14.751+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27188 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:20.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:19 smithi171 conmon[35325]: debug 2022-01-31T20:08:19.756+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27297 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:25.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:24 smithi171 conmon[35325]: debug 2022-01-31T20:08:24.761+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27405 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:30.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:29 smithi171 conmon[35325]: debug 2022-01-31T20:08:30.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:29 smithi171 conmon[35325]: 2022-01-31T20:08:29.767+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27514 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:35.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:34 smithi171 conmon[35325]: debug 2022-01-31T20:08:34.771+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27622 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:40.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:39 smithi171 conmon[35325]: debug 2022-01-31T20:08:39.776+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27721 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:45.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:44 smithi171 conmon[35325]: debug 2022-01-31T20:08:44.780+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27829 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:50.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:49 smithi171 conmon[35325]: debug 2022-01-31T20:08:49.785+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 27938 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:08:55.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:54 smithi171 conmon[35325]: debug 2022-01-31T20:08:54.789+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28046 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:00.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:08:59 smithi171 conmon[35325]: debug 2022-01-31T20:08:59.793+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28155 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:04 smithi171 conmon[35325]: debug 2022-01-31T20:09:04.797+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28263 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:10.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:09 smithi171 conmon[35325]: debug 2022-01-31T20:09:09.803+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28371 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:15.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:14 smithi171 conmon[35325]: debug 2022-01-31T20:09:14.807+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28480 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:20.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:19 smithi171 conmon[35325]: debug 2022-01-31T20:09:19.811+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28588 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:25.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:24 smithi171 conmon[35325]: debug 2022-01-31T20:09:24.816+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28697 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:30.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:29 smithi171 conmon[35325]: debug 2022-01-31T20:09:29.820+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28805 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:35.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:34 smithi171 conmon[35325]: debug 2022-01-31T20:09:34.825+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 28914 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:40.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:39 smithi171 conmon[35325]: debug 2022-01-31T20:09:39.830+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29022 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:45.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:44 smithi171 conmon[35325]: debug 2022-01-31T20:09:44.834+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29131 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:50.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:49 smithi171 conmon[35325]: debug 2022-01-31T20:09:49.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29238 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:09:55.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:54 smithi171 conmon[35325]: debug 2022-01-31T20:09:54.844+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29348 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:00.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:09:59 smithi171 conmon[35325]: debug 2022-01-31T20:09:59.848+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29455 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:04 smithi171 conmon[35325]: debug 2022-01-31T20:10:04.853+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29565 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:10.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:09 smithi171 conmon[35325]: debug 2022-01-31T20:10:09.858+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29672 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:15.001 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:14 smithi171 conmon[35325]: debug 2022-01-31T20:10:14.863+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29782 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:20.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:19 smithi171 conmon[35325]: debug 2022-01-31T20:10:19.867+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29889 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:25.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:24 smithi171 conmon[35325]: debug 2022-01-31T20:10:24.872+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 29999 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:30.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:29 smithi171 conmon[35325]: debug 2022-01-31T20:10:29.877+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30106 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:35.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:34 smithi171 conmon[35325]: debug 2022-01-31T20:10:34.880+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30216 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:40.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:39 smithi171 conmon[35325]: debug 2022-01-31T20:10:39.885+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30323 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:45.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:44 smithi171 conmon[35325]: debug 2022-01-31T20:10:44.891+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30433 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:46.452 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:10:46.449+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:10:50.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:49 smithi171 conmon[35325]: debug 2022-01-31T20:10:49.892+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30540 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:10:55.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:10:54 smithi171 conmon[35325]: debug 2022-01-31T20:10:54.901+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30650 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:00.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:00 smithi171 conmon[35325]: debug 2022-01-31T20:10:59.905+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30757 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:05.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:04 smithi171 conmon[35325]: debug 2022-01-31T20:11:04.910+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30867 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:10.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:09 smithi171 conmon[35325]: debug 2022-01-31T20:11:09.915+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 30974 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:15.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:14 smithi171 conmon[35325]: debug 2022-01-31T20:11:14.919+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31084 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:20.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:19 smithi171 conmon[35325]: debug 2022-01-31T20:11:19.923+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31191 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:25.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:24 smithi171 conmon[35325]: debug 2022-01-31T20:11:24.928+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31301 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:30.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:29 smithi171 conmon[35325]: debug 2022-01-31T20:11:29.933+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31404 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:35.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:34 smithi171 conmon[35325]: debug 2022-01-31T20:11:34.939+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31514 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:40.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:39 smithi171 conmon[35325]: debug 2022-01-31T20:11:39.944+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31622 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:45.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:44 smithi171 conmon[35325]: debug 2022-01-31T20:11:44.949+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31731 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:50.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:49 smithi171 conmon[35325]: debug 2022-01-31T20:11:49.954+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31839 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:11:55.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:54 smithi171 conmon[35325]: debug 2022-01-31T20:11:54.959+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 31948 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:00.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:11:59 smithi171 conmon[35325]: debug 2022-01-31T20:11:59.964+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32056 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:05.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:04 smithi171 conmon[35325]: debug 2022-01-31T20:12:04.969+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32165 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:10.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:09 smithi171 conmon[35325]: debug 2022-01-31T20:12:09.973+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32326 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:15.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:14 smithi171 conmon[35325]: debug 2022-01-31T20:12:14.978+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32435 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:20.352 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:19 smithi171 conmon[35325]: debug 2022-01-31T20:12:19.984+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32543 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:25.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:24 smithi171 conmon[35325]: debug 2022-01-31T20:12:24.989+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32652 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:30.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:29 smithi171 conmon[35325]: debug 2022-01-31T20:12:29.994+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32760 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:35.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:35 smithi171 conmon[35325]: debug 2022-01-31T20:12:34.999+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32869 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:40.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:40 smithi171 conmon[35325]: debug 2022-01-31T20:12:40.004+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 32977 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:45.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:45 smithi171 conmon[35325]: debug 2022-01-31T20:12:45.009+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33082 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:50.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:50 smithi171 conmon[35325]: debug 2022-01-31T20:12:50.010+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33190 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:12:55.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:12:55 smithi171 conmon[35325]: debug 2022-01-31T20:12:55.019+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33299 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:00.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:00 smithi171 conmon[35325]: debug 2022-01-31T20:13:00.024+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33407 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:05.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:05 smithi171 conmon[35325]: debug 2022-01-31T20:13:05.029+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33516 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:10.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:10 smithi171 conmon[35325]: debug 2022-01-31T20:13:10.035+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33624 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:15.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:15 smithi171 conmon[35325]: debug 2022-01-31T20:13:15.040+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33732 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:20.322 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:20 smithi171 conmon[35325]: debug 2022-01-31T20:13:20.046+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33841 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:25.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:25 smithi171 conmon[35325]: debug 2022-01-31T20:13:25.051+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 33948 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:30.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:30 smithi171 conmon[35325]: debug 2022-01-31T20:13:30.056+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34058 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:35.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:35 smithi171 conmon[35325]: debug 2022-01-31T20:13:35.061+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34165 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:40.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:40 smithi171 conmon[35325]: debug 2022-01-31T20:13:40.067+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34274 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:45.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:45 smithi171 conmon[35325]: debug 2022-01-31T20:13:45.071+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34381 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:50.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:50 smithi171 conmon[35325]: debug 2022-01-31T20:13:50.076+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34491 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:13:55.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:13:55 smithi171 conmon[35325]: debug 2022-01-31T20:13:55.083+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34598 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:00.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:00 smithi171 conmon[35325]: debug 2022-01-31T20:14:00.088+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34708 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:05.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:05 smithi171 conmon[35325]: debug 2022-01-31T20:14:05.093+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34815 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:10.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:10 smithi171 conmon[35325]: debug 2022-01-31T20:14:10.098+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 34925 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:15.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:15 smithi171 conmon[35325]: debug 2022-01-31T20:14:15.103+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35032 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:20.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:20 smithi171 conmon[35325]: debug 2022-01-31T20:14:20.109+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35142 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:25 smithi171 conmon[35325]: debug 2022-01-31T20:14:25.114+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35249 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:30.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:30 smithi171 conmon[35325]: debug 2022-01-31T20:14:30.119+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35359 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:35 smithi171 conmon[35325]: debug 2022-01-31T20:14:35.124+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35466 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:40.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:40 smithi171 conmon[35325]: debug 2022-01-31T20:14:40.129+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35576 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:43.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:43 smithi171 conmon[35325]: debug 2022-01-31T20:14:43.074+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:14:43.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:43 smithi171 conmon[35325]: debug 2022-01-31T20:14:43.275+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:14:44.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:43 smithi171 conmon[35325]: debug 2022-01-31T20:14:43.676+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:14:44.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:44 smithi171 conmon[35325]: debug 2022-01-31T20:14:44.477+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:14:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:45 smithi171 conmon[35325]: debug 2022-01-31T20:14:45.136+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35683 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:46.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:46 smithi171 conmon[35325]: debug 2022-01-31T20:14:46.078+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:14:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:49 smithi171 conmon[35325]: debug 2022-01-31T20:14:49.279+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:14:50.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:50 smithi171 conmon[35325]: debug 2022-01-31T20:14:50.142+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35793 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:55.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:55 smithi171 conmon[35325]: debug 2022-01-31T20:14:55.147+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 35900 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:14:56.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:14:55 smithi171 conmon[35325]: debug 2022-01-31T20:14:55.680+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:15:00.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:00 smithi171 conmon[35325]: debug 2022-01-31T20:15:00.153+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36010 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:05 smithi171 conmon[35325]: debug 2022-01-31T20:15:05.159+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36117 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:08 smithi171 conmon[35325]: debug 2022-01-31T20:15:08.481+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:15:10.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:10 smithi171 conmon[35325]: debug 2022-01-31T20:15:10.164+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36227 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:15.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:15 smithi171 conmon[35325]: debug 2022-01-31T20:15:15.169+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36334 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:20.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:20 smithi171 conmon[35325]: debug 2022-01-31T20:15:20.174+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36444 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:23 smithi171 conmon[35325]: debug 2022-01-31T20:15:23.481+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:15:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:25 smithi171 conmon[35325]: debug 2022-01-31T20:15:25.179+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36551 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:30.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:30 smithi171 conmon[35325]: debug 2022-01-31T20:15:30.186+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36661 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:35 smithi171 conmon[35325]: debug 2022-01-31T20:15:35.191+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36768 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:38 smithi171 conmon[35325]: debug 2022-01-31T20:15:38.482+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:15:40.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:40 smithi171 conmon[35325]: debug 2022-01-31T20:15:40.197+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36878 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:45 smithi171 conmon[35325]: debug 2022-01-31T20:15:45.202+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 36985 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:15:46.451+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:15:50.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:50 smithi171 conmon[35325]: debug 2022-01-31T20:15:50.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37095 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:15:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:53 smithi171 conmon[35325]: debug 2022-01-31T20:15:53.483+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:15:55.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:15:55 smithi171 conmon[35325]: debug 2022-01-31T20:15:55.213+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37202 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:00.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:00 smithi171 conmon[35325]: debug 2022-01-31T20:16:00.219+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37312 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:05 smithi171 conmon[35325]: debug 2022-01-31T20:16:05.223+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37419 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:08 smithi171 conmon[35325]: debug 2022-01-31T20:16:08.483+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:16:10.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:10 smithi171 conmon[35325]: debug 2022-01-31T20:16:10.229+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37529 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:15.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:15 smithi171 conmon[35325]: debug 2022-01-31T20:16:15.234+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37636 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:20.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:20 smithi171 conmon[35325]: debug 2022-01-31T20:16:20.241+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37746 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:23 smithi171 conmon[35325]: debug 2022-01-31T20:16:23.484+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:16:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:25 smithi171 conmon[35325]: debug 2022-01-31T20:16:25.246+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37853 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:30.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:30 smithi171 conmon[35325]: debug 2022-01-31T20:16:30.252+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 37963 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:35 smithi171 conmon[35325]: debug 2022-01-31T20:16:35.258+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38070 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:38 smithi171 conmon[35325]: debug 2022-01-31T20:16:38.485+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:16:40.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:40 smithi171 conmon[35325]: debug 2022-01-31T20:16:40.263+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38180 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:45 smithi171 conmon[35325]: debug 2022-01-31T20:16:45.269+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38287 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:50.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:50 smithi171 conmon[35325]: debug 2022-01-31T20:16:50.274+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38396 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:16:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:53 smithi171 conmon[35325]: debug 2022-01-31T20:16:53.485+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:16:55.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:16:55 smithi171 conmon[35325]: debug 2022-01-31T20:16:55.279+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38504 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:00.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:00 smithi171 conmon[35325]: debug 2022-01-31T20:17:00.285+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38613 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:05 smithi171 conmon[35325]: debug 2022-01-31T20:17:05.292+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38721 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:08 smithi171 conmon[35325]: debug 2022-01-31T20:17:08.486+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:17:10.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:10 smithi171 conmon[35325]: debug 2022-01-31T20:17:10.297+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38830 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:15.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:15 smithi171 conmon[35325]: debug 2022-01-31T20:17:15.303+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 38938 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:20.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:20 smithi171 conmon[35325]: debug 2022-01-31T20:17:20.309+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39047 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:23 smithi171 conmon[35325]: debug 2022-01-31T20:17:23.487+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:17:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:25 smithi171 conmon[35325]: debug 2022-01-31T20:17:25.315+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39151 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:30.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:30 smithi171 conmon[35325]: debug 2022-01-31T20:17:30.321+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39260 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:35 smithi171 conmon[35325]: debug 2022-01-31T20:17:35.327+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39364 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:38 smithi171 conmon[35325]: debug 2022-01-31T20:17:38.488+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:17:40.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:40 smithi171 conmon[35325]: debug 2022-01-31T20:17:40.332+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39472 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:45 smithi171 conmon[35325]: debug 2022-01-31T20:17:45.337+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39581 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:50.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:50 smithi171 conmon[35325]: debug 2022-01-31T20:17:50.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39689 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:17:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:53 smithi171 conmon[35325]: debug 2022-01-31T20:17:53.488+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:17:55.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:17:55 smithi171 conmon[35325]: debug 2022-01-31T20:17:55.350+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39794 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:00.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:00 smithi171 conmon[35325]: debug 2022-01-31T20:18:00.356+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 39902 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:05.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:05 smithi171 conmon[35325]: debug 2022-01-31T20:18:05.362+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40011 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:08 smithi171 conmon[35325]: debug 2022-01-31T20:18:08.489+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:18:10.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:10 smithi171 conmon[35325]: debug 2022-01-31T20:18:10.368+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40115 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:15.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:15 smithi171 conmon[35325]: debug 2022-01-31T20:18:15.374+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40224 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:20.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:20 smithi171 conmon[35325]: debug 2022-01-31T20:18:20.380+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40332 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:23 smithi171 conmon[35325]: debug 2022-01-31T20:18:23.490+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:18:25.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:25 smithi171 conmon[35325]: debug 2022-01-31T20:18:25.386+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40441 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:30.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:30 smithi171 conmon[35325]: debug 2022-01-31T20:18:30.390+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:35.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:35 smithi171 conmon[35325]: debug 2022-01-31T20:18:35.396+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40658 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:38 smithi171 conmon[35325]: debug 2022-01-31T20:18:38.490+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:18:40.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:40 smithi171 conmon[35325]: debug 2022-01-31T20:18:40.403+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40766 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:45.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:45 smithi171 conmon[35325]: debug 2022-01-31T20:18:45.410+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40875 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:50.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:50 smithi171 conmon[35325]: debug 2022-01-31T20:18:50.416+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 40983 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:18:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:53 smithi171 conmon[35325]: debug 2022-01-31T20:18:53.491+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:18:55.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:18:55 smithi171 conmon[35325]: debug 2022-01-31T20:18:55.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41092 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:00.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:00 smithi171 conmon[35325]: debug 2022-01-31T20:19:00.428+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41200 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:05.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:05 smithi171 conmon[35325]: debug 2022-01-31T20:19:05.434+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41309 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:08 smithi171 conmon[35325]: debug 2022-01-31T20:19:08.492+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:19:10.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:10 smithi171 conmon[35325]: debug 2022-01-31T20:19:10.440+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41417 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:15.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:15 smithi171 conmon[35325]: debug 2022-01-31T20:19:15.445+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41530 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:20.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:20 smithi171 conmon[35325]: debug 2022-01-31T20:19:20.451+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41638 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:23 smithi171 conmon[35325]: debug 2022-01-31T20:19:23.492+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:19:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:25 smithi171 conmon[35325]: debug 2022-01-31T20:19:25.458+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41744 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:30.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:30 smithi171 conmon[35325]: debug 2022-01-31T20:19:30.464+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41852 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:35.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:35 smithi171 conmon[35325]: debug 2022-01-31T20:19:35.470+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 41961 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:38 smithi171 conmon[35325]: debug 2022-01-31T20:19:38.493+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:19:40.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:40 smithi171 conmon[35325]: debug 2022-01-31T20:19:40.477+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42069 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:45.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:45 smithi171 conmon[35325]: debug 2022-01-31T20:19:45.482+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42178 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:50.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:50 smithi171 conmon[35325]: debug 2022-01-31T20:19:50.489+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42286 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:19:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:53 smithi171 conmon[35325]: debug 2022-01-31T20:19:53.494+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:19:55.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:19:55 smithi171 conmon[35325]: debug 2022-01-31T20:19:55.495+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42395 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:00.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:00 smithi171 conmon[35325]: debug 2022-01-31T20:20:00.500+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42503 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:05.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:05 smithi171 conmon[35325]: debug 2022-01-31T20:20:05.506+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42612 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:08 smithi171 conmon[35325]: debug 2022-01-31T20:20:08.494+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:20:10.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:10 smithi171 conmon[35325]: debug 2022-01-31T20:20:10.512+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42720 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:15.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:15 smithi171 conmon[35325]: debug 2022-01-31T20:20:15.519+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42829 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:20.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:20 smithi171 conmon[35325]: debug 2022-01-31T20:20:20.525+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 42937 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:23 smithi171 conmon[35325]: debug 2022-01-31T20:20:23.495+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:20:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:25 smithi171 conmon[35325]: debug 2022-01-31T20:20:25.531+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43046 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:30.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:30 smithi171 conmon[35325]: debug 2022-01-31T20:20:30.538+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43154 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:35.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:35 smithi171 conmon[35325]: debug 2022-01-31T20:20:35.544+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43263 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:38 smithi171 conmon[35325]: debug 2022-01-31T20:20:38.496+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:20:40.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:40 smithi171 conmon[35325]: debug 2022-01-31T20:20:40.551+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43371 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:45.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:45 smithi171 conmon[35325]: debug 2022-01-31T20:20:45.557+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43479 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:20:46.452+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:20:50.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:50 smithi171 conmon[35325]: debug 2022-01-31T20:20:50.562+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43589 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:20:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:53 smithi171 conmon[35325]: debug 2022-01-31T20:20:53.496+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:20:55.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:20:55 smithi171 conmon[35325]: debug 2022-01-31T20:20:55.568+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43698 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:00.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:00 smithi171 conmon[35325]: debug 2022-01-31T20:21:00.575+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:05.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:05 smithi171 conmon[35325]: debug 2022-01-31T20:21:05.582+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 43918 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:08 smithi171 conmon[35325]: debug 2022-01-31T20:21:08.497+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:21:10.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:10 smithi171 conmon[35325]: debug 2022-01-31T20:21:10.588+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44028 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:15.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:15 smithi171 conmon[35325]: debug 2022-01-31T20:21:15.595+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44137 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:20.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:20 smithi171 conmon[35325]: debug 2022-01-31T20:21:20.602+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44247 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:23 smithi171 conmon[35325]: debug 2022-01-31T20:21:23.498+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:21:25.719 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:25 smithi171 conmon[35325]: debug 2022-01-31T20:21:25.609+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44355 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:31.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:30 smithi171 conmon[35325]: debug 2022-01-31T20:21:30.615+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44466 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:36.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:35 smithi171 conmon[35325]: debug 2022-01-31T20:21:35.620+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44574 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:38 smithi171 conmon[35325]: debug 2022-01-31T20:21:38.498+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:21:41.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:40 smithi171 conmon[35325]: debug 2022-01-31T20:21:40.627+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44686 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:46.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:45 smithi171 conmon[35325]: debug 2022-01-31T20:21:45.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44793 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:51.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:50 smithi171 conmon[35325]: debug 2022-01-31T20:21:50.640+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 44905 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:21:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:53 smithi171 conmon[35325]: debug 2022-01-31T20:21:53.499+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:21:56.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:21:55 smithi171 conmon[35325]: debug 2022-01-31T20:21:55.646+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45012 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:01.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:00 smithi171 conmon[35325]: debug 2022-01-31T20:22:00.653+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45124 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:06.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:05 smithi171 conmon[35325]: debug 2022-01-31T20:22:05.659+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45231 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:08 smithi171 conmon[35325]: debug 2022-01-31T20:22:08.500+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:22:11.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:10 smithi171 conmon[35325]: debug 2022-01-31T20:22:10.666+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45343 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:16.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:15 smithi171 conmon[35325]: debug 2022-01-31T20:22:15.673+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45454 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:21.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:20 smithi171 conmon[35325]: debug 2022-01-31T20:22:20.679+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45566 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:23 smithi171 conmon[35325]: debug 2022-01-31T20:22:23.500+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:22:26.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:25 smithi171 conmon[35325]: debug 2022-01-31T20:22:25.685+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45674 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:31.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:30 smithi171 conmon[35325]: debug 2022-01-31T20:22:30.692+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45785 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:36.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:35 smithi171 conmon[35325]: debug 2022-01-31T20:22:35.699+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 45892 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:38 smithi171 conmon[35325]: debug 2022-01-31T20:22:38.501+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:22:41.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:40 smithi171 conmon[35325]: debug 2022-01-31T20:22:40.706+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46004 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:46.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:45 smithi171 conmon[35325]: debug 2022-01-31T20:22:45.712+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:51.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:50 smithi171 conmon[35325]: debug 2022-01-31T20:22:50.719+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46223 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:22:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:53 smithi171 conmon[35325]: debug 2022-01-31T20:22:53.502+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:22:56.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:22:55 smithi171 conmon[35325]: debug 2022-01-31T20:22:55.725+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46331 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:01.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:00 smithi171 conmon[35325]: debug 2022-01-31T20:23:00.733+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46438 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:06.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:05 smithi171 conmon[35325]: debug 2022-01-31T20:23:05.739+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46546 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:08 smithi171 conmon[35325]: debug 2022-01-31T20:23:08.502+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:23:11.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:10 smithi171 conmon[35325]: debug 2022-01-31T20:23:10.745+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46657 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:16.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:15 smithi171 conmon[35325]: debug 2022-01-31T20:23:15.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46765 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:21.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:20 smithi171 conmon[35325]: debug 2022-01-31T20:23:20.760+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46876 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:23 smithi171 conmon[35325]: debug 2022-01-31T20:23:23.503+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:23:26.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:25 smithi171 conmon[35325]: debug 2022-01-31T20:23:25.766+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 46983 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:31.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:30 smithi171 conmon[35325]: debug 2022-01-31T20:23:30.773+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47087 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:36.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:35 smithi171 conmon[35325]: debug 2022-01-31T20:23:35.780+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47195 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:38 smithi171 conmon[35325]: debug 2022-01-31T20:23:38.504+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:23:41.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:40 smithi171 conmon[35325]: debug 2022-01-31T20:23:40.787+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47306 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:46.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:45 smithi171 conmon[35325]: debug 2022-01-31T20:23:45.794+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47414 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:51.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:50 smithi171 conmon[35325]: debug 2022-01-31T20:23:50.800+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47525 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:23:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:53 smithi171 conmon[35325]: debug 2022-01-31T20:23:53.505+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:23:56.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:23:55 smithi171 conmon[35325]: debug 2022-01-31T20:23:55.806+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47633 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:01.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:00 smithi171 conmon[35325]: debug 2022-01-31T20:24:00.812+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47744 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:06.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:05 smithi171 conmon[35325]: debug 2022-01-31T20:24:05.820+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47852 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:08 smithi171 conmon[35325]: debug 2022-01-31T20:24:08.505+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:24:11.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:10 smithi171 conmon[35325]: debug 2022-01-31T20:24:10.827+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 47963 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:16.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:15 smithi171 conmon[35325]: debug 2022-01-31T20:24:15.834+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48071 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:21.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:20 smithi171 conmon[35325]: debug 2022-01-31T20:24:20.840+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48182 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:23 smithi171 conmon[35325]: debug 2022-01-31T20:24:23.506+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:24:26.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:25 smithi171 conmon[35325]: debug 2022-01-31T20:24:25.847+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48290 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:31.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:30 smithi171 conmon[35325]: debug 2022-01-31T20:24:30.854+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48401 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:36.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:35 smithi171 conmon[35325]: debug 2022-01-31T20:24:35.861+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48509 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:38 smithi171 conmon[35325]: debug 2022-01-31T20:24:38.507+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:24:41.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:40 smithi171 conmon[35325]: debug 2022-01-31T20:24:40.867+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48621 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:46.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:45 smithi171 conmon[35325]: debug 2022-01-31T20:24:45.874+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48729 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:51.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:50 smithi171 conmon[35325]: debug 2022-01-31T20:24:50.881+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48839 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:24:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:53 smithi171 conmon[35325]: debug 2022-01-31T20:24:53.507+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:24:56.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:24:55 smithi171 conmon[35325]: debug 2022-01-31T20:24:55.889+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 48946 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:01.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:00 smithi171 conmon[35325]: debug 2022-01-31T20:25:00.895+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49053 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:06.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:05 smithi171 conmon[35325]: debug 2022-01-31T20:25:05.903+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49161 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:08 smithi171 conmon[35325]: debug 2022-01-31T20:25:08.508+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:25:11.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:10 smithi171 conmon[35325]: debug 2022-01-31T20:25:10.910+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49271 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:16.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:15 smithi171 conmon[35325]: debug 2022-01-31T20:25:15.918+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49381 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:21.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:20 smithi171 conmon[35325]: debug 2022-01-31T20:25:20.924+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49491 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:23 smithi171 conmon[35325]: debug 2022-01-31T20:25:23.509+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:25:26.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:25 smithi171 conmon[35325]: debug 2022-01-31T20:25:25.932+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49600 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:31.328 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:30 smithi171 conmon[35325]: debug 2022-01-31T20:25:30.938+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49709 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:36.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:35 smithi171 conmon[35325]: debug 2022-01-31T20:25:35.944+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49819 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:38 smithi171 conmon[35325]: debug 2022-01-31T20:25:38.509+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:25:41.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:40 smithi171 conmon[35325]: debug 2022-01-31T20:25:40.953+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 49928 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:46.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:45 smithi171 conmon[35325]: debug 2022-01-31T20:25:45.960+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50038 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:25:46.453+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:25:51.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:50 smithi171 conmon[35325]: debug 2022-01-31T20:25:50.967+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50147 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:25:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:53 smithi171 conmon[35325]: debug 2022-01-31T20:25:53.510+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:25:56.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:25:55 smithi171 conmon[35325]: debug 2022-01-31T20:25:55.974+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50257 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:01.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:00 smithi171 conmon[35325]: debug 2022-01-31T20:26:00.982+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50366 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:06.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:05 smithi171 conmon[35325]: debug 2022-01-31T20:26:05.989+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50476 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:08 smithi171 conmon[35325]: debug 2022-01-31T20:26:08.512+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:26:11.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:10 smithi171 conmon[35325]: debug 2022-01-31T20:26:10.996+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50585 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:16.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:16 smithi171 conmon[35325]: debug 2022-01-31T20:26:16.002+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50696 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:21.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:21 smithi171 conmon[35325]: debug 2022-01-31T20:26:21.009+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50804 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:23 smithi171 conmon[35325]: debug 2022-01-31T20:26:23.512+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:26:26.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:26 smithi171 conmon[35325]: debug 2022-01-31T20:26:26.016+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 50915 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:31.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:31 smithi171 conmon[35325]: debug 2022-01-31T20:26:31.024+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51023 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:36.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:36 smithi171 conmon[35325]: debug 2022-01-31T20:26:36.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51134 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:38.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:38 smithi171 conmon[35325]: debug 2022-01-31T20:26:38.513+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:26:41.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:41 smithi171 conmon[35325]: debug 2022-01-31T20:26:41.038+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51242 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:46.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:46 smithi171 conmon[35325]: debug 2022-01-31T20:26:46.045+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51353 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:51.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:51 smithi171 conmon[35325]: debug 2022-01-31T20:26:51.052+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:26:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:53 smithi171 conmon[35325]: debug 2022-01-31T20:26:53.514+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:26:56.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:26:56 smithi171 conmon[35325]: debug 2022-01-31T20:26:56.059+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:01.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:01 smithi171 conmon[35325]: debug 2022-01-31T20:27:01.066+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51680 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:06.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:06 smithi171 conmon[35325]: debug 2022-01-31T20:27:06.072+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51791 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:08 smithi171 conmon[35325]: debug 2022-01-31T20:27:08.514+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:27:11.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:11 smithi171 conmon[35325]: debug 2022-01-31T20:27:11.079+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 51899 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:16.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:16 smithi171 conmon[35325]: debug 2022-01-31T20:27:16.087+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52010 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:21.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:21 smithi171 conmon[35325]: debug 2022-01-31T20:27:21.094+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52118 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:23 smithi171 conmon[35325]: debug 2022-01-31T20:27:23.515+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:27:26.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:26 smithi171 conmon[35325]: debug 2022-01-31T20:27:26.102+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52229 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:31.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:31 smithi171 conmon[35325]: debug 2022-01-31T20:27:31.109+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52337 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:36.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:36 smithi171 conmon[35325]: debug 2022-01-31T20:27:36.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52448 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:38 smithi171 conmon[35325]: debug 2022-01-31T20:27:38.516+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:27:41.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:41 smithi171 conmon[35325]: debug 2022-01-31T20:27:41.124+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52556 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:46.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:46 smithi171 conmon[35325]: debug 2022-01-31T20:27:46.131+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52667 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:51.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:51 smithi171 conmon[35325]: debug 2022-01-31T20:27:51.137+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52775 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:27:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:53 smithi171 conmon[35325]: debug 2022-01-31T20:27:53.516+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:27:56.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:27:56 smithi171 conmon[35325]: debug 2022-01-31T20:27:56.145+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:01.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:01 smithi171 conmon[35325]: debug 2022-01-31T20:28:01.153+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 52994 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:06.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:06 smithi171 conmon[35325]: debug 2022-01-31T20:28:06.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53105 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:08 smithi171 conmon[35325]: debug 2022-01-31T20:28:08.517+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:28:11.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:11 smithi171 conmon[35325]: debug 2022-01-31T20:28:11.169+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53213 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:16.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:16 smithi171 conmon[35325]: debug 2022-01-31T20:28:16.176+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53325 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:21.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:21 smithi171 conmon[35325]: debug 2022-01-31T20:28:21.184+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53433 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:23 smithi171 conmon[35325]: debug 2022-01-31T20:28:23.518+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:28:26.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:26 smithi171 conmon[35325]: debug 2022-01-31T20:28:26.191+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53544 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:31.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:31 smithi171 conmon[35325]: debug 2022-01-31T20:28:31.198+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53652 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:36.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:36 smithi171 conmon[35325]: debug 2022-01-31T20:28:36.205+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53763 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:38 smithi171 conmon[35325]: debug 2022-01-31T20:28:38.518+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:28:41.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:41 smithi171 conmon[35325]: debug 2022-01-31T20:28:41.212+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53871 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:46.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:46 smithi171 conmon[35325]: debug 2022-01-31T20:28:46.220+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 53982 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:51.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:51 smithi171 conmon[35325]: debug 2022-01-31T20:28:51.228+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54091 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:28:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:53 smithi171 conmon[35325]: debug 2022-01-31T20:28:53.519+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:28:56.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:28:56 smithi171 conmon[35325]: debug 2022-01-31T20:28:56.236+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54203 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:01.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:01 smithi171 conmon[35325]: debug 2022-01-31T20:29:01.243+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54311 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:06.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:06 smithi171 conmon[35325]: debug 2022-01-31T20:29:06.251+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54421 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:08 smithi171 conmon[35325]: debug 2022-01-31T20:29:08.520+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:29:11.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:11 smithi171 conmon[35325]: debug 2022-01-31T20:29:11.258+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54530 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:16.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:16 smithi171 conmon[35325]: debug 2022-01-31T20:29:16.266+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54641 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:21.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:21 smithi171 conmon[35325]: debug 2022-01-31T20:29:21.273+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54750 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:23 smithi171 conmon[35325]: debug 2022-01-31T20:29:23.521+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:29:26.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:26 smithi171 conmon[35325]: debug 2022-01-31T20:29:26.280+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54860 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:31.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:31 smithi171 conmon[35325]: debug 2022-01-31T20:29:31.287+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 54969 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:36.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:36 smithi171 conmon[35325]: debug 2022-01-31T20:29:36.297+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55079 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:38 smithi171 conmon[35325]: debug 2022-01-31T20:29:38.521+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:29:41.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:41 smithi171 conmon[35325]: debug 2022-01-31T20:29:41.305+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55188 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:46.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:46 smithi171 conmon[35325]: debug 2022-01-31T20:29:46.312+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55298 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:51.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:51 smithi171 conmon[35325]: debug 2022-01-31T20:29:51.320+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55407 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:29:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:53 smithi171 conmon[35325]: debug 2022-01-31T20:29:53.522+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:29:56.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:29:56 smithi171 conmon[35325]: debug 2022-01-31T20:29:56.327+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55516 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:01.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:01 smithi171 conmon[35325]: debug 2022-01-31T20:30:01.336+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55626 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:06.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:06 smithi171 conmon[35325]: debug 2022-01-31T20:30:06.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55735 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:08 smithi171 conmon[35325]: debug 2022-01-31T20:30:08.523+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:30:11.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:11 smithi171 conmon[35325]: debug 2022-01-31T20:30:11.350+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55845 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:16.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:16 smithi171 conmon[35325]: debug 2022-01-31T20:30:16.358+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 55954 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:21.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:21 smithi171 conmon[35325]: debug 2022-01-31T20:30:21.365+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56064 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:23 smithi171 conmon[35325]: debug 2022-01-31T20:30:23.523+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:30:26.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:26 smithi171 conmon[35325]: debug 2022-01-31T20:30:26.374+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56173 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:31.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:31 smithi171 conmon[35325]: debug 2022-01-31T20:30:31.382+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56283 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:36.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:36 smithi171 conmon[35325]: debug 2022-01-31T20:30:36.389+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56392 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:38.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:38 smithi171 conmon[35325]: debug 2022-01-31T20:30:38.524+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:30:41.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:41 smithi171 conmon[35325]: debug 2022-01-31T20:30:41.397+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56502 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:30:46.454+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:30:46.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:46 smithi171 conmon[35325]: debug 2022-01-31T20:30:46.404+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56611 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:51.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:51 smithi171 conmon[35325]: debug 2022-01-31T20:30:51.412+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56721 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:30:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:53 smithi171 conmon[35325]: debug 2022-01-31T20:30:53.525+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:30:56.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:30:56 smithi171 conmon[35325]: debug 2022-01-31T20:30:56.420+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56830 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:01.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:01 smithi171 conmon[35325]: debug 2022-01-31T20:31:01.427+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 56940 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:06.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:06 smithi171 conmon[35325]: debug 2022-01-31T20:31:06.435+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57049 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:08 smithi171 conmon[35325]: debug 2022-01-31T20:31:08.525+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:31:11.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:11 smithi171 conmon[35325]: debug 2022-01-31T20:31:11.444+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57159 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:16.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:16 smithi171 conmon[35325]: debug 2022-01-31T20:31:16.452+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57273 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:21.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:21 smithi171 conmon[35325]: debug 2022-01-31T20:31:21.460+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57383 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:23 smithi171 conmon[35325]: debug 2022-01-31T20:31:23.526+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:31:26.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:26 smithi171 conmon[35325]: debug 2022-01-31T20:31:26.467+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57492 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:31.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:31 smithi171 conmon[35325]: debug 2022-01-31T20:31:31.475+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57602 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:36.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:36 smithi171 conmon[35325]: debug 2022-01-31T20:31:36.484+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57711 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:38 smithi171 conmon[35325]: debug 2022-01-31T20:31:38.527+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:31:41.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:41 smithi171 conmon[35325]: debug 2022-01-31T20:31:41.491+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57821 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:46.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:46 smithi171 conmon[35325]: debug 2022-01-31T20:31:46.498+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 57930 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:51.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:51 smithi171 conmon[35325]: debug 2022-01-31T20:31:51.506+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58040 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:31:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:53 smithi171 conmon[35325]: debug 2022-01-31T20:31:53.527+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:31:56.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:31:56 smithi171 conmon[35325]: debug 2022-01-31T20:31:56.515+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58150 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:01.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:01 smithi171 conmon[35325]: debug 2022-01-31T20:32:01.523+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58260 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:06.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:06 smithi171 conmon[35325]: debug 2022-01-31T20:32:06.534+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58369 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:08 smithi171 conmon[35325]: debug 2022-01-31T20:32:08.528+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:32:11.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:11 smithi171 conmon[35325]: debug 2022-01-31T20:32:11.542+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58479 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:16.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:16 smithi171 conmon[35325]: debug 2022-01-31T20:32:16.550+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58588 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:21.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:21 smithi171 conmon[35325]: debug 2022-01-31T20:32:21.558+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58698 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:23 smithi171 conmon[35325]: debug 2022-01-31T20:32:23.529+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:32:26.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:26 smithi171 conmon[35325]: debug 2022-01-31T20:32:26.566+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58807 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:31.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:31 smithi171 conmon[35325]: debug 2022-01-31T20:32:31.573+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 58918 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:36.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:36 smithi171 conmon[35325]: debug 2022-01-31T20:32:36.582+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59027 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:38 smithi171 conmon[35325]: debug 2022-01-31T20:32:38.529+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:32:41.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:41 smithi171 conmon[35325]: debug 2022-01-31T20:32:41.591+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59137 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:46.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:46 smithi171 conmon[35325]: debug 2022-01-31T20:32:46.599+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59246 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:51.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:51 smithi171 conmon[35325]: debug 2022-01-31T20:32:51.607+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59357 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:32:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:53 smithi171 conmon[35325]: debug 2022-01-31T20:32:53.530+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:32:57.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:32:56 smithi171 conmon[35325]: debug 2022-01-31T20:32:56.616+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59466 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:02.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:01 smithi171 conmon[35325]: debug 2022-01-31T20:33:01.624+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59578 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:07.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:06 smithi171 conmon[35325]: debug 2022-01-31T20:33:06.648+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59687 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:08 smithi171 conmon[35325]: debug 2022-01-31T20:33:08.531+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:33:12.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:11 smithi171 conmon[35325]: debug 2022-01-31T20:33:11.657+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59796 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:17.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:16 smithi171 conmon[35325]: debug 2022-01-31T20:33:16.664+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 59901 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:22.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:21 smithi171 conmon[35325]: debug 2022-01-31T20:33:21.672+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60012 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:23 smithi171 conmon[35325]: debug 2022-01-31T20:33:23.531+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:33:27.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:26 smithi171 conmon[35325]: debug 2022-01-31T20:33:26.681+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60121 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:32.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:31 smithi171 conmon[35325]: debug 2022-01-31T20:33:31.690+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60230 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:36.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:36 smithi171 conmon[35325]: debug 2022-01-31T20:33:36.698+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60340 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:38.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:38 smithi171 conmon[35325]: debug 2022-01-31T20:33:38.532+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:33:42.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:41 smithi171 conmon[35325]: debug 2022-01-31T20:33:41.706+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60449 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:47.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:46 smithi171 conmon[35325]: debug 2022-01-31T20:33:46.714+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60559 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:52.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:51 smithi171 conmon[35325]: debug 2022-01-31T20:33:51.722+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60668 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:33:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:53 smithi171 conmon[35325]: debug 2022-01-31T20:33:53.533+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:33:57.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:33:56 smithi171 conmon[35325]: debug 2022-01-31T20:33:56.730+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60778 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:02.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:01 smithi171 conmon[35325]: debug 2022-01-31T20:34:01.738+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:07.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:06 smithi171 conmon[35325]: debug 2022-01-31T20:34:06.747+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 60997 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:08 smithi171 conmon[35325]: debug 2022-01-31T20:34:08.533+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:34:12.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:11 smithi171 conmon[35325]: debug 2022-01-31T20:34:11.756+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61107 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:17.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:16 smithi171 conmon[35325]: debug 2022-01-31T20:34:16.765+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61221 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:22.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:21 smithi171 conmon[35325]: debug 2022-01-31T20:34:21.773+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61330 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:23 smithi171 conmon[35325]: debug 2022-01-31T20:34:23.534+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:34:27.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:26 smithi171 conmon[35325]: debug 2022-01-31T20:34:26.781+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61440 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:32.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:31 smithi171 conmon[35325]: debug 2022-01-31T20:34:31.790+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:37.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:36 smithi171 conmon[35325]: debug 2022-01-31T20:34:36.799+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61656 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:38.808 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:38 smithi171 conmon[35325]: debug 2022-01-31T20:34:38.535+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:34:42.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:41 smithi171 conmon[35325]: debug 2022-01-31T20:34:41.807+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61765 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:47.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:46 smithi171 conmon[35325]: debug 2022-01-31T20:34:46.814+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61876 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:52.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:51 smithi171 conmon[35325]: debug 2022-01-31T20:34:51.822+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 61984 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:34:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:53 smithi171 conmon[35325]: debug 2022-01-31T20:34:53.535+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:34:57.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:34:56 smithi171 conmon[35325]: debug 2022-01-31T20:34:56.831+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62095 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:02.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:01 smithi171 conmon[35325]: debug 2022-01-31T20:35:01.840+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62203 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:07.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:06 smithi171 conmon[35325]: debug 2022-01-31T20:35:06.848+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62314 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:08 smithi171 conmon[35325]: debug 2022-01-31T20:35:08.536+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:35:12.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:11 smithi171 conmon[35325]: debug 2022-01-31T20:35:11.850+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62423 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:17.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:16 smithi171 conmon[35325]: debug 2022-01-31T20:35:16.864+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62534 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:22.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:21 smithi171 conmon[35325]: debug 2022-01-31T20:35:21.873+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62642 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:23 smithi171 conmon[35325]: debug 2022-01-31T20:35:23.537+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:35:27.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:26 smithi171 conmon[35325]: debug 2022-01-31T20:35:26.881+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62753 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:32.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:31 smithi171 conmon[35325]: debug 2022-01-31T20:35:31.889+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62861 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:37.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:36 smithi171 conmon[35325]: debug 2022-01-31T20:35:36.897+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 62972 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:38 smithi171 conmon[35325]: debug 2022-01-31T20:35:38.537+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:35:42.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:41 smithi171 conmon[35325]: debug 2022-01-31T20:35:41.907+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63080 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:46.453 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:35:46.454+0000 7f12a7b8e700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:35:46.454 INFO:teuthology.orchestra.run.smithi167.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-01-31T20:35:46.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph orch ps' 2022-01-31T20:35:47.327 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:46 smithi171 conmon[35325]: debug 2022-01-31T20:35:46.916+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63191 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:52.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:51 smithi171 conmon[35325]: debug 2022-01-31T20:35:51.925+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63299 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:35:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:53 smithi171 conmon[35325]: debug 2022-01-31T20:35:53.538+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:35:57.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:35:56 smithi171 conmon[35325]: debug 2022-01-31T20:35:56.934+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63410 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:02.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:01 smithi171 conmon[35325]: debug 2022-01-31T20:36:01.943+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63518 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:07.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:06 smithi171 conmon[35325]: debug 2022-01-31T20:36:06.952+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63629 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:08 smithi171 conmon[35325]: debug 2022-01-31T20:36:08.539+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:36:12.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:11 smithi171 conmon[35325]: debug 2022-01-31T20:36:11.970+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63737 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:17.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:16 smithi171 conmon[35325]: debug 2022-01-31T20:36:16.979+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63848 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:22.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:21 smithi171 conmon[35325]: debug 2022-01-31T20:36:21.986+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 63956 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:23 smithi171 conmon[35325]: debug 2022-01-31T20:36:23.539+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:36:27.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:26 smithi171 conmon[35325]: debug 2022-01-31T20:36:26.995+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64067 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:32.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:32 smithi171 conmon[35325]: debug 2022-01-31T20:36:32.004+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64175 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:37.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:37 smithi171 conmon[35325]: debug 2022-01-31T20:36:37.013+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64286 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:38 smithi171 conmon[35325]: debug 2022-01-31T20:36:38.540+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:36:42.304 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:42 smithi171 conmon[35325]: debug 2022-01-31T20:36:42.022+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64394 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:47.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:47 smithi171 conmon[35325]: debug 2022-01-31T20:36:47.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64505 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:52.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:52 smithi171 conmon[35325]: debug 2022-01-31T20:36:52.040+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64614 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:36:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:53 smithi171 conmon[35325]: debug 2022-01-31T20:36:53.541+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:36:57.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:36:57 smithi171 conmon[35325]: debug 2022-01-31T20:36:57.049+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64724 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:02.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:02 smithi171 conmon[35325]: debug 2022-01-31T20:37:02.058+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64829 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:07.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:07 smithi171 conmon[35325]: debug 2022-01-31T20:37:07.066+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 64939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:08 smithi171 conmon[35325]: debug 2022-01-31T20:37:08.541+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:37:12.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:12 smithi171 conmon[35325]: debug 2022-01-31T20:37:12.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:12 smithi171 conmon[35325]: 2022-01-31T20:37:12.076+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65048 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:17.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:17 smithi171 conmon[35325]: debug 2022-01-31T20:37:17.086+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65161 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:22.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:22 smithi171 conmon[35325]: debug 2022-01-31T20:37:22.095+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65270 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:23 smithi171 conmon[35325]: debug 2022-01-31T20:37:23.542+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:37:27.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:27 smithi171 conmon[35325]: debug 2022-01-31T20:37:27.104+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65380 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:32.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:32 smithi171 conmon[35325]: debug 2022-01-31T20:37:32.114+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65489 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:37.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:37 smithi171 conmon[35325]: debug 2022-01-31T20:37:37.123+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:38 smithi171 conmon[35325]: debug 2022-01-31T20:37:38.543+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:37:42.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:42 smithi171 conmon[35325]: debug 2022-01-31T20:37:42.135+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65704 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:47.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:47 smithi171 conmon[35325]: debug 2022-01-31T20:37:47.145+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65814 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:52.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:52 smithi171 conmon[35325]: debug 2022-01-31T20:37:52.153+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 65923 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:37:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:53 smithi171 conmon[35325]: debug 2022-01-31T20:37:53.543+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:37:57.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:37:57 smithi171 conmon[35325]: debug 2022-01-31T20:37:57.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66033 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:02.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:02 smithi171 conmon[35325]: debug 2022-01-31T20:38:02.171+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66142 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:07.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:07 smithi171 conmon[35325]: debug 2022-01-31T20:38:07.181+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66252 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:08 smithi171 conmon[35325]: debug 2022-01-31T20:38:08.544+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:38:12.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:12 smithi171 conmon[35325]: debug 2022-01-31T20:38:12.207+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66361 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:17.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:17 smithi171 conmon[35325]: debug 2022-01-31T20:38:17.216+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66471 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:22.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:22 smithi171 conmon[35325]: debug 2022-01-31T20:38:22.225+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66580 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:23 smithi171 conmon[35325]: debug 2022-01-31T20:38:23.545+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:38:27.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:27 smithi171 conmon[35325]: debug 2022-01-31T20:38:27.235+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66689 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:32.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:32 smithi171 conmon[35325]: debug 2022-01-31T20:38:32.244+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66799 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:37.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:37 smithi171 conmon[35325]: debug 2022-01-31T20:38:37.253+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 66908 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:38 smithi171 conmon[35325]: debug 2022-01-31T20:38:38.545+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:38:42.536 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:42 smithi171 conmon[35325]: debug 2022-01-31T20:38:42.261+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67018 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:47.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:47 smithi171 conmon[35325]: debug 2022-01-31T20:38:47.270+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67127 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:52.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:52 smithi171 conmon[35325]: debug 2022-01-31T20:38:52.280+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67237 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:38:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:53 smithi171 conmon[35325]: debug 2022-01-31T20:38:53.547+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:38:57.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:38:57 smithi171 conmon[35325]: debug 2022-01-31T20:38:57.289+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67345 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:02.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:02 smithi171 conmon[35325]: debug 2022-01-31T20:39:02.298+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67456 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:07.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:07 smithi171 conmon[35325]: debug 2022-01-31T20:39:07.308+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67564 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:08 smithi171 conmon[35325]: debug 2022-01-31T20:39:08.548+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:39:12.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:12 smithi171 conmon[35325]: debug 2022-01-31T20:39:12.317+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67675 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:17.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:17 smithi171 conmon[35325]: debug 2022-01-31T20:39:17.328+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67783 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:22.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:22 smithi171 conmon[35325]: debug 2022-01-31T20:39:22.338+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 67894 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:23 smithi171 conmon[35325]: debug 2022-01-31T20:39:23.549+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:39:27.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:27 smithi171 conmon[35325]: debug 2022-01-31T20:39:27.346+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68002 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:32.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:32 smithi171 conmon[35325]: debug 2022-01-31T20:39:32.355+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68113 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:37.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:37 smithi171 conmon[35325]: debug 2022-01-31T20:39:37.365+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68221 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:38 smithi171 conmon[35325]: debug 2022-01-31T20:39:38.549+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:39:42.656 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:42 smithi171 conmon[35325]: debug 2022-01-31T20:39:42.374+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68332 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:47.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:47 smithi171 conmon[35325]: debug 2022-01-31T20:39:47.383+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68440 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:52.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:52 smithi171 conmon[35325]: debug 2022-01-31T20:39:52.393+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68551 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:39:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:53 smithi171 conmon[35325]: debug 2022-01-31T20:39:53.550+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:39:57.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:39:57 smithi171 conmon[35325]: debug 2022-01-31T20:39:57.402+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68659 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:02.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:02 smithi171 conmon[35325]: debug 2022-01-31T20:40:02.413+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68771 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:07.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:07 smithi171 conmon[35325]: debug 2022-01-31T20:40:07.423+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68875 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:08 smithi171 conmon[35325]: debug 2022-01-31T20:40:08.551+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:40:12.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:12 smithi171 conmon[35325]: debug 2022-01-31T20:40:12.433+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 68986 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:17.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:17 smithi171 conmon[35325]: debug 2022-01-31T20:40:17.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:17 smithi171 conmon[35325]: 2022-01-31T20:40:17.442+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69100 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:22.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:22 smithi171 conmon[35325]: debug 2022-01-31T20:40:22.451+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69211 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:23 smithi171 conmon[35325]: debug 2022-01-31T20:40:23.551+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:40:27.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:27 smithi171 conmon[35325]: debug 2022-01-31T20:40:27.462+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69319 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:32.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:32 smithi171 conmon[35325]: debug 2022-01-31T20:40:32.471+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69430 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:37.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:37 smithi171 conmon[35325]: debug 2022-01-31T20:40:37.481+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69538 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:38 smithi171 conmon[35325]: debug 2022-01-31T20:40:38.552+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:40:42.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:42 smithi171 conmon[35325]: debug 2022-01-31T20:40:42.490+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69649 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:47.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:47 smithi171 conmon[35325]: debug 2022-01-31T20:40:47.499+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69757 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:48.218 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:40:48.216+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:40:52.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:52 smithi171 conmon[35325]: debug 2022-01-31T20:40:52.509+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69868 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:40:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:53 smithi171 conmon[35325]: debug 2022-01-31T20:40:53.553+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:40:57.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:40:57 smithi171 conmon[35325]: debug 2022-01-31T20:40:57.519+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 69973 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:02.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:02 smithi171 conmon[35325]: debug 2022-01-31T20:41:02.527+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70084 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:07.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:07 smithi171 conmon[35325]: debug 2022-01-31T20:41:07.537+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:08 smithi171 conmon[35325]: debug 2022-01-31T20:41:08.553+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:41:12.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:12 smithi171 conmon[35325]: debug 2022-01-31T20:41:12.547+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70303 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:17.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:17 smithi171 conmon[35325]: debug 2022-01-31T20:41:17.558+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70411 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:22.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:22 smithi171 conmon[35325]: debug 2022-01-31T20:41:22.568+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70522 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:23 smithi171 conmon[35325]: debug 2022-01-31T20:41:23.555+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:41:27.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:27 smithi171 conmon[35325]: debug 2022-01-31T20:41:27.578+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70630 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:32.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:32 smithi171 conmon[35325]: debug 2022-01-31T20:41:32.588+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70741 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:37.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:37 smithi171 conmon[35325]: debug 2022-01-31T20:41:37.598+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70849 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:38 smithi171 conmon[35325]: debug 2022-01-31T20:41:38.556+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:41:43.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:42 smithi171 conmon[35325]: debug 2022-01-31T20:41:42.607+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 70960 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:48.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:47 smithi171 conmon[35325]: debug 2022-01-31T20:41:47.615+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71068 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:53.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:52 smithi171 conmon[35325]: debug 2022-01-31T20:41:52.625+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71179 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:41:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:53 smithi171 conmon[35325]: debug 2022-01-31T20:41:53.556+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:41:58.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:41:57 smithi171 conmon[35325]: debug 2022-01-31T20:41:57.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71287 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:03.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:02 smithi171 conmon[35325]: debug 2022-01-31T20:42:02.646+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71399 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:08.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:07 smithi171 conmon[35325]: debug 2022-01-31T20:42:07.656+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71507 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:08 smithi171 conmon[35325]: debug 2022-01-31T20:42:08.557+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:42:13.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:12 smithi171 conmon[35325]: debug 2022-01-31T20:42:12.666+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71619 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:18.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:17 smithi171 conmon[35325]: debug 2022-01-31T20:42:17.692+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71727 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:23.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:22 smithi171 conmon[35325]: debug 2022-01-31T20:42:22.702+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71838 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:23 smithi171 conmon[35325]: debug 2022-01-31T20:42:23.558+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:42:28.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:27 smithi171 conmon[35325]: debug 2022-01-31T20:42:27.712+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 71946 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:33.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:32 smithi171 conmon[35325]: debug 2022-01-31T20:42:32.722+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72057 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:38.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:37 smithi171 conmon[35325]: debug 2022-01-31T20:42:37.731+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72165 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:38 smithi171 conmon[35325]: debug 2022-01-31T20:42:38.558+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:42:43.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:42 smithi171 conmon[35325]: debug 2022-01-31T20:42:42.740+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72276 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:48.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:47 smithi171 conmon[35325]: debug 2022-01-31T20:42:47.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72384 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:53.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:52 smithi171 conmon[35325]: debug 2022-01-31T20:42:52.762+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72495 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:42:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:53 smithi171 conmon[35325]: debug 2022-01-31T20:42:53.559+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:42:58.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:42:57 smithi171 conmon[35325]: debug 2022-01-31T20:42:57.771+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72604 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:03.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:02 smithi171 conmon[35325]: debug 2022-01-31T20:43:02.781+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72714 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:08.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:07 smithi171 conmon[35325]: debug 2022-01-31T20:43:07.791+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72824 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:08 smithi171 conmon[35325]: debug 2022-01-31T20:43:08.560+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:43:13.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:12 smithi171 conmon[35325]: debug 2022-01-31T20:43:12.801+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 72934 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:18.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:17 smithi171 conmon[35325]: debug 2022-01-31T20:43:17.812+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73051 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:23.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:22 smithi171 conmon[35325]: debug 2022-01-31T20:43:22.821+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73161 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:23 smithi171 conmon[35325]: debug 2022-01-31T20:43:23.560+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:43:28.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:27 smithi171 conmon[35325]: debug 2022-01-31T20:43:27.831+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73271 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:33.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:32 smithi171 conmon[35325]: debug 2022-01-31T20:43:32.841+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73380 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:38.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:37 smithi171 conmon[35325]: debug 2022-01-31T20:43:37.851+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73490 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:38 smithi171 conmon[35325]: debug 2022-01-31T20:43:38.561+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:43:43.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:42 smithi171 conmon[35325]: debug 2022-01-31T20:43:42.861+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:48.354 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:47 smithi171 conmon[35325]: debug 2022-01-31T20:43:47.871+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73709 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:53.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:52 smithi171 conmon[35325]: debug 2022-01-31T20:43:52.880+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73817 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:43:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:53 smithi171 conmon[35325]: debug 2022-01-31T20:43:53.562+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:43:58.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:43:57 smithi171 conmon[35325]: debug 2022-01-31T20:43:57.891+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 73928 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:03.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:02 smithi171 conmon[35325]: debug 2022-01-31T20:44:02.901+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74036 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:08.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:07 smithi171 conmon[35325]: debug 2022-01-31T20:44:07.910+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74147 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:08 smithi171 conmon[35325]: debug 2022-01-31T20:44:08.562+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:44:13.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:12 smithi171 conmon[35325]: debug 2022-01-31T20:44:12.920+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74255 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:18.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:17 smithi171 conmon[35325]: debug 2022-01-31T20:44:17.931+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74366 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:18.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:17 smithi171 conmon[35325]: 2022-01-31T20:44:23.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:22 smithi171 conmon[35325]: debug 2022-01-31T20:44:22.942+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74475 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:23 smithi171 conmon[35325]: debug 2022-01-31T20:44:23.563+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:44:28.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:27 smithi171 conmon[35325]: debug 2022-01-31T20:44:27.952+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74586 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:33.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:32 smithi171 conmon[35325]: debug 2022-01-31T20:44:32.962+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74695 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:38.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:37 smithi171 conmon[35325]: debug 2022-01-31T20:44:37.972+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74805 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:38 smithi171 conmon[35325]: debug 2022-01-31T20:44:38.564+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:44:41.263 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-01-31T20:44:41.265 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-01-31T20:44:43.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:42 smithi171 conmon[35325]: debug 2022-01-31T20:44:42.983+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 74914 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:48.088 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:47 smithi171 conmon[35325]: debug 2022-01-31T20:44:47.993+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75024 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:53.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:53 smithi171 conmon[35325]: debug 2022-01-31T20:44:53.003+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75133 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:44:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:53 smithi171 conmon[35325]: debug 2022-01-31T20:44:53.564+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:44:56.225 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-01-31T20:44:56.226 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-01-31T20:44:58.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:44:58 smithi171 conmon[35325]: debug 2022-01-31T20:44:58.012+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75243 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:03.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:03 smithi171 conmon[35325]: debug 2022-01-31T20:45:03.023+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75352 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:06 smithi167 conmon[49112]: debug 2022-01-31T20:45:06.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:06.565378+0000) 2022-01-31T20:45:07.276 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:07 smithi167 conmon[54076]: debug 2022-01-31T20:45:07.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:07.236216+0000) 2022-01-31T20:45:07.277 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:06 smithi167 conmon[60316]: debug 2022-01-31T20:45:06.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:06.987244+0000) 2022-01-31T20:45:07.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:07 smithi171 conmon[51620]: debug 2022-01-31T20:45:07.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:07.342823+0000) 2022-01-31T20:45:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:07 smithi171 conmon[46715]: debug 2022-01-31T20:45:07.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:07.263473+0000) 2022-01-31T20:45:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:07 smithi171 conmon[41853]: debug 2022-01-31T20:45:07.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:07.471212+0000) 2022-01-31T20:45:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:07 smithi167 conmon[49112]: debug 2022-01-31T20:45:07.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:07.565581+0000) 2022-01-31T20:45:08.277 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:08 smithi167 conmon[49112]: debug 2022-01-31T20:45:08.027+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.029179+0000) 2022-01-31T20:45:08.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:08 smithi167 conmon[54076]: debug 2022-01-31T20:45:08.027+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.028725+0000) 2022-01-31T20:45:08.278 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:08 smithi167 conmon[54076]: debug 2022-01-31T20:45:08.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.236455+0000) 2022-01-31T20:45:08.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:07 smithi167 conmon[60316]: debug 2022-01-31T20:45:07.985+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:07.987432+0000) 2022-01-31T20:45:08.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:08 smithi167 conmon[60316]: debug 2022-01-31T20:45:08.027+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.028942+0000) 2022-01-31T20:45:08.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[35325]: debug 2022-01-31T20:45:08.034+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75463 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:08.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[41853]: debug 2022-01-31T20:45:08.026+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.027416+0000) 2022-01-31T20:45:08.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[51620]: debug 2022-01-31T20:45:08.027+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.027905+0000) 2022-01-31T20:45:08.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[46715]: debug 2022-01-31T20:45:08.028+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.029014+0000) 2022-01-31T20:45:08.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[46715]: debug 2022-01-31T20:45:08.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.263667+0000) 2022-01-31T20:45:08.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[41853]: debug 2022-01-31T20:45:08.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.471392+0000) 2022-01-31T20:45:08.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[51620]: debug 2022-01-31T20:45:08.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.342991+0000) 2022-01-31T20:45:08.591 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[35325]: debug 2022-01-31T20:45:08.565+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:45:08.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:08 smithi171 conmon[35325]: 2022-01-31T20:45:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:08 smithi167 conmon[49112]: debug 2022-01-31T20:45:08.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.565708+0000) 2022-01-31T20:45:09.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:09 smithi167 conmon[54076]: debug 2022-01-31T20:45:09.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:09.236651+0000) 2022-01-31T20:45:09.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:08 smithi167 conmon[60316]: debug 2022-01-31T20:45:08.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:08.987622+0000) 2022-01-31T20:45:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:09 smithi171 conmon[46715]: debug 2022-01-31T20:45:09.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:09.263845+0000) 2022-01-31T20:45:09.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:09 smithi171 conmon[51620]: debug 2022-01-31T20:45:09.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:09.343153+0000) 2022-01-31T20:45:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:09 smithi171 conmon[41853]: debug 2022-01-31T20:45:09.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:09.471547+0000) 2022-01-31T20:45:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:09 smithi167 conmon[49112]: debug 2022-01-31T20:45:09.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:09.565825+0000) 2022-01-31T20:45:10.277 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:09 smithi167 conmon[60316]: debug 2022-01-31T20:45:09.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:09.987804+0000) 2022-01-31T20:45:10.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:10 smithi167 conmon[54076]: debug 2022-01-31T20:45:10.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:10.236783+0000) 2022-01-31T20:45:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:10 smithi171 conmon[46715]: debug 2022-01-31T20:45:10.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:10.264032+0000) 2022-01-31T20:45:10.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:10 smithi171 conmon[51620]: debug 2022-01-31T20:45:10.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:10.343371+0000) 2022-01-31T20:45:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:10 smithi171 conmon[41853]: debug 2022-01-31T20:45:10.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:10.471783+0000) 2022-01-31T20:45:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:10 smithi167 conmon[49112]: debug 2022-01-31T20:45:10.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:10.565981+0000) 2022-01-31T20:45:11.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:11 smithi167 conmon[54076]: debug 2022-01-31T20:45:11.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:11.236882+0000) 2022-01-31T20:45:11.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:10 smithi167 conmon[60316]: debug 2022-01-31T20:45:10.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:10.987988+0000) 2022-01-31T20:45:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:11 smithi171 conmon[46715]: debug 2022-01-31T20:45:11.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:11.264179+0000) 2022-01-31T20:45:11.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:11 smithi171 conmon[51620]: debug 2022-01-31T20:45:11.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:11.343552+0000) 2022-01-31T20:45:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:11 smithi171 conmon[41853]: debug 2022-01-31T20:45:11.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:11.471967+0000) 2022-01-31T20:45:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:11 smithi167 conmon[49112]: debug 2022-01-31T20:45:11.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:11.566156+0000) 2022-01-31T20:45:12.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:12 smithi167 conmon[54076]: debug 2022-01-31T20:45:12.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:12.236982+0000) 2022-01-31T20:45:12.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:11 smithi167 conmon[60316]: debug 2022-01-31T20:45:11.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:11.988204+0000) 2022-01-31T20:45:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:12 smithi171 conmon[46715]: debug 2022-01-31T20:45:12.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:12.264377+0000) 2022-01-31T20:45:12.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:12 smithi171 conmon[51620]: debug 2022-01-31T20:45:12.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:12.343740+0000) 2022-01-31T20:45:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:12 smithi171 conmon[41853]: debug 2022-01-31T20:45:12.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:12.472162+0000) 2022-01-31T20:45:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:12 smithi167 conmon[49112]: debug 2022-01-31T20:45:12.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:12.566302+0000) 2022-01-31T20:45:13.278 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:13 smithi167 conmon[49112]: debug 2022-01-31T20:45:13.036+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.038461+0000) 2022-01-31T20:45:13.278 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:13 smithi167 conmon[54076]: debug 2022-01-31T20:45:13.037+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.038508+0000) 2022-01-31T20:45:13.278 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:13 smithi167 conmon[54076]: debug 2022-01-31T20:45:13.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.237114+0000) 2022-01-31T20:45:13.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:12 smithi167 conmon[60316]: debug 2022-01-31T20:45:12.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:12.988413+0000) 2022-01-31T20:45:13.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:13 smithi167 conmon[60316]: debug 2022-01-31T20:45:13.037+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.038625+0000) 2022-01-31T20:45:13.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[35325]: debug 2022-01-31T20:45:13.044+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:13.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[41853]: debug 2022-01-31T20:45:13.036+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.037312+0000) 2022-01-31T20:45:13.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[46715]: debug 2022-01-31T20:45:13.036+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.037736+0000) 2022-01-31T20:45:13.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[46715]: debug 2022-01-31T20:45:13.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.264520+0000) 2022-01-31T20:45:13.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[51620]: debug 2022-01-31T20:45:13.038+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.039080+0000) 2022-01-31T20:45:13.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[41853]: debug 2022-01-31T20:45:13.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.472278+0000) 2022-01-31T20:45:13.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:13 smithi171 conmon[51620]: debug 2022-01-31T20:45:13.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.343941+0000) 2022-01-31T20:45:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:13 smithi167 conmon[49112]: debug 2022-01-31T20:45:13.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.566540+0000) 2022-01-31T20:45:14.278 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:14 smithi167 conmon[54076]: debug 2022-01-31T20:45:14.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:14.237284+0000) 2022-01-31T20:45:14.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:13 smithi167 conmon[60316]: debug 2022-01-31T20:45:13.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:13.988536+0000) 2022-01-31T20:45:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:14 smithi171 conmon[46715]: debug 2022-01-31T20:45:14.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:14.264688+0000) 2022-01-31T20:45:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:14 smithi171 conmon[51620]: debug 2022-01-31T20:45:14.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:14.344069+0000) 2022-01-31T20:45:14.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:14 smithi171 conmon[41853]: debug 2022-01-31T20:45:14.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:14.472451+0000) 2022-01-31T20:45:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:14 smithi167 conmon[49112]: debug 2022-01-31T20:45:14.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:14.566590+0000) 2022-01-31T20:45:15.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:14 smithi167 conmon[60316]: debug 2022-01-31T20:45:14.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:14.988689+0000) 2022-01-31T20:45:15.278 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:15 smithi167 conmon[54076]: debug 2022-01-31T20:45:15.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:15.237490+0000) 2022-01-31T20:45:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:15 smithi171 conmon[46715]: debug 2022-01-31T20:45:15.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:15.264837+0000) 2022-01-31T20:45:15.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:15 smithi171 conmon[51620]: debug 2022-01-31T20:45:15.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:15.344217+0000) 2022-01-31T20:45:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:15 smithi171 conmon[41853]: debug 2022-01-31T20:45:15.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:15.472673+0000) 2022-01-31T20:45:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:15 smithi167 conmon[49112]: debug 2022-01-31T20:45:15.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:15.566745+0000) 2022-01-31T20:45:16.278 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:16 smithi167 conmon[54076]: debug 2022-01-31T20:45:16.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:16.237637+0000) 2022-01-31T20:45:16.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:15 smithi167 conmon[60316]: debug 2022-01-31T20:45:15.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:15.988852+0000) 2022-01-31T20:45:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:16 smithi171 conmon[46715]: debug 2022-01-31T20:45:16.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:16.264988+0000) 2022-01-31T20:45:16.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:16 smithi171 conmon[51620]: debug 2022-01-31T20:45:16.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:16.344334+0000) 2022-01-31T20:45:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:16 smithi171 conmon[41853]: debug 2022-01-31T20:45:16.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:16.472852+0000) 2022-01-31T20:45:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:16 smithi167 conmon[49112]: debug 2022-01-31T20:45:16.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:16.566960+0000) 2022-01-31T20:45:17.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:17 smithi167 conmon[54076]: debug 2022-01-31T20:45:17.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:17.237819+0000) 2022-01-31T20:45:17.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:16 smithi167 conmon[60316]: debug 2022-01-31T20:45:16.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:16.989015+0000) 2022-01-31T20:45:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:17 smithi171 conmon[46715]: debug 2022-01-31T20:45:17.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:17.265129+0000) 2022-01-31T20:45:17.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:17 smithi171 conmon[51620]: debug 2022-01-31T20:45:17.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:17.344512+0000) 2022-01-31T20:45:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:17 smithi171 conmon[41853]: debug 2022-01-31T20:45:17.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:17.473021+0000) 2022-01-31T20:45:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:17 smithi167 conmon[49112]: debug 2022-01-31T20:45:17.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:17.567173+0000) 2022-01-31T20:45:18.278 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:18 smithi167 conmon[49112]: debug 2022-01-31T20:45:18.047+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.047791+0000) 2022-01-31T20:45:18.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:17 smithi167 conmon[60316]: debug 2022-01-31T20:45:17.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:17.989163+0000) 2022-01-31T20:45:18.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:18 smithi167 conmon[60316]: debug 2022-01-31T20:45:18.048+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.049244+0000) 2022-01-31T20:45:18.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:18 smithi167 conmon[54076]: debug 2022-01-31T20:45:18.048+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.048967+0000) 2022-01-31T20:45:18.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:18 smithi167 conmon[54076]: debug 2022-01-31T20:45:18.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.238008+0000) 2022-01-31T20:45:18.343 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[35325]: debug 2022-01-31T20:45:18.054+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75682 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:18.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[41853]: debug 2022-01-31T20:45:18.047+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.047877+0000) 2022-01-31T20:45:18.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[51620]: debug 2022-01-31T20:45:18.047+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.048284+0000) 2022-01-31T20:45:18.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[46715]: debug 2022-01-31T20:45:18.047+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.047963+0000) 2022-01-31T20:45:18.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[46715]: debug 2022-01-31T20:45:18.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.265327+0000) 2022-01-31T20:45:18.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[41853]: debug 2022-01-31T20:45:18.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.473183+0000) 2022-01-31T20:45:18.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:18 smithi171 conmon[51620]: debug 2022-01-31T20:45:18.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.344732+0000) 2022-01-31T20:45:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:18 smithi167 conmon[49112]: debug 2022-01-31T20:45:18.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.567391+0000) 2022-01-31T20:45:19.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:19 smithi167 conmon[54076]: debug 2022-01-31T20:45:19.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:19.238127+0000) 2022-01-31T20:45:19.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:18 smithi167 conmon[60316]: debug 2022-01-31T20:45:18.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:18.989375+0000) 2022-01-31T20:45:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:19 smithi171 conmon[46715]: debug 2022-01-31T20:45:19.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:19.265481+0000) 2022-01-31T20:45:19.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:19 smithi171 conmon[51620]: debug 2022-01-31T20:45:19.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:19.344910+0000) 2022-01-31T20:45:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:19 smithi171 conmon[41853]: debug 2022-01-31T20:45:19.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:19.473329+0000) 2022-01-31T20:45:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:19 smithi167 conmon[49112]: debug 2022-01-31T20:45:19.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:19.567575+0000) 2022-01-31T20:45:20.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:19 smithi167 conmon[60316]: debug 2022-01-31T20:45:19.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:19.989575+0000) 2022-01-31T20:45:20.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:20 smithi167 conmon[54076]: debug 2022-01-31T20:45:20.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:20.238323+0000) 2022-01-31T20:45:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:20 smithi171 conmon[46715]: debug 2022-01-31T20:45:20.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:20.265644+0000) 2022-01-31T20:45:20.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:20 smithi171 conmon[51620]: debug 2022-01-31T20:45:20.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:20.345075+0000) 2022-01-31T20:45:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:20 smithi171 conmon[41853]: debug 2022-01-31T20:45:20.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:20.473511+0000) 2022-01-31T20:45:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:20 smithi167 conmon[49112]: debug 2022-01-31T20:45:20.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:20.567735+0000) 2022-01-31T20:45:21.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:21 smithi167 conmon[54076]: debug 2022-01-31T20:45:21.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:21.238538+0000) 2022-01-31T20:45:21.280 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:20 smithi167 conmon[60316]: debug 2022-01-31T20:45:20.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:20.989731+0000) 2022-01-31T20:45:21.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:21 smithi171 conmon[51620]: debug 2022-01-31T20:45:21.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:21.345294+0000) 2022-01-31T20:45:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:21 smithi171 conmon[46715]: debug 2022-01-31T20:45:21.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:21.265842+0000) 2022-01-31T20:45:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:21 smithi171 conmon[41853]: debug 2022-01-31T20:45:21.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:21.473700+0000) 2022-01-31T20:45:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:21 smithi167 conmon[49112]: debug 2022-01-31T20:45:21.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:21.567893+0000) 2022-01-31T20:45:22.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:21 smithi167 conmon[60316]: debug 2022-01-31T20:45:21.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:21.989908+0000) 2022-01-31T20:45:22.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:22 smithi167 conmon[54076]: debug 2022-01-31T20:45:22.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:22.238685+0000) 2022-01-31T20:45:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:22 smithi171 conmon[46715]: debug 2022-01-31T20:45:22.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:22.266023+0000) 2022-01-31T20:45:22.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:22 smithi171 conmon[51620]: debug 2022-01-31T20:45:22.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:22.345477+0000) 2022-01-31T20:45:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:22 smithi171 conmon[41853]: debug 2022-01-31T20:45:22.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:22.473882+0000) 2022-01-31T20:45:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:22 smithi167 conmon[49112]: debug 2022-01-31T20:45:22.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:22.568057+0000) 2022-01-31T20:45:23.279 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[49112]: debug 2022-01-31T20:45:23.280 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[49112]: 2022-01-31T20:45:23.064+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.064973+0000) 2022-01-31T20:45:23.280 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[49112]: 2022-01-31T20:45:23.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[54076]: debug 2022-01-31T20:45:23.065+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.065506+0000) 2022-01-31T20:45:23.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[54076]: debug 2022-01-31T20:45:23.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.238793+0000) 2022-01-31T20:45:23.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:22 smithi167 conmon[60316]: debug 2022-01-31T20:45:22.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:22.990073+0000) 2022-01-31T20:45:23.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[60316]: debug 2022-01-31T20:45:23.064+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.065146+0000) 2022-01-31T20:45:23.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[41853]: debug 2022-01-31T20:45:23.056+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.057702+0000) 2022-01-31T20:45:23.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[51620]: debug 2022-01-31T20:45:23.057+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.057839+0000) 2022-01-31T20:45:23.345 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[35325]: debug 2022-01-31T20:45:23.065+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75791 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:23.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[46715]: debug 2022-01-31T20:45:23.057+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.058501+0000) 2022-01-31T20:45:23.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[46715]: debug 2022-01-31T20:45:23.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.266213+0000) 2022-01-31T20:45:23.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[41853]: debug 2022-01-31T20:45:23.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.474061+0000) 2022-01-31T20:45:23.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[51620]: debug 2022-01-31T20:45:23.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.345656+0000) 2022-01-31T20:45:23.595 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[35325]: debug 2022-01-31T20:45:23.596 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:23 smithi171 conmon[35325]: 2022-01-31T20:45:23.566+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:45:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[49112]: debug 2022-01-31T20:45:23.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.568252+0000) 2022-01-31T20:45:24.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:24 smithi167 conmon[54076]: debug 2022-01-31T20:45:24.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:24.238916+0000) 2022-01-31T20:45:24.280 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:23 smithi167 conmon[60316]: debug 2022-01-31T20:45:23.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:23.990223+0000) 2022-01-31T20:45:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:24 smithi171 conmon[46715]: debug 2022-01-31T20:45:24.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:24.266376+0000) 2022-01-31T20:45:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:24 smithi171 conmon[51620]: debug 2022-01-31T20:45:24.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:24.345804+0000) 2022-01-31T20:45:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:24 smithi171 conmon[41853]: debug 2022-01-31T20:45:24.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:24.474244+0000) 2022-01-31T20:45:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:24 smithi167 conmon[49112]: debug 2022-01-31T20:45:24.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:24.568409+0000) 2022-01-31T20:45:25.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:25 smithi167 conmon[54076]: debug 2022-01-31T20:45:25.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:25.239039+0000) 2022-01-31T20:45:25.280 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:24 smithi167 conmon[60316]: debug 2022-01-31T20:45:24.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:24.990372+0000) 2022-01-31T20:45:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:25 smithi171 conmon[46715]: debug 2022-01-31T20:45:25.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:25.266581+0000) 2022-01-31T20:45:25.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:25 smithi171 conmon[51620]: debug 2022-01-31T20:45:25.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:25.345985+0000) 2022-01-31T20:45:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:25 smithi171 conmon[41853]: debug 2022-01-31T20:45:25.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:25.474446+0000) 2022-01-31T20:45:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:25 smithi167 conmon[49112]: debug 2022-01-31T20:45:25.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:25.568589+0000) 2022-01-31T20:45:26.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:26 smithi167 conmon[54076]: debug 2022-01-31T20:45:26.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:26.239212+0000) 2022-01-31T20:45:26.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:25 smithi167 conmon[60316]: debug 2022-01-31T20:45:25.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:25.990556+0000) 2022-01-31T20:45:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:26 smithi171 conmon[46715]: debug 2022-01-31T20:45:26.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:26.266790+0000) 2022-01-31T20:45:26.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:26 smithi171 conmon[51620]: debug 2022-01-31T20:45:26.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:26.346169+0000) 2022-01-31T20:45:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:26 smithi171 conmon[41853]: debug 2022-01-31T20:45:26.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:26.474632+0000) 2022-01-31T20:45:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:26 smithi167 conmon[49112]: debug 2022-01-31T20:45:26.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:26.568710+0000) 2022-01-31T20:45:27.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:27 smithi167 conmon[54076]: debug 2022-01-31T20:45:27.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:27.239386+0000) 2022-01-31T20:45:27.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:26 smithi167 conmon[60316]: debug 2022-01-31T20:45:26.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:26.990744+0000) 2022-01-31T20:45:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:27 smithi171 conmon[46715]: debug 2022-01-31T20:45:27.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:27.266944+0000) 2022-01-31T20:45:27.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:27 smithi171 conmon[51620]: debug 2022-01-31T20:45:27.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:27.346327+0000) 2022-01-31T20:45:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:27 smithi171 conmon[41853]: debug 2022-01-31T20:45:27.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:27.474779+0000) 2022-01-31T20:45:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:27 smithi167 conmon[49112]: debug 2022-01-31T20:45:27.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:27.568916+0000) 2022-01-31T20:45:28.280 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:28 smithi167 conmon[49112]: debug 2022-01-31T20:45:28.069+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.069826+0000) 2022-01-31T20:45:28.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:28 smithi167 conmon[54076]: debug 2022-01-31T20:45:28.069+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.070006+0000) 2022-01-31T20:45:28.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:28 smithi167 conmon[54076]: debug 2022-01-31T20:45:28.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.239576+0000) 2022-01-31T20:45:28.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:27 smithi167 conmon[60316]: debug 2022-01-31T20:45:27.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:27.990936+0000) 2022-01-31T20:45:28.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:28 smithi167 conmon[60316]: debug 2022-01-31T20:45:28.068+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.069357+0000) 2022-01-31T20:45:28.345 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[35325]: debug 2022-01-31T20:45:28.075+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 75902 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:28.345 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[41853]: debug 2022-01-31T20:45:28.067+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.068341+0000) 2022-01-31T20:45:28.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[51620]: debug 2022-01-31T20:45:28.067+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.067804+0000) 2022-01-31T20:45:28.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[46715]: debug 2022-01-31T20:45:28.067+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.068615+0000) 2022-01-31T20:45:28.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[46715]: debug 2022-01-31T20:45:28.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.267094+0000) 2022-01-31T20:45:28.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[51620]: debug 2022-01-31T20:45:28.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.346528+0000) 2022-01-31T20:45:28.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:28 smithi171 conmon[41853]: debug 2022-01-31T20:45:28.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.474967+0000) 2022-01-31T20:45:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:28 smithi167 conmon[49112]: debug 2022-01-31T20:45:28.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.569120+0000) 2022-01-31T20:45:29.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:29 smithi167 conmon[54076]: debug 2022-01-31T20:45:29.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:29.239706+0000) 2022-01-31T20:45:29.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:28 smithi167 conmon[60316]: debug 2022-01-31T20:45:28.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:28.991142+0000) 2022-01-31T20:45:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:29 smithi171 conmon[46715]: debug 2022-01-31T20:45:29.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:29.267256+0000) 2022-01-31T20:45:29.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:29 smithi171 conmon[51620]: debug 2022-01-31T20:45:29.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:29.346697+0000) 2022-01-31T20:45:29.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:29 smithi171 conmon[41853]: debug 2022-01-31T20:45:29.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:29.475108+0000) 2022-01-31T20:45:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:29 smithi167 conmon[49112]: debug 2022-01-31T20:45:29.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:29.569278+0000) 2022-01-31T20:45:30.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:30 smithi167 conmon[54076]: debug 2022-01-31T20:45:30.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:30.239874+0000) 2022-01-31T20:45:30.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:29 smithi167 conmon[60316]: debug 2022-01-31T20:45:29.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:29.991357+0000) 2022-01-31T20:45:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:30 smithi171 conmon[46715]: debug 2022-01-31T20:45:30.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:30.267443+0000) 2022-01-31T20:45:30.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:30 smithi171 conmon[51620]: debug 2022-01-31T20:45:30.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:30.346938+0000) 2022-01-31T20:45:30.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:30 smithi171 conmon[41853]: debug 2022-01-31T20:45:30.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:30.475294+0000) 2022-01-31T20:45:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:30 smithi167 conmon[49112]: debug 2022-01-31T20:45:30.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:30.569469+0000) 2022-01-31T20:45:31.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:31 smithi167 conmon[54076]: debug 2022-01-31T20:45:31.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:31.240051+0000) 2022-01-31T20:45:31.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:30 smithi167 conmon[60316]: debug 2022-01-31T20:45:30.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:30.991554+0000) 2022-01-31T20:45:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:31 smithi171 conmon[46715]: debug 2022-01-31T20:45:31.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:31.267595+0000) 2022-01-31T20:45:31.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:31 smithi171 conmon[51620]: debug 2022-01-31T20:45:31.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:31.347083+0000) 2022-01-31T20:45:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:31 smithi171 conmon[41853]: debug 2022-01-31T20:45:31.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:31.475496+0000) 2022-01-31T20:45:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:31 smithi167 conmon[49112]: debug 2022-01-31T20:45:31.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:31.569619+0000) 2022-01-31T20:45:32.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:32 smithi167 conmon[54076]: debug 2022-01-31T20:45:32.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:32.240220+0000) 2022-01-31T20:45:32.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:31 smithi167 conmon[60316]: debug 2022-01-31T20:45:31.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:31.991708+0000) 2022-01-31T20:45:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:32 smithi171 conmon[46715]: debug 2022-01-31T20:45:32.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:32.267698+0000) 2022-01-31T20:45:32.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:32 smithi171 conmon[51620]: debug 2022-01-31T20:45:32.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:32.347274+0000) 2022-01-31T20:45:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:32 smithi171 conmon[41853]: debug 2022-01-31T20:45:32.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:32.475666+0000) 2022-01-31T20:45:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:32 smithi167 conmon[49112]: debug 2022-01-31T20:45:32.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:32.569775+0000) 2022-01-31T20:45:33.281 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:33 smithi167 conmon[49112]: debug 2022-01-31T20:45:33.078+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.078760+0000) 2022-01-31T20:45:33.282 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:33 smithi167 conmon[54076]: debug 2022-01-31T20:45:33.078+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.078867+0000) 2022-01-31T20:45:33.282 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:33 smithi167 conmon[54076]: debug 2022-01-31T20:45:33.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.240394+0000) 2022-01-31T20:45:33.283 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:32 smithi167 conmon[60316]: debug 2022-01-31T20:45:32.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:32.991870+0000) 2022-01-31T20:45:33.283 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:33 smithi167 conmon[60316]: debug 2022-01-31T20:45:33.079+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.079821+0000) 2022-01-31T20:45:33.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[41853]: debug 2022-01-31T20:45:33.078+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.079395+0000) 2022-01-31T20:45:33.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[51620]: debug 2022-01-31T20:45:33.077+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.077970+0000) 2022-01-31T20:45:33.347 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[35325]: debug 2022-01-31T20:45:33.085+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76010 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:33.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[46715]: debug 2022-01-31T20:45:33.077+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.078573+0000) 2022-01-31T20:45:33.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[46715]: debug 2022-01-31T20:45:33.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.267845+0000) 2022-01-31T20:45:33.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[41853]: debug 2022-01-31T20:45:33.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.475869+0000) 2022-01-31T20:45:33.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:33 smithi171 conmon[51620]: debug 2022-01-31T20:45:33.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.347473+0000) 2022-01-31T20:45:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:33 smithi167 conmon[49112]: debug 2022-01-31T20:45:33.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.569933+0000) 2022-01-31T20:45:34.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:34 smithi167 conmon[54076]: debug 2022-01-31T20:45:34.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:34.240582+0000) 2022-01-31T20:45:34.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:33 smithi167 conmon[60316]: debug 2022-01-31T20:45:33.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:33.991969+0000) 2022-01-31T20:45:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:34 smithi171 conmon[46715]: debug 2022-01-31T20:45:34.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:34.267956+0000) 2022-01-31T20:45:34.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:34 smithi171 conmon[51620]: debug 2022-01-31T20:45:34.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:34.347603+0000) 2022-01-31T20:45:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:34 smithi171 conmon[41853]: debug 2022-01-31T20:45:34.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:34.476020+0000) 2022-01-31T20:45:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:34 smithi167 conmon[49112]: debug 2022-01-31T20:45:34.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:34.570095+0000) 2022-01-31T20:45:35.281 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:35 smithi167 conmon[54076]: debug 2022-01-31T20:45:35.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:35.240776+0000) 2022-01-31T20:45:35.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:34 smithi167 conmon[60316]: debug 2022-01-31T20:45:34.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:34.992064+0000) 2022-01-31T20:45:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:35 smithi171 conmon[46715]: debug 2022-01-31T20:45:35.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:35.268119+0000) 2022-01-31T20:45:35.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:35 smithi171 conmon[51620]: debug 2022-01-31T20:45:35.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:35.347782+0000) 2022-01-31T20:45:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:35 smithi171 conmon[41853]: debug 2022-01-31T20:45:35.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:35.476202+0000) 2022-01-31T20:45:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:35 smithi167 conmon[49112]: debug 2022-01-31T20:45:35.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:35.570325+0000) 2022-01-31T20:45:36.282 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:36 smithi167 conmon[54076]: debug 2022-01-31T20:45:36.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:36.240905+0000) 2022-01-31T20:45:36.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:35 smithi167 conmon[60316]: debug 2022-01-31T20:45:35.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:35.992235+0000) 2022-01-31T20:45:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:36 smithi171 conmon[46715]: debug 2022-01-31T20:45:36.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:36.268276+0000) 2022-01-31T20:45:36.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:36 smithi171 conmon[51620]: debug 2022-01-31T20:45:36.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:36.347942+0000) 2022-01-31T20:45:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:36 smithi171 conmon[41853]: debug 2022-01-31T20:45:36.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:36.476388+0000) 2022-01-31T20:45:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:36 smithi167 conmon[49112]: debug 2022-01-31T20:45:36.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:36.570547+0000) 2022-01-31T20:45:37.282 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:37 smithi167 conmon[54076]: debug 2022-01-31T20:45:37.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:37.241060+0000) 2022-01-31T20:45:37.282 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:36 smithi167 conmon[60316]: debug 2022-01-31T20:45:36.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:36.992395+0000) 2022-01-31T20:45:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:37 smithi171 conmon[46715]: debug 2022-01-31T20:45:37.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:37.268489+0000) 2022-01-31T20:45:37.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:37 smithi171 conmon[51620]: debug 2022-01-31T20:45:37.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:37.348128+0000) 2022-01-31T20:45:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:37 smithi171 conmon[41853]: debug 2022-01-31T20:45:37.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:37.476578+0000) 2022-01-31T20:45:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:37 smithi167 conmon[49112]: debug 2022-01-31T20:45:37.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:37.570695+0000) 2022-01-31T20:45:38.282 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:38 smithi167 conmon[49112]: debug 2022-01-31T20:45:38.089+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.089534+0000) 2022-01-31T20:45:38.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:38 smithi167 conmon[54076]: debug 2022-01-31T20:45:38.088+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.089214+0000) 2022-01-31T20:45:38.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:38 smithi167 conmon[54076]: debug 2022-01-31T20:45:38.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.241247+0000) 2022-01-31T20:45:38.283 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:37 smithi167 conmon[60316]: debug 2022-01-31T20:45:37.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:37.992544+0000) 2022-01-31T20:45:38.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:38 smithi167 conmon[60316]: debug 2022-01-31T20:45:38.089+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.089844+0000) 2022-01-31T20:45:38.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[35325]: debug 2022-01-31T20:45:38.095+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76121 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:38.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[46715]: debug 2022-01-31T20:45:38.088+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.089201+0000) 2022-01-31T20:45:38.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[46715]: debug 2022-01-31T20:45:38.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.268740+0000) 2022-01-31T20:45:38.348 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[41853]: debug 2022-01-31T20:45:38.089+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.089857+0000) 2022-01-31T20:45:38.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[51620]: debug 2022-01-31T20:45:38.087+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.088248+0000) 2022-01-31T20:45:38.597 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[35325]: debug 2022-01-31T20:45:38.567+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:45:38.597 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[41853]: debug 2022-01-31T20:45:38.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.476738+0000) 2022-01-31T20:45:38.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:38 smithi171 conmon[51620]: debug 2022-01-31T20:45:38.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.348319+0000) 2022-01-31T20:45:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:38 smithi167 conmon[49112]: debug 2022-01-31T20:45:38.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.570831+0000) 2022-01-31T20:45:39.282 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:39 smithi167 conmon[54076]: debug 2022-01-31T20:45:39.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:39.241365+0000) 2022-01-31T20:45:39.283 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:38 smithi167 conmon[60316]: debug 2022-01-31T20:45:38.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:38.992690+0000) 2022-01-31T20:45:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:39 smithi171 conmon[46715]: debug 2022-01-31T20:45:39.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:39.268910+0000) 2022-01-31T20:45:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:39 smithi171 conmon[51620]: debug 2022-01-31T20:45:39.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:39.348471+0000) 2022-01-31T20:45:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:39 smithi171 conmon[41853]: debug 2022-01-31T20:45:39.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:39.476867+0000) 2022-01-31T20:45:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:39 smithi167 conmon[49112]: debug 2022-01-31T20:45:39.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:39.570955+0000) 2022-01-31T20:45:40.282 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:40 smithi167 conmon[54076]: debug 2022-01-31T20:45:40.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:40.241535+0000) 2022-01-31T20:45:40.283 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:39 smithi167 conmon[60316]: debug 2022-01-31T20:45:39.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:39.992879+0000) 2022-01-31T20:45:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:40 smithi171 conmon[46715]: debug 2022-01-31T20:45:40.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:40.269060+0000) 2022-01-31T20:45:40.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:40 smithi171 conmon[51620]: debug 2022-01-31T20:45:40.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:40.348676+0000) 2022-01-31T20:45:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:40 smithi171 conmon[41853]: debug 2022-01-31T20:45:40.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:40.477033+0000) 2022-01-31T20:45:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:40 smithi167 conmon[49112]: debug 2022-01-31T20:45:40.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:40.571146+0000) 2022-01-31T20:45:41.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:41 smithi167 conmon[54076]: debug 2022-01-31T20:45:41.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:41.241729+0000) 2022-01-31T20:45:41.283 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:40 smithi167 conmon[60316]: debug 2022-01-31T20:45:40.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:40.993030+0000) 2022-01-31T20:45:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:41 smithi171 conmon[46715]: debug 2022-01-31T20:45:41.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:41.269218+0000) 2022-01-31T20:45:41.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:41 smithi171 conmon[51620]: debug 2022-01-31T20:45:41.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:41.348866+0000) 2022-01-31T20:45:41.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:41 smithi171 conmon[41853]: debug 2022-01-31T20:45:41.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:41.477237+0000) 2022-01-31T20:45:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:41 smithi167 conmon[49112]: debug 2022-01-31T20:45:41.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:41.571356+0000) 2022-01-31T20:45:42.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:42 smithi167 conmon[54076]: debug 2022-01-31T20:45:42.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:42.241861+0000) 2022-01-31T20:45:42.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:41 smithi167 conmon[60316]: debug 2022-01-31T20:45:41.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:41.993201+0000) 2022-01-31T20:45:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:42 smithi171 conmon[46715]: debug 2022-01-31T20:45:42.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:42.269429+0000) 2022-01-31T20:45:42.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:42 smithi171 conmon[51620]: debug 2022-01-31T20:45:42.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:42.349067+0000) 2022-01-31T20:45:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:42 smithi171 conmon[41853]: debug 2022-01-31T20:45:42.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:42.477430+0000) 2022-01-31T20:45:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:42 smithi167 conmon[49112]: debug 2022-01-31T20:45:42.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:42.571575+0000) 2022-01-31T20:45:43.283 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:43 smithi167 conmon[49112]: debug 2022-01-31T20:45:43.098+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.100029+0000) 2022-01-31T20:45:43.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:43 smithi167 conmon[54076]: debug 2022-01-31T20:45:43.098+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.099564+0000) 2022-01-31T20:45:43.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:43 smithi167 conmon[54076]: debug 2022-01-31T20:45:43.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.242077+0000) 2022-01-31T20:45:43.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:42 smithi167 conmon[60316]: debug 2022-01-31T20:45:42.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:42.993375+0000) 2022-01-31T20:45:43.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:43 smithi167 conmon[60316]: debug 2022-01-31T20:45:43.097+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.099217+0000) 2022-01-31T20:45:43.347 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[35325]: debug 2022-01-31T20:45:43.105+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76229 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:43.348 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[41853]: debug 2022-01-31T20:45:43.098+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.099821+0000) 2022-01-31T20:45:43.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[46715]: debug 2022-01-31T20:45:43.097+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.099478+0000) 2022-01-31T20:45:43.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[46715]: debug 2022-01-31T20:45:43.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.269615+0000) 2022-01-31T20:45:43.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[51620]: debug 2022-01-31T20:45:43.096+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.098066+0000) 2022-01-31T20:45:43.597 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[41853]: debug 2022-01-31T20:45:43.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.477630+0000) 2022-01-31T20:45:43.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:43 smithi171 conmon[51620]: debug 2022-01-31T20:45:43.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.349241+0000) 2022-01-31T20:45:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:43 smithi167 conmon[49112]: debug 2022-01-31T20:45:43.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.571738+0000) 2022-01-31T20:45:44.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:44 smithi167 conmon[54076]: debug 2022-01-31T20:45:44.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:44.242194+0000) 2022-01-31T20:45:44.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:43 smithi167 conmon[60316]: debug 2022-01-31T20:45:43.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:43.993577+0000) 2022-01-31T20:45:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:44 smithi171 conmon[46715]: debug 2022-01-31T20:45:44.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:44.269784+0000) 2022-01-31T20:45:44.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:44 smithi171 conmon[51620]: debug 2022-01-31T20:45:44.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:44.349376+0000) 2022-01-31T20:45:44.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:44 smithi171 conmon[41853]: debug 2022-01-31T20:45:44.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:44.477763+0000) 2022-01-31T20:45:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:44 smithi167 conmon[49112]: debug 2022-01-31T20:45:44.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:44.571857+0000) 2022-01-31T20:45:45.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:45 smithi167 conmon[54076]: debug 2022-01-31T20:45:45.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:45.242311+0000) 2022-01-31T20:45:45.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:44 smithi167 conmon[60316]: debug 2022-01-31T20:45:44.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:44.993742+0000) 2022-01-31T20:45:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:45 smithi171 conmon[46715]: debug 2022-01-31T20:45:45.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:45.269971+0000) 2022-01-31T20:45:45.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:45 smithi171 conmon[51620]: debug 2022-01-31T20:45:45.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:45.349561+0000) 2022-01-31T20:45:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:45 smithi171 conmon[41853]: debug 2022-01-31T20:45:45.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:45.477916+0000) 2022-01-31T20:45:45.948 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:45 smithi167 conmon[49112]: debug 2022-01-31T20:45:45.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:45.572062+0000) 2022-01-31T20:45:46.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:46 smithi167 conmon[54076]: debug 2022-01-31T20:45:46.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:46.242497+0000) 2022-01-31T20:45:46.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:45 smithi167 conmon[60316]: debug 2022-01-31T20:45:45.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:45.993895+0000) 2022-01-31T20:45:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:46 smithi171 conmon[46715]: debug 2022-01-31T20:45:46.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:46.270187+0000) 2022-01-31T20:45:46.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:46 smithi171 conmon[51620]: debug 2022-01-31T20:45:46.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:46.349747+0000) 2022-01-31T20:45:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:46 smithi171 conmon[41853]: debug 2022-01-31T20:45:46.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:46.478073+0000) 2022-01-31T20:45:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:46 smithi167 conmon[49112]: debug 2022-01-31T20:45:46.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:46.572269+0000) 2022-01-31T20:45:47.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:47 smithi167 conmon[54076]: debug 2022-01-31T20:45:47.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:47.242640+0000) 2022-01-31T20:45:47.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:46 smithi167 conmon[60316]: debug 2022-01-31T20:45:46.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:46.994075+0000) 2022-01-31T20:45:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:47 smithi171 conmon[46715]: debug 2022-01-31T20:45:47.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:47.270390+0000) 2022-01-31T20:45:47.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:47 smithi171 conmon[51620]: debug 2022-01-31T20:45:47.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:47.349899+0000) 2022-01-31T20:45:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:47 smithi171 conmon[41853]: debug 2022-01-31T20:45:47.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:47.478263+0000) 2022-01-31T20:45:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:47 smithi167 conmon[49112]: debug 2022-01-31T20:45:47.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:47.572520+0000) 2022-01-31T20:45:48.218 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:45:48.217+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:45:48.284 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:48 smithi167 conmon[49112]: debug 2022-01-31T20:45:48.109+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.111250+0000) 2022-01-31T20:45:48.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:48 smithi167 conmon[54076]: debug 2022-01-31T20:45:48.109+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.111438+0000) 2022-01-31T20:45:48.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:48 smithi167 conmon[54076]: debug 2022-01-31T20:45:48.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.242816+0000) 2022-01-31T20:45:48.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:47 smithi167 conmon[60316]: debug 2022-01-31T20:45:47.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:47.994209+0000) 2022-01-31T20:45:48.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:48 smithi167 conmon[60316]: debug 2022-01-31T20:45:48.108+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.110335+0000) 2022-01-31T20:45:48.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[35325]: debug 2022-01-31T20:45:48.115+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76340 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:48.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[41853]: debug 2022-01-31T20:45:48.108+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.110478+0000) 2022-01-31T20:45:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[46715]: debug 2022-01-31T20:45:48.108+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.110179+0000) 2022-01-31T20:45:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[46715]: debug 2022-01-31T20:45:48.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.270619+0000) 2022-01-31T20:45:48.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[51620]: debug 2022-01-31T20:45:48.108+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.109921+0000) 2022-01-31T20:45:48.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[51620]: debug 2022-01-31T20:45:48.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.350102+0000) 2022-01-31T20:45:48.736 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:48 smithi171 conmon[41853]: debug 2022-01-31T20:45:48.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.478454+0000) 2022-01-31T20:45:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:48 smithi167 conmon[49112]: debug 2022-01-31T20:45:48.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.572695+0000) 2022-01-31T20:45:49.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:49 smithi167 conmon[54076]: debug 2022-01-31T20:45:49.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:49.242951+0000) 2022-01-31T20:45:49.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:48 smithi167 conmon[60316]: debug 2022-01-31T20:45:48.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:48.994383+0000) 2022-01-31T20:45:49.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:49 smithi171 conmon[41853]: debug 2022-01-31T20:45:49.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:49.478621+0000) 2022-01-31T20:45:49.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:49 smithi171 conmon[46715]: debug 2022-01-31T20:45:49.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:49.270787+0000) 2022-01-31T20:45:49.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:49 smithi171 conmon[51620]: debug 2022-01-31T20:45:49.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:49.350274+0000) 2022-01-31T20:45:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:49 smithi167 conmon[49112]: debug 2022-01-31T20:45:49.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:49.572840+0000) 2022-01-31T20:45:50.241 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:49 smithi167 conmon[60316]: debug 2022-01-31T20:45:49.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:49.994551+0000) 2022-01-31T20:45:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:50 smithi171 conmon[46715]: debug 2022-01-31T20:45:50.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:50.270958+0000) 2022-01-31T20:45:50.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:50 smithi171 conmon[51620]: debug 2022-01-31T20:45:50.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:50.350482+0000) 2022-01-31T20:45:50.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:50 smithi167 conmon[54076]: debug 2022-01-31T20:45:50.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:50.243126+0000) 2022-01-31T20:45:50.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:50 smithi171 conmon[41853]: debug 2022-01-31T20:45:50.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:50.478810+0000) 2022-01-31T20:45:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:50 smithi167 conmon[49112]: debug 2022-01-31T20:45:50.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:50.572990+0000) 2022-01-31T20:45:51.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:51 smithi167 conmon[54076]: debug 2022-01-31T20:45:51.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:51.243334+0000) 2022-01-31T20:45:51.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:50 smithi167 conmon[60316]: debug 2022-01-31T20:45:50.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:50.994726+0000) 2022-01-31T20:45:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:51 smithi171 conmon[46715]: debug 2022-01-31T20:45:51.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:51.271160+0000) 2022-01-31T20:45:51.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:51 smithi171 conmon[51620]: debug 2022-01-31T20:45:51.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:51.350663+0000) 2022-01-31T20:45:51.685 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:51 smithi171 conmon[41853]: debug 2022-01-31T20:45:51.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:51.478954+0000) 2022-01-31T20:45:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:51 smithi167 conmon[49112]: debug 2022-01-31T20:45:51.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:51.573087+0000) 2022-01-31T20:45:52.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:52 smithi167 conmon[54076]: debug 2022-01-31T20:45:52.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:52.243544+0000) 2022-01-31T20:45:52.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:51 smithi167 conmon[60316]: debug 2022-01-31T20:45:51.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:51.994883+0000) 2022-01-31T20:45:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:52 smithi171 conmon[46715]: debug 2022-01-31T20:45:52.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:52.271321+0000) 2022-01-31T20:45:52.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:52 smithi171 conmon[51620]: debug 2022-01-31T20:45:52.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:52.350823+0000) 2022-01-31T20:45:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:52 smithi171 conmon[41853]: debug 2022-01-31T20:45:52.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:52.479132+0000) 2022-01-31T20:45:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:52 smithi167 conmon[49112]: debug 2022-01-31T20:45:52.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:52.573255+0000) 2022-01-31T20:45:53.285 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:53 smithi167 conmon[49112]: debug 2022-01-31T20:45:53.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.121276+0000) 2022-01-31T20:45:53.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:52 smithi167 conmon[60316]: debug 2022-01-31T20:45:52.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:52.995091+0000) 2022-01-31T20:45:53.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:53 smithi167 conmon[60316]: debug 2022-01-31T20:45:53.119+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.121071+0000) 2022-01-31T20:45:53.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:53 smithi167 conmon[54076]: debug 2022-01-31T20:45:53.119+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.120822+0000) 2022-01-31T20:45:53.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:53 smithi167 conmon[54076]: debug 2022-01-31T20:45:53.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.243691+0000) 2022-01-31T20:45:53.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[35325]: debug 2022-01-31T20:45:53.126+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76448 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[41853]: debug 2022-01-31T20:45:53.119+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.120531+0000) 2022-01-31T20:45:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[51620]: debug 2022-01-31T20:45:53.118+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.119590+0000) 2022-01-31T20:45:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[51620]: debug 2022-01-31T20:45:53.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.350950+0000) 2022-01-31T20:45:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[46715]: debug 2022-01-31T20:45:53.118+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.119454+0000) 2022-01-31T20:45:53.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[46715]: debug 2022-01-31T20:45:53.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.271546+0000) 2022-01-31T20:45:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[35325]: debug 2022-01-31T20:45:53.567+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:45:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:53 smithi171 conmon[41853]: debug 2022-01-31T20:45:53.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.479287+0000) 2022-01-31T20:45:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:53 smithi167 conmon[49112]: debug 2022-01-31T20:45:53.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.573460+0000) 2022-01-31T20:45:54.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:54 smithi167 conmon[54076]: debug 2022-01-31T20:45:54.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:54.243865+0000) 2022-01-31T20:45:54.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:53 smithi167 conmon[60316]: debug 2022-01-31T20:45:53.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:53.995251+0000) 2022-01-31T20:45:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:54 smithi171 conmon[46715]: debug 2022-01-31T20:45:54.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:54.271718+0000) 2022-01-31T20:45:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:54 smithi171 conmon[51620]: debug 2022-01-31T20:45:54.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:54.351100+0000) 2022-01-31T20:45:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:54 smithi171 conmon[41853]: debug 2022-01-31T20:45:54.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:54.479494+0000) 2022-01-31T20:45:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:54 smithi167 conmon[49112]: debug 2022-01-31T20:45:54.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:54.573599+0000) 2022-01-31T20:45:55.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:55 smithi167 conmon[54076]: debug 2022-01-31T20:45:55.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:55.244017+0000) 2022-01-31T20:45:55.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:54 smithi167 conmon[60316]: debug 2022-01-31T20:45:54.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:54.995433+0000) 2022-01-31T20:45:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:55 smithi171 conmon[46715]: debug 2022-01-31T20:45:55.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:55.271929+0000) 2022-01-31T20:45:55.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:55 smithi171 conmon[51620]: debug 2022-01-31T20:45:55.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:55.351279+0000) 2022-01-31T20:45:55.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:55 smithi171 conmon[41853]: debug 2022-01-31T20:45:55.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:55.479698+0000) 2022-01-31T20:45:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:55 smithi167 conmon[49112]: debug 2022-01-31T20:45:55.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:55.573714+0000) 2022-01-31T20:45:56.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:56 smithi167 conmon[54076]: debug 2022-01-31T20:45:56.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:56.244158+0000) 2022-01-31T20:45:56.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:55 smithi167 conmon[60316]: debug 2022-01-31T20:45:55.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:55.995637+0000) 2022-01-31T20:45:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:56 smithi171 conmon[46715]: debug 2022-01-31T20:45:56.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:56.272120+0000) 2022-01-31T20:45:56.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:56 smithi171 conmon[51620]: debug 2022-01-31T20:45:56.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:56.351460+0000) 2022-01-31T20:45:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:56 smithi171 conmon[41853]: debug 2022-01-31T20:45:56.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:56.479941+0000) 2022-01-31T20:45:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:56 smithi167 conmon[49112]: debug 2022-01-31T20:45:56.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:56.573867+0000) 2022-01-31T20:45:57.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:57 smithi167 conmon[54076]: debug 2022-01-31T20:45:57.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:57.244371+0000) 2022-01-31T20:45:57.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:56 smithi167 conmon[60316]: debug 2022-01-31T20:45:56.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:56.995821+0000) 2022-01-31T20:45:57.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:57 smithi171 conmon[51620]: debug 2022-01-31T20:45:57.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:57.351636+0000) 2022-01-31T20:45:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:57 smithi171 conmon[46715]: debug 2022-01-31T20:45:57.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:57.272307+0000) 2022-01-31T20:45:57.686 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:57 smithi171 conmon[41853]: debug 2022-01-31T20:45:57.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:57.480098+0000) 2022-01-31T20:45:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:57 smithi167 conmon[49112]: debug 2022-01-31T20:45:57.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:57.574071+0000) 2022-01-31T20:45:58.285 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:58 smithi167 conmon[49112]: debug 2022-01-31T20:45:58.130+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.132009+0000) 2022-01-31T20:45:58.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:58 smithi167 conmon[54076]: debug 2022-01-31T20:45:58.129+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.131137+0000) 2022-01-31T20:45:58.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:58 smithi167 conmon[54076]: debug 2022-01-31T20:45:58.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.244595+0000) 2022-01-31T20:45:58.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:57 smithi167 conmon[60316]: debug 2022-01-31T20:45:57.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:57.996022+0000) 2022-01-31T20:45:58.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:58 smithi167 conmon[60316]: debug 2022-01-31T20:45:58.129+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.130942+0000) 2022-01-31T20:45:58.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[35325]: debug 2022-01-31T20:45:58.136+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76559 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:45:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[41853]: debug 2022-01-31T20:45:58.129+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.129811+0000) 2022-01-31T20:45:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[46715]: debug 2022-01-31T20:45:58.129+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.130038+0000) 2022-01-31T20:45:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[46715]: debug 2022-01-31T20:45:58.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.272471+0000) 2022-01-31T20:45:58.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[51620]: debug 2022-01-31T20:45:58.130+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.131638+0000) 2022-01-31T20:45:58.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[51620]: debug 2022-01-31T20:45:58.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.351854+0000) 2022-01-31T20:45:58.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:58 smithi171 conmon[41853]: debug 2022-01-31T20:45:58.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.480256+0000) 2022-01-31T20:45:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:58 smithi167 conmon[49112]: debug 2022-01-31T20:45:58.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.574249+0000) 2022-01-31T20:45:59.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:45:59 smithi167 conmon[54076]: debug 2022-01-31T20:45:59.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:59.244723+0000) 2022-01-31T20:45:59.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:58 smithi167 conmon[60316]: debug 2022-01-31T20:45:58.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:58.996186+0000) 2022-01-31T20:45:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:45:59 smithi171 conmon[46715]: debug 2022-01-31T20:45:59.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:59.272616+0000) 2022-01-31T20:45:59.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:45:59 smithi171 conmon[51620]: debug 2022-01-31T20:45:59.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:59.352007+0000) 2022-01-31T20:45:59.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:45:59 smithi171 conmon[41853]: debug 2022-01-31T20:45:59.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:59.480426+0000) 2022-01-31T20:45:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:45:59 smithi167 conmon[49112]: debug 2022-01-31T20:45:59.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:59.574401+0000) 2022-01-31T20:46:00.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:00 smithi167 conmon[54076]: debug 2022-01-31T20:46:00.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:00.244846+0000) 2022-01-31T20:46:00.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:45:59 smithi167 conmon[60316]: debug 2022-01-31T20:45:59.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:45:59.996365+0000) 2022-01-31T20:46:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:00 smithi171 conmon[46715]: debug 2022-01-31T20:46:00.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:00.272840+0000) 2022-01-31T20:46:00.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:00 smithi171 conmon[51620]: debug 2022-01-31T20:46:00.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:00.352226+0000) 2022-01-31T20:46:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:00 smithi171 conmon[41853]: debug 2022-01-31T20:46:00.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:00.480625+0000) 2022-01-31T20:46:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:00 smithi167 conmon[49112]: debug 2022-01-31T20:46:00.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:00.574602+0000) 2022-01-31T20:46:01.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:01 smithi167 conmon[54076]: debug 2022-01-31T20:46:01.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:01.245071+0000) 2022-01-31T20:46:01.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:00 smithi167 conmon[60316]: debug 2022-01-31T20:46:00.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:00.996521+0000) 2022-01-31T20:46:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:01 smithi171 conmon[46715]: debug 2022-01-31T20:46:01.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:01.272998+0000) 2022-01-31T20:46:01.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:01 smithi171 conmon[51620]: debug 2022-01-31T20:46:01.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:01.352433+0000) 2022-01-31T20:46:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:01 smithi171 conmon[41853]: debug 2022-01-31T20:46:01.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:01.480814+0000) 2022-01-31T20:46:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:01 smithi167 conmon[49112]: debug 2022-01-31T20:46:01.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:01.574779+0000) 2022-01-31T20:46:02.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:02 smithi167 conmon[54076]: debug 2022-01-31T20:46:02.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:02.245304+0000) 2022-01-31T20:46:02.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:01 smithi167 conmon[60316]: debug 2022-01-31T20:46:01.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:01.996684+0000) 2022-01-31T20:46:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:02 smithi171 conmon[46715]: debug 2022-01-31T20:46:02.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:02.273182+0000) 2022-01-31T20:46:02.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:02 smithi171 conmon[51620]: debug 2022-01-31T20:46:02.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:02.352643+0000) 2022-01-31T20:46:02.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:02 smithi171 conmon[41853]: debug 2022-01-31T20:46:02.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:02.480998+0000) 2022-01-31T20:46:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:02 smithi167 conmon[49112]: debug 2022-01-31T20:46:02.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:02.574926+0000) 2022-01-31T20:46:03.286 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:03 smithi167 conmon[49112]: debug 2022-01-31T20:46:03.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.141452+0000) 2022-01-31T20:46:03.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:03 smithi167 conmon[54076]: debug 2022-01-31T20:46:03.139+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.140645+0000) 2022-01-31T20:46:03.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:03 smithi167 conmon[54076]: debug 2022-01-31T20:46:03.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.245492+0000) 2022-01-31T20:46:03.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:02 smithi167 conmon[60316]: debug 2022-01-31T20:46:02.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:02.996841+0000) 2022-01-31T20:46:03.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:03 smithi167 conmon[60316]: debug 2022-01-31T20:46:03.139+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.141270+0000) 2022-01-31T20:46:03.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[35325]: debug 2022-01-31T20:46:03.148+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76667 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:03.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[41853]: debug 2022-01-31T20:46:03.139+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.140283+0000) 2022-01-31T20:46:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[46715]: debug 2022-01-31T20:46:03.139+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.140171+0000) 2022-01-31T20:46:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[46715]: debug 2022-01-31T20:46:03.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.273328+0000) 2022-01-31T20:46:03.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[51620]: debug 2022-01-31T20:46:03.139+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.139988+0000) 2022-01-31T20:46:03.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[51620]: debug 2022-01-31T20:46:03.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.352827+0000) 2022-01-31T20:46:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:03 smithi171 conmon[41853]: debug 2022-01-31T20:46:03.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.481190+0000) 2022-01-31T20:46:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:03 smithi167 conmon[49112]: debug 2022-01-31T20:46:03.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.575119+0000) 2022-01-31T20:46:04.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:04 smithi167 conmon[54076]: debug 2022-01-31T20:46:04.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:04.245700+0000) 2022-01-31T20:46:04.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:03 smithi167 conmon[60316]: debug 2022-01-31T20:46:03.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:03.997031+0000) 2022-01-31T20:46:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:04 smithi171 conmon[46715]: debug 2022-01-31T20:46:04.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:04.273468+0000) 2022-01-31T20:46:04.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:04 smithi171 conmon[51620]: debug 2022-01-31T20:46:04.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:04.352923+0000) 2022-01-31T20:46:04.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:04 smithi171 conmon[41853]: debug 2022-01-31T20:46:04.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:04.481355+0000) 2022-01-31T20:46:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:04 smithi167 conmon[49112]: debug 2022-01-31T20:46:04.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:04.575329+0000) 2022-01-31T20:46:05.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:05 smithi167 conmon[54076]: debug 2022-01-31T20:46:05.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:05.245890+0000) 2022-01-31T20:46:05.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:04 smithi167 conmon[60316]: debug 2022-01-31T20:46:04.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:04.997196+0000) 2022-01-31T20:46:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:05 smithi171 conmon[46715]: debug 2022-01-31T20:46:05.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:05.273657+0000) 2022-01-31T20:46:05.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:05 smithi171 conmon[51620]: debug 2022-01-31T20:46:05.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:05.353047+0000) 2022-01-31T20:46:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:05 smithi171 conmon[41853]: debug 2022-01-31T20:46:05.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:05.481578+0000) 2022-01-31T20:46:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:05 smithi167 conmon[49112]: debug 2022-01-31T20:46:05.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:05.575513+0000) 2022-01-31T20:46:06.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:06 smithi167 conmon[54076]: debug 2022-01-31T20:46:06.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:06.246014+0000) 2022-01-31T20:46:06.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:05 smithi167 conmon[60316]: debug 2022-01-31T20:46:05.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:05.997392+0000) 2022-01-31T20:46:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:06 smithi171 conmon[46715]: debug 2022-01-31T20:46:06.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:06.273824+0000) 2022-01-31T20:46:06.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:06 smithi171 conmon[51620]: debug 2022-01-31T20:46:06.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:06.353150+0000) 2022-01-31T20:46:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:06 smithi171 conmon[41853]: debug 2022-01-31T20:46:06.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:06.481785+0000) 2022-01-31T20:46:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:06 smithi167 conmon[49112]: debug 2022-01-31T20:46:06.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:06.575661+0000) 2022-01-31T20:46:07.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:07 smithi167 conmon[54076]: debug 2022-01-31T20:46:07.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:07.246190+0000) 2022-01-31T20:46:07.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:06 smithi167 conmon[60316]: debug 2022-01-31T20:46:06.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:06.997577+0000) 2022-01-31T20:46:07.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:07 smithi171 conmon[51620]: debug 2022-01-31T20:46:07.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:07.353267+0000) 2022-01-31T20:46:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:07 smithi171 conmon[46715]: debug 2022-01-31T20:46:07.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:07.274004+0000) 2022-01-31T20:46:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:07 smithi171 conmon[41853]: debug 2022-01-31T20:46:07.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:07.481946+0000) 2022-01-31T20:46:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:07 smithi167 conmon[49112]: debug 2022-01-31T20:46:07.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:07.575850+0000) 2022-01-31T20:46:08.287 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[49112]: debug 2022-01-31T20:46:08.152+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.152563+0000) 2022-01-31T20:46:08.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[54076]: debug 2022-01-31T20:46:08.150+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.151179+0000) 2022-01-31T20:46:08.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[54076]: debug 2022-01-31T20:46:08.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.246407+0000) 2022-01-31T20:46:08.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:07 smithi167 conmon[60316]: debug 2022-01-31T20:46:07.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:07.997735+0000) 2022-01-31T20:46:08.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[60316]: debug 2022-01-31T20:46:08.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[60316]: 2022-01-31T20:46:08.151+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.152405+0000) 2022-01-31T20:46:08.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[35325]: debug 2022-01-31T20:46:08.158+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76778 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[41853]: debug 2022-01-31T20:46:08.151+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.151815+0000) 2022-01-31T20:46:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[46715]: debug 2022-01-31T20:46:08.151+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.151741+0000) 2022-01-31T20:46:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[46715]: debug 2022-01-31T20:46:08.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.274198+0000) 2022-01-31T20:46:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[51620]: debug 2022-01-31T20:46:08.150+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.151351+0000) 2022-01-31T20:46:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[51620]: debug 2022-01-31T20:46:08.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.353430+0000) 2022-01-31T20:46:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[35325]: debug 2022-01-31T20:46:08.568+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:46:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:08 smithi171 conmon[41853]: debug 2022-01-31T20:46:08.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.482131+0000) 2022-01-31T20:46:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[49112]: debug 2022-01-31T20:46:08.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.576043+0000) 2022-01-31T20:46:09.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:08 smithi167 conmon[60316]: debug 2022-01-31T20:46:08.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:08.997934+0000) 2022-01-31T20:46:09.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:09 smithi167 conmon[54076]: debug 2022-01-31T20:46:09.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:09.246542+0000) 2022-01-31T20:46:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:09 smithi171 conmon[46715]: debug 2022-01-31T20:46:09.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:09.274340+0000) 2022-01-31T20:46:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:09 smithi171 conmon[51620]: debug 2022-01-31T20:46:09.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:09.353576+0000) 2022-01-31T20:46:09.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:09 smithi171 conmon[41853]: debug 2022-01-31T20:46:09.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:09.482302+0000) 2022-01-31T20:46:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:09 smithi167 conmon[49112]: debug 2022-01-31T20:46:09.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:09.576212+0000) 2022-01-31T20:46:10.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:10 smithi167 conmon[54076]: debug 2022-01-31T20:46:10.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:10.246690+0000) 2022-01-31T20:46:10.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:09 smithi167 conmon[60316]: debug 2022-01-31T20:46:09.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:09.998122+0000) 2022-01-31T20:46:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:10 smithi171 conmon[46715]: debug 2022-01-31T20:46:10.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:10.274468+0000) 2022-01-31T20:46:10.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:10 smithi171 conmon[51620]: debug 2022-01-31T20:46:10.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:10.353768+0000) 2022-01-31T20:46:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:10 smithi171 conmon[41853]: debug 2022-01-31T20:46:10.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:10.482511+0000) 2022-01-31T20:46:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:10 smithi167 conmon[49112]: debug 2022-01-31T20:46:10.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:10.576429+0000) 2022-01-31T20:46:11.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:11 smithi167 conmon[54076]: debug 2022-01-31T20:46:11.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:11.246852+0000) 2022-01-31T20:46:11.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:10 smithi167 conmon[60316]: debug 2022-01-31T20:46:10.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:10.998284+0000) 2022-01-31T20:46:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:11 smithi171 conmon[46715]: debug 2022-01-31T20:46:11.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:11.274627+0000) 2022-01-31T20:46:11.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:11 smithi171 conmon[51620]: debug 2022-01-31T20:46:11.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:11.353928+0000) 2022-01-31T20:46:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:11 smithi171 conmon[41853]: debug 2022-01-31T20:46:11.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:11.482690+0000) 2022-01-31T20:46:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:11 smithi167 conmon[49112]: debug 2022-01-31T20:46:11.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:11.576558+0000) 2022-01-31T20:46:12.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:12 smithi167 conmon[54076]: debug 2022-01-31T20:46:12.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:12.247010+0000) 2022-01-31T20:46:12.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:11 smithi167 conmon[60316]: debug 2022-01-31T20:46:11.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:11.998482+0000) 2022-01-31T20:46:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:12 smithi171 conmon[46715]: debug 2022-01-31T20:46:12.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:12.274781+0000) 2022-01-31T20:46:12.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:12 smithi171 conmon[51620]: debug 2022-01-31T20:46:12.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:12.354114+0000) 2022-01-31T20:46:12.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:12 smithi171 conmon[41853]: debug 2022-01-31T20:46:12.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:12.482860+0000) 2022-01-31T20:46:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:12 smithi167 conmon[49112]: debug 2022-01-31T20:46:12.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:12.576708+0000) 2022-01-31T20:46:13.288 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:13 smithi167 conmon[49112]: debug 2022-01-31T20:46:13.162+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.163059+0000) 2022-01-31T20:46:13.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:13 smithi167 conmon[54076]: debug 2022-01-31T20:46:13.162+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.162770+0000) 2022-01-31T20:46:13.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:13 smithi167 conmon[54076]: debug 2022-01-31T20:46:13.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.247201+0000) 2022-01-31T20:46:13.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:12 smithi167 conmon[60316]: debug 2022-01-31T20:46:12.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:12.998683+0000) 2022-01-31T20:46:13.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:13 smithi167 conmon[60316]: debug 2022-01-31T20:46:13.163+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.164148+0000) 2022-01-31T20:46:13.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[35325]: debug 2022-01-31T20:46:13.170+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 76886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:13.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[41853]: debug 2022-01-31T20:46:13.161+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.162475+0000) 2022-01-31T20:46:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[46715]: debug 2022-01-31T20:46:13.163+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.163823+0000) 2022-01-31T20:46:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[46715]: debug 2022-01-31T20:46:13.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.274966+0000) 2022-01-31T20:46:13.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[51620]: debug 2022-01-31T20:46:13.161+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.162357+0000) 2022-01-31T20:46:13.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[51620]: debug 2022-01-31T20:46:13.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.354274+0000) 2022-01-31T20:46:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:13 smithi171 conmon[41853]: debug 2022-01-31T20:46:13.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.483058+0000) 2022-01-31T20:46:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:13 smithi167 conmon[49112]: debug 2022-01-31T20:46:13.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.576864+0000) 2022-01-31T20:46:14.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:14 smithi167 conmon[54076]: debug 2022-01-31T20:46:14.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:14.247388+0000) 2022-01-31T20:46:14.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:13 smithi167 conmon[60316]: debug 2022-01-31T20:46:13.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:13.998804+0000) 2022-01-31T20:46:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:14 smithi171 conmon[46715]: debug 2022-01-31T20:46:14.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:14.275135+0000) 2022-01-31T20:46:14.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:14 smithi171 conmon[51620]: debug 2022-01-31T20:46:14.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:14.354419+0000) 2022-01-31T20:46:14.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:14 smithi171 conmon[41853]: debug 2022-01-31T20:46:14.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:14.483215+0000) 2022-01-31T20:46:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:14 smithi167 conmon[49112]: debug 2022-01-31T20:46:14.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:14.577025+0000) 2022-01-31T20:46:15.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:15 smithi167 conmon[54076]: debug 2022-01-31T20:46:15.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:15.247588+0000) 2022-01-31T20:46:15.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:14 smithi167 conmon[60316]: debug 2022-01-31T20:46:14.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:14.998953+0000) 2022-01-31T20:46:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:15 smithi171 conmon[46715]: debug 2022-01-31T20:46:15.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:15.275343+0000) 2022-01-31T20:46:15.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:15 smithi171 conmon[51620]: debug 2022-01-31T20:46:15.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:15.354618+0000) 2022-01-31T20:46:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:15 smithi171 conmon[41853]: debug 2022-01-31T20:46:15.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:15.483404+0000) 2022-01-31T20:46:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:15 smithi167 conmon[49112]: debug 2022-01-31T20:46:15.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:15.577229+0000) 2022-01-31T20:46:16.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:16 smithi167 conmon[54076]: debug 2022-01-31T20:46:16.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:16.247754+0000) 2022-01-31T20:46:16.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:15 smithi167 conmon[60316]: debug 2022-01-31T20:46:15.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:15.999099+0000) 2022-01-31T20:46:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:16 smithi171 conmon[46715]: debug 2022-01-31T20:46:16.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:16.275549+0000) 2022-01-31T20:46:16.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:16 smithi171 conmon[51620]: debug 2022-01-31T20:46:16.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:16.354820+0000) 2022-01-31T20:46:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:16 smithi171 conmon[41853]: debug 2022-01-31T20:46:16.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:16.483615+0000) 2022-01-31T20:46:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:16 smithi167 conmon[49112]: debug 2022-01-31T20:46:16.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:16.577453+0000) 2022-01-31T20:46:17.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:17 smithi167 conmon[54076]: debug 2022-01-31T20:46:17.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:17.247861+0000) 2022-01-31T20:46:17.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:16 smithi167 conmon[60316]: debug 2022-01-31T20:46:16.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:16.999259+0000) 2022-01-31T20:46:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:17 smithi171 conmon[46715]: debug 2022-01-31T20:46:17.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:17.275776+0000) 2022-01-31T20:46:17.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:17 smithi171 conmon[51620]: debug 2022-01-31T20:46:17.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:17.354943+0000) 2022-01-31T20:46:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:17 smithi171 conmon[41853]: debug 2022-01-31T20:46:17.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:17.483792+0000) 2022-01-31T20:46:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:17 smithi167 conmon[49112]: debug 2022-01-31T20:46:17.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:17.577598+0000) 2022-01-31T20:46:18.289 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:18 smithi167 conmon[49112]: debug 2022-01-31T20:46:18.173+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.174384+0000) 2022-01-31T20:46:18.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:18 smithi167 conmon[54076]: debug 2022-01-31T20:46:18.174+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.175082+0000) 2022-01-31T20:46:18.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:18 smithi167 conmon[54076]: debug 2022-01-31T20:46:18.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.248029+0000) 2022-01-31T20:46:18.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:17 smithi167 conmon[60316]: debug 2022-01-31T20:46:17.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:17.999495+0000) 2022-01-31T20:46:18.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:18 smithi167 conmon[60316]: debug 2022-01-31T20:46:18.173+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.174286+0000) 2022-01-31T20:46:18.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[35325]: debug 2022-01-31T20:46:18.180+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77002 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[41853]: debug 2022-01-31T20:46:18.172+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.173501+0000) 2022-01-31T20:46:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[46715]: debug 2022-01-31T20:46:18.174+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.175421+0000) 2022-01-31T20:46:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[46715]: debug 2022-01-31T20:46:18.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.275963+0000) 2022-01-31T20:46:18.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[51620]: debug 2022-01-31T20:46:18.172+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.172929+0000) 2022-01-31T20:46:18.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[51620]: debug 2022-01-31T20:46:18.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.355046+0000) 2022-01-31T20:46:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:18 smithi171 conmon[41853]: debug 2022-01-31T20:46:18.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.483980+0000) 2022-01-31T20:46:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:18 smithi167 conmon[49112]: debug 2022-01-31T20:46:18.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.577748+0000) 2022-01-31T20:46:19.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:19 smithi167 conmon[54076]: debug 2022-01-31T20:46:19.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:19.248239+0000) 2022-01-31T20:46:19.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:18 smithi167 conmon[60316]: debug 2022-01-31T20:46:18.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:18.999699+0000) 2022-01-31T20:46:19.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:19 smithi171 conmon[46715]: debug 2022-01-31T20:46:19.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:19.276120+0000) 2022-01-31T20:46:19.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:19 smithi171 conmon[51620]: debug 2022-01-31T20:46:19.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:19.355144+0000) 2022-01-31T20:46:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:19 smithi171 conmon[41853]: debug 2022-01-31T20:46:19.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:19.484086+0000) 2022-01-31T20:46:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:19 smithi167 conmon[49112]: debug 2022-01-31T20:46:19.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:19.577892+0000) 2022-01-31T20:46:20.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:20 smithi167 conmon[60316]: debug 2022-01-31T20:46:19.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:19.999927+0000) 2022-01-31T20:46:20.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:20 smithi167 conmon[54076]: debug 2022-01-31T20:46:20.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:20.248447+0000) 2022-01-31T20:46:20.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:20 smithi171 conmon[46715]: debug 2022-01-31T20:46:20.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:20.276307+0000) 2022-01-31T20:46:20.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:20 smithi171 conmon[51620]: debug 2022-01-31T20:46:20.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:20.355308+0000) 2022-01-31T20:46:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:20 smithi171 conmon[41853]: debug 2022-01-31T20:46:20.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:20.484192+0000) 2022-01-31T20:46:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:20 smithi167 conmon[49112]: debug 2022-01-31T20:46:20.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:20.578058+0000) 2022-01-31T20:46:21.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:21 smithi167 conmon[54076]: debug 2022-01-31T20:46:21.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:21.248694+0000) 2022-01-31T20:46:21.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:21 smithi167 conmon[60316]: debug 2022-01-31T20:46:20.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:21.000088+0000) 2022-01-31T20:46:21.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:21 smithi171 conmon[46715]: debug 2022-01-31T20:46:21.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:21.276468+0000) 2022-01-31T20:46:21.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:21 smithi171 conmon[51620]: debug 2022-01-31T20:46:21.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:21.355463+0000) 2022-01-31T20:46:21.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:21 smithi171 conmon[41853]: debug 2022-01-31T20:46:21.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:21.484377+0000) 2022-01-31T20:46:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:21 smithi167 conmon[49112]: debug 2022-01-31T20:46:21.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:21.578229+0000) 2022-01-31T20:46:22.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:22 smithi167 conmon[54076]: debug 2022-01-31T20:46:22.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:22.248882+0000) 2022-01-31T20:46:22.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:22 smithi167 conmon[60316]: debug 2022-01-31T20:46:21.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:22.000243+0000) 2022-01-31T20:46:22.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:22 smithi171 conmon[46715]: debug 2022-01-31T20:46:22.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:22.276620+0000) 2022-01-31T20:46:22.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:22 smithi171 conmon[51620]: debug 2022-01-31T20:46:22.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:22.355644+0000) 2022-01-31T20:46:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:22 smithi171 conmon[41853]: debug 2022-01-31T20:46:22.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:22.484583+0000) 2022-01-31T20:46:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:22 smithi167 conmon[49112]: debug 2022-01-31T20:46:22.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:22.578417+0000) 2022-01-31T20:46:23.290 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:23 smithi167 conmon[49112]: debug 2022-01-31T20:46:23.183+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.184049+0000) 2022-01-31T20:46:23.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:23 smithi167 conmon[54076]: debug 2022-01-31T20:46:23.184+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.185189+0000) 2022-01-31T20:46:23.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:23 smithi167 conmon[54076]: debug 2022-01-31T20:46:23.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.249087+0000) 2022-01-31T20:46:23.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:23 smithi167 conmon[60316]: debug 2022-01-31T20:46:22.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.000386+0000) 2022-01-31T20:46:23.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:23 smithi167 conmon[60316]: debug 2022-01-31T20:46:23.183+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.184183+0000) 2022-01-31T20:46:23.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[41853]: debug 2022-01-31T20:46:23.182+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.183723+0000) 2022-01-31T20:46:23.484 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[35325]: debug 2022-01-31T20:46:23.192+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77110 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:23.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[46715]: debug 2022-01-31T20:46:23.184+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.185193+0000) 2022-01-31T20:46:23.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[46715]: debug 2022-01-31T20:46:23.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.276735+0000) 2022-01-31T20:46:23.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[51620]: debug 2022-01-31T20:46:23.183+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.184735+0000) 2022-01-31T20:46:23.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[51620]: debug 2022-01-31T20:46:23.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.355809+0000) 2022-01-31T20:46:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[35325]: debug 2022-01-31T20:46:23.569+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:46:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:23 smithi171 conmon[41853]: debug 2022-01-31T20:46:23.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.484783+0000) 2022-01-31T20:46:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:23 smithi167 conmon[49112]: debug 2022-01-31T20:46:23.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:23.578559+0000) 2022-01-31T20:46:24.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:24 smithi167 conmon[54076]: debug 2022-01-31T20:46:24.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:24.249232+0000) 2022-01-31T20:46:24.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:24 smithi167 conmon[60316]: debug 2022-01-31T20:46:24.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:24.000572+0000) 2022-01-31T20:46:24.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:24 smithi171 conmon[46715]: debug 2022-01-31T20:46:24.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:24.276903+0000) 2022-01-31T20:46:24.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:24 smithi171 conmon[51620]: debug 2022-01-31T20:46:24.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:24.355976+0000) 2022-01-31T20:46:24.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:24 smithi171 conmon[41853]: debug 2022-01-31T20:46:24.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:24.484946+0000) 2022-01-31T20:46:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:24 smithi167 conmon[49112]: debug 2022-01-31T20:46:24.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:24.578668+0000) 2022-01-31T20:46:25.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:25 smithi167 conmon[54076]: debug 2022-01-31T20:46:25.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:25.249416+0000) 2022-01-31T20:46:25.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:25 smithi167 conmon[60316]: debug 2022-01-31T20:46:25.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:25.000732+0000) 2022-01-31T20:46:25.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:25 smithi171 conmon[46715]: debug 2022-01-31T20:46:25.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:25.277111+0000) 2022-01-31T20:46:25.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:25 smithi171 conmon[51620]: debug 2022-01-31T20:46:25.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:25.356133+0000) 2022-01-31T20:46:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:25 smithi171 conmon[41853]: debug 2022-01-31T20:46:25.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:25.485100+0000) 2022-01-31T20:46:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:25 smithi167 conmon[49112]: debug 2022-01-31T20:46:25.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:25.578835+0000) 2022-01-31T20:46:26.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:26 smithi167 conmon[54076]: debug 2022-01-31T20:46:26.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:26.249579+0000) 2022-01-31T20:46:26.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:26 smithi167 conmon[60316]: debug 2022-01-31T20:46:26.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:26.000844+0000) 2022-01-31T20:46:26.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:26 smithi171 conmon[46715]: debug 2022-01-31T20:46:26.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:26.277239+0000) 2022-01-31T20:46:26.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:26 smithi171 conmon[51620]: debug 2022-01-31T20:46:26.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:26.356335+0000) 2022-01-31T20:46:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:26 smithi171 conmon[41853]: debug 2022-01-31T20:46:26.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:26.485193+0000) 2022-01-31T20:46:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:26 smithi167 conmon[49112]: debug 2022-01-31T20:46:26.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:26.579023+0000) 2022-01-31T20:46:27.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:27 smithi167 conmon[54076]: debug 2022-01-31T20:46:27.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:27.249774+0000) 2022-01-31T20:46:27.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:27 smithi167 conmon[60316]: debug 2022-01-31T20:46:27.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:27.001027+0000) 2022-01-31T20:46:27.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:27 smithi171 conmon[46715]: debug 2022-01-31T20:46:27.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:27.277396+0000) 2022-01-31T20:46:27.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:27 smithi171 conmon[51620]: debug 2022-01-31T20:46:27.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:27.356501+0000) 2022-01-31T20:46:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:27 smithi171 conmon[41853]: debug 2022-01-31T20:46:27.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:27.485346+0000) 2022-01-31T20:46:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:27 smithi167 conmon[49112]: debug 2022-01-31T20:46:27.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:27.579171+0000) 2022-01-31T20:46:28.291 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:28 smithi167 conmon[49112]: debug 2022-01-31T20:46:28.194+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.195179+0000) 2022-01-31T20:46:28.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:28 smithi167 conmon[54076]: debug 2022-01-31T20:46:28.195+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.195721+0000) 2022-01-31T20:46:28.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:28 smithi167 conmon[54076]: debug 2022-01-31T20:46:28.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.249927+0000) 2022-01-31T20:46:28.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:28 smithi167 conmon[60316]: debug 2022-01-31T20:46:28.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.001213+0000) 2022-01-31T20:46:28.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:28 smithi167 conmon[60316]: debug 2022-01-31T20:46:28.195+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.195880+0000) 2022-01-31T20:46:28.484 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[35325]: debug 2022-01-31T20:46:28.201+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77221 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:28.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[41853]: debug 2022-01-31T20:46:28.192+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.194561+0000) 2022-01-31T20:46:28.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[46715]: debug 2022-01-31T20:46:28.194+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.196211+0000) 2022-01-31T20:46:28.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[46715]: debug 2022-01-31T20:46:28.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.277542+0000) 2022-01-31T20:46:28.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[51620]: debug 2022-01-31T20:46:28.193+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.195333+0000) 2022-01-31T20:46:28.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[51620]: debug 2022-01-31T20:46:28.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.356689+0000) 2022-01-31T20:46:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:28 smithi171 conmon[41853]: debug 2022-01-31T20:46:28.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.485558+0000) 2022-01-31T20:46:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:28 smithi167 conmon[49112]: debug 2022-01-31T20:46:28.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:28.579346+0000) 2022-01-31T20:46:29.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:29 smithi167 conmon[54076]: debug 2022-01-31T20:46:29.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:29.250063+0000) 2022-01-31T20:46:29.291 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:29 smithi167 conmon[60316]: debug 2022-01-31T20:46:29.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:29.001384+0000) 2022-01-31T20:46:29.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:29 smithi171 conmon[46715]: debug 2022-01-31T20:46:29.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:29.277696+0000) 2022-01-31T20:46:29.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:29 smithi171 conmon[51620]: debug 2022-01-31T20:46:29.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:29.356853+0000) 2022-01-31T20:46:29.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:29 smithi171 conmon[41853]: debug 2022-01-31T20:46:29.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:29.485714+0000) 2022-01-31T20:46:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:29 smithi167 conmon[49112]: debug 2022-01-31T20:46:29.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:29.579474+0000) 2022-01-31T20:46:30.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:30 smithi167 conmon[54076]: debug 2022-01-31T20:46:30.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:30.250263+0000) 2022-01-31T20:46:30.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:30 smithi167 conmon[60316]: debug 2022-01-31T20:46:30.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:30.001544+0000) 2022-01-31T20:46:30.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:30 smithi171 conmon[46715]: debug 2022-01-31T20:46:30.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:30.277904+0000) 2022-01-31T20:46:30.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:30 smithi171 conmon[51620]: debug 2022-01-31T20:46:30.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:30.357024+0000) 2022-01-31T20:46:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:30 smithi171 conmon[41853]: debug 2022-01-31T20:46:30.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:30.485938+0000) 2022-01-31T20:46:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:30 smithi167 conmon[49112]: debug 2022-01-31T20:46:30.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:30.579635+0000) 2022-01-31T20:46:31.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:31 smithi167 conmon[54076]: debug 2022-01-31T20:46:31.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:31.250422+0000) 2022-01-31T20:46:31.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:31 smithi167 conmon[60316]: debug 2022-01-31T20:46:31.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:31.001708+0000) 2022-01-31T20:46:31.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:31 smithi171 conmon[46715]: debug 2022-01-31T20:46:31.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:31.278096+0000) 2022-01-31T20:46:31.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:31 smithi171 conmon[51620]: debug 2022-01-31T20:46:31.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:31.357177+0000) 2022-01-31T20:46:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:31 smithi171 conmon[41853]: debug 2022-01-31T20:46:31.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:31.486128+0000) 2022-01-31T20:46:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:31 smithi167 conmon[49112]: debug 2022-01-31T20:46:31.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:31.579768+0000) 2022-01-31T20:46:32.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:32 smithi167 conmon[54076]: debug 2022-01-31T20:46:32.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:32.250577+0000) 2022-01-31T20:46:32.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:32 smithi167 conmon[60316]: debug 2022-01-31T20:46:32.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:32.001915+0000) 2022-01-31T20:46:32.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:32 smithi171 conmon[46715]: debug 2022-01-31T20:46:32.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:32.278282+0000) 2022-01-31T20:46:32.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:32 smithi171 conmon[51620]: debug 2022-01-31T20:46:32.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:32.357389+0000) 2022-01-31T20:46:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:32 smithi171 conmon[41853]: debug 2022-01-31T20:46:32.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:32.486341+0000) 2022-01-31T20:46:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:32 smithi167 conmon[49112]: debug 2022-01-31T20:46:32.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:32.579973+0000) 2022-01-31T20:46:33.292 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:33 smithi167 conmon[49112]: debug 2022-01-31T20:46:33.204+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.205625+0000) 2022-01-31T20:46:33.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:33 smithi167 conmon[54076]: debug 2022-01-31T20:46:33.203+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.204952+0000) 2022-01-31T20:46:33.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:33 smithi167 conmon[54076]: debug 2022-01-31T20:46:33.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.250726+0000) 2022-01-31T20:46:33.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:33 smithi167 conmon[60316]: debug 2022-01-31T20:46:33.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.002084+0000) 2022-01-31T20:46:33.294 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:33 smithi167 conmon[60316]: debug 2022-01-31T20:46:33.203+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.205460+0000) 2022-01-31T20:46:33.485 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[35325]: debug 2022-01-31T20:46:33.211+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77329 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:33.485 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[41853]: debug 2022-01-31T20:46:33.203+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.204830+0000) 2022-01-31T20:46:33.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[46715]: debug 2022-01-31T20:46:33.204+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.206361+0000) 2022-01-31T20:46:33.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[46715]: debug 2022-01-31T20:46:33.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.278422+0000) 2022-01-31T20:46:33.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[51620]: debug 2022-01-31T20:46:33.202+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.204589+0000) 2022-01-31T20:46:33.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[51620]: debug 2022-01-31T20:46:33.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.357579+0000) 2022-01-31T20:46:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:33 smithi171 conmon[41853]: debug 2022-01-31T20:46:33.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.486496+0000) 2022-01-31T20:46:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:33 smithi167 conmon[49112]: debug 2022-01-31T20:46:33.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:33.580169+0000) 2022-01-31T20:46:34.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:34 smithi167 conmon[54076]: debug 2022-01-31T20:46:34.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:34.250837+0000) 2022-01-31T20:46:34.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:34 smithi167 conmon[60316]: debug 2022-01-31T20:46:34.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:34.002291+0000) 2022-01-31T20:46:34.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:34 smithi171 conmon[46715]: debug 2022-01-31T20:46:34.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:34.278577+0000) 2022-01-31T20:46:34.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:34 smithi171 conmon[51620]: debug 2022-01-31T20:46:34.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:34.357736+0000) 2022-01-31T20:46:34.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:34 smithi171 conmon[41853]: debug 2022-01-31T20:46:34.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:34.486632+0000) 2022-01-31T20:46:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:34 smithi167 conmon[49112]: debug 2022-01-31T20:46:34.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:34.580334+0000) 2022-01-31T20:46:35.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:35 smithi167 conmon[54076]: debug 2022-01-31T20:46:35.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:35.251048+0000) 2022-01-31T20:46:35.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:35 smithi167 conmon[60316]: debug 2022-01-31T20:46:35.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:35.002482+0000) 2022-01-31T20:46:35.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:35 smithi171 conmon[46715]: debug 2022-01-31T20:46:35.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:35.278793+0000) 2022-01-31T20:46:35.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:35 smithi171 conmon[51620]: debug 2022-01-31T20:46:35.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:35.357934+0000) 2022-01-31T20:46:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:35 smithi171 conmon[41853]: debug 2022-01-31T20:46:35.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:35.486860+0000) 2022-01-31T20:46:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:35 smithi167 conmon[49112]: debug 2022-01-31T20:46:35.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:35.580555+0000) 2022-01-31T20:46:36.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:36 smithi167 conmon[54076]: debug 2022-01-31T20:46:36.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:36.251243+0000) 2022-01-31T20:46:36.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:36 smithi167 conmon[60316]: debug 2022-01-31T20:46:36.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:36.002685+0000) 2022-01-31T20:46:36.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:36 smithi171 conmon[46715]: debug 2022-01-31T20:46:36.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:36.278999+0000) 2022-01-31T20:46:36.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:36 smithi171 conmon[51620]: debug 2022-01-31T20:46:36.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:36.358100+0000) 2022-01-31T20:46:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:36 smithi171 conmon[41853]: debug 2022-01-31T20:46:36.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:36.487031+0000) 2022-01-31T20:46:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:36 smithi167 conmon[49112]: debug 2022-01-31T20:46:36.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:36.580676+0000) 2022-01-31T20:46:37.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:37 smithi167 conmon[54076]: debug 2022-01-31T20:46:37.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:37.251458+0000) 2022-01-31T20:46:37.294 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:37 smithi167 conmon[60316]: debug 2022-01-31T20:46:37.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:37.002801+0000) 2022-01-31T20:46:37.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:37 smithi171 conmon[46715]: debug 2022-01-31T20:46:37.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:37.279191+0000) 2022-01-31T20:46:37.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:37 smithi171 conmon[51620]: debug 2022-01-31T20:46:37.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:37.358234+0000) 2022-01-31T20:46:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:37 smithi171 conmon[41853]: debug 2022-01-31T20:46:37.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:37.487192+0000) 2022-01-31T20:46:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:37 smithi167 conmon[49112]: debug 2022-01-31T20:46:37.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:37.580790+0000) 2022-01-31T20:46:38.293 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[49112]: debug 2022-01-31T20:46:38.214+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.215759+0000) 2022-01-31T20:46:38.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[54076]: debug 2022-01-31T20:46:38.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[54076]: 2022-01-31T20:46:38.214+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.215527+0000) 2022-01-31T20:46:38.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[54076]: debug 2022-01-31T20:46:38.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.251609+0000) 2022-01-31T20:46:38.294 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[60316]: debug 2022-01-31T20:46:38.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.002920+0000) 2022-01-31T20:46:38.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[60316]: debug 2022-01-31T20:46:38.213+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.215470+0000) 2022-01-31T20:46:38.485 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[35325]: debug 2022-01-31T20:46:38.221+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77440 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:38.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[41853]: debug 2022-01-31T20:46:38.214+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.215761+0000) 2022-01-31T20:46:38.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[46715]: debug 2022-01-31T20:46:38.214+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.216543+0000) 2022-01-31T20:46:38.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[46715]: debug 2022-01-31T20:46:38.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.279428+0000) 2022-01-31T20:46:38.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[51620]: debug 2022-01-31T20:46:38.214+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.216311+0000) 2022-01-31T20:46:38.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[51620]: debug 2022-01-31T20:46:38.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.358412+0000) 2022-01-31T20:46:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[41853]: debug 2022-01-31T20:46:38.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.487344+0000) 2022-01-31T20:46:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:38 smithi171 conmon[35325]: debug 2022-01-31T20:46:38.569+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:46:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:38 smithi167 conmon[49112]: debug 2022-01-31T20:46:38.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:38.580928+0000) 2022-01-31T20:46:39.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:39 smithi167 conmon[54076]: debug 2022-01-31T20:46:39.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:39.251747+0000) 2022-01-31T20:46:39.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:39 smithi167 conmon[60316]: debug 2022-01-31T20:46:39.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:39.003112+0000) 2022-01-31T20:46:39.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:39 smithi171 conmon[46715]: debug 2022-01-31T20:46:39.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:39.279608+0000) 2022-01-31T20:46:39.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:39 smithi171 conmon[51620]: debug 2022-01-31T20:46:39.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:39.358590+0000) 2022-01-31T20:46:39.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:39 smithi171 conmon[41853]: debug 2022-01-31T20:46:39.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:39.487512+0000) 2022-01-31T20:46:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:39 smithi167 conmon[49112]: debug 2022-01-31T20:46:39.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:39.581084+0000) 2022-01-31T20:46:40.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:40 smithi167 conmon[54076]: debug 2022-01-31T20:46:40.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:40.251920+0000) 2022-01-31T20:46:40.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:40 smithi167 conmon[60316]: debug 2022-01-31T20:46:40.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:40.003282+0000) 2022-01-31T20:46:40.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:40 smithi171 conmon[46715]: debug 2022-01-31T20:46:40.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:40.279793+0000) 2022-01-31T20:46:40.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:40 smithi171 conmon[51620]: debug 2022-01-31T20:46:40.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:40.358773+0000) 2022-01-31T20:46:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:40 smithi171 conmon[41853]: debug 2022-01-31T20:46:40.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:40.487691+0000) 2022-01-31T20:46:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:40 smithi167 conmon[49112]: debug 2022-01-31T20:46:40.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:40.581278+0000) 2022-01-31T20:46:41.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:41 smithi167 conmon[54076]: debug 2022-01-31T20:46:41.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:41.252091+0000) 2022-01-31T20:46:41.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:41 smithi167 conmon[60316]: debug 2022-01-31T20:46:41.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:41.003432+0000) 2022-01-31T20:46:41.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:41 smithi171 conmon[46715]: debug 2022-01-31T20:46:41.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:41.279986+0000) 2022-01-31T20:46:41.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:41 smithi171 conmon[51620]: debug 2022-01-31T20:46:41.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:41.358954+0000) 2022-01-31T20:46:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:41 smithi171 conmon[41853]: debug 2022-01-31T20:46:41.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:41.487899+0000) 2022-01-31T20:46:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:41 smithi167 conmon[49112]: debug 2022-01-31T20:46:41.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:41.581458+0000) 2022-01-31T20:46:42.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:42 smithi167 conmon[54076]: debug 2022-01-31T20:46:42.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:42.252283+0000) 2022-01-31T20:46:42.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:42 smithi167 conmon[60316]: debug 2022-01-31T20:46:42.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:42.003605+0000) 2022-01-31T20:46:42.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:42 smithi171 conmon[46715]: debug 2022-01-31T20:46:42.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:42.280168+0000) 2022-01-31T20:46:42.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:42 smithi171 conmon[51620]: debug 2022-01-31T20:46:42.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:42.359169+0000) 2022-01-31T20:46:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:42 smithi171 conmon[41853]: debug 2022-01-31T20:46:42.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:42.488052+0000) 2022-01-31T20:46:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:42 smithi167 conmon[49112]: debug 2022-01-31T20:46:42.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:42.581601+0000) 2022-01-31T20:46:43.294 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[49112]: debug 2022-01-31T20:46:43.224+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.225782+0000) 2022-01-31T20:46:43.294 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[49112]: 2022-01-31T20:46:43.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[54076]: debug 2022-01-31T20:46:43.224+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.226107+0000) 2022-01-31T20:46:43.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[54076]: debug 2022-01-31T20:46:43.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.252467+0000) 2022-01-31T20:46:43.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[60316]: debug 2022-01-31T20:46:43.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.003761+0000) 2022-01-31T20:46:43.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[60316]: debug 2022-01-31T20:46:43.225+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.226660+0000) 2022-01-31T20:46:43.487 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[35325]: debug 2022-01-31T20:46:43.233+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77548 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:43.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[41853]: debug 2022-01-31T20:46:43.225+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.226414+0000) 2022-01-31T20:46:43.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[46715]: debug 2022-01-31T20:46:43.226+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.226935+0000) 2022-01-31T20:46:43.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[46715]: debug 2022-01-31T20:46:43.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.280361+0000) 2022-01-31T20:46:43.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[51620]: debug 2022-01-31T20:46:43.225+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.226309+0000) 2022-01-31T20:46:43.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[51620]: debug 2022-01-31T20:46:43.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.359352+0000) 2022-01-31T20:46:43.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:43 smithi171 conmon[41853]: debug 2022-01-31T20:46:43.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.488208+0000) 2022-01-31T20:46:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:43 smithi167 conmon[49112]: debug 2022-01-31T20:46:43.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:43.581736+0000) 2022-01-31T20:46:44.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:44 smithi167 conmon[54076]: debug 2022-01-31T20:46:44.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:44.252633+0000) 2022-01-31T20:46:44.294 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:44 smithi167 conmon[60316]: debug 2022-01-31T20:46:44.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:44.003937+0000) 2022-01-31T20:46:44.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:44 smithi171 conmon[46715]: debug 2022-01-31T20:46:44.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:44.280507+0000) 2022-01-31T20:46:44.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:44 smithi171 conmon[51620]: debug 2022-01-31T20:46:44.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:44.359515+0000) 2022-01-31T20:46:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:44 smithi171 conmon[41853]: debug 2022-01-31T20:46:44.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:44.488418+0000) 2022-01-31T20:46:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:44 smithi167 conmon[49112]: debug 2022-01-31T20:46:44.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:44.581856+0000) 2022-01-31T20:46:45.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:45 smithi167 conmon[54076]: debug 2022-01-31T20:46:45.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:45.252829+0000) 2022-01-31T20:46:45.294 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:45 smithi167 conmon[60316]: debug 2022-01-31T20:46:45.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:45.004027+0000) 2022-01-31T20:46:45.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:45 smithi171 conmon[46715]: debug 2022-01-31T20:46:45.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:45.280669+0000) 2022-01-31T20:46:45.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:45 smithi171 conmon[51620]: debug 2022-01-31T20:46:45.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:45.359673+0000) 2022-01-31T20:46:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:45 smithi171 conmon[41853]: debug 2022-01-31T20:46:45.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:45.488570+0000) 2022-01-31T20:46:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:45 smithi167 conmon[49112]: debug 2022-01-31T20:46:45.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:45.582055+0000) 2022-01-31T20:46:46.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:46 smithi167 conmon[54076]: debug 2022-01-31T20:46:46.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:46.253013+0000) 2022-01-31T20:46:46.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:46 smithi167 conmon[60316]: debug 2022-01-31T20:46:46.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:46.004182+0000) 2022-01-31T20:46:46.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:46 smithi171 conmon[46715]: debug 2022-01-31T20:46:46.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:46.280843+0000) 2022-01-31T20:46:46.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:46 smithi171 conmon[51620]: debug 2022-01-31T20:46:46.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:46.359824+0000) 2022-01-31T20:46:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:46 smithi171 conmon[41853]: debug 2022-01-31T20:46:46.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:46.488762+0000) 2022-01-31T20:46:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:46 smithi167 conmon[49112]: debug 2022-01-31T20:46:46.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:46.582172+0000) 2022-01-31T20:46:47.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:47 smithi167 conmon[54076]: debug 2022-01-31T20:46:47.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:47.253211+0000) 2022-01-31T20:46:47.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:47 smithi167 conmon[60316]: debug 2022-01-31T20:46:47.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:47.004359+0000) 2022-01-31T20:46:47.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:47 smithi171 conmon[46715]: debug 2022-01-31T20:46:47.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:47.281006+0000) 2022-01-31T20:46:47.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:47 smithi171 conmon[51620]: debug 2022-01-31T20:46:47.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:47.360002+0000) 2022-01-31T20:46:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:47 smithi171 conmon[41853]: debug 2022-01-31T20:46:47.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:47.488915+0000) 2022-01-31T20:46:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:47 smithi167 conmon[49112]: debug 2022-01-31T20:46:47.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:47.582379+0000) 2022-01-31T20:46:48.294 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:48 smithi167 conmon[49112]: debug 2022-01-31T20:46:48.236+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.237885+0000) 2022-01-31T20:46:48.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:48 smithi167 conmon[54076]: debug 2022-01-31T20:46:48.235+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.237000+0000) 2022-01-31T20:46:48.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:48 smithi167 conmon[54076]: debug 2022-01-31T20:46:48.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.253361+0000) 2022-01-31T20:46:48.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:48 smithi167 conmon[60316]: debug 2022-01-31T20:46:48.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.004573+0000) 2022-01-31T20:46:48.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:48 smithi167 conmon[60316]: debug 2022-01-31T20:46:48.236+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.238127+0000) 2022-01-31T20:46:48.487 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[35325]: debug 2022-01-31T20:46:48.244+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77660 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:48.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[41853]: debug 2022-01-31T20:46:48.236+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.237055+0000) 2022-01-31T20:46:48.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[46715]: debug 2022-01-31T20:46:48.236+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.237104+0000) 2022-01-31T20:46:48.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[46715]: debug 2022-01-31T20:46:48.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.281167+0000) 2022-01-31T20:46:48.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[51620]: debug 2022-01-31T20:46:48.236+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.237283+0000) 2022-01-31T20:46:48.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[51620]: debug 2022-01-31T20:46:48.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.360214+0000) 2022-01-31T20:46:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:48 smithi171 conmon[41853]: debug 2022-01-31T20:46:48.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.489074+0000) 2022-01-31T20:46:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:48 smithi167 conmon[49112]: debug 2022-01-31T20:46:48.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:48.582575+0000) 2022-01-31T20:46:49.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:49 smithi167 conmon[54076]: debug 2022-01-31T20:46:49.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:49.253530+0000) 2022-01-31T20:46:49.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:49 smithi167 conmon[60316]: debug 2022-01-31T20:46:49.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:49.004771+0000) 2022-01-31T20:46:49.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:49 smithi171 conmon[46715]: debug 2022-01-31T20:46:49.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:49.281349+0000) 2022-01-31T20:46:49.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:49 smithi171 conmon[51620]: debug 2022-01-31T20:46:49.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:49.360384+0000) 2022-01-31T20:46:49.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:49 smithi171 conmon[41853]: debug 2022-01-31T20:46:49.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:49.489254+0000) 2022-01-31T20:46:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:49 smithi167 conmon[49112]: debug 2022-01-31T20:46:49.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:49.582696+0000) 2022-01-31T20:46:50.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:50 smithi167 conmon[54076]: debug 2022-01-31T20:46:50.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:50.253643+0000) 2022-01-31T20:46:50.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:50 smithi167 conmon[60316]: debug 2022-01-31T20:46:50.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:50.004957+0000) 2022-01-31T20:46:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:50 smithi171 conmon[41853]: debug 2022-01-31T20:46:50.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:50.489414+0000) 2022-01-31T20:46:50.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:50 smithi171 conmon[46715]: debug 2022-01-31T20:46:50.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:50.281555+0000) 2022-01-31T20:46:50.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:50 smithi171 conmon[51620]: debug 2022-01-31T20:46:50.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:50.360574+0000) 2022-01-31T20:46:50.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:50 smithi167 conmon[49112]: debug 2022-01-31T20:46:50.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:50.582848+0000) 2022-01-31T20:46:51.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:51 smithi167 conmon[54076]: debug 2022-01-31T20:46:51.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:51.253803+0000) 2022-01-31T20:46:51.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:51 smithi167 conmon[60316]: debug 2022-01-31T20:46:51.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:51.005085+0000) 2022-01-31T20:46:51.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:51 smithi171 conmon[46715]: debug 2022-01-31T20:46:51.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:51.281779+0000) 2022-01-31T20:46:51.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:51 smithi171 conmon[51620]: debug 2022-01-31T20:46:51.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:51.360672+0000) 2022-01-31T20:46:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:51 smithi171 conmon[41853]: debug 2022-01-31T20:46:51.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:51.489645+0000) 2022-01-31T20:46:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:51 smithi167 conmon[49112]: debug 2022-01-31T20:46:51.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:51.583060+0000) 2022-01-31T20:46:52.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:52 smithi167 conmon[54076]: debug 2022-01-31T20:46:52.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:52.253952+0000) 2022-01-31T20:46:52.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:52 smithi167 conmon[60316]: debug 2022-01-31T20:46:52.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:52.005252+0000) 2022-01-31T20:46:52.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:52 smithi171 conmon[46715]: debug 2022-01-31T20:46:52.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:52.281969+0000) 2022-01-31T20:46:52.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:52 smithi171 conmon[51620]: debug 2022-01-31T20:46:52.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:52.360848+0000) 2022-01-31T20:46:52.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:52 smithi171 conmon[41853]: debug 2022-01-31T20:46:52.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:52.489848+0000) 2022-01-31T20:46:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:52 smithi167 conmon[49112]: debug 2022-01-31T20:46:52.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:52.583267+0000) 2022-01-31T20:46:53.296 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:53 smithi167 conmon[49112]: debug 2022-01-31T20:46:53.245+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.247299+0000) 2022-01-31T20:46:53.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:53 smithi167 conmon[54076]: debug 2022-01-31T20:46:53.245+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.247176+0000) 2022-01-31T20:46:53.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:53 smithi167 conmon[54076]: debug 2022-01-31T20:46:53.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.254108+0000) 2022-01-31T20:46:53.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:53 smithi167 conmon[60316]: debug 2022-01-31T20:46:53.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.005434+0000) 2022-01-31T20:46:53.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:53 smithi167 conmon[60316]: debug 2022-01-31T20:46:53.246+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.248286+0000) 2022-01-31T20:46:53.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[41853]: debug 2022-01-31T20:46:53.246+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.247609+0000) 2022-01-31T20:46:53.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[51620]: debug 2022-01-31T20:46:53.246+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.247465+0000) 2022-01-31T20:46:53.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[51620]: debug 2022-01-31T20:46:53.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.361023+0000) 2022-01-31T20:46:53.489 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[35325]: debug 2022-01-31T20:46:53.254+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77768 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:53.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[46715]: debug 2022-01-31T20:46:53.246+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.247021+0000) 2022-01-31T20:46:53.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[46715]: debug 2022-01-31T20:46:53.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.282099+0000) 2022-01-31T20:46:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[35325]: debug 2022-01-31T20:46:53.570+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:46:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:53 smithi171 conmon[41853]: debug 2022-01-31T20:46:53.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.490038+0000) 2022-01-31T20:46:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:53 smithi167 conmon[49112]: debug 2022-01-31T20:46:53.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:53.583495+0000) 2022-01-31T20:46:54.295 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:54 smithi167 conmon[54076]: debug 2022-01-31T20:46:54.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:54.254259+0000) 2022-01-31T20:46:54.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:54 smithi167 conmon[60316]: debug 2022-01-31T20:46:54.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:54.005598+0000) 2022-01-31T20:46:54.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:54 smithi171 conmon[46715]: debug 2022-01-31T20:46:54.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:54.282200+0000) 2022-01-31T20:46:54.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:54 smithi171 conmon[51620]: debug 2022-01-31T20:46:54.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:54.361160+0000) 2022-01-31T20:46:54.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:54 smithi171 conmon[41853]: debug 2022-01-31T20:46:54.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:54.490155+0000) 2022-01-31T20:46:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:54 smithi167 conmon[49112]: debug 2022-01-31T20:46:54.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:54.583610+0000) 2022-01-31T20:46:55.296 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:55 smithi167 conmon[54076]: debug 2022-01-31T20:46:55.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:55.254457+0000) 2022-01-31T20:46:55.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:55 smithi167 conmon[60316]: debug 2022-01-31T20:46:55.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:55.005699+0000) 2022-01-31T20:46:55.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:55 smithi171 conmon[46715]: debug 2022-01-31T20:46:55.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:55.282377+0000) 2022-01-31T20:46:55.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:55 smithi171 conmon[51620]: debug 2022-01-31T20:46:55.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:55.361347+0000) 2022-01-31T20:46:55.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:55 smithi171 conmon[41853]: debug 2022-01-31T20:46:55.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:55.490334+0000) 2022-01-31T20:46:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:55 smithi167 conmon[49112]: debug 2022-01-31T20:46:55.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:55.583750+0000) 2022-01-31T20:46:56.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:56 smithi167 conmon[60316]: debug 2022-01-31T20:46:56.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:56.005873+0000) 2022-01-31T20:46:56.296 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:56 smithi167 conmon[54076]: debug 2022-01-31T20:46:56.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:56.254636+0000) 2022-01-31T20:46:56.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:56 smithi171 conmon[46715]: debug 2022-01-31T20:46:56.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:56.282569+0000) 2022-01-31T20:46:56.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:56 smithi171 conmon[51620]: debug 2022-01-31T20:46:56.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:56.361529+0000) 2022-01-31T20:46:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:56 smithi171 conmon[41853]: debug 2022-01-31T20:46:56.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:56.490514+0000) 2022-01-31T20:46:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:56 smithi167 conmon[49112]: debug 2022-01-31T20:46:56.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:56.583959+0000) 2022-01-31T20:46:57.296 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:57 smithi167 conmon[54076]: debug 2022-01-31T20:46:57.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:57.254821+0000) 2022-01-31T20:46:57.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:57 smithi167 conmon[60316]: debug 2022-01-31T20:46:57.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:57.006080+0000) 2022-01-31T20:46:57.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:57 smithi171 conmon[46715]: debug 2022-01-31T20:46:57.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:57.282767+0000) 2022-01-31T20:46:57.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:57 smithi171 conmon[51620]: debug 2022-01-31T20:46:57.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:57.361682+0000) 2022-01-31T20:46:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:57 smithi171 conmon[41853]: debug 2022-01-31T20:46:57.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:57.490692+0000) 2022-01-31T20:46:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:57 smithi167 conmon[49112]: debug 2022-01-31T20:46:57.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:57.584135+0000) 2022-01-31T20:46:58.254 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:58 smithi167 conmon[54076]: debug 2022-01-31T20:46:58.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.255043+0000) 2022-01-31T20:46:58.256 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:58 smithi167 conmon[60316]: debug 2022-01-31T20:46:58.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.006270+0000) 2022-01-31T20:46:58.489 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[35325]: debug 2022-01-31T20:46:58.264+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77879 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:46:58.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[41853]: debug 2022-01-31T20:46:58.257+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.257982+0000) 2022-01-31T20:46:58.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[46715]: debug 2022-01-31T20:46:58.257+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.258254+0000) 2022-01-31T20:46:58.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[46715]: debug 2022-01-31T20:46:58.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.282909+0000) 2022-01-31T20:46:58.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[51620]: debug 2022-01-31T20:46:58.256+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.257084+0000) 2022-01-31T20:46:58.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[51620]: debug 2022-01-31T20:46:58.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.361810+0000) 2022-01-31T20:46:58.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:58 smithi167 conmon[54076]: debug 2022-01-31T20:46:58.256+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.256598+0000) 2022-01-31T20:46:58.573 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:58 smithi167 conmon[49112]: debug 2022-01-31T20:46:58.257+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.257846+0000) 2022-01-31T20:46:58.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:58 smithi167 conmon[60316]: debug 2022-01-31T20:46:58.257+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.257893+0000) 2022-01-31T20:46:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:58 smithi171 conmon[41853]: debug 2022-01-31T20:46:58.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.490860+0000) 2022-01-31T20:46:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:58 smithi167 conmon[49112]: debug 2022-01-31T20:46:58.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:58.584339+0000) 2022-01-31T20:46:59.296 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:46:59 smithi167 conmon[54076]: debug 2022-01-31T20:46:59.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:59.255247+0000) 2022-01-31T20:46:59.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:46:59 smithi167 conmon[60316]: debug 2022-01-31T20:46:59.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:59.006423+0000) 2022-01-31T20:46:59.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:46:59 smithi171 conmon[46715]: debug 2022-01-31T20:46:59.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:59.283056+0000) 2022-01-31T20:46:59.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:46:59 smithi171 conmon[51620]: debug 2022-01-31T20:46:59.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:59.361958+0000) 2022-01-31T20:46:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:46:59 smithi171 conmon[41853]: debug 2022-01-31T20:46:59.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:59.490958+0000) 2022-01-31T20:46:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:46:59 smithi167 conmon[49112]: debug 2022-01-31T20:46:59.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:46:59.584540+0000) 2022-01-31T20:47:00.296 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:00 smithi167 conmon[54076]: debug 2022-01-31T20:47:00.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:00.255402+0000) 2022-01-31T20:47:00.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:00 smithi167 conmon[60316]: debug 2022-01-31T20:47:00.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:00.006600+0000) 2022-01-31T20:47:00.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:00 smithi171 conmon[46715]: debug 2022-01-31T20:47:00.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:00.283178+0000) 2022-01-31T20:47:00.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:00 smithi171 conmon[51620]: debug 2022-01-31T20:47:00.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:00.362120+0000) 2022-01-31T20:47:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:00 smithi171 conmon[41853]: debug 2022-01-31T20:47:00.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:00.491085+0000) 2022-01-31T20:47:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:00 smithi167 conmon[49112]: debug 2022-01-31T20:47:00.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:00.584688+0000) 2022-01-31T20:47:01.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:01 smithi167 conmon[60316]: debug 2022-01-31T20:47:01.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:01.006764+0000) 2022-01-31T20:47:01.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:01 smithi167 conmon[54076]: debug 2022-01-31T20:47:01.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:01.255554+0000) 2022-01-31T20:47:01.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:01 smithi171 conmon[46715]: debug 2022-01-31T20:47:01.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:01.283354+0000) 2022-01-31T20:47:01.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:01 smithi171 conmon[51620]: debug 2022-01-31T20:47:01.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:01.362307+0000) 2022-01-31T20:47:01.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:01 smithi171 conmon[41853]: debug 2022-01-31T20:47:01.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:01.491240+0000) 2022-01-31T20:47:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:01 smithi167 conmon[49112]: debug 2022-01-31T20:47:01.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:01.584852+0000) 2022-01-31T20:47:02.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:02 smithi167 conmon[60316]: debug 2022-01-31T20:47:02.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:02.006983+0000) 2022-01-31T20:47:02.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:02 smithi167 conmon[54076]: debug 2022-01-31T20:47:02.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:02.255694+0000) 2022-01-31T20:47:02.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:02 smithi171 conmon[46715]: debug 2022-01-31T20:47:02.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:02.283521+0000) 2022-01-31T20:47:02.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:02 smithi171 conmon[51620]: debug 2022-01-31T20:47:02.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:02.362473+0000) 2022-01-31T20:47:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:02 smithi171 conmon[41853]: debug 2022-01-31T20:47:02.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:02.491412+0000) 2022-01-31T20:47:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:02 smithi167 conmon[49112]: debug 2022-01-31T20:47:02.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:02.584984+0000) 2022-01-31T20:47:03.265 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[54076]: debug 2022-01-31T20:47:03.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.255866+0000) 2022-01-31T20:47:03.265 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[60316]: debug 2022-01-31T20:47:03.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.007136+0000) 2022-01-31T20:47:03.490 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[35325]: debug 2022-01-31T20:47:03.274+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 77987 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:03.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[41853]: debug 2022-01-31T20:47:03.267+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.268466+0000) 2022-01-31T20:47:03.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[46715]: debug 2022-01-31T20:47:03.268+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.269000+0000) 2022-01-31T20:47:03.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[46715]: debug 2022-01-31T20:47:03.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.283651+0000) 2022-01-31T20:47:03.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[51620]: debug 2022-01-31T20:47:03.266+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.267106+0000) 2022-01-31T20:47:03.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[51620]: debug 2022-01-31T20:47:03.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.362664+0000) 2022-01-31T20:47:03.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[54076]: debug 2022-01-31T20:47:03.266+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.267310+0000) 2022-01-31T20:47:03.574 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[49112]: debug 2022-01-31T20:47:03.574 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[49112]: 2022-01-31T20:47:03.267+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.268229+0000) 2022-01-31T20:47:03.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[60316]: debug 2022-01-31T20:47:03.267+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.268118+0000) 2022-01-31T20:47:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:03 smithi171 conmon[41853]: debug 2022-01-31T20:47:03.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.491567+0000) 2022-01-31T20:47:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:03 smithi167 conmon[49112]: debug 2022-01-31T20:47:03.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:03.585168+0000) 2022-01-31T20:47:04.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:04 smithi167 conmon[54076]: debug 2022-01-31T20:47:04.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:04.256066+0000) 2022-01-31T20:47:04.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:04 smithi167 conmon[60316]: debug 2022-01-31T20:47:04.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:04.007329+0000) 2022-01-31T20:47:04.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:04 smithi171 conmon[46715]: debug 2022-01-31T20:47:04.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:04.283796+0000) 2022-01-31T20:47:04.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:04 smithi171 conmon[51620]: debug 2022-01-31T20:47:04.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:04.362829+0000) 2022-01-31T20:47:04.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:04 smithi171 conmon[41853]: debug 2022-01-31T20:47:04.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:04.491706+0000) 2022-01-31T20:47:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:04 smithi167 conmon[49112]: debug 2022-01-31T20:47:04.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:04.585266+0000) 2022-01-31T20:47:05.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:05 smithi167 conmon[60316]: debug 2022-01-31T20:47:05.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:05.007431+0000) 2022-01-31T20:47:05.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:05 smithi167 conmon[54076]: debug 2022-01-31T20:47:05.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:05.256221+0000) 2022-01-31T20:47:05.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:05 smithi171 conmon[46715]: debug 2022-01-31T20:47:05.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:05.284027+0000) 2022-01-31T20:47:05.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:05 smithi171 conmon[51620]: debug 2022-01-31T20:47:05.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:05.363019+0000) 2022-01-31T20:47:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:05 smithi171 conmon[41853]: debug 2022-01-31T20:47:05.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:05.491935+0000) 2022-01-31T20:47:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:05 smithi167 conmon[49112]: debug 2022-01-31T20:47:05.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:05.585428+0000) 2022-01-31T20:47:06.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:06 smithi167 conmon[54076]: debug 2022-01-31T20:47:06.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:06.256397+0000) 2022-01-31T20:47:06.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:06 smithi167 conmon[60316]: debug 2022-01-31T20:47:06.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:06.007608+0000) 2022-01-31T20:47:06.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:06 smithi171 conmon[46715]: debug 2022-01-31T20:47:06.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:06.284236+0000) 2022-01-31T20:47:06.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:06 smithi171 conmon[51620]: debug 2022-01-31T20:47:06.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:06.363174+0000) 2022-01-31T20:47:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:06 smithi171 conmon[41853]: debug 2022-01-31T20:47:06.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:06.492122+0000) 2022-01-31T20:47:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:06 smithi167 conmon[49112]: debug 2022-01-31T20:47:06.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:06.585590+0000) 2022-01-31T20:47:07.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:07 smithi167 conmon[60316]: debug 2022-01-31T20:47:07.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:07.007789+0000) 2022-01-31T20:47:07.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:07 smithi167 conmon[54076]: debug 2022-01-31T20:47:07.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:07.256545+0000) 2022-01-31T20:47:07.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:07 smithi171 conmon[46715]: debug 2022-01-31T20:47:07.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:07.284450+0000) 2022-01-31T20:47:07.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:07 smithi171 conmon[51620]: debug 2022-01-31T20:47:07.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:07.363345+0000) 2022-01-31T20:47:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:07 smithi171 conmon[41853]: debug 2022-01-31T20:47:07.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:07.492323+0000) 2022-01-31T20:47:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:07 smithi167 conmon[49112]: debug 2022-01-31T20:47:07.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:07.585717+0000) 2022-01-31T20:47:08.276 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:08 smithi167 conmon[60316]: debug 2022-01-31T20:47:08.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.007949+0000) 2022-01-31T20:47:08.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:08 smithi167 conmon[54076]: debug 2022-01-31T20:47:08.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.256656+0000) 2022-01-31T20:47:08.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[41853]: debug 2022-01-31T20:47:08.277+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.278056+0000) 2022-01-31T20:47:08.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[46715]: debug 2022-01-31T20:47:08.278+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.279631+0000) 2022-01-31T20:47:08.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[46715]: debug 2022-01-31T20:47:08.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.284638+0000) 2022-01-31T20:47:08.493 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[35325]: debug 2022-01-31T20:47:08.285+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78098 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:08.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[51620]: debug 2022-01-31T20:47:08.276+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.277654+0000) 2022-01-31T20:47:08.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[51620]: debug 2022-01-31T20:47:08.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.363567+0000) 2022-01-31T20:47:08.574 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:08 smithi167 conmon[49112]: debug 2022-01-31T20:47:08.278+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.278620+0000) 2022-01-31T20:47:08.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:08 smithi167 conmon[54076]: debug 2022-01-31T20:47:08.278+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.279019+0000) 2022-01-31T20:47:08.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:08 smithi167 conmon[60316]: debug 2022-01-31T20:47:08.278+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.278539+0000) 2022-01-31T20:47:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[35325]: debug 2022-01-31T20:47:08.571+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:47:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:08 smithi171 conmon[41853]: debug 2022-01-31T20:47:08.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.492524+0000) 2022-01-31T20:47:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:08 smithi167 conmon[49112]: debug 2022-01-31T20:47:08.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:08.585892+0000) 2022-01-31T20:47:09.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:09 smithi167 conmon[54076]: debug 2022-01-31T20:47:09.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:09.256809+0000) 2022-01-31T20:47:09.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:09 smithi167 conmon[60316]: debug 2022-01-31T20:47:09.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:09.008078+0000) 2022-01-31T20:47:09.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:09 smithi171 conmon[46715]: debug 2022-01-31T20:47:09.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:09.284812+0000) 2022-01-31T20:47:09.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:09 smithi171 conmon[51620]: debug 2022-01-31T20:47:09.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:09.363736+0000) 2022-01-31T20:47:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:09 smithi171 conmon[41853]: debug 2022-01-31T20:47:09.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:09.492675+0000) 2022-01-31T20:47:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:09 smithi167 conmon[49112]: debug 2022-01-31T20:47:09.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:09.586067+0000) 2022-01-31T20:47:10.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:10 smithi167 conmon[54076]: debug 2022-01-31T20:47:10.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:10.256946+0000) 2022-01-31T20:47:10.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:10 smithi167 conmon[60316]: debug 2022-01-31T20:47:10.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:10.008228+0000) 2022-01-31T20:47:10.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:10 smithi171 conmon[46715]: debug 2022-01-31T20:47:10.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:10.284998+0000) 2022-01-31T20:47:10.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:10 smithi171 conmon[51620]: debug 2022-01-31T20:47:10.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:10.363964+0000) 2022-01-31T20:47:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:10 smithi171 conmon[41853]: debug 2022-01-31T20:47:10.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:10.492868+0000) 2022-01-31T20:47:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:10 smithi167 conmon[49112]: debug 2022-01-31T20:47:10.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:10.586270+0000) 2022-01-31T20:47:11.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:11 smithi167 conmon[54076]: debug 2022-01-31T20:47:11.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:11.257076+0000) 2022-01-31T20:47:11.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:11 smithi167 conmon[60316]: debug 2022-01-31T20:47:11.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:11.008384+0000) 2022-01-31T20:47:11.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:11 smithi171 conmon[46715]: debug 2022-01-31T20:47:11.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:11.285200+0000) 2022-01-31T20:47:11.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:11 smithi171 conmon[51620]: debug 2022-01-31T20:47:11.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:11.364150+0000) 2022-01-31T20:47:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:11 smithi171 conmon[41853]: debug 2022-01-31T20:47:11.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:11.493040+0000) 2022-01-31T20:47:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:11 smithi167 conmon[49112]: debug 2022-01-31T20:47:11.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:11.586492+0000) 2022-01-31T20:47:12.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:12 smithi167 conmon[54076]: debug 2022-01-31T20:47:12.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:12.257235+0000) 2022-01-31T20:47:12.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:12 smithi167 conmon[60316]: debug 2022-01-31T20:47:12.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:12.008538+0000) 2022-01-31T20:47:12.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:12 smithi171 conmon[46715]: debug 2022-01-31T20:47:12.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:12.285355+0000) 2022-01-31T20:47:12.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:12 smithi171 conmon[51620]: debug 2022-01-31T20:47:12.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:12.364331+0000) 2022-01-31T20:47:12.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:12 smithi171 conmon[41853]: debug 2022-01-31T20:47:12.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:12.493213+0000) 2022-01-31T20:47:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:12 smithi167 conmon[49112]: debug 2022-01-31T20:47:12.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:12.586656+0000) 2022-01-31T20:47:13.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:13 smithi167 conmon[54076]: debug 2022-01-31T20:47:13.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.257428+0000) 2022-01-31T20:47:13.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:13 smithi167 conmon[60316]: debug 2022-01-31T20:47:13.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.008633+0000) 2022-01-31T20:47:13.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[41853]: debug 2022-01-31T20:47:13.287+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.287807+0000) 2022-01-31T20:47:13.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[51620]: debug 2022-01-31T20:47:13.287+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.288156+0000) 2022-01-31T20:47:13.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[51620]: debug 2022-01-31T20:47:13.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.364513+0000) 2022-01-31T20:47:13.493 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[35325]: debug 2022-01-31T20:47:13.295+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78206 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:13.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[46715]: debug 2022-01-31T20:47:13.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.285542+0000) 2022-01-31T20:47:13.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[46715]: debug 2022-01-31T20:47:13.287+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.288700+0000) 2022-01-31T20:47:13.575 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:13 smithi167 conmon[49112]: debug 2022-01-31T20:47:13.288+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.288810+0000) 2022-01-31T20:47:13.576 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:13 smithi167 conmon[60316]: debug 2022-01-31T20:47:13.287+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.288432+0000) 2022-01-31T20:47:13.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:13 smithi167 conmon[54076]: debug 2022-01-31T20:47:13.288+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.288870+0000) 2022-01-31T20:47:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:13 smithi171 conmon[41853]: debug 2022-01-31T20:47:13.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.493363+0000) 2022-01-31T20:47:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:13 smithi167 conmon[49112]: debug 2022-01-31T20:47:13.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:13.586822+0000) 2022-01-31T20:47:14.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:14 smithi167 conmon[54076]: debug 2022-01-31T20:47:14.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:14.257581+0000) 2022-01-31T20:47:14.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:14 smithi167 conmon[60316]: debug 2022-01-31T20:47:14.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:14.008777+0000) 2022-01-31T20:47:14.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:14 smithi171 conmon[46715]: debug 2022-01-31T20:47:14.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:14.285738+0000) 2022-01-31T20:47:14.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:14 smithi171 conmon[51620]: debug 2022-01-31T20:47:14.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:14.364661+0000) 2022-01-31T20:47:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:14 smithi171 conmon[41853]: debug 2022-01-31T20:47:14.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:14.493516+0000) 2022-01-31T20:47:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:14 smithi167 conmon[49112]: debug 2022-01-31T20:47:14.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:14.586920+0000) 2022-01-31T20:47:15.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:15 smithi167 conmon[60316]: debug 2022-01-31T20:47:15.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:15.008896+0000) 2022-01-31T20:47:15.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:15 smithi167 conmon[54076]: debug 2022-01-31T20:47:15.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:15.257709+0000) 2022-01-31T20:47:15.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:15 smithi171 conmon[46715]: debug 2022-01-31T20:47:15.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:15.285931+0000) 2022-01-31T20:47:15.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:15 smithi171 conmon[51620]: debug 2022-01-31T20:47:15.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:15.364822+0000) 2022-01-31T20:47:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:15 smithi171 conmon[41853]: debug 2022-01-31T20:47:15.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:15.493672+0000) 2022-01-31T20:47:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:15 smithi167 conmon[49112]: debug 2022-01-31T20:47:15.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:15.587082+0000) 2022-01-31T20:47:16.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:16 smithi167 conmon[60316]: debug 2022-01-31T20:47:16.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:16.009053+0000) 2022-01-31T20:47:16.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:16 smithi167 conmon[54076]: debug 2022-01-31T20:47:16.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:16.257945+0000) 2022-01-31T20:47:16.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:16 smithi171 conmon[46715]: debug 2022-01-31T20:47:16.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:16.286108+0000) 2022-01-31T20:47:16.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:16 smithi171 conmon[51620]: debug 2022-01-31T20:47:16.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:16.364995+0000) 2022-01-31T20:47:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:16 smithi171 conmon[41853]: debug 2022-01-31T20:47:16.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:16.493832+0000) 2022-01-31T20:47:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:16 smithi167 conmon[49112]: debug 2022-01-31T20:47:16.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:16.587281+0000) 2022-01-31T20:47:17.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:17 smithi167 conmon[60316]: debug 2022-01-31T20:47:17.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:17.009182+0000) 2022-01-31T20:47:17.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:17 smithi167 conmon[54076]: debug 2022-01-31T20:47:17.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:17.258123+0000) 2022-01-31T20:47:17.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:17 smithi171 conmon[46715]: debug 2022-01-31T20:47:17.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:17.286322+0000) 2022-01-31T20:47:17.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:17 smithi171 conmon[51620]: debug 2022-01-31T20:47:17.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:17.365171+0000) 2022-01-31T20:47:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:17 smithi171 conmon[41853]: debug 2022-01-31T20:47:17.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:17.494031+0000) 2022-01-31T20:47:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:17 smithi167 conmon[49112]: debug 2022-01-31T20:47:17.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:17.587482+0000) 2022-01-31T20:47:18.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:18 smithi167 conmon[60316]: debug 2022-01-31T20:47:18.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.009416+0000) 2022-01-31T20:47:18.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:18 smithi167 conmon[54076]: debug 2022-01-31T20:47:18.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.258340+0000) 2022-01-31T20:47:18.492 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[35325]: debug 2022-01-31T20:47:18.305+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78317 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:18.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[41853]: debug 2022-01-31T20:47:18.295+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.297661+0000) 2022-01-31T20:47:18.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[46715]: debug 2022-01-31T20:47:18.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.286582+0000) 2022-01-31T20:47:18.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[46715]: debug 2022-01-31T20:47:18.296+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.297988+0000) 2022-01-31T20:47:18.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[51620]: debug 2022-01-31T20:47:18.296+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.298617+0000) 2022-01-31T20:47:18.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[51620]: debug 2022-01-31T20:47:18.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.365331+0000) 2022-01-31T20:47:18.576 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:18 smithi167 conmon[49112]: debug 2022-01-31T20:47:18.298+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.299265+0000) 2022-01-31T20:47:18.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:18 smithi167 conmon[54076]: debug 2022-01-31T20:47:18.299+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.299732+0000) 2022-01-31T20:47:18.577 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:18 smithi167 conmon[60316]: debug 2022-01-31T20:47:18.297+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.298405+0000) 2022-01-31T20:47:18.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:18 smithi171 conmon[41853]: debug 2022-01-31T20:47:18.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.494205+0000) 2022-01-31T20:47:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:18 smithi167 conmon[49112]: debug 2022-01-31T20:47:18.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:18.587649+0000) 2022-01-31T20:47:19.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:19 smithi167 conmon[54076]: debug 2022-01-31T20:47:19.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:19.258484+0000) 2022-01-31T20:47:19.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:19 smithi167 conmon[60316]: debug 2022-01-31T20:47:19.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:19.009591+0000) 2022-01-31T20:47:19.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:19 smithi171 conmon[46715]: debug 2022-01-31T20:47:19.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:19.286745+0000) 2022-01-31T20:47:19.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:19 smithi171 conmon[51620]: debug 2022-01-31T20:47:19.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:19.365470+0000) 2022-01-31T20:47:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:19 smithi171 conmon[41853]: debug 2022-01-31T20:47:19.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:19.494356+0000) 2022-01-31T20:47:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:19 smithi167 conmon[49112]: debug 2022-01-31T20:47:19.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:19.587781+0000) 2022-01-31T20:47:20.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:20 smithi167 conmon[54076]: debug 2022-01-31T20:47:20.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:20.258588+0000) 2022-01-31T20:47:20.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:20 smithi167 conmon[60316]: debug 2022-01-31T20:47:20.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:20.009786+0000) 2022-01-31T20:47:20.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:20 smithi171 conmon[46715]: debug 2022-01-31T20:47:20.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:20.286890+0000) 2022-01-31T20:47:20.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:20 smithi171 conmon[51620]: debug 2022-01-31T20:47:20.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:20.365653+0000) 2022-01-31T20:47:20.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:20 smithi171 conmon[41853]: debug 2022-01-31T20:47:20.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:20.494541+0000) 2022-01-31T20:47:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:20 smithi167 conmon[49112]: debug 2022-01-31T20:47:20.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:20.587929+0000) 2022-01-31T20:47:21.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:21 smithi167 conmon[54076]: debug 2022-01-31T20:47:21.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:21.258817+0000) 2022-01-31T20:47:21.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:21 smithi167 conmon[60316]: debug 2022-01-31T20:47:21.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:21.009981+0000) 2022-01-31T20:47:21.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:21 smithi171 conmon[46715]: debug 2022-01-31T20:47:21.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:21.287077+0000) 2022-01-31T20:47:21.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:21 smithi171 conmon[51620]: debug 2022-01-31T20:47:21.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:21.365850+0000) 2022-01-31T20:47:21.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:21 smithi171 conmon[41853]: debug 2022-01-31T20:47:21.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:21.494704+0000) 2022-01-31T20:47:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:21 smithi167 conmon[49112]: debug 2022-01-31T20:47:21.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:21.588069+0000) 2022-01-31T20:47:22.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:22 smithi167 conmon[54076]: debug 2022-01-31T20:47:22.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:22.258989+0000) 2022-01-31T20:47:22.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:22 smithi167 conmon[60316]: debug 2022-01-31T20:47:22.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:22.010183+0000) 2022-01-31T20:47:22.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:22 smithi171 conmon[46715]: debug 2022-01-31T20:47:22.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:22.287196+0000) 2022-01-31T20:47:22.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:22 smithi171 conmon[51620]: debug 2022-01-31T20:47:22.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:22.366004+0000) 2022-01-31T20:47:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:22 smithi171 conmon[41853]: debug 2022-01-31T20:47:22.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:22.494883+0000) 2022-01-31T20:47:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:22 smithi167 conmon[49112]: debug 2022-01-31T20:47:22.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:22.588248+0000) 2022-01-31T20:47:23.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:23 smithi167 conmon[54076]: debug 2022-01-31T20:47:23.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.259196+0000) 2022-01-31T20:47:23.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:23 smithi167 conmon[60316]: debug 2022-01-31T20:47:23.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.010339+0000) 2022-01-31T20:47:23.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[41853]: debug 2022-01-31T20:47:23.308+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.310373+0000) 2022-01-31T20:47:23.494 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[35325]: debug 2022-01-31T20:47:23.332+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78425 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:23.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[46715]: debug 2022-01-31T20:47:23.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.287374+0000) 2022-01-31T20:47:23.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[46715]: debug 2022-01-31T20:47:23.307+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.309144+0000) 2022-01-31T20:47:23.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[51620]: debug 2022-01-31T20:47:23.308+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.310403+0000) 2022-01-31T20:47:23.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[51620]: debug 2022-01-31T20:47:23.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.366187+0000) 2022-01-31T20:47:23.576 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:23 smithi167 conmon[49112]: debug 2022-01-31T20:47:23.308+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.309926+0000) 2022-01-31T20:47:23.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:23 smithi167 conmon[54076]: debug 2022-01-31T20:47:23.308+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.310244+0000) 2022-01-31T20:47:23.578 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:23 smithi167 conmon[60316]: debug 2022-01-31T20:47:23.308+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.310340+0000) 2022-01-31T20:47:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[35325]: debug 2022-01-31T20:47:23.571+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:47:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:23 smithi171 conmon[41853]: debug 2022-01-31T20:47:23.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.495051+0000) 2022-01-31T20:47:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:23 smithi167 conmon[49112]: debug 2022-01-31T20:47:23.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:23.588460+0000) 2022-01-31T20:47:24.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:24 smithi167 conmon[54076]: debug 2022-01-31T20:47:24.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:24.259400+0000) 2022-01-31T20:47:24.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:24 smithi167 conmon[60316]: debug 2022-01-31T20:47:24.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:24.010532+0000) 2022-01-31T20:47:24.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:24 smithi171 conmon[46715]: debug 2022-01-31T20:47:24.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:24.287601+0000) 2022-01-31T20:47:24.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:24 smithi171 conmon[51620]: debug 2022-01-31T20:47:24.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:24.366322+0000) 2022-01-31T20:47:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:24 smithi171 conmon[41853]: debug 2022-01-31T20:47:24.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:24.495186+0000) 2022-01-31T20:47:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:24 smithi167 conmon[49112]: debug 2022-01-31T20:47:24.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:24.588600+0000) 2022-01-31T20:47:25.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:25 smithi167 conmon[54076]: debug 2022-01-31T20:47:25.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:25.259585+0000) 2022-01-31T20:47:25.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:25 smithi167 conmon[60316]: debug 2022-01-31T20:47:25.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:25.010627+0000) 2022-01-31T20:47:25.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:25 smithi171 conmon[46715]: debug 2022-01-31T20:47:25.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:25.287746+0000) 2022-01-31T20:47:25.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:25 smithi171 conmon[51620]: debug 2022-01-31T20:47:25.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:25.366479+0000) 2022-01-31T20:47:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:25 smithi171 conmon[41853]: debug 2022-01-31T20:47:25.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:25.495349+0000) 2022-01-31T20:47:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:25 smithi167 conmon[49112]: debug 2022-01-31T20:47:25.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:25.588755+0000) 2022-01-31T20:47:26.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:26 smithi167 conmon[54076]: debug 2022-01-31T20:47:26.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:26.259743+0000) 2022-01-31T20:47:26.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:26 smithi167 conmon[60316]: debug 2022-01-31T20:47:26.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:26.010831+0000) 2022-01-31T20:47:26.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:26 smithi171 conmon[46715]: debug 2022-01-31T20:47:26.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:26.287921+0000) 2022-01-31T20:47:26.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:26 smithi171 conmon[51620]: debug 2022-01-31T20:47:26.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:26.366664+0000) 2022-01-31T20:47:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:26 smithi171 conmon[41853]: debug 2022-01-31T20:47:26.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:26.495503+0000) 2022-01-31T20:47:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:26 smithi167 conmon[49112]: debug 2022-01-31T20:47:26.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:26.588924+0000) 2022-01-31T20:47:27.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:27 smithi167 conmon[54076]: debug 2022-01-31T20:47:27.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:27.259876+0000) 2022-01-31T20:47:27.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:27 smithi167 conmon[60316]: debug 2022-01-31T20:47:27.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:27.010979+0000) 2022-01-31T20:47:27.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:27 smithi171 conmon[46715]: debug 2022-01-31T20:47:27.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:27.288112+0000) 2022-01-31T20:47:27.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:27 smithi171 conmon[51620]: debug 2022-01-31T20:47:27.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:27.366852+0000) 2022-01-31T20:47:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:27 smithi171 conmon[41853]: debug 2022-01-31T20:47:27.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:27.495631+0000) 2022-01-31T20:47:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:27 smithi167 conmon[49112]: debug 2022-01-31T20:47:27.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:27.589133+0000) 2022-01-31T20:47:28.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:28 smithi167 conmon[54076]: debug 2022-01-31T20:47:28.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.260057+0000) 2022-01-31T20:47:28.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:28 smithi167 conmon[60316]: debug 2022-01-31T20:47:28.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.011158+0000) 2022-01-31T20:47:28.494 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[35325]: debug 2022-01-31T20:47:28.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78536 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:28.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[41853]: debug 2022-01-31T20:47:28.335+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.335925+0000) 2022-01-31T20:47:28.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[46715]: debug 2022-01-31T20:47:28.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.288325+0000) 2022-01-31T20:47:28.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[46715]: debug 2022-01-31T20:47:28.335+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.336265+0000) 2022-01-31T20:47:28.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[51620]: debug 2022-01-31T20:47:28.337+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.337927+0000) 2022-01-31T20:47:28.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[51620]: debug 2022-01-31T20:47:28.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.366991+0000) 2022-01-31T20:47:28.578 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:28 smithi167 conmon[49112]: debug 2022-01-31T20:47:28.335+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.337151+0000) 2022-01-31T20:47:28.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:28 smithi167 conmon[54076]: debug 2022-01-31T20:47:28.335+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.337344+0000) 2022-01-31T20:47:28.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:28 smithi167 conmon[60316]: debug 2022-01-31T20:47:28.336+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.337704+0000) 2022-01-31T20:47:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:28 smithi171 conmon[41853]: debug 2022-01-31T20:47:28.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.495815+0000) 2022-01-31T20:47:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:28 smithi167 conmon[49112]: debug 2022-01-31T20:47:28.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:28.589315+0000) 2022-01-31T20:47:29.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:29 smithi167 conmon[54076]: debug 2022-01-31T20:47:29.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:29.260264+0000) 2022-01-31T20:47:29.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:29 smithi167 conmon[60316]: debug 2022-01-31T20:47:29.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:29.011285+0000) 2022-01-31T20:47:29.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:29 smithi171 conmon[46715]: debug 2022-01-31T20:47:29.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:29.288507+0000) 2022-01-31T20:47:29.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:29 smithi171 conmon[51620]: debug 2022-01-31T20:47:29.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:29.367116+0000) 2022-01-31T20:47:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:29 smithi171 conmon[41853]: debug 2022-01-31T20:47:29.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:29.495973+0000) 2022-01-31T20:47:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:29 smithi167 conmon[49112]: debug 2022-01-31T20:47:29.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:29.589480+0000) 2022-01-31T20:47:30.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:30 smithi167 conmon[54076]: debug 2022-01-31T20:47:30.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:30.260403+0000) 2022-01-31T20:47:30.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:30 smithi167 conmon[60316]: debug 2022-01-31T20:47:30.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:30.011455+0000) 2022-01-31T20:47:30.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:30 smithi171 conmon[46715]: debug 2022-01-31T20:47:30.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:30.288657+0000) 2022-01-31T20:47:30.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:30 smithi171 conmon[51620]: debug 2022-01-31T20:47:30.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:30.367289+0000) 2022-01-31T20:47:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:30 smithi171 conmon[41853]: debug 2022-01-31T20:47:30.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:30.496123+0000) 2022-01-31T20:47:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:30 smithi167 conmon[49112]: debug 2022-01-31T20:47:30.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:30.589673+0000) 2022-01-31T20:47:31.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:31 smithi167 conmon[54076]: debug 2022-01-31T20:47:31.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:31.260607+0000) 2022-01-31T20:47:31.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:31 smithi167 conmon[60316]: debug 2022-01-31T20:47:31.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:31.011625+0000) 2022-01-31T20:47:31.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:31 smithi171 conmon[46715]: debug 2022-01-31T20:47:31.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:31.288843+0000) 2022-01-31T20:47:31.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:31 smithi171 conmon[51620]: debug 2022-01-31T20:47:31.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:31.367483+0000) 2022-01-31T20:47:31.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:31 smithi171 conmon[41853]: debug 2022-01-31T20:47:31.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:31.496305+0000) 2022-01-31T20:47:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:31 smithi167 conmon[49112]: debug 2022-01-31T20:47:31.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:31.589837+0000) 2022-01-31T20:47:32.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:32 smithi167 conmon[54076]: debug 2022-01-31T20:47:32.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:32.260765+0000) 2022-01-31T20:47:32.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:32 smithi167 conmon[60316]: debug 2022-01-31T20:47:32.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:32.011775+0000) 2022-01-31T20:47:32.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:32 smithi171 conmon[46715]: debug 2022-01-31T20:47:32.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:32.289022+0000) 2022-01-31T20:47:32.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:32 smithi171 conmon[51620]: debug 2022-01-31T20:47:32.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:32.367632+0000) 2022-01-31T20:47:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:32 smithi171 conmon[41853]: debug 2022-01-31T20:47:32.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:32.496497+0000) 2022-01-31T20:47:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:32 smithi167 conmon[49112]: debug 2022-01-31T20:47:32.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:32.589999+0000) 2022-01-31T20:47:33.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:33 smithi167 conmon[54076]: debug 2022-01-31T20:47:33.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.260925+0000) 2022-01-31T20:47:33.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:33 smithi167 conmon[60316]: debug 2022-01-31T20:47:33.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.011871+0000) 2022-01-31T20:47:33.495 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[35325]: debug 2022-01-31T20:47:33.354+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78644 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:33.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[41853]: debug 2022-01-31T20:47:33.345+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.346191+0000) 2022-01-31T20:47:33.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[46715]: debug 2022-01-31T20:47:33.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.289180+0000) 2022-01-31T20:47:33.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[46715]: debug 2022-01-31T20:47:33.346+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.347257+0000) 2022-01-31T20:47:33.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[51620]: debug 2022-01-31T20:47:33.346+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.347405+0000) 2022-01-31T20:47:33.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[51620]: debug 2022-01-31T20:47:33.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.367832+0000) 2022-01-31T20:47:33.578 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:33 smithi167 conmon[49112]: debug 2022-01-31T20:47:33.346+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.347735+0000) 2022-01-31T20:47:33.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:33 smithi167 conmon[54076]: debug 2022-01-31T20:47:33.346+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.347924+0000) 2022-01-31T20:47:33.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:33 smithi167 conmon[60316]: debug 2022-01-31T20:47:33.347+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.348528+0000) 2022-01-31T20:47:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:33 smithi171 conmon[41853]: debug 2022-01-31T20:47:33.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.496682+0000) 2022-01-31T20:47:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:33 smithi167 conmon[49112]: debug 2022-01-31T20:47:33.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:33.590220+0000) 2022-01-31T20:47:34.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:34 smithi167 conmon[54076]: debug 2022-01-31T20:47:34.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:34.261115+0000) 2022-01-31T20:47:34.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:34 smithi167 conmon[60316]: debug 2022-01-31T20:47:34.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:34.012046+0000) 2022-01-31T20:47:34.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:34 smithi171 conmon[46715]: debug 2022-01-31T20:47:34.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:34.289311+0000) 2022-01-31T20:47:34.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:34 smithi171 conmon[51620]: debug 2022-01-31T20:47:34.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:34.368011+0000) 2022-01-31T20:47:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:34 smithi171 conmon[41853]: debug 2022-01-31T20:47:34.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:34.496851+0000) 2022-01-31T20:47:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:34 smithi167 conmon[49112]: debug 2022-01-31T20:47:34.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:34.590396+0000) 2022-01-31T20:47:35.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:35 smithi167 conmon[54076]: debug 2022-01-31T20:47:35.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:35.261257+0000) 2022-01-31T20:47:35.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:35 smithi167 conmon[60316]: debug 2022-01-31T20:47:35.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:35.012156+0000) 2022-01-31T20:47:35.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:35 smithi171 conmon[46715]: debug 2022-01-31T20:47:35.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:35.289486+0000) 2022-01-31T20:47:35.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:35 smithi171 conmon[51620]: debug 2022-01-31T20:47:35.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:35.368184+0000) 2022-01-31T20:47:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:35 smithi171 conmon[41853]: debug 2022-01-31T20:47:35.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:35.497028+0000) 2022-01-31T20:47:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:35 smithi167 conmon[49112]: debug 2022-01-31T20:47:35.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:35.590595+0000) 2022-01-31T20:47:36.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:36 smithi167 conmon[54076]: debug 2022-01-31T20:47:36.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:36.261453+0000) 2022-01-31T20:47:36.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:36 smithi167 conmon[60316]: debug 2022-01-31T20:47:36.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:36.012315+0000) 2022-01-31T20:47:36.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:36 smithi171 conmon[46715]: debug 2022-01-31T20:47:36.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:36.289695+0000) 2022-01-31T20:47:36.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:36 smithi171 conmon[51620]: debug 2022-01-31T20:47:36.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:36.368368+0000) 2022-01-31T20:47:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:36 smithi171 conmon[41853]: debug 2022-01-31T20:47:36.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:36.497223+0000) 2022-01-31T20:47:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:36 smithi167 conmon[49112]: debug 2022-01-31T20:47:36.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:36.590725+0000) 2022-01-31T20:47:37.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:37 smithi167 conmon[54076]: debug 2022-01-31T20:47:37.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:37.261670+0000) 2022-01-31T20:47:37.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:37 smithi167 conmon[60316]: debug 2022-01-31T20:47:37.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:37.012440+0000) 2022-01-31T20:47:37.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:37 smithi171 conmon[46715]: debug 2022-01-31T20:47:37.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:37.289929+0000) 2022-01-31T20:47:37.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:37 smithi171 conmon[51620]: debug 2022-01-31T20:47:37.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:37.368529+0000) 2022-01-31T20:47:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:37 smithi171 conmon[41853]: debug 2022-01-31T20:47:37.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:37.497376+0000) 2022-01-31T20:47:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:37 smithi167 conmon[49112]: debug 2022-01-31T20:47:37.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:37.590840+0000) 2022-01-31T20:47:38.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:38 smithi167 conmon[54076]: debug 2022-01-31T20:47:38.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.261825+0000) 2022-01-31T20:47:38.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:38 smithi167 conmon[60316]: debug 2022-01-31T20:47:38.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.012598+0000) 2022-01-31T20:47:38.496 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[35325]: debug 2022-01-31T20:47:38.365+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78755 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:38.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[41853]: debug 2022-01-31T20:47:38.356+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.357523+0000) 2022-01-31T20:47:38.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[46715]: debug 2022-01-31T20:47:38.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.290106+0000) 2022-01-31T20:47:38.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[46715]: debug 2022-01-31T20:47:38.357+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.358119+0000) 2022-01-31T20:47:38.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[51620]: debug 2022-01-31T20:47:38.357+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.357886+0000) 2022-01-31T20:47:38.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[51620]: debug 2022-01-31T20:47:38.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.368889+0000) 2022-01-31T20:47:38.579 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:38 smithi167 conmon[49112]: debug 2022-01-31T20:47:38.357+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.358822+0000) 2022-01-31T20:47:38.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:38 smithi167 conmon[54076]: debug 2022-01-31T20:47:38.358+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.359971+0000) 2022-01-31T20:47:38.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:38 smithi167 conmon[60316]: debug 2022-01-31T20:47:38.358+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.360261+0000) 2022-01-31T20:47:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[35325]: debug 2022-01-31T20:47:38.572+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:47:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:38 smithi171 conmon[41853]: debug 2022-01-31T20:47:38.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.497537+0000) 2022-01-31T20:47:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:38 smithi167 conmon[49112]: debug 2022-01-31T20:47:38.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:38.591045+0000) 2022-01-31T20:47:39.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:39 smithi167 conmon[54076]: debug 2022-01-31T20:47:39.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:39.262005+0000) 2022-01-31T20:47:39.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:39 smithi167 conmon[60316]: debug 2022-01-31T20:47:39.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:39.012738+0000) 2022-01-31T20:47:39.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:39 smithi171 conmon[46715]: debug 2022-01-31T20:47:39.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:39.290325+0000) 2022-01-31T20:47:39.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:39 smithi171 conmon[51620]: debug 2022-01-31T20:47:39.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:39.369034+0000) 2022-01-31T20:47:39.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:39 smithi171 conmon[41853]: debug 2022-01-31T20:47:39.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:39.497668+0000) 2022-01-31T20:47:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:39 smithi167 conmon[49112]: debug 2022-01-31T20:47:39.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:39.591219+0000) 2022-01-31T20:47:40.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:40 smithi167 conmon[54076]: debug 2022-01-31T20:47:40.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:40.262164+0000) 2022-01-31T20:47:40.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:40 smithi167 conmon[60316]: debug 2022-01-31T20:47:40.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:40.012930+0000) 2022-01-31T20:47:40.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:40 smithi171 conmon[46715]: debug 2022-01-31T20:47:40.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:40.290475+0000) 2022-01-31T20:47:40.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:40 smithi171 conmon[51620]: debug 2022-01-31T20:47:40.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:40.369239+0000) 2022-01-31T20:47:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:40 smithi171 conmon[41853]: debug 2022-01-31T20:47:40.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:40.497770+0000) 2022-01-31T20:47:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:40 smithi167 conmon[49112]: debug 2022-01-31T20:47:40.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:40.591454+0000) 2022-01-31T20:47:41.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:41 smithi167 conmon[54076]: debug 2022-01-31T20:47:41.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:41.262372+0000) 2022-01-31T20:47:41.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:41 smithi167 conmon[60316]: debug 2022-01-31T20:47:41.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:41.013087+0000) 2022-01-31T20:47:41.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:41 smithi171 conmon[46715]: debug 2022-01-31T20:47:41.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:41.290661+0000) 2022-01-31T20:47:41.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:41 smithi171 conmon[51620]: debug 2022-01-31T20:47:41.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:41.369442+0000) 2022-01-31T20:47:41.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:41 smithi171 conmon[41853]: debug 2022-01-31T20:47:41.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:41.497891+0000) 2022-01-31T20:47:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:41 smithi167 conmon[49112]: debug 2022-01-31T20:47:41.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:41.591605+0000) 2022-01-31T20:47:42.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:42 smithi167 conmon[54076]: debug 2022-01-31T20:47:42.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:42.262576+0000) 2022-01-31T20:47:42.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:42 smithi167 conmon[60316]: debug 2022-01-31T20:47:42.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:42.013262+0000) 2022-01-31T20:47:42.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:42 smithi171 conmon[46715]: debug 2022-01-31T20:47:42.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:42.290845+0000) 2022-01-31T20:47:42.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:42 smithi171 conmon[51620]: debug 2022-01-31T20:47:42.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:42.369659+0000) 2022-01-31T20:47:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:42 smithi171 conmon[41853]: debug 2022-01-31T20:47:42.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:42.498055+0000) 2022-01-31T20:47:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:42 smithi167 conmon[49112]: debug 2022-01-31T20:47:42.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:42.591720+0000) 2022-01-31T20:47:43.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:43 smithi167 conmon[54076]: debug 2022-01-31T20:47:43.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.262706+0000) 2022-01-31T20:47:43.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:43 smithi167 conmon[60316]: debug 2022-01-31T20:47:43.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.013409+0000) 2022-01-31T20:47:43.375 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[41853]: debug 2022-01-31T20:47:43.367+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.368655+0000) 2022-01-31T20:47:43.375 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[46715]: debug 2022-01-31T20:47:43.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.291051+0000) 2022-01-31T20:47:43.376 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[46715]: debug 2022-01-31T20:47:43.367+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.368527+0000) 2022-01-31T20:47:43.376 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[51620]: debug 2022-01-31T20:47:43.367+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.368201+0000) 2022-01-31T20:47:43.376 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[51620]: debug 2022-01-31T20:47:43.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.369785+0000) 2022-01-31T20:47:43.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:43 smithi167 conmon[49112]: debug 2022-01-31T20:47:43.368+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.369678+0000) 2022-01-31T20:47:43.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:43 smithi167 conmon[54076]: debug 2022-01-31T20:47:43.368+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.369782+0000) 2022-01-31T20:47:43.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:43 smithi167 conmon[60316]: debug 2022-01-31T20:47:43.368+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.370059+0000) 2022-01-31T20:47:43.757 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[35325]: debug 2022-01-31T20:47:43.375+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78863 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:43.757 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:43 smithi171 conmon[41853]: debug 2022-01-31T20:47:43.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.498261+0000) 2022-01-31T20:47:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:43 smithi167 conmon[49112]: debug 2022-01-31T20:47:43.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:43.591881+0000) 2022-01-31T20:47:44.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:44 smithi167 conmon[54076]: debug 2022-01-31T20:47:44.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:44.262908+0000) 2022-01-31T20:47:44.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:44 smithi167 conmon[60316]: debug 2022-01-31T20:47:44.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:44.013571+0000) 2022-01-31T20:47:44.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:44 smithi171 conmon[46715]: debug 2022-01-31T20:47:44.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:44.291266+0000) 2022-01-31T20:47:44.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:44 smithi171 conmon[51620]: debug 2022-01-31T20:47:44.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:44.369911+0000) 2022-01-31T20:47:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:44 smithi171 conmon[41853]: debug 2022-01-31T20:47:44.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:44.498433+0000) 2022-01-31T20:47:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:44 smithi167 conmon[49112]: debug 2022-01-31T20:47:44.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:44.592025+0000) 2022-01-31T20:47:45.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:45 smithi167 conmon[54076]: debug 2022-01-31T20:47:45.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:45.263071+0000) 2022-01-31T20:47:45.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:45 smithi167 conmon[60316]: debug 2022-01-31T20:47:45.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:45.013749+0000) 2022-01-31T20:47:45.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:45 smithi171 conmon[46715]: debug 2022-01-31T20:47:45.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:45.291432+0000) 2022-01-31T20:47:45.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:45 smithi171 conmon[51620]: debug 2022-01-31T20:47:45.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:45.370074+0000) 2022-01-31T20:47:45.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:45 smithi171 conmon[41853]: debug 2022-01-31T20:47:45.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:45.498613+0000) 2022-01-31T20:47:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:45 smithi167 conmon[49112]: debug 2022-01-31T20:47:45.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:45.592207+0000) 2022-01-31T20:47:46.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:46 smithi167 conmon[60316]: debug 2022-01-31T20:47:46.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:46.013945+0000) 2022-01-31T20:47:46.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:46 smithi167 conmon[54076]: debug 2022-01-31T20:47:46.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:46.263259+0000) 2022-01-31T20:47:46.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:46 smithi171 conmon[46715]: debug 2022-01-31T20:47:46.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:46.291607+0000) 2022-01-31T20:47:46.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:46 smithi171 conmon[51620]: debug 2022-01-31T20:47:46.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:46.370194+0000) 2022-01-31T20:47:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:46 smithi171 conmon[41853]: debug 2022-01-31T20:47:46.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:46.498746+0000) 2022-01-31T20:47:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:46 smithi167 conmon[49112]: debug 2022-01-31T20:47:46.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:46.592368+0000) 2022-01-31T20:47:47.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:47 smithi167 conmon[60316]: debug 2022-01-31T20:47:47.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:47.014161+0000) 2022-01-31T20:47:47.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:47 smithi167 conmon[54076]: debug 2022-01-31T20:47:47.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:47.263413+0000) 2022-01-31T20:47:47.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:47 smithi171 conmon[46715]: debug 2022-01-31T20:47:47.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:47.291795+0000) 2022-01-31T20:47:47.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:47 smithi171 conmon[51620]: debug 2022-01-31T20:47:47.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:47.370385+0000) 2022-01-31T20:47:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:47 smithi171 conmon[41853]: debug 2022-01-31T20:47:47.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:47.498889+0000) 2022-01-31T20:47:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:47 smithi167 conmon[49112]: debug 2022-01-31T20:47:47.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:47.592556+0000) 2022-01-31T20:47:48.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:48 smithi167 conmon[60316]: debug 2022-01-31T20:47:48.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.014351+0000) 2022-01-31T20:47:48.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:48 smithi167 conmon[54076]: debug 2022-01-31T20:47:48.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.263633+0000) 2022-01-31T20:47:48.376 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[46715]: debug 2022-01-31T20:47:48.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.291952+0000) 2022-01-31T20:47:48.377 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[51620]: debug 2022-01-31T20:47:48.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.370577+0000) 2022-01-31T20:47:48.377 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[51620]: debug 2022-01-31T20:47:48.377+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.378285+0000) 2022-01-31T20:47:48.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:48 smithi167 conmon[49112]: debug 2022-01-31T20:47:48.379+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.380156+0000) 2022-01-31T20:47:48.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:48 smithi167 conmon[54076]: debug 2022-01-31T20:47:48.379+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.380001+0000) 2022-01-31T20:47:48.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:48 smithi167 conmon[60316]: debug 2022-01-31T20:47:48.380+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.380722+0000) 2022-01-31T20:47:48.705 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[35325]: debug 2022-01-31T20:47:48.387+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 78975 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:48.706 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[46715]: debug 2022-01-31T20:47:48.378+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.379517+0000) 2022-01-31T20:47:48.706 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[41853]: debug 2022-01-31T20:47:48.378+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.378962+0000) 2022-01-31T20:47:48.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:48 smithi171 conmon[41853]: debug 2022-01-31T20:47:48.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.499030+0000) 2022-01-31T20:47:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:48 smithi167 conmon[49112]: debug 2022-01-31T20:47:48.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:48.592688+0000) 2022-01-31T20:47:49.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:49 smithi167 conmon[54076]: debug 2022-01-31T20:47:49.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:49.263815+0000) 2022-01-31T20:47:49.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:49 smithi167 conmon[60316]: debug 2022-01-31T20:47:49.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:49.014582+0000) 2022-01-31T20:47:49.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:49 smithi171 conmon[46715]: debug 2022-01-31T20:47:49.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:49.292163+0000) 2022-01-31T20:47:49.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:49 smithi171 conmon[51620]: debug 2022-01-31T20:47:49.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:49.370708+0000) 2022-01-31T20:47:49.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:49 smithi167 conmon[49112]: debug 2022-01-31T20:47:49.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:49.592847+0000) 2022-01-31T20:47:49.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:49 smithi171 conmon[41853]: debug 2022-01-31T20:47:49.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:49.499172+0000) 2022-01-31T20:47:50.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:50 smithi167 conmon[54076]: debug 2022-01-31T20:47:50.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:50.263961+0000) 2022-01-31T20:47:50.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:50 smithi167 conmon[60316]: debug 2022-01-31T20:47:50.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:50.014758+0000) 2022-01-31T20:47:50.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:50 smithi171 conmon[51620]: debug 2022-01-31T20:47:50.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:50.370861+0000) 2022-01-31T20:47:50.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:50 smithi171 conmon[46715]: debug 2022-01-31T20:47:50.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:50.292320+0000) 2022-01-31T20:47:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:50 smithi171 conmon[41853]: debug 2022-01-31T20:47:50.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:50.499376+0000) 2022-01-31T20:47:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:50 smithi167 conmon[49112]: debug 2022-01-31T20:47:50.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:50.592990+0000) 2022-01-31T20:47:51.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:51 smithi167 conmon[54076]: debug 2022-01-31T20:47:51.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:51.264147+0000) 2022-01-31T20:47:51.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:51 smithi167 conmon[60316]: debug 2022-01-31T20:47:51.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:51.014964+0000) 2022-01-31T20:47:51.400 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:51 smithi171 conmon[46715]: debug 2022-01-31T20:47:51.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:51.292504+0000) 2022-01-31T20:47:51.401 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:51 smithi171 conmon[51620]: debug 2022-01-31T20:47:51.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:51.371045+0000) 2022-01-31T20:47:51.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:51 smithi167 conmon[49112]: debug 2022-01-31T20:47:51.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:51.593328+0000) 2022-01-31T20:47:51.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:51 smithi171 conmon[41853]: debug 2022-01-31T20:47:51.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:51.499551+0000) 2022-01-31T20:47:52.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:52 smithi167 conmon[54076]: debug 2022-01-31T20:47:52.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:52.264351+0000) 2022-01-31T20:47:52.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:52 smithi167 conmon[60316]: debug 2022-01-31T20:47:52.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:52.015125+0000) 2022-01-31T20:47:52.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:52 smithi171 conmon[46715]: debug 2022-01-31T20:47:52.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:52.292670+0000) 2022-01-31T20:47:52.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:52 smithi171 conmon[51620]: debug 2022-01-31T20:47:52.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:52.371141+0000) 2022-01-31T20:47:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:52 smithi171 conmon[41853]: debug 2022-01-31T20:47:52.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:52.499710+0000) 2022-01-31T20:47:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:52 smithi167 conmon[49112]: debug 2022-01-31T20:47:52.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:52.593535+0000) 2022-01-31T20:47:53.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:53 smithi167 conmon[54076]: debug 2022-01-31T20:47:53.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.264540+0000) 2022-01-31T20:47:53.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:53 smithi167 conmon[60316]: debug 2022-01-31T20:47:53.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.015256+0000) 2022-01-31T20:47:53.388 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[46715]: debug 2022-01-31T20:47:53.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.292856+0000) 2022-01-31T20:47:53.389 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[51620]: debug 2022-01-31T20:47:53.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.371338+0000) 2022-01-31T20:47:53.582 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:53 smithi167 conmon[49112]: debug 2022-01-31T20:47:53.391+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.392107+0000) 2022-01-31T20:47:53.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:53 smithi167 conmon[54076]: debug 2022-01-31T20:47:53.389+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.389847+0000) 2022-01-31T20:47:53.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:53 smithi167 conmon[60316]: debug 2022-01-31T20:47:53.391+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.392390+0000) 2022-01-31T20:47:53.706 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[46715]: debug 2022-01-31T20:47:53.389+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.390518+0000) 2022-01-31T20:47:53.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[51620]: debug 2022-01-31T20:47:53.389+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.390433+0000) 2022-01-31T20:47:53.707 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[35325]: debug 2022-01-31T20:47:53.397+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79083 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:53.707 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[35325]: debug 2022-01-31T20:47:53.573+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:47:53.708 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[41853]: debug 2022-01-31T20:47:53.389+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.389897+0000) 2022-01-31T20:47:53.708 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:53 smithi171 conmon[41853]: debug 2022-01-31T20:47:53.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.499858+0000) 2022-01-31T20:47:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:53 smithi167 conmon[49112]: debug 2022-01-31T20:47:53.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:53.593703+0000) 2022-01-31T20:47:54.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:54 smithi167 conmon[54076]: debug 2022-01-31T20:47:54.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:54.264751+0000) 2022-01-31T20:47:54.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:54 smithi167 conmon[60316]: debug 2022-01-31T20:47:54.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:54.015392+0000) 2022-01-31T20:47:54.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:54 smithi171 conmon[46715]: debug 2022-01-31T20:47:54.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:54.293030+0000) 2022-01-31T20:47:54.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:54 smithi171 conmon[51620]: debug 2022-01-31T20:47:54.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:54.371553+0000) 2022-01-31T20:47:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:54 smithi171 conmon[41853]: debug 2022-01-31T20:47:54.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:54.500018+0000) 2022-01-31T20:47:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:54 smithi167 conmon[49112]: debug 2022-01-31T20:47:54.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:54.593833+0000) 2022-01-31T20:47:55.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:55 smithi167 conmon[54076]: debug 2022-01-31T20:47:55.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:55.264891+0000) 2022-01-31T20:47:55.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:55 smithi167 conmon[60316]: debug 2022-01-31T20:47:55.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:55.015525+0000) 2022-01-31T20:47:55.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:55 smithi171 conmon[46715]: debug 2022-01-31T20:47:55.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:55.293205+0000) 2022-01-31T20:47:55.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:55 smithi171 conmon[51620]: debug 2022-01-31T20:47:55.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:55.371660+0000) 2022-01-31T20:47:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:55 smithi171 conmon[41853]: debug 2022-01-31T20:47:55.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:55.500184+0000) 2022-01-31T20:47:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:55 smithi167 conmon[49112]: debug 2022-01-31T20:47:55.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:55.594072+0000) 2022-01-31T20:47:56.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:56 smithi167 conmon[54076]: debug 2022-01-31T20:47:56.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:56.265043+0000) 2022-01-31T20:47:56.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:56 smithi167 conmon[60316]: debug 2022-01-31T20:47:56.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:56.015707+0000) 2022-01-31T20:47:56.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:56 smithi171 conmon[46715]: debug 2022-01-31T20:47:56.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:56.293339+0000) 2022-01-31T20:47:56.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:56 smithi171 conmon[51620]: debug 2022-01-31T20:47:56.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:56.371843+0000) 2022-01-31T20:47:56.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:56 smithi171 conmon[41853]: debug 2022-01-31T20:47:56.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:56.500387+0000) 2022-01-31T20:47:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:56 smithi167 conmon[49112]: debug 2022-01-31T20:47:56.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:56.594256+0000) 2022-01-31T20:47:57.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:57 smithi167 conmon[54076]: debug 2022-01-31T20:47:57.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:57.265193+0000) 2022-01-31T20:47:57.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:57 smithi167 conmon[60316]: debug 2022-01-31T20:47:57.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:57.015873+0000) 2022-01-31T20:47:57.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:57 smithi171 conmon[46715]: debug 2022-01-31T20:47:57.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:57.293542+0000) 2022-01-31T20:47:57.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:57 smithi171 conmon[51620]: debug 2022-01-31T20:47:57.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:57.371997+0000) 2022-01-31T20:47:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:57 smithi171 conmon[41853]: debug 2022-01-31T20:47:57.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:57.500573+0000) 2022-01-31T20:47:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:57 smithi167 conmon[49112]: debug 2022-01-31T20:47:57.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:57.594440+0000) 2022-01-31T20:47:58.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:58 smithi167 conmon[54076]: debug 2022-01-31T20:47:58.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.265394+0000) 2022-01-31T20:47:58.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:58 smithi167 conmon[60316]: debug 2022-01-31T20:47:58.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.016037+0000) 2022-01-31T20:47:58.399 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[46715]: debug 2022-01-31T20:47:58.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.293742+0000) 2022-01-31T20:47:58.399 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[51620]: debug 2022-01-31T20:47:58.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.372189+0000) 2022-01-31T20:47:58.400 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[41853]: debug 2022-01-31T20:47:58.399+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.400463+0000) 2022-01-31T20:47:58.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:58 smithi167 conmon[54076]: debug 2022-01-31T20:47:58.400+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.400811+0000) 2022-01-31T20:47:58.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:58 smithi167 conmon[60316]: debug 2022-01-31T20:47:58.401+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.402048+0000) 2022-01-31T20:47:58.586 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:58 smithi167 conmon[49112]: debug 2022-01-31T20:47:58.401+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.402169+0000) 2022-01-31T20:47:58.707 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[35325]: debug 2022-01-31T20:47:58.407+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79194 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:47:58.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[41853]: debug 2022-01-31T20:47:58.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.500738+0000) 2022-01-31T20:47:58.708 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[46715]: debug 2022-01-31T20:47:58.400+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.400999+0000) 2022-01-31T20:47:58.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:58 smithi171 conmon[51620]: debug 2022-01-31T20:47:58.400+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.401235+0000) 2022-01-31T20:47:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:58 smithi167 conmon[49112]: debug 2022-01-31T20:47:58.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:58.594573+0000) 2022-01-31T20:47:59.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:47:59 smithi167 conmon[60316]: debug 2022-01-31T20:47:59.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:59.016203+0000) 2022-01-31T20:47:59.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:47:59 smithi167 conmon[54076]: debug 2022-01-31T20:47:59.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:59.265607+0000) 2022-01-31T20:47:59.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:47:59 smithi171 conmon[46715]: debug 2022-01-31T20:47:59.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:59.293963+0000) 2022-01-31T20:47:59.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:47:59 smithi171 conmon[51620]: debug 2022-01-31T20:47:59.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:59.372362+0000) 2022-01-31T20:47:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:47:59 smithi171 conmon[41853]: debug 2022-01-31T20:47:59.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:59.500879+0000) 2022-01-31T20:47:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:47:59 smithi167 conmon[49112]: debug 2022-01-31T20:47:59.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:47:59.594682+0000) 2022-01-31T20:48:00.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:00 smithi167 conmon[60316]: debug 2022-01-31T20:48:00.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:00.016377+0000) 2022-01-31T20:48:00.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:00 smithi167 conmon[54076]: debug 2022-01-31T20:48:00.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:00.265792+0000) 2022-01-31T20:48:00.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:00 smithi171 conmon[46715]: debug 2022-01-31T20:48:00.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:00.294114+0000) 2022-01-31T20:48:00.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:00 smithi171 conmon[51620]: debug 2022-01-31T20:48:00.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:00.372499+0000) 2022-01-31T20:48:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:00 smithi171 conmon[41853]: debug 2022-01-31T20:48:00.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:00.501061+0000) 2022-01-31T20:48:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:00 smithi167 conmon[49112]: debug 2022-01-31T20:48:00.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:00.594804+0000) 2022-01-31T20:48:01.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:01 smithi167 conmon[54076]: debug 2022-01-31T20:48:01.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:01.265975+0000) 2022-01-31T20:48:01.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:01 smithi167 conmon[60316]: debug 2022-01-31T20:48:01.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:01.016582+0000) 2022-01-31T20:48:01.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:01 smithi171 conmon[46715]: debug 2022-01-31T20:48:01.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:01.294281+0000) 2022-01-31T20:48:01.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:01 smithi171 conmon[51620]: debug 2022-01-31T20:48:01.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:01.372699+0000) 2022-01-31T20:48:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:01 smithi171 conmon[41853]: debug 2022-01-31T20:48:01.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:01.501214+0000) 2022-01-31T20:48:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:01 smithi167 conmon[49112]: debug 2022-01-31T20:48:01.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:01.595021+0000) 2022-01-31T20:48:02.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:02 smithi167 conmon[54076]: debug 2022-01-31T20:48:02.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:02.266176+0000) 2022-01-31T20:48:02.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:02 smithi167 conmon[60316]: debug 2022-01-31T20:48:02.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:02.016798+0000) 2022-01-31T20:48:02.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:02 smithi171 conmon[46715]: debug 2022-01-31T20:48:02.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:02.294400+0000) 2022-01-31T20:48:02.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:02 smithi171 conmon[51620]: debug 2022-01-31T20:48:02.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:02.372912+0000) 2022-01-31T20:48:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:02 smithi171 conmon[41853]: debug 2022-01-31T20:48:02.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:02.501351+0000) 2022-01-31T20:48:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:02 smithi167 conmon[49112]: debug 2022-01-31T20:48:02.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:02.595201+0000) 2022-01-31T20:48:03.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:03 smithi167 conmon[60316]: debug 2022-01-31T20:48:03.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.016947+0000) 2022-01-31T20:48:03.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:03 smithi167 conmon[54076]: debug 2022-01-31T20:48:03.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.266349+0000) 2022-01-31T20:48:03.408 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[46715]: debug 2022-01-31T20:48:03.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.294606+0000) 2022-01-31T20:48:03.409 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[51620]: debug 2022-01-31T20:48:03.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.373094+0000) 2022-01-31T20:48:03.584 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:03 smithi167 conmon[49112]: debug 2022-01-31T20:48:03.411+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.412399+0000) 2022-01-31T20:48:03.585 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:03 smithi167 conmon[54076]: debug 2022-01-31T20:48:03.411+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.412206+0000) 2022-01-31T20:48:03.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:03 smithi167 conmon[60316]: debug 2022-01-31T20:48:03.411+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.411541+0000) 2022-01-31T20:48:03.708 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[35325]: debug 2022-01-31T20:48:03.416+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79302 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:03.709 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[46715]: debug 2022-01-31T20:48:03.408+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.410192+0000) 2022-01-31T20:48:03.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[51620]: debug 2022-01-31T20:48:03.409+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.411025+0000) 2022-01-31T20:48:03.710 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[41853]: debug 2022-01-31T20:48:03.410+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.411998+0000) 2022-01-31T20:48:03.711 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:03 smithi171 conmon[41853]: debug 2022-01-31T20:48:03.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.501478+0000) 2022-01-31T20:48:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:03 smithi167 conmon[49112]: debug 2022-01-31T20:48:03.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:03.595417+0000) 2022-01-31T20:48:04.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:04 smithi167 conmon[54076]: debug 2022-01-31T20:48:04.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:04.266514+0000) 2022-01-31T20:48:04.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:04 smithi167 conmon[60316]: debug 2022-01-31T20:48:04.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:04.017124+0000) 2022-01-31T20:48:04.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:04 smithi171 conmon[46715]: debug 2022-01-31T20:48:04.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:04.294795+0000) 2022-01-31T20:48:04.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:04 smithi171 conmon[51620]: debug 2022-01-31T20:48:04.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:04.373268+0000) 2022-01-31T20:48:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:04 smithi171 conmon[41853]: debug 2022-01-31T20:48:04.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:04.501617+0000) 2022-01-31T20:48:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:04 smithi167 conmon[49112]: debug 2022-01-31T20:48:04.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:04.595529+0000) 2022-01-31T20:48:05.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:05 smithi167 conmon[54076]: debug 2022-01-31T20:48:05.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:05.266624+0000) 2022-01-31T20:48:05.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:05 smithi167 conmon[60316]: debug 2022-01-31T20:48:05.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:05.017283+0000) 2022-01-31T20:48:05.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:05 smithi171 conmon[46715]: debug 2022-01-31T20:48:05.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:05.294911+0000) 2022-01-31T20:48:05.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:05 smithi171 conmon[51620]: debug 2022-01-31T20:48:05.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:05.373420+0000) 2022-01-31T20:48:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:05 smithi171 conmon[41853]: debug 2022-01-31T20:48:05.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:05.501804+0000) 2022-01-31T20:48:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:05 smithi167 conmon[49112]: debug 2022-01-31T20:48:05.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:05.595662+0000) 2022-01-31T20:48:06.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:06 smithi167 conmon[54076]: debug 2022-01-31T20:48:06.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:06.266747+0000) 2022-01-31T20:48:06.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:06 smithi167 conmon[60316]: debug 2022-01-31T20:48:06.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:06.017468+0000) 2022-01-31T20:48:06.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:06 smithi171 conmon[46715]: debug 2022-01-31T20:48:06.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:06.295096+0000) 2022-01-31T20:48:06.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:06 smithi171 conmon[51620]: debug 2022-01-31T20:48:06.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:06.373609+0000) 2022-01-31T20:48:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:06 smithi171 conmon[41853]: debug 2022-01-31T20:48:06.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:06.501989+0000) 2022-01-31T20:48:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:06 smithi167 conmon[49112]: debug 2022-01-31T20:48:06.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:06.595789+0000) 2022-01-31T20:48:07.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:07 smithi167 conmon[54076]: debug 2022-01-31T20:48:07.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:07.266973+0000) 2022-01-31T20:48:07.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:07 smithi167 conmon[60316]: debug 2022-01-31T20:48:07.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:07.017665+0000) 2022-01-31T20:48:07.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:07 smithi171 conmon[46715]: debug 2022-01-31T20:48:07.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:07.295263+0000) 2022-01-31T20:48:07.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:07 smithi171 conmon[51620]: debug 2022-01-31T20:48:07.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:07.373794+0000) 2022-01-31T20:48:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:07 smithi171 conmon[41853]: debug 2022-01-31T20:48:07.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:07.502175+0000) 2022-01-31T20:48:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:07 smithi167 conmon[49112]: debug 2022-01-31T20:48:07.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:07.595912+0000) 2022-01-31T20:48:08.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:08 smithi167 conmon[54076]: debug 2022-01-31T20:48:08.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.267160+0000) 2022-01-31T20:48:08.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:08 smithi167 conmon[60316]: debug 2022-01-31T20:48:08.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.017869+0000) 2022-01-31T20:48:08.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[46715]: debug 2022-01-31T20:48:08.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.295420+0000) 2022-01-31T20:48:08.419 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[51620]: debug 2022-01-31T20:48:08.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.374003+0000) 2022-01-31T20:48:08.586 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:08 smithi167 conmon[49112]: debug 2022-01-31T20:48:08.421+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.421504+0000) 2022-01-31T20:48:08.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:08 smithi167 conmon[54076]: debug 2022-01-31T20:48:08.421+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.422175+0000) 2022-01-31T20:48:08.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:08 smithi167 conmon[60316]: debug 2022-01-31T20:48:08.421+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.421994+0000) 2022-01-31T20:48:08.709 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[46715]: debug 2022-01-31T20:48:08.418+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.420347+0000) 2022-01-31T20:48:08.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[51620]: debug 2022-01-31T20:48:08.419+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.420937+0000) 2022-01-31T20:48:08.710 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[35325]: debug 2022-01-31T20:48:08.427+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79413 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:08.710 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[35325]: debug 2022-01-31T20:48:08.573+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:48:08.711 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[41853]: debug 2022-01-31T20:48:08.419+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.421309+0000) 2022-01-31T20:48:08.711 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:08 smithi171 conmon[41853]: debug 2022-01-31T20:48:08.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.502398+0000) 2022-01-31T20:48:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:08 smithi167 conmon[49112]: debug 2022-01-31T20:48:08.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:08.596055+0000) 2022-01-31T20:48:09.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:09 smithi167 conmon[54076]: debug 2022-01-31T20:48:09.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:09.267262+0000) 2022-01-31T20:48:09.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:09 smithi167 conmon[60316]: debug 2022-01-31T20:48:09.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:09.018072+0000) 2022-01-31T20:48:09.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:09 smithi171 conmon[51620]: debug 2022-01-31T20:48:09.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:09.374181+0000) 2022-01-31T20:48:09.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:09 smithi171 conmon[46715]: debug 2022-01-31T20:48:09.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:09.295619+0000) 2022-01-31T20:48:09.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:09 smithi171 conmon[41853]: debug 2022-01-31T20:48:09.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:09.502550+0000) 2022-01-31T20:48:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:09 smithi167 conmon[49112]: debug 2022-01-31T20:48:09.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:09.596176+0000) 2022-01-31T20:48:10.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:10 smithi167 conmon[54076]: debug 2022-01-31T20:48:10.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:10.267423+0000) 2022-01-31T20:48:10.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:10 smithi167 conmon[60316]: debug 2022-01-31T20:48:10.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:10.018249+0000) 2022-01-31T20:48:10.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:10 smithi171 conmon[46715]: debug 2022-01-31T20:48:10.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:10.295781+0000) 2022-01-31T20:48:10.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:10 smithi171 conmon[51620]: debug 2022-01-31T20:48:10.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:10.374353+0000) 2022-01-31T20:48:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:10 smithi171 conmon[41853]: debug 2022-01-31T20:48:10.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:10.502728+0000) 2022-01-31T20:48:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:10 smithi167 conmon[49112]: debug 2022-01-31T20:48:10.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:10.596286+0000) 2022-01-31T20:48:11.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:11 smithi167 conmon[54076]: debug 2022-01-31T20:48:11.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:11.267596+0000) 2022-01-31T20:48:11.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:11 smithi167 conmon[60316]: debug 2022-01-31T20:48:11.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:11.018421+0000) 2022-01-31T20:48:11.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:11 smithi171 conmon[46715]: debug 2022-01-31T20:48:11.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:11.295971+0000) 2022-01-31T20:48:11.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:11 smithi171 conmon[51620]: debug 2022-01-31T20:48:11.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:11.374546+0000) 2022-01-31T20:48:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:11 smithi171 conmon[41853]: debug 2022-01-31T20:48:11.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:11.502953+0000) 2022-01-31T20:48:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:11 smithi167 conmon[49112]: debug 2022-01-31T20:48:11.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:11.596418+0000) 2022-01-31T20:48:12.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:12 smithi167 conmon[54076]: debug 2022-01-31T20:48:12.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:12.267765+0000) 2022-01-31T20:48:12.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:12 smithi167 conmon[60316]: debug 2022-01-31T20:48:12.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:12.018604+0000) 2022-01-31T20:48:12.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:12 smithi171 conmon[46715]: debug 2022-01-31T20:48:12.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:12.296130+0000) 2022-01-31T20:48:12.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:12 smithi171 conmon[51620]: debug 2022-01-31T20:48:12.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:12.374679+0000) 2022-01-31T20:48:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:12 smithi171 conmon[41853]: debug 2022-01-31T20:48:12.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:12.503112+0000) 2022-01-31T20:48:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:12 smithi167 conmon[49112]: debug 2022-01-31T20:48:12.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:12.596550+0000) 2022-01-31T20:48:13.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:13 smithi167 conmon[54076]: debug 2022-01-31T20:48:13.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.267966+0000) 2022-01-31T20:48:13.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:13 smithi167 conmon[60316]: debug 2022-01-31T20:48:13.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.018778+0000) 2022-01-31T20:48:13.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[46715]: debug 2022-01-31T20:48:13.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.296292+0000) 2022-01-31T20:48:13.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[51620]: debug 2022-01-31T20:48:13.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.374809+0000) 2022-01-31T20:48:13.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:13 smithi167 conmon[54076]: debug 2022-01-31T20:48:13.430+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.431514+0000) 2022-01-31T20:48:13.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:13 smithi167 conmon[60316]: debug 2022-01-31T20:48:13.431+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.432863+0000) 2022-01-31T20:48:13.587 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:13 smithi167 conmon[49112]: debug 2022-01-31T20:48:13.430+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.432286+0000) 2022-01-31T20:48:13.710 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[35325]: debug 2022-01-31T20:48:13.439+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79520 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:13.710 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[41853]: debug 2022-01-31T20:48:13.431+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.432108+0000) 2022-01-31T20:48:13.711 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[41853]: debug 2022-01-31T20:48:13.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.503240+0000) 2022-01-31T20:48:13.711 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[46715]: debug 2022-01-31T20:48:13.430+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.431014+0000) 2022-01-31T20:48:13.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:13 smithi171 conmon[51620]: debug 2022-01-31T20:48:13.430+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.431713+0000) 2022-01-31T20:48:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:13 smithi167 conmon[49112]: debug 2022-01-31T20:48:13.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:13.596685+0000) 2022-01-31T20:48:14.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:14 smithi167 conmon[54076]: debug 2022-01-31T20:48:14.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:14.268134+0000) 2022-01-31T20:48:14.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:14 smithi167 conmon[60316]: debug 2022-01-31T20:48:14.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:14.018986+0000) 2022-01-31T20:48:14.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:14 smithi171 conmon[51620]: debug 2022-01-31T20:48:14.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:14.374927+0000) 2022-01-31T20:48:14.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:14 smithi171 conmon[46715]: debug 2022-01-31T20:48:14.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:14.296481+0000) 2022-01-31T20:48:14.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:14 smithi171 conmon[41853]: debug 2022-01-31T20:48:14.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:14.503407+0000) 2022-01-31T20:48:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:14 smithi167 conmon[49112]: debug 2022-01-31T20:48:14.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:14.596786+0000) 2022-01-31T20:48:15.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:15 smithi167 conmon[60316]: debug 2022-01-31T20:48:15.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:15.019124+0000) 2022-01-31T20:48:15.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:15 smithi167 conmon[54076]: debug 2022-01-31T20:48:15.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:15.268280+0000) 2022-01-31T20:48:15.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:15 smithi171 conmon[46715]: debug 2022-01-31T20:48:15.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:15.296644+0000) 2022-01-31T20:48:15.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:15 smithi171 conmon[51620]: debug 2022-01-31T20:48:15.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:15.375010+0000) 2022-01-31T20:48:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:15 smithi171 conmon[41853]: debug 2022-01-31T20:48:15.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:15.503607+0000) 2022-01-31T20:48:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:15 smithi167 conmon[49112]: debug 2022-01-31T20:48:15.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:15.596906+0000) 2022-01-31T20:48:16.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:16 smithi167 conmon[60316]: debug 2022-01-31T20:48:16.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:16.019279+0000) 2022-01-31T20:48:16.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:16 smithi167 conmon[54076]: debug 2022-01-31T20:48:16.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:16.268469+0000) 2022-01-31T20:48:16.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:16 smithi171 conmon[46715]: debug 2022-01-31T20:48:16.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:16.296831+0000) 2022-01-31T20:48:16.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:16 smithi171 conmon[51620]: debug 2022-01-31T20:48:16.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:16.375139+0000) 2022-01-31T20:48:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:16 smithi171 conmon[41853]: debug 2022-01-31T20:48:16.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:16.503787+0000) 2022-01-31T20:48:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:16 smithi167 conmon[49112]: debug 2022-01-31T20:48:16.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:16.597072+0000) 2022-01-31T20:48:17.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:17 smithi167 conmon[60316]: debug 2022-01-31T20:48:17.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:17.019474+0000) 2022-01-31T20:48:17.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:17 smithi167 conmon[54076]: debug 2022-01-31T20:48:17.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:17.268665+0000) 2022-01-31T20:48:17.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:17 smithi171 conmon[46715]: debug 2022-01-31T20:48:17.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:17.297033+0000) 2022-01-31T20:48:17.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:17 smithi171 conmon[51620]: debug 2022-01-31T20:48:17.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:17.375309+0000) 2022-01-31T20:48:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:17 smithi171 conmon[41853]: debug 2022-01-31T20:48:17.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:17.503932+0000) 2022-01-31T20:48:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:17 smithi167 conmon[49112]: debug 2022-01-31T20:48:17.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:17.597273+0000) 2022-01-31T20:48:18.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:18 smithi167 conmon[60316]: debug 2022-01-31T20:48:18.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.019717+0000) 2022-01-31T20:48:18.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:18 smithi167 conmon[54076]: debug 2022-01-31T20:48:18.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.268873+0000) 2022-01-31T20:48:18.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[46715]: debug 2022-01-31T20:48:18.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.297248+0000) 2022-01-31T20:48:18.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[51620]: debug 2022-01-31T20:48:18.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.375455+0000) 2022-01-31T20:48:18.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[51620]: debug 2022-01-31T20:48:18.441+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.441921+0000) 2022-01-31T20:48:18.587 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:18 smithi167 conmon[49112]: debug 2022-01-31T20:48:18.441+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.443005+0000) 2022-01-31T20:48:18.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:18 smithi167 conmon[54076]: debug 2022-01-31T20:48:18.441+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.442813+0000) 2022-01-31T20:48:18.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:18 smithi167 conmon[60316]: debug 2022-01-31T20:48:18.441+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.443214+0000) 2022-01-31T20:48:18.711 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[35325]: debug 2022-01-31T20:48:18.450+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79630 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:18.711 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[46715]: debug 2022-01-31T20:48:18.441+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.442227+0000) 2022-01-31T20:48:18.712 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[41853]: debug 2022-01-31T20:48:18.442+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.443391+0000) 2022-01-31T20:48:18.712 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:18 smithi171 conmon[41853]: debug 2022-01-31T20:48:18.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.504113+0000) 2022-01-31T20:48:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:18 smithi167 conmon[49112]: debug 2022-01-31T20:48:18.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:18.597456+0000) 2022-01-31T20:48:19.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:19 smithi167 conmon[60316]: debug 2022-01-31T20:48:19.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:19.019908+0000) 2022-01-31T20:48:19.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:19 smithi167 conmon[54076]: debug 2022-01-31T20:48:19.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:19.269060+0000) 2022-01-31T20:48:19.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:19 smithi171 conmon[46715]: debug 2022-01-31T20:48:19.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:19.297414+0000) 2022-01-31T20:48:19.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:19 smithi171 conmon[51620]: debug 2022-01-31T20:48:19.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:19.375628+0000) 2022-01-31T20:48:19.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:19 smithi171 conmon[41853]: debug 2022-01-31T20:48:19.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:19.504249+0000) 2022-01-31T20:48:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:19 smithi167 conmon[49112]: debug 2022-01-31T20:48:19.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:19.597583+0000) 2022-01-31T20:48:20.271 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:20 smithi167 conmon[54076]: debug 2022-01-31T20:48:20.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:20.269210+0000) 2022-01-31T20:48:20.271 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:20 smithi167 conmon[60316]: debug 2022-01-31T20:48:20.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:20.020087+0000) 2022-01-31T20:48:20.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:20 smithi171 conmon[46715]: debug 2022-01-31T20:48:20.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:20.297566+0000) 2022-01-31T20:48:20.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:20 smithi171 conmon[51620]: debug 2022-01-31T20:48:20.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:20.375794+0000) 2022-01-31T20:48:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:20 smithi171 conmon[41853]: debug 2022-01-31T20:48:20.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:20.504404+0000) 2022-01-31T20:48:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:20 smithi167 conmon[49112]: debug 2022-01-31T20:48:20.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:20.597698+0000) 2022-01-31T20:48:21.271 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:21 smithi167 conmon[54076]: debug 2022-01-31T20:48:21.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:21.269396+0000) 2022-01-31T20:48:21.271 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:21 smithi167 conmon[60316]: debug 2022-01-31T20:48:21.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:21.020267+0000) 2022-01-31T20:48:21.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:21 smithi171 conmon[46715]: debug 2022-01-31T20:48:21.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:21.297766+0000) 2022-01-31T20:48:21.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:21 smithi171 conmon[51620]: debug 2022-01-31T20:48:21.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:21.375966+0000) 2022-01-31T20:48:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:21 smithi171 conmon[41853]: debug 2022-01-31T20:48:21.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:21.504564+0000) 2022-01-31T20:48:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:21 smithi167 conmon[49112]: debug 2022-01-31T20:48:21.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:21.597828+0000) 2022-01-31T20:48:22.271 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:22 smithi167 conmon[54076]: debug 2022-01-31T20:48:22.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:22.269582+0000) 2022-01-31T20:48:22.272 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:22 smithi167 conmon[60316]: debug 2022-01-31T20:48:22.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:22.020424+0000) 2022-01-31T20:48:22.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:22 smithi171 conmon[46715]: debug 2022-01-31T20:48:22.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:22.297960+0000) 2022-01-31T20:48:22.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:22 smithi171 conmon[51620]: debug 2022-01-31T20:48:22.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:22.376137+0000) 2022-01-31T20:48:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:22 smithi171 conmon[41853]: debug 2022-01-31T20:48:22.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:22.504706+0000) 2022-01-31T20:48:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:22 smithi167 conmon[49112]: debug 2022-01-31T20:48:22.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:22.598003+0000) 2022-01-31T20:48:23.271 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:23 smithi167 conmon[54076]: debug 2022-01-31T20:48:23.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.269745+0000) 2022-01-31T20:48:23.272 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:23 smithi167 conmon[60316]: debug 2022-01-31T20:48:23.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.020618+0000) 2022-01-31T20:48:23.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[46715]: debug 2022-01-31T20:48:23.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.298123+0000) 2022-01-31T20:48:23.451 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[51620]: debug 2022-01-31T20:48:23.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.376332+0000) 2022-01-31T20:48:23.588 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:23 smithi167 conmon[49112]: debug 2022-01-31T20:48:23.459+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.461095+0000) 2022-01-31T20:48:23.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:23 smithi167 conmon[54076]: debug 2022-01-31T20:48:23.452+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.454123+0000) 2022-01-31T20:48:23.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:23 smithi167 conmon[60316]: debug 2022-01-31T20:48:23.459+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.461127+0000) 2022-01-31T20:48:23.711 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[46715]: debug 2022-01-31T20:48:23.451+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.452411+0000) 2022-01-31T20:48:23.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[51620]: debug 2022-01-31T20:48:23.452+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.453389+0000) 2022-01-31T20:48:23.713 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[35325]: debug 2022-01-31T20:48:23.462+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79739 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:23.713 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[35325]: debug 2022-01-31T20:48:23.574+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:48:23.713 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[41853]: debug 2022-01-31T20:48:23.453+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.454129+0000) 2022-01-31T20:48:23.713 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:23 smithi171 conmon[41853]: debug 2022-01-31T20:48:23.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.504804+0000) 2022-01-31T20:48:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:23 smithi167 conmon[49112]: debug 2022-01-31T20:48:23.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:23.598202+0000) 2022-01-31T20:48:24.272 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:24 smithi167 conmon[54076]: debug 2022-01-31T20:48:24.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:24.269848+0000) 2022-01-31T20:48:24.272 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:24 smithi167 conmon[60316]: debug 2022-01-31T20:48:24.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:24.020816+0000) 2022-01-31T20:48:24.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:24 smithi171 conmon[46715]: debug 2022-01-31T20:48:24.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:24.298284+0000) 2022-01-31T20:48:24.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:24 smithi171 conmon[51620]: debug 2022-01-31T20:48:24.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:24.376487+0000) 2022-01-31T20:48:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:24 smithi171 conmon[41853]: debug 2022-01-31T20:48:24.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:24.504942+0000) 2022-01-31T20:48:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:24 smithi167 conmon[49112]: debug 2022-01-31T20:48:24.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:24.598379+0000) 2022-01-31T20:48:25.275 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:25 smithi167 conmon[54076]: debug 2022-01-31T20:48:25.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:25.269987+0000) 2022-01-31T20:48:25.275 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:25 smithi167 conmon[60316]: debug 2022-01-31T20:48:25.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:25.020980+0000) 2022-01-31T20:48:25.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:25 smithi171 conmon[46715]: debug 2022-01-31T20:48:25.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:25.298417+0000) 2022-01-31T20:48:25.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:25 smithi171 conmon[51620]: debug 2022-01-31T20:48:25.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:25.376662+0000) 2022-01-31T20:48:25.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:25 smithi171 conmon[41853]: debug 2022-01-31T20:48:25.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:25.505088+0000) 2022-01-31T20:48:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:25 smithi167 conmon[49112]: debug 2022-01-31T20:48:25.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:25.598579+0000) 2022-01-31T20:48:26.275 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:26 smithi167 conmon[54076]: debug 2022-01-31T20:48:26.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:26.270171+0000) 2022-01-31T20:48:26.276 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:26 smithi167 conmon[60316]: debug 2022-01-31T20:48:26.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:26.021137+0000) 2022-01-31T20:48:26.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:26 smithi171 conmon[46715]: debug 2022-01-31T20:48:26.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:26.298578+0000) 2022-01-31T20:48:26.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:26 smithi171 conmon[51620]: debug 2022-01-31T20:48:26.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:26.376836+0000) 2022-01-31T20:48:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:26 smithi171 conmon[41853]: debug 2022-01-31T20:48:26.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:26.505240+0000) 2022-01-31T20:48:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:26 smithi167 conmon[49112]: debug 2022-01-31T20:48:26.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:26.598784+0000) 2022-01-31T20:48:27.275 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:27 smithi167 conmon[54076]: debug 2022-01-31T20:48:27.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:27.270322+0000) 2022-01-31T20:48:27.276 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:27 smithi167 conmon[60316]: debug 2022-01-31T20:48:27.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:27.021338+0000) 2022-01-31T20:48:27.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:27 smithi171 conmon[46715]: debug 2022-01-31T20:48:27.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:27.298773+0000) 2022-01-31T20:48:27.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:27 smithi171 conmon[51620]: debug 2022-01-31T20:48:27.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:27.377018+0000) 2022-01-31T20:48:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:27 smithi171 conmon[41853]: debug 2022-01-31T20:48:27.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:27.505440+0000) 2022-01-31T20:48:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:27 smithi167 conmon[49112]: debug 2022-01-31T20:48:27.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:27.598983+0000) 2022-01-31T20:48:28.276 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:28 smithi167 conmon[54076]: debug 2022-01-31T20:48:28.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.270591+0000) 2022-01-31T20:48:28.276 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:28 smithi167 conmon[60316]: debug 2022-01-31T20:48:28.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.021511+0000) 2022-01-31T20:48:28.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[46715]: debug 2022-01-31T20:48:28.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.298896+0000) 2022-01-31T20:48:28.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[51620]: debug 2022-01-31T20:48:28.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.377211+0000) 2022-01-31T20:48:28.589 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:28 smithi167 conmon[49112]: debug 2022-01-31T20:48:28.465+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.467316+0000) 2022-01-31T20:48:28.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:28 smithi167 conmon[54076]: debug 2022-01-31T20:48:28.465+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.466791+0000) 2022-01-31T20:48:28.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:28 smithi167 conmon[60316]: debug 2022-01-31T20:48:28.465+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.467421+0000) 2022-01-31T20:48:28.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[35325]: debug 2022-01-31T20:48:28.473+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79849 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:28.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[46715]: debug 2022-01-31T20:48:28.465+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.465974+0000) 2022-01-31T20:48:28.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[51620]: debug 2022-01-31T20:48:28.465+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.466278+0000) 2022-01-31T20:48:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[41853]: debug 2022-01-31T20:48:28.466+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.466939+0000) 2022-01-31T20:48:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:28 smithi171 conmon[41853]: debug 2022-01-31T20:48:28.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.505648+0000) 2022-01-31T20:48:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:28 smithi167 conmon[49112]: debug 2022-01-31T20:48:28.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:28.599197+0000) 2022-01-31T20:48:29.276 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:29 smithi167 conmon[54076]: debug 2022-01-31T20:48:29.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:29.270792+0000) 2022-01-31T20:48:29.276 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:29 smithi167 conmon[60316]: debug 2022-01-31T20:48:29.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:29.021667+0000) 2022-01-31T20:48:29.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:29 smithi171 conmon[46715]: debug 2022-01-31T20:48:29.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:29.299063+0000) 2022-01-31T20:48:29.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:29 smithi171 conmon[51620]: debug 2022-01-31T20:48:29.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:29.377371+0000) 2022-01-31T20:48:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:29 smithi171 conmon[41853]: debug 2022-01-31T20:48:29.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:29.505801+0000) 2022-01-31T20:48:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:29 smithi167 conmon[49112]: debug 2022-01-31T20:48:29.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:29.599388+0000) 2022-01-31T20:48:30.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:30 smithi167 conmon[54076]: debug 2022-01-31T20:48:30.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:30.270904+0000) 2022-01-31T20:48:30.280 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:30 smithi167 conmon[60316]: debug 2022-01-31T20:48:30.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:30.021843+0000) 2022-01-31T20:48:30.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:30 smithi171 conmon[46715]: debug 2022-01-31T20:48:30.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:30.299239+0000) 2022-01-31T20:48:30.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:30 smithi171 conmon[51620]: debug 2022-01-31T20:48:30.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:30.377525+0000) 2022-01-31T20:48:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:30 smithi171 conmon[41853]: debug 2022-01-31T20:48:30.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:30.505961+0000) 2022-01-31T20:48:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:30 smithi167 conmon[49112]: debug 2022-01-31T20:48:30.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:30.599581+0000) 2022-01-31T20:48:31.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:31 smithi167 conmon[54076]: debug 2022-01-31T20:48:31.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:31.271083+0000) 2022-01-31T20:48:31.280 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:31 smithi167 conmon[60316]: debug 2022-01-31T20:48:31.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:31.022027+0000) 2022-01-31T20:48:31.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:31 smithi171 conmon[46715]: debug 2022-01-31T20:48:31.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:31.299384+0000) 2022-01-31T20:48:31.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:31 smithi171 conmon[51620]: debug 2022-01-31T20:48:31.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:31.377730+0000) 2022-01-31T20:48:31.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:31 smithi171 conmon[41853]: debug 2022-01-31T20:48:31.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:31.506092+0000) 2022-01-31T20:48:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:31 smithi167 conmon[49112]: debug 2022-01-31T20:48:31.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:31.599730+0000) 2022-01-31T20:48:32.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:32 smithi167 conmon[54076]: debug 2022-01-31T20:48:32.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:32.271238+0000) 2022-01-31T20:48:32.280 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:32 smithi167 conmon[60316]: debug 2022-01-31T20:48:32.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:32.022247+0000) 2022-01-31T20:48:32.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:32 smithi171 conmon[46715]: debug 2022-01-31T20:48:32.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:32.299569+0000) 2022-01-31T20:48:32.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:32 smithi171 conmon[51620]: debug 2022-01-31T20:48:32.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:32.377849+0000) 2022-01-31T20:48:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:32 smithi171 conmon[41853]: debug 2022-01-31T20:48:32.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:32.506277+0000) 2022-01-31T20:48:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:32 smithi167 conmon[49112]: debug 2022-01-31T20:48:32.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:32.599865+0000) 2022-01-31T20:48:33.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:33 smithi167 conmon[54076]: debug 2022-01-31T20:48:33.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.271428+0000) 2022-01-31T20:48:33.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:33 smithi167 conmon[60316]: debug 2022-01-31T20:48:33.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.022446+0000) 2022-01-31T20:48:33.474 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[46715]: debug 2022-01-31T20:48:33.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.299787+0000) 2022-01-31T20:48:33.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[51620]: debug 2022-01-31T20:48:33.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.378003+0000) 2022-01-31T20:48:33.590 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:33 smithi167 conmon[49112]: debug 2022-01-31T20:48:33.476+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.477647+0000) 2022-01-31T20:48:33.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:33 smithi167 conmon[54076]: debug 2022-01-31T20:48:33.476+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.477526+0000) 2022-01-31T20:48:33.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:33 smithi167 conmon[60316]: debug 2022-01-31T20:48:33.475+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.477405+0000) 2022-01-31T20:48:33.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[46715]: debug 2022-01-31T20:48:33.476+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.476912+0000) 2022-01-31T20:48:33.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[51620]: debug 2022-01-31T20:48:33.476+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.477200+0000) 2022-01-31T20:48:33.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[35325]: debug 2022-01-31T20:48:33.484+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 79958 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[41853]: debug 2022-01-31T20:48:33.477+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.477781+0000) 2022-01-31T20:48:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:33 smithi171 conmon[41853]: debug 2022-01-31T20:48:33.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.506479+0000) 2022-01-31T20:48:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:33 smithi167 conmon[49112]: debug 2022-01-31T20:48:33.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:33.600079+0000) 2022-01-31T20:48:34.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:34 smithi167 conmon[54076]: debug 2022-01-31T20:48:34.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:34.271635+0000) 2022-01-31T20:48:34.281 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:34 smithi167 conmon[60316]: debug 2022-01-31T20:48:34.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:34.022631+0000) 2022-01-31T20:48:34.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:34 smithi171 conmon[46715]: debug 2022-01-31T20:48:34.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:34.299973+0000) 2022-01-31T20:48:34.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:34 smithi171 conmon[51620]: debug 2022-01-31T20:48:34.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:34.378191+0000) 2022-01-31T20:48:34.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:34 smithi171 conmon[41853]: debug 2022-01-31T20:48:34.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:34.506644+0000) 2022-01-31T20:48:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:34 smithi167 conmon[49112]: debug 2022-01-31T20:48:34.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:34.600256+0000) 2022-01-31T20:48:35.283 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:35 smithi167 conmon[54076]: debug 2022-01-31T20:48:35.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:35.271785+0000) 2022-01-31T20:48:35.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:35 smithi167 conmon[60316]: debug 2022-01-31T20:48:35.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:35.022781+0000) 2022-01-31T20:48:35.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:35 smithi171 conmon[46715]: debug 2022-01-31T20:48:35.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:35.300131+0000) 2022-01-31T20:48:35.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:35 smithi171 conmon[51620]: debug 2022-01-31T20:48:35.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:35.378344+0000) 2022-01-31T20:48:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:35 smithi171 conmon[41853]: debug 2022-01-31T20:48:35.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:35.506799+0000) 2022-01-31T20:48:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:35 smithi167 conmon[49112]: debug 2022-01-31T20:48:35.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:35.600489+0000) 2022-01-31T20:48:36.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:36 smithi167 conmon[54076]: debug 2022-01-31T20:48:36.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:36.271903+0000) 2022-01-31T20:48:36.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:36 smithi167 conmon[60316]: debug 2022-01-31T20:48:36.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:36.022935+0000) 2022-01-31T20:48:36.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:36 smithi171 conmon[51620]: debug 2022-01-31T20:48:36.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:36.378528+0000) 2022-01-31T20:48:36.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:36 smithi171 conmon[46715]: debug 2022-01-31T20:48:36.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:36.300339+0000) 2022-01-31T20:48:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:36 smithi171 conmon[41853]: debug 2022-01-31T20:48:36.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:36.506994+0000) 2022-01-31T20:48:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:36 smithi167 conmon[49112]: debug 2022-01-31T20:48:36.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:36.600654+0000) 2022-01-31T20:48:37.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:37 smithi167 conmon[54076]: debug 2022-01-31T20:48:37.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:37.272096+0000) 2022-01-31T20:48:37.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:37 smithi167 conmon[60316]: debug 2022-01-31T20:48:37.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:37.023112+0000) 2022-01-31T20:48:37.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:37 smithi171 conmon[46715]: debug 2022-01-31T20:48:37.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:37.300544+0000) 2022-01-31T20:48:37.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:37 smithi171 conmon[51620]: debug 2022-01-31T20:48:37.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:37.378683+0000) 2022-01-31T20:48:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:37 smithi171 conmon[41853]: debug 2022-01-31T20:48:37.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:37.507181+0000) 2022-01-31T20:48:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:37 smithi167 conmon[49112]: debug 2022-01-31T20:48:37.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:37.600801+0000) 2022-01-31T20:48:38.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:38 smithi167 conmon[60316]: debug 2022-01-31T20:48:38.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.023295+0000) 2022-01-31T20:48:38.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:38 smithi167 conmon[54076]: debug 2022-01-31T20:48:38.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.272315+0000) 2022-01-31T20:48:38.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[46715]: debug 2022-01-31T20:48:38.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.300698+0000) 2022-01-31T20:48:38.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[51620]: debug 2022-01-31T20:48:38.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.378848+0000) 2022-01-31T20:48:38.591 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:38 smithi167 conmon[49112]: debug 2022-01-31T20:48:38.486+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.487193+0000) 2022-01-31T20:48:38.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:38 smithi167 conmon[54076]: debug 2022-01-31T20:48:38.487+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.488417+0000) 2022-01-31T20:48:38.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:38 smithi167 conmon[60316]: debug 2022-01-31T20:48:38.487+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.487522+0000) 2022-01-31T20:48:38.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[46715]: debug 2022-01-31T20:48:38.486+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.487334+0000) 2022-01-31T20:48:38.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[51620]: debug 2022-01-31T20:48:38.488+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.488888+0000) 2022-01-31T20:48:38.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[35325]: debug 2022-01-31T20:48:38.494+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80068 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:38.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[35325]: debug 2022-01-31T20:48:38.575+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:48:38.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[41853]: debug 2022-01-31T20:48:38.486+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.487663+0000) 2022-01-31T20:48:38.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:38 smithi171 conmon[41853]: debug 2022-01-31T20:48:38.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.507382+0000) 2022-01-31T20:48:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:38 smithi167 conmon[49112]: debug 2022-01-31T20:48:38.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:38.600987+0000) 2022-01-31T20:48:39.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:39 smithi167 conmon[54076]: debug 2022-01-31T20:48:39.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:39.272533+0000) 2022-01-31T20:48:39.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:39 smithi167 conmon[60316]: debug 2022-01-31T20:48:39.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:39.023476+0000) 2022-01-31T20:48:39.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:39 smithi171 conmon[46715]: debug 2022-01-31T20:48:39.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:39.300891+0000) 2022-01-31T20:48:39.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:39 smithi171 conmon[51620]: debug 2022-01-31T20:48:39.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:39.379072+0000) 2022-01-31T20:48:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:39 smithi171 conmon[41853]: debug 2022-01-31T20:48:39.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:39.507550+0000) 2022-01-31T20:48:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:39 smithi167 conmon[49112]: debug 2022-01-31T20:48:39.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:39.601144+0000) 2022-01-31T20:48:40.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:40 smithi167 conmon[54076]: debug 2022-01-31T20:48:40.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:40.272685+0000) 2022-01-31T20:48:40.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:40 smithi167 conmon[60316]: debug 2022-01-31T20:48:40.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:40.023675+0000) 2022-01-31T20:48:40.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:40 smithi171 conmon[46715]: debug 2022-01-31T20:48:40.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:40.301081+0000) 2022-01-31T20:48:40.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:40 smithi171 conmon[51620]: debug 2022-01-31T20:48:40.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:40.379235+0000) 2022-01-31T20:48:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:40 smithi171 conmon[41853]: debug 2022-01-31T20:48:40.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:40.507748+0000) 2022-01-31T20:48:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:40 smithi167 conmon[49112]: debug 2022-01-31T20:48:40.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:40.601326+0000) 2022-01-31T20:48:41.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:41 smithi167 conmon[54076]: debug 2022-01-31T20:48:41.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:41.272884+0000) 2022-01-31T20:48:41.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:41 smithi167 conmon[60316]: debug 2022-01-31T20:48:41.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:41.023857+0000) 2022-01-31T20:48:41.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:41 smithi171 conmon[46715]: debug 2022-01-31T20:48:41.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:41.301238+0000) 2022-01-31T20:48:41.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:41 smithi171 conmon[51620]: debug 2022-01-31T20:48:41.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:41.379429+0000) 2022-01-31T20:48:41.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:41 smithi171 conmon[41853]: debug 2022-01-31T20:48:41.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:41.507917+0000) 2022-01-31T20:48:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:41 smithi167 conmon[49112]: debug 2022-01-31T20:48:41.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:41.601424+0000) 2022-01-31T20:48:42.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:42 smithi167 conmon[60316]: debug 2022-01-31T20:48:42.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:42.024057+0000) 2022-01-31T20:48:42.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:42 smithi167 conmon[54076]: debug 2022-01-31T20:48:42.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:42.273035+0000) 2022-01-31T20:48:42.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:42 smithi171 conmon[46715]: debug 2022-01-31T20:48:42.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:42.301398+0000) 2022-01-31T20:48:42.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:42 smithi171 conmon[51620]: debug 2022-01-31T20:48:42.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:42.379626+0000) 2022-01-31T20:48:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:42 smithi171 conmon[41853]: debug 2022-01-31T20:48:42.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:42.508117+0000) 2022-01-31T20:48:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:42 smithi167 conmon[49112]: debug 2022-01-31T20:48:42.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:42.601525+0000) 2022-01-31T20:48:43.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:43 smithi167 conmon[54076]: debug 2022-01-31T20:48:43.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.273237+0000) 2022-01-31T20:48:43.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:43 smithi167 conmon[60316]: debug 2022-01-31T20:48:43.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.024257+0000) 2022-01-31T20:48:43.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[46715]: debug 2022-01-31T20:48:43.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.301578+0000) 2022-01-31T20:48:43.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[51620]: debug 2022-01-31T20:48:43.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.379850+0000) 2022-01-31T20:48:43.592 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:43 smithi167 conmon[49112]: debug 2022-01-31T20:48:43.498+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.499246+0000) 2022-01-31T20:48:43.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:43 smithi167 conmon[54076]: debug 2022-01-31T20:48:43.498+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.498774+0000) 2022-01-31T20:48:43.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:43 smithi167 conmon[60316]: debug 2022-01-31T20:48:43.497+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.498369+0000) 2022-01-31T20:48:43.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[35325]: debug 2022-01-31T20:48:43.505+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80177 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:43.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[46715]: debug 2022-01-31T20:48:43.497+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.498421+0000) 2022-01-31T20:48:43.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[51620]: debug 2022-01-31T20:48:43.498+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.499264+0000) 2022-01-31T20:48:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[41853]: debug 2022-01-31T20:48:43.497+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.498082+0000) 2022-01-31T20:48:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:43 smithi171 conmon[41853]: debug 2022-01-31T20:48:43.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.508220+0000) 2022-01-31T20:48:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:43 smithi167 conmon[49112]: debug 2022-01-31T20:48:43.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:43.601660+0000) 2022-01-31T20:48:44.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:44 smithi167 conmon[54076]: debug 2022-01-31T20:48:44.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:44.273399+0000) 2022-01-31T20:48:44.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:44 smithi167 conmon[60316]: debug 2022-01-31T20:48:44.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:44.024608+0000) 2022-01-31T20:48:44.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:44 smithi171 conmon[46715]: debug 2022-01-31T20:48:44.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:44.301744+0000) 2022-01-31T20:48:44.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:44 smithi171 conmon[51620]: debug 2022-01-31T20:48:44.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:44.380047+0000) 2022-01-31T20:48:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:44 smithi171 conmon[41853]: debug 2022-01-31T20:48:44.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:44.508376+0000) 2022-01-31T20:48:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:44 smithi167 conmon[49112]: debug 2022-01-31T20:48:44.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:44.601778+0000) 2022-01-31T20:48:45.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:45 smithi167 conmon[54076]: debug 2022-01-31T20:48:45.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:45.273545+0000) 2022-01-31T20:48:45.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:45 smithi167 conmon[60316]: debug 2022-01-31T20:48:45.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:45.024718+0000) 2022-01-31T20:48:45.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:45 smithi171 conmon[46715]: debug 2022-01-31T20:48:45.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:45.301939+0000) 2022-01-31T20:48:45.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:45 smithi171 conmon[51620]: debug 2022-01-31T20:48:45.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:45.380227+0000) 2022-01-31T20:48:45.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:45 smithi171 conmon[41853]: debug 2022-01-31T20:48:45.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:45.508553+0000) 2022-01-31T20:48:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:45 smithi167 conmon[49112]: debug 2022-01-31T20:48:45.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:45.601941+0000) 2022-01-31T20:48:46.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:46 smithi167 conmon[54076]: debug 2022-01-31T20:48:46.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:46.273693+0000) 2022-01-31T20:48:46.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:46 smithi167 conmon[60316]: debug 2022-01-31T20:48:46.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:46.024909+0000) 2022-01-31T20:48:46.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:46 smithi171 conmon[46715]: debug 2022-01-31T20:48:46.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:46.302117+0000) 2022-01-31T20:48:46.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:46 smithi171 conmon[51620]: debug 2022-01-31T20:48:46.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:46.380442+0000) 2022-01-31T20:48:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:46 smithi171 conmon[41853]: debug 2022-01-31T20:48:46.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:46.508769+0000) 2022-01-31T20:48:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:46 smithi167 conmon[49112]: debug 2022-01-31T20:48:46.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:46.602142+0000) 2022-01-31T20:48:47.292 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:47 smithi167 conmon[54076]: debug 2022-01-31T20:48:47.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:47.273820+0000) 2022-01-31T20:48:47.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:47 smithi167 conmon[60316]: debug 2022-01-31T20:48:47.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:47.025113+0000) 2022-01-31T20:48:47.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:47 smithi171 conmon[46715]: debug 2022-01-31T20:48:47.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:47.302326+0000) 2022-01-31T20:48:47.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:47 smithi171 conmon[51620]: debug 2022-01-31T20:48:47.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:47.380570+0000) 2022-01-31T20:48:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:47 smithi171 conmon[41853]: debug 2022-01-31T20:48:47.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:47.508964+0000) 2022-01-31T20:48:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:47 smithi167 conmon[49112]: debug 2022-01-31T20:48:47.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:47.602360+0000) 2022-01-31T20:48:48.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:48 smithi167 conmon[60316]: debug 2022-01-31T20:48:48.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.025296+0000) 2022-01-31T20:48:48.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:48 smithi167 conmon[54076]: debug 2022-01-31T20:48:48.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.274057+0000) 2022-01-31T20:48:48.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[46715]: debug 2022-01-31T20:48:48.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.302564+0000) 2022-01-31T20:48:48.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[51620]: debug 2022-01-31T20:48:48.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.380789+0000) 2022-01-31T20:48:48.593 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:48 smithi167 conmon[49112]: debug 2022-01-31T20:48:48.508+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.509396+0000) 2022-01-31T20:48:48.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:48 smithi167 conmon[54076]: debug 2022-01-31T20:48:48.509+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.509645+0000) 2022-01-31T20:48:48.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:48 smithi167 conmon[60316]: debug 2022-01-31T20:48:48.508+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.508736+0000) 2022-01-31T20:48:48.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[35325]: debug 2022-01-31T20:48:48.515+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80287 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:48.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[46715]: debug 2022-01-31T20:48:48.506+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.508412+0000) 2022-01-31T20:48:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[41853]: debug 2022-01-31T20:48:48.506+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.508569+0000) 2022-01-31T20:48:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[41853]: debug 2022-01-31T20:48:48.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.509182+0000) 2022-01-31T20:48:48.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:48 smithi171 conmon[51620]: debug 2022-01-31T20:48:48.507+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.508930+0000) 2022-01-31T20:48:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:48 smithi167 conmon[49112]: debug 2022-01-31T20:48:48.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:48.602563+0000) 2022-01-31T20:48:49.293 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:49 smithi167 conmon[60316]: debug 2022-01-31T20:48:49.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:49.025466+0000) 2022-01-31T20:48:49.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:49 smithi167 conmon[54076]: debug 2022-01-31T20:48:49.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:49.274214+0000) 2022-01-31T20:48:49.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:49 smithi171 conmon[46715]: debug 2022-01-31T20:48:49.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:49.302729+0000) 2022-01-31T20:48:49.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:49 smithi171 conmon[51620]: debug 2022-01-31T20:48:49.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:49.380954+0000) 2022-01-31T20:48:49.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:49 smithi171 conmon[41853]: debug 2022-01-31T20:48:49.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:49.509335+0000) 2022-01-31T20:48:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:49 smithi167 conmon[49112]: debug 2022-01-31T20:48:49.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:49.602670+0000) 2022-01-31T20:48:50.296 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:50 smithi167 conmon[54076]: debug 2022-01-31T20:48:50.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:50.274371+0000) 2022-01-31T20:48:50.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:50 smithi167 conmon[60316]: debug 2022-01-31T20:48:50.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:50.025665+0000) 2022-01-31T20:48:50.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:50 smithi171 conmon[46715]: debug 2022-01-31T20:48:50.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:50.302900+0000) 2022-01-31T20:48:50.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:50 smithi171 conmon[51620]: debug 2022-01-31T20:48:50.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:50.381068+0000) 2022-01-31T20:48:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:50 smithi171 conmon[41853]: debug 2022-01-31T20:48:50.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:50.509471+0000) 2022-01-31T20:48:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:50 smithi167 conmon[49112]: debug 2022-01-31T20:48:50.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:50.602866+0000) 2022-01-31T20:48:51.296 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:51 smithi167 conmon[60316]: debug 2022-01-31T20:48:51.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:51.025824+0000) 2022-01-31T20:48:51.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:51 smithi167 conmon[54076]: debug 2022-01-31T20:48:51.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:51.274558+0000) 2022-01-31T20:48:51.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:51 smithi171 conmon[46715]: debug 2022-01-31T20:48:51.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:51.303059+0000) 2022-01-31T20:48:51.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:51 smithi171 conmon[51620]: debug 2022-01-31T20:48:51.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:51.381253+0000) 2022-01-31T20:48:51.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:51 smithi171 conmon[41853]: debug 2022-01-31T20:48:51.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:51.509622+0000) 2022-01-31T20:48:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:51 smithi167 conmon[49112]: debug 2022-01-31T20:48:51.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:51.603071+0000) 2022-01-31T20:48:52.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:52 smithi167 conmon[60316]: debug 2022-01-31T20:48:52.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:52.025959+0000) 2022-01-31T20:48:52.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:52 smithi167 conmon[54076]: debug 2022-01-31T20:48:52.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:52.274750+0000) 2022-01-31T20:48:52.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:52 smithi171 conmon[46715]: debug 2022-01-31T20:48:52.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:52.303193+0000) 2022-01-31T20:48:52.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:52 smithi171 conmon[51620]: debug 2022-01-31T20:48:52.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:52.381445+0000) 2022-01-31T20:48:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:52 smithi171 conmon[41853]: debug 2022-01-31T20:48:52.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:52.509807+0000) 2022-01-31T20:48:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:52 smithi167 conmon[49112]: debug 2022-01-31T20:48:52.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:52.603244+0000) 2022-01-31T20:48:53.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:53 smithi167 conmon[54076]: debug 2022-01-31T20:48:53.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.274869+0000) 2022-01-31T20:48:53.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:53 smithi167 conmon[60316]: debug 2022-01-31T20:48:53.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.026075+0000) 2022-01-31T20:48:53.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[46715]: debug 2022-01-31T20:48:53.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.303375+0000) 2022-01-31T20:48:53.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[51620]: debug 2022-01-31T20:48:53.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.381659+0000) 2022-01-31T20:48:53.593 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:53 smithi167 conmon[49112]: debug 2022-01-31T20:48:53.519+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.520048+0000) 2022-01-31T20:48:53.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:53 smithi167 conmon[54076]: debug 2022-01-31T20:48:53.519+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.520188+0000) 2022-01-31T20:48:53.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:53 smithi167 conmon[60316]: debug 2022-01-31T20:48:53.519+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.519789+0000) 2022-01-31T20:48:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[35325]: debug 2022-01-31T20:48:53.526+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80396 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[35325]: debug 2022-01-31T20:48:53.575+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:48:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[41853]: debug 2022-01-31T20:48:53.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.509972+0000) 2022-01-31T20:48:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[41853]: debug 2022-01-31T20:48:53.517+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.519637+0000) 2022-01-31T20:48:53.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[46715]: debug 2022-01-31T20:48:53.518+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.520547+0000) 2022-01-31T20:48:53.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:53 smithi171 conmon[51620]: debug 2022-01-31T20:48:53.518+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.519866+0000) 2022-01-31T20:48:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:53 smithi167 conmon[49112]: debug 2022-01-31T20:48:53.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:53.603476+0000) 2022-01-31T20:48:54.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:54 smithi167 conmon[54076]: debug 2022-01-31T20:48:54.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:54.275066+0000) 2022-01-31T20:48:54.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:54 smithi167 conmon[60316]: debug 2022-01-31T20:48:54.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:54.026239+0000) 2022-01-31T20:48:54.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:54 smithi171 conmon[46715]: debug 2022-01-31T20:48:54.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:54.303567+0000) 2022-01-31T20:48:54.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:54 smithi171 conmon[51620]: debug 2022-01-31T20:48:54.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:54.381852+0000) 2022-01-31T20:48:54.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:54 smithi171 conmon[41853]: debug 2022-01-31T20:48:54.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:54.510120+0000) 2022-01-31T20:48:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:54 smithi167 conmon[49112]: debug 2022-01-31T20:48:54.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:54.603605+0000) 2022-01-31T20:48:55.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:55 smithi167 conmon[54076]: debug 2022-01-31T20:48:55.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:55.275187+0000) 2022-01-31T20:48:55.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:55 smithi167 conmon[60316]: debug 2022-01-31T20:48:55.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:55.026388+0000) 2022-01-31T20:48:55.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:55 smithi171 conmon[46715]: debug 2022-01-31T20:48:55.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:55.304018+0000) 2022-01-31T20:48:55.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:55 smithi171 conmon[51620]: debug 2022-01-31T20:48:55.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:55.381990+0000) 2022-01-31T20:48:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:55 smithi171 conmon[41853]: debug 2022-01-31T20:48:55.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:55.510276+0000) 2022-01-31T20:48:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:55 smithi167 conmon[49112]: debug 2022-01-31T20:48:55.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:55.603743+0000) 2022-01-31T20:48:56.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:56 smithi167 conmon[60316]: debug 2022-01-31T20:48:56.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:56.026602+0000) 2022-01-31T20:48:56.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:56 smithi167 conmon[54076]: debug 2022-01-31T20:48:56.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:56.275380+0000) 2022-01-31T20:48:56.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:56 smithi171 conmon[46715]: debug 2022-01-31T20:48:56.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:56.304725+0000) 2022-01-31T20:48:56.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:56 smithi171 conmon[51620]: debug 2022-01-31T20:48:56.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:56.382160+0000) 2022-01-31T20:48:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:56 smithi171 conmon[41853]: debug 2022-01-31T20:48:56.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:56.510506+0000) 2022-01-31T20:48:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:56 smithi167 conmon[49112]: debug 2022-01-31T20:48:56.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:56.603916+0000) 2022-01-31T20:48:57.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:57 smithi167 conmon[54076]: debug 2022-01-31T20:48:57.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:57.275514+0000) 2022-01-31T20:48:57.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:57 smithi167 conmon[60316]: debug 2022-01-31T20:48:57.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:57.026771+0000) 2022-01-31T20:48:57.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:57 smithi171 conmon[46715]: debug 2022-01-31T20:48:57.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:57.304942+0000) 2022-01-31T20:48:57.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:57 smithi171 conmon[51620]: debug 2022-01-31T20:48:57.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:57.382349+0000) 2022-01-31T20:48:57.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:57 smithi171 conmon[41853]: debug 2022-01-31T20:48:57.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:57.510692+0000) 2022-01-31T20:48:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:57 smithi167 conmon[49112]: debug 2022-01-31T20:48:57.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:57.604126+0000) 2022-01-31T20:48:58.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:58 smithi167 conmon[54076]: debug 2022-01-31T20:48:58.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.275647+0000) 2022-01-31T20:48:58.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:58 smithi167 conmon[60316]: debug 2022-01-31T20:48:58.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.026967+0000) 2022-01-31T20:48:58.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[46715]: debug 2022-01-31T20:48:58.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.305161+0000) 2022-01-31T20:48:58.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[51620]: debug 2022-01-31T20:48:58.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.382560+0000) 2022-01-31T20:48:58.594 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:58 smithi167 conmon[49112]: debug 2022-01-31T20:48:58.529+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.531190+0000) 2022-01-31T20:48:58.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:58 smithi167 conmon[54076]: debug 2022-01-31T20:48:58.529+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.530944+0000) 2022-01-31T20:48:58.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:58 smithi167 conmon[60316]: debug 2022-01-31T20:48:58.529+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.531092+0000) 2022-01-31T20:48:58.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[35325]: debug 2022-01-31T20:48:58.537+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80506 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:48:58.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[46715]: debug 2022-01-31T20:48:58.530+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.531989+0000) 2022-01-31T20:48:58.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[51620]: debug 2022-01-31T20:48:58.529+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.531482+0000) 2022-01-31T20:48:58.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[41853]: debug 2022-01-31T20:48:58.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.510889+0000) 2022-01-31T20:48:58.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:58 smithi171 conmon[41853]: debug 2022-01-31T20:48:58.528+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.530097+0000) 2022-01-31T20:48:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:58 smithi167 conmon[49112]: debug 2022-01-31T20:48:58.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:58.604324+0000) 2022-01-31T20:48:59.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:48:59 smithi167 conmon[54076]: debug 2022-01-31T20:48:59.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:59.275819+0000) 2022-01-31T20:48:59.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:48:59 smithi167 conmon[60316]: debug 2022-01-31T20:48:59.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:59.027202+0000) 2022-01-31T20:48:59.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:48:59 smithi171 conmon[46715]: debug 2022-01-31T20:48:59.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:59.305392+0000) 2022-01-31T20:48:59.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:48:59 smithi171 conmon[51620]: debug 2022-01-31T20:48:59.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:59.382776+0000) 2022-01-31T20:48:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:48:59 smithi171 conmon[41853]: debug 2022-01-31T20:48:59.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:59.511050+0000) 2022-01-31T20:48:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:48:59 smithi167 conmon[49112]: debug 2022-01-31T20:48:59.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:48:59.604541+0000) 2022-01-31T20:49:00.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:00 smithi167 conmon[54076]: debug 2022-01-31T20:49:00.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:00.275980+0000) 2022-01-31T20:49:00.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:00 smithi167 conmon[60316]: debug 2022-01-31T20:49:00.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:00.027396+0000) 2022-01-31T20:49:00.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:00 smithi171 conmon[46715]: debug 2022-01-31T20:49:00.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:00.305539+0000) 2022-01-31T20:49:00.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:00 smithi171 conmon[51620]: debug 2022-01-31T20:49:00.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:00.382927+0000) 2022-01-31T20:49:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:00 smithi171 conmon[41853]: debug 2022-01-31T20:49:00.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:00.511200+0000) 2022-01-31T20:49:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:00 smithi167 conmon[49112]: debug 2022-01-31T20:49:00.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:00.604663+0000) 2022-01-31T20:49:01.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:01 smithi167 conmon[54076]: debug 2022-01-31T20:49:01.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:01.276145+0000) 2022-01-31T20:49:01.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:01 smithi167 conmon[60316]: debug 2022-01-31T20:49:01.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:01.027557+0000) 2022-01-31T20:49:01.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:01 smithi171 conmon[46715]: debug 2022-01-31T20:49:01.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:01.305685+0000) 2022-01-31T20:49:01.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:01 smithi171 conmon[51620]: debug 2022-01-31T20:49:01.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:01.383071+0000) 2022-01-31T20:49:01.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:01 smithi171 conmon[41853]: debug 2022-01-31T20:49:01.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:01.511384+0000) 2022-01-31T20:49:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:01 smithi167 conmon[49112]: debug 2022-01-31T20:49:01.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:01.604816+0000) 2022-01-31T20:49:02.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:02 smithi167 conmon[54076]: debug 2022-01-31T20:49:02.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:02.276334+0000) 2022-01-31T20:49:02.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:02 smithi167 conmon[60316]: debug 2022-01-31T20:49:02.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:02.027673+0000) 2022-01-31T20:49:02.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:02 smithi171 conmon[46715]: debug 2022-01-31T20:49:02.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:02.305840+0000) 2022-01-31T20:49:02.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:02 smithi171 conmon[51620]: debug 2022-01-31T20:49:02.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:02.383259+0000) 2022-01-31T20:49:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:02 smithi171 conmon[41853]: debug 2022-01-31T20:49:02.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:02.511585+0000) 2022-01-31T20:49:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:02 smithi167 conmon[49112]: debug 2022-01-31T20:49:02.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:02.605024+0000) 2022-01-31T20:49:03.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:03 smithi167 conmon[54076]: debug 2022-01-31T20:49:03.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.276518+0000) 2022-01-31T20:49:03.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:03 smithi167 conmon[60316]: debug 2022-01-31T20:49:03.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.027860+0000) 2022-01-31T20:49:03.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[46715]: debug 2022-01-31T20:49:03.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.305973+0000) 2022-01-31T20:49:03.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[51620]: debug 2022-01-31T20:49:03.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.383467+0000) 2022-01-31T20:49:03.595 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:03 smithi167 conmon[49112]: debug 2022-01-31T20:49:03.540+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.542037+0000) 2022-01-31T20:49:03.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:03 smithi167 conmon[54076]: debug 2022-01-31T20:49:03.540+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.542452+0000) 2022-01-31T20:49:03.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:03 smithi167 conmon[60316]: debug 2022-01-31T20:49:03.539+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.541347+0000) 2022-01-31T20:49:03.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[35325]: debug 2022-01-31T20:49:03.548+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80615 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:03.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[46715]: debug 2022-01-31T20:49:03.541+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.542644+0000) 2022-01-31T20:49:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[41853]: debug 2022-01-31T20:49:03.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.511793+0000) 2022-01-31T20:49:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[41853]: debug 2022-01-31T20:49:03.541+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.542270+0000) 2022-01-31T20:49:03.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:03 smithi171 conmon[51620]: debug 2022-01-31T20:49:03.540+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.541632+0000) 2022-01-31T20:49:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:03 smithi167 conmon[49112]: debug 2022-01-31T20:49:03.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:03.605220+0000) 2022-01-31T20:49:04.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:04 smithi167 conmon[54076]: debug 2022-01-31T20:49:04.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:04.276662+0000) 2022-01-31T20:49:04.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:04 smithi167 conmon[60316]: debug 2022-01-31T20:49:04.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:04.028052+0000) 2022-01-31T20:49:04.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:04 smithi171 conmon[51620]: debug 2022-01-31T20:49:04.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:04.383630+0000) 2022-01-31T20:49:04.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:04 smithi171 conmon[46715]: debug 2022-01-31T20:49:04.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:04.306137+0000) 2022-01-31T20:49:04.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:04 smithi171 conmon[41853]: debug 2022-01-31T20:49:04.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:04.511978+0000) 2022-01-31T20:49:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:04 smithi167 conmon[49112]: debug 2022-01-31T20:49:04.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:04.605408+0000) 2022-01-31T20:49:05.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:05 smithi167 conmon[54076]: debug 2022-01-31T20:49:05.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:05.276844+0000) 2022-01-31T20:49:05.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:05 smithi167 conmon[60316]: debug 2022-01-31T20:49:05.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:05.028211+0000) 2022-01-31T20:49:05.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:05 smithi171 conmon[46715]: debug 2022-01-31T20:49:05.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:05.306325+0000) 2022-01-31T20:49:05.647 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:05 smithi171 conmon[51620]: debug 2022-01-31T20:49:05.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:05.383797+0000) 2022-01-31T20:49:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:05 smithi171 conmon[41853]: debug 2022-01-31T20:49:05.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:05.512085+0000) 2022-01-31T20:49:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:05 smithi167 conmon[49112]: debug 2022-01-31T20:49:05.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:05.605600+0000) 2022-01-31T20:49:06.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:06 smithi167 conmon[54076]: debug 2022-01-31T20:49:06.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:06.277033+0000) 2022-01-31T20:49:06.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:06 smithi167 conmon[60316]: debug 2022-01-31T20:49:06.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:06.028396+0000) 2022-01-31T20:49:06.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:06 smithi171 conmon[46715]: debug 2022-01-31T20:49:06.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:06.306515+0000) 2022-01-31T20:49:06.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:06 smithi171 conmon[51620]: debug 2022-01-31T20:49:06.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:06.384002+0000) 2022-01-31T20:49:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:06 smithi171 conmon[41853]: debug 2022-01-31T20:49:06.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:06.512233+0000) 2022-01-31T20:49:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:06 smithi167 conmon[49112]: debug 2022-01-31T20:49:06.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:06.605820+0000) 2022-01-31T20:49:07.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:07 smithi167 conmon[54076]: debug 2022-01-31T20:49:07.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:07.277237+0000) 2022-01-31T20:49:07.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:07 smithi167 conmon[60316]: debug 2022-01-31T20:49:07.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:07.028581+0000) 2022-01-31T20:49:07.392 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:07 smithi171 conmon[51620]: debug 2022-01-31T20:49:07.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:07.384149+0000) 2022-01-31T20:49:07.392 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:07 smithi171 conmon[46715]: debug 2022-01-31T20:49:07.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:07.306668+0000) 2022-01-31T20:49:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:07 smithi171 conmon[41853]: debug 2022-01-31T20:49:07.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:07.512386+0000) 2022-01-31T20:49:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:07 smithi167 conmon[49112]: debug 2022-01-31T20:49:07.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:07.605960+0000) 2022-01-31T20:49:08.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:08 smithi167 conmon[54076]: debug 2022-01-31T20:49:08.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.277432+0000) 2022-01-31T20:49:08.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:08 smithi167 conmon[60316]: debug 2022-01-31T20:49:08.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.028769+0000) 2022-01-31T20:49:08.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[46715]: debug 2022-01-31T20:49:08.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.306794+0000) 2022-01-31T20:49:08.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[51620]: debug 2022-01-31T20:49:08.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.384367+0000) 2022-01-31T20:49:08.596 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:08 smithi167 conmon[49112]: debug 2022-01-31T20:49:08.551+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.552960+0000) 2022-01-31T20:49:08.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:08 smithi167 conmon[54076]: debug 2022-01-31T20:49:08.551+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.552713+0000) 2022-01-31T20:49:08.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:08 smithi167 conmon[60316]: debug 2022-01-31T20:49:08.550+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.551731+0000) 2022-01-31T20:49:08.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[46715]: debug 2022-01-31T20:49:08.552+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.552982+0000) 2022-01-31T20:49:08.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[51620]: debug 2022-01-31T20:49:08.551+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.552081+0000) 2022-01-31T20:49:08.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[35325]: debug 2022-01-31T20:49:08.560+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80724 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:08.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[35325]: debug 2022-01-31T20:49:08.576+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:49:08.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[41853]: debug 2022-01-31T20:49:08.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.512572+0000) 2022-01-31T20:49:08.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:08 smithi171 conmon[41853]: debug 2022-01-31T20:49:08.551+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.552272+0000) 2022-01-31T20:49:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:08 smithi167 conmon[49112]: debug 2022-01-31T20:49:08.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:08.606125+0000) 2022-01-31T20:49:09.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:09 smithi167 conmon[54076]: debug 2022-01-31T20:49:09.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:09.277603+0000) 2022-01-31T20:49:09.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:09 smithi167 conmon[60316]: debug 2022-01-31T20:49:09.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:09.028953+0000) 2022-01-31T20:49:09.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:09 smithi171 conmon[46715]: debug 2022-01-31T20:49:09.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:09.307026+0000) 2022-01-31T20:49:09.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:09 smithi171 conmon[51620]: debug 2022-01-31T20:49:09.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:09.384522+0000) 2022-01-31T20:49:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:09 smithi171 conmon[41853]: debug 2022-01-31T20:49:09.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:09.512764+0000) 2022-01-31T20:49:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:09 smithi167 conmon[49112]: debug 2022-01-31T20:49:09.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:09.606302+0000) 2022-01-31T20:49:10.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:10 smithi167 conmon[60316]: debug 2022-01-31T20:49:10.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:10.029085+0000) 2022-01-31T20:49:10.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:10 smithi167 conmon[54076]: debug 2022-01-31T20:49:10.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:10.277741+0000) 2022-01-31T20:49:10.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:10 smithi171 conmon[46715]: debug 2022-01-31T20:49:10.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:10.307178+0000) 2022-01-31T20:49:10.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:10 smithi171 conmon[51620]: debug 2022-01-31T20:49:10.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:10.384664+0000) 2022-01-31T20:49:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:10 smithi171 conmon[41853]: debug 2022-01-31T20:49:10.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:10.512912+0000) 2022-01-31T20:49:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:10 smithi167 conmon[49112]: debug 2022-01-31T20:49:10.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:10.606554+0000) 2022-01-31T20:49:11.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:11 smithi167 conmon[54076]: debug 2022-01-31T20:49:11.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:11.277900+0000) 2022-01-31T20:49:11.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:11 smithi167 conmon[60316]: debug 2022-01-31T20:49:11.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:11.029295+0000) 2022-01-31T20:49:11.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:11 smithi171 conmon[46715]: debug 2022-01-31T20:49:11.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:11.307364+0000) 2022-01-31T20:49:11.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:11 smithi171 conmon[51620]: debug 2022-01-31T20:49:11.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:11.384849+0000) 2022-01-31T20:49:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:11 smithi171 conmon[41853]: debug 2022-01-31T20:49:11.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:11.513106+0000) 2022-01-31T20:49:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:11 smithi167 conmon[49112]: debug 2022-01-31T20:49:11.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:11.606713+0000) 2022-01-31T20:49:12.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:12 smithi167 conmon[54076]: debug 2022-01-31T20:49:12.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:12.278000+0000) 2022-01-31T20:49:12.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:12 smithi167 conmon[60316]: debug 2022-01-31T20:49:12.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:12.029477+0000) 2022-01-31T20:49:12.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:12 smithi171 conmon[46715]: debug 2022-01-31T20:49:12.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:12.307543+0000) 2022-01-31T20:49:12.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:12 smithi171 conmon[51620]: debug 2022-01-31T20:49:12.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:12.385027+0000) 2022-01-31T20:49:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:12 smithi171 conmon[41853]: debug 2022-01-31T20:49:12.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:12.513268+0000) 2022-01-31T20:49:12.921 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:12 smithi167 conmon[49112]: debug 2022-01-31T20:49:12.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:12.606868+0000) 2022-01-31T20:49:13.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:13 smithi167 conmon[54076]: debug 2022-01-31T20:49:13.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.278170+0000) 2022-01-31T20:49:13.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:13 smithi167 conmon[60316]: debug 2022-01-31T20:49:13.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.029680+0000) 2022-01-31T20:49:13.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[46715]: debug 2022-01-31T20:49:13.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.307790+0000) 2022-01-31T20:49:13.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[51620]: debug 2022-01-31T20:49:13.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.385193+0000) 2022-01-31T20:49:13.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:13 smithi167 conmon[54076]: debug 2022-01-31T20:49:13.563+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.564620+0000) 2022-01-31T20:49:13.567 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:13 smithi167 conmon[60316]: debug 2022-01-31T20:49:13.561+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.563432+0000) 2022-01-31T20:49:13.567 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:13 smithi167 conmon[49112]: debug 2022-01-31T20:49:13.562+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.563912+0000) 2022-01-31T20:49:13.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[35325]: debug 2022-01-31T20:49:13.571+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80835 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:13.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[46715]: debug 2022-01-31T20:49:13.564+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.565024+0000) 2022-01-31T20:49:13.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[51620]: debug 2022-01-31T20:49:13.563+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.563795+0000) 2022-01-31T20:49:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[41853]: debug 2022-01-31T20:49:13.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.513424+0000) 2022-01-31T20:49:13.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:13 smithi171 conmon[41853]: debug 2022-01-31T20:49:13.562+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.563494+0000) 2022-01-31T20:49:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:13 smithi167 conmon[49112]: debug 2022-01-31T20:49:13.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:13.607038+0000) 2022-01-31T20:49:14.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:14 smithi167 conmon[54076]: debug 2022-01-31T20:49:14.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:14.278305+0000) 2022-01-31T20:49:14.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:14 smithi167 conmon[60316]: debug 2022-01-31T20:49:14.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:14.029834+0000) 2022-01-31T20:49:14.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:14 smithi171 conmon[46715]: debug 2022-01-31T20:49:14.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:14.308009+0000) 2022-01-31T20:49:14.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:14 smithi171 conmon[51620]: debug 2022-01-31T20:49:14.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:14.385361+0000) 2022-01-31T20:49:14.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:14 smithi171 conmon[41853]: debug 2022-01-31T20:49:14.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:14.513613+0000) 2022-01-31T20:49:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:14 smithi167 conmon[49112]: debug 2022-01-31T20:49:14.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:14.607194+0000) 2022-01-31T20:49:15.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:15 smithi167 conmon[54076]: debug 2022-01-31T20:49:15.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:15.278403+0000) 2022-01-31T20:49:15.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:15 smithi167 conmon[60316]: debug 2022-01-31T20:49:15.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:15.029948+0000) 2022-01-31T20:49:15.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:15 smithi171 conmon[46715]: debug 2022-01-31T20:49:15.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:15.308155+0000) 2022-01-31T20:49:15.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:15 smithi171 conmon[51620]: debug 2022-01-31T20:49:15.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:15.385511+0000) 2022-01-31T20:49:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:15 smithi171 conmon[41853]: debug 2022-01-31T20:49:15.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:15.513746+0000) 2022-01-31T20:49:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:15 smithi167 conmon[49112]: debug 2022-01-31T20:49:15.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:15.607322+0000) 2022-01-31T20:49:16.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:16 smithi167 conmon[54076]: debug 2022-01-31T20:49:16.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:16.278559+0000) 2022-01-31T20:49:16.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:16 smithi167 conmon[60316]: debug 2022-01-31T20:49:16.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:16.030138+0000) 2022-01-31T20:49:16.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:16 smithi171 conmon[46715]: debug 2022-01-31T20:49:16.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:16.308321+0000) 2022-01-31T20:49:16.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:16 smithi171 conmon[51620]: debug 2022-01-31T20:49:16.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:16.385701+0000) 2022-01-31T20:49:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:16 smithi171 conmon[41853]: debug 2022-01-31T20:49:16.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:16.513949+0000) 2022-01-31T20:49:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:16 smithi167 conmon[49112]: debug 2022-01-31T20:49:16.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:16.607476+0000) 2022-01-31T20:49:17.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:17 smithi167 conmon[54076]: debug 2022-01-31T20:49:17.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:17.278735+0000) 2022-01-31T20:49:17.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:17 smithi167 conmon[60316]: debug 2022-01-31T20:49:17.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:17.030293+0000) 2022-01-31T20:49:17.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:17 smithi171 conmon[46715]: debug 2022-01-31T20:49:17.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:17.308526+0000) 2022-01-31T20:49:17.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:17 smithi171 conmon[51620]: debug 2022-01-31T20:49:17.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:17.385833+0000) 2022-01-31T20:49:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:17 smithi171 conmon[41853]: debug 2022-01-31T20:49:17.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:17.514163+0000) 2022-01-31T20:49:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:17 smithi167 conmon[49112]: debug 2022-01-31T20:49:17.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:17.607675+0000) 2022-01-31T20:49:18.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:18 smithi167 conmon[60316]: debug 2022-01-31T20:49:18.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.030425+0000) 2022-01-31T20:49:18.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:18 smithi167 conmon[54076]: debug 2022-01-31T20:49:18.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.278933+0000) 2022-01-31T20:49:18.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[46715]: debug 2022-01-31T20:49:18.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.308716+0000) 2022-01-31T20:49:18.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[51620]: debug 2022-01-31T20:49:18.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.386046+0000) 2022-01-31T20:49:18.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[35325]: debug 2022-01-31T20:49:18.583+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 80949 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:18.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[46715]: debug 2022-01-31T20:49:18.574+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.575553+0000) 2022-01-31T20:49:18.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[51620]: debug 2022-01-31T20:49:18.575+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.576454+0000) 2022-01-31T20:49:18.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[41853]: debug 2022-01-31T20:49:18.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.514347+0000) 2022-01-31T20:49:18.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:18 smithi171 conmon[41853]: debug 2022-01-31T20:49:18.573+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.574631+0000) 2022-01-31T20:49:18.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:18 smithi167 conmon[54076]: debug 2022-01-31T20:49:18.574+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.576096+0000) 2022-01-31T20:49:18.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:18 smithi167 conmon[60316]: debug 2022-01-31T20:49:18.573+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.575084+0000) 2022-01-31T20:49:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:18 smithi167 conmon[49112]: debug 2022-01-31T20:49:18.574+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.575521+0000) 2022-01-31T20:49:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:18 smithi167 conmon[49112]: debug 2022-01-31T20:49:18.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:18.607896+0000) 2022-01-31T20:49:19.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:19 smithi167 conmon[54076]: debug 2022-01-31T20:49:19.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:19.279163+0000) 2022-01-31T20:49:19.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:19 smithi167 conmon[60316]: debug 2022-01-31T20:49:19.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:19.030594+0000) 2022-01-31T20:49:19.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:19 smithi171 conmon[46715]: debug 2022-01-31T20:49:19.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:19.308890+0000) 2022-01-31T20:49:19.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:19 smithi171 conmon[51620]: debug 2022-01-31T20:49:19.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:19.386163+0000) 2022-01-31T20:49:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:19 smithi171 conmon[41853]: debug 2022-01-31T20:49:19.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:19.514521+0000) 2022-01-31T20:49:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:19 smithi167 conmon[49112]: debug 2022-01-31T20:49:19.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:19.608000+0000) 2022-01-31T20:49:20.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:20 smithi167 conmon[54076]: debug 2022-01-31T20:49:20.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:20.279273+0000) 2022-01-31T20:49:20.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:20 smithi167 conmon[60316]: debug 2022-01-31T20:49:20.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:20.030756+0000) 2022-01-31T20:49:20.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:20 smithi171 conmon[46715]: debug 2022-01-31T20:49:20.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:20.309018+0000) 2022-01-31T20:49:20.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:20 smithi171 conmon[51620]: debug 2022-01-31T20:49:20.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:20.386331+0000) 2022-01-31T20:49:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:20 smithi171 conmon[41853]: debug 2022-01-31T20:49:20.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:20.514673+0000) 2022-01-31T20:49:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:20 smithi167 conmon[49112]: debug 2022-01-31T20:49:20.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:20.608117+0000) 2022-01-31T20:49:21.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:21 smithi167 conmon[54076]: debug 2022-01-31T20:49:21.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:21.279481+0000) 2022-01-31T20:49:21.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:21 smithi167 conmon[60316]: debug 2022-01-31T20:49:21.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:21.030928+0000) 2022-01-31T20:49:21.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:21 smithi171 conmon[51620]: debug 2022-01-31T20:49:21.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:21.386526+0000) 2022-01-31T20:49:21.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:21 smithi171 conmon[46715]: debug 2022-01-31T20:49:21.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:21.309174+0000) 2022-01-31T20:49:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:21 smithi171 conmon[41853]: debug 2022-01-31T20:49:21.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:21.514844+0000) 2022-01-31T20:49:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:21 smithi167 conmon[49112]: debug 2022-01-31T20:49:21.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:21.608256+0000) 2022-01-31T20:49:22.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:22 smithi167 conmon[54076]: debug 2022-01-31T20:49:22.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:22.279705+0000) 2022-01-31T20:49:22.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:22 smithi167 conmon[60316]: debug 2022-01-31T20:49:22.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:22.031105+0000) 2022-01-31T20:49:22.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:22 smithi171 conmon[46715]: debug 2022-01-31T20:49:22.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:22.309329+0000) 2022-01-31T20:49:22.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:22 smithi171 conmon[51620]: debug 2022-01-31T20:49:22.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:22.386685+0000) 2022-01-31T20:49:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:22 smithi171 conmon[41853]: debug 2022-01-31T20:49:22.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:22.515044+0000) 2022-01-31T20:49:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:22 smithi167 conmon[49112]: debug 2022-01-31T20:49:22.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:22.608461+0000) 2022-01-31T20:49:23.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:23 smithi167 conmon[54076]: debug 2022-01-31T20:49:23.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.279916+0000) 2022-01-31T20:49:23.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:23 smithi167 conmon[60316]: debug 2022-01-31T20:49:23.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.031303+0000) 2022-01-31T20:49:23.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[46715]: debug 2022-01-31T20:49:23.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.309535+0000) 2022-01-31T20:49:23.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[51620]: debug 2022-01-31T20:49:23.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.386865+0000) 2022-01-31T20:49:23.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[46715]: debug 2022-01-31T20:49:23.586+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.587725+0000) 2022-01-31T20:49:23.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[51620]: debug 2022-01-31T20:49:23.586+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.587163+0000) 2022-01-31T20:49:23.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[35325]: debug 2022-01-31T20:49:23.577+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:49:23.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[35325]: debug 2022-01-31T20:49:23.595+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81060 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:23.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[41853]: debug 2022-01-31T20:49:23.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.515237+0000) 2022-01-31T20:49:23.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:23 smithi171 conmon[41853]: debug 2022-01-31T20:49:23.584+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.585623+0000) 2022-01-31T20:49:23.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:23 smithi167 conmon[60316]: debug 2022-01-31T20:49:23.586+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.587329+0000) 2022-01-31T20:49:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:23 smithi167 conmon[49112]: debug 2022-01-31T20:49:23.592+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.593364+0000) 2022-01-31T20:49:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:23 smithi167 conmon[49112]: debug 2022-01-31T20:49:23.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.608627+0000) 2022-01-31T20:49:23.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:23 smithi167 conmon[54076]: debug 2022-01-31T20:49:23.586+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:23.586811+0000) 2022-01-31T20:49:24.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:24 smithi167 conmon[54076]: debug 2022-01-31T20:49:24.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:24.280108+0000) 2022-01-31T20:49:24.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:24 smithi167 conmon[60316]: debug 2022-01-31T20:49:24.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:24.031528+0000) 2022-01-31T20:49:24.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:24 smithi171 conmon[46715]: debug 2022-01-31T20:49:24.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:24.309711+0000) 2022-01-31T20:49:24.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:24 smithi171 conmon[51620]: debug 2022-01-31T20:49:24.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:24.387035+0000) 2022-01-31T20:49:24.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:24 smithi171 conmon[41853]: debug 2022-01-31T20:49:24.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:24.515443+0000) 2022-01-31T20:49:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:24 smithi167 conmon[49112]: debug 2022-01-31T20:49:24.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:24.608750+0000) 2022-01-31T20:49:25.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:25 smithi167 conmon[60316]: debug 2022-01-31T20:49:25.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:25.031682+0000) 2022-01-31T20:49:25.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:25 smithi167 conmon[54076]: debug 2022-01-31T20:49:25.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:25.280243+0000) 2022-01-31T20:49:25.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:25 smithi171 conmon[46715]: debug 2022-01-31T20:49:25.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:25.309831+0000) 2022-01-31T20:49:25.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:25 smithi171 conmon[51620]: debug 2022-01-31T20:49:25.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:25.387180+0000) 2022-01-31T20:49:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:25 smithi171 conmon[41853]: debug 2022-01-31T20:49:25.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:25.515590+0000) 2022-01-31T20:49:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:25 smithi167 conmon[49112]: debug 2022-01-31T20:49:25.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:25.608926+0000) 2022-01-31T20:49:26.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:26 smithi167 conmon[54076]: debug 2022-01-31T20:49:26.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:26.280438+0000) 2022-01-31T20:49:26.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:26 smithi167 conmon[60316]: debug 2022-01-31T20:49:26.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:26.031856+0000) 2022-01-31T20:49:26.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:26 smithi171 conmon[46715]: debug 2022-01-31T20:49:26.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:26.309989+0000) 2022-01-31T20:49:26.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:26 smithi171 conmon[51620]: debug 2022-01-31T20:49:26.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:26.387385+0000) 2022-01-31T20:49:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:26 smithi171 conmon[41853]: debug 2022-01-31T20:49:26.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:26.515716+0000) 2022-01-31T20:49:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:26 smithi167 conmon[49112]: debug 2022-01-31T20:49:26.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:26.609090+0000) 2022-01-31T20:49:27.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:27 smithi167 conmon[54076]: debug 2022-01-31T20:49:27.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:27.280649+0000) 2022-01-31T20:49:27.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:27 smithi167 conmon[60316]: debug 2022-01-31T20:49:27.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:27.032021+0000) 2022-01-31T20:49:27.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:27 smithi171 conmon[46715]: debug 2022-01-31T20:49:27.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:27.310168+0000) 2022-01-31T20:49:27.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:27 smithi171 conmon[51620]: debug 2022-01-31T20:49:27.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:27.387608+0000) 2022-01-31T20:49:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:27 smithi171 conmon[41853]: debug 2022-01-31T20:49:27.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:27.515891+0000) 2022-01-31T20:49:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:27 smithi167 conmon[49112]: debug 2022-01-31T20:49:27.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:27.609224+0000) 2022-01-31T20:49:28.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:28 smithi167 conmon[54076]: debug 2022-01-31T20:49:28.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.280843+0000) 2022-01-31T20:49:28.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:28 smithi167 conmon[60316]: debug 2022-01-31T20:49:28.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.032196+0000) 2022-01-31T20:49:28.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[46715]: debug 2022-01-31T20:49:28.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.310365+0000) 2022-01-31T20:49:28.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[51620]: debug 2022-01-31T20:49:28.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.387814+0000) 2022-01-31T20:49:28.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[46715]: debug 2022-01-31T20:49:28.599+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.599785+0000) 2022-01-31T20:49:28.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[51620]: debug 2022-01-31T20:49:28.597+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.598665+0000) 2022-01-31T20:49:28.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[35325]: debug 2022-01-31T20:49:28.605+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81169 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:28.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[41853]: debug 2022-01-31T20:49:28.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.516040+0000) 2022-01-31T20:49:28.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:28 smithi171 conmon[41853]: debug 2022-01-31T20:49:28.597+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.597980+0000) 2022-01-31T20:49:28.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:28 smithi167 conmon[54076]: debug 2022-01-31T20:49:28.598+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.598936+0000) 2022-01-31T20:49:28.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:28 smithi167 conmon[60316]: debug 2022-01-31T20:49:28.599+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.599585+0000) 2022-01-31T20:49:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:28 smithi167 conmon[49112]: debug 2022-01-31T20:49:28.598+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.599260+0000) 2022-01-31T20:49:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:28 smithi167 conmon[49112]: debug 2022-01-31T20:49:28.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:28.609399+0000) 2022-01-31T20:49:29.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:29 smithi167 conmon[60316]: debug 2022-01-31T20:49:29.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:29.032395+0000) 2022-01-31T20:49:29.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:29 smithi167 conmon[54076]: debug 2022-01-31T20:49:29.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:29.281047+0000) 2022-01-31T20:49:29.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:29 smithi171 conmon[46715]: debug 2022-01-31T20:49:29.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:29.310555+0000) 2022-01-31T20:49:29.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:29 smithi171 conmon[51620]: debug 2022-01-31T20:49:29.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:29.387972+0000) 2022-01-31T20:49:29.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:29 smithi171 conmon[41853]: debug 2022-01-31T20:49:29.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:29.516225+0000) 2022-01-31T20:49:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:29 smithi167 conmon[49112]: debug 2022-01-31T20:49:29.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:29.609601+0000) 2022-01-31T20:49:30.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:30 smithi167 conmon[54076]: debug 2022-01-31T20:49:30.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:30.281214+0000) 2022-01-31T20:49:30.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:30 smithi167 conmon[60316]: debug 2022-01-31T20:49:30.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:30.032588+0000) 2022-01-31T20:49:30.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:30 smithi171 conmon[46715]: debug 2022-01-31T20:49:30.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:30.310713+0000) 2022-01-31T20:49:30.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:30 smithi171 conmon[51620]: debug 2022-01-31T20:49:30.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:30.388129+0000) 2022-01-31T20:49:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:30 smithi171 conmon[41853]: debug 2022-01-31T20:49:30.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:30.516380+0000) 2022-01-31T20:49:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:30 smithi167 conmon[49112]: debug 2022-01-31T20:49:30.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:30.609747+0000) 2022-01-31T20:49:31.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:31 smithi167 conmon[54076]: debug 2022-01-31T20:49:31.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:31.281418+0000) 2022-01-31T20:49:31.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:31 smithi167 conmon[60316]: debug 2022-01-31T20:49:31.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:31.032788+0000) 2022-01-31T20:49:31.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:31 smithi171 conmon[46715]: debug 2022-01-31T20:49:31.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:31.310948+0000) 2022-01-31T20:49:31.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:31 smithi171 conmon[51620]: debug 2022-01-31T20:49:31.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:31.388315+0000) 2022-01-31T20:49:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:31 smithi171 conmon[41853]: debug 2022-01-31T20:49:31.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:31.516477+0000) 2022-01-31T20:49:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:31 smithi167 conmon[49112]: debug 2022-01-31T20:49:31.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:31.609954+0000) 2022-01-31T20:49:32.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:32 smithi167 conmon[54076]: debug 2022-01-31T20:49:32.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:32.281548+0000) 2022-01-31T20:49:32.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:32 smithi167 conmon[60316]: debug 2022-01-31T20:49:32.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:32.032968+0000) 2022-01-31T20:49:32.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:32 smithi171 conmon[51620]: debug 2022-01-31T20:49:32.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:32.388516+0000) 2022-01-31T20:49:32.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:32 smithi171 conmon[46715]: debug 2022-01-31T20:49:32.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:32.311106+0000) 2022-01-31T20:49:32.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:32 smithi171 conmon[41853]: debug 2022-01-31T20:49:32.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:32.516645+0000) 2022-01-31T20:49:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:32 smithi167 conmon[49112]: debug 2022-01-31T20:49:32.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:32.610171+0000) 2022-01-31T20:49:33.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:33 smithi167 conmon[54076]: debug 2022-01-31T20:49:33.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.281663+0000) 2022-01-31T20:49:33.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:33 smithi167 conmon[60316]: debug 2022-01-31T20:49:33.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.033155+0000) 2022-01-31T20:49:33.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[46715]: debug 2022-01-31T20:49:33.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.311274+0000) 2022-01-31T20:49:33.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[51620]: debug 2022-01-31T20:49:33.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.388678+0000) 2022-01-31T20:49:33.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[35325]: debug 2022-01-31T20:49:33.617+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81279 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:33.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[46715]: debug 2022-01-31T20:49:33.609+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.610560+0000) 2022-01-31T20:49:33.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[51620]: debug 2022-01-31T20:49:33.608+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.608927+0000) 2022-01-31T20:49:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[41853]: debug 2022-01-31T20:49:33.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.516839+0000) 2022-01-31T20:49:33.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:33 smithi171 conmon[41853]: debug 2022-01-31T20:49:33.608+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.609409+0000) 2022-01-31T20:49:33.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:33 smithi167 conmon[54076]: debug 2022-01-31T20:49:33.608+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.608847+0000) 2022-01-31T20:49:33.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:33 smithi167 conmon[60316]: debug 2022-01-31T20:49:33.609+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.609883+0000) 2022-01-31T20:49:33.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:33 smithi167 conmon[49112]: debug 2022-01-31T20:49:33.609+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.609591+0000) 2022-01-31T20:49:33.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:33 smithi167 conmon[49112]: debug 2022-01-31T20:49:33.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:33.610345+0000) 2022-01-31T20:49:34.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:34 smithi167 conmon[54076]: debug 2022-01-31T20:49:34.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:34.281858+0000) 2022-01-31T20:49:34.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:34 smithi167 conmon[60316]: debug 2022-01-31T20:49:34.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:34.033343+0000) 2022-01-31T20:49:34.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:34 smithi171 conmon[46715]: debug 2022-01-31T20:49:34.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:34.311444+0000) 2022-01-31T20:49:34.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:34 smithi171 conmon[51620]: debug 2022-01-31T20:49:34.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:34.388834+0000) 2022-01-31T20:49:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:34 smithi171 conmon[41853]: debug 2022-01-31T20:49:34.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:34.517019+0000) 2022-01-31T20:49:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:34 smithi167 conmon[49112]: debug 2022-01-31T20:49:34.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:34.610526+0000) 2022-01-31T20:49:35.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:35 smithi167 conmon[54076]: debug 2022-01-31T20:49:35.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:35.282036+0000) 2022-01-31T20:49:35.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:35 smithi167 conmon[60316]: debug 2022-01-31T20:49:35.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:35.033494+0000) 2022-01-31T20:49:35.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:35 smithi171 conmon[46715]: debug 2022-01-31T20:49:35.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:35.311594+0000) 2022-01-31T20:49:35.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:35 smithi171 conmon[51620]: debug 2022-01-31T20:49:35.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:35.388961+0000) 2022-01-31T20:49:35.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:35 smithi171 conmon[41853]: debug 2022-01-31T20:49:35.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:35.517171+0000) 2022-01-31T20:49:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:35 smithi167 conmon[49112]: debug 2022-01-31T20:49:35.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:35.610693+0000) 2022-01-31T20:49:36.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:36 smithi167 conmon[54076]: debug 2022-01-31T20:49:36.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:36.282245+0000) 2022-01-31T20:49:36.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:36 smithi167 conmon[60316]: debug 2022-01-31T20:49:36.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:36.033652+0000) 2022-01-31T20:49:36.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:36 smithi171 conmon[51620]: debug 2022-01-31T20:49:36.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:36.389082+0000) 2022-01-31T20:49:36.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:36 smithi171 conmon[46715]: debug 2022-01-31T20:49:36.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:36.311782+0000) 2022-01-31T20:49:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:36 smithi171 conmon[41853]: debug 2022-01-31T20:49:36.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:36.517354+0000) 2022-01-31T20:49:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:36 smithi167 conmon[49112]: debug 2022-01-31T20:49:36.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:36.610844+0000) 2022-01-31T20:49:37.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:37 smithi167 conmon[54076]: debug 2022-01-31T20:49:37.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:37.282366+0000) 2022-01-31T20:49:37.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:37 smithi167 conmon[60316]: debug 2022-01-31T20:49:37.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:37.033814+0000) 2022-01-31T20:49:37.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:37 smithi171 conmon[51620]: debug 2022-01-31T20:49:37.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:37.389272+0000) 2022-01-31T20:49:37.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:37 smithi171 conmon[46715]: debug 2022-01-31T20:49:37.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:37.311973+0000) 2022-01-31T20:49:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:37 smithi171 conmon[41853]: debug 2022-01-31T20:49:37.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:37.517525+0000) 2022-01-31T20:49:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:37 smithi167 conmon[49112]: debug 2022-01-31T20:49:37.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:37.611033+0000) 2022-01-31T20:49:38.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:38 smithi167 conmon[60316]: debug 2022-01-31T20:49:38.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.033984+0000) 2022-01-31T20:49:38.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:38 smithi167 conmon[54076]: debug 2022-01-31T20:49:38.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.282585+0000) 2022-01-31T20:49:38.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[46715]: debug 2022-01-31T20:49:38.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.312131+0000) 2022-01-31T20:49:38.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[51620]: debug 2022-01-31T20:49:38.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.389439+0000) 2022-01-31T20:49:38.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[51620]: debug 2022-01-31T20:49:38.618+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.620050+0000) 2022-01-31T20:49:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[35325]: debug 2022-01-31T20:49:38.577+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:49:38.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[35325]: debug 2022-01-31T20:49:38.627+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81388 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[41853]: debug 2022-01-31T20:49:38.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.517743+0000) 2022-01-31T20:49:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[41853]: debug 2022-01-31T20:49:38.618+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.620289+0000) 2022-01-31T20:49:38.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:38 smithi171 conmon[46715]: debug 2022-01-31T20:49:38.620+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.622016+0000) 2022-01-31T20:49:38.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:38 smithi167 conmon[54076]: debug 2022-01-31T20:49:38.619+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.620208+0000) 2022-01-31T20:49:38.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:38 smithi167 conmon[60316]: debug 2022-01-31T20:49:38.621+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.621513+0000) 2022-01-31T20:49:38.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:38 smithi167 conmon[49112]: debug 2022-01-31T20:49:38.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.611240+0000) 2022-01-31T20:49:38.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:38 smithi167 conmon[49112]: debug 2022-01-31T20:49:38.620+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:38.621001+0000) 2022-01-31T20:49:39.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:39 smithi167 conmon[54076]: debug 2022-01-31T20:49:39.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:39.282770+0000) 2022-01-31T20:49:39.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:39 smithi167 conmon[60316]: debug 2022-01-31T20:49:39.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:39.034175+0000) 2022-01-31T20:49:39.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:39 smithi171 conmon[46715]: debug 2022-01-31T20:49:39.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:39.312316+0000) 2022-01-31T20:49:39.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:39 smithi171 conmon[51620]: debug 2022-01-31T20:49:39.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:39.389597+0000) 2022-01-31T20:49:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:39 smithi171 conmon[41853]: debug 2022-01-31T20:49:39.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:39.517927+0000) 2022-01-31T20:49:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:39 smithi167 conmon[49112]: debug 2022-01-31T20:49:39.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:39.611458+0000) 2022-01-31T20:49:40.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:40 smithi167 conmon[54076]: debug 2022-01-31T20:49:40.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:40.282908+0000) 2022-01-31T20:49:40.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:40 smithi167 conmon[60316]: debug 2022-01-31T20:49:40.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:40.034330+0000) 2022-01-31T20:49:40.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:40 smithi171 conmon[46715]: debug 2022-01-31T20:49:40.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:40.312468+0000) 2022-01-31T20:49:40.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:40 smithi171 conmon[51620]: debug 2022-01-31T20:49:40.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:40.389749+0000) 2022-01-31T20:49:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:40 smithi171 conmon[41853]: debug 2022-01-31T20:49:40.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:40.518068+0000) 2022-01-31T20:49:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:40 smithi167 conmon[49112]: debug 2022-01-31T20:49:40.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:40.611582+0000) 2022-01-31T20:49:41.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:41 smithi167 conmon[54076]: debug 2022-01-31T20:49:41.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:41.283089+0000) 2022-01-31T20:49:41.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:41 smithi167 conmon[60316]: debug 2022-01-31T20:49:41.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:41.034516+0000) 2022-01-31T20:49:41.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:41 smithi171 conmon[46715]: debug 2022-01-31T20:49:41.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:41.312663+0000) 2022-01-31T20:49:41.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:41 smithi171 conmon[51620]: debug 2022-01-31T20:49:41.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:41.389952+0000) 2022-01-31T20:49:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:41 smithi171 conmon[41853]: debug 2022-01-31T20:49:41.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:41.518235+0000) 2022-01-31T20:49:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:41 smithi167 conmon[49112]: debug 2022-01-31T20:49:41.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:41.611735+0000) 2022-01-31T20:49:42.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:42 smithi167 conmon[54076]: debug 2022-01-31T20:49:42.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:42.283314+0000) 2022-01-31T20:49:42.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:42 smithi167 conmon[60316]: debug 2022-01-31T20:49:42.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:42.034704+0000) 2022-01-31T20:49:42.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:42 smithi171 conmon[46715]: debug 2022-01-31T20:49:42.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:42.312839+0000) 2022-01-31T20:49:42.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:42 smithi171 conmon[51620]: debug 2022-01-31T20:49:42.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:42.390137+0000) 2022-01-31T20:49:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:42 smithi171 conmon[41853]: debug 2022-01-31T20:49:42.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:42.518441+0000) 2022-01-31T20:49:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:42 smithi167 conmon[49112]: debug 2022-01-31T20:49:42.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:42.611893+0000) 2022-01-31T20:49:43.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:43 smithi167 conmon[54076]: debug 2022-01-31T20:49:43.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.283517+0000) 2022-01-31T20:49:43.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:43 smithi167 conmon[60316]: debug 2022-01-31T20:49:43.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.034871+0000) 2022-01-31T20:49:43.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[46715]: debug 2022-01-31T20:49:43.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.313036+0000) 2022-01-31T20:49:43.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[51620]: debug 2022-01-31T20:49:43.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.390319+0000) 2022-01-31T20:49:43.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[35325]: debug 2022-01-31T20:49:43.638+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81498 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:43.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[46715]: debug 2022-01-31T20:49:43.629+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.631658+0000) 2022-01-31T20:49:43.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[51620]: debug 2022-01-31T20:49:43.628+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.630587+0000) 2022-01-31T20:49:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[41853]: debug 2022-01-31T20:49:43.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.518629+0000) 2022-01-31T20:49:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:43 smithi171 conmon[41853]: debug 2022-01-31T20:49:43.630+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.632132+0000) 2022-01-31T20:49:43.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:43 smithi167 conmon[54076]: debug 2022-01-31T20:49:43.630+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.631449+0000) 2022-01-31T20:49:43.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:43 smithi167 conmon[60316]: debug 2022-01-31T20:49:43.630+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.630991+0000) 2022-01-31T20:49:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:43 smithi167 conmon[49112]: debug 2022-01-31T20:49:43.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.612114+0000) 2022-01-31T20:49:43.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:43 smithi167 conmon[49112]: debug 2022-01-31T20:49:43.631+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:43.631741+0000) 2022-01-31T20:49:44.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:44 smithi167 conmon[60316]: debug 2022-01-31T20:49:44.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:44.035080+0000) 2022-01-31T20:49:44.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:44 smithi167 conmon[54076]: debug 2022-01-31T20:49:44.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:44.283652+0000) 2022-01-31T20:49:44.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:44 smithi171 conmon[51620]: debug 2022-01-31T20:49:44.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:44.390505+0000) 2022-01-31T20:49:44.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:44 smithi171 conmon[46715]: debug 2022-01-31T20:49:44.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:44.313207+0000) 2022-01-31T20:49:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:44 smithi171 conmon[41853]: debug 2022-01-31T20:49:44.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:44.518823+0000) 2022-01-31T20:49:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:44 smithi167 conmon[49112]: debug 2022-01-31T20:49:44.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:44.612275+0000) 2022-01-31T20:49:45.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:45 smithi167 conmon[54076]: debug 2022-01-31T20:49:45.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:45.283773+0000) 2022-01-31T20:49:45.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:45 smithi167 conmon[60316]: debug 2022-01-31T20:49:45.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:45.035249+0000) 2022-01-31T20:49:45.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:45 smithi171 conmon[46715]: debug 2022-01-31T20:49:45.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:45.313369+0000) 2022-01-31T20:49:45.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:45 smithi171 conmon[51620]: debug 2022-01-31T20:49:45.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:45.390653+0000) 2022-01-31T20:49:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:45 smithi171 conmon[41853]: debug 2022-01-31T20:49:45.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:45.518979+0000) 2022-01-31T20:49:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:45 smithi167 conmon[49112]: debug 2022-01-31T20:49:45.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:45.612414+0000) 2022-01-31T20:49:46.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:46 smithi167 conmon[54076]: debug 2022-01-31T20:49:46.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:46.283935+0000) 2022-01-31T20:49:46.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:46 smithi167 conmon[60316]: debug 2022-01-31T20:49:46.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:46.035440+0000) 2022-01-31T20:49:46.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:46 smithi171 conmon[46715]: debug 2022-01-31T20:49:46.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:46.313563+0000) 2022-01-31T20:49:46.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:46 smithi171 conmon[51620]: debug 2022-01-31T20:49:46.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:46.390843+0000) 2022-01-31T20:49:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:46 smithi171 conmon[41853]: debug 2022-01-31T20:49:46.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:46.519165+0000) 2022-01-31T20:49:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:46 smithi167 conmon[49112]: debug 2022-01-31T20:49:46.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:46.612541+0000) 2022-01-31T20:49:47.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:47 smithi167 conmon[60316]: debug 2022-01-31T20:49:47.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:47.035657+0000) 2022-01-31T20:49:47.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:47 smithi167 conmon[54076]: debug 2022-01-31T20:49:47.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:47.284100+0000) 2022-01-31T20:49:47.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:47 smithi171 conmon[46715]: debug 2022-01-31T20:49:47.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:47.313793+0000) 2022-01-31T20:49:47.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:47 smithi171 conmon[51620]: debug 2022-01-31T20:49:47.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:47.390985+0000) 2022-01-31T20:49:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:47 smithi171 conmon[41853]: debug 2022-01-31T20:49:47.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:47.519368+0000) 2022-01-31T20:49:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:47 smithi167 conmon[49112]: debug 2022-01-31T20:49:47.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:47.612697+0000) 2022-01-31T20:49:48.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:48 smithi167 conmon[60316]: debug 2022-01-31T20:49:48.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.035783+0000) 2022-01-31T20:49:48.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:48 smithi167 conmon[54076]: debug 2022-01-31T20:49:48.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.284303+0000) 2022-01-31T20:49:48.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[46715]: debug 2022-01-31T20:49:48.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.313923+0000) 2022-01-31T20:49:48.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[51620]: debug 2022-01-31T20:49:48.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.391179+0000) 2022-01-31T20:49:48.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[35325]: debug 2022-01-31T20:49:48.650+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81608 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[41853]: debug 2022-01-31T20:49:48.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.519543+0000) 2022-01-31T20:49:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[41853]: debug 2022-01-31T20:49:48.642+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.643727+0000) 2022-01-31T20:49:48.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[46715]: debug 2022-01-31T20:49:48.641+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.642328+0000) 2022-01-31T20:49:48.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:48 smithi171 conmon[51620]: debug 2022-01-31T20:49:48.641+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.641875+0000) 2022-01-31T20:49:48.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:48 smithi167 conmon[60316]: debug 2022-01-31T20:49:48.641+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.643166+0000) 2022-01-31T20:49:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:48 smithi167 conmon[49112]: debug 2022-01-31T20:49:48.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.612873+0000) 2022-01-31T20:49:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:48 smithi167 conmon[49112]: debug 2022-01-31T20:49:48.642+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.643602+0000) 2022-01-31T20:49:48.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:48 smithi167 conmon[54076]: debug 2022-01-31T20:49:48.641+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:48.642771+0000) 2022-01-31T20:49:49.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:49 smithi167 conmon[54076]: debug 2022-01-31T20:49:49.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:49.284517+0000) 2022-01-31T20:49:49.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:49 smithi167 conmon[60316]: debug 2022-01-31T20:49:49.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:49.035951+0000) 2022-01-31T20:49:49.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:49 smithi171 conmon[46715]: debug 2022-01-31T20:49:49.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:49.314107+0000) 2022-01-31T20:49:49.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:49 smithi171 conmon[51620]: debug 2022-01-31T20:49:49.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:49.391383+0000) 2022-01-31T20:49:49.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:49 smithi171 conmon[41853]: debug 2022-01-31T20:49:49.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:49.519767+0000) 2022-01-31T20:49:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:49 smithi167 conmon[49112]: debug 2022-01-31T20:49:49.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:49.613059+0000) 2022-01-31T20:49:50.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:50 smithi167 conmon[54076]: debug 2022-01-31T20:49:50.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:50.284679+0000) 2022-01-31T20:49:50.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:50 smithi167 conmon[60316]: debug 2022-01-31T20:49:50.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:50.036132+0000) 2022-01-31T20:49:50.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:50 smithi171 conmon[46715]: debug 2022-01-31T20:49:50.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:50.314279+0000) 2022-01-31T20:49:50.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:50 smithi171 conmon[51620]: debug 2022-01-31T20:49:50.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:50.391541+0000) 2022-01-31T20:49:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:50 smithi171 conmon[41853]: debug 2022-01-31T20:49:50.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:50.519900+0000) 2022-01-31T20:49:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:50 smithi167 conmon[49112]: debug 2022-01-31T20:49:50.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:50.613202+0000) 2022-01-31T20:49:51.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:51 smithi167 conmon[54076]: debug 2022-01-31T20:49:51.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:51.284861+0000) 2022-01-31T20:49:51.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:51 smithi167 conmon[60316]: debug 2022-01-31T20:49:51.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:51.036294+0000) 2022-01-31T20:49:51.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:51 smithi171 conmon[46715]: debug 2022-01-31T20:49:51.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:51.314459+0000) 2022-01-31T20:49:51.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:51 smithi171 conmon[51620]: debug 2022-01-31T20:49:51.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:51.391717+0000) 2022-01-31T20:49:51.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:51 smithi171 conmon[41853]: debug 2022-01-31T20:49:51.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:51.520021+0000) 2022-01-31T20:49:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:51 smithi167 conmon[49112]: debug 2022-01-31T20:49:51.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:51.613378+0000) 2022-01-31T20:49:52.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:52 smithi167 conmon[54076]: debug 2022-01-31T20:49:52.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:52.285019+0000) 2022-01-31T20:49:52.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:52 smithi167 conmon[60316]: debug 2022-01-31T20:49:52.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:52.036465+0000) 2022-01-31T20:49:52.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:52 smithi171 conmon[46715]: debug 2022-01-31T20:49:52.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:52.314642+0000) 2022-01-31T20:49:52.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:52 smithi171 conmon[51620]: debug 2022-01-31T20:49:52.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:52.391856+0000) 2022-01-31T20:49:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:52 smithi171 conmon[41853]: debug 2022-01-31T20:49:52.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:52.520231+0000) 2022-01-31T20:49:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:52 smithi167 conmon[49112]: debug 2022-01-31T20:49:52.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:52.613591+0000) 2022-01-31T20:49:53.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:53 smithi167 conmon[54076]: debug 2022-01-31T20:49:53.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.285158+0000) 2022-01-31T20:49:53.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:53 smithi167 conmon[60316]: debug 2022-01-31T20:49:53.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.036624+0000) 2022-01-31T20:49:53.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[46715]: debug 2022-01-31T20:49:53.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.314817+0000) 2022-01-31T20:49:53.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[51620]: debug 2022-01-31T20:49:53.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.392016+0000) 2022-01-31T20:49:53.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[46715]: debug 2022-01-31T20:49:53.653+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.654053+0000) 2022-01-31T20:49:53.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[51620]: debug 2022-01-31T20:49:53.652+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.652955+0000) 2022-01-31T20:49:53.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[35325]: debug 2022-01-31T20:49:53.578+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:49:53.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[35325]: debug 2022-01-31T20:49:53.660+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81718 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[41853]: debug 2022-01-31T20:49:53.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.520409+0000) 2022-01-31T20:49:53.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:53 smithi171 conmon[41853]: debug 2022-01-31T20:49:53.653+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.653904+0000) 2022-01-31T20:49:53.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:53 smithi167 conmon[60316]: debug 2022-01-31T20:49:53.652+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.654300+0000) 2022-01-31T20:49:53.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:53 smithi167 conmon[54076]: debug 2022-01-31T20:49:53.651+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.653170+0000) 2022-01-31T20:49:53.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:53 smithi167 conmon[49112]: debug 2022-01-31T20:49:53.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.613725+0000) 2022-01-31T20:49:53.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:53 smithi167 conmon[49112]: debug 2022-01-31T20:49:53.653+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:53.654867+0000) 2022-01-31T20:49:54.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:54 smithi167 conmon[54076]: debug 2022-01-31T20:49:54.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:54.285319+0000) 2022-01-31T20:49:54.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:54 smithi167 conmon[60316]: debug 2022-01-31T20:49:54.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:54.036785+0000) 2022-01-31T20:49:54.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:54 smithi171 conmon[51620]: debug 2022-01-31T20:49:54.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:54.392205+0000) 2022-01-31T20:49:54.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:54 smithi171 conmon[46715]: debug 2022-01-31T20:49:54.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:54 smithi171 conmon[46715]: 2022-01-31T20:49:54.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:54.315012+0000) 2022-01-31T20:49:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:54 smithi171 conmon[41853]: debug 2022-01-31T20:49:54.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:54.520598+0000) 2022-01-31T20:49:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:54 smithi167 conmon[49112]: debug 2022-01-31T20:49:54.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:54.613885+0000) 2022-01-31T20:49:55.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:55 smithi167 conmon[54076]: debug 2022-01-31T20:49:55.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:55.285462+0000) 2022-01-31T20:49:55.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:55 smithi167 conmon[60316]: debug 2022-01-31T20:49:55.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:55.036936+0000) 2022-01-31T20:49:55.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:55 smithi171 conmon[46715]: debug 2022-01-31T20:49:55.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:55.315154+0000) 2022-01-31T20:49:55.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:55 smithi171 conmon[51620]: debug 2022-01-31T20:49:55.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:55.392340+0000) 2022-01-31T20:49:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:55 smithi171 conmon[41853]: debug 2022-01-31T20:49:55.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:55.520750+0000) 2022-01-31T20:49:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:55 smithi167 conmon[49112]: debug 2022-01-31T20:49:55.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:55.614044+0000) 2022-01-31T20:49:56.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:56 smithi167 conmon[60316]: debug 2022-01-31T20:49:56.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:56.037129+0000) 2022-01-31T20:49:56.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:56 smithi167 conmon[54076]: debug 2022-01-31T20:49:56.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:56.285575+0000) 2022-01-31T20:49:56.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:56 smithi171 conmon[46715]: debug 2022-01-31T20:49:56.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:56.315337+0000) 2022-01-31T20:49:56.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:56 smithi171 conmon[51620]: debug 2022-01-31T20:49:56.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:56.392556+0000) 2022-01-31T20:49:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:56 smithi171 conmon[41853]: debug 2022-01-31T20:49:56.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:56.520899+0000) 2022-01-31T20:49:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:56 smithi167 conmon[49112]: debug 2022-01-31T20:49:56.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:56.614236+0000) 2022-01-31T20:49:57.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:57 smithi167 conmon[54076]: debug 2022-01-31T20:49:57.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:57.285737+0000) 2022-01-31T20:49:57.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:57 smithi167 conmon[60316]: debug 2022-01-31T20:49:57.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:57.037352+0000) 2022-01-31T20:49:57.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:57 smithi171 conmon[46715]: debug 2022-01-31T20:49:57.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:57.315543+0000) 2022-01-31T20:49:57.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:57 smithi171 conmon[51620]: debug 2022-01-31T20:49:57.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:57.392766+0000) 2022-01-31T20:49:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:57 smithi171 conmon[41853]: debug 2022-01-31T20:49:57.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:57.521028+0000) 2022-01-31T20:49:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:57 smithi167 conmon[49112]: debug 2022-01-31T20:49:57.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:57.614420+0000) 2022-01-31T20:49:58.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:58 smithi167 conmon[54076]: debug 2022-01-31T20:49:58.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.285951+0000) 2022-01-31T20:49:58.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:58 smithi167 conmon[60316]: debug 2022-01-31T20:49:58.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.037469+0000) 2022-01-31T20:49:58.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[51620]: debug 2022-01-31T20:49:58.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.392959+0000) 2022-01-31T20:49:58.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[46715]: debug 2022-01-31T20:49:58.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.315775+0000) 2022-01-31T20:49:58.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[51620]: debug 2022-01-31T20:49:58.662+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.663429+0000) 2022-01-31T20:49:58.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[35325]: debug 2022-01-31T20:49:58.672+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81827 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:49:58.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[46715]: debug 2022-01-31T20:49:58.663+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.664548+0000) 2022-01-31T20:49:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[41853]: debug 2022-01-31T20:49:58.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.521261+0000) 2022-01-31T20:49:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:58 smithi171 conmon[41853]: debug 2022-01-31T20:49:58.662+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.663637+0000) 2022-01-31T20:49:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:58 smithi167 conmon[49112]: debug 2022-01-31T20:49:58.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.614606+0000) 2022-01-31T20:49:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:58 smithi167 conmon[49112]: debug 2022-01-31T20:49:58.663+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.664857+0000) 2022-01-31T20:49:58.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:58 smithi167 conmon[54076]: debug 2022-01-31T20:49:58.663+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.665146+0000) 2022-01-31T20:49:58.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:58 smithi167 conmon[60316]: debug 2022-01-31T20:49:58.663+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:58.664534+0000) 2022-01-31T20:49:59.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:49:59 smithi167 conmon[54076]: debug 2022-01-31T20:49:59.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:59.286184+0000) 2022-01-31T20:49:59.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:49:59 smithi167 conmon[60316]: debug 2022-01-31T20:49:59.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:59.037684+0000) 2022-01-31T20:49:59.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:49:59 smithi171 conmon[46715]: debug 2022-01-31T20:49:59.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:59.315967+0000) 2022-01-31T20:49:59.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:49:59 smithi171 conmon[51620]: debug 2022-01-31T20:49:59.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:59.393144+0000) 2022-01-31T20:49:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:49:59 smithi171 conmon[41853]: debug 2022-01-31T20:49:59.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:59.521450+0000) 2022-01-31T20:49:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:49:59 smithi167 conmon[49112]: debug 2022-01-31T20:49:59.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:49:59.614763+0000) 2022-01-31T20:50:00.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:00 smithi167 conmon[54076]: debug 2022-01-31T20:50:00.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:00.286355+0000) 2022-01-31T20:50:00.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:00 smithi167 conmon[60316]: debug 2022-01-31T20:50:00.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:00.037816+0000) 2022-01-31T20:50:00.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:00 smithi171 conmon[46715]: debug 2022-01-31T20:50:00.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:00.316092+0000) 2022-01-31T20:50:00.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:00 smithi171 conmon[51620]: debug 2022-01-31T20:50:00.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:00.393294+0000) 2022-01-31T20:50:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:00 smithi171 conmon[41853]: debug 2022-01-31T20:50:00.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:00.521627+0000) 2022-01-31T20:50:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:00 smithi167 conmon[49112]: debug 2022-01-31T20:50:00.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:00.614861+0000) 2022-01-31T20:50:01.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:01 smithi167 conmon[54076]: debug 2022-01-31T20:50:01.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:01.286581+0000) 2022-01-31T20:50:01.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:01 smithi167 conmon[60316]: debug 2022-01-31T20:50:01.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:01.037978+0000) 2022-01-31T20:50:01.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:01 smithi171 conmon[46715]: debug 2022-01-31T20:50:01.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:01.316257+0000) 2022-01-31T20:50:01.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:01 smithi171 conmon[51620]: debug 2022-01-31T20:50:01.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:01.393442+0000) 2022-01-31T20:50:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:01 smithi171 conmon[41853]: debug 2022-01-31T20:50:01.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:01.521785+0000) 2022-01-31T20:50:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:01 smithi167 conmon[49112]: debug 2022-01-31T20:50:01.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:01.615034+0000) 2022-01-31T20:50:02.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:02 smithi167 conmon[54076]: debug 2022-01-31T20:50:02.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:02.286803+0000) 2022-01-31T20:50:02.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:02 smithi167 conmon[60316]: debug 2022-01-31T20:50:02.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:02.038183+0000) 2022-01-31T20:50:02.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:02 smithi171 conmon[46715]: debug 2022-01-31T20:50:02.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:02.316445+0000) 2022-01-31T20:50:02.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:02 smithi171 conmon[51620]: debug 2022-01-31T20:50:02.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:02.393645+0000) 2022-01-31T20:50:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:02 smithi171 conmon[41853]: debug 2022-01-31T20:50:02.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:02.521987+0000) 2022-01-31T20:50:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:02 smithi167 conmon[49112]: debug 2022-01-31T20:50:02.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:02.615243+0000) 2022-01-31T20:50:03.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:03 smithi167 conmon[54076]: debug 2022-01-31T20:50:03.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.287012+0000) 2022-01-31T20:50:03.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:03 smithi167 conmon[60316]: debug 2022-01-31T20:50:03.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.038385+0000) 2022-01-31T20:50:03.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[46715]: debug 2022-01-31T20:50:03.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.316646+0000) 2022-01-31T20:50:03.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[51620]: debug 2022-01-31T20:50:03.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.393804+0000) 2022-01-31T20:50:03.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[35325]: debug 2022-01-31T20:50:03.683+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 81937 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[41853]: debug 2022-01-31T20:50:03.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.522192+0000) 2022-01-31T20:50:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[41853]: debug 2022-01-31T20:50:03.674+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.675715+0000) 2022-01-31T20:50:03.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[46715]: debug 2022-01-31T20:50:03.675+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.676718+0000) 2022-01-31T20:50:03.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:03 smithi171 conmon[51620]: debug 2022-01-31T20:50:03.674+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.675350+0000) 2022-01-31T20:50:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:03 smithi167 conmon[49112]: debug 2022-01-31T20:50:03.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.615405+0000) 2022-01-31T20:50:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:03 smithi167 conmon[49112]: debug 2022-01-31T20:50:03.675+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.676703+0000) 2022-01-31T20:50:03.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:03 smithi167 conmon[54076]: debug 2022-01-31T20:50:03.675+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.676489+0000) 2022-01-31T20:50:03.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:03 smithi167 conmon[60316]: debug 2022-01-31T20:50:03.674+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:03.676184+0000) 2022-01-31T20:50:04.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:04 smithi167 conmon[54076]: debug 2022-01-31T20:50:04.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:04.287171+0000) 2022-01-31T20:50:04.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:04 smithi167 conmon[60316]: debug 2022-01-31T20:50:04.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:04.038546+0000) 2022-01-31T20:50:04.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:04 smithi171 conmon[46715]: debug 2022-01-31T20:50:04.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:04.316831+0000) 2022-01-31T20:50:04.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:04 smithi171 conmon[51620]: debug 2022-01-31T20:50:04.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:04.393986+0000) 2022-01-31T20:50:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:04 smithi171 conmon[41853]: debug 2022-01-31T20:50:04.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:04.522352+0000) 2022-01-31T20:50:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:04 smithi167 conmon[49112]: debug 2022-01-31T20:50:04.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:04.615608+0000) 2022-01-31T20:50:05.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:05 smithi167 conmon[54076]: debug 2022-01-31T20:50:05.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:05.287336+0000) 2022-01-31T20:50:05.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:05 smithi167 conmon[60316]: debug 2022-01-31T20:50:05.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:05.038661+0000) 2022-01-31T20:50:05.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:05 smithi171 conmon[46715]: debug 2022-01-31T20:50:05.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:05.316967+0000) 2022-01-31T20:50:05.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:05 smithi171 conmon[51620]: debug 2022-01-31T20:50:05.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:05.394170+0000) 2022-01-31T20:50:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:05 smithi171 conmon[41853]: debug 2022-01-31T20:50:05.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:05.522504+0000) 2022-01-31T20:50:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:05 smithi167 conmon[49112]: debug 2022-01-31T20:50:05.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:05.615743+0000) 2022-01-31T20:50:06.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:06 smithi167 conmon[54076]: debug 2022-01-31T20:50:06.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:06.287491+0000) 2022-01-31T20:50:06.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:06 smithi167 conmon[60316]: debug 2022-01-31T20:50:06.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:06.038811+0000) 2022-01-31T20:50:06.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:06 smithi171 conmon[46715]: debug 2022-01-31T20:50:06.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:06.317165+0000) 2022-01-31T20:50:06.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:06 smithi171 conmon[51620]: debug 2022-01-31T20:50:06.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:06.394356+0000) 2022-01-31T20:50:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:06 smithi171 conmon[41853]: debug 2022-01-31T20:50:06.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:06.522715+0000) 2022-01-31T20:50:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:06 smithi167 conmon[49112]: debug 2022-01-31T20:50:06.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:06.615868+0000) 2022-01-31T20:50:07.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:07 smithi167 conmon[54076]: debug 2022-01-31T20:50:07.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:07.287705+0000) 2022-01-31T20:50:07.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:07 smithi167 conmon[60316]: debug 2022-01-31T20:50:07.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:07.038956+0000) 2022-01-31T20:50:07.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:07 smithi171 conmon[46715]: debug 2022-01-31T20:50:07.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:07.317371+0000) 2022-01-31T20:50:07.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:07 smithi171 conmon[51620]: debug 2022-01-31T20:50:07.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:07.394560+0000) 2022-01-31T20:50:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:07 smithi171 conmon[41853]: debug 2022-01-31T20:50:07.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:07.522937+0000) 2022-01-31T20:50:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:07 smithi167 conmon[49112]: debug 2022-01-31T20:50:07.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:07.616051+0000) 2022-01-31T20:50:08.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:08 smithi167 conmon[54076]: debug 2022-01-31T20:50:08.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.287940+0000) 2022-01-31T20:50:08.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:08 smithi167 conmon[60316]: debug 2022-01-31T20:50:08.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.039106+0000) 2022-01-31T20:50:08.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[46715]: debug 2022-01-31T20:50:08.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.317540+0000) 2022-01-31T20:50:08.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[51620]: debug 2022-01-31T20:50:08.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.394796+0000) 2022-01-31T20:50:08.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[46715]: debug 2022-01-31T20:50:08.687+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.688266+0000) 2022-01-31T20:50:08.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[51620]: debug 2022-01-31T20:50:08.685+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.686164+0000) 2022-01-31T20:50:08.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[35325]: debug 2022-01-31T20:50:08.579+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:50:08.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[35325]: debug 2022-01-31T20:50:08.694+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82046 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:08.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[41853]: debug 2022-01-31T20:50:08.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.523091+0000) 2022-01-31T20:50:08.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:08 smithi171 conmon[41853]: debug 2022-01-31T20:50:08.686+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.687061+0000) 2022-01-31T20:50:08.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:08 smithi167 conmon[54076]: debug 2022-01-31T20:50:08.685+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.687261+0000) 2022-01-31T20:50:08.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:08 smithi167 conmon[60316]: debug 2022-01-31T20:50:08.685+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.687323+0000) 2022-01-31T20:50:08.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:08 smithi167 conmon[49112]: debug 2022-01-31T20:50:08.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.616257+0000) 2022-01-31T20:50:08.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:08 smithi167 conmon[49112]: debug 2022-01-31T20:50:08.685+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:08.686594+0000) 2022-01-31T20:50:09.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:09 smithi167 conmon[60316]: debug 2022-01-31T20:50:09.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:09.039290+0000) 2022-01-31T20:50:09.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:09 smithi167 conmon[54076]: debug 2022-01-31T20:50:09.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:09.288074+0000) 2022-01-31T20:50:09.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:09 smithi171 conmon[46715]: debug 2022-01-31T20:50:09.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:09.317718+0000) 2022-01-31T20:50:09.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:09 smithi171 conmon[51620]: debug 2022-01-31T20:50:09.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:09.394913+0000) 2022-01-31T20:50:09.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:09 smithi171 conmon[41853]: debug 2022-01-31T20:50:09.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:09.523260+0000) 2022-01-31T20:50:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:09 smithi167 conmon[49112]: debug 2022-01-31T20:50:09.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:09.616454+0000) 2022-01-31T20:50:10.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:10 smithi167 conmon[54076]: debug 2022-01-31T20:50:10.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:10.288249+0000) 2022-01-31T20:50:10.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:10 smithi167 conmon[60316]: debug 2022-01-31T20:50:10.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:10.039489+0000) 2022-01-31T20:50:10.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:10 smithi171 conmon[46715]: debug 2022-01-31T20:50:10.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:10.317872+0000) 2022-01-31T20:50:10.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:10 smithi171 conmon[51620]: debug 2022-01-31T20:50:10.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:10.395047+0000) 2022-01-31T20:50:10.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:10 smithi171 conmon[41853]: debug 2022-01-31T20:50:10.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:10.523400+0000) 2022-01-31T20:50:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:10 smithi167 conmon[49112]: debug 2022-01-31T20:50:10.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:10.616601+0000) 2022-01-31T20:50:11.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:11 smithi167 conmon[54076]: debug 2022-01-31T20:50:11.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:11.288454+0000) 2022-01-31T20:50:11.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:11 smithi167 conmon[60316]: debug 2022-01-31T20:50:11.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:11.039650+0000) 2022-01-31T20:50:11.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:11 smithi171 conmon[46715]: debug 2022-01-31T20:50:11.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:11.318052+0000) 2022-01-31T20:50:11.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:11 smithi171 conmon[51620]: debug 2022-01-31T20:50:11.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:11.395206+0000) 2022-01-31T20:50:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:11 smithi171 conmon[41853]: debug 2022-01-31T20:50:11.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:11.523607+0000) 2022-01-31T20:50:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:11 smithi167 conmon[49112]: debug 2022-01-31T20:50:11.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:11.616743+0000) 2022-01-31T20:50:12.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:12 smithi167 conmon[54076]: debug 2022-01-31T20:50:12.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:12.288561+0000) 2022-01-31T20:50:12.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:12 smithi167 conmon[60316]: debug 2022-01-31T20:50:12.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:12.039800+0000) 2022-01-31T20:50:12.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:12 smithi171 conmon[46715]: debug 2022-01-31T20:50:12.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:12.318255+0000) 2022-01-31T20:50:12.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:12 smithi171 conmon[51620]: debug 2022-01-31T20:50:12.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:12.395361+0000) 2022-01-31T20:50:12.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:12 smithi171 conmon[41853]: debug 2022-01-31T20:50:12.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:12.523804+0000) 2022-01-31T20:50:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:12 smithi167 conmon[49112]: debug 2022-01-31T20:50:12.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:12.616903+0000) 2022-01-31T20:50:13.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:13 smithi167 conmon[54076]: debug 2022-01-31T20:50:13.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.288722+0000) 2022-01-31T20:50:13.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:13 smithi167 conmon[60316]: debug 2022-01-31T20:50:13.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.040008+0000) 2022-01-31T20:50:13.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[46715]: debug 2022-01-31T20:50:13.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.318449+0000) 2022-01-31T20:50:13.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[51620]: debug 2022-01-31T20:50:13.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.395549+0000) 2022-01-31T20:50:13.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[35325]: debug 2022-01-31T20:50:13.705+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82156 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[41853]: debug 2022-01-31T20:50:13.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.523962+0000) 2022-01-31T20:50:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[41853]: debug 2022-01-31T20:50:13.696+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.697431+0000) 2022-01-31T20:50:13.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[46715]: debug 2022-01-31T20:50:13.698+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.698845+0000) 2022-01-31T20:50:13.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:13 smithi171 conmon[51620]: debug 2022-01-31T20:50:13.696+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.697545+0000) 2022-01-31T20:50:13.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:13 smithi167 conmon[54076]: debug 2022-01-31T20:50:13.697+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.698096+0000) 2022-01-31T20:50:13.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:13 smithi167 conmon[60316]: debug 2022-01-31T20:50:13.697+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.698305+0000) 2022-01-31T20:50:13.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:13 smithi167 conmon[49112]: debug 2022-01-31T20:50:13.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.617100+0000) 2022-01-31T20:50:13.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:13 smithi167 conmon[49112]: debug 2022-01-31T20:50:13.697+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:13.698417+0000) 2022-01-31T20:50:14.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:14 smithi167 conmon[54076]: debug 2022-01-31T20:50:14.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:14.288877+0000) 2022-01-31T20:50:14.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:14 smithi167 conmon[60316]: debug 2022-01-31T20:50:14.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:14.040137+0000) 2022-01-31T20:50:14.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:14 smithi171 conmon[46715]: debug 2022-01-31T20:50:14.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:14.318638+0000) 2022-01-31T20:50:14.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:14 smithi171 conmon[51620]: debug 2022-01-31T20:50:14.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:14.395734+0000) 2022-01-31T20:50:14.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:14 smithi171 conmon[41853]: debug 2022-01-31T20:50:14.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:14.524116+0000) 2022-01-31T20:50:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:14 smithi167 conmon[49112]: debug 2022-01-31T20:50:14.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:14.617286+0000) 2022-01-31T20:50:15.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:15 smithi167 conmon[54076]: debug 2022-01-31T20:50:15.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:15.289026+0000) 2022-01-31T20:50:15.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:15 smithi167 conmon[60316]: debug 2022-01-31T20:50:15.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:15.040262+0000) 2022-01-31T20:50:15.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:15 smithi171 conmon[46715]: debug 2022-01-31T20:50:15.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:15.318790+0000) 2022-01-31T20:50:15.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:15 smithi171 conmon[51620]: debug 2022-01-31T20:50:15.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:15.395929+0000) 2022-01-31T20:50:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:15 smithi171 conmon[41853]: debug 2022-01-31T20:50:15.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:15.524257+0000) 2022-01-31T20:50:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:15 smithi167 conmon[49112]: debug 2022-01-31T20:50:15.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:15.617441+0000) 2022-01-31T20:50:16.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:16 smithi167 conmon[54076]: debug 2022-01-31T20:50:16.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:16.289161+0000) 2022-01-31T20:50:16.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:16 smithi167 conmon[60316]: debug 2022-01-31T20:50:16.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:16.040413+0000) 2022-01-31T20:50:16.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:16 smithi171 conmon[46715]: debug 2022-01-31T20:50:16.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:16.318974+0000) 2022-01-31T20:50:16.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:16 smithi171 conmon[51620]: debug 2022-01-31T20:50:16.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:16.396111+0000) 2022-01-31T20:50:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:16 smithi171 conmon[41853]: debug 2022-01-31T20:50:16.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:16.524416+0000) 2022-01-31T20:50:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:16 smithi167 conmon[49112]: debug 2022-01-31T20:50:16.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:16.617601+0000) 2022-01-31T20:50:17.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:17 smithi167 conmon[54076]: debug 2022-01-31T20:50:17.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:17.289354+0000) 2022-01-31T20:50:17.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:17 smithi167 conmon[60316]: debug 2022-01-31T20:50:17.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:17.040587+0000) 2022-01-31T20:50:17.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:17 smithi171 conmon[46715]: debug 2022-01-31T20:50:17.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:17.319134+0000) 2022-01-31T20:50:17.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:17 smithi171 conmon[51620]: debug 2022-01-31T20:50:17.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:17.396292+0000) 2022-01-31T20:50:17.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:17 smithi171 conmon[41853]: debug 2022-01-31T20:50:17.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:17.524564+0000) 2022-01-31T20:50:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:17 smithi167 conmon[49112]: debug 2022-01-31T20:50:17.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:17.617753+0000) 2022-01-31T20:50:18.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:18 smithi167 conmon[54076]: debug 2022-01-31T20:50:18.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.289514+0000) 2022-01-31T20:50:18.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:18 smithi167 conmon[60316]: debug 2022-01-31T20:50:18.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.040745+0000) 2022-01-31T20:50:18.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[46715]: debug 2022-01-31T20:50:18.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.319319+0000) 2022-01-31T20:50:18.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[51620]: debug 2022-01-31T20:50:18.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.396530+0000) 2022-01-31T20:50:18.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[51620]: debug 2022-01-31T20:50:18.707+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.708718+0000) 2022-01-31T20:50:18.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[35325]: debug 2022-01-31T20:50:18.716+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82265 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[41853]: debug 2022-01-31T20:50:18.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.524802+0000) 2022-01-31T20:50:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[41853]: debug 2022-01-31T20:50:18.707+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.708070+0000) 2022-01-31T20:50:18.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:18 smithi171 conmon[46715]: debug 2022-01-31T20:50:18.708+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.709553+0000) 2022-01-31T20:50:18.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:18 smithi167 conmon[54076]: debug 2022-01-31T20:50:18.708+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.709039+0000) 2022-01-31T20:50:18.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:18 smithi167 conmon[60316]: debug 2022-01-31T20:50:18.708+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.709322+0000) 2022-01-31T20:50:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:18 smithi167 conmon[49112]: debug 2022-01-31T20:50:18.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.617972+0000) 2022-01-31T20:50:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:18 smithi167 conmon[49112]: debug 2022-01-31T20:50:18.708+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:18.708860+0000) 2022-01-31T20:50:19.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:19 smithi167 conmon[54076]: debug 2022-01-31T20:50:19.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:19.289670+0000) 2022-01-31T20:50:19.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:19 smithi167 conmon[60316]: debug 2022-01-31T20:50:19.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:19.040933+0000) 2022-01-31T20:50:19.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:19 smithi171 conmon[46715]: debug 2022-01-31T20:50:19.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:19.319466+0000) 2022-01-31T20:50:19.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:19 smithi171 conmon[51620]: debug 2022-01-31T20:50:19.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:19.396694+0000) 2022-01-31T20:50:19.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:19 smithi171 conmon[41853]: debug 2022-01-31T20:50:19.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:19.524977+0000) 2022-01-31T20:50:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:19 smithi167 conmon[49112]: debug 2022-01-31T20:50:19.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:19.618135+0000) 2022-01-31T20:50:20.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:20 smithi167 conmon[54076]: debug 2022-01-31T20:50:20.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:20.289789+0000) 2022-01-31T20:50:20.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:20 smithi167 conmon[60316]: debug 2022-01-31T20:50:20.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:20.041118+0000) 2022-01-31T20:50:20.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:20 smithi171 conmon[46715]: debug 2022-01-31T20:50:20.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:20.319638+0000) 2022-01-31T20:50:20.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:20 smithi171 conmon[51620]: debug 2022-01-31T20:50:20.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:20.396834+0000) 2022-01-31T20:50:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:20 smithi171 conmon[41853]: debug 2022-01-31T20:50:20.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:20.525126+0000) 2022-01-31T20:50:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:20 smithi167 conmon[49112]: debug 2022-01-31T20:50:20.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:20.618306+0000) 2022-01-31T20:50:21.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:21 smithi167 conmon[54076]: debug 2022-01-31T20:50:21.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:21.289938+0000) 2022-01-31T20:50:21.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:21 smithi167 conmon[60316]: debug 2022-01-31T20:50:21.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:21.041312+0000) 2022-01-31T20:50:21.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:21 smithi171 conmon[46715]: debug 2022-01-31T20:50:21.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:21.319820+0000) 2022-01-31T20:50:21.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:21 smithi171 conmon[51620]: debug 2022-01-31T20:50:21.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:21.396994+0000) 2022-01-31T20:50:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:21 smithi171 conmon[41853]: debug 2022-01-31T20:50:21.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:21.525350+0000) 2022-01-31T20:50:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:21 smithi167 conmon[49112]: debug 2022-01-31T20:50:21.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:21.618534+0000) 2022-01-31T20:50:22.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:22 smithi167 conmon[54076]: debug 2022-01-31T20:50:22.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:22.290072+0000) 2022-01-31T20:50:22.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:22 smithi167 conmon[60316]: debug 2022-01-31T20:50:22.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:22.041438+0000) 2022-01-31T20:50:22.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:22 smithi171 conmon[46715]: debug 2022-01-31T20:50:22.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:22.320004+0000) 2022-01-31T20:50:22.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:22 smithi171 conmon[51620]: debug 2022-01-31T20:50:22.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:22.397147+0000) 2022-01-31T20:50:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:22 smithi171 conmon[41853]: debug 2022-01-31T20:50:22.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:22.525547+0000) 2022-01-31T20:50:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:22 smithi167 conmon[49112]: debug 2022-01-31T20:50:22.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:22.618698+0000) 2022-01-31T20:50:23.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:23 smithi167 conmon[54076]: debug 2022-01-31T20:50:23.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.290195+0000) 2022-01-31T20:50:23.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:23 smithi167 conmon[60316]: debug 2022-01-31T20:50:23.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.041632+0000) 2022-01-31T20:50:23.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[46715]: debug 2022-01-31T20:50:23.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.320159+0000) 2022-01-31T20:50:23.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[51620]: debug 2022-01-31T20:50:23.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.397318+0000) 2022-01-31T20:50:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[41853]: debug 2022-01-31T20:50:23.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.525732+0000) 2022-01-31T20:50:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[41853]: debug 2022-01-31T20:50:23.717+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.719310+0000) 2022-01-31T20:50:23.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[46715]: debug 2022-01-31T20:50:23.718+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.720624+0000) 2022-01-31T20:50:23.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[51620]: debug 2022-01-31T20:50:23.718+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.720518+0000) 2022-01-31T20:50:23.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[35325]: debug 2022-01-31T20:50:23.579+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:50:23.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:23 smithi171 conmon[35325]: debug 2022-01-31T20:50:23.727+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82375 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:23.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:23 smithi167 conmon[54076]: debug 2022-01-31T20:50:23.726+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.726899+0000) 2022-01-31T20:50:23.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:23 smithi167 conmon[60316]: debug 2022-01-31T20:50:23.726+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.727085+0000) 2022-01-31T20:50:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:23 smithi167 conmon[49112]: debug 2022-01-31T20:50:23.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.618858+0000) 2022-01-31T20:50:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:23 smithi167 conmon[49112]: debug 2022-01-31T20:50:23.719+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:23.720158+0000) 2022-01-31T20:50:24.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:24 smithi167 conmon[54076]: debug 2022-01-31T20:50:24.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:24.290358+0000) 2022-01-31T20:50:24.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:24 smithi167 conmon[60316]: debug 2022-01-31T20:50:24.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:24.041784+0000) 2022-01-31T20:50:24.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:24 smithi171 conmon[46715]: debug 2022-01-31T20:50:24.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:24.320328+0000) 2022-01-31T20:50:24.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:24 smithi171 conmon[51620]: debug 2022-01-31T20:50:24.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:24.397488+0000) 2022-01-31T20:50:24.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:24 smithi171 conmon[41853]: debug 2022-01-31T20:50:24.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:24.525968+0000) 2022-01-31T20:50:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:24 smithi167 conmon[49112]: debug 2022-01-31T20:50:24.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:24.619085+0000) 2022-01-31T20:50:25.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:25 smithi167 conmon[54076]: debug 2022-01-31T20:50:25.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:25.290529+0000) 2022-01-31T20:50:25.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:25 smithi167 conmon[60316]: debug 2022-01-31T20:50:25.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:25.041939+0000) 2022-01-31T20:50:25.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:25 smithi171 conmon[46715]: debug 2022-01-31T20:50:25.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:25.320483+0000) 2022-01-31T20:50:25.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:25 smithi171 conmon[51620]: debug 2022-01-31T20:50:25.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:25.397609+0000) 2022-01-31T20:50:25.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:25 smithi171 conmon[41853]: debug 2022-01-31T20:50:25.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:25.526126+0000) 2022-01-31T20:50:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:25 smithi167 conmon[49112]: debug 2022-01-31T20:50:25.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:25.619255+0000) 2022-01-31T20:50:26.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:26 smithi167 conmon[54076]: debug 2022-01-31T20:50:26.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:26.290660+0000) 2022-01-31T20:50:26.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:26 smithi167 conmon[60316]: debug 2022-01-31T20:50:26.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:26.042099+0000) 2022-01-31T20:50:26.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:26 smithi171 conmon[46715]: debug 2022-01-31T20:50:26.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:26.320670+0000) 2022-01-31T20:50:26.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:26 smithi171 conmon[51620]: debug 2022-01-31T20:50:26.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:26.397773+0000) 2022-01-31T20:50:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:26 smithi171 conmon[41853]: debug 2022-01-31T20:50:26.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:26.526309+0000) 2022-01-31T20:50:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:26 smithi167 conmon[49112]: debug 2022-01-31T20:50:26.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:26.619497+0000) 2022-01-31T20:50:27.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:27 smithi167 conmon[54076]: debug 2022-01-31T20:50:27.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:27.290835+0000) 2022-01-31T20:50:27.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:27 smithi167 conmon[60316]: debug 2022-01-31T20:50:27.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:27.042264+0000) 2022-01-31T20:50:27.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:27 smithi171 conmon[46715]: debug 2022-01-31T20:50:27.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:27.320846+0000) 2022-01-31T20:50:27.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:27 smithi171 conmon[51620]: debug 2022-01-31T20:50:27.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:27.397978+0000) 2022-01-31T20:50:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:27 smithi171 conmon[41853]: debug 2022-01-31T20:50:27.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:27.526492+0000) 2022-01-31T20:50:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:27 smithi167 conmon[49112]: debug 2022-01-31T20:50:27.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:27.619654+0000) 2022-01-31T20:50:28.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:28 smithi167 conmon[54076]: debug 2022-01-31T20:50:28.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.291047+0000) 2022-01-31T20:50:28.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:28 smithi167 conmon[60316]: debug 2022-01-31T20:50:28.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.042418+0000) 2022-01-31T20:50:28.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[46715]: debug 2022-01-31T20:50:28.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.321017+0000) 2022-01-31T20:50:28.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[51620]: debug 2022-01-31T20:50:28.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.398154+0000) 2022-01-31T20:50:28.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[35325]: debug 2022-01-31T20:50:28.737+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82484 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[41853]: debug 2022-01-31T20:50:28.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.526684+0000) 2022-01-31T20:50:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[41853]: debug 2022-01-31T20:50:28.729+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.731436+0000) 2022-01-31T20:50:28.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[46715]: debug 2022-01-31T20:50:28.730+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.732276+0000) 2022-01-31T20:50:28.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:28 smithi171 conmon[51620]: debug 2022-01-31T20:50:28.730+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.732040+0000) 2022-01-31T20:50:28.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:28 smithi167 conmon[54076]: debug 2022-01-31T20:50:28.732+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.732759+0000) 2022-01-31T20:50:28.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:28 smithi167 conmon[60316]: debug 2022-01-31T20:50:28.731+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.732190+0000) 2022-01-31T20:50:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:28 smithi167 conmon[49112]: debug 2022-01-31T20:50:28.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.619805+0000) 2022-01-31T20:50:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:28 smithi167 conmon[49112]: debug 2022-01-31T20:50:28.737+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:28.738227+0000) 2022-01-31T20:50:29.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:29 smithi167 conmon[60316]: debug 2022-01-31T20:50:29.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:29.042607+0000) 2022-01-31T20:50:29.333 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:29 smithi167 conmon[54076]: debug 2022-01-31T20:50:29.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:29.291248+0000) 2022-01-31T20:50:29.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:29 smithi171 conmon[51620]: debug 2022-01-31T20:50:29.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:29.398326+0000) 2022-01-31T20:50:29.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:29 smithi171 conmon[46715]: debug 2022-01-31T20:50:29.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:29.321213+0000) 2022-01-31T20:50:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:29 smithi171 conmon[41853]: debug 2022-01-31T20:50:29.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:29.526870+0000) 2022-01-31T20:50:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:29 smithi167 conmon[49112]: debug 2022-01-31T20:50:29.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:29.619999+0000) 2022-01-31T20:50:30.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:30 smithi167 conmon[54076]: debug 2022-01-31T20:50:30.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:30.291400+0000) 2022-01-31T20:50:30.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:30 smithi167 conmon[60316]: debug 2022-01-31T20:50:30.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:30.042808+0000) 2022-01-31T20:50:30.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:30 smithi171 conmon[46715]: debug 2022-01-31T20:50:30.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:30.321372+0000) 2022-01-31T20:50:30.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:30 smithi171 conmon[51620]: debug 2022-01-31T20:50:30.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:30.398466+0000) 2022-01-31T20:50:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:30 smithi171 conmon[41853]: debug 2022-01-31T20:50:30.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:30.526965+0000) 2022-01-31T20:50:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:30 smithi167 conmon[49112]: debug 2022-01-31T20:50:30.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:30.620188+0000) 2022-01-31T20:50:31.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:31 smithi167 conmon[54076]: debug 2022-01-31T20:50:31.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:31.291611+0000) 2022-01-31T20:50:31.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:31 smithi167 conmon[60316]: debug 2022-01-31T20:50:31.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:31.042973+0000) 2022-01-31T20:50:31.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:31 smithi171 conmon[46715]: debug 2022-01-31T20:50:31.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:31.321526+0000) 2022-01-31T20:50:31.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:31 smithi171 conmon[51620]: debug 2022-01-31T20:50:31.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:31.398654+0000) 2022-01-31T20:50:31.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:31 smithi171 conmon[41853]: debug 2022-01-31T20:50:31.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:31.527079+0000) 2022-01-31T20:50:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:31 smithi167 conmon[49112]: debug 2022-01-31T20:50:31.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:31.620410+0000) 2022-01-31T20:50:32.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:32 smithi167 conmon[54076]: debug 2022-01-31T20:50:32.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:32.291760+0000) 2022-01-31T20:50:32.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:32 smithi167 conmon[60316]: debug 2022-01-31T20:50:32.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:32.043168+0000) 2022-01-31T20:50:32.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:32 smithi171 conmon[46715]: debug 2022-01-31T20:50:32.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:32.321709+0000) 2022-01-31T20:50:32.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:32 smithi171 conmon[51620]: debug 2022-01-31T20:50:32.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:32.398807+0000) 2022-01-31T20:50:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:32 smithi171 conmon[41853]: debug 2022-01-31T20:50:32.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:32.527288+0000) 2022-01-31T20:50:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:32 smithi167 conmon[49112]: debug 2022-01-31T20:50:32.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:32.620603+0000) 2022-01-31T20:50:33.333 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:33 smithi167 conmon[54076]: debug 2022-01-31T20:50:33.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.291920+0000) 2022-01-31T20:50:33.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:33 smithi167 conmon[60316]: debug 2022-01-31T20:50:33.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.043359+0000) 2022-01-31T20:50:33.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[46715]: debug 2022-01-31T20:50:33.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.321939+0000) 2022-01-31T20:50:33.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[51620]: debug 2022-01-31T20:50:33.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.398981+0000) 2022-01-31T20:50:33.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[35325]: debug 2022-01-31T20:50:33.749+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82594 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[41853]: debug 2022-01-31T20:50:33.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.527498+0000) 2022-01-31T20:50:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[41853]: debug 2022-01-31T20:50:33.741+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.741839+0000) 2022-01-31T20:50:33.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[46715]: debug 2022-01-31T20:50:33.741+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.742692+0000) 2022-01-31T20:50:33.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:33 smithi171 conmon[51620]: debug 2022-01-31T20:50:33.742+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.742999+0000) 2022-01-31T20:50:33.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:33 smithi167 conmon[54076]: debug 2022-01-31T20:50:33.742+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.743467+0000) 2022-01-31T20:50:33.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:33 smithi167 conmon[60316]: debug 2022-01-31T20:50:33.742+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.742796+0000) 2022-01-31T20:50:33.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:33 smithi167 conmon[49112]: debug 2022-01-31T20:50:33.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.620765+0000) 2022-01-31T20:50:33.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:33 smithi167 conmon[49112]: debug 2022-01-31T20:50:33.742+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:33.743388+0000) 2022-01-31T20:50:34.333 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:34 smithi167 conmon[54076]: debug 2022-01-31T20:50:34.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:34.292086+0000) 2022-01-31T20:50:34.334 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:34 smithi167 conmon[60316]: debug 2022-01-31T20:50:34.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:34.043522+0000) 2022-01-31T20:50:34.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:34 smithi171 conmon[51620]: debug 2022-01-31T20:50:34.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:34.399167+0000) 2022-01-31T20:50:34.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:34 smithi171 conmon[46715]: debug 2022-01-31T20:50:34.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:34.322058+0000) 2022-01-31T20:50:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:34 smithi171 conmon[41853]: debug 2022-01-31T20:50:34.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:34.527685+0000) 2022-01-31T20:50:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:34 smithi167 conmon[49112]: debug 2022-01-31T20:50:34.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:34.620961+0000) 2022-01-31T20:50:35.333 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:35 smithi167 conmon[54076]: debug 2022-01-31T20:50:35.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:35.292237+0000) 2022-01-31T20:50:35.334 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:35 smithi167 conmon[60316]: debug 2022-01-31T20:50:35.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:35.043656+0000) 2022-01-31T20:50:35.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:35 smithi171 conmon[46715]: debug 2022-01-31T20:50:35.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:35.322205+0000) 2022-01-31T20:50:35.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:35 smithi171 conmon[51620]: debug 2022-01-31T20:50:35.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:35.399288+0000) 2022-01-31T20:50:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:35 smithi171 conmon[41853]: debug 2022-01-31T20:50:35.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:35.527792+0000) 2022-01-31T20:50:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:35 smithi167 conmon[49112]: debug 2022-01-31T20:50:35.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:35.621086+0000) 2022-01-31T20:50:36.333 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:36 smithi167 conmon[54076]: debug 2022-01-31T20:50:36.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:36.292443+0000) 2022-01-31T20:50:36.334 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:36 smithi167 conmon[60316]: debug 2022-01-31T20:50:36.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:36.043847+0000) 2022-01-31T20:50:36.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:36 smithi171 conmon[51620]: debug 2022-01-31T20:50:36.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:36.399435+0000) 2022-01-31T20:50:36.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:36 smithi171 conmon[46715]: debug 2022-01-31T20:50:36.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:36.322384+0000) 2022-01-31T20:50:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:36 smithi171 conmon[41853]: debug 2022-01-31T20:50:36.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:36.527991+0000) 2022-01-31T20:50:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:36 smithi167 conmon[49112]: debug 2022-01-31T20:50:36.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:36.621237+0000) 2022-01-31T20:50:37.334 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:37 smithi167 conmon[54076]: debug 2022-01-31T20:50:37.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:37.292642+0000) 2022-01-31T20:50:37.334 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:37 smithi167 conmon[60316]: debug 2022-01-31T20:50:37.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:37.043990+0000) 2022-01-31T20:50:37.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:37 smithi171 conmon[46715]: debug 2022-01-31T20:50:37.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:37.322539+0000) 2022-01-31T20:50:37.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:37 smithi171 conmon[51620]: debug 2022-01-31T20:50:37.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:37.399614+0000) 2022-01-31T20:50:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:37 smithi171 conmon[41853]: debug 2022-01-31T20:50:37.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:37.528181+0000) 2022-01-31T20:50:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:37 smithi167 conmon[49112]: debug 2022-01-31T20:50:37.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:37.621404+0000) 2022-01-31T20:50:38.334 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:38 smithi167 conmon[54076]: debug 2022-01-31T20:50:38.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.292830+0000) 2022-01-31T20:50:38.334 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:38 smithi167 conmon[60316]: debug 2022-01-31T20:50:38.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.044196+0000) 2022-01-31T20:50:38.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[46715]: debug 2022-01-31T20:50:38.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.322740+0000) 2022-01-31T20:50:38.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[51620]: debug 2022-01-31T20:50:38.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.399849+0000) 2022-01-31T20:50:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[35325]: debug 2022-01-31T20:50:38.580+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:50:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[35325]: debug 2022-01-31T20:50:38.761+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82703 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:38.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[46715]: debug 2022-01-31T20:50:38.752+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.753342+0000) 2022-01-31T20:50:38.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[51620]: debug 2022-01-31T20:50:38.752+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.753261+0000) 2022-01-31T20:50:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[41853]: debug 2022-01-31T20:50:38.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.528347+0000) 2022-01-31T20:50:38.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:38 smithi171 conmon[41853]: debug 2022-01-31T20:50:38.751+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.752712+0000) 2022-01-31T20:50:38.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:38 smithi167 conmon[54076]: debug 2022-01-31T20:50:38.752+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.754327+0000) 2022-01-31T20:50:38.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:38 smithi167 conmon[60316]: debug 2022-01-31T20:50:38.751+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.753462+0000) 2022-01-31T20:50:38.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:38 smithi167 conmon[49112]: debug 2022-01-31T20:50:38.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.621516+0000) 2022-01-31T20:50:38.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:38 smithi167 conmon[49112]: debug 2022-01-31T20:50:38.752+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:38.753527+0000) 2022-01-31T20:50:39.334 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:39 smithi167 conmon[54076]: debug 2022-01-31T20:50:39.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:39.293029+0000) 2022-01-31T20:50:39.334 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:39 smithi167 conmon[60316]: debug 2022-01-31T20:50:39.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:39.044392+0000) 2022-01-31T20:50:39.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:39 smithi171 conmon[46715]: debug 2022-01-31T20:50:39.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:39.322948+0000) 2022-01-31T20:50:39.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:39 smithi171 conmon[51620]: debug 2022-01-31T20:50:39.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:39.399981+0000) 2022-01-31T20:50:39.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:39 smithi171 conmon[41853]: debug 2022-01-31T20:50:39.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:39.528501+0000) 2022-01-31T20:50:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:39 smithi167 conmon[49112]: debug 2022-01-31T20:50:39.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:39.621656+0000) 2022-01-31T20:50:40.334 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:40 smithi167 conmon[54076]: debug 2022-01-31T20:50:40.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:40.293166+0000) 2022-01-31T20:50:40.335 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:40 smithi167 conmon[60316]: debug 2022-01-31T20:50:40.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:40.044546+0000) 2022-01-31T20:50:40.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:40 smithi171 conmon[46715]: debug 2022-01-31T20:50:40.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:40.323077+0000) 2022-01-31T20:50:40.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:40 smithi171 conmon[51620]: debug 2022-01-31T20:50:40.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:40.400103+0000) 2022-01-31T20:50:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:40 smithi171 conmon[41853]: debug 2022-01-31T20:50:40.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:40.528676+0000) 2022-01-31T20:50:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:40 smithi167 conmon[49112]: debug 2022-01-31T20:50:40.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:40.621780+0000) 2022-01-31T20:50:41.334 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:41 smithi167 conmon[54076]: debug 2022-01-31T20:50:41.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:41.293374+0000) 2022-01-31T20:50:41.335 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:41 smithi167 conmon[60316]: debug 2022-01-31T20:50:41.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:41.044736+0000) 2022-01-31T20:50:41.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:41 smithi171 conmon[46715]: debug 2022-01-31T20:50:41.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:41.323232+0000) 2022-01-31T20:50:41.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:41 smithi171 conmon[51620]: debug 2022-01-31T20:50:41.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:41.400197+0000) 2022-01-31T20:50:41.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:41 smithi171 conmon[41853]: debug 2022-01-31T20:50:41.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:41.528863+0000) 2022-01-31T20:50:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:41 smithi167 conmon[49112]: debug 2022-01-31T20:50:41.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:41.621961+0000) 2022-01-31T20:50:42.335 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:42 smithi167 conmon[54076]: debug 2022-01-31T20:50:42.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:42.293549+0000) 2022-01-31T20:50:42.335 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:42 smithi167 conmon[60316]: debug 2022-01-31T20:50:42.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:42.044866+0000) 2022-01-31T20:50:42.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:42 smithi171 conmon[46715]: debug 2022-01-31T20:50:42.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:42.323389+0000) 2022-01-31T20:50:42.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:42 smithi171 conmon[51620]: debug 2022-01-31T20:50:42.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:42.400293+0000) 2022-01-31T20:50:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:42 smithi171 conmon[41853]: debug 2022-01-31T20:50:42.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:42.529017+0000) 2022-01-31T20:50:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:42 smithi167 conmon[49112]: debug 2022-01-31T20:50:42.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:42.622133+0000) 2022-01-31T20:50:43.335 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:43 smithi167 conmon[54076]: debug 2022-01-31T20:50:43.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.293732+0000) 2022-01-31T20:50:43.335 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:43 smithi167 conmon[60316]: debug 2022-01-31T20:50:43.044+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.045007+0000) 2022-01-31T20:50:43.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[46715]: debug 2022-01-31T20:50:43.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.323551+0000) 2022-01-31T20:50:43.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[51620]: debug 2022-01-31T20:50:43.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.400403+0000) 2022-01-31T20:50:43.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[51620]: debug 2022-01-31T20:50:43.764+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.764880+0000) 2022-01-31T20:50:43.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[35325]: debug 2022-01-31T20:50:43.771+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82814 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[41853]: debug 2022-01-31T20:50:43.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.529203+0000) 2022-01-31T20:50:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[41853]: debug 2022-01-31T20:50:43.762+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.763571+0000) 2022-01-31T20:50:43.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:43 smithi171 conmon[46715]: debug 2022-01-31T20:50:43.763+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.764278+0000) 2022-01-31T20:50:43.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:43 smithi167 conmon[54076]: debug 2022-01-31T20:50:43.764+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.765896+0000) 2022-01-31T20:50:43.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:43 smithi167 conmon[60316]: debug 2022-01-31T20:50:43.763+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.764722+0000) 2022-01-31T20:50:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:43 smithi167 conmon[49112]: debug 2022-01-31T20:50:43.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.622350+0000) 2022-01-31T20:50:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:43 smithi167 conmon[49112]: debug 2022-01-31T20:50:43.762+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:43.764485+0000) 2022-01-31T20:50:44.335 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:44 smithi167 conmon[54076]: debug 2022-01-31T20:50:44.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:44.293928+0000) 2022-01-31T20:50:44.335 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:44 smithi167 conmon[60316]: debug 2022-01-31T20:50:44.044+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:44.045172+0000) 2022-01-31T20:50:44.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:44 smithi171 conmon[46715]: debug 2022-01-31T20:50:44.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:44.323719+0000) 2022-01-31T20:50:44.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:44 smithi171 conmon[51620]: debug 2022-01-31T20:50:44.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:44.400545+0000) 2022-01-31T20:50:44.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:44 smithi171 conmon[41853]: debug 2022-01-31T20:50:44.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:44.529385+0000) 2022-01-31T20:50:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:44 smithi167 conmon[49112]: debug 2022-01-31T20:50:44.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:44.622587+0000) 2022-01-31T20:50:45.336 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:45 smithi167 conmon[54076]: debug 2022-01-31T20:50:45.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:45.294074+0000) 2022-01-31T20:50:45.337 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:45 smithi167 conmon[60316]: debug 2022-01-31T20:50:45.044+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:45.045338+0000) 2022-01-31T20:50:45.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:45 smithi171 conmon[46715]: debug 2022-01-31T20:50:45.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:45.323907+0000) 2022-01-31T20:50:45.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:45 smithi171 conmon[51620]: debug 2022-01-31T20:50:45.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:45.400678+0000) 2022-01-31T20:50:45.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:45 smithi171 conmon[41853]: debug 2022-01-31T20:50:45.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:45.529478+0000) 2022-01-31T20:50:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:45 smithi167 conmon[49112]: debug 2022-01-31T20:50:45.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:45.622703+0000) 2022-01-31T20:50:46.335 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:46 smithi167 conmon[54076]: debug 2022-01-31T20:50:46.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:46.294241+0000) 2022-01-31T20:50:46.336 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:46 smithi167 conmon[60316]: debug 2022-01-31T20:50:46.044+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:46.045548+0000) 2022-01-31T20:50:46.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:46 smithi171 conmon[46715]: debug 2022-01-31T20:50:46.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:46.324093+0000) 2022-01-31T20:50:46.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:46 smithi171 conmon[51620]: debug 2022-01-31T20:50:46.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:46.400840+0000) 2022-01-31T20:50:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:46 smithi171 conmon[41853]: debug 2022-01-31T20:50:46.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:46.529634+0000) 2022-01-31T20:50:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:46 smithi167 conmon[49112]: debug 2022-01-31T20:50:46.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:46.622853+0000) 2022-01-31T20:50:47.336 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:47 smithi167 conmon[54076]: debug 2022-01-31T20:50:47.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:47.294442+0000) 2022-01-31T20:50:47.336 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:47 smithi167 conmon[60316]: debug 2022-01-31T20:50:47.044+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:47.045765+0000) 2022-01-31T20:50:47.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:47 smithi171 conmon[46715]: debug 2022-01-31T20:50:47.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:47.324256+0000) 2022-01-31T20:50:47.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:47 smithi171 conmon[51620]: debug 2022-01-31T20:50:47.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:47.400988+0000) 2022-01-31T20:50:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:47 smithi171 conmon[41853]: debug 2022-01-31T20:50:47.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:47.529819+0000) 2022-01-31T20:50:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:47 smithi167 conmon[49112]: debug 2022-01-31T20:50:47.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:47.623064+0000) 2022-01-31T20:50:48.218 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:50:48.218+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:50:48.336 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:48 smithi167 conmon[54076]: debug 2022-01-31T20:50:48.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.294611+0000) 2022-01-31T20:50:48.337 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:48 smithi167 conmon[60316]: debug 2022-01-31T20:50:48.045+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.045947+0000) 2022-01-31T20:50:48.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[46715]: debug 2022-01-31T20:50:48.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.324470+0000) 2022-01-31T20:50:48.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[51620]: debug 2022-01-31T20:50:48.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.401198+0000) 2022-01-31T20:50:48.781 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[46715]: debug 2022-01-31T20:50:48.773+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.774669+0000) 2022-01-31T20:50:48.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[51620]: debug 2022-01-31T20:50:48.774+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.775497+0000) 2022-01-31T20:50:48.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[41853]: debug 2022-01-31T20:50:48.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.529969+0000) 2022-01-31T20:50:48.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[41853]: debug 2022-01-31T20:50:48.775+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.776184+0000) 2022-01-31T20:50:48.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:48 smithi167 conmon[54076]: debug 2022-01-31T20:50:48.773+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.774997+0000) 2022-01-31T20:50:48.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:48 smithi167 conmon[60316]: debug 2022-01-31T20:50:48.774+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.776002+0000) 2022-01-31T20:50:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:48 smithi167 conmon[49112]: debug 2022-01-31T20:50:48.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.623259+0000) 2022-01-31T20:50:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:48 smithi167 conmon[49112]: debug 2022-01-31T20:50:48.773+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:48.775239+0000) 2022-01-31T20:50:49.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:48 smithi171 conmon[35325]: debug 2022-01-31T20:50:48.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 82923 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:49.336 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:49 smithi167 conmon[54076]: debug 2022-01-31T20:50:49.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:49.294814+0000) 2022-01-31T20:50:49.336 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:49 smithi167 conmon[60316]: debug 2022-01-31T20:50:49.044+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:49.046151+0000) 2022-01-31T20:50:49.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:49 smithi171 conmon[46715]: debug 2022-01-31T20:50:49.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:49.324661+0000) 2022-01-31T20:50:49.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:49 smithi171 conmon[51620]: debug 2022-01-31T20:50:49.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:49.401385+0000) 2022-01-31T20:50:49.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:49 smithi171 conmon[41853]: debug 2022-01-31T20:50:49.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:49.530149+0000) 2022-01-31T20:50:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:49 smithi167 conmon[49112]: debug 2022-01-31T20:50:49.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:49.623441+0000) 2022-01-31T20:50:50.336 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:50 smithi167 conmon[54076]: debug 2022-01-31T20:50:50.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:50.294973+0000) 2022-01-31T20:50:50.337 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:50 smithi167 conmon[60316]: debug 2022-01-31T20:50:50.045+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:50.046327+0000) 2022-01-31T20:50:50.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:50 smithi171 conmon[46715]: debug 2022-01-31T20:50:50.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:50.324812+0000) 2022-01-31T20:50:50.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:50 smithi171 conmon[51620]: debug 2022-01-31T20:50:50.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:50.401538+0000) 2022-01-31T20:50:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:50 smithi171 conmon[41853]: debug 2022-01-31T20:50:50.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:50.530340+0000) 2022-01-31T20:50:50.922 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:50 smithi167 conmon[49112]: debug 2022-01-31T20:50:50.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:50.623598+0000) 2022-01-31T20:50:51.337 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:51 smithi167 conmon[54076]: debug 2022-01-31T20:50:51.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:51.295175+0000) 2022-01-31T20:50:51.337 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:51 smithi167 conmon[60316]: debug 2022-01-31T20:50:51.045+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:51.046513+0000) 2022-01-31T20:50:51.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:51 smithi171 conmon[46715]: debug 2022-01-31T20:50:51.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:51.324996+0000) 2022-01-31T20:50:51.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:51 smithi171 conmon[51620]: debug 2022-01-31T20:50:51.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:51.401748+0000) 2022-01-31T20:50:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:51 smithi171 conmon[41853]: debug 2022-01-31T20:50:51.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:51.530521+0000) 2022-01-31T20:50:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:51 smithi167 conmon[49112]: debug 2022-01-31T20:50:51.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:51.623727+0000) 2022-01-31T20:50:52.337 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:52 smithi167 conmon[54076]: debug 2022-01-31T20:50:52.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:52.295378+0000) 2022-01-31T20:50:52.337 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:52 smithi167 conmon[60316]: debug 2022-01-31T20:50:52.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:52.046714+0000) 2022-01-31T20:50:52.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:52 smithi171 conmon[46715]: debug 2022-01-31T20:50:52.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:52.325181+0000) 2022-01-31T20:50:52.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:52 smithi171 conmon[51620]: debug 2022-01-31T20:50:52.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:52.401991+0000) 2022-01-31T20:50:52.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:52 smithi171 conmon[41853]: debug 2022-01-31T20:50:52.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:52.530719+0000) 2022-01-31T20:50:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:52 smithi167 conmon[49112]: debug 2022-01-31T20:50:52.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:52.623847+0000) 2022-01-31T20:50:53.337 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:53 smithi167 conmon[54076]: debug 2022-01-31T20:50:53.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.295564+0000) 2022-01-31T20:50:53.337 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:53 smithi167 conmon[60316]: debug 2022-01-31T20:50:53.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.046882+0000) 2022-01-31T20:50:53.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[46715]: debug 2022-01-31T20:50:53.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.325369+0000) 2022-01-31T20:50:53.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[51620]: debug 2022-01-31T20:50:53.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.402140+0000) 2022-01-31T20:50:53.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[41853]: debug 2022-01-31T20:50:53.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.530937+0000) 2022-01-31T20:50:53.784 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[35325]: debug 2022-01-31T20:50:53.581+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:50:53.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:53 smithi167 conmon[54076]: debug 2022-01-31T20:50:53.785+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.787015+0000) 2022-01-31T20:50:53.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:53 smithi167 conmon[49112]: debug 2022-01-31T20:50:53.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.623988+0000) 2022-01-31T20:50:53.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:53 smithi167 conmon[49112]: debug 2022-01-31T20:50:53.784+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.785832+0000) 2022-01-31T20:50:53.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:53 smithi167 conmon[60316]: debug 2022-01-31T20:50:53.785+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.787435+0000) 2022-01-31T20:50:54.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[35325]: debug 2022-01-31T20:50:53.794+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83035 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:54.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[41853]: debug 2022-01-31T20:50:53.786+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.787148+0000) 2022-01-31T20:50:54.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[46715]: debug 2022-01-31T20:50:53.784+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.784874+0000) 2022-01-31T20:50:54.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:53 smithi171 conmon[51620]: debug 2022-01-31T20:50:53.785+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:53.786201+0000) 2022-01-31T20:50:54.337 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:54 smithi167 conmon[54076]: debug 2022-01-31T20:50:54.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:54.295745+0000) 2022-01-31T20:50:54.338 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:54 smithi167 conmon[60316]: debug 2022-01-31T20:50:54.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:54.047072+0000) 2022-01-31T20:50:54.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:54 smithi171 conmon[51620]: debug 2022-01-31T20:50:54.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:54.402300+0000) 2022-01-31T20:50:54.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:54 smithi171 conmon[46715]: debug 2022-01-31T20:50:54.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:54.325589+0000) 2022-01-31T20:50:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:54 smithi171 conmon[41853]: debug 2022-01-31T20:50:54.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:54.531084+0000) 2022-01-31T20:50:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:54 smithi167 conmon[49112]: debug 2022-01-31T20:50:54.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:54.624187+0000) 2022-01-31T20:50:55.337 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:55 smithi167 conmon[54076]: debug 2022-01-31T20:50:55.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:55.295913+0000) 2022-01-31T20:50:55.338 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:55 smithi167 conmon[60316]: debug 2022-01-31T20:50:55.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:55.047234+0000) 2022-01-31T20:50:55.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:55 smithi171 conmon[46715]: debug 2022-01-31T20:50:55.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:55.325739+0000) 2022-01-31T20:50:55.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:55 smithi171 conmon[51620]: debug 2022-01-31T20:50:55.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:55.402470+0000) 2022-01-31T20:50:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:55 smithi171 conmon[41853]: debug 2022-01-31T20:50:55.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:55.531236+0000) 2022-01-31T20:50:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:55 smithi167 conmon[49112]: debug 2022-01-31T20:50:55.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:55.624356+0000) 2022-01-31T20:50:56.337 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:56 smithi167 conmon[54076]: debug 2022-01-31T20:50:56.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:56.296078+0000) 2022-01-31T20:50:56.338 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:56 smithi167 conmon[60316]: debug 2022-01-31T20:50:56.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:56.047395+0000) 2022-01-31T20:50:56.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:56 smithi171 conmon[46715]: debug 2022-01-31T20:50:56.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:56.325931+0000) 2022-01-31T20:50:56.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:56 smithi171 conmon[51620]: debug 2022-01-31T20:50:56.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:56.402653+0000) 2022-01-31T20:50:56.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:56 smithi171 conmon[41853]: debug 2022-01-31T20:50:56.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:56.531417+0000) 2022-01-31T20:50:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:56 smithi167 conmon[49112]: debug 2022-01-31T20:50:56.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:56.624561+0000) 2022-01-31T20:50:57.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:57 smithi167 conmon[60316]: debug 2022-01-31T20:50:57.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:57.047536+0000) 2022-01-31T20:50:57.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:57 smithi167 conmon[54076]: debug 2022-01-31T20:50:57.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:57.296221+0000) 2022-01-31T20:50:57.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:57 smithi171 conmon[46715]: debug 2022-01-31T20:50:57.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:57.326117+0000) 2022-01-31T20:50:57.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:57 smithi171 conmon[51620]: debug 2022-01-31T20:50:57.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:57.402853+0000) 2022-01-31T20:50:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:57 smithi171 conmon[41853]: debug 2022-01-31T20:50:57.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:57.531577+0000) 2022-01-31T20:50:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:57 smithi167 conmon[49112]: debug 2022-01-31T20:50:57.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:57.624688+0000) 2022-01-31T20:50:58.338 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:58 smithi167 conmon[54076]: debug 2022-01-31T20:50:58.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.296422+0000) 2022-01-31T20:50:58.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:58 smithi167 conmon[60316]: debug 2022-01-31T20:50:58.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.047646+0000) 2022-01-31T20:50:58.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[46715]: debug 2022-01-31T20:50:58.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.326296+0000) 2022-01-31T20:50:58.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[51620]: debug 2022-01-31T20:50:58.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.403048+0000) 2022-01-31T20:50:58.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[41853]: debug 2022-01-31T20:50:58.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.531763+0000) 2022-01-31T20:50:58.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[51620]: debug 2022-01-31T20:50:58.796+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.797256+0000) 2022-01-31T20:50:58.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:58 smithi167 conmon[54076]: debug 2022-01-31T20:50:58.797+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.799098+0000) 2022-01-31T20:50:58.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:58 smithi167 conmon[60316]: debug 2022-01-31T20:50:58.797+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.798984+0000) 2022-01-31T20:50:58.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:58 smithi167 conmon[49112]: debug 2022-01-31T20:50:58.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.624846+0000) 2022-01-31T20:50:58.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:58 smithi167 conmon[49112]: debug 2022-01-31T20:50:58.796+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.797927+0000) 2022-01-31T20:50:59.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[35325]: debug 2022-01-31T20:50:58.805+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83143 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:50:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[41853]: debug 2022-01-31T20:50:58.797+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.798457+0000) 2022-01-31T20:50:59.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:58 smithi171 conmon[46715]: debug 2022-01-31T20:50:58.797+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:58.798726+0000) 2022-01-31T20:50:59.338 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:50:59 smithi167 conmon[60316]: debug 2022-01-31T20:50:59.046+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:59.047794+0000) 2022-01-31T20:50:59.338 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:50:59 smithi167 conmon[54076]: debug 2022-01-31T20:50:59.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:59.296606+0000) 2022-01-31T20:50:59.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:50:59 smithi171 conmon[46715]: debug 2022-01-31T20:50:59.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:59.326452+0000) 2022-01-31T20:50:59.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:50:59 smithi171 conmon[51620]: debug 2022-01-31T20:50:59.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:59.403226+0000) 2022-01-31T20:50:59.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:50:59 smithi171 conmon[41853]: debug 2022-01-31T20:50:59.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:59.531918+0000) 2022-01-31T20:50:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:50:59 smithi167 conmon[49112]: debug 2022-01-31T20:50:59.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:50:59.625058+0000) 2022-01-31T20:51:00.338 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:00 smithi167 conmon[54076]: debug 2022-01-31T20:51:00.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:00.296700+0000) 2022-01-31T20:51:00.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:00 smithi167 conmon[60316]: debug 2022-01-31T20:51:00.047+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:00.047964+0000) 2022-01-31T20:51:00.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:00 smithi171 conmon[46715]: debug 2022-01-31T20:51:00.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:00.326569+0000) 2022-01-31T20:51:00.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:00 smithi171 conmon[51620]: debug 2022-01-31T20:51:00.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:00.403362+0000) 2022-01-31T20:51:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:00 smithi171 conmon[41853]: debug 2022-01-31T20:51:00.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:00.532059+0000) 2022-01-31T20:51:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:00 smithi167 conmon[49112]: debug 2022-01-31T20:51:00.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:00.625244+0000) 2022-01-31T20:51:01.338 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:01 smithi167 conmon[54076]: debug 2022-01-31T20:51:01.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:01.296856+0000) 2022-01-31T20:51:01.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:01 smithi167 conmon[60316]: debug 2022-01-31T20:51:01.047+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:01.048114+0000) 2022-01-31T20:51:01.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:01 smithi171 conmon[46715]: debug 2022-01-31T20:51:01.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:01.326704+0000) 2022-01-31T20:51:01.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:01 smithi171 conmon[51620]: debug 2022-01-31T20:51:01.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:01.403561+0000) 2022-01-31T20:51:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:01 smithi171 conmon[41853]: debug 2022-01-31T20:51:01.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:01.532241+0000) 2022-01-31T20:51:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:01 smithi167 conmon[49112]: debug 2022-01-31T20:51:01.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:01.625427+0000) 2022-01-31T20:51:02.339 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:02 smithi167 conmon[54076]: debug 2022-01-31T20:51:02.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:02.297057+0000) 2022-01-31T20:51:02.340 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:02 smithi167 conmon[60316]: debug 2022-01-31T20:51:02.047+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:02.048329+0000) 2022-01-31T20:51:02.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:02 smithi171 conmon[46715]: debug 2022-01-31T20:51:02.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:02.326860+0000) 2022-01-31T20:51:02.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:02 smithi171 conmon[51620]: debug 2022-01-31T20:51:02.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:02.403779+0000) 2022-01-31T20:51:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:02 smithi171 conmon[41853]: debug 2022-01-31T20:51:02.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:02.532474+0000) 2022-01-31T20:51:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:02 smithi167 conmon[49112]: debug 2022-01-31T20:51:02.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:02.625596+0000) 2022-01-31T20:51:03.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:03 smithi167 conmon[60316]: debug 2022-01-31T20:51:03.048+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.048531+0000) 2022-01-31T20:51:03.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:03 smithi167 conmon[54076]: debug 2022-01-31T20:51:03.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.297242+0000) 2022-01-31T20:51:03.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[51620]: debug 2022-01-31T20:51:03.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.403960+0000) 2022-01-31T20:51:03.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[46715]: debug 2022-01-31T20:51:03.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.326999+0000) 2022-01-31T20:51:03.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[51620]: debug 2022-01-31T20:51:03.807+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.808467+0000) 2022-01-31T20:51:03.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[41853]: debug 2022-01-31T20:51:03.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.532649+0000) 2022-01-31T20:51:03.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[41853]: debug 2022-01-31T20:51:03.807+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.808276+0000) 2022-01-31T20:51:03.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:03 smithi167 conmon[54076]: debug 2022-01-31T20:51:03.809+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.809760+0000) 2022-01-31T20:51:03.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:03 smithi167 conmon[60316]: debug 2022-01-31T20:51:03.809+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.809560+0000) 2022-01-31T20:51:03.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:03 smithi167 conmon[49112]: debug 2022-01-31T20:51:03.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.625733+0000) 2022-01-31T20:51:03.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:03 smithi167 conmon[49112]: debug 2022-01-31T20:51:03.807+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.808388+0000) 2022-01-31T20:51:04.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[35325]: debug 2022-01-31T20:51:03.817+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83254 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:04.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:03 smithi171 conmon[46715]: debug 2022-01-31T20:51:03.808+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:03.809425+0000) 2022-01-31T20:51:04.339 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:04 smithi167 conmon[54076]: debug 2022-01-31T20:51:04.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:04.297435+0000) 2022-01-31T20:51:04.340 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:04 smithi167 conmon[60316]: debug 2022-01-31T20:51:04.048+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:04.048707+0000) 2022-01-31T20:51:04.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:04 smithi171 conmon[46715]: debug 2022-01-31T20:51:04.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:04.327114+0000) 2022-01-31T20:51:04.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:04 smithi171 conmon[51620]: debug 2022-01-31T20:51:04.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:04.404118+0000) 2022-01-31T20:51:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:04 smithi171 conmon[41853]: debug 2022-01-31T20:51:04.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:04.532848+0000) 2022-01-31T20:51:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:04 smithi167 conmon[49112]: debug 2022-01-31T20:51:04.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:04.625889+0000) 2022-01-31T20:51:05.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:05 smithi167 conmon[60316]: debug 2022-01-31T20:51:05.048+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:05.048862+0000) 2022-01-31T20:51:05.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:05 smithi167 conmon[54076]: debug 2022-01-31T20:51:05.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:05.297588+0000) 2022-01-31T20:51:05.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:05 smithi171 conmon[46715]: debug 2022-01-31T20:51:05.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:05.327259+0000) 2022-01-31T20:51:05.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:05 smithi171 conmon[51620]: debug 2022-01-31T20:51:05.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:05.404261+0000) 2022-01-31T20:51:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:05 smithi171 conmon[41853]: debug 2022-01-31T20:51:05.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:05.532999+0000) 2022-01-31T20:51:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:05 smithi167 conmon[49112]: debug 2022-01-31T20:51:05.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:05.626074+0000) 2022-01-31T20:51:06.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:06 smithi167 conmon[60316]: debug 2022-01-31T20:51:06.048+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:06.049062+0000) 2022-01-31T20:51:06.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:06 smithi167 conmon[54076]: debug 2022-01-31T20:51:06.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:06.297774+0000) 2022-01-31T20:51:06.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:06 smithi171 conmon[51620]: debug 2022-01-31T20:51:06.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:06.404448+0000) 2022-01-31T20:51:06.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:06 smithi171 conmon[46715]: debug 2022-01-31T20:51:06.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:06.327468+0000) 2022-01-31T20:51:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:06 smithi171 conmon[41853]: debug 2022-01-31T20:51:06.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:06.533134+0000) 2022-01-31T20:51:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:06 smithi167 conmon[49112]: debug 2022-01-31T20:51:06.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:06.626257+0000) 2022-01-31T20:51:07.339 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:07 smithi167 conmon[54076]: debug 2022-01-31T20:51:07.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:07.298008+0000) 2022-01-31T20:51:07.340 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:07 smithi167 conmon[60316]: debug 2022-01-31T20:51:07.048+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:07.049246+0000) 2022-01-31T20:51:07.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:07 smithi171 conmon[46715]: debug 2022-01-31T20:51:07.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:07.327615+0000) 2022-01-31T20:51:07.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:07 smithi171 conmon[51620]: debug 2022-01-31T20:51:07.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:07.404655+0000) 2022-01-31T20:51:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:07 smithi171 conmon[41853]: debug 2022-01-31T20:51:07.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:07.533312+0000) 2022-01-31T20:51:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:07 smithi167 conmon[49112]: debug 2022-01-31T20:51:07.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:07.626431+0000) 2022-01-31T20:51:08.340 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:08 smithi167 conmon[60316]: debug 2022-01-31T20:51:08.048+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.049431+0000) 2022-01-31T20:51:08.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:08 smithi167 conmon[54076]: debug 2022-01-31T20:51:08.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.298190+0000) 2022-01-31T20:51:08.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[46715]: debug 2022-01-31T20:51:08.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.327797+0000) 2022-01-31T20:51:08.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[51620]: debug 2022-01-31T20:51:08.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.404802+0000) 2022-01-31T20:51:08.818 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[35325]: debug 2022-01-31T20:51:08.581+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:51:08.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[41853]: debug 2022-01-31T20:51:08.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.533478+0000) 2022-01-31T20:51:08.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[41853]: debug 2022-01-31T20:51:08.818+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.819830+0000) 2022-01-31T20:51:08.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:08 smithi167 conmon[54076]: debug 2022-01-31T20:51:08.821+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.821996+0000) 2022-01-31T20:51:08.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:08 smithi167 conmon[60316]: debug 2022-01-31T20:51:08.819+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.820412+0000) 2022-01-31T20:51:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:08 smithi167 conmon[49112]: debug 2022-01-31T20:51:08.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.626564+0000) 2022-01-31T20:51:08.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:08 smithi167 conmon[49112]: debug 2022-01-31T20:51:08.821+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.821702+0000) 2022-01-31T20:51:09.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[35325]: debug 2022-01-31T20:51:08.827+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83362 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:09.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[46715]: debug 2022-01-31T20:51:08.820+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.821837+0000) 2022-01-31T20:51:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:08 smithi171 conmon[51620]: debug 2022-01-31T20:51:08.818+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:08.820096+0000) 2022-01-31T20:51:09.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:09 smithi167 conmon[54076]: debug 2022-01-31T20:51:09.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:09.298400+0000) 2022-01-31T20:51:09.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:09 smithi167 conmon[60316]: debug 2022-01-31T20:51:09.049+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:09.049597+0000) 2022-01-31T20:51:09.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:09 smithi171 conmon[51620]: debug 2022-01-31T20:51:09.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:09.404952+0000) 2022-01-31T20:51:09.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:09 smithi171 conmon[46715]: debug 2022-01-31T20:51:09.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:09.327960+0000) 2022-01-31T20:51:09.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:09 smithi171 conmon[41853]: debug 2022-01-31T20:51:09.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:09.533635+0000) 2022-01-31T20:51:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:09 smithi167 conmon[49112]: debug 2022-01-31T20:51:09.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:09.626673+0000) 2022-01-31T20:51:10.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:10 smithi167 conmon[54076]: debug 2022-01-31T20:51:10.298+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:10.298551+0000) 2022-01-31T20:51:10.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:10 smithi167 conmon[60316]: debug 2022-01-31T20:51:10.049+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:10.049733+0000) 2022-01-31T20:51:10.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:10 smithi171 conmon[46715]: debug 2022-01-31T20:51:10.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:10.328109+0000) 2022-01-31T20:51:10.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:10 smithi171 conmon[51620]: debug 2022-01-31T20:51:10.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:10.405099+0000) 2022-01-31T20:51:10.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:10 smithi171 conmon[41853]: debug 2022-01-31T20:51:10.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:10.533790+0000) 2022-01-31T20:51:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:10 smithi167 conmon[49112]: debug 2022-01-31T20:51:10.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:10.626784+0000) 2022-01-31T20:51:11.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:11 smithi167 conmon[54076]: debug 2022-01-31T20:51:11.298+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:11.298705+0000) 2022-01-31T20:51:11.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:11 smithi167 conmon[60316]: debug 2022-01-31T20:51:11.049+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:11.049911+0000) 2022-01-31T20:51:11.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:11 smithi171 conmon[46715]: debug 2022-01-31T20:51:11.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:11.328264+0000) 2022-01-31T20:51:11.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:11 smithi171 conmon[51620]: debug 2022-01-31T20:51:11.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:11.405277+0000) 2022-01-31T20:51:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:11 smithi171 conmon[41853]: debug 2022-01-31T20:51:11.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:11.533973+0000) 2022-01-31T20:51:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:11 smithi167 conmon[49112]: debug 2022-01-31T20:51:11.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:11.626976+0000) 2022-01-31T20:51:12.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:12 smithi167 conmon[54076]: debug 2022-01-31T20:51:12.298+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:12.298868+0000) 2022-01-31T20:51:12.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:12 smithi167 conmon[60316]: debug 2022-01-31T20:51:12.049+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:12.050066+0000) 2022-01-31T20:51:12.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:12 smithi171 conmon[46715]: debug 2022-01-31T20:51:12.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:12.328447+0000) 2022-01-31T20:51:12.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:12 smithi171 conmon[51620]: debug 2022-01-31T20:51:12.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:12.405433+0000) 2022-01-31T20:51:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:12 smithi171 conmon[41853]: debug 2022-01-31T20:51:12.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:12.534174+0000) 2022-01-31T20:51:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:12 smithi167 conmon[49112]: debug 2022-01-31T20:51:12.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:12.627179+0000) 2022-01-31T20:51:13.341 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:13 smithi167 conmon[54076]: debug 2022-01-31T20:51:13.298+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.299053+0000) 2022-01-31T20:51:13.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:13 smithi167 conmon[60316]: debug 2022-01-31T20:51:13.049+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.050235+0000) 2022-01-31T20:51:13.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[46715]: debug 2022-01-31T20:51:13.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.328654+0000) 2022-01-31T20:51:13.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[51620]: debug 2022-01-31T20:51:13.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.405559+0000) 2022-01-31T20:51:13.830 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[41853]: debug 2022-01-31T20:51:13.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.534374+0000) 2022-01-31T20:51:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:13 smithi167 conmon[49112]: debug 2022-01-31T20:51:13.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.627418+0000) 2022-01-31T20:51:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:13 smithi167 conmon[49112]: debug 2022-01-31T20:51:13.831+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.831870+0000) 2022-01-31T20:51:13.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:13 smithi167 conmon[54076]: debug 2022-01-31T20:51:13.832+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.832774+0000) 2022-01-31T20:51:13.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:13 smithi167 conmon[60316]: debug 2022-01-31T20:51:13.830+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.831380+0000) 2022-01-31T20:51:14.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[35325]: debug 2022-01-31T20:51:13.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83473 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[41853]: debug 2022-01-31T20:51:13.829+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.831728+0000) 2022-01-31T20:51:14.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[46715]: debug 2022-01-31T20:51:13.831+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.833140+0000) 2022-01-31T20:51:14.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:13 smithi171 conmon[51620]: debug 2022-01-31T20:51:13.830+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:13.832267+0000) 2022-01-31T20:51:14.341 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:14 smithi167 conmon[54076]: debug 2022-01-31T20:51:14.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:14.299254+0000) 2022-01-31T20:51:14.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:14 smithi167 conmon[60316]: debug 2022-01-31T20:51:14.049+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:14.050434+0000) 2022-01-31T20:51:14.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:14 smithi171 conmon[46715]: debug 2022-01-31T20:51:14.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:14.328833+0000) 2022-01-31T20:51:14.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:14 smithi171 conmon[51620]: debug 2022-01-31T20:51:14.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:14.405714+0000) 2022-01-31T20:51:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:14 smithi171 conmon[41853]: debug 2022-01-31T20:51:14.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:14.534530+0000) 2022-01-31T20:51:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:14 smithi167 conmon[49112]: debug 2022-01-31T20:51:14.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:14.627588+0000) 2022-01-31T20:51:15.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:15 smithi167 conmon[60316]: debug 2022-01-31T20:51:15.050+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:15.050589+0000) 2022-01-31T20:51:15.341 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:15 smithi167 conmon[54076]: debug 2022-01-31T20:51:15.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:15.299438+0000) 2022-01-31T20:51:15.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:15 smithi171 conmon[46715]: debug 2022-01-31T20:51:15.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:15.328950+0000) 2022-01-31T20:51:15.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:15 smithi171 conmon[51620]: debug 2022-01-31T20:51:15.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:15.405863+0000) 2022-01-31T20:51:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:15 smithi171 conmon[41853]: debug 2022-01-31T20:51:15.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:15.534622+0000) 2022-01-31T20:51:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:15 smithi167 conmon[49112]: debug 2022-01-31T20:51:15.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:15.627702+0000) 2022-01-31T20:51:16.341 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:16 smithi167 conmon[54076]: debug 2022-01-31T20:51:16.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:16.299621+0000) 2022-01-31T20:51:16.342 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:16 smithi167 conmon[60316]: debug 2022-01-31T20:51:16.050+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:16.050744+0000) 2022-01-31T20:51:16.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:16 smithi171 conmon[46715]: debug 2022-01-31T20:51:16.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:16.329106+0000) 2022-01-31T20:51:16.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:16 smithi171 conmon[51620]: debug 2022-01-31T20:51:16.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:16.406030+0000) 2022-01-31T20:51:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:16 smithi171 conmon[41853]: debug 2022-01-31T20:51:16.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:16.534792+0000) 2022-01-31T20:51:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:16 smithi167 conmon[49112]: debug 2022-01-31T20:51:16.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:16.627826+0000) 2022-01-31T20:51:17.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:17 smithi167 conmon[60316]: debug 2022-01-31T20:51:17.050+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:17.050923+0000) 2022-01-31T20:51:17.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:17 smithi167 conmon[54076]: debug 2022-01-31T20:51:17.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:17.299780+0000) 2022-01-31T20:51:17.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:17 smithi171 conmon[46715]: debug 2022-01-31T20:51:17.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:17.329264+0000) 2022-01-31T20:51:17.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:17 smithi171 conmon[51620]: debug 2022-01-31T20:51:17.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:17.406247+0000) 2022-01-31T20:51:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:17 smithi171 conmon[41853]: debug 2022-01-31T20:51:17.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:17.534977+0000) 2022-01-31T20:51:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:17 smithi167 conmon[49112]: debug 2022-01-31T20:51:17.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:17.628015+0000) 2022-01-31T20:51:18.341 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:18 smithi167 conmon[54076]: debug 2022-01-31T20:51:18.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.299961+0000) 2022-01-31T20:51:18.342 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:18 smithi167 conmon[60316]: debug 2022-01-31T20:51:18.050+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.051075+0000) 2022-01-31T20:51:18.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[46715]: debug 2022-01-31T20:51:18.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.329379+0000) 2022-01-31T20:51:18.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[51620]: debug 2022-01-31T20:51:18.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.406423+0000) 2022-01-31T20:51:18.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[41853]: debug 2022-01-31T20:51:18.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.535129+0000) 2022-01-31T20:51:18.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:18 smithi167 conmon[54076]: debug 2022-01-31T20:51:18.843+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.843679+0000) 2022-01-31T20:51:18.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:18 smithi167 conmon[60316]: debug 2022-01-31T20:51:18.843+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.844403+0000) 2022-01-31T20:51:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:18 smithi167 conmon[49112]: debug 2022-01-31T20:51:18.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.628214+0000) 2022-01-31T20:51:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:18 smithi167 conmon[49112]: debug 2022-01-31T20:51:18.841+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.842224+0000) 2022-01-31T20:51:19.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[35325]: debug 2022-01-31T20:51:18.849+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83581 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[41853]: debug 2022-01-31T20:51:18.841+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.843070+0000) 2022-01-31T20:51:19.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[46715]: debug 2022-01-31T20:51:18.842+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.844101+0000) 2022-01-31T20:51:19.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:18 smithi171 conmon[51620]: debug 2022-01-31T20:51:18.842+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:18.843851+0000) 2022-01-31T20:51:19.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:19 smithi167 conmon[54076]: debug 2022-01-31T20:51:19.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:19.300148+0000) 2022-01-31T20:51:19.342 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:19 smithi167 conmon[60316]: debug 2022-01-31T20:51:19.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:19.051264+0000) 2022-01-31T20:51:19.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:19 smithi171 conmon[46715]: debug 2022-01-31T20:51:19.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:19.329564+0000) 2022-01-31T20:51:19.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:19 smithi171 conmon[51620]: debug 2022-01-31T20:51:19.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:19.406631+0000) 2022-01-31T20:51:19.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:19 smithi171 conmon[41853]: debug 2022-01-31T20:51:19.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:19.535331+0000) 2022-01-31T20:51:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:19 smithi167 conmon[49112]: debug 2022-01-31T20:51:19.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:19.628399+0000) 2022-01-31T20:51:20.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:20 smithi167 conmon[54076]: debug 2022-01-31T20:51:20.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:20.300297+0000) 2022-01-31T20:51:20.342 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:20 smithi167 conmon[60316]: debug 2022-01-31T20:51:20.050+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:20.051461+0000) 2022-01-31T20:51:20.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:20 smithi171 conmon[51620]: debug 2022-01-31T20:51:20.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:20.406813+0000) 2022-01-31T20:51:20.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:20 smithi171 conmon[46715]: debug 2022-01-31T20:51:20.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:20.329745+0000) 2022-01-31T20:51:20.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:20 smithi171 conmon[41853]: debug 2022-01-31T20:51:20.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:20.535481+0000) 2022-01-31T20:51:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:20 smithi167 conmon[49112]: debug 2022-01-31T20:51:20.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:20.628554+0000) 2022-01-31T20:51:21.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:21 smithi167 conmon[54076]: debug 2022-01-31T20:51:21.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:21.300486+0000) 2022-01-31T20:51:21.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:21 smithi167 conmon[60316]: debug 2022-01-31T20:51:21.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:21.051659+0000) 2022-01-31T20:51:21.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:21 smithi171 conmon[51620]: debug 2022-01-31T20:51:21.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:21.406993+0000) 2022-01-31T20:51:21.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:21 smithi171 conmon[46715]: debug 2022-01-31T20:51:21.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:21.329918+0000) 2022-01-31T20:51:21.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:21 smithi171 conmon[41853]: debug 2022-01-31T20:51:21.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:21.535659+0000) 2022-01-31T20:51:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:21 smithi167 conmon[49112]: debug 2022-01-31T20:51:21.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:21.628685+0000) 2022-01-31T20:51:22.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:22 smithi167 conmon[54076]: debug 2022-01-31T20:51:22.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:22.300699+0000) 2022-01-31T20:51:22.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:22 smithi167 conmon[60316]: debug 2022-01-31T20:51:22.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:22.051850+0000) 2022-01-31T20:51:22.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:22 smithi171 conmon[46715]: debug 2022-01-31T20:51:22.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:22.330112+0000) 2022-01-31T20:51:22.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:22 smithi171 conmon[51620]: debug 2022-01-31T20:51:22.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:22.407158+0000) 2022-01-31T20:51:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:22 smithi171 conmon[41853]: debug 2022-01-31T20:51:22.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:22.535839+0000) 2022-01-31T20:51:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:22 smithi167 conmon[49112]: debug 2022-01-31T20:51:22.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:22.628817+0000) 2022-01-31T20:51:23.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:23 smithi167 conmon[54076]: debug 2022-01-31T20:51:23.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.300875+0000) 2022-01-31T20:51:23.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:23 smithi167 conmon[60316]: debug 2022-01-31T20:51:23.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.052011+0000) 2022-01-31T20:51:23.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[46715]: debug 2022-01-31T20:51:23.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.330300+0000) 2022-01-31T20:51:23.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[51620]: debug 2022-01-31T20:51:23.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.407313+0000) 2022-01-31T20:51:23.851 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[35325]: debug 2022-01-31T20:51:23.582+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:51:23.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[41853]: debug 2022-01-31T20:51:23.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.535992+0000) 2022-01-31T20:51:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:23 smithi167 conmon[49112]: debug 2022-01-31T20:51:23.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.629005+0000) 2022-01-31T20:51:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:23 smithi167 conmon[49112]: debug 2022-01-31T20:51:23.853+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.853608+0000) 2022-01-31T20:51:23.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:23 smithi167 conmon[54076]: debug 2022-01-31T20:51:23.854+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.854899+0000) 2022-01-31T20:51:23.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:23 smithi167 conmon[60316]: debug 2022-01-31T20:51:23.854+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.855079+0000) 2022-01-31T20:51:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[41853]: debug 2022-01-31T20:51:23.852+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.853196+0000) 2022-01-31T20:51:24.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[46715]: debug 2022-01-31T20:51:23.853+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.854341+0000) 2022-01-31T20:51:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[51620]: debug 2022-01-31T20:51:23.852+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:23.852970+0000) 2022-01-31T20:51:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:23 smithi171 conmon[35325]: debug 2022-01-31T20:51:23.877+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83692 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:24.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:24 smithi167 conmon[60316]: debug 2022-01-31T20:51:24.052+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:24.052216+0000) 2022-01-31T20:51:24.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:24 smithi167 conmon[54076]: debug 2022-01-31T20:51:24.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:24.301084+0000) 2022-01-31T20:51:24.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:24 smithi171 conmon[46715]: debug 2022-01-31T20:51:24.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:24.330459+0000) 2022-01-31T20:51:24.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:24 smithi171 conmon[51620]: debug 2022-01-31T20:51:24.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:24.407501+0000) 2022-01-31T20:51:24.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:24 smithi171 conmon[41853]: debug 2022-01-31T20:51:24.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:24.536154+0000) 2022-01-31T20:51:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:24 smithi167 conmon[49112]: debug 2022-01-31T20:51:24.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:24.629179+0000) 2022-01-31T20:51:25.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:25 smithi167 conmon[54076]: debug 2022-01-31T20:51:25.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:25.301260+0000) 2022-01-31T20:51:25.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:25 smithi167 conmon[60316]: debug 2022-01-31T20:51:25.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:25.052326+0000) 2022-01-31T20:51:25.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:25 smithi171 conmon[46715]: debug 2022-01-31T20:51:25.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:25.330554+0000) 2022-01-31T20:51:25.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:25 smithi171 conmon[51620]: debug 2022-01-31T20:51:25.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:25.407653+0000) 2022-01-31T20:51:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:25 smithi171 conmon[41853]: debug 2022-01-31T20:51:25.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:25.536311+0000) 2022-01-31T20:51:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:25 smithi167 conmon[49112]: debug 2022-01-31T20:51:25.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:25.629331+0000) 2022-01-31T20:51:26.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:26 smithi167 conmon[54076]: debug 2022-01-31T20:51:26.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:26.301448+0000) 2022-01-31T20:51:26.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:26 smithi167 conmon[60316]: debug 2022-01-31T20:51:26.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:26.052542+0000) 2022-01-31T20:51:26.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:26 smithi171 conmon[46715]: debug 2022-01-31T20:51:26.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:26.330709+0000) 2022-01-31T20:51:26.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:26 smithi171 conmon[51620]: debug 2022-01-31T20:51:26.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:26.407837+0000) 2022-01-31T20:51:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:26 smithi171 conmon[41853]: debug 2022-01-31T20:51:26.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:26.536496+0000) 2022-01-31T20:51:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:26 smithi167 conmon[49112]: debug 2022-01-31T20:51:26.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:26.629518+0000) 2022-01-31T20:51:27.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:27 smithi167 conmon[54076]: debug 2022-01-31T20:51:27.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:27.301602+0000) 2022-01-31T20:51:27.344 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:27 smithi167 conmon[60316]: debug 2022-01-31T20:51:27.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:27.052764+0000) 2022-01-31T20:51:27.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:27 smithi171 conmon[46715]: debug 2022-01-31T20:51:27.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:27.330912+0000) 2022-01-31T20:51:27.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:27 smithi171 conmon[51620]: debug 2022-01-31T20:51:27.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:27.408021+0000) 2022-01-31T20:51:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:27 smithi171 conmon[41853]: debug 2022-01-31T20:51:27.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:27.536700+0000) 2022-01-31T20:51:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:27 smithi167 conmon[49112]: debug 2022-01-31T20:51:27.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:27.629650+0000) 2022-01-31T20:51:28.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:28 smithi167 conmon[54076]: debug 2022-01-31T20:51:28.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.301816+0000) 2022-01-31T20:51:28.344 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:28 smithi167 conmon[60316]: debug 2022-01-31T20:51:28.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.052941+0000) 2022-01-31T20:51:28.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[46715]: debug 2022-01-31T20:51:28.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.331134+0000) 2022-01-31T20:51:28.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[51620]: debug 2022-01-31T20:51:28.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.408224+0000) 2022-01-31T20:51:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[41853]: debug 2022-01-31T20:51:28.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.536894+0000) 2022-01-31T20:51:28.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:28 smithi167 conmon[49112]: debug 2022-01-31T20:51:28.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.629770+0000) 2022-01-31T20:51:29.167 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[35325]: debug 2022-01-31T20:51:28.889+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83800 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:29.167 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[41853]: debug 2022-01-31T20:51:28.879+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.880650+0000) 2022-01-31T20:51:29.168 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[46715]: debug 2022-01-31T20:51:28.880+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.881283+0000) 2022-01-31T20:51:29.168 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:28 smithi171 conmon[51620]: debug 2022-01-31T20:51:28.879+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.880195+0000) 2022-01-31T20:51:29.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:28 smithi167 conmon[49112]: debug 2022-01-31T20:51:28.879+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.880928+0000) 2022-01-31T20:51:29.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:28 smithi167 conmon[54076]: debug 2022-01-31T20:51:28.879+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.881151+0000) 2022-01-31T20:51:29.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:28 smithi167 conmon[60316]: debug 2022-01-31T20:51:28.880+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:28.881533+0000) 2022-01-31T20:51:29.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:29 smithi167 conmon[60316]: debug 2022-01-31T20:51:29.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:29.053105+0000) 2022-01-31T20:51:29.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:29 smithi171 conmon[46715]: debug 2022-01-31T20:51:29.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:29.331359+0000) 2022-01-31T20:51:29.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:29 smithi171 conmon[51620]: debug 2022-01-31T20:51:29.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:29.408407+0000) 2022-01-31T20:51:29.620 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:29 smithi167 conmon[54076]: debug 2022-01-31T20:51:29.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:29.301947+0000) 2022-01-31T20:51:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:29 smithi171 conmon[41853]: debug 2022-01-31T20:51:29.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:29.537099+0000) 2022-01-31T20:51:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:29 smithi167 conmon[49112]: debug 2022-01-31T20:51:29.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:29.629954+0000) 2022-01-31T20:51:30.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:30 smithi167 conmon[54076]: debug 2022-01-31T20:51:30.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:30.302091+0000) 2022-01-31T20:51:30.344 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:30 smithi167 conmon[60316]: debug 2022-01-31T20:51:30.052+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:30.053270+0000) 2022-01-31T20:51:30.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:30 smithi171 conmon[46715]: debug 2022-01-31T20:51:30.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:30.331535+0000) 2022-01-31T20:51:30.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:30 smithi171 conmon[51620]: debug 2022-01-31T20:51:30.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:30.408561+0000) 2022-01-31T20:51:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:30 smithi171 conmon[41853]: debug 2022-01-31T20:51:30.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:30.537275+0000) 2022-01-31T20:51:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:30 smithi167 conmon[49112]: debug 2022-01-31T20:51:30.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:30.630132+0000) 2022-01-31T20:51:31.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:31 smithi167 conmon[54076]: debug 2022-01-31T20:51:31.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:31.302274+0000) 2022-01-31T20:51:31.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:31 smithi167 conmon[60316]: debug 2022-01-31T20:51:31.051+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:31.053429+0000) 2022-01-31T20:51:31.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:31 smithi171 conmon[46715]: debug 2022-01-31T20:51:31.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:31.331693+0000) 2022-01-31T20:51:31.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:31 smithi171 conmon[51620]: debug 2022-01-31T20:51:31.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:31.408768+0000) 2022-01-31T20:51:31.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:31 smithi171 conmon[41853]: debug 2022-01-31T20:51:31.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:31.537456+0000) 2022-01-31T20:51:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:31 smithi167 conmon[49112]: debug 2022-01-31T20:51:31.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:31.630337+0000) 2022-01-31T20:51:32.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:32 smithi167 conmon[54076]: debug 2022-01-31T20:51:32.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:32.302435+0000) 2022-01-31T20:51:32.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:32 smithi167 conmon[60316]: debug 2022-01-31T20:51:32.052+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:32.053641+0000) 2022-01-31T20:51:32.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:32 smithi171 conmon[46715]: debug 2022-01-31T20:51:32.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:32.331829+0000) 2022-01-31T20:51:32.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:32 smithi171 conmon[51620]: debug 2022-01-31T20:51:32.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:32.408915+0000) 2022-01-31T20:51:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:32 smithi171 conmon[41853]: debug 2022-01-31T20:51:32.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:32.537650+0000) 2022-01-31T20:51:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:32 smithi167 conmon[49112]: debug 2022-01-31T20:51:32.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:32.630546+0000) 2022-01-31T20:51:33.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:33 smithi167 conmon[60316]: debug 2022-01-31T20:51:33.052+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.053825+0000) 2022-01-31T20:51:33.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:33 smithi167 conmon[54076]: debug 2022-01-31T20:51:33.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.302593+0000) 2022-01-31T20:51:33.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[46715]: debug 2022-01-31T20:51:33.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.331979+0000) 2022-01-31T20:51:33.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[51620]: debug 2022-01-31T20:51:33.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.409069+0000) 2022-01-31T20:51:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[41853]: debug 2022-01-31T20:51:33.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.537835+0000) 2022-01-31T20:51:33.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:33 smithi167 conmon[49112]: debug 2022-01-31T20:51:33.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.630678+0000) 2022-01-31T20:51:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:33 smithi167 conmon[49112]: debug 2022-01-31T20:51:33.891+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.893402+0000) 2022-01-31T20:51:34.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:33 smithi167 conmon[54076]: debug 2022-01-31T20:51:33.891+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.893413+0000) 2022-01-31T20:51:34.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:33 smithi167 conmon[60316]: debug 2022-01-31T20:51:33.891+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.892790+0000) 2022-01-31T20:51:34.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:34 smithi167 conmon[60316]: debug 2022-01-31T20:51:34.052+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:34.053978+0000) 2022-01-31T20:51:34.169 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[35325]: debug 2022-01-31T20:51:33.901+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 83912 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:34.170 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[41853]: debug 2022-01-31T20:51:33.891+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.892131+0000) 2022-01-31T20:51:34.170 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[46715]: debug 2022-01-31T20:51:33.892+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.893572+0000) 2022-01-31T20:51:34.171 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:33 smithi171 conmon[51620]: debug 2022-01-31T20:51:33.891+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:33.892573+0000) 2022-01-31T20:51:34.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:34 smithi171 conmon[46715]: debug 2022-01-31T20:51:34.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:34.332184+0000) 2022-01-31T20:51:34.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:34 smithi171 conmon[51620]: debug 2022-01-31T20:51:34.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:34.409195+0000) 2022-01-31T20:51:34.621 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:34 smithi167 conmon[54076]: debug 2022-01-31T20:51:34.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:34.302783+0000) 2022-01-31T20:51:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:34 smithi171 conmon[41853]: debug 2022-01-31T20:51:34.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:34.537990+0000) 2022-01-31T20:51:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:34 smithi167 conmon[49112]: debug 2022-01-31T20:51:34.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:34.630829+0000) 2022-01-31T20:51:35.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:35 smithi167 conmon[54076]: debug 2022-01-31T20:51:35.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:35.302969+0000) 2022-01-31T20:51:35.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:35 smithi167 conmon[60316]: debug 2022-01-31T20:51:35.053+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:35.054140+0000) 2022-01-31T20:51:35.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:35 smithi171 conmon[46715]: debug 2022-01-31T20:51:35.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:35.332319+0000) 2022-01-31T20:51:35.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:35 smithi171 conmon[51620]: debug 2022-01-31T20:51:35.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:35.409382+0000) 2022-01-31T20:51:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:35 smithi171 conmon[41853]: debug 2022-01-31T20:51:35.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:35.538113+0000) 2022-01-31T20:51:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:35 smithi167 conmon[49112]: debug 2022-01-31T20:51:35.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:35.630979+0000) 2022-01-31T20:51:36.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:36 smithi167 conmon[60316]: debug 2022-01-31T20:51:36.053+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:36.054347+0000) 2022-01-31T20:51:36.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:36 smithi167 conmon[54076]: debug 2022-01-31T20:51:36.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:36.303157+0000) 2022-01-31T20:51:36.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:36 smithi171 conmon[46715]: debug 2022-01-31T20:51:36.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:36.332482+0000) 2022-01-31T20:51:36.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:36 smithi171 conmon[51620]: debug 2022-01-31T20:51:36.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:36.409586+0000) 2022-01-31T20:51:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:36 smithi171 conmon[41853]: debug 2022-01-31T20:51:36.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:36.538264+0000) 2022-01-31T20:51:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:36 smithi167 conmon[49112]: debug 2022-01-31T20:51:36.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:36.631149+0000) 2022-01-31T20:51:37.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:37 smithi167 conmon[60316]: debug 2022-01-31T20:51:37.053+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:37.054535+0000) 2022-01-31T20:51:37.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:37 smithi167 conmon[54076]: debug 2022-01-31T20:51:37.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:37.303358+0000) 2022-01-31T20:51:37.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:37 smithi171 conmon[46715]: debug 2022-01-31T20:51:37.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:37.332659+0000) 2022-01-31T20:51:37.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:37 smithi171 conmon[51620]: debug 2022-01-31T20:51:37.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:37.409800+0000) 2022-01-31T20:51:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:37 smithi171 conmon[41853]: debug 2022-01-31T20:51:37.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:37.538395+0000) 2022-01-31T20:51:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:37 smithi167 conmon[49112]: debug 2022-01-31T20:51:37.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:37.631332+0000) 2022-01-31T20:51:38.345 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:38 smithi167 conmon[60316]: debug 2022-01-31T20:51:38.053+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.054696+0000) 2022-01-31T20:51:38.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:38 smithi167 conmon[54076]: debug 2022-01-31T20:51:38.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.303530+0000) 2022-01-31T20:51:38.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[46715]: debug 2022-01-31T20:51:38.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.332797+0000) 2022-01-31T20:51:38.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[51620]: debug 2022-01-31T20:51:38.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.410000+0000) 2022-01-31T20:51:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[41853]: debug 2022-01-31T20:51:38.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.538541+0000) 2022-01-31T20:51:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[35325]: debug 2022-01-31T20:51:38.583+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:51:38.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:38 smithi167 conmon[49112]: debug 2022-01-31T20:51:38.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.631531+0000) 2022-01-31T20:51:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:38 smithi167 conmon[49112]: debug 2022-01-31T20:51:38.902+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.904237+0000) 2022-01-31T20:51:39.248 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:38 smithi167 conmon[54076]: debug 2022-01-31T20:51:38.903+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.905268+0000) 2022-01-31T20:51:39.248 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:38 smithi167 conmon[60316]: debug 2022-01-31T20:51:38.903+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.904712+0000) 2022-01-31T20:51:39.249 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:39 smithi167 conmon[60316]: debug 2022-01-31T20:51:39.054+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:39.054859+0000) 2022-01-31T20:51:39.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[35325]: debug 2022-01-31T20:51:38.912+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84017 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:39.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[41853]: debug 2022-01-31T20:51:38.903+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.904059+0000) 2022-01-31T20:51:39.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[46715]: debug 2022-01-31T20:51:38.904+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.905691+0000) 2022-01-31T20:51:39.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:38 smithi171 conmon[51620]: debug 2022-01-31T20:51:38.904+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:38.905146+0000) 2022-01-31T20:51:39.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:39 smithi171 conmon[46715]: debug 2022-01-31T20:51:39.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:39.332960+0000) 2022-01-31T20:51:39.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:39 smithi171 conmon[51620]: debug 2022-01-31T20:51:39.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:39.410188+0000) 2022-01-31T20:51:39.622 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:39 smithi167 conmon[54076]: debug 2022-01-31T20:51:39.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:39.303632+0000) 2022-01-31T20:51:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:39 smithi171 conmon[41853]: debug 2022-01-31T20:51:39.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:39.538709+0000) 2022-01-31T20:51:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:39 smithi167 conmon[49112]: debug 2022-01-31T20:51:39.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:39.631682+0000) 2022-01-31T20:51:40.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:40 smithi167 conmon[54076]: debug 2022-01-31T20:51:40.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:40.303710+0000) 2022-01-31T20:51:40.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:40 smithi167 conmon[60316]: debug 2022-01-31T20:51:40.053+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:40.055052+0000) 2022-01-31T20:51:40.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:40 smithi171 conmon[46715]: debug 2022-01-31T20:51:40.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:40.333104+0000) 2022-01-31T20:51:40.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:40 smithi171 conmon[51620]: debug 2022-01-31T20:51:40.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:40.410349+0000) 2022-01-31T20:51:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:40 smithi171 conmon[41853]: debug 2022-01-31T20:51:40.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:40.538855+0000) 2022-01-31T20:51:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:40 smithi167 conmon[49112]: debug 2022-01-31T20:51:40.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:40.631797+0000) 2022-01-31T20:51:41.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:41 smithi167 conmon[54076]: debug 2022-01-31T20:51:41.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:41.303908+0000) 2022-01-31T20:51:41.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:41 smithi167 conmon[60316]: debug 2022-01-31T20:51:41.054+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:41.055212+0000) 2022-01-31T20:51:41.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:41 smithi171 conmon[46715]: debug 2022-01-31T20:51:41.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:41.333302+0000) 2022-01-31T20:51:41.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:41 smithi171 conmon[51620]: debug 2022-01-31T20:51:41.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:41.410508+0000) 2022-01-31T20:51:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:41 smithi171 conmon[41853]: debug 2022-01-31T20:51:41.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:41.539061+0000) 2022-01-31T20:51:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:41 smithi167 conmon[49112]: debug 2022-01-31T20:51:41.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:41.632028+0000) 2022-01-31T20:51:42.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:42 smithi167 conmon[54076]: debug 2022-01-31T20:51:42.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:42.304055+0000) 2022-01-31T20:51:42.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:42 smithi167 conmon[60316]: debug 2022-01-31T20:51:42.054+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:42.055422+0000) 2022-01-31T20:51:42.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:42 smithi171 conmon[46715]: debug 2022-01-31T20:51:42.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:42.333504+0000) 2022-01-31T20:51:42.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:42 smithi171 conmon[51620]: debug 2022-01-31T20:51:42.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:42.410672+0000) 2022-01-31T20:51:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:42 smithi171 conmon[41853]: debug 2022-01-31T20:51:42.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:42.539221+0000) 2022-01-31T20:51:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:42 smithi167 conmon[49112]: debug 2022-01-31T20:51:42.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:42.632156+0000) 2022-01-31T20:51:43.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:43 smithi167 conmon[54076]: debug 2022-01-31T20:51:43.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.304235+0000) 2022-01-31T20:51:43.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:43 smithi167 conmon[60316]: debug 2022-01-31T20:51:43.054+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.055607+0000) 2022-01-31T20:51:43.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[46715]: debug 2022-01-31T20:51:43.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.333696+0000) 2022-01-31T20:51:43.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[51620]: debug 2022-01-31T20:51:43.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.410835+0000) 2022-01-31T20:51:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[41853]: debug 2022-01-31T20:51:43.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.539382+0000) 2022-01-31T20:51:43.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:43 smithi167 conmon[49112]: debug 2022-01-31T20:51:43.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.632327+0000) 2022-01-31T20:51:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:43 smithi167 conmon[49112]: debug 2022-01-31T20:51:43.913+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.915380+0000) 2022-01-31T20:51:44.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:43 smithi167 conmon[54076]: debug 2022-01-31T20:51:43.914+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.915977+0000) 2022-01-31T20:51:44.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:43 smithi167 conmon[60316]: debug 2022-01-31T20:51:43.914+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.916210+0000) 2022-01-31T20:51:44.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:44 smithi167 conmon[60316]: debug 2022-01-31T20:51:44.055+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:44.055795+0000) 2022-01-31T20:51:44.170 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[35325]: debug 2022-01-31T20:51:43.923+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84128 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:44.170 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[46715]: debug 2022-01-31T20:51:43.915+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.916132+0000) 2022-01-31T20:51:44.171 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[51620]: debug 2022-01-31T20:51:43.915+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.916512+0000) 2022-01-31T20:51:44.171 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:43 smithi171 conmon[41853]: debug 2022-01-31T20:51:43.915+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:43.915889+0000) 2022-01-31T20:51:44.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:44 smithi171 conmon[46715]: debug 2022-01-31T20:51:44.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:44.333895+0000) 2022-01-31T20:51:44.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:44 smithi171 conmon[51620]: debug 2022-01-31T20:51:44.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:44.411024+0000) 2022-01-31T20:51:44.622 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:44 smithi167 conmon[54076]: debug 2022-01-31T20:51:44.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:44.304378+0000) 2022-01-31T20:51:44.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:44 smithi171 conmon[41853]: debug 2022-01-31T20:51:44.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:44.539561+0000) 2022-01-31T20:51:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:44 smithi167 conmon[49112]: debug 2022-01-31T20:51:44.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:44.632522+0000) 2022-01-31T20:51:45.347 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:45 smithi167 conmon[54076]: debug 2022-01-31T20:51:45.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:45.304485+0000) 2022-01-31T20:51:45.348 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:45 smithi167 conmon[60316]: debug 2022-01-31T20:51:45.055+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:45.055911+0000) 2022-01-31T20:51:45.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:45 smithi171 conmon[46715]: debug 2022-01-31T20:51:45.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:45.334046+0000) 2022-01-31T20:51:45.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:45 smithi171 conmon[51620]: debug 2022-01-31T20:51:45.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:45.411168+0000) 2022-01-31T20:51:45.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:45 smithi171 conmon[41853]: debug 2022-01-31T20:51:45.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:45.539691+0000) 2022-01-31T20:51:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:45 smithi167 conmon[49112]: debug 2022-01-31T20:51:45.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:45.632638+0000) 2022-01-31T20:51:46.347 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:46 smithi167 conmon[54076]: debug 2022-01-31T20:51:46.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:46.304683+0000) 2022-01-31T20:51:46.348 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:46 smithi167 conmon[60316]: debug 2022-01-31T20:51:46.055+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:46.056123+0000) 2022-01-31T20:51:46.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:46 smithi171 conmon[46715]: debug 2022-01-31T20:51:46.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:46.334220+0000) 2022-01-31T20:51:46.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:46 smithi171 conmon[51620]: debug 2022-01-31T20:51:46.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:46.411349+0000) 2022-01-31T20:51:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:46 smithi171 conmon[41853]: debug 2022-01-31T20:51:46.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:46.539851+0000) 2022-01-31T20:51:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:46 smithi167 conmon[49112]: debug 2022-01-31T20:51:46.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:46.632788+0000) 2022-01-31T20:51:47.347 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:47 smithi167 conmon[54076]: debug 2022-01-31T20:51:47.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:47.304870+0000) 2022-01-31T20:51:47.348 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:47 smithi167 conmon[60316]: debug 2022-01-31T20:51:47.055+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:47.056314+0000) 2022-01-31T20:51:47.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:47 smithi171 conmon[46715]: debug 2022-01-31T20:51:47.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:47.334383+0000) 2022-01-31T20:51:47.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:47 smithi171 conmon[51620]: debug 2022-01-31T20:51:47.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:47.411522+0000) 2022-01-31T20:51:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:47 smithi171 conmon[41853]: debug 2022-01-31T20:51:47.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:47.540035+0000) 2022-01-31T20:51:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:47 smithi167 conmon[49112]: debug 2022-01-31T20:51:47.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:47.632964+0000) 2022-01-31T20:51:48.347 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:48 smithi167 conmon[54076]: debug 2022-01-31T20:51:48.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.305087+0000) 2022-01-31T20:51:48.348 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:48 smithi167 conmon[60316]: debug 2022-01-31T20:51:48.055+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.056495+0000) 2022-01-31T20:51:48.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[46715]: debug 2022-01-31T20:51:48.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.334612+0000) 2022-01-31T20:51:48.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[51620]: debug 2022-01-31T20:51:48.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.411743+0000) 2022-01-31T20:51:48.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[41853]: debug 2022-01-31T20:51:48.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.540193+0000) 2022-01-31T20:51:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:48 smithi167 conmon[49112]: debug 2022-01-31T20:51:48.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.633121+0000) 2022-01-31T20:51:49.303 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:48 smithi167 conmon[49112]: debug 2022-01-31T20:51:48.926+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.927121+0000) 2022-01-31T20:51:49.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:48 smithi167 conmon[54076]: debug 2022-01-31T20:51:48.926+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.926849+0000) 2022-01-31T20:51:49.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:48 smithi167 conmon[60316]: debug 2022-01-31T20:51:48.926+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.927365+0000) 2022-01-31T20:51:49.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:49 smithi167 conmon[60316]: debug 2022-01-31T20:51:49.056+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:49.056661+0000) 2022-01-31T20:51:49.333 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[35325]: debug 2022-01-31T20:51:48.934+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84236 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:49.334 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[41853]: debug 2022-01-31T20:51:48.925+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.926322+0000) 2022-01-31T20:51:49.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[46715]: debug 2022-01-31T20:51:48.925+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.926486+0000) 2022-01-31T20:51:49.335 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:48 smithi171 conmon[51620]: debug 2022-01-31T20:51:48.925+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:48.926171+0000) 2022-01-31T20:51:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:49 smithi171 conmon[41853]: debug 2022-01-31T20:51:49.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:49.540345+0000) 2022-01-31T20:51:49.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:49 smithi171 conmon[46715]: debug 2022-01-31T20:51:49.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:49.334778+0000) 2022-01-31T20:51:49.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:49 smithi171 conmon[51620]: debug 2022-01-31T20:51:49.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:49.411952+0000) 2022-01-31T20:51:49.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:49 smithi167 conmon[54076]: debug 2022-01-31T20:51:49.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:49.305248+0000) 2022-01-31T20:51:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:49 smithi167 conmon[49112]: debug 2022-01-31T20:51:49.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:49.633304+0000) 2022-01-31T20:51:50.348 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:50 smithi167 conmon[54076]: debug 2022-01-31T20:51:50.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:50.305399+0000) 2022-01-31T20:51:50.348 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:50 smithi167 conmon[60316]: debug 2022-01-31T20:51:50.056+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:50.056833+0000) 2022-01-31T20:51:50.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:50 smithi171 conmon[46715]: debug 2022-01-31T20:51:50.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:50.334909+0000) 2022-01-31T20:51:50.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:50 smithi171 conmon[51620]: debug 2022-01-31T20:51:50.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:50.412107+0000) 2022-01-31T20:51:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:50 smithi171 conmon[41853]: debug 2022-01-31T20:51:50.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:50.540494+0000) 2022-01-31T20:51:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:50 smithi167 conmon[49112]: debug 2022-01-31T20:51:50.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:50.633457+0000) 2022-01-31T20:51:51.348 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:51 smithi167 conmon[54076]: debug 2022-01-31T20:51:51.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:51.305562+0000) 2022-01-31T20:51:51.349 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:51 smithi167 conmon[60316]: debug 2022-01-31T20:51:51.056+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:51.057021+0000) 2022-01-31T20:51:51.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:51 smithi171 conmon[46715]: debug 2022-01-31T20:51:51.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:51.335069+0000) 2022-01-31T20:51:51.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:51 smithi171 conmon[51620]: debug 2022-01-31T20:51:51.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:51.412292+0000) 2022-01-31T20:51:51.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:51 smithi171 conmon[41853]: debug 2022-01-31T20:51:51.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:51.540675+0000) 2022-01-31T20:51:51.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:51 smithi167 conmon[49112]: debug 2022-01-31T20:51:51.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:51.633596+0000) 2022-01-31T20:51:52.348 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:52 smithi167 conmon[54076]: debug 2022-01-31T20:51:52.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:52.305659+0000) 2022-01-31T20:51:52.349 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:52 smithi167 conmon[60316]: debug 2022-01-31T20:51:52.056+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:52.057175+0000) 2022-01-31T20:51:52.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:52 smithi171 conmon[46715]: debug 2022-01-31T20:51:52.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:52.335253+0000) 2022-01-31T20:51:52.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:52 smithi171 conmon[51620]: debug 2022-01-31T20:51:52.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:52.412438+0000) 2022-01-31T20:51:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:52 smithi171 conmon[41853]: debug 2022-01-31T20:51:52.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:52.540871+0000) 2022-01-31T20:51:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:52 smithi167 conmon[49112]: debug 2022-01-31T20:51:52.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:52.633730+0000) 2022-01-31T20:51:53.348 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:53 smithi167 conmon[54076]: debug 2022-01-31T20:51:53.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.305809+0000) 2022-01-31T20:51:53.349 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:53 smithi167 conmon[60316]: debug 2022-01-31T20:51:53.056+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.057341+0000) 2022-01-31T20:51:53.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[46715]: debug 2022-01-31T20:51:53.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.335464+0000) 2022-01-31T20:51:53.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[51620]: debug 2022-01-31T20:51:53.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.412627+0000) 2022-01-31T20:51:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[41853]: debug 2022-01-31T20:51:53.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.541028+0000) 2022-01-31T20:51:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[35325]: debug 2022-01-31T20:51:53.584+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:51:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:53 smithi167 conmon[49112]: debug 2022-01-31T20:51:53.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.633868+0000) 2022-01-31T20:51:54.304 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[35325]: debug 2022-01-31T20:51:53.943+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84348 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:54.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[41853]: debug 2022-01-31T20:51:53.935+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.936861+0000) 2022-01-31T20:51:54.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[46715]: debug 2022-01-31T20:51:53.935+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.937195+0000) 2022-01-31T20:51:54.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:53 smithi171 conmon[51620]: debug 2022-01-31T20:51:53.935+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.937391+0000) 2022-01-31T20:51:54.306 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:53 smithi167 conmon[49112]: debug 2022-01-31T20:51:53.937+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.938152+0000) 2022-01-31T20:51:54.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:53 smithi167 conmon[54076]: debug 2022-01-31T20:51:53.938+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.938596+0000) 2022-01-31T20:51:54.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:53 smithi167 conmon[60316]: debug 2022-01-31T20:51:53.938+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:53.938512+0000) 2022-01-31T20:51:54.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:54 smithi167 conmon[60316]: debug 2022-01-31T20:51:54.057+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:54.057551+0000) 2022-01-31T20:51:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:54 smithi171 conmon[41853]: debug 2022-01-31T20:51:54.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:54.541198+0000) 2022-01-31T20:51:54.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:54 smithi171 conmon[46715]: debug 2022-01-31T20:51:54.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:54.335670+0000) 2022-01-31T20:51:54.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:54 smithi171 conmon[51620]: debug 2022-01-31T20:51:54.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:54.412807+0000) 2022-01-31T20:51:54.624 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:54 smithi167 conmon[54076]: debug 2022-01-31T20:51:54.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:54.305980+0000) 2022-01-31T20:51:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:54 smithi167 conmon[49112]: debug 2022-01-31T20:51:54.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:54.634045+0000) 2022-01-31T20:51:55.349 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:55 smithi167 conmon[54076]: debug 2022-01-31T20:51:55.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:55.306119+0000) 2022-01-31T20:51:55.350 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:55 smithi167 conmon[60316]: debug 2022-01-31T20:51:55.057+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:55.057656+0000) 2022-01-31T20:51:55.411 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:55 smithi171 conmon[46715]: debug 2022-01-31T20:51:55.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:55.335817+0000) 2022-01-31T20:51:55.664 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:55 smithi171 conmon[51620]: debug 2022-01-31T20:51:55.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:55.412953+0000) 2022-01-31T20:51:55.664 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:55 smithi171 conmon[41853]: debug 2022-01-31T20:51:55.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:55.541354+0000) 2022-01-31T20:51:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:55 smithi167 conmon[49112]: debug 2022-01-31T20:51:55.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:55.634214+0000) 2022-01-31T20:51:56.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:56 smithi167 conmon[54076]: debug 2022-01-31T20:51:56.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:56.306282+0000) 2022-01-31T20:51:56.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:56 smithi167 conmon[60316]: debug 2022-01-31T20:51:56.057+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:56.057845+0000) 2022-01-31T20:51:56.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:56 smithi171 conmon[46715]: debug 2022-01-31T20:51:56.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:56.335980+0000) 2022-01-31T20:51:56.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:56 smithi171 conmon[51620]: debug 2022-01-31T20:51:56.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:56.413132+0000) 2022-01-31T20:51:56.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:56 smithi171 conmon[41853]: debug 2022-01-31T20:51:56.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:56.541542+0000) 2022-01-31T20:51:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:56 smithi167 conmon[49112]: debug 2022-01-31T20:51:56.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:56.634419+0000) 2022-01-31T20:51:57.349 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:57 smithi167 conmon[54076]: debug 2022-01-31T20:51:57.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:57.306467+0000) 2022-01-31T20:51:57.350 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:57 smithi167 conmon[60316]: debug 2022-01-31T20:51:57.057+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:57.058054+0000) 2022-01-31T20:51:57.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:57 smithi171 conmon[46715]: debug 2022-01-31T20:51:57.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:57.336083+0000) 2022-01-31T20:51:57.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:57 smithi171 conmon[51620]: debug 2022-01-31T20:51:57.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:57.413323+0000) 2022-01-31T20:51:57.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:57 smithi171 conmon[41853]: debug 2022-01-31T20:51:57.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:57.541724+0000) 2022-01-31T20:51:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:57 smithi167 conmon[49112]: debug 2022-01-31T20:51:57.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:57.634568+0000) 2022-01-31T20:51:58.349 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:58 smithi167 conmon[54076]: debug 2022-01-31T20:51:58.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.306597+0000) 2022-01-31T20:51:58.350 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:58 smithi167 conmon[60316]: debug 2022-01-31T20:51:58.057+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.058243+0000) 2022-01-31T20:51:58.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[46715]: debug 2022-01-31T20:51:58.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.336197+0000) 2022-01-31T20:51:58.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[51620]: debug 2022-01-31T20:51:58.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.413504+0000) 2022-01-31T20:51:58.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[41853]: debug 2022-01-31T20:51:58.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.541930+0000) 2022-01-31T20:51:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:58 smithi167 conmon[49112]: debug 2022-01-31T20:51:58.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.634691+0000) 2022-01-31T20:51:59.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:58 smithi167 conmon[54076]: debug 2022-01-31T20:51:58.948+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.948952+0000) 2022-01-31T20:51:59.306 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:58 smithi167 conmon[49112]: debug 2022-01-31T20:51:58.947+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.948278+0000) 2022-01-31T20:51:59.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:58 smithi167 conmon[60316]: debug 2022-01-31T20:51:58.948+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.948728+0000) 2022-01-31T20:51:59.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:51:59 smithi167 conmon[60316]: debug 2022-01-31T20:51:59.057+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:59.058416+0000) 2022-01-31T20:51:59.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[41853]: debug 2022-01-31T20:51:58.945+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.947450+0000) 2022-01-31T20:51:59.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[46715]: debug 2022-01-31T20:51:58.947+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.948839+0000) 2022-01-31T20:51:59.336 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[35325]: debug 2022-01-31T20:51:58.954+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84456 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:51:59.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:58 smithi171 conmon[51620]: debug 2022-01-31T20:51:58.945+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:58.947342+0000) 2022-01-31T20:51:59.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:51:59 smithi171 conmon[46715]: debug 2022-01-31T20:51:59.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:59.336382+0000) 2022-01-31T20:51:59.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:51:59 smithi171 conmon[51620]: debug 2022-01-31T20:51:59.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:59.413690+0000) 2022-01-31T20:51:59.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:51:59 smithi171 conmon[41853]: debug 2022-01-31T20:51:59.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:59.542113+0000) 2022-01-31T20:51:59.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:51:59 smithi167 conmon[54076]: debug 2022-01-31T20:51:59.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:59.306745+0000) 2022-01-31T20:51:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:51:59 smithi167 conmon[49112]: debug 2022-01-31T20:51:59.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:51:59.634849+0000) 2022-01-31T20:52:00.348 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:00 smithi167 conmon[54076]: debug 2022-01-31T20:52:00.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:00.306933+0000) 2022-01-31T20:52:00.348 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:00 smithi167 conmon[60316]: debug 2022-01-31T20:52:00.058+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:00.058609+0000) 2022-01-31T20:52:00.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:00 smithi171 conmon[46715]: debug 2022-01-31T20:52:00.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:00.336555+0000) 2022-01-31T20:52:00.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:00 smithi171 conmon[51620]: debug 2022-01-31T20:52:00.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:00.413840+0000) 2022-01-31T20:52:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:00 smithi171 conmon[41853]: debug 2022-01-31T20:52:00.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:00.542267+0000) 2022-01-31T20:52:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:00 smithi167 conmon[49112]: debug 2022-01-31T20:52:00.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:00.634995+0000) 2022-01-31T20:52:01.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:01 smithi167 conmon[54076]: debug 2022-01-31T20:52:01.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:01.307140+0000) 2022-01-31T20:52:01.350 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:01 smithi167 conmon[60316]: debug 2022-01-31T20:52:01.058+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:01.058800+0000) 2022-01-31T20:52:01.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:01 smithi171 conmon[46715]: debug 2022-01-31T20:52:01.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:01.336740+0000) 2022-01-31T20:52:01.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:01 smithi171 conmon[51620]: debug 2022-01-31T20:52:01.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:01.414037+0000) 2022-01-31T20:52:01.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:01 smithi171 conmon[41853]: debug 2022-01-31T20:52:01.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:01.542475+0000) 2022-01-31T20:52:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:01 smithi167 conmon[49112]: debug 2022-01-31T20:52:01.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:01.635171+0000) 2022-01-31T20:52:02.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:02 smithi167 conmon[54076]: debug 2022-01-31T20:52:02.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:02.307325+0000) 2022-01-31T20:52:02.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:02 smithi167 conmon[60316]: debug 2022-01-31T20:52:02.058+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:02.058997+0000) 2022-01-31T20:52:02.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:02 smithi171 conmon[46715]: debug 2022-01-31T20:52:02.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:02.337015+0000) 2022-01-31T20:52:02.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:02 smithi171 conmon[51620]: debug 2022-01-31T20:52:02.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:02.414227+0000) 2022-01-31T20:52:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:02 smithi171 conmon[41853]: debug 2022-01-31T20:52:02.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:02.542683+0000) 2022-01-31T20:52:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:02 smithi167 conmon[49112]: debug 2022-01-31T20:52:02.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:02.635335+0000) 2022-01-31T20:52:03.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:03 smithi167 conmon[54076]: debug 2022-01-31T20:52:03.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.307477+0000) 2022-01-31T20:52:03.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:03 smithi167 conmon[60316]: debug 2022-01-31T20:52:03.058+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.059179+0000) 2022-01-31T20:52:03.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[46715]: debug 2022-01-31T20:52:03.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.337148+0000) 2022-01-31T20:52:03.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[51620]: debug 2022-01-31T20:52:03.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.414384+0000) 2022-01-31T20:52:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[41853]: debug 2022-01-31T20:52:03.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.542844+0000) 2022-01-31T20:52:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:03 smithi167 conmon[49112]: debug 2022-01-31T20:52:03.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.635494+0000) 2022-01-31T20:52:04.306 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:03 smithi167 conmon[49112]: debug 2022-01-31T20:52:03.958+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.958915+0000) 2022-01-31T20:52:04.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:03 smithi167 conmon[54076]: debug 2022-01-31T20:52:03.959+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.959960+0000) 2022-01-31T20:52:04.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:03 smithi167 conmon[60316]: debug 2022-01-31T20:52:03.959+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.959599+0000) 2022-01-31T20:52:04.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:04 smithi167 conmon[60316]: debug 2022-01-31T20:52:04.058+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:04.059332+0000) 2022-01-31T20:52:04.335 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[35325]: debug 2022-01-31T20:52:03.965+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84567 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:04.336 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[41853]: debug 2022-01-31T20:52:03.956+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.958021+0000) 2022-01-31T20:52:04.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[46715]: debug 2022-01-31T20:52:03.958+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.960367+0000) 2022-01-31T20:52:04.337 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:03 smithi171 conmon[51620]: debug 2022-01-31T20:52:03.957+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:03.959033+0000) 2022-01-31T20:52:04.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:04 smithi171 conmon[51620]: debug 2022-01-31T20:52:04.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:04.414503+0000) 2022-01-31T20:52:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:04 smithi171 conmon[41853]: debug 2022-01-31T20:52:04.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:04.542988+0000) 2022-01-31T20:52:04.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:04 smithi171 conmon[46715]: debug 2022-01-31T20:52:04.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:04.337320+0000) 2022-01-31T20:52:04.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:04 smithi167 conmon[54076]: debug 2022-01-31T20:52:04.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:04.307657+0000) 2022-01-31T20:52:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:04 smithi167 conmon[49112]: debug 2022-01-31T20:52:04.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:04.635659+0000) 2022-01-31T20:52:05.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:05 smithi167 conmon[54076]: debug 2022-01-31T20:52:05.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:05.307830+0000) 2022-01-31T20:52:05.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:05 smithi167 conmon[60316]: debug 2022-01-31T20:52:05.059+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:05.059487+0000) 2022-01-31T20:52:05.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:05 smithi171 conmon[46715]: debug 2022-01-31T20:52:05.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:05.337487+0000) 2022-01-31T20:52:05.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:05 smithi171 conmon[51620]: debug 2022-01-31T20:52:05.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:05.414669+0000) 2022-01-31T20:52:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:05 smithi171 conmon[41853]: debug 2022-01-31T20:52:05.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:05.543146+0000) 2022-01-31T20:52:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:05 smithi167 conmon[49112]: debug 2022-01-31T20:52:05.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:05.635748+0000) 2022-01-31T20:52:06.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:06 smithi167 conmon[54076]: debug 2022-01-31T20:52:06.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:06.308033+0000) 2022-01-31T20:52:06.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:06 smithi167 conmon[60316]: debug 2022-01-31T20:52:06.059+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:06.059704+0000) 2022-01-31T20:52:06.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:06 smithi171 conmon[46715]: debug 2022-01-31T20:52:06.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:06.337653+0000) 2022-01-31T20:52:06.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:06 smithi171 conmon[51620]: debug 2022-01-31T20:52:06.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:06.414864+0000) 2022-01-31T20:52:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:06 smithi171 conmon[41853]: debug 2022-01-31T20:52:06.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:06.543324+0000) 2022-01-31T20:52:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:06 smithi167 conmon[49112]: debug 2022-01-31T20:52:06.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:06.635867+0000) 2022-01-31T20:52:07.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:07 smithi167 conmon[54076]: debug 2022-01-31T20:52:07.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:07.308242+0000) 2022-01-31T20:52:07.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:07 smithi167 conmon[60316]: debug 2022-01-31T20:52:07.059+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:07.059912+0000) 2022-01-31T20:52:07.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:07 smithi171 conmon[46715]: debug 2022-01-31T20:52:07.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:07.337815+0000) 2022-01-31T20:52:07.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:07 smithi171 conmon[51620]: debug 2022-01-31T20:52:07.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:07.415058+0000) 2022-01-31T20:52:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:07 smithi171 conmon[41853]: debug 2022-01-31T20:52:07.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:07.543503+0000) 2022-01-31T20:52:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:07 smithi167 conmon[49112]: debug 2022-01-31T20:52:07.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:07.636032+0000) 2022-01-31T20:52:08.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:08 smithi167 conmon[54076]: debug 2022-01-31T20:52:08.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.308442+0000) 2022-01-31T20:52:08.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:08 smithi167 conmon[60316]: debug 2022-01-31T20:52:08.059+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.060130+0000) 2022-01-31T20:52:08.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[46715]: debug 2022-01-31T20:52:08.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.337983+0000) 2022-01-31T20:52:08.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[51620]: debug 2022-01-31T20:52:08.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.415264+0000) 2022-01-31T20:52:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[35325]: debug 2022-01-31T20:52:08.584+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:52:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[41853]: debug 2022-01-31T20:52:08.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.543685+0000) 2022-01-31T20:52:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:08 smithi167 conmon[49112]: debug 2022-01-31T20:52:08.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.636187+0000) 2022-01-31T20:52:09.307 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:08 smithi167 conmon[49112]: debug 2022-01-31T20:52:08.969+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.970232+0000) 2022-01-31T20:52:09.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:08 smithi167 conmon[54076]: debug 2022-01-31T20:52:08.968+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.969301+0000) 2022-01-31T20:52:09.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:08 smithi167 conmon[60316]: debug 2022-01-31T20:52:08.970+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.970510+0000) 2022-01-31T20:52:09.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:09 smithi167 conmon[60316]: debug 2022-01-31T20:52:09.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:09.060256+0000) 2022-01-31T20:52:09.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[51620]: debug 2022-01-31T20:52:08.968+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.969364+0000) 2022-01-31T20:52:09.337 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[35325]: debug 2022-01-31T20:52:08.977+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84675 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:09.337 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[41853]: debug 2022-01-31T20:52:08.968+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.969606+0000) 2022-01-31T20:52:09.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:08 smithi171 conmon[46715]: debug 2022-01-31T20:52:08.969+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:08.970695+0000) 2022-01-31T20:52:09.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:09 smithi171 conmon[51620]: debug 2022-01-31T20:52:09.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:09.415441+0000) 2022-01-31T20:52:09.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:09 smithi171 conmon[41853]: debug 2022-01-31T20:52:09.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:09.543905+0000) 2022-01-31T20:52:09.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:09 smithi171 conmon[46715]: debug 2022-01-31T20:52:09.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:09.338193+0000) 2022-01-31T20:52:09.626 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:09 smithi167 conmon[54076]: debug 2022-01-31T20:52:09.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:09.308611+0000) 2022-01-31T20:52:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:09 smithi167 conmon[49112]: debug 2022-01-31T20:52:09.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:09.636358+0000) 2022-01-31T20:52:10.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:10 smithi167 conmon[54076]: debug 2022-01-31T20:52:10.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:10.308747+0000) 2022-01-31T20:52:10.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:10 smithi167 conmon[60316]: debug 2022-01-31T20:52:10.059+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:10.060469+0000) 2022-01-31T20:52:10.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:10 smithi171 conmon[46715]: debug 2022-01-31T20:52:10.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:10.338365+0000) 2022-01-31T20:52:10.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:10 smithi171 conmon[51620]: debug 2022-01-31T20:52:10.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:10.415591+0000) 2022-01-31T20:52:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:10 smithi171 conmon[41853]: debug 2022-01-31T20:52:10.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:10.544077+0000) 2022-01-31T20:52:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:10 smithi167 conmon[49112]: debug 2022-01-31T20:52:10.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:10.636537+0000) 2022-01-31T20:52:11.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:11 smithi167 conmon[54076]: debug 2022-01-31T20:52:11.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:11.308925+0000) 2022-01-31T20:52:11.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:11 smithi167 conmon[60316]: debug 2022-01-31T20:52:11.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:11.060652+0000) 2022-01-31T20:52:11.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:11 smithi171 conmon[46715]: debug 2022-01-31T20:52:11.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:11.338573+0000) 2022-01-31T20:52:11.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:11 smithi171 conmon[51620]: debug 2022-01-31T20:52:11.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:11.415794+0000) 2022-01-31T20:52:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:11 smithi171 conmon[41853]: debug 2022-01-31T20:52:11.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:11.544276+0000) 2022-01-31T20:52:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:11 smithi167 conmon[49112]: debug 2022-01-31T20:52:11.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:11.636726+0000) 2022-01-31T20:52:12.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:12 smithi167 conmon[54076]: debug 2022-01-31T20:52:12.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:12.309109+0000) 2022-01-31T20:52:12.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:12 smithi167 conmon[60316]: debug 2022-01-31T20:52:12.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:12.060836+0000) 2022-01-31T20:52:12.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:12 smithi171 conmon[46715]: debug 2022-01-31T20:52:12.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:12.338781+0000) 2022-01-31T20:52:12.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:12 smithi171 conmon[51620]: debug 2022-01-31T20:52:12.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:12.415978+0000) 2022-01-31T20:52:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:12 smithi171 conmon[41853]: debug 2022-01-31T20:52:12.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:12.544470+0000) 2022-01-31T20:52:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:12 smithi167 conmon[49112]: debug 2022-01-31T20:52:12.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:12.636842+0000) 2022-01-31T20:52:13.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:13 smithi167 conmon[54076]: debug 2022-01-31T20:52:13.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.309299+0000) 2022-01-31T20:52:13.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:13 smithi167 conmon[60316]: debug 2022-01-31T20:52:13.059+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.061064+0000) 2022-01-31T20:52:13.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[46715]: debug 2022-01-31T20:52:13.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.338969+0000) 2022-01-31T20:52:13.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[51620]: debug 2022-01-31T20:52:13.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.416155+0000) 2022-01-31T20:52:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[41853]: debug 2022-01-31T20:52:13.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.544665+0000) 2022-01-31T20:52:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:13 smithi167 conmon[49112]: debug 2022-01-31T20:52:13.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.637019+0000) 2022-01-31T20:52:14.308 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:13 smithi167 conmon[49112]: debug 2022-01-31T20:52:13.979+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.980837+0000) 2022-01-31T20:52:14.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:13 smithi167 conmon[54076]: debug 2022-01-31T20:52:13.980+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.981817+0000) 2022-01-31T20:52:14.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:13 smithi167 conmon[60316]: debug 2022-01-31T20:52:13.980+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.981650+0000) 2022-01-31T20:52:14.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:14 smithi167 conmon[60316]: debug 2022-01-31T20:52:14.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:14.061238+0000) 2022-01-31T20:52:14.337 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[35325]: debug 2022-01-31T20:52:13.988+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84786 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:14.338 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[41853]: debug 2022-01-31T20:52:13.979+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.980716+0000) 2022-01-31T20:52:14.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[46715]: debug 2022-01-31T20:52:13.981+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.982013+0000) 2022-01-31T20:52:14.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:13 smithi171 conmon[51620]: debug 2022-01-31T20:52:13.980+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:13.981679+0000) 2022-01-31T20:52:14.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:14 smithi171 conmon[46715]: debug 2022-01-31T20:52:14.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:14.339134+0000) 2022-01-31T20:52:14.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:14 smithi171 conmon[51620]: debug 2022-01-31T20:52:14.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:14.416310+0000) 2022-01-31T20:52:14.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:14 smithi171 conmon[41853]: debug 2022-01-31T20:52:14.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:14.544843+0000) 2022-01-31T20:52:14.627 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:14 smithi167 conmon[54076]: debug 2022-01-31T20:52:14.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:14.309494+0000) 2022-01-31T20:52:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:14 smithi167 conmon[49112]: debug 2022-01-31T20:52:14.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:14.637229+0000) 2022-01-31T20:52:15.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:15 smithi167 conmon[54076]: debug 2022-01-31T20:52:15.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:15.309683+0000) 2022-01-31T20:52:15.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:15 smithi167 conmon[60316]: debug 2022-01-31T20:52:15.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:15.061387+0000) 2022-01-31T20:52:15.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:15 smithi171 conmon[46715]: debug 2022-01-31T20:52:15.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:15.339272+0000) 2022-01-31T20:52:15.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:15 smithi171 conmon[51620]: debug 2022-01-31T20:52:15.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:15.416469+0000) 2022-01-31T20:52:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:15 smithi171 conmon[41853]: debug 2022-01-31T20:52:15.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:15.544987+0000) 2022-01-31T20:52:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:15 smithi167 conmon[49112]: debug 2022-01-31T20:52:15.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:15.637398+0000) 2022-01-31T20:52:16.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:16 smithi167 conmon[54076]: debug 2022-01-31T20:52:16.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:16.309892+0000) 2022-01-31T20:52:16.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:16 smithi167 conmon[60316]: debug 2022-01-31T20:52:16.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:16.061558+0000) 2022-01-31T20:52:16.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:16 smithi171 conmon[46715]: debug 2022-01-31T20:52:16.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:16.339454+0000) 2022-01-31T20:52:16.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:16 smithi171 conmon[51620]: debug 2022-01-31T20:52:16.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:16.416656+0000) 2022-01-31T20:52:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:16 smithi171 conmon[41853]: debug 2022-01-31T20:52:16.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:16.545199+0000) 2022-01-31T20:52:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:16 smithi167 conmon[49112]: debug 2022-01-31T20:52:16.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:16.637590+0000) 2022-01-31T20:52:17.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:17 smithi167 conmon[54076]: debug 2022-01-31T20:52:17.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:17.310083+0000) 2022-01-31T20:52:17.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:17 smithi167 conmon[60316]: debug 2022-01-31T20:52:17.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:17.061698+0000) 2022-01-31T20:52:17.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:17 smithi171 conmon[46715]: debug 2022-01-31T20:52:17.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:17.339657+0000) 2022-01-31T20:52:17.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:17 smithi171 conmon[51620]: debug 2022-01-31T20:52:17.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:17.416816+0000) 2022-01-31T20:52:17.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:17 smithi171 conmon[41853]: debug 2022-01-31T20:52:17.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:17.545404+0000) 2022-01-31T20:52:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:17 smithi167 conmon[49112]: debug 2022-01-31T20:52:17.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:17.637727+0000) 2022-01-31T20:52:18.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:18 smithi167 conmon[54076]: debug 2022-01-31T20:52:18.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.310293+0000) 2022-01-31T20:52:18.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:18 smithi167 conmon[60316]: debug 2022-01-31T20:52:18.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.061866+0000) 2022-01-31T20:52:18.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:18 smithi171 conmon[46715]: debug 2022-01-31T20:52:18.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.339826+0000) 2022-01-31T20:52:18.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:18 smithi171 conmon[51620]: debug 2022-01-31T20:52:18.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.416987+0000) 2022-01-31T20:52:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:18 smithi171 conmon[41853]: debug 2022-01-31T20:52:18.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.545587+0000) 2022-01-31T20:52:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:18 smithi167 conmon[49112]: debug 2022-01-31T20:52:18.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.637863+0000) 2022-01-31T20:52:19.309 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:18 smithi167 conmon[49112]: debug 2022-01-31T20:52:18.991+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.992997+0000) 2022-01-31T20:52:19.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:18 smithi167 conmon[54076]: debug 2022-01-31T20:52:18.990+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.992338+0000) 2022-01-31T20:52:19.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:18 smithi167 conmon[60316]: debug 2022-01-31T20:52:18.991+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.993364+0000) 2022-01-31T20:52:19.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:19 smithi167 conmon[60316]: debug 2022-01-31T20:52:19.061+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:19.062018+0000) 2022-01-31T20:52:19.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:18 smithi171 conmon[46715]: debug 2022-01-31T20:52:18.991+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.992712+0000) 2022-01-31T20:52:19.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:19 smithi171 conmon[35325]: debug 2022-01-31T20:52:18.999+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 84904 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:19.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:18 smithi171 conmon[41853]: debug 2022-01-31T20:52:18.991+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.991849+0000) 2022-01-31T20:52:19.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:18 smithi171 conmon[51620]: debug 2022-01-31T20:52:18.992+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:18.993039+0000) 2022-01-31T20:52:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:19 smithi171 conmon[41853]: debug 2022-01-31T20:52:19.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:19.545774+0000) 2022-01-31T20:52:19.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:19 smithi171 conmon[46715]: debug 2022-01-31T20:52:19.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:19.340011+0000) 2022-01-31T20:52:19.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:19 smithi171 conmon[51620]: debug 2022-01-31T20:52:19.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:19.417170+0000) 2022-01-31T20:52:19.628 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:19 smithi167 conmon[54076]: debug 2022-01-31T20:52:19.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:19.310480+0000) 2022-01-31T20:52:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:19 smithi167 conmon[49112]: debug 2022-01-31T20:52:19.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:19.637960+0000) 2022-01-31T20:52:20.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:20 smithi167 conmon[54076]: debug 2022-01-31T20:52:20.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:20.310673+0000) 2022-01-31T20:52:20.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:20 smithi167 conmon[60316]: debug 2022-01-31T20:52:20.060+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:20.062219+0000) 2022-01-31T20:52:20.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:20 smithi171 conmon[46715]: debug 2022-01-31T20:52:20.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:20.340162+0000) 2022-01-31T20:52:20.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:20 smithi171 conmon[51620]: debug 2022-01-31T20:52:20.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:20.417320+0000) 2022-01-31T20:52:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:20 smithi171 conmon[41853]: debug 2022-01-31T20:52:20.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:20.545921+0000) 2022-01-31T20:52:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:20 smithi167 conmon[49112]: debug 2022-01-31T20:52:20.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:20.638078+0000) 2022-01-31T20:52:21.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:21 smithi167 conmon[54076]: debug 2022-01-31T20:52:21.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:21.310837+0000) 2022-01-31T20:52:21.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:21 smithi167 conmon[60316]: debug 2022-01-31T20:52:21.061+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:21.062376+0000) 2022-01-31T20:52:21.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:21 smithi171 conmon[46715]: debug 2022-01-31T20:52:21.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:21.340347+0000) 2022-01-31T20:52:21.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:21 smithi171 conmon[51620]: debug 2022-01-31T20:52:21.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:21.417488+0000) 2022-01-31T20:52:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:21 smithi171 conmon[41853]: debug 2022-01-31T20:52:21.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:21.546124+0000) 2022-01-31T20:52:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:21 smithi167 conmon[49112]: debug 2022-01-31T20:52:21.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:21.638263+0000) 2022-01-31T20:52:22.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:22 smithi167 conmon[54076]: debug 2022-01-31T20:52:22.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:22.310992+0000) 2022-01-31T20:52:22.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:22 smithi167 conmon[60316]: debug 2022-01-31T20:52:22.061+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:22.062533+0000) 2022-01-31T20:52:22.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:22 smithi171 conmon[46715]: debug 2022-01-31T20:52:22.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:22.340523+0000) 2022-01-31T20:52:22.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:22 smithi171 conmon[51620]: debug 2022-01-31T20:52:22.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:22.417621+0000) 2022-01-31T20:52:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:22 smithi171 conmon[41853]: debug 2022-01-31T20:52:22.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:22.546308+0000) 2022-01-31T20:52:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:22 smithi167 conmon[49112]: debug 2022-01-31T20:52:22.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:22.638451+0000) 2022-01-31T20:52:23.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:23 smithi167 conmon[54076]: debug 2022-01-31T20:52:23.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:23.311174+0000) 2022-01-31T20:52:23.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:23 smithi167 conmon[60316]: debug 2022-01-31T20:52:23.061+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:23.062739+0000) 2022-01-31T20:52:23.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:23 smithi171 conmon[46715]: debug 2022-01-31T20:52:23.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:23.340709+0000) 2022-01-31T20:52:23.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:23 smithi171 conmon[51620]: debug 2022-01-31T20:52:23.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:23.417794+0000) 2022-01-31T20:52:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:23 smithi171 conmon[35325]: debug 2022-01-31T20:52:23.585+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:52:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:23 smithi171 conmon[41853]: debug 2022-01-31T20:52:23.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:23.546486+0000) 2022-01-31T20:52:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:23 smithi167 conmon[49112]: debug 2022-01-31T20:52:23.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:23.638633+0000) 2022-01-31T20:52:24.310 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:24 smithi167 conmon[49112]: debug 2022-01-31T20:52:24.002+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.004150+0000) 2022-01-31T20:52:24.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:24 smithi167 conmon[54076]: debug 2022-01-31T20:52:24.001+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.003048+0000) 2022-01-31T20:52:24.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:24 smithi167 conmon[60316]: debug 2022-01-31T20:52:24.002+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.003991+0000) 2022-01-31T20:52:24.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:24 smithi167 conmon[60316]: debug 2022-01-31T20:52:24.061+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.062910+0000) 2022-01-31T20:52:24.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[35325]: debug 2022-01-31T20:52:24.010+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85015 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:24.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[41853]: debug 2022-01-31T20:52:24.002+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.003238+0000) 2022-01-31T20:52:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[46715]: debug 2022-01-31T20:52:24.003+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.004127+0000) 2022-01-31T20:52:24.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[51620]: debug 2022-01-31T20:52:24.003+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.003769+0000) 2022-01-31T20:52:24.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[51620]: debug 2022-01-31T20:52:24.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.417976+0000) 2022-01-31T20:52:24.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[41853]: debug 2022-01-31T20:52:24.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.546710+0000) 2022-01-31T20:52:24.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:24 smithi171 conmon[46715]: debug 2022-01-31T20:52:24.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.340874+0000) 2022-01-31T20:52:24.629 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:24 smithi167 conmon[54076]: debug 2022-01-31T20:52:24.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.311383+0000) 2022-01-31T20:52:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:24 smithi167 conmon[49112]: debug 2022-01-31T20:52:24.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:24.638789+0000) 2022-01-31T20:52:25.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:25 smithi167 conmon[54076]: debug 2022-01-31T20:52:25.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:25.311554+0000) 2022-01-31T20:52:25.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:25 smithi167 conmon[60316]: debug 2022-01-31T20:52:25.061+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:25.063073+0000) 2022-01-31T20:52:25.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:25 smithi171 conmon[46715]: debug 2022-01-31T20:52:25.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:25.341028+0000) 2022-01-31T20:52:25.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:25 smithi171 conmon[51620]: debug 2022-01-31T20:52:25.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:25.418146+0000) 2022-01-31T20:52:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:25 smithi171 conmon[41853]: debug 2022-01-31T20:52:25.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:25.546850+0000) 2022-01-31T20:52:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:25 smithi167 conmon[49112]: debug 2022-01-31T20:52:25.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:25.638910+0000) 2022-01-31T20:52:26.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:26 smithi167 conmon[54076]: debug 2022-01-31T20:52:26.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:26.311687+0000) 2022-01-31T20:52:26.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:26 smithi167 conmon[60316]: debug 2022-01-31T20:52:26.062+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:26.063262+0000) 2022-01-31T20:52:26.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:26 smithi171 conmon[46715]: debug 2022-01-31T20:52:26.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:26.341211+0000) 2022-01-31T20:52:26.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:26 smithi171 conmon[51620]: debug 2022-01-31T20:52:26.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:26.418335+0000) 2022-01-31T20:52:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:26 smithi171 conmon[41853]: debug 2022-01-31T20:52:26.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:26.546993+0000) 2022-01-31T20:52:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:26 smithi167 conmon[49112]: debug 2022-01-31T20:52:26.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:26.639066+0000) 2022-01-31T20:52:27.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:27 smithi167 conmon[54076]: debug 2022-01-31T20:52:27.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:27.311860+0000) 2022-01-31T20:52:27.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:27 smithi167 conmon[60316]: debug 2022-01-31T20:52:27.062+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:27.063423+0000) 2022-01-31T20:52:27.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:27 smithi171 conmon[46715]: debug 2022-01-31T20:52:27.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:27.341412+0000) 2022-01-31T20:52:27.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:27 smithi171 conmon[51620]: debug 2022-01-31T20:52:27.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:27.418506+0000) 2022-01-31T20:52:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:27 smithi171 conmon[41853]: debug 2022-01-31T20:52:27.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:27.547182+0000) 2022-01-31T20:52:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:27 smithi167 conmon[49112]: debug 2022-01-31T20:52:27.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:27.639273+0000) 2022-01-31T20:52:28.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:28 smithi167 conmon[54076]: debug 2022-01-31T20:52:28.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:28.312062+0000) 2022-01-31T20:52:28.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:28 smithi167 conmon[60316]: debug 2022-01-31T20:52:28.062+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:28.063603+0000) 2022-01-31T20:52:28.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:28 smithi171 conmon[46715]: debug 2022-01-31T20:52:28.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:28.341640+0000) 2022-01-31T20:52:28.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:28 smithi171 conmon[51620]: debug 2022-01-31T20:52:28.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:28.418710+0000) 2022-01-31T20:52:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:28 smithi171 conmon[41853]: debug 2022-01-31T20:52:28.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:28.547386+0000) 2022-01-31T20:52:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:28 smithi167 conmon[49112]: debug 2022-01-31T20:52:28.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:28.639434+0000) 2022-01-31T20:52:29.310 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:29 smithi167 conmon[49112]: debug 2022-01-31T20:52:29.019+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.020610+0000) 2022-01-31T20:52:29.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:29 smithi167 conmon[54076]: debug 2022-01-31T20:52:29.013+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.014753+0000) 2022-01-31T20:52:29.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:29 smithi167 conmon[60316]: debug 2022-01-31T20:52:29.013+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.015449+0000) 2022-01-31T20:52:29.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:29 smithi167 conmon[60316]: debug 2022-01-31T20:52:29.062+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.063753+0000) 2022-01-31T20:52:29.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[46715]: debug 2022-01-31T20:52:29.013+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.014467+0000) 2022-01-31T20:52:29.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[51620]: debug 2022-01-31T20:52:29.013+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.013820+0000) 2022-01-31T20:52:29.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[35325]: debug 2022-01-31T20:52:29.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[35325]: 2022-01-31T20:52:29.024+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85123 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:29.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[41853]: debug 2022-01-31T20:52:29.014+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.015017+0000) 2022-01-31T20:52:29.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[51620]: debug 2022-01-31T20:52:29.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.418907+0000) 2022-01-31T20:52:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[41853]: debug 2022-01-31T20:52:29.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.547583+0000) 2022-01-31T20:52:29.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:29 smithi171 conmon[46715]: debug 2022-01-31T20:52:29.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.341798+0000) 2022-01-31T20:52:29.629 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:29 smithi167 conmon[54076]: debug 2022-01-31T20:52:29.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.312197+0000) 2022-01-31T20:52:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:29 smithi167 conmon[49112]: debug 2022-01-31T20:52:29.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:29.639555+0000) 2022-01-31T20:52:30.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:30 smithi167 conmon[54076]: debug 2022-01-31T20:52:30.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:30.312350+0000) 2022-01-31T20:52:30.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:30 smithi167 conmon[60316]: debug 2022-01-31T20:52:30.062+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:30.063941+0000) 2022-01-31T20:52:30.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:30 smithi171 conmon[46715]: debug 2022-01-31T20:52:30.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:30.341952+0000) 2022-01-31T20:52:30.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:30 smithi171 conmon[51620]: debug 2022-01-31T20:52:30.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:30.419058+0000) 2022-01-31T20:52:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:30 smithi171 conmon[41853]: debug 2022-01-31T20:52:30.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:30.547701+0000) 2022-01-31T20:52:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:30 smithi167 conmon[49112]: debug 2022-01-31T20:52:30.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:30.639662+0000) 2022-01-31T20:52:31.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:31 smithi167 conmon[54076]: debug 2022-01-31T20:52:31.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:31.312539+0000) 2022-01-31T20:52:31.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:31 smithi167 conmon[60316]: debug 2022-01-31T20:52:31.063+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:31.064144+0000) 2022-01-31T20:52:31.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:31 smithi171 conmon[51620]: debug 2022-01-31T20:52:31.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:31.419238+0000) 2022-01-31T20:52:31.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:31 smithi171 conmon[46715]: debug 2022-01-31T20:52:31.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:31.342109+0000) 2022-01-31T20:52:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:31 smithi171 conmon[41853]: debug 2022-01-31T20:52:31.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:31.547820+0000) 2022-01-31T20:52:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:31 smithi167 conmon[49112]: debug 2022-01-31T20:52:31.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:31.639869+0000) 2022-01-31T20:52:32.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:32 smithi167 conmon[54076]: debug 2022-01-31T20:52:32.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:32.312723+0000) 2022-01-31T20:52:32.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:32 smithi167 conmon[60316]: debug 2022-01-31T20:52:32.063+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:32.064299+0000) 2022-01-31T20:52:32.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:32 smithi171 conmon[51620]: debug 2022-01-31T20:52:32.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:32.419412+0000) 2022-01-31T20:52:32.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:32 smithi171 conmon[46715]: debug 2022-01-31T20:52:32.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:32.342265+0000) 2022-01-31T20:52:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:32 smithi171 conmon[41853]: debug 2022-01-31T20:52:32.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:32.548003+0000) 2022-01-31T20:52:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:32 smithi167 conmon[49112]: debug 2022-01-31T20:52:32.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:32.640070+0000) 2022-01-31T20:52:33.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:33 smithi167 conmon[54076]: debug 2022-01-31T20:52:33.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:33.312912+0000) 2022-01-31T20:52:33.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:33 smithi167 conmon[60316]: debug 2022-01-31T20:52:33.063+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:33.064488+0000) 2022-01-31T20:52:33.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:33 smithi171 conmon[46715]: debug 2022-01-31T20:52:33.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:33.342450+0000) 2022-01-31T20:52:33.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:33 smithi171 conmon[51620]: debug 2022-01-31T20:52:33.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:33.419609+0000) 2022-01-31T20:52:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:33 smithi171 conmon[41853]: debug 2022-01-31T20:52:33.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:33.548125+0000) 2022-01-31T20:52:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:33 smithi167 conmon[49112]: debug 2022-01-31T20:52:33.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:33.640256+0000) 2022-01-31T20:52:34.311 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:34 smithi167 conmon[49112]: debug 2022-01-31T20:52:34.026+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.028352+0000) 2022-01-31T20:52:34.312 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:34 smithi167 conmon[54076]: debug 2022-01-31T20:52:34.026+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.027610+0000) 2022-01-31T20:52:34.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:34 smithi167 conmon[60316]: debug 2022-01-31T20:52:34.027+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.028616+0000) 2022-01-31T20:52:34.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:34 smithi167 conmon[60316]: debug 2022-01-31T20:52:34.063+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.064578+0000) 2022-01-31T20:52:34.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[35325]: debug 2022-01-31T20:52:34.035+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85235 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:34.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[41853]: debug 2022-01-31T20:52:34.027+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.028652+0000) 2022-01-31T20:52:34.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[46715]: debug 2022-01-31T20:52:34.026+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.027435+0000) 2022-01-31T20:52:34.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[51620]: debug 2022-01-31T20:52:34.027+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.028207+0000) 2022-01-31T20:52:34.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[51620]: debug 2022-01-31T20:52:34.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.419804+0000) 2022-01-31T20:52:34.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[46715]: debug 2022-01-31T20:52:34.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.342586+0000) 2022-01-31T20:52:34.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:34 smithi171 conmon[41853]: debug 2022-01-31T20:52:34.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.548329+0000) 2022-01-31T20:52:34.631 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:34 smithi167 conmon[54076]: debug 2022-01-31T20:52:34.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.313100+0000) 2022-01-31T20:52:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:34 smithi167 conmon[49112]: debug 2022-01-31T20:52:34.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:34.640462+0000) 2022-01-31T20:52:35.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:35 smithi167 conmon[54076]: debug 2022-01-31T20:52:35.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:35.313260+0000) 2022-01-31T20:52:35.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:35 smithi167 conmon[60316]: debug 2022-01-31T20:52:35.064+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:35.064760+0000) 2022-01-31T20:52:35.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:35 smithi171 conmon[46715]: debug 2022-01-31T20:52:35.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:35.342779+0000) 2022-01-31T20:52:35.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:35 smithi171 conmon[51620]: debug 2022-01-31T20:52:35.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:35.419930+0000) 2022-01-31T20:52:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:35 smithi171 conmon[41853]: debug 2022-01-31T20:52:35.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:35.548430+0000) 2022-01-31T20:52:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:35 smithi167 conmon[49112]: debug 2022-01-31T20:52:35.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:35.640605+0000) 2022-01-31T20:52:36.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:36 smithi167 conmon[54076]: debug 2022-01-31T20:52:36.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:36.313460+0000) 2022-01-31T20:52:36.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:36 smithi167 conmon[60316]: debug 2022-01-31T20:52:36.064+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:36.064958+0000) 2022-01-31T20:52:36.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:36 smithi171 conmon[46715]: debug 2022-01-31T20:52:36.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:36.342970+0000) 2022-01-31T20:52:36.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:36 smithi171 conmon[51620]: debug 2022-01-31T20:52:36.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:36.420106+0000) 2022-01-31T20:52:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:36 smithi171 conmon[41853]: debug 2022-01-31T20:52:36.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:36.548622+0000) 2022-01-31T20:52:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:36 smithi167 conmon[49112]: debug 2022-01-31T20:52:36.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:36.640803+0000) 2022-01-31T20:52:37.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:37 smithi167 conmon[54076]: debug 2022-01-31T20:52:37.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:37.313545+0000) 2022-01-31T20:52:37.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:37 smithi167 conmon[60316]: debug 2022-01-31T20:52:37.064+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:37.065089+0000) 2022-01-31T20:52:37.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:37 smithi171 conmon[46715]: debug 2022-01-31T20:52:37.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:37.343098+0000) 2022-01-31T20:52:37.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:37 smithi171 conmon[51620]: debug 2022-01-31T20:52:37.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:37.420255+0000) 2022-01-31T20:52:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:37 smithi171 conmon[41853]: debug 2022-01-31T20:52:37.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:37.548838+0000) 2022-01-31T20:52:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:37 smithi167 conmon[49112]: debug 2022-01-31T20:52:37.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:37.641015+0000) 2022-01-31T20:52:38.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:38 smithi167 conmon[54076]: debug 2022-01-31T20:52:38.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:38.313701+0000) 2022-01-31T20:52:38.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:38 smithi167 conmon[60316]: debug 2022-01-31T20:52:38.064+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:38.065168+0000) 2022-01-31T20:52:38.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:38 smithi171 conmon[46715]: debug 2022-01-31T20:52:38.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:38.343306+0000) 2022-01-31T20:52:38.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:38 smithi171 conmon[51620]: debug 2022-01-31T20:52:38.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:38.420432+0000) 2022-01-31T20:52:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:38 smithi171 conmon[35325]: debug 2022-01-31T20:52:38.586+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:52:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:38 smithi171 conmon[41853]: debug 2022-01-31T20:52:38.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:38.549009+0000) 2022-01-31T20:52:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:38 smithi167 conmon[49112]: debug 2022-01-31T20:52:38.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:38.641219+0000) 2022-01-31T20:52:39.312 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:39 smithi167 conmon[49112]: debug 2022-01-31T20:52:39.037+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.038092+0000) 2022-01-31T20:52:39.313 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:39 smithi167 conmon[54076]: debug 2022-01-31T20:52:39.038+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.038567+0000) 2022-01-31T20:52:39.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:39 smithi167 conmon[60316]: debug 2022-01-31T20:52:39.038+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.039474+0000) 2022-01-31T20:52:39.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:39 smithi167 conmon[60316]: debug 2022-01-31T20:52:39.064+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.065363+0000) 2022-01-31T20:52:39.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[35325]: debug 2022-01-31T20:52:39.046+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85343 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:39.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[41853]: debug 2022-01-31T20:52:39.038+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.039038+0000) 2022-01-31T20:52:39.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[46715]: debug 2022-01-31T20:52:39.037+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.038413+0000) 2022-01-31T20:52:39.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[51620]: debug 2022-01-31T20:52:39.038+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.039410+0000) 2022-01-31T20:52:39.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[46715]: debug 2022-01-31T20:52:39.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.343456+0000) 2022-01-31T20:52:39.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[51620]: debug 2022-01-31T20:52:39.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.420606+0000) 2022-01-31T20:52:39.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:39 smithi171 conmon[41853]: debug 2022-01-31T20:52:39.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.549138+0000) 2022-01-31T20:52:39.632 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:39 smithi167 conmon[54076]: debug 2022-01-31T20:52:39.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.313886+0000) 2022-01-31T20:52:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:39 smithi167 conmon[49112]: debug 2022-01-31T20:52:39.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:39.641414+0000) 2022-01-31T20:52:40.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:40 smithi167 conmon[54076]: debug 2022-01-31T20:52:40.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:40.314058+0000) 2022-01-31T20:52:40.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:40 smithi167 conmon[60316]: debug 2022-01-31T20:52:40.065+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:40.065581+0000) 2022-01-31T20:52:40.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:40 smithi171 conmon[46715]: debug 2022-01-31T20:52:40.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:40.343636+0000) 2022-01-31T20:52:40.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:40 smithi171 conmon[51620]: debug 2022-01-31T20:52:40.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:40.420791+0000) 2022-01-31T20:52:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:40 smithi171 conmon[41853]: debug 2022-01-31T20:52:40.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:40.549301+0000) 2022-01-31T20:52:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:40 smithi167 conmon[49112]: debug 2022-01-31T20:52:40.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:40.641551+0000) 2022-01-31T20:52:41.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:41 smithi167 conmon[54076]: debug 2022-01-31T20:52:41.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:41.314264+0000) 2022-01-31T20:52:41.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:41 smithi167 conmon[60316]: debug 2022-01-31T20:52:41.065+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:41.065757+0000) 2022-01-31T20:52:41.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:41 smithi171 conmon[46715]: debug 2022-01-31T20:52:41.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:41.343820+0000) 2022-01-31T20:52:41.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:41 smithi171 conmon[51620]: debug 2022-01-31T20:52:41.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:41.420892+0000) 2022-01-31T20:52:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:41 smithi171 conmon[41853]: debug 2022-01-31T20:52:41.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:41.549459+0000) 2022-01-31T20:52:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:41 smithi167 conmon[49112]: debug 2022-01-31T20:52:41.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:41.641709+0000) 2022-01-31T20:52:42.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:42 smithi167 conmon[54076]: debug 2022-01-31T20:52:42.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:42.314419+0000) 2022-01-31T20:52:42.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:42 smithi167 conmon[60316]: debug 2022-01-31T20:52:42.065+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:42.065954+0000) 2022-01-31T20:52:42.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:42 smithi171 conmon[46715]: debug 2022-01-31T20:52:42.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:42.343940+0000) 2022-01-31T20:52:42.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:42 smithi171 conmon[51620]: debug 2022-01-31T20:52:42.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:42.421062+0000) 2022-01-31T20:52:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:42 smithi171 conmon[41853]: debug 2022-01-31T20:52:42.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:42.549657+0000) 2022-01-31T20:52:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:42 smithi167 conmon[49112]: debug 2022-01-31T20:52:42.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:42.641880+0000) 2022-01-31T20:52:43.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:43 smithi167 conmon[54076]: debug 2022-01-31T20:52:43.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:43.314576+0000) 2022-01-31T20:52:43.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:43 smithi167 conmon[60316]: debug 2022-01-31T20:52:43.065+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:43.066149+0000) 2022-01-31T20:52:43.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:43 smithi171 conmon[46715]: debug 2022-01-31T20:52:43.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:43.344116+0000) 2022-01-31T20:52:43.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:43 smithi171 conmon[51620]: debug 2022-01-31T20:52:43.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:43.421260+0000) 2022-01-31T20:52:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:43 smithi171 conmon[41853]: debug 2022-01-31T20:52:43.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:43.549829+0000) 2022-01-31T20:52:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:43 smithi167 conmon[49112]: debug 2022-01-31T20:52:43.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:43.642083+0000) 2022-01-31T20:52:44.313 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:44 smithi167 conmon[49112]: debug 2022-01-31T20:52:44.049+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.050106+0000) 2022-01-31T20:52:44.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:44 smithi167 conmon[54076]: debug 2022-01-31T20:52:44.049+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.050429+0000) 2022-01-31T20:52:44.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:44 smithi167 conmon[60316]: debug 2022-01-31T20:52:44.049+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.049997+0000) 2022-01-31T20:52:44.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:44 smithi167 conmon[60316]: debug 2022-01-31T20:52:44.065+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.066296+0000) 2022-01-31T20:52:44.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[35325]: debug 2022-01-31T20:52:44.056+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85454 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:44.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[41853]: debug 2022-01-31T20:52:44.048+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.050448+0000) 2022-01-31T20:52:44.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[46715]: debug 2022-01-31T20:52:44.047+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.048804+0000) 2022-01-31T20:52:44.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[51620]: debug 2022-01-31T20:52:44.048+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.049868+0000) 2022-01-31T20:52:44.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[51620]: debug 2022-01-31T20:52:44.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.421435+0000) 2022-01-31T20:52:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[41853]: debug 2022-01-31T20:52:44.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.550021+0000) 2022-01-31T20:52:44.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:44 smithi171 conmon[46715]: debug 2022-01-31T20:52:44.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.344321+0000) 2022-01-31T20:52:44.632 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:44 smithi167 conmon[54076]: debug 2022-01-31T20:52:44.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.314712+0000) 2022-01-31T20:52:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:44 smithi167 conmon[49112]: debug 2022-01-31T20:52:44.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:44.642288+0000) 2022-01-31T20:52:45.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:45 smithi167 conmon[54076]: debug 2022-01-31T20:52:45.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:45.314851+0000) 2022-01-31T20:52:45.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:45 smithi167 conmon[60316]: debug 2022-01-31T20:52:45.065+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:45.066445+0000) 2022-01-31T20:52:45.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:45 smithi171 conmon[46715]: debug 2022-01-31T20:52:45.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:45.344470+0000) 2022-01-31T20:52:45.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:45 smithi171 conmon[51620]: debug 2022-01-31T20:52:45.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:45.421556+0000) 2022-01-31T20:52:45.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:45 smithi171 conmon[41853]: debug 2022-01-31T20:52:45.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:45.550212+0000) 2022-01-31T20:52:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:45 smithi167 conmon[49112]: debug 2022-01-31T20:52:45.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:45.642493+0000) 2022-01-31T20:52:46.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:46 smithi167 conmon[54076]: debug 2022-01-31T20:52:46.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:46.315055+0000) 2022-01-31T20:52:46.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:46 smithi167 conmon[60316]: debug 2022-01-31T20:52:46.066+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:46.066680+0000) 2022-01-31T20:52:46.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:46 smithi171 conmon[46715]: debug 2022-01-31T20:52:46.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:46.344610+0000) 2022-01-31T20:52:46.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:46 smithi171 conmon[51620]: debug 2022-01-31T20:52:46.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:46.421707+0000) 2022-01-31T20:52:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:46 smithi171 conmon[41853]: debug 2022-01-31T20:52:46.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:46.550356+0000) 2022-01-31T20:52:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:46 smithi167 conmon[49112]: debug 2022-01-31T20:52:46.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:46.642628+0000) 2022-01-31T20:52:47.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:47 smithi167 conmon[54076]: debug 2022-01-31T20:52:47.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:47.315274+0000) 2022-01-31T20:52:47.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:47 smithi167 conmon[60316]: debug 2022-01-31T20:52:47.066+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:47.066867+0000) 2022-01-31T20:52:47.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:47 smithi171 conmon[46715]: debug 2022-01-31T20:52:47.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:47.344788+0000) 2022-01-31T20:52:47.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:47 smithi171 conmon[51620]: debug 2022-01-31T20:52:47.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:47.421877+0000) 2022-01-31T20:52:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:47 smithi171 conmon[41853]: debug 2022-01-31T20:52:47.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:47.550539+0000) 2022-01-31T20:52:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:47 smithi167 conmon[49112]: debug 2022-01-31T20:52:47.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:47.642780+0000) 2022-01-31T20:52:48.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:48 smithi167 conmon[54076]: debug 2022-01-31T20:52:48.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:48.315490+0000) 2022-01-31T20:52:48.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:48 smithi167 conmon[60316]: debug 2022-01-31T20:52:48.066+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:48.067073+0000) 2022-01-31T20:52:48.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:48 smithi171 conmon[46715]: debug 2022-01-31T20:52:48.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:48.344979+0000) 2022-01-31T20:52:48.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:48 smithi171 conmon[51620]: debug 2022-01-31T20:52:48.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:48.422056+0000) 2022-01-31T20:52:48.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:48 smithi171 conmon[41853]: debug 2022-01-31T20:52:48.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:48.550721+0000) 2022-01-31T20:52:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:48 smithi167 conmon[49112]: debug 2022-01-31T20:52:48.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:48.642969+0000) 2022-01-31T20:52:49.314 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:49 smithi167 conmon[49112]: debug 2022-01-31T20:52:49.061+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.061854+0000) 2022-01-31T20:52:49.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:49 smithi167 conmon[54076]: debug 2022-01-31T20:52:49.061+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.061638+0000) 2022-01-31T20:52:49.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:49 smithi167 conmon[60316]: debug 2022-01-31T20:52:49.060+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.061270+0000) 2022-01-31T20:52:49.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:49 smithi167 conmon[60316]: debug 2022-01-31T20:52:49.066+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.067198+0000) 2022-01-31T20:52:49.343 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[35325]: debug 2022-01-31T20:52:49.068+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85559 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:49.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[41853]: debug 2022-01-31T20:52:49.058+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.060631+0000) 2022-01-31T20:52:49.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[51620]: debug 2022-01-31T20:52:49.059+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.061580+0000) 2022-01-31T20:52:49.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[46715]: debug 2022-01-31T20:52:49.058+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.060417+0000) 2022-01-31T20:52:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[41853]: debug 2022-01-31T20:52:49.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.550925+0000) 2022-01-31T20:52:49.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[51620]: debug 2022-01-31T20:52:49.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.422233+0000) 2022-01-31T20:52:49.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:49 smithi171 conmon[46715]: debug 2022-01-31T20:52:49.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.345177+0000) 2022-01-31T20:52:49.633 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:49 smithi167 conmon[54076]: debug 2022-01-31T20:52:49.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.315654+0000) 2022-01-31T20:52:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:49 smithi167 conmon[49112]: debug 2022-01-31T20:52:49.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:49.643184+0000) 2022-01-31T20:52:50.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:50 smithi167 conmon[54076]: debug 2022-01-31T20:52:50.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:50.315801+0000) 2022-01-31T20:52:50.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:50 smithi167 conmon[60316]: debug 2022-01-31T20:52:50.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:50.067377+0000) 2022-01-31T20:52:50.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:50 smithi171 conmon[46715]: debug 2022-01-31T20:52:50.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:50.345322+0000) 2022-01-31T20:52:50.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:50 smithi171 conmon[51620]: debug 2022-01-31T20:52:50.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:50.422368+0000) 2022-01-31T20:52:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:50 smithi171 conmon[41853]: debug 2022-01-31T20:52:50.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:50.551073+0000) 2022-01-31T20:52:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:50 smithi167 conmon[49112]: debug 2022-01-31T20:52:50.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:50.643363+0000) 2022-01-31T20:52:51.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:51 smithi167 conmon[54076]: debug 2022-01-31T20:52:51.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:51.315963+0000) 2022-01-31T20:52:51.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:51 smithi167 conmon[60316]: debug 2022-01-31T20:52:51.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:51.067541+0000) 2022-01-31T20:52:51.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:51 smithi171 conmon[46715]: debug 2022-01-31T20:52:51.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:51.345504+0000) 2022-01-31T20:52:51.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:51 smithi171 conmon[51620]: debug 2022-01-31T20:52:51.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:51.422553+0000) 2022-01-31T20:52:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:51 smithi171 conmon[41853]: debug 2022-01-31T20:52:51.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:51.551262+0000) 2022-01-31T20:52:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:51 smithi167 conmon[49112]: debug 2022-01-31T20:52:51.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:51.643540+0000) 2022-01-31T20:52:52.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:52 smithi167 conmon[54076]: debug 2022-01-31T20:52:52.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:52.316142+0000) 2022-01-31T20:52:52.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:52 smithi167 conmon[60316]: debug 2022-01-31T20:52:52.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:52.067733+0000) 2022-01-31T20:52:52.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:52 smithi171 conmon[46715]: debug 2022-01-31T20:52:52.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:52.345723+0000) 2022-01-31T20:52:52.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:52 smithi171 conmon[51620]: debug 2022-01-31T20:52:52.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:52.422747+0000) 2022-01-31T20:52:52.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:52 smithi171 conmon[41853]: debug 2022-01-31T20:52:52.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:52.551477+0000) 2022-01-31T20:52:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:52 smithi167 conmon[49112]: debug 2022-01-31T20:52:52.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:52.643693+0000) 2022-01-31T20:52:53.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:53 smithi167 conmon[54076]: debug 2022-01-31T20:52:53.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:53.316347+0000) 2022-01-31T20:52:53.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:53 smithi167 conmon[60316]: debug 2022-01-31T20:52:53.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:53.067897+0000) 2022-01-31T20:52:53.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:53 smithi171 conmon[46715]: debug 2022-01-31T20:52:53.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:53.345941+0000) 2022-01-31T20:52:53.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:53 smithi171 conmon[51620]: debug 2022-01-31T20:52:53.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:53.422923+0000) 2022-01-31T20:52:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:53 smithi171 conmon[35325]: debug 2022-01-31T20:52:53.586+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:52:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:53 smithi171 conmon[41853]: debug 2022-01-31T20:52:53.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:53.551608+0000) 2022-01-31T20:52:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:53 smithi167 conmon[49112]: debug 2022-01-31T20:52:53.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:53.643855+0000) 2022-01-31T20:52:54.345 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[35325]: debug 2022-01-31T20:52:54.080+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85669 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:54.345 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[41853]: debug 2022-01-31T20:52:54.071+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.072641+0000) 2022-01-31T20:52:54.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[46715]: debug 2022-01-31T20:52:54.072+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.072852+0000) 2022-01-31T20:52:54.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[51620]: debug 2022-01-31T20:52:54.072+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.073449+0000) 2022-01-31T20:52:54.360 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:54 smithi167 conmon[49112]: debug 2022-01-31T20:52:54.073+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.073829+0000) 2022-01-31T20:52:54.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:54 smithi167 conmon[54076]: debug 2022-01-31T20:52:54.072+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.073349+0000) 2022-01-31T20:52:54.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:54 smithi167 conmon[54076]: debug 2022-01-31T20:52:54.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.316523+0000) 2022-01-31T20:52:54.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:54 smithi167 conmon[60316]: debug 2022-01-31T20:52:54.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.068055+0000) 2022-01-31T20:52:54.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:54 smithi167 conmon[60316]: debug 2022-01-31T20:52:54.072+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.073238+0000) 2022-01-31T20:52:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[41853]: debug 2022-01-31T20:52:54.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.551782+0000) 2022-01-31T20:52:54.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[46715]: debug 2022-01-31T20:52:54.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.346113+0000) 2022-01-31T20:52:54.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:54 smithi171 conmon[51620]: debug 2022-01-31T20:52:54.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.423104+0000) 2022-01-31T20:52:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:54 smithi167 conmon[49112]: debug 2022-01-31T20:52:54.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:54.644039+0000) 2022-01-31T20:52:55.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:55 smithi167 conmon[54076]: debug 2022-01-31T20:52:55.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:55.316679+0000) 2022-01-31T20:52:55.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:55 smithi167 conmon[60316]: debug 2022-01-31T20:52:55.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:55.068236+0000) 2022-01-31T20:52:55.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:55 smithi171 conmon[46715]: debug 2022-01-31T20:52:55.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:55.346241+0000) 2022-01-31T20:52:55.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:55 smithi171 conmon[51620]: debug 2022-01-31T20:52:55.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:55.423292+0000) 2022-01-31T20:52:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:55 smithi171 conmon[41853]: debug 2022-01-31T20:52:55.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:55.551900+0000) 2022-01-31T20:52:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:55 smithi167 conmon[49112]: debug 2022-01-31T20:52:55.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:55.644221+0000) 2022-01-31T20:52:56.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:56 smithi167 conmon[54076]: debug 2022-01-31T20:52:56.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:56.316813+0000) 2022-01-31T20:52:56.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:56 smithi167 conmon[60316]: debug 2022-01-31T20:52:56.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:56.068419+0000) 2022-01-31T20:52:56.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:56 smithi171 conmon[46715]: debug 2022-01-31T20:52:56.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:56.346399+0000) 2022-01-31T20:52:56.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:56 smithi171 conmon[51620]: debug 2022-01-31T20:52:56.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:56.423441+0000) 2022-01-31T20:52:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:56 smithi171 conmon[41853]: debug 2022-01-31T20:52:56.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:56.552057+0000) 2022-01-31T20:52:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:56 smithi167 conmon[49112]: debug 2022-01-31T20:52:56.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:56.644374+0000) 2022-01-31T20:52:57.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:57 smithi167 conmon[60316]: debug 2022-01-31T20:52:57.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:57.068645+0000) 2022-01-31T20:52:57.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:57 smithi171 conmon[46715]: debug 2022-01-31T20:52:57.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:57.346566+0000) 2022-01-31T20:52:57.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:57 smithi171 conmon[51620]: debug 2022-01-31T20:52:57.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:57.423640+0000) 2022-01-31T20:52:57.635 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:57 smithi167 conmon[54076]: debug 2022-01-31T20:52:57.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:57.316996+0000) 2022-01-31T20:52:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:57 smithi171 conmon[41853]: debug 2022-01-31T20:52:57.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:57.552242+0000) 2022-01-31T20:52:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:57 smithi167 conmon[49112]: debug 2022-01-31T20:52:57.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:57.644556+0000) 2022-01-31T20:52:58.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:58 smithi167 conmon[54076]: debug 2022-01-31T20:52:58.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:58.317198+0000) 2022-01-31T20:52:58.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:58 smithi167 conmon[60316]: debug 2022-01-31T20:52:58.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:58.068815+0000) 2022-01-31T20:52:58.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:58 smithi171 conmon[46715]: debug 2022-01-31T20:52:58.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:58.346782+0000) 2022-01-31T20:52:58.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:58 smithi171 conmon[51620]: debug 2022-01-31T20:52:58.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:58.423858+0000) 2022-01-31T20:52:58.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:58 smithi171 conmon[41853]: debug 2022-01-31T20:52:58.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:58.552429+0000) 2022-01-31T20:52:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:58 smithi167 conmon[49112]: debug 2022-01-31T20:52:58.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:58.644714+0000) 2022-01-31T20:52:59.345 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[35325]: debug 2022-01-31T20:52:59.091+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85778 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:52:59.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[41853]: debug 2022-01-31T20:52:59.082+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.083443+0000) 2022-01-31T20:52:59.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[46715]: debug 2022-01-31T20:52:59.083+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.083786+0000) 2022-01-31T20:52:59.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[51620]: debug 2022-01-31T20:52:59.083+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.084602+0000) 2022-01-31T20:52:59.360 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:59 smithi167 conmon[49112]: debug 2022-01-31T20:52:59.084+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.085018+0000) 2022-01-31T20:52:59.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:59 smithi167 conmon[54076]: debug 2022-01-31T20:52:59.083+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.084040+0000) 2022-01-31T20:52:59.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:52:59 smithi167 conmon[54076]: debug 2022-01-31T20:52:59.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.317387+0000) 2022-01-31T20:52:59.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:59 smithi167 conmon[60316]: debug 2022-01-31T20:52:59.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.069001+0000) 2022-01-31T20:52:59.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:52:59 smithi167 conmon[60316]: debug 2022-01-31T20:52:59.084+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.084514+0000) 2022-01-31T20:52:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[41853]: debug 2022-01-31T20:52:59.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.552621+0000) 2022-01-31T20:52:59.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[46715]: debug 2022-01-31T20:52:59.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.346927+0000) 2022-01-31T20:52:59.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:52:59 smithi171 conmon[51620]: debug 2022-01-31T20:52:59.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.424036+0000) 2022-01-31T20:52:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:52:59 smithi167 conmon[49112]: debug 2022-01-31T20:52:59.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:52:59.644856+0000) 2022-01-31T20:53:00.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:00 smithi167 conmon[54076]: debug 2022-01-31T20:53:00.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:00.317564+0000) 2022-01-31T20:53:00.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:00 smithi167 conmon[60316]: debug 2022-01-31T20:53:00.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:00.069190+0000) 2022-01-31T20:53:00.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:00 smithi171 conmon[46715]: debug 2022-01-31T20:53:00.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:00.347082+0000) 2022-01-31T20:53:00.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:00 smithi171 conmon[51620]: debug 2022-01-31T20:53:00.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:00.424161+0000) 2022-01-31T20:53:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:00 smithi171 conmon[41853]: debug 2022-01-31T20:53:00.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:00.552797+0000) 2022-01-31T20:53:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:00 smithi167 conmon[49112]: debug 2022-01-31T20:53:00.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:00.645013+0000) 2022-01-31T20:53:01.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:01 smithi167 conmon[54076]: debug 2022-01-31T20:53:01.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:01.317715+0000) 2022-01-31T20:53:01.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:01 smithi167 conmon[60316]: debug 2022-01-31T20:53:01.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:01.069321+0000) 2022-01-31T20:53:01.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:01 smithi171 conmon[46715]: debug 2022-01-31T20:53:01.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:01.347270+0000) 2022-01-31T20:53:01.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:01 smithi171 conmon[51620]: debug 2022-01-31T20:53:01.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:01.424278+0000) 2022-01-31T20:53:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:01 smithi171 conmon[41853]: debug 2022-01-31T20:53:01.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:01.552977+0000) 2022-01-31T20:53:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:01 smithi167 conmon[49112]: debug 2022-01-31T20:53:01.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:01.645226+0000) 2022-01-31T20:53:02.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:02 smithi167 conmon[54076]: debug 2022-01-31T20:53:02.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:02.317935+0000) 2022-01-31T20:53:02.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:02 smithi167 conmon[60316]: debug 2022-01-31T20:53:02.067+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:02.069486+0000) 2022-01-31T20:53:02.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:02 smithi171 conmon[46715]: debug 2022-01-31T20:53:02.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:02.347440+0000) 2022-01-31T20:53:02.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:02 smithi171 conmon[51620]: debug 2022-01-31T20:53:02.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:02.424463+0000) 2022-01-31T20:53:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:02 smithi171 conmon[41853]: debug 2022-01-31T20:53:02.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:02.553195+0000) 2022-01-31T20:53:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:02 smithi167 conmon[49112]: debug 2022-01-31T20:53:02.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:02.645434+0000) 2022-01-31T20:53:03.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:03 smithi167 conmon[54076]: debug 2022-01-31T20:53:03.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:03.318105+0000) 2022-01-31T20:53:03.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:03 smithi167 conmon[60316]: debug 2022-01-31T20:53:03.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:03.069700+0000) 2022-01-31T20:53:03.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:03 smithi171 conmon[46715]: debug 2022-01-31T20:53:03.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:03.347593+0000) 2022-01-31T20:53:03.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:03 smithi171 conmon[51620]: debug 2022-01-31T20:53:03.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:03.424654+0000) 2022-01-31T20:53:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:03 smithi171 conmon[41853]: debug 2022-01-31T20:53:03.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:03.553351+0000) 2022-01-31T20:53:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:03 smithi167 conmon[49112]: debug 2022-01-31T20:53:03.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:03.645618+0000) 2022-01-31T20:53:04.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[35325]: debug 2022-01-31T20:53:04.102+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85890 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:04.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[41853]: debug 2022-01-31T20:53:04.093+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.094401+0000) 2022-01-31T20:53:04.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[46715]: debug 2022-01-31T20:53:04.093+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.093878+0000) 2022-01-31T20:53:04.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[51620]: debug 2022-01-31T20:53:04.093+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.094626+0000) 2022-01-31T20:53:04.348 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:04 smithi167 conmon[49112]: debug 2022-01-31T20:53:04.094+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.095805+0000) 2022-01-31T20:53:04.349 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:04 smithi167 conmon[60316]: debug 2022-01-31T20:53:04.068+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.069903+0000) 2022-01-31T20:53:04.349 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:04 smithi167 conmon[60316]: debug 2022-01-31T20:53:04.093+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.095009+0000) 2022-01-31T20:53:04.349 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:04 smithi167 conmon[54076]: debug 2022-01-31T20:53:04.093+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.095317+0000) 2022-01-31T20:53:04.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:04 smithi167 conmon[54076]: debug 2022-01-31T20:53:04.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.318323+0000) 2022-01-31T20:53:04.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[46715]: debug 2022-01-31T20:53:04.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.347791+0000) 2022-01-31T20:53:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[41853]: debug 2022-01-31T20:53:04.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.553558+0000) 2022-01-31T20:53:04.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:04 smithi171 conmon[51620]: debug 2022-01-31T20:53:04.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.424844+0000) 2022-01-31T20:53:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:04 smithi167 conmon[49112]: debug 2022-01-31T20:53:04.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:04.645821+0000) 2022-01-31T20:53:05.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:05 smithi167 conmon[54076]: debug 2022-01-31T20:53:05.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:05.318477+0000) 2022-01-31T20:53:05.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:05 smithi167 conmon[60316]: debug 2022-01-31T20:53:05.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:05.070083+0000) 2022-01-31T20:53:05.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:05 smithi171 conmon[46715]: debug 2022-01-31T20:53:05.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:05.347974+0000) 2022-01-31T20:53:05.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:05 smithi171 conmon[51620]: debug 2022-01-31T20:53:05.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:05.424998+0000) 2022-01-31T20:53:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:05 smithi171 conmon[41853]: debug 2022-01-31T20:53:05.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:05.553738+0000) 2022-01-31T20:53:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:05 smithi167 conmon[49112]: debug 2022-01-31T20:53:05.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:05.645959+0000) 2022-01-31T20:53:06.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:06 smithi167 conmon[54076]: debug 2022-01-31T20:53:06.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:06.318655+0000) 2022-01-31T20:53:06.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:06 smithi167 conmon[60316]: debug 2022-01-31T20:53:06.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:06.070209+0000) 2022-01-31T20:53:06.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:06 smithi171 conmon[46715]: debug 2022-01-31T20:53:06.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:06.348176+0000) 2022-01-31T20:53:06.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:06 smithi171 conmon[51620]: debug 2022-01-31T20:53:06.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:06.425174+0000) 2022-01-31T20:53:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:06 smithi171 conmon[41853]: debug 2022-01-31T20:53:06.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:06.553959+0000) 2022-01-31T20:53:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:06 smithi167 conmon[49112]: debug 2022-01-31T20:53:06.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:06.646131+0000) 2022-01-31T20:53:07.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:07 smithi167 conmon[54076]: debug 2022-01-31T20:53:07.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:07.318795+0000) 2022-01-31T20:53:07.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:07 smithi167 conmon[60316]: debug 2022-01-31T20:53:07.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:07.070375+0000) 2022-01-31T20:53:07.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:07 smithi171 conmon[46715]: debug 2022-01-31T20:53:07.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:07.348383+0000) 2022-01-31T20:53:07.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:07 smithi171 conmon[51620]: debug 2022-01-31T20:53:07.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:07.425325+0000) 2022-01-31T20:53:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:07 smithi171 conmon[41853]: debug 2022-01-31T20:53:07.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:07.554134+0000) 2022-01-31T20:53:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:07 smithi167 conmon[49112]: debug 2022-01-31T20:53:07.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:07.646281+0000) 2022-01-31T20:53:08.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:08 smithi167 conmon[54076]: debug 2022-01-31T20:53:08.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:08.319000+0000) 2022-01-31T20:53:08.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:08 smithi167 conmon[60316]: debug 2022-01-31T20:53:08.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:08.070529+0000) 2022-01-31T20:53:08.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:08 smithi171 conmon[46715]: debug 2022-01-31T20:53:08.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:08.348623+0000) 2022-01-31T20:53:08.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:08 smithi171 conmon[51620]: debug 2022-01-31T20:53:08.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:08.425525+0000) 2022-01-31T20:53:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:08 smithi171 conmon[41853]: debug 2022-01-31T20:53:08.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:08.554317+0000) 2022-01-31T20:53:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:08 smithi171 conmon[35325]: debug 2022-01-31T20:53:08.587+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:53:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:08 smithi167 conmon[49112]: debug 2022-01-31T20:53:08.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:08.646392+0000) 2022-01-31T20:53:09.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[35325]: debug 2022-01-31T20:53:09.113+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 85998 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[41853]: debug 2022-01-31T20:53:09.104+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.105443+0000) 2022-01-31T20:53:09.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[51620]: debug 2022-01-31T20:53:09.105+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.106076+0000) 2022-01-31T20:53:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[46715]: debug 2022-01-31T20:53:09.105+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.105950+0000) 2022-01-31T20:53:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[46715]: debug 2022-01-31T20:53:09.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.348828+0000) 2022-01-31T20:53:09.362 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:09 smithi167 conmon[49112]: debug 2022-01-31T20:53:09.105+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.106704+0000) 2022-01-31T20:53:09.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:09 smithi167 conmon[54076]: debug 2022-01-31T20:53:09.105+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.106830+0000) 2022-01-31T20:53:09.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:09 smithi167 conmon[54076]: debug 2022-01-31T20:53:09.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.319160+0000) 2022-01-31T20:53:09.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:09 smithi167 conmon[60316]: debug 2022-01-31T20:53:09.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.070690+0000) 2022-01-31T20:53:09.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:09 smithi167 conmon[60316]: debug 2022-01-31T20:53:09.103+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.105365+0000) 2022-01-31T20:53:09.676 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[41853]: debug 2022-01-31T20:53:09.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.554505+0000) 2022-01-31T20:53:09.677 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:09 smithi171 conmon[51620]: debug 2022-01-31T20:53:09.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.425680+0000) 2022-01-31T20:53:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:09 smithi167 conmon[49112]: debug 2022-01-31T20:53:09.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:09.646580+0000) 2022-01-31T20:53:10.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:10 smithi167 conmon[54076]: debug 2022-01-31T20:53:10.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:10.319307+0000) 2022-01-31T20:53:10.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:10 smithi167 conmon[60316]: debug 2022-01-31T20:53:10.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:10.070833+0000) 2022-01-31T20:53:10.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:10 smithi171 conmon[46715]: debug 2022-01-31T20:53:10.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:10.348969+0000) 2022-01-31T20:53:10.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:10 smithi171 conmon[51620]: debug 2022-01-31T20:53:10.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:10.425810+0000) 2022-01-31T20:53:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:10 smithi171 conmon[41853]: debug 2022-01-31T20:53:10.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:10.554698+0000) 2022-01-31T20:53:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:10 smithi167 conmon[49112]: debug 2022-01-31T20:53:10.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:10.646704+0000) 2022-01-31T20:53:11.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:11 smithi167 conmon[54076]: debug 2022-01-31T20:53:11.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:11.319463+0000) 2022-01-31T20:53:11.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:11 smithi167 conmon[60316]: debug 2022-01-31T20:53:11.069+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:11.070961+0000) 2022-01-31T20:53:11.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:11 smithi171 conmon[46715]: debug 2022-01-31T20:53:11.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:11.349188+0000) 2022-01-31T20:53:11.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:11 smithi171 conmon[51620]: debug 2022-01-31T20:53:11.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:11.425926+0000) 2022-01-31T20:53:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:11 smithi171 conmon[41853]: debug 2022-01-31T20:53:11.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:11.554903+0000) 2022-01-31T20:53:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:11 smithi167 conmon[49112]: debug 2022-01-31T20:53:11.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:11.646842+0000) 2022-01-31T20:53:12.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:12 smithi167 conmon[60316]: debug 2022-01-31T20:53:12.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:12.071169+0000) 2022-01-31T20:53:12.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:12 smithi167 conmon[54076]: debug 2022-01-31T20:53:12.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:12.319677+0000) 2022-01-31T20:53:12.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:12 smithi171 conmon[46715]: debug 2022-01-31T20:53:12.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:12.349370+0000) 2022-01-31T20:53:12.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:12 smithi171 conmon[51620]: debug 2022-01-31T20:53:12.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:12.426055+0000) 2022-01-31T20:53:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:12 smithi171 conmon[41853]: debug 2022-01-31T20:53:12.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:12.555094+0000) 2022-01-31T20:53:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:12 smithi167 conmon[49112]: debug 2022-01-31T20:53:12.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:12.646942+0000) 2022-01-31T20:53:13.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:13 smithi167 conmon[54076]: debug 2022-01-31T20:53:13.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:13.319876+0000) 2022-01-31T20:53:13.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:13 smithi167 conmon[60316]: debug 2022-01-31T20:53:13.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:13.071362+0000) 2022-01-31T20:53:13.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:13 smithi171 conmon[46715]: debug 2022-01-31T20:53:13.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:13.349580+0000) 2022-01-31T20:53:13.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:13 smithi171 conmon[51620]: debug 2022-01-31T20:53:13.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:13.426163+0000) 2022-01-31T20:53:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:13 smithi171 conmon[41853]: debug 2022-01-31T20:53:13.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:13.555296+0000) 2022-01-31T20:53:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:13 smithi167 conmon[49112]: debug 2022-01-31T20:53:13.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:13.647103+0000) 2022-01-31T20:53:14.364 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:14 smithi167 conmon[49112]: debug 2022-01-31T20:53:14.115+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.117054+0000) 2022-01-31T20:53:14.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:14 smithi167 conmon[54076]: debug 2022-01-31T20:53:14.116+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.117956+0000) 2022-01-31T20:53:14.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:14 smithi167 conmon[54076]: debug 2022-01-31T20:53:14.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.320037+0000) 2022-01-31T20:53:14.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:14 smithi167 conmon[60316]: debug 2022-01-31T20:53:14.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.071550+0000) 2022-01-31T20:53:14.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:14 smithi167 conmon[60316]: debug 2022-01-31T20:53:14.115+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.117466+0000) 2022-01-31T20:53:14.424 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[35325]: debug 2022-01-31T20:53:14.124+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86109 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:14.425 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[41853]: debug 2022-01-31T20:53:14.116+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.116838+0000) 2022-01-31T20:53:14.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[46715]: debug 2022-01-31T20:53:14.116+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.116805+0000) 2022-01-31T20:53:14.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[46715]: debug 2022-01-31T20:53:14.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.349821+0000) 2022-01-31T20:53:14.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[51620]: debug 2022-01-31T20:53:14.116+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.117279+0000) 2022-01-31T20:53:14.677 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[41853]: debug 2022-01-31T20:53:14.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.555500+0000) 2022-01-31T20:53:14.678 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:14 smithi171 conmon[51620]: debug 2022-01-31T20:53:14.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.426344+0000) 2022-01-31T20:53:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:14 smithi167 conmon[49112]: debug 2022-01-31T20:53:14.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:14.647285+0000) 2022-01-31T20:53:15.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:15 smithi167 conmon[54076]: debug 2022-01-31T20:53:15.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:15.320209+0000) 2022-01-31T20:53:15.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:15 smithi167 conmon[60316]: debug 2022-01-31T20:53:15.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:15.071754+0000) 2022-01-31T20:53:15.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:15 smithi171 conmon[46715]: debug 2022-01-31T20:53:15.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:15.349978+0000) 2022-01-31T20:53:15.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:15 smithi171 conmon[51620]: debug 2022-01-31T20:53:15.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:15.426489+0000) 2022-01-31T20:53:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:15 smithi171 conmon[41853]: debug 2022-01-31T20:53:15.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:15.555685+0000) 2022-01-31T20:53:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:15 smithi167 conmon[49112]: debug 2022-01-31T20:53:15.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:15.647482+0000) 2022-01-31T20:53:16.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:16 smithi167 conmon[54076]: debug 2022-01-31T20:53:16.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:16.320406+0000) 2022-01-31T20:53:16.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:16 smithi167 conmon[60316]: debug 2022-01-31T20:53:16.071+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:16.071895+0000) 2022-01-31T20:53:16.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:16 smithi171 conmon[46715]: debug 2022-01-31T20:53:16.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:16.350140+0000) 2022-01-31T20:53:16.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:16 smithi171 conmon[51620]: debug 2022-01-31T20:53:16.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:16.426664+0000) 2022-01-31T20:53:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:16 smithi171 conmon[41853]: debug 2022-01-31T20:53:16.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:16.555868+0000) 2022-01-31T20:53:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:16 smithi167 conmon[49112]: debug 2022-01-31T20:53:16.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:16.647609+0000) 2022-01-31T20:53:17.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:17 smithi167 conmon[54076]: debug 2022-01-31T20:53:17.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:17.320633+0000) 2022-01-31T20:53:17.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:17 smithi167 conmon[60316]: debug 2022-01-31T20:53:17.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:17.072116+0000) 2022-01-31T20:53:17.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:17 smithi171 conmon[46715]: debug 2022-01-31T20:53:17.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:17.350297+0000) 2022-01-31T20:53:17.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:17 smithi171 conmon[51620]: debug 2022-01-31T20:53:17.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:17.426837+0000) 2022-01-31T20:53:17.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:17 smithi171 conmon[41853]: debug 2022-01-31T20:53:17.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:17.556028+0000) 2022-01-31T20:53:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:17 smithi167 conmon[49112]: debug 2022-01-31T20:53:17.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:17.647830+0000) 2022-01-31T20:53:18.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:18 smithi167 conmon[54076]: debug 2022-01-31T20:53:18.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:18.320790+0000) 2022-01-31T20:53:18.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:18 smithi167 conmon[60316]: debug 2022-01-31T20:53:18.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:18.072245+0000) 2022-01-31T20:53:18.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:18 smithi171 conmon[46715]: debug 2022-01-31T20:53:18.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:18.350470+0000) 2022-01-31T20:53:18.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:18 smithi171 conmon[51620]: debug 2022-01-31T20:53:18.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:18.426987+0000) 2022-01-31T20:53:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:18 smithi171 conmon[41853]: debug 2022-01-31T20:53:18.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:18.556205+0000) 2022-01-31T20:53:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:18 smithi167 conmon[49112]: debug 2022-01-31T20:53:18.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:18.647992+0000) 2022-01-31T20:53:19.364 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:19 smithi167 conmon[49112]: debug 2022-01-31T20:53:19.127+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.128848+0000) 2022-01-31T20:53:19.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:19 smithi167 conmon[54076]: debug 2022-01-31T20:53:19.127+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.129098+0000) 2022-01-31T20:53:19.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:19 smithi167 conmon[54076]: debug 2022-01-31T20:53:19.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.320949+0000) 2022-01-31T20:53:19.366 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:19 smithi167 conmon[60316]: debug 2022-01-31T20:53:19.070+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.072412+0000) 2022-01-31T20:53:19.366 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:19 smithi167 conmon[60316]: debug 2022-01-31T20:53:19.126+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.128406+0000) 2022-01-31T20:53:19.425 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[35325]: debug 2022-01-31T20:53:19.135+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86217 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:19.426 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[41853]: debug 2022-01-31T20:53:19.127+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.128249+0000) 2022-01-31T20:53:19.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[51620]: debug 2022-01-31T20:53:19.127+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.128703+0000) 2022-01-31T20:53:19.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[46715]: debug 2022-01-31T20:53:19.127+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.127984+0000) 2022-01-31T20:53:19.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[46715]: debug 2022-01-31T20:53:19.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.350702+0000) 2022-01-31T20:53:19.678 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[51620]: debug 2022-01-31T20:53:19.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.427136+0000) 2022-01-31T20:53:19.678 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:19 smithi171 conmon[41853]: debug 2022-01-31T20:53:19.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.556345+0000) 2022-01-31T20:53:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:19 smithi167 conmon[49112]: debug 2022-01-31T20:53:19.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:19.648185+0000) 2022-01-31T20:53:20.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:20 smithi167 conmon[54076]: debug 2022-01-31T20:53:20.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:20.321102+0000) 2022-01-31T20:53:20.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:20 smithi167 conmon[60316]: debug 2022-01-31T20:53:20.071+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:20.072574+0000) 2022-01-31T20:53:20.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:20 smithi171 conmon[46715]: debug 2022-01-31T20:53:20.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:20.350856+0000) 2022-01-31T20:53:20.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:20 smithi171 conmon[51620]: debug 2022-01-31T20:53:20.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:20.427268+0000) 2022-01-31T20:53:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:20 smithi171 conmon[41853]: debug 2022-01-31T20:53:20.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:20.556480+0000) 2022-01-31T20:53:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:20 smithi167 conmon[49112]: debug 2022-01-31T20:53:20.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:20.648354+0000) 2022-01-31T20:53:21.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:21 smithi167 conmon[54076]: debug 2022-01-31T20:53:21.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:21.321294+0000) 2022-01-31T20:53:21.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:21 smithi167 conmon[60316]: debug 2022-01-31T20:53:21.071+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:21.072674+0000) 2022-01-31T20:53:21.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:21 smithi171 conmon[51620]: debug 2022-01-31T20:53:21.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:21.427401+0000) 2022-01-31T20:53:21.675 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:21 smithi171 conmon[46715]: debug 2022-01-31T20:53:21.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:21.351041+0000) 2022-01-31T20:53:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:21 smithi171 conmon[41853]: debug 2022-01-31T20:53:21.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:21.556670+0000) 2022-01-31T20:53:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:21 smithi167 conmon[49112]: debug 2022-01-31T20:53:21.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:21.648470+0000) 2022-01-31T20:53:22.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:22 smithi167 conmon[54076]: debug 2022-01-31T20:53:22.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:22.321494+0000) 2022-01-31T20:53:22.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:22 smithi167 conmon[60316]: debug 2022-01-31T20:53:22.071+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:22.072799+0000) 2022-01-31T20:53:22.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:22 smithi171 conmon[46715]: debug 2022-01-31T20:53:22.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:22.351200+0000) 2022-01-31T20:53:22.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:22 smithi171 conmon[51620]: debug 2022-01-31T20:53:22.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:22.427580+0000) 2022-01-31T20:53:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:22 smithi171 conmon[41853]: debug 2022-01-31T20:53:22.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:22.556825+0000) 2022-01-31T20:53:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:22 smithi167 conmon[49112]: debug 2022-01-31T20:53:22.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:22.648600+0000) 2022-01-31T20:53:23.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:23 smithi167 conmon[60316]: debug 2022-01-31T20:53:23.072+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:23.072936+0000) 2022-01-31T20:53:23.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:23 smithi167 conmon[54076]: debug 2022-01-31T20:53:23.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:23.321679+0000) 2022-01-31T20:53:23.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:23 smithi171 conmon[46715]: debug 2022-01-31T20:53:23.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:23.351386+0000) 2022-01-31T20:53:23.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:23 smithi171 conmon[51620]: debug 2022-01-31T20:53:23.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:23.427770+0000) 2022-01-31T20:53:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:23 smithi171 conmon[35325]: debug 2022-01-31T20:53:23.588+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:53:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:23 smithi171 conmon[41853]: debug 2022-01-31T20:53:23.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:23.557007+0000) 2022-01-31T20:53:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:23 smithi167 conmon[49112]: debug 2022-01-31T20:53:23.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:23.648768+0000) 2022-01-31T20:53:24.365 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:24 smithi167 conmon[49112]: debug 2022-01-31T20:53:24.138+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.140178+0000) 2022-01-31T20:53:24.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:24 smithi167 conmon[54076]: debug 2022-01-31T20:53:24.138+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.139509+0000) 2022-01-31T20:53:24.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:24 smithi167 conmon[54076]: debug 2022-01-31T20:53:24.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.321837+0000) 2022-01-31T20:53:24.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:24 smithi167 conmon[60316]: debug 2022-01-31T20:53:24.072+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.073101+0000) 2022-01-31T20:53:24.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:24 smithi167 conmon[60316]: debug 2022-01-31T20:53:24.137+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.138951+0000) 2022-01-31T20:53:24.426 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[35325]: debug 2022-01-31T20:53:24.147+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86328 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:24.427 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[41853]: debug 2022-01-31T20:53:24.138+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.139727+0000) 2022-01-31T20:53:24.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[51620]: debug 2022-01-31T20:53:24.138+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.139193+0000) 2022-01-31T20:53:24.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[46715]: debug 2022-01-31T20:53:24.138+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.139040+0000) 2022-01-31T20:53:24.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[46715]: debug 2022-01-31T20:53:24.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.351593+0000) 2022-01-31T20:53:24.679 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[51620]: debug 2022-01-31T20:53:24.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.427915+0000) 2022-01-31T20:53:24.679 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:24 smithi171 conmon[41853]: debug 2022-01-31T20:53:24.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.557163+0000) 2022-01-31T20:53:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:24 smithi167 conmon[49112]: debug 2022-01-31T20:53:24.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:24.648947+0000) 2022-01-31T20:53:25.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:25 smithi167 conmon[54076]: debug 2022-01-31T20:53:25.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:25.322029+0000) 2022-01-31T20:53:25.366 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:25 smithi167 conmon[60316]: debug 2022-01-31T20:53:25.072+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:25.073295+0000) 2022-01-31T20:53:25.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:25 smithi171 conmon[46715]: debug 2022-01-31T20:53:25.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:25.351782+0000) 2022-01-31T20:53:25.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:25 smithi171 conmon[51620]: debug 2022-01-31T20:53:25.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:25.428034+0000) 2022-01-31T20:53:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:25 smithi171 conmon[41853]: debug 2022-01-31T20:53:25.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:25.557302+0000) 2022-01-31T20:53:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:25 smithi167 conmon[49112]: debug 2022-01-31T20:53:25.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:25.649105+0000) 2022-01-31T20:53:26.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:26 smithi167 conmon[54076]: debug 2022-01-31T20:53:26.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:26.322241+0000) 2022-01-31T20:53:26.366 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:26 smithi167 conmon[60316]: debug 2022-01-31T20:53:26.072+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:26.073451+0000) 2022-01-31T20:53:26.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:26 smithi171 conmon[46715]: debug 2022-01-31T20:53:26.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:26.351966+0000) 2022-01-31T20:53:26.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:26 smithi171 conmon[51620]: debug 2022-01-31T20:53:26.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:26.428218+0000) 2022-01-31T20:53:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:26 smithi171 conmon[41853]: debug 2022-01-31T20:53:26.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:26.557484+0000) 2022-01-31T20:53:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:26 smithi167 conmon[49112]: debug 2022-01-31T20:53:26.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:26.649311+0000) 2022-01-31T20:53:27.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:27 smithi167 conmon[54076]: debug 2022-01-31T20:53:27.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:27.322471+0000) 2022-01-31T20:53:27.366 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:27 smithi167 conmon[60316]: debug 2022-01-31T20:53:27.073+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:27.073659+0000) 2022-01-31T20:53:27.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:27 smithi171 conmon[46715]: debug 2022-01-31T20:53:27.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:27.352155+0000) 2022-01-31T20:53:27.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:27 smithi171 conmon[51620]: debug 2022-01-31T20:53:27.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:27.428404+0000) 2022-01-31T20:53:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:27 smithi171 conmon[41853]: debug 2022-01-31T20:53:27.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:27.557671+0000) 2022-01-31T20:53:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:27 smithi167 conmon[49112]: debug 2022-01-31T20:53:27.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:27.649488+0000) 2022-01-31T20:53:28.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:28 smithi167 conmon[54076]: debug 2022-01-31T20:53:28.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:28.322698+0000) 2022-01-31T20:53:28.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:28 smithi167 conmon[60316]: debug 2022-01-31T20:53:28.073+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:28.073854+0000) 2022-01-31T20:53:28.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:28 smithi171 conmon[46715]: debug 2022-01-31T20:53:28.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:28.352347+0000) 2022-01-31T20:53:28.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:28 smithi171 conmon[51620]: debug 2022-01-31T20:53:28.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:28.428549+0000) 2022-01-31T20:53:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:28 smithi171 conmon[41853]: debug 2022-01-31T20:53:28.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:28.557865+0000) 2022-01-31T20:53:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:28 smithi167 conmon[49112]: debug 2022-01-31T20:53:28.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:28.649652+0000) 2022-01-31T20:53:29.366 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:29 smithi167 conmon[49112]: debug 2022-01-31T20:53:29.150+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.151686+0000) 2022-01-31T20:53:29.367 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:29 smithi167 conmon[54076]: debug 2022-01-31T20:53:29.150+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.150947+0000) 2022-01-31T20:53:29.367 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:29 smithi167 conmon[54076]: debug 2022-01-31T20:53:29.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.322851+0000) 2022-01-31T20:53:29.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:29 smithi167 conmon[60316]: debug 2022-01-31T20:53:29.073+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.074011+0000) 2022-01-31T20:53:29.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:29 smithi167 conmon[60316]: debug 2022-01-31T20:53:29.150+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.150572+0000) 2022-01-31T20:53:29.427 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[41853]: debug 2022-01-31T20:53:29.148+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.150214+0000) 2022-01-31T20:53:29.428 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[35325]: debug 2022-01-31T20:53:29.159+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86436 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:29.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[46715]: debug 2022-01-31T20:53:29.148+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.150329+0000) 2022-01-31T20:53:29.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[46715]: debug 2022-01-31T20:53:29.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.352529+0000) 2022-01-31T20:53:29.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[51620]: debug 2022-01-31T20:53:29.149+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.151254+0000) 2022-01-31T20:53:29.680 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[41853]: debug 2022-01-31T20:53:29.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.558026+0000) 2022-01-31T20:53:29.680 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:29 smithi171 conmon[51620]: debug 2022-01-31T20:53:29.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.428704+0000) 2022-01-31T20:53:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:29 smithi167 conmon[49112]: debug 2022-01-31T20:53:29.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:29.649816+0000) 2022-01-31T20:53:30.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:30 smithi167 conmon[54076]: debug 2022-01-31T20:53:30.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:30.322984+0000) 2022-01-31T20:53:30.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:30 smithi167 conmon[60316]: debug 2022-01-31T20:53:30.073+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:30.074150+0000) 2022-01-31T20:53:30.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:30 smithi171 conmon[46715]: debug 2022-01-31T20:53:30.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:30.352690+0000) 2022-01-31T20:53:30.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:30 smithi171 conmon[51620]: debug 2022-01-31T20:53:30.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:30.428866+0000) 2022-01-31T20:53:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:30 smithi171 conmon[41853]: debug 2022-01-31T20:53:30.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:30.558118+0000) 2022-01-31T20:53:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:30 smithi167 conmon[49112]: debug 2022-01-31T20:53:30.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:30.649940+0000) 2022-01-31T20:53:31.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:31 smithi167 conmon[60316]: debug 2022-01-31T20:53:31.073+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:31.074280+0000) 2022-01-31T20:53:31.367 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:31 smithi167 conmon[54076]: debug 2022-01-31T20:53:31.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:31.323141+0000) 2022-01-31T20:53:31.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:31 smithi171 conmon[46715]: debug 2022-01-31T20:53:31.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:31.352892+0000) 2022-01-31T20:53:31.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:31 smithi171 conmon[51620]: debug 2022-01-31T20:53:31.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:31.429027+0000) 2022-01-31T20:53:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:31 smithi171 conmon[41853]: debug 2022-01-31T20:53:31.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:31.558275+0000) 2022-01-31T20:53:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:31 smithi167 conmon[49112]: debug 2022-01-31T20:53:31.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:31.650138+0000) 2022-01-31T20:53:32.367 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:32 smithi167 conmon[54076]: debug 2022-01-31T20:53:32.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:32.323330+0000) 2022-01-31T20:53:32.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:32 smithi167 conmon[60316]: debug 2022-01-31T20:53:32.073+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:32.074457+0000) 2022-01-31T20:53:32.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:32 smithi171 conmon[46715]: debug 2022-01-31T20:53:32.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:32.353070+0000) 2022-01-31T20:53:32.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:32 smithi171 conmon[51620]: debug 2022-01-31T20:53:32.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:32.429208+0000) 2022-01-31T20:53:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:32 smithi171 conmon[41853]: debug 2022-01-31T20:53:32.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:32.558483+0000) 2022-01-31T20:53:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:32 smithi167 conmon[49112]: debug 2022-01-31T20:53:32.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:32.650339+0000) 2022-01-31T20:53:33.367 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:33 smithi167 conmon[54076]: debug 2022-01-31T20:53:33.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:33.323539+0000) 2022-01-31T20:53:33.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:33 smithi167 conmon[60316]: debug 2022-01-31T20:53:33.074+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:33.074679+0000) 2022-01-31T20:53:33.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:33 smithi171 conmon[46715]: debug 2022-01-31T20:53:33.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:33.353283+0000) 2022-01-31T20:53:33.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:33 smithi171 conmon[51620]: debug 2022-01-31T20:53:33.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:33.429358+0000) 2022-01-31T20:53:33.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:33 smithi171 conmon[41853]: debug 2022-01-31T20:53:33.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:33.558695+0000) 2022-01-31T20:53:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:33 smithi167 conmon[49112]: debug 2022-01-31T20:53:33.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:33.650533+0000) 2022-01-31T20:53:34.367 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:34 smithi167 conmon[49112]: debug 2022-01-31T20:53:34.163+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.164454+0000) 2022-01-31T20:53:34.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:34 smithi167 conmon[60316]: debug 2022-01-31T20:53:34.074+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.074822+0000) 2022-01-31T20:53:34.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:34 smithi167 conmon[60316]: debug 2022-01-31T20:53:34.162+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.163296+0000) 2022-01-31T20:53:34.368 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:34 smithi167 conmon[54076]: debug 2022-01-31T20:53:34.163+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.163803+0000) 2022-01-31T20:53:34.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:34 smithi167 conmon[54076]: debug 2022-01-31T20:53:34.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.323750+0000) 2022-01-31T20:53:34.428 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[35325]: debug 2022-01-31T20:53:34.170+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86546 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:34.428 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[41853]: debug 2022-01-31T20:53:34.162+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.164380+0000) 2022-01-31T20:53:34.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[51620]: debug 2022-01-31T20:53:34.162+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.164233+0000) 2022-01-31T20:53:34.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[46715]: debug 2022-01-31T20:53:34.161+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.162964+0000) 2022-01-31T20:53:34.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[46715]: debug 2022-01-31T20:53:34.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.353468+0000) 2022-01-31T20:53:34.680 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[41853]: debug 2022-01-31T20:53:34.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.558872+0000) 2022-01-31T20:53:34.681 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:34 smithi171 conmon[51620]: debug 2022-01-31T20:53:34.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.429589+0000) 2022-01-31T20:53:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:34 smithi167 conmon[49112]: debug 2022-01-31T20:53:34.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:34.650707+0000) 2022-01-31T20:53:35.367 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:35 smithi167 conmon[54076]: debug 2022-01-31T20:53:35.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:35.323913+0000) 2022-01-31T20:53:35.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:35 smithi167 conmon[60316]: debug 2022-01-31T20:53:35.074+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:35.075031+0000) 2022-01-31T20:53:35.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:35 smithi171 conmon[46715]: debug 2022-01-31T20:53:35.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:35.353588+0000) 2022-01-31T20:53:35.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:35 smithi171 conmon[51620]: debug 2022-01-31T20:53:35.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:35.429749+0000) 2022-01-31T20:53:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:35 smithi171 conmon[41853]: debug 2022-01-31T20:53:35.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:35.559034+0000) 2022-01-31T20:53:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:35 smithi167 conmon[49112]: debug 2022-01-31T20:53:35.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:35.650803+0000) 2022-01-31T20:53:36.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:36 smithi167 conmon[60316]: debug 2022-01-31T20:53:36.074+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:36.075191+0000) 2022-01-31T20:53:36.368 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:36 smithi167 conmon[54076]: debug 2022-01-31T20:53:36.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:36.324118+0000) 2022-01-31T20:53:36.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:36 smithi171 conmon[46715]: debug 2022-01-31T20:53:36.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:36.353773+0000) 2022-01-31T20:53:36.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:36 smithi171 conmon[51620]: debug 2022-01-31T20:53:36.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:36.429924+0000) 2022-01-31T20:53:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:36 smithi171 conmon[41853]: debug 2022-01-31T20:53:36.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:36.559217+0000) 2022-01-31T20:53:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:36 smithi167 conmon[49112]: debug 2022-01-31T20:53:36.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:36.650974+0000) 2022-01-31T20:53:37.368 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:37 smithi167 conmon[54076]: debug 2022-01-31T20:53:37.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:37.324296+0000) 2022-01-31T20:53:37.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:37 smithi167 conmon[60316]: debug 2022-01-31T20:53:37.074+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:37.075423+0000) 2022-01-31T20:53:37.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:37 smithi171 conmon[46715]: debug 2022-01-31T20:53:37.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:37.353951+0000) 2022-01-31T20:53:37.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:37 smithi171 conmon[51620]: debug 2022-01-31T20:53:37.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:37.430081+0000) 2022-01-31T20:53:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:37 smithi171 conmon[41853]: debug 2022-01-31T20:53:37.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:37.559370+0000) 2022-01-31T20:53:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:37 smithi167 conmon[49112]: debug 2022-01-31T20:53:37.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:37.651145+0000) 2022-01-31T20:53:38.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:38 smithi167 conmon[60316]: debug 2022-01-31T20:53:38.075+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:38.075647+0000) 2022-01-31T20:53:38.368 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:38 smithi167 conmon[54076]: debug 2022-01-31T20:53:38.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:38.324469+0000) 2022-01-31T20:53:38.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:38 smithi171 conmon[46715]: debug 2022-01-31T20:53:38.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:38.354117+0000) 2022-01-31T20:53:38.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:38 smithi171 conmon[51620]: debug 2022-01-31T20:53:38.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:38.430286+0000) 2022-01-31T20:53:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:38 smithi171 conmon[41853]: debug 2022-01-31T20:53:38.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:38.559549+0000) 2022-01-31T20:53:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:38 smithi171 conmon[35325]: debug 2022-01-31T20:53:38.588+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:53:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:38 smithi167 conmon[49112]: debug 2022-01-31T20:53:38.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:38.651306+0000) 2022-01-31T20:53:39.368 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:39 smithi167 conmon[49112]: debug 2022-01-31T20:53:39.174+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.175255+0000) 2022-01-31T20:53:39.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:39 smithi167 conmon[54076]: debug 2022-01-31T20:53:39.174+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.174998+0000) 2022-01-31T20:53:39.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:39 smithi167 conmon[54076]: debug 2022-01-31T20:53:39.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.324706+0000) 2022-01-31T20:53:39.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:39 smithi167 conmon[60316]: debug 2022-01-31T20:53:39.075+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.075828+0000) 2022-01-31T20:53:39.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:39 smithi167 conmon[60316]: debug 2022-01-31T20:53:39.175+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.175629+0000) 2022-01-31T20:53:39.428 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[35325]: debug 2022-01-31T20:53:39.182+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86655 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:39.429 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[41853]: debug 2022-01-31T20:53:39.173+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.174007+0000) 2022-01-31T20:53:39.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[51620]: debug 2022-01-31T20:53:39.173+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.174724+0000) 2022-01-31T20:53:39.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[46715]: debug 2022-01-31T20:53:39.173+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.174497+0000) 2022-01-31T20:53:39.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[46715]: debug 2022-01-31T20:53:39.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.354283+0000) 2022-01-31T20:53:39.682 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[51620]: debug 2022-01-31T20:53:39.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.430459+0000) 2022-01-31T20:53:39.682 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:39 smithi171 conmon[41853]: debug 2022-01-31T20:53:39.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.559774+0000) 2022-01-31T20:53:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:39 smithi167 conmon[49112]: debug 2022-01-31T20:53:39.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:39.651522+0000) 2022-01-31T20:53:40.368 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:40 smithi167 conmon[54076]: debug 2022-01-31T20:53:40.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:40.324852+0000) 2022-01-31T20:53:40.369 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:40 smithi167 conmon[60316]: debug 2022-01-31T20:53:40.075+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:40.075998+0000) 2022-01-31T20:53:40.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:40 smithi171 conmon[46715]: debug 2022-01-31T20:53:40.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:40.354436+0000) 2022-01-31T20:53:40.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:40 smithi171 conmon[51620]: debug 2022-01-31T20:53:40.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:40.430633+0000) 2022-01-31T20:53:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:40 smithi171 conmon[41853]: debug 2022-01-31T20:53:40.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:40.559913+0000) 2022-01-31T20:53:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:40 smithi167 conmon[49112]: debug 2022-01-31T20:53:40.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:40.651626+0000) 2022-01-31T20:53:41.368 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:41 smithi167 conmon[54076]: debug 2022-01-31T20:53:41.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:41.325051+0000) 2022-01-31T20:53:41.369 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:41 smithi167 conmon[60316]: debug 2022-01-31T20:53:41.075+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:41.076166+0000) 2022-01-31T20:53:41.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:41 smithi171 conmon[46715]: debug 2022-01-31T20:53:41.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:41.354627+0000) 2022-01-31T20:53:41.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:41 smithi171 conmon[51620]: debug 2022-01-31T20:53:41.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:41.430814+0000) 2022-01-31T20:53:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:41 smithi171 conmon[41853]: debug 2022-01-31T20:53:41.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:41.560102+0000) 2022-01-31T20:53:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:41 smithi167 conmon[49112]: debug 2022-01-31T20:53:41.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:41.651732+0000) 2022-01-31T20:53:42.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:42 smithi167 conmon[54076]: debug 2022-01-31T20:53:42.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:42.325208+0000) 2022-01-31T20:53:42.369 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:42 smithi167 conmon[60316]: debug 2022-01-31T20:53:42.075+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:42.076375+0000) 2022-01-31T20:53:42.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:42 smithi171 conmon[46715]: debug 2022-01-31T20:53:42.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:42.354841+0000) 2022-01-31T20:53:42.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:42 smithi171 conmon[51620]: debug 2022-01-31T20:53:42.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:42.431005+0000) 2022-01-31T20:53:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:42 smithi171 conmon[41853]: debug 2022-01-31T20:53:42.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:42.560326+0000) 2022-01-31T20:53:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:42 smithi167 conmon[49112]: debug 2022-01-31T20:53:42.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:42.651862+0000) 2022-01-31T20:53:43.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:43 smithi167 conmon[54076]: debug 2022-01-31T20:53:43.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:43.325370+0000) 2022-01-31T20:53:43.369 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:43 smithi167 conmon[60316]: debug 2022-01-31T20:53:43.076+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:43.076525+0000) 2022-01-31T20:53:43.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:43 smithi171 conmon[46715]: debug 2022-01-31T20:53:43.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:43.355007+0000) 2022-01-31T20:53:43.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:43 smithi171 conmon[51620]: debug 2022-01-31T20:53:43.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:43.431207+0000) 2022-01-31T20:53:43.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:43 smithi171 conmon[41853]: debug 2022-01-31T20:53:43.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:43.560531+0000) 2022-01-31T20:53:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:43 smithi167 conmon[49112]: debug 2022-01-31T20:53:43.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:43.652035+0000) 2022-01-31T20:53:44.369 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:44 smithi167 conmon[49112]: debug 2022-01-31T20:53:44.185+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.186238+0000) 2022-01-31T20:53:44.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:44 smithi167 conmon[60316]: debug 2022-01-31T20:53:44.076+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.076699+0000) 2022-01-31T20:53:44.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:44 smithi167 conmon[60316]: debug 2022-01-31T20:53:44.185+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.186410+0000) 2022-01-31T20:53:44.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:44 smithi167 conmon[54076]: debug 2022-01-31T20:53:44.185+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.185765+0000) 2022-01-31T20:53:44.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:44 smithi167 conmon[54076]: debug 2022-01-31T20:53:44.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.325545+0000) 2022-01-31T20:53:44.559 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[35325]: debug 2022-01-31T20:53:44.194+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86765 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:44.560 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[41853]: debug 2022-01-31T20:53:44.184+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.185614+0000) 2022-01-31T20:53:44.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[46715]: debug 2022-01-31T20:53:44.184+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.185413+0000) 2022-01-31T20:53:44.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[46715]: debug 2022-01-31T20:53:44.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.355166+0000) 2022-01-31T20:53:44.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[51620]: debug 2022-01-31T20:53:44.184+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.185382+0000) 2022-01-31T20:53:44.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[51620]: debug 2022-01-31T20:53:44.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.431384+0000) 2022-01-31T20:53:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:44 smithi171 conmon[41853]: debug 2022-01-31T20:53:44.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.560707+0000) 2022-01-31T20:53:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:44 smithi167 conmon[49112]: debug 2022-01-31T20:53:44.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:44.652249+0000) 2022-01-31T20:53:45.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:45 smithi167 conmon[54076]: debug 2022-01-31T20:53:45.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:45.325704+0000) 2022-01-31T20:53:45.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:45 smithi167 conmon[60316]: debug 2022-01-31T20:53:45.076+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:45.076886+0000) 2022-01-31T20:53:45.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:45 smithi171 conmon[46715]: debug 2022-01-31T20:53:45.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:45.355313+0000) 2022-01-31T20:53:45.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:45 smithi171 conmon[51620]: debug 2022-01-31T20:53:45.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:45.431540+0000) 2022-01-31T20:53:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:45 smithi171 conmon[41853]: debug 2022-01-31T20:53:45.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:45.560841+0000) 2022-01-31T20:53:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:45 smithi167 conmon[49112]: debug 2022-01-31T20:53:45.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:45.652392+0000) 2022-01-31T20:53:46.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:46 smithi167 conmon[54076]: debug 2022-01-31T20:53:46.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:46.325805+0000) 2022-01-31T20:53:46.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:46 smithi167 conmon[60316]: debug 2022-01-31T20:53:46.076+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:46.077069+0000) 2022-01-31T20:53:46.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:46 smithi171 conmon[46715]: debug 2022-01-31T20:53:46.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:46.355498+0000) 2022-01-31T20:53:46.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:46 smithi171 conmon[51620]: debug 2022-01-31T20:53:46.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:46.431734+0000) 2022-01-31T20:53:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:46 smithi171 conmon[41853]: debug 2022-01-31T20:53:46.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:46.560965+0000) 2022-01-31T20:53:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:46 smithi167 conmon[49112]: debug 2022-01-31T20:53:46.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:46.652538+0000) 2022-01-31T20:53:47.370 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:47 smithi167 conmon[54076]: debug 2022-01-31T20:53:47.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:47.325954+0000) 2022-01-31T20:53:47.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:47 smithi167 conmon[60316]: debug 2022-01-31T20:53:47.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:47.077251+0000) 2022-01-31T20:53:47.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:47 smithi171 conmon[46715]: debug 2022-01-31T20:53:47.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:47.355656+0000) 2022-01-31T20:53:47.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:47 smithi171 conmon[51620]: debug 2022-01-31T20:53:47.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:47.431898+0000) 2022-01-31T20:53:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:47 smithi171 conmon[41853]: debug 2022-01-31T20:53:47.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:47.561098+0000) 2022-01-31T20:53:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:47 smithi167 conmon[49112]: debug 2022-01-31T20:53:47.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:47.652671+0000) 2022-01-31T20:53:48.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:48 smithi167 conmon[60316]: debug 2022-01-31T20:53:48.076+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:48.077460+0000) 2022-01-31T20:53:48.370 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:48 smithi167 conmon[54076]: debug 2022-01-31T20:53:48.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:48.326142+0000) 2022-01-31T20:53:48.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:48 smithi171 conmon[46715]: debug 2022-01-31T20:53:48.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:48.355858+0000) 2022-01-31T20:53:48.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:48 smithi171 conmon[51620]: debug 2022-01-31T20:53:48.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:48.432071+0000) 2022-01-31T20:53:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:48 smithi171 conmon[41853]: debug 2022-01-31T20:53:48.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:48.561256+0000) 2022-01-31T20:53:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:48 smithi167 conmon[49112]: debug 2022-01-31T20:53:48.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:48.652816+0000) 2022-01-31T20:53:49.370 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:49 smithi167 conmon[49112]: debug 2022-01-31T20:53:49.197+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.197928+0000) 2022-01-31T20:53:49.370 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:49 smithi167 conmon[54076]: debug 2022-01-31T20:53:49.197+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.198370+0000) 2022-01-31T20:53:49.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:49 smithi167 conmon[54076]: debug 2022-01-31T20:53:49.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.326350+0000) 2022-01-31T20:53:49.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:49 smithi167 conmon[60316]: debug 2022-01-31T20:53:49.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.077618+0000) 2022-01-31T20:53:49.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:49 smithi167 conmon[60316]: debug 2022-01-31T20:53:49.198+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.198818+0000) 2022-01-31T20:53:49.560 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[35325]: debug 2022-01-31T20:53:49.205+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86874 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:49.560 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[41853]: debug 2022-01-31T20:53:49.196+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.196861+0000) 2022-01-31T20:53:49.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[46715]: debug 2022-01-31T20:53:49.196+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.197041+0000) 2022-01-31T20:53:49.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[46715]: debug 2022-01-31T20:53:49.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.356001+0000) 2022-01-31T20:53:49.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[51620]: debug 2022-01-31T20:53:49.197+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.197875+0000) 2022-01-31T20:53:49.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[51620]: debug 2022-01-31T20:53:49.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.432264+0000) 2022-01-31T20:53:49.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:49 smithi171 conmon[41853]: debug 2022-01-31T20:53:49.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.561385+0000) 2022-01-31T20:53:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:49 smithi167 conmon[49112]: debug 2022-01-31T20:53:49.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:49.652989+0000) 2022-01-31T20:53:50.370 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:50 smithi167 conmon[54076]: debug 2022-01-31T20:53:50.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:50.326541+0000) 2022-01-31T20:53:50.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:50 smithi167 conmon[60316]: debug 2022-01-31T20:53:50.076+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:50.077843+0000) 2022-01-31T20:53:50.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:50 smithi171 conmon[46715]: debug 2022-01-31T20:53:50.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:50.356151+0000) 2022-01-31T20:53:50.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:50 smithi171 conmon[51620]: debug 2022-01-31T20:53:50.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:50.432392+0000) 2022-01-31T20:53:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:50 smithi171 conmon[41853]: debug 2022-01-31T20:53:50.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:50.561543+0000) 2022-01-31T20:53:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:50 smithi167 conmon[49112]: debug 2022-01-31T20:53:50.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:50.653181+0000) 2022-01-31T20:53:51.370 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:51 smithi167 conmon[54076]: debug 2022-01-31T20:53:51.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:51.326728+0000) 2022-01-31T20:53:51.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:51 smithi167 conmon[60316]: debug 2022-01-31T20:53:51.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:51.078009+0000) 2022-01-31T20:53:51.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:51 smithi171 conmon[46715]: debug 2022-01-31T20:53:51.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:51.356340+0000) 2022-01-31T20:53:51.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:51 smithi171 conmon[51620]: debug 2022-01-31T20:53:51.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:51.432571+0000) 2022-01-31T20:53:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:51 smithi171 conmon[41853]: debug 2022-01-31T20:53:51.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:51.561742+0000) 2022-01-31T20:53:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:51 smithi167 conmon[49112]: debug 2022-01-31T20:53:51.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:51.653396+0000) 2022-01-31T20:53:52.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:52 smithi167 conmon[54076]: debug 2022-01-31T20:53:52.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:52.326939+0000) 2022-01-31T20:53:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:52 smithi167 conmon[60316]: debug 2022-01-31T20:53:52.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:52.078194+0000) 2022-01-31T20:53:52.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:52 smithi171 conmon[46715]: debug 2022-01-31T20:53:52.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:52.356570+0000) 2022-01-31T20:53:52.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:52 smithi171 conmon[51620]: debug 2022-01-31T20:53:52.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:52.432849+0000) 2022-01-31T20:53:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:52 smithi171 conmon[41853]: debug 2022-01-31T20:53:52.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:52.561942+0000) 2022-01-31T20:53:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:52 smithi167 conmon[49112]: debug 2022-01-31T20:53:52.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:52.653580+0000) 2022-01-31T20:53:53.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:53 smithi167 conmon[60316]: debug 2022-01-31T20:53:53.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:53.078368+0000) 2022-01-31T20:53:53.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:53 smithi167 conmon[54076]: debug 2022-01-31T20:53:53.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:53.327127+0000) 2022-01-31T20:53:53.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:53 smithi171 conmon[46715]: debug 2022-01-31T20:53:53.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:53.356766+0000) 2022-01-31T20:53:53.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:53 smithi171 conmon[51620]: debug 2022-01-31T20:53:53.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:53.433024+0000) 2022-01-31T20:53:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:53 smithi171 conmon[35325]: debug 2022-01-31T20:53:53.589+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:53:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:53 smithi171 conmon[41853]: debug 2022-01-31T20:53:53.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:53.562107+0000) 2022-01-31T20:53:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:53 smithi167 conmon[49112]: debug 2022-01-31T20:53:53.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:53.653747+0000) 2022-01-31T20:53:54.371 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:54 smithi167 conmon[49112]: debug 2022-01-31T20:53:54.207+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.208785+0000) 2022-01-31T20:53:54.372 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:54 smithi167 conmon[60316]: debug 2022-01-31T20:53:54.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.078579+0000) 2022-01-31T20:53:54.372 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:54 smithi167 conmon[60316]: debug 2022-01-31T20:53:54.207+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.209434+0000) 2022-01-31T20:53:54.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:54 smithi167 conmon[54076]: debug 2022-01-31T20:53:54.207+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.209349+0000) 2022-01-31T20:53:54.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:54 smithi167 conmon[54076]: debug 2022-01-31T20:53:54.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.327283+0000) 2022-01-31T20:53:54.560 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[35325]: debug 2022-01-31T20:53:54.216+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 86984 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:54.561 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[41853]: debug 2022-01-31T20:53:54.207+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.208583+0000) 2022-01-31T20:53:54.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[51620]: debug 2022-01-31T20:53:54.207+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.208312+0000) 2022-01-31T20:53:54.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[51620]: debug 2022-01-31T20:53:54.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.433196+0000) 2022-01-31T20:53:54.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[46715]: debug 2022-01-31T20:53:54.208+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.208956+0000) 2022-01-31T20:53:54.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[46715]: debug 2022-01-31T20:53:54.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.356931+0000) 2022-01-31T20:53:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:54 smithi171 conmon[41853]: debug 2022-01-31T20:53:54.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.562263+0000) 2022-01-31T20:53:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:54 smithi167 conmon[49112]: debug 2022-01-31T20:53:54.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:54.653871+0000) 2022-01-31T20:53:55.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:55 smithi167 conmon[54076]: debug 2022-01-31T20:53:55.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:55.327413+0000) 2022-01-31T20:53:55.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:55 smithi167 conmon[60316]: debug 2022-01-31T20:53:55.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:55.078727+0000) 2022-01-31T20:53:55.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:55 smithi171 conmon[46715]: debug 2022-01-31T20:53:55.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:55.357051+0000) 2022-01-31T20:53:55.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:55 smithi171 conmon[51620]: debug 2022-01-31T20:53:55.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:55.433315+0000) 2022-01-31T20:53:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:55 smithi171 conmon[41853]: debug 2022-01-31T20:53:55.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:55.562434+0000) 2022-01-31T20:53:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:55 smithi167 conmon[49112]: debug 2022-01-31T20:53:55.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:55.654024+0000) 2022-01-31T20:53:56.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:56 smithi167 conmon[60316]: debug 2022-01-31T20:53:56.077+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:56.078863+0000) 2022-01-31T20:53:56.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:56 smithi167 conmon[54076]: debug 2022-01-31T20:53:56.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:56.327603+0000) 2022-01-31T20:53:56.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:56 smithi171 conmon[46715]: debug 2022-01-31T20:53:56.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:56.357208+0000) 2022-01-31T20:53:56.456 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:56 smithi171 conmon[51620]: debug 2022-01-31T20:53:56.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:56.433512+0000) 2022-01-31T20:53:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:56 smithi171 conmon[41853]: debug 2022-01-31T20:53:56.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:56.562614+0000) 2022-01-31T20:53:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:56 smithi167 conmon[49112]: debug 2022-01-31T20:53:56.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:56.654227+0000) 2022-01-31T20:53:57.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:57 smithi167 conmon[54076]: debug 2022-01-31T20:53:57.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:57.327777+0000) 2022-01-31T20:53:57.372 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:57 smithi167 conmon[60316]: debug 2022-01-31T20:53:57.078+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:57.079046+0000) 2022-01-31T20:53:57.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:57 smithi171 conmon[46715]: debug 2022-01-31T20:53:57.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:57.357376+0000) 2022-01-31T20:53:57.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:57 smithi171 conmon[51620]: debug 2022-01-31T20:53:57.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:57.433713+0000) 2022-01-31T20:53:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:57 smithi171 conmon[41853]: debug 2022-01-31T20:53:57.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:57.562791+0000) 2022-01-31T20:53:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:57 smithi167 conmon[49112]: debug 2022-01-31T20:53:57.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:57.654385+0000) 2022-01-31T20:53:58.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:58 smithi167 conmon[60316]: debug 2022-01-31T20:53:58.078+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:58.079201+0000) 2022-01-31T20:53:58.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:58 smithi171 conmon[46715]: debug 2022-01-31T20:53:58.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:58.357567+0000) 2022-01-31T20:53:58.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:58 smithi171 conmon[51620]: debug 2022-01-31T20:53:58.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:58.433933+0000) 2022-01-31T20:53:58.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:58 smithi167 conmon[54076]: debug 2022-01-31T20:53:58.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:58.327903+0000) 2022-01-31T20:53:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:58 smithi171 conmon[41853]: debug 2022-01-31T20:53:58.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:58.562972+0000) 2022-01-31T20:53:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:58 smithi167 conmon[49112]: debug 2022-01-31T20:53:58.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:58.654585+0000) 2022-01-31T20:53:59.372 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:59 smithi167 conmon[49112]: debug 2022-01-31T20:53:59.219+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.220705+0000) 2022-01-31T20:53:59.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:59 smithi167 conmon[54076]: debug 2022-01-31T20:53:59.219+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.220520+0000) 2022-01-31T20:53:59.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:53:59 smithi167 conmon[54076]: debug 2022-01-31T20:53:59.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.328040+0000) 2022-01-31T20:53:59.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:59 smithi167 conmon[60316]: debug 2022-01-31T20:53:59.078+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.079387+0000) 2022-01-31T20:53:59.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:53:59 smithi167 conmon[60316]: debug 2022-01-31T20:53:59.218+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.220202+0000) 2022-01-31T20:53:59.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[41853]: debug 2022-01-31T20:53:59.219+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.219954+0000) 2022-01-31T20:53:59.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[46715]: debug 2022-01-31T20:53:59.218+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.219734+0000) 2022-01-31T20:53:59.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[46715]: debug 2022-01-31T20:53:59.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.357747+0000) 2022-01-31T20:53:59.563 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[35325]: debug 2022-01-31T20:53:59.228+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87093 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:53:59.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[51620]: debug 2022-01-31T20:53:59.219+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.220410+0000) 2022-01-31T20:53:59.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[51620]: debug 2022-01-31T20:53:59.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.434088+0000) 2022-01-31T20:53:59.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:53:59 smithi171 conmon[41853]: debug 2022-01-31T20:53:59.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.563154+0000) 2022-01-31T20:53:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:53:59 smithi167 conmon[49112]: debug 2022-01-31T20:53:59.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:53:59.654739+0000) 2022-01-31T20:54:00.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:00 smithi167 conmon[54076]: debug 2022-01-31T20:54:00.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:00.328169+0000) 2022-01-31T20:54:00.372 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:00 smithi167 conmon[60316]: debug 2022-01-31T20:54:00.078+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:00.079569+0000) 2022-01-31T20:54:00.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:00 smithi171 conmon[46715]: debug 2022-01-31T20:54:00.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:00.357924+0000) 2022-01-31T20:54:00.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:00 smithi171 conmon[51620]: debug 2022-01-31T20:54:00.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:00.434191+0000) 2022-01-31T20:54:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:00 smithi171 conmon[41853]: debug 2022-01-31T20:54:00.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:00.563321+0000) 2022-01-31T20:54:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:00 smithi167 conmon[49112]: debug 2022-01-31T20:54:00.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:00.654849+0000) 2022-01-31T20:54:01.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:01 smithi167 conmon[54076]: debug 2022-01-31T20:54:01.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:01.328294+0000) 2022-01-31T20:54:01.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:01 smithi167 conmon[60316]: debug 2022-01-31T20:54:01.078+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:01.079740+0000) 2022-01-31T20:54:01.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:01 smithi171 conmon[46715]: debug 2022-01-31T20:54:01.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:01.358146+0000) 2022-01-31T20:54:01.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:01 smithi171 conmon[51620]: debug 2022-01-31T20:54:01.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:01.434377+0000) 2022-01-31T20:54:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:01 smithi171 conmon[41853]: debug 2022-01-31T20:54:01.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:01.563523+0000) 2022-01-31T20:54:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:01 smithi167 conmon[49112]: debug 2022-01-31T20:54:01.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:01.655011+0000) 2022-01-31T20:54:02.372 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:02 smithi167 conmon[60316]: debug 2022-01-31T20:54:02.079+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:02.079937+0000) 2022-01-31T20:54:02.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:02 smithi167 conmon[54076]: debug 2022-01-31T20:54:02.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:02.328476+0000) 2022-01-31T20:54:02.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:02 smithi171 conmon[46715]: debug 2022-01-31T20:54:02.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:02.358278+0000) 2022-01-31T20:54:02.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:02 smithi171 conmon[51620]: debug 2022-01-31T20:54:02.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:02.434583+0000) 2022-01-31T20:54:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:02 smithi171 conmon[41853]: debug 2022-01-31T20:54:02.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:02.563770+0000) 2022-01-31T20:54:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:02 smithi167 conmon[49112]: debug 2022-01-31T20:54:02.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:02.655256+0000) 2022-01-31T20:54:03.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:03 smithi167 conmon[54076]: debug 2022-01-31T20:54:03.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:03.328683+0000) 2022-01-31T20:54:03.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:03 smithi167 conmon[60316]: debug 2022-01-31T20:54:03.079+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:03.080119+0000) 2022-01-31T20:54:03.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:03 smithi171 conmon[46715]: debug 2022-01-31T20:54:03.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:03.358459+0000) 2022-01-31T20:54:03.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:03 smithi171 conmon[51620]: debug 2022-01-31T20:54:03.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:03.434779+0000) 2022-01-31T20:54:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:03 smithi171 conmon[41853]: debug 2022-01-31T20:54:03.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:03.563934+0000) 2022-01-31T20:54:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:03 smithi167 conmon[49112]: debug 2022-01-31T20:54:03.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:03.655377+0000) 2022-01-31T20:54:04.373 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:04 smithi167 conmon[49112]: debug 2022-01-31T20:54:04.231+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.232739+0000) 2022-01-31T20:54:04.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:04 smithi167 conmon[54076]: debug 2022-01-31T20:54:04.231+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.232576+0000) 2022-01-31T20:54:04.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:04 smithi167 conmon[54076]: debug 2022-01-31T20:54:04.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.328816+0000) 2022-01-31T20:54:04.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:04 smithi167 conmon[60316]: debug 2022-01-31T20:54:04.079+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.080319+0000) 2022-01-31T20:54:04.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:04 smithi167 conmon[60316]: debug 2022-01-31T20:54:04.230+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.231648+0000) 2022-01-31T20:54:04.563 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[35325]: debug 2022-01-31T20:54:04.239+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87203 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:04.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[41853]: debug 2022-01-31T20:54:04.231+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.232106+0000) 2022-01-31T20:54:04.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[46715]: debug 2022-01-31T20:54:04.231+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.232494+0000) 2022-01-31T20:54:04.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[46715]: debug 2022-01-31T20:54:04.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.358614+0000) 2022-01-31T20:54:04.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[51620]: debug 2022-01-31T20:54:04.231+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.231960+0000) 2022-01-31T20:54:04.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[51620]: debug 2022-01-31T20:54:04.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.434967+0000) 2022-01-31T20:54:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:04 smithi171 conmon[41853]: debug 2022-01-31T20:54:04.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.564076+0000) 2022-01-31T20:54:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:04 smithi167 conmon[49112]: debug 2022-01-31T20:54:04.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:04.655572+0000) 2022-01-31T20:54:05.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:05 smithi167 conmon[60316]: debug 2022-01-31T20:54:05.079+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:05.080529+0000) 2022-01-31T20:54:05.339 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:05 smithi167 conmon[54076]: debug 2022-01-31T20:54:05.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:05.328966+0000) 2022-01-31T20:54:05.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:05 smithi171 conmon[46715]: debug 2022-01-31T20:54:05.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:05.358794+0000) 2022-01-31T20:54:05.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:05 smithi171 conmon[51620]: debug 2022-01-31T20:54:05.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:05.435137+0000) 2022-01-31T20:54:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:05 smithi171 conmon[41853]: debug 2022-01-31T20:54:05.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:05.564167+0000) 2022-01-31T20:54:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:05 smithi167 conmon[49112]: debug 2022-01-31T20:54:05.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:05.655673+0000) 2022-01-31T20:54:06.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:06 smithi167 conmon[54076]: debug 2022-01-31T20:54:06.328+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:06.329117+0000) 2022-01-31T20:54:06.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:06 smithi167 conmon[60316]: debug 2022-01-31T20:54:06.080+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:06.080702+0000) 2022-01-31T20:54:06.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:06 smithi171 conmon[51620]: debug 2022-01-31T20:54:06.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:06.435323+0000) 2022-01-31T20:54:06.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:06 smithi171 conmon[46715]: debug 2022-01-31T20:54:06.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:06.358923+0000) 2022-01-31T20:54:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:06 smithi171 conmon[41853]: debug 2022-01-31T20:54:06.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:06.564328+0000) 2022-01-31T20:54:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:06 smithi167 conmon[49112]: debug 2022-01-31T20:54:06.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:06.655829+0000) 2022-01-31T20:54:07.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:07 smithi167 conmon[54076]: debug 2022-01-31T20:54:07.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:07.329231+0000) 2022-01-31T20:54:07.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:07 smithi167 conmon[60316]: debug 2022-01-31T20:54:07.080+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:07.080909+0000) 2022-01-31T20:54:07.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:07 smithi171 conmon[51620]: debug 2022-01-31T20:54:07.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:07.435452+0000) 2022-01-31T20:54:07.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:07 smithi171 conmon[46715]: debug 2022-01-31T20:54:07.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:07.359076+0000) 2022-01-31T20:54:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:07 smithi171 conmon[41853]: debug 2022-01-31T20:54:07.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:07.564489+0000) 2022-01-31T20:54:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:07 smithi167 conmon[49112]: debug 2022-01-31T20:54:07.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:07.655971+0000) 2022-01-31T20:54:08.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:08 smithi167 conmon[60316]: debug 2022-01-31T20:54:08.080+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:08.081121+0000) 2022-01-31T20:54:08.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:08 smithi167 conmon[54076]: debug 2022-01-31T20:54:08.328+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:08.329456+0000) 2022-01-31T20:54:08.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:08 smithi171 conmon[46715]: debug 2022-01-31T20:54:08.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:08.359275+0000) 2022-01-31T20:54:08.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:08 smithi171 conmon[51620]: debug 2022-01-31T20:54:08.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:08.435626+0000) 2022-01-31T20:54:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:08 smithi171 conmon[41853]: debug 2022-01-31T20:54:08.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:08.564646+0000) 2022-01-31T20:54:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:08 smithi171 conmon[35325]: debug 2022-01-31T20:54:08.590+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:54:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:08 smithi167 conmon[49112]: debug 2022-01-31T20:54:08.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:08.656177+0000) 2022-01-31T20:54:09.374 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:09 smithi167 conmon[49112]: debug 2022-01-31T20:54:09.241+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.243449+0000) 2022-01-31T20:54:09.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:09 smithi167 conmon[54076]: debug 2022-01-31T20:54:09.241+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.243040+0000) 2022-01-31T20:54:09.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:09 smithi167 conmon[54076]: debug 2022-01-31T20:54:09.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.329681+0000) 2022-01-31T20:54:09.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:09 smithi167 conmon[60316]: debug 2022-01-31T20:54:09.079+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.081356+0000) 2022-01-31T20:54:09.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:09 smithi167 conmon[60316]: debug 2022-01-31T20:54:09.241+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.243341+0000) 2022-01-31T20:54:09.563 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[35325]: debug 2022-01-31T20:54:09.251+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87312 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:09.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[41853]: debug 2022-01-31T20:54:09.243+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.243962+0000) 2022-01-31T20:54:09.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[46715]: debug 2022-01-31T20:54:09.242+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.243319+0000) 2022-01-31T20:54:09.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[46715]: debug 2022-01-31T20:54:09.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.359442+0000) 2022-01-31T20:54:09.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[51620]: debug 2022-01-31T20:54:09.241+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.242495+0000) 2022-01-31T20:54:09.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[51620]: debug 2022-01-31T20:54:09.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.435813+0000) 2022-01-31T20:54:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:09 smithi171 conmon[41853]: debug 2022-01-31T20:54:09.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.564801+0000) 2022-01-31T20:54:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:09 smithi167 conmon[49112]: debug 2022-01-31T20:54:09.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:09.656359+0000) 2022-01-31T20:54:10.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:10 smithi167 conmon[54076]: debug 2022-01-31T20:54:10.328+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:10.329789+0000) 2022-01-31T20:54:10.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:10 smithi167 conmon[60316]: debug 2022-01-31T20:54:10.080+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:10.081564+0000) 2022-01-31T20:54:10.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:10 smithi171 conmon[46715]: debug 2022-01-31T20:54:10.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:10.359589+0000) 2022-01-31T20:54:10.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:10 smithi171 conmon[51620]: debug 2022-01-31T20:54:10.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:10.435968+0000) 2022-01-31T20:54:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:10 smithi171 conmon[41853]: debug 2022-01-31T20:54:10.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:10.564952+0000) 2022-01-31T20:54:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:10 smithi167 conmon[49112]: debug 2022-01-31T20:54:10.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:10.656545+0000) 2022-01-31T20:54:11.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:11 smithi167 conmon[54076]: debug 2022-01-31T20:54:11.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:11.329977+0000) 2022-01-31T20:54:11.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:11 smithi167 conmon[60316]: debug 2022-01-31T20:54:11.080+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:11.081670+0000) 2022-01-31T20:54:11.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:11 smithi171 conmon[46715]: debug 2022-01-31T20:54:11.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:11.359760+0000) 2022-01-31T20:54:11.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:11 smithi171 conmon[51620]: debug 2022-01-31T20:54:11.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:11.436138+0000) 2022-01-31T20:54:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:11 smithi171 conmon[41853]: debug 2022-01-31T20:54:11.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:11.565132+0000) 2022-01-31T20:54:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:11 smithi167 conmon[49112]: debug 2022-01-31T20:54:11.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:11.656648+0000) 2022-01-31T20:54:12.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:12 smithi167 conmon[54076]: debug 2022-01-31T20:54:12.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:12.330168+0000) 2022-01-31T20:54:12.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:12 smithi167 conmon[60316]: debug 2022-01-31T20:54:12.081+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:12.081878+0000) 2022-01-31T20:54:12.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:12 smithi171 conmon[46715]: debug 2022-01-31T20:54:12.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:12.359921+0000) 2022-01-31T20:54:12.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:12 smithi171 conmon[51620]: debug 2022-01-31T20:54:12.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:12.436263+0000) 2022-01-31T20:54:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:12 smithi171 conmon[41853]: debug 2022-01-31T20:54:12.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:12.565320+0000) 2022-01-31T20:54:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:12 smithi167 conmon[49112]: debug 2022-01-31T20:54:12.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:12.656771+0000) 2022-01-31T20:54:13.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:13 smithi167 conmon[54076]: debug 2022-01-31T20:54:13.328+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:13.330347+0000) 2022-01-31T20:54:13.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:13 smithi167 conmon[60316]: debug 2022-01-31T20:54:13.081+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:13.082073+0000) 2022-01-31T20:54:13.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:13 smithi171 conmon[46715]: debug 2022-01-31T20:54:13.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:13.360045+0000) 2022-01-31T20:54:13.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:13 smithi171 conmon[51620]: debug 2022-01-31T20:54:13.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:13.436437+0000) 2022-01-31T20:54:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:13 smithi171 conmon[41853]: debug 2022-01-31T20:54:13.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:13.565525+0000) 2022-01-31T20:54:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:13 smithi167 conmon[49112]: debug 2022-01-31T20:54:13.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:13.656929+0000) 2022-01-31T20:54:14.375 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:14 smithi167 conmon[49112]: debug 2022-01-31T20:54:14.253+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.254390+0000) 2022-01-31T20:54:14.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:14 smithi167 conmon[54076]: debug 2022-01-31T20:54:14.253+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.253950+0000) 2022-01-31T20:54:14.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:14 smithi167 conmon[54076]: debug 2022-01-31T20:54:14.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.330549+0000) 2022-01-31T20:54:14.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:14 smithi167 conmon[60316]: debug 2022-01-31T20:54:14.081+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.082247+0000) 2022-01-31T20:54:14.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:14 smithi167 conmon[60316]: debug 2022-01-31T20:54:14.254+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.255111+0000) 2022-01-31T20:54:14.564 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[35325]: debug 2022-01-31T20:54:14.261+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87423 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:14.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[41853]: debug 2022-01-31T20:54:14.253+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.255502+0000) 2022-01-31T20:54:14.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[46715]: debug 2022-01-31T20:54:14.252+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.254302+0000) 2022-01-31T20:54:14.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[46715]: debug 2022-01-31T20:54:14.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.360205+0000) 2022-01-31T20:54:14.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[51620]: debug 2022-01-31T20:54:14.253+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.255043+0000) 2022-01-31T20:54:14.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[51620]: debug 2022-01-31T20:54:14.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.436590+0000) 2022-01-31T20:54:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:14 smithi171 conmon[41853]: debug 2022-01-31T20:54:14.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.565692+0000) 2022-01-31T20:54:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:14 smithi167 conmon[49112]: debug 2022-01-31T20:54:14.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:14.657094+0000) 2022-01-31T20:54:15.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:15 smithi167 conmon[60316]: debug 2022-01-31T20:54:15.081+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:15.082432+0000) 2022-01-31T20:54:15.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:15 smithi167 conmon[54076]: debug 2022-01-31T20:54:15.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:15.330646+0000) 2022-01-31T20:54:15.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:15 smithi171 conmon[46715]: debug 2022-01-31T20:54:15.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:15.360371+0000) 2022-01-31T20:54:15.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:15 smithi171 conmon[51620]: debug 2022-01-31T20:54:15.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:15.436680+0000) 2022-01-31T20:54:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:15 smithi171 conmon[41853]: debug 2022-01-31T20:54:15.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:15.565862+0000) 2022-01-31T20:54:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:15 smithi167 conmon[49112]: debug 2022-01-31T20:54:15.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:15.657244+0000) 2022-01-31T20:54:16.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:16 smithi167 conmon[54076]: debug 2022-01-31T20:54:16.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:16.331345+0000) 2022-01-31T20:54:16.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:16 smithi167 conmon[60316]: debug 2022-01-31T20:54:16.082+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:16.082541+0000) 2022-01-31T20:54:16.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:16 smithi171 conmon[46715]: debug 2022-01-31T20:54:16.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:16.360559+0000) 2022-01-31T20:54:16.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:16 smithi171 conmon[51620]: debug 2022-01-31T20:54:16.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:16.436827+0000) 2022-01-31T20:54:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:16 smithi171 conmon[41853]: debug 2022-01-31T20:54:16.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:16.566051+0000) 2022-01-31T20:54:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:16 smithi167 conmon[49112]: debug 2022-01-31T20:54:16.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:16.657420+0000) 2022-01-31T20:54:17.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:17 smithi167 conmon[54076]: debug 2022-01-31T20:54:17.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:17.331535+0000) 2022-01-31T20:54:17.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:17 smithi167 conmon[60316]: debug 2022-01-31T20:54:17.082+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:17.082741+0000) 2022-01-31T20:54:17.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:17 smithi171 conmon[46715]: debug 2022-01-31T20:54:17.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:17.360715+0000) 2022-01-31T20:54:17.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:17 smithi171 conmon[51620]: debug 2022-01-31T20:54:17.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:17.437001+0000) 2022-01-31T20:54:17.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:17 smithi171 conmon[41853]: debug 2022-01-31T20:54:17.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:17.566229+0000) 2022-01-31T20:54:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:17 smithi167 conmon[49112]: debug 2022-01-31T20:54:17.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:17.657591+0000) 2022-01-31T20:54:18.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:18 smithi167 conmon[60316]: debug 2022-01-31T20:54:18.082+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:18.082925+0000) 2022-01-31T20:54:18.376 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:18 smithi167 conmon[54076]: debug 2022-01-31T20:54:18.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:18.331760+0000) 2022-01-31T20:54:18.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:18 smithi171 conmon[46715]: debug 2022-01-31T20:54:18.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:18.360903+0000) 2022-01-31T20:54:18.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:18 smithi171 conmon[51620]: debug 2022-01-31T20:54:18.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:18.437182+0000) 2022-01-31T20:54:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:18 smithi171 conmon[41853]: debug 2022-01-31T20:54:18.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:18.566434+0000) 2022-01-31T20:54:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:18 smithi167 conmon[49112]: debug 2022-01-31T20:54:18.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:18.657722+0000) 2022-01-31T20:54:19.375 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:19 smithi167 conmon[49112]: debug 2022-01-31T20:54:19.265+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.266170+0000) 2022-01-31T20:54:19.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:19 smithi167 conmon[60316]: debug 2022-01-31T20:54:19.082+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.083106+0000) 2022-01-31T20:54:19.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:19 smithi167 conmon[60316]: debug 2022-01-31T20:54:19.266+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.266768+0000) 2022-01-31T20:54:19.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:19 smithi167 conmon[54076]: debug 2022-01-31T20:54:19.264+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.265266+0000) 2022-01-31T20:54:19.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:19 smithi167 conmon[54076]: debug 2022-01-31T20:54:19.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.331864+0000) 2022-01-31T20:54:19.565 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[35325]: debug 2022-01-31T20:54:19.272+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87532 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:19.566 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[41853]: debug 2022-01-31T20:54:19.265+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.266821+0000) 2022-01-31T20:54:19.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[46715]: debug 2022-01-31T20:54:19.263+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.265502+0000) 2022-01-31T20:54:19.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[46715]: debug 2022-01-31T20:54:19.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.361054+0000) 2022-01-31T20:54:19.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[51620]: debug 2022-01-31T20:54:19.264+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.266601+0000) 2022-01-31T20:54:19.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[51620]: debug 2022-01-31T20:54:19.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.437361+0000) 2022-01-31T20:54:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:19 smithi171 conmon[41853]: debug 2022-01-31T20:54:19.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.566597+0000) 2022-01-31T20:54:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:19 smithi167 conmon[49112]: debug 2022-01-31T20:54:19.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:19.657841+0000) 2022-01-31T20:54:20.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:20 smithi167 conmon[54076]: debug 2022-01-31T20:54:20.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:20.332013+0000) 2022-01-31T20:54:20.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:20 smithi167 conmon[60316]: debug 2022-01-31T20:54:20.082+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:20.083283+0000) 2022-01-31T20:54:20.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:20 smithi171 conmon[46715]: debug 2022-01-31T20:54:20.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:20.361205+0000) 2022-01-31T20:54:20.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:20 smithi171 conmon[51620]: debug 2022-01-31T20:54:20.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:20.437518+0000) 2022-01-31T20:54:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:20 smithi171 conmon[41853]: debug 2022-01-31T20:54:20.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:20.566746+0000) 2022-01-31T20:54:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:20 smithi167 conmon[49112]: debug 2022-01-31T20:54:20.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:20.657963+0000) 2022-01-31T20:54:21.376 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:21 smithi167 conmon[54076]: debug 2022-01-31T20:54:21.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:21.332201+0000) 2022-01-31T20:54:21.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:21 smithi167 conmon[60316]: debug 2022-01-31T20:54:21.082+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:21.083451+0000) 2022-01-31T20:54:21.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:21 smithi171 conmon[46715]: debug 2022-01-31T20:54:21.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:21.361424+0000) 2022-01-31T20:54:21.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:21 smithi171 conmon[51620]: debug 2022-01-31T20:54:21.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:21.437658+0000) 2022-01-31T20:54:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:21 smithi171 conmon[41853]: debug 2022-01-31T20:54:21.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:21.566957+0000) 2022-01-31T20:54:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:21 smithi167 conmon[49112]: debug 2022-01-31T20:54:21.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:21.658168+0000) 2022-01-31T20:54:22.376 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:22 smithi167 conmon[54076]: debug 2022-01-31T20:54:22.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:22.332384+0000) 2022-01-31T20:54:22.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:22 smithi167 conmon[60316]: debug 2022-01-31T20:54:22.083+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:22.083596+0000) 2022-01-31T20:54:22.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:22 smithi171 conmon[46715]: debug 2022-01-31T20:54:22.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:22.361593+0000) 2022-01-31T20:54:22.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:22 smithi171 conmon[51620]: debug 2022-01-31T20:54:22.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:22.437825+0000) 2022-01-31T20:54:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:22 smithi171 conmon[41853]: debug 2022-01-31T20:54:22.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:22.567146+0000) 2022-01-31T20:54:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:22 smithi167 conmon[49112]: debug 2022-01-31T20:54:22.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:22.658394+0000) 2022-01-31T20:54:23.376 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:23 smithi167 conmon[54076]: debug 2022-01-31T20:54:23.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:23.332556+0000) 2022-01-31T20:54:23.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:23 smithi167 conmon[60316]: debug 2022-01-31T20:54:23.083+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:23.083782+0000) 2022-01-31T20:54:23.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:23 smithi171 conmon[46715]: debug 2022-01-31T20:54:23.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:23.361760+0000) 2022-01-31T20:54:23.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:23 smithi171 conmon[51620]: debug 2022-01-31T20:54:23.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:23.437965+0000) 2022-01-31T20:54:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:23 smithi171 conmon[41853]: debug 2022-01-31T20:54:23.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:23.567330+0000) 2022-01-31T20:54:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:23 smithi171 conmon[35325]: debug 2022-01-31T20:54:23.590+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:54:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:23 smithi167 conmon[49112]: debug 2022-01-31T20:54:23.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:23.658573+0000) 2022-01-31T20:54:24.376 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:24 smithi167 conmon[49112]: debug 2022-01-31T20:54:24.277+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.278447+0000) 2022-01-31T20:54:24.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:24 smithi167 conmon[54076]: debug 2022-01-31T20:54:24.277+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.278077+0000) 2022-01-31T20:54:24.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:24 smithi167 conmon[54076]: debug 2022-01-31T20:54:24.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.332749+0000) 2022-01-31T20:54:24.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:24 smithi167 conmon[60316]: debug 2022-01-31T20:54:24.083+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.083976+0000) 2022-01-31T20:54:24.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:24 smithi167 conmon[60316]: debug 2022-01-31T20:54:24.277+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.278114+0000) 2022-01-31T20:54:24.566 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[35325]: debug 2022-01-31T20:54:24.285+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87643 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:24.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[41853]: debug 2022-01-31T20:54:24.276+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.277400+0000) 2022-01-31T20:54:24.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[46715]: debug 2022-01-31T20:54:24.275+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.276458+0000) 2022-01-31T20:54:24.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[46715]: debug 2022-01-31T20:54:24.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.361862+0000) 2022-01-31T20:54:24.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[51620]: debug 2022-01-31T20:54:24.277+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.277800+0000) 2022-01-31T20:54:24.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[51620]: debug 2022-01-31T20:54:24.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.438140+0000) 2022-01-31T20:54:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:24 smithi171 conmon[41853]: debug 2022-01-31T20:54:24.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.567496+0000) 2022-01-31T20:54:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:24 smithi167 conmon[49112]: debug 2022-01-31T20:54:24.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:24.658716+0000) 2022-01-31T20:54:25.376 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:25 smithi167 conmon[54076]: debug 2022-01-31T20:54:25.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:25.332859+0000) 2022-01-31T20:54:25.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:25 smithi167 conmon[60316]: debug 2022-01-31T20:54:25.083+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:25.084165+0000) 2022-01-31T20:54:25.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:25 smithi171 conmon[46715]: debug 2022-01-31T20:54:25.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:25.362013+0000) 2022-01-31T20:54:25.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:25 smithi171 conmon[51620]: debug 2022-01-31T20:54:25.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:25.438276+0000) 2022-01-31T20:54:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:25 smithi171 conmon[41853]: debug 2022-01-31T20:54:25.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:25.567626+0000) 2022-01-31T20:54:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:25 smithi167 conmon[49112]: debug 2022-01-31T20:54:25.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:25.658804+0000) 2022-01-31T20:54:26.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:26 smithi167 conmon[60316]: debug 2022-01-31T20:54:26.084+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:26.084324+0000) 2022-01-31T20:54:26.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:26 smithi167 conmon[54076]: debug 2022-01-31T20:54:26.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:26.333011+0000) 2022-01-31T20:54:26.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:26 smithi171 conmon[46715]: debug 2022-01-31T20:54:26.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:26.362137+0000) 2022-01-31T20:54:26.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:26 smithi171 conmon[51620]: debug 2022-01-31T20:54:26.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:26.438429+0000) 2022-01-31T20:54:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:26 smithi171 conmon[41853]: debug 2022-01-31T20:54:26.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:26.567837+0000) 2022-01-31T20:54:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:26 smithi167 conmon[49112]: debug 2022-01-31T20:54:26.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:26.658931+0000) 2022-01-31T20:54:27.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:27 smithi167 conmon[60316]: debug 2022-01-31T20:54:27.084+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:27.084551+0000) 2022-01-31T20:54:27.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:27 smithi167 conmon[54076]: debug 2022-01-31T20:54:27.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:27.333176+0000) 2022-01-31T20:54:27.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:27 smithi171 conmon[46715]: debug 2022-01-31T20:54:27.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:27.362337+0000) 2022-01-31T20:54:27.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:27 smithi171 conmon[51620]: debug 2022-01-31T20:54:27.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:27.438581+0000) 2022-01-31T20:54:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:27 smithi171 conmon[41853]: debug 2022-01-31T20:54:27.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:27.567976+0000) 2022-01-31T20:54:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:27 smithi167 conmon[49112]: debug 2022-01-31T20:54:27.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:27.659088+0000) 2022-01-31T20:54:28.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:28 smithi167 conmon[54076]: debug 2022-01-31T20:54:28.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:28.333341+0000) 2022-01-31T20:54:28.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:28 smithi167 conmon[60316]: debug 2022-01-31T20:54:28.084+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:28.084756+0000) 2022-01-31T20:54:28.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:28 smithi171 conmon[46715]: debug 2022-01-31T20:54:28.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:28.362543+0000) 2022-01-31T20:54:28.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:28 smithi171 conmon[51620]: debug 2022-01-31T20:54:28.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:28.438766+0000) 2022-01-31T20:54:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:28 smithi171 conmon[41853]: debug 2022-01-31T20:54:28.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:28.568128+0000) 2022-01-31T20:54:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:28 smithi167 conmon[49112]: debug 2022-01-31T20:54:28.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:28.659183+0000) 2022-01-31T20:54:29.378 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:29 smithi167 conmon[49112]: debug 2022-01-31T20:54:29.295+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.295824+0000) 2022-01-31T20:54:29.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:29 smithi167 conmon[54076]: debug 2022-01-31T20:54:29.296+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.296943+0000) 2022-01-31T20:54:29.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:29 smithi167 conmon[54076]: debug 2022-01-31T20:54:29.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.333550+0000) 2022-01-31T20:54:29.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:29 smithi167 conmon[60316]: debug 2022-01-31T20:54:29.084+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.084946+0000) 2022-01-31T20:54:29.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:29 smithi167 conmon[60316]: debug 2022-01-31T20:54:29.295+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.295891+0000) 2022-01-31T20:54:29.567 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[35325]: debug 2022-01-31T20:54:29.298+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87752 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:29.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[41853]: debug 2022-01-31T20:54:29.286+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.287721+0000) 2022-01-31T20:54:29.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[46715]: debug 2022-01-31T20:54:29.287+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.288092+0000) 2022-01-31T20:54:29.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[46715]: debug 2022-01-31T20:54:29.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.362743+0000) 2022-01-31T20:54:29.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[51620]: debug 2022-01-31T20:54:29.287+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.288636+0000) 2022-01-31T20:54:29.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[51620]: debug 2022-01-31T20:54:29.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.438889+0000) 2022-01-31T20:54:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:29 smithi171 conmon[41853]: debug 2022-01-31T20:54:29.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.568316+0000) 2022-01-31T20:54:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:29 smithi167 conmon[49112]: debug 2022-01-31T20:54:29.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:29.659376+0000) 2022-01-31T20:54:30.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:30 smithi167 conmon[54076]: debug 2022-01-31T20:54:30.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:30.333674+0000) 2022-01-31T20:54:30.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:30 smithi167 conmon[60316]: debug 2022-01-31T20:54:30.084+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:30.085139+0000) 2022-01-31T20:54:30.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:30 smithi171 conmon[46715]: debug 2022-01-31T20:54:30.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:30.362905+0000) 2022-01-31T20:54:30.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:30 smithi171 conmon[51620]: debug 2022-01-31T20:54:30.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:30.439010+0000) 2022-01-31T20:54:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:30 smithi171 conmon[41853]: debug 2022-01-31T20:54:30.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:30.568468+0000) 2022-01-31T20:54:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:30 smithi167 conmon[49112]: debug 2022-01-31T20:54:30.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:30.659577+0000) 2022-01-31T20:54:31.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:31 smithi167 conmon[60316]: debug 2022-01-31T20:54:31.084+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:31.085323+0000) 2022-01-31T20:54:31.378 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:31 smithi167 conmon[54076]: debug 2022-01-31T20:54:31.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:31.333881+0000) 2022-01-31T20:54:31.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:31 smithi171 conmon[46715]: debug 2022-01-31T20:54:31.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:31.363070+0000) 2022-01-31T20:54:31.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:31 smithi171 conmon[51620]: debug 2022-01-31T20:54:31.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:31.439141+0000) 2022-01-31T20:54:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:31 smithi171 conmon[41853]: debug 2022-01-31T20:54:31.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:31.568695+0000) 2022-01-31T20:54:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:31 smithi167 conmon[49112]: debug 2022-01-31T20:54:31.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:31.659741+0000) 2022-01-31T20:54:32.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:32 smithi167 conmon[60316]: debug 2022-01-31T20:54:32.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:32.085542+0000) 2022-01-31T20:54:32.378 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:32 smithi167 conmon[54076]: debug 2022-01-31T20:54:32.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:32.334092+0000) 2022-01-31T20:54:32.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:32 smithi171 conmon[46715]: debug 2022-01-31T20:54:32.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:32.363255+0000) 2022-01-31T20:54:32.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:32 smithi171 conmon[51620]: debug 2022-01-31T20:54:32.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:32.439265+0000) 2022-01-31T20:54:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:32 smithi171 conmon[41853]: debug 2022-01-31T20:54:32.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:32.568898+0000) 2022-01-31T20:54:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:32 smithi167 conmon[49112]: debug 2022-01-31T20:54:32.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:32.659912+0000) 2022-01-31T20:54:33.378 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:33 smithi167 conmon[54076]: debug 2022-01-31T20:54:33.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:33.334274+0000) 2022-01-31T20:54:33.379 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:33 smithi167 conmon[60316]: debug 2022-01-31T20:54:33.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:33.085693+0000) 2022-01-31T20:54:33.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:33 smithi171 conmon[46715]: debug 2022-01-31T20:54:33.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:33.363407+0000) 2022-01-31T20:54:33.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:33 smithi171 conmon[51620]: debug 2022-01-31T20:54:33.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:33.439433+0000) 2022-01-31T20:54:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:33 smithi171 conmon[41853]: debug 2022-01-31T20:54:33.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:33.569047+0000) 2022-01-31T20:54:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:33 smithi167 conmon[49112]: debug 2022-01-31T20:54:33.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:33.660149+0000) 2022-01-31T20:54:34.378 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:34 smithi167 conmon[49112]: debug 2022-01-31T20:54:34.301+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.301679+0000) 2022-01-31T20:54:34.379 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:34 smithi167 conmon[60316]: debug 2022-01-31T20:54:34.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.085847+0000) 2022-01-31T20:54:34.379 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:34 smithi167 conmon[60316]: debug 2022-01-31T20:54:34.302+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.302547+0000) 2022-01-31T20:54:34.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:34 smithi167 conmon[54076]: debug 2022-01-31T20:54:34.302+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.302797+0000) 2022-01-31T20:54:34.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:34 smithi167 conmon[54076]: debug 2022-01-31T20:54:34.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.334405+0000) 2022-01-31T20:54:34.567 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[35325]: debug 2022-01-31T20:54:34.309+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87863 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:34.568 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[41853]: debug 2022-01-31T20:54:34.301+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.302713+0000) 2022-01-31T20:54:34.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[51620]: debug 2022-01-31T20:54:34.300+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.301270+0000) 2022-01-31T20:54:34.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[51620]: debug 2022-01-31T20:54:34.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.439625+0000) 2022-01-31T20:54:34.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[46715]: debug 2022-01-31T20:54:34.300+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.301195+0000) 2022-01-31T20:54:34.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[46715]: debug 2022-01-31T20:54:34.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.363591+0000) 2022-01-31T20:54:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:34 smithi171 conmon[41853]: debug 2022-01-31T20:54:34.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.569165+0000) 2022-01-31T20:54:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:34 smithi167 conmon[49112]: debug 2022-01-31T20:54:34.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:34.660357+0000) 2022-01-31T20:54:35.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:35 smithi167 conmon[60316]: debug 2022-01-31T20:54:35.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:35.086053+0000) 2022-01-31T20:54:35.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:35 smithi167 conmon[54076]: debug 2022-01-31T20:54:35.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:35.334570+0000) 2022-01-31T20:54:35.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:35 smithi171 conmon[46715]: debug 2022-01-31T20:54:35.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:35.363748+0000) 2022-01-31T20:54:35.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:35 smithi171 conmon[51620]: debug 2022-01-31T20:54:35.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:35.439790+0000) 2022-01-31T20:54:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:35 smithi171 conmon[41853]: debug 2022-01-31T20:54:35.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:35.569311+0000) 2022-01-31T20:54:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:35 smithi167 conmon[49112]: debug 2022-01-31T20:54:35.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:35.660479+0000) 2022-01-31T20:54:36.378 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:36 smithi167 conmon[54076]: debug 2022-01-31T20:54:36.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:36.334717+0000) 2022-01-31T20:54:36.379 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:36 smithi167 conmon[60316]: debug 2022-01-31T20:54:36.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:36.086164+0000) 2022-01-31T20:54:36.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:36 smithi171 conmon[46715]: debug 2022-01-31T20:54:36.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:36.363938+0000) 2022-01-31T20:54:36.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:36 smithi171 conmon[51620]: debug 2022-01-31T20:54:36.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:36.439998+0000) 2022-01-31T20:54:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:36 smithi171 conmon[41853]: debug 2022-01-31T20:54:36.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:36.569462+0000) 2022-01-31T20:54:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:36 smithi167 conmon[49112]: debug 2022-01-31T20:54:36.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:36.660676+0000) 2022-01-31T20:54:37.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:37 smithi167 conmon[54076]: debug 2022-01-31T20:54:37.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:37.334849+0000) 2022-01-31T20:54:37.379 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:37 smithi167 conmon[60316]: debug 2022-01-31T20:54:37.086+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:37.086377+0000) 2022-01-31T20:54:37.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:37 smithi171 conmon[46715]: debug 2022-01-31T20:54:37.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:37.364102+0000) 2022-01-31T20:54:37.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:37 smithi171 conmon[51620]: debug 2022-01-31T20:54:37.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:37.440194+0000) 2022-01-31T20:54:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:37 smithi171 conmon[41853]: debug 2022-01-31T20:54:37.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:37.569635+0000) 2022-01-31T20:54:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:37 smithi167 conmon[49112]: debug 2022-01-31T20:54:37.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:37.660872+0000) 2022-01-31T20:54:38.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:38 smithi167 conmon[54076]: debug 2022-01-31T20:54:38.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:38.335055+0000) 2022-01-31T20:54:38.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:38 smithi167 conmon[60316]: debug 2022-01-31T20:54:38.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:38.086557+0000) 2022-01-31T20:54:38.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:38 smithi171 conmon[46715]: debug 2022-01-31T20:54:38.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:38.364330+0000) 2022-01-31T20:54:38.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:38 smithi171 conmon[51620]: debug 2022-01-31T20:54:38.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:38.440343+0000) 2022-01-31T20:54:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:38 smithi171 conmon[35325]: debug 2022-01-31T20:54:38.591+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:54:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:38 smithi171 conmon[41853]: debug 2022-01-31T20:54:38.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:38.569818+0000) 2022-01-31T20:54:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:38 smithi167 conmon[49112]: debug 2022-01-31T20:54:38.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:38.660995+0000) 2022-01-31T20:54:39.379 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:39 smithi167 conmon[49112]: debug 2022-01-31T20:54:39.311+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.312609+0000) 2022-01-31T20:54:39.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:39 smithi167 conmon[54076]: debug 2022-01-31T20:54:39.312+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.313524+0000) 2022-01-31T20:54:39.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:39 smithi167 conmon[54076]: debug 2022-01-31T20:54:39.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.335207+0000) 2022-01-31T20:54:39.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:39 smithi167 conmon[60316]: debug 2022-01-31T20:54:39.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.086756+0000) 2022-01-31T20:54:39.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:39 smithi167 conmon[60316]: debug 2022-01-31T20:54:39.311+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.313076+0000) 2022-01-31T20:54:39.568 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[35325]: debug 2022-01-31T20:54:39.320+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 87972 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:39.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[41853]: debug 2022-01-31T20:54:39.312+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.313464+0000) 2022-01-31T20:54:39.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[46715]: debug 2022-01-31T20:54:39.311+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.312474+0000) 2022-01-31T20:54:39.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[46715]: debug 2022-01-31T20:54:39.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.364525+0000) 2022-01-31T20:54:39.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[51620]: debug 2022-01-31T20:54:39.311+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.312186+0000) 2022-01-31T20:54:39.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[51620]: debug 2022-01-31T20:54:39.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.440492+0000) 2022-01-31T20:54:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:39 smithi171 conmon[41853]: debug 2022-01-31T20:54:39.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.569997+0000) 2022-01-31T20:54:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:39 smithi167 conmon[49112]: debug 2022-01-31T20:54:39.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:39.661171+0000) 2022-01-31T20:54:40.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:40 smithi167 conmon[54076]: debug 2022-01-31T20:54:40.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:40.335387+0000) 2022-01-31T20:54:40.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:40 smithi167 conmon[60316]: debug 2022-01-31T20:54:40.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:40.086959+0000) 2022-01-31T20:54:40.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:40 smithi171 conmon[51620]: debug 2022-01-31T20:54:40.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:40.440639+0000) 2022-01-31T20:54:40.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:40 smithi171 conmon[46715]: debug 2022-01-31T20:54:40.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:40.364660+0000) 2022-01-31T20:54:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:40 smithi171 conmon[41853]: debug 2022-01-31T20:54:40.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:40.570142+0000) 2022-01-31T20:54:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:40 smithi167 conmon[49112]: debug 2022-01-31T20:54:40.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:40.661304+0000) 2022-01-31T20:54:41.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:41 smithi167 conmon[54076]: debug 2022-01-31T20:54:41.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:41.335583+0000) 2022-01-31T20:54:41.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:41 smithi167 conmon[60316]: debug 2022-01-31T20:54:41.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:41.087099+0000) 2022-01-31T20:54:41.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:41 smithi171 conmon[46715]: debug 2022-01-31T20:54:41.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:41.364824+0000) 2022-01-31T20:54:41.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:41 smithi171 conmon[51620]: debug 2022-01-31T20:54:41.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:41.440773+0000) 2022-01-31T20:54:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:41 smithi171 conmon[41853]: debug 2022-01-31T20:54:41.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:41.570317+0000) 2022-01-31T20:54:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:41 smithi167 conmon[49112]: debug 2022-01-31T20:54:41.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:41.661551+0000) 2022-01-31T20:54:42.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:42 smithi167 conmon[54076]: debug 2022-01-31T20:54:42.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:42.335736+0000) 2022-01-31T20:54:42.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:42 smithi167 conmon[60316]: debug 2022-01-31T20:54:42.085+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:42.087307+0000) 2022-01-31T20:54:42.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:42 smithi171 conmon[46715]: debug 2022-01-31T20:54:42.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:42.365035+0000) 2022-01-31T20:54:42.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:42 smithi171 conmon[51620]: debug 2022-01-31T20:54:42.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:42.440946+0000) 2022-01-31T20:54:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:42 smithi171 conmon[41853]: debug 2022-01-31T20:54:42.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:42.570508+0000) 2022-01-31T20:54:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:42 smithi167 conmon[49112]: debug 2022-01-31T20:54:42.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:42.661711+0000) 2022-01-31T20:54:43.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:43 smithi167 conmon[54076]: debug 2022-01-31T20:54:43.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:43.335928+0000) 2022-01-31T20:54:43.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:43 smithi167 conmon[60316]: debug 2022-01-31T20:54:43.086+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:43.087489+0000) 2022-01-31T20:54:43.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:43 smithi171 conmon[46715]: debug 2022-01-31T20:54:43.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:43.365213+0000) 2022-01-31T20:54:43.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:43 smithi171 conmon[51620]: debug 2022-01-31T20:54:43.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:43.441125+0000) 2022-01-31T20:54:43.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:43 smithi171 conmon[41853]: debug 2022-01-31T20:54:43.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:43.570690+0000) 2022-01-31T20:54:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:43 smithi167 conmon[49112]: debug 2022-01-31T20:54:43.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:43.661837+0000) 2022-01-31T20:54:44.380 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:44 smithi167 conmon[49112]: debug 2022-01-31T20:54:44.322+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.323921+0000) 2022-01-31T20:54:44.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:44 smithi167 conmon[54076]: debug 2022-01-31T20:54:44.324+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.325760+0000) 2022-01-31T20:54:44.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:44 smithi167 conmon[54076]: debug 2022-01-31T20:54:44.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.336069+0000) 2022-01-31T20:54:44.382 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:44 smithi167 conmon[60316]: debug 2022-01-31T20:54:44.086+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.087690+0000) 2022-01-31T20:54:44.382 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:44 smithi167 conmon[60316]: debug 2022-01-31T20:54:44.324+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.325660+0000) 2022-01-31T20:54:44.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[41853]: debug 2022-01-31T20:54:44.323+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.324406+0000) 2022-01-31T20:54:44.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[46715]: debug 2022-01-31T20:54:44.323+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.324014+0000) 2022-01-31T20:54:44.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[46715]: debug 2022-01-31T20:54:44.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.365427+0000) 2022-01-31T20:54:44.570 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[35325]: debug 2022-01-31T20:54:44.332+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88082 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:44.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[51620]: debug 2022-01-31T20:54:44.322+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.323720+0000) 2022-01-31T20:54:44.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[51620]: debug 2022-01-31T20:54:44.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.441256+0000) 2022-01-31T20:54:44.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:44 smithi171 conmon[41853]: debug 2022-01-31T20:54:44.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.570898+0000) 2022-01-31T20:54:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:44 smithi167 conmon[49112]: debug 2022-01-31T20:54:44.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:44.662022+0000) 2022-01-31T20:54:45.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:45 smithi167 conmon[54076]: debug 2022-01-31T20:54:45.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:45.336168+0000) 2022-01-31T20:54:45.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:45 smithi167 conmon[60316]: debug 2022-01-31T20:54:45.086+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:45.087867+0000) 2022-01-31T20:54:45.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:45 smithi171 conmon[46715]: debug 2022-01-31T20:54:45.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:45.365594+0000) 2022-01-31T20:54:45.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:45 smithi171 conmon[51620]: debug 2022-01-31T20:54:45.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:45.441383+0000) 2022-01-31T20:54:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:45 smithi171 conmon[41853]: debug 2022-01-31T20:54:45.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:45.571044+0000) 2022-01-31T20:54:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:45 smithi167 conmon[49112]: debug 2022-01-31T20:54:45.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:45.662223+0000) 2022-01-31T20:54:46.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:46 smithi167 conmon[54076]: debug 2022-01-31T20:54:46.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:46.336311+0000) 2022-01-31T20:54:46.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:46 smithi167 conmon[60316]: debug 2022-01-31T20:54:46.087+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:46.088025+0000) 2022-01-31T20:54:46.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:46 smithi171 conmon[46715]: debug 2022-01-31T20:54:46.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:46.365806+0000) 2022-01-31T20:54:46.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:46 smithi171 conmon[51620]: debug 2022-01-31T20:54:46.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:46.441519+0000) 2022-01-31T20:54:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:46 smithi171 conmon[41853]: debug 2022-01-31T20:54:46.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:46.571247+0000) 2022-01-31T20:54:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:46 smithi167 conmon[49112]: debug 2022-01-31T20:54:46.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:46.662425+0000) 2022-01-31T20:54:47.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:47 smithi167 conmon[54076]: debug 2022-01-31T20:54:47.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:47.336450+0000) 2022-01-31T20:54:47.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:47 smithi167 conmon[60316]: debug 2022-01-31T20:54:47.087+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:47.088215+0000) 2022-01-31T20:54:47.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:47 smithi171 conmon[46715]: debug 2022-01-31T20:54:47.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:47.365994+0000) 2022-01-31T20:54:47.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:47 smithi171 conmon[51620]: debug 2022-01-31T20:54:47.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:47.441688+0000) 2022-01-31T20:54:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:47 smithi171 conmon[41853]: debug 2022-01-31T20:54:47.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:47.571430+0000) 2022-01-31T20:54:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:47 smithi167 conmon[49112]: debug 2022-01-31T20:54:47.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:47.662610+0000) 2022-01-31T20:54:48.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:48 smithi167 conmon[54076]: debug 2022-01-31T20:54:48.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:48.336615+0000) 2022-01-31T20:54:48.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:48 smithi167 conmon[60316]: debug 2022-01-31T20:54:48.087+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:48.088397+0000) 2022-01-31T20:54:48.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:48 smithi171 conmon[46715]: debug 2022-01-31T20:54:48.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:48.366157+0000) 2022-01-31T20:54:48.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:48 smithi171 conmon[51620]: debug 2022-01-31T20:54:48.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:48.441901+0000) 2022-01-31T20:54:48.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:48 smithi171 conmon[41853]: debug 2022-01-31T20:54:48.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:48.571626+0000) 2022-01-31T20:54:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:48 smithi167 conmon[49112]: debug 2022-01-31T20:54:48.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:48.662764+0000) 2022-01-31T20:54:49.339 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:49 smithi167 conmon[49112]: debug 2022-01-31T20:54:49.334+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.335978+0000) 2022-01-31T20:54:49.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:49 smithi167 conmon[54076]: debug 2022-01-31T20:54:49.335+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.336722+0000) 2022-01-31T20:54:49.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:49 smithi167 conmon[54076]: debug 2022-01-31T20:54:49.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.336752+0000) 2022-01-31T20:54:49.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:49 smithi167 conmon[60316]: debug 2022-01-31T20:54:49.087+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.088580+0000) 2022-01-31T20:54:49.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:49 smithi167 conmon[60316]: debug 2022-01-31T20:54:49.334+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.335795+0000) 2022-01-31T20:54:49.570 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[35325]: debug 2022-01-31T20:54:49.344+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:49.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[41853]: debug 2022-01-31T20:54:49.335+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.335878+0000) 2022-01-31T20:54:49.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[51620]: debug 2022-01-31T20:54:49.335+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.336250+0000) 2022-01-31T20:54:49.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[51620]: debug 2022-01-31T20:54:49.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.442080+0000) 2022-01-31T20:54:49.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[46715]: debug 2022-01-31T20:54:49.335+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.336080+0000) 2022-01-31T20:54:49.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[46715]: debug 2022-01-31T20:54:49.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.366276+0000) 2022-01-31T20:54:49.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:49 smithi171 conmon[41853]: debug 2022-01-31T20:54:49.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.571859+0000) 2022-01-31T20:54:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:49 smithi167 conmon[49112]: debug 2022-01-31T20:54:49.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:49.662899+0000) 2022-01-31T20:54:50.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:50 smithi167 conmon[60316]: debug 2022-01-31T20:54:50.087+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:50.088719+0000) 2022-01-31T20:54:50.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:50 smithi167 conmon[54076]: debug 2022-01-31T20:54:50.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:50.336916+0000) 2022-01-31T20:54:50.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:50 smithi171 conmon[46715]: debug 2022-01-31T20:54:50.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:50.366449+0000) 2022-01-31T20:54:50.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:50 smithi171 conmon[51620]: debug 2022-01-31T20:54:50.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:50.442237+0000) 2022-01-31T20:54:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:50 smithi171 conmon[41853]: debug 2022-01-31T20:54:50.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:50.572014+0000) 2022-01-31T20:54:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:50 smithi167 conmon[49112]: debug 2022-01-31T20:54:50.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:50.663045+0000) 2022-01-31T20:54:51.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:51 smithi167 conmon[54076]: debug 2022-01-31T20:54:51.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:51.337124+0000) 2022-01-31T20:54:51.382 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:51 smithi167 conmon[60316]: debug 2022-01-31T20:54:51.088+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:51.088886+0000) 2022-01-31T20:54:51.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:51 smithi171 conmon[46715]: debug 2022-01-31T20:54:51.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:51.366639+0000) 2022-01-31T20:54:51.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:51 smithi171 conmon[51620]: debug 2022-01-31T20:54:51.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:51.442396+0000) 2022-01-31T20:54:51.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:51 smithi171 conmon[41853]: debug 2022-01-31T20:54:51.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:51.572131+0000) 2022-01-31T20:54:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:51 smithi167 conmon[49112]: debug 2022-01-31T20:54:51.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:51.663248+0000) 2022-01-31T20:54:52.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:52 smithi167 conmon[54076]: debug 2022-01-31T20:54:52.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:52.337280+0000) 2022-01-31T20:54:52.382 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:52 smithi167 conmon[60316]: debug 2022-01-31T20:54:52.088+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:52.089057+0000) 2022-01-31T20:54:52.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:52 smithi171 conmon[46715]: debug 2022-01-31T20:54:52.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:52.366826+0000) 2022-01-31T20:54:52.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:52 smithi171 conmon[51620]: debug 2022-01-31T20:54:52.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:52.442592+0000) 2022-01-31T20:54:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:52 smithi171 conmon[41853]: debug 2022-01-31T20:54:52.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:52.572350+0000) 2022-01-31T20:54:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:52 smithi167 conmon[49112]: debug 2022-01-31T20:54:52.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:52.663429+0000) 2022-01-31T20:54:53.382 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:53 smithi167 conmon[54076]: debug 2022-01-31T20:54:53.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:53.337433+0000) 2022-01-31T20:54:53.382 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:53 smithi167 conmon[60316]: debug 2022-01-31T20:54:53.087+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:53.089239+0000) 2022-01-31T20:54:53.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:53 smithi171 conmon[46715]: debug 2022-01-31T20:54:53.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:53.366982+0000) 2022-01-31T20:54:53.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:53 smithi171 conmon[51620]: debug 2022-01-31T20:54:53.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:53.442780+0000) 2022-01-31T20:54:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:53 smithi171 conmon[35325]: debug 2022-01-31T20:54:53.592+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:54:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:53 smithi171 conmon[41853]: debug 2022-01-31T20:54:53.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:53.572509+0000) 2022-01-31T20:54:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:53 smithi167 conmon[49112]: debug 2022-01-31T20:54:53.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:53.663574+0000) 2022-01-31T20:54:54.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:54 smithi167 conmon[54076]: debug 2022-01-31T20:54:54.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.337589+0000) 2022-01-31T20:54:54.346 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:54 smithi167 conmon[60316]: debug 2022-01-31T20:54:54.088+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.089443+0000) 2022-01-31T20:54:54.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:54 smithi167 conmon[60316]: debug 2022-01-31T20:54:54.346+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.347610+0000) 2022-01-31T20:54:54.571 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[35325]: debug 2022-01-31T20:54:54.356+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88303 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:54.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[41853]: debug 2022-01-31T20:54:54.347+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.348081+0000) 2022-01-31T20:54:54.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[51620]: debug 2022-01-31T20:54:54.346+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.347507+0000) 2022-01-31T20:54:54.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[51620]: debug 2022-01-31T20:54:54.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.442908+0000) 2022-01-31T20:54:54.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[46715]: debug 2022-01-31T20:54:54.346+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.346925+0000) 2022-01-31T20:54:54.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[46715]: debug 2022-01-31T20:54:54.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.367091+0000) 2022-01-31T20:54:54.647 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:54 smithi167 conmon[49112]: debug 2022-01-31T20:54:54.347+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.348901+0000) 2022-01-31T20:54:54.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:54 smithi167 conmon[54076]: debug 2022-01-31T20:54:54.347+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.348646+0000) 2022-01-31T20:54:54.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:54 smithi171 conmon[41853]: debug 2022-01-31T20:54:54.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.572720+0000) 2022-01-31T20:54:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:54 smithi167 conmon[49112]: debug 2022-01-31T20:54:54.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:54.663697+0000) 2022-01-31T20:54:55.382 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:55 smithi167 conmon[54076]: debug 2022-01-31T20:54:55.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:55.337718+0000) 2022-01-31T20:54:55.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:55 smithi167 conmon[60316]: debug 2022-01-31T20:54:55.088+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:55.089649+0000) 2022-01-31T20:54:55.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:55 smithi171 conmon[46715]: debug 2022-01-31T20:54:55.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:55.367191+0000) 2022-01-31T20:54:55.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:55 smithi171 conmon[51620]: debug 2022-01-31T20:54:55.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:55.443047+0000) 2022-01-31T20:54:55.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:55 smithi171 conmon[41853]: debug 2022-01-31T20:54:55.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:55.572875+0000) 2022-01-31T20:54:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:55 smithi167 conmon[49112]: debug 2022-01-31T20:54:55.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:55.663797+0000) 2022-01-31T20:54:56.382 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:56 smithi167 conmon[54076]: debug 2022-01-31T20:54:56.337+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:56.337881+0000) 2022-01-31T20:54:56.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:56 smithi167 conmon[60316]: debug 2022-01-31T20:54:56.089+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:56.089803+0000) 2022-01-31T20:54:56.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:56 smithi171 conmon[46715]: debug 2022-01-31T20:54:56.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:56.367406+0000) 2022-01-31T20:54:56.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:56 smithi171 conmon[51620]: debug 2022-01-31T20:54:56.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:56.443238+0000) 2022-01-31T20:54:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:56 smithi171 conmon[41853]: debug 2022-01-31T20:54:56.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:56.573032+0000) 2022-01-31T20:54:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:56 smithi167 conmon[49112]: debug 2022-01-31T20:54:56.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:56.663961+0000) 2022-01-31T20:54:57.382 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:57 smithi167 conmon[54076]: debug 2022-01-31T20:54:57.337+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:57.338074+0000) 2022-01-31T20:54:57.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:57 smithi167 conmon[60316]: debug 2022-01-31T20:54:57.088+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:57.089997+0000) 2022-01-31T20:54:57.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:57 smithi171 conmon[46715]: debug 2022-01-31T20:54:57.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:57.367603+0000) 2022-01-31T20:54:57.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:57 smithi171 conmon[51620]: debug 2022-01-31T20:54:57.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:57.443409+0000) 2022-01-31T20:54:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:57 smithi171 conmon[41853]: debug 2022-01-31T20:54:57.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:57.573210+0000) 2022-01-31T20:54:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:57 smithi167 conmon[49112]: debug 2022-01-31T20:54:57.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:57.664145+0000) 2022-01-31T20:54:58.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:58 smithi167 conmon[54076]: debug 2022-01-31T20:54:58.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:58.338315+0000) 2022-01-31T20:54:58.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:58 smithi167 conmon[60316]: debug 2022-01-31T20:54:58.089+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:58.090190+0000) 2022-01-31T20:54:58.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:58 smithi171 conmon[46715]: debug 2022-01-31T20:54:58.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:58.367833+0000) 2022-01-31T20:54:58.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:58 smithi171 conmon[51620]: debug 2022-01-31T20:54:58.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:58.443609+0000) 2022-01-31T20:54:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:58 smithi171 conmon[41853]: debug 2022-01-31T20:54:58.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:58.573393+0000) 2022-01-31T20:54:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:58 smithi167 conmon[49112]: debug 2022-01-31T20:54:58.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:58.664346+0000) 2022-01-31T20:54:59.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:59 smithi167 conmon[54076]: debug 2022-01-31T20:54:59.337+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.338489+0000) 2022-01-31T20:54:59.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:59 smithi167 conmon[60316]: debug 2022-01-31T20:54:59.089+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.090377+0000) 2022-01-31T20:54:59.572 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[35325]: debug 2022-01-31T20:54:59.366+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88413 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:54:59.572 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[41853]: debug 2022-01-31T20:54:59.358+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.360456+0000) 2022-01-31T20:54:59.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[46715]: debug 2022-01-31T20:54:59.357+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.359712+0000) 2022-01-31T20:54:59.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[46715]: debug 2022-01-31T20:54:59.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.368005+0000) 2022-01-31T20:54:59.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[51620]: debug 2022-01-31T20:54:59.357+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.358980+0000) 2022-01-31T20:54:59.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[51620]: debug 2022-01-31T20:54:59.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.443816+0000) 2022-01-31T20:54:59.652 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:59 smithi167 conmon[49112]: debug 2022-01-31T20:54:59.356+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.360237+0000) 2022-01-31T20:54:59.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:54:59 smithi167 conmon[54076]: debug 2022-01-31T20:54:59.356+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.359442+0000) 2022-01-31T20:54:59.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:54:59 smithi167 conmon[60316]: debug 2022-01-31T20:54:59.356+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.360357+0000) 2022-01-31T20:54:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:54:59 smithi171 conmon[41853]: debug 2022-01-31T20:54:59.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.573575+0000) 2022-01-31T20:54:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:54:59 smithi167 conmon[49112]: debug 2022-01-31T20:54:59.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:54:59.664572+0000) 2022-01-31T20:55:00.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:00 smithi167 conmon[54076]: debug 2022-01-31T20:55:00.337+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:00.338676+0000) 2022-01-31T20:55:00.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:00 smithi167 conmon[60316]: debug 2022-01-31T20:55:00.089+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:00.090563+0000) 2022-01-31T20:55:00.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:00 smithi171 conmon[46715]: debug 2022-01-31T20:55:00.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:00.368145+0000) 2022-01-31T20:55:00.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:00 smithi171 conmon[51620]: debug 2022-01-31T20:55:00.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:00.443972+0000) 2022-01-31T20:55:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:00 smithi171 conmon[41853]: debug 2022-01-31T20:55:00.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:00.573706+0000) 2022-01-31T20:55:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:00 smithi167 conmon[49112]: debug 2022-01-31T20:55:00.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:00.664678+0000) 2022-01-31T20:55:01.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:01 smithi167 conmon[54076]: debug 2022-01-31T20:55:01.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:01.338765+0000) 2022-01-31T20:55:01.384 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:01 smithi167 conmon[60316]: debug 2022-01-31T20:55:01.090+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:01.090744+0000) 2022-01-31T20:55:01.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:01 smithi171 conmon[46715]: debug 2022-01-31T20:55:01.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:01.368325+0000) 2022-01-31T20:55:01.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:01 smithi171 conmon[51620]: debug 2022-01-31T20:55:01.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:01.444148+0000) 2022-01-31T20:55:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:01 smithi171 conmon[41853]: debug 2022-01-31T20:55:01.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:01.573898+0000) 2022-01-31T20:55:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:01 smithi167 conmon[49112]: debug 2022-01-31T20:55:01.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:01.664863+0000) 2022-01-31T20:55:02.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:02 smithi167 conmon[60316]: debug 2022-01-31T20:55:02.090+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:02.090875+0000) 2022-01-31T20:55:02.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:02 smithi167 conmon[54076]: debug 2022-01-31T20:55:02.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:02.338925+0000) 2022-01-31T20:55:02.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:02 smithi171 conmon[46715]: debug 2022-01-31T20:55:02.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:02.368483+0000) 2022-01-31T20:55:02.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:02 smithi171 conmon[51620]: debug 2022-01-31T20:55:02.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:02.444332+0000) 2022-01-31T20:55:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:02 smithi171 conmon[41853]: debug 2022-01-31T20:55:02.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:02.574095+0000) 2022-01-31T20:55:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:02 smithi167 conmon[49112]: debug 2022-01-31T20:55:02.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:02.665019+0000) 2022-01-31T20:55:03.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:03 smithi167 conmon[54076]: debug 2022-01-31T20:55:03.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:03.339102+0000) 2022-01-31T20:55:03.384 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:03 smithi167 conmon[60316]: debug 2022-01-31T20:55:03.090+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:03.091051+0000) 2022-01-31T20:55:03.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:03 smithi171 conmon[46715]: debug 2022-01-31T20:55:03.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:03.368669+0000) 2022-01-31T20:55:03.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:03 smithi171 conmon[51620]: debug 2022-01-31T20:55:03.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:03.444465+0000) 2022-01-31T20:55:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:03 smithi171 conmon[41853]: debug 2022-01-31T20:55:03.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:03.574302+0000) 2022-01-31T20:55:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:03 smithi167 conmon[49112]: debug 2022-01-31T20:55:03.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:03.665176+0000) 2022-01-31T20:55:04.368 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:04 smithi167 conmon[60316]: debug 2022-01-31T20:55:04.090+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.091202+0000) 2022-01-31T20:55:04.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:04 smithi167 conmon[54076]: debug 2022-01-31T20:55:04.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.339282+0000) 2022-01-31T20:55:04.573 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[35325]: debug 2022-01-31T20:55:04.377+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88523 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:04.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[41853]: debug 2022-01-31T20:55:04.368+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.370403+0000) 2022-01-31T20:55:04.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[46715]: debug 2022-01-31T20:55:04.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.368896+0000) 2022-01-31T20:55:04.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[46715]: debug 2022-01-31T20:55:04.369+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.371120+0000) 2022-01-31T20:55:04.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[51620]: debug 2022-01-31T20:55:04.368+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.370722+0000) 2022-01-31T20:55:04.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[51620]: debug 2022-01-31T20:55:04.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.444684+0000) 2022-01-31T20:55:04.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:04 smithi167 conmon[54076]: debug 2022-01-31T20:55:04.370+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.371447+0000) 2022-01-31T20:55:04.657 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:04 smithi167 conmon[49112]: debug 2022-01-31T20:55:04.370+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.371217+0000) 2022-01-31T20:55:04.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:04 smithi167 conmon[60316]: debug 2022-01-31T20:55:04.370+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.370952+0000) 2022-01-31T20:55:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:04 smithi171 conmon[41853]: debug 2022-01-31T20:55:04.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.574475+0000) 2022-01-31T20:55:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:04 smithi167 conmon[49112]: debug 2022-01-31T20:55:04.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:04.665351+0000) 2022-01-31T20:55:05.385 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:05 smithi167 conmon[54076]: debug 2022-01-31T20:55:05.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:05.339478+0000) 2022-01-31T20:55:05.385 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:05 smithi167 conmon[60316]: debug 2022-01-31T20:55:05.090+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:05.091363+0000) 2022-01-31T20:55:05.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:05 smithi171 conmon[46715]: debug 2022-01-31T20:55:05.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:05.369019+0000) 2022-01-31T20:55:05.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:05 smithi171 conmon[51620]: debug 2022-01-31T20:55:05.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:05.444806+0000) 2022-01-31T20:55:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:05 smithi171 conmon[41853]: debug 2022-01-31T20:55:05.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:05.574630+0000) 2022-01-31T20:55:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:05 smithi167 conmon[49112]: debug 2022-01-31T20:55:05.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:05.665460+0000) 2022-01-31T20:55:06.385 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:06 smithi167 conmon[54076]: debug 2022-01-31T20:55:06.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:06.339652+0000) 2022-01-31T20:55:06.385 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:06 smithi167 conmon[60316]: debug 2022-01-31T20:55:06.091+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:06.091540+0000) 2022-01-31T20:55:06.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:06 smithi171 conmon[46715]: debug 2022-01-31T20:55:06.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:06.369140+0000) 2022-01-31T20:55:06.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:06 smithi171 conmon[51620]: debug 2022-01-31T20:55:06.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:06.444954+0000) 2022-01-31T20:55:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:06 smithi171 conmon[41853]: debug 2022-01-31T20:55:06.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:06.574830+0000) 2022-01-31T20:55:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:06 smithi167 conmon[49112]: debug 2022-01-31T20:55:06.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:06.665600+0000) 2022-01-31T20:55:07.385 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:07 smithi167 conmon[54076]: debug 2022-01-31T20:55:07.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:07.339811+0000) 2022-01-31T20:55:07.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:07 smithi167 conmon[60316]: debug 2022-01-31T20:55:07.091+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:07.091722+0000) 2022-01-31T20:55:07.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:07 smithi171 conmon[46715]: debug 2022-01-31T20:55:07.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:07.369315+0000) 2022-01-31T20:55:07.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:07 smithi171 conmon[51620]: debug 2022-01-31T20:55:07.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:07.445122+0000) 2022-01-31T20:55:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:07 smithi171 conmon[41853]: debug 2022-01-31T20:55:07.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:07.575012+0000) 2022-01-31T20:55:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:07 smithi167 conmon[49112]: debug 2022-01-31T20:55:07.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:07.665746+0000) 2022-01-31T20:55:08.385 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:08 smithi167 conmon[54076]: debug 2022-01-31T20:55:08.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:08.340056+0000) 2022-01-31T20:55:08.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:08 smithi167 conmon[60316]: debug 2022-01-31T20:55:08.091+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:08.091903+0000) 2022-01-31T20:55:08.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:08 smithi171 conmon[46715]: debug 2022-01-31T20:55:08.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:08.369534+0000) 2022-01-31T20:55:08.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:08 smithi171 conmon[51620]: debug 2022-01-31T20:55:08.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:08.445324+0000) 2022-01-31T20:55:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:08 smithi171 conmon[41853]: debug 2022-01-31T20:55:08.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:08.575209+0000) 2022-01-31T20:55:08.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:08 smithi171 conmon[35325]: debug 2022-01-31T20:55:08.592+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:55:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:08 smithi167 conmon[49112]: debug 2022-01-31T20:55:08.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:08.665897+0000) 2022-01-31T20:55:09.380 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:09 smithi167 conmon[54076]: debug 2022-01-31T20:55:09.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.340262+0000) 2022-01-31T20:55:09.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:09 smithi167 conmon[60316]: debug 2022-01-31T20:55:09.091+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.092084+0000) 2022-01-31T20:55:09.574 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[35325]: debug 2022-01-31T20:55:09.389+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88633 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:09.575 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[41853]: debug 2022-01-31T20:55:09.379+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.381434+0000) 2022-01-31T20:55:09.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[46715]: debug 2022-01-31T20:55:09.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.369701+0000) 2022-01-31T20:55:09.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[46715]: debug 2022-01-31T20:55:09.380+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.382095+0000) 2022-01-31T20:55:09.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[51620]: debug 2022-01-31T20:55:09.379+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.381528+0000) 2022-01-31T20:55:09.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[51620]: debug 2022-01-31T20:55:09.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.445523+0000) 2022-01-31T20:55:09.657 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:09 smithi167 conmon[49112]: debug 2022-01-31T20:55:09.382+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.383063+0000) 2022-01-31T20:55:09.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:09 smithi167 conmon[54076]: debug 2022-01-31T20:55:09.382+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.382854+0000) 2022-01-31T20:55:09.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:09 smithi167 conmon[60316]: debug 2022-01-31T20:55:09.381+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.382347+0000) 2022-01-31T20:55:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:09 smithi171 conmon[41853]: debug 2022-01-31T20:55:09.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.575446+0000) 2022-01-31T20:55:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:09 smithi167 conmon[49112]: debug 2022-01-31T20:55:09.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:09.666046+0000) 2022-01-31T20:55:10.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:10 smithi167 conmon[54076]: debug 2022-01-31T20:55:10.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:10.340473+0000) 2022-01-31T20:55:10.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:10 smithi167 conmon[60316]: debug 2022-01-31T20:55:10.091+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:10.092269+0000) 2022-01-31T20:55:10.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:10 smithi171 conmon[46715]: debug 2022-01-31T20:55:10.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:10.369927+0000) 2022-01-31T20:55:10.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:10 smithi171 conmon[51620]: debug 2022-01-31T20:55:10.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:10.445684+0000) 2022-01-31T20:55:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:10 smithi171 conmon[41853]: debug 2022-01-31T20:55:10.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:10.575628+0000) 2022-01-31T20:55:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:10 smithi167 conmon[49112]: debug 2022-01-31T20:55:10.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:10.666206+0000) 2022-01-31T20:55:11.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:11 smithi167 conmon[54076]: debug 2022-01-31T20:55:11.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:11.340647+0000) 2022-01-31T20:55:11.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:11 smithi167 conmon[60316]: debug 2022-01-31T20:55:11.092+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:11.092402+0000) 2022-01-31T20:55:11.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:11 smithi171 conmon[46715]: debug 2022-01-31T20:55:11.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:11.370083+0000) 2022-01-31T20:55:11.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:11 smithi171 conmon[51620]: debug 2022-01-31T20:55:11.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:11.445880+0000) 2022-01-31T20:55:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:11 smithi171 conmon[41853]: debug 2022-01-31T20:55:11.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:11.575817+0000) 2022-01-31T20:55:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:11 smithi167 conmon[49112]: debug 2022-01-31T20:55:11.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:11.666388+0000) 2022-01-31T20:55:12.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:12 smithi167 conmon[54076]: debug 2022-01-31T20:55:12.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:12.340831+0000) 2022-01-31T20:55:12.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:12 smithi167 conmon[60316]: debug 2022-01-31T20:55:12.092+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:12.092577+0000) 2022-01-31T20:55:12.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:12 smithi171 conmon[46715]: debug 2022-01-31T20:55:12.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:12.370271+0000) 2022-01-31T20:55:12.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:12 smithi171 conmon[51620]: debug 2022-01-31T20:55:12.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:12.446001+0000) 2022-01-31T20:55:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:12 smithi171 conmon[41853]: debug 2022-01-31T20:55:12.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:12.576013+0000) 2022-01-31T20:55:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:12 smithi167 conmon[49112]: debug 2022-01-31T20:55:12.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:12.666618+0000) 2022-01-31T20:55:13.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:13 smithi167 conmon[54076]: debug 2022-01-31T20:55:13.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:13.340989+0000) 2022-01-31T20:55:13.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:13 smithi167 conmon[60316]: debug 2022-01-31T20:55:13.092+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:13.092761+0000) 2022-01-31T20:55:13.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:13 smithi171 conmon[46715]: debug 2022-01-31T20:55:13.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:13.370454+0000) 2022-01-31T20:55:13.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:13 smithi171 conmon[51620]: debug 2022-01-31T20:55:13.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:13.446183+0000) 2022-01-31T20:55:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:13 smithi171 conmon[41853]: debug 2022-01-31T20:55:13.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:13.576201+0000) 2022-01-31T20:55:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:13 smithi167 conmon[49112]: debug 2022-01-31T20:55:13.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:13.666729+0000) 2022-01-31T20:55:14.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:14 smithi167 conmon[54076]: debug 2022-01-31T20:55:14.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.341166+0000) 2022-01-31T20:55:14.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:14 smithi167 conmon[60316]: debug 2022-01-31T20:55:14.092+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.092945+0000) 2022-01-31T20:55:14.575 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[35325]: debug 2022-01-31T20:55:14.401+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88743 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:14.575 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[41853]: debug 2022-01-31T20:55:14.391+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.392651+0000) 2022-01-31T20:55:14.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[46715]: debug 2022-01-31T20:55:14.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.370640+0000) 2022-01-31T20:55:14.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[46715]: debug 2022-01-31T20:55:14.392+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.393055+0000) 2022-01-31T20:55:14.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[51620]: debug 2022-01-31T20:55:14.392+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.392806+0000) 2022-01-31T20:55:14.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[51620]: debug 2022-01-31T20:55:14.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.446404+0000) 2022-01-31T20:55:14.658 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:14 smithi167 conmon[49112]: debug 2022-01-31T20:55:14.393+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.394442+0000) 2022-01-31T20:55:14.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:14 smithi167 conmon[54076]: debug 2022-01-31T20:55:14.393+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.393797+0000) 2022-01-31T20:55:14.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:14 smithi167 conmon[60316]: debug 2022-01-31T20:55:14.393+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.394011+0000) 2022-01-31T20:55:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:14 smithi171 conmon[41853]: debug 2022-01-31T20:55:14.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.576357+0000) 2022-01-31T20:55:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:14 smithi167 conmon[49112]: debug 2022-01-31T20:55:14.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:14.666893+0000) 2022-01-31T20:55:15.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:15 smithi167 conmon[54076]: debug 2022-01-31T20:55:15.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:15.341347+0000) 2022-01-31T20:55:15.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:15 smithi167 conmon[60316]: debug 2022-01-31T20:55:15.092+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:15.093135+0000) 2022-01-31T20:55:15.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:15 smithi171 conmon[46715]: debug 2022-01-31T20:55:15.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:15.370798+0000) 2022-01-31T20:55:15.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:15 smithi171 conmon[51620]: debug 2022-01-31T20:55:15.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:15.446593+0000) 2022-01-31T20:55:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:15 smithi171 conmon[41853]: debug 2022-01-31T20:55:15.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:15.576505+0000) 2022-01-31T20:55:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:15 smithi167 conmon[49112]: debug 2022-01-31T20:55:15.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:15.667048+0000) 2022-01-31T20:55:16.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:16 smithi167 conmon[54076]: debug 2022-01-31T20:55:16.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:16.341563+0000) 2022-01-31T20:55:16.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:16 smithi167 conmon[60316]: debug 2022-01-31T20:55:16.092+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:16.093275+0000) 2022-01-31T20:55:16.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:16 smithi171 conmon[51620]: debug 2022-01-31T20:55:16.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:16.446802+0000) 2022-01-31T20:55:16.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:16 smithi171 conmon[46715]: debug 2022-01-31T20:55:16.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:16.370955+0000) 2022-01-31T20:55:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:16 smithi171 conmon[41853]: debug 2022-01-31T20:55:16.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:16.576689+0000) 2022-01-31T20:55:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:16 smithi167 conmon[49112]: debug 2022-01-31T20:55:16.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:16.667220+0000) 2022-01-31T20:55:17.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:17 smithi167 conmon[54076]: debug 2022-01-31T20:55:17.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:17.341676+0000) 2022-01-31T20:55:17.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:17 smithi167 conmon[60316]: debug 2022-01-31T20:55:17.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:17.093430+0000) 2022-01-31T20:55:17.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:17 smithi171 conmon[46715]: debug 2022-01-31T20:55:17.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:17.371091+0000) 2022-01-31T20:55:17.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:17 smithi171 conmon[51620]: debug 2022-01-31T20:55:17.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:17.447004+0000) 2022-01-31T20:55:17.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:17 smithi171 conmon[41853]: debug 2022-01-31T20:55:17.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:17.576957+0000) 2022-01-31T20:55:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:17 smithi167 conmon[49112]: debug 2022-01-31T20:55:17.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:17.667428+0000) 2022-01-31T20:55:18.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:18 smithi167 conmon[54076]: debug 2022-01-31T20:55:18.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:18.341851+0000) 2022-01-31T20:55:18.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:18 smithi167 conmon[60316]: debug 2022-01-31T20:55:18.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:18.093586+0000) 2022-01-31T20:55:18.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:18 smithi171 conmon[46715]: debug 2022-01-31T20:55:18.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:18.371258+0000) 2022-01-31T20:55:18.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:18 smithi171 conmon[51620]: debug 2022-01-31T20:55:18.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:18.447208+0000) 2022-01-31T20:55:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:18 smithi171 conmon[41853]: debug 2022-01-31T20:55:18.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:18.577075+0000) 2022-01-31T20:55:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:18 smithi167 conmon[49112]: debug 2022-01-31T20:55:18.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:18.667580+0000) 2022-01-31T20:55:19.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:19 smithi167 conmon[54076]: debug 2022-01-31T20:55:19.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.342008+0000) 2022-01-31T20:55:19.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:19 smithi167 conmon[60316]: debug 2022-01-31T20:55:19.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.093763+0000) 2022-01-31T20:55:19.575 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[35325]: debug 2022-01-31T20:55:19.413+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88856 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:19.576 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[41853]: debug 2022-01-31T20:55:19.403+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.404553+0000) 2022-01-31T20:55:19.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[51620]: debug 2022-01-31T20:55:19.405+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.406140+0000) 2022-01-31T20:55:19.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[51620]: debug 2022-01-31T20:55:19.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.447386+0000) 2022-01-31T20:55:19.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[46715]: debug 2022-01-31T20:55:19.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.371445+0000) 2022-01-31T20:55:19.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[46715]: debug 2022-01-31T20:55:19.403+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.403917+0000) 2022-01-31T20:55:19.663 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:19 smithi167 conmon[49112]: debug 2022-01-31T20:55:19.405+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.405987+0000) 2022-01-31T20:55:19.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:19 smithi167 conmon[54076]: debug 2022-01-31T20:55:19.404+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.405383+0000) 2022-01-31T20:55:19.664 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:19 smithi167 conmon[60316]: debug 2022-01-31T20:55:19.405+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.405834+0000) 2022-01-31T20:55:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:19 smithi171 conmon[41853]: debug 2022-01-31T20:55:19.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.577278+0000) 2022-01-31T20:55:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:19 smithi167 conmon[49112]: debug 2022-01-31T20:55:19.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:19.667707+0000) 2022-01-31T20:55:20.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:20 smithi167 conmon[54076]: debug 2022-01-31T20:55:20.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:20.342227+0000) 2022-01-31T20:55:20.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:20 smithi167 conmon[60316]: debug 2022-01-31T20:55:20.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:20.093951+0000) 2022-01-31T20:55:20.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:20 smithi171 conmon[51620]: debug 2022-01-31T20:55:20.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:20.447491+0000) 2022-01-31T20:55:20.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:20 smithi171 conmon[46715]: debug 2022-01-31T20:55:20.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:20.371637+0000) 2022-01-31T20:55:20.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:20 smithi171 conmon[41853]: debug 2022-01-31T20:55:20.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:20.577430+0000) 2022-01-31T20:55:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:20 smithi167 conmon[49112]: debug 2022-01-31T20:55:20.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:20.667810+0000) 2022-01-31T20:55:21.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:21 smithi167 conmon[60316]: debug 2022-01-31T20:55:21.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:21.094079+0000) 2022-01-31T20:55:21.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:21 smithi167 conmon[54076]: debug 2022-01-31T20:55:21.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:21.342394+0000) 2022-01-31T20:55:21.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:21 smithi171 conmon[46715]: debug 2022-01-31T20:55:21.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:21.371772+0000) 2022-01-31T20:55:21.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:21 smithi171 conmon[51620]: debug 2022-01-31T20:55:21.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:21.447626+0000) 2022-01-31T20:55:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:21 smithi171 conmon[41853]: debug 2022-01-31T20:55:21.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:21.577605+0000) 2022-01-31T20:55:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:21 smithi167 conmon[49112]: debug 2022-01-31T20:55:21.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:21.667983+0000) 2022-01-31T20:55:22.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:22 smithi167 conmon[54076]: debug 2022-01-31T20:55:22.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:22.342583+0000) 2022-01-31T20:55:22.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:22 smithi167 conmon[60316]: debug 2022-01-31T20:55:22.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:22.094255+0000) 2022-01-31T20:55:22.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:22 smithi171 conmon[46715]: debug 2022-01-31T20:55:22.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:22.371925+0000) 2022-01-31T20:55:22.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:22 smithi171 conmon[51620]: debug 2022-01-31T20:55:22.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:22.447788+0000) 2022-01-31T20:55:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:22 smithi171 conmon[41853]: debug 2022-01-31T20:55:22.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:22.577820+0000) 2022-01-31T20:55:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:22 smithi167 conmon[49112]: debug 2022-01-31T20:55:22.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:22.668184+0000) 2022-01-31T20:55:23.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:23 smithi167 conmon[54076]: debug 2022-01-31T20:55:23.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:23.342726+0000) 2022-01-31T20:55:23.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:23 smithi167 conmon[60316]: debug 2022-01-31T20:55:23.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:23.094442+0000) 2022-01-31T20:55:23.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:23 smithi171 conmon[46715]: debug 2022-01-31T20:55:23.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:23.372109+0000) 2022-01-31T20:55:23.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:23 smithi171 conmon[51620]: debug 2022-01-31T20:55:23.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:23.447954+0000) 2022-01-31T20:55:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:23 smithi171 conmon[35325]: debug 2022-01-31T20:55:23.593+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:55:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:23 smithi171 conmon[41853]: debug 2022-01-31T20:55:23.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:23.578010+0000) 2022-01-31T20:55:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:23 smithi167 conmon[49112]: debug 2022-01-31T20:55:23.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:23.668385+0000) 2022-01-31T20:55:24.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:24 smithi167 conmon[54076]: debug 2022-01-31T20:55:24.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.342917+0000) 2022-01-31T20:55:24.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:24 smithi167 conmon[60316]: debug 2022-01-31T20:55:24.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.094659+0000) 2022-01-31T20:55:24.623 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[35325]: debug 2022-01-31T20:55:24.424+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 88966 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:24.623 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[41853]: debug 2022-01-31T20:55:24.415+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.416465+0000) 2022-01-31T20:55:24.624 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[46715]: debug 2022-01-31T20:55:24.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.372320+0000) 2022-01-31T20:55:24.624 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[46715]: debug 2022-01-31T20:55:24.415+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.416205+0000) 2022-01-31T20:55:24.625 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[51620]: debug 2022-01-31T20:55:24.416+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.417446+0000) 2022-01-31T20:55:24.625 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[51620]: debug 2022-01-31T20:55:24.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.448121+0000) 2022-01-31T20:55:24.660 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:24 smithi167 conmon[49112]: debug 2022-01-31T20:55:24.416+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.416590+0000) 2022-01-31T20:55:24.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:24 smithi167 conmon[54076]: debug 2022-01-31T20:55:24.416+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.417009+0000) 2022-01-31T20:55:24.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:24 smithi167 conmon[60316]: debug 2022-01-31T20:55:24.417+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.418021+0000) 2022-01-31T20:55:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:24 smithi171 conmon[41853]: debug 2022-01-31T20:55:24.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.578157+0000) 2022-01-31T20:55:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:24 smithi167 conmon[49112]: debug 2022-01-31T20:55:24.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:24.668586+0000) 2022-01-31T20:55:25.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:25 smithi167 conmon[54076]: debug 2022-01-31T20:55:25.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:25.343129+0000) 2022-01-31T20:55:25.389 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:25 smithi167 conmon[60316]: debug 2022-01-31T20:55:25.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:25.094818+0000) 2022-01-31T20:55:25.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:25 smithi171 conmon[46715]: debug 2022-01-31T20:55:25.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:25.372495+0000) 2022-01-31T20:55:25.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:25 smithi171 conmon[51620]: debug 2022-01-31T20:55:25.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:25.448297+0000) 2022-01-31T20:55:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:25 smithi171 conmon[41853]: debug 2022-01-31T20:55:25.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:25.578306+0000) 2022-01-31T20:55:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:25 smithi167 conmon[49112]: debug 2022-01-31T20:55:25.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:25.668676+0000) 2022-01-31T20:55:26.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:26 smithi167 conmon[54076]: debug 2022-01-31T20:55:26.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:26.343287+0000) 2022-01-31T20:55:26.389 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:26 smithi167 conmon[60316]: debug 2022-01-31T20:55:26.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:26.094952+0000) 2022-01-31T20:55:26.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:26 smithi171 conmon[46715]: debug 2022-01-31T20:55:26.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:26.372648+0000) 2022-01-31T20:55:26.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:26 smithi171 conmon[51620]: debug 2022-01-31T20:55:26.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:26.448458+0000) 2022-01-31T20:55:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:26 smithi171 conmon[41853]: debug 2022-01-31T20:55:26.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:26.578502+0000) 2022-01-31T20:55:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:26 smithi167 conmon[49112]: debug 2022-01-31T20:55:26.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:26.668799+0000) 2022-01-31T20:55:27.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:27 smithi167 conmon[54076]: debug 2022-01-31T20:55:27.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:27.343478+0000) 2022-01-31T20:55:27.389 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:27 smithi167 conmon[60316]: debug 2022-01-31T20:55:27.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:27.095102+0000) 2022-01-31T20:55:27.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:27 smithi171 conmon[46715]: debug 2022-01-31T20:55:27.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:27.372833+0000) 2022-01-31T20:55:27.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:27 smithi171 conmon[51620]: debug 2022-01-31T20:55:27.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:27.448604+0000) 2022-01-31T20:55:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:27 smithi171 conmon[41853]: debug 2022-01-31T20:55:27.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:27.578685+0000) 2022-01-31T20:55:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:27 smithi171 conmon[41853]: 2022-01-31T20:55:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:27 smithi167 conmon[49112]: debug 2022-01-31T20:55:27.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:27.668954+0000) 2022-01-31T20:55:28.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:28 smithi167 conmon[54076]: debug 2022-01-31T20:55:28.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:28.343657+0000) 2022-01-31T20:55:28.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:28 smithi167 conmon[60316]: debug 2022-01-31T20:55:28.093+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:28.095283+0000) 2022-01-31T20:55:28.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:28 smithi171 conmon[46715]: debug 2022-01-31T20:55:28.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:28.372972+0000) 2022-01-31T20:55:28.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:28 smithi171 conmon[51620]: debug 2022-01-31T20:55:28.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:28.448824+0000) 2022-01-31T20:55:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:28 smithi171 conmon[41853]: debug 2022-01-31T20:55:28.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:28.578864+0000) 2022-01-31T20:55:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:28 smithi167 conmon[49112]: debug 2022-01-31T20:55:28.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:28.669128+0000) 2022-01-31T20:55:29.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:29 smithi167 conmon[54076]: debug 2022-01-31T20:55:29.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.343812+0000) 2022-01-31T20:55:29.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:29 smithi167 conmon[60316]: debug 2022-01-31T20:55:29.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.095437+0000) 2022-01-31T20:55:29.577 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[35325]: debug 2022-01-31T20:55:29.452+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89076 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:29.578 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[41853]: debug 2022-01-31T20:55:29.426+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.427135+0000) 2022-01-31T20:55:29.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[46715]: debug 2022-01-31T20:55:29.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.373148+0000) 2022-01-31T20:55:29.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[46715]: debug 2022-01-31T20:55:29.426+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.431900+0000) 2022-01-31T20:55:29.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[51620]: debug 2022-01-31T20:55:29.426+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.428108+0000) 2022-01-31T20:55:29.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[51620]: debug 2022-01-31T20:55:29.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.449839+0000) 2022-01-31T20:55:29.661 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:29 smithi167 conmon[49112]: debug 2022-01-31T20:55:29.426+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.427945+0000) 2022-01-31T20:55:29.661 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:29 smithi167 conmon[54076]: debug 2022-01-31T20:55:29.426+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.427842+0000) 2022-01-31T20:55:29.662 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:29 smithi167 conmon[60316]: debug 2022-01-31T20:55:29.433+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.434564+0000) 2022-01-31T20:55:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:29 smithi171 conmon[41853]: debug 2022-01-31T20:55:29.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.579019+0000) 2022-01-31T20:55:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:29 smithi167 conmon[49112]: debug 2022-01-31T20:55:29.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:29.669336+0000) 2022-01-31T20:55:30.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:30 smithi167 conmon[54076]: debug 2022-01-31T20:55:30.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:30.344022+0000) 2022-01-31T20:55:30.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:30 smithi167 conmon[60316]: debug 2022-01-31T20:55:30.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:30.095656+0000) 2022-01-31T20:55:30.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:30 smithi171 conmon[46715]: debug 2022-01-31T20:55:30.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:30.373306+0000) 2022-01-31T20:55:30.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:30 smithi171 conmon[51620]: debug 2022-01-31T20:55:30.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:30.449929+0000) 2022-01-31T20:55:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:30 smithi171 conmon[41853]: debug 2022-01-31T20:55:30.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:30.579119+0000) 2022-01-31T20:55:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:30 smithi167 conmon[49112]: debug 2022-01-31T20:55:30.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:30.669491+0000) 2022-01-31T20:55:31.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:31 smithi167 conmon[54076]: debug 2022-01-31T20:55:31.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:31.344167+0000) 2022-01-31T20:55:31.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:31 smithi167 conmon[60316]: debug 2022-01-31T20:55:31.094+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:31.095862+0000) 2022-01-31T20:55:31.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:31 smithi171 conmon[46715]: debug 2022-01-31T20:55:31.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:31.373494+0000) 2022-01-31T20:55:31.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:31 smithi171 conmon[51620]: debug 2022-01-31T20:55:31.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:31.450046+0000) 2022-01-31T20:55:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:31 smithi171 conmon[41853]: debug 2022-01-31T20:55:31.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:31.579299+0000) 2022-01-31T20:55:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:31 smithi167 conmon[49112]: debug 2022-01-31T20:55:31.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:31.669647+0000) 2022-01-31T20:55:32.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:32 smithi167 conmon[54076]: debug 2022-01-31T20:55:32.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:32.344346+0000) 2022-01-31T20:55:32.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:32 smithi167 conmon[60316]: debug 2022-01-31T20:55:32.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:32.096084+0000) 2022-01-31T20:55:32.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:32 smithi171 conmon[46715]: debug 2022-01-31T20:55:32.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:32.373710+0000) 2022-01-31T20:55:32.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:32 smithi171 conmon[51620]: debug 2022-01-31T20:55:32.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:32.450244+0000) 2022-01-31T20:55:32.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:32 smithi171 conmon[41853]: debug 2022-01-31T20:55:32.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:32.579485+0000) 2022-01-31T20:55:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:32 smithi167 conmon[49112]: debug 2022-01-31T20:55:32.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:32.669829+0000) 2022-01-31T20:55:33.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:33 smithi167 conmon[54076]: debug 2022-01-31T20:55:33.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:33.344443+0000) 2022-01-31T20:55:33.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:33 smithi167 conmon[60316]: debug 2022-01-31T20:55:33.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:33.096291+0000) 2022-01-31T20:55:33.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:33 smithi171 conmon[46715]: debug 2022-01-31T20:55:33.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:33.373936+0000) 2022-01-31T20:55:33.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:33 smithi171 conmon[51620]: debug 2022-01-31T20:55:33.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:33.450429+0000) 2022-01-31T20:55:33.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:33 smithi171 conmon[41853]: debug 2022-01-31T20:55:33.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:33.579667+0000) 2022-01-31T20:55:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:33 smithi167 conmon[49112]: debug 2022-01-31T20:55:33.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:33.669982+0000) 2022-01-31T20:55:34.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:34 smithi167 conmon[54076]: debug 2022-01-31T20:55:34.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.344601+0000) 2022-01-31T20:55:34.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:34 smithi167 conmon[60316]: debug 2022-01-31T20:55:34.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.096484+0000) 2022-01-31T20:55:34.578 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[35325]: debug 2022-01-31T20:55:34.463+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89186 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:34.579 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[41853]: debug 2022-01-31T20:55:34.454+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.455197+0000) 2022-01-31T20:55:34.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[46715]: debug 2022-01-31T20:55:34.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.374155+0000) 2022-01-31T20:55:34.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[46715]: debug 2022-01-31T20:55:34.454+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.455621+0000) 2022-01-31T20:55:34.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[51620]: debug 2022-01-31T20:55:34.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.450636+0000) 2022-01-31T20:55:34.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[51620]: debug 2022-01-31T20:55:34.455+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.456275+0000) 2022-01-31T20:55:34.662 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:34 smithi167 conmon[49112]: debug 2022-01-31T20:55:34.454+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.456351+0000) 2022-01-31T20:55:34.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:34 smithi167 conmon[54076]: debug 2022-01-31T20:55:34.454+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.456046+0000) 2022-01-31T20:55:34.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:34 smithi167 conmon[60316]: debug 2022-01-31T20:55:34.455+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.457445+0000) 2022-01-31T20:55:34.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:34 smithi171 conmon[41853]: debug 2022-01-31T20:55:34.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.579832+0000) 2022-01-31T20:55:34.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:34 smithi167 conmon[49112]: debug 2022-01-31T20:55:34.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:34.670189+0000) 2022-01-31T20:55:35.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:35 smithi167 conmon[54076]: debug 2022-01-31T20:55:35.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:35.344784+0000) 2022-01-31T20:55:35.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:35 smithi167 conmon[60316]: debug 2022-01-31T20:55:35.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:35.096638+0000) 2022-01-31T20:55:35.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:35 smithi171 conmon[46715]: debug 2022-01-31T20:55:35.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:35.374292+0000) 2022-01-31T20:55:35.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:35 smithi171 conmon[51620]: debug 2022-01-31T20:55:35.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:35.450867+0000) 2022-01-31T20:55:35.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:35 smithi171 conmon[41853]: debug 2022-01-31T20:55:35.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:35.579969+0000) 2022-01-31T20:55:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:35 smithi167 conmon[49112]: debug 2022-01-31T20:55:35.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:35.670284+0000) 2022-01-31T20:55:36.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:36 smithi167 conmon[54076]: debug 2022-01-31T20:55:36.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:36.344956+0000) 2022-01-31T20:55:36.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:36 smithi167 conmon[60316]: debug 2022-01-31T20:55:36.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:36.096805+0000) 2022-01-31T20:55:36.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:36 smithi171 conmon[46715]: debug 2022-01-31T20:55:36.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:36.374427+0000) 2022-01-31T20:55:36.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:36 smithi171 conmon[51620]: debug 2022-01-31T20:55:36.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:36.450948+0000) 2022-01-31T20:55:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:36 smithi171 conmon[41853]: debug 2022-01-31T20:55:36.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:36.580089+0000) 2022-01-31T20:55:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:36 smithi167 conmon[49112]: debug 2022-01-31T20:55:36.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:36.670454+0000) 2022-01-31T20:55:37.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:37 smithi167 conmon[54076]: debug 2022-01-31T20:55:37.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:37.345177+0000) 2022-01-31T20:55:37.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:37 smithi167 conmon[60316]: debug 2022-01-31T20:55:37.096+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:37.096998+0000) 2022-01-31T20:55:37.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:37 smithi171 conmon[46715]: debug 2022-01-31T20:55:37.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:37.374613+0000) 2022-01-31T20:55:37.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:37 smithi171 conmon[51620]: debug 2022-01-31T20:55:37.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:37.451062+0000) 2022-01-31T20:55:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:37 smithi171 conmon[41853]: debug 2022-01-31T20:55:37.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:37.580260+0000) 2022-01-31T20:55:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:37 smithi167 conmon[49112]: debug 2022-01-31T20:55:37.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:37.670594+0000) 2022-01-31T20:55:38.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:38 smithi167 conmon[54076]: debug 2022-01-31T20:55:38.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:38.345348+0000) 2022-01-31T20:55:38.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:38 smithi167 conmon[60316]: debug 2022-01-31T20:55:38.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:38.097226+0000) 2022-01-31T20:55:38.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:38 smithi171 conmon[46715]: debug 2022-01-31T20:55:38.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:38.374849+0000) 2022-01-31T20:55:38.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:38 smithi171 conmon[51620]: debug 2022-01-31T20:55:38.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:38.451278+0000) 2022-01-31T20:55:38.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:38 smithi171 conmon[41853]: debug 2022-01-31T20:55:38.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:38.580427+0000) 2022-01-31T20:55:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:38 smithi171 conmon[35325]: debug 2022-01-31T20:55:38.594+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:55:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:38 smithi167 conmon[49112]: debug 2022-01-31T20:55:38.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:38.670735+0000) 2022-01-31T20:55:39.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:39 smithi167 conmon[54076]: debug 2022-01-31T20:55:39.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.345497+0000) 2022-01-31T20:55:39.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:39 smithi167 conmon[60316]: debug 2022-01-31T20:55:39.095+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.097430+0000) 2022-01-31T20:55:39.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[46715]: debug 2022-01-31T20:55:39.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.374976+0000) 2022-01-31T20:55:39.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[46715]: debug 2022-01-31T20:55:39.465+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.466510+0000) 2022-01-31T20:55:39.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[51620]: debug 2022-01-31T20:55:39.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.451414+0000) 2022-01-31T20:55:39.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[51620]: debug 2022-01-31T20:55:39.466+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.466897+0000) 2022-01-31T20:55:39.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[41853]: debug 2022-01-31T20:55:39.465+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.466225+0000) 2022-01-31T20:55:39.663 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:39 smithi167 conmon[49112]: debug 2022-01-31T20:55:39.465+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.467186+0000) 2022-01-31T20:55:39.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:39 smithi167 conmon[54076]: debug 2022-01-31T20:55:39.465+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.466858+0000) 2022-01-31T20:55:39.664 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:39 smithi167 conmon[60316]: debug 2022-01-31T20:55:39.466+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.468053+0000) 2022-01-31T20:55:39.787 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[35325]: debug 2022-01-31T20:55:39.476+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89296 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:39.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:39 smithi171 conmon[41853]: debug 2022-01-31T20:55:39.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.580612+0000) 2022-01-31T20:55:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:39 smithi167 conmon[49112]: debug 2022-01-31T20:55:39.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:39.670902+0000) 2022-01-31T20:55:40.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:40 smithi167 conmon[54076]: debug 2022-01-31T20:55:40.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:40.345669+0000) 2022-01-31T20:55:40.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:40 smithi167 conmon[60316]: debug 2022-01-31T20:55:40.096+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:40.097623+0000) 2022-01-31T20:55:40.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:40 smithi171 conmon[46715]: debug 2022-01-31T20:55:40.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:40.375139+0000) 2022-01-31T20:55:40.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:40 smithi171 conmon[51620]: debug 2022-01-31T20:55:40.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:40.451593+0000) 2022-01-31T20:55:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:40 smithi171 conmon[41853]: debug 2022-01-31T20:55:40.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:40.580774+0000) 2022-01-31T20:55:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:40 smithi167 conmon[49112]: debug 2022-01-31T20:55:40.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:40.671063+0000) 2022-01-31T20:55:41.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:41 smithi167 conmon[54076]: debug 2022-01-31T20:55:41.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:41.345782+0000) 2022-01-31T20:55:41.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:41 smithi167 conmon[60316]: debug 2022-01-31T20:55:41.097+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:41.097772+0000) 2022-01-31T20:55:41.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:41 smithi171 conmon[46715]: debug 2022-01-31T20:55:41.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:41.375297+0000) 2022-01-31T20:55:41.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:41 smithi171 conmon[51620]: debug 2022-01-31T20:55:41.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:41.451811+0000) 2022-01-31T20:55:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:41 smithi171 conmon[41853]: debug 2022-01-31T20:55:41.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:41.580999+0000) 2022-01-31T20:55:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:41 smithi167 conmon[49112]: debug 2022-01-31T20:55:41.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:41.671191+0000) 2022-01-31T20:55:42.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:42 smithi167 conmon[60316]: debug 2022-01-31T20:55:42.097+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:42.097962+0000) 2022-01-31T20:55:42.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:42 smithi167 conmon[54076]: debug 2022-01-31T20:55:42.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:42.345964+0000) 2022-01-31T20:55:42.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:42 smithi171 conmon[46715]: debug 2022-01-31T20:55:42.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:42.375486+0000) 2022-01-31T20:55:42.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:42 smithi171 conmon[51620]: debug 2022-01-31T20:55:42.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:42.452015+0000) 2022-01-31T20:55:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:42 smithi171 conmon[41853]: debug 2022-01-31T20:55:42.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:42.581213+0000) 2022-01-31T20:55:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:42 smithi167 conmon[49112]: debug 2022-01-31T20:55:42.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:42.671364+0000) 2022-01-31T20:55:43.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:43 smithi167 conmon[54076]: debug 2022-01-31T20:55:43.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:43.346147+0000) 2022-01-31T20:55:43.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:43 smithi167 conmon[60316]: debug 2022-01-31T20:55:43.097+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:43.098149+0000) 2022-01-31T20:55:43.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:43 smithi171 conmon[46715]: debug 2022-01-31T20:55:43.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:43.375663+0000) 2022-01-31T20:55:43.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:43 smithi171 conmon[51620]: debug 2022-01-31T20:55:43.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:43.452217+0000) 2022-01-31T20:55:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:43 smithi171 conmon[41853]: debug 2022-01-31T20:55:43.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:43.581371+0000) 2022-01-31T20:55:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:43 smithi167 conmon[49112]: debug 2022-01-31T20:55:43.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:43.671592+0000) 2022-01-31T20:55:44.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:44 smithi167 conmon[54076]: debug 2022-01-31T20:55:44.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.346342+0000) 2022-01-31T20:55:44.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:44 smithi167 conmon[60316]: debug 2022-01-31T20:55:44.097+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.098301+0000) 2022-01-31T20:55:44.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[46715]: debug 2022-01-31T20:55:44.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.375915+0000) 2022-01-31T20:55:44.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[51620]: debug 2022-01-31T20:55:44.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.452398+0000) 2022-01-31T20:55:44.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[41853]: debug 2022-01-31T20:55:44.477+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.478484+0000) 2022-01-31T20:55:44.663 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:44 smithi167 conmon[49112]: debug 2022-01-31T20:55:44.478+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.479553+0000) 2022-01-31T20:55:44.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:44 smithi167 conmon[54076]: debug 2022-01-31T20:55:44.477+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.479446+0000) 2022-01-31T20:55:44.664 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:44 smithi167 conmon[60316]: debug 2022-01-31T20:55:44.478+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.480139+0000) 2022-01-31T20:55:44.787 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[35325]: debug 2022-01-31T20:55:44.487+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89406 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:44.788 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[46715]: debug 2022-01-31T20:55:44.478+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.479004+0000) 2022-01-31T20:55:44.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[41853]: debug 2022-01-31T20:55:44.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.581530+0000) 2022-01-31T20:55:44.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:44 smithi171 conmon[51620]: debug 2022-01-31T20:55:44.478+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.479522+0000) 2022-01-31T20:55:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:44 smithi167 conmon[49112]: debug 2022-01-31T20:55:44.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:44.671702+0000) 2022-01-31T20:55:45.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:45 smithi167 conmon[60316]: debug 2022-01-31T20:55:45.097+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:45.098486+0000) 2022-01-31T20:55:45.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:45 smithi167 conmon[54076]: debug 2022-01-31T20:55:45.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:45.346536+0000) 2022-01-31T20:55:45.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:45 smithi171 conmon[46715]: debug 2022-01-31T20:55:45.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:45.376129+0000) 2022-01-31T20:55:45.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:45 smithi171 conmon[51620]: debug 2022-01-31T20:55:45.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:45.452587+0000) 2022-01-31T20:55:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:45 smithi171 conmon[41853]: debug 2022-01-31T20:55:45.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:45.581704+0000) 2022-01-31T20:55:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:45 smithi167 conmon[49112]: debug 2022-01-31T20:55:45.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:45.671822+0000) 2022-01-31T20:55:46.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:46 smithi167 conmon[54076]: debug 2022-01-31T20:55:46.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:46.346650+0000) 2022-01-31T20:55:46.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:46 smithi167 conmon[60316]: debug 2022-01-31T20:55:46.098+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:46.098681+0000) 2022-01-31T20:55:46.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:46 smithi171 conmon[46715]: debug 2022-01-31T20:55:46.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:46.376291+0000) 2022-01-31T20:55:46.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:46 smithi171 conmon[51620]: debug 2022-01-31T20:55:46.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:46.452692+0000) 2022-01-31T20:55:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:46 smithi171 conmon[41853]: debug 2022-01-31T20:55:46.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:46.581859+0000) 2022-01-31T20:55:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:46 smithi167 conmon[49112]: debug 2022-01-31T20:55:46.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:46.672000+0000) 2022-01-31T20:55:47.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:47 smithi167 conmon[54076]: debug 2022-01-31T20:55:47.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:47.346802+0000) 2022-01-31T20:55:47.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:47 smithi167 conmon[60316]: debug 2022-01-31T20:55:47.098+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:47.098853+0000) 2022-01-31T20:55:47.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:47 smithi171 conmon[46715]: debug 2022-01-31T20:55:47.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:47.376477+0000) 2022-01-31T20:55:47.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:47 smithi171 conmon[51620]: debug 2022-01-31T20:55:47.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:47.452905+0000) 2022-01-31T20:55:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:47 smithi171 conmon[41853]: debug 2022-01-31T20:55:47.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:47.582043+0000) 2022-01-31T20:55:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:47 smithi167 conmon[49112]: debug 2022-01-31T20:55:47.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:47.672188+0000) 2022-01-31T20:55:48.218 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T20:55:48.219+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T20:55:48.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:48 smithi167 conmon[54076]: debug 2022-01-31T20:55:48.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:48.347013+0000) 2022-01-31T20:55:48.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:48 smithi167 conmon[60316]: debug 2022-01-31T20:55:48.097+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:48.098981+0000) 2022-01-31T20:55:48.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:48 smithi171 conmon[46715]: debug 2022-01-31T20:55:48.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:48.376671+0000) 2022-01-31T20:55:48.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:48 smithi171 conmon[51620]: debug 2022-01-31T20:55:48.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:48.453078+0000) 2022-01-31T20:55:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:48 smithi171 conmon[41853]: debug 2022-01-31T20:55:48.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:48.582221+0000) 2022-01-31T20:55:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:48 smithi167 conmon[49112]: debug 2022-01-31T20:55:48.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:48.672370+0000) 2022-01-31T20:55:49.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:49 smithi167 conmon[54076]: debug 2022-01-31T20:55:49.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.347171+0000) 2022-01-31T20:55:49.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:49 smithi167 conmon[60316]: debug 2022-01-31T20:55:49.098+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.099152+0000) 2022-01-31T20:55:49.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[46715]: debug 2022-01-31T20:55:49.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.376857+0000) 2022-01-31T20:55:49.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[51620]: debug 2022-01-31T20:55:49.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.453303+0000) 2022-01-31T20:55:49.664 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:49 smithi167 conmon[49112]: debug 2022-01-31T20:55:49.488+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.489877+0000) 2022-01-31T20:55:49.665 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:49 smithi167 conmon[60316]: debug 2022-01-31T20:55:49.489+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.490846+0000) 2022-01-31T20:55:49.665 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:49 smithi167 conmon[54076]: debug 2022-01-31T20:55:49.490+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.491593+0000) 2022-01-31T20:55:49.788 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[46715]: debug 2022-01-31T20:55:49.488+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.490393+0000) 2022-01-31T20:55:49.789 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[35325]: debug 2022-01-31T20:55:49.498+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89516 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:49.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[41853]: debug 2022-01-31T20:55:49.489+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.490789+0000) 2022-01-31T20:55:49.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[41853]: debug 2022-01-31T20:55:49.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.582385+0000) 2022-01-31T20:55:49.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:49 smithi171 conmon[51620]: debug 2022-01-31T20:55:49.488+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.490176+0000) 2022-01-31T20:55:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:49 smithi167 conmon[49112]: debug 2022-01-31T20:55:49.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:49.672555+0000) 2022-01-31T20:55:50.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:50 smithi167 conmon[54076]: debug 2022-01-31T20:55:50.346+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:50.347370+0000) 2022-01-31T20:55:50.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:50 smithi167 conmon[60316]: debug 2022-01-31T20:55:50.098+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:50.099327+0000) 2022-01-31T20:55:50.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:50 smithi171 conmon[46715]: debug 2022-01-31T20:55:50.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:50.377007+0000) 2022-01-31T20:55:50.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:50 smithi171 conmon[51620]: debug 2022-01-31T20:55:50.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:50.453474+0000) 2022-01-31T20:55:50.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:50 smithi171 conmon[41853]: debug 2022-01-31T20:55:50.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:50.582547+0000) 2022-01-31T20:55:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:50 smithi167 conmon[49112]: debug 2022-01-31T20:55:50.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:50.672675+0000) 2022-01-31T20:55:51.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:51 smithi167 conmon[54076]: debug 2022-01-31T20:55:51.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:51.347540+0000) 2022-01-31T20:55:51.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:51 smithi167 conmon[60316]: debug 2022-01-31T20:55:51.098+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:51.099475+0000) 2022-01-31T20:55:51.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:51 smithi171 conmon[46715]: debug 2022-01-31T20:55:51.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:51.377171+0000) 2022-01-31T20:55:51.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:51 smithi171 conmon[51620]: debug 2022-01-31T20:55:51.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:51.453622+0000) 2022-01-31T20:55:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:51 smithi171 conmon[41853]: debug 2022-01-31T20:55:51.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:51.582778+0000) 2022-01-31T20:55:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:51 smithi167 conmon[49112]: debug 2022-01-31T20:55:51.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:51.672829+0000) 2022-01-31T20:55:52.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:52 smithi167 conmon[60316]: debug 2022-01-31T20:55:52.099+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:52.099603+0000) 2022-01-31T20:55:52.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:52 smithi167 conmon[54076]: debug 2022-01-31T20:55:52.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:52.347719+0000) 2022-01-31T20:55:52.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:52 smithi171 conmon[46715]: debug 2022-01-31T20:55:52.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:52.377377+0000) 2022-01-31T20:55:52.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:52 smithi171 conmon[51620]: debug 2022-01-31T20:55:52.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:52.453744+0000) 2022-01-31T20:55:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:52 smithi171 conmon[41853]: debug 2022-01-31T20:55:52.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:52.582923+0000) 2022-01-31T20:55:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:52 smithi167 conmon[49112]: debug 2022-01-31T20:55:52.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:52.673026+0000) 2022-01-31T20:55:53.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:53 smithi167 conmon[54076]: debug 2022-01-31T20:55:53.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:53.347921+0000) 2022-01-31T20:55:53.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:53 smithi167 conmon[60316]: debug 2022-01-31T20:55:53.099+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:53.099786+0000) 2022-01-31T20:55:53.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:53 smithi171 conmon[46715]: debug 2022-01-31T20:55:53.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:53.377589+0000) 2022-01-31T20:55:53.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:53 smithi171 conmon[51620]: debug 2022-01-31T20:55:53.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:53.453875+0000) 2022-01-31T20:55:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:53 smithi171 conmon[41853]: debug 2022-01-31T20:55:53.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:53.583080+0000) 2022-01-31T20:55:53.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:53 smithi171 conmon[35325]: debug 2022-01-31T20:55:53.594+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:55:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:53 smithi167 conmon[49112]: debug 2022-01-31T20:55:53.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:53.673271+0000) 2022-01-31T20:55:54.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:54 smithi167 conmon[60316]: debug 2022-01-31T20:55:54.099+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.099972+0000) 2022-01-31T20:55:54.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:54 smithi167 conmon[54076]: debug 2022-01-31T20:55:54.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.348117+0000) 2022-01-31T20:55:54.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[46715]: debug 2022-01-31T20:55:54.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.377776+0000) 2022-01-31T20:55:54.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[51620]: debug 2022-01-31T20:55:54.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.454084+0000) 2022-01-31T20:55:54.665 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:54 smithi167 conmon[49112]: debug 2022-01-31T20:55:54.503+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.503623+0000) 2022-01-31T20:55:54.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:54 smithi167 conmon[54076]: debug 2022-01-31T20:55:54.502+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.502826+0000) 2022-01-31T20:55:54.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:54 smithi167 conmon[60316]: debug 2022-01-31T20:55:54.501+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.502094+0000) 2022-01-31T20:55:54.789 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[35325]: debug 2022-01-31T20:55:54.509+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89626 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:54.790 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[46715]: debug 2022-01-31T20:55:54.500+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.502458+0000) 2022-01-31T20:55:54.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[51620]: debug 2022-01-31T20:55:54.500+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.502441+0000) 2022-01-31T20:55:54.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[41853]: debug 2022-01-31T20:55:54.500+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.501778+0000) 2022-01-31T20:55:54.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:54 smithi171 conmon[41853]: debug 2022-01-31T20:55:54.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.583270+0000) 2022-01-31T20:55:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:54 smithi167 conmon[49112]: debug 2022-01-31T20:55:54.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:54.673400+0000) 2022-01-31T20:55:55.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:55 smithi167 conmon[54076]: debug 2022-01-31T20:55:55.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:55.348297+0000) 2022-01-31T20:55:55.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:55 smithi167 conmon[60316]: debug 2022-01-31T20:55:55.099+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:55.100110+0000) 2022-01-31T20:55:55.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:55 smithi171 conmon[46715]: debug 2022-01-31T20:55:55.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:55.377971+0000) 2022-01-31T20:55:55.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:55 smithi171 conmon[51620]: debug 2022-01-31T20:55:55.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:55.454268+0000) 2022-01-31T20:55:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:55 smithi171 conmon[41853]: debug 2022-01-31T20:55:55.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:55.583458+0000) 2022-01-31T20:55:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:55 smithi167 conmon[49112]: debug 2022-01-31T20:55:55.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:55.673586+0000) 2022-01-31T20:55:56.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:56 smithi167 conmon[54076]: debug 2022-01-31T20:55:56.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:56.348449+0000) 2022-01-31T20:55:56.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:56 smithi167 conmon[60316]: debug 2022-01-31T20:55:56.099+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:56.100263+0000) 2022-01-31T20:55:56.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:56 smithi171 conmon[46715]: debug 2022-01-31T20:55:56.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:56.378110+0000) 2022-01-31T20:55:56.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:56 smithi171 conmon[51620]: debug 2022-01-31T20:55:56.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:56.454431+0000) 2022-01-31T20:55:56.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:56 smithi171 conmon[41853]: debug 2022-01-31T20:55:56.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:56.583647+0000) 2022-01-31T20:55:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:56 smithi167 conmon[49112]: debug 2022-01-31T20:55:56.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:56.673698+0000) 2022-01-31T20:55:57.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:57 smithi167 conmon[54076]: debug 2022-01-31T20:55:57.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:57.348610+0000) 2022-01-31T20:55:57.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:57 smithi167 conmon[60316]: debug 2022-01-31T20:55:57.100+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:57.100450+0000) 2022-01-31T20:55:57.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:57 smithi171 conmon[46715]: debug 2022-01-31T20:55:57.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:57.378267+0000) 2022-01-31T20:55:57.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:57 smithi171 conmon[51620]: debug 2022-01-31T20:55:57.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:57.454591+0000) 2022-01-31T20:55:57.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:57 smithi171 conmon[41853]: debug 2022-01-31T20:55:57.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:57.583833+0000) 2022-01-31T20:55:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:57 smithi167 conmon[49112]: debug 2022-01-31T20:55:57.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:57.673823+0000) 2022-01-31T20:55:58.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:58 smithi167 conmon[54076]: debug 2022-01-31T20:55:58.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:58.348814+0000) 2022-01-31T20:55:58.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:58 smithi167 conmon[60316]: debug 2022-01-31T20:55:58.100+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:58.100607+0000) 2022-01-31T20:55:58.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:58 smithi171 conmon[46715]: debug 2022-01-31T20:55:58.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:58.378471+0000) 2022-01-31T20:55:58.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:58 smithi171 conmon[51620]: debug 2022-01-31T20:55:58.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:58.454816+0000) 2022-01-31T20:55:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:58 smithi171 conmon[41853]: debug 2022-01-31T20:55:58.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:58.584036+0000) 2022-01-31T20:55:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:58 smithi167 conmon[49112]: debug 2022-01-31T20:55:58.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:58.674012+0000) 2022-01-31T20:55:59.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:59 smithi167 conmon[60316]: debug 2022-01-31T20:55:59.100+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.100766+0000) 2022-01-31T20:55:59.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:59 smithi167 conmon[54076]: debug 2022-01-31T20:55:59.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.349028+0000) 2022-01-31T20:55:59.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[46715]: debug 2022-01-31T20:55:59.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.378650+0000) 2022-01-31T20:55:59.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[51620]: debug 2022-01-31T20:55:59.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.454961+0000) 2022-01-31T20:55:59.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:55:59 smithi167 conmon[54076]: debug 2022-01-31T20:55:59.511+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.513682+0000) 2022-01-31T20:55:59.666 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:59 smithi167 conmon[49112]: debug 2022-01-31T20:55:59.511+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.515417+0000) 2022-01-31T20:55:59.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:55:59 smithi167 conmon[60316]: debug 2022-01-31T20:55:59.517+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.517613+0000) 2022-01-31T20:55:59.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[41853]: debug 2022-01-31T20:55:59.513+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.514140+0000) 2022-01-31T20:55:59.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[41853]: debug 2022-01-31T20:55:59.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.584190+0000) 2022-01-31T20:55:59.791 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[46715]: debug 2022-01-31T20:55:59.512+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.513365+0000) 2022-01-31T20:55:59.791 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[35325]: debug 2022-01-31T20:55:59.522+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89736 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:55:59.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:55:59 smithi171 conmon[51620]: debug 2022-01-31T20:55:59.513+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.514231+0000) 2022-01-31T20:55:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:55:59 smithi167 conmon[49112]: debug 2022-01-31T20:55:59.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:55:59.674204+0000) 2022-01-31T20:56:00.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:00 smithi167 conmon[54076]: debug 2022-01-31T20:56:00.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:00.349229+0000) 2022-01-31T20:56:00.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:00 smithi167 conmon[60316]: debug 2022-01-31T20:56:00.100+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:00.100982+0000) 2022-01-31T20:56:00.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:00 smithi171 conmon[46715]: debug 2022-01-31T20:56:00.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:00.378841+0000) 2022-01-31T20:56:00.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:00 smithi171 conmon[51620]: debug 2022-01-31T20:56:00.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:00.455141+0000) 2022-01-31T20:56:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:00 smithi171 conmon[41853]: debug 2022-01-31T20:56:00.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:00.584356+0000) 2022-01-31T20:56:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:00 smithi167 conmon[49112]: debug 2022-01-31T20:56:00.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:00.674365+0000) 2022-01-31T20:56:01.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:01 smithi167 conmon[60316]: debug 2022-01-31T20:56:01.100+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:01.101128+0000) 2022-01-31T20:56:01.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:01 smithi167 conmon[54076]: debug 2022-01-31T20:56:01.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:01.349387+0000) 2022-01-31T20:56:01.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:01 smithi171 conmon[46715]: debug 2022-01-31T20:56:01.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:01.378994+0000) 2022-01-31T20:56:01.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:01 smithi171 conmon[51620]: debug 2022-01-31T20:56:01.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:01.455285+0000) 2022-01-31T20:56:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:01 smithi171 conmon[41853]: debug 2022-01-31T20:56:01.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:01.584535+0000) 2022-01-31T20:56:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:01 smithi167 conmon[49112]: debug 2022-01-31T20:56:01.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:01.674579+0000) 2022-01-31T20:56:02.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:02 smithi167 conmon[54076]: debug 2022-01-31T20:56:02.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:02.349597+0000) 2022-01-31T20:56:02.396 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:02 smithi167 conmon[60316]: debug 2022-01-31T20:56:02.100+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:02.101285+0000) 2022-01-31T20:56:02.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:02 smithi171 conmon[46715]: debug 2022-01-31T20:56:02.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:02.379176+0000) 2022-01-31T20:56:02.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:02 smithi171 conmon[51620]: debug 2022-01-31T20:56:02.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:02.455495+0000) 2022-01-31T20:56:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:02 smithi171 conmon[41853]: debug 2022-01-31T20:56:02.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:02.584720+0000) 2022-01-31T20:56:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:02 smithi167 conmon[49112]: debug 2022-01-31T20:56:02.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:02.674717+0000) 2022-01-31T20:56:03.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:03 smithi167 conmon[60316]: debug 2022-01-31T20:56:03.101+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:03.101469+0000) 2022-01-31T20:56:03.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:03 smithi167 conmon[54076]: debug 2022-01-31T20:56:03.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:03.349802+0000) 2022-01-31T20:56:03.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:03 smithi171 conmon[46715]: debug 2022-01-31T20:56:03.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:03.379399+0000) 2022-01-31T20:56:03.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:03 smithi171 conmon[51620]: debug 2022-01-31T20:56:03.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:03.455654+0000) 2022-01-31T20:56:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:03 smithi171 conmon[41853]: debug 2022-01-31T20:56:03.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:03.584887+0000) 2022-01-31T20:56:03.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:03 smithi167 conmon[49112]: debug 2022-01-31T20:56:03.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:03.674828+0000) 2022-01-31T20:56:04.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:04 smithi167 conmon[54076]: debug 2022-01-31T20:56:04.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.350002+0000) 2022-01-31T20:56:04.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:04 smithi167 conmon[60316]: debug 2022-01-31T20:56:04.101+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.101635+0000) 2022-01-31T20:56:04.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[46715]: debug 2022-01-31T20:56:04.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.379601+0000) 2022-01-31T20:56:04.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[51620]: debug 2022-01-31T20:56:04.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.455812+0000) 2022-01-31T20:56:04.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:04 smithi167 conmon[49112]: debug 2022-01-31T20:56:04.526+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.526482+0000) 2022-01-31T20:56:04.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:04 smithi167 conmon[54076]: debug 2022-01-31T20:56:04.525+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.525555+0000) 2022-01-31T20:56:04.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:04 smithi167 conmon[60316]: debug 2022-01-31T20:56:04.526+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.526698+0000) 2022-01-31T20:56:04.791 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[35325]: debug 2022-01-31T20:56:04.534+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89846 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:04.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[41853]: debug 2022-01-31T20:56:04.525+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.526412+0000) 2022-01-31T20:56:04.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[41853]: debug 2022-01-31T20:56:04.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.585099+0000) 2022-01-31T20:56:04.793 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[46715]: debug 2022-01-31T20:56:04.524+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.525089+0000) 2022-01-31T20:56:04.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:04 smithi171 conmon[51620]: debug 2022-01-31T20:56:04.524+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.525239+0000) 2022-01-31T20:56:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:04 smithi167 conmon[49112]: debug 2022-01-31T20:56:04.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:04.674993+0000) 2022-01-31T20:56:05.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:05 smithi167 conmon[54076]: debug 2022-01-31T20:56:05.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:05.350143+0000) 2022-01-31T20:56:05.664 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:05 smithi167 conmon[60316]: debug 2022-01-31T20:56:05.101+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:05.101820+0000) 2022-01-31T20:56:05.666 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:05 smithi171 conmon[46715]: debug 2022-01-31T20:56:05.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:05.379810+0000) 2022-01-31T20:56:05.666 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:05 smithi171 conmon[51620]: debug 2022-01-31T20:56:05.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:05.455972+0000) 2022-01-31T20:56:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:05 smithi171 conmon[41853]: debug 2022-01-31T20:56:05.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:05.585246+0000) 2022-01-31T20:56:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:05 smithi167 conmon[49112]: debug 2022-01-31T20:56:05.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:05.675156+0000) 2022-01-31T20:56:06.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:06 smithi167 conmon[54076]: debug 2022-01-31T20:56:06.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:06.350301+0000) 2022-01-31T20:56:06.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:06 smithi167 conmon[60316]: debug 2022-01-31T20:56:06.101+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:06.101979+0000) 2022-01-31T20:56:06.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:06 smithi171 conmon[46715]: debug 2022-01-31T20:56:06.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:06.379976+0000) 2022-01-31T20:56:06.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:06 smithi171 conmon[51620]: debug 2022-01-31T20:56:06.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:06.456123+0000) 2022-01-31T20:56:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:06 smithi171 conmon[41853]: debug 2022-01-31T20:56:06.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:06.585434+0000) 2022-01-31T20:56:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:06 smithi167 conmon[49112]: debug 2022-01-31T20:56:06.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:06.675314+0000) 2022-01-31T20:56:07.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:07 smithi167 conmon[54076]: debug 2022-01-31T20:56:07.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:07.350558+0000) 2022-01-31T20:56:07.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:07 smithi167 conmon[60316]: debug 2022-01-31T20:56:07.101+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:07.102202+0000) 2022-01-31T20:56:07.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:07 smithi171 conmon[46715]: debug 2022-01-31T20:56:07.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:07.380122+0000) 2022-01-31T20:56:07.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:07 smithi171 conmon[51620]: debug 2022-01-31T20:56:07.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:07.456344+0000) 2022-01-31T20:56:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:07 smithi171 conmon[41853]: debug 2022-01-31T20:56:07.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:07.585636+0000) 2022-01-31T20:56:08.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:07 smithi167 conmon[49112]: debug 2022-01-31T20:56:07.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:07.675494+0000) 2022-01-31T20:56:08.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:08 smithi167 conmon[54076]: debug 2022-01-31T20:56:08.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:08.350767+0000) 2022-01-31T20:56:08.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:08 smithi167 conmon[60316]: debug 2022-01-31T20:56:08.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:08.102381+0000) 2022-01-31T20:56:08.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:08 smithi171 conmon[46715]: debug 2022-01-31T20:56:08.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:08.380340+0000) 2022-01-31T20:56:08.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:08 smithi171 conmon[51620]: debug 2022-01-31T20:56:08.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:08.456530+0000) 2022-01-31T20:56:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:08 smithi171 conmon[35325]: debug 2022-01-31T20:56:08.595+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:56:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:08 smithi171 conmon[41853]: debug 2022-01-31T20:56:08.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:08.585831+0000) 2022-01-31T20:56:09.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:08 smithi167 conmon[49112]: debug 2022-01-31T20:56:08.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:08.675628+0000) 2022-01-31T20:56:09.396 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:09 smithi167 conmon[60316]: debug 2022-01-31T20:56:09.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.102546+0000) 2022-01-31T20:56:09.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:09 smithi167 conmon[54076]: debug 2022-01-31T20:56:09.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.350969+0000) 2022-01-31T20:56:09.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[46715]: debug 2022-01-31T20:56:09.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.380500+0000) 2022-01-31T20:56:09.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[51620]: debug 2022-01-31T20:56:09.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.456714+0000) 2022-01-31T20:56:09.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[41853]: debug 2022-01-31T20:56:09.536+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.537016+0000) 2022-01-31T20:56:09.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:09 smithi167 conmon[54076]: debug 2022-01-31T20:56:09.536+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.537460+0000) 2022-01-31T20:56:09.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:09 smithi167 conmon[49112]: debug 2022-01-31T20:56:09.537+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.538216+0000) 2022-01-31T20:56:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:09 smithi167 conmon[60316]: debug 2022-01-31T20:56:09.538+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.538660+0000) 2022-01-31T20:56:09.792 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[35325]: debug 2022-01-31T20:56:09.545+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 89956 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:09.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[41853]: debug 2022-01-31T20:56:09.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.585992+0000) 2022-01-31T20:56:09.793 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[46715]: debug 2022-01-31T20:56:09.536+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.536880+0000) 2022-01-31T20:56:09.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:09 smithi171 conmon[51620]: debug 2022-01-31T20:56:09.536+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.537161+0000) 2022-01-31T20:56:10.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:09 smithi167 conmon[49112]: debug 2022-01-31T20:56:09.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:09.675736+0000) 2022-01-31T20:56:10.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:10 smithi167 conmon[60316]: debug 2022-01-31T20:56:10.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:10.102742+0000) 2022-01-31T20:56:10.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:10 smithi167 conmon[54076]: debug 2022-01-31T20:56:10.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:10.351166+0000) 2022-01-31T20:56:10.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:10 smithi171 conmon[46715]: debug 2022-01-31T20:56:10.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:10.380691+0000) 2022-01-31T20:56:10.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:10 smithi171 conmon[51620]: debug 2022-01-31T20:56:10.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:10.456922+0000) 2022-01-31T20:56:10.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:10 smithi171 conmon[41853]: debug 2022-01-31T20:56:10.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:10.586128+0000) 2022-01-31T20:56:11.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:10 smithi167 conmon[49112]: debug 2022-01-31T20:56:10.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:10.675853+0000) 2022-01-31T20:56:11.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:11 smithi167 conmon[54076]: debug 2022-01-31T20:56:11.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:11.351321+0000) 2022-01-31T20:56:11.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:11 smithi167 conmon[60316]: debug 2022-01-31T20:56:11.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:11.102870+0000) 2022-01-31T20:56:11.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:11 smithi171 conmon[46715]: debug 2022-01-31T20:56:11.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:11.380897+0000) 2022-01-31T20:56:11.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:11 smithi171 conmon[51620]: debug 2022-01-31T20:56:11.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:11.457083+0000) 2022-01-31T20:56:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:11 smithi171 conmon[41853]: debug 2022-01-31T20:56:11.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:11.586299+0000) 2022-01-31T20:56:12.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:11 smithi167 conmon[49112]: debug 2022-01-31T20:56:11.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:11.676051+0000) 2022-01-31T20:56:12.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:12 smithi167 conmon[54076]: debug 2022-01-31T20:56:12.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:12.351511+0000) 2022-01-31T20:56:12.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:12 smithi167 conmon[60316]: debug 2022-01-31T20:56:12.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:12.103075+0000) 2022-01-31T20:56:12.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:12 smithi171 conmon[51620]: debug 2022-01-31T20:56:12.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:12.457258+0000) 2022-01-31T20:56:12.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:12 smithi171 conmon[46715]: debug 2022-01-31T20:56:12.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:12.381082+0000) 2022-01-31T20:56:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:12 smithi171 conmon[41853]: debug 2022-01-31T20:56:12.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:12.586492+0000) 2022-01-31T20:56:13.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:12 smithi167 conmon[49112]: debug 2022-01-31T20:56:12.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:12.676246+0000) 2022-01-31T20:56:13.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:13 smithi167 conmon[54076]: debug 2022-01-31T20:56:13.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:13.351690+0000) 2022-01-31T20:56:13.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:13 smithi167 conmon[60316]: debug 2022-01-31T20:56:13.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:13.103229+0000) 2022-01-31T20:56:13.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:13 smithi171 conmon[46715]: debug 2022-01-31T20:56:13.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:13.381300+0000) 2022-01-31T20:56:13.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:13 smithi171 conmon[51620]: debug 2022-01-31T20:56:13.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:13.457413+0000) 2022-01-31T20:56:13.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:13 smithi171 conmon[41853]: debug 2022-01-31T20:56:13.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:13.586674+0000) 2022-01-31T20:56:14.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:13 smithi167 conmon[49112]: debug 2022-01-31T20:56:13.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:13.676439+0000) 2022-01-31T20:56:14.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:14 smithi167 conmon[54076]: debug 2022-01-31T20:56:14.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.351889+0000) 2022-01-31T20:56:14.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:14 smithi167 conmon[60316]: debug 2022-01-31T20:56:14.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.103445+0000) 2022-01-31T20:56:14.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[46715]: debug 2022-01-31T20:56:14.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.381498+0000) 2022-01-31T20:56:14.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[51620]: debug 2022-01-31T20:56:14.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.457613+0000) 2022-01-31T20:56:14.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:14 smithi167 conmon[49112]: debug 2022-01-31T20:56:14.549+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.549747+0000) 2022-01-31T20:56:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:14 smithi167 conmon[54076]: debug 2022-01-31T20:56:14.548+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.549174+0000) 2022-01-31T20:56:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:14 smithi167 conmon[60316]: debug 2022-01-31T20:56:14.549+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.550335+0000) 2022-01-31T20:56:14.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[46715]: debug 2022-01-31T20:56:14.548+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.548866+0000) 2022-01-31T20:56:14.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[51620]: debug 2022-01-31T20:56:14.549+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.549861+0000) 2022-01-31T20:56:14.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[35325]: debug 2022-01-31T20:56:14.557+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90066 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[41853]: debug 2022-01-31T20:56:14.548+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.549419+0000) 2022-01-31T20:56:14.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:14 smithi171 conmon[41853]: debug 2022-01-31T20:56:14.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.586837+0000) 2022-01-31T20:56:15.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:14 smithi167 conmon[49112]: debug 2022-01-31T20:56:14.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:14.676611+0000) 2022-01-31T20:56:15.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:15 smithi167 conmon[54076]: debug 2022-01-31T20:56:15.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:15.352096+0000) 2022-01-31T20:56:15.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:15 smithi167 conmon[60316]: debug 2022-01-31T20:56:15.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:15.103637+0000) 2022-01-31T20:56:15.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:15 smithi171 conmon[46715]: debug 2022-01-31T20:56:15.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:15.381665+0000) 2022-01-31T20:56:15.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:15 smithi171 conmon[51620]: debug 2022-01-31T20:56:15.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:15.457821+0000) 2022-01-31T20:56:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:15 smithi171 conmon[41853]: debug 2022-01-31T20:56:15.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:15.586978+0000) 2022-01-31T20:56:16.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:15 smithi167 conmon[49112]: debug 2022-01-31T20:56:15.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:15.676729+0000) 2022-01-31T20:56:16.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:16 smithi167 conmon[54076]: debug 2022-01-31T20:56:16.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:16.352240+0000) 2022-01-31T20:56:16.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:16 smithi167 conmon[60316]: debug 2022-01-31T20:56:16.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:16.103778+0000) 2022-01-31T20:56:16.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:16 smithi171 conmon[51620]: debug 2022-01-31T20:56:16.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:16.457973+0000) 2022-01-31T20:56:16.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:16 smithi171 conmon[46715]: debug 2022-01-31T20:56:16.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:16.381810+0000) 2022-01-31T20:56:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:16 smithi171 conmon[41853]: debug 2022-01-31T20:56:16.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:16.587078+0000) 2022-01-31T20:56:17.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:16 smithi167 conmon[49112]: debug 2022-01-31T20:56:16.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:16.676868+0000) 2022-01-31T20:56:17.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:17 smithi167 conmon[60316]: debug 2022-01-31T20:56:17.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:17.103967+0000) 2022-01-31T20:56:17.399 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:17 smithi167 conmon[54076]: debug 2022-01-31T20:56:17.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:17.352391+0000) 2022-01-31T20:56:17.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:17 smithi171 conmon[46715]: debug 2022-01-31T20:56:17.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:17.381967+0000) 2022-01-31T20:56:17.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:17 smithi171 conmon[51620]: debug 2022-01-31T20:56:17.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:17.458157+0000) 2022-01-31T20:56:17.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:17 smithi171 conmon[41853]: debug 2022-01-31T20:56:17.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:17.587263+0000) 2022-01-31T20:56:18.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:17 smithi167 conmon[49112]: debug 2022-01-31T20:56:17.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:17.677053+0000) 2022-01-31T20:56:18.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:18 smithi167 conmon[60316]: debug 2022-01-31T20:56:18.102+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:18.104174+0000) 2022-01-31T20:56:18.399 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:18 smithi167 conmon[54076]: debug 2022-01-31T20:56:18.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:18.352567+0000) 2022-01-31T20:56:18.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:18 smithi171 conmon[46715]: debug 2022-01-31T20:56:18.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:18.382148+0000) 2022-01-31T20:56:18.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:18 smithi171 conmon[51620]: debug 2022-01-31T20:56:18.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:18.458354+0000) 2022-01-31T20:56:18.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:18 smithi171 conmon[41853]: debug 2022-01-31T20:56:18.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:18.587447+0000) 2022-01-31T20:56:19.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:18 smithi167 conmon[49112]: debug 2022-01-31T20:56:18.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:18.677204+0000) 2022-01-31T20:56:19.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:19 smithi167 conmon[54076]: debug 2022-01-31T20:56:19.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.352721+0000) 2022-01-31T20:56:19.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:19 smithi167 conmon[60316]: debug 2022-01-31T20:56:19.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.104340+0000) 2022-01-31T20:56:19.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[46715]: debug 2022-01-31T20:56:19.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.382394+0000) 2022-01-31T20:56:19.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[51620]: debug 2022-01-31T20:56:19.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.458620+0000) 2022-01-31T20:56:19.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:19 smithi167 conmon[49112]: debug 2022-01-31T20:56:19.560+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.562462+0000) 2022-01-31T20:56:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:19 smithi167 conmon[54076]: debug 2022-01-31T20:56:19.560+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.562334+0000) 2022-01-31T20:56:19.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:19 smithi167 conmon[60316]: debug 2022-01-31T20:56:19.561+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.562853+0000) 2022-01-31T20:56:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[35325]: debug 2022-01-31T20:56:19.569+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90176 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:19.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[46715]: debug 2022-01-31T20:56:19.560+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.561612+0000) 2022-01-31T20:56:19.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[51620]: debug 2022-01-31T20:56:19.562+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.563186+0000) 2022-01-31T20:56:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[41853]: debug 2022-01-31T20:56:19.561+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.561888+0000) 2022-01-31T20:56:19.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:19 smithi171 conmon[41853]: debug 2022-01-31T20:56:19.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.587619+0000) 2022-01-31T20:56:20.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:19 smithi167 conmon[49112]: debug 2022-01-31T20:56:19.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:19.677412+0000) 2022-01-31T20:56:20.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:20 smithi167 conmon[60316]: debug 2022-01-31T20:56:20.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:20.104521+0000) 2022-01-31T20:56:20.399 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:20 smithi167 conmon[54076]: debug 2022-01-31T20:56:20.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:20.352857+0000) 2022-01-31T20:56:20.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:20 smithi171 conmon[46715]: debug 2022-01-31T20:56:20.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:20.382555+0000) 2022-01-31T20:56:20.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:20 smithi171 conmon[51620]: debug 2022-01-31T20:56:20.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:20.458786+0000) 2022-01-31T20:56:20.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:20 smithi171 conmon[41853]: debug 2022-01-31T20:56:20.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:20.587793+0000) 2022-01-31T20:56:21.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:20 smithi167 conmon[49112]: debug 2022-01-31T20:56:20.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:20.677558+0000) 2022-01-31T20:56:21.399 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:21 smithi167 conmon[54076]: debug 2022-01-31T20:56:21.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:21.353024+0000) 2022-01-31T20:56:21.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:21 smithi167 conmon[60316]: debug 2022-01-31T20:56:21.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:21.104696+0000) 2022-01-31T20:56:21.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:21 smithi171 conmon[46715]: debug 2022-01-31T20:56:21.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:21.382651+0000) 2022-01-31T20:56:21.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:21 smithi171 conmon[51620]: debug 2022-01-31T20:56:21.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:21.458952+0000) 2022-01-31T20:56:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:21 smithi171 conmon[41853]: debug 2022-01-31T20:56:21.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:21.587980+0000) 2022-01-31T20:56:22.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:21 smithi167 conmon[49112]: debug 2022-01-31T20:56:21.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:21.677677+0000) 2022-01-31T20:56:22.399 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:22 smithi167 conmon[54076]: debug 2022-01-31T20:56:22.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:22.353203+0000) 2022-01-31T20:56:22.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:22 smithi167 conmon[60316]: debug 2022-01-31T20:56:22.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:22.104903+0000) 2022-01-31T20:56:22.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:22 smithi171 conmon[46715]: debug 2022-01-31T20:56:22.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:22.382806+0000) 2022-01-31T20:56:22.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:22 smithi171 conmon[51620]: debug 2022-01-31T20:56:22.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:22.459104+0000) 2022-01-31T20:56:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:22 smithi171 conmon[41853]: debug 2022-01-31T20:56:22.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:22.588138+0000) 2022-01-31T20:56:23.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:22 smithi167 conmon[49112]: debug 2022-01-31T20:56:22.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:22.677853+0000) 2022-01-31T20:56:23.399 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:23 smithi167 conmon[54076]: debug 2022-01-31T20:56:23.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:23.353386+0000) 2022-01-31T20:56:23.400 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:23 smithi167 conmon[60316]: debug 2022-01-31T20:56:23.104+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:23.105071+0000) 2022-01-31T20:56:23.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:23 smithi171 conmon[46715]: debug 2022-01-31T20:56:23.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:23.382913+0000) 2022-01-31T20:56:23.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:23 smithi171 conmon[51620]: debug 2022-01-31T20:56:23.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:23.459284+0000) 2022-01-31T20:56:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:23 smithi171 conmon[35325]: debug 2022-01-31T20:56:23.596+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:56:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:23 smithi171 conmon[41853]: debug 2022-01-31T20:56:23.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:23.588310+0000) 2022-01-31T20:56:24.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:23 smithi167 conmon[49112]: debug 2022-01-31T20:56:23.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:23.677975+0000) 2022-01-31T20:56:24.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:24 smithi167 conmon[60316]: debug 2022-01-31T20:56:24.103+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.105281+0000) 2022-01-31T20:56:24.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:24 smithi167 conmon[54076]: debug 2022-01-31T20:56:24.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.353592+0000) 2022-01-31T20:56:24.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[51620]: debug 2022-01-31T20:56:24.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.459446+0000) 2022-01-31T20:56:24.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[46715]: debug 2022-01-31T20:56:24.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.383020+0000) 2022-01-31T20:56:24.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:24 smithi167 conmon[49112]: debug 2022-01-31T20:56:24.570+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.572445+0000) 2022-01-31T20:56:24.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:24 smithi167 conmon[60316]: debug 2022-01-31T20:56:24.572+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.574123+0000) 2022-01-31T20:56:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:24 smithi167 conmon[54076]: debug 2022-01-31T20:56:24.572+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.573878+0000) 2022-01-31T20:56:24.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[35325]: debug 2022-01-31T20:56:24.581+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90286 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:24.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[46715]: debug 2022-01-31T20:56:24.572+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.573059+0000) 2022-01-31T20:56:24.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[51620]: debug 2022-01-31T20:56:24.571+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.572554+0000) 2022-01-31T20:56:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[41853]: debug 2022-01-31T20:56:24.572+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.573540+0000) 2022-01-31T20:56:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:24 smithi171 conmon[41853]: debug 2022-01-31T20:56:24.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.588454+0000) 2022-01-31T20:56:25.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:24 smithi167 conmon[49112]: debug 2022-01-31T20:56:24.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:24.678157+0000) 2022-01-31T20:56:25.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:25 smithi167 conmon[54076]: debug 2022-01-31T20:56:25.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:25.353757+0000) 2022-01-31T20:56:25.400 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:25 smithi167 conmon[60316]: debug 2022-01-31T20:56:25.104+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:25.105466+0000) 2022-01-31T20:56:25.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:25 smithi171 conmon[46715]: debug 2022-01-31T20:56:25.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:25.383188+0000) 2022-01-31T20:56:25.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:25 smithi171 conmon[51620]: debug 2022-01-31T20:56:25.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:25.459688+0000) 2022-01-31T20:56:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:25 smithi171 conmon[41853]: debug 2022-01-31T20:56:25.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:25.588637+0000) 2022-01-31T20:56:26.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:25 smithi167 conmon[49112]: debug 2022-01-31T20:56:25.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:25.678288+0000) 2022-01-31T20:56:26.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:26 smithi167 conmon[54076]: debug 2022-01-31T20:56:26.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:26.353853+0000) 2022-01-31T20:56:26.400 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:26 smithi167 conmon[60316]: debug 2022-01-31T20:56:26.104+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:26.105599+0000) 2022-01-31T20:56:26.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:26 smithi171 conmon[46715]: debug 2022-01-31T20:56:26.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:26.383334+0000) 2022-01-31T20:56:26.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:26 smithi171 conmon[51620]: debug 2022-01-31T20:56:26.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:26.459819+0000) 2022-01-31T20:56:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:26 smithi171 conmon[41853]: debug 2022-01-31T20:56:26.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:26.588822+0000) 2022-01-31T20:56:27.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:26 smithi167 conmon[49112]: debug 2022-01-31T20:56:26.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:26.678396+0000) 2022-01-31T20:56:27.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:27 smithi167 conmon[54076]: debug 2022-01-31T20:56:27.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:27.354030+0000) 2022-01-31T20:56:27.400 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:27 smithi167 conmon[60316]: debug 2022-01-31T20:56:27.104+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:27.105733+0000) 2022-01-31T20:56:27.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:27 smithi171 conmon[46715]: debug 2022-01-31T20:56:27.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:27.383546+0000) 2022-01-31T20:56:27.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:27 smithi171 conmon[51620]: debug 2022-01-31T20:56:27.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:27.460012+0000) 2022-01-31T20:56:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:27 smithi171 conmon[41853]: debug 2022-01-31T20:56:27.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:27.588946+0000) 2022-01-31T20:56:28.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:27 smithi167 conmon[49112]: debug 2022-01-31T20:56:27.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:27.678547+0000) 2022-01-31T20:56:28.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:28 smithi167 conmon[54076]: debug 2022-01-31T20:56:28.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:28.354211+0000) 2022-01-31T20:56:28.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:28 smithi167 conmon[60316]: debug 2022-01-31T20:56:28.105+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:28.105898+0000) 2022-01-31T20:56:28.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:28 smithi171 conmon[46715]: debug 2022-01-31T20:56:28.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:28.383697+0000) 2022-01-31T20:56:28.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:28 smithi171 conmon[51620]: debug 2022-01-31T20:56:28.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:28.460125+0000) 2022-01-31T20:56:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:28 smithi171 conmon[41853]: debug 2022-01-31T20:56:28.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:28.589104+0000) 2022-01-31T20:56:29.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:28 smithi167 conmon[49112]: debug 2022-01-31T20:56:28.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:28.678651+0000) 2022-01-31T20:56:29.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:29 smithi167 conmon[54076]: debug 2022-01-31T20:56:29.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.354418+0000) 2022-01-31T20:56:29.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:29 smithi167 conmon[60316]: debug 2022-01-31T20:56:29.105+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.106039+0000) 2022-01-31T20:56:29.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[46715]: debug 2022-01-31T20:56:29.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.383826+0000) 2022-01-31T20:56:29.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[51620]: debug 2022-01-31T20:56:29.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.460302+0000) 2022-01-31T20:56:29.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:29 smithi167 conmon[49112]: debug 2022-01-31T20:56:29.582+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.583855+0000) 2022-01-31T20:56:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:29 smithi167 conmon[54076]: debug 2022-01-31T20:56:29.583+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.585215+0000) 2022-01-31T20:56:29.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:29 smithi167 conmon[60316]: debug 2022-01-31T20:56:29.583+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.585070+0000) 2022-01-31T20:56:29.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[35325]: debug 2022-01-31T20:56:29.609+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90396 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:29.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[46715]: debug 2022-01-31T20:56:29.582+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.583630+0000) 2022-01-31T20:56:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[41853]: debug 2022-01-31T20:56:29.583+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.583967+0000) 2022-01-31T20:56:29.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[41853]: debug 2022-01-31T20:56:29.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.589276+0000) 2022-01-31T20:56:29.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:29 smithi171 conmon[51620]: debug 2022-01-31T20:56:29.584+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.585425+0000) 2022-01-31T20:56:30.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:29 smithi167 conmon[49112]: debug 2022-01-31T20:56:29.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:29.678760+0000) 2022-01-31T20:56:30.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:30 smithi167 conmon[60316]: debug 2022-01-31T20:56:30.105+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:30.106218+0000) 2022-01-31T20:56:30.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:30 smithi167 conmon[54076]: debug 2022-01-31T20:56:30.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:30.354602+0000) 2022-01-31T20:56:30.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:30 smithi171 conmon[46715]: debug 2022-01-31T20:56:30.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:30.383993+0000) 2022-01-31T20:56:30.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:30 smithi171 conmon[51620]: debug 2022-01-31T20:56:30.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:30.460508+0000) 2022-01-31T20:56:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:30 smithi171 conmon[41853]: debug 2022-01-31T20:56:30.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:30.589429+0000) 2022-01-31T20:56:31.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:30 smithi167 conmon[49112]: debug 2022-01-31T20:56:30.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:30.678906+0000) 2022-01-31T20:56:31.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:31 smithi167 conmon[54076]: debug 2022-01-31T20:56:31.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:31.354715+0000) 2022-01-31T20:56:31.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:31 smithi167 conmon[60316]: debug 2022-01-31T20:56:31.105+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:31.106343+0000) 2022-01-31T20:56:31.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:31 smithi171 conmon[46715]: debug 2022-01-31T20:56:31.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:31.384200+0000) 2022-01-31T20:56:31.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:31 smithi171 conmon[51620]: debug 2022-01-31T20:56:31.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:31.460650+0000) 2022-01-31T20:56:31.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:31 smithi171 conmon[41853]: debug 2022-01-31T20:56:31.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:31.589652+0000) 2022-01-31T20:56:32.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:31 smithi167 conmon[49112]: debug 2022-01-31T20:56:31.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:31.679094+0000) 2022-01-31T20:56:32.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:32 smithi167 conmon[54076]: debug 2022-01-31T20:56:32.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:32.354926+0000) 2022-01-31T20:56:32.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:32 smithi167 conmon[60316]: debug 2022-01-31T20:56:32.105+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:32.106517+0000) 2022-01-31T20:56:32.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:32 smithi171 conmon[46715]: debug 2022-01-31T20:56:32.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:32.384376+0000) 2022-01-31T20:56:32.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:32 smithi171 conmon[51620]: debug 2022-01-31T20:56:32.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:32.460830+0000) 2022-01-31T20:56:32.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:32 smithi171 conmon[41853]: debug 2022-01-31T20:56:32.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:32.589826+0000) 2022-01-31T20:56:33.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:32 smithi167 conmon[49112]: debug 2022-01-31T20:56:32.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:32.679238+0000) 2022-01-31T20:56:33.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:33 smithi167 conmon[54076]: debug 2022-01-31T20:56:33.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:33.355110+0000) 2022-01-31T20:56:33.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:33 smithi167 conmon[60316]: debug 2022-01-31T20:56:33.105+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:33.106661+0000) 2022-01-31T20:56:33.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:33 smithi171 conmon[46715]: debug 2022-01-31T20:56:33.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:33.384532+0000) 2022-01-31T20:56:33.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:33 smithi171 conmon[51620]: debug 2022-01-31T20:56:33.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:33.460985+0000) 2022-01-31T20:56:33.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:33 smithi171 conmon[41853]: debug 2022-01-31T20:56:33.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:33.589984+0000) 2022-01-31T20:56:34.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:33 smithi167 conmon[49112]: debug 2022-01-31T20:56:33.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:33.679454+0000) 2022-01-31T20:56:34.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:34 smithi167 conmon[60316]: debug 2022-01-31T20:56:34.106+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.106852+0000) 2022-01-31T20:56:34.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:34 smithi167 conmon[54076]: debug 2022-01-31T20:56:34.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.355284+0000) 2022-01-31T20:56:34.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[46715]: debug 2022-01-31T20:56:34.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.384711+0000) 2022-01-31T20:56:34.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[51620]: debug 2022-01-31T20:56:34.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.461168+0000) 2022-01-31T20:56:34.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:34 smithi167 conmon[49112]: debug 2022-01-31T20:56:34.635+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.636987+0000) 2022-01-31T20:56:34.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:34 smithi167 conmon[54076]: debug 2022-01-31T20:56:34.611+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.613465+0000) 2022-01-31T20:56:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:34 smithi167 conmon[60316]: debug 2022-01-31T20:56:34.611+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.613345+0000) 2022-01-31T20:56:34.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[46715]: debug 2022-01-31T20:56:34.610+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.612040+0000) 2022-01-31T20:56:34.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[51620]: debug 2022-01-31T20:56:34.611+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.613725+0000) 2022-01-31T20:56:34.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[35325]: debug 2022-01-31T20:56:34.620+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90506 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:34.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[41853]: debug 2022-01-31T20:56:34.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.590150+0000) 2022-01-31T20:56:34.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:34 smithi171 conmon[41853]: debug 2022-01-31T20:56:34.610+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.612520+0000) 2022-01-31T20:56:35.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:34 smithi167 conmon[49112]: debug 2022-01-31T20:56:34.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:34.679607+0000) 2022-01-31T20:56:35.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:35 smithi167 conmon[60316]: debug 2022-01-31T20:56:35.106+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:35.107070+0000) 2022-01-31T20:56:35.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:35 smithi167 conmon[54076]: debug 2022-01-31T20:56:35.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:35.355497+0000) 2022-01-31T20:56:35.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:35 smithi171 conmon[46715]: debug 2022-01-31T20:56:35.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:35.384862+0000) 2022-01-31T20:56:35.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:35 smithi171 conmon[51620]: debug 2022-01-31T20:56:35.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:35.461352+0000) 2022-01-31T20:56:35.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:35 smithi171 conmon[41853]: debug 2022-01-31T20:56:35.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:35.590306+0000) 2022-01-31T20:56:36.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:35 smithi167 conmon[49112]: debug 2022-01-31T20:56:35.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:35.679740+0000) 2022-01-31T20:56:36.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:36 smithi167 conmon[54076]: debug 2022-01-31T20:56:36.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:36.355644+0000) 2022-01-31T20:56:36.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:36 smithi167 conmon[60316]: debug 2022-01-31T20:56:36.106+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:36.107232+0000) 2022-01-31T20:56:36.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:36 smithi171 conmon[46715]: debug 2022-01-31T20:56:36.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:36.385014+0000) 2022-01-31T20:56:36.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:36 smithi171 conmon[51620]: debug 2022-01-31T20:56:36.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:36.461500+0000) 2022-01-31T20:56:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:36 smithi171 conmon[41853]: debug 2022-01-31T20:56:36.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:36.590384+0000) 2022-01-31T20:56:37.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:36 smithi167 conmon[49112]: debug 2022-01-31T20:56:36.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:36.679888+0000) 2022-01-31T20:56:37.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:37 smithi167 conmon[54076]: debug 2022-01-31T20:56:37.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:37.355800+0000) 2022-01-31T20:56:37.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:37 smithi167 conmon[60316]: debug 2022-01-31T20:56:37.106+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:37.107388+0000) 2022-01-31T20:56:37.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:37 smithi171 conmon[46715]: debug 2022-01-31T20:56:37.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:37.385201+0000) 2022-01-31T20:56:37.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:37 smithi171 conmon[51620]: debug 2022-01-31T20:56:37.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:37.461686+0000) 2022-01-31T20:56:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:37 smithi171 conmon[41853]: debug 2022-01-31T20:56:37.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:37.590540+0000) 2022-01-31T20:56:38.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:37 smithi167 conmon[49112]: debug 2022-01-31T20:56:37.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:37.679986+0000) 2022-01-31T20:56:38.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:38 smithi167 conmon[54076]: debug 2022-01-31T20:56:38.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:38.356027+0000) 2022-01-31T20:56:38.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:38 smithi167 conmon[60316]: debug 2022-01-31T20:56:38.106+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:38.107545+0000) 2022-01-31T20:56:38.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:38 smithi171 conmon[46715]: debug 2022-01-31T20:56:38.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:38.385427+0000) 2022-01-31T20:56:38.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:38 smithi171 conmon[51620]: debug 2022-01-31T20:56:38.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:38.461888+0000) 2022-01-31T20:56:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:38 smithi171 conmon[41853]: debug 2022-01-31T20:56:38.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:38.590700+0000) 2022-01-31T20:56:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:38 smithi171 conmon[35325]: debug 2022-01-31T20:56:38.596+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:56:39.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:38 smithi167 conmon[49112]: debug 2022-01-31T20:56:38.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:38.680159+0000) 2022-01-31T20:56:39.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:39 smithi167 conmon[54076]: debug 2022-01-31T20:56:39.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.356214+0000) 2022-01-31T20:56:39.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:39 smithi167 conmon[60316]: debug 2022-01-31T20:56:39.107+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.107728+0000) 2022-01-31T20:56:39.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[46715]: debug 2022-01-31T20:56:39.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.385587+0000) 2022-01-31T20:56:39.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[51620]: debug 2022-01-31T20:56:39.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.462074+0000) 2022-01-31T20:56:39.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:39 smithi167 conmon[49112]: debug 2022-01-31T20:56:39.624+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.626011+0000) 2022-01-31T20:56:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:39 smithi167 conmon[54076]: debug 2022-01-31T20:56:39.623+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.624566+0000) 2022-01-31T20:56:39.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:39 smithi167 conmon[60316]: debug 2022-01-31T20:56:39.623+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.625131+0000) 2022-01-31T20:56:39.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[46715]: debug 2022-01-31T20:56:39.622+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.624132+0000) 2022-01-31T20:56:39.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[51620]: debug 2022-01-31T20:56:39.623+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.624987+0000) 2022-01-31T20:56:39.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[35325]: debug 2022-01-31T20:56:39.632+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90616 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[41853]: debug 2022-01-31T20:56:39.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.590859+0000) 2022-01-31T20:56:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:39 smithi171 conmon[41853]: debug 2022-01-31T20:56:39.622+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.624730+0000) 2022-01-31T20:56:40.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:39 smithi167 conmon[49112]: debug 2022-01-31T20:56:39.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:39.680337+0000) 2022-01-31T20:56:40.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:40 smithi167 conmon[60316]: debug 2022-01-31T20:56:40.107+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:40.107921+0000) 2022-01-31T20:56:40.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:40 smithi167 conmon[54076]: debug 2022-01-31T20:56:40.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:40.356398+0000) 2022-01-31T20:56:40.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:40 smithi171 conmon[51620]: debug 2022-01-31T20:56:40.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:40.462264+0000) 2022-01-31T20:56:40.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:40 smithi171 conmon[46715]: debug 2022-01-31T20:56:40.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:40.385779+0000) 2022-01-31T20:56:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:40 smithi171 conmon[41853]: debug 2022-01-31T20:56:40.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:40.591042+0000) 2022-01-31T20:56:41.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:40 smithi167 conmon[49112]: debug 2022-01-31T20:56:40.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:40.680480+0000) 2022-01-31T20:56:41.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:41 smithi167 conmon[60316]: debug 2022-01-31T20:56:41.107+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:41.108093+0000) 2022-01-31T20:56:41.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:41 smithi167 conmon[54076]: debug 2022-01-31T20:56:41.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:41.356550+0000) 2022-01-31T20:56:41.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:41 smithi171 conmon[46715]: debug 2022-01-31T20:56:41.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:41.385923+0000) 2022-01-31T20:56:41.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:41 smithi171 conmon[51620]: debug 2022-01-31T20:56:41.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:41.462400+0000) 2022-01-31T20:56:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:41 smithi171 conmon[41853]: debug 2022-01-31T20:56:41.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:41.591224+0000) 2022-01-31T20:56:42.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:41 smithi167 conmon[49112]: debug 2022-01-31T20:56:41.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:41.680667+0000) 2022-01-31T20:56:42.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:42 smithi167 conmon[60316]: debug 2022-01-31T20:56:42.107+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:42.108220+0000) 2022-01-31T20:56:42.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:42 smithi167 conmon[54076]: debug 2022-01-31T20:56:42.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:42.356701+0000) 2022-01-31T20:56:42.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:42 smithi171 conmon[46715]: debug 2022-01-31T20:56:42.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:42.386101+0000) 2022-01-31T20:56:42.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:42 smithi171 conmon[51620]: debug 2022-01-31T20:56:42.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:42.462584+0000) 2022-01-31T20:56:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:42 smithi171 conmon[41853]: debug 2022-01-31T20:56:42.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:42.591443+0000) 2022-01-31T20:56:43.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:42 smithi167 conmon[49112]: debug 2022-01-31T20:56:42.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:42.680781+0000) 2022-01-31T20:56:43.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:43 smithi167 conmon[54076]: debug 2022-01-31T20:56:43.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:43.356813+0000) 2022-01-31T20:56:43.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:43 smithi167 conmon[60316]: debug 2022-01-31T20:56:43.107+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:43.108377+0000) 2022-01-31T20:56:43.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:43 smithi171 conmon[46715]: debug 2022-01-31T20:56:43.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:43.386291+0000) 2022-01-31T20:56:43.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:43 smithi171 conmon[51620]: debug 2022-01-31T20:56:43.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:43.462789+0000) 2022-01-31T20:56:43.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:43 smithi171 conmon[41853]: debug 2022-01-31T20:56:43.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:43.591628+0000) 2022-01-31T20:56:44.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:43 smithi167 conmon[49112]: debug 2022-01-31T20:56:43.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:43.680957+0000) 2022-01-31T20:56:44.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:44 smithi167 conmon[60316]: debug 2022-01-31T20:56:44.108+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.108557+0000) 2022-01-31T20:56:44.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:44 smithi167 conmon[54076]: debug 2022-01-31T20:56:44.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.357006+0000) 2022-01-31T20:56:44.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[46715]: debug 2022-01-31T20:56:44.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.386484+0000) 2022-01-31T20:56:44.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[51620]: debug 2022-01-31T20:56:44.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.462995+0000) 2022-01-31T20:56:44.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:44 smithi167 conmon[49112]: debug 2022-01-31T20:56:44.637+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.637610+0000) 2022-01-31T20:56:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:44 smithi167 conmon[54076]: debug 2022-01-31T20:56:44.636+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.636860+0000) 2022-01-31T20:56:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:44 smithi167 conmon[60316]: debug 2022-01-31T20:56:44.636+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.637007+0000) 2022-01-31T20:56:44.861 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[35325]: debug 2022-01-31T20:56:44.644+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90726 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:44.862 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[46715]: debug 2022-01-31T20:56:44.634+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.635655+0000) 2022-01-31T20:56:44.862 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[51620]: debug 2022-01-31T20:56:44.635+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.636485+0000) 2022-01-31T20:56:44.863 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[41853]: debug 2022-01-31T20:56:44.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.591837+0000) 2022-01-31T20:56:44.863 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:44 smithi171 conmon[41853]: debug 2022-01-31T20:56:44.635+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.635787+0000) 2022-01-31T20:56:45.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:44 smithi167 conmon[49112]: debug 2022-01-31T20:56:44.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:44.681120+0000) 2022-01-31T20:56:45.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:45 smithi167 conmon[54076]: debug 2022-01-31T20:56:45.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:45.357202+0000) 2022-01-31T20:56:45.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:45 smithi167 conmon[60316]: debug 2022-01-31T20:56:45.108+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:45.108728+0000) 2022-01-31T20:56:45.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:45 smithi171 conmon[51620]: debug 2022-01-31T20:56:45.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:45.463175+0000) 2022-01-31T20:56:45.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:45 smithi171 conmon[46715]: debug 2022-01-31T20:56:45.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:45.386693+0000) 2022-01-31T20:56:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:45 smithi171 conmon[41853]: debug 2022-01-31T20:56:45.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:45.592039+0000) 2022-01-31T20:56:46.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:45 smithi167 conmon[49112]: debug 2022-01-31T20:56:45.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:45.681279+0000) 2022-01-31T20:56:46.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:46 smithi167 conmon[54076]: debug 2022-01-31T20:56:46.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:46.357359+0000) 2022-01-31T20:56:46.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:46 smithi167 conmon[60316]: debug 2022-01-31T20:56:46.108+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:46.108873+0000) 2022-01-31T20:56:46.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:46 smithi171 conmon[46715]: debug 2022-01-31T20:56:46.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:46.386839+0000) 2022-01-31T20:56:46.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:46 smithi171 conmon[51620]: debug 2022-01-31T20:56:46.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:46.463328+0000) 2022-01-31T20:56:46.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:46 smithi171 conmon[41853]: debug 2022-01-31T20:56:46.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:46.592186+0000) 2022-01-31T20:56:47.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:46 smithi167 conmon[49112]: debug 2022-01-31T20:56:46.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:46.681430+0000) 2022-01-31T20:56:47.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:47 smithi167 conmon[54076]: debug 2022-01-31T20:56:47.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:47.357574+0000) 2022-01-31T20:56:47.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:47 smithi167 conmon[60316]: debug 2022-01-31T20:56:47.108+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:47.109058+0000) 2022-01-31T20:56:47.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:47 smithi171 conmon[46715]: debug 2022-01-31T20:56:47.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:47.387052+0000) 2022-01-31T20:56:47.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:47 smithi171 conmon[51620]: debug 2022-01-31T20:56:47.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:47.463505+0000) 2022-01-31T20:56:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:47 smithi171 conmon[41853]: debug 2022-01-31T20:56:47.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:47.592372+0000) 2022-01-31T20:56:48.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:47 smithi167 conmon[49112]: debug 2022-01-31T20:56:47.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:47.681541+0000) 2022-01-31T20:56:48.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:48 smithi167 conmon[54076]: debug 2022-01-31T20:56:48.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:48.357800+0000) 2022-01-31T20:56:48.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:48 smithi167 conmon[60316]: debug 2022-01-31T20:56:48.108+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:48.109246+0000) 2022-01-31T20:56:48.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:48 smithi171 conmon[46715]: debug 2022-01-31T20:56:48.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:48.387231+0000) 2022-01-31T20:56:48.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:48 smithi171 conmon[51620]: debug 2022-01-31T20:56:48.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:48.463708+0000) 2022-01-31T20:56:48.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:48 smithi171 conmon[41853]: debug 2022-01-31T20:56:48.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:48.592591+0000) 2022-01-31T20:56:49.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:48 smithi167 conmon[49112]: debug 2022-01-31T20:56:48.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:48.681679+0000) 2022-01-31T20:56:49.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:49 smithi167 conmon[54076]: debug 2022-01-31T20:56:49.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.357955+0000) 2022-01-31T20:56:49.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:49 smithi167 conmon[60316]: debug 2022-01-31T20:56:49.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.109425+0000) 2022-01-31T20:56:49.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[46715]: debug 2022-01-31T20:56:49.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.387432+0000) 2022-01-31T20:56:49.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[51620]: debug 2022-01-31T20:56:49.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.463834+0000) 2022-01-31T20:56:49.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:49 smithi167 conmon[49112]: debug 2022-01-31T20:56:49.648+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.648578+0000) 2022-01-31T20:56:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:49 smithi167 conmon[54076]: debug 2022-01-31T20:56:49.647+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.648272+0000) 2022-01-31T20:56:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:49 smithi167 conmon[60316]: debug 2022-01-31T20:56:49.647+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.648083+0000) 2022-01-31T20:56:49.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[46715]: debug 2022-01-31T20:56:49.648+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.649521+0000) 2022-01-31T20:56:49.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[51620]: debug 2022-01-31T20:56:49.646+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.647276+0000) 2022-01-31T20:56:49.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[35325]: debug 2022-01-31T20:56:49.657+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90836 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[41853]: debug 2022-01-31T20:56:49.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.592770+0000) 2022-01-31T20:56:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:49 smithi171 conmon[41853]: debug 2022-01-31T20:56:49.647+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.648483+0000) 2022-01-31T20:56:50.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:49 smithi167 conmon[49112]: debug 2022-01-31T20:56:49.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:49.681850+0000) 2022-01-31T20:56:50.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:50 smithi167 conmon[54076]: debug 2022-01-31T20:56:50.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:50.358118+0000) 2022-01-31T20:56:50.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:50 smithi167 conmon[60316]: debug 2022-01-31T20:56:50.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:50.109592+0000) 2022-01-31T20:56:50.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:50 smithi171 conmon[46715]: debug 2022-01-31T20:56:50.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:50.387622+0000) 2022-01-31T20:56:50.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:50 smithi171 conmon[51620]: debug 2022-01-31T20:56:50.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:50.463959+0000) 2022-01-31T20:56:50.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:50 smithi171 conmon[41853]: debug 2022-01-31T20:56:50.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:50.592884+0000) 2022-01-31T20:56:51.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:50 smithi167 conmon[49112]: debug 2022-01-31T20:56:50.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:50.682037+0000) 2022-01-31T20:56:51.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:51 smithi167 conmon[54076]: debug 2022-01-31T20:56:51.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:51.358293+0000) 2022-01-31T20:56:51.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:51 smithi167 conmon[60316]: debug 2022-01-31T20:56:51.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:51.109709+0000) 2022-01-31T20:56:51.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:51 smithi171 conmon[46715]: debug 2022-01-31T20:56:51.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:51.387780+0000) 2022-01-31T20:56:51.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:51 smithi171 conmon[51620]: debug 2022-01-31T20:56:51.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:51.464036+0000) 2022-01-31T20:56:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:51 smithi171 conmon[41853]: debug 2022-01-31T20:56:51.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:51.593004+0000) 2022-01-31T20:56:52.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:51 smithi167 conmon[49112]: debug 2022-01-31T20:56:51.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:51.682211+0000) 2022-01-31T20:56:52.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:52 smithi167 conmon[54076]: debug 2022-01-31T20:56:52.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:52.358440+0000) 2022-01-31T20:56:52.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:52 smithi167 conmon[60316]: debug 2022-01-31T20:56:52.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:52.109896+0000) 2022-01-31T20:56:52.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:52 smithi171 conmon[46715]: debug 2022-01-31T20:56:52.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:52.387924+0000) 2022-01-31T20:56:52.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:52 smithi171 conmon[51620]: debug 2022-01-31T20:56:52.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:52.464136+0000) 2022-01-31T20:56:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:52 smithi171 conmon[41853]: debug 2022-01-31T20:56:52.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:52.593131+0000) 2022-01-31T20:56:53.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:52 smithi167 conmon[49112]: debug 2022-01-31T20:56:52.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:52.682419+0000) 2022-01-31T20:56:53.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:53 smithi167 conmon[54076]: debug 2022-01-31T20:56:53.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:53.358599+0000) 2022-01-31T20:56:53.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:53 smithi167 conmon[60316]: debug 2022-01-31T20:56:53.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:53.110037+0000) 2022-01-31T20:56:53.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:53 smithi171 conmon[46715]: debug 2022-01-31T20:56:53.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:53.388087+0000) 2022-01-31T20:56:53.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:53 smithi171 conmon[51620]: debug 2022-01-31T20:56:53.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:53.464278+0000) 2022-01-31T20:56:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:53 smithi171 conmon[41853]: debug 2022-01-31T20:56:53.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:53.593291+0000) 2022-01-31T20:56:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:53 smithi171 conmon[35325]: debug 2022-01-31T20:56:53.597+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:56:54.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:53 smithi167 conmon[49112]: debug 2022-01-31T20:56:53.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:53.682629+0000) 2022-01-31T20:56:54.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:54 smithi167 conmon[54076]: debug 2022-01-31T20:56:54.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.358761+0000) 2022-01-31T20:56:54.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:54 smithi167 conmon[60316]: debug 2022-01-31T20:56:54.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.110211+0000) 2022-01-31T20:56:54.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[46715]: debug 2022-01-31T20:56:54.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.388244+0000) 2022-01-31T20:56:54.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[51620]: debug 2022-01-31T20:56:54.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.464431+0000) 2022-01-31T20:56:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[35325]: debug 2022-01-31T20:56:54.669+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 90948 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:54.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[46715]: debug 2022-01-31T20:56:54.660+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.661244+0000) 2022-01-31T20:56:54.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[51620]: debug 2022-01-31T20:56:54.659+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.660699+0000) 2022-01-31T20:56:54.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[41853]: debug 2022-01-31T20:56:54.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.593490+0000) 2022-01-31T20:56:54.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:54 smithi171 conmon[41853]: debug 2022-01-31T20:56:54.660+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.661362+0000) 2022-01-31T20:56:54.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:54 smithi167 conmon[54076]: debug 2022-01-31T20:56:54.660+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.661012+0000) 2022-01-31T20:56:54.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:54 smithi167 conmon[60316]: debug 2022-01-31T20:56:54.659+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.660483+0000) 2022-01-31T20:56:54.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:54 smithi167 conmon[49112]: debug 2022-01-31T20:56:54.661+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.661790+0000) 2022-01-31T20:56:54.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:54 smithi167 conmon[49112]: debug 2022-01-31T20:56:54.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:54.682764+0000) 2022-01-31T20:56:55.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:55 smithi167 conmon[54076]: debug 2022-01-31T20:56:55.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:55.358916+0000) 2022-01-31T20:56:55.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:55 smithi167 conmon[60316]: debug 2022-01-31T20:56:55.109+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:55.110419+0000) 2022-01-31T20:56:55.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:55 smithi171 conmon[51620]: debug 2022-01-31T20:56:55.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:55.464612+0000) 2022-01-31T20:56:55.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:55 smithi171 conmon[46715]: debug 2022-01-31T20:56:55.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:55.388344+0000) 2022-01-31T20:56:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:55 smithi171 conmon[41853]: debug 2022-01-31T20:56:55.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:55.593666+0000) 2022-01-31T20:56:56.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:55 smithi167 conmon[49112]: debug 2022-01-31T20:56:55.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:55.682859+0000) 2022-01-31T20:56:56.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:56 smithi167 conmon[54076]: debug 2022-01-31T20:56:56.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:56.359088+0000) 2022-01-31T20:56:56.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:56 smithi167 conmon[60316]: debug 2022-01-31T20:56:56.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:56.110575+0000) 2022-01-31T20:56:56.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:56 smithi171 conmon[46715]: debug 2022-01-31T20:56:56.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:56.388486+0000) 2022-01-31T20:56:56.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:56 smithi171 conmon[51620]: debug 2022-01-31T20:56:56.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:56.464796+0000) 2022-01-31T20:56:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:56 smithi171 conmon[41853]: debug 2022-01-31T20:56:56.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:56.593785+0000) 2022-01-31T20:56:57.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:56 smithi167 conmon[49112]: debug 2022-01-31T20:56:56.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:56.683059+0000) 2022-01-31T20:56:57.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:57 smithi167 conmon[54076]: debug 2022-01-31T20:56:57.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:57.359278+0000) 2022-01-31T20:56:57.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:57 smithi167 conmon[60316]: debug 2022-01-31T20:56:57.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:57.110723+0000) 2022-01-31T20:56:57.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:57 smithi171 conmon[46715]: debug 2022-01-31T20:56:57.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:57.388680+0000) 2022-01-31T20:56:57.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:57 smithi171 conmon[51620]: debug 2022-01-31T20:56:57.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:57.464956+0000) 2022-01-31T20:56:57.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:57 smithi171 conmon[41853]: debug 2022-01-31T20:56:57.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:57.593944+0000) 2022-01-31T20:56:58.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:57 smithi167 conmon[49112]: debug 2022-01-31T20:56:57.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:57.683210+0000) 2022-01-31T20:56:58.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:58 smithi167 conmon[54076]: debug 2022-01-31T20:56:58.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:58.359444+0000) 2022-01-31T20:56:58.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:58 smithi167 conmon[60316]: debug 2022-01-31T20:56:58.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:58.110862+0000) 2022-01-31T20:56:58.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:58 smithi171 conmon[46715]: debug 2022-01-31T20:56:58.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:58.388900+0000) 2022-01-31T20:56:58.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:58 smithi171 conmon[51620]: debug 2022-01-31T20:56:58.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:58.465119+0000) 2022-01-31T20:56:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:58 smithi171 conmon[41853]: debug 2022-01-31T20:56:58.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:58.594126+0000) 2022-01-31T20:56:59.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:58 smithi167 conmon[49112]: debug 2022-01-31T20:56:58.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:58.683443+0000) 2022-01-31T20:56:59.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:59 smithi167 conmon[54076]: debug 2022-01-31T20:56:59.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.359646+0000) 2022-01-31T20:56:59.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:59 smithi167 conmon[60316]: debug 2022-01-31T20:56:59.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.111023+0000) 2022-01-31T20:56:59.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[46715]: debug 2022-01-31T20:56:59.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.389072+0000) 2022-01-31T20:56:59.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[51620]: debug 2022-01-31T20:56:59.465+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.465930+0000) 2022-01-31T20:56:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[35325]: debug 2022-01-31T20:56:59.681+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91058 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:56:59.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[46715]: debug 2022-01-31T20:56:59.671+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.672398+0000) 2022-01-31T20:56:59.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[51620]: debug 2022-01-31T20:56:59.672+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.672779+0000) 2022-01-31T20:56:59.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[41853]: debug 2022-01-31T20:56:59.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.594339+0000) 2022-01-31T20:56:59.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:56:59 smithi171 conmon[41853]: debug 2022-01-31T20:56:59.672+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.673185+0000) 2022-01-31T20:57:00.109 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:56:59 smithi167 conmon[54076]: debug 2022-01-31T20:56:59.672+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.672894+0000) 2022-01-31T20:57:00.110 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:56:59 smithi167 conmon[60316]: debug 2022-01-31T20:56:59.672+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.672532+0000) 2022-01-31T20:57:00.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:59 smithi167 conmon[49112]: debug 2022-01-31T20:56:59.672+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.673337+0000) 2022-01-31T20:57:00.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:56:59 smithi167 conmon[49112]: debug 2022-01-31T20:56:59.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:56:59.683534+0000) 2022-01-31T20:57:00.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:00 smithi167 conmon[54076]: debug 2022-01-31T20:57:00.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:00.359840+0000) 2022-01-31T20:57:00.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:00 smithi167 conmon[60316]: debug 2022-01-31T20:57:00.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:00.111218+0000) 2022-01-31T20:57:00.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:00 smithi171 conmon[46715]: debug 2022-01-31T20:57:00.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:00.389221+0000) 2022-01-31T20:57:00.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:00 smithi171 conmon[51620]: debug 2022-01-31T20:57:00.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:00.466608+0000) 2022-01-31T20:57:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:00 smithi171 conmon[41853]: debug 2022-01-31T20:57:00.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:00.594546+0000) 2022-01-31T20:57:01.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:00 smithi167 conmon[49112]: debug 2022-01-31T20:57:00.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:00.683582+0000) 2022-01-31T20:57:01.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:01 smithi167 conmon[54076]: debug 2022-01-31T20:57:01.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:01.359977+0000) 2022-01-31T20:57:01.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:01 smithi167 conmon[60316]: debug 2022-01-31T20:57:01.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:01.111382+0000) 2022-01-31T20:57:01.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:01 smithi171 conmon[46715]: debug 2022-01-31T20:57:01.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:01.389404+0000) 2022-01-31T20:57:01.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:01 smithi171 conmon[51620]: debug 2022-01-31T20:57:01.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:01.466767+0000) 2022-01-31T20:57:01.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:01 smithi171 conmon[41853]: debug 2022-01-31T20:57:01.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:01.594720+0000) 2022-01-31T20:57:02.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:01 smithi167 conmon[49112]: debug 2022-01-31T20:57:01.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:01.683761+0000) 2022-01-31T20:57:02.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:02 smithi167 conmon[54076]: debug 2022-01-31T20:57:02.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:02.360141+0000) 2022-01-31T20:57:02.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:02 smithi167 conmon[60316]: debug 2022-01-31T20:57:02.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:02.111593+0000) 2022-01-31T20:57:02.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:02 smithi171 conmon[46715]: debug 2022-01-31T20:57:02.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:02.389579+0000) 2022-01-31T20:57:02.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:02 smithi171 conmon[51620]: debug 2022-01-31T20:57:02.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:02.466927+0000) 2022-01-31T20:57:02.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:02 smithi171 conmon[41853]: debug 2022-01-31T20:57:02.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:02.594967+0000) 2022-01-31T20:57:03.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:02 smithi167 conmon[49112]: debug 2022-01-31T20:57:02.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:02.683915+0000) 2022-01-31T20:57:03.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:03 smithi167 conmon[54076]: debug 2022-01-31T20:57:03.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:03.360336+0000) 2022-01-31T20:57:03.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:03 smithi167 conmon[60316]: debug 2022-01-31T20:57:03.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:03.111798+0000) 2022-01-31T20:57:03.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:03 smithi171 conmon[46715]: debug 2022-01-31T20:57:03.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:03.389730+0000) 2022-01-31T20:57:03.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:03 smithi171 conmon[51620]: debug 2022-01-31T20:57:03.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:03.467109+0000) 2022-01-31T20:57:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:03 smithi171 conmon[41853]: debug 2022-01-31T20:57:03.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:03.595119+0000) 2022-01-31T20:57:04.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:03 smithi167 conmon[49112]: debug 2022-01-31T20:57:03.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:03.684113+0000) 2022-01-31T20:57:04.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:04 smithi167 conmon[54076]: debug 2022-01-31T20:57:04.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.360507+0000) 2022-01-31T20:57:04.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:04 smithi167 conmon[60316]: debug 2022-01-31T20:57:04.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.111981+0000) 2022-01-31T20:57:04.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[46715]: debug 2022-01-31T20:57:04.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.389945+0000) 2022-01-31T20:57:04.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[51620]: debug 2022-01-31T20:57:04.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.467309+0000) 2022-01-31T20:57:04.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[46715]: debug 2022-01-31T20:57:04.683+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.684113+0000) 2022-01-31T20:57:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[35325]: debug 2022-01-31T20:57:04.693+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91168 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[41853]: debug 2022-01-31T20:57:04.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.595265+0000) 2022-01-31T20:57:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[41853]: debug 2022-01-31T20:57:04.684+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.685701+0000) 2022-01-31T20:57:04.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:04 smithi171 conmon[51620]: debug 2022-01-31T20:57:04.684+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.685275+0000) 2022-01-31T20:57:05.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:04 smithi167 conmon[49112]: debug 2022-01-31T20:57:04.684+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.685709+0000) 2022-01-31T20:57:05.111 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:04 smithi167 conmon[54076]: debug 2022-01-31T20:57:04.683+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.685450+0000) 2022-01-31T20:57:05.111 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:04 smithi167 conmon[60316]: debug 2022-01-31T20:57:04.683+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:04.685164+0000) 2022-01-31T20:57:05.111 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:05 smithi167 conmon[60316]: debug 2022-01-31T20:57:05.110+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:05.112115+0000) 2022-01-31T20:57:05.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:05 smithi167 conmon[54076]: debug 2022-01-31T20:57:05.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:05.360652+0000) 2022-01-31T20:57:05.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:05 smithi171 conmon[46715]: debug 2022-01-31T20:57:05.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:05.390109+0000) 2022-01-31T20:57:05.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:05 smithi171 conmon[51620]: debug 2022-01-31T20:57:05.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:05.467437+0000) 2022-01-31T20:57:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:05 smithi171 conmon[41853]: debug 2022-01-31T20:57:05.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:05.595458+0000) 2022-01-31T20:57:06.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:06 smithi167 conmon[54076]: debug 2022-01-31T20:57:06.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:06.360794+0000) 2022-01-31T20:57:06.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:06 smithi167 conmon[60316]: debug 2022-01-31T20:57:06.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:06.112220+0000) 2022-01-31T20:57:06.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:06 smithi171 conmon[46715]: debug 2022-01-31T20:57:06.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:06.390252+0000) 2022-01-31T20:57:06.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:06 smithi171 conmon[51620]: debug 2022-01-31T20:57:06.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:06.467518+0000) 2022-01-31T20:57:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:06 smithi171 conmon[41853]: debug 2022-01-31T20:57:06.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:06.595566+0000) 2022-01-31T20:57:07.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:07 smithi167 conmon[54076]: debug 2022-01-31T20:57:07.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:07.360956+0000) 2022-01-31T20:57:07.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:07 smithi167 conmon[60316]: debug 2022-01-31T20:57:07.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:07.112405+0000) 2022-01-31T20:57:07.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:07 smithi171 conmon[46715]: debug 2022-01-31T20:57:07.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:07.390383+0000) 2022-01-31T20:57:07.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:07 smithi171 conmon[51620]: debug 2022-01-31T20:57:07.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:07.467634+0000) 2022-01-31T20:57:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:07 smithi171 conmon[41853]: debug 2022-01-31T20:57:07.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:07.595743+0000) 2022-01-31T20:57:08.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:08 smithi167 conmon[54076]: debug 2022-01-31T20:57:08.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:08.361162+0000) 2022-01-31T20:57:08.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:08 smithi167 conmon[60316]: debug 2022-01-31T20:57:08.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:08.112583+0000) 2022-01-31T20:57:08.682 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:08 smithi171 conmon[35325]: debug 2022-01-31T20:57:08.598+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:57:08.683 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:08 smithi171 conmon[41853]: debug 2022-01-31T20:57:08.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:08.595893+0000) 2022-01-31T20:57:08.683 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:08 smithi171 conmon[46715]: debug 2022-01-31T20:57:08.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:08.390485+0000) 2022-01-31T20:57:08.684 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:08 smithi171 conmon[51620]: debug 2022-01-31T20:57:08.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:08.467794+0000) 2022-01-31T20:57:09.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:09 smithi167 conmon[54076]: debug 2022-01-31T20:57:09.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.361365+0000) 2022-01-31T20:57:09.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:09 smithi167 conmon[60316]: debug 2022-01-31T20:57:09.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.112725+0000) 2022-01-31T20:57:09.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[46715]: debug 2022-01-31T20:57:09.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.390656+0000) 2022-01-31T20:57:09.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[51620]: debug 2022-01-31T20:57:09.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.467953+0000) 2022-01-31T20:57:09.697 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:09 smithi167 conmon[54076]: debug 2022-01-31T20:57:09.695+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.697394+0000) 2022-01-31T20:57:09.697 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:09 smithi167 conmon[60316]: debug 2022-01-31T20:57:09.694+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.696047+0000) 2022-01-31T20:57:09.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[51620]: debug 2022-01-31T20:57:09.694+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.695666+0000) 2022-01-31T20:57:09.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[35325]: debug 2022-01-31T20:57:09.703+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91278 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[41853]: debug 2022-01-31T20:57:09.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.596032+0000) 2022-01-31T20:57:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[41853]: debug 2022-01-31T20:57:09.695+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.696665+0000) 2022-01-31T20:57:09.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:09 smithi171 conmon[46715]: debug 2022-01-31T20:57:09.694+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.695596+0000) 2022-01-31T20:57:10.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:09 smithi167 conmon[49112]: debug 2022-01-31T20:57:09.696+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:09.697922+0000) 2022-01-31T20:57:10.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:10 smithi167 conmon[54076]: debug 2022-01-31T20:57:10.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:10.361571+0000) 2022-01-31T20:57:10.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:10 smithi167 conmon[60316]: debug 2022-01-31T20:57:10.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:10.112909+0000) 2022-01-31T20:57:10.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:10 smithi171 conmon[46715]: debug 2022-01-31T20:57:10.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:10.390787+0000) 2022-01-31T20:57:10.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:10 smithi171 conmon[51620]: debug 2022-01-31T20:57:10.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:10.468102+0000) 2022-01-31T20:57:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:10 smithi171 conmon[41853]: debug 2022-01-31T20:57:10.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:10.596195+0000) 2022-01-31T20:57:11.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:11 smithi167 conmon[54076]: debug 2022-01-31T20:57:11.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:11.361667+0000) 2022-01-31T20:57:11.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:11 smithi167 conmon[60316]: debug 2022-01-31T20:57:11.112+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:11.113059+0000) 2022-01-31T20:57:11.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:11 smithi171 conmon[46715]: debug 2022-01-31T20:57:11.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:11.390929+0000) 2022-01-31T20:57:11.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:11 smithi171 conmon[51620]: debug 2022-01-31T20:57:11.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:11.468224+0000) 2022-01-31T20:57:11.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:11 smithi167 conmon[49112]: debug 2022-01-31T20:57:11.433+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:11.434445+0000) 2022-01-31T20:57:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:11 smithi171 conmon[41853]: debug 2022-01-31T20:57:11.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:11.596344+0000) 2022-01-31T20:57:12.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:12 smithi167 conmon[54076]: debug 2022-01-31T20:57:12.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:12.361842+0000) 2022-01-31T20:57:12.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:12 smithi167 conmon[60316]: debug 2022-01-31T20:57:12.112+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:12.113268+0000) 2022-01-31T20:57:12.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:12 smithi171 conmon[46715]: debug 2022-01-31T20:57:12.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:12.391114+0000) 2022-01-31T20:57:12.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:12 smithi171 conmon[51620]: debug 2022-01-31T20:57:12.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:12.468359+0000) 2022-01-31T20:57:12.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:12 smithi167 conmon[49112]: debug 2022-01-31T20:57:12.433+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:12.434671+0000) 2022-01-31T20:57:12.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:12 smithi171 conmon[41853]: debug 2022-01-31T20:57:12.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:12.596518+0000) 2022-01-31T20:57:13.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:13 smithi167 conmon[54076]: debug 2022-01-31T20:57:13.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:13.361975+0000) 2022-01-31T20:57:13.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:13 smithi167 conmon[60316]: debug 2022-01-31T20:57:13.111+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:13.113449+0000) 2022-01-31T20:57:13.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:13 smithi171 conmon[46715]: debug 2022-01-31T20:57:13.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:13.391280+0000) 2022-01-31T20:57:13.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:13 smithi171 conmon[51620]: debug 2022-01-31T20:57:13.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:13.468537+0000) 2022-01-31T20:57:13.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:13 smithi167 conmon[49112]: debug 2022-01-31T20:57:13.433+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:13.434860+0000) 2022-01-31T20:57:13.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:13 smithi171 conmon[41853]: debug 2022-01-31T20:57:13.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:13.596671+0000) 2022-01-31T20:57:14.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:14 smithi167 conmon[54076]: debug 2022-01-31T20:57:14.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.362161+0000) 2022-01-31T20:57:14.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:14 smithi167 conmon[60316]: debug 2022-01-31T20:57:14.112+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.113651+0000) 2022-01-31T20:57:14.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[46715]: debug 2022-01-31T20:57:14.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.391445+0000) 2022-01-31T20:57:14.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[51620]: debug 2022-01-31T20:57:14.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.468692+0000) 2022-01-31T20:57:14.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:14 smithi167 conmon[49112]: debug 2022-01-31T20:57:14.433+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.435063+0000) 2022-01-31T20:57:14.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[35325]: debug 2022-01-31T20:57:14.716+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91389 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:14.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[46715]: debug 2022-01-31T20:57:14.706+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.706940+0000) 2022-01-31T20:57:14.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[51620]: debug 2022-01-31T20:57:14.705+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.706487+0000) 2022-01-31T20:57:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[41853]: debug 2022-01-31T20:57:14.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.596878+0000) 2022-01-31T20:57:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:14 smithi171 conmon[41853]: debug 2022-01-31T20:57:14.706+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.707301+0000) 2022-01-31T20:57:15.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:14 smithi167 conmon[49112]: debug 2022-01-31T20:57:14.706+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.707630+0000) 2022-01-31T20:57:15.112 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:14 smithi167 conmon[54076]: debug 2022-01-31T20:57:14.706+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.708358+0000) 2022-01-31T20:57:15.113 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:14 smithi167 conmon[60316]: debug 2022-01-31T20:57:14.706+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:14.708020+0000) 2022-01-31T20:57:15.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:15 smithi167 conmon[54076]: debug 2022-01-31T20:57:15.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:15.362332+0000) 2022-01-31T20:57:15.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:15 smithi167 conmon[60316]: debug 2022-01-31T20:57:15.113+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:15.113844+0000) 2022-01-31T20:57:15.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:15 smithi171 conmon[46715]: debug 2022-01-31T20:57:15.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:15.391617+0000) 2022-01-31T20:57:15.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:15 smithi171 conmon[51620]: debug 2022-01-31T20:57:15.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:15.468833+0000) 2022-01-31T20:57:15.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:15 smithi167 conmon[49112]: debug 2022-01-31T20:57:15.433+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:15.435259+0000) 2022-01-31T20:57:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:15 smithi171 conmon[41853]: debug 2022-01-31T20:57:15.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:15.597081+0000) 2022-01-31T20:57:16.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:16 smithi167 conmon[54076]: debug 2022-01-31T20:57:16.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:16.362531+0000) 2022-01-31T20:57:16.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:16 smithi167 conmon[60316]: debug 2022-01-31T20:57:16.112+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:16.113938+0000) 2022-01-31T20:57:16.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:16 smithi171 conmon[46715]: debug 2022-01-31T20:57:16.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:16.391779+0000) 2022-01-31T20:57:16.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:16 smithi171 conmon[51620]: debug 2022-01-31T20:57:16.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:16.468916+0000) 2022-01-31T20:57:16.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:16 smithi167 conmon[49112]: debug 2022-01-31T20:57:16.434+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:16.435388+0000) 2022-01-31T20:57:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:16 smithi171 conmon[41853]: debug 2022-01-31T20:57:16.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:16.597220+0000) 2022-01-31T20:57:17.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:17 smithi167 conmon[54076]: debug 2022-01-31T20:57:17.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:17.362728+0000) 2022-01-31T20:57:17.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:17 smithi167 conmon[60316]: debug 2022-01-31T20:57:17.113+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:17.114101+0000) 2022-01-31T20:57:17.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:17 smithi171 conmon[46715]: debug 2022-01-31T20:57:17.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:17.391944+0000) 2022-01-31T20:57:17.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:17 smithi171 conmon[51620]: debug 2022-01-31T20:57:17.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:17.469093+0000) 2022-01-31T20:57:17.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:17 smithi167 conmon[49112]: debug 2022-01-31T20:57:17.434+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:17.435575+0000) 2022-01-31T20:57:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:17 smithi171 conmon[41853]: debug 2022-01-31T20:57:17.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:17.597415+0000) 2022-01-31T20:57:18.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:18 smithi167 conmon[54076]: debug 2022-01-31T20:57:18.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:18.362919+0000) 2022-01-31T20:57:18.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:18 smithi167 conmon[60316]: debug 2022-01-31T20:57:18.113+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:18.114284+0000) 2022-01-31T20:57:18.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:18 smithi171 conmon[46715]: debug 2022-01-31T20:57:18.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:18.392121+0000) 2022-01-31T20:57:18.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:18 smithi171 conmon[51620]: debug 2022-01-31T20:57:18.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:18.469287+0000) 2022-01-31T20:57:18.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:18 smithi167 conmon[49112]: debug 2022-01-31T20:57:18.435+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:18.435828+0000) 2022-01-31T20:57:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:18 smithi171 conmon[41853]: debug 2022-01-31T20:57:18.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:18.597600+0000) 2022-01-31T20:57:19.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[54076]: debug 2022-01-31T20:57:19.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.363081+0000) 2022-01-31T20:57:19.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[60316]: debug 2022-01-31T20:57:19.113+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.114438+0000) 2022-01-31T20:57:19.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[60316]: 2022-01-31T20:57:19.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[46715]: debug 2022-01-31T20:57:19.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.392342+0000) 2022-01-31T20:57:19.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[51620]: debug 2022-01-31T20:57:19.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.469472+0000) 2022-01-31T20:57:19.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[49112]: debug 2022-01-31T20:57:19.434+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.436014+0000) 2022-01-31T20:57:19.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[35325]: debug 2022-01-31T20:57:19.727+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91499 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:19.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[51620]: debug 2022-01-31T20:57:19.717+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.718913+0000) 2022-01-31T20:57:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[41853]: debug 2022-01-31T20:57:19.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.597792+0000) 2022-01-31T20:57:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[41853]: debug 2022-01-31T20:57:19.717+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.719643+0000) 2022-01-31T20:57:19.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:19 smithi171 conmon[46715]: debug 2022-01-31T20:57:19.718+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.720261+0000) 2022-01-31T20:57:20.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[49112]: debug 2022-01-31T20:57:19.718+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.719528+0000) 2022-01-31T20:57:20.114 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[54076]: debug 2022-01-31T20:57:19.718+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.719886+0000) 2022-01-31T20:57:20.115 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:19 smithi167 conmon[60316]: debug 2022-01-31T20:57:19.718+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:19.720476+0000) 2022-01-31T20:57:20.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:20 smithi167 conmon[54076]: debug 2022-01-31T20:57:20.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:20.363280+0000) 2022-01-31T20:57:20.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:20 smithi167 conmon[60316]: debug 2022-01-31T20:57:20.113+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:20.114660+0000) 2022-01-31T20:57:20.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:20 smithi171 conmon[46715]: debug 2022-01-31T20:57:20.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:20.392722+0000) 2022-01-31T20:57:20.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:20 smithi171 conmon[51620]: debug 2022-01-31T20:57:20.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:20.469620+0000) 2022-01-31T20:57:20.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:20 smithi167 conmon[49112]: debug 2022-01-31T20:57:20.434+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:20.436189+0000) 2022-01-31T20:57:20.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:20 smithi171 conmon[41853]: debug 2022-01-31T20:57:20.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:20.597973+0000) 2022-01-31T20:57:21.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:21 smithi167 conmon[60316]: debug 2022-01-31T20:57:21.114+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:21.114792+0000) 2022-01-31T20:57:21.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:21 smithi167 conmon[54076]: debug 2022-01-31T20:57:21.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:21.363445+0000) 2022-01-31T20:57:21.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:21 smithi171 conmon[46715]: debug 2022-01-31T20:57:21.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:21.392808+0000) 2022-01-31T20:57:21.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:21 smithi171 conmon[51620]: debug 2022-01-31T20:57:21.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:21.469783+0000) 2022-01-31T20:57:21.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:21 smithi167 conmon[49112]: debug 2022-01-31T20:57:21.434+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:21.436344+0000) 2022-01-31T20:57:21.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:21 smithi171 conmon[41853]: debug 2022-01-31T20:57:21.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:21.598093+0000) 2022-01-31T20:57:22.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:22 smithi167 conmon[60316]: debug 2022-01-31T20:57:22.113+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:22.114937+0000) 2022-01-31T20:57:22.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:22 smithi167 conmon[54076]: debug 2022-01-31T20:57:22.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:22.363668+0000) 2022-01-31T20:57:22.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:22 smithi171 conmon[46715]: debug 2022-01-31T20:57:22.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:22.392988+0000) 2022-01-31T20:57:22.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:22 smithi171 conmon[51620]: debug 2022-01-31T20:57:22.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:22.469950+0000) 2022-01-31T20:57:22.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:22 smithi167 conmon[49112]: debug 2022-01-31T20:57:22.435+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:22.436537+0000) 2022-01-31T20:57:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:22 smithi171 conmon[41853]: debug 2022-01-31T20:57:22.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:22.598266+0000) 2022-01-31T20:57:23.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:23 smithi167 conmon[54076]: debug 2022-01-31T20:57:23.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:23.363829+0000) 2022-01-31T20:57:23.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:23 smithi167 conmon[60316]: debug 2022-01-31T20:57:23.114+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:23.115136+0000) 2022-01-31T20:57:23.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:23 smithi171 conmon[46715]: debug 2022-01-31T20:57:23.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:23.393179+0000) 2022-01-31T20:57:23.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:23 smithi171 conmon[51620]: debug 2022-01-31T20:57:23.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:23.470083+0000) 2022-01-31T20:57:23.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:23 smithi167 conmon[49112]: debug 2022-01-31T20:57:23.435+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:23.436719+0000) 2022-01-31T20:57:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:23 smithi171 conmon[35325]: debug 2022-01-31T20:57:23.598+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:57:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:23 smithi171 conmon[41853]: debug 2022-01-31T20:57:23.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:23.598445+0000) 2022-01-31T20:57:24.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:24 smithi167 conmon[60316]: debug 2022-01-31T20:57:24.114+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.115338+0000) 2022-01-31T20:57:24.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:24 smithi167 conmon[54076]: debug 2022-01-31T20:57:24.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.364030+0000) 2022-01-31T20:57:24.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[46715]: debug 2022-01-31T20:57:24.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.393345+0000) 2022-01-31T20:57:24.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[51620]: debug 2022-01-31T20:57:24.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.470268+0000) 2022-01-31T20:57:24.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:24 smithi167 conmon[49112]: debug 2022-01-31T20:57:24.436+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.436909+0000) 2022-01-31T20:57:24.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[46715]: debug 2022-01-31T20:57:24.729+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.730934+0000) 2022-01-31T20:57:24.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[51620]: debug 2022-01-31T20:57:24.729+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.731271+0000) 2022-01-31T20:57:24.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[35325]: debug 2022-01-31T20:57:24.738+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91609 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[41853]: debug 2022-01-31T20:57:24.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.598619+0000) 2022-01-31T20:57:24.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:24 smithi171 conmon[41853]: debug 2022-01-31T20:57:24.729+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.731515+0000) 2022-01-31T20:57:25.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:24 smithi167 conmon[49112]: debug 2022-01-31T20:57:24.730+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.732075+0000) 2022-01-31T20:57:25.114 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:24 smithi167 conmon[54076]: debug 2022-01-31T20:57:24.730+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.731574+0000) 2022-01-31T20:57:25.115 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:24 smithi167 conmon[60316]: debug 2022-01-31T20:57:24.730+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:24.731785+0000) 2022-01-31T20:57:25.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:25 smithi167 conmon[54076]: debug 2022-01-31T20:57:25.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:25.364216+0000) 2022-01-31T20:57:25.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:25 smithi167 conmon[60316]: debug 2022-01-31T20:57:25.114+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:25.115575+0000) 2022-01-31T20:57:25.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:25 smithi171 conmon[46715]: debug 2022-01-31T20:57:25.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:25.393511+0000) 2022-01-31T20:57:25.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:25 smithi171 conmon[51620]: debug 2022-01-31T20:57:25.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:25.470476+0000) 2022-01-31T20:57:25.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:25 smithi167 conmon[49112]: debug 2022-01-31T20:57:25.435+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:25.437073+0000) 2022-01-31T20:57:25.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:25 smithi171 conmon[41853]: debug 2022-01-31T20:57:25.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:25.598739+0000) 2022-01-31T20:57:26.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:26 smithi167 conmon[54076]: debug 2022-01-31T20:57:26.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:26.364393+0000) 2022-01-31T20:57:26.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:26 smithi167 conmon[60316]: debug 2022-01-31T20:57:26.115+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:26.115713+0000) 2022-01-31T20:57:26.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:26 smithi171 conmon[46715]: debug 2022-01-31T20:57:26.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:26.393620+0000) 2022-01-31T20:57:26.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:26 smithi171 conmon[51620]: debug 2022-01-31T20:57:26.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:26.470621+0000) 2022-01-31T20:57:26.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:26 smithi167 conmon[49112]: debug 2022-01-31T20:57:26.436+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:26.437216+0000) 2022-01-31T20:57:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:26 smithi171 conmon[41853]: debug 2022-01-31T20:57:26.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:26.598921+0000) 2022-01-31T20:57:27.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:27 smithi167 conmon[54076]: debug 2022-01-31T20:57:27.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:27.364578+0000) 2022-01-31T20:57:27.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:27 smithi167 conmon[60316]: debug 2022-01-31T20:57:27.114+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:27.115860+0000) 2022-01-31T20:57:27.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:27 smithi171 conmon[46715]: debug 2022-01-31T20:57:27.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:27.393804+0000) 2022-01-31T20:57:27.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:27 smithi171 conmon[51620]: debug 2022-01-31T20:57:27.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:27.470825+0000) 2022-01-31T20:57:27.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:27 smithi167 conmon[49112]: debug 2022-01-31T20:57:27.436+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:27.437368+0000) 2022-01-31T20:57:27.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:27 smithi171 conmon[41853]: debug 2022-01-31T20:57:27.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:27.599050+0000) 2022-01-31T20:57:28.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:28 smithi167 conmon[54076]: debug 2022-01-31T20:57:28.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:28.364778+0000) 2022-01-31T20:57:28.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:28 smithi167 conmon[60316]: debug 2022-01-31T20:57:28.115+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:28.116045+0000) 2022-01-31T20:57:28.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:28 smithi171 conmon[46715]: debug 2022-01-31T20:57:28.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:28.393908+0000) 2022-01-31T20:57:28.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:28 smithi171 conmon[51620]: debug 2022-01-31T20:57:28.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:28.471008+0000) 2022-01-31T20:57:28.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:28 smithi167 conmon[49112]: debug 2022-01-31T20:57:28.437+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:28.437579+0000) 2022-01-31T20:57:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:28 smithi171 conmon[41853]: debug 2022-01-31T20:57:28.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:28.599235+0000) 2022-01-31T20:57:29.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:29 smithi167 conmon[54076]: debug 2022-01-31T20:57:29.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.364939+0000) 2022-01-31T20:57:29.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:29 smithi167 conmon[60316]: debug 2022-01-31T20:57:29.115+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.116220+0000) 2022-01-31T20:57:29.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[46715]: debug 2022-01-31T20:57:29.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.394020+0000) 2022-01-31T20:57:29.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[51620]: debug 2022-01-31T20:57:29.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.471210+0000) 2022-01-31T20:57:29.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:29 smithi167 conmon[49112]: debug 2022-01-31T20:57:29.437+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.437784+0000) 2022-01-31T20:57:29.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[46715]: debug 2022-01-31T20:57:29.742+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.742959+0000) 2022-01-31T20:57:29.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[51620]: debug 2022-01-31T20:57:29.742+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.743738+0000) 2022-01-31T20:57:29.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[35325]: debug 2022-01-31T20:57:29.753+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91720 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[41853]: debug 2022-01-31T20:57:29.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.599416+0000) 2022-01-31T20:57:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:29 smithi171 conmon[41853]: debug 2022-01-31T20:57:29.741+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.742309+0000) 2022-01-31T20:57:30.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:29 smithi167 conmon[49112]: debug 2022-01-31T20:57:29.742+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.743278+0000) 2022-01-31T20:57:30.115 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:29 smithi167 conmon[54076]: debug 2022-01-31T20:57:29.743+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.743596+0000) 2022-01-31T20:57:30.115 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:29 smithi167 conmon[60316]: debug 2022-01-31T20:57:29.741+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:29.742438+0000) 2022-01-31T20:57:30.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:30 smithi167 conmon[54076]: debug 2022-01-31T20:57:30.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:30.365118+0000) 2022-01-31T20:57:30.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:30 smithi167 conmon[60316]: debug 2022-01-31T20:57:30.115+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:30.116457+0000) 2022-01-31T20:57:30.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:30 smithi171 conmon[46715]: debug 2022-01-31T20:57:30.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:30.394212+0000) 2022-01-31T20:57:30.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:30 smithi171 conmon[51620]: debug 2022-01-31T20:57:30.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:30.471360+0000) 2022-01-31T20:57:30.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:30 smithi167 conmon[49112]: debug 2022-01-31T20:57:30.437+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:30.437986+0000) 2022-01-31T20:57:30.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:30 smithi171 conmon[41853]: debug 2022-01-31T20:57:30.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:30.599604+0000) 2022-01-31T20:57:31.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:31 smithi167 conmon[54076]: debug 2022-01-31T20:57:31.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:31.365264+0000) 2022-01-31T20:57:31.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:31 smithi167 conmon[60316]: debug 2022-01-31T20:57:31.116+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:31.116657+0000) 2022-01-31T20:57:31.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:31 smithi171 conmon[46715]: debug 2022-01-31T20:57:31.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:31.394345+0000) 2022-01-31T20:57:31.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:31 smithi171 conmon[51620]: debug 2022-01-31T20:57:31.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:31.471495+0000) 2022-01-31T20:57:31.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:31 smithi167 conmon[49112]: debug 2022-01-31T20:57:31.437+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:31.438143+0000) 2022-01-31T20:57:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:31 smithi171 conmon[41853]: debug 2022-01-31T20:57:31.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:31.599764+0000) 2022-01-31T20:57:32.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:32 smithi167 conmon[54076]: debug 2022-01-31T20:57:32.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:32.365482+0000) 2022-01-31T20:57:32.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:32 smithi167 conmon[60316]: debug 2022-01-31T20:57:32.116+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:32.116836+0000) 2022-01-31T20:57:32.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:32 smithi171 conmon[46715]: debug 2022-01-31T20:57:32.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:32.394504+0000) 2022-01-31T20:57:32.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:32 smithi171 conmon[51620]: debug 2022-01-31T20:57:32.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:32.471655+0000) 2022-01-31T20:57:32.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:32 smithi167 conmon[49112]: debug 2022-01-31T20:57:32.437+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:32.438367+0000) 2022-01-31T20:57:32.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:32 smithi171 conmon[41853]: debug 2022-01-31T20:57:32.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:32.599928+0000) 2022-01-31T20:57:33.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:33 smithi167 conmon[54076]: debug 2022-01-31T20:57:33.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:33.365687+0000) 2022-01-31T20:57:33.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:33 smithi167 conmon[60316]: debug 2022-01-31T20:57:33.116+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:33.117006+0000) 2022-01-31T20:57:33.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:33 smithi171 conmon[46715]: debug 2022-01-31T20:57:33.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:33.394673+0000) 2022-01-31T20:57:33.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:33 smithi171 conmon[51620]: debug 2022-01-31T20:57:33.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:33.471809+0000) 2022-01-31T20:57:33.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:33 smithi167 conmon[49112]: debug 2022-01-31T20:57:33.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:33.438535+0000) 2022-01-31T20:57:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:33 smithi171 conmon[41853]: debug 2022-01-31T20:57:33.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:33.600073+0000) 2022-01-31T20:57:34.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:34 smithi167 conmon[54076]: debug 2022-01-31T20:57:34.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.365841+0000) 2022-01-31T20:57:34.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:34 smithi167 conmon[60316]: debug 2022-01-31T20:57:34.116+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.117160+0000) 2022-01-31T20:57:34.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[51620]: debug 2022-01-31T20:57:34.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.471989+0000) 2022-01-31T20:57:34.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[46715]: debug 2022-01-31T20:57:34.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.394860+0000) 2022-01-31T20:57:34.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:34 smithi167 conmon[49112]: debug 2022-01-31T20:57:34.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.438698+0000) 2022-01-31T20:57:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[35325]: debug 2022-01-31T20:57:34.766+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91831 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:34.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[46715]: debug 2022-01-31T20:57:34.755+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.756542+0000) 2022-01-31T20:57:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[41853]: debug 2022-01-31T20:57:34.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.600222+0000) 2022-01-31T20:57:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[41853]: debug 2022-01-31T20:57:34.756+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.756972+0000) 2022-01-31T20:57:34.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:34 smithi171 conmon[51620]: debug 2022-01-31T20:57:34.755+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.756718+0000) 2022-01-31T20:57:35.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:34 smithi167 conmon[49112]: debug 2022-01-31T20:57:34.757+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.757610+0000) 2022-01-31T20:57:35.116 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:34 smithi167 conmon[54076]: debug 2022-01-31T20:57:34.757+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.757763+0000) 2022-01-31T20:57:35.116 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:34 smithi167 conmon[60316]: debug 2022-01-31T20:57:34.757+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:34.757893+0000) 2022-01-31T20:57:35.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:35 smithi167 conmon[54076]: debug 2022-01-31T20:57:35.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:35.365970+0000) 2022-01-31T20:57:35.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:35 smithi167 conmon[60316]: debug 2022-01-31T20:57:35.116+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:35.117362+0000) 2022-01-31T20:57:35.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:35 smithi171 conmon[46715]: debug 2022-01-31T20:57:35.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:35.395028+0000) 2022-01-31T20:57:35.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:35 smithi171 conmon[51620]: debug 2022-01-31T20:57:35.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:35.472199+0000) 2022-01-31T20:57:35.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:35 smithi167 conmon[49112]: debug 2022-01-31T20:57:35.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:35.438865+0000) 2022-01-31T20:57:35.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:35 smithi171 conmon[41853]: debug 2022-01-31T20:57:35.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:35.600397+0000) 2022-01-31T20:57:36.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:36 smithi167 conmon[54076]: debug 2022-01-31T20:57:36.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:36.366108+0000) 2022-01-31T20:57:36.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:36 smithi167 conmon[60316]: debug 2022-01-31T20:57:36.117+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:36.117546+0000) 2022-01-31T20:57:36.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:36 smithi171 conmon[46715]: debug 2022-01-31T20:57:36.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:36.395194+0000) 2022-01-31T20:57:36.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:36 smithi171 conmon[51620]: debug 2022-01-31T20:57:36.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:36.472363+0000) 2022-01-31T20:57:36.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:36 smithi167 conmon[49112]: debug 2022-01-31T20:57:36.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:36.438973+0000) 2022-01-31T20:57:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:36 smithi171 conmon[41853]: debug 2022-01-31T20:57:36.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:36.600530+0000) 2022-01-31T20:57:37.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:37 smithi167 conmon[54076]: debug 2022-01-31T20:57:37.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:37.366267+0000) 2022-01-31T20:57:37.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:37 smithi167 conmon[60316]: debug 2022-01-31T20:57:37.117+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:37.117684+0000) 2022-01-31T20:57:37.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:37 smithi171 conmon[46715]: debug 2022-01-31T20:57:37.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:37.395381+0000) 2022-01-31T20:57:37.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:37 smithi171 conmon[51620]: debug 2022-01-31T20:57:37.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:37.472519+0000) 2022-01-31T20:57:37.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:37 smithi167 conmon[49112]: debug 2022-01-31T20:57:37.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:37.439074+0000) 2022-01-31T20:57:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:37 smithi171 conmon[41853]: debug 2022-01-31T20:57:37.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:37.600737+0000) 2022-01-31T20:57:38.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:38 smithi167 conmon[54076]: debug 2022-01-31T20:57:38.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:38.366505+0000) 2022-01-31T20:57:38.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:38 smithi167 conmon[60316]: debug 2022-01-31T20:57:38.117+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:38.117870+0000) 2022-01-31T20:57:38.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:38 smithi171 conmon[46715]: debug 2022-01-31T20:57:38.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:38.395577+0000) 2022-01-31T20:57:38.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:38 smithi171 conmon[51620]: debug 2022-01-31T20:57:38.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:38.472735+0000) 2022-01-31T20:57:38.672 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:38 smithi167 conmon[49112]: debug 2022-01-31T20:57:38.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:38.439241+0000) 2022-01-31T20:57:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:38 smithi171 conmon[41853]: debug 2022-01-31T20:57:38.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:38.600909+0000) 2022-01-31T20:57:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:38 smithi171 conmon[35325]: debug 2022-01-31T20:57:38.599+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:57:39.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:39 smithi167 conmon[54076]: debug 2022-01-31T20:57:39.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.366682+0000) 2022-01-31T20:57:39.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:39 smithi167 conmon[60316]: debug 2022-01-31T20:57:39.117+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.118097+0000) 2022-01-31T20:57:39.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[46715]: debug 2022-01-31T20:57:39.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.395731+0000) 2022-01-31T20:57:39.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[51620]: debug 2022-01-31T20:57:39.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.472951+0000) 2022-01-31T20:57:39.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:39 smithi167 conmon[49112]: debug 2022-01-31T20:57:39.438+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.439469+0000) 2022-01-31T20:57:39.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[41853]: debug 2022-01-31T20:57:39.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.601078+0000) 2022-01-31T20:57:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[41853]: debug 2022-01-31T20:57:39.768+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.768968+0000) 2022-01-31T20:57:39.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[46715]: debug 2022-01-31T20:57:39.767+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.768639+0000) 2022-01-31T20:57:39.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[35325]: debug 2022-01-31T20:57:39.778+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 91941 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:39.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:39 smithi171 conmon[51620]: debug 2022-01-31T20:57:39.768+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.769219+0000) 2022-01-31T20:57:40.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:39 smithi167 conmon[49112]: debug 2022-01-31T20:57:39.769+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.769538+0000) 2022-01-31T20:57:40.117 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:39 smithi167 conmon[60316]: debug 2022-01-31T20:57:39.770+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.770645+0000) 2022-01-31T20:57:40.117 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:39 smithi167 conmon[54076]: debug 2022-01-31T20:57:39.769+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:39.769823+0000) 2022-01-31T20:57:40.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:40 smithi167 conmon[54076]: debug 2022-01-31T20:57:40.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:40.366892+0000) 2022-01-31T20:57:40.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:40 smithi167 conmon[60316]: debug 2022-01-31T20:57:40.117+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:40.118294+0000) 2022-01-31T20:57:40.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:40 smithi171 conmon[46715]: debug 2022-01-31T20:57:40.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:40.395902+0000) 2022-01-31T20:57:40.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:40 smithi171 conmon[51620]: debug 2022-01-31T20:57:40.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:40.473139+0000) 2022-01-31T20:57:40.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:40 smithi167 conmon[49112]: debug 2022-01-31T20:57:40.439+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:40.439577+0000) 2022-01-31T20:57:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:40 smithi171 conmon[41853]: debug 2022-01-31T20:57:40.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:40.601227+0000) 2022-01-31T20:57:41.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:41 smithi167 conmon[54076]: debug 2022-01-31T20:57:41.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:41.367064+0000) 2022-01-31T20:57:41.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:41 smithi167 conmon[60316]: debug 2022-01-31T20:57:41.117+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:41.118417+0000) 2022-01-31T20:57:41.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:41 smithi171 conmon[46715]: debug 2022-01-31T20:57:41.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:41.396057+0000) 2022-01-31T20:57:41.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:41 smithi171 conmon[51620]: debug 2022-01-31T20:57:41.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:41.473321+0000) 2022-01-31T20:57:41.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:41 smithi167 conmon[49112]: debug 2022-01-31T20:57:41.439+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:41.439690+0000) 2022-01-31T20:57:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:41 smithi171 conmon[41853]: debug 2022-01-31T20:57:41.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:41.601401+0000) 2022-01-31T20:57:42.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:42 smithi167 conmon[54076]: debug 2022-01-31T20:57:42.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:42.367244+0000) 2022-01-31T20:57:42.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:42 smithi167 conmon[60316]: debug 2022-01-31T20:57:42.118+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:42.118567+0000) 2022-01-31T20:57:42.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:42 smithi171 conmon[51620]: debug 2022-01-31T20:57:42.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:42.473526+0000) 2022-01-31T20:57:42.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:42 smithi171 conmon[46715]: debug 2022-01-31T20:57:42.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:42.396240+0000) 2022-01-31T20:57:42.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:42 smithi167 conmon[49112]: debug 2022-01-31T20:57:42.439+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:42.439840+0000) 2022-01-31T20:57:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:42 smithi171 conmon[41853]: debug 2022-01-31T20:57:42.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:42.601618+0000) 2022-01-31T20:57:43.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:43 smithi167 conmon[54076]: debug 2022-01-31T20:57:43.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:43.367423+0000) 2022-01-31T20:57:43.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:43 smithi167 conmon[60316]: debug 2022-01-31T20:57:43.118+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:43.118748+0000) 2022-01-31T20:57:43.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:43 smithi171 conmon[46715]: debug 2022-01-31T20:57:43.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:43.396400+0000) 2022-01-31T20:57:43.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:43 smithi171 conmon[51620]: debug 2022-01-31T20:57:43.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:43.473729+0000) 2022-01-31T20:57:43.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:43 smithi167 conmon[49112]: debug 2022-01-31T20:57:43.439+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:43.440033+0000) 2022-01-31T20:57:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:43 smithi171 conmon[41853]: debug 2022-01-31T20:57:43.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:43.601819+0000) 2022-01-31T20:57:44.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:44 smithi167 conmon[54076]: debug 2022-01-31T20:57:44.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.367629+0000) 2022-01-31T20:57:44.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:44 smithi167 conmon[60316]: debug 2022-01-31T20:57:44.118+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.118905+0000) 2022-01-31T20:57:44.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[46715]: debug 2022-01-31T20:57:44.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.396552+0000) 2022-01-31T20:57:44.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[51620]: debug 2022-01-31T20:57:44.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.473962+0000) 2022-01-31T20:57:44.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:44 smithi167 conmon[49112]: debug 2022-01-31T20:57:44.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.440252+0000) 2022-01-31T20:57:44.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[46715]: debug 2022-01-31T20:57:44.780+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.781500+0000) 2022-01-31T20:57:44.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[35325]: debug 2022-01-31T20:57:44.790+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92051 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[41853]: debug 2022-01-31T20:57:44.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.602027+0000) 2022-01-31T20:57:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[41853]: debug 2022-01-31T20:57:44.781+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.782231+0000) 2022-01-31T20:57:44.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:44 smithi171 conmon[51620]: debug 2022-01-31T20:57:44.780+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.781398+0000) 2022-01-31T20:57:45.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:44 smithi167 conmon[49112]: debug 2022-01-31T20:57:44.781+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.782375+0000) 2022-01-31T20:57:45.118 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:44 smithi167 conmon[54076]: debug 2022-01-31T20:57:44.781+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.782057+0000) 2022-01-31T20:57:45.118 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:44 smithi167 conmon[60316]: debug 2022-01-31T20:57:44.782+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:44.783036+0000) 2022-01-31T20:57:45.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:45 smithi167 conmon[60316]: debug 2022-01-31T20:57:45.118+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:45.119077+0000) 2022-01-31T20:57:45.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:45 smithi167 conmon[54076]: debug 2022-01-31T20:57:45.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:45.367820+0000) 2022-01-31T20:57:45.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:45 smithi171 conmon[46715]: debug 2022-01-31T20:57:45.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:45.396771+0000) 2022-01-31T20:57:45.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:45 smithi171 conmon[51620]: debug 2022-01-31T20:57:45.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:45.474149+0000) 2022-01-31T20:57:45.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:45 smithi167 conmon[49112]: debug 2022-01-31T20:57:45.439+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:45.440463+0000) 2022-01-31T20:57:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:45 smithi171 conmon[41853]: debug 2022-01-31T20:57:45.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:45.602173+0000) 2022-01-31T20:57:46.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:46 smithi167 conmon[54076]: debug 2022-01-31T20:57:46.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:46.367966+0000) 2022-01-31T20:57:46.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:46 smithi167 conmon[60316]: debug 2022-01-31T20:57:46.118+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:46.119234+0000) 2022-01-31T20:57:46.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:46 smithi171 conmon[46715]: debug 2022-01-31T20:57:46.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:46.396944+0000) 2022-01-31T20:57:46.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:46 smithi171 conmon[51620]: debug 2022-01-31T20:57:46.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:46.474287+0000) 2022-01-31T20:57:46.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:46 smithi167 conmon[49112]: debug 2022-01-31T20:57:46.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:46.440595+0000) 2022-01-31T20:57:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:46 smithi171 conmon[41853]: debug 2022-01-31T20:57:46.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:46.602330+0000) 2022-01-31T20:57:47.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:47 smithi167 conmon[54076]: debug 2022-01-31T20:57:47.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:47.368125+0000) 2022-01-31T20:57:47.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:47 smithi167 conmon[60316]: debug 2022-01-31T20:57:47.118+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:47.119401+0000) 2022-01-31T20:57:47.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:47 smithi171 conmon[46715]: debug 2022-01-31T20:57:47.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:47.397122+0000) 2022-01-31T20:57:47.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:47 smithi171 conmon[51620]: debug 2022-01-31T20:57:47.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:47.474443+0000) 2022-01-31T20:57:47.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:47 smithi167 conmon[49112]: debug 2022-01-31T20:57:47.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:47.440709+0000) 2022-01-31T20:57:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:47 smithi171 conmon[41853]: debug 2022-01-31T20:57:47.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:47.602512+0000) 2022-01-31T20:57:48.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:48 smithi167 conmon[54076]: debug 2022-01-31T20:57:48.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:48.368339+0000) 2022-01-31T20:57:48.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:48 smithi167 conmon[60316]: debug 2022-01-31T20:57:48.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:48.119539+0000) 2022-01-31T20:57:48.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:48 smithi171 conmon[46715]: debug 2022-01-31T20:57:48.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:48.397313+0000) 2022-01-31T20:57:48.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:48 smithi171 conmon[51620]: debug 2022-01-31T20:57:48.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:48.474602+0000) 2022-01-31T20:57:48.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:48 smithi167 conmon[49112]: debug 2022-01-31T20:57:48.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:48.440896+0000) 2022-01-31T20:57:48.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:48 smithi171 conmon[41853]: debug 2022-01-31T20:57:48.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:48.602711+0000) 2022-01-31T20:57:49.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:49 smithi167 conmon[54076]: debug 2022-01-31T20:57:49.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.368511+0000) 2022-01-31T20:57:49.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:49 smithi167 conmon[60316]: debug 2022-01-31T20:57:49.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.119711+0000) 2022-01-31T20:57:49.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[46715]: debug 2022-01-31T20:57:49.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.397486+0000) 2022-01-31T20:57:49.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[51620]: debug 2022-01-31T20:57:49.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.474772+0000) 2022-01-31T20:57:49.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:49 smithi167 conmon[49112]: debug 2022-01-31T20:57:49.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.441080+0000) 2022-01-31T20:57:49.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[35325]: debug 2022-01-31T20:57:49.802+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:49.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[46715]: debug 2022-01-31T20:57:49.792+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.793252+0000) 2022-01-31T20:57:49.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[51620]: debug 2022-01-31T20:57:49.792+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.793394+0000) 2022-01-31T20:57:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[41853]: debug 2022-01-31T20:57:49.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.602879+0000) 2022-01-31T20:57:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:49 smithi171 conmon[41853]: debug 2022-01-31T20:57:49.792+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.793737+0000) 2022-01-31T20:57:50.118 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:49 smithi167 conmon[54076]: debug 2022-01-31T20:57:49.794+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.794956+0000) 2022-01-31T20:57:50.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:49 smithi167 conmon[49112]: debug 2022-01-31T20:57:49.794+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.795287+0000) 2022-01-31T20:57:50.119 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:49 smithi167 conmon[60316]: debug 2022-01-31T20:57:49.794+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:49.794894+0000) 2022-01-31T20:57:50.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:50 smithi167 conmon[54076]: debug 2022-01-31T20:57:50.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:50.368673+0000) 2022-01-31T20:57:50.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:50 smithi167 conmon[60316]: debug 2022-01-31T20:57:50.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:50.119872+0000) 2022-01-31T20:57:50.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:50 smithi171 conmon[46715]: debug 2022-01-31T20:57:50.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:50.397661+0000) 2022-01-31T20:57:50.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:50 smithi171 conmon[51620]: debug 2022-01-31T20:57:50.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:50.474931+0000) 2022-01-31T20:57:50.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:50 smithi167 conmon[49112]: debug 2022-01-31T20:57:50.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:50.441285+0000) 2022-01-31T20:57:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:50 smithi171 conmon[41853]: debug 2022-01-31T20:57:50.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:50.603073+0000) 2022-01-31T20:57:51.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:51 smithi167 conmon[54076]: debug 2022-01-31T20:57:51.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:51.368824+0000) 2022-01-31T20:57:51.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:51 smithi167 conmon[60316]: debug 2022-01-31T20:57:51.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:51.119982+0000) 2022-01-31T20:57:51.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:51 smithi171 conmon[46715]: debug 2022-01-31T20:57:51.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:51.397800+0000) 2022-01-31T20:57:51.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:51 smithi171 conmon[51620]: debug 2022-01-31T20:57:51.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:51.475040+0000) 2022-01-31T20:57:51.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:51 smithi167 conmon[49112]: debug 2022-01-31T20:57:51.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:51.441438+0000) 2022-01-31T20:57:51.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:51 smithi171 conmon[41853]: debug 2022-01-31T20:57:51.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:51.603246+0000) 2022-01-31T20:57:52.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:52 smithi167 conmon[54076]: debug 2022-01-31T20:57:52.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:52.368987+0000) 2022-01-31T20:57:52.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:52 smithi167 conmon[60316]: debug 2022-01-31T20:57:52.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:52.120146+0000) 2022-01-31T20:57:52.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:52 smithi171 conmon[46715]: debug 2022-01-31T20:57:52.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:52.397976+0000) 2022-01-31T20:57:52.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:52 smithi171 conmon[51620]: debug 2022-01-31T20:57:52.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:52.475198+0000) 2022-01-31T20:57:52.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:52 smithi167 conmon[49112]: debug 2022-01-31T20:57:52.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:52.441574+0000) 2022-01-31T20:57:52.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:52 smithi171 conmon[41853]: debug 2022-01-31T20:57:52.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:52.603474+0000) 2022-01-31T20:57:53.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:53 smithi167 conmon[54076]: debug 2022-01-31T20:57:53.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:53.369156+0000) 2022-01-31T20:57:53.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:53 smithi167 conmon[60316]: debug 2022-01-31T20:57:53.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:53.120344+0000) 2022-01-31T20:57:53.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:53 smithi171 conmon[46715]: debug 2022-01-31T20:57:53.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:53.398168+0000) 2022-01-31T20:57:53.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:53 smithi171 conmon[51620]: debug 2022-01-31T20:57:53.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:53.475391+0000) 2022-01-31T20:57:53.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:53 smithi167 conmon[49112]: debug 2022-01-31T20:57:53.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:53.441729+0000) 2022-01-31T20:57:53.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:53 smithi171 conmon[35325]: debug 2022-01-31T20:57:53.600+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:57:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:53 smithi171 conmon[41853]: debug 2022-01-31T20:57:53.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:53.603580+0000) 2022-01-31T20:57:54.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:54 smithi167 conmon[54076]: debug 2022-01-31T20:57:54.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.369344+0000) 2022-01-31T20:57:54.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:54 smithi167 conmon[60316]: debug 2022-01-31T20:57:54.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.120445+0000) 2022-01-31T20:57:54.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[46715]: debug 2022-01-31T20:57:54.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.398298+0000) 2022-01-31T20:57:54.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[51620]: debug 2022-01-31T20:57:54.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.475591+0000) 2022-01-31T20:57:54.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:54 smithi167 conmon[49112]: debug 2022-01-31T20:57:54.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.441947+0000) 2022-01-31T20:57:54.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[35325]: debug 2022-01-31T20:57:54.813+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92271 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:54.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[46715]: debug 2022-01-31T20:57:54.804+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.805104+0000) 2022-01-31T20:57:54.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[51620]: debug 2022-01-31T20:57:54.804+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.805002+0000) 2022-01-31T20:57:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[41853]: debug 2022-01-31T20:57:54.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.603714+0000) 2022-01-31T20:57:54.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:54 smithi171 conmon[41853]: debug 2022-01-31T20:57:54.804+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.804815+0000) 2022-01-31T20:57:55.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:54 smithi167 conmon[49112]: debug 2022-01-31T20:57:54.804+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.806304+0000) 2022-01-31T20:57:55.120 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:54 smithi167 conmon[54076]: debug 2022-01-31T20:57:54.804+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.806383+0000) 2022-01-31T20:57:55.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:54 smithi167 conmon[60316]: debug 2022-01-31T20:57:54.804+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:54.805819+0000) 2022-01-31T20:57:55.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:55 smithi167 conmon[54076]: debug 2022-01-31T20:57:55.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:55.369574+0000) 2022-01-31T20:57:55.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:55 smithi167 conmon[60316]: debug 2022-01-31T20:57:55.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:55.120619+0000) 2022-01-31T20:57:55.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:55 smithi171 conmon[46715]: debug 2022-01-31T20:57:55.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:55.398452+0000) 2022-01-31T20:57:55.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:55 smithi171 conmon[51620]: debug 2022-01-31T20:57:55.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:55.475756+0000) 2022-01-31T20:57:55.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:55 smithi167 conmon[49112]: debug 2022-01-31T20:57:55.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:55.442106+0000) 2022-01-31T20:57:55.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:55 smithi171 conmon[41853]: debug 2022-01-31T20:57:55.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:55.603870+0000) 2022-01-31T20:57:56.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:56 smithi167 conmon[60316]: debug 2022-01-31T20:57:56.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:56.120777+0000) 2022-01-31T20:57:56.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:56 smithi167 conmon[54076]: debug 2022-01-31T20:57:56.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:56.369737+0000) 2022-01-31T20:57:56.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:56 smithi171 conmon[46715]: debug 2022-01-31T20:57:56.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:56.398594+0000) 2022-01-31T20:57:56.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:56 smithi171 conmon[51620]: debug 2022-01-31T20:57:56.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:56.475893+0000) 2022-01-31T20:57:56.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:56 smithi167 conmon[49112]: debug 2022-01-31T20:57:56.440+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:56.442310+0000) 2022-01-31T20:57:56.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:56 smithi171 conmon[41853]: debug 2022-01-31T20:57:56.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:56.603981+0000) 2022-01-31T20:57:57.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:57 smithi167 conmon[54076]: debug 2022-01-31T20:57:57.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:57.369944+0000) 2022-01-31T20:57:57.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:57 smithi167 conmon[60316]: debug 2022-01-31T20:57:57.119+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:57.120891+0000) 2022-01-31T20:57:57.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:57 smithi171 conmon[46715]: debug 2022-01-31T20:57:57.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:57.398779+0000) 2022-01-31T20:57:57.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:57 smithi171 conmon[51620]: debug 2022-01-31T20:57:57.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:57.476033+0000) 2022-01-31T20:57:57.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:57 smithi167 conmon[49112]: debug 2022-01-31T20:57:57.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:57.442527+0000) 2022-01-31T20:57:57.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:57 smithi171 conmon[41853]: debug 2022-01-31T20:57:57.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:57.604178+0000) 2022-01-31T20:57:58.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:58 smithi167 conmon[54076]: debug 2022-01-31T20:57:58.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:58.370127+0000) 2022-01-31T20:57:58.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:58 smithi167 conmon[60316]: debug 2022-01-31T20:57:58.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:58.121105+0000) 2022-01-31T20:57:58.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:58 smithi171 conmon[46715]: debug 2022-01-31T20:57:58.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:58.398963+0000) 2022-01-31T20:57:58.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:58 smithi171 conmon[51620]: debug 2022-01-31T20:57:58.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:58.476210+0000) 2022-01-31T20:57:58.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:58 smithi167 conmon[49112]: debug 2022-01-31T20:57:58.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:58.442700+0000) 2022-01-31T20:57:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:58 smithi171 conmon[41853]: debug 2022-01-31T20:57:58.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:58.604303+0000) 2022-01-31T20:57:59.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:59 smithi167 conmon[54076]: debug 2022-01-31T20:57:59.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.370266+0000) 2022-01-31T20:57:59.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:59 smithi167 conmon[60316]: debug 2022-01-31T20:57:59.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.121319+0000) 2022-01-31T20:57:59.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[46715]: debug 2022-01-31T20:57:59.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.399146+0000) 2022-01-31T20:57:59.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[51620]: debug 2022-01-31T20:57:59.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.476416+0000) 2022-01-31T20:57:59.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:59 smithi167 conmon[49112]: debug 2022-01-31T20:57:59.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.442867+0000) 2022-01-31T20:57:59.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[35325]: debug 2022-01-31T20:57:59.825+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92381 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:57:59.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[46715]: debug 2022-01-31T20:57:59.816+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.816922+0000) 2022-01-31T20:57:59.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[51620]: debug 2022-01-31T20:57:59.815+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.816441+0000) 2022-01-31T20:57:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[41853]: debug 2022-01-31T20:57:59.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.604459+0000) 2022-01-31T20:57:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:57:59 smithi171 conmon[41853]: debug 2022-01-31T20:57:59.816+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.817253+0000) 2022-01-31T20:58:00.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:57:59 smithi167 conmon[49112]: debug 2022-01-31T20:57:59.816+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.818020+0000) 2022-01-31T20:58:00.120 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:57:59 smithi167 conmon[54076]: debug 2022-01-31T20:57:59.816+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.818292+0000) 2022-01-31T20:58:00.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:57:59 smithi167 conmon[60316]: debug 2022-01-31T20:57:59.815+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:57:59.817395+0000) 2022-01-31T20:58:00.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:00 smithi167 conmon[60316]: debug 2022-01-31T20:58:00.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:00.121555+0000) 2022-01-31T20:58:00.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:00 smithi167 conmon[54076]: debug 2022-01-31T20:58:00.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:00.370367+0000) 2022-01-31T20:58:00.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:00 smithi171 conmon[46715]: debug 2022-01-31T20:58:00.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:00.399348+0000) 2022-01-31T20:58:00.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:00 smithi171 conmon[51620]: debug 2022-01-31T20:58:00.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:00.476597+0000) 2022-01-31T20:58:00.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:00 smithi167 conmon[49112]: debug 2022-01-31T20:58:00.441+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:00.443068+0000) 2022-01-31T20:58:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:00 smithi171 conmon[41853]: debug 2022-01-31T20:58:00.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:00.604606+0000) 2022-01-31T20:58:01.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:01 smithi167 conmon[54076]: debug 2022-01-31T20:58:01.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:01.370524+0000) 2022-01-31T20:58:01.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:01 smithi167 conmon[60316]: debug 2022-01-31T20:58:01.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:01.121708+0000) 2022-01-31T20:58:01.475 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:01 smithi171 conmon[46715]: debug 2022-01-31T20:58:01.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:01.399494+0000) 2022-01-31T20:58:01.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:01 smithi167 conmon[49112]: debug 2022-01-31T20:58:01.442+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:01.443210+0000) 2022-01-31T20:58:01.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:01 smithi171 conmon[51620]: debug 2022-01-31T20:58:01.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:01.476737+0000) 2022-01-31T20:58:01.728 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:01 smithi171 conmon[41853]: debug 2022-01-31T20:58:01.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:01.604709+0000) 2022-01-31T20:58:02.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:02 smithi167 conmon[54076]: debug 2022-01-31T20:58:02.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:02.370711+0000) 2022-01-31T20:58:02.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:02 smithi167 conmon[60316]: debug 2022-01-31T20:58:02.121+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:02.121896+0000) 2022-01-31T20:58:02.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:02 smithi171 conmon[46715]: debug 2022-01-31T20:58:02.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:02.399644+0000) 2022-01-31T20:58:02.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:02 smithi171 conmon[51620]: debug 2022-01-31T20:58:02.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:02.476891+0000) 2022-01-31T20:58:02.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:02 smithi167 conmon[49112]: debug 2022-01-31T20:58:02.442+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:02.443376+0000) 2022-01-31T20:58:02.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:02 smithi171 conmon[41853]: debug 2022-01-31T20:58:02.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:02.604914+0000) 2022-01-31T20:58:03.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:03 smithi167 conmon[54076]: debug 2022-01-31T20:58:03.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:03.370871+0000) 2022-01-31T20:58:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:03 smithi167 conmon[60316]: debug 2022-01-31T20:58:03.120+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:03.122042+0000) 2022-01-31T20:58:03.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:03 smithi171 conmon[46715]: debug 2022-01-31T20:58:03.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:03.399862+0000) 2022-01-31T20:58:03.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:03 smithi171 conmon[51620]: debug 2022-01-31T20:58:03.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:03.477040+0000) 2022-01-31T20:58:03.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:03 smithi167 conmon[49112]: debug 2022-01-31T20:58:03.442+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:03.443582+0000) 2022-01-31T20:58:03.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:03 smithi171 conmon[41853]: debug 2022-01-31T20:58:03.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:03.605071+0000) 2022-01-31T20:58:04.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:04 smithi167 conmon[54076]: debug 2022-01-31T20:58:04.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.371056+0000) 2022-01-31T20:58:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:04 smithi167 conmon[60316]: debug 2022-01-31T20:58:04.121+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.122224+0000) 2022-01-31T20:58:04.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[46715]: debug 2022-01-31T20:58:04.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.400048+0000) 2022-01-31T20:58:04.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[51620]: debug 2022-01-31T20:58:04.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.477233+0000) 2022-01-31T20:58:04.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:04 smithi167 conmon[49112]: debug 2022-01-31T20:58:04.442+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.443733+0000) 2022-01-31T20:58:04.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[35325]: debug 2022-01-31T20:58:04.836+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92491 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:04.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[46715]: debug 2022-01-31T20:58:04.827+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.829659+0000) 2022-01-31T20:58:04.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[51620]: debug 2022-01-31T20:58:04.826+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.828582+0000) 2022-01-31T20:58:04.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[41853]: debug 2022-01-31T20:58:04.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.605278+0000) 2022-01-31T20:58:04.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:04 smithi171 conmon[41853]: debug 2022-01-31T20:58:04.828+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.830207+0000) 2022-01-31T20:58:05.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:04 smithi167 conmon[49112]: debug 2022-01-31T20:58:04.827+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.829423+0000) 2022-01-31T20:58:05.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:04 smithi167 conmon[54076]: debug 2022-01-31T20:58:04.828+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.829704+0000) 2022-01-31T20:58:05.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:04 smithi167 conmon[60316]: debug 2022-01-31T20:58:04.828+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:04.830306+0000) 2022-01-31T20:58:05.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:05 smithi167 conmon[54076]: debug 2022-01-31T20:58:05.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:05.371237+0000) 2022-01-31T20:58:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:05 smithi167 conmon[60316]: debug 2022-01-31T20:58:05.121+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:05.122424+0000) 2022-01-31T20:58:05.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:05 smithi171 conmon[46715]: debug 2022-01-31T20:58:05.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:05.400257+0000) 2022-01-31T20:58:05.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:05 smithi171 conmon[51620]: debug 2022-01-31T20:58:05.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:05.477426+0000) 2022-01-31T20:58:05.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:05 smithi167 conmon[49112]: debug 2022-01-31T20:58:05.443+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:05.443936+0000) 2022-01-31T20:58:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:05 smithi171 conmon[41853]: debug 2022-01-31T20:58:05.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:05.605461+0000) 2022-01-31T20:58:06.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:06 smithi167 conmon[54076]: debug 2022-01-31T20:58:06.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:06.371344+0000) 2022-01-31T20:58:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:06 smithi167 conmon[60316]: debug 2022-01-31T20:58:06.121+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:06.122609+0000) 2022-01-31T20:58:06.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:06 smithi171 conmon[46715]: debug 2022-01-31T20:58:06.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:06.400384+0000) 2022-01-31T20:58:06.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:06 smithi171 conmon[51620]: debug 2022-01-31T20:58:06.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:06.477565+0000) 2022-01-31T20:58:06.688 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:06 smithi167 conmon[49112]: debug 2022-01-31T20:58:06.443+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:06.444103+0000) 2022-01-31T20:58:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:06 smithi171 conmon[41853]: debug 2022-01-31T20:58:06.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:06.605569+0000) 2022-01-31T20:58:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:07 smithi167 conmon[54076]: debug 2022-01-31T20:58:07.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:07.371548+0000) 2022-01-31T20:58:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:07 smithi167 conmon[60316]: debug 2022-01-31T20:58:07.121+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:07.122756+0000) 2022-01-31T20:58:07.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:07 smithi171 conmon[46715]: debug 2022-01-31T20:58:07.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:07.400542+0000) 2022-01-31T20:58:07.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:07 smithi171 conmon[51620]: debug 2022-01-31T20:58:07.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:07.477793+0000) 2022-01-31T20:58:07.688 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:07 smithi167 conmon[49112]: debug 2022-01-31T20:58:07.442+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:07.444288+0000) 2022-01-31T20:58:07.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:07 smithi171 conmon[41853]: debug 2022-01-31T20:58:07.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:07.605765+0000) 2022-01-31T20:58:08.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:08 smithi167 conmon[54076]: debug 2022-01-31T20:58:08.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:08.371747+0000) 2022-01-31T20:58:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:08 smithi167 conmon[60316]: debug 2022-01-31T20:58:08.122+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:08.122960+0000) 2022-01-31T20:58:08.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:08 smithi171 conmon[46715]: debug 2022-01-31T20:58:08.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:08.400713+0000) 2022-01-31T20:58:08.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:08 smithi171 conmon[51620]: debug 2022-01-31T20:58:08.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:08.478006+0000) 2022-01-31T20:58:08.688 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:08 smithi167 conmon[49112]: debug 2022-01-31T20:58:08.443+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:08.444474+0000) 2022-01-31T20:58:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:08 smithi171 conmon[35325]: debug 2022-01-31T20:58:08.600+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:58:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:08 smithi171 conmon[41853]: debug 2022-01-31T20:58:08.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:08.605879+0000) 2022-01-31T20:58:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:09 smithi167 conmon[60316]: debug 2022-01-31T20:58:09.122+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.123149+0000) 2022-01-31T20:58:09.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:09 smithi167 conmon[54076]: debug 2022-01-31T20:58:09.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.371946+0000) 2022-01-31T20:58:09.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[46715]: debug 2022-01-31T20:58:09.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.400867+0000) 2022-01-31T20:58:09.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[51620]: debug 2022-01-31T20:58:09.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.478212+0000) 2022-01-31T20:58:09.688 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:09 smithi167 conmon[49112]: debug 2022-01-31T20:58:09.443+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.444654+0000) 2022-01-31T20:58:09.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[46715]: debug 2022-01-31T20:58:09.839+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.840918+0000) 2022-01-31T20:58:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[41853]: debug 2022-01-31T20:58:09.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.606105+0000) 2022-01-31T20:58:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[41853]: debug 2022-01-31T20:58:09.840+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.842198+0000) 2022-01-31T20:58:09.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[51620]: debug 2022-01-31T20:58:09.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[51620]: 2022-01-31T20:58:09.840+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.841824+0000) 2022-01-31T20:58:09.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:09 smithi171 conmon[35325]: debug 2022-01-31T20:58:09.849+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92602 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:10.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:09 smithi167 conmon[49112]: debug 2022-01-31T20:58:09.840+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.841609+0000) 2022-01-31T20:58:10.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:09 smithi167 conmon[54076]: debug 2022-01-31T20:58:09.839+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.840817+0000) 2022-01-31T20:58:10.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:09 smithi167 conmon[60316]: debug 2022-01-31T20:58:09.840+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:09.841717+0000) 2022-01-31T20:58:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:10 smithi167 conmon[54076]: debug 2022-01-31T20:58:10.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:10.372148+0000) 2022-01-31T20:58:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:10 smithi167 conmon[60316]: debug 2022-01-31T20:58:10.122+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:10.123360+0000) 2022-01-31T20:58:10.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:10 smithi171 conmon[46715]: debug 2022-01-31T20:58:10.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:10.401021+0000) 2022-01-31T20:58:10.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:10 smithi171 conmon[51620]: debug 2022-01-31T20:58:10.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:10.478361+0000) 2022-01-31T20:58:10.688 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:10 smithi167 conmon[49112]: debug 2022-01-31T20:58:10.444+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:10.444840+0000) 2022-01-31T20:58:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:10 smithi171 conmon[41853]: debug 2022-01-31T20:58:10.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:10.606232+0000) 2022-01-31T20:58:11.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:11 smithi167 conmon[54076]: debug 2022-01-31T20:58:11.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:11.372297+0000) 2022-01-31T20:58:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:11 smithi167 conmon[60316]: debug 2022-01-31T20:58:11.122+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:11.123546+0000) 2022-01-31T20:58:11.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:11 smithi171 conmon[51620]: debug 2022-01-31T20:58:11.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:11.478509+0000) 2022-01-31T20:58:11.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:11 smithi171 conmon[46715]: debug 2022-01-31T20:58:11.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:11.401139+0000) 2022-01-31T20:58:11.688 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:11 smithi167 conmon[49112]: debug 2022-01-31T20:58:11.444+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:11.444969+0000) 2022-01-31T20:58:11.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:11 smithi171 conmon[41853]: debug 2022-01-31T20:58:11.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:11.606351+0000) 2022-01-31T20:58:12.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:12 smithi167 conmon[54076]: debug 2022-01-31T20:58:12.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:12.372473+0000) 2022-01-31T20:58:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:12 smithi167 conmon[60316]: debug 2022-01-31T20:58:12.123+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:12.123731+0000) 2022-01-31T20:58:12.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:12 smithi171 conmon[46715]: debug 2022-01-31T20:58:12.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:12.401344+0000) 2022-01-31T20:58:12.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:12 smithi171 conmon[51620]: debug 2022-01-31T20:58:12.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:12.478699+0000) 2022-01-31T20:58:12.689 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:12 smithi167 conmon[49112]: debug 2022-01-31T20:58:12.443+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:12.445178+0000) 2022-01-31T20:58:12.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:12 smithi171 conmon[41853]: debug 2022-01-31T20:58:12.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:12.606520+0000) 2022-01-31T20:58:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:13 smithi167 conmon[54076]: debug 2022-01-31T20:58:13.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:13.372674+0000) 2022-01-31T20:58:13.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:13 smithi167 conmon[60316]: debug 2022-01-31T20:58:13.123+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:13.123933+0000) 2022-01-31T20:58:13.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:13 smithi171 conmon[46715]: debug 2022-01-31T20:58:13.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:13.401531+0000) 2022-01-31T20:58:13.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:13 smithi171 conmon[51620]: debug 2022-01-31T20:58:13.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:13.478840+0000) 2022-01-31T20:58:13.689 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:13 smithi167 conmon[49112]: debug 2022-01-31T20:58:13.444+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:13.445336+0000) 2022-01-31T20:58:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:13 smithi171 conmon[41853]: debug 2022-01-31T20:58:13.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:13.606673+0000) 2022-01-31T20:58:14.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:14 smithi167 conmon[54076]: debug 2022-01-31T20:58:14.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.372901+0000) 2022-01-31T20:58:14.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:14 smithi167 conmon[60316]: debug 2022-01-31T20:58:14.123+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.124097+0000) 2022-01-31T20:58:14.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[46715]: debug 2022-01-31T20:58:14.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.401717+0000) 2022-01-31T20:58:14.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[51620]: debug 2022-01-31T20:58:14.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.479043+0000) 2022-01-31T20:58:14.689 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:14 smithi167 conmon[49112]: debug 2022-01-31T20:58:14.444+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.445543+0000) 2022-01-31T20:58:14.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[46715]: debug 2022-01-31T20:58:14.852+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.853785+0000) 2022-01-31T20:58:14.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[51620]: debug 2022-01-31T20:58:14.852+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.854714+0000) 2022-01-31T20:58:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[41853]: debug 2022-01-31T20:58:14.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.606853+0000) 2022-01-31T20:58:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[41853]: debug 2022-01-31T20:58:14.852+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.854449+0000) 2022-01-31T20:58:15.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:14 smithi167 conmon[49112]: debug 2022-01-31T20:58:14.853+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.855144+0000) 2022-01-31T20:58:15.123 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:14 smithi167 conmon[54076]: debug 2022-01-31T20:58:14.852+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.853663+0000) 2022-01-31T20:58:15.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:14 smithi167 conmon[60316]: debug 2022-01-31T20:58:14.852+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:14.853952+0000) 2022-01-31T20:58:15.247 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:14 smithi171 conmon[35325]: debug 2022-01-31T20:58:14.861+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92711 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:15.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:15 smithi167 conmon[54076]: debug 2022-01-31T20:58:15.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:15.373086+0000) 2022-01-31T20:58:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:15 smithi167 conmon[60316]: debug 2022-01-31T20:58:15.123+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:15.124289+0000) 2022-01-31T20:58:15.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:15 smithi171 conmon[46715]: debug 2022-01-31T20:58:15.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:15.401925+0000) 2022-01-31T20:58:15.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:15 smithi171 conmon[51620]: debug 2022-01-31T20:58:15.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:15.479206+0000) 2022-01-31T20:58:15.689 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:15 smithi167 conmon[49112]: debug 2022-01-31T20:58:15.445+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:15.445741+0000) 2022-01-31T20:58:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:15 smithi171 conmon[41853]: debug 2022-01-31T20:58:15.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:15.607020+0000) 2022-01-31T20:58:16.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:16 smithi167 conmon[54076]: debug 2022-01-31T20:58:16.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:16.373219+0000) 2022-01-31T20:58:16.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:16 smithi167 conmon[60316]: debug 2022-01-31T20:58:16.123+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:16.124443+0000) 2022-01-31T20:58:16.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:16 smithi171 conmon[46715]: debug 2022-01-31T20:58:16.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:16.402083+0000) 2022-01-31T20:58:16.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:16 smithi171 conmon[51620]: debug 2022-01-31T20:58:16.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:16.479396+0000) 2022-01-31T20:58:16.689 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:16 smithi167 conmon[49112]: debug 2022-01-31T20:58:16.445+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:16.445838+0000) 2022-01-31T20:58:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:16 smithi171 conmon[41853]: debug 2022-01-31T20:58:16.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:16.607189+0000) 2022-01-31T20:58:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:17 smithi167 conmon[54076]: debug 2022-01-31T20:58:17.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:17.373402+0000) 2022-01-31T20:58:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:17 smithi167 conmon[60316]: debug 2022-01-31T20:58:17.124+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:17.124546+0000) 2022-01-31T20:58:17.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:17 smithi171 conmon[46715]: debug 2022-01-31T20:58:17.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:17.402275+0000) 2022-01-31T20:58:17.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:17 smithi171 conmon[51620]: debug 2022-01-31T20:58:17.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:17.479590+0000) 2022-01-31T20:58:17.690 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:17 smithi167 conmon[49112]: debug 2022-01-31T20:58:17.445+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:17.446046+0000) 2022-01-31T20:58:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:17 smithi171 conmon[41853]: debug 2022-01-31T20:58:17.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:17.607353+0000) 2022-01-31T20:58:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:18 smithi167 conmon[54076]: debug 2022-01-31T20:58:18.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:18.373560+0000) 2022-01-31T20:58:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:18 smithi167 conmon[60316]: debug 2022-01-31T20:58:18.124+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:18.124736+0000) 2022-01-31T20:58:18.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:18 smithi171 conmon[46715]: debug 2022-01-31T20:58:18.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:18.402467+0000) 2022-01-31T20:58:18.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:18 smithi171 conmon[51620]: debug 2022-01-31T20:58:18.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:18.479789+0000) 2022-01-31T20:58:18.690 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:18 smithi167 conmon[49112]: debug 2022-01-31T20:58:18.445+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:18.446226+0000) 2022-01-31T20:58:19.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:18 smithi171 conmon[41853]: debug 2022-01-31T20:58:18.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:18.607535+0000) 2022-01-31T20:58:19.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:19 smithi167 conmon[54076]: debug 2022-01-31T20:58:19.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.373715+0000) 2022-01-31T20:58:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:19 smithi167 conmon[60316]: debug 2022-01-31T20:58:19.124+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.124956+0000) 2022-01-31T20:58:19.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[46715]: debug 2022-01-31T20:58:19.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.402654+0000) 2022-01-31T20:58:19.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[51620]: debug 2022-01-31T20:58:19.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.479921+0000) 2022-01-31T20:58:19.690 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:19 smithi167 conmon[49112]: debug 2022-01-31T20:58:19.445+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.446389+0000) 2022-01-31T20:58:19.863 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[41853]: debug 2022-01-31T20:58:19.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.607725+0000) 2022-01-31T20:58:20.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:19 smithi167 conmon[49112]: debug 2022-01-31T20:58:19.865+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.866345+0000) 2022-01-31T20:58:20.124 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:19 smithi167 conmon[54076]: debug 2022-01-31T20:58:19.865+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.865818+0000) 2022-01-31T20:58:20.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:19 smithi167 conmon[60316]: debug 2022-01-31T20:58:19.866+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.866587+0000) 2022-01-31T20:58:20.248 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[35325]: debug 2022-01-31T20:58:19.874+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92828 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:20.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[46715]: debug 2022-01-31T20:58:19.864+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.865594+0000) 2022-01-31T20:58:20.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[51620]: debug 2022-01-31T20:58:19.865+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.865930+0000) 2022-01-31T20:58:20.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:19 smithi171 conmon[41853]: debug 2022-01-31T20:58:19.865+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:19.866698+0000) 2022-01-31T20:58:20.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:20 smithi167 conmon[54076]: debug 2022-01-31T20:58:20.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:20.373908+0000) 2022-01-31T20:58:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:20 smithi167 conmon[60316]: debug 2022-01-31T20:58:20.124+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:20.125183+0000) 2022-01-31T20:58:20.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:20 smithi171 conmon[51620]: debug 2022-01-31T20:58:20.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:20.480081+0000) 2022-01-31T20:58:20.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:20 smithi171 conmon[46715]: debug 2022-01-31T20:58:20.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:20.402838+0000) 2022-01-31T20:58:20.690 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:20 smithi167 conmon[49112]: debug 2022-01-31T20:58:20.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:20.446523+0000) 2022-01-31T20:58:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:20 smithi171 conmon[41853]: debug 2022-01-31T20:58:20.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:20.607860+0000) 2022-01-31T20:58:21.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:21 smithi167 conmon[54076]: debug 2022-01-31T20:58:21.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:21.374060+0000) 2022-01-31T20:58:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:21 smithi167 conmon[60316]: debug 2022-01-31T20:58:21.124+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:21.125345+0000) 2022-01-31T20:58:21.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:21 smithi171 conmon[51620]: debug 2022-01-31T20:58:21.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:21.480250+0000) 2022-01-31T20:58:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:21 smithi171 conmon[46715]: debug 2022-01-31T20:58:21.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:21.402960+0000) 2022-01-31T20:58:21.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:21 smithi167 conmon[49112]: debug 2022-01-31T20:58:21.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:21.446609+0000) 2022-01-31T20:58:22.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:21 smithi171 conmon[41853]: debug 2022-01-31T20:58:21.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:21.607986+0000) 2022-01-31T20:58:22.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:22 smithi167 conmon[54076]: debug 2022-01-31T20:58:22.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:22.374264+0000) 2022-01-31T20:58:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:22 smithi167 conmon[60316]: debug 2022-01-31T20:58:22.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:22.125513+0000) 2022-01-31T20:58:22.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:22 smithi171 conmon[51620]: debug 2022-01-31T20:58:22.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:22.480436+0000) 2022-01-31T20:58:22.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:22 smithi171 conmon[46715]: debug 2022-01-31T20:58:22.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:22.403089+0000) 2022-01-31T20:58:22.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:22 smithi167 conmon[49112]: debug 2022-01-31T20:58:22.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:22.446715+0000) 2022-01-31T20:58:23.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:22 smithi171 conmon[41853]: debug 2022-01-31T20:58:22.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:22.608098+0000) 2022-01-31T20:58:23.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:23 smithi167 conmon[54076]: debug 2022-01-31T20:58:23.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:23.374448+0000) 2022-01-31T20:58:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:23 smithi167 conmon[60316]: debug 2022-01-31T20:58:23.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:23.125666+0000) 2022-01-31T20:58:23.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:23 smithi171 conmon[46715]: debug 2022-01-31T20:58:23.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:23.403276+0000) 2022-01-31T20:58:23.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:23 smithi171 conmon[51620]: debug 2022-01-31T20:58:23.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:23.480652+0000) 2022-01-31T20:58:23.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:23 smithi167 conmon[49112]: debug 2022-01-31T20:58:23.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:23.446853+0000) 2022-01-31T20:58:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:23 smithi171 conmon[35325]: debug 2022-01-31T20:58:23.601+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:58:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:23 smithi171 conmon[41853]: debug 2022-01-31T20:58:23.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:23.608228+0000) 2022-01-31T20:58:24.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:24 smithi167 conmon[60316]: debug 2022-01-31T20:58:24.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.125827+0000) 2022-01-31T20:58:24.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:24 smithi167 conmon[54076]: debug 2022-01-31T20:58:24.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.374628+0000) 2022-01-31T20:58:24.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[46715]: debug 2022-01-31T20:58:24.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.403435+0000) 2022-01-31T20:58:24.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[51620]: debug 2022-01-31T20:58:24.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.480831+0000) 2022-01-31T20:58:24.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:24 smithi167 conmon[49112]: debug 2022-01-31T20:58:24.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.447063+0000) 2022-01-31T20:58:24.875 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[41853]: debug 2022-01-31T20:58:24.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.608343+0000) 2022-01-31T20:58:25.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:24 smithi167 conmon[49112]: debug 2022-01-31T20:58:24.877+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.878201+0000) 2022-01-31T20:58:25.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:24 smithi167 conmon[54076]: debug 2022-01-31T20:58:24.878+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.878857+0000) 2022-01-31T20:58:25.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:24 smithi167 conmon[60316]: debug 2022-01-31T20:58:24.877+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.877979+0000) 2022-01-31T20:58:25.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[35325]: debug 2022-01-31T20:58:24.886+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 92938 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:25.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[41853]: debug 2022-01-31T20:58:24.877+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.877935+0000) 2022-01-31T20:58:25.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[46715]: debug 2022-01-31T20:58:24.876+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.877224+0000) 2022-01-31T20:58:25.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:24 smithi171 conmon[51620]: debug 2022-01-31T20:58:24.877+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:24.877761+0000) 2022-01-31T20:58:25.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:25 smithi167 conmon[54076]: debug 2022-01-31T20:58:25.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:25.374842+0000) 2022-01-31T20:58:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:25 smithi167 conmon[60316]: debug 2022-01-31T20:58:25.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:25.126018+0000) 2022-01-31T20:58:25.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:25 smithi171 conmon[46715]: debug 2022-01-31T20:58:25.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:25.403619+0000) 2022-01-31T20:58:25.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:25 smithi171 conmon[51620]: debug 2022-01-31T20:58:25.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:25.480973+0000) 2022-01-31T20:58:25.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:25 smithi167 conmon[49112]: debug 2022-01-31T20:58:25.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:25.447244+0000) 2022-01-31T20:58:26.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:25 smithi171 conmon[41853]: debug 2022-01-31T20:58:25.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:25.608513+0000) 2022-01-31T20:58:26.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:26 smithi167 conmon[54076]: debug 2022-01-31T20:58:26.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:26.375026+0000) 2022-01-31T20:58:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:26 smithi167 conmon[60316]: debug 2022-01-31T20:58:26.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:26.126137+0000) 2022-01-31T20:58:26.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:26 smithi171 conmon[46715]: debug 2022-01-31T20:58:26.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:26.403785+0000) 2022-01-31T20:58:26.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:26 smithi171 conmon[51620]: debug 2022-01-31T20:58:26.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:26.481117+0000) 2022-01-31T20:58:26.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:26 smithi167 conmon[49112]: debug 2022-01-31T20:58:26.446+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:26.447385+0000) 2022-01-31T20:58:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:26 smithi171 conmon[41853]: debug 2022-01-31T20:58:26.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:26.608629+0000) 2022-01-31T20:58:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:27 smithi167 conmon[60316]: debug 2022-01-31T20:58:27.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:27.126266+0000) 2022-01-31T20:58:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:27 smithi167 conmon[54076]: debug 2022-01-31T20:58:27.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:27.375206+0000) 2022-01-31T20:58:27.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:27 smithi171 conmon[46715]: debug 2022-01-31T20:58:27.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:27.403974+0000) 2022-01-31T20:58:27.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:27 smithi171 conmon[51620]: debug 2022-01-31T20:58:27.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:27.481275+0000) 2022-01-31T20:58:27.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:27 smithi167 conmon[49112]: debug 2022-01-31T20:58:27.447+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:27.447580+0000) 2022-01-31T20:58:28.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:27 smithi171 conmon[41853]: debug 2022-01-31T20:58:27.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:27.608748+0000) 2022-01-31T20:58:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:28 smithi167 conmon[60316]: debug 2022-01-31T20:58:28.125+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:28.126415+0000) 2022-01-31T20:58:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:28 smithi167 conmon[54076]: debug 2022-01-31T20:58:28.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:28.375401+0000) 2022-01-31T20:58:28.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:28 smithi171 conmon[46715]: debug 2022-01-31T20:58:28.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:28.404191+0000) 2022-01-31T20:58:28.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:28 smithi171 conmon[51620]: debug 2022-01-31T20:58:28.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:28.481472+0000) 2022-01-31T20:58:28.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:28 smithi167 conmon[49112]: debug 2022-01-31T20:58:28.447+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:28.447763+0000) 2022-01-31T20:58:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:28 smithi171 conmon[41853]: debug 2022-01-31T20:58:28.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:28.608899+0000) 2022-01-31T20:58:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:29 smithi167 conmon[60316]: debug 2022-01-31T20:58:29.126+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.126576+0000) 2022-01-31T20:58:29.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:29 smithi167 conmon[54076]: debug 2022-01-31T20:58:29.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.375569+0000) 2022-01-31T20:58:29.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[46715]: debug 2022-01-31T20:58:29.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.404401+0000) 2022-01-31T20:58:29.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[51620]: debug 2022-01-31T20:58:29.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.481655+0000) 2022-01-31T20:58:29.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:29 smithi167 conmon[49112]: debug 2022-01-31T20:58:29.447+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.447903+0000) 2022-01-31T20:58:29.887 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[41853]: debug 2022-01-31T20:58:29.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.609057+0000) 2022-01-31T20:58:30.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:29 smithi167 conmon[49112]: debug 2022-01-31T20:58:29.889+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.890331+0000) 2022-01-31T20:58:30.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:29 smithi167 conmon[54076]: debug 2022-01-31T20:58:29.890+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.890982+0000) 2022-01-31T20:58:30.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:29 smithi167 conmon[60316]: debug 2022-01-31T20:58:29.889+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.889823+0000) 2022-01-31T20:58:30.250 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[35325]: debug 2022-01-31T20:58:29.898+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93045 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:30.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[41853]: debug 2022-01-31T20:58:29.889+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.889792+0000) 2022-01-31T20:58:30.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[46715]: debug 2022-01-31T20:58:29.888+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.888989+0000) 2022-01-31T20:58:30.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:29 smithi171 conmon[51620]: debug 2022-01-31T20:58:29.889+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:29.890738+0000) 2022-01-31T20:58:30.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:30 smithi167 conmon[60316]: debug 2022-01-31T20:58:30.126+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:30.126761+0000) 2022-01-31T20:58:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:30 smithi167 conmon[54076]: debug 2022-01-31T20:58:30.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:30.375713+0000) 2022-01-31T20:58:30.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:30 smithi171 conmon[46715]: debug 2022-01-31T20:58:30.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:30.404584+0000) 2022-01-31T20:58:30.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:30 smithi171 conmon[51620]: debug 2022-01-31T20:58:30.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:30.481812+0000) 2022-01-31T20:58:30.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:30 smithi167 conmon[49112]: debug 2022-01-31T20:58:30.447+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:30.448050+0000) 2022-01-31T20:58:31.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:30 smithi171 conmon[41853]: debug 2022-01-31T20:58:30.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:30.609236+0000) 2022-01-31T20:58:31.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:31 smithi167 conmon[60316]: debug 2022-01-31T20:58:31.126+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:31.126886+0000) 2022-01-31T20:58:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:31 smithi167 conmon[54076]: debug 2022-01-31T20:58:31.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:31.375828+0000) 2022-01-31T20:58:31.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:31 smithi171 conmon[46715]: debug 2022-01-31T20:58:31.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:31.404723+0000) 2022-01-31T20:58:31.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:31 smithi171 conmon[51620]: debug 2022-01-31T20:58:31.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:31.481958+0000) 2022-01-31T20:58:31.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:31 smithi167 conmon[49112]: debug 2022-01-31T20:58:31.447+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:31.448191+0000) 2022-01-31T20:58:32.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:31 smithi171 conmon[41853]: debug 2022-01-31T20:58:31.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:31.609342+0000) 2022-01-31T20:58:32.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:32 smithi167 conmon[60316]: debug 2022-01-31T20:58:32.126+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:32.127053+0000) 2022-01-31T20:58:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:32 smithi167 conmon[54076]: debug 2022-01-31T20:58:32.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:32.375971+0000) 2022-01-31T20:58:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:32 smithi171 conmon[46715]: debug 2022-01-31T20:58:32.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:32.404897+0000) 2022-01-31T20:58:32.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:32 smithi171 conmon[51620]: debug 2022-01-31T20:58:32.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:32.482068+0000) 2022-01-31T20:58:32.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:32 smithi167 conmon[49112]: debug 2022-01-31T20:58:32.447+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:32.448324+0000) 2022-01-31T20:58:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:32 smithi171 conmon[41853]: debug 2022-01-31T20:58:32.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:32.609546+0000) 2022-01-31T20:58:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:33 smithi167 conmon[60316]: debug 2022-01-31T20:58:33.126+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:33.127212+0000) 2022-01-31T20:58:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:33 smithi167 conmon[54076]: debug 2022-01-31T20:58:33.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:33.376098+0000) 2022-01-31T20:58:33.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:33 smithi171 conmon[46715]: debug 2022-01-31T20:58:33.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:33.405065+0000) 2022-01-31T20:58:33.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:33 smithi171 conmon[51620]: debug 2022-01-31T20:58:33.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:33.482252+0000) 2022-01-31T20:58:33.693 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:33 smithi167 conmon[49112]: debug 2022-01-31T20:58:33.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:33.448495+0000) 2022-01-31T20:58:34.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:33 smithi171 conmon[41853]: debug 2022-01-31T20:58:33.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:33.609694+0000) 2022-01-31T20:58:34.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:34 smithi167 conmon[60316]: debug 2022-01-31T20:58:34.126+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.127355+0000) 2022-01-31T20:58:34.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:34 smithi167 conmon[54076]: debug 2022-01-31T20:58:34.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.376272+0000) 2022-01-31T20:58:34.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[46715]: debug 2022-01-31T20:58:34.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.405228+0000) 2022-01-31T20:58:34.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[51620]: debug 2022-01-31T20:58:34.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.482435+0000) 2022-01-31T20:58:34.693 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:34 smithi167 conmon[49112]: debug 2022-01-31T20:58:34.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.448712+0000) 2022-01-31T20:58:34.899 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[41853]: debug 2022-01-31T20:58:34.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.609832+0000) 2022-01-31T20:58:35.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:34 smithi167 conmon[49112]: debug 2022-01-31T20:58:34.901+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.901559+0000) 2022-01-31T20:58:35.126 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:34 smithi167 conmon[54076]: debug 2022-01-31T20:58:34.901+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.902170+0000) 2022-01-31T20:58:35.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:34 smithi167 conmon[60316]: debug 2022-01-31T20:58:34.909+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.910257+0000) 2022-01-31T20:58:35.251 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[35325]: debug 2022-01-31T20:58:34.926+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93155 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:35.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[41853]: debug 2022-01-31T20:58:34.900+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.901167+0000) 2022-01-31T20:58:35.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[46715]: debug 2022-01-31T20:58:34.900+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.901016+0000) 2022-01-31T20:58:35.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:34 smithi171 conmon[51620]: debug 2022-01-31T20:58:34.900+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:34.901280+0000) 2022-01-31T20:58:35.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:35 smithi167 conmon[60316]: debug 2022-01-31T20:58:35.127+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:35.127524+0000) 2022-01-31T20:58:35.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:35 smithi167 conmon[54076]: debug 2022-01-31T20:58:35.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:35.376471+0000) 2022-01-31T20:58:35.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:35 smithi171 conmon[46715]: debug 2022-01-31T20:58:35.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:35.405378+0000) 2022-01-31T20:58:35.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:35 smithi171 conmon[51620]: debug 2022-01-31T20:58:35.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:35.482620+0000) 2022-01-31T20:58:35.693 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:35 smithi167 conmon[49112]: debug 2022-01-31T20:58:35.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:35.448886+0000) 2022-01-31T20:58:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:35 smithi171 conmon[41853]: debug 2022-01-31T20:58:35.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:35.609984+0000) 2022-01-31T20:58:36.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:36 smithi167 conmon[60316]: debug 2022-01-31T20:58:36.127+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:36.127675+0000) 2022-01-31T20:58:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:36 smithi167 conmon[54076]: debug 2022-01-31T20:58:36.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:36.376611+0000) 2022-01-31T20:58:36.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:36 smithi171 conmon[46715]: debug 2022-01-31T20:58:36.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:36.405527+0000) 2022-01-31T20:58:36.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:36 smithi171 conmon[51620]: debug 2022-01-31T20:58:36.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:36.482782+0000) 2022-01-31T20:58:36.693 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:36 smithi167 conmon[49112]: debug 2022-01-31T20:58:36.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:36.449059+0000) 2022-01-31T20:58:37.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:36 smithi171 conmon[41853]: debug 2022-01-31T20:58:36.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:36.610080+0000) 2022-01-31T20:58:37.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:37 smithi167 conmon[60316]: debug 2022-01-31T20:58:37.127+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:37.127853+0000) 2022-01-31T20:58:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:37 smithi167 conmon[54076]: debug 2022-01-31T20:58:37.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:37.376792+0000) 2022-01-31T20:58:37.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:37 smithi171 conmon[46715]: debug 2022-01-31T20:58:37.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:37.405692+0000) 2022-01-31T20:58:37.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:37 smithi171 conmon[51620]: debug 2022-01-31T20:58:37.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:37.482940+0000) 2022-01-31T20:58:37.693 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:37 smithi167 conmon[49112]: debug 2022-01-31T20:58:37.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:37.449211+0000) 2022-01-31T20:58:38.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:37 smithi171 conmon[41853]: debug 2022-01-31T20:58:37.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:37.610235+0000) 2022-01-31T20:58:38.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:38 smithi167 conmon[60316]: debug 2022-01-31T20:58:38.127+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:38.128035+0000) 2022-01-31T20:58:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:38 smithi167 conmon[54076]: debug 2022-01-31T20:58:38.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:38.377016+0000) 2022-01-31T20:58:38.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:38 smithi171 conmon[46715]: debug 2022-01-31T20:58:38.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:38.405863+0000) 2022-01-31T20:58:38.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:38 smithi171 conmon[51620]: debug 2022-01-31T20:58:38.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:38.483161+0000) 2022-01-31T20:58:38.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:38 smithi167 conmon[49112]: debug 2022-01-31T20:58:38.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:38.449325+0000) 2022-01-31T20:58:38.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:38 smithi171 conmon[35325]: debug 2022-01-31T20:58:38.602+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:58:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:38 smithi171 conmon[41853]: debug 2022-01-31T20:58:38.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:38.610388+0000) 2022-01-31T20:58:39.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:39 smithi167 conmon[60316]: debug 2022-01-31T20:58:39.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.128214+0000) 2022-01-31T20:58:39.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:39 smithi167 conmon[54076]: debug 2022-01-31T20:58:39.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.377222+0000) 2022-01-31T20:58:39.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[46715]: debug 2022-01-31T20:58:39.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.406053+0000) 2022-01-31T20:58:39.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[51620]: debug 2022-01-31T20:58:39.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.483336+0000) 2022-01-31T20:58:39.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:39 smithi167 conmon[49112]: debug 2022-01-31T20:58:39.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.449523+0000) 2022-01-31T20:58:39.927 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[41853]: debug 2022-01-31T20:58:39.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.610510+0000) 2022-01-31T20:58:40.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:39 smithi167 conmon[49112]: debug 2022-01-31T20:58:39.929+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.930094+0000) 2022-01-31T20:58:40.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:39 smithi167 conmon[60316]: debug 2022-01-31T20:58:39.930+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.931003+0000) 2022-01-31T20:58:40.127 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:39 smithi167 conmon[54076]: debug 2022-01-31T20:58:39.930+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.930766+0000) 2022-01-31T20:58:40.252 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[35325]: debug 2022-01-31T20:58:39.938+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93266 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:40.252 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[41853]: debug 2022-01-31T20:58:39.930+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.931291+0000) 2022-01-31T20:58:40.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[46715]: debug 2022-01-31T20:58:39.929+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.930386+0000) 2022-01-31T20:58:40.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:39 smithi171 conmon[51620]: debug 2022-01-31T20:58:39.928+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:39.929488+0000) 2022-01-31T20:58:40.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:40 smithi167 conmon[54076]: debug 2022-01-31T20:58:40.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:40.377396+0000) 2022-01-31T20:58:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:40 smithi167 conmon[60316]: debug 2022-01-31T20:58:40.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:40.128387+0000) 2022-01-31T20:58:40.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:40 smithi171 conmon[51620]: debug 2022-01-31T20:58:40.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:40.483498+0000) 2022-01-31T20:58:40.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:40 smithi171 conmon[46715]: debug 2022-01-31T20:58:40.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:40.406215+0000) 2022-01-31T20:58:40.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:40 smithi167 conmon[49112]: debug 2022-01-31T20:58:40.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:40.449723+0000) 2022-01-31T20:58:41.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:40 smithi171 conmon[41853]: debug 2022-01-31T20:58:40.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:40.610700+0000) 2022-01-31T20:58:41.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:41 smithi167 conmon[60316]: debug 2022-01-31T20:58:41.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:41.128536+0000) 2022-01-31T20:58:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:41 smithi167 conmon[54076]: debug 2022-01-31T20:58:41.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:41.377521+0000) 2022-01-31T20:58:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:41 smithi171 conmon[46715]: debug 2022-01-31T20:58:41.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:41.406338+0000) 2022-01-31T20:58:41.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:41 smithi171 conmon[51620]: debug 2022-01-31T20:58:41.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:41.483647+0000) 2022-01-31T20:58:41.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:41 smithi167 conmon[49112]: debug 2022-01-31T20:58:41.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:41.449874+0000) 2022-01-31T20:58:42.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:41 smithi171 conmon[41853]: debug 2022-01-31T20:58:41.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:41.610832+0000) 2022-01-31T20:58:42.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:42 smithi167 conmon[60316]: debug 2022-01-31T20:58:42.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:42.128814+0000) 2022-01-31T20:58:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:42 smithi167 conmon[54076]: debug 2022-01-31T20:58:42.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:42.377670+0000) 2022-01-31T20:58:42.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:42 smithi171 conmon[46715]: debug 2022-01-31T20:58:42.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:42.406517+0000) 2022-01-31T20:58:42.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:42 smithi171 conmon[51620]: debug 2022-01-31T20:58:42.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:42.483820+0000) 2022-01-31T20:58:42.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:42 smithi167 conmon[49112]: debug 2022-01-31T20:58:42.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:42.450039+0000) 2022-01-31T20:58:43.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:42 smithi171 conmon[41853]: debug 2022-01-31T20:58:42.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:42.611004+0000) 2022-01-31T20:58:43.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:43 smithi167 conmon[54076]: debug 2022-01-31T20:58:43.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:43.377852+0000) 2022-01-31T20:58:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:43 smithi167 conmon[60316]: debug 2022-01-31T20:58:43.127+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:43.128968+0000) 2022-01-31T20:58:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:43 smithi171 conmon[46715]: debug 2022-01-31T20:58:43.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:43.406713+0000) 2022-01-31T20:58:43.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:43 smithi171 conmon[51620]: debug 2022-01-31T20:58:43.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:43.483979+0000) 2022-01-31T20:58:43.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:43 smithi167 conmon[49112]: debug 2022-01-31T20:58:43.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:43.450184+0000) 2022-01-31T20:58:44.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:43 smithi171 conmon[41853]: debug 2022-01-31T20:58:43.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:43.611214+0000) 2022-01-31T20:58:44.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:44 smithi167 conmon[54076]: debug 2022-01-31T20:58:44.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.378019+0000) 2022-01-31T20:58:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:44 smithi167 conmon[60316]: debug 2022-01-31T20:58:44.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.129161+0000) 2022-01-31T20:58:44.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[46715]: debug 2022-01-31T20:58:44.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.406908+0000) 2022-01-31T20:58:44.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[51620]: debug 2022-01-31T20:58:44.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.484163+0000) 2022-01-31T20:58:44.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:44 smithi167 conmon[49112]: debug 2022-01-31T20:58:44.448+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.450427+0000) 2022-01-31T20:58:44.940 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[41853]: debug 2022-01-31T20:58:44.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.611415+0000) 2022-01-31T20:58:45.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:44 smithi167 conmon[49112]: debug 2022-01-31T20:58:44.942+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.944086+0000) 2022-01-31T20:58:45.128 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:44 smithi167 conmon[54076]: debug 2022-01-31T20:58:44.941+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.943289+0000) 2022-01-31T20:58:45.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:44 smithi167 conmon[60316]: debug 2022-01-31T20:58:44.942+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.943924+0000) 2022-01-31T20:58:45.253 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[35325]: debug 2022-01-31T20:58:44.950+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93375 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:45.253 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[41853]: debug 2022-01-31T20:58:44.942+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.943608+0000) 2022-01-31T20:58:45.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[46715]: debug 2022-01-31T20:58:44.941+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.941826+0000) 2022-01-31T20:58:45.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:44 smithi171 conmon[51620]: debug 2022-01-31T20:58:44.940+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:44.941547+0000) 2022-01-31T20:58:45.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:45 smithi167 conmon[54076]: debug 2022-01-31T20:58:45.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:45.378204+0000) 2022-01-31T20:58:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:45 smithi167 conmon[60316]: debug 2022-01-31T20:58:45.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:45.129340+0000) 2022-01-31T20:58:45.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:45 smithi171 conmon[46715]: debug 2022-01-31T20:58:45.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:45.407096+0000) 2022-01-31T20:58:45.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:45 smithi171 conmon[51620]: debug 2022-01-31T20:58:45.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:45.484322+0000) 2022-01-31T20:58:45.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:45 smithi167 conmon[49112]: debug 2022-01-31T20:58:45.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:45.450544+0000) 2022-01-31T20:58:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:45 smithi171 conmon[41853]: debug 2022-01-31T20:58:45.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:45.611603+0000) 2022-01-31T20:58:46.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:46 smithi167 conmon[54076]: debug 2022-01-31T20:58:46.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:46.378377+0000) 2022-01-31T20:58:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:46 smithi167 conmon[60316]: debug 2022-01-31T20:58:46.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:46.129517+0000) 2022-01-31T20:58:46.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:46 smithi171 conmon[46715]: debug 2022-01-31T20:58:46.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:46.407253+0000) 2022-01-31T20:58:46.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:46 smithi171 conmon[51620]: debug 2022-01-31T20:58:46.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:46.484447+0000) 2022-01-31T20:58:46.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:46 smithi167 conmon[49112]: debug 2022-01-31T20:58:46.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:46.450652+0000) 2022-01-31T20:58:47.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:46 smithi171 conmon[41853]: debug 2022-01-31T20:58:46.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:46.611787+0000) 2022-01-31T20:58:47.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:47 smithi167 conmon[54076]: debug 2022-01-31T20:58:47.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:47.378559+0000) 2022-01-31T20:58:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:47 smithi167 conmon[60316]: debug 2022-01-31T20:58:47.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:47.129660+0000) 2022-01-31T20:58:47.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:47 smithi171 conmon[46715]: debug 2022-01-31T20:58:47.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:47.407449+0000) 2022-01-31T20:58:47.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:47 smithi171 conmon[51620]: debug 2022-01-31T20:58:47.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:47.484654+0000) 2022-01-31T20:58:47.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:47 smithi167 conmon[49112]: debug 2022-01-31T20:58:47.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:47.450833+0000) 2022-01-31T20:58:48.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:47 smithi171 conmon[41853]: debug 2022-01-31T20:58:47.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:47.611941+0000) 2022-01-31T20:58:48.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:48 smithi167 conmon[54076]: debug 2022-01-31T20:58:48.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:48.378719+0000) 2022-01-31T20:58:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:48 smithi167 conmon[60316]: debug 2022-01-31T20:58:48.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:48.129772+0000) 2022-01-31T20:58:48.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:48 smithi171 conmon[51620]: debug 2022-01-31T20:58:48.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:48.484851+0000) 2022-01-31T20:58:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:48 smithi171 conmon[46715]: debug 2022-01-31T20:58:48.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:48.407626+0000) 2022-01-31T20:58:48.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:48 smithi167 conmon[49112]: debug 2022-01-31T20:58:48.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:48.451014+0000) 2022-01-31T20:58:49.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:48 smithi171 conmon[41853]: debug 2022-01-31T20:58:48.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:48.612131+0000) 2022-01-31T20:58:49.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:49 smithi167 conmon[54076]: debug 2022-01-31T20:58:49.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.378865+0000) 2022-01-31T20:58:49.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:49 smithi167 conmon[60316]: debug 2022-01-31T20:58:49.128+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.129940+0000) 2022-01-31T20:58:49.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[51620]: debug 2022-01-31T20:58:49.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.485034+0000) 2022-01-31T20:58:49.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[46715]: debug 2022-01-31T20:58:49.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.407822+0000) 2022-01-31T20:58:49.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:49 smithi167 conmon[49112]: debug 2022-01-31T20:58:49.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.451217+0000) 2022-01-31T20:58:49.952 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[41853]: debug 2022-01-31T20:58:49.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.612296+0000) 2022-01-31T20:58:49.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[51620]: debug 2022-01-31T20:58:49.952+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.953718+0000) 2022-01-31T20:58:50.254 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[35325]: debug 2022-01-31T20:58:49.962+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93486 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:50.254 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[41853]: debug 2022-01-31T20:58:49.954+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.955194+0000) 2022-01-31T20:58:50.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:49 smithi171 conmon[46715]: debug 2022-01-31T20:58:49.953+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.953863+0000) 2022-01-31T20:58:50.349 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:49 smithi167 conmon[49112]: debug 2022-01-31T20:58:49.952+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.954144+0000) 2022-01-31T20:58:50.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:49 smithi167 conmon[54076]: debug 2022-01-31T20:58:49.953+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.955268+0000) 2022-01-31T20:58:50.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:49 smithi167 conmon[60316]: debug 2022-01-31T20:58:49.954+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:49.955600+0000) 2022-01-31T20:58:50.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:50 smithi167 conmon[60316]: debug 2022-01-31T20:58:50.129+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:50.130092+0000) 2022-01-31T20:58:50.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:50 smithi171 conmon[46715]: debug 2022-01-31T20:58:50.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:50.408005+0000) 2022-01-31T20:58:50.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:50 smithi171 conmon[51620]: debug 2022-01-31T20:58:50.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:50.485222+0000) 2022-01-31T20:58:50.641 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:50 smithi167 conmon[49112]: debug 2022-01-31T20:58:50.449+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:50.451427+0000) 2022-01-31T20:58:50.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:50 smithi167 conmon[54076]: debug 2022-01-31T20:58:50.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:50.379032+0000) 2022-01-31T20:58:51.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:50 smithi171 conmon[41853]: debug 2022-01-31T20:58:50.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:50.612499+0000) 2022-01-31T20:58:51.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:51 smithi167 conmon[54076]: debug 2022-01-31T20:58:51.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:51.379177+0000) 2022-01-31T20:58:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:51 smithi167 conmon[60316]: debug 2022-01-31T20:58:51.129+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:51.130240+0000) 2022-01-31T20:58:51.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:51 smithi171 conmon[46715]: debug 2022-01-31T20:58:51.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:51.408166+0000) 2022-01-31T20:58:51.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:51 smithi171 conmon[51620]: debug 2022-01-31T20:58:51.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:51.485370+0000) 2022-01-31T20:58:51.696 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:51 smithi167 conmon[49112]: debug 2022-01-31T20:58:51.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:51.451572+0000) 2022-01-31T20:58:52.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:51 smithi171 conmon[41853]: debug 2022-01-31T20:58:51.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:51.612649+0000) 2022-01-31T20:58:52.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:52 smithi167 conmon[54076]: debug 2022-01-31T20:58:52.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:52.379366+0000) 2022-01-31T20:58:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:52 smithi167 conmon[60316]: debug 2022-01-31T20:58:52.129+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:52.130425+0000) 2022-01-31T20:58:52.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:52 smithi171 conmon[46715]: debug 2022-01-31T20:58:52.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:52.408300+0000) 2022-01-31T20:58:52.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:52 smithi171 conmon[51620]: debug 2022-01-31T20:58:52.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:52.485550+0000) 2022-01-31T20:58:52.696 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:52 smithi167 conmon[49112]: debug 2022-01-31T20:58:52.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:52.451718+0000) 2022-01-31T20:58:53.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:52 smithi171 conmon[41853]: debug 2022-01-31T20:58:52.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:52.612853+0000) 2022-01-31T20:58:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:53 smithi167 conmon[54076]: debug 2022-01-31T20:58:53.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:53.379544+0000) 2022-01-31T20:58:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:53 smithi167 conmon[60316]: debug 2022-01-31T20:58:53.129+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:53.130605+0000) 2022-01-31T20:58:53.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:53 smithi171 conmon[46715]: debug 2022-01-31T20:58:53.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:53.408454+0000) 2022-01-31T20:58:53.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:53 smithi171 conmon[51620]: debug 2022-01-31T20:58:53.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:53.485729+0000) 2022-01-31T20:58:53.696 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:53 smithi167 conmon[49112]: debug 2022-01-31T20:58:53.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:53.451826+0000) 2022-01-31T20:58:53.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:53 smithi171 conmon[41853]: debug 2022-01-31T20:58:53.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:53.613025+0000) 2022-01-31T20:58:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:53 smithi171 conmon[35325]: debug 2022-01-31T20:58:53.602+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:58:54.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:54 smithi167 conmon[54076]: debug 2022-01-31T20:58:54.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.379706+0000) 2022-01-31T20:58:54.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:54 smithi167 conmon[60316]: debug 2022-01-31T20:58:54.129+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.130807+0000) 2022-01-31T20:58:54.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[46715]: debug 2022-01-31T20:58:54.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.408604+0000) 2022-01-31T20:58:54.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[51620]: debug 2022-01-31T20:58:54.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.485939+0000) 2022-01-31T20:58:54.696 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:54 smithi167 conmon[49112]: debug 2022-01-31T20:58:54.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.451950+0000) 2022-01-31T20:58:54.964 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[41853]: debug 2022-01-31T20:58:54.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.613196+0000) 2022-01-31T20:58:55.255 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[35325]: debug 2022-01-31T20:58:54.974+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93595 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:58:55.255 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[41853]: debug 2022-01-31T20:58:54.964+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.966649+0000) 2022-01-31T20:58:55.256 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[46715]: debug 2022-01-31T20:58:54.964+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.966069+0000) 2022-01-31T20:58:55.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:54 smithi171 conmon[51620]: debug 2022-01-31T20:58:54.964+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.965963+0000) 2022-01-31T20:58:55.350 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:54 smithi167 conmon[49112]: debug 2022-01-31T20:58:54.964+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.965835+0000) 2022-01-31T20:58:55.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:54 smithi167 conmon[54076]: debug 2022-01-31T20:58:54.965+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.967331+0000) 2022-01-31T20:58:55.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:54 smithi167 conmon[60316]: debug 2022-01-31T20:58:54.966+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:54.968309+0000) 2022-01-31T20:58:55.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:55 smithi167 conmon[60316]: debug 2022-01-31T20:58:55.129+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:55.130949+0000) 2022-01-31T20:58:55.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:55 smithi171 conmon[46715]: debug 2022-01-31T20:58:55.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:55.408792+0000) 2022-01-31T20:58:55.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:55 smithi171 conmon[51620]: debug 2022-01-31T20:58:55.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:55.486097+0000) 2022-01-31T20:58:55.647 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:55 smithi167 conmon[49112]: debug 2022-01-31T20:58:55.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:55.452060+0000) 2022-01-31T20:58:55.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:55 smithi167 conmon[54076]: debug 2022-01-31T20:58:55.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:55.379862+0000) 2022-01-31T20:58:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:55 smithi171 conmon[41853]: debug 2022-01-31T20:58:55.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:55.613395+0000) 2022-01-31T20:58:56.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:56 smithi167 conmon[54076]: debug 2022-01-31T20:58:56.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:56.380040+0000) 2022-01-31T20:58:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:56 smithi167 conmon[60316]: debug 2022-01-31T20:58:56.130+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:56.131054+0000) 2022-01-31T20:58:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:56 smithi171 conmon[46715]: debug 2022-01-31T20:58:56.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:56.408941+0000) 2022-01-31T20:58:56.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:56 smithi171 conmon[51620]: debug 2022-01-31T20:58:56.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:56.486233+0000) 2022-01-31T20:58:56.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:56 smithi167 conmon[49112]: debug 2022-01-31T20:58:56.451+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:56.452211+0000) 2022-01-31T20:58:57.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:56 smithi171 conmon[41853]: debug 2022-01-31T20:58:56.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:56.613565+0000) 2022-01-31T20:58:57.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:57 smithi167 conmon[54076]: debug 2022-01-31T20:58:57.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:57.380248+0000) 2022-01-31T20:58:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:57 smithi167 conmon[60316]: debug 2022-01-31T20:58:57.130+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:57.131207+0000) 2022-01-31T20:58:57.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:57 smithi171 conmon[46715]: debug 2022-01-31T20:58:57.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:57.409121+0000) 2022-01-31T20:58:57.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:57 smithi171 conmon[51620]: debug 2022-01-31T20:58:57.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:57.486417+0000) 2022-01-31T20:58:57.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:57 smithi167 conmon[49112]: debug 2022-01-31T20:58:57.450+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:57.452365+0000) 2022-01-31T20:58:58.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:57 smithi171 conmon[41853]: debug 2022-01-31T20:58:57.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:57.613743+0000) 2022-01-31T20:58:58.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:58 smithi167 conmon[54076]: debug 2022-01-31T20:58:58.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:58.380509+0000) 2022-01-31T20:58:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:58 smithi167 conmon[60316]: debug 2022-01-31T20:58:58.130+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:58.131383+0000) 2022-01-31T20:58:58.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:58 smithi171 conmon[46715]: debug 2022-01-31T20:58:58.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:58.409304+0000) 2022-01-31T20:58:58.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:58 smithi171 conmon[51620]: debug 2022-01-31T20:58:58.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:58.486601+0000) 2022-01-31T20:58:58.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:58 smithi167 conmon[49112]: debug 2022-01-31T20:58:58.451+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:58.452578+0000) 2022-01-31T20:58:59.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:58 smithi171 conmon[41853]: debug 2022-01-31T20:58:58.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:58.613907+0000) 2022-01-31T20:58:59.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:59 smithi167 conmon[54076]: debug 2022-01-31T20:58:59.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.380683+0000) 2022-01-31T20:58:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:59 smithi167 conmon[60316]: debug 2022-01-31T20:58:59.130+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.131552+0000) 2022-01-31T20:58:59.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[46715]: debug 2022-01-31T20:58:59.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.409483+0000) 2022-01-31T20:58:59.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[51620]: debug 2022-01-31T20:58:59.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.486764+0000) 2022-01-31T20:58:59.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:59 smithi167 conmon[49112]: debug 2022-01-31T20:58:59.451+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.452781+0000) 2022-01-31T20:58:59.976 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[41853]: debug 2022-01-31T20:58:59.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.614060+0000) 2022-01-31T20:59:00.256 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[35325]: debug 2022-01-31T20:58:59.986+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93706 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:00.256 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[41853]: debug 2022-01-31T20:58:59.976+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.978265+0000) 2022-01-31T20:59:00.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[46715]: debug 2022-01-31T20:58:59.977+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.979391+0000) 2022-01-31T20:59:00.257 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:58:59 smithi171 conmon[51620]: debug 2022-01-31T20:58:59.976+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.978062+0000) 2022-01-31T20:59:00.351 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:58:59 smithi167 conmon[49112]: debug 2022-01-31T20:58:59.978+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.980424+0000) 2022-01-31T20:59:00.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:58:59 smithi167 conmon[54076]: debug 2022-01-31T20:58:59.977+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.978752+0000) 2022-01-31T20:59:00.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:58:59 smithi167 conmon[60316]: debug 2022-01-31T20:58:59.978+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:58:59.979592+0000) 2022-01-31T20:59:00.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:00 smithi167 conmon[60316]: debug 2022-01-31T20:59:00.131+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:00.131764+0000) 2022-01-31T20:59:00.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:00 smithi171 conmon[51620]: debug 2022-01-31T20:59:00.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:00.486916+0000) 2022-01-31T20:59:00.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:00 smithi171 conmon[46715]: debug 2022-01-31T20:59:00.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:00.409709+0000) 2022-01-31T20:59:00.652 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:00 smithi167 conmon[49112]: debug 2022-01-31T20:59:00.452+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:00.452998+0000) 2022-01-31T20:59:00.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:00 smithi167 conmon[54076]: debug 2022-01-31T20:59:00.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:00.380868+0000) 2022-01-31T20:59:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:00 smithi171 conmon[41853]: debug 2022-01-31T20:59:00.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:00.614209+0000) 2022-01-31T20:59:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:01 smithi167 conmon[54076]: debug 2022-01-31T20:59:01.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:01.381015+0000) 2022-01-31T20:59:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:01 smithi167 conmon[60316]: debug 2022-01-31T20:59:01.130+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:01.131911+0000) 2022-01-31T20:59:01.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:01 smithi171 conmon[46715]: debug 2022-01-31T20:59:01.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:01.409830+0000) 2022-01-31T20:59:01.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:01 smithi171 conmon[51620]: debug 2022-01-31T20:59:01.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:01.487076+0000) 2022-01-31T20:59:01.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:01 smithi167 conmon[49112]: debug 2022-01-31T20:59:01.451+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:01.453172+0000) 2022-01-31T20:59:02.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:01 smithi171 conmon[41853]: debug 2022-01-31T20:59:01.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:01.614368+0000) 2022-01-31T20:59:02.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:02 smithi167 conmon[54076]: debug 2022-01-31T20:59:02.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:02.381232+0000) 2022-01-31T20:59:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:02 smithi167 conmon[60316]: debug 2022-01-31T20:59:02.131+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:02.132103+0000) 2022-01-31T20:59:02.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:02 smithi167 conmon[49112]: debug 2022-01-31T20:59:02.451+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:02.453388+0000) 2022-01-31T20:59:02.738 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:02 smithi171 conmon[41853]: debug 2022-01-31T20:59:02.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:02.614567+0000) 2022-01-31T20:59:02.739 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:02 smithi171 conmon[46715]: debug 2022-01-31T20:59:02.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:02.409986+0000) 2022-01-31T20:59:02.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:02 smithi171 conmon[51620]: debug 2022-01-31T20:59:02.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:02.487220+0000) 2022-01-31T20:59:03.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:03 smithi167 conmon[54076]: debug 2022-01-31T20:59:03.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:03.381426+0000) 2022-01-31T20:59:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:03 smithi167 conmon[60316]: debug 2022-01-31T20:59:03.131+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:03.132267+0000) 2022-01-31T20:59:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:03 smithi171 conmon[46715]: debug 2022-01-31T20:59:03.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:03.410210+0000) 2022-01-31T20:59:03.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:03 smithi171 conmon[51620]: debug 2022-01-31T20:59:03.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:03.487375+0000) 2022-01-31T20:59:03.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:03 smithi167 conmon[49112]: debug 2022-01-31T20:59:03.452+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:03.453585+0000) 2022-01-31T20:59:04.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:03 smithi171 conmon[41853]: debug 2022-01-31T20:59:03.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:03.614735+0000) 2022-01-31T20:59:04.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:04 smithi167 conmon[54076]: debug 2022-01-31T20:59:04.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.381598+0000) 2022-01-31T20:59:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:04 smithi167 conmon[60316]: debug 2022-01-31T20:59:04.131+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.132471+0000) 2022-01-31T20:59:04.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:04 smithi171 conmon[46715]: debug 2022-01-31T20:59:04.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.410372+0000) 2022-01-31T20:59:04.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:04 smithi171 conmon[51620]: debug 2022-01-31T20:59:04.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.487564+0000) 2022-01-31T20:59:04.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:04 smithi167 conmon[49112]: debug 2022-01-31T20:59:04.452+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.453779+0000) 2022-01-31T20:59:04.988 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:04 smithi171 conmon[41853]: debug 2022-01-31T20:59:04.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.614868+0000) 2022-01-31T20:59:04.989 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:04 smithi171 conmon[41853]: debug 2022-01-31T20:59:04.989+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.990405+0000) 2022-01-31T20:59:05.257 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:05 smithi171 conmon[35325]: debug 2022-01-31T20:59:04.999+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93816 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:05.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:04 smithi171 conmon[46715]: debug 2022-01-31T20:59:04.990+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.991273+0000) 2022-01-31T20:59:05.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:04 smithi171 conmon[51620]: debug 2022-01-31T20:59:04.991+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.991985+0000) 2022-01-31T20:59:05.306 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:04 smithi167 conmon[49112]: debug 2022-01-31T20:59:04.990+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.991772+0000) 2022-01-31T20:59:05.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:04 smithi167 conmon[54076]: debug 2022-01-31T20:59:04.990+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.992124+0000) 2022-01-31T20:59:05.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:04 smithi167 conmon[60316]: debug 2022-01-31T20:59:04.989+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:04.991198+0000) 2022-01-31T20:59:05.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:05 smithi167 conmon[60316]: debug 2022-01-31T20:59:05.132+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:05.132666+0000) 2022-01-31T20:59:05.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:05 smithi171 conmon[46715]: debug 2022-01-31T20:59:05.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:05.410533+0000) 2022-01-31T20:59:05.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:05 smithi171 conmon[51620]: debug 2022-01-31T20:59:05.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:05.487771+0000) 2022-01-31T20:59:05.658 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:05 smithi167 conmon[49112]: debug 2022-01-31T20:59:05.453+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:05.453957+0000) 2022-01-31T20:59:05.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:05 smithi167 conmon[54076]: debug 2022-01-31T20:59:05.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:05.381759+0000) 2022-01-31T20:59:06.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:05 smithi171 conmon[41853]: debug 2022-01-31T20:59:05.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:05.614983+0000) 2022-01-31T20:59:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:06 smithi167 conmon[54076]: debug 2022-01-31T20:59:06.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:06.381895+0000) 2022-01-31T20:59:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:06 smithi167 conmon[60316]: debug 2022-01-31T20:59:06.132+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:06.132826+0000) 2022-01-31T20:59:06.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:06 smithi171 conmon[46715]: debug 2022-01-31T20:59:06.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:06.410658+0000) 2022-01-31T20:59:06.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:06 smithi171 conmon[51620]: debug 2022-01-31T20:59:06.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:06.487944+0000) 2022-01-31T20:59:06.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:06 smithi167 conmon[49112]: debug 2022-01-31T20:59:06.453+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:06.454105+0000) 2022-01-31T20:59:07.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:06 smithi171 conmon[41853]: debug 2022-01-31T20:59:06.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:06.615088+0000) 2022-01-31T20:59:07.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:07 smithi167 conmon[54076]: debug 2022-01-31T20:59:07.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:07.382056+0000) 2022-01-31T20:59:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:07 smithi167 conmon[60316]: debug 2022-01-31T20:59:07.132+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:07.132991+0000) 2022-01-31T20:59:07.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:07 smithi171 conmon[46715]: debug 2022-01-31T20:59:07.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:07.410833+0000) 2022-01-31T20:59:07.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:07 smithi171 conmon[51620]: debug 2022-01-31T20:59:07.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:07.488171+0000) 2022-01-31T20:59:07.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:07 smithi167 conmon[49112]: debug 2022-01-31T20:59:07.453+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:07.454255+0000) 2022-01-31T20:59:08.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:07 smithi171 conmon[41853]: debug 2022-01-31T20:59:07.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:07.615275+0000) 2022-01-31T20:59:08.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:08 smithi167 conmon[54076]: debug 2022-01-31T20:59:08.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:08.382217+0000) 2022-01-31T20:59:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:08 smithi167 conmon[60316]: debug 2022-01-31T20:59:08.132+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:08.133197+0000) 2022-01-31T20:59:08.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:08 smithi171 conmon[46715]: debug 2022-01-31T20:59:08.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:08.411032+0000) 2022-01-31T20:59:08.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:08 smithi171 conmon[51620]: debug 2022-01-31T20:59:08.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:08.488305+0000) 2022-01-31T20:59:08.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:08 smithi167 conmon[49112]: debug 2022-01-31T20:59:08.453+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:08.454464+0000) 2022-01-31T20:59:08.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:08 smithi171 conmon[35325]: debug 2022-01-31T20:59:08.603+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:59:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:08 smithi171 conmon[41853]: debug 2022-01-31T20:59:08.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:08.615456+0000) 2022-01-31T20:59:09.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:09 smithi167 conmon[54076]: debug 2022-01-31T20:59:09.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:09.382400+0000) 2022-01-31T20:59:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:09 smithi167 conmon[60316]: debug 2022-01-31T20:59:09.132+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:09.133401+0000) 2022-01-31T20:59:09.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:09 smithi171 conmon[46715]: debug 2022-01-31T20:59:09.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:09.411219+0000) 2022-01-31T20:59:09.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:09 smithi171 conmon[51620]: debug 2022-01-31T20:59:09.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:09.488467+0000) 2022-01-31T20:59:09.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:09 smithi167 conmon[49112]: debug 2022-01-31T20:59:09.454+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:09.454660+0000) 2022-01-31T20:59:10.001 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:09 smithi171 conmon[41853]: debug 2022-01-31T20:59:09.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:09.615617+0000) 2022-01-31T20:59:10.258 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[35325]: debug 2022-01-31T20:59:10.012+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 93922 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:10.258 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[41853]: debug 2022-01-31T20:59:10.002+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.002786+0000) 2022-01-31T20:59:10.259 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[46715]: debug 2022-01-31T20:59:10.002+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.003181+0000) 2022-01-31T20:59:10.259 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[51620]: debug 2022-01-31T20:59:10.003+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.004509+0000) 2022-01-31T20:59:10.352 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:10 smithi167 conmon[49112]: debug 2022-01-31T20:59:10.002+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.002924+0000) 2022-01-31T20:59:10.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:10 smithi167 conmon[54076]: debug 2022-01-31T20:59:10.003+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.004030+0000) 2022-01-31T20:59:10.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:10 smithi167 conmon[60316]: debug 2022-01-31T20:59:10.002+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.003057+0000) 2022-01-31T20:59:10.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:10 smithi167 conmon[60316]: debug 2022-01-31T20:59:10.133+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.133630+0000) 2022-01-31T20:59:10.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[46715]: debug 2022-01-31T20:59:10.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.411431+0000) 2022-01-31T20:59:10.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[51620]: debug 2022-01-31T20:59:10.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.488656+0000) 2022-01-31T20:59:10.663 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:10 smithi167 conmon[49112]: debug 2022-01-31T20:59:10.454+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.454840+0000) 2022-01-31T20:59:10.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:10 smithi167 conmon[54076]: debug 2022-01-31T20:59:10.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.382597+0000) 2022-01-31T20:59:11.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:10 smithi171 conmon[41853]: debug 2022-01-31T20:59:10.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:10.615834+0000) 2022-01-31T20:59:11.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:11 smithi167 conmon[54076]: debug 2022-01-31T20:59:11.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:11.382741+0000) 2022-01-31T20:59:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:11 smithi167 conmon[60316]: debug 2022-01-31T20:59:11.133+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:11.133771+0000) 2022-01-31T20:59:11.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:11 smithi171 conmon[46715]: debug 2022-01-31T20:59:11.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:11.411570+0000) 2022-01-31T20:59:11.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:11 smithi171 conmon[51620]: debug 2022-01-31T20:59:11.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:11.488847+0000) 2022-01-31T20:59:11.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:11 smithi167 conmon[49112]: debug 2022-01-31T20:59:11.454+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:11.455029+0000) 2022-01-31T20:59:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:11 smithi171 conmon[41853]: debug 2022-01-31T20:59:11.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:11.615967+0000) 2022-01-31T20:59:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:12 smithi167 conmon[60316]: debug 2022-01-31T20:59:12.133+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:12.133891+0000) 2022-01-31T20:59:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:12 smithi167 conmon[54076]: debug 2022-01-31T20:59:12.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:12.382950+0000) 2022-01-31T20:59:12.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:12 smithi171 conmon[46715]: debug 2022-01-31T20:59:12.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:12.411783+0000) 2022-01-31T20:59:12.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:12 smithi171 conmon[51620]: debug 2022-01-31T20:59:12.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:12.489031+0000) 2022-01-31T20:59:12.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:12 smithi167 conmon[49112]: debug 2022-01-31T20:59:12.454+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:12.455252+0000) 2022-01-31T20:59:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:12 smithi171 conmon[41853]: debug 2022-01-31T20:59:12.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:12.616120+0000) 2022-01-31T20:59:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:13 smithi167 conmon[54076]: debug 2022-01-31T20:59:13.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:13.383156+0000) 2022-01-31T20:59:13.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:13 smithi167 conmon[60316]: debug 2022-01-31T20:59:13.133+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:13.134082+0000) 2022-01-31T20:59:13.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:13 smithi171 conmon[46715]: debug 2022-01-31T20:59:13.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:13.411979+0000) 2022-01-31T20:59:13.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:13 smithi171 conmon[51620]: debug 2022-01-31T20:59:13.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:13.489214+0000) 2022-01-31T20:59:13.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:13 smithi167 conmon[49112]: debug 2022-01-31T20:59:13.454+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:13.455431+0000) 2022-01-31T20:59:14.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:13 smithi171 conmon[41853]: debug 2022-01-31T20:59:13.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:13.616298+0000) 2022-01-31T20:59:14.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:14 smithi167 conmon[54076]: debug 2022-01-31T20:59:14.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:14.383316+0000) 2022-01-31T20:59:14.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:14 smithi167 conmon[60316]: debug 2022-01-31T20:59:14.133+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:14.134269+0000) 2022-01-31T20:59:14.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:14 smithi171 conmon[51620]: debug 2022-01-31T20:59:14.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:14.489403+0000) 2022-01-31T20:59:14.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:14 smithi171 conmon[46715]: debug 2022-01-31T20:59:14.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:14.412146+0000) 2022-01-31T20:59:14.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:14 smithi167 conmon[49112]: debug 2022-01-31T20:59:14.455+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:14.455611+0000) 2022-01-31T20:59:15.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:14 smithi171 conmon[41853]: debug 2022-01-31T20:59:14.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:14.616507+0000) 2022-01-31T20:59:15.353 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:15 smithi167 conmon[49112]: debug 2022-01-31T20:59:15.014+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.014627+0000) 2022-01-31T20:59:15.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:15 smithi167 conmon[54076]: debug 2022-01-31T20:59:15.015+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.015718+0000) 2022-01-31T20:59:15.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:15 smithi167 conmon[60316]: debug 2022-01-31T20:59:15.014+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.015003+0000) 2022-01-31T20:59:15.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:15 smithi167 conmon[60316]: debug 2022-01-31T20:59:15.133+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.134426+0000) 2022-01-31T20:59:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[46715]: debug 2022-01-31T20:59:15.014+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.014806+0000) 2022-01-31T20:59:15.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[51620]: debug 2022-01-31T20:59:15.015+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.016313+0000) 2022-01-31T20:59:15.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[35325]: debug 2022-01-31T20:59:15.024+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94032 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:15.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[41853]: debug 2022-01-31T20:59:15.014+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.015672+0000) 2022-01-31T20:59:15.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:15 smithi167 conmon[49112]: debug 2022-01-31T20:59:15.455+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.455795+0000) 2022-01-31T20:59:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:15 smithi167 conmon[54076]: debug 2022-01-31T20:59:15.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.383540+0000) 2022-01-31T20:59:15.741 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[46715]: debug 2022-01-31T20:59:15.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.412335+0000) 2022-01-31T20:59:15.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[51620]: debug 2022-01-31T20:59:15.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.489554+0000) 2022-01-31T20:59:15.743 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:15 smithi171 conmon[41853]: debug 2022-01-31T20:59:15.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:15.616661+0000) 2022-01-31T20:59:16.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:16 smithi167 conmon[54076]: debug 2022-01-31T20:59:16.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:16.383736+0000) 2022-01-31T20:59:16.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:16 smithi167 conmon[60316]: debug 2022-01-31T20:59:16.134+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:16.134599+0000) 2022-01-31T20:59:16.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:16 smithi171 conmon[46715]: debug 2022-01-31T20:59:16.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:16.412505+0000) 2022-01-31T20:59:16.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:16 smithi171 conmon[51620]: debug 2022-01-31T20:59:16.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:16.489700+0000) 2022-01-31T20:59:16.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:16 smithi167 conmon[49112]: debug 2022-01-31T20:59:16.455+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:16.455906+0000) 2022-01-31T20:59:17.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:16 smithi171 conmon[41853]: debug 2022-01-31T20:59:16.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:16.616827+0000) 2022-01-31T20:59:17.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:17 smithi167 conmon[54076]: debug 2022-01-31T20:59:17.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:17.383920+0000) 2022-01-31T20:59:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:17 smithi167 conmon[60316]: debug 2022-01-31T20:59:17.134+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:17.134801+0000) 2022-01-31T20:59:17.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:17 smithi171 conmon[46715]: debug 2022-01-31T20:59:17.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:17.412661+0000) 2022-01-31T20:59:17.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:17 smithi171 conmon[51620]: debug 2022-01-31T20:59:17.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:17.489822+0000) 2022-01-31T20:59:17.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:17 smithi167 conmon[49112]: debug 2022-01-31T20:59:17.455+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:17.456073+0000) 2022-01-31T20:59:18.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:17 smithi171 conmon[41853]: debug 2022-01-31T20:59:17.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:17.616983+0000) 2022-01-31T20:59:18.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:18 smithi167 conmon[54076]: debug 2022-01-31T20:59:18.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:18.384117+0000) 2022-01-31T20:59:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:18 smithi167 conmon[60316]: debug 2022-01-31T20:59:18.134+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:18.134998+0000) 2022-01-31T20:59:18.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:18 smithi171 conmon[46715]: debug 2022-01-31T20:59:18.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:18.412830+0000) 2022-01-31T20:59:18.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:18 smithi171 conmon[51620]: debug 2022-01-31T20:59:18.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:18.489946+0000) 2022-01-31T20:59:18.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:18 smithi167 conmon[49112]: debug 2022-01-31T20:59:18.455+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:18.456265+0000) 2022-01-31T20:59:19.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:18 smithi171 conmon[41853]: debug 2022-01-31T20:59:18.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:18.617160+0000) 2022-01-31T20:59:19.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:19 smithi167 conmon[54076]: debug 2022-01-31T20:59:19.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:19.384331+0000) 2022-01-31T20:59:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:19 smithi167 conmon[60316]: debug 2022-01-31T20:59:19.134+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:19.135131+0000) 2022-01-31T20:59:19.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:19 smithi171 conmon[46715]: debug 2022-01-31T20:59:19.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:19.413015+0000) 2022-01-31T20:59:19.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:19 smithi171 conmon[51620]: debug 2022-01-31T20:59:19.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:19.490102+0000) 2022-01-31T20:59:19.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:19 smithi167 conmon[49112]: debug 2022-01-31T20:59:19.455+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:19.456388+0000) 2022-01-31T20:59:20.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:19 smithi171 conmon[41853]: debug 2022-01-31T20:59:19.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:19.617309+0000) 2022-01-31T20:59:20.354 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:20 smithi167 conmon[49112]: debug 2022-01-31T20:59:20.027+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.028120+0000) 2022-01-31T20:59:20.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:20 smithi167 conmon[54076]: debug 2022-01-31T20:59:20.027+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.028244+0000) 2022-01-31T20:59:20.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:20 smithi167 conmon[60316]: debug 2022-01-31T20:59:20.026+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.027285+0000) 2022-01-31T20:59:20.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:20 smithi167 conmon[60316]: debug 2022-01-31T20:59:20.134+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.135308+0000) 2022-01-31T20:59:20.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[35325]: debug 2022-01-31T20:59:20.035+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94142 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:20.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[41853]: debug 2022-01-31T20:59:20.026+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.026926+0000) 2022-01-31T20:59:20.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[46715]: debug 2022-01-31T20:59:20.026+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.027217+0000) 2022-01-31T20:59:20.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[51620]: debug 2022-01-31T20:59:20.025+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.026622+0000) 2022-01-31T20:59:20.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:20 smithi167 conmon[49112]: debug 2022-01-31T20:59:20.456+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.456576+0000) 2022-01-31T20:59:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:20 smithi167 conmon[54076]: debug 2022-01-31T20:59:20.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.384521+0000) 2022-01-31T20:59:20.741 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[46715]: debug 2022-01-31T20:59:20.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.413205+0000) 2022-01-31T20:59:20.742 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[41853]: debug 2022-01-31T20:59:20.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.617471+0000) 2022-01-31T20:59:20.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:20 smithi171 conmon[51620]: debug 2022-01-31T20:59:20.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:20.490292+0000) 2022-01-31T20:59:21.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:21 smithi167 conmon[54076]: debug 2022-01-31T20:59:21.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:21.384630+0000) 2022-01-31T20:59:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:21 smithi167 conmon[60316]: debug 2022-01-31T20:59:21.135+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:21.135437+0000) 2022-01-31T20:59:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:21 smithi171 conmon[46715]: debug 2022-01-31T20:59:21.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:21.413361+0000) 2022-01-31T20:59:21.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:21 smithi171 conmon[51620]: debug 2022-01-31T20:59:21.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:21.490420+0000) 2022-01-31T20:59:21.701 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:21 smithi167 conmon[49112]: debug 2022-01-31T20:59:21.456+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:21.456727+0000) 2022-01-31T20:59:22.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:21 smithi171 conmon[41853]: debug 2022-01-31T20:59:21.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:21.617627+0000) 2022-01-31T20:59:22.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:22 smithi167 conmon[54076]: debug 2022-01-31T20:59:22.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:22.384811+0000) 2022-01-31T20:59:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:22 smithi167 conmon[60316]: debug 2022-01-31T20:59:22.135+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:22.135650+0000) 2022-01-31T20:59:22.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:22 smithi171 conmon[46715]: debug 2022-01-31T20:59:22.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:22.413517+0000) 2022-01-31T20:59:22.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:22 smithi171 conmon[51620]: debug 2022-01-31T20:59:22.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:22.490583+0000) 2022-01-31T20:59:22.680 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:22 smithi167 conmon[49112]: debug 2022-01-31T20:59:22.456+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:22.456848+0000) 2022-01-31T20:59:23.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:22 smithi171 conmon[41853]: debug 2022-01-31T20:59:22.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:22.617812+0000) 2022-01-31T20:59:23.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:23 smithi167 conmon[54076]: debug 2022-01-31T20:59:23.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:23.384998+0000) 2022-01-31T20:59:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:23 smithi167 conmon[60316]: debug 2022-01-31T20:59:23.135+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:23.135864+0000) 2022-01-31T20:59:23.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:23 smithi171 conmon[46715]: debug 2022-01-31T20:59:23.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:23.413657+0000) 2022-01-31T20:59:23.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:23 smithi171 conmon[51620]: debug 2022-01-31T20:59:23.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:23.490786+0000) 2022-01-31T20:59:23.680 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:23 smithi167 conmon[49112]: debug 2022-01-31T20:59:23.456+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:23.457030+0000) 2022-01-31T20:59:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:23 smithi171 conmon[35325]: debug 2022-01-31T20:59:23.604+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:59:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:23 smithi171 conmon[41853]: debug 2022-01-31T20:59:23.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:23.617973+0000) 2022-01-31T20:59:24.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:24 smithi167 conmon[54076]: debug 2022-01-31T20:59:24.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:24.385157+0000) 2022-01-31T20:59:24.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:24 smithi167 conmon[60316]: debug 2022-01-31T20:59:24.135+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:24.136092+0000) 2022-01-31T20:59:24.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:24 smithi171 conmon[46715]: debug 2022-01-31T20:59:24.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:24.413815+0000) 2022-01-31T20:59:24.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:24 smithi171 conmon[51620]: debug 2022-01-31T20:59:24.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:24.490901+0000) 2022-01-31T20:59:24.680 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:24 smithi167 conmon[49112]: debug 2022-01-31T20:59:24.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:24.457212+0000) 2022-01-31T20:59:25.039 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:24 smithi171 conmon[41853]: debug 2022-01-31T20:59:24.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:24.618161+0000) 2022-01-31T20:59:25.040 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[41853]: debug 2022-01-31T20:59:25.040+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.041526+0000) 2022-01-31T20:59:25.356 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:25 smithi167 conmon[49112]: debug 2022-01-31T20:59:25.041+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.041618+0000) 2022-01-31T20:59:25.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[35325]: debug 2022-01-31T20:59:25.048+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94253 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:25.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:25 smithi167 conmon[54076]: debug 2022-01-31T20:59:25.042+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.042930+0000) 2022-01-31T20:59:25.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[46715]: debug 2022-01-31T20:59:25.041+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.041844+0000) 2022-01-31T20:59:25.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:25 smithi167 conmon[60316]: debug 2022-01-31T20:59:25.041+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.041748+0000) 2022-01-31T20:59:25.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:25 smithi167 conmon[60316]: debug 2022-01-31T20:59:25.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.136232+0000) 2022-01-31T20:59:25.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[51620]: debug 2022-01-31T20:59:25.040+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.040976+0000) 2022-01-31T20:59:25.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:25 smithi167 conmon[49112]: debug 2022-01-31T20:59:25.456+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.457418+0000) 2022-01-31T20:59:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:25 smithi167 conmon[54076]: debug 2022-01-31T20:59:25.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.385281+0000) 2022-01-31T20:59:25.742 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[46715]: debug 2022-01-31T20:59:25.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.414027+0000) 2022-01-31T20:59:25.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[51620]: debug 2022-01-31T20:59:25.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.491029+0000) 2022-01-31T20:59:25.744 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:25 smithi171 conmon[41853]: debug 2022-01-31T20:59:25.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:25.618347+0000) 2022-01-31T20:59:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:26 smithi167 conmon[60316]: debug 2022-01-31T20:59:26.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:26.136374+0000) 2022-01-31T20:59:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:26 smithi167 conmon[54076]: debug 2022-01-31T20:59:26.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:26.385427+0000) 2022-01-31T20:59:26.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:26 smithi171 conmon[46715]: debug 2022-01-31T20:59:26.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:26.414182+0000) 2022-01-31T20:59:26.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:26 smithi171 conmon[51620]: debug 2022-01-31T20:59:26.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:26.491170+0000) 2022-01-31T20:59:26.684 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:26 smithi167 conmon[49112]: debug 2022-01-31T20:59:26.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:26.457599+0000) 2022-01-31T20:59:27.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:26 smithi171 conmon[41853]: debug 2022-01-31T20:59:26.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:26.618533+0000) 2022-01-31T20:59:27.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:27 smithi167 conmon[54076]: debug 2022-01-31T20:59:27.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:27.385601+0000) 2022-01-31T20:59:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:27 smithi167 conmon[60316]: debug 2022-01-31T20:59:27.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:27.136534+0000) 2022-01-31T20:59:27.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:27 smithi171 conmon[46715]: debug 2022-01-31T20:59:27.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:27.414330+0000) 2022-01-31T20:59:27.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:27 smithi171 conmon[51620]: debug 2022-01-31T20:59:27.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:27.491377+0000) 2022-01-31T20:59:27.686 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:27 smithi167 conmon[49112]: debug 2022-01-31T20:59:27.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:27.457767+0000) 2022-01-31T20:59:28.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:27 smithi171 conmon[41853]: debug 2022-01-31T20:59:27.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:27.618736+0000) 2022-01-31T20:59:28.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:28 smithi167 conmon[54076]: debug 2022-01-31T20:59:28.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:28.385795+0000) 2022-01-31T20:59:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:28 smithi167 conmon[60316]: debug 2022-01-31T20:59:28.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:28.136698+0000) 2022-01-31T20:59:28.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:28 smithi171 conmon[46715]: debug 2022-01-31T20:59:28.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:28.414496+0000) 2022-01-31T20:59:28.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:28 smithi171 conmon[51620]: debug 2022-01-31T20:59:28.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:28.491570+0000) 2022-01-31T20:59:28.686 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:28 smithi167 conmon[49112]: debug 2022-01-31T20:59:28.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:28.457951+0000) 2022-01-31T20:59:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:28 smithi171 conmon[41853]: debug 2022-01-31T20:59:28.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:28.618905+0000) 2022-01-31T20:59:29.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:29 smithi167 conmon[54076]: debug 2022-01-31T20:59:29.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:29.385959+0000) 2022-01-31T20:59:29.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:29 smithi167 conmon[60316]: debug 2022-01-31T20:59:29.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:29.136859+0000) 2022-01-31T20:59:29.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:29 smithi171 conmon[46715]: debug 2022-01-31T20:59:29.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:29.414671+0000) 2022-01-31T20:59:29.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:29 smithi171 conmon[51620]: debug 2022-01-31T20:59:29.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:29.491749+0000) 2022-01-31T20:59:29.686 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:29 smithi167 conmon[49112]: debug 2022-01-31T20:59:29.458+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:29.458140+0000) 2022-01-31T20:59:30.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:29 smithi171 conmon[41853]: debug 2022-01-31T20:59:29.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:29.619091+0000) 2022-01-31T20:59:30.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[51620]: debug 2022-01-31T20:59:30.050+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.050965+0000) 2022-01-31T20:59:30.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[35325]: debug 2022-01-31T20:59:30.060+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94363 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:30.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[41853]: debug 2022-01-31T20:59:30.051+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.051846+0000) 2022-01-31T20:59:30.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[46715]: debug 2022-01-31T20:59:30.050+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.051637+0000) 2022-01-31T20:59:30.357 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:30 smithi167 conmon[49112]: debug 2022-01-31T20:59:30.051+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.051951+0000) 2022-01-31T20:59:30.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:30 smithi167 conmon[54076]: debug 2022-01-31T20:59:30.052+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.052824+0000) 2022-01-31T20:59:30.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:30 smithi167 conmon[60316]: debug 2022-01-31T20:59:30.051+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.052185+0000) 2022-01-31T20:59:30.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:30 smithi167 conmon[60316]: debug 2022-01-31T20:59:30.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.137060+0000) 2022-01-31T20:59:30.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:30 smithi167 conmon[49112]: debug 2022-01-31T20:59:30.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.458323+0000) 2022-01-31T20:59:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:30 smithi167 conmon[54076]: debug 2022-01-31T20:59:30.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.386170+0000) 2022-01-31T20:59:30.743 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[41853]: debug 2022-01-31T20:59:30.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.619280+0000) 2022-01-31T20:59:30.744 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[46715]: debug 2022-01-31T20:59:30.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.414836+0000) 2022-01-31T20:59:30.744 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:30 smithi171 conmon[51620]: debug 2022-01-31T20:59:30.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:30.491955+0000) 2022-01-31T20:59:31.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:31 smithi167 conmon[54076]: debug 2022-01-31T20:59:31.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:31.386290+0000) 2022-01-31T20:59:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:31 smithi167 conmon[60316]: debug 2022-01-31T20:59:31.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:31.137155+0000) 2022-01-31T20:59:31.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:31 smithi171 conmon[51620]: debug 2022-01-31T20:59:31.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:31.492112+0000) 2022-01-31T20:59:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:31 smithi171 conmon[46715]: debug 2022-01-31T20:59:31.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:31.414956+0000) 2022-01-31T20:59:31.690 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:31 smithi167 conmon[49112]: debug 2022-01-31T20:59:31.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:31.458464+0000) 2022-01-31T20:59:32.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:31 smithi171 conmon[41853]: debug 2022-01-31T20:59:31.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:31.619451+0000) 2022-01-31T20:59:32.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:32 smithi167 conmon[54076]: debug 2022-01-31T20:59:32.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:32.386454+0000) 2022-01-31T20:59:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:32 smithi167 conmon[60316]: debug 2022-01-31T20:59:32.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:32.137350+0000) 2022-01-31T20:59:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:32 smithi171 conmon[46715]: debug 2022-01-31T20:59:32.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:32.415088+0000) 2022-01-31T20:59:32.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:32 smithi171 conmon[51620]: debug 2022-01-31T20:59:32.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:32.492301+0000) 2022-01-31T20:59:32.691 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:32 smithi167 conmon[49112]: debug 2022-01-31T20:59:32.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:32.458667+0000) 2022-01-31T20:59:33.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:32 smithi171 conmon[41853]: debug 2022-01-31T20:59:32.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:32.619661+0000) 2022-01-31T20:59:33.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:33 smithi167 conmon[54076]: debug 2022-01-31T20:59:33.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:33.386667+0000) 2022-01-31T20:59:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:33 smithi167 conmon[60316]: debug 2022-01-31T20:59:33.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:33.137544+0000) 2022-01-31T20:59:33.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:33 smithi171 conmon[46715]: debug 2022-01-31T20:59:33.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:33.415299+0000) 2022-01-31T20:59:33.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:33 smithi171 conmon[51620]: debug 2022-01-31T20:59:33.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:33.492503+0000) 2022-01-31T20:59:33.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:33 smithi167 conmon[49112]: debug 2022-01-31T20:59:33.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:33.458805+0000) 2022-01-31T20:59:34.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:33 smithi171 conmon[41853]: debug 2022-01-31T20:59:33.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:33.619849+0000) 2022-01-31T20:59:34.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:34 smithi167 conmon[54076]: debug 2022-01-31T20:59:34.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:34.386802+0000) 2022-01-31T20:59:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:34 smithi167 conmon[60316]: debug 2022-01-31T20:59:34.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:34.137693+0000) 2022-01-31T20:59:34.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:34 smithi171 conmon[46715]: debug 2022-01-31T20:59:34.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:34.415478+0000) 2022-01-31T20:59:34.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:34 smithi171 conmon[51620]: debug 2022-01-31T20:59:34.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:34.492626+0000) 2022-01-31T20:59:34.692 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:34 smithi167 conmon[49112]: debug 2022-01-31T20:59:34.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:34.459014+0000) 2022-01-31T20:59:35.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:34 smithi171 conmon[41853]: debug 2022-01-31T20:59:34.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:34.619996+0000) 2022-01-31T20:59:35.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[35325]: debug 2022-01-31T20:59:35.074+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94473 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:35.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[41853]: debug 2022-01-31T20:59:35.061+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.063772+0000) 2022-01-31T20:59:35.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[46715]: debug 2022-01-31T20:59:35.061+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.064345+0000) 2022-01-31T20:59:35.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[51620]: debug 2022-01-31T20:59:35.061+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.062988+0000) 2022-01-31T20:59:35.359 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:35 smithi167 conmon[49112]: debug 2022-01-31T20:59:35.062+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.063626+0000) 2022-01-31T20:59:35.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:35 smithi167 conmon[54076]: debug 2022-01-31T20:59:35.061+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.063444+0000) 2022-01-31T20:59:35.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:35 smithi167 conmon[60316]: debug 2022-01-31T20:59:35.069+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.070494+0000) 2022-01-31T20:59:35.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:35 smithi167 conmon[60316]: debug 2022-01-31T20:59:35.136+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.137803+0000) 2022-01-31T20:59:35.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:35 smithi167 conmon[49112]: debug 2022-01-31T20:59:35.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.459192+0000) 2022-01-31T20:59:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:35 smithi167 conmon[54076]: debug 2022-01-31T20:59:35.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.386935+0000) 2022-01-31T20:59:35.744 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[41853]: debug 2022-01-31T20:59:35.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.620140+0000) 2022-01-31T20:59:35.744 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[46715]: debug 2022-01-31T20:59:35.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.415603+0000) 2022-01-31T20:59:35.745 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:35 smithi171 conmon[51620]: debug 2022-01-31T20:59:35.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:35.492836+0000) 2022-01-31T20:59:36.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:36 smithi167 conmon[54076]: debug 2022-01-31T20:59:36.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:36.387095+0000) 2022-01-31T20:59:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:36 smithi167 conmon[60316]: debug 2022-01-31T20:59:36.137+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:36.137944+0000) 2022-01-31T20:59:36.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:36 smithi171 conmon[46715]: debug 2022-01-31T20:59:36.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:36.415712+0000) 2022-01-31T20:59:36.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:36 smithi171 conmon[51620]: debug 2022-01-31T20:59:36.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:36.492985+0000) 2022-01-31T20:59:36.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:36 smithi167 conmon[49112]: debug 2022-01-31T20:59:36.457+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:36.459339+0000) 2022-01-31T20:59:37.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:36 smithi171 conmon[41853]: debug 2022-01-31T20:59:36.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:36.620297+0000) 2022-01-31T20:59:37.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:37 smithi167 conmon[54076]: debug 2022-01-31T20:59:37.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:37.387304+0000) 2022-01-31T20:59:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:37 smithi167 conmon[60316]: debug 2022-01-31T20:59:37.137+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:37.138097+0000) 2022-01-31T20:59:37.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:37 smithi171 conmon[46715]: debug 2022-01-31T20:59:37.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:37.415839+0000) 2022-01-31T20:59:37.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:37 smithi171 conmon[51620]: debug 2022-01-31T20:59:37.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:37.493170+0000) 2022-01-31T20:59:37.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:37 smithi167 conmon[49112]: debug 2022-01-31T20:59:37.458+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:37.459541+0000) 2022-01-31T20:59:38.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:37 smithi171 conmon[41853]: debug 2022-01-31T20:59:37.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:37.620500+0000) 2022-01-31T20:59:38.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:38 smithi167 conmon[54076]: debug 2022-01-31T20:59:38.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:38.387486+0000) 2022-01-31T20:59:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:38 smithi167 conmon[60316]: debug 2022-01-31T20:59:38.137+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:38.138279+0000) 2022-01-31T20:59:38.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:38 smithi171 conmon[46715]: debug 2022-01-31T20:59:38.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:38.415937+0000) 2022-01-31T20:59:38.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:38 smithi171 conmon[51620]: debug 2022-01-31T20:59:38.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:38.493349+0000) 2022-01-31T20:59:38.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:38 smithi167 conmon[49112]: debug 2022-01-31T20:59:38.458+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:38.459703+0000) 2022-01-31T20:59:38.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:38 smithi171 conmon[41853]: debug 2022-01-31T20:59:38.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:38.620704+0000) 2022-01-31T20:59:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:38 smithi171 conmon[35325]: debug 2022-01-31T20:59:38.604+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:59:39.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:39 smithi167 conmon[54076]: debug 2022-01-31T20:59:39.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:39.387690+0000) 2022-01-31T20:59:39.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:39 smithi167 conmon[60316]: debug 2022-01-31T20:59:39.137+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:39.138452+0000) 2022-01-31T20:59:39.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:39 smithi171 conmon[46715]: debug 2022-01-31T20:59:39.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:39.416112+0000) 2022-01-31T20:59:39.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:39 smithi171 conmon[51620]: debug 2022-01-31T20:59:39.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:39.493530+0000) 2022-01-31T20:59:39.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:39 smithi167 conmon[49112]: debug 2022-01-31T20:59:39.458+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:39.459886+0000) 2022-01-31T20:59:40.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:39 smithi171 conmon[41853]: debug 2022-01-31T20:59:39.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:39.620901+0000) 2022-01-31T20:59:40.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[41853]: debug 2022-01-31T20:59:40.075+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.077234+0000) 2022-01-31T20:59:40.077 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[51620]: debug 2022-01-31T20:59:40.076+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.077838+0000) 2022-01-31T20:59:40.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[35325]: debug 2022-01-31T20:59:40.085+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94583 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[46715]: debug 2022-01-31T20:59:40.077+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.079030+0000) 2022-01-31T20:59:40.358 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:40 smithi167 conmon[49112]: debug 2022-01-31T20:59:40.077+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.079366+0000) 2022-01-31T20:59:40.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:40 smithi167 conmon[54076]: debug 2022-01-31T20:59:40.077+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.078946+0000) 2022-01-31T20:59:40.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:40 smithi167 conmon[60316]: debug 2022-01-31T20:59:40.076+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.078338+0000) 2022-01-31T20:59:40.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:40 smithi167 conmon[60316]: debug 2022-01-31T20:59:40.137+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.138649+0000) 2022-01-31T20:59:40.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:40 smithi167 conmon[49112]: debug 2022-01-31T20:59:40.458+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.460088+0000) 2022-01-31T20:59:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:40 smithi167 conmon[54076]: debug 2022-01-31T20:59:40.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.387840+0000) 2022-01-31T20:59:40.745 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[41853]: debug 2022-01-31T20:59:40.620+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.621077+0000) 2022-01-31T20:59:40.745 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[46715]: debug 2022-01-31T20:59:40.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.416291+0000) 2022-01-31T20:59:40.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:40 smithi171 conmon[51620]: debug 2022-01-31T20:59:40.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:40.493711+0000) 2022-01-31T20:59:41.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:41 smithi167 conmon[54076]: debug 2022-01-31T20:59:41.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:41.387946+0000) 2022-01-31T20:59:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:41 smithi167 conmon[60316]: debug 2022-01-31T20:59:41.137+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:41.138736+0000) 2022-01-31T20:59:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:41 smithi171 conmon[46715]: debug 2022-01-31T20:59:41.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:41.416436+0000) 2022-01-31T20:59:41.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:41 smithi171 conmon[51620]: debug 2022-01-31T20:59:41.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:41.493828+0000) 2022-01-31T20:59:41.702 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:41 smithi167 conmon[49112]: debug 2022-01-31T20:59:41.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:41.460240+0000) 2022-01-31T20:59:42.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:41 smithi171 conmon[41853]: debug 2022-01-31T20:59:41.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:41.621233+0000) 2022-01-31T20:59:42.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:42 smithi167 conmon[54076]: debug 2022-01-31T20:59:42.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:42.388098+0000) 2022-01-31T20:59:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:42 smithi167 conmon[60316]: debug 2022-01-31T20:59:42.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:42.138900+0000) 2022-01-31T20:59:42.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:42 smithi171 conmon[46715]: debug 2022-01-31T20:59:42.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:42.416622+0000) 2022-01-31T20:59:42.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:42 smithi171 conmon[51620]: debug 2022-01-31T20:59:42.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:42.493989+0000) 2022-01-31T20:59:42.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:42 smithi167 conmon[49112]: debug 2022-01-31T20:59:42.458+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:42.460449+0000) 2022-01-31T20:59:43.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:42 smithi171 conmon[41853]: debug 2022-01-31T20:59:42.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:42.621388+0000) 2022-01-31T20:59:43.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:43 smithi167 conmon[54076]: debug 2022-01-31T20:59:43.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:43.388253+0000) 2022-01-31T20:59:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:43 smithi167 conmon[60316]: debug 2022-01-31T20:59:43.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:43.139085+0000) 2022-01-31T20:59:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:43 smithi171 conmon[46715]: debug 2022-01-31T20:59:43.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:43.416754+0000) 2022-01-31T20:59:43.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:43 smithi171 conmon[51620]: debug 2022-01-31T20:59:43.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:43.494144+0000) 2022-01-31T20:59:43.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:43 smithi167 conmon[49112]: debug 2022-01-31T20:59:43.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:43.460576+0000) 2022-01-31T20:59:44.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:43 smithi171 conmon[41853]: debug 2022-01-31T20:59:43.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:43.621559+0000) 2022-01-31T20:59:44.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:44 smithi167 conmon[54076]: debug 2022-01-31T20:59:44.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:44.388479+0000) 2022-01-31T20:59:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:44 smithi167 conmon[60316]: debug 2022-01-31T20:59:44.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:44.139264+0000) 2022-01-31T20:59:44.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:44 smithi171 conmon[46715]: debug 2022-01-31T20:59:44.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:44.416955+0000) 2022-01-31T20:59:44.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:44 smithi171 conmon[51620]: debug 2022-01-31T20:59:44.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:44.494255+0000) 2022-01-31T20:59:44.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:44 smithi167 conmon[49112]: debug 2022-01-31T20:59:44.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:44.460760+0000) 2022-01-31T20:59:45.087 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:44 smithi171 conmon[41853]: debug 2022-01-31T20:59:44.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:44.621774+0000) 2022-01-31T20:59:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[46715]: debug 2022-01-31T20:59:45.088+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.090281+0000) 2022-01-31T20:59:45.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[35325]: debug 2022-01-31T20:59:45.097+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94693 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:45.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[41853]: debug 2022-01-31T20:59:45.088+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.090356+0000) 2022-01-31T20:59:45.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[51620]: debug 2022-01-31T20:59:45.088+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.089990+0000) 2022-01-31T20:59:45.359 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:45 smithi167 conmon[49112]: debug 2022-01-31T20:59:45.088+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.089734+0000) 2022-01-31T20:59:45.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:45 smithi167 conmon[54076]: debug 2022-01-31T20:59:45.089+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.090699+0000) 2022-01-31T20:59:45.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:45 smithi167 conmon[60316]: debug 2022-01-31T20:59:45.088+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.089819+0000) 2022-01-31T20:59:45.360 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:45 smithi167 conmon[60316]: debug 2022-01-31T20:59:45.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.139428+0000) 2022-01-31T20:59:45.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:45 smithi167 conmon[49112]: debug 2022-01-31T20:59:45.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.460933+0000) 2022-01-31T20:59:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:45 smithi167 conmon[54076]: debug 2022-01-31T20:59:45.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.388594+0000) 2022-01-31T20:59:45.745 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[41853]: debug 2022-01-31T20:59:45.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.621944+0000) 2022-01-31T20:59:45.746 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[46715]: debug 2022-01-31T20:59:45.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.417114+0000) 2022-01-31T20:59:45.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:45 smithi171 conmon[51620]: debug 2022-01-31T20:59:45.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:45.494438+0000) 2022-01-31T20:59:46.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:46 smithi167 conmon[54076]: debug 2022-01-31T20:59:46.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:46.388754+0000) 2022-01-31T20:59:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:46 smithi167 conmon[60316]: debug 2022-01-31T20:59:46.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:46.139616+0000) 2022-01-31T20:59:46.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:46 smithi171 conmon[46715]: debug 2022-01-31T20:59:46.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:46.417283+0000) 2022-01-31T20:59:46.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:46 smithi171 conmon[51620]: debug 2022-01-31T20:59:46.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:46.494587+0000) 2022-01-31T20:59:46.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:46 smithi167 conmon[49112]: debug 2022-01-31T20:59:46.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:46.461046+0000) 2022-01-31T20:59:47.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:46 smithi171 conmon[41853]: debug 2022-01-31T20:59:46.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:46.622121+0000) 2022-01-31T20:59:47.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:47 smithi167 conmon[54076]: debug 2022-01-31T20:59:47.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:47.388926+0000) 2022-01-31T20:59:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:47 smithi167 conmon[60316]: debug 2022-01-31T20:59:47.139+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:47.139801+0000) 2022-01-31T20:59:47.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:47 smithi171 conmon[46715]: debug 2022-01-31T20:59:47.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:47.417469+0000) 2022-01-31T20:59:47.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:47 smithi171 conmon[51620]: debug 2022-01-31T20:59:47.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:47.494758+0000) 2022-01-31T20:59:47.705 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:47 smithi167 conmon[49112]: debug 2022-01-31T20:59:47.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:47.461234+0000) 2022-01-31T20:59:48.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:47 smithi171 conmon[41853]: debug 2022-01-31T20:59:47.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:47.622329+0000) 2022-01-31T20:59:48.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:48 smithi167 conmon[54076]: debug 2022-01-31T20:59:48.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:48.389118+0000) 2022-01-31T20:59:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:48 smithi167 conmon[60316]: debug 2022-01-31T20:59:48.139+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:48.139960+0000) 2022-01-31T20:59:48.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:48 smithi171 conmon[51620]: debug 2022-01-31T20:59:48.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:48.494942+0000) 2022-01-31T20:59:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:48 smithi171 conmon[46715]: debug 2022-01-31T20:59:48.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:48.417683+0000) 2022-01-31T20:59:48.705 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:48 smithi167 conmon[49112]: debug 2022-01-31T20:59:48.459+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:48.461427+0000) 2022-01-31T20:59:49.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:48 smithi171 conmon[41853]: debug 2022-01-31T20:59:48.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:48.622511+0000) 2022-01-31T20:59:49.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:49 smithi167 conmon[54076]: debug 2022-01-31T20:59:49.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:49.389323+0000) 2022-01-31T20:59:49.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:49 smithi167 conmon[60316]: debug 2022-01-31T20:59:49.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:49.140088+0000) 2022-01-31T20:59:49.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:49 smithi171 conmon[51620]: debug 2022-01-31T20:59:49.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:49.495138+0000) 2022-01-31T20:59:49.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:49 smithi171 conmon[46715]: debug 2022-01-31T20:59:49.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:49.417836+0000) 2022-01-31T20:59:49.705 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:49 smithi167 conmon[49112]: debug 2022-01-31T20:59:49.460+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:49.461613+0000) 2022-01-31T20:59:50.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:49 smithi171 conmon[41853]: debug 2022-01-31T20:59:49.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:49.622661+0000) 2022-01-31T20:59:50.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[35325]: debug 2022-01-31T20:59:50.111+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94803 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:50.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[41853]: debug 2022-01-31T20:59:50.101+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.102016+0000) 2022-01-31T20:59:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[46715]: debug 2022-01-31T20:59:50.100+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.101655+0000) 2022-01-31T20:59:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[51620]: debug 2022-01-31T20:59:50.101+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.102286+0000) 2022-01-31T20:59:50.360 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:50 smithi167 conmon[49112]: debug 2022-01-31T20:59:50.101+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.102973+0000) 2022-01-31T20:59:50.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:50 smithi167 conmon[54076]: debug 2022-01-31T20:59:50.101+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.102817+0000) 2022-01-31T20:59:50.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:50 smithi167 conmon[60316]: debug 2022-01-31T20:59:50.101+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.103064+0000) 2022-01-31T20:59:50.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:50 smithi167 conmon[60316]: debug 2022-01-31T20:59:50.138+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.140291+0000) 2022-01-31T20:59:50.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:50 smithi167 conmon[54076]: debug 2022-01-31T20:59:50.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.389494+0000) 2022-01-31T20:59:50.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:50 smithi167 conmon[49112]: debug 2022-01-31T20:59:50.460+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.461761+0000) 2022-01-31T20:59:50.746 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[46715]: debug 2022-01-31T20:59:50.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.417963+0000) 2022-01-31T20:59:50.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[51620]: debug 2022-01-31T20:59:50.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.495316+0000) 2022-01-31T20:59:50.747 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:50 smithi171 conmon[41853]: debug 2022-01-31T20:59:50.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:50.622808+0000) 2022-01-31T20:59:51.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:51 smithi167 conmon[60316]: debug 2022-01-31T20:59:51.139+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:51.140434+0000) 2022-01-31T20:59:51.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:51 smithi167 conmon[54076]: debug 2022-01-31T20:59:51.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:51.389661+0000) 2022-01-31T20:59:51.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:51 smithi171 conmon[46715]: debug 2022-01-31T20:59:51.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:51.418062+0000) 2022-01-31T20:59:51.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:51 smithi171 conmon[51620]: debug 2022-01-31T20:59:51.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:51.495475+0000) 2022-01-31T20:59:51.706 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:51 smithi167 conmon[49112]: debug 2022-01-31T20:59:51.460+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:51.461892+0000) 2022-01-31T20:59:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:51 smithi171 conmon[41853]: debug 2022-01-31T20:59:51.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:51.622934+0000) 2022-01-31T20:59:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:52 smithi167 conmon[54076]: debug 2022-01-31T20:59:52.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:52.389823+0000) 2022-01-31T20:59:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:52 smithi167 conmon[60316]: debug 2022-01-31T20:59:52.139+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:52.140568+0000) 2022-01-31T20:59:52.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:52 smithi171 conmon[46715]: debug 2022-01-31T20:59:52.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:52.418244+0000) 2022-01-31T20:59:52.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:52 smithi171 conmon[51620]: debug 2022-01-31T20:59:52.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:52.495628+0000) 2022-01-31T20:59:52.706 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:52 smithi167 conmon[49112]: debug 2022-01-31T20:59:52.460+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:52.462100+0000) 2022-01-31T20:59:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:52 smithi171 conmon[41853]: debug 2022-01-31T20:59:52.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:52.623134+0000) 2022-01-31T20:59:53.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:53 smithi167 conmon[54076]: debug 2022-01-31T20:59:53.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:53.390019+0000) 2022-01-31T20:59:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:53 smithi167 conmon[60316]: debug 2022-01-31T20:59:53.140+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:53.140707+0000) 2022-01-31T20:59:53.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:53 smithi171 conmon[46715]: debug 2022-01-31T20:59:53.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:53.418438+0000) 2022-01-31T20:59:53.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:53 smithi171 conmon[51620]: debug 2022-01-31T20:59:53.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:53.495841+0000) 2022-01-31T20:59:53.706 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:53 smithi167 conmon[49112]: debug 2022-01-31T20:59:53.461+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:53.462259+0000) 2022-01-31T20:59:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:53 smithi171 conmon[35325]: debug 2022-01-31T20:59:53.605+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T20:59:53.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:53 smithi171 conmon[41853]: debug 2022-01-31T20:59:53.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:53.623322+0000) 2022-01-31T20:59:54.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:54 smithi167 conmon[54076]: debug 2022-01-31T20:59:54.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:54.390209+0000) 2022-01-31T20:59:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:54 smithi167 conmon[60316]: debug 2022-01-31T20:59:54.140+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:54.140868+0000) 2022-01-31T20:59:54.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:54 smithi171 conmon[46715]: debug 2022-01-31T20:59:54.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:54.418591+0000) 2022-01-31T20:59:54.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:54 smithi171 conmon[51620]: debug 2022-01-31T20:59:54.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:54.496025+0000) 2022-01-31T20:59:54.706 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:54 smithi167 conmon[49112]: debug 2022-01-31T20:59:54.460+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:54.462405+0000) 2022-01-31T20:59:55.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:54 smithi171 conmon[41853]: debug 2022-01-31T20:59:54.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:54.623476+0000) 2022-01-31T20:59:55.388 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:55 smithi167 conmon[49112]: debug 2022-01-31T20:59:55.114+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.116069+0000) 2022-01-31T20:59:55.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:55 smithi167 conmon[54076]: debug 2022-01-31T20:59:55.114+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.115533+0000) 2022-01-31T20:59:55.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:55 smithi167 conmon[60316]: debug 2022-01-31T20:59:55.114+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.116386+0000) 2022-01-31T20:59:55.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:55 smithi167 conmon[60316]: debug 2022-01-31T20:59:55.140+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.141063+0000) 2022-01-31T20:59:55.417 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[35325]: debug 2022-01-31T20:59:55.123+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 94913 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T20:59:55.417 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[41853]: debug 2022-01-31T20:59:55.113+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.114153+0000) 2022-01-31T20:59:55.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[46715]: debug 2022-01-31T20:59:55.114+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.115074+0000) 2022-01-31T20:59:55.418 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[51620]: debug 2022-01-31T20:59:55.114+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.115421+0000) 2022-01-31T20:59:55.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:55 smithi167 conmon[49112]: debug 2022-01-31T20:59:55.461+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.462546+0000) 2022-01-31T20:59:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:55 smithi167 conmon[54076]: debug 2022-01-31T20:59:55.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.390402+0000) 2022-01-31T20:59:55.747 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[46715]: debug 2022-01-31T20:59:55.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.418745+0000) 2022-01-31T20:59:55.748 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[41853]: debug 2022-01-31T20:59:55.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.623675+0000) 2022-01-31T20:59:55.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:55 smithi171 conmon[51620]: debug 2022-01-31T20:59:55.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:55.496196+0000) 2022-01-31T20:59:56.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:56 smithi167 conmon[54076]: debug 2022-01-31T20:59:56.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:56.390556+0000) 2022-01-31T20:59:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:56 smithi167 conmon[60316]: debug 2022-01-31T20:59:56.140+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:56.141205+0000) 2022-01-31T20:59:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:56 smithi171 conmon[46715]: debug 2022-01-31T20:59:56.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:56.418930+0000) 2022-01-31T20:59:56.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:56 smithi171 conmon[51620]: debug 2022-01-31T20:59:56.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:56.496411+0000) 2022-01-31T20:59:56.706 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:56 smithi167 conmon[49112]: debug 2022-01-31T20:59:56.462+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:56.462660+0000) 2022-01-31T20:59:57.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:56 smithi171 conmon[41853]: debug 2022-01-31T20:59:56.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:56.623838+0000) 2022-01-31T20:59:57.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:57 smithi167 conmon[54076]: debug 2022-01-31T20:59:57.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:57.390740+0000) 2022-01-31T20:59:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:57 smithi167 conmon[60316]: debug 2022-01-31T20:59:57.140+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:57.141330+0000) 2022-01-31T20:59:57.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:57 smithi171 conmon[46715]: debug 2022-01-31T20:59:57.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:57.419059+0000) 2022-01-31T20:59:57.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:57 smithi171 conmon[51620]: debug 2022-01-31T20:59:57.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:57.496608+0000) 2022-01-31T20:59:57.707 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:57 smithi167 conmon[49112]: debug 2022-01-31T20:59:57.462+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:57.462794+0000) 2022-01-31T20:59:58.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:57 smithi171 conmon[41853]: debug 2022-01-31T20:59:57.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:57.623981+0000) 2022-01-31T20:59:58.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:58 smithi167 conmon[54076]: debug 2022-01-31T20:59:58.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:58.390957+0000) 2022-01-31T20:59:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:58 smithi167 conmon[60316]: debug 2022-01-31T20:59:58.140+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:58.141485+0000) 2022-01-31T20:59:58.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:58 smithi171 conmon[46715]: debug 2022-01-31T20:59:58.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:58.419266+0000) 2022-01-31T20:59:58.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:58 smithi171 conmon[51620]: debug 2022-01-31T20:59:58.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:58.496783+0000) 2022-01-31T20:59:58.707 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:58 smithi167 conmon[49112]: debug 2022-01-31T20:59:58.462+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:58.462963+0000) 2022-01-31T20:59:59.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:58 smithi171 conmon[41853]: debug 2022-01-31T20:59:58.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:58.624125+0000) 2022-01-31T20:59:59.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 20:59:59 smithi167 conmon[54076]: debug 2022-01-31T20:59:59.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:59.391141+0000) 2022-01-31T20:59:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 20:59:59 smithi167 conmon[60316]: debug 2022-01-31T20:59:59.141+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:59.141651+0000) 2022-01-31T20:59:59.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 20:59:59 smithi171 conmon[46715]: debug 2022-01-31T20:59:59.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:59.419443+0000) 2022-01-31T20:59:59.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 20:59:59 smithi171 conmon[51620]: debug 2022-01-31T20:59:59.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:59.496956+0000) 2022-01-31T20:59:59.707 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 20:59:59 smithi167 conmon[49112]: debug 2022-01-31T20:59:59.462+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:59.463146+0000) 2022-01-31T21:00:00.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 20:59:59 smithi171 conmon[41853]: debug 2022-01-31T20:59:59.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T19:59:59.624331+0000) 2022-01-31T21:00:00.389 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:00 smithi167 conmon[49112]: debug 2022-01-31T21:00:00.127+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.128069+0000) 2022-01-31T21:00:00.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:00 smithi167 conmon[54076]: debug 2022-01-31T21:00:00.127+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.128306+0000) 2022-01-31T21:00:00.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:00 smithi167 conmon[60316]: debug 2022-01-31T21:00:00.127+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.127638+0000) 2022-01-31T21:00:00.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:00 smithi167 conmon[60316]: debug 2022-01-31T21:00:00.142+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.142773+0000) 2022-01-31T21:00:00.417 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[35325]: debug 2022-01-31T21:00:00.136+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95025 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:00.418 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[41853]: debug 2022-01-31T21:00:00.125+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.126421+0000) 2022-01-31T21:00:00.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[46715]: debug 2022-01-31T21:00:00.126+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.127460+0000) 2022-01-31T21:00:00.419 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[51620]: debug 2022-01-31T21:00:00.127+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.127885+0000) 2022-01-31T21:00:00.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:00 smithi167 conmon[49112]: debug 2022-01-31T21:00:00.462+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.463296+0000) 2022-01-31T21:00:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:00 smithi167 conmon[54076]: debug 2022-01-31T21:00:00.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.391278+0000) 2022-01-31T21:00:00.748 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[41853]: debug 2022-01-31T21:00:00.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.624464+0000) 2022-01-31T21:00:00.749 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[46715]: debug 2022-01-31T21:00:00.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.419650+0000) 2022-01-31T21:00:00.749 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:00 smithi171 conmon[51620]: debug 2022-01-31T21:00:00.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:00.497087+0000) 2022-01-31T21:00:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:01 smithi167 conmon[54076]: debug 2022-01-31T21:00:01.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:01.391436+0000) 2022-01-31T21:00:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:01 smithi167 conmon[60316]: debug 2022-01-31T21:00:01.142+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:01.142867+0000) 2022-01-31T21:00:01.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:01 smithi171 conmon[46715]: debug 2022-01-31T21:00:01.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:01.419806+0000) 2022-01-31T21:00:01.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:01 smithi171 conmon[51620]: debug 2022-01-31T21:00:01.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:01.497261+0000) 2022-01-31T21:00:01.707 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:01 smithi167 conmon[49112]: debug 2022-01-31T21:00:01.462+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:01.463479+0000) 2022-01-31T21:00:02.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:01 smithi171 conmon[41853]: debug 2022-01-31T21:00:01.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:01.624660+0000) 2022-01-31T21:00:02.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:02 smithi167 conmon[54076]: debug 2022-01-31T21:00:02.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:02.391597+0000) 2022-01-31T21:00:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:02 smithi167 conmon[60316]: debug 2022-01-31T21:00:02.142+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:02.143029+0000) 2022-01-31T21:00:02.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:02 smithi171 conmon[46715]: debug 2022-01-31T21:00:02.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:02.419965+0000) 2022-01-31T21:00:02.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:02 smithi171 conmon[51620]: debug 2022-01-31T21:00:02.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:02.497446+0000) 2022-01-31T21:00:02.707 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:02 smithi167 conmon[49112]: debug 2022-01-31T21:00:02.463+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:02.463580+0000) 2022-01-31T21:00:03.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:02 smithi171 conmon[41853]: debug 2022-01-31T21:00:02.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:02.624777+0000) 2022-01-31T21:00:03.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:03 smithi167 conmon[54076]: debug 2022-01-31T21:00:03.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:03.391711+0000) 2022-01-31T21:00:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:03 smithi167 conmon[60316]: debug 2022-01-31T21:00:03.142+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:03.143214+0000) 2022-01-31T21:00:03.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:03 smithi171 conmon[46715]: debug 2022-01-31T21:00:03.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:03.420160+0000) 2022-01-31T21:00:03.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:03 smithi171 conmon[51620]: debug 2022-01-31T21:00:03.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:03.497641+0000) 2022-01-31T21:00:03.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:03 smithi167 conmon[49112]: debug 2022-01-31T21:00:03.463+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:03.463708+0000) 2022-01-31T21:00:04.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:03 smithi171 conmon[41853]: debug 2022-01-31T21:00:03.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:03.624955+0000) 2022-01-31T21:00:04.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:04 smithi167 conmon[54076]: debug 2022-01-31T21:00:04.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:04.391814+0000) 2022-01-31T21:00:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:04 smithi167 conmon[60316]: debug 2022-01-31T21:00:04.143+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:04.143418+0000) 2022-01-31T21:00:04.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:04 smithi171 conmon[46715]: debug 2022-01-31T21:00:04.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:04.420314+0000) 2022-01-31T21:00:04.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:04 smithi171 conmon[51620]: debug 2022-01-31T21:00:04.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:04.497789+0000) 2022-01-31T21:00:04.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:04 smithi167 conmon[49112]: debug 2022-01-31T21:00:04.463+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:04.463852+0000) 2022-01-31T21:00:05.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:04 smithi171 conmon[41853]: debug 2022-01-31T21:00:04.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:04.625106+0000) 2022-01-31T21:00:05.390 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:05 smithi167 conmon[49112]: debug 2022-01-31T21:00:05.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.139685+0000) 2022-01-31T21:00:05.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:05 smithi167 conmon[54076]: debug 2022-01-31T21:00:05.139+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.140446+0000) 2022-01-31T21:00:05.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:05 smithi167 conmon[60316]: debug 2022-01-31T21:00:05.139+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.140220+0000) 2022-01-31T21:00:05.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:05 smithi167 conmon[60316]: debug 2022-01-31T21:00:05.143+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.143525+0000) 2022-01-31T21:00:05.418 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[35325]: debug 2022-01-31T21:00:05.148+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95134 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:05.419 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[41853]: debug 2022-01-31T21:00:05.137+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.138654+0000) 2022-01-31T21:00:05.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[46715]: debug 2022-01-31T21:00:05.138+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.139384+0000) 2022-01-31T21:00:05.420 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[51620]: debug 2022-01-31T21:00:05.138+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.139600+0000) 2022-01-31T21:00:05.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:05 smithi167 conmon[49112]: debug 2022-01-31T21:00:05.463+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.464047+0000) 2022-01-31T21:00:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:05 smithi167 conmon[54076]: debug 2022-01-31T21:00:05.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.391991+0000) 2022-01-31T21:00:05.749 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[46715]: debug 2022-01-31T21:00:05.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.420478+0000) 2022-01-31T21:00:05.749 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[51620]: debug 2022-01-31T21:00:05.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.497915+0000) 2022-01-31T21:00:05.750 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:05 smithi171 conmon[41853]: debug 2022-01-31T21:00:05.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:05.625282+0000) 2022-01-31T21:00:06.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:06 smithi167 conmon[54076]: debug 2022-01-31T21:00:06.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:06.392109+0000) 2022-01-31T21:00:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:06 smithi167 conmon[60316]: debug 2022-01-31T21:00:06.143+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:06.143668+0000) 2022-01-31T21:00:06.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:06 smithi171 conmon[46715]: debug 2022-01-31T21:00:06.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:06.420615+0000) 2022-01-31T21:00:06.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:06 smithi171 conmon[51620]: debug 2022-01-31T21:00:06.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:06.498085+0000) 2022-01-31T21:00:06.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:06 smithi167 conmon[49112]: debug 2022-01-31T21:00:06.463+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:06.464216+0000) 2022-01-31T21:00:07.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:06 smithi171 conmon[41853]: debug 2022-01-31T21:00:06.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:06.625450+0000) 2022-01-31T21:00:07.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:07 smithi167 conmon[54076]: debug 2022-01-31T21:00:07.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:07.392279+0000) 2022-01-31T21:00:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:07 smithi167 conmon[60316]: debug 2022-01-31T21:00:07.143+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:07.143825+0000) 2022-01-31T21:00:07.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:07 smithi171 conmon[46715]: debug 2022-01-31T21:00:07.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:07.420814+0000) 2022-01-31T21:00:07.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:07 smithi171 conmon[51620]: debug 2022-01-31T21:00:07.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:07.498248+0000) 2022-01-31T21:00:07.709 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:07 smithi167 conmon[49112]: debug 2022-01-31T21:00:07.463+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:07.464439+0000) 2022-01-31T21:00:08.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:07 smithi171 conmon[41853]: debug 2022-01-31T21:00:07.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:07.625623+0000) 2022-01-31T21:00:08.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:08 smithi167 conmon[60316]: debug 2022-01-31T21:00:08.143+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:08.144008+0000) 2022-01-31T21:00:08.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:08 smithi167 conmon[54076]: debug 2022-01-31T21:00:08.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:08.392478+0000) 2022-01-31T21:00:08.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:08 smithi171 conmon[46715]: debug 2022-01-31T21:00:08.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:08.420981+0000) 2022-01-31T21:00:08.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:08 smithi171 conmon[51620]: debug 2022-01-31T21:00:08.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:08.498355+0000) 2022-01-31T21:00:08.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:08 smithi167 conmon[49112]: debug 2022-01-31T21:00:08.464+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:08.464617+0000) 2022-01-31T21:00:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:08 smithi171 conmon[35325]: debug 2022-01-31T21:00:08.606+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:00:08.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:08 smithi171 conmon[41853]: debug 2022-01-31T21:00:08.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:08.625826+0000) 2022-01-31T21:00:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:09 smithi167 conmon[60316]: debug 2022-01-31T21:00:09.143+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:09.144213+0000) 2022-01-31T21:00:09.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:09 smithi167 conmon[54076]: debug 2022-01-31T21:00:09.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:09.392650+0000) 2022-01-31T21:00:09.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:09 smithi171 conmon[46715]: debug 2022-01-31T21:00:09.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:09.421171+0000) 2022-01-31T21:00:09.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:09 smithi171 conmon[51620]: debug 2022-01-31T21:00:09.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:09.498490+0000) 2022-01-31T21:00:09.709 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:09 smithi167 conmon[49112]: debug 2022-01-31T21:00:09.464+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:09.464772+0000) 2022-01-31T21:00:10.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:09 smithi171 conmon[41853]: debug 2022-01-31T21:00:09.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:09.626027+0000) 2022-01-31T21:00:10.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:10 smithi167 conmon[49112]: debug 2022-01-31T21:00:10.152+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.152536+0000) 2022-01-31T21:00:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:10 smithi167 conmon[54076]: debug 2022-01-31T21:00:10.151+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.152329+0000) 2022-01-31T21:00:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:10 smithi167 conmon[54076]: debug 2022-01-31T21:00:10.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.392821+0000) 2022-01-31T21:00:10.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:10 smithi167 conmon[60316]: debug 2022-01-31T21:00:10.144+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.144421+0000) 2022-01-31T21:00:10.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:10 smithi167 conmon[60316]: debug 2022-01-31T21:00:10.151+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.152156+0000) 2022-01-31T21:00:10.419 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[35325]: debug 2022-01-31T21:00:10.160+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95245 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:10.420 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[41853]: debug 2022-01-31T21:00:10.150+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.150904+0000) 2022-01-31T21:00:10.420 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[46715]: debug 2022-01-31T21:00:10.150+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.151648+0000) 2022-01-31T21:00:10.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[51620]: debug 2022-01-31T21:00:10.151+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.152446+0000) 2022-01-31T21:00:10.709 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:10 smithi167 conmon[49112]: debug 2022-01-31T21:00:10.464+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.464917+0000) 2022-01-31T21:00:10.750 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[46715]: debug 2022-01-31T21:00:10.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.421361+0000) 2022-01-31T21:00:10.750 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[41853]: debug 2022-01-31T21:00:10.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.626180+0000) 2022-01-31T21:00:10.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:10 smithi171 conmon[51620]: debug 2022-01-31T21:00:10.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:10.498601+0000) 2022-01-31T21:00:11.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:11 smithi167 conmon[54076]: debug 2022-01-31T21:00:11.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:11.392951+0000) 2022-01-31T21:00:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:11 smithi167 conmon[60316]: debug 2022-01-31T21:00:11.144+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:11.144608+0000) 2022-01-31T21:00:11.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:11 smithi171 conmon[46715]: debug 2022-01-31T21:00:11.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:11.421491+0000) 2022-01-31T21:00:11.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:11 smithi171 conmon[51620]: debug 2022-01-31T21:00:11.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:11.498745+0000) 2022-01-31T21:00:11.709 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:11 smithi167 conmon[49112]: debug 2022-01-31T21:00:11.464+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:11.465054+0000) 2022-01-31T21:00:12.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:11 smithi171 conmon[41853]: debug 2022-01-31T21:00:11.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:11.626360+0000) 2022-01-31T21:00:12.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:12 smithi167 conmon[60316]: debug 2022-01-31T21:00:12.144+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:12.144752+0000) 2022-01-31T21:00:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:12 smithi167 conmon[54076]: debug 2022-01-31T21:00:12.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:12.393147+0000) 2022-01-31T21:00:12.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:12 smithi171 conmon[46715]: debug 2022-01-31T21:00:12.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:12.421688+0000) 2022-01-31T21:00:12.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:12 smithi171 conmon[51620]: debug 2022-01-31T21:00:12.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:12.498916+0000) 2022-01-31T21:00:12.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:12 smithi167 conmon[49112]: debug 2022-01-31T21:00:12.464+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:12.465209+0000) 2022-01-31T21:00:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:12 smithi171 conmon[41853]: debug 2022-01-31T21:00:12.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:12.626583+0000) 2022-01-31T21:00:13.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:13 smithi167 conmon[60316]: debug 2022-01-31T21:00:13.144+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:13.144921+0000) 2022-01-31T21:00:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:13 smithi167 conmon[54076]: debug 2022-01-31T21:00:13.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:13.393306+0000) 2022-01-31T21:00:13.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:13 smithi171 conmon[46715]: debug 2022-01-31T21:00:13.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:13.421907+0000) 2022-01-31T21:00:13.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:13 smithi171 conmon[51620]: debug 2022-01-31T21:00:13.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:13.499090+0000) 2022-01-31T21:00:13.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:13 smithi167 conmon[49112]: debug 2022-01-31T21:00:13.464+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:13.465399+0000) 2022-01-31T21:00:14.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:13 smithi171 conmon[41853]: debug 2022-01-31T21:00:13.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:13.626787+0000) 2022-01-31T21:00:14.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:14 smithi167 conmon[60316]: debug 2022-01-31T21:00:14.144+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:14.145080+0000) 2022-01-31T21:00:14.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:14 smithi167 conmon[54076]: debug 2022-01-31T21:00:14.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:14.393537+0000) 2022-01-31T21:00:14.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:14 smithi171 conmon[46715]: debug 2022-01-31T21:00:14.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:14.422064+0000) 2022-01-31T21:00:14.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:14 smithi171 conmon[51620]: debug 2022-01-31T21:00:14.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:14.499244+0000) 2022-01-31T21:00:14.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:14 smithi167 conmon[49112]: debug 2022-01-31T21:00:14.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:14.465602+0000) 2022-01-31T21:00:15.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:14 smithi171 conmon[41853]: debug 2022-01-31T21:00:14.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:14.626981+0000) 2022-01-31T21:00:15.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:15 smithi167 conmon[49112]: debug 2022-01-31T21:00:15.163+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.163872+0000) 2022-01-31T21:00:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:15 smithi167 conmon[60316]: debug 2022-01-31T21:00:15.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.145242+0000) 2022-01-31T21:00:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:15 smithi167 conmon[60316]: debug 2022-01-31T21:00:15.163+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.163734+0000) 2022-01-31T21:00:15.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:15 smithi167 conmon[54076]: debug 2022-01-31T21:00:15.163+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.163655+0000) 2022-01-31T21:00:15.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:15 smithi167 conmon[54076]: debug 2022-01-31T21:00:15.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.393696+0000) 2022-01-31T21:00:15.420 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[35325]: debug 2022-01-31T21:00:15.172+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95355 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:15.420 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[41853]: debug 2022-01-31T21:00:15.163+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.164172+0000) 2022-01-31T21:00:15.421 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[46715]: debug 2022-01-31T21:00:15.163+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.163834+0000) 2022-01-31T21:00:15.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[51620]: debug 2022-01-31T21:00:15.163+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.164239+0000) 2022-01-31T21:00:15.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:15 smithi167 conmon[49112]: debug 2022-01-31T21:00:15.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.465795+0000) 2022-01-31T21:00:15.751 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[46715]: debug 2022-01-31T21:00:15.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.422222+0000) 2022-01-31T21:00:15.751 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[41853]: debug 2022-01-31T21:00:15.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.627190+0000) 2022-01-31T21:00:15.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:15 smithi171 conmon[51620]: debug 2022-01-31T21:00:15.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:15.499449+0000) 2022-01-31T21:00:16.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:16 smithi167 conmon[60316]: debug 2022-01-31T21:00:16.144+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:16.145448+0000) 2022-01-31T21:00:16.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:16 smithi167 conmon[54076]: debug 2022-01-31T21:00:16.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:16.393825+0000) 2022-01-31T21:00:16.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:16 smithi171 conmon[46715]: debug 2022-01-31T21:00:16.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:16.422385+0000) 2022-01-31T21:00:16.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:16 smithi171 conmon[51620]: debug 2022-01-31T21:00:16.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:16.499604+0000) 2022-01-31T21:00:16.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:16 smithi167 conmon[49112]: debug 2022-01-31T21:00:16.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:16.465940+0000) 2022-01-31T21:00:17.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:16 smithi171 conmon[41853]: debug 2022-01-31T21:00:16.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:16.627354+0000) 2022-01-31T21:00:17.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:17 smithi167 conmon[60316]: debug 2022-01-31T21:00:17.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:17.145606+0000) 2022-01-31T21:00:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:17 smithi167 conmon[54076]: debug 2022-01-31T21:00:17.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:17.393950+0000) 2022-01-31T21:00:17.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:17 smithi171 conmon[46715]: debug 2022-01-31T21:00:17.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:17.422592+0000) 2022-01-31T21:00:17.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:17 smithi171 conmon[51620]: debug 2022-01-31T21:00:17.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:17.499786+0000) 2022-01-31T21:00:17.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:17 smithi167 conmon[49112]: debug 2022-01-31T21:00:17.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:17.466105+0000) 2022-01-31T21:00:18.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:17 smithi171 conmon[41853]: debug 2022-01-31T21:00:17.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:17.627566+0000) 2022-01-31T21:00:18.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:18 smithi167 conmon[60316]: debug 2022-01-31T21:00:18.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:18.145769+0000) 2022-01-31T21:00:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:18 smithi167 conmon[54076]: debug 2022-01-31T21:00:18.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:18.394082+0000) 2022-01-31T21:00:18.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:18 smithi171 conmon[46715]: debug 2022-01-31T21:00:18.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:18.422809+0000) 2022-01-31T21:00:18.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:18 smithi171 conmon[51620]: debug 2022-01-31T21:00:18.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:18.499981+0000) 2022-01-31T21:00:18.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:18 smithi167 conmon[49112]: debug 2022-01-31T21:00:18.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:18.466307+0000) 2022-01-31T21:00:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:18 smithi171 conmon[41853]: debug 2022-01-31T21:00:18.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:18.627785+0000) 2022-01-31T21:00:19.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:19 smithi167 conmon[60316]: debug 2022-01-31T21:00:19.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:19.145925+0000) 2022-01-31T21:00:19.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:19 smithi167 conmon[54076]: debug 2022-01-31T21:00:19.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:19.394262+0000) 2022-01-31T21:00:19.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:19 smithi171 conmon[46715]: debug 2022-01-31T21:00:19.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:19.422993+0000) 2022-01-31T21:00:19.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:19 smithi171 conmon[51620]: debug 2022-01-31T21:00:19.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:19.500174+0000) 2022-01-31T21:00:19.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:19 smithi167 conmon[49112]: debug 2022-01-31T21:00:19.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:19.466486+0000) 2022-01-31T21:00:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:19 smithi171 conmon[41853]: debug 2022-01-31T21:00:19.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:19.628160+0000) 2022-01-31T21:00:20.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:20 smithi167 conmon[49112]: debug 2022-01-31T21:00:20.174+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.176038+0000) 2022-01-31T21:00:20.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:20 smithi167 conmon[60316]: debug 2022-01-31T21:00:20.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.146129+0000) 2022-01-31T21:00:20.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:20 smithi167 conmon[60316]: debug 2022-01-31T21:00:20.174+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.175902+0000) 2022-01-31T21:00:20.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:20 smithi167 conmon[54076]: debug 2022-01-31T21:00:20.174+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.175709+0000) 2022-01-31T21:00:20.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:20 smithi167 conmon[54076]: debug 2022-01-31T21:00:20.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.394469+0000) 2022-01-31T21:00:20.498 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[35325]: debug 2022-01-31T21:00:20.184+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95464 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:20.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[41853]: debug 2022-01-31T21:00:20.175+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.176136+0000) 2022-01-31T21:00:20.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[46715]: debug 2022-01-31T21:00:20.175+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.176410+0000) 2022-01-31T21:00:20.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[46715]: debug 2022-01-31T21:00:20.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.423178+0000) 2022-01-31T21:00:20.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[51620]: debug 2022-01-31T21:00:20.174+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.174986+0000) 2022-01-31T21:00:20.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:20 smithi167 conmon[49112]: debug 2022-01-31T21:00:20.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.466715+0000) 2022-01-31T21:00:20.751 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[41853]: debug 2022-01-31T21:00:20.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.628320+0000) 2022-01-31T21:00:20.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:20 smithi171 conmon[51620]: debug 2022-01-31T21:00:20.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:20.500361+0000) 2022-01-31T21:00:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:21 smithi167 conmon[60316]: debug 2022-01-31T21:00:21.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:21.146342+0000) 2022-01-31T21:00:21.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:21 smithi167 conmon[54076]: debug 2022-01-31T21:00:21.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:21.394667+0000) 2022-01-31T21:00:21.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:21 smithi171 conmon[46715]: debug 2022-01-31T21:00:21.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:21.423318+0000) 2022-01-31T21:00:21.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:21 smithi171 conmon[51620]: debug 2022-01-31T21:00:21.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:21.500511+0000) 2022-01-31T21:00:21.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:21 smithi167 conmon[49112]: debug 2022-01-31T21:00:21.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:21.466844+0000) 2022-01-31T21:00:22.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:21 smithi171 conmon[41853]: debug 2022-01-31T21:00:21.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:21.628484+0000) 2022-01-31T21:00:22.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:22 smithi167 conmon[60316]: debug 2022-01-31T21:00:22.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:22.146524+0000) 2022-01-31T21:00:22.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:22 smithi167 conmon[54076]: debug 2022-01-31T21:00:22.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:22.394823+0000) 2022-01-31T21:00:22.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:22 smithi171 conmon[46715]: debug 2022-01-31T21:00:22.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:22.423508+0000) 2022-01-31T21:00:22.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:22 smithi171 conmon[51620]: debug 2022-01-31T21:00:22.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:22.500660+0000) 2022-01-31T21:00:22.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:22 smithi167 conmon[49112]: debug 2022-01-31T21:00:22.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:22.467000+0000) 2022-01-31T21:00:23.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:22 smithi171 conmon[41853]: debug 2022-01-31T21:00:22.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:22.628661+0000) 2022-01-31T21:00:23.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:23 smithi167 conmon[60316]: debug 2022-01-31T21:00:23.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:23.146729+0000) 2022-01-31T21:00:23.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:23 smithi167 conmon[54076]: debug 2022-01-31T21:00:23.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:23.395019+0000) 2022-01-31T21:00:23.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:23 smithi171 conmon[46715]: debug 2022-01-31T21:00:23.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:23.423690+0000) 2022-01-31T21:00:23.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:23 smithi171 conmon[51620]: debug 2022-01-31T21:00:23.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:23.500839+0000) 2022-01-31T21:00:23.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:23 smithi167 conmon[49112]: debug 2022-01-31T21:00:23.465+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:23.467146+0000) 2022-01-31T21:00:24.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:23 smithi171 conmon[35325]: debug 2022-01-31T21:00:23.606+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:00:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:23 smithi171 conmon[41853]: debug 2022-01-31T21:00:23.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:23.628848+0000) 2022-01-31T21:00:24.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:24 smithi167 conmon[60316]: debug 2022-01-31T21:00:24.145+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:24.146915+0000) 2022-01-31T21:00:24.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:24 smithi167 conmon[54076]: debug 2022-01-31T21:00:24.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:24.395208+0000) 2022-01-31T21:00:24.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:24 smithi171 conmon[46715]: debug 2022-01-31T21:00:24.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:24.423874+0000) 2022-01-31T21:00:24.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:24 smithi171 conmon[51620]: debug 2022-01-31T21:00:24.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:24.501014+0000) 2022-01-31T21:00:24.712 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:24 smithi167 conmon[49112]: debug 2022-01-31T21:00:24.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:24.467339+0000) 2022-01-31T21:00:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:24 smithi171 conmon[41853]: debug 2022-01-31T21:00:24.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:24.629013+0000) 2022-01-31T21:00:25.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[49112]: debug 2022-01-31T21:00:25.187+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.188873+0000) 2022-01-31T21:00:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[54076]: debug 2022-01-31T21:00:25.186+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.188089+0000) 2022-01-31T21:00:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[54076]: debug 2022-01-31T21:00:25.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.395400+0000) 2022-01-31T21:00:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[60316]: debug 2022-01-31T21:00:25.146+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.147075+0000) 2022-01-31T21:00:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[60316]: debug 2022-01-31T21:00:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[60316]: 2022-01-31T21:00:25.186+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.187992+0000) 2022-01-31T21:00:25.499 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[35325]: debug 2022-01-31T21:00:25.195+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95566 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:25.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[41853]: debug 2022-01-31T21:00:25.188+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.189772+0000) 2022-01-31T21:00:25.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[46715]: debug 2022-01-31T21:00:25.187+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.188951+0000) 2022-01-31T21:00:25.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[46715]: debug 2022-01-31T21:00:25.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.424031+0000) 2022-01-31T21:00:25.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[51620]: debug 2022-01-31T21:00:25.185+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.187515+0000) 2022-01-31T21:00:25.712 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:25 smithi167 conmon[49112]: debug 2022-01-31T21:00:25.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.467534+0000) 2022-01-31T21:00:25.752 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[41853]: debug 2022-01-31T21:00:25.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.629202+0000) 2022-01-31T21:00:25.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:25 smithi171 conmon[51620]: debug 2022-01-31T21:00:25.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:25.501188+0000) 2022-01-31T21:00:26.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:26 smithi167 conmon[60316]: debug 2022-01-31T21:00:26.146+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:26.147226+0000) 2022-01-31T21:00:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:26 smithi167 conmon[54076]: debug 2022-01-31T21:00:26.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:26.395574+0000) 2022-01-31T21:00:26.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:26 smithi171 conmon[46715]: debug 2022-01-31T21:00:26.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:26.424208+0000) 2022-01-31T21:00:26.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:26 smithi171 conmon[51620]: debug 2022-01-31T21:00:26.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:26.501332+0000) 2022-01-31T21:00:26.712 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:26 smithi167 conmon[49112]: debug 2022-01-31T21:00:26.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:26.467683+0000) 2022-01-31T21:00:27.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:26 smithi171 conmon[41853]: debug 2022-01-31T21:00:26.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:26.629373+0000) 2022-01-31T21:00:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:27 smithi167 conmon[60316]: debug 2022-01-31T21:00:27.146+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:27.147376+0000) 2022-01-31T21:00:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:27 smithi167 conmon[54076]: debug 2022-01-31T21:00:27.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:27.395734+0000) 2022-01-31T21:00:27.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:27 smithi171 conmon[46715]: debug 2022-01-31T21:00:27.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:27.424390+0000) 2022-01-31T21:00:27.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:27 smithi171 conmon[51620]: debug 2022-01-31T21:00:27.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:27.501517+0000) 2022-01-31T21:00:27.712 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:27 smithi167 conmon[49112]: debug 2022-01-31T21:00:27.466+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:27.467872+0000) 2022-01-31T21:00:28.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:27 smithi171 conmon[41853]: debug 2022-01-31T21:00:27.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:27.629557+0000) 2022-01-31T21:00:28.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:28 smithi167 conmon[60316]: debug 2022-01-31T21:00:28.146+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:28.147570+0000) 2022-01-31T21:00:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:28 smithi167 conmon[54076]: debug 2022-01-31T21:00:28.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:28.395893+0000) 2022-01-31T21:00:28.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:28 smithi171 conmon[46715]: debug 2022-01-31T21:00:28.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:28.424591+0000) 2022-01-31T21:00:28.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:28 smithi171 conmon[51620]: debug 2022-01-31T21:00:28.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:28.501708+0000) 2022-01-31T21:00:28.712 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:28 smithi167 conmon[49112]: debug 2022-01-31T21:00:28.467+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:28.468058+0000) 2022-01-31T21:00:29.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:28 smithi171 conmon[41853]: debug 2022-01-31T21:00:28.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:28.629760+0000) 2022-01-31T21:00:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:29 smithi167 conmon[60316]: debug 2022-01-31T21:00:29.146+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:29.147725+0000) 2022-01-31T21:00:29.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:29 smithi167 conmon[54076]: debug 2022-01-31T21:00:29.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:29.396054+0000) 2022-01-31T21:00:29.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:29 smithi171 conmon[46715]: debug 2022-01-31T21:00:29.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:29.424807+0000) 2022-01-31T21:00:29.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:29 smithi171 conmon[51620]: debug 2022-01-31T21:00:29.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:29.501865+0000) 2022-01-31T21:00:29.712 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:29 smithi167 conmon[49112]: debug 2022-01-31T21:00:29.467+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:29.468248+0000) 2022-01-31T21:00:30.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:29 smithi171 conmon[41853]: debug 2022-01-31T21:00:29.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:29.629914+0000) 2022-01-31T21:00:30.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:30 smithi167 conmon[49112]: debug 2022-01-31T21:00:30.199+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.200575+0000) 2022-01-31T21:00:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:30 smithi167 conmon[54076]: debug 2022-01-31T21:00:30.199+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.200754+0000) 2022-01-31T21:00:30.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:30 smithi167 conmon[54076]: debug 2022-01-31T21:00:30.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.396242+0000) 2022-01-31T21:00:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:30 smithi167 conmon[60316]: debug 2022-01-31T21:00:30.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.147898+0000) 2022-01-31T21:00:30.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:30 smithi167 conmon[60316]: debug 2022-01-31T21:00:30.198+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.200053+0000) 2022-01-31T21:00:30.500 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[35325]: debug 2022-01-31T21:00:30.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95676 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:30.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[41853]: debug 2022-01-31T21:00:30.198+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.200691+0000) 2022-01-31T21:00:30.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[51620]: debug 2022-01-31T21:00:30.197+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.199491+0000) 2022-01-31T21:00:30.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[46715]: debug 2022-01-31T21:00:30.198+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.200263+0000) 2022-01-31T21:00:30.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[46715]: debug 2022-01-31T21:00:30.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.424928+0000) 2022-01-31T21:00:30.713 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:30 smithi167 conmon[49112]: debug 2022-01-31T21:00:30.467+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.468443+0000) 2022-01-31T21:00:30.753 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[41853]: debug 2022-01-31T21:00:30.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.630084+0000) 2022-01-31T21:00:30.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:30 smithi171 conmon[51620]: debug 2022-01-31T21:00:30.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:30.502067+0000) 2022-01-31T21:00:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:31 smithi167 conmon[54076]: debug 2022-01-31T21:00:31.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:31.396386+0000) 2022-01-31T21:00:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:31 smithi167 conmon[60316]: debug 2022-01-31T21:00:31.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:31.148052+0000) 2022-01-31T21:00:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:31 smithi171 conmon[46715]: debug 2022-01-31T21:00:31.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:31.425091+0000) 2022-01-31T21:00:31.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:31 smithi171 conmon[51620]: debug 2022-01-31T21:00:31.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:31.502240+0000) 2022-01-31T21:00:31.713 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:31 smithi167 conmon[49112]: debug 2022-01-31T21:00:31.467+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:31.468613+0000) 2022-01-31T21:00:32.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:31 smithi171 conmon[41853]: debug 2022-01-31T21:00:31.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:31.630241+0000) 2022-01-31T21:00:32.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:32 smithi167 conmon[60316]: debug 2022-01-31T21:00:32.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:32.148170+0000) 2022-01-31T21:00:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:32 smithi167 conmon[54076]: debug 2022-01-31T21:00:32.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:32.396573+0000) 2022-01-31T21:00:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:32 smithi171 conmon[46715]: debug 2022-01-31T21:00:32.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:32.425240+0000) 2022-01-31T21:00:32.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:32 smithi171 conmon[51620]: debug 2022-01-31T21:00:32.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:32.502441+0000) 2022-01-31T21:00:32.713 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:32 smithi167 conmon[49112]: debug 2022-01-31T21:00:32.468+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:32.468790+0000) 2022-01-31T21:00:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:32 smithi171 conmon[41853]: debug 2022-01-31T21:00:32.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:32.630453+0000) 2022-01-31T21:00:33.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:33 smithi167 conmon[54076]: debug 2022-01-31T21:00:33.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:33.396741+0000) 2022-01-31T21:00:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:33 smithi167 conmon[60316]: debug 2022-01-31T21:00:33.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:33.148327+0000) 2022-01-31T21:00:33.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:33 smithi171 conmon[51620]: debug 2022-01-31T21:00:33.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:33.502625+0000) 2022-01-31T21:00:33.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:33 smithi171 conmon[46715]: debug 2022-01-31T21:00:33.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:33.425414+0000) 2022-01-31T21:00:33.713 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:33 smithi167 conmon[49112]: debug 2022-01-31T21:00:33.468+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:33.468969+0000) 2022-01-31T21:00:34.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:33 smithi171 conmon[41853]: debug 2022-01-31T21:00:33.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:33.630655+0000) 2022-01-31T21:00:34.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:34 smithi167 conmon[54076]: debug 2022-01-31T21:00:34.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:34.396938+0000) 2022-01-31T21:00:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:34 smithi167 conmon[60316]: debug 2022-01-31T21:00:34.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:34.148519+0000) 2022-01-31T21:00:34.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:34 smithi171 conmon[46715]: debug 2022-01-31T21:00:34.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:34.425574+0000) 2022-01-31T21:00:34.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:34 smithi171 conmon[51620]: debug 2022-01-31T21:00:34.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:34.502837+0000) 2022-01-31T21:00:34.713 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:34 smithi167 conmon[49112]: debug 2022-01-31T21:00:34.468+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:34.469141+0000) 2022-01-31T21:00:35.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:34 smithi171 conmon[41853]: debug 2022-01-31T21:00:34.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:34.630853+0000) 2022-01-31T21:00:35.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:35 smithi167 conmon[49112]: debug 2022-01-31T21:00:35.211+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.212932+0000) 2022-01-31T21:00:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:35 smithi167 conmon[60316]: debug 2022-01-31T21:00:35.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.148639+0000) 2022-01-31T21:00:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:35 smithi167 conmon[60316]: debug 2022-01-31T21:00:35.212+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.213670+0000) 2022-01-31T21:00:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:35 smithi167 conmon[54076]: debug 2022-01-31T21:00:35.218+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.219966+0000) 2022-01-31T21:00:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:35 smithi167 conmon[54076]: debug 2022-01-31T21:00:35.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.397112+0000) 2022-01-31T21:00:35.501 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[35325]: debug 2022-01-31T21:00:35.224+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95786 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:35.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[41853]: debug 2022-01-31T21:00:35.213+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.213797+0000) 2022-01-31T21:00:35.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[51620]: debug 2022-01-31T21:00:35.212+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.213596+0000) 2022-01-31T21:00:35.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[46715]: debug 2022-01-31T21:00:35.212+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.213523+0000) 2022-01-31T21:00:35.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[46715]: debug 2022-01-31T21:00:35.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.425788+0000) 2022-01-31T21:00:35.713 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:35 smithi167 conmon[49112]: debug 2022-01-31T21:00:35.467+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.469355+0000) 2022-01-31T21:00:35.754 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[41853]: debug 2022-01-31T21:00:35.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.631036+0000) 2022-01-31T21:00:35.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:35 smithi171 conmon[51620]: debug 2022-01-31T21:00:35.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:35.502997+0000) 2022-01-31T21:00:36.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:36 smithi167 conmon[54076]: debug 2022-01-31T21:00:36.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:36.397249+0000) 2022-01-31T21:00:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:36 smithi167 conmon[60316]: debug 2022-01-31T21:00:36.148+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:36.148814+0000) 2022-01-31T21:00:36.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:36 smithi171 conmon[46715]: debug 2022-01-31T21:00:36.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:36.425884+0000) 2022-01-31T21:00:36.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:36 smithi171 conmon[51620]: debug 2022-01-31T21:00:36.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:36.503140+0000) 2022-01-31T21:00:36.714 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:36 smithi167 conmon[49112]: debug 2022-01-31T21:00:36.468+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:36.469526+0000) 2022-01-31T21:00:37.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:36 smithi171 conmon[41853]: debug 2022-01-31T21:00:36.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:36.631137+0000) 2022-01-31T21:00:37.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:37 smithi167 conmon[54076]: debug 2022-01-31T21:00:37.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:37.397446+0000) 2022-01-31T21:00:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:37 smithi167 conmon[60316]: debug 2022-01-31T21:00:37.148+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:37.148968+0000) 2022-01-31T21:00:37.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:37 smithi171 conmon[46715]: debug 2022-01-31T21:00:37.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:37.426049+0000) 2022-01-31T21:00:37.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:37 smithi171 conmon[51620]: debug 2022-01-31T21:00:37.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:37.503326+0000) 2022-01-31T21:00:37.714 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:37 smithi167 conmon[49112]: debug 2022-01-31T21:00:37.468+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:37.469731+0000) 2022-01-31T21:00:38.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:37 smithi171 conmon[41853]: debug 2022-01-31T21:00:37.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:37.631292+0000) 2022-01-31T21:00:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:38 smithi167 conmon[54076]: debug 2022-01-31T21:00:38.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:38.397610+0000) 2022-01-31T21:00:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:38 smithi167 conmon[60316]: debug 2022-01-31T21:00:38.147+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:38.149171+0000) 2022-01-31T21:00:38.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:38 smithi171 conmon[46715]: debug 2022-01-31T21:00:38.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:38.426262+0000) 2022-01-31T21:00:38.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:38 smithi171 conmon[51620]: debug 2022-01-31T21:00:38.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:38.503559+0000) 2022-01-31T21:00:38.714 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:38 smithi167 conmon[49112]: debug 2022-01-31T21:00:38.469+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:38.469922+0000) 2022-01-31T21:00:39.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:38 smithi171 conmon[35325]: debug 2022-01-31T21:00:38.607+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:00:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:38 smithi171 conmon[41853]: debug 2022-01-31T21:00:38.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:38.631451+0000) 2022-01-31T21:00:39.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:39 smithi167 conmon[60316]: debug 2022-01-31T21:00:39.148+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:39.149345+0000) 2022-01-31T21:00:39.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:39 smithi167 conmon[54076]: debug 2022-01-31T21:00:39.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:39.397769+0000) 2022-01-31T21:00:39.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:39 smithi171 conmon[46715]: debug 2022-01-31T21:00:39.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:39.426464+0000) 2022-01-31T21:00:39.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:39 smithi171 conmon[51620]: debug 2022-01-31T21:00:39.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:39.503743+0000) 2022-01-31T21:00:39.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:39 smithi167 conmon[49112]: debug 2022-01-31T21:00:39.469+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:39.470105+0000) 2022-01-31T21:00:40.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:39 smithi171 conmon[41853]: debug 2022-01-31T21:00:39.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:39.631668+0000) 2022-01-31T21:00:40.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:40 smithi167 conmon[49112]: debug 2022-01-31T21:00:40.225+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.227215+0000) 2022-01-31T21:00:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:40 smithi167 conmon[60316]: debug 2022-01-31T21:00:40.148+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.149542+0000) 2022-01-31T21:00:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:40 smithi167 conmon[60316]: debug 2022-01-31T21:00:40.225+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.227372+0000) 2022-01-31T21:00:40.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:40 smithi167 conmon[54076]: debug 2022-01-31T21:00:40.226+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.227676+0000) 2022-01-31T21:00:40.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:40 smithi167 conmon[54076]: debug 2022-01-31T21:00:40.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.397980+0000) 2022-01-31T21:00:40.502 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[35325]: debug 2022-01-31T21:00:40.236+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 95896 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:40.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[41853]: debug 2022-01-31T21:00:40.227+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.228607+0000) 2022-01-31T21:00:40.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[51620]: debug 2022-01-31T21:00:40.227+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.228200+0000) 2022-01-31T21:00:40.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[46715]: debug 2022-01-31T21:00:40.227+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.227998+0000) 2022-01-31T21:00:40.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[46715]: debug 2022-01-31T21:00:40.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.426625+0000) 2022-01-31T21:00:40.714 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:40 smithi167 conmon[49112]: debug 2022-01-31T21:00:40.469+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.470281+0000) 2022-01-31T21:00:40.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[51620]: debug 2022-01-31T21:00:40.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.503947+0000) 2022-01-31T21:00:40.755 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:40 smithi171 conmon[41853]: debug 2022-01-31T21:00:40.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:40.631879+0000) 2022-01-31T21:00:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:41 smithi167 conmon[54076]: debug 2022-01-31T21:00:41.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:41.398131+0000) 2022-01-31T21:00:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:41 smithi167 conmon[60316]: debug 2022-01-31T21:00:41.149+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:41.149759+0000) 2022-01-31T21:00:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:41 smithi171 conmon[46715]: debug 2022-01-31T21:00:41.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:41.426788+0000) 2022-01-31T21:00:41.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:41 smithi171 conmon[51620]: debug 2022-01-31T21:00:41.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:41.504081+0000) 2022-01-31T21:00:41.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:41 smithi167 conmon[49112]: debug 2022-01-31T21:00:41.469+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:41.470432+0000) 2022-01-31T21:00:42.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:41 smithi171 conmon[41853]: debug 2022-01-31T21:00:41.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:41.632030+0000) 2022-01-31T21:00:42.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:42 smithi167 conmon[54076]: debug 2022-01-31T21:00:42.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:42.398333+0000) 2022-01-31T21:00:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:42 smithi167 conmon[60316]: debug 2022-01-31T21:00:42.149+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:42.149912+0000) 2022-01-31T21:00:42.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:42 smithi171 conmon[46715]: debug 2022-01-31T21:00:42.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:42.426976+0000) 2022-01-31T21:00:42.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:42 smithi171 conmon[51620]: debug 2022-01-31T21:00:42.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:42.504249+0000) 2022-01-31T21:00:42.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:42 smithi167 conmon[49112]: debug 2022-01-31T21:00:42.470+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:42.470593+0000) 2022-01-31T21:00:43.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:42 smithi171 conmon[41853]: debug 2022-01-31T21:00:42.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:42.632249+0000) 2022-01-31T21:00:43.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:43 smithi167 conmon[54076]: debug 2022-01-31T21:00:43.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:43.398522+0000) 2022-01-31T21:00:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:43 smithi167 conmon[60316]: debug 2022-01-31T21:00:43.149+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:43.150072+0000) 2022-01-31T21:00:43.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:43 smithi171 conmon[46715]: debug 2022-01-31T21:00:43.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:43.427138+0000) 2022-01-31T21:00:43.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:43 smithi171 conmon[51620]: debug 2022-01-31T21:00:43.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:43.504382+0000) 2022-01-31T21:00:43.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:43 smithi167 conmon[49112]: debug 2022-01-31T21:00:43.470+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:43.470721+0000) 2022-01-31T21:00:44.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:43 smithi171 conmon[41853]: debug 2022-01-31T21:00:43.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:43.632429+0000) 2022-01-31T21:00:44.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:44 smithi167 conmon[54076]: debug 2022-01-31T21:00:44.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:44.398634+0000) 2022-01-31T21:00:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:44 smithi167 conmon[60316]: debug 2022-01-31T21:00:44.149+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:44.150252+0000) 2022-01-31T21:00:44.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:44 smithi171 conmon[46715]: debug 2022-01-31T21:00:44.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:44.427316+0000) 2022-01-31T21:00:44.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:44 smithi171 conmon[51620]: debug 2022-01-31T21:00:44.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:44.504547+0000) 2022-01-31T21:00:44.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:44 smithi167 conmon[49112]: debug 2022-01-31T21:00:44.470+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:44.470873+0000) 2022-01-31T21:00:45.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:44 smithi171 conmon[41853]: debug 2022-01-31T21:00:44.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:44.632584+0000) 2022-01-31T21:00:45.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:45 smithi167 conmon[49112]: debug 2022-01-31T21:00:45.240+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.240592+0000) 2022-01-31T21:00:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:45 smithi167 conmon[54076]: debug 2022-01-31T21:00:45.240+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.241008+0000) 2022-01-31T21:00:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:45 smithi167 conmon[54076]: debug 2022-01-31T21:00:45.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.398797+0000) 2022-01-31T21:00:45.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:45 smithi167 conmon[60316]: debug 2022-01-31T21:00:45.149+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.150431+0000) 2022-01-31T21:00:45.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:45 smithi167 conmon[60316]: debug 2022-01-31T21:00:45.239+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.239736+0000) 2022-01-31T21:00:45.503 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[35325]: debug 2022-01-31T21:00:45.248+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96006 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:45.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[41853]: debug 2022-01-31T21:00:45.239+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.240593+0000) 2022-01-31T21:00:45.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[46715]: debug 2022-01-31T21:00:45.238+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.239173+0000) 2022-01-31T21:00:45.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[46715]: debug 2022-01-31T21:00:45.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.427500+0000) 2022-01-31T21:00:45.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[51620]: debug 2022-01-31T21:00:45.239+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.240267+0000) 2022-01-31T21:00:45.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:45 smithi167 conmon[49112]: debug 2022-01-31T21:00:45.470+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.471057+0000) 2022-01-31T21:00:45.756 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[41853]: debug 2022-01-31T21:00:45.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.632770+0000) 2022-01-31T21:00:45.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:45 smithi171 conmon[51620]: debug 2022-01-31T21:00:45.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:45.504749+0000) 2022-01-31T21:00:46.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:46 smithi167 conmon[54076]: debug 2022-01-31T21:00:46.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:46.398953+0000) 2022-01-31T21:00:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:46 smithi167 conmon[60316]: debug 2022-01-31T21:00:46.150+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:46.150612+0000) 2022-01-31T21:00:46.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:46 smithi171 conmon[46715]: debug 2022-01-31T21:00:46.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:46.427676+0000) 2022-01-31T21:00:46.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:46 smithi171 conmon[51620]: debug 2022-01-31T21:00:46.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:46.504919+0000) 2022-01-31T21:00:46.715 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:46 smithi167 conmon[49112]: debug 2022-01-31T21:00:46.470+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:46.471223+0000) 2022-01-31T21:00:47.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:46 smithi171 conmon[41853]: debug 2022-01-31T21:00:46.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:46.632902+0000) 2022-01-31T21:00:47.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:47 smithi167 conmon[54076]: debug 2022-01-31T21:00:47.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:47.399164+0000) 2022-01-31T21:00:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:47 smithi167 conmon[60316]: debug 2022-01-31T21:00:47.150+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:47.150781+0000) 2022-01-31T21:00:47.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:47 smithi171 conmon[46715]: debug 2022-01-31T21:00:47.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:47.427901+0000) 2022-01-31T21:00:47.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:47 smithi171 conmon[51620]: debug 2022-01-31T21:00:47.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:47.505104+0000) 2022-01-31T21:00:47.716 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:47 smithi167 conmon[49112]: debug 2022-01-31T21:00:47.470+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:47.471385+0000) 2022-01-31T21:00:48.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:47 smithi171 conmon[41853]: debug 2022-01-31T21:00:47.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:47.633076+0000) 2022-01-31T21:00:48.219 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T21:00:48.220+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T21:00:48.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:48 smithi167 conmon[54076]: debug 2022-01-31T21:00:48.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:48.399361+0000) 2022-01-31T21:00:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:48 smithi167 conmon[60316]: debug 2022-01-31T21:00:48.150+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:48.150966+0000) 2022-01-31T21:00:48.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:48 smithi171 conmon[46715]: debug 2022-01-31T21:00:48.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:48.428038+0000) 2022-01-31T21:00:48.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:48 smithi171 conmon[51620]: debug 2022-01-31T21:00:48.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:48.505284+0000) 2022-01-31T21:00:48.716 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:48 smithi167 conmon[49112]: debug 2022-01-31T21:00:48.471+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:48.471568+0000) 2022-01-31T21:00:49.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:48 smithi171 conmon[41853]: debug 2022-01-31T21:00:48.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:48.633215+0000) 2022-01-31T21:00:49.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:49 smithi167 conmon[54076]: debug 2022-01-31T21:00:49.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:49.399557+0000) 2022-01-31T21:00:49.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:49 smithi167 conmon[60316]: debug 2022-01-31T21:00:49.150+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:49.151150+0000) 2022-01-31T21:00:49.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:49 smithi171 conmon[46715]: debug 2022-01-31T21:00:49.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:49.428203+0000) 2022-01-31T21:00:49.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:49 smithi171 conmon[51620]: debug 2022-01-31T21:00:49.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:49.505470+0000) 2022-01-31T21:00:49.716 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:49 smithi167 conmon[49112]: debug 2022-01-31T21:00:49.471+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:49.471753+0000) 2022-01-31T21:00:50.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:49 smithi171 conmon[41853]: debug 2022-01-31T21:00:49.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:49.633397+0000) 2022-01-31T21:00:50.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:50 smithi167 conmon[49112]: debug 2022-01-31T21:00:50.252+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.253012+0000) 2022-01-31T21:00:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:50 smithi167 conmon[54076]: debug 2022-01-31T21:00:50.251+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.252466+0000) 2022-01-31T21:00:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:50 smithi167 conmon[54076]: debug 2022-01-31T21:00:50.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.399726+0000) 2022-01-31T21:00:50.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:50 smithi167 conmon[60316]: debug 2022-01-31T21:00:50.150+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.151348+0000) 2022-01-31T21:00:50.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:50 smithi167 conmon[60316]: debug 2022-01-31T21:00:50.251+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.252319+0000) 2022-01-31T21:00:50.504 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[35325]: debug 2022-01-31T21:00:50.260+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96116 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:50.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[41853]: debug 2022-01-31T21:00:50.250+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.251693+0000) 2022-01-31T21:00:50.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[46715]: debug 2022-01-31T21:00:50.250+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.250832+0000) 2022-01-31T21:00:50.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[46715]: debug 2022-01-31T21:00:50.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.428337+0000) 2022-01-31T21:00:50.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[51620]: debug 2022-01-31T21:00:50.251+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.251893+0000) 2022-01-31T21:00:50.716 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:50 smithi167 conmon[49112]: debug 2022-01-31T21:00:50.471+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.471944+0000) 2022-01-31T21:00:50.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[51620]: debug 2022-01-31T21:00:50.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.505686+0000) 2022-01-31T21:00:50.757 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:50 smithi171 conmon[41853]: debug 2022-01-31T21:00:50.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:50.633595+0000) 2022-01-31T21:00:51.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:51 smithi167 conmon[54076]: debug 2022-01-31T21:00:51.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:51.399875+0000) 2022-01-31T21:00:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:51 smithi167 conmon[60316]: debug 2022-01-31T21:00:51.151+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:51.151530+0000) 2022-01-31T21:00:51.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:51 smithi171 conmon[46715]: debug 2022-01-31T21:00:51.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:51.428484+0000) 2022-01-31T21:00:51.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:51 smithi171 conmon[51620]: debug 2022-01-31T21:00:51.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:51.505833+0000) 2022-01-31T21:00:51.716 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:51 smithi167 conmon[49112]: debug 2022-01-31T21:00:51.471+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:51.472112+0000) 2022-01-31T21:00:52.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:51 smithi171 conmon[41853]: debug 2022-01-31T21:00:51.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:51.633783+0000) 2022-01-31T21:00:52.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:52 smithi167 conmon[54076]: debug 2022-01-31T21:00:52.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:52.400002+0000) 2022-01-31T21:00:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:52 smithi167 conmon[60316]: debug 2022-01-31T21:00:52.151+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:52.151663+0000) 2022-01-31T21:00:52.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:52 smithi171 conmon[46715]: debug 2022-01-31T21:00:52.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:52.428637+0000) 2022-01-31T21:00:52.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:52 smithi171 conmon[51620]: debug 2022-01-31T21:00:52.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:52.505948+0000) 2022-01-31T21:00:52.716 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:52 smithi167 conmon[49112]: debug 2022-01-31T21:00:52.471+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:52.472301+0000) 2022-01-31T21:00:53.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:52 smithi171 conmon[41853]: debug 2022-01-31T21:00:52.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:52.633935+0000) 2022-01-31T21:00:53.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:53 smithi167 conmon[54076]: debug 2022-01-31T21:00:53.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:53.400200+0000) 2022-01-31T21:00:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:53 smithi167 conmon[60316]: debug 2022-01-31T21:00:53.151+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:53.151790+0000) 2022-01-31T21:00:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:53 smithi171 conmon[46715]: debug 2022-01-31T21:00:53.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:53.428771+0000) 2022-01-31T21:00:53.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:53 smithi171 conmon[51620]: debug 2022-01-31T21:00:53.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:53.506123+0000) 2022-01-31T21:00:53.717 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:53 smithi167 conmon[49112]: debug 2022-01-31T21:00:53.472+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:53.472495+0000) 2022-01-31T21:00:54.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:53 smithi171 conmon[35325]: debug 2022-01-31T21:00:53.608+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:00:54.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:53 smithi171 conmon[41853]: debug 2022-01-31T21:00:53.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:53.634055+0000) 2022-01-31T21:00:54.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:54 smithi167 conmon[54076]: debug 2022-01-31T21:00:54.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:54.400400+0000) 2022-01-31T21:00:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:54 smithi167 conmon[60316]: debug 2022-01-31T21:00:54.151+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:54.151950+0000) 2022-01-31T21:00:54.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:54 smithi171 conmon[46715]: debug 2022-01-31T21:00:54.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:54.428926+0000) 2022-01-31T21:00:54.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:54 smithi171 conmon[51620]: debug 2022-01-31T21:00:54.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:54.506313+0000) 2022-01-31T21:00:54.717 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:54 smithi167 conmon[49112]: debug 2022-01-31T21:00:54.472+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:54.472703+0000) 2022-01-31T21:00:55.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:54 smithi171 conmon[41853]: debug 2022-01-31T21:00:54.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:54.634256+0000) 2022-01-31T21:00:55.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:55 smithi167 conmon[49112]: debug 2022-01-31T21:00:55.264+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.264569+0000) 2022-01-31T21:00:55.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:55 smithi167 conmon[54076]: debug 2022-01-31T21:00:55.263+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.264104+0000) 2022-01-31T21:00:55.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:55 smithi167 conmon[54076]: debug 2022-01-31T21:00:55.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.400589+0000) 2022-01-31T21:00:55.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:55 smithi167 conmon[60316]: debug 2022-01-31T21:00:55.151+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.152161+0000) 2022-01-31T21:00:55.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:55 smithi167 conmon[60316]: debug 2022-01-31T21:00:55.263+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.263806+0000) 2022-01-31T21:00:55.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[35325]: debug 2022-01-31T21:00:55.272+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96226 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:00:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[41853]: debug 2022-01-31T21:00:55.263+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.263813+0000) 2022-01-31T21:00:55.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[51620]: debug 2022-01-31T21:00:55.262+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.263045+0000) 2022-01-31T21:00:55.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[51620]: debug 2022-01-31T21:00:55.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.506514+0000) 2022-01-31T21:00:55.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[46715]: debug 2022-01-31T21:00:55.262+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.263005+0000) 2022-01-31T21:00:55.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[46715]: debug 2022-01-31T21:00:55.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.429073+0000) 2022-01-31T21:00:55.717 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:55 smithi167 conmon[49112]: debug 2022-01-31T21:00:55.472+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.472869+0000) 2022-01-31T21:00:56.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:55 smithi171 conmon[41853]: debug 2022-01-31T21:00:55.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:55.634436+0000) 2022-01-31T21:00:56.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:56 smithi167 conmon[54076]: debug 2022-01-31T21:00:56.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:56.400747+0000) 2022-01-31T21:00:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:56 smithi167 conmon[60316]: debug 2022-01-31T21:00:56.152+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:56.152348+0000) 2022-01-31T21:00:56.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:56 smithi171 conmon[51620]: debug 2022-01-31T21:00:56.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:56.506625+0000) 2022-01-31T21:00:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:56 smithi171 conmon[46715]: debug 2022-01-31T21:00:56.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:56.429228+0000) 2022-01-31T21:00:56.717 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:56 smithi167 conmon[49112]: debug 2022-01-31T21:00:56.472+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:56.473023+0000) 2022-01-31T21:00:57.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:56 smithi171 conmon[41853]: debug 2022-01-31T21:00:56.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:56.634591+0000) 2022-01-31T21:00:57.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:57 smithi167 conmon[54076]: debug 2022-01-31T21:00:57.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:57.400939+0000) 2022-01-31T21:00:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:57 smithi167 conmon[60316]: debug 2022-01-31T21:00:57.152+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:57.152522+0000) 2022-01-31T21:00:57.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:57 smithi171 conmon[46715]: debug 2022-01-31T21:00:57.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:57.429419+0000) 2022-01-31T21:00:57.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:57 smithi171 conmon[51620]: debug 2022-01-31T21:00:57.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:57.506807+0000) 2022-01-31T21:00:57.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:57 smithi167 conmon[49112]: debug 2022-01-31T21:00:57.472+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:57.473187+0000) 2022-01-31T21:00:58.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:57 smithi171 conmon[41853]: debug 2022-01-31T21:00:57.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:57.634720+0000) 2022-01-31T21:00:58.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:58 smithi167 conmon[54076]: debug 2022-01-31T21:00:58.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:58.401120+0000) 2022-01-31T21:00:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:58 smithi167 conmon[60316]: debug 2022-01-31T21:00:58.152+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:58.152716+0000) 2022-01-31T21:00:58.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:58 smithi171 conmon[46715]: debug 2022-01-31T21:00:58.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:58.429640+0000) 2022-01-31T21:00:58.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:58 smithi171 conmon[51620]: debug 2022-01-31T21:00:58.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:58.507005+0000) 2022-01-31T21:00:58.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:58 smithi167 conmon[49112]: debug 2022-01-31T21:00:58.472+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:58.473377+0000) 2022-01-31T21:00:59.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:58 smithi171 conmon[41853]: debug 2022-01-31T21:00:58.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:58.634819+0000) 2022-01-31T21:00:59.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:00:59 smithi167 conmon[54076]: debug 2022-01-31T21:00:59.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:59.401349+0000) 2022-01-31T21:00:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:00:59 smithi167 conmon[60316]: debug 2022-01-31T21:00:59.152+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:59.152868+0000) 2022-01-31T21:00:59.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:00:59 smithi171 conmon[46715]: debug 2022-01-31T21:00:59.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:59.429824+0000) 2022-01-31T21:00:59.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:00:59 smithi171 conmon[51620]: debug 2022-01-31T21:00:59.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:59.507213+0000) 2022-01-31T21:00:59.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:00:59 smithi167 conmon[49112]: debug 2022-01-31T21:00:59.473+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:59.473595+0000) 2022-01-31T21:01:00.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:00:59 smithi171 conmon[41853]: debug 2022-01-31T21:00:59.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:00:59.634935+0000) 2022-01-31T21:01:00.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:00 smithi167 conmon[49112]: debug 2022-01-31T21:01:00.275+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.276481+0000) 2022-01-31T21:01:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:00 smithi167 conmon[54076]: debug 2022-01-31T21:01:00.276+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.277363+0000) 2022-01-31T21:01:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:00 smithi167 conmon[54076]: debug 2022-01-31T21:01:00.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.401542+0000) 2022-01-31T21:01:00.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:00 smithi167 conmon[60316]: debug 2022-01-31T21:01:00.152+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.153107+0000) 2022-01-31T21:01:00.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:00 smithi167 conmon[60316]: debug 2022-01-31T21:01:00.275+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.276193+0000) 2022-01-31T21:01:00.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[46715]: debug 2022-01-31T21:01:00.274+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.275434+0000) 2022-01-31T21:01:00.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[46715]: debug 2022-01-31T21:01:00.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.429984+0000) 2022-01-31T21:01:00.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[35325]: debug 2022-01-31T21:01:00.284+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96336 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:00.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[41853]: debug 2022-01-31T21:01:00.275+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.276645+0000) 2022-01-31T21:01:00.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[51620]: debug 2022-01-31T21:01:00.275+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.276033+0000) 2022-01-31T21:01:00.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[51620]: debug 2022-01-31T21:01:00.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.507377+0000) 2022-01-31T21:01:00.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:00 smithi167 conmon[49112]: debug 2022-01-31T21:01:00.473+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.473808+0000) 2022-01-31T21:01:01.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:00 smithi171 conmon[41853]: debug 2022-01-31T21:01:00.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:00.635064+0000) 2022-01-31T21:01:01.400 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:01 smithi167 conmon[60316]: debug 2022-01-31T21:01:01.152+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:01.153289+0000) 2022-01-31T21:01:01.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:01 smithi171 conmon[46715]: debug 2022-01-31T21:01:01.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:01.430141+0000) 2022-01-31T21:01:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:01 smithi167 conmon[54076]: debug 2022-01-31T21:01:01.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:01.401690+0000) 2022-01-31T21:01:01.669 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:01 smithi167 conmon[49112]: debug 2022-01-31T21:01:01.473+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:01.473961+0000) 2022-01-31T21:01:01.759 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:01 smithi171 conmon[41853]: debug 2022-01-31T21:01:01.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:01.635211+0000) 2022-01-31T21:01:01.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:01 smithi171 conmon[51620]: debug 2022-01-31T21:01:01.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:01.507519+0000) 2022-01-31T21:01:02.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:02 smithi167 conmon[54076]: debug 2022-01-31T21:01:02.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:02.401846+0000) 2022-01-31T21:01:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:02 smithi167 conmon[60316]: debug 2022-01-31T21:01:02.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:02.153459+0000) 2022-01-31T21:01:02.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:02 smithi171 conmon[46715]: debug 2022-01-31T21:01:02.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:02.430338+0000) 2022-01-31T21:01:02.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:02 smithi171 conmon[51620]: debug 2022-01-31T21:01:02.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:02.507710+0000) 2022-01-31T21:01:02.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:02 smithi167 conmon[49112]: debug 2022-01-31T21:01:02.473+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:02.474150+0000) 2022-01-31T21:01:02.936 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:02 smithi171 conmon[41853]: debug 2022-01-31T21:01:02.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:02.635355+0000) 2022-01-31T21:01:03.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:03 smithi167 conmon[54076]: debug 2022-01-31T21:01:03.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:03.402005+0000) 2022-01-31T21:01:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:03 smithi167 conmon[60316]: debug 2022-01-31T21:01:03.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:03.153638+0000) 2022-01-31T21:01:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:03 smithi171 conmon[46715]: debug 2022-01-31T21:01:03.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:03.430559+0000) 2022-01-31T21:01:03.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:03 smithi171 conmon[51620]: debug 2022-01-31T21:01:03.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:03.507934+0000) 2022-01-31T21:01:03.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:03 smithi167 conmon[49112]: debug 2022-01-31T21:01:03.473+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:03.474308+0000) 2022-01-31T21:01:04.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:03 smithi171 conmon[41853]: debug 2022-01-31T21:01:03.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:03.635531+0000) 2022-01-31T21:01:04.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:04 smithi167 conmon[54076]: debug 2022-01-31T21:01:04.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:04.402142+0000) 2022-01-31T21:01:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:04 smithi167 conmon[60316]: debug 2022-01-31T21:01:04.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:04.153756+0000) 2022-01-31T21:01:04.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:04 smithi171 conmon[46715]: debug 2022-01-31T21:01:04.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:04.430796+0000) 2022-01-31T21:01:04.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:04 smithi171 conmon[51620]: debug 2022-01-31T21:01:04.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:04.508119+0000) 2022-01-31T21:01:04.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:04 smithi167 conmon[49112]: debug 2022-01-31T21:01:04.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:04.474519+0000) 2022-01-31T21:01:05.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:04 smithi171 conmon[41853]: debug 2022-01-31T21:01:04.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:04.635653+0000) 2022-01-31T21:01:05.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:05 smithi167 conmon[49112]: debug 2022-01-31T21:01:05.287+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.287965+0000) 2022-01-31T21:01:05.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:05 smithi167 conmon[54076]: debug 2022-01-31T21:01:05.287+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.288308+0000) 2022-01-31T21:01:05.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:05 smithi167 conmon[54076]: debug 2022-01-31T21:01:05.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.402292+0000) 2022-01-31T21:01:05.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:05 smithi167 conmon[60316]: debug 2022-01-31T21:01:05.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.153890+0000) 2022-01-31T21:01:05.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:05 smithi167 conmon[60316]: debug 2022-01-31T21:01:05.288+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.289008+0000) 2022-01-31T21:01:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[35325]: debug 2022-01-31T21:01:05.297+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96446 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[41853]: debug 2022-01-31T21:01:05.286+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.287687+0000) 2022-01-31T21:01:05.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[51620]: debug 2022-01-31T21:01:05.286+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.287524+0000) 2022-01-31T21:01:05.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[51620]: debug 2022-01-31T21:01:05.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.508288+0000) 2022-01-31T21:01:05.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[46715]: debug 2022-01-31T21:01:05.287+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.288169+0000) 2022-01-31T21:01:05.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[46715]: debug 2022-01-31T21:01:05.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.430988+0000) 2022-01-31T21:01:05.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:05 smithi167 conmon[49112]: debug 2022-01-31T21:01:05.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.474673+0000) 2022-01-31T21:01:06.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:05 smithi171 conmon[41853]: debug 2022-01-31T21:01:05.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:05.635826+0000) 2022-01-31T21:01:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:06 smithi167 conmon[54076]: debug 2022-01-31T21:01:06.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:06.402395+0000) 2022-01-31T21:01:06.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:06 smithi167 conmon[60316]: debug 2022-01-31T21:01:06.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:06.154064+0000) 2022-01-31T21:01:06.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:06 smithi171 conmon[51620]: debug 2022-01-31T21:01:06.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:06.508424+0000) 2022-01-31T21:01:06.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:06 smithi171 conmon[46715]: debug 2022-01-31T21:01:06.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:06.431155+0000) 2022-01-31T21:01:06.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:06 smithi167 conmon[49112]: debug 2022-01-31T21:01:06.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:06.474846+0000) 2022-01-31T21:01:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:06 smithi171 conmon[41853]: debug 2022-01-31T21:01:06.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:06.635941+0000) 2022-01-31T21:01:07.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:07 smithi167 conmon[54076]: debug 2022-01-31T21:01:07.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:07.402596+0000) 2022-01-31T21:01:07.679 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:07 smithi167 conmon[60316]: debug 2022-01-31T21:01:07.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:07.154239+0000) 2022-01-31T21:01:07.680 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:07 smithi171 conmon[51620]: debug 2022-01-31T21:01:07.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:07.508609+0000) 2022-01-31T21:01:07.680 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:07 smithi171 conmon[46715]: debug 2022-01-31T21:01:07.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:07.431330+0000) 2022-01-31T21:01:07.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:07 smithi167 conmon[49112]: debug 2022-01-31T21:01:07.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:07.475057+0000) 2022-01-31T21:01:08.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:07 smithi171 conmon[41853]: debug 2022-01-31T21:01:07.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:07.636057+0000) 2022-01-31T21:01:08.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:08 smithi167 conmon[54076]: debug 2022-01-31T21:01:08.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:08.402768+0000) 2022-01-31T21:01:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:08 smithi167 conmon[60316]: debug 2022-01-31T21:01:08.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:08.154438+0000) 2022-01-31T21:01:08.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:08 smithi171 conmon[51620]: debug 2022-01-31T21:01:08.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:08.508804+0000) 2022-01-31T21:01:08.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:08 smithi171 conmon[46715]: debug 2022-01-31T21:01:08.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:08.431545+0000) 2022-01-31T21:01:08.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:08 smithi167 conmon[49112]: debug 2022-01-31T21:01:08.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:08.475206+0000) 2022-01-31T21:01:09.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:08 smithi171 conmon[35325]: debug 2022-01-31T21:01:08.609+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:01:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:08 smithi171 conmon[41853]: debug 2022-01-31T21:01:08.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:08.636254+0000) 2022-01-31T21:01:09.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:09 smithi167 conmon[54076]: debug 2022-01-31T21:01:09.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:09.402902+0000) 2022-01-31T21:01:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:09 smithi167 conmon[60316]: debug 2022-01-31T21:01:09.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:09.154610+0000) 2022-01-31T21:01:09.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:09 smithi171 conmon[46715]: debug 2022-01-31T21:01:09.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:09.431795+0000) 2022-01-31T21:01:09.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:09 smithi171 conmon[51620]: debug 2022-01-31T21:01:09.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:09.508989+0000) 2022-01-31T21:01:09.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:09 smithi167 conmon[49112]: debug 2022-01-31T21:01:09.473+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:09.475413+0000) 2022-01-31T21:01:10.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:09 smithi171 conmon[41853]: debug 2022-01-31T21:01:09.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:09.636456+0000) 2022-01-31T21:01:10.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:10 smithi167 conmon[49112]: debug 2022-01-31T21:01:10.299+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.301205+0000) 2022-01-31T21:01:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:10 smithi167 conmon[60316]: debug 2022-01-31T21:01:10.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.154735+0000) 2022-01-31T21:01:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:10 smithi167 conmon[60316]: debug 2022-01-31T21:01:10.299+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.301356+0000) 2022-01-31T21:01:10.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:10 smithi167 conmon[54076]: debug 2022-01-31T21:01:10.300+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.301711+0000) 2022-01-31T21:01:10.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:10 smithi167 conmon[54076]: debug 2022-01-31T21:01:10.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.403087+0000) 2022-01-31T21:01:10.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[35325]: debug 2022-01-31T21:01:10.309+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96556 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[41853]: debug 2022-01-31T21:01:10.298+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.300130+0000) 2022-01-31T21:01:10.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[51620]: debug 2022-01-31T21:01:10.298+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.300206+0000) 2022-01-31T21:01:10.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[51620]: debug 2022-01-31T21:01:10.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.509171+0000) 2022-01-31T21:01:10.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[46715]: debug 2022-01-31T21:01:10.299+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.300797+0000) 2022-01-31T21:01:10.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[46715]: debug 2022-01-31T21:01:10.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.431977+0000) 2022-01-31T21:01:10.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:10 smithi167 conmon[49112]: debug 2022-01-31T21:01:10.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.475639+0000) 2022-01-31T21:01:11.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:10 smithi171 conmon[41853]: debug 2022-01-31T21:01:10.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:10.636606+0000) 2022-01-31T21:01:11.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:11 smithi167 conmon[54076]: debug 2022-01-31T21:01:11.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:11.403264+0000) 2022-01-31T21:01:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:11 smithi167 conmon[60316]: debug 2022-01-31T21:01:11.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:11.154918+0000) 2022-01-31T21:01:11.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:11 smithi171 conmon[46715]: debug 2022-01-31T21:01:11.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:11.432135+0000) 2022-01-31T21:01:11.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:11 smithi171 conmon[51620]: debug 2022-01-31T21:01:11.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:11.509327+0000) 2022-01-31T21:01:11.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:11 smithi167 conmon[49112]: debug 2022-01-31T21:01:11.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:11.475758+0000) 2022-01-31T21:01:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:11 smithi171 conmon[41853]: debug 2022-01-31T21:01:11.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:11.636766+0000) 2022-01-31T21:01:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:12 smithi167 conmon[54076]: debug 2022-01-31T21:01:12.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:12.403453+0000) 2022-01-31T21:01:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:12 smithi167 conmon[60316]: debug 2022-01-31T21:01:12.154+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:12.155096+0000) 2022-01-31T21:01:12.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:12 smithi171 conmon[46715]: debug 2022-01-31T21:01:12.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:12.432347+0000) 2022-01-31T21:01:12.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:12 smithi171 conmon[51620]: debug 2022-01-31T21:01:12.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:12.509533+0000) 2022-01-31T21:01:12.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:12 smithi167 conmon[49112]: debug 2022-01-31T21:01:12.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:12.475971+0000) 2022-01-31T21:01:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:12 smithi171 conmon[41853]: debug 2022-01-31T21:01:12.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:12.636951+0000) 2022-01-31T21:01:13.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:13 smithi167 conmon[54076]: debug 2022-01-31T21:01:13.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:13.403598+0000) 2022-01-31T21:01:13.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:13 smithi167 conmon[60316]: debug 2022-01-31T21:01:13.153+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:13.155325+0000) 2022-01-31T21:01:13.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:13 smithi171 conmon[46715]: debug 2022-01-31T21:01:13.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:13.432553+0000) 2022-01-31T21:01:13.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:13 smithi171 conmon[51620]: debug 2022-01-31T21:01:13.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:13.509779+0000) 2022-01-31T21:01:13.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:13 smithi167 conmon[49112]: debug 2022-01-31T21:01:13.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:13.476157+0000) 2022-01-31T21:01:14.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:13 smithi171 conmon[41853]: debug 2022-01-31T21:01:13.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:13.637105+0000) 2022-01-31T21:01:14.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:14 smithi167 conmon[54076]: debug 2022-01-31T21:01:14.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:14.403782+0000) 2022-01-31T21:01:14.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:14 smithi167 conmon[60316]: debug 2022-01-31T21:01:14.154+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:14.155467+0000) 2022-01-31T21:01:14.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:14 smithi171 conmon[46715]: debug 2022-01-31T21:01:14.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:14.432711+0000) 2022-01-31T21:01:14.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:14 smithi171 conmon[51620]: debug 2022-01-31T21:01:14.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:14.509959+0000) 2022-01-31T21:01:14.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:14 smithi167 conmon[49112]: debug 2022-01-31T21:01:14.474+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:14.476351+0000) 2022-01-31T21:01:15.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:14 smithi171 conmon[41853]: debug 2022-01-31T21:01:14.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:14.637315+0000) 2022-01-31T21:01:15.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:15 smithi167 conmon[49112]: debug 2022-01-31T21:01:15.312+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.314431+0000) 2022-01-31T21:01:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:15 smithi167 conmon[54076]: debug 2022-01-31T21:01:15.313+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.314736+0000) 2022-01-31T21:01:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:15 smithi167 conmon[54076]: debug 2022-01-31T21:01:15.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.403960+0000) 2022-01-31T21:01:15.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:15 smithi167 conmon[60316]: debug 2022-01-31T21:01:15.154+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.155681+0000) 2022-01-31T21:01:15.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:15 smithi167 conmon[60316]: debug 2022-01-31T21:01:15.312+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.314116+0000) 2022-01-31T21:01:15.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[35325]: debug 2022-01-31T21:01:15.322+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96666 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[41853]: debug 2022-01-31T21:01:15.311+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.313653+0000) 2022-01-31T21:01:15.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[51620]: debug 2022-01-31T21:01:15.311+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.313444+0000) 2022-01-31T21:01:15.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[51620]: debug 2022-01-31T21:01:15.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.510125+0000) 2022-01-31T21:01:15.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[46715]: debug 2022-01-31T21:01:15.312+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.314384+0000) 2022-01-31T21:01:15.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[46715]: debug 2022-01-31T21:01:15.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.432885+0000) 2022-01-31T21:01:15.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:15 smithi167 conmon[49112]: debug 2022-01-31T21:01:15.475+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.476543+0000) 2022-01-31T21:01:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:15 smithi171 conmon[41853]: debug 2022-01-31T21:01:15.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:15.637462+0000) 2022-01-31T21:01:16.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:16 smithi167 conmon[54076]: debug 2022-01-31T21:01:16.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:16.404119+0000) 2022-01-31T21:01:16.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:16 smithi167 conmon[60316]: debug 2022-01-31T21:01:16.154+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:16.155810+0000) 2022-01-31T21:01:16.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:16 smithi171 conmon[46715]: debug 2022-01-31T21:01:16.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:16.433026+0000) 2022-01-31T21:01:16.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:16 smithi171 conmon[51620]: debug 2022-01-31T21:01:16.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:16.510223+0000) 2022-01-31T21:01:16.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:16 smithi167 conmon[49112]: debug 2022-01-31T21:01:16.475+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:16.476723+0000) 2022-01-31T21:01:17.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:16 smithi171 conmon[41853]: debug 2022-01-31T21:01:16.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:16.637603+0000) 2022-01-31T21:01:17.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:17 smithi167 conmon[54076]: debug 2022-01-31T21:01:17.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:17.404298+0000) 2022-01-31T21:01:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:17 smithi167 conmon[60316]: debug 2022-01-31T21:01:17.155+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:17.155963+0000) 2022-01-31T21:01:17.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:17 smithi171 conmon[46715]: debug 2022-01-31T21:01:17.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:17.433195+0000) 2022-01-31T21:01:17.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:17 smithi171 conmon[51620]: debug 2022-01-31T21:01:17.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:17.510350+0000) 2022-01-31T21:01:17.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:17 smithi167 conmon[49112]: debug 2022-01-31T21:01:17.475+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:17.476881+0000) 2022-01-31T21:01:18.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:17 smithi171 conmon[41853]: debug 2022-01-31T21:01:17.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:17.637822+0000) 2022-01-31T21:01:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:18 smithi167 conmon[54076]: debug 2022-01-31T21:01:18.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:18.404494+0000) 2022-01-31T21:01:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:18 smithi167 conmon[60316]: debug 2022-01-31T21:01:18.154+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:18.156166+0000) 2022-01-31T21:01:18.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:18 smithi171 conmon[51620]: debug 2022-01-31T21:01:18.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:18.510506+0000) 2022-01-31T21:01:18.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:18 smithi171 conmon[46715]: debug 2022-01-31T21:01:18.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:18.433389+0000) 2022-01-31T21:01:18.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:18 smithi167 conmon[49112]: debug 2022-01-31T21:01:18.475+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:18.477086+0000) 2022-01-31T21:01:19.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:18 smithi171 conmon[41853]: debug 2022-01-31T21:01:18.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:18.638007+0000) 2022-01-31T21:01:19.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:19 smithi167 conmon[54076]: debug 2022-01-31T21:01:19.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:19.404692+0000) 2022-01-31T21:01:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:19 smithi167 conmon[60316]: debug 2022-01-31T21:01:19.155+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:19.156332+0000) 2022-01-31T21:01:19.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:19 smithi171 conmon[46715]: debug 2022-01-31T21:01:19.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:19.433599+0000) 2022-01-31T21:01:19.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:19 smithi171 conmon[51620]: debug 2022-01-31T21:01:19.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:19.510709+0000) 2022-01-31T21:01:19.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:19 smithi167 conmon[49112]: debug 2022-01-31T21:01:19.476+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:19.477244+0000) 2022-01-31T21:01:20.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:19 smithi171 conmon[41853]: debug 2022-01-31T21:01:19.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:19.638205+0000) 2022-01-31T21:01:20.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:20 smithi167 conmon[49112]: debug 2022-01-31T21:01:20.324+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.326440+0000) 2022-01-31T21:01:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:20 smithi167 conmon[54076]: debug 2022-01-31T21:01:20.325+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.326910+0000) 2022-01-31T21:01:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:20 smithi167 conmon[54076]: debug 2022-01-31T21:01:20.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.404918+0000) 2022-01-31T21:01:20.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:20 smithi167 conmon[60316]: debug 2022-01-31T21:01:20.155+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.156531+0000) 2022-01-31T21:01:20.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:20 smithi167 conmon[60316]: debug 2022-01-31T21:01:20.326+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.327836+0000) 2022-01-31T21:01:20.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[35325]: debug 2022-01-31T21:01:20.335+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96783 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[41853]: debug 2022-01-31T21:01:20.325+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.327071+0000) 2022-01-31T21:01:20.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[46715]: debug 2022-01-31T21:01:20.324+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.326055+0000) 2022-01-31T21:01:20.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[46715]: debug 2022-01-31T21:01:20.432+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.433808+0000) 2022-01-31T21:01:20.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[51620]: debug 2022-01-31T21:01:20.324+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.325849+0000) 2022-01-31T21:01:20.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[51620]: debug 2022-01-31T21:01:20.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.510935+0000) 2022-01-31T21:01:20.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:20 smithi167 conmon[49112]: debug 2022-01-31T21:01:20.476+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.477415+0000) 2022-01-31T21:01:21.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:20 smithi171 conmon[41853]: debug 2022-01-31T21:01:20.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:20.638384+0000) 2022-01-31T21:01:21.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:21 smithi167 conmon[60316]: debug 2022-01-31T21:01:21.155+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:21.156714+0000) 2022-01-31T21:01:21.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:21 smithi167 conmon[54076]: debug 2022-01-31T21:01:21.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:21.405064+0000) 2022-01-31T21:01:21.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:21 smithi171 conmon[46715]: debug 2022-01-31T21:01:21.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:21.433968+0000) 2022-01-31T21:01:21.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:21 smithi171 conmon[51620]: debug 2022-01-31T21:01:21.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:21.511098+0000) 2022-01-31T21:01:21.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:21 smithi167 conmon[49112]: debug 2022-01-31T21:01:21.476+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:21.477631+0000) 2022-01-31T21:01:22.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:21 smithi171 conmon[41853]: debug 2022-01-31T21:01:21.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:21.638531+0000) 2022-01-31T21:01:22.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:22 smithi167 conmon[54076]: debug 2022-01-31T21:01:22.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:22.405268+0000) 2022-01-31T21:01:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:22 smithi167 conmon[60316]: debug 2022-01-31T21:01:22.156+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:22.156884+0000) 2022-01-31T21:01:22.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:22 smithi171 conmon[46715]: debug 2022-01-31T21:01:22.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:22.434138+0000) 2022-01-31T21:01:22.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:22 smithi171 conmon[51620]: debug 2022-01-31T21:01:22.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:22.511293+0000) 2022-01-31T21:01:22.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:22 smithi167 conmon[49112]: debug 2022-01-31T21:01:22.476+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:22.477734+0000) 2022-01-31T21:01:23.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:22 smithi171 conmon[41853]: debug 2022-01-31T21:01:22.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:22.638725+0000) 2022-01-31T21:01:23.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:23 smithi167 conmon[54076]: debug 2022-01-31T21:01:23.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:23.405444+0000) 2022-01-31T21:01:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:23 smithi167 conmon[60316]: debug 2022-01-31T21:01:23.156+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:23.157076+0000) 2022-01-31T21:01:23.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:23 smithi171 conmon[46715]: debug 2022-01-31T21:01:23.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:23.434308+0000) 2022-01-31T21:01:23.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:23 smithi171 conmon[51620]: debug 2022-01-31T21:01:23.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:23.511446+0000) 2022-01-31T21:01:23.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:23 smithi167 conmon[49112]: debug 2022-01-31T21:01:23.476+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:23.477860+0000) 2022-01-31T21:01:24.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:23 smithi171 conmon[35325]: debug 2022-01-31T21:01:23.609+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:01:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:23 smithi171 conmon[41853]: debug 2022-01-31T21:01:23.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:23.638897+0000) 2022-01-31T21:01:24.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:24 smithi167 conmon[54076]: debug 2022-01-31T21:01:24.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:24.405606+0000) 2022-01-31T21:01:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:24 smithi167 conmon[60316]: debug 2022-01-31T21:01:24.156+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:24.157280+0000) 2022-01-31T21:01:24.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:24 smithi171 conmon[51620]: debug 2022-01-31T21:01:24.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:24.511636+0000) 2022-01-31T21:01:24.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:24 smithi171 conmon[46715]: debug 2022-01-31T21:01:24.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:24.434518+0000) 2022-01-31T21:01:24.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:24 smithi167 conmon[49112]: debug 2022-01-31T21:01:24.477+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:24.478060+0000) 2022-01-31T21:01:25.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:24 smithi171 conmon[41853]: debug 2022-01-31T21:01:24.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:24.639078+0000) 2022-01-31T21:01:25.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:25 smithi167 conmon[49112]: debug 2022-01-31T21:01:25.336+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.338343+0000) 2022-01-31T21:01:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:25 smithi167 conmon[54076]: debug 2022-01-31T21:01:25.337+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.339120+0000) 2022-01-31T21:01:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:25 smithi167 conmon[54076]: debug 2022-01-31T21:01:25.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.405798+0000) 2022-01-31T21:01:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:25 smithi167 conmon[60316]: debug 2022-01-31T21:01:25.156+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.157464+0000) 2022-01-31T21:01:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:25 smithi167 conmon[60316]: debug 2022-01-31T21:01:25.338+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.339996+0000) 2022-01-31T21:01:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[35325]: debug 2022-01-31T21:01:25.347+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 96893 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[41853]: debug 2022-01-31T21:01:25.337+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.338422+0000) 2022-01-31T21:01:25.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[51620]: debug 2022-01-31T21:01:25.337+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.338149+0000) 2022-01-31T21:01:25.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[51620]: debug 2022-01-31T21:01:25.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.511836+0000) 2022-01-31T21:01:25.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[46715]: debug 2022-01-31T21:01:25.338+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.338823+0000) 2022-01-31T21:01:25.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[46715]: debug 2022-01-31T21:01:25.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.434705+0000) 2022-01-31T21:01:25.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:25 smithi167 conmon[49112]: debug 2022-01-31T21:01:25.477+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.478254+0000) 2022-01-31T21:01:26.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:25 smithi171 conmon[41853]: debug 2022-01-31T21:01:25.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:25.639257+0000) 2022-01-31T21:01:26.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:26 smithi167 conmon[54076]: debug 2022-01-31T21:01:26.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:26.405967+0000) 2022-01-31T21:01:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:26 smithi167 conmon[60316]: debug 2022-01-31T21:01:26.156+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:26.157667+0000) 2022-01-31T21:01:26.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:26 smithi171 conmon[46715]: debug 2022-01-31T21:01:26.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:26.434850+0000) 2022-01-31T21:01:26.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:26 smithi171 conmon[51620]: debug 2022-01-31T21:01:26.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:26.511995+0000) 2022-01-31T21:01:26.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:26 smithi167 conmon[49112]: debug 2022-01-31T21:01:26.477+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:26.478404+0000) 2022-01-31T21:01:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:26 smithi171 conmon[41853]: debug 2022-01-31T21:01:26.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:26.639391+0000) 2022-01-31T21:01:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:27 smithi167 conmon[60316]: debug 2022-01-31T21:01:27.157+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:27.157816+0000) 2022-01-31T21:01:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:27 smithi167 conmon[54076]: debug 2022-01-31T21:01:27.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:27.406121+0000) 2022-01-31T21:01:27.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:27 smithi171 conmon[46715]: debug 2022-01-31T21:01:27.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:27.434999+0000) 2022-01-31T21:01:27.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:27 smithi171 conmon[51620]: debug 2022-01-31T21:01:27.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:27.512178+0000) 2022-01-31T21:01:27.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:27 smithi167 conmon[49112]: debug 2022-01-31T21:01:27.477+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:27.478556+0000) 2022-01-31T21:01:28.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:27 smithi171 conmon[41853]: debug 2022-01-31T21:01:27.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:27.639546+0000) 2022-01-31T21:01:28.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:28 smithi167 conmon[54076]: debug 2022-01-31T21:01:28.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:28.406261+0000) 2022-01-31T21:01:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:28 smithi167 conmon[60316]: debug 2022-01-31T21:01:28.157+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:28.157972+0000) 2022-01-31T21:01:28.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:28 smithi171 conmon[46715]: debug 2022-01-31T21:01:28.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:28.435165+0000) 2022-01-31T21:01:28.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:28 smithi171 conmon[51620]: debug 2022-01-31T21:01:28.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:28.512398+0000) 2022-01-31T21:01:28.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:28 smithi167 conmon[49112]: debug 2022-01-31T21:01:28.478+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:28.478742+0000) 2022-01-31T21:01:29.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:28 smithi171 conmon[41853]: debug 2022-01-31T21:01:28.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:28.639783+0000) 2022-01-31T21:01:29.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:29 smithi167 conmon[54076]: debug 2022-01-31T21:01:29.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:29.406432+0000) 2022-01-31T21:01:29.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:29 smithi167 conmon[60316]: debug 2022-01-31T21:01:29.157+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:29.158161+0000) 2022-01-31T21:01:29.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:29 smithi171 conmon[46715]: debug 2022-01-31T21:01:29.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:29.435299+0000) 2022-01-31T21:01:29.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:29 smithi171 conmon[51620]: debug 2022-01-31T21:01:29.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:29.512582+0000) 2022-01-31T21:01:29.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:29 smithi167 conmon[49112]: debug 2022-01-31T21:01:29.477+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:29.478931+0000) 2022-01-31T21:01:30.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:29 smithi171 conmon[41853]: debug 2022-01-31T21:01:29.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:29.639968+0000) 2022-01-31T21:01:30.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:30 smithi167 conmon[49112]: debug 2022-01-31T21:01:30.349+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.350852+0000) 2022-01-31T21:01:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:30 smithi167 conmon[54076]: debug 2022-01-31T21:01:30.350+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.351554+0000) 2022-01-31T21:01:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:30 smithi167 conmon[54076]: debug 2022-01-31T21:01:30.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.406611+0000) 2022-01-31T21:01:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:30 smithi167 conmon[60316]: debug 2022-01-31T21:01:30.157+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.158413+0000) 2022-01-31T21:01:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:30 smithi167 conmon[60316]: debug 2022-01-31T21:01:30.349+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.350990+0000) 2022-01-31T21:01:30.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[41853]: debug 2022-01-31T21:01:30.350+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.351168+0000) 2022-01-31T21:01:30.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[35325]: debug 2022-01-31T21:01:30.360+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97003 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:30.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[46715]: debug 2022-01-31T21:01:30.351+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.351972+0000) 2022-01-31T21:01:30.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[46715]: debug 2022-01-31T21:01:30.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.435528+0000) 2022-01-31T21:01:30.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[51620]: debug 2022-01-31T21:01:30.349+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.350238+0000) 2022-01-31T21:01:30.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[51620]: debug 2022-01-31T21:01:30.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.512773+0000) 2022-01-31T21:01:30.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:30 smithi167 conmon[49112]: debug 2022-01-31T21:01:30.478+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.479143+0000) 2022-01-31T21:01:31.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:30 smithi171 conmon[41853]: debug 2022-01-31T21:01:30.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:30.640181+0000) 2022-01-31T21:01:31.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:31 smithi167 conmon[54076]: debug 2022-01-31T21:01:31.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:31.406783+0000) 2022-01-31T21:01:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:31 smithi167 conmon[60316]: debug 2022-01-31T21:01:31.157+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:31.158548+0000) 2022-01-31T21:01:31.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:31 smithi171 conmon[51620]: debug 2022-01-31T21:01:31.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:31.512886+0000) 2022-01-31T21:01:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:31 smithi171 conmon[46715]: debug 2022-01-31T21:01:31.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:31.435686+0000) 2022-01-31T21:01:31.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:31 smithi167 conmon[49112]: debug 2022-01-31T21:01:31.478+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:31.479293+0000) 2022-01-31T21:01:32.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:31 smithi171 conmon[41853]: debug 2022-01-31T21:01:31.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:31.640306+0000) 2022-01-31T21:01:32.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:32 smithi167 conmon[54076]: debug 2022-01-31T21:01:32.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:32.406961+0000) 2022-01-31T21:01:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:32 smithi167 conmon[60316]: debug 2022-01-31T21:01:32.158+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:32.158687+0000) 2022-01-31T21:01:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:32 smithi171 conmon[46715]: debug 2022-01-31T21:01:32.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:32.435839+0000) 2022-01-31T21:01:32.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:32 smithi171 conmon[51620]: debug 2022-01-31T21:01:32.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:32.513068+0000) 2022-01-31T21:01:32.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:32 smithi167 conmon[49112]: debug 2022-01-31T21:01:32.478+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:32.479519+0000) 2022-01-31T21:01:33.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:32 smithi171 conmon[41853]: debug 2022-01-31T21:01:32.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:32.640467+0000) 2022-01-31T21:01:33.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:33 smithi167 conmon[54076]: debug 2022-01-31T21:01:33.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:33.407184+0000) 2022-01-31T21:01:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:33 smithi167 conmon[60316]: debug 2022-01-31T21:01:33.158+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:33.158884+0000) 2022-01-31T21:01:33.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:33 smithi171 conmon[46715]: debug 2022-01-31T21:01:33.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:33.435994+0000) 2022-01-31T21:01:33.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:33 smithi171 conmon[51620]: debug 2022-01-31T21:01:33.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:33.513268+0000) 2022-01-31T21:01:33.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:33 smithi167 conmon[49112]: debug 2022-01-31T21:01:33.479+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:33.479703+0000) 2022-01-31T21:01:34.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:33 smithi171 conmon[41853]: debug 2022-01-31T21:01:33.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:33.640666+0000) 2022-01-31T21:01:34.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:34 smithi167 conmon[54076]: debug 2022-01-31T21:01:34.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:34.407385+0000) 2022-01-31T21:01:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:34 smithi167 conmon[60316]: debug 2022-01-31T21:01:34.158+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:34.159080+0000) 2022-01-31T21:01:34.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:34 smithi171 conmon[46715]: debug 2022-01-31T21:01:34.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:34.436179+0000) 2022-01-31T21:01:34.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:34 smithi171 conmon[51620]: debug 2022-01-31T21:01:34.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:34.513444+0000) 2022-01-31T21:01:34.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:34 smithi167 conmon[49112]: debug 2022-01-31T21:01:34.479+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:34.479913+0000) 2022-01-31T21:01:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:34 smithi171 conmon[41853]: debug 2022-01-31T21:01:34.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:34.640803+0000) 2022-01-31T21:01:35.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:35 smithi167 conmon[49112]: debug 2022-01-31T21:01:35.363+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.363879+0000) 2022-01-31T21:01:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:35 smithi167 conmon[60316]: debug 2022-01-31T21:01:35.158+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.159245+0000) 2022-01-31T21:01:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:35 smithi167 conmon[60316]: debug 2022-01-31T21:01:35.364+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.365060+0000) 2022-01-31T21:01:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:35 smithi167 conmon[54076]: debug 2022-01-31T21:01:35.364+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.364911+0000) 2022-01-31T21:01:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:35 smithi167 conmon[54076]: debug 2022-01-31T21:01:35.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.407577+0000) 2022-01-31T21:01:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[35325]: debug 2022-01-31T21:01:35.374+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97113 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[41853]: debug 2022-01-31T21:01:35.362+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.363582+0000) 2022-01-31T21:01:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[46715]: debug 2022-01-31T21:01:35.363+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.364018+0000) 2022-01-31T21:01:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[46715]: debug 2022-01-31T21:01:35.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.436306+0000) 2022-01-31T21:01:35.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[51620]: debug 2022-01-31T21:01:35.362+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.363651+0000) 2022-01-31T21:01:35.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[51620]: debug 2022-01-31T21:01:35.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.513645+0000) 2022-01-31T21:01:35.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:35 smithi167 conmon[49112]: debug 2022-01-31T21:01:35.479+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.480126+0000) 2022-01-31T21:01:36.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:35 smithi171 conmon[41853]: debug 2022-01-31T21:01:35.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:35.640919+0000) 2022-01-31T21:01:36.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:36 smithi167 conmon[54076]: debug 2022-01-31T21:01:36.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:36.407724+0000) 2022-01-31T21:01:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:36 smithi167 conmon[60316]: debug 2022-01-31T21:01:36.158+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:36.159428+0000) 2022-01-31T21:01:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:36 smithi167 conmon[60316]: 2022-01-31T21:01:36.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:36 smithi171 conmon[46715]: debug 2022-01-31T21:01:36.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:36.436419+0000) 2022-01-31T21:01:36.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:36 smithi171 conmon[51620]: debug 2022-01-31T21:01:36.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:36.513788+0000) 2022-01-31T21:01:36.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:36 smithi167 conmon[49112]: debug 2022-01-31T21:01:36.479+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:36.480260+0000) 2022-01-31T21:01:37.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:36 smithi171 conmon[41853]: debug 2022-01-31T21:01:36.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:36.641049+0000) 2022-01-31T21:01:37.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:37 smithi167 conmon[54076]: debug 2022-01-31T21:01:37.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:37.407879+0000) 2022-01-31T21:01:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:37 smithi167 conmon[60316]: debug 2022-01-31T21:01:37.159+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:37.159583+0000) 2022-01-31T21:01:37.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:37 smithi171 conmon[46715]: debug 2022-01-31T21:01:37.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:37.436577+0000) 2022-01-31T21:01:37.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:37 smithi171 conmon[51620]: debug 2022-01-31T21:01:37.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:37.513908+0000) 2022-01-31T21:01:37.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:37 smithi167 conmon[49112]: debug 2022-01-31T21:01:37.479+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:37.480396+0000) 2022-01-31T21:01:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:37 smithi171 conmon[41853]: debug 2022-01-31T21:01:37.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:37.641201+0000) 2022-01-31T21:01:38.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:38 smithi167 conmon[54076]: debug 2022-01-31T21:01:38.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:38.408076+0000) 2022-01-31T21:01:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:38 smithi167 conmon[60316]: debug 2022-01-31T21:01:38.159+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:38.159763+0000) 2022-01-31T21:01:38.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:38 smithi171 conmon[46715]: debug 2022-01-31T21:01:38.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:38.436805+0000) 2022-01-31T21:01:38.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:38 smithi171 conmon[51620]: debug 2022-01-31T21:01:38.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:38.514044+0000) 2022-01-31T21:01:38.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:38 smithi167 conmon[49112]: debug 2022-01-31T21:01:38.480+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:38.480565+0000) 2022-01-31T21:01:39.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:38 smithi171 conmon[35325]: debug 2022-01-31T21:01:38.610+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:01:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:38 smithi171 conmon[41853]: debug 2022-01-31T21:01:38.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:38.641378+0000) 2022-01-31T21:01:39.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:39 smithi167 conmon[54076]: debug 2022-01-31T21:01:39.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:39.408265+0000) 2022-01-31T21:01:39.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:39 smithi167 conmon[60316]: debug 2022-01-31T21:01:39.159+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:39.159934+0000) 2022-01-31T21:01:39.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:39 smithi171 conmon[46715]: debug 2022-01-31T21:01:39.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:39.437023+0000) 2022-01-31T21:01:39.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:39 smithi171 conmon[51620]: debug 2022-01-31T21:01:39.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:39.514203+0000) 2022-01-31T21:01:39.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:39 smithi167 conmon[49112]: debug 2022-01-31T21:01:39.480+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:39.480761+0000) 2022-01-31T21:01:40.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:39 smithi171 conmon[41853]: debug 2022-01-31T21:01:39.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:39.641559+0000) 2022-01-31T21:01:40.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:40 smithi167 conmon[49112]: debug 2022-01-31T21:01:40.376+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.377275+0000) 2022-01-31T21:01:40.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:40 smithi167 conmon[60316]: debug 2022-01-31T21:01:40.159+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.160161+0000) 2022-01-31T21:01:40.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:40 smithi167 conmon[60316]: debug 2022-01-31T21:01:40.376+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.377462+0000) 2022-01-31T21:01:40.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:40 smithi167 conmon[54076]: debug 2022-01-31T21:01:40.377+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.378328+0000) 2022-01-31T21:01:40.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:40 smithi167 conmon[54076]: debug 2022-01-31T21:01:40.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.408401+0000) 2022-01-31T21:01:40.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[35325]: debug 2022-01-31T21:01:40.386+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97223 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[41853]: debug 2022-01-31T21:01:40.377+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.378277+0000) 2022-01-31T21:01:40.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[46715]: debug 2022-01-31T21:01:40.376+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.377563+0000) 2022-01-31T21:01:40.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[46715]: debug 2022-01-31T21:01:40.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.437221+0000) 2022-01-31T21:01:40.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[51620]: debug 2022-01-31T21:01:40.376+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.376939+0000) 2022-01-31T21:01:40.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[51620]: debug 2022-01-31T21:01:40.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.514392+0000) 2022-01-31T21:01:40.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:40 smithi167 conmon[49112]: debug 2022-01-31T21:01:40.480+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.480963+0000) 2022-01-31T21:01:41.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:40 smithi171 conmon[41853]: debug 2022-01-31T21:01:40.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:40.641708+0000) 2022-01-31T21:01:41.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:41 smithi167 conmon[54076]: debug 2022-01-31T21:01:41.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:41.408598+0000) 2022-01-31T21:01:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:41 smithi167 conmon[60316]: debug 2022-01-31T21:01:41.159+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:41.160346+0000) 2022-01-31T21:01:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:41 smithi171 conmon[46715]: debug 2022-01-31T21:01:41.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:41.437377+0000) 2022-01-31T21:01:41.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:41 smithi171 conmon[51620]: debug 2022-01-31T21:01:41.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:41.514548+0000) 2022-01-31T21:01:41.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:41 smithi167 conmon[49112]: debug 2022-01-31T21:01:41.480+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:41.481137+0000) 2022-01-31T21:01:42.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:41 smithi171 conmon[41853]: debug 2022-01-31T21:01:41.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:41.641856+0000) 2022-01-31T21:01:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:42 smithi167 conmon[60316]: debug 2022-01-31T21:01:42.160+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:42.160522+0000) 2022-01-31T21:01:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:42 smithi167 conmon[54076]: debug 2022-01-31T21:01:42.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:42.408746+0000) 2022-01-31T21:01:42.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:42 smithi171 conmon[46715]: debug 2022-01-31T21:01:42.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:42.437567+0000) 2022-01-31T21:01:42.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:42 smithi171 conmon[51620]: debug 2022-01-31T21:01:42.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:42.514735+0000) 2022-01-31T21:01:42.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:42 smithi167 conmon[49112]: debug 2022-01-31T21:01:42.480+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:42.481364+0000) 2022-01-31T21:01:43.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:42 smithi171 conmon[41853]: debug 2022-01-31T21:01:42.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:42.642004+0000) 2022-01-31T21:01:43.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:43 smithi167 conmon[54076]: debug 2022-01-31T21:01:43.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:43.408902+0000) 2022-01-31T21:01:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:43 smithi167 conmon[60316]: debug 2022-01-31T21:01:43.160+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:43.160722+0000) 2022-01-31T21:01:43.609 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:43 smithi171 conmon[46715]: debug 2022-01-31T21:01:43.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:43.437784+0000) 2022-01-31T21:01:43.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:43 smithi171 conmon[51620]: debug 2022-01-31T21:01:43.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:43.514881+0000) 2022-01-31T21:01:43.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:43 smithi167 conmon[49112]: debug 2022-01-31T21:01:43.481+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:43.481539+0000) 2022-01-31T21:01:44.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:43 smithi171 conmon[41853]: debug 2022-01-31T21:01:43.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:43.642161+0000) 2022-01-31T21:01:44.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:44 smithi167 conmon[54076]: debug 2022-01-31T21:01:44.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:44.409099+0000) 2022-01-31T21:01:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:44 smithi167 conmon[60316]: debug 2022-01-31T21:01:44.160+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:44.160934+0000) 2022-01-31T21:01:44.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:44 smithi171 conmon[46715]: debug 2022-01-31T21:01:44.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:44.437945+0000) 2022-01-31T21:01:44.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:44 smithi171 conmon[51620]: debug 2022-01-31T21:01:44.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:44.515056+0000) 2022-01-31T21:01:44.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:44 smithi167 conmon[49112]: debug 2022-01-31T21:01:44.481+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:44.481749+0000) 2022-01-31T21:01:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:44 smithi171 conmon[41853]: debug 2022-01-31T21:01:44.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:44.642313+0000) 2022-01-31T21:01:45.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:45 smithi167 conmon[49112]: debug 2022-01-31T21:01:45.390+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.390611+0000) 2022-01-31T21:01:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:45 smithi167 conmon[60316]: debug 2022-01-31T21:01:45.160+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.161116+0000) 2022-01-31T21:01:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:45 smithi167 conmon[60316]: debug 2022-01-31T21:01:45.389+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.389619+0000) 2022-01-31T21:01:45.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:45 smithi167 conmon[54076]: debug 2022-01-31T21:01:45.390+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.391083+0000) 2022-01-31T21:01:45.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:45 smithi167 conmon[54076]: debug 2022-01-31T21:01:45.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.409231+0000) 2022-01-31T21:01:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[35325]: debug 2022-01-31T21:01:45.398+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97333 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[41853]: debug 2022-01-31T21:01:45.389+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.390690+0000) 2022-01-31T21:01:45.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[51620]: debug 2022-01-31T21:01:45.389+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.389781+0000) 2022-01-31T21:01:45.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[51620]: debug 2022-01-31T21:01:45.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.515238+0000) 2022-01-31T21:01:45.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[46715]: debug 2022-01-31T21:01:45.389+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.390244+0000) 2022-01-31T21:01:45.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[46715]: debug 2022-01-31T21:01:45.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.438080+0000) 2022-01-31T21:01:45.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:45 smithi167 conmon[49112]: debug 2022-01-31T21:01:45.481+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.481897+0000) 2022-01-31T21:01:46.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:45 smithi171 conmon[41853]: debug 2022-01-31T21:01:45.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:45.642475+0000) 2022-01-31T21:01:46.423 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:46 smithi167 conmon[54076]: debug 2022-01-31T21:01:46.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:46.409381+0000) 2022-01-31T21:01:46.423 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:46 smithi167 conmon[60316]: debug 2022-01-31T21:01:46.160+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:46.161294+0000) 2022-01-31T21:01:46.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:46 smithi171 conmon[51620]: debug 2022-01-31T21:01:46.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:46.515403+0000) 2022-01-31T21:01:46.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:46 smithi171 conmon[46715]: debug 2022-01-31T21:01:46.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:46.438221+0000) 2022-01-31T21:01:46.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:46 smithi167 conmon[49112]: debug 2022-01-31T21:01:46.481+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:46.482038+0000) 2022-01-31T21:01:47.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:46 smithi171 conmon[41853]: debug 2022-01-31T21:01:46.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:46.642643+0000) 2022-01-31T21:01:47.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:47 smithi167 conmon[54076]: debug 2022-01-31T21:01:47.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:47.409557+0000) 2022-01-31T21:01:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:47 smithi167 conmon[60316]: debug 2022-01-31T21:01:47.161+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:47.161455+0000) 2022-01-31T21:01:47.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:47 smithi171 conmon[51620]: debug 2022-01-31T21:01:47.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:47.515583+0000) 2022-01-31T21:01:47.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:47 smithi171 conmon[46715]: debug 2022-01-31T21:01:47.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:47.438411+0000) 2022-01-31T21:01:47.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:47 smithi167 conmon[49112]: debug 2022-01-31T21:01:47.481+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:47.482188+0000) 2022-01-31T21:01:48.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:47 smithi171 conmon[41853]: debug 2022-01-31T21:01:47.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:47.642855+0000) 2022-01-31T21:01:48.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:48 smithi167 conmon[60316]: debug 2022-01-31T21:01:48.161+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:48.161654+0000) 2022-01-31T21:01:48.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:48 smithi167 conmon[54076]: debug 2022-01-31T21:01:48.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:48.409747+0000) 2022-01-31T21:01:48.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:48 smithi171 conmon[51620]: debug 2022-01-31T21:01:48.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:48.515764+0000) 2022-01-31T21:01:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:48 smithi171 conmon[46715]: debug 2022-01-31T21:01:48.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:48.438609+0000) 2022-01-31T21:01:48.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:48 smithi167 conmon[49112]: debug 2022-01-31T21:01:48.481+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:48.482329+0000) 2022-01-31T21:01:49.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:48 smithi171 conmon[41853]: debug 2022-01-31T21:01:48.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:48.643032+0000) 2022-01-31T21:01:49.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:49 smithi167 conmon[54076]: debug 2022-01-31T21:01:49.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:49.409961+0000) 2022-01-31T21:01:49.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:49 smithi167 conmon[60316]: debug 2022-01-31T21:01:49.161+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:49.161796+0000) 2022-01-31T21:01:49.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:49 smithi171 conmon[51620]: debug 2022-01-31T21:01:49.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:49.515916+0000) 2022-01-31T21:01:49.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:49 smithi171 conmon[46715]: debug 2022-01-31T21:01:49.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:49.438805+0000) 2022-01-31T21:01:49.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:49 smithi167 conmon[49112]: debug 2022-01-31T21:01:49.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:49.482551+0000) 2022-01-31T21:01:50.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:49 smithi171 conmon[41853]: debug 2022-01-31T21:01:49.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:49.643183+0000) 2022-01-31T21:01:50.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:50 smithi167 conmon[49112]: debug 2022-01-31T21:01:50.402+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.403186+0000) 2022-01-31T21:01:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:50 smithi167 conmon[54076]: debug 2022-01-31T21:01:50.402+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.403128+0000) 2022-01-31T21:01:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:50 smithi167 conmon[54076]: debug 2022-01-31T21:01:50.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.410116+0000) 2022-01-31T21:01:50.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:50 smithi167 conmon[60316]: debug 2022-01-31T21:01:50.161+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.162024+0000) 2022-01-31T21:01:50.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:50 smithi167 conmon[60316]: debug 2022-01-31T21:01:50.403+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.403494+0000) 2022-01-31T21:01:50.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[35325]: debug 2022-01-31T21:01:50.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97443 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[41853]: debug 2022-01-31T21:01:50.402+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.403303+0000) 2022-01-31T21:01:50.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[51620]: debug 2022-01-31T21:01:50.400+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.401684+0000) 2022-01-31T21:01:50.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[51620]: debug 2022-01-31T21:01:50.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.516070+0000) 2022-01-31T21:01:50.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[46715]: debug 2022-01-31T21:01:50.401+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.402252+0000) 2022-01-31T21:01:50.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[46715]: debug 2022-01-31T21:01:50.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.438982+0000) 2022-01-31T21:01:50.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:50 smithi167 conmon[49112]: debug 2022-01-31T21:01:50.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.482707+0000) 2022-01-31T21:01:51.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:50 smithi171 conmon[41853]: debug 2022-01-31T21:01:50.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:50.643366+0000) 2022-01-31T21:01:51.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:51 smithi167 conmon[54076]: debug 2022-01-31T21:01:51.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:51.410260+0000) 2022-01-31T21:01:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:51 smithi167 conmon[60316]: debug 2022-01-31T21:01:51.161+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:51.162190+0000) 2022-01-31T21:01:51.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:51 smithi171 conmon[46715]: debug 2022-01-31T21:01:51.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:51.439137+0000) 2022-01-31T21:01:51.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:51 smithi171 conmon[51620]: debug 2022-01-31T21:01:51.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:51.516214+0000) 2022-01-31T21:01:51.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:51 smithi167 conmon[49112]: debug 2022-01-31T21:01:51.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:51.482852+0000) 2022-01-31T21:01:52.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:51 smithi171 conmon[41853]: debug 2022-01-31T21:01:51.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:51.643514+0000) 2022-01-31T21:01:52.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:52 smithi167 conmon[54076]: debug 2022-01-31T21:01:52.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:52.410441+0000) 2022-01-31T21:01:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:52 smithi167 conmon[60316]: debug 2022-01-31T21:01:52.161+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:52.162364+0000) 2022-01-31T21:01:52.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:52 smithi171 conmon[51620]: debug 2022-01-31T21:01:52.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:52.516391+0000) 2022-01-31T21:01:52.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:52 smithi171 conmon[46715]: debug 2022-01-31T21:01:52.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:52.439309+0000) 2022-01-31T21:01:52.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:52 smithi167 conmon[49112]: debug 2022-01-31T21:01:52.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:52.483056+0000) 2022-01-31T21:01:53.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:52 smithi171 conmon[41853]: debug 2022-01-31T21:01:52.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:52.643714+0000) 2022-01-31T21:01:53.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:53 smithi167 conmon[60316]: debug 2022-01-31T21:01:53.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:53.162526+0000) 2022-01-31T21:01:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:53 smithi167 conmon[54076]: debug 2022-01-31T21:01:53.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:53.410602+0000) 2022-01-31T21:01:53.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:53 smithi171 conmon[46715]: debug 2022-01-31T21:01:53.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:53.439486+0000) 2022-01-31T21:01:53.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:53 smithi171 conmon[51620]: debug 2022-01-31T21:01:53.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:53.516585+0000) 2022-01-31T21:01:53.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:53 smithi167 conmon[49112]: debug 2022-01-31T21:01:53.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:53.483255+0000) 2022-01-31T21:01:54.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:53 smithi171 conmon[41853]: debug 2022-01-31T21:01:53.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:53.643879+0000) 2022-01-31T21:01:54.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:53 smithi171 conmon[35325]: debug 2022-01-31T21:01:53.611+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:01:54.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:54 smithi167 conmon[54076]: debug 2022-01-31T21:01:54.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:54.410747+0000) 2022-01-31T21:01:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:54 smithi167 conmon[60316]: debug 2022-01-31T21:01:54.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:54.162669+0000) 2022-01-31T21:01:54.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:54 smithi171 conmon[46715]: debug 2022-01-31T21:01:54.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:54.439682+0000) 2022-01-31T21:01:54.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:54 smithi171 conmon[51620]: debug 2022-01-31T21:01:54.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:54.516794+0000) 2022-01-31T21:01:54.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:54 smithi167 conmon[49112]: debug 2022-01-31T21:01:54.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:54.483458+0000) 2022-01-31T21:01:55.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:54 smithi171 conmon[41853]: debug 2022-01-31T21:01:54.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:54.644054+0000) 2022-01-31T21:01:55.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:55 smithi167 conmon[54076]: debug 2022-01-31T21:01:55.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.410899+0000) 2022-01-31T21:01:55.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:55 smithi167 conmon[60316]: debug 2022-01-31T21:01:55.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.162843+0000) 2022-01-31T21:01:55.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[35325]: debug 2022-01-31T21:01:55.424+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97553 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:01:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[41853]: debug 2022-01-31T21:01:55.414+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.414781+0000) 2022-01-31T21:01:55.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[46715]: debug 2022-01-31T21:01:55.414+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.414916+0000) 2022-01-31T21:01:55.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[46715]: debug 2022-01-31T21:01:55.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.439810+0000) 2022-01-31T21:01:55.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[51620]: debug 2022-01-31T21:01:55.413+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.413909+0000) 2022-01-31T21:01:55.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[51620]: debug 2022-01-31T21:01:55.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.516965+0000) 2022-01-31T21:01:55.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:55 smithi167 conmon[49112]: debug 2022-01-31T21:01:55.414+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.415164+0000) 2022-01-31T21:01:55.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:55 smithi167 conmon[49112]: debug 2022-01-31T21:01:55.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.483595+0000) 2022-01-31T21:01:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:55 smithi167 conmon[54076]: debug 2022-01-31T21:01:55.414+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.414602+0000) 2022-01-31T21:01:55.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:55 smithi167 conmon[60316]: debug 2022-01-31T21:01:55.414+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.415026+0000) 2022-01-31T21:01:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:55 smithi171 conmon[41853]: debug 2022-01-31T21:01:55.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:55.644243+0000) 2022-01-31T21:01:56.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:56 smithi167 conmon[54076]: debug 2022-01-31T21:01:56.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:56.411057+0000) 2022-01-31T21:01:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:56 smithi167 conmon[60316]: debug 2022-01-31T21:01:56.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:56.163023+0000) 2022-01-31T21:01:56.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:56 smithi171 conmon[51620]: debug 2022-01-31T21:01:56.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:56.517148+0000) 2022-01-31T21:01:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:56 smithi171 conmon[46715]: debug 2022-01-31T21:01:56.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:56.439977+0000) 2022-01-31T21:01:56.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:56 smithi167 conmon[49112]: debug 2022-01-31T21:01:56.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:56.483741+0000) 2022-01-31T21:01:57.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:56 smithi171 conmon[41853]: debug 2022-01-31T21:01:56.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:56.644413+0000) 2022-01-31T21:01:57.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:57 smithi167 conmon[60316]: debug 2022-01-31T21:01:57.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:57.163192+0000) 2022-01-31T21:01:57.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:57 smithi167 conmon[54076]: debug 2022-01-31T21:01:57.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:57.411191+0000) 2022-01-31T21:01:57.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:57 smithi171 conmon[46715]: debug 2022-01-31T21:01:57.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:57.440185+0000) 2022-01-31T21:01:57.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:57 smithi171 conmon[51620]: debug 2022-01-31T21:01:57.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:57.517318+0000) 2022-01-31T21:01:57.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:57 smithi167 conmon[49112]: debug 2022-01-31T21:01:57.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:57.483888+0000) 2022-01-31T21:01:58.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:57 smithi171 conmon[41853]: debug 2022-01-31T21:01:57.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:57.644601+0000) 2022-01-31T21:01:58.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:58 smithi167 conmon[54076]: debug 2022-01-31T21:01:58.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:58.411392+0000) 2022-01-31T21:01:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:58 smithi167 conmon[60316]: debug 2022-01-31T21:01:58.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:58.163378+0000) 2022-01-31T21:01:58.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:58 smithi171 conmon[46715]: debug 2022-01-31T21:01:58.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:58.440370+0000) 2022-01-31T21:01:58.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:58 smithi171 conmon[51620]: debug 2022-01-31T21:01:58.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:58.517543+0000) 2022-01-31T21:01:58.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:58 smithi167 conmon[49112]: debug 2022-01-31T21:01:58.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:58.484050+0000) 2022-01-31T21:01:59.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:58 smithi171 conmon[41853]: debug 2022-01-31T21:01:58.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:58.644777+0000) 2022-01-31T21:01:59.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:01:59 smithi167 conmon[54076]: debug 2022-01-31T21:01:59.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:59.411601+0000) 2022-01-31T21:01:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:01:59 smithi167 conmon[60316]: debug 2022-01-31T21:01:59.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:59.163581+0000) 2022-01-31T21:01:59.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:01:59 smithi171 conmon[46715]: debug 2022-01-31T21:01:59.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:59.440580+0000) 2022-01-31T21:01:59.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:01:59 smithi171 conmon[51620]: debug 2022-01-31T21:01:59.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:59.517701+0000) 2022-01-31T21:01:59.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:01:59 smithi167 conmon[49112]: debug 2022-01-31T21:01:59.482+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:59.484226+0000) 2022-01-31T21:02:00.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:01:59 smithi171 conmon[41853]: debug 2022-01-31T21:01:59.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:01:59.644906+0000) 2022-01-31T21:02:00.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:00 smithi167 conmon[54076]: debug 2022-01-31T21:02:00.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.411735+0000) 2022-01-31T21:02:00.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:00 smithi167 conmon[60316]: debug 2022-01-31T21:02:00.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.163709+0000) 2022-01-31T21:02:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[41853]: debug 2022-01-31T21:02:00.426+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.428310+0000) 2022-01-31T21:02:00.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[35325]: debug 2022-01-31T21:02:00.436+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97663 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:00.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[46715]: debug 2022-01-31T21:02:00.427+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.429499+0000) 2022-01-31T21:02:00.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[46715]: debug 2022-01-31T21:02:00.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.440810+0000) 2022-01-31T21:02:00.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[51620]: debug 2022-01-31T21:02:00.425+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.427650+0000) 2022-01-31T21:02:00.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[51620]: debug 2022-01-31T21:02:00.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.517924+0000) 2022-01-31T21:02:00.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:00 smithi167 conmon[49112]: debug 2022-01-31T21:02:00.427+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.428798+0000) 2022-01-31T21:02:00.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:00 smithi167 conmon[49112]: debug 2022-01-31T21:02:00.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.484419+0000) 2022-01-31T21:02:00.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:00 smithi167 conmon[54076]: debug 2022-01-31T21:02:00.426+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.428018+0000) 2022-01-31T21:02:00.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:00 smithi167 conmon[60316]: debug 2022-01-31T21:02:00.426+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.428161+0000) 2022-01-31T21:02:01.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:00 smithi171 conmon[41853]: debug 2022-01-31T21:02:00.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:00.645074+0000) 2022-01-31T21:02:01.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:01 smithi167 conmon[54076]: debug 2022-01-31T21:02:01.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:01.411912+0000) 2022-01-31T21:02:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:01 smithi167 conmon[60316]: debug 2022-01-31T21:02:01.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:01.163913+0000) 2022-01-31T21:02:01.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:01 smithi171 conmon[46715]: debug 2022-01-31T21:02:01.440+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:01.440953+0000) 2022-01-31T21:02:01.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:01 smithi171 conmon[51620]: debug 2022-01-31T21:02:01.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:01.518037+0000) 2022-01-31T21:02:01.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:01 smithi167 conmon[49112]: debug 2022-01-31T21:02:01.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:01.484612+0000) 2022-01-31T21:02:02.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:01 smithi171 conmon[41853]: debug 2022-01-31T21:02:01.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:01.645230+0000) 2022-01-31T21:02:02.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:02 smithi167 conmon[54076]: debug 2022-01-31T21:02:02.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:02.412092+0000) 2022-01-31T21:02:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:02 smithi167 conmon[60316]: debug 2022-01-31T21:02:02.163+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:02.164058+0000) 2022-01-31T21:02:02.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:02 smithi171 conmon[46715]: debug 2022-01-31T21:02:02.440+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:02.441143+0000) 2022-01-31T21:02:02.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:02 smithi171 conmon[51620]: debug 2022-01-31T21:02:02.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:02.518222+0000) 2022-01-31T21:02:02.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:02 smithi167 conmon[49112]: debug 2022-01-31T21:02:02.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:02.484803+0000) 2022-01-31T21:02:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:02 smithi171 conmon[41853]: debug 2022-01-31T21:02:02.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:02.645433+0000) 2022-01-31T21:02:03.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:03 smithi167 conmon[54076]: debug 2022-01-31T21:02:03.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:03.412295+0000) 2022-01-31T21:02:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:03 smithi167 conmon[60316]: debug 2022-01-31T21:02:03.162+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:03.164197+0000) 2022-01-31T21:02:03.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:03 smithi171 conmon[46715]: debug 2022-01-31T21:02:03.440+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:03.441301+0000) 2022-01-31T21:02:03.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:03 smithi171 conmon[51620]: debug 2022-01-31T21:02:03.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:03.518409+0000) 2022-01-31T21:02:03.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:03 smithi167 conmon[49112]: debug 2022-01-31T21:02:03.483+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:03.484960+0000) 2022-01-31T21:02:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:03 smithi171 conmon[41853]: debug 2022-01-31T21:02:03.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:03.645633+0000) 2022-01-31T21:02:04.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:04 smithi167 conmon[54076]: debug 2022-01-31T21:02:04.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:04.412539+0000) 2022-01-31T21:02:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:04 smithi167 conmon[60316]: debug 2022-01-31T21:02:04.163+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:04.164382+0000) 2022-01-31T21:02:04.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:04 smithi171 conmon[46715]: debug 2022-01-31T21:02:04.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:04.441401+0000) 2022-01-31T21:02:04.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:04 smithi171 conmon[51620]: debug 2022-01-31T21:02:04.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:04.518610+0000) 2022-01-31T21:02:04.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:04 smithi167 conmon[49112]: debug 2022-01-31T21:02:04.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:04.485166+0000) 2022-01-31T21:02:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:04 smithi171 conmon[41853]: debug 2022-01-31T21:02:04.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:04.645804+0000) 2022-01-31T21:02:05.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:05 smithi167 conmon[54076]: debug 2022-01-31T21:02:05.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.412743+0000) 2022-01-31T21:02:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:05 smithi167 conmon[60316]: debug 2022-01-31T21:02:05.163+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.164573+0000) 2022-01-31T21:02:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:05 smithi171 conmon[41853]: debug 2022-01-31T21:02:05.438+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.440571+0000) 2022-01-31T21:02:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:05 smithi171 conmon[35325]: debug 2022-01-31T21:02:05.449+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97773 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:05.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:05 smithi171 conmon[46715]: debug 2022-01-31T21:02:05.440+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.442332+0000) 2022-01-31T21:02:05.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:05 smithi171 conmon[51620]: debug 2022-01-31T21:02:05.438+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.440575+0000) 2022-01-31T21:02:05.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:05 smithi171 conmon[51620]: debug 2022-01-31T21:02:05.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.518787+0000) 2022-01-31T21:02:05.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:05 smithi167 conmon[54076]: debug 2022-01-31T21:02:05.438+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.440022+0000) 2022-01-31T21:02:05.730 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:05 smithi167 conmon[49112]: debug 2022-01-31T21:02:05.440+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.441532+0000) 2022-01-31T21:02:05.730 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:05 smithi167 conmon[49112]: debug 2022-01-31T21:02:05.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.485358+0000) 2022-01-31T21:02:05.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:05 smithi167 conmon[60316]: debug 2022-01-31T21:02:05.439+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.440751+0000) 2022-01-31T21:02:06.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:05 smithi171 conmon[41853]: debug 2022-01-31T21:02:05.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:05.645958+0000) 2022-01-31T21:02:06.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:06 smithi167 conmon[54076]: debug 2022-01-31T21:02:06.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:06.412893+0000) 2022-01-31T21:02:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:06 smithi167 conmon[60316]: debug 2022-01-31T21:02:06.163+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:06.164761+0000) 2022-01-31T21:02:06.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:06 smithi171 conmon[51620]: debug 2022-01-31T21:02:06.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:06.518934+0000) 2022-01-31T21:02:06.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:06 smithi167 conmon[49112]: debug 2022-01-31T21:02:06.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:06.485529+0000) 2022-01-31T21:02:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:06 smithi171 conmon[41853]: debug 2022-01-31T21:02:06.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:06.646069+0000) 2022-01-31T21:02:07.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:07 smithi167 conmon[54076]: debug 2022-01-31T21:02:07.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:07.413038+0000) 2022-01-31T21:02:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:07 smithi167 conmon[60316]: debug 2022-01-31T21:02:07.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:07.164912+0000) 2022-01-31T21:02:07.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:07 smithi171 conmon[51620]: debug 2022-01-31T21:02:07.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:07.519120+0000) 2022-01-31T21:02:07.730 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:07 smithi167 conmon[49112]: debug 2022-01-31T21:02:07.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:07.485704+0000) 2022-01-31T21:02:08.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:07 smithi171 conmon[41853]: debug 2022-01-31T21:02:07.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:07.646229+0000) 2022-01-31T21:02:08.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:08 smithi167 conmon[54076]: debug 2022-01-31T21:02:08.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:08.413239+0000) 2022-01-31T21:02:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:08 smithi167 conmon[60316]: debug 2022-01-31T21:02:08.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:08.165074+0000) 2022-01-31T21:02:08.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:08 smithi171 conmon[51620]: debug 2022-01-31T21:02:08.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:08.519345+0000) 2022-01-31T21:02:08.730 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:08 smithi167 conmon[49112]: debug 2022-01-31T21:02:08.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:08.485871+0000) 2022-01-31T21:02:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:08 smithi171 conmon[41853]: debug 2022-01-31T21:02:08.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:08.646429+0000) 2022-01-31T21:02:09.233 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:08 smithi171 conmon[35325]: debug 2022-01-31T21:02:08.611+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:02:09.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:09 smithi167 conmon[54076]: debug 2022-01-31T21:02:09.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:09.413433+0000) 2022-01-31T21:02:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:09 smithi167 conmon[60316]: debug 2022-01-31T21:02:09.163+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:09.165278+0000) 2022-01-31T21:02:09.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:09 smithi171 conmon[51620]: debug 2022-01-31T21:02:09.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:09.519549+0000) 2022-01-31T21:02:09.730 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:09 smithi167 conmon[49112]: debug 2022-01-31T21:02:09.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:09.486058+0000) 2022-01-31T21:02:10.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:09 smithi171 conmon[41853]: debug 2022-01-31T21:02:09.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:09.646611+0000) 2022-01-31T21:02:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:10 smithi167 conmon[54076]: debug 2022-01-31T21:02:10.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.413568+0000) 2022-01-31T21:02:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:10 smithi167 conmon[60316]: debug 2022-01-31T21:02:10.163+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.165432+0000) 2022-01-31T21:02:10.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:10 smithi171 conmon[35325]: debug 2022-01-31T21:02:10.463+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97884 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:10 smithi171 conmon[41853]: debug 2022-01-31T21:02:10.452+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.453087+0000) 2022-01-31T21:02:10.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:10 smithi171 conmon[46715]: debug 2022-01-31T21:02:10.453+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.454420+0000) 2022-01-31T21:02:10.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:10 smithi171 conmon[51620]: debug 2022-01-31T21:02:10.453+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.454348+0000) 2022-01-31T21:02:10.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:10 smithi171 conmon[51620]: debug 2022-01-31T21:02:10.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.519704+0000) 2022-01-31T21:02:10.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:10 smithi167 conmon[54076]: debug 2022-01-31T21:02:10.453+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.454520+0000) 2022-01-31T21:02:10.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:10 smithi167 conmon[60316]: debug 2022-01-31T21:02:10.451+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.453337+0000) 2022-01-31T21:02:10.733 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:10 smithi167 conmon[49112]: debug 2022-01-31T21:02:10.452+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.453545+0000) 2022-01-31T21:02:10.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:10 smithi167 conmon[49112]: debug 2022-01-31T21:02:10.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.486265+0000) 2022-01-31T21:02:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:10 smithi171 conmon[41853]: debug 2022-01-31T21:02:10.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:10.646808+0000) 2022-01-31T21:02:11.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:11 smithi167 conmon[54076]: debug 2022-01-31T21:02:11.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:11.413668+0000) 2022-01-31T21:02:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:11 smithi167 conmon[60316]: debug 2022-01-31T21:02:11.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:11.165613+0000) 2022-01-31T21:02:11.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:11 smithi171 conmon[51620]: debug 2022-01-31T21:02:11.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:11.519853+0000) 2022-01-31T21:02:11.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:11 smithi167 conmon[49112]: debug 2022-01-31T21:02:11.484+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:11.486442+0000) 2022-01-31T21:02:12.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:11 smithi171 conmon[41853]: debug 2022-01-31T21:02:11.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:11.646948+0000) 2022-01-31T21:02:12.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:12 smithi167 conmon[54076]: debug 2022-01-31T21:02:12.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:12.413814+0000) 2022-01-31T21:02:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:12 smithi167 conmon[60316]: debug 2022-01-31T21:02:12.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:12.165720+0000) 2022-01-31T21:02:12.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:12 smithi171 conmon[46715]: debug 2022-01-31T21:02:12.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:12.191802+0000) 2022-01-31T21:02:12.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:12 smithi167 conmon[49112]: debug 2022-01-31T21:02:12.485+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:12.486632+0000) 2022-01-31T21:02:12.771 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:12 smithi171 conmon[51620]: debug 2022-01-31T21:02:12.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:12.520036+0000) 2022-01-31T21:02:12.772 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:12 smithi171 conmon[41853]: debug 2022-01-31T21:02:12.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:12.647128+0000) 2022-01-31T21:02:13.421 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:13 smithi167 conmon[54076]: debug 2022-01-31T21:02:13.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:13.414030+0000) 2022-01-31T21:02:13.422 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:13 smithi167 conmon[60316]: debug 2022-01-31T21:02:13.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:13.165841+0000) 2022-01-31T21:02:13.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:13 smithi171 conmon[46715]: debug 2022-01-31T21:02:13.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:13.191980+0000) 2022-01-31T21:02:13.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:13 smithi167 conmon[49112]: debug 2022-01-31T21:02:13.486+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:13.486774+0000) 2022-01-31T21:02:13.771 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:13 smithi171 conmon[41853]: debug 2022-01-31T21:02:13.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:13.647314+0000) 2022-01-31T21:02:13.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:13 smithi171 conmon[51620]: debug 2022-01-31T21:02:13.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:13.520195+0000) 2022-01-31T21:02:14.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:14 smithi167 conmon[54076]: debug 2022-01-31T21:02:14.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:14.414225+0000) 2022-01-31T21:02:14.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:14 smithi167 conmon[60316]: debug 2022-01-31T21:02:14.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:14.166000+0000) 2022-01-31T21:02:14.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:14 smithi171 conmon[46715]: debug 2022-01-31T21:02:14.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:14.192175+0000) 2022-01-31T21:02:14.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:14 smithi167 conmon[49112]: debug 2022-01-31T21:02:14.486+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:14.486953+0000) 2022-01-31T21:02:14.771 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:14 smithi171 conmon[51620]: debug 2022-01-31T21:02:14.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:14.520384+0000) 2022-01-31T21:02:14.772 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:14 smithi171 conmon[41853]: debug 2022-01-31T21:02:14.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:14.647493+0000) 2022-01-31T21:02:15.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:15 smithi167 conmon[54076]: debug 2022-01-31T21:02:15.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.414409+0000) 2022-01-31T21:02:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:15 smithi167 conmon[60316]: debug 2022-01-31T21:02:15.165+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.166185+0000) 2022-01-31T21:02:15.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[46715]: debug 2022-01-31T21:02:15.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.192358+0000) 2022-01-31T21:02:15.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:15 smithi167 conmon[49112]: debug 2022-01-31T21:02:15.466+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.467597+0000) 2022-01-31T21:02:15.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:15 smithi167 conmon[49112]: debug 2022-01-31T21:02:15.485+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.487109+0000) 2022-01-31T21:02:15.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:15 smithi167 conmon[54076]: debug 2022-01-31T21:02:15.466+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.468371+0000) 2022-01-31T21:02:15.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:15 smithi167 conmon[60316]: debug 2022-01-31T21:02:15.465+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.466705+0000) 2022-01-31T21:02:15.772 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[35325]: debug 2022-01-31T21:02:15.475+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 97994 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:15.772 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[41853]: debug 2022-01-31T21:02:15.465+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.466462+0000) 2022-01-31T21:02:15.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[41853]: debug 2022-01-31T21:02:15.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.647677+0000) 2022-01-31T21:02:15.773 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[46715]: debug 2022-01-31T21:02:15.466+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.466912+0000) 2022-01-31T21:02:15.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[51620]: debug 2022-01-31T21:02:15.467+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.467944+0000) 2022-01-31T21:02:15.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:15 smithi171 conmon[51620]: debug 2022-01-31T21:02:15.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:15.520585+0000) 2022-01-31T21:02:16.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:16 smithi167 conmon[60316]: debug 2022-01-31T21:02:16.164+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:16.166365+0000) 2022-01-31T21:02:16.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:16 smithi167 conmon[54076]: debug 2022-01-31T21:02:16.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:16.414592+0000) 2022-01-31T21:02:16.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:16 smithi171 conmon[46715]: debug 2022-01-31T21:02:16.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:16.192541+0000) 2022-01-31T21:02:16.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:16 smithi167 conmon[49112]: debug 2022-01-31T21:02:16.485+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:16.487267+0000) 2022-01-31T21:02:16.772 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:16 smithi171 conmon[41853]: debug 2022-01-31T21:02:16.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:16.647858+0000) 2022-01-31T21:02:16.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:16 smithi171 conmon[51620]: debug 2022-01-31T21:02:16.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:16.520736+0000) 2022-01-31T21:02:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:17 smithi167 conmon[60316]: debug 2022-01-31T21:02:17.165+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:17.166539+0000) 2022-01-31T21:02:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:17 smithi167 conmon[54076]: debug 2022-01-31T21:02:17.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:17.414745+0000) 2022-01-31T21:02:17.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:17 smithi171 conmon[46715]: debug 2022-01-31T21:02:17.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:17.192713+0000) 2022-01-31T21:02:17.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:17 smithi167 conmon[49112]: debug 2022-01-31T21:02:17.486+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:17.487445+0000) 2022-01-31T21:02:17.772 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:17 smithi171 conmon[41853]: debug 2022-01-31T21:02:17.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:17.648012+0000) 2022-01-31T21:02:17.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:17 smithi171 conmon[51620]: debug 2022-01-31T21:02:17.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:17.520880+0000) 2022-01-31T21:02:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:18 smithi167 conmon[54076]: debug 2022-01-31T21:02:18.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:18.414960+0000) 2022-01-31T21:02:18.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:18 smithi167 conmon[60316]: debug 2022-01-31T21:02:18.166+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:18.166752+0000) 2022-01-31T21:02:18.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:18 smithi171 conmon[46715]: debug 2022-01-31T21:02:18.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:18.192912+0000) 2022-01-31T21:02:18.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:18 smithi167 conmon[49112]: debug 2022-01-31T21:02:18.486+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:18.487602+0000) 2022-01-31T21:02:18.772 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:18 smithi171 conmon[41853]: debug 2022-01-31T21:02:18.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:18.648165+0000) 2022-01-31T21:02:18.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:18 smithi171 conmon[51620]: debug 2022-01-31T21:02:18.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:18.521054+0000) 2022-01-31T21:02:19.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:19 smithi167 conmon[54076]: debug 2022-01-31T21:02:19.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:19.415168+0000) 2022-01-31T21:02:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:19 smithi167 conmon[60316]: debug 2022-01-31T21:02:19.166+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:19.166908+0000) 2022-01-31T21:02:19.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:19 smithi171 conmon[46715]: debug 2022-01-31T21:02:19.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:19.193093+0000) 2022-01-31T21:02:19.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:19 smithi167 conmon[49112]: debug 2022-01-31T21:02:19.487+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:19.487764+0000) 2022-01-31T21:02:19.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:19 smithi171 conmon[51620]: debug 2022-01-31T21:02:19.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:19.521211+0000) 2022-01-31T21:02:19.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:19 smithi171 conmon[41853]: debug 2022-01-31T21:02:19.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:19.648345+0000) 2022-01-31T21:02:20.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:20 smithi167 conmon[60316]: debug 2022-01-31T21:02:20.166+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.167146+0000) 2022-01-31T21:02:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:20 smithi167 conmon[54076]: debug 2022-01-31T21:02:20.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.415455+0000) 2022-01-31T21:02:20.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[46715]: debug 2022-01-31T21:02:20.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.193373+0000) 2022-01-31T21:02:20.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:20 smithi167 conmon[49112]: debug 2022-01-31T21:02:20.478+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.479606+0000) 2022-01-31T21:02:20.733 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:20 smithi167 conmon[49112]: debug 2022-01-31T21:02:20.486+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.487859+0000) 2022-01-31T21:02:20.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:20 smithi167 conmon[54076]: debug 2022-01-31T21:02:20.478+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.480347+0000) 2022-01-31T21:02:20.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:20 smithi167 conmon[60316]: debug 2022-01-31T21:02:20.478+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.479855+0000) 2022-01-31T21:02:20.773 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[35325]: debug 2022-01-31T21:02:20.488+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98104 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:20.773 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[46715]: debug 2022-01-31T21:02:20.478+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.479005+0000) 2022-01-31T21:02:20.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[41853]: debug 2022-01-31T21:02:20.477+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.478604+0000) 2022-01-31T21:02:20.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[41853]: debug 2022-01-31T21:02:20.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.648505+0000) 2022-01-31T21:02:20.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[51620]: debug 2022-01-31T21:02:20.479+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.480116+0000) 2022-01-31T21:02:20.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:20 smithi171 conmon[51620]: debug 2022-01-31T21:02:20.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:20.521413+0000) 2022-01-31T21:02:21.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:21 smithi167 conmon[54076]: debug 2022-01-31T21:02:21.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:21.415587+0000) 2022-01-31T21:02:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:21 smithi167 conmon[60316]: debug 2022-01-31T21:02:21.166+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:21.167329+0000) 2022-01-31T21:02:21.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:21 smithi171 conmon[46715]: debug 2022-01-31T21:02:21.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:21.193543+0000) 2022-01-31T21:02:21.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:21 smithi167 conmon[49112]: debug 2022-01-31T21:02:21.487+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:21.487984+0000) 2022-01-31T21:02:21.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:21 smithi171 conmon[51620]: debug 2022-01-31T21:02:21.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:21.521575+0000) 2022-01-31T21:02:21.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:21 smithi171 conmon[41853]: debug 2022-01-31T21:02:21.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:21.648659+0000) 2022-01-31T21:02:22.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:22 smithi167 conmon[54076]: debug 2022-01-31T21:02:22.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:22.415718+0000) 2022-01-31T21:02:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:22 smithi167 conmon[60316]: debug 2022-01-31T21:02:22.166+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:22.167481+0000) 2022-01-31T21:02:22.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:22 smithi171 conmon[46715]: debug 2022-01-31T21:02:22.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:22.193699+0000) 2022-01-31T21:02:22.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:22 smithi167 conmon[49112]: debug 2022-01-31T21:02:22.487+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:22.488161+0000) 2022-01-31T21:02:22.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:22 smithi171 conmon[51620]: debug 2022-01-31T21:02:22.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:22.521796+0000) 2022-01-31T21:02:22.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:22 smithi171 conmon[41853]: debug 2022-01-31T21:02:22.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:22.648854+0000) 2022-01-31T21:02:23.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:23 smithi167 conmon[54076]: debug 2022-01-31T21:02:23.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:23.415911+0000) 2022-01-31T21:02:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:23 smithi167 conmon[60316]: debug 2022-01-31T21:02:23.167+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:23.167651+0000) 2022-01-31T21:02:23.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:23 smithi171 conmon[46715]: debug 2022-01-31T21:02:23.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:23.193907+0000) 2022-01-31T21:02:23.733 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:23 smithi167 conmon[49112]: debug 2022-01-31T21:02:23.487+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:23.488362+0000) 2022-01-31T21:02:23.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:23 smithi171 conmon[51620]: debug 2022-01-31T21:02:23.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:23.521978+0000) 2022-01-31T21:02:23.773 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:23 smithi171 conmon[35325]: debug 2022-01-31T21:02:23.612+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:02:23.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:23 smithi171 conmon[41853]: debug 2022-01-31T21:02:23.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:23.649039+0000) 2022-01-31T21:02:24.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:24 smithi167 conmon[54076]: debug 2022-01-31T21:02:24.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:24.416091+0000) 2022-01-31T21:02:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:24 smithi167 conmon[60316]: debug 2022-01-31T21:02:24.167+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:24.167863+0000) 2022-01-31T21:02:24.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:24 smithi171 conmon[46715]: debug 2022-01-31T21:02:24.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:24.194095+0000) 2022-01-31T21:02:24.733 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:24 smithi167 conmon[49112]: debug 2022-01-31T21:02:24.488+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:24.488560+0000) 2022-01-31T21:02:24.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:24 smithi171 conmon[41853]: debug 2022-01-31T21:02:24.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:24.649226+0000) 2022-01-31T21:02:24.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:24 smithi171 conmon[51620]: debug 2022-01-31T21:02:24.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:24.522078+0000) 2022-01-31T21:02:25.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:25 smithi167 conmon[54076]: debug 2022-01-31T21:02:25.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.416296+0000) 2022-01-31T21:02:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:25 smithi167 conmon[60316]: debug 2022-01-31T21:02:25.167+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.168058+0000) 2022-01-31T21:02:25.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[46715]: debug 2022-01-31T21:02:25.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.194247+0000) 2022-01-31T21:02:25.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[41853]: debug 2022-01-31T21:02:25.490+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.491519+0000) 2022-01-31T21:02:25.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:25 smithi167 conmon[54076]: debug 2022-01-31T21:02:25.492+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.492718+0000) 2022-01-31T21:02:25.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:25 smithi167 conmon[49112]: debug 2022-01-31T21:02:25.488+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.488725+0000) 2022-01-31T21:02:25.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:25 smithi167 conmon[49112]: debug 2022-01-31T21:02:25.491+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.492063+0000) 2022-01-31T21:02:25.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:25 smithi167 conmon[60316]: debug 2022-01-31T21:02:25.491+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.491778+0000) 2022-01-31T21:02:25.773 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[46715]: debug 2022-01-31T21:02:25.491+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.491763+0000) 2022-01-31T21:02:25.774 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[35325]: debug 2022-01-31T21:02:25.501+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98214 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:25.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[41853]: debug 2022-01-31T21:02:25.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.649430+0000) 2022-01-31T21:02:25.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[51620]: debug 2022-01-31T21:02:25.492+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.493063+0000) 2022-01-31T21:02:25.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:25 smithi171 conmon[51620]: debug 2022-01-31T21:02:25.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:25.522240+0000) 2022-01-31T21:02:26.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:26 smithi167 conmon[54076]: debug 2022-01-31T21:02:26.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:26.416462+0000) 2022-01-31T21:02:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:26 smithi167 conmon[60316]: debug 2022-01-31T21:02:26.167+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:26.168238+0000) 2022-01-31T21:02:26.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:26 smithi171 conmon[46715]: debug 2022-01-31T21:02:26.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:26.194437+0000) 2022-01-31T21:02:26.733 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:26 smithi167 conmon[49112]: debug 2022-01-31T21:02:26.488+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:26.488852+0000) 2022-01-31T21:02:26.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:26 smithi171 conmon[41853]: debug 2022-01-31T21:02:26.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:26.649599+0000) 2022-01-31T21:02:26.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:26 smithi171 conmon[51620]: debug 2022-01-31T21:02:26.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:26.522392+0000) 2022-01-31T21:02:27.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:27 smithi167 conmon[54076]: debug 2022-01-31T21:02:27.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:27.416599+0000) 2022-01-31T21:02:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:27 smithi167 conmon[60316]: debug 2022-01-31T21:02:27.167+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:27.168392+0000) 2022-01-31T21:02:27.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:27 smithi171 conmon[46715]: debug 2022-01-31T21:02:27.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:27.194586+0000) 2022-01-31T21:02:27.733 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:27 smithi167 conmon[49112]: debug 2022-01-31T21:02:27.488+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:27.488991+0000) 2022-01-31T21:02:27.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:27 smithi171 conmon[41853]: debug 2022-01-31T21:02:27.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:27.649782+0000) 2022-01-31T21:02:27.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:27 smithi171 conmon[51620]: debug 2022-01-31T21:02:27.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:27.522574+0000) 2022-01-31T21:02:28.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:28 smithi167 conmon[54076]: debug 2022-01-31T21:02:28.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:28.416786+0000) 2022-01-31T21:02:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:28 smithi167 conmon[60316]: debug 2022-01-31T21:02:28.168+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:28.168579+0000) 2022-01-31T21:02:28.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:28 smithi171 conmon[46715]: debug 2022-01-31T21:02:28.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:28.194785+0000) 2022-01-31T21:02:28.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:28 smithi167 conmon[49112]: debug 2022-01-31T21:02:28.488+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:28.489153+0000) 2022-01-31T21:02:28.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:28 smithi171 conmon[41853]: debug 2022-01-31T21:02:28.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:28.649935+0000) 2022-01-31T21:02:28.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:28 smithi171 conmon[51620]: debug 2022-01-31T21:02:28.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:28.522779+0000) 2022-01-31T21:02:29.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:29 smithi167 conmon[54076]: debug 2022-01-31T21:02:29.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:29.416971+0000) 2022-01-31T21:02:29.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:29 smithi167 conmon[60316]: debug 2022-01-31T21:02:29.168+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:29.168769+0000) 2022-01-31T21:02:29.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:29 smithi171 conmon[46715]: debug 2022-01-31T21:02:29.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:29.194927+0000) 2022-01-31T21:02:29.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:29 smithi167 conmon[49112]: debug 2022-01-31T21:02:29.488+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:29.489310+0000) 2022-01-31T21:02:29.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:29 smithi171 conmon[41853]: debug 2022-01-31T21:02:29.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:29.650118+0000) 2022-01-31T21:02:29.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:29 smithi171 conmon[51620]: debug 2022-01-31T21:02:29.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:29.522966+0000) 2022-01-31T21:02:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:30 smithi167 conmon[54076]: debug 2022-01-31T21:02:30.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.417133+0000) 2022-01-31T21:02:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:30 smithi167 conmon[60316]: debug 2022-01-31T21:02:30.168+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.168970+0000) 2022-01-31T21:02:30.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[46715]: debug 2022-01-31T21:02:30.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.195116+0000) 2022-01-31T21:02:30.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:30 smithi167 conmon[49112]: debug 2022-01-31T21:02:30.489+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.489491+0000) 2022-01-31T21:02:30.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:30 smithi167 conmon[49112]: debug 2022-01-31T21:02:30.504+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.504613+0000) 2022-01-31T21:02:30.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:30 smithi167 conmon[54076]: debug 2022-01-31T21:02:30.504+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.504991+0000) 2022-01-31T21:02:30.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:30 smithi167 conmon[60316]: debug 2022-01-31T21:02:30.504+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.504905+0000) 2022-01-31T21:02:30.774 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[35325]: debug 2022-01-31T21:02:30.513+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98324 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:30.775 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[41853]: debug 2022-01-31T21:02:30.503+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.504575+0000) 2022-01-31T21:02:30.775 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[41853]: debug 2022-01-31T21:02:30.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.650314+0000) 2022-01-31T21:02:30.776 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[46715]: debug 2022-01-31T21:02:30.503+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.504071+0000) 2022-01-31T21:02:30.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[51620]: debug 2022-01-31T21:02:30.503+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.504499+0000) 2022-01-31T21:02:30.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:30 smithi171 conmon[51620]: debug 2022-01-31T21:02:30.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:30.523095+0000) 2022-01-31T21:02:31.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:31 smithi167 conmon[54076]: debug 2022-01-31T21:02:31.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:31.417329+0000) 2022-01-31T21:02:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:31 smithi167 conmon[60316]: debug 2022-01-31T21:02:31.168+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:31.169177+0000) 2022-01-31T21:02:31.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:31 smithi171 conmon[46715]: debug 2022-01-31T21:02:31.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:31.195291+0000) 2022-01-31T21:02:31.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:31 smithi167 conmon[49112]: debug 2022-01-31T21:02:31.489+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:31.489691+0000) 2022-01-31T21:02:31.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:31 smithi171 conmon[41853]: debug 2022-01-31T21:02:31.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:31.650514+0000) 2022-01-31T21:02:31.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:31 smithi171 conmon[51620]: debug 2022-01-31T21:02:31.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:31.523222+0000) 2022-01-31T21:02:32.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:32 smithi167 conmon[54076]: debug 2022-01-31T21:02:32.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:32.417512+0000) 2022-01-31T21:02:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:32 smithi167 conmon[60316]: debug 2022-01-31T21:02:32.168+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:32.169292+0000) 2022-01-31T21:02:32.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:32 smithi171 conmon[46715]: debug 2022-01-31T21:02:32.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:32.195440+0000) 2022-01-31T21:02:32.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:32 smithi167 conmon[49112]: debug 2022-01-31T21:02:32.489+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:32.489832+0000) 2022-01-31T21:02:32.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:32 smithi171 conmon[41853]: debug 2022-01-31T21:02:32.649+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:32.650704+0000) 2022-01-31T21:02:32.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:32 smithi171 conmon[51620]: debug 2022-01-31T21:02:32.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:32.523365+0000) 2022-01-31T21:02:33.463 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:33 smithi167 conmon[54076]: debug 2022-01-31T21:02:33.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:33.417637+0000) 2022-01-31T21:02:33.464 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:33 smithi167 conmon[60316]: debug 2022-01-31T21:02:33.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:33.169476+0000) 2022-01-31T21:02:33.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:33 smithi171 conmon[46715]: debug 2022-01-31T21:02:33.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:33.195642+0000) 2022-01-31T21:02:33.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:33 smithi167 conmon[49112]: debug 2022-01-31T21:02:33.489+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:33.490015+0000) 2022-01-31T21:02:33.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:33 smithi171 conmon[51620]: debug 2022-01-31T21:02:33.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:33.523519+0000) 2022-01-31T21:02:33.775 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:33 smithi171 conmon[41853]: debug 2022-01-31T21:02:33.650+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:33.650902+0000) 2022-01-31T21:02:34.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:34 smithi167 conmon[54076]: debug 2022-01-31T21:02:34.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:34.417786+0000) 2022-01-31T21:02:34.464 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:34 smithi167 conmon[60316]: debug 2022-01-31T21:02:34.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:34.169683+0000) 2022-01-31T21:02:34.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:34 smithi171 conmon[46715]: debug 2022-01-31T21:02:34.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:34.195865+0000) 2022-01-31T21:02:34.735 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:34 smithi167 conmon[49112]: debug 2022-01-31T21:02:34.489+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:34.490242+0000) 2022-01-31T21:02:34.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:34 smithi171 conmon[51620]: debug 2022-01-31T21:02:34.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:34.523722+0000) 2022-01-31T21:02:34.775 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:34 smithi171 conmon[41853]: debug 2022-01-31T21:02:34.650+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:34.651048+0000) 2022-01-31T21:02:35.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:35 smithi167 conmon[54076]: debug 2022-01-31T21:02:35.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.417976+0000) 2022-01-31T21:02:35.464 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:35 smithi167 conmon[60316]: debug 2022-01-31T21:02:35.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.169802+0000) 2022-01-31T21:02:35.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[46715]: debug 2022-01-31T21:02:35.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.196032+0000) 2022-01-31T21:02:35.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:35 smithi167 conmon[54076]: debug 2022-01-31T21:02:35.522+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.523446+0000) 2022-01-31T21:02:35.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:35 smithi167 conmon[60316]: debug 2022-01-31T21:02:35.515+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.515984+0000) 2022-01-31T21:02:35.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:35 smithi167 conmon[49112]: debug 2022-01-31T21:02:35.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.490404+0000) 2022-01-31T21:02:35.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:35 smithi167 conmon[49112]: debug 2022-01-31T21:02:35.522+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.523204+0000) 2022-01-31T21:02:35.775 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[35325]: debug 2022-01-31T21:02:35.527+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98434 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:35.776 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[46715]: debug 2022-01-31T21:02:35.515+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.516721+0000) 2022-01-31T21:02:35.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[41853]: debug 2022-01-31T21:02:35.516+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.517049+0000) 2022-01-31T21:02:35.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[41853]: debug 2022-01-31T21:02:35.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.651241+0000) 2022-01-31T21:02:35.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[51620]: debug 2022-01-31T21:02:35.515+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.516414+0000) 2022-01-31T21:02:35.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[51620]: debug 2022-01-31T21:02:35.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:35 smithi171 conmon[51620]: 2022-01-31T21:02:35.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:35.523907+0000) 2022-01-31T21:02:36.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:36 smithi167 conmon[54076]: debug 2022-01-31T21:02:36.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:36.418162+0000) 2022-01-31T21:02:36.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:36 smithi167 conmon[60316]: debug 2022-01-31T21:02:36.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:36.169919+0000) 2022-01-31T21:02:36.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:36 smithi171 conmon[46715]: debug 2022-01-31T21:02:36.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:36.196201+0000) 2022-01-31T21:02:36.735 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:36 smithi167 conmon[49112]: debug 2022-01-31T21:02:36.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:36.490556+0000) 2022-01-31T21:02:36.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:36 smithi171 conmon[51620]: debug 2022-01-31T21:02:36.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:36.524073+0000) 2022-01-31T21:02:36.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:36 smithi171 conmon[41853]: debug 2022-01-31T21:02:36.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:36.651401+0000) 2022-01-31T21:02:37.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:37 smithi167 conmon[54076]: debug 2022-01-31T21:02:37.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:37.418314+0000) 2022-01-31T21:02:37.464 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:37 smithi167 conmon[60316]: debug 2022-01-31T21:02:37.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:37.170023+0000) 2022-01-31T21:02:37.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:37 smithi171 conmon[46715]: debug 2022-01-31T21:02:37.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:37.196343+0000) 2022-01-31T21:02:37.735 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:37 smithi167 conmon[49112]: debug 2022-01-31T21:02:37.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:37.490671+0000) 2022-01-31T21:02:37.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:37 smithi171 conmon[51620]: debug 2022-01-31T21:02:37.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:37.524259+0000) 2022-01-31T21:02:37.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:37 smithi171 conmon[41853]: debug 2022-01-31T21:02:37.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:37.651574+0000) 2022-01-31T21:02:38.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:38 smithi167 conmon[54076]: debug 2022-01-31T21:02:38.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:38.418488+0000) 2022-01-31T21:02:38.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:38 smithi167 conmon[60316]: debug 2022-01-31T21:02:38.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:38.170132+0000) 2022-01-31T21:02:38.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:38 smithi171 conmon[46715]: debug 2022-01-31T21:02:38.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:38.196530+0000) 2022-01-31T21:02:38.735 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:38 smithi167 conmon[49112]: debug 2022-01-31T21:02:38.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:38.490823+0000) 2022-01-31T21:02:38.776 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:38 smithi171 conmon[35325]: debug 2022-01-31T21:02:38.613+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:02:38.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:38 smithi171 conmon[41853]: debug 2022-01-31T21:02:38.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:38.651791+0000) 2022-01-31T21:02:38.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:38 smithi171 conmon[51620]: debug 2022-01-31T21:02:38.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:38.524466+0000) 2022-01-31T21:02:39.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:39 smithi167 conmon[54076]: debug 2022-01-31T21:02:39.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:39.418636+0000) 2022-01-31T21:02:39.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:39 smithi167 conmon[60316]: debug 2022-01-31T21:02:39.169+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:39.170299+0000) 2022-01-31T21:02:39.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:39 smithi171 conmon[46715]: debug 2022-01-31T21:02:39.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:39.196718+0000) 2022-01-31T21:02:39.735 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:39 smithi167 conmon[49112]: debug 2022-01-31T21:02:39.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:39.490955+0000) 2022-01-31T21:02:39.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:39 smithi171 conmon[41853]: debug 2022-01-31T21:02:39.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:39.652001+0000) 2022-01-31T21:02:39.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:39 smithi171 conmon[51620]: debug 2022-01-31T21:02:39.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:39.524667+0000) 2022-01-31T21:02:40.465 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:40 smithi167 conmon[54076]: debug 2022-01-31T21:02:40.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.418831+0000) 2022-01-31T21:02:40.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:40 smithi167 conmon[60316]: debug 2022-01-31T21:02:40.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.170528+0000) 2022-01-31T21:02:40.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[46715]: debug 2022-01-31T21:02:40.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.196844+0000) 2022-01-31T21:02:40.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:40 smithi167 conmon[54076]: debug 2022-01-31T21:02:40.530+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.531451+0000) 2022-01-31T21:02:40.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:40 smithi167 conmon[60316]: debug 2022-01-31T21:02:40.536+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.537311+0000) 2022-01-31T21:02:40.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:40 smithi167 conmon[49112]: debug 2022-01-31T21:02:40.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.491112+0000) 2022-01-31T21:02:40.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:40 smithi167 conmon[49112]: debug 2022-01-31T21:02:40.531+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.531888+0000) 2022-01-31T21:02:40.776 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[35325]: debug 2022-01-31T21:02:40.540+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98544 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:40.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[41853]: debug 2022-01-31T21:02:40.531+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.531856+0000) 2022-01-31T21:02:40.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[41853]: debug 2022-01-31T21:02:40.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.652132+0000) 2022-01-31T21:02:40.777 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[46715]: debug 2022-01-31T21:02:40.530+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.531077+0000) 2022-01-31T21:02:40.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[51620]: debug 2022-01-31T21:02:40.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.524842+0000) 2022-01-31T21:02:40.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:40 smithi171 conmon[51620]: debug 2022-01-31T21:02:40.529+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:40.530447+0000) 2022-01-31T21:02:41.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:41 smithi167 conmon[60316]: debug 2022-01-31T21:02:41.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:41.170683+0000) 2022-01-31T21:02:41.465 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:41 smithi167 conmon[54076]: debug 2022-01-31T21:02:41.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:41.419055+0000) 2022-01-31T21:02:41.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:41 smithi171 conmon[46715]: debug 2022-01-31T21:02:41.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:41.197001+0000) 2022-01-31T21:02:41.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:41 smithi167 conmon[49112]: debug 2022-01-31T21:02:41.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:41.491296+0000) 2022-01-31T21:02:41.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:41 smithi171 conmon[41853]: debug 2022-01-31T21:02:41.652+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:41.652267+0000) 2022-01-31T21:02:41.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:41 smithi171 conmon[51620]: debug 2022-01-31T21:02:41.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:41.525024+0000) 2022-01-31T21:02:42.465 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:42 smithi167 conmon[54076]: debug 2022-01-31T21:02:42.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:42.419227+0000) 2022-01-31T21:02:42.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:42 smithi167 conmon[60316]: debug 2022-01-31T21:02:42.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:42.170830+0000) 2022-01-31T21:02:42.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:42 smithi171 conmon[46715]: debug 2022-01-31T21:02:42.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:42.197151+0000) 2022-01-31T21:02:42.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:42 smithi167 conmon[49112]: debug 2022-01-31T21:02:42.490+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:42.491455+0000) 2022-01-31T21:02:42.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:42 smithi171 conmon[41853]: debug 2022-01-31T21:02:42.652+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:42.652446+0000) 2022-01-31T21:02:42.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:42 smithi171 conmon[51620]: debug 2022-01-31T21:02:42.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:42.525187+0000) 2022-01-31T21:02:43.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:43 smithi167 conmon[60316]: debug 2022-01-31T21:02:43.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:43.170983+0000) 2022-01-31T21:02:43.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:43 smithi167 conmon[54076]: debug 2022-01-31T21:02:43.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:43.419386+0000) 2022-01-31T21:02:43.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:43 smithi171 conmon[46715]: debug 2022-01-31T21:02:43.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:43.197345+0000) 2022-01-31T21:02:43.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:43 smithi167 conmon[49112]: debug 2022-01-31T21:02:43.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:43.491603+0000) 2022-01-31T21:02:43.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:43 smithi171 conmon[51620]: debug 2022-01-31T21:02:43.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:43.525400+0000) 2022-01-31T21:02:43.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:43 smithi171 conmon[41853]: debug 2022-01-31T21:02:43.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:43.652618+0000) 2022-01-31T21:02:44.465 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:44 smithi167 conmon[54076]: debug 2022-01-31T21:02:44.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:44.419568+0000) 2022-01-31T21:02:44.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:44 smithi167 conmon[60316]: debug 2022-01-31T21:02:44.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:44.171113+0000) 2022-01-31T21:02:44.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:44 smithi171 conmon[46715]: debug 2022-01-31T21:02:44.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:44.197538+0000) 2022-01-31T21:02:44.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:44 smithi167 conmon[49112]: debug 2022-01-31T21:02:44.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:44.491775+0000) 2022-01-31T21:02:44.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:44 smithi171 conmon[51620]: debug 2022-01-31T21:02:44.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:44.525597+0000) 2022-01-31T21:02:44.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:44 smithi171 conmon[41853]: debug 2022-01-31T21:02:44.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:44.652794+0000) 2022-01-31T21:02:45.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:45 smithi167 conmon[60316]: debug 2022-01-31T21:02:45.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.171338+0000) 2022-01-31T21:02:45.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:45 smithi167 conmon[54076]: debug 2022-01-31T21:02:45.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.419718+0000) 2022-01-31T21:02:45.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[46715]: debug 2022-01-31T21:02:45.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.197722+0000) 2022-01-31T21:02:45.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:45 smithi167 conmon[54076]: debug 2022-01-31T21:02:45.542+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.543316+0000) 2022-01-31T21:02:45.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:45 smithi167 conmon[60316]: debug 2022-01-31T21:02:45.544+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.544532+0000) 2022-01-31T21:02:45.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:45 smithi167 conmon[49112]: debug 2022-01-31T21:02:45.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.491930+0000) 2022-01-31T21:02:45.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:45 smithi167 conmon[49112]: debug 2022-01-31T21:02:45.542+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.543433+0000) 2022-01-31T21:02:45.777 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[35325]: debug 2022-01-31T21:02:45.552+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98654 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:45.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[41853]: debug 2022-01-31T21:02:45.542+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.544687+0000) 2022-01-31T21:02:45.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[41853]: debug 2022-01-31T21:02:45.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.652973+0000) 2022-01-31T21:02:45.778 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[46715]: debug 2022-01-31T21:02:45.540+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.542632+0000) 2022-01-31T21:02:45.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[51620]: debug 2022-01-31T21:02:45.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.525791+0000) 2022-01-31T21:02:45.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:45 smithi171 conmon[51620]: debug 2022-01-31T21:02:45.542+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:45.544320+0000) 2022-01-31T21:02:46.465 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:46 smithi167 conmon[54076]: debug 2022-01-31T21:02:46.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:46.419869+0000) 2022-01-31T21:02:46.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:46 smithi167 conmon[60316]: debug 2022-01-31T21:02:46.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:46.171488+0000) 2022-01-31T21:02:46.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:46 smithi171 conmon[46715]: debug 2022-01-31T21:02:46.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:46.197924+0000) 2022-01-31T21:02:46.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:46 smithi167 conmon[49112]: debug 2022-01-31T21:02:46.492+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:46.492119+0000) 2022-01-31T21:02:46.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:46 smithi171 conmon[41853]: debug 2022-01-31T21:02:46.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:46.653117+0000) 2022-01-31T21:02:46.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:46 smithi171 conmon[51620]: debug 2022-01-31T21:02:46.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:46.525951+0000) 2022-01-31T21:02:47.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:47 smithi167 conmon[60316]: debug 2022-01-31T21:02:47.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:47.171599+0000) 2022-01-31T21:02:47.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:47 smithi167 conmon[54076]: debug 2022-01-31T21:02:47.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:47.420052+0000) 2022-01-31T21:02:47.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:47 smithi171 conmon[46715]: debug 2022-01-31T21:02:47.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:47.198069+0000) 2022-01-31T21:02:47.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:47 smithi167 conmon[49112]: debug 2022-01-31T21:02:47.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:47.492266+0000) 2022-01-31T21:02:47.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:47 smithi171 conmon[51620]: debug 2022-01-31T21:02:47.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:47.526090+0000) 2022-01-31T21:02:47.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:47 smithi171 conmon[41853]: debug 2022-01-31T21:02:47.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:47.653306+0000) 2022-01-31T21:02:48.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:48 smithi167 conmon[60316]: debug 2022-01-31T21:02:48.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:48.171759+0000) 2022-01-31T21:02:48.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:48 smithi167 conmon[54076]: debug 2022-01-31T21:02:48.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:48.420238+0000) 2022-01-31T21:02:48.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:48 smithi171 conmon[46715]: debug 2022-01-31T21:02:48.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:48.198245+0000) 2022-01-31T21:02:48.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:48 smithi167 conmon[49112]: debug 2022-01-31T21:02:48.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:48.492419+0000) 2022-01-31T21:02:48.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:48 smithi171 conmon[41853]: debug 2022-01-31T21:02:48.652+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:48.653456+0000) 2022-01-31T21:02:48.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:48 smithi171 conmon[51620]: debug 2022-01-31T21:02:48.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:48.526282+0000) 2022-01-31T21:02:49.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:49 smithi167 conmon[54076]: debug 2022-01-31T21:02:49.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:49.420395+0000) 2022-01-31T21:02:49.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:49 smithi167 conmon[60316]: debug 2022-01-31T21:02:49.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:49.171906+0000) 2022-01-31T21:02:49.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:49 smithi171 conmon[46715]: debug 2022-01-31T21:02:49.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:49.198416+0000) 2022-01-31T21:02:49.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:49 smithi167 conmon[49112]: debug 2022-01-31T21:02:49.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:49.492616+0000) 2022-01-31T21:02:49.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:49 smithi171 conmon[51620]: debug 2022-01-31T21:02:49.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:49.526482+0000) 2022-01-31T21:02:49.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:49 smithi171 conmon[41853]: debug 2022-01-31T21:02:49.651+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:49.653630+0000) 2022-01-31T21:02:50.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[60316]: debug 2022-01-31T21:02:50.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.172083+0000) 2022-01-31T21:02:50.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[54076]: debug 2022-01-31T21:02:50.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.420599+0000) 2022-01-31T21:02:50.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[46715]: debug 2022-01-31T21:02:50.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.198616+0000) 2022-01-31T21:02:50.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[49112]: debug 2022-01-31T21:02:50.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.492798+0000) 2022-01-31T21:02:50.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[49112]: debug 2022-01-31T21:02:50.555+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.557005+0000) 2022-01-31T21:02:50.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[60316]: debug 2022-01-31T21:02:50.556+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.558067+0000) 2022-01-31T21:02:50.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[54076]: debug 2022-01-31T21:02:50.556+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.557612+0000) 2022-01-31T21:02:50.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:50 smithi167 conmon[54076]: 2022-01-31T21:02:50.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[35325]: debug 2022-01-31T21:02:50.565+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98764 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:50.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[41853]: debug 2022-01-31T21:02:50.556+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.557910+0000) 2022-01-31T21:02:50.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[41853]: debug 2022-01-31T21:02:50.652+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.653811+0000) 2022-01-31T21:02:50.779 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[46715]: debug 2022-01-31T21:02:50.554+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.556672+0000) 2022-01-31T21:02:50.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[51620]: debug 2022-01-31T21:02:50.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.526680+0000) 2022-01-31T21:02:50.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:50 smithi171 conmon[51620]: debug 2022-01-31T21:02:50.555+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:50.557119+0000) 2022-01-31T21:02:51.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:51 smithi167 conmon[60316]: debug 2022-01-31T21:02:51.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:51.172261+0000) 2022-01-31T21:02:51.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:51 smithi167 conmon[54076]: debug 2022-01-31T21:02:51.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:51.420744+0000) 2022-01-31T21:02:51.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:51 smithi171 conmon[46715]: debug 2022-01-31T21:02:51.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:51.198808+0000) 2022-01-31T21:02:51.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:51 smithi167 conmon[49112]: debug 2022-01-31T21:02:51.492+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:51.492935+0000) 2022-01-31T21:02:51.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:51 smithi171 conmon[41853]: debug 2022-01-31T21:02:51.652+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:51.653956+0000) 2022-01-31T21:02:51.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:51 smithi171 conmon[51620]: debug 2022-01-31T21:02:51.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:51.526844+0000) 2022-01-31T21:02:52.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:52 smithi167 conmon[54076]: debug 2022-01-31T21:02:52.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:52.420898+0000) 2022-01-31T21:02:52.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:52 smithi167 conmon[60316]: debug 2022-01-31T21:02:52.170+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:52.172397+0000) 2022-01-31T21:02:52.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:52 smithi171 conmon[46715]: debug 2022-01-31T21:02:52.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:52.198978+0000) 2022-01-31T21:02:52.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:52 smithi167 conmon[49112]: debug 2022-01-31T21:02:52.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:52.493035+0000) 2022-01-31T21:02:52.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:52 smithi171 conmon[41853]: debug 2022-01-31T21:02:52.653+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:52.654157+0000) 2022-01-31T21:02:52.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:52 smithi171 conmon[51620]: debug 2022-01-31T21:02:52.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:52.526989+0000) 2022-01-31T21:02:53.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:53 smithi167 conmon[60316]: debug 2022-01-31T21:02:53.171+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:53.172619+0000) 2022-01-31T21:02:53.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:53 smithi167 conmon[54076]: debug 2022-01-31T21:02:53.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:53.421087+0000) 2022-01-31T21:02:53.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:53 smithi171 conmon[46715]: debug 2022-01-31T21:02:53.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:53.199182+0000) 2022-01-31T21:02:53.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:53 smithi167 conmon[49112]: debug 2022-01-31T21:02:53.491+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:53.493247+0000) 2022-01-31T21:02:53.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:53 smithi171 conmon[41853]: debug 2022-01-31T21:02:53.653+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:53.654324+0000) 2022-01-31T21:02:53.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:53 smithi171 conmon[35325]: debug 2022-01-31T21:02:53.613+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:02:53.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:53 smithi171 conmon[51620]: debug 2022-01-31T21:02:53.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:53.527189+0000) 2022-01-31T21:02:54.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:54 smithi167 conmon[54076]: debug 2022-01-31T21:02:54.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:54.421249+0000) 2022-01-31T21:02:54.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:54 smithi167 conmon[60316]: debug 2022-01-31T21:02:54.171+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:54.172778+0000) 2022-01-31T21:02:54.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:54 smithi171 conmon[46715]: debug 2022-01-31T21:02:54.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:54.199352+0000) 2022-01-31T21:02:54.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:54 smithi167 conmon[49112]: debug 2022-01-31T21:02:54.492+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:54.493445+0000) 2022-01-31T21:02:54.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:54 smithi171 conmon[41853]: debug 2022-01-31T21:02:54.653+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:54.654497+0000) 2022-01-31T21:02:54.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:54 smithi171 conmon[51620]: debug 2022-01-31T21:02:54.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:54.527366+0000) 2022-01-31T21:02:55.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:55 smithi167 conmon[54076]: debug 2022-01-31T21:02:55.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.421453+0000) 2022-01-31T21:02:55.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:55 smithi167 conmon[60316]: debug 2022-01-31T21:02:55.171+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.172909+0000) 2022-01-31T21:02:55.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[46715]: debug 2022-01-31T21:02:55.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.199504+0000) 2022-01-31T21:02:55.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:55 smithi167 conmon[54076]: debug 2022-01-31T21:02:55.568+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.570186+0000) 2022-01-31T21:02:55.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:55 smithi167 conmon[60316]: debug 2022-01-31T21:02:55.568+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.569680+0000) 2022-01-31T21:02:55.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:55 smithi167 conmon[49112]: debug 2022-01-31T21:02:55.492+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.493598+0000) 2022-01-31T21:02:55.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:55 smithi167 conmon[49112]: debug 2022-01-31T21:02:55.567+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.569205+0000) 2022-01-31T21:02:55.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[35325]: debug 2022-01-31T21:02:55.578+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98874 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:02:55.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[41853]: debug 2022-01-31T21:02:55.569+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.570406+0000) 2022-01-31T21:02:55.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[41853]: debug 2022-01-31T21:02:55.653+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.654606+0000) 2022-01-31T21:02:55.780 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[46715]: debug 2022-01-31T21:02:55.567+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.568521+0000) 2022-01-31T21:02:55.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[51620]: debug 2022-01-31T21:02:55.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.527539+0000) 2022-01-31T21:02:55.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[51620]: debug 2022-01-31T21:02:55.569+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:55.569874+0000) 2022-01-31T21:02:55.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:55 smithi171 conmon[51620]: 2022-01-31T21:02:56.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:56 smithi167 conmon[60316]: debug 2022-01-31T21:02:56.171+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:56.173049+0000) 2022-01-31T21:02:56.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:56 smithi167 conmon[54076]: debug 2022-01-31T21:02:56.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:56.421622+0000) 2022-01-31T21:02:56.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:56 smithi171 conmon[46715]: debug 2022-01-31T21:02:56.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:56.199687+0000) 2022-01-31T21:02:56.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:56 smithi167 conmon[49112]: debug 2022-01-31T21:02:56.492+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:56.493810+0000) 2022-01-31T21:02:56.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:56 smithi171 conmon[41853]: debug 2022-01-31T21:02:56.653+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:56.654724+0000) 2022-01-31T21:02:56.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:56 smithi171 conmon[51620]: debug 2022-01-31T21:02:56.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:56.527729+0000) 2022-01-31T21:02:57.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:57 smithi167 conmon[54076]: debug 2022-01-31T21:02:57.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:57.421774+0000) 2022-01-31T21:02:57.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:57 smithi167 conmon[60316]: debug 2022-01-31T21:02:57.171+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:57.173236+0000) 2022-01-31T21:02:57.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:57 smithi171 conmon[46715]: debug 2022-01-31T21:02:57.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:57.199807+0000) 2022-01-31T21:02:57.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:57 smithi167 conmon[49112]: debug 2022-01-31T21:02:57.493+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:57.493962+0000) 2022-01-31T21:02:57.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:57 smithi171 conmon[41853]: debug 2022-01-31T21:02:57.654+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:57.654908+0000) 2022-01-31T21:02:57.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:57 smithi171 conmon[51620]: debug 2022-01-31T21:02:57.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:57.527868+0000) 2022-01-31T21:02:58.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:58 smithi167 conmon[54076]: debug 2022-01-31T21:02:58.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:58.421974+0000) 2022-01-31T21:02:58.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:58 smithi167 conmon[60316]: debug 2022-01-31T21:02:58.171+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:58.173445+0000) 2022-01-31T21:02:58.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:58 smithi171 conmon[46715]: debug 2022-01-31T21:02:58.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:58.199977+0000) 2022-01-31T21:02:58.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:58 smithi167 conmon[49112]: debug 2022-01-31T21:02:58.493+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:58.494147+0000) 2022-01-31T21:02:58.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:58 smithi171 conmon[51620]: debug 2022-01-31T21:02:58.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:58.527996+0000) 2022-01-31T21:02:58.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:58 smithi171 conmon[41853]: debug 2022-01-31T21:02:58.654+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:58.655082+0000) 2022-01-31T21:02:59.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:02:59 smithi167 conmon[54076]: debug 2022-01-31T21:02:59.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:59.422193+0000) 2022-01-31T21:02:59.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:02:59 smithi167 conmon[60316]: debug 2022-01-31T21:02:59.172+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:59.173636+0000) 2022-01-31T21:02:59.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:02:59 smithi171 conmon[46715]: debug 2022-01-31T21:02:59.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:59.200132+0000) 2022-01-31T21:02:59.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:02:59 smithi167 conmon[49112]: debug 2022-01-31T21:02:59.492+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:59.494356+0000) 2022-01-31T21:02:59.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:02:59 smithi171 conmon[41853]: debug 2022-01-31T21:02:59.654+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:59.655262+0000) 2022-01-31T21:02:59.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:02:59 smithi171 conmon[51620]: debug 2022-01-31T21:02:59.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:02:59.528204+0000) 2022-01-31T21:03:00.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[54076]: debug 2022-01-31T21:03:00.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.422400+0000) 2022-01-31T21:03:00.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[60316]: debug 2022-01-31T21:03:00.172+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.173803+0000) 2022-01-31T21:03:00.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[46715]: debug 2022-01-31T21:03:00.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.200318+0000) 2022-01-31T21:03:00.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[49112]: debug 2022-01-31T21:03:00.493+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.494527+0000) 2022-01-31T21:03:00.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[49112]: debug 2022-01-31T21:03:00.579+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.580977+0000) 2022-01-31T21:03:00.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[54076]: debug 2022-01-31T21:03:00.579+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.581291+0000) 2022-01-31T21:03:00.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[60316]: debug 2022-01-31T21:03:00.580+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.581779+0000) 2022-01-31T21:03:00.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:00 smithi167 conmon[60316]: 2022-01-31T21:03:00.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[35325]: debug 2022-01-31T21:03:00.591+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 98984 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:00.780 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[46715]: debug 2022-01-31T21:03:00.581+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.582035+0000) 2022-01-31T21:03:00.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[41853]: debug 2022-01-31T21:03:00.581+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.582432+0000) 2022-01-31T21:03:00.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[41853]: debug 2022-01-31T21:03:00.654+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.655433+0000) 2022-01-31T21:03:00.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[51620]: debug 2022-01-31T21:03:00.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.528389+0000) 2022-01-31T21:03:00.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:00 smithi171 conmon[51620]: debug 2022-01-31T21:03:00.580+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:00.580847+0000) 2022-01-31T21:03:01.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:01 smithi167 conmon[54076]: debug 2022-01-31T21:03:01.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:01.422608+0000) 2022-01-31T21:03:01.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:01 smithi167 conmon[60316]: debug 2022-01-31T21:03:01.172+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:01.173962+0000) 2022-01-31T21:03:01.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:01 smithi171 conmon[46715]: debug 2022-01-31T21:03:01.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:01.200505+0000) 2022-01-31T21:03:01.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:01 smithi167 conmon[49112]: debug 2022-01-31T21:03:01.493+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:01.494672+0000) 2022-01-31T21:03:01.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:01 smithi171 conmon[41853]: debug 2022-01-31T21:03:01.654+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:01.655611+0000) 2022-01-31T21:03:01.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:01 smithi171 conmon[51620]: debug 2022-01-31T21:03:01.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:01.528609+0000) 2022-01-31T21:03:02.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:02 smithi167 conmon[60316]: debug 2022-01-31T21:03:02.172+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:02.174121+0000) 2022-01-31T21:03:02.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:02 smithi167 conmon[54076]: debug 2022-01-31T21:03:02.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:02.422724+0000) 2022-01-31T21:03:02.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:02 smithi171 conmon[46715]: debug 2022-01-31T21:03:02.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:02.200656+0000) 2022-01-31T21:03:02.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:02 smithi167 conmon[49112]: debug 2022-01-31T21:03:02.494+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:02.494838+0000) 2022-01-31T21:03:02.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:02 smithi171 conmon[41853]: debug 2022-01-31T21:03:02.655+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:02.655844+0000) 2022-01-31T21:03:02.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:02 smithi171 conmon[51620]: debug 2022-01-31T21:03:02.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:02.528746+0000) 2022-01-31T21:03:03.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:03 smithi167 conmon[54076]: debug 2022-01-31T21:03:03.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:03.422826+0000) 2022-01-31T21:03:03.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:03 smithi167 conmon[60316]: debug 2022-01-31T21:03:03.172+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:03.174330+0000) 2022-01-31T21:03:03.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:03 smithi171 conmon[46715]: debug 2022-01-31T21:03:03.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:03.200845+0000) 2022-01-31T21:03:03.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:03 smithi167 conmon[49112]: debug 2022-01-31T21:03:03.494+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:03.495049+0000) 2022-01-31T21:03:03.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:03 smithi171 conmon[41853]: debug 2022-01-31T21:03:03.655+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:03.656050+0000) 2022-01-31T21:03:03.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:03 smithi171 conmon[51620]: debug 2022-01-31T21:03:03.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:03.528891+0000) 2022-01-31T21:03:04.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:04 smithi167 conmon[54076]: debug 2022-01-31T21:03:04.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:04.422983+0000) 2022-01-31T21:03:04.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:04 smithi167 conmon[60316]: debug 2022-01-31T21:03:04.173+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:04.174530+0000) 2022-01-31T21:03:04.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:04 smithi171 conmon[46715]: debug 2022-01-31T21:03:04.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:04.200969+0000) 2022-01-31T21:03:04.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:04 smithi167 conmon[49112]: debug 2022-01-31T21:03:04.493+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:04.495277+0000) 2022-01-31T21:03:04.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:04 smithi171 conmon[51620]: debug 2022-01-31T21:03:04.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:04 smithi171 conmon[51620]: 2022-01-31T21:03:04.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:04.529000+0000) 2022-01-31T21:03:04.781 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:04 smithi171 conmon[41853]: debug 2022-01-31T21:03:04.655+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:04.656228+0000) 2022-01-31T21:03:05.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:05 smithi167 conmon[54076]: debug 2022-01-31T21:03:05.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.423190+0000) 2022-01-31T21:03:05.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:05 smithi167 conmon[60316]: debug 2022-01-31T21:03:05.173+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.174684+0000) 2022-01-31T21:03:05.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[46715]: debug 2022-01-31T21:03:05.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.201156+0000) 2022-01-31T21:03:05.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:05 smithi167 conmon[49112]: debug 2022-01-31T21:03:05.494+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.495473+0000) 2022-01-31T21:03:05.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:05 smithi167 conmon[49112]: debug 2022-01-31T21:03:05.593+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.595179+0000) 2022-01-31T21:03:05.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:05 smithi167 conmon[54076]: debug 2022-01-31T21:03:05.593+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.594601+0000) 2022-01-31T21:03:05.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:05 smithi167 conmon[60316]: debug 2022-01-31T21:03:05.593+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.594852+0000) 2022-01-31T21:03:05.780 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[35325]: debug 2022-01-31T21:03:05.603+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99095 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:05.781 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[46715]: debug 2022-01-31T21:03:05.593+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.594285+0000) 2022-01-31T21:03:05.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[51620]: debug 2022-01-31T21:03:05.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.529142+0000) 2022-01-31T21:03:05.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[51620]: debug 2022-01-31T21:03:05.593+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.594422+0000) 2022-01-31T21:03:05.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[41853]: debug 2022-01-31T21:03:05.593+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.594032+0000) 2022-01-31T21:03:05.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:05 smithi171 conmon[41853]: debug 2022-01-31T21:03:05.655+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:05.656441+0000) 2022-01-31T21:03:06.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:06 smithi167 conmon[54076]: debug 2022-01-31T21:03:06.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:06.423366+0000) 2022-01-31T21:03:06.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:06 smithi167 conmon[60316]: debug 2022-01-31T21:03:06.173+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:06.174801+0000) 2022-01-31T21:03:06.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:06 smithi171 conmon[46715]: debug 2022-01-31T21:03:06.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:06.201321+0000) 2022-01-31T21:03:06.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:06 smithi167 conmon[49112]: debug 2022-01-31T21:03:06.494+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:06.495662+0000) 2022-01-31T21:03:06.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:06 smithi171 conmon[41853]: debug 2022-01-31T21:03:06.656+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:06.656568+0000) 2022-01-31T21:03:06.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:06 smithi171 conmon[51620]: debug 2022-01-31T21:03:06.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:06.529360+0000) 2022-01-31T21:03:07.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:07 smithi167 conmon[54076]: debug 2022-01-31T21:03:07.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:07.423538+0000) 2022-01-31T21:03:07.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:07 smithi167 conmon[60316]: debug 2022-01-31T21:03:07.173+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:07.174943+0000) 2022-01-31T21:03:07.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:07 smithi171 conmon[46715]: debug 2022-01-31T21:03:07.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:07.201481+0000) 2022-01-31T21:03:07.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:07 smithi167 conmon[49112]: debug 2022-01-31T21:03:07.495+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:07.495820+0000) 2022-01-31T21:03:07.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:07 smithi171 conmon[41853]: debug 2022-01-31T21:03:07.656+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:07.656777+0000) 2022-01-31T21:03:07.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:07 smithi171 conmon[51620]: debug 2022-01-31T21:03:07.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:07.529533+0000) 2022-01-31T21:03:08.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:08 smithi167 conmon[54076]: debug 2022-01-31T21:03:08.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:08.423725+0000) 2022-01-31T21:03:08.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:08 smithi167 conmon[60316]: debug 2022-01-31T21:03:08.173+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:08.175166+0000) 2022-01-31T21:03:08.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:08 smithi171 conmon[46715]: debug 2022-01-31T21:03:08.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:08.201670+0000) 2022-01-31T21:03:08.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:08 smithi167 conmon[49112]: debug 2022-01-31T21:03:08.495+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:08.495983+0000) 2022-01-31T21:03:08.781 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:08 smithi171 conmon[41853]: debug 2022-01-31T21:03:08.656+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:08.656960+0000) 2022-01-31T21:03:08.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:08 smithi171 conmon[51620]: debug 2022-01-31T21:03:08.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:08.529741+0000) 2022-01-31T21:03:08.782 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:08 smithi171 conmon[35325]: debug 2022-01-31T21:03:08.614+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:03:09.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:09 smithi167 conmon[54076]: debug 2022-01-31T21:03:09.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:09.423924+0000) 2022-01-31T21:03:09.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:09 smithi167 conmon[60316]: debug 2022-01-31T21:03:09.173+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:09.175322+0000) 2022-01-31T21:03:09.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:09 smithi171 conmon[46715]: debug 2022-01-31T21:03:09.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:09.201866+0000) 2022-01-31T21:03:09.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:09 smithi167 conmon[49112]: debug 2022-01-31T21:03:09.495+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:09.496142+0000) 2022-01-31T21:03:09.781 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:09 smithi171 conmon[41853]: debug 2022-01-31T21:03:09.656+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:09.657150+0000) 2022-01-31T21:03:09.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:09 smithi171 conmon[51620]: debug 2022-01-31T21:03:09.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:09.529928+0000) 2022-01-31T21:03:10.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:10 smithi167 conmon[54076]: debug 2022-01-31T21:03:10.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.424118+0000) 2022-01-31T21:03:10.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:10 smithi167 conmon[60316]: debug 2022-01-31T21:03:10.174+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.175548+0000) 2022-01-31T21:03:10.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[46715]: debug 2022-01-31T21:03:10.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.202060+0000) 2022-01-31T21:03:10.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:10 smithi167 conmon[49112]: debug 2022-01-31T21:03:10.495+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.496342+0000) 2022-01-31T21:03:10.925 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:10 smithi167 conmon[49112]: debug 2022-01-31T21:03:10.606+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.607981+0000) 2022-01-31T21:03:10.926 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:10 smithi167 conmon[54076]: debug 2022-01-31T21:03:10.606+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.607790+0000) 2022-01-31T21:03:10.927 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:10 smithi167 conmon[60316]: debug 2022-01-31T21:03:10.605+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.607367+0000) 2022-01-31T21:03:10.928 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[35325]: debug 2022-01-31T21:03:10.616+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99205 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:10.928 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[41853]: debug 2022-01-31T21:03:10.606+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.607608+0000) 2022-01-31T21:03:10.929 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[41853]: debug 2022-01-31T21:03:10.656+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.657393+0000) 2022-01-31T21:03:10.929 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[46715]: debug 2022-01-31T21:03:10.606+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.607011+0000) 2022-01-31T21:03:10.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[51620]: debug 2022-01-31T21:03:10.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.530096+0000) 2022-01-31T21:03:10.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:10 smithi171 conmon[51620]: debug 2022-01-31T21:03:10.605+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:10.606534+0000) 2022-01-31T21:03:11.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:11 smithi167 conmon[54076]: debug 2022-01-31T21:03:11.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:11.424332+0000) 2022-01-31T21:03:11.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:11 smithi167 conmon[60316]: debug 2022-01-31T21:03:11.175+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:11.175654+0000) 2022-01-31T21:03:11.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:11 smithi171 conmon[46715]: debug 2022-01-31T21:03:11.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:11.202235+0000) 2022-01-31T21:03:11.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:11 smithi167 conmon[49112]: debug 2022-01-31T21:03:11.496+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:11.496534+0000) 2022-01-31T21:03:11.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:11 smithi171 conmon[51620]: debug 2022-01-31T21:03:11.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:11.530252+0000) 2022-01-31T21:03:11.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:11 smithi171 conmon[41853]: debug 2022-01-31T21:03:11.656+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:11.657569+0000) 2022-01-31T21:03:12.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:12 smithi167 conmon[54076]: debug 2022-01-31T21:03:12.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:12.424487+0000) 2022-01-31T21:03:12.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:12 smithi167 conmon[60316]: debug 2022-01-31T21:03:12.175+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:12.175746+0000) 2022-01-31T21:03:12.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:12 smithi171 conmon[46715]: debug 2022-01-31T21:03:12.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:12.202386+0000) 2022-01-31T21:03:12.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:12 smithi167 conmon[49112]: debug 2022-01-31T21:03:12.496+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:12.496690+0000) 2022-01-31T21:03:12.781 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:12 smithi171 conmon[41853]: debug 2022-01-31T21:03:12.657+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:12.657785+0000) 2022-01-31T21:03:12.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:12 smithi171 conmon[51620]: debug 2022-01-31T21:03:12.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:12.530401+0000) 2022-01-31T21:03:13.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:13 smithi167 conmon[54076]: debug 2022-01-31T21:03:13.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:13.424694+0000) 2022-01-31T21:03:13.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:13 smithi167 conmon[60316]: debug 2022-01-31T21:03:13.175+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:13.175879+0000) 2022-01-31T21:03:13.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:13 smithi171 conmon[46715]: debug 2022-01-31T21:03:13.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:13.202562+0000) 2022-01-31T21:03:13.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:13 smithi167 conmon[49112]: debug 2022-01-31T21:03:13.496+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:13.496843+0000) 2022-01-31T21:03:13.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:13 smithi171 conmon[41853]: debug 2022-01-31T21:03:13.657+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:13.657963+0000) 2022-01-31T21:03:13.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:13 smithi171 conmon[51620]: debug 2022-01-31T21:03:13.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:13.530555+0000) 2022-01-31T21:03:14.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:14 smithi167 conmon[54076]: debug 2022-01-31T21:03:14.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:14.424912+0000) 2022-01-31T21:03:14.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:14 smithi167 conmon[60316]: debug 2022-01-31T21:03:14.175+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:14.176032+0000) 2022-01-31T21:03:14.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:14 smithi171 conmon[46715]: debug 2022-01-31T21:03:14.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:14.202733+0000) 2022-01-31T21:03:14.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:14 smithi167 conmon[49112]: debug 2022-01-31T21:03:14.496+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:14.497033+0000) 2022-01-31T21:03:14.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:14 smithi171 conmon[51620]: debug 2022-01-31T21:03:14.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:14.530750+0000) 2022-01-31T21:03:14.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:14 smithi171 conmon[41853]: debug 2022-01-31T21:03:14.657+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:14.658097+0000) 2022-01-31T21:03:15.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[54076]: debug 2022-01-31T21:03:15.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.425044+0000) 2022-01-31T21:03:15.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[60316]: debug 2022-01-31T21:03:15.175+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.176188+0000) 2022-01-31T21:03:15.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[46715]: debug 2022-01-31T21:03:15.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.202965+0000) 2022-01-31T21:03:15.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[54076]: debug 2022-01-31T21:03:15.619+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.620467+0000) 2022-01-31T21:03:15.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[49112]: debug 2022-01-31T21:03:15.496+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.497216+0000) 2022-01-31T21:03:15.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[49112]: debug 2022-01-31T21:03:15.620+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.620564+0000) 2022-01-31T21:03:15.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[60316]: debug 2022-01-31T21:03:15.619+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.620056+0000) 2022-01-31T21:03:15.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:15 smithi167 conmon[60316]: 2022-01-31T21:03:15.782 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[35325]: debug 2022-01-31T21:03:15.629+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99315 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:15.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[41853]: debug 2022-01-31T21:03:15.618+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.619436+0000) 2022-01-31T21:03:15.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[41853]: debug 2022-01-31T21:03:15.657+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.658254+0000) 2022-01-31T21:03:15.784 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[46715]: debug 2022-01-31T21:03:15.618+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.619647+0000) 2022-01-31T21:03:15.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[51620]: debug 2022-01-31T21:03:15.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.530970+0000) 2022-01-31T21:03:15.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:15 smithi171 conmon[51620]: debug 2022-01-31T21:03:15.618+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:15.619193+0000) 2022-01-31T21:03:16.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:16 smithi167 conmon[54076]: debug 2022-01-31T21:03:16.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:16.425214+0000) 2022-01-31T21:03:16.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:16 smithi167 conmon[60316]: debug 2022-01-31T21:03:16.175+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:16.176404+0000) 2022-01-31T21:03:16.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:16 smithi171 conmon[46715]: debug 2022-01-31T21:03:16.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:16.203172+0000) 2022-01-31T21:03:16.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:16 smithi167 conmon[49112]: debug 2022-01-31T21:03:16.496+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:16.497400+0000) 2022-01-31T21:03:16.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:16 smithi171 conmon[51620]: debug 2022-01-31T21:03:16.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:16.531118+0000) 2022-01-31T21:03:16.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:16 smithi171 conmon[41853]: debug 2022-01-31T21:03:16.657+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:16.658381+0000) 2022-01-31T21:03:17.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:17 smithi167 conmon[54076]: debug 2022-01-31T21:03:17.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:17.425309+0000) 2022-01-31T21:03:17.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:17 smithi167 conmon[60316]: debug 2022-01-31T21:03:17.176+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:17.176595+0000) 2022-01-31T21:03:17.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:17 smithi171 conmon[46715]: debug 2022-01-31T21:03:17.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:17.203329+0000) 2022-01-31T21:03:17.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:17 smithi167 conmon[49112]: debug 2022-01-31T21:03:17.497+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:17.497572+0000) 2022-01-31T21:03:17.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:17 smithi171 conmon[51620]: debug 2022-01-31T21:03:17.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:17.531273+0000) 2022-01-31T21:03:17.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:17 smithi171 conmon[41853]: debug 2022-01-31T21:03:17.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:17.658479+0000) 2022-01-31T21:03:18.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:18 smithi167 conmon[54076]: debug 2022-01-31T21:03:18.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:18.425467+0000) 2022-01-31T21:03:18.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:18 smithi167 conmon[60316]: debug 2022-01-31T21:03:18.176+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:18.176723+0000) 2022-01-31T21:03:18.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:18 smithi171 conmon[46715]: debug 2022-01-31T21:03:18.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:18.203514+0000) 2022-01-31T21:03:18.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:18 smithi167 conmon[49112]: debug 2022-01-31T21:03:18.497+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:18.497766+0000) 2022-01-31T21:03:18.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:18 smithi171 conmon[41853]: debug 2022-01-31T21:03:18.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:18.658640+0000) 2022-01-31T21:03:18.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:18 smithi171 conmon[51620]: debug 2022-01-31T21:03:18.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:18.531478+0000) 2022-01-31T21:03:19.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:19 smithi167 conmon[54076]: debug 2022-01-31T21:03:19.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:19.425612+0000) 2022-01-31T21:03:19.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:19 smithi167 conmon[60316]: debug 2022-01-31T21:03:19.176+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:19.176887+0000) 2022-01-31T21:03:19.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:19 smithi171 conmon[46715]: debug 2022-01-31T21:03:19.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:19.203700+0000) 2022-01-31T21:03:19.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:19 smithi167 conmon[49112]: debug 2022-01-31T21:03:19.497+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:19.497871+0000) 2022-01-31T21:03:19.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:19 smithi171 conmon[51620]: debug 2022-01-31T21:03:19.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:19.531698+0000) 2022-01-31T21:03:19.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:19 smithi171 conmon[41853]: debug 2022-01-31T21:03:19.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:19.658777+0000) 2022-01-31T21:03:20.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:20 smithi167 conmon[54076]: debug 2022-01-31T21:03:20.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.425803+0000) 2022-01-31T21:03:20.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:20 smithi167 conmon[60316]: debug 2022-01-31T21:03:20.176+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.177088+0000) 2022-01-31T21:03:20.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[46715]: debug 2022-01-31T21:03:20.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.203902+0000) 2022-01-31T21:03:20.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:20 smithi167 conmon[54076]: debug 2022-01-31T21:03:20.633+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.633861+0000) 2022-01-31T21:03:20.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:20 smithi167 conmon[49112]: debug 2022-01-31T21:03:20.497+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.498078+0000) 2022-01-31T21:03:20.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:20 smithi167 conmon[49112]: debug 2022-01-31T21:03:20.632+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.633423+0000) 2022-01-31T21:03:20.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:20 smithi167 conmon[60316]: debug 2022-01-31T21:03:20.632+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.632759+0000) 2022-01-31T21:03:20.783 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[35325]: debug 2022-01-31T21:03:20.641+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99425 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:20.784 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[46715]: debug 2022-01-31T21:03:20.631+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.632375+0000) 2022-01-31T21:03:20.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[41853]: debug 2022-01-31T21:03:20.632+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.633505+0000) 2022-01-31T21:03:20.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[41853]: debug 2022-01-31T21:03:20.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.658924+0000) 2022-01-31T21:03:20.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[51620]: debug 2022-01-31T21:03:20.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.531840+0000) 2022-01-31T21:03:20.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:20 smithi171 conmon[51620]: debug 2022-01-31T21:03:20.631+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:20.632412+0000) 2022-01-31T21:03:21.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:21 smithi167 conmon[54076]: debug 2022-01-31T21:03:21.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:21.425959+0000) 2022-01-31T21:03:21.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:21 smithi167 conmon[60316]: debug 2022-01-31T21:03:21.176+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:21.177315+0000) 2022-01-31T21:03:21.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:21 smithi171 conmon[46715]: debug 2022-01-31T21:03:21.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:21.204106+0000) 2022-01-31T21:03:21.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:21 smithi167 conmon[49112]: debug 2022-01-31T21:03:21.497+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:21.498276+0000) 2022-01-31T21:03:21.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:21 smithi171 conmon[51620]: debug 2022-01-31T21:03:21.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:21.531991+0000) 2022-01-31T21:03:21.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:21 smithi171 conmon[41853]: debug 2022-01-31T21:03:21.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:21.659103+0000) 2022-01-31T21:03:22.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:22 smithi167 conmon[54076]: debug 2022-01-31T21:03:22.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:22.426131+0000) 2022-01-31T21:03:22.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:22 smithi167 conmon[60316]: debug 2022-01-31T21:03:22.176+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:22.177475+0000) 2022-01-31T21:03:22.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:22 smithi171 conmon[46715]: debug 2022-01-31T21:03:22.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:22.204235+0000) 2022-01-31T21:03:22.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:22 smithi167 conmon[49112]: debug 2022-01-31T21:03:22.497+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:22.498458+0000) 2022-01-31T21:03:22.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:22 smithi171 conmon[41853]: debug 2022-01-31T21:03:22.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:22.659299+0000) 2022-01-31T21:03:22.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:22 smithi171 conmon[51620]: debug 2022-01-31T21:03:22.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:22.532134+0000) 2022-01-31T21:03:23.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:23 smithi167 conmon[54076]: debug 2022-01-31T21:03:23.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:23.426367+0000) 2022-01-31T21:03:23.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:23 smithi167 conmon[60316]: debug 2022-01-31T21:03:23.177+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:23.177640+0000) 2022-01-31T21:03:23.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:23 smithi171 conmon[46715]: debug 2022-01-31T21:03:23.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:23.204423+0000) 2022-01-31T21:03:23.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:23 smithi167 conmon[49112]: debug 2022-01-31T21:03:23.498+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:23.498616+0000) 2022-01-31T21:03:23.783 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:23 smithi171 conmon[35325]: debug 2022-01-31T21:03:23.615+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:03:23.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:23 smithi171 conmon[41853]: debug 2022-01-31T21:03:23.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:23.659481+0000) 2022-01-31T21:03:23.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:23 smithi171 conmon[51620]: debug 2022-01-31T21:03:23.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:23.532252+0000) 2022-01-31T21:03:24.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:24 smithi167 conmon[54076]: debug 2022-01-31T21:03:24.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:24.426556+0000) 2022-01-31T21:03:24.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:24 smithi167 conmon[60316]: debug 2022-01-31T21:03:24.177+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:24.177795+0000) 2022-01-31T21:03:24.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:24 smithi171 conmon[46715]: debug 2022-01-31T21:03:24.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:24.204622+0000) 2022-01-31T21:03:24.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:24 smithi167 conmon[49112]: debug 2022-01-31T21:03:24.498+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:24.498775+0000) 2022-01-31T21:03:24.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:24 smithi171 conmon[51620]: debug 2022-01-31T21:03:24.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:24.532463+0000) 2022-01-31T21:03:24.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:24 smithi171 conmon[41853]: debug 2022-01-31T21:03:24.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:24.659670+0000) 2022-01-31T21:03:25.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:25 smithi167 conmon[54076]: debug 2022-01-31T21:03:25.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.426713+0000) 2022-01-31T21:03:25.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:25 smithi167 conmon[60316]: debug 2022-01-31T21:03:25.177+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.177964+0000) 2022-01-31T21:03:25.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[46715]: debug 2022-01-31T21:03:25.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.204819+0000) 2022-01-31T21:03:25.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:25 smithi167 conmon[60316]: debug 2022-01-31T21:03:25.644+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.645026+0000) 2022-01-31T21:03:25.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:25 smithi167 conmon[49112]: debug 2022-01-31T21:03:25.498+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.498976+0000) 2022-01-31T21:03:25.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:25 smithi167 conmon[49112]: debug 2022-01-31T21:03:25.644+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.644896+0000) 2022-01-31T21:03:25.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:25 smithi167 conmon[54076]: debug 2022-01-31T21:03:25.645+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.646094+0000) 2022-01-31T21:03:25.784 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[35325]: debug 2022-01-31T21:03:25.654+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99535 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:25.784 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[46715]: debug 2022-01-31T21:03:25.645+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.645791+0000) 2022-01-31T21:03:25.785 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[41853]: debug 2022-01-31T21:03:25.644+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.645664+0000) 2022-01-31T21:03:25.785 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[41853]: debug 2022-01-31T21:03:25.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.659769+0000) 2022-01-31T21:03:25.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[51620]: debug 2022-01-31T21:03:25.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.532621+0000) 2022-01-31T21:03:25.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:25 smithi171 conmon[51620]: debug 2022-01-31T21:03:25.644+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:25.645201+0000) 2022-01-31T21:03:26.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:26 smithi167 conmon[60316]: debug 2022-01-31T21:03:26.177+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:26.178143+0000) 2022-01-31T21:03:26.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:26 smithi167 conmon[54076]: debug 2022-01-31T21:03:26.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:26.426869+0000) 2022-01-31T21:03:26.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:26 smithi171 conmon[46715]: debug 2022-01-31T21:03:26.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:26.204989+0000) 2022-01-31T21:03:26.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:26 smithi167 conmon[49112]: debug 2022-01-31T21:03:26.498+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:26.499140+0000) 2022-01-31T21:03:26.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:26 smithi171 conmon[41853]: debug 2022-01-31T21:03:26.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:26.659921+0000) 2022-01-31T21:03:26.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:26 smithi171 conmon[51620]: debug 2022-01-31T21:03:26.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:26.532811+0000) 2022-01-31T21:03:27.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:27 smithi167 conmon[60316]: debug 2022-01-31T21:03:27.177+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:27.178304+0000) 2022-01-31T21:03:27.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:27 smithi167 conmon[54076]: debug 2022-01-31T21:03:27.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:27.427039+0000) 2022-01-31T21:03:27.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:27 smithi171 conmon[46715]: debug 2022-01-31T21:03:27.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:27.205120+0000) 2022-01-31T21:03:27.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:27 smithi167 conmon[49112]: debug 2022-01-31T21:03:27.498+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:27.499330+0000) 2022-01-31T21:03:27.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:27 smithi171 conmon[41853]: debug 2022-01-31T21:03:27.660+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:27.660109+0000) 2022-01-31T21:03:27.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:27 smithi171 conmon[51620]: debug 2022-01-31T21:03:27.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:27.532981+0000) 2022-01-31T21:03:28.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:28 smithi167 conmon[60316]: debug 2022-01-31T21:03:28.177+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:28.178439+0000) 2022-01-31T21:03:28.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:28 smithi167 conmon[54076]: debug 2022-01-31T21:03:28.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:28.427229+0000) 2022-01-31T21:03:28.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:28 smithi171 conmon[46715]: debug 2022-01-31T21:03:28.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:28.205304+0000) 2022-01-31T21:03:28.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:28 smithi167 conmon[49112]: debug 2022-01-31T21:03:28.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:28.499531+0000) 2022-01-31T21:03:28.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:28 smithi171 conmon[51620]: debug 2022-01-31T21:03:28.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:28.533135+0000) 2022-01-31T21:03:28.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:28 smithi171 conmon[41853]: debug 2022-01-31T21:03:28.660+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:28.660366+0000) 2022-01-31T21:03:29.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:29 smithi167 conmon[60316]: debug 2022-01-31T21:03:29.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:29.178604+0000) 2022-01-31T21:03:29.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:29 smithi167 conmon[54076]: debug 2022-01-31T21:03:29.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:29.427441+0000) 2022-01-31T21:03:29.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:29 smithi171 conmon[46715]: debug 2022-01-31T21:03:29.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:29.205496+0000) 2022-01-31T21:03:29.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:29 smithi167 conmon[49112]: debug 2022-01-31T21:03:29.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:29.499736+0000) 2022-01-31T21:03:29.784 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:29 smithi171 conmon[41853]: debug 2022-01-31T21:03:29.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:29.660572+0000) 2022-01-31T21:03:29.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:29 smithi171 conmon[51620]: debug 2022-01-31T21:03:29.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:29.533282+0000) 2022-01-31T21:03:30.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:30 smithi167 conmon[60316]: debug 2022-01-31T21:03:30.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.178753+0000) 2022-01-31T21:03:30.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:30 smithi167 conmon[54076]: debug 2022-01-31T21:03:30.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.427641+0000) 2022-01-31T21:03:30.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[46715]: debug 2022-01-31T21:03:30.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.205700+0000) 2022-01-31T21:03:30.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:30 smithi167 conmon[54076]: debug 2022-01-31T21:03:30.657+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.658325+0000) 2022-01-31T21:03:30.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:30 smithi167 conmon[60316]: debug 2022-01-31T21:03:30.657+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.658175+0000) 2022-01-31T21:03:30.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:30 smithi167 conmon[49112]: debug 2022-01-31T21:03:30.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.499923+0000) 2022-01-31T21:03:30.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:30 smithi167 conmon[49112]: debug 2022-01-31T21:03:30.658+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.658573+0000) 2022-01-31T21:03:30.785 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[35325]: debug 2022-01-31T21:03:30.666+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99645 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:30.785 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[46715]: debug 2022-01-31T21:03:30.656+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.658574+0000) 2022-01-31T21:03:30.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[41853]: debug 2022-01-31T21:03:30.656+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.657863+0000) 2022-01-31T21:03:30.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[41853]: debug 2022-01-31T21:03:30.658+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.660699+0000) 2022-01-31T21:03:30.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[51620]: debug 2022-01-31T21:03:30.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.533485+0000) 2022-01-31T21:03:30.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:30 smithi171 conmon[51620]: debug 2022-01-31T21:03:30.655+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:30.657407+0000) 2022-01-31T21:03:31.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:31 smithi167 conmon[54076]: debug 2022-01-31T21:03:31.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:31.427794+0000) 2022-01-31T21:03:31.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:31 smithi167 conmon[60316]: debug 2022-01-31T21:03:31.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:31.178904+0000) 2022-01-31T21:03:31.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:31 smithi171 conmon[46715]: debug 2022-01-31T21:03:31.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:31.205873+0000) 2022-01-31T21:03:31.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:31 smithi167 conmon[49112]: debug 2022-01-31T21:03:31.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:31.500110+0000) 2022-01-31T21:03:31.785 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:31 smithi171 conmon[41853]: debug 2022-01-31T21:03:31.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:31.660796+0000) 2022-01-31T21:03:31.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:31 smithi171 conmon[51620]: debug 2022-01-31T21:03:31.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:31.533676+0000) 2022-01-31T21:03:32.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:32 smithi167 conmon[60316]: debug 2022-01-31T21:03:32.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:32.179044+0000) 2022-01-31T21:03:32.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:32 smithi167 conmon[54076]: debug 2022-01-31T21:03:32.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:32.427889+0000) 2022-01-31T21:03:32.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:32 smithi171 conmon[46715]: debug 2022-01-31T21:03:32.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:32.205987+0000) 2022-01-31T21:03:32.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:32 smithi167 conmon[49112]: debug 2022-01-31T21:03:32.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:32.500296+0000) 2022-01-31T21:03:32.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:32 smithi171 conmon[51620]: debug 2022-01-31T21:03:32.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:32.533830+0000) 2022-01-31T21:03:32.785 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:32 smithi171 conmon[41853]: debug 2022-01-31T21:03:32.660+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:32.660991+0000) 2022-01-31T21:03:33.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:33 smithi167 conmon[54076]: debug 2022-01-31T21:03:33.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:33.428011+0000) 2022-01-31T21:03:33.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:33 smithi167 conmon[60316]: debug 2022-01-31T21:03:33.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:33.179227+0000) 2022-01-31T21:03:33.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:33 smithi171 conmon[46715]: debug 2022-01-31T21:03:33.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:33.206154+0000) 2022-01-31T21:03:33.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:33 smithi167 conmon[49112]: debug 2022-01-31T21:03:33.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:33.500480+0000) 2022-01-31T21:03:33.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:33 smithi171 conmon[51620]: debug 2022-01-31T21:03:33.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:33.534001+0000) 2022-01-31T21:03:33.785 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:33 smithi171 conmon[41853]: debug 2022-01-31T21:03:33.660+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:33.661197+0000) 2022-01-31T21:03:34.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:34 smithi167 conmon[54076]: debug 2022-01-31T21:03:34.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:34.428170+0000) 2022-01-31T21:03:34.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:34 smithi167 conmon[60316]: debug 2022-01-31T21:03:34.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:34.179437+0000) 2022-01-31T21:03:34.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:34 smithi171 conmon[46715]: debug 2022-01-31T21:03:34.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:34.206356+0000) 2022-01-31T21:03:34.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:34 smithi167 conmon[49112]: debug 2022-01-31T21:03:34.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:34.500672+0000) 2022-01-31T21:03:34.785 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:34 smithi171 conmon[41853]: debug 2022-01-31T21:03:34.659+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:34.661369+0000) 2022-01-31T21:03:34.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:34 smithi171 conmon[51620]: debug 2022-01-31T21:03:34.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:34.534158+0000) 2022-01-31T21:03:35.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:35 smithi167 conmon[54076]: debug 2022-01-31T21:03:35.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.428387+0000) 2022-01-31T21:03:35.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:35 smithi167 conmon[60316]: debug 2022-01-31T21:03:35.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.179584+0000) 2022-01-31T21:03:35.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[46715]: debug 2022-01-31T21:03:35.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.206523+0000) 2022-01-31T21:03:35.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:35 smithi167 conmon[54076]: debug 2022-01-31T21:03:35.669+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.669797+0000) 2022-01-31T21:03:35.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:35 smithi167 conmon[60316]: debug 2022-01-31T21:03:35.672+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.672531+0000) 2022-01-31T21:03:35.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:35 smithi167 conmon[49112]: debug 2022-01-31T21:03:35.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.500831+0000) 2022-01-31T21:03:35.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:35 smithi167 conmon[49112]: debug 2022-01-31T21:03:35.670+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.670722+0000) 2022-01-31T21:03:35.786 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[35325]: debug 2022-01-31T21:03:35.694+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99755 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:35.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[41853]: debug 2022-01-31T21:03:35.660+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.661544+0000) 2022-01-31T21:03:35.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[41853]: debug 2022-01-31T21:03:35.666+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.670509+0000) 2022-01-31T21:03:35.787 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[46715]: debug 2022-01-31T21:03:35.666+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.671402+0000) 2022-01-31T21:03:35.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[51620]: debug 2022-01-31T21:03:35.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.534345+0000) 2022-01-31T21:03:35.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:35 smithi171 conmon[51620]: debug 2022-01-31T21:03:35.666+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:35.670594+0000) 2022-01-31T21:03:36.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:36 smithi167 conmon[54076]: debug 2022-01-31T21:03:36.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:36.428595+0000) 2022-01-31T21:03:36.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:36 smithi167 conmon[60316]: debug 2022-01-31T21:03:36.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:36.179771+0000) 2022-01-31T21:03:36.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:36 smithi171 conmon[46715]: debug 2022-01-31T21:03:36.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:36.206701+0000) 2022-01-31T21:03:36.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:36 smithi167 conmon[49112]: debug 2022-01-31T21:03:36.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:36.501031+0000) 2022-01-31T21:03:36.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:36 smithi171 conmon[51620]: debug 2022-01-31T21:03:36.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:36.534550+0000) 2022-01-31T21:03:36.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:36 smithi171 conmon[41853]: debug 2022-01-31T21:03:36.660+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:36.661737+0000) 2022-01-31T21:03:37.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:37 smithi167 conmon[54076]: debug 2022-01-31T21:03:37.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:37.428750+0000) 2022-01-31T21:03:37.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:37 smithi167 conmon[60316]: debug 2022-01-31T21:03:37.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:37.179959+0000) 2022-01-31T21:03:37.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:37 smithi171 conmon[46715]: debug 2022-01-31T21:03:37.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:37.206927+0000) 2022-01-31T21:03:37.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:37 smithi167 conmon[49112]: debug 2022-01-31T21:03:37.499+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:37.501187+0000) 2022-01-31T21:03:37.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:37 smithi171 conmon[41853]: debug 2022-01-31T21:03:37.661+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:37.661875+0000) 2022-01-31T21:03:37.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:37 smithi171 conmon[51620]: debug 2022-01-31T21:03:37.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:37.534716+0000) 2022-01-31T21:03:38.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:38 smithi167 conmon[54076]: debug 2022-01-31T21:03:38.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:38.428879+0000) 2022-01-31T21:03:38.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:38 smithi167 conmon[60316]: debug 2022-01-31T21:03:38.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:38.180133+0000) 2022-01-31T21:03:38.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:38 smithi171 conmon[46715]: debug 2022-01-31T21:03:38.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:38.207118+0000) 2022-01-31T21:03:38.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:38 smithi167 conmon[49112]: debug 2022-01-31T21:03:38.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:38.501383+0000) 2022-01-31T21:03:38.786 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:38 smithi171 conmon[35325]: debug 2022-01-31T21:03:38.615+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:03:38.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:38 smithi171 conmon[41853]: debug 2022-01-31T21:03:38.661+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:38.662033+0000) 2022-01-31T21:03:38.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:38 smithi171 conmon[51620]: debug 2022-01-31T21:03:38.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:38.534945+0000) 2022-01-31T21:03:39.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:39 smithi167 conmon[54076]: debug 2022-01-31T21:03:39.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:39.429060+0000) 2022-01-31T21:03:39.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:39 smithi167 conmon[60316]: debug 2022-01-31T21:03:39.178+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:39.180342+0000) 2022-01-31T21:03:39.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:39 smithi171 conmon[46715]: debug 2022-01-31T21:03:39.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:39.207279+0000) 2022-01-31T21:03:39.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:39 smithi167 conmon[49112]: debug 2022-01-31T21:03:39.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:39.501556+0000) 2022-01-31T21:03:39.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:39 smithi171 conmon[41853]: debug 2022-01-31T21:03:39.661+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:39.662239+0000) 2022-01-31T21:03:39.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:39 smithi171 conmon[51620]: debug 2022-01-31T21:03:39.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:39.535087+0000) 2022-01-31T21:03:40.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:40 smithi167 conmon[54076]: debug 2022-01-31T21:03:40.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.429252+0000) 2022-01-31T21:03:40.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:40 smithi167 conmon[60316]: debug 2022-01-31T21:03:40.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.180574+0000) 2022-01-31T21:03:40.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[46715]: debug 2022-01-31T21:03:40.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.207468+0000) 2022-01-31T21:03:40.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:40 smithi167 conmon[49112]: debug 2022-01-31T21:03:40.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.501719+0000) 2022-01-31T21:03:40.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:40 smithi167 conmon[49112]: debug 2022-01-31T21:03:40.698+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.699559+0000) 2022-01-31T21:03:40.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:40 smithi167 conmon[54076]: debug 2022-01-31T21:03:40.697+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.698734+0000) 2022-01-31T21:03:40.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:40 smithi167 conmon[60316]: debug 2022-01-31T21:03:40.698+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.699873+0000) 2022-01-31T21:03:40.786 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[35325]: debug 2022-01-31T21:03:40.708+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99865 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:40.787 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[46715]: debug 2022-01-31T21:03:40.698+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.699789+0000) 2022-01-31T21:03:40.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[41853]: debug 2022-01-31T21:03:40.661+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.662415+0000) 2022-01-31T21:03:40.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[41853]: debug 2022-01-31T21:03:40.697+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.699132+0000) 2022-01-31T21:03:40.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[51620]: debug 2022-01-31T21:03:40.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.535302+0000) 2022-01-31T21:03:40.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:40 smithi171 conmon[51620]: debug 2022-01-31T21:03:40.697+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:40.698937+0000) 2022-01-31T21:03:41.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:41 smithi167 conmon[54076]: debug 2022-01-31T21:03:41.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:41.429464+0000) 2022-01-31T21:03:41.619 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:41 smithi167 conmon[60316]: debug 2022-01-31T21:03:41.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:41.180708+0000) 2022-01-31T21:03:41.620 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:41 smithi171 conmon[46715]: debug 2022-01-31T21:03:41.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:41.207683+0000) 2022-01-31T21:03:41.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:41 smithi167 conmon[49112]: debug 2022-01-31T21:03:41.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:41.501898+0000) 2022-01-31T21:03:41.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:41 smithi171 conmon[41853]: debug 2022-01-31T21:03:41.661+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:41.662594+0000) 2022-01-31T21:03:41.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:41 smithi171 conmon[51620]: debug 2022-01-31T21:03:41.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:41.535487+0000) 2022-01-31T21:03:42.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:42 smithi167 conmon[54076]: debug 2022-01-31T21:03:42.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:42.429621+0000) 2022-01-31T21:03:42.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:42 smithi167 conmon[60316]: debug 2022-01-31T21:03:42.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:42.180820+0000) 2022-01-31T21:03:42.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:42 smithi171 conmon[46715]: debug 2022-01-31T21:03:42.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:42.207828+0000) 2022-01-31T21:03:42.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:42 smithi167 conmon[49112]: debug 2022-01-31T21:03:42.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:42.502023+0000) 2022-01-31T21:03:42.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:42 smithi171 conmon[41853]: debug 2022-01-31T21:03:42.662+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:42.662782+0000) 2022-01-31T21:03:42.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:42 smithi171 conmon[51620]: debug 2022-01-31T21:03:42.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:42.535634+0000) 2022-01-31T21:03:43.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:43 smithi167 conmon[54076]: debug 2022-01-31T21:03:43.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:43.429809+0000) 2022-01-31T21:03:43.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:43 smithi167 conmon[60316]: debug 2022-01-31T21:03:43.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:43.180960+0000) 2022-01-31T21:03:43.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:43 smithi171 conmon[46715]: debug 2022-01-31T21:03:43.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:43.208071+0000) 2022-01-31T21:03:43.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:43 smithi167 conmon[49112]: debug 2022-01-31T21:03:43.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:43.502181+0000) 2022-01-31T21:03:43.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:43 smithi171 conmon[41853]: debug 2022-01-31T21:03:43.662+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:43.662971+0000) 2022-01-31T21:03:43.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:43 smithi171 conmon[51620]: debug 2022-01-31T21:03:43.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:43.535801+0000) 2022-01-31T21:03:44.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:44 smithi167 conmon[54076]: debug 2022-01-31T21:03:44.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:44.430008+0000) 2022-01-31T21:03:44.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:44 smithi167 conmon[60316]: debug 2022-01-31T21:03:44.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:44.181192+0000) 2022-01-31T21:03:44.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:44 smithi171 conmon[46715]: debug 2022-01-31T21:03:44.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:44.208280+0000) 2022-01-31T21:03:44.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:44 smithi167 conmon[49112]: debug 2022-01-31T21:03:44.500+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:44.502404+0000) 2022-01-31T21:03:44.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:44 smithi171 conmon[41853]: debug 2022-01-31T21:03:44.662+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:44.663164+0000) 2022-01-31T21:03:44.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:44 smithi171 conmon[51620]: debug 2022-01-31T21:03:44.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:44.535953+0000) 2022-01-31T21:03:45.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:45 smithi167 conmon[54076]: debug 2022-01-31T21:03:45.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.430143+0000) 2022-01-31T21:03:45.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:45 smithi167 conmon[60316]: debug 2022-01-31T21:03:45.179+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.181397+0000) 2022-01-31T21:03:45.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[46715]: debug 2022-01-31T21:03:45.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.208439+0000) 2022-01-31T21:03:45.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:45 smithi167 conmon[49112]: debug 2022-01-31T21:03:45.501+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.502626+0000) 2022-01-31T21:03:45.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:45 smithi167 conmon[49112]: debug 2022-01-31T21:03:45.711+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.712992+0000) 2022-01-31T21:03:45.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:45 smithi167 conmon[54076]: debug 2022-01-31T21:03:45.711+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.712674+0000) 2022-01-31T21:03:45.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:45 smithi167 conmon[60316]: debug 2022-01-31T21:03:45.712+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.713983+0000) 2022-01-31T21:03:45.787 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[35325]: debug 2022-01-31T21:03:45.721+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 99976 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:45.788 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[46715]: debug 2022-01-31T21:03:45.711+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.712139+0000) 2022-01-31T21:03:45.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[41853]: debug 2022-01-31T21:03:45.662+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.663329+0000) 2022-01-31T21:03:45.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[41853]: debug 2022-01-31T21:03:45.712+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.713102+0000) 2022-01-31T21:03:45.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[51620]: debug 2022-01-31T21:03:45.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.536138+0000) 2022-01-31T21:03:45.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:45 smithi171 conmon[51620]: debug 2022-01-31T21:03:45.711+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:45.711820+0000) 2022-01-31T21:03:46.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:46 smithi167 conmon[54076]: debug 2022-01-31T21:03:46.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:46.430339+0000) 2022-01-31T21:03:46.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:46 smithi167 conmon[60316]: debug 2022-01-31T21:03:46.180+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:46.181595+0000) 2022-01-31T21:03:46.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:46 smithi171 conmon[46715]: debug 2022-01-31T21:03:46.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:46.208602+0000) 2022-01-31T21:03:46.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:46 smithi167 conmon[49112]: debug 2022-01-31T21:03:46.501+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:46.502797+0000) 2022-01-31T21:03:46.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:46 smithi171 conmon[41853]: debug 2022-01-31T21:03:46.662+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:46.663471+0000) 2022-01-31T21:03:46.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:46 smithi171 conmon[51620]: debug 2022-01-31T21:03:46.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:46.536318+0000) 2022-01-31T21:03:47.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:47 smithi167 conmon[54076]: debug 2022-01-31T21:03:47.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:47.430464+0000) 2022-01-31T21:03:47.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:47 smithi167 conmon[60316]: debug 2022-01-31T21:03:47.180+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:47.181709+0000) 2022-01-31T21:03:47.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:47 smithi171 conmon[46715]: debug 2022-01-31T21:03:47.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:47.208748+0000) 2022-01-31T21:03:47.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:47 smithi167 conmon[49112]: debug 2022-01-31T21:03:47.502+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:47.502957+0000) 2022-01-31T21:03:47.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:47 smithi171 conmon[41853]: debug 2022-01-31T21:03:47.662+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:47.663616+0000) 2022-01-31T21:03:47.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:47 smithi171 conmon[51620]: debug 2022-01-31T21:03:47.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:47.536478+0000) 2022-01-31T21:03:48.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:48 smithi167 conmon[54076]: debug 2022-01-31T21:03:48.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:48.430665+0000) 2022-01-31T21:03:48.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:48 smithi167 conmon[60316]: debug 2022-01-31T21:03:48.180+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:48.181831+0000) 2022-01-31T21:03:48.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:48 smithi171 conmon[46715]: debug 2022-01-31T21:03:48.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:48.208914+0000) 2022-01-31T21:03:48.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:48 smithi167 conmon[49112]: debug 2022-01-31T21:03:48.502+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:48.503105+0000) 2022-01-31T21:03:48.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:48 smithi171 conmon[41853]: debug 2022-01-31T21:03:48.663+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:48.663805+0000) 2022-01-31T21:03:48.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:48 smithi171 conmon[51620]: debug 2022-01-31T21:03:48.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:48.536693+0000) 2022-01-31T21:03:49.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:49 smithi167 conmon[54076]: debug 2022-01-31T21:03:49.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:49.430850+0000) 2022-01-31T21:03:49.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:49 smithi167 conmon[60316]: debug 2022-01-31T21:03:49.180+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:49.181998+0000) 2022-01-31T21:03:49.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:49 smithi171 conmon[46715]: debug 2022-01-31T21:03:49.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:49.209097+0000) 2022-01-31T21:03:49.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:49 smithi167 conmon[49112]: debug 2022-01-31T21:03:49.501+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:49.503282+0000) 2022-01-31T21:03:49.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:49 smithi171 conmon[41853]: debug 2022-01-31T21:03:49.663+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:49.663923+0000) 2022-01-31T21:03:49.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:49 smithi171 conmon[51620]: debug 2022-01-31T21:03:49.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:49.536859+0000) 2022-01-31T21:03:50.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:50 smithi167 conmon[54076]: debug 2022-01-31T21:03:50.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.431008+0000) 2022-01-31T21:03:50.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:50 smithi167 conmon[60316]: debug 2022-01-31T21:03:50.180+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.182219+0000) 2022-01-31T21:03:50.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[46715]: debug 2022-01-31T21:03:50.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.209296+0000) 2022-01-31T21:03:50.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:50 smithi167 conmon[49112]: debug 2022-01-31T21:03:50.501+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.503479+0000) 2022-01-31T21:03:50.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:50 smithi167 conmon[49112]: debug 2022-01-31T21:03:50.724+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.726091+0000) 2022-01-31T21:03:50.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:50 smithi167 conmon[54076]: debug 2022-01-31T21:03:50.723+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.724805+0000) 2022-01-31T21:03:50.729 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:50 smithi167 conmon[60316]: debug 2022-01-31T21:03:50.724+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.726465+0000) 2022-01-31T21:03:50.788 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[35325]: debug 2022-01-31T21:03:50.734+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100086 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:50.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[41853]: debug 2022-01-31T21:03:50.663+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.664111+0000) 2022-01-31T21:03:50.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[41853]: debug 2022-01-31T21:03:50.725+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.725911+0000) 2022-01-31T21:03:50.790 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[46715]: debug 2022-01-31T21:03:50.724+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.725095+0000) 2022-01-31T21:03:50.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[51620]: debug 2022-01-31T21:03:50.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.537051+0000) 2022-01-31T21:03:50.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:50 smithi171 conmon[51620]: debug 2022-01-31T21:03:50.723+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:50.724567+0000) 2022-01-31T21:03:51.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:51 smithi167 conmon[60316]: debug 2022-01-31T21:03:51.181+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:51.182433+0000) 2022-01-31T21:03:51.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:51 smithi171 conmon[46715]: debug 2022-01-31T21:03:51.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:51.209446+0000) 2022-01-31T21:03:51.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:51 smithi167 conmon[49112]: debug 2022-01-31T21:03:51.502+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:51.503719+0000) 2022-01-31T21:03:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:51 smithi167 conmon[54076]: debug 2022-01-31T21:03:51.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:51.431164+0000) 2022-01-31T21:03:51.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:51 smithi171 conmon[41853]: debug 2022-01-31T21:03:51.663+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:51.664294+0000) 2022-01-31T21:03:51.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:51 smithi171 conmon[51620]: debug 2022-01-31T21:03:51.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:51.537205+0000) 2022-01-31T21:03:52.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:52 smithi167 conmon[54076]: debug 2022-01-31T21:03:52.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:52.431310+0000) 2022-01-31T21:03:52.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:52 smithi167 conmon[60316]: debug 2022-01-31T21:03:52.181+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:52.182595+0000) 2022-01-31T21:03:52.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:52 smithi171 conmon[46715]: debug 2022-01-31T21:03:52.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:52.209597+0000) 2022-01-31T21:03:52.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:52 smithi167 conmon[49112]: debug 2022-01-31T21:03:52.503+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:52.503869+0000) 2022-01-31T21:03:52.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:52 smithi171 conmon[41853]: debug 2022-01-31T21:03:52.663+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:52.664470+0000) 2022-01-31T21:03:52.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:52 smithi171 conmon[51620]: debug 2022-01-31T21:03:52.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:52.537371+0000) 2022-01-31T21:03:53.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:53 smithi167 conmon[54076]: debug 2022-01-31T21:03:53.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:53.431470+0000) 2022-01-31T21:03:53.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:53 smithi167 conmon[60316]: debug 2022-01-31T21:03:53.181+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:53.182719+0000) 2022-01-31T21:03:53.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:53 smithi171 conmon[46715]: debug 2022-01-31T21:03:53.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:53.209829+0000) 2022-01-31T21:03:53.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:53 smithi167 conmon[49112]: debug 2022-01-31T21:03:53.502+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:53.504050+0000) 2022-01-31T21:03:53.789 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:53 smithi171 conmon[35325]: debug 2022-01-31T21:03:53.617+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:03:53.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:53 smithi171 conmon[51620]: debug 2022-01-31T21:03:53.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:53.537558+0000) 2022-01-31T21:03:53.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:53 smithi171 conmon[41853]: debug 2022-01-31T21:03:53.664+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:53.664654+0000) 2022-01-31T21:03:54.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:54 smithi167 conmon[54076]: debug 2022-01-31T21:03:54.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:54.431679+0000) 2022-01-31T21:03:54.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:54 smithi167 conmon[60316]: debug 2022-01-31T21:03:54.181+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:54.182886+0000) 2022-01-31T21:03:54.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:54 smithi171 conmon[46715]: debug 2022-01-31T21:03:54.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:54.210006+0000) 2022-01-31T21:03:54.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:54 smithi167 conmon[49112]: debug 2022-01-31T21:03:54.503+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:54.504170+0000) 2022-01-31T21:03:54.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:54 smithi171 conmon[51620]: debug 2022-01-31T21:03:54.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:54.537736+0000) 2022-01-31T21:03:54.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:54 smithi171 conmon[41853]: debug 2022-01-31T21:03:54.664+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:54.664834+0000) 2022-01-31T21:03:55.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:55 smithi167 conmon[54076]: debug 2022-01-31T21:03:55.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.431870+0000) 2022-01-31T21:03:55.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:55 smithi167 conmon[60316]: debug 2022-01-31T21:03:55.181+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.183038+0000) 2022-01-31T21:03:55.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[46715]: debug 2022-01-31T21:03:55.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.210180+0000) 2022-01-31T21:03:55.735 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:55 smithi167 conmon[49112]: debug 2022-01-31T21:03:55.502+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.504355+0000) 2022-01-31T21:03:55.789 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[35325]: debug 2022-01-31T21:03:55.746+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100197 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:03:55.790 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[46715]: debug 2022-01-31T21:03:55.737+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.738165+0000) 2022-01-31T21:03:55.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[41853]: debug 2022-01-31T21:03:55.664+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.665005+0000) 2022-01-31T21:03:55.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[41853]: debug 2022-01-31T21:03:55.736+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.737707+0000) 2022-01-31T21:03:55.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[51620]: debug 2022-01-31T21:03:55.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.537935+0000) 2022-01-31T21:03:55.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:55 smithi171 conmon[51620]: debug 2022-01-31T21:03:55.736+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.737097+0000) 2022-01-31T21:03:55.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:55 smithi167 conmon[49112]: debug 2022-01-31T21:03:55.737+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.739039+0000) 2022-01-31T21:03:55.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:55 smithi167 conmon[54076]: debug 2022-01-31T21:03:55.736+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.738387+0000) 2022-01-31T21:03:55.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:55 smithi167 conmon[60316]: debug 2022-01-31T21:03:55.737+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:55.738869+0000) 2022-01-31T21:03:56.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:56 smithi167 conmon[60316]: debug 2022-01-31T21:03:56.181+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:56.183212+0000) 2022-01-31T21:03:56.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:56 smithi171 conmon[46715]: debug 2022-01-31T21:03:56.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:56.210398+0000) 2022-01-31T21:03:56.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:56 smithi167 conmon[49112]: debug 2022-01-31T21:03:56.503+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:56.504487+0000) 2022-01-31T21:03:56.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:56 smithi167 conmon[54076]: debug 2022-01-31T21:03:56.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:56.432062+0000) 2022-01-31T21:03:56.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:56 smithi171 conmon[51620]: debug 2022-01-31T21:03:56.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:56.538133+0000) 2022-01-31T21:03:56.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:56 smithi171 conmon[41853]: debug 2022-01-31T21:03:56.664+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:56.665181+0000) 2022-01-31T21:03:57.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:57 smithi167 conmon[60316]: debug 2022-01-31T21:03:57.182+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:57.183357+0000) 2022-01-31T21:03:57.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:57 smithi171 conmon[46715]: debug 2022-01-31T21:03:57.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:57.210568+0000) 2022-01-31T21:03:57.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:57 smithi167 conmon[49112]: debug 2022-01-31T21:03:57.504+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:57.504653+0000) 2022-01-31T21:03:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:57 smithi167 conmon[54076]: debug 2022-01-31T21:03:57.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:57.432191+0000) 2022-01-31T21:03:57.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:57 smithi171 conmon[51620]: debug 2022-01-31T21:03:57.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:57.538294+0000) 2022-01-31T21:03:57.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:57 smithi171 conmon[41853]: debug 2022-01-31T21:03:57.664+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:57.665276+0000) 2022-01-31T21:03:58.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:58 smithi167 conmon[60316]: debug 2022-01-31T21:03:58.182+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:58.183532+0000) 2022-01-31T21:03:58.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:58 smithi171 conmon[46715]: debug 2022-01-31T21:03:58.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:58.210786+0000) 2022-01-31T21:03:58.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:58 smithi167 conmon[49112]: debug 2022-01-31T21:03:58.503+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:58.504791+0000) 2022-01-31T21:03:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:58 smithi167 conmon[54076]: debug 2022-01-31T21:03:58.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:58.432395+0000) 2022-01-31T21:03:58.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:58 smithi171 conmon[41853]: debug 2022-01-31T21:03:58.665+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:58.665443+0000) 2022-01-31T21:03:58.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:58 smithi171 conmon[51620]: debug 2022-01-31T21:03:58.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:58.538505+0000) 2022-01-31T21:03:59.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:03:59 smithi167 conmon[60316]: debug 2022-01-31T21:03:59.183+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:59.183716+0000) 2022-01-31T21:03:59.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:03:59 smithi171 conmon[46715]: debug 2022-01-31T21:03:59.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:59.210943+0000) 2022-01-31T21:03:59.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:03:59 smithi167 conmon[49112]: debug 2022-01-31T21:03:59.504+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:59.504985+0000) 2022-01-31T21:03:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:03:59 smithi167 conmon[54076]: debug 2022-01-31T21:03:59.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:59.432582+0000) 2022-01-31T21:03:59.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:03:59 smithi171 conmon[41853]: debug 2022-01-31T21:03:59.665+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:59.665593+0000) 2022-01-31T21:03:59.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:03:59 smithi171 conmon[51620]: debug 2022-01-31T21:03:59.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:03:59.538687+0000) 2022-01-31T21:04:00.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:00 smithi167 conmon[60316]: debug 2022-01-31T21:04:00.183+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.183916+0000) 2022-01-31T21:04:00.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[46715]: debug 2022-01-31T21:04:00.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.211117+0000) 2022-01-31T21:04:00.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:00 smithi167 conmon[49112]: debug 2022-01-31T21:04:00.504+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.505185+0000) 2022-01-31T21:04:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:00 smithi167 conmon[54076]: debug 2022-01-31T21:04:00.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.432748+0000) 2022-01-31T21:04:00.790 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[35325]: debug 2022-01-31T21:04:00.759+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100307 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:00.791 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[46715]: debug 2022-01-31T21:04:00.749+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.750251+0000) 2022-01-31T21:04:00.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[51620]: debug 2022-01-31T21:04:00.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.538908+0000) 2022-01-31T21:04:00.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[51620]: debug 2022-01-31T21:04:00.748+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.749514+0000) 2022-01-31T21:04:00.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[41853]: debug 2022-01-31T21:04:00.665+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.665787+0000) 2022-01-31T21:04:00.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:00 smithi171 conmon[41853]: debug 2022-01-31T21:04:00.749+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.750430+0000) 2022-01-31T21:04:01.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:00 smithi167 conmon[49112]: debug 2022-01-31T21:04:00.749+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.750821+0000) 2022-01-31T21:04:01.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:00 smithi167 conmon[54076]: debug 2022-01-31T21:04:00.749+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.750952+0000) 2022-01-31T21:04:01.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:00 smithi167 conmon[60316]: debug 2022-01-31T21:04:00.748+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:00.750450+0000) 2022-01-31T21:04:01.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:01 smithi167 conmon[60316]: debug 2022-01-31T21:04:01.183+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:01.184122+0000) 2022-01-31T21:04:01.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:01 smithi171 conmon[46715]: debug 2022-01-31T21:04:01.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:01.211298+0000) 2022-01-31T21:04:01.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:01 smithi167 conmon[49112]: debug 2022-01-31T21:04:01.504+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:01.505371+0000) 2022-01-31T21:04:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:01 smithi167 conmon[54076]: debug 2022-01-31T21:04:01.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:01.432884+0000) 2022-01-31T21:04:01.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:01 smithi171 conmon[41853]: debug 2022-01-31T21:04:01.665+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:01.665994+0000) 2022-01-31T21:04:01.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:01 smithi171 conmon[51620]: debug 2022-01-31T21:04:01.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:01.539071+0000) 2022-01-31T21:04:02.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:02 smithi167 conmon[60316]: debug 2022-01-31T21:04:02.183+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:02.184252+0000) 2022-01-31T21:04:02.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:02 smithi171 conmon[46715]: debug 2022-01-31T21:04:02.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:02.211417+0000) 2022-01-31T21:04:02.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:02 smithi167 conmon[54076]: debug 2022-01-31T21:04:02.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:02.432998+0000) 2022-01-31T21:04:02.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:02 smithi167 conmon[49112]: debug 2022-01-31T21:04:02.505+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:02.505539+0000) 2022-01-31T21:04:02.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:02 smithi171 conmon[41853]: debug 2022-01-31T21:04:02.665+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:02.666165+0000) 2022-01-31T21:04:02.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:02 smithi171 conmon[51620]: debug 2022-01-31T21:04:02.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:02.539223+0000) 2022-01-31T21:04:03.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:03 smithi167 conmon[60316]: debug 2022-01-31T21:04:03.183+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:03.184408+0000) 2022-01-31T21:04:03.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:03 smithi171 conmon[46715]: debug 2022-01-31T21:04:03.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:03.211586+0000) 2022-01-31T21:04:03.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:03 smithi167 conmon[54076]: debug 2022-01-31T21:04:03.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:03.433176+0000) 2022-01-31T21:04:03.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:03 smithi167 conmon[49112]: debug 2022-01-31T21:04:03.505+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:03.505713+0000) 2022-01-31T21:04:03.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:03 smithi171 conmon[51620]: debug 2022-01-31T21:04:03.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:03.539420+0000) 2022-01-31T21:04:03.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:03 smithi171 conmon[41853]: debug 2022-01-31T21:04:03.666+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:03.666396+0000) 2022-01-31T21:04:04.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:04 smithi167 conmon[60316]: debug 2022-01-31T21:04:04.184+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:04.184555+0000) 2022-01-31T21:04:04.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:04 smithi171 conmon[46715]: debug 2022-01-31T21:04:04.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:04.211786+0000) 2022-01-31T21:04:04.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:04 smithi167 conmon[49112]: debug 2022-01-31T21:04:04.505+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:04.505917+0000) 2022-01-31T21:04:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:04 smithi167 conmon[54076]: debug 2022-01-31T21:04:04.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:04.433363+0000) 2022-01-31T21:04:04.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:04 smithi171 conmon[51620]: debug 2022-01-31T21:04:04.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:04.539626+0000) 2022-01-31T21:04:04.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:04 smithi171 conmon[41853]: debug 2022-01-31T21:04:04.665+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:04.666599+0000) 2022-01-31T21:04:05.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:05 smithi167 conmon[60316]: debug 2022-01-31T21:04:05.184+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.184707+0000) 2022-01-31T21:04:05.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[46715]: debug 2022-01-31T21:04:05.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.211987+0000) 2022-01-31T21:04:05.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:05 smithi167 conmon[49112]: debug 2022-01-31T21:04:05.505+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.506067+0000) 2022-01-31T21:04:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:05 smithi167 conmon[54076]: debug 2022-01-31T21:04:05.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.433556+0000) 2022-01-31T21:04:05.791 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[35325]: debug 2022-01-31T21:04:05.771+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100417 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:05.792 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[46715]: debug 2022-01-31T21:04:05.761+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.762563+0000) 2022-01-31T21:04:05.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[41853]: debug 2022-01-31T21:04:05.666+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.666780+0000) 2022-01-31T21:04:05.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[41853]: debug 2022-01-31T21:04:05.762+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.763321+0000) 2022-01-31T21:04:05.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[51620]: debug 2022-01-31T21:04:05.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.539822+0000) 2022-01-31T21:04:05.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:05 smithi171 conmon[51620]: debug 2022-01-31T21:04:05.761+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.762087+0000) 2022-01-31T21:04:06.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:05 smithi167 conmon[49112]: debug 2022-01-31T21:04:05.763+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.763580+0000) 2022-01-31T21:04:06.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:05 smithi167 conmon[54076]: debug 2022-01-31T21:04:05.762+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.763040+0000) 2022-01-31T21:04:06.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:05 smithi167 conmon[60316]: debug 2022-01-31T21:04:05.761+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:05.762371+0000) 2022-01-31T21:04:06.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:06 smithi167 conmon[60316]: debug 2022-01-31T21:04:06.184+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:06.184892+0000) 2022-01-31T21:04:06.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:06 smithi171 conmon[46715]: debug 2022-01-31T21:04:06.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:06.212194+0000) 2022-01-31T21:04:06.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:06 smithi167 conmon[49112]: debug 2022-01-31T21:04:06.505+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:06.506267+0000) 2022-01-31T21:04:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:06 smithi167 conmon[54076]: debug 2022-01-31T21:04:06.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:06.433715+0000) 2022-01-31T21:04:06.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:06 smithi171 conmon[51620]: debug 2022-01-31T21:04:06.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:06.539949+0000) 2022-01-31T21:04:06.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:06 smithi171 conmon[41853]: debug 2022-01-31T21:04:06.666+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:06.666955+0000) 2022-01-31T21:04:07.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:07 smithi167 conmon[60316]: debug 2022-01-31T21:04:07.184+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:07.185044+0000) 2022-01-31T21:04:07.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:07 smithi171 conmon[46715]: debug 2022-01-31T21:04:07.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:07.212341+0000) 2022-01-31T21:04:07.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:07 smithi167 conmon[49112]: debug 2022-01-31T21:04:07.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:07.506412+0000) 2022-01-31T21:04:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:07 smithi167 conmon[54076]: debug 2022-01-31T21:04:07.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:07.433870+0000) 2022-01-31T21:04:07.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:07 smithi171 conmon[41853]: debug 2022-01-31T21:04:07.666+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:07.667072+0000) 2022-01-31T21:04:07.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:07 smithi171 conmon[51620]: debug 2022-01-31T21:04:07.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:07.540112+0000) 2022-01-31T21:04:08.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:08 smithi167 conmon[60316]: debug 2022-01-31T21:04:08.184+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:08.185249+0000) 2022-01-31T21:04:08.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:08 smithi171 conmon[46715]: debug 2022-01-31T21:04:08.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:08.212498+0000) 2022-01-31T21:04:08.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:08 smithi167 conmon[49112]: debug 2022-01-31T21:04:08.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:08.506540+0000) 2022-01-31T21:04:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:08 smithi167 conmon[54076]: debug 2022-01-31T21:04:08.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:08.434090+0000) 2022-01-31T21:04:08.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:08 smithi171 conmon[51620]: debug 2022-01-31T21:04:08.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:08.540338+0000) 2022-01-31T21:04:08.792 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:08 smithi171 conmon[35325]: debug 2022-01-31T21:04:08.618+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:04:08.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:08 smithi171 conmon[41853]: debug 2022-01-31T21:04:08.666+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:08.667197+0000) 2022-01-31T21:04:09.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:09 smithi167 conmon[60316]: debug 2022-01-31T21:04:09.185+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:09.185434+0000) 2022-01-31T21:04:09.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:09 smithi171 conmon[46715]: debug 2022-01-31T21:04:09.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:09.212705+0000) 2022-01-31T21:04:09.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:09 smithi167 conmon[49112]: debug 2022-01-31T21:04:09.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:09.506691+0000) 2022-01-31T21:04:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:09 smithi167 conmon[54076]: debug 2022-01-31T21:04:09.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:09.434267+0000) 2022-01-31T21:04:09.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:09 smithi171 conmon[41853]: debug 2022-01-31T21:04:09.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:09.667353+0000) 2022-01-31T21:04:09.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:09 smithi171 conmon[51620]: debug 2022-01-31T21:04:09.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:09.540549+0000) 2022-01-31T21:04:10.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:10 smithi167 conmon[60316]: debug 2022-01-31T21:04:10.185+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.185684+0000) 2022-01-31T21:04:10.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[46715]: debug 2022-01-31T21:04:10.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.212920+0000) 2022-01-31T21:04:10.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:10 smithi167 conmon[49112]: debug 2022-01-31T21:04:10.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.506880+0000) 2022-01-31T21:04:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:10 smithi167 conmon[54076]: debug 2022-01-31T21:04:10.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.434478+0000) 2022-01-31T21:04:10.793 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[35325]: debug 2022-01-31T21:04:10.784+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100523 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:10.793 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[46715]: debug 2022-01-31T21:04:10.773+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.774493+0000) 2022-01-31T21:04:10.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[41853]: debug 2022-01-31T21:04:10.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.667567+0000) 2022-01-31T21:04:10.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[41853]: debug 2022-01-31T21:04:10.774+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.775560+0000) 2022-01-31T21:04:10.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[51620]: debug 2022-01-31T21:04:10.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.540766+0000) 2022-01-31T21:04:10.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:10 smithi171 conmon[51620]: debug 2022-01-31T21:04:10.774+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.775586+0000) 2022-01-31T21:04:11.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:10 smithi167 conmon[49112]: debug 2022-01-31T21:04:10.775+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.775635+0000) 2022-01-31T21:04:11.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:10 smithi167 conmon[54076]: debug 2022-01-31T21:04:10.775+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.776093+0000) 2022-01-31T21:04:11.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:10 smithi167 conmon[60316]: debug 2022-01-31T21:04:10.774+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:10.774852+0000) 2022-01-31T21:04:11.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:11 smithi167 conmon[60316]: debug 2022-01-31T21:04:11.185+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:11.185874+0000) 2022-01-31T21:04:11.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:11 smithi171 conmon[46715]: debug 2022-01-31T21:04:11.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:11.213112+0000) 2022-01-31T21:04:11.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:11 smithi167 conmon[54076]: debug 2022-01-31T21:04:11.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:11.434710+0000) 2022-01-31T21:04:11.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:11 smithi167 conmon[49112]: debug 2022-01-31T21:04:11.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:11.507072+0000) 2022-01-31T21:04:11.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:11 smithi171 conmon[41853]: debug 2022-01-31T21:04:11.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:11.667769+0000) 2022-01-31T21:04:11.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:11 smithi171 conmon[51620]: debug 2022-01-31T21:04:11.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:11.540917+0000) 2022-01-31T21:04:12.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:12 smithi167 conmon[60316]: debug 2022-01-31T21:04:12.185+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:12.186029+0000) 2022-01-31T21:04:12.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:12 smithi171 conmon[46715]: debug 2022-01-31T21:04:12.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:12.213267+0000) 2022-01-31T21:04:12.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:12 smithi167 conmon[49112]: debug 2022-01-31T21:04:12.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:12.507238+0000) 2022-01-31T21:04:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:12 smithi167 conmon[54076]: debug 2022-01-31T21:04:12.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:12.434873+0000) 2022-01-31T21:04:12.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:12 smithi171 conmon[41853]: debug 2022-01-31T21:04:12.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:12.667960+0000) 2022-01-31T21:04:12.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:12 smithi171 conmon[51620]: debug 2022-01-31T21:04:12.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:12.541061+0000) 2022-01-31T21:04:13.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:13 smithi167 conmon[60316]: debug 2022-01-31T21:04:13.185+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:13.186213+0000) 2022-01-31T21:04:13.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:13 smithi171 conmon[46715]: debug 2022-01-31T21:04:13.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:13.213525+0000) 2022-01-31T21:04:13.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:13 smithi167 conmon[49112]: debug 2022-01-31T21:04:13.506+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:13.507417+0000) 2022-01-31T21:04:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:13 smithi167 conmon[54076]: debug 2022-01-31T21:04:13.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:13.435036+0000) 2022-01-31T21:04:13.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:13 smithi171 conmon[41853]: debug 2022-01-31T21:04:13.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:13.668174+0000) 2022-01-31T21:04:13.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:13 smithi171 conmon[51620]: debug 2022-01-31T21:04:13.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:13.541251+0000) 2022-01-31T21:04:14.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:14 smithi167 conmon[60316]: debug 2022-01-31T21:04:14.185+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:14.186342+0000) 2022-01-31T21:04:14.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:14 smithi171 conmon[46715]: debug 2022-01-31T21:04:14.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:14.213685+0000) 2022-01-31T21:04:14.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:14 smithi167 conmon[49112]: debug 2022-01-31T21:04:14.507+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:14.507658+0000) 2022-01-31T21:04:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:14 smithi167 conmon[54076]: debug 2022-01-31T21:04:14.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:14.435199+0000) 2022-01-31T21:04:14.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:14 smithi171 conmon[41853]: debug 2022-01-31T21:04:14.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:14.668325+0000) 2022-01-31T21:04:14.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:14 smithi171 conmon[51620]: debug 2022-01-31T21:04:14.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:14.541403+0000) 2022-01-31T21:04:15.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:15 smithi167 conmon[60316]: debug 2022-01-31T21:04:15.186+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.186536+0000) 2022-01-31T21:04:15.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[46715]: debug 2022-01-31T21:04:15.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.213867+0000) 2022-01-31T21:04:15.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:15 smithi167 conmon[49112]: debug 2022-01-31T21:04:15.507+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.507817+0000) 2022-01-31T21:04:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:15 smithi167 conmon[54076]: debug 2022-01-31T21:04:15.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.435384+0000) 2022-01-31T21:04:15.793 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[46715]: debug 2022-01-31T21:04:15.787+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.788232+0000) 2022-01-31T21:04:15.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[41853]: debug 2022-01-31T21:04:15.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.668487+0000) 2022-01-31T21:04:15.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[41853]: debug 2022-01-31T21:04:15.787+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.788088+0000) 2022-01-31T21:04:15.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[51620]: debug 2022-01-31T21:04:15.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.541554+0000) 2022-01-31T21:04:15.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[51620]: debug 2022-01-31T21:04:15.787+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.788417+0000) 2022-01-31T21:04:16.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:15 smithi167 conmon[49112]: debug 2022-01-31T21:04:15.787+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.787753+0000) 2022-01-31T21:04:16.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:15 smithi167 conmon[54076]: debug 2022-01-31T21:04:15.787+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.788483+0000) 2022-01-31T21:04:16.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:15 smithi167 conmon[60316]: debug 2022-01-31T21:04:15.788+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:15.789385+0000) 2022-01-31T21:04:16.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:15 smithi171 conmon[35325]: debug 2022-01-31T21:04:15.798+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100634 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:16.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:16 smithi167 conmon[60316]: debug 2022-01-31T21:04:16.186+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:16.186734+0000) 2022-01-31T21:04:16.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:16 smithi171 conmon[46715]: debug 2022-01-31T21:04:16.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:16.214023+0000) 2022-01-31T21:04:16.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:16 smithi167 conmon[49112]: debug 2022-01-31T21:04:16.507+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:16.507972+0000) 2022-01-31T21:04:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:16 smithi167 conmon[54076]: debug 2022-01-31T21:04:16.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:16.435581+0000) 2022-01-31T21:04:16.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:16 smithi171 conmon[51620]: debug 2022-01-31T21:04:16.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:16.541715+0000) 2022-01-31T21:04:16.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:16 smithi171 conmon[41853]: debug 2022-01-31T21:04:16.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:16 smithi171 conmon[41853]: 2022-01-31T21:04:16.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:16.668685+0000) 2022-01-31T21:04:17.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:17 smithi167 conmon[60316]: debug 2022-01-31T21:04:17.186+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:17.186867+0000) 2022-01-31T21:04:17.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:17 smithi171 conmon[46715]: debug 2022-01-31T21:04:17.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:17.214169+0000) 2022-01-31T21:04:17.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:17 smithi167 conmon[49112]: debug 2022-01-31T21:04:17.507+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:17.508132+0000) 2022-01-31T21:04:17.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:17 smithi167 conmon[54076]: debug 2022-01-31T21:04:17.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:17.435697+0000) 2022-01-31T21:04:17.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:17 smithi171 conmon[41853]: debug 2022-01-31T21:04:17.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:17.668832+0000) 2022-01-31T21:04:17.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:17 smithi171 conmon[51620]: debug 2022-01-31T21:04:17.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:17.541856+0000) 2022-01-31T21:04:18.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:18 smithi167 conmon[60316]: debug 2022-01-31T21:04:18.186+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:18.187063+0000) 2022-01-31T21:04:18.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:18 smithi171 conmon[46715]: debug 2022-01-31T21:04:18.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:18.214356+0000) 2022-01-31T21:04:18.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:18 smithi167 conmon[49112]: debug 2022-01-31T21:04:18.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:18.508311+0000) 2022-01-31T21:04:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:18 smithi167 conmon[54076]: debug 2022-01-31T21:04:18.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:18.435909+0000) 2022-01-31T21:04:18.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:18 smithi171 conmon[41853]: debug 2022-01-31T21:04:18.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:18.669047+0000) 2022-01-31T21:04:18.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:18 smithi171 conmon[51620]: debug 2022-01-31T21:04:18.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:18.542050+0000) 2022-01-31T21:04:19.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:19 smithi167 conmon[60316]: debug 2022-01-31T21:04:19.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:19.187247+0000) 2022-01-31T21:04:19.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:19 smithi171 conmon[46715]: debug 2022-01-31T21:04:19.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:19.214573+0000) 2022-01-31T21:04:19.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:19 smithi167 conmon[49112]: debug 2022-01-31T21:04:19.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:19.508465+0000) 2022-01-31T21:04:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:19 smithi167 conmon[54076]: debug 2022-01-31T21:04:19.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:19.436102+0000) 2022-01-31T21:04:19.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:19 smithi171 conmon[41853]: debug 2022-01-31T21:04:19.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:19.669233+0000) 2022-01-31T21:04:19.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:19 smithi171 conmon[51620]: debug 2022-01-31T21:04:19.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:19.542223+0000) 2022-01-31T21:04:20.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:20 smithi167 conmon[60316]: debug 2022-01-31T21:04:20.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.187425+0000) 2022-01-31T21:04:20.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[46715]: debug 2022-01-31T21:04:20.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.214758+0000) 2022-01-31T21:04:20.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:20 smithi167 conmon[49112]: debug 2022-01-31T21:04:20.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.508647+0000) 2022-01-31T21:04:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:20 smithi167 conmon[54076]: debug 2022-01-31T21:04:20.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.436291+0000) 2022-01-31T21:04:20.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[41853]: debug 2022-01-31T21:04:20.667+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.669408+0000) 2022-01-31T21:04:20.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[51620]: debug 2022-01-31T21:04:20.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.542365+0000) 2022-01-31T21:04:21.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:20 smithi167 conmon[49112]: debug 2022-01-31T21:04:20.800+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.801224+0000) 2022-01-31T21:04:21.027 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:20 smithi167 conmon[54076]: debug 2022-01-31T21:04:20.801+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.801920+0000) 2022-01-31T21:04:21.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:20 smithi167 conmon[60316]: debug 2022-01-31T21:04:20.801+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.802233+0000) 2022-01-31T21:04:21.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[35325]: debug 2022-01-31T21:04:20.809+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100750 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[41853]: debug 2022-01-31T21:04:20.799+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.801641+0000) 2022-01-31T21:04:21.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[46715]: debug 2022-01-31T21:04:20.800+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.802082+0000) 2022-01-31T21:04:21.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:20 smithi171 conmon[51620]: debug 2022-01-31T21:04:20.799+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:20.801305+0000) 2022-01-31T21:04:21.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:21 smithi167 conmon[60316]: debug 2022-01-31T21:04:21.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:21.187588+0000) 2022-01-31T21:04:21.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:21 smithi171 conmon[46715]: debug 2022-01-31T21:04:21.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:21.214922+0000) 2022-01-31T21:04:21.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:21 smithi167 conmon[49112]: debug 2022-01-31T21:04:21.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:21.508809+0000) 2022-01-31T21:04:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:21 smithi167 conmon[54076]: debug 2022-01-31T21:04:21.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:21.436446+0000) 2022-01-31T21:04:21.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:21 smithi171 conmon[41853]: debug 2022-01-31T21:04:21.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:21.669602+0000) 2022-01-31T21:04:21.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:21 smithi171 conmon[51620]: debug 2022-01-31T21:04:21.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:21.542574+0000) 2022-01-31T21:04:22.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:22 smithi167 conmon[60316]: debug 2022-01-31T21:04:22.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:22.187721+0000) 2022-01-31T21:04:22.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:22 smithi171 conmon[46715]: debug 2022-01-31T21:04:22.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:22.215071+0000) 2022-01-31T21:04:22.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:22 smithi167 conmon[49112]: debug 2022-01-31T21:04:22.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:22.508937+0000) 2022-01-31T21:04:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:22 smithi167 conmon[54076]: debug 2022-01-31T21:04:22.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:22.436602+0000) 2022-01-31T21:04:22.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:22 smithi171 conmon[51620]: debug 2022-01-31T21:04:22.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:22.542772+0000) 2022-01-31T21:04:22.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:22 smithi171 conmon[41853]: debug 2022-01-31T21:04:22.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:22.669760+0000) 2022-01-31T21:04:23.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:23 smithi167 conmon[60316]: debug 2022-01-31T21:04:23.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:23.187904+0000) 2022-01-31T21:04:23.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:23 smithi171 conmon[46715]: debug 2022-01-31T21:04:23.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:23.215273+0000) 2022-01-31T21:04:23.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:23 smithi167 conmon[49112]: debug 2022-01-31T21:04:23.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:23.509067+0000) 2022-01-31T21:04:23.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:23 smithi167 conmon[54076]: debug 2022-01-31T21:04:23.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:23.436807+0000) 2022-01-31T21:04:23.794 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:23 smithi171 conmon[35325]: debug 2022-01-31T21:04:23.618+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:04:23.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:23 smithi171 conmon[51620]: debug 2022-01-31T21:04:23.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:23.542987+0000) 2022-01-31T21:04:23.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:23 smithi171 conmon[41853]: debug 2022-01-31T21:04:23.669+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:23.669907+0000) 2022-01-31T21:04:24.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:24 smithi167 conmon[60316]: debug 2022-01-31T21:04:24.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:24.188125+0000) 2022-01-31T21:04:24.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:24 smithi171 conmon[46715]: debug 2022-01-31T21:04:24.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:24.215411+0000) 2022-01-31T21:04:24.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:24 smithi167 conmon[49112]: debug 2022-01-31T21:04:24.509+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:24.509249+0000) 2022-01-31T21:04:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:24 smithi167 conmon[54076]: debug 2022-01-31T21:04:24.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:24.437004+0000) 2022-01-31T21:04:24.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:24 smithi171 conmon[41853]: debug 2022-01-31T21:04:24.668+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:24.670023+0000) 2022-01-31T21:04:24.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:24 smithi171 conmon[51620]: debug 2022-01-31T21:04:24.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:24.543189+0000) 2022-01-31T21:04:25.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:25 smithi167 conmon[60316]: debug 2022-01-31T21:04:25.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.188327+0000) 2022-01-31T21:04:25.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[46715]: debug 2022-01-31T21:04:25.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.215566+0000) 2022-01-31T21:04:25.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:25 smithi167 conmon[49112]: debug 2022-01-31T21:04:25.507+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.509409+0000) 2022-01-31T21:04:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:25 smithi167 conmon[54076]: debug 2022-01-31T21:04:25.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.437195+0000) 2022-01-31T21:04:25.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[41853]: debug 2022-01-31T21:04:25.669+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.670207+0000) 2022-01-31T21:04:25.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[51620]: debug 2022-01-31T21:04:25.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.543363+0000) 2022-01-31T21:04:26.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:25 smithi167 conmon[49112]: debug 2022-01-31T21:04:25.814+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.815536+0000) 2022-01-31T21:04:26.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:25 smithi167 conmon[60316]: debug 2022-01-31T21:04:25.813+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.814748+0000) 2022-01-31T21:04:26.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:25 smithi167 conmon[54076]: debug 2022-01-31T21:04:25.812+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.814350+0000) 2022-01-31T21:04:26.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[35325]: debug 2022-01-31T21:04:25.823+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100862 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[41853]: debug 2022-01-31T21:04:25.812+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.814434+0000) 2022-01-31T21:04:26.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[46715]: debug 2022-01-31T21:04:25.813+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.814767+0000) 2022-01-31T21:04:26.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:25 smithi171 conmon[51620]: debug 2022-01-31T21:04:25.812+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:25.814137+0000) 2022-01-31T21:04:26.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:26 smithi167 conmon[60316]: debug 2022-01-31T21:04:26.186+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:26.188453+0000) 2022-01-31T21:04:26.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:26 smithi171 conmon[46715]: debug 2022-01-31T21:04:26.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:26.215786+0000) 2022-01-31T21:04:26.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:26 smithi167 conmon[49112]: debug 2022-01-31T21:04:26.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:26.509594+0000) 2022-01-31T21:04:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:26 smithi167 conmon[54076]: debug 2022-01-31T21:04:26.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:26.437397+0000) 2022-01-31T21:04:26.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:26 smithi171 conmon[51620]: debug 2022-01-31T21:04:26.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:26.543539+0000) 2022-01-31T21:04:26.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:26 smithi171 conmon[41853]: debug 2022-01-31T21:04:26.669+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:26.670393+0000) 2022-01-31T21:04:27.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:27 smithi167 conmon[60316]: debug 2022-01-31T21:04:27.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:27.188573+0000) 2022-01-31T21:04:27.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:27 smithi171 conmon[46715]: debug 2022-01-31T21:04:27.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:27.215936+0000) 2022-01-31T21:04:27.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:27 smithi167 conmon[49112]: debug 2022-01-31T21:04:27.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:27.509749+0000) 2022-01-31T21:04:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:27 smithi167 conmon[54076]: debug 2022-01-31T21:04:27.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:27.437516+0000) 2022-01-31T21:04:27.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:27 smithi171 conmon[41853]: debug 2022-01-31T21:04:27.669+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:27.670539+0000) 2022-01-31T21:04:27.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:27 smithi171 conmon[51620]: debug 2022-01-31T21:04:27.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:27.543687+0000) 2022-01-31T21:04:28.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:28 smithi167 conmon[60316]: debug 2022-01-31T21:04:28.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:28.188705+0000) 2022-01-31T21:04:28.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:28 smithi171 conmon[46715]: debug 2022-01-31T21:04:28.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:28.216135+0000) 2022-01-31T21:04:28.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:28 smithi167 conmon[49112]: debug 2022-01-31T21:04:28.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:28.509944+0000) 2022-01-31T21:04:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:28 smithi167 conmon[54076]: debug 2022-01-31T21:04:28.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:28.437664+0000) 2022-01-31T21:04:28.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:28 smithi171 conmon[41853]: debug 2022-01-31T21:04:28.669+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:28.670651+0000) 2022-01-31T21:04:28.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:28 smithi171 conmon[51620]: debug 2022-01-31T21:04:28.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:28.543824+0000) 2022-01-31T21:04:29.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:29 smithi167 conmon[60316]: debug 2022-01-31T21:04:29.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:29.188865+0000) 2022-01-31T21:04:29.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:29 smithi171 conmon[46715]: debug 2022-01-31T21:04:29.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:29.216344+0000) 2022-01-31T21:04:29.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:29 smithi167 conmon[54076]: debug 2022-01-31T21:04:29.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:29.437807+0000) 2022-01-31T21:04:29.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:29 smithi167 conmon[49112]: debug 2022-01-31T21:04:29.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:29.510131+0000) 2022-01-31T21:04:29.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:29 smithi171 conmon[51620]: debug 2022-01-31T21:04:29.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:29.543981+0000) 2022-01-31T21:04:29.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:29 smithi171 conmon[41853]: debug 2022-01-31T21:04:29.670+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:29.670840+0000) 2022-01-31T21:04:30.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:30 smithi167 conmon[60316]: debug 2022-01-31T21:04:30.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.189017+0000) 2022-01-31T21:04:30.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[46715]: debug 2022-01-31T21:04:30.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.216572+0000) 2022-01-31T21:04:30.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:30 smithi167 conmon[54076]: debug 2022-01-31T21:04:30.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.437960+0000) 2022-01-31T21:04:30.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:30 smithi167 conmon[49112]: debug 2022-01-31T21:04:30.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.510333+0000) 2022-01-31T21:04:30.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[51620]: debug 2022-01-31T21:04:30.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.544142+0000) 2022-01-31T21:04:30.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[41853]: debug 2022-01-31T21:04:30.670+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.671018+0000) 2022-01-31T21:04:31.037 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:30 smithi167 conmon[49112]: debug 2022-01-31T21:04:30.827+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.829242+0000) 2022-01-31T21:04:31.038 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:30 smithi167 conmon[54076]: debug 2022-01-31T21:04:30.826+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.828201+0000) 2022-01-31T21:04:31.038 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:30 smithi167 conmon[60316]: debug 2022-01-31T21:04:30.826+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.828010+0000) 2022-01-31T21:04:31.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[46715]: debug 2022-01-31T21:04:30.827+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.828419+0000) 2022-01-31T21:04:31.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[35325]: debug 2022-01-31T21:04:30.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 100972 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[41853]: debug 2022-01-31T21:04:30.827+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.827785+0000) 2022-01-31T21:04:31.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:30 smithi171 conmon[51620]: debug 2022-01-31T21:04:30.828+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:30.828887+0000) 2022-01-31T21:04:31.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:31 smithi167 conmon[60316]: debug 2022-01-31T21:04:31.187+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:31.189165+0000) 2022-01-31T21:04:31.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:31 smithi171 conmon[46715]: debug 2022-01-31T21:04:31.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:31.216776+0000) 2022-01-31T21:04:31.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:31 smithi167 conmon[49112]: debug 2022-01-31T21:04:31.508+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:31.510495+0000) 2022-01-31T21:04:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:31 smithi167 conmon[54076]: debug 2022-01-31T21:04:31.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:31.438110+0000) 2022-01-31T21:04:31.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:31 smithi171 conmon[51620]: debug 2022-01-31T21:04:31.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:31.544302+0000) 2022-01-31T21:04:31.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:31 smithi171 conmon[41853]: debug 2022-01-31T21:04:31.670+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:31.671200+0000) 2022-01-31T21:04:32.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:32 smithi167 conmon[60316]: debug 2022-01-31T21:04:32.188+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:32.189341+0000) 2022-01-31T21:04:32.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:32 smithi171 conmon[46715]: debug 2022-01-31T21:04:32.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:32.216939+0000) 2022-01-31T21:04:32.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:32 smithi167 conmon[49112]: debug 2022-01-31T21:04:32.509+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:32.510593+0000) 2022-01-31T21:04:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:32 smithi167 conmon[54076]: debug 2022-01-31T21:04:32.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:32.438276+0000) 2022-01-31T21:04:32.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:32 smithi171 conmon[41853]: debug 2022-01-31T21:04:32.670+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:32.671382+0000) 2022-01-31T21:04:32.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:32 smithi171 conmon[51620]: debug 2022-01-31T21:04:32.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:32.544464+0000) 2022-01-31T21:04:33.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:33 smithi167 conmon[60316]: debug 2022-01-31T21:04:33.188+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:33.189544+0000) 2022-01-31T21:04:33.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:33 smithi171 conmon[46715]: debug 2022-01-31T21:04:33.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:33.217165+0000) 2022-01-31T21:04:33.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:33 smithi167 conmon[49112]: debug 2022-01-31T21:04:33.509+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:33.510766+0000) 2022-01-31T21:04:33.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:33 smithi167 conmon[54076]: debug 2022-01-31T21:04:33.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:33.438457+0000) 2022-01-31T21:04:33.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:33 smithi171 conmon[41853]: debug 2022-01-31T21:04:33.670+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:33.671537+0000) 2022-01-31T21:04:33.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:33 smithi171 conmon[51620]: debug 2022-01-31T21:04:33.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:33.544653+0000) 2022-01-31T21:04:34.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:34 smithi167 conmon[60316]: debug 2022-01-31T21:04:34.188+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:34.189697+0000) 2022-01-31T21:04:34.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:34 smithi171 conmon[46715]: debug 2022-01-31T21:04:34.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:34.217351+0000) 2022-01-31T21:04:34.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:34 smithi167 conmon[49112]: debug 2022-01-31T21:04:34.510+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:34.510946+0000) 2022-01-31T21:04:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:34 smithi167 conmon[54076]: debug 2022-01-31T21:04:34.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:34.438638+0000) 2022-01-31T21:04:34.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:34 smithi171 conmon[51620]: debug 2022-01-31T21:04:34.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:34.544848+0000) 2022-01-31T21:04:34.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:34 smithi171 conmon[41853]: debug 2022-01-31T21:04:34.670+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:34.671660+0000) 2022-01-31T21:04:35.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:35 smithi167 conmon[60316]: debug 2022-01-31T21:04:35.189+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.189874+0000) 2022-01-31T21:04:35.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[46715]: debug 2022-01-31T21:04:35.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.217532+0000) 2022-01-31T21:04:35.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:35 smithi167 conmon[49112]: debug 2022-01-31T21:04:35.510+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.511084+0000) 2022-01-31T21:04:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:35 smithi167 conmon[54076]: debug 2022-01-31T21:04:35.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.438749+0000) 2022-01-31T21:04:35.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[41853]: debug 2022-01-31T21:04:35.671+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.671830+0000) 2022-01-31T21:04:35.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[51620]: debug 2022-01-31T21:04:35.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.544983+0000) 2022-01-31T21:04:36.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:35 smithi167 conmon[49112]: debug 2022-01-31T21:04:35.847+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.848799+0000) 2022-01-31T21:04:36.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:35 smithi167 conmon[54076]: debug 2022-01-31T21:04:35.840+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.841578+0000) 2022-01-31T21:04:36.044 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:35 smithi167 conmon[60316]: debug 2022-01-31T21:04:35.847+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.848670+0000) 2022-01-31T21:04:36.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[35325]: debug 2022-01-31T21:04:35.853+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101083 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[41853]: debug 2022-01-31T21:04:35.840+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.841605+0000) 2022-01-31T21:04:36.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[46715]: debug 2022-01-31T21:04:35.841+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.842482+0000) 2022-01-31T21:04:36.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:35 smithi171 conmon[51620]: debug 2022-01-31T21:04:35.840+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:35.841259+0000) 2022-01-31T21:04:36.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:36 smithi167 conmon[60316]: debug 2022-01-31T21:04:36.189+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:36.190073+0000) 2022-01-31T21:04:36.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:36 smithi171 conmon[46715]: debug 2022-01-31T21:04:36.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:36.217729+0000) 2022-01-31T21:04:36.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:36 smithi167 conmon[49112]: debug 2022-01-31T21:04:36.510+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:36.511250+0000) 2022-01-31T21:04:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:36 smithi167 conmon[54076]: debug 2022-01-31T21:04:36.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:36.438867+0000) 2022-01-31T21:04:36.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:36 smithi171 conmon[41853]: debug 2022-01-31T21:04:36.671+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:36.671992+0000) 2022-01-31T21:04:36.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:36 smithi171 conmon[51620]: debug 2022-01-31T21:04:36.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:36.545186+0000) 2022-01-31T21:04:37.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:37 smithi167 conmon[60316]: debug 2022-01-31T21:04:37.188+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:37.190220+0000) 2022-01-31T21:04:37.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:37 smithi171 conmon[46715]: debug 2022-01-31T21:04:37.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:37.217905+0000) 2022-01-31T21:04:37.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:37 smithi167 conmon[49112]: debug 2022-01-31T21:04:37.510+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:37.511402+0000) 2022-01-31T21:04:37.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:37 smithi167 conmon[54076]: debug 2022-01-31T21:04:37.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:37.439005+0000) 2022-01-31T21:04:37.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:37 smithi171 conmon[41853]: debug 2022-01-31T21:04:37.671+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:37.672140+0000) 2022-01-31T21:04:37.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:37 smithi171 conmon[51620]: debug 2022-01-31T21:04:37.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:37.545341+0000) 2022-01-31T21:04:38.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:38 smithi167 conmon[60316]: debug 2022-01-31T21:04:38.188+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:38.190406+0000) 2022-01-31T21:04:38.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:38 smithi171 conmon[46715]: debug 2022-01-31T21:04:38.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:38.218116+0000) 2022-01-31T21:04:38.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:38 smithi167 conmon[49112]: debug 2022-01-31T21:04:38.510+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:38.511604+0000) 2022-01-31T21:04:38.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:38 smithi167 conmon[54076]: debug 2022-01-31T21:04:38.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:38.439206+0000) 2022-01-31T21:04:38.796 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:38 smithi171 conmon[35325]: debug 2022-01-31T21:04:38.619+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:04:38.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:38 smithi171 conmon[41853]: debug 2022-01-31T21:04:38.671+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:38.672323+0000) 2022-01-31T21:04:38.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:38 smithi171 conmon[51620]: debug 2022-01-31T21:04:38.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:38.545564+0000) 2022-01-31T21:04:39.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:39 smithi167 conmon[60316]: debug 2022-01-31T21:04:39.189+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:39.190597+0000) 2022-01-31T21:04:39.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:39 smithi171 conmon[46715]: debug 2022-01-31T21:04:39.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:39.218304+0000) 2022-01-31T21:04:39.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:39 smithi167 conmon[49112]: debug 2022-01-31T21:04:39.511+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:39.511765+0000) 2022-01-31T21:04:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:39 smithi167 conmon[54076]: debug 2022-01-31T21:04:39.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:39.439418+0000) 2022-01-31T21:04:39.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:39 smithi171 conmon[41853]: debug 2022-01-31T21:04:39.671+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:39.672553+0000) 2022-01-31T21:04:39.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:39 smithi171 conmon[51620]: debug 2022-01-31T21:04:39.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:39.545727+0000) 2022-01-31T21:04:40.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[60316]: debug 2022-01-31T21:04:40.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.190743+0000) 2022-01-31T21:04:40.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[46715]: debug 2022-01-31T21:04:40.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.218458+0000) 2022-01-31T21:04:40.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[49112]: debug 2022-01-31T21:04:40.511+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.511924+0000) 2022-01-31T21:04:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[54076]: debug 2022-01-31T21:04:40.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.439555+0000) 2022-01-31T21:04:40.796 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[41853]: debug 2022-01-31T21:04:40.672+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.672737+0000) 2022-01-31T21:04:40.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[51620]: debug 2022-01-31T21:04:40.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.545853+0000) 2022-01-31T21:04:41.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[54076]: debug 2022-01-31T21:04:40.855+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.857362+0000) 2022-01-31T21:04:41.049 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[60316]: debug 2022-01-31T21:04:40.856+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.857842+0000) 2022-01-31T21:04:41.050 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[49112]: debug 2022-01-31T21:04:40.856+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.858162+0000) 2022-01-31T21:04:41.050 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:40 smithi167 conmon[49112]: 2022-01-31T21:04:41.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[35325]: debug 2022-01-31T21:04:40.866+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101193 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[41853]: debug 2022-01-31T21:04:40.855+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.856719+0000) 2022-01-31T21:04:41.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[46715]: debug 2022-01-31T21:04:40.857+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.858028+0000) 2022-01-31T21:04:41.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:40 smithi171 conmon[51620]: debug 2022-01-31T21:04:40.856+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:40.857532+0000) 2022-01-31T21:04:41.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:41 smithi167 conmon[60316]: debug 2022-01-31T21:04:41.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:41.190896+0000) 2022-01-31T21:04:41.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:41 smithi171 conmon[46715]: debug 2022-01-31T21:04:41.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:41.218623+0000) 2022-01-31T21:04:41.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:41 smithi167 conmon[49112]: debug 2022-01-31T21:04:41.511+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:41.512158+0000) 2022-01-31T21:04:41.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:41 smithi167 conmon[54076]: debug 2022-01-31T21:04:41.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:41.439703+0000) 2022-01-31T21:04:41.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:41 smithi171 conmon[51620]: debug 2022-01-31T21:04:41.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:41.546039+0000) 2022-01-31T21:04:41.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:41 smithi171 conmon[41853]: debug 2022-01-31T21:04:41.672+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:41.672946+0000) 2022-01-31T21:04:42.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:42 smithi167 conmon[60316]: debug 2022-01-31T21:04:42.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:42.191060+0000) 2022-01-31T21:04:42.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:42 smithi171 conmon[46715]: debug 2022-01-31T21:04:42.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:42.218777+0000) 2022-01-31T21:04:42.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:42 smithi167 conmon[49112]: debug 2022-01-31T21:04:42.511+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:42.512335+0000) 2022-01-31T21:04:42.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:42 smithi167 conmon[54076]: debug 2022-01-31T21:04:42.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:42.439857+0000) 2022-01-31T21:04:42.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:42 smithi171 conmon[41853]: debug 2022-01-31T21:04:42.672+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:42.673165+0000) 2022-01-31T21:04:42.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:42 smithi171 conmon[51620]: debug 2022-01-31T21:04:42.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:42.546185+0000) 2022-01-31T21:04:43.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:43 smithi167 conmon[60316]: debug 2022-01-31T21:04:43.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:43.191253+0000) 2022-01-31T21:04:43.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:43 smithi171 conmon[46715]: debug 2022-01-31T21:04:43.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:43.218934+0000) 2022-01-31T21:04:43.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:43 smithi167 conmon[49112]: debug 2022-01-31T21:04:43.511+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:43.512491+0000) 2022-01-31T21:04:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:43 smithi167 conmon[54076]: debug 2022-01-31T21:04:43.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:43.440053+0000) 2022-01-31T21:04:43.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:43 smithi171 conmon[51620]: debug 2022-01-31T21:04:43.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:43.546387+0000) 2022-01-31T21:04:43.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:43 smithi171 conmon[41853]: debug 2022-01-31T21:04:43.672+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:43.673361+0000) 2022-01-31T21:04:44.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:44 smithi167 conmon[60316]: debug 2022-01-31T21:04:44.189+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:44.191449+0000) 2022-01-31T21:04:44.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:44 smithi171 conmon[46715]: debug 2022-01-31T21:04:44.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:44.219044+0000) 2022-01-31T21:04:44.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:44 smithi167 conmon[49112]: debug 2022-01-31T21:04:44.512+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:44.512722+0000) 2022-01-31T21:04:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:44 smithi167 conmon[54076]: debug 2022-01-31T21:04:44.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:44.440245+0000) 2022-01-31T21:04:44.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:44 smithi171 conmon[51620]: debug 2022-01-31T21:04:44.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:44.546583+0000) 2022-01-31T21:04:44.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:44 smithi171 conmon[41853]: debug 2022-01-31T21:04:44.673+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:44.673534+0000) 2022-01-31T21:04:45.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:45 smithi167 conmon[60316]: debug 2022-01-31T21:04:45.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.191664+0000) 2022-01-31T21:04:45.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[46715]: debug 2022-01-31T21:04:45.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.219199+0000) 2022-01-31T21:04:45.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:45 smithi167 conmon[49112]: debug 2022-01-31T21:04:45.512+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.512902+0000) 2022-01-31T21:04:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:45 smithi167 conmon[54076]: debug 2022-01-31T21:04:45.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.440450+0000) 2022-01-31T21:04:45.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[51620]: debug 2022-01-31T21:04:45.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.546789+0000) 2022-01-31T21:04:45.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[41853]: debug 2022-01-31T21:04:45.672+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.673719+0000) 2022-01-31T21:04:46.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:45 smithi167 conmon[49112]: debug 2022-01-31T21:04:45.868+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.870202+0000) 2022-01-31T21:04:46.056 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:45 smithi167 conmon[54076]: debug 2022-01-31T21:04:45.869+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.871261+0000) 2022-01-31T21:04:46.057 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:45 smithi167 conmon[60316]: debug 2022-01-31T21:04:45.868+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.870411+0000) 2022-01-31T21:04:46.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[41853]: debug 2022-01-31T21:04:45.868+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.869649+0000) 2022-01-31T21:04:46.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[51620]: debug 2022-01-31T21:04:45.870+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.871528+0000) 2022-01-31T21:04:46.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[35325]: debug 2022-01-31T21:04:45.879+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101304 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:46.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:45 smithi171 conmon[46715]: debug 2022-01-31T21:04:45.870+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:45.871251+0000) 2022-01-31T21:04:46.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:46 smithi167 conmon[60316]: debug 2022-01-31T21:04:46.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:46.191808+0000) 2022-01-31T21:04:46.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:46 smithi171 conmon[46715]: debug 2022-01-31T21:04:46.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:46.219409+0000) 2022-01-31T21:04:46.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:46 smithi167 conmon[49112]: debug 2022-01-31T21:04:46.512+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:46.513085+0000) 2022-01-31T21:04:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:46 smithi167 conmon[54076]: debug 2022-01-31T21:04:46.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:46.440595+0000) 2022-01-31T21:04:46.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:46 smithi171 conmon[51620]: debug 2022-01-31T21:04:46.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:46.546992+0000) 2022-01-31T21:04:46.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:46 smithi171 conmon[41853]: debug 2022-01-31T21:04:46.673+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:46.673923+0000) 2022-01-31T21:04:47.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:47 smithi167 conmon[60316]: debug 2022-01-31T21:04:47.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:47.191963+0000) 2022-01-31T21:04:47.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:47 smithi171 conmon[46715]: debug 2022-01-31T21:04:47.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:47.219579+0000) 2022-01-31T21:04:47.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:47 smithi167 conmon[49112]: debug 2022-01-31T21:04:47.512+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:47.513230+0000) 2022-01-31T21:04:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:47 smithi167 conmon[54076]: debug 2022-01-31T21:04:47.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:47.440715+0000) 2022-01-31T21:04:47.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:47 smithi171 conmon[41853]: debug 2022-01-31T21:04:47.673+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:47.674050+0000) 2022-01-31T21:04:47.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:47 smithi171 conmon[51620]: debug 2022-01-31T21:04:47.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:47.547175+0000) 2022-01-31T21:04:48.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:48 smithi167 conmon[60316]: debug 2022-01-31T21:04:48.191+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:48.192128+0000) 2022-01-31T21:04:48.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:48 smithi171 conmon[46715]: debug 2022-01-31T21:04:48.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:48.219766+0000) 2022-01-31T21:04:48.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:48 smithi167 conmon[49112]: debug 2022-01-31T21:04:48.512+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:48.513386+0000) 2022-01-31T21:04:48.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:48 smithi167 conmon[54076]: debug 2022-01-31T21:04:48.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:48.440935+0000) 2022-01-31T21:04:48.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:48 smithi171 conmon[41853]: debug 2022-01-31T21:04:48.673+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:48.674144+0000) 2022-01-31T21:04:48.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:48 smithi171 conmon[51620]: debug 2022-01-31T21:04:48.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:48.547368+0000) 2022-01-31T21:04:49.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:49 smithi167 conmon[60316]: debug 2022-01-31T21:04:49.190+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:49.192337+0000) 2022-01-31T21:04:49.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:49 smithi171 conmon[46715]: debug 2022-01-31T21:04:49.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:49.219897+0000) 2022-01-31T21:04:49.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:49 smithi167 conmon[49112]: debug 2022-01-31T21:04:49.512+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:49.513565+0000) 2022-01-31T21:04:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:49 smithi167 conmon[54076]: debug 2022-01-31T21:04:49.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:49.441080+0000) 2022-01-31T21:04:49.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:49 smithi171 conmon[51620]: debug 2022-01-31T21:04:49.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:49.547552+0000) 2022-01-31T21:04:49.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:49 smithi171 conmon[41853]: debug 2022-01-31T21:04:49.673+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:49.674303+0000) 2022-01-31T21:04:50.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:50 smithi167 conmon[60316]: debug 2022-01-31T21:04:50.192+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.192539+0000) 2022-01-31T21:04:50.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[46715]: debug 2022-01-31T21:04:50.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.219997+0000) 2022-01-31T21:04:50.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:50 smithi167 conmon[49112]: debug 2022-01-31T21:04:50.513+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.513698+0000) 2022-01-31T21:04:50.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:50 smithi167 conmon[54076]: debug 2022-01-31T21:04:50.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.441273+0000) 2022-01-31T21:04:50.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[51620]: debug 2022-01-31T21:04:50.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.547735+0000) 2022-01-31T21:04:50.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[41853]: debug 2022-01-31T21:04:50.674+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.674462+0000) 2022-01-31T21:04:51.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:50 smithi167 conmon[49112]: debug 2022-01-31T21:04:50.883+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.883917+0000) 2022-01-31T21:04:51.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:50 smithi167 conmon[54076]: debug 2022-01-31T21:04:50.882+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.882862+0000) 2022-01-31T21:04:51.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:50 smithi167 conmon[60316]: debug 2022-01-31T21:04:50.882+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.882701+0000) 2022-01-31T21:04:51.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[35325]: debug 2022-01-31T21:04:50.892+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101413 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[41853]: debug 2022-01-31T21:04:50.882+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.883463+0000) 2022-01-31T21:04:51.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[46715]: debug 2022-01-31T21:04:50.882+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.883727+0000) 2022-01-31T21:04:51.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:50 smithi171 conmon[51620]: debug 2022-01-31T21:04:50.883+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:50.884116+0000) 2022-01-31T21:04:51.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:51 smithi167 conmon[60316]: debug 2022-01-31T21:04:51.192+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:51.192730+0000) 2022-01-31T21:04:51.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:51 smithi171 conmon[46715]: debug 2022-01-31T21:04:51.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:51.220128+0000) 2022-01-31T21:04:51.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:51 smithi167 conmon[54076]: debug 2022-01-31T21:04:51.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:51.441474+0000) 2022-01-31T21:04:51.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:51 smithi167 conmon[49112]: debug 2022-01-31T21:04:51.513+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:51.513843+0000) 2022-01-31T21:04:51.798 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:51 smithi171 conmon[41853]: debug 2022-01-31T21:04:51.674+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:51.674618+0000) 2022-01-31T21:04:51.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:51 smithi171 conmon[51620]: debug 2022-01-31T21:04:51.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:51.547946+0000) 2022-01-31T21:04:52.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:52 smithi167 conmon[60316]: debug 2022-01-31T21:04:52.192+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:52.192889+0000) 2022-01-31T21:04:52.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:52 smithi171 conmon[46715]: debug 2022-01-31T21:04:52.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:52.220261+0000) 2022-01-31T21:04:52.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:52 smithi167 conmon[49112]: debug 2022-01-31T21:04:52.513+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:52.513993+0000) 2022-01-31T21:04:52.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:52 smithi167 conmon[54076]: debug 2022-01-31T21:04:52.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:52.441616+0000) 2022-01-31T21:04:52.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:52 smithi171 conmon[51620]: debug 2022-01-31T21:04:52.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:52.548114+0000) 2022-01-31T21:04:52.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:52 smithi171 conmon[41853]: debug 2022-01-31T21:04:52.674+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:52.674799+0000) 2022-01-31T21:04:53.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:53 smithi167 conmon[60316]: debug 2022-01-31T21:04:53.192+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:53.193053+0000) 2022-01-31T21:04:53.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:53 smithi171 conmon[46715]: debug 2022-01-31T21:04:53.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:53.220437+0000) 2022-01-31T21:04:53.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:53 smithi167 conmon[49112]: debug 2022-01-31T21:04:53.513+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:53.514173+0000) 2022-01-31T21:04:53.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:53 smithi167 conmon[54076]: debug 2022-01-31T21:04:53.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:53.441818+0000) 2022-01-31T21:04:53.798 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:53 smithi171 conmon[35325]: debug 2022-01-31T21:04:53.620+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:04:53.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:53 smithi171 conmon[41853]: debug 2022-01-31T21:04:53.674+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:53.674999+0000) 2022-01-31T21:04:53.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:53 smithi171 conmon[51620]: debug 2022-01-31T21:04:53.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:53.548311+0000) 2022-01-31T21:04:54.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:54 smithi167 conmon[60316]: debug 2022-01-31T21:04:54.192+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:54.193211+0000) 2022-01-31T21:04:54.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:54 smithi171 conmon[46715]: debug 2022-01-31T21:04:54.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:54.220616+0000) 2022-01-31T21:04:54.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:54 smithi167 conmon[49112]: debug 2022-01-31T21:04:54.513+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:54.514314+0000) 2022-01-31T21:04:54.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:54 smithi167 conmon[54076]: debug 2022-01-31T21:04:54.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:54.441982+0000) 2022-01-31T21:04:54.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:54 smithi171 conmon[51620]: debug 2022-01-31T21:04:54.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:54.548488+0000) 2022-01-31T21:04:54.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:54 smithi171 conmon[41853]: debug 2022-01-31T21:04:54.674+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:54.675199+0000) 2022-01-31T21:04:55.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:55 smithi167 conmon[60316]: debug 2022-01-31T21:04:55.192+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.193404+0000) 2022-01-31T21:04:55.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[46715]: debug 2022-01-31T21:04:55.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.220782+0000) 2022-01-31T21:04:55.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:55 smithi167 conmon[49112]: debug 2022-01-31T21:04:55.513+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.514469+0000) 2022-01-31T21:04:55.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:55 smithi167 conmon[54076]: debug 2022-01-31T21:04:55.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.442148+0000) 2022-01-31T21:04:55.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[51620]: debug 2022-01-31T21:04:55.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.548676+0000) 2022-01-31T21:04:55.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[41853]: debug 2022-01-31T21:04:55.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.675402+0000) 2022-01-31T21:04:56.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:55 smithi167 conmon[49112]: debug 2022-01-31T21:04:55.896+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.897422+0000) 2022-01-31T21:04:56.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:55 smithi167 conmon[54076]: debug 2022-01-31T21:04:55.895+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.896180+0000) 2022-01-31T21:04:56.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:55 smithi167 conmon[60316]: debug 2022-01-31T21:04:55.895+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.896072+0000) 2022-01-31T21:04:56.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[35325]: debug 2022-01-31T21:04:55.905+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101524 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:04:56.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[46715]: debug 2022-01-31T21:04:55.895+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.896040+0000) 2022-01-31T21:04:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[41853]: debug 2022-01-31T21:04:55.895+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.896388+0000) 2022-01-31T21:04:56.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:55 smithi171 conmon[51620]: debug 2022-01-31T21:04:55.895+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:55.896508+0000) 2022-01-31T21:04:56.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:56 smithi167 conmon[60316]: debug 2022-01-31T21:04:56.193+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:56.193565+0000) 2022-01-31T21:04:56.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:56 smithi171 conmon[46715]: debug 2022-01-31T21:04:56.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:56.220965+0000) 2022-01-31T21:04:56.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:56 smithi167 conmon[49112]: debug 2022-01-31T21:04:56.514+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:56.514639+0000) 2022-01-31T21:04:56.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:56 smithi167 conmon[54076]: debug 2022-01-31T21:04:56.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:56.442368+0000) 2022-01-31T21:04:56.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:56 smithi171 conmon[41853]: debug 2022-01-31T21:04:56.674+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:56.675574+0000) 2022-01-31T21:04:56.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:56 smithi171 conmon[51620]: debug 2022-01-31T21:04:56.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:56.548865+0000) 2022-01-31T21:04:57.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:57 smithi167 conmon[60316]: debug 2022-01-31T21:04:57.193+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:57.193701+0000) 2022-01-31T21:04:57.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:57 smithi171 conmon[46715]: debug 2022-01-31T21:04:57.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:57.221100+0000) 2022-01-31T21:04:57.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:57 smithi167 conmon[49112]: debug 2022-01-31T21:04:57.514+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:57.514764+0000) 2022-01-31T21:04:57.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:57 smithi167 conmon[54076]: debug 2022-01-31T21:04:57.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:57.442540+0000) 2022-01-31T21:04:57.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:57 smithi171 conmon[41853]: debug 2022-01-31T21:04:57.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:57.675701+0000) 2022-01-31T21:04:57.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:57 smithi171 conmon[51620]: debug 2022-01-31T21:04:57.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:57.548995+0000) 2022-01-31T21:04:58.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:58 smithi167 conmon[60316]: debug 2022-01-31T21:04:58.193+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:58.193909+0000) 2022-01-31T21:04:58.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:58 smithi171 conmon[46715]: debug 2022-01-31T21:04:58.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:58.221277+0000) 2022-01-31T21:04:58.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:58 smithi167 conmon[49112]: debug 2022-01-31T21:04:58.514+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:58.514920+0000) 2022-01-31T21:04:58.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:58 smithi167 conmon[54076]: debug 2022-01-31T21:04:58.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:58.442742+0000) 2022-01-31T21:04:58.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:58 smithi171 conmon[51620]: debug 2022-01-31T21:04:58.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:58.549158+0000) 2022-01-31T21:04:58.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:58 smithi171 conmon[41853]: debug 2022-01-31T21:04:58.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:58.675852+0000) 2022-01-31T21:04:59.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:04:59 smithi167 conmon[60316]: debug 2022-01-31T21:04:59.193+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:59.194075+0000) 2022-01-31T21:04:59.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:04:59 smithi171 conmon[46715]: debug 2022-01-31T21:04:59.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:59.221413+0000) 2022-01-31T21:04:59.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:04:59 smithi167 conmon[49112]: debug 2022-01-31T21:04:59.514+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:59.515082+0000) 2022-01-31T21:04:59.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:04:59 smithi167 conmon[54076]: debug 2022-01-31T21:04:59.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:59.442879+0000) 2022-01-31T21:04:59.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:04:59 smithi171 conmon[51620]: debug 2022-01-31T21:04:59.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:59.549348+0000) 2022-01-31T21:04:59.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:04:59 smithi171 conmon[41853]: debug 2022-01-31T21:04:59.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:04:59.675987+0000) 2022-01-31T21:05:00.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:00 smithi167 conmon[60316]: debug 2022-01-31T21:05:00.193+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.194255+0000) 2022-01-31T21:05:00.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[46715]: debug 2022-01-31T21:05:00.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.221579+0000) 2022-01-31T21:05:00.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:00 smithi167 conmon[49112]: debug 2022-01-31T21:05:00.514+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.515294+0000) 2022-01-31T21:05:00.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:00 smithi167 conmon[54076]: debug 2022-01-31T21:05:00.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.443038+0000) 2022-01-31T21:05:00.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[51620]: debug 2022-01-31T21:05:00.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.549510+0000) 2022-01-31T21:05:00.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[41853]: debug 2022-01-31T21:05:00.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.676101+0000) 2022-01-31T21:05:01.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:00 smithi167 conmon[49112]: debug 2022-01-31T21:05:00.909+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.910003+0000) 2022-01-31T21:05:01.073 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:00 smithi167 conmon[54076]: debug 2022-01-31T21:05:00.909+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.910313+0000) 2022-01-31T21:05:01.074 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:00 smithi167 conmon[60316]: debug 2022-01-31T21:05:00.908+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.908513+0000) 2022-01-31T21:05:01.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[35325]: debug 2022-01-31T21:05:00.918+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101633 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[41853]: debug 2022-01-31T21:05:00.909+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.910176+0000) 2022-01-31T21:05:01.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[46715]: debug 2022-01-31T21:05:00.908+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.909452+0000) 2022-01-31T21:05:01.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:00 smithi171 conmon[51620]: debug 2022-01-31T21:05:00.908+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:00.908964+0000) 2022-01-31T21:05:01.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:01 smithi167 conmon[60316]: debug 2022-01-31T21:05:01.194+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:01.194418+0000) 2022-01-31T21:05:01.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:01 smithi171 conmon[46715]: debug 2022-01-31T21:05:01.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:01.221796+0000) 2022-01-31T21:05:01.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:01 smithi167 conmon[49112]: debug 2022-01-31T21:05:01.515+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:01.515528+0000) 2022-01-31T21:05:01.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:01 smithi167 conmon[54076]: debug 2022-01-31T21:05:01.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:01.443225+0000) 2022-01-31T21:05:01.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:01 smithi171 conmon[51620]: debug 2022-01-31T21:05:01.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:01.549662+0000) 2022-01-31T21:05:01.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:01 smithi171 conmon[41853]: debug 2022-01-31T21:05:01.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:01.676345+0000) 2022-01-31T21:05:02.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:02 smithi167 conmon[60316]: debug 2022-01-31T21:05:02.194+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:02.194588+0000) 2022-01-31T21:05:02.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:02 smithi171 conmon[46715]: debug 2022-01-31T21:05:02.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:02.221948+0000) 2022-01-31T21:05:02.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:02 smithi167 conmon[49112]: debug 2022-01-31T21:05:02.515+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:02.515673+0000) 2022-01-31T21:05:02.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:02 smithi167 conmon[54076]: debug 2022-01-31T21:05:02.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:02.443396+0000) 2022-01-31T21:05:02.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:02 smithi171 conmon[51620]: debug 2022-01-31T21:05:02.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:02.549807+0000) 2022-01-31T21:05:02.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:02 smithi171 conmon[41853]: debug 2022-01-31T21:05:02.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:02.676525+0000) 2022-01-31T21:05:03.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:03 smithi167 conmon[60316]: debug 2022-01-31T21:05:03.194+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:03.194774+0000) 2022-01-31T21:05:03.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:03 smithi171 conmon[46715]: debug 2022-01-31T21:05:03.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:03.222167+0000) 2022-01-31T21:05:03.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:03 smithi167 conmon[49112]: debug 2022-01-31T21:05:03.515+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:03.515871+0000) 2022-01-31T21:05:03.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:03 smithi167 conmon[54076]: debug 2022-01-31T21:05:03.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:03.443589+0000) 2022-01-31T21:05:03.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:03 smithi171 conmon[51620]: debug 2022-01-31T21:05:03.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:03.549979+0000) 2022-01-31T21:05:03.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:03 smithi171 conmon[41853]: debug 2022-01-31T21:05:03.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:03.676674+0000) 2022-01-31T21:05:04.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:04 smithi167 conmon[60316]: debug 2022-01-31T21:05:04.194+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:04.194964+0000) 2022-01-31T21:05:04.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:04 smithi171 conmon[46715]: debug 2022-01-31T21:05:04.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:04.222358+0000) 2022-01-31T21:05:04.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:04 smithi167 conmon[49112]: debug 2022-01-31T21:05:04.515+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:04.516036+0000) 2022-01-31T21:05:04.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:04 smithi167 conmon[54076]: debug 2022-01-31T21:05:04.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:04.443742+0000) 2022-01-31T21:05:04.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:04 smithi171 conmon[51620]: debug 2022-01-31T21:05:04.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:04.550136+0000) 2022-01-31T21:05:04.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:04 smithi171 conmon[41853]: debug 2022-01-31T21:05:04.675+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:04.676828+0000) 2022-01-31T21:05:05.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:05 smithi167 conmon[60316]: debug 2022-01-31T21:05:05.194+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.195105+0000) 2022-01-31T21:05:05.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[46715]: debug 2022-01-31T21:05:05.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.222514+0000) 2022-01-31T21:05:05.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:05 smithi167 conmon[49112]: debug 2022-01-31T21:05:05.515+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.516197+0000) 2022-01-31T21:05:05.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:05 smithi167 conmon[54076]: debug 2022-01-31T21:05:05.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.443904+0000) 2022-01-31T21:05:05.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[41853]: debug 2022-01-31T21:05:05.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.676989+0000) 2022-01-31T21:05:05.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[51620]: debug 2022-01-31T21:05:05.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.550326+0000) 2022-01-31T21:05:06.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:05 smithi167 conmon[49112]: debug 2022-01-31T21:05:05.921+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.922337+0000) 2022-01-31T21:05:06.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:05 smithi167 conmon[60316]: debug 2022-01-31T21:05:05.920+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.921354+0000) 2022-01-31T21:05:06.081 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:05 smithi167 conmon[54076]: debug 2022-01-31T21:05:05.921+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.922158+0000) 2022-01-31T21:05:06.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[35325]: debug 2022-01-31T21:05:05.930+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101743 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[41853]: debug 2022-01-31T21:05:05.921+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.923116+0000) 2022-01-31T21:05:06.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[46715]: debug 2022-01-31T21:05:05.920+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.922669+0000) 2022-01-31T21:05:06.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:05 smithi171 conmon[51620]: debug 2022-01-31T21:05:05.919+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:05.921601+0000) 2022-01-31T21:05:06.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:06 smithi167 conmon[60316]: debug 2022-01-31T21:05:06.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:06.195269+0000) 2022-01-31T21:05:06.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:06 smithi171 conmon[46715]: debug 2022-01-31T21:05:06.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:06.222697+0000) 2022-01-31T21:05:06.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:06 smithi167 conmon[49112]: debug 2022-01-31T21:05:06.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:06.516380+0000) 2022-01-31T21:05:06.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:06 smithi167 conmon[54076]: debug 2022-01-31T21:05:06.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:06.444040+0000) 2022-01-31T21:05:06.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:06 smithi171 conmon[41853]: debug 2022-01-31T21:05:06.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:06.677146+0000) 2022-01-31T21:05:06.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:06 smithi171 conmon[51620]: debug 2022-01-31T21:05:06.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:06.550468+0000) 2022-01-31T21:05:07.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:07 smithi167 conmon[60316]: debug 2022-01-31T21:05:07.194+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:07.195359+0000) 2022-01-31T21:05:07.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:07 smithi171 conmon[46715]: debug 2022-01-31T21:05:07.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:07.222827+0000) 2022-01-31T21:05:07.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:07 smithi167 conmon[49112]: debug 2022-01-31T21:05:07.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:07.516537+0000) 2022-01-31T21:05:07.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:07 smithi167 conmon[54076]: debug 2022-01-31T21:05:07.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:07.444157+0000) 2022-01-31T21:05:07.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:07 smithi171 conmon[51620]: debug 2022-01-31T21:05:07.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:07.550623+0000) 2022-01-31T21:05:07.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:07 smithi171 conmon[41853]: debug 2022-01-31T21:05:07.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:07.677296+0000) 2022-01-31T21:05:08.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:08 smithi167 conmon[60316]: debug 2022-01-31T21:05:08.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:08.195529+0000) 2022-01-31T21:05:08.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:08 smithi171 conmon[46715]: debug 2022-01-31T21:05:08.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:08.222976+0000) 2022-01-31T21:05:08.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:08 smithi167 conmon[49112]: debug 2022-01-31T21:05:08.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:08.516716+0000) 2022-01-31T21:05:08.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:08 smithi167 conmon[54076]: debug 2022-01-31T21:05:08.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:08.444349+0000) 2022-01-31T21:05:08.801 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:08 smithi171 conmon[35325]: debug 2022-01-31T21:05:08.620+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:05:08.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:08 smithi171 conmon[41853]: debug 2022-01-31T21:05:08.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:08.677440+0000) 2022-01-31T21:05:08.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:08 smithi171 conmon[51620]: debug 2022-01-31T21:05:08.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:08.550821+0000) 2022-01-31T21:05:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:09 smithi167 conmon[60316]: debug 2022-01-31T21:05:09.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:09.195689+0000) 2022-01-31T21:05:09.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:09 smithi171 conmon[46715]: debug 2022-01-31T21:05:09.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:09.223179+0000) 2022-01-31T21:05:09.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:09 smithi167 conmon[49112]: debug 2022-01-31T21:05:09.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:09.516920+0000) 2022-01-31T21:05:09.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:09 smithi167 conmon[54076]: debug 2022-01-31T21:05:09.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:09.444566+0000) 2022-01-31T21:05:09.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:09 smithi171 conmon[51620]: debug 2022-01-31T21:05:09.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:09.551012+0000) 2022-01-31T21:05:09.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:09 smithi171 conmon[41853]: debug 2022-01-31T21:05:09.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:09.677618+0000) 2022-01-31T21:05:10.362 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:10 smithi167 conmon[60316]: debug 2022-01-31T21:05:10.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.195867+0000) 2022-01-31T21:05:10.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[46715]: debug 2022-01-31T21:05:10.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.223305+0000) 2022-01-31T21:05:10.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:10 smithi167 conmon[49112]: debug 2022-01-31T21:05:10.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.517064+0000) 2022-01-31T21:05:10.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:10 smithi167 conmon[54076]: debug 2022-01-31T21:05:10.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.444726+0000) 2022-01-31T21:05:10.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[51620]: debug 2022-01-31T21:05:10.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.551201+0000) 2022-01-31T21:05:10.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[41853]: debug 2022-01-31T21:05:10.676+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.677804+0000) 2022-01-31T21:05:11.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:10 smithi167 conmon[49112]: debug 2022-01-31T21:05:10.935+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.936312+0000) 2022-01-31T21:05:11.087 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:10 smithi167 conmon[54076]: debug 2022-01-31T21:05:10.934+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.935012+0000) 2022-01-31T21:05:11.087 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:10 smithi167 conmon[60316]: debug 2022-01-31T21:05:10.935+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.935920+0000) 2022-01-31T21:05:11.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[35325]: debug 2022-01-31T21:05:10.943+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101853 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[41853]: debug 2022-01-31T21:05:10.934+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.935882+0000) 2022-01-31T21:05:11.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[46715]: debug 2022-01-31T21:05:10.934+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.936328+0000) 2022-01-31T21:05:11.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:10 smithi171 conmon[51620]: debug 2022-01-31T21:05:10.933+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:10.934879+0000) 2022-01-31T21:05:11.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:11 smithi167 conmon[60316]: debug 2022-01-31T21:05:11.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:11.196074+0000) 2022-01-31T21:05:11.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:11 smithi171 conmon[46715]: debug 2022-01-31T21:05:11.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:11.223453+0000) 2022-01-31T21:05:11.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:11 smithi167 conmon[49112]: debug 2022-01-31T21:05:11.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:11.517211+0000) 2022-01-31T21:05:11.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:11 smithi167 conmon[54076]: debug 2022-01-31T21:05:11.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:11.444847+0000) 2022-01-31T21:05:11.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:11 smithi171 conmon[41853]: debug 2022-01-31T21:05:11.677+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:11.678008+0000) 2022-01-31T21:05:11.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:11 smithi171 conmon[51620]: debug 2022-01-31T21:05:11.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:11.551391+0000) 2022-01-31T21:05:12.346 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:12 smithi167 conmon[60316]: debug 2022-01-31T21:05:12.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:12.196219+0000) 2022-01-31T21:05:12.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:12 smithi171 conmon[46715]: debug 2022-01-31T21:05:12.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:12.223593+0000) 2022-01-31T21:05:12.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:12 smithi167 conmon[49112]: debug 2022-01-31T21:05:12.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:12.517368+0000) 2022-01-31T21:05:12.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:12 smithi167 conmon[54076]: debug 2022-01-31T21:05:12.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:12.445003+0000) 2022-01-31T21:05:12.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:12 smithi171 conmon[41853]: debug 2022-01-31T21:05:12.677+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:12.678213+0000) 2022-01-31T21:05:12.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:12 smithi171 conmon[51620]: debug 2022-01-31T21:05:12.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:12.551571+0000) 2022-01-31T21:05:13.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:13 smithi167 conmon[60316]: debug 2022-01-31T21:05:13.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:13.196401+0000) 2022-01-31T21:05:13.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:13 smithi171 conmon[46715]: debug 2022-01-31T21:05:13.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:13.223767+0000) 2022-01-31T21:05:13.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:13 smithi167 conmon[49112]: debug 2022-01-31T21:05:13.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:13.517549+0000) 2022-01-31T21:05:13.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:13 smithi167 conmon[54076]: debug 2022-01-31T21:05:13.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:13.445175+0000) 2022-01-31T21:05:13.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:13 smithi171 conmon[51620]: debug 2022-01-31T21:05:13.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:13.551775+0000) 2022-01-31T21:05:13.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:13 smithi171 conmon[41853]: debug 2022-01-31T21:05:13.677+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:13.678379+0000) 2022-01-31T21:05:14.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:14 smithi167 conmon[60316]: debug 2022-01-31T21:05:14.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:14.196579+0000) 2022-01-31T21:05:14.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:14 smithi171 conmon[46715]: debug 2022-01-31T21:05:14.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:14.223955+0000) 2022-01-31T21:05:14.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:14 smithi167 conmon[49112]: debug 2022-01-31T21:05:14.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:14.517732+0000) 2022-01-31T21:05:14.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:14 smithi167 conmon[54076]: debug 2022-01-31T21:05:14.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:14.445360+0000) 2022-01-31T21:05:14.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:14 smithi171 conmon[41853]: debug 2022-01-31T21:05:14.677+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:14.678578+0000) 2022-01-31T21:05:14.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:14 smithi171 conmon[51620]: debug 2022-01-31T21:05:14.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:14.551951+0000) 2022-01-31T21:05:15.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:15 smithi167 conmon[60316]: debug 2022-01-31T21:05:15.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.196776+0000) 2022-01-31T21:05:15.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[46715]: debug 2022-01-31T21:05:15.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.224134+0000) 2022-01-31T21:05:15.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:15 smithi167 conmon[49112]: debug 2022-01-31T21:05:15.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.517865+0000) 2022-01-31T21:05:15.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:15 smithi167 conmon[54076]: debug 2022-01-31T21:05:15.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.445556+0000) 2022-01-31T21:05:15.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[41853]: debug 2022-01-31T21:05:15.678+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.678784+0000) 2022-01-31T21:05:15.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[51620]: debug 2022-01-31T21:05:15.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.552138+0000) 2022-01-31T21:05:16.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:15 smithi167 conmon[49112]: debug 2022-01-31T21:05:15.946+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.948324+0000) 2022-01-31T21:05:16.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:15 smithi167 conmon[54076]: debug 2022-01-31T21:05:15.945+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.947283+0000) 2022-01-31T21:05:16.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:15 smithi167 conmon[60316]: debug 2022-01-31T21:05:15.946+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.947803+0000) 2022-01-31T21:05:16.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[41853]: debug 2022-01-31T21:05:15.947+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.948669+0000) 2022-01-31T21:05:16.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[46715]: debug 2022-01-31T21:05:15.948+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.949112+0000) 2022-01-31T21:05:16.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[35325]: debug 2022-01-31T21:05:15.957+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 101963 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:16.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:15 smithi171 conmon[51620]: debug 2022-01-31T21:05:15.948+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:15.948855+0000) 2022-01-31T21:05:16.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:16 smithi167 conmon[60316]: debug 2022-01-31T21:05:16.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:16.196982+0000) 2022-01-31T21:05:16.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:16 smithi171 conmon[46715]: debug 2022-01-31T21:05:16.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:16.224341+0000) 2022-01-31T21:05:16.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:16 smithi167 conmon[49112]: debug 2022-01-31T21:05:16.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:16.517985+0000) 2022-01-31T21:05:16.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:16 smithi167 conmon[54076]: debug 2022-01-31T21:05:16.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:16.445707+0000) 2022-01-31T21:05:16.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:16 smithi171 conmon[51620]: debug 2022-01-31T21:05:16.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:16.552319+0000) 2022-01-31T21:05:16.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:16 smithi171 conmon[41853]: debug 2022-01-31T21:05:16.678+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:16.678998+0000) 2022-01-31T21:05:17.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:17 smithi167 conmon[60316]: debug 2022-01-31T21:05:17.195+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:17.197125+0000) 2022-01-31T21:05:17.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:17 smithi171 conmon[46715]: debug 2022-01-31T21:05:17.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:17.224507+0000) 2022-01-31T21:05:17.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:17 smithi167 conmon[49112]: debug 2022-01-31T21:05:17.516+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:17.518130+0000) 2022-01-31T21:05:17.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:17 smithi167 conmon[54076]: debug 2022-01-31T21:05:17.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:17.445850+0000) 2022-01-31T21:05:17.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:17 smithi171 conmon[51620]: debug 2022-01-31T21:05:17.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:17.552473+0000) 2022-01-31T21:05:17.803 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:17 smithi171 conmon[41853]: debug 2022-01-31T21:05:17.678+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:17.679155+0000) 2022-01-31T21:05:18.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:18 smithi167 conmon[60316]: debug 2022-01-31T21:05:18.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:18.197230+0000) 2022-01-31T21:05:18.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:18 smithi171 conmon[46715]: debug 2022-01-31T21:05:18.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:18.224669+0000) 2022-01-31T21:05:18.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:18 smithi167 conmon[49112]: debug 2022-01-31T21:05:18.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:18.518306+0000) 2022-01-31T21:05:18.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:18 smithi167 conmon[54076]: debug 2022-01-31T21:05:18.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:18.446076+0000) 2022-01-31T21:05:18.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:18 smithi171 conmon[41853]: debug 2022-01-31T21:05:18.678+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:18.679312+0000) 2022-01-31T21:05:18.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:18 smithi171 conmon[51620]: debug 2022-01-31T21:05:18.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:18.552672+0000) 2022-01-31T21:05:19.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:19 smithi167 conmon[60316]: debug 2022-01-31T21:05:19.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:19.197420+0000) 2022-01-31T21:05:19.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:19 smithi171 conmon[46715]: debug 2022-01-31T21:05:19.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:19.224846+0000) 2022-01-31T21:05:19.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:19 smithi167 conmon[49112]: debug 2022-01-31T21:05:19.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:19.518526+0000) 2022-01-31T21:05:19.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:19 smithi167 conmon[54076]: debug 2022-01-31T21:05:19.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:19.446299+0000) 2022-01-31T21:05:19.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:19 smithi171 conmon[51620]: debug 2022-01-31T21:05:19.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:19.552864+0000) 2022-01-31T21:05:19.803 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:19 smithi171 conmon[41853]: debug 2022-01-31T21:05:19.678+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:19.679515+0000) 2022-01-31T21:05:20.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:20 smithi167 conmon[60316]: debug 2022-01-31T21:05:20.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.197627+0000) 2022-01-31T21:05:20.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[46715]: debug 2022-01-31T21:05:20.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.225031+0000) 2022-01-31T21:05:20.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:20 smithi167 conmon[49112]: debug 2022-01-31T21:05:20.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.518709+0000) 2022-01-31T21:05:20.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:20 smithi167 conmon[54076]: debug 2022-01-31T21:05:20.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.446529+0000) 2022-01-31T21:05:20.803 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[41853]: debug 2022-01-31T21:05:20.679+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.679789+0000) 2022-01-31T21:05:20.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[51620]: debug 2022-01-31T21:05:20.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.553029+0000) 2022-01-31T21:05:21.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:20 smithi167 conmon[49112]: debug 2022-01-31T21:05:20.960+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.962064+0000) 2022-01-31T21:05:21.098 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:20 smithi167 conmon[54076]: debug 2022-01-31T21:05:20.958+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.960402+0000) 2022-01-31T21:05:21.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:20 smithi167 conmon[60316]: debug 2022-01-31T21:05:20.960+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.961591+0000) 2022-01-31T21:05:21.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[41853]: debug 2022-01-31T21:05:20.959+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.960672+0000) 2022-01-31T21:05:21.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[46715]: debug 2022-01-31T21:05:20.960+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.960815+0000) 2022-01-31T21:05:21.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[35325]: debug 2022-01-31T21:05:20.970+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102073 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:21.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:20 smithi171 conmon[51620]: debug 2022-01-31T21:05:20.962+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:20.962918+0000) 2022-01-31T21:05:21.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:21 smithi167 conmon[60316]: debug 2022-01-31T21:05:21.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:21.197799+0000) 2022-01-31T21:05:21.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:21 smithi171 conmon[46715]: debug 2022-01-31T21:05:21.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:21.225214+0000) 2022-01-31T21:05:21.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:21 smithi167 conmon[49112]: debug 2022-01-31T21:05:21.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:21.518856+0000) 2022-01-31T21:05:21.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:21 smithi167 conmon[54076]: debug 2022-01-31T21:05:21.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:21.446679+0000) 2022-01-31T21:05:21.803 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:21 smithi171 conmon[41853]: debug 2022-01-31T21:05:21.679+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:21.679938+0000) 2022-01-31T21:05:21.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:21 smithi171 conmon[51620]: debug 2022-01-31T21:05:21.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:21.553192+0000) 2022-01-31T21:05:22.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:22 smithi167 conmon[60316]: debug 2022-01-31T21:05:22.197+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:22.197953+0000) 2022-01-31T21:05:22.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:22 smithi171 conmon[46715]: debug 2022-01-31T21:05:22.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:22.225365+0000) 2022-01-31T21:05:22.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:22 smithi167 conmon[49112]: debug 2022-01-31T21:05:22.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:22.519042+0000) 2022-01-31T21:05:22.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:22 smithi167 conmon[54076]: debug 2022-01-31T21:05:22.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:22.446835+0000) 2022-01-31T21:05:22.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:22 smithi171 conmon[51620]: debug 2022-01-31T21:05:22.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:22.553380+0000) 2022-01-31T21:05:22.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:22 smithi171 conmon[41853]: debug 2022-01-31T21:05:22.679+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:22.680112+0000) 2022-01-31T21:05:23.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:23 smithi167 conmon[60316]: debug 2022-01-31T21:05:23.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:23.198157+0000) 2022-01-31T21:05:23.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:23 smithi171 conmon[46715]: debug 2022-01-31T21:05:23.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:23.225559+0000) 2022-01-31T21:05:23.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:23 smithi167 conmon[49112]: debug 2022-01-31T21:05:23.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:23.519224+0000) 2022-01-31T21:05:23.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:23 smithi167 conmon[54076]: debug 2022-01-31T21:05:23.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:23.447019+0000) 2022-01-31T21:05:23.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:23 smithi171 conmon[41853]: debug 2022-01-31T21:05:23.679+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:23.680269+0000) 2022-01-31T21:05:23.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:23 smithi171 conmon[51620]: debug 2022-01-31T21:05:23.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:23.553577+0000) 2022-01-31T21:05:23.805 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:23 smithi171 conmon[35325]: debug 2022-01-31T21:05:23.621+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:05:24.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:24 smithi167 conmon[60316]: debug 2022-01-31T21:05:24.196+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:24.198340+0000) 2022-01-31T21:05:24.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:24 smithi171 conmon[46715]: debug 2022-01-31T21:05:24.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:24.225711+0000) 2022-01-31T21:05:24.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:24 smithi167 conmon[49112]: debug 2022-01-31T21:05:24.517+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:24.519452+0000) 2022-01-31T21:05:24.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:24 smithi167 conmon[54076]: debug 2022-01-31T21:05:24.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:24.447169+0000) 2022-01-31T21:05:24.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:24 smithi171 conmon[41853]: debug 2022-01-31T21:05:24.679+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:24.680472+0000) 2022-01-31T21:05:24.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:24 smithi171 conmon[51620]: debug 2022-01-31T21:05:24.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:24.553788+0000) 2022-01-31T21:05:25.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:25 smithi167 conmon[60316]: debug 2022-01-31T21:05:25.197+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.198540+0000) 2022-01-31T21:05:25.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[46715]: debug 2022-01-31T21:05:25.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.225869+0000) 2022-01-31T21:05:25.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:25 smithi167 conmon[54076]: debug 2022-01-31T21:05:25.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.447352+0000) 2022-01-31T21:05:25.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:25 smithi167 conmon[49112]: debug 2022-01-31T21:05:25.518+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.519663+0000) 2022-01-31T21:05:25.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[41853]: debug 2022-01-31T21:05:25.680+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.680682+0000) 2022-01-31T21:05:25.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[51620]: debug 2022-01-31T21:05:25.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.553972+0000) 2022-01-31T21:05:26.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:25 smithi167 conmon[49112]: debug 2022-01-31T21:05:25.974+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.975938+0000) 2022-01-31T21:05:26.105 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:25 smithi167 conmon[54076]: debug 2022-01-31T21:05:25.972+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.973920+0000) 2022-01-31T21:05:26.105 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:25 smithi167 conmon[60316]: debug 2022-01-31T21:05:25.972+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.974296+0000) 2022-01-31T21:05:26.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[35325]: debug 2022-01-31T21:05:25.983+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102184 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:26.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[41853]: debug 2022-01-31T21:05:25.974+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.975377+0000) 2022-01-31T21:05:26.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[46715]: debug 2022-01-31T21:05:25.973+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.974023+0000) 2022-01-31T21:05:26.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:25 smithi171 conmon[51620]: debug 2022-01-31T21:05:25.973+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:25.974678+0000) 2022-01-31T21:05:26.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:26 smithi167 conmon[60316]: debug 2022-01-31T21:05:26.197+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:26.198723+0000) 2022-01-31T21:05:26.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:26 smithi171 conmon[46715]: debug 2022-01-31T21:05:26.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:26.226057+0000) 2022-01-31T21:05:26.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:26 smithi167 conmon[49112]: debug 2022-01-31T21:05:26.518+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:26.519810+0000) 2022-01-31T21:05:26.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:26 smithi167 conmon[54076]: debug 2022-01-31T21:05:26.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:26.447545+0000) 2022-01-31T21:05:26.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:26 smithi171 conmon[41853]: debug 2022-01-31T21:05:26.680+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:26.680860+0000) 2022-01-31T21:05:26.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:26 smithi171 conmon[51620]: debug 2022-01-31T21:05:26.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:26.554156+0000) 2022-01-31T21:05:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:27 smithi167 conmon[60316]: debug 2022-01-31T21:05:27.198+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:27.198923+0000) 2022-01-31T21:05:27.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:27 smithi171 conmon[46715]: debug 2022-01-31T21:05:27.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:27.226190+0000) 2022-01-31T21:05:27.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:27 smithi167 conmon[49112]: debug 2022-01-31T21:05:27.519+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:27.519983+0000) 2022-01-31T21:05:27.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:27 smithi167 conmon[54076]: debug 2022-01-31T21:05:27.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:27.447696+0000) 2022-01-31T21:05:27.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:27 smithi171 conmon[41853]: debug 2022-01-31T21:05:27.680+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:27.681025+0000) 2022-01-31T21:05:27.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:27 smithi171 conmon[51620]: debug 2022-01-31T21:05:27.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:27.554305+0000) 2022-01-31T21:05:28.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:28 smithi167 conmon[60316]: debug 2022-01-31T21:05:28.197+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:28.199104+0000) 2022-01-31T21:05:28.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:28 smithi171 conmon[46715]: debug 2022-01-31T21:05:28.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:28.226345+0000) 2022-01-31T21:05:28.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:28 smithi167 conmon[49112]: debug 2022-01-31T21:05:28.519+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:28.520137+0000) 2022-01-31T21:05:28.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:28 smithi167 conmon[54076]: debug 2022-01-31T21:05:28.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:28.447863+0000) 2022-01-31T21:05:28.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:28 smithi171 conmon[51620]: debug 2022-01-31T21:05:28.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:28.554497+0000) 2022-01-31T21:05:28.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:28 smithi171 conmon[41853]: debug 2022-01-31T21:05:28.680+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:28.681180+0000) 2022-01-31T21:05:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:29 smithi167 conmon[60316]: debug 2022-01-31T21:05:29.198+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:29.199261+0000) 2022-01-31T21:05:29.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:29 smithi171 conmon[46715]: debug 2022-01-31T21:05:29.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:29.226497+0000) 2022-01-31T21:05:29.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:29 smithi167 conmon[49112]: debug 2022-01-31T21:05:29.518+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:29.520349+0000) 2022-01-31T21:05:29.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:29 smithi167 conmon[54076]: debug 2022-01-31T21:05:29.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:29.447995+0000) 2022-01-31T21:05:29.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:29 smithi171 conmon[41853]: debug 2022-01-31T21:05:29.680+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:29.681383+0000) 2022-01-31T21:05:29.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:29 smithi171 conmon[51620]: debug 2022-01-31T21:05:29.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:29.554682+0000) 2022-01-31T21:05:30.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:30 smithi167 conmon[60316]: debug 2022-01-31T21:05:30.198+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.199452+0000) 2022-01-31T21:05:30.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[46715]: debug 2022-01-31T21:05:30.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.226654+0000) 2022-01-31T21:05:30.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:30 smithi167 conmon[49112]: debug 2022-01-31T21:05:30.519+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.520575+0000) 2022-01-31T21:05:30.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:30 smithi167 conmon[54076]: debug 2022-01-31T21:05:30.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.448176+0000) 2022-01-31T21:05:30.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[41853]: debug 2022-01-31T21:05:30.680+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.681586+0000) 2022-01-31T21:05:30.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[51620]: debug 2022-01-31T21:05:30.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.554861+0000) 2022-01-31T21:05:31.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[35325]: debug 2022-01-31T21:05:30.996+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102294 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[41853]: debug 2022-01-31T21:05:30.987+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.988072+0000) 2022-01-31T21:05:31.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[46715]: debug 2022-01-31T21:05:30.986+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.987667+0000) 2022-01-31T21:05:31.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:30 smithi171 conmon[51620]: debug 2022-01-31T21:05:30.986+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.986742+0000) 2022-01-31T21:05:31.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:30 smithi167 conmon[49112]: debug 2022-01-31T21:05:30.985+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.987466+0000) 2022-01-31T21:05:31.111 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:30 smithi167 conmon[54076]: debug 2022-01-31T21:05:30.985+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.986909+0000) 2022-01-31T21:05:31.111 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:30 smithi167 conmon[60316]: debug 2022-01-31T21:05:30.986+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:30.987726+0000) 2022-01-31T21:05:31.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:31 smithi167 conmon[60316]: debug 2022-01-31T21:05:31.198+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:31.199650+0000) 2022-01-31T21:05:31.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:31 smithi171 conmon[46715]: debug 2022-01-31T21:05:31.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:31.226810+0000) 2022-01-31T21:05:31.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:31 smithi167 conmon[49112]: debug 2022-01-31T21:05:31.520+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:31.520783+0000) 2022-01-31T21:05:31.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:31 smithi167 conmon[54076]: debug 2022-01-31T21:05:31.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:31.448375+0000) 2022-01-31T21:05:31.804 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:31 smithi171 conmon[41853]: debug 2022-01-31T21:05:31.681+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:31.681804+0000) 2022-01-31T21:05:31.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:31 smithi171 conmon[51620]: debug 2022-01-31T21:05:31.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:31.555052+0000) 2022-01-31T21:05:32.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:32 smithi167 conmon[60316]: debug 2022-01-31T21:05:32.198+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:32.199809+0000) 2022-01-31T21:05:32.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:32 smithi171 conmon[46715]: debug 2022-01-31T21:05:32.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:32.226963+0000) 2022-01-31T21:05:32.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:32 smithi167 conmon[49112]: debug 2022-01-31T21:05:32.519+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:32.520942+0000) 2022-01-31T21:05:32.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:32 smithi167 conmon[54076]: debug 2022-01-31T21:05:32.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:32.448549+0000) 2022-01-31T21:05:32.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:32 smithi171 conmon[51620]: debug 2022-01-31T21:05:32.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:32.555236+0000) 2022-01-31T21:05:32.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:32 smithi171 conmon[41853]: debug 2022-01-31T21:05:32.681+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:32.681979+0000) 2022-01-31T21:05:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:33 smithi167 conmon[60316]: debug 2022-01-31T21:05:33.198+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:33.200015+0000) 2022-01-31T21:05:33.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:33 smithi171 conmon[46715]: debug 2022-01-31T21:05:33.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:33.227193+0000) 2022-01-31T21:05:33.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:33 smithi167 conmon[49112]: debug 2022-01-31T21:05:33.520+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:33.521142+0000) 2022-01-31T21:05:33.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:33 smithi167 conmon[54076]: debug 2022-01-31T21:05:33.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:33.448732+0000) 2022-01-31T21:05:33.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:33 smithi171 conmon[41853]: debug 2022-01-31T21:05:33.681+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:33.682202+0000) 2022-01-31T21:05:33.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:33 smithi171 conmon[51620]: debug 2022-01-31T21:05:33.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:33.555420+0000) 2022-01-31T21:05:34.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:34 smithi167 conmon[60316]: debug 2022-01-31T21:05:34.199+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:34.200179+0000) 2022-01-31T21:05:34.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:34 smithi171 conmon[46715]: debug 2022-01-31T21:05:34.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:34.227401+0000) 2022-01-31T21:05:34.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:34 smithi167 conmon[49112]: debug 2022-01-31T21:05:34.519+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:34.521356+0000) 2022-01-31T21:05:34.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:34 smithi167 conmon[54076]: debug 2022-01-31T21:05:34.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:34.448886+0000) 2022-01-31T21:05:34.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:34 smithi171 conmon[51620]: debug 2022-01-31T21:05:34.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:34.555577+0000) 2022-01-31T21:05:34.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:34 smithi171 conmon[41853]: debug 2022-01-31T21:05:34.681+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:34.682424+0000) 2022-01-31T21:05:35.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:35 smithi167 conmon[60316]: debug 2022-01-31T21:05:35.199+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.200344+0000) 2022-01-31T21:05:35.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:35 smithi171 conmon[46715]: debug 2022-01-31T21:05:35.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.227587+0000) 2022-01-31T21:05:35.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:35 smithi167 conmon[49112]: debug 2022-01-31T21:05:35.520+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.521531+0000) 2022-01-31T21:05:35.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:35 smithi167 conmon[54076]: debug 2022-01-31T21:05:35.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.449050+0000) 2022-01-31T21:05:35.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:35 smithi171 conmon[51620]: debug 2022-01-31T21:05:35.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.555750+0000) 2022-01-31T21:05:35.806 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:35 smithi171 conmon[41853]: debug 2022-01-31T21:05:35.681+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.682595+0000) 2022-01-31T21:05:36.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:36 smithi171 conmon[35325]: debug 2022-01-31T21:05:36.010+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102405 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:36 smithi171 conmon[41853]: debug 2022-01-31T21:05:35.999+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.000001+0000) 2022-01-31T21:05:36.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:36 smithi171 conmon[46715]: debug 2022-01-31T21:05:36.000+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.001334+0000) 2022-01-31T21:05:36.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:35 smithi171 conmon[51620]: debug 2022-01-31T21:05:35.998+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.999572+0000) 2022-01-31T21:05:36.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:36 smithi167 conmon[49112]: debug 2022-01-31T21:05:35.999+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.001158+0000) 2022-01-31T21:05:36.117 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:35 smithi167 conmon[54076]: debug 2022-01-31T21:05:35.998+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:35.999850+0000) 2022-01-31T21:05:36.117 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:36 smithi167 conmon[60316]: debug 2022-01-31T21:05:35.998+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.000278+0000) 2022-01-31T21:05:36.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:36 smithi167 conmon[60316]: debug 2022-01-31T21:05:36.199+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.200531+0000) 2022-01-31T21:05:36.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:36 smithi171 conmon[46715]: debug 2022-01-31T21:05:36.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.227796+0000) 2022-01-31T21:05:36.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:36 smithi167 conmon[49112]: debug 2022-01-31T21:05:36.521+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.521712+0000) 2022-01-31T21:05:36.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:36 smithi167 conmon[54076]: debug 2022-01-31T21:05:36.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.449237+0000) 2022-01-31T21:05:36.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:36 smithi171 conmon[41853]: debug 2022-01-31T21:05:36.682+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.682742+0000) 2022-01-31T21:05:36.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:36 smithi171 conmon[51620]: debug 2022-01-31T21:05:36.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:36.555912+0000) 2022-01-31T21:05:37.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:37 smithi167 conmon[60316]: debug 2022-01-31T21:05:37.200+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:37.200687+0000) 2022-01-31T21:05:37.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:37 smithi171 conmon[46715]: debug 2022-01-31T21:05:37.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:37.227944+0000) 2022-01-31T21:05:37.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:37 smithi167 conmon[49112]: debug 2022-01-31T21:05:37.521+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:37.521865+0000) 2022-01-31T21:05:37.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:37 smithi167 conmon[54076]: debug 2022-01-31T21:05:37.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:37.449399+0000) 2022-01-31T21:05:37.806 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:37 smithi171 conmon[41853]: debug 2022-01-31T21:05:37.682+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:37.682928+0000) 2022-01-31T21:05:37.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:37 smithi171 conmon[51620]: debug 2022-01-31T21:05:37.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:37.556105+0000) 2022-01-31T21:05:38.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:38 smithi167 conmon[60316]: debug 2022-01-31T21:05:38.200+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:38.200872+0000) 2022-01-31T21:05:38.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:38 smithi171 conmon[46715]: debug 2022-01-31T21:05:38.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:38.228123+0000) 2022-01-31T21:05:38.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:38 smithi167 conmon[49112]: debug 2022-01-31T21:05:38.521+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:38.522082+0000) 2022-01-31T21:05:38.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:38 smithi167 conmon[54076]: debug 2022-01-31T21:05:38.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:38.449583+0000) 2022-01-31T21:05:38.806 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:38 smithi171 conmon[35325]: debug 2022-01-31T21:05:38.622+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:05:38.806 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:38 smithi171 conmon[41853]: debug 2022-01-31T21:05:38.682+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:38.683110+0000) 2022-01-31T21:05:38.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:38 smithi171 conmon[51620]: debug 2022-01-31T21:05:38.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:38.556271+0000) 2022-01-31T21:05:39.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:39 smithi167 conmon[60316]: debug 2022-01-31T21:05:39.200+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:39.201030+0000) 2022-01-31T21:05:39.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:39 smithi171 conmon[46715]: debug 2022-01-31T21:05:39.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:39.228313+0000) 2022-01-31T21:05:39.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:39 smithi167 conmon[49112]: debug 2022-01-31T21:05:39.521+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:39.522257+0000) 2022-01-31T21:05:39.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:39 smithi167 conmon[54076]: debug 2022-01-31T21:05:39.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:39.449774+0000) 2022-01-31T21:05:39.806 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:39 smithi171 conmon[41853]: debug 2022-01-31T21:05:39.682+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:39.683333+0000) 2022-01-31T21:05:39.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:39 smithi171 conmon[51620]: debug 2022-01-31T21:05:39.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:39.556435+0000) 2022-01-31T21:05:40.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:40 smithi167 conmon[60316]: debug 2022-01-31T21:05:40.200+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:40.201259+0000) 2022-01-31T21:05:40.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:40 smithi171 conmon[46715]: debug 2022-01-31T21:05:40.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:40.228489+0000) 2022-01-31T21:05:40.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:40 smithi167 conmon[49112]: debug 2022-01-31T21:05:40.521+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:40.522470+0000) 2022-01-31T21:05:40.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:40 smithi167 conmon[54076]: debug 2022-01-31T21:05:40.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:40.449939+0000) 2022-01-31T21:05:40.806 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:40 smithi171 conmon[41853]: debug 2022-01-31T21:05:40.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:40.683516+0000) 2022-01-31T21:05:40.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:40 smithi171 conmon[51620]: debug 2022-01-31T21:05:40.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:40.556633+0000) 2022-01-31T21:05:41.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:41 smithi167 conmon[49112]: debug 2022-01-31T21:05:41.014+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.014569+0000) 2022-01-31T21:05:41.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:41 smithi167 conmon[60316]: debug 2022-01-31T21:05:41.013+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.013529+0000) 2022-01-31T21:05:41.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:41 smithi167 conmon[54076]: debug 2022-01-31T21:05:41.020+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.020821+0000) 2022-01-31T21:05:41.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[35325]: debug 2022-01-31T21:05:41.026+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102515 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[41853]: debug 2022-01-31T21:05:41.012+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.012871+0000) 2022-01-31T21:05:41.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[46715]: debug 2022-01-31T21:05:41.013+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.014461+0000) 2022-01-31T21:05:41.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[51620]: debug 2022-01-31T21:05:41.012+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.013143+0000) 2022-01-31T21:05:41.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:41 smithi167 conmon[60316]: debug 2022-01-31T21:05:41.200+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.201447+0000) 2022-01-31T21:05:41.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[46715]: debug 2022-01-31T21:05:41.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.228647+0000) 2022-01-31T21:05:41.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:41 smithi167 conmon[49112]: debug 2022-01-31T21:05:41.522+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.522688+0000) 2022-01-31T21:05:41.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:41 smithi167 conmon[54076]: debug 2022-01-31T21:05:41.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.450129+0000) 2022-01-31T21:05:41.806 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[41853]: debug 2022-01-31T21:05:41.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.683700+0000) 2022-01-31T21:05:41.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:41 smithi171 conmon[51620]: debug 2022-01-31T21:05:41.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:41.556819+0000) 2022-01-31T21:05:42.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:42 smithi167 conmon[60316]: debug 2022-01-31T21:05:42.201+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:42.201649+0000) 2022-01-31T21:05:42.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:42 smithi171 conmon[46715]: debug 2022-01-31T21:05:42.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:42.228801+0000) 2022-01-31T21:05:42.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:42 smithi167 conmon[49112]: debug 2022-01-31T21:05:42.522+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:42.522843+0000) 2022-01-31T21:05:42.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:42 smithi167 conmon[54076]: debug 2022-01-31T21:05:42.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:42.450346+0000) 2022-01-31T21:05:42.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:42 smithi171 conmon[41853]: debug 2022-01-31T21:05:42.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:42.683869+0000) 2022-01-31T21:05:42.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:42 smithi171 conmon[51620]: debug 2022-01-31T21:05:42.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:42.556979+0000) 2022-01-31T21:05:43.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:43 smithi167 conmon[60316]: debug 2022-01-31T21:05:43.201+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:43.201809+0000) 2022-01-31T21:05:43.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:43 smithi171 conmon[46715]: debug 2022-01-31T21:05:43.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:43.229008+0000) 2022-01-31T21:05:43.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:43 smithi167 conmon[49112]: debug 2022-01-31T21:05:43.522+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:43.523043+0000) 2022-01-31T21:05:43.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:43 smithi167 conmon[54076]: debug 2022-01-31T21:05:43.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:43.450514+0000) 2022-01-31T21:05:43.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:43 smithi171 conmon[41853]: debug 2022-01-31T21:05:43.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:43.684020+0000) 2022-01-31T21:05:43.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:43 smithi171 conmon[51620]: debug 2022-01-31T21:05:43.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:43.557155+0000) 2022-01-31T21:05:44.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:44 smithi167 conmon[60316]: debug 2022-01-31T21:05:44.201+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:44.202022+0000) 2022-01-31T21:05:44.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:44 smithi171 conmon[46715]: debug 2022-01-31T21:05:44.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:44.229202+0000) 2022-01-31T21:05:44.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:44 smithi167 conmon[49112]: debug 2022-01-31T21:05:44.522+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:44.523229+0000) 2022-01-31T21:05:44.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:44 smithi167 conmon[54076]: debug 2022-01-31T21:05:44.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:44.450641+0000) 2022-01-31T21:05:44.863 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:44 smithi171 conmon[41853]: debug 2022-01-31T21:05:44.684+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:44.684178+0000) 2022-01-31T21:05:44.863 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:44 smithi171 conmon[51620]: debug 2022-01-31T21:05:44.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:44.557300+0000) 2022-01-31T21:05:45.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:45 smithi167 conmon[60316]: debug 2022-01-31T21:05:45.201+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:45.202221+0000) 2022-01-31T21:05:45.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:45 smithi171 conmon[46715]: debug 2022-01-31T21:05:45.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:45.229342+0000) 2022-01-31T21:05:45.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:45 smithi167 conmon[49112]: debug 2022-01-31T21:05:45.523+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:45.523435+0000) 2022-01-31T21:05:45.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:45 smithi167 conmon[54076]: debug 2022-01-31T21:05:45.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:45.450787+0000) 2022-01-31T21:05:45.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:45 smithi171 conmon[41853]: debug 2022-01-31T21:05:45.684+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:45.684329+0000) 2022-01-31T21:05:45.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:45 smithi171 conmon[51620]: debug 2022-01-31T21:05:45.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:45.557456+0000) 2022-01-31T21:05:46.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[35325]: debug 2022-01-31T21:05:46.039+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102625 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[41853]: debug 2022-01-31T21:05:46.028+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.029231+0000) 2022-01-31T21:05:46.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[46715]: debug 2022-01-31T21:05:46.029+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.030197+0000) 2022-01-31T21:05:46.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[51620]: debug 2022-01-31T21:05:46.029+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.029986+0000) 2022-01-31T21:05:46.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:46 smithi167 conmon[49112]: debug 2022-01-31T21:05:46.030+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.031188+0000) 2022-01-31T21:05:46.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:46 smithi167 conmon[54076]: debug 2022-01-31T21:05:46.030+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.030975+0000) 2022-01-31T21:05:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:46 smithi167 conmon[60316]: debug 2022-01-31T21:05:46.030+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.030841+0000) 2022-01-31T21:05:46.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:46 smithi167 conmon[60316]: debug 2022-01-31T21:05:46.201+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.202428+0000) 2022-01-31T21:05:46.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[46715]: debug 2022-01-31T21:05:46.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.229545+0000) 2022-01-31T21:05:46.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:46 smithi167 conmon[49112]: debug 2022-01-31T21:05:46.523+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.523622+0000) 2022-01-31T21:05:46.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:46 smithi167 conmon[54076]: debug 2022-01-31T21:05:46.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.450981+0000) 2022-01-31T21:05:46.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[51620]: debug 2022-01-31T21:05:46.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.557661+0000) 2022-01-31T21:05:46.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:46 smithi171 conmon[41853]: debug 2022-01-31T21:05:46.684+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:46.684520+0000) 2022-01-31T21:05:47.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:47 smithi167 conmon[60316]: debug 2022-01-31T21:05:47.202+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:47.202581+0000) 2022-01-31T21:05:47.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:47 smithi171 conmon[46715]: debug 2022-01-31T21:05:47.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:47.229695+0000) 2022-01-31T21:05:47.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:47 smithi167 conmon[49112]: debug 2022-01-31T21:05:47.523+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:47.523757+0000) 2022-01-31T21:05:47.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:47 smithi167 conmon[54076]: debug 2022-01-31T21:05:47.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:47.451148+0000) 2022-01-31T21:05:47.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:47 smithi171 conmon[51620]: debug 2022-01-31T21:05:47.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:47.557813+0000) 2022-01-31T21:05:47.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:47 smithi171 conmon[41853]: debug 2022-01-31T21:05:47.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:47.684687+0000) 2022-01-31T21:05:48.220 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T21:05:48.220+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T21:05:48.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:48 smithi167 conmon[60316]: debug 2022-01-31T21:05:48.202+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:48.202768+0000) 2022-01-31T21:05:48.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:48 smithi171 conmon[46715]: debug 2022-01-31T21:05:48.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:48.229886+0000) 2022-01-31T21:05:48.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:48 smithi167 conmon[49112]: debug 2022-01-31T21:05:48.523+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:48.523939+0000) 2022-01-31T21:05:48.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:48 smithi167 conmon[54076]: debug 2022-01-31T21:05:48.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:48.451372+0000) 2022-01-31T21:05:48.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:48 smithi171 conmon[51620]: debug 2022-01-31T21:05:48.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:48.558015+0000) 2022-01-31T21:05:48.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:48 smithi171 conmon[41853]: debug 2022-01-31T21:05:48.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:48.684939+0000) 2022-01-31T21:05:49.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:49 smithi167 conmon[60316]: debug 2022-01-31T21:05:49.202+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:49.202955+0000) 2022-01-31T21:05:49.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:49 smithi171 conmon[46715]: debug 2022-01-31T21:05:49.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:49.230064+0000) 2022-01-31T21:05:49.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:49 smithi167 conmon[49112]: debug 2022-01-31T21:05:49.523+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:49.524154+0000) 2022-01-31T21:05:49.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:49 smithi167 conmon[54076]: debug 2022-01-31T21:05:49.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:49.451516+0000) 2022-01-31T21:05:49.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:49 smithi171 conmon[51620]: debug 2022-01-31T21:05:49.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:49.558202+0000) 2022-01-31T21:05:49.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:49 smithi171 conmon[41853]: debug 2022-01-31T21:05:49.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:49.685064+0000) 2022-01-31T21:05:50.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:50 smithi167 conmon[60316]: debug 2022-01-31T21:05:50.202+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:50.203161+0000) 2022-01-31T21:05:50.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:50 smithi171 conmon[46715]: debug 2022-01-31T21:05:50.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:50.230261+0000) 2022-01-31T21:05:50.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:50 smithi167 conmon[49112]: debug 2022-01-31T21:05:50.523+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:50.524368+0000) 2022-01-31T21:05:50.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:50 smithi167 conmon[54076]: debug 2022-01-31T21:05:50.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:50.451643+0000) 2022-01-31T21:05:50.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:50 smithi171 conmon[51620]: debug 2022-01-31T21:05:50.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:50.558351+0000) 2022-01-31T21:05:50.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:50 smithi171 conmon[41853]: debug 2022-01-31T21:05:50.683+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:50.685265+0000) 2022-01-31T21:05:51.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[35325]: debug 2022-01-31T21:05:51.051+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102735 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[41853]: debug 2022-01-31T21:05:51.040+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.042462+0000) 2022-01-31T21:05:51.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[46715]: debug 2022-01-31T21:05:51.041+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.042871+0000) 2022-01-31T21:05:51.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[51620]: debug 2022-01-31T21:05:51.040+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.042161+0000) 2022-01-31T21:05:51.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:51 smithi167 conmon[49112]: debug 2022-01-31T21:05:51.043+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.043714+0000) 2022-01-31T21:05:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:51 smithi167 conmon[54076]: debug 2022-01-31T21:05:51.043+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.044123+0000) 2022-01-31T21:05:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:51 smithi167 conmon[60316]: debug 2022-01-31T21:05:51.043+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.044236+0000) 2022-01-31T21:05:51.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:51 smithi167 conmon[60316]: debug 2022-01-31T21:05:51.202+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.203324+0000) 2022-01-31T21:05:51.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[46715]: debug 2022-01-31T21:05:51.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.230441+0000) 2022-01-31T21:05:51.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:51 smithi167 conmon[49112]: debug 2022-01-31T21:05:51.524+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.524579+0000) 2022-01-31T21:05:51.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:51 smithi167 conmon[54076]: debug 2022-01-31T21:05:51.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.451873+0000) 2022-01-31T21:05:51.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[41853]: debug 2022-01-31T21:05:51.684+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.685427+0000) 2022-01-31T21:05:51.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:51 smithi171 conmon[51620]: debug 2022-01-31T21:05:51.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:51.558516+0000) 2022-01-31T21:05:52.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:52 smithi167 conmon[60316]: debug 2022-01-31T21:05:52.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:52.203466+0000) 2022-01-31T21:05:52.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:52 smithi171 conmon[46715]: debug 2022-01-31T21:05:52.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:52.230592+0000) 2022-01-31T21:05:52.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:52 smithi167 conmon[49112]: debug 2022-01-31T21:05:52.524+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:52.524724+0000) 2022-01-31T21:05:52.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:52 smithi167 conmon[54076]: debug 2022-01-31T21:05:52.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:52.452051+0000) 2022-01-31T21:05:52.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:52 smithi171 conmon[41853]: debug 2022-01-31T21:05:52.684+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:52.685608+0000) 2022-01-31T21:05:52.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:52 smithi171 conmon[51620]: debug 2022-01-31T21:05:52.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:52.558668+0000) 2022-01-31T21:05:53.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:53 smithi167 conmon[60316]: debug 2022-01-31T21:05:53.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:53.203679+0000) 2022-01-31T21:05:53.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:53 smithi171 conmon[46715]: debug 2022-01-31T21:05:53.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:53.230792+0000) 2022-01-31T21:05:53.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:53 smithi167 conmon[49112]: debug 2022-01-31T21:05:53.524+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:53.524913+0000) 2022-01-31T21:05:53.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:53 smithi167 conmon[54076]: debug 2022-01-31T21:05:53.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:53.452234+0000) 2022-01-31T21:05:53.809 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:53 smithi171 conmon[35325]: debug 2022-01-31T21:05:53.622+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:05:53.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:53 smithi171 conmon[51620]: debug 2022-01-31T21:05:53.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:53.558853+0000) 2022-01-31T21:05:53.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:53 smithi171 conmon[41853]: debug 2022-01-31T21:05:53.684+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:53.685816+0000) 2022-01-31T21:05:54.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:54 smithi167 conmon[60316]: debug 2022-01-31T21:05:54.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:54.203842+0000) 2022-01-31T21:05:54.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:54 smithi171 conmon[46715]: debug 2022-01-31T21:05:54.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:54.231014+0000) 2022-01-31T21:05:54.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:54 smithi167 conmon[49112]: debug 2022-01-31T21:05:54.524+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:54.525141+0000) 2022-01-31T21:05:54.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:54 smithi167 conmon[54076]: debug 2022-01-31T21:05:54.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:54.452448+0000) 2022-01-31T21:05:54.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:54 smithi171 conmon[51620]: debug 2022-01-31T21:05:54.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:54.559038+0000) 2022-01-31T21:05:54.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:54 smithi171 conmon[41853]: debug 2022-01-31T21:05:54.685+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:54.685998+0000) 2022-01-31T21:05:55.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:55 smithi167 conmon[60316]: debug 2022-01-31T21:05:55.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:55.204036+0000) 2022-01-31T21:05:55.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:55 smithi171 conmon[46715]: debug 2022-01-31T21:05:55.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:55.231195+0000) 2022-01-31T21:05:55.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:55 smithi167 conmon[49112]: debug 2022-01-31T21:05:55.524+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:55.525351+0000) 2022-01-31T21:05:55.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:55 smithi167 conmon[54076]: debug 2022-01-31T21:05:55.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:55.452658+0000) 2022-01-31T21:05:55.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:55 smithi171 conmon[41853]: debug 2022-01-31T21:05:55.685+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:55.686195+0000) 2022-01-31T21:05:55.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:55 smithi171 conmon[51620]: debug 2022-01-31T21:05:55.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:55.559204+0000) 2022-01-31T21:05:56.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[41853]: debug 2022-01-31T21:05:56.053+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.055121+0000) 2022-01-31T21:05:56.065 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[46715]: debug 2022-01-31T21:05:56.054+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.055782+0000) 2022-01-31T21:05:56.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[51620]: debug 2022-01-31T21:05:56.053+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.054956+0000) 2022-01-31T21:05:56.337 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[35325]: debug 2022-01-31T21:05:56.064+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102845 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:05:56.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[46715]: debug 2022-01-31T21:05:56.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.231353+0000) 2022-01-31T21:05:56.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:56 smithi167 conmon[49112]: debug 2022-01-31T21:05:56.056+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.056828+0000) 2022-01-31T21:05:56.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:56 smithi167 conmon[54076]: debug 2022-01-31T21:05:56.056+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.056970+0000) 2022-01-31T21:05:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:56 smithi167 conmon[60316]: debug 2022-01-31T21:05:56.056+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.056734+0000) 2022-01-31T21:05:56.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:56 smithi167 conmon[60316]: debug 2022-01-31T21:05:56.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.204241+0000) 2022-01-31T21:05:56.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[51620]: debug 2022-01-31T21:05:56.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.559359+0000) 2022-01-31T21:05:56.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:56 smithi167 conmon[49112]: debug 2022-01-31T21:05:56.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.525539+0000) 2022-01-31T21:05:56.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:56 smithi167 conmon[54076]: debug 2022-01-31T21:05:56.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.452860+0000) 2022-01-31T21:05:57.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:56 smithi171 conmon[41853]: debug 2022-01-31T21:05:56.685+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:56.686382+0000) 2022-01-31T21:05:57.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:57 smithi167 conmon[60316]: debug 2022-01-31T21:05:57.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:57.204384+0000) 2022-01-31T21:05:57.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:57 smithi171 conmon[46715]: debug 2022-01-31T21:05:57.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:57.231513+0000) 2022-01-31T21:05:57.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:57 smithi167 conmon[49112]: debug 2022-01-31T21:05:57.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:57.525657+0000) 2022-01-31T21:05:57.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:57 smithi167 conmon[54076]: debug 2022-01-31T21:05:57.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:57.453007+0000) 2022-01-31T21:05:57.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:57 smithi171 conmon[41853]: debug 2022-01-31T21:05:57.685+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:57.686507+0000) 2022-01-31T21:05:57.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:57 smithi171 conmon[51620]: debug 2022-01-31T21:05:57.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:57.559547+0000) 2022-01-31T21:05:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:58 smithi167 conmon[60316]: debug 2022-01-31T21:05:58.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:58.204574+0000) 2022-01-31T21:05:58.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:58 smithi171 conmon[46715]: debug 2022-01-31T21:05:58.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:58.231719+0000) 2022-01-31T21:05:58.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:58 smithi167 conmon[49112]: debug 2022-01-31T21:05:58.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:58.525860+0000) 2022-01-31T21:05:58.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:58 smithi167 conmon[54076]: debug 2022-01-31T21:05:58.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:58.453167+0000) 2022-01-31T21:05:58.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:58 smithi171 conmon[41853]: debug 2022-01-31T21:05:58.685+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:58.686686+0000) 2022-01-31T21:05:58.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:58 smithi171 conmon[51620]: debug 2022-01-31T21:05:58.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:58.559779+0000) 2022-01-31T21:05:59.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:05:59 smithi167 conmon[60316]: debug 2022-01-31T21:05:59.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:59.204756+0000) 2022-01-31T21:05:59.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:05:59 smithi171 conmon[46715]: debug 2022-01-31T21:05:59.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:59.231949+0000) 2022-01-31T21:05:59.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:05:59 smithi167 conmon[49112]: debug 2022-01-31T21:05:59.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:59.526055+0000) 2022-01-31T21:05:59.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:05:59 smithi167 conmon[54076]: debug 2022-01-31T21:05:59.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:59.453340+0000) 2022-01-31T21:05:59.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:05:59 smithi171 conmon[41853]: debug 2022-01-31T21:05:59.686+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:59.686851+0000) 2022-01-31T21:05:59.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:05:59 smithi171 conmon[51620]: debug 2022-01-31T21:05:59.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:05:59.559997+0000) 2022-01-31T21:06:00.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:00 smithi167 conmon[60316]: debug 2022-01-31T21:06:00.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:00.204960+0000) 2022-01-31T21:06:00.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:00 smithi171 conmon[46715]: debug 2022-01-31T21:06:00.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:00.232106+0000) 2022-01-31T21:06:00.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:00 smithi167 conmon[49112]: debug 2022-01-31T21:06:00.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:00.526193+0000) 2022-01-31T21:06:00.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:00 smithi167 conmon[54076]: debug 2022-01-31T21:06:00.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:00.453473+0000) 2022-01-31T21:06:00.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:00 smithi171 conmon[41853]: debug 2022-01-31T21:06:00.686+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:00.686968+0000) 2022-01-31T21:06:00.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:00 smithi171 conmon[51620]: debug 2022-01-31T21:06:00.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:00.560184+0000) 2022-01-31T21:06:01.337 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[35325]: debug 2022-01-31T21:06:01.078+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 102955 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:01.338 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[41853]: debug 2022-01-31T21:06:01.066+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.067661+0000) 2022-01-31T21:06:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[46715]: debug 2022-01-31T21:06:01.068+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.069104+0000) 2022-01-31T21:06:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[46715]: debug 2022-01-31T21:06:01.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.232232+0000) 2022-01-31T21:06:01.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[51620]: debug 2022-01-31T21:06:01.067+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.068637+0000) 2022-01-31T21:06:01.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:01 smithi167 conmon[49112]: debug 2022-01-31T21:06:01.067+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.068430+0000) 2022-01-31T21:06:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:01 smithi167 conmon[54076]: debug 2022-01-31T21:06:01.069+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.069686+0000) 2022-01-31T21:06:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:01 smithi167 conmon[60316]: debug 2022-01-31T21:06:01.068+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.069361+0000) 2022-01-31T21:06:01.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:01 smithi167 conmon[60316]: debug 2022-01-31T21:06:01.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.205119+0000) 2022-01-31T21:06:01.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[51620]: debug 2022-01-31T21:06:01.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.560358+0000) 2022-01-31T21:06:01.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:01 smithi167 conmon[54076]: debug 2022-01-31T21:06:01.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.453635+0000) 2022-01-31T21:06:01.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:01 smithi167 conmon[49112]: debug 2022-01-31T21:06:01.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.526356+0000) 2022-01-31T21:06:02.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:01 smithi171 conmon[41853]: debug 2022-01-31T21:06:01.686+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:01.687149+0000) 2022-01-31T21:06:02.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:02 smithi167 conmon[60316]: debug 2022-01-31T21:06:02.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:02.205263+0000) 2022-01-31T21:06:02.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:02 smithi171 conmon[46715]: debug 2022-01-31T21:06:02.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:02.232374+0000) 2022-01-31T21:06:02.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:02 smithi167 conmon[49112]: debug 2022-01-31T21:06:02.526+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:02.526537+0000) 2022-01-31T21:06:02.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:02 smithi167 conmon[54076]: debug 2022-01-31T21:06:02.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:02.453747+0000) 2022-01-31T21:06:02.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:02 smithi171 conmon[41853]: debug 2022-01-31T21:06:02.686+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:02.687328+0000) 2022-01-31T21:06:02.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:02 smithi171 conmon[51620]: debug 2022-01-31T21:06:02.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:02.560515+0000) 2022-01-31T21:06:03.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:03 smithi167 conmon[60316]: debug 2022-01-31T21:06:03.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:03.205468+0000) 2022-01-31T21:06:03.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:03 smithi171 conmon[46715]: debug 2022-01-31T21:06:03.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:03.232583+0000) 2022-01-31T21:06:03.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:03 smithi167 conmon[49112]: debug 2022-01-31T21:06:03.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:03.526750+0000) 2022-01-31T21:06:03.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:03 smithi167 conmon[54076]: debug 2022-01-31T21:06:03.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:03.453933+0000) 2022-01-31T21:06:03.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:03 smithi171 conmon[41853]: debug 2022-01-31T21:06:03.686+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:03.687487+0000) 2022-01-31T21:06:03.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:03 smithi171 conmon[51620]: debug 2022-01-31T21:06:03.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:03.560704+0000) 2022-01-31T21:06:04.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:04 smithi167 conmon[60316]: debug 2022-01-31T21:06:04.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:04.205678+0000) 2022-01-31T21:06:04.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:04 smithi171 conmon[46715]: debug 2022-01-31T21:06:04.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:04.232768+0000) 2022-01-31T21:06:04.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:04 smithi167 conmon[49112]: debug 2022-01-31T21:06:04.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:04.526955+0000) 2022-01-31T21:06:04.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:04 smithi167 conmon[54076]: debug 2022-01-31T21:06:04.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:04.454138+0000) 2022-01-31T21:06:04.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:04 smithi171 conmon[51620]: debug 2022-01-31T21:06:04.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:04.560905+0000) 2022-01-31T21:06:04.811 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:04 smithi171 conmon[41853]: debug 2022-01-31T21:06:04.686+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:04.687638+0000) 2022-01-31T21:06:05.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:05 smithi167 conmon[60316]: debug 2022-01-31T21:06:05.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:05.205863+0000) 2022-01-31T21:06:05.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:05 smithi171 conmon[46715]: debug 2022-01-31T21:06:05.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:05.232949+0000) 2022-01-31T21:06:05.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:05 smithi167 conmon[49112]: debug 2022-01-31T21:06:05.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:05.527132+0000) 2022-01-31T21:06:05.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:05 smithi167 conmon[54076]: debug 2022-01-31T21:06:05.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:05.454368+0000) 2022-01-31T21:06:05.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:05 smithi171 conmon[41853]: debug 2022-01-31T21:06:05.687+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:05.687823+0000) 2022-01-31T21:06:05.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:05 smithi171 conmon[51620]: debug 2022-01-31T21:06:05.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:05.561085+0000) 2022-01-31T21:06:06.338 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[35325]: debug 2022-01-31T21:06:06.091+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103066 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:06.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[41853]: debug 2022-01-31T21:06:06.081+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.081799+0000) 2022-01-31T21:06:06.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[51620]: debug 2022-01-31T21:06:06.081+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.082030+0000) 2022-01-31T21:06:06.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[46715]: debug 2022-01-31T21:06:06.081+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.082469+0000) 2022-01-31T21:06:06.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[46715]: debug 2022-01-31T21:06:06.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.233115+0000) 2022-01-31T21:06:06.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:06 smithi167 conmon[49112]: debug 2022-01-31T21:06:06.081+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.083331+0000) 2022-01-31T21:06:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:06 smithi167 conmon[54076]: debug 2022-01-31T21:06:06.081+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.083299+0000) 2022-01-31T21:06:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:06 smithi167 conmon[60316]: debug 2022-01-31T21:06:06.082+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.083637+0000) 2022-01-31T21:06:06.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:06 smithi167 conmon[60316]: debug 2022-01-31T21:06:06.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.206073+0000) 2022-01-31T21:06:06.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[51620]: debug 2022-01-31T21:06:06.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.561263+0000) 2022-01-31T21:06:06.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:06 smithi167 conmon[49112]: debug 2022-01-31T21:06:06.525+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.527328+0000) 2022-01-31T21:06:06.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:06 smithi167 conmon[54076]: debug 2022-01-31T21:06:06.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.454538+0000) 2022-01-31T21:06:07.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:06 smithi171 conmon[41853]: debug 2022-01-31T21:06:06.687+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:06.688035+0000) 2022-01-31T21:06:07.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:07 smithi167 conmon[60316]: debug 2022-01-31T21:06:07.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:07.206245+0000) 2022-01-31T21:06:07.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:07 smithi171 conmon[46715]: debug 2022-01-31T21:06:07.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:07.233253+0000) 2022-01-31T21:06:07.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:07 smithi167 conmon[49112]: debug 2022-01-31T21:06:07.526+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:07.527487+0000) 2022-01-31T21:06:07.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:07 smithi167 conmon[54076]: debug 2022-01-31T21:06:07.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:07.454698+0000) 2022-01-31T21:06:07.811 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:07 smithi171 conmon[41853]: debug 2022-01-31T21:06:07.687+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:07.688201+0000) 2022-01-31T21:06:07.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:07 smithi171 conmon[51620]: debug 2022-01-31T21:06:07.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:07.561397+0000) 2022-01-31T21:06:08.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:08 smithi167 conmon[60316]: debug 2022-01-31T21:06:08.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:08.206406+0000) 2022-01-31T21:06:08.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:08 smithi171 conmon[46715]: debug 2022-01-31T21:06:08.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:08.233341+0000) 2022-01-31T21:06:08.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:08 smithi167 conmon[49112]: debug 2022-01-31T21:06:08.526+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:08.527654+0000) 2022-01-31T21:06:08.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:08 smithi167 conmon[54076]: debug 2022-01-31T21:06:08.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:08.454883+0000) 2022-01-31T21:06:08.811 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:08 smithi171 conmon[35325]: debug 2022-01-31T21:06:08.623+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:06:08.811 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:08 smithi171 conmon[41853]: debug 2022-01-31T21:06:08.687+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:08.688382+0000) 2022-01-31T21:06:08.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:08 smithi171 conmon[51620]: debug 2022-01-31T21:06:08.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:08.561613+0000) 2022-01-31T21:06:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:09 smithi167 conmon[60316]: debug 2022-01-31T21:06:09.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:09.206613+0000) 2022-01-31T21:06:09.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:09 smithi171 conmon[46715]: debug 2022-01-31T21:06:09.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:09.233497+0000) 2022-01-31T21:06:09.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:09 smithi167 conmon[49112]: debug 2022-01-31T21:06:09.526+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:09.527838+0000) 2022-01-31T21:06:09.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:09 smithi167 conmon[54076]: debug 2022-01-31T21:06:09.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:09.455091+0000) 2022-01-31T21:06:09.811 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:09 smithi171 conmon[41853]: debug 2022-01-31T21:06:09.687+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:09.688518+0000) 2022-01-31T21:06:09.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:09 smithi171 conmon[51620]: debug 2022-01-31T21:06:09.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:09.561817+0000) 2022-01-31T21:06:10.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:10 smithi167 conmon[60316]: debug 2022-01-31T21:06:10.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:10.206783+0000) 2022-01-31T21:06:10.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:10 smithi171 conmon[46715]: debug 2022-01-31T21:06:10.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:10.233715+0000) 2022-01-31T21:06:10.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:10 smithi167 conmon[54076]: debug 2022-01-31T21:06:10.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:10.455278+0000) 2022-01-31T21:06:10.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:10 smithi167 conmon[49112]: debug 2022-01-31T21:06:10.526+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:10.528013+0000) 2022-01-31T21:06:10.811 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:10 smithi171 conmon[41853]: debug 2022-01-31T21:06:10.687+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:10.688656+0000) 2022-01-31T21:06:10.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:10 smithi171 conmon[51620]: debug 2022-01-31T21:06:10.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:10.561969+0000) 2022-01-31T21:06:11.356 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:11 smithi167 conmon[49112]: debug 2022-01-31T21:06:11.094+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.095526+0000) 2022-01-31T21:06:11.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[35325]: debug 2022-01-31T21:06:11.104+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103176 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:11.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:11 smithi167 conmon[54076]: debug 2022-01-31T21:06:11.094+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.095835+0000) 2022-01-31T21:06:11.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[41853]: debug 2022-01-31T21:06:11.095+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.096016+0000) 2022-01-31T21:06:11.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:11 smithi167 conmon[60316]: debug 2022-01-31T21:06:11.094+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.096206+0000) 2022-01-31T21:06:11.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:11 smithi167 conmon[60316]: debug 2022-01-31T21:06:11.206+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.206937+0000) 2022-01-31T21:06:11.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[51620]: debug 2022-01-31T21:06:11.094+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.095632+0000) 2022-01-31T21:06:11.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[46715]: debug 2022-01-31T21:06:11.095+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.095952+0000) 2022-01-31T21:06:11.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[46715]: debug 2022-01-31T21:06:11.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.233925+0000) 2022-01-31T21:06:11.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:11 smithi167 conmon[49112]: debug 2022-01-31T21:06:11.526+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.528211+0000) 2022-01-31T21:06:11.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:11 smithi167 conmon[54076]: debug 2022-01-31T21:06:11.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.455418+0000) 2022-01-31T21:06:11.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[51620]: debug 2022-01-31T21:06:11.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.562171+0000) 2022-01-31T21:06:11.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:11 smithi171 conmon[41853]: debug 2022-01-31T21:06:11.688+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:11.688856+0000) 2022-01-31T21:06:12.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:12 smithi167 conmon[60316]: debug 2022-01-31T21:06:12.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:12.207099+0000) 2022-01-31T21:06:12.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:12 smithi171 conmon[46715]: debug 2022-01-31T21:06:12.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:12.234083+0000) 2022-01-31T21:06:12.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:12 smithi167 conmon[49112]: debug 2022-01-31T21:06:12.527+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:12.528366+0000) 2022-01-31T21:06:12.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:12 smithi167 conmon[54076]: debug 2022-01-31T21:06:12.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:12.455564+0000) 2022-01-31T21:06:12.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:12 smithi171 conmon[51620]: debug 2022-01-31T21:06:12.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:12.562324+0000) 2022-01-31T21:06:12.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:12 smithi171 conmon[41853]: debug 2022-01-31T21:06:12.688+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:12.689040+0000) 2022-01-31T21:06:13.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:13 smithi167 conmon[60316]: debug 2022-01-31T21:06:13.206+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:13.207281+0000) 2022-01-31T21:06:13.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:13 smithi171 conmon[46715]: debug 2022-01-31T21:06:13.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:13.234314+0000) 2022-01-31T21:06:13.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:13 smithi167 conmon[49112]: debug 2022-01-31T21:06:13.527+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:13.528548+0000) 2022-01-31T21:06:13.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:13 smithi167 conmon[54076]: debug 2022-01-31T21:06:13.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:13.455781+0000) 2022-01-31T21:06:13.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:13 smithi171 conmon[51620]: debug 2022-01-31T21:06:13.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:13.562474+0000) 2022-01-31T21:06:13.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:13 smithi171 conmon[41853]: debug 2022-01-31T21:06:13.688+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:13.689197+0000) 2022-01-31T21:06:14.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:14 smithi167 conmon[60316]: debug 2022-01-31T21:06:14.206+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:14.207410+0000) 2022-01-31T21:06:14.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:14 smithi171 conmon[46715]: debug 2022-01-31T21:06:14.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:14.234465+0000) 2022-01-31T21:06:14.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:14 smithi167 conmon[49112]: debug 2022-01-31T21:06:14.527+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:14.528743+0000) 2022-01-31T21:06:14.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:14 smithi167 conmon[54076]: debug 2022-01-31T21:06:14.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:14.455937+0000) 2022-01-31T21:06:14.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:14 smithi171 conmon[41853]: debug 2022-01-31T21:06:14.688+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:14.689351+0000) 2022-01-31T21:06:14.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:14 smithi171 conmon[51620]: debug 2022-01-31T21:06:14.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:14.562626+0000) 2022-01-31T21:06:15.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:15 smithi167 conmon[60316]: debug 2022-01-31T21:06:15.206+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:15.207591+0000) 2022-01-31T21:06:15.617 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:15 smithi171 conmon[46715]: debug 2022-01-31T21:06:15.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:15.234664+0000) 2022-01-31T21:06:15.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:15 smithi167 conmon[49112]: debug 2022-01-31T21:06:15.527+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:15.528889+0000) 2022-01-31T21:06:15.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:15 smithi167 conmon[54076]: debug 2022-01-31T21:06:15.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:15.456124+0000) 2022-01-31T21:06:15.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:15 smithi171 conmon[51620]: debug 2022-01-31T21:06:15.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:15.562806+0000) 2022-01-31T21:06:15.813 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:15 smithi171 conmon[41853]: debug 2022-01-31T21:06:15.689+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:15.689555+0000) 2022-01-31T21:06:16.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[35325]: debug 2022-01-31T21:06:16.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103287 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:16.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[41853]: debug 2022-01-31T21:06:16.107+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.108545+0000) 2022-01-31T21:06:16.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[51620]: debug 2022-01-31T21:06:16.106+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.107304+0000) 2022-01-31T21:06:16.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[46715]: debug 2022-01-31T21:06:16.108+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.108847+0000) 2022-01-31T21:06:16.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[46715]: debug 2022-01-31T21:06:16.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.234826+0000) 2022-01-31T21:06:16.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:16 smithi167 conmon[49112]: debug 2022-01-31T21:06:16.106+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.107938+0000) 2022-01-31T21:06:16.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:16 smithi167 conmon[54076]: debug 2022-01-31T21:06:16.106+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.108414+0000) 2022-01-31T21:06:16.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:16 smithi167 conmon[60316]: debug 2022-01-31T21:06:16.106+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.107566+0000) 2022-01-31T21:06:16.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:16 smithi167 conmon[60316]: debug 2022-01-31T21:06:16.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.207782+0000) 2022-01-31T21:06:16.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:16 smithi167 conmon[49112]: debug 2022-01-31T21:06:16.528+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.529037+0000) 2022-01-31T21:06:16.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:16 smithi167 conmon[54076]: debug 2022-01-31T21:06:16.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.456333+0000) 2022-01-31T21:06:16.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[41853]: debug 2022-01-31T21:06:16.688+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.689712+0000) 2022-01-31T21:06:16.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:16 smithi171 conmon[51620]: debug 2022-01-31T21:06:16.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:16.562966+0000) 2022-01-31T21:06:17.433 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:17 smithi167 conmon[60316]: debug 2022-01-31T21:06:17.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:17.207955+0000) 2022-01-31T21:06:17.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:17 smithi171 conmon[46715]: debug 2022-01-31T21:06:17.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:17.234962+0000) 2022-01-31T21:06:17.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:17 smithi167 conmon[49112]: debug 2022-01-31T21:06:17.528+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:17.529178+0000) 2022-01-31T21:06:17.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:17 smithi167 conmon[54076]: debug 2022-01-31T21:06:17.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:17.456515+0000) 2022-01-31T21:06:17.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:17 smithi171 conmon[41853]: debug 2022-01-31T21:06:17.689+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:17.689859+0000) 2022-01-31T21:06:17.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:17 smithi171 conmon[51620]: debug 2022-01-31T21:06:17.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:17.563119+0000) 2022-01-31T21:06:18.433 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:18 smithi167 conmon[60316]: debug 2022-01-31T21:06:18.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:18.208139+0000) 2022-01-31T21:06:18.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:18 smithi171 conmon[46715]: debug 2022-01-31T21:06:18.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:18.235149+0000) 2022-01-31T21:06:18.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:18 smithi167 conmon[49112]: debug 2022-01-31T21:06:18.528+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:18.529333+0000) 2022-01-31T21:06:18.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:18 smithi167 conmon[54076]: debug 2022-01-31T21:06:18.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:18.456719+0000) 2022-01-31T21:06:18.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:18 smithi171 conmon[51620]: debug 2022-01-31T21:06:18.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:18.563302+0000) 2022-01-31T21:06:18.813 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:18 smithi171 conmon[41853]: debug 2022-01-31T21:06:18.689+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:18.690009+0000) 2022-01-31T21:06:19.433 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:19 smithi167 conmon[60316]: debug 2022-01-31T21:06:19.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:19.208334+0000) 2022-01-31T21:06:19.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:19 smithi171 conmon[46715]: debug 2022-01-31T21:06:19.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:19.235283+0000) 2022-01-31T21:06:19.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:19 smithi167 conmon[49112]: debug 2022-01-31T21:06:19.528+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:19.529529+0000) 2022-01-31T21:06:19.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:19 smithi167 conmon[54076]: debug 2022-01-31T21:06:19.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:19.456880+0000) 2022-01-31T21:06:19.813 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:19 smithi171 conmon[41853]: debug 2022-01-31T21:06:19.689+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:19.690157+0000) 2022-01-31T21:06:19.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:19 smithi171 conmon[51620]: debug 2022-01-31T21:06:19.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:19.563482+0000) 2022-01-31T21:06:20.433 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:20 smithi167 conmon[60316]: debug 2022-01-31T21:06:20.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:20.208524+0000) 2022-01-31T21:06:20.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:20 smithi171 conmon[46715]: debug 2022-01-31T21:06:20.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:20.235445+0000) 2022-01-31T21:06:20.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:20 smithi167 conmon[49112]: debug 2022-01-31T21:06:20.528+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:20.529670+0000) 2022-01-31T21:06:20.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:20 smithi167 conmon[54076]: debug 2022-01-31T21:06:20.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:20.457067+0000) 2022-01-31T21:06:20.813 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:20 smithi171 conmon[41853]: debug 2022-01-31T21:06:20.689+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:20.690330+0000) 2022-01-31T21:06:20.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:20 smithi171 conmon[51620]: debug 2022-01-31T21:06:20.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:20.563632+0000) 2022-01-31T21:06:21.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:21 smithi167 conmon[49112]: debug 2022-01-31T21:06:21.120+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.122026+0000) 2022-01-31T21:06:21.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:21 smithi167 conmon[54076]: debug 2022-01-31T21:06:21.120+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.122437+0000) 2022-01-31T21:06:21.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:21 smithi167 conmon[60316]: debug 2022-01-31T21:06:21.119+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.121082+0000) 2022-01-31T21:06:21.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:21 smithi167 conmon[60316]: debug 2022-01-31T21:06:21.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.208685+0000) 2022-01-31T21:06:21.562 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[35325]: debug 2022-01-31T21:06:21.131+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103397 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:21.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[41853]: debug 2022-01-31T21:06:21.120+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.121367+0000) 2022-01-31T21:06:21.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[51620]: debug 2022-01-31T21:06:21.119+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.120669+0000) 2022-01-31T21:06:21.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[46715]: debug 2022-01-31T21:06:21.120+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.121412+0000) 2022-01-31T21:06:21.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[46715]: debug 2022-01-31T21:06:21.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.235659+0000) 2022-01-31T21:06:21.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:21 smithi167 conmon[49112]: debug 2022-01-31T21:06:21.529+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.529871+0000) 2022-01-31T21:06:21.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:21 smithi167 conmon[54076]: debug 2022-01-31T21:06:21.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.457274+0000) 2022-01-31T21:06:21.813 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[41853]: debug 2022-01-31T21:06:21.690+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.690540+0000) 2022-01-31T21:06:21.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:21 smithi171 conmon[51620]: debug 2022-01-31T21:06:21.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:21.563810+0000) 2022-01-31T21:06:22.434 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:22 smithi167 conmon[60316]: debug 2022-01-31T21:06:22.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:22.208827+0000) 2022-01-31T21:06:22.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:22 smithi171 conmon[46715]: debug 2022-01-31T21:06:22.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:22.235814+0000) 2022-01-31T21:06:22.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:22 smithi167 conmon[49112]: debug 2022-01-31T21:06:22.529+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:22.530025+0000) 2022-01-31T21:06:22.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:22 smithi167 conmon[54076]: debug 2022-01-31T21:06:22.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:22.457414+0000) 2022-01-31T21:06:22.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:22 smithi171 conmon[41853]: debug 2022-01-31T21:06:22.690+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:22.690738+0000) 2022-01-31T21:06:22.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:22 smithi171 conmon[51620]: debug 2022-01-31T21:06:22.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:22.563967+0000) 2022-01-31T21:06:23.434 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:23 smithi167 conmon[60316]: debug 2022-01-31T21:06:23.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:23.209036+0000) 2022-01-31T21:06:23.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:23 smithi171 conmon[46715]: debug 2022-01-31T21:06:23.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:23.236031+0000) 2022-01-31T21:06:23.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:23 smithi167 conmon[49112]: debug 2022-01-31T21:06:23.528+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:23.530249+0000) 2022-01-31T21:06:23.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:23 smithi167 conmon[54076]: debug 2022-01-31T21:06:23.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:23.457577+0000) 2022-01-31T21:06:23.813 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:23 smithi171 conmon[35325]: debug 2022-01-31T21:06:23.624+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:06:23.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:23 smithi171 conmon[41853]: debug 2022-01-31T21:06:23.690+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:23.690974+0000) 2022-01-31T21:06:23.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:23 smithi171 conmon[51620]: debug 2022-01-31T21:06:23.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:23.564151+0000) 2022-01-31T21:06:24.434 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:24 smithi167 conmon[60316]: debug 2022-01-31T21:06:24.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:24.209234+0000) 2022-01-31T21:06:24.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:24 smithi171 conmon[46715]: debug 2022-01-31T21:06:24.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:24.236215+0000) 2022-01-31T21:06:24.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:24 smithi167 conmon[49112]: debug 2022-01-31T21:06:24.529+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:24.530475+0000) 2022-01-31T21:06:24.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:24 smithi167 conmon[54076]: debug 2022-01-31T21:06:24.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:24.457771+0000) 2022-01-31T21:06:24.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:24 smithi171 conmon[41853]: debug 2022-01-31T21:06:24.690+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:24.691165+0000) 2022-01-31T21:06:24.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:24 smithi171 conmon[51620]: debug 2022-01-31T21:06:24.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:24.564337+0000) 2022-01-31T21:06:25.434 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:25 smithi167 conmon[60316]: debug 2022-01-31T21:06:25.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:25.209425+0000) 2022-01-31T21:06:25.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:25 smithi171 conmon[46715]: debug 2022-01-31T21:06:25.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:25.236399+0000) 2022-01-31T21:06:25.690 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:25 smithi167 conmon[49112]: debug 2022-01-31T21:06:25.530+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:25.530700+0000) 2022-01-31T21:06:25.691 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:25 smithi167 conmon[54076]: debug 2022-01-31T21:06:25.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:25.457963+0000) 2022-01-31T21:06:25.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:25 smithi171 conmon[41853]: debug 2022-01-31T21:06:25.690+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:25.691346+0000) 2022-01-31T21:06:25.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:25 smithi171 conmon[51620]: debug 2022-01-31T21:06:25.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:25.564513+0000) 2022-01-31T21:06:26.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:26 smithi167 conmon[49112]: debug 2022-01-31T21:06:26.133+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.135415+0000) 2022-01-31T21:06:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:26 smithi167 conmon[54076]: debug 2022-01-31T21:06:26.133+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.135482+0000) 2022-01-31T21:06:26.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:26 smithi167 conmon[60316]: debug 2022-01-31T21:06:26.133+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.135146+0000) 2022-01-31T21:06:26.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:26 smithi167 conmon[60316]: debug 2022-01-31T21:06:26.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.209614+0000) 2022-01-31T21:06:26.563 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[35325]: debug 2022-01-31T21:06:26.144+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103508 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:26.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[41853]: debug 2022-01-31T21:06:26.134+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.135621+0000) 2022-01-31T21:06:26.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[51620]: debug 2022-01-31T21:06:26.134+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.135044+0000) 2022-01-31T21:06:26.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[46715]: debug 2022-01-31T21:06:26.133+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.134652+0000) 2022-01-31T21:06:26.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[46715]: debug 2022-01-31T21:06:26.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.236560+0000) 2022-01-31T21:06:26.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:26 smithi167 conmon[49112]: debug 2022-01-31T21:06:26.530+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.530869+0000) 2022-01-31T21:06:26.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:26 smithi167 conmon[54076]: debug 2022-01-31T21:06:26.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.458122+0000) 2022-01-31T21:06:26.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[51620]: debug 2022-01-31T21:06:26.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.564709+0000) 2022-01-31T21:06:26.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:26 smithi171 conmon[41853]: debug 2022-01-31T21:06:26.691+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:26.691569+0000) 2022-01-31T21:06:27.435 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:27 smithi167 conmon[60316]: debug 2022-01-31T21:06:27.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:27.209787+0000) 2022-01-31T21:06:27.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:27 smithi171 conmon[46715]: debug 2022-01-31T21:06:27.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:27.236685+0000) 2022-01-31T21:06:27.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:27 smithi167 conmon[49112]: debug 2022-01-31T21:06:27.530+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:27.531050+0000) 2022-01-31T21:06:27.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:27 smithi167 conmon[54076]: debug 2022-01-31T21:06:27.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:27.458280+0000) 2022-01-31T21:06:27.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:27 smithi171 conmon[41853]: debug 2022-01-31T21:06:27.690+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:27.691737+0000) 2022-01-31T21:06:27.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:27 smithi171 conmon[51620]: debug 2022-01-31T21:06:27.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:27.564857+0000) 2022-01-31T21:06:28.435 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:28 smithi167 conmon[60316]: debug 2022-01-31T21:06:28.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:28.209965+0000) 2022-01-31T21:06:28.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:28 smithi171 conmon[46715]: debug 2022-01-31T21:06:28.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:28.236840+0000) 2022-01-31T21:06:28.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:28 smithi167 conmon[54076]: debug 2022-01-31T21:06:28.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:28.458476+0000) 2022-01-31T21:06:28.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:28 smithi167 conmon[49112]: debug 2022-01-31T21:06:28.530+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:28.531253+0000) 2022-01-31T21:06:28.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:28 smithi171 conmon[41853]: debug 2022-01-31T21:06:28.691+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:28.691867+0000) 2022-01-31T21:06:28.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:28 smithi171 conmon[51620]: debug 2022-01-31T21:06:28.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:28.565019+0000) 2022-01-31T21:06:29.435 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:29 smithi167 conmon[60316]: debug 2022-01-31T21:06:29.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:29.210125+0000) 2022-01-31T21:06:29.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:29 smithi171 conmon[46715]: debug 2022-01-31T21:06:29.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:29.237040+0000) 2022-01-31T21:06:29.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:29 smithi167 conmon[49112]: debug 2022-01-31T21:06:29.530+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:29.531487+0000) 2022-01-31T21:06:29.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:29 smithi167 conmon[54076]: debug 2022-01-31T21:06:29.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:29.458684+0000) 2022-01-31T21:06:29.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:29 smithi171 conmon[41853]: debug 2022-01-31T21:06:29.691+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:29.692053+0000) 2022-01-31T21:06:29.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:29 smithi171 conmon[51620]: debug 2022-01-31T21:06:29.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:29.565152+0000) 2022-01-31T21:06:30.435 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:30 smithi167 conmon[60316]: debug 2022-01-31T21:06:30.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:30.210325+0000) 2022-01-31T21:06:30.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:30 smithi171 conmon[46715]: debug 2022-01-31T21:06:30.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:30.237232+0000) 2022-01-31T21:06:30.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:30 smithi167 conmon[49112]: debug 2022-01-31T21:06:30.531+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:30.531674+0000) 2022-01-31T21:06:30.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:30 smithi167 conmon[54076]: debug 2022-01-31T21:06:30.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:30.458852+0000) 2022-01-31T21:06:30.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:30 smithi171 conmon[51620]: debug 2022-01-31T21:06:30.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:30.565309+0000) 2022-01-31T21:06:30.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:30 smithi171 conmon[41853]: debug 2022-01-31T21:06:30.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:30.692225+0000) 2022-01-31T21:06:31.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:31 smithi167 conmon[49112]: debug 2022-01-31T21:06:31.147+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.148070+0000) 2022-01-31T21:06:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:31 smithi167 conmon[54076]: debug 2022-01-31T21:06:31.148+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.148674+0000) 2022-01-31T21:06:31.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:31 smithi167 conmon[60316]: debug 2022-01-31T21:06:31.148+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.148525+0000) 2022-01-31T21:06:31.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:31 smithi167 conmon[60316]: debug 2022-01-31T21:06:31.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.210549+0000) 2022-01-31T21:06:31.564 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[35325]: debug 2022-01-31T21:06:31.157+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103617 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:31.564 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[41853]: debug 2022-01-31T21:06:31.148+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.148828+0000) 2022-01-31T21:06:31.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[51620]: debug 2022-01-31T21:06:31.146+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.147516+0000) 2022-01-31T21:06:31.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[46715]: debug 2022-01-31T21:06:31.146+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.147595+0000) 2022-01-31T21:06:31.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[46715]: debug 2022-01-31T21:06:31.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.237377+0000) 2022-01-31T21:06:31.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:31 smithi167 conmon[49112]: debug 2022-01-31T21:06:31.531+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.531895+0000) 2022-01-31T21:06:31.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:31 smithi167 conmon[54076]: debug 2022-01-31T21:06:31.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.459005+0000) 2022-01-31T21:06:31.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[51620]: debug 2022-01-31T21:06:31.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.565498+0000) 2022-01-31T21:06:31.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:31 smithi171 conmon[41853]: debug 2022-01-31T21:06:31.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:31.692378+0000) 2022-01-31T21:06:32.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:32 smithi167 conmon[60316]: debug 2022-01-31T21:06:32.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:32.210689+0000) 2022-01-31T21:06:32.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:32 smithi171 conmon[46715]: debug 2022-01-31T21:06:32.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:32.237535+0000) 2022-01-31T21:06:32.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:32 smithi167 conmon[49112]: debug 2022-01-31T21:06:32.531+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:32.532065+0000) 2022-01-31T21:06:32.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:32 smithi167 conmon[54076]: debug 2022-01-31T21:06:32.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:32.459150+0000) 2022-01-31T21:06:32.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:32 smithi171 conmon[51620]: debug 2022-01-31T21:06:32.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:32.565653+0000) 2022-01-31T21:06:32.816 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:32 smithi171 conmon[41853]: debug 2022-01-31T21:06:32.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:32.692585+0000) 2022-01-31T21:06:33.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:33 smithi167 conmon[60316]: debug 2022-01-31T21:06:33.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:33.210867+0000) 2022-01-31T21:06:33.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:33 smithi171 conmon[46715]: debug 2022-01-31T21:06:33.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:33.237737+0000) 2022-01-31T21:06:33.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:33 smithi167 conmon[49112]: debug 2022-01-31T21:06:33.531+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:33.532278+0000) 2022-01-31T21:06:33.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:33 smithi167 conmon[54076]: debug 2022-01-31T21:06:33.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:33.459324+0000) 2022-01-31T21:06:33.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:33 smithi171 conmon[51620]: debug 2022-01-31T21:06:33.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:33.565832+0000) 2022-01-31T21:06:33.816 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:33 smithi171 conmon[41853]: debug 2022-01-31T21:06:33.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:33.692770+0000) 2022-01-31T21:06:34.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:34 smithi167 conmon[60316]: debug 2022-01-31T21:06:34.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:34.211082+0000) 2022-01-31T21:06:34.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:34 smithi171 conmon[46715]: debug 2022-01-31T21:06:34.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:34.237934+0000) 2022-01-31T21:06:34.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:34 smithi167 conmon[49112]: debug 2022-01-31T21:06:34.531+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:34.532474+0000) 2022-01-31T21:06:34.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:34 smithi167 conmon[54076]: debug 2022-01-31T21:06:34.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:34.459560+0000) 2022-01-31T21:06:34.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:34 smithi171 conmon[41853]: debug 2022-01-31T21:06:34.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:34.692869+0000) 2022-01-31T21:06:34.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:34 smithi171 conmon[51620]: debug 2022-01-31T21:06:34.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:34.565981+0000) 2022-01-31T21:06:35.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:35 smithi167 conmon[60316]: debug 2022-01-31T21:06:35.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:35.211274+0000) 2022-01-31T21:06:35.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:35 smithi171 conmon[46715]: debug 2022-01-31T21:06:35.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:35.238093+0000) 2022-01-31T21:06:35.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:35 smithi167 conmon[49112]: debug 2022-01-31T21:06:35.532+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:35.532670+0000) 2022-01-31T21:06:35.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:35 smithi167 conmon[54076]: debug 2022-01-31T21:06:35.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:35.459732+0000) 2022-01-31T21:06:35.816 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:35 smithi171 conmon[41853]: debug 2022-01-31T21:06:35.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:35.693034+0000) 2022-01-31T21:06:35.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:35 smithi171 conmon[51620]: debug 2022-01-31T21:06:35.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:35.566137+0000) 2022-01-31T21:06:36.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:36 smithi167 conmon[49112]: debug 2022-01-31T21:06:36.160+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.161090+0000) 2022-01-31T21:06:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:36 smithi167 conmon[54076]: debug 2022-01-31T21:06:36.161+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.161714+0000) 2022-01-31T21:06:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:36 smithi167 conmon[60316]: debug 2022-01-31T21:06:36.161+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.162243+0000) 2022-01-31T21:06:36.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:36 smithi167 conmon[60316]: debug 2022-01-31T21:06:36.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.211428+0000) 2022-01-31T21:06:36.564 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[35325]: debug 2022-01-31T21:06:36.170+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103727 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:36.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[41853]: debug 2022-01-31T21:06:36.159+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.161316+0000) 2022-01-31T21:06:36.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[51620]: debug 2022-01-31T21:06:36.159+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.161280+0000) 2022-01-31T21:06:36.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[46715]: debug 2022-01-31T21:06:36.158+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.160669+0000) 2022-01-31T21:06:36.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[46715]: debug 2022-01-31T21:06:36.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.238245+0000) 2022-01-31T21:06:36.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:36 smithi167 conmon[49112]: debug 2022-01-31T21:06:36.532+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.532846+0000) 2022-01-31T21:06:36.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:36 smithi167 conmon[54076]: debug 2022-01-31T21:06:36.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.459872+0000) 2022-01-31T21:06:36.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[51620]: debug 2022-01-31T21:06:36.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.566321+0000) 2022-01-31T21:06:36.817 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:36 smithi171 conmon[41853]: debug 2022-01-31T21:06:36.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:36.693214+0000) 2022-01-31T21:06:37.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:37 smithi167 conmon[60316]: debug 2022-01-31T21:06:37.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:37.211540+0000) 2022-01-31T21:06:37.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:37 smithi171 conmon[46715]: debug 2022-01-31T21:06:37.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:37.238390+0000) 2022-01-31T21:06:37.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:37 smithi167 conmon[49112]: debug 2022-01-31T21:06:37.532+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:37.532984+0000) 2022-01-31T21:06:37.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:37 smithi167 conmon[54076]: debug 2022-01-31T21:06:37.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:37.460018+0000) 2022-01-31T21:06:37.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:37 smithi171 conmon[51620]: debug 2022-01-31T21:06:37.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:37.566475+0000) 2022-01-31T21:06:37.816 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:37 smithi171 conmon[41853]: debug 2022-01-31T21:06:37.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:37.693347+0000) 2022-01-31T21:06:38.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:38 smithi167 conmon[60316]: debug 2022-01-31T21:06:38.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:38.211709+0000) 2022-01-31T21:06:38.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:38 smithi171 conmon[46715]: debug 2022-01-31T21:06:38.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:38.238593+0000) 2022-01-31T21:06:38.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:38 smithi167 conmon[49112]: debug 2022-01-31T21:06:38.532+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:38.533182+0000) 2022-01-31T21:06:38.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:38 smithi167 conmon[54076]: debug 2022-01-31T21:06:38.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:38.460187+0000) 2022-01-31T21:06:38.817 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:38 smithi171 conmon[35325]: debug 2022-01-31T21:06:38.624+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:06:38.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:38 smithi171 conmon[41853]: debug 2022-01-31T21:06:38.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:38.693520+0000) 2022-01-31T21:06:38.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:38 smithi171 conmon[51620]: debug 2022-01-31T21:06:38.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:38.566670+0000) 2022-01-31T21:06:39.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:39 smithi167 conmon[60316]: debug 2022-01-31T21:06:39.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:39.211897+0000) 2022-01-31T21:06:39.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:39 smithi171 conmon[46715]: debug 2022-01-31T21:06:39.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:39.238787+0000) 2022-01-31T21:06:39.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:39 smithi167 conmon[49112]: debug 2022-01-31T21:06:39.532+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:39.533403+0000) 2022-01-31T21:06:39.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:39 smithi167 conmon[54076]: debug 2022-01-31T21:06:39.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:39.460379+0000) 2022-01-31T21:06:39.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:39 smithi171 conmon[51620]: debug 2022-01-31T21:06:39.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:39.566882+0000) 2022-01-31T21:06:39.817 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:39 smithi171 conmon[41853]: debug 2022-01-31T21:06:39.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:39.693688+0000) 2022-01-31T21:06:40.459 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:40 smithi167 conmon[60316]: debug 2022-01-31T21:06:40.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:40.212097+0000) 2022-01-31T21:06:40.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:40 smithi171 conmon[46715]: debug 2022-01-31T21:06:40.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:40.239003+0000) 2022-01-31T21:06:40.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:40 smithi167 conmon[49112]: debug 2022-01-31T21:06:40.533+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:40.533592+0000) 2022-01-31T21:06:40.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:40 smithi167 conmon[54076]: debug 2022-01-31T21:06:40.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:40.460551+0000) 2022-01-31T21:06:40.816 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:40 smithi171 conmon[41853]: debug 2022-01-31T21:06:40.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:40.693873+0000) 2022-01-31T21:06:40.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:40 smithi171 conmon[51620]: debug 2022-01-31T21:06:40.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:40.567058+0000) 2022-01-31T21:06:41.437 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:41 smithi167 conmon[49112]: debug 2022-01-31T21:06:41.174+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.174634+0000) 2022-01-31T21:06:41.438 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:41 smithi167 conmon[54076]: debug 2022-01-31T21:06:41.180+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.181411+0000) 2022-01-31T21:06:41.438 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:41 smithi167 conmon[60316]: debug 2022-01-31T21:06:41.180+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.181263+0000) 2022-01-31T21:06:41.439 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:41 smithi167 conmon[60316]: debug 2022-01-31T21:06:41.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.212274+0000) 2022-01-31T21:06:41.565 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[35325]: debug 2022-01-31T21:06:41.199+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103837 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:41.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[41853]: debug 2022-01-31T21:06:41.173+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.175083+0000) 2022-01-31T21:06:41.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[46715]: debug 2022-01-31T21:06:41.173+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.175171+0000) 2022-01-31T21:06:41.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[46715]: debug 2022-01-31T21:06:41.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.239205+0000) 2022-01-31T21:06:41.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[51620]: debug 2022-01-31T21:06:41.173+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.174872+0000) 2022-01-31T21:06:41.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:41 smithi167 conmon[49112]: debug 2022-01-31T21:06:41.533+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.533798+0000) 2022-01-31T21:06:41.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:41 smithi167 conmon[54076]: debug 2022-01-31T21:06:41.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.460773+0000) 2022-01-31T21:06:41.817 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[41853]: debug 2022-01-31T21:06:41.693+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.694054+0000) 2022-01-31T21:06:41.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:41 smithi171 conmon[51620]: debug 2022-01-31T21:06:41.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:41.567224+0000) 2022-01-31T21:06:42.459 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:42 smithi167 conmon[60316]: debug 2022-01-31T21:06:42.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:42.212449+0000) 2022-01-31T21:06:42.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:42 smithi171 conmon[46715]: debug 2022-01-31T21:06:42.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:42.239369+0000) 2022-01-31T21:06:42.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:42 smithi167 conmon[49112]: debug 2022-01-31T21:06:42.533+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:42.533954+0000) 2022-01-31T21:06:42.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:42 smithi167 conmon[54076]: debug 2022-01-31T21:06:42.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:42.460930+0000) 2022-01-31T21:06:42.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:42 smithi171 conmon[51620]: debug 2022-01-31T21:06:42.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:42.567357+0000) 2022-01-31T21:06:42.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:42 smithi171 conmon[41853]: debug 2022-01-31T21:06:42.693+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:42.694222+0000) 2022-01-31T21:06:43.459 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:43 smithi167 conmon[60316]: debug 2022-01-31T21:06:43.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:43.212664+0000) 2022-01-31T21:06:43.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:43 smithi171 conmon[46715]: debug 2022-01-31T21:06:43.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:43.239553+0000) 2022-01-31T21:06:43.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:43 smithi167 conmon[49112]: debug 2022-01-31T21:06:43.533+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:43.534129+0000) 2022-01-31T21:06:43.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:43 smithi167 conmon[54076]: debug 2022-01-31T21:06:43.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:43.461138+0000) 2022-01-31T21:06:43.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:43 smithi171 conmon[51620]: debug 2022-01-31T21:06:43.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:43.567528+0000) 2022-01-31T21:06:43.817 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:43 smithi171 conmon[41853]: debug 2022-01-31T21:06:43.692+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:43.694372+0000) 2022-01-31T21:06:44.459 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:44 smithi167 conmon[60316]: debug 2022-01-31T21:06:44.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:44.212798+0000) 2022-01-31T21:06:44.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:44 smithi171 conmon[46715]: debug 2022-01-31T21:06:44.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:44.239779+0000) 2022-01-31T21:06:44.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:44 smithi167 conmon[49112]: debug 2022-01-31T21:06:44.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:44.534327+0000) 2022-01-31T21:06:44.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:44 smithi167 conmon[54076]: debug 2022-01-31T21:06:44.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:44.461333+0000) 2022-01-31T21:06:44.817 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:44 smithi171 conmon[41853]: debug 2022-01-31T21:06:44.693+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:44.694562+0000) 2022-01-31T21:06:44.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:44 smithi171 conmon[51620]: debug 2022-01-31T21:06:44.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:44.567682+0000) 2022-01-31T21:06:45.459 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:45 smithi167 conmon[60316]: debug 2022-01-31T21:06:45.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:45.212921+0000) 2022-01-31T21:06:45.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:45 smithi171 conmon[46715]: debug 2022-01-31T21:06:45.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:45.239953+0000) 2022-01-31T21:06:45.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:45 smithi167 conmon[49112]: debug 2022-01-31T21:06:45.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:45.534533+0000) 2022-01-31T21:06:45.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:45 smithi167 conmon[54076]: debug 2022-01-31T21:06:45.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:45.461505+0000) 2022-01-31T21:06:45.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:45 smithi171 conmon[51620]: debug 2022-01-31T21:06:45.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:45.567874+0000) 2022-01-31T21:06:45.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:45 smithi171 conmon[41853]: debug 2022-01-31T21:06:45.693+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:45.694745+0000) 2022-01-31T21:06:46.460 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:46 smithi167 conmon[49112]: debug 2022-01-31T21:06:46.202+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.202828+0000) 2022-01-31T21:06:46.460 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:46 smithi167 conmon[54076]: debug 2022-01-31T21:06:46.204+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.204889+0000) 2022-01-31T21:06:46.461 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:46 smithi167 conmon[60316]: debug 2022-01-31T21:06:46.203+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.204070+0000) 2022-01-31T21:06:46.461 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:46 smithi167 conmon[60316]: debug 2022-01-31T21:06:46.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.213082+0000) 2022-01-31T21:06:46.566 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[35325]: debug 2022-01-31T21:06:46.213+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 103947 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:46.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[41853]: debug 2022-01-31T21:06:46.202+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.204478+0000) 2022-01-31T21:06:46.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[51620]: debug 2022-01-31T21:06:46.202+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.203899+0000) 2022-01-31T21:06:46.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[46715]: debug 2022-01-31T21:06:46.201+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.203632+0000) 2022-01-31T21:06:46.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[46715]: debug 2022-01-31T21:06:46.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.240106+0000) 2022-01-31T21:06:46.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:46 smithi167 conmon[49112]: debug 2022-01-31T21:06:46.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.534675+0000) 2022-01-31T21:06:46.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:46 smithi167 conmon[54076]: debug 2022-01-31T21:06:46.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.461610+0000) 2022-01-31T21:06:46.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[41853]: debug 2022-01-31T21:06:46.694+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.694959+0000) 2022-01-31T21:06:46.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:46 smithi171 conmon[51620]: debug 2022-01-31T21:06:46.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:46.568088+0000) 2022-01-31T21:06:47.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:47 smithi167 conmon[54076]: debug 2022-01-31T21:06:47.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:47.461739+0000) 2022-01-31T21:06:47.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:47 smithi167 conmon[60316]: debug 2022-01-31T21:06:47.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:47.213233+0000) 2022-01-31T21:06:47.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:47 smithi171 conmon[46715]: debug 2022-01-31T21:06:47.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:47.240282+0000) 2022-01-31T21:06:47.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:47 smithi167 conmon[49112]: debug 2022-01-31T21:06:47.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:47.534827+0000) 2022-01-31T21:06:47.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:47 smithi171 conmon[41853]: debug 2022-01-31T21:06:47.694+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:47.695133+0000) 2022-01-31T21:06:47.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:47 smithi171 conmon[51620]: debug 2022-01-31T21:06:47.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:47.568241+0000) 2022-01-31T21:06:48.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:48 smithi167 conmon[54076]: debug 2022-01-31T21:06:48.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:48.461914+0000) 2022-01-31T21:06:48.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:48 smithi167 conmon[60316]: debug 2022-01-31T21:06:48.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:48.213418+0000) 2022-01-31T21:06:48.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:48 smithi171 conmon[46715]: debug 2022-01-31T21:06:48.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:48.240437+0000) 2022-01-31T21:06:48.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:48 smithi167 conmon[49112]: debug 2022-01-31T21:06:48.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:48.535023+0000) 2022-01-31T21:06:48.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:48 smithi171 conmon[51620]: debug 2022-01-31T21:06:48.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:48.568462+0000) 2022-01-31T21:06:48.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:48 smithi171 conmon[41853]: debug 2022-01-31T21:06:48.694+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:48.695337+0000) 2022-01-31T21:06:49.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:49 smithi167 conmon[54076]: debug 2022-01-31T21:06:49.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:49.462037+0000) 2022-01-31T21:06:49.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:49 smithi167 conmon[60316]: debug 2022-01-31T21:06:49.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:49.213580+0000) 2022-01-31T21:06:49.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:49 smithi171 conmon[46715]: debug 2022-01-31T21:06:49.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:49.240634+0000) 2022-01-31T21:06:49.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:49 smithi167 conmon[49112]: debug 2022-01-31T21:06:49.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:49.535198+0000) 2022-01-31T21:06:49.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:49 smithi171 conmon[41853]: debug 2022-01-31T21:06:49.694+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:49.695510+0000) 2022-01-31T21:06:49.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:49 smithi171 conmon[51620]: debug 2022-01-31T21:06:49.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:49.568652+0000) 2022-01-31T21:06:50.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:50 smithi167 conmon[54076]: debug 2022-01-31T21:06:50.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:50.462225+0000) 2022-01-31T21:06:50.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:50 smithi167 conmon[60316]: debug 2022-01-31T21:06:50.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:50.213726+0000) 2022-01-31T21:06:50.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:50 smithi171 conmon[46715]: debug 2022-01-31T21:06:50.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:50.240821+0000) 2022-01-31T21:06:50.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:50 smithi167 conmon[49112]: debug 2022-01-31T21:06:50.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:50.535404+0000) 2022-01-31T21:06:50.818 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:50 smithi171 conmon[41853]: debug 2022-01-31T21:06:50.694+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:50.695696+0000) 2022-01-31T21:06:50.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:50 smithi171 conmon[51620]: debug 2022-01-31T21:06:50.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:50.568808+0000) 2022-01-31T21:06:51.509 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:51 smithi167 conmon[49112]: debug 2022-01-31T21:06:51.215+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.216316+0000) 2022-01-31T21:06:51.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:51 smithi167 conmon[54076]: debug 2022-01-31T21:06:51.216+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.217221+0000) 2022-01-31T21:06:51.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:51 smithi167 conmon[54076]: debug 2022-01-31T21:06:51.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.462389+0000) 2022-01-31T21:06:51.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:51 smithi167 conmon[60316]: debug 2022-01-31T21:06:51.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.213888+0000) 2022-01-31T21:06:51.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:51 smithi167 conmon[60316]: debug 2022-01-31T21:06:51.216+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.217026+0000) 2022-01-31T21:06:51.567 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[35325]: debug 2022-01-31T21:06:51.226+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104057 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:51.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[41853]: debug 2022-01-31T21:06:51.216+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.217470+0000) 2022-01-31T21:06:51.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[51620]: debug 2022-01-31T21:06:51.216+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.217149+0000) 2022-01-31T21:06:51.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[46715]: debug 2022-01-31T21:06:51.215+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.216252+0000) 2022-01-31T21:06:51.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[46715]: debug 2022-01-31T21:06:51.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.240968+0000) 2022-01-31T21:06:51.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:51 smithi167 conmon[49112]: debug 2022-01-31T21:06:51.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.535580+0000) 2022-01-31T21:06:51.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[51620]: debug 2022-01-31T21:06:51.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.568958+0000) 2022-01-31T21:06:51.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:51 smithi171 conmon[41853]: debug 2022-01-31T21:06:51.695+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:51.695904+0000) 2022-01-31T21:06:52.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:52 smithi167 conmon[54076]: debug 2022-01-31T21:06:52.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:52.462574+0000) 2022-01-31T21:06:52.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:52 smithi167 conmon[60316]: debug 2022-01-31T21:06:52.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:52.214090+0000) 2022-01-31T21:06:52.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:52 smithi171 conmon[46715]: debug 2022-01-31T21:06:52.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:52.241123+0000) 2022-01-31T21:06:52.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:52 smithi167 conmon[49112]: debug 2022-01-31T21:06:52.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:52.535701+0000) 2022-01-31T21:06:52.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:52 smithi171 conmon[51620]: debug 2022-01-31T21:06:52.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:52.569087+0000) 2022-01-31T21:06:52.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:52 smithi171 conmon[41853]: debug 2022-01-31T21:06:52.695+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:52.696074+0000) 2022-01-31T21:06:53.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:53 smithi167 conmon[54076]: debug 2022-01-31T21:06:53.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:53.462768+0000) 2022-01-31T21:06:53.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:53 smithi167 conmon[60316]: debug 2022-01-31T21:06:53.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:53.214258+0000) 2022-01-31T21:06:53.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:53 smithi171 conmon[46715]: debug 2022-01-31T21:06:53.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:53.241338+0000) 2022-01-31T21:06:53.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:53 smithi167 conmon[49112]: debug 2022-01-31T21:06:53.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:53.535872+0000) 2022-01-31T21:06:53.819 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:53 smithi171 conmon[35325]: debug 2022-01-31T21:06:53.625+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:06:53.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:53 smithi171 conmon[51620]: debug 2022-01-31T21:06:53.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:53.569299+0000) 2022-01-31T21:06:53.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:53 smithi171 conmon[41853]: debug 2022-01-31T21:06:53.695+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:53.696227+0000) 2022-01-31T21:06:54.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:54 smithi167 conmon[54076]: debug 2022-01-31T21:06:54.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:54.462931+0000) 2022-01-31T21:06:54.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:54 smithi167 conmon[60316]: debug 2022-01-31T21:06:54.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:54.214423+0000) 2022-01-31T21:06:54.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:54 smithi171 conmon[46715]: debug 2022-01-31T21:06:54.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:54.241505+0000) 2022-01-31T21:06:54.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:54 smithi167 conmon[49112]: debug 2022-01-31T21:06:54.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:54.536116+0000) 2022-01-31T21:06:54.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:54 smithi171 conmon[51620]: debug 2022-01-31T21:06:54.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:54.569486+0000) 2022-01-31T21:06:54.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:54 smithi171 conmon[41853]: debug 2022-01-31T21:06:54.695+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:54.696382+0000) 2022-01-31T21:06:55.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:55 smithi167 conmon[54076]: debug 2022-01-31T21:06:55.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:55.463081+0000) 2022-01-31T21:06:55.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:55 smithi167 conmon[60316]: debug 2022-01-31T21:06:55.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:55.214609+0000) 2022-01-31T21:06:55.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:55 smithi171 conmon[46715]: debug 2022-01-31T21:06:55.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:55.241657+0000) 2022-01-31T21:06:55.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:55 smithi167 conmon[49112]: debug 2022-01-31T21:06:55.534+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:55.536237+0000) 2022-01-31T21:06:55.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:55 smithi171 conmon[51620]: debug 2022-01-31T21:06:55.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:55.569672+0000) 2022-01-31T21:06:55.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:55 smithi171 conmon[41853]: debug 2022-01-31T21:06:55.695+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:55.696592+0000) 2022-01-31T21:06:56.509 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:56 smithi167 conmon[49112]: debug 2022-01-31T21:06:56.228+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.230085+0000) 2022-01-31T21:06:56.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:56 smithi167 conmon[60316]: debug 2022-01-31T21:06:56.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.214796+0000) 2022-01-31T21:06:56.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:56 smithi167 conmon[60316]: debug 2022-01-31T21:06:56.228+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.230457+0000) 2022-01-31T21:06:56.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:56 smithi167 conmon[54076]: debug 2022-01-31T21:06:56.229+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.230772+0000) 2022-01-31T21:06:56.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:56 smithi167 conmon[54076]: debug 2022-01-31T21:06:56.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.463248+0000) 2022-01-31T21:06:56.568 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[35325]: debug 2022-01-31T21:06:56.240+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104168 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:06:56.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[41853]: debug 2022-01-31T21:06:56.228+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.229228+0000) 2022-01-31T21:06:56.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[51620]: debug 2022-01-31T21:06:56.229+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.229742+0000) 2022-01-31T21:06:56.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[46715]: debug 2022-01-31T21:06:56.228+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.228915+0000) 2022-01-31T21:06:56.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[46715]: debug 2022-01-31T21:06:56.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.241781+0000) 2022-01-31T21:06:56.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:56 smithi167 conmon[49112]: debug 2022-01-31T21:06:56.535+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.536376+0000) 2022-01-31T21:06:56.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[41853]: debug 2022-01-31T21:06:56.696+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.696802+0000) 2022-01-31T21:06:56.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:56 smithi171 conmon[51620]: debug 2022-01-31T21:06:56.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:56.569820+0000) 2022-01-31T21:06:57.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:57 smithi167 conmon[54076]: debug 2022-01-31T21:06:57.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:57.463366+0000) 2022-01-31T21:06:57.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:57 smithi167 conmon[60316]: debug 2022-01-31T21:06:57.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:57.214984+0000) 2022-01-31T21:06:57.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:57 smithi171 conmon[46715]: debug 2022-01-31T21:06:57.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:57.241896+0000) 2022-01-31T21:06:57.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:57 smithi167 conmon[49112]: debug 2022-01-31T21:06:57.535+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:57.536455+0000) 2022-01-31T21:06:57.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:57 smithi171 conmon[41853]: debug 2022-01-31T21:06:57.696+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:57.696982+0000) 2022-01-31T21:06:57.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:57 smithi171 conmon[51620]: debug 2022-01-31T21:06:57.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:57.569980+0000) 2022-01-31T21:06:58.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:58 smithi167 conmon[54076]: debug 2022-01-31T21:06:58.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:58.463526+0000) 2022-01-31T21:06:58.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:58 smithi167 conmon[60316]: debug 2022-01-31T21:06:58.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:58.215132+0000) 2022-01-31T21:06:58.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:58 smithi171 conmon[46715]: debug 2022-01-31T21:06:58.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:58.242042+0000) 2022-01-31T21:06:58.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:58 smithi167 conmon[49112]: debug 2022-01-31T21:06:58.535+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:58.536598+0000) 2022-01-31T21:06:58.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:58 smithi171 conmon[41853]: debug 2022-01-31T21:06:58.696+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:58.697109+0000) 2022-01-31T21:06:58.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:58 smithi171 conmon[51620]: debug 2022-01-31T21:06:58.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:58.570130+0000) 2022-01-31T21:06:59.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:06:59 smithi167 conmon[54076]: debug 2022-01-31T21:06:59.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:59.463732+0000) 2022-01-31T21:06:59.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:06:59 smithi167 conmon[60316]: debug 2022-01-31T21:06:59.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:59.215309+0000) 2022-01-31T21:06:59.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:06:59 smithi171 conmon[46715]: debug 2022-01-31T21:06:59.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:59.242188+0000) 2022-01-31T21:06:59.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:06:59 smithi167 conmon[49112]: debug 2022-01-31T21:06:59.535+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:59.536778+0000) 2022-01-31T21:06:59.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:06:59 smithi171 conmon[41853]: debug 2022-01-31T21:06:59.696+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:59.697263+0000) 2022-01-31T21:06:59.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:06:59 smithi171 conmon[51620]: debug 2022-01-31T21:06:59.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:06:59.570321+0000) 2022-01-31T21:07:00.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:00 smithi167 conmon[60316]: debug 2022-01-31T21:07:00.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:00.215526+0000) 2022-01-31T21:07:00.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:00 smithi167 conmon[54076]: debug 2022-01-31T21:07:00.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:00.463941+0000) 2022-01-31T21:07:00.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:00 smithi171 conmon[46715]: debug 2022-01-31T21:07:00.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:00.242335+0000) 2022-01-31T21:07:00.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:00 smithi167 conmon[49112]: debug 2022-01-31T21:07:00.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:00.536969+0000) 2022-01-31T21:07:00.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:00 smithi171 conmon[41853]: debug 2022-01-31T21:07:00.696+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:00.697462+0000) 2022-01-31T21:07:00.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:00 smithi171 conmon[51620]: debug 2022-01-31T21:07:00.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:00.570502+0000) 2022-01-31T21:07:01.510 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:01 smithi167 conmon[49112]: debug 2022-01-31T21:07:01.243+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.244821+0000) 2022-01-31T21:07:01.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:01 smithi167 conmon[60316]: debug 2022-01-31T21:07:01.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.215711+0000) 2022-01-31T21:07:01.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:01 smithi167 conmon[60316]: debug 2022-01-31T21:07:01.242+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.244201+0000) 2022-01-31T21:07:01.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:01 smithi167 conmon[54076]: debug 2022-01-31T21:07:01.243+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.244986+0000) 2022-01-31T21:07:01.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:01 smithi167 conmon[54076]: debug 2022-01-31T21:07:01.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.464179+0000) 2022-01-31T21:07:01.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:01 smithi171 conmon[41853]: debug 2022-01-31T21:07:01.242+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.243068+0000) 2022-01-31T21:07:01.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:01 smithi171 conmon[46715]: debug 2022-01-31T21:07:01.243+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.244460+0000) 2022-01-31T21:07:01.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:01 smithi171 conmon[51620]: debug 2022-01-31T21:07:01.242+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.242938+0000) 2022-01-31T21:07:01.571 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:01 smithi171 conmon[35325]: debug 2022-01-31T21:07:01.253+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104278 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:01.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:01 smithi167 conmon[49112]: debug 2022-01-31T21:07:01.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.537175+0000) 2022-01-31T21:07:01.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:01 smithi171 conmon[41853]: debug 2022-01-31T21:07:01.696+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.697660+0000) 2022-01-31T21:07:01.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:01 smithi171 conmon[51620]: debug 2022-01-31T21:07:01.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:01.570698+0000) 2022-01-31T21:07:02.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:02 smithi167 conmon[54076]: debug 2022-01-31T21:07:02.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:02.464283+0000) 2022-01-31T21:07:02.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:02 smithi167 conmon[60316]: debug 2022-01-31T21:07:02.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:02.215852+0000) 2022-01-31T21:07:02.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:02 smithi171 conmon[51620]: debug 2022-01-31T21:07:02.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:02.570876+0000) 2022-01-31T21:07:02.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:02 smithi167 conmon[49112]: debug 2022-01-31T21:07:02.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:02.537359+0000) 2022-01-31T21:07:03.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:02 smithi171 conmon[41853]: debug 2022-01-31T21:07:02.697+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:02.697853+0000) 2022-01-31T21:07:03.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:03 smithi167 conmon[54076]: debug 2022-01-31T21:07:03.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:03.464447+0000) 2022-01-31T21:07:03.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:03 smithi167 conmon[60316]: debug 2022-01-31T21:07:03.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:03.216032+0000) 2022-01-31T21:07:03.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:03 smithi171 conmon[51620]: debug 2022-01-31T21:07:03.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:03.571057+0000) 2022-01-31T21:07:03.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:03 smithi167 conmon[49112]: debug 2022-01-31T21:07:03.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:03.537568+0000) 2022-01-31T21:07:04.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:03 smithi171 conmon[41853]: debug 2022-01-31T21:07:03.697+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:03.698038+0000) 2022-01-31T21:07:04.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:04 smithi167 conmon[54076]: debug 2022-01-31T21:07:04.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:04.464597+0000) 2022-01-31T21:07:04.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:04 smithi167 conmon[60316]: debug 2022-01-31T21:07:04.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:04.216226+0000) 2022-01-31T21:07:04.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:04 smithi171 conmon[51620]: debug 2022-01-31T21:07:04.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:04.571270+0000) 2022-01-31T21:07:04.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:04 smithi167 conmon[49112]: debug 2022-01-31T21:07:04.537+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:04.537778+0000) 2022-01-31T21:07:05.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:04 smithi171 conmon[41853]: debug 2022-01-31T21:07:04.697+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:04.698218+0000) 2022-01-31T21:07:05.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:05 smithi167 conmon[60316]: debug 2022-01-31T21:07:05.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:05.216351+0000) 2022-01-31T21:07:05.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:05 smithi167 conmon[54076]: debug 2022-01-31T21:07:05.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:05.464713+0000) 2022-01-31T21:07:05.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:05 smithi171 conmon[51620]: debug 2022-01-31T21:07:05.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:05.571425+0000) 2022-01-31T21:07:05.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:05 smithi167 conmon[49112]: debug 2022-01-31T21:07:05.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:05.537952+0000) 2022-01-31T21:07:06.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:05 smithi171 conmon[41853]: debug 2022-01-31T21:07:05.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:05.698368+0000) 2022-01-31T21:07:06.511 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:06 smithi167 conmon[49112]: debug 2022-01-31T21:07:06.256+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.258023+0000) 2022-01-31T21:07:06.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:06 smithi167 conmon[54076]: debug 2022-01-31T21:07:06.255+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.257331+0000) 2022-01-31T21:07:06.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:06 smithi167 conmon[54076]: debug 2022-01-31T21:07:06.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.464868+0000) 2022-01-31T21:07:06.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:06 smithi167 conmon[60316]: debug 2022-01-31T21:07:06.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.216519+0000) 2022-01-31T21:07:06.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:06 smithi167 conmon[60316]: debug 2022-01-31T21:07:06.255+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.256819+0000) 2022-01-31T21:07:06.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:06 smithi171 conmon[35325]: debug 2022-01-31T21:07:06.266+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104388 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:06.570 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:06 smithi171 conmon[41853]: debug 2022-01-31T21:07:06.256+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.257325+0000) 2022-01-31T21:07:06.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:06 smithi171 conmon[46715]: debug 2022-01-31T21:07:06.256+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.257708+0000) 2022-01-31T21:07:06.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:06 smithi171 conmon[51620]: debug 2022-01-31T21:07:06.255+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.256337+0000) 2022-01-31T21:07:06.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:06 smithi167 conmon[49112]: debug 2022-01-31T21:07:06.537+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.538134+0000) 2022-01-31T21:07:06.821 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:06 smithi171 conmon[41853]: debug 2022-01-31T21:07:06.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.698523+0000) 2022-01-31T21:07:06.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:06 smithi171 conmon[51620]: debug 2022-01-31T21:07:06.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:06.571554+0000) 2022-01-31T21:07:07.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:07 smithi167 conmon[54076]: debug 2022-01-31T21:07:07.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:07.465038+0000) 2022-01-31T21:07:07.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:07 smithi167 conmon[60316]: debug 2022-01-31T21:07:07.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:07.216669+0000) 2022-01-31T21:07:07.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:07 smithi171 conmon[51620]: debug 2022-01-31T21:07:07.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:07.571670+0000) 2022-01-31T21:07:07.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:07 smithi167 conmon[49112]: debug 2022-01-31T21:07:07.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:07.538278+0000) 2022-01-31T21:07:07.990 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:07 smithi171 conmon[41853]: debug 2022-01-31T21:07:07.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:07.698669+0000) 2022-01-31T21:07:08.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:07 smithi171 conmon[46715]: debug 2022-01-31T21:07:07.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:07.992802+0000) 2022-01-31T21:07:08.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:08 smithi167 conmon[54076]: debug 2022-01-31T21:07:08.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:08.465220+0000) 2022-01-31T21:07:08.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:08 smithi167 conmon[60316]: debug 2022-01-31T21:07:08.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:08.216839+0000) 2022-01-31T21:07:08.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:08 smithi171 conmon[51620]: debug 2022-01-31T21:07:08.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:08.571804+0000) 2022-01-31T21:07:08.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:08 smithi167 conmon[49112]: debug 2022-01-31T21:07:08.536+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:08.538452+0000) 2022-01-31T21:07:08.904 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:08 smithi171 conmon[35325]: debug 2022-01-31T21:07:08.626+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:07:08.905 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:08 smithi171 conmon[41853]: debug 2022-01-31T21:07:08.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:08.698801+0000) 2022-01-31T21:07:09.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:08 smithi171 conmon[46715]: debug 2022-01-31T21:07:08.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:08.993021+0000) 2022-01-31T21:07:09.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:09 smithi167 conmon[54076]: debug 2022-01-31T21:07:09.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:09.465419+0000) 2022-01-31T21:07:09.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:09 smithi167 conmon[60316]: debug 2022-01-31T21:07:09.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:09.217029+0000) 2022-01-31T21:07:09.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:09 smithi171 conmon[51620]: debug 2022-01-31T21:07:09.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:09.571902+0000) 2022-01-31T21:07:09.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:09 smithi167 conmon[49112]: debug 2022-01-31T21:07:09.538+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:09.538678+0000) 2022-01-31T21:07:09.991 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:09 smithi171 conmon[41853]: debug 2022-01-31T21:07:09.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:09.698960+0000) 2022-01-31T21:07:10.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:09 smithi171 conmon[46715]: debug 2022-01-31T21:07:09.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:09.993234+0000) 2022-01-31T21:07:10.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:10 smithi167 conmon[54076]: debug 2022-01-31T21:07:10.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:10.465591+0000) 2022-01-31T21:07:10.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:10 smithi167 conmon[60316]: debug 2022-01-31T21:07:10.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:10.217254+0000) 2022-01-31T21:07:10.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:10 smithi171 conmon[51620]: debug 2022-01-31T21:07:10.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:10.572013+0000) 2022-01-31T21:07:10.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:10 smithi167 conmon[49112]: debug 2022-01-31T21:07:10.538+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:10.538842+0000) 2022-01-31T21:07:10.991 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:10 smithi171 conmon[41853]: debug 2022-01-31T21:07:10.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:10.699141+0000) 2022-01-31T21:07:11.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:10 smithi171 conmon[46715]: debug 2022-01-31T21:07:10.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:10.993436+0000) 2022-01-31T21:07:11.512 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:11 smithi167 conmon[49112]: debug 2022-01-31T21:07:11.269+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.270515+0000) 2022-01-31T21:07:11.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:11 smithi167 conmon[60316]: debug 2022-01-31T21:07:11.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.217438+0000) 2022-01-31T21:07:11.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:11 smithi167 conmon[60316]: debug 2022-01-31T21:07:11.268+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.269702+0000) 2022-01-31T21:07:11.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:11 smithi167 conmon[54076]: debug 2022-01-31T21:07:11.268+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.270430+0000) 2022-01-31T21:07:11.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:11 smithi167 conmon[54076]: debug 2022-01-31T21:07:11.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.465783+0000) 2022-01-31T21:07:11.570 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[35325]: debug 2022-01-31T21:07:11.279+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104499 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:11.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[41853]: debug 2022-01-31T21:07:11.269+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.270625+0000) 2022-01-31T21:07:11.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[46715]: debug 2022-01-31T21:07:11.270+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.270947+0000) 2022-01-31T21:07:11.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[51620]: debug 2022-01-31T21:07:11.268+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.269637+0000) 2022-01-31T21:07:11.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:11 smithi167 conmon[49112]: debug 2022-01-31T21:07:11.538+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.539041+0000) 2022-01-31T21:07:12.136 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[41853]: debug 2022-01-31T21:07:11.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.699323+0000) 2022-01-31T21:07:12.137 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[51620]: debug 2022-01-31T21:07:11.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.572197+0000) 2022-01-31T21:07:12.137 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:11 smithi171 conmon[46715]: debug 2022-01-31T21:07:11.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:11.993626+0000) 2022-01-31T21:07:12.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:12 smithi167 conmon[54076]: debug 2022-01-31T21:07:12.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:12.465937+0000) 2022-01-31T21:07:12.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:12 smithi167 conmon[60316]: debug 2022-01-31T21:07:12.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:12.217616+0000) 2022-01-31T21:07:12.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:12 smithi171 conmon[51620]: debug 2022-01-31T21:07:12.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:12.572355+0000) 2022-01-31T21:07:12.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:12 smithi167 conmon[49112]: debug 2022-01-31T21:07:12.537+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:12.539212+0000) 2022-01-31T21:07:12.992 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:12 smithi171 conmon[41853]: debug 2022-01-31T21:07:12.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:12.699497+0000) 2022-01-31T21:07:13.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:12 smithi171 conmon[46715]: debug 2022-01-31T21:07:12.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:12.993787+0000) 2022-01-31T21:07:13.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:13 smithi167 conmon[54076]: debug 2022-01-31T21:07:13.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:13.466040+0000) 2022-01-31T21:07:13.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:13 smithi167 conmon[60316]: debug 2022-01-31T21:07:13.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:13.217747+0000) 2022-01-31T21:07:13.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:13 smithi171 conmon[51620]: debug 2022-01-31T21:07:13.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:13.572560+0000) 2022-01-31T21:07:13.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:13 smithi167 conmon[49112]: debug 2022-01-31T21:07:13.538+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:13.539397+0000) 2022-01-31T21:07:13.992 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:13 smithi171 conmon[41853]: debug 2022-01-31T21:07:13.698+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:13.699696+0000) 2022-01-31T21:07:14.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:13 smithi171 conmon[46715]: debug 2022-01-31T21:07:13.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:13.993983+0000) 2022-01-31T21:07:14.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:14 smithi167 conmon[54076]: debug 2022-01-31T21:07:14.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:14.466239+0000) 2022-01-31T21:07:14.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:14 smithi167 conmon[60316]: debug 2022-01-31T21:07:14.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:14.217910+0000) 2022-01-31T21:07:14.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:14 smithi171 conmon[51620]: debug 2022-01-31T21:07:14.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:14.572776+0000) 2022-01-31T21:07:14.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:14 smithi167 conmon[49112]: debug 2022-01-31T21:07:14.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:14.539595+0000) 2022-01-31T21:07:14.992 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:14 smithi171 conmon[41853]: debug 2022-01-31T21:07:14.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:14.699865+0000) 2022-01-31T21:07:15.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:14 smithi171 conmon[46715]: debug 2022-01-31T21:07:14.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:14.994203+0000) 2022-01-31T21:07:15.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:15 smithi167 conmon[54076]: debug 2022-01-31T21:07:15.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:15.466424+0000) 2022-01-31T21:07:15.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:15 smithi167 conmon[60316]: debug 2022-01-31T21:07:15.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:15.218072+0000) 2022-01-31T21:07:15.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:15 smithi171 conmon[51620]: debug 2022-01-31T21:07:15.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:15.572983+0000) 2022-01-31T21:07:15.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:15 smithi167 conmon[49112]: debug 2022-01-31T21:07:15.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:15.539781+0000) 2022-01-31T21:07:15.992 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:15 smithi171 conmon[41853]: debug 2022-01-31T21:07:15.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:15.699987+0000) 2022-01-31T21:07:16.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:15 smithi171 conmon[46715]: debug 2022-01-31T21:07:15.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:15.994384+0000) 2022-01-31T21:07:16.513 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:16 smithi167 conmon[49112]: debug 2022-01-31T21:07:16.282+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.283020+0000) 2022-01-31T21:07:16.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:16 smithi167 conmon[54076]: debug 2022-01-31T21:07:16.282+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.283399+0000) 2022-01-31T21:07:16.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:16 smithi167 conmon[54076]: debug 2022-01-31T21:07:16.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.466631+0000) 2022-01-31T21:07:16.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:16 smithi167 conmon[60316]: debug 2022-01-31T21:07:16.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.218227+0000) 2022-01-31T21:07:16.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:16 smithi167 conmon[60316]: debug 2022-01-31T21:07:16.283+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.283999+0000) 2022-01-31T21:07:16.571 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[35325]: debug 2022-01-31T21:07:16.292+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104608 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:16.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[46715]: debug 2022-01-31T21:07:16.282+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.282931+0000) 2022-01-31T21:07:16.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[51620]: debug 2022-01-31T21:07:16.282+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.282897+0000) 2022-01-31T21:07:16.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[41853]: debug 2022-01-31T21:07:16.283+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.284353+0000) 2022-01-31T21:07:16.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:16 smithi167 conmon[49112]: debug 2022-01-31T21:07:16.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.539984+0000) 2022-01-31T21:07:16.823 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[51620]: debug 2022-01-31T21:07:16.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.573193+0000) 2022-01-31T21:07:16.823 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[41853]: debug 2022-01-31T21:07:16.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.700190+0000) 2022-01-31T21:07:17.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:16 smithi171 conmon[46715]: debug 2022-01-31T21:07:16.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:16.994530+0000) 2022-01-31T21:07:17.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:17 smithi167 conmon[54076]: debug 2022-01-31T21:07:17.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:17.466731+0000) 2022-01-31T21:07:17.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:17 smithi167 conmon[60316]: debug 2022-01-31T21:07:17.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:17.218448+0000) 2022-01-31T21:07:17.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:17 smithi171 conmon[51620]: debug 2022-01-31T21:07:17.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:17.573350+0000) 2022-01-31T21:07:17.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:17 smithi167 conmon[49112]: debug 2022-01-31T21:07:17.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:17.540156+0000) 2022-01-31T21:07:17.992 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:17 smithi171 conmon[41853]: debug 2022-01-31T21:07:17.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:17.700343+0000) 2022-01-31T21:07:18.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:17 smithi171 conmon[46715]: debug 2022-01-31T21:07:17.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:17.994691+0000) 2022-01-31T21:07:18.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:18 smithi167 conmon[54076]: debug 2022-01-31T21:07:18.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:18.466914+0000) 2022-01-31T21:07:18.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:18 smithi167 conmon[60316]: debug 2022-01-31T21:07:18.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:18.218610+0000) 2022-01-31T21:07:18.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:18 smithi171 conmon[51620]: debug 2022-01-31T21:07:18.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:18.573548+0000) 2022-01-31T21:07:18.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:18 smithi167 conmon[49112]: debug 2022-01-31T21:07:18.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:18.540315+0000) 2022-01-31T21:07:18.993 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:18 smithi171 conmon[41853]: debug 2022-01-31T21:07:18.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:18.700496+0000) 2022-01-31T21:07:19.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:18 smithi171 conmon[46715]: debug 2022-01-31T21:07:18.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:18.994849+0000) 2022-01-31T21:07:19.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:19 smithi167 conmon[54076]: debug 2022-01-31T21:07:19.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:19.467096+0000) 2022-01-31T21:07:19.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:19 smithi167 conmon[60316]: debug 2022-01-31T21:07:19.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:19.218787+0000) 2022-01-31T21:07:19.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:19 smithi171 conmon[51620]: debug 2022-01-31T21:07:19.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:19.573715+0000) 2022-01-31T21:07:19.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:19 smithi167 conmon[49112]: debug 2022-01-31T21:07:19.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:19.540488+0000) 2022-01-31T21:07:19.993 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:19 smithi171 conmon[41853]: debug 2022-01-31T21:07:19.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:19.700672+0000) 2022-01-31T21:07:20.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:19 smithi171 conmon[46715]: debug 2022-01-31T21:07:19.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:19.995018+0000) 2022-01-31T21:07:20.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:20 smithi167 conmon[54076]: debug 2022-01-31T21:07:20.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:20.467230+0000) 2022-01-31T21:07:20.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:20 smithi167 conmon[60316]: debug 2022-01-31T21:07:20.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:20.218962+0000) 2022-01-31T21:07:20.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:20 smithi171 conmon[51620]: debug 2022-01-31T21:07:20.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:20.573871+0000) 2022-01-31T21:07:20.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:20 smithi167 conmon[49112]: debug 2022-01-31T21:07:20.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:20.540715+0000) 2022-01-31T21:07:20.993 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:20 smithi171 conmon[41853]: debug 2022-01-31T21:07:20.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:20.700835+0000) 2022-01-31T21:07:21.293 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:20 smithi171 conmon[46715]: debug 2022-01-31T21:07:20.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:20.995204+0000) 2022-01-31T21:07:21.514 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:21 smithi167 conmon[49112]: debug 2022-01-31T21:07:21.295+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.296248+0000) 2022-01-31T21:07:21.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:21 smithi167 conmon[54076]: debug 2022-01-31T21:07:21.296+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.297172+0000) 2022-01-31T21:07:21.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:21 smithi167 conmon[54076]: debug 2022-01-31T21:07:21.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.467418+0000) 2022-01-31T21:07:21.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:21 smithi167 conmon[60316]: debug 2022-01-31T21:07:21.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.219075+0000) 2022-01-31T21:07:21.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:21 smithi167 conmon[60316]: debug 2022-01-31T21:07:21.296+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.296959+0000) 2022-01-31T21:07:21.572 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[41853]: debug 2022-01-31T21:07:21.295+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.297257+0000) 2022-01-31T21:07:21.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[46715]: debug 2022-01-31T21:07:21.294+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.296125+0000) 2022-01-31T21:07:21.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[51620]: debug 2022-01-31T21:07:21.294+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.296197+0000) 2022-01-31T21:07:21.573 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[35325]: debug 2022-01-31T21:07:21.304+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104730 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:21.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:21 smithi167 conmon[49112]: debug 2022-01-31T21:07:21.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.540923+0000) 2022-01-31T21:07:21.824 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[41853]: debug 2022-01-31T21:07:21.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.701017+0000) 2022-01-31T21:07:21.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[51620]: debug 2022-01-31T21:07:21.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.574077+0000) 2022-01-31T21:07:22.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:21 smithi171 conmon[46715]: debug 2022-01-31T21:07:21.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:21.995388+0000) 2022-01-31T21:07:22.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:22 smithi167 conmon[54076]: debug 2022-01-31T21:07:22.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:22.467597+0000) 2022-01-31T21:07:22.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:22 smithi167 conmon[60316]: debug 2022-01-31T21:07:22.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:22.219268+0000) 2022-01-31T21:07:22.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:22 smithi171 conmon[51620]: debug 2022-01-31T21:07:22.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:22.574256+0000) 2022-01-31T21:07:22.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:22 smithi167 conmon[49112]: debug 2022-01-31T21:07:22.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:22.541071+0000) 2022-01-31T21:07:22.993 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:22 smithi171 conmon[41853]: debug 2022-01-31T21:07:22.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:22.701225+0000) 2022-01-31T21:07:23.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:22 smithi171 conmon[46715]: debug 2022-01-31T21:07:22.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:22.995486+0000) 2022-01-31T21:07:23.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:23 smithi167 conmon[54076]: debug 2022-01-31T21:07:23.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:23.467761+0000) 2022-01-31T21:07:23.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:23 smithi167 conmon[60316]: debug 2022-01-31T21:07:23.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:23.219424+0000) 2022-01-31T21:07:23.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:23 smithi171 conmon[51620]: debug 2022-01-31T21:07:23.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:23.574480+0000) 2022-01-31T21:07:23.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:23 smithi167 conmon[49112]: debug 2022-01-31T21:07:23.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:23.541254+0000) 2022-01-31T21:07:23.907 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:23 smithi171 conmon[35325]: debug 2022-01-31T21:07:23.626+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:07:23.908 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:23 smithi171 conmon[41853]: debug 2022-01-31T21:07:23.699+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:23.701376+0000) 2022-01-31T21:07:24.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:23 smithi171 conmon[46715]: debug 2022-01-31T21:07:23.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:23.995648+0000) 2022-01-31T21:07:24.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:24 smithi167 conmon[54076]: debug 2022-01-31T21:07:24.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:24.467942+0000) 2022-01-31T21:07:24.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:24 smithi167 conmon[60316]: debug 2022-01-31T21:07:24.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:24.219588+0000) 2022-01-31T21:07:24.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:24 smithi171 conmon[51620]: debug 2022-01-31T21:07:24.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:24.574700+0000) 2022-01-31T21:07:24.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:24 smithi167 conmon[49112]: debug 2022-01-31T21:07:24.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:24.541439+0000) 2022-01-31T21:07:24.994 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:24 smithi171 conmon[41853]: debug 2022-01-31T21:07:24.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:24.701553+0000) 2022-01-31T21:07:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:24 smithi171 conmon[46715]: debug 2022-01-31T21:07:24.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:24.995832+0000) 2022-01-31T21:07:25.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:25 smithi167 conmon[54076]: debug 2022-01-31T21:07:25.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:25.468084+0000) 2022-01-31T21:07:25.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:25 smithi167 conmon[60316]: debug 2022-01-31T21:07:25.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:25.219744+0000) 2022-01-31T21:07:25.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:25 smithi171 conmon[51620]: debug 2022-01-31T21:07:25.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:25.574876+0000) 2022-01-31T21:07:25.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:25 smithi167 conmon[49112]: debug 2022-01-31T21:07:25.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:25.541587+0000) 2022-01-31T21:07:25.907 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:25 smithi171 conmon[41853]: debug 2022-01-31T21:07:25.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:25.701779+0000) 2022-01-31T21:07:26.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:25 smithi171 conmon[46715]: debug 2022-01-31T21:07:25.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:25.996043+0000) 2022-01-31T21:07:26.515 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:26 smithi167 conmon[49112]: debug 2022-01-31T21:07:26.309+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.309861+0000) 2022-01-31T21:07:26.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:26 smithi167 conmon[54076]: debug 2022-01-31T21:07:26.308+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.309188+0000) 2022-01-31T21:07:26.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:26 smithi167 conmon[54076]: debug 2022-01-31T21:07:26.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.468282+0000) 2022-01-31T21:07:26.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:26 smithi167 conmon[60316]: debug 2022-01-31T21:07:26.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.219898+0000) 2022-01-31T21:07:26.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:26 smithi167 conmon[60316]: debug 2022-01-31T21:07:26.309+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.309673+0000) 2022-01-31T21:07:26.573 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[35325]: debug 2022-01-31T21:07:26.318+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104839 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:26.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[41853]: debug 2022-01-31T21:07:26.308+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.310542+0000) 2022-01-31T21:07:26.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[46715]: debug 2022-01-31T21:07:26.308+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.310342+0000) 2022-01-31T21:07:26.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[51620]: debug 2022-01-31T21:07:26.307+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.309043+0000) 2022-01-31T21:07:26.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:26 smithi167 conmon[49112]: debug 2022-01-31T21:07:26.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.541751+0000) 2022-01-31T21:07:26.825 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[41853]: debug 2022-01-31T21:07:26.701+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.701963+0000) 2022-01-31T21:07:26.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[51620]: debug 2022-01-31T21:07:26.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.575078+0000) 2022-01-31T21:07:27.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:26 smithi171 conmon[46715]: debug 2022-01-31T21:07:26.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:26.996251+0000) 2022-01-31T21:07:27.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:27 smithi167 conmon[54076]: debug 2022-01-31T21:07:27.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:27.468440+0000) 2022-01-31T21:07:27.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:27 smithi167 conmon[60316]: debug 2022-01-31T21:07:27.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:27.220083+0000) 2022-01-31T21:07:27.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:27 smithi171 conmon[51620]: debug 2022-01-31T21:07:27.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:27.575219+0000) 2022-01-31T21:07:27.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:27 smithi167 conmon[49112]: debug 2022-01-31T21:07:27.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:27.541896+0000) 2022-01-31T21:07:27.994 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:27 smithi171 conmon[41853]: debug 2022-01-31T21:07:27.701+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:27.702147+0000) 2022-01-31T21:07:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:27 smithi171 conmon[46715]: debug 2022-01-31T21:07:27.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:27.996410+0000) 2022-01-31T21:07:28.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:28 smithi167 conmon[54076]: debug 2022-01-31T21:07:28.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:28.468698+0000) 2022-01-31T21:07:28.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:28 smithi167 conmon[60316]: debug 2022-01-31T21:07:28.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:28.220228+0000) 2022-01-31T21:07:28.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:28 smithi171 conmon[51620]: debug 2022-01-31T21:07:28.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:28.575413+0000) 2022-01-31T21:07:28.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:28 smithi167 conmon[49112]: debug 2022-01-31T21:07:28.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:28.542085+0000) 2022-01-31T21:07:28.994 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:28 smithi171 conmon[41853]: debug 2022-01-31T21:07:28.700+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:28.702301+0000) 2022-01-31T21:07:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:28 smithi171 conmon[46715]: debug 2022-01-31T21:07:28.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:28.996573+0000) 2022-01-31T21:07:29.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:29 smithi167 conmon[54076]: debug 2022-01-31T21:07:29.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:29.468906+0000) 2022-01-31T21:07:29.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:29 smithi167 conmon[60316]: debug 2022-01-31T21:07:29.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:29.220409+0000) 2022-01-31T21:07:29.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:29 smithi171 conmon[51620]: debug 2022-01-31T21:07:29.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:29.575621+0000) 2022-01-31T21:07:29.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:29 smithi167 conmon[49112]: debug 2022-01-31T21:07:29.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:29.542305+0000) 2022-01-31T21:07:29.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:29 smithi171 conmon[41853]: debug 2022-01-31T21:07:29.701+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:29.702482+0000) 2022-01-31T21:07:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:29 smithi171 conmon[46715]: debug 2022-01-31T21:07:29.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:29.996720+0000) 2022-01-31T21:07:30.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:30 smithi167 conmon[54076]: debug 2022-01-31T21:07:30.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:30.469130+0000) 2022-01-31T21:07:30.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:30 smithi167 conmon[60316]: debug 2022-01-31T21:07:30.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:30.220605+0000) 2022-01-31T21:07:30.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:30 smithi171 conmon[51620]: debug 2022-01-31T21:07:30.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:30.575845+0000) 2022-01-31T21:07:30.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:30 smithi167 conmon[49112]: debug 2022-01-31T21:07:30.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:30.542471+0000) 2022-01-31T21:07:30.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:30 smithi171 conmon[41853]: debug 2022-01-31T21:07:30.701+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:30.702648+0000) 2022-01-31T21:07:31.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:30 smithi171 conmon[46715]: debug 2022-01-31T21:07:30.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:30.996915+0000) 2022-01-31T21:07:31.516 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:31 smithi167 conmon[49112]: debug 2022-01-31T21:07:31.323+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.323990+0000) 2022-01-31T21:07:31.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:31 smithi167 conmon[60316]: debug 2022-01-31T21:07:31.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.220780+0000) 2022-01-31T21:07:31.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:31 smithi167 conmon[60316]: debug 2022-01-31T21:07:31.322+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.323255+0000) 2022-01-31T21:07:31.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:31 smithi167 conmon[54076]: debug 2022-01-31T21:07:31.322+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.322529+0000) 2022-01-31T21:07:31.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:31 smithi167 conmon[54076]: debug 2022-01-31T21:07:31.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.469340+0000) 2022-01-31T21:07:31.574 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[35325]: debug 2022-01-31T21:07:31.331+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 104950 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:31.574 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[41853]: debug 2022-01-31T21:07:31.322+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.324100+0000) 2022-01-31T21:07:31.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[51620]: debug 2022-01-31T21:07:31.321+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.322903+0000) 2022-01-31T21:07:31.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[46715]: debug 2022-01-31T21:07:31.321+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.323127+0000) 2022-01-31T21:07:31.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:31 smithi167 conmon[49112]: debug 2022-01-31T21:07:31.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.542673+0000) 2022-01-31T21:07:31.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[51620]: debug 2022-01-31T21:07:31.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.576030+0000) 2022-01-31T21:07:31.826 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[41853]: debug 2022-01-31T21:07:31.702+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.702869+0000) 2022-01-31T21:07:32.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:31 smithi171 conmon[46715]: debug 2022-01-31T21:07:31.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:31.997072+0000) 2022-01-31T21:07:32.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:32 smithi167 conmon[54076]: debug 2022-01-31T21:07:32.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:32.469553+0000) 2022-01-31T21:07:32.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:32 smithi167 conmon[60316]: debug 2022-01-31T21:07:32.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:32.220925+0000) 2022-01-31T21:07:32.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:32 smithi171 conmon[51620]: debug 2022-01-31T21:07:32.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:32.576172+0000) 2022-01-31T21:07:32.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:32 smithi167 conmon[49112]: debug 2022-01-31T21:07:32.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:32.542789+0000) 2022-01-31T21:07:32.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:32 smithi171 conmon[41853]: debug 2022-01-31T21:07:32.702+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:32.703034+0000) 2022-01-31T21:07:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:32 smithi171 conmon[46715]: debug 2022-01-31T21:07:32.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:32.997197+0000) 2022-01-31T21:07:33.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:33 smithi167 conmon[54076]: debug 2022-01-31T21:07:33.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:33.469679+0000) 2022-01-31T21:07:33.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:33 smithi167 conmon[60316]: debug 2022-01-31T21:07:33.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:33.221078+0000) 2022-01-31T21:07:33.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:33 smithi171 conmon[51620]: debug 2022-01-31T21:07:33.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:33.576325+0000) 2022-01-31T21:07:33.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:33 smithi167 conmon[49112]: debug 2022-01-31T21:07:33.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:33.542965+0000) 2022-01-31T21:07:33.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:33 smithi171 conmon[41853]: debug 2022-01-31T21:07:33.702+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:33.703214+0000) 2022-01-31T21:07:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:33 smithi171 conmon[46715]: debug 2022-01-31T21:07:33.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:33.997353+0000) 2022-01-31T21:07:34.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:34 smithi167 conmon[54076]: debug 2022-01-31T21:07:34.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:34.469810+0000) 2022-01-31T21:07:34.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:34 smithi167 conmon[60316]: debug 2022-01-31T21:07:34.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:34.221298+0000) 2022-01-31T21:07:34.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:34 smithi171 conmon[51620]: debug 2022-01-31T21:07:34.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:34.576495+0000) 2022-01-31T21:07:34.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:34 smithi167 conmon[49112]: debug 2022-01-31T21:07:34.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:34.543157+0000) 2022-01-31T21:07:34.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:34 smithi171 conmon[41853]: debug 2022-01-31T21:07:34.702+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:34.703390+0000) 2022-01-31T21:07:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:34 smithi171 conmon[46715]: debug 2022-01-31T21:07:34.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:34.997542+0000) 2022-01-31T21:07:35.516 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:35 smithi167 conmon[60316]: debug 2022-01-31T21:07:35.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:35.221456+0000) 2022-01-31T21:07:35.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:35 smithi167 conmon[54076]: debug 2022-01-31T21:07:35.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:35.469977+0000) 2022-01-31T21:07:35.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:35 smithi171 conmon[51620]: debug 2022-01-31T21:07:35.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:35.576650+0000) 2022-01-31T21:07:35.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:35 smithi167 conmon[49112]: debug 2022-01-31T21:07:35.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:35.543335+0000) 2022-01-31T21:07:35.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:35 smithi171 conmon[41853]: debug 2022-01-31T21:07:35.702+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:35.703550+0000) 2022-01-31T21:07:36.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:35 smithi171 conmon[46715]: debug 2022-01-31T21:07:35.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:35.997721+0000) 2022-01-31T21:07:36.516 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:36 smithi167 conmon[49112]: debug 2022-01-31T21:07:36.335+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.335891+0000) 2022-01-31T21:07:36.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:36 smithi167 conmon[54076]: debug 2022-01-31T21:07:36.335+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.335655+0000) 2022-01-31T21:07:36.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:36 smithi167 conmon[54076]: debug 2022-01-31T21:07:36.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.470177+0000) 2022-01-31T21:07:36.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:36 smithi167 conmon[60316]: debug 2022-01-31T21:07:36.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.221607+0000) 2022-01-31T21:07:36.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:36 smithi167 conmon[60316]: debug 2022-01-31T21:07:36.334+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.335353+0000) 2022-01-31T21:07:36.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[35325]: debug 2022-01-31T21:07:36.345+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105059 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[41853]: debug 2022-01-31T21:07:36.334+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.335237+0000) 2022-01-31T21:07:36.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[46715]: debug 2022-01-31T21:07:36.334+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.335125+0000) 2022-01-31T21:07:36.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[51620]: debug 2022-01-31T21:07:36.335+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.335784+0000) 2022-01-31T21:07:36.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[51620]: debug 2022-01-31T21:07:36.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.576774+0000) 2022-01-31T21:07:36.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:36 smithi167 conmon[49112]: debug 2022-01-31T21:07:36.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.543489+0000) 2022-01-31T21:07:36.996 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[41853]: debug 2022-01-31T21:07:36.703+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.703738+0000) 2022-01-31T21:07:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:36 smithi171 conmon[46715]: debug 2022-01-31T21:07:36.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:36.997905+0000) 2022-01-31T21:07:37.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:37 smithi167 conmon[54076]: debug 2022-01-31T21:07:37.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:37.470335+0000) 2022-01-31T21:07:37.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:37 smithi167 conmon[60316]: debug 2022-01-31T21:07:37.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:37.221790+0000) 2022-01-31T21:07:37.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:37 smithi171 conmon[51620]: debug 2022-01-31T21:07:37.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:37.576914+0000) 2022-01-31T21:07:37.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:37 smithi167 conmon[49112]: debug 2022-01-31T21:07:37.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:37.543651+0000) 2022-01-31T21:07:37.996 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:37 smithi171 conmon[41853]: debug 2022-01-31T21:07:37.703+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:37.703862+0000) 2022-01-31T21:07:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:37 smithi171 conmon[46715]: debug 2022-01-31T21:07:37.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:37.998076+0000) 2022-01-31T21:07:38.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:38 smithi167 conmon[60316]: debug 2022-01-31T21:07:38.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:38.221936+0000) 2022-01-31T21:07:38.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:38 smithi167 conmon[54076]: debug 2022-01-31T21:07:38.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:38.470527+0000) 2022-01-31T21:07:38.626 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:38 smithi171 conmon[51620]: debug 2022-01-31T21:07:38.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:38.577107+0000) 2022-01-31T21:07:38.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:38 smithi167 conmon[49112]: debug 2022-01-31T21:07:38.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:38.543830+0000) 2022-01-31T21:07:38.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:38 smithi171 conmon[35325]: debug 2022-01-31T21:07:38.627+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:07:38.910 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:38 smithi171 conmon[41853]: debug 2022-01-31T21:07:38.703+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:38.703987+0000) 2022-01-31T21:07:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:38 smithi171 conmon[46715]: debug 2022-01-31T21:07:38.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:38.998267+0000) 2022-01-31T21:07:39.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:39 smithi167 conmon[60316]: debug 2022-01-31T21:07:39.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:39.222112+0000) 2022-01-31T21:07:39.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:39 smithi167 conmon[54076]: debug 2022-01-31T21:07:39.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:39.470717+0000) 2022-01-31T21:07:39.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:39 smithi171 conmon[51620]: debug 2022-01-31T21:07:39.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:39.577309+0000) 2022-01-31T21:07:39.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:39 smithi167 conmon[49112]: debug 2022-01-31T21:07:39.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:39.543989+0000) 2022-01-31T21:07:39.996 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:39 smithi171 conmon[41853]: debug 2022-01-31T21:07:39.703+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:39.704169+0000) 2022-01-31T21:07:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:39 smithi171 conmon[46715]: debug 2022-01-31T21:07:39.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:39.998457+0000) 2022-01-31T21:07:40.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:40 smithi167 conmon[60316]: debug 2022-01-31T21:07:40.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:40.222274+0000) 2022-01-31T21:07:40.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:40 smithi167 conmon[54076]: debug 2022-01-31T21:07:40.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:40.470903+0000) 2022-01-31T21:07:40.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:40 smithi171 conmon[51620]: debug 2022-01-31T21:07:40.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:40.577496+0000) 2022-01-31T21:07:40.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:40 smithi167 conmon[49112]: debug 2022-01-31T21:07:40.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:40.544178+0000) 2022-01-31T21:07:40.996 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:40 smithi171 conmon[41853]: debug 2022-01-31T21:07:40.703+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:40.704386+0000) 2022-01-31T21:07:41.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:40 smithi171 conmon[46715]: debug 2022-01-31T21:07:40.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:40.998639+0000) 2022-01-31T21:07:41.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[41853]: debug 2022-01-31T21:07:41.347+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.348072+0000) 2022-01-31T21:07:41.517 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:41 smithi167 conmon[49112]: debug 2022-01-31T21:07:41.347+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.349272+0000) 2022-01-31T21:07:41.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:41 smithi167 conmon[60316]: debug 2022-01-31T21:07:41.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.222420+0000) 2022-01-31T21:07:41.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:41 smithi167 conmon[60316]: debug 2022-01-31T21:07:41.348+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.349526+0000) 2022-01-31T21:07:41.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:41 smithi167 conmon[54076]: debug 2022-01-31T21:07:41.348+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.350292+0000) 2022-01-31T21:07:41.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:41 smithi167 conmon[54076]: debug 2022-01-31T21:07:41.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.471078+0000) 2022-01-31T21:07:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[46715]: debug 2022-01-31T21:07:41.348+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.349163+0000) 2022-01-31T21:07:41.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[51620]: debug 2022-01-31T21:07:41.349+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.349929+0000) 2022-01-31T21:07:41.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[51620]: debug 2022-01-31T21:07:41.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.577678+0000) 2022-01-31T21:07:41.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[35325]: debug 2022-01-31T21:07:41.362+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105170 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:41.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:41 smithi167 conmon[49112]: debug 2022-01-31T21:07:41.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.544360+0000) 2022-01-31T21:07:41.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[41853]: debug 2022-01-31T21:07:41.703+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.704538+0000) 2022-01-31T21:07:42.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:41 smithi171 conmon[46715]: debug 2022-01-31T21:07:41.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:41.998795+0000) 2022-01-31T21:07:42.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:42 smithi167 conmon[54076]: debug 2022-01-31T21:07:42.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:42.471160+0000) 2022-01-31T21:07:42.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:42 smithi167 conmon[60316]: debug 2022-01-31T21:07:42.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:42.222614+0000) 2022-01-31T21:07:42.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:42 smithi171 conmon[51620]: debug 2022-01-31T21:07:42.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:42.577850+0000) 2022-01-31T21:07:42.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:42 smithi167 conmon[49112]: debug 2022-01-31T21:07:42.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:42.544456+0000) 2022-01-31T21:07:42.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:42 smithi171 conmon[41853]: debug 2022-01-31T21:07:42.704+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:42.704719+0000) 2022-01-31T21:07:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:42 smithi171 conmon[46715]: debug 2022-01-31T21:07:42.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:42.998952+0000) 2022-01-31T21:07:43.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:43 smithi167 conmon[54076]: debug 2022-01-31T21:07:43.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:43.471322+0000) 2022-01-31T21:07:43.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:43 smithi167 conmon[60316]: debug 2022-01-31T21:07:43.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:43.222755+0000) 2022-01-31T21:07:43.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:43 smithi171 conmon[51620]: debug 2022-01-31T21:07:43.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:43.578011+0000) 2022-01-31T21:07:43.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:43 smithi167 conmon[49112]: debug 2022-01-31T21:07:43.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:43.544667+0000) 2022-01-31T21:07:43.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:43 smithi171 conmon[41853]: debug 2022-01-31T21:07:43.704+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:43.704849+0000) 2022-01-31T21:07:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:43 smithi171 conmon[46715]: debug 2022-01-31T21:07:43.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:43.999124+0000) 2022-01-31T21:07:44.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:44 smithi167 conmon[54076]: debug 2022-01-31T21:07:44.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:44.471533+0000) 2022-01-31T21:07:44.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:44 smithi167 conmon[60316]: debug 2022-01-31T21:07:44.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:44.222964+0000) 2022-01-31T21:07:44.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:44 smithi171 conmon[51620]: debug 2022-01-31T21:07:44.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:44.578196+0000) 2022-01-31T21:07:44.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:44 smithi167 conmon[49112]: debug 2022-01-31T21:07:44.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:44.544849+0000) 2022-01-31T21:07:44.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:44 smithi171 conmon[41853]: debug 2022-01-31T21:07:44.704+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:44.705005+0000) 2022-01-31T21:07:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:44 smithi171 conmon[46715]: debug 2022-01-31T21:07:44.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:44.999314+0000) 2022-01-31T21:07:45.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:45 smithi167 conmon[54076]: debug 2022-01-31T21:07:45.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:45.471690+0000) 2022-01-31T21:07:45.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:45 smithi167 conmon[60316]: debug 2022-01-31T21:07:45.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:45.223168+0000) 2022-01-31T21:07:45.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:45 smithi171 conmon[51620]: debug 2022-01-31T21:07:45.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:45.578400+0000) 2022-01-31T21:07:45.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:45 smithi167 conmon[49112]: debug 2022-01-31T21:07:45.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:45.545035+0000) 2022-01-31T21:07:45.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:45 smithi171 conmon[41853]: debug 2022-01-31T21:07:45.704+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:45.705156+0000) 2022-01-31T21:07:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:45 smithi171 conmon[46715]: debug 2022-01-31T21:07:45.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:45.999472+0000) 2022-01-31T21:07:46.518 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:46 smithi167 conmon[49112]: debug 2022-01-31T21:07:46.364+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.365645+0000) 2022-01-31T21:07:46.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:46 smithi167 conmon[54076]: debug 2022-01-31T21:07:46.365+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.367332+0000) 2022-01-31T21:07:46.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:46 smithi167 conmon[54076]: debug 2022-01-31T21:07:46.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.471884+0000) 2022-01-31T21:07:46.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:46 smithi167 conmon[60316]: debug 2022-01-31T21:07:46.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.223325+0000) 2022-01-31T21:07:46.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:46 smithi167 conmon[60316]: debug 2022-01-31T21:07:46.365+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.366658+0000) 2022-01-31T21:07:46.703 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[35325]: debug 2022-01-31T21:07:46.375+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105279 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:46.704 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[41853]: debug 2022-01-31T21:07:46.366+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.366875+0000) 2022-01-31T21:07:46.704 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[46715]: debug 2022-01-31T21:07:46.364+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.365295+0000) 2022-01-31T21:07:46.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[51620]: debug 2022-01-31T21:07:46.365+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.366289+0000) 2022-01-31T21:07:46.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[51620]: debug 2022-01-31T21:07:46.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.578592+0000) 2022-01-31T21:07:46.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:46 smithi167 conmon[49112]: debug 2022-01-31T21:07:46.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.545223+0000) 2022-01-31T21:07:46.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[41853]: debug 2022-01-31T21:07:46.704+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.705344+0000) 2022-01-31T21:07:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:46 smithi171 conmon[46715]: debug 2022-01-31T21:07:46.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:46.999634+0000) 2022-01-31T21:07:47.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:47 smithi167 conmon[54076]: debug 2022-01-31T21:07:47.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:47.472064+0000) 2022-01-31T21:07:47.519 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:47 smithi167 conmon[60316]: debug 2022-01-31T21:07:47.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:47.223474+0000) 2022-01-31T21:07:47.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:47 smithi171 conmon[51620]: debug 2022-01-31T21:07:47.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:47.578765+0000) 2022-01-31T21:07:47.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:47 smithi167 conmon[49112]: debug 2022-01-31T21:07:47.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:47.545379+0000) 2022-01-31T21:07:47.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:47 smithi171 conmon[41853]: debug 2022-01-31T21:07:47.705+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:47.705501+0000) 2022-01-31T21:07:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:47 smithi171 conmon[46715]: debug 2022-01-31T21:07:47.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:47.999794+0000) 2022-01-31T21:07:48.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:48 smithi167 conmon[60316]: debug 2022-01-31T21:07:48.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:48.223642+0000) 2022-01-31T21:07:48.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:48 smithi167 conmon[54076]: debug 2022-01-31T21:07:48.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:48.472263+0000) 2022-01-31T21:07:48.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:48 smithi171 conmon[51620]: debug 2022-01-31T21:07:48.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:48.578950+0000) 2022-01-31T21:07:48.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:48 smithi167 conmon[49112]: debug 2022-01-31T21:07:48.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:48.545605+0000) 2022-01-31T21:07:48.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:48 smithi171 conmon[41853]: debug 2022-01-31T21:07:48.705+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:48.705691+0000) 2022-01-31T21:07:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:49 smithi171 conmon[46715]: debug 2022-01-31T21:07:48.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:49.000000+0000) 2022-01-31T21:07:49.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:49 smithi167 conmon[54076]: debug 2022-01-31T21:07:49.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:49.472421+0000) 2022-01-31T21:07:49.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:49 smithi167 conmon[60316]: debug 2022-01-31T21:07:49.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:49.223799+0000) 2022-01-31T21:07:49.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:49 smithi171 conmon[51620]: debug 2022-01-31T21:07:49.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:49.579135+0000) 2022-01-31T21:07:49.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:49 smithi167 conmon[49112]: debug 2022-01-31T21:07:49.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:49.545815+0000) 2022-01-31T21:07:49.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:49 smithi171 conmon[41853]: debug 2022-01-31T21:07:49.705+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:49.705850+0000) 2022-01-31T21:07:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:50 smithi171 conmon[46715]: debug 2022-01-31T21:07:49.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:50.000181+0000) 2022-01-31T21:07:50.519 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:50 smithi167 conmon[60316]: debug 2022-01-31T21:07:50.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:50.223998+0000) 2022-01-31T21:07:50.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:50 smithi167 conmon[54076]: debug 2022-01-31T21:07:50.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:50.472516+0000) 2022-01-31T21:07:50.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:50 smithi171 conmon[51620]: debug 2022-01-31T21:07:50.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:50.579323+0000) 2022-01-31T21:07:50.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:50 smithi167 conmon[49112]: debug 2022-01-31T21:07:50.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:50.545960+0000) 2022-01-31T21:07:50.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:50 smithi171 conmon[41853]: debug 2022-01-31T21:07:50.705+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:50.706045+0000) 2022-01-31T21:07:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[46715]: debug 2022-01-31T21:07:50.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.000332+0000) 2022-01-31T21:07:51.519 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:51 smithi167 conmon[49112]: debug 2022-01-31T21:07:51.378+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.380401+0000) 2022-01-31T21:07:51.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:51 smithi167 conmon[54076]: debug 2022-01-31T21:07:51.377+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.379459+0000) 2022-01-31T21:07:51.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:51 smithi167 conmon[54076]: debug 2022-01-31T21:07:51.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.472621+0000) 2022-01-31T21:07:51.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:51 smithi167 conmon[60316]: debug 2022-01-31T21:07:51.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.224149+0000) 2022-01-31T21:07:51.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:51 smithi167 conmon[60316]: debug 2022-01-31T21:07:51.378+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.380092+0000) 2022-01-31T21:07:51.704 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[35325]: debug 2022-01-31T21:07:51.388+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105390 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:51.705 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[41853]: debug 2022-01-31T21:07:51.378+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.379593+0000) 2022-01-31T21:07:51.705 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[46715]: debug 2022-01-31T21:07:51.378+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.379098+0000) 2022-01-31T21:07:51.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[51620]: debug 2022-01-31T21:07:51.378+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.379201+0000) 2022-01-31T21:07:51.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[51620]: debug 2022-01-31T21:07:51.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.579503+0000) 2022-01-31T21:07:51.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:51 smithi167 conmon[49112]: debug 2022-01-31T21:07:51.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.546073+0000) 2022-01-31T21:07:51.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:51 smithi171 conmon[41853]: debug 2022-01-31T21:07:51.705+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:51.706259+0000) 2022-01-31T21:07:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:52 smithi171 conmon[46715]: debug 2022-01-31T21:07:52.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:52.000519+0000) 2022-01-31T21:07:52.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:52 smithi167 conmon[54076]: debug 2022-01-31T21:07:52.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:52.472733+0000) 2022-01-31T21:07:52.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:52 smithi167 conmon[60316]: debug 2022-01-31T21:07:52.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:52.224303+0000) 2022-01-31T21:07:52.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:52 smithi171 conmon[51620]: debug 2022-01-31T21:07:52.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:52.579661+0000) 2022-01-31T21:07:52.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:52 smithi167 conmon[49112]: debug 2022-01-31T21:07:52.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:52.546217+0000) 2022-01-31T21:07:52.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:52 smithi171 conmon[41853]: debug 2022-01-31T21:07:52.706+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:52.706431+0000) 2022-01-31T21:07:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:53 smithi171 conmon[46715]: debug 2022-01-31T21:07:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:53 smithi171 conmon[46715]: 2022-01-31T21:07:52.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:53.000687+0000) 2022-01-31T21:07:53.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:53 smithi167 conmon[54076]: debug 2022-01-31T21:07:53.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:53.472936+0000) 2022-01-31T21:07:53.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:53 smithi167 conmon[60316]: debug 2022-01-31T21:07:53.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:53.224457+0000) 2022-01-31T21:07:53.627 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:53 smithi171 conmon[51620]: debug 2022-01-31T21:07:53.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:53.579837+0000) 2022-01-31T21:07:53.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:53 smithi167 conmon[49112]: debug 2022-01-31T21:07:53.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:53.546434+0000) 2022-01-31T21:07:53.912 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:53 smithi171 conmon[35325]: debug 2022-01-31T21:07:53.628+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:07:53.913 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:53 smithi171 conmon[41853]: debug 2022-01-31T21:07:53.706+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:53.706652+0000) 2022-01-31T21:07:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:54 smithi171 conmon[46715]: debug 2022-01-31T21:07:54.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:54.000887+0000) 2022-01-31T21:07:54.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:54 smithi167 conmon[54076]: debug 2022-01-31T21:07:54.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:54.473100+0000) 2022-01-31T21:07:54.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:54 smithi167 conmon[60316]: debug 2022-01-31T21:07:54.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:54.224679+0000) 2022-01-31T21:07:54.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:54 smithi171 conmon[51620]: debug 2022-01-31T21:07:54.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:54.580005+0000) 2022-01-31T21:07:54.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:54 smithi167 conmon[49112]: debug 2022-01-31T21:07:54.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:54.546624+0000) 2022-01-31T21:07:54.999 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:54 smithi171 conmon[41853]: debug 2022-01-31T21:07:54.706+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:54.706805+0000) 2022-01-31T21:07:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:55 smithi171 conmon[46715]: debug 2022-01-31T21:07:55.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:55.001046+0000) 2022-01-31T21:07:55.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:55 smithi167 conmon[54076]: debug 2022-01-31T21:07:55.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:55.473293+0000) 2022-01-31T21:07:55.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:55 smithi167 conmon[60316]: debug 2022-01-31T21:07:55.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:55.224859+0000) 2022-01-31T21:07:55.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:55 smithi171 conmon[51620]: debug 2022-01-31T21:07:55.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:55.580191+0000) 2022-01-31T21:07:55.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:55 smithi167 conmon[49112]: debug 2022-01-31T21:07:55.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:55.546787+0000) 2022-01-31T21:07:55.999 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:55 smithi171 conmon[41853]: debug 2022-01-31T21:07:55.706+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:55.706988+0000) 2022-01-31T21:07:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[46715]: debug 2022-01-31T21:07:56.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.001199+0000) 2022-01-31T21:07:56.520 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:56 smithi167 conmon[49112]: debug 2022-01-31T21:07:56.390+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.392039+0000) 2022-01-31T21:07:56.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:56 smithi167 conmon[54076]: debug 2022-01-31T21:07:56.391+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.392830+0000) 2022-01-31T21:07:56.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:56 smithi167 conmon[54076]: debug 2022-01-31T21:07:56.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.473481+0000) 2022-01-31T21:07:56.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:56 smithi167 conmon[60316]: debug 2022-01-31T21:07:56.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.225063+0000) 2022-01-31T21:07:56.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:56 smithi167 conmon[60316]: debug 2022-01-31T21:07:56.391+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.393103+0000) 2022-01-31T21:07:56.705 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[35325]: debug 2022-01-31T21:07:56.401+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105499 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:07:56.706 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[41853]: debug 2022-01-31T21:07:56.391+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.392492+0000) 2022-01-31T21:07:56.706 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[46715]: debug 2022-01-31T21:07:56.391+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.391910+0000) 2022-01-31T21:07:56.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[51620]: debug 2022-01-31T21:07:56.391+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.392054+0000) 2022-01-31T21:07:56.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[51620]: debug 2022-01-31T21:07:56.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.580349+0000) 2022-01-31T21:07:56.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:56 smithi167 conmon[49112]: debug 2022-01-31T21:07:56.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.546968+0000) 2022-01-31T21:07:56.999 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:56 smithi171 conmon[41853]: debug 2022-01-31T21:07:56.706+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:56.707139+0000) 2022-01-31T21:07:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:57 smithi171 conmon[46715]: debug 2022-01-31T21:07:57.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:57.001402+0000) 2022-01-31T21:07:57.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:57 smithi167 conmon[60316]: debug 2022-01-31T21:07:57.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:57.225250+0000) 2022-01-31T21:07:57.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:57 smithi167 conmon[54076]: debug 2022-01-31T21:07:57.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:57.473531+0000) 2022-01-31T21:07:57.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:57 smithi171 conmon[51620]: debug 2022-01-31T21:07:57.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:57.580509+0000) 2022-01-31T21:07:57.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:57 smithi167 conmon[49112]: debug 2022-01-31T21:07:57.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:57.547114+0000) 2022-01-31T21:07:57.999 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:57 smithi171 conmon[41853]: debug 2022-01-31T21:07:57.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:57.707298+0000) 2022-01-31T21:07:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:58 smithi171 conmon[46715]: debug 2022-01-31T21:07:58.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:58.001554+0000) 2022-01-31T21:07:58.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:58 smithi167 conmon[54076]: debug 2022-01-31T21:07:58.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:58.473646+0000) 2022-01-31T21:07:58.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:58 smithi167 conmon[60316]: debug 2022-01-31T21:07:58.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:58.225428+0000) 2022-01-31T21:07:58.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:58 smithi171 conmon[51620]: debug 2022-01-31T21:07:58.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:58.580673+0000) 2022-01-31T21:07:58.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:58 smithi167 conmon[49112]: debug 2022-01-31T21:07:58.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:58.547289+0000) 2022-01-31T21:07:59.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:58 smithi171 conmon[41853]: debug 2022-01-31T21:07:58.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:58.707481+0000) 2022-01-31T21:07:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:07:59 smithi171 conmon[46715]: debug 2022-01-31T21:07:59.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:59.001741+0000) 2022-01-31T21:07:59.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:07:59 smithi167 conmon[60316]: debug 2022-01-31T21:07:59.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:59.225612+0000) 2022-01-31T21:07:59.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:07:59 smithi167 conmon[54076]: debug 2022-01-31T21:07:59.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:59.473822+0000) 2022-01-31T21:07:59.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:07:59 smithi171 conmon[51620]: debug 2022-01-31T21:07:59.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:59.580842+0000) 2022-01-31T21:07:59.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:07:59 smithi167 conmon[49112]: debug 2022-01-31T21:07:59.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:59.547519+0000) 2022-01-31T21:08:00.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:07:59 smithi171 conmon[41853]: debug 2022-01-31T21:07:59.706+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:07:59.707625+0000) 2022-01-31T21:08:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:00 smithi171 conmon[46715]: debug 2022-01-31T21:08:00.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:00.001973+0000) 2022-01-31T21:08:00.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:00 smithi167 conmon[60316]: debug 2022-01-31T21:08:00.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:00.225825+0000) 2022-01-31T21:08:00.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:00 smithi167 conmon[54076]: debug 2022-01-31T21:08:00.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:00.474010+0000) 2022-01-31T21:08:00.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:00 smithi171 conmon[51620]: debug 2022-01-31T21:08:00.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:00.581000+0000) 2022-01-31T21:08:00.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:00 smithi167 conmon[49112]: debug 2022-01-31T21:08:00.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:00.547722+0000) 2022-01-31T21:08:01.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:00 smithi171 conmon[41853]: debug 2022-01-31T21:08:00.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:00.707748+0000) 2022-01-31T21:08:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[46715]: debug 2022-01-31T21:08:01.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.002155+0000) 2022-01-31T21:08:01.521 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:01 smithi167 conmon[49112]: debug 2022-01-31T21:08:01.404+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.406311+0000) 2022-01-31T21:08:01.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:01 smithi167 conmon[60316]: debug 2022-01-31T21:08:01.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.226034+0000) 2022-01-31T21:08:01.522 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:01 smithi167 conmon[60316]: debug 2022-01-31T21:08:01.403+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.405375+0000) 2022-01-31T21:08:01.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:01 smithi167 conmon[54076]: debug 2022-01-31T21:08:01.404+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.405859+0000) 2022-01-31T21:08:01.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:01 smithi167 conmon[54076]: debug 2022-01-31T21:08:01.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.474175+0000) 2022-01-31T21:08:01.706 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[35325]: debug 2022-01-31T21:08:01.414+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105610 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:01.706 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[46715]: debug 2022-01-31T21:08:01.404+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.405205+0000) 2022-01-31T21:08:01.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[41853]: debug 2022-01-31T21:08:01.404+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.404823+0000) 2022-01-31T21:08:01.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[51620]: debug 2022-01-31T21:08:01.404+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.405685+0000) 2022-01-31T21:08:01.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[51620]: debug 2022-01-31T21:08:01.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.581233+0000) 2022-01-31T21:08:01.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:01 smithi167 conmon[49112]: debug 2022-01-31T21:08:01.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.547931+0000) 2022-01-31T21:08:02.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:01 smithi171 conmon[41853]: debug 2022-01-31T21:08:01.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:01.707867+0000) 2022-01-31T21:08:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:02 smithi171 conmon[46715]: debug 2022-01-31T21:08:02.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:02.002301+0000) 2022-01-31T21:08:02.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:02 smithi167 conmon[54076]: debug 2022-01-31T21:08:02.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:02.474323+0000) 2022-01-31T21:08:02.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:02 smithi167 conmon[60316]: debug 2022-01-31T21:08:02.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:02.226211+0000) 2022-01-31T21:08:02.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:02 smithi171 conmon[51620]: debug 2022-01-31T21:08:02.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:02.581419+0000) 2022-01-31T21:08:02.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:02 smithi167 conmon[49112]: debug 2022-01-31T21:08:02.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:02.548081+0000) 2022-01-31T21:08:03.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:02 smithi171 conmon[41853]: debug 2022-01-31T21:08:02.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:02.707987+0000) 2022-01-31T21:08:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:03 smithi171 conmon[46715]: debug 2022-01-31T21:08:03.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:03.002464+0000) 2022-01-31T21:08:03.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:03 smithi167 conmon[60316]: debug 2022-01-31T21:08:03.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:03.226389+0000) 2022-01-31T21:08:03.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:03 smithi171 conmon[51620]: debug 2022-01-31T21:08:03.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:03.581628+0000) 2022-01-31T21:08:03.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:03 smithi167 conmon[49112]: debug 2022-01-31T21:08:03.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:03.548245+0000) 2022-01-31T21:08:03.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:03 smithi167 conmon[54076]: debug 2022-01-31T21:08:03.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:03.474457+0000) 2022-01-31T21:08:04.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:03 smithi171 conmon[41853]: debug 2022-01-31T21:08:03.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:03.708085+0000) 2022-01-31T21:08:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:04 smithi171 conmon[46715]: debug 2022-01-31T21:08:04.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:04.002624+0000) 2022-01-31T21:08:04.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:04 smithi167 conmon[60316]: debug 2022-01-31T21:08:04.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:04.226545+0000) 2022-01-31T21:08:04.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:04 smithi171 conmon[51620]: debug 2022-01-31T21:08:04.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:04.581829+0000) 2022-01-31T21:08:04.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:04 smithi167 conmon[54076]: debug 2022-01-31T21:08:04.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:04.474621+0000) 2022-01-31T21:08:04.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:04 smithi167 conmon[49112]: debug 2022-01-31T21:08:04.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:04.548412+0000) 2022-01-31T21:08:05.001 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:04 smithi171 conmon[41853]: debug 2022-01-31T21:08:04.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:04.708190+0000) 2022-01-31T21:08:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:05 smithi171 conmon[46715]: debug 2022-01-31T21:08:05.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:05.002790+0000) 2022-01-31T21:08:05.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:05 smithi167 conmon[60316]: debug 2022-01-31T21:08:05.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:05.226711+0000) 2022-01-31T21:08:05.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:05 smithi171 conmon[51620]: debug 2022-01-31T21:08:05.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:05.581989+0000) 2022-01-31T21:08:05.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:05 smithi167 conmon[54076]: debug 2022-01-31T21:08:05.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:05.474771+0000) 2022-01-31T21:08:05.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:05 smithi167 conmon[49112]: debug 2022-01-31T21:08:05.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:05.548598+0000) 2022-01-31T21:08:06.001 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:05 smithi171 conmon[41853]: debug 2022-01-31T21:08:05.708+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:05.708342+0000) 2022-01-31T21:08:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[46715]: debug 2022-01-31T21:08:06.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.002992+0000) 2022-01-31T21:08:06.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:06 smithi167 conmon[60316]: debug 2022-01-31T21:08:06.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.226914+0000) 2022-01-31T21:08:06.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:06 smithi167 conmon[49112]: debug 2022-01-31T21:08:06.418+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.418739+0000) 2022-01-31T21:08:06.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:06 smithi167 conmon[49112]: debug 2022-01-31T21:08:06.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.548786+0000) 2022-01-31T21:08:06.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:06 smithi167 conmon[54076]: debug 2022-01-31T21:08:06.418+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.419055+0000) 2022-01-31T21:08:06.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:06 smithi167 conmon[54076]: debug 2022-01-31T21:08:06.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.474981+0000) 2022-01-31T21:08:06.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:06 smithi167 conmon[60316]: debug 2022-01-31T21:08:06.418+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.418496+0000) 2022-01-31T21:08:06.707 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[35325]: debug 2022-01-31T21:08:06.428+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105719 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:06.707 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[46715]: debug 2022-01-31T21:08:06.418+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.419599+0000) 2022-01-31T21:08:06.708 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[41853]: debug 2022-01-31T21:08:06.417+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.417915+0000) 2022-01-31T21:08:06.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[51620]: debug 2022-01-31T21:08:06.417+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.418731+0000) 2022-01-31T21:08:06.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[51620]: debug 2022-01-31T21:08:06.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.582170+0000) 2022-01-31T21:08:07.001 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:06 smithi171 conmon[41853]: debug 2022-01-31T21:08:06.708+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:06.708498+0000) 2022-01-31T21:08:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:07 smithi171 conmon[46715]: debug 2022-01-31T21:08:07.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:07.003156+0000) 2022-01-31T21:08:07.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:07 smithi167 conmon[60316]: debug 2022-01-31T21:08:07.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:07.227072+0000) 2022-01-31T21:08:07.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:07 smithi171 conmon[51620]: debug 2022-01-31T21:08:07.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:07.582315+0000) 2022-01-31T21:08:07.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:07 smithi167 conmon[49112]: debug 2022-01-31T21:08:07.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:07.548966+0000) 2022-01-31T21:08:07.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:07 smithi167 conmon[54076]: debug 2022-01-31T21:08:07.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:07.475146+0000) 2022-01-31T21:08:08.001 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:07 smithi171 conmon[41853]: debug 2022-01-31T21:08:07.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:07.708647+0000) 2022-01-31T21:08:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:08 smithi171 conmon[46715]: debug 2022-01-31T21:08:08.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:08.003306+0000) 2022-01-31T21:08:08.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:08 smithi167 conmon[60316]: debug 2022-01-31T21:08:08.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:08.227238+0000) 2022-01-31T21:08:08.628 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:08 smithi171 conmon[51620]: debug 2022-01-31T21:08:08.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:08.582517+0000) 2022-01-31T21:08:08.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:08 smithi167 conmon[49112]: debug 2022-01-31T21:08:08.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:08.549151+0000) 2022-01-31T21:08:08.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:08 smithi167 conmon[54076]: debug 2022-01-31T21:08:08.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:08.475344+0000) 2022-01-31T21:08:08.915 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:08 smithi171 conmon[35325]: debug 2022-01-31T21:08:08.628+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:08:08.915 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:08 smithi171 conmon[41853]: debug 2022-01-31T21:08:08.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:08.708819+0000) 2022-01-31T21:08:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:09 smithi171 conmon[46715]: debug 2022-01-31T21:08:09.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:09.003510+0000) 2022-01-31T21:08:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:09 smithi167 conmon[60316]: debug 2022-01-31T21:08:09.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:09.227449+0000) 2022-01-31T21:08:09.664 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:09 smithi171 conmon[51620]: debug 2022-01-31T21:08:09.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:09.582702+0000) 2022-01-31T21:08:09.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:09 smithi167 conmon[49112]: debug 2022-01-31T21:08:09.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:09.549337+0000) 2022-01-31T21:08:09.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:09 smithi167 conmon[54076]: debug 2022-01-31T21:08:09.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:09.475556+0000) 2022-01-31T21:08:10.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:09 smithi171 conmon[41853]: debug 2022-01-31T21:08:09.707+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:09.708999+0000) 2022-01-31T21:08:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:10 smithi171 conmon[46715]: debug 2022-01-31T21:08:10.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:10.003694+0000) 2022-01-31T21:08:10.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:10 smithi167 conmon[60316]: debug 2022-01-31T21:08:10.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:10.227655+0000) 2022-01-31T21:08:10.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:10 smithi171 conmon[51620]: debug 2022-01-31T21:08:10.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:10.582880+0000) 2022-01-31T21:08:10.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:10 smithi167 conmon[49112]: debug 2022-01-31T21:08:10.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:10.549490+0000) 2022-01-31T21:08:10.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:10 smithi167 conmon[54076]: debug 2022-01-31T21:08:10.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:10.475772+0000) 2022-01-31T21:08:11.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:10 smithi171 conmon[41853]: debug 2022-01-31T21:08:10.708+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:10.709221+0000) 2022-01-31T21:08:11.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[46715]: debug 2022-01-31T21:08:11.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.003899+0000) 2022-01-31T21:08:11.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:11 smithi167 conmon[60316]: debug 2022-01-31T21:08:11.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.227804+0000) 2022-01-31T21:08:11.691 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:11 smithi167 conmon[60316]: debug 2022-01-31T21:08:11.440+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.440984+0000) 2022-01-31T21:08:11.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:11 smithi167 conmon[49112]: debug 2022-01-31T21:08:11.433+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.433747+0000) 2022-01-31T21:08:11.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:11 smithi167 conmon[49112]: debug 2022-01-31T21:08:11.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.549718+0000) 2022-01-31T21:08:11.695 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:11 smithi167 conmon[54076]: debug 2022-01-31T21:08:11.432+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.432855+0000) 2022-01-31T21:08:11.695 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:11 smithi167 conmon[54076]: debug 2022-01-31T21:08:11.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.475969+0000) 2022-01-31T21:08:11.707 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[35325]: debug 2022-01-31T21:08:11.440+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105830 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:11.708 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[41853]: debug 2022-01-31T21:08:11.429+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.431335+0000) 2022-01-31T21:08:11.708 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[46715]: debug 2022-01-31T21:08:11.430+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.431937+0000) 2022-01-31T21:08:11.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[51620]: debug 2022-01-31T21:08:11.429+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.431042+0000) 2022-01-31T21:08:11.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[51620]: debug 2022-01-31T21:08:11.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.582994+0000) 2022-01-31T21:08:12.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:11 smithi171 conmon[41853]: debug 2022-01-31T21:08:11.708+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:11.709397+0000) 2022-01-31T21:08:12.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:12 smithi167 conmon[60316]: debug 2022-01-31T21:08:12.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:12.228010+0000) 2022-01-31T21:08:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:12 smithi171 conmon[46715]: debug 2022-01-31T21:08:12.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:12.004090+0000) 2022-01-31T21:08:12.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:12 smithi171 conmon[51620]: debug 2022-01-31T21:08:12.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:12.583079+0000) 2022-01-31T21:08:12.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:12 smithi167 conmon[49112]: debug 2022-01-31T21:08:12.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:12.549885+0000) 2022-01-31T21:08:12.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:12 smithi167 conmon[54076]: debug 2022-01-31T21:08:12.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:12.476126+0000) 2022-01-31T21:08:13.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:12 smithi171 conmon[41853]: debug 2022-01-31T21:08:12.708+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:12.709569+0000) 2022-01-31T21:08:13.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:13 smithi171 conmon[46715]: debug 2022-01-31T21:08:13.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:13.004268+0000) 2022-01-31T21:08:13.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:13 smithi167 conmon[60316]: debug 2022-01-31T21:08:13.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:13.228152+0000) 2022-01-31T21:08:13.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:13 smithi171 conmon[51620]: debug 2022-01-31T21:08:13.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:13.583211+0000) 2022-01-31T21:08:13.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:13 smithi167 conmon[49112]: debug 2022-01-31T21:08:13.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:13.550060+0000) 2022-01-31T21:08:13.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:13 smithi167 conmon[54076]: debug 2022-01-31T21:08:13.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:13.476332+0000) 2022-01-31T21:08:14.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:13 smithi171 conmon[41853]: debug 2022-01-31T21:08:13.708+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:13.709724+0000) 2022-01-31T21:08:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:14 smithi171 conmon[46715]: debug 2022-01-31T21:08:14.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:14.004457+0000) 2022-01-31T21:08:14.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:14 smithi167 conmon[60316]: debug 2022-01-31T21:08:14.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:14.228343+0000) 2022-01-31T21:08:14.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:14 smithi171 conmon[51620]: debug 2022-01-31T21:08:14.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:14.583379+0000) 2022-01-31T21:08:14.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:14 smithi167 conmon[49112]: debug 2022-01-31T21:08:14.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:14.550232+0000) 2022-01-31T21:08:14.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:14 smithi167 conmon[54076]: debug 2022-01-31T21:08:14.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:14.476484+0000) 2022-01-31T21:08:15.003 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:14 smithi171 conmon[41853]: debug 2022-01-31T21:08:14.709+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:14.709929+0000) 2022-01-31T21:08:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:15 smithi171 conmon[46715]: debug 2022-01-31T21:08:15.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:15.004660+0000) 2022-01-31T21:08:15.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:15 smithi167 conmon[60316]: debug 2022-01-31T21:08:15.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:15.228539+0000) 2022-01-31T21:08:15.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:15 smithi171 conmon[51620]: debug 2022-01-31T21:08:15.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:15.583611+0000) 2022-01-31T21:08:15.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:15 smithi167 conmon[49112]: debug 2022-01-31T21:08:15.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:15.550390+0000) 2022-01-31T21:08:15.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:15 smithi167 conmon[54076]: debug 2022-01-31T21:08:15.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:15.476672+0000) 2022-01-31T21:08:16.003 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:15 smithi171 conmon[41853]: debug 2022-01-31T21:08:15.709+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:15.710144+0000) 2022-01-31T21:08:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[46715]: debug 2022-01-31T21:08:16.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.004872+0000) 2022-01-31T21:08:16.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:16 smithi167 conmon[60316]: debug 2022-01-31T21:08:16.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.228727+0000) 2022-01-31T21:08:16.709 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[35325]: debug 2022-01-31T21:08:16.453+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 105939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:16.710 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[41853]: debug 2022-01-31T21:08:16.443+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.445127+0000) 2022-01-31T21:08:16.710 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[46715]: debug 2022-01-31T21:08:16.442+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.444642+0000) 2022-01-31T21:08:16.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[51620]: debug 2022-01-31T21:08:16.443+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.444999+0000) 2022-01-31T21:08:16.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[51620]: debug 2022-01-31T21:08:16.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.583814+0000) 2022-01-31T21:08:16.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:16 smithi167 conmon[60316]: debug 2022-01-31T21:08:16.445+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.446118+0000) 2022-01-31T21:08:16.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:16 smithi167 conmon[49112]: debug 2022-01-31T21:08:16.446+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.446546+0000) 2022-01-31T21:08:16.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:16 smithi167 conmon[49112]: debug 2022-01-31T21:08:16.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.550577+0000) 2022-01-31T21:08:16.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:16 smithi167 conmon[54076]: debug 2022-01-31T21:08:16.444+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.444765+0000) 2022-01-31T21:08:16.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:16 smithi167 conmon[54076]: debug 2022-01-31T21:08:16.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.476820+0000) 2022-01-31T21:08:17.003 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:16 smithi171 conmon[41853]: debug 2022-01-31T21:08:16.709+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:16.710349+0000) 2022-01-31T21:08:17.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:17 smithi171 conmon[46715]: debug 2022-01-31T21:08:17.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:17.005082+0000) 2022-01-31T21:08:17.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:17 smithi167 conmon[60316]: debug 2022-01-31T21:08:17.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:17.228936+0000) 2022-01-31T21:08:17.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:17 smithi171 conmon[51620]: debug 2022-01-31T21:08:17.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:17.583959+0000) 2022-01-31T21:08:17.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:17 smithi167 conmon[49112]: debug 2022-01-31T21:08:17.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:17.550727+0000) 2022-01-31T21:08:17.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:17 smithi167 conmon[54076]: debug 2022-01-31T21:08:17.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:17.476921+0000) 2022-01-31T21:08:18.003 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:17 smithi171 conmon[41853]: debug 2022-01-31T21:08:17.709+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:17.710498+0000) 2022-01-31T21:08:18.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:18 smithi167 conmon[60316]: debug 2022-01-31T21:08:18.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:18.229093+0000) 2022-01-31T21:08:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:18 smithi171 conmon[46715]: debug 2022-01-31T21:08:18.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:18.005254+0000) 2022-01-31T21:08:18.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:18 smithi171 conmon[51620]: debug 2022-01-31T21:08:18.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:18.584096+0000) 2022-01-31T21:08:18.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:18 smithi167 conmon[49112]: debug 2022-01-31T21:08:18.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:18.550914+0000) 2022-01-31T21:08:18.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:18 smithi167 conmon[54076]: debug 2022-01-31T21:08:18.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:18.477103+0000) 2022-01-31T21:08:19.003 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:18 smithi171 conmon[41853]: debug 2022-01-31T21:08:18.709+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:18.710669+0000) 2022-01-31T21:08:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:19 smithi171 conmon[46715]: debug 2022-01-31T21:08:19.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:19.005454+0000) 2022-01-31T21:08:19.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:19 smithi167 conmon[60316]: debug 2022-01-31T21:08:19.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:19.229274+0000) 2022-01-31T21:08:19.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:19 smithi171 conmon[51620]: debug 2022-01-31T21:08:19.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:19.584255+0000) 2022-01-31T21:08:19.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:19 smithi167 conmon[49112]: debug 2022-01-31T21:08:19.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:19.551125+0000) 2022-01-31T21:08:19.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:19 smithi167 conmon[54076]: debug 2022-01-31T21:08:19.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:19.477289+0000) 2022-01-31T21:08:20.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:19 smithi171 conmon[41853]: debug 2022-01-31T21:08:19.710+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:19.710829+0000) 2022-01-31T21:08:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:20 smithi171 conmon[46715]: debug 2022-01-31T21:08:20.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:20.005588+0000) 2022-01-31T21:08:20.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:20 smithi167 conmon[60316]: debug 2022-01-31T21:08:20.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:20.229473+0000) 2022-01-31T21:08:20.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:20 smithi171 conmon[51620]: debug 2022-01-31T21:08:20.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:20.584383+0000) 2022-01-31T21:08:20.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:20 smithi167 conmon[49112]: debug 2022-01-31T21:08:20.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:20.551315+0000) 2022-01-31T21:08:20.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:20 smithi167 conmon[54076]: debug 2022-01-31T21:08:20.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:20.477443+0000) 2022-01-31T21:08:21.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:20 smithi171 conmon[41853]: debug 2022-01-31T21:08:20.710+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:20.711010+0000) 2022-01-31T21:08:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[46715]: debug 2022-01-31T21:08:21.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.005828+0000) 2022-01-31T21:08:21.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:21 smithi167 conmon[60316]: debug 2022-01-31T21:08:21.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.229741+0000) 2022-01-31T21:08:21.709 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[35325]: debug 2022-01-31T21:08:21.468+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106050 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:21.710 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[41853]: debug 2022-01-31T21:08:21.458+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.459197+0000) 2022-01-31T21:08:21.710 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[46715]: debug 2022-01-31T21:08:21.456+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.457234+0000) 2022-01-31T21:08:21.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[51620]: debug 2022-01-31T21:08:21.458+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.459714+0000) 2022-01-31T21:08:21.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[51620]: debug 2022-01-31T21:08:21.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.584565+0000) 2022-01-31T21:08:21.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:21 smithi167 conmon[60316]: debug 2022-01-31T21:08:21.458+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.459281+0000) 2022-01-31T21:08:21.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:21 smithi167 conmon[49112]: debug 2022-01-31T21:08:21.459+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.459890+0000) 2022-01-31T21:08:21.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:21 smithi167 conmon[49112]: debug 2022-01-31T21:08:21.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.551529+0000) 2022-01-31T21:08:21.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:21 smithi167 conmon[54076]: debug 2022-01-31T21:08:21.458+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.458563+0000) 2022-01-31T21:08:21.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:21 smithi167 conmon[54076]: debug 2022-01-31T21:08:21.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.477533+0000) 2022-01-31T21:08:22.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:21 smithi171 conmon[41853]: debug 2022-01-31T21:08:21.710+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:21.711201+0000) 2022-01-31T21:08:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:22 smithi171 conmon[46715]: debug 2022-01-31T21:08:22.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:22.006000+0000) 2022-01-31T21:08:22.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:22 smithi167 conmon[60316]: debug 2022-01-31T21:08:22.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:22.229894+0000) 2022-01-31T21:08:22.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:22 smithi171 conmon[51620]: debug 2022-01-31T21:08:22.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:22.584696+0000) 2022-01-31T21:08:22.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:22 smithi167 conmon[49112]: debug 2022-01-31T21:08:22.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:22.551659+0000) 2022-01-31T21:08:22.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:22 smithi167 conmon[54076]: debug 2022-01-31T21:08:22.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:22.477657+0000) 2022-01-31T21:08:23.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:22 smithi171 conmon[41853]: debug 2022-01-31T21:08:22.710+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:22.711363+0000) 2022-01-31T21:08:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:23 smithi171 conmon[46715]: debug 2022-01-31T21:08:23.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:23.006151+0000) 2022-01-31T21:08:23.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:23 smithi167 conmon[60316]: debug 2022-01-31T21:08:23.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:23.230072+0000) 2022-01-31T21:08:23.628 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:23 smithi171 conmon[51620]: debug 2022-01-31T21:08:23.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:23.584854+0000) 2022-01-31T21:08:23.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:23 smithi167 conmon[49112]: debug 2022-01-31T21:08:23.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:23.551861+0000) 2022-01-31T21:08:23.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:23 smithi167 conmon[54076]: debug 2022-01-31T21:08:23.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:23.477824+0000) 2022-01-31T21:08:23.918 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:23 smithi171 conmon[35325]: debug 2022-01-31T21:08:23.629+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:08:23.918 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:23 smithi171 conmon[41853]: debug 2022-01-31T21:08:23.710+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:23.711545+0000) 2022-01-31T21:08:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:24 smithi171 conmon[46715]: debug 2022-01-31T21:08:24.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:24.006303+0000) 2022-01-31T21:08:24.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:24 smithi167 conmon[60316]: debug 2022-01-31T21:08:24.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:24.230259+0000) 2022-01-31T21:08:24.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:24 smithi171 conmon[51620]: debug 2022-01-31T21:08:24.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:24.585068+0000) 2022-01-31T21:08:24.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:24 smithi167 conmon[49112]: debug 2022-01-31T21:08:24.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:24.552076+0000) 2022-01-31T21:08:24.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:24 smithi167 conmon[54076]: debug 2022-01-31T21:08:24.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:24.478010+0000) 2022-01-31T21:08:25.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:24 smithi171 conmon[41853]: debug 2022-01-31T21:08:24.711+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:24.711709+0000) 2022-01-31T21:08:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:25 smithi171 conmon[46715]: debug 2022-01-31T21:08:25.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:25.006450+0000) 2022-01-31T21:08:25.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:25 smithi167 conmon[60316]: debug 2022-01-31T21:08:25.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:25.230467+0000) 2022-01-31T21:08:25.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:25 smithi171 conmon[51620]: debug 2022-01-31T21:08:25.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:25.585249+0000) 2022-01-31T21:08:25.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:25 smithi167 conmon[49112]: debug 2022-01-31T21:08:25.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:25.552290+0000) 2022-01-31T21:08:25.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:25 smithi167 conmon[54076]: debug 2022-01-31T21:08:25.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:25.478195+0000) 2022-01-31T21:08:26.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:25 smithi171 conmon[41853]: debug 2022-01-31T21:08:25.711+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:25.711870+0000) 2022-01-31T21:08:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[46715]: debug 2022-01-31T21:08:26.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.006584+0000) 2022-01-31T21:08:26.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:26 smithi167 conmon[60316]: debug 2022-01-31T21:08:26.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.230664+0000) 2022-01-31T21:08:26.710 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[41853]: debug 2022-01-31T21:08:26.471+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.472514+0000) 2022-01-31T21:08:26.711 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[46715]: debug 2022-01-31T21:08:26.470+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.471436+0000) 2022-01-31T21:08:26.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[51620]: debug 2022-01-31T21:08:26.471+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.472214+0000) 2022-01-31T21:08:26.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[51620]: debug 2022-01-31T21:08:26.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.585456+0000) 2022-01-31T21:08:26.712 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[35325]: debug 2022-01-31T21:08:26.481+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106159 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:26.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:26 smithi167 conmon[49112]: debug 2022-01-31T21:08:26.472+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.473027+0000) 2022-01-31T21:08:26.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:26 smithi167 conmon[49112]: debug 2022-01-31T21:08:26.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.552453+0000) 2022-01-31T21:08:26.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:26 smithi167 conmon[54076]: debug 2022-01-31T21:08:26.471+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.471978+0000) 2022-01-31T21:08:26.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:26 smithi167 conmon[54076]: debug 2022-01-31T21:08:26.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.478330+0000) 2022-01-31T21:08:26.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:26 smithi167 conmon[60316]: debug 2022-01-31T21:08:26.472+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.473149+0000) 2022-01-31T21:08:27.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:26 smithi171 conmon[41853]: debug 2022-01-31T21:08:26.711+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:26.712028+0000) 2022-01-31T21:08:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:27 smithi171 conmon[46715]: debug 2022-01-31T21:08:27.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:27.006744+0000) 2022-01-31T21:08:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:27 smithi167 conmon[60316]: debug 2022-01-31T21:08:27.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:27.230859+0000) 2022-01-31T21:08:27.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:27 smithi171 conmon[51620]: debug 2022-01-31T21:08:27.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:27.585597+0000) 2022-01-31T21:08:27.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:27 smithi167 conmon[49112]: debug 2022-01-31T21:08:27.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:27.552590+0000) 2022-01-31T21:08:27.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:27 smithi167 conmon[54076]: debug 2022-01-31T21:08:27.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:27.478465+0000) 2022-01-31T21:08:28.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:27 smithi171 conmon[41853]: debug 2022-01-31T21:08:27.711+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:27.712189+0000) 2022-01-31T21:08:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:28 smithi171 conmon[46715]: debug 2022-01-31T21:08:28.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:28.006892+0000) 2022-01-31T21:08:28.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:28 smithi167 conmon[60316]: debug 2022-01-31T21:08:28.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:28.231012+0000) 2022-01-31T21:08:28.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:28 smithi171 conmon[51620]: debug 2022-01-31T21:08:28.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:28.585794+0000) 2022-01-31T21:08:28.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:28 smithi167 conmon[49112]: debug 2022-01-31T21:08:28.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:28.552744+0000) 2022-01-31T21:08:28.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:28 smithi167 conmon[54076]: debug 2022-01-31T21:08:28.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:28.478682+0000) 2022-01-31T21:08:29.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:28 smithi171 conmon[41853]: debug 2022-01-31T21:08:28.711+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:28.712374+0000) 2022-01-31T21:08:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:29 smithi171 conmon[46715]: debug 2022-01-31T21:08:29.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:29.007090+0000) 2022-01-31T21:08:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:29 smithi167 conmon[60316]: debug 2022-01-31T21:08:29.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:29.231174+0000) 2022-01-31T21:08:29.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:29 smithi171 conmon[51620]: debug 2022-01-31T21:08:29.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:29.585919+0000) 2022-01-31T21:08:29.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:29 smithi167 conmon[49112]: debug 2022-01-31T21:08:29.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:29.552963+0000) 2022-01-31T21:08:29.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:29 smithi167 conmon[54076]: debug 2022-01-31T21:08:29.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:29.478872+0000) 2022-01-31T21:08:30.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:29 smithi171 conmon[41853]: debug 2022-01-31T21:08:29.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:29.712558+0000) 2022-01-31T21:08:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:30 smithi171 conmon[46715]: debug 2022-01-31T21:08:30.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:30.007244+0000) 2022-01-31T21:08:30.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:30 smithi167 conmon[60316]: debug 2022-01-31T21:08:30.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:30.231362+0000) 2022-01-31T21:08:30.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:30 smithi171 conmon[51620]: debug 2022-01-31T21:08:30.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:30.586081+0000) 2022-01-31T21:08:30.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:30 smithi167 conmon[49112]: debug 2022-01-31T21:08:30.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:30.553175+0000) 2022-01-31T21:08:30.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:30 smithi167 conmon[54076]: debug 2022-01-31T21:08:30.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:30.479051+0000) 2022-01-31T21:08:31.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:30 smithi171 conmon[41853]: debug 2022-01-31T21:08:30.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:30.712740+0000) 2022-01-31T21:08:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[46715]: debug 2022-01-31T21:08:31.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.007405+0000) 2022-01-31T21:08:31.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:31 smithi167 conmon[60316]: debug 2022-01-31T21:08:31.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.231536+0000) 2022-01-31T21:08:31.711 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[35325]: debug 2022-01-31T21:08:31.495+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106270 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:31.712 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[41853]: debug 2022-01-31T21:08:31.484+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.485431+0000) 2022-01-31T21:08:31.712 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[46715]: debug 2022-01-31T21:08:31.485+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.485782+0000) 2022-01-31T21:08:31.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[51620]: debug 2022-01-31T21:08:31.485+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.486143+0000) 2022-01-31T21:08:31.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[51620]: debug 2022-01-31T21:08:31.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.586267+0000) 2022-01-31T21:08:31.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:31 smithi167 conmon[60316]: debug 2022-01-31T21:08:31.483+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.485236+0000) 2022-01-31T21:08:31.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:31 smithi167 conmon[49112]: debug 2022-01-31T21:08:31.485+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.486587+0000) 2022-01-31T21:08:31.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:31 smithi167 conmon[49112]: debug 2022-01-31T21:08:31.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.553320+0000) 2022-01-31T21:08:31.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:31 smithi167 conmon[54076]: debug 2022-01-31T21:08:31.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.479239+0000) 2022-01-31T21:08:31.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:31 smithi167 conmon[54076]: debug 2022-01-31T21:08:31.483+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.485001+0000) 2022-01-31T21:08:32.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:31 smithi171 conmon[41853]: debug 2022-01-31T21:08:31.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:31.712902+0000) 2022-01-31T21:08:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:32 smithi171 conmon[46715]: debug 2022-01-31T21:08:32.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:32.007597+0000) 2022-01-31T21:08:32.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:32 smithi167 conmon[60316]: debug 2022-01-31T21:08:32.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:32.231662+0000) 2022-01-31T21:08:32.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:32 smithi171 conmon[51620]: debug 2022-01-31T21:08:32.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:32.586433+0000) 2022-01-31T21:08:32.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:32 smithi167 conmon[49112]: debug 2022-01-31T21:08:32.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:32.553476+0000) 2022-01-31T21:08:32.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:32 smithi167 conmon[54076]: debug 2022-01-31T21:08:32.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:32.479422+0000) 2022-01-31T21:08:33.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:32 smithi171 conmon[41853]: debug 2022-01-31T21:08:32.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:32.713079+0000) 2022-01-31T21:08:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:33 smithi171 conmon[46715]: debug 2022-01-31T21:08:33.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:33.007721+0000) 2022-01-31T21:08:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:33 smithi167 conmon[60316]: debug 2022-01-31T21:08:33.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:33.231807+0000) 2022-01-31T21:08:33.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:33 smithi171 conmon[51620]: debug 2022-01-31T21:08:33.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:33.586636+0000) 2022-01-31T21:08:33.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:33 smithi167 conmon[49112]: debug 2022-01-31T21:08:33.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:33.553698+0000) 2022-01-31T21:08:33.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:33 smithi167 conmon[54076]: debug 2022-01-31T21:08:33.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:33.479567+0000) 2022-01-31T21:08:34.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:33 smithi171 conmon[41853]: debug 2022-01-31T21:08:33.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:33.713262+0000) 2022-01-31T21:08:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:34 smithi171 conmon[46715]: debug 2022-01-31T21:08:34.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:34.007912+0000) 2022-01-31T21:08:34.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:34 smithi167 conmon[60316]: debug 2022-01-31T21:08:34.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:34.231997+0000) 2022-01-31T21:08:34.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:34 smithi171 conmon[51620]: debug 2022-01-31T21:08:34.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:34.586793+0000) 2022-01-31T21:08:34.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:34 smithi167 conmon[49112]: debug 2022-01-31T21:08:34.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:34.553858+0000) 2022-01-31T21:08:34.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:34 smithi167 conmon[54076]: debug 2022-01-31T21:08:34.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:34.479723+0000) 2022-01-31T21:08:35.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:34 smithi171 conmon[41853]: debug 2022-01-31T21:08:34.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:34.713416+0000) 2022-01-31T21:08:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:35 smithi171 conmon[46715]: debug 2022-01-31T21:08:35.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:35.008096+0000) 2022-01-31T21:08:35.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:35 smithi167 conmon[60316]: debug 2022-01-31T21:08:35.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:35.232188+0000) 2022-01-31T21:08:35.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:35 smithi171 conmon[51620]: debug 2022-01-31T21:08:35.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:35.586972+0000) 2022-01-31T21:08:35.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:35 smithi167 conmon[49112]: debug 2022-01-31T21:08:35.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:35.554056+0000) 2022-01-31T21:08:35.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:35 smithi167 conmon[54076]: debug 2022-01-31T21:08:35.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:35.479898+0000) 2022-01-31T21:08:36.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:35 smithi171 conmon[41853]: debug 2022-01-31T21:08:35.712+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:35.713562+0000) 2022-01-31T21:08:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[46715]: debug 2022-01-31T21:08:36.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.008278+0000) 2022-01-31T21:08:36.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[60316]: debug 2022-01-31T21:08:36.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.232339+0000) 2022-01-31T21:08:36.712 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[35325]: debug 2022-01-31T21:08:36.508+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106379 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:36.712 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[41853]: debug 2022-01-31T21:08:36.498+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.499011+0000) 2022-01-31T21:08:36.713 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[46715]: debug 2022-01-31T21:08:36.498+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.499127+0000) 2022-01-31T21:08:36.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[51620]: debug 2022-01-31T21:08:36.499+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.500104+0000) 2022-01-31T21:08:36.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[51620]: debug 2022-01-31T21:08:36.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.587191+0000) 2022-01-31T21:08:36.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[49112]: debug 2022-01-31T21:08:36.498+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.499641+0000) 2022-01-31T21:08:36.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[49112]: debug 2022-01-31T21:08:36.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.554255+0000) 2022-01-31T21:08:36.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[54076]: debug 2022-01-31T21:08:36.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.480089+0000) 2022-01-31T21:08:36.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[54076]: debug 2022-01-31T21:08:36.498+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.500031+0000) 2022-01-31T21:08:36.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[60316]: debug 2022-01-31T21:08:36.497+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.499487+0000) 2022-01-31T21:08:36.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:36 smithi167 conmon[60316]: 2022-01-31T21:08:37.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:36 smithi171 conmon[41853]: debug 2022-01-31T21:08:36.713+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:36.713772+0000) 2022-01-31T21:08:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:37 smithi171 conmon[46715]: debug 2022-01-31T21:08:37.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:37.008449+0000) 2022-01-31T21:08:37.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:37 smithi167 conmon[60316]: debug 2022-01-31T21:08:37.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:37.232543+0000) 2022-01-31T21:08:37.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:37 smithi171 conmon[51620]: debug 2022-01-31T21:08:37.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:37.587377+0000) 2022-01-31T21:08:37.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:37 smithi167 conmon[49112]: debug 2022-01-31T21:08:37.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:37.554393+0000) 2022-01-31T21:08:37.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:37 smithi167 conmon[54076]: debug 2022-01-31T21:08:37.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:37.480222+0000) 2022-01-31T21:08:38.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:37 smithi171 conmon[41853]: debug 2022-01-31T21:08:37.713+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:37.713921+0000) 2022-01-31T21:08:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:38 smithi171 conmon[46715]: debug 2022-01-31T21:08:38.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:38.008587+0000) 2022-01-31T21:08:38.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:38 smithi167 conmon[60316]: debug 2022-01-31T21:08:38.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:38.232665+0000) 2022-01-31T21:08:38.629 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:38 smithi171 conmon[51620]: debug 2022-01-31T21:08:38.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:38.587567+0000) 2022-01-31T21:08:38.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:38 smithi167 conmon[49112]: debug 2022-01-31T21:08:38.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:38.554550+0000) 2022-01-31T21:08:38.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:38 smithi167 conmon[54076]: debug 2022-01-31T21:08:38.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:38.480375+0000) 2022-01-31T21:08:38.921 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:38 smithi171 conmon[35325]: debug 2022-01-31T21:08:38.630+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:08:38.921 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:38 smithi171 conmon[41853]: debug 2022-01-31T21:08:38.713+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:38.714080+0000) 2022-01-31T21:08:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:39 smithi171 conmon[46715]: debug 2022-01-31T21:08:39.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:39.008830+0000) 2022-01-31T21:08:39.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:39 smithi167 conmon[60316]: debug 2022-01-31T21:08:39.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:39.232830+0000) 2022-01-31T21:08:39.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:39 smithi171 conmon[51620]: debug 2022-01-31T21:08:39.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:39.587748+0000) 2022-01-31T21:08:39.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:39 smithi167 conmon[49112]: debug 2022-01-31T21:08:39.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:39.554728+0000) 2022-01-31T21:08:39.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:39 smithi167 conmon[54076]: debug 2022-01-31T21:08:39.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:39.480581+0000) 2022-01-31T21:08:40.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:39 smithi171 conmon[41853]: debug 2022-01-31T21:08:39.713+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:39.714287+0000) 2022-01-31T21:08:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:40 smithi171 conmon[46715]: debug 2022-01-31T21:08:40.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:40.008987+0000) 2022-01-31T21:08:40.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:40 smithi167 conmon[60316]: debug 2022-01-31T21:08:40.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:40.233049+0000) 2022-01-31T21:08:40.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:40 smithi171 conmon[51620]: debug 2022-01-31T21:08:40.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:40.587951+0000) 2022-01-31T21:08:40.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:40 smithi167 conmon[49112]: debug 2022-01-31T21:08:40.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:40.554914+0000) 2022-01-31T21:08:40.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:40 smithi167 conmon[54076]: debug 2022-01-31T21:08:40.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:40.480785+0000) 2022-01-31T21:08:41.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:40 smithi171 conmon[41853]: debug 2022-01-31T21:08:40.714+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:40.714441+0000) 2022-01-31T21:08:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[46715]: debug 2022-01-31T21:08:41.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.009138+0000) 2022-01-31T21:08:41.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:41 smithi167 conmon[60316]: debug 2022-01-31T21:08:41.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.233240+0000) 2022-01-31T21:08:41.713 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[35325]: debug 2022-01-31T21:08:41.537+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106490 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:41.713 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[41853]: debug 2022-01-31T21:08:41.511+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.512427+0000) 2022-01-31T21:08:41.714 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[46715]: debug 2022-01-31T21:08:41.511+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.512217+0000) 2022-01-31T21:08:41.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[51620]: debug 2022-01-31T21:08:41.511+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.512566+0000) 2022-01-31T21:08:41.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[51620]: debug 2022-01-31T21:08:41.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.588112+0000) 2022-01-31T21:08:41.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:41 smithi167 conmon[54076]: debug 2022-01-31T21:08:41.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.480922+0000) 2022-01-31T21:08:41.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:41 smithi167 conmon[54076]: debug 2022-01-31T21:08:41.510+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.511709+0000) 2022-01-31T21:08:41.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:41 smithi167 conmon[49112]: debug 2022-01-31T21:08:41.517+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.519199+0000) 2022-01-31T21:08:41.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:41 smithi167 conmon[49112]: debug 2022-01-31T21:08:41.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.555154+0000) 2022-01-31T21:08:41.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:41 smithi167 conmon[60316]: debug 2022-01-31T21:08:41.517+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.519366+0000) 2022-01-31T21:08:42.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:41 smithi171 conmon[41853]: debug 2022-01-31T21:08:41.714+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:41.714620+0000) 2022-01-31T21:08:42.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:42 smithi171 conmon[46715]: debug 2022-01-31T21:08:42.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:42.009323+0000) 2022-01-31T21:08:42.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:42 smithi167 conmon[60316]: debug 2022-01-31T21:08:42.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:42.233429+0000) 2022-01-31T21:08:42.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:42 smithi171 conmon[51620]: debug 2022-01-31T21:08:42.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:42.588241+0000) 2022-01-31T21:08:42.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:42 smithi167 conmon[49112]: debug 2022-01-31T21:08:42.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:42.555249+0000) 2022-01-31T21:08:42.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:42 smithi167 conmon[54076]: debug 2022-01-31T21:08:42.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:42.481118+0000) 2022-01-31T21:08:43.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:42 smithi171 conmon[41853]: debug 2022-01-31T21:08:42.714+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:42.714821+0000) 2022-01-31T21:08:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:43 smithi171 conmon[46715]: debug 2022-01-31T21:08:43.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:43.009495+0000) 2022-01-31T21:08:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:43 smithi167 conmon[60316]: debug 2022-01-31T21:08:43.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:43.233578+0000) 2022-01-31T21:08:43.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:43 smithi171 conmon[51620]: debug 2022-01-31T21:08:43.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:43.588449+0000) 2022-01-31T21:08:43.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:43 smithi167 conmon[49112]: debug 2022-01-31T21:08:43.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:43.555431+0000) 2022-01-31T21:08:43.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:43 smithi167 conmon[54076]: debug 2022-01-31T21:08:43.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:43.481261+0000) 2022-01-31T21:08:44.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:43 smithi171 conmon[41853]: debug 2022-01-31T21:08:43.714+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:43.714954+0000) 2022-01-31T21:08:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:44 smithi171 conmon[46715]: debug 2022-01-31T21:08:44.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:44.009682+0000) 2022-01-31T21:08:44.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:44 smithi167 conmon[60316]: debug 2022-01-31T21:08:44.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:44.233750+0000) 2022-01-31T21:08:44.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:44 smithi171 conmon[51620]: debug 2022-01-31T21:08:44.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:44.588639+0000) 2022-01-31T21:08:44.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:44 smithi167 conmon[54076]: debug 2022-01-31T21:08:44.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:44.481470+0000) 2022-01-31T21:08:44.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:44 smithi167 conmon[49112]: debug 2022-01-31T21:08:44.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:44.555685+0000) 2022-01-31T21:08:45.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:44 smithi171 conmon[41853]: debug 2022-01-31T21:08:44.714+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:44.715154+0000) 2022-01-31T21:08:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:45 smithi171 conmon[46715]: debug 2022-01-31T21:08:45.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:45.009827+0000) 2022-01-31T21:08:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:45 smithi167 conmon[60316]: debug 2022-01-31T21:08:45.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:45.233890+0000) 2022-01-31T21:08:45.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:45 smithi171 conmon[51620]: debug 2022-01-31T21:08:45.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:45.588807+0000) 2022-01-31T21:08:45.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:45 smithi167 conmon[49112]: debug 2022-01-31T21:08:45.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:45.555870+0000) 2022-01-31T21:08:45.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:45 smithi167 conmon[54076]: debug 2022-01-31T21:08:45.481+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:45.481703+0000) 2022-01-31T21:08:46.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:45 smithi171 conmon[41853]: debug 2022-01-31T21:08:45.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:45.715341+0000) 2022-01-31T21:08:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[46715]: debug 2022-01-31T21:08:46.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.010008+0000) 2022-01-31T21:08:46.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:46 smithi167 conmon[60316]: debug 2022-01-31T21:08:46.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.234116+0000) 2022-01-31T21:08:46.714 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[41853]: debug 2022-01-31T21:08:46.539+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.540611+0000) 2022-01-31T21:08:46.714 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[46715]: debug 2022-01-31T21:08:46.540+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.540745+0000) 2022-01-31T21:08:46.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[51620]: debug 2022-01-31T21:08:46.541+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.542056+0000) 2022-01-31T21:08:46.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[51620]: debug 2022-01-31T21:08:46.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.588965+0000) 2022-01-31T21:08:46.715 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[35325]: debug 2022-01-31T21:08:46.550+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106598 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:46.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:46 smithi167 conmon[54076]: debug 2022-01-31T21:08:46.481+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.481891+0000) 2022-01-31T21:08:46.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:46 smithi167 conmon[54076]: debug 2022-01-31T21:08:46.540+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.541728+0000) 2022-01-31T21:08:46.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:46 smithi167 conmon[60316]: debug 2022-01-31T21:08:46.540+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.541618+0000) 2022-01-31T21:08:46.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:46 smithi167 conmon[49112]: debug 2022-01-31T21:08:46.540+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.542476+0000) 2022-01-31T21:08:46.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:46 smithi167 conmon[49112]: debug 2022-01-31T21:08:46.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.555970+0000) 2022-01-31T21:08:47.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:46 smithi171 conmon[41853]: debug 2022-01-31T21:08:46.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:46.715526+0000) 2022-01-31T21:08:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:47 smithi171 conmon[46715]: debug 2022-01-31T21:08:47.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:47.010213+0000) 2022-01-31T21:08:47.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:47 smithi167 conmon[60316]: debug 2022-01-31T21:08:47.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:47.234275+0000) 2022-01-31T21:08:47.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:47 smithi171 conmon[51620]: debug 2022-01-31T21:08:47.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:47.589095+0000) 2022-01-31T21:08:47.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:47 smithi167 conmon[49112]: debug 2022-01-31T21:08:47.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:47.556114+0000) 2022-01-31T21:08:47.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:47 smithi167 conmon[54076]: debug 2022-01-31T21:08:47.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:47.482103+0000) 2022-01-31T21:08:48.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:47 smithi171 conmon[41853]: debug 2022-01-31T21:08:47.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:47.715683+0000) 2022-01-31T21:08:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:48 smithi171 conmon[46715]: debug 2022-01-31T21:08:48.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:48.010394+0000) 2022-01-31T21:08:48.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:48 smithi167 conmon[60316]: debug 2022-01-31T21:08:48.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:48.234428+0000) 2022-01-31T21:08:48.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:48 smithi171 conmon[51620]: debug 2022-01-31T21:08:48.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:48.589311+0000) 2022-01-31T21:08:48.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:48 smithi167 conmon[49112]: debug 2022-01-31T21:08:48.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:48.556270+0000) 2022-01-31T21:08:48.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:48 smithi167 conmon[54076]: debug 2022-01-31T21:08:48.481+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:48.482293+0000) 2022-01-31T21:08:49.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:48 smithi171 conmon[41853]: debug 2022-01-31T21:08:48.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:48.715868+0000) 2022-01-31T21:08:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:49 smithi171 conmon[46715]: debug 2022-01-31T21:08:49.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:49.010581+0000) 2022-01-31T21:08:49.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:49 smithi167 conmon[60316]: debug 2022-01-31T21:08:49.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:49.234609+0000) 2022-01-31T21:08:49.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:49 smithi171 conmon[51620]: debug 2022-01-31T21:08:49.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:49.589536+0000) 2022-01-31T21:08:49.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:49 smithi167 conmon[49112]: debug 2022-01-31T21:08:49.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:49.556429+0000) 2022-01-31T21:08:49.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:49 smithi167 conmon[54076]: debug 2022-01-31T21:08:49.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:49.482484+0000) 2022-01-31T21:08:50.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:49 smithi171 conmon[41853]: debug 2022-01-31T21:08:49.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:49.716051+0000) 2022-01-31T21:08:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:50 smithi171 conmon[46715]: debug 2022-01-31T21:08:50.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:50.010789+0000) 2022-01-31T21:08:50.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:50 smithi167 conmon[60316]: debug 2022-01-31T21:08:50.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:50.234831+0000) 2022-01-31T21:08:50.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:50 smithi171 conmon[51620]: debug 2022-01-31T21:08:50.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:50.589716+0000) 2022-01-31T21:08:50.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:50 smithi167 conmon[49112]: debug 2022-01-31T21:08:50.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:50.556611+0000) 2022-01-31T21:08:50.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:50 smithi167 conmon[54076]: debug 2022-01-31T21:08:50.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:50.482692+0000) 2022-01-31T21:08:51.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:50 smithi171 conmon[41853]: debug 2022-01-31T21:08:50.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:50.716247+0000) 2022-01-31T21:08:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[46715]: debug 2022-01-31T21:08:51.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.010972+0000) 2022-01-31T21:08:51.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:51 smithi167 conmon[60316]: debug 2022-01-31T21:08:51.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.235018+0000) 2022-01-31T21:08:51.714 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[35325]: debug 2022-01-31T21:08:51.564+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106701 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:51.715 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[41853]: debug 2022-01-31T21:08:51.553+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.554619+0000) 2022-01-31T21:08:51.715 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[46715]: debug 2022-01-31T21:08:51.553+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.554325+0000) 2022-01-31T21:08:51.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[51620]: debug 2022-01-31T21:08:51.554+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.555179+0000) 2022-01-31T21:08:51.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[51620]: debug 2022-01-31T21:08:51.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.589858+0000) 2022-01-31T21:08:51.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:51 smithi167 conmon[60316]: debug 2022-01-31T21:08:51.553+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.554825+0000) 2022-01-31T21:08:51.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:51 smithi167 conmon[49112]: debug 2022-01-31T21:08:51.554+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.555517+0000) 2022-01-31T21:08:51.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:51 smithi167 conmon[49112]: debug 2022-01-31T21:08:51.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.556755+0000) 2022-01-31T21:08:51.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:51 smithi167 conmon[54076]: debug 2022-01-31T21:08:51.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.482837+0000) 2022-01-31T21:08:51.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:51 smithi167 conmon[54076]: debug 2022-01-31T21:08:51.553+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.554978+0000) 2022-01-31T21:08:52.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:51 smithi171 conmon[41853]: debug 2022-01-31T21:08:51.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:51.716428+0000) 2022-01-31T21:08:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:52 smithi171 conmon[46715]: debug 2022-01-31T21:08:52.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:52.011151+0000) 2022-01-31T21:08:52.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:52 smithi167 conmon[60316]: debug 2022-01-31T21:08:52.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:52.235175+0000) 2022-01-31T21:08:52.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:52 smithi171 conmon[51620]: debug 2022-01-31T21:08:52.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:52.590013+0000) 2022-01-31T21:08:52.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:52 smithi167 conmon[49112]: debug 2022-01-31T21:08:52.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:52.556921+0000) 2022-01-31T21:08:52.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:52 smithi167 conmon[54076]: debug 2022-01-31T21:08:52.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:52.482953+0000) 2022-01-31T21:08:53.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:52 smithi171 conmon[41853]: debug 2022-01-31T21:08:52.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:52.716599+0000) 2022-01-31T21:08:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:53 smithi171 conmon[46715]: debug 2022-01-31T21:08:53.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:53.011307+0000) 2022-01-31T21:08:53.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:53 smithi167 conmon[60316]: debug 2022-01-31T21:08:53.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:53.235309+0000) 2022-01-31T21:08:53.630 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:53 smithi171 conmon[51620]: debug 2022-01-31T21:08:53.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:53.590192+0000) 2022-01-31T21:08:53.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:53 smithi167 conmon[49112]: debug 2022-01-31T21:08:53.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:53.557114+0000) 2022-01-31T21:08:53.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:53 smithi167 conmon[54076]: debug 2022-01-31T21:08:53.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:53.483113+0000) 2022-01-31T21:08:53.923 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:53 smithi171 conmon[35325]: debug 2022-01-31T21:08:53.630+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:08:53.923 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:53 smithi171 conmon[41853]: debug 2022-01-31T21:08:53.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:53.716774+0000) 2022-01-31T21:08:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:54 smithi171 conmon[46715]: debug 2022-01-31T21:08:54.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:54.011486+0000) 2022-01-31T21:08:54.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:54 smithi167 conmon[60316]: debug 2022-01-31T21:08:54.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:54.235480+0000) 2022-01-31T21:08:54.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:54 smithi171 conmon[51620]: debug 2022-01-31T21:08:54.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:54.590385+0000) 2022-01-31T21:08:54.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:54 smithi167 conmon[49112]: debug 2022-01-31T21:08:54.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:54.557349+0000) 2022-01-31T21:08:54.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:54 smithi167 conmon[54076]: debug 2022-01-31T21:08:54.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:54.483269+0000) 2022-01-31T21:08:55.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:54 smithi171 conmon[41853]: debug 2022-01-31T21:08:54.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:54.716875+0000) 2022-01-31T21:08:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:55 smithi171 conmon[46715]: debug 2022-01-31T21:08:55.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:55.011664+0000) 2022-01-31T21:08:55.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:55 smithi167 conmon[60316]: debug 2022-01-31T21:08:55.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:55.235688+0000) 2022-01-31T21:08:55.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:55 smithi171 conmon[51620]: debug 2022-01-31T21:08:55.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:55.590572+0000) 2022-01-31T21:08:55.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:55 smithi167 conmon[49112]: debug 2022-01-31T21:08:55.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:55.557519+0000) 2022-01-31T21:08:55.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:55 smithi167 conmon[54076]: debug 2022-01-31T21:08:55.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:55.483452+0000) 2022-01-31T21:08:56.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:55 smithi171 conmon[41853]: debug 2022-01-31T21:08:55.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:55.717021+0000) 2022-01-31T21:08:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[46715]: debug 2022-01-31T21:08:56.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.011825+0000) 2022-01-31T21:08:56.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:56 smithi167 conmon[60316]: debug 2022-01-31T21:08:56.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.235868+0000) 2022-01-31T21:08:56.715 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[35325]: debug 2022-01-31T21:08:56.576+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106810 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:08:56.716 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[41853]: debug 2022-01-31T21:08:56.565+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.567377+0000) 2022-01-31T21:08:56.716 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[46715]: debug 2022-01-31T21:08:56.566+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.567808+0000) 2022-01-31T21:08:56.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[51620]: debug 2022-01-31T21:08:56.567+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.569064+0000) 2022-01-31T21:08:56.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[51620]: debug 2022-01-31T21:08:56.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.590772+0000) 2022-01-31T21:08:56.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:56 smithi167 conmon[60316]: debug 2022-01-31T21:08:56.566+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.567320+0000) 2022-01-31T21:08:56.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:56 smithi167 conmon[49112]: debug 2022-01-31T21:08:56.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.557710+0000) 2022-01-31T21:08:56.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:56 smithi167 conmon[49112]: debug 2022-01-31T21:08:56.567+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.568221+0000) 2022-01-31T21:08:56.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:56 smithi167 conmon[54076]: debug 2022-01-31T21:08:56.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.483595+0000) 2022-01-31T21:08:56.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:56 smithi167 conmon[54076]: debug 2022-01-31T21:08:56.567+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.568355+0000) 2022-01-31T21:08:57.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:56 smithi171 conmon[41853]: debug 2022-01-31T21:08:56.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:56.717176+0000) 2022-01-31T21:08:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:57 smithi171 conmon[46715]: debug 2022-01-31T21:08:57.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:57.012001+0000) 2022-01-31T21:08:57.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:57 smithi167 conmon[60316]: debug 2022-01-31T21:08:57.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:57.236053+0000) 2022-01-31T21:08:57.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:57 smithi171 conmon[51620]: debug 2022-01-31T21:08:57.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:57.590888+0000) 2022-01-31T21:08:57.803 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:57 smithi167 conmon[49112]: debug 2022-01-31T21:08:57.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:57.557922+0000) 2022-01-31T21:08:57.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:57 smithi167 conmon[54076]: debug 2022-01-31T21:08:57.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:57.483740+0000) 2022-01-31T21:08:58.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:57 smithi171 conmon[41853]: debug 2022-01-31T21:08:57.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:57.717324+0000) 2022-01-31T21:08:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:58 smithi171 conmon[46715]: debug 2022-01-31T21:08:58.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:58.012123+0000) 2022-01-31T21:08:58.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:58 smithi167 conmon[60316]: debug 2022-01-31T21:08:58.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:58.236221+0000) 2022-01-31T21:08:58.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:58 smithi171 conmon[51620]: debug 2022-01-31T21:08:58.590+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:58.591084+0000) 2022-01-31T21:08:58.803 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:58 smithi167 conmon[49112]: debug 2022-01-31T21:08:58.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:58.558077+0000) 2022-01-31T21:08:58.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:58 smithi167 conmon[54076]: debug 2022-01-31T21:08:58.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:58.483939+0000) 2022-01-31T21:08:59.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:58 smithi171 conmon[41853]: debug 2022-01-31T21:08:58.715+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:58.717481+0000) 2022-01-31T21:08:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:08:59 smithi171 conmon[46715]: debug 2022-01-31T21:08:59.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:59.012340+0000) 2022-01-31T21:08:59.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:08:59 smithi167 conmon[60316]: debug 2022-01-31T21:08:59.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:59.236393+0000) 2022-01-31T21:08:59.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:08:59 smithi171 conmon[51620]: debug 2022-01-31T21:08:59.590+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:59.591265+0000) 2022-01-31T21:08:59.803 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:08:59 smithi167 conmon[49112]: debug 2022-01-31T21:08:59.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:59.558298+0000) 2022-01-31T21:08:59.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:08:59 smithi167 conmon[54076]: debug 2022-01-31T21:08:59.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:59.484136+0000) 2022-01-31T21:09:00.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:08:59 smithi171 conmon[41853]: debug 2022-01-31T21:08:59.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:08:59.717637+0000) 2022-01-31T21:09:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:00 smithi171 conmon[46715]: debug 2022-01-31T21:09:00.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:00.012523+0000) 2022-01-31T21:09:00.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:00 smithi167 conmon[60316]: debug 2022-01-31T21:09:00.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:00.236578+0000) 2022-01-31T21:09:00.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:00 smithi171 conmon[51620]: debug 2022-01-31T21:09:00.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:00.591448+0000) 2022-01-31T21:09:00.803 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:00 smithi167 conmon[49112]: debug 2022-01-31T21:09:00.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:00.558536+0000) 2022-01-31T21:09:00.804 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:00 smithi167 conmon[54076]: debug 2022-01-31T21:09:00.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:00.484323+0000) 2022-01-31T21:09:01.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:00 smithi171 conmon[41853]: debug 2022-01-31T21:09:00.716+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:00.717782+0000) 2022-01-31T21:09:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[46715]: debug 2022-01-31T21:09:01.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.012736+0000) 2022-01-31T21:09:01.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:01 smithi167 conmon[60316]: debug 2022-01-31T21:09:01.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.236733+0000) 2022-01-31T21:09:01.716 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[35325]: debug 2022-01-31T21:09:01.590+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 106920 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:01.717 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[41853]: debug 2022-01-31T21:09:01.578+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.580399+0000) 2022-01-31T21:09:01.717 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[46715]: debug 2022-01-31T21:09:01.579+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.581303+0000) 2022-01-31T21:09:01.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[51620]: debug 2022-01-31T21:09:01.580+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.581877+0000) 2022-01-31T21:09:01.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[51620]: debug 2022-01-31T21:09:01.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.591623+0000) 2022-01-31T21:09:01.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:01 smithi167 conmon[60316]: debug 2022-01-31T21:09:01.580+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.581031+0000) 2022-01-31T21:09:01.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:01 smithi167 conmon[49112]: debug 2022-01-31T21:09:01.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.558716+0000) 2022-01-31T21:09:01.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:01 smithi167 conmon[49112]: debug 2022-01-31T21:09:01.580+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.581348+0000) 2022-01-31T21:09:01.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:01 smithi167 conmon[54076]: debug 2022-01-31T21:09:01.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.484531+0000) 2022-01-31T21:09:01.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:01 smithi167 conmon[54076]: debug 2022-01-31T21:09:01.580+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.581435+0000) 2022-01-31T21:09:02.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:01 smithi171 conmon[41853]: debug 2022-01-31T21:09:01.717+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:01.717964+0000) 2022-01-31T21:09:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:02 smithi171 conmon[46715]: debug 2022-01-31T21:09:02.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:02.012888+0000) 2022-01-31T21:09:02.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:02 smithi167 conmon[60316]: debug 2022-01-31T21:09:02.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:02.236863+0000) 2022-01-31T21:09:02.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:02 smithi171 conmon[51620]: debug 2022-01-31T21:09:02.590+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:02.591791+0000) 2022-01-31T21:09:02.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:02 smithi167 conmon[54076]: debug 2022-01-31T21:09:02.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:02.484672+0000) 2022-01-31T21:09:02.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:02 smithi167 conmon[49112]: debug 2022-01-31T21:09:02.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:02.558893+0000) 2022-01-31T21:09:03.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:02 smithi171 conmon[41853]: debug 2022-01-31T21:09:02.717+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:02.718143+0000) 2022-01-31T21:09:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:03 smithi171 conmon[46715]: debug 2022-01-31T21:09:03.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:03.013040+0000) 2022-01-31T21:09:03.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:03 smithi167 conmon[60316]: debug 2022-01-31T21:09:03.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:03.237050+0000) 2022-01-31T21:09:03.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:03 smithi171 conmon[51620]: debug 2022-01-31T21:09:03.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:03.591979+0000) 2022-01-31T21:09:03.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:03 smithi167 conmon[54076]: debug 2022-01-31T21:09:03.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:03.484824+0000) 2022-01-31T21:09:03.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:03 smithi167 conmon[49112]: debug 2022-01-31T21:09:03.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:03.559040+0000) 2022-01-31T21:09:04.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:03 smithi171 conmon[41853]: debug 2022-01-31T21:09:03.717+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:03.718331+0000) 2022-01-31T21:09:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:04 smithi171 conmon[46715]: debug 2022-01-31T21:09:04.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:04.013218+0000) 2022-01-31T21:09:04.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:04 smithi167 conmon[60316]: debug 2022-01-31T21:09:04.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:04.237207+0000) 2022-01-31T21:09:04.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:04 smithi171 conmon[51620]: debug 2022-01-31T21:09:04.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:04.592169+0000) 2022-01-31T21:09:04.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:04 smithi167 conmon[49112]: debug 2022-01-31T21:09:04.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:04.559278+0000) 2022-01-31T21:09:04.804 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:04 smithi167 conmon[54076]: debug 2022-01-31T21:09:04.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:04.484984+0000) 2022-01-31T21:09:05.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:04 smithi171 conmon[41853]: debug 2022-01-31T21:09:04.717+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:04.718509+0000) 2022-01-31T21:09:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:05 smithi171 conmon[46715]: debug 2022-01-31T21:09:05.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:05.013404+0000) 2022-01-31T21:09:05.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:05 smithi167 conmon[60316]: debug 2022-01-31T21:09:05.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:05.237359+0000) 2022-01-31T21:09:05.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:05 smithi171 conmon[51620]: debug 2022-01-31T21:09:05.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:05.592370+0000) 2022-01-31T21:09:05.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:05 smithi167 conmon[49112]: debug 2022-01-31T21:09:05.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:05.559470+0000) 2022-01-31T21:09:05.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:05 smithi167 conmon[54076]: debug 2022-01-31T21:09:05.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:05.485111+0000) 2022-01-31T21:09:06.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:05 smithi171 conmon[41853]: debug 2022-01-31T21:09:05.717+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:05.718688+0000) 2022-01-31T21:09:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[46715]: debug 2022-01-31T21:09:06.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.013583+0000) 2022-01-31T21:09:06.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:06 smithi167 conmon[60316]: debug 2022-01-31T21:09:06.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.237573+0000) 2022-01-31T21:09:06.717 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[35325]: debug 2022-01-31T21:09:06.604+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107026 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:06.717 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[41853]: debug 2022-01-31T21:09:06.593+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.594025+0000) 2022-01-31T21:09:06.718 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[46715]: debug 2022-01-31T21:09:06.592+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.593512+0000) 2022-01-31T21:09:06.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[51620]: debug 2022-01-31T21:09:06.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.592606+0000) 2022-01-31T21:09:06.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[51620]: debug 2022-01-31T21:09:06.594+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.595241+0000) 2022-01-31T21:09:06.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:06 smithi167 conmon[49112]: debug 2022-01-31T21:09:06.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.559671+0000) 2022-01-31T21:09:06.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:06 smithi167 conmon[49112]: debug 2022-01-31T21:09:06.594+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.595360+0000) 2022-01-31T21:09:06.805 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:06 smithi167 conmon[60316]: debug 2022-01-31T21:09:06.594+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.595033+0000) 2022-01-31T21:09:06.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:06 smithi167 conmon[54076]: debug 2022-01-31T21:09:06.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.485295+0000) 2022-01-31T21:09:06.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:06 smithi167 conmon[54076]: debug 2022-01-31T21:09:06.594+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.595148+0000) 2022-01-31T21:09:07.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:06 smithi171 conmon[41853]: debug 2022-01-31T21:09:06.718+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:06.718914+0000) 2022-01-31T21:09:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:07 smithi171 conmon[46715]: debug 2022-01-31T21:09:07.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:07.013806+0000) 2022-01-31T21:09:07.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:07 smithi167 conmon[60316]: debug 2022-01-31T21:09:07.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:07.237719+0000) 2022-01-31T21:09:07.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:07 smithi171 conmon[51620]: debug 2022-01-31T21:09:07.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:07.592793+0000) 2022-01-31T21:09:07.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:07 smithi167 conmon[49112]: debug 2022-01-31T21:09:07.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:07.559845+0000) 2022-01-31T21:09:07.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:07 smithi167 conmon[54076]: debug 2022-01-31T21:09:07.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:07.485523+0000) 2022-01-31T21:09:08.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:07 smithi171 conmon[41853]: debug 2022-01-31T21:09:07.718+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:07.719067+0000) 2022-01-31T21:09:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:08 smithi171 conmon[46715]: debug 2022-01-31T21:09:08.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:08.013949+0000) 2022-01-31T21:09:08.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:08 smithi167 conmon[60316]: debug 2022-01-31T21:09:08.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:08.237875+0000) 2022-01-31T21:09:08.630 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:08 smithi171 conmon[51620]: debug 2022-01-31T21:09:08.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:08.592991+0000) 2022-01-31T21:09:08.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:08 smithi167 conmon[49112]: debug 2022-01-31T21:09:08.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:08.560002+0000) 2022-01-31T21:09:08.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:08 smithi167 conmon[54076]: debug 2022-01-31T21:09:08.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:08.485718+0000) 2022-01-31T21:09:08.925 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:08 smithi171 conmon[41853]: debug 2022-01-31T21:09:08.718+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:08.719195+0000) 2022-01-31T21:09:08.926 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:08 smithi171 conmon[35325]: debug 2022-01-31T21:09:08.631+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:09:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:09 smithi171 conmon[46715]: debug 2022-01-31T21:09:09.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:09.014148+0000) 2022-01-31T21:09:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:09 smithi167 conmon[60316]: debug 2022-01-31T21:09:09.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:09.238064+0000) 2022-01-31T21:09:09.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:09 smithi171 conmon[51620]: debug 2022-01-31T21:09:09.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:09.593182+0000) 2022-01-31T21:09:09.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:09 smithi167 conmon[49112]: debug 2022-01-31T21:09:09.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:09.560224+0000) 2022-01-31T21:09:09.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:09 smithi167 conmon[54076]: debug 2022-01-31T21:09:09.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:09.485903+0000) 2022-01-31T21:09:10.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:09 smithi171 conmon[41853]: debug 2022-01-31T21:09:09.718+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:09.719355+0000) 2022-01-31T21:09:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:10 smithi171 conmon[46715]: debug 2022-01-31T21:09:10.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:10.014366+0000) 2022-01-31T21:09:10.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:10 smithi167 conmon[60316]: debug 2022-01-31T21:09:10.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:10.238225+0000) 2022-01-31T21:09:10.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:10 smithi171 conmon[51620]: debug 2022-01-31T21:09:10.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:10.593324+0000) 2022-01-31T21:09:10.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:10 smithi167 conmon[49112]: debug 2022-01-31T21:09:10.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:10.560423+0000) 2022-01-31T21:09:10.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:10 smithi167 conmon[54076]: debug 2022-01-31T21:09:10.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:10.486061+0000) 2022-01-31T21:09:11.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:10 smithi171 conmon[41853]: debug 2022-01-31T21:09:10.718+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:10.719553+0000) 2022-01-31T21:09:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[46715]: debug 2022-01-31T21:09:11.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.014529+0000) 2022-01-31T21:09:11.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:11 smithi167 conmon[60316]: debug 2022-01-31T21:09:11.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.238407+0000) 2022-01-31T21:09:11.718 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[35325]: debug 2022-01-31T21:09:11.618+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107136 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:11.718 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[41853]: debug 2022-01-31T21:09:11.606+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.607682+0000) 2022-01-31T21:09:11.719 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[46715]: debug 2022-01-31T21:09:11.607+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.608384+0000) 2022-01-31T21:09:11.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[51620]: debug 2022-01-31T21:09:11.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.593529+0000) 2022-01-31T21:09:11.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[51620]: debug 2022-01-31T21:09:11.607+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.608672+0000) 2022-01-31T21:09:11.805 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:11 smithi167 conmon[60316]: debug 2022-01-31T21:09:11.608+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.608702+0000) 2022-01-31T21:09:11.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:11 smithi167 conmon[49112]: debug 2022-01-31T21:09:11.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.560574+0000) 2022-01-31T21:09:11.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:11 smithi167 conmon[49112]: debug 2022-01-31T21:09:11.607+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.607766+0000) 2022-01-31T21:09:11.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:11 smithi167 conmon[54076]: debug 2022-01-31T21:09:11.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.486248+0000) 2022-01-31T21:09:11.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:11 smithi167 conmon[54076]: debug 2022-01-31T21:09:11.607+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.607873+0000) 2022-01-31T21:09:12.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:11 smithi171 conmon[41853]: debug 2022-01-31T21:09:11.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:11.719681+0000) 2022-01-31T21:09:12.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:12 smithi171 conmon[46715]: debug 2022-01-31T21:09:12.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:12.014680+0000) 2022-01-31T21:09:12.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:12 smithi167 conmon[60316]: debug 2022-01-31T21:09:12.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:12.238558+0000) 2022-01-31T21:09:12.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:12 smithi171 conmon[51620]: debug 2022-01-31T21:09:12.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:12.593731+0000) 2022-01-31T21:09:12.805 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:12 smithi167 conmon[54076]: debug 2022-01-31T21:09:12.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:12.486374+0000) 2022-01-31T21:09:12.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:12 smithi167 conmon[49112]: debug 2022-01-31T21:09:12.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:12.560766+0000) 2022-01-31T21:09:13.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:12 smithi171 conmon[41853]: debug 2022-01-31T21:09:12.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:12.719840+0000) 2022-01-31T21:09:13.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:13 smithi171 conmon[46715]: debug 2022-01-31T21:09:13.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:13.014805+0000) 2022-01-31T21:09:13.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:13 smithi167 conmon[60316]: debug 2022-01-31T21:09:13.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:13.238705+0000) 2022-01-31T21:09:13.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:13 smithi171 conmon[51620]: debug 2022-01-31T21:09:13.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:13.593917+0000) 2022-01-31T21:09:13.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:13 smithi167 conmon[49112]: debug 2022-01-31T21:09:13.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:13.560922+0000) 2022-01-31T21:09:13.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:13 smithi167 conmon[54076]: debug 2022-01-31T21:09:13.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:13.486534+0000) 2022-01-31T21:09:14.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:13 smithi171 conmon[41853]: debug 2022-01-31T21:09:13.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:13.720032+0000) 2022-01-31T21:09:14.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:14 smithi171 conmon[46715]: debug 2022-01-31T21:09:14.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:14.014947+0000) 2022-01-31T21:09:14.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:14 smithi167 conmon[60316]: debug 2022-01-31T21:09:14.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:14.238844+0000) 2022-01-31T21:09:14.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:14 smithi171 conmon[51620]: debug 2022-01-31T21:09:14.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:14.594084+0000) 2022-01-31T21:09:14.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:14 smithi167 conmon[49112]: debug 2022-01-31T21:09:14.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:14.561111+0000) 2022-01-31T21:09:14.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:14 smithi167 conmon[54076]: debug 2022-01-31T21:09:14.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:14.486726+0000) 2022-01-31T21:09:15.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:14 smithi171 conmon[41853]: debug 2022-01-31T21:09:14.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:14.720175+0000) 2022-01-31T21:09:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:15 smithi171 conmon[46715]: debug 2022-01-31T21:09:15.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:15.015098+0000) 2022-01-31T21:09:15.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:15 smithi167 conmon[60316]: debug 2022-01-31T21:09:15.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:15.239034+0000) 2022-01-31T21:09:15.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:15 smithi171 conmon[51620]: debug 2022-01-31T21:09:15.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:15.594248+0000) 2022-01-31T21:09:15.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:15 smithi167 conmon[49112]: debug 2022-01-31T21:09:15.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:15.561304+0000) 2022-01-31T21:09:15.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:15 smithi167 conmon[54076]: debug 2022-01-31T21:09:15.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:15.486911+0000) 2022-01-31T21:09:16.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:15 smithi171 conmon[41853]: debug 2022-01-31T21:09:15.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:15.720304+0000) 2022-01-31T21:09:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[46715]: debug 2022-01-31T21:09:16.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.015276+0000) 2022-01-31T21:09:16.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:16 smithi167 conmon[60316]: debug 2022-01-31T21:09:16.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.239259+0000) 2022-01-31T21:09:16.630 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[41853]: debug 2022-01-31T21:09:16.619+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.620361+0000) 2022-01-31T21:09:16.631 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[46715]: debug 2022-01-31T21:09:16.620+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.621366+0000) 2022-01-31T21:09:16.631 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[51620]: debug 2022-01-31T21:09:16.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.594435+0000) 2022-01-31T21:09:16.632 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[51620]: debug 2022-01-31T21:09:16.621+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.622158+0000) 2022-01-31T21:09:16.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:16 smithi167 conmon[49112]: debug 2022-01-31T21:09:16.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.561464+0000) 2022-01-31T21:09:16.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:16 smithi167 conmon[49112]: debug 2022-01-31T21:09:16.620+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.621524+0000) 2022-01-31T21:09:16.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:16 smithi167 conmon[54076]: debug 2022-01-31T21:09:16.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.487114+0000) 2022-01-31T21:09:16.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:16 smithi167 conmon[54076]: debug 2022-01-31T21:09:16.619+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.621403+0000) 2022-01-31T21:09:16.808 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:16 smithi167 conmon[60316]: debug 2022-01-31T21:09:16.619+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.620910+0000) 2022-01-31T21:09:16.927 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[35325]: debug 2022-01-31T21:09:16.631+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107246 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:16.928 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:16 smithi171 conmon[41853]: debug 2022-01-31T21:09:16.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:16.720463+0000) 2022-01-31T21:09:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:17 smithi171 conmon[46715]: debug 2022-01-31T21:09:17.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:17.015512+0000) 2022-01-31T21:09:17.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:17 smithi167 conmon[60316]: debug 2022-01-31T21:09:17.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:17.239437+0000) 2022-01-31T21:09:17.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:17 smithi171 conmon[51620]: debug 2022-01-31T21:09:17.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:17.594620+0000) 2022-01-31T21:09:17.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:17 smithi167 conmon[49112]: debug 2022-01-31T21:09:17.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:17.561698+0000) 2022-01-31T21:09:17.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:17 smithi167 conmon[54076]: debug 2022-01-31T21:09:17.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:17.487308+0000) 2022-01-31T21:09:18.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:17 smithi171 conmon[41853]: debug 2022-01-31T21:09:17.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:17.720601+0000) 2022-01-31T21:09:18.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:18 smithi171 conmon[46715]: debug 2022-01-31T21:09:18.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:18.015706+0000) 2022-01-31T21:09:18.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:18 smithi167 conmon[60316]: debug 2022-01-31T21:09:18.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:18.239556+0000) 2022-01-31T21:09:18.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:18 smithi171 conmon[51620]: debug 2022-01-31T21:09:18.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:18.594778+0000) 2022-01-31T21:09:18.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:18 smithi167 conmon[49112]: debug 2022-01-31T21:09:18.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:18.561869+0000) 2022-01-31T21:09:18.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:18 smithi167 conmon[54076]: debug 2022-01-31T21:09:18.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:18.487531+0000) 2022-01-31T21:09:19.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:18 smithi171 conmon[41853]: debug 2022-01-31T21:09:18.719+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:18.720710+0000) 2022-01-31T21:09:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:19 smithi171 conmon[46715]: debug 2022-01-31T21:09:19.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:19.015944+0000) 2022-01-31T21:09:19.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:19 smithi167 conmon[60316]: debug 2022-01-31T21:09:19.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:19.239683+0000) 2022-01-31T21:09:19.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:19 smithi171 conmon[51620]: debug 2022-01-31T21:09:19.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:19.594908+0000) 2022-01-31T21:09:19.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:19 smithi167 conmon[49112]: debug 2022-01-31T21:09:19.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:19.562076+0000) 2022-01-31T21:09:19.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:19 smithi167 conmon[54076]: debug 2022-01-31T21:09:19.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:19.487685+0000) 2022-01-31T21:09:20.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:19 smithi171 conmon[41853]: debug 2022-01-31T21:09:19.720+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:19.720864+0000) 2022-01-31T21:09:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:20 smithi171 conmon[46715]: debug 2022-01-31T21:09:20.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:20.016077+0000) 2022-01-31T21:09:20.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:20 smithi167 conmon[60316]: debug 2022-01-31T21:09:20.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:20.239868+0000) 2022-01-31T21:09:20.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:20 smithi171 conmon[51620]: debug 2022-01-31T21:09:20.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:20.595063+0000) 2022-01-31T21:09:20.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:20 smithi167 conmon[49112]: debug 2022-01-31T21:09:20.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:20.562244+0000) 2022-01-31T21:09:20.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:20 smithi167 conmon[54076]: debug 2022-01-31T21:09:20.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:20.487897+0000) 2022-01-31T21:09:21.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:20 smithi171 conmon[41853]: debug 2022-01-31T21:09:20.720+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:20.721015+0000) 2022-01-31T21:09:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[46715]: debug 2022-01-31T21:09:21.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.016268+0000) 2022-01-31T21:09:21.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:21 smithi167 conmon[60316]: debug 2022-01-31T21:09:21.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.240080+0000) 2022-01-31T21:09:21.632 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[51620]: debug 2022-01-31T21:09:21.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.595249+0000) 2022-01-31T21:09:21.807 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:21 smithi167 conmon[60316]: debug 2022-01-31T21:09:21.632+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.634221+0000) 2022-01-31T21:09:21.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:21 smithi167 conmon[49112]: debug 2022-01-31T21:09:21.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.562398+0000) 2022-01-31T21:09:21.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:21 smithi167 conmon[49112]: debug 2022-01-31T21:09:21.634+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.636113+0000) 2022-01-31T21:09:21.808 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:21 smithi167 conmon[54076]: debug 2022-01-31T21:09:21.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.488054+0000) 2022-01-31T21:09:21.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:21 smithi167 conmon[54076]: debug 2022-01-31T21:09:21.633+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.635219+0000) 2022-01-31T21:09:21.928 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[35325]: debug 2022-01-31T21:09:21.644+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107356 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:21.929 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[41853]: debug 2022-01-31T21:09:21.634+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.634939+0000) 2022-01-31T21:09:21.929 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[41853]: debug 2022-01-31T21:09:21.720+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.721195+0000) 2022-01-31T21:09:21.929 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[46715]: debug 2022-01-31T21:09:21.633+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.634005+0000) 2022-01-31T21:09:21.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:21 smithi171 conmon[51620]: debug 2022-01-31T21:09:21.634+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:21.635454+0000) 2022-01-31T21:09:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:22 smithi171 conmon[46715]: debug 2022-01-31T21:09:22.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:22.016450+0000) 2022-01-31T21:09:22.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:22 smithi167 conmon[60316]: debug 2022-01-31T21:09:22.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:22.240297+0000) 2022-01-31T21:09:22.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:22 smithi171 conmon[51620]: debug 2022-01-31T21:09:22.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:22.595437+0000) 2022-01-31T21:09:22.807 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:22 smithi167 conmon[54076]: debug 2022-01-31T21:09:22.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:22.488259+0000) 2022-01-31T21:09:22.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:22 smithi167 conmon[49112]: debug 2022-01-31T21:09:22.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:22.562587+0000) 2022-01-31T21:09:23.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:22 smithi171 conmon[41853]: debug 2022-01-31T21:09:22.720+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:22.721394+0000) 2022-01-31T21:09:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:23 smithi171 conmon[46715]: debug 2022-01-31T21:09:23.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:23.016566+0000) 2022-01-31T21:09:23.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:23 smithi167 conmon[60316]: debug 2022-01-31T21:09:23.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:23.240441+0000) 2022-01-31T21:09:23.631 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:23 smithi171 conmon[51620]: debug 2022-01-31T21:09:23.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:23.595613+0000) 2022-01-31T21:09:23.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:23 smithi167 conmon[49112]: debug 2022-01-31T21:09:23.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:23.562762+0000) 2022-01-31T21:09:23.808 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:23 smithi167 conmon[54076]: debug 2022-01-31T21:09:23.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:23.488440+0000) 2022-01-31T21:09:23.928 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:23 smithi171 conmon[35325]: debug 2022-01-31T21:09:23.632+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:09:23.929 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:23 smithi171 conmon[41853]: debug 2022-01-31T21:09:23.720+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:23.721562+0000) 2022-01-31T21:09:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:24 smithi171 conmon[46715]: debug 2022-01-31T21:09:24.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:24.016723+0000) 2022-01-31T21:09:24.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:24 smithi167 conmon[60316]: debug 2022-01-31T21:09:24.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:24.240640+0000) 2022-01-31T21:09:24.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:24 smithi171 conmon[51620]: debug 2022-01-31T21:09:24.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:24.595789+0000) 2022-01-31T21:09:24.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:24 smithi167 conmon[49112]: debug 2022-01-31T21:09:24.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:24.562973+0000) 2022-01-31T21:09:24.808 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:24 smithi167 conmon[54076]: debug 2022-01-31T21:09:24.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:24.488566+0000) 2022-01-31T21:09:25.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:24 smithi171 conmon[41853]: debug 2022-01-31T21:09:24.721+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:24.721747+0000) 2022-01-31T21:09:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:25 smithi171 conmon[46715]: debug 2022-01-31T21:09:25.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:25.016851+0000) 2022-01-31T21:09:25.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:25 smithi167 conmon[60316]: debug 2022-01-31T21:09:25.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:25.240781+0000) 2022-01-31T21:09:25.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:25 smithi171 conmon[51620]: debug 2022-01-31T21:09:25.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:25.595992+0000) 2022-01-31T21:09:25.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:25 smithi167 conmon[49112]: debug 2022-01-31T21:09:25.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:25.563160+0000) 2022-01-31T21:09:25.808 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:25 smithi167 conmon[54076]: debug 2022-01-31T21:09:25.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:25.488673+0000) 2022-01-31T21:09:26.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:25 smithi171 conmon[41853]: debug 2022-01-31T21:09:25.721+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:25.721946+0000) 2022-01-31T21:09:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[46715]: debug 2022-01-31T21:09:26.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.016969+0000) 2022-01-31T21:09:26.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:26 smithi167 conmon[60316]: debug 2022-01-31T21:09:26.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.240979+0000) 2022-01-31T21:09:26.645 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[51620]: debug 2022-01-31T21:09:26.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.596149+0000) 2022-01-31T21:09:26.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:26 smithi167 conmon[49112]: debug 2022-01-31T21:09:26.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.563346+0000) 2022-01-31T21:09:26.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:26 smithi167 conmon[49112]: debug 2022-01-31T21:09:26.647+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.648534+0000) 2022-01-31T21:09:26.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:26 smithi167 conmon[54076]: debug 2022-01-31T21:09:26.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.488843+0000) 2022-01-31T21:09:26.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:26 smithi167 conmon[54076]: debug 2022-01-31T21:09:26.647+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.648658+0000) 2022-01-31T21:09:26.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:26 smithi167 conmon[60316]: debug 2022-01-31T21:09:26.646+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.648194+0000) 2022-01-31T21:09:26.929 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[35325]: debug 2022-01-31T21:09:26.658+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107466 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:26.929 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[46715]: debug 2022-01-31T21:09:26.646+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.647291+0000) 2022-01-31T21:09:26.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[51620]: debug 2022-01-31T21:09:26.648+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.648946+0000) 2022-01-31T21:09:26.930 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[41853]: debug 2022-01-31T21:09:26.648+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.649139+0000) 2022-01-31T21:09:26.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:26 smithi171 conmon[41853]: debug 2022-01-31T21:09:26.721+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:26.722103+0000) 2022-01-31T21:09:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:27 smithi171 conmon[46715]: debug 2022-01-31T21:09:27.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:27.017160+0000) 2022-01-31T21:09:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:27 smithi167 conmon[60316]: debug 2022-01-31T21:09:27.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:27.241165+0000) 2022-01-31T21:09:27.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:27 smithi167 conmon[49112]: debug 2022-01-31T21:09:27.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:27.563570+0000) 2022-01-31T21:09:27.704 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:27 smithi167 conmon[54076]: debug 2022-01-31T21:09:27.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:27.489010+0000) 2022-01-31T21:09:27.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:27 smithi171 conmon[51620]: debug 2022-01-31T21:09:27.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:27.596305+0000) 2022-01-31T21:09:28.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:27 smithi171 conmon[41853]: debug 2022-01-31T21:09:27.721+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:27.722255+0000) 2022-01-31T21:09:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:28 smithi171 conmon[46715]: debug 2022-01-31T21:09:28.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:28.017329+0000) 2022-01-31T21:09:28.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:28 smithi167 conmon[60316]: debug 2022-01-31T21:09:28.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:28.241322+0000) 2022-01-31T21:09:28.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:28 smithi171 conmon[51620]: debug 2022-01-31T21:09:28.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:28.596493+0000) 2022-01-31T21:09:28.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:28 smithi167 conmon[49112]: debug 2022-01-31T21:09:28.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:28.563720+0000) 2022-01-31T21:09:28.808 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:28 smithi167 conmon[54076]: debug 2022-01-31T21:09:28.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:28.489224+0000) 2022-01-31T21:09:29.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:28 smithi171 conmon[41853]: debug 2022-01-31T21:09:28.721+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:28.722451+0000) 2022-01-31T21:09:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:29 smithi171 conmon[46715]: debug 2022-01-31T21:09:29.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:29.017498+0000) 2022-01-31T21:09:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:29 smithi167 conmon[60316]: debug 2022-01-31T21:09:29.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:29.241527+0000) 2022-01-31T21:09:29.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:29 smithi171 conmon[51620]: debug 2022-01-31T21:09:29.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:29.596642+0000) 2022-01-31T21:09:29.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:29 smithi167 conmon[49112]: debug 2022-01-31T21:09:29.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:29.563899+0000) 2022-01-31T21:09:29.808 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:29 smithi167 conmon[54076]: debug 2022-01-31T21:09:29.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:29.489409+0000) 2022-01-31T21:09:30.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:29 smithi171 conmon[41853]: debug 2022-01-31T21:09:29.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:29.722604+0000) 2022-01-31T21:09:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:30 smithi171 conmon[46715]: debug 2022-01-31T21:09:30.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:30.017677+0000) 2022-01-31T21:09:30.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:30 smithi167 conmon[60316]: debug 2022-01-31T21:09:30.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:30.241740+0000) 2022-01-31T21:09:30.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:30 smithi171 conmon[51620]: debug 2022-01-31T21:09:30.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:30.596826+0000) 2022-01-31T21:09:30.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:30 smithi167 conmon[49112]: debug 2022-01-31T21:09:30.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:30.564104+0000) 2022-01-31T21:09:30.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:30 smithi167 conmon[54076]: debug 2022-01-31T21:09:30.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:30.489569+0000) 2022-01-31T21:09:31.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:30 smithi171 conmon[41853]: debug 2022-01-31T21:09:30.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:30.722770+0000) 2022-01-31T21:09:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[46715]: debug 2022-01-31T21:09:31.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.017900+0000) 2022-01-31T21:09:31.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:31 smithi167 conmon[60316]: debug 2022-01-31T21:09:31.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.241917+0000) 2022-01-31T21:09:31.659 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[51620]: debug 2022-01-31T21:09:31.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.597024+0000) 2022-01-31T21:09:31.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:31 smithi167 conmon[60316]: debug 2022-01-31T21:09:31.661+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.663037+0000) 2022-01-31T21:09:31.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:31 smithi167 conmon[49112]: debug 2022-01-31T21:09:31.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.564255+0000) 2022-01-31T21:09:31.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:31 smithi167 conmon[49112]: debug 2022-01-31T21:09:31.661+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.662843+0000) 2022-01-31T21:09:31.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:31 smithi167 conmon[54076]: debug 2022-01-31T21:09:31.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.489755+0000) 2022-01-31T21:09:31.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:31 smithi167 conmon[54076]: debug 2022-01-31T21:09:31.661+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.662725+0000) 2022-01-31T21:09:31.930 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[35325]: debug 2022-01-31T21:09:31.672+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107576 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:31.930 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[46715]: debug 2022-01-31T21:09:31.661+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.662300+0000) 2022-01-31T21:09:31.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[51620]: debug 2022-01-31T21:09:31.662+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.663530+0000) 2022-01-31T21:09:31.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[41853]: debug 2022-01-31T21:09:31.662+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.663398+0000) 2022-01-31T21:09:31.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:31 smithi171 conmon[41853]: debug 2022-01-31T21:09:31.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:31.722891+0000) 2022-01-31T21:09:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:32 smithi171 conmon[46715]: debug 2022-01-31T21:09:32.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:32.018050+0000) 2022-01-31T21:09:32.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:32 smithi167 conmon[60316]: debug 2022-01-31T21:09:32.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:32.242104+0000) 2022-01-31T21:09:32.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:32 smithi171 conmon[51620]: debug 2022-01-31T21:09:32.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:32.597183+0000) 2022-01-31T21:09:32.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:32 smithi167 conmon[54076]: debug 2022-01-31T21:09:32.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:32.489953+0000) 2022-01-31T21:09:32.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:32 smithi167 conmon[49112]: debug 2022-01-31T21:09:32.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:32.564436+0000) 2022-01-31T21:09:33.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:32 smithi171 conmon[41853]: debug 2022-01-31T21:09:32.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:32.723046+0000) 2022-01-31T21:09:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:33 smithi171 conmon[46715]: debug 2022-01-31T21:09:33.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:33.018208+0000) 2022-01-31T21:09:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:33 smithi167 conmon[60316]: debug 2022-01-31T21:09:33.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:33.242249+0000) 2022-01-31T21:09:33.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:33 smithi171 conmon[51620]: debug 2022-01-31T21:09:33.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:33.597337+0000) 2022-01-31T21:09:33.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:33 smithi167 conmon[49112]: debug 2022-01-31T21:09:33.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:33.564586+0000) 2022-01-31T21:09:33.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:33 smithi167 conmon[54076]: debug 2022-01-31T21:09:33.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:33.490140+0000) 2022-01-31T21:09:34.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:33 smithi171 conmon[41853]: debug 2022-01-31T21:09:33.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:33.723201+0000) 2022-01-31T21:09:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:34 smithi171 conmon[46715]: debug 2022-01-31T21:09:34.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:34.018398+0000) 2022-01-31T21:09:34.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:34 smithi167 conmon[60316]: debug 2022-01-31T21:09:34.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:34.242437+0000) 2022-01-31T21:09:34.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:34 smithi171 conmon[51620]: debug 2022-01-31T21:09:34.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:34.597518+0000) 2022-01-31T21:09:34.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:34 smithi167 conmon[49112]: debug 2022-01-31T21:09:34.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:34.564786+0000) 2022-01-31T21:09:34.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:34 smithi167 conmon[54076]: debug 2022-01-31T21:09:34.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:34.490365+0000) 2022-01-31T21:09:35.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:34 smithi171 conmon[41853]: debug 2022-01-31T21:09:34.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:34.723352+0000) 2022-01-31T21:09:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:35 smithi171 conmon[46715]: debug 2022-01-31T21:09:35.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:35.018584+0000) 2022-01-31T21:09:35.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:35 smithi167 conmon[60316]: debug 2022-01-31T21:09:35.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:35.242652+0000) 2022-01-31T21:09:35.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:35 smithi171 conmon[51620]: debug 2022-01-31T21:09:35.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:35.597694+0000) 2022-01-31T21:09:35.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:35 smithi167 conmon[49112]: debug 2022-01-31T21:09:35.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:35.565010+0000) 2022-01-31T21:09:35.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:35 smithi167 conmon[54076]: debug 2022-01-31T21:09:35.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:35.490593+0000) 2022-01-31T21:09:36.017 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:35 smithi171 conmon[41853]: debug 2022-01-31T21:09:35.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:35.723511+0000) 2022-01-31T21:09:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[46715]: debug 2022-01-31T21:09:36.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.018787+0000) 2022-01-31T21:09:36.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:36 smithi167 conmon[60316]: debug 2022-01-31T21:09:36.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.242805+0000) 2022-01-31T21:09:36.673 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[51620]: debug 2022-01-31T21:09:36.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.597854+0000) 2022-01-31T21:09:36.809 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:36 smithi167 conmon[60316]: debug 2022-01-31T21:09:36.674+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.675760+0000) 2022-01-31T21:09:36.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:36 smithi167 conmon[49112]: debug 2022-01-31T21:09:36.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.565197+0000) 2022-01-31T21:09:36.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:36 smithi167 conmon[49112]: debug 2022-01-31T21:09:36.675+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.676896+0000) 2022-01-31T21:09:36.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:36 smithi167 conmon[54076]: debug 2022-01-31T21:09:36.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.490796+0000) 2022-01-31T21:09:36.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:36 smithi167 conmon[54076]: debug 2022-01-31T21:09:36.674+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.675996+0000) 2022-01-31T21:09:36.930 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[35325]: debug 2022-01-31T21:09:36.685+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107686 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:36.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[41853]: debug 2022-01-31T21:09:36.675+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.676472+0000) 2022-01-31T21:09:36.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[41853]: debug 2022-01-31T21:09:36.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.723698+0000) 2022-01-31T21:09:36.932 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[46715]: debug 2022-01-31T21:09:36.674+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.674815+0000) 2022-01-31T21:09:36.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:36 smithi171 conmon[51620]: debug 2022-01-31T21:09:36.676+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:36.676766+0000) 2022-01-31T21:09:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:37 smithi171 conmon[46715]: debug 2022-01-31T21:09:37.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:37.018995+0000) 2022-01-31T21:09:37.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:37 smithi167 conmon[60316]: debug 2022-01-31T21:09:37.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:37.242962+0000) 2022-01-31T21:09:37.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:37 smithi171 conmon[51620]: debug 2022-01-31T21:09:37.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:37.598048+0000) 2022-01-31T21:09:37.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:37 smithi167 conmon[49112]: debug 2022-01-31T21:09:37.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:37.565355+0000) 2022-01-31T21:09:37.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:37 smithi167 conmon[54076]: debug 2022-01-31T21:09:37.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:37.491005+0000) 2022-01-31T21:09:38.017 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:37 smithi171 conmon[41853]: debug 2022-01-31T21:09:37.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:37.723822+0000) 2022-01-31T21:09:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:38 smithi171 conmon[46715]: debug 2022-01-31T21:09:38.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:38.019164+0000) 2022-01-31T21:09:38.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:38 smithi167 conmon[60316]: debug 2022-01-31T21:09:38.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:38.243115+0000) 2022-01-31T21:09:38.632 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:38 smithi171 conmon[51620]: debug 2022-01-31T21:09:38.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:38.598196+0000) 2022-01-31T21:09:38.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:38 smithi167 conmon[49112]: debug 2022-01-31T21:09:38.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:38.565536+0000) 2022-01-31T21:09:38.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:38 smithi167 conmon[54076]: debug 2022-01-31T21:09:38.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:38.491189+0000) 2022-01-31T21:09:38.931 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:38 smithi171 conmon[35325]: debug 2022-01-31T21:09:38.633+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:09:38.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:38 smithi171 conmon[41853]: debug 2022-01-31T21:09:38.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:38.723978+0000) 2022-01-31T21:09:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:39 smithi171 conmon[46715]: debug 2022-01-31T21:09:39.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:39.019354+0000) 2022-01-31T21:09:39.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:39 smithi167 conmon[60316]: debug 2022-01-31T21:09:39.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:39.243332+0000) 2022-01-31T21:09:39.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:39 smithi171 conmon[51620]: debug 2022-01-31T21:09:39.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:39.598408+0000) 2022-01-31T21:09:39.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:39 smithi167 conmon[49112]: debug 2022-01-31T21:09:39.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:39.565739+0000) 2022-01-31T21:09:39.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:39 smithi167 conmon[54076]: debug 2022-01-31T21:09:39.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:39.491379+0000) 2022-01-31T21:09:40.017 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:39 smithi171 conmon[41853]: debug 2022-01-31T21:09:39.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:39.724159+0000) 2022-01-31T21:09:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:40 smithi171 conmon[46715]: debug 2022-01-31T21:09:40.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:40.019527+0000) 2022-01-31T21:09:40.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:40 smithi167 conmon[60316]: debug 2022-01-31T21:09:40.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:40.243570+0000) 2022-01-31T21:09:40.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:40 smithi171 conmon[51620]: debug 2022-01-31T21:09:40.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:40.598600+0000) 2022-01-31T21:09:40.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:40 smithi167 conmon[49112]: debug 2022-01-31T21:09:40.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:40.565925+0000) 2022-01-31T21:09:40.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:40 smithi167 conmon[54076]: debug 2022-01-31T21:09:40.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:40.491566+0000) 2022-01-31T21:09:41.017 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:40 smithi171 conmon[41853]: debug 2022-01-31T21:09:40.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:40.724312+0000) 2022-01-31T21:09:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[46715]: debug 2022-01-31T21:09:41.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.019682+0000) 2022-01-31T21:09:41.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:41 smithi167 conmon[60316]: debug 2022-01-31T21:09:41.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.243761+0000) 2022-01-31T21:09:41.686 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[51620]: debug 2022-01-31T21:09:41.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.598747+0000) 2022-01-31T21:09:41.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:41 smithi167 conmon[54076]: debug 2022-01-31T21:09:41.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.491726+0000) 2022-01-31T21:09:41.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:41 smithi167 conmon[54076]: debug 2022-01-31T21:09:41.689+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.689544+0000) 2022-01-31T21:09:41.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:41 smithi167 conmon[60316]: debug 2022-01-31T21:09:41.688+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.688585+0000) 2022-01-31T21:09:41.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:41 smithi167 conmon[49112]: debug 2022-01-31T21:09:41.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.566090+0000) 2022-01-31T21:09:41.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:41 smithi167 conmon[49112]: debug 2022-01-31T21:09:41.687+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.688440+0000) 2022-01-31T21:09:42.018 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[35325]: debug 2022-01-31T21:09:41.714+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107796 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:42.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[41853]: debug 2022-01-31T21:09:41.688+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.690478+0000) 2022-01-31T21:09:42.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[41853]: debug 2022-01-31T21:09:41.722+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.724438+0000) 2022-01-31T21:09:42.019 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[46715]: debug 2022-01-31T21:09:41.686+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.688307+0000) 2022-01-31T21:09:42.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:41 smithi171 conmon[51620]: debug 2022-01-31T21:09:41.687+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:41.689670+0000) 2022-01-31T21:09:42.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:42 smithi171 conmon[46715]: debug 2022-01-31T21:09:42.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:42.019884+0000) 2022-01-31T21:09:42.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:42 smithi167 conmon[60316]: debug 2022-01-31T21:09:42.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:42.243977+0000) 2022-01-31T21:09:42.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:42 smithi171 conmon[51620]: debug 2022-01-31T21:09:42.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:42.598909+0000) 2022-01-31T21:09:42.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:42 smithi167 conmon[49112]: debug 2022-01-31T21:09:42.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:42.566270+0000) 2022-01-31T21:09:42.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:42 smithi167 conmon[54076]: debug 2022-01-31T21:09:42.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:42.491855+0000) 2022-01-31T21:09:43.018 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:42 smithi171 conmon[41853]: debug 2022-01-31T21:09:42.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:42.724582+0000) 2022-01-31T21:09:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:43 smithi171 conmon[46715]: debug 2022-01-31T21:09:43.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:43.020032+0000) 2022-01-31T21:09:43.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:43 smithi167 conmon[60316]: debug 2022-01-31T21:09:43.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:43.244076+0000) 2022-01-31T21:09:43.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:43 smithi171 conmon[51620]: debug 2022-01-31T21:09:43.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:43.599074+0000) 2022-01-31T21:09:43.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:43 smithi167 conmon[49112]: debug 2022-01-31T21:09:43.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:43.566428+0000) 2022-01-31T21:09:43.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:43 smithi167 conmon[54076]: debug 2022-01-31T21:09:43.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:43.492003+0000) 2022-01-31T21:09:44.018 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:43 smithi171 conmon[41853]: debug 2022-01-31T21:09:43.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:43.724787+0000) 2022-01-31T21:09:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:44 smithi171 conmon[46715]: debug 2022-01-31T21:09:44.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:44.020189+0000) 2022-01-31T21:09:44.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:44 smithi167 conmon[60316]: debug 2022-01-31T21:09:44.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:44.244240+0000) 2022-01-31T21:09:44.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:44 smithi171 conmon[51620]: debug 2022-01-31T21:09:44.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:44.599261+0000) 2022-01-31T21:09:44.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:44 smithi167 conmon[49112]: debug 2022-01-31T21:09:44.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:44.566632+0000) 2022-01-31T21:09:44.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:44 smithi167 conmon[54076]: debug 2022-01-31T21:09:44.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:44.492200+0000) 2022-01-31T21:09:45.018 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:44 smithi171 conmon[41853]: debug 2022-01-31T21:09:44.724+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:44.724956+0000) 2022-01-31T21:09:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:45 smithi171 conmon[46715]: debug 2022-01-31T21:09:45.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:45.020386+0000) 2022-01-31T21:09:45.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:45 smithi167 conmon[60316]: debug 2022-01-31T21:09:45.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:45.244442+0000) 2022-01-31T21:09:45.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:45 smithi171 conmon[51620]: debug 2022-01-31T21:09:45.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:45.599472+0000) 2022-01-31T21:09:45.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:45 smithi167 conmon[49112]: debug 2022-01-31T21:09:45.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:45.566782+0000) 2022-01-31T21:09:45.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:45 smithi167 conmon[54076]: debug 2022-01-31T21:09:45.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:45.492391+0000) 2022-01-31T21:09:46.018 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:45 smithi171 conmon[41853]: debug 2022-01-31T21:09:45.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:45.725088+0000) 2022-01-31T21:09:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[46715]: debug 2022-01-31T21:09:46.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.020542+0000) 2022-01-31T21:09:46.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:46 smithi167 conmon[60316]: debug 2022-01-31T21:09:46.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.244595+0000) 2022-01-31T21:09:46.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[51620]: debug 2022-01-31T21:09:46.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.599656+0000) 2022-01-31T21:09:46.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[51620]: debug 2022-01-31T21:09:46.716+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.717805+0000) 2022-01-31T21:09:46.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:46 smithi167 conmon[60316]: debug 2022-01-31T21:09:46.717+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.718363+0000) 2022-01-31T21:09:46.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:46 smithi167 conmon[49112]: debug 2022-01-31T21:09:46.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.566943+0000) 2022-01-31T21:09:46.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:46 smithi167 conmon[49112]: debug 2022-01-31T21:09:46.718+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.718664+0000) 2022-01-31T21:09:46.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:46 smithi167 conmon[54076]: debug 2022-01-31T21:09:46.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.492574+0000) 2022-01-31T21:09:46.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:46 smithi167 conmon[54076]: debug 2022-01-31T21:09:46.718+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.719110+0000) 2022-01-31T21:09:47.019 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[35325]: debug 2022-01-31T21:09:46.727+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 107907 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:47.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[41853]: debug 2022-01-31T21:09:46.717+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.719505+0000) 2022-01-31T21:09:47.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[41853]: debug 2022-01-31T21:09:46.723+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.725226+0000) 2022-01-31T21:09:47.020 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:46 smithi171 conmon[46715]: debug 2022-01-31T21:09:46.716+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:46.718096+0000) 2022-01-31T21:09:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:47 smithi171 conmon[46715]: debug 2022-01-31T21:09:47.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:47.020731+0000) 2022-01-31T21:09:47.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:47 smithi167 conmon[60316]: debug 2022-01-31T21:09:47.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:47.244783+0000) 2022-01-31T21:09:47.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:47 smithi171 conmon[51620]: debug 2022-01-31T21:09:47.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:47.599834+0000) 2022-01-31T21:09:47.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:47 smithi167 conmon[49112]: debug 2022-01-31T21:09:47.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:47.567144+0000) 2022-01-31T21:09:47.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:47 smithi167 conmon[54076]: debug 2022-01-31T21:09:47.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:47.492779+0000) 2022-01-31T21:09:48.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:47 smithi171 conmon[41853]: debug 2022-01-31T21:09:47.724+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:47.725369+0000) 2022-01-31T21:09:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:48 smithi171 conmon[46715]: debug 2022-01-31T21:09:48.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:48.020887+0000) 2022-01-31T21:09:48.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:48 smithi167 conmon[60316]: debug 2022-01-31T21:09:48.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:48.244930+0000) 2022-01-31T21:09:48.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:48 smithi171 conmon[51620]: debug 2022-01-31T21:09:48.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:48.600009+0000) 2022-01-31T21:09:48.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:48 smithi167 conmon[49112]: debug 2022-01-31T21:09:48.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:48.567296+0000) 2022-01-31T21:09:48.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:48 smithi167 conmon[54076]: debug 2022-01-31T21:09:48.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:48.492930+0000) 2022-01-31T21:09:49.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:48 smithi171 conmon[41853]: debug 2022-01-31T21:09:48.724+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:48.725564+0000) 2022-01-31T21:09:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:49 smithi171 conmon[46715]: debug 2022-01-31T21:09:49.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:49.021067+0000) 2022-01-31T21:09:49.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:49 smithi167 conmon[60316]: debug 2022-01-31T21:09:49.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:49.245072+0000) 2022-01-31T21:09:49.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:49 smithi171 conmon[51620]: debug 2022-01-31T21:09:49.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:49.600193+0000) 2022-01-31T21:09:49.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:49 smithi167 conmon[49112]: debug 2022-01-31T21:09:49.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:49.567541+0000) 2022-01-31T21:09:49.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:49 smithi167 conmon[54076]: debug 2022-01-31T21:09:49.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:49.493132+0000) 2022-01-31T21:09:50.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:49 smithi171 conmon[41853]: debug 2022-01-31T21:09:49.724+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:49.725768+0000) 2022-01-31T21:09:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:50 smithi171 conmon[46715]: debug 2022-01-31T21:09:50.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:50.021222+0000) 2022-01-31T21:09:50.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:50 smithi167 conmon[60316]: debug 2022-01-31T21:09:50.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:50.245252+0000) 2022-01-31T21:09:50.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:50 smithi171 conmon[51620]: debug 2022-01-31T21:09:50.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:50.600360+0000) 2022-01-31T21:09:50.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:50 smithi167 conmon[54076]: debug 2022-01-31T21:09:50.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:50.493290+0000) 2022-01-31T21:09:50.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:50 smithi167 conmon[49112]: debug 2022-01-31T21:09:50.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:50.567731+0000) 2022-01-31T21:09:51.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:50 smithi171 conmon[41853]: debug 2022-01-31T21:09:50.725+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:50.725945+0000) 2022-01-31T21:09:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[46715]: debug 2022-01-31T21:09:51.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.021432+0000) 2022-01-31T21:09:51.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:51 smithi167 conmon[60316]: debug 2022-01-31T21:09:51.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.245440+0000) 2022-01-31T21:09:51.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[51620]: debug 2022-01-31T21:09:51.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.600508+0000) 2022-01-31T21:09:51.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:51 smithi167 conmon[54076]: debug 2022-01-31T21:09:51.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.493468+0000) 2022-01-31T21:09:51.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:51 smithi167 conmon[54076]: debug 2022-01-31T21:09:51.732+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.732594+0000) 2022-01-31T21:09:51.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:51 smithi167 conmon[60316]: debug 2022-01-31T21:09:51.731+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.732362+0000) 2022-01-31T21:09:51.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:51 smithi167 conmon[49112]: debug 2022-01-31T21:09:51.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.567871+0000) 2022-01-31T21:09:51.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:51 smithi167 conmon[49112]: debug 2022-01-31T21:09:51.732+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.733116+0000) 2022-01-31T21:09:52.019 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[35325]: debug 2022-01-31T21:09:51.741+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108017 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:52.020 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[46715]: debug 2022-01-31T21:09:51.730+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.731669+0000) 2022-01-31T21:09:52.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[51620]: debug 2022-01-31T21:09:51.731+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.732492+0000) 2022-01-31T21:09:52.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[41853]: debug 2022-01-31T21:09:51.725+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.726126+0000) 2022-01-31T21:09:52.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:51 smithi171 conmon[41853]: debug 2022-01-31T21:09:51.731+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:51.732701+0000) 2022-01-31T21:09:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:52 smithi171 conmon[46715]: debug 2022-01-31T21:09:52.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:52.021609+0000) 2022-01-31T21:09:52.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:52 smithi167 conmon[60316]: debug 2022-01-31T21:09:52.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:52.245666+0000) 2022-01-31T21:09:52.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:52 smithi171 conmon[51620]: debug 2022-01-31T21:09:52.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:52.600681+0000) 2022-01-31T21:09:52.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:52 smithi167 conmon[49112]: debug 2022-01-31T21:09:52.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:52.568074+0000) 2022-01-31T21:09:52.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:52 smithi167 conmon[54076]: debug 2022-01-31T21:09:52.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:52.493617+0000) 2022-01-31T21:09:53.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:52 smithi171 conmon[41853]: debug 2022-01-31T21:09:52.725+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:52.726324+0000) 2022-01-31T21:09:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:53 smithi171 conmon[46715]: debug 2022-01-31T21:09:53.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:53.021746+0000) 2022-01-31T21:09:53.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:53 smithi167 conmon[60316]: debug 2022-01-31T21:09:53.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:53.245792+0000) 2022-01-31T21:09:53.724 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:53 smithi171 conmon[35325]: debug 2022-01-31T21:09:53.633+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:09:53.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:53 smithi171 conmon[51620]: debug 2022-01-31T21:09:53.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:53.600814+0000) 2022-01-31T21:09:53.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:53 smithi167 conmon[49112]: debug 2022-01-31T21:09:53.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:53.568225+0000) 2022-01-31T21:09:53.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:53 smithi167 conmon[54076]: debug 2022-01-31T21:09:53.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:53.493738+0000) 2022-01-31T21:09:54.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:53 smithi171 conmon[41853]: debug 2022-01-31T21:09:53.725+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:53.726499+0000) 2022-01-31T21:09:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:54 smithi171 conmon[46715]: debug 2022-01-31T21:09:54.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:54.021982+0000) 2022-01-31T21:09:54.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:54 smithi167 conmon[60316]: debug 2022-01-31T21:09:54.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:54.245957+0000) 2022-01-31T21:09:54.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:54 smithi171 conmon[51620]: debug 2022-01-31T21:09:54.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:54.600972+0000) 2022-01-31T21:09:54.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:54 smithi167 conmon[49112]: debug 2022-01-31T21:09:54.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:54.568427+0000) 2022-01-31T21:09:54.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:54 smithi167 conmon[54076]: debug 2022-01-31T21:09:54.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:54.493888+0000) 2022-01-31T21:09:55.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:54 smithi171 conmon[41853]: debug 2022-01-31T21:09:54.725+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:54.726679+0000) 2022-01-31T21:09:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:55 smithi171 conmon[46715]: debug 2022-01-31T21:09:55.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:55.022169+0000) 2022-01-31T21:09:55.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:55 smithi167 conmon[60316]: debug 2022-01-31T21:09:55.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:55.246147+0000) 2022-01-31T21:09:55.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:55 smithi171 conmon[51620]: debug 2022-01-31T21:09:55.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:55.601155+0000) 2022-01-31T21:09:55.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:55 smithi167 conmon[49112]: debug 2022-01-31T21:09:55.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:55.568610+0000) 2022-01-31T21:09:55.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:55 smithi167 conmon[54076]: debug 2022-01-31T21:09:55.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:55.494072+0000) 2022-01-31T21:09:56.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:55 smithi171 conmon[41853]: debug 2022-01-31T21:09:55.726+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:55.726863+0000) 2022-01-31T21:09:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[46715]: debug 2022-01-31T21:09:56.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.022329+0000) 2022-01-31T21:09:56.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:56 smithi167 conmon[60316]: debug 2022-01-31T21:09:56.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.246327+0000) 2022-01-31T21:09:56.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[51620]: debug 2022-01-31T21:09:56.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.601331+0000) 2022-01-31T21:09:56.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:56 smithi167 conmon[49112]: debug 2022-01-31T21:09:56.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.568796+0000) 2022-01-31T21:09:56.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:56 smithi167 conmon[54076]: debug 2022-01-31T21:09:56.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.494290+0000) 2022-01-31T21:09:56.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:56 smithi167 conmon[49112]: debug 2022-01-31T21:09:56.745+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.745491+0000) 2022-01-31T21:09:56.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:56 smithi167 conmon[54076]: debug 2022-01-31T21:09:56.744+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.744706+0000) 2022-01-31T21:09:56.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:56 smithi167 conmon[60316]: debug 2022-01-31T21:09:56.745+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.745696+0000) 2022-01-31T21:09:57.020 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[35325]: debug 2022-01-31T21:09:56.755+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108127 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:09:57.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[41853]: debug 2022-01-31T21:09:56.726+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.727043+0000) 2022-01-31T21:09:57.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[41853]: debug 2022-01-31T21:09:56.744+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.744848+0000) 2022-01-31T21:09:57.022 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[46715]: debug 2022-01-31T21:09:56.744+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.745679+0000) 2022-01-31T21:09:57.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:56 smithi171 conmon[51620]: debug 2022-01-31T21:09:56.745+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:56.746026+0000) 2022-01-31T21:09:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:57 smithi171 conmon[46715]: debug 2022-01-31T21:09:57.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:57.022483+0000) 2022-01-31T21:09:57.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:57 smithi167 conmon[60316]: debug 2022-01-31T21:09:57.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:57.246530+0000) 2022-01-31T21:09:57.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:57 smithi171 conmon[51620]: debug 2022-01-31T21:09:57.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:57.601512+0000) 2022-01-31T21:09:57.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:57 smithi167 conmon[49112]: debug 2022-01-31T21:09:57.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:57.568977+0000) 2022-01-31T21:09:57.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:57 smithi167 conmon[54076]: debug 2022-01-31T21:09:57.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:57.494436+0000) 2022-01-31T21:09:58.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:57 smithi171 conmon[41853]: debug 2022-01-31T21:09:57.726+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:57.727202+0000) 2022-01-31T21:09:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:58 smithi171 conmon[46715]: debug 2022-01-31T21:09:58.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:58.022672+0000) 2022-01-31T21:09:58.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:58 smithi167 conmon[60316]: debug 2022-01-31T21:09:58.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:58.246656+0000) 2022-01-31T21:09:58.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:58 smithi171 conmon[51620]: debug 2022-01-31T21:09:58.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:58.601679+0000) 2022-01-31T21:09:58.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:58 smithi167 conmon[49112]: debug 2022-01-31T21:09:58.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:58.569112+0000) 2022-01-31T21:09:58.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:58 smithi167 conmon[54076]: debug 2022-01-31T21:09:58.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:58.494588+0000) 2022-01-31T21:09:59.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:58 smithi171 conmon[41853]: debug 2022-01-31T21:09:58.726+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:58.727352+0000) 2022-01-31T21:09:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:09:59 smithi171 conmon[46715]: debug 2022-01-31T21:09:59.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:59.022867+0000) 2022-01-31T21:09:59.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:09:59 smithi167 conmon[60316]: debug 2022-01-31T21:09:59.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:59.246828+0000) 2022-01-31T21:09:59.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:09:59 smithi171 conmon[51620]: debug 2022-01-31T21:09:59.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:59.601824+0000) 2022-01-31T21:09:59.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:09:59 smithi167 conmon[54076]: debug 2022-01-31T21:09:59.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:59.494688+0000) 2022-01-31T21:09:59.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:09:59 smithi167 conmon[49112]: debug 2022-01-31T21:09:59.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:59.569272+0000) 2022-01-31T21:10:00.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:09:59 smithi171 conmon[41853]: debug 2022-01-31T21:09:59.726+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:09:59.727558+0000) 2022-01-31T21:10:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:00 smithi171 conmon[46715]: debug 2022-01-31T21:10:00.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:00.022999+0000) 2022-01-31T21:10:00.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:00 smithi167 conmon[60316]: debug 2022-01-31T21:10:00.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:00.247010+0000) 2022-01-31T21:10:00.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:00 smithi171 conmon[51620]: debug 2022-01-31T21:10:00.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:00.602024+0000) 2022-01-31T21:10:00.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:00 smithi167 conmon[49112]: debug 2022-01-31T21:10:00.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:00.569475+0000) 2022-01-31T21:10:00.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:00 smithi167 conmon[54076]: debug 2022-01-31T21:10:00.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:00.494860+0000) 2022-01-31T21:10:01.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:00 smithi171 conmon[41853]: debug 2022-01-31T21:10:00.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:00.727735+0000) 2022-01-31T21:10:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[46715]: debug 2022-01-31T21:10:01.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.023155+0000) 2022-01-31T21:10:01.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:01 smithi167 conmon[60316]: debug 2022-01-31T21:10:01.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.247138+0000) 2022-01-31T21:10:01.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[51620]: debug 2022-01-31T21:10:01.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.602179+0000) 2022-01-31T21:10:01.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:01 smithi167 conmon[49112]: debug 2022-01-31T21:10:01.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.569669+0000) 2022-01-31T21:10:01.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:01 smithi167 conmon[54076]: debug 2022-01-31T21:10:01.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.495065+0000) 2022-01-31T21:10:02.022 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[35325]: debug 2022-01-31T21:10:01.768+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108237 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:02.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[41853]: debug 2022-01-31T21:10:01.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.727899+0000) 2022-01-31T21:10:02.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[41853]: debug 2022-01-31T21:10:01.756+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.757606+0000) 2022-01-31T21:10:02.024 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[46715]: debug 2022-01-31T21:10:01.756+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.757558+0000) 2022-01-31T21:10:02.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:01 smithi171 conmon[51620]: debug 2022-01-31T21:10:01.758+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.759419+0000) 2022-01-31T21:10:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:01 smithi167 conmon[49112]: debug 2022-01-31T21:10:01.758+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.758933+0000) 2022-01-31T21:10:02.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:01 smithi167 conmon[54076]: debug 2022-01-31T21:10:01.758+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.758716+0000) 2022-01-31T21:10:02.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:01 smithi167 conmon[60316]: debug 2022-01-31T21:10:01.757+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:01.758234+0000) 2022-01-31T21:10:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:02 smithi171 conmon[46715]: debug 2022-01-31T21:10:02.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:02.023371+0000) 2022-01-31T21:10:02.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:02 smithi167 conmon[60316]: debug 2022-01-31T21:10:02.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:02.247314+0000) 2022-01-31T21:10:02.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:02 smithi171 conmon[51620]: debug 2022-01-31T21:10:02.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:02.602364+0000) 2022-01-31T21:10:02.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:02 smithi167 conmon[49112]: debug 2022-01-31T21:10:02.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:02.569825+0000) 2022-01-31T21:10:02.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:02 smithi167 conmon[54076]: debug 2022-01-31T21:10:02.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:02.495261+0000) 2022-01-31T21:10:03.021 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:02 smithi171 conmon[41853]: debug 2022-01-31T21:10:02.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:02.728056+0000) 2022-01-31T21:10:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:03 smithi171 conmon[46715]: debug 2022-01-31T21:10:03.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:03.023530+0000) 2022-01-31T21:10:03.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:03 smithi167 conmon[60316]: debug 2022-01-31T21:10:03.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:03.247464+0000) 2022-01-31T21:10:03.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:03 smithi171 conmon[51620]: debug 2022-01-31T21:10:03.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:03.602516+0000) 2022-01-31T21:10:03.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:03 smithi167 conmon[54076]: debug 2022-01-31T21:10:03.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:03.495435+0000) 2022-01-31T21:10:03.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:03 smithi167 conmon[49112]: debug 2022-01-31T21:10:03.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:03.570005+0000) 2022-01-31T21:10:04.022 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:03 smithi171 conmon[41853]: debug 2022-01-31T21:10:03.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:03.728192+0000) 2022-01-31T21:10:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:04 smithi171 conmon[46715]: debug 2022-01-31T21:10:04.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:04.023689+0000) 2022-01-31T21:10:04.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:04 smithi167 conmon[60316]: debug 2022-01-31T21:10:04.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:04.247621+0000) 2022-01-31T21:10:04.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:04 smithi171 conmon[51620]: debug 2022-01-31T21:10:04.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:04.602725+0000) 2022-01-31T21:10:04.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:04 smithi167 conmon[49112]: debug 2022-01-31T21:10:04.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:04.570192+0000) 2022-01-31T21:10:04.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:04 smithi167 conmon[54076]: debug 2022-01-31T21:10:04.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:04.495601+0000) 2022-01-31T21:10:05.022 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:04 smithi171 conmon[41853]: debug 2022-01-31T21:10:04.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:04.728359+0000) 2022-01-31T21:10:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:05 smithi171 conmon[46715]: debug 2022-01-31T21:10:05.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:05.023848+0000) 2022-01-31T21:10:05.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:05 smithi167 conmon[60316]: debug 2022-01-31T21:10:05.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:05.247812+0000) 2022-01-31T21:10:05.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:05 smithi171 conmon[51620]: debug 2022-01-31T21:10:05.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:05.602946+0000) 2022-01-31T21:10:05.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:05 smithi167 conmon[49112]: debug 2022-01-31T21:10:05.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:05.570402+0000) 2022-01-31T21:10:05.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:05 smithi167 conmon[54076]: debug 2022-01-31T21:10:05.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:05.495782+0000) 2022-01-31T21:10:06.022 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:05 smithi171 conmon[41853]: debug 2022-01-31T21:10:05.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:05.728568+0000) 2022-01-31T21:10:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[46715]: debug 2022-01-31T21:10:06.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.024029+0000) 2022-01-31T21:10:06.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:06 smithi167 conmon[60316]: debug 2022-01-31T21:10:06.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.248018+0000) 2022-01-31T21:10:06.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[51620]: debug 2022-01-31T21:10:06.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.603162+0000) 2022-01-31T21:10:06.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:06 smithi167 conmon[49112]: debug 2022-01-31T21:10:06.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.570565+0000) 2022-01-31T21:10:06.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:06 smithi167 conmon[54076]: debug 2022-01-31T21:10:06.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.495979+0000) 2022-01-31T21:10:07.022 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[46715]: debug 2022-01-31T21:10:06.771+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.771925+0000) 2022-01-31T21:10:07.023 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[35325]: debug 2022-01-31T21:10:06.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108347 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:07.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[41853]: debug 2022-01-31T21:10:06.727+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.728741+0000) 2022-01-31T21:10:07.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[41853]: debug 2022-01-31T21:10:06.770+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.771665+0000) 2022-01-31T21:10:07.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:06 smithi171 conmon[51620]: debug 2022-01-31T21:10:06.772+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.772916+0000) 2022-01-31T21:10:07.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:06 smithi167 conmon[49112]: debug 2022-01-31T21:10:06.770+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.772193+0000) 2022-01-31T21:10:07.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:06 smithi167 conmon[54076]: debug 2022-01-31T21:10:06.770+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.772030+0000) 2022-01-31T21:10:07.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:06 smithi167 conmon[60316]: debug 2022-01-31T21:10:06.771+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:06.772668+0000) 2022-01-31T21:10:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:07 smithi171 conmon[46715]: debug 2022-01-31T21:10:07.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:07.024205+0000) 2022-01-31T21:10:07.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:07 smithi167 conmon[60316]: debug 2022-01-31T21:10:07.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:07.248202+0000) 2022-01-31T21:10:07.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:07 smithi171 conmon[51620]: debug 2022-01-31T21:10:07.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:07.603357+0000) 2022-01-31T21:10:07.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:07 smithi167 conmon[49112]: debug 2022-01-31T21:10:07.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:07.570729+0000) 2022-01-31T21:10:07.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:07 smithi167 conmon[54076]: debug 2022-01-31T21:10:07.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:07.496182+0000) 2022-01-31T21:10:08.022 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:07 smithi171 conmon[41853]: debug 2022-01-31T21:10:07.728+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:07.728930+0000) 2022-01-31T21:10:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:08 smithi171 conmon[46715]: debug 2022-01-31T21:10:08.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:08.024376+0000) 2022-01-31T21:10:08.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:08 smithi167 conmon[60316]: debug 2022-01-31T21:10:08.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:08.248334+0000) 2022-01-31T21:10:08.727 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:08 smithi171 conmon[35325]: debug 2022-01-31T21:10:08.634+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:10:08.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:08 smithi171 conmon[51620]: debug 2022-01-31T21:10:08.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:08.603532+0000) 2022-01-31T21:10:08.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:08 smithi167 conmon[49112]: debug 2022-01-31T21:10:08.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:08.570902+0000) 2022-01-31T21:10:08.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:08 smithi167 conmon[54076]: debug 2022-01-31T21:10:08.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:08.496382+0000) 2022-01-31T21:10:09.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:08 smithi171 conmon[41853]: debug 2022-01-31T21:10:08.728+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:08.729076+0000) 2022-01-31T21:10:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:09 smithi171 conmon[46715]: debug 2022-01-31T21:10:09.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:09.024577+0000) 2022-01-31T21:10:09.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:09 smithi167 conmon[60316]: debug 2022-01-31T21:10:09.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:09.248497+0000) 2022-01-31T21:10:09.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:09 smithi171 conmon[51620]: debug 2022-01-31T21:10:09.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:09.603688+0000) 2022-01-31T21:10:09.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:09 smithi167 conmon[49112]: debug 2022-01-31T21:10:09.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:09.571018+0000) 2022-01-31T21:10:09.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:09 smithi167 conmon[54076]: debug 2022-01-31T21:10:09.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:09.496594+0000) 2022-01-31T21:10:10.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:09 smithi171 conmon[41853]: debug 2022-01-31T21:10:09.728+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:09.729253+0000) 2022-01-31T21:10:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:10 smithi171 conmon[46715]: debug 2022-01-31T21:10:10.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:10.024713+0000) 2022-01-31T21:10:10.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:10 smithi167 conmon[60316]: debug 2022-01-31T21:10:10.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:10.248633+0000) 2022-01-31T21:10:10.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:10 smithi171 conmon[51620]: debug 2022-01-31T21:10:10.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:10.603842+0000) 2022-01-31T21:10:10.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:10 smithi167 conmon[49112]: debug 2022-01-31T21:10:10.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:10.571222+0000) 2022-01-31T21:10:10.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:10 smithi167 conmon[54076]: debug 2022-01-31T21:10:10.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:10.496723+0000) 2022-01-31T21:10:11.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:10 smithi171 conmon[41853]: debug 2022-01-31T21:10:10.729+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:10.729474+0000) 2022-01-31T21:10:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[46715]: debug 2022-01-31T21:10:11.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.024884+0000) 2022-01-31T21:10:11.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:11 smithi167 conmon[60316]: debug 2022-01-31T21:10:11.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.248761+0000) 2022-01-31T21:10:11.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[51620]: debug 2022-01-31T21:10:11.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.604030+0000) 2022-01-31T21:10:11.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:11 smithi167 conmon[49112]: debug 2022-01-31T21:10:11.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.571400+0000) 2022-01-31T21:10:11.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:11 smithi167 conmon[54076]: debug 2022-01-31T21:10:11.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.496886+0000) 2022-01-31T21:10:12.023 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[35325]: debug 2022-01-31T21:10:11.795+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108457 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:12.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[41853]: debug 2022-01-31T21:10:11.728+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.729623+0000) 2022-01-31T21:10:12.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[41853]: debug 2022-01-31T21:10:11.784+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.785203+0000) 2022-01-31T21:10:12.025 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[46715]: debug 2022-01-31T21:10:11.784+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.785671+0000) 2022-01-31T21:10:12.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:11 smithi171 conmon[51620]: debug 2022-01-31T21:10:11.785+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.786198+0000) 2022-01-31T21:10:12.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:11 smithi167 conmon[60316]: debug 2022-01-31T21:10:11.783+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.785334+0000) 2022-01-31T21:10:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:11 smithi167 conmon[49112]: debug 2022-01-31T21:10:11.784+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.786094+0000) 2022-01-31T21:10:12.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:11 smithi167 conmon[54076]: debug 2022-01-31T21:10:11.784+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:11.785992+0000) 2022-01-31T21:10:12.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:12 smithi171 conmon[46715]: debug 2022-01-31T21:10:12.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:12.025070+0000) 2022-01-31T21:10:12.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:12 smithi167 conmon[60316]: debug 2022-01-31T21:10:12.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:12.248928+0000) 2022-01-31T21:10:12.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:12 smithi171 conmon[51620]: debug 2022-01-31T21:10:12.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:12.604229+0000) 2022-01-31T21:10:12.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:12 smithi167 conmon[49112]: debug 2022-01-31T21:10:12.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:12.571586+0000) 2022-01-31T21:10:12.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:12 smithi167 conmon[54076]: debug 2022-01-31T21:10:12.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:12.497266+0000) 2022-01-31T21:10:13.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:12 smithi171 conmon[41853]: debug 2022-01-31T21:10:12.729+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:12.729824+0000) 2022-01-31T21:10:13.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:13 smithi171 conmon[46715]: debug 2022-01-31T21:10:13.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:13.025218+0000) 2022-01-31T21:10:13.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:13 smithi167 conmon[60316]: debug 2022-01-31T21:10:13.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:13.249073+0000) 2022-01-31T21:10:13.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:13 smithi171 conmon[51620]: debug 2022-01-31T21:10:13.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:13.604380+0000) 2022-01-31T21:10:13.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:13 smithi167 conmon[49112]: debug 2022-01-31T21:10:13.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:13.571771+0000) 2022-01-31T21:10:13.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:13 smithi167 conmon[54076]: debug 2022-01-31T21:10:13.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:13.497420+0000) 2022-01-31T21:10:14.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:13 smithi171 conmon[41853]: debug 2022-01-31T21:10:13.729+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:13.729977+0000) 2022-01-31T21:10:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:14 smithi171 conmon[46715]: debug 2022-01-31T21:10:14.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:14.025347+0000) 2022-01-31T21:10:14.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:14 smithi167 conmon[60316]: debug 2022-01-31T21:10:14.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:14.249264+0000) 2022-01-31T21:10:14.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:14 smithi171 conmon[51620]: debug 2022-01-31T21:10:14.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:14.604561+0000) 2022-01-31T21:10:14.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:14 smithi167 conmon[49112]: debug 2022-01-31T21:10:14.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:14.571970+0000) 2022-01-31T21:10:14.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:14 smithi167 conmon[54076]: debug 2022-01-31T21:10:14.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:14.497619+0000) 2022-01-31T21:10:15.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:14 smithi171 conmon[41853]: debug 2022-01-31T21:10:14.729+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:14.730147+0000) 2022-01-31T21:10:15.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:15 smithi171 conmon[46715]: debug 2022-01-31T21:10:15.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:15.025534+0000) 2022-01-31T21:10:15.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:15 smithi167 conmon[60316]: debug 2022-01-31T21:10:15.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:15.249447+0000) 2022-01-31T21:10:15.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:15 smithi171 conmon[51620]: debug 2022-01-31T21:10:15.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:15.604749+0000) 2022-01-31T21:10:15.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:15 smithi167 conmon[54076]: debug 2022-01-31T21:10:15.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:15.497789+0000) 2022-01-31T21:10:15.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:15 smithi167 conmon[49112]: debug 2022-01-31T21:10:15.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:15.572155+0000) 2022-01-31T21:10:16.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:15 smithi171 conmon[41853]: debug 2022-01-31T21:10:15.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:15.730283+0000) 2022-01-31T21:10:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[46715]: debug 2022-01-31T21:10:16.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.025683+0000) 2022-01-31T21:10:16.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[60316]: debug 2022-01-31T21:10:16.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.249653+0000) 2022-01-31T21:10:16.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[51620]: debug 2022-01-31T21:10:16.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.604978+0000) 2022-01-31T21:10:16.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[49112]: debug 2022-01-31T21:10:16.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.572334+0000) 2022-01-31T21:10:16.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[54076]: debug 2022-01-31T21:10:16.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.497991+0000) 2022-01-31T21:10:17.024 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[35325]: debug 2022-01-31T21:10:16.808+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108567 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:17.024 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[46715]: debug 2022-01-31T21:10:16.797+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.798278+0000) 2022-01-31T21:10:17.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[41853]: debug 2022-01-31T21:10:16.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.730456+0000) 2022-01-31T21:10:17.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[41853]: debug 2022-01-31T21:10:16.798+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.799237+0000) 2022-01-31T21:10:17.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:16 smithi171 conmon[51620]: debug 2022-01-31T21:10:16.798+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.798811+0000) 2022-01-31T21:10:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[49112]: debug 2022-01-31T21:10:16.798+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.799612+0000) 2022-01-31T21:10:17.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[60316]: debug 2022-01-31T21:10:16.796+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.798388+0000) 2022-01-31T21:10:17.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[54076]: debug 2022-01-31T21:10:17.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:16 smithi167 conmon[54076]: 2022-01-31T21:10:16.797+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:16.799256+0000) 2022-01-31T21:10:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:17 smithi171 conmon[46715]: debug 2022-01-31T21:10:17.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:17.025835+0000) 2022-01-31T21:10:17.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:17 smithi167 conmon[60316]: debug 2022-01-31T21:10:17.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:17.249780+0000) 2022-01-31T21:10:17.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:17 smithi171 conmon[51620]: debug 2022-01-31T21:10:17.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:17.605133+0000) 2022-01-31T21:10:17.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:17 smithi167 conmon[49112]: debug 2022-01-31T21:10:17.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:17.572531+0000) 2022-01-31T21:10:17.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:17 smithi167 conmon[54076]: debug 2022-01-31T21:10:17.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:17.498152+0000) 2022-01-31T21:10:18.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:17 smithi171 conmon[41853]: debug 2022-01-31T21:10:17.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:17.730635+0000) 2022-01-31T21:10:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:18 smithi171 conmon[46715]: debug 2022-01-31T21:10:18.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:18.025987+0000) 2022-01-31T21:10:18.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:18 smithi167 conmon[60316]: debug 2022-01-31T21:10:18.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:18.249945+0000) 2022-01-31T21:10:18.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:18 smithi171 conmon[51620]: debug 2022-01-31T21:10:18.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:18.605251+0000) 2022-01-31T21:10:18.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:18 smithi167 conmon[49112]: debug 2022-01-31T21:10:18.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:18.572630+0000) 2022-01-31T21:10:18.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:18 smithi167 conmon[54076]: debug 2022-01-31T21:10:18.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:18.498325+0000) 2022-01-31T21:10:19.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:18 smithi171 conmon[41853]: debug 2022-01-31T21:10:18.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:18.730799+0000) 2022-01-31T21:10:19.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:19 smithi171 conmon[46715]: debug 2022-01-31T21:10:19.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:19.026205+0000) 2022-01-31T21:10:19.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:19 smithi167 conmon[60316]: debug 2022-01-31T21:10:19.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:19.250120+0000) 2022-01-31T21:10:19.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:19 smithi171 conmon[51620]: debug 2022-01-31T21:10:19.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:19.605395+0000) 2022-01-31T21:10:19.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:19 smithi167 conmon[49112]: debug 2022-01-31T21:10:19.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:19.572828+0000) 2022-01-31T21:10:19.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:19 smithi167 conmon[54076]: debug 2022-01-31T21:10:19.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:19.498521+0000) 2022-01-31T21:10:20.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:19 smithi171 conmon[41853]: debug 2022-01-31T21:10:19.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:19.730928+0000) 2022-01-31T21:10:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:20 smithi171 conmon[46715]: debug 2022-01-31T21:10:20.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:20.026422+0000) 2022-01-31T21:10:20.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:20 smithi167 conmon[60316]: debug 2022-01-31T21:10:20.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:20.250252+0000) 2022-01-31T21:10:20.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:20 smithi171 conmon[51620]: debug 2022-01-31T21:10:20.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:20.605565+0000) 2022-01-31T21:10:20.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:20 smithi167 conmon[49112]: debug 2022-01-31T21:10:20.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:20.573038+0000) 2022-01-31T21:10:20.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:20 smithi167 conmon[54076]: debug 2022-01-31T21:10:20.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:20.498746+0000) 2022-01-31T21:10:21.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:20 smithi171 conmon[41853]: debug 2022-01-31T21:10:20.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:20.731084+0000) 2022-01-31T21:10:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[46715]: debug 2022-01-31T21:10:21.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.026534+0000) 2022-01-31T21:10:21.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:21 smithi167 conmon[60316]: debug 2022-01-31T21:10:21.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.250422+0000) 2022-01-31T21:10:21.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[51620]: debug 2022-01-31T21:10:21.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.605749+0000) 2022-01-31T21:10:21.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:21 smithi167 conmon[49112]: debug 2022-01-31T21:10:21.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.573219+0000) 2022-01-31T21:10:21.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:21 smithi167 conmon[54076]: debug 2022-01-31T21:10:21.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.498925+0000) 2022-01-31T21:10:22.025 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[46715]: debug 2022-01-31T21:10:21.811+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.811784+0000) 2022-01-31T21:10:22.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[51620]: debug 2022-01-31T21:10:21.810+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.811689+0000) 2022-01-31T21:10:22.026 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[35325]: debug 2022-01-31T21:10:21.822+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108683 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:22.026 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[41853]: debug 2022-01-31T21:10:21.731+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.731268+0000) 2022-01-31T21:10:22.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:21 smithi171 conmon[41853]: debug 2022-01-31T21:10:21.811+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.812417+0000) 2022-01-31T21:10:22.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:21 smithi167 conmon[49112]: debug 2022-01-31T21:10:21.811+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.813044+0000) 2022-01-31T21:10:22.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:21 smithi167 conmon[60316]: debug 2022-01-31T21:10:21.811+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.812785+0000) 2022-01-31T21:10:22.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:21 smithi167 conmon[54076]: debug 2022-01-31T21:10:21.810+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:21.811987+0000) 2022-01-31T21:10:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:22 smithi171 conmon[46715]: debug 2022-01-31T21:10:22.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:22.026695+0000) 2022-01-31T21:10:22.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:22 smithi167 conmon[60316]: debug 2022-01-31T21:10:22.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:22.250612+0000) 2022-01-31T21:10:22.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:22 smithi171 conmon[51620]: debug 2022-01-31T21:10:22.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:22.605952+0000) 2022-01-31T21:10:22.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:22 smithi167 conmon[49112]: debug 2022-01-31T21:10:22.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:22.573402+0000) 2022-01-31T21:10:22.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:22 smithi167 conmon[54076]: debug 2022-01-31T21:10:22.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:22.499103+0000) 2022-01-31T21:10:23.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:22 smithi171 conmon[41853]: debug 2022-01-31T21:10:22.731+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:22.731461+0000) 2022-01-31T21:10:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:23 smithi171 conmon[46715]: debug 2022-01-31T21:10:23.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:23.026803+0000) 2022-01-31T21:10:23.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:23 smithi167 conmon[60316]: debug 2022-01-31T21:10:23.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:23.250737+0000) 2022-01-31T21:10:23.730 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:23 smithi171 conmon[35325]: debug 2022-01-31T21:10:23.635+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:10:23.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:23 smithi171 conmon[51620]: debug 2022-01-31T21:10:23.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:23.606098+0000) 2022-01-31T21:10:23.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:23 smithi167 conmon[49112]: debug 2022-01-31T21:10:23.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:23.573573+0000) 2022-01-31T21:10:23.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:23 smithi167 conmon[54076]: debug 2022-01-31T21:10:23.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:23.499238+0000) 2022-01-31T21:10:24.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:23 smithi171 conmon[41853]: debug 2022-01-31T21:10:23.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:23.731634+0000) 2022-01-31T21:10:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:24 smithi171 conmon[46715]: debug 2022-01-31T21:10:24.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:24.026984+0000) 2022-01-31T21:10:24.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:24 smithi167 conmon[60316]: debug 2022-01-31T21:10:24.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:24.250934+0000) 2022-01-31T21:10:24.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:24 smithi171 conmon[51620]: debug 2022-01-31T21:10:24.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:24.606280+0000) 2022-01-31T21:10:24.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:24 smithi167 conmon[49112]: debug 2022-01-31T21:10:24.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:24.573775+0000) 2022-01-31T21:10:24.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:24 smithi167 conmon[54076]: debug 2022-01-31T21:10:24.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:24.499405+0000) 2022-01-31T21:10:25.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:24 smithi171 conmon[41853]: debug 2022-01-31T21:10:24.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:24.731796+0000) 2022-01-31T21:10:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:25 smithi171 conmon[46715]: debug 2022-01-31T21:10:25.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:25.027167+0000) 2022-01-31T21:10:25.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:25 smithi167 conmon[60316]: debug 2022-01-31T21:10:25.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:25.251119+0000) 2022-01-31T21:10:25.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:25 smithi171 conmon[51620]: debug 2022-01-31T21:10:25.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:25.606456+0000) 2022-01-31T21:10:25.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:25 smithi167 conmon[49112]: debug 2022-01-31T21:10:25.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:25.573923+0000) 2022-01-31T21:10:25.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:25 smithi167 conmon[54076]: debug 2022-01-31T21:10:25.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:25.499612+0000) 2022-01-31T21:10:26.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:25 smithi171 conmon[41853]: debug 2022-01-31T21:10:25.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:25.731976+0000) 2022-01-31T21:10:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[46715]: debug 2022-01-31T21:10:26.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.027323+0000) 2022-01-31T21:10:26.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:26 smithi167 conmon[60316]: debug 2022-01-31T21:10:26.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.251312+0000) 2022-01-31T21:10:26.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[51620]: debug 2022-01-31T21:10:26.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.606648+0000) 2022-01-31T21:10:26.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:26 smithi167 conmon[49112]: debug 2022-01-31T21:10:26.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.574072+0000) 2022-01-31T21:10:26.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:26 smithi167 conmon[54076]: debug 2022-01-31T21:10:26.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.499814+0000) 2022-01-31T21:10:27.026 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[35325]: debug 2022-01-31T21:10:26.834+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108788 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:27.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[51620]: debug 2022-01-31T21:10:26.823+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.825198+0000) 2022-01-31T21:10:27.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[41853]: debug 2022-01-31T21:10:26.730+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.732105+0000) 2022-01-31T21:10:27.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[41853]: debug 2022-01-31T21:10:26.823+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.825719+0000) 2022-01-31T21:10:27.027 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:26 smithi171 conmon[46715]: debug 2022-01-31T21:10:26.823+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.825088+0000) 2022-01-31T21:10:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:26 smithi167 conmon[49112]: debug 2022-01-31T21:10:26.824+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.826369+0000) 2022-01-31T21:10:27.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:26 smithi167 conmon[54076]: debug 2022-01-31T21:10:26.824+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.825597+0000) 2022-01-31T21:10:27.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:26 smithi167 conmon[60316]: debug 2022-01-31T21:10:26.825+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:26.827283+0000) 2022-01-31T21:10:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:27 smithi171 conmon[46715]: debug 2022-01-31T21:10:27.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:27.027510+0000) 2022-01-31T21:10:27.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:27 smithi167 conmon[60316]: debug 2022-01-31T21:10:27.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:27.251530+0000) 2022-01-31T21:10:27.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:27 smithi171 conmon[51620]: debug 2022-01-31T21:10:27.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:27.606841+0000) 2022-01-31T21:10:27.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:27 smithi167 conmon[49112]: debug 2022-01-31T21:10:27.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:27.574234+0000) 2022-01-31T21:10:27.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:27 smithi167 conmon[54076]: debug 2022-01-31T21:10:27.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:27.499992+0000) 2022-01-31T21:10:28.025 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:27 smithi171 conmon[41853]: debug 2022-01-31T21:10:27.731+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:27.732262+0000) 2022-01-31T21:10:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:28 smithi171 conmon[46715]: debug 2022-01-31T21:10:28.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:28.027641+0000) 2022-01-31T21:10:28.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:28 smithi167 conmon[60316]: debug 2022-01-31T21:10:28.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:28.251648+0000) 2022-01-31T21:10:28.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:28 smithi171 conmon[51620]: debug 2022-01-31T21:10:28.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:28.607029+0000) 2022-01-31T21:10:28.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:28 smithi167 conmon[49112]: debug 2022-01-31T21:10:28.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:28.574359+0000) 2022-01-31T21:10:28.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:28 smithi167 conmon[54076]: debug 2022-01-31T21:10:28.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:28.500197+0000) 2022-01-31T21:10:29.026 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:28 smithi171 conmon[41853]: debug 2022-01-31T21:10:28.731+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:28.732413+0000) 2022-01-31T21:10:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:29 smithi171 conmon[46715]: debug 2022-01-31T21:10:29.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:29.027772+0000) 2022-01-31T21:10:29.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:29 smithi167 conmon[60316]: debug 2022-01-31T21:10:29.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:29.251804+0000) 2022-01-31T21:10:29.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:29 smithi171 conmon[51620]: debug 2022-01-31T21:10:29.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:29.607182+0000) 2022-01-31T21:10:29.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:29 smithi167 conmon[49112]: debug 2022-01-31T21:10:29.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:29.574534+0000) 2022-01-31T21:10:29.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:29 smithi167 conmon[54076]: debug 2022-01-31T21:10:29.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:29.500393+0000) 2022-01-31T21:10:30.026 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:29 smithi171 conmon[41853]: debug 2022-01-31T21:10:29.731+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:29.732589+0000) 2022-01-31T21:10:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:30 smithi171 conmon[46715]: debug 2022-01-31T21:10:30.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:30.027927+0000) 2022-01-31T21:10:30.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:30 smithi167 conmon[60316]: debug 2022-01-31T21:10:30.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:30.252007+0000) 2022-01-31T21:10:30.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:30 smithi171 conmon[51620]: debug 2022-01-31T21:10:30.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:30.607338+0000) 2022-01-31T21:10:30.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:30 smithi167 conmon[49112]: debug 2022-01-31T21:10:30.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:30.574741+0000) 2022-01-31T21:10:30.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:30 smithi167 conmon[54076]: debug 2022-01-31T21:10:30.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:30.500578+0000) 2022-01-31T21:10:31.026 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:30 smithi171 conmon[41853]: debug 2022-01-31T21:10:30.731+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:30.732783+0000) 2022-01-31T21:10:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[46715]: debug 2022-01-31T21:10:31.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.028088+0000) 2022-01-31T21:10:31.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:31 smithi167 conmon[60316]: debug 2022-01-31T21:10:31.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.252207+0000) 2022-01-31T21:10:31.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[51620]: debug 2022-01-31T21:10:31.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.607538+0000) 2022-01-31T21:10:31.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:31 smithi167 conmon[49112]: debug 2022-01-31T21:10:31.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.574887+0000) 2022-01-31T21:10:31.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:31 smithi167 conmon[54076]: debug 2022-01-31T21:10:31.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.500720+0000) 2022-01-31T21:10:32.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[51620]: debug 2022-01-31T21:10:31.836+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.838427+0000) 2022-01-31T21:10:32.027 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[35325]: debug 2022-01-31T21:10:31.848+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 108899 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:32.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[41853]: debug 2022-01-31T21:10:31.732+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.732971+0000) 2022-01-31T21:10:32.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[41853]: debug 2022-01-31T21:10:31.838+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.840193+0000) 2022-01-31T21:10:32.028 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:31 smithi171 conmon[46715]: debug 2022-01-31T21:10:31.836+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.838586+0000) 2022-01-31T21:10:32.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:31 smithi167 conmon[49112]: debug 2022-01-31T21:10:31.838+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.838966+0000) 2022-01-31T21:10:32.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:31 smithi167 conmon[54076]: debug 2022-01-31T21:10:31.837+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.838268+0000) 2022-01-31T21:10:32.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:31 smithi167 conmon[60316]: debug 2022-01-31T21:10:31.839+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:31.839743+0000) 2022-01-31T21:10:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:32 smithi171 conmon[46715]: debug 2022-01-31T21:10:32.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:32.028239+0000) 2022-01-31T21:10:32.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:32 smithi167 conmon[60316]: debug 2022-01-31T21:10:32.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:32.252373+0000) 2022-01-31T21:10:32.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:32 smithi171 conmon[51620]: debug 2022-01-31T21:10:32.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:32.607694+0000) 2022-01-31T21:10:32.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:32 smithi167 conmon[49112]: debug 2022-01-31T21:10:32.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:32.574991+0000) 2022-01-31T21:10:32.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:32 smithi167 conmon[54076]: debug 2022-01-31T21:10:32.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:32.500939+0000) 2022-01-31T21:10:33.026 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:32 smithi171 conmon[41853]: debug 2022-01-31T21:10:32.732+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:32.733191+0000) 2022-01-31T21:10:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:33 smithi171 conmon[46715]: debug 2022-01-31T21:10:33.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:33.028395+0000) 2022-01-31T21:10:33.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:33 smithi167 conmon[60316]: debug 2022-01-31T21:10:33.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:33.252458+0000) 2022-01-31T21:10:33.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:33 smithi171 conmon[51620]: debug 2022-01-31T21:10:33.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:33.607856+0000) 2022-01-31T21:10:33.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:33 smithi167 conmon[49112]: debug 2022-01-31T21:10:33.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:33.575110+0000) 2022-01-31T21:10:33.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:33 smithi167 conmon[54076]: debug 2022-01-31T21:10:33.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:33.501093+0000) 2022-01-31T21:10:34.026 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:33 smithi171 conmon[41853]: debug 2022-01-31T21:10:33.732+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:33.733316+0000) 2022-01-31T21:10:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:34 smithi171 conmon[46715]: debug 2022-01-31T21:10:34.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:34.028602+0000) 2022-01-31T21:10:34.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:34 smithi167 conmon[60316]: debug 2022-01-31T21:10:34.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:34.252641+0000) 2022-01-31T21:10:34.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:34 smithi171 conmon[51620]: debug 2022-01-31T21:10:34.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:34.608034+0000) 2022-01-31T21:10:34.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:34 smithi167 conmon[49112]: debug 2022-01-31T21:10:34.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:34.575283+0000) 2022-01-31T21:10:34.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:34 smithi167 conmon[54076]: debug 2022-01-31T21:10:34.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:34.501278+0000) 2022-01-31T21:10:35.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:34 smithi171 conmon[41853]: debug 2022-01-31T21:10:34.732+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:34.733496+0000) 2022-01-31T21:10:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:35 smithi171 conmon[46715]: debug 2022-01-31T21:10:35.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:35.028787+0000) 2022-01-31T21:10:35.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:35 smithi167 conmon[60316]: debug 2022-01-31T21:10:35.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:35.252794+0000) 2022-01-31T21:10:35.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:35 smithi171 conmon[51620]: debug 2022-01-31T21:10:35.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:35.608223+0000) 2022-01-31T21:10:35.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:35 smithi167 conmon[54076]: debug 2022-01-31T21:10:35.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:35.501457+0000) 2022-01-31T21:10:35.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:35 smithi167 conmon[49112]: debug 2022-01-31T21:10:35.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:35.575492+0000) 2022-01-31T21:10:36.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:35 smithi171 conmon[41853]: debug 2022-01-31T21:10:35.732+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:35.733738+0000) 2022-01-31T21:10:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[46715]: debug 2022-01-31T21:10:36.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.028944+0000) 2022-01-31T21:10:36.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:36 smithi167 conmon[60316]: debug 2022-01-31T21:10:36.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.252972+0000) 2022-01-31T21:10:36.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[51620]: debug 2022-01-31T21:10:36.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.608408+0000) 2022-01-31T21:10:36.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:36 smithi167 conmon[49112]: debug 2022-01-31T21:10:36.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.575662+0000) 2022-01-31T21:10:36.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:36 smithi167 conmon[54076]: debug 2022-01-31T21:10:36.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.501665+0000) 2022-01-31T21:10:37.027 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[35325]: debug 2022-01-31T21:10:36.863+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109009 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:37.028 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[46715]: debug 2022-01-31T21:10:36.852+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.853002+0000) 2022-01-31T21:10:37.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[51620]: debug 2022-01-31T21:10:36.851+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.852131+0000) 2022-01-31T21:10:37.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[41853]: debug 2022-01-31T21:10:36.733+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.733910+0000) 2022-01-31T21:10:37.029 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:36 smithi171 conmon[41853]: debug 2022-01-31T21:10:36.853+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.853809+0000) 2022-01-31T21:10:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:36 smithi167 conmon[49112]: debug 2022-01-31T21:10:36.851+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.852389+0000) 2022-01-31T21:10:37.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:36 smithi167 conmon[54076]: debug 2022-01-31T21:10:36.852+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.852704+0000) 2022-01-31T21:10:37.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:36 smithi167 conmon[60316]: debug 2022-01-31T21:10:36.853+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:36.853594+0000) 2022-01-31T21:10:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:37 smithi171 conmon[46715]: debug 2022-01-31T21:10:37.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:37.029069+0000) 2022-01-31T21:10:37.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:37 smithi167 conmon[60316]: debug 2022-01-31T21:10:37.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:37.253102+0000) 2022-01-31T21:10:37.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:37 smithi171 conmon[51620]: debug 2022-01-31T21:10:37.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:37.608617+0000) 2022-01-31T21:10:37.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:37 smithi167 conmon[54076]: debug 2022-01-31T21:10:37.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:37.501843+0000) 2022-01-31T21:10:37.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:37 smithi167 conmon[49112]: debug 2022-01-31T21:10:37.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:37.575845+0000) 2022-01-31T21:10:38.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:37 smithi171 conmon[41853]: debug 2022-01-31T21:10:37.733+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:37.734048+0000) 2022-01-31T21:10:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:38 smithi171 conmon[46715]: debug 2022-01-31T21:10:38.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:38.029210+0000) 2022-01-31T21:10:38.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:38 smithi167 conmon[60316]: debug 2022-01-31T21:10:38.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:38.253259+0000) 2022-01-31T21:10:38.732 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:38 smithi171 conmon[35325]: debug 2022-01-31T21:10:38.635+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:10:38.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:38 smithi171 conmon[51620]: debug 2022-01-31T21:10:38.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:38.608788+0000) 2022-01-31T21:10:38.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:38 smithi167 conmon[54076]: debug 2022-01-31T21:10:38.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:38.502038+0000) 2022-01-31T21:10:38.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:38 smithi167 conmon[49112]: debug 2022-01-31T21:10:38.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:38.576002+0000) 2022-01-31T21:10:39.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:38 smithi171 conmon[41853]: debug 2022-01-31T21:10:38.733+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:38.734189+0000) 2022-01-31T21:10:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:39 smithi171 conmon[46715]: debug 2022-01-31T21:10:39.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:39.029386+0000) 2022-01-31T21:10:39.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:39 smithi167 conmon[60316]: debug 2022-01-31T21:10:39.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:39.253427+0000) 2022-01-31T21:10:39.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:39 smithi171 conmon[51620]: debug 2022-01-31T21:10:39.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:39.608952+0000) 2022-01-31T21:10:39.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:39 smithi167 conmon[49112]: debug 2022-01-31T21:10:39.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:39.576185+0000) 2022-01-31T21:10:39.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:39 smithi167 conmon[54076]: debug 2022-01-31T21:10:39.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:39.502151+0000) 2022-01-31T21:10:40.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:39 smithi171 conmon[41853]: debug 2022-01-31T21:10:39.733+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:39.734347+0000) 2022-01-31T21:10:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:40 smithi171 conmon[46715]: debug 2022-01-31T21:10:40.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:40.029537+0000) 2022-01-31T21:10:40.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:40 smithi167 conmon[60316]: debug 2022-01-31T21:10:40.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:40.253593+0000) 2022-01-31T21:10:40.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:40 smithi171 conmon[51620]: debug 2022-01-31T21:10:40.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:40.609094+0000) 2022-01-31T21:10:40.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:40 smithi167 conmon[49112]: debug 2022-01-31T21:10:40.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:40.576390+0000) 2022-01-31T21:10:40.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:40 smithi167 conmon[54076]: debug 2022-01-31T21:10:40.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:40.502327+0000) 2022-01-31T21:10:41.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:40 smithi171 conmon[41853]: debug 2022-01-31T21:10:40.733+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:40.734513+0000) 2022-01-31T21:10:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[46715]: debug 2022-01-31T21:10:41.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.029762+0000) 2022-01-31T21:10:41.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:41 smithi167 conmon[60316]: debug 2022-01-31T21:10:41.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.253784+0000) 2022-01-31T21:10:41.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[51620]: debug 2022-01-31T21:10:41.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.609292+0000) 2022-01-31T21:10:41.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:41 smithi167 conmon[49112]: debug 2022-01-31T21:10:41.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.576579+0000) 2022-01-31T21:10:41.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:41 smithi167 conmon[54076]: debug 2022-01-31T21:10:41.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.502529+0000) 2022-01-31T21:10:42.028 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[35325]: debug 2022-01-31T21:10:41.879+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109120 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:42.029 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[46715]: debug 2022-01-31T21:10:41.866+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.867102+0000) 2022-01-31T21:10:42.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[51620]: debug 2022-01-31T21:10:41.866+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.867320+0000) 2022-01-31T21:10:42.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[41853]: debug 2022-01-31T21:10:41.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.734663+0000) 2022-01-31T21:10:42.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:41 smithi171 conmon[41853]: debug 2022-01-31T21:10:41.867+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.868271+0000) 2022-01-31T21:10:42.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:41 smithi167 conmon[49112]: debug 2022-01-31T21:10:41.867+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.867813+0000) 2022-01-31T21:10:42.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:41 smithi167 conmon[54076]: debug 2022-01-31T21:10:41.866+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.866978+0000) 2022-01-31T21:10:42.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:41 smithi167 conmon[60316]: debug 2022-01-31T21:10:41.891+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:41.892122+0000) 2022-01-31T21:10:42.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:42 smithi171 conmon[46715]: debug 2022-01-31T21:10:42.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:42.029955+0000) 2022-01-31T21:10:42.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:42 smithi167 conmon[60316]: debug 2022-01-31T21:10:42.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:42.253963+0000) 2022-01-31T21:10:42.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:42 smithi171 conmon[51620]: debug 2022-01-31T21:10:42.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:42.609456+0000) 2022-01-31T21:10:42.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:42 smithi167 conmon[49112]: debug 2022-01-31T21:10:42.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:42.576675+0000) 2022-01-31T21:10:42.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:42 smithi167 conmon[54076]: debug 2022-01-31T21:10:42.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:42.502660+0000) 2022-01-31T21:10:43.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:42 smithi171 conmon[41853]: debug 2022-01-31T21:10:42.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:42.734861+0000) 2022-01-31T21:10:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:43 smithi171 conmon[46715]: debug 2022-01-31T21:10:43.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:43.030073+0000) 2022-01-31T21:10:43.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:43 smithi167 conmon[60316]: debug 2022-01-31T21:10:43.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:43.254139+0000) 2022-01-31T21:10:43.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:43 smithi171 conmon[51620]: debug 2022-01-31T21:10:43.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:43.609571+0000) 2022-01-31T21:10:43.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:43 smithi167 conmon[49112]: debug 2022-01-31T21:10:43.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:43.576820+0000) 2022-01-31T21:10:43.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:43 smithi167 conmon[54076]: debug 2022-01-31T21:10:43.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:43.502763+0000) 2022-01-31T21:10:44.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:43 smithi171 conmon[41853]: debug 2022-01-31T21:10:43.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:43.734943+0000) 2022-01-31T21:10:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:44 smithi171 conmon[46715]: debug 2022-01-31T21:10:44.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:44.030226+0000) 2022-01-31T21:10:44.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:44 smithi167 conmon[60316]: debug 2022-01-31T21:10:44.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:44.254316+0000) 2022-01-31T21:10:44.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:44 smithi171 conmon[51620]: debug 2022-01-31T21:10:44.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:44.609736+0000) 2022-01-31T21:10:44.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:44 smithi167 conmon[49112]: debug 2022-01-31T21:10:44.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:44.577003+0000) 2022-01-31T21:10:44.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:44 smithi167 conmon[54076]: debug 2022-01-31T21:10:44.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:44.502920+0000) 2022-01-31T21:10:45.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:44 smithi171 conmon[41853]: debug 2022-01-31T21:10:44.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:44.735095+0000) 2022-01-31T21:10:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:45 smithi171 conmon[46715]: debug 2022-01-31T21:10:45.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:45.030355+0000) 2022-01-31T21:10:45.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:45 smithi167 conmon[60316]: debug 2022-01-31T21:10:45.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:45.254496+0000) 2022-01-31T21:10:45.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:45 smithi171 conmon[51620]: debug 2022-01-31T21:10:45.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:45.609958+0000) 2022-01-31T21:10:45.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:45 smithi167 conmon[49112]: debug 2022-01-31T21:10:45.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:45.577188+0000) 2022-01-31T21:10:45.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:45 smithi167 conmon[54076]: debug 2022-01-31T21:10:45.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:45.503060+0000) 2022-01-31T21:10:46.028 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:45 smithi171 conmon[41853]: debug 2022-01-31T21:10:45.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:45.735283+0000) 2022-01-31T21:10:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[46715]: debug 2022-01-31T21:10:46.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.030494+0000) 2022-01-31T21:10:46.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:46 smithi167 conmon[60316]: debug 2022-01-31T21:10:46.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.254710+0000) 2022-01-31T21:10:46.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[51620]: debug 2022-01-31T21:10:46.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.610078+0000) 2022-01-31T21:10:46.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:46 smithi167 conmon[49112]: debug 2022-01-31T21:10:46.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.577350+0000) 2022-01-31T21:10:46.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:46 smithi167 conmon[54076]: debug 2022-01-31T21:10:46.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.503179+0000) 2022-01-31T21:10:47.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[35325]: debug 2022-01-31T21:10:46.893+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109230 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:47.029 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[41853]: debug 2022-01-31T21:10:46.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.735464+0000) 2022-01-31T21:10:47.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[41853]: debug 2022-01-31T21:10:46.882+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.883657+0000) 2022-01-31T21:10:47.030 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[46715]: debug 2022-01-31T21:10:46.881+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.882336+0000) 2022-01-31T21:10:47.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:46 smithi171 conmon[51620]: debug 2022-01-31T21:10:46.881+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.882627+0000) 2022-01-31T21:10:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:46 smithi167 conmon[49112]: debug 2022-01-31T21:10:46.889+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.889950+0000) 2022-01-31T21:10:47.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:46 smithi167 conmon[54076]: debug 2022-01-31T21:10:46.882+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.883010+0000) 2022-01-31T21:10:47.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:46 smithi167 conmon[60316]: debug 2022-01-31T21:10:46.882+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:46.882906+0000) 2022-01-31T21:10:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:47 smithi171 conmon[46715]: debug 2022-01-31T21:10:47.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:47.030652+0000) 2022-01-31T21:10:47.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:47 smithi167 conmon[60316]: debug 2022-01-31T21:10:47.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:47.254919+0000) 2022-01-31T21:10:47.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:47 smithi171 conmon[51620]: debug 2022-01-31T21:10:47.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:47.610268+0000) 2022-01-31T21:10:47.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:47 smithi167 conmon[54076]: debug 2022-01-31T21:10:47.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:47.503380+0000) 2022-01-31T21:10:47.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:47 smithi167 conmon[49112]: debug 2022-01-31T21:10:47.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:47.577528+0000) 2022-01-31T21:10:48.029 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:47 smithi171 conmon[41853]: debug 2022-01-31T21:10:47.734+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:47.735667+0000) 2022-01-31T21:10:48.220 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T21:10:48.221+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T21:10:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:48 smithi171 conmon[46715]: debug 2022-01-31T21:10:48.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:48.030790+0000) 2022-01-31T21:10:48.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:48 smithi167 conmon[60316]: debug 2022-01-31T21:10:48.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:48.255076+0000) 2022-01-31T21:10:48.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:48 smithi171 conmon[51620]: debug 2022-01-31T21:10:48.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:48.610442+0000) 2022-01-31T21:10:48.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:48 smithi167 conmon[49112]: debug 2022-01-31T21:10:48.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:48.577696+0000) 2022-01-31T21:10:48.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:48 smithi167 conmon[54076]: debug 2022-01-31T21:10:48.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:48.503530+0000) 2022-01-31T21:10:49.029 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:48 smithi171 conmon[41853]: debug 2022-01-31T21:10:48.735+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:48.735817+0000) 2022-01-31T21:10:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:49 smithi171 conmon[46715]: debug 2022-01-31T21:10:49.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:49.030985+0000) 2022-01-31T21:10:49.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:49 smithi167 conmon[60316]: debug 2022-01-31T21:10:49.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:49.255285+0000) 2022-01-31T21:10:49.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:49 smithi171 conmon[51620]: debug 2022-01-31T21:10:49.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:49.610623+0000) 2022-01-31T21:10:49.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:49 smithi167 conmon[54076]: debug 2022-01-31T21:10:49.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:49.503690+0000) 2022-01-31T21:10:49.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:49 smithi167 conmon[49112]: debug 2022-01-31T21:10:49.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:49.577898+0000) 2022-01-31T21:10:50.029 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:49 smithi171 conmon[41853]: debug 2022-01-31T21:10:49.735+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:49.735950+0000) 2022-01-31T21:10:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:50 smithi171 conmon[46715]: debug 2022-01-31T21:10:50.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:50.031127+0000) 2022-01-31T21:10:50.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:50 smithi167 conmon[60316]: debug 2022-01-31T21:10:50.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:50.255478+0000) 2022-01-31T21:10:50.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:50 smithi171 conmon[51620]: debug 2022-01-31T21:10:50.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:50.610807+0000) 2022-01-31T21:10:50.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:50 smithi167 conmon[54076]: debug 2022-01-31T21:10:50.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:50.503882+0000) 2022-01-31T21:10:50.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:50 smithi167 conmon[49112]: debug 2022-01-31T21:10:50.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:50.578102+0000) 2022-01-31T21:10:51.029 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:50 smithi171 conmon[41853]: debug 2022-01-31T21:10:50.735+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:50.736098+0000) 2022-01-31T21:10:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[46715]: debug 2022-01-31T21:10:51.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.031285+0000) 2022-01-31T21:10:51.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[60316]: debug 2022-01-31T21:10:51.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.255710+0000) 2022-01-31T21:10:51.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[51620]: debug 2022-01-31T21:10:51.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.610967+0000) 2022-01-31T21:10:51.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[54076]: debug 2022-01-31T21:10:51.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.504060+0000) 2022-01-31T21:10:51.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[49112]: debug 2022-01-31T21:10:51.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.578274+0000) 2022-01-31T21:10:52.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[35325]: debug 2022-01-31T21:10:51.906+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109341 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:52.030 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[46715]: debug 2022-01-31T21:10:51.895+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.896242+0000) 2022-01-31T21:10:52.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[51620]: debug 2022-01-31T21:10:51.895+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.896371+0000) 2022-01-31T21:10:52.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[41853]: debug 2022-01-31T21:10:51.735+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.736256+0000) 2022-01-31T21:10:52.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:51 smithi171 conmon[41853]: debug 2022-01-31T21:10:51.897+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.897853+0000) 2022-01-31T21:10:52.167 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[54076]: debug 2022-01-31T21:10:51.895+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.896469+0000) 2022-01-31T21:10:52.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[60316]: debug 2022-01-31T21:10:51.896+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.897255+0000) 2022-01-31T21:10:52.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[49112]: debug 2022-01-31T21:10:52.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:51 smithi167 conmon[49112]: 2022-01-31T21:10:51.896+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:51.896811+0000) 2022-01-31T21:10:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:52 smithi171 conmon[46715]: debug 2022-01-31T21:10:52.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:52.031426+0000) 2022-01-31T21:10:52.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:52 smithi167 conmon[60316]: debug 2022-01-31T21:10:52.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:52.255912+0000) 2022-01-31T21:10:52.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:52 smithi171 conmon[51620]: debug 2022-01-31T21:10:52.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:52.611143+0000) 2022-01-31T21:10:52.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:52 smithi167 conmon[49112]: debug 2022-01-31T21:10:52.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:52.578405+0000) 2022-01-31T21:10:52.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:52 smithi167 conmon[54076]: debug 2022-01-31T21:10:52.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:52.504265+0000) 2022-01-31T21:10:53.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:52 smithi171 conmon[41853]: debug 2022-01-31T21:10:52.735+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:52.736452+0000) 2022-01-31T21:10:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:53 smithi171 conmon[46715]: debug 2022-01-31T21:10:53.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:53.031527+0000) 2022-01-31T21:10:53.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:53 smithi167 conmon[60316]: debug 2022-01-31T21:10:53.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:53.256027+0000) 2022-01-31T21:10:53.735 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:53 smithi171 conmon[35325]: debug 2022-01-31T21:10:53.636+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:10:53.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:53 smithi171 conmon[51620]: debug 2022-01-31T21:10:53.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:53.611325+0000) 2022-01-31T21:10:53.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:53 smithi167 conmon[49112]: debug 2022-01-31T21:10:53.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:53.578564+0000) 2022-01-31T21:10:53.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:53 smithi167 conmon[54076]: debug 2022-01-31T21:10:53.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:53.504407+0000) 2022-01-31T21:10:54.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:53 smithi171 conmon[41853]: debug 2022-01-31T21:10:53.735+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:53.736627+0000) 2022-01-31T21:10:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:54 smithi171 conmon[46715]: debug 2022-01-31T21:10:54.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:54.031684+0000) 2022-01-31T21:10:54.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:54 smithi167 conmon[60316]: debug 2022-01-31T21:10:54.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:54.256209+0000) 2022-01-31T21:10:54.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:54 smithi171 conmon[51620]: debug 2022-01-31T21:10:54.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:54.611512+0000) 2022-01-31T21:10:54.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:54 smithi167 conmon[54076]: debug 2022-01-31T21:10:54.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:54.504546+0000) 2022-01-31T21:10:54.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:54 smithi167 conmon[49112]: debug 2022-01-31T21:10:54.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:54.578788+0000) 2022-01-31T21:10:55.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:54 smithi171 conmon[41853]: debug 2022-01-31T21:10:54.736+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:54.736793+0000) 2022-01-31T21:10:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:55 smithi171 conmon[46715]: debug 2022-01-31T21:10:55.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:55.031860+0000) 2022-01-31T21:10:55.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:55 smithi167 conmon[60316]: debug 2022-01-31T21:10:55.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:55.256336+0000) 2022-01-31T21:10:55.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:55 smithi171 conmon[51620]: debug 2022-01-31T21:10:55.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:55.611699+0000) 2022-01-31T21:10:55.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:55 smithi167 conmon[49112]: debug 2022-01-31T21:10:55.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:55.579004+0000) 2022-01-31T21:10:55.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:55 smithi167 conmon[54076]: debug 2022-01-31T21:10:55.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:55.504661+0000) 2022-01-31T21:10:56.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:55 smithi171 conmon[41853]: debug 2022-01-31T21:10:55.736+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:55.736974+0000) 2022-01-31T21:10:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[46715]: debug 2022-01-31T21:10:56.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.031981+0000) 2022-01-31T21:10:56.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:56 smithi167 conmon[60316]: debug 2022-01-31T21:10:56.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.256520+0000) 2022-01-31T21:10:56.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[51620]: debug 2022-01-31T21:10:56.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.611907+0000) 2022-01-31T21:10:56.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:56 smithi167 conmon[54076]: debug 2022-01-31T21:10:56.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.504860+0000) 2022-01-31T21:10:56.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:56 smithi167 conmon[49112]: debug 2022-01-31T21:10:56.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.579174+0000) 2022-01-31T21:10:57.031 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[35325]: debug 2022-01-31T21:10:56.920+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109452 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:10:57.127 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[46715]: debug 2022-01-31T21:10:56.910+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.910992+0000) 2022-01-31T21:10:57.127 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[51620]: debug 2022-01-31T21:10:56.909+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.910301+0000) 2022-01-31T21:10:57.128 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[41853]: debug 2022-01-31T21:10:56.736+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.737178+0000) 2022-01-31T21:10:57.128 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:56 smithi171 conmon[41853]: debug 2022-01-31T21:10:56.910+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.911348+0000) 2022-01-31T21:10:57.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:56 smithi167 conmon[49112]: debug 2022-01-31T21:10:56.909+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.911045+0000) 2022-01-31T21:10:57.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:56 smithi167 conmon[54076]: debug 2022-01-31T21:10:56.909+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.911126+0000) 2022-01-31T21:10:57.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:56 smithi167 conmon[60316]: debug 2022-01-31T21:10:56.908+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:56.910449+0000) 2022-01-31T21:10:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:57 smithi171 conmon[46715]: debug 2022-01-31T21:10:57.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:57.032098+0000) 2022-01-31T21:10:57.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:57 smithi167 conmon[60316]: debug 2022-01-31T21:10:57.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:57.256701+0000) 2022-01-31T21:10:57.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:57 smithi171 conmon[51620]: debug 2022-01-31T21:10:57.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:57.612103+0000) 2022-01-31T21:10:57.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:57 smithi167 conmon[54076]: debug 2022-01-31T21:10:57.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:57.505040+0000) 2022-01-31T21:10:57.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:57 smithi167 conmon[49112]: debug 2022-01-31T21:10:57.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:57.579361+0000) 2022-01-31T21:10:58.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:57 smithi171 conmon[41853]: debug 2022-01-31T21:10:57.737+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:57.737365+0000) 2022-01-31T21:10:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:58 smithi171 conmon[46715]: debug 2022-01-31T21:10:58.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:58.032253+0000) 2022-01-31T21:10:58.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:58 smithi167 conmon[60316]: debug 2022-01-31T21:10:58.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:58.256827+0000) 2022-01-31T21:10:58.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:58 smithi171 conmon[51620]: debug 2022-01-31T21:10:58.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:58.612274+0000) 2022-01-31T21:10:58.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:58 smithi167 conmon[49112]: debug 2022-01-31T21:10:58.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:58.579537+0000) 2022-01-31T21:10:58.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:58 smithi167 conmon[54076]: debug 2022-01-31T21:10:58.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:58.505221+0000) 2022-01-31T21:10:59.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:58 smithi171 conmon[41853]: debug 2022-01-31T21:10:58.737+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:58.737530+0000) 2022-01-31T21:10:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:10:59 smithi171 conmon[46715]: debug 2022-01-31T21:10:59.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:59.032468+0000) 2022-01-31T21:10:59.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:10:59 smithi167 conmon[60316]: debug 2022-01-31T21:10:59.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:59.256922+0000) 2022-01-31T21:10:59.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:10:59 smithi171 conmon[51620]: debug 2022-01-31T21:10:59.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:59.612388+0000) 2022-01-31T21:10:59.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:10:59 smithi167 conmon[49112]: debug 2022-01-31T21:10:59.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:59.579725+0000) 2022-01-31T21:10:59.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:10:59 smithi167 conmon[54076]: debug 2022-01-31T21:10:59.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:59.505410+0000) 2022-01-31T21:11:00.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:10:59 smithi171 conmon[41853]: debug 2022-01-31T21:10:59.737+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:10:59.737706+0000) 2022-01-31T21:11:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:00 smithi171 conmon[46715]: debug 2022-01-31T21:11:00.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:00.032678+0000) 2022-01-31T21:11:00.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:00 smithi167 conmon[60316]: debug 2022-01-31T21:11:00.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:00.257116+0000) 2022-01-31T21:11:00.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:00 smithi171 conmon[51620]: debug 2022-01-31T21:11:00.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:00.612525+0000) 2022-01-31T21:11:00.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:00 smithi167 conmon[49112]: debug 2022-01-31T21:11:00.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:00.579909+0000) 2022-01-31T21:11:00.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:00 smithi167 conmon[54076]: debug 2022-01-31T21:11:00.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:00.505594+0000) 2022-01-31T21:11:01.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:00 smithi171 conmon[41853]: debug 2022-01-31T21:11:00.737+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:00.737875+0000) 2022-01-31T21:11:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[46715]: debug 2022-01-31T21:11:01.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.032794+0000) 2022-01-31T21:11:01.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:01 smithi167 conmon[60316]: debug 2022-01-31T21:11:01.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.257273+0000) 2022-01-31T21:11:01.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[51620]: debug 2022-01-31T21:11:01.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.612680+0000) 2022-01-31T21:11:01.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:01 smithi167 conmon[49112]: debug 2022-01-31T21:11:01.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.580118+0000) 2022-01-31T21:11:01.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:01 smithi167 conmon[54076]: debug 2022-01-31T21:11:01.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.505750+0000) 2022-01-31T21:11:02.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[51620]: debug 2022-01-31T21:11:01.922+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.923215+0000) 2022-01-31T21:11:02.034 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[35325]: debug 2022-01-31T21:11:01.933+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109563 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:02.034 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[41853]: debug 2022-01-31T21:11:01.737+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.738065+0000) 2022-01-31T21:11:02.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[41853]: debug 2022-01-31T21:11:01.923+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.924660+0000) 2022-01-31T21:11:02.035 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:01 smithi171 conmon[46715]: debug 2022-01-31T21:11:01.922+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.923476+0000) 2022-01-31T21:11:02.035 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:02 smithi171 conmon[46715]: debug 2022-01-31T21:11:02.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:02.032929+0000) 2022-01-31T21:11:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:01 smithi167 conmon[49112]: debug 2022-01-31T21:11:01.923+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.924506+0000) 2022-01-31T21:11:02.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:01 smithi167 conmon[54076]: debug 2022-01-31T21:11:01.923+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.925006+0000) 2022-01-31T21:11:02.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:01 smithi167 conmon[60316]: debug 2022-01-31T21:11:01.922+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:01.924152+0000) 2022-01-31T21:11:02.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:02 smithi167 conmon[60316]: debug 2022-01-31T21:11:02.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:02.257432+0000) 2022-01-31T21:11:02.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:02 smithi171 conmon[51620]: debug 2022-01-31T21:11:02.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:02.612889+0000) 2022-01-31T21:11:02.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:02 smithi167 conmon[49112]: debug 2022-01-31T21:11:02.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:02.580275+0000) 2022-01-31T21:11:02.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:02 smithi167 conmon[54076]: debug 2022-01-31T21:11:02.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:02.505936+0000) 2022-01-31T21:11:03.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:02 smithi171 conmon[41853]: debug 2022-01-31T21:11:02.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:02.738243+0000) 2022-01-31T21:11:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:03 smithi171 conmon[46715]: debug 2022-01-31T21:11:03.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:03.033067+0000) 2022-01-31T21:11:03.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:03 smithi167 conmon[60316]: debug 2022-01-31T21:11:03.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:03.257584+0000) 2022-01-31T21:11:03.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:03 smithi171 conmon[51620]: debug 2022-01-31T21:11:03.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:03.613044+0000) 2022-01-31T21:11:03.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:03 smithi167 conmon[49112]: debug 2022-01-31T21:11:03.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:03.580428+0000) 2022-01-31T21:11:03.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:03 smithi167 conmon[54076]: debug 2022-01-31T21:11:03.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:03.506043+0000) 2022-01-31T21:11:04.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:03 smithi171 conmon[41853]: debug 2022-01-31T21:11:03.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:03.738399+0000) 2022-01-31T21:11:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:04 smithi171 conmon[46715]: debug 2022-01-31T21:11:04.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:04.033256+0000) 2022-01-31T21:11:04.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:04 smithi167 conmon[60316]: debug 2022-01-31T21:11:04.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:04.257761+0000) 2022-01-31T21:11:04.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:04 smithi171 conmon[51620]: debug 2022-01-31T21:11:04.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:04.613244+0000) 2022-01-31T21:11:04.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:04 smithi167 conmon[49112]: debug 2022-01-31T21:11:04.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:04.580600+0000) 2022-01-31T21:11:04.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:04 smithi167 conmon[54076]: debug 2022-01-31T21:11:04.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:04.506199+0000) 2022-01-31T21:11:05.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:04 smithi171 conmon[41853]: debug 2022-01-31T21:11:04.737+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:04.738569+0000) 2022-01-31T21:11:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:05 smithi171 conmon[46715]: debug 2022-01-31T21:11:05.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:05.033465+0000) 2022-01-31T21:11:05.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:05 smithi167 conmon[60316]: debug 2022-01-31T21:11:05.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:05.257923+0000) 2022-01-31T21:11:05.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:05 smithi171 conmon[51620]: debug 2022-01-31T21:11:05.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:05.613393+0000) 2022-01-31T21:11:05.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:05 smithi167 conmon[49112]: debug 2022-01-31T21:11:05.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:05.580737+0000) 2022-01-31T21:11:05.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:05 smithi167 conmon[54076]: debug 2022-01-31T21:11:05.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:05.506382+0000) 2022-01-31T21:11:06.032 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:05 smithi171 conmon[41853]: debug 2022-01-31T21:11:05.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:05.738728+0000) 2022-01-31T21:11:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[46715]: debug 2022-01-31T21:11:06.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.033673+0000) 2022-01-31T21:11:06.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:06 smithi167 conmon[60316]: debug 2022-01-31T21:11:06.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.258129+0000) 2022-01-31T21:11:06.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[51620]: debug 2022-01-31T21:11:06.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.613597+0000) 2022-01-31T21:11:06.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:06 smithi167 conmon[49112]: debug 2022-01-31T21:11:06.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.580896+0000) 2022-01-31T21:11:06.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:06 smithi167 conmon[54076]: debug 2022-01-31T21:11:06.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.506582+0000) 2022-01-31T21:11:07.032 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[35325]: debug 2022-01-31T21:11:06.946+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109672 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:07.033 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[41853]: debug 2022-01-31T21:11:06.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.738848+0000) 2022-01-31T21:11:07.033 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[41853]: debug 2022-01-31T21:11:06.936+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.937098+0000) 2022-01-31T21:11:07.033 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[46715]: debug 2022-01-31T21:11:06.935+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.936500+0000) 2022-01-31T21:11:07.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:06 smithi171 conmon[51620]: debug 2022-01-31T21:11:06.936+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.936744+0000) 2022-01-31T21:11:07.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:06 smithi167 conmon[49112]: debug 2022-01-31T21:11:06.935+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.937271+0000) 2022-01-31T21:11:07.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:06 smithi167 conmon[54076]: debug 2022-01-31T21:11:06.936+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.937968+0000) 2022-01-31T21:11:07.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:06 smithi167 conmon[60316]: debug 2022-01-31T21:11:06.936+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:06.937810+0000) 2022-01-31T21:11:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:07 smithi171 conmon[46715]: debug 2022-01-31T21:11:07.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:07.033880+0000) 2022-01-31T21:11:07.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:07 smithi167 conmon[60316]: debug 2022-01-31T21:11:07.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:07.258282+0000) 2022-01-31T21:11:07.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:07 smithi171 conmon[51620]: debug 2022-01-31T21:11:07.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:07.613780+0000) 2022-01-31T21:11:07.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:07 smithi167 conmon[49112]: debug 2022-01-31T21:11:07.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:07.581124+0000) 2022-01-31T21:11:07.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:07 smithi167 conmon[54076]: debug 2022-01-31T21:11:07.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:07.506742+0000) 2022-01-31T21:11:08.032 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:07 smithi171 conmon[41853]: debug 2022-01-31T21:11:07.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:07.739002+0000) 2022-01-31T21:11:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:08 smithi171 conmon[46715]: debug 2022-01-31T21:11:08.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:08.034037+0000) 2022-01-31T21:11:08.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:08 smithi167 conmon[60316]: debug 2022-01-31T21:11:08.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:08.258429+0000) 2022-01-31T21:11:08.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:08 smithi171 conmon[51620]: debug 2022-01-31T21:11:08.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:08.613933+0000) 2022-01-31T21:11:08.738 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:08 smithi171 conmon[35325]: debug 2022-01-31T21:11:08.637+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:11:08.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:08 smithi167 conmon[49112]: debug 2022-01-31T21:11:08.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:08.581276+0000) 2022-01-31T21:11:08.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:08 smithi167 conmon[54076]: debug 2022-01-31T21:11:08.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:08.506953+0000) 2022-01-31T21:11:09.032 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:08 smithi171 conmon[41853]: debug 2022-01-31T21:11:08.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:08.739107+0000) 2022-01-31T21:11:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:09 smithi171 conmon[46715]: debug 2022-01-31T21:11:09.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:09.034210+0000) 2022-01-31T21:11:09.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:09 smithi167 conmon[60316]: debug 2022-01-31T21:11:09.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:09.258591+0000) 2022-01-31T21:11:09.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:09 smithi171 conmon[51620]: debug 2022-01-31T21:11:09.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:09.614113+0000) 2022-01-31T21:11:09.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:09 smithi167 conmon[54076]: debug 2022-01-31T21:11:09.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:09.507164+0000) 2022-01-31T21:11:09.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:09 smithi167 conmon[49112]: debug 2022-01-31T21:11:09.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:09.581458+0000) 2022-01-31T21:11:10.032 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:09 smithi171 conmon[41853]: debug 2022-01-31T21:11:09.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:09.739282+0000) 2022-01-31T21:11:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:10 smithi171 conmon[46715]: debug 2022-01-31T21:11:10.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:10.034360+0000) 2022-01-31T21:11:10.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:10 smithi167 conmon[60316]: debug 2022-01-31T21:11:10.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:10.258739+0000) 2022-01-31T21:11:10.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:10 smithi171 conmon[51620]: debug 2022-01-31T21:11:10.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:10.614260+0000) 2022-01-31T21:11:10.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:10 smithi167 conmon[49112]: debug 2022-01-31T21:11:10.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:10.581627+0000) 2022-01-31T21:11:10.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:10 smithi167 conmon[54076]: debug 2022-01-31T21:11:10.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:10.507360+0000) 2022-01-31T21:11:11.032 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:10 smithi171 conmon[41853]: debug 2022-01-31T21:11:10.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:10.739484+0000) 2022-01-31T21:11:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[46715]: debug 2022-01-31T21:11:11.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.034513+0000) 2022-01-31T21:11:11.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:11 smithi167 conmon[60316]: debug 2022-01-31T21:11:11.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.258872+0000) 2022-01-31T21:11:11.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[51620]: debug 2022-01-31T21:11:11.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.614473+0000) 2022-01-31T21:11:11.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:11 smithi167 conmon[49112]: debug 2022-01-31T21:11:11.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.581780+0000) 2022-01-31T21:11:11.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:11 smithi167 conmon[54076]: debug 2022-01-31T21:11:11.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.507538+0000) 2022-01-31T21:11:12.033 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[35325]: debug 2022-01-31T21:11:11.959+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109783 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:12.033 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[46715]: debug 2022-01-31T21:11:11.948+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.950150+0000) 2022-01-31T21:11:12.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[51620]: debug 2022-01-31T21:11:11.948+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.950565+0000) 2022-01-31T21:11:12.034 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[41853]: debug 2022-01-31T21:11:11.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.739682+0000) 2022-01-31T21:11:12.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:11 smithi171 conmon[41853]: debug 2022-01-31T21:11:11.949+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.950841+0000) 2022-01-31T21:11:12.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:11 smithi167 conmon[49112]: debug 2022-01-31T21:11:11.948+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.949879+0000) 2022-01-31T21:11:12.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:11 smithi167 conmon[54076]: debug 2022-01-31T21:11:11.950+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.951512+0000) 2022-01-31T21:11:12.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:11 smithi167 conmon[60316]: debug 2022-01-31T21:11:11.949+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:11.950617+0000) 2022-01-31T21:11:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:12 smithi171 conmon[46715]: debug 2022-01-31T21:11:12.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:12.034709+0000) 2022-01-31T21:11:12.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:12 smithi167 conmon[60316]: debug 2022-01-31T21:11:12.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:12.259050+0000) 2022-01-31T21:11:12.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:12 smithi171 conmon[51620]: debug 2022-01-31T21:11:12.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:12.614660+0000) 2022-01-31T21:11:12.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:12 smithi167 conmon[49112]: debug 2022-01-31T21:11:12.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:12.581956+0000) 2022-01-31T21:11:12.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:12 smithi167 conmon[54076]: debug 2022-01-31T21:11:12.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:12.507742+0000) 2022-01-31T21:11:13.033 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:12 smithi171 conmon[41853]: debug 2022-01-31T21:11:12.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:12.739818+0000) 2022-01-31T21:11:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:13 smithi171 conmon[46715]: debug 2022-01-31T21:11:13.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:13.034845+0000) 2022-01-31T21:11:13.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:13 smithi167 conmon[60316]: debug 2022-01-31T21:11:13.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:13.259211+0000) 2022-01-31T21:11:13.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:13 smithi171 conmon[51620]: debug 2022-01-31T21:11:13.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:13.614792+0000) 2022-01-31T21:11:13.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:13 smithi167 conmon[49112]: debug 2022-01-31T21:11:13.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:13.582099+0000) 2022-01-31T21:11:13.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:13 smithi167 conmon[54076]: debug 2022-01-31T21:11:13.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:13.507894+0000) 2022-01-31T21:11:14.033 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:13 smithi171 conmon[41853]: debug 2022-01-31T21:11:13.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:13.739966+0000) 2022-01-31T21:11:14.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:14 smithi171 conmon[46715]: debug 2022-01-31T21:11:14.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:14.035012+0000) 2022-01-31T21:11:14.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:14 smithi167 conmon[60316]: debug 2022-01-31T21:11:14.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:14.259395+0000) 2022-01-31T21:11:14.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:14 smithi171 conmon[51620]: debug 2022-01-31T21:11:14.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:14.615000+0000) 2022-01-31T21:11:14.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:14 smithi167 conmon[49112]: debug 2022-01-31T21:11:14.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:14.582259+0000) 2022-01-31T21:11:14.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:14 smithi167 conmon[54076]: debug 2022-01-31T21:11:14.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:14.508058+0000) 2022-01-31T21:11:15.033 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:14 smithi171 conmon[41853]: debug 2022-01-31T21:11:14.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:14.740132+0000) 2022-01-31T21:11:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:15 smithi171 conmon[46715]: debug 2022-01-31T21:11:15.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:15.035194+0000) 2022-01-31T21:11:15.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:15 smithi167 conmon[60316]: debug 2022-01-31T21:11:15.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:15.259556+0000) 2022-01-31T21:11:15.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:15 smithi171 conmon[51620]: debug 2022-01-31T21:11:15.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:15.615189+0000) 2022-01-31T21:11:15.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:15 smithi167 conmon[49112]: debug 2022-01-31T21:11:15.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:15.582404+0000) 2022-01-31T21:11:15.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:15 smithi167 conmon[54076]: debug 2022-01-31T21:11:15.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:15.508215+0000) 2022-01-31T21:11:16.033 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:15 smithi171 conmon[41853]: debug 2022-01-31T21:11:15.738+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:15.740260+0000) 2022-01-31T21:11:16.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[46715]: debug 2022-01-31T21:11:16.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.035397+0000) 2022-01-31T21:11:16.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:16 smithi167 conmon[60316]: debug 2022-01-31T21:11:16.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.259712+0000) 2022-01-31T21:11:16.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[51620]: debug 2022-01-31T21:11:16.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.615379+0000) 2022-01-31T21:11:16.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:16 smithi167 conmon[54076]: debug 2022-01-31T21:11:16.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.508403+0000) 2022-01-31T21:11:16.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:16 smithi167 conmon[49112]: debug 2022-01-31T21:11:16.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.582557+0000) 2022-01-31T21:11:17.034 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[35325]: debug 2022-01-31T21:11:16.973+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 109892 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:17.035 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[46715]: debug 2022-01-31T21:11:16.962+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.964277+0000) 2022-01-31T21:11:17.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[41853]: debug 2022-01-31T21:11:16.739+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.740458+0000) 2022-01-31T21:11:17.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[41853]: debug 2022-01-31T21:11:16.963+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.965000+0000) 2022-01-31T21:11:17.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:16 smithi171 conmon[51620]: debug 2022-01-31T21:11:16.962+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.964636+0000) 2022-01-31T21:11:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:16 smithi167 conmon[49112]: debug 2022-01-31T21:11:16.963+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.964424+0000) 2022-01-31T21:11:17.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:16 smithi167 conmon[54076]: debug 2022-01-31T21:11:16.964+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.964624+0000) 2022-01-31T21:11:17.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:16 smithi167 conmon[60316]: debug 2022-01-31T21:11:16.964+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:16.964591+0000) 2022-01-31T21:11:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:17 smithi171 conmon[46715]: debug 2022-01-31T21:11:17.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:17.035599+0000) 2022-01-31T21:11:17.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:17 smithi167 conmon[60316]: debug 2022-01-31T21:11:17.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:17.259932+0000) 2022-01-31T21:11:17.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:17 smithi171 conmon[51620]: debug 2022-01-31T21:11:17.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:17.615582+0000) 2022-01-31T21:11:17.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:17 smithi167 conmon[54076]: debug 2022-01-31T21:11:17.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:17.508566+0000) 2022-01-31T21:11:17.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:17 smithi167 conmon[49112]: debug 2022-01-31T21:11:17.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:17.582677+0000) 2022-01-31T21:11:18.034 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:17 smithi171 conmon[41853]: debug 2022-01-31T21:11:17.739+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:17.740662+0000) 2022-01-31T21:11:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:18 smithi171 conmon[46715]: debug 2022-01-31T21:11:18.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:18.035739+0000) 2022-01-31T21:11:18.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:18 smithi167 conmon[60316]: debug 2022-01-31T21:11:18.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:18.260012+0000) 2022-01-31T21:11:18.697 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:18 smithi171 conmon[51620]: debug 2022-01-31T21:11:18.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:18.615774+0000) 2022-01-31T21:11:18.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:18 smithi167 conmon[49112]: debug 2022-01-31T21:11:18.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:18.582830+0000) 2022-01-31T21:11:18.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:18 smithi167 conmon[54076]: debug 2022-01-31T21:11:18.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:18.508741+0000) 2022-01-31T21:11:19.034 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:18 smithi171 conmon[41853]: debug 2022-01-31T21:11:18.740+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:18.740842+0000) 2022-01-31T21:11:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:19 smithi171 conmon[46715]: debug 2022-01-31T21:11:19.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:19.036006+0000) 2022-01-31T21:11:19.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:19 smithi167 conmon[60316]: debug 2022-01-31T21:11:19.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:19.260187+0000) 2022-01-31T21:11:19.698 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:19 smithi171 conmon[51620]: debug 2022-01-31T21:11:19.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:19.615976+0000) 2022-01-31T21:11:19.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:19 smithi167 conmon[49112]: debug 2022-01-31T21:11:19.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:19.583019+0000) 2022-01-31T21:11:19.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:19 smithi167 conmon[54076]: debug 2022-01-31T21:11:19.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:19.508923+0000) 2022-01-31T21:11:20.034 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:19 smithi171 conmon[41853]: debug 2022-01-31T21:11:19.740+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:19.740983+0000) 2022-01-31T21:11:20.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:20 smithi171 conmon[46715]: debug 2022-01-31T21:11:20.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:20.036190+0000) 2022-01-31T21:11:20.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:20 smithi167 conmon[60316]: debug 2022-01-31T21:11:20.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:20.260380+0000) 2022-01-31T21:11:20.698 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:20 smithi171 conmon[51620]: debug 2022-01-31T21:11:20.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:20.616124+0000) 2022-01-31T21:11:20.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:20 smithi167 conmon[49112]: debug 2022-01-31T21:11:20.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:20.583255+0000) 2022-01-31T21:11:20.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:20 smithi167 conmon[54076]: debug 2022-01-31T21:11:20.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:20.509108+0000) 2022-01-31T21:11:21.034 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:20 smithi171 conmon[41853]: debug 2022-01-31T21:11:20.740+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:20.741183+0000) 2022-01-31T21:11:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[46715]: debug 2022-01-31T21:11:21.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.036390+0000) 2022-01-31T21:11:21.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:21 smithi167 conmon[60316]: debug 2022-01-31T21:11:21.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.260598+0000) 2022-01-31T21:11:21.698 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[51620]: debug 2022-01-31T21:11:21.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.616367+0000) 2022-01-31T21:11:21.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:21 smithi167 conmon[49112]: debug 2022-01-31T21:11:21.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.583521+0000) 2022-01-31T21:11:21.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:21 smithi167 conmon[54076]: debug 2022-01-31T21:11:21.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.509353+0000) 2022-01-31T21:11:21.975 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[41853]: debug 2022-01-31T21:11:21.740+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.741408+0000) 2022-01-31T21:11:22.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:21 smithi167 conmon[49112]: debug 2022-01-31T21:11:21.976+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.977237+0000) 2022-01-31T21:11:22.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:21 smithi167 conmon[54076]: debug 2022-01-31T21:11:21.977+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.978316+0000) 2022-01-31T21:11:22.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:21 smithi167 conmon[60316]: debug 2022-01-31T21:11:21.978+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.978632+0000) 2022-01-31T21:11:22.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[35325]: debug 2022-01-31T21:11:21.988+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110003 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:22.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[41853]: debug 2022-01-31T21:11:21.978+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.979116+0000) 2022-01-31T21:11:22.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[51620]: debug 2022-01-31T21:11:21.977+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.978531+0000) 2022-01-31T21:11:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:21 smithi171 conmon[46715]: debug 2022-01-31T21:11:21.977+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:21.978245+0000) 2022-01-31T21:11:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:22 smithi171 conmon[46715]: debug 2022-01-31T21:11:22.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:22.036616+0000) 2022-01-31T21:11:22.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:22 smithi167 conmon[60316]: debug 2022-01-31T21:11:22.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:22.260786+0000) 2022-01-31T21:11:22.698 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:22 smithi171 conmon[51620]: debug 2022-01-31T21:11:22.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:22.616552+0000) 2022-01-31T21:11:22.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:22 smithi167 conmon[49112]: debug 2022-01-31T21:11:22.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:22.583710+0000) 2022-01-31T21:11:22.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:22 smithi167 conmon[54076]: debug 2022-01-31T21:11:22.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:22.509548+0000) 2022-01-31T21:11:23.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:22 smithi171 conmon[41853]: debug 2022-01-31T21:11:22.740+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:22.741602+0000) 2022-01-31T21:11:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:23 smithi171 conmon[46715]: debug 2022-01-31T21:11:23.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:23.036718+0000) 2022-01-31T21:11:23.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:23 smithi167 conmon[60316]: debug 2022-01-31T21:11:23.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:23.260882+0000) 2022-01-31T21:11:23.698 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:23 smithi171 conmon[35325]: debug 2022-01-31T21:11:23.637+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:11:23.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:23 smithi171 conmon[51620]: debug 2022-01-31T21:11:23.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:23.616680+0000) 2022-01-31T21:11:23.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:23 smithi167 conmon[54076]: debug 2022-01-31T21:11:23.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:23.509726+0000) 2022-01-31T21:11:23.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:23 smithi167 conmon[49112]: debug 2022-01-31T21:11:23.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:23.583884+0000) 2022-01-31T21:11:24.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:23 smithi171 conmon[41853]: debug 2022-01-31T21:11:23.741+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:23.741788+0000) 2022-01-31T21:11:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:24 smithi171 conmon[46715]: debug 2022-01-31T21:11:24.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:24.036847+0000) 2022-01-31T21:11:24.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:24 smithi167 conmon[60316]: debug 2022-01-31T21:11:24.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:24.261020+0000) 2022-01-31T21:11:24.698 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:24 smithi171 conmon[51620]: debug 2022-01-31T21:11:24.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:24.616842+0000) 2022-01-31T21:11:24.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:24 smithi167 conmon[54076]: debug 2022-01-31T21:11:24.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:24.509910+0000) 2022-01-31T21:11:24.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:24 smithi167 conmon[49112]: debug 2022-01-31T21:11:24.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:24.584034+0000) 2022-01-31T21:11:25.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:24 smithi171 conmon[41853]: debug 2022-01-31T21:11:24.741+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:24.741973+0000) 2022-01-31T21:11:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:25 smithi171 conmon[46715]: debug 2022-01-31T21:11:25.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:25.037039+0000) 2022-01-31T21:11:25.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:25 smithi167 conmon[60316]: debug 2022-01-31T21:11:25.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:25.261208+0000) 2022-01-31T21:11:25.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:25 smithi171 conmon[51620]: debug 2022-01-31T21:11:25.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:25.616998+0000) 2022-01-31T21:11:25.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:25 smithi167 conmon[54076]: debug 2022-01-31T21:11:25.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:25.510094+0000) 2022-01-31T21:11:25.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:25 smithi167 conmon[49112]: debug 2022-01-31T21:11:25.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:25.584238+0000) 2022-01-31T21:11:26.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:25 smithi171 conmon[41853]: debug 2022-01-31T21:11:25.741+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:25.742160+0000) 2022-01-31T21:11:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:26 smithi171 conmon[46715]: debug 2022-01-31T21:11:26.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.037200+0000) 2022-01-31T21:11:26.508 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:26 smithi167 conmon[60316]: debug 2022-01-31T21:11:26.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.261413+0000) 2022-01-31T21:11:26.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:26 smithi171 conmon[51620]: debug 2022-01-31T21:11:26.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.617174+0000) 2022-01-31T21:11:26.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:26 smithi167 conmon[54076]: debug 2022-01-31T21:11:26.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.510220+0000) 2022-01-31T21:11:26.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:26 smithi167 conmon[49112]: debug 2022-01-31T21:11:26.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.584476+0000) 2022-01-31T21:11:26.989 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:26 smithi171 conmon[41853]: debug 2022-01-31T21:11:26.741+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.742319+0000) 2022-01-31T21:11:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:26 smithi167 conmon[49112]: debug 2022-01-31T21:11:26.990+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.991208+0000) 2022-01-31T21:11:27.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:26 smithi167 conmon[60316]: debug 2022-01-31T21:11:26.991+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.991595+0000) 2022-01-31T21:11:27.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:26 smithi167 conmon[54076]: debug 2022-01-31T21:11:26.992+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.992713+0000) 2022-01-31T21:11:27.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:27 smithi171 conmon[35325]: debug 2022-01-31T21:11:27.001+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:27.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:26 smithi171 conmon[41853]: debug 2022-01-31T21:11:26.990+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.991134+0000) 2022-01-31T21:11:27.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:26 smithi171 conmon[46715]: debug 2022-01-31T21:11:26.991+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.992139+0000) 2022-01-31T21:11:27.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:27 smithi171 conmon[46715]: debug 2022-01-31T21:11:27.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:27.037375+0000) 2022-01-31T21:11:27.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:26 smithi171 conmon[51620]: debug 2022-01-31T21:11:26.991+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:26.992425+0000) 2022-01-31T21:11:27.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:27 smithi167 conmon[60316]: debug 2022-01-31T21:11:27.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:27.261622+0000) 2022-01-31T21:11:27.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:27 smithi171 conmon[51620]: debug 2022-01-31T21:11:27.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:27.617382+0000) 2022-01-31T21:11:27.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:27 smithi167 conmon[49112]: debug 2022-01-31T21:11:27.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:27.584636+0000) 2022-01-31T21:11:27.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:27 smithi167 conmon[54076]: debug 2022-01-31T21:11:27.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:27.510402+0000) 2022-01-31T21:11:28.035 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:27 smithi171 conmon[41853]: debug 2022-01-31T21:11:27.741+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:27.742517+0000) 2022-01-31T21:11:28.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:28 smithi171 conmon[46715]: debug 2022-01-31T21:11:28.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:28.037512+0000) 2022-01-31T21:11:28.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:28 smithi167 conmon[60316]: debug 2022-01-31T21:11:28.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:28.261753+0000) 2022-01-31T21:11:28.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:28 smithi171 conmon[51620]: debug 2022-01-31T21:11:28.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:28.617534+0000) 2022-01-31T21:11:28.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:28 smithi167 conmon[54076]: debug 2022-01-31T21:11:28.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:28.510581+0000) 2022-01-31T21:11:28.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:28 smithi167 conmon[49112]: debug 2022-01-31T21:11:28.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:28.584742+0000) 2022-01-31T21:11:29.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:28 smithi171 conmon[41853]: debug 2022-01-31T21:11:28.741+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:28.742646+0000) 2022-01-31T21:11:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:29 smithi171 conmon[46715]: debug 2022-01-31T21:11:29.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:29.037735+0000) 2022-01-31T21:11:29.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:29 smithi167 conmon[60316]: debug 2022-01-31T21:11:29.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:29.261911+0000) 2022-01-31T21:11:29.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:29 smithi171 conmon[51620]: debug 2022-01-31T21:11:29.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:29.617712+0000) 2022-01-31T21:11:29.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:29 smithi167 conmon[49112]: debug 2022-01-31T21:11:29.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:29.584935+0000) 2022-01-31T21:11:29.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:29 smithi167 conmon[54076]: debug 2022-01-31T21:11:29.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:29.510778+0000) 2022-01-31T21:11:30.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:29 smithi171 conmon[41853]: debug 2022-01-31T21:11:29.742+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:29.742811+0000) 2022-01-31T21:11:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:30 smithi171 conmon[46715]: debug 2022-01-31T21:11:30.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:30.037957+0000) 2022-01-31T21:11:30.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:30 smithi167 conmon[60316]: debug 2022-01-31T21:11:30.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:30.262112+0000) 2022-01-31T21:11:30.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:30 smithi171 conmon[51620]: debug 2022-01-31T21:11:30.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:30.617947+0000) 2022-01-31T21:11:30.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:30 smithi167 conmon[49112]: debug 2022-01-31T21:11:30.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:30.585139+0000) 2022-01-31T21:11:30.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:30 smithi167 conmon[54076]: debug 2022-01-31T21:11:30.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:30.510971+0000) 2022-01-31T21:11:31.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:30 smithi171 conmon[41853]: debug 2022-01-31T21:11:30.742+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:30.743018+0000) 2022-01-31T21:11:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:31 smithi171 conmon[46715]: debug 2022-01-31T21:11:31.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:31.038128+0000) 2022-01-31T21:11:31.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:31 smithi167 conmon[60316]: debug 2022-01-31T21:11:31.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:31.262277+0000) 2022-01-31T21:11:31.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:31 smithi171 conmon[51620]: debug 2022-01-31T21:11:31.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:31.618131+0000) 2022-01-31T21:11:31.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:31 smithi167 conmon[54076]: debug 2022-01-31T21:11:31.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:31.511142+0000) 2022-01-31T21:11:31.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:31 smithi167 conmon[49112]: debug 2022-01-31T21:11:31.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:31.585326+0000) 2022-01-31T21:11:32.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:31 smithi171 conmon[41853]: debug 2022-01-31T21:11:31.742+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:31.743188+0000) 2022-01-31T21:11:32.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:32 smithi167 conmon[49112]: debug 2022-01-31T21:11:32.004+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.005227+0000) 2022-01-31T21:11:32.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:32 smithi167 conmon[54076]: debug 2022-01-31T21:11:32.005+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.005814+0000) 2022-01-31T21:11:32.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:32 smithi167 conmon[60316]: debug 2022-01-31T21:11:32.005+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.005540+0000) 2022-01-31T21:11:32.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[41853]: debug 2022-01-31T21:11:32.004+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.004940+0000) 2022-01-31T21:11:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[46715]: debug 2022-01-31T21:11:32.004+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.005730+0000) 2022-01-31T21:11:32.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[46715]: debug 2022-01-31T21:11:32.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.038274+0000) 2022-01-31T21:11:32.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[51620]: debug 2022-01-31T21:11:32.004+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.004816+0000) 2022-01-31T21:11:32.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[35325]: debug 2022-01-31T21:11:32.016+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110222 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:32.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:32 smithi167 conmon[60316]: debug 2022-01-31T21:11:32.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.262397+0000) 2022-01-31T21:11:32.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[51620]: debug 2022-01-31T21:11:32.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.618306+0000) 2022-01-31T21:11:32.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:32 smithi167 conmon[49112]: debug 2022-01-31T21:11:32.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.585485+0000) 2022-01-31T21:11:32.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:32 smithi167 conmon[54076]: debug 2022-01-31T21:11:32.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.511241+0000) 2022-01-31T21:11:33.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:32 smithi171 conmon[41853]: debug 2022-01-31T21:11:32.742+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:32.743431+0000) 2022-01-31T21:11:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:33 smithi171 conmon[46715]: debug 2022-01-31T21:11:33.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:33.038437+0000) 2022-01-31T21:11:33.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:33 smithi167 conmon[60316]: debug 2022-01-31T21:11:33.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:33.262603+0000) 2022-01-31T21:11:33.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:33 smithi171 conmon[51620]: debug 2022-01-31T21:11:33.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:33.618466+0000) 2022-01-31T21:11:33.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:33 smithi167 conmon[49112]: debug 2022-01-31T21:11:33.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:33.585622+0000) 2022-01-31T21:11:33.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:33 smithi167 conmon[54076]: debug 2022-01-31T21:11:33.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:33.511381+0000) 2022-01-31T21:11:34.037 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:33 smithi171 conmon[41853]: debug 2022-01-31T21:11:33.742+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:33.743600+0000) 2022-01-31T21:11:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:34 smithi171 conmon[46715]: debug 2022-01-31T21:11:34.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:34.038637+0000) 2022-01-31T21:11:34.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:34 smithi167 conmon[60316]: debug 2022-01-31T21:11:34.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:34.262780+0000) 2022-01-31T21:11:34.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:34 smithi171 conmon[51620]: debug 2022-01-31T21:11:34.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:34.618653+0000) 2022-01-31T21:11:34.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:34 smithi167 conmon[49112]: debug 2022-01-31T21:11:34.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:34.585826+0000) 2022-01-31T21:11:34.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:34 smithi167 conmon[54076]: debug 2022-01-31T21:11:34.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:34.511542+0000) 2022-01-31T21:11:35.037 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:34 smithi171 conmon[41853]: debug 2022-01-31T21:11:34.743+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:34.743749+0000) 2022-01-31T21:11:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:35 smithi171 conmon[46715]: debug 2022-01-31T21:11:35.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:35.038856+0000) 2022-01-31T21:11:35.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:35 smithi167 conmon[60316]: debug 2022-01-31T21:11:35.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:35.262881+0000) 2022-01-31T21:11:35.700 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:35 smithi171 conmon[51620]: debug 2022-01-31T21:11:35.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:35.618832+0000) 2022-01-31T21:11:35.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:35 smithi167 conmon[49112]: debug 2022-01-31T21:11:35.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:35.586005+0000) 2022-01-31T21:11:35.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:35 smithi167 conmon[54076]: debug 2022-01-31T21:11:35.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:35.511741+0000) 2022-01-31T21:11:36.037 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:35 smithi171 conmon[41853]: debug 2022-01-31T21:11:35.743+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:35.743927+0000) 2022-01-31T21:11:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:36 smithi171 conmon[46715]: debug 2022-01-31T21:11:36.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:36.039039+0000) 2022-01-31T21:11:36.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:36 smithi167 conmon[60316]: debug 2022-01-31T21:11:36.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:36.262999+0000) 2022-01-31T21:11:36.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:36 smithi171 conmon[51620]: debug 2022-01-31T21:11:36.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:36.618982+0000) 2022-01-31T21:11:36.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:36 smithi167 conmon[49112]: debug 2022-01-31T21:11:36.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:36.586213+0000) 2022-01-31T21:11:36.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:36 smithi167 conmon[54076]: debug 2022-01-31T21:11:36.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:36.511947+0000) 2022-01-31T21:11:37.017 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:36 smithi171 conmon[41853]: debug 2022-01-31T21:11:36.743+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:36.744118+0000) 2022-01-31T21:11:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:37 smithi167 conmon[49112]: debug 2022-01-31T21:11:37.019+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.020233+0000) 2022-01-31T21:11:37.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:37 smithi167 conmon[54076]: debug 2022-01-31T21:11:37.020+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.020789+0000) 2022-01-31T21:11:37.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:37 smithi167 conmon[60316]: debug 2022-01-31T21:11:37.018+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.019113+0000) 2022-01-31T21:11:37.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[35325]: debug 2022-01-31T21:11:37.030+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110332 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:37.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[41853]: debug 2022-01-31T21:11:37.019+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.020533+0000) 2022-01-31T21:11:37.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[51620]: debug 2022-01-31T21:11:37.019+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.020342+0000) 2022-01-31T21:11:37.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[46715]: debug 2022-01-31T21:11:37.019+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.019974+0000) 2022-01-31T21:11:37.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[46715]: debug 2022-01-31T21:11:37.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.039172+0000) 2022-01-31T21:11:37.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:37 smithi167 conmon[60316]: debug 2022-01-31T21:11:37.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.263156+0000) 2022-01-31T21:11:37.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[51620]: debug 2022-01-31T21:11:37.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.619168+0000) 2022-01-31T21:11:37.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:37 smithi167 conmon[54076]: debug 2022-01-31T21:11:37.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.512108+0000) 2022-01-31T21:11:37.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:37 smithi167 conmon[49112]: debug 2022-01-31T21:11:37.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.586372+0000) 2022-01-31T21:11:38.037 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:37 smithi171 conmon[41853]: debug 2022-01-31T21:11:37.743+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:37.744272+0000) 2022-01-31T21:11:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:38 smithi171 conmon[46715]: debug 2022-01-31T21:11:38.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:38.039296+0000) 2022-01-31T21:11:38.510 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:38 smithi167 conmon[60316]: debug 2022-01-31T21:11:38.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:38.263314+0000) 2022-01-31T21:11:38.701 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:38 smithi171 conmon[35325]: debug 2022-01-31T21:11:38.638+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:11:38.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:38 smithi171 conmon[51620]: debug 2022-01-31T21:11:38.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:38.619354+0000) 2022-01-31T21:11:38.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:38 smithi167 conmon[49112]: debug 2022-01-31T21:11:38.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:38.586531+0000) 2022-01-31T21:11:38.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:38 smithi167 conmon[54076]: debug 2022-01-31T21:11:38.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:38.512277+0000) 2022-01-31T21:11:39.037 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:38 smithi171 conmon[41853]: debug 2022-01-31T21:11:38.743+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:38.744404+0000) 2022-01-31T21:11:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:39 smithi171 conmon[46715]: debug 2022-01-31T21:11:39.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:39.039475+0000) 2022-01-31T21:11:39.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:39 smithi167 conmon[60316]: debug 2022-01-31T21:11:39.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:39.263505+0000) 2022-01-31T21:11:39.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:39 smithi171 conmon[51620]: debug 2022-01-31T21:11:39.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:39.619503+0000) 2022-01-31T21:11:39.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:39 smithi167 conmon[49112]: debug 2022-01-31T21:11:39.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:39.586713+0000) 2022-01-31T21:11:39.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:39 smithi167 conmon[54076]: debug 2022-01-31T21:11:39.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:39.512445+0000) 2022-01-31T21:11:40.038 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:39 smithi171 conmon[41853]: debug 2022-01-31T21:11:39.743+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:39.744591+0000) 2022-01-31T21:11:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:40 smithi171 conmon[46715]: debug 2022-01-31T21:11:40.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:40.039676+0000) 2022-01-31T21:11:40.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:40 smithi167 conmon[60316]: debug 2022-01-31T21:11:40.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:40.263683+0000) 2022-01-31T21:11:40.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:40 smithi171 conmon[51620]: debug 2022-01-31T21:11:40.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:40.619687+0000) 2022-01-31T21:11:40.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:40 smithi167 conmon[49112]: debug 2022-01-31T21:11:40.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:40.586867+0000) 2022-01-31T21:11:40.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:40 smithi167 conmon[54076]: debug 2022-01-31T21:11:40.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:40.512684+0000) 2022-01-31T21:11:41.038 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:40 smithi171 conmon[41853]: debug 2022-01-31T21:11:40.744+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:40.744793+0000) 2022-01-31T21:11:41.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:41 smithi171 conmon[46715]: debug 2022-01-31T21:11:41.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:41.039839+0000) 2022-01-31T21:11:41.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:41 smithi167 conmon[60316]: debug 2022-01-31T21:11:41.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:41.263841+0000) 2022-01-31T21:11:41.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:41 smithi171 conmon[51620]: debug 2022-01-31T21:11:41.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:41.619872+0000) 2022-01-31T21:11:41.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:41 smithi167 conmon[49112]: debug 2022-01-31T21:11:41.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:41.587049+0000) 2022-01-31T21:11:41.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:41 smithi167 conmon[54076]: debug 2022-01-31T21:11:41.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:41.512902+0000) 2022-01-31T21:11:42.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:41 smithi171 conmon[41853]: debug 2022-01-31T21:11:41.744+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:41.744993+0000) 2022-01-31T21:11:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:42 smithi167 conmon[49112]: debug 2022-01-31T21:11:42.031+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.033412+0000) 2022-01-31T21:11:42.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:42 smithi167 conmon[54076]: debug 2022-01-31T21:11:42.032+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.033882+0000) 2022-01-31T21:11:42.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:42 smithi167 conmon[60316]: debug 2022-01-31T21:11:42.032+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.033636+0000) 2022-01-31T21:11:42.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[35325]: debug 2022-01-31T21:11:42.043+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110442 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:42.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[41853]: debug 2022-01-31T21:11:42.033+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.034106+0000) 2022-01-31T21:11:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[46715]: debug 2022-01-31T21:11:42.033+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.034135+0000) 2022-01-31T21:11:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[46715]: debug 2022-01-31T21:11:42.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.039959+0000) 2022-01-31T21:11:42.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[51620]: debug 2022-01-31T21:11:42.032+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.033569+0000) 2022-01-31T21:11:42.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:42 smithi167 conmon[60316]: debug 2022-01-31T21:11:42.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.263996+0000) 2022-01-31T21:11:42.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[51620]: debug 2022-01-31T21:11:42.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.620086+0000) 2022-01-31T21:11:42.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:42 smithi167 conmon[49112]: debug 2022-01-31T21:11:42.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.587235+0000) 2022-01-31T21:11:42.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:42 smithi167 conmon[54076]: debug 2022-01-31T21:11:42.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.513089+0000) 2022-01-31T21:11:43.038 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:42 smithi171 conmon[41853]: debug 2022-01-31T21:11:42.744+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:42.745194+0000) 2022-01-31T21:11:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:43 smithi171 conmon[46715]: debug 2022-01-31T21:11:43.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:43.040089+0000) 2022-01-31T21:11:43.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:43 smithi167 conmon[60316]: debug 2022-01-31T21:11:43.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:43.264144+0000) 2022-01-31T21:11:43.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:43 smithi171 conmon[51620]: debug 2022-01-31T21:11:43.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:43.620244+0000) 2022-01-31T21:11:43.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:43 smithi167 conmon[49112]: debug 2022-01-31T21:11:43.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:43.587376+0000) 2022-01-31T21:11:43.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:43 smithi167 conmon[54076]: debug 2022-01-31T21:11:43.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:43.513238+0000) 2022-01-31T21:11:44.038 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:43 smithi171 conmon[41853]: debug 2022-01-31T21:11:43.744+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:43.745308+0000) 2022-01-31T21:11:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:44 smithi171 conmon[46715]: debug 2022-01-31T21:11:44.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:44.040238+0000) 2022-01-31T21:11:44.511 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:44 smithi167 conmon[60316]: debug 2022-01-31T21:11:44.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:44.264345+0000) 2022-01-31T21:11:44.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:44 smithi171 conmon[51620]: debug 2022-01-31T21:11:44.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:44.620446+0000) 2022-01-31T21:11:44.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:44 smithi167 conmon[54076]: debug 2022-01-31T21:11:44.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:44.513429+0000) 2022-01-31T21:11:44.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:44 smithi167 conmon[49112]: debug 2022-01-31T21:11:44.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:44.587575+0000) 2022-01-31T21:11:45.038 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:44 smithi171 conmon[41853]: debug 2022-01-31T21:11:44.744+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:44.745482+0000) 2022-01-31T21:11:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:45 smithi171 conmon[46715]: debug 2022-01-31T21:11:45.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:45.040417+0000) 2022-01-31T21:11:45.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:45 smithi167 conmon[60316]: debug 2022-01-31T21:11:45.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:45.264516+0000) 2022-01-31T21:11:45.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:45 smithi171 conmon[51620]: debug 2022-01-31T21:11:45.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:45.620583+0000) 2022-01-31T21:11:45.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:45 smithi167 conmon[49112]: debug 2022-01-31T21:11:45.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:45.587701+0000) 2022-01-31T21:11:45.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:45 smithi167 conmon[54076]: debug 2022-01-31T21:11:45.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:45.513654+0000) 2022-01-31T21:11:46.038 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:45 smithi171 conmon[41853]: debug 2022-01-31T21:11:45.744+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:45.745690+0000) 2022-01-31T21:11:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:46 smithi171 conmon[46715]: debug 2022-01-31T21:11:46.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:46.040573+0000) 2022-01-31T21:11:46.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:46 smithi167 conmon[60316]: debug 2022-01-31T21:11:46.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:46.264686+0000) 2022-01-31T21:11:46.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:46 smithi171 conmon[51620]: debug 2022-01-31T21:11:46.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:46.620738+0000) 2022-01-31T21:11:46.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:46 smithi167 conmon[49112]: debug 2022-01-31T21:11:46.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:46.587860+0000) 2022-01-31T21:11:46.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:46 smithi167 conmon[54076]: debug 2022-01-31T21:11:46.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:46.513844+0000) 2022-01-31T21:11:47.039 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:46 smithi171 conmon[41853]: debug 2022-01-31T21:11:46.745+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:46.745873+0000) 2022-01-31T21:11:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:47 smithi167 conmon[49112]: debug 2022-01-31T21:11:47.045+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.046538+0000) 2022-01-31T21:11:47.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:47 smithi167 conmon[54076]: debug 2022-01-31T21:11:47.053+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.054636+0000) 2022-01-31T21:11:47.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:47 smithi167 conmon[60316]: debug 2022-01-31T21:11:47.051+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.053470+0000) 2022-01-31T21:11:47.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[35325]: debug 2022-01-31T21:11:47.060+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110553 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:47.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[41853]: debug 2022-01-31T21:11:47.044+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.046800+0000) 2022-01-31T21:11:47.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[51620]: debug 2022-01-31T21:11:47.044+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.047456+0000) 2022-01-31T21:11:47.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[46715]: debug 2022-01-31T21:11:47.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.040790+0000) 2022-01-31T21:11:47.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[46715]: debug 2022-01-31T21:11:47.044+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.047428+0000) 2022-01-31T21:11:47.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:47 smithi167 conmon[60316]: debug 2022-01-31T21:11:47.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.264794+0000) 2022-01-31T21:11:47.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[51620]: debug 2022-01-31T21:11:47.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.620970+0000) 2022-01-31T21:11:47.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:47 smithi167 conmon[49112]: debug 2022-01-31T21:11:47.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.588005+0000) 2022-01-31T21:11:47.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:47 smithi167 conmon[54076]: debug 2022-01-31T21:11:47.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.514032+0000) 2022-01-31T21:11:48.039 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:47 smithi171 conmon[41853]: debug 2022-01-31T21:11:47.745+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:47.746071+0000) 2022-01-31T21:11:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:48 smithi171 conmon[46715]: debug 2022-01-31T21:11:48.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:48.040996+0000) 2022-01-31T21:11:48.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:48 smithi167 conmon[60316]: debug 2022-01-31T21:11:48.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:48.264900+0000) 2022-01-31T21:11:48.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:48 smithi171 conmon[51620]: debug 2022-01-31T21:11:48.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:48.621089+0000) 2022-01-31T21:11:48.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:48 smithi167 conmon[49112]: debug 2022-01-31T21:11:48.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:48.588141+0000) 2022-01-31T21:11:48.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:48 smithi167 conmon[54076]: debug 2022-01-31T21:11:48.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:48.514231+0000) 2022-01-31T21:11:49.039 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:48 smithi171 conmon[41853]: debug 2022-01-31T21:11:48.745+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:48.746191+0000) 2022-01-31T21:11:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:49 smithi171 conmon[46715]: debug 2022-01-31T21:11:49.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:49.041183+0000) 2022-01-31T21:11:49.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:49 smithi167 conmon[60316]: debug 2022-01-31T21:11:49.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:49.265084+0000) 2022-01-31T21:11:49.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:49 smithi171 conmon[51620]: debug 2022-01-31T21:11:49.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:49.621244+0000) 2022-01-31T21:11:49.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:49 smithi167 conmon[49112]: debug 2022-01-31T21:11:49.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:49.588323+0000) 2022-01-31T21:11:49.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:49 smithi167 conmon[54076]: debug 2022-01-31T21:11:49.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:49.514382+0000) 2022-01-31T21:11:50.039 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:49 smithi171 conmon[41853]: debug 2022-01-31T21:11:49.745+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:49.746351+0000) 2022-01-31T21:11:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:50 smithi171 conmon[46715]: debug 2022-01-31T21:11:50.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:50.041379+0000) 2022-01-31T21:11:50.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:50 smithi167 conmon[60316]: debug 2022-01-31T21:11:50.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:50.265246+0000) 2022-01-31T21:11:50.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:50 smithi171 conmon[51620]: debug 2022-01-31T21:11:50.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:50.621445+0000) 2022-01-31T21:11:50.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:50 smithi167 conmon[49112]: debug 2022-01-31T21:11:50.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:50.588522+0000) 2022-01-31T21:11:50.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:50 smithi167 conmon[54076]: debug 2022-01-31T21:11:50.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:50.514597+0000) 2022-01-31T21:11:51.039 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:50 smithi171 conmon[41853]: debug 2022-01-31T21:11:50.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:50.746533+0000) 2022-01-31T21:11:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:51 smithi171 conmon[46715]: debug 2022-01-31T21:11:51.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:51.041535+0000) 2022-01-31T21:11:51.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:51 smithi167 conmon[60316]: debug 2022-01-31T21:11:51.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:51.265404+0000) 2022-01-31T21:11:51.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:51 smithi171 conmon[51620]: debug 2022-01-31T21:11:51.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:51.621633+0000) 2022-01-31T21:11:51.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:51 smithi167 conmon[49112]: debug 2022-01-31T21:11:51.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:51.588674+0000) 2022-01-31T21:11:51.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:51 smithi167 conmon[54076]: debug 2022-01-31T21:11:51.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:51.514784+0000) 2022-01-31T21:11:52.040 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:51 smithi171 conmon[41853]: debug 2022-01-31T21:11:51.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:51.746749+0000) 2022-01-31T21:11:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:52 smithi167 conmon[49112]: debug 2022-01-31T21:11:52.062+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.063554+0000) 2022-01-31T21:11:52.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:52 smithi167 conmon[54076]: debug 2022-01-31T21:11:52.062+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.064375+0000) 2022-01-31T21:11:52.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:52 smithi167 conmon[60316]: debug 2022-01-31T21:11:52.061+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.063355+0000) 2022-01-31T21:11:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[41853]: debug 2022-01-31T21:11:52.061+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.062681+0000) 2022-01-31T21:11:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[46715]: debug 2022-01-31T21:11:52.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.041686+0000) 2022-01-31T21:11:52.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[46715]: debug 2022-01-31T21:11:52.063+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.064145+0000) 2022-01-31T21:11:52.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[35325]: debug 2022-01-31T21:11:52.073+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110664 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:52.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[51620]: debug 2022-01-31T21:11:52.062+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.063458+0000) 2022-01-31T21:11:52.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:52 smithi167 conmon[60316]: debug 2022-01-31T21:11:52.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.265593+0000) 2022-01-31T21:11:52.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[51620]: debug 2022-01-31T21:11:52.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.621813+0000) 2022-01-31T21:11:52.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:52 smithi167 conmon[49112]: debug 2022-01-31T21:11:52.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.588853+0000) 2022-01-31T21:11:52.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:52 smithi167 conmon[54076]: debug 2022-01-31T21:11:52.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.514912+0000) 2022-01-31T21:11:53.040 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:52 smithi171 conmon[41853]: debug 2022-01-31T21:11:52.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:52.746878+0000) 2022-01-31T21:11:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:53 smithi171 conmon[46715]: debug 2022-01-31T21:11:53.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:53.041889+0000) 2022-01-31T21:11:53.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:53 smithi167 conmon[60316]: debug 2022-01-31T21:11:53.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:53.265766+0000) 2022-01-31T21:11:53.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:53 smithi171 conmon[51620]: debug 2022-01-31T21:11:53.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:53.621975+0000) 2022-01-31T21:11:53.704 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:53 smithi171 conmon[35325]: debug 2022-01-31T21:11:53.639+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:11:53.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:53 smithi167 conmon[49112]: debug 2022-01-31T21:11:53.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:53.589029+0000) 2022-01-31T21:11:53.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:53 smithi167 conmon[54076]: debug 2022-01-31T21:11:53.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:53.515059+0000) 2022-01-31T21:11:54.040 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:53 smithi171 conmon[41853]: debug 2022-01-31T21:11:53.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:53.747018+0000) 2022-01-31T21:11:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:54 smithi171 conmon[46715]: debug 2022-01-31T21:11:54.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:54.042095+0000) 2022-01-31T21:11:54.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:54 smithi167 conmon[60316]: debug 2022-01-31T21:11:54.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:54.265915+0000) 2022-01-31T21:11:54.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:54 smithi171 conmon[51620]: debug 2022-01-31T21:11:54.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:54.622126+0000) 2022-01-31T21:11:54.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:54 smithi167 conmon[49112]: debug 2022-01-31T21:11:54.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:54.589246+0000) 2022-01-31T21:11:54.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:54 smithi167 conmon[54076]: debug 2022-01-31T21:11:54.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:54.515222+0000) 2022-01-31T21:11:55.040 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:54 smithi171 conmon[41853]: debug 2022-01-31T21:11:54.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:54.747194+0000) 2022-01-31T21:11:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:55 smithi171 conmon[46715]: debug 2022-01-31T21:11:55.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:55.042315+0000) 2022-01-31T21:11:55.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:55 smithi167 conmon[60316]: debug 2022-01-31T21:11:55.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:55.266113+0000) 2022-01-31T21:11:55.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:55 smithi171 conmon[51620]: debug 2022-01-31T21:11:55.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:55.622298+0000) 2022-01-31T21:11:55.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:55 smithi167 conmon[49112]: debug 2022-01-31T21:11:55.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:55.589443+0000) 2022-01-31T21:11:55.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:55 smithi167 conmon[54076]: debug 2022-01-31T21:11:55.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:55.515414+0000) 2022-01-31T21:11:56.040 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:55 smithi171 conmon[41853]: debug 2022-01-31T21:11:55.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:55.747375+0000) 2022-01-31T21:11:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:56 smithi171 conmon[46715]: debug 2022-01-31T21:11:56.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:56.042487+0000) 2022-01-31T21:11:56.513 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:56 smithi167 conmon[60316]: debug 2022-01-31T21:11:56.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:56.266273+0000) 2022-01-31T21:11:56.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:56 smithi171 conmon[51620]: debug 2022-01-31T21:11:56.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:56.622467+0000) 2022-01-31T21:11:56.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:56 smithi167 conmon[49112]: debug 2022-01-31T21:11:56.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:56.589612+0000) 2022-01-31T21:11:56.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:56 smithi167 conmon[54076]: debug 2022-01-31T21:11:56.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:56.515553+0000) 2022-01-31T21:11:57.041 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:56 smithi171 conmon[41853]: debug 2022-01-31T21:11:56.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:56.747571+0000) 2022-01-31T21:11:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[49112]: debug 2022-01-31T21:11:57.076+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.078100+0000) 2022-01-31T21:11:57.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[54076]: debug 2022-01-31T21:11:57.076+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.077561+0000) 2022-01-31T21:11:57.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[54076]: 2022-01-31T21:11:57.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[60316]: debug 2022-01-31T21:11:57.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[60316]: 2022-01-31T21:11:57.075+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.077135+0000) 2022-01-31T21:11:57.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[35325]: debug 2022-01-31T21:11:57.086+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110774 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:11:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[41853]: debug 2022-01-31T21:11:57.075+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.076902+0000) 2022-01-31T21:11:57.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[51620]: debug 2022-01-31T21:11:57.074+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.076348+0000) 2022-01-31T21:11:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[46715]: debug 2022-01-31T21:11:57.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.042709+0000) 2022-01-31T21:11:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[46715]: debug 2022-01-31T21:11:57.075+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.077223+0000) 2022-01-31T21:11:57.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[60316]: debug 2022-01-31T21:11:57.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.266433+0000) 2022-01-31T21:11:57.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[60316]: 2022-01-31T21:11:57.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[51620]: debug 2022-01-31T21:11:57.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.622603+0000) 2022-01-31T21:11:57.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[49112]: debug 2022-01-31T21:11:57.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.589752+0000) 2022-01-31T21:11:57.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:57 smithi167 conmon[54076]: debug 2022-01-31T21:11:57.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.515707+0000) 2022-01-31T21:11:58.041 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:57 smithi171 conmon[41853]: debug 2022-01-31T21:11:57.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:57.747766+0000) 2022-01-31T21:11:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:58 smithi171 conmon[46715]: debug 2022-01-31T21:11:58.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:58.042909+0000) 2022-01-31T21:11:58.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:58 smithi167 conmon[60316]: debug 2022-01-31T21:11:58.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:58.266580+0000) 2022-01-31T21:11:58.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:58 smithi171 conmon[51620]: debug 2022-01-31T21:11:58.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:58.622778+0000) 2022-01-31T21:11:58.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:58 smithi167 conmon[49112]: debug 2022-01-31T21:11:58.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:58.589903+0000) 2022-01-31T21:11:58.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:58 smithi167 conmon[54076]: debug 2022-01-31T21:11:58.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:58.515885+0000) 2022-01-31T21:11:59.041 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:58 smithi171 conmon[41853]: debug 2022-01-31T21:11:58.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:58.747888+0000) 2022-01-31T21:11:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:11:59 smithi171 conmon[46715]: debug 2022-01-31T21:11:59.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:59.043085+0000) 2022-01-31T21:11:59.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:11:59 smithi167 conmon[60316]: debug 2022-01-31T21:11:59.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:59.266771+0000) 2022-01-31T21:11:59.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:11:59 smithi171 conmon[51620]: debug 2022-01-31T21:11:59.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:59.622936+0000) 2022-01-31T21:11:59.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:11:59 smithi167 conmon[49112]: debug 2022-01-31T21:11:59.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:59.590112+0000) 2022-01-31T21:11:59.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:11:59 smithi167 conmon[54076]: debug 2022-01-31T21:11:59.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:59.516066+0000) 2022-01-31T21:12:00.041 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:11:59 smithi171 conmon[41853]: debug 2022-01-31T21:11:59.746+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:11:59.748056+0000) 2022-01-31T21:12:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:00 smithi171 conmon[46715]: debug 2022-01-31T21:12:00.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:00.043270+0000) 2022-01-31T21:12:00.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:00 smithi167 conmon[60316]: debug 2022-01-31T21:12:00.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:00.266983+0000) 2022-01-31T21:12:00.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:00 smithi171 conmon[51620]: debug 2022-01-31T21:12:00.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:00.623107+0000) 2022-01-31T21:12:00.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:00 smithi167 conmon[54076]: debug 2022-01-31T21:12:00.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:00.516257+0000) 2022-01-31T21:12:00.836 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:00 smithi167 conmon[49112]: debug 2022-01-31T21:12:00.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:00.590308+0000) 2022-01-31T21:12:01.041 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:00 smithi171 conmon[41853]: debug 2022-01-31T21:12:00.747+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:00.748257+0000) 2022-01-31T21:12:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:01 smithi171 conmon[46715]: debug 2022-01-31T21:12:01.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:01.043474+0000) 2022-01-31T21:12:01.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:01 smithi167 conmon[60316]: debug 2022-01-31T21:12:01.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:01.267139+0000) 2022-01-31T21:12:01.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:01 smithi171 conmon[51620]: debug 2022-01-31T21:12:01.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:01.623309+0000) 2022-01-31T21:12:01.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:01 smithi167 conmon[49112]: debug 2022-01-31T21:12:01.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:01.590510+0000) 2022-01-31T21:12:01.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:01 smithi167 conmon[54076]: debug 2022-01-31T21:12:01.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:01.516442+0000) 2022-01-31T21:12:02.042 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:01 smithi171 conmon[41853]: debug 2022-01-31T21:12:01.747+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:01.748431+0000) 2022-01-31T21:12:02.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[35325]: debug 2022-01-31T21:12:02.100+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110884 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[41853]: debug 2022-01-31T21:12:02.089+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.090868+0000) 2022-01-31T21:12:02.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[51620]: debug 2022-01-31T21:12:02.088+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.090129+0000) 2022-01-31T21:12:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[46715]: debug 2022-01-31T21:12:02.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.043654+0000) 2022-01-31T21:12:02.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[46715]: debug 2022-01-31T21:12:02.088+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.090584+0000) 2022-01-31T21:12:02.373 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:02 smithi167 conmon[49112]: debug 2022-01-31T21:12:02.090+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.091763+0000) 2022-01-31T21:12:02.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:02 smithi167 conmon[54076]: debug 2022-01-31T21:12:02.090+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.091974+0000) 2022-01-31T21:12:02.374 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:02 smithi167 conmon[60316]: debug 2022-01-31T21:12:02.089+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.091481+0000) 2022-01-31T21:12:02.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:02 smithi167 conmon[60316]: debug 2022-01-31T21:12:02.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.267319+0000) 2022-01-31T21:12:02.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:02 smithi167 conmon[54076]: debug 2022-01-31T21:12:02.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.516678+0000) 2022-01-31T21:12:02.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:02 smithi167 conmon[49112]: debug 2022-01-31T21:12:02.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.590651+0000) 2022-01-31T21:12:02.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[51620]: debug 2022-01-31T21:12:02.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.623437+0000) 2022-01-31T21:12:03.042 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:02 smithi171 conmon[41853]: debug 2022-01-31T21:12:02.747+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:02.748622+0000) 2022-01-31T21:12:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:03 smithi171 conmon[46715]: debug 2022-01-31T21:12:03.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:03.043837+0000) 2022-01-31T21:12:03.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:03 smithi167 conmon[60316]: debug 2022-01-31T21:12:03.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:03.267477+0000) 2022-01-31T21:12:03.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:03 smithi167 conmon[49112]: debug 2022-01-31T21:12:03.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:03.590790+0000) 2022-01-31T21:12:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:03 smithi167 conmon[54076]: debug 2022-01-31T21:12:03.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:03.516864+0000) 2022-01-31T21:12:03.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:03 smithi171 conmon[51620]: debug 2022-01-31T21:12:03.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:03.623648+0000) 2022-01-31T21:12:04.042 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:03 smithi171 conmon[41853]: debug 2022-01-31T21:12:03.747+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:03.748803+0000) 2022-01-31T21:12:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:04 smithi171 conmon[46715]: debug 2022-01-31T21:12:04.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:04.043972+0000) 2022-01-31T21:12:04.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:04 smithi167 conmon[60316]: debug 2022-01-31T21:12:04.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:04.267706+0000) 2022-01-31T21:12:04.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:04 smithi167 conmon[49112]: debug 2022-01-31T21:12:04.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:04.590998+0000) 2022-01-31T21:12:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:04 smithi167 conmon[54076]: debug 2022-01-31T21:12:04.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:04.517078+0000) 2022-01-31T21:12:04.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:04 smithi171 conmon[51620]: debug 2022-01-31T21:12:04.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:04.623828+0000) 2022-01-31T21:12:05.043 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:04 smithi171 conmon[41853]: debug 2022-01-31T21:12:04.748+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:04.748947+0000) 2022-01-31T21:12:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:05 smithi171 conmon[46715]: debug 2022-01-31T21:12:05.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:05.044303+0000) 2022-01-31T21:12:05.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:05 smithi167 conmon[60316]: debug 2022-01-31T21:12:05.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:05.267883+0000) 2022-01-31T21:12:05.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:05 smithi167 conmon[49112]: debug 2022-01-31T21:12:05.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:05.591211+0000) 2022-01-31T21:12:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:05 smithi167 conmon[54076]: debug 2022-01-31T21:12:05.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:05.517295+0000) 2022-01-31T21:12:05.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:05 smithi171 conmon[51620]: debug 2022-01-31T21:12:05.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:05.623966+0000) 2022-01-31T21:12:06.042 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:05 smithi171 conmon[41853]: debug 2022-01-31T21:12:05.748+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:05.749129+0000) 2022-01-31T21:12:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:06 smithi171 conmon[46715]: debug 2022-01-31T21:12:06.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:06.044405+0000) 2022-01-31T21:12:06.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:06 smithi167 conmon[60316]: debug 2022-01-31T21:12:06.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:06.268067+0000) 2022-01-31T21:12:06.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:06 smithi167 conmon[49112]: debug 2022-01-31T21:12:06.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:06.591394+0000) 2022-01-31T21:12:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:06 smithi167 conmon[54076]: debug 2022-01-31T21:12:06.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:06.517465+0000) 2022-01-31T21:12:06.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:06 smithi171 conmon[51620]: debug 2022-01-31T21:12:06.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:06.624150+0000) 2022-01-31T21:12:07.043 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:06 smithi171 conmon[41853]: debug 2022-01-31T21:12:06.748+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:06.749310+0000) 2022-01-31T21:12:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[41853]: debug 2022-01-31T21:12:07.104+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.104828+0000) 2022-01-31T21:12:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[46715]: debug 2022-01-31T21:12:07.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.044605+0000) 2022-01-31T21:12:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[46715]: debug 2022-01-31T21:12:07.103+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.104109+0000) 2022-01-31T21:12:07.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[51620]: debug 2022-01-31T21:12:07.103+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.104181+0000) 2022-01-31T21:12:07.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[35325]: debug 2022-01-31T21:12:07.115+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 110994 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:07.380 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:07 smithi167 conmon[49112]: debug 2022-01-31T21:12:07.104+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.105272+0000) 2022-01-31T21:12:07.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:07 smithi167 conmon[54076]: debug 2022-01-31T21:12:07.104+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.105366+0000) 2022-01-31T21:12:07.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:07 smithi167 conmon[60316]: debug 2022-01-31T21:12:07.103+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.104450+0000) 2022-01-31T21:12:07.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:07 smithi167 conmon[60316]: debug 2022-01-31T21:12:07.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.268254+0000) 2022-01-31T21:12:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:07 smithi167 conmon[54076]: debug 2022-01-31T21:12:07.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.517604+0000) 2022-01-31T21:12:07.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:07 smithi167 conmon[49112]: debug 2022-01-31T21:12:07.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.591554+0000) 2022-01-31T21:12:07.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[51620]: debug 2022-01-31T21:12:07.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.624354+0000) 2022-01-31T21:12:08.043 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:07 smithi171 conmon[41853]: debug 2022-01-31T21:12:07.748+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:07.749514+0000) 2022-01-31T21:12:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:08 smithi171 conmon[46715]: debug 2022-01-31T21:12:08.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:08.044798+0000) 2022-01-31T21:12:08.384 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:08 smithi167 conmon[60316]: debug 2022-01-31T21:12:08.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:08.268437+0000) 2022-01-31T21:12:08.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:08 smithi167 conmon[49112]: debug 2022-01-31T21:12:08.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:08.591706+0000) 2022-01-31T21:12:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:08 smithi167 conmon[54076]: debug 2022-01-31T21:12:08.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:08.517730+0000) 2022-01-31T21:12:08.706 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:08 smithi171 conmon[35325]: debug 2022-01-31T21:12:08.639+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:12:08.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:08 smithi171 conmon[51620]: debug 2022-01-31T21:12:08.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:08.624553+0000) 2022-01-31T21:12:09.043 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:08 smithi171 conmon[41853]: debug 2022-01-31T21:12:08.748+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:08.749683+0000) 2022-01-31T21:12:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:09 smithi171 conmon[46715]: debug 2022-01-31T21:12:09.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:09.044955+0000) 2022-01-31T21:12:09.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:09 smithi167 conmon[60316]: debug 2022-01-31T21:12:09.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:09.268641+0000) 2022-01-31T21:12:09.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:09 smithi167 conmon[49112]: debug 2022-01-31T21:12:09.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:09.591912+0000) 2022-01-31T21:12:09.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:09 smithi167 conmon[54076]: debug 2022-01-31T21:12:09.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:09.517854+0000) 2022-01-31T21:12:09.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:09 smithi171 conmon[51620]: debug 2022-01-31T21:12:09.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:09.624736+0000) 2022-01-31T21:12:10.043 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:09 smithi171 conmon[41853]: debug 2022-01-31T21:12:09.749+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:09.749869+0000) 2022-01-31T21:12:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:10 smithi171 conmon[46715]: debug 2022-01-31T21:12:10.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:10.045141+0000) 2022-01-31T21:12:10.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:10 smithi167 conmon[60316]: debug 2022-01-31T21:12:10.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:10.268763+0000) 2022-01-31T21:12:10.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:10 smithi167 conmon[49112]: debug 2022-01-31T21:12:10.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:10.592096+0000) 2022-01-31T21:12:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:10 smithi167 conmon[54076]: debug 2022-01-31T21:12:10.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:10.517972+0000) 2022-01-31T21:12:10.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:10 smithi171 conmon[51620]: debug 2022-01-31T21:12:10.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:10.624893+0000) 2022-01-31T21:12:11.043 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:10 smithi171 conmon[41853]: debug 2022-01-31T21:12:10.749+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:10.750057+0000) 2022-01-31T21:12:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:11 smithi171 conmon[46715]: debug 2022-01-31T21:12:11.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:11.045326+0000) 2022-01-31T21:12:11.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:11 smithi167 conmon[60316]: debug 2022-01-31T21:12:11.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:11.268926+0000) 2022-01-31T21:12:11.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:11 smithi167 conmon[49112]: debug 2022-01-31T21:12:11.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:11.592280+0000) 2022-01-31T21:12:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:11 smithi167 conmon[54076]: debug 2022-01-31T21:12:11.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:11.518143+0000) 2022-01-31T21:12:11.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:11 smithi171 conmon[51620]: debug 2022-01-31T21:12:11.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:11.625032+0000) 2022-01-31T21:12:12.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:11 smithi171 conmon[41853]: debug 2022-01-31T21:12:11.749+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:11.750250+0000) 2022-01-31T21:12:12.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[35325]: debug 2022-01-31T21:12:12.128+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111105 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[46715]: debug 2022-01-31T21:12:12.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.045511+0000) 2022-01-31T21:12:12.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[46715]: debug 2022-01-31T21:12:12.117+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.117767+0000) 2022-01-31T21:12:12.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[41853]: debug 2022-01-31T21:12:12.117+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.117993+0000) 2022-01-31T21:12:12.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[51620]: debug 2022-01-31T21:12:12.117+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.117873+0000) 2022-01-31T21:12:12.387 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:12 smithi167 conmon[49112]: debug 2022-01-31T21:12:12.118+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.118985+0000) 2022-01-31T21:12:12.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:12 smithi167 conmon[54076]: debug 2022-01-31T21:12:12.118+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.119078+0000) 2022-01-31T21:12:12.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:12 smithi167 conmon[60316]: debug 2022-01-31T21:12:12.118+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.118890+0000) 2022-01-31T21:12:12.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:12 smithi167 conmon[60316]: debug 2022-01-31T21:12:12.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.269107+0000) 2022-01-31T21:12:12.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:12 smithi167 conmon[49112]: debug 2022-01-31T21:12:12.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.592423+0000) 2022-01-31T21:12:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:12 smithi167 conmon[54076]: debug 2022-01-31T21:12:12.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.518348+0000) 2022-01-31T21:12:12.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[51620]: debug 2022-01-31T21:12:12.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.625234+0000) 2022-01-31T21:12:13.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:12 smithi171 conmon[41853]: debug 2022-01-31T21:12:12.749+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:12.750432+0000) 2022-01-31T21:12:13.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:13 smithi171 conmon[46715]: debug 2022-01-31T21:12:13.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:13.045636+0000) 2022-01-31T21:12:13.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:13 smithi167 conmon[60316]: debug 2022-01-31T21:12:13.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:13.269254+0000) 2022-01-31T21:12:13.656 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:13 smithi171 conmon[51620]: debug 2022-01-31T21:12:13.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:13.625328+0000) 2022-01-31T21:12:13.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:13 smithi167 conmon[49112]: debug 2022-01-31T21:12:13.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:13.592595+0000) 2022-01-31T21:12:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:13 smithi167 conmon[54076]: debug 2022-01-31T21:12:13.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:13.518462+0000) 2022-01-31T21:12:14.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:13 smithi171 conmon[41853]: debug 2022-01-31T21:12:13.749+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:13.750610+0000) 2022-01-31T21:12:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:14 smithi171 conmon[46715]: debug 2022-01-31T21:12:14.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:14.045720+0000) 2022-01-31T21:12:14.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:14 smithi167 conmon[60316]: debug 2022-01-31T21:12:14.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:14.269462+0000) 2022-01-31T21:12:14.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:14 smithi167 conmon[49112]: debug 2022-01-31T21:12:14.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:14.592795+0000) 2022-01-31T21:12:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:14 smithi167 conmon[54076]: debug 2022-01-31T21:12:14.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:14.518663+0000) 2022-01-31T21:12:14.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:14 smithi171 conmon[51620]: debug 2022-01-31T21:12:14.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:14.625509+0000) 2022-01-31T21:12:15.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:14 smithi171 conmon[41853]: debug 2022-01-31T21:12:14.750+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:14.750797+0000) 2022-01-31T21:12:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:15 smithi171 conmon[46715]: debug 2022-01-31T21:12:15.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:15.045890+0000) 2022-01-31T21:12:15.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:15 smithi167 conmon[60316]: debug 2022-01-31T21:12:15.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:15.269628+0000) 2022-01-31T21:12:15.651 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:15 smithi167 conmon[49112]: debug 2022-01-31T21:12:15.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:15.593001+0000) 2022-01-31T21:12:15.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:15 smithi167 conmon[54076]: debug 2022-01-31T21:12:15.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:15.518856+0000) 2022-01-31T21:12:15.876 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:15 smithi171 conmon[41853]: debug 2022-01-31T21:12:15.750+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:15.750978+0000) 2022-01-31T21:12:15.877 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:15 smithi171 conmon[51620]: debug 2022-01-31T21:12:15.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:15.625673+0000) 2022-01-31T21:12:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:16 smithi171 conmon[46715]: debug 2022-01-31T21:12:16.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:16.046048+0000) 2022-01-31T21:12:16.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:16 smithi167 conmon[60316]: debug 2022-01-31T21:12:16.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:16.269815+0000) 2022-01-31T21:12:16.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:16 smithi167 conmon[49112]: debug 2022-01-31T21:12:16.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:16.593096+0000) 2022-01-31T21:12:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:16 smithi167 conmon[54076]: debug 2022-01-31T21:12:16.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:16.519052+0000) 2022-01-31T21:12:16.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:16 smithi171 conmon[51620]: debug 2022-01-31T21:12:16.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:16.625830+0000) 2022-01-31T21:12:17.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:16 smithi171 conmon[41853]: debug 2022-01-31T21:12:16.750+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:16.751158+0000) 2022-01-31T21:12:17.307 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:17 smithi167 conmon[49112]: debug 2022-01-31T21:12:17.132+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.132693+0000) 2022-01-31T21:12:17.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:17 smithi167 conmon[54076]: debug 2022-01-31T21:12:17.132+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.132577+0000) 2022-01-31T21:12:17.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:17 smithi167 conmon[60316]: debug 2022-01-31T21:12:17.131+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.132302+0000) 2022-01-31T21:12:17.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:17 smithi167 conmon[60316]: debug 2022-01-31T21:12:17.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.270019+0000) 2022-01-31T21:12:17.344 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[35325]: debug 2022-01-31T21:12:17.142+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111214 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:17.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[41853]: debug 2022-01-31T21:12:17.130+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.131423+0000) 2022-01-31T21:12:17.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[51620]: debug 2022-01-31T21:12:17.131+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.132043+0000) 2022-01-31T21:12:17.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[46715]: debug 2022-01-31T21:12:17.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.046239+0000) 2022-01-31T21:12:17.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[46715]: debug 2022-01-31T21:12:17.131+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.131780+0000) 2022-01-31T21:12:17.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:17 smithi167 conmon[49112]: debug 2022-01-31T21:12:17.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.593272+0000) 2022-01-31T21:12:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:17 smithi167 conmon[54076]: debug 2022-01-31T21:12:17.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.519219+0000) 2022-01-31T21:12:17.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[51620]: debug 2022-01-31T21:12:17.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.626030+0000) 2022-01-31T21:12:18.045 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:17 smithi171 conmon[41853]: debug 2022-01-31T21:12:17.750+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:17.751362+0000) 2022-01-31T21:12:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:18 smithi171 conmon[46715]: debug 2022-01-31T21:12:18.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:18.046443+0000) 2022-01-31T21:12:18.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:18 smithi167 conmon[60316]: debug 2022-01-31T21:12:18.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:18.270206+0000) 2022-01-31T21:12:18.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:18 smithi167 conmon[49112]: debug 2022-01-31T21:12:18.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:18.593445+0000) 2022-01-31T21:12:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:18 smithi167 conmon[54076]: debug 2022-01-31T21:12:18.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:18.519383+0000) 2022-01-31T21:12:18.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:18 smithi171 conmon[51620]: debug 2022-01-31T21:12:18.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:18.626207+0000) 2022-01-31T21:12:19.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:18 smithi171 conmon[41853]: debug 2022-01-31T21:12:18.750+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:18.751521+0000) 2022-01-31T21:12:19.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:19 smithi167 conmon[60316]: debug 2022-01-31T21:12:19.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:19.270367+0000) 2022-01-31T21:12:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:19 smithi171 conmon[46715]: debug 2022-01-31T21:12:19.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:19.046614+0000) 2022-01-31T21:12:19.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:19 smithi167 conmon[49112]: debug 2022-01-31T21:12:19.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:19.593687+0000) 2022-01-31T21:12:19.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:19 smithi167 conmon[54076]: debug 2022-01-31T21:12:19.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:19.519573+0000) 2022-01-31T21:12:19.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:19 smithi171 conmon[51620]: debug 2022-01-31T21:12:19.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:19.626387+0000) 2022-01-31T21:12:20.045 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:19 smithi171 conmon[41853]: debug 2022-01-31T21:12:19.750+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:19.751686+0000) 2022-01-31T21:12:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:20 smithi171 conmon[46715]: debug 2022-01-31T21:12:20.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:20.046823+0000) 2022-01-31T21:12:20.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:20 smithi167 conmon[60316]: debug 2022-01-31T21:12:20.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:20.270570+0000) 2022-01-31T21:12:20.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:20 smithi167 conmon[49112]: debug 2022-01-31T21:12:20.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:20.593818+0000) 2022-01-31T21:12:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:20 smithi167 conmon[54076]: debug 2022-01-31T21:12:20.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:20.519761+0000) 2022-01-31T21:12:20.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:20 smithi171 conmon[51620]: debug 2022-01-31T21:12:20.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:20.626549+0000) 2022-01-31T21:12:21.045 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:20 smithi171 conmon[41853]: debug 2022-01-31T21:12:20.751+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:20.751855+0000) 2022-01-31T21:12:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:21 smithi171 conmon[46715]: debug 2022-01-31T21:12:21.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:21.046919+0000) 2022-01-31T21:12:21.400 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:21 smithi167 conmon[60316]: debug 2022-01-31T21:12:21.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:21.270763+0000) 2022-01-31T21:12:21.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:21 smithi167 conmon[49112]: debug 2022-01-31T21:12:21.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:21.593980+0000) 2022-01-31T21:12:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:21 smithi167 conmon[54076]: debug 2022-01-31T21:12:21.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:21.519918+0000) 2022-01-31T21:12:21.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:21 smithi171 conmon[51620]: debug 2022-01-31T21:12:21.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:21.626772+0000) 2022-01-31T21:12:22.045 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:21 smithi171 conmon[41853]: debug 2022-01-31T21:12:21.751+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:21.752068+0000) 2022-01-31T21:12:22.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[41853]: debug 2022-01-31T21:12:22.145+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.145861+0000) 2022-01-31T21:12:22.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[35325]: debug 2022-01-31T21:12:22.156+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111325 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[46715]: debug 2022-01-31T21:12:22.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.047020+0000) 2022-01-31T21:12:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[46715]: debug 2022-01-31T21:12:22.146+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.147131+0000) 2022-01-31T21:12:22.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[51620]: debug 2022-01-31T21:12:22.144+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.145257+0000) 2022-01-31T21:12:22.400 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:22 smithi167 conmon[49112]: debug 2022-01-31T21:12:22.146+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.146752+0000) 2022-01-31T21:12:22.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:22 smithi167 conmon[54076]: debug 2022-01-31T21:12:22.146+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.146939+0000) 2022-01-31T21:12:22.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:22 smithi167 conmon[60316]: debug 2022-01-31T21:12:22.145+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.146361+0000) 2022-01-31T21:12:22.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:22 smithi167 conmon[60316]: debug 2022-01-31T21:12:22.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.270938+0000) 2022-01-31T21:12:22.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:22 smithi167 conmon[49112]: debug 2022-01-31T21:12:22.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.594164+0000) 2022-01-31T21:12:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:22 smithi167 conmon[54076]: debug 2022-01-31T21:12:22.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.520063+0000) 2022-01-31T21:12:22.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[51620]: debug 2022-01-31T21:12:22.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.626973+0000) 2022-01-31T21:12:23.045 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:22 smithi171 conmon[41853]: debug 2022-01-31T21:12:22.751+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:22.752286+0000) 2022-01-31T21:12:23.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:23 smithi167 conmon[60316]: debug 2022-01-31T21:12:23.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:23.271101+0000) 2022-01-31T21:12:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:23 smithi171 conmon[46715]: debug 2022-01-31T21:12:23.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:23.047178+0000) 2022-01-31T21:12:23.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:23 smithi167 conmon[49112]: debug 2022-01-31T21:12:23.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:23.594319+0000) 2022-01-31T21:12:23.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:23 smithi167 conmon[54076]: debug 2022-01-31T21:12:23.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:23.520163+0000) 2022-01-31T21:12:23.709 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:23 smithi171 conmon[35325]: debug 2022-01-31T21:12:23.640+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:12:23.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:23 smithi171 conmon[51620]: debug 2022-01-31T21:12:23.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:23.627097+0000) 2022-01-31T21:12:24.045 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:23 smithi171 conmon[41853]: debug 2022-01-31T21:12:23.751+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:23.752439+0000) 2022-01-31T21:12:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:24 smithi171 conmon[46715]: debug 2022-01-31T21:12:24.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:24.047334+0000) 2022-01-31T21:12:24.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:24 smithi167 conmon[60316]: debug 2022-01-31T21:12:24.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:24.271291+0000) 2022-01-31T21:12:24.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:24 smithi167 conmon[49112]: debug 2022-01-31T21:12:24.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:24.594538+0000) 2022-01-31T21:12:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:24 smithi167 conmon[54076]: debug 2022-01-31T21:12:24.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:24.520339+0000) 2022-01-31T21:12:24.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:24 smithi171 conmon[51620]: debug 2022-01-31T21:12:24.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:24.627286+0000) 2022-01-31T21:12:25.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:24 smithi171 conmon[41853]: debug 2022-01-31T21:12:24.751+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:24.752589+0000) 2022-01-31T21:12:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:25 smithi171 conmon[46715]: debug 2022-01-31T21:12:25.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:25.047520+0000) 2022-01-31T21:12:25.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:25 smithi167 conmon[60316]: debug 2022-01-31T21:12:25.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:25.271457+0000) 2022-01-31T21:12:25.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:25 smithi167 conmon[49112]: debug 2022-01-31T21:12:25.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:25.594699+0000) 2022-01-31T21:12:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:25 smithi167 conmon[54076]: debug 2022-01-31T21:12:25.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:25.520561+0000) 2022-01-31T21:12:25.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:25 smithi171 conmon[51620]: debug 2022-01-31T21:12:25.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:25.627475+0000) 2022-01-31T21:12:26.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:25 smithi171 conmon[41853]: debug 2022-01-31T21:12:25.752+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:25.752782+0000) 2022-01-31T21:12:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:26 smithi171 conmon[46715]: debug 2022-01-31T21:12:26.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:26.047707+0000) 2022-01-31T21:12:26.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:26 smithi167 conmon[60316]: debug 2022-01-31T21:12:26.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:26.271635+0000) 2022-01-31T21:12:26.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:26 smithi167 conmon[49112]: debug 2022-01-31T21:12:26.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:26.594826+0000) 2022-01-31T21:12:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:26 smithi167 conmon[54076]: debug 2022-01-31T21:12:26.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:26.520716+0000) 2022-01-31T21:12:26.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:26 smithi171 conmon[51620]: debug 2022-01-31T21:12:26.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:26.627649+0000) 2022-01-31T21:12:27.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:26 smithi171 conmon[41853]: debug 2022-01-31T21:12:26.752+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:26.752959+0000) 2022-01-31T21:12:27.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[35325]: debug 2022-01-31T21:12:27.170+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111435 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:27.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[41853]: debug 2022-01-31T21:12:27.158+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.158958+0000) 2022-01-31T21:12:27.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[51620]: debug 2022-01-31T21:12:27.158+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.159411+0000) 2022-01-31T21:12:27.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[46715]: debug 2022-01-31T21:12:27.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.047905+0000) 2022-01-31T21:12:27.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[46715]: debug 2022-01-31T21:12:27.159+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.160401+0000) 2022-01-31T21:12:27.407 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:27 smithi167 conmon[49112]: debug 2022-01-31T21:12:27.159+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.159778+0000) 2022-01-31T21:12:27.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:27 smithi167 conmon[54076]: debug 2022-01-31T21:12:27.160+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.160681+0000) 2022-01-31T21:12:27.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:27 smithi167 conmon[60316]: debug 2022-01-31T21:12:27.160+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.160585+0000) 2022-01-31T21:12:27.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:27 smithi167 conmon[60316]: debug 2022-01-31T21:12:27.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.271797+0000) 2022-01-31T21:12:27.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:27 smithi167 conmon[49112]: debug 2022-01-31T21:12:27.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.594987+0000) 2022-01-31T21:12:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:27 smithi167 conmon[54076]: debug 2022-01-31T21:12:27.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.520916+0000) 2022-01-31T21:12:27.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[51620]: debug 2022-01-31T21:12:27.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.627830+0000) 2022-01-31T21:12:28.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:27 smithi171 conmon[41853]: debug 2022-01-31T21:12:27.752+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:27.753157+0000) 2022-01-31T21:12:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:28 smithi171 conmon[46715]: debug 2022-01-31T21:12:28.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:28.048074+0000) 2022-01-31T21:12:28.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:28 smithi167 conmon[60316]: debug 2022-01-31T21:12:28.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:28.271957+0000) 2022-01-31T21:12:28.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:28 smithi167 conmon[49112]: debug 2022-01-31T21:12:28.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:28.595191+0000) 2022-01-31T21:12:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:28 smithi167 conmon[54076]: debug 2022-01-31T21:12:28.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:28.521099+0000) 2022-01-31T21:12:28.709 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:28 smithi171 conmon[51620]: debug 2022-01-31T21:12:28.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:28.628004+0000) 2022-01-31T21:12:29.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:28 smithi171 conmon[41853]: debug 2022-01-31T21:12:28.752+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:28.753313+0000) 2022-01-31T21:12:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:29 smithi171 conmon[46715]: debug 2022-01-31T21:12:29.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:29.048183+0000) 2022-01-31T21:12:29.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:29 smithi167 conmon[60316]: debug 2022-01-31T21:12:29.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:29.272140+0000) 2022-01-31T21:12:29.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:29 smithi167 conmon[49112]: debug 2022-01-31T21:12:29.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:29.595399+0000) 2022-01-31T21:12:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:29 smithi167 conmon[54076]: debug 2022-01-31T21:12:29.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:29.521301+0000) 2022-01-31T21:12:29.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:29 smithi171 conmon[51620]: debug 2022-01-31T21:12:29.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:29.628120+0000) 2022-01-31T21:12:30.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:29 smithi171 conmon[41853]: debug 2022-01-31T21:12:29.752+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:29.753468+0000) 2022-01-31T21:12:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:30 smithi171 conmon[46715]: debug 2022-01-31T21:12:30.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:30.048376+0000) 2022-01-31T21:12:30.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:30 smithi167 conmon[60316]: debug 2022-01-31T21:12:30.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:30.272377+0000) 2022-01-31T21:12:30.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:30 smithi167 conmon[49112]: debug 2022-01-31T21:12:30.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:30.595580+0000) 2022-01-31T21:12:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:30 smithi167 conmon[54076]: debug 2022-01-31T21:12:30.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:30.521516+0000) 2022-01-31T21:12:30.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:30 smithi171 conmon[51620]: debug 2022-01-31T21:12:30.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:30.628311+0000) 2022-01-31T21:12:31.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:30 smithi171 conmon[41853]: debug 2022-01-31T21:12:30.752+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:30.753659+0000) 2022-01-31T21:12:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:31 smithi171 conmon[46715]: debug 2022-01-31T21:12:31.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:31.048534+0000) 2022-01-31T21:12:31.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:31 smithi167 conmon[60316]: debug 2022-01-31T21:12:31.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:31.272537+0000) 2022-01-31T21:12:31.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:31 smithi167 conmon[49112]: debug 2022-01-31T21:12:31.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:31.595766+0000) 2022-01-31T21:12:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:31 smithi167 conmon[54076]: debug 2022-01-31T21:12:31.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:31.521701+0000) 2022-01-31T21:12:31.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:31 smithi171 conmon[51620]: debug 2022-01-31T21:12:31.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:31.628498+0000) 2022-01-31T21:12:32.047 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:31 smithi171 conmon[41853]: debug 2022-01-31T21:12:31.753+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:31.753844+0000) 2022-01-31T21:12:32.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[35325]: debug 2022-01-31T21:12:32.183+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111546 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:32.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[41853]: debug 2022-01-31T21:12:32.172+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.173079+0000) 2022-01-31T21:12:32.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[51620]: debug 2022-01-31T21:12:32.172+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.173227+0000) 2022-01-31T21:12:32.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[46715]: debug 2022-01-31T21:12:32.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.048721+0000) 2022-01-31T21:12:32.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[46715]: debug 2022-01-31T21:12:32.172+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.172802+0000) 2022-01-31T21:12:32.414 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[49112]: debug 2022-01-31T21:12:32.173+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.174648+0000) 2022-01-31T21:12:32.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[54076]: debug 2022-01-31T21:12:32.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[54076]: 2022-01-31T21:12:32.172+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.174175+0000) 2022-01-31T21:12:32.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[60316]: debug 2022-01-31T21:12:32.172+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.173537+0000) 2022-01-31T21:12:32.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[60316]: debug 2022-01-31T21:12:32.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.272684+0000) 2022-01-31T21:12:32.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[49112]: debug 2022-01-31T21:12:32.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.595966+0000) 2022-01-31T21:12:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:32 smithi167 conmon[54076]: debug 2022-01-31T21:12:32.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.521890+0000) 2022-01-31T21:12:32.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[51620]: debug 2022-01-31T21:12:32.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.628627+0000) 2022-01-31T21:12:33.047 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:32 smithi171 conmon[41853]: debug 2022-01-31T21:12:32.753+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:32.754054+0000) 2022-01-31T21:12:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:33 smithi171 conmon[46715]: debug 2022-01-31T21:12:33.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:33.048905+0000) 2022-01-31T21:12:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:33 smithi167 conmon[60316]: debug 2022-01-31T21:12:33.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:33.272890+0000) 2022-01-31T21:12:33.710 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:33 smithi171 conmon[51620]: debug 2022-01-31T21:12:33.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:33.628832+0000) 2022-01-31T21:12:33.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:33 smithi167 conmon[49112]: debug 2022-01-31T21:12:33.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:33.596147+0000) 2022-01-31T21:12:33.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:33 smithi167 conmon[54076]: debug 2022-01-31T21:12:33.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:33.522043+0000) 2022-01-31T21:12:34.047 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:33 smithi171 conmon[41853]: debug 2022-01-31T21:12:33.753+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:33.754186+0000) 2022-01-31T21:12:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:34 smithi171 conmon[46715]: debug 2022-01-31T21:12:34.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:34.049023+0000) 2022-01-31T21:12:34.436 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:34 smithi167 conmon[60316]: debug 2022-01-31T21:12:34.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:34.273069+0000) 2022-01-31T21:12:34.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:34 smithi171 conmon[51620]: debug 2022-01-31T21:12:34.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:34.628964+0000) 2022-01-31T21:12:34.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:34 smithi167 conmon[49112]: debug 2022-01-31T21:12:34.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:34.596378+0000) 2022-01-31T21:12:34.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:34 smithi167 conmon[54076]: debug 2022-01-31T21:12:34.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:34.522192+0000) 2022-01-31T21:12:35.047 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:34 smithi171 conmon[41853]: debug 2022-01-31T21:12:34.753+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:34.754335+0000) 2022-01-31T21:12:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:35 smithi171 conmon[46715]: debug 2022-01-31T21:12:35.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:35.049154+0000) 2022-01-31T21:12:35.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:35 smithi167 conmon[60316]: debug 2022-01-31T21:12:35.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:35.273301+0000) 2022-01-31T21:12:35.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:35 smithi171 conmon[51620]: debug 2022-01-31T21:12:35.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:35.629134+0000) 2022-01-31T21:12:35.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:35 smithi167 conmon[49112]: debug 2022-01-31T21:12:35.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:35.596538+0000) 2022-01-31T21:12:35.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:35 smithi167 conmon[54076]: debug 2022-01-31T21:12:35.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:35.522368+0000) 2022-01-31T21:12:36.047 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:35 smithi171 conmon[41853]: debug 2022-01-31T21:12:35.753+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:35.754494+0000) 2022-01-31T21:12:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:36 smithi171 conmon[46715]: debug 2022-01-31T21:12:36.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:36.049328+0000) 2022-01-31T21:12:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:36 smithi167 conmon[60316]: debug 2022-01-31T21:12:36.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:36.273490+0000) 2022-01-31T21:12:36.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:36 smithi171 conmon[51620]: debug 2022-01-31T21:12:36.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:36.629282+0000) 2022-01-31T21:12:36.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:36 smithi167 conmon[49112]: debug 2022-01-31T21:12:36.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:36.596654+0000) 2022-01-31T21:12:36.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:36 smithi167 conmon[54076]: debug 2022-01-31T21:12:36.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:36.522554+0000) 2022-01-31T21:12:37.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:36 smithi171 conmon[41853]: debug 2022-01-31T21:12:36.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:36.754648+0000) 2022-01-31T21:12:37.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[35325]: debug 2022-01-31T21:12:37.197+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111655 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:37.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[51620]: debug 2022-01-31T21:12:37.186+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.186813+0000) 2022-01-31T21:12:37.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[41853]: debug 2022-01-31T21:12:37.186+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.187104+0000) 2022-01-31T21:12:37.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[46715]: debug 2022-01-31T21:12:37.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.049535+0000) 2022-01-31T21:12:37.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[46715]: debug 2022-01-31T21:12:37.186+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.187039+0000) 2022-01-31T21:12:37.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:37 smithi167 conmon[49112]: debug 2022-01-31T21:12:37.186+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.187909+0000) 2022-01-31T21:12:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:37 smithi167 conmon[54076]: debug 2022-01-31T21:12:37.186+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.188087+0000) 2022-01-31T21:12:37.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:37 smithi167 conmon[60316]: debug 2022-01-31T21:12:37.186+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.187620+0000) 2022-01-31T21:12:37.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:37 smithi167 conmon[60316]: debug 2022-01-31T21:12:37.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.273683+0000) 2022-01-31T21:12:37.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[51620]: debug 2022-01-31T21:12:37.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.629466+0000) 2022-01-31T21:12:37.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:37 smithi167 conmon[49112]: debug 2022-01-31T21:12:37.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.596810+0000) 2022-01-31T21:12:37.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:37 smithi167 conmon[54076]: debug 2022-01-31T21:12:37.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.522714+0000) 2022-01-31T21:12:38.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:37 smithi171 conmon[41853]: debug 2022-01-31T21:12:37.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:37.754848+0000) 2022-01-31T21:12:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:38 smithi171 conmon[46715]: debug 2022-01-31T21:12:38.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:38.049765+0000) 2022-01-31T21:12:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:38 smithi167 conmon[60316]: debug 2022-01-31T21:12:38.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:38.273842+0000) 2022-01-31T21:12:38.711 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:38 smithi171 conmon[35325]: debug 2022-01-31T21:12:38.641+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:12:38.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:38 smithi171 conmon[51620]: debug 2022-01-31T21:12:38.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:38.629608+0000) 2022-01-31T21:12:38.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:38 smithi167 conmon[49112]: debug 2022-01-31T21:12:38.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:38.596921+0000) 2022-01-31T21:12:38.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:38 smithi167 conmon[54076]: debug 2022-01-31T21:12:38.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:38.522844+0000) 2022-01-31T21:12:39.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:38 smithi171 conmon[41853]: debug 2022-01-31T21:12:38.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:38.754997+0000) 2022-01-31T21:12:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:39 smithi171 conmon[46715]: debug 2022-01-31T21:12:39.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:39.049933+0000) 2022-01-31T21:12:39.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:39 smithi167 conmon[60316]: debug 2022-01-31T21:12:39.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:39.273998+0000) 2022-01-31T21:12:39.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:39 smithi171 conmon[51620]: debug 2022-01-31T21:12:39.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:39.629834+0000) 2022-01-31T21:12:39.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:39 smithi167 conmon[54076]: debug 2022-01-31T21:12:39.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:39.522942+0000) 2022-01-31T21:12:39.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:39 smithi167 conmon[49112]: debug 2022-01-31T21:12:39.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:39.597117+0000) 2022-01-31T21:12:40.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:39 smithi171 conmon[41853]: debug 2022-01-31T21:12:39.755+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:39.755181+0000) 2022-01-31T21:12:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:40 smithi171 conmon[46715]: debug 2022-01-31T21:12:40.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:40.050119+0000) 2022-01-31T21:12:40.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:40 smithi167 conmon[60316]: debug 2022-01-31T21:12:40.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:40.274217+0000) 2022-01-31T21:12:40.711 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:40 smithi171 conmon[51620]: debug 2022-01-31T21:12:40.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:40.630011+0000) 2022-01-31T21:12:40.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:40 smithi167 conmon[49112]: debug 2022-01-31T21:12:40.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:40.597321+0000) 2022-01-31T21:12:40.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:40 smithi167 conmon[54076]: debug 2022-01-31T21:12:40.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:40.523099+0000) 2022-01-31T21:12:41.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:40 smithi171 conmon[41853]: debug 2022-01-31T21:12:40.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:40.755293+0000) 2022-01-31T21:12:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:41 smithi171 conmon[46715]: debug 2022-01-31T21:12:41.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:41.050322+0000) 2022-01-31T21:12:41.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:41 smithi167 conmon[60316]: debug 2022-01-31T21:12:41.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:41.274370+0000) 2022-01-31T21:12:41.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:41 smithi171 conmon[51620]: debug 2022-01-31T21:12:41.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:41.630185+0000) 2022-01-31T21:12:41.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:41 smithi167 conmon[54076]: debug 2022-01-31T21:12:41.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:41.523318+0000) 2022-01-31T21:12:41.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:41 smithi167 conmon[49112]: debug 2022-01-31T21:12:41.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:41.597520+0000) 2022-01-31T21:12:42.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:41 smithi171 conmon[41853]: debug 2022-01-31T21:12:41.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:41.755422+0000) 2022-01-31T21:12:42.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[35325]: debug 2022-01-31T21:12:42.210+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111767 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:42.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[41853]: debug 2022-01-31T21:12:42.199+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.200781+0000) 2022-01-31T21:12:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[46715]: debug 2022-01-31T21:12:42.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.050507+0000) 2022-01-31T21:12:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[46715]: debug 2022-01-31T21:12:42.198+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.200598+0000) 2022-01-31T21:12:42.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[51620]: debug 2022-01-31T21:12:42.198+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.200433+0000) 2022-01-31T21:12:42.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[49112]: debug 2022-01-31T21:12:42.200+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.201881+0000) 2022-01-31T21:12:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[60316]: debug 2022-01-31T21:12:42.199+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.200964+0000) 2022-01-31T21:12:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[60316]: debug 2022-01-31T21:12:42.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.274613+0000) 2022-01-31T21:12:42.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[54076]: debug 2022-01-31T21:12:42.199+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.200835+0000) 2022-01-31T21:12:42.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[51620]: debug 2022-01-31T21:12:42.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.630334+0000) 2022-01-31T21:12:42.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[54076]: debug 2022-01-31T21:12:42.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.523432+0000) 2022-01-31T21:12:42.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[54076]: 2022-01-31T21:12:42.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:42 smithi167 conmon[49112]: debug 2022-01-31T21:12:42.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.597656+0000) 2022-01-31T21:12:43.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:42 smithi171 conmon[41853]: debug 2022-01-31T21:12:42.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:42.755572+0000) 2022-01-31T21:12:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:43 smithi171 conmon[46715]: debug 2022-01-31T21:12:43.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:43.050732+0000) 2022-01-31T21:12:43.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:43 smithi167 conmon[60316]: debug 2022-01-31T21:12:43.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:43.274797+0000) 2022-01-31T21:12:43.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:43 smithi171 conmon[51620]: debug 2022-01-31T21:12:43.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:43.630471+0000) 2022-01-31T21:12:43.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:43 smithi167 conmon[54076]: debug 2022-01-31T21:12:43.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:43.523587+0000) 2022-01-31T21:12:43.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:43 smithi167 conmon[49112]: debug 2022-01-31T21:12:43.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:43.597790+0000) 2022-01-31T21:12:44.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:43 smithi171 conmon[41853]: debug 2022-01-31T21:12:43.753+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:43.755734+0000) 2022-01-31T21:12:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:44 smithi171 conmon[46715]: debug 2022-01-31T21:12:44.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:44.050929+0000) 2022-01-31T21:12:44.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:44 smithi167 conmon[60316]: debug 2022-01-31T21:12:44.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:44.274975+0000) 2022-01-31T21:12:44.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:44 smithi171 conmon[51620]: debug 2022-01-31T21:12:44.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:44.630664+0000) 2022-01-31T21:12:44.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:44 smithi167 conmon[49112]: debug 2022-01-31T21:12:44.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:44.597960+0000) 2022-01-31T21:12:44.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:44 smithi167 conmon[54076]: debug 2022-01-31T21:12:44.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:44.523804+0000) 2022-01-31T21:12:45.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:44 smithi171 conmon[41853]: debug 2022-01-31T21:12:44.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:44.755897+0000) 2022-01-31T21:12:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:45 smithi171 conmon[46715]: debug 2022-01-31T21:12:45.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:45.051064+0000) 2022-01-31T21:12:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:45 smithi167 conmon[60316]: debug 2022-01-31T21:12:45.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:45.275181+0000) 2022-01-31T21:12:45.712 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:45 smithi171 conmon[51620]: debug 2022-01-31T21:12:45.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:45.630788+0000) 2022-01-31T21:12:45.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:45 smithi167 conmon[49112]: debug 2022-01-31T21:12:45.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:45.598154+0000) 2022-01-31T21:12:45.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:45 smithi167 conmon[54076]: debug 2022-01-31T21:12:45.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:45.523951+0000) 2022-01-31T21:12:46.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:45 smithi171 conmon[41853]: debug 2022-01-31T21:12:45.755+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:45.756078+0000) 2022-01-31T21:12:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:46 smithi171 conmon[46715]: debug 2022-01-31T21:12:46.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:46.051234+0000) 2022-01-31T21:12:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:46 smithi167 conmon[60316]: debug 2022-01-31T21:12:46.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:46.275376+0000) 2022-01-31T21:12:46.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:46 smithi171 conmon[51620]: debug 2022-01-31T21:12:46.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:46.630965+0000) 2022-01-31T21:12:46.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:46 smithi167 conmon[49112]: debug 2022-01-31T21:12:46.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:46.598342+0000) 2022-01-31T21:12:46.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:46 smithi167 conmon[54076]: debug 2022-01-31T21:12:46.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:46.524087+0000) 2022-01-31T21:12:47.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:46 smithi171 conmon[41853]: debug 2022-01-31T21:12:46.755+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:46.756258+0000) 2022-01-31T21:12:47.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[41853]: debug 2022-01-31T21:12:47.211+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.213818+0000) 2022-01-31T21:12:47.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[35325]: debug 2022-01-31T21:12:47.228+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111876 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:47.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[46715]: debug 2022-01-31T21:12:47.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.051416+0000) 2022-01-31T21:12:47.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[46715]: debug 2022-01-31T21:12:47.211+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.214849+0000) 2022-01-31T21:12:47.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[51620]: debug 2022-01-31T21:12:47.211+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.214626+0000) 2022-01-31T21:12:47.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:47 smithi167 conmon[49112]: debug 2022-01-31T21:12:47.214+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.215569+0000) 2022-01-31T21:12:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:47 smithi167 conmon[54076]: debug 2022-01-31T21:12:47.213+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.215022+0000) 2022-01-31T21:12:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:47 smithi167 conmon[60316]: debug 2022-01-31T21:12:47.212+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.214208+0000) 2022-01-31T21:12:47.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:47 smithi167 conmon[60316]: debug 2022-01-31T21:12:47.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.275510+0000) 2022-01-31T21:12:47.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[51620]: debug 2022-01-31T21:12:47.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.631155+0000) 2022-01-31T21:12:47.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:47 smithi167 conmon[54076]: debug 2022-01-31T21:12:47.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.524267+0000) 2022-01-31T21:12:47.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:47 smithi167 conmon[49112]: debug 2022-01-31T21:12:47.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.598527+0000) 2022-01-31T21:12:48.050 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:47 smithi171 conmon[41853]: debug 2022-01-31T21:12:47.755+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:47.756418+0000) 2022-01-31T21:12:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:48 smithi171 conmon[46715]: debug 2022-01-31T21:12:48.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:48.051574+0000) 2022-01-31T21:12:48.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:48 smithi167 conmon[60316]: debug 2022-01-31T21:12:48.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:48.275683+0000) 2022-01-31T21:12:48.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:48 smithi171 conmon[51620]: debug 2022-01-31T21:12:48.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:48.631382+0000) 2022-01-31T21:12:48.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:48 smithi167 conmon[49112]: debug 2022-01-31T21:12:48.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:48.598666+0000) 2022-01-31T21:12:48.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:48 smithi167 conmon[54076]: debug 2022-01-31T21:12:48.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:48.524413+0000) 2022-01-31T21:12:49.050 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:48 smithi171 conmon[41853]: debug 2022-01-31T21:12:48.754+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:48.756568+0000) 2022-01-31T21:12:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:49 smithi171 conmon[46715]: debug 2022-01-31T21:12:49.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:49.051804+0000) 2022-01-31T21:12:49.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:49 smithi167 conmon[60316]: debug 2022-01-31T21:12:49.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:49.275870+0000) 2022-01-31T21:12:49.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:49 smithi171 conmon[51620]: debug 2022-01-31T21:12:49.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:49.631561+0000) 2022-01-31T21:12:49.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:49 smithi167 conmon[49112]: debug 2022-01-31T21:12:49.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:49.598828+0000) 2022-01-31T21:12:49.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:49 smithi167 conmon[54076]: debug 2022-01-31T21:12:49.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:49.524600+0000) 2022-01-31T21:12:50.050 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:49 smithi171 conmon[41853]: debug 2022-01-31T21:12:49.755+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:49.756750+0000) 2022-01-31T21:12:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:50 smithi171 conmon[46715]: debug 2022-01-31T21:12:50.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:50.051984+0000) 2022-01-31T21:12:50.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:50 smithi167 conmon[60316]: debug 2022-01-31T21:12:50.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:50.276091+0000) 2022-01-31T21:12:50.713 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:50 smithi171 conmon[51620]: debug 2022-01-31T21:12:50.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:50.631781+0000) 2022-01-31T21:12:50.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:50 smithi167 conmon[49112]: debug 2022-01-31T21:12:50.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:50.599028+0000) 2022-01-31T21:12:50.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:50 smithi167 conmon[54076]: debug 2022-01-31T21:12:50.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:50.524799+0000) 2022-01-31T21:12:51.050 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:50 smithi171 conmon[41853]: debug 2022-01-31T21:12:50.756+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:50.756929+0000) 2022-01-31T21:12:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:51 smithi171 conmon[46715]: debug 2022-01-31T21:12:51.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:51.052135+0000) 2022-01-31T21:12:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:51 smithi167 conmon[60316]: debug 2022-01-31T21:12:51.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:51.276256+0000) 2022-01-31T21:12:51.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:51 smithi171 conmon[51620]: debug 2022-01-31T21:12:51.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:51.631953+0000) 2022-01-31T21:12:51.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:51 smithi167 conmon[49112]: debug 2022-01-31T21:12:51.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:51.599189+0000) 2022-01-31T21:12:51.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:51 smithi167 conmon[54076]: debug 2022-01-31T21:12:51.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:51.524973+0000) 2022-01-31T21:12:52.050 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:51 smithi171 conmon[41853]: debug 2022-01-31T21:12:51.756+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:51.757081+0000) 2022-01-31T21:12:52.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[35325]: debug 2022-01-31T21:12:52.242+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 111987 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[41853]: debug 2022-01-31T21:12:52.232+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.232841+0000) 2022-01-31T21:12:52.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[51620]: debug 2022-01-31T21:12:52.230+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.231652+0000) 2022-01-31T21:12:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[46715]: debug 2022-01-31T21:12:52.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.052313+0000) 2022-01-31T21:12:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[46715]: debug 2022-01-31T21:12:52.231+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.232526+0000) 2022-01-31T21:12:52.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:52 smithi167 conmon[49112]: debug 2022-01-31T21:12:52.231+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.233415+0000) 2022-01-31T21:12:52.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:52 smithi167 conmon[54076]: debug 2022-01-31T21:12:52.230+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.232334+0000) 2022-01-31T21:12:52.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:52 smithi167 conmon[60316]: debug 2022-01-31T21:12:52.232+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.233646+0000) 2022-01-31T21:12:52.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:52 smithi167 conmon[60316]: debug 2022-01-31T21:12:52.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.276422+0000) 2022-01-31T21:12:52.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[51620]: debug 2022-01-31T21:12:52.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.632158+0000) 2022-01-31T21:12:52.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:52 smithi167 conmon[54076]: debug 2022-01-31T21:12:52.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.525123+0000) 2022-01-31T21:12:52.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:52 smithi167 conmon[49112]: debug 2022-01-31T21:12:52.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.599342+0000) 2022-01-31T21:12:53.050 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:52 smithi171 conmon[41853]: debug 2022-01-31T21:12:52.756+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:52.757276+0000) 2022-01-31T21:12:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:53 smithi171 conmon[46715]: debug 2022-01-31T21:12:53.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:53.052445+0000) 2022-01-31T21:12:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:53 smithi167 conmon[60316]: debug 2022-01-31T21:12:53.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:53.276592+0000) 2022-01-31T21:12:53.714 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:53 smithi171 conmon[35325]: debug 2022-01-31T21:12:53.641+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:12:53.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:53 smithi171 conmon[51620]: debug 2022-01-31T21:12:53.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:53.632298+0000) 2022-01-31T21:12:53.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:53 smithi167 conmon[49112]: debug 2022-01-31T21:12:53.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:53.599467+0000) 2022-01-31T21:12:53.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:53 smithi167 conmon[54076]: debug 2022-01-31T21:12:53.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:53.525250+0000) 2022-01-31T21:12:54.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:53 smithi171 conmon[41853]: debug 2022-01-31T21:12:53.756+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:53.757466+0000) 2022-01-31T21:12:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:54 smithi171 conmon[46715]: debug 2022-01-31T21:12:54.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:54.052586+0000) 2022-01-31T21:12:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:54 smithi167 conmon[60316]: debug 2022-01-31T21:12:54.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:54.276777+0000) 2022-01-31T21:12:54.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:54 smithi171 conmon[51620]: debug 2022-01-31T21:12:54.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:54.632499+0000) 2022-01-31T21:12:54.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:54 smithi167 conmon[49112]: debug 2022-01-31T21:12:54.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:54.599660+0000) 2022-01-31T21:12:54.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:54 smithi167 conmon[54076]: debug 2022-01-31T21:12:54.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:54.525423+0000) 2022-01-31T21:12:55.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:54 smithi171 conmon[41853]: debug 2022-01-31T21:12:54.756+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:54.757649+0000) 2022-01-31T21:12:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:55 smithi171 conmon[46715]: debug 2022-01-31T21:12:55.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:55.052785+0000) 2022-01-31T21:12:55.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:55 smithi167 conmon[60316]: debug 2022-01-31T21:12:55.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:55.276998+0000) 2022-01-31T21:12:55.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:55 smithi171 conmon[51620]: debug 2022-01-31T21:12:55.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:55.632685+0000) 2022-01-31T21:12:55.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:55 smithi167 conmon[49112]: debug 2022-01-31T21:12:55.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:55.599846+0000) 2022-01-31T21:12:55.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:55 smithi167 conmon[54076]: debug 2022-01-31T21:12:55.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:55.525523+0000) 2022-01-31T21:12:56.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:55 smithi171 conmon[41853]: debug 2022-01-31T21:12:55.757+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:55.757859+0000) 2022-01-31T21:12:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:56 smithi171 conmon[46715]: debug 2022-01-31T21:12:56.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:56.052973+0000) 2022-01-31T21:12:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:56 smithi167 conmon[60316]: debug 2022-01-31T21:12:56.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:56.277150+0000) 2022-01-31T21:12:56.714 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:56 smithi171 conmon[51620]: debug 2022-01-31T21:12:56.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:56.632840+0000) 2022-01-31T21:12:56.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:56 smithi167 conmon[49112]: debug 2022-01-31T21:12:56.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:56.600032+0000) 2022-01-31T21:12:56.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:56 smithi167 conmon[54076]: debug 2022-01-31T21:12:56.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:56.525642+0000) 2022-01-31T21:12:57.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:56 smithi171 conmon[41853]: debug 2022-01-31T21:12:56.757+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:56.758012+0000) 2022-01-31T21:12:57.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[35325]: debug 2022-01-31T21:12:57.256+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112096 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:12:57.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[41853]: debug 2022-01-31T21:12:57.245+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.246267+0000) 2022-01-31T21:12:57.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[51620]: debug 2022-01-31T21:12:57.244+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.245522+0000) 2022-01-31T21:12:57.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[46715]: debug 2022-01-31T21:12:57.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.053177+0000) 2022-01-31T21:12:57.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[46715]: debug 2022-01-31T21:12:57.246+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.246869+0000) 2022-01-31T21:12:57.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:57 smithi167 conmon[49112]: debug 2022-01-31T21:12:57.246+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.246525+0000) 2022-01-31T21:12:57.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:57 smithi167 conmon[54076]: debug 2022-01-31T21:12:57.245+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.246356+0000) 2022-01-31T21:12:57.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:57 smithi167 conmon[60316]: debug 2022-01-31T21:12:57.246+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.247220+0000) 2022-01-31T21:12:57.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:57 smithi167 conmon[60316]: debug 2022-01-31T21:12:57.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.277326+0000) 2022-01-31T21:12:57.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[51620]: debug 2022-01-31T21:12:57.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.633007+0000) 2022-01-31T21:12:57.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:57 smithi167 conmon[54076]: debug 2022-01-31T21:12:57.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.525834+0000) 2022-01-31T21:12:57.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:57 smithi167 conmon[49112]: debug 2022-01-31T21:12:57.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.600228+0000) 2022-01-31T21:12:58.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:57 smithi171 conmon[41853]: debug 2022-01-31T21:12:57.757+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:57.758194+0000) 2022-01-31T21:12:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:58 smithi171 conmon[46715]: debug 2022-01-31T21:12:58.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:58.053371+0000) 2022-01-31T21:12:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:58 smithi167 conmon[60316]: debug 2022-01-31T21:12:58.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:58.277481+0000) 2022-01-31T21:12:58.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:58 smithi171 conmon[51620]: debug 2022-01-31T21:12:58.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:58.633214+0000) 2022-01-31T21:12:58.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:58 smithi167 conmon[54076]: debug 2022-01-31T21:12:58.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:58.526065+0000) 2022-01-31T21:12:58.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:58 smithi167 conmon[49112]: debug 2022-01-31T21:12:58.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:58.600377+0000) 2022-01-31T21:12:59.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:58 smithi171 conmon[41853]: debug 2022-01-31T21:12:58.757+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:58.758362+0000) 2022-01-31T21:12:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:12:59 smithi171 conmon[46715]: debug 2022-01-31T21:12:59.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:59.053539+0000) 2022-01-31T21:12:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:12:59 smithi167 conmon[60316]: debug 2022-01-31T21:12:59.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:59.277636+0000) 2022-01-31T21:12:59.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:12:59 smithi171 conmon[51620]: debug 2022-01-31T21:12:59.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:59.633427+0000) 2022-01-31T21:12:59.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:12:59 smithi167 conmon[49112]: debug 2022-01-31T21:12:59.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:59.600597+0000) 2022-01-31T21:12:59.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:12:59 smithi167 conmon[54076]: debug 2022-01-31T21:12:59.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:59.526268+0000) 2022-01-31T21:13:00.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:12:59 smithi171 conmon[41853]: debug 2022-01-31T21:12:59.757+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:12:59.758533+0000) 2022-01-31T21:13:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:00 smithi171 conmon[46715]: debug 2022-01-31T21:13:00.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:00.053724+0000) 2022-01-31T21:13:00.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:00 smithi167 conmon[60316]: debug 2022-01-31T21:13:00.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:00.277792+0000) 2022-01-31T21:13:00.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:00 smithi171 conmon[51620]: debug 2022-01-31T21:13:00.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:00.633612+0000) 2022-01-31T21:13:00.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:00 smithi167 conmon[54076]: debug 2022-01-31T21:13:00.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:00.526474+0000) 2022-01-31T21:13:00.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:00 smithi167 conmon[49112]: debug 2022-01-31T21:13:00.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:00.600797+0000) 2022-01-31T21:13:01.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:00 smithi171 conmon[41853]: debug 2022-01-31T21:13:00.758+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:00.758688+0000) 2022-01-31T21:13:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:01 smithi171 conmon[46715]: debug 2022-01-31T21:13:01.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:01.053891+0000) 2022-01-31T21:13:01.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:01 smithi167 conmon[60316]: debug 2022-01-31T21:13:01.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:01.277977+0000) 2022-01-31T21:13:01.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:01 smithi171 conmon[51620]: debug 2022-01-31T21:13:01.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:01.633824+0000) 2022-01-31T21:13:01.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:01 smithi167 conmon[49112]: debug 2022-01-31T21:13:01.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:01.600984+0000) 2022-01-31T21:13:01.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:01 smithi167 conmon[54076]: debug 2022-01-31T21:13:01.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:01.526621+0000) 2022-01-31T21:13:02.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:01 smithi171 conmon[41853]: debug 2022-01-31T21:13:01.758+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:01.758872+0000) 2022-01-31T21:13:02.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[35325]: debug 2022-01-31T21:13:02.271+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112206 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[41853]: debug 2022-01-31T21:13:02.259+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.259878+0000) 2022-01-31T21:13:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[46715]: debug 2022-01-31T21:13:02.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.054086+0000) 2022-01-31T21:13:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[46715]: debug 2022-01-31T21:13:02.259+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.260267+0000) 2022-01-31T21:13:02.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[51620]: debug 2022-01-31T21:13:02.258+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.259201+0000) 2022-01-31T21:13:02.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:02 smithi167 conmon[49112]: debug 2022-01-31T21:13:02.259+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.260118+0000) 2022-01-31T21:13:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:02 smithi167 conmon[54076]: debug 2022-01-31T21:13:02.259+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.259956+0000) 2022-01-31T21:13:02.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:02 smithi167 conmon[60316]: debug 2022-01-31T21:13:02.260+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.260580+0000) 2022-01-31T21:13:02.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:02 smithi167 conmon[60316]: debug 2022-01-31T21:13:02.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.278117+0000) 2022-01-31T21:13:02.715 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[51620]: debug 2022-01-31T21:13:02.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.633985+0000) 2022-01-31T21:13:02.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:02 smithi167 conmon[49112]: debug 2022-01-31T21:13:02.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.601208+0000) 2022-01-31T21:13:02.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:02 smithi167 conmon[54076]: debug 2022-01-31T21:13:02.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.526801+0000) 2022-01-31T21:13:03.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:02 smithi171 conmon[41853]: debug 2022-01-31T21:13:02.758+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:02.759082+0000) 2022-01-31T21:13:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:03 smithi171 conmon[46715]: debug 2022-01-31T21:13:03.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:03.054190+0000) 2022-01-31T21:13:03.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:03 smithi167 conmon[60316]: debug 2022-01-31T21:13:03.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:03.278286+0000) 2022-01-31T21:13:03.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:03 smithi171 conmon[51620]: debug 2022-01-31T21:13:03.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:03.634140+0000) 2022-01-31T21:13:03.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:03 smithi167 conmon[49112]: debug 2022-01-31T21:13:03.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:03.601364+0000) 2022-01-31T21:13:03.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:03 smithi167 conmon[54076]: debug 2022-01-31T21:13:03.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:03.526941+0000) 2022-01-31T21:13:04.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:03 smithi171 conmon[41853]: debug 2022-01-31T21:13:03.758+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:03.759165+0000) 2022-01-31T21:13:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:04 smithi171 conmon[46715]: debug 2022-01-31T21:13:04.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:04.054332+0000) 2022-01-31T21:13:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:04 smithi167 conmon[60316]: debug 2022-01-31T21:13:04.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:04.278492+0000) 2022-01-31T21:13:04.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:04 smithi171 conmon[51620]: debug 2022-01-31T21:13:04.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:04.634327+0000) 2022-01-31T21:13:04.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:04 smithi167 conmon[49112]: debug 2022-01-31T21:13:04.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:04.601552+0000) 2022-01-31T21:13:04.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:04 smithi167 conmon[54076]: debug 2022-01-31T21:13:04.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:04.527152+0000) 2022-01-31T21:13:05.053 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:04 smithi171 conmon[41853]: debug 2022-01-31T21:13:04.758+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:04.759285+0000) 2022-01-31T21:13:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:05 smithi171 conmon[46715]: debug 2022-01-31T21:13:05.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:05.054514+0000) 2022-01-31T21:13:05.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:05 smithi167 conmon[60316]: debug 2022-01-31T21:13:05.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:05.278737+0000) 2022-01-31T21:13:05.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:05 smithi171 conmon[51620]: debug 2022-01-31T21:13:05.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:05.634510+0000) 2022-01-31T21:13:05.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:05 smithi167 conmon[49112]: debug 2022-01-31T21:13:05.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:05.601706+0000) 2022-01-31T21:13:05.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:05 smithi167 conmon[54076]: debug 2022-01-31T21:13:05.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:05.527348+0000) 2022-01-31T21:13:06.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:05 smithi171 conmon[41853]: debug 2022-01-31T21:13:05.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:05.759489+0000) 2022-01-31T21:13:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:06 smithi171 conmon[46715]: debug 2022-01-31T21:13:06.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:06.054694+0000) 2022-01-31T21:13:06.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:06 smithi167 conmon[60316]: debug 2022-01-31T21:13:06.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:06.278900+0000) 2022-01-31T21:13:06.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:06 smithi171 conmon[51620]: debug 2022-01-31T21:13:06.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:06.634682+0000) 2022-01-31T21:13:06.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:06 smithi167 conmon[49112]: debug 2022-01-31T21:13:06.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:06.601904+0000) 2022-01-31T21:13:06.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:06 smithi167 conmon[54076]: debug 2022-01-31T21:13:06.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:06.527518+0000) 2022-01-31T21:13:07.053 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:06 smithi171 conmon[41853]: debug 2022-01-31T21:13:06.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:06.759646+0000) 2022-01-31T21:13:07.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[35325]: debug 2022-01-31T21:13:07.285+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112316 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[41853]: debug 2022-01-31T21:13:07.273+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.274083+0000) 2022-01-31T21:13:07.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[51620]: debug 2022-01-31T21:13:07.274+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.275117+0000) 2022-01-31T21:13:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[46715]: debug 2022-01-31T21:13:07.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.054931+0000) 2022-01-31T21:13:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[46715]: debug 2022-01-31T21:13:07.275+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.276069+0000) 2022-01-31T21:13:07.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:07 smithi167 conmon[49112]: debug 2022-01-31T21:13:07.274+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.274806+0000) 2022-01-31T21:13:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:07 smithi167 conmon[54076]: debug 2022-01-31T21:13:07.275+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.275496+0000) 2022-01-31T21:13:07.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:07 smithi167 conmon[60316]: debug 2022-01-31T21:13:07.275+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.275833+0000) 2022-01-31T21:13:07.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:07 smithi167 conmon[60316]: debug 2022-01-31T21:13:07.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.279021+0000) 2022-01-31T21:13:07.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[51620]: debug 2022-01-31T21:13:07.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.634875+0000) 2022-01-31T21:13:07.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:07 smithi167 conmon[49112]: debug 2022-01-31T21:13:07.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.602055+0000) 2022-01-31T21:13:07.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:07 smithi167 conmon[54076]: debug 2022-01-31T21:13:07.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.527709+0000) 2022-01-31T21:13:08.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:07 smithi171 conmon[41853]: debug 2022-01-31T21:13:07.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:07.759801+0000) 2022-01-31T21:13:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:08 smithi171 conmon[46715]: debug 2022-01-31T21:13:08.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:08.055103+0000) 2022-01-31T21:13:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:08 smithi167 conmon[60316]: debug 2022-01-31T21:13:08.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:08.279167+0000) 2022-01-31T21:13:08.716 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:08 smithi171 conmon[51620]: debug 2022-01-31T21:13:08.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:08.635050+0000) 2022-01-31T21:13:08.717 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:08 smithi171 conmon[35325]: debug 2022-01-31T21:13:08.642+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:13:08.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:08 smithi167 conmon[49112]: debug 2022-01-31T21:13:08.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:08.602228+0000) 2022-01-31T21:13:08.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:08 smithi167 conmon[54076]: debug 2022-01-31T21:13:08.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:08.527886+0000) 2022-01-31T21:13:09.053 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:08 smithi171 conmon[41853]: debug 2022-01-31T21:13:08.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:08.759950+0000) 2022-01-31T21:13:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:09 smithi171 conmon[46715]: debug 2022-01-31T21:13:09.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:09.055280+0000) 2022-01-31T21:13:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:09 smithi167 conmon[60316]: debug 2022-01-31T21:13:09.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:09.279354+0000) 2022-01-31T21:13:09.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:09 smithi171 conmon[51620]: debug 2022-01-31T21:13:09.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:09.635237+0000) 2022-01-31T21:13:09.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:09 smithi167 conmon[49112]: debug 2022-01-31T21:13:09.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:09.602421+0000) 2022-01-31T21:13:09.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:09 smithi167 conmon[54076]: debug 2022-01-31T21:13:09.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:09.528050+0000) 2022-01-31T21:13:10.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:09 smithi171 conmon[41853]: debug 2022-01-31T21:13:09.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:09.760147+0000) 2022-01-31T21:13:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:10 smithi171 conmon[46715]: debug 2022-01-31T21:13:10.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:10.055464+0000) 2022-01-31T21:13:10.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:10 smithi167 conmon[60316]: debug 2022-01-31T21:13:10.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:10.279555+0000) 2022-01-31T21:13:10.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:10 smithi171 conmon[51620]: debug 2022-01-31T21:13:10.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:10.635419+0000) 2022-01-31T21:13:10.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:10 smithi167 conmon[49112]: debug 2022-01-31T21:13:10.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:10.602672+0000) 2022-01-31T21:13:10.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:10 smithi167 conmon[54076]: debug 2022-01-31T21:13:10.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:10.528178+0000) 2022-01-31T21:13:11.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:10 smithi171 conmon[41853]: debug 2022-01-31T21:13:10.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:10.760308+0000) 2022-01-31T21:13:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:11 smithi171 conmon[46715]: debug 2022-01-31T21:13:11.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:11.055646+0000) 2022-01-31T21:13:11.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:11 smithi167 conmon[60316]: debug 2022-01-31T21:13:11.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:11.279737+0000) 2022-01-31T21:13:11.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:11 smithi171 conmon[51620]: debug 2022-01-31T21:13:11.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:11.635591+0000) 2022-01-31T21:13:11.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:11 smithi167 conmon[49112]: debug 2022-01-31T21:13:11.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:11.602844+0000) 2022-01-31T21:13:11.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:11 smithi167 conmon[54076]: debug 2022-01-31T21:13:11.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:11.528335+0000) 2022-01-31T21:13:12.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:11 smithi171 conmon[41853]: debug 2022-01-31T21:13:11.760+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:11.760425+0000) 2022-01-31T21:13:12.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:12 smithi167 conmon[60316]: debug 2022-01-31T21:13:12.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.279927+0000) 2022-01-31T21:13:12.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[51620]: debug 2022-01-31T21:13:12.287+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.287758+0000) 2022-01-31T21:13:12.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[35325]: debug 2022-01-31T21:13:12.298+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112426 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:12.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[41853]: debug 2022-01-31T21:13:12.287+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.288126+0000) 2022-01-31T21:13:12.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[46715]: debug 2022-01-31T21:13:12.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.055834+0000) 2022-01-31T21:13:12.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[46715]: debug 2022-01-31T21:13:12.288+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.289196+0000) 2022-01-31T21:13:12.579 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:12 smithi167 conmon[49112]: debug 2022-01-31T21:13:12.288+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.288597+0000) 2022-01-31T21:13:12.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:12 smithi167 conmon[60316]: debug 2022-01-31T21:13:12.288+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.289108+0000) 2022-01-31T21:13:12.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:12 smithi167 conmon[54076]: debug 2022-01-31T21:13:12.288+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.288927+0000) 2022-01-31T21:13:12.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:12 smithi167 conmon[54076]: debug 2022-01-31T21:13:12.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.528486+0000) 2022-01-31T21:13:12.717 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[51620]: debug 2022-01-31T21:13:12.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.635748+0000) 2022-01-31T21:13:12.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:12 smithi167 conmon[49112]: debug 2022-01-31T21:13:12.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.603011+0000) 2022-01-31T21:13:13.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:12 smithi171 conmon[41853]: debug 2022-01-31T21:13:12.759+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:12.760644+0000) 2022-01-31T21:13:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:13 smithi171 conmon[46715]: debug 2022-01-31T21:13:13.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:13.056022+0000) 2022-01-31T21:13:13.527 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:13 smithi167 conmon[60316]: debug 2022-01-31T21:13:13.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:13.280116+0000) 2022-01-31T21:13:13.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:13 smithi171 conmon[51620]: debug 2022-01-31T21:13:13.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:13.635938+0000) 2022-01-31T21:13:13.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:13 smithi167 conmon[49112]: debug 2022-01-31T21:13:13.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:13.603175+0000) 2022-01-31T21:13:13.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:13 smithi167 conmon[54076]: debug 2022-01-31T21:13:13.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:13.528688+0000) 2022-01-31T21:13:14.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:13 smithi171 conmon[41853]: debug 2022-01-31T21:13:13.760+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:13.760819+0000) 2022-01-31T21:13:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:14 smithi171 conmon[46715]: debug 2022-01-31T21:13:14.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:14.056102+0000) 2022-01-31T21:13:14.527 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:14 smithi167 conmon[60316]: debug 2022-01-31T21:13:14.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:14.280244+0000) 2022-01-31T21:13:14.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:14 smithi171 conmon[51620]: debug 2022-01-31T21:13:14.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:14.636149+0000) 2022-01-31T21:13:14.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:14 smithi167 conmon[49112]: debug 2022-01-31T21:13:14.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:14.603383+0000) 2022-01-31T21:13:14.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:14 smithi167 conmon[54076]: debug 2022-01-31T21:13:14.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:14.528849+0000) 2022-01-31T21:13:15.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:14 smithi171 conmon[41853]: debug 2022-01-31T21:13:14.760+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:14.760987+0000) 2022-01-31T21:13:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:15 smithi171 conmon[46715]: debug 2022-01-31T21:13:15.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:15.056291+0000) 2022-01-31T21:13:15.527 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:15 smithi167 conmon[60316]: debug 2022-01-31T21:13:15.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:15.280406+0000) 2022-01-31T21:13:15.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:15 smithi171 conmon[51620]: debug 2022-01-31T21:13:15.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:15.636316+0000) 2022-01-31T21:13:15.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:15 smithi167 conmon[49112]: debug 2022-01-31T21:13:15.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:15.603551+0000) 2022-01-31T21:13:15.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:15 smithi167 conmon[54076]: debug 2022-01-31T21:13:15.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:15.529002+0000) 2022-01-31T21:13:16.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:15 smithi171 conmon[41853]: debug 2022-01-31T21:13:15.760+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:15.761193+0000) 2022-01-31T21:13:16.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:16 smithi171 conmon[46715]: debug 2022-01-31T21:13:16.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:16.056474+0000) 2022-01-31T21:13:16.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:16 smithi167 conmon[60316]: debug 2022-01-31T21:13:16.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:16.280569+0000) 2022-01-31T21:13:16.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:16 smithi171 conmon[51620]: debug 2022-01-31T21:13:16.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:16.636474+0000) 2022-01-31T21:13:16.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:16 smithi167 conmon[49112]: debug 2022-01-31T21:13:16.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:16.603726+0000) 2022-01-31T21:13:16.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:16 smithi167 conmon[54076]: debug 2022-01-31T21:13:16.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:16.529176+0000) 2022-01-31T21:13:17.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:16 smithi171 conmon[41853]: debug 2022-01-31T21:13:16.760+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:16.761378+0000) 2022-01-31T21:13:17.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[41853]: debug 2022-01-31T21:13:17.300+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.301718+0000) 2022-01-31T21:13:17.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[51620]: debug 2022-01-31T21:13:17.301+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.302647+0000) 2022-01-31T21:13:17.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[46715]: debug 2022-01-31T21:13:17.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.056651+0000) 2022-01-31T21:13:17.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[46715]: debug 2022-01-31T21:13:17.302+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.303633+0000) 2022-01-31T21:13:17.527 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:17 smithi167 conmon[49112]: debug 2022-01-31T21:13:17.301+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.302322+0000) 2022-01-31T21:13:17.528 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:17 smithi167 conmon[54076]: debug 2022-01-31T21:13:17.301+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.302226+0000) 2022-01-31T21:13:17.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:17 smithi167 conmon[60316]: debug 2022-01-31T21:13:17.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.280704+0000) 2022-01-31T21:13:17.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:17 smithi167 conmon[60316]: debug 2022-01-31T21:13:17.302+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.302859+0000) 2022-01-31T21:13:17.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[35325]: debug 2022-01-31T21:13:17.313+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112532 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:17.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:17 smithi167 conmon[49112]: debug 2022-01-31T21:13:17.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.603899+0000) 2022-01-31T21:13:17.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:17 smithi167 conmon[54076]: debug 2022-01-31T21:13:17.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.529367+0000) 2022-01-31T21:13:17.887 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[41853]: debug 2022-01-31T21:13:17.760+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.761591+0000) 2022-01-31T21:13:17.887 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:17 smithi171 conmon[51620]: debug 2022-01-31T21:13:17.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:17.636670+0000) 2022-01-31T21:13:18.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:18 smithi171 conmon[46715]: debug 2022-01-31T21:13:18.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:18.056836+0000) 2022-01-31T21:13:18.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:18 smithi167 conmon[60316]: debug 2022-01-31T21:13:18.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:18.280837+0000) 2022-01-31T21:13:18.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:18 smithi171 conmon[51620]: debug 2022-01-31T21:13:18.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:18.636867+0000) 2022-01-31T21:13:18.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:18 smithi167 conmon[49112]: debug 2022-01-31T21:13:18.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:18.604055+0000) 2022-01-31T21:13:18.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:18 smithi167 conmon[54076]: debug 2022-01-31T21:13:18.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:18.529495+0000) 2022-01-31T21:13:19.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:18 smithi171 conmon[41853]: debug 2022-01-31T21:13:18.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:18.761768+0000) 2022-01-31T21:13:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:19 smithi171 conmon[46715]: debug 2022-01-31T21:13:19.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:19.057009+0000) 2022-01-31T21:13:19.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:19 smithi167 conmon[60316]: debug 2022-01-31T21:13:19.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:19.280974+0000) 2022-01-31T21:13:19.718 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:19 smithi171 conmon[51620]: debug 2022-01-31T21:13:19.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:19.637055+0000) 2022-01-31T21:13:19.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:19 smithi167 conmon[49112]: debug 2022-01-31T21:13:19.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:19.604231+0000) 2022-01-31T21:13:19.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:19 smithi167 conmon[54076]: debug 2022-01-31T21:13:19.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:19.529699+0000) 2022-01-31T21:13:20.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:19 smithi171 conmon[41853]: debug 2022-01-31T21:13:19.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:19.761857+0000) 2022-01-31T21:13:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:20 smithi171 conmon[46715]: debug 2022-01-31T21:13:20.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:20.057191+0000) 2022-01-31T21:13:20.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:20 smithi167 conmon[60316]: debug 2022-01-31T21:13:20.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:20.281192+0000) 2022-01-31T21:13:20.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:20 smithi171 conmon[51620]: debug 2022-01-31T21:13:20.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:20.637235+0000) 2022-01-31T21:13:20.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:20 smithi167 conmon[49112]: debug 2022-01-31T21:13:20.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:20.604416+0000) 2022-01-31T21:13:20.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:20 smithi167 conmon[54076]: debug 2022-01-31T21:13:20.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:20.529894+0000) 2022-01-31T21:13:21.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:20 smithi171 conmon[41853]: debug 2022-01-31T21:13:20.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:20.761959+0000) 2022-01-31T21:13:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:21 smithi171 conmon[46715]: debug 2022-01-31T21:13:21.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:21.057371+0000) 2022-01-31T21:13:21.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:21 smithi167 conmon[60316]: debug 2022-01-31T21:13:21.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:21.281384+0000) 2022-01-31T21:13:21.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:21 smithi171 conmon[51620]: debug 2022-01-31T21:13:21.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:21.637390+0000) 2022-01-31T21:13:21.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:21 smithi167 conmon[49112]: debug 2022-01-31T21:13:21.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:21.604601+0000) 2022-01-31T21:13:21.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:21 smithi167 conmon[54076]: debug 2022-01-31T21:13:21.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:21.530083+0000) 2022-01-31T21:13:22.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:21 smithi171 conmon[41853]: debug 2022-01-31T21:13:21.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:21.762110+0000) 2022-01-31T21:13:22.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[46715]: debug 2022-01-31T21:13:22.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.057514+0000) 2022-01-31T21:13:22.528 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[49112]: debug 2022-01-31T21:13:22.315+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.317088+0000) 2022-01-31T21:13:22.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[60316]: debug 2022-01-31T21:13:22.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.281598+0000) 2022-01-31T21:13:22.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[60316]: debug 2022-01-31T21:13:22.315+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.316831+0000) 2022-01-31T21:13:22.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[54076]: debug 2022-01-31T21:13:22.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[54076]: 2022-01-31T21:13:22.314+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.315940+0000) 2022-01-31T21:13:22.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[35325]: debug 2022-01-31T21:13:22.327+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112652 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[41853]: debug 2022-01-31T21:13:22.316+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.316761+0000) 2022-01-31T21:13:22.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[46715]: debug 2022-01-31T21:13:22.316+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.317447+0000) 2022-01-31T21:13:22.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[51620]: debug 2022-01-31T21:13:22.316+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.316852+0000) 2022-01-31T21:13:22.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[49112]: debug 2022-01-31T21:13:22.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.604758+0000) 2022-01-31T21:13:22.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:22 smithi167 conmon[54076]: debug 2022-01-31T21:13:22.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.530287+0000) 2022-01-31T21:13:22.887 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[41853]: debug 2022-01-31T21:13:22.762+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.762287+0000) 2022-01-31T21:13:22.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:22 smithi171 conmon[51620]: debug 2022-01-31T21:13:22.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:22.637608+0000) 2022-01-31T21:13:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:23 smithi171 conmon[46715]: debug 2022-01-31T21:13:23.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:23.057671+0000) 2022-01-31T21:13:23.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:23 smithi167 conmon[60316]: debug 2022-01-31T21:13:23.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:23.281754+0000) 2022-01-31T21:13:23.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:23 smithi171 conmon[51620]: debug 2022-01-31T21:13:23.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:23.637777+0000) 2022-01-31T21:13:23.719 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:23 smithi171 conmon[35325]: debug 2022-01-31T21:13:23.643+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:13:23.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:23 smithi167 conmon[49112]: debug 2022-01-31T21:13:23.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:23.604869+0000) 2022-01-31T21:13:23.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:23 smithi167 conmon[54076]: debug 2022-01-31T21:13:23.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:23.530466+0000) 2022-01-31T21:13:24.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:23 smithi171 conmon[41853]: debug 2022-01-31T21:13:23.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:23.762442+0000) 2022-01-31T21:13:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:24 smithi171 conmon[46715]: debug 2022-01-31T21:13:24.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:24.057826+0000) 2022-01-31T21:13:24.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:24 smithi167 conmon[60316]: debug 2022-01-31T21:13:24.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:24.281914+0000) 2022-01-31T21:13:24.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:24 smithi171 conmon[51620]: debug 2022-01-31T21:13:24.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:24.637940+0000) 2022-01-31T21:13:24.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:24 smithi167 conmon[49112]: debug 2022-01-31T21:13:24.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:24.605070+0000) 2022-01-31T21:13:24.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:24 smithi167 conmon[54076]: debug 2022-01-31T21:13:24.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:24.530659+0000) 2022-01-31T21:13:25.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:24 smithi171 conmon[41853]: debug 2022-01-31T21:13:24.762+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:24.762595+0000) 2022-01-31T21:13:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:25 smithi171 conmon[46715]: debug 2022-01-31T21:13:25.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:25.057985+0000) 2022-01-31T21:13:25.530 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:25 smithi167 conmon[60316]: debug 2022-01-31T21:13:25.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:25.282100+0000) 2022-01-31T21:13:25.719 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:25 smithi171 conmon[51620]: debug 2022-01-31T21:13:25.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:25.638070+0000) 2022-01-31T21:13:25.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:25 smithi167 conmon[49112]: debug 2022-01-31T21:13:25.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:25.605247+0000) 2022-01-31T21:13:25.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:25 smithi167 conmon[54076]: debug 2022-01-31T21:13:25.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:25.530847+0000) 2022-01-31T21:13:26.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:25 smithi171 conmon[41853]: debug 2022-01-31T21:13:25.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:25.762776+0000) 2022-01-31T21:13:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:26 smithi171 conmon[46715]: debug 2022-01-31T21:13:26.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:26.058169+0000) 2022-01-31T21:13:26.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:26 smithi167 conmon[60316]: debug 2022-01-31T21:13:26.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:26.282259+0000) 2022-01-31T21:13:26.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:26 smithi171 conmon[51620]: debug 2022-01-31T21:13:26.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:26.638205+0000) 2022-01-31T21:13:26.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:26 smithi167 conmon[49112]: debug 2022-01-31T21:13:26.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:26.605460+0000) 2022-01-31T21:13:26.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:26 smithi167 conmon[54076]: debug 2022-01-31T21:13:26.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:26.531063+0000) 2022-01-31T21:13:27.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:26 smithi171 conmon[41853]: debug 2022-01-31T21:13:26.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:26.762934+0000) 2022-01-31T21:13:27.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[46715]: debug 2022-01-31T21:13:27.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.058333+0000) 2022-01-31T21:13:27.329 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[41853]: debug 2022-01-31T21:13:27.328+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.329849+0000) 2022-01-31T21:13:27.530 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:27 smithi167 conmon[49112]: debug 2022-01-31T21:13:27.330+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.331728+0000) 2022-01-31T21:13:27.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:27 smithi167 conmon[54076]: debug 2022-01-31T21:13:27.329+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.331252+0000) 2022-01-31T21:13:27.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:27 smithi167 conmon[60316]: debug 2022-01-31T21:13:27.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.282444+0000) 2022-01-31T21:13:27.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:27 smithi167 conmon[60316]: debug 2022-01-31T21:13:27.329+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.330621+0000) 2022-01-31T21:13:27.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[35325]: debug 2022-01-31T21:13:27.339+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112762 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:27.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[46715]: debug 2022-01-31T21:13:27.329+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.330852+0000) 2022-01-31T21:13:27.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[51620]: debug 2022-01-31T21:13:27.328+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.330169+0000) 2022-01-31T21:13:27.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:27 smithi167 conmon[49112]: debug 2022-01-31T21:13:27.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.605671+0000) 2022-01-31T21:13:27.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:27 smithi167 conmon[54076]: debug 2022-01-31T21:13:27.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.531191+0000) 2022-01-31T21:13:27.889 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[51620]: debug 2022-01-31T21:13:27.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.638314+0000) 2022-01-31T21:13:27.890 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:27 smithi171 conmon[41853]: debug 2022-01-31T21:13:27.762+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:27.763114+0000) 2022-01-31T21:13:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:28 smithi171 conmon[46715]: debug 2022-01-31T21:13:28.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:28.058485+0000) 2022-01-31T21:13:28.529 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:28 smithi167 conmon[60316]: debug 2022-01-31T21:13:28.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:28.282649+0000) 2022-01-31T21:13:28.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:28 smithi171 conmon[51620]: debug 2022-01-31T21:13:28.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:28.638487+0000) 2022-01-31T21:13:28.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:28 smithi167 conmon[49112]: debug 2022-01-31T21:13:28.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:28.605777+0000) 2022-01-31T21:13:28.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:28 smithi167 conmon[54076]: debug 2022-01-31T21:13:28.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:28.531357+0000) 2022-01-31T21:13:29.057 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:28 smithi171 conmon[41853]: debug 2022-01-31T21:13:28.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:28.763252+0000) 2022-01-31T21:13:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:29 smithi171 conmon[46715]: debug 2022-01-31T21:13:29.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:29.058664+0000) 2022-01-31T21:13:29.530 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:29 smithi167 conmon[60316]: debug 2022-01-31T21:13:29.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:29.282794+0000) 2022-01-31T21:13:29.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:29 smithi171 conmon[51620]: debug 2022-01-31T21:13:29.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:29.638642+0000) 2022-01-31T21:13:29.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:29 smithi167 conmon[49112]: debug 2022-01-31T21:13:29.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:29.605932+0000) 2022-01-31T21:13:29.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:29 smithi167 conmon[54076]: debug 2022-01-31T21:13:29.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:29.531566+0000) 2022-01-31T21:13:30.057 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:29 smithi171 conmon[41853]: debug 2022-01-31T21:13:29.761+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:29.763427+0000) 2022-01-31T21:13:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:30 smithi171 conmon[46715]: debug 2022-01-31T21:13:30.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:30.058856+0000) 2022-01-31T21:13:30.530 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:30 smithi167 conmon[60316]: debug 2022-01-31T21:13:30.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:30.283013+0000) 2022-01-31T21:13:30.688 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:30 smithi171 conmon[51620]: debug 2022-01-31T21:13:30.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:30.638796+0000) 2022-01-31T21:13:30.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:30 smithi167 conmon[49112]: debug 2022-01-31T21:13:30.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:30.606114+0000) 2022-01-31T21:13:30.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:30 smithi167 conmon[54076]: debug 2022-01-31T21:13:30.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:30.531721+0000) 2022-01-31T21:13:31.057 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:30 smithi171 conmon[41853]: debug 2022-01-31T21:13:30.762+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:30.763629+0000) 2022-01-31T21:13:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:31 smithi171 conmon[46715]: debug 2022-01-31T21:13:31.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:31.059009+0000) 2022-01-31T21:13:31.530 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:31 smithi167 conmon[60316]: debug 2022-01-31T21:13:31.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:31.283193+0000) 2022-01-31T21:13:31.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:31 smithi171 conmon[51620]: debug 2022-01-31T21:13:31.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:31.638897+0000) 2022-01-31T21:13:31.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:31 smithi167 conmon[49112]: debug 2022-01-31T21:13:31.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:31.606303+0000) 2022-01-31T21:13:31.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:31 smithi167 conmon[54076]: debug 2022-01-31T21:13:31.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:31.531876+0000) 2022-01-31T21:13:32.057 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:31 smithi171 conmon[41853]: debug 2022-01-31T21:13:31.762+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:31.763795+0000) 2022-01-31T21:13:32.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[46715]: debug 2022-01-31T21:13:32.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.059219+0000) 2022-01-31T21:13:32.530 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:32 smithi167 conmon[49112]: debug 2022-01-31T21:13:32.342+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.344183+0000) 2022-01-31T21:13:32.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:32 smithi167 conmon[54076]: debug 2022-01-31T21:13:32.343+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.344873+0000) 2022-01-31T21:13:32.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:32 smithi167 conmon[60316]: debug 2022-01-31T21:13:32.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.283357+0000) 2022-01-31T21:13:32.532 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:32 smithi167 conmon[60316]: debug 2022-01-31T21:13:32.342+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.343546+0000) 2022-01-31T21:13:32.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[35325]: debug 2022-01-31T21:13:32.353+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112873 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[41853]: debug 2022-01-31T21:13:32.341+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.343704+0000) 2022-01-31T21:13:32.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[46715]: debug 2022-01-31T21:13:32.341+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.343186+0000) 2022-01-31T21:13:32.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[51620]: debug 2022-01-31T21:13:32.340+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.342693+0000) 2022-01-31T21:13:32.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:32 smithi167 conmon[49112]: debug 2022-01-31T21:13:32.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.606493+0000) 2022-01-31T21:13:32.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:32 smithi167 conmon[54076]: debug 2022-01-31T21:13:32.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.532072+0000) 2022-01-31T21:13:32.889 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[51620]: debug 2022-01-31T21:13:32.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.639071+0000) 2022-01-31T21:13:32.889 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:32 smithi171 conmon[41853]: debug 2022-01-31T21:13:32.763+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:32.763991+0000) 2022-01-31T21:13:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:33 smithi171 conmon[46715]: debug 2022-01-31T21:13:33.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:33.059437+0000) 2022-01-31T21:13:33.530 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:33 smithi167 conmon[60316]: debug 2022-01-31T21:13:33.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:33.283525+0000) 2022-01-31T21:13:33.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:33 smithi171 conmon[51620]: debug 2022-01-31T21:13:33.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:33.639231+0000) 2022-01-31T21:13:33.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:33 smithi167 conmon[49112]: debug 2022-01-31T21:13:33.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:33.606618+0000) 2022-01-31T21:13:33.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:33 smithi167 conmon[54076]: debug 2022-01-31T21:13:33.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:33.532203+0000) 2022-01-31T21:13:34.057 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:33 smithi171 conmon[41853]: debug 2022-01-31T21:13:33.762+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:33.764104+0000) 2022-01-31T21:13:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:34 smithi171 conmon[46715]: debug 2022-01-31T21:13:34.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:34.059495+0000) 2022-01-31T21:13:34.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:34 smithi167 conmon[60316]: debug 2022-01-31T21:13:34.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:34.283678+0000) 2022-01-31T21:13:34.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:34 smithi171 conmon[51620]: debug 2022-01-31T21:13:34.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:34.639433+0000) 2022-01-31T21:13:34.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:34 smithi167 conmon[49112]: debug 2022-01-31T21:13:34.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:34.606757+0000) 2022-01-31T21:13:34.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:34 smithi167 conmon[54076]: debug 2022-01-31T21:13:34.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:34.532369+0000) 2022-01-31T21:13:35.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:34 smithi171 conmon[41853]: debug 2022-01-31T21:13:34.763+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:34.764239+0000) 2022-01-31T21:13:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:35 smithi171 conmon[46715]: debug 2022-01-31T21:13:35.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:35.059618+0000) 2022-01-31T21:13:35.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:35 smithi167 conmon[60316]: debug 2022-01-31T21:13:35.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:35.283869+0000) 2022-01-31T21:13:35.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:35 smithi171 conmon[51620]: debug 2022-01-31T21:13:35.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:35.639615+0000) 2022-01-31T21:13:35.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:35 smithi167 conmon[49112]: debug 2022-01-31T21:13:35.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:35.606912+0000) 2022-01-31T21:13:35.978 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:35 smithi167 conmon[54076]: debug 2022-01-31T21:13:35.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:35.532572+0000) 2022-01-31T21:13:36.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:35 smithi171 conmon[41853]: debug 2022-01-31T21:13:35.763+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:35.764413+0000) 2022-01-31T21:13:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:36 smithi171 conmon[46715]: debug 2022-01-31T21:13:36.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:36.059784+0000) 2022-01-31T21:13:36.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:36 smithi167 conmon[60316]: debug 2022-01-31T21:13:36.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:36.284066+0000) 2022-01-31T21:13:36.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:36 smithi171 conmon[51620]: debug 2022-01-31T21:13:36.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:36.639727+0000) 2022-01-31T21:13:36.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:36 smithi167 conmon[54076]: debug 2022-01-31T21:13:36.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:36.532715+0000) 2022-01-31T21:13:36.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:36 smithi167 conmon[49112]: debug 2022-01-31T21:13:36.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:36.607108+0000) 2022-01-31T21:13:37.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:36 smithi171 conmon[41853]: debug 2022-01-31T21:13:36.763+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:36.764632+0000) 2022-01-31T21:13:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[46715]: debug 2022-01-31T21:13:37.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.059969+0000) 2022-01-31T21:13:37.531 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:37 smithi167 conmon[49112]: debug 2022-01-31T21:13:37.357+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.358800+0000) 2022-01-31T21:13:37.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:37 smithi167 conmon[54076]: debug 2022-01-31T21:13:37.357+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.358977+0000) 2022-01-31T21:13:37.532 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:37 smithi167 conmon[60316]: debug 2022-01-31T21:13:37.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.284238+0000) 2022-01-31T21:13:37.532 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:37 smithi167 conmon[60316]: debug 2022-01-31T21:13:37.357+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.359170+0000) 2022-01-31T21:13:37.638 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[35325]: debug 2022-01-31T21:13:37.368+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 112985 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:37.639 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[41853]: debug 2022-01-31T21:13:37.357+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.358315+0000) 2022-01-31T21:13:37.641 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[46715]: debug 2022-01-31T21:13:37.357+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.358236+0000) 2022-01-31T21:13:37.642 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[51620]: debug 2022-01-31T21:13:37.356+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.357516+0000) 2022-01-31T21:13:37.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:37 smithi167 conmon[49112]: debug 2022-01-31T21:13:37.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.607279+0000) 2022-01-31T21:13:37.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:37 smithi167 conmon[54076]: debug 2022-01-31T21:13:37.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.532862+0000) 2022-01-31T21:13:37.890 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[41853]: debug 2022-01-31T21:13:37.764+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.764843+0000) 2022-01-31T21:13:37.890 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:37 smithi171 conmon[51620]: debug 2022-01-31T21:13:37.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:37.639954+0000) 2022-01-31T21:13:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:38 smithi171 conmon[46715]: debug 2022-01-31T21:13:38.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:38.060149+0000) 2022-01-31T21:13:38.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:38 smithi167 conmon[60316]: debug 2022-01-31T21:13:38.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:38.284447+0000) 2022-01-31T21:13:38.722 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:38 smithi171 conmon[35325]: debug 2022-01-31T21:13:38.643+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:13:38.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:38 smithi171 conmon[51620]: debug 2022-01-31T21:13:38.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:38.640091+0000) 2022-01-31T21:13:38.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:38 smithi167 conmon[49112]: debug 2022-01-31T21:13:38.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:38.607438+0000) 2022-01-31T21:13:38.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:38 smithi167 conmon[54076]: debug 2022-01-31T21:13:38.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:38.533036+0000) 2022-01-31T21:13:39.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:38 smithi171 conmon[41853]: debug 2022-01-31T21:13:38.764+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:38.764993+0000) 2022-01-31T21:13:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:39 smithi171 conmon[46715]: debug 2022-01-31T21:13:39.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:39.060357+0000) 2022-01-31T21:13:39.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:39 smithi167 conmon[60316]: debug 2022-01-31T21:13:39.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:39.284603+0000) 2022-01-31T21:13:39.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:39 smithi171 conmon[51620]: debug 2022-01-31T21:13:39.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:39.640271+0000) 2022-01-31T21:13:39.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:39 smithi167 conmon[54076]: debug 2022-01-31T21:13:39.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:39.533180+0000) 2022-01-31T21:13:39.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:39 smithi167 conmon[49112]: debug 2022-01-31T21:13:39.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:39.607580+0000) 2022-01-31T21:13:40.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:39 smithi171 conmon[41853]: debug 2022-01-31T21:13:39.764+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:39.765194+0000) 2022-01-31T21:13:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:40 smithi171 conmon[46715]: debug 2022-01-31T21:13:40.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:40.060512+0000) 2022-01-31T21:13:40.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:40 smithi167 conmon[60316]: debug 2022-01-31T21:13:40.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:40.284772+0000) 2022-01-31T21:13:40.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:40 smithi171 conmon[51620]: debug 2022-01-31T21:13:40.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:40.640447+0000) 2022-01-31T21:13:40.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:40 smithi167 conmon[49112]: debug 2022-01-31T21:13:40.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:40.607742+0000) 2022-01-31T21:13:40.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:40 smithi167 conmon[54076]: debug 2022-01-31T21:13:40.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:40.533342+0000) 2022-01-31T21:13:41.059 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:40 smithi171 conmon[41853]: debug 2022-01-31T21:13:40.764+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:40.765335+0000) 2022-01-31T21:13:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:41 smithi171 conmon[46715]: debug 2022-01-31T21:13:41.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:41.060701+0000) 2022-01-31T21:13:41.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:41 smithi167 conmon[60316]: debug 2022-01-31T21:13:41.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:41.284931+0000) 2022-01-31T21:13:41.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:41 smithi171 conmon[51620]: debug 2022-01-31T21:13:41.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:41.640648+0000) 2022-01-31T21:13:41.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:41 smithi167 conmon[49112]: debug 2022-01-31T21:13:41.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:41.607909+0000) 2022-01-31T21:13:41.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:41 smithi167 conmon[54076]: debug 2022-01-31T21:13:41.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:41.533513+0000) 2022-01-31T21:13:42.059 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:41 smithi171 conmon[41853]: debug 2022-01-31T21:13:41.764+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:41.765521+0000) 2022-01-31T21:13:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[46715]: debug 2022-01-31T21:13:42.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.060887+0000) 2022-01-31T21:13:42.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:42 smithi167 conmon[54076]: debug 2022-01-31T21:13:42.370+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.371829+0000) 2022-01-31T21:13:42.532 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:42 smithi167 conmon[49112]: debug 2022-01-31T21:13:42.371+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.372877+0000) 2022-01-31T21:13:42.533 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:42 smithi167 conmon[60316]: debug 2022-01-31T21:13:42.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.285091+0000) 2022-01-31T21:13:42.533 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:42 smithi167 conmon[60316]: debug 2022-01-31T21:13:42.371+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.372841+0000) 2022-01-31T21:13:42.639 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[35325]: debug 2022-01-31T21:13:42.382+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113094 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:42.639 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[41853]: debug 2022-01-31T21:13:42.372+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.373293+0000) 2022-01-31T21:13:42.640 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[46715]: debug 2022-01-31T21:13:42.371+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.372252+0000) 2022-01-31T21:13:42.640 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[51620]: debug 2022-01-31T21:13:42.370+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.371404+0000) 2022-01-31T21:13:42.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:42 smithi167 conmon[54076]: debug 2022-01-31T21:13:42.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.533669+0000) 2022-01-31T21:13:42.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:42 smithi167 conmon[49112]: debug 2022-01-31T21:13:42.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.608122+0000) 2022-01-31T21:13:42.891 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[41853]: debug 2022-01-31T21:13:42.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.765692+0000) 2022-01-31T21:13:42.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:42 smithi171 conmon[51620]: debug 2022-01-31T21:13:42.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:42.640806+0000) 2022-01-31T21:13:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:43 smithi171 conmon[46715]: debug 2022-01-31T21:13:43.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:43.061064+0000) 2022-01-31T21:13:43.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:43 smithi167 conmon[60316]: debug 2022-01-31T21:13:43.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:43.285290+0000) 2022-01-31T21:13:43.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:43 smithi171 conmon[51620]: debug 2022-01-31T21:13:43.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:43.640890+0000) 2022-01-31T21:13:43.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:43 smithi167 conmon[49112]: debug 2022-01-31T21:13:43.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:43.608340+0000) 2022-01-31T21:13:43.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:43 smithi167 conmon[54076]: debug 2022-01-31T21:13:43.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:43.533773+0000) 2022-01-31T21:13:44.059 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:43 smithi171 conmon[41853]: debug 2022-01-31T21:13:43.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:43.765828+0000) 2022-01-31T21:13:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:44 smithi171 conmon[46715]: debug 2022-01-31T21:13:44.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:44.061199+0000) 2022-01-31T21:13:44.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:44 smithi167 conmon[60316]: debug 2022-01-31T21:13:44.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:44.285465+0000) 2022-01-31T21:13:44.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:44 smithi171 conmon[51620]: debug 2022-01-31T21:13:44.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:44.641057+0000) 2022-01-31T21:13:44.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:44 smithi167 conmon[49112]: debug 2022-01-31T21:13:44.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:44.608592+0000) 2022-01-31T21:13:44.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:44 smithi167 conmon[54076]: debug 2022-01-31T21:13:44.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:44.533944+0000) 2022-01-31T21:13:45.059 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:44 smithi171 conmon[41853]: debug 2022-01-31T21:13:44.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:44.765961+0000) 2022-01-31T21:13:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:45 smithi171 conmon[46715]: debug 2022-01-31T21:13:45.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:45.061374+0000) 2022-01-31T21:13:45.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:45 smithi167 conmon[60316]: debug 2022-01-31T21:13:45.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:45.285640+0000) 2022-01-31T21:13:45.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:45 smithi171 conmon[51620]: debug 2022-01-31T21:13:45.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:45.641291+0000) 2022-01-31T21:13:45.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:45 smithi167 conmon[49112]: debug 2022-01-31T21:13:45.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:45.608780+0000) 2022-01-31T21:13:45.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:45 smithi167 conmon[54076]: debug 2022-01-31T21:13:45.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:45.534108+0000) 2022-01-31T21:13:46.060 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:45 smithi171 conmon[41853]: debug 2022-01-31T21:13:45.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:45.766147+0000) 2022-01-31T21:13:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:46 smithi171 conmon[46715]: debug 2022-01-31T21:13:46.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:46.061606+0000) 2022-01-31T21:13:46.518 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:46 smithi167 conmon[60316]: debug 2022-01-31T21:13:46.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:46.285769+0000) 2022-01-31T21:13:46.723 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:46 smithi171 conmon[51620]: debug 2022-01-31T21:13:46.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:46.641462+0000) 2022-01-31T21:13:46.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:46 smithi167 conmon[49112]: debug 2022-01-31T21:13:46.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:46.608960+0000) 2022-01-31T21:13:46.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:46 smithi167 conmon[54076]: debug 2022-01-31T21:13:46.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:46.534329+0000) 2022-01-31T21:13:47.060 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:46 smithi171 conmon[41853]: debug 2022-01-31T21:13:46.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:46.766347+0000) 2022-01-31T21:13:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[46715]: debug 2022-01-31T21:13:47.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.061809+0000) 2022-01-31T21:13:47.518 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:47 smithi167 conmon[49112]: debug 2022-01-31T21:13:47.410+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.411415+0000) 2022-01-31T21:13:47.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:47 smithi167 conmon[54076]: debug 2022-01-31T21:13:47.385+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.386018+0000) 2022-01-31T21:13:47.519 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:47 smithi167 conmon[60316]: debug 2022-01-31T21:13:47.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.285937+0000) 2022-01-31T21:13:47.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:47 smithi167 conmon[60316]: debug 2022-01-31T21:13:47.385+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.385516+0000) 2022-01-31T21:13:47.640 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[35325]: debug 2022-01-31T21:13:47.400+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113206 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:47.640 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[41853]: debug 2022-01-31T21:13:47.386+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.387594+0000) 2022-01-31T21:13:47.641 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[46715]: debug 2022-01-31T21:13:47.385+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.386712+0000) 2022-01-31T21:13:47.641 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[51620]: debug 2022-01-31T21:13:47.385+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.385877+0000) 2022-01-31T21:13:47.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:47 smithi167 conmon[54076]: debug 2022-01-31T21:13:47.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.534578+0000) 2022-01-31T21:13:47.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:47 smithi167 conmon[49112]: debug 2022-01-31T21:13:47.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.609169+0000) 2022-01-31T21:13:47.892 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[41853]: debug 2022-01-31T21:13:47.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.766545+0000) 2022-01-31T21:13:47.892 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:47 smithi171 conmon[51620]: debug 2022-01-31T21:13:47.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:47.641652+0000) 2022-01-31T21:13:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:48 smithi171 conmon[46715]: debug 2022-01-31T21:13:48.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:48.062016+0000) 2022-01-31T21:13:48.522 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:48 smithi167 conmon[60316]: debug 2022-01-31T21:13:48.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:48.286088+0000) 2022-01-31T21:13:48.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:48 smithi171 conmon[51620]: debug 2022-01-31T21:13:48.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:48.641833+0000) 2022-01-31T21:13:48.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:48 smithi167 conmon[54076]: debug 2022-01-31T21:13:48.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:48.534758+0000) 2022-01-31T21:13:48.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:48 smithi167 conmon[49112]: debug 2022-01-31T21:13:48.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:48.609323+0000) 2022-01-31T21:13:49.060 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:48 smithi171 conmon[41853]: debug 2022-01-31T21:13:48.765+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:48.766673+0000) 2022-01-31T21:13:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:49 smithi171 conmon[46715]: debug 2022-01-31T21:13:49.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:49.062214+0000) 2022-01-31T21:13:49.525 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:49 smithi167 conmon[60316]: debug 2022-01-31T21:13:49.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:49.286230+0000) 2022-01-31T21:13:49.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:49 smithi171 conmon[51620]: debug 2022-01-31T21:13:49.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:49.642026+0000) 2022-01-31T21:13:49.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:49 smithi167 conmon[54076]: debug 2022-01-31T21:13:49.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:49.534920+0000) 2022-01-31T21:13:49.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:49 smithi167 conmon[49112]: debug 2022-01-31T21:13:49.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:49.609530+0000) 2022-01-31T21:13:50.060 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:49 smithi171 conmon[41853]: debug 2022-01-31T21:13:49.766+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:49.766832+0000) 2022-01-31T21:13:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:50 smithi171 conmon[46715]: debug 2022-01-31T21:13:50.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:50.062427+0000) 2022-01-31T21:13:50.525 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:50 smithi167 conmon[60316]: debug 2022-01-31T21:13:50.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:50.286448+0000) 2022-01-31T21:13:50.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:50 smithi171 conmon[51620]: debug 2022-01-31T21:13:50.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:50.642183+0000) 2022-01-31T21:13:50.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:50 smithi167 conmon[54076]: debug 2022-01-31T21:13:50.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:50.535071+0000) 2022-01-31T21:13:50.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:50 smithi167 conmon[49112]: debug 2022-01-31T21:13:50.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:50.609717+0000) 2022-01-31T21:13:51.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:50 smithi171 conmon[41853]: debug 2022-01-31T21:13:50.766+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:50.766990+0000) 2022-01-31T21:13:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:51 smithi171 conmon[46715]: debug 2022-01-31T21:13:51.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:51.062585+0000) 2022-01-31T21:13:51.525 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:51 smithi167 conmon[60316]: debug 2022-01-31T21:13:51.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:51.286649+0000) 2022-01-31T21:13:51.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:51 smithi171 conmon[51620]: debug 2022-01-31T21:13:51.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:51.642376+0000) 2022-01-31T21:13:51.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:51 smithi167 conmon[54076]: debug 2022-01-31T21:13:51.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:51.535272+0000) 2022-01-31T21:13:51.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:51 smithi167 conmon[49112]: debug 2022-01-31T21:13:51.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:51.609908+0000) 2022-01-31T21:13:52.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:51 smithi171 conmon[41853]: debug 2022-01-31T21:13:51.766+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:51.767151+0000) 2022-01-31T21:13:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[46715]: debug 2022-01-31T21:13:52.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.062794+0000) 2022-01-31T21:13:52.525 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:52 smithi167 conmon[49112]: debug 2022-01-31T21:13:52.405+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.405756+0000) 2022-01-31T21:13:52.526 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:52 smithi167 conmon[54076]: debug 2022-01-31T21:13:52.403+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.403572+0000) 2022-01-31T21:13:52.526 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:52 smithi167 conmon[60316]: debug 2022-01-31T21:13:52.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.286837+0000) 2022-01-31T21:13:52.527 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:52 smithi167 conmon[60316]: debug 2022-01-31T21:13:52.402+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.403336+0000) 2022-01-31T21:13:52.724 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[35325]: debug 2022-01-31T21:13:52.414+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113316 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:52.725 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[41853]: debug 2022-01-31T21:13:52.403+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.404075+0000) 2022-01-31T21:13:52.725 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[46715]: debug 2022-01-31T21:13:52.404+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.404835+0000) 2022-01-31T21:13:52.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[51620]: debug 2022-01-31T21:13:52.402+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.403111+0000) 2022-01-31T21:13:52.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[51620]: debug 2022-01-31T21:13:52.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.642579+0000) 2022-01-31T21:13:52.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:52 smithi167 conmon[49112]: debug 2022-01-31T21:13:52.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.610074+0000) 2022-01-31T21:13:52.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:52 smithi167 conmon[54076]: debug 2022-01-31T21:13:52.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.535454+0000) 2022-01-31T21:13:53.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:52 smithi171 conmon[41853]: debug 2022-01-31T21:13:52.766+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:52.767338+0000) 2022-01-31T21:13:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:53 smithi171 conmon[46715]: debug 2022-01-31T21:13:53.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:53.062944+0000) 2022-01-31T21:13:53.528 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:53 smithi167 conmon[60316]: debug 2022-01-31T21:13:53.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:53.287022+0000) 2022-01-31T21:13:53.725 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:53 smithi171 conmon[35325]: debug 2022-01-31T21:13:53.644+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:13:53.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:53 smithi171 conmon[51620]: debug 2022-01-31T21:13:53.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:53.642725+0000) 2022-01-31T21:13:53.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:53 smithi167 conmon[49112]: debug 2022-01-31T21:13:53.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:53.610199+0000) 2022-01-31T21:13:53.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:53 smithi167 conmon[54076]: debug 2022-01-31T21:13:53.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:53.535552+0000) 2022-01-31T21:13:54.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:53 smithi171 conmon[41853]: debug 2022-01-31T21:13:53.766+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:53.767496+0000) 2022-01-31T21:13:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:54 smithi171 conmon[46715]: debug 2022-01-31T21:13:54.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:54.063061+0000) 2022-01-31T21:13:54.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:54 smithi167 conmon[60316]: debug 2022-01-31T21:13:54.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:54.287186+0000) 2022-01-31T21:13:54.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:54 smithi171 conmon[51620]: debug 2022-01-31T21:13:54.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:54.642919+0000) 2022-01-31T21:13:54.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:54 smithi167 conmon[49112]: debug 2022-01-31T21:13:54.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:54.610382+0000) 2022-01-31T21:13:54.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:54 smithi167 conmon[54076]: debug 2022-01-31T21:13:54.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:54.535660+0000) 2022-01-31T21:13:55.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:54 smithi171 conmon[41853]: debug 2022-01-31T21:13:54.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:54.767690+0000) 2022-01-31T21:13:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:55 smithi171 conmon[46715]: debug 2022-01-31T21:13:55.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:55.063219+0000) 2022-01-31T21:13:55.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:55 smithi167 conmon[60316]: debug 2022-01-31T21:13:55.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:55.287373+0000) 2022-01-31T21:13:55.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:55 smithi171 conmon[51620]: debug 2022-01-31T21:13:55.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:55.643077+0000) 2022-01-31T21:13:55.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:55 smithi167 conmon[49112]: debug 2022-01-31T21:13:55.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:55.610556+0000) 2022-01-31T21:13:55.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:55 smithi167 conmon[54076]: debug 2022-01-31T21:13:55.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:55.535791+0000) 2022-01-31T21:13:56.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:55 smithi171 conmon[41853]: debug 2022-01-31T21:13:55.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:55.767874+0000) 2022-01-31T21:13:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:56 smithi171 conmon[46715]: debug 2022-01-31T21:13:56.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:56.063395+0000) 2022-01-31T21:13:56.532 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:56 smithi167 conmon[60316]: debug 2022-01-31T21:13:56.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:56.287549+0000) 2022-01-31T21:13:56.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:56 smithi171 conmon[51620]: debug 2022-01-31T21:13:56.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:56.643217+0000) 2022-01-31T21:13:56.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:56 smithi167 conmon[49112]: debug 2022-01-31T21:13:56.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:56.610773+0000) 2022-01-31T21:13:56.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:56 smithi167 conmon[54076]: debug 2022-01-31T21:13:56.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:56.535972+0000) 2022-01-31T21:13:57.062 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:56 smithi171 conmon[41853]: debug 2022-01-31T21:13:56.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:56.768029+0000) 2022-01-31T21:13:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[46715]: debug 2022-01-31T21:13:57.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.063555+0000) 2022-01-31T21:13:57.532 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:57 smithi167 conmon[49112]: debug 2022-01-31T21:13:57.419+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.419626+0000) 2022-01-31T21:13:57.533 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:57 smithi167 conmon[54076]: debug 2022-01-31T21:13:57.418+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.418596+0000) 2022-01-31T21:13:57.533 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:57 smithi167 conmon[60316]: debug 2022-01-31T21:13:57.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.287765+0000) 2022-01-31T21:13:57.533 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:57 smithi167 conmon[60316]: debug 2022-01-31T21:13:57.418+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.418693+0000) 2022-01-31T21:13:57.725 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[35325]: debug 2022-01-31T21:13:57.428+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113426 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:13:57.726 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[41853]: debug 2022-01-31T21:13:57.417+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.418277+0000) 2022-01-31T21:13:57.726 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[46715]: debug 2022-01-31T21:13:57.418+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.419200+0000) 2022-01-31T21:13:57.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[51620]: debug 2022-01-31T21:13:57.417+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.418136+0000) 2022-01-31T21:13:57.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[51620]: debug 2022-01-31T21:13:57.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.643427+0000) 2022-01-31T21:13:57.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:57 smithi167 conmon[49112]: debug 2022-01-31T21:13:57.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.610940+0000) 2022-01-31T21:13:57.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:57 smithi167 conmon[54076]: debug 2022-01-31T21:13:57.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.536071+0000) 2022-01-31T21:13:58.062 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:57 smithi171 conmon[41853]: debug 2022-01-31T21:13:57.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:57.768129+0000) 2022-01-31T21:13:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:58 smithi171 conmon[46715]: debug 2022-01-31T21:13:58.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:58.063743+0000) 2022-01-31T21:13:58.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:58 smithi167 conmon[60316]: debug 2022-01-31T21:13:58.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:58.287979+0000) 2022-01-31T21:13:58.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:58 smithi171 conmon[51620]: debug 2022-01-31T21:13:58.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:58.643600+0000) 2022-01-31T21:13:58.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:58 smithi167 conmon[49112]: debug 2022-01-31T21:13:58.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:58.611071+0000) 2022-01-31T21:13:58.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:58 smithi167 conmon[54076]: debug 2022-01-31T21:13:58.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:58.536251+0000) 2022-01-31T21:13:59.062 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:58 smithi171 conmon[41853]: debug 2022-01-31T21:13:58.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:58.768284+0000) 2022-01-31T21:13:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:13:59 smithi171 conmon[46715]: debug 2022-01-31T21:13:59.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:59.063931+0000) 2022-01-31T21:13:59.534 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:13:59 smithi167 conmon[60316]: debug 2022-01-31T21:13:59.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:59.288137+0000) 2022-01-31T21:13:59.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:13:59 smithi171 conmon[51620]: debug 2022-01-31T21:13:59.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:59.643844+0000) 2022-01-31T21:13:59.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:13:59 smithi167 conmon[49112]: debug 2022-01-31T21:13:59.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:59.611200+0000) 2022-01-31T21:13:59.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:13:59 smithi167 conmon[54076]: debug 2022-01-31T21:13:59.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:59.536403+0000) 2022-01-31T21:14:00.062 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:13:59 smithi171 conmon[41853]: debug 2022-01-31T21:13:59.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:13:59.768487+0000) 2022-01-31T21:14:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:00 smithi171 conmon[46715]: debug 2022-01-31T21:14:00.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:00.064083+0000) 2022-01-31T21:14:00.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:00 smithi167 conmon[60316]: debug 2022-01-31T21:14:00.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:00.288322+0000) 2022-01-31T21:14:00.725 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:00 smithi171 conmon[51620]: debug 2022-01-31T21:14:00.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:00.644003+0000) 2022-01-31T21:14:00.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:00 smithi167 conmon[49112]: debug 2022-01-31T21:14:00.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:00.611401+0000) 2022-01-31T21:14:00.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:00 smithi167 conmon[54076]: debug 2022-01-31T21:14:00.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:00.536629+0000) 2022-01-31T21:14:01.062 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:00 smithi171 conmon[41853]: debug 2022-01-31T21:14:00.767+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:00.768630+0000) 2022-01-31T21:14:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:01 smithi171 conmon[46715]: debug 2022-01-31T21:14:01.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:01.064226+0000) 2022-01-31T21:14:01.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:01 smithi167 conmon[60316]: debug 2022-01-31T21:14:01.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:01.288541+0000) 2022-01-31T21:14:01.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:01 smithi171 conmon[51620]: debug 2022-01-31T21:14:01.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:01.644157+0000) 2022-01-31T21:14:01.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:01 smithi167 conmon[54076]: debug 2022-01-31T21:14:01.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:01.536844+0000) 2022-01-31T21:14:01.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:01 smithi167 conmon[49112]: debug 2022-01-31T21:14:01.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:01.611535+0000) 2022-01-31T21:14:02.062 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:01 smithi171 conmon[41853]: debug 2022-01-31T21:14:01.768+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:01.768796+0000) 2022-01-31T21:14:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[46715]: debug 2022-01-31T21:14:02.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.064406+0000) 2022-01-31T21:14:02.535 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:02 smithi167 conmon[49112]: debug 2022-01-31T21:14:02.432+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.432620+0000) 2022-01-31T21:14:02.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:02 smithi167 conmon[54076]: debug 2022-01-31T21:14:02.432+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.432508+0000) 2022-01-31T21:14:02.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:02 smithi167 conmon[60316]: debug 2022-01-31T21:14:02.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.288738+0000) 2022-01-31T21:14:02.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:02 smithi167 conmon[60316]: debug 2022-01-31T21:14:02.432+0000 7f4305d8b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.433440+0000) 2022-01-31T21:14:02.726 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[35325]: debug 2022-01-31T21:14:02.443+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113536 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:02.727 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[41853]: debug 2022-01-31T21:14:02.430+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.431700+0000) 2022-01-31T21:14:02.727 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[46715]: debug 2022-01-31T21:14:02.431+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.432673+0000) 2022-01-31T21:14:02.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[51620]: debug 2022-01-31T21:14:02.431+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.432229+0000) 2022-01-31T21:14:02.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[51620]: debug 2022-01-31T21:14:02.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.644340+0000) 2022-01-31T21:14:02.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:02 smithi167 conmon[49112]: debug 2022-01-31T21:14:02.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.611691+0000) 2022-01-31T21:14:02.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:02 smithi167 conmon[54076]: debug 2022-01-31T21:14:02.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.537043+0000) 2022-01-31T21:14:03.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:02 smithi171 conmon[41853]: debug 2022-01-31T21:14:02.768+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:02.769004+0000) 2022-01-31T21:14:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:03 smithi171 conmon[46715]: debug 2022-01-31T21:14:03.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:03.064594+0000) 2022-01-31T21:14:03.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:03 smithi167 conmon[60316]: debug 2022-01-31T21:14:03.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:03.288924+0000) 2022-01-31T21:14:03.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:03 smithi171 conmon[51620]: debug 2022-01-31T21:14:03.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:03.644493+0000) 2022-01-31T21:14:03.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:03 smithi167 conmon[49112]: debug 2022-01-31T21:14:03.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:03.611862+0000) 2022-01-31T21:14:03.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:03 smithi167 conmon[54076]: debug 2022-01-31T21:14:03.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:03.537233+0000) 2022-01-31T21:14:04.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:03 smithi171 conmon[41853]: debug 2022-01-31T21:14:03.768+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:03.769128+0000) 2022-01-31T21:14:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:04 smithi171 conmon[46715]: debug 2022-01-31T21:14:04.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:04.064747+0000) 2022-01-31T21:14:04.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:04 smithi167 conmon[60316]: debug 2022-01-31T21:14:04.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:04.289063+0000) 2022-01-31T21:14:04.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:04 smithi171 conmon[51620]: debug 2022-01-31T21:14:04.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:04.644676+0000) 2022-01-31T21:14:04.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:04 smithi167 conmon[49112]: debug 2022-01-31T21:14:04.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:04.612075+0000) 2022-01-31T21:14:04.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:04 smithi167 conmon[54076]: debug 2022-01-31T21:14:04.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:04.537394+0000) 2022-01-31T21:14:05.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:04 smithi171 conmon[41853]: debug 2022-01-31T21:14:04.768+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:04.769302+0000) 2022-01-31T21:14:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:05 smithi171 conmon[46715]: debug 2022-01-31T21:14:05.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:05.064888+0000) 2022-01-31T21:14:05.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:05 smithi167 conmon[60316]: debug 2022-01-31T21:14:05.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:05.289221+0000) 2022-01-31T21:14:05.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:05 smithi171 conmon[51620]: debug 2022-01-31T21:14:05.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:05.644853+0000) 2022-01-31T21:14:05.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:05 smithi167 conmon[49112]: debug 2022-01-31T21:14:05.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:05.612263+0000) 2022-01-31T21:14:05.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:05 smithi167 conmon[54076]: debug 2022-01-31T21:14:05.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:05.537537+0000) 2022-01-31T21:14:06.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:05 smithi171 conmon[41853]: debug 2022-01-31T21:14:05.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:05.769491+0000) 2022-01-31T21:14:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:06 smithi171 conmon[46715]: debug 2022-01-31T21:14:06.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:06.065034+0000) 2022-01-31T21:14:06.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:06 smithi167 conmon[60316]: debug 2022-01-31T21:14:06.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:06.289405+0000) 2022-01-31T21:14:06.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:06 smithi171 conmon[51620]: debug 2022-01-31T21:14:06.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:06.645018+0000) 2022-01-31T21:14:06.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:06 smithi167 conmon[49112]: debug 2022-01-31T21:14:06.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:06.612466+0000) 2022-01-31T21:14:06.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:06 smithi167 conmon[54076]: debug 2022-01-31T21:14:06.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:06.537736+0000) 2022-01-31T21:14:07.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:06 smithi171 conmon[41853]: debug 2022-01-31T21:14:06.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:06.769707+0000) 2022-01-31T21:14:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[46715]: debug 2022-01-31T21:14:07.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.065220+0000) 2022-01-31T21:14:07.536 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:07 smithi167 conmon[49112]: debug 2022-01-31T21:14:07.445+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.447125+0000) 2022-01-31T21:14:07.537 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:07 smithi167 conmon[54076]: debug 2022-01-31T21:14:07.445+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.446862+0000) 2022-01-31T21:14:07.537 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:07 smithi167 conmon[60316]: debug 2022-01-31T21:14:07.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.289592+0000) 2022-01-31T21:14:07.537 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:07 smithi167 conmon[60316]: debug 2022-01-31T21:14:07.446+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.447596+0000) 2022-01-31T21:14:07.727 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[35325]: debug 2022-01-31T21:14:07.456+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113646 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:07.728 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[41853]: debug 2022-01-31T21:14:07.445+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.445856+0000) 2022-01-31T21:14:07.728 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[46715]: debug 2022-01-31T21:14:07.445+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.446541+0000) 2022-01-31T21:14:07.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[51620]: debug 2022-01-31T21:14:07.445+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.446658+0000) 2022-01-31T21:14:07.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[51620]: debug 2022-01-31T21:14:07.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.645204+0000) 2022-01-31T21:14:07.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:07 smithi167 conmon[49112]: debug 2022-01-31T21:14:07.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.612672+0000) 2022-01-31T21:14:07.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:07 smithi167 conmon[54076]: debug 2022-01-31T21:14:07.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.537917+0000) 2022-01-31T21:14:08.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:07 smithi171 conmon[41853]: debug 2022-01-31T21:14:07.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:07.769865+0000) 2022-01-31T21:14:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:08 smithi171 conmon[46715]: debug 2022-01-31T21:14:08.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:08.065398+0000) 2022-01-31T21:14:08.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:08 smithi167 conmon[60316]: debug 2022-01-31T21:14:08.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:08.289713+0000) 2022-01-31T21:14:08.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:08 smithi171 conmon[51620]: debug 2022-01-31T21:14:08.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:08.645380+0000) 2022-01-31T21:14:08.728 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:08 smithi171 conmon[35325]: debug 2022-01-31T21:14:08.645+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:14:08.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:08 smithi167 conmon[49112]: debug 2022-01-31T21:14:08.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:08.612830+0000) 2022-01-31T21:14:08.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:08 smithi167 conmon[54076]: debug 2022-01-31T21:14:08.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:08.538100+0000) 2022-01-31T21:14:09.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:08 smithi171 conmon[41853]: debug 2022-01-31T21:14:08.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:08.770004+0000) 2022-01-31T21:14:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:09 smithi171 conmon[46715]: debug 2022-01-31T21:14:09.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:09.065571+0000) 2022-01-31T21:14:09.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:09 smithi167 conmon[60316]: debug 2022-01-31T21:14:09.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:09.289826+0000) 2022-01-31T21:14:09.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:09 smithi171 conmon[51620]: debug 2022-01-31T21:14:09.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:09.645544+0000) 2022-01-31T21:14:09.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:09 smithi167 conmon[49112]: debug 2022-01-31T21:14:09.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:09.613017+0000) 2022-01-31T21:14:09.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:09 smithi167 conmon[54076]: debug 2022-01-31T21:14:09.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:09.538285+0000) 2022-01-31T21:14:10.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:09 smithi171 conmon[41853]: debug 2022-01-31T21:14:09.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:09.770167+0000) 2022-01-31T21:14:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:10 smithi171 conmon[46715]: debug 2022-01-31T21:14:10.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:10.065800+0000) 2022-01-31T21:14:10.537 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:10 smithi167 conmon[60316]: debug 2022-01-31T21:14:10.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:10.290064+0000) 2022-01-31T21:14:10.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:10 smithi171 conmon[51620]: debug 2022-01-31T21:14:10.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:10.645770+0000) 2022-01-31T21:14:10.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:10 smithi167 conmon[49112]: debug 2022-01-31T21:14:10.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:10.613207+0000) 2022-01-31T21:14:10.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:10 smithi167 conmon[54076]: debug 2022-01-31T21:14:10.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:10.538495+0000) 2022-01-31T21:14:11.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:10 smithi171 conmon[41853]: debug 2022-01-31T21:14:10.770+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:10.770325+0000) 2022-01-31T21:14:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:11 smithi171 conmon[46715]: debug 2022-01-31T21:14:11.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:11.065950+0000) 2022-01-31T21:14:11.537 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:11 smithi167 conmon[60316]: debug 2022-01-31T21:14:11.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:11.290285+0000) 2022-01-31T21:14:11.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:11 smithi171 conmon[51620]: debug 2022-01-31T21:14:11.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:11.645897+0000) 2022-01-31T21:14:11.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:11 smithi167 conmon[49112]: debug 2022-01-31T21:14:11.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:11.613396+0000) 2022-01-31T21:14:11.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:11 smithi167 conmon[54076]: debug 2022-01-31T21:14:11.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:11.538704+0000) 2022-01-31T21:14:12.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:11 smithi171 conmon[41853]: debug 2022-01-31T21:14:11.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:11.770483+0000) 2022-01-31T21:14:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[46715]: debug 2022-01-31T21:14:12.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.066143+0000) 2022-01-31T21:14:12.537 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:12 smithi167 conmon[49112]: debug 2022-01-31T21:14:12.459+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.460839+0000) 2022-01-31T21:14:12.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:12 smithi167 conmon[54076]: debug 2022-01-31T21:14:12.459+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.461304+0000) 2022-01-31T21:14:12.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:12 smithi167 conmon[60316]: debug 2022-01-31T21:14:12.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.290482+0000) 2022-01-31T21:14:12.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:12 smithi167 conmon[60316]: debug 2022-01-31T21:14:12.459+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.460605+0000) 2022-01-31T21:14:12.728 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[35325]: debug 2022-01-31T21:14:12.470+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113756 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:12.729 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[41853]: debug 2022-01-31T21:14:12.458+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.459999+0000) 2022-01-31T21:14:12.729 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[46715]: debug 2022-01-31T21:14:12.458+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.459749+0000) 2022-01-31T21:14:12.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[51620]: debug 2022-01-31T21:14:12.458+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.460356+0000) 2022-01-31T21:14:12.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[51620]: debug 2022-01-31T21:14:12.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.645999+0000) 2022-01-31T21:14:12.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:12 smithi167 conmon[49112]: debug 2022-01-31T21:14:12.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.613591+0000) 2022-01-31T21:14:12.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:12 smithi167 conmon[54076]: debug 2022-01-31T21:14:12.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.538891+0000) 2022-01-31T21:14:13.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:12 smithi171 conmon[41853]: debug 2022-01-31T21:14:12.768+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:12.770671+0000) 2022-01-31T21:14:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:13 smithi171 conmon[46715]: debug 2022-01-31T21:14:13.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:13.066349+0000) 2022-01-31T21:14:13.537 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:13 smithi167 conmon[60316]: debug 2022-01-31T21:14:13.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:13.290677+0000) 2022-01-31T21:14:13.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:13 smithi171 conmon[51620]: debug 2022-01-31T21:14:13.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:13.646082+0000) 2022-01-31T21:14:13.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:13 smithi167 conmon[49112]: debug 2022-01-31T21:14:13.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:13.613737+0000) 2022-01-31T21:14:13.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:13 smithi167 conmon[54076]: debug 2022-01-31T21:14:13.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:13.539044+0000) 2022-01-31T21:14:14.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:13 smithi171 conmon[41853]: debug 2022-01-31T21:14:13.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:13.770827+0000) 2022-01-31T21:14:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:14 smithi171 conmon[46715]: debug 2022-01-31T21:14:14.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:14.066497+0000) 2022-01-31T21:14:14.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:14 smithi167 conmon[60316]: debug 2022-01-31T21:14:14.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:14.290822+0000) 2022-01-31T21:14:14.720 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:14 smithi171 conmon[51620]: debug 2022-01-31T21:14:14.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:14.646181+0000) 2022-01-31T21:14:14.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:14 smithi167 conmon[49112]: debug 2022-01-31T21:14:14.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:14.613980+0000) 2022-01-31T21:14:14.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:14 smithi167 conmon[54076]: debug 2022-01-31T21:14:14.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:14.539208+0000) 2022-01-31T21:14:15.000 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:14 smithi171 conmon[41853]: debug 2022-01-31T21:14:14.769+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:14.771004+0000) 2022-01-31T21:14:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:15 smithi171 conmon[46715]: debug 2022-01-31T21:14:15.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:15.066650+0000) 2022-01-31T21:14:15.537 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:15 smithi167 conmon[60316]: debug 2022-01-31T21:14:15.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:15.290995+0000) 2022-01-31T21:14:15.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:15 smithi171 conmon[51620]: debug 2022-01-31T21:14:15.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:15.646325+0000) 2022-01-31T21:14:15.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:15 smithi167 conmon[49112]: debug 2022-01-31T21:14:15.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:15.614157+0000) 2022-01-31T21:14:15.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:15 smithi167 conmon[54076]: debug 2022-01-31T21:14:15.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:15.539423+0000) 2022-01-31T21:14:16.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:15 smithi171 conmon[41853]: debug 2022-01-31T21:14:15.770+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:15.771207+0000) 2022-01-31T21:14:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:16 smithi171 conmon[46715]: debug 2022-01-31T21:14:16.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:16.066826+0000) 2022-01-31T21:14:16.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:16 smithi167 conmon[60316]: debug 2022-01-31T21:14:16.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:16.291199+0000) 2022-01-31T21:14:16.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:16 smithi171 conmon[51620]: debug 2022-01-31T21:14:16.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:16.646485+0000) 2022-01-31T21:14:16.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:16 smithi167 conmon[49112]: debug 2022-01-31T21:14:16.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:16.614384+0000) 2022-01-31T21:14:16.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:16 smithi167 conmon[54076]: debug 2022-01-31T21:14:16.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:16.539590+0000) 2022-01-31T21:14:17.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:16 smithi171 conmon[41853]: debug 2022-01-31T21:14:16.770+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:16.771391+0000) 2022-01-31T21:14:17.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[46715]: debug 2022-01-31T21:14:17.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.066979+0000) 2022-01-31T21:14:17.538 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:17 smithi167 conmon[49112]: debug 2022-01-31T21:14:17.473+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.474668+0000) 2022-01-31T21:14:17.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:17 smithi167 conmon[54076]: debug 2022-01-31T21:14:17.474+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.475917+0000) 2022-01-31T21:14:17.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:17 smithi167 conmon[60316]: debug 2022-01-31T21:14:17.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.291406+0000) 2022-01-31T21:14:17.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:17 smithi167 conmon[60316]: debug 2022-01-31T21:14:17.473+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.475039+0000) 2022-01-31T21:14:17.729 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[35325]: debug 2022-01-31T21:14:17.484+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113866 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:17.729 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[41853]: debug 2022-01-31T21:14:17.473+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.474760+0000) 2022-01-31T21:14:17.730 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[46715]: debug 2022-01-31T21:14:17.472+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.474191+0000) 2022-01-31T21:14:17.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[51620]: debug 2022-01-31T21:14:17.473+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.475364+0000) 2022-01-31T21:14:17.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[51620]: debug 2022-01-31T21:14:17.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.646645+0000) 2022-01-31T21:14:17.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:17 smithi167 conmon[54076]: debug 2022-01-31T21:14:17.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.539753+0000) 2022-01-31T21:14:17.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:17 smithi167 conmon[49112]: debug 2022-01-31T21:14:17.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.614558+0000) 2022-01-31T21:14:18.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:17 smithi171 conmon[41853]: debug 2022-01-31T21:14:17.770+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:17.771572+0000) 2022-01-31T21:14:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:18 smithi171 conmon[46715]: debug 2022-01-31T21:14:18.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:18.067134+0000) 2022-01-31T21:14:18.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:18 smithi167 conmon[60316]: debug 2022-01-31T21:14:18.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:18.291590+0000) 2022-01-31T21:14:18.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:18 smithi171 conmon[51620]: debug 2022-01-31T21:14:18.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:18.646806+0000) 2022-01-31T21:14:18.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:18 smithi167 conmon[49112]: debug 2022-01-31T21:14:18.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:18.614676+0000) 2022-01-31T21:14:18.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:18 smithi167 conmon[54076]: debug 2022-01-31T21:14:18.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:18.539927+0000) 2022-01-31T21:14:19.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:18 smithi171 conmon[41853]: debug 2022-01-31T21:14:18.770+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:18.771722+0000) 2022-01-31T21:14:19.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:19 smithi171 conmon[46715]: debug 2022-01-31T21:14:19.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:19.067317+0000) 2022-01-31T21:14:19.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:19 smithi167 conmon[60316]: debug 2022-01-31T21:14:19.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:19.291732+0000) 2022-01-31T21:14:19.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:19 smithi171 conmon[51620]: debug 2022-01-31T21:14:19.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:19.646982+0000) 2022-01-31T21:14:19.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:19 smithi167 conmon[49112]: debug 2022-01-31T21:14:19.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:19.614777+0000) 2022-01-31T21:14:19.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:19 smithi167 conmon[54076]: debug 2022-01-31T21:14:19.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:19.540110+0000) 2022-01-31T21:14:20.066 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:19 smithi171 conmon[41853]: debug 2022-01-31T21:14:19.770+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:19.771899+0000) 2022-01-31T21:14:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:20 smithi171 conmon[46715]: debug 2022-01-31T21:14:20.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:20.067501+0000) 2022-01-31T21:14:20.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:20 smithi167 conmon[60316]: debug 2022-01-31T21:14:20.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:20.291934+0000) 2022-01-31T21:14:20.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:20 smithi171 conmon[51620]: debug 2022-01-31T21:14:20.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:20.647126+0000) 2022-01-31T21:14:20.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:20 smithi167 conmon[49112]: debug 2022-01-31T21:14:20.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:20.614909+0000) 2022-01-31T21:14:20.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:20 smithi167 conmon[54076]: debug 2022-01-31T21:14:20.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:20.540303+0000) 2022-01-31T21:14:21.066 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:20 smithi171 conmon[41853]: debug 2022-01-31T21:14:20.771+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:20.772110+0000) 2022-01-31T21:14:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:21 smithi171 conmon[46715]: debug 2022-01-31T21:14:21.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:21.067656+0000) 2022-01-31T21:14:21.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:21 smithi167 conmon[60316]: debug 2022-01-31T21:14:21.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:21.292126+0000) 2022-01-31T21:14:21.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:21 smithi171 conmon[51620]: debug 2022-01-31T21:14:21.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:21.647291+0000) 2022-01-31T21:14:21.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:21 smithi167 conmon[49112]: debug 2022-01-31T21:14:21.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:21.615087+0000) 2022-01-31T21:14:21.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:21 smithi167 conmon[54076]: debug 2022-01-31T21:14:21.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:21.540481+0000) 2022-01-31T21:14:22.066 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:21 smithi171 conmon[41853]: debug 2022-01-31T21:14:21.771+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:21.772262+0000) 2022-01-31T21:14:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[46715]: debug 2022-01-31T21:14:22.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.067908+0000) 2022-01-31T21:14:22.539 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:22 smithi167 conmon[49112]: debug 2022-01-31T21:14:22.487+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.488717+0000) 2022-01-31T21:14:22.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:22 smithi167 conmon[54076]: debug 2022-01-31T21:14:22.487+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.489019+0000) 2022-01-31T21:14:22.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:22 smithi167 conmon[60316]: debug 2022-01-31T21:14:22.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.292393+0000) 2022-01-31T21:14:22.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:22 smithi167 conmon[60316]: debug 2022-01-31T21:14:22.488+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.490242+0000) 2022-01-31T21:14:22.730 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[46715]: debug 2022-01-31T21:14:22.489+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.490812+0000) 2022-01-31T21:14:22.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[51620]: debug 2022-01-31T21:14:22.487+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.489306+0000) 2022-01-31T21:14:22.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[51620]: debug 2022-01-31T21:14:22.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.647455+0000) 2022-01-31T21:14:22.731 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[35325]: debug 2022-01-31T21:14:22.499+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 113976 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:22.732 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[41853]: debug 2022-01-31T21:14:22.489+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.490115+0000) 2022-01-31T21:14:22.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:22 smithi167 conmon[49112]: debug 2022-01-31T21:14:22.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.615245+0000) 2022-01-31T21:14:22.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:22 smithi167 conmon[54076]: debug 2022-01-31T21:14:22.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.540666+0000) 2022-01-31T21:14:23.067 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:22 smithi171 conmon[41853]: debug 2022-01-31T21:14:22.771+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:22.772444+0000) 2022-01-31T21:14:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:23 smithi171 conmon[46715]: debug 2022-01-31T21:14:23.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:23.068071+0000) 2022-01-31T21:14:23.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:23 smithi167 conmon[60316]: debug 2022-01-31T21:14:23.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:23.292532+0000) 2022-01-31T21:14:23.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:23 smithi167 conmon[49112]: debug 2022-01-31T21:14:23.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:23.615398+0000) 2022-01-31T21:14:23.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:23 smithi167 conmon[54076]: debug 2022-01-31T21:14:23.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:23.540838+0000) 2022-01-31T21:14:23.898 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:23 smithi171 conmon[35325]: debug 2022-01-31T21:14:23.645+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:14:23.899 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:23 smithi171 conmon[41853]: debug 2022-01-31T21:14:23.771+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:23.772601+0000) 2022-01-31T21:14:23.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:23 smithi171 conmon[51620]: debug 2022-01-31T21:14:23.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:23.647551+0000) 2022-01-31T21:14:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:24 smithi171 conmon[46715]: debug 2022-01-31T21:14:24.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:24.068223+0000) 2022-01-31T21:14:24.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:24 smithi167 conmon[60316]: debug 2022-01-31T21:14:24.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:24.292674+0000) 2022-01-31T21:14:24.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:24 smithi171 conmon[51620]: debug 2022-01-31T21:14:24.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:24.647722+0000) 2022-01-31T21:14:24.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:24 smithi167 conmon[49112]: debug 2022-01-31T21:14:24.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:24.615599+0000) 2022-01-31T21:14:24.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:24 smithi167 conmon[54076]: debug 2022-01-31T21:14:24.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:24.541043+0000) 2022-01-31T21:14:25.067 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:24 smithi171 conmon[41853]: debug 2022-01-31T21:14:24.772+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:24.772776+0000) 2022-01-31T21:14:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:25 smithi171 conmon[46715]: debug 2022-01-31T21:14:25.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:25.068432+0000) 2022-01-31T21:14:25.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:25 smithi167 conmon[60316]: debug 2022-01-31T21:14:25.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:25.292827+0000) 2022-01-31T21:14:25.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:25 smithi171 conmon[51620]: debug 2022-01-31T21:14:25.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:25.647890+0000) 2022-01-31T21:14:25.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:25 smithi167 conmon[49112]: debug 2022-01-31T21:14:25.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:25.615738+0000) 2022-01-31T21:14:25.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:25 smithi167 conmon[54076]: debug 2022-01-31T21:14:25.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:25.541249+0000) 2022-01-31T21:14:26.067 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:25 smithi171 conmon[41853]: debug 2022-01-31T21:14:25.772+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:25.772928+0000) 2022-01-31T21:14:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:26 smithi171 conmon[46715]: debug 2022-01-31T21:14:26.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:26.068654+0000) 2022-01-31T21:14:26.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:26 smithi167 conmon[60316]: debug 2022-01-31T21:14:26.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:26.292984+0000) 2022-01-31T21:14:26.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:26 smithi171 conmon[51620]: debug 2022-01-31T21:14:26.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:26.647996+0000) 2022-01-31T21:14:26.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:26 smithi167 conmon[49112]: debug 2022-01-31T21:14:26.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:26.615897+0000) 2022-01-31T21:14:26.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:26 smithi167 conmon[54076]: debug 2022-01-31T21:14:26.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:26.541415+0000) 2022-01-31T21:14:27.067 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:26 smithi171 conmon[41853]: debug 2022-01-31T21:14:26.772+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:26.773085+0000) 2022-01-31T21:14:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[46715]: debug 2022-01-31T21:14:27.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.068832+0000) 2022-01-31T21:14:27.540 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:27 smithi167 conmon[49112]: debug 2022-01-31T21:14:27.502+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.504356+0000) 2022-01-31T21:14:27.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:27 smithi167 conmon[54076]: debug 2022-01-31T21:14:27.501+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.503439+0000) 2022-01-31T21:14:27.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:27 smithi167 conmon[60316]: debug 2022-01-31T21:14:27.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.293173+0000) 2022-01-31T21:14:27.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:27 smithi167 conmon[60316]: debug 2022-01-31T21:14:27.502+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.503980+0000) 2022-01-31T21:14:27.730 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[35325]: debug 2022-01-31T21:14:27.514+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114086 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:27.731 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[41853]: debug 2022-01-31T21:14:27.503+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.504714+0000) 2022-01-31T21:14:27.732 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[46715]: debug 2022-01-31T21:14:27.503+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.504091+0000) 2022-01-31T21:14:27.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[51620]: debug 2022-01-31T21:14:27.503+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.504424+0000) 2022-01-31T21:14:27.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[51620]: debug 2022-01-31T21:14:27.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.648174+0000) 2022-01-31T21:14:27.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:27 smithi167 conmon[49112]: debug 2022-01-31T21:14:27.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.616082+0000) 2022-01-31T21:14:27.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:27 smithi167 conmon[54076]: debug 2022-01-31T21:14:27.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.541606+0000) 2022-01-31T21:14:28.067 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:27 smithi171 conmon[41853]: debug 2022-01-31T21:14:27.772+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:27.773206+0000) 2022-01-31T21:14:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:28 smithi171 conmon[46715]: debug 2022-01-31T21:14:28.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:28.068982+0000) 2022-01-31T21:14:28.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:28 smithi167 conmon[60316]: debug 2022-01-31T21:14:28.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:28.293331+0000) 2022-01-31T21:14:28.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:28 smithi171 conmon[51620]: debug 2022-01-31T21:14:28.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:28.648348+0000) 2022-01-31T21:14:28.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:28 smithi167 conmon[49112]: debug 2022-01-31T21:14:28.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:28.616231+0000) 2022-01-31T21:14:28.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:28 smithi167 conmon[54076]: debug 2022-01-31T21:14:28.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:28.541764+0000) 2022-01-31T21:14:29.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:28 smithi171 conmon[41853]: debug 2022-01-31T21:14:28.772+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:28.773365+0000) 2022-01-31T21:14:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:29 smithi171 conmon[46715]: debug 2022-01-31T21:14:29.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:29.069187+0000) 2022-01-31T21:14:29.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:29 smithi167 conmon[60316]: debug 2022-01-31T21:14:29.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:29.293508+0000) 2022-01-31T21:14:29.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:29 smithi171 conmon[51620]: debug 2022-01-31T21:14:29.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:29.648552+0000) 2022-01-31T21:14:29.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:29 smithi167 conmon[49112]: debug 2022-01-31T21:14:29.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:29.616415+0000) 2022-01-31T21:14:29.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:29 smithi167 conmon[54076]: debug 2022-01-31T21:14:29.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:29.541882+0000) 2022-01-31T21:14:30.067 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:29 smithi171 conmon[41853]: debug 2022-01-31T21:14:29.772+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:29.773564+0000) 2022-01-31T21:14:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:30 smithi171 conmon[46715]: debug 2022-01-31T21:14:30.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:30.069362+0000) 2022-01-31T21:14:30.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:30 smithi167 conmon[60316]: debug 2022-01-31T21:14:30.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:30.293672+0000) 2022-01-31T21:14:30.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:30 smithi171 conmon[51620]: debug 2022-01-31T21:14:30.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:30.648743+0000) 2022-01-31T21:14:30.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:30 smithi167 conmon[54076]: debug 2022-01-31T21:14:30.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:30.542086+0000) 2022-01-31T21:14:30.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:30 smithi167 conmon[49112]: debug 2022-01-31T21:14:30.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:30.616574+0000) 2022-01-31T21:14:31.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:30 smithi171 conmon[41853]: debug 2022-01-31T21:14:30.773+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:30.773728+0000) 2022-01-31T21:14:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:31 smithi171 conmon[46715]: debug 2022-01-31T21:14:31.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:31.069541+0000) 2022-01-31T21:14:31.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:31 smithi167 conmon[60316]: debug 2022-01-31T21:14:31.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:31.293812+0000) 2022-01-31T21:14:31.731 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:31 smithi171 conmon[51620]: debug 2022-01-31T21:14:31.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:31.648930+0000) 2022-01-31T21:14:31.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:31 smithi167 conmon[49112]: debug 2022-01-31T21:14:31.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:31.616697+0000) 2022-01-31T21:14:31.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:31 smithi167 conmon[54076]: debug 2022-01-31T21:14:31.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:31.542281+0000) 2022-01-31T21:14:32.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:31 smithi171 conmon[41853]: debug 2022-01-31T21:14:31.773+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:31.773927+0000) 2022-01-31T21:14:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[46715]: debug 2022-01-31T21:14:32.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.069748+0000) 2022-01-31T21:14:32.541 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:32 smithi167 conmon[49112]: debug 2022-01-31T21:14:32.518+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.518548+0000) 2022-01-31T21:14:32.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:32 smithi167 conmon[54076]: debug 2022-01-31T21:14:32.518+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.519359+0000) 2022-01-31T21:14:32.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:32 smithi167 conmon[60316]: debug 2022-01-31T21:14:32.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.293966+0000) 2022-01-31T21:14:32.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:32 smithi167 conmon[60316]: debug 2022-01-31T21:14:32.518+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.519138+0000) 2022-01-31T21:14:32.732 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[35325]: debug 2022-01-31T21:14:32.528+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114196 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:32.733 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[41853]: debug 2022-01-31T21:14:32.518+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.519185+0000) 2022-01-31T21:14:32.733 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[46715]: debug 2022-01-31T21:14:32.518+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.519398+0000) 2022-01-31T21:14:32.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[51620]: debug 2022-01-31T21:14:32.519+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.519828+0000) 2022-01-31T21:14:32.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[51620]: debug 2022-01-31T21:14:32.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.649032+0000) 2022-01-31T21:14:32.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:32 smithi167 conmon[54076]: debug 2022-01-31T21:14:32.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.542408+0000) 2022-01-31T21:14:32.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:32 smithi167 conmon[49112]: debug 2022-01-31T21:14:32.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.617079+0000) 2022-01-31T21:14:33.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:32 smithi171 conmon[41853]: debug 2022-01-31T21:14:32.773+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:32.774123+0000) 2022-01-31T21:14:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:33 smithi171 conmon[46715]: debug 2022-01-31T21:14:33.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:33.069929+0000) 2022-01-31T21:14:33.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:33 smithi167 conmon[60316]: debug 2022-01-31T21:14:33.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:33.294152+0000) 2022-01-31T21:14:33.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:33 smithi171 conmon[51620]: debug 2022-01-31T21:14:33.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:33.649187+0000) 2022-01-31T21:14:33.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:33 smithi167 conmon[49112]: debug 2022-01-31T21:14:33.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:33.617539+0000) 2022-01-31T21:14:33.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:33 smithi167 conmon[54076]: debug 2022-01-31T21:14:33.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:33.542528+0000) 2022-01-31T21:14:34.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:33 smithi171 conmon[41853]: debug 2022-01-31T21:14:33.773+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:33.774292+0000) 2022-01-31T21:14:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:34 smithi171 conmon[46715]: debug 2022-01-31T21:14:34.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:34.070067+0000) 2022-01-31T21:14:34.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:34 smithi167 conmon[60316]: debug 2022-01-31T21:14:34.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:34.294312+0000) 2022-01-31T21:14:34.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:34 smithi171 conmon[51620]: debug 2022-01-31T21:14:34.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:34.649365+0000) 2022-01-31T21:14:34.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:34 smithi167 conmon[54076]: debug 2022-01-31T21:14:34.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:34.542721+0000) 2022-01-31T21:14:34.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:34 smithi167 conmon[49112]: debug 2022-01-31T21:14:34.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:34.617760+0000) 2022-01-31T21:14:35.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:34 smithi171 conmon[41853]: debug 2022-01-31T21:14:34.773+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:34.774495+0000) 2022-01-31T21:14:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:35 smithi171 conmon[46715]: debug 2022-01-31T21:14:35.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:35.070250+0000) 2022-01-31T21:14:35.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:35 smithi167 conmon[60316]: debug 2022-01-31T21:14:35.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:35.294529+0000) 2022-01-31T21:14:35.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:35 smithi171 conmon[51620]: debug 2022-01-31T21:14:35.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:35.649524+0000) 2022-01-31T21:14:35.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:35 smithi167 conmon[49112]: debug 2022-01-31T21:14:35.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:35.617914+0000) 2022-01-31T21:14:35.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:35 smithi167 conmon[54076]: debug 2022-01-31T21:14:35.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:35.542908+0000) 2022-01-31T21:14:36.068 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:35 smithi171 conmon[41853]: debug 2022-01-31T21:14:35.774+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:35.774697+0000) 2022-01-31T21:14:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:36 smithi171 conmon[46715]: debug 2022-01-31T21:14:36.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:36.070436+0000) 2022-01-31T21:14:36.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:36 smithi167 conmon[60316]: debug 2022-01-31T21:14:36.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:36.294713+0000) 2022-01-31T21:14:36.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:36 smithi171 conmon[51620]: debug 2022-01-31T21:14:36.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:36.649708+0000) 2022-01-31T21:14:36.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:36 smithi167 conmon[49112]: debug 2022-01-31T21:14:36.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:36.618086+0000) 2022-01-31T21:14:36.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:36 smithi167 conmon[54076]: debug 2022-01-31T21:14:36.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:36.543086+0000) 2022-01-31T21:14:37.069 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:36 smithi171 conmon[41853]: debug 2022-01-31T21:14:36.774+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:36.774872+0000) 2022-01-31T21:14:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[46715]: debug 2022-01-31T21:14:37.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.070620+0000) 2022-01-31T21:14:37.530 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:37 smithi167 conmon[60316]: debug 2022-01-31T21:14:37.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.294935+0000) 2022-01-31T21:14:37.732 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[35325]: debug 2022-01-31T21:14:37.543+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114306 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:37.733 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[46715]: debug 2022-01-31T21:14:37.531+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.532230+0000) 2022-01-31T21:14:37.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[51620]: debug 2022-01-31T21:14:37.532+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.533169+0000) 2022-01-31T21:14:37.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[51620]: debug 2022-01-31T21:14:37.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.649832+0000) 2022-01-31T21:14:37.734 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[41853]: debug 2022-01-31T21:14:37.532+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.533232+0000) 2022-01-31T21:14:37.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:37 smithi167 conmon[49112]: debug 2022-01-31T21:14:37.532+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.532796+0000) 2022-01-31T21:14:37.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:37 smithi167 conmon[49112]: debug 2022-01-31T21:14:37.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.618297+0000) 2022-01-31T21:14:37.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:37 smithi167 conmon[60316]: debug 2022-01-31T21:14:37.532+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.533408+0000) 2022-01-31T21:14:37.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:37 smithi167 conmon[54076]: debug 2022-01-31T21:14:37.532+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.532970+0000) 2022-01-31T21:14:37.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:37 smithi167 conmon[54076]: debug 2022-01-31T21:14:37.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.543272+0000) 2022-01-31T21:14:38.069 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:37 smithi171 conmon[41853]: debug 2022-01-31T21:14:37.774+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:37.774988+0000) 2022-01-31T21:14:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:38 smithi171 conmon[46715]: debug 2022-01-31T21:14:38.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:38.070783+0000) 2022-01-31T21:14:38.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:38 smithi167 conmon[60316]: debug 2022-01-31T21:14:38.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:38.295125+0000) 2022-01-31T21:14:38.732 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:38 smithi171 conmon[35325]: debug 2022-01-31T21:14:38.646+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:14:38.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:38 smithi171 conmon[51620]: debug 2022-01-31T21:14:38.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:38.649949+0000) 2022-01-31T21:14:38.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:38 smithi167 conmon[49112]: debug 2022-01-31T21:14:38.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:38.618470+0000) 2022-01-31T21:14:38.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:38 smithi167 conmon[54076]: debug 2022-01-31T21:14:38.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:38.543433+0000) 2022-01-31T21:14:39.069 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:38 smithi171 conmon[41853]: debug 2022-01-31T21:14:38.774+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:38.775138+0000) 2022-01-31T21:14:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:39 smithi171 conmon[46715]: debug 2022-01-31T21:14:39.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:39.070983+0000) 2022-01-31T21:14:39.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:39 smithi167 conmon[60316]: debug 2022-01-31T21:14:39.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:39.295278+0000) 2022-01-31T21:14:39.732 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:39 smithi171 conmon[51620]: debug 2022-01-31T21:14:39.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:39.650108+0000) 2022-01-31T21:14:39.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:39 smithi167 conmon[49112]: debug 2022-01-31T21:14:39.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:39.618678+0000) 2022-01-31T21:14:39.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:39 smithi167 conmon[54076]: debug 2022-01-31T21:14:39.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:39.543562+0000) 2022-01-31T21:14:40.069 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:39 smithi171 conmon[41853]: debug 2022-01-31T21:14:39.774+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:39.775281+0000) 2022-01-31T21:14:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:40 smithi171 conmon[46715]: debug 2022-01-31T21:14:40.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:40.071165+0000) 2022-01-31T21:14:40.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:40 smithi167 conmon[60316]: debug 2022-01-31T21:14:40.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:40.295497+0000) 2022-01-31T21:14:40.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:40 smithi171 conmon[51620]: debug 2022-01-31T21:14:40.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:40.650289+0000) 2022-01-31T21:14:40.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:40 smithi167 conmon[54076]: debug 2022-01-31T21:14:40.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:40.543696+0000) 2022-01-31T21:14:40.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:40 smithi167 conmon[49112]: debug 2022-01-31T21:14:40.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:40.618838+0000) 2022-01-31T21:14:41.069 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:40 smithi171 conmon[41853]: debug 2022-01-31T21:14:40.775+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:40.775462+0000) 2022-01-31T21:14:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:41 smithi171 conmon[46715]: debug 2022-01-31T21:14:41.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:41.071341+0000) 2022-01-31T21:14:41.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:41 smithi167 conmon[60316]: debug 2022-01-31T21:14:41.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:41.295730+0000) 2022-01-31T21:14:41.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:41 smithi171 conmon[51620]: debug 2022-01-31T21:14:41.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:41.650474+0000) 2022-01-31T21:14:41.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:41 smithi167 conmon[49112]: debug 2022-01-31T21:14:41.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:41.619023+0000) 2022-01-31T21:14:41.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:41 smithi167 conmon[54076]: debug 2022-01-31T21:14:41.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:41.543867+0000) 2022-01-31T21:14:42.070 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:41 smithi171 conmon[41853]: debug 2022-01-31T21:14:41.775+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:41.775639+0000) 2022-01-31T21:14:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[46715]: debug 2022-01-31T21:14:42.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.071510+0000) 2022-01-31T21:14:42.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:42 smithi167 conmon[60316]: debug 2022-01-31T21:14:42.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.295939+0000) 2022-01-31T21:14:42.733 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[35325]: debug 2022-01-31T21:14:42.558+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114416 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:42.734 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[41853]: debug 2022-01-31T21:14:42.546+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.547031+0000) 2022-01-31T21:14:42.735 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[46715]: debug 2022-01-31T21:14:42.546+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.547416+0000) 2022-01-31T21:14:42.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[51620]: debug 2022-01-31T21:14:42.546+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.547604+0000) 2022-01-31T21:14:42.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[51620]: debug 2022-01-31T21:14:42.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.650649+0000) 2022-01-31T21:14:42.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:42 smithi167 conmon[60316]: debug 2022-01-31T21:14:42.547+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.547732+0000) 2022-01-31T21:14:42.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:42 smithi167 conmon[49112]: debug 2022-01-31T21:14:42.547+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.548114+0000) 2022-01-31T21:14:42.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:42 smithi167 conmon[49112]: debug 2022-01-31T21:14:42.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.619236+0000) 2022-01-31T21:14:42.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:42 smithi167 conmon[54076]: debug 2022-01-31T21:14:42.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.544054+0000) 2022-01-31T21:14:42.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:42 smithi167 conmon[54076]: debug 2022-01-31T21:14:42.547+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.547542+0000) 2022-01-31T21:14:43.070 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:42 smithi171 conmon[41853]: debug 2022-01-31T21:14:42.775+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:42.775843+0000) 2022-01-31T21:14:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:43 smithi171 conmon[46715]: debug 2022-01-31T21:14:43.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:43.071687+0000) 2022-01-31T21:14:43.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:43 smithi167 conmon[60316]: debug 2022-01-31T21:14:43.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:43.296130+0000) 2022-01-31T21:14:43.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:43 smithi171 conmon[51620]: debug 2022-01-31T21:14:43.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:43.650806+0000) 2022-01-31T21:14:43.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:43 smithi167 conmon[49112]: debug 2022-01-31T21:14:43.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:43.619390+0000) 2022-01-31T21:14:43.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:43 smithi167 conmon[54076]: debug 2022-01-31T21:14:43.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:43.544226+0000) 2022-01-31T21:14:44.070 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:43 smithi171 conmon[41853]: debug 2022-01-31T21:14:43.775+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:43.775957+0000) 2022-01-31T21:14:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:44 smithi171 conmon[46715]: debug 2022-01-31T21:14:44.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:44.071854+0000) 2022-01-31T21:14:44.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:44 smithi167 conmon[60316]: debug 2022-01-31T21:14:44.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:44.296285+0000) 2022-01-31T21:14:44.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:44 smithi171 conmon[51620]: debug 2022-01-31T21:14:44.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:44.651016+0000) 2022-01-31T21:14:44.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:44 smithi167 conmon[49112]: debug 2022-01-31T21:14:44.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:44.619571+0000) 2022-01-31T21:14:44.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:44 smithi167 conmon[54076]: debug 2022-01-31T21:14:44.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:44.544382+0000) 2022-01-31T21:14:45.070 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:44 smithi171 conmon[41853]: debug 2022-01-31T21:14:44.775+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:44.776150+0000) 2022-01-31T21:14:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:45 smithi171 conmon[46715]: debug 2022-01-31T21:14:45.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:45.072018+0000) 2022-01-31T21:14:45.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:45 smithi167 conmon[60316]: debug 2022-01-31T21:14:45.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:45.296481+0000) 2022-01-31T21:14:45.733 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:45 smithi171 conmon[51620]: debug 2022-01-31T21:14:45.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:45.651199+0000) 2022-01-31T21:14:45.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:45 smithi167 conmon[49112]: debug 2022-01-31T21:14:45.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:45.619782+0000) 2022-01-31T21:14:45.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:45 smithi167 conmon[54076]: debug 2022-01-31T21:14:45.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:45.544588+0000) 2022-01-31T21:14:46.070 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:45 smithi171 conmon[41853]: debug 2022-01-31T21:14:45.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:45.776337+0000) 2022-01-31T21:14:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:46 smithi171 conmon[46715]: debug 2022-01-31T21:14:46.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:46.072135+0000) 2022-01-31T21:14:46.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:46 smithi167 conmon[60316]: debug 2022-01-31T21:14:46.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:46.296580+0000) 2022-01-31T21:14:46.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:46 smithi171 conmon[51620]: debug 2022-01-31T21:14:46.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:46.651389+0000) 2022-01-31T21:14:46.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:46 smithi167 conmon[49112]: debug 2022-01-31T21:14:46.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:46.619973+0000) 2022-01-31T21:14:46.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:46 smithi167 conmon[54076]: debug 2022-01-31T21:14:46.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:46.544753+0000) 2022-01-31T21:14:47.070 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:46 smithi171 conmon[41853]: debug 2022-01-31T21:14:46.775+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:46.776519+0000) 2022-01-31T21:14:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[46715]: debug 2022-01-31T21:14:47.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.072332+0000) 2022-01-31T21:14:47.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:47 smithi167 conmon[60316]: debug 2022-01-31T21:14:47.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.296741+0000) 2022-01-31T21:14:47.734 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[35325]: debug 2022-01-31T21:14:47.575+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114526 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:47.735 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[41853]: debug 2022-01-31T21:14:47.560+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.561568+0000) 2022-01-31T21:14:47.736 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[46715]: debug 2022-01-31T21:14:47.561+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.561994+0000) 2022-01-31T21:14:47.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[51620]: debug 2022-01-31T21:14:47.562+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.562801+0000) 2022-01-31T21:14:47.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[51620]: debug 2022-01-31T21:14:47.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.651567+0000) 2022-01-31T21:14:47.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:47 smithi167 conmon[49112]: debug 2022-01-31T21:14:47.562+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.563096+0000) 2022-01-31T21:14:47.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:47 smithi167 conmon[49112]: debug 2022-01-31T21:14:47.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.620179+0000) 2022-01-31T21:14:47.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:47 smithi167 conmon[54076]: debug 2022-01-31T21:14:47.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.544888+0000) 2022-01-31T21:14:47.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:47 smithi167 conmon[54076]: debug 2022-01-31T21:14:47.569+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.569835+0000) 2022-01-31T21:14:47.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:47 smithi167 conmon[60316]: debug 2022-01-31T21:14:47.562+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.562613+0000) 2022-01-31T21:14:48.071 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:47 smithi171 conmon[41853]: debug 2022-01-31T21:14:47.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:47.776706+0000) 2022-01-31T21:14:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:48 smithi171 conmon[46715]: debug 2022-01-31T21:14:48.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:48.072461+0000) 2022-01-31T21:14:48.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:48 smithi167 conmon[60316]: debug 2022-01-31T21:14:48.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:48.296934+0000) 2022-01-31T21:14:48.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:48 smithi171 conmon[51620]: debug 2022-01-31T21:14:48.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:48.651739+0000) 2022-01-31T21:14:48.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:48 smithi167 conmon[54076]: debug 2022-01-31T21:14:48.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:48.545049+0000) 2022-01-31T21:14:48.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:48 smithi167 conmon[49112]: debug 2022-01-31T21:14:48.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:48.620313+0000) 2022-01-31T21:14:49.071 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:48 smithi171 conmon[41853]: debug 2022-01-31T21:14:48.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:48.776846+0000) 2022-01-31T21:14:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:49 smithi171 conmon[46715]: debug 2022-01-31T21:14:49.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:49.072653+0000) 2022-01-31T21:14:49.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:49 smithi167 conmon[60316]: debug 2022-01-31T21:14:49.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:49.297081+0000) 2022-01-31T21:14:49.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:49 smithi171 conmon[51620]: debug 2022-01-31T21:14:49.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:49.651909+0000) 2022-01-31T21:14:49.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:49 smithi167 conmon[49112]: debug 2022-01-31T21:14:49.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:49.620490+0000) 2022-01-31T21:14:49.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:49 smithi167 conmon[54076]: debug 2022-01-31T21:14:49.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:49.545179+0000) 2022-01-31T21:14:50.071 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:49 smithi171 conmon[41853]: debug 2022-01-31T21:14:49.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:49.777029+0000) 2022-01-31T21:14:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:50 smithi171 conmon[46715]: debug 2022-01-31T21:14:50.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:50.072863+0000) 2022-01-31T21:14:50.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:50 smithi167 conmon[60316]: debug 2022-01-31T21:14:50.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:50.297227+0000) 2022-01-31T21:14:50.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:50 smithi171 conmon[51620]: debug 2022-01-31T21:14:50.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:50.652135+0000) 2022-01-31T21:14:50.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:50 smithi167 conmon[54076]: debug 2022-01-31T21:14:50.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:50.545392+0000) 2022-01-31T21:14:50.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:50 smithi167 conmon[49112]: debug 2022-01-31T21:14:50.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:50.620706+0000) 2022-01-31T21:14:51.071 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:50 smithi171 conmon[41853]: debug 2022-01-31T21:14:50.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:50.777210+0000) 2022-01-31T21:14:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:51 smithi171 conmon[46715]: debug 2022-01-31T21:14:51.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:51.073037+0000) 2022-01-31T21:14:51.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:51 smithi167 conmon[60316]: debug 2022-01-31T21:14:51.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:51.297404+0000) 2022-01-31T21:14:51.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:51 smithi171 conmon[51620]: debug 2022-01-31T21:14:51.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:51.652284+0000) 2022-01-31T21:14:51.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:51 smithi167 conmon[54076]: debug 2022-01-31T21:14:51.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:51.545610+0000) 2022-01-31T21:14:51.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:51 smithi167 conmon[49112]: debug 2022-01-31T21:14:51.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:51.620831+0000) 2022-01-31T21:14:52.071 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:51 smithi171 conmon[41853]: debug 2022-01-31T21:14:51.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:51.777359+0000) 2022-01-31T21:14:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[46715]: debug 2022-01-31T21:14:52.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.073224+0000) 2022-01-31T21:14:52.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:52 smithi167 conmon[60316]: debug 2022-01-31T21:14:52.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.297605+0000) 2022-01-31T21:14:52.735 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[35325]: debug 2022-01-31T21:14:52.589+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114636 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:52.735 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[41853]: debug 2022-01-31T21:14:52.578+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.579419+0000) 2022-01-31T21:14:52.736 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[46715]: debug 2022-01-31T21:14:52.578+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.578863+0000) 2022-01-31T21:14:52.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[51620]: debug 2022-01-31T21:14:52.579+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.580582+0000) 2022-01-31T21:14:52.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[51620]: debug 2022-01-31T21:14:52.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.652465+0000) 2022-01-31T21:14:52.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:52 smithi167 conmon[49112]: debug 2022-01-31T21:14:52.578+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.579213+0000) 2022-01-31T21:14:52.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:52 smithi167 conmon[49112]: debug 2022-01-31T21:14:52.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.621000+0000) 2022-01-31T21:14:52.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:52 smithi167 conmon[54076]: debug 2022-01-31T21:14:52.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.545792+0000) 2022-01-31T21:14:52.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:52 smithi167 conmon[54076]: debug 2022-01-31T21:14:52.578+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.579121+0000) 2022-01-31T21:14:52.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:52 smithi167 conmon[60316]: debug 2022-01-31T21:14:52.580+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.580984+0000) 2022-01-31T21:14:53.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:52 smithi171 conmon[41853]: debug 2022-01-31T21:14:52.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:52.777557+0000) 2022-01-31T21:14:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:53 smithi171 conmon[46715]: debug 2022-01-31T21:14:53.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:53.073402+0000) 2022-01-31T21:14:53.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:53 smithi167 conmon[60316]: debug 2022-01-31T21:14:53.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:53.297759+0000) 2022-01-31T21:14:53.735 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:53 smithi171 conmon[35325]: debug 2022-01-31T21:14:53.647+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:14:53.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:53 smithi171 conmon[51620]: debug 2022-01-31T21:14:53.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:53.652614+0000) 2022-01-31T21:14:53.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:53 smithi167 conmon[49112]: debug 2022-01-31T21:14:53.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:53.621185+0000) 2022-01-31T21:14:53.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:53 smithi167 conmon[54076]: debug 2022-01-31T21:14:53.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:53.545986+0000) 2022-01-31T21:14:54.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:53 smithi171 conmon[41853]: debug 2022-01-31T21:14:53.776+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:53.777718+0000) 2022-01-31T21:14:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:54 smithi171 conmon[46715]: debug 2022-01-31T21:14:54.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:54.073560+0000) 2022-01-31T21:14:54.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:54 smithi167 conmon[60316]: debug 2022-01-31T21:14:54.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:54.297898+0000) 2022-01-31T21:14:54.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:54 smithi171 conmon[51620]: debug 2022-01-31T21:14:54.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:54.652798+0000) 2022-01-31T21:14:54.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:54 smithi167 conmon[49112]: debug 2022-01-31T21:14:54.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:54.621391+0000) 2022-01-31T21:14:54.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:54 smithi167 conmon[54076]: debug 2022-01-31T21:14:54.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:54.546133+0000) 2022-01-31T21:14:55.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:54 smithi171 conmon[41853]: debug 2022-01-31T21:14:54.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:54.777894+0000) 2022-01-31T21:14:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:55 smithi171 conmon[46715]: debug 2022-01-31T21:14:55.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:55.073739+0000) 2022-01-31T21:14:55.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:55 smithi167 conmon[60316]: debug 2022-01-31T21:14:55.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:55.298106+0000) 2022-01-31T21:14:55.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:55 smithi171 conmon[51620]: debug 2022-01-31T21:14:55.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:55.653013+0000) 2022-01-31T21:14:55.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:55 smithi167 conmon[49112]: debug 2022-01-31T21:14:55.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:55.621598+0000) 2022-01-31T21:14:55.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:55 smithi167 conmon[54076]: debug 2022-01-31T21:14:55.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:55.546308+0000) 2022-01-31T21:14:56.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:55 smithi171 conmon[41853]: debug 2022-01-31T21:14:55.778+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:55.778076+0000) 2022-01-31T21:14:56.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:56 smithi171 conmon[46715]: debug 2022-01-31T21:14:56.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:56.073946+0000) 2022-01-31T21:14:56.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:56 smithi167 conmon[60316]: debug 2022-01-31T21:14:56.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:56.298288+0000) 2022-01-31T21:14:56.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:56 smithi171 conmon[51620]: debug 2022-01-31T21:14:56.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:56.653182+0000) 2022-01-31T21:14:56.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:56 smithi167 conmon[49112]: debug 2022-01-31T21:14:56.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:56.621777+0000) 2022-01-31T21:14:56.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:56 smithi167 conmon[54076]: debug 2022-01-31T21:14:56.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:56.546496+0000) 2022-01-31T21:14:57.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:56 smithi171 conmon[41853]: debug 2022-01-31T21:14:56.778+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:56.778243+0000) 2022-01-31T21:14:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[46715]: debug 2022-01-31T21:14:57.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.074104+0000) 2022-01-31T21:14:57.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:57 smithi167 conmon[60316]: debug 2022-01-31T21:14:57.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.298486+0000) 2022-01-31T21:14:57.735 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[35325]: debug 2022-01-31T21:14:57.602+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114746 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:14:57.736 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[41853]: debug 2022-01-31T21:14:57.591+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.592852+0000) 2022-01-31T21:14:57.736 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[46715]: debug 2022-01-31T21:14:57.591+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.593114+0000) 2022-01-31T21:14:57.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[51620]: debug 2022-01-31T21:14:57.592+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.594409+0000) 2022-01-31T21:14:57.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[51620]: debug 2022-01-31T21:14:57.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.653386+0000) 2022-01-31T21:14:57.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:57 smithi167 conmon[60316]: debug 2022-01-31T21:14:57.591+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.593416+0000) 2022-01-31T21:14:57.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:57 smithi167 conmon[49112]: debug 2022-01-31T21:14:57.591+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.593334+0000) 2022-01-31T21:14:57.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:57 smithi167 conmon[49112]: debug 2022-01-31T21:14:57.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.621952+0000) 2022-01-31T21:14:57.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:57 smithi167 conmon[54076]: debug 2022-01-31T21:14:57.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.546698+0000) 2022-01-31T21:14:57.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:57 smithi167 conmon[54076]: debug 2022-01-31T21:14:57.591+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.593129+0000) 2022-01-31T21:14:58.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:57 smithi171 conmon[41853]: debug 2022-01-31T21:14:57.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:57.778427+0000) 2022-01-31T21:14:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:58 smithi171 conmon[46715]: debug 2022-01-31T21:14:58.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:58.074275+0000) 2022-01-31T21:14:58.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:58 smithi167 conmon[60316]: debug 2022-01-31T21:14:58.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:58.298674+0000) 2022-01-31T21:14:58.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:58 smithi171 conmon[51620]: debug 2022-01-31T21:14:58.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:58.653557+0000) 2022-01-31T21:14:58.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:58 smithi167 conmon[49112]: debug 2022-01-31T21:14:58.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:58.622058+0000) 2022-01-31T21:14:58.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:58 smithi167 conmon[54076]: debug 2022-01-31T21:14:58.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:58.546871+0000) 2022-01-31T21:14:59.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:58 smithi171 conmon[41853]: debug 2022-01-31T21:14:58.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:58.778562+0000) 2022-01-31T21:14:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:14:59 smithi171 conmon[46715]: debug 2022-01-31T21:14:59.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:59.074448+0000) 2022-01-31T21:14:59.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:14:59 smithi167 conmon[60316]: debug 2022-01-31T21:14:59.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:59.298868+0000) 2022-01-31T21:14:59.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:14:59 smithi171 conmon[51620]: debug 2022-01-31T21:14:59.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:59.653714+0000) 2022-01-31T21:14:59.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:14:59 smithi167 conmon[49112]: debug 2022-01-31T21:14:59.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:59.622247+0000) 2022-01-31T21:14:59.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:14:59 smithi167 conmon[54076]: debug 2022-01-31T21:14:59.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:59.547032+0000) 2022-01-31T21:15:00.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:14:59 smithi171 conmon[41853]: debug 2022-01-31T21:14:59.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:14:59.778750+0000) 2022-01-31T21:15:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:00 smithi171 conmon[46715]: debug 2022-01-31T21:15:00.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:00.074628+0000) 2022-01-31T21:15:00.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:00 smithi167 conmon[60316]: debug 2022-01-31T21:15:00.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:00.299032+0000) 2022-01-31T21:15:00.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:00 smithi171 conmon[51620]: debug 2022-01-31T21:15:00.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:00.653907+0000) 2022-01-31T21:15:00.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:00 smithi167 conmon[49112]: debug 2022-01-31T21:15:00.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:00.622429+0000) 2022-01-31T21:15:00.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:00 smithi167 conmon[54076]: debug 2022-01-31T21:15:00.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:00.547184+0000) 2022-01-31T21:15:01.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:00 smithi171 conmon[41853]: debug 2022-01-31T21:15:00.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:00.778892+0000) 2022-01-31T21:15:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:01 smithi171 conmon[46715]: debug 2022-01-31T21:15:01.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:01.074796+0000) 2022-01-31T21:15:01.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:01 smithi167 conmon[60316]: debug 2022-01-31T21:15:01.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:01.299189+0000) 2022-01-31T21:15:01.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:01 smithi171 conmon[51620]: debug 2022-01-31T21:15:01.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:01.654056+0000) 2022-01-31T21:15:01.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:01 smithi167 conmon[54076]: debug 2022-01-31T21:15:01.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:01.547343+0000) 2022-01-31T21:15:01.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:01 smithi167 conmon[49112]: debug 2022-01-31T21:15:01.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:01.622545+0000) 2022-01-31T21:15:02.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:01 smithi171 conmon[41853]: debug 2022-01-31T21:15:01.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:01.779009+0000) 2022-01-31T21:15:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[46715]: debug 2022-01-31T21:15:02.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.074962+0000) 2022-01-31T21:15:02.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:02 smithi167 conmon[60316]: debug 2022-01-31T21:15:02.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.299379+0000) 2022-01-31T21:15:02.736 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[35325]: debug 2022-01-31T21:15:02.617+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114856 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:02.737 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[41853]: debug 2022-01-31T21:15:02.605+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.607197+0000) 2022-01-31T21:15:02.737 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[46715]: debug 2022-01-31T21:15:02.605+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.607277+0000) 2022-01-31T21:15:02.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[51620]: debug 2022-01-31T21:15:02.606+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.607848+0000) 2022-01-31T21:15:02.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[51620]: debug 2022-01-31T21:15:02.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.654194+0000) 2022-01-31T21:15:02.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:02 smithi167 conmon[49112]: debug 2022-01-31T21:15:02.606+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.607760+0000) 2022-01-31T21:15:02.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:02 smithi167 conmon[49112]: debug 2022-01-31T21:15:02.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.622685+0000) 2022-01-31T21:15:02.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:02 smithi167 conmon[60316]: debug 2022-01-31T21:15:02.605+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.606975+0000) 2022-01-31T21:15:02.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:02 smithi167 conmon[54076]: debug 2022-01-31T21:15:02.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.547549+0000) 2022-01-31T21:15:02.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:02 smithi167 conmon[54076]: debug 2022-01-31T21:15:02.604+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.606268+0000) 2022-01-31T21:15:03.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:02 smithi171 conmon[41853]: debug 2022-01-31T21:15:02.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:02.779196+0000) 2022-01-31T21:15:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:03 smithi171 conmon[46715]: debug 2022-01-31T21:15:03.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:03.075155+0000) 2022-01-31T21:15:03.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:03 smithi167 conmon[60316]: debug 2022-01-31T21:15:03.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:03.299595+0000) 2022-01-31T21:15:03.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:03 smithi171 conmon[51620]: debug 2022-01-31T21:15:03.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:03.654336+0000) 2022-01-31T21:15:03.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:03 smithi167 conmon[49112]: debug 2022-01-31T21:15:03.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:03.622832+0000) 2022-01-31T21:15:03.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:03 smithi167 conmon[54076]: debug 2022-01-31T21:15:03.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:03.547728+0000) 2022-01-31T21:15:04.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:03 smithi171 conmon[41853]: debug 2022-01-31T21:15:03.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:03.779331+0000) 2022-01-31T21:15:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:04 smithi171 conmon[46715]: debug 2022-01-31T21:15:04.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:04.075303+0000) 2022-01-31T21:15:04.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:04 smithi167 conmon[60316]: debug 2022-01-31T21:15:04.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:04.299759+0000) 2022-01-31T21:15:04.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:04 smithi171 conmon[51620]: debug 2022-01-31T21:15:04.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:04.654497+0000) 2022-01-31T21:15:04.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:04 smithi167 conmon[54076]: debug 2022-01-31T21:15:04.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:04.547889+0000) 2022-01-31T21:15:04.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:04 smithi167 conmon[49112]: debug 2022-01-31T21:15:04.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:04.623031+0000) 2022-01-31T21:15:05.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:04 smithi171 conmon[41853]: debug 2022-01-31T21:15:04.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:04.779494+0000) 2022-01-31T21:15:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:05 smithi171 conmon[46715]: debug 2022-01-31T21:15:05.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:05.075506+0000) 2022-01-31T21:15:05.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:05 smithi167 conmon[60316]: debug 2022-01-31T21:15:05.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:05.299975+0000) 2022-01-31T21:15:05.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:05 smithi171 conmon[51620]: debug 2022-01-31T21:15:05.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:05.654649+0000) 2022-01-31T21:15:05.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:05 smithi167 conmon[49112]: debug 2022-01-31T21:15:05.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:05.623191+0000) 2022-01-31T21:15:05.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:05 smithi167 conmon[54076]: debug 2022-01-31T21:15:05.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:05.548081+0000) 2022-01-31T21:15:06.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:05 smithi171 conmon[41853]: debug 2022-01-31T21:15:05.777+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:05.779588+0000) 2022-01-31T21:15:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:06 smithi171 conmon[46715]: debug 2022-01-31T21:15:06.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:06.075692+0000) 2022-01-31T21:15:06.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:06 smithi167 conmon[60316]: debug 2022-01-31T21:15:06.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:06.300138+0000) 2022-01-31T21:15:06.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:06 smithi171 conmon[51620]: debug 2022-01-31T21:15:06.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:06.654832+0000) 2022-01-31T21:15:06.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:06 smithi167 conmon[49112]: debug 2022-01-31T21:15:06.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:06.623373+0000) 2022-01-31T21:15:06.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:06 smithi167 conmon[54076]: debug 2022-01-31T21:15:06.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:06.548274+0000) 2022-01-31T21:15:07.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:06 smithi171 conmon[41853]: debug 2022-01-31T21:15:06.778+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:06.779768+0000) 2022-01-31T21:15:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[46715]: debug 2022-01-31T21:15:07.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.075898+0000) 2022-01-31T21:15:07.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:07 smithi167 conmon[60316]: debug 2022-01-31T21:15:07.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.300343+0000) 2022-01-31T21:15:07.737 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[35325]: debug 2022-01-31T21:15:07.633+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 114966 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:07.738 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[41853]: debug 2022-01-31T21:15:07.620+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.621928+0000) 2022-01-31T21:15:07.739 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[46715]: debug 2022-01-31T21:15:07.619+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.621502+0000) 2022-01-31T21:15:07.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[51620]: debug 2022-01-31T21:15:07.621+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.622763+0000) 2022-01-31T21:15:07.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[51620]: debug 2022-01-31T21:15:07.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.654959+0000) 2022-01-31T21:15:07.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:07 smithi167 conmon[54076]: debug 2022-01-31T21:15:07.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.548474+0000) 2022-01-31T21:15:08.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:07 smithi167 conmon[54076]: debug 2022-01-31T21:15:07.620+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.621720+0000) 2022-01-31T21:15:08.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:07 smithi167 conmon[60316]: debug 2022-01-31T21:15:07.620+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.622093+0000) 2022-01-31T21:15:08.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:07 smithi167 conmon[49112]: debug 2022-01-31T21:15:07.620+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.621859+0000) 2022-01-31T21:15:08.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:07 smithi167 conmon[49112]: debug 2022-01-31T21:15:07.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.623515+0000) 2022-01-31T21:15:08.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:07 smithi171 conmon[41853]: debug 2022-01-31T21:15:07.779+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:07.779929+0000) 2022-01-31T21:15:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:08 smithi171 conmon[46715]: debug 2022-01-31T21:15:08.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:08.076066+0000) 2022-01-31T21:15:08.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:08 smithi167 conmon[60316]: debug 2022-01-31T21:15:08.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:08.300526+0000) 2022-01-31T21:15:08.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:08 smithi171 conmon[51620]: debug 2022-01-31T21:15:08.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:08.655079+0000) 2022-01-31T21:15:08.738 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:08 smithi171 conmon[35325]: debug 2022-01-31T21:15:08.647+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:15:08.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:08 smithi167 conmon[49112]: debug 2022-01-31T21:15:08.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:08.623669+0000) 2022-01-31T21:15:08.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:08 smithi167 conmon[54076]: debug 2022-01-31T21:15:08.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:08.548678+0000) 2022-01-31T21:15:09.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:08 smithi171 conmon[41853]: debug 2022-01-31T21:15:08.779+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:08.780047+0000) 2022-01-31T21:15:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:09 smithi171 conmon[46715]: debug 2022-01-31T21:15:09.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:09.076169+0000) 2022-01-31T21:15:09.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:09 smithi167 conmon[60316]: debug 2022-01-31T21:15:09.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:09.300687+0000) 2022-01-31T21:15:09.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:09 smithi171 conmon[51620]: debug 2022-01-31T21:15:09.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:09.655274+0000) 2022-01-31T21:15:09.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:09 smithi167 conmon[49112]: debug 2022-01-31T21:15:09.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:09.623856+0000) 2022-01-31T21:15:09.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:09 smithi167 conmon[54076]: debug 2022-01-31T21:15:09.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:09.548780+0000) 2022-01-31T21:15:10.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:09 smithi171 conmon[41853]: debug 2022-01-31T21:15:09.779+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:09.780226+0000) 2022-01-31T21:15:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:10 smithi171 conmon[46715]: debug 2022-01-31T21:15:10.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:10.076336+0000) 2022-01-31T21:15:10.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:10 smithi167 conmon[60316]: debug 2022-01-31T21:15:10.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:10.300890+0000) 2022-01-31T21:15:10.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:10 smithi171 conmon[51620]: debug 2022-01-31T21:15:10.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:10.655458+0000) 2022-01-31T21:15:10.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:10 smithi167 conmon[49112]: debug 2022-01-31T21:15:10.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:10.624040+0000) 2022-01-31T21:15:10.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:10 smithi167 conmon[54076]: debug 2022-01-31T21:15:10.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:10.548937+0000) 2022-01-31T21:15:11.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:10 smithi171 conmon[41853]: debug 2022-01-31T21:15:10.779+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:10.780395+0000) 2022-01-31T21:15:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:11 smithi171 conmon[46715]: debug 2022-01-31T21:15:11.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:11.076467+0000) 2022-01-31T21:15:11.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:11 smithi167 conmon[60316]: debug 2022-01-31T21:15:11.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:11.301087+0000) 2022-01-31T21:15:11.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:11 smithi171 conmon[51620]: debug 2022-01-31T21:15:11.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:11.655642+0000) 2022-01-31T21:15:11.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:11 smithi167 conmon[49112]: debug 2022-01-31T21:15:11.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:11.624241+0000) 2022-01-31T21:15:11.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:11 smithi167 conmon[54076]: debug 2022-01-31T21:15:11.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:11.549122+0000) 2022-01-31T21:15:12.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:11 smithi171 conmon[41853]: debug 2022-01-31T21:15:11.779+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:11.780518+0000) 2022-01-31T21:15:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[46715]: debug 2022-01-31T21:15:12.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.076650+0000) 2022-01-31T21:15:12.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:12 smithi167 conmon[60316]: debug 2022-01-31T21:15:12.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.301310+0000) 2022-01-31T21:15:12.738 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[35325]: debug 2022-01-31T21:15:12.648+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115077 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:12.739 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[41853]: debug 2022-01-31T21:15:12.635+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.636162+0000) 2022-01-31T21:15:12.739 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[46715]: debug 2022-01-31T21:15:12.634+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.635599+0000) 2022-01-31T21:15:12.740 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[51620]: debug 2022-01-31T21:15:12.636+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.636946+0000) 2022-01-31T21:15:12.740 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[51620]: debug 2022-01-31T21:15:12.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.655771+0000) 2022-01-31T21:15:12.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:12 smithi167 conmon[60316]: debug 2022-01-31T21:15:12.635+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.636810+0000) 2022-01-31T21:15:12.868 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:12 smithi167 conmon[49112]: debug 2022-01-31T21:15:12.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.624401+0000) 2022-01-31T21:15:12.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:12 smithi167 conmon[49112]: debug 2022-01-31T21:15:12.635+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.636693+0000) 2022-01-31T21:15:12.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:12 smithi167 conmon[54076]: debug 2022-01-31T21:15:12.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.549317+0000) 2022-01-31T21:15:12.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:12 smithi167 conmon[54076]: debug 2022-01-31T21:15:12.635+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.637384+0000) 2022-01-31T21:15:13.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:12 smithi171 conmon[41853]: debug 2022-01-31T21:15:12.780+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:12.780695+0000) 2022-01-31T21:15:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:13 smithi171 conmon[46715]: debug 2022-01-31T21:15:13.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:13.076863+0000) 2022-01-31T21:15:13.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:13 smithi167 conmon[60316]: debug 2022-01-31T21:15:13.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:13.301449+0000) 2022-01-31T21:15:13.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:13 smithi171 conmon[51620]: debug 2022-01-31T21:15:13.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:13.655937+0000) 2022-01-31T21:15:13.868 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:13 smithi167 conmon[49112]: debug 2022-01-31T21:15:13.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:13.624553+0000) 2022-01-31T21:15:13.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:13 smithi167 conmon[54076]: debug 2022-01-31T21:15:13.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:13.549474+0000) 2022-01-31T21:15:14.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:13 smithi171 conmon[41853]: debug 2022-01-31T21:15:13.780+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:13.780835+0000) 2022-01-31T21:15:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:14 smithi171 conmon[46715]: debug 2022-01-31T21:15:14.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:14.077009+0000) 2022-01-31T21:15:14.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:14 smithi167 conmon[60316]: debug 2022-01-31T21:15:14.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:14.301527+0000) 2022-01-31T21:15:14.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:14 smithi171 conmon[51620]: debug 2022-01-31T21:15:14.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:14.656173+0000) 2022-01-31T21:15:14.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:14 smithi167 conmon[54076]: debug 2022-01-31T21:15:14.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:14.549636+0000) 2022-01-31T21:15:14.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:14 smithi167 conmon[49112]: debug 2022-01-31T21:15:14.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:14.624724+0000) 2022-01-31T21:15:15.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:14 smithi171 conmon[41853]: debug 2022-01-31T21:15:14.780+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:14.780938+0000) 2022-01-31T21:15:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:15 smithi171 conmon[46715]: debug 2022-01-31T21:15:15.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:15.077168+0000) 2022-01-31T21:15:15.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:15 smithi167 conmon[60316]: debug 2022-01-31T21:15:15.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:15.301727+0000) 2022-01-31T21:15:15.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:15 smithi171 conmon[51620]: debug 2022-01-31T21:15:15.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:15.656291+0000) 2022-01-31T21:15:15.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:15 smithi167 conmon[54076]: debug 2022-01-31T21:15:15.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:15.549788+0000) 2022-01-31T21:15:15.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:15 smithi167 conmon[49112]: debug 2022-01-31T21:15:15.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:15.624884+0000) 2022-01-31T21:15:16.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:15 smithi171 conmon[41853]: debug 2022-01-31T21:15:15.780+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:15.781067+0000) 2022-01-31T21:15:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:16 smithi171 conmon[46715]: debug 2022-01-31T21:15:16.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:16.077375+0000) 2022-01-31T21:15:16.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:16 smithi167 conmon[60316]: debug 2022-01-31T21:15:16.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:16.301903+0000) 2022-01-31T21:15:16.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:16 smithi171 conmon[51620]: debug 2022-01-31T21:15:16.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:16.656497+0000) 2022-01-31T21:15:16.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:16 smithi167 conmon[54076]: debug 2022-01-31T21:15:16.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:16.549975+0000) 2022-01-31T21:15:16.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:16 smithi167 conmon[49112]: debug 2022-01-31T21:15:16.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:16.625059+0000) 2022-01-31T21:15:17.075 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:16 smithi171 conmon[41853]: debug 2022-01-31T21:15:16.780+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:16.781218+0000) 2022-01-31T21:15:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[46715]: debug 2022-01-31T21:15:17.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.077509+0000) 2022-01-31T21:15:17.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:17 smithi167 conmon[60316]: debug 2022-01-31T21:15:17.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.302092+0000) 2022-01-31T21:15:17.739 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[35325]: debug 2022-01-31T21:15:17.662+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115187 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:17.740 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[41853]: debug 2022-01-31T21:15:17.650+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.651371+0000) 2022-01-31T21:15:17.740 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[46715]: debug 2022-01-31T21:15:17.649+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.650668+0000) 2022-01-31T21:15:17.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[51620]: debug 2022-01-31T21:15:17.652+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.653014+0000) 2022-01-31T21:15:17.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[51620]: debug 2022-01-31T21:15:17.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.656629+0000) 2022-01-31T21:15:17.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:17 smithi167 conmon[54076]: debug 2022-01-31T21:15:17.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.550155+0000) 2022-01-31T21:15:17.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:17 smithi167 conmon[54076]: debug 2022-01-31T21:15:17.651+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.652576+0000) 2022-01-31T21:15:17.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:17 smithi167 conmon[60316]: debug 2022-01-31T21:15:17.651+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.653387+0000) 2022-01-31T21:15:17.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:17 smithi167 conmon[49112]: debug 2022-01-31T21:15:17.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.625265+0000) 2022-01-31T21:15:17.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:17 smithi167 conmon[49112]: debug 2022-01-31T21:15:17.651+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.653208+0000) 2022-01-31T21:15:18.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:17 smithi171 conmon[41853]: debug 2022-01-31T21:15:17.780+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:17.781433+0000) 2022-01-31T21:15:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:18 smithi171 conmon[46715]: debug 2022-01-31T21:15:18.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:18.077661+0000) 2022-01-31T21:15:18.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:18 smithi167 conmon[60316]: debug 2022-01-31T21:15:18.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:18.302250+0000) 2022-01-31T21:15:18.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:18 smithi171 conmon[51620]: debug 2022-01-31T21:15:18.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:18.656828+0000) 2022-01-31T21:15:18.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:18 smithi167 conmon[49112]: debug 2022-01-31T21:15:18.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:18.625439+0000) 2022-01-31T21:15:18.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:18 smithi167 conmon[54076]: debug 2022-01-31T21:15:18.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:18.550349+0000) 2022-01-31T21:15:19.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:18 smithi171 conmon[41853]: debug 2022-01-31T21:15:18.781+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:18.781587+0000) 2022-01-31T21:15:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:19 smithi171 conmon[46715]: debug 2022-01-31T21:15:19.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:19.077832+0000) 2022-01-31T21:15:19.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:19 smithi167 conmon[60316]: debug 2022-01-31T21:15:19.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:19.302404+0000) 2022-01-31T21:15:19.740 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:19 smithi171 conmon[51620]: debug 2022-01-31T21:15:19.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:19.657027+0000) 2022-01-31T21:15:19.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:19 smithi167 conmon[49112]: debug 2022-01-31T21:15:19.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:19.625578+0000) 2022-01-31T21:15:19.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:19 smithi167 conmon[54076]: debug 2022-01-31T21:15:19.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:19.550533+0000) 2022-01-31T21:15:20.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:19 smithi171 conmon[41853]: debug 2022-01-31T21:15:19.781+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:19.781799+0000) 2022-01-31T21:15:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:20 smithi171 conmon[46715]: debug 2022-01-31T21:15:20.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:20.078037+0000) 2022-01-31T21:15:20.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:20 smithi167 conmon[60316]: debug 2022-01-31T21:15:20.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:20.302601+0000) 2022-01-31T21:15:20.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:20 smithi171 conmon[51620]: debug 2022-01-31T21:15:20.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:20.657233+0000) 2022-01-31T21:15:20.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:20 smithi167 conmon[49112]: debug 2022-01-31T21:15:20.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:20.625725+0000) 2022-01-31T21:15:20.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:20 smithi167 conmon[54076]: debug 2022-01-31T21:15:20.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:20.550690+0000) 2022-01-31T21:15:21.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:20 smithi171 conmon[41853]: debug 2022-01-31T21:15:20.781+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:20.782005+0000) 2022-01-31T21:15:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:21 smithi171 conmon[46715]: debug 2022-01-31T21:15:21.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:21.078221+0000) 2022-01-31T21:15:21.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:21 smithi167 conmon[60316]: debug 2022-01-31T21:15:21.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:21.302698+0000) 2022-01-31T21:15:21.740 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:21 smithi171 conmon[51620]: debug 2022-01-31T21:15:21.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:21.657406+0000) 2022-01-31T21:15:21.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:21 smithi167 conmon[49112]: debug 2022-01-31T21:15:21.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:21.625908+0000) 2022-01-31T21:15:21.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:21 smithi167 conmon[54076]: debug 2022-01-31T21:15:21.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:21.550902+0000) 2022-01-31T21:15:22.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:21 smithi171 conmon[41853]: debug 2022-01-31T21:15:21.781+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:21.782185+0000) 2022-01-31T21:15:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[46715]: debug 2022-01-31T21:15:22.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.078409+0000) 2022-01-31T21:15:22.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:22 smithi167 conmon[60316]: debug 2022-01-31T21:15:22.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.302882+0000) 2022-01-31T21:15:22.740 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[35325]: debug 2022-01-31T21:15:22.677+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115297 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:22.741 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[41853]: debug 2022-01-31T21:15:22.664+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.664875+0000) 2022-01-31T21:15:22.741 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[46715]: debug 2022-01-31T21:15:22.664+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.665002+0000) 2022-01-31T21:15:22.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[51620]: debug 2022-01-31T21:15:22.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.657621+0000) 2022-01-31T21:15:22.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[51620]: debug 2022-01-31T21:15:22.666+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.666853+0000) 2022-01-31T21:15:22.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:22 smithi167 conmon[54076]: debug 2022-01-31T21:15:22.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.551117+0000) 2022-01-31T21:15:22.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:22 smithi167 conmon[54076]: debug 2022-01-31T21:15:22.665+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.666004+0000) 2022-01-31T21:15:22.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:22 smithi167 conmon[60316]: debug 2022-01-31T21:15:22.666+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.666738+0000) 2022-01-31T21:15:22.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:22 smithi167 conmon[49112]: debug 2022-01-31T21:15:22.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.626080+0000) 2022-01-31T21:15:22.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:22 smithi167 conmon[49112]: debug 2022-01-31T21:15:22.666+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.666615+0000) 2022-01-31T21:15:23.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:22 smithi171 conmon[41853]: debug 2022-01-31T21:15:22.781+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:22.782397+0000) 2022-01-31T21:15:23.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:23 smithi171 conmon[46715]: debug 2022-01-31T21:15:23.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:23.078516+0000) 2022-01-31T21:15:23.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:23 smithi167 conmon[60316]: debug 2022-01-31T21:15:23.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:23.303073+0000) 2022-01-31T21:15:23.740 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:23 smithi171 conmon[35325]: debug 2022-01-31T21:15:23.648+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:15:23.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:23 smithi171 conmon[51620]: debug 2022-01-31T21:15:23.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:23.657830+0000) 2022-01-31T21:15:23.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:23 smithi167 conmon[49112]: debug 2022-01-31T21:15:23.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:23.626257+0000) 2022-01-31T21:15:23.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:23 smithi167 conmon[54076]: debug 2022-01-31T21:15:23.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:23.551280+0000) 2022-01-31T21:15:24.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:23 smithi171 conmon[41853]: debug 2022-01-31T21:15:23.782+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:23.782569+0000) 2022-01-31T21:15:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:24 smithi171 conmon[46715]: debug 2022-01-31T21:15:24.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:24.078643+0000) 2022-01-31T21:15:24.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:24 smithi167 conmon[60316]: debug 2022-01-31T21:15:24.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:24.303242+0000) 2022-01-31T21:15:24.740 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:24 smithi171 conmon[51620]: debug 2022-01-31T21:15:24.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:24.657985+0000) 2022-01-31T21:15:24.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:24 smithi167 conmon[49112]: debug 2022-01-31T21:15:24.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:24.626429+0000) 2022-01-31T21:15:24.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:24 smithi167 conmon[54076]: debug 2022-01-31T21:15:24.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:24.551420+0000) 2022-01-31T21:15:25.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:24 smithi171 conmon[41853]: debug 2022-01-31T21:15:24.782+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:24.782766+0000) 2022-01-31T21:15:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:25 smithi171 conmon[46715]: debug 2022-01-31T21:15:25.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:25.078767+0000) 2022-01-31T21:15:25.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:25 smithi167 conmon[60316]: debug 2022-01-31T21:15:25.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:25.303398+0000) 2022-01-31T21:15:25.740 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:25 smithi171 conmon[51620]: debug 2022-01-31T21:15:25.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:25.658187+0000) 2022-01-31T21:15:25.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:25 smithi167 conmon[49112]: debug 2022-01-31T21:15:25.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:25.626620+0000) 2022-01-31T21:15:25.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:25 smithi167 conmon[54076]: debug 2022-01-31T21:15:25.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:25.551606+0000) 2022-01-31T21:15:26.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:25 smithi171 conmon[41853]: debug 2022-01-31T21:15:25.782+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:25.782939+0000) 2022-01-31T21:15:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:26 smithi171 conmon[46715]: debug 2022-01-31T21:15:26.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:26.078942+0000) 2022-01-31T21:15:26.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:26 smithi167 conmon[60316]: debug 2022-01-31T21:15:26.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:26.303556+0000) 2022-01-31T21:15:26.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:26 smithi171 conmon[51620]: debug 2022-01-31T21:15:26.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:26.658370+0000) 2022-01-31T21:15:26.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:26 smithi167 conmon[49112]: debug 2022-01-31T21:15:26.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:26.626799+0000) 2022-01-31T21:15:26.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:26 smithi167 conmon[54076]: debug 2022-01-31T21:15:26.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:26.551762+0000) 2022-01-31T21:15:27.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:26 smithi171 conmon[41853]: debug 2022-01-31T21:15:26.782+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:26.783113+0000) 2022-01-31T21:15:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[46715]: debug 2022-01-31T21:15:27.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.079122+0000) 2022-01-31T21:15:27.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:27 smithi167 conmon[60316]: debug 2022-01-31T21:15:27.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.303765+0000) 2022-01-31T21:15:27.741 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[35325]: debug 2022-01-31T21:15:27.691+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115407 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:27.742 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[41853]: debug 2022-01-31T21:15:27.679+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.680152+0000) 2022-01-31T21:15:27.742 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[46715]: debug 2022-01-31T21:15:27.679+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.680055+0000) 2022-01-31T21:15:27.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[51620]: debug 2022-01-31T21:15:27.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.658559+0000) 2022-01-31T21:15:27.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[51620]: debug 2022-01-31T21:15:27.680+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.681064+0000) 2022-01-31T21:15:27.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:27 smithi167 conmon[60316]: debug 2022-01-31T21:15:27.681+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.681638+0000) 2022-01-31T21:15:27.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:27 smithi167 conmon[49112]: debug 2022-01-31T21:15:27.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.626998+0000) 2022-01-31T21:15:27.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:27 smithi167 conmon[49112]: debug 2022-01-31T21:15:27.680+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.681475+0000) 2022-01-31T21:15:27.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:27 smithi167 conmon[54076]: debug 2022-01-31T21:15:27.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.551946+0000) 2022-01-31T21:15:27.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:27 smithi167 conmon[54076]: debug 2022-01-31T21:15:27.680+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.681272+0000) 2022-01-31T21:15:28.077 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:27 smithi171 conmon[41853]: debug 2022-01-31T21:15:27.782+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:27.783305+0000) 2022-01-31T21:15:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:28 smithi171 conmon[46715]: debug 2022-01-31T21:15:28.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:28.079281+0000) 2022-01-31T21:15:28.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:28 smithi167 conmon[60316]: debug 2022-01-31T21:15:28.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:28.303917+0000) 2022-01-31T21:15:28.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:28 smithi171 conmon[51620]: debug 2022-01-31T21:15:28.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:28.658734+0000) 2022-01-31T21:15:28.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:28 smithi167 conmon[49112]: debug 2022-01-31T21:15:28.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:28.627176+0000) 2022-01-31T21:15:28.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:28 smithi167 conmon[54076]: debug 2022-01-31T21:15:28.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:28.552111+0000) 2022-01-31T21:15:29.078 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:28 smithi171 conmon[41853]: debug 2022-01-31T21:15:28.783+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:28.783441+0000) 2022-01-31T21:15:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:29 smithi171 conmon[46715]: debug 2022-01-31T21:15:29.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:29.079460+0000) 2022-01-31T21:15:29.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:29 smithi167 conmon[60316]: debug 2022-01-31T21:15:29.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:29.304066+0000) 2022-01-31T21:15:29.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:29 smithi171 conmon[51620]: debug 2022-01-31T21:15:29.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:29.658890+0000) 2022-01-31T21:15:29.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:29 smithi167 conmon[49112]: debug 2022-01-31T21:15:29.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:29.627361+0000) 2022-01-31T21:15:29.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:29 smithi167 conmon[54076]: debug 2022-01-31T21:15:29.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:29.552228+0000) 2022-01-31T21:15:30.078 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:29 smithi171 conmon[41853]: debug 2022-01-31T21:15:29.783+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:29.783605+0000) 2022-01-31T21:15:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:30 smithi171 conmon[46715]: debug 2022-01-31T21:15:30.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:30.079645+0000) 2022-01-31T21:15:30.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:30 smithi167 conmon[60316]: debug 2022-01-31T21:15:30.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:30.304247+0000) 2022-01-31T21:15:30.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:30 smithi171 conmon[51620]: debug 2022-01-31T21:15:30.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:30.659069+0000) 2022-01-31T21:15:30.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:30 smithi167 conmon[49112]: debug 2022-01-31T21:15:30.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:30.627538+0000) 2022-01-31T21:15:30.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:30 smithi167 conmon[54076]: debug 2022-01-31T21:15:30.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:30.552388+0000) 2022-01-31T21:15:31.078 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:30 smithi171 conmon[41853]: debug 2022-01-31T21:15:30.783+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:30.783829+0000) 2022-01-31T21:15:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:31 smithi171 conmon[46715]: debug 2022-01-31T21:15:31.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:31.079847+0000) 2022-01-31T21:15:31.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:31 smithi167 conmon[60316]: debug 2022-01-31T21:15:31.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:31.304410+0000) 2022-01-31T21:15:31.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:31 smithi171 conmon[51620]: debug 2022-01-31T21:15:31.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:31.659250+0000) 2022-01-31T21:15:31.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:31 smithi167 conmon[49112]: debug 2022-01-31T21:15:31.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:31.627698+0000) 2022-01-31T21:15:31.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:31 smithi167 conmon[54076]: debug 2022-01-31T21:15:31.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:31.552571+0000) 2022-01-31T21:15:32.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:31 smithi171 conmon[41853]: debug 2022-01-31T21:15:31.783+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:31.783984+0000) 2022-01-31T21:15:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[46715]: debug 2022-01-31T21:15:32.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.080049+0000) 2022-01-31T21:15:32.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:32 smithi167 conmon[60316]: debug 2022-01-31T21:15:32.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.304585+0000) 2022-01-31T21:15:32.705 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[41853]: debug 2022-01-31T21:15:32.693+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.694198+0000) 2022-01-31T21:15:32.705 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[46715]: debug 2022-01-31T21:15:32.693+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.693850+0000) 2022-01-31T21:15:32.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[51620]: debug 2022-01-31T21:15:32.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.659401+0000) 2022-01-31T21:15:32.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[51620]: debug 2022-01-31T21:15:32.694+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.695422+0000) 2022-01-31T21:15:32.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:32 smithi167 conmon[54076]: debug 2022-01-31T21:15:32.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.552772+0000) 2022-01-31T21:15:32.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:32 smithi167 conmon[54076]: debug 2022-01-31T21:15:32.694+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.694674+0000) 2022-01-31T21:15:32.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:32 smithi167 conmon[60316]: debug 2022-01-31T21:15:32.695+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.695664+0000) 2022-01-31T21:15:32.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:32 smithi167 conmon[49112]: debug 2022-01-31T21:15:32.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.627892+0000) 2022-01-31T21:15:32.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:32 smithi167 conmon[49112]: debug 2022-01-31T21:15:32.695+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.695854+0000) 2022-01-31T21:15:33.078 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[35325]: debug 2022-01-31T21:15:32.705+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115517 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:33.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:32 smithi171 conmon[41853]: debug 2022-01-31T21:15:32.783+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:32.784213+0000) 2022-01-31T21:15:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:33 smithi171 conmon[46715]: debug 2022-01-31T21:15:33.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:33.080228+0000) 2022-01-31T21:15:33.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:33 smithi167 conmon[60316]: debug 2022-01-31T21:15:33.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:33.304785+0000) 2022-01-31T21:15:33.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:33 smithi171 conmon[51620]: debug 2022-01-31T21:15:33.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:33.659585+0000) 2022-01-31T21:15:33.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:33 smithi167 conmon[49112]: debug 2022-01-31T21:15:33.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:33.628112+0000) 2022-01-31T21:15:33.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:33 smithi167 conmon[54076]: debug 2022-01-31T21:15:33.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:33.552933+0000) 2022-01-31T21:15:34.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:33 smithi171 conmon[41853]: debug 2022-01-31T21:15:33.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:33.784367+0000) 2022-01-31T21:15:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:34 smithi171 conmon[46715]: debug 2022-01-31T21:15:34.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:34.080401+0000) 2022-01-31T21:15:34.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:34 smithi167 conmon[60316]: debug 2022-01-31T21:15:34.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:34.304953+0000) 2022-01-31T21:15:34.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:34 smithi171 conmon[51620]: debug 2022-01-31T21:15:34.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:34.659723+0000) 2022-01-31T21:15:34.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:34 smithi167 conmon[49112]: debug 2022-01-31T21:15:34.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:34.628260+0000) 2022-01-31T21:15:34.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:34 smithi167 conmon[54076]: debug 2022-01-31T21:15:34.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:34.553047+0000) 2022-01-31T21:15:35.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:34 smithi171 conmon[41853]: debug 2022-01-31T21:15:34.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:34.784553+0000) 2022-01-31T21:15:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:35 smithi171 conmon[46715]: debug 2022-01-31T21:15:35.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:35.080603+0000) 2022-01-31T21:15:35.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:35 smithi167 conmon[60316]: debug 2022-01-31T21:15:35.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:35.305137+0000) 2022-01-31T21:15:35.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:35 smithi171 conmon[51620]: debug 2022-01-31T21:15:35.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:35.659941+0000) 2022-01-31T21:15:35.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:35 smithi167 conmon[49112]: debug 2022-01-31T21:15:35.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:35.628416+0000) 2022-01-31T21:15:35.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:35 smithi167 conmon[54076]: debug 2022-01-31T21:15:35.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:35.553220+0000) 2022-01-31T21:15:36.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:35 smithi171 conmon[41853]: debug 2022-01-31T21:15:35.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:35.784716+0000) 2022-01-31T21:15:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:36 smithi171 conmon[46715]: debug 2022-01-31T21:15:36.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:36.080787+0000) 2022-01-31T21:15:36.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:36 smithi167 conmon[60316]: debug 2022-01-31T21:15:36.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:36.305369+0000) 2022-01-31T21:15:36.742 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:36 smithi171 conmon[51620]: debug 2022-01-31T21:15:36.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:36.660150+0000) 2022-01-31T21:15:36.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:36 smithi167 conmon[49112]: debug 2022-01-31T21:15:36.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:36.628593+0000) 2022-01-31T21:15:36.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:36 smithi167 conmon[54076]: debug 2022-01-31T21:15:36.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:36.553334+0000) 2022-01-31T21:15:37.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:36 smithi171 conmon[41853]: debug 2022-01-31T21:15:36.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:36.784870+0000) 2022-01-31T21:15:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[46715]: debug 2022-01-31T21:15:37.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.080965+0000) 2022-01-31T21:15:37.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:37 smithi167 conmon[60316]: debug 2022-01-31T21:15:37.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.305516+0000) 2022-01-31T21:15:37.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[51620]: debug 2022-01-31T21:15:37.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.660353+0000) 2022-01-31T21:15:37.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:37 smithi167 conmon[60316]: debug 2022-01-31T21:15:37.709+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.710013+0000) 2022-01-31T21:15:37.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:37 smithi167 conmon[49112]: debug 2022-01-31T21:15:37.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.628782+0000) 2022-01-31T21:15:37.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:37 smithi167 conmon[49112]: debug 2022-01-31T21:15:37.710+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.710508+0000) 2022-01-31T21:15:37.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:37 smithi167 conmon[54076]: debug 2022-01-31T21:15:37.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.553540+0000) 2022-01-31T21:15:37.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:37 smithi167 conmon[54076]: debug 2022-01-31T21:15:37.708+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.708727+0000) 2022-01-31T21:15:38.079 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[35325]: debug 2022-01-31T21:15:37.720+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115627 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:38.080 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[46715]: debug 2022-01-31T21:15:37.707+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.708321+0000) 2022-01-31T21:15:38.080 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[51620]: debug 2022-01-31T21:15:37.708+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.709015+0000) 2022-01-31T21:15:38.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[41853]: debug 2022-01-31T21:15:37.709+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.709872+0000) 2022-01-31T21:15:38.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:37 smithi171 conmon[41853]: debug 2022-01-31T21:15:37.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:37.785020+0000) 2022-01-31T21:15:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:38 smithi171 conmon[46715]: debug 2022-01-31T21:15:38.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:38.081120+0000) 2022-01-31T21:15:38.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:38 smithi167 conmon[60316]: debug 2022-01-31T21:15:38.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:38.305674+0000) 2022-01-31T21:15:38.743 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:38 smithi171 conmon[35325]: debug 2022-01-31T21:15:38.649+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:15:38.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:38 smithi171 conmon[51620]: debug 2022-01-31T21:15:38.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:38.660571+0000) 2022-01-31T21:15:38.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:38 smithi167 conmon[49112]: debug 2022-01-31T21:15:38.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:38.628984+0000) 2022-01-31T21:15:38.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:38 smithi167 conmon[54076]: debug 2022-01-31T21:15:38.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:38.553682+0000) 2022-01-31T21:15:39.079 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:38 smithi171 conmon[41853]: debug 2022-01-31T21:15:38.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:38.785178+0000) 2022-01-31T21:15:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:39 smithi171 conmon[46715]: debug 2022-01-31T21:15:39.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:39.081316+0000) 2022-01-31T21:15:39.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:39 smithi167 conmon[60316]: debug 2022-01-31T21:15:39.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:39.305862+0000) 2022-01-31T21:15:39.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:39 smithi171 conmon[51620]: debug 2022-01-31T21:15:39.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:39.660717+0000) 2022-01-31T21:15:39.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:39 smithi167 conmon[54076]: debug 2022-01-31T21:15:39.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:39.553828+0000) 2022-01-31T21:15:39.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:39 smithi167 conmon[49112]: debug 2022-01-31T21:15:39.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:39.629125+0000) 2022-01-31T21:15:40.080 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:39 smithi171 conmon[41853]: debug 2022-01-31T21:15:39.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:39.785358+0000) 2022-01-31T21:15:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:40 smithi171 conmon[46715]: debug 2022-01-31T21:15:40.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:40.081492+0000) 2022-01-31T21:15:40.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:40 smithi167 conmon[60316]: debug 2022-01-31T21:15:40.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:40.306075+0000) 2022-01-31T21:15:40.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:40 smithi171 conmon[51620]: debug 2022-01-31T21:15:40.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:40.660940+0000) 2022-01-31T21:15:40.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:40 smithi167 conmon[49112]: debug 2022-01-31T21:15:40.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:40.629315+0000) 2022-01-31T21:15:40.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:40 smithi167 conmon[54076]: debug 2022-01-31T21:15:40.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:40.554047+0000) 2022-01-31T21:15:41.080 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:40 smithi171 conmon[41853]: debug 2022-01-31T21:15:40.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:40.785542+0000) 2022-01-31T21:15:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:41 smithi171 conmon[46715]: debug 2022-01-31T21:15:41.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:41.081645+0000) 2022-01-31T21:15:41.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:41 smithi167 conmon[60316]: debug 2022-01-31T21:15:41.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:41.306342+0000) 2022-01-31T21:15:41.743 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:41 smithi171 conmon[51620]: debug 2022-01-31T21:15:41.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:41.661115+0000) 2022-01-31T21:15:41.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:41 smithi167 conmon[54076]: debug 2022-01-31T21:15:41.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:41.554236+0000) 2022-01-31T21:15:41.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:41 smithi167 conmon[49112]: debug 2022-01-31T21:15:41.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:41.629523+0000) 2022-01-31T21:15:42.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:41 smithi171 conmon[41853]: debug 2022-01-31T21:15:41.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:41.785760+0000) 2022-01-31T21:15:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[46715]: debug 2022-01-31T21:15:42.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.081833+0000) 2022-01-31T21:15:42.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:42 smithi167 conmon[60316]: debug 2022-01-31T21:15:42.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.306519+0000) 2022-01-31T21:15:42.722 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[51620]: debug 2022-01-31T21:15:42.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.661315+0000) 2022-01-31T21:15:42.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:42 smithi167 conmon[54076]: debug 2022-01-31T21:15:42.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.554424+0000) 2022-01-31T21:15:42.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:42 smithi167 conmon[54076]: debug 2022-01-31T21:15:42.724+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.724630+0000) 2022-01-31T21:15:42.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:42 smithi167 conmon[60316]: debug 2022-01-31T21:15:42.725+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.725722+0000) 2022-01-31T21:15:42.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:42 smithi167 conmon[49112]: debug 2022-01-31T21:15:42.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.629702+0000) 2022-01-31T21:15:42.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:42 smithi167 conmon[49112]: debug 2022-01-31T21:15:42.723+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.724261+0000) 2022-01-31T21:15:43.080 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[35325]: debug 2022-01-31T21:15:42.734+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115737 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:43.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[41853]: debug 2022-01-31T21:15:42.722+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.724407+0000) 2022-01-31T21:15:43.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[41853]: debug 2022-01-31T21:15:42.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.785976+0000) 2022-01-31T21:15:43.081 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[46715]: debug 2022-01-31T21:15:42.721+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.723691+0000) 2022-01-31T21:15:43.082 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:42 smithi171 conmon[51620]: debug 2022-01-31T21:15:42.722+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:42.724687+0000) 2022-01-31T21:15:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:43 smithi171 conmon[46715]: debug 2022-01-31T21:15:43.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:43.081997+0000) 2022-01-31T21:15:43.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:43 smithi167 conmon[60316]: debug 2022-01-31T21:15:43.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:43.306719+0000) 2022-01-31T21:15:43.744 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:43 smithi171 conmon[51620]: debug 2022-01-31T21:15:43.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:43.661497+0000) 2022-01-31T21:15:43.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:43 smithi167 conmon[49112]: debug 2022-01-31T21:15:43.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:43.629916+0000) 2022-01-31T21:15:43.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:43 smithi167 conmon[54076]: debug 2022-01-31T21:15:43.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:43.554619+0000) 2022-01-31T21:15:44.080 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:43 smithi171 conmon[41853]: debug 2022-01-31T21:15:43.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:43.786136+0000) 2022-01-31T21:15:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:44 smithi171 conmon[46715]: debug 2022-01-31T21:15:44.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:44.082140+0000) 2022-01-31T21:15:44.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:44 smithi167 conmon[60316]: debug 2022-01-31T21:15:44.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:44.306833+0000) 2022-01-31T21:15:44.744 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:44 smithi171 conmon[51620]: debug 2022-01-31T21:15:44.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:44.661652+0000) 2022-01-31T21:15:44.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:44 smithi167 conmon[54076]: debug 2022-01-31T21:15:44.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:44.554736+0000) 2022-01-31T21:15:44.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:44 smithi167 conmon[49112]: debug 2022-01-31T21:15:44.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:44.630084+0000) 2022-01-31T21:15:45.080 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:44 smithi171 conmon[41853]: debug 2022-01-31T21:15:44.784+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:44.786315+0000) 2022-01-31T21:15:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:45 smithi171 conmon[46715]: debug 2022-01-31T21:15:45.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:45.082322+0000) 2022-01-31T21:15:45.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:45 smithi167 conmon[60316]: debug 2022-01-31T21:15:45.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:45.307001+0000) 2022-01-31T21:15:45.744 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:45 smithi171 conmon[51620]: debug 2022-01-31T21:15:45.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:45.661859+0000) 2022-01-31T21:15:45.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:45 smithi167 conmon[49112]: debug 2022-01-31T21:15:45.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:45.630261+0000) 2022-01-31T21:15:45.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:45 smithi167 conmon[54076]: debug 2022-01-31T21:15:45.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:45.554909+0000) 2022-01-31T21:15:46.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:45 smithi171 conmon[41853]: debug 2022-01-31T21:15:45.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:45.786499+0000) 2022-01-31T21:15:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:46 smithi171 conmon[46715]: debug 2022-01-31T21:15:46.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:46.082478+0000) 2022-01-31T21:15:46.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:46 smithi167 conmon[60316]: debug 2022-01-31T21:15:46.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:46.307209+0000) 2022-01-31T21:15:46.744 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:46 smithi171 conmon[51620]: debug 2022-01-31T21:15:46.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:46.662051+0000) 2022-01-31T21:15:46.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:46 smithi167 conmon[54076]: debug 2022-01-31T21:15:46.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:46.555067+0000) 2022-01-31T21:15:46.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:46 smithi167 conmon[49112]: debug 2022-01-31T21:15:46.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:46.630451+0000) 2022-01-31T21:15:47.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:46 smithi171 conmon[41853]: debug 2022-01-31T21:15:46.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:46.786676+0000) 2022-01-31T21:15:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[46715]: debug 2022-01-31T21:15:47.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.082686+0000) 2022-01-31T21:15:47.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:47 smithi167 conmon[60316]: debug 2022-01-31T21:15:47.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.307404+0000) 2022-01-31T21:15:47.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[51620]: debug 2022-01-31T21:15:47.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.662214+0000) 2022-01-31T21:15:47.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:47 smithi167 conmon[60316]: debug 2022-01-31T21:15:47.737+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.739330+0000) 2022-01-31T21:15:47.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:47 smithi167 conmon[49112]: debug 2022-01-31T21:15:47.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.630641+0000) 2022-01-31T21:15:47.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:47 smithi167 conmon[49112]: debug 2022-01-31T21:15:47.737+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.739255+0000) 2022-01-31T21:15:47.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:47 smithi167 conmon[54076]: debug 2022-01-31T21:15:47.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.555266+0000) 2022-01-31T21:15:47.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:47 smithi167 conmon[54076]: debug 2022-01-31T21:15:47.737+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.738543+0000) 2022-01-31T21:15:48.081 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[35325]: debug 2022-01-31T21:15:47.764+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115847 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:48.082 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[46715]: debug 2022-01-31T21:15:47.737+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.739412+0000) 2022-01-31T21:15:48.082 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[51620]: debug 2022-01-31T21:15:47.738+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.740212+0000) 2022-01-31T21:15:48.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[41853]: debug 2022-01-31T21:15:47.736+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.737909+0000) 2022-01-31T21:15:48.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:47 smithi171 conmon[41853]: debug 2022-01-31T21:15:47.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:47.786864+0000) 2022-01-31T21:15:48.221 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T21:15:48.221+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T21:15:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:48 smithi171 conmon[46715]: debug 2022-01-31T21:15:48.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:48.082868+0000) 2022-01-31T21:15:48.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:48 smithi167 conmon[60316]: debug 2022-01-31T21:15:48.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:48.307590+0000) 2022-01-31T21:15:48.744 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:48 smithi171 conmon[51620]: debug 2022-01-31T21:15:48.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:48.662379+0000) 2022-01-31T21:15:48.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:48 smithi167 conmon[49112]: debug 2022-01-31T21:15:48.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:48.630814+0000) 2022-01-31T21:15:48.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:48 smithi167 conmon[54076]: debug 2022-01-31T21:15:48.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:48.555485+0000) 2022-01-31T21:15:49.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:48 smithi171 conmon[41853]: debug 2022-01-31T21:15:48.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:48.787011+0000) 2022-01-31T21:15:49.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:49 smithi171 conmon[46715]: debug 2022-01-31T21:15:49.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:49.083058+0000) 2022-01-31T21:15:49.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:49 smithi167 conmon[60316]: debug 2022-01-31T21:15:49.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:49.307744+0000) 2022-01-31T21:15:49.745 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:49 smithi171 conmon[51620]: debug 2022-01-31T21:15:49.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:49.662530+0000) 2022-01-31T21:15:49.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:49 smithi167 conmon[49112]: debug 2022-01-31T21:15:49.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:49.630925+0000) 2022-01-31T21:15:49.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:49 smithi167 conmon[54076]: debug 2022-01-31T21:15:49.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:49.555679+0000) 2022-01-31T21:15:50.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:49 smithi171 conmon[41853]: debug 2022-01-31T21:15:49.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:49.787168+0000) 2022-01-31T21:15:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:50 smithi171 conmon[46715]: debug 2022-01-31T21:15:50.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:50.083217+0000) 2022-01-31T21:15:50.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:50 smithi167 conmon[60316]: debug 2022-01-31T21:15:50.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:50.307978+0000) 2022-01-31T21:15:50.745 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:50 smithi171 conmon[51620]: debug 2022-01-31T21:15:50.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:50.662736+0000) 2022-01-31T21:15:50.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:50 smithi167 conmon[49112]: debug 2022-01-31T21:15:50.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:50.631067+0000) 2022-01-31T21:15:50.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:50 smithi167 conmon[54076]: debug 2022-01-31T21:15:50.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:50.555875+0000) 2022-01-31T21:15:51.081 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:50 smithi171 conmon[41853]: debug 2022-01-31T21:15:50.786+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:50.787325+0000) 2022-01-31T21:15:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:51 smithi171 conmon[46715]: debug 2022-01-31T21:15:51.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:51.083351+0000) 2022-01-31T21:15:51.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:51 smithi167 conmon[60316]: debug 2022-01-31T21:15:51.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:51.308140+0000) 2022-01-31T21:15:51.745 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:51 smithi171 conmon[51620]: debug 2022-01-31T21:15:51.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:51.662941+0000) 2022-01-31T21:15:51.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:51 smithi167 conmon[49112]: debug 2022-01-31T21:15:51.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:51.631244+0000) 2022-01-31T21:15:51.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:51 smithi167 conmon[54076]: debug 2022-01-31T21:15:51.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:51.556024+0000) 2022-01-31T21:15:52.082 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:51 smithi171 conmon[41853]: debug 2022-01-31T21:15:51.786+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:51.787527+0000) 2022-01-31T21:15:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[46715]: debug 2022-01-31T21:15:52.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.083460+0000) 2022-01-31T21:15:52.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:52 smithi167 conmon[60316]: debug 2022-01-31T21:15:52.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.308328+0000) 2022-01-31T21:15:52.745 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[51620]: debug 2022-01-31T21:15:52.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.663071+0000) 2022-01-31T21:15:52.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:52 smithi167 conmon[60316]: debug 2022-01-31T21:15:52.767+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.768615+0000) 2022-01-31T21:15:52.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:52 smithi167 conmon[49112]: debug 2022-01-31T21:15:52.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.631453+0000) 2022-01-31T21:15:52.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:52 smithi167 conmon[49112]: debug 2022-01-31T21:15:52.773+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.775231+0000) 2022-01-31T21:15:52.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:52 smithi167 conmon[54076]: debug 2022-01-31T21:15:52.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.556226+0000) 2022-01-31T21:15:52.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:52 smithi167 conmon[54076]: debug 2022-01-31T21:15:52.768+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.769759+0000) 2022-01-31T21:15:53.082 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[35325]: debug 2022-01-31T21:15:52.778+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 115957 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:53.082 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[46715]: debug 2022-01-31T21:15:52.767+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.769498+0000) 2022-01-31T21:15:53.083 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[51620]: debug 2022-01-31T21:15:52.768+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.770091+0000) 2022-01-31T21:15:53.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[41853]: debug 2022-01-31T21:15:52.766+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.768103+0000) 2022-01-31T21:15:53.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:52 smithi171 conmon[41853]: debug 2022-01-31T21:15:52.785+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:52.787672+0000) 2022-01-31T21:15:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:53 smithi171 conmon[46715]: debug 2022-01-31T21:15:53.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:53.083618+0000) 2022-01-31T21:15:53.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:53 smithi167 conmon[60316]: debug 2022-01-31T21:15:53.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:53.308514+0000) 2022-01-31T21:15:53.745 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:53 smithi171 conmon[51620]: debug 2022-01-31T21:15:53.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:53.663236+0000) 2022-01-31T21:15:53.746 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:53 smithi171 conmon[35325]: debug 2022-01-31T21:15:53.649+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:15:53.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:53 smithi167 conmon[49112]: debug 2022-01-31T21:15:53.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:53.631613+0000) 2022-01-31T21:15:53.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:53 smithi167 conmon[54076]: debug 2022-01-31T21:15:53.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:53.556431+0000) 2022-01-31T21:15:54.082 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:53 smithi171 conmon[41853]: debug 2022-01-31T21:15:53.787+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:53.787848+0000) 2022-01-31T21:15:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:54 smithi171 conmon[46715]: debug 2022-01-31T21:15:54.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:54.083736+0000) 2022-01-31T21:15:54.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:54 smithi167 conmon[60316]: debug 2022-01-31T21:15:54.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:54.308683+0000) 2022-01-31T21:15:54.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:54 smithi171 conmon[51620]: debug 2022-01-31T21:15:54.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:54.663351+0000) 2022-01-31T21:15:54.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:54 smithi167 conmon[49112]: debug 2022-01-31T21:15:54.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:54.631736+0000) 2022-01-31T21:15:54.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:54 smithi167 conmon[54076]: debug 2022-01-31T21:15:54.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:54.556584+0000) 2022-01-31T21:15:55.082 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:54 smithi171 conmon[41853]: debug 2022-01-31T21:15:54.787+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:54.788033+0000) 2022-01-31T21:15:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:55 smithi171 conmon[46715]: debug 2022-01-31T21:15:55.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:55.083937+0000) 2022-01-31T21:15:55.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:55 smithi167 conmon[60316]: debug 2022-01-31T21:15:55.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:55.308875+0000) 2022-01-31T21:15:55.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:55 smithi171 conmon[51620]: debug 2022-01-31T21:15:55.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:55.663499+0000) 2022-01-31T21:15:55.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:55 smithi167 conmon[49112]: debug 2022-01-31T21:15:55.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:55.631913+0000) 2022-01-31T21:15:55.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:55 smithi167 conmon[54076]: debug 2022-01-31T21:15:55.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:55.556746+0000) 2022-01-31T21:15:56.082 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:55 smithi171 conmon[41853]: debug 2022-01-31T21:15:55.787+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:55.788251+0000) 2022-01-31T21:15:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:56 smithi171 conmon[46715]: debug 2022-01-31T21:15:56.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:56.084115+0000) 2022-01-31T21:15:56.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:56 smithi167 conmon[60316]: debug 2022-01-31T21:15:56.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:56.309055+0000) 2022-01-31T21:15:56.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:56 smithi171 conmon[51620]: debug 2022-01-31T21:15:56.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:56.663683+0000) 2022-01-31T21:15:56.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:56 smithi167 conmon[49112]: debug 2022-01-31T21:15:56.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:56.632069+0000) 2022-01-31T21:15:56.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:56 smithi167 conmon[54076]: debug 2022-01-31T21:15:56.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:56.556905+0000) 2022-01-31T21:15:57.082 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:56 smithi171 conmon[41853]: debug 2022-01-31T21:15:56.787+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:56.788490+0000) 2022-01-31T21:15:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[46715]: debug 2022-01-31T21:15:57.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.084279+0000) 2022-01-31T21:15:57.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:57 smithi167 conmon[60316]: debug 2022-01-31T21:15:57.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.309244+0000) 2022-01-31T21:15:57.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[51620]: debug 2022-01-31T21:15:57.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.663869+0000) 2022-01-31T21:15:57.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:57 smithi167 conmon[60316]: debug 2022-01-31T21:15:57.781+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.782980+0000) 2022-01-31T21:15:57.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:57 smithi167 conmon[49112]: debug 2022-01-31T21:15:57.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.632248+0000) 2022-01-31T21:15:57.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:57 smithi167 conmon[49112]: debug 2022-01-31T21:15:57.782+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.783787+0000) 2022-01-31T21:15:57.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:57 smithi167 conmon[54076]: debug 2022-01-31T21:15:57.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.557095+0000) 2022-01-31T21:15:57.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:57 smithi167 conmon[54076]: debug 2022-01-31T21:15:57.781+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.783163+0000) 2022-01-31T21:15:58.083 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[35325]: debug 2022-01-31T21:15:57.794+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116067 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:15:58.083 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[46715]: debug 2022-01-31T21:15:57.782+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.783215+0000) 2022-01-31T21:15:58.084 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[51620]: debug 2022-01-31T21:15:57.783+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.784106+0000) 2022-01-31T21:15:58.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[41853]: debug 2022-01-31T21:15:57.782+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.783406+0000) 2022-01-31T21:15:58.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:57 smithi171 conmon[41853]: debug 2022-01-31T21:15:57.787+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:57.788675+0000) 2022-01-31T21:15:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:58 smithi171 conmon[46715]: debug 2022-01-31T21:15:58.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:58.084448+0000) 2022-01-31T21:15:58.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:58 smithi167 conmon[60316]: debug 2022-01-31T21:15:58.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:58.309452+0000) 2022-01-31T21:15:58.746 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:58 smithi171 conmon[51620]: debug 2022-01-31T21:15:58.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:58.664049+0000) 2022-01-31T21:15:58.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:58 smithi167 conmon[49112]: debug 2022-01-31T21:15:58.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:58.632435+0000) 2022-01-31T21:15:58.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:58 smithi167 conmon[54076]: debug 2022-01-31T21:15:58.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:58.557234+0000) 2022-01-31T21:15:59.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:58 smithi171 conmon[41853]: debug 2022-01-31T21:15:58.788+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:58.788829+0000) 2022-01-31T21:15:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:15:59 smithi171 conmon[46715]: debug 2022-01-31T21:15:59.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:59.084647+0000) 2022-01-31T21:15:59.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:15:59 smithi167 conmon[60316]: debug 2022-01-31T21:15:59.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:59.309664+0000) 2022-01-31T21:15:59.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:15:59 smithi171 conmon[51620]: debug 2022-01-31T21:15:59.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:59.664188+0000) 2022-01-31T21:15:59.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:15:59 smithi167 conmon[54076]: debug 2022-01-31T21:15:59.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:59.557396+0000) 2022-01-31T21:15:59.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:15:59 smithi167 conmon[49112]: debug 2022-01-31T21:15:59.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:59.632579+0000) 2022-01-31T21:16:00.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:15:59 smithi171 conmon[41853]: debug 2022-01-31T21:15:59.788+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:15:59.788999+0000) 2022-01-31T21:16:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:00 smithi171 conmon[46715]: debug 2022-01-31T21:16:00.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:00.084851+0000) 2022-01-31T21:16:00.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:00 smithi167 conmon[60316]: debug 2022-01-31T21:16:00.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:00.309829+0000) 2022-01-31T21:16:00.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:00 smithi171 conmon[51620]: debug 2022-01-31T21:16:00.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:00.664402+0000) 2022-01-31T21:16:00.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:00 smithi167 conmon[54076]: debug 2022-01-31T21:16:00.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:00.557556+0000) 2022-01-31T21:16:00.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:00 smithi167 conmon[49112]: debug 2022-01-31T21:16:00.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:00.632761+0000) 2022-01-31T21:16:01.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:00 smithi171 conmon[41853]: debug 2022-01-31T21:16:00.788+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:00.789176+0000) 2022-01-31T21:16:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:01 smithi171 conmon[46715]: debug 2022-01-31T21:16:01.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:01.085009+0000) 2022-01-31T21:16:01.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:01 smithi167 conmon[60316]: debug 2022-01-31T21:16:01.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:01.310014+0000) 2022-01-31T21:16:01.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:01 smithi171 conmon[51620]: debug 2022-01-31T21:16:01.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:01.664570+0000) 2022-01-31T21:16:01.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:01 smithi167 conmon[49112]: debug 2022-01-31T21:16:01.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:01.632932+0000) 2022-01-31T21:16:01.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:01 smithi167 conmon[54076]: debug 2022-01-31T21:16:01.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:01.557728+0000) 2022-01-31T21:16:02.083 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:01 smithi171 conmon[41853]: debug 2022-01-31T21:16:01.788+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:01.789353+0000) 2022-01-31T21:16:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[46715]: debug 2022-01-31T21:16:02.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.085193+0000) 2022-01-31T21:16:02.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:02 smithi167 conmon[60316]: debug 2022-01-31T21:16:02.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.310183+0000) 2022-01-31T21:16:02.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[51620]: debug 2022-01-31T21:16:02.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.664767+0000) 2022-01-31T21:16:02.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:02 smithi167 conmon[60316]: debug 2022-01-31T21:16:02.795+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.797084+0000) 2022-01-31T21:16:02.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:02 smithi167 conmon[49112]: debug 2022-01-31T21:16:02.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.633084+0000) 2022-01-31T21:16:02.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:02 smithi167 conmon[49112]: debug 2022-01-31T21:16:02.797+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.798869+0000) 2022-01-31T21:16:02.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:02 smithi167 conmon[54076]: debug 2022-01-31T21:16:02.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.557924+0000) 2022-01-31T21:16:02.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:02 smithi167 conmon[54076]: debug 2022-01-31T21:16:02.796+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.797748+0000) 2022-01-31T21:16:03.084 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[35325]: debug 2022-01-31T21:16:02.808+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116177 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:03.084 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[46715]: debug 2022-01-31T21:16:02.796+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.796735+0000) 2022-01-31T21:16:03.085 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[51620]: debug 2022-01-31T21:16:02.797+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.798443+0000) 2022-01-31T21:16:03.085 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[41853]: debug 2022-01-31T21:16:02.788+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.789569+0000) 2022-01-31T21:16:03.085 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:02 smithi171 conmon[41853]: debug 2022-01-31T21:16:02.797+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:02.798570+0000) 2022-01-31T21:16:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:03 smithi171 conmon[46715]: debug 2022-01-31T21:16:03.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:03.085399+0000) 2022-01-31T21:16:03.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:03 smithi167 conmon[60316]: debug 2022-01-31T21:16:03.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:03.310361+0000) 2022-01-31T21:16:03.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:03 smithi171 conmon[51620]: debug 2022-01-31T21:16:03.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:03.664948+0000) 2022-01-31T21:16:03.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:03 smithi167 conmon[49112]: debug 2022-01-31T21:16:03.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:03.633254+0000) 2022-01-31T21:16:03.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:03 smithi167 conmon[54076]: debug 2022-01-31T21:16:03.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:03.558107+0000) 2022-01-31T21:16:04.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:03 smithi171 conmon[41853]: debug 2022-01-31T21:16:03.789+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:03.789780+0000) 2022-01-31T21:16:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:04 smithi171 conmon[46715]: debug 2022-01-31T21:16:04.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:04.085579+0000) 2022-01-31T21:16:04.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:04 smithi167 conmon[60316]: debug 2022-01-31T21:16:04.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:04.310493+0000) 2022-01-31T21:16:04.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:04 smithi171 conmon[51620]: debug 2022-01-31T21:16:04.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:04.665064+0000) 2022-01-31T21:16:04.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:04 smithi167 conmon[49112]: debug 2022-01-31T21:16:04.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:04.633407+0000) 2022-01-31T21:16:04.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:04 smithi167 conmon[54076]: debug 2022-01-31T21:16:04.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:04.558244+0000) 2022-01-31T21:16:05.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:04 smithi171 conmon[41853]: debug 2022-01-31T21:16:04.789+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:04.789913+0000) 2022-01-31T21:16:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:05 smithi171 conmon[46715]: debug 2022-01-31T21:16:05.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:05.085721+0000) 2022-01-31T21:16:05.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:05 smithi167 conmon[60316]: debug 2022-01-31T21:16:05.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:05.310743+0000) 2022-01-31T21:16:05.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:05 smithi171 conmon[51620]: debug 2022-01-31T21:16:05.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:05.665247+0000) 2022-01-31T21:16:05.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:05 smithi167 conmon[54076]: debug 2022-01-31T21:16:05.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:05.558458+0000) 2022-01-31T21:16:05.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:05 smithi167 conmon[49112]: debug 2022-01-31T21:16:05.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:05.633583+0000) 2022-01-31T21:16:06.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:05 smithi171 conmon[41853]: debug 2022-01-31T21:16:05.789+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:05.790134+0000) 2022-01-31T21:16:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:06 smithi171 conmon[46715]: debug 2022-01-31T21:16:06.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:06.085943+0000) 2022-01-31T21:16:06.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:06 smithi167 conmon[60316]: debug 2022-01-31T21:16:06.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:06.310934+0000) 2022-01-31T21:16:06.747 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:06 smithi171 conmon[51620]: debug 2022-01-31T21:16:06.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:06.665402+0000) 2022-01-31T21:16:06.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:06 smithi167 conmon[49112]: debug 2022-01-31T21:16:06.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:06.633803+0000) 2022-01-31T21:16:06.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:06 smithi167 conmon[54076]: debug 2022-01-31T21:16:06.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:06.558668+0000) 2022-01-31T21:16:07.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:06 smithi171 conmon[41853]: debug 2022-01-31T21:16:06.789+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:06.790311+0000) 2022-01-31T21:16:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[46715]: debug 2022-01-31T21:16:07.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.086108+0000) 2022-01-31T21:16:07.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:07 smithi167 conmon[60316]: debug 2022-01-31T21:16:07.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.311096+0000) 2022-01-31T21:16:07.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[51620]: debug 2022-01-31T21:16:07.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.665592+0000) 2022-01-31T21:16:07.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:07 smithi167 conmon[49112]: debug 2022-01-31T21:16:07.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.633984+0000) 2022-01-31T21:16:07.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:07 smithi167 conmon[54076]: debug 2022-01-31T21:16:07.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.558859+0000) 2022-01-31T21:16:08.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:07 smithi167 conmon[49112]: debug 2022-01-31T21:16:07.811+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.812618+0000) 2022-01-31T21:16:08.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:07 smithi167 conmon[54076]: debug 2022-01-31T21:16:07.811+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.812683+0000) 2022-01-31T21:16:08.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:07 smithi167 conmon[60316]: debug 2022-01-31T21:16:07.810+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.811872+0000) 2022-01-31T21:16:08.084 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[46715]: debug 2022-01-31T21:16:07.810+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.811674+0000) 2022-01-31T21:16:08.085 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[51620]: debug 2022-01-31T21:16:07.811+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.812133+0000) 2022-01-31T21:16:08.085 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[35325]: debug 2022-01-31T21:16:07.822+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116288 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:08.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[41853]: debug 2022-01-31T21:16:07.790+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.790516+0000) 2022-01-31T21:16:08.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:07 smithi171 conmon[41853]: debug 2022-01-31T21:16:07.812+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:07.813191+0000) 2022-01-31T21:16:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:08 smithi171 conmon[46715]: debug 2022-01-31T21:16:08.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:08.086291+0000) 2022-01-31T21:16:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:08 smithi167 conmon[60316]: debug 2022-01-31T21:16:08.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:08.311253+0000) 2022-01-31T21:16:08.748 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:08 smithi171 conmon[35325]: debug 2022-01-31T21:16:08.650+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:16:08.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:08 smithi171 conmon[51620]: debug 2022-01-31T21:16:08.665+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:08.665813+0000) 2022-01-31T21:16:08.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:08 smithi167 conmon[49112]: debug 2022-01-31T21:16:08.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:08.634157+0000) 2022-01-31T21:16:08.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:08 smithi167 conmon[54076]: debug 2022-01-31T21:16:08.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:08.559066+0000) 2022-01-31T21:16:09.084 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:08 smithi171 conmon[41853]: debug 2022-01-31T21:16:08.790+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:08.790701+0000) 2022-01-31T21:16:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:09 smithi171 conmon[46715]: debug 2022-01-31T21:16:09.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:09.086459+0000) 2022-01-31T21:16:09.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:09 smithi167 conmon[60316]: debug 2022-01-31T21:16:09.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:09.311408+0000) 2022-01-31T21:16:09.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:09 smithi171 conmon[51620]: debug 2022-01-31T21:16:09.665+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:09.665914+0000) 2022-01-31T21:16:09.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:09 smithi167 conmon[49112]: debug 2022-01-31T21:16:09.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:09.634322+0000) 2022-01-31T21:16:09.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:09 smithi167 conmon[54076]: debug 2022-01-31T21:16:09.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:09.559219+0000) 2022-01-31T21:16:10.085 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:09 smithi171 conmon[41853]: debug 2022-01-31T21:16:09.790+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:09.790846+0000) 2022-01-31T21:16:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:10 smithi171 conmon[46715]: debug 2022-01-31T21:16:10.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:10.086675+0000) 2022-01-31T21:16:10.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:10 smithi167 conmon[60316]: debug 2022-01-31T21:16:10.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:10.311604+0000) 2022-01-31T21:16:10.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:10 smithi171 conmon[51620]: debug 2022-01-31T21:16:10.665+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:10.666095+0000) 2022-01-31T21:16:10.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:10 smithi167 conmon[49112]: debug 2022-01-31T21:16:10.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:10.634530+0000) 2022-01-31T21:16:10.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:10 smithi167 conmon[54076]: debug 2022-01-31T21:16:10.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:10.559410+0000) 2022-01-31T21:16:11.085 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:10 smithi171 conmon[41853]: debug 2022-01-31T21:16:10.790+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:10.791028+0000) 2022-01-31T21:16:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:11 smithi171 conmon[46715]: debug 2022-01-31T21:16:11.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:11.086887+0000) 2022-01-31T21:16:11.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:11 smithi167 conmon[60316]: debug 2022-01-31T21:16:11.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:11.311807+0000) 2022-01-31T21:16:11.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:11 smithi171 conmon[51620]: debug 2022-01-31T21:16:11.665+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:11.666319+0000) 2022-01-31T21:16:11.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:11 smithi167 conmon[49112]: debug 2022-01-31T21:16:11.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:11.634737+0000) 2022-01-31T21:16:11.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:11 smithi167 conmon[54076]: debug 2022-01-31T21:16:11.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:11.559599+0000) 2022-01-31T21:16:12.085 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:11 smithi171 conmon[41853]: debug 2022-01-31T21:16:11.790+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:11.791207+0000) 2022-01-31T21:16:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[46715]: debug 2022-01-31T21:16:12.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.087077+0000) 2022-01-31T21:16:12.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:12 smithi167 conmon[60316]: debug 2022-01-31T21:16:12.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.311991+0000) 2022-01-31T21:16:12.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[51620]: debug 2022-01-31T21:16:12.666+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.666501+0000) 2022-01-31T21:16:12.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:12 smithi167 conmon[49112]: debug 2022-01-31T21:16:12.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.634911+0000) 2022-01-31T21:16:12.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:12 smithi167 conmon[54076]: debug 2022-01-31T21:16:12.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.559799+0000) 2022-01-31T21:16:13.086 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[35325]: debug 2022-01-31T21:16:12.837+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116398 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:13.086 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[46715]: debug 2022-01-31T21:16:12.825+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.825824+0000) 2022-01-31T21:16:13.087 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[51620]: debug 2022-01-31T21:16:12.825+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.826312+0000) 2022-01-31T21:16:13.087 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[41853]: debug 2022-01-31T21:16:12.791+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.791401+0000) 2022-01-31T21:16:13.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:12 smithi171 conmon[41853]: debug 2022-01-31T21:16:12.825+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.826499+0000) 2022-01-31T21:16:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:12 smithi167 conmon[49112]: debug 2022-01-31T21:16:12.826+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.827084+0000) 2022-01-31T21:16:13.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:12 smithi167 conmon[54076]: debug 2022-01-31T21:16:12.826+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.826695+0000) 2022-01-31T21:16:13.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:12 smithi167 conmon[60316]: debug 2022-01-31T21:16:12.826+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:12.827251+0000) 2022-01-31T21:16:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:13 smithi171 conmon[46715]: debug 2022-01-31T21:16:13.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:13.087265+0000) 2022-01-31T21:16:13.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:13 smithi167 conmon[60316]: debug 2022-01-31T21:16:13.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:13.312205+0000) 2022-01-31T21:16:13.749 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:13 smithi171 conmon[51620]: debug 2022-01-31T21:16:13.666+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:13.666693+0000) 2022-01-31T21:16:13.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:13 smithi167 conmon[49112]: debug 2022-01-31T21:16:13.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:13.635090+0000) 2022-01-31T21:16:13.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:13 smithi167 conmon[54076]: debug 2022-01-31T21:16:13.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:13.559978+0000) 2022-01-31T21:16:14.085 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:13 smithi171 conmon[41853]: debug 2022-01-31T21:16:13.791+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:13.791552+0000) 2022-01-31T21:16:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:14 smithi171 conmon[46715]: debug 2022-01-31T21:16:14.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:14.087411+0000) 2022-01-31T21:16:14.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:14 smithi167 conmon[60316]: debug 2022-01-31T21:16:14.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:14.312366+0000) 2022-01-31T21:16:14.749 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:14 smithi171 conmon[51620]: debug 2022-01-31T21:16:14.666+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:14.666836+0000) 2022-01-31T21:16:14.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:14 smithi167 conmon[49112]: debug 2022-01-31T21:16:14.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:14.635259+0000) 2022-01-31T21:16:14.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:14 smithi167 conmon[54076]: debug 2022-01-31T21:16:14.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:14.560151+0000) 2022-01-31T21:16:15.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:14 smithi171 conmon[41853]: debug 2022-01-31T21:16:14.791+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:14.791707+0000) 2022-01-31T21:16:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:15 smithi171 conmon[46715]: debug 2022-01-31T21:16:15.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:15.087568+0000) 2022-01-31T21:16:15.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:15 smithi167 conmon[60316]: debug 2022-01-31T21:16:15.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:15.312574+0000) 2022-01-31T21:16:15.749 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:15 smithi171 conmon[51620]: debug 2022-01-31T21:16:15.666+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:15.667046+0000) 2022-01-31T21:16:15.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:15 smithi167 conmon[49112]: debug 2022-01-31T21:16:15.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:15.635428+0000) 2022-01-31T21:16:15.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:15 smithi167 conmon[54076]: debug 2022-01-31T21:16:15.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:15.560335+0000) 2022-01-31T21:16:16.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:15 smithi171 conmon[41853]: debug 2022-01-31T21:16:15.791+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:15.791931+0000) 2022-01-31T21:16:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:16 smithi171 conmon[46715]: debug 2022-01-31T21:16:16.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:16.087789+0000) 2022-01-31T21:16:16.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:16 smithi167 conmon[60316]: debug 2022-01-31T21:16:16.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:16.312735+0000) 2022-01-31T21:16:16.749 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:16 smithi171 conmon[51620]: debug 2022-01-31T21:16:16.666+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:16.667273+0000) 2022-01-31T21:16:16.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:16 smithi167 conmon[49112]: debug 2022-01-31T21:16:16.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:16.635623+0000) 2022-01-31T21:16:16.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:16 smithi167 conmon[54076]: debug 2022-01-31T21:16:16.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:16.560544+0000) 2022-01-31T21:16:17.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:16 smithi171 conmon[41853]: debug 2022-01-31T21:16:16.791+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:16.792124+0000) 2022-01-31T21:16:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[46715]: debug 2022-01-31T21:16:17.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.087978+0000) 2022-01-31T21:16:17.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[60316]: debug 2022-01-31T21:16:17.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.312899+0000) 2022-01-31T21:16:17.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[49112]: debug 2022-01-31T21:16:17.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.635796+0000) 2022-01-31T21:16:17.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[54076]: debug 2022-01-31T21:16:17.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.560647+0000) 2022-01-31T21:16:17.917 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[35325]: debug 2022-01-31T21:16:17.851+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116509 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:17.918 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[46715]: debug 2022-01-31T21:16:17.839+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.840485+0000) 2022-01-31T21:16:17.918 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[41853]: debug 2022-01-31T21:16:17.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.792307+0000) 2022-01-31T21:16:17.919 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[41853]: debug 2022-01-31T21:16:17.839+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.839871+0000) 2022-01-31T21:16:17.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[51620]: debug 2022-01-31T21:16:17.666+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.667492+0000) 2022-01-31T21:16:17.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:17 smithi171 conmon[51620]: debug 2022-01-31T21:16:17.840+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.841014+0000) 2022-01-31T21:16:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[49112]: debug 2022-01-31T21:16:17.840+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.840751+0000) 2022-01-31T21:16:18.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[54076]: debug 2022-01-31T21:16:17.840+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.841302+0000) 2022-01-31T21:16:18.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[60316]: debug 2022-01-31T21:16:18.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:17 smithi167 conmon[60316]: 2022-01-31T21:16:17.840+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:17.841376+0000) 2022-01-31T21:16:18.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:18 smithi171 conmon[46715]: debug 2022-01-31T21:16:18.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:18.088150+0000) 2022-01-31T21:16:18.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:18 smithi167 conmon[60316]: debug 2022-01-31T21:16:18.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:18.313089+0000) 2022-01-31T21:16:18.750 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:18 smithi171 conmon[51620]: debug 2022-01-31T21:16:18.667+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:18.667715+0000) 2022-01-31T21:16:18.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:18 smithi167 conmon[49112]: debug 2022-01-31T21:16:18.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:18.635982+0000) 2022-01-31T21:16:18.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:18 smithi167 conmon[54076]: debug 2022-01-31T21:16:18.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:18.560800+0000) 2022-01-31T21:16:19.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:18 smithi171 conmon[41853]: debug 2022-01-31T21:16:18.791+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:18.792483+0000) 2022-01-31T21:16:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:19 smithi171 conmon[46715]: debug 2022-01-31T21:16:19.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:19.088322+0000) 2022-01-31T21:16:19.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:19 smithi167 conmon[60316]: debug 2022-01-31T21:16:19.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:19.313248+0000) 2022-01-31T21:16:19.750 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:19 smithi171 conmon[51620]: debug 2022-01-31T21:16:19.667+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:19.667865+0000) 2022-01-31T21:16:19.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:19 smithi167 conmon[49112]: debug 2022-01-31T21:16:19.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:19.636090+0000) 2022-01-31T21:16:19.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:19 smithi167 conmon[54076]: debug 2022-01-31T21:16:19.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:19.560948+0000) 2022-01-31T21:16:20.087 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:19 smithi171 conmon[41853]: debug 2022-01-31T21:16:19.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:19.792634+0000) 2022-01-31T21:16:20.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:20 smithi171 conmon[46715]: debug 2022-01-31T21:16:20.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:20.088506+0000) 2022-01-31T21:16:20.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:20 smithi167 conmon[60316]: debug 2022-01-31T21:16:20.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:20.313402+0000) 2022-01-31T21:16:20.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:20 smithi171 conmon[51620]: debug 2022-01-31T21:16:20.667+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:20.668049+0000) 2022-01-31T21:16:20.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:20 smithi167 conmon[49112]: debug 2022-01-31T21:16:20.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:20.636272+0000) 2022-01-31T21:16:20.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:20 smithi167 conmon[54076]: debug 2022-01-31T21:16:20.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:20.561085+0000) 2022-01-31T21:16:21.087 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:20 smithi171 conmon[41853]: debug 2022-01-31T21:16:20.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:20.792820+0000) 2022-01-31T21:16:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:21 smithi171 conmon[46715]: debug 2022-01-31T21:16:21.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:21.088662+0000) 2022-01-31T21:16:21.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:21 smithi167 conmon[60316]: debug 2022-01-31T21:16:21.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:21.313580+0000) 2022-01-31T21:16:21.750 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:21 smithi171 conmon[51620]: debug 2022-01-31T21:16:21.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:21.668206+0000) 2022-01-31T21:16:21.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:21 smithi167 conmon[49112]: debug 2022-01-31T21:16:21.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:21.636423+0000) 2022-01-31T21:16:21.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:21 smithi167 conmon[54076]: debug 2022-01-31T21:16:21.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:21.561252+0000) 2022-01-31T21:16:22.087 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:21 smithi171 conmon[41853]: debug 2022-01-31T21:16:21.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:21.792980+0000) 2022-01-31T21:16:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[46715]: debug 2022-01-31T21:16:22.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.088846+0000) 2022-01-31T21:16:22.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:22 smithi167 conmon[60316]: debug 2022-01-31T21:16:22.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.313678+0000) 2022-01-31T21:16:22.750 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[51620]: debug 2022-01-31T21:16:22.667+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.668415+0000) 2022-01-31T21:16:22.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:22 smithi167 conmon[49112]: debug 2022-01-31T21:16:22.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.636608+0000) 2022-01-31T21:16:22.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:22 smithi167 conmon[54076]: debug 2022-01-31T21:16:22.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.561448+0000) 2022-01-31T21:16:23.087 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[35325]: debug 2022-01-31T21:16:22.865+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116623 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:23.088 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[46715]: debug 2022-01-31T21:16:22.853+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.854170+0000) 2022-01-31T21:16:23.088 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[51620]: debug 2022-01-31T21:16:22.854+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.855269+0000) 2022-01-31T21:16:23.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[41853]: debug 2022-01-31T21:16:22.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.793149+0000) 2022-01-31T21:16:23.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:22 smithi171 conmon[41853]: debug 2022-01-31T21:16:22.854+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.855605+0000) 2022-01-31T21:16:23.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:22 smithi167 conmon[49112]: debug 2022-01-31T21:16:22.854+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.855092+0000) 2022-01-31T21:16:23.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:22 smithi167 conmon[54076]: debug 2022-01-31T21:16:22.854+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.854738+0000) 2022-01-31T21:16:23.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:22 smithi167 conmon[60316]: debug 2022-01-31T21:16:22.855+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:22.855906+0000) 2022-01-31T21:16:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:23 smithi171 conmon[46715]: debug 2022-01-31T21:16:23.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:23.088930+0000) 2022-01-31T21:16:23.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:23 smithi167 conmon[60316]: debug 2022-01-31T21:16:23.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:23.313847+0000) 2022-01-31T21:16:23.751 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:23 smithi171 conmon[35325]: debug 2022-01-31T21:16:23.651+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:16:23.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:23 smithi171 conmon[51620]: debug 2022-01-31T21:16:23.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:23.668620+0000) 2022-01-31T21:16:23.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:23 smithi167 conmon[49112]: debug 2022-01-31T21:16:23.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:23.636781+0000) 2022-01-31T21:16:23.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:23 smithi167 conmon[54076]: debug 2022-01-31T21:16:23.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:23.561636+0000) 2022-01-31T21:16:24.087 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:23 smithi171 conmon[41853]: debug 2022-01-31T21:16:23.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:23.793308+0000) 2022-01-31T21:16:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:24 smithi171 conmon[46715]: debug 2022-01-31T21:16:24.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:24.089084+0000) 2022-01-31T21:16:24.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:24 smithi167 conmon[60316]: debug 2022-01-31T21:16:24.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:24.313982+0000) 2022-01-31T21:16:24.750 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:24 smithi171 conmon[51620]: debug 2022-01-31T21:16:24.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:24.668808+0000) 2022-01-31T21:16:24.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:24 smithi167 conmon[49112]: debug 2022-01-31T21:16:24.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:24.636880+0000) 2022-01-31T21:16:24.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:24 smithi167 conmon[54076]: debug 2022-01-31T21:16:24.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:24.561793+0000) 2022-01-31T21:16:25.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:24 smithi171 conmon[41853]: debug 2022-01-31T21:16:24.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:24.793472+0000) 2022-01-31T21:16:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:25 smithi171 conmon[46715]: debug 2022-01-31T21:16:25.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:25.089238+0000) 2022-01-31T21:16:25.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:25 smithi167 conmon[60316]: debug 2022-01-31T21:16:25.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:25.314139+0000) 2022-01-31T21:16:25.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:25 smithi171 conmon[51620]: debug 2022-01-31T21:16:25.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:25.668958+0000) 2022-01-31T21:16:25.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:25 smithi167 conmon[49112]: debug 2022-01-31T21:16:25.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:25.637039+0000) 2022-01-31T21:16:25.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:25 smithi167 conmon[54076]: debug 2022-01-31T21:16:25.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:25.561968+0000) 2022-01-31T21:16:26.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:25 smithi171 conmon[41853]: debug 2022-01-31T21:16:25.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:25.793645+0000) 2022-01-31T21:16:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:26 smithi171 conmon[46715]: debug 2022-01-31T21:16:26.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:26.089443+0000) 2022-01-31T21:16:26.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:26 smithi167 conmon[60316]: debug 2022-01-31T21:16:26.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:26.314297+0000) 2022-01-31T21:16:26.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:26 smithi171 conmon[51620]: debug 2022-01-31T21:16:26.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:26.669116+0000) 2022-01-31T21:16:26.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:26 smithi167 conmon[54076]: debug 2022-01-31T21:16:26.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:26.562166+0000) 2022-01-31T21:16:26.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:26 smithi167 conmon[49112]: debug 2022-01-31T21:16:26.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:26.637227+0000) 2022-01-31T21:16:27.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:26 smithi171 conmon[41853]: debug 2022-01-31T21:16:26.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:26.793806+0000) 2022-01-31T21:16:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[46715]: debug 2022-01-31T21:16:27.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.089599+0000) 2022-01-31T21:16:27.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:27 smithi167 conmon[60316]: debug 2022-01-31T21:16:27.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.314436+0000) 2022-01-31T21:16:27.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[51620]: debug 2022-01-31T21:16:27.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.669288+0000) 2022-01-31T21:16:27.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:27 smithi167 conmon[49112]: debug 2022-01-31T21:16:27.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.637409+0000) 2022-01-31T21:16:27.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:27 smithi167 conmon[54076]: debug 2022-01-31T21:16:27.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.562377+0000) 2022-01-31T21:16:28.088 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[35325]: debug 2022-01-31T21:16:27.880+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116733 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:28.089 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[46715]: debug 2022-01-31T21:16:27.868+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.869690+0000) 2022-01-31T21:16:28.089 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[51620]: debug 2022-01-31T21:16:27.868+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.869379+0000) 2022-01-31T21:16:28.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[41853]: debug 2022-01-31T21:16:27.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.793963+0000) 2022-01-31T21:16:28.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:27 smithi171 conmon[41853]: debug 2022-01-31T21:16:27.869+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.869965+0000) 2022-01-31T21:16:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:27 smithi167 conmon[49112]: debug 2022-01-31T21:16:27.868+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.868599+0000) 2022-01-31T21:16:28.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:27 smithi167 conmon[54076]: debug 2022-01-31T21:16:27.869+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.869583+0000) 2022-01-31T21:16:28.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:27 smithi167 conmon[60316]: debug 2022-01-31T21:16:27.869+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:27.869894+0000) 2022-01-31T21:16:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:28 smithi171 conmon[46715]: debug 2022-01-31T21:16:28.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:28.089806+0000) 2022-01-31T21:16:28.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:28 smithi167 conmon[60316]: debug 2022-01-31T21:16:28.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:28.314632+0000) 2022-01-31T21:16:28.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:28 smithi171 conmon[51620]: debug 2022-01-31T21:16:28.667+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:28.669516+0000) 2022-01-31T21:16:28.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:28 smithi167 conmon[49112]: debug 2022-01-31T21:16:28.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:28.637602+0000) 2022-01-31T21:16:28.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:28 smithi167 conmon[54076]: debug 2022-01-31T21:16:28.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:28.562558+0000) 2022-01-31T21:16:29.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:28 smithi171 conmon[41853]: debug 2022-01-31T21:16:28.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:28.794144+0000) 2022-01-31T21:16:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:29 smithi171 conmon[46715]: debug 2022-01-31T21:16:29.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:29.089974+0000) 2022-01-31T21:16:29.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:29 smithi167 conmon[60316]: debug 2022-01-31T21:16:29.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:29.314804+0000) 2022-01-31T21:16:29.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:29 smithi171 conmon[51620]: debug 2022-01-31T21:16:29.667+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:29.669694+0000) 2022-01-31T21:16:29.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:29 smithi167 conmon[49112]: debug 2022-01-31T21:16:29.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:29.637741+0000) 2022-01-31T21:16:29.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:29 smithi167 conmon[54076]: debug 2022-01-31T21:16:29.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:29.562739+0000) 2022-01-31T21:16:30.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:29 smithi171 conmon[41853]: debug 2022-01-31T21:16:29.792+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:29.794292+0000) 2022-01-31T21:16:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:30 smithi171 conmon[46715]: debug 2022-01-31T21:16:30.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:30.090192+0000) 2022-01-31T21:16:30.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:30 smithi167 conmon[60316]: debug 2022-01-31T21:16:30.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:30.315048+0000) 2022-01-31T21:16:30.751 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:30 smithi171 conmon[51620]: debug 2022-01-31T21:16:30.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:30.669857+0000) 2022-01-31T21:16:30.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:30 smithi167 conmon[49112]: debug 2022-01-31T21:16:30.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:30.637925+0000) 2022-01-31T21:16:30.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:30 smithi167 conmon[54076]: debug 2022-01-31T21:16:30.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:30.562904+0000) 2022-01-31T21:16:31.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:30 smithi171 conmon[41853]: debug 2022-01-31T21:16:30.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:30.794417+0000) 2022-01-31T21:16:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:31 smithi171 conmon[46715]: debug 2022-01-31T21:16:31.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:31.090368+0000) 2022-01-31T21:16:31.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:31 smithi167 conmon[60316]: debug 2022-01-31T21:16:31.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:31.315209+0000) 2022-01-31T21:16:31.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:31 smithi171 conmon[51620]: debug 2022-01-31T21:16:31.669+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:31.670014+0000) 2022-01-31T21:16:31.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:31 smithi167 conmon[49112]: debug 2022-01-31T21:16:31.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:31.638128+0000) 2022-01-31T21:16:31.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:31 smithi167 conmon[54076]: debug 2022-01-31T21:16:31.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:31.563091+0000) 2022-01-31T21:16:32.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:31 smithi171 conmon[41853]: debug 2022-01-31T21:16:31.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:31.794570+0000) 2022-01-31T21:16:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[46715]: debug 2022-01-31T21:16:32.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.090559+0000) 2022-01-31T21:16:32.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:32 smithi167 conmon[60316]: debug 2022-01-31T21:16:32.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.315321+0000) 2022-01-31T21:16:32.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[51620]: debug 2022-01-31T21:16:32.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.670220+0000) 2022-01-31T21:16:32.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:32 smithi167 conmon[49112]: debug 2022-01-31T21:16:32.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.638319+0000) 2022-01-31T21:16:32.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:32 smithi167 conmon[54076]: debug 2022-01-31T21:16:32.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.563293+0000) 2022-01-31T21:16:32.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:32 smithi167 conmon[54076]: debug 2022-01-31T21:16:32.882+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.883469+0000) 2022-01-31T21:16:32.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:32 smithi167 conmon[60316]: debug 2022-01-31T21:16:32.883+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.883720+0000) 2022-01-31T21:16:33.089 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[35325]: debug 2022-01-31T21:16:32.893+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116843 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:33.090 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[46715]: debug 2022-01-31T21:16:32.882+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.884529+0000) 2022-01-31T21:16:33.090 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[51620]: debug 2022-01-31T21:16:32.881+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.883042+0000) 2022-01-31T21:16:33.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[41853]: debug 2022-01-31T21:16:32.793+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.794788+0000) 2022-01-31T21:16:33.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:32 smithi171 conmon[41853]: debug 2022-01-31T21:16:32.881+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.883337+0000) 2022-01-31T21:16:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:32 smithi167 conmon[49112]: debug 2022-01-31T21:16:32.883+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:32.883923+0000) 2022-01-31T21:16:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:33 smithi171 conmon[46715]: debug 2022-01-31T21:16:33.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:33.090767+0000) 2022-01-31T21:16:33.562 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:33 smithi167 conmon[60316]: debug 2022-01-31T21:16:33.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:33.315537+0000) 2022-01-31T21:16:33.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:33 smithi171 conmon[51620]: debug 2022-01-31T21:16:33.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:33.670452+0000) 2022-01-31T21:16:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:33 smithi167 conmon[49112]: debug 2022-01-31T21:16:33.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:33.638438+0000) 2022-01-31T21:16:33.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:33 smithi167 conmon[54076]: debug 2022-01-31T21:16:33.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:33.563533+0000) 2022-01-31T21:16:34.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:33 smithi171 conmon[41853]: debug 2022-01-31T21:16:33.794+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:33.794965+0000) 2022-01-31T21:16:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:34 smithi171 conmon[46715]: debug 2022-01-31T21:16:34.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:34.090947+0000) 2022-01-31T21:16:34.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:34 smithi167 conmon[60316]: debug 2022-01-31T21:16:34.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:34.315685+0000) 2022-01-31T21:16:34.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:34 smithi171 conmon[51620]: debug 2022-01-31T21:16:34.668+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:34.670595+0000) 2022-01-31T21:16:34.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:34 smithi167 conmon[49112]: debug 2022-01-31T21:16:34.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:34.638597+0000) 2022-01-31T21:16:34.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:34 smithi167 conmon[54076]: debug 2022-01-31T21:16:34.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:34.563686+0000) 2022-01-31T21:16:35.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:34 smithi171 conmon[41853]: debug 2022-01-31T21:16:34.794+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:34.795105+0000) 2022-01-31T21:16:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:35 smithi171 conmon[46715]: debug 2022-01-31T21:16:35.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:35.091147+0000) 2022-01-31T21:16:35.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:35 smithi167 conmon[60316]: debug 2022-01-31T21:16:35.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:35.315849+0000) 2022-01-31T21:16:35.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:35 smithi171 conmon[51620]: debug 2022-01-31T21:16:35.669+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:35.670795+0000) 2022-01-31T21:16:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:35 smithi167 conmon[49112]: debug 2022-01-31T21:16:35.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:35.638776+0000) 2022-01-31T21:16:35.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:35 smithi167 conmon[54076]: debug 2022-01-31T21:16:35.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:35.563842+0000) 2022-01-31T21:16:36.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:35 smithi171 conmon[41853]: debug 2022-01-31T21:16:35.794+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:35.795262+0000) 2022-01-31T21:16:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:36 smithi171 conmon[46715]: debug 2022-01-31T21:16:36.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:36.091308+0000) 2022-01-31T21:16:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:36 smithi167 conmon[60316]: debug 2022-01-31T21:16:36.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:36.316002+0000) 2022-01-31T21:16:36.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:36 smithi171 conmon[51620]: debug 2022-01-31T21:16:36.669+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:36.670930+0000) 2022-01-31T21:16:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:36 smithi167 conmon[49112]: debug 2022-01-31T21:16:36.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:36.638912+0000) 2022-01-31T21:16:36.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:36 smithi167 conmon[54076]: debug 2022-01-31T21:16:36.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:36.564035+0000) 2022-01-31T21:16:37.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:36 smithi171 conmon[41853]: debug 2022-01-31T21:16:36.794+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:36.795439+0000) 2022-01-31T21:16:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[46715]: debug 2022-01-31T21:16:37.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.091493+0000) 2022-01-31T21:16:37.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:37 smithi167 conmon[60316]: debug 2022-01-31T21:16:37.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.316213+0000) 2022-01-31T21:16:37.752 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[51620]: debug 2022-01-31T21:16:37.670+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.671081+0000) 2022-01-31T21:16:37.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:37 smithi167 conmon[54076]: debug 2022-01-31T21:16:37.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.564220+0000) 2022-01-31T21:16:37.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:37 smithi167 conmon[49112]: debug 2022-01-31T21:16:37.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.639060+0000) 2022-01-31T21:16:38.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[35325]: debug 2022-01-31T21:16:37.907+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 116954 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:38.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[41853]: debug 2022-01-31T21:16:37.794+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.795618+0000) 2022-01-31T21:16:38.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[41853]: debug 2022-01-31T21:16:37.895+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.897314+0000) 2022-01-31T21:16:38.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[46715]: debug 2022-01-31T21:16:37.895+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.897548+0000) 2022-01-31T21:16:38.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:37 smithi171 conmon[51620]: debug 2022-01-31T21:16:37.895+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.897660+0000) 2022-01-31T21:16:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:37 smithi167 conmon[49112]: debug 2022-01-31T21:16:37.896+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.898414+0000) 2022-01-31T21:16:38.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:37 smithi167 conmon[54076]: debug 2022-01-31T21:16:37.895+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.897254+0000) 2022-01-31T21:16:38.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:37 smithi167 conmon[60316]: debug 2022-01-31T21:16:37.896+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:37.897544+0000) 2022-01-31T21:16:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:38 smithi171 conmon[46715]: debug 2022-01-31T21:16:38.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:38.091707+0000) 2022-01-31T21:16:38.562 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:38 smithi167 conmon[60316]: debug 2022-01-31T21:16:38.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:38.316388+0000) 2022-01-31T21:16:38.753 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:38 smithi171 conmon[35325]: debug 2022-01-31T21:16:38.651+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:16:38.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:38 smithi171 conmon[51620]: debug 2022-01-31T21:16:38.670+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:38.671300+0000) 2022-01-31T21:16:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:38 smithi167 conmon[49112]: debug 2022-01-31T21:16:38.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:38.639264+0000) 2022-01-31T21:16:38.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:38 smithi167 conmon[54076]: debug 2022-01-31T21:16:38.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:38.564419+0000) 2022-01-31T21:16:39.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:38 smithi171 conmon[41853]: debug 2022-01-31T21:16:38.794+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:38.795800+0000) 2022-01-31T21:16:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:39 smithi171 conmon[46715]: debug 2022-01-31T21:16:39.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:39.091898+0000) 2022-01-31T21:16:39.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:39 smithi167 conmon[60316]: debug 2022-01-31T21:16:39.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:39.316540+0000) 2022-01-31T21:16:39.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:39 smithi171 conmon[51620]: debug 2022-01-31T21:16:39.670+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:39.671436+0000) 2022-01-31T21:16:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:39 smithi167 conmon[49112]: debug 2022-01-31T21:16:39.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:39.639432+0000) 2022-01-31T21:16:39.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:39 smithi167 conmon[54076]: debug 2022-01-31T21:16:39.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:39.564564+0000) 2022-01-31T21:16:40.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:39 smithi171 conmon[41853]: debug 2022-01-31T21:16:39.795+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:39.795959+0000) 2022-01-31T21:16:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:40 smithi171 conmon[46715]: debug 2022-01-31T21:16:40.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:40.092074+0000) 2022-01-31T21:16:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:40 smithi167 conmon[60316]: debug 2022-01-31T21:16:40.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:40.316711+0000) 2022-01-31T21:16:40.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:40 smithi171 conmon[51620]: debug 2022-01-31T21:16:40.670+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:40.671637+0000) 2022-01-31T21:16:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:40 smithi167 conmon[49112]: debug 2022-01-31T21:16:40.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:40.639580+0000) 2022-01-31T21:16:40.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:40 smithi167 conmon[54076]: debug 2022-01-31T21:16:40.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:40.564733+0000) 2022-01-31T21:16:41.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:40 smithi171 conmon[41853]: debug 2022-01-31T21:16:40.795+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:40.796150+0000) 2022-01-31T21:16:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:41 smithi171 conmon[46715]: debug 2022-01-31T21:16:41.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:41.092234+0000) 2022-01-31T21:16:41.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:41 smithi167 conmon[60316]: debug 2022-01-31T21:16:41.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:41.316963+0000) 2022-01-31T21:16:41.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:41 smithi171 conmon[51620]: debug 2022-01-31T21:16:41.671+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:41.671813+0000) 2022-01-31T21:16:41.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:41 smithi167 conmon[54076]: debug 2022-01-31T21:16:41.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:41.564923+0000) 2022-01-31T21:16:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:41 smithi167 conmon[49112]: debug 2022-01-31T21:16:41.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:41.639731+0000) 2022-01-31T21:16:42.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:41 smithi171 conmon[41853]: debug 2022-01-31T21:16:41.795+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:41.796343+0000) 2022-01-31T21:16:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[46715]: debug 2022-01-31T21:16:42.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.092392+0000) 2022-01-31T21:16:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:42 smithi167 conmon[60316]: debug 2022-01-31T21:16:42.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.317113+0000) 2022-01-31T21:16:42.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[51620]: debug 2022-01-31T21:16:42.671+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.671939+0000) 2022-01-31T21:16:42.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:42 smithi167 conmon[49112]: debug 2022-01-31T21:16:42.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.639878+0000) 2022-01-31T21:16:42.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:42 smithi167 conmon[54076]: debug 2022-01-31T21:16:42.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.565084+0000) 2022-01-31T21:16:43.091 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[35325]: debug 2022-01-31T21:16:42.923+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117064 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:43.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[46715]: debug 2022-01-31T21:16:42.912+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.912938+0000) 2022-01-31T21:16:43.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[51620]: debug 2022-01-31T21:16:42.910+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.911607+0000) 2022-01-31T21:16:43.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[41853]: debug 2022-01-31T21:16:42.795+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.796553+0000) 2022-01-31T21:16:43.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:42 smithi171 conmon[41853]: debug 2022-01-31T21:16:42.911+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.912045+0000) 2022-01-31T21:16:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:42 smithi167 conmon[49112]: debug 2022-01-31T21:16:42.911+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.912531+0000) 2022-01-31T21:16:43.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:42 smithi167 conmon[54076]: debug 2022-01-31T21:16:42.909+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.911302+0000) 2022-01-31T21:16:43.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:42 smithi167 conmon[60316]: debug 2022-01-31T21:16:42.910+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:42.911542+0000) 2022-01-31T21:16:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:43 smithi171 conmon[46715]: debug 2022-01-31T21:16:43.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:43.092583+0000) 2022-01-31T21:16:43.564 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:43 smithi167 conmon[60316]: debug 2022-01-31T21:16:43.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:43.317308+0000) 2022-01-31T21:16:43.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:43 smithi171 conmon[51620]: debug 2022-01-31T21:16:43.671+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:43.672086+0000) 2022-01-31T21:16:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:43 smithi167 conmon[49112]: debug 2022-01-31T21:16:43.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:43.640094+0000) 2022-01-31T21:16:43.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:43 smithi167 conmon[54076]: debug 2022-01-31T21:16:43.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:43.565260+0000) 2022-01-31T21:16:44.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:43 smithi171 conmon[41853]: debug 2022-01-31T21:16:43.796+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:43.796742+0000) 2022-01-31T21:16:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:44 smithi171 conmon[46715]: debug 2022-01-31T21:16:44.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:44.092787+0000) 2022-01-31T21:16:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:44 smithi167 conmon[60316]: debug 2022-01-31T21:16:44.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:44.317457+0000) 2022-01-31T21:16:44.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:44 smithi171 conmon[51620]: debug 2022-01-31T21:16:44.671+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:44.672224+0000) 2022-01-31T21:16:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:44 smithi167 conmon[49112]: debug 2022-01-31T21:16:44.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:44.640305+0000) 2022-01-31T21:16:44.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:44 smithi167 conmon[54076]: debug 2022-01-31T21:16:44.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:44.565377+0000) 2022-01-31T21:16:45.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:44 smithi171 conmon[41853]: debug 2022-01-31T21:16:44.796+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:44.796868+0000) 2022-01-31T21:16:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:45 smithi171 conmon[46715]: debug 2022-01-31T21:16:45.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:45.092983+0000) 2022-01-31T21:16:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:45 smithi167 conmon[60316]: debug 2022-01-31T21:16:45.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:45.317614+0000) 2022-01-31T21:16:45.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:45 smithi171 conmon[51620]: debug 2022-01-31T21:16:45.671+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:45.672398+0000) 2022-01-31T21:16:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:45 smithi167 conmon[49112]: debug 2022-01-31T21:16:45.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:45.640553+0000) 2022-01-31T21:16:45.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:45 smithi167 conmon[54076]: debug 2022-01-31T21:16:45.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:45.565540+0000) 2022-01-31T21:16:46.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:45 smithi171 conmon[41853]: debug 2022-01-31T21:16:45.796+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:45.797075+0000) 2022-01-31T21:16:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:46 smithi171 conmon[46715]: debug 2022-01-31T21:16:46.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:46.093113+0000) 2022-01-31T21:16:46.564 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:46 smithi167 conmon[60316]: debug 2022-01-31T21:16:46.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:46.317748+0000) 2022-01-31T21:16:46.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:46 smithi171 conmon[51620]: debug 2022-01-31T21:16:46.671+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:46.672577+0000) 2022-01-31T21:16:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:46 smithi167 conmon[49112]: debug 2022-01-31T21:16:46.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:46.640764+0000) 2022-01-31T21:16:46.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:46 smithi167 conmon[54076]: debug 2022-01-31T21:16:46.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:46.565678+0000) 2022-01-31T21:16:47.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:46 smithi171 conmon[41853]: debug 2022-01-31T21:16:46.796+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:46.797259+0000) 2022-01-31T21:16:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[46715]: debug 2022-01-31T21:16:47.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.093270+0000) 2022-01-31T21:16:47.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:47 smithi167 conmon[60316]: debug 2022-01-31T21:16:47.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.317869+0000) 2022-01-31T21:16:47.754 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[51620]: debug 2022-01-31T21:16:47.672+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.672778+0000) 2022-01-31T21:16:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:47 smithi167 conmon[49112]: debug 2022-01-31T21:16:47.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.640975+0000) 2022-01-31T21:16:47.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:47 smithi167 conmon[54076]: debug 2022-01-31T21:16:47.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.565834+0000) 2022-01-31T21:16:48.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[35325]: debug 2022-01-31T21:16:47.942+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117175 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:48.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[46715]: debug 2022-01-31T21:16:47.927+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.928563+0000) 2022-01-31T21:16:48.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[51620]: debug 2022-01-31T21:16:47.925+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.926595+0000) 2022-01-31T21:16:48.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[41853]: debug 2022-01-31T21:16:47.796+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.797419+0000) 2022-01-31T21:16:48.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:47 smithi171 conmon[41853]: debug 2022-01-31T21:16:47.926+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.927499+0000) 2022-01-31T21:16:48.296 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:47 smithi167 conmon[49112]: debug 2022-01-31T21:16:47.925+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.926957+0000) 2022-01-31T21:16:48.297 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:47 smithi167 conmon[54076]: debug 2022-01-31T21:16:47.925+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.926711+0000) 2022-01-31T21:16:48.297 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:47 smithi167 conmon[60316]: debug 2022-01-31T21:16:47.924+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:47.926432+0000) 2022-01-31T21:16:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:48 smithi171 conmon[46715]: debug 2022-01-31T21:16:48.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:48.093444+0000) 2022-01-31T21:16:48.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:48 smithi167 conmon[60316]: debug 2022-01-31T21:16:48.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:48.318037+0000) 2022-01-31T21:16:48.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:48 smithi171 conmon[51620]: debug 2022-01-31T21:16:48.672+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:48.672991+0000) 2022-01-31T21:16:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:48 smithi167 conmon[49112]: debug 2022-01-31T21:16:48.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:48.641183+0000) 2022-01-31T21:16:48.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:48 smithi167 conmon[54076]: debug 2022-01-31T21:16:48.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:48.566028+0000) 2022-01-31T21:16:49.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:48 smithi171 conmon[41853]: debug 2022-01-31T21:16:48.796+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:48.797547+0000) 2022-01-31T21:16:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:49 smithi171 conmon[46715]: debug 2022-01-31T21:16:49.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:49.093606+0000) 2022-01-31T21:16:49.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:49 smithi167 conmon[60316]: debug 2022-01-31T21:16:49.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:49.318151+0000) 2022-01-31T21:16:49.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:49 smithi171 conmon[51620]: debug 2022-01-31T21:16:49.672+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:49.673139+0000) 2022-01-31T21:16:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:49 smithi167 conmon[49112]: debug 2022-01-31T21:16:49.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:49.641343+0000) 2022-01-31T21:16:49.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:49 smithi167 conmon[54076]: debug 2022-01-31T21:16:49.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:49.566162+0000) 2022-01-31T21:16:50.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:49 smithi171 conmon[41853]: debug 2022-01-31T21:16:49.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:49.797686+0000) 2022-01-31T21:16:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:50 smithi171 conmon[46715]: debug 2022-01-31T21:16:50.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:50.093831+0000) 2022-01-31T21:16:50.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:50 smithi167 conmon[60316]: debug 2022-01-31T21:16:50.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:50.318351+0000) 2022-01-31T21:16:50.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:50 smithi171 conmon[51620]: debug 2022-01-31T21:16:50.672+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:50.673342+0000) 2022-01-31T21:16:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:50 smithi167 conmon[49112]: debug 2022-01-31T21:16:50.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:50.641533+0000) 2022-01-31T21:16:50.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:50 smithi167 conmon[54076]: debug 2022-01-31T21:16:50.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:50.566370+0000) 2022-01-31T21:16:51.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:50 smithi171 conmon[41853]: debug 2022-01-31T21:16:50.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:50.797896+0000) 2022-01-31T21:16:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:51 smithi171 conmon[46715]: debug 2022-01-31T21:16:51.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:51.093984+0000) 2022-01-31T21:16:51.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:51 smithi167 conmon[60316]: debug 2022-01-31T21:16:51.317+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:51.318535+0000) 2022-01-31T21:16:51.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:51 smithi171 conmon[51620]: debug 2022-01-31T21:16:51.672+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:51.673542+0000) 2022-01-31T21:16:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:51 smithi167 conmon[49112]: debug 2022-01-31T21:16:51.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:51.641701+0000) 2022-01-31T21:16:51.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:51 smithi167 conmon[54076]: debug 2022-01-31T21:16:51.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:51.566564+0000) 2022-01-31T21:16:52.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:51 smithi171 conmon[41853]: debug 2022-01-31T21:16:51.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:51.798077+0000) 2022-01-31T21:16:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[46715]: debug 2022-01-31T21:16:52.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.094134+0000) 2022-01-31T21:16:52.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:52 smithi167 conmon[60316]: debug 2022-01-31T21:16:52.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.318721+0000) 2022-01-31T21:16:52.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[51620]: debug 2022-01-31T21:16:52.673+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.673747+0000) 2022-01-31T21:16:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:52 smithi167 conmon[49112]: debug 2022-01-31T21:16:52.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.641833+0000) 2022-01-31T21:16:52.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:52 smithi167 conmon[54076]: debug 2022-01-31T21:16:52.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.566747+0000) 2022-01-31T21:16:53.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[35325]: debug 2022-01-31T21:16:52.956+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117284 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:53.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[46715]: debug 2022-01-31T21:16:52.945+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.946686+0000) 2022-01-31T21:16:53.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[41853]: debug 2022-01-31T21:16:52.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.798277+0000) 2022-01-31T21:16:53.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[41853]: debug 2022-01-31T21:16:52.945+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.946694+0000) 2022-01-31T21:16:53.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:52 smithi171 conmon[51620]: debug 2022-01-31T21:16:52.944+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.945373+0000) 2022-01-31T21:16:53.297 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:52 smithi167 conmon[49112]: debug 2022-01-31T21:16:52.944+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.945921+0000) 2022-01-31T21:16:53.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:52 smithi167 conmon[54076]: debug 2022-01-31T21:16:52.944+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.945493+0000) 2022-01-31T21:16:53.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:52 smithi167 conmon[60316]: debug 2022-01-31T21:16:52.943+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:52.945193+0000) 2022-01-31T21:16:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:53 smithi171 conmon[46715]: debug 2022-01-31T21:16:53.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:53.094311+0000) 2022-01-31T21:16:53.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:53 smithi167 conmon[60316]: debug 2022-01-31T21:16:53.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:53.318922+0000) 2022-01-31T21:16:53.755 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:53 smithi171 conmon[35325]: debug 2022-01-31T21:16:53.652+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:16:53.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:53 smithi171 conmon[51620]: debug 2022-01-31T21:16:53.673+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:53.673924+0000) 2022-01-31T21:16:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:53 smithi167 conmon[49112]: debug 2022-01-31T21:16:53.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:53.641984+0000) 2022-01-31T21:16:53.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:53 smithi167 conmon[54076]: debug 2022-01-31T21:16:53.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:53.566927+0000) 2022-01-31T21:16:54.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:53 smithi171 conmon[41853]: debug 2022-01-31T21:16:53.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:53.798432+0000) 2022-01-31T21:16:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:54 smithi171 conmon[46715]: debug 2022-01-31T21:16:54.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:54.094477+0000) 2022-01-31T21:16:54.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:54 smithi167 conmon[60316]: debug 2022-01-31T21:16:54.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:54.319097+0000) 2022-01-31T21:16:54.755 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:54 smithi171 conmon[51620]: debug 2022-01-31T21:16:54.673+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:54.674044+0000) 2022-01-31T21:16:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:54 smithi167 conmon[49112]: debug 2022-01-31T21:16:54.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:54.642181+0000) 2022-01-31T21:16:54.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:54 smithi167 conmon[54076]: debug 2022-01-31T21:16:54.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:54.567132+0000) 2022-01-31T21:16:55.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:54 smithi171 conmon[41853]: debug 2022-01-31T21:16:54.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:54.798543+0000) 2022-01-31T21:16:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:55 smithi171 conmon[46715]: debug 2022-01-31T21:16:55.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:55.094632+0000) 2022-01-31T21:16:55.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:55 smithi167 conmon[60316]: debug 2022-01-31T21:16:55.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:55.319283+0000) 2022-01-31T21:16:55.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:55 smithi171 conmon[51620]: debug 2022-01-31T21:16:55.673+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:55.674196+0000) 2022-01-31T21:16:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:55 smithi167 conmon[49112]: debug 2022-01-31T21:16:55.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:55.642367+0000) 2022-01-31T21:16:55.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:55 smithi167 conmon[54076]: debug 2022-01-31T21:16:55.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:55.567345+0000) 2022-01-31T21:16:56.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:55 smithi171 conmon[41853]: debug 2022-01-31T21:16:55.797+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:55.798723+0000) 2022-01-31T21:16:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:56 smithi171 conmon[46715]: debug 2022-01-31T21:16:56.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:56.094837+0000) 2022-01-31T21:16:56.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:56 smithi167 conmon[60316]: debug 2022-01-31T21:16:56.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:56.319490+0000) 2022-01-31T21:16:56.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:56 smithi171 conmon[51620]: debug 2022-01-31T21:16:56.673+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:56.674387+0000) 2022-01-31T21:16:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:56 smithi167 conmon[49112]: debug 2022-01-31T21:16:56.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:56.642584+0000) 2022-01-31T21:16:56.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:56 smithi167 conmon[54076]: debug 2022-01-31T21:16:56.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:56.567518+0000) 2022-01-31T21:16:57.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:56 smithi171 conmon[41853]: debug 2022-01-31T21:16:56.798+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:56.798942+0000) 2022-01-31T21:16:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[46715]: debug 2022-01-31T21:16:57.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.095017+0000) 2022-01-31T21:16:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:57 smithi167 conmon[60316]: debug 2022-01-31T21:16:57.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.319735+0000) 2022-01-31T21:16:57.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[51620]: debug 2022-01-31T21:16:57.674+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.674570+0000) 2022-01-31T21:16:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:57 smithi167 conmon[49112]: debug 2022-01-31T21:16:57.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.642736+0000) 2022-01-31T21:16:57.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:57 smithi167 conmon[54076]: debug 2022-01-31T21:16:57.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.567643+0000) 2022-01-31T21:16:58.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[35325]: debug 2022-01-31T21:16:57.970+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117395 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:16:58.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[46715]: debug 2022-01-31T21:16:57.960+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.961341+0000) 2022-01-31T21:16:58.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[51620]: debug 2022-01-31T21:16:57.958+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.959728+0000) 2022-01-31T21:16:58.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[41853]: debug 2022-01-31T21:16:57.798+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.799040+0000) 2022-01-31T21:16:58.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:57 smithi171 conmon[41853]: debug 2022-01-31T21:16:57.959+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.959859+0000) 2022-01-31T21:16:58.298 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:57 smithi167 conmon[49112]: debug 2022-01-31T21:16:57.960+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.961264+0000) 2022-01-31T21:16:58.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:57 smithi167 conmon[54076]: debug 2022-01-31T21:16:57.959+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.960397+0000) 2022-01-31T21:16:58.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:57 smithi167 conmon[60316]: debug 2022-01-31T21:16:57.960+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:57.960661+0000) 2022-01-31T21:16:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:58 smithi171 conmon[46715]: debug 2022-01-31T21:16:58.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:58.095154+0000) 2022-01-31T21:16:58.566 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:58 smithi167 conmon[60316]: debug 2022-01-31T21:16:58.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:58.319833+0000) 2022-01-31T21:16:58.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:58 smithi171 conmon[51620]: debug 2022-01-31T21:16:58.674+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:58.674792+0000) 2022-01-31T21:16:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:58 smithi167 conmon[49112]: debug 2022-01-31T21:16:58.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:58.642929+0000) 2022-01-31T21:16:58.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:58 smithi167 conmon[54076]: debug 2022-01-31T21:16:58.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:58.567841+0000) 2022-01-31T21:16:59.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:58 smithi171 conmon[41853]: debug 2022-01-31T21:16:58.798+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:58.799142+0000) 2022-01-31T21:16:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:16:59 smithi171 conmon[46715]: debug 2022-01-31T21:16:59.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:59.095341+0000) 2022-01-31T21:16:59.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:16:59 smithi167 conmon[60316]: debug 2022-01-31T21:16:59.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:59.319968+0000) 2022-01-31T21:16:59.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:16:59 smithi171 conmon[51620]: debug 2022-01-31T21:16:59.674+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:59.674945+0000) 2022-01-31T21:16:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:16:59 smithi167 conmon[49112]: debug 2022-01-31T21:16:59.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:59.643120+0000) 2022-01-31T21:16:59.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:16:59 smithi167 conmon[54076]: debug 2022-01-31T21:16:59.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:59.567997+0000) 2022-01-31T21:17:00.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:16:59 smithi171 conmon[41853]: debug 2022-01-31T21:16:59.798+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:16:59.799311+0000) 2022-01-31T21:17:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:00 smithi171 conmon[46715]: debug 2022-01-31T21:17:00.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:00.095504+0000) 2022-01-31T21:17:00.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:00 smithi167 conmon[60316]: debug 2022-01-31T21:17:00.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:00.320175+0000) 2022-01-31T21:17:00.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:00 smithi171 conmon[51620]: debug 2022-01-31T21:17:00.674+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:00.675083+0000) 2022-01-31T21:17:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:00 smithi167 conmon[49112]: debug 2022-01-31T21:17:00.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:00.643349+0000) 2022-01-31T21:17:00.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:00 smithi167 conmon[54076]: debug 2022-01-31T21:17:00.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:00.568192+0000) 2022-01-31T21:17:01.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:00 smithi171 conmon[41853]: debug 2022-01-31T21:17:00.799+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:00.799496+0000) 2022-01-31T21:17:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:01 smithi171 conmon[46715]: debug 2022-01-31T21:17:01.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:01.095659+0000) 2022-01-31T21:17:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:01 smithi167 conmon[60316]: debug 2022-01-31T21:17:01.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:01.320339+0000) 2022-01-31T21:17:01.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:01 smithi171 conmon[51620]: debug 2022-01-31T21:17:01.674+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:01.675268+0000) 2022-01-31T21:17:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:01 smithi167 conmon[49112]: debug 2022-01-31T21:17:01.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:01.643589+0000) 2022-01-31T21:17:01.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:01 smithi167 conmon[54076]: debug 2022-01-31T21:17:01.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:01.568353+0000) 2022-01-31T21:17:02.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:01 smithi171 conmon[41853]: debug 2022-01-31T21:17:01.799+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:01.799671+0000) 2022-01-31T21:17:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[46715]: debug 2022-01-31T21:17:02.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.095827+0000) 2022-01-31T21:17:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:02 smithi167 conmon[60316]: debug 2022-01-31T21:17:02.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.320495+0000) 2022-01-31T21:17:02.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[51620]: debug 2022-01-31T21:17:02.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.675417+0000) 2022-01-31T21:17:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:02 smithi167 conmon[49112]: debug 2022-01-31T21:17:02.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.643775+0000) 2022-01-31T21:17:02.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:02 smithi167 conmon[54076]: debug 2022-01-31T21:17:02.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.568536+0000) 2022-01-31T21:17:03.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[35325]: debug 2022-01-31T21:17:02.985+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117505 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:03.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[46715]: debug 2022-01-31T21:17:02.973+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.974631+0000) 2022-01-31T21:17:03.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[51620]: debug 2022-01-31T21:17:02.973+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.974376+0000) 2022-01-31T21:17:03.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[41853]: debug 2022-01-31T21:17:02.799+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.799909+0000) 2022-01-31T21:17:03.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:02 smithi171 conmon[41853]: debug 2022-01-31T21:17:02.972+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.973505+0000) 2022-01-31T21:17:03.299 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:02 smithi167 conmon[49112]: debug 2022-01-31T21:17:02.974+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.974531+0000) 2022-01-31T21:17:03.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:02 smithi167 conmon[54076]: debug 2022-01-31T21:17:02.973+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.974302+0000) 2022-01-31T21:17:03.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:02 smithi167 conmon[60316]: debug 2022-01-31T21:17:02.974+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:02.974610+0000) 2022-01-31T21:17:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:03 smithi171 conmon[46715]: debug 2022-01-31T21:17:03.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:03.096011+0000) 2022-01-31T21:17:03.567 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:03 smithi167 conmon[60316]: debug 2022-01-31T21:17:03.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:03.320696+0000) 2022-01-31T21:17:03.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:03 smithi171 conmon[51620]: debug 2022-01-31T21:17:03.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:03.675598+0000) 2022-01-31T21:17:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:03 smithi167 conmon[49112]: debug 2022-01-31T21:17:03.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:03.643954+0000) 2022-01-31T21:17:03.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:03 smithi167 conmon[54076]: debug 2022-01-31T21:17:03.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:03.568688+0000) 2022-01-31T21:17:04.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:03 smithi171 conmon[41853]: debug 2022-01-31T21:17:03.799+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:03.800050+0000) 2022-01-31T21:17:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:04 smithi171 conmon[46715]: debug 2022-01-31T21:17:04.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:04.096164+0000) 2022-01-31T21:17:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:04 smithi167 conmon[60316]: debug 2022-01-31T21:17:04.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:04.320852+0000) 2022-01-31T21:17:04.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:04 smithi171 conmon[51620]: debug 2022-01-31T21:17:04.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:04.675785+0000) 2022-01-31T21:17:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:04 smithi167 conmon[49112]: debug 2022-01-31T21:17:04.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:04.644145+0000) 2022-01-31T21:17:04.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:04 smithi167 conmon[54076]: debug 2022-01-31T21:17:04.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:04.568841+0000) 2022-01-31T21:17:05.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:04 smithi171 conmon[41853]: debug 2022-01-31T21:17:04.799+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:04.800199+0000) 2022-01-31T21:17:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:05 smithi171 conmon[46715]: debug 2022-01-31T21:17:05.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:05.096369+0000) 2022-01-31T21:17:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:05 smithi167 conmon[60316]: debug 2022-01-31T21:17:05.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:05.321042+0000) 2022-01-31T21:17:05.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:05 smithi171 conmon[51620]: debug 2022-01-31T21:17:05.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:05.675963+0000) 2022-01-31T21:17:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:05 smithi167 conmon[49112]: debug 2022-01-31T21:17:05.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:05.644383+0000) 2022-01-31T21:17:05.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:05 smithi167 conmon[54076]: debug 2022-01-31T21:17:05.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:05.568964+0000) 2022-01-31T21:17:06.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:05 smithi171 conmon[41853]: debug 2022-01-31T21:17:05.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:05.800389+0000) 2022-01-31T21:17:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:06 smithi171 conmon[46715]: debug 2022-01-31T21:17:06.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:06.096550+0000) 2022-01-31T21:17:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:06 smithi167 conmon[60316]: debug 2022-01-31T21:17:06.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:06.321173+0000) 2022-01-31T21:17:06.758 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:06 smithi171 conmon[51620]: debug 2022-01-31T21:17:06.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:06.676145+0000) 2022-01-31T21:17:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:06 smithi167 conmon[49112]: debug 2022-01-31T21:17:06.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:06.644580+0000) 2022-01-31T21:17:06.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:06 smithi167 conmon[54076]: debug 2022-01-31T21:17:06.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:06.569143+0000) 2022-01-31T21:17:07.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:06 smithi171 conmon[41853]: debug 2022-01-31T21:17:06.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:06.800593+0000) 2022-01-31T21:17:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:07 smithi171 conmon[46715]: debug 2022-01-31T21:17:07.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.096722+0000) 2022-01-31T21:17:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:07 smithi167 conmon[60316]: debug 2022-01-31T21:17:07.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.321275+0000) 2022-01-31T21:17:07.758 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:07 smithi171 conmon[51620]: debug 2022-01-31T21:17:07.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.676314+0000) 2022-01-31T21:17:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:07 smithi167 conmon[49112]: debug 2022-01-31T21:17:07.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.644751+0000) 2022-01-31T21:17:07.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:07 smithi167 conmon[54076]: debug 2022-01-31T21:17:07.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.569298+0000) 2022-01-31T21:17:08.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:08 smithi171 conmon[35325]: debug 2022-01-31T21:17:08.000+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117616 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:08.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:07 smithi171 conmon[46715]: debug 2022-01-31T21:17:07.988+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.988834+0000) 2022-01-31T21:17:08.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:07 smithi171 conmon[51620]: debug 2022-01-31T21:17:07.988+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.988884+0000) 2022-01-31T21:17:08.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:07 smithi171 conmon[41853]: debug 2022-01-31T21:17:07.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.800773+0000) 2022-01-31T21:17:08.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:07 smithi171 conmon[41853]: debug 2022-01-31T21:17:07.987+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.988280+0000) 2022-01-31T21:17:08.300 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:07 smithi167 conmon[49112]: debug 2022-01-31T21:17:07.989+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.989555+0000) 2022-01-31T21:17:08.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:07 smithi167 conmon[54076]: debug 2022-01-31T21:17:07.988+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.989485+0000) 2022-01-31T21:17:08.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:07 smithi167 conmon[60316]: debug 2022-01-31T21:17:07.990+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:07.990883+0000) 2022-01-31T21:17:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:08 smithi171 conmon[46715]: debug 2022-01-31T21:17:08.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:08.096867+0000) 2022-01-31T21:17:08.567 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:08 smithi167 conmon[60316]: debug 2022-01-31T21:17:08.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:08.321440+0000) 2022-01-31T21:17:08.758 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:08 smithi171 conmon[35325]: debug 2022-01-31T21:17:08.653+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:17:08.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:08 smithi171 conmon[51620]: debug 2022-01-31T21:17:08.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:08.676507+0000) 2022-01-31T21:17:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:08 smithi167 conmon[49112]: debug 2022-01-31T21:17:08.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:08.644946+0000) 2022-01-31T21:17:08.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:08 smithi167 conmon[54076]: debug 2022-01-31T21:17:08.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:08.569475+0000) 2022-01-31T21:17:09.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:08 smithi171 conmon[41853]: debug 2022-01-31T21:17:08.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:08.800941+0000) 2022-01-31T21:17:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:09 smithi171 conmon[46715]: debug 2022-01-31T21:17:09.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:09.096979+0000) 2022-01-31T21:17:09.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:09 smithi167 conmon[60316]: debug 2022-01-31T21:17:09.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:09.321595+0000) 2022-01-31T21:17:09.758 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:09 smithi171 conmon[51620]: debug 2022-01-31T21:17:09.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:09.676664+0000) 2022-01-31T21:17:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:09 smithi167 conmon[49112]: debug 2022-01-31T21:17:09.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:09.645131+0000) 2022-01-31T21:17:09.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:09 smithi167 conmon[54076]: debug 2022-01-31T21:17:09.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:09.569618+0000) 2022-01-31T21:17:10.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:09 smithi171 conmon[41853]: debug 2022-01-31T21:17:09.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:09.801060+0000) 2022-01-31T21:17:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:10 smithi171 conmon[46715]: debug 2022-01-31T21:17:10.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:10.097110+0000) 2022-01-31T21:17:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:10 smithi167 conmon[60316]: debug 2022-01-31T21:17:10.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:10.321796+0000) 2022-01-31T21:17:10.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:10 smithi171 conmon[51620]: debug 2022-01-31T21:17:10.676+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:10.676845+0000) 2022-01-31T21:17:10.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:10 smithi167 conmon[49112]: debug 2022-01-31T21:17:10.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:10.645364+0000) 2022-01-31T21:17:10.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:10 smithi167 conmon[54076]: debug 2022-01-31T21:17:10.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:10.569806+0000) 2022-01-31T21:17:11.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:10 smithi171 conmon[41853]: debug 2022-01-31T21:17:10.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:10.801240+0000) 2022-01-31T21:17:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:11 smithi171 conmon[46715]: debug 2022-01-31T21:17:11.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:11.097216+0000) 2022-01-31T21:17:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:11 smithi167 conmon[60316]: debug 2022-01-31T21:17:11.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:11.321967+0000) 2022-01-31T21:17:11.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:11 smithi171 conmon[51620]: debug 2022-01-31T21:17:11.676+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:11.676971+0000) 2022-01-31T21:17:11.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:11 smithi167 conmon[49112]: debug 2022-01-31T21:17:11.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:11.645604+0000) 2022-01-31T21:17:11.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:11 smithi167 conmon[54076]: debug 2022-01-31T21:17:11.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:11.569989+0000) 2022-01-31T21:17:12.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:11 smithi171 conmon[41853]: debug 2022-01-31T21:17:11.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:11.801440+0000) 2022-01-31T21:17:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:12 smithi171 conmon[46715]: debug 2022-01-31T21:17:12.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:12.097373+0000) 2022-01-31T21:17:12.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:12 smithi167 conmon[60316]: debug 2022-01-31T21:17:12.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:12.322129+0000) 2022-01-31T21:17:12.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:12 smithi171 conmon[51620]: debug 2022-01-31T21:17:12.676+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:12.677094+0000) 2022-01-31T21:17:12.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:12 smithi167 conmon[49112]: debug 2022-01-31T21:17:12.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:12.645759+0000) 2022-01-31T21:17:12.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:12 smithi167 conmon[54076]: debug 2022-01-31T21:17:12.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:12.570146+0000) 2022-01-31T21:17:13.014 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[46715]: debug 2022-01-31T21:17:13.002+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.003243+0000) 2022-01-31T21:17:13.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[51620]: debug 2022-01-31T21:17:13.002+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.003136+0000) 2022-01-31T21:17:13.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:12 smithi171 conmon[41853]: debug 2022-01-31T21:17:12.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:12.801656+0000) 2022-01-31T21:17:13.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[41853]: debug 2022-01-31T21:17:13.002+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.003475+0000) 2022-01-31T21:17:13.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:13 smithi167 conmon[49112]: debug 2022-01-31T21:17:13.002+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.003104+0000) 2022-01-31T21:17:13.074 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:13 smithi167 conmon[54076]: debug 2022-01-31T21:17:13.004+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.004857+0000) 2022-01-31T21:17:13.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:13 smithi167 conmon[60316]: debug 2022-01-31T21:17:13.003+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.004417+0000) 2022-01-31T21:17:13.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[35325]: debug 2022-01-31T21:17:13.014+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117727 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[46715]: debug 2022-01-31T21:17:13.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.097557+0000) 2022-01-31T21:17:13.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:13 smithi167 conmon[60316]: debug 2022-01-31T21:17:13.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.322307+0000) 2022-01-31T21:17:13.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[51620]: debug 2022-01-31T21:17:13.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.677306+0000) 2022-01-31T21:17:13.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:13 smithi167 conmon[49112]: debug 2022-01-31T21:17:13.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.645993+0000) 2022-01-31T21:17:13.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:13 smithi167 conmon[54076]: debug 2022-01-31T21:17:13.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.570307+0000) 2022-01-31T21:17:14.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:13 smithi171 conmon[41853]: debug 2022-01-31T21:17:13.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:13.801836+0000) 2022-01-31T21:17:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:14 smithi171 conmon[46715]: debug 2022-01-31T21:17:14.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:14.097671+0000) 2022-01-31T21:17:14.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:14 smithi167 conmon[60316]: debug 2022-01-31T21:17:14.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:14.322469+0000) 2022-01-31T21:17:14.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:14 smithi171 conmon[51620]: debug 2022-01-31T21:17:14.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:14.677478+0000) 2022-01-31T21:17:14.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:14 smithi167 conmon[49112]: debug 2022-01-31T21:17:14.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:14.646153+0000) 2022-01-31T21:17:14.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:14 smithi167 conmon[54076]: debug 2022-01-31T21:17:14.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:14.570467+0000) 2022-01-31T21:17:15.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:14 smithi171 conmon[41853]: debug 2022-01-31T21:17:14.800+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:14.801982+0000) 2022-01-31T21:17:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:15 smithi171 conmon[46715]: debug 2022-01-31T21:17:15.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:15.097828+0000) 2022-01-31T21:17:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:15 smithi167 conmon[60316]: debug 2022-01-31T21:17:15.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:15.322706+0000) 2022-01-31T21:17:15.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:15 smithi171 conmon[51620]: debug 2022-01-31T21:17:15.675+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:15.677661+0000) 2022-01-31T21:17:15.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:15 smithi167 conmon[49112]: debug 2022-01-31T21:17:15.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:15.646342+0000) 2022-01-31T21:17:15.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:15 smithi167 conmon[54076]: debug 2022-01-31T21:17:15.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:15.570674+0000) 2022-01-31T21:17:16.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:15 smithi171 conmon[41853]: debug 2022-01-31T21:17:15.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:15.802155+0000) 2022-01-31T21:17:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:16 smithi171 conmon[46715]: debug 2022-01-31T21:17:16.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:16.097985+0000) 2022-01-31T21:17:16.569 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:16 smithi167 conmon[60316]: debug 2022-01-31T21:17:16.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:16.322910+0000) 2022-01-31T21:17:16.760 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:16 smithi171 conmon[51620]: debug 2022-01-31T21:17:16.676+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:16.677832+0000) 2022-01-31T21:17:16.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:16 smithi167 conmon[54076]: debug 2022-01-31T21:17:16.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:16.570857+0000) 2022-01-31T21:17:16.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:16 smithi167 conmon[49112]: debug 2022-01-31T21:17:16.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:16.646524+0000) 2022-01-31T21:17:17.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:16 smithi171 conmon[41853]: debug 2022-01-31T21:17:16.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:16.802359+0000) 2022-01-31T21:17:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:17 smithi171 conmon[46715]: debug 2022-01-31T21:17:17.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:17.098104+0000) 2022-01-31T21:17:17.569 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:17 smithi167 conmon[60316]: debug 2022-01-31T21:17:17.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:17.323059+0000) 2022-01-31T21:17:17.760 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:17 smithi171 conmon[51620]: debug 2022-01-31T21:17:17.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:17.678035+0000) 2022-01-31T21:17:17.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:17 smithi167 conmon[54076]: debug 2022-01-31T21:17:17.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:17.571002+0000) 2022-01-31T21:17:17.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:17 smithi167 conmon[49112]: debug 2022-01-31T21:17:17.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:17.646677+0000) 2022-01-31T21:17:18.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[51620]: debug 2022-01-31T21:17:18.015+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.017580+0000) 2022-01-31T21:17:18.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:17 smithi171 conmon[41853]: debug 2022-01-31T21:17:17.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:17.802537+0000) 2022-01-31T21:17:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:18 smithi167 conmon[49112]: debug 2022-01-31T21:17:18.018+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.018715+0000) 2022-01-31T21:17:18.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:18 smithi167 conmon[54076]: debug 2022-01-31T21:17:18.016+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.017326+0000) 2022-01-31T21:17:18.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:18 smithi167 conmon[60316]: debug 2022-01-31T21:17:18.017+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.017735+0000) 2022-01-31T21:17:18.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[35325]: debug 2022-01-31T21:17:18.028+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117839 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[41853]: debug 2022-01-31T21:17:18.016+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.017907+0000) 2022-01-31T21:17:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[46715]: debug 2022-01-31T21:17:18.016+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.017905+0000) 2022-01-31T21:17:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[46715]: debug 2022-01-31T21:17:18.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.098249+0000) 2022-01-31T21:17:18.569 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:18 smithi167 conmon[60316]: debug 2022-01-31T21:17:18.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.323251+0000) 2022-01-31T21:17:18.760 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[51620]: debug 2022-01-31T21:17:18.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.678238+0000) 2022-01-31T21:17:18.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:18 smithi167 conmon[49112]: debug 2022-01-31T21:17:18.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.646886+0000) 2022-01-31T21:17:18.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:18 smithi167 conmon[54076]: debug 2022-01-31T21:17:18.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.571176+0000) 2022-01-31T21:17:19.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:18 smithi171 conmon[41853]: debug 2022-01-31T21:17:18.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:18.802664+0000) 2022-01-31T21:17:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:19 smithi171 conmon[46715]: debug 2022-01-31T21:17:19.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:19.098399+0000) 2022-01-31T21:17:19.569 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:19 smithi167 conmon[60316]: debug 2022-01-31T21:17:19.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:19.323406+0000) 2022-01-31T21:17:19.760 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:19 smithi171 conmon[51620]: debug 2022-01-31T21:17:19.676+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:19.678403+0000) 2022-01-31T21:17:19.831 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:19 smithi167 conmon[49112]: debug 2022-01-31T21:17:19.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:19.647058+0000) 2022-01-31T21:17:19.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:19 smithi167 conmon[54076]: debug 2022-01-31T21:17:19.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:19.571313+0000) 2022-01-31T21:17:20.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:19 smithi171 conmon[41853]: debug 2022-01-31T21:17:19.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:19.802808+0000) 2022-01-31T21:17:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:20 smithi171 conmon[46715]: debug 2022-01-31T21:17:20.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:20.098594+0000) 2022-01-31T21:17:20.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:20 smithi167 conmon[60316]: debug 2022-01-31T21:17:20.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:20.323602+0000) 2022-01-31T21:17:20.760 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:20 smithi171 conmon[51620]: debug 2022-01-31T21:17:20.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:20.678529+0000) 2022-01-31T21:17:20.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:20 smithi167 conmon[49112]: debug 2022-01-31T21:17:20.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:20.647264+0000) 2022-01-31T21:17:20.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:20 smithi167 conmon[54076]: debug 2022-01-31T21:17:20.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:20.571513+0000) 2022-01-31T21:17:21.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:20 smithi171 conmon[41853]: debug 2022-01-31T21:17:20.801+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:20.802944+0000) 2022-01-31T21:17:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:21 smithi171 conmon[46715]: debug 2022-01-31T21:17:21.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:21.098727+0000) 2022-01-31T21:17:21.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:21 smithi167 conmon[60316]: debug 2022-01-31T21:17:21.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:21.323790+0000) 2022-01-31T21:17:21.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:21 smithi171 conmon[51620]: debug 2022-01-31T21:17:21.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:21.678686+0000) 2022-01-31T21:17:21.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:21 smithi167 conmon[49112]: debug 2022-01-31T21:17:21.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:21.647483+0000) 2022-01-31T21:17:21.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:21 smithi167 conmon[54076]: debug 2022-01-31T21:17:21.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:21.571679+0000) 2022-01-31T21:17:22.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:21 smithi171 conmon[41853]: debug 2022-01-31T21:17:21.802+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:21.803129+0000) 2022-01-31T21:17:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:22 smithi171 conmon[46715]: debug 2022-01-31T21:17:22.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:22.098862+0000) 2022-01-31T21:17:22.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:22 smithi167 conmon[60316]: debug 2022-01-31T21:17:22.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:22.323975+0000) 2022-01-31T21:17:22.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:22 smithi171 conmon[51620]: debug 2022-01-31T21:17:22.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:22.678928+0000) 2022-01-31T21:17:22.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:22 smithi167 conmon[54076]: debug 2022-01-31T21:17:22.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:22.571902+0000) 2022-01-31T21:17:22.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:22 smithi167 conmon[49112]: debug 2022-01-31T21:17:22.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:22.647720+0000) 2022-01-31T21:17:23.031 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:22 smithi171 conmon[41853]: debug 2022-01-31T21:17:22.802+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:22.803456+0000) 2022-01-31T21:17:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:23 smithi167 conmon[49112]: debug 2022-01-31T21:17:23.031+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.033027+0000) 2022-01-31T21:17:23.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:23 smithi167 conmon[54076]: debug 2022-01-31T21:17:23.030+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.032184+0000) 2022-01-31T21:17:23.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:23 smithi167 conmon[60316]: debug 2022-01-31T21:17:23.031+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.032725+0000) 2022-01-31T21:17:23.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[35325]: debug 2022-01-31T21:17:23.042+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 117948 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:23.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[41853]: debug 2022-01-31T21:17:23.030+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.032254+0000) 2022-01-31T21:17:23.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[51620]: debug 2022-01-31T21:17:23.031+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.033151+0000) 2022-01-31T21:17:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[46715]: debug 2022-01-31T21:17:23.030+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.032485+0000) 2022-01-31T21:17:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[46715]: debug 2022-01-31T21:17:23.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.099074+0000) 2022-01-31T21:17:23.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:23 smithi167 conmon[60316]: debug 2022-01-31T21:17:23.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.324176+0000) 2022-01-31T21:17:23.761 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[35325]: debug 2022-01-31T21:17:23.653+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:17:23.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[51620]: debug 2022-01-31T21:17:23.678+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.679055+0000) 2022-01-31T21:17:23.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:23 smithi167 conmon[49112]: debug 2022-01-31T21:17:23.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.647875+0000) 2022-01-31T21:17:23.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:23 smithi167 conmon[54076]: debug 2022-01-31T21:17:23.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.572082+0000) 2022-01-31T21:17:24.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:23 smithi171 conmon[41853]: debug 2022-01-31T21:17:23.802+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:23.803597+0000) 2022-01-31T21:17:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:24 smithi171 conmon[46715]: debug 2022-01-31T21:17:24.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:24.099206+0000) 2022-01-31T21:17:24.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:24 smithi167 conmon[60316]: debug 2022-01-31T21:17:24.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:24.324293+0000) 2022-01-31T21:17:24.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:24 smithi171 conmon[51620]: debug 2022-01-31T21:17:24.677+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:24.679208+0000) 2022-01-31T21:17:24.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:24 smithi167 conmon[49112]: debug 2022-01-31T21:17:24.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:24.648076+0000) 2022-01-31T21:17:24.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:24 smithi167 conmon[54076]: debug 2022-01-31T21:17:24.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:24.572210+0000) 2022-01-31T21:17:25.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:24 smithi171 conmon[41853]: debug 2022-01-31T21:17:24.802+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:24.803736+0000) 2022-01-31T21:17:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:25 smithi171 conmon[46715]: debug 2022-01-31T21:17:25.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:25.099380+0000) 2022-01-31T21:17:25.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:25 smithi167 conmon[60316]: debug 2022-01-31T21:17:25.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:25.324395+0000) 2022-01-31T21:17:25.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:25 smithi171 conmon[51620]: debug 2022-01-31T21:17:25.678+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:25.679411+0000) 2022-01-31T21:17:25.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:25 smithi167 conmon[49112]: debug 2022-01-31T21:17:25.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:25.648318+0000) 2022-01-31T21:17:25.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:25 smithi167 conmon[54076]: debug 2022-01-31T21:17:25.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:25.572370+0000) 2022-01-31T21:17:26.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:25 smithi171 conmon[41853]: debug 2022-01-31T21:17:25.803+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:25.803955+0000) 2022-01-31T21:17:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:26 smithi171 conmon[46715]: debug 2022-01-31T21:17:26.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:26.099594+0000) 2022-01-31T21:17:26.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:26 smithi167 conmon[60316]: debug 2022-01-31T21:17:26.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:26.324555+0000) 2022-01-31T21:17:26.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:26 smithi171 conmon[51620]: debug 2022-01-31T21:17:26.678+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:26.679617+0000) 2022-01-31T21:17:26.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:26 smithi167 conmon[49112]: debug 2022-01-31T21:17:26.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:26.648560+0000) 2022-01-31T21:17:26.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:26 smithi167 conmon[54076]: debug 2022-01-31T21:17:26.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:26.572559+0000) 2022-01-31T21:17:27.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:26 smithi171 conmon[41853]: debug 2022-01-31T21:17:26.803+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:26.804142+0000) 2022-01-31T21:17:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:27 smithi171 conmon[46715]: debug 2022-01-31T21:17:27.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:27.099772+0000) 2022-01-31T21:17:27.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:27 smithi167 conmon[60316]: debug 2022-01-31T21:17:27.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:27.324769+0000) 2022-01-31T21:17:27.761 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:27 smithi171 conmon[51620]: debug 2022-01-31T21:17:27.679+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:27.679828+0000) 2022-01-31T21:17:27.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:27 smithi167 conmon[54076]: debug 2022-01-31T21:17:27.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:27.572746+0000) 2022-01-31T21:17:27.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:27 smithi167 conmon[49112]: debug 2022-01-31T21:17:27.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:27.648744+0000) 2022-01-31T21:17:28.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:27 smithi171 conmon[41853]: debug 2022-01-31T21:17:27.803+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:27.804318+0000) 2022-01-31T21:17:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:28 smithi167 conmon[49112]: debug 2022-01-31T21:17:28.045+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.047117+0000) 2022-01-31T21:17:28.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:28 smithi167 conmon[54076]: debug 2022-01-31T21:17:28.045+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.047474+0000) 2022-01-31T21:17:28.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:28 smithi167 conmon[60316]: debug 2022-01-31T21:17:28.045+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.046754+0000) 2022-01-31T21:17:28.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[35325]: debug 2022-01-31T21:17:28.057+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118060 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:28.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[41853]: debug 2022-01-31T21:17:28.045+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.046412+0000) 2022-01-31T21:17:28.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[51620]: debug 2022-01-31T21:17:28.046+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.046878+0000) 2022-01-31T21:17:28.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[46715]: debug 2022-01-31T21:17:28.046+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.046934+0000) 2022-01-31T21:17:28.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[46715]: debug 2022-01-31T21:17:28.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.099922+0000) 2022-01-31T21:17:28.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:28 smithi167 conmon[60316]: debug 2022-01-31T21:17:28.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.324957+0000) 2022-01-31T21:17:28.762 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[51620]: debug 2022-01-31T21:17:28.679+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.679998+0000) 2022-01-31T21:17:28.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:28 smithi167 conmon[49112]: debug 2022-01-31T21:17:28.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.648923+0000) 2022-01-31T21:17:28.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:28 smithi167 conmon[54076]: debug 2022-01-31T21:17:28.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.572942+0000) 2022-01-31T21:17:29.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:28 smithi171 conmon[41853]: debug 2022-01-31T21:17:28.803+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:28.804491+0000) 2022-01-31T21:17:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:29 smithi171 conmon[46715]: debug 2022-01-31T21:17:29.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:29.100132+0000) 2022-01-31T21:17:29.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:29 smithi167 conmon[60316]: debug 2022-01-31T21:17:29.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:29.325161+0000) 2022-01-31T21:17:29.762 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:29 smithi171 conmon[51620]: debug 2022-01-31T21:17:29.679+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:29.680141+0000) 2022-01-31T21:17:29.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:29 smithi167 conmon[49112]: debug 2022-01-31T21:17:29.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:29.649080+0000) 2022-01-31T21:17:29.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:29 smithi167 conmon[54076]: debug 2022-01-31T21:17:29.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:29.573129+0000) 2022-01-31T21:17:30.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:29 smithi171 conmon[41853]: debug 2022-01-31T21:17:29.803+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:29.804626+0000) 2022-01-31T21:17:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:30 smithi171 conmon[46715]: debug 2022-01-31T21:17:30.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:30.100318+0000) 2022-01-31T21:17:30.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:30 smithi167 conmon[60316]: debug 2022-01-31T21:17:30.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:30.325358+0000) 2022-01-31T21:17:30.762 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:30 smithi171 conmon[51620]: debug 2022-01-31T21:17:30.679+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:30.680296+0000) 2022-01-31T21:17:30.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:30 smithi167 conmon[49112]: debug 2022-01-31T21:17:30.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:30.649317+0000) 2022-01-31T21:17:30.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:30 smithi167 conmon[54076]: debug 2022-01-31T21:17:30.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:30.573335+0000) 2022-01-31T21:17:31.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:30 smithi171 conmon[41853]: debug 2022-01-31T21:17:30.804+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:30.804806+0000) 2022-01-31T21:17:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:31 smithi171 conmon[46715]: debug 2022-01-31T21:17:31.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:31.100503+0000) 2022-01-31T21:17:31.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:31 smithi167 conmon[60316]: debug 2022-01-31T21:17:31.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:31.325564+0000) 2022-01-31T21:17:31.762 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:31 smithi171 conmon[51620]: debug 2022-01-31T21:17:31.679+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:31.680417+0000) 2022-01-31T21:17:31.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:31 smithi167 conmon[54076]: debug 2022-01-31T21:17:31.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:31.573522+0000) 2022-01-31T21:17:31.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:31 smithi167 conmon[49112]: debug 2022-01-31T21:17:31.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:31.649495+0000) 2022-01-31T21:17:32.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:31 smithi171 conmon[41853]: debug 2022-01-31T21:17:31.804+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:31.804962+0000) 2022-01-31T21:17:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:32 smithi171 conmon[46715]: debug 2022-01-31T21:17:32.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:32.100661+0000) 2022-01-31T21:17:32.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:32 smithi167 conmon[60316]: debug 2022-01-31T21:17:32.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:32.325718+0000) 2022-01-31T21:17:32.763 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:32 smithi171 conmon[51620]: debug 2022-01-31T21:17:32.679+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:32.680576+0000) 2022-01-31T21:17:32.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:32 smithi167 conmon[54076]: debug 2022-01-31T21:17:32.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:32.573675+0000) 2022-01-31T21:17:32.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:32 smithi167 conmon[49112]: debug 2022-01-31T21:17:32.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:32.649729+0000) 2022-01-31T21:17:33.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:32 smithi171 conmon[41853]: debug 2022-01-31T21:17:32.804+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:32.805153+0000) 2022-01-31T21:17:33.059 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[46715]: debug 2022-01-31T21:17:33.059+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.060249+0000) 2022-01-31T21:17:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:33 smithi167 conmon[49112]: debug 2022-01-31T21:17:33.060+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.061960+0000) 2022-01-31T21:17:33.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:33 smithi167 conmon[54076]: debug 2022-01-31T21:17:33.060+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.061594+0000) 2022-01-31T21:17:33.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:33 smithi167 conmon[60316]: debug 2022-01-31T21:17:33.059+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.060939+0000) 2022-01-31T21:17:33.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[35325]: debug 2022-01-31T21:17:33.072+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118168 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:33.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[41853]: debug 2022-01-31T21:17:33.060+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.061373+0000) 2022-01-31T21:17:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[46715]: debug 2022-01-31T21:17:33.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.100840+0000) 2022-01-31T21:17:33.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[51620]: debug 2022-01-31T21:17:33.060+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.061728+0000) 2022-01-31T21:17:33.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:33 smithi167 conmon[60316]: debug 2022-01-31T21:17:33.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.325906+0000) 2022-01-31T21:17:33.763 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[51620]: debug 2022-01-31T21:17:33.680+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.680738+0000) 2022-01-31T21:17:33.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:33 smithi167 conmon[49112]: debug 2022-01-31T21:17:33.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.649937+0000) 2022-01-31T21:17:33.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:33 smithi167 conmon[54076]: debug 2022-01-31T21:17:33.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.573829+0000) 2022-01-31T21:17:34.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:33 smithi171 conmon[41853]: debug 2022-01-31T21:17:33.804+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:33.805357+0000) 2022-01-31T21:17:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:34 smithi171 conmon[46715]: debug 2022-01-31T21:17:34.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:34.100997+0000) 2022-01-31T21:17:34.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:34 smithi167 conmon[60316]: debug 2022-01-31T21:17:34.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:34.326059+0000) 2022-01-31T21:17:34.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:34 smithi167 conmon[54076]: debug 2022-01-31T21:17:34.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:34.573965+0000) 2022-01-31T21:17:34.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:34 smithi167 conmon[49112]: debug 2022-01-31T21:17:34.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:34.650112+0000) 2022-01-31T21:17:34.932 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:34 smithi171 conmon[41853]: debug 2022-01-31T21:17:34.804+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:34.805523+0000) 2022-01-31T21:17:34.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:34 smithi171 conmon[51620]: debug 2022-01-31T21:17:34.680+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:34.680876+0000) 2022-01-31T21:17:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:35 smithi171 conmon[46715]: debug 2022-01-31T21:17:35.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:35.101186+0000) 2022-01-31T21:17:35.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:35 smithi167 conmon[60316]: debug 2022-01-31T21:17:35.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:35.326236+0000) 2022-01-31T21:17:35.763 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:35 smithi171 conmon[51620]: debug 2022-01-31T21:17:35.680+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:35.681057+0000) 2022-01-31T21:17:35.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:35 smithi167 conmon[49112]: debug 2022-01-31T21:17:35.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:35.650262+0000) 2022-01-31T21:17:35.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:35 smithi167 conmon[54076]: debug 2022-01-31T21:17:35.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:35.574150+0000) 2022-01-31T21:17:36.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:35 smithi171 conmon[41853]: debug 2022-01-31T21:17:35.805+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:35.805678+0000) 2022-01-31T21:17:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:36 smithi171 conmon[46715]: debug 2022-01-31T21:17:36.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:36.101414+0000) 2022-01-31T21:17:36.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:36 smithi167 conmon[60316]: debug 2022-01-31T21:17:36.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:36.326435+0000) 2022-01-31T21:17:36.763 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:36 smithi171 conmon[51620]: debug 2022-01-31T21:17:36.680+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:36.681229+0000) 2022-01-31T21:17:36.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:36 smithi167 conmon[49112]: debug 2022-01-31T21:17:36.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:36.650457+0000) 2022-01-31T21:17:36.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:36 smithi167 conmon[54076]: debug 2022-01-31T21:17:36.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:36.574377+0000) 2022-01-31T21:17:37.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:36 smithi171 conmon[41853]: debug 2022-01-31T21:17:36.805+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:36.805914+0000) 2022-01-31T21:17:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:37 smithi171 conmon[46715]: debug 2022-01-31T21:17:37.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:37.101604+0000) 2022-01-31T21:17:37.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:37 smithi167 conmon[60316]: debug 2022-01-31T21:17:37.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:37.326611+0000) 2022-01-31T21:17:37.763 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:37 smithi171 conmon[51620]: debug 2022-01-31T21:17:37.680+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:37.681368+0000) 2022-01-31T21:17:37.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:37 smithi167 conmon[49112]: debug 2022-01-31T21:17:37.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:37.650634+0000) 2022-01-31T21:17:37.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:37 smithi167 conmon[54076]: debug 2022-01-31T21:17:37.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:37.574589+0000) 2022-01-31T21:17:38.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:37 smithi171 conmon[41853]: debug 2022-01-31T21:17:37.805+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:37.806120+0000) 2022-01-31T21:17:38.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[49112]: debug 2022-01-31T21:17:38.075+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.077373+0000) 2022-01-31T21:17:38.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[60316]: debug 2022-01-31T21:17:38.074+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.076014+0000) 2022-01-31T21:17:38.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[54076]: debug 2022-01-31T21:17:38.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[54076]: 2022-01-31T21:17:38.074+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.076209+0000) 2022-01-31T21:17:38.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[35325]: debug 2022-01-31T21:17:38.087+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118280 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:38.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[41853]: debug 2022-01-31T21:17:38.075+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.075859+0000) 2022-01-31T21:17:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[46715]: debug 2022-01-31T21:17:38.074+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.075128+0000) 2022-01-31T21:17:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[46715]: debug 2022-01-31T21:17:38.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.101790+0000) 2022-01-31T21:17:38.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[51620]: debug 2022-01-31T21:17:38.075+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.076547+0000) 2022-01-31T21:17:38.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[60316]: debug 2022-01-31T21:17:38.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.326780+0000) 2022-01-31T21:17:38.764 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[35325]: debug 2022-01-31T21:17:38.654+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:17:38.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[51620]: debug 2022-01-31T21:17:38.680+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.681584+0000) 2022-01-31T21:17:38.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[49112]: debug 2022-01-31T21:17:38.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.650784+0000) 2022-01-31T21:17:38.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:38 smithi167 conmon[54076]: debug 2022-01-31T21:17:38.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.574753+0000) 2022-01-31T21:17:39.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:38 smithi171 conmon[41853]: debug 2022-01-31T21:17:38.805+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:38.806318+0000) 2022-01-31T21:17:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:39 smithi171 conmon[46715]: debug 2022-01-31T21:17:39.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:39.101956+0000) 2022-01-31T21:17:39.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:39 smithi167 conmon[60316]: debug 2022-01-31T21:17:39.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:39.326923+0000) 2022-01-31T21:17:39.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:39 smithi171 conmon[51620]: debug 2022-01-31T21:17:39.681+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:39.681738+0000) 2022-01-31T21:17:39.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:39 smithi167 conmon[49112]: debug 2022-01-31T21:17:39.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:39.650921+0000) 2022-01-31T21:17:39.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:39 smithi167 conmon[54076]: debug 2022-01-31T21:17:39.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:39.574916+0000) 2022-01-31T21:17:40.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:39 smithi171 conmon[41853]: debug 2022-01-31T21:17:39.805+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:39.806472+0000) 2022-01-31T21:17:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:40 smithi171 conmon[46715]: debug 2022-01-31T21:17:40.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:40.102136+0000) 2022-01-31T21:17:40.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:40 smithi167 conmon[60316]: debug 2022-01-31T21:17:40.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:40.327101+0000) 2022-01-31T21:17:40.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:40 smithi171 conmon[51620]: debug 2022-01-31T21:17:40.681+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:40.681878+0000) 2022-01-31T21:17:40.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:40 smithi167 conmon[49112]: debug 2022-01-31T21:17:40.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:40.651123+0000) 2022-01-31T21:17:40.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:40 smithi167 conmon[54076]: debug 2022-01-31T21:17:40.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:40.575078+0000) 2022-01-31T21:17:41.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:40 smithi171 conmon[41853]: debug 2022-01-31T21:17:40.806+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:40.806661+0000) 2022-01-31T21:17:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:41 smithi171 conmon[46715]: debug 2022-01-31T21:17:41.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:41.102322+0000) 2022-01-31T21:17:41.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:41 smithi167 conmon[60316]: debug 2022-01-31T21:17:41.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:41.327304+0000) 2022-01-31T21:17:41.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:41 smithi171 conmon[51620]: debug 2022-01-31T21:17:41.681+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:41.682040+0000) 2022-01-31T21:17:41.926 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:41 smithi167 conmon[54076]: debug 2022-01-31T21:17:41.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:41.575265+0000) 2022-01-31T21:17:41.927 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:41 smithi167 conmon[49112]: debug 2022-01-31T21:17:41.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:41.651343+0000) 2022-01-31T21:17:42.101 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:41 smithi171 conmon[41853]: debug 2022-01-31T21:17:41.806+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:41.806824+0000) 2022-01-31T21:17:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:42 smithi171 conmon[46715]: debug 2022-01-31T21:17:42.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:42.102503+0000) 2022-01-31T21:17:42.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:42 smithi167 conmon[60316]: debug 2022-01-31T21:17:42.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:42.327524+0000) 2022-01-31T21:17:42.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:42 smithi171 conmon[51620]: debug 2022-01-31T21:17:42.681+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:42.682197+0000) 2022-01-31T21:17:42.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:42 smithi167 conmon[54076]: debug 2022-01-31T21:17:42.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:42.575461+0000) 2022-01-31T21:17:42.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:42 smithi167 conmon[49112]: debug 2022-01-31T21:17:42.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:42.651555+0000) 2022-01-31T21:17:43.088 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:42 smithi171 conmon[41853]: debug 2022-01-31T21:17:42.806+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:42.806962+0000) 2022-01-31T21:17:43.089 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[46715]: debug 2022-01-31T21:17:43.089+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.090054+0000) 2022-01-31T21:17:43.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:43 smithi167 conmon[49112]: debug 2022-01-31T21:17:43.090+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.091993+0000) 2022-01-31T21:17:43.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:43 smithi167 conmon[54076]: debug 2022-01-31T21:17:43.089+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.090896+0000) 2022-01-31T21:17:43.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:43 smithi167 conmon[60316]: debug 2022-01-31T21:17:43.090+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.092082+0000) 2022-01-31T21:17:43.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[35325]: debug 2022-01-31T21:17:43.102+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118388 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:43.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[41853]: debug 2022-01-31T21:17:43.090+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.090732+0000) 2022-01-31T21:17:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[46715]: debug 2022-01-31T21:17:43.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.102660+0000) 2022-01-31T21:17:43.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[51620]: debug 2022-01-31T21:17:43.090+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.091596+0000) 2022-01-31T21:17:43.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:43 smithi167 conmon[60316]: debug 2022-01-31T21:17:43.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.327700+0000) 2022-01-31T21:17:43.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[51620]: debug 2022-01-31T21:17:43.681+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.682369+0000) 2022-01-31T21:17:43.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:43 smithi167 conmon[49112]: debug 2022-01-31T21:17:43.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.651756+0000) 2022-01-31T21:17:43.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:43 smithi167 conmon[54076]: debug 2022-01-31T21:17:43.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.575680+0000) 2022-01-31T21:17:44.101 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:43 smithi171 conmon[41853]: debug 2022-01-31T21:17:43.806+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:43.807143+0000) 2022-01-31T21:17:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:44 smithi171 conmon[46715]: debug 2022-01-31T21:17:44.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:44.102811+0000) 2022-01-31T21:17:44.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:44 smithi167 conmon[60316]: debug 2022-01-31T21:17:44.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:44.327869+0000) 2022-01-31T21:17:44.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:44 smithi171 conmon[51620]: debug 2022-01-31T21:17:44.682+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:44.682525+0000) 2022-01-31T21:17:44.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:44 smithi167 conmon[49112]: debug 2022-01-31T21:17:44.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:44.651926+0000) 2022-01-31T21:17:44.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:44 smithi167 conmon[54076]: debug 2022-01-31T21:17:44.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:44.575854+0000) 2022-01-31T21:17:45.101 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:44 smithi171 conmon[41853]: debug 2022-01-31T21:17:44.806+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:44.807290+0000) 2022-01-31T21:17:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:45 smithi171 conmon[46715]: debug 2022-01-31T21:17:45.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:45.102976+0000) 2022-01-31T21:17:45.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:45 smithi167 conmon[60316]: debug 2022-01-31T21:17:45.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:45.328038+0000) 2022-01-31T21:17:45.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:45 smithi171 conmon[51620]: debug 2022-01-31T21:17:45.682+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:45.682731+0000) 2022-01-31T21:17:45.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:45 smithi167 conmon[49112]: debug 2022-01-31T21:17:45.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:45.652102+0000) 2022-01-31T21:17:45.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:45 smithi167 conmon[54076]: debug 2022-01-31T21:17:45.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:45.576060+0000) 2022-01-31T21:17:46.101 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:45 smithi171 conmon[41853]: debug 2022-01-31T21:17:45.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:45.807457+0000) 2022-01-31T21:17:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:46 smithi171 conmon[46715]: debug 2022-01-31T21:17:46.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:46.103130+0000) 2022-01-31T21:17:46.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:46 smithi167 conmon[60316]: debug 2022-01-31T21:17:46.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:46.328171+0000) 2022-01-31T21:17:46.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:46 smithi171 conmon[51620]: debug 2022-01-31T21:17:46.682+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:46.682928+0000) 2022-01-31T21:17:46.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:46 smithi167 conmon[49112]: debug 2022-01-31T21:17:46.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:46.652288+0000) 2022-01-31T21:17:46.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:46 smithi167 conmon[54076]: debug 2022-01-31T21:17:46.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:46.576250+0000) 2022-01-31T21:17:47.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:46 smithi171 conmon[41853]: debug 2022-01-31T21:17:46.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:46.807615+0000) 2022-01-31T21:17:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:47 smithi171 conmon[46715]: debug 2022-01-31T21:17:47.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:47.103291+0000) 2022-01-31T21:17:47.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:47 smithi167 conmon[60316]: debug 2022-01-31T21:17:47.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:47.328350+0000) 2022-01-31T21:17:47.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:47 smithi171 conmon[51620]: debug 2022-01-31T21:17:47.682+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:47.683132+0000) 2022-01-31T21:17:47.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:47 smithi167 conmon[49112]: debug 2022-01-31T21:17:47.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:47.652457+0000) 2022-01-31T21:17:47.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:47 smithi167 conmon[54076]: debug 2022-01-31T21:17:47.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:47.576381+0000) 2022-01-31T21:17:48.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:47 smithi171 conmon[41853]: debug 2022-01-31T21:17:47.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:47.807767+0000) 2022-01-31T21:17:48.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:48 smithi167 conmon[49112]: debug 2022-01-31T21:17:48.106+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.107240+0000) 2022-01-31T21:17:48.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:48 smithi167 conmon[54076]: debug 2022-01-31T21:17:48.105+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.106202+0000) 2022-01-31T21:17:48.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:48 smithi167 conmon[60316]: debug 2022-01-31T21:17:48.105+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.105979+0000) 2022-01-31T21:17:48.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[35325]: debug 2022-01-31T21:17:48.116+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118500 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:48.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[41853]: debug 2022-01-31T21:17:48.106+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.106793+0000) 2022-01-31T21:17:48.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[51620]: debug 2022-01-31T21:17:48.106+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.107050+0000) 2022-01-31T21:17:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[46715]: debug 2022-01-31T21:17:48.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.103490+0000) 2022-01-31T21:17:48.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[46715]: debug 2022-01-31T21:17:48.104+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.105602+0000) 2022-01-31T21:17:48.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:48 smithi167 conmon[60316]: debug 2022-01-31T21:17:48.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.328537+0000) 2022-01-31T21:17:48.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[51620]: debug 2022-01-31T21:17:48.683+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.683354+0000) 2022-01-31T21:17:48.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:48 smithi167 conmon[49112]: debug 2022-01-31T21:17:48.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.652601+0000) 2022-01-31T21:17:48.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:48 smithi167 conmon[54076]: debug 2022-01-31T21:17:48.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.576575+0000) 2022-01-31T21:17:49.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:48 smithi171 conmon[41853]: debug 2022-01-31T21:17:48.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:48.807866+0000) 2022-01-31T21:17:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:49 smithi171 conmon[46715]: debug 2022-01-31T21:17:49.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:49.103717+0000) 2022-01-31T21:17:49.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:49 smithi167 conmon[60316]: debug 2022-01-31T21:17:49.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:49.328685+0000) 2022-01-31T21:17:49.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:49 smithi171 conmon[51620]: debug 2022-01-31T21:17:49.682+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:49.683522+0000) 2022-01-31T21:17:49.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:49 smithi167 conmon[49112]: debug 2022-01-31T21:17:49.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:49.652718+0000) 2022-01-31T21:17:49.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:49 smithi167 conmon[54076]: debug 2022-01-31T21:17:49.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:49.576669+0000) 2022-01-31T21:17:50.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:49 smithi171 conmon[41853]: debug 2022-01-31T21:17:49.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:49.807983+0000) 2022-01-31T21:17:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:50 smithi171 conmon[46715]: debug 2022-01-31T21:17:50.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:50.103837+0000) 2022-01-31T21:17:50.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:50 smithi167 conmon[60316]: debug 2022-01-31T21:17:50.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:50.328858+0000) 2022-01-31T21:17:50.766 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:50 smithi171 conmon[51620]: debug 2022-01-31T21:17:50.683+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:50.683706+0000) 2022-01-31T21:17:50.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:50 smithi167 conmon[49112]: debug 2022-01-31T21:17:50.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:50.652891+0000) 2022-01-31T21:17:50.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:50 smithi167 conmon[54076]: debug 2022-01-31T21:17:50.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:50.576791+0000) 2022-01-31T21:17:51.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:50 smithi171 conmon[41853]: debug 2022-01-31T21:17:50.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:50.808172+0000) 2022-01-31T21:17:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:51 smithi171 conmon[46715]: debug 2022-01-31T21:17:51.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:51.104040+0000) 2022-01-31T21:17:51.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:51 smithi167 conmon[60316]: debug 2022-01-31T21:17:51.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:51.329048+0000) 2022-01-31T21:17:51.766 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:51 smithi171 conmon[51620]: debug 2022-01-31T21:17:51.683+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:51.683847+0000) 2022-01-31T21:17:51.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:51 smithi167 conmon[54076]: debug 2022-01-31T21:17:51.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:51.576947+0000) 2022-01-31T21:17:51.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:51 smithi167 conmon[49112]: debug 2022-01-31T21:17:51.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:51.653083+0000) 2022-01-31T21:17:52.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:51 smithi171 conmon[41853]: debug 2022-01-31T21:17:51.808+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:51.808359+0000) 2022-01-31T21:17:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:52 smithi171 conmon[46715]: debug 2022-01-31T21:17:52.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:52.104189+0000) 2022-01-31T21:17:52.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:52 smithi167 conmon[60316]: debug 2022-01-31T21:17:52.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:52.329208+0000) 2022-01-31T21:17:52.766 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:52 smithi171 conmon[51620]: debug 2022-01-31T21:17:52.683+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:52.683993+0000) 2022-01-31T21:17:52.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:52 smithi167 conmon[49112]: debug 2022-01-31T21:17:52.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:52.653287+0000) 2022-01-31T21:17:52.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:52 smithi167 conmon[54076]: debug 2022-01-31T21:17:52.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:52.577139+0000) 2022-01-31T21:17:53.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:52 smithi171 conmon[41853]: debug 2022-01-31T21:17:52.807+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:52.808554+0000) 2022-01-31T21:17:53.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:53 smithi167 conmon[54076]: debug 2022-01-31T21:17:53.126+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.126887+0000) 2022-01-31T21:17:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:53 smithi167 conmon[49112]: debug 2022-01-31T21:17:53.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.119980+0000) 2022-01-31T21:17:53.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:53 smithi167 conmon[60316]: debug 2022-01-31T21:17:53.118+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.119242+0000) 2022-01-31T21:17:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[41853]: debug 2022-01-31T21:17:53.119+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.120289+0000) 2022-01-31T21:17:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[46715]: debug 2022-01-31T21:17:53.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.104331+0000) 2022-01-31T21:17:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[46715]: debug 2022-01-31T21:17:53.118+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.119437+0000) 2022-01-31T21:17:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[51620]: debug 2022-01-31T21:17:53.119+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.120521+0000) 2022-01-31T21:17:53.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[35325]: debug 2022-01-31T21:17:53.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[35325]: 2022-01-31T21:17:53.134+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118609 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:53.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:53 smithi167 conmon[60316]: debug 2022-01-31T21:17:53.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.329367+0000) 2022-01-31T21:17:53.766 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[51620]: debug 2022-01-31T21:17:53.683+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.684128+0000) 2022-01-31T21:17:53.767 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[35325]: debug 2022-01-31T21:17:53.655+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:17:53.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:53 smithi167 conmon[54076]: debug 2022-01-31T21:17:53.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.577232+0000) 2022-01-31T21:17:53.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:53 smithi167 conmon[49112]: debug 2022-01-31T21:17:53.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.653415+0000) 2022-01-31T21:17:54.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:53 smithi171 conmon[41853]: debug 2022-01-31T21:17:53.808+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:53.808740+0000) 2022-01-31T21:17:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:54 smithi171 conmon[46715]: debug 2022-01-31T21:17:54.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:54.104482+0000) 2022-01-31T21:17:54.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:54 smithi167 conmon[60316]: debug 2022-01-31T21:17:54.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:54.329485+0000) 2022-01-31T21:17:54.766 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:54 smithi171 conmon[51620]: debug 2022-01-31T21:17:54.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:54.684281+0000) 2022-01-31T21:17:54.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:54 smithi167 conmon[49112]: debug 2022-01-31T21:17:54.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:54.653583+0000) 2022-01-31T21:17:54.884 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:54 smithi167 conmon[54076]: debug 2022-01-31T21:17:54.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:54.577396+0000) 2022-01-31T21:17:55.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:54 smithi171 conmon[41853]: debug 2022-01-31T21:17:54.808+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:54.808909+0000) 2022-01-31T21:17:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:55 smithi171 conmon[46715]: debug 2022-01-31T21:17:55.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:55.104620+0000) 2022-01-31T21:17:55.576 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:55 smithi167 conmon[60316]: debug 2022-01-31T21:17:55.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:55.329629+0000) 2022-01-31T21:17:55.766 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:55 smithi171 conmon[51620]: debug 2022-01-31T21:17:55.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:55.684471+0000) 2022-01-31T21:17:55.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:55 smithi167 conmon[54076]: debug 2022-01-31T21:17:55.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:55.577635+0000) 2022-01-31T21:17:55.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:55 smithi167 conmon[49112]: debug 2022-01-31T21:17:55.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:55.653732+0000) 2022-01-31T21:17:56.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:55 smithi171 conmon[41853]: debug 2022-01-31T21:17:55.808+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:55.809101+0000) 2022-01-31T21:17:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:56 smithi171 conmon[46715]: debug 2022-01-31T21:17:56.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:56.104782+0000) 2022-01-31T21:17:56.576 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:56 smithi167 conmon[60316]: debug 2022-01-31T21:17:56.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:56.329780+0000) 2022-01-31T21:17:56.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:56 smithi171 conmon[51620]: debug 2022-01-31T21:17:56.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:56.684652+0000) 2022-01-31T21:17:56.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:56 smithi167 conmon[54076]: debug 2022-01-31T21:17:56.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:56.577857+0000) 2022-01-31T21:17:56.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:56 smithi167 conmon[49112]: debug 2022-01-31T21:17:56.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:56.653881+0000) 2022-01-31T21:17:57.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:56 smithi171 conmon[41853]: debug 2022-01-31T21:17:56.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:56.809305+0000) 2022-01-31T21:17:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:57 smithi171 conmon[46715]: debug 2022-01-31T21:17:57.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:57.104915+0000) 2022-01-31T21:17:57.576 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:57 smithi167 conmon[60316]: debug 2022-01-31T21:17:57.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:57.329949+0000) 2022-01-31T21:17:57.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:57 smithi171 conmon[51620]: debug 2022-01-31T21:17:57.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:57.684865+0000) 2022-01-31T21:17:57.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:57 smithi167 conmon[54076]: debug 2022-01-31T21:17:57.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:57.578035+0000) 2022-01-31T21:17:57.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:57 smithi167 conmon[49112]: debug 2022-01-31T21:17:57.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:57.654079+0000) 2022-01-31T21:17:58.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:57 smithi171 conmon[41853]: debug 2022-01-31T21:17:57.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:57.809516+0000) 2022-01-31T21:17:58.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:58 smithi167 conmon[49112]: debug 2022-01-31T21:17:58.137+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.138136+0000) 2022-01-31T21:17:58.140 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:58 smithi167 conmon[54076]: debug 2022-01-31T21:17:58.138+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.138717+0000) 2022-01-31T21:17:58.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:58 smithi167 conmon[60316]: debug 2022-01-31T21:17:58.137+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.137836+0000) 2022-01-31T21:17:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[41853]: debug 2022-01-31T21:17:58.137+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.138217+0000) 2022-01-31T21:17:58.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[51620]: debug 2022-01-31T21:17:58.137+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.138319+0000) 2022-01-31T21:17:58.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[35325]: debug 2022-01-31T21:17:58.148+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118722 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:17:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[46715]: debug 2022-01-31T21:17:58.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.105118+0000) 2022-01-31T21:17:58.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[46715]: debug 2022-01-31T21:17:58.135+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.136572+0000) 2022-01-31T21:17:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:58 smithi167 conmon[60316]: debug 2022-01-31T21:17:58.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.330080+0000) 2022-01-31T21:17:58.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[51620]: debug 2022-01-31T21:17:58.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.685072+0000) 2022-01-31T21:17:58.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:58 smithi167 conmon[49112]: debug 2022-01-31T21:17:58.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.654305+0000) 2022-01-31T21:17:58.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:58 smithi167 conmon[54076]: debug 2022-01-31T21:17:58.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.578231+0000) 2022-01-31T21:17:59.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:58 smithi171 conmon[41853]: debug 2022-01-31T21:17:58.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:58.809708+0000) 2022-01-31T21:17:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:17:59 smithi171 conmon[46715]: debug 2022-01-31T21:17:59.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:59.105313+0000) 2022-01-31T21:17:59.576 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:17:59 smithi167 conmon[60316]: debug 2022-01-31T21:17:59.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:59.330197+0000) 2022-01-31T21:17:59.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:17:59 smithi171 conmon[51620]: debug 2022-01-31T21:17:59.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:59.685247+0000) 2022-01-31T21:17:59.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:17:59 smithi167 conmon[49112]: debug 2022-01-31T21:17:59.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:59.654474+0000) 2022-01-31T21:17:59.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:17:59 smithi167 conmon[54076]: debug 2022-01-31T21:17:59.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:59.578378+0000) 2022-01-31T21:18:00.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:17:59 smithi171 conmon[41853]: debug 2022-01-31T21:17:59.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:17:59.809851+0000) 2022-01-31T21:18:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:00 smithi171 conmon[46715]: debug 2022-01-31T21:18:00.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:00.105468+0000) 2022-01-31T21:18:00.577 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:00 smithi167 conmon[60316]: debug 2022-01-31T21:18:00.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:00.330376+0000) 2022-01-31T21:18:00.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:00 smithi171 conmon[51620]: debug 2022-01-31T21:18:00.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:00.685406+0000) 2022-01-31T21:18:00.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:00 smithi167 conmon[49112]: debug 2022-01-31T21:18:00.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:00.654605+0000) 2022-01-31T21:18:00.894 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:00 smithi167 conmon[54076]: debug 2022-01-31T21:18:00.578+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:00.578600+0000) 2022-01-31T21:18:01.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:00 smithi171 conmon[41853]: debug 2022-01-31T21:18:00.808+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:00.810031+0000) 2022-01-31T21:18:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:01 smithi171 conmon[46715]: debug 2022-01-31T21:18:01.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:01.105602+0000) 2022-01-31T21:18:01.577 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:01 smithi167 conmon[60316]: debug 2022-01-31T21:18:01.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:01.330554+0000) 2022-01-31T21:18:01.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:01 smithi171 conmon[51620]: debug 2022-01-31T21:18:01.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:01.685581+0000) 2022-01-31T21:18:01.894 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:01 smithi167 conmon[54076]: debug 2022-01-31T21:18:01.578+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:01.578754+0000) 2022-01-31T21:18:01.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:01 smithi167 conmon[49112]: debug 2022-01-31T21:18:01.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:01.654752+0000) 2022-01-31T21:18:02.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:01 smithi171 conmon[41853]: debug 2022-01-31T21:18:01.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:01.810220+0000) 2022-01-31T21:18:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:02 smithi171 conmon[46715]: debug 2022-01-31T21:18:02.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:02.105763+0000) 2022-01-31T21:18:02.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:02 smithi167 conmon[60316]: debug 2022-01-31T21:18:02.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:02.330686+0000) 2022-01-31T21:18:02.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:02 smithi171 conmon[51620]: debug 2022-01-31T21:18:02.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:02.685724+0000) 2022-01-31T21:18:02.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:02 smithi167 conmon[49112]: debug 2022-01-31T21:18:02.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:02.654956+0000) 2022-01-31T21:18:02.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:02 smithi167 conmon[54076]: debug 2022-01-31T21:18:02.578+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:02.578915+0000) 2022-01-31T21:18:03.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:02 smithi171 conmon[41853]: debug 2022-01-31T21:18:02.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:02.810436+0000) 2022-01-31T21:18:03.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[35325]: debug 2022-01-31T21:18:03.162+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118831 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:03.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[41853]: debug 2022-01-31T21:18:03.150+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.152331+0000) 2022-01-31T21:18:03.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[51620]: debug 2022-01-31T21:18:03.151+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.153224+0000) 2022-01-31T21:18:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[46715]: debug 2022-01-31T21:18:03.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.105938+0000) 2022-01-31T21:18:03.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[46715]: debug 2022-01-31T21:18:03.150+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.152278+0000) 2022-01-31T21:18:03.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:03 smithi167 conmon[49112]: debug 2022-01-31T21:18:03.151+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.151833+0000) 2022-01-31T21:18:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:03 smithi167 conmon[54076]: debug 2022-01-31T21:18:03.152+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.152527+0000) 2022-01-31T21:18:03.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:03 smithi167 conmon[60316]: debug 2022-01-31T21:18:03.152+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.152908+0000) 2022-01-31T21:18:03.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:03 smithi167 conmon[60316]: debug 2022-01-31T21:18:03.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.330882+0000) 2022-01-31T21:18:03.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[51620]: debug 2022-01-31T21:18:03.684+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.685945+0000) 2022-01-31T21:18:03.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:03 smithi167 conmon[49112]: debug 2022-01-31T21:18:03.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.655148+0000) 2022-01-31T21:18:03.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:03 smithi167 conmon[54076]: debug 2022-01-31T21:18:03.578+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.579099+0000) 2022-01-31T21:18:04.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:03 smithi171 conmon[41853]: debug 2022-01-31T21:18:03.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:03.810631+0000) 2022-01-31T21:18:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:04 smithi171 conmon[46715]: debug 2022-01-31T21:18:04.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:04.106112+0000) 2022-01-31T21:18:04.577 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:04 smithi167 conmon[60316]: debug 2022-01-31T21:18:04.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:04.331035+0000) 2022-01-31T21:18:04.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:04 smithi171 conmon[51620]: debug 2022-01-31T21:18:04.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:04.686082+0000) 2022-01-31T21:18:04.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:04 smithi167 conmon[49112]: debug 2022-01-31T21:18:04.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:04.655346+0000) 2022-01-31T21:18:04.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:04 smithi167 conmon[54076]: debug 2022-01-31T21:18:04.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:04.579255+0000) 2022-01-31T21:18:05.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:04 smithi171 conmon[41853]: debug 2022-01-31T21:18:04.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:04.810782+0000) 2022-01-31T21:18:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:05 smithi171 conmon[46715]: debug 2022-01-31T21:18:05.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:05.106263+0000) 2022-01-31T21:18:05.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:05 smithi167 conmon[60316]: debug 2022-01-31T21:18:05.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:05.331197+0000) 2022-01-31T21:18:05.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:05 smithi171 conmon[51620]: debug 2022-01-31T21:18:05.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:05.686240+0000) 2022-01-31T21:18:05.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:05 smithi167 conmon[49112]: debug 2022-01-31T21:18:05.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:05.655557+0000) 2022-01-31T21:18:05.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:05 smithi167 conmon[54076]: debug 2022-01-31T21:18:05.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:05.579475+0000) 2022-01-31T21:18:06.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:05 smithi171 conmon[41853]: debug 2022-01-31T21:18:05.810+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:05.810987+0000) 2022-01-31T21:18:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:06 smithi171 conmon[46715]: debug 2022-01-31T21:18:06.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:06.106399+0000) 2022-01-31T21:18:06.578 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:06 smithi167 conmon[60316]: debug 2022-01-31T21:18:06.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:06.331394+0000) 2022-01-31T21:18:06.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:06 smithi171 conmon[51620]: debug 2022-01-31T21:18:06.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:06.686393+0000) 2022-01-31T21:18:06.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:06 smithi167 conmon[49112]: debug 2022-01-31T21:18:06.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:06.655722+0000) 2022-01-31T21:18:06.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:06 smithi167 conmon[54076]: debug 2022-01-31T21:18:06.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:06.579675+0000) 2022-01-31T21:18:07.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:06 smithi171 conmon[41853]: debug 2022-01-31T21:18:06.810+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:06.811167+0000) 2022-01-31T21:18:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:07 smithi171 conmon[46715]: debug 2022-01-31T21:18:07.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:07.106587+0000) 2022-01-31T21:18:07.578 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:07 smithi167 conmon[60316]: debug 2022-01-31T21:18:07.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:07.331494+0000) 2022-01-31T21:18:07.769 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:07 smithi171 conmon[51620]: debug 2022-01-31T21:18:07.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:07.686557+0000) 2022-01-31T21:18:07.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:07 smithi167 conmon[49112]: debug 2022-01-31T21:18:07.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:07.655881+0000) 2022-01-31T21:18:07.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:07 smithi167 conmon[54076]: debug 2022-01-31T21:18:07.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:07.579855+0000) 2022-01-31T21:18:08.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:07 smithi171 conmon[41853]: debug 2022-01-31T21:18:07.810+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:07.811385+0000) 2022-01-31T21:18:08.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[35325]: debug 2022-01-31T21:18:08.177+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 118944 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[41853]: debug 2022-01-31T21:18:08.165+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.167267+0000) 2022-01-31T21:18:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[51620]: debug 2022-01-31T21:18:08.166+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.168003+0000) 2022-01-31T21:18:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[46715]: debug 2022-01-31T21:18:08.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.106785+0000) 2022-01-31T21:18:08.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[46715]: debug 2022-01-31T21:18:08.165+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.166878+0000) 2022-01-31T21:18:08.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:08 smithi167 conmon[49112]: debug 2022-01-31T21:18:08.167+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.167877+0000) 2022-01-31T21:18:08.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:08 smithi167 conmon[54076]: debug 2022-01-31T21:18:08.167+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.168027+0000) 2022-01-31T21:18:08.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:08 smithi167 conmon[60316]: debug 2022-01-31T21:18:08.168+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.168517+0000) 2022-01-31T21:18:08.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:08 smithi167 conmon[60316]: debug 2022-01-31T21:18:08.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.331674+0000) 2022-01-31T21:18:08.769 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[35325]: debug 2022-01-31T21:18:08.655+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:18:08.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[51620]: debug 2022-01-31T21:18:08.685+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.686772+0000) 2022-01-31T21:18:08.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:08 smithi167 conmon[49112]: debug 2022-01-31T21:18:08.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.656025+0000) 2022-01-31T21:18:08.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:08 smithi167 conmon[54076]: debug 2022-01-31T21:18:08.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.580010+0000) 2022-01-31T21:18:09.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:08 smithi171 conmon[41853]: debug 2022-01-31T21:18:08.809+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:08.811540+0000) 2022-01-31T21:18:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:09 smithi171 conmon[46715]: debug 2022-01-31T21:18:09.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:09.106979+0000) 2022-01-31T21:18:09.578 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:09 smithi167 conmon[60316]: debug 2022-01-31T21:18:09.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:09.331857+0000) 2022-01-31T21:18:09.769 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:09 smithi171 conmon[51620]: debug 2022-01-31T21:18:09.686+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:09.686933+0000) 2022-01-31T21:18:09.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:09 smithi167 conmon[49112]: debug 2022-01-31T21:18:09.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:09.656178+0000) 2022-01-31T21:18:09.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:09 smithi167 conmon[54076]: debug 2022-01-31T21:18:09.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:09.580159+0000) 2022-01-31T21:18:10.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:09 smithi171 conmon[41853]: debug 2022-01-31T21:18:09.810+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:09.811692+0000) 2022-01-31T21:18:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:10 smithi171 conmon[46715]: debug 2022-01-31T21:18:10.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:10.107105+0000) 2022-01-31T21:18:10.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:10 smithi167 conmon[60316]: debug 2022-01-31T21:18:10.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:10.332030+0000) 2022-01-31T21:18:10.769 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:10 smithi171 conmon[51620]: debug 2022-01-31T21:18:10.686+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:10.687119+0000) 2022-01-31T21:18:10.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:10 smithi167 conmon[49112]: debug 2022-01-31T21:18:10.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:10.656353+0000) 2022-01-31T21:18:10.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:10 smithi167 conmon[54076]: debug 2022-01-31T21:18:10.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:10.580362+0000) 2022-01-31T21:18:11.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:10 smithi171 conmon[41853]: debug 2022-01-31T21:18:10.811+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:10.811884+0000) 2022-01-31T21:18:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:11 smithi171 conmon[46715]: debug 2022-01-31T21:18:11.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:11.107256+0000) 2022-01-31T21:18:11.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:11 smithi167 conmon[60316]: debug 2022-01-31T21:18:11.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:11.332165+0000) 2022-01-31T21:18:11.769 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:11 smithi171 conmon[51620]: debug 2022-01-31T21:18:11.686+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:11.687290+0000) 2022-01-31T21:18:11.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:11 smithi167 conmon[49112]: debug 2022-01-31T21:18:11.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:11.656583+0000) 2022-01-31T21:18:11.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:11 smithi167 conmon[54076]: debug 2022-01-31T21:18:11.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:11.580544+0000) 2022-01-31T21:18:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:11 smithi171 conmon[41853]: debug 2022-01-31T21:18:11.811+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:11.812070+0000) 2022-01-31T21:18:12.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:12 smithi171 conmon[46715]: debug 2022-01-31T21:18:12.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:12.107474+0000) 2022-01-31T21:18:12.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:12 smithi167 conmon[60316]: debug 2022-01-31T21:18:12.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:12.332309+0000) 2022-01-31T21:18:12.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:12 smithi171 conmon[51620]: debug 2022-01-31T21:18:12.686+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:12.687444+0000) 2022-01-31T21:18:12.951 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:12 smithi167 conmon[49112]: debug 2022-01-31T21:18:12.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:12.656697+0000) 2022-01-31T21:18:12.952 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:12 smithi167 conmon[54076]: debug 2022-01-31T21:18:12.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:12.580706+0000) 2022-01-31T21:18:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:12 smithi171 conmon[41853]: debug 2022-01-31T21:18:12.811+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:12.812261+0000) 2022-01-31T21:18:13.481 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[35325]: debug 2022-01-31T21:18:13.193+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119051 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:13.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[41853]: debug 2022-01-31T21:18:13.181+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.182363+0000) 2022-01-31T21:18:13.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[51620]: debug 2022-01-31T21:18:13.181+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.182091+0000) 2022-01-31T21:18:13.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[46715]: debug 2022-01-31T21:18:13.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.107656+0000) 2022-01-31T21:18:13.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[46715]: debug 2022-01-31T21:18:13.181+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.181762+0000) 2022-01-31T21:18:13.579 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:13 smithi167 conmon[49112]: debug 2022-01-31T21:18:13.179+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.181311+0000) 2022-01-31T21:18:13.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:13 smithi167 conmon[54076]: debug 2022-01-31T21:18:13.180+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.181994+0000) 2022-01-31T21:18:13.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:13 smithi167 conmon[60316]: debug 2022-01-31T21:18:13.181+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.182935+0000) 2022-01-31T21:18:13.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:13 smithi167 conmon[60316]: debug 2022-01-31T21:18:13.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.332443+0000) 2022-01-31T21:18:13.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[51620]: debug 2022-01-31T21:18:13.686+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.687630+0000) 2022-01-31T21:18:13.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:13 smithi167 conmon[49112]: debug 2022-01-31T21:18:13.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.656906+0000) 2022-01-31T21:18:13.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:13 smithi167 conmon[54076]: debug 2022-01-31T21:18:13.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.580890+0000) 2022-01-31T21:18:14.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:13 smithi171 conmon[41853]: debug 2022-01-31T21:18:13.811+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:13.812424+0000) 2022-01-31T21:18:14.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:14 smithi171 conmon[46715]: debug 2022-01-31T21:18:14.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:14.107782+0000) 2022-01-31T21:18:14.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:14 smithi167 conmon[60316]: debug 2022-01-31T21:18:14.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:14.332586+0000) 2022-01-31T21:18:14.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:14 smithi171 conmon[51620]: debug 2022-01-31T21:18:14.687+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:14.687784+0000) 2022-01-31T21:18:14.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:14 smithi167 conmon[54076]: debug 2022-01-31T21:18:14.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:14.581046+0000) 2022-01-31T21:18:14.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:14 smithi167 conmon[49112]: debug 2022-01-31T21:18:14.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:14.657083+0000) 2022-01-31T21:18:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:14 smithi171 conmon[41853]: debug 2022-01-31T21:18:14.811+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:14.812596+0000) 2022-01-31T21:18:15.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:15 smithi171 conmon[46715]: debug 2022-01-31T21:18:15.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:15.107879+0000) 2022-01-31T21:18:15.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:15 smithi167 conmon[60316]: debug 2022-01-31T21:18:15.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:15.332748+0000) 2022-01-31T21:18:15.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:15 smithi171 conmon[51620]: debug 2022-01-31T21:18:15.687+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:15.687941+0000) 2022-01-31T21:18:15.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:15 smithi167 conmon[49112]: debug 2022-01-31T21:18:15.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:15.657301+0000) 2022-01-31T21:18:15.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:15 smithi167 conmon[54076]: debug 2022-01-31T21:18:15.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:15.581228+0000) 2022-01-31T21:18:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:15 smithi171 conmon[41853]: debug 2022-01-31T21:18:15.812+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:15.812802+0000) 2022-01-31T21:18:16.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:16 smithi171 conmon[46715]: debug 2022-01-31T21:18:16.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:16.108016+0000) 2022-01-31T21:18:16.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:16 smithi167 conmon[60316]: debug 2022-01-31T21:18:16.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:16.332917+0000) 2022-01-31T21:18:16.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:16 smithi171 conmon[51620]: debug 2022-01-31T21:18:16.687+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:16.688102+0000) 2022-01-31T21:18:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:16 smithi167 conmon[49112]: debug 2022-01-31T21:18:16.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:16.657514+0000) 2022-01-31T21:18:16.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:16 smithi167 conmon[54076]: debug 2022-01-31T21:18:16.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:16.581431+0000) 2022-01-31T21:18:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:16 smithi171 conmon[41853]: debug 2022-01-31T21:18:16.812+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:16.813032+0000) 2022-01-31T21:18:17.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:17 smithi171 conmon[46715]: debug 2022-01-31T21:18:17.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:17.108189+0000) 2022-01-31T21:18:17.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:17 smithi167 conmon[60316]: debug 2022-01-31T21:18:17.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:17.333103+0000) 2022-01-31T21:18:17.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:17 smithi171 conmon[51620]: debug 2022-01-31T21:18:17.687+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:17.688291+0000) 2022-01-31T21:18:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:17 smithi167 conmon[49112]: debug 2022-01-31T21:18:17.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:17.657701+0000) 2022-01-31T21:18:17.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:17 smithi167 conmon[54076]: debug 2022-01-31T21:18:17.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:17.581616+0000) 2022-01-31T21:18:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:17 smithi171 conmon[41853]: debug 2022-01-31T21:18:17.812+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:17.813214+0000) 2022-01-31T21:18:18.482 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[35325]: debug 2022-01-31T21:18:18.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119163 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:18.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[41853]: debug 2022-01-31T21:18:18.196+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.197599+0000) 2022-01-31T21:18:18.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[46715]: debug 2022-01-31T21:18:18.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.108348+0000) 2022-01-31T21:18:18.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[46715]: debug 2022-01-31T21:18:18.195+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.196043+0000) 2022-01-31T21:18:18.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[51620]: debug 2022-01-31T21:18:18.195+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.196377+0000) 2022-01-31T21:18:18.580 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:18 smithi167 conmon[49112]: debug 2022-01-31T21:18:18.195+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.197281+0000) 2022-01-31T21:18:18.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:18 smithi167 conmon[54076]: debug 2022-01-31T21:18:18.195+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.196907+0000) 2022-01-31T21:18:18.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:18 smithi167 conmon[60316]: debug 2022-01-31T21:18:18.195+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.196989+0000) 2022-01-31T21:18:18.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:18 smithi167 conmon[60316]: debug 2022-01-31T21:18:18.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.333289+0000) 2022-01-31T21:18:18.770 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[51620]: debug 2022-01-31T21:18:18.687+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.688492+0000) 2022-01-31T21:18:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:18 smithi167 conmon[49112]: debug 2022-01-31T21:18:18.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.660445+0000) 2022-01-31T21:18:18.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:18 smithi167 conmon[54076]: debug 2022-01-31T21:18:18.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.581832+0000) 2022-01-31T21:18:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:18 smithi171 conmon[41853]: debug 2022-01-31T21:18:18.812+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:18.813415+0000) 2022-01-31T21:18:19.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:19 smithi171 conmon[46715]: debug 2022-01-31T21:18:19.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:19.108520+0000) 2022-01-31T21:18:19.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:19 smithi167 conmon[60316]: debug 2022-01-31T21:18:19.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:19.333379+0000) 2022-01-31T21:18:19.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:19 smithi171 conmon[51620]: debug 2022-01-31T21:18:19.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:19.688669+0000) 2022-01-31T21:18:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:19 smithi167 conmon[49112]: debug 2022-01-31T21:18:19.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:19.660565+0000) 2022-01-31T21:18:19.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:19 smithi167 conmon[54076]: debug 2022-01-31T21:18:19.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:19.581984+0000) 2022-01-31T21:18:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:19 smithi171 conmon[41853]: debug 2022-01-31T21:18:19.812+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:19.813551+0000) 2022-01-31T21:18:20.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:20 smithi171 conmon[46715]: debug 2022-01-31T21:18:20.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:20.108609+0000) 2022-01-31T21:18:20.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:20 smithi167 conmon[60316]: debug 2022-01-31T21:18:20.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:20.333572+0000) 2022-01-31T21:18:20.771 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:20 smithi171 conmon[51620]: debug 2022-01-31T21:18:20.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:20.688847+0000) 2022-01-31T21:18:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:20 smithi167 conmon[49112]: debug 2022-01-31T21:18:20.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:20.660729+0000) 2022-01-31T21:18:20.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:20 smithi167 conmon[54076]: debug 2022-01-31T21:18:20.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:20.582180+0000) 2022-01-31T21:18:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:20 smithi171 conmon[41853]: debug 2022-01-31T21:18:20.813+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:20.813719+0000) 2022-01-31T21:18:21.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:21 smithi171 conmon[46715]: debug 2022-01-31T21:18:21.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:21.108746+0000) 2022-01-31T21:18:21.580 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:21 smithi167 conmon[60316]: debug 2022-01-31T21:18:21.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:21.333751+0000) 2022-01-31T21:18:21.771 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:21 smithi171 conmon[51620]: debug 2022-01-31T21:18:21.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:21.688981+0000) 2022-01-31T21:18:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:21 smithi167 conmon[49112]: debug 2022-01-31T21:18:21.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:21.660849+0000) 2022-01-31T21:18:21.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:21 smithi167 conmon[54076]: debug 2022-01-31T21:18:21.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:21.582365+0000) 2022-01-31T21:18:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:21 smithi171 conmon[41853]: debug 2022-01-31T21:18:21.813+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:21.813877+0000) 2022-01-31T21:18:22.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:22 smithi171 conmon[46715]: debug 2022-01-31T21:18:22.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:22.108887+0000) 2022-01-31T21:18:22.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:22 smithi167 conmon[60316]: debug 2022-01-31T21:18:22.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:22.333985+0000) 2022-01-31T21:18:22.771 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:22 smithi171 conmon[51620]: debug 2022-01-31T21:18:22.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:22.689163+0000) 2022-01-31T21:18:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:22 smithi167 conmon[49112]: debug 2022-01-31T21:18:22.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:22.661032+0000) 2022-01-31T21:18:22.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:22 smithi167 conmon[54076]: debug 2022-01-31T21:18:22.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:22.582555+0000) 2022-01-31T21:18:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:22 smithi171 conmon[41853]: debug 2022-01-31T21:18:22.813+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:22.814065+0000) 2022-01-31T21:18:23.483 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[35325]: debug 2022-01-31T21:18:23.222+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119271 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:23.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[41853]: debug 2022-01-31T21:18:23.211+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.212131+0000) 2022-01-31T21:18:23.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[46715]: debug 2022-01-31T21:18:23.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.108989+0000) 2022-01-31T21:18:23.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[46715]: debug 2022-01-31T21:18:23.210+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.210848+0000) 2022-01-31T21:18:23.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[51620]: debug 2022-01-31T21:18:23.211+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.211848+0000) 2022-01-31T21:18:23.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:23 smithi167 conmon[49112]: debug 2022-01-31T21:18:23.211+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.212661+0000) 2022-01-31T21:18:23.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:23 smithi167 conmon[54076]: debug 2022-01-31T21:18:23.210+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.212030+0000) 2022-01-31T21:18:23.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:23 smithi167 conmon[60316]: debug 2022-01-31T21:18:23.210+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.212396+0000) 2022-01-31T21:18:23.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:23 smithi167 conmon[60316]: debug 2022-01-31T21:18:23.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.334175+0000) 2022-01-31T21:18:23.771 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[35325]: debug 2022-01-31T21:18:23.656+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:18:23.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[51620]: debug 2022-01-31T21:18:23.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.689320+0000) 2022-01-31T21:18:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:23 smithi167 conmon[49112]: debug 2022-01-31T21:18:23.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.661196+0000) 2022-01-31T21:18:23.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:23 smithi167 conmon[54076]: debug 2022-01-31T21:18:23.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.582752+0000) 2022-01-31T21:18:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:23 smithi171 conmon[41853]: debug 2022-01-31T21:18:23.813+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:23.814242+0000) 2022-01-31T21:18:24.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:24 smithi171 conmon[46715]: debug 2022-01-31T21:18:24.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:24.109105+0000) 2022-01-31T21:18:24.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:24 smithi167 conmon[60316]: debug 2022-01-31T21:18:24.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:24.334356+0000) 2022-01-31T21:18:24.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:24 smithi171 conmon[51620]: debug 2022-01-31T21:18:24.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:24.689475+0000) 2022-01-31T21:18:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:24 smithi167 conmon[49112]: debug 2022-01-31T21:18:24.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:24.661372+0000) 2022-01-31T21:18:24.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:24 smithi167 conmon[54076]: debug 2022-01-31T21:18:24.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:24.582905+0000) 2022-01-31T21:18:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:24 smithi171 conmon[41853]: debug 2022-01-31T21:18:24.813+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:24.814392+0000) 2022-01-31T21:18:25.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:25 smithi171 conmon[46715]: debug 2022-01-31T21:18:25.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:25.109241+0000) 2022-01-31T21:18:25.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:25 smithi167 conmon[60316]: debug 2022-01-31T21:18:25.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:25.334536+0000) 2022-01-31T21:18:25.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:25 smithi171 conmon[51620]: debug 2022-01-31T21:18:25.688+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:25.689660+0000) 2022-01-31T21:18:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:25 smithi167 conmon[49112]: debug 2022-01-31T21:18:25.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:25.661588+0000) 2022-01-31T21:18:25.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:25 smithi167 conmon[54076]: debug 2022-01-31T21:18:25.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:25.583097+0000) 2022-01-31T21:18:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:25 smithi171 conmon[41853]: debug 2022-01-31T21:18:25.814+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:25.814576+0000) 2022-01-31T21:18:26.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:26 smithi171 conmon[46715]: debug 2022-01-31T21:18:26.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:26.109402+0000) 2022-01-31T21:18:26.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:26 smithi167 conmon[60316]: debug 2022-01-31T21:18:26.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:26.334654+0000) 2022-01-31T21:18:26.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:26 smithi171 conmon[51620]: debug 2022-01-31T21:18:26.689+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:26.689860+0000) 2022-01-31T21:18:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:26 smithi167 conmon[49112]: debug 2022-01-31T21:18:26.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:26.661774+0000) 2022-01-31T21:18:26.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:26 smithi167 conmon[54076]: debug 2022-01-31T21:18:26.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:26.583291+0000) 2022-01-31T21:18:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:26 smithi171 conmon[41853]: debug 2022-01-31T21:18:26.814+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:26.814741+0000) 2022-01-31T21:18:27.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:27 smithi171 conmon[46715]: debug 2022-01-31T21:18:27.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:27.109586+0000) 2022-01-31T21:18:27.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:27 smithi167 conmon[60316]: debug 2022-01-31T21:18:27.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:27.334931+0000) 2022-01-31T21:18:27.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:27 smithi171 conmon[51620]: debug 2022-01-31T21:18:27.689+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:27.690046+0000) 2022-01-31T21:18:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:27 smithi167 conmon[49112]: debug 2022-01-31T21:18:27.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:27.661961+0000) 2022-01-31T21:18:27.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:27 smithi167 conmon[54076]: debug 2022-01-31T21:18:27.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:27.583480+0000) 2022-01-31T21:18:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:27 smithi171 conmon[41853]: debug 2022-01-31T21:18:27.814+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:27.814958+0000) 2022-01-31T21:18:28.484 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[35325]: debug 2022-01-31T21:18:28.237+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119383 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:28.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[41853]: debug 2022-01-31T21:18:28.225+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.225984+0000) 2022-01-31T21:18:28.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[51620]: debug 2022-01-31T21:18:28.224+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.225166+0000) 2022-01-31T21:18:28.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[46715]: debug 2022-01-31T21:18:28.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.109784+0000) 2022-01-31T21:18:28.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[46715]: debug 2022-01-31T21:18:28.224+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.225092+0000) 2022-01-31T21:18:28.582 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:28 smithi167 conmon[49112]: debug 2022-01-31T21:18:28.225+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.226633+0000) 2022-01-31T21:18:28.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:28 smithi167 conmon[54076]: debug 2022-01-31T21:18:28.223+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.225014+0000) 2022-01-31T21:18:28.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:28 smithi167 conmon[60316]: debug 2022-01-31T21:18:28.225+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.226722+0000) 2022-01-31T21:18:28.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:28 smithi167 conmon[60316]: debug 2022-01-31T21:18:28.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.335128+0000) 2022-01-31T21:18:28.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[51620]: debug 2022-01-31T21:18:28.689+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.690246+0000) 2022-01-31T21:18:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:28 smithi167 conmon[49112]: debug 2022-01-31T21:18:28.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.662133+0000) 2022-01-31T21:18:28.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:28 smithi167 conmon[54076]: debug 2022-01-31T21:18:28.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.583686+0000) 2022-01-31T21:18:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:28 smithi171 conmon[41853]: debug 2022-01-31T21:18:28.814+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:28.815129+0000) 2022-01-31T21:18:29.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:29 smithi171 conmon[46715]: debug 2022-01-31T21:18:29.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:29.109950+0000) 2022-01-31T21:18:29.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:29 smithi167 conmon[60316]: debug 2022-01-31T21:18:29.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:29.335261+0000) 2022-01-31T21:18:29.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:29 smithi171 conmon[51620]: debug 2022-01-31T21:18:29.689+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:29.690384+0000) 2022-01-31T21:18:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:29 smithi167 conmon[49112]: debug 2022-01-31T21:18:29.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:29.662302+0000) 2022-01-31T21:18:29.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:29 smithi167 conmon[54076]: debug 2022-01-31T21:18:29.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:29.583825+0000) 2022-01-31T21:18:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:29 smithi171 conmon[41853]: debug 2022-01-31T21:18:29.814+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:29.815266+0000) 2022-01-31T21:18:30.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:30 smithi171 conmon[46715]: debug 2022-01-31T21:18:30.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:30.110053+0000) 2022-01-31T21:18:30.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:30 smithi167 conmon[60316]: debug 2022-01-31T21:18:30.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:30.335488+0000) 2022-01-31T21:18:30.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:30 smithi171 conmon[51620]: debug 2022-01-31T21:18:30.690+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:30.690545+0000) 2022-01-31T21:18:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:30 smithi167 conmon[49112]: debug 2022-01-31T21:18:30.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:30.662485+0000) 2022-01-31T21:18:30.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:30 smithi167 conmon[54076]: debug 2022-01-31T21:18:30.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:30.583988+0000) 2022-01-31T21:18:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:30 smithi171 conmon[41853]: debug 2022-01-31T21:18:30.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:30.815442+0000) 2022-01-31T21:18:31.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:31 smithi171 conmon[46715]: debug 2022-01-31T21:18:31.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:31.110257+0000) 2022-01-31T21:18:31.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:31 smithi167 conmon[60316]: debug 2022-01-31T21:18:31.334+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:31.335607+0000) 2022-01-31T21:18:31.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:31 smithi171 conmon[51620]: debug 2022-01-31T21:18:31.690+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:31.690726+0000) 2022-01-31T21:18:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:31 smithi167 conmon[49112]: debug 2022-01-31T21:18:31.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:31.662656+0000) 2022-01-31T21:18:31.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:31 smithi167 conmon[54076]: debug 2022-01-31T21:18:31.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:31.584143+0000) 2022-01-31T21:18:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:31 smithi171 conmon[41853]: debug 2022-01-31T21:18:31.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:31.815635+0000) 2022-01-31T21:18:32.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:32 smithi171 conmon[46715]: debug 2022-01-31T21:18:32.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:32.110461+0000) 2022-01-31T21:18:32.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:32 smithi167 conmon[60316]: debug 2022-01-31T21:18:32.334+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:32.335750+0000) 2022-01-31T21:18:32.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:32 smithi171 conmon[51620]: debug 2022-01-31T21:18:32.690+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:32.690961+0000) 2022-01-31T21:18:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:32 smithi167 conmon[49112]: debug 2022-01-31T21:18:32.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:32.662819+0000) 2022-01-31T21:18:32.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:32 smithi167 conmon[54076]: debug 2022-01-31T21:18:32.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:32.584297+0000) 2022-01-31T21:18:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:32 smithi171 conmon[41853]: debug 2022-01-31T21:18:32.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:32.815797+0000) 2022-01-31T21:18:33.485 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[35325]: debug 2022-01-31T21:18:33.252+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119491 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:33.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[41853]: debug 2022-01-31T21:18:33.240+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.241055+0000) 2022-01-31T21:18:33.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[51620]: debug 2022-01-31T21:18:33.240+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.240813+0000) 2022-01-31T21:18:33.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[46715]: debug 2022-01-31T21:18:33.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.110631+0000) 2022-01-31T21:18:33.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[46715]: debug 2022-01-31T21:18:33.240+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.241658+0000) 2022-01-31T21:18:33.583 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:33 smithi167 conmon[49112]: debug 2022-01-31T21:18:33.240+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.241917+0000) 2022-01-31T21:18:33.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:33 smithi167 conmon[54076]: debug 2022-01-31T21:18:33.239+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.240672+0000) 2022-01-31T21:18:33.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:33 smithi167 conmon[60316]: debug 2022-01-31T21:18:33.240+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.241790+0000) 2022-01-31T21:18:33.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:33 smithi167 conmon[60316]: debug 2022-01-31T21:18:33.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.335940+0000) 2022-01-31T21:18:33.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[51620]: debug 2022-01-31T21:18:33.690+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.691113+0000) 2022-01-31T21:18:33.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:33 smithi167 conmon[54076]: debug 2022-01-31T21:18:33.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.584495+0000) 2022-01-31T21:18:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:33 smithi167 conmon[49112]: debug 2022-01-31T21:18:33.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.663011+0000) 2022-01-31T21:18:34.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:33 smithi171 conmon[41853]: debug 2022-01-31T21:18:33.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:33.815975+0000) 2022-01-31T21:18:34.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:34 smithi171 conmon[46715]: debug 2022-01-31T21:18:34.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:34.110849+0000) 2022-01-31T21:18:34.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:34 smithi167 conmon[60316]: debug 2022-01-31T21:18:34.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:34.336089+0000) 2022-01-31T21:18:34.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:34 smithi171 conmon[51620]: debug 2022-01-31T21:18:34.690+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:34.691309+0000) 2022-01-31T21:18:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:34 smithi167 conmon[49112]: debug 2022-01-31T21:18:34.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:34.663220+0000) 2022-01-31T21:18:34.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:34 smithi167 conmon[54076]: debug 2022-01-31T21:18:34.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:34.584666+0000) 2022-01-31T21:18:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:34 smithi171 conmon[41853]: debug 2022-01-31T21:18:34.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:34.816127+0000) 2022-01-31T21:18:35.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:35 smithi171 conmon[46715]: debug 2022-01-31T21:18:35.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:35.110965+0000) 2022-01-31T21:18:35.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:35 smithi167 conmon[60316]: debug 2022-01-31T21:18:35.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:35.336248+0000) 2022-01-31T21:18:35.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:35 smithi171 conmon[51620]: debug 2022-01-31T21:18:35.691+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:35.691510+0000) 2022-01-31T21:18:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:35 smithi167 conmon[49112]: debug 2022-01-31T21:18:35.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:35.663442+0000) 2022-01-31T21:18:35.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:35 smithi167 conmon[54076]: debug 2022-01-31T21:18:35.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:35.584828+0000) 2022-01-31T21:18:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:35 smithi171 conmon[41853]: debug 2022-01-31T21:18:35.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:35.816224+0000) 2022-01-31T21:18:36.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:36 smithi171 conmon[46715]: debug 2022-01-31T21:18:36.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:36.111092+0000) 2022-01-31T21:18:36.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:36 smithi167 conmon[60316]: debug 2022-01-31T21:18:36.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:36.336427+0000) 2022-01-31T21:18:36.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:36 smithi171 conmon[51620]: debug 2022-01-31T21:18:36.691+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:36.691694+0000) 2022-01-31T21:18:36.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:36 smithi167 conmon[54076]: debug 2022-01-31T21:18:36.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:36.584958+0000) 2022-01-31T21:18:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:36 smithi167 conmon[49112]: debug 2022-01-31T21:18:36.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:36.663626+0000) 2022-01-31T21:18:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:36 smithi171 conmon[41853]: debug 2022-01-31T21:18:36.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:36.816396+0000) 2022-01-31T21:18:37.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:37 smithi171 conmon[46715]: debug 2022-01-31T21:18:37.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:37.111319+0000) 2022-01-31T21:18:37.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:37 smithi167 conmon[60316]: debug 2022-01-31T21:18:37.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:37.336610+0000) 2022-01-31T21:18:37.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:37 smithi171 conmon[51620]: debug 2022-01-31T21:18:37.691+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:37.691892+0000) 2022-01-31T21:18:37.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:37 smithi167 conmon[54076]: debug 2022-01-31T21:18:37.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:37.585111+0000) 2022-01-31T21:18:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:37 smithi167 conmon[49112]: debug 2022-01-31T21:18:37.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:37.663765+0000) 2022-01-31T21:18:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:37 smithi171 conmon[41853]: debug 2022-01-31T21:18:37.815+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:37.816602+0000) 2022-01-31T21:18:38.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[51620]: debug 2022-01-31T21:18:38.255+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.256265+0000) 2022-01-31T21:18:38.486 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[35325]: debug 2022-01-31T21:18:38.266+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:38.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[41853]: debug 2022-01-31T21:18:38.255+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.255869+0000) 2022-01-31T21:18:38.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[46715]: debug 2022-01-31T21:18:38.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.111525+0000) 2022-01-31T21:18:38.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[46715]: debug 2022-01-31T21:18:38.254+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.254801+0000) 2022-01-31T21:18:38.583 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:38 smithi167 conmon[49112]: debug 2022-01-31T21:18:38.255+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.255859+0000) 2022-01-31T21:18:38.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:38 smithi167 conmon[54076]: debug 2022-01-31T21:18:38.254+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.255102+0000) 2022-01-31T21:18:38.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:38 smithi167 conmon[60316]: debug 2022-01-31T21:18:38.255+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.255697+0000) 2022-01-31T21:18:38.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:38 smithi167 conmon[60316]: debug 2022-01-31T21:18:38.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.336805+0000) 2022-01-31T21:18:38.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[51620]: debug 2022-01-31T21:18:38.691+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.692066+0000) 2022-01-31T21:18:38.775 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[35325]: debug 2022-01-31T21:18:38.657+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:18:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:38 smithi167 conmon[49112]: debug 2022-01-31T21:18:38.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.663963+0000) 2022-01-31T21:18:38.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:38 smithi167 conmon[54076]: debug 2022-01-31T21:18:38.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.585312+0000) 2022-01-31T21:18:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:38 smithi171 conmon[41853]: debug 2022-01-31T21:18:38.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:38.816832+0000) 2022-01-31T21:18:39.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:39 smithi171 conmon[46715]: debug 2022-01-31T21:18:39.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:39.111762+0000) 2022-01-31T21:18:39.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:39 smithi167 conmon[60316]: debug 2022-01-31T21:18:39.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:39.336955+0000) 2022-01-31T21:18:39.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:39 smithi171 conmon[51620]: debug 2022-01-31T21:18:39.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:39.692198+0000) 2022-01-31T21:18:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:39 smithi167 conmon[49112]: debug 2022-01-31T21:18:39.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:39.664145+0000) 2022-01-31T21:18:39.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:39 smithi167 conmon[54076]: debug 2022-01-31T21:18:39.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:39.585452+0000) 2022-01-31T21:18:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:39 smithi171 conmon[41853]: debug 2022-01-31T21:18:39.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:39.816965+0000) 2022-01-31T21:18:40.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:40 smithi171 conmon[46715]: debug 2022-01-31T21:18:40.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:40.111874+0000) 2022-01-31T21:18:40.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:40 smithi167 conmon[60316]: debug 2022-01-31T21:18:40.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:40.337157+0000) 2022-01-31T21:18:40.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:40 smithi171 conmon[51620]: debug 2022-01-31T21:18:40.691+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:40.692372+0000) 2022-01-31T21:18:40.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:40 smithi167 conmon[54076]: debug 2022-01-31T21:18:40.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:40.585654+0000) 2022-01-31T21:18:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:40 smithi167 conmon[49112]: debug 2022-01-31T21:18:40.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:40.664271+0000) 2022-01-31T21:18:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:40 smithi171 conmon[41853]: debug 2022-01-31T21:18:40.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:40.817173+0000) 2022-01-31T21:18:41.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:41 smithi171 conmon[46715]: debug 2022-01-31T21:18:41.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:41.112026+0000) 2022-01-31T21:18:41.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:41 smithi167 conmon[60316]: debug 2022-01-31T21:18:41.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:41.337312+0000) 2022-01-31T21:18:41.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:41 smithi171 conmon[51620]: debug 2022-01-31T21:18:41.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:41.692555+0000) 2022-01-31T21:18:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:41 smithi167 conmon[49112]: debug 2022-01-31T21:18:41.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:41.664394+0000) 2022-01-31T21:18:41.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:41 smithi167 conmon[54076]: debug 2022-01-31T21:18:41.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:41.585822+0000) 2022-01-31T21:18:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:41 smithi171 conmon[41853]: debug 2022-01-31T21:18:41.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:41.817328+0000) 2022-01-31T21:18:42.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:42 smithi171 conmon[46715]: debug 2022-01-31T21:18:42.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:42.112215+0000) 2022-01-31T21:18:42.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:42 smithi167 conmon[60316]: debug 2022-01-31T21:18:42.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:42.337467+0000) 2022-01-31T21:18:42.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:42 smithi171 conmon[51620]: debug 2022-01-31T21:18:42.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:42.692788+0000) 2022-01-31T21:18:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:42 smithi167 conmon[49112]: debug 2022-01-31T21:18:42.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:42.664521+0000) 2022-01-31T21:18:42.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:42 smithi167 conmon[54076]: debug 2022-01-31T21:18:42.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:42.585985+0000) 2022-01-31T21:18:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:42 smithi171 conmon[41853]: debug 2022-01-31T21:18:42.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:42.817550+0000) 2022-01-31T21:18:43.487 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[35325]: debug 2022-01-31T21:18:43.280+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119707 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:43.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[41853]: debug 2022-01-31T21:18:43.269+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.270306+0000) 2022-01-31T21:18:43.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[46715]: debug 2022-01-31T21:18:43.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.112394+0000) 2022-01-31T21:18:43.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[46715]: debug 2022-01-31T21:18:43.268+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.269338+0000) 2022-01-31T21:18:43.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[51620]: debug 2022-01-31T21:18:43.269+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.269836+0000) 2022-01-31T21:18:43.584 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:43 smithi167 conmon[49112]: debug 2022-01-31T21:18:43.269+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.269777+0000) 2022-01-31T21:18:43.585 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:43 smithi167 conmon[54076]: debug 2022-01-31T21:18:43.269+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.270280+0000) 2022-01-31T21:18:43.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:43 smithi167 conmon[60316]: debug 2022-01-31T21:18:43.269+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.269858+0000) 2022-01-31T21:18:43.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:43 smithi167 conmon[60316]: debug 2022-01-31T21:18:43.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.337587+0000) 2022-01-31T21:18:43.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[51620]: debug 2022-01-31T21:18:43.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.692973+0000) 2022-01-31T21:18:44.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:43 smithi167 conmon[49112]: debug 2022-01-31T21:18:43.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.664707+0000) 2022-01-31T21:18:44.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:43 smithi167 conmon[54076]: debug 2022-01-31T21:18:43.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.586124+0000) 2022-01-31T21:18:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:43 smithi171 conmon[41853]: debug 2022-01-31T21:18:43.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:43.817724+0000) 2022-01-31T21:18:44.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:44 smithi171 conmon[46715]: debug 2022-01-31T21:18:44.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:44.112524+0000) 2022-01-31T21:18:44.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:44 smithi167 conmon[60316]: debug 2022-01-31T21:18:44.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:44.337682+0000) 2022-01-31T21:18:44.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:44 smithi171 conmon[51620]: debug 2022-01-31T21:18:44.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:44.693133+0000) 2022-01-31T21:18:44.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:44 smithi167 conmon[54076]: debug 2022-01-31T21:18:44.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:44.586243+0000) 2022-01-31T21:18:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:44 smithi167 conmon[49112]: debug 2022-01-31T21:18:44.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:44.664874+0000) 2022-01-31T21:18:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:44 smithi171 conmon[41853]: debug 2022-01-31T21:18:44.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:44.817812+0000) 2022-01-31T21:18:45.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:45 smithi171 conmon[46715]: debug 2022-01-31T21:18:45.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:45.112702+0000) 2022-01-31T21:18:45.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:45 smithi167 conmon[60316]: debug 2022-01-31T21:18:45.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:45.337846+0000) 2022-01-31T21:18:45.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:45 smithi171 conmon[51620]: debug 2022-01-31T21:18:45.693+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:45.693333+0000) 2022-01-31T21:18:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:45 smithi167 conmon[49112]: debug 2022-01-31T21:18:45.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:45.665041+0000) 2022-01-31T21:18:45.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:45 smithi167 conmon[54076]: debug 2022-01-31T21:18:45.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:45.586351+0000) 2022-01-31T21:18:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:45 smithi171 conmon[41853]: debug 2022-01-31T21:18:45.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:45.817982+0000) 2022-01-31T21:18:46.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:46 smithi171 conmon[46715]: debug 2022-01-31T21:18:46.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:46.112862+0000) 2022-01-31T21:18:46.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:46 smithi167 conmon[60316]: debug 2022-01-31T21:18:46.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:46.338048+0000) 2022-01-31T21:18:46.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:46 smithi171 conmon[51620]: debug 2022-01-31T21:18:46.691+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:46.693508+0000) 2022-01-31T21:18:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:46 smithi167 conmon[49112]: debug 2022-01-31T21:18:46.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:46.665244+0000) 2022-01-31T21:18:46.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:46 smithi167 conmon[54076]: debug 2022-01-31T21:18:46.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:46.586546+0000) 2022-01-31T21:18:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:46 smithi171 conmon[41853]: debug 2022-01-31T21:18:46.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:46.818127+0000) 2022-01-31T21:18:47.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:47 smithi171 conmon[46715]: debug 2022-01-31T21:18:47.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:47.113008+0000) 2022-01-31T21:18:47.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:47 smithi167 conmon[60316]: debug 2022-01-31T21:18:47.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:47.338172+0000) 2022-01-31T21:18:47.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:47 smithi171 conmon[51620]: debug 2022-01-31T21:18:47.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:47.693671+0000) 2022-01-31T21:18:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:47 smithi167 conmon[49112]: debug 2022-01-31T21:18:47.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:47.665475+0000) 2022-01-31T21:18:47.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:47 smithi167 conmon[54076]: debug 2022-01-31T21:18:47.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:47.586743+0000) 2022-01-31T21:18:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:47 smithi171 conmon[41853]: debug 2022-01-31T21:18:47.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:47.818266+0000) 2022-01-31T21:18:48.488 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[35325]: debug 2022-01-31T21:18:48.293+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119820 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:48.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[41853]: debug 2022-01-31T21:18:48.282+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.284135+0000) 2022-01-31T21:18:48.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[51620]: debug 2022-01-31T21:18:48.281+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.283360+0000) 2022-01-31T21:18:48.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[46715]: debug 2022-01-31T21:18:48.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.113155+0000) 2022-01-31T21:18:48.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[46715]: debug 2022-01-31T21:18:48.281+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.283707+0000) 2022-01-31T21:18:48.585 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:48 smithi167 conmon[49112]: debug 2022-01-31T21:18:48.282+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.283321+0000) 2022-01-31T21:18:48.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:48 smithi167 conmon[54076]: debug 2022-01-31T21:18:48.283+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.284296+0000) 2022-01-31T21:18:48.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:48 smithi167 conmon[60316]: debug 2022-01-31T21:18:48.283+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.283519+0000) 2022-01-31T21:18:48.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:48 smithi167 conmon[60316]: debug 2022-01-31T21:18:48.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.338380+0000) 2022-01-31T21:18:48.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[51620]: debug 2022-01-31T21:18:48.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.693852+0000) 2022-01-31T21:18:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:48 smithi167 conmon[49112]: debug 2022-01-31T21:18:48.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.665708+0000) 2022-01-31T21:18:48.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:48 smithi167 conmon[54076]: debug 2022-01-31T21:18:48.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.586963+0000) 2022-01-31T21:18:49.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:48 smithi171 conmon[41853]: debug 2022-01-31T21:18:48.816+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:48.818422+0000) 2022-01-31T21:18:49.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:49 smithi171 conmon[46715]: debug 2022-01-31T21:18:49.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:49.113379+0000) 2022-01-31T21:18:49.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:49 smithi167 conmon[60316]: debug 2022-01-31T21:18:49.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:49.338574+0000) 2022-01-31T21:18:49.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:49 smithi171 conmon[51620]: debug 2022-01-31T21:18:49.693+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:49.694006+0000) 2022-01-31T21:18:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:49 smithi167 conmon[49112]: debug 2022-01-31T21:18:49.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:49.665877+0000) 2022-01-31T21:18:49.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:49 smithi167 conmon[54076]: debug 2022-01-31T21:18:49.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:49.587148+0000) 2022-01-31T21:18:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:49 smithi171 conmon[41853]: debug 2022-01-31T21:18:49.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:49.818577+0000) 2022-01-31T21:18:50.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:50 smithi171 conmon[46715]: debug 2022-01-31T21:18:50.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:50.113545+0000) 2022-01-31T21:18:50.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:50 smithi167 conmon[60316]: debug 2022-01-31T21:18:50.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:50.338778+0000) 2022-01-31T21:18:50.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:50 smithi171 conmon[51620]: debug 2022-01-31T21:18:50.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:50.694150+0000) 2022-01-31T21:18:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:50 smithi167 conmon[49112]: debug 2022-01-31T21:18:50.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:50.666099+0000) 2022-01-31T21:18:50.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:50 smithi167 conmon[54076]: debug 2022-01-31T21:18:50.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:50.587361+0000) 2022-01-31T21:18:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:50 smithi171 conmon[41853]: debug 2022-01-31T21:18:50.817+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:50.818785+0000) 2022-01-31T21:18:51.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:51 smithi171 conmon[46715]: debug 2022-01-31T21:18:51.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:51.113722+0000) 2022-01-31T21:18:51.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:51 smithi167 conmon[60316]: debug 2022-01-31T21:18:51.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:51.338975+0000) 2022-01-31T21:18:51.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:51 smithi171 conmon[51620]: debug 2022-01-31T21:18:51.693+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:51.694354+0000) 2022-01-31T21:18:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:51 smithi167 conmon[49112]: debug 2022-01-31T21:18:51.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:51.666295+0000) 2022-01-31T21:18:51.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:51 smithi167 conmon[54076]: debug 2022-01-31T21:18:51.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:51.587547+0000) 2022-01-31T21:18:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:51 smithi171 conmon[41853]: debug 2022-01-31T21:18:51.818+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:51.818964+0000) 2022-01-31T21:18:52.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:52 smithi171 conmon[46715]: debug 2022-01-31T21:18:52.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:52.113888+0000) 2022-01-31T21:18:52.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:52 smithi167 conmon[60316]: debug 2022-01-31T21:18:52.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:52.339206+0000) 2022-01-31T21:18:52.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:52 smithi171 conmon[51620]: debug 2022-01-31T21:18:52.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:52.694532+0000) 2022-01-31T21:18:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:52 smithi167 conmon[49112]: debug 2022-01-31T21:18:52.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:52.666458+0000) 2022-01-31T21:18:52.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:52 smithi167 conmon[54076]: debug 2022-01-31T21:18:52.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:52.587712+0000) 2022-01-31T21:18:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:52 smithi171 conmon[41853]: debug 2022-01-31T21:18:52.818+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:52.819138+0000) 2022-01-31T21:18:53.489 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[35325]: debug 2022-01-31T21:18:53.324+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 119928 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:53.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[41853]: debug 2022-01-31T21:18:53.296+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.298244+0000) 2022-01-31T21:18:53.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[51620]: debug 2022-01-31T21:18:53.296+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.298701+0000) 2022-01-31T21:18:53.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[46715]: debug 2022-01-31T21:18:53.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.114043+0000) 2022-01-31T21:18:53.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[46715]: debug 2022-01-31T21:18:53.296+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.298700+0000) 2022-01-31T21:18:53.586 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:53 smithi167 conmon[49112]: debug 2022-01-31T21:18:53.298+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.299060+0000) 2022-01-31T21:18:53.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:53 smithi167 conmon[54076]: debug 2022-01-31T21:18:53.299+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.299543+0000) 2022-01-31T21:18:53.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:53 smithi167 conmon[60316]: debug 2022-01-31T21:18:53.298+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.298929+0000) 2022-01-31T21:18:53.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:53 smithi167 conmon[60316]: debug 2022-01-31T21:18:53.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.339374+0000) 2022-01-31T21:18:53.777 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[35325]: debug 2022-01-31T21:18:53.657+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:18:53.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[51620]: debug 2022-01-31T21:18:53.692+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.694710+0000) 2022-01-31T21:18:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:53 smithi167 conmon[49112]: debug 2022-01-31T21:18:53.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.666595+0000) 2022-01-31T21:18:53.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:53 smithi167 conmon[54076]: debug 2022-01-31T21:18:53.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.587864+0000) 2022-01-31T21:18:54.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:53 smithi171 conmon[41853]: debug 2022-01-31T21:18:53.818+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:53.819327+0000) 2022-01-31T21:18:54.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:54 smithi171 conmon[46715]: debug 2022-01-31T21:18:54.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:54.114226+0000) 2022-01-31T21:18:54.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:54 smithi167 conmon[60316]: debug 2022-01-31T21:18:54.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:54.339540+0000) 2022-01-31T21:18:54.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:54 smithi171 conmon[51620]: debug 2022-01-31T21:18:54.694+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:54.694900+0000) 2022-01-31T21:18:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:54 smithi167 conmon[49112]: debug 2022-01-31T21:18:54.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:54.666730+0000) 2022-01-31T21:18:54.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:54 smithi167 conmon[54076]: debug 2022-01-31T21:18:54.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:54.587970+0000) 2022-01-31T21:18:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:54 smithi171 conmon[41853]: debug 2022-01-31T21:18:54.818+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:54.819479+0000) 2022-01-31T21:18:55.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:55 smithi171 conmon[46715]: debug 2022-01-31T21:18:55.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:55.114368+0000) 2022-01-31T21:18:55.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:55 smithi167 conmon[60316]: debug 2022-01-31T21:18:55.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:55.339682+0000) 2022-01-31T21:18:55.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:55 smithi171 conmon[51620]: debug 2022-01-31T21:18:55.694+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:55.695048+0000) 2022-01-31T21:18:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:55 smithi167 conmon[49112]: debug 2022-01-31T21:18:55.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:55.666943+0000) 2022-01-31T21:18:55.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:55 smithi167 conmon[54076]: debug 2022-01-31T21:18:55.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:55.588176+0000) 2022-01-31T21:18:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:55 smithi171 conmon[41853]: debug 2022-01-31T21:18:55.818+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:55.819667+0000) 2022-01-31T21:18:56.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:56 smithi171 conmon[46715]: debug 2022-01-31T21:18:56.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:56.114570+0000) 2022-01-31T21:18:56.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:56 smithi167 conmon[60316]: debug 2022-01-31T21:18:56.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:56.339870+0000) 2022-01-31T21:18:56.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:56 smithi171 conmon[51620]: debug 2022-01-31T21:18:56.694+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:56.695201+0000) 2022-01-31T21:18:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:56 smithi167 conmon[49112]: debug 2022-01-31T21:18:56.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:56.667155+0000) 2022-01-31T21:18:56.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:56 smithi167 conmon[54076]: debug 2022-01-31T21:18:56.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:56.588362+0000) 2022-01-31T21:18:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:56 smithi171 conmon[41853]: debug 2022-01-31T21:18:56.819+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:56.819831+0000) 2022-01-31T21:18:57.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:57 smithi171 conmon[46715]: debug 2022-01-31T21:18:57.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:57.114746+0000) 2022-01-31T21:18:57.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:57 smithi167 conmon[60316]: debug 2022-01-31T21:18:57.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:57.340058+0000) 2022-01-31T21:18:57.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:57 smithi171 conmon[51620]: debug 2022-01-31T21:18:57.694+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:57.695380+0000) 2022-01-31T21:18:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:57 smithi167 conmon[49112]: debug 2022-01-31T21:18:57.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:57.667348+0000) 2022-01-31T21:18:57.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:57 smithi167 conmon[54076]: debug 2022-01-31T21:18:57.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:57.588541+0000) 2022-01-31T21:18:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:57 smithi171 conmon[41853]: debug 2022-01-31T21:18:57.819+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:57.820019+0000) 2022-01-31T21:18:58.490 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[35325]: debug 2022-01-31T21:18:58.339+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120040 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:18:58.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[41853]: debug 2022-01-31T21:18:58.328+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.329025+0000) 2022-01-31T21:18:58.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[46715]: debug 2022-01-31T21:18:58.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.114903+0000) 2022-01-31T21:18:58.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[46715]: debug 2022-01-31T21:18:58.327+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.328263+0000) 2022-01-31T21:18:58.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[51620]: debug 2022-01-31T21:18:58.328+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.329430+0000) 2022-01-31T21:18:58.587 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:58 smithi167 conmon[49112]: debug 2022-01-31T21:18:58.328+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.329276+0000) 2022-01-31T21:18:58.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:58 smithi167 conmon[54076]: debug 2022-01-31T21:18:58.328+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.328991+0000) 2022-01-31T21:18:58.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:58 smithi167 conmon[60316]: debug 2022-01-31T21:18:58.328+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.329171+0000) 2022-01-31T21:18:58.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:58 smithi167 conmon[60316]: debug 2022-01-31T21:18:58.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.340231+0000) 2022-01-31T21:18:58.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[51620]: debug 2022-01-31T21:18:58.694+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.695580+0000) 2022-01-31T21:18:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:58 smithi167 conmon[49112]: debug 2022-01-31T21:18:58.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.667544+0000) 2022-01-31T21:18:58.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:58 smithi167 conmon[54076]: debug 2022-01-31T21:18:58.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.588694+0000) 2022-01-31T21:18:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:58 smithi171 conmon[41853]: debug 2022-01-31T21:18:58.819+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:58.820172+0000) 2022-01-31T21:18:59.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:18:59 smithi171 conmon[46715]: debug 2022-01-31T21:18:59.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:59.115068+0000) 2022-01-31T21:18:59.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:18:59 smithi167 conmon[60316]: debug 2022-01-31T21:18:59.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:59.340380+0000) 2022-01-31T21:18:59.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:18:59 smithi171 conmon[51620]: debug 2022-01-31T21:18:59.694+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:59.695737+0000) 2022-01-31T21:18:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:18:59 smithi167 conmon[49112]: debug 2022-01-31T21:18:59.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:59.667646+0000) 2022-01-31T21:18:59.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:18:59 smithi167 conmon[54076]: debug 2022-01-31T21:18:59.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:59.588860+0000) 2022-01-31T21:19:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:18:59 smithi171 conmon[41853]: debug 2022-01-31T21:18:59.819+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:18:59.820328+0000) 2022-01-31T21:19:00.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:00 smithi171 conmon[46715]: debug 2022-01-31T21:19:00.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:00.115190+0000) 2022-01-31T21:19:00.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:00 smithi167 conmon[60316]: debug 2022-01-31T21:19:00.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:00.340593+0000) 2022-01-31T21:19:00.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:00 smithi171 conmon[51620]: debug 2022-01-31T21:19:00.695+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:00.695885+0000) 2022-01-31T21:19:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:00 smithi167 conmon[49112]: debug 2022-01-31T21:19:00.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:00.667807+0000) 2022-01-31T21:19:00.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:00 smithi167 conmon[54076]: debug 2022-01-31T21:19:00.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:00.589048+0000) 2022-01-31T21:19:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:00 smithi171 conmon[41853]: debug 2022-01-31T21:19:00.819+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:00.820494+0000) 2022-01-31T21:19:01.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:01 smithi171 conmon[46715]: debug 2022-01-31T21:19:01.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:01.115369+0000) 2022-01-31T21:19:01.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:01 smithi167 conmon[60316]: debug 2022-01-31T21:19:01.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:01.340714+0000) 2022-01-31T21:19:01.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:01 smithi171 conmon[51620]: debug 2022-01-31T21:19:01.695+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:01.696062+0000) 2022-01-31T21:19:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:01 smithi167 conmon[49112]: debug 2022-01-31T21:19:01.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:01.667974+0000) 2022-01-31T21:19:01.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:01 smithi167 conmon[54076]: debug 2022-01-31T21:19:01.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:01.589223+0000) 2022-01-31T21:19:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:01 smithi171 conmon[41853]: debug 2022-01-31T21:19:01.819+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:01.820654+0000) 2022-01-31T21:19:02.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:02 smithi171 conmon[46715]: debug 2022-01-31T21:19:02.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:02.115578+0000) 2022-01-31T21:19:02.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:02 smithi167 conmon[60316]: debug 2022-01-31T21:19:02.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:02.340934+0000) 2022-01-31T21:19:02.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:02 smithi171 conmon[51620]: debug 2022-01-31T21:19:02.695+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:02.696211+0000) 2022-01-31T21:19:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:02 smithi167 conmon[49112]: debug 2022-01-31T21:19:02.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:02.668176+0000) 2022-01-31T21:19:02.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:02 smithi167 conmon[54076]: debug 2022-01-31T21:19:02.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:02.589451+0000) 2022-01-31T21:19:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:02 smithi171 conmon[41853]: debug 2022-01-31T21:19:02.820+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:02.820814+0000) 2022-01-31T21:19:03.491 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[35325]: debug 2022-01-31T21:19:03.354+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120148 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:03.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[41853]: debug 2022-01-31T21:19:03.341+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.342640+0000) 2022-01-31T21:19:03.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[51620]: debug 2022-01-31T21:19:03.342+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.343322+0000) 2022-01-31T21:19:03.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[46715]: debug 2022-01-31T21:19:03.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.115805+0000) 2022-01-31T21:19:03.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[46715]: debug 2022-01-31T21:19:03.341+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.342615+0000) 2022-01-31T21:19:03.588 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:03 smithi167 conmon[49112]: debug 2022-01-31T21:19:03.341+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.342998+0000) 2022-01-31T21:19:03.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:03 smithi167 conmon[54076]: debug 2022-01-31T21:19:03.341+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.342536+0000) 2022-01-31T21:19:03.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:03 smithi167 conmon[60316]: debug 2022-01-31T21:19:03.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.341068+0000) 2022-01-31T21:19:03.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:03 smithi167 conmon[60316]: debug 2022-01-31T21:19:03.342+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.344133+0000) 2022-01-31T21:19:03.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[51620]: debug 2022-01-31T21:19:03.695+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.696411+0000) 2022-01-31T21:19:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:03 smithi167 conmon[49112]: debug 2022-01-31T21:19:03.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.668366+0000) 2022-01-31T21:19:03.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:03 smithi167 conmon[54076]: debug 2022-01-31T21:19:03.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.589647+0000) 2022-01-31T21:19:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:03 smithi171 conmon[41853]: debug 2022-01-31T21:19:03.820+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:03.820974+0000) 2022-01-31T21:19:04.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:04 smithi171 conmon[46715]: debug 2022-01-31T21:19:04.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:04.115941+0000) 2022-01-31T21:19:04.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:04 smithi167 conmon[60316]: debug 2022-01-31T21:19:04.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:04.341258+0000) 2022-01-31T21:19:04.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:04 smithi171 conmon[51620]: debug 2022-01-31T21:19:04.695+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:04.696546+0000) 2022-01-31T21:19:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:04 smithi167 conmon[49112]: debug 2022-01-31T21:19:04.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:04.668575+0000) 2022-01-31T21:19:04.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:04 smithi167 conmon[54076]: debug 2022-01-31T21:19:04.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:04.589767+0000) 2022-01-31T21:19:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:04 smithi171 conmon[41853]: debug 2022-01-31T21:19:04.820+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:04.821115+0000) 2022-01-31T21:19:05.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:05 smithi171 conmon[46715]: debug 2022-01-31T21:19:05.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:05.116084+0000) 2022-01-31T21:19:05.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:05 smithi167 conmon[60316]: debug 2022-01-31T21:19:05.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:05.341421+0000) 2022-01-31T21:19:05.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:05 smithi171 conmon[51620]: debug 2022-01-31T21:19:05.695+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:05.696699+0000) 2022-01-31T21:19:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:05 smithi167 conmon[49112]: debug 2022-01-31T21:19:05.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:05.668756+0000) 2022-01-31T21:19:05.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:05 smithi167 conmon[54076]: debug 2022-01-31T21:19:05.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:05.589939+0000) 2022-01-31T21:19:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:05 smithi171 conmon[41853]: debug 2022-01-31T21:19:05.820+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:05.821299+0000) 2022-01-31T21:19:06.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:06 smithi171 conmon[46715]: debug 2022-01-31T21:19:06.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:06.116243+0000) 2022-01-31T21:19:06.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:06 smithi167 conmon[60316]: debug 2022-01-31T21:19:06.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:06.341613+0000) 2022-01-31T21:19:06.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:06 smithi171 conmon[51620]: debug 2022-01-31T21:19:06.696+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:06.696867+0000) 2022-01-31T21:19:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:06 smithi167 conmon[49112]: debug 2022-01-31T21:19:06.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:06.668942+0000) 2022-01-31T21:19:06.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:06 smithi167 conmon[54076]: debug 2022-01-31T21:19:06.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:06.590091+0000) 2022-01-31T21:19:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:06 smithi171 conmon[41853]: debug 2022-01-31T21:19:06.820+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:06.821444+0000) 2022-01-31T21:19:07.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:07 smithi171 conmon[46715]: debug 2022-01-31T21:19:07.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:07.116414+0000) 2022-01-31T21:19:07.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:07 smithi167 conmon[60316]: debug 2022-01-31T21:19:07.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:07.341747+0000) 2022-01-31T21:19:07.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:07 smithi171 conmon[51620]: debug 2022-01-31T21:19:07.696+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:07.696994+0000) 2022-01-31T21:19:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:07 smithi167 conmon[49112]: debug 2022-01-31T21:19:07.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:07.669115+0000) 2022-01-31T21:19:07.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:07 smithi167 conmon[54076]: debug 2022-01-31T21:19:07.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:07.590268+0000) 2022-01-31T21:19:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:07 smithi171 conmon[41853]: debug 2022-01-31T21:19:07.820+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:07.821586+0000) 2022-01-31T21:19:08.369 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[41853]: debug 2022-01-31T21:19:08.356+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.357612+0000) 2022-01-31T21:19:08.369 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[46715]: debug 2022-01-31T21:19:08.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.116617+0000) 2022-01-31T21:19:08.370 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[46715]: debug 2022-01-31T21:19:08.357+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.357742+0000) 2022-01-31T21:19:08.370 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[51620]: debug 2022-01-31T21:19:08.356+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.357151+0000) 2022-01-31T21:19:08.589 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:08 smithi167 conmon[49112]: debug 2022-01-31T21:19:08.356+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.357537+0000) 2022-01-31T21:19:08.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:08 smithi167 conmon[54076]: debug 2022-01-31T21:19:08.356+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.358123+0000) 2022-01-31T21:19:08.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:08 smithi167 conmon[60316]: debug 2022-01-31T21:19:08.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.341871+0000) 2022-01-31T21:19:08.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:08 smithi167 conmon[60316]: debug 2022-01-31T21:19:08.357+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.358875+0000) 2022-01-31T21:19:08.658 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[35325]: debug 2022-01-31T21:19:08.369+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120260 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:08 smithi167 conmon[49112]: debug 2022-01-31T21:19:08.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.669323+0000) 2022-01-31T21:19:08.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:08 smithi167 conmon[54076]: debug 2022-01-31T21:19:08.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.590469+0000) 2022-01-31T21:19:08.947 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[35325]: debug 2022-01-31T21:19:08.659+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:19:08.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[51620]: debug 2022-01-31T21:19:08.696+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.697163+0000) 2022-01-31T21:19:08.948 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:08 smithi171 conmon[41853]: debug 2022-01-31T21:19:08.821+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:08.821783+0000) 2022-01-31T21:19:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:09 smithi171 conmon[46715]: debug 2022-01-31T21:19:09.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:09.116815+0000) 2022-01-31T21:19:09.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:09 smithi167 conmon[60316]: debug 2022-01-31T21:19:09.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:09.342067+0000) 2022-01-31T21:19:09.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:09 smithi171 conmon[51620]: debug 2022-01-31T21:19:09.696+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:09.697372+0000) 2022-01-31T21:19:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:09 smithi167 conmon[49112]: debug 2022-01-31T21:19:09.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:09.669494+0000) 2022-01-31T21:19:09.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:09 smithi167 conmon[54076]: debug 2022-01-31T21:19:09.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:09.590610+0000) 2022-01-31T21:19:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:09 smithi171 conmon[41853]: debug 2022-01-31T21:19:09.821+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:09.821931+0000) 2022-01-31T21:19:10.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:10 smithi171 conmon[46715]: debug 2022-01-31T21:19:10.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:10.116960+0000) 2022-01-31T21:19:10.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:10 smithi167 conmon[60316]: debug 2022-01-31T21:19:10.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:10.342268+0000) 2022-01-31T21:19:10.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:10 smithi171 conmon[51620]: debug 2022-01-31T21:19:10.696+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:10.697598+0000) 2022-01-31T21:19:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:10 smithi167 conmon[49112]: debug 2022-01-31T21:19:10.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:10.669729+0000) 2022-01-31T21:19:10.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:10 smithi167 conmon[54076]: debug 2022-01-31T21:19:10.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:10.590793+0000) 2022-01-31T21:19:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:10 smithi171 conmon[41853]: debug 2022-01-31T21:19:10.821+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:10.822150+0000) 2022-01-31T21:19:11.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:11 smithi171 conmon[46715]: debug 2022-01-31T21:19:11.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:11.117108+0000) 2022-01-31T21:19:11.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:11 smithi167 conmon[60316]: debug 2022-01-31T21:19:11.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:11.342472+0000) 2022-01-31T21:19:11.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:11 smithi171 conmon[51620]: debug 2022-01-31T21:19:11.697+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:11.697803+0000) 2022-01-31T21:19:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:11 smithi167 conmon[49112]: debug 2022-01-31T21:19:11.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:11.669919+0000) 2022-01-31T21:19:11.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:11 smithi167 conmon[54076]: debug 2022-01-31T21:19:11.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:11.590972+0000) 2022-01-31T21:19:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:11 smithi171 conmon[41853]: debug 2022-01-31T21:19:11.821+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:11.822351+0000) 2022-01-31T21:19:12.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:12 smithi171 conmon[46715]: debug 2022-01-31T21:19:12.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:12.117273+0000) 2022-01-31T21:19:12.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:12 smithi167 conmon[60316]: debug 2022-01-31T21:19:12.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:12.342654+0000) 2022-01-31T21:19:12.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:12 smithi171 conmon[51620]: debug 2022-01-31T21:19:12.697+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:12.697984+0000) 2022-01-31T21:19:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:12 smithi167 conmon[49112]: debug 2022-01-31T21:19:12.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:12.670118+0000) 2022-01-31T21:19:12.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:12 smithi167 conmon[54076]: debug 2022-01-31T21:19:12.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:12.591134+0000) 2022-01-31T21:19:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:12 smithi171 conmon[41853]: debug 2022-01-31T21:19:12.821+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:12.822499+0000) 2022-01-31T21:19:13.371 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[46715]: debug 2022-01-31T21:19:13.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.117446+0000) 2022-01-31T21:19:13.372 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[41853]: debug 2022-01-31T21:19:13.372+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.372826+0000) 2022-01-31T21:19:13.590 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:13 smithi167 conmon[49112]: debug 2022-01-31T21:19:13.371+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.373026+0000) 2022-01-31T21:19:13.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:13 smithi167 conmon[54076]: debug 2022-01-31T21:19:13.372+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.374454+0000) 2022-01-31T21:19:13.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:13 smithi167 conmon[60316]: debug 2022-01-31T21:19:13.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.342817+0000) 2022-01-31T21:19:13.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:13 smithi167 conmon[60316]: debug 2022-01-31T21:19:13.371+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.372806+0000) 2022-01-31T21:19:13.696 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[46715]: debug 2022-01-31T21:19:13.372+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.373713+0000) 2022-01-31T21:19:13.697 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[35325]: debug 2022-01-31T21:19:13.384+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120368 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:13.698 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[51620]: debug 2022-01-31T21:19:13.372+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.373039+0000) 2022-01-31T21:19:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:13 smithi167 conmon[49112]: debug 2022-01-31T21:19:13.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.670295+0000) 2022-01-31T21:19:13.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:13 smithi167 conmon[54076]: debug 2022-01-31T21:19:13.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.591335+0000) 2022-01-31T21:19:13.948 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[41853]: debug 2022-01-31T21:19:13.822+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.822679+0000) 2022-01-31T21:19:13.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:13 smithi171 conmon[51620]: debug 2022-01-31T21:19:13.697+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:13.698160+0000) 2022-01-31T21:19:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:14 smithi171 conmon[46715]: debug 2022-01-31T21:19:14.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:14.117635+0000) 2022-01-31T21:19:14.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:14 smithi167 conmon[60316]: debug 2022-01-31T21:19:14.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:14.342986+0000) 2022-01-31T21:19:14.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:14 smithi171 conmon[51620]: debug 2022-01-31T21:19:14.697+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:14.698307+0000) 2022-01-31T21:19:15.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:14 smithi167 conmon[49112]: debug 2022-01-31T21:19:14.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:14.670464+0000) 2022-01-31T21:19:15.144 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:14 smithi171 conmon[41853]: debug 2022-01-31T21:19:14.822+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:14.822831+0000) 2022-01-31T21:19:15.144 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:14 smithi167 conmon[54076]: debug 2022-01-31T21:19:14.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:14.591479+0000) 2022-01-31T21:19:15.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:15 smithi171 conmon[46715]: debug 2022-01-31T21:19:15.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:15.117830+0000) 2022-01-31T21:19:15.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:15 smithi167 conmon[60316]: debug 2022-01-31T21:19:15.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:15.343142+0000) 2022-01-31T21:19:15.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:15 smithi171 conmon[51620]: debug 2022-01-31T21:19:15.698+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:15.698461+0000) 2022-01-31T21:19:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:15 smithi167 conmon[49112]: debug 2022-01-31T21:19:15.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:15.670653+0000) 2022-01-31T21:19:15.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:15 smithi167 conmon[54076]: debug 2022-01-31T21:19:15.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:15.591703+0000) 2022-01-31T21:19:16.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:15 smithi171 conmon[41853]: debug 2022-01-31T21:19:15.822+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:15.823020+0000) 2022-01-31T21:19:16.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:16 smithi171 conmon[46715]: debug 2022-01-31T21:19:16.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:16.118008+0000) 2022-01-31T21:19:16.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:16 smithi167 conmon[60316]: debug 2022-01-31T21:19:16.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:16.343351+0000) 2022-01-31T21:19:16.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:16 smithi171 conmon[51620]: debug 2022-01-31T21:19:16.698+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:16.698644+0000) 2022-01-31T21:19:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:16 smithi167 conmon[49112]: debug 2022-01-31T21:19:16.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:16.670845+0000) 2022-01-31T21:19:16.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:16 smithi167 conmon[54076]: debug 2022-01-31T21:19:16.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:16.591865+0000) 2022-01-31T21:19:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:16 smithi171 conmon[41853]: debug 2022-01-31T21:19:16.822+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:16.823205+0000) 2022-01-31T21:19:17.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:17 smithi171 conmon[46715]: debug 2022-01-31T21:19:17.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:17.118111+0000) 2022-01-31T21:19:17.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:17 smithi167 conmon[60316]: debug 2022-01-31T21:19:17.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:17.343522+0000) 2022-01-31T21:19:17.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:17 smithi171 conmon[51620]: debug 2022-01-31T21:19:17.698+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:17.698825+0000) 2022-01-31T21:19:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:17 smithi167 conmon[49112]: debug 2022-01-31T21:19:17.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:17.671028+0000) 2022-01-31T21:19:17.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:17 smithi167 conmon[54076]: debug 2022-01-31T21:19:17.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:17.592048+0000) 2022-01-31T21:19:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:17 smithi171 conmon[41853]: debug 2022-01-31T21:19:17.823+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:17.823393+0000) 2022-01-31T21:19:18.385 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[46715]: debug 2022-01-31T21:19:18.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.118312+0000) 2022-01-31T21:19:18.590 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:18 smithi167 conmon[49112]: debug 2022-01-31T21:19:18.386+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.387627+0000) 2022-01-31T21:19:18.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:18 smithi167 conmon[54076]: debug 2022-01-31T21:19:18.387+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.388787+0000) 2022-01-31T21:19:18.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:18 smithi167 conmon[60316]: debug 2022-01-31T21:19:18.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.343673+0000) 2022-01-31T21:19:18.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:18 smithi167 conmon[60316]: debug 2022-01-31T21:19:18.386+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.387885+0000) 2022-01-31T21:19:18.697 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[35325]: debug 2022-01-31T21:19:18.400+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120480 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:18.698 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[41853]: debug 2022-01-31T21:19:18.387+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.388179+0000) 2022-01-31T21:19:18.698 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[46715]: debug 2022-01-31T21:19:18.387+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.387787+0000) 2022-01-31T21:19:18.699 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[51620]: debug 2022-01-31T21:19:18.387+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.388614+0000) 2022-01-31T21:19:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:18 smithi167 conmon[49112]: debug 2022-01-31T21:19:18.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.671182+0000) 2022-01-31T21:19:18.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:18 smithi167 conmon[54076]: debug 2022-01-31T21:19:18.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.592224+0000) 2022-01-31T21:19:18.949 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[41853]: debug 2022-01-31T21:19:18.822+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.823590+0000) 2022-01-31T21:19:18.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:18 smithi171 conmon[51620]: debug 2022-01-31T21:19:18.698+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:18.699022+0000) 2022-01-31T21:19:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:19 smithi171 conmon[46715]: debug 2022-01-31T21:19:19.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:19.118506+0000) 2022-01-31T21:19:19.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:19 smithi167 conmon[60316]: debug 2022-01-31T21:19:19.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:19.343805+0000) 2022-01-31T21:19:19.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:19 smithi171 conmon[51620]: debug 2022-01-31T21:19:19.698+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:19.699172+0000) 2022-01-31T21:19:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:19 smithi167 conmon[49112]: debug 2022-01-31T21:19:19.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:19.671331+0000) 2022-01-31T21:19:19.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:19 smithi167 conmon[54076]: debug 2022-01-31T21:19:19.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:19.592387+0000) 2022-01-31T21:19:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:19 smithi171 conmon[41853]: debug 2022-01-31T21:19:19.823+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:19.823775+0000) 2022-01-31T21:19:20.384 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:20 smithi171 conmon[46715]: debug 2022-01-31T21:19:20.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:20.118655+0000) 2022-01-31T21:19:20.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:20 smithi167 conmon[60316]: debug 2022-01-31T21:19:20.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:20.344000+0000) 2022-01-31T21:19:20.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:20 smithi171 conmon[51620]: debug 2022-01-31T21:19:20.699+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:20.699333+0000) 2022-01-31T21:19:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:20 smithi167 conmon[49112]: debug 2022-01-31T21:19:20.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:20.671542+0000) 2022-01-31T21:19:20.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:20 smithi167 conmon[54076]: debug 2022-01-31T21:19:20.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:20.592589+0000) 2022-01-31T21:19:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:20 smithi171 conmon[41853]: debug 2022-01-31T21:19:20.823+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:20.823982+0000) 2022-01-31T21:19:21.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:21 smithi171 conmon[46715]: debug 2022-01-31T21:19:21.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:21.118861+0000) 2022-01-31T21:19:21.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:21 smithi167 conmon[60316]: debug 2022-01-31T21:19:21.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:21.344151+0000) 2022-01-31T21:19:21.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:21 smithi171 conmon[51620]: debug 2022-01-31T21:19:21.699+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:21.699506+0000) 2022-01-31T21:19:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:21 smithi167 conmon[49112]: debug 2022-01-31T21:19:21.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:21.671739+0000) 2022-01-31T21:19:21.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:21 smithi167 conmon[54076]: debug 2022-01-31T21:19:21.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:21.592711+0000) 2022-01-31T21:19:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:21 smithi171 conmon[41853]: debug 2022-01-31T21:19:21.823+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:21.824160+0000) 2022-01-31T21:19:22.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:22 smithi171 conmon[46715]: debug 2022-01-31T21:19:22.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:22.119022+0000) 2022-01-31T21:19:22.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:22 smithi167 conmon[60316]: debug 2022-01-31T21:19:22.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:22.344325+0000) 2022-01-31T21:19:22.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:22 smithi171 conmon[51620]: debug 2022-01-31T21:19:22.699+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:22.699693+0000) 2022-01-31T21:19:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:22 smithi167 conmon[49112]: debug 2022-01-31T21:19:22.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:22.671888+0000) 2022-01-31T21:19:22.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:22 smithi167 conmon[54076]: debug 2022-01-31T21:19:22.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:22.592888+0000) 2022-01-31T21:19:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:22 smithi171 conmon[41853]: debug 2022-01-31T21:19:22.824+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:22.824334+0000) 2022-01-31T21:19:23.401 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[46715]: debug 2022-01-31T21:19:23.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.119216+0000) 2022-01-31T21:19:23.591 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:23 smithi167 conmon[49112]: debug 2022-01-31T21:19:23.402+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.403610+0000) 2022-01-31T21:19:23.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:23 smithi167 conmon[54076]: debug 2022-01-31T21:19:23.403+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.404880+0000) 2022-01-31T21:19:23.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:23 smithi167 conmon[60316]: debug 2022-01-31T21:19:23.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.344543+0000) 2022-01-31T21:19:23.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:23 smithi167 conmon[60316]: debug 2022-01-31T21:19:23.402+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.404126+0000) 2022-01-31T21:19:23.659 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[41853]: debug 2022-01-31T21:19:23.403+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.404000+0000) 2022-01-31T21:19:23.660 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[46715]: debug 2022-01-31T21:19:23.403+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.403800+0000) 2022-01-31T21:19:23.660 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[51620]: debug 2022-01-31T21:19:23.404+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.404893+0000) 2022-01-31T21:19:23.661 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[35325]: debug 2022-01-31T21:19:23.415+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:23 smithi167 conmon[49112]: debug 2022-01-31T21:19:23.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.672071+0000) 2022-01-31T21:19:23.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:23 smithi167 conmon[54076]: debug 2022-01-31T21:19:23.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.593051+0000) 2022-01-31T21:19:23.950 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[35325]: debug 2022-01-31T21:19:23.660+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:19:23.950 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[41853]: debug 2022-01-31T21:19:23.823+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.824502+0000) 2022-01-31T21:19:23.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:23 smithi171 conmon[51620]: debug 2022-01-31T21:19:23.699+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:23.699852+0000) 2022-01-31T21:19:24.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:24 smithi171 conmon[46715]: debug 2022-01-31T21:19:24.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:24.119420+0000) 2022-01-31T21:19:24.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:24 smithi167 conmon[60316]: debug 2022-01-31T21:19:24.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:24.344716+0000) 2022-01-31T21:19:24.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:24 smithi171 conmon[51620]: debug 2022-01-31T21:19:24.699+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:24.700002+0000) 2022-01-31T21:19:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:24 smithi167 conmon[49112]: debug 2022-01-31T21:19:24.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:24.672241+0000) 2022-01-31T21:19:24.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:24 smithi167 conmon[54076]: debug 2022-01-31T21:19:24.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:24.593210+0000) 2022-01-31T21:19:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:24 smithi171 conmon[41853]: debug 2022-01-31T21:19:24.824+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:24.824653+0000) 2022-01-31T21:19:25.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:25 smithi171 conmon[46715]: debug 2022-01-31T21:19:25.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:25.119605+0000) 2022-01-31T21:19:25.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:25 smithi167 conmon[60316]: debug 2022-01-31T21:19:25.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:25.344923+0000) 2022-01-31T21:19:25.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:25 smithi171 conmon[51620]: debug 2022-01-31T21:19:25.699+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:25.700189+0000) 2022-01-31T21:19:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:25 smithi167 conmon[49112]: debug 2022-01-31T21:19:25.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:25.672423+0000) 2022-01-31T21:19:25.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:25 smithi167 conmon[54076]: debug 2022-01-31T21:19:25.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:25.593424+0000) 2022-01-31T21:19:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:25 smithi171 conmon[41853]: debug 2022-01-31T21:19:25.824+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:25.824841+0000) 2022-01-31T21:19:26.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:26 smithi171 conmon[46715]: debug 2022-01-31T21:19:26.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:26.119801+0000) 2022-01-31T21:19:26.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:26 smithi167 conmon[60316]: debug 2022-01-31T21:19:26.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:26.345595+0000) 2022-01-31T21:19:26.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:26 smithi171 conmon[51620]: debug 2022-01-31T21:19:26.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:26.700371+0000) 2022-01-31T21:19:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:26 smithi167 conmon[49112]: debug 2022-01-31T21:19:26.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:26.672566+0000) 2022-01-31T21:19:26.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:26 smithi167 conmon[54076]: debug 2022-01-31T21:19:26.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:26.593582+0000) 2022-01-31T21:19:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:26 smithi171 conmon[41853]: debug 2022-01-31T21:19:26.824+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:26.825052+0000) 2022-01-31T21:19:27.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:27 smithi171 conmon[46715]: debug 2022-01-31T21:19:27.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:27.119935+0000) 2022-01-31T21:19:27.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:27 smithi167 conmon[60316]: debug 2022-01-31T21:19:27.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:27.345748+0000) 2022-01-31T21:19:27.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:27 smithi171 conmon[51620]: debug 2022-01-31T21:19:27.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:27.700562+0000) 2022-01-31T21:19:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:27 smithi167 conmon[49112]: debug 2022-01-31T21:19:27.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:27.672744+0000) 2022-01-31T21:19:27.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:27 smithi167 conmon[54076]: debug 2022-01-31T21:19:27.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:27.593711+0000) 2022-01-31T21:19:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:27 smithi171 conmon[41853]: debug 2022-01-31T21:19:27.824+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:27.825231+0000) 2022-01-31T21:19:28.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[46715]: debug 2022-01-31T21:19:28.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.120061+0000) 2022-01-31T21:19:28.592 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:28 smithi167 conmon[49112]: debug 2022-01-31T21:19:28.418+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.418847+0000) 2022-01-31T21:19:28.593 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:28 smithi167 conmon[54076]: debug 2022-01-31T21:19:28.419+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.419879+0000) 2022-01-31T21:19:28.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:28 smithi167 conmon[60316]: debug 2022-01-31T21:19:28.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.345896+0000) 2022-01-31T21:19:28.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:28 smithi167 conmon[60316]: debug 2022-01-31T21:19:28.419+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.419747+0000) 2022-01-31T21:19:28.699 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[35325]: debug 2022-01-31T21:19:28.430+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120711 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:28.700 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[41853]: debug 2022-01-31T21:19:28.417+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.418345+0000) 2022-01-31T21:19:28.700 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[46715]: debug 2022-01-31T21:19:28.418+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.418951+0000) 2022-01-31T21:19:28.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[51620]: debug 2022-01-31T21:19:28.418+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.419711+0000) 2022-01-31T21:19:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:28 smithi167 conmon[49112]: debug 2022-01-31T21:19:28.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.672903+0000) 2022-01-31T21:19:28.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:28 smithi167 conmon[54076]: debug 2022-01-31T21:19:28.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.593879+0000) 2022-01-31T21:19:28.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[51620]: debug 2022-01-31T21:19:28.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.700772+0000) 2022-01-31T21:19:28.951 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:28 smithi171 conmon[41853]: debug 2022-01-31T21:19:28.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:28.825385+0000) 2022-01-31T21:19:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:29 smithi171 conmon[46715]: debug 2022-01-31T21:19:29.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:29.120225+0000) 2022-01-31T21:19:29.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:29 smithi167 conmon[60316]: debug 2022-01-31T21:19:29.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:29.346100+0000) 2022-01-31T21:19:29.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:29 smithi171 conmon[51620]: debug 2022-01-31T21:19:29.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:29.700931+0000) 2022-01-31T21:19:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:29 smithi167 conmon[49112]: debug 2022-01-31T21:19:29.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:29.673061+0000) 2022-01-31T21:19:29.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:29 smithi167 conmon[54076]: debug 2022-01-31T21:19:29.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:29.594036+0000) 2022-01-31T21:19:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:29 smithi171 conmon[41853]: debug 2022-01-31T21:19:29.824+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:29.825557+0000) 2022-01-31T21:19:30.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:30 smithi171 conmon[46715]: debug 2022-01-31T21:19:30.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:30.120377+0000) 2022-01-31T21:19:30.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:30 smithi167 conmon[60316]: debug 2022-01-31T21:19:30.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:30.346256+0000) 2022-01-31T21:19:30.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:30 smithi171 conmon[51620]: debug 2022-01-31T21:19:30.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:30.701117+0000) 2022-01-31T21:19:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:30 smithi167 conmon[49112]: debug 2022-01-31T21:19:30.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:30.673268+0000) 2022-01-31T21:19:30.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:30 smithi167 conmon[54076]: debug 2022-01-31T21:19:30.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:30.594254+0000) 2022-01-31T21:19:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:30 smithi171 conmon[41853]: debug 2022-01-31T21:19:30.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:30.825718+0000) 2022-01-31T21:19:31.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:31 smithi171 conmon[46715]: debug 2022-01-31T21:19:31.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:31.120548+0000) 2022-01-31T21:19:31.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:31 smithi167 conmon[60316]: debug 2022-01-31T21:19:31.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:31.346490+0000) 2022-01-31T21:19:31.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:31 smithi171 conmon[51620]: debug 2022-01-31T21:19:31.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:31.701324+0000) 2022-01-31T21:19:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:31 smithi167 conmon[49112]: debug 2022-01-31T21:19:31.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:31.673526+0000) 2022-01-31T21:19:31.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:31 smithi167 conmon[54076]: debug 2022-01-31T21:19:31.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:31.594460+0000) 2022-01-31T21:19:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:31 smithi171 conmon[41853]: debug 2022-01-31T21:19:31.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:31.825904+0000) 2022-01-31T21:19:32.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:32 smithi171 conmon[46715]: debug 2022-01-31T21:19:32.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:32.120734+0000) 2022-01-31T21:19:32.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:32 smithi167 conmon[60316]: debug 2022-01-31T21:19:32.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:32.346679+0000) 2022-01-31T21:19:32.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:32 smithi171 conmon[51620]: debug 2022-01-31T21:19:32.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:32.701523+0000) 2022-01-31T21:19:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:32 smithi167 conmon[49112]: debug 2022-01-31T21:19:32.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:32.673702+0000) 2022-01-31T21:19:32.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:32 smithi167 conmon[54076]: debug 2022-01-31T21:19:32.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:32.594645+0000) 2022-01-31T21:19:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:32 smithi171 conmon[41853]: debug 2022-01-31T21:19:32.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:32.826115+0000) 2022-01-31T21:19:33.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[46715]: debug 2022-01-31T21:19:33.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.120920+0000) 2022-01-31T21:19:33.593 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:33 smithi167 conmon[49112]: debug 2022-01-31T21:19:33.433+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.434198+0000) 2022-01-31T21:19:33.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:33 smithi167 conmon[54076]: debug 2022-01-31T21:19:33.433+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.434442+0000) 2022-01-31T21:19:33.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:33 smithi167 conmon[60316]: debug 2022-01-31T21:19:33.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.346845+0000) 2022-01-31T21:19:33.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:33 smithi167 conmon[60316]: debug 2022-01-31T21:19:33.433+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.434068+0000) 2022-01-31T21:19:33.700 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[35325]: debug 2022-01-31T21:19:33.444+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120819 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:33.701 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[41853]: debug 2022-01-31T21:19:33.432+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.434297+0000) 2022-01-31T21:19:33.701 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[46715]: debug 2022-01-31T21:19:33.431+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.433249+0000) 2022-01-31T21:19:33.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[51620]: debug 2022-01-31T21:19:33.702 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[51620]: 2022-01-31T21:19:33.432+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.434568+0000) 2022-01-31T21:19:33.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:33 smithi167 conmon[54076]: debug 2022-01-31T21:19:33.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.594777+0000) 2022-01-31T21:19:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:33 smithi167 conmon[49112]: debug 2022-01-31T21:19:33.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.673856+0000) 2022-01-31T21:19:33.951 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[41853]: debug 2022-01-31T21:19:33.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.826289+0000) 2022-01-31T21:19:33.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:33 smithi171 conmon[51620]: debug 2022-01-31T21:19:33.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:33.701705+0000) 2022-01-31T21:19:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:34 smithi171 conmon[46715]: debug 2022-01-31T21:19:34.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:34.121071+0000) 2022-01-31T21:19:34.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:34 smithi167 conmon[60316]: debug 2022-01-31T21:19:34.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:34.347008+0000) 2022-01-31T21:19:34.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:34 smithi171 conmon[51620]: debug 2022-01-31T21:19:34.700+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:34.701853+0000) 2022-01-31T21:19:34.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:34 smithi167 conmon[54076]: debug 2022-01-31T21:19:34.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:34.594955+0000) 2022-01-31T21:19:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:34 smithi167 conmon[49112]: debug 2022-01-31T21:19:34.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:34.674043+0000) 2022-01-31T21:19:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:34 smithi171 conmon[41853]: debug 2022-01-31T21:19:34.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:34.826424+0000) 2022-01-31T21:19:35.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:35 smithi171 conmon[46715]: debug 2022-01-31T21:19:35.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:35.121236+0000) 2022-01-31T21:19:35.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:35 smithi167 conmon[60316]: debug 2022-01-31T21:19:35.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:35.347184+0000) 2022-01-31T21:19:35.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:35 smithi171 conmon[51620]: debug 2022-01-31T21:19:35.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:35.702036+0000) 2022-01-31T21:19:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:35 smithi167 conmon[49112]: debug 2022-01-31T21:19:35.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:35.674229+0000) 2022-01-31T21:19:35.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:35 smithi167 conmon[54076]: debug 2022-01-31T21:19:35.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:35.595053+0000) 2022-01-31T21:19:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:35 smithi171 conmon[41853]: debug 2022-01-31T21:19:35.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:35.826604+0000) 2022-01-31T21:19:36.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:36 smithi171 conmon[46715]: debug 2022-01-31T21:19:36.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:36.121430+0000) 2022-01-31T21:19:36.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:36 smithi167 conmon[60316]: debug 2022-01-31T21:19:36.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:36.347373+0000) 2022-01-31T21:19:36.769 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:36 smithi171 conmon[51620]: debug 2022-01-31T21:19:36.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:36.702220+0000) 2022-01-31T21:19:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:36 smithi167 conmon[49112]: debug 2022-01-31T21:19:36.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:36.674424+0000) 2022-01-31T21:19:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:36 smithi167 conmon[49112]: 2022-01-31T21:19:36.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:36 smithi167 conmon[54076]: debug 2022-01-31T21:19:36.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:36.595245+0000) 2022-01-31T21:19:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:36 smithi171 conmon[41853]: debug 2022-01-31T21:19:36.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:36.826793+0000) 2022-01-31T21:19:37.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:37 smithi171 conmon[46715]: debug 2022-01-31T21:19:37.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:37.121584+0000) 2022-01-31T21:19:37.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:37 smithi167 conmon[60316]: debug 2022-01-31T21:19:37.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:37.347531+0000) 2022-01-31T21:19:37.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:37 smithi171 conmon[51620]: debug 2022-01-31T21:19:37.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:37.702366+0000) 2022-01-31T21:19:37.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:37 smithi167 conmon[54076]: debug 2022-01-31T21:19:37.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:37.595386+0000) 2022-01-31T21:19:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:37 smithi167 conmon[49112]: debug 2022-01-31T21:19:37.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:37.674524+0000) 2022-01-31T21:19:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:37 smithi171 conmon[41853]: debug 2022-01-31T21:19:37.826+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:37.826984+0000) 2022-01-31T21:19:38.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[46715]: debug 2022-01-31T21:19:38.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.121736+0000) 2022-01-31T21:19:38.594 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:38 smithi167 conmon[49112]: debug 2022-01-31T21:19:38.449+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.450354+0000) 2022-01-31T21:19:38.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:38 smithi167 conmon[54076]: debug 2022-01-31T21:19:38.449+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.450184+0000) 2022-01-31T21:19:38.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:38 smithi167 conmon[60316]: debug 2022-01-31T21:19:38.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.347660+0000) 2022-01-31T21:19:38.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:38 smithi167 conmon[60316]: debug 2022-01-31T21:19:38.449+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.449495+0000) 2022-01-31T21:19:38.701 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[46715]: debug 2022-01-31T21:19:38.446+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.448564+0000) 2022-01-31T21:19:38.701 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[51620]: debug 2022-01-31T21:19:38.448+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.450371+0000) 2022-01-31T21:19:38.702 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[35325]: debug 2022-01-31T21:19:38.459+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 120931 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:38.702 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[35325]: debug 2022-01-31T21:19:38.660+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:19:38.703 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[41853]: debug 2022-01-31T21:19:38.447+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.448769+0000) 2022-01-31T21:19:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:38 smithi167 conmon[49112]: debug 2022-01-31T21:19:38.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.674655+0000) 2022-01-31T21:19:38.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:38 smithi167 conmon[54076]: debug 2022-01-31T21:19:38.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.595567+0000) 2022-01-31T21:19:38.952 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[41853]: debug 2022-01-31T21:19:38.825+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.827137+0000) 2022-01-31T21:19:38.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:38 smithi171 conmon[51620]: debug 2022-01-31T21:19:38.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:38.702562+0000) 2022-01-31T21:19:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:39 smithi171 conmon[46715]: debug 2022-01-31T21:19:39.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:39.121868+0000) 2022-01-31T21:19:39.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:39 smithi167 conmon[60316]: debug 2022-01-31T21:19:39.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:39.347768+0000) 2022-01-31T21:19:39.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:39 smithi171 conmon[51620]: debug 2022-01-31T21:19:39.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:39.702717+0000) 2022-01-31T21:19:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:39 smithi167 conmon[49112]: debug 2022-01-31T21:19:39.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:39.674766+0000) 2022-01-31T21:19:39.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:39 smithi167 conmon[54076]: debug 2022-01-31T21:19:39.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:39.595707+0000) 2022-01-31T21:19:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:39 smithi171 conmon[41853]: debug 2022-01-31T21:19:39.826+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:39.827252+0000) 2022-01-31T21:19:40.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:40 smithi171 conmon[46715]: debug 2022-01-31T21:19:40.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:40.121999+0000) 2022-01-31T21:19:40.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:40 smithi167 conmon[60316]: debug 2022-01-31T21:19:40.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:40.347951+0000) 2022-01-31T21:19:40.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:40 smithi171 conmon[51620]: debug 2022-01-31T21:19:40.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:40.702891+0000) 2022-01-31T21:19:40.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:40 smithi167 conmon[54076]: debug 2022-01-31T21:19:40.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:40.595906+0000) 2022-01-31T21:19:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:40 smithi167 conmon[49112]: debug 2022-01-31T21:19:40.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:40.674937+0000) 2022-01-31T21:19:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:40 smithi171 conmon[41853]: debug 2022-01-31T21:19:40.826+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:40.827380+0000) 2022-01-31T21:19:41.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:41 smithi171 conmon[46715]: debug 2022-01-31T21:19:41.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:41.122165+0000) 2022-01-31T21:19:41.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:41 smithi167 conmon[60316]: debug 2022-01-31T21:19:41.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:41.348169+0000) 2022-01-31T21:19:41.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:41 smithi171 conmon[51620]: debug 2022-01-31T21:19:41.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:41.703070+0000) 2022-01-31T21:19:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:41 smithi167 conmon[49112]: debug 2022-01-31T21:19:41.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:41.675122+0000) 2022-01-31T21:19:41.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:41 smithi167 conmon[54076]: debug 2022-01-31T21:19:41.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:41.596124+0000) 2022-01-31T21:19:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:41 smithi171 conmon[41853]: debug 2022-01-31T21:19:41.826+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:41.827474+0000) 2022-01-31T21:19:42.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:42 smithi171 conmon[46715]: debug 2022-01-31T21:19:42.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:42.122347+0000) 2022-01-31T21:19:42.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:42 smithi167 conmon[60316]: debug 2022-01-31T21:19:42.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:42.348398+0000) 2022-01-31T21:19:42.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:42 smithi171 conmon[51620]: debug 2022-01-31T21:19:42.702+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:42.703274+0000) 2022-01-31T21:19:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:42 smithi167 conmon[49112]: debug 2022-01-31T21:19:42.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:42.675301+0000) 2022-01-31T21:19:42.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:42 smithi167 conmon[54076]: debug 2022-01-31T21:19:42.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:42.596334+0000) 2022-01-31T21:19:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:42 smithi171 conmon[41853]: debug 2022-01-31T21:19:42.826+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:42.827656+0000) 2022-01-31T21:19:43.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[46715]: debug 2022-01-31T21:19:43.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.122497+0000) 2022-01-31T21:19:43.595 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:43 smithi167 conmon[49112]: debug 2022-01-31T21:19:43.464+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.464942+0000) 2022-01-31T21:19:43.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:43 smithi167 conmon[54076]: debug 2022-01-31T21:19:43.464+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.465048+0000) 2022-01-31T21:19:43.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:43 smithi167 conmon[60316]: debug 2022-01-31T21:19:43.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.348607+0000) 2022-01-31T21:19:43.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:43 smithi167 conmon[60316]: debug 2022-01-31T21:19:43.462+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.463238+0000) 2022-01-31T21:19:43.786 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[35325]: debug 2022-01-31T21:19:43.474+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121041 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:43.786 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[46715]: debug 2022-01-31T21:19:43.461+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.462948+0000) 2022-01-31T21:19:43.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[41853]: debug 2022-01-31T21:19:43.461+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.463532+0000) 2022-01-31T21:19:43.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[51620]: debug 2022-01-31T21:19:43.462+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.464462+0000) 2022-01-31T21:19:43.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[51620]: debug 2022-01-31T21:19:43.701+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.703502+0000) 2022-01-31T21:19:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:43 smithi167 conmon[49112]: debug 2022-01-31T21:19:43.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.675461+0000) 2022-01-31T21:19:43.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:43 smithi167 conmon[54076]: debug 2022-01-31T21:19:43.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.596536+0000) 2022-01-31T21:19:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:43 smithi171 conmon[41853]: debug 2022-01-31T21:19:43.827+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:43.827838+0000) 2022-01-31T21:19:44.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:44 smithi171 conmon[46715]: debug 2022-01-31T21:19:44.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:44.122668+0000) 2022-01-31T21:19:44.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:44 smithi167 conmon[60316]: debug 2022-01-31T21:19:44.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:44.348801+0000) 2022-01-31T21:19:44.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:44 smithi171 conmon[51620]: debug 2022-01-31T21:19:44.702+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:44.703672+0000) 2022-01-31T21:19:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:44 smithi167 conmon[49112]: debug 2022-01-31T21:19:44.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:44.675644+0000) 2022-01-31T21:19:44.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:44 smithi167 conmon[54076]: debug 2022-01-31T21:19:44.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:44.596672+0000) 2022-01-31T21:19:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:44 smithi171 conmon[41853]: debug 2022-01-31T21:19:44.827+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:44.827962+0000) 2022-01-31T21:19:45.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:45 smithi171 conmon[46715]: debug 2022-01-31T21:19:45.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:45.122812+0000) 2022-01-31T21:19:45.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:45 smithi167 conmon[60316]: debug 2022-01-31T21:19:45.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:45.348925+0000) 2022-01-31T21:19:45.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:45 smithi171 conmon[51620]: debug 2022-01-31T21:19:45.703+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:45.703850+0000) 2022-01-31T21:19:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:45 smithi167 conmon[49112]: debug 2022-01-31T21:19:45.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:45.675845+0000) 2022-01-31T21:19:45.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:45 smithi167 conmon[54076]: debug 2022-01-31T21:19:45.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:45.596774+0000) 2022-01-31T21:19:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:45 smithi171 conmon[41853]: debug 2022-01-31T21:19:45.827+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:45.828137+0000) 2022-01-31T21:19:46.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:46 smithi171 conmon[46715]: debug 2022-01-31T21:19:46.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:46.122981+0000) 2022-01-31T21:19:46.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:46 smithi167 conmon[60316]: debug 2022-01-31T21:19:46.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:46.349086+0000) 2022-01-31T21:19:46.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:46 smithi171 conmon[51620]: debug 2022-01-31T21:19:46.703+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:46.703999+0000) 2022-01-31T21:19:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:46 smithi167 conmon[49112]: debug 2022-01-31T21:19:46.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:46.676033+0000) 2022-01-31T21:19:46.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:46 smithi167 conmon[54076]: debug 2022-01-31T21:19:46.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:46.596934+0000) 2022-01-31T21:19:47.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:46 smithi171 conmon[41853]: debug 2022-01-31T21:19:46.827+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:46.828347+0000) 2022-01-31T21:19:47.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:47 smithi171 conmon[46715]: debug 2022-01-31T21:19:47.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:47.123139+0000) 2022-01-31T21:19:47.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:47 smithi167 conmon[60316]: debug 2022-01-31T21:19:47.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:47.349303+0000) 2022-01-31T21:19:47.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:47 smithi171 conmon[51620]: debug 2022-01-31T21:19:47.703+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:47.704153+0000) 2022-01-31T21:19:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:47 smithi167 conmon[49112]: debug 2022-01-31T21:19:47.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:47.676229+0000) 2022-01-31T21:19:47.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:47 smithi167 conmon[54076]: debug 2022-01-31T21:19:47.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:47.597124+0000) 2022-01-31T21:19:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:47 smithi171 conmon[41853]: debug 2022-01-31T21:19:47.827+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:47.828559+0000) 2022-01-31T21:19:48.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[46715]: debug 2022-01-31T21:19:48.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.123302+0000) 2022-01-31T21:19:48.596 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:48 smithi167 conmon[49112]: debug 2022-01-31T21:19:48.477+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.478607+0000) 2022-01-31T21:19:48.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:48 smithi167 conmon[54076]: debug 2022-01-31T21:19:48.478+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.479644+0000) 2022-01-31T21:19:48.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:48 smithi167 conmon[60316]: debug 2022-01-31T21:19:48.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.349529+0000) 2022-01-31T21:19:48.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:48 smithi167 conmon[60316]: debug 2022-01-31T21:19:48.478+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.479785+0000) 2022-01-31T21:19:48.787 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[35325]: debug 2022-01-31T21:19:48.490+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121152 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:48.787 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[41853]: debug 2022-01-31T21:19:48.477+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.478204+0000) 2022-01-31T21:19:48.788 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[46715]: debug 2022-01-31T21:19:48.477+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.478363+0000) 2022-01-31T21:19:48.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[51620]: debug 2022-01-31T21:19:48.479+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.479968+0000) 2022-01-31T21:19:48.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[51620]: debug 2022-01-31T21:19:48.703+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.704353+0000) 2022-01-31T21:19:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:48 smithi167 conmon[49112]: debug 2022-01-31T21:19:48.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.676378+0000) 2022-01-31T21:19:48.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:48 smithi167 conmon[54076]: debug 2022-01-31T21:19:48.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.597331+0000) 2022-01-31T21:19:49.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:48 smithi171 conmon[41853]: debug 2022-01-31T21:19:48.828+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:48.828794+0000) 2022-01-31T21:19:49.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:49 smithi171 conmon[46715]: debug 2022-01-31T21:19:49.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:49.123540+0000) 2022-01-31T21:19:49.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:49 smithi167 conmon[60316]: debug 2022-01-31T21:19:49.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:49.349686+0000) 2022-01-31T21:19:49.786 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:49 smithi171 conmon[51620]: debug 2022-01-31T21:19:49.703+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:49.704484+0000) 2022-01-31T21:19:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:49 smithi167 conmon[49112]: debug 2022-01-31T21:19:49.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:49.676581+0000) 2022-01-31T21:19:49.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:49 smithi167 conmon[54076]: debug 2022-01-31T21:19:49.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:49.597518+0000) 2022-01-31T21:19:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:49 smithi171 conmon[41853]: debug 2022-01-31T21:19:49.828+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:49.828900+0000) 2022-01-31T21:19:50.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:50 smithi171 conmon[46715]: debug 2022-01-31T21:19:50.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:50.123712+0000) 2022-01-31T21:19:50.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:50 smithi167 conmon[60316]: debug 2022-01-31T21:19:50.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:50.349893+0000) 2022-01-31T21:19:50.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:50 smithi171 conmon[51620]: debug 2022-01-31T21:19:50.703+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:50.704649+0000) 2022-01-31T21:19:50.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:50 smithi167 conmon[54076]: debug 2022-01-31T21:19:50.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:50.597675+0000) 2022-01-31T21:19:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:50 smithi167 conmon[49112]: debug 2022-01-31T21:19:50.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:50.676729+0000) 2022-01-31T21:19:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:50 smithi171 conmon[41853]: debug 2022-01-31T21:19:50.828+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:50.829101+0000) 2022-01-31T21:19:51.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:51 smithi171 conmon[46715]: debug 2022-01-31T21:19:51.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:51.123865+0000) 2022-01-31T21:19:51.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:51 smithi167 conmon[60316]: debug 2022-01-31T21:19:51.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:51.350077+0000) 2022-01-31T21:19:51.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:51 smithi171 conmon[51620]: debug 2022-01-31T21:19:51.704+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:51.704837+0000) 2022-01-31T21:19:51.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:51 smithi167 conmon[54076]: debug 2022-01-31T21:19:51.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:51.597870+0000) 2022-01-31T21:19:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:51 smithi167 conmon[49112]: debug 2022-01-31T21:19:51.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:51.676884+0000) 2022-01-31T21:19:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:51 smithi171 conmon[41853]: debug 2022-01-31T21:19:51.828+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:51.829293+0000) 2022-01-31T21:19:52.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:52 smithi171 conmon[46715]: debug 2022-01-31T21:19:52.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:52.124042+0000) 2022-01-31T21:19:52.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:52 smithi167 conmon[60316]: debug 2022-01-31T21:19:52.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:52.350283+0000) 2022-01-31T21:19:52.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:52 smithi171 conmon[51620]: debug 2022-01-31T21:19:52.704+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:52.704941+0000) 2022-01-31T21:19:52.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:52 smithi167 conmon[54076]: debug 2022-01-31T21:19:52.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:52.598058+0000) 2022-01-31T21:19:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:52 smithi167 conmon[49112]: debug 2022-01-31T21:19:52.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:52.677096+0000) 2022-01-31T21:19:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:52 smithi171 conmon[41853]: debug 2022-01-31T21:19:52.828+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:52.829511+0000) 2022-01-31T21:19:53.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[46715]: debug 2022-01-31T21:19:53.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.124232+0000) 2022-01-31T21:19:53.596 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:53 smithi167 conmon[49112]: debug 2022-01-31T21:19:53.491+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.494813+0000) 2022-01-31T21:19:53.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:53 smithi167 conmon[54076]: debug 2022-01-31T21:19:53.491+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.494400+0000) 2022-01-31T21:19:53.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:53 smithi167 conmon[60316]: debug 2022-01-31T21:19:53.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.350489+0000) 2022-01-31T21:19:53.598 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:53 smithi167 conmon[60316]: debug 2022-01-31T21:19:53.491+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.494708+0000) 2022-01-31T21:19:53.787 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[35325]: debug 2022-01-31T21:19:53.510+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121261 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:53.788 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[35325]: debug 2022-01-31T21:19:53.661+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:19:53.788 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[41853]: debug 2022-01-31T21:19:53.491+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.493198+0000) 2022-01-31T21:19:53.789 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[46715]: debug 2022-01-31T21:19:53.491+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.493798+0000) 2022-01-31T21:19:53.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[51620]: debug 2022-01-31T21:19:53.491+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.494845+0000) 2022-01-31T21:19:53.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[51620]: debug 2022-01-31T21:19:53.704+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.705117+0000) 2022-01-31T21:19:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:53 smithi167 conmon[49112]: debug 2022-01-31T21:19:53.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.677286+0000) 2022-01-31T21:19:53.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:53 smithi167 conmon[54076]: debug 2022-01-31T21:19:53.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.598191+0000) 2022-01-31T21:19:54.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:53 smithi171 conmon[41853]: debug 2022-01-31T21:19:53.828+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:53.829713+0000) 2022-01-31T21:19:54.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:54 smithi171 conmon[46715]: debug 2022-01-31T21:19:54.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:54.124391+0000) 2022-01-31T21:19:54.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:54 smithi167 conmon[60316]: debug 2022-01-31T21:19:54.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:54.350712+0000) 2022-01-31T21:19:54.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:54 smithi171 conmon[51620]: debug 2022-01-31T21:19:54.704+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:54.705288+0000) 2022-01-31T21:19:54.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:54 smithi167 conmon[54076]: debug 2022-01-31T21:19:54.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:54.598336+0000) 2022-01-31T21:19:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:54 smithi167 conmon[49112]: debug 2022-01-31T21:19:54.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:54.677469+0000) 2022-01-31T21:19:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:54 smithi171 conmon[41853]: debug 2022-01-31T21:19:54.829+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:54.829905+0000) 2022-01-31T21:19:55.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:55 smithi171 conmon[46715]: debug 2022-01-31T21:19:55.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:55.124534+0000) 2022-01-31T21:19:55.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:55 smithi167 conmon[60316]: debug 2022-01-31T21:19:55.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:55.350890+0000) 2022-01-31T21:19:55.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:55 smithi171 conmon[51620]: debug 2022-01-31T21:19:55.704+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:55.705471+0000) 2022-01-31T21:19:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:55 smithi167 conmon[49112]: debug 2022-01-31T21:19:55.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:55.677670+0000) 2022-01-31T21:19:55.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:55 smithi167 conmon[54076]: debug 2022-01-31T21:19:55.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:55.598547+0000) 2022-01-31T21:19:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:55 smithi171 conmon[41853]: debug 2022-01-31T21:19:55.829+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:55.830042+0000) 2022-01-31T21:19:56.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:56 smithi171 conmon[46715]: debug 2022-01-31T21:19:56.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:56.124695+0000) 2022-01-31T21:19:56.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:56 smithi167 conmon[60316]: debug 2022-01-31T21:19:56.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:56.351108+0000) 2022-01-31T21:19:56.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:56 smithi171 conmon[51620]: debug 2022-01-31T21:19:56.705+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:56.705650+0000) 2022-01-31T21:19:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:56 smithi167 conmon[49112]: debug 2022-01-31T21:19:56.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:56.677808+0000) 2022-01-31T21:19:56.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:56 smithi167 conmon[54076]: debug 2022-01-31T21:19:56.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:56.598707+0000) 2022-01-31T21:19:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:56 smithi171 conmon[41853]: debug 2022-01-31T21:19:56.829+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:56.830168+0000) 2022-01-31T21:19:57.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:57 smithi171 conmon[46715]: debug 2022-01-31T21:19:57.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:57.124846+0000) 2022-01-31T21:19:57.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:57 smithi167 conmon[60316]: debug 2022-01-31T21:19:57.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:57.351313+0000) 2022-01-31T21:19:57.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:57 smithi171 conmon[51620]: debug 2022-01-31T21:19:57.705+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:57.705833+0000) 2022-01-31T21:19:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:57 smithi167 conmon[49112]: debug 2022-01-31T21:19:57.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:57.678025+0000) 2022-01-31T21:19:57.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:57 smithi167 conmon[54076]: debug 2022-01-31T21:19:57.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:57.598893+0000) 2022-01-31T21:19:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:57 smithi171 conmon[41853]: debug 2022-01-31T21:19:57.829+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:57.830320+0000) 2022-01-31T21:19:58.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[46715]: debug 2022-01-31T21:19:58.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.125024+0000) 2022-01-31T21:19:58.598 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:58 smithi167 conmon[49112]: debug 2022-01-31T21:19:58.512+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.514148+0000) 2022-01-31T21:19:58.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:58 smithi167 conmon[54076]: debug 2022-01-31T21:19:58.512+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.514460+0000) 2022-01-31T21:19:58.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:58 smithi167 conmon[60316]: debug 2022-01-31T21:19:58.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.351541+0000) 2022-01-31T21:19:58.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:58 smithi167 conmon[60316]: debug 2022-01-31T21:19:58.512+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.514229+0000) 2022-01-31T21:19:58.788 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[35325]: debug 2022-01-31T21:19:58.525+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121371 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:19:58.789 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[46715]: debug 2022-01-31T21:19:58.513+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.513951+0000) 2022-01-31T21:19:58.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[41853]: debug 2022-01-31T21:19:58.513+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.514493+0000) 2022-01-31T21:19:58.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[51620]: debug 2022-01-31T21:19:58.514+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.515332+0000) 2022-01-31T21:19:58.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[51620]: debug 2022-01-31T21:19:58.705+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.706015+0000) 2022-01-31T21:19:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:58 smithi167 conmon[49112]: debug 2022-01-31T21:19:58.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.678250+0000) 2022-01-31T21:19:58.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:58 smithi167 conmon[54076]: debug 2022-01-31T21:19:58.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.599074+0000) 2022-01-31T21:19:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:58 smithi171 conmon[41853]: debug 2022-01-31T21:19:58.829+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:58.830479+0000) 2022-01-31T21:19:59.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:19:59 smithi171 conmon[46715]: debug 2022-01-31T21:19:59.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:59.125253+0000) 2022-01-31T21:19:59.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:19:59 smithi167 conmon[60316]: debug 2022-01-31T21:19:59.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:59.351726+0000) 2022-01-31T21:19:59.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:19:59 smithi171 conmon[51620]: debug 2022-01-31T21:19:59.705+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:59.706170+0000) 2022-01-31T21:19:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:19:59 smithi167 conmon[49112]: debug 2022-01-31T21:19:59.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:59.678417+0000) 2022-01-31T21:19:59.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:19:59 smithi167 conmon[54076]: debug 2022-01-31T21:19:59.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:59.599221+0000) 2022-01-31T21:20:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:19:59 smithi171 conmon[41853]: debug 2022-01-31T21:19:59.830+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:19:59.830633+0000) 2022-01-31T21:20:00.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:00 smithi171 conmon[46715]: debug 2022-01-31T21:20:00.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:00.125416+0000) 2022-01-31T21:20:00.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:00 smithi167 conmon[60316]: debug 2022-01-31T21:20:00.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:00.351916+0000) 2022-01-31T21:20:00.788 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:00 smithi171 conmon[51620]: debug 2022-01-31T21:20:00.705+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:00.706322+0000) 2022-01-31T21:20:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:00 smithi167 conmon[49112]: debug 2022-01-31T21:20:00.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:00.678631+0000) 2022-01-31T21:20:00.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:00 smithi167 conmon[54076]: debug 2022-01-31T21:20:00.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:00.599427+0000) 2022-01-31T21:20:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:00 smithi171 conmon[41853]: debug 2022-01-31T21:20:00.830+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:00.830840+0000) 2022-01-31T21:20:01.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:01 smithi171 conmon[46715]: debug 2022-01-31T21:20:01.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:01.125577+0000) 2022-01-31T21:20:01.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:01 smithi167 conmon[54076]: debug 2022-01-31T21:20:01.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:01.599613+0000) 2022-01-31T21:20:01.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:01 smithi167 conmon[60316]: debug 2022-01-31T21:20:01.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:01.352123+0000) 2022-01-31T21:20:01.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:01 smithi171 conmon[51620]: debug 2022-01-31T21:20:01.706+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:01.706514+0000) 2022-01-31T21:20:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:01 smithi167 conmon[49112]: debug 2022-01-31T21:20:01.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:01.678767+0000) 2022-01-31T21:20:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:01 smithi171 conmon[41853]: debug 2022-01-31T21:20:01.830+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:01.831017+0000) 2022-01-31T21:20:02.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:02 smithi171 conmon[46715]: debug 2022-01-31T21:20:02.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:02.125777+0000) 2022-01-31T21:20:02.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:02 smithi167 conmon[54076]: debug 2022-01-31T21:20:02.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:02.599825+0000) 2022-01-31T21:20:02.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:02 smithi167 conmon[60316]: debug 2022-01-31T21:20:02.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:02.352329+0000) 2022-01-31T21:20:02.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:02 smithi171 conmon[51620]: debug 2022-01-31T21:20:02.706+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:02.706714+0000) 2022-01-31T21:20:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:02 smithi167 conmon[49112]: debug 2022-01-31T21:20:02.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:02.678925+0000) 2022-01-31T21:20:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:02 smithi171 conmon[41853]: debug 2022-01-31T21:20:02.830+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:02.831235+0000) 2022-01-31T21:20:03.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[46715]: debug 2022-01-31T21:20:03.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.125990+0000) 2022-01-31T21:20:03.654 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:03 smithi167 conmon[49112]: debug 2022-01-31T21:20:03.527+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.528953+0000) 2022-01-31T21:20:03.654 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:03 smithi167 conmon[54076]: debug 2022-01-31T21:20:03.528+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.529612+0000) 2022-01-31T21:20:03.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:03 smithi167 conmon[54076]: debug 2022-01-31T21:20:03.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.600007+0000) 2022-01-31T21:20:03.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:03 smithi167 conmon[60316]: debug 2022-01-31T21:20:03.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.352489+0000) 2022-01-31T21:20:03.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:03 smithi167 conmon[60316]: debug 2022-01-31T21:20:03.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.529302+0000) 2022-01-31T21:20:03.789 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[35325]: debug 2022-01-31T21:20:03.540+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121481 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:03.789 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[46715]: debug 2022-01-31T21:20:03.528+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.528984+0000) 2022-01-31T21:20:03.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[41853]: debug 2022-01-31T21:20:03.529+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.529842+0000) 2022-01-31T21:20:03.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[51620]: debug 2022-01-31T21:20:03.528+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.529724+0000) 2022-01-31T21:20:03.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[51620]: debug 2022-01-31T21:20:03.706+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.706903+0000) 2022-01-31T21:20:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:03 smithi167 conmon[49112]: debug 2022-01-31T21:20:03.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.679148+0000) 2022-01-31T21:20:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:03 smithi171 conmon[41853]: debug 2022-01-31T21:20:03.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:03.831419+0000) 2022-01-31T21:20:04.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:04 smithi171 conmon[46715]: debug 2022-01-31T21:20:04.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:04.126174+0000) 2022-01-31T21:20:04.654 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:04 smithi167 conmon[54076]: debug 2022-01-31T21:20:04.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:04.600155+0000) 2022-01-31T21:20:04.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:04 smithi167 conmon[60316]: debug 2022-01-31T21:20:04.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:04.352654+0000) 2022-01-31T21:20:04.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:04 smithi171 conmon[51620]: debug 2022-01-31T21:20:04.706+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:04.707054+0000) 2022-01-31T21:20:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:04 smithi167 conmon[49112]: debug 2022-01-31T21:20:04.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:04.679352+0000) 2022-01-31T21:20:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:04 smithi171 conmon[41853]: debug 2022-01-31T21:20:04.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:04.831568+0000) 2022-01-31T21:20:05.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:05 smithi171 conmon[46715]: debug 2022-01-31T21:20:05.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:05.126350+0000) 2022-01-31T21:20:05.654 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:05 smithi167 conmon[54076]: debug 2022-01-31T21:20:05.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:05.600347+0000) 2022-01-31T21:20:05.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:05 smithi167 conmon[60316]: debug 2022-01-31T21:20:05.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:05.352812+0000) 2022-01-31T21:20:05.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:05 smithi171 conmon[51620]: debug 2022-01-31T21:20:05.706+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:05.707259+0000) 2022-01-31T21:20:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:05 smithi167 conmon[49112]: debug 2022-01-31T21:20:05.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:05.679556+0000) 2022-01-31T21:20:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:05 smithi171 conmon[41853]: debug 2022-01-31T21:20:05.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:05.831765+0000) 2022-01-31T21:20:06.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:06 smithi171 conmon[46715]: debug 2022-01-31T21:20:06.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:06.126556+0000) 2022-01-31T21:20:06.654 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:06 smithi167 conmon[54076]: debug 2022-01-31T21:20:06.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:06.600566+0000) 2022-01-31T21:20:06.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:06 smithi167 conmon[60316]: debug 2022-01-31T21:20:06.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:06.352952+0000) 2022-01-31T21:20:06.789 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:06 smithi171 conmon[51620]: debug 2022-01-31T21:20:06.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:06.707421+0000) 2022-01-31T21:20:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:06 smithi167 conmon[49112]: debug 2022-01-31T21:20:06.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:06.679757+0000) 2022-01-31T21:20:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:06 smithi171 conmon[41853]: debug 2022-01-31T21:20:06.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:06.831965+0000) 2022-01-31T21:20:07.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:07 smithi171 conmon[46715]: debug 2022-01-31T21:20:07.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:07.126773+0000) 2022-01-31T21:20:07.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:07 smithi167 conmon[54076]: debug 2022-01-31T21:20:07.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:07.600760+0000) 2022-01-31T21:20:07.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:07 smithi167 conmon[60316]: debug 2022-01-31T21:20:07.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:07.353109+0000) 2022-01-31T21:20:07.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:07 smithi171 conmon[51620]: debug 2022-01-31T21:20:07.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:07.707616+0000) 2022-01-31T21:20:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:07 smithi167 conmon[49112]: debug 2022-01-31T21:20:07.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:07.679919+0000) 2022-01-31T21:20:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:07 smithi171 conmon[41853]: debug 2022-01-31T21:20:07.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:07.832116+0000) 2022-01-31T21:20:08.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[46715]: debug 2022-01-31T21:20:08.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.126960+0000) 2022-01-31T21:20:08.655 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:08 smithi167 conmon[49112]: debug 2022-01-31T21:20:08.543+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.544642+0000) 2022-01-31T21:20:08.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:08 smithi167 conmon[54076]: debug 2022-01-31T21:20:08.542+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.544419+0000) 2022-01-31T21:20:08.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:08 smithi167 conmon[54076]: debug 2022-01-31T21:20:08.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.601002+0000) 2022-01-31T21:20:08.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:08 smithi167 conmon[60316]: debug 2022-01-31T21:20:08.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.353315+0000) 2022-01-31T21:20:08.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:08 smithi167 conmon[60316]: debug 2022-01-31T21:20:08.542+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.544485+0000) 2022-01-31T21:20:08.790 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[41853]: debug 2022-01-31T21:20:08.544+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.545319+0000) 2022-01-31T21:20:08.791 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[46715]: debug 2022-01-31T21:20:08.543+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.543900+0000) 2022-01-31T21:20:08.791 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[35325]: debug 2022-01-31T21:20:08.556+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121591 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:08.792 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[35325]: debug 2022-01-31T21:20:08.662+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:20:08.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[51620]: debug 2022-01-31T21:20:08.544+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.545219+0000) 2022-01-31T21:20:08.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[51620]: debug 2022-01-31T21:20:08.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.707842+0000) 2022-01-31T21:20:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:08 smithi167 conmon[49112]: debug 2022-01-31T21:20:08.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.680120+0000) 2022-01-31T21:20:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:08 smithi171 conmon[41853]: debug 2022-01-31T21:20:08.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:08.832246+0000) 2022-01-31T21:20:09.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:09 smithi171 conmon[46715]: debug 2022-01-31T21:20:09.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:09.127187+0000) 2022-01-31T21:20:09.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:09 smithi167 conmon[54076]: debug 2022-01-31T21:20:09.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:09.601131+0000) 2022-01-31T21:20:09.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:09 smithi167 conmon[60316]: debug 2022-01-31T21:20:09.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:09.353530+0000) 2022-01-31T21:20:09.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:09 smithi171 conmon[51620]: debug 2022-01-31T21:20:09.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:09.707941+0000) 2022-01-31T21:20:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:09 smithi167 conmon[49112]: debug 2022-01-31T21:20:09.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:09.680279+0000) 2022-01-31T21:20:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:09 smithi171 conmon[41853]: debug 2022-01-31T21:20:09.832+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:09.832397+0000) 2022-01-31T21:20:10.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:10 smithi171 conmon[46715]: debug 2022-01-31T21:20:10.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:10.127334+0000) 2022-01-31T21:20:10.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:10 smithi167 conmon[54076]: debug 2022-01-31T21:20:10.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:10.601267+0000) 2022-01-31T21:20:10.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:10 smithi167 conmon[60316]: debug 2022-01-31T21:20:10.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:10.353707+0000) 2022-01-31T21:20:10.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:10 smithi171 conmon[51620]: debug 2022-01-31T21:20:10.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:10.708115+0000) 2022-01-31T21:20:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:10 smithi167 conmon[49112]: debug 2022-01-31T21:20:10.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:10.680458+0000) 2022-01-31T21:20:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:10 smithi171 conmon[41853]: debug 2022-01-31T21:20:10.831+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:10.832617+0000) 2022-01-31T21:20:11.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:11 smithi171 conmon[46715]: debug 2022-01-31T21:20:11.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:11.127529+0000) 2022-01-31T21:20:11.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:11 smithi167 conmon[54076]: debug 2022-01-31T21:20:11.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:11.601421+0000) 2022-01-31T21:20:11.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:11 smithi167 conmon[60316]: debug 2022-01-31T21:20:11.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:11.353848+0000) 2022-01-31T21:20:11.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:11 smithi171 conmon[51620]: debug 2022-01-31T21:20:11.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:11.708295+0000) 2022-01-31T21:20:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:11 smithi167 conmon[49112]: debug 2022-01-31T21:20:11.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:11.680675+0000) 2022-01-31T21:20:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:11 smithi171 conmon[41853]: debug 2022-01-31T21:20:11.832+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:11.832802+0000) 2022-01-31T21:20:12.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:12 smithi171 conmon[46715]: debug 2022-01-31T21:20:12.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:12.127686+0000) 2022-01-31T21:20:12.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:12 smithi167 conmon[60316]: debug 2022-01-31T21:20:12.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:12.354044+0000) 2022-01-31T21:20:12.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:12 smithi167 conmon[54076]: debug 2022-01-31T21:20:12.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:12.601641+0000) 2022-01-31T21:20:12.790 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:12 smithi171 conmon[51620]: debug 2022-01-31T21:20:12.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:12.708493+0000) 2022-01-31T21:20:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:12 smithi167 conmon[49112]: debug 2022-01-31T21:20:12.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:12.680849+0000) 2022-01-31T21:20:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:12 smithi171 conmon[41853]: debug 2022-01-31T21:20:12.832+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:12.833022+0000) 2022-01-31T21:20:13.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[46715]: debug 2022-01-31T21:20:13.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.127835+0000) 2022-01-31T21:20:13.656 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:13 smithi167 conmon[49112]: debug 2022-01-31T21:20:13.559+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.560288+0000) 2022-01-31T21:20:13.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:13 smithi167 conmon[60316]: debug 2022-01-31T21:20:13.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.354253+0000) 2022-01-31T21:20:13.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:13 smithi167 conmon[60316]: debug 2022-01-31T21:20:13.559+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.559942+0000) 2022-01-31T21:20:13.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:13 smithi167 conmon[54076]: debug 2022-01-31T21:20:13.559+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.560432+0000) 2022-01-31T21:20:13.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:13 smithi167 conmon[54076]: debug 2022-01-31T21:20:13.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.601834+0000) 2022-01-31T21:20:13.791 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[35325]: debug 2022-01-31T21:20:13.571+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121701 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:13.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[41853]: debug 2022-01-31T21:20:13.558+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.559381+0000) 2022-01-31T21:20:13.792 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[46715]: debug 2022-01-31T21:20:13.558+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.559667+0000) 2022-01-31T21:20:13.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[51620]: debug 2022-01-31T21:20:13.558+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.559501+0000) 2022-01-31T21:20:13.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[51620]: debug 2022-01-31T21:20:13.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.708634+0000) 2022-01-31T21:20:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:13 smithi167 conmon[49112]: debug 2022-01-31T21:20:13.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.681013+0000) 2022-01-31T21:20:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:13 smithi171 conmon[41853]: debug 2022-01-31T21:20:13.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:13.833210+0000) 2022-01-31T21:20:14.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:14 smithi171 conmon[46715]: debug 2022-01-31T21:20:14.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:14.128025+0000) 2022-01-31T21:20:14.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:14 smithi167 conmon[54076]: debug 2022-01-31T21:20:14.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:14.601951+0000) 2022-01-31T21:20:14.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:14 smithi167 conmon[60316]: debug 2022-01-31T21:20:14.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:14.354416+0000) 2022-01-31T21:20:14.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:14 smithi171 conmon[51620]: debug 2022-01-31T21:20:14.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:14.708723+0000) 2022-01-31T21:20:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:14 smithi167 conmon[49112]: debug 2022-01-31T21:20:14.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:14.681199+0000) 2022-01-31T21:20:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:14 smithi171 conmon[41853]: debug 2022-01-31T21:20:14.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:14.833384+0000) 2022-01-31T21:20:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:15 smithi167 conmon[60316]: debug 2022-01-31T21:20:15.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:15.354575+0000) 2022-01-31T21:20:15.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:15 smithi171 conmon[46715]: debug 2022-01-31T21:20:15.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:15.128181+0000) 2022-01-31T21:20:15.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:15 smithi171 conmon[51620]: debug 2022-01-31T21:20:15.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:15.708904+0000) 2022-01-31T21:20:15.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:15 smithi167 conmon[54076]: debug 2022-01-31T21:20:15.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:15.602154+0000) 2022-01-31T21:20:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:15 smithi167 conmon[49112]: debug 2022-01-31T21:20:15.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:15.681401+0000) 2022-01-31T21:20:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:15 smithi171 conmon[41853]: debug 2022-01-31T21:20:15.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:15.833568+0000) 2022-01-31T21:20:16.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:16 smithi171 conmon[46715]: debug 2022-01-31T21:20:16.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:16.128383+0000) 2022-01-31T21:20:16.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:16 smithi167 conmon[60316]: debug 2022-01-31T21:20:16.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:16.354723+0000) 2022-01-31T21:20:16.897 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:16 smithi171 conmon[51620]: debug 2022-01-31T21:20:16.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:16.709085+0000) 2022-01-31T21:20:16.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:16 smithi167 conmon[54076]: debug 2022-01-31T21:20:16.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:16.602344+0000) 2022-01-31T21:20:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:16 smithi167 conmon[49112]: debug 2022-01-31T21:20:16.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:16.681585+0000) 2022-01-31T21:20:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:16 smithi171 conmon[41853]: debug 2022-01-31T21:20:16.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:16.833726+0000) 2022-01-31T21:20:17.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:17 smithi167 conmon[60316]: debug 2022-01-31T21:20:17.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:17.354915+0000) 2022-01-31T21:20:17.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:17 smithi171 conmon[46715]: debug 2022-01-31T21:20:17.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:17.128563+0000) 2022-01-31T21:20:17.791 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:17 smithi171 conmon[51620]: debug 2022-01-31T21:20:17.709+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:17.709263+0000) 2022-01-31T21:20:17.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:17 smithi167 conmon[54076]: debug 2022-01-31T21:20:17.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:17.602543+0000) 2022-01-31T21:20:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:17 smithi167 conmon[49112]: debug 2022-01-31T21:20:17.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:17.681777+0000) 2022-01-31T21:20:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:17 smithi171 conmon[41853]: debug 2022-01-31T21:20:17.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:17.833829+0000) 2022-01-31T21:20:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:18 smithi167 conmon[60316]: debug 2022-01-31T21:20:18.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.355139+0000) 2022-01-31T21:20:18.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[46715]: debug 2022-01-31T21:20:18.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.128723+0000) 2022-01-31T21:20:18.791 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[35325]: debug 2022-01-31T21:20:18.586+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121811 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:18.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[41853]: debug 2022-01-31T21:20:18.575+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.576280+0000) 2022-01-31T21:20:18.793 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[46715]: debug 2022-01-31T21:20:18.575+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.575765+0000) 2022-01-31T21:20:18.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[51620]: debug 2022-01-31T21:20:18.573+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.574135+0000) 2022-01-31T21:20:18.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[51620]: debug 2022-01-31T21:20:18.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.709440+0000) 2022-01-31T21:20:18.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:18 smithi167 conmon[60316]: debug 2022-01-31T21:20:18.575+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.576005+0000) 2022-01-31T21:20:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:18 smithi167 conmon[49112]: debug 2022-01-31T21:20:18.575+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.575677+0000) 2022-01-31T21:20:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:18 smithi167 conmon[49112]: debug 2022-01-31T21:20:18.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.681899+0000) 2022-01-31T21:20:18.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:18 smithi167 conmon[54076]: debug 2022-01-31T21:20:18.574+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.575380+0000) 2022-01-31T21:20:18.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:18 smithi167 conmon[54076]: debug 2022-01-31T21:20:18.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.602687+0000) 2022-01-31T21:20:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:18 smithi171 conmon[41853]: debug 2022-01-31T21:20:18.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:18.833983+0000) 2022-01-31T21:20:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:19 smithi167 conmon[60316]: debug 2022-01-31T21:20:19.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:19.355344+0000) 2022-01-31T21:20:19.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:19 smithi171 conmon[46715]: debug 2022-01-31T21:20:19.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:19.128913+0000) 2022-01-31T21:20:19.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:19 smithi171 conmon[51620]: debug 2022-01-31T21:20:19.707+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:19.709613+0000) 2022-01-31T21:20:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:19 smithi167 conmon[49112]: debug 2022-01-31T21:20:19.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:19.682060+0000) 2022-01-31T21:20:19.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:19 smithi167 conmon[54076]: debug 2022-01-31T21:20:19.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:19.602830+0000) 2022-01-31T21:20:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:19 smithi171 conmon[41853]: debug 2022-01-31T21:20:19.832+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:19.834121+0000) 2022-01-31T21:20:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:20 smithi167 conmon[60316]: debug 2022-01-31T21:20:20.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:20.355517+0000) 2022-01-31T21:20:20.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:20 smithi171 conmon[46715]: debug 2022-01-31T21:20:20.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:20.129087+0000) 2022-01-31T21:20:20.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:20 smithi171 conmon[51620]: debug 2022-01-31T21:20:20.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:20.709816+0000) 2022-01-31T21:20:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:20 smithi167 conmon[49112]: debug 2022-01-31T21:20:20.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:20.682264+0000) 2022-01-31T21:20:20.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:20 smithi167 conmon[54076]: debug 2022-01-31T21:20:20.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:20.603036+0000) 2022-01-31T21:20:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:20 smithi171 conmon[41853]: debug 2022-01-31T21:20:20.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:20.834285+0000) 2022-01-31T21:20:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:21 smithi167 conmon[60316]: debug 2022-01-31T21:20:21.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:21.355689+0000) 2022-01-31T21:20:21.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:21 smithi171 conmon[46715]: debug 2022-01-31T21:20:21.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:21.129249+0000) 2022-01-31T21:20:21.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:21 smithi171 conmon[51620]: debug 2022-01-31T21:20:21.709+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:21.710001+0000) 2022-01-31T21:20:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:21 smithi167 conmon[49112]: debug 2022-01-31T21:20:21.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:21.682449+0000) 2022-01-31T21:20:21.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:21 smithi167 conmon[54076]: debug 2022-01-31T21:20:21.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:21.603183+0000) 2022-01-31T21:20:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:21 smithi171 conmon[41853]: debug 2022-01-31T21:20:21.832+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:21.834406+0000) 2022-01-31T21:20:22.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:22 smithi167 conmon[60316]: debug 2022-01-31T21:20:22.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:22.355880+0000) 2022-01-31T21:20:22.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:22 smithi171 conmon[46715]: debug 2022-01-31T21:20:22.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:22.130832+0000) 2022-01-31T21:20:22.792 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:22 smithi171 conmon[51620]: debug 2022-01-31T21:20:22.708+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:22.710193+0000) 2022-01-31T21:20:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:22 smithi167 conmon[49112]: debug 2022-01-31T21:20:22.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:22.682575+0000) 2022-01-31T21:20:22.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:22 smithi167 conmon[54076]: debug 2022-01-31T21:20:22.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:22.603359+0000) 2022-01-31T21:20:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:22 smithi171 conmon[41853]: debug 2022-01-31T21:20:22.832+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:22.834628+0000) 2022-01-31T21:20:23.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[46715]: debug 2022-01-31T21:20:23.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.131107+0000) 2022-01-31T21:20:23.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:23 smithi167 conmon[60316]: debug 2022-01-31T21:20:23.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.356128+0000) 2022-01-31T21:20:23.792 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[46715]: debug 2022-01-31T21:20:23.587+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.589592+0000) 2022-01-31T21:20:23.793 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[35325]: debug 2022-01-31T21:20:23.601+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 121921 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:23.793 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[35325]: debug 2022-01-31T21:20:23.662+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:20:23.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[41853]: debug 2022-01-31T21:20:23.589+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.590948+0000) 2022-01-31T21:20:23.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[51620]: debug 2022-01-31T21:20:23.587+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.589709+0000) 2022-01-31T21:20:23.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[51620]: debug 2022-01-31T21:20:23.709+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.710399+0000) 2022-01-31T21:20:23.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:23 smithi167 conmon[60316]: debug 2022-01-31T21:20:23.590+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.590664+0000) 2022-01-31T21:20:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:23 smithi167 conmon[49112]: debug 2022-01-31T21:20:23.590+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.591148+0000) 2022-01-31T21:20:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:23 smithi167 conmon[49112]: debug 2022-01-31T21:20:23.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.682729+0000) 2022-01-31T21:20:23.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:23 smithi167 conmon[54076]: debug 2022-01-31T21:20:23.591+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.591541+0000) 2022-01-31T21:20:23.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:23 smithi167 conmon[54076]: debug 2022-01-31T21:20:23.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.603555+0000) 2022-01-31T21:20:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:23 smithi171 conmon[41853]: debug 2022-01-31T21:20:23.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:23.834843+0000) 2022-01-31T21:20:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:24 smithi167 conmon[60316]: debug 2022-01-31T21:20:24.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:24.356272+0000) 2022-01-31T21:20:24.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:24 smithi171 conmon[46715]: debug 2022-01-31T21:20:24.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:24.131266+0000) 2022-01-31T21:20:24.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:24 smithi171 conmon[51620]: debug 2022-01-31T21:20:24.709+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:24.710545+0000) 2022-01-31T21:20:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:24 smithi167 conmon[49112]: debug 2022-01-31T21:20:24.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:24.682908+0000) 2022-01-31T21:20:24.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:24 smithi167 conmon[54076]: debug 2022-01-31T21:20:24.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:24.603643+0000) 2022-01-31T21:20:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:24 smithi171 conmon[41853]: debug 2022-01-31T21:20:24.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:24.834936+0000) 2022-01-31T21:20:25.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:25 smithi167 conmon[60316]: debug 2022-01-31T21:20:25.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:25.356458+0000) 2022-01-31T21:20:25.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:25 smithi171 conmon[46715]: debug 2022-01-31T21:20:25.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:25.131422+0000) 2022-01-31T21:20:25.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:25 smithi171 conmon[51620]: debug 2022-01-31T21:20:25.709+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:25.710771+0000) 2022-01-31T21:20:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:25 smithi167 conmon[49112]: debug 2022-01-31T21:20:25.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:25.683089+0000) 2022-01-31T21:20:25.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:25 smithi167 conmon[54076]: debug 2022-01-31T21:20:25.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:25.603807+0000) 2022-01-31T21:20:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:25 smithi171 conmon[41853]: debug 2022-01-31T21:20:25.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:25.835038+0000) 2022-01-31T21:20:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:26 smithi167 conmon[60316]: debug 2022-01-31T21:20:26.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:26.356663+0000) 2022-01-31T21:20:26.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:26 smithi171 conmon[46715]: debug 2022-01-31T21:20:26.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:26.131610+0000) 2022-01-31T21:20:26.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:26 smithi171 conmon[51620]: debug 2022-01-31T21:20:26.710+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:26.710970+0000) 2022-01-31T21:20:26.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:26 smithi167 conmon[54076]: debug 2022-01-31T21:20:26.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:26.603974+0000) 2022-01-31T21:20:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:26 smithi167 conmon[49112]: debug 2022-01-31T21:20:26.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:26.683256+0000) 2022-01-31T21:20:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:26 smithi171 conmon[41853]: debug 2022-01-31T21:20:26.834+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:26.835156+0000) 2022-01-31T21:20:27.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:27 smithi167 conmon[60316]: debug 2022-01-31T21:20:27.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:27.357006+0000) 2022-01-31T21:20:27.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:27 smithi171 conmon[46715]: debug 2022-01-31T21:20:27.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:27.131803+0000) 2022-01-31T21:20:27.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:27 smithi171 conmon[51620]: debug 2022-01-31T21:20:27.710+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:27.711190+0000) 2022-01-31T21:20:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:27 smithi167 conmon[49112]: debug 2022-01-31T21:20:27.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:27.683447+0000) 2022-01-31T21:20:27.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:27 smithi167 conmon[54076]: debug 2022-01-31T21:20:27.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:27.604161+0000) 2022-01-31T21:20:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:27 smithi171 conmon[41853]: debug 2022-01-31T21:20:27.833+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:27.835358+0000) 2022-01-31T21:20:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:28 smithi167 conmon[60316]: debug 2022-01-31T21:20:28.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.357792+0000) 2022-01-31T21:20:28.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[46715]: debug 2022-01-31T21:20:28.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.131988+0000) 2022-01-31T21:20:28.793 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[35325]: debug 2022-01-31T21:20:28.616+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122031 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:28.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[41853]: debug 2022-01-31T21:20:28.603+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.604812+0000) 2022-01-31T21:20:28.794 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[46715]: debug 2022-01-31T21:20:28.602+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.604634+0000) 2022-01-31T21:20:28.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[51620]: debug 2022-01-31T21:20:28.604+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.605773+0000) 2022-01-31T21:20:28.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[51620]: debug 2022-01-31T21:20:28.709+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.711404+0000) 2022-01-31T21:20:28.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:28 smithi167 conmon[60316]: debug 2022-01-31T21:20:28.604+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.605319+0000) 2022-01-31T21:20:28.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:28 smithi167 conmon[54076]: debug 2022-01-31T21:20:28.606+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.606691+0000) 2022-01-31T21:20:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:28 smithi167 conmon[49112]: debug 2022-01-31T21:20:28.605+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.605556+0000) 2022-01-31T21:20:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:28 smithi167 conmon[49112]: debug 2022-01-31T21:20:28.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.683588+0000) 2022-01-31T21:20:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:28 smithi171 conmon[41853]: debug 2022-01-31T21:20:28.834+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:28.835540+0000) 2022-01-31T21:20:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:29 smithi167 conmon[60316]: debug 2022-01-31T21:20:29.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:29.357988+0000) 2022-01-31T21:20:29.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:29 smithi171 conmon[46715]: debug 2022-01-31T21:20:29.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:29.132206+0000) 2022-01-31T21:20:29.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:29 smithi171 conmon[51620]: debug 2022-01-31T21:20:29.710+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:29.711557+0000) 2022-01-31T21:20:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:29 smithi167 conmon[49112]: debug 2022-01-31T21:20:29.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:29.683679+0000) 2022-01-31T21:20:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:29 smithi171 conmon[41853]: debug 2022-01-31T21:20:29.834+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:29.835659+0000) 2022-01-31T21:20:30.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:30 smithi167 conmon[60316]: debug 2022-01-31T21:20:30.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:30.358166+0000) 2022-01-31T21:20:30.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:30 smithi171 conmon[46715]: debug 2022-01-31T21:20:30.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:30.132343+0000) 2022-01-31T21:20:30.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:30 smithi171 conmon[51620]: debug 2022-01-31T21:20:30.710+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:30.711719+0000) 2022-01-31T21:20:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:30 smithi167 conmon[49112]: debug 2022-01-31T21:20:30.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:30.683809+0000) 2022-01-31T21:20:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:30 smithi171 conmon[41853]: debug 2022-01-31T21:20:30.835+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:30.835814+0000) 2022-01-31T21:20:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:31 smithi167 conmon[60316]: debug 2022-01-31T21:20:31.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:31.358399+0000) 2022-01-31T21:20:31.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:31 smithi171 conmon[46715]: debug 2022-01-31T21:20:31.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:31.132497+0000) 2022-01-31T21:20:31.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:31 smithi171 conmon[51620]: debug 2022-01-31T21:20:31.711+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:31.711904+0000) 2022-01-31T21:20:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:31 smithi167 conmon[49112]: debug 2022-01-31T21:20:31.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:31.684005+0000) 2022-01-31T21:20:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:31 smithi171 conmon[41853]: debug 2022-01-31T21:20:31.835+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:31.836022+0000) 2022-01-31T21:20:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:32 smithi167 conmon[60316]: debug 2022-01-31T21:20:32.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:32.358604+0000) 2022-01-31T21:20:32.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:32 smithi171 conmon[46715]: debug 2022-01-31T21:20:32.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:32.132676+0000) 2022-01-31T21:20:32.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:32 smithi171 conmon[51620]: debug 2022-01-31T21:20:32.711+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:32.712099+0000) 2022-01-31T21:20:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:32 smithi167 conmon[49112]: debug 2022-01-31T21:20:32.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:32.684216+0000) 2022-01-31T21:20:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:32 smithi171 conmon[41853]: debug 2022-01-31T21:20:32.835+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:32.836211+0000) 2022-01-31T21:20:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:33 smithi167 conmon[60316]: debug 2022-01-31T21:20:33.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.358805+0000) 2022-01-31T21:20:33.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[46715]: debug 2022-01-31T21:20:33.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.132907+0000) 2022-01-31T21:20:33.794 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[35325]: debug 2022-01-31T21:20:33.631+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122141 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:33.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[41853]: debug 2022-01-31T21:20:33.619+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.620120+0000) 2022-01-31T21:20:33.795 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[46715]: debug 2022-01-31T21:20:33.619+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.619907+0000) 2022-01-31T21:20:33.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[51620]: debug 2022-01-31T21:20:33.620+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.620852+0000) 2022-01-31T21:20:33.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[51620]: debug 2022-01-31T21:20:33.711+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.712276+0000) 2022-01-31T21:20:33.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:33 smithi167 conmon[60316]: debug 2022-01-31T21:20:33.619+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.620131+0000) 2022-01-31T21:20:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:33 smithi167 conmon[49112]: debug 2022-01-31T21:20:33.620+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.621184+0000) 2022-01-31T21:20:33.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:33 smithi167 conmon[49112]: debug 2022-01-31T21:20:33.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.684398+0000) 2022-01-31T21:20:33.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:33 smithi167 conmon[54076]: debug 2022-01-31T21:20:33.619+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.620396+0000) 2022-01-31T21:20:34.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:33 smithi171 conmon[41853]: debug 2022-01-31T21:20:33.835+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:33.836360+0000) 2022-01-31T21:20:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:34 smithi167 conmon[60316]: debug 2022-01-31T21:20:34.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:34.359013+0000) 2022-01-31T21:20:34.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:34 smithi171 conmon[46715]: debug 2022-01-31T21:20:34.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:34.133135+0000) 2022-01-31T21:20:34.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:34 smithi171 conmon[51620]: debug 2022-01-31T21:20:34.711+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:34.712444+0000) 2022-01-31T21:20:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:34 smithi167 conmon[49112]: debug 2022-01-31T21:20:34.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:34.684560+0000) 2022-01-31T21:20:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:34 smithi171 conmon[41853]: debug 2022-01-31T21:20:34.835+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:34.836514+0000) 2022-01-31T21:20:35.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:35 smithi167 conmon[54076]: debug 2022-01-31T21:20:35.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:35.354589+0000) 2022-01-31T21:20:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:35 smithi167 conmon[60316]: debug 2022-01-31T21:20:35.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:35.359169+0000) 2022-01-31T21:20:35.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:35 smithi171 conmon[46715]: debug 2022-01-31T21:20:35.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:35.133263+0000) 2022-01-31T21:20:35.794 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:35 smithi171 conmon[51620]: debug 2022-01-31T21:20:35.711+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:35.712627+0000) 2022-01-31T21:20:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:35 smithi167 conmon[49112]: debug 2022-01-31T21:20:35.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:35.684690+0000) 2022-01-31T21:20:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:35 smithi171 conmon[41853]: debug 2022-01-31T21:20:35.836+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:35.836699+0000) 2022-01-31T21:20:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:36 smithi167 conmon[54076]: debug 2022-01-31T21:20:36.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:36.354754+0000) 2022-01-31T21:20:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:36 smithi167 conmon[60316]: debug 2022-01-31T21:20:36.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:36.359348+0000) 2022-01-31T21:20:36.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:36 smithi171 conmon[46715]: debug 2022-01-31T21:20:36.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:36.133417+0000) 2022-01-31T21:20:36.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:36 smithi171 conmon[51620]: debug 2022-01-31T21:20:36.712+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:36.712817+0000) 2022-01-31T21:20:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:36 smithi167 conmon[49112]: debug 2022-01-31T21:20:36.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:36.684892+0000) 2022-01-31T21:20:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:36 smithi171 conmon[41853]: debug 2022-01-31T21:20:36.836+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:36.836908+0000) 2022-01-31T21:20:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:37 smithi167 conmon[54076]: debug 2022-01-31T21:20:37.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:37.354931+0000) 2022-01-31T21:20:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:37 smithi167 conmon[60316]: debug 2022-01-31T21:20:37.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:37.359563+0000) 2022-01-31T21:20:37.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:37 smithi171 conmon[46715]: debug 2022-01-31T21:20:37.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:37.133550+0000) 2022-01-31T21:20:37.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:37 smithi171 conmon[51620]: debug 2022-01-31T21:20:37.712+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:37.712946+0000) 2022-01-31T21:20:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:37 smithi167 conmon[49112]: debug 2022-01-31T21:20:37.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:37.685086+0000) 2022-01-31T21:20:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:37 smithi171 conmon[41853]: debug 2022-01-31T21:20:37.836+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:37.837036+0000) 2022-01-31T21:20:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:38 smithi167 conmon[54076]: debug 2022-01-31T21:20:38.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.355140+0000) 2022-01-31T21:20:38.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:38 smithi167 conmon[60316]: debug 2022-01-31T21:20:38.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.359742+0000) 2022-01-31T21:20:38.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[46715]: debug 2022-01-31T21:20:38.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.133701+0000) 2022-01-31T21:20:38.795 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[41853]: debug 2022-01-31T21:20:38.633+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.634436+0000) 2022-01-31T21:20:38.796 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[46715]: debug 2022-01-31T21:20:38.634+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.635430+0000) 2022-01-31T21:20:38.796 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[35325]: debug 2022-01-31T21:20:38.646+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122251 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:38.796 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[35325]: debug 2022-01-31T21:20:38.663+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:20:38.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[51620]: debug 2022-01-31T21:20:38.634+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.634826+0000) 2022-01-31T21:20:38.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[51620]: debug 2022-01-31T21:20:38.712+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.713067+0000) 2022-01-31T21:20:38.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:38 smithi167 conmon[54076]: debug 2022-01-31T21:20:38.634+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.635674+0000) 2022-01-31T21:20:38.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:38 smithi167 conmon[60316]: debug 2022-01-31T21:20:38.633+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.634878+0000) 2022-01-31T21:20:38.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:38 smithi167 conmon[49112]: debug 2022-01-31T21:20:38.634+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.636143+0000) 2022-01-31T21:20:38.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:38 smithi167 conmon[49112]: debug 2022-01-31T21:20:38.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.685285+0000) 2022-01-31T21:20:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:38 smithi171 conmon[41853]: debug 2022-01-31T21:20:38.836+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:38.837187+0000) 2022-01-31T21:20:39.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:39 smithi167 conmon[54076]: debug 2022-01-31T21:20:39.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:39.355346+0000) 2022-01-31T21:20:39.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:39 smithi167 conmon[60316]: debug 2022-01-31T21:20:39.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:39.359895+0000) 2022-01-31T21:20:39.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:39 smithi171 conmon[46715]: debug 2022-01-31T21:20:39.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:39.133860+0000) 2022-01-31T21:20:39.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:39 smithi171 conmon[51620]: debug 2022-01-31T21:20:39.712+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:39.713201+0000) 2022-01-31T21:20:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:39 smithi167 conmon[49112]: debug 2022-01-31T21:20:39.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:39.685450+0000) 2022-01-31T21:20:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:39 smithi171 conmon[41853]: debug 2022-01-31T21:20:39.836+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:39.837338+0000) 2022-01-31T21:20:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:40 smithi167 conmon[54076]: debug 2022-01-31T21:20:40.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:40.355534+0000) 2022-01-31T21:20:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:40 smithi167 conmon[60316]: debug 2022-01-31T21:20:40.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:40.360081+0000) 2022-01-31T21:20:40.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:40 smithi171 conmon[46715]: debug 2022-01-31T21:20:40.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:40.134033+0000) 2022-01-31T21:20:40.795 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:40 smithi171 conmon[51620]: debug 2022-01-31T21:20:40.712+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:40.713346+0000) 2022-01-31T21:20:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:40 smithi167 conmon[49112]: debug 2022-01-31T21:20:40.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:40.685618+0000) 2022-01-31T21:20:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:40 smithi171 conmon[41853]: debug 2022-01-31T21:20:40.836+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:40.837526+0000) 2022-01-31T21:20:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:41 smithi167 conmon[54076]: debug 2022-01-31T21:20:41.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:41.355725+0000) 2022-01-31T21:20:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:41 smithi167 conmon[60316]: debug 2022-01-31T21:20:41.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:41.360289+0000) 2022-01-31T21:20:41.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:41 smithi171 conmon[46715]: debug 2022-01-31T21:20:41.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:41.134224+0000) 2022-01-31T21:20:41.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:41 smithi171 conmon[51620]: debug 2022-01-31T21:20:41.712+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:41.713495+0000) 2022-01-31T21:20:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:41 smithi167 conmon[49112]: debug 2022-01-31T21:20:41.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:41.685790+0000) 2022-01-31T21:20:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:41 smithi171 conmon[41853]: debug 2022-01-31T21:20:41.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:41.837706+0000) 2022-01-31T21:20:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:42 smithi167 conmon[54076]: debug 2022-01-31T21:20:42.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:42.355880+0000) 2022-01-31T21:20:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:42 smithi167 conmon[60316]: debug 2022-01-31T21:20:42.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:42.360459+0000) 2022-01-31T21:20:42.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:42 smithi171 conmon[46715]: debug 2022-01-31T21:20:42.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:42.134379+0000) 2022-01-31T21:20:42.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:42 smithi171 conmon[51620]: debug 2022-01-31T21:20:42.713+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:42.713674+0000) 2022-01-31T21:20:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:42 smithi167 conmon[49112]: debug 2022-01-31T21:20:42.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:42.685943+0000) 2022-01-31T21:20:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:42 smithi171 conmon[41853]: debug 2022-01-31T21:20:42.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:42.837850+0000) 2022-01-31T21:20:43.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:43 smithi167 conmon[54076]: debug 2022-01-31T21:20:43.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.356029+0000) 2022-01-31T21:20:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:43 smithi167 conmon[60316]: debug 2022-01-31T21:20:43.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.360632+0000) 2022-01-31T21:20:43.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[46715]: debug 2022-01-31T21:20:43.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.134511+0000) 2022-01-31T21:20:43.796 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[35325]: debug 2022-01-31T21:20:43.661+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122361 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:43.797 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[46715]: debug 2022-01-31T21:20:43.649+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.650679+0000) 2022-01-31T21:20:43.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[41853]: debug 2022-01-31T21:20:43.648+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.649013+0000) 2022-01-31T21:20:43.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[51620]: debug 2022-01-31T21:20:43.648+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.649340+0000) 2022-01-31T21:20:43.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[51620]: debug 2022-01-31T21:20:43.713+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.713826+0000) 2022-01-31T21:20:43.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:43 smithi167 conmon[54076]: debug 2022-01-31T21:20:43.649+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.650577+0000) 2022-01-31T21:20:43.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:43 smithi167 conmon[60316]: debug 2022-01-31T21:20:43.648+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.649628+0000) 2022-01-31T21:20:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:43 smithi167 conmon[49112]: debug 2022-01-31T21:20:43.649+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.650914+0000) 2022-01-31T21:20:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:43 smithi167 conmon[49112]: debug 2022-01-31T21:20:43.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.686132+0000) 2022-01-31T21:20:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:43 smithi171 conmon[41853]: debug 2022-01-31T21:20:43.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:43.837976+0000) 2022-01-31T21:20:44.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:44 smithi167 conmon[54076]: debug 2022-01-31T21:20:44.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:44.356192+0000) 2022-01-31T21:20:44.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:44 smithi167 conmon[60316]: debug 2022-01-31T21:20:44.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:44.360787+0000) 2022-01-31T21:20:44.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:44 smithi171 conmon[46715]: debug 2022-01-31T21:20:44.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:44.134665+0000) 2022-01-31T21:20:44.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:44 smithi171 conmon[51620]: debug 2022-01-31T21:20:44.713+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:44.713967+0000) 2022-01-31T21:20:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:44 smithi167 conmon[49112]: debug 2022-01-31T21:20:44.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:44.686321+0000) 2022-01-31T21:20:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:44 smithi171 conmon[41853]: debug 2022-01-31T21:20:44.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:44.838133+0000) 2022-01-31T21:20:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:45 smithi167 conmon[54076]: debug 2022-01-31T21:20:45.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:45.356340+0000) 2022-01-31T21:20:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:45 smithi167 conmon[60316]: debug 2022-01-31T21:20:45.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:45.360981+0000) 2022-01-31T21:20:45.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:45 smithi171 conmon[46715]: debug 2022-01-31T21:20:45.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:45.134857+0000) 2022-01-31T21:20:45.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:45 smithi171 conmon[51620]: debug 2022-01-31T21:20:45.713+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:45.714097+0000) 2022-01-31T21:20:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:45 smithi167 conmon[49112]: debug 2022-01-31T21:20:45.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:45.686553+0000) 2022-01-31T21:20:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:45 smithi171 conmon[41853]: debug 2022-01-31T21:20:45.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:45.838343+0000) 2022-01-31T21:20:46.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:46 smithi171 conmon[46715]: debug 2022-01-31T21:20:46.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:46.134983+0000) 2022-01-31T21:20:46.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:46 smithi167 conmon[54076]: debug 2022-01-31T21:20:46.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:46.356492+0000) 2022-01-31T21:20:46.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:46 smithi167 conmon[60316]: debug 2022-01-31T21:20:46.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:46.361140+0000) 2022-01-31T21:20:46.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:46 smithi171 conmon[51620]: debug 2022-01-31T21:20:46.713+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:46.714268+0000) 2022-01-31T21:20:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:46 smithi167 conmon[49112]: debug 2022-01-31T21:20:46.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:46.686739+0000) 2022-01-31T21:20:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:46 smithi171 conmon[41853]: debug 2022-01-31T21:20:46.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:46.838432+0000) 2022-01-31T21:20:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:47 smithi167 conmon[54076]: debug 2022-01-31T21:20:47.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:47.356671+0000) 2022-01-31T21:20:47.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:47 smithi167 conmon[60316]: debug 2022-01-31T21:20:47.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:47.361343+0000) 2022-01-31T21:20:47.622 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:47 smithi171 conmon[46715]: debug 2022-01-31T21:20:47.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:47.135173+0000) 2022-01-31T21:20:47.796 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:47 smithi171 conmon[51620]: debug 2022-01-31T21:20:47.713+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:47.714447+0000) 2022-01-31T21:20:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:47 smithi167 conmon[49112]: debug 2022-01-31T21:20:47.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:47.686901+0000) 2022-01-31T21:20:48.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:47 smithi171 conmon[41853]: debug 2022-01-31T21:20:47.838+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:47.838582+0000) 2022-01-31T21:20:48.222 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T21:20:48.222+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T21:20:48.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:48 smithi167 conmon[54076]: debug 2022-01-31T21:20:48.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.356819+0000) 2022-01-31T21:20:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:48 smithi167 conmon[60316]: debug 2022-01-31T21:20:48.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.361459+0000) 2022-01-31T21:20:48.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[46715]: debug 2022-01-31T21:20:48.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.135380+0000) 2022-01-31T21:20:48.797 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[35325]: debug 2022-01-31T21:20:48.676+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122471 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:48.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[41853]: debug 2022-01-31T21:20:48.663+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.664615+0000) 2022-01-31T21:20:48.798 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[46715]: debug 2022-01-31T21:20:48.664+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.665491+0000) 2022-01-31T21:20:48.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[51620]: debug 2022-01-31T21:20:48.663+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.663906+0000) 2022-01-31T21:20:48.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[51620]: debug 2022-01-31T21:20:48.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.714622+0000) 2022-01-31T21:20:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:48 smithi167 conmon[49112]: debug 2022-01-31T21:20:48.663+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.665470+0000) 2022-01-31T21:20:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:48 smithi167 conmon[49112]: debug 2022-01-31T21:20:48.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.687074+0000) 2022-01-31T21:20:48.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:48 smithi167 conmon[54076]: debug 2022-01-31T21:20:48.664+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.665735+0000) 2022-01-31T21:20:48.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:48 smithi167 conmon[60316]: debug 2022-01-31T21:20:48.663+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.665019+0000) 2022-01-31T21:20:49.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:48 smithi171 conmon[41853]: debug 2022-01-31T21:20:48.837+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:48.838771+0000) 2022-01-31T21:20:49.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:49 smithi167 conmon[54076]: debug 2022-01-31T21:20:49.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:49.357038+0000) 2022-01-31T21:20:49.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:49 smithi167 conmon[60316]: debug 2022-01-31T21:20:49.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:49.361636+0000) 2022-01-31T21:20:49.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:49 smithi171 conmon[46715]: debug 2022-01-31T21:20:49.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:49.135602+0000) 2022-01-31T21:20:49.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:49 smithi171 conmon[51620]: debug 2022-01-31T21:20:49.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:49.714787+0000) 2022-01-31T21:20:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:49 smithi171 conmon[41853]: debug 2022-01-31T21:20:49.838+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:49.838903+0000) 2022-01-31T21:20:50.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:49 smithi167 conmon[49112]: debug 2022-01-31T21:20:49.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:49.687206+0000) 2022-01-31T21:20:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:50 smithi167 conmon[54076]: debug 2022-01-31T21:20:50.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:50.357211+0000) 2022-01-31T21:20:50.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:50 smithi167 conmon[60316]: debug 2022-01-31T21:20:50.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:50.361790+0000) 2022-01-31T21:20:50.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:50 smithi171 conmon[46715]: debug 2022-01-31T21:20:50.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:50.135748+0000) 2022-01-31T21:20:50.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:50 smithi171 conmon[51620]: debug 2022-01-31T21:20:50.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:50.714920+0000) 2022-01-31T21:20:51.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:50 smithi171 conmon[41853]: debug 2022-01-31T21:20:50.838+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:50.839055+0000) 2022-01-31T21:20:51.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:50 smithi167 conmon[49112]: debug 2022-01-31T21:20:50.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:50.687373+0000) 2022-01-31T21:20:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:51 smithi167 conmon[54076]: debug 2022-01-31T21:20:51.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:51.357396+0000) 2022-01-31T21:20:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:51 smithi167 conmon[60316]: debug 2022-01-31T21:20:51.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:51.361894+0000) 2022-01-31T21:20:51.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:51 smithi171 conmon[46715]: debug 2022-01-31T21:20:51.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:51.135907+0000) 2022-01-31T21:20:51.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:51 smithi171 conmon[51620]: debug 2022-01-31T21:20:51.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:51.715034+0000) 2022-01-31T21:20:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:51 smithi171 conmon[41853]: debug 2022-01-31T21:20:51.838+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:51.839237+0000) 2022-01-31T21:20:52.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:51 smithi167 conmon[49112]: debug 2022-01-31T21:20:51.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:51.687576+0000) 2022-01-31T21:20:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:52 smithi167 conmon[54076]: debug 2022-01-31T21:20:52.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:52.357613+0000) 2022-01-31T21:20:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:52 smithi167 conmon[60316]: debug 2022-01-31T21:20:52.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:52.362029+0000) 2022-01-31T21:20:52.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:52 smithi171 conmon[46715]: debug 2022-01-31T21:20:52.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:52.136082+0000) 2022-01-31T21:20:52.797 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:52 smithi171 conmon[51620]: debug 2022-01-31T21:20:52.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:52.715213+0000) 2022-01-31T21:20:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:52 smithi171 conmon[41853]: debug 2022-01-31T21:20:52.839+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:52.839450+0000) 2022-01-31T21:20:53.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:52 smithi167 conmon[49112]: debug 2022-01-31T21:20:52.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:52.687772+0000) 2022-01-31T21:20:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[54076]: debug 2022-01-31T21:20:53.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.357767+0000) 2022-01-31T21:20:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[60316]: debug 2022-01-31T21:20:53.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.362199+0000) 2022-01-31T21:20:53.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[46715]: debug 2022-01-31T21:20:53.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.136268+0000) 2022-01-31T21:20:53.797 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[41853]: debug 2022-01-31T21:20:53.678+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.679379+0000) 2022-01-31T21:20:53.798 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[46715]: debug 2022-01-31T21:20:53.679+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.679909+0000) 2022-01-31T21:20:53.799 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[35325]: debug 2022-01-31T21:20:53.664+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:20:53.799 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[35325]: debug 2022-01-31T21:20:53.707+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122581 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:53.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[51620]: debug 2022-01-31T21:20:53.678+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.679209+0000) 2022-01-31T21:20:53.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[51620]: debug 2022-01-31T21:20:53.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.715308+0000) 2022-01-31T21:20:54.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:53 smithi171 conmon[41853]: debug 2022-01-31T21:20:53.839+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.839664+0000) 2022-01-31T21:20:54.112 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[54076]: debug 2022-01-31T21:20:53.678+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.679575+0000) 2022-01-31T21:20:54.113 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[60316]: debug 2022-01-31T21:20:53.677+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.679359+0000) 2022-01-31T21:20:54.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[49112]: debug 2022-01-31T21:20:53.679+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.680652+0000) 2022-01-31T21:20:54.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[49112]: debug 2022-01-31T21:20:53.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:53.687937+0000) 2022-01-31T21:20:54.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:53 smithi167 conmon[49112]: 2022-01-31T21:20:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:54 smithi167 conmon[60316]: debug 2022-01-31T21:20:54.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:54.362364+0000) 2022-01-31T21:20:54.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:54 smithi167 conmon[54076]: debug 2022-01-31T21:20:54.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:54.357898+0000) 2022-01-31T21:20:54.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:54 smithi171 conmon[46715]: debug 2022-01-31T21:20:54.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:54.136457+0000) 2022-01-31T21:20:54.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:54 smithi171 conmon[51620]: debug 2022-01-31T21:20:54.714+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:54.715470+0000) 2022-01-31T21:20:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:54 smithi171 conmon[41853]: debug 2022-01-31T21:20:54.839+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:54.839804+0000) 2022-01-31T21:20:55.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:54 smithi167 conmon[49112]: debug 2022-01-31T21:20:54.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:54.688104+0000) 2022-01-31T21:20:55.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:55 smithi167 conmon[54076]: debug 2022-01-31T21:20:55.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:55.358048+0000) 2022-01-31T21:20:55.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:55 smithi167 conmon[60316]: debug 2022-01-31T21:20:55.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:55.362537+0000) 2022-01-31T21:20:55.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:55 smithi171 conmon[46715]: debug 2022-01-31T21:20:55.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:55.136616+0000) 2022-01-31T21:20:55.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:55 smithi171 conmon[51620]: debug 2022-01-31T21:20:55.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:55.715675+0000) 2022-01-31T21:20:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:55 smithi171 conmon[41853]: debug 2022-01-31T21:20:55.839+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:55.839962+0000) 2022-01-31T21:20:56.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:55 smithi167 conmon[49112]: debug 2022-01-31T21:20:55.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:55.688304+0000) 2022-01-31T21:20:56.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:56 smithi167 conmon[54076]: debug 2022-01-31T21:20:56.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:56.358261+0000) 2022-01-31T21:20:56.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:56 smithi167 conmon[60316]: debug 2022-01-31T21:20:56.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:56.362682+0000) 2022-01-31T21:20:56.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:56 smithi171 conmon[46715]: debug 2022-01-31T21:20:56.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:56.136804+0000) 2022-01-31T21:20:56.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:56 smithi171 conmon[51620]: debug 2022-01-31T21:20:56.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:56.715814+0000) 2022-01-31T21:20:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:56 smithi171 conmon[41853]: debug 2022-01-31T21:20:56.839+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:56.840121+0000) 2022-01-31T21:20:57.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:56 smithi167 conmon[49112]: debug 2022-01-31T21:20:56.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:56.688530+0000) 2022-01-31T21:20:57.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:57 smithi167 conmon[54076]: debug 2022-01-31T21:20:57.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:57.358464+0000) 2022-01-31T21:20:57.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:57 smithi167 conmon[60316]: debug 2022-01-31T21:20:57.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:57.362844+0000) 2022-01-31T21:20:57.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:57 smithi171 conmon[46715]: debug 2022-01-31T21:20:57.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:57.136953+0000) 2022-01-31T21:20:57.798 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:57 smithi171 conmon[51620]: debug 2022-01-31T21:20:57.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:57.715971+0000) 2022-01-31T21:20:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:57 smithi171 conmon[41853]: debug 2022-01-31T21:20:57.840+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:57.840303+0000) 2022-01-31T21:20:58.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:57 smithi167 conmon[49112]: debug 2022-01-31T21:20:57.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:57.688673+0000) 2022-01-31T21:20:58.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:58 smithi167 conmon[54076]: debug 2022-01-31T21:20:58.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.358662+0000) 2022-01-31T21:20:58.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:58 smithi167 conmon[60316]: debug 2022-01-31T21:20:58.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.362998+0000) 2022-01-31T21:20:58.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[46715]: debug 2022-01-31T21:20:58.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.137108+0000) 2022-01-31T21:20:58.798 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[35325]: debug 2022-01-31T21:20:58.722+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122692 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:20:58.799 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[41853]: debug 2022-01-31T21:20:58.709+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.710477+0000) 2022-01-31T21:20:58.800 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[46715]: debug 2022-01-31T21:20:58.710+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.710929+0000) 2022-01-31T21:20:58.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[51620]: debug 2022-01-31T21:20:58.709+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.710285+0000) 2022-01-31T21:20:58.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[51620]: debug 2022-01-31T21:20:58.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.716138+0000) 2022-01-31T21:20:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:58 smithi171 conmon[41853]: debug 2022-01-31T21:20:58.840+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.840493+0000) 2022-01-31T21:20:59.113 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:58 smithi167 conmon[54076]: debug 2022-01-31T21:20:58.709+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.711295+0000) 2022-01-31T21:20:59.114 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:58 smithi167 conmon[60316]: debug 2022-01-31T21:20:58.709+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.711184+0000) 2022-01-31T21:20:59.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:58 smithi167 conmon[49112]: debug 2022-01-31T21:20:58.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.688835+0000) 2022-01-31T21:20:59.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:58 smithi167 conmon[49112]: debug 2022-01-31T21:20:58.709+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:58.711344+0000) 2022-01-31T21:20:59.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:20:59 smithi167 conmon[60316]: debug 2022-01-31T21:20:59.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:59.363177+0000) 2022-01-31T21:20:59.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:20:59 smithi167 conmon[54076]: debug 2022-01-31T21:20:59.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:59.358819+0000) 2022-01-31T21:20:59.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:20:59 smithi171 conmon[46715]: debug 2022-01-31T21:20:59.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:59.137325+0000) 2022-01-31T21:20:59.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:20:59 smithi171 conmon[51620]: debug 2022-01-31T21:20:59.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:59.716304+0000) 2022-01-31T21:21:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:20:59 smithi171 conmon[41853]: debug 2022-01-31T21:20:59.839+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:59.840670+0000) 2022-01-31T21:21:00.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:20:59 smithi167 conmon[49112]: debug 2022-01-31T21:20:59.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:20:59.689029+0000) 2022-01-31T21:21:00.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:00 smithi167 conmon[60316]: debug 2022-01-31T21:21:00.362+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:00.363349+0000) 2022-01-31T21:21:00.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:00 smithi167 conmon[54076]: debug 2022-01-31T21:21:00.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:00.358957+0000) 2022-01-31T21:21:00.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:00 smithi171 conmon[46715]: debug 2022-01-31T21:21:00.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:00.137469+0000) 2022-01-31T21:21:00.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:00 smithi171 conmon[51620]: debug 2022-01-31T21:21:00.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:00.716506+0000) 2022-01-31T21:21:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:00 smithi171 conmon[41853]: debug 2022-01-31T21:21:00.840+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:00.840869+0000) 2022-01-31T21:21:01.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:00 smithi167 conmon[49112]: debug 2022-01-31T21:21:00.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:00.689160+0000) 2022-01-31T21:21:01.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:01 smithi167 conmon[60316]: debug 2022-01-31T21:21:01.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:01.363559+0000) 2022-01-31T21:21:01.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:01 smithi167 conmon[54076]: debug 2022-01-31T21:21:01.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:01.359174+0000) 2022-01-31T21:21:01.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:01 smithi171 conmon[46715]: debug 2022-01-31T21:21:01.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:01.137631+0000) 2022-01-31T21:21:01.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:01 smithi171 conmon[51620]: debug 2022-01-31T21:21:01.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:01.716689+0000) 2022-01-31T21:21:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:01 smithi171 conmon[41853]: debug 2022-01-31T21:21:01.840+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:01.841073+0000) 2022-01-31T21:21:02.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:01 smithi167 conmon[49112]: debug 2022-01-31T21:21:01.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:01.689344+0000) 2022-01-31T21:21:02.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:02 smithi167 conmon[60316]: debug 2022-01-31T21:21:02.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:02.363682+0000) 2022-01-31T21:21:02.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:02 smithi167 conmon[54076]: debug 2022-01-31T21:21:02.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:02.359371+0000) 2022-01-31T21:21:02.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:02 smithi171 conmon[46715]: debug 2022-01-31T21:21:02.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:02.137822+0000) 2022-01-31T21:21:02.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:02 smithi171 conmon[51620]: debug 2022-01-31T21:21:02.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:02.716875+0000) 2022-01-31T21:21:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:02 smithi171 conmon[41853]: debug 2022-01-31T21:21:02.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:02.841263+0000) 2022-01-31T21:21:03.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:02 smithi167 conmon[49112]: debug 2022-01-31T21:21:02.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:02.689540+0000) 2022-01-31T21:21:03.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:03 smithi167 conmon[54076]: debug 2022-01-31T21:21:03.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.359553+0000) 2022-01-31T21:21:03.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:03 smithi167 conmon[60316]: debug 2022-01-31T21:21:03.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.363814+0000) 2022-01-31T21:21:03.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[46715]: debug 2022-01-31T21:21:03.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.137976+0000) 2022-01-31T21:21:03.799 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[35325]: debug 2022-01-31T21:21:03.737+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122802 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:03.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[41853]: debug 2022-01-31T21:21:03.725+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.725788+0000) 2022-01-31T21:21:03.801 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[46715]: debug 2022-01-31T21:21:03.726+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.726747+0000) 2022-01-31T21:21:03.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[51620]: debug 2022-01-31T21:21:03.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.717022+0000) 2022-01-31T21:21:03.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[51620]: debug 2022-01-31T21:21:03.724+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.725525+0000) 2022-01-31T21:21:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:03 smithi171 conmon[41853]: debug 2022-01-31T21:21:03.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.841447+0000) 2022-01-31T21:21:04.114 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:03 smithi167 conmon[54076]: debug 2022-01-31T21:21:03.725+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.726315+0000) 2022-01-31T21:21:04.114 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:03 smithi167 conmon[60316]: debug 2022-01-31T21:21:03.726+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.726541+0000) 2022-01-31T21:21:04.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:03 smithi167 conmon[49112]: debug 2022-01-31T21:21:03.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.689730+0000) 2022-01-31T21:21:04.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:03 smithi167 conmon[49112]: debug 2022-01-31T21:21:03.725+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:03.725983+0000) 2022-01-31T21:21:04.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:04 smithi167 conmon[54076]: debug 2022-01-31T21:21:04.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:04.359669+0000) 2022-01-31T21:21:04.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:04 smithi167 conmon[60316]: debug 2022-01-31T21:21:04.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:04.363986+0000) 2022-01-31T21:21:04.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:04 smithi171 conmon[46715]: debug 2022-01-31T21:21:04.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:04.138162+0000) 2022-01-31T21:21:04.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:04 smithi171 conmon[51620]: debug 2022-01-31T21:21:04.717+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:04.717209+0000) 2022-01-31T21:21:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:04 smithi171 conmon[41853]: debug 2022-01-31T21:21:04.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:04.841621+0000) 2022-01-31T21:21:05.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:04 smithi167 conmon[49112]: debug 2022-01-31T21:21:04.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:04.689922+0000) 2022-01-31T21:21:05.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:05 smithi167 conmon[54076]: debug 2022-01-31T21:21:05.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:05.359806+0000) 2022-01-31T21:21:05.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:05 smithi167 conmon[60316]: debug 2022-01-31T21:21:05.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:05.364159+0000) 2022-01-31T21:21:05.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:05 smithi171 conmon[46715]: debug 2022-01-31T21:21:05.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:05.138315+0000) 2022-01-31T21:21:05.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:05 smithi171 conmon[51620]: debug 2022-01-31T21:21:05.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:05.717381+0000) 2022-01-31T21:21:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:05 smithi171 conmon[41853]: debug 2022-01-31T21:21:05.840+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:05.841819+0000) 2022-01-31T21:21:06.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:05 smithi167 conmon[49112]: debug 2022-01-31T21:21:05.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:05.690092+0000) 2022-01-31T21:21:06.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:06 smithi167 conmon[54076]: debug 2022-01-31T21:21:06.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:06.359989+0000) 2022-01-31T21:21:06.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:06 smithi167 conmon[60316]: debug 2022-01-31T21:21:06.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:06.364300+0000) 2022-01-31T21:21:06.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:06 smithi171 conmon[46715]: debug 2022-01-31T21:21:06.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:06.138489+0000) 2022-01-31T21:21:06.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:06 smithi171 conmon[51620]: debug 2022-01-31T21:21:06.715+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:06.717584+0000) 2022-01-31T21:21:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:06 smithi171 conmon[41853]: debug 2022-01-31T21:21:06.840+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:06.841972+0000) 2022-01-31T21:21:07.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:06 smithi167 conmon[49112]: debug 2022-01-31T21:21:06.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:06.690290+0000) 2022-01-31T21:21:07.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:07 smithi167 conmon[54076]: debug 2022-01-31T21:21:07.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:07.360194+0000) 2022-01-31T21:21:07.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:07 smithi167 conmon[60316]: debug 2022-01-31T21:21:07.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:07.364521+0000) 2022-01-31T21:21:07.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:07 smithi171 conmon[46715]: debug 2022-01-31T21:21:07.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:07.138623+0000) 2022-01-31T21:21:07.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:07 smithi171 conmon[51620]: debug 2022-01-31T21:21:07.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:07.717791+0000) 2022-01-31T21:21:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:07 smithi171 conmon[41853]: debug 2022-01-31T21:21:07.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:07.842176+0000) 2022-01-31T21:21:08.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:07 smithi167 conmon[49112]: debug 2022-01-31T21:21:07.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:07.690535+0000) 2022-01-31T21:21:08.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:08 smithi167 conmon[54076]: debug 2022-01-31T21:21:08.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.360406+0000) 2022-01-31T21:21:08.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:08 smithi167 conmon[60316]: debug 2022-01-31T21:21:08.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.364662+0000) 2022-01-31T21:21:08.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[46715]: debug 2022-01-31T21:21:08.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.138778+0000) 2022-01-31T21:21:08.800 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[41853]: debug 2022-01-31T21:21:08.739+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.741253+0000) 2022-01-31T21:21:08.801 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[46715]: debug 2022-01-31T21:21:08.739+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.741421+0000) 2022-01-31T21:21:08.801 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[35325]: debug 2022-01-31T21:21:08.664+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:21:08.802 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[35325]: debug 2022-01-31T21:21:08.751+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 122912 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:08.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[51620]: debug 2022-01-31T21:21:08.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.717965+0000) 2022-01-31T21:21:08.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[51620]: debug 2022-01-31T21:21:08.739+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.740784+0000) 2022-01-31T21:21:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:08 smithi171 conmon[41853]: debug 2022-01-31T21:21:08.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.842361+0000) 2022-01-31T21:21:09.115 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:08 smithi167 conmon[54076]: debug 2022-01-31T21:21:08.740+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.741359+0000) 2022-01-31T21:21:09.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:08 smithi167 conmon[49112]: debug 2022-01-31T21:21:08.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.690690+0000) 2022-01-31T21:21:09.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:08 smithi167 conmon[49112]: debug 2022-01-31T21:21:08.740+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.741187+0000) 2022-01-31T21:21:09.116 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:08 smithi167 conmon[60316]: debug 2022-01-31T21:21:08.740+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:08.741428+0000) 2022-01-31T21:21:09.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:09 smithi167 conmon[54076]: debug 2022-01-31T21:21:09.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:09.360585+0000) 2022-01-31T21:21:09.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:09 smithi167 conmon[60316]: debug 2022-01-31T21:21:09.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:09.364789+0000) 2022-01-31T21:21:09.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:09 smithi171 conmon[46715]: debug 2022-01-31T21:21:09.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:09.138907+0000) 2022-01-31T21:21:09.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:09 smithi171 conmon[51620]: debug 2022-01-31T21:21:09.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:09.718121+0000) 2022-01-31T21:21:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:09 smithi171 conmon[41853]: debug 2022-01-31T21:21:09.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:09.842516+0000) 2022-01-31T21:21:10.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:09 smithi167 conmon[49112]: debug 2022-01-31T21:21:09.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:09.690897+0000) 2022-01-31T21:21:10.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:10 smithi167 conmon[54076]: debug 2022-01-31T21:21:10.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:10.360776+0000) 2022-01-31T21:21:10.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:10 smithi167 conmon[60316]: debug 2022-01-31T21:21:10.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:10.364980+0000) 2022-01-31T21:21:10.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:10 smithi171 conmon[46715]: debug 2022-01-31T21:21:10.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:10.139024+0000) 2022-01-31T21:21:10.800 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:10 smithi171 conmon[51620]: debug 2022-01-31T21:21:10.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:10.718290+0000) 2022-01-31T21:21:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:10 smithi171 conmon[41853]: debug 2022-01-31T21:21:10.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:10.842699+0000) 2022-01-31T21:21:11.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:10 smithi167 conmon[49112]: debug 2022-01-31T21:21:10.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:10.691088+0000) 2022-01-31T21:21:11.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:11 smithi167 conmon[60316]: debug 2022-01-31T21:21:11.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:11.365185+0000) 2022-01-31T21:21:11.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:11 smithi167 conmon[54076]: debug 2022-01-31T21:21:11.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:11.360928+0000) 2022-01-31T21:21:11.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:11 smithi171 conmon[46715]: debug 2022-01-31T21:21:11.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:11.139183+0000) 2022-01-31T21:21:11.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:11 smithi171 conmon[51620]: debug 2022-01-31T21:21:11.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:11.718462+0000) 2022-01-31T21:21:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:11 smithi171 conmon[41853]: debug 2022-01-31T21:21:11.842+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:11.842923+0000) 2022-01-31T21:21:12.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:11 smithi167 conmon[49112]: debug 2022-01-31T21:21:11.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:11.691290+0000) 2022-01-31T21:21:12.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:12 smithi167 conmon[60316]: debug 2022-01-31T21:21:12.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:12.365374+0000) 2022-01-31T21:21:12.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:12 smithi167 conmon[54076]: debug 2022-01-31T21:21:12.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:12.361115+0000) 2022-01-31T21:21:12.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:12 smithi171 conmon[46715]: debug 2022-01-31T21:21:12.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:12.139354+0000) 2022-01-31T21:21:12.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:12 smithi171 conmon[51620]: debug 2022-01-31T21:21:12.716+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:12.718586+0000) 2022-01-31T21:21:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:12 smithi171 conmon[41853]: debug 2022-01-31T21:21:12.842+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:12.843133+0000) 2022-01-31T21:21:13.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:12 smithi167 conmon[49112]: debug 2022-01-31T21:21:12.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:12.691477+0000) 2022-01-31T21:21:13.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:13 smithi167 conmon[54076]: debug 2022-01-31T21:21:13.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.361307+0000) 2022-01-31T21:21:13.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:13 smithi167 conmon[60316]: debug 2022-01-31T21:21:13.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.365559+0000) 2022-01-31T21:21:13.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[46715]: debug 2022-01-31T21:21:13.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.139545+0000) 2022-01-31T21:21:13.801 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[35325]: debug 2022-01-31T21:21:13.766+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123022 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:13.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[41853]: debug 2022-01-31T21:21:13.754+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.755775+0000) 2022-01-31T21:21:13.803 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[46715]: debug 2022-01-31T21:21:13.754+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.756545+0000) 2022-01-31T21:21:13.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[51620]: debug 2022-01-31T21:21:13.717+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.718775+0000) 2022-01-31T21:21:13.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[51620]: debug 2022-01-31T21:21:13.753+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.755249+0000) 2022-01-31T21:21:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:13 smithi171 conmon[41853]: debug 2022-01-31T21:21:13.841+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.843353+0000) 2022-01-31T21:21:14.116 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:13 smithi167 conmon[54076]: debug 2022-01-31T21:21:13.755+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.756175+0000) 2022-01-31T21:21:14.116 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:13 smithi167 conmon[60316]: debug 2022-01-31T21:21:13.755+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.755897+0000) 2022-01-31T21:21:14.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:13 smithi167 conmon[49112]: debug 2022-01-31T21:21:13.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.691618+0000) 2022-01-31T21:21:14.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:13 smithi167 conmon[49112]: debug 2022-01-31T21:21:13.755+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:13.755726+0000) 2022-01-31T21:21:14.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:14 smithi167 conmon[54076]: debug 2022-01-31T21:21:14.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:14.361458+0000) 2022-01-31T21:21:14.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:14 smithi167 conmon[60316]: debug 2022-01-31T21:21:14.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:14.365719+0000) 2022-01-31T21:21:14.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:14 smithi171 conmon[46715]: debug 2022-01-31T21:21:14.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:14.139768+0000) 2022-01-31T21:21:14.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:14 smithi171 conmon[51620]: debug 2022-01-31T21:21:14.718+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:14.718951+0000) 2022-01-31T21:21:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:14 smithi171 conmon[41853]: debug 2022-01-31T21:21:14.842+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:14.843486+0000) 2022-01-31T21:21:15.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:14 smithi167 conmon[49112]: debug 2022-01-31T21:21:14.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:14.691786+0000) 2022-01-31T21:21:15.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:15 smithi167 conmon[60316]: debug 2022-01-31T21:21:15.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:15.365855+0000) 2022-01-31T21:21:15.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:15 smithi167 conmon[54076]: debug 2022-01-31T21:21:15.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:15.361561+0000) 2022-01-31T21:21:15.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:15 smithi171 conmon[46715]: debug 2022-01-31T21:21:15.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:15.139886+0000) 2022-01-31T21:21:15.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:15 smithi171 conmon[51620]: debug 2022-01-31T21:21:15.718+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:15.719107+0000) 2022-01-31T21:21:15.984 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:15 smithi167 conmon[49112]: debug 2022-01-31T21:21:15.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:15.691976+0000) 2022-01-31T21:21:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:15 smithi171 conmon[41853]: debug 2022-01-31T21:21:15.842+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:15.843640+0000) 2022-01-31T21:21:16.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:16 smithi171 conmon[46715]: debug 2022-01-31T21:21:16.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:16.140070+0000) 2022-01-31T21:21:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:16 smithi167 conmon[54076]: debug 2022-01-31T21:21:16.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:16.361680+0000) 2022-01-31T21:21:16.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:16 smithi167 conmon[60316]: debug 2022-01-31T21:21:16.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:16.366054+0000) 2022-01-31T21:21:16.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:16 smithi171 conmon[51620]: debug 2022-01-31T21:21:16.718+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:16.719298+0000) 2022-01-31T21:21:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:16 smithi171 conmon[41853]: debug 2022-01-31T21:21:16.843+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:16.843839+0000) 2022-01-31T21:21:17.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:16 smithi167 conmon[49112]: debug 2022-01-31T21:21:16.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:16.692171+0000) 2022-01-31T21:21:17.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:17 smithi167 conmon[54076]: debug 2022-01-31T21:21:17.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:17.361855+0000) 2022-01-31T21:21:17.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:17 smithi167 conmon[60316]: debug 2022-01-31T21:21:17.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:17.366263+0000) 2022-01-31T21:21:17.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:17 smithi171 conmon[46715]: debug 2022-01-31T21:21:17.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:17.140260+0000) 2022-01-31T21:21:17.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:17 smithi171 conmon[51620]: debug 2022-01-31T21:21:17.718+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:17.719460+0000) 2022-01-31T21:21:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:17 smithi171 conmon[41853]: debug 2022-01-31T21:21:17.843+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:17.844025+0000) 2022-01-31T21:21:18.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:17 smithi167 conmon[49112]: debug 2022-01-31T21:21:17.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:17.692363+0000) 2022-01-31T21:21:18.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:18 smithi167 conmon[54076]: debug 2022-01-31T21:21:18.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.362036+0000) 2022-01-31T21:21:18.683 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:18 smithi167 conmon[60316]: debug 2022-01-31T21:21:18.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.366402+0000) 2022-01-31T21:21:18.684 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[46715]: debug 2022-01-31T21:21:18.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.140443+0000) 2022-01-31T21:21:18.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[51620]: debug 2022-01-31T21:21:18.718+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.719606+0000) 2022-01-31T21:21:19.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[35325]: debug 2022-01-31T21:21:18.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123132 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:19.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[46715]: debug 2022-01-31T21:21:18.769+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.770739+0000) 2022-01-31T21:21:19.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[51620]: debug 2022-01-31T21:21:18.769+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.770350+0000) 2022-01-31T21:21:19.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[41853]: debug 2022-01-31T21:21:18.770+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.771604+0000) 2022-01-31T21:21:19.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:18 smithi171 conmon[41853]: debug 2022-01-31T21:21:18.843+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.844229+0000) 2022-01-31T21:21:19.117 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:18 smithi167 conmon[54076]: debug 2022-01-31T21:21:18.770+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.770675+0000) 2022-01-31T21:21:19.117 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:18 smithi167 conmon[60316]: debug 2022-01-31T21:21:18.770+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.771067+0000) 2022-01-31T21:21:19.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:18 smithi167 conmon[49112]: debug 2022-01-31T21:21:18.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.692587+0000) 2022-01-31T21:21:19.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:18 smithi167 conmon[49112]: debug 2022-01-31T21:21:18.771+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:18.771494+0000) 2022-01-31T21:21:19.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:19 smithi167 conmon[60316]: debug 2022-01-31T21:21:19.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:19.366576+0000) 2022-01-31T21:21:19.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:19 smithi167 conmon[54076]: debug 2022-01-31T21:21:19.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:19.362262+0000) 2022-01-31T21:21:19.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:19 smithi171 conmon[46715]: debug 2022-01-31T21:21:19.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:19.140640+0000) 2022-01-31T21:21:19.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:19 smithi171 conmon[51620]: debug 2022-01-31T21:21:19.719+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:19.719770+0000) 2022-01-31T21:21:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:19 smithi171 conmon[41853]: debug 2022-01-31T21:21:19.843+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:19.844399+0000) 2022-01-31T21:21:20.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:19 smithi167 conmon[49112]: debug 2022-01-31T21:21:19.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:19.692749+0000) 2022-01-31T21:21:20.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:20 smithi167 conmon[54076]: debug 2022-01-31T21:21:20.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:20.362384+0000) 2022-01-31T21:21:20.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:20 smithi167 conmon[60316]: debug 2022-01-31T21:21:20.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:20.366750+0000) 2022-01-31T21:21:20.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:20 smithi171 conmon[46715]: debug 2022-01-31T21:21:20.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:20.140798+0000) 2022-01-31T21:21:20.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:20 smithi171 conmon[51620]: debug 2022-01-31T21:21:20.719+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:20.719953+0000) 2022-01-31T21:21:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:20 smithi171 conmon[41853]: debug 2022-01-31T21:21:20.843+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:20.844567+0000) 2022-01-31T21:21:21.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:20 smithi167 conmon[49112]: debug 2022-01-31T21:21:20.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:20.692889+0000) 2022-01-31T21:21:21.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:21 smithi167 conmon[54076]: debug 2022-01-31T21:21:21.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:21.362575+0000) 2022-01-31T21:21:21.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:21 smithi167 conmon[60316]: debug 2022-01-31T21:21:21.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:21.366871+0000) 2022-01-31T21:21:21.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:21 smithi171 conmon[46715]: debug 2022-01-31T21:21:21.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:21.141013+0000) 2022-01-31T21:21:21.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:21 smithi171 conmon[51620]: debug 2022-01-31T21:21:21.719+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:21.720130+0000) 2022-01-31T21:21:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:21 smithi171 conmon[41853]: debug 2022-01-31T21:21:21.844+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:21.844723+0000) 2022-01-31T21:21:22.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:21 smithi167 conmon[49112]: debug 2022-01-31T21:21:21.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:21.693055+0000) 2022-01-31T21:21:22.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:22 smithi167 conmon[54076]: debug 2022-01-31T21:21:22.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:22.362762+0000) 2022-01-31T21:21:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:22 smithi167 conmon[60316]: debug 2022-01-31T21:21:22.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:22.367076+0000) 2022-01-31T21:21:22.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:22 smithi171 conmon[46715]: debug 2022-01-31T21:21:22.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:22.141218+0000) 2022-01-31T21:21:22.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:22 smithi171 conmon[51620]: debug 2022-01-31T21:21:22.719+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:22.720301+0000) 2022-01-31T21:21:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:22 smithi171 conmon[41853]: debug 2022-01-31T21:21:22.844+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:22.844995+0000) 2022-01-31T21:21:23.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:22 smithi167 conmon[49112]: debug 2022-01-31T21:21:22.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:22.693228+0000) 2022-01-31T21:21:23.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:23 smithi167 conmon[54076]: debug 2022-01-31T21:21:23.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.362925+0000) 2022-01-31T21:21:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:23 smithi167 conmon[60316]: debug 2022-01-31T21:21:23.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.367287+0000) 2022-01-31T21:21:23.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[46715]: debug 2022-01-31T21:21:23.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.141443+0000) 2022-01-31T21:21:23.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[51620]: debug 2022-01-31T21:21:23.719+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.720483+0000) 2022-01-31T21:21:23.785 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[35325]: debug 2022-01-31T21:21:23.665+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:21:23.785 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[46715]: debug 2022-01-31T21:21:23.784+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.785445+0000) 2022-01-31T21:21:24.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:23 smithi167 conmon[54076]: debug 2022-01-31T21:21:23.785+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.786117+0000) 2022-01-31T21:21:24.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:23 smithi167 conmon[60316]: debug 2022-01-31T21:21:23.786+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.787226+0000) 2022-01-31T21:21:24.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:23 smithi167 conmon[49112]: debug 2022-01-31T21:21:23.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.693406+0000) 2022-01-31T21:21:24.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:23 smithi167 conmon[49112]: debug 2022-01-31T21:21:23.786+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.786652+0000) 2022-01-31T21:21:24.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[51620]: debug 2022-01-31T21:21:23.785+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.785865+0000) 2022-01-31T21:21:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[35325]: debug 2022-01-31T21:21:23.797+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123242 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:24.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[41853]: debug 2022-01-31T21:21:23.786+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.787227+0000) 2022-01-31T21:21:24.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:23 smithi171 conmon[41853]: debug 2022-01-31T21:21:23.844+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:23.845152+0000) 2022-01-31T21:21:24.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:24 smithi167 conmon[54076]: debug 2022-01-31T21:21:24.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:24.363071+0000) 2022-01-31T21:21:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:24 smithi167 conmon[60316]: debug 2022-01-31T21:21:24.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:24.367465+0000) 2022-01-31T21:21:24.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:24 smithi171 conmon[46715]: debug 2022-01-31T21:21:24.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:24.141647+0000) 2022-01-31T21:21:24.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:24 smithi171 conmon[51620]: debug 2022-01-31T21:21:24.719+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:24.720662+0000) 2022-01-31T21:21:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:24 smithi171 conmon[41853]: debug 2022-01-31T21:21:24.844+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:24.845319+0000) 2022-01-31T21:21:25.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:24 smithi167 conmon[49112]: debug 2022-01-31T21:21:24.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:24.693604+0000) 2022-01-31T21:21:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:25 smithi167 conmon[54076]: debug 2022-01-31T21:21:25.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:25.363239+0000) 2022-01-31T21:21:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:25 smithi167 conmon[60316]: debug 2022-01-31T21:21:25.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:25.367524+0000) 2022-01-31T21:21:25.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:25 smithi171 conmon[46715]: debug 2022-01-31T21:21:25.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:25.141799+0000) 2022-01-31T21:21:25.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:25 smithi171 conmon[51620]: debug 2022-01-31T21:21:25.720+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:25.720800+0000) 2022-01-31T21:21:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:25 smithi171 conmon[41853]: debug 2022-01-31T21:21:25.844+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:25.845531+0000) 2022-01-31T21:21:26.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:25 smithi167 conmon[49112]: debug 2022-01-31T21:21:25.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:25.693776+0000) 2022-01-31T21:21:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:26 smithi167 conmon[54076]: debug 2022-01-31T21:21:26.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:26.363451+0000) 2022-01-31T21:21:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:26 smithi167 conmon[60316]: debug 2022-01-31T21:21:26.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:26.367673+0000) 2022-01-31T21:21:26.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:26 smithi171 conmon[46715]: debug 2022-01-31T21:21:26.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:26.141989+0000) 2022-01-31T21:21:26.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:26 smithi171 conmon[51620]: debug 2022-01-31T21:21:26.720+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:26.720919+0000) 2022-01-31T21:21:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:26 smithi171 conmon[41853]: debug 2022-01-31T21:21:26.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:26.845659+0000) 2022-01-31T21:21:27.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:26 smithi167 conmon[49112]: debug 2022-01-31T21:21:26.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:26.693945+0000) 2022-01-31T21:21:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:27 smithi167 conmon[60316]: debug 2022-01-31T21:21:27.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:27.367795+0000) 2022-01-31T21:21:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:27 smithi167 conmon[54076]: debug 2022-01-31T21:21:27.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:27.363610+0000) 2022-01-31T21:21:27.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:27 smithi171 conmon[46715]: debug 2022-01-31T21:21:27.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:27.142146+0000) 2022-01-31T21:21:27.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:27 smithi171 conmon[51620]: debug 2022-01-31T21:21:27.720+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:27.721079+0000) 2022-01-31T21:21:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:27 smithi171 conmon[41853]: debug 2022-01-31T21:21:27.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:27.845829+0000) 2022-01-31T21:21:28.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:27 smithi167 conmon[49112]: debug 2022-01-31T21:21:27.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:27.694170+0000) 2022-01-31T21:21:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:28 smithi167 conmon[54076]: debug 2022-01-31T21:21:28.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.363770+0000) 2022-01-31T21:21:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:28 smithi167 conmon[60316]: debug 2022-01-31T21:21:28.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.367943+0000) 2022-01-31T21:21:28.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[46715]: debug 2022-01-31T21:21:28.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.142337+0000) 2022-01-31T21:21:28.799 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[46715]: debug 2022-01-31T21:21:28.799+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.799964+0000) 2022-01-31T21:21:28.799 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[51620]: debug 2022-01-31T21:21:28.720+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.721280+0000) 2022-01-31T21:21:29.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[35325]: debug 2022-01-31T21:21:28.812+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123352 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[41853]: debug 2022-01-31T21:21:28.800+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.801709+0000) 2022-01-31T21:21:29.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[41853]: debug 2022-01-31T21:21:28.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.845927+0000) 2022-01-31T21:21:29.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:28 smithi171 conmon[51620]: debug 2022-01-31T21:21:28.799+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.800459+0000) 2022-01-31T21:21:29.119 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:28 smithi167 conmon[54076]: debug 2022-01-31T21:21:28.800+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.801604+0000) 2022-01-31T21:21:29.119 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:28 smithi167 conmon[60316]: debug 2022-01-31T21:21:28.799+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.801391+0000) 2022-01-31T21:21:29.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:28 smithi167 conmon[49112]: debug 2022-01-31T21:21:28.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.694381+0000) 2022-01-31T21:21:29.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:28 smithi167 conmon[49112]: debug 2022-01-31T21:21:28.799+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:28.801235+0000) 2022-01-31T21:21:29.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:29 smithi167 conmon[54076]: debug 2022-01-31T21:21:29.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:29.363913+0000) 2022-01-31T21:21:29.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:29 smithi167 conmon[60316]: debug 2022-01-31T21:21:29.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:29.368176+0000) 2022-01-31T21:21:29.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:29 smithi171 conmon[46715]: debug 2022-01-31T21:21:29.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:29.142549+0000) 2022-01-31T21:21:29.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:29 smithi171 conmon[51620]: debug 2022-01-31T21:21:29.720+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:29.721464+0000) 2022-01-31T21:21:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:29 smithi171 conmon[41853]: debug 2022-01-31T21:21:29.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:29.846056+0000) 2022-01-31T21:21:30.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:29 smithi167 conmon[49112]: debug 2022-01-31T21:21:29.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:29.694574+0000) 2022-01-31T21:21:30.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:30 smithi167 conmon[54076]: debug 2022-01-31T21:21:30.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:30.364008+0000) 2022-01-31T21:21:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:30 smithi167 conmon[60316]: debug 2022-01-31T21:21:30.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:30.368411+0000) 2022-01-31T21:21:30.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:30 smithi171 conmon[46715]: debug 2022-01-31T21:21:30.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:30.142720+0000) 2022-01-31T21:21:30.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:30 smithi171 conmon[51620]: debug 2022-01-31T21:21:30.720+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:30.721640+0000) 2022-01-31T21:21:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:30 smithi171 conmon[41853]: debug 2022-01-31T21:21:30.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:30.846187+0000) 2022-01-31T21:21:31.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:30 smithi167 conmon[49112]: debug 2022-01-31T21:21:30.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:30.694703+0000) 2022-01-31T21:21:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:31 smithi167 conmon[60316]: debug 2022-01-31T21:21:31.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:31.368772+0000) 2022-01-31T21:21:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:31 smithi167 conmon[54076]: debug 2022-01-31T21:21:31.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:31.364197+0000) 2022-01-31T21:21:31.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:31 smithi171 conmon[46715]: debug 2022-01-31T21:21:31.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:31.142925+0000) 2022-01-31T21:21:31.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:31 smithi171 conmon[51620]: debug 2022-01-31T21:21:31.721+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:31.721820+0000) 2022-01-31T21:21:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:31 smithi171 conmon[41853]: debug 2022-01-31T21:21:31.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:31.846334+0000) 2022-01-31T21:21:32.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:31 smithi167 conmon[49112]: debug 2022-01-31T21:21:31.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:31.694827+0000) 2022-01-31T21:21:32.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:32 smithi167 conmon[54076]: debug 2022-01-31T21:21:32.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:32.364418+0000) 2022-01-31T21:21:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:32 smithi167 conmon[60316]: debug 2022-01-31T21:21:32.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:32.369534+0000) 2022-01-31T21:21:32.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:32 smithi171 conmon[46715]: debug 2022-01-31T21:21:32.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:32.143099+0000) 2022-01-31T21:21:32.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:32 smithi171 conmon[51620]: debug 2022-01-31T21:21:32.721+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:32.721964+0000) 2022-01-31T21:21:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:32 smithi171 conmon[41853]: debug 2022-01-31T21:21:32.846+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:32.846523+0000) 2022-01-31T21:21:33.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:32 smithi167 conmon[49112]: debug 2022-01-31T21:21:32.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:32.694918+0000) 2022-01-31T21:21:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:33 smithi167 conmon[60316]: debug 2022-01-31T21:21:33.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.369716+0000) 2022-01-31T21:21:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:33 smithi167 conmon[54076]: debug 2022-01-31T21:21:33.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.364620+0000) 2022-01-31T21:21:33.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[46715]: debug 2022-01-31T21:21:33.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.143378+0000) 2022-01-31T21:21:33.804 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[51620]: debug 2022-01-31T21:21:33.721+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.722107+0000) 2022-01-31T21:21:34.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[35325]: debug 2022-01-31T21:21:33.827+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123462 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:34.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[46715]: debug 2022-01-31T21:21:33.815+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.816048+0000) 2022-01-31T21:21:34.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[51620]: debug 2022-01-31T21:21:33.814+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.815080+0000) 2022-01-31T21:21:34.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[41853]: debug 2022-01-31T21:21:33.816+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.816911+0000) 2022-01-31T21:21:34.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:33 smithi171 conmon[41853]: debug 2022-01-31T21:21:33.845+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.846704+0000) 2022-01-31T21:21:34.119 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:33 smithi167 conmon[54076]: debug 2022-01-31T21:21:33.814+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.816232+0000) 2022-01-31T21:21:34.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:33 smithi167 conmon[60316]: debug 2022-01-31T21:21:33.814+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.815778+0000) 2022-01-31T21:21:34.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:33 smithi167 conmon[49112]: debug 2022-01-31T21:21:33.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.695162+0000) 2022-01-31T21:21:34.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:33 smithi167 conmon[49112]: debug 2022-01-31T21:21:33.814+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:33.816185+0000) 2022-01-31T21:21:34.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:34 smithi167 conmon[54076]: debug 2022-01-31T21:21:34.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:34.364741+0000) 2022-01-31T21:21:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:34 smithi167 conmon[60316]: debug 2022-01-31T21:21:34.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:34.369857+0000) 2022-01-31T21:21:34.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:34 smithi171 conmon[46715]: debug 2022-01-31T21:21:34.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:34.143583+0000) 2022-01-31T21:21:34.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:34 smithi171 conmon[51620]: debug 2022-01-31T21:21:34.721+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:34.722272+0000) 2022-01-31T21:21:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:34 smithi171 conmon[41853]: debug 2022-01-31T21:21:34.846+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:34.846873+0000) 2022-01-31T21:21:35.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:34 smithi167 conmon[49112]: debug 2022-01-31T21:21:34.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:34.695278+0000) 2022-01-31T21:21:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:35 smithi167 conmon[60316]: debug 2022-01-31T21:21:35.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:35.370001+0000) 2022-01-31T21:21:35.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:35 smithi167 conmon[54076]: debug 2022-01-31T21:21:35.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:35.364896+0000) 2022-01-31T21:21:35.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:35 smithi171 conmon[46715]: debug 2022-01-31T21:21:35.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:35.143723+0000) 2022-01-31T21:21:35.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:35 smithi171 conmon[51620]: debug 2022-01-31T21:21:35.722+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:35.722416+0000) 2022-01-31T21:21:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:35 smithi171 conmon[41853]: debug 2022-01-31T21:21:35.846+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:35.847068+0000) 2022-01-31T21:21:36.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:35 smithi167 conmon[49112]: debug 2022-01-31T21:21:35.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:35.695407+0000) 2022-01-31T21:21:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:36 smithi167 conmon[54076]: debug 2022-01-31T21:21:36.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:36.365112+0000) 2022-01-31T21:21:36.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:36 smithi167 conmon[60316]: debug 2022-01-31T21:21:36.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:36.370185+0000) 2022-01-31T21:21:36.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:36 smithi171 conmon[46715]: debug 2022-01-31T21:21:36.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:36.143869+0000) 2022-01-31T21:21:36.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:36 smithi171 conmon[51620]: debug 2022-01-31T21:21:36.722+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:36.722594+0000) 2022-01-31T21:21:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:36 smithi171 conmon[41853]: debug 2022-01-31T21:21:36.846+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:36.847257+0000) 2022-01-31T21:21:37.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:36 smithi167 conmon[49112]: debug 2022-01-31T21:21:36.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:36.695611+0000) 2022-01-31T21:21:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:37 smithi167 conmon[54076]: debug 2022-01-31T21:21:37.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:37.365271+0000) 2022-01-31T21:21:37.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:37 smithi167 conmon[60316]: debug 2022-01-31T21:21:37.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:37.370398+0000) 2022-01-31T21:21:37.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:37 smithi171 conmon[46715]: debug 2022-01-31T21:21:37.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:37.144023+0000) 2022-01-31T21:21:37.805 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:37 smithi171 conmon[51620]: debug 2022-01-31T21:21:37.721+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:37.722731+0000) 2022-01-31T21:21:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:37 smithi171 conmon[41853]: debug 2022-01-31T21:21:37.846+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:37.847442+0000) 2022-01-31T21:21:38.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:37 smithi167 conmon[49112]: debug 2022-01-31T21:21:37.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:37.695781+0000) 2022-01-31T21:21:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:38 smithi167 conmon[60316]: debug 2022-01-31T21:21:38.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.370593+0000) 2022-01-31T21:21:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:38 smithi167 conmon[54076]: debug 2022-01-31T21:21:38.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.365474+0000) 2022-01-31T21:21:38.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[46715]: debug 2022-01-31T21:21:38.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.144195+0000) 2022-01-31T21:21:38.805 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[35325]: debug 2022-01-31T21:21:38.666+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:21:38.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[51620]: debug 2022-01-31T21:21:38.722+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.722908+0000) 2022-01-31T21:21:39.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[35325]: debug 2022-01-31T21:21:38.842+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:39.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[46715]: debug 2022-01-31T21:21:38.829+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.830565+0000) 2022-01-31T21:21:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[51620]: debug 2022-01-31T21:21:38.829+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.830310+0000) 2022-01-31T21:21:39.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[41853]: debug 2022-01-31T21:21:38.831+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.831902+0000) 2022-01-31T21:21:39.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:38 smithi171 conmon[41853]: debug 2022-01-31T21:21:38.846+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.847581+0000) 2022-01-31T21:21:39.120 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:38 smithi167 conmon[54076]: debug 2022-01-31T21:21:38.828+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.830107+0000) 2022-01-31T21:21:39.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:38 smithi167 conmon[60316]: debug 2022-01-31T21:21:38.830+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.831607+0000) 2022-01-31T21:21:39.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:38 smithi167 conmon[49112]: debug 2022-01-31T21:21:38.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.696007+0000) 2022-01-31T21:21:39.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:38 smithi167 conmon[49112]: debug 2022-01-31T21:21:38.828+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:38.830471+0000) 2022-01-31T21:21:39.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:39 smithi167 conmon[54076]: debug 2022-01-31T21:21:39.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:39.365679+0000) 2022-01-31T21:21:39.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:39 smithi167 conmon[60316]: debug 2022-01-31T21:21:39.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:39.370742+0000) 2022-01-31T21:21:39.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:39 smithi171 conmon[46715]: debug 2022-01-31T21:21:39.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:39.144385+0000) 2022-01-31T21:21:39.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:39 smithi171 conmon[51620]: debug 2022-01-31T21:21:39.722+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:39.723084+0000) 2022-01-31T21:21:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:39 smithi171 conmon[41853]: debug 2022-01-31T21:21:39.847+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:39.847726+0000) 2022-01-31T21:21:40.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:39 smithi167 conmon[49112]: debug 2022-01-31T21:21:39.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:39.696217+0000) 2022-01-31T21:21:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:40 smithi167 conmon[60316]: debug 2022-01-31T21:21:40.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:40.370889+0000) 2022-01-31T21:21:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:40 smithi167 conmon[54076]: debug 2022-01-31T21:21:40.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:40.365804+0000) 2022-01-31T21:21:40.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:40 smithi171 conmon[46715]: debug 2022-01-31T21:21:40.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:40.144561+0000) 2022-01-31T21:21:40.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:40 smithi171 conmon[51620]: debug 2022-01-31T21:21:40.722+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:40.723239+0000) 2022-01-31T21:21:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:40 smithi171 conmon[41853]: debug 2022-01-31T21:21:40.847+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:40.847902+0000) 2022-01-31T21:21:41.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:40 smithi167 conmon[49112]: debug 2022-01-31T21:21:40.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:40.696390+0000) 2022-01-31T21:21:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:41 smithi167 conmon[60316]: debug 2022-01-31T21:21:41.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:41.371027+0000) 2022-01-31T21:21:41.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:41 smithi167 conmon[54076]: debug 2022-01-31T21:21:41.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:41.365954+0000) 2022-01-31T21:21:41.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:41 smithi171 conmon[46715]: debug 2022-01-31T21:21:41.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:41.144727+0000) 2022-01-31T21:21:41.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:41 smithi171 conmon[51620]: debug 2022-01-31T21:21:41.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:41.723393+0000) 2022-01-31T21:21:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:41 smithi171 conmon[41853]: debug 2022-01-31T21:21:41.847+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:41.848056+0000) 2022-01-31T21:21:42.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:41 smithi167 conmon[49112]: debug 2022-01-31T21:21:41.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:41.696610+0000) 2022-01-31T21:21:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:42 smithi167 conmon[60316]: debug 2022-01-31T21:21:42.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:42.371207+0000) 2022-01-31T21:21:42.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:42 smithi167 conmon[54076]: debug 2022-01-31T21:21:42.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:42.366112+0000) 2022-01-31T21:21:42.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:42 smithi171 conmon[46715]: debug 2022-01-31T21:21:42.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:42.144874+0000) 2022-01-31T21:21:42.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:42 smithi171 conmon[51620]: debug 2022-01-31T21:21:42.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:42.723537+0000) 2022-01-31T21:21:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:42 smithi171 conmon[41853]: debug 2022-01-31T21:21:42.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:42.848270+0000) 2022-01-31T21:21:43.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:42 smithi167 conmon[49112]: debug 2022-01-31T21:21:42.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:42.696823+0000) 2022-01-31T21:21:43.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:43 smithi167 conmon[54076]: debug 2022-01-31T21:21:43.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.366334+0000) 2022-01-31T21:21:43.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:43 smithi167 conmon[60316]: debug 2022-01-31T21:21:43.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.371417+0000) 2022-01-31T21:21:43.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[46715]: debug 2022-01-31T21:21:43.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.144982+0000) 2022-01-31T21:21:43.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[51620]: debug 2022-01-31T21:21:43.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.723718+0000) 2022-01-31T21:21:44.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[35325]: debug 2022-01-31T21:21:43.858+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123682 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[41853]: debug 2022-01-31T21:21:43.846+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.846776+0000) 2022-01-31T21:21:44.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[41853]: debug 2022-01-31T21:21:43.847+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.848369+0000) 2022-01-31T21:21:44.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[46715]: debug 2022-01-31T21:21:43.845+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.846721+0000) 2022-01-31T21:21:44.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:43 smithi171 conmon[51620]: debug 2022-01-31T21:21:43.844+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.845142+0000) 2022-01-31T21:21:44.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:43 smithi167 conmon[54076]: debug 2022-01-31T21:21:43.844+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.846390+0000) 2022-01-31T21:21:44.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:43 smithi167 conmon[60316]: debug 2022-01-31T21:21:43.844+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.846079+0000) 2022-01-31T21:21:44.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:43 smithi167 conmon[49112]: debug 2022-01-31T21:21:43.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.697007+0000) 2022-01-31T21:21:44.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:43 smithi167 conmon[49112]: debug 2022-01-31T21:21:43.844+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:43.846427+0000) 2022-01-31T21:21:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:44 smithi167 conmon[60316]: debug 2022-01-31T21:21:44.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:44.371624+0000) 2022-01-31T21:21:44.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:44 smithi167 conmon[54076]: debug 2022-01-31T21:21:44.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:44.366545+0000) 2022-01-31T21:21:44.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:44 smithi171 conmon[46715]: debug 2022-01-31T21:21:44.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:44.145155+0000) 2022-01-31T21:21:44.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:44 smithi171 conmon[51620]: debug 2022-01-31T21:21:44.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:44.723880+0000) 2022-01-31T21:21:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:44 smithi171 conmon[41853]: debug 2022-01-31T21:21:44.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:44.848516+0000) 2022-01-31T21:21:45.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:44 smithi167 conmon[49112]: debug 2022-01-31T21:21:44.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:44.697188+0000) 2022-01-31T21:21:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:45 smithi167 conmon[54076]: debug 2022-01-31T21:21:45.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:45.366674+0000) 2022-01-31T21:21:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:45 smithi167 conmon[60316]: debug 2022-01-31T21:21:45.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:45.371735+0000) 2022-01-31T21:21:45.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:45 smithi171 conmon[46715]: debug 2022-01-31T21:21:45.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:45.145304+0000) 2022-01-31T21:21:45.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:45 smithi171 conmon[51620]: debug 2022-01-31T21:21:45.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:45.723976+0000) 2022-01-31T21:21:45.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:45 smithi167 conmon[49112]: debug 2022-01-31T21:21:45.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:45.697375+0000) 2022-01-31T21:21:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:45 smithi171 conmon[41853]: debug 2022-01-31T21:21:45.847+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:45.848617+0000) 2022-01-31T21:21:46.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:46 smithi171 conmon[46715]: debug 2022-01-31T21:21:46.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:46.145447+0000) 2022-01-31T21:21:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:46 smithi167 conmon[54076]: debug 2022-01-31T21:21:46.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:46.366839+0000) 2022-01-31T21:21:46.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:46 smithi167 conmon[60316]: debug 2022-01-31T21:21:46.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:46.371889+0000) 2022-01-31T21:21:46.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:46 smithi171 conmon[51620]: debug 2022-01-31T21:21:46.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:46.724129+0000) 2022-01-31T21:21:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:46 smithi171 conmon[41853]: debug 2022-01-31T21:21:46.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:46.848799+0000) 2022-01-31T21:21:47.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:46 smithi167 conmon[49112]: debug 2022-01-31T21:21:46.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:46.697609+0000) 2022-01-31T21:21:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:47 smithi167 conmon[54076]: debug 2022-01-31T21:21:47.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:47.366996+0000) 2022-01-31T21:21:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:47 smithi167 conmon[60316]: debug 2022-01-31T21:21:47.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:47.372087+0000) 2022-01-31T21:21:47.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:47 smithi171 conmon[46715]: debug 2022-01-31T21:21:47.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:47.145627+0000) 2022-01-31T21:21:47.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:47 smithi171 conmon[51620]: debug 2022-01-31T21:21:47.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:47.724307+0000) 2022-01-31T21:21:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:47 smithi171 conmon[41853]: debug 2022-01-31T21:21:47.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:47.849009+0000) 2022-01-31T21:21:48.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:47 smithi167 conmon[49112]: debug 2022-01-31T21:21:47.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:47.697769+0000) 2022-01-31T21:21:48.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:48 smithi167 conmon[54076]: debug 2022-01-31T21:21:48.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.367179+0000) 2022-01-31T21:21:48.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:48 smithi167 conmon[60316]: debug 2022-01-31T21:21:48.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.372256+0000) 2022-01-31T21:21:48.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[46715]: debug 2022-01-31T21:21:48.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.145814+0000) 2022-01-31T21:21:48.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[51620]: debug 2022-01-31T21:21:48.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.724533+0000) 2022-01-31T21:21:49.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[35325]: debug 2022-01-31T21:21:48.873+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123792 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:49.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[46715]: debug 2022-01-31T21:21:48.860+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.861331+0000) 2022-01-31T21:21:49.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[41853]: debug 2022-01-31T21:21:48.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.849190+0000) 2022-01-31T21:21:49.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[41853]: debug 2022-01-31T21:21:48.860+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.861571+0000) 2022-01-31T21:21:49.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:48 smithi171 conmon[51620]: debug 2022-01-31T21:21:48.860+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.861588+0000) 2022-01-31T21:21:49.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:48 smithi167 conmon[54076]: debug 2022-01-31T21:21:48.861+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.862832+0000) 2022-01-31T21:21:49.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:48 smithi167 conmon[60316]: debug 2022-01-31T21:21:48.860+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.862407+0000) 2022-01-31T21:21:49.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:48 smithi167 conmon[49112]: debug 2022-01-31T21:21:48.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.697981+0000) 2022-01-31T21:21:49.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:48 smithi167 conmon[49112]: debug 2022-01-31T21:21:48.860+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:48.862127+0000) 2022-01-31T21:21:49.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:49 smithi167 conmon[54076]: debug 2022-01-31T21:21:49.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:49.367347+0000) 2022-01-31T21:21:49.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:49 smithi167 conmon[60316]: debug 2022-01-31T21:21:49.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:49.372430+0000) 2022-01-31T21:21:49.658 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:49 smithi171 conmon[46715]: debug 2022-01-31T21:21:49.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:49.146038+0000) 2022-01-31T21:21:49.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:49 smithi171 conmon[51620]: debug 2022-01-31T21:21:49.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:49.724708+0000) 2022-01-31T21:21:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:49 smithi171 conmon[41853]: debug 2022-01-31T21:21:49.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:49.849372+0000) 2022-01-31T21:21:50.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:49 smithi167 conmon[49112]: debug 2022-01-31T21:21:49.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:49.698176+0000) 2022-01-31T21:21:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:50 smithi167 conmon[54076]: debug 2022-01-31T21:21:50.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:50.367494+0000) 2022-01-31T21:21:50.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:50 smithi167 conmon[60316]: debug 2022-01-31T21:21:50.371+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:50.372589+0000) 2022-01-31T21:21:50.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:50 smithi171 conmon[46715]: debug 2022-01-31T21:21:50.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:50.146191+0000) 2022-01-31T21:21:50.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:50 smithi171 conmon[51620]: debug 2022-01-31T21:21:50.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:50.724887+0000) 2022-01-31T21:21:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:50 smithi171 conmon[41853]: debug 2022-01-31T21:21:50.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:50.849525+0000) 2022-01-31T21:21:51.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:50 smithi167 conmon[49112]: debug 2022-01-31T21:21:50.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:50.698384+0000) 2022-01-31T21:21:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:51 smithi167 conmon[54076]: debug 2022-01-31T21:21:51.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:51.367670+0000) 2022-01-31T21:21:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:51 smithi167 conmon[60316]: debug 2022-01-31T21:21:51.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:51.372693+0000) 2022-01-31T21:21:51.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:51 smithi171 conmon[46715]: debug 2022-01-31T21:21:51.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:51.146372+0000) 2022-01-31T21:21:51.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:51 smithi171 conmon[51620]: debug 2022-01-31T21:21:51.723+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:51.725012+0000) 2022-01-31T21:21:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:51 smithi171 conmon[41853]: debug 2022-01-31T21:21:51.847+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:51.849726+0000) 2022-01-31T21:21:52.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:51 smithi167 conmon[49112]: debug 2022-01-31T21:21:51.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:51.698562+0000) 2022-01-31T21:21:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:52 smithi167 conmon[54076]: debug 2022-01-31T21:21:52.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:52.367818+0000) 2022-01-31T21:21:52.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:52 smithi167 conmon[60316]: debug 2022-01-31T21:21:52.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:52.372848+0000) 2022-01-31T21:21:52.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:52 smithi171 conmon[46715]: debug 2022-01-31T21:21:52.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:52.146503+0000) 2022-01-31T21:21:52.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:52 smithi171 conmon[51620]: debug 2022-01-31T21:21:52.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:52.725139+0000) 2022-01-31T21:21:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:52 smithi171 conmon[41853]: debug 2022-01-31T21:21:52.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:52.849905+0000) 2022-01-31T21:21:53.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:52 smithi167 conmon[49112]: debug 2022-01-31T21:21:52.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:52.698754+0000) 2022-01-31T21:21:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:53 smithi167 conmon[54076]: debug 2022-01-31T21:21:53.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.368012+0000) 2022-01-31T21:21:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:53 smithi167 conmon[60316]: debug 2022-01-31T21:21:53.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.373018+0000) 2022-01-31T21:21:53.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[46715]: debug 2022-01-31T21:21:53.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.146608+0000) 2022-01-31T21:21:53.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[51620]: debug 2022-01-31T21:21:53.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.725265+0000) 2022-01-31T21:21:53.808 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[35325]: debug 2022-01-31T21:21:53.666+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:21:54.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[46715]: debug 2022-01-31T21:21:53.874+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.876272+0000) 2022-01-31T21:21:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[51620]: debug 2022-01-31T21:21:53.874+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.876469+0000) 2022-01-31T21:21:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[35325]: debug 2022-01-31T21:21:53.902+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 123903 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:54.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[41853]: debug 2022-01-31T21:21:53.848+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.850082+0000) 2022-01-31T21:21:54.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:53 smithi171 conmon[41853]: debug 2022-01-31T21:21:53.874+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.876366+0000) 2022-01-31T21:21:54.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:53 smithi167 conmon[49112]: debug 2022-01-31T21:21:53.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.698968+0000) 2022-01-31T21:21:54.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:53 smithi167 conmon[49112]: debug 2022-01-31T21:21:53.876+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.877478+0000) 2022-01-31T21:21:54.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:53 smithi167 conmon[54076]: debug 2022-01-31T21:21:53.875+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.876237+0000) 2022-01-31T21:21:54.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:53 smithi167 conmon[60316]: debug 2022-01-31T21:21:53.877+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:53.877915+0000) 2022-01-31T21:21:54.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:54 smithi167 conmon[54076]: debug 2022-01-31T21:21:54.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:54.368116+0000) 2022-01-31T21:21:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:54 smithi167 conmon[60316]: debug 2022-01-31T21:21:54.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:54.373212+0000) 2022-01-31T21:21:54.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:54 smithi171 conmon[46715]: debug 2022-01-31T21:21:54.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:54.146802+0000) 2022-01-31T21:21:54.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:54 smithi171 conmon[51620]: debug 2022-01-31T21:21:54.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:54.725441+0000) 2022-01-31T21:21:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:54 smithi171 conmon[41853]: debug 2022-01-31T21:21:54.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:54.850248+0000) 2022-01-31T21:21:55.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:54 smithi167 conmon[49112]: debug 2022-01-31T21:21:54.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:54.699214+0000) 2022-01-31T21:21:55.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:55 smithi167 conmon[54076]: debug 2022-01-31T21:21:55.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:55.368256+0000) 2022-01-31T21:21:55.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:55 smithi167 conmon[60316]: debug 2022-01-31T21:21:55.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:55.373395+0000) 2022-01-31T21:21:55.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:55 smithi171 conmon[46715]: debug 2022-01-31T21:21:55.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:55.146930+0000) 2022-01-31T21:21:55.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:55 smithi171 conmon[51620]: debug 2022-01-31T21:21:55.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:55.725590+0000) 2022-01-31T21:21:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:55 smithi171 conmon[41853]: debug 2022-01-31T21:21:55.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:55.850382+0000) 2022-01-31T21:21:56.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:55 smithi167 conmon[49112]: debug 2022-01-31T21:21:55.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:55.699376+0000) 2022-01-31T21:21:56.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:56 smithi167 conmon[54076]: debug 2022-01-31T21:21:56.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:56.368462+0000) 2022-01-31T21:21:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:56 smithi167 conmon[60316]: debug 2022-01-31T21:21:56.373+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:56.373552+0000) 2022-01-31T21:21:56.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:56 smithi171 conmon[46715]: debug 2022-01-31T21:21:56.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:56.147082+0000) 2022-01-31T21:21:56.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:56 smithi171 conmon[51620]: debug 2022-01-31T21:21:56.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:56.725786+0000) 2022-01-31T21:21:57.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:56 smithi171 conmon[41853]: debug 2022-01-31T21:21:56.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:56.850578+0000) 2022-01-31T21:21:57.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:56 smithi167 conmon[49112]: debug 2022-01-31T21:21:56.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:56.699559+0000) 2022-01-31T21:21:57.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:57 smithi167 conmon[54076]: debug 2022-01-31T21:21:57.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:57.368665+0000) 2022-01-31T21:21:57.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:57 smithi167 conmon[60316]: debug 2022-01-31T21:21:57.373+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:57.373685+0000) 2022-01-31T21:21:57.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:57 smithi171 conmon[46715]: debug 2022-01-31T21:21:57.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:57.147243+0000) 2022-01-31T21:21:57.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:57 smithi171 conmon[51620]: debug 2022-01-31T21:21:57.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:57.725921+0000) 2022-01-31T21:21:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:57 smithi171 conmon[41853]: debug 2022-01-31T21:21:57.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:57.850790+0000) 2022-01-31T21:21:58.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:57 smithi167 conmon[49112]: debug 2022-01-31T21:21:57.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:57.699723+0000) 2022-01-31T21:21:58.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:58 smithi167 conmon[54076]: debug 2022-01-31T21:21:58.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.368827+0000) 2022-01-31T21:21:58.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:58 smithi167 conmon[60316]: debug 2022-01-31T21:21:58.373+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.373828+0000) 2022-01-31T21:21:58.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[46715]: debug 2022-01-31T21:21:58.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.147430+0000) 2022-01-31T21:21:58.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[51620]: debug 2022-01-31T21:21:58.725+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.726093+0000) 2022-01-31T21:21:59.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[35325]: debug 2022-01-31T21:21:58.918+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124013 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:21:59.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[46715]: debug 2022-01-31T21:21:58.905+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.907036+0000) 2022-01-31T21:21:59.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[51620]: debug 2022-01-31T21:21:58.904+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.906252+0000) 2022-01-31T21:21:59.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[41853]: debug 2022-01-31T21:21:58.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.850967+0000) 2022-01-31T21:21:59.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:58 smithi171 conmon[41853]: debug 2022-01-31T21:21:58.906+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.908033+0000) 2022-01-31T21:21:59.124 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:58 smithi167 conmon[54076]: debug 2022-01-31T21:21:58.907+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.907575+0000) 2022-01-31T21:21:59.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:58 smithi167 conmon[60316]: debug 2022-01-31T21:21:58.907+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.908034+0000) 2022-01-31T21:21:59.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:58 smithi167 conmon[49112]: debug 2022-01-31T21:21:58.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.699912+0000) 2022-01-31T21:21:59.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:58 smithi167 conmon[49112]: debug 2022-01-31T21:21:58.906+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:58.907266+0000) 2022-01-31T21:21:59.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:21:59 smithi167 conmon[54076]: debug 2022-01-31T21:21:59.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:59.369032+0000) 2022-01-31T21:21:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:21:59 smithi167 conmon[60316]: debug 2022-01-31T21:21:59.373+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:59.373985+0000) 2022-01-31T21:21:59.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:21:59 smithi171 conmon[46715]: debug 2022-01-31T21:21:59.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:59.147644+0000) 2022-01-31T21:21:59.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:21:59 smithi171 conmon[51620]: debug 2022-01-31T21:21:59.724+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:59.726271+0000) 2022-01-31T21:22:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:21:59 smithi171 conmon[41853]: debug 2022-01-31T21:21:59.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:59.851161+0000) 2022-01-31T21:22:00.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:21:59 smithi167 conmon[49112]: debug 2022-01-31T21:21:59.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:21:59.700147+0000) 2022-01-31T21:22:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:00 smithi167 conmon[54076]: debug 2022-01-31T21:22:00.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:00.369191+0000) 2022-01-31T21:22:00.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:00 smithi167 conmon[60316]: debug 2022-01-31T21:22:00.373+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:00.374191+0000) 2022-01-31T21:22:00.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:00 smithi171 conmon[46715]: debug 2022-01-31T21:22:00.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:00.147773+0000) 2022-01-31T21:22:00.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:00 smithi171 conmon[51620]: debug 2022-01-31T21:22:00.725+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:00.726427+0000) 2022-01-31T21:22:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:00 smithi171 conmon[41853]: debug 2022-01-31T21:22:00.849+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:00.851340+0000) 2022-01-31T21:22:01.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:00 smithi167 conmon[49112]: debug 2022-01-31T21:22:00.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:00.700336+0000) 2022-01-31T21:22:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:01 smithi167 conmon[54076]: debug 2022-01-31T21:22:01.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:01.369388+0000) 2022-01-31T21:22:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:01 smithi167 conmon[60316]: debug 2022-01-31T21:22:01.373+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:01.374401+0000) 2022-01-31T21:22:01.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:01 smithi171 conmon[46715]: debug 2022-01-31T21:22:01.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:01.147897+0000) 2022-01-31T21:22:01.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:01 smithi171 conmon[51620]: debug 2022-01-31T21:22:01.725+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:01.726584+0000) 2022-01-31T21:22:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:01 smithi171 conmon[41853]: debug 2022-01-31T21:22:01.850+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:01.851535+0000) 2022-01-31T21:22:02.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:01 smithi167 conmon[49112]: debug 2022-01-31T21:22:01.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:01.700585+0000) 2022-01-31T21:22:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:02 smithi167 conmon[54076]: debug 2022-01-31T21:22:02.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:02.369574+0000) 2022-01-31T21:22:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:02 smithi167 conmon[60316]: debug 2022-01-31T21:22:02.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:02.374559+0000) 2022-01-31T21:22:02.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:02 smithi171 conmon[46715]: debug 2022-01-31T21:22:02.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:02.148083+0000) 2022-01-31T21:22:02.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:02 smithi171 conmon[51620]: debug 2022-01-31T21:22:02.725+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:02.726774+0000) 2022-01-31T21:22:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:02 smithi171 conmon[41853]: debug 2022-01-31T21:22:02.850+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:02.851734+0000) 2022-01-31T21:22:03.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:02 smithi167 conmon[49112]: debug 2022-01-31T21:22:02.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:02.700812+0000) 2022-01-31T21:22:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:03 smithi167 conmon[54076]: debug 2022-01-31T21:22:03.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.369780+0000) 2022-01-31T21:22:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:03 smithi167 conmon[60316]: debug 2022-01-31T21:22:03.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.374713+0000) 2022-01-31T21:22:03.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[46715]: debug 2022-01-31T21:22:03.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.148223+0000) 2022-01-31T21:22:03.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[51620]: debug 2022-01-31T21:22:03.726+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.726962+0000) 2022-01-31T21:22:04.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[35325]: debug 2022-01-31T21:22:03.934+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124124 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:04.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[46715]: debug 2022-01-31T21:22:03.921+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.922608+0000) 2022-01-31T21:22:04.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[51620]: debug 2022-01-31T21:22:03.920+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.921550+0000) 2022-01-31T21:22:04.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[41853]: debug 2022-01-31T21:22:03.851+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.851864+0000) 2022-01-31T21:22:04.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:03 smithi171 conmon[41853]: debug 2022-01-31T21:22:03.921+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.922491+0000) 2022-01-31T21:22:04.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:03 smithi167 conmon[49112]: debug 2022-01-31T21:22:03.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.701014+0000) 2022-01-31T21:22:04.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:03 smithi167 conmon[49112]: debug 2022-01-31T21:22:03.922+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.923210+0000) 2022-01-31T21:22:04.126 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:03 smithi167 conmon[54076]: debug 2022-01-31T21:22:03.921+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.922431+0000) 2022-01-31T21:22:04.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:03 smithi167 conmon[60316]: debug 2022-01-31T21:22:03.922+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:03.922979+0000) 2022-01-31T21:22:04.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:04 smithi167 conmon[54076]: debug 2022-01-31T21:22:04.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:04.369999+0000) 2022-01-31T21:22:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:04 smithi167 conmon[60316]: debug 2022-01-31T21:22:04.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:04.374867+0000) 2022-01-31T21:22:04.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:04 smithi171 conmon[46715]: debug 2022-01-31T21:22:04.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:04.148405+0000) 2022-01-31T21:22:04.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:04 smithi171 conmon[51620]: debug 2022-01-31T21:22:04.726+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:04.727121+0000) 2022-01-31T21:22:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:04 smithi171 conmon[41853]: debug 2022-01-31T21:22:04.851+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:04.852030+0000) 2022-01-31T21:22:05.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:04 smithi167 conmon[49112]: debug 2022-01-31T21:22:04.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:04.701232+0000) 2022-01-31T21:22:05.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:05 smithi167 conmon[54076]: debug 2022-01-31T21:22:05.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:05.370154+0000) 2022-01-31T21:22:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:05 smithi167 conmon[60316]: debug 2022-01-31T21:22:05.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:05.375041+0000) 2022-01-31T21:22:05.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:05 smithi171 conmon[46715]: debug 2022-01-31T21:22:05.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:05.148567+0000) 2022-01-31T21:22:05.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:05 smithi171 conmon[51620]: debug 2022-01-31T21:22:05.726+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:05.727212+0000) 2022-01-31T21:22:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:05 smithi171 conmon[41853]: debug 2022-01-31T21:22:05.851+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:05.852174+0000) 2022-01-31T21:22:06.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:05 smithi167 conmon[49112]: debug 2022-01-31T21:22:05.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:05.701419+0000) 2022-01-31T21:22:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:06 smithi167 conmon[60316]: debug 2022-01-31T21:22:06.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:06.375240+0000) 2022-01-31T21:22:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:06 smithi167 conmon[54076]: debug 2022-01-31T21:22:06.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:06.370341+0000) 2022-01-31T21:22:06.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:06 smithi171 conmon[46715]: debug 2022-01-31T21:22:06.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:06.148787+0000) 2022-01-31T21:22:06.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:06 smithi171 conmon[51620]: debug 2022-01-31T21:22:06.726+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:06.727366+0000) 2022-01-31T21:22:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:06 smithi171 conmon[41853]: debug 2022-01-31T21:22:06.851+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:06.852347+0000) 2022-01-31T21:22:07.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:06 smithi167 conmon[49112]: debug 2022-01-31T21:22:06.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:06.701622+0000) 2022-01-31T21:22:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:07 smithi167 conmon[54076]: debug 2022-01-31T21:22:07.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:07.370565+0000) 2022-01-31T21:22:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:07 smithi167 conmon[60316]: debug 2022-01-31T21:22:07.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:07.375439+0000) 2022-01-31T21:22:07.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:07 smithi171 conmon[46715]: debug 2022-01-31T21:22:07.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:07.148964+0000) 2022-01-31T21:22:07.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:07 smithi171 conmon[51620]: debug 2022-01-31T21:22:07.726+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:07.727562+0000) 2022-01-31T21:22:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:07 smithi171 conmon[41853]: debug 2022-01-31T21:22:07.851+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:07.852552+0000) 2022-01-31T21:22:08.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:07 smithi167 conmon[49112]: debug 2022-01-31T21:22:07.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:07.701813+0000) 2022-01-31T21:22:08.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:08 smithi167 conmon[54076]: debug 2022-01-31T21:22:08.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.370754+0000) 2022-01-31T21:22:08.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:08 smithi167 conmon[60316]: debug 2022-01-31T21:22:08.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.375607+0000) 2022-01-31T21:22:08.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[46715]: debug 2022-01-31T21:22:08.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.149158+0000) 2022-01-31T21:22:08.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[51620]: debug 2022-01-31T21:22:08.727+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.727788+0000) 2022-01-31T21:22:08.811 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[35325]: debug 2022-01-31T21:22:08.667+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:22:08.940 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:08 smithi167 conmon[49112]: debug 2022-01-31T21:22:08.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.701976+0000) 2022-01-31T21:22:08.941 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:08 smithi167 conmon[49112]: debug 2022-01-31T21:22:08.938+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.939394+0000) 2022-01-31T21:22:08.941 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:08 smithi167 conmon[54076]: debug 2022-01-31T21:22:08.937+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.937528+0000) 2022-01-31T21:22:08.942 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:08 smithi167 conmon[60316]: debug 2022-01-31T21:22:08.938+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.938611+0000) 2022-01-31T21:22:09.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[35325]: debug 2022-01-31T21:22:08.949+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124233 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:09.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[46715]: debug 2022-01-31T21:22:08.937+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.937893+0000) 2022-01-31T21:22:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[51620]: debug 2022-01-31T21:22:08.936+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.937207+0000) 2022-01-31T21:22:09.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[41853]: debug 2022-01-31T21:22:08.852+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.852770+0000) 2022-01-31T21:22:09.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:08 smithi171 conmon[41853]: debug 2022-01-31T21:22:08.936+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:08.937387+0000) 2022-01-31T21:22:09.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:09 smithi171 conmon[46715]: debug 2022-01-31T21:22:09.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:09.149397+0000) 2022-01-31T21:22:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:09 smithi167 conmon[54076]: debug 2022-01-31T21:22:09.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:09.370996+0000) 2022-01-31T21:22:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:09 smithi167 conmon[60316]: debug 2022-01-31T21:22:09.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:09.375776+0000) 2022-01-31T21:22:09.810 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:09 smithi171 conmon[51620]: debug 2022-01-31T21:22:09.727+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:09.727939+0000) 2022-01-31T21:22:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:09 smithi171 conmon[41853]: debug 2022-01-31T21:22:09.852+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:09.852976+0000) 2022-01-31T21:22:10.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:09 smithi167 conmon[49112]: debug 2022-01-31T21:22:09.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:09.702120+0000) 2022-01-31T21:22:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:10 smithi167 conmon[54076]: debug 2022-01-31T21:22:10.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:10.371166+0000) 2022-01-31T21:22:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:10 smithi167 conmon[60316]: debug 2022-01-31T21:22:10.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:10.375930+0000) 2022-01-31T21:22:10.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:10 smithi171 conmon[46715]: debug 2022-01-31T21:22:10.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:10.149549+0000) 2022-01-31T21:22:10.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:10 smithi171 conmon[51620]: debug 2022-01-31T21:22:10.727+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:10.728101+0000) 2022-01-31T21:22:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:10 smithi171 conmon[41853]: debug 2022-01-31T21:22:10.852+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:10.853113+0000) 2022-01-31T21:22:11.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:10 smithi167 conmon[49112]: debug 2022-01-31T21:22:10.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:10.702259+0000) 2022-01-31T21:22:11.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:11 smithi167 conmon[54076]: debug 2022-01-31T21:22:11.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:11.371385+0000) 2022-01-31T21:22:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:11 smithi167 conmon[60316]: debug 2022-01-31T21:22:11.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:11.376096+0000) 2022-01-31T21:22:11.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:11 smithi171 conmon[46715]: debug 2022-01-31T21:22:11.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:11.149730+0000) 2022-01-31T21:22:11.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:11 smithi171 conmon[51620]: debug 2022-01-31T21:22:11.727+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:11.728282+0000) 2022-01-31T21:22:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:11 smithi171 conmon[41853]: debug 2022-01-31T21:22:11.852+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:11.853317+0000) 2022-01-31T21:22:12.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:11 smithi167 conmon[49112]: debug 2022-01-31T21:22:11.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:11.702457+0000) 2022-01-31T21:22:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:12 smithi167 conmon[54076]: debug 2022-01-31T21:22:12.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:12.371609+0000) 2022-01-31T21:22:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:12 smithi167 conmon[60316]: debug 2022-01-31T21:22:12.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:12.376267+0000) 2022-01-31T21:22:12.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:12 smithi171 conmon[46715]: debug 2022-01-31T21:22:12.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:12.149886+0000) 2022-01-31T21:22:12.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:12 smithi171 conmon[51620]: debug 2022-01-31T21:22:12.727+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:12.728458+0000) 2022-01-31T21:22:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:12 smithi171 conmon[41853]: debug 2022-01-31T21:22:12.852+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:12.853513+0000) 2022-01-31T21:22:13.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:12 smithi167 conmon[49112]: debug 2022-01-31T21:22:12.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:12.702642+0000) 2022-01-31T21:22:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:13 smithi167 conmon[54076]: debug 2022-01-31T21:22:13.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.371757+0000) 2022-01-31T21:22:13.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:13 smithi167 conmon[60316]: debug 2022-01-31T21:22:13.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.376421+0000) 2022-01-31T21:22:13.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[46715]: debug 2022-01-31T21:22:13.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.149964+0000) 2022-01-31T21:22:13.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[51620]: debug 2022-01-31T21:22:13.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.728610+0000) 2022-01-31T21:22:13.951 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:13 smithi167 conmon[49112]: debug 2022-01-31T21:22:13.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.702809+0000) 2022-01-31T21:22:14.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[51620]: debug 2022-01-31T21:22:13.951+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.952267+0000) 2022-01-31T21:22:14.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[35325]: debug 2022-01-31T21:22:13.964+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124344 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:14.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[41853]: debug 2022-01-31T21:22:13.853+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.853697+0000) 2022-01-31T21:22:14.109 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[41853]: debug 2022-01-31T21:22:13.952+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.953073+0000) 2022-01-31T21:22:14.109 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:13 smithi171 conmon[46715]: debug 2022-01-31T21:22:13.952+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.952776+0000) 2022-01-31T21:22:14.297 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:13 smithi167 conmon[49112]: debug 2022-01-31T21:22:13.952+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.953420+0000) 2022-01-31T21:22:14.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:13 smithi167 conmon[54076]: debug 2022-01-31T21:22:13.952+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.953275+0000) 2022-01-31T21:22:14.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:13 smithi167 conmon[60316]: debug 2022-01-31T21:22:13.953+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:13.953522+0000) 2022-01-31T21:22:14.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:14 smithi171 conmon[46715]: debug 2022-01-31T21:22:14.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:14.150132+0000) 2022-01-31T21:22:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:14 smithi167 conmon[54076]: debug 2022-01-31T21:22:14.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:14.371969+0000) 2022-01-31T21:22:14.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:14 smithi167 conmon[60316]: debug 2022-01-31T21:22:14.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:14.376588+0000) 2022-01-31T21:22:14.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:14 smithi171 conmon[51620]: debug 2022-01-31T21:22:14.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:14.728734+0000) 2022-01-31T21:22:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:14 smithi171 conmon[41853]: debug 2022-01-31T21:22:14.853+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:14.853824+0000) 2022-01-31T21:22:15.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:14 smithi167 conmon[49112]: debug 2022-01-31T21:22:14.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:14.702968+0000) 2022-01-31T21:22:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:15 smithi167 conmon[54076]: debug 2022-01-31T21:22:15.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:15.372144+0000) 2022-01-31T21:22:15.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:15 smithi167 conmon[60316]: debug 2022-01-31T21:22:15.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:15.376708+0000) 2022-01-31T21:22:15.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:15 smithi171 conmon[46715]: debug 2022-01-31T21:22:15.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:15.150293+0000) 2022-01-31T21:22:15.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:15 smithi171 conmon[51620]: debug 2022-01-31T21:22:15.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:15.728909+0000) 2022-01-31T21:22:15.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:15 smithi167 conmon[49112]: debug 2022-01-31T21:22:15.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:15.703107+0000) 2022-01-31T21:22:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:15 smithi171 conmon[41853]: debug 2022-01-31T21:22:15.853+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:15.853952+0000) 2022-01-31T21:22:16.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:16 smithi171 conmon[46715]: debug 2022-01-31T21:22:16.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:16.150471+0000) 2022-01-31T21:22:16.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:16 smithi167 conmon[60316]: debug 2022-01-31T21:22:16.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:16.376903+0000) 2022-01-31T21:22:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:16 smithi167 conmon[54076]: debug 2022-01-31T21:22:16.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:16.372274+0000) 2022-01-31T21:22:16.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:16 smithi171 conmon[51620]: debug 2022-01-31T21:22:16.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:16.729114+0000) 2022-01-31T21:22:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:16 smithi171 conmon[41853]: debug 2022-01-31T21:22:16.853+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:16.854133+0000) 2022-01-31T21:22:17.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:16 smithi167 conmon[49112]: debug 2022-01-31T21:22:16.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:16.703327+0000) 2022-01-31T21:22:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:17 smithi167 conmon[60316]: debug 2022-01-31T21:22:17.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:17.377096+0000) 2022-01-31T21:22:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:17 smithi167 conmon[54076]: debug 2022-01-31T21:22:17.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:17.372434+0000) 2022-01-31T21:22:17.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:17 smithi171 conmon[46715]: debug 2022-01-31T21:22:17.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:17.150656+0000) 2022-01-31T21:22:17.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:17 smithi171 conmon[51620]: debug 2022-01-31T21:22:17.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:17.729340+0000) 2022-01-31T21:22:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:17 smithi171 conmon[41853]: debug 2022-01-31T21:22:17.853+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:17.854334+0000) 2022-01-31T21:22:18.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:17 smithi167 conmon[49112]: debug 2022-01-31T21:22:17.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:17.703572+0000) 2022-01-31T21:22:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:18 smithi167 conmon[54076]: debug 2022-01-31T21:22:18.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.372562+0000) 2022-01-31T21:22:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:18 smithi167 conmon[60316]: debug 2022-01-31T21:22:18.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.377304+0000) 2022-01-31T21:22:18.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[46715]: debug 2022-01-31T21:22:18.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.150825+0000) 2022-01-31T21:22:18.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[51620]: debug 2022-01-31T21:22:18.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.729461+0000) 2022-01-31T21:22:18.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:18 smithi167 conmon[49112]: debug 2022-01-31T21:22:18.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.703782+0000) 2022-01-31T21:22:19.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[35325]: debug 2022-01-31T21:22:18.980+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124453 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:19.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[46715]: debug 2022-01-31T21:22:18.966+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.967635+0000) 2022-01-31T21:22:19.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[51620]: debug 2022-01-31T21:22:18.966+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.966931+0000) 2022-01-31T21:22:19.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[41853]: debug 2022-01-31T21:22:18.853+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.854517+0000) 2022-01-31T21:22:19.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:18 smithi171 conmon[41853]: debug 2022-01-31T21:22:18.966+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.967528+0000) 2022-01-31T21:22:19.305 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:18 smithi167 conmon[49112]: debug 2022-01-31T21:22:18.967+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.968700+0000) 2022-01-31T21:22:19.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:18 smithi167 conmon[54076]: debug 2022-01-31T21:22:18.966+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.968144+0000) 2022-01-31T21:22:19.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:18 smithi167 conmon[60316]: debug 2022-01-31T21:22:18.967+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:18.968824+0000) 2022-01-31T21:22:19.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:19 smithi171 conmon[46715]: debug 2022-01-31T21:22:19.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:19.151003+0000) 2022-01-31T21:22:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:19 smithi167 conmon[54076]: debug 2022-01-31T21:22:19.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:19.372738+0000) 2022-01-31T21:22:19.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:19 smithi167 conmon[60316]: debug 2022-01-31T21:22:19.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:19.377545+0000) 2022-01-31T21:22:19.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:19 smithi171 conmon[51620]: debug 2022-01-31T21:22:19.728+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:19.729654+0000) 2022-01-31T21:22:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:19 smithi171 conmon[41853]: debug 2022-01-31T21:22:19.854+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:19.854716+0000) 2022-01-31T21:22:20.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:19 smithi167 conmon[49112]: debug 2022-01-31T21:22:19.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:19.704011+0000) 2022-01-31T21:22:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:20 smithi167 conmon[60316]: debug 2022-01-31T21:22:20.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:20.377729+0000) 2022-01-31T21:22:20.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:20 smithi167 conmon[54076]: debug 2022-01-31T21:22:20.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:20.372895+0000) 2022-01-31T21:22:20.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:20 smithi171 conmon[46715]: debug 2022-01-31T21:22:20.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:20.151156+0000) 2022-01-31T21:22:20.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:20 smithi171 conmon[51620]: debug 2022-01-31T21:22:20.729+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:20.729833+0000) 2022-01-31T21:22:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:20 smithi171 conmon[41853]: debug 2022-01-31T21:22:20.854+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:20.854867+0000) 2022-01-31T21:22:21.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:20 smithi167 conmon[49112]: debug 2022-01-31T21:22:20.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:20.704168+0000) 2022-01-31T21:22:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:21 smithi167 conmon[60316]: debug 2022-01-31T21:22:21.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:21.377897+0000) 2022-01-31T21:22:21.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:21 smithi167 conmon[54076]: debug 2022-01-31T21:22:21.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:21.373091+0000) 2022-01-31T21:22:21.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:21 smithi171 conmon[46715]: debug 2022-01-31T21:22:21.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:21.151348+0000) 2022-01-31T21:22:21.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:21 smithi171 conmon[51620]: debug 2022-01-31T21:22:21.729+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:21.730039+0000) 2022-01-31T21:22:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:21 smithi171 conmon[41853]: debug 2022-01-31T21:22:21.854+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:21.855054+0000) 2022-01-31T21:22:22.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:21 smithi167 conmon[49112]: debug 2022-01-31T21:22:21.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:21.704380+0000) 2022-01-31T21:22:22.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:22 smithi167 conmon[54076]: debug 2022-01-31T21:22:22.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:22.373279+0000) 2022-01-31T21:22:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:22 smithi167 conmon[60316]: debug 2022-01-31T21:22:22.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:22.378082+0000) 2022-01-31T21:22:22.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:22 smithi171 conmon[46715]: debug 2022-01-31T21:22:22.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:22.151506+0000) 2022-01-31T21:22:22.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:22 smithi171 conmon[51620]: debug 2022-01-31T21:22:22.729+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:22.730230+0000) 2022-01-31T21:22:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:22 smithi171 conmon[41853]: debug 2022-01-31T21:22:22.854+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:22.855230+0000) 2022-01-31T21:22:23.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:22 smithi167 conmon[49112]: debug 2022-01-31T21:22:22.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:22.704581+0000) 2022-01-31T21:22:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:23 smithi167 conmon[60316]: debug 2022-01-31T21:22:23.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.378291+0000) 2022-01-31T21:22:23.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:23 smithi167 conmon[54076]: debug 2022-01-31T21:22:23.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.373480+0000) 2022-01-31T21:22:23.465 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[46715]: debug 2022-01-31T21:22:23.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.151605+0000) 2022-01-31T21:22:23.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[51620]: debug 2022-01-31T21:22:23.730+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.730386+0000) 2022-01-31T21:22:23.814 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[35325]: debug 2022-01-31T21:22:23.668+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:22:23.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:23 smithi167 conmon[49112]: debug 2022-01-31T21:22:23.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.704770+0000) 2022-01-31T21:22:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[35325]: debug 2022-01-31T21:22:23.995+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124570 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:24.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[46715]: debug 2022-01-31T21:22:23.983+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.983944+0000) 2022-01-31T21:22:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[51620]: debug 2022-01-31T21:22:23.982+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.983230+0000) 2022-01-31T21:22:24.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[41853]: debug 2022-01-31T21:22:23.854+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.855410+0000) 2022-01-31T21:22:24.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:23 smithi171 conmon[41853]: debug 2022-01-31T21:22:23.982+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.983046+0000) 2022-01-31T21:22:24.314 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:23 smithi167 conmon[49112]: debug 2022-01-31T21:22:23.982+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.984384+0000) 2022-01-31T21:22:24.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:23 smithi167 conmon[54076]: debug 2022-01-31T21:22:23.981+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.982702+0000) 2022-01-31T21:22:24.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:23 smithi167 conmon[60316]: debug 2022-01-31T21:22:23.982+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:23.983897+0000) 2022-01-31T21:22:24.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:24 smithi171 conmon[46715]: debug 2022-01-31T21:22:24.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:24.151793+0000) 2022-01-31T21:22:24.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:24 smithi167 conmon[60316]: debug 2022-01-31T21:22:24.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:24.378495+0000) 2022-01-31T21:22:24.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:24 smithi167 conmon[54076]: debug 2022-01-31T21:22:24.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:24.373673+0000) 2022-01-31T21:22:24.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:24 smithi171 conmon[51620]: debug 2022-01-31T21:22:24.729+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:24.730596+0000) 2022-01-31T21:22:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:24 smithi171 conmon[41853]: debug 2022-01-31T21:22:24.855+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:24.855602+0000) 2022-01-31T21:22:25.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:24 smithi167 conmon[49112]: debug 2022-01-31T21:22:24.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:24.704983+0000) 2022-01-31T21:22:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:25 smithi167 conmon[54076]: debug 2022-01-31T21:22:25.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:25.373828+0000) 2022-01-31T21:22:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:25 smithi167 conmon[60316]: debug 2022-01-31T21:22:25.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:25.378610+0000) 2022-01-31T21:22:25.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:25 smithi171 conmon[46715]: debug 2022-01-31T21:22:25.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:25.151889+0000) 2022-01-31T21:22:25.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:25 smithi171 conmon[51620]: debug 2022-01-31T21:22:25.730+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:25.730746+0000) 2022-01-31T21:22:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:25 smithi171 conmon[41853]: debug 2022-01-31T21:22:25.855+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:25.855788+0000) 2022-01-31T21:22:26.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:25 smithi167 conmon[49112]: debug 2022-01-31T21:22:25.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:25.705174+0000) 2022-01-31T21:22:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:26 smithi167 conmon[54076]: debug 2022-01-31T21:22:26.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:26.374057+0000) 2022-01-31T21:22:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:26 smithi167 conmon[60316]: debug 2022-01-31T21:22:26.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:26.378717+0000) 2022-01-31T21:22:26.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:26 smithi171 conmon[46715]: debug 2022-01-31T21:22:26.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:26.152024+0000) 2022-01-31T21:22:26.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:26 smithi171 conmon[51620]: debug 2022-01-31T21:22:26.730+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:26.730880+0000) 2022-01-31T21:22:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:26 smithi171 conmon[41853]: debug 2022-01-31T21:22:26.855+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:26.855937+0000) 2022-01-31T21:22:27.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:26 smithi167 conmon[49112]: debug 2022-01-31T21:22:26.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:26.705419+0000) 2022-01-31T21:22:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:27 smithi167 conmon[54076]: debug 2022-01-31T21:22:27.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:27.374247+0000) 2022-01-31T21:22:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:27 smithi167 conmon[60316]: debug 2022-01-31T21:22:27.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:27.378835+0000) 2022-01-31T21:22:27.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:27 smithi171 conmon[46715]: debug 2022-01-31T21:22:27.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:27.152186+0000) 2022-01-31T21:22:27.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:27 smithi171 conmon[51620]: debug 2022-01-31T21:22:27.730+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:27.731050+0000) 2022-01-31T21:22:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:27 smithi171 conmon[41853]: debug 2022-01-31T21:22:27.855+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:27.856089+0000) 2022-01-31T21:22:28.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:27 smithi167 conmon[49112]: debug 2022-01-31T21:22:27.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:27.705592+0000) 2022-01-31T21:22:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:28 smithi167 conmon[54076]: debug 2022-01-31T21:22:28.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.374449+0000) 2022-01-31T21:22:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:28 smithi167 conmon[60316]: debug 2022-01-31T21:22:28.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.379006+0000) 2022-01-31T21:22:28.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:28 smithi171 conmon[46715]: debug 2022-01-31T21:22:28.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.152365+0000) 2022-01-31T21:22:28.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:28 smithi171 conmon[51620]: debug 2022-01-31T21:22:28.730+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.731194+0000) 2022-01-31T21:22:28.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:28 smithi167 conmon[49112]: debug 2022-01-31T21:22:28.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.705750+0000) 2022-01-31T21:22:29.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:29 smithi171 conmon[35325]: debug 2022-01-31T21:22:29.010+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124679 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:29.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:28 smithi171 conmon[46715]: debug 2022-01-31T21:22:28.999+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.999796+0000) 2022-01-31T21:22:29.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:28 smithi171 conmon[51620]: debug 2022-01-31T21:22:28.998+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.999554+0000) 2022-01-31T21:22:29.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:28 smithi171 conmon[41853]: debug 2022-01-31T21:22:28.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.856285+0000) 2022-01-31T21:22:29.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:28 smithi171 conmon[41853]: debug 2022-01-31T21:22:28.997+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.998225+0000) 2022-01-31T21:22:29.322 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:28 smithi167 conmon[49112]: debug 2022-01-31T21:22:28.998+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.999576+0000) 2022-01-31T21:22:29.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:28 smithi167 conmon[54076]: debug 2022-01-31T21:22:28.996+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.997986+0000) 2022-01-31T21:22:29.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:28 smithi167 conmon[60316]: debug 2022-01-31T21:22:28.997+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:28.999182+0000) 2022-01-31T21:22:29.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:29 smithi171 conmon[46715]: debug 2022-01-31T21:22:29.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:29.152569+0000) 2022-01-31T21:22:29.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:29 smithi167 conmon[60316]: debug 2022-01-31T21:22:29.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:29.379176+0000) 2022-01-31T21:22:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:29 smithi167 conmon[54076]: debug 2022-01-31T21:22:29.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:29.374608+0000) 2022-01-31T21:22:29.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:29 smithi171 conmon[51620]: debug 2022-01-31T21:22:29.730+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:29.731349+0000) 2022-01-31T21:22:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:29 smithi171 conmon[41853]: debug 2022-01-31T21:22:29.855+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:29.856484+0000) 2022-01-31T21:22:30.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:29 smithi167 conmon[49112]: debug 2022-01-31T21:22:29.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:29.705933+0000) 2022-01-31T21:22:30.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:30 smithi167 conmon[60316]: debug 2022-01-31T21:22:30.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:30.379397+0000) 2022-01-31T21:22:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:30 smithi167 conmon[54076]: debug 2022-01-31T21:22:30.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:30.374755+0000) 2022-01-31T21:22:30.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:30 smithi171 conmon[46715]: debug 2022-01-31T21:22:30.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:30.152775+0000) 2022-01-31T21:22:30.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:30 smithi171 conmon[51620]: debug 2022-01-31T21:22:30.731+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:30.731502+0000) 2022-01-31T21:22:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:30 smithi171 conmon[41853]: debug 2022-01-31T21:22:30.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:30.856599+0000) 2022-01-31T21:22:31.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:30 smithi167 conmon[49112]: debug 2022-01-31T21:22:30.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:30.706108+0000) 2022-01-31T21:22:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:31 smithi167 conmon[60316]: debug 2022-01-31T21:22:31.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:31.379586+0000) 2022-01-31T21:22:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:31 smithi167 conmon[54076]: debug 2022-01-31T21:22:31.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:31.374968+0000) 2022-01-31T21:22:31.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:31 smithi171 conmon[46715]: debug 2022-01-31T21:22:31.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:31.152931+0000) 2022-01-31T21:22:31.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:31 smithi171 conmon[51620]: debug 2022-01-31T21:22:31.731+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:31.731707+0000) 2022-01-31T21:22:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:31 smithi171 conmon[41853]: debug 2022-01-31T21:22:31.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:31.856729+0000) 2022-01-31T21:22:32.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:31 smithi167 conmon[49112]: debug 2022-01-31T21:22:31.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:31.706352+0000) 2022-01-31T21:22:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:32 smithi167 conmon[60316]: debug 2022-01-31T21:22:32.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:32.379724+0000) 2022-01-31T21:22:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:32 smithi167 conmon[54076]: debug 2022-01-31T21:22:32.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:32.375175+0000) 2022-01-31T21:22:32.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:32 smithi171 conmon[46715]: debug 2022-01-31T21:22:32.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:32.153147+0000) 2022-01-31T21:22:32.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:32 smithi171 conmon[51620]: debug 2022-01-31T21:22:32.731+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:32.731967+0000) 2022-01-31T21:22:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:32 smithi171 conmon[41853]: debug 2022-01-31T21:22:32.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:32.856896+0000) 2022-01-31T21:22:33.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:32 smithi167 conmon[49112]: debug 2022-01-31T21:22:32.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:32.706571+0000) 2022-01-31T21:22:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:33 smithi167 conmon[60316]: debug 2022-01-31T21:22:33.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:33.379847+0000) 2022-01-31T21:22:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:33 smithi167 conmon[54076]: debug 2022-01-31T21:22:33.373+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:33.375382+0000) 2022-01-31T21:22:33.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:33 smithi171 conmon[46715]: debug 2022-01-31T21:22:33.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:33.153331+0000) 2022-01-31T21:22:33.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:33 smithi171 conmon[51620]: debug 2022-01-31T21:22:33.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:33.732142+0000) 2022-01-31T21:22:34.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:33 smithi167 conmon[49112]: debug 2022-01-31T21:22:33.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:33.706721+0000) 2022-01-31T21:22:34.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[35325]: debug 2022-01-31T21:22:34.025+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124786 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:34.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[46715]: debug 2022-01-31T21:22:34.014+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.015493+0000) 2022-01-31T21:22:34.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[51620]: debug 2022-01-31T21:22:34.013+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.013759+0000) 2022-01-31T21:22:34.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:33 smithi171 conmon[41853]: debug 2022-01-31T21:22:33.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:33.857056+0000) 2022-01-31T21:22:34.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[41853]: debug 2022-01-31T21:22:34.012+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.013448+0000) 2022-01-31T21:22:34.330 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:34 smithi167 conmon[49112]: debug 2022-01-31T21:22:34.012+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.013703+0000) 2022-01-31T21:22:34.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:34 smithi167 conmon[54076]: debug 2022-01-31T21:22:34.013+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.014697+0000) 2022-01-31T21:22:34.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:34 smithi167 conmon[60316]: debug 2022-01-31T21:22:34.013+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.014835+0000) 2022-01-31T21:22:34.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[46715]: debug 2022-01-31T21:22:34.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.153526+0000) 2022-01-31T21:22:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:34 smithi167 conmon[54076]: debug 2022-01-31T21:22:34.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.375570+0000) 2022-01-31T21:22:34.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:34 smithi167 conmon[60316]: debug 2022-01-31T21:22:34.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.379994+0000) 2022-01-31T21:22:34.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[51620]: debug 2022-01-31T21:22:34.731+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.732332+0000) 2022-01-31T21:22:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:34 smithi171 conmon[41853]: debug 2022-01-31T21:22:34.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.857234+0000) 2022-01-31T21:22:35.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:34 smithi167 conmon[49112]: debug 2022-01-31T21:22:34.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:34.706891+0000) 2022-01-31T21:22:35.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:35 smithi167 conmon[54076]: debug 2022-01-31T21:22:35.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:35.375681+0000) 2022-01-31T21:22:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:35 smithi167 conmon[60316]: debug 2022-01-31T21:22:35.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:35.380164+0000) 2022-01-31T21:22:35.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:35 smithi171 conmon[46715]: debug 2022-01-31T21:22:35.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:35.153705+0000) 2022-01-31T21:22:35.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:35 smithi171 conmon[51620]: debug 2022-01-31T21:22:35.731+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:35.732487+0000) 2022-01-31T21:22:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:35 smithi171 conmon[41853]: debug 2022-01-31T21:22:35.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:35.857377+0000) 2022-01-31T21:22:36.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:35 smithi167 conmon[49112]: debug 2022-01-31T21:22:35.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:35.707027+0000) 2022-01-31T21:22:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:36 smithi167 conmon[54076]: debug 2022-01-31T21:22:36.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:36.375839+0000) 2022-01-31T21:22:36.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:36 smithi167 conmon[60316]: debug 2022-01-31T21:22:36.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:36.380340+0000) 2022-01-31T21:22:36.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:36 smithi171 conmon[46715]: debug 2022-01-31T21:22:36.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:36.153916+0000) 2022-01-31T21:22:36.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:36 smithi171 conmon[51620]: debug 2022-01-31T21:22:36.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:36.732666+0000) 2022-01-31T21:22:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:36 smithi171 conmon[41853]: debug 2022-01-31T21:22:36.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:36.857560+0000) 2022-01-31T21:22:37.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:36 smithi167 conmon[49112]: debug 2022-01-31T21:22:36.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:36.707229+0000) 2022-01-31T21:22:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:37 smithi167 conmon[54076]: debug 2022-01-31T21:22:37.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:37.375997+0000) 2022-01-31T21:22:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:37 smithi167 conmon[60316]: debug 2022-01-31T21:22:37.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:37.380540+0000) 2022-01-31T21:22:37.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:37 smithi171 conmon[46715]: debug 2022-01-31T21:22:37.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:37.154075+0000) 2022-01-31T21:22:37.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:37 smithi171 conmon[51620]: debug 2022-01-31T21:22:37.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:37.732847+0000) 2022-01-31T21:22:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:37 smithi171 conmon[41853]: debug 2022-01-31T21:22:37.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:37.857741+0000) 2022-01-31T21:22:38.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:37 smithi167 conmon[49112]: debug 2022-01-31T21:22:37.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:37.707450+0000) 2022-01-31T21:22:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:38 smithi167 conmon[54076]: debug 2022-01-31T21:22:38.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:38.376199+0000) 2022-01-31T21:22:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:38 smithi167 conmon[60316]: debug 2022-01-31T21:22:38.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:38.380727+0000) 2022-01-31T21:22:38.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:38 smithi171 conmon[46715]: debug 2022-01-31T21:22:38.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:38.154262+0000) 2022-01-31T21:22:38.816 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:38 smithi171 conmon[35325]: debug 2022-01-31T21:22:38.668+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:22:38.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:38 smithi171 conmon[51620]: debug 2022-01-31T21:22:38.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:38.733048+0000) 2022-01-31T21:22:39.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:38 smithi167 conmon[49112]: debug 2022-01-31T21:22:38.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:38.707628+0000) 2022-01-31T21:22:39.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[35325]: debug 2022-01-31T21:22:39.040+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 124895 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:39.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[46715]: debug 2022-01-31T21:22:39.027+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.029682+0000) 2022-01-31T21:22:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[51620]: debug 2022-01-31T21:22:39.026+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.028722+0000) 2022-01-31T21:22:39.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:38 smithi171 conmon[41853]: debug 2022-01-31T21:22:38.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:38.857909+0000) 2022-01-31T21:22:39.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[41853]: debug 2022-01-31T21:22:39.026+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.028335+0000) 2022-01-31T21:22:39.337 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:39 smithi167 conmon[49112]: debug 2022-01-31T21:22:39.028+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.029910+0000) 2022-01-31T21:22:39.338 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:39 smithi167 conmon[54076]: debug 2022-01-31T21:22:39.027+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.028878+0000) 2022-01-31T21:22:39.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:39 smithi167 conmon[60316]: debug 2022-01-31T21:22:39.028+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.030150+0000) 2022-01-31T21:22:39.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[46715]: debug 2022-01-31T21:22:39.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.154408+0000) 2022-01-31T21:22:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:39 smithi167 conmon[54076]: debug 2022-01-31T21:22:39.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.376414+0000) 2022-01-31T21:22:39.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:39 smithi167 conmon[60316]: debug 2022-01-31T21:22:39.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.380917+0000) 2022-01-31T21:22:39.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[51620]: debug 2022-01-31T21:22:39.731+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.733276+0000) 2022-01-31T21:22:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:39 smithi171 conmon[41853]: debug 2022-01-31T21:22:39.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.858094+0000) 2022-01-31T21:22:40.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:39 smithi167 conmon[49112]: debug 2022-01-31T21:22:39.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:39.707759+0000) 2022-01-31T21:22:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:40 smithi167 conmon[54076]: debug 2022-01-31T21:22:40.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:40.376560+0000) 2022-01-31T21:22:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:40 smithi167 conmon[60316]: debug 2022-01-31T21:22:40.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:40.381096+0000) 2022-01-31T21:22:40.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:40 smithi171 conmon[46715]: debug 2022-01-31T21:22:40.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:40.154566+0000) 2022-01-31T21:22:40.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:40 smithi171 conmon[51620]: debug 2022-01-31T21:22:40.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:40.733422+0000) 2022-01-31T21:22:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:40 smithi171 conmon[41853]: debug 2022-01-31T21:22:40.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:40.858240+0000) 2022-01-31T21:22:41.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:40 smithi167 conmon[49112]: debug 2022-01-31T21:22:40.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:40.707867+0000) 2022-01-31T21:22:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:41 smithi167 conmon[54076]: debug 2022-01-31T21:22:41.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:41.376744+0000) 2022-01-31T21:22:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:41 smithi167 conmon[60316]: debug 2022-01-31T21:22:41.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:41.381304+0000) 2022-01-31T21:22:41.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:41 smithi171 conmon[46715]: debug 2022-01-31T21:22:41.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:41.154762+0000) 2022-01-31T21:22:41.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:41 smithi171 conmon[51620]: debug 2022-01-31T21:22:41.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:41.733599+0000) 2022-01-31T21:22:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:41 smithi171 conmon[41853]: debug 2022-01-31T21:22:41.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:41.858371+0000) 2022-01-31T21:22:42.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:41 smithi167 conmon[49112]: debug 2022-01-31T21:22:41.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:41.708084+0000) 2022-01-31T21:22:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:42 smithi167 conmon[60316]: debug 2022-01-31T21:22:42.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:42.381534+0000) 2022-01-31T21:22:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:42 smithi167 conmon[54076]: debug 2022-01-31T21:22:42.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:42.376900+0000) 2022-01-31T21:22:42.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:42 smithi171 conmon[46715]: debug 2022-01-31T21:22:42.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:42.154945+0000) 2022-01-31T21:22:42.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:42 smithi171 conmon[51620]: debug 2022-01-31T21:22:42.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:42.733803+0000) 2022-01-31T21:22:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:42 smithi171 conmon[41853]: debug 2022-01-31T21:22:42.856+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:42.858546+0000) 2022-01-31T21:22:43.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:42 smithi167 conmon[49112]: debug 2022-01-31T21:22:42.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:42.708261+0000) 2022-01-31T21:22:43.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:43 smithi167 conmon[54076]: debug 2022-01-31T21:22:43.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:43.377077+0000) 2022-01-31T21:22:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:43 smithi167 conmon[60316]: debug 2022-01-31T21:22:43.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:43.381674+0000) 2022-01-31T21:22:43.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:43 smithi171 conmon[46715]: debug 2022-01-31T21:22:43.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:43.155119+0000) 2022-01-31T21:22:43.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:43 smithi171 conmon[51620]: debug 2022-01-31T21:22:43.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:43.733944+0000) 2022-01-31T21:22:44.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:43 smithi167 conmon[49112]: debug 2022-01-31T21:22:43.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:43.708459+0000) 2022-01-31T21:22:44.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[35325]: debug 2022-01-31T21:22:44.056+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125006 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:44.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[46715]: debug 2022-01-31T21:22:44.043+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.045506+0000) 2022-01-31T21:22:44.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[51620]: debug 2022-01-31T21:22:44.044+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.046004+0000) 2022-01-31T21:22:44.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:43 smithi171 conmon[41853]: debug 2022-01-31T21:22:43.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:43.858697+0000) 2022-01-31T21:22:44.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[41853]: debug 2022-01-31T21:22:44.042+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.044606+0000) 2022-01-31T21:22:44.346 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:44 smithi167 conmon[49112]: debug 2022-01-31T21:22:44.044+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.045309+0000) 2022-01-31T21:22:44.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:44 smithi167 conmon[54076]: debug 2022-01-31T21:22:44.046+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.046633+0000) 2022-01-31T21:22:44.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:44 smithi167 conmon[60316]: debug 2022-01-31T21:22:44.045+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.045919+0000) 2022-01-31T21:22:44.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[46715]: debug 2022-01-31T21:22:44.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.155263+0000) 2022-01-31T21:22:44.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:44 smithi167 conmon[60316]: debug 2022-01-31T21:22:44.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.381828+0000) 2022-01-31T21:22:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:44 smithi167 conmon[54076]: debug 2022-01-31T21:22:44.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.377273+0000) 2022-01-31T21:22:44.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[51620]: debug 2022-01-31T21:22:44.732+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.734136+0000) 2022-01-31T21:22:45.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:44 smithi171 conmon[41853]: debug 2022-01-31T21:22:44.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.858838+0000) 2022-01-31T21:22:45.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:44 smithi167 conmon[49112]: debug 2022-01-31T21:22:44.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:44.708672+0000) 2022-01-31T21:22:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:45 smithi167 conmon[54076]: debug 2022-01-31T21:22:45.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:45.377478+0000) 2022-01-31T21:22:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:45 smithi167 conmon[60316]: debug 2022-01-31T21:22:45.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:45.381966+0000) 2022-01-31T21:22:45.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:45 smithi171 conmon[46715]: debug 2022-01-31T21:22:45.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:45.155420+0000) 2022-01-31T21:22:45.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:45 smithi171 conmon[51620]: debug 2022-01-31T21:22:45.733+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:45.734290+0000) 2022-01-31T21:22:45.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:45 smithi167 conmon[49112]: debug 2022-01-31T21:22:45.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:45.708803+0000) 2022-01-31T21:22:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:45 smithi171 conmon[41853]: debug 2022-01-31T21:22:45.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:45.858912+0000) 2022-01-31T21:22:46.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:46 smithi171 conmon[46715]: debug 2022-01-31T21:22:46.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:46.155600+0000) 2022-01-31T21:22:46.674 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:46 smithi167 conmon[60316]: debug 2022-01-31T21:22:46.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:46.382171+0000) 2022-01-31T21:22:46.675 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:46 smithi167 conmon[54076]: debug 2022-01-31T21:22:46.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:46.377675+0000) 2022-01-31T21:22:46.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:46 smithi171 conmon[51620]: debug 2022-01-31T21:22:46.733+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:46.734444+0000) 2022-01-31T21:22:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:46 smithi171 conmon[41853]: debug 2022-01-31T21:22:46.857+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:46.859058+0000) 2022-01-31T21:22:47.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:46 smithi167 conmon[49112]: debug 2022-01-31T21:22:46.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:46.709045+0000) 2022-01-31T21:22:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:47 smithi167 conmon[60316]: debug 2022-01-31T21:22:47.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:47.382360+0000) 2022-01-31T21:22:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:47 smithi167 conmon[54076]: debug 2022-01-31T21:22:47.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:47.377869+0000) 2022-01-31T21:22:47.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:47 smithi171 conmon[46715]: debug 2022-01-31T21:22:47.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:47.155769+0000) 2022-01-31T21:22:47.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:47 smithi171 conmon[51620]: debug 2022-01-31T21:22:47.733+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:47.734650+0000) 2022-01-31T21:22:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:47 smithi171 conmon[41853]: debug 2022-01-31T21:22:47.858+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:47.859285+0000) 2022-01-31T21:22:48.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:47 smithi167 conmon[49112]: debug 2022-01-31T21:22:47.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:47.709273+0000) 2022-01-31T21:22:48.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:48 smithi167 conmon[54076]: debug 2022-01-31T21:22:48.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:48.378038+0000) 2022-01-31T21:22:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:48 smithi167 conmon[60316]: debug 2022-01-31T21:22:48.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:48.382562+0000) 2022-01-31T21:22:48.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:48 smithi171 conmon[46715]: debug 2022-01-31T21:22:48.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:48.155950+0000) 2022-01-31T21:22:48.817 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:48 smithi171 conmon[51620]: debug 2022-01-31T21:22:48.734+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:48.734863+0000) 2022-01-31T21:22:49.058 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:48 smithi167 conmon[49112]: debug 2022-01-31T21:22:48.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:48.709472+0000) 2022-01-31T21:22:49.071 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[46715]: debug 2022-01-31T21:22:49.059+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.060771+0000) 2022-01-31T21:22:49.072 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[51620]: debug 2022-01-31T21:22:49.058+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.059749+0000) 2022-01-31T21:22:49.072 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:48 smithi171 conmon[41853]: debug 2022-01-31T21:22:48.858+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:48.859497+0000) 2022-01-31T21:22:49.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[41853]: debug 2022-01-31T21:22:49.058+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.060109+0000) 2022-01-31T21:22:49.355 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:49 smithi167 conmon[49112]: debug 2022-01-31T21:22:49.060+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.060679+0000) 2022-01-31T21:22:49.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:49 smithi167 conmon[54076]: debug 2022-01-31T21:22:49.060+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.061157+0000) 2022-01-31T21:22:49.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:49 smithi167 conmon[60316]: debug 2022-01-31T21:22:49.059+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.060223+0000) 2022-01-31T21:22:49.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[46715]: debug 2022-01-31T21:22:49.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.156184+0000) 2022-01-31T21:22:49.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[35325]: debug 2022-01-31T21:22:49.071+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125115 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:49 smithi167 conmon[60316]: debug 2022-01-31T21:22:49.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.382699+0000) 2022-01-31T21:22:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:49 smithi167 conmon[54076]: debug 2022-01-31T21:22:49.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.378241+0000) 2022-01-31T21:22:49.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[51620]: debug 2022-01-31T21:22:49.734+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.735048+0000) 2022-01-31T21:22:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:49 smithi171 conmon[41853]: debug 2022-01-31T21:22:49.858+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.859650+0000) 2022-01-31T21:22:50.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:49 smithi167 conmon[49112]: debug 2022-01-31T21:22:49.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:49.709708+0000) 2022-01-31T21:22:50.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:50 smithi167 conmon[60316]: debug 2022-01-31T21:22:50.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:50.382867+0000) 2022-01-31T21:22:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:50 smithi167 conmon[54076]: debug 2022-01-31T21:22:50.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:50.378429+0000) 2022-01-31T21:22:50.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:50 smithi171 conmon[46715]: debug 2022-01-31T21:22:50.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:50.156371+0000) 2022-01-31T21:22:50.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:50 smithi171 conmon[51620]: debug 2022-01-31T21:22:50.734+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:50.735174+0000) 2022-01-31T21:22:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:50 smithi171 conmon[41853]: debug 2022-01-31T21:22:50.859+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:50.859833+0000) 2022-01-31T21:22:51.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:50 smithi167 conmon[49112]: debug 2022-01-31T21:22:50.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:50.709839+0000) 2022-01-31T21:22:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:51 smithi167 conmon[60316]: debug 2022-01-31T21:22:51.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:51.383075+0000) 2022-01-31T21:22:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:51 smithi167 conmon[54076]: debug 2022-01-31T21:22:51.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:51.378638+0000) 2022-01-31T21:22:51.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:51 smithi171 conmon[46715]: debug 2022-01-31T21:22:51.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:51.156541+0000) 2022-01-31T21:22:51.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:51 smithi171 conmon[51620]: debug 2022-01-31T21:22:51.734+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:51.735309+0000) 2022-01-31T21:22:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:51 smithi171 conmon[41853]: debug 2022-01-31T21:22:51.859+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:51.859984+0000) 2022-01-31T21:22:52.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:51 smithi167 conmon[49112]: debug 2022-01-31T21:22:51.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:51.710061+0000) 2022-01-31T21:22:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:52 smithi167 conmon[54076]: debug 2022-01-31T21:22:52.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:52.378806+0000) 2022-01-31T21:22:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:52 smithi167 conmon[60316]: debug 2022-01-31T21:22:52.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:52.383252+0000) 2022-01-31T21:22:52.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:52 smithi171 conmon[46715]: debug 2022-01-31T21:22:52.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:52.156743+0000) 2022-01-31T21:22:52.818 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:52 smithi171 conmon[51620]: debug 2022-01-31T21:22:52.734+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:52.735463+0000) 2022-01-31T21:22:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:52 smithi171 conmon[41853]: debug 2022-01-31T21:22:52.859+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:52.860178+0000) 2022-01-31T21:22:53.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:52 smithi167 conmon[49112]: debug 2022-01-31T21:22:52.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:52.710296+0000) 2022-01-31T21:22:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:53 smithi167 conmon[60316]: debug 2022-01-31T21:22:53.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:53.383435+0000) 2022-01-31T21:22:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:53 smithi167 conmon[54076]: debug 2022-01-31T21:22:53.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:53.378994+0000) 2022-01-31T21:22:53.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:53 smithi171 conmon[46715]: debug 2022-01-31T21:22:53.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:53.156867+0000) 2022-01-31T21:22:53.818 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:53 smithi171 conmon[35325]: debug 2022-01-31T21:22:53.669+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:22:53.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:53 smithi171 conmon[51620]: debug 2022-01-31T21:22:53.734+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:53.735648+0000) 2022-01-31T21:22:54.073 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:53 smithi171 conmon[41853]: debug 2022-01-31T21:22:53.859+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:53.860378+0000) 2022-01-31T21:22:54.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:53 smithi167 conmon[49112]: debug 2022-01-31T21:22:53.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:53.710483+0000) 2022-01-31T21:22:54.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[35325]: debug 2022-01-31T21:22:54.090+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125222 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[41853]: debug 2022-01-31T21:22:54.074+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.075022+0000) 2022-01-31T21:22:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[46715]: debug 2022-01-31T21:22:54.075+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.076503+0000) 2022-01-31T21:22:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[46715]: debug 2022-01-31T21:22:54.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.157003+0000) 2022-01-31T21:22:54.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[51620]: debug 2022-01-31T21:22:54.075+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.076690+0000) 2022-01-31T21:22:54.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:54 smithi167 conmon[54076]: debug 2022-01-31T21:22:54.075+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.075924+0000) 2022-01-31T21:22:54.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:54 smithi167 conmon[60316]: debug 2022-01-31T21:22:54.076+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.077417+0000) 2022-01-31T21:22:54.364 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:54 smithi167 conmon[49112]: debug 2022-01-31T21:22:54.083+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.084405+0000) 2022-01-31T21:22:54.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:54 smithi167 conmon[54076]: debug 2022-01-31T21:22:54.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.379199+0000) 2022-01-31T21:22:54.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:54 smithi167 conmon[60316]: debug 2022-01-31T21:22:54.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.383649+0000) 2022-01-31T21:22:54.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[51620]: debug 2022-01-31T21:22:54.735+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.735862+0000) 2022-01-31T21:22:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:54 smithi171 conmon[41853]: debug 2022-01-31T21:22:54.859+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.860554+0000) 2022-01-31T21:22:55.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:54 smithi167 conmon[49112]: debug 2022-01-31T21:22:54.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:54.710611+0000) 2022-01-31T21:22:55.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:55 smithi167 conmon[60316]: debug 2022-01-31T21:22:55.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:55.383747+0000) 2022-01-31T21:22:55.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:55 smithi167 conmon[54076]: debug 2022-01-31T21:22:55.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:55.379339+0000) 2022-01-31T21:22:55.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:55 smithi171 conmon[46715]: debug 2022-01-31T21:22:55.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:55.157130+0000) 2022-01-31T21:22:55.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:55 smithi171 conmon[51620]: debug 2022-01-31T21:22:55.735+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:55.735961+0000) 2022-01-31T21:22:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:55 smithi171 conmon[41853]: debug 2022-01-31T21:22:55.860+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:55.860692+0000) 2022-01-31T21:22:56.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:55 smithi167 conmon[49112]: debug 2022-01-31T21:22:55.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:55.710746+0000) 2022-01-31T21:22:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:56 smithi167 conmon[60316]: debug 2022-01-31T21:22:56.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:56.383941+0000) 2022-01-31T21:22:56.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:56 smithi167 conmon[54076]: debug 2022-01-31T21:22:56.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:56.379571+0000) 2022-01-31T21:22:56.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:56 smithi171 conmon[46715]: debug 2022-01-31T21:22:56.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:56.157286+0000) 2022-01-31T21:22:56.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:56 smithi171 conmon[51620]: debug 2022-01-31T21:22:56.735+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:56.736142+0000) 2022-01-31T21:22:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:56 smithi171 conmon[41853]: debug 2022-01-31T21:22:56.860+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:56.860845+0000) 2022-01-31T21:22:57.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:56 smithi167 conmon[49112]: debug 2022-01-31T21:22:56.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:56.710905+0000) 2022-01-31T21:22:57.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:57 smithi167 conmon[54076]: debug 2022-01-31T21:22:57.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:57.379716+0000) 2022-01-31T21:22:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:57 smithi167 conmon[60316]: debug 2022-01-31T21:22:57.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:57.384159+0000) 2022-01-31T21:22:57.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:57 smithi171 conmon[46715]: debug 2022-01-31T21:22:57.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:57.157488+0000) 2022-01-31T21:22:57.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:57 smithi171 conmon[51620]: debug 2022-01-31T21:22:57.735+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:57.736330+0000) 2022-01-31T21:22:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:57 smithi171 conmon[41853]: debug 2022-01-31T21:22:57.860+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:57.860989+0000) 2022-01-31T21:22:58.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:57 smithi167 conmon[49112]: debug 2022-01-31T21:22:57.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:57.711077+0000) 2022-01-31T21:22:58.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:58 smithi167 conmon[54076]: debug 2022-01-31T21:22:58.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:58.379858+0000) 2022-01-31T21:22:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:58 smithi167 conmon[60316]: debug 2022-01-31T21:22:58.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:58.384366+0000) 2022-01-31T21:22:58.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:58 smithi171 conmon[46715]: debug 2022-01-31T21:22:58.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:58.157678+0000) 2022-01-31T21:22:58.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:58 smithi171 conmon[51620]: debug 2022-01-31T21:22:58.735+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:58.736511+0000) 2022-01-31T21:22:59.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:58 smithi171 conmon[41853]: debug 2022-01-31T21:22:58.860+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:58.861175+0000) 2022-01-31T21:22:59.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:58 smithi167 conmon[49112]: debug 2022-01-31T21:22:58.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:58.711318+0000) 2022-01-31T21:22:59.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[35325]: debug 2022-01-31T21:22:59.106+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125333 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:22:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[41853]: debug 2022-01-31T21:22:59.092+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.093634+0000) 2022-01-31T21:22:59.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[46715]: debug 2022-01-31T21:22:59.094+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.094787+0000) 2022-01-31T21:22:59.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[46715]: debug 2022-01-31T21:22:59.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.157849+0000) 2022-01-31T21:22:59.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[51620]: debug 2022-01-31T21:22:59.093+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.093881+0000) 2022-01-31T21:22:59.365 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:59 smithi167 conmon[54076]: debug 2022-01-31T21:22:59.100+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.101227+0000) 2022-01-31T21:22:59.365 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:59 smithi167 conmon[60316]: debug 2022-01-31T21:22:59.092+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.095321+0000) 2022-01-31T21:22:59.366 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:59 smithi167 conmon[49112]: debug 2022-01-31T21:22:59.092+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.094593+0000) 2022-01-31T21:22:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:22:59 smithi167 conmon[54076]: debug 2022-01-31T21:22:59.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.380025+0000) 2022-01-31T21:22:59.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:22:59 smithi167 conmon[60316]: debug 2022-01-31T21:22:59.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.384573+0000) 2022-01-31T21:22:59.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[51620]: debug 2022-01-31T21:22:59.735+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.736681+0000) 2022-01-31T21:23:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:22:59 smithi171 conmon[41853]: debug 2022-01-31T21:22:59.860+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.861355+0000) 2022-01-31T21:23:00.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:22:59 smithi167 conmon[49112]: debug 2022-01-31T21:22:59.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:22:59.711494+0000) 2022-01-31T21:23:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:00 smithi167 conmon[54076]: debug 2022-01-31T21:23:00.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:00.380158+0000) 2022-01-31T21:23:00.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:00 smithi167 conmon[60316]: debug 2022-01-31T21:23:00.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:00.384682+0000) 2022-01-31T21:23:00.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:00 smithi171 conmon[46715]: debug 2022-01-31T21:23:00.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:00.158019+0000) 2022-01-31T21:23:00.819 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:00 smithi171 conmon[51620]: debug 2022-01-31T21:23:00.736+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:00.736860+0000) 2022-01-31T21:23:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:00 smithi171 conmon[41853]: debug 2022-01-31T21:23:00.860+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:00.861543+0000) 2022-01-31T21:23:01.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:00 smithi167 conmon[49112]: debug 2022-01-31T21:23:00.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:00.711641+0000) 2022-01-31T21:23:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:01 smithi167 conmon[54076]: debug 2022-01-31T21:23:01.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:01.380339+0000) 2022-01-31T21:23:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:01 smithi167 conmon[60316]: debug 2022-01-31T21:23:01.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:01.384808+0000) 2022-01-31T21:23:01.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:01 smithi171 conmon[46715]: debug 2022-01-31T21:23:01.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:01.158123+0000) 2022-01-31T21:23:01.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:01 smithi171 conmon[51620]: debug 2022-01-31T21:23:01.736+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:01.737052+0000) 2022-01-31T21:23:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:01 smithi171 conmon[41853]: debug 2022-01-31T21:23:01.861+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:01.861747+0000) 2022-01-31T21:23:02.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:01 smithi167 conmon[49112]: debug 2022-01-31T21:23:01.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:01.711782+0000) 2022-01-31T21:23:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:02 smithi167 conmon[60316]: debug 2022-01-31T21:23:02.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:02.384942+0000) 2022-01-31T21:23:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:02 smithi167 conmon[54076]: debug 2022-01-31T21:23:02.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:02.380585+0000) 2022-01-31T21:23:02.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:02 smithi171 conmon[46715]: debug 2022-01-31T21:23:02.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:02.158302+0000) 2022-01-31T21:23:02.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:02 smithi171 conmon[51620]: debug 2022-01-31T21:23:02.736+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:02.737202+0000) 2022-01-31T21:23:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:02 smithi171 conmon[41853]: debug 2022-01-31T21:23:02.861+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:02.861923+0000) 2022-01-31T21:23:03.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:02 smithi167 conmon[49112]: debug 2022-01-31T21:23:02.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:02.711997+0000) 2022-01-31T21:23:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:03 smithi167 conmon[60316]: debug 2022-01-31T21:23:03.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:03.385084+0000) 2022-01-31T21:23:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:03 smithi167 conmon[54076]: debug 2022-01-31T21:23:03.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:03.380793+0000) 2022-01-31T21:23:03.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:03 smithi171 conmon[46715]: debug 2022-01-31T21:23:03.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:03.158466+0000) 2022-01-31T21:23:03.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:03 smithi171 conmon[51620]: debug 2022-01-31T21:23:03.736+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:03.737380+0000) 2022-01-31T21:23:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:03 smithi171 conmon[41853]: debug 2022-01-31T21:23:03.861+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:03.862078+0000) 2022-01-31T21:23:04.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:03 smithi167 conmon[49112]: debug 2022-01-31T21:23:03.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:03.712156+0000) 2022-01-31T21:23:04.366 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:04 smithi167 conmon[49112]: debug 2022-01-31T21:23:04.108+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.109655+0000) 2022-01-31T21:23:04.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:04 smithi167 conmon[54076]: debug 2022-01-31T21:23:04.108+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.110207+0000) 2022-01-31T21:23:04.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:04 smithi167 conmon[60316]: debug 2022-01-31T21:23:04.108+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.110141+0000) 2022-01-31T21:23:04.538 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[35325]: debug 2022-01-31T21:23:04.122+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125445 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:04.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[41853]: debug 2022-01-31T21:23:04.109+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.110449+0000) 2022-01-31T21:23:04.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[51620]: debug 2022-01-31T21:23:04.109+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.109793+0000) 2022-01-31T21:23:04.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[46715]: debug 2022-01-31T21:23:04.108+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.109352+0000) 2022-01-31T21:23:04.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[46715]: debug 2022-01-31T21:23:04.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.158646+0000) 2022-01-31T21:23:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:04 smithi167 conmon[54076]: debug 2022-01-31T21:23:04.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.380977+0000) 2022-01-31T21:23:04.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:04 smithi167 conmon[60316]: debug 2022-01-31T21:23:04.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.385236+0000) 2022-01-31T21:23:04.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[51620]: debug 2022-01-31T21:23:04.736+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.737581+0000) 2022-01-31T21:23:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:04 smithi171 conmon[41853]: debug 2022-01-31T21:23:04.861+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.862258+0000) 2022-01-31T21:23:05.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:04 smithi167 conmon[49112]: debug 2022-01-31T21:23:04.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:04.712343+0000) 2022-01-31T21:23:05.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:05 smithi167 conmon[54076]: debug 2022-01-31T21:23:05.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:05.381137+0000) 2022-01-31T21:23:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:05 smithi167 conmon[60316]: debug 2022-01-31T21:23:05.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:05.385342+0000) 2022-01-31T21:23:05.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:05 smithi171 conmon[46715]: debug 2022-01-31T21:23:05.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:05.158815+0000) 2022-01-31T21:23:05.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:05 smithi171 conmon[51620]: debug 2022-01-31T21:23:05.737+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:05.737705+0000) 2022-01-31T21:23:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:05 smithi171 conmon[41853]: debug 2022-01-31T21:23:05.861+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:05.862418+0000) 2022-01-31T21:23:06.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:05 smithi167 conmon[49112]: debug 2022-01-31T21:23:05.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:05.712551+0000) 2022-01-31T21:23:06.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:06 smithi167 conmon[54076]: debug 2022-01-31T21:23:06.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:06.381312+0000) 2022-01-31T21:23:06.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:06 smithi167 conmon[60316]: debug 2022-01-31T21:23:06.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:06.385540+0000) 2022-01-31T21:23:06.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:06 smithi171 conmon[46715]: debug 2022-01-31T21:23:06.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:06.158950+0000) 2022-01-31T21:23:06.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:06 smithi171 conmon[51620]: debug 2022-01-31T21:23:06.737+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:06.737862+0000) 2022-01-31T21:23:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:06 smithi171 conmon[41853]: debug 2022-01-31T21:23:06.861+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:06.862593+0000) 2022-01-31T21:23:07.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:06 smithi167 conmon[49112]: debug 2022-01-31T21:23:06.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:06.712773+0000) 2022-01-31T21:23:07.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:07 smithi167 conmon[54076]: debug 2022-01-31T21:23:07.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:07.381528+0000) 2022-01-31T21:23:07.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:07 smithi167 conmon[60316]: debug 2022-01-31T21:23:07.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:07.385708+0000) 2022-01-31T21:23:07.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:07 smithi171 conmon[46715]: debug 2022-01-31T21:23:07.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:07.159108+0000) 2022-01-31T21:23:07.820 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:07 smithi171 conmon[51620]: debug 2022-01-31T21:23:07.737+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:07.738014+0000) 2022-01-31T21:23:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:07 smithi171 conmon[41853]: debug 2022-01-31T21:23:07.862+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:07.862792+0000) 2022-01-31T21:23:08.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:07 smithi167 conmon[49112]: debug 2022-01-31T21:23:07.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:07.712980+0000) 2022-01-31T21:23:08.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:08 smithi167 conmon[54076]: debug 2022-01-31T21:23:08.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:08.381684+0000) 2022-01-31T21:23:08.388 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:08 smithi167 conmon[60316]: debug 2022-01-31T21:23:08.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:08.385833+0000) 2022-01-31T21:23:08.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:08 smithi171 conmon[46715]: debug 2022-01-31T21:23:08.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:08.159265+0000) 2022-01-31T21:23:08.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:08 smithi171 conmon[51620]: debug 2022-01-31T21:23:08.737+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:08.738205+0000) 2022-01-31T21:23:08.821 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:08 smithi171 conmon[35325]: debug 2022-01-31T21:23:08.670+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:23:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:08 smithi171 conmon[41853]: debug 2022-01-31T21:23:08.862+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:08.862943+0000) 2022-01-31T21:23:09.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:08 smithi167 conmon[49112]: debug 2022-01-31T21:23:08.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:08.713159+0000) 2022-01-31T21:23:09.380 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:09 smithi167 conmon[49112]: debug 2022-01-31T21:23:09.124+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.125814+0000) 2022-01-31T21:23:09.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:09 smithi167 conmon[54076]: debug 2022-01-31T21:23:09.124+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.126126+0000) 2022-01-31T21:23:09.381 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:09 smithi167 conmon[60316]: debug 2022-01-31T21:23:09.124+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.125677+0000) 2022-01-31T21:23:09.540 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[35325]: debug 2022-01-31T21:23:09.137+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125554 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:09.540 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[41853]: debug 2022-01-31T21:23:09.125+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.126553+0000) 2022-01-31T21:23:09.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[51620]: debug 2022-01-31T21:23:09.123+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.124635+0000) 2022-01-31T21:23:09.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[46715]: debug 2022-01-31T21:23:09.124+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.125004+0000) 2022-01-31T21:23:09.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[46715]: debug 2022-01-31T21:23:09.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.159476+0000) 2022-01-31T21:23:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:09 smithi167 conmon[54076]: debug 2022-01-31T21:23:09.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.381882+0000) 2022-01-31T21:23:09.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:09 smithi167 conmon[60316]: debug 2022-01-31T21:23:09.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.386021+0000) 2022-01-31T21:23:09.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[51620]: debug 2022-01-31T21:23:09.737+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.738391+0000) 2022-01-31T21:23:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:09 smithi171 conmon[41853]: debug 2022-01-31T21:23:09.862+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.863081+0000) 2022-01-31T21:23:10.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:09 smithi167 conmon[49112]: debug 2022-01-31T21:23:09.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:09.713354+0000) 2022-01-31T21:23:10.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:10 smithi167 conmon[54076]: debug 2022-01-31T21:23:10.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:10.382009+0000) 2022-01-31T21:23:10.392 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:10 smithi167 conmon[60316]: debug 2022-01-31T21:23:10.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:10.386217+0000) 2022-01-31T21:23:10.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:10 smithi171 conmon[46715]: debug 2022-01-31T21:23:10.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:10.159614+0000) 2022-01-31T21:23:10.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:10 smithi171 conmon[51620]: debug 2022-01-31T21:23:10.738+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:10.738539+0000) 2022-01-31T21:23:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:10 smithi171 conmon[41853]: debug 2022-01-31T21:23:10.862+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:10.863235+0000) 2022-01-31T21:23:11.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:10 smithi167 conmon[49112]: debug 2022-01-31T21:23:10.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:10.713552+0000) 2022-01-31T21:23:11.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:11 smithi167 conmon[54076]: debug 2022-01-31T21:23:11.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:11.382144+0000) 2022-01-31T21:23:11.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:11 smithi167 conmon[60316]: debug 2022-01-31T21:23:11.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:11.386421+0000) 2022-01-31T21:23:11.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:11 smithi171 conmon[46715]: debug 2022-01-31T21:23:11.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:11.159802+0000) 2022-01-31T21:23:11.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:11 smithi171 conmon[51620]: debug 2022-01-31T21:23:11.738+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:11.738697+0000) 2022-01-31T21:23:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:11 smithi171 conmon[41853]: debug 2022-01-31T21:23:11.862+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:11.863437+0000) 2022-01-31T21:23:12.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:11 smithi167 conmon[49112]: debug 2022-01-31T21:23:11.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:11.713804+0000) 2022-01-31T21:23:12.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:12 smithi167 conmon[54076]: debug 2022-01-31T21:23:12.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:12.382320+0000) 2022-01-31T21:23:12.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:12 smithi167 conmon[60316]: debug 2022-01-31T21:23:12.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:12.386597+0000) 2022-01-31T21:23:12.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:12 smithi171 conmon[46715]: debug 2022-01-31T21:23:12.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:12.159965+0000) 2022-01-31T21:23:12.821 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:12 smithi171 conmon[51620]: debug 2022-01-31T21:23:12.738+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:12.738904+0000) 2022-01-31T21:23:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:12 smithi171 conmon[41853]: debug 2022-01-31T21:23:12.863+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:12.863633+0000) 2022-01-31T21:23:13.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:12 smithi167 conmon[49112]: debug 2022-01-31T21:23:12.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:12.713997+0000) 2022-01-31T21:23:13.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:13 smithi167 conmon[54076]: debug 2022-01-31T21:23:13.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:13.382542+0000) 2022-01-31T21:23:13.396 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:13 smithi167 conmon[60316]: debug 2022-01-31T21:23:13.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:13.386718+0000) 2022-01-31T21:23:13.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:13 smithi171 conmon[46715]: debug 2022-01-31T21:23:13.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:13.160071+0000) 2022-01-31T21:23:13.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:13 smithi171 conmon[51620]: debug 2022-01-31T21:23:13.738+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:13.738992+0000) 2022-01-31T21:23:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:13 smithi171 conmon[41853]: debug 2022-01-31T21:23:13.863+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:13.863840+0000) 2022-01-31T21:23:14.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:13 smithi167 conmon[49112]: debug 2022-01-31T21:23:13.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:13.714208+0000) 2022-01-31T21:23:14.395 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:14 smithi167 conmon[49112]: debug 2022-01-31T21:23:14.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.141256+0000) 2022-01-31T21:23:14.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:14 smithi167 conmon[54076]: debug 2022-01-31T21:23:14.139+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.141175+0000) 2022-01-31T21:23:14.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:14 smithi167 conmon[54076]: debug 2022-01-31T21:23:14.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.382747+0000) 2022-01-31T21:23:14.396 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:14 smithi167 conmon[60316]: debug 2022-01-31T21:23:14.139+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.140955+0000) 2022-01-31T21:23:14.397 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:14 smithi167 conmon[60316]: debug 2022-01-31T21:23:14.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.386889+0000) 2022-01-31T21:23:14.541 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[35325]: debug 2022-01-31T21:23:14.152+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125665 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:14.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[41853]: debug 2022-01-31T21:23:14.139+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.140458+0000) 2022-01-31T21:23:14.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[51620]: debug 2022-01-31T21:23:14.140+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.140798+0000) 2022-01-31T21:23:14.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[46715]: debug 2022-01-31T21:23:14.140+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.141548+0000) 2022-01-31T21:23:14.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[46715]: debug 2022-01-31T21:23:14.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.160189+0000) 2022-01-31T21:23:14.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[51620]: debug 2022-01-31T21:23:14.738+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.739181+0000) 2022-01-31T21:23:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:14 smithi171 conmon[41853]: debug 2022-01-31T21:23:14.863+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.864008+0000) 2022-01-31T21:23:15.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:14 smithi167 conmon[49112]: debug 2022-01-31T21:23:14.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:14.714424+0000) 2022-01-31T21:23:15.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:15 smithi167 conmon[54076]: debug 2022-01-31T21:23:15.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:15.382925+0000) 2022-01-31T21:23:15.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:15 smithi167 conmon[60316]: debug 2022-01-31T21:23:15.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:15.387030+0000) 2022-01-31T21:23:15.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:15 smithi171 conmon[46715]: debug 2022-01-31T21:23:15.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:15.160339+0000) 2022-01-31T21:23:15.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:15 smithi171 conmon[51620]: debug 2022-01-31T21:23:15.739+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:15.739338+0000) 2022-01-31T21:23:16.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:15 smithi167 conmon[49112]: debug 2022-01-31T21:23:15.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:15.714601+0000) 2022-01-31T21:23:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:15 smithi171 conmon[41853]: debug 2022-01-31T21:23:15.863+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:15.864120+0000) 2022-01-31T21:23:16.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:16 smithi171 conmon[46715]: debug 2022-01-31T21:23:16.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:16.160517+0000) 2022-01-31T21:23:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:16 smithi167 conmon[54076]: debug 2022-01-31T21:23:16.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:16.383116+0000) 2022-01-31T21:23:16.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:16 smithi167 conmon[60316]: debug 2022-01-31T21:23:16.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:16.387211+0000) 2022-01-31T21:23:16.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:16 smithi171 conmon[51620]: debug 2022-01-31T21:23:16.739+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:16.739492+0000) 2022-01-31T21:23:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:16 smithi171 conmon[41853]: debug 2022-01-31T21:23:16.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:16.864277+0000) 2022-01-31T21:23:17.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:16 smithi167 conmon[49112]: debug 2022-01-31T21:23:16.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:16.714812+0000) 2022-01-31T21:23:17.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:17 smithi167 conmon[54076]: debug 2022-01-31T21:23:17.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:17.383309+0000) 2022-01-31T21:23:17.402 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:17 smithi167 conmon[60316]: debug 2022-01-31T21:23:17.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:17.387418+0000) 2022-01-31T21:23:17.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:17 smithi171 conmon[46715]: debug 2022-01-31T21:23:17.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:17.160705+0000) 2022-01-31T21:23:17.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:17 smithi171 conmon[51620]: debug 2022-01-31T21:23:17.739+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:17.739648+0000) 2022-01-31T21:23:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:17 smithi171 conmon[41853]: debug 2022-01-31T21:23:17.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:17.864430+0000) 2022-01-31T21:23:18.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:17 smithi167 conmon[49112]: debug 2022-01-31T21:23:17.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:17.714965+0000) 2022-01-31T21:23:18.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:18 smithi167 conmon[54076]: debug 2022-01-31T21:23:18.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:18.383528+0000) 2022-01-31T21:23:18.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:18 smithi167 conmon[60316]: debug 2022-01-31T21:23:18.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:18.387605+0000) 2022-01-31T21:23:18.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:18 smithi171 conmon[46715]: debug 2022-01-31T21:23:18.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:18.160904+0000) 2022-01-31T21:23:18.822 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:18 smithi171 conmon[51620]: debug 2022-01-31T21:23:18.739+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:18.739817+0000) 2022-01-31T21:23:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:18 smithi171 conmon[41853]: debug 2022-01-31T21:23:18.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:18.864615+0000) 2022-01-31T21:23:19.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:18 smithi167 conmon[49112]: debug 2022-01-31T21:23:18.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:18.715178+0000) 2022-01-31T21:23:19.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:19 smithi167 conmon[49112]: debug 2022-01-31T21:23:19.155+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.156698+0000) 2022-01-31T21:23:19.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:19 smithi167 conmon[60316]: debug 2022-01-31T21:23:19.155+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.156834+0000) 2022-01-31T21:23:19.403 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:19 smithi167 conmon[60316]: debug 2022-01-31T21:23:19.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.387739+0000) 2022-01-31T21:23:19.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:19 smithi167 conmon[54076]: debug 2022-01-31T21:23:19.154+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.156168+0000) 2022-01-31T21:23:19.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:19 smithi167 conmon[54076]: debug 2022-01-31T21:23:19.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.383677+0000) 2022-01-31T21:23:19.542 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[35325]: debug 2022-01-31T21:23:19.167+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125774 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:19.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[41853]: debug 2022-01-31T21:23:19.154+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.155465+0000) 2022-01-31T21:23:19.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[51620]: debug 2022-01-31T21:23:19.155+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.156002+0000) 2022-01-31T21:23:19.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[46715]: debug 2022-01-31T21:23:19.155+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.155875+0000) 2022-01-31T21:23:19.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[46715]: debug 2022-01-31T21:23:19.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.161028+0000) 2022-01-31T21:23:19.823 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[51620]: debug 2022-01-31T21:23:19.739+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.739993+0000) 2022-01-31T21:23:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:19 smithi171 conmon[41853]: debug 2022-01-31T21:23:19.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.864800+0000) 2022-01-31T21:23:20.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:19 smithi167 conmon[49112]: debug 2022-01-31T21:23:19.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:19.715373+0000) 2022-01-31T21:23:20.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:20 smithi167 conmon[60316]: debug 2022-01-31T21:23:20.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:20.387885+0000) 2022-01-31T21:23:20.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:20 smithi167 conmon[54076]: debug 2022-01-31T21:23:20.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:20.383887+0000) 2022-01-31T21:23:20.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:20 smithi171 conmon[46715]: debug 2022-01-31T21:23:20.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:20.161137+0000) 2022-01-31T21:23:20.823 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:20 smithi171 conmon[51620]: debug 2022-01-31T21:23:20.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:20.740174+0000) 2022-01-31T21:23:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:20 smithi171 conmon[41853]: debug 2022-01-31T21:23:20.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:20.864910+0000) 2022-01-31T21:23:21.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:20 smithi167 conmon[49112]: debug 2022-01-31T21:23:20.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:20.715560+0000) 2022-01-31T21:23:21.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:21 smithi167 conmon[54076]: debug 2022-01-31T21:23:21.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:21.384088+0000) 2022-01-31T21:23:21.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:21 smithi167 conmon[60316]: debug 2022-01-31T21:23:21.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:21.388070+0000) 2022-01-31T21:23:21.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:21 smithi171 conmon[46715]: debug 2022-01-31T21:23:21.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:21.161294+0000) 2022-01-31T21:23:21.823 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:21 smithi171 conmon[51620]: debug 2022-01-31T21:23:21.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:21.740360+0000) 2022-01-31T21:23:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:21 smithi171 conmon[41853]: debug 2022-01-31T21:23:21.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:21.865064+0000) 2022-01-31T21:23:22.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:21 smithi167 conmon[49112]: debug 2022-01-31T21:23:21.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:21.715792+0000) 2022-01-31T21:23:22.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:22 smithi167 conmon[54076]: debug 2022-01-31T21:23:22.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:22.384257+0000) 2022-01-31T21:23:22.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:22 smithi167 conmon[60316]: debug 2022-01-31T21:23:22.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:22.388242+0000) 2022-01-31T21:23:22.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:22 smithi171 conmon[46715]: debug 2022-01-31T21:23:22.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:22.161502+0000) 2022-01-31T21:23:22.823 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:22 smithi171 conmon[51620]: debug 2022-01-31T21:23:22.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:22.740517+0000) 2022-01-31T21:23:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:22 smithi171 conmon[41853]: debug 2022-01-31T21:23:22.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:22.865245+0000) 2022-01-31T21:23:23.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:22 smithi167 conmon[49112]: debug 2022-01-31T21:23:22.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:22.715965+0000) 2022-01-31T21:23:23.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:23 smithi167 conmon[54076]: debug 2022-01-31T21:23:23.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:23.384490+0000) 2022-01-31T21:23:23.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:23 smithi167 conmon[60316]: debug 2022-01-31T21:23:23.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:23.388451+0000) 2022-01-31T21:23:23.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:23 smithi171 conmon[46715]: debug 2022-01-31T21:23:23.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:23.161608+0000) 2022-01-31T21:23:23.823 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:23 smithi171 conmon[35325]: debug 2022-01-31T21:23:23.671+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:23:23.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:23 smithi171 conmon[51620]: debug 2022-01-31T21:23:23.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:23.740824+0000) 2022-01-31T21:23:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:23 smithi171 conmon[41853]: debug 2022-01-31T21:23:23.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:23.865436+0000) 2022-01-31T21:23:24.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:23 smithi167 conmon[49112]: debug 2022-01-31T21:23:23.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:23.716146+0000) 2022-01-31T21:23:24.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:24 smithi167 conmon[49112]: debug 2022-01-31T21:23:24.169+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.171399+0000) 2022-01-31T21:23:24.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:24 smithi167 conmon[54076]: debug 2022-01-31T21:23:24.170+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.171982+0000) 2022-01-31T21:23:24.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:24 smithi167 conmon[54076]: debug 2022-01-31T21:23:24.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.384674+0000) 2022-01-31T21:23:24.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:24 smithi167 conmon[60316]: debug 2022-01-31T21:23:24.169+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.170805+0000) 2022-01-31T21:23:24.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:24 smithi167 conmon[60316]: debug 2022-01-31T21:23:24.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.388579+0000) 2022-01-31T21:23:24.543 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[35325]: debug 2022-01-31T21:23:24.182+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:24.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[41853]: debug 2022-01-31T21:23:24.169+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.171001+0000) 2022-01-31T21:23:24.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[46715]: debug 2022-01-31T21:23:24.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.161867+0000) 2022-01-31T21:23:24.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[46715]: debug 2022-01-31T21:23:24.168+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.170192+0000) 2022-01-31T21:23:24.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[51620]: debug 2022-01-31T21:23:24.170+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.171997+0000) 2022-01-31T21:23:24.823 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[51620]: debug 2022-01-31T21:23:24.739+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.740964+0000) 2022-01-31T21:23:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:24 smithi171 conmon[41853]: debug 2022-01-31T21:23:24.863+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.865629+0000) 2022-01-31T21:23:25.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:24 smithi167 conmon[49112]: debug 2022-01-31T21:23:24.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:24.716311+0000) 2022-01-31T21:23:25.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:25 smithi167 conmon[54076]: debug 2022-01-31T21:23:25.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:25.384797+0000) 2022-01-31T21:23:25.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:25 smithi167 conmon[60316]: debug 2022-01-31T21:23:25.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:25.388717+0000) 2022-01-31T21:23:25.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:25 smithi171 conmon[46715]: debug 2022-01-31T21:23:25.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:25.162003+0000) 2022-01-31T21:23:25.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:25 smithi171 conmon[51620]: debug 2022-01-31T21:23:25.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:25.741145+0000) 2022-01-31T21:23:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:25 smithi171 conmon[41853]: debug 2022-01-31T21:23:25.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:25.865780+0000) 2022-01-31T21:23:26.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:25 smithi167 conmon[49112]: debug 2022-01-31T21:23:25.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:25.716520+0000) 2022-01-31T21:23:26.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:26 smithi167 conmon[54076]: debug 2022-01-31T21:23:26.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:26.384947+0000) 2022-01-31T21:23:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:26 smithi167 conmon[60316]: debug 2022-01-31T21:23:26.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:26.389021+0000) 2022-01-31T21:23:26.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:26 smithi171 conmon[46715]: debug 2022-01-31T21:23:26.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:26.162104+0000) 2022-01-31T21:23:26.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:26 smithi171 conmon[51620]: debug 2022-01-31T21:23:26.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:26.741325+0000) 2022-01-31T21:23:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:26 smithi171 conmon[41853]: debug 2022-01-31T21:23:26.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:26.865941+0000) 2022-01-31T21:23:27.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:26 smithi167 conmon[49112]: debug 2022-01-31T21:23:26.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:26.716709+0000) 2022-01-31T21:23:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:27 smithi167 conmon[54076]: debug 2022-01-31T21:23:27.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:27.385121+0000) 2022-01-31T21:23:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:27 smithi167 conmon[60316]: debug 2022-01-31T21:23:27.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:27.389456+0000) 2022-01-31T21:23:27.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:27 smithi171 conmon[46715]: debug 2022-01-31T21:23:27.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:27.162265+0000) 2022-01-31T21:23:27.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:27 smithi171 conmon[51620]: debug 2022-01-31T21:23:27.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:27.741523+0000) 2022-01-31T21:23:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:27 smithi171 conmon[41853]: debug 2022-01-31T21:23:27.865+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:27.866106+0000) 2022-01-31T21:23:28.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:27 smithi167 conmon[49112]: debug 2022-01-31T21:23:27.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:27.716903+0000) 2022-01-31T21:23:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:28 smithi167 conmon[54076]: debug 2022-01-31T21:23:28.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:28.385297+0000) 2022-01-31T21:23:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:28 smithi167 conmon[60316]: debug 2022-01-31T21:23:28.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:28.389657+0000) 2022-01-31T21:23:28.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:28 smithi171 conmon[46715]: debug 2022-01-31T21:23:28.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:28.162428+0000) 2022-01-31T21:23:28.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:28 smithi171 conmon[51620]: debug 2022-01-31T21:23:28.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:28.741741+0000) 2022-01-31T21:23:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:28 smithi171 conmon[41853]: debug 2022-01-31T21:23:28.865+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:28.866297+0000) 2022-01-31T21:23:29.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:28 smithi167 conmon[49112]: debug 2022-01-31T21:23:28.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:28.717123+0000) 2022-01-31T21:23:29.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:29 smithi167 conmon[49112]: debug 2022-01-31T21:23:29.186+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.186985+0000) 2022-01-31T21:23:29.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:29 smithi167 conmon[54076]: debug 2022-01-31T21:23:29.187+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.187754+0000) 2022-01-31T21:23:29.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:29 smithi167 conmon[54076]: debug 2022-01-31T21:23:29.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.385522+0000) 2022-01-31T21:23:29.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:29 smithi167 conmon[60316]: debug 2022-01-31T21:23:29.186+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.186613+0000) 2022-01-31T21:23:29.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:29 smithi167 conmon[60316]: debug 2022-01-31T21:23:29.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.389812+0000) 2022-01-31T21:23:29.544 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[35325]: debug 2022-01-31T21:23:29.197+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 125994 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:29.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[41853]: debug 2022-01-31T21:23:29.183+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.185721+0000) 2022-01-31T21:23:29.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[51620]: debug 2022-01-31T21:23:29.185+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.187618+0000) 2022-01-31T21:23:29.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[46715]: debug 2022-01-31T21:23:29.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.162640+0000) 2022-01-31T21:23:29.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[46715]: debug 2022-01-31T21:23:29.184+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.186368+0000) 2022-01-31T21:23:29.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[51620]: debug 2022-01-31T21:23:29.740+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.741871+0000) 2022-01-31T21:23:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:29 smithi171 conmon[41853]: debug 2022-01-31T21:23:29.864+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.866479+0000) 2022-01-31T21:23:30.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:29 smithi167 conmon[49112]: debug 2022-01-31T21:23:29.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:29.717343+0000) 2022-01-31T21:23:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:30 smithi167 conmon[54076]: debug 2022-01-31T21:23:30.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:30.385639+0000) 2022-01-31T21:23:30.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:30 smithi167 conmon[60316]: debug 2022-01-31T21:23:30.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:30.389969+0000) 2022-01-31T21:23:30.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:30 smithi171 conmon[46715]: debug 2022-01-31T21:23:30.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:30.162847+0000) 2022-01-31T21:23:30.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:30 smithi171 conmon[51620]: debug 2022-01-31T21:23:30.741+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:30.742048+0000) 2022-01-31T21:23:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:30 smithi171 conmon[41853]: debug 2022-01-31T21:23:30.865+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:30.866670+0000) 2022-01-31T21:23:31.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:30 smithi167 conmon[49112]: debug 2022-01-31T21:23:30.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:30.717512+0000) 2022-01-31T21:23:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:31 smithi167 conmon[54076]: debug 2022-01-31T21:23:31.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:31.385801+0000) 2022-01-31T21:23:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:31 smithi167 conmon[60316]: debug 2022-01-31T21:23:31.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:31.390120+0000) 2022-01-31T21:23:31.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:31 smithi171 conmon[46715]: debug 2022-01-31T21:23:31.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:31.162952+0000) 2022-01-31T21:23:31.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:31 smithi171 conmon[51620]: debug 2022-01-31T21:23:31.741+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:31.742252+0000) 2022-01-31T21:23:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:31 smithi171 conmon[41853]: debug 2022-01-31T21:23:31.865+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:31.866860+0000) 2022-01-31T21:23:32.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:31 smithi167 conmon[49112]: debug 2022-01-31T21:23:31.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:31.717670+0000) 2022-01-31T21:23:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:32 smithi167 conmon[60316]: debug 2022-01-31T21:23:32.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:32.390328+0000) 2022-01-31T21:23:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:32 smithi167 conmon[54076]: debug 2022-01-31T21:23:32.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:32.385988+0000) 2022-01-31T21:23:32.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:32 smithi171 conmon[46715]: debug 2022-01-31T21:23:32.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:32.163144+0000) 2022-01-31T21:23:32.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:32 smithi171 conmon[51620]: debug 2022-01-31T21:23:32.741+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:32.742414+0000) 2022-01-31T21:23:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:32 smithi171 conmon[41853]: debug 2022-01-31T21:23:32.866+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:32.867049+0000) 2022-01-31T21:23:33.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:32 smithi167 conmon[49112]: debug 2022-01-31T21:23:32.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:32.717888+0000) 2022-01-31T21:23:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:33 smithi167 conmon[54076]: debug 2022-01-31T21:23:33.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:33.386182+0000) 2022-01-31T21:23:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:33 smithi167 conmon[60316]: debug 2022-01-31T21:23:33.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:33.390477+0000) 2022-01-31T21:23:33.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:33 smithi171 conmon[46715]: debug 2022-01-31T21:23:33.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:33.163325+0000) 2022-01-31T21:23:33.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:33 smithi171 conmon[51620]: debug 2022-01-31T21:23:33.741+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:33.742623+0000) 2022-01-31T21:23:34.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:33 smithi171 conmon[41853]: debug 2022-01-31T21:23:33.866+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:33.867224+0000) 2022-01-31T21:23:34.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:33 smithi167 conmon[49112]: debug 2022-01-31T21:23:33.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:33.718100+0000) 2022-01-31T21:23:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:34 smithi167 conmon[60316]: debug 2022-01-31T21:23:34.201+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.202020+0000) 2022-01-31T21:23:34.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:34 smithi167 conmon[60316]: debug 2022-01-31T21:23:34.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.390657+0000) 2022-01-31T21:23:34.420 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:34 smithi167 conmon[49112]: debug 2022-01-31T21:23:34.201+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.202325+0000) 2022-01-31T21:23:34.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:34 smithi167 conmon[54076]: debug 2022-01-31T21:23:34.201+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.202242+0000) 2022-01-31T21:23:34.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:34 smithi167 conmon[54076]: debug 2022-01-31T21:23:34.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.386406+0000) 2022-01-31T21:23:34.545 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[35325]: debug 2022-01-31T21:23:34.213+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126107 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:34.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[41853]: debug 2022-01-31T21:23:34.200+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.201741+0000) 2022-01-31T21:23:34.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[51620]: debug 2022-01-31T21:23:34.200+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.202301+0000) 2022-01-31T21:23:34.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[46715]: debug 2022-01-31T21:23:34.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.163484+0000) 2022-01-31T21:23:34.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[46715]: debug 2022-01-31T21:23:34.200+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.202670+0000) 2022-01-31T21:23:34.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[51620]: debug 2022-01-31T21:23:34.742+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.742823+0000) 2022-01-31T21:23:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:34 smithi171 conmon[41853]: debug 2022-01-31T21:23:34.866+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.867394+0000) 2022-01-31T21:23:35.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:34 smithi167 conmon[49112]: debug 2022-01-31T21:23:34.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:34.718335+0000) 2022-01-31T21:23:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:35 smithi167 conmon[60316]: debug 2022-01-31T21:23:35.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:35.390819+0000) 2022-01-31T21:23:35.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:35 smithi167 conmon[54076]: debug 2022-01-31T21:23:35.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:35.386590+0000) 2022-01-31T21:23:35.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:35 smithi171 conmon[46715]: debug 2022-01-31T21:23:35.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:35.163642+0000) 2022-01-31T21:23:35.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:35 smithi171 conmon[51620]: debug 2022-01-31T21:23:35.742+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:35.743115+0000) 2022-01-31T21:23:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:35 smithi171 conmon[41853]: debug 2022-01-31T21:23:35.866+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:35.867577+0000) 2022-01-31T21:23:36.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:35 smithi167 conmon[49112]: debug 2022-01-31T21:23:35.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:35.718541+0000) 2022-01-31T21:23:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:36 smithi167 conmon[60316]: debug 2022-01-31T21:23:36.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:36.391018+0000) 2022-01-31T21:23:36.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:36 smithi167 conmon[54076]: debug 2022-01-31T21:23:36.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:36.386773+0000) 2022-01-31T21:23:36.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:36 smithi171 conmon[46715]: debug 2022-01-31T21:23:36.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:36.163801+0000) 2022-01-31T21:23:36.826 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:36 smithi171 conmon[51620]: debug 2022-01-31T21:23:36.742+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:36.743323+0000) 2022-01-31T21:23:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:36 smithi171 conmon[41853]: debug 2022-01-31T21:23:36.867+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:36.867766+0000) 2022-01-31T21:23:37.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:36 smithi167 conmon[49112]: debug 2022-01-31T21:23:36.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:36.718766+0000) 2022-01-31T21:23:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:37 smithi167 conmon[60316]: debug 2022-01-31T21:23:37.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:37.391218+0000) 2022-01-31T21:23:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:37 smithi167 conmon[54076]: debug 2022-01-31T21:23:37.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:37.386939+0000) 2022-01-31T21:23:37.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:37 smithi171 conmon[46715]: debug 2022-01-31T21:23:37.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:37.163971+0000) 2022-01-31T21:23:37.826 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:37 smithi171 conmon[51620]: debug 2022-01-31T21:23:37.742+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:37.743493+0000) 2022-01-31T21:23:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:37 smithi171 conmon[41853]: debug 2022-01-31T21:23:37.867+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:37.867935+0000) 2022-01-31T21:23:38.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:37 smithi167 conmon[49112]: debug 2022-01-31T21:23:37.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:37.718945+0000) 2022-01-31T21:23:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:38 smithi167 conmon[54076]: debug 2022-01-31T21:23:38.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:38.387118+0000) 2022-01-31T21:23:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:38 smithi167 conmon[60316]: debug 2022-01-31T21:23:38.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:38.391440+0000) 2022-01-31T21:23:38.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:38 smithi171 conmon[46715]: debug 2022-01-31T21:23:38.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:38.164160+0000) 2022-01-31T21:23:38.826 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:38 smithi171 conmon[35325]: debug 2022-01-31T21:23:38.671+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:23:38.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:38 smithi171 conmon[51620]: debug 2022-01-31T21:23:38.743+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:38.743747+0000) 2022-01-31T21:23:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:38 smithi171 conmon[41853]: debug 2022-01-31T21:23:38.867+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:38.868115+0000) 2022-01-31T21:23:39.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:38 smithi167 conmon[49112]: debug 2022-01-31T21:23:38.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:38.719125+0000) 2022-01-31T21:23:39.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:39 smithi167 conmon[49112]: debug 2022-01-31T21:23:39.217+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.217894+0000) 2022-01-31T21:23:39.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:39 smithi167 conmon[60316]: debug 2022-01-31T21:23:39.217+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.218358+0000) 2022-01-31T21:23:39.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:39 smithi167 conmon[60316]: debug 2022-01-31T21:23:39.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.391607+0000) 2022-01-31T21:23:39.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:39 smithi167 conmon[54076]: debug 2022-01-31T21:23:39.218+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.218745+0000) 2022-01-31T21:23:39.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:39 smithi167 conmon[54076]: debug 2022-01-31T21:23:39.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.387353+0000) 2022-01-31T21:23:39.458 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[35325]: debug 2022-01-31T21:23:39.230+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126215 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:39.458 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[41853]: debug 2022-01-31T21:23:39.217+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.218623+0000) 2022-01-31T21:23:39.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[46715]: debug 2022-01-31T21:23:39.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.164380+0000) 2022-01-31T21:23:39.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[46715]: debug 2022-01-31T21:23:39.217+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.218233+0000) 2022-01-31T21:23:39.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[51620]: debug 2022-01-31T21:23:39.217+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.218030+0000) 2022-01-31T21:23:39.826 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[51620]: debug 2022-01-31T21:23:39.743+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.743926+0000) 2022-01-31T21:23:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:39 smithi171 conmon[41853]: debug 2022-01-31T21:23:39.867+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.868257+0000) 2022-01-31T21:23:40.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:39 smithi167 conmon[49112]: debug 2022-01-31T21:23:39.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:39.719317+0000) 2022-01-31T21:23:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:40 smithi167 conmon[54076]: debug 2022-01-31T21:23:40.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:40.387519+0000) 2022-01-31T21:23:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:40 smithi167 conmon[60316]: debug 2022-01-31T21:23:40.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:40.391736+0000) 2022-01-31T21:23:40.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:40 smithi171 conmon[46715]: debug 2022-01-31T21:23:40.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:40.164594+0000) 2022-01-31T21:23:40.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:40 smithi171 conmon[51620]: debug 2022-01-31T21:23:40.743+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:40.744078+0000) 2022-01-31T21:23:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:40 smithi171 conmon[41853]: debug 2022-01-31T21:23:40.867+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:40.868411+0000) 2022-01-31T21:23:41.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:40 smithi167 conmon[49112]: debug 2022-01-31T21:23:40.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:40.719531+0000) 2022-01-31T21:23:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:41 smithi167 conmon[54076]: debug 2022-01-31T21:23:41.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:41.387679+0000) 2022-01-31T21:23:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:41 smithi167 conmon[60316]: debug 2022-01-31T21:23:41.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:41.391846+0000) 2022-01-31T21:23:41.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:41 smithi171 conmon[46715]: debug 2022-01-31T21:23:41.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:41.164731+0000) 2022-01-31T21:23:41.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:41 smithi171 conmon[51620]: debug 2022-01-31T21:23:41.743+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:41.744217+0000) 2022-01-31T21:23:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:41 smithi171 conmon[41853]: debug 2022-01-31T21:23:41.867+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:41.868635+0000) 2022-01-31T21:23:42.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:41 smithi167 conmon[49112]: debug 2022-01-31T21:23:41.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:41.719724+0000) 2022-01-31T21:23:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:42 smithi167 conmon[54076]: debug 2022-01-31T21:23:42.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:42.387835+0000) 2022-01-31T21:23:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:42 smithi167 conmon[60316]: debug 2022-01-31T21:23:42.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:42.392002+0000) 2022-01-31T21:23:42.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:42 smithi171 conmon[46715]: debug 2022-01-31T21:23:42.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:42.164861+0000) 2022-01-31T21:23:42.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:42 smithi171 conmon[51620]: debug 2022-01-31T21:23:42.743+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:42.744397+0000) 2022-01-31T21:23:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:42 smithi171 conmon[41853]: debug 2022-01-31T21:23:42.868+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:42.868839+0000) 2022-01-31T21:23:43.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:42 smithi167 conmon[49112]: debug 2022-01-31T21:23:42.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:42.719887+0000) 2022-01-31T21:23:43.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:43 smithi167 conmon[54076]: debug 2022-01-31T21:23:43.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:43.387956+0000) 2022-01-31T21:23:43.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:43 smithi167 conmon[60316]: debug 2022-01-31T21:23:43.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:43.392208+0000) 2022-01-31T21:23:43.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:43 smithi171 conmon[46715]: debug 2022-01-31T21:23:43.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:43.164980+0000) 2022-01-31T21:23:43.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:43 smithi171 conmon[51620]: debug 2022-01-31T21:23:43.743+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:43.744586+0000) 2022-01-31T21:23:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:43 smithi171 conmon[41853]: debug 2022-01-31T21:23:43.868+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:43.869033+0000) 2022-01-31T21:23:44.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:43 smithi167 conmon[49112]: debug 2022-01-31T21:23:43.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:43.720116+0000) 2022-01-31T21:23:44.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:44 smithi167 conmon[49112]: debug 2022-01-31T21:23:44.233+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.233824+0000) 2022-01-31T21:23:44.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:44 smithi167 conmon[54076]: debug 2022-01-31T21:23:44.233+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.233613+0000) 2022-01-31T21:23:44.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:44 smithi167 conmon[54076]: debug 2022-01-31T21:23:44.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.388153+0000) 2022-01-31T21:23:44.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:44 smithi167 conmon[60316]: debug 2022-01-31T21:23:44.233+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.233532+0000) 2022-01-31T21:23:44.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:44 smithi167 conmon[60316]: debug 2022-01-31T21:23:44.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.392396+0000) 2022-01-31T21:23:44.547 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[35325]: debug 2022-01-31T21:23:44.245+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126328 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:44.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[41853]: debug 2022-01-31T21:23:44.232+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.233135+0000) 2022-01-31T21:23:44.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[51620]: debug 2022-01-31T21:23:44.232+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.232778+0000) 2022-01-31T21:23:44.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[46715]: debug 2022-01-31T21:23:44.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.165202+0000) 2022-01-31T21:23:44.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[46715]: debug 2022-01-31T21:23:44.233+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.234567+0000) 2022-01-31T21:23:44.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[51620]: debug 2022-01-31T21:23:44.744+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.744801+0000) 2022-01-31T21:23:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:44 smithi171 conmon[41853]: debug 2022-01-31T21:23:44.868+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.869210+0000) 2022-01-31T21:23:45.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:44 smithi167 conmon[49112]: debug 2022-01-31T21:23:44.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:44.720348+0000) 2022-01-31T21:23:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:45 smithi167 conmon[54076]: debug 2022-01-31T21:23:45.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:45.388285+0000) 2022-01-31T21:23:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:45 smithi167 conmon[60316]: debug 2022-01-31T21:23:45.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:45.392571+0000) 2022-01-31T21:23:45.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:45 smithi171 conmon[46715]: debug 2022-01-31T21:23:45.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:45.165388+0000) 2022-01-31T21:23:45.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:45 smithi171 conmon[51620]: debug 2022-01-31T21:23:45.744+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:45.744975+0000) 2022-01-31T21:23:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:45 smithi171 conmon[41853]: debug 2022-01-31T21:23:45.868+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:45.869378+0000) 2022-01-31T21:23:46.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:45 smithi167 conmon[49112]: debug 2022-01-31T21:23:45.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:45.720549+0000) 2022-01-31T21:23:46.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:46 smithi167 conmon[54076]: debug 2022-01-31T21:23:46.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:46.388435+0000) 2022-01-31T21:23:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:46 smithi167 conmon[60316]: debug 2022-01-31T21:23:46.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:46.392692+0000) 2022-01-31T21:23:46.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:46 smithi171 conmon[46715]: debug 2022-01-31T21:23:46.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:46.165524+0000) 2022-01-31T21:23:46.828 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:46 smithi171 conmon[51620]: debug 2022-01-31T21:23:46.744+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:46.745187+0000) 2022-01-31T21:23:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:46 smithi171 conmon[41853]: debug 2022-01-31T21:23:46.868+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:46.869581+0000) 2022-01-31T21:23:47.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:46 smithi167 conmon[49112]: debug 2022-01-31T21:23:46.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:46.720729+0000) 2022-01-31T21:23:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:47 smithi167 conmon[60316]: debug 2022-01-31T21:23:47.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:47.392841+0000) 2022-01-31T21:23:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:47 smithi167 conmon[54076]: debug 2022-01-31T21:23:47.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:47.388621+0000) 2022-01-31T21:23:47.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:47 smithi171 conmon[46715]: debug 2022-01-31T21:23:47.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:47.165727+0000) 2022-01-31T21:23:47.828 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:47 smithi171 conmon[51620]: debug 2022-01-31T21:23:47.744+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:47.745397+0000) 2022-01-31T21:23:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:47 smithi171 conmon[41853]: debug 2022-01-31T21:23:47.869+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:47.869786+0000) 2022-01-31T21:23:48.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:47 smithi167 conmon[49112]: debug 2022-01-31T21:23:47.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:47.720944+0000) 2022-01-31T21:23:48.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:48 smithi167 conmon[54076]: debug 2022-01-31T21:23:48.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:48.388742+0000) 2022-01-31T21:23:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:48 smithi167 conmon[60316]: debug 2022-01-31T21:23:48.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:48.393026+0000) 2022-01-31T21:23:48.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:48 smithi171 conmon[46715]: debug 2022-01-31T21:23:48.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:48.165937+0000) 2022-01-31T21:23:48.828 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:48 smithi171 conmon[51620]: debug 2022-01-31T21:23:48.744+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:48.745533+0000) 2022-01-31T21:23:49.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:48 smithi171 conmon[41853]: debug 2022-01-31T21:23:48.869+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:48.869965+0000) 2022-01-31T21:23:49.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:48 smithi167 conmon[49112]: debug 2022-01-31T21:23:48.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:48.721170+0000) 2022-01-31T21:23:49.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:49 smithi167 conmon[49112]: debug 2022-01-31T21:23:49.248+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.248992+0000) 2022-01-31T21:23:49.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:49 smithi167 conmon[54076]: debug 2022-01-31T21:23:49.249+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.250180+0000) 2022-01-31T21:23:49.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:49 smithi167 conmon[54076]: debug 2022-01-31T21:23:49.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.388923+0000) 2022-01-31T21:23:49.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:49 smithi167 conmon[60316]: debug 2022-01-31T21:23:49.248+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.249250+0000) 2022-01-31T21:23:49.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:49 smithi167 conmon[60316]: debug 2022-01-31T21:23:49.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.393229+0000) 2022-01-31T21:23:49.547 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[35325]: debug 2022-01-31T21:23:49.260+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126437 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:49.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[41853]: debug 2022-01-31T21:23:49.248+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.249328+0000) 2022-01-31T21:23:49.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[46715]: debug 2022-01-31T21:23:49.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.166097+0000) 2022-01-31T21:23:49.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[46715]: debug 2022-01-31T21:23:49.250+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.250804+0000) 2022-01-31T21:23:49.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[51620]: debug 2022-01-31T21:23:49.247+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.248738+0000) 2022-01-31T21:23:49.828 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[51620]: debug 2022-01-31T21:23:49.744+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.745701+0000) 2022-01-31T21:23:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:49 smithi171 conmon[41853]: debug 2022-01-31T21:23:49.869+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.870156+0000) 2022-01-31T21:23:50.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:49 smithi167 conmon[49112]: debug 2022-01-31T21:23:49.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:49.721351+0000) 2022-01-31T21:23:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:50 smithi167 conmon[54076]: debug 2022-01-31T21:23:50.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:50.389106+0000) 2022-01-31T21:23:50.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:50 smithi167 conmon[60316]: debug 2022-01-31T21:23:50.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:50.393428+0000) 2022-01-31T21:23:50.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:50 smithi171 conmon[46715]: debug 2022-01-31T21:23:50.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:50.166295+0000) 2022-01-31T21:23:50.828 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:50 smithi171 conmon[51620]: debug 2022-01-31T21:23:50.745+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:50.745844+0000) 2022-01-31T21:23:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:50 smithi171 conmon[41853]: debug 2022-01-31T21:23:50.869+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:50.870317+0000) 2022-01-31T21:23:51.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:50 smithi167 conmon[49112]: debug 2022-01-31T21:23:50.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:50.721549+0000) 2022-01-31T21:23:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:51 smithi167 conmon[54076]: debug 2022-01-31T21:23:51.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:51.389287+0000) 2022-01-31T21:23:51.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:51 smithi167 conmon[60316]: debug 2022-01-31T21:23:51.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:51.393596+0000) 2022-01-31T21:23:51.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:51 smithi171 conmon[46715]: debug 2022-01-31T21:23:51.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:51.166451+0000) 2022-01-31T21:23:51.828 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:51 smithi171 conmon[51620]: debug 2022-01-31T21:23:51.745+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:51.746030+0000) 2022-01-31T21:23:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:51 smithi171 conmon[41853]: debug 2022-01-31T21:23:51.869+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:51.870528+0000) 2022-01-31T21:23:52.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:51 smithi167 conmon[49112]: debug 2022-01-31T21:23:51.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:51.721712+0000) 2022-01-31T21:23:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:52 smithi167 conmon[54076]: debug 2022-01-31T21:23:52.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:52.389465+0000) 2022-01-31T21:23:52.751 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:52 smithi167 conmon[60316]: debug 2022-01-31T21:23:52.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:52.393763+0000) 2022-01-31T21:23:52.752 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:52 smithi171 conmon[46715]: debug 2022-01-31T21:23:52.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:52.166637+0000) 2022-01-31T21:23:52.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:52 smithi171 conmon[51620]: debug 2022-01-31T21:23:52.745+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:52.746221+0000) 2022-01-31T21:23:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:52 smithi171 conmon[41853]: debug 2022-01-31T21:23:52.870+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:52.870748+0000) 2022-01-31T21:23:53.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:52 smithi167 conmon[49112]: debug 2022-01-31T21:23:52.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:52.721872+0000) 2022-01-31T21:23:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:53 smithi167 conmon[54076]: debug 2022-01-31T21:23:53.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:53.389591+0000) 2022-01-31T21:23:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:53 smithi167 conmon[60316]: debug 2022-01-31T21:23:53.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:53.393957+0000) 2022-01-31T21:23:53.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:53 smithi171 conmon[46715]: debug 2022-01-31T21:23:53.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:53.166858+0000) 2022-01-31T21:23:53.829 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:53 smithi171 conmon[35325]: debug 2022-01-31T21:23:53.672+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:23:53.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:53 smithi171 conmon[51620]: debug 2022-01-31T21:23:53.745+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:53.746381+0000) 2022-01-31T21:23:54.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:53 smithi171 conmon[41853]: debug 2022-01-31T21:23:53.870+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:53.870945+0000) 2022-01-31T21:23:54.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:53 smithi167 conmon[49112]: debug 2022-01-31T21:23:53.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:53.722049+0000) 2022-01-31T21:23:54.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[49112]: debug 2022-01-31T21:23:54.263+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.264609+0000) 2022-01-31T21:23:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[60316]: debug 2022-01-31T21:23:54.263+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.264662+0000) 2022-01-31T21:23:54.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[60316]: debug 2022-01-31T21:23:54.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.394145+0000) 2022-01-31T21:23:54.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[54076]: debug 2022-01-31T21:23:54.262+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.264415+0000) 2022-01-31T21:23:54.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[54076]: 2022-01-31T21:23:54.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[54076]: debug 2022-01-31T21:23:54.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.389734+0000) 2022-01-31T21:23:54.549 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[35325]: debug 2022-01-31T21:23:54.276+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:54.549 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[41853]: debug 2022-01-31T21:23:54.262+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.263476+0000) 2022-01-31T21:23:54.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[46715]: debug 2022-01-31T21:23:54.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.167061+0000) 2022-01-31T21:23:54.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[46715]: debug 2022-01-31T21:23:54.263+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.264724+0000) 2022-01-31T21:23:54.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[51620]: debug 2022-01-31T21:23:54.262+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.263419+0000) 2022-01-31T21:23:54.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[51620]: debug 2022-01-31T21:23:54.745+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.746567+0000) 2022-01-31T21:23:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:54 smithi171 conmon[41853]: debug 2022-01-31T21:23:54.870+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.871086+0000) 2022-01-31T21:23:55.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:54 smithi167 conmon[49112]: debug 2022-01-31T21:23:54.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:54.722215+0000) 2022-01-31T21:23:55.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:55 smithi167 conmon[54076]: debug 2022-01-31T21:23:55.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:55.389887+0000) 2022-01-31T21:23:55.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:55 smithi167 conmon[60316]: debug 2022-01-31T21:23:55.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:55.394302+0000) 2022-01-31T21:23:55.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:55 smithi171 conmon[46715]: debug 2022-01-31T21:23:55.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:55.167271+0000) 2022-01-31T21:23:55.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:55 smithi171 conmon[51620]: debug 2022-01-31T21:23:55.746+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:55.746777+0000) 2022-01-31T21:23:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:55 smithi171 conmon[41853]: debug 2022-01-31T21:23:55.870+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:55.871225+0000) 2022-01-31T21:23:56.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:55 smithi167 conmon[49112]: debug 2022-01-31T21:23:55.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:55.722407+0000) 2022-01-31T21:23:56.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:56 smithi167 conmon[54076]: debug 2022-01-31T21:23:56.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:56.390078+0000) 2022-01-31T21:23:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:56 smithi167 conmon[60316]: debug 2022-01-31T21:23:56.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:56.394490+0000) 2022-01-31T21:23:56.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:56 smithi171 conmon[46715]: debug 2022-01-31T21:23:56.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:56.167420+0000) 2022-01-31T21:23:56.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:56 smithi171 conmon[51620]: debug 2022-01-31T21:23:56.746+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:56.746951+0000) 2022-01-31T21:23:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:56 smithi171 conmon[41853]: debug 2022-01-31T21:23:56.870+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:56.871464+0000) 2022-01-31T21:23:57.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:56 smithi167 conmon[49112]: debug 2022-01-31T21:23:56.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:56.722604+0000) 2022-01-31T21:23:57.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:57 smithi167 conmon[54076]: debug 2022-01-31T21:23:57.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:57.390263+0000) 2022-01-31T21:23:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:57 smithi167 conmon[60316]: debug 2022-01-31T21:23:57.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:57.394645+0000) 2022-01-31T21:23:57.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:57 smithi171 conmon[46715]: debug 2022-01-31T21:23:57.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:57.167606+0000) 2022-01-31T21:23:57.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:57 smithi171 conmon[51620]: debug 2022-01-31T21:23:57.746+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:57.747177+0000) 2022-01-31T21:23:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:57 smithi171 conmon[41853]: debug 2022-01-31T21:23:57.871+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:57.871618+0000) 2022-01-31T21:23:58.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:57 smithi167 conmon[49112]: debug 2022-01-31T21:23:57.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:57.722760+0000) 2022-01-31T21:23:58.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:58 smithi167 conmon[54076]: debug 2022-01-31T21:23:58.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:58.390455+0000) 2022-01-31T21:23:58.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:58 smithi167 conmon[60316]: debug 2022-01-31T21:23:58.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:58.394797+0000) 2022-01-31T21:23:58.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:58 smithi171 conmon[46715]: debug 2022-01-31T21:23:58.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:58.167825+0000) 2022-01-31T21:23:58.829 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:58 smithi171 conmon[51620]: debug 2022-01-31T21:23:58.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:58.747393+0000) 2022-01-31T21:23:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:58 smithi171 conmon[41853]: debug 2022-01-31T21:23:58.871+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:58.871835+0000) 2022-01-31T21:23:59.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:58 smithi167 conmon[49112]: debug 2022-01-31T21:23:58.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:58.722877+0000) 2022-01-31T21:23:59.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:59 smithi167 conmon[49112]: debug 2022-01-31T21:23:59.277+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.279274+0000) 2022-01-31T21:23:59.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:59 smithi167 conmon[54076]: debug 2022-01-31T21:23:59.285+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.286841+0000) 2022-01-31T21:23:59.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:23:59 smithi167 conmon[54076]: debug 2022-01-31T21:23:59.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.390678+0000) 2022-01-31T21:23:59.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:59 smithi167 conmon[60316]: debug 2022-01-31T21:23:59.279+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.280929+0000) 2022-01-31T21:23:59.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:23:59 smithi167 conmon[60316]: debug 2022-01-31T21:23:59.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.395004+0000) 2022-01-31T21:23:59.550 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[35325]: debug 2022-01-31T21:23:59.295+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126657 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:23:59.550 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[41853]: debug 2022-01-31T21:23:59.279+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.279762+0000) 2022-01-31T21:23:59.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[51620]: debug 2022-01-31T21:23:59.278+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.279492+0000) 2022-01-31T21:23:59.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[46715]: debug 2022-01-31T21:23:59.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.168059+0000) 2022-01-31T21:23:59.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[46715]: debug 2022-01-31T21:23:59.279+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.280630+0000) 2022-01-31T21:23:59.830 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[51620]: debug 2022-01-31T21:23:59.746+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.747568+0000) 2022-01-31T21:24:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:23:59 smithi171 conmon[41853]: debug 2022-01-31T21:23:59.871+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.871997+0000) 2022-01-31T21:24:00.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:23:59 smithi167 conmon[49112]: debug 2022-01-31T21:23:59.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:23:59.723069+0000) 2022-01-31T21:24:00.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:00 smithi167 conmon[60316]: debug 2022-01-31T21:24:00.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:00.395182+0000) 2022-01-31T21:24:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:00 smithi167 conmon[54076]: debug 2022-01-31T21:24:00.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:00.390849+0000) 2022-01-31T21:24:00.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:00 smithi171 conmon[46715]: debug 2022-01-31T21:24:00.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:00.168268+0000) 2022-01-31T21:24:00.830 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:00 smithi171 conmon[51620]: debug 2022-01-31T21:24:00.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:00.747720+0000) 2022-01-31T21:24:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:00 smithi171 conmon[41853]: debug 2022-01-31T21:24:00.871+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:00.872113+0000) 2022-01-31T21:24:01.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:00 smithi167 conmon[49112]: debug 2022-01-31T21:24:00.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:00.723240+0000) 2022-01-31T21:24:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:01 smithi167 conmon[54076]: debug 2022-01-31T21:24:01.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:01.391004+0000) 2022-01-31T21:24:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:01 smithi167 conmon[60316]: debug 2022-01-31T21:24:01.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:01.395383+0000) 2022-01-31T21:24:01.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:01 smithi171 conmon[46715]: debug 2022-01-31T21:24:01.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:01.168365+0000) 2022-01-31T21:24:01.830 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:01 smithi171 conmon[51620]: debug 2022-01-31T21:24:01.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:01.747921+0000) 2022-01-31T21:24:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:01 smithi171 conmon[41853]: debug 2022-01-31T21:24:01.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:01.872265+0000) 2022-01-31T21:24:02.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:01 smithi167 conmon[49112]: debug 2022-01-31T21:24:01.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:01.723466+0000) 2022-01-31T21:24:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:02 smithi167 conmon[54076]: debug 2022-01-31T21:24:02.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:02.391201+0000) 2022-01-31T21:24:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:02 smithi167 conmon[60316]: debug 2022-01-31T21:24:02.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:02.395577+0000) 2022-01-31T21:24:02.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:02 smithi171 conmon[46715]: debug 2022-01-31T21:24:02.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:02.168515+0000) 2022-01-31T21:24:02.830 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:02 smithi171 conmon[51620]: debug 2022-01-31T21:24:02.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:02.748112+0000) 2022-01-31T21:24:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:02 smithi171 conmon[41853]: debug 2022-01-31T21:24:02.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:02.872457+0000) 2022-01-31T21:24:03.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:02 smithi167 conmon[49112]: debug 2022-01-31T21:24:02.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:02.723675+0000) 2022-01-31T21:24:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:03 smithi167 conmon[54076]: debug 2022-01-31T21:24:03.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:03.391331+0000) 2022-01-31T21:24:03.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:03 smithi167 conmon[60316]: debug 2022-01-31T21:24:03.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:03.395679+0000) 2022-01-31T21:24:03.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:03 smithi171 conmon[46715]: debug 2022-01-31T21:24:03.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:03.168694+0000) 2022-01-31T21:24:03.830 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:03 smithi171 conmon[51620]: debug 2022-01-31T21:24:03.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:03.748284+0000) 2022-01-31T21:24:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:03 smithi171 conmon[41853]: debug 2022-01-31T21:24:03.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:03.872611+0000) 2022-01-31T21:24:04.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:03 smithi167 conmon[49112]: debug 2022-01-31T21:24:03.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:03.723845+0000) 2022-01-31T21:24:04.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:04 smithi167 conmon[49112]: debug 2022-01-31T21:24:04.297+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.298881+0000) 2022-01-31T21:24:04.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:04 smithi167 conmon[54076]: debug 2022-01-31T21:24:04.298+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.299600+0000) 2022-01-31T21:24:04.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:04 smithi167 conmon[54076]: debug 2022-01-31T21:24:04.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.391446+0000) 2022-01-31T21:24:04.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:04 smithi167 conmon[60316]: debug 2022-01-31T21:24:04.298+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.299846+0000) 2022-01-31T21:24:04.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:04 smithi167 conmon[60316]: debug 2022-01-31T21:24:04.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.395832+0000) 2022-01-31T21:24:04.551 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[35325]: debug 2022-01-31T21:24:04.310+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126770 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:04.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[41853]: debug 2022-01-31T21:24:04.297+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.298595+0000) 2022-01-31T21:24:04.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[51620]: debug 2022-01-31T21:24:04.298+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.299291+0000) 2022-01-31T21:24:04.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[46715]: debug 2022-01-31T21:24:04.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.168859+0000) 2022-01-31T21:24:04.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[46715]: debug 2022-01-31T21:24:04.297+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.298724+0000) 2022-01-31T21:24:04.831 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[51620]: debug 2022-01-31T21:24:04.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.748477+0000) 2022-01-31T21:24:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:04 smithi171 conmon[41853]: debug 2022-01-31T21:24:04.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.872775+0000) 2022-01-31T21:24:05.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:04 smithi167 conmon[49112]: debug 2022-01-31T21:24:04.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:04.724052+0000) 2022-01-31T21:24:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:05 smithi167 conmon[60316]: debug 2022-01-31T21:24:05.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:05.395989+0000) 2022-01-31T21:24:05.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:05 smithi167 conmon[54076]: debug 2022-01-31T21:24:05.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:05.391547+0000) 2022-01-31T21:24:05.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:05 smithi171 conmon[46715]: debug 2022-01-31T21:24:05.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:05.168960+0000) 2022-01-31T21:24:05.831 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:05 smithi171 conmon[51620]: debug 2022-01-31T21:24:05.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:05.748640+0000) 2022-01-31T21:24:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:05 smithi171 conmon[41853]: debug 2022-01-31T21:24:05.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:05.872889+0000) 2022-01-31T21:24:06.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:05 smithi167 conmon[49112]: debug 2022-01-31T21:24:05.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:05.724193+0000) 2022-01-31T21:24:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:06 smithi167 conmon[60316]: debug 2022-01-31T21:24:06.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:06.396165+0000) 2022-01-31T21:24:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:06 smithi167 conmon[54076]: debug 2022-01-31T21:24:06.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:06.391713+0000) 2022-01-31T21:24:06.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:06 smithi171 conmon[46715]: debug 2022-01-31T21:24:06.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:06.169055+0000) 2022-01-31T21:24:06.831 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:06 smithi171 conmon[51620]: debug 2022-01-31T21:24:06.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:06.748833+0000) 2022-01-31T21:24:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:06 smithi171 conmon[41853]: debug 2022-01-31T21:24:06.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:06.873072+0000) 2022-01-31T21:24:07.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:06 smithi167 conmon[49112]: debug 2022-01-31T21:24:06.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:06.724404+0000) 2022-01-31T21:24:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:07 smithi167 conmon[54076]: debug 2022-01-31T21:24:07.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:07.391870+0000) 2022-01-31T21:24:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:07 smithi167 conmon[60316]: debug 2022-01-31T21:24:07.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:07.396366+0000) 2022-01-31T21:24:07.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:07 smithi171 conmon[46715]: debug 2022-01-31T21:24:07.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:07.169229+0000) 2022-01-31T21:24:07.831 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:07 smithi171 conmon[51620]: debug 2022-01-31T21:24:07.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:07.748987+0000) 2022-01-31T21:24:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:07 smithi171 conmon[41853]: debug 2022-01-31T21:24:07.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:07.873256+0000) 2022-01-31T21:24:08.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:07 smithi167 conmon[49112]: debug 2022-01-31T21:24:07.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:07.724606+0000) 2022-01-31T21:24:08.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:08 smithi167 conmon[54076]: debug 2022-01-31T21:24:08.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:08.392088+0000) 2022-01-31T21:24:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:08 smithi167 conmon[60316]: debug 2022-01-31T21:24:08.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:08.396598+0000) 2022-01-31T21:24:08.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:08 smithi171 conmon[46715]: debug 2022-01-31T21:24:08.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:08.169416+0000) 2022-01-31T21:24:08.831 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:08 smithi171 conmon[35325]: debug 2022-01-31T21:24:08.673+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:24:08.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:08 smithi171 conmon[51620]: debug 2022-01-31T21:24:08.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:08.749151+0000) 2022-01-31T21:24:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:08 smithi171 conmon[41853]: debug 2022-01-31T21:24:08.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:08.873439+0000) 2022-01-31T21:24:09.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:08 smithi167 conmon[49112]: debug 2022-01-31T21:24:08.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:08.724766+0000) 2022-01-31T21:24:09.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:09 smithi167 conmon[49112]: debug 2022-01-31T21:24:09.311+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.313426+0000) 2022-01-31T21:24:09.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:09 smithi167 conmon[54076]: debug 2022-01-31T21:24:09.312+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.314398+0000) 2022-01-31T21:24:09.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:09 smithi167 conmon[54076]: debug 2022-01-31T21:24:09.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.392257+0000) 2022-01-31T21:24:09.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:09 smithi167 conmon[60316]: debug 2022-01-31T21:24:09.313+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.314619+0000) 2022-01-31T21:24:09.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:09 smithi167 conmon[60316]: debug 2022-01-31T21:24:09.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.396743+0000) 2022-01-31T21:24:09.551 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[35325]: debug 2022-01-31T21:24:09.325+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126878 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:09.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[41853]: debug 2022-01-31T21:24:09.313+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.313804+0000) 2022-01-31T21:24:09.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[51620]: debug 2022-01-31T21:24:09.313+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.314517+0000) 2022-01-31T21:24:09.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[46715]: debug 2022-01-31T21:24:09.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.169585+0000) 2022-01-31T21:24:09.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[46715]: debug 2022-01-31T21:24:09.314+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.314834+0000) 2022-01-31T21:24:09.831 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[51620]: debug 2022-01-31T21:24:09.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.749340+0000) 2022-01-31T21:24:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:09 smithi171 conmon[41853]: debug 2022-01-31T21:24:09.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.873590+0000) 2022-01-31T21:24:10.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:09 smithi167 conmon[49112]: debug 2022-01-31T21:24:09.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:09.724942+0000) 2022-01-31T21:24:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:10 smithi167 conmon[54076]: debug 2022-01-31T21:24:10.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:10.392426+0000) 2022-01-31T21:24:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:10 smithi167 conmon[60316]: debug 2022-01-31T21:24:10.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:10.396884+0000) 2022-01-31T21:24:10.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:10 smithi171 conmon[46715]: debug 2022-01-31T21:24:10.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:10.169768+0000) 2022-01-31T21:24:10.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:10 smithi171 conmon[51620]: debug 2022-01-31T21:24:10.747+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:10.749493+0000) 2022-01-31T21:24:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:10 smithi171 conmon[41853]: debug 2022-01-31T21:24:10.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:10.873750+0000) 2022-01-31T21:24:11.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:10 smithi167 conmon[49112]: debug 2022-01-31T21:24:10.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:10.725046+0000) 2022-01-31T21:24:11.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:11 smithi167 conmon[54076]: debug 2022-01-31T21:24:11.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:11.392616+0000) 2022-01-31T21:24:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:11 smithi167 conmon[60316]: debug 2022-01-31T21:24:11.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:11.397075+0000) 2022-01-31T21:24:11.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:11 smithi171 conmon[46715]: debug 2022-01-31T21:24:11.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:11.169928+0000) 2022-01-31T21:24:11.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:11 smithi171 conmon[51620]: debug 2022-01-31T21:24:11.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:11.749682+0000) 2022-01-31T21:24:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:11 smithi171 conmon[41853]: debug 2022-01-31T21:24:11.872+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:11.873926+0000) 2022-01-31T21:24:12.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:11 smithi167 conmon[49112]: debug 2022-01-31T21:24:11.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:11.725246+0000) 2022-01-31T21:24:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:12 smithi167 conmon[54076]: debug 2022-01-31T21:24:12.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:12.392804+0000) 2022-01-31T21:24:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:12 smithi167 conmon[60316]: debug 2022-01-31T21:24:12.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:12.397300+0000) 2022-01-31T21:24:12.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:12 smithi171 conmon[46715]: debug 2022-01-31T21:24:12.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:12.170118+0000) 2022-01-31T21:24:12.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:12 smithi171 conmon[51620]: debug 2022-01-31T21:24:12.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:12.749860+0000) 2022-01-31T21:24:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:12 smithi171 conmon[41853]: debug 2022-01-31T21:24:12.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:12.874098+0000) 2022-01-31T21:24:13.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:12 smithi167 conmon[49112]: debug 2022-01-31T21:24:12.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:12.725480+0000) 2022-01-31T21:24:13.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:13 smithi167 conmon[60316]: debug 2022-01-31T21:24:13.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:13.397484+0000) 2022-01-31T21:24:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:13 smithi167 conmon[54076]: debug 2022-01-31T21:24:13.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:13.392985+0000) 2022-01-31T21:24:13.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:13 smithi171 conmon[46715]: debug 2022-01-31T21:24:13.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:13.170328+0000) 2022-01-31T21:24:13.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:13 smithi171 conmon[51620]: debug 2022-01-31T21:24:13.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:13.749984+0000) 2022-01-31T21:24:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:13 smithi171 conmon[41853]: debug 2022-01-31T21:24:13.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:13.874260+0000) 2022-01-31T21:24:14.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:13 smithi167 conmon[49112]: debug 2022-01-31T21:24:13.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:13.725604+0000) 2022-01-31T21:24:14.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:14 smithi167 conmon[49112]: debug 2022-01-31T21:24:14.327+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.328903+0000) 2022-01-31T21:24:14.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:14 smithi167 conmon[54076]: debug 2022-01-31T21:24:14.328+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.330172+0000) 2022-01-31T21:24:14.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:14 smithi167 conmon[54076]: debug 2022-01-31T21:24:14.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.393138+0000) 2022-01-31T21:24:14.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:14 smithi167 conmon[60316]: debug 2022-01-31T21:24:14.328+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.329755+0000) 2022-01-31T21:24:14.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:14 smithi167 conmon[60316]: debug 2022-01-31T21:24:14.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.397627+0000) 2022-01-31T21:24:14.552 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[35325]: debug 2022-01-31T21:24:14.340+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 126990 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:14.553 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[41853]: debug 2022-01-31T21:24:14.326+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.328281+0000) 2022-01-31T21:24:14.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[46715]: debug 2022-01-31T21:24:14.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.170522+0000) 2022-01-31T21:24:14.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[46715]: debug 2022-01-31T21:24:14.328+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.329962+0000) 2022-01-31T21:24:14.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[51620]: debug 2022-01-31T21:24:14.328+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.329934+0000) 2022-01-31T21:24:14.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[51620]: debug 2022-01-31T21:24:14.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.750165+0000) 2022-01-31T21:24:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:14 smithi171 conmon[41853]: debug 2022-01-31T21:24:14.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.874385+0000) 2022-01-31T21:24:15.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:14 smithi167 conmon[49112]: debug 2022-01-31T21:24:14.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:14.725752+0000) 2022-01-31T21:24:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:15 smithi167 conmon[54076]: debug 2022-01-31T21:24:15.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:15.393320+0000) 2022-01-31T21:24:15.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:15 smithi167 conmon[60316]: debug 2022-01-31T21:24:15.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:15.397733+0000) 2022-01-31T21:24:15.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:15 smithi171 conmon[46715]: debug 2022-01-31T21:24:15.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:15.170674+0000) 2022-01-31T21:24:15.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:15 smithi171 conmon[51620]: debug 2022-01-31T21:24:15.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:15.750326+0000) 2022-01-31T21:24:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:15 smithi171 conmon[41853]: debug 2022-01-31T21:24:15.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:15.874548+0000) 2022-01-31T21:24:16.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:15 smithi167 conmon[49112]: debug 2022-01-31T21:24:15.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:15.725888+0000) 2022-01-31T21:24:16.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:16 smithi167 conmon[54076]: debug 2022-01-31T21:24:16.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:16.393474+0000) 2022-01-31T21:24:16.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:16 smithi167 conmon[60316]: debug 2022-01-31T21:24:16.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:16.397885+0000) 2022-01-31T21:24:16.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:16 smithi171 conmon[46715]: debug 2022-01-31T21:24:16.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:16.170838+0000) 2022-01-31T21:24:16.833 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:16 smithi171 conmon[51620]: debug 2022-01-31T21:24:16.748+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:16.750512+0000) 2022-01-31T21:24:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:16 smithi171 conmon[41853]: debug 2022-01-31T21:24:16.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:16.874704+0000) 2022-01-31T21:24:17.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:16 smithi167 conmon[49112]: debug 2022-01-31T21:24:16.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:16.726081+0000) 2022-01-31T21:24:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:17 smithi167 conmon[54076]: debug 2022-01-31T21:24:17.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:17.393665+0000) 2022-01-31T21:24:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:17 smithi167 conmon[60316]: debug 2022-01-31T21:24:17.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:17.398054+0000) 2022-01-31T21:24:17.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:17 smithi171 conmon[46715]: debug 2022-01-31T21:24:17.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:17.171000+0000) 2022-01-31T21:24:17.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:17 smithi171 conmon[51620]: debug 2022-01-31T21:24:17.749+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:17.750694+0000) 2022-01-31T21:24:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:17 smithi171 conmon[41853]: debug 2022-01-31T21:24:17.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:17.874894+0000) 2022-01-31T21:24:18.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:17 smithi167 conmon[49112]: debug 2022-01-31T21:24:17.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:17.726310+0000) 2022-01-31T21:24:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:18 smithi167 conmon[54076]: debug 2022-01-31T21:24:18.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:18.393861+0000) 2022-01-31T21:24:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:18 smithi167 conmon[60316]: debug 2022-01-31T21:24:18.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:18.398234+0000) 2022-01-31T21:24:18.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:18 smithi171 conmon[46715]: debug 2022-01-31T21:24:18.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:18.171220+0000) 2022-01-31T21:24:18.833 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:18 smithi171 conmon[51620]: debug 2022-01-31T21:24:18.750+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:18.750905+0000) 2022-01-31T21:24:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:18 smithi171 conmon[41853]: debug 2022-01-31T21:24:18.873+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:18.875035+0000) 2022-01-31T21:24:19.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:18 smithi167 conmon[49112]: debug 2022-01-31T21:24:18.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:18.726497+0000) 2022-01-31T21:24:19.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:19 smithi167 conmon[49112]: debug 2022-01-31T21:24:19.344+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.345067+0000) 2022-01-31T21:24:19.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:19 smithi167 conmon[54076]: debug 2022-01-31T21:24:19.344+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.345214+0000) 2022-01-31T21:24:19.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:19 smithi167 conmon[54076]: debug 2022-01-31T21:24:19.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.394045+0000) 2022-01-31T21:24:19.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:19 smithi167 conmon[60316]: debug 2022-01-31T21:24:19.344+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.345464+0000) 2022-01-31T21:24:19.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:19 smithi167 conmon[60316]: debug 2022-01-31T21:24:19.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.398448+0000) 2022-01-31T21:24:19.553 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[35325]: debug 2022-01-31T21:24:19.356+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127098 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:19.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[41853]: debug 2022-01-31T21:24:19.342+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.344029+0000) 2022-01-31T21:24:19.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[51620]: debug 2022-01-31T21:24:19.343+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.345118+0000) 2022-01-31T21:24:19.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[46715]: debug 2022-01-31T21:24:19.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.171403+0000) 2022-01-31T21:24:19.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[46715]: debug 2022-01-31T21:24:19.344+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.345748+0000) 2022-01-31T21:24:19.833 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[51620]: debug 2022-01-31T21:24:19.750+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.751118+0000) 2022-01-31T21:24:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:19 smithi171 conmon[41853]: debug 2022-01-31T21:24:19.874+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.875229+0000) 2022-01-31T21:24:20.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:19 smithi167 conmon[49112]: debug 2022-01-31T21:24:19.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:19.726698+0000) 2022-01-31T21:24:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:20 smithi167 conmon[60316]: debug 2022-01-31T21:24:20.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:20.398618+0000) 2022-01-31T21:24:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:20 smithi167 conmon[54076]: debug 2022-01-31T21:24:20.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:20.394218+0000) 2022-01-31T21:24:20.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:20 smithi171 conmon[46715]: debug 2022-01-31T21:24:20.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:20.171586+0000) 2022-01-31T21:24:20.833 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:20 smithi171 conmon[51620]: debug 2022-01-31T21:24:20.750+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:20.751305+0000) 2022-01-31T21:24:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:20 smithi171 conmon[41853]: debug 2022-01-31T21:24:20.874+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:20.875365+0000) 2022-01-31T21:24:21.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:20 smithi167 conmon[49112]: debug 2022-01-31T21:24:20.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:20.726839+0000) 2022-01-31T21:24:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:21 smithi167 conmon[60316]: debug 2022-01-31T21:24:21.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:21.398746+0000) 2022-01-31T21:24:21.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:21 smithi167 conmon[54076]: debug 2022-01-31T21:24:21.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:21.394413+0000) 2022-01-31T21:24:21.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:21 smithi171 conmon[46715]: debug 2022-01-31T21:24:21.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:21.171773+0000) 2022-01-31T21:24:21.833 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:21 smithi171 conmon[51620]: debug 2022-01-31T21:24:21.750+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:21.751516+0000) 2022-01-31T21:24:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:21 smithi171 conmon[41853]: debug 2022-01-31T21:24:21.874+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:21.875572+0000) 2022-01-31T21:24:22.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:21 smithi167 conmon[49112]: debug 2022-01-31T21:24:21.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:21.727008+0000) 2022-01-31T21:24:22.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:22 smithi167 conmon[54076]: debug 2022-01-31T21:24:22.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:22.394593+0000) 2022-01-31T21:24:22.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:22 smithi167 conmon[60316]: debug 2022-01-31T21:24:22.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:22.398895+0000) 2022-01-31T21:24:22.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:22 smithi171 conmon[46715]: debug 2022-01-31T21:24:22.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:22.171937+0000) 2022-01-31T21:24:22.834 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:22 smithi171 conmon[51620]: debug 2022-01-31T21:24:22.750+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:22.751636+0000) 2022-01-31T21:24:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:22 smithi171 conmon[41853]: debug 2022-01-31T21:24:22.875+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:22.875784+0000) 2022-01-31T21:24:23.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:22 smithi167 conmon[49112]: debug 2022-01-31T21:24:22.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:22.727203+0000) 2022-01-31T21:24:23.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:23 smithi167 conmon[54076]: debug 2022-01-31T21:24:23.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:23.394761+0000) 2022-01-31T21:24:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:23 smithi167 conmon[60316]: debug 2022-01-31T21:24:23.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:23.399070+0000) 2022-01-31T21:24:23.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:23 smithi171 conmon[46715]: debug 2022-01-31T21:24:23.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:23.172068+0000) 2022-01-31T21:24:23.834 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:23 smithi171 conmon[35325]: debug 2022-01-31T21:24:23.673+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:24:23.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:23 smithi171 conmon[51620]: debug 2022-01-31T21:24:23.751+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:23.751838+0000) 2022-01-31T21:24:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:23 smithi171 conmon[41853]: debug 2022-01-31T21:24:23.875+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:23.875928+0000) 2022-01-31T21:24:24.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:23 smithi167 conmon[49112]: debug 2022-01-31T21:24:23.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:23.727446+0000) 2022-01-31T21:24:24.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:24 smithi167 conmon[49112]: debug 2022-01-31T21:24:24.359+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.360425+0000) 2022-01-31T21:24:24.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:24 smithi167 conmon[54076]: debug 2022-01-31T21:24:24.359+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.360357+0000) 2022-01-31T21:24:24.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:24 smithi167 conmon[54076]: debug 2022-01-31T21:24:24.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.394911+0000) 2022-01-31T21:24:24.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:24 smithi167 conmon[60316]: debug 2022-01-31T21:24:24.360+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.361191+0000) 2022-01-31T21:24:24.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:24 smithi167 conmon[60316]: debug 2022-01-31T21:24:24.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.399274+0000) 2022-01-31T21:24:24.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[41853]: debug 2022-01-31T21:24:24.359+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.360000+0000) 2022-01-31T21:24:24.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[51620]: debug 2022-01-31T21:24:24.361+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.361984+0000) 2022-01-31T21:24:24.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[46715]: debug 2022-01-31T21:24:24.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.172273+0000) 2022-01-31T21:24:24.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[46715]: debug 2022-01-31T21:24:24.359+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.360540+0000) 2022-01-31T21:24:24.514 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[35325]: debug 2022-01-31T21:24:24.372+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127210 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:24.834 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[51620]: debug 2022-01-31T21:24:24.751+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.752062+0000) 2022-01-31T21:24:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:24 smithi171 conmon[41853]: debug 2022-01-31T21:24:24.875+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.876112+0000) 2022-01-31T21:24:25.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:24 smithi167 conmon[49112]: debug 2022-01-31T21:24:24.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:24.727663+0000) 2022-01-31T21:24:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:25 smithi167 conmon[54076]: debug 2022-01-31T21:24:25.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:25.395079+0000) 2022-01-31T21:24:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:25 smithi167 conmon[60316]: debug 2022-01-31T21:24:25.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:25.399457+0000) 2022-01-31T21:24:25.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:25 smithi171 conmon[46715]: debug 2022-01-31T21:24:25.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:25.172418+0000) 2022-01-31T21:24:25.834 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:25 smithi171 conmon[51620]: debug 2022-01-31T21:24:25.751+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:25.752233+0000) 2022-01-31T21:24:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:25 smithi171 conmon[41853]: debug 2022-01-31T21:24:25.875+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:25.876289+0000) 2022-01-31T21:24:26.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:25 smithi167 conmon[49112]: debug 2022-01-31T21:24:25.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:25.727825+0000) 2022-01-31T21:24:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:26 smithi167 conmon[54076]: debug 2022-01-31T21:24:26.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:26.395262+0000) 2022-01-31T21:24:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:26 smithi167 conmon[60316]: debug 2022-01-31T21:24:26.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:26.399610+0000) 2022-01-31T21:24:26.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:26 smithi171 conmon[46715]: debug 2022-01-31T21:24:26.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:26.172543+0000) 2022-01-31T21:24:26.834 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:26 smithi171 conmon[51620]: debug 2022-01-31T21:24:26.751+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:26.752421+0000) 2022-01-31T21:24:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:26 smithi171 conmon[41853]: debug 2022-01-31T21:24:26.875+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:26.876473+0000) 2022-01-31T21:24:27.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:26 smithi167 conmon[49112]: debug 2022-01-31T21:24:26.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:26.727969+0000) 2022-01-31T21:24:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:27 smithi167 conmon[54076]: debug 2022-01-31T21:24:27.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:27.395481+0000) 2022-01-31T21:24:27.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:27 smithi167 conmon[60316]: debug 2022-01-31T21:24:27.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:27.399806+0000) 2022-01-31T21:24:27.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:27 smithi171 conmon[46715]: debug 2022-01-31T21:24:27.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:27.172729+0000) 2022-01-31T21:24:27.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:27 smithi171 conmon[51620]: debug 2022-01-31T21:24:27.751+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:27.752607+0000) 2022-01-31T21:24:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:27 smithi171 conmon[41853]: debug 2022-01-31T21:24:27.876+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:27.876682+0000) 2022-01-31T21:24:28.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:27 smithi167 conmon[49112]: debug 2022-01-31T21:24:27.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:27.728110+0000) 2022-01-31T21:24:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:28 smithi167 conmon[54076]: debug 2022-01-31T21:24:28.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:28.395668+0000) 2022-01-31T21:24:28.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:28 smithi167 conmon[60316]: debug 2022-01-31T21:24:28.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:28.399924+0000) 2022-01-31T21:24:28.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:28 smithi171 conmon[46715]: debug 2022-01-31T21:24:28.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:28.172927+0000) 2022-01-31T21:24:28.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:28 smithi171 conmon[51620]: debug 2022-01-31T21:24:28.752+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:28.752807+0000) 2022-01-31T21:24:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:28 smithi171 conmon[41853]: debug 2022-01-31T21:24:28.876+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:28.876893+0000) 2022-01-31T21:24:29.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:28 smithi167 conmon[49112]: debug 2022-01-31T21:24:28.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:28.728319+0000) 2022-01-31T21:24:29.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:29 smithi167 conmon[49112]: debug 2022-01-31T21:24:29.376+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.376873+0000) 2022-01-31T21:24:29.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:29 smithi167 conmon[54076]: debug 2022-01-31T21:24:29.375+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.376044+0000) 2022-01-31T21:24:29.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:29 smithi167 conmon[54076]: debug 2022-01-31T21:24:29.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.395803+0000) 2022-01-31T21:24:29.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:29 smithi167 conmon[60316]: debug 2022-01-31T21:24:29.376+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.376909+0000) 2022-01-31T21:24:29.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:29 smithi167 conmon[60316]: debug 2022-01-31T21:24:29.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.400057+0000) 2022-01-31T21:24:29.555 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[35325]: debug 2022-01-31T21:24:29.388+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127314 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:29.556 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[41853]: debug 2022-01-31T21:24:29.374+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.375669+0000) 2022-01-31T21:24:29.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[51620]: debug 2022-01-31T21:24:29.376+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.377332+0000) 2022-01-31T21:24:29.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[46715]: debug 2022-01-31T21:24:29.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.173079+0000) 2022-01-31T21:24:29.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[46715]: debug 2022-01-31T21:24:29.374+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.375401+0000) 2022-01-31T21:24:29.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[51620]: debug 2022-01-31T21:24:29.752+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.753005+0000) 2022-01-31T21:24:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:29 smithi171 conmon[41853]: debug 2022-01-31T21:24:29.876+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.877058+0000) 2022-01-31T21:24:30.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:29 smithi167 conmon[49112]: debug 2022-01-31T21:24:29.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:29.728545+0000) 2022-01-31T21:24:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:30 smithi167 conmon[54076]: debug 2022-01-31T21:24:30.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:30.395927+0000) 2022-01-31T21:24:30.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:30 smithi167 conmon[60316]: debug 2022-01-31T21:24:30.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:30.400246+0000) 2022-01-31T21:24:30.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:30 smithi171 conmon[46715]: debug 2022-01-31T21:24:30.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:30.173258+0000) 2022-01-31T21:24:30.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:30 smithi171 conmon[51620]: debug 2022-01-31T21:24:30.752+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:30.753161+0000) 2022-01-31T21:24:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:30 smithi171 conmon[41853]: debug 2022-01-31T21:24:30.876+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:30.877207+0000) 2022-01-31T21:24:31.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:30 smithi167 conmon[49112]: debug 2022-01-31T21:24:30.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:30.728679+0000) 2022-01-31T21:24:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:31 smithi167 conmon[54076]: debug 2022-01-31T21:24:31.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:31.396129+0000) 2022-01-31T21:24:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:31 smithi167 conmon[60316]: debug 2022-01-31T21:24:31.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:31.400452+0000) 2022-01-31T21:24:31.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:31 smithi171 conmon[46715]: debug 2022-01-31T21:24:31.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:31.173441+0000) 2022-01-31T21:24:31.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:31 smithi171 conmon[51620]: debug 2022-01-31T21:24:31.752+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:31.753380+0000) 2022-01-31T21:24:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:31 smithi171 conmon[41853]: debug 2022-01-31T21:24:31.876+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:31.877355+0000) 2022-01-31T21:24:32.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:31 smithi167 conmon[49112]: debug 2022-01-31T21:24:31.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:31.728867+0000) 2022-01-31T21:24:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:32 smithi167 conmon[54076]: debug 2022-01-31T21:24:32.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:32.396363+0000) 2022-01-31T21:24:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:32 smithi167 conmon[60316]: debug 2022-01-31T21:24:32.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:32.400664+0000) 2022-01-31T21:24:32.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:32 smithi171 conmon[46715]: debug 2022-01-31T21:24:32.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:32.173595+0000) 2022-01-31T21:24:32.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:32 smithi171 conmon[51620]: debug 2022-01-31T21:24:32.753+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:32.753582+0000) 2022-01-31T21:24:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:32 smithi171 conmon[41853]: debug 2022-01-31T21:24:32.876+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:32.877553+0000) 2022-01-31T21:24:33.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:32 smithi167 conmon[49112]: debug 2022-01-31T21:24:32.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:32.729080+0000) 2022-01-31T21:24:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:33 smithi167 conmon[54076]: debug 2022-01-31T21:24:33.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:33.396559+0000) 2022-01-31T21:24:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:33 smithi167 conmon[60316]: debug 2022-01-31T21:24:33.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:33.400841+0000) 2022-01-31T21:24:33.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:33 smithi171 conmon[46715]: debug 2022-01-31T21:24:33.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:33.173808+0000) 2022-01-31T21:24:33.836 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:33 smithi171 conmon[51620]: debug 2022-01-31T21:24:33.753+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:33.753784+0000) 2022-01-31T21:24:34.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:33 smithi171 conmon[41853]: debug 2022-01-31T21:24:33.877+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:33.877746+0000) 2022-01-31T21:24:34.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:33 smithi167 conmon[49112]: debug 2022-01-31T21:24:33.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:33.729260+0000) 2022-01-31T21:24:34.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:34 smithi167 conmon[49112]: debug 2022-01-31T21:24:34.392+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.392828+0000) 2022-01-31T21:24:34.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:34 smithi167 conmon[54076]: debug 2022-01-31T21:24:34.391+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.391618+0000) 2022-01-31T21:24:34.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:34 smithi167 conmon[54076]: debug 2022-01-31T21:24:34.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.396669+0000) 2022-01-31T21:24:34.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:34 smithi167 conmon[60316]: debug 2022-01-31T21:24:34.392+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.393242+0000) 2022-01-31T21:24:34.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:34 smithi167 conmon[60316]: debug 2022-01-31T21:24:34.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.401009+0000) 2022-01-31T21:24:34.556 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[35325]: debug 2022-01-31T21:24:34.404+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127426 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:34.557 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[41853]: debug 2022-01-31T21:24:34.390+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.391013+0000) 2022-01-31T21:24:34.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[51620]: debug 2022-01-31T21:24:34.391+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.391818+0000) 2022-01-31T21:24:34.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[46715]: debug 2022-01-31T21:24:34.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.174017+0000) 2022-01-31T21:24:34.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[46715]: debug 2022-01-31T21:24:34.390+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.391316+0000) 2022-01-31T21:24:34.836 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[51620]: debug 2022-01-31T21:24:34.753+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.753975+0000) 2022-01-31T21:24:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:34 smithi171 conmon[41853]: debug 2022-01-31T21:24:34.877+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.877957+0000) 2022-01-31T21:24:35.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:34 smithi167 conmon[49112]: debug 2022-01-31T21:24:34.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:34.729464+0000) 2022-01-31T21:24:35.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:35 smithi167 conmon[54076]: debug 2022-01-31T21:24:35.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:35.396751+0000) 2022-01-31T21:24:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:35 smithi167 conmon[60316]: debug 2022-01-31T21:24:35.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:35.401146+0000) 2022-01-31T21:24:35.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:35 smithi171 conmon[46715]: debug 2022-01-31T21:24:35.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:35.174157+0000) 2022-01-31T21:24:35.836 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:35 smithi171 conmon[51620]: debug 2022-01-31T21:24:35.753+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:35.754191+0000) 2022-01-31T21:24:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:35 smithi171 conmon[41853]: debug 2022-01-31T21:24:35.877+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:35.878109+0000) 2022-01-31T21:24:36.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:35 smithi167 conmon[49112]: debug 2022-01-31T21:24:35.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:35.729621+0000) 2022-01-31T21:24:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:36 smithi167 conmon[60316]: debug 2022-01-31T21:24:36.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:36.401304+0000) 2022-01-31T21:24:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:36 smithi167 conmon[54076]: debug 2022-01-31T21:24:36.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:36.396916+0000) 2022-01-31T21:24:36.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:36 smithi171 conmon[46715]: debug 2022-01-31T21:24:36.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:36.174305+0000) 2022-01-31T21:24:36.836 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:36 smithi171 conmon[51620]: debug 2022-01-31T21:24:36.753+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:36.754381+0000) 2022-01-31T21:24:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:36 smithi171 conmon[41853]: debug 2022-01-31T21:24:36.877+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:36.878329+0000) 2022-01-31T21:24:37.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:36 smithi167 conmon[49112]: debug 2022-01-31T21:24:36.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:36.729804+0000) 2022-01-31T21:24:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:37 smithi167 conmon[54076]: debug 2022-01-31T21:24:37.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:37.397102+0000) 2022-01-31T21:24:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:37 smithi167 conmon[60316]: debug 2022-01-31T21:24:37.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:37.401492+0000) 2022-01-31T21:24:37.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:37 smithi171 conmon[46715]: debug 2022-01-31T21:24:37.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:37.174504+0000) 2022-01-31T21:24:37.836 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:37 smithi171 conmon[51620]: debug 2022-01-31T21:24:37.753+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:37.754552+0000) 2022-01-31T21:24:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:37 smithi171 conmon[41853]: debug 2022-01-31T21:24:37.878+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:37.878513+0000) 2022-01-31T21:24:38.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:37 smithi167 conmon[49112]: debug 2022-01-31T21:24:37.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:37.730005+0000) 2022-01-31T21:24:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:38 smithi167 conmon[60316]: debug 2022-01-31T21:24:38.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:38.401664+0000) 2022-01-31T21:24:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:38 smithi167 conmon[54076]: debug 2022-01-31T21:24:38.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:38.397296+0000) 2022-01-31T21:24:38.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:38 smithi171 conmon[46715]: debug 2022-01-31T21:24:38.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:38.174655+0000) 2022-01-31T21:24:38.837 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:38 smithi171 conmon[35325]: debug 2022-01-31T21:24:38.674+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:24:38.837 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:38 smithi171 conmon[51620]: debug 2022-01-31T21:24:38.754+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:38.754771+0000) 2022-01-31T21:24:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:38 smithi171 conmon[41853]: debug 2022-01-31T21:24:38.878+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:38.878693+0000) 2022-01-31T21:24:39.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:38 smithi167 conmon[49112]: debug 2022-01-31T21:24:38.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:38.730232+0000) 2022-01-31T21:24:39.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:39 smithi167 conmon[54076]: debug 2022-01-31T21:24:39.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.397476+0000) 2022-01-31T21:24:39.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:39 smithi167 conmon[60316]: debug 2022-01-31T21:24:39.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.401813+0000) 2022-01-31T21:24:39.557 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[35325]: debug 2022-01-31T21:24:39.419+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127534 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:39.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[41853]: debug 2022-01-31T21:24:39.406+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.406967+0000) 2022-01-31T21:24:39.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[51620]: debug 2022-01-31T21:24:39.406+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.406823+0000) 2022-01-31T21:24:39.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[46715]: debug 2022-01-31T21:24:39.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.174789+0000) 2022-01-31T21:24:39.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[46715]: debug 2022-01-31T21:24:39.406+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.407311+0000) 2022-01-31T21:24:39.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:39 smithi167 conmon[49112]: debug 2022-01-31T21:24:39.407+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.408267+0000) 2022-01-31T21:24:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:39 smithi167 conmon[54076]: debug 2022-01-31T21:24:39.407+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.407989+0000) 2022-01-31T21:24:39.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:39 smithi167 conmon[60316]: debug 2022-01-31T21:24:39.408+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.408537+0000) 2022-01-31T21:24:39.837 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[51620]: debug 2022-01-31T21:24:39.754+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.754938+0000) 2022-01-31T21:24:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:39 smithi171 conmon[41853]: debug 2022-01-31T21:24:39.878+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.878853+0000) 2022-01-31T21:24:40.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:39 smithi167 conmon[49112]: debug 2022-01-31T21:24:39.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:39.730444+0000) 2022-01-31T21:24:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:40 smithi167 conmon[54076]: debug 2022-01-31T21:24:40.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:40.397632+0000) 2022-01-31T21:24:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:40 smithi167 conmon[60316]: debug 2022-01-31T21:24:40.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:40.402036+0000) 2022-01-31T21:24:40.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:40 smithi171 conmon[46715]: debug 2022-01-31T21:24:40.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:40.174946+0000) 2022-01-31T21:24:40.837 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:40 smithi171 conmon[51620]: debug 2022-01-31T21:24:40.754+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:40.755049+0000) 2022-01-31T21:24:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:40 smithi171 conmon[41853]: debug 2022-01-31T21:24:40.878+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:40.878993+0000) 2022-01-31T21:24:41.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:40 smithi167 conmon[49112]: debug 2022-01-31T21:24:40.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:40.730588+0000) 2022-01-31T21:24:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:41 smithi167 conmon[54076]: debug 2022-01-31T21:24:41.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:41.397779+0000) 2022-01-31T21:24:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:41 smithi167 conmon[60316]: debug 2022-01-31T21:24:41.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:41.402182+0000) 2022-01-31T21:24:41.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:41 smithi171 conmon[46715]: debug 2022-01-31T21:24:41.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:41.175078+0000) 2022-01-31T21:24:41.837 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:41 smithi171 conmon[51620]: debug 2022-01-31T21:24:41.754+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:41.755234+0000) 2022-01-31T21:24:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:41 smithi171 conmon[41853]: debug 2022-01-31T21:24:41.878+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:41.879205+0000) 2022-01-31T21:24:42.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:41 smithi167 conmon[49112]: debug 2022-01-31T21:24:41.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:41.730761+0000) 2022-01-31T21:24:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:42 smithi167 conmon[54076]: debug 2022-01-31T21:24:42.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:42.397940+0000) 2022-01-31T21:24:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:42 smithi167 conmon[60316]: debug 2022-01-31T21:24:42.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:42.402350+0000) 2022-01-31T21:24:42.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:42 smithi171 conmon[46715]: debug 2022-01-31T21:24:42.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:42.175225+0000) 2022-01-31T21:24:42.837 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:42 smithi171 conmon[51620]: debug 2022-01-31T21:24:42.755+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:42.755419+0000) 2022-01-31T21:24:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:42 smithi171 conmon[41853]: debug 2022-01-31T21:24:42.879+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:42.879426+0000) 2022-01-31T21:24:43.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:42 smithi167 conmon[49112]: debug 2022-01-31T21:24:42.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:42.730999+0000) 2022-01-31T21:24:43.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:43 smithi167 conmon[54076]: debug 2022-01-31T21:24:43.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:43.398124+0000) 2022-01-31T21:24:43.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:43 smithi167 conmon[60316]: debug 2022-01-31T21:24:43.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:43.402559+0000) 2022-01-31T21:24:43.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:43 smithi171 conmon[46715]: debug 2022-01-31T21:24:43.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:43.175419+0000) 2022-01-31T21:24:43.837 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:43 smithi171 conmon[51620]: debug 2022-01-31T21:24:43.755+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:43.755611+0000) 2022-01-31T21:24:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:43 smithi171 conmon[41853]: debug 2022-01-31T21:24:43.879+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:43.879583+0000) 2022-01-31T21:24:44.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:43 smithi167 conmon[49112]: debug 2022-01-31T21:24:43.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:43.731237+0000) 2022-01-31T21:24:44.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:44 smithi167 conmon[54076]: debug 2022-01-31T21:24:44.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.398283+0000) 2022-01-31T21:24:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:44 smithi167 conmon[60316]: debug 2022-01-31T21:24:44.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.402693+0000) 2022-01-31T21:24:44.434 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[41853]: debug 2022-01-31T21:24:44.421+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.422630+0000) 2022-01-31T21:24:44.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[46715]: debug 2022-01-31T21:24:44.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.175607+0000) 2022-01-31T21:24:44.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[46715]: debug 2022-01-31T21:24:44.422+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.422936+0000) 2022-01-31T21:24:44.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[51620]: debug 2022-01-31T21:24:44.422+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.422739+0000) 2022-01-31T21:24:44.717 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:44 smithi167 conmon[49112]: debug 2022-01-31T21:24:44.420+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.422211+0000) 2022-01-31T21:24:44.717 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:44 smithi167 conmon[54076]: debug 2022-01-31T21:24:44.421+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.423223+0000) 2022-01-31T21:24:44.718 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:44 smithi167 conmon[60316]: debug 2022-01-31T21:24:44.422+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.423718+0000) 2022-01-31T21:24:44.754 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[35325]: debug 2022-01-31T21:24:44.434+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127646 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[41853]: debug 2022-01-31T21:24:44.879+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.879786+0000) 2022-01-31T21:24:45.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:44 smithi171 conmon[51620]: debug 2022-01-31T21:24:44.755+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.755807+0000) 2022-01-31T21:24:45.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:44 smithi167 conmon[49112]: debug 2022-01-31T21:24:44.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:44.731480+0000) 2022-01-31T21:24:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:45 smithi167 conmon[60316]: debug 2022-01-31T21:24:45.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:45.402849+0000) 2022-01-31T21:24:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:45 smithi167 conmon[54076]: debug 2022-01-31T21:24:45.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:45.398492+0000) 2022-01-31T21:24:45.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:45 smithi171 conmon[46715]: debug 2022-01-31T21:24:45.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:45.175791+0000) 2022-01-31T21:24:45.838 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:45 smithi171 conmon[51620]: debug 2022-01-31T21:24:45.755+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:45.755968+0000) 2022-01-31T21:24:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:45 smithi171 conmon[41853]: debug 2022-01-31T21:24:45.879+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:45.879901+0000) 2022-01-31T21:24:46.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:45 smithi167 conmon[49112]: debug 2022-01-31T21:24:45.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:45.731659+0000) 2022-01-31T21:24:46.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:46 smithi167 conmon[54076]: debug 2022-01-31T21:24:46.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:46.398597+0000) 2022-01-31T21:24:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:46 smithi167 conmon[60316]: debug 2022-01-31T21:24:46.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:46.402969+0000) 2022-01-31T21:24:46.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:46 smithi171 conmon[46715]: debug 2022-01-31T21:24:46.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:46.175963+0000) 2022-01-31T21:24:46.838 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:46 smithi171 conmon[51620]: debug 2022-01-31T21:24:46.755+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:46.756178+0000) 2022-01-31T21:24:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:46 smithi171 conmon[41853]: debug 2022-01-31T21:24:46.879+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:46.880088+0000) 2022-01-31T21:24:47.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:46 smithi167 conmon[49112]: debug 2022-01-31T21:24:46.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:46.731814+0000) 2022-01-31T21:24:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:47 smithi167 conmon[60316]: debug 2022-01-31T21:24:47.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:47.403172+0000) 2022-01-31T21:24:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:47 smithi167 conmon[54076]: debug 2022-01-31T21:24:47.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:47.398797+0000) 2022-01-31T21:24:47.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:47 smithi171 conmon[46715]: debug 2022-01-31T21:24:47.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:47.176152+0000) 2022-01-31T21:24:47.838 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:47 smithi171 conmon[51620]: debug 2022-01-31T21:24:47.756+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:47.756401+0000) 2022-01-31T21:24:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:47 smithi171 conmon[41853]: debug 2022-01-31T21:24:47.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:47.880277+0000) 2022-01-31T21:24:48.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:47 smithi167 conmon[49112]: debug 2022-01-31T21:24:47.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:47.731924+0000) 2022-01-31T21:24:48.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:48 smithi167 conmon[60316]: debug 2022-01-31T21:24:48.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:48.403370+0000) 2022-01-31T21:24:48.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:48 smithi167 conmon[54076]: debug 2022-01-31T21:24:48.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:48.399006+0000) 2022-01-31T21:24:48.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:48 smithi171 conmon[46715]: debug 2022-01-31T21:24:48.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:48.176305+0000) 2022-01-31T21:24:48.838 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:48 smithi171 conmon[51620]: debug 2022-01-31T21:24:48.755+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:48.756607+0000) 2022-01-31T21:24:49.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:48 smithi171 conmon[41853]: debug 2022-01-31T21:24:48.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:48.880454+0000) 2022-01-31T21:24:49.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:48 smithi167 conmon[49112]: debug 2022-01-31T21:24:48.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:48.732065+0000) 2022-01-31T21:24:49.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:49 smithi167 conmon[60316]: debug 2022-01-31T21:24:49.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.403586+0000) 2022-01-31T21:24:49.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:49 smithi167 conmon[54076]: debug 2022-01-31T21:24:49.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.399230+0000) 2022-01-31T21:24:49.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[46715]: debug 2022-01-31T21:24:49.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.176502+0000) 2022-01-31T21:24:49.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:49 smithi167 conmon[49112]: debug 2022-01-31T21:24:49.437+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.439003+0000) 2022-01-31T21:24:49.718 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:49 smithi167 conmon[54076]: debug 2022-01-31T21:24:49.437+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.438576+0000) 2022-01-31T21:24:49.719 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:49 smithi167 conmon[60316]: debug 2022-01-31T21:24:49.437+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.439136+0000) 2022-01-31T21:24:49.755 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[35325]: debug 2022-01-31T21:24:49.449+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127754 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:49.756 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[41853]: debug 2022-01-31T21:24:49.436+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.437334+0000) 2022-01-31T21:24:49.756 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[46715]: debug 2022-01-31T21:24:49.438+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.438805+0000) 2022-01-31T21:24:49.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[51620]: debug 2022-01-31T21:24:49.437+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.437767+0000) 2022-01-31T21:24:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[41853]: debug 2022-01-31T21:24:49.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.880648+0000) 2022-01-31T21:24:50.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:49 smithi171 conmon[51620]: debug 2022-01-31T21:24:49.756+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.756822+0000) 2022-01-31T21:24:50.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:49 smithi167 conmon[49112]: debug 2022-01-31T21:24:49.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:49.732284+0000) 2022-01-31T21:24:50.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:50 smithi167 conmon[60316]: debug 2022-01-31T21:24:50.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:50.403764+0000) 2022-01-31T21:24:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:50 smithi167 conmon[54076]: debug 2022-01-31T21:24:50.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:50.399371+0000) 2022-01-31T21:24:50.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:50 smithi171 conmon[46715]: debug 2022-01-31T21:24:50.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:50.176690+0000) 2022-01-31T21:24:50.838 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:50 smithi171 conmon[51620]: debug 2022-01-31T21:24:50.756+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:50.756973+0000) 2022-01-31T21:24:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:50 smithi171 conmon[41853]: debug 2022-01-31T21:24:50.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:50.880783+0000) 2022-01-31T21:24:51.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:50 smithi167 conmon[49112]: debug 2022-01-31T21:24:50.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:50.732437+0000) 2022-01-31T21:24:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:51 smithi167 conmon[60316]: debug 2022-01-31T21:24:51.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:51.403887+0000) 2022-01-31T21:24:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:51 smithi167 conmon[54076]: debug 2022-01-31T21:24:51.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:51.399542+0000) 2022-01-31T21:24:51.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:51 smithi171 conmon[46715]: debug 2022-01-31T21:24:51.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:51.176827+0000) 2022-01-31T21:24:51.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:51 smithi171 conmon[51620]: debug 2022-01-31T21:24:51.756+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:51.757158+0000) 2022-01-31T21:24:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:51 smithi171 conmon[41853]: debug 2022-01-31T21:24:51.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:51.881004+0000) 2022-01-31T21:24:52.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:51 smithi167 conmon[49112]: debug 2022-01-31T21:24:51.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:51.732619+0000) 2022-01-31T21:24:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:52 smithi167 conmon[60316]: debug 2022-01-31T21:24:52.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:52.404062+0000) 2022-01-31T21:24:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:52 smithi167 conmon[54076]: debug 2022-01-31T21:24:52.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:52.399731+0000) 2022-01-31T21:24:52.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:52 smithi171 conmon[46715]: debug 2022-01-31T21:24:52.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:52.176976+0000) 2022-01-31T21:24:52.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:52 smithi171 conmon[51620]: debug 2022-01-31T21:24:52.757+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:52.757346+0000) 2022-01-31T21:24:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:52 smithi171 conmon[41853]: debug 2022-01-31T21:24:52.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:52.881206+0000) 2022-01-31T21:24:53.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:52 smithi167 conmon[49112]: debug 2022-01-31T21:24:52.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:52.732822+0000) 2022-01-31T21:24:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:53 smithi167 conmon[60316]: debug 2022-01-31T21:24:53.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:53.404291+0000) 2022-01-31T21:24:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:53 smithi167 conmon[54076]: debug 2022-01-31T21:24:53.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:53.399918+0000) 2022-01-31T21:24:53.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:53 smithi171 conmon[46715]: debug 2022-01-31T21:24:53.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:53.177120+0000) 2022-01-31T21:24:53.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:53 smithi171 conmon[35325]: debug 2022-01-31T21:24:53.675+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:24:53.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:53 smithi171 conmon[51620]: debug 2022-01-31T21:24:53.756+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:53.757551+0000) 2022-01-31T21:24:54.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:53 smithi171 conmon[41853]: debug 2022-01-31T21:24:53.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:53.881386+0000) 2022-01-31T21:24:54.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:53 smithi167 conmon[49112]: debug 2022-01-31T21:24:53.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:53.732998+0000) 2022-01-31T21:24:54.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:54 smithi167 conmon[60316]: debug 2022-01-31T21:24:54.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.404497+0000) 2022-01-31T21:24:54.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:54 smithi167 conmon[54076]: debug 2022-01-31T21:24:54.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.400148+0000) 2022-01-31T21:24:54.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[46715]: debug 2022-01-31T21:24:54.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.177283+0000) 2022-01-31T21:24:54.451 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[41853]: debug 2022-01-31T21:24:54.451+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.452235+0000) 2022-01-31T21:24:54.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:54 smithi167 conmon[49112]: debug 2022-01-31T21:24:54.451+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.452866+0000) 2022-01-31T21:24:54.719 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:54 smithi167 conmon[54076]: debug 2022-01-31T21:24:54.452+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.454136+0000) 2022-01-31T21:24:54.720 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:54 smithi167 conmon[60316]: debug 2022-01-31T21:24:54.452+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.453856+0000) 2022-01-31T21:24:54.756 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[35325]: debug 2022-01-31T21:24:54.464+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127866 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:54.757 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[46715]: debug 2022-01-31T21:24:54.452+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.453735+0000) 2022-01-31T21:24:54.758 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[51620]: debug 2022-01-31T21:24:54.452+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.452826+0000) 2022-01-31T21:24:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[41853]: debug 2022-01-31T21:24:54.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.881572+0000) 2022-01-31T21:24:55.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:54 smithi171 conmon[51620]: debug 2022-01-31T21:24:54.757+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.757788+0000) 2022-01-31T21:24:55.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:54 smithi167 conmon[49112]: debug 2022-01-31T21:24:54.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:54.733176+0000) 2022-01-31T21:24:55.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:55 smithi167 conmon[60316]: debug 2022-01-31T21:24:55.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:55.404635+0000) 2022-01-31T21:24:55.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:55 smithi167 conmon[54076]: debug 2022-01-31T21:24:55.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:55.400357+0000) 2022-01-31T21:24:55.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:55 smithi171 conmon[46715]: debug 2022-01-31T21:24:55.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:55.177428+0000) 2022-01-31T21:24:55.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:55 smithi171 conmon[51620]: debug 2022-01-31T21:24:55.757+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:55.757941+0000) 2022-01-31T21:24:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:55 smithi171 conmon[41853]: debug 2022-01-31T21:24:55.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:55.881773+0000) 2022-01-31T21:24:56.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:55 smithi167 conmon[49112]: debug 2022-01-31T21:24:55.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:55.733284+0000) 2022-01-31T21:24:56.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:56 smithi167 conmon[60316]: debug 2022-01-31T21:24:56.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:56.404788+0000) 2022-01-31T21:24:56.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:56 smithi167 conmon[54076]: debug 2022-01-31T21:24:56.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:56.400544+0000) 2022-01-31T21:24:56.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:56 smithi171 conmon[46715]: debug 2022-01-31T21:24:56.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:56.177580+0000) 2022-01-31T21:24:56.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:56 smithi171 conmon[51620]: debug 2022-01-31T21:24:56.758+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:56.758105+0000) 2022-01-31T21:24:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:56 smithi171 conmon[41853]: debug 2022-01-31T21:24:56.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:56.881961+0000) 2022-01-31T21:24:57.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:56 smithi167 conmon[49112]: debug 2022-01-31T21:24:56.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:56.733495+0000) 2022-01-31T21:24:57.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:57 smithi167 conmon[54076]: debug 2022-01-31T21:24:57.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:57.400716+0000) 2022-01-31T21:24:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:57 smithi167 conmon[60316]: debug 2022-01-31T21:24:57.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:57.404946+0000) 2022-01-31T21:24:57.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:57 smithi171 conmon[46715]: debug 2022-01-31T21:24:57.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:57.177785+0000) 2022-01-31T21:24:57.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:57 smithi171 conmon[51620]: debug 2022-01-31T21:24:57.757+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:57.758278+0000) 2022-01-31T21:24:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:57 smithi171 conmon[41853]: debug 2022-01-31T21:24:57.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:57.882122+0000) 2022-01-31T21:24:58.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:57 smithi167 conmon[49112]: debug 2022-01-31T21:24:57.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:57.733663+0000) 2022-01-31T21:24:58.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:58 smithi167 conmon[54076]: debug 2022-01-31T21:24:58.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:58.400882+0000) 2022-01-31T21:24:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:58 smithi167 conmon[60316]: debug 2022-01-31T21:24:58.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:58.405110+0000) 2022-01-31T21:24:58.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:58 smithi171 conmon[46715]: debug 2022-01-31T21:24:58.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:58.178001+0000) 2022-01-31T21:24:58.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:58 smithi171 conmon[51620]: debug 2022-01-31T21:24:58.757+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:58.758481+0000) 2022-01-31T21:24:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:58 smithi171 conmon[41853]: debug 2022-01-31T21:24:58.880+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:58.882309+0000) 2022-01-31T21:24:59.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:58 smithi167 conmon[49112]: debug 2022-01-31T21:24:58.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:58.733808+0000) 2022-01-31T21:24:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:59 smithi167 conmon[60316]: debug 2022-01-31T21:24:59.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.405297+0000) 2022-01-31T21:24:59.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:59 smithi167 conmon[54076]: debug 2022-01-31T21:24:59.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.401091+0000) 2022-01-31T21:24:59.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[46715]: debug 2022-01-31T21:24:59.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.178189+0000) 2022-01-31T21:24:59.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:59 smithi167 conmon[49112]: debug 2022-01-31T21:24:59.466+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.467680+0000) 2022-01-31T21:24:59.721 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:24:59 smithi167 conmon[54076]: debug 2022-01-31T21:24:59.467+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.468763+0000) 2022-01-31T21:24:59.721 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:24:59 smithi167 conmon[60316]: debug 2022-01-31T21:24:59.468+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.469579+0000) 2022-01-31T21:24:59.757 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[35325]: debug 2022-01-31T21:24:59.757 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[35325]: 2022-01-31T21:24:59.482+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 127974 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:24:59.758 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[41853]: debug 2022-01-31T21:24:59.465+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.467700+0000) 2022-01-31T21:24:59.758 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[46715]: debug 2022-01-31T21:24:59.467+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.469446+0000) 2022-01-31T21:24:59.759 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[51620]: debug 2022-01-31T21:24:59.466+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.468444+0000) 2022-01-31T21:25:00.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[51620]: debug 2022-01-31T21:24:59.756+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.758650+0000) 2022-01-31T21:25:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:24:59 smithi171 conmon[41853]: debug 2022-01-31T21:24:59.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.882461+0000) 2022-01-31T21:25:00.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:24:59 smithi167 conmon[49112]: debug 2022-01-31T21:24:59.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:24:59.734004+0000) 2022-01-31T21:25:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:00 smithi167 conmon[54076]: debug 2022-01-31T21:25:00.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:00.401280+0000) 2022-01-31T21:25:00.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:00 smithi167 conmon[60316]: debug 2022-01-31T21:25:00.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:00.405468+0000) 2022-01-31T21:25:00.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:00 smithi171 conmon[46715]: debug 2022-01-31T21:25:00.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:00.178368+0000) 2022-01-31T21:25:00.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:00 smithi171 conmon[51620]: debug 2022-01-31T21:25:00.757+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:00.758819+0000) 2022-01-31T21:25:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:00 smithi171 conmon[41853]: debug 2022-01-31T21:25:00.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:00.882600+0000) 2022-01-31T21:25:01.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:00 smithi167 conmon[49112]: debug 2022-01-31T21:25:00.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:00.734190+0000) 2022-01-31T21:25:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:01 smithi167 conmon[54076]: debug 2022-01-31T21:25:01.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:01.401463+0000) 2022-01-31T21:25:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:01 smithi167 conmon[60316]: debug 2022-01-31T21:25:01.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:01.405600+0000) 2022-01-31T21:25:01.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:01 smithi171 conmon[46715]: debug 2022-01-31T21:25:01.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:01.178476+0000) 2022-01-31T21:25:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:01 smithi171 conmon[51620]: debug 2022-01-31T21:25:01.758+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:01.759017+0000) 2022-01-31T21:25:02.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:01 smithi171 conmon[41853]: debug 2022-01-31T21:25:01.881+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:01.882828+0000) 2022-01-31T21:25:02.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:01 smithi167 conmon[49112]: debug 2022-01-31T21:25:01.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:01.734383+0000) 2022-01-31T21:25:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:02 smithi167 conmon[54076]: debug 2022-01-31T21:25:02.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:02.401621+0000) 2022-01-31T21:25:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:02 smithi167 conmon[60316]: debug 2022-01-31T21:25:02.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:02.405729+0000) 2022-01-31T21:25:02.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:02 smithi171 conmon[46715]: debug 2022-01-31T21:25:02.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:02.178632+0000) 2022-01-31T21:25:02.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:02 smithi171 conmon[51620]: debug 2022-01-31T21:25:02.758+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:02.759197+0000) 2022-01-31T21:25:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:02 smithi171 conmon[41853]: debug 2022-01-31T21:25:02.882+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:02.882989+0000) 2022-01-31T21:25:03.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:02 smithi167 conmon[49112]: debug 2022-01-31T21:25:02.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:02.734617+0000) 2022-01-31T21:25:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:03 smithi167 conmon[54076]: debug 2022-01-31T21:25:03.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:03.401812+0000) 2022-01-31T21:25:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:03 smithi167 conmon[60316]: debug 2022-01-31T21:25:03.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:03.405899+0000) 2022-01-31T21:25:03.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:03 smithi171 conmon[46715]: debug 2022-01-31T21:25:03.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:03.178839+0000) 2022-01-31T21:25:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:03 smithi171 conmon[51620]: debug 2022-01-31T21:25:03.758+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:03.759395+0000) 2022-01-31T21:25:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:03 smithi171 conmon[41853]: debug 2022-01-31T21:25:03.882+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:03.883169+0000) 2022-01-31T21:25:04.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:03 smithi167 conmon[49112]: debug 2022-01-31T21:25:03.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:03.734794+0000) 2022-01-31T21:25:04.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:04 smithi167 conmon[54076]: debug 2022-01-31T21:25:04.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.401975+0000) 2022-01-31T21:25:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:04 smithi167 conmon[60316]: debug 2022-01-31T21:25:04.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.406119+0000) 2022-01-31T21:25:04.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[46715]: debug 2022-01-31T21:25:04.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.179042+0000) 2022-01-31T21:25:04.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[51620]: debug 2022-01-31T21:25:04.486+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.488325+0000) 2022-01-31T21:25:04.720 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:04 smithi167 conmon[49112]: debug 2022-01-31T21:25:04.486+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.487547+0000) 2022-01-31T21:25:04.721 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:04 smithi167 conmon[54076]: debug 2022-01-31T21:25:04.487+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.488590+0000) 2022-01-31T21:25:04.721 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:04 smithi167 conmon[60316]: debug 2022-01-31T21:25:04.486+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.487600+0000) 2022-01-31T21:25:04.758 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[35325]: debug 2022-01-31T21:25:04.498+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128086 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:04.759 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[41853]: debug 2022-01-31T21:25:04.484+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.486631+0000) 2022-01-31T21:25:04.759 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[46715]: debug 2022-01-31T21:25:04.486+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.488204+0000) 2022-01-31T21:25:05.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[51620]: debug 2022-01-31T21:25:04.758+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.759575+0000) 2022-01-31T21:25:05.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:04 smithi171 conmon[41853]: debug 2022-01-31T21:25:04.882+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.883330+0000) 2022-01-31T21:25:05.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:04 smithi167 conmon[49112]: debug 2022-01-31T21:25:04.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:04.735010+0000) 2022-01-31T21:25:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:05 smithi167 conmon[60316]: debug 2022-01-31T21:25:05.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:05.406305+0000) 2022-01-31T21:25:05.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:05 smithi167 conmon[54076]: debug 2022-01-31T21:25:05.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:05.402129+0000) 2022-01-31T21:25:05.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:05 smithi171 conmon[46715]: debug 2022-01-31T21:25:05.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:05.179245+0000) 2022-01-31T21:25:05.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:05 smithi171 conmon[51620]: debug 2022-01-31T21:25:05.758+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:05.759702+0000) 2022-01-31T21:25:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:05 smithi171 conmon[41853]: debug 2022-01-31T21:25:05.882+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:05.883469+0000) 2022-01-31T21:25:06.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:05 smithi167 conmon[49112]: debug 2022-01-31T21:25:05.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:05.735172+0000) 2022-01-31T21:25:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:06 smithi167 conmon[54076]: debug 2022-01-31T21:25:06.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:06.402283+0000) 2022-01-31T21:25:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:06 smithi167 conmon[60316]: debug 2022-01-31T21:25:06.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:06.406476+0000) 2022-01-31T21:25:06.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:06 smithi171 conmon[46715]: debug 2022-01-31T21:25:06.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:06.179398+0000) 2022-01-31T21:25:06.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:06 smithi171 conmon[51620]: debug 2022-01-31T21:25:06.759+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:06.759925+0000) 2022-01-31T21:25:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:06 smithi171 conmon[41853]: debug 2022-01-31T21:25:06.882+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:06.883653+0000) 2022-01-31T21:25:07.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:06 smithi167 conmon[49112]: debug 2022-01-31T21:25:06.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:06.735377+0000) 2022-01-31T21:25:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:07 smithi167 conmon[60316]: debug 2022-01-31T21:25:07.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:07.406617+0000) 2022-01-31T21:25:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:07 smithi167 conmon[54076]: debug 2022-01-31T21:25:07.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:07.402482+0000) 2022-01-31T21:25:07.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:07 smithi171 conmon[46715]: debug 2022-01-31T21:25:07.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:07.179600+0000) 2022-01-31T21:25:07.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:07 smithi171 conmon[51620]: debug 2022-01-31T21:25:07.759+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:07.760111+0000) 2022-01-31T21:25:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:07 smithi171 conmon[41853]: debug 2022-01-31T21:25:07.883+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:07.883843+0000) 2022-01-31T21:25:08.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:07 smithi167 conmon[49112]: debug 2022-01-31T21:25:07.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:07.735588+0000) 2022-01-31T21:25:08.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:08 smithi167 conmon[54076]: debug 2022-01-31T21:25:08.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:08.402714+0000) 2022-01-31T21:25:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:08 smithi167 conmon[60316]: debug 2022-01-31T21:25:08.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:08.406821+0000) 2022-01-31T21:25:08.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:08 smithi171 conmon[46715]: debug 2022-01-31T21:25:08.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:08.179802+0000) 2022-01-31T21:25:08.842 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:08 smithi171 conmon[35325]: debug 2022-01-31T21:25:08.675+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:25:08.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:08 smithi171 conmon[51620]: debug 2022-01-31T21:25:08.759+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:08.760314+0000) 2022-01-31T21:25:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:08 smithi171 conmon[41853]: debug 2022-01-31T21:25:08.883+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:08.883986+0000) 2022-01-31T21:25:09.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:08 smithi167 conmon[49112]: debug 2022-01-31T21:25:08.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:08.735774+0000) 2022-01-31T21:25:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[60316]: debug 2022-01-31T21:25:09.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.407033+0000) 2022-01-31T21:25:09.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[54076]: debug 2022-01-31T21:25:09.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.402917+0000) 2022-01-31T21:25:09.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[46715]: debug 2022-01-31T21:25:09.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.179955+0000) 2022-01-31T21:25:09.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[49112]: debug 2022-01-31T21:25:09.502+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.502804+0000) 2022-01-31T21:25:09.722 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[60316]: debug 2022-01-31T21:25:09.501+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.502231+0000) 2022-01-31T21:25:09.722 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[54076]: debug 2022-01-31T21:25:09.723 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[54076]: 2022-01-31T21:25:09.503+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.503576+0000) 2022-01-31T21:25:09.759 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[35325]: debug 2022-01-31T21:25:09.514+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128194 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:09.759 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[46715]: debug 2022-01-31T21:25:09.501+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.501955+0000) 2022-01-31T21:25:09.760 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[51620]: debug 2022-01-31T21:25:09.502+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.502744+0000) 2022-01-31T21:25:09.760 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[41853]: debug 2022-01-31T21:25:09.501+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.502120+0000) 2022-01-31T21:25:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[41853]: debug 2022-01-31T21:25:09.883+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.884168+0000) 2022-01-31T21:25:10.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:09 smithi171 conmon[51620]: debug 2022-01-31T21:25:09.759+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.760513+0000) 2022-01-31T21:25:10.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:09 smithi167 conmon[49112]: debug 2022-01-31T21:25:09.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:09.736028+0000) 2022-01-31T21:25:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:10 smithi167 conmon[54076]: debug 2022-01-31T21:25:10.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:10.403114+0000) 2022-01-31T21:25:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:10 smithi167 conmon[60316]: debug 2022-01-31T21:25:10.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:10.407231+0000) 2022-01-31T21:25:10.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:10 smithi171 conmon[46715]: debug 2022-01-31T21:25:10.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:10.180081+0000) 2022-01-31T21:25:10.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:10 smithi171 conmon[51620]: debug 2022-01-31T21:25:10.759+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:10.760695+0000) 2022-01-31T21:25:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:10 smithi171 conmon[41853]: debug 2022-01-31T21:25:10.883+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:10.884309+0000) 2022-01-31T21:25:11.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:10 smithi167 conmon[49112]: debug 2022-01-31T21:25:10.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:10.736211+0000) 2022-01-31T21:25:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:11 smithi167 conmon[60316]: debug 2022-01-31T21:25:11.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:11.407400+0000) 2022-01-31T21:25:11.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:11 smithi167 conmon[54076]: debug 2022-01-31T21:25:11.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:11.403296+0000) 2022-01-31T21:25:11.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:11 smithi171 conmon[46715]: debug 2022-01-31T21:25:11.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:11.180219+0000) 2022-01-31T21:25:11.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:11 smithi171 conmon[51620]: debug 2022-01-31T21:25:11.760+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:11.760894+0000) 2022-01-31T21:25:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:11 smithi171 conmon[41853]: debug 2022-01-31T21:25:11.883+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:11.884488+0000) 2022-01-31T21:25:12.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:11 smithi167 conmon[49112]: debug 2022-01-31T21:25:11.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:11.736437+0000) 2022-01-31T21:25:12.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:12 smithi167 conmon[54076]: debug 2022-01-31T21:25:12.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:12.403525+0000) 2022-01-31T21:25:12.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:12 smithi167 conmon[60316]: debug 2022-01-31T21:25:12.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:12.407581+0000) 2022-01-31T21:25:12.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:12 smithi171 conmon[46715]: debug 2022-01-31T21:25:12.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:12.180422+0000) 2022-01-31T21:25:12.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:12 smithi171 conmon[51620]: debug 2022-01-31T21:25:12.760+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:12.761081+0000) 2022-01-31T21:25:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:12 smithi171 conmon[41853]: debug 2022-01-31T21:25:12.883+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:12.884706+0000) 2022-01-31T21:25:13.158 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:12 smithi167 conmon[49112]: debug 2022-01-31T21:25:12.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:12.736610+0000) 2022-01-31T21:25:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:13 smithi167 conmon[54076]: debug 2022-01-31T21:25:13.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:13.403707+0000) 2022-01-31T21:25:13.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:13 smithi167 conmon[60316]: debug 2022-01-31T21:25:13.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:13.407732+0000) 2022-01-31T21:25:13.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:13 smithi171 conmon[46715]: debug 2022-01-31T21:25:13.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:13.180607+0000) 2022-01-31T21:25:13.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:13 smithi171 conmon[51620]: debug 2022-01-31T21:25:13.760+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:13.761275+0000) 2022-01-31T21:25:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:13 smithi171 conmon[41853]: debug 2022-01-31T21:25:13.884+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:13.884924+0000) 2022-01-31T21:25:14.158 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:13 smithi167 conmon[49112]: debug 2022-01-31T21:25:13.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:13.736745+0000) 2022-01-31T21:25:14.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:14 smithi167 conmon[60316]: debug 2022-01-31T21:25:14.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.407918+0000) 2022-01-31T21:25:14.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:14 smithi167 conmon[54076]: debug 2022-01-31T21:25:14.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.403865+0000) 2022-01-31T21:25:14.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[46715]: debug 2022-01-31T21:25:14.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.180796+0000) 2022-01-31T21:25:14.722 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:14 smithi167 conmon[49112]: debug 2022-01-31T21:25:14.518+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.519132+0000) 2022-01-31T21:25:14.723 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:14 smithi167 conmon[54076]: debug 2022-01-31T21:25:14.518+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.518957+0000) 2022-01-31T21:25:14.723 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:14 smithi167 conmon[60316]: debug 2022-01-31T21:25:14.517+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.517989+0000) 2022-01-31T21:25:14.843 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[41853]: debug 2022-01-31T21:25:14.517+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.518405+0000) 2022-01-31T21:25:14.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[46715]: debug 2022-01-31T21:25:14.516+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.517385+0000) 2022-01-31T21:25:14.844 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[35325]: debug 2022-01-31T21:25:14.530+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128306 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:14.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[51620]: debug 2022-01-31T21:25:14.517+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.518686+0000) 2022-01-31T21:25:14.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[51620]: debug 2022-01-31T21:25:14.760+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.761505+0000) 2022-01-31T21:25:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:14 smithi171 conmon[41853]: debug 2022-01-31T21:25:14.884+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.885094+0000) 2022-01-31T21:25:15.158 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:14 smithi167 conmon[49112]: debug 2022-01-31T21:25:14.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:14.736982+0000) 2022-01-31T21:25:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:15 smithi167 conmon[54076]: debug 2022-01-31T21:25:15.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:15.404074+0000) 2022-01-31T21:25:15.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:15 smithi167 conmon[60316]: debug 2022-01-31T21:25:15.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:15.408094+0000) 2022-01-31T21:25:15.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:15 smithi171 conmon[46715]: debug 2022-01-31T21:25:15.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:15.180957+0000) 2022-01-31T21:25:15.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:15 smithi171 conmon[51620]: debug 2022-01-31T21:25:15.760+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:15.761656+0000) 2022-01-31T21:25:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:15 smithi171 conmon[41853]: debug 2022-01-31T21:25:15.884+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:15.885256+0000) 2022-01-31T21:25:16.158 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:15 smithi167 conmon[49112]: debug 2022-01-31T21:25:15.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:15.737169+0000) 2022-01-31T21:25:16.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:16 smithi167 conmon[54076]: debug 2022-01-31T21:25:16.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:16.404246+0000) 2022-01-31T21:25:16.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:16 smithi167 conmon[60316]: debug 2022-01-31T21:25:16.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:16.408289+0000) 2022-01-31T21:25:16.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:16 smithi171 conmon[46715]: debug 2022-01-31T21:25:16.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:16.181111+0000) 2022-01-31T21:25:16.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:16 smithi171 conmon[51620]: debug 2022-01-31T21:25:16.761+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:16.761844+0000) 2022-01-31T21:25:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:16 smithi171 conmon[41853]: debug 2022-01-31T21:25:16.884+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:16.885409+0000) 2022-01-31T21:25:17.158 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:16 smithi167 conmon[49112]: debug 2022-01-31T21:25:16.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:16.737377+0000) 2022-01-31T21:25:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:17 smithi167 conmon[54076]: debug 2022-01-31T21:25:17.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:17.404449+0000) 2022-01-31T21:25:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:17 smithi167 conmon[60316]: debug 2022-01-31T21:25:17.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:17.408382+0000) 2022-01-31T21:25:17.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:17 smithi171 conmon[46715]: debug 2022-01-31T21:25:17.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:17.181293+0000) 2022-01-31T21:25:17.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:17 smithi171 conmon[51620]: debug 2022-01-31T21:25:17.761+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:17.762038+0000) 2022-01-31T21:25:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:17 smithi171 conmon[41853]: debug 2022-01-31T21:25:17.884+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:17.885595+0000) 2022-01-31T21:25:18.159 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:17 smithi167 conmon[49112]: debug 2022-01-31T21:25:17.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:17.737582+0000) 2022-01-31T21:25:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:18 smithi167 conmon[54076]: debug 2022-01-31T21:25:18.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:18.404653+0000) 2022-01-31T21:25:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:18 smithi167 conmon[60316]: debug 2022-01-31T21:25:18.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:18.408565+0000) 2022-01-31T21:25:18.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:18 smithi171 conmon[46715]: debug 2022-01-31T21:25:18.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:18.181503+0000) 2022-01-31T21:25:18.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:18 smithi171 conmon[51620]: debug 2022-01-31T21:25:18.761+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:18.762265+0000) 2022-01-31T21:25:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:18 smithi171 conmon[41853]: debug 2022-01-31T21:25:18.885+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:18.885759+0000) 2022-01-31T21:25:19.159 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:18 smithi167 conmon[49112]: debug 2022-01-31T21:25:18.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:18.737762+0000) 2022-01-31T21:25:19.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:19 smithi167 conmon[54076]: debug 2022-01-31T21:25:19.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.404851+0000) 2022-01-31T21:25:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:19 smithi167 conmon[60316]: debug 2022-01-31T21:25:19.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.408738+0000) 2022-01-31T21:25:19.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[46715]: debug 2022-01-31T21:25:19.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.181659+0000) 2022-01-31T21:25:19.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:19 smithi167 conmon[49112]: debug 2022-01-31T21:25:19.534+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.534740+0000) 2022-01-31T21:25:19.724 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:19 smithi167 conmon[54076]: debug 2022-01-31T21:25:19.533+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.533990+0000) 2022-01-31T21:25:19.724 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:19 smithi167 conmon[60316]: debug 2022-01-31T21:25:19.533+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.534310+0000) 2022-01-31T21:25:19.844 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[35325]: debug 2022-01-31T21:25:19.545+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128415 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:19.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[41853]: debug 2022-01-31T21:25:19.533+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.534453+0000) 2022-01-31T21:25:19.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[46715]: debug 2022-01-31T21:25:19.533+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.533770+0000) 2022-01-31T21:25:19.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[51620]: debug 2022-01-31T21:25:19.532+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.533327+0000) 2022-01-31T21:25:19.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[51620]: debug 2022-01-31T21:25:19.761+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.762448+0000) 2022-01-31T21:25:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:19 smithi171 conmon[41853]: debug 2022-01-31T21:25:19.885+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.885940+0000) 2022-01-31T21:25:20.159 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:19 smithi167 conmon[49112]: debug 2022-01-31T21:25:19.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:19.737982+0000) 2022-01-31T21:25:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:20 smithi167 conmon[54076]: debug 2022-01-31T21:25:20.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:20.405034+0000) 2022-01-31T21:25:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:20 smithi167 conmon[60316]: debug 2022-01-31T21:25:20.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:20.408884+0000) 2022-01-31T21:25:20.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:20 smithi171 conmon[46715]: debug 2022-01-31T21:25:20.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:20.181809+0000) 2022-01-31T21:25:20.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:20 smithi171 conmon[51620]: debug 2022-01-31T21:25:20.762+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:20.762586+0000) 2022-01-31T21:25:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:20 smithi171 conmon[41853]: debug 2022-01-31T21:25:20.885+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:20.886126+0000) 2022-01-31T21:25:21.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:20 smithi167 conmon[49112]: debug 2022-01-31T21:25:20.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:20.738168+0000) 2022-01-31T21:25:21.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:21 smithi167 conmon[54076]: debug 2022-01-31T21:25:21.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:21.405191+0000) 2022-01-31T21:25:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:21 smithi167 conmon[60316]: debug 2022-01-31T21:25:21.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:21.409053+0000) 2022-01-31T21:25:21.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:21 smithi171 conmon[46715]: debug 2022-01-31T21:25:21.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:21.181956+0000) 2022-01-31T21:25:21.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:21 smithi171 conmon[51620]: debug 2022-01-31T21:25:21.762+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:21.762765+0000) 2022-01-31T21:25:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:21 smithi171 conmon[41853]: debug 2022-01-31T21:25:21.885+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:21.886329+0000) 2022-01-31T21:25:22.159 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:21 smithi167 conmon[49112]: debug 2022-01-31T21:25:21.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:21.738333+0000) 2022-01-31T21:25:22.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:22 smithi167 conmon[54076]: debug 2022-01-31T21:25:22.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:22.405366+0000) 2022-01-31T21:25:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:22 smithi167 conmon[60316]: debug 2022-01-31T21:25:22.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:22.409259+0000) 2022-01-31T21:25:22.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:22 smithi171 conmon[46715]: debug 2022-01-31T21:25:22.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:22.182145+0000) 2022-01-31T21:25:22.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:22 smithi171 conmon[51620]: debug 2022-01-31T21:25:22.762+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:22.762948+0000) 2022-01-31T21:25:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:22 smithi171 conmon[41853]: debug 2022-01-31T21:25:22.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:22.886489+0000) 2022-01-31T21:25:23.159 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:22 smithi167 conmon[49112]: debug 2022-01-31T21:25:22.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:22.738605+0000) 2022-01-31T21:25:23.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:23 smithi167 conmon[54076]: debug 2022-01-31T21:25:23.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:23.405553+0000) 2022-01-31T21:25:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:23 smithi167 conmon[60316]: debug 2022-01-31T21:25:23.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:23.409435+0000) 2022-01-31T21:25:23.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:23 smithi171 conmon[46715]: debug 2022-01-31T21:25:23.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:23.182319+0000) 2022-01-31T21:25:23.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:23 smithi171 conmon[35325]: debug 2022-01-31T21:25:23.676+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:25:23.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:23 smithi171 conmon[51620]: debug 2022-01-31T21:25:23.762+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:23.763137+0000) 2022-01-31T21:25:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:23 smithi171 conmon[41853]: debug 2022-01-31T21:25:23.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:23.886645+0000) 2022-01-31T21:25:24.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:23 smithi167 conmon[49112]: debug 2022-01-31T21:25:23.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:23.738793+0000) 2022-01-31T21:25:24.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:24 smithi167 conmon[54076]: debug 2022-01-31T21:25:24.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.405689+0000) 2022-01-31T21:25:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:24 smithi167 conmon[60316]: debug 2022-01-31T21:25:24.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.409570+0000) 2022-01-31T21:25:24.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[46715]: debug 2022-01-31T21:25:24.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.182495+0000) 2022-01-31T21:25:24.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:24 smithi167 conmon[49112]: debug 2022-01-31T21:25:24.549+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.550232+0000) 2022-01-31T21:25:24.724 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:24 smithi167 conmon[54076]: debug 2022-01-31T21:25:24.549+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.549958+0000) 2022-01-31T21:25:24.725 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:24 smithi167 conmon[60316]: debug 2022-01-31T21:25:24.548+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.548784+0000) 2022-01-31T21:25:24.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[35325]: debug 2022-01-31T21:25:24.561+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128537 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:24.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[41853]: debug 2022-01-31T21:25:24.548+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.549268+0000) 2022-01-31T21:25:24.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[46715]: debug 2022-01-31T21:25:24.548+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.548826+0000) 2022-01-31T21:25:24.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[51620]: debug 2022-01-31T21:25:24.549+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.550061+0000) 2022-01-31T21:25:24.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[51620]: debug 2022-01-31T21:25:24.762+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.763320+0000) 2022-01-31T21:25:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:24 smithi171 conmon[41853]: debug 2022-01-31T21:25:24.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.886767+0000) 2022-01-31T21:25:25.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:24 smithi167 conmon[49112]: debug 2022-01-31T21:25:24.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:24.738984+0000) 2022-01-31T21:25:25.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:25 smithi167 conmon[54076]: debug 2022-01-31T21:25:25.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:25.405865+0000) 2022-01-31T21:25:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:25 smithi167 conmon[60316]: debug 2022-01-31T21:25:25.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:25.409707+0000) 2022-01-31T21:25:25.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:25 smithi171 conmon[46715]: debug 2022-01-31T21:25:25.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:25.182695+0000) 2022-01-31T21:25:25.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:25 smithi171 conmon[51620]: debug 2022-01-31T21:25:25.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:25.763500+0000) 2022-01-31T21:25:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:25 smithi171 conmon[41853]: debug 2022-01-31T21:25:25.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:25.886900+0000) 2022-01-31T21:25:26.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:25 smithi167 conmon[49112]: debug 2022-01-31T21:25:25.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:25.739167+0000) 2022-01-31T21:25:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:26 smithi167 conmon[54076]: debug 2022-01-31T21:25:26.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:26.406023+0000) 2022-01-31T21:25:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:26 smithi167 conmon[60316]: debug 2022-01-31T21:25:26.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:26.409844+0000) 2022-01-31T21:25:26.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:26 smithi171 conmon[46715]: debug 2022-01-31T21:25:26.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:26.182850+0000) 2022-01-31T21:25:26.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:26 smithi171 conmon[51620]: debug 2022-01-31T21:25:26.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:26.763635+0000) 2022-01-31T21:25:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:26 smithi171 conmon[41853]: debug 2022-01-31T21:25:26.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:26.887089+0000) 2022-01-31T21:25:27.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:26 smithi167 conmon[49112]: debug 2022-01-31T21:25:26.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:26.739354+0000) 2022-01-31T21:25:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:27 smithi167 conmon[60316]: debug 2022-01-31T21:25:27.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:27.409939+0000) 2022-01-31T21:25:27.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:27 smithi167 conmon[54076]: debug 2022-01-31T21:25:27.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:27.406186+0000) 2022-01-31T21:25:27.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:27 smithi171 conmon[46715]: debug 2022-01-31T21:25:27.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:27.183011+0000) 2022-01-31T21:25:27.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:27 smithi171 conmon[51620]: debug 2022-01-31T21:25:27.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:27.763838+0000) 2022-01-31T21:25:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:27 smithi171 conmon[41853]: debug 2022-01-31T21:25:27.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:27.887293+0000) 2022-01-31T21:25:28.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:27 smithi167 conmon[49112]: debug 2022-01-31T21:25:27.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:27.739543+0000) 2022-01-31T21:25:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:28 smithi167 conmon[54076]: debug 2022-01-31T21:25:28.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:28.406367+0000) 2022-01-31T21:25:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:28 smithi167 conmon[60316]: debug 2022-01-31T21:25:28.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:28.410146+0000) 2022-01-31T21:25:28.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:28 smithi171 conmon[46715]: debug 2022-01-31T21:25:28.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:28.183176+0000) 2022-01-31T21:25:28.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:28 smithi171 conmon[51620]: debug 2022-01-31T21:25:28.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:28.764030+0000) 2022-01-31T21:25:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:28 smithi171 conmon[41853]: debug 2022-01-31T21:25:28.886+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:28.887489+0000) 2022-01-31T21:25:29.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:28 smithi167 conmon[49112]: debug 2022-01-31T21:25:28.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:28.739685+0000) 2022-01-31T21:25:29.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:29 smithi167 conmon[60316]: debug 2022-01-31T21:25:29.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.410338+0000) 2022-01-31T21:25:29.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:29 smithi167 conmon[54076]: debug 2022-01-31T21:25:29.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.406580+0000) 2022-01-31T21:25:29.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[46715]: debug 2022-01-31T21:25:29.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.183416+0000) 2022-01-31T21:25:29.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:29 smithi167 conmon[49112]: debug 2022-01-31T21:25:29.563+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.564113+0000) 2022-01-31T21:25:29.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:29 smithi167 conmon[54076]: debug 2022-01-31T21:25:29.565+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.565957+0000) 2022-01-31T21:25:29.727 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:29 smithi167 conmon[60316]: debug 2022-01-31T21:25:29.564+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.564940+0000) 2022-01-31T21:25:29.846 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[35325]: debug 2022-01-31T21:25:29.577+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128646 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:29.846 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[41853]: debug 2022-01-31T21:25:29.564+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.565312+0000) 2022-01-31T21:25:29.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[46715]: debug 2022-01-31T21:25:29.562+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.563718+0000) 2022-01-31T21:25:29.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[51620]: debug 2022-01-31T21:25:29.564+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.565579+0000) 2022-01-31T21:25:29.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[51620]: debug 2022-01-31T21:25:29.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.764150+0000) 2022-01-31T21:25:30.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:29 smithi171 conmon[41853]: debug 2022-01-31T21:25:29.887+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.887684+0000) 2022-01-31T21:25:30.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:29 smithi167 conmon[49112]: debug 2022-01-31T21:25:29.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:29.739912+0000) 2022-01-31T21:25:30.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:30 smithi167 conmon[60316]: debug 2022-01-31T21:25:30.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:30.410576+0000) 2022-01-31T21:25:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:30 smithi167 conmon[54076]: debug 2022-01-31T21:25:30.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:30.406764+0000) 2022-01-31T21:25:30.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:30 smithi171 conmon[46715]: debug 2022-01-31T21:25:30.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:30.183543+0000) 2022-01-31T21:25:30.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:30 smithi171 conmon[51620]: debug 2022-01-31T21:25:30.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:30.764323+0000) 2022-01-31T21:25:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:30 smithi171 conmon[41853]: debug 2022-01-31T21:25:30.887+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:30.887828+0000) 2022-01-31T21:25:31.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:30 smithi167 conmon[49112]: debug 2022-01-31T21:25:30.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:30.740073+0000) 2022-01-31T21:25:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:31 smithi167 conmon[54076]: debug 2022-01-31T21:25:31.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:31.406897+0000) 2022-01-31T21:25:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:31 smithi167 conmon[60316]: debug 2022-01-31T21:25:31.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:31.410682+0000) 2022-01-31T21:25:31.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:31 smithi171 conmon[46715]: debug 2022-01-31T21:25:31.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:31.183698+0000) 2022-01-31T21:25:31.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:31 smithi171 conmon[51620]: debug 2022-01-31T21:25:31.763+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:31.764489+0000) 2022-01-31T21:25:32.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:31 smithi171 conmon[41853]: debug 2022-01-31T21:25:31.887+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:31.887979+0000) 2022-01-31T21:25:32.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:31 smithi167 conmon[49112]: debug 2022-01-31T21:25:31.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:31.740283+0000) 2022-01-31T21:25:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:32 smithi167 conmon[54076]: debug 2022-01-31T21:25:32.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:32.407078+0000) 2022-01-31T21:25:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:32 smithi167 conmon[60316]: debug 2022-01-31T21:25:32.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:32.410852+0000) 2022-01-31T21:25:32.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:32 smithi171 conmon[46715]: debug 2022-01-31T21:25:32.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:32.183832+0000) 2022-01-31T21:25:32.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:32 smithi171 conmon[51620]: debug 2022-01-31T21:25:32.764+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:32.764657+0000) 2022-01-31T21:25:33.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:32 smithi171 conmon[41853]: debug 2022-01-31T21:25:32.887+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:32.888172+0000) 2022-01-31T21:25:33.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:32 smithi167 conmon[49112]: debug 2022-01-31T21:25:32.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:32.740496+0000) 2022-01-31T21:25:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:33 smithi167 conmon[54076]: debug 2022-01-31T21:25:33.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:33.407217+0000) 2022-01-31T21:25:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:33 smithi167 conmon[60316]: debug 2022-01-31T21:25:33.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:33.410970+0000) 2022-01-31T21:25:33.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:33 smithi171 conmon[46715]: debug 2022-01-31T21:25:33.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:33.183984+0000) 2022-01-31T21:25:33.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:33 smithi171 conmon[51620]: debug 2022-01-31T21:25:33.764+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:33.764851+0000) 2022-01-31T21:25:34.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:33 smithi171 conmon[41853]: debug 2022-01-31T21:25:33.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:33.888386+0000) 2022-01-31T21:25:34.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:33 smithi167 conmon[49112]: debug 2022-01-31T21:25:33.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:33.740674+0000) 2022-01-31T21:25:34.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:34 smithi167 conmon[54076]: debug 2022-01-31T21:25:34.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.407377+0000) 2022-01-31T21:25:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:34 smithi167 conmon[60316]: debug 2022-01-31T21:25:34.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.411160+0000) 2022-01-31T21:25:34.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[46715]: debug 2022-01-31T21:25:34.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.184202+0000) 2022-01-31T21:25:34.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:34 smithi167 conmon[49112]: debug 2022-01-31T21:25:34.579+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.580879+0000) 2022-01-31T21:25:34.726 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:34 smithi167 conmon[54076]: debug 2022-01-31T21:25:34.580+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.581691+0000) 2022-01-31T21:25:34.727 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:34 smithi167 conmon[60316]: debug 2022-01-31T21:25:34.579+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.580604+0000) 2022-01-31T21:25:34.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[46715]: debug 2022-01-31T21:25:34.579+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.580111+0000) 2022-01-31T21:25:34.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[51620]: debug 2022-01-31T21:25:34.581+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.581972+0000) 2022-01-31T21:25:34.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[51620]: debug 2022-01-31T21:25:34.764+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.765039+0000) 2022-01-31T21:25:34.848 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[35325]: debug 2022-01-31T21:25:34.594+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128757 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:34.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[41853]: debug 2022-01-31T21:25:34.580+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.581224+0000) 2022-01-31T21:25:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:34 smithi171 conmon[41853]: debug 2022-01-31T21:25:34.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.888568+0000) 2022-01-31T21:25:35.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:34 smithi167 conmon[49112]: debug 2022-01-31T21:25:34.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:34.740822+0000) 2022-01-31T21:25:35.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:35 smithi167 conmon[54076]: debug 2022-01-31T21:25:35.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:35.407589+0000) 2022-01-31T21:25:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:35 smithi167 conmon[60316]: debug 2022-01-31T21:25:35.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:35.411362+0000) 2022-01-31T21:25:35.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:35 smithi171 conmon[46715]: debug 2022-01-31T21:25:35.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:35.184411+0000) 2022-01-31T21:25:35.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:35 smithi171 conmon[51620]: debug 2022-01-31T21:25:35.764+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:35.765197+0000) 2022-01-31T21:25:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:35 smithi171 conmon[41853]: debug 2022-01-31T21:25:35.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:35.888735+0000) 2022-01-31T21:25:36.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:35 smithi167 conmon[49112]: debug 2022-01-31T21:25:35.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:35.741029+0000) 2022-01-31T21:25:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:36 smithi167 conmon[54076]: debug 2022-01-31T21:25:36.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:36.407704+0000) 2022-01-31T21:25:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:36 smithi167 conmon[60316]: debug 2022-01-31T21:25:36.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:36.411554+0000) 2022-01-31T21:25:36.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:36 smithi171 conmon[46715]: debug 2022-01-31T21:25:36.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:36.184560+0000) 2022-01-31T21:25:36.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:36 smithi171 conmon[51620]: debug 2022-01-31T21:25:36.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:36.765396+0000) 2022-01-31T21:25:37.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:36 smithi171 conmon[41853]: debug 2022-01-31T21:25:36.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:36.888965+0000) 2022-01-31T21:25:37.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:36 smithi167 conmon[49112]: debug 2022-01-31T21:25:36.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:36.741232+0000) 2022-01-31T21:25:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:37 smithi167 conmon[54076]: debug 2022-01-31T21:25:37.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:37.407865+0000) 2022-01-31T21:25:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:37 smithi167 conmon[60316]: debug 2022-01-31T21:25:37.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:37.411734+0000) 2022-01-31T21:25:37.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:37 smithi171 conmon[46715]: debug 2022-01-31T21:25:37.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:37.184721+0000) 2022-01-31T21:25:37.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:37 smithi171 conmon[51620]: debug 2022-01-31T21:25:37.764+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:37.765570+0000) 2022-01-31T21:25:38.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:37 smithi171 conmon[41853]: debug 2022-01-31T21:25:37.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:37.889183+0000) 2022-01-31T21:25:38.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:37 smithi167 conmon[49112]: debug 2022-01-31T21:25:37.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:37.741442+0000) 2022-01-31T21:25:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:38 smithi167 conmon[54076]: debug 2022-01-31T21:25:38.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:38.408026+0000) 2022-01-31T21:25:38.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:38 smithi167 conmon[60316]: debug 2022-01-31T21:25:38.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:38.411924+0000) 2022-01-31T21:25:38.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:38 smithi171 conmon[46715]: debug 2022-01-31T21:25:38.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:38.184877+0000) 2022-01-31T21:25:38.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:38 smithi171 conmon[35325]: debug 2022-01-31T21:25:38.677+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:25:38.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:38 smithi171 conmon[51620]: debug 2022-01-31T21:25:38.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:38.765777+0000) 2022-01-31T21:25:39.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:38 smithi171 conmon[41853]: debug 2022-01-31T21:25:38.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:38.889301+0000) 2022-01-31T21:25:39.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:38 smithi167 conmon[49112]: debug 2022-01-31T21:25:38.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:38.741613+0000) 2022-01-31T21:25:39.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:39 smithi167 conmon[54076]: debug 2022-01-31T21:25:39.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.408243+0000) 2022-01-31T21:25:39.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:39 smithi167 conmon[60316]: debug 2022-01-31T21:25:39.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.412128+0000) 2022-01-31T21:25:39.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[46715]: debug 2022-01-31T21:25:39.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.185100+0000) 2022-01-31T21:25:39.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:39 smithi167 conmon[49112]: debug 2022-01-31T21:25:39.596+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.598270+0000) 2022-01-31T21:25:39.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:39 smithi167 conmon[54076]: debug 2022-01-31T21:25:39.596+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.598405+0000) 2022-01-31T21:25:39.728 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:39 smithi167 conmon[60316]: debug 2022-01-31T21:25:39.596+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.598031+0000) 2022-01-31T21:25:39.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[35325]: debug 2022-01-31T21:25:39.610+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128866 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:39.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[41853]: debug 2022-01-31T21:25:39.597+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.598118+0000) 2022-01-31T21:25:39.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[46715]: debug 2022-01-31T21:25:39.595+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.596498+0000) 2022-01-31T21:25:39.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[51620]: debug 2022-01-31T21:25:39.596+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.597408+0000) 2022-01-31T21:25:39.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[51620]: debug 2022-01-31T21:25:39.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.765984+0000) 2022-01-31T21:25:40.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:39 smithi171 conmon[41853]: debug 2022-01-31T21:25:39.888+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.889502+0000) 2022-01-31T21:25:40.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:39 smithi167 conmon[49112]: debug 2022-01-31T21:25:39.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:39.741793+0000) 2022-01-31T21:25:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:40 smithi167 conmon[54076]: debug 2022-01-31T21:25:40.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:40.408402+0000) 2022-01-31T21:25:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:40 smithi167 conmon[60316]: debug 2022-01-31T21:25:40.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:40.412320+0000) 2022-01-31T21:25:40.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:40 smithi171 conmon[46715]: debug 2022-01-31T21:25:40.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:40.185260+0000) 2022-01-31T21:25:40.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:40 smithi171 conmon[51620]: debug 2022-01-31T21:25:40.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:40.766145+0000) 2022-01-31T21:25:41.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:40 smithi171 conmon[41853]: debug 2022-01-31T21:25:40.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:40.889688+0000) 2022-01-31T21:25:41.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:40 smithi167 conmon[49112]: debug 2022-01-31T21:25:40.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:40.741951+0000) 2022-01-31T21:25:41.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:41 smithi167 conmon[54076]: debug 2022-01-31T21:25:41.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:41.408582+0000) 2022-01-31T21:25:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:41 smithi167 conmon[60316]: debug 2022-01-31T21:25:41.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:41.412437+0000) 2022-01-31T21:25:41.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:41 smithi171 conmon[46715]: debug 2022-01-31T21:25:41.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:41.185402+0000) 2022-01-31T21:25:41.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:41 smithi171 conmon[51620]: debug 2022-01-31T21:25:41.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:41.766333+0000) 2022-01-31T21:25:42.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:41 smithi171 conmon[41853]: debug 2022-01-31T21:25:41.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:41.889850+0000) 2022-01-31T21:25:42.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:41 smithi167 conmon[49112]: debug 2022-01-31T21:25:41.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:41.742165+0000) 2022-01-31T21:25:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:42 smithi167 conmon[54076]: debug 2022-01-31T21:25:42.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:42.408786+0000) 2022-01-31T21:25:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:42 smithi167 conmon[60316]: debug 2022-01-31T21:25:42.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:42.412521+0000) 2022-01-31T21:25:42.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:42 smithi171 conmon[46715]: debug 2022-01-31T21:25:42.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:42.185585+0000) 2022-01-31T21:25:42.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:42 smithi171 conmon[51620]: debug 2022-01-31T21:25:42.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:42.766523+0000) 2022-01-31T21:25:43.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:42 smithi171 conmon[41853]: debug 2022-01-31T21:25:42.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:42.889973+0000) 2022-01-31T21:25:43.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:42 smithi167 conmon[49112]: debug 2022-01-31T21:25:42.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:42.742366+0000) 2022-01-31T21:25:43.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:43 smithi167 conmon[54076]: debug 2022-01-31T21:25:43.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:43.408980+0000) 2022-01-31T21:25:43.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:43 smithi167 conmon[60316]: debug 2022-01-31T21:25:43.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:43.412643+0000) 2022-01-31T21:25:43.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:43 smithi171 conmon[46715]: debug 2022-01-31T21:25:43.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:43.185774+0000) 2022-01-31T21:25:43.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:43 smithi171 conmon[51620]: debug 2022-01-31T21:25:43.766+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:43.766726+0000) 2022-01-31T21:25:44.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:43 smithi171 conmon[41853]: debug 2022-01-31T21:25:43.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:43.890136+0000) 2022-01-31T21:25:44.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:43 smithi167 conmon[49112]: debug 2022-01-31T21:25:43.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:43.742596+0000) 2022-01-31T21:25:44.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[54076]: debug 2022-01-31T21:25:44.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.409192+0000) 2022-01-31T21:25:44.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[60316]: debug 2022-01-31T21:25:44.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.412814+0000) 2022-01-31T21:25:44.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[46715]: debug 2022-01-31T21:25:44.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.185909+0000) 2022-01-31T21:25:44.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[54076]: debug 2022-01-31T21:25:44.613+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.614822+0000) 2022-01-31T21:25:44.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[49112]: debug 2022-01-31T21:25:44.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[49112]: 2022-01-31T21:25:44.612+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.614300+0000) 2022-01-31T21:25:44.729 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[60316]: debug 2022-01-31T21:25:44.613+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.614720+0000) 2022-01-31T21:25:44.848 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[35325]: debug 2022-01-31T21:25:44.625+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 128977 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:44.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[41853]: debug 2022-01-31T21:25:44.612+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.613812+0000) 2022-01-31T21:25:44.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[46715]: debug 2022-01-31T21:25:44.611+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.613456+0000) 2022-01-31T21:25:44.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[51620]: debug 2022-01-31T21:25:44.611+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.612781+0000) 2022-01-31T21:25:44.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[51620]: debug 2022-01-31T21:25:44.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.766874+0000) 2022-01-31T21:25:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:44 smithi171 conmon[41853]: debug 2022-01-31T21:25:44.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.890332+0000) 2022-01-31T21:25:45.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:44 smithi167 conmon[49112]: debug 2022-01-31T21:25:44.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:44.742820+0000) 2022-01-31T21:25:45.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:45 smithi167 conmon[60316]: debug 2022-01-31T21:25:45.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:45.413008+0000) 2022-01-31T21:25:45.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:45 smithi167 conmon[54076]: debug 2022-01-31T21:25:45.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:45.409396+0000) 2022-01-31T21:25:45.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:45 smithi171 conmon[46715]: debug 2022-01-31T21:25:45.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:45.186090+0000) 2022-01-31T21:25:45.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:45 smithi171 conmon[51620]: debug 2022-01-31T21:25:45.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:45.767029+0000) 2022-01-31T21:25:46.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:45 smithi171 conmon[41853]: debug 2022-01-31T21:25:45.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:45.890468+0000) 2022-01-31T21:25:46.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:45 smithi167 conmon[49112]: debug 2022-01-31T21:25:45.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:45.742976+0000) 2022-01-31T21:25:46.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:46 smithi167 conmon[54076]: debug 2022-01-31T21:25:46.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:46.409551+0000) 2022-01-31T21:25:46.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:46 smithi167 conmon[60316]: debug 2022-01-31T21:25:46.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:46.413182+0000) 2022-01-31T21:25:46.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:46 smithi171 conmon[46715]: debug 2022-01-31T21:25:46.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:46.186235+0000) 2022-01-31T21:25:46.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:46 smithi171 conmon[51620]: debug 2022-01-31T21:25:46.766+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:46.767224+0000) 2022-01-31T21:25:47.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:46 smithi171 conmon[41853]: debug 2022-01-31T21:25:46.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:46.890654+0000) 2022-01-31T21:25:47.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:46 smithi167 conmon[49112]: debug 2022-01-31T21:25:46.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:46.743157+0000) 2022-01-31T21:25:47.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:47 smithi167 conmon[60316]: debug 2022-01-31T21:25:47.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:47.413370+0000) 2022-01-31T21:25:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:47 smithi167 conmon[54076]: debug 2022-01-31T21:25:47.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:47.409725+0000) 2022-01-31T21:25:47.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:47 smithi171 conmon[46715]: debug 2022-01-31T21:25:47.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:47.186392+0000) 2022-01-31T21:25:47.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:47 smithi171 conmon[51620]: debug 2022-01-31T21:25:47.766+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:47.767414+0000) 2022-01-31T21:25:48.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:47 smithi171 conmon[41853]: debug 2022-01-31T21:25:47.889+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:47.890842+0000) 2022-01-31T21:25:48.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:47 smithi167 conmon[49112]: debug 2022-01-31T21:25:47.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:47.743387+0000) 2022-01-31T21:25:48.223 INFO:teuthology.orchestra.run.smithi167.stderr:2022-01-31T21:25:48.223+0000 7fd128bc0700 0 monclient(hunting): authenticate timed out after 300 2022-01-31T21:25:48.223 INFO:teuthology.orchestra.run.smithi167.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-01-31T21:25:48.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:48 smithi167 conmon[54076]: debug 2022-01-31T21:25:48.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:48.409893+0000) 2022-01-31T21:25:48.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:48 smithi167 conmon[60316]: debug 2022-01-31T21:25:48.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:48.413547+0000) 2022-01-31T21:25:48.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:48 smithi171 conmon[46715]: debug 2022-01-31T21:25:48.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:48.186553+0000) 2022-01-31T21:25:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-31T21:25:48.585 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa/tasks/cephadm.py", line 1101, in shell extra_cephadm_args=args) File "/home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa/tasks/cephadm.py", line 47, in _shell **kwargs File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph orch ps'" 2022-01-31T21:25:48.683 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c9f2eae8b19e42f49caab78f3ce3021b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa/tasks/cephadm.py", line 1101, in shell extra_cephadm_args=args) File "/home/teuthworker/src/github.com_batrick_ceph_e7bdcb25f366a81de92a121020949aeab8ce71c7/qa/tasks/cephadm.py", line 47, in _shell **kwargs File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1c5a1df6-82cd-11ec-8c35-001a4aab830c -e sha1=29e1fc1722aa5915b44828a5ad02ec45ce760aa3 -- bash -c 'ceph orch ps'" 2022-01-31T21:25:48.687 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2022-01-31T21:25:48.696 INFO:tasks.kclient:Unmounting kernel clients... 2022-01-31T21:25:48.697 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2022-01-31T21:25:48.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-01-31T21:25:48.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo umount /home/ubuntu/cephtest/mnt.0 2022-01-31T21:25:48.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:48 smithi171 conmon[51620]: debug 2022-01-31T21:25:48.765+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:48.767599+0000) 2022-01-31T21:25:49.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:48 smithi171 conmon[41853]: debug 2022-01-31T21:25:48.890+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:48.891044+0000) 2022-01-31T21:25:49.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:48 smithi167 conmon[49112]: debug 2022-01-31T21:25:48.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:48.743525+0000) 2022-01-31T21:25:49.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:49 smithi167 conmon[54076]: debug 2022-01-31T21:25:49.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.410029+0000) 2022-01-31T21:25:49.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:49 smithi167 conmon[60316]: debug 2022-01-31T21:25:49.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.413670+0000) 2022-01-31T21:25:49.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[46715]: debug 2022-01-31T21:25:49.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.186743+0000) 2022-01-31T21:25:49.728 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:49 smithi167 conmon[49112]: debug 2022-01-31T21:25:49.627+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.629222+0000) 2022-01-31T21:25:49.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:49 smithi167 conmon[54076]: debug 2022-01-31T21:25:49.629+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.631216+0000) 2022-01-31T21:25:49.729 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:49 smithi167 conmon[60316]: debug 2022-01-31T21:25:49.629+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.630776+0000) 2022-01-31T21:25:49.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[35325]: debug 2022-01-31T21:25:49.641+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129086 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:49.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[41853]: debug 2022-01-31T21:25:49.629+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.630890+0000) 2022-01-31T21:25:49.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[46715]: debug 2022-01-31T21:25:49.627+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.629302+0000) 2022-01-31T21:25:49.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[51620]: debug 2022-01-31T21:25:49.627+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.629423+0000) 2022-01-31T21:25:49.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[51620]: debug 2022-01-31T21:25:49.766+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.767787+0000) 2022-01-31T21:25:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:49 smithi171 conmon[41853]: debug 2022-01-31T21:25:49.890+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.891284+0000) 2022-01-31T21:25:50.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:49 smithi167 conmon[49112]: debug 2022-01-31T21:25:49.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:49.743740+0000) 2022-01-31T21:25:50.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:50 smithi167 conmon[54076]: debug 2022-01-31T21:25:50.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:50.410243+0000) 2022-01-31T21:25:50.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:50 smithi167 conmon[60316]: debug 2022-01-31T21:25:50.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:50.413861+0000) 2022-01-31T21:25:50.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:50 smithi171 conmon[46715]: debug 2022-01-31T21:25:50.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:50.186922+0000) 2022-01-31T21:25:50.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:50 smithi171 conmon[51620]: debug 2022-01-31T21:25:50.767+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:50.767950+0000) 2022-01-31T21:25:51.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:50 smithi171 conmon[41853]: debug 2022-01-31T21:25:50.890+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:50.891400+0000) 2022-01-31T21:25:51.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:50 smithi167 conmon[49112]: debug 2022-01-31T21:25:50.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:50.743875+0000) 2022-01-31T21:25:51.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:51 smithi167 conmon[54076]: debug 2022-01-31T21:25:51.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:51.410398+0000) 2022-01-31T21:25:51.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:51 smithi167 conmon[60316]: debug 2022-01-31T21:25:51.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:51.414048+0000) 2022-01-31T21:25:51.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:51 smithi171 conmon[46715]: debug 2022-01-31T21:25:51.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:51.187087+0000) 2022-01-31T21:25:51.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:51 smithi171 conmon[51620]: debug 2022-01-31T21:25:51.767+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:51.768155+0000) 2022-01-31T21:25:52.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:51 smithi171 conmon[41853]: debug 2022-01-31T21:25:51.890+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:51.891591+0000) 2022-01-31T21:25:52.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:51 smithi167 conmon[49112]: debug 2022-01-31T21:25:51.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:51.744086+0000) 2022-01-31T21:25:52.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:52 smithi167 conmon[54076]: debug 2022-01-31T21:25:52.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:52.410591+0000) 2022-01-31T21:25:52.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:52 smithi167 conmon[60316]: debug 2022-01-31T21:25:52.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:52.414223+0000) 2022-01-31T21:25:52.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:52 smithi171 conmon[46715]: debug 2022-01-31T21:25:52.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:52.187294+0000) 2022-01-31T21:25:52.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:52 smithi171 conmon[51620]: debug 2022-01-31T21:25:52.766+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:52.768349+0000) 2022-01-31T21:25:53.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:52 smithi171 conmon[41853]: debug 2022-01-31T21:25:52.890+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:52.891767+0000) 2022-01-31T21:25:53.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:52 smithi167 conmon[49112]: debug 2022-01-31T21:25:52.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:52.744306+0000) 2022-01-31T21:25:53.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:53 smithi167 conmon[54076]: debug 2022-01-31T21:25:53.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:53.410757+0000) 2022-01-31T21:25:53.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:53 smithi167 conmon[60316]: debug 2022-01-31T21:25:53.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:53.414431+0000) 2022-01-31T21:25:53.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:53 smithi171 conmon[46715]: debug 2022-01-31T21:25:53.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:53.187502+0000) 2022-01-31T21:25:53.850 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:53 smithi171 conmon[35325]: debug 2022-01-31T21:25:53.677+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:25:53.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:53 smithi171 conmon[51620]: debug 2022-01-31T21:25:53.767+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:53.768522+0000) 2022-01-31T21:25:54.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:53 smithi171 conmon[41853]: debug 2022-01-31T21:25:53.891+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:53.891919+0000) 2022-01-31T21:25:54.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:53 smithi167 conmon[49112]: debug 2022-01-31T21:25:53.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:53.744559+0000) 2022-01-31T21:25:54.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:54 smithi167 conmon[54076]: debug 2022-01-31T21:25:54.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.410901+0000) 2022-01-31T21:25:54.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:54 smithi167 conmon[60316]: debug 2022-01-31T21:25:54.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.414580+0000) 2022-01-31T21:25:54.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[46715]: debug 2022-01-31T21:25:54.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.187714+0000) 2022-01-31T21:25:54.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:54 smithi167 conmon[49112]: debug 2022-01-31T21:25:54.645+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.646060+0000) 2022-01-31T21:25:54.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:54 smithi167 conmon[54076]: debug 2022-01-31T21:25:54.646+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.647193+0000) 2022-01-31T21:25:54.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:54 smithi167 conmon[60316]: debug 2022-01-31T21:25:54.645+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.646338+0000) 2022-01-31T21:25:54.850 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[35325]: debug 2022-01-31T21:25:54.659+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129197 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:54.851 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[41853]: debug 2022-01-31T21:25:54.645+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.647290+0000) 2022-01-31T21:25:54.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[46715]: debug 2022-01-31T21:25:54.644+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.646513+0000) 2022-01-31T21:25:54.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[51620]: debug 2022-01-31T21:25:54.644+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.646142+0000) 2022-01-31T21:25:54.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[51620]: debug 2022-01-31T21:25:54.767+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.768691+0000) 2022-01-31T21:25:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:54 smithi171 conmon[41853]: debug 2022-01-31T21:25:54.891+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.892078+0000) 2022-01-31T21:25:55.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:54 smithi167 conmon[49112]: debug 2022-01-31T21:25:54.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:54.744681+0000) 2022-01-31T21:25:55.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:55 smithi167 conmon[54076]: debug 2022-01-31T21:25:55.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:55.411093+0000) 2022-01-31T21:25:55.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:55 smithi167 conmon[60316]: debug 2022-01-31T21:25:55.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:55.414704+0000) 2022-01-31T21:25:55.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:55 smithi171 conmon[46715]: debug 2022-01-31T21:25:55.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:55.187926+0000) 2022-01-31T21:25:55.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:55 smithi171 conmon[51620]: debug 2022-01-31T21:25:55.768+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:55.768871+0000) 2022-01-31T21:25:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:55 smithi171 conmon[41853]: debug 2022-01-31T21:25:55.891+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:55.892264+0000) 2022-01-31T21:25:56.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:55 smithi167 conmon[49112]: debug 2022-01-31T21:25:55.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:55.744810+0000) 2022-01-31T21:25:56.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:56 smithi167 conmon[54076]: debug 2022-01-31T21:25:56.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:56.411284+0000) 2022-01-31T21:25:56.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:56 smithi167 conmon[60316]: debug 2022-01-31T21:25:56.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:56.414836+0000) 2022-01-31T21:25:56.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:56 smithi171 conmon[46715]: debug 2022-01-31T21:25:56.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:56.188067+0000) 2022-01-31T21:25:56.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:56 smithi171 conmon[51620]: debug 2022-01-31T21:25:56.768+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:56.769076+0000) 2022-01-31T21:25:57.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:56 smithi171 conmon[41853]: debug 2022-01-31T21:25:56.891+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:56.892408+0000) 2022-01-31T21:25:57.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:56 smithi167 conmon[49112]: debug 2022-01-31T21:25:56.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:56.745024+0000) 2022-01-31T21:25:57.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:57 smithi167 conmon[54076]: debug 2022-01-31T21:25:57.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:57.411467+0000) 2022-01-31T21:25:57.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:57 smithi167 conmon[60316]: debug 2022-01-31T21:25:57.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:57.415001+0000) 2022-01-31T21:25:57.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:57 smithi171 conmon[46715]: debug 2022-01-31T21:25:57.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:57.188242+0000) 2022-01-31T21:25:57.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:57 smithi171 conmon[51620]: debug 2022-01-31T21:25:57.768+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:57.769285+0000) 2022-01-31T21:25:58.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:57 smithi171 conmon[41853]: debug 2022-01-31T21:25:57.891+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:57.892611+0000) 2022-01-31T21:25:58.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:57 smithi167 conmon[49112]: debug 2022-01-31T21:25:57.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:57.745256+0000) 2022-01-31T21:25:58.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:58 smithi167 conmon[54076]: debug 2022-01-31T21:25:58.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:58.411633+0000) 2022-01-31T21:25:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:58 smithi167 conmon[60316]: debug 2022-01-31T21:25:58.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:58.415203+0000) 2022-01-31T21:25:58.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:58 smithi171 conmon[46715]: debug 2022-01-31T21:25:58.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:58.188429+0000) 2022-01-31T21:25:58.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:58 smithi171 conmon[51620]: debug 2022-01-31T21:25:58.768+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:58.769503+0000) 2022-01-31T21:25:59.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:58 smithi171 conmon[41853]: debug 2022-01-31T21:25:58.892+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:58.892858+0000) 2022-01-31T21:25:59.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:58 smithi167 conmon[49112]: debug 2022-01-31T21:25:58.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:58.745479+0000) 2022-01-31T21:25:59.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:59 smithi167 conmon[60316]: debug 2022-01-31T21:25:59.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.415381+0000) 2022-01-31T21:25:59.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:59 smithi167 conmon[54076]: debug 2022-01-31T21:25:59.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.411844+0000) 2022-01-31T21:25:59.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[46715]: debug 2022-01-31T21:25:59.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.188618+0000) 2022-01-31T21:25:59.730 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:59 smithi167 conmon[49112]: debug 2022-01-31T21:25:59.662+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.662824+0000) 2022-01-31T21:25:59.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:25:59 smithi167 conmon[54076]: debug 2022-01-31T21:25:59.661+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.662373+0000) 2022-01-31T21:25:59.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:25:59 smithi167 conmon[60316]: debug 2022-01-31T21:25:59.668+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.669310+0000) 2022-01-31T21:25:59.851 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[41853]: debug 2022-01-31T21:25:59.661+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.661942+0000) 2022-01-31T21:25:59.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[46715]: debug 2022-01-31T21:25:59.662+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.663093+0000) 2022-01-31T21:25:59.852 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[35325]: debug 2022-01-31T21:25:59.678+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129306 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:25:59.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[51620]: debug 2022-01-31T21:25:59.662+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.663142+0000) 2022-01-31T21:25:59.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[51620]: debug 2022-01-31T21:25:59.768+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.769668+0000) 2022-01-31T21:26:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:25:59 smithi171 conmon[41853]: debug 2022-01-31T21:25:59.892+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.893044+0000) 2022-01-31T21:26:00.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:25:59 smithi167 conmon[49112]: debug 2022-01-31T21:25:59.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:25:59.745674+0000) 2022-01-31T21:26:00.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:00 smithi167 conmon[60316]: debug 2022-01-31T21:26:00.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:00.415579+0000) 2022-01-31T21:26:00.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:00 smithi167 conmon[54076]: debug 2022-01-31T21:26:00.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:00.412027+0000) 2022-01-31T21:26:00.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:00 smithi171 conmon[46715]: debug 2022-01-31T21:26:00.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:00.188785+0000) 2022-01-31T21:26:00.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:00 smithi171 conmon[51620]: debug 2022-01-31T21:26:00.769+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:00.769825+0000) 2022-01-31T21:26:01.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:00 smithi171 conmon[41853]: debug 2022-01-31T21:26:00.892+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:00.893181+0000) 2022-01-31T21:26:01.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:00 smithi167 conmon[49112]: debug 2022-01-31T21:26:00.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:00.745838+0000) 2022-01-31T21:26:01.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:01 smithi167 conmon[60316]: debug 2022-01-31T21:26:01.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:01.415722+0000) 2022-01-31T21:26:01.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:01 smithi167 conmon[54076]: debug 2022-01-31T21:26:01.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:01.412200+0000) 2022-01-31T21:26:01.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:01 smithi171 conmon[46715]: debug 2022-01-31T21:26:01.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:01.188898+0000) 2022-01-31T21:26:01.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:01 smithi171 conmon[51620]: debug 2022-01-31T21:26:01.769+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:01.769970+0000) 2022-01-31T21:26:02.111 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:01 smithi171 conmon[41853]: debug 2022-01-31T21:26:01.892+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:01.893370+0000) 2022-01-31T21:26:02.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:01 smithi167 conmon[49112]: debug 2022-01-31T21:26:01.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:01.746052+0000) 2022-01-31T21:26:02.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:02 smithi167 conmon[60316]: debug 2022-01-31T21:26:02.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:02.415916+0000) 2022-01-31T21:26:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:02 smithi167 conmon[54076]: debug 2022-01-31T21:26:02.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:02.412387+0000) 2022-01-31T21:26:02.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:02 smithi171 conmon[46715]: debug 2022-01-31T21:26:02.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:02.189075+0000) 2022-01-31T21:26:02.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:02 smithi171 conmon[51620]: debug 2022-01-31T21:26:02.769+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:02.770122+0000) 2022-01-31T21:26:03.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:02 smithi171 conmon[41853]: debug 2022-01-31T21:26:02.892+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:02.893509+0000) 2022-01-31T21:26:03.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:02 smithi167 conmon[49112]: debug 2022-01-31T21:26:02.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:02.746300+0000) 2022-01-31T21:26:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:03 smithi167 conmon[60316]: debug 2022-01-31T21:26:03.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:03.416064+0000) 2022-01-31T21:26:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:03 smithi167 conmon[54076]: debug 2022-01-31T21:26:03.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:03.412597+0000) 2022-01-31T21:26:03.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:03 smithi171 conmon[46715]: debug 2022-01-31T21:26:03.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:03.189266+0000) 2022-01-31T21:26:03.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:03 smithi171 conmon[51620]: debug 2022-01-31T21:26:03.769+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:03.770313+0000) 2022-01-31T21:26:04.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:03 smithi171 conmon[41853]: debug 2022-01-31T21:26:03.893+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:03.893670+0000) 2022-01-31T21:26:04.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:03 smithi167 conmon[49112]: debug 2022-01-31T21:26:03.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:03.746551+0000) 2022-01-31T21:26:04.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:04 smithi167 conmon[54076]: debug 2022-01-31T21:26:04.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.412784+0000) 2022-01-31T21:26:04.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:04 smithi167 conmon[60316]: debug 2022-01-31T21:26:04.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.416245+0000) 2022-01-31T21:26:04.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[46715]: debug 2022-01-31T21:26:04.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.189428+0000) 2022-01-31T21:26:04.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:04 smithi167 conmon[49112]: debug 2022-01-31T21:26:04.681+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.682172+0000) 2022-01-31T21:26:04.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:04 smithi167 conmon[54076]: debug 2022-01-31T21:26:04.681+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.681920+0000) 2022-01-31T21:26:04.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:04 smithi167 conmon[60316]: debug 2022-01-31T21:26:04.682+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.683415+0000) 2022-01-31T21:26:04.852 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[35325]: debug 2022-01-31T21:26:04.694+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129417 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:04.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[41853]: debug 2022-01-31T21:26:04.680+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.681463+0000) 2022-01-31T21:26:04.853 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[46715]: debug 2022-01-31T21:26:04.680+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.681599+0000) 2022-01-31T21:26:04.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[51620]: debug 2022-01-31T21:26:04.681+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.682030+0000) 2022-01-31T21:26:04.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[51620]: debug 2022-01-31T21:26:04.769+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.770567+0000) 2022-01-31T21:26:05.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:04 smithi171 conmon[41853]: debug 2022-01-31T21:26:04.893+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.893875+0000) 2022-01-31T21:26:05.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:04 smithi167 conmon[49112]: debug 2022-01-31T21:26:04.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:04.746741+0000) 2022-01-31T21:26:05.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:05 smithi167 conmon[54076]: debug 2022-01-31T21:26:05.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:05.412964+0000) 2022-01-31T21:26:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:05 smithi167 conmon[60316]: debug 2022-01-31T21:26:05.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:05.416468+0000) 2022-01-31T21:26:05.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:05 smithi171 conmon[46715]: debug 2022-01-31T21:26:05.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:05.189634+0000) 2022-01-31T21:26:05.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:05 smithi171 conmon[51620]: debug 2022-01-31T21:26:05.770+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:05.770719+0000) 2022-01-31T21:26:06.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:05 smithi171 conmon[41853]: debug 2022-01-31T21:26:05.893+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:05.893980+0000) 2022-01-31T21:26:06.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:05 smithi167 conmon[49112]: debug 2022-01-31T21:26:05.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:05.746872+0000) 2022-01-31T21:26:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:06 smithi167 conmon[60316]: debug 2022-01-31T21:26:06.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:06.416584+0000) 2022-01-31T21:26:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:06 smithi167 conmon[54076]: debug 2022-01-31T21:26:06.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:06.413103+0000) 2022-01-31T21:26:06.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:06 smithi171 conmon[46715]: debug 2022-01-31T21:26:06.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:06.189793+0000) 2022-01-31T21:26:06.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:06 smithi171 conmon[51620]: debug 2022-01-31T21:26:06.770+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:06.770939+0000) 2022-01-31T21:26:07.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:06 smithi171 conmon[41853]: debug 2022-01-31T21:26:06.893+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:06.894164+0000) 2022-01-31T21:26:07.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:06 smithi167 conmon[49112]: debug 2022-01-31T21:26:06.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:06.747088+0000) 2022-01-31T21:26:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:07 smithi167 conmon[54076]: debug 2022-01-31T21:26:07.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:07.413213+0000) 2022-01-31T21:26:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:07 smithi167 conmon[60316]: debug 2022-01-31T21:26:07.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:07.416716+0000) 2022-01-31T21:26:07.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:07 smithi171 conmon[46715]: debug 2022-01-31T21:26:07.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:07.189956+0000) 2022-01-31T21:26:07.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:07 smithi171 conmon[51620]: debug 2022-01-31T21:26:07.770+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:07.771136+0000) 2022-01-31T21:26:08.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:07 smithi171 conmon[41853]: debug 2022-01-31T21:26:07.893+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:07.894324+0000) 2022-01-31T21:26:08.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:07 smithi167 conmon[49112]: debug 2022-01-31T21:26:07.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:07.747291+0000) 2022-01-31T21:26:08.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:08 smithi167 conmon[54076]: debug 2022-01-31T21:26:08.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:08.413398+0000) 2022-01-31T21:26:08.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:08 smithi167 conmon[60316]: debug 2022-01-31T21:26:08.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:08.416846+0000) 2022-01-31T21:26:08.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:08 smithi171 conmon[46715]: debug 2022-01-31T21:26:08.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:08.190162+0000) 2022-01-31T21:26:08.852 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:08 smithi171 conmon[51620]: debug 2022-01-31T21:26:08.770+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:08.771293+0000) 2022-01-31T21:26:08.853 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:08 smithi171 conmon[35325]: debug 2022-01-31T21:26:08.678+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:26:09.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:08 smithi171 conmon[41853]: debug 2022-01-31T21:26:08.893+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:08.894502+0000) 2022-01-31T21:26:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:08 smithi167 conmon[49112]: debug 2022-01-31T21:26:08.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:08.747493+0000) 2022-01-31T21:26:09.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:09 smithi167 conmon[54076]: debug 2022-01-31T21:26:09.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.413596+0000) 2022-01-31T21:26:09.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:09 smithi167 conmon[60316]: debug 2022-01-31T21:26:09.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.416943+0000) 2022-01-31T21:26:09.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[46715]: debug 2022-01-31T21:26:09.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.190375+0000) 2022-01-31T21:26:09.732 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:09 smithi167 conmon[49112]: debug 2022-01-31T21:26:09.697+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.698176+0000) 2022-01-31T21:26:09.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:09 smithi167 conmon[60316]: debug 2022-01-31T21:26:09.698+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.699176+0000) 2022-01-31T21:26:09.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:09 smithi167 conmon[54076]: debug 2022-01-31T21:26:09.697+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.698429+0000) 2022-01-31T21:26:09.853 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[35325]: debug 2022-01-31T21:26:09.711+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129526 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:09.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[41853]: debug 2022-01-31T21:26:09.696+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.697577+0000) 2022-01-31T21:26:09.854 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[46715]: debug 2022-01-31T21:26:09.697+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.698105+0000) 2022-01-31T21:26:09.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[51620]: debug 2022-01-31T21:26:09.696+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.697428+0000) 2022-01-31T21:26:09.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[51620]: debug 2022-01-31T21:26:09.770+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.771502+0000) 2022-01-31T21:26:10.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:09 smithi171 conmon[41853]: debug 2022-01-31T21:26:09.894+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.894690+0000) 2022-01-31T21:26:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:09 smithi167 conmon[49112]: debug 2022-01-31T21:26:09.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:09.747655+0000) 2022-01-31T21:26:10.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:10 smithi167 conmon[54076]: debug 2022-01-31T21:26:10.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:10.413781+0000) 2022-01-31T21:26:10.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:10 smithi167 conmon[60316]: debug 2022-01-31T21:26:10.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:10.417093+0000) 2022-01-31T21:26:10.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:10 smithi171 conmon[46715]: debug 2022-01-31T21:26:10.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:10.190566+0000) 2022-01-31T21:26:10.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:10 smithi171 conmon[51620]: debug 2022-01-31T21:26:10.771+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:10.771687+0000) 2022-01-31T21:26:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:10 smithi171 conmon[41853]: debug 2022-01-31T21:26:10.894+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:10.894826+0000) 2022-01-31T21:26:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:10 smithi167 conmon[49112]: debug 2022-01-31T21:26:10.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:10.747777+0000) 2022-01-31T21:26:11.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:11 smithi167 conmon[54076]: debug 2022-01-31T21:26:11.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:11.413932+0000) 2022-01-31T21:26:11.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:11 smithi167 conmon[60316]: debug 2022-01-31T21:26:11.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:11.417236+0000) 2022-01-31T21:26:11.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:11 smithi171 conmon[46715]: debug 2022-01-31T21:26:11.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:11.190699+0000) 2022-01-31T21:26:11.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:11 smithi171 conmon[51620]: debug 2022-01-31T21:26:11.771+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:11.771839+0000) 2022-01-31T21:26:12.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:11 smithi171 conmon[41853]: debug 2022-01-31T21:26:11.894+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:11.895016+0000) 2022-01-31T21:26:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:11 smithi167 conmon[49112]: debug 2022-01-31T21:26:11.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:11.747933+0000) 2022-01-31T21:26:12.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:12 smithi167 conmon[54076]: debug 2022-01-31T21:26:12.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:12.414132+0000) 2022-01-31T21:26:12.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:12 smithi167 conmon[60316]: debug 2022-01-31T21:26:12.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:12.417416+0000) 2022-01-31T21:26:12.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:12 smithi171 conmon[46715]: debug 2022-01-31T21:26:12.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:12.190899+0000) 2022-01-31T21:26:12.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:12 smithi171 conmon[51620]: debug 2022-01-31T21:26:12.771+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:12.772011+0000) 2022-01-31T21:26:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:12 smithi171 conmon[41853]: debug 2022-01-31T21:26:12.894+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:12.895228+0000) 2022-01-31T21:26:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:12 smithi167 conmon[49112]: debug 2022-01-31T21:26:12.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:12.748101+0000) 2022-01-31T21:26:13.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:13 smithi167 conmon[54076]: debug 2022-01-31T21:26:13.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:13.414342+0000) 2022-01-31T21:26:13.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:13 smithi167 conmon[60316]: debug 2022-01-31T21:26:13.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:13.417611+0000) 2022-01-31T21:26:13.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:13 smithi171 conmon[46715]: debug 2022-01-31T21:26:13.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:13.191089+0000) 2022-01-31T21:26:13.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:13 smithi171 conmon[51620]: debug 2022-01-31T21:26:13.771+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:13.772165+0000) 2022-01-31T21:26:14.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:13 smithi171 conmon[41853]: debug 2022-01-31T21:26:13.894+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:13.895351+0000) 2022-01-31T21:26:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:13 smithi167 conmon[49112]: debug 2022-01-31T21:26:13.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:13.748332+0000) 2022-01-31T21:26:14.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:14 smithi167 conmon[54076]: debug 2022-01-31T21:26:14.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.414559+0000) 2022-01-31T21:26:14.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:14 smithi167 conmon[60316]: debug 2022-01-31T21:26:14.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.417739+0000) 2022-01-31T21:26:14.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[46715]: debug 2022-01-31T21:26:14.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.191283+0000) 2022-01-31T21:26:14.734 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:14 smithi167 conmon[49112]: debug 2022-01-31T21:26:14.714+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.714783+0000) 2022-01-31T21:26:14.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:14 smithi167 conmon[54076]: debug 2022-01-31T21:26:14.713+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.714184+0000) 2022-01-31T21:26:14.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:14 smithi167 conmon[60316]: debug 2022-01-31T21:26:14.715+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.716041+0000) 2022-01-31T21:26:14.853 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[35325]: debug 2022-01-31T21:26:14.726+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129636 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:14.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[41853]: debug 2022-01-31T21:26:14.714+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.715512+0000) 2022-01-31T21:26:14.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[46715]: debug 2022-01-31T21:26:14.713+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.714019+0000) 2022-01-31T21:26:14.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[51620]: debug 2022-01-31T21:26:14.713+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.713879+0000) 2022-01-31T21:26:14.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[51620]: debug 2022-01-31T21:26:14.771+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.772374+0000) 2022-01-31T21:26:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:14 smithi171 conmon[41853]: debug 2022-01-31T21:26:14.895+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.895546+0000) 2022-01-31T21:26:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:14 smithi167 conmon[49112]: debug 2022-01-31T21:26:14.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:14.748569+0000) 2022-01-31T21:26:15.468 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:15 smithi167 conmon[60316]: debug 2022-01-31T21:26:15.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:15.417857+0000) 2022-01-31T21:26:15.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:15 smithi167 conmon[54076]: debug 2022-01-31T21:26:15.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:15.414722+0000) 2022-01-31T21:26:15.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:15 smithi171 conmon[46715]: debug 2022-01-31T21:26:15.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:15.191455+0000) 2022-01-31T21:26:15.853 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:15 smithi171 conmon[51620]: debug 2022-01-31T21:26:15.772+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:15.772530+0000) 2022-01-31T21:26:16.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:15 smithi171 conmon[41853]: debug 2022-01-31T21:26:15.895+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:15.895715+0000) 2022-01-31T21:26:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:15 smithi167 conmon[49112]: debug 2022-01-31T21:26:15.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:15.748735+0000) 2022-01-31T21:26:16.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:16 smithi167 conmon[54076]: debug 2022-01-31T21:26:16.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:16.414841+0000) 2022-01-31T21:26:16.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:16 smithi167 conmon[60316]: debug 2022-01-31T21:26:16.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:16.418029+0000) 2022-01-31T21:26:16.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:16 smithi171 conmon[46715]: debug 2022-01-31T21:26:16.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:16.191624+0000) 2022-01-31T21:26:16.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:16 smithi171 conmon[51620]: debug 2022-01-31T21:26:16.772+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:16.772733+0000) 2022-01-31T21:26:17.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:16 smithi171 conmon[41853]: debug 2022-01-31T21:26:16.895+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:16.895891+0000) 2022-01-31T21:26:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:16 smithi167 conmon[49112]: debug 2022-01-31T21:26:16.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:16.748919+0000) 2022-01-31T21:26:17.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:17 smithi167 conmon[54076]: debug 2022-01-31T21:26:17.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:17.415026+0000) 2022-01-31T21:26:17.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:17 smithi167 conmon[60316]: debug 2022-01-31T21:26:17.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:17.418204+0000) 2022-01-31T21:26:17.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:17 smithi171 conmon[46715]: debug 2022-01-31T21:26:17.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:17.191805+0000) 2022-01-31T21:26:17.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:17 smithi171 conmon[51620]: debug 2022-01-31T21:26:17.772+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:17.772920+0000) 2022-01-31T21:26:18.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:17 smithi171 conmon[41853]: debug 2022-01-31T21:26:17.895+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:17.896071+0000) 2022-01-31T21:26:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:17 smithi167 conmon[49112]: debug 2022-01-31T21:26:17.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:17.749120+0000) 2022-01-31T21:26:18.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:18 smithi167 conmon[54076]: debug 2022-01-31T21:26:18.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:18.415213+0000) 2022-01-31T21:26:18.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:18 smithi167 conmon[60316]: debug 2022-01-31T21:26:18.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:18.418385+0000) 2022-01-31T21:26:18.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:18 smithi171 conmon[46715]: debug 2022-01-31T21:26:18.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:18.191960+0000) 2022-01-31T21:26:18.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:18 smithi171 conmon[51620]: debug 2022-01-31T21:26:18.772+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:18.773092+0000) 2022-01-31T21:26:19.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:18 smithi171 conmon[41853]: debug 2022-01-31T21:26:18.895+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:18.896253+0000) 2022-01-31T21:26:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:18 smithi167 conmon[49112]: debug 2022-01-31T21:26:18.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:18.749340+0000) 2022-01-31T21:26:19.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:19 smithi167 conmon[54076]: debug 2022-01-31T21:26:19.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.415436+0000) 2022-01-31T21:26:19.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:19 smithi167 conmon[60316]: debug 2022-01-31T21:26:19.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.418573+0000) 2022-01-31T21:26:19.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[46715]: debug 2022-01-31T21:26:19.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.192159+0000) 2022-01-31T21:26:19.854 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[35325]: debug 2022-01-31T21:26:19.743+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129746 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:19.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[46715]: debug 2022-01-31T21:26:19.728+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.729660+0000) 2022-01-31T21:26:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[41853]: debug 2022-01-31T21:26:19.729+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.730679+0000) 2022-01-31T21:26:19.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[51620]: debug 2022-01-31T21:26:19.729+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.730303+0000) 2022-01-31T21:26:19.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[51620]: debug 2022-01-31T21:26:19.772+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.773272+0000) 2022-01-31T21:26:20.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:19 smithi171 conmon[41853]: debug 2022-01-31T21:26:19.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.896452+0000) 2022-01-31T21:26:20.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:19 smithi167 conmon[60316]: debug 2022-01-31T21:26:19.728+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.729580+0000) 2022-01-31T21:26:20.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:19 smithi167 conmon[54076]: debug 2022-01-31T21:26:19.728+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.730105+0000) 2022-01-31T21:26:20.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:19 smithi167 conmon[49112]: debug 2022-01-31T21:26:19.728+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.730155+0000) 2022-01-31T21:26:20.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:19 smithi167 conmon[49112]: debug 2022-01-31T21:26:19.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:19.749492+0000) 2022-01-31T21:26:20.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:20 smithi167 conmon[60316]: debug 2022-01-31T21:26:20.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:20.418693+0000) 2022-01-31T21:26:20.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:20 smithi167 conmon[54076]: debug 2022-01-31T21:26:20.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:20.415602+0000) 2022-01-31T21:26:20.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:20 smithi171 conmon[46715]: debug 2022-01-31T21:26:20.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:20.192343+0000) 2022-01-31T21:26:20.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:20 smithi171 conmon[51620]: debug 2022-01-31T21:26:20.773+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:20.773441+0000) 2022-01-31T21:26:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:20 smithi171 conmon[41853]: debug 2022-01-31T21:26:20.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:20.896598+0000) 2022-01-31T21:26:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:20 smithi167 conmon[49112]: debug 2022-01-31T21:26:20.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:20.749643+0000) 2022-01-31T21:26:21.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:21 smithi167 conmon[60316]: debug 2022-01-31T21:26:21.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:21.418819+0000) 2022-01-31T21:26:21.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:21 smithi167 conmon[54076]: debug 2022-01-31T21:26:21.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:21.415744+0000) 2022-01-31T21:26:21.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:21 smithi171 conmon[46715]: debug 2022-01-31T21:26:21.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:21.192490+0000) 2022-01-31T21:26:21.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:21 smithi171 conmon[51620]: debug 2022-01-31T21:26:21.772+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:21.773675+0000) 2022-01-31T21:26:22.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:21 smithi171 conmon[41853]: debug 2022-01-31T21:26:21.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:21.896807+0000) 2022-01-31T21:26:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:21 smithi167 conmon[49112]: debug 2022-01-31T21:26:21.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:21.749787+0000) 2022-01-31T21:26:22.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:22 smithi167 conmon[60316]: debug 2022-01-31T21:26:22.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:22.418977+0000) 2022-01-31T21:26:22.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:22 smithi167 conmon[54076]: debug 2022-01-31T21:26:22.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:22.415949+0000) 2022-01-31T21:26:22.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:22 smithi171 conmon[46715]: debug 2022-01-31T21:26:22.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:22.192665+0000) 2022-01-31T21:26:22.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:22 smithi171 conmon[51620]: debug 2022-01-31T21:26:22.773+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:22.773905+0000) 2022-01-31T21:26:23.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:22 smithi171 conmon[41853]: debug 2022-01-31T21:26:22.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:22.896980+0000) 2022-01-31T21:26:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:22 smithi167 conmon[49112]: debug 2022-01-31T21:26:22.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:22.749956+0000) 2022-01-31T21:26:23.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:23 smithi167 conmon[60316]: debug 2022-01-31T21:26:23.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:23.419187+0000) 2022-01-31T21:26:23.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:23 smithi167 conmon[54076]: debug 2022-01-31T21:26:23.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:23.416115+0000) 2022-01-31T21:26:23.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:23 smithi171 conmon[46715]: debug 2022-01-31T21:26:23.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:23.192865+0000) 2022-01-31T21:26:23.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:23 smithi171 conmon[35325]: debug 2022-01-31T21:26:23.679+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:26:23.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:23 smithi171 conmon[51620]: debug 2022-01-31T21:26:23.773+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:23.774122+0000) 2022-01-31T21:26:24.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:23 smithi171 conmon[41853]: debug 2022-01-31T21:26:23.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:23.897186+0000) 2022-01-31T21:26:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:23 smithi167 conmon[49112]: debug 2022-01-31T21:26:23.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:23.750184+0000) 2022-01-31T21:26:24.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:24 smithi167 conmon[54076]: debug 2022-01-31T21:26:24.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.416368+0000) 2022-01-31T21:26:24.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:24 smithi167 conmon[60316]: debug 2022-01-31T21:26:24.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.419395+0000) 2022-01-31T21:26:24.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[46715]: debug 2022-01-31T21:26:24.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.193020+0000) 2022-01-31T21:26:24.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[35325]: debug 2022-01-31T21:26:24.759+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129856 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:24.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[46715]: debug 2022-01-31T21:26:24.746+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.747578+0000) 2022-01-31T21:26:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[41853]: debug 2022-01-31T21:26:24.745+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.746535+0000) 2022-01-31T21:26:24.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[51620]: debug 2022-01-31T21:26:24.745+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.745852+0000) 2022-01-31T21:26:24.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[51620]: debug 2022-01-31T21:26:24.773+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.774315+0000) 2022-01-31T21:26:25.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:24 smithi171 conmon[41853]: debug 2022-01-31T21:26:24.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.897385+0000) 2022-01-31T21:26:25.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:24 smithi167 conmon[54076]: debug 2022-01-31T21:26:24.745+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.747204+0000) 2022-01-31T21:26:25.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:24 smithi167 conmon[60316]: debug 2022-01-31T21:26:24.745+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.746872+0000) 2022-01-31T21:26:25.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:24 smithi167 conmon[49112]: debug 2022-01-31T21:26:24.745+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.747345+0000) 2022-01-31T21:26:25.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:24 smithi167 conmon[49112]: debug 2022-01-31T21:26:24.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:24.750350+0000) 2022-01-31T21:26:25.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:25 smithi167 conmon[54076]: debug 2022-01-31T21:26:25.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:25.416532+0000) 2022-01-31T21:26:25.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:25 smithi167 conmon[60316]: debug 2022-01-31T21:26:25.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:25.419553+0000) 2022-01-31T21:26:25.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:25 smithi171 conmon[46715]: debug 2022-01-31T21:26:25.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:25.193161+0000) 2022-01-31T21:26:25.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:25 smithi171 conmon[51620]: debug 2022-01-31T21:26:25.773+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:25.774460+0000) 2022-01-31T21:26:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:25 smithi171 conmon[41853]: debug 2022-01-31T21:26:25.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:25.897537+0000) 2022-01-31T21:26:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:25 smithi167 conmon[49112]: debug 2022-01-31T21:26:25.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:25.750494+0000) 2022-01-31T21:26:26.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:26 smithi167 conmon[54076]: debug 2022-01-31T21:26:26.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:26.416680+0000) 2022-01-31T21:26:26.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:26 smithi167 conmon[60316]: debug 2022-01-31T21:26:26.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:26.419697+0000) 2022-01-31T21:26:26.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:26 smithi171 conmon[46715]: debug 2022-01-31T21:26:26.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:26.193304+0000) 2022-01-31T21:26:26.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:26 smithi171 conmon[51620]: debug 2022-01-31T21:26:26.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:26.774653+0000) 2022-01-31T21:26:27.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:26 smithi171 conmon[41853]: debug 2022-01-31T21:26:26.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:26.897719+0000) 2022-01-31T21:26:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:26 smithi167 conmon[49112]: debug 2022-01-31T21:26:26.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:26.750711+0000) 2022-01-31T21:26:27.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:27 smithi167 conmon[54076]: debug 2022-01-31T21:26:27.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:27.416829+0000) 2022-01-31T21:26:27.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:27 smithi167 conmon[60316]: debug 2022-01-31T21:26:27.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:27.419820+0000) 2022-01-31T21:26:27.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:27 smithi171 conmon[46715]: debug 2022-01-31T21:26:27.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:27.193443+0000) 2022-01-31T21:26:27.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:27 smithi171 conmon[51620]: debug 2022-01-31T21:26:27.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:27.774854+0000) 2022-01-31T21:26:28.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:27 smithi171 conmon[41853]: debug 2022-01-31T21:26:27.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:27.897890+0000) 2022-01-31T21:26:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:27 smithi167 conmon[49112]: debug 2022-01-31T21:26:27.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:27.750870+0000) 2022-01-31T21:26:28.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:28 smithi167 conmon[54076]: debug 2022-01-31T21:26:28.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:28.417016+0000) 2022-01-31T21:26:28.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:28 smithi167 conmon[60316]: debug 2022-01-31T21:26:28.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:28.420017+0000) 2022-01-31T21:26:28.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:28 smithi171 conmon[46715]: debug 2022-01-31T21:26:28.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:28.193644+0000) 2022-01-31T21:26:28.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:28 smithi171 conmon[51620]: debug 2022-01-31T21:26:28.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:28.775058+0000) 2022-01-31T21:26:29.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:28 smithi171 conmon[41853]: debug 2022-01-31T21:26:28.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:28.898045+0000) 2022-01-31T21:26:29.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:28 smithi167 conmon[49112]: debug 2022-01-31T21:26:28.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:28.751114+0000) 2022-01-31T21:26:29.471 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:29 smithi167 conmon[54076]: debug 2022-01-31T21:26:29.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.417192+0000) 2022-01-31T21:26:29.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:29 smithi167 conmon[60316]: debug 2022-01-31T21:26:29.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.420175+0000) 2022-01-31T21:26:29.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[46715]: debug 2022-01-31T21:26:29.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.193819+0000) 2022-01-31T21:26:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[41853]: debug 2022-01-31T21:26:29.763+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.763933+0000) 2022-01-31T21:26:29.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[46715]: debug 2022-01-31T21:26:29.762+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.762962+0000) 2022-01-31T21:26:29.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[35325]: debug 2022-01-31T21:26:29.776+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 129966 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:29.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[51620]: debug 2022-01-31T21:26:29.761+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.762047+0000) 2022-01-31T21:26:29.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[51620]: debug 2022-01-31T21:26:29.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.775246+0000) 2022-01-31T21:26:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:29 smithi167 conmon[49112]: debug 2022-01-31T21:26:29.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.751317+0000) 2022-01-31T21:26:30.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:29 smithi167 conmon[49112]: debug 2022-01-31T21:26:29.761+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.762987+0000) 2022-01-31T21:26:30.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:29 smithi167 conmon[54076]: debug 2022-01-31T21:26:29.761+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.762680+0000) 2022-01-31T21:26:30.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:29 smithi167 conmon[60316]: debug 2022-01-31T21:26:29.761+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.762492+0000) 2022-01-31T21:26:30.192 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:29 smithi171 conmon[41853]: debug 2022-01-31T21:26:29.898+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:29.898179+0000) 2022-01-31T21:26:30.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:30 smithi167 conmon[54076]: debug 2022-01-31T21:26:30.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:30.417401+0000) 2022-01-31T21:26:30.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:30 smithi167 conmon[60316]: debug 2022-01-31T21:26:30.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:30.420382+0000) 2022-01-31T21:26:30.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:30 smithi171 conmon[46715]: debug 2022-01-31T21:26:30.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:30.193980+0000) 2022-01-31T21:26:30.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:30 smithi171 conmon[51620]: debug 2022-01-31T21:26:30.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:30.775441+0000) 2022-01-31T21:26:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:30 smithi167 conmon[49112]: debug 2022-01-31T21:26:30.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:30.751549+0000) 2022-01-31T21:26:31.297 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:30 smithi171 conmon[41853]: debug 2022-01-31T21:26:30.896+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:30.898280+0000) 2022-01-31T21:26:31.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:31 smithi167 conmon[54076]: debug 2022-01-31T21:26:31.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:31.417571+0000) 2022-01-31T21:26:31.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:31 smithi167 conmon[60316]: debug 2022-01-31T21:26:31.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:31.420554+0000) 2022-01-31T21:26:31.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:31 smithi171 conmon[46715]: debug 2022-01-31T21:26:31.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:31.194116+0000) 2022-01-31T21:26:31.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:31 smithi171 conmon[51620]: debug 2022-01-31T21:26:31.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:31.775644+0000) 2022-01-31T21:26:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:31 smithi167 conmon[49112]: debug 2022-01-31T21:26:31.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:31.751695+0000) 2022-01-31T21:26:32.193 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:31 smithi171 conmon[41853]: debug 2022-01-31T21:26:31.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:31.898407+0000) 2022-01-31T21:26:32.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:32 smithi167 conmon[54076]: debug 2022-01-31T21:26:32.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:32.417761+0000) 2022-01-31T21:26:32.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:32 smithi167 conmon[60316]: debug 2022-01-31T21:26:32.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:32.420683+0000) 2022-01-31T21:26:32.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:32 smithi171 conmon[46715]: debug 2022-01-31T21:26:32.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:32.194325+0000) 2022-01-31T21:26:32.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:32 smithi171 conmon[51620]: debug 2022-01-31T21:26:32.774+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:32.775841+0000) 2022-01-31T21:26:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:32 smithi167 conmon[49112]: debug 2022-01-31T21:26:32.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:32.751873+0000) 2022-01-31T21:26:33.193 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:32 smithi171 conmon[41853]: debug 2022-01-31T21:26:32.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:32.898539+0000) 2022-01-31T21:26:33.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:33 smithi167 conmon[54076]: debug 2022-01-31T21:26:33.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:33.417887+0000) 2022-01-31T21:26:33.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:33 smithi167 conmon[60316]: debug 2022-01-31T21:26:33.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:33.420849+0000) 2022-01-31T21:26:33.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:33 smithi171 conmon[46715]: debug 2022-01-31T21:26:33.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:33.194515+0000) 2022-01-31T21:26:33.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:33 smithi171 conmon[51620]: debug 2022-01-31T21:26:33.775+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:33.776024+0000) 2022-01-31T21:26:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:33 smithi167 conmon[49112]: debug 2022-01-31T21:26:33.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:33.751987+0000) 2022-01-31T21:26:34.193 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:33 smithi171 conmon[41853]: debug 2022-01-31T21:26:33.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:33.898688+0000) 2022-01-31T21:26:34.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:34 smithi167 conmon[54076]: debug 2022-01-31T21:26:34.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.418063+0000) 2022-01-31T21:26:34.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:34 smithi167 conmon[60316]: debug 2022-01-31T21:26:34.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.421059+0000) 2022-01-31T21:26:34.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[46715]: debug 2022-01-31T21:26:34.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.194709+0000) 2022-01-31T21:26:34.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[35325]: debug 2022-01-31T21:26:34.791+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130077 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[41853]: debug 2022-01-31T21:26:34.778+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.780601+0000) 2022-01-31T21:26:34.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[46715]: debug 2022-01-31T21:26:34.777+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.779077+0000) 2022-01-31T21:26:34.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[51620]: debug 2022-01-31T21:26:34.775+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.776206+0000) 2022-01-31T21:26:34.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[51620]: debug 2022-01-31T21:26:34.777+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.779314+0000) 2022-01-31T21:26:35.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:34 smithi167 conmon[54076]: debug 2022-01-31T21:26:34.778+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.779761+0000) 2022-01-31T21:26:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:34 smithi167 conmon[49112]: debug 2022-01-31T21:26:34.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.752176+0000) 2022-01-31T21:26:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:34 smithi167 conmon[49112]: debug 2022-01-31T21:26:34.778+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.780376+0000) 2022-01-31T21:26:35.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:34 smithi167 conmon[60316]: debug 2022-01-31T21:26:34.779+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.780686+0000) 2022-01-31T21:26:35.193 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:34 smithi171 conmon[41853]: debug 2022-01-31T21:26:34.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:34.898902+0000) 2022-01-31T21:26:35.472 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:35 smithi167 conmon[60316]: debug 2022-01-31T21:26:35.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:35.421259+0000) 2022-01-31T21:26:35.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:35 smithi167 conmon[54076]: debug 2022-01-31T21:26:35.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:35.418261+0000) 2022-01-31T21:26:35.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:35 smithi171 conmon[46715]: debug 2022-01-31T21:26:35.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:35.194928+0000) 2022-01-31T21:26:35.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:35 smithi171 conmon[51620]: debug 2022-01-31T21:26:35.775+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:35.776399+0000) 2022-01-31T21:26:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:35 smithi167 conmon[49112]: debug 2022-01-31T21:26:35.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:35.752311+0000) 2022-01-31T21:26:36.193 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:35 smithi171 conmon[41853]: debug 2022-01-31T21:26:35.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:35.898997+0000) 2022-01-31T21:26:36.472 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:36 smithi167 conmon[54076]: debug 2022-01-31T21:26:36.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:36.418444+0000) 2022-01-31T21:26:36.473 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:36 smithi167 conmon[60316]: debug 2022-01-31T21:26:36.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:36.421434+0000) 2022-01-31T21:26:36.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:36 smithi171 conmon[46715]: debug 2022-01-31T21:26:36.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:36.195082+0000) 2022-01-31T21:26:36.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:36 smithi171 conmon[51620]: debug 2022-01-31T21:26:36.775+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:36.776536+0000) 2022-01-31T21:26:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:36 smithi167 conmon[49112]: debug 2022-01-31T21:26:36.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:36.752432+0000) 2022-01-31T21:26:37.193 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:36 smithi171 conmon[41853]: debug 2022-01-31T21:26:36.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:36.899121+0000) 2022-01-31T21:26:37.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:37 smithi167 conmon[54076]: debug 2022-01-31T21:26:37.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:37.418646+0000) 2022-01-31T21:26:37.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:37 smithi167 conmon[60316]: debug 2022-01-31T21:26:37.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:37.421626+0000) 2022-01-31T21:26:37.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:37 smithi171 conmon[46715]: debug 2022-01-31T21:26:37.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:37.195276+0000) 2022-01-31T21:26:37.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:37 smithi171 conmon[51620]: debug 2022-01-31T21:26:37.775+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:37.776702+0000) 2022-01-31T21:26:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:37 smithi167 conmon[49112]: debug 2022-01-31T21:26:37.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:37.752576+0000) 2022-01-31T21:26:38.194 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:37 smithi171 conmon[41853]: debug 2022-01-31T21:26:37.898+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:37.899305+0000) 2022-01-31T21:26:38.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:38 smithi167 conmon[54076]: debug 2022-01-31T21:26:38.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:38.418837+0000) 2022-01-31T21:26:38.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:38 smithi167 conmon[60316]: debug 2022-01-31T21:26:38.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:38.421776+0000) 2022-01-31T21:26:38.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:38 smithi171 conmon[46715]: debug 2022-01-31T21:26:38.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:38.195452+0000) 2022-01-31T21:26:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:38 smithi171 conmon[35325]: debug 2022-01-31T21:26:38.679+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:26:38.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:38 smithi171 conmon[51620]: debug 2022-01-31T21:26:38.776+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:38.776862+0000) 2022-01-31T21:26:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:38 smithi167 conmon[49112]: debug 2022-01-31T21:26:38.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:38.752694+0000) 2022-01-31T21:26:39.194 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:38 smithi171 conmon[41853]: debug 2022-01-31T21:26:38.898+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:38.899475+0000) 2022-01-31T21:26:39.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[54076]: debug 2022-01-31T21:26:39.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.419044+0000) 2022-01-31T21:26:39.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[60316]: debug 2022-01-31T21:26:39.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.421961+0000) 2022-01-31T21:26:39.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[46715]: debug 2022-01-31T21:26:39.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.195662+0000) 2022-01-31T21:26:39.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[35325]: debug 2022-01-31T21:26:39.807+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130189 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[41853]: debug 2022-01-31T21:26:39.794+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.796611+0000) 2022-01-31T21:26:39.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[46715]: debug 2022-01-31T21:26:39.793+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.795100+0000) 2022-01-31T21:26:39.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[51620]: debug 2022-01-31T21:26:39.775+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.776980+0000) 2022-01-31T21:26:39.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[51620]: debug 2022-01-31T21:26:39.794+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.796525+0000) 2022-01-31T21:26:40.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[54076]: debug 2022-01-31T21:26:39.794+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.796072+0000) 2022-01-31T21:26:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[49112]: debug 2022-01-31T21:26:39.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.752853+0000) 2022-01-31T21:26:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[49112]: debug 2022-01-31T21:26:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[49112]: 2022-01-31T21:26:39.794+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.795638+0000) 2022-01-31T21:26:40.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:39 smithi167 conmon[60316]: debug 2022-01-31T21:26:39.796+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.797505+0000) 2022-01-31T21:26:40.194 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:39 smithi171 conmon[41853]: debug 2022-01-31T21:26:39.897+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:39.899645+0000) 2022-01-31T21:26:40.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:40 smithi167 conmon[54076]: debug 2022-01-31T21:26:40.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:40.419232+0000) 2022-01-31T21:26:40.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:40 smithi167 conmon[60316]: debug 2022-01-31T21:26:40.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:40.422209+0000) 2022-01-31T21:26:40.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:40 smithi171 conmon[46715]: debug 2022-01-31T21:26:40.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:40.195856+0000) 2022-01-31T21:26:40.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:40 smithi171 conmon[51620]: debug 2022-01-31T21:26:40.776+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:40.777162+0000) 2022-01-31T21:26:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:40 smithi167 conmon[49112]: debug 2022-01-31T21:26:40.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:40.753053+0000) 2022-01-31T21:26:41.194 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:40 smithi171 conmon[41853]: debug 2022-01-31T21:26:40.898+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:40.899800+0000) 2022-01-31T21:26:41.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:41 smithi167 conmon[54076]: debug 2022-01-31T21:26:41.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:41.419402+0000) 2022-01-31T21:26:41.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:41 smithi167 conmon[60316]: debug 2022-01-31T21:26:41.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:41.422374+0000) 2022-01-31T21:26:41.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:41 smithi171 conmon[46715]: debug 2022-01-31T21:26:41.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:41.195972+0000) 2022-01-31T21:26:41.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:41 smithi171 conmon[51620]: debug 2022-01-31T21:26:41.776+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:41.777331+0000) 2022-01-31T21:26:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:41 smithi167 conmon[49112]: debug 2022-01-31T21:26:41.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:41.753169+0000) 2022-01-31T21:26:42.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:41 smithi171 conmon[41853]: debug 2022-01-31T21:26:41.899+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:41.899975+0000) 2022-01-31T21:26:42.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:42 smithi167 conmon[54076]: debug 2022-01-31T21:26:42.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:42.419570+0000) 2022-01-31T21:26:42.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:42 smithi167 conmon[60316]: debug 2022-01-31T21:26:42.421+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:42.422549+0000) 2022-01-31T21:26:42.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:42 smithi171 conmon[46715]: debug 2022-01-31T21:26:42.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:42.196164+0000) 2022-01-31T21:26:42.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:42 smithi171 conmon[51620]: debug 2022-01-31T21:26:42.776+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:42.777555+0000) 2022-01-31T21:26:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:42 smithi167 conmon[49112]: debug 2022-01-31T21:26:42.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:42.753383+0000) 2022-01-31T21:26:43.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:42 smithi171 conmon[41853]: debug 2022-01-31T21:26:42.899+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:42.900155+0000) 2022-01-31T21:26:43.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:43 smithi167 conmon[54076]: debug 2022-01-31T21:26:43.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:43.419769+0000) 2022-01-31T21:26:43.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:43 smithi167 conmon[60316]: debug 2022-01-31T21:26:43.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:43.422703+0000) 2022-01-31T21:26:43.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:43 smithi171 conmon[46715]: debug 2022-01-31T21:26:43.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:43.196351+0000) 2022-01-31T21:26:43.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:43 smithi171 conmon[51620]: debug 2022-01-31T21:26:43.777+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:43.777801+0000) 2022-01-31T21:26:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:43 smithi167 conmon[49112]: debug 2022-01-31T21:26:43.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:43.753622+0000) 2022-01-31T21:26:44.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:43 smithi171 conmon[41853]: debug 2022-01-31T21:26:43.899+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:43.900310+0000) 2022-01-31T21:26:44.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:44 smithi167 conmon[54076]: debug 2022-01-31T21:26:44.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.419971+0000) 2022-01-31T21:26:44.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:44 smithi167 conmon[60316]: debug 2022-01-31T21:26:44.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.422901+0000) 2022-01-31T21:26:44.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[46715]: debug 2022-01-31T21:26:44.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.196567+0000) 2022-01-31T21:26:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[41853]: debug 2022-01-31T21:26:44.811+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.812260+0000) 2022-01-31T21:26:44.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[46715]: debug 2022-01-31T21:26:44.810+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.811481+0000) 2022-01-31T21:26:44.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[35325]: debug 2022-01-31T21:26:44.824+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130300 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:44.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[51620]: debug 2022-01-31T21:26:44.777+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.777982+0000) 2022-01-31T21:26:44.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[51620]: debug 2022-01-31T21:26:44.811+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.811988+0000) 2022-01-31T21:26:45.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:44 smithi167 conmon[60316]: debug 2022-01-31T21:26:44.812+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.812675+0000) 2022-01-31T21:26:45.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:44 smithi167 conmon[49112]: debug 2022-01-31T21:26:44.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.753765+0000) 2022-01-31T21:26:45.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:44 smithi167 conmon[49112]: debug 2022-01-31T21:26:44.811+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.811797+0000) 2022-01-31T21:26:45.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:44 smithi167 conmon[54076]: debug 2022-01-31T21:26:44.812+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.812671+0000) 2022-01-31T21:26:45.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:44 smithi171 conmon[41853]: debug 2022-01-31T21:26:44.899+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:44.900493+0000) 2022-01-31T21:26:45.474 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:45 smithi167 conmon[60316]: debug 2022-01-31T21:26:45.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:45.423084+0000) 2022-01-31T21:26:45.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:45 smithi167 conmon[54076]: debug 2022-01-31T21:26:45.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:45.420186+0000) 2022-01-31T21:26:45.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:45 smithi171 conmon[46715]: debug 2022-01-31T21:26:45.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:45.196775+0000) 2022-01-31T21:26:45.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:45 smithi171 conmon[51620]: debug 2022-01-31T21:26:45.777+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:45.778159+0000) 2022-01-31T21:26:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:45 smithi167 conmon[49112]: debug 2022-01-31T21:26:45.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:45.753964+0000) 2022-01-31T21:26:46.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:45 smithi171 conmon[41853]: debug 2022-01-31T21:26:45.899+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:45.900631+0000) 2022-01-31T21:26:46.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:46 smithi167 conmon[54076]: debug 2022-01-31T21:26:46.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:46.420345+0000) 2022-01-31T21:26:46.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:46 smithi167 conmon[60316]: debug 2022-01-31T21:26:46.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:46.423275+0000) 2022-01-31T21:26:46.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:46 smithi171 conmon[46715]: debug 2022-01-31T21:26:46.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:46.196943+0000) 2022-01-31T21:26:46.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:46 smithi171 conmon[51620]: debug 2022-01-31T21:26:46.777+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:46.778323+0000) 2022-01-31T21:26:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:46 smithi167 conmon[49112]: debug 2022-01-31T21:26:46.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:46.754176+0000) 2022-01-31T21:26:47.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:46 smithi171 conmon[41853]: debug 2022-01-31T21:26:46.900+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:46.900822+0000) 2022-01-31T21:26:47.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:47 smithi167 conmon[54076]: debug 2022-01-31T21:26:47.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:47.420529+0000) 2022-01-31T21:26:47.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:47 smithi167 conmon[60316]: debug 2022-01-31T21:26:47.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:47.423481+0000) 2022-01-31T21:26:47.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:47 smithi171 conmon[46715]: debug 2022-01-31T21:26:47.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:47.197063+0000) 2022-01-31T21:26:47.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:47 smithi171 conmon[51620]: debug 2022-01-31T21:26:47.777+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:47.778506+0000) 2022-01-31T21:26:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:47 smithi167 conmon[49112]: debug 2022-01-31T21:26:47.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:47.754361+0000) 2022-01-31T21:26:48.195 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:47 smithi171 conmon[41853]: debug 2022-01-31T21:26:47.900+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:47.901006+0000) 2022-01-31T21:26:48.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:48 smithi167 conmon[54076]: debug 2022-01-31T21:26:48.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:48.420692+0000) 2022-01-31T21:26:48.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:48 smithi167 conmon[60316]: debug 2022-01-31T21:26:48.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:48.423652+0000) 2022-01-31T21:26:48.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:48 smithi171 conmon[46715]: debug 2022-01-31T21:26:48.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:48.197264+0000) 2022-01-31T21:26:48.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:48 smithi171 conmon[51620]: debug 2022-01-31T21:26:48.777+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:48.778709+0000) 2022-01-31T21:26:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:48 smithi167 conmon[49112]: debug 2022-01-31T21:26:48.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:48.754467+0000) 2022-01-31T21:26:49.196 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:48 smithi171 conmon[41853]: debug 2022-01-31T21:26:48.900+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:48.901183+0000) 2022-01-31T21:26:49.475 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:49 smithi167 conmon[54076]: debug 2022-01-31T21:26:49.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.420859+0000) 2022-01-31T21:26:49.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:49 smithi167 conmon[60316]: debug 2022-01-31T21:26:49.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.423836+0000) 2022-01-31T21:26:49.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[46715]: debug 2022-01-31T21:26:49.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.197507+0000) 2022-01-31T21:26:49.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[46715]: debug 2022-01-31T21:26:49.826+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.827667+0000) 2022-01-31T21:26:49.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[51620]: debug 2022-01-31T21:26:49.778+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.778912+0000) 2022-01-31T21:26:49.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[51620]: debug 2022-01-31T21:26:49.827+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.827977+0000) 2022-01-31T21:26:49.843 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[41853]: debug 2022-01-31T21:26:49.827+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.827891+0000) 2022-01-31T21:26:50.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[35325]: debug 2022-01-31T21:26:49.841+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130409 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:50.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:49 smithi171 conmon[41853]: debug 2022-01-31T21:26:49.900+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.901388+0000) 2022-01-31T21:26:50.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:49 smithi167 conmon[54076]: debug 2022-01-31T21:26:49.828+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.829111+0000) 2022-01-31T21:26:50.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:49 smithi167 conmon[60316]: debug 2022-01-31T21:26:49.828+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.828884+0000) 2022-01-31T21:26:50.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:49 smithi167 conmon[49112]: debug 2022-01-31T21:26:49.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.754616+0000) 2022-01-31T21:26:50.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:49 smithi167 conmon[49112]: debug 2022-01-31T21:26:49.828+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:49.828649+0000) 2022-01-31T21:26:50.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:50 smithi167 conmon[60316]: debug 2022-01-31T21:26:50.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:50.424035+0000) 2022-01-31T21:26:50.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:50 smithi167 conmon[54076]: debug 2022-01-31T21:26:50.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:50.421021+0000) 2022-01-31T21:26:50.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:50 smithi171 conmon[46715]: debug 2022-01-31T21:26:50.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:50.197694+0000) 2022-01-31T21:26:50.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:50 smithi171 conmon[51620]: debug 2022-01-31T21:26:50.778+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:50.779121+0000) 2022-01-31T21:26:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:50 smithi167 conmon[49112]: debug 2022-01-31T21:26:50.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:50.754807+0000) 2022-01-31T21:26:51.196 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:50 smithi171 conmon[41853]: debug 2022-01-31T21:26:50.900+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:50.901546+0000) 2022-01-31T21:26:51.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:51 smithi167 conmon[60316]: debug 2022-01-31T21:26:51.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:51.424154+0000) 2022-01-31T21:26:51.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:51 smithi167 conmon[54076]: debug 2022-01-31T21:26:51.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:51.421176+0000) 2022-01-31T21:26:51.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:51 smithi171 conmon[46715]: debug 2022-01-31T21:26:51.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:51.197851+0000) 2022-01-31T21:26:51.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:51 smithi171 conmon[51620]: debug 2022-01-31T21:26:51.778+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:51.779276+0000) 2022-01-31T21:26:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:51 smithi167 conmon[49112]: debug 2022-01-31T21:26:51.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:51.754956+0000) 2022-01-31T21:26:52.196 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:51 smithi171 conmon[41853]: debug 2022-01-31T21:26:51.901+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:51.901732+0000) 2022-01-31T21:26:52.475 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:52 smithi167 conmon[60316]: debug 2022-01-31T21:26:52.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:52.424371+0000) 2022-01-31T21:26:52.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:52 smithi167 conmon[54076]: debug 2022-01-31T21:26:52.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:52.421380+0000) 2022-01-31T21:26:52.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:52 smithi171 conmon[46715]: debug 2022-01-31T21:26:52.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:52.198054+0000) 2022-01-31T21:26:52.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:52 smithi171 conmon[51620]: debug 2022-01-31T21:26:52.778+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:52.779448+0000) 2022-01-31T21:26:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:52 smithi167 conmon[49112]: debug 2022-01-31T21:26:52.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:52.755136+0000) 2022-01-31T21:26:53.197 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:52 smithi171 conmon[41853]: debug 2022-01-31T21:26:52.901+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:52.901943+0000) 2022-01-31T21:26:53.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:53 smithi167 conmon[60316]: debug 2022-01-31T21:26:53.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:53.424585+0000) 2022-01-31T21:26:53.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:53 smithi167 conmon[54076]: debug 2022-01-31T21:26:53.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:53.421556+0000) 2022-01-31T21:26:53.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:53 smithi171 conmon[46715]: debug 2022-01-31T21:26:53.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:53.198249+0000) 2022-01-31T21:26:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:53 smithi171 conmon[35325]: debug 2022-01-31T21:26:53.680+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:26:53.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:53 smithi171 conmon[51620]: debug 2022-01-31T21:26:53.778+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:53.779624+0000) 2022-01-31T21:26:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:53 smithi167 conmon[49112]: debug 2022-01-31T21:26:53.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:53.755361+0000) 2022-01-31T21:26:54.197 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:53 smithi171 conmon[41853]: debug 2022-01-31T21:26:53.901+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:53.902100+0000) 2022-01-31T21:26:54.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:54 smithi167 conmon[60316]: debug 2022-01-31T21:26:54.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.424724+0000) 2022-01-31T21:26:54.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:54 smithi167 conmon[54076]: debug 2022-01-31T21:26:54.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.421728+0000) 2022-01-31T21:26:54.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[46715]: debug 2022-01-31T21:26:54.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.198459+0000) 2022-01-31T21:26:54.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[46715]: debug 2022-01-31T21:26:54.842+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.843553+0000) 2022-01-31T21:26:54.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[51620]: debug 2022-01-31T21:26:54.779+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.779848+0000) 2022-01-31T21:26:55.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[35325]: debug 2022-01-31T21:26:54.857+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130520 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:26:55.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[51620]: debug 2022-01-31T21:26:54.843+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.844368+0000) 2022-01-31T21:26:55.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[41853]: debug 2022-01-31T21:26:54.843+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.844015+0000) 2022-01-31T21:26:55.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:54 smithi171 conmon[41853]: debug 2022-01-31T21:26:54.901+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.902229+0000) 2022-01-31T21:26:55.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:54 smithi167 conmon[54076]: debug 2022-01-31T21:26:54.845+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.845934+0000) 2022-01-31T21:26:55.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:54 smithi167 conmon[60316]: debug 2022-01-31T21:26:54.844+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.845167+0000) 2022-01-31T21:26:55.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:54 smithi167 conmon[49112]: debug 2022-01-31T21:26:54.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.755591+0000) 2022-01-31T21:26:55.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:54 smithi167 conmon[49112]: debug 2022-01-31T21:26:54.845+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:54.845709+0000) 2022-01-31T21:26:55.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:55 smithi167 conmon[60316]: debug 2022-01-31T21:26:55.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:55.424894+0000) 2022-01-31T21:26:55.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:55 smithi167 conmon[54076]: debug 2022-01-31T21:26:55.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:55.421873+0000) 2022-01-31T21:26:55.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:55 smithi171 conmon[46715]: debug 2022-01-31T21:26:55.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:55.198661+0000) 2022-01-31T21:26:55.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:55 smithi171 conmon[51620]: debug 2022-01-31T21:26:55.779+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:55.779995+0000) 2022-01-31T21:26:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:55 smithi167 conmon[49112]: debug 2022-01-31T21:26:55.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:55.755773+0000) 2022-01-31T21:26:56.197 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:55 smithi171 conmon[41853]: debug 2022-01-31T21:26:55.901+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:55.902386+0000) 2022-01-31T21:26:56.476 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:56 smithi167 conmon[60316]: debug 2022-01-31T21:26:56.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:56.425056+0000) 2022-01-31T21:26:56.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:56 smithi167 conmon[54076]: debug 2022-01-31T21:26:56.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:56.422045+0000) 2022-01-31T21:26:56.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:56 smithi171 conmon[46715]: debug 2022-01-31T21:26:56.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:56.198843+0000) 2022-01-31T21:26:56.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:56 smithi171 conmon[51620]: debug 2022-01-31T21:26:56.779+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:56.780130+0000) 2022-01-31T21:26:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:56 smithi167 conmon[49112]: debug 2022-01-31T21:26:56.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:56.755931+0000) 2022-01-31T21:26:57.197 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:56 smithi171 conmon[41853]: debug 2022-01-31T21:26:56.902+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:56.902591+0000) 2022-01-31T21:26:57.476 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:57 smithi167 conmon[54076]: debug 2022-01-31T21:26:57.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:57.422276+0000) 2022-01-31T21:26:57.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:57 smithi167 conmon[60316]: debug 2022-01-31T21:26:57.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:57.425261+0000) 2022-01-31T21:26:57.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:57 smithi171 conmon[46715]: debug 2022-01-31T21:26:57.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:57.199032+0000) 2022-01-31T21:26:57.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:57 smithi171 conmon[51620]: debug 2022-01-31T21:26:57.779+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:57.780330+0000) 2022-01-31T21:26:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:57 smithi167 conmon[49112]: debug 2022-01-31T21:26:57.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:57.756194+0000) 2022-01-31T21:26:58.197 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:57 smithi171 conmon[41853]: debug 2022-01-31T21:26:57.902+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:57.902775+0000) 2022-01-31T21:26:58.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:58 smithi167 conmon[60316]: debug 2022-01-31T21:26:58.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:58.425479+0000) 2022-01-31T21:26:58.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:58 smithi167 conmon[54076]: debug 2022-01-31T21:26:58.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:58.422448+0000) 2022-01-31T21:26:58.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:58 smithi171 conmon[46715]: debug 2022-01-31T21:26:58.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:58.199220+0000) 2022-01-31T21:26:58.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:58 smithi171 conmon[51620]: debug 2022-01-31T21:26:58.780+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:58.780537+0000) 2022-01-31T21:26:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:58 smithi167 conmon[49112]: debug 2022-01-31T21:26:58.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:58.756391+0000) 2022-01-31T21:26:59.198 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:58 smithi171 conmon[41853]: debug 2022-01-31T21:26:58.902+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:58.902913+0000) 2022-01-31T21:26:59.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:59 smithi167 conmon[60316]: debug 2022-01-31T21:26:59.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.425653+0000) 2022-01-31T21:26:59.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:59 smithi167 conmon[54076]: debug 2022-01-31T21:26:59.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.422629+0000) 2022-01-31T21:26:59.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[46715]: debug 2022-01-31T21:26:59.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.199415+0000) 2022-01-31T21:26:59.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[51620]: debug 2022-01-31T21:26:59.780+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.780769+0000) 2022-01-31T21:27:00.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:26:59 smithi167 conmon[54076]: debug 2022-01-31T21:26:59.868+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.869375+0000) 2022-01-31T21:27:00.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:26:59 smithi167 conmon[60316]: debug 2022-01-31T21:26:59.860+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.861046+0000) 2022-01-31T21:27:00.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:59 smithi167 conmon[49112]: debug 2022-01-31T21:26:59.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.756602+0000) 2022-01-31T21:27:00.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:26:59 smithi167 conmon[49112]: debug 2022-01-31T21:26:59.860+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.860964+0000) 2022-01-31T21:27:00.198 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[35325]: debug 2022-01-31T21:26:59.878+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130630 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:00.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[46715]: debug 2022-01-31T21:26:59.859+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.860608+0000) 2022-01-31T21:27:00.200 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[51620]: debug 2022-01-31T21:26:59.859+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.860457+0000) 2022-01-31T21:27:00.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[41853]: debug 2022-01-31T21:26:59.859+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.860376+0000) 2022-01-31T21:27:00.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:26:59 smithi171 conmon[41853]: debug 2022-01-31T21:26:59.902+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:26:59.903030+0000) 2022-01-31T21:27:00.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:00 smithi167 conmon[60316]: debug 2022-01-31T21:27:00.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:00.425824+0000) 2022-01-31T21:27:00.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:00 smithi167 conmon[54076]: debug 2022-01-31T21:27:00.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:00.422828+0000) 2022-01-31T21:27:00.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:00 smithi171 conmon[46715]: debug 2022-01-31T21:27:00.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:00.199591+0000) 2022-01-31T21:27:00.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:00 smithi171 conmon[51620]: debug 2022-01-31T21:27:00.780+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:00.780938+0000) 2022-01-31T21:27:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:00 smithi167 conmon[49112]: debug 2022-01-31T21:27:00.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:00.756819+0000) 2022-01-31T21:27:01.198 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:00 smithi171 conmon[41853]: debug 2022-01-31T21:27:00.902+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:00.903187+0000) 2022-01-31T21:27:01.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:01 smithi167 conmon[54076]: debug 2022-01-31T21:27:01.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:01.422990+0000) 2022-01-31T21:27:01.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:01 smithi167 conmon[60316]: debug 2022-01-31T21:27:01.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:01.425970+0000) 2022-01-31T21:27:01.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:01 smithi171 conmon[46715]: debug 2022-01-31T21:27:01.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:01.199724+0000) 2022-01-31T21:27:01.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:01 smithi171 conmon[51620]: debug 2022-01-31T21:27:01.780+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:01.781098+0000) 2022-01-31T21:27:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:01 smithi167 conmon[49112]: debug 2022-01-31T21:27:01.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:01.757021+0000) 2022-01-31T21:27:02.198 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:01 smithi171 conmon[41853]: debug 2022-01-31T21:27:01.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:01.903376+0000) 2022-01-31T21:27:02.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:02 smithi167 conmon[60316]: debug 2022-01-31T21:27:02.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:02.426185+0000) 2022-01-31T21:27:02.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:02 smithi167 conmon[54076]: debug 2022-01-31T21:27:02.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:02.423196+0000) 2022-01-31T21:27:02.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:02 smithi171 conmon[46715]: debug 2022-01-31T21:27:02.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:02.199910+0000) 2022-01-31T21:27:02.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:02 smithi171 conmon[51620]: debug 2022-01-31T21:27:02.780+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:02.781283+0000) 2022-01-31T21:27:03.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:02 smithi167 conmon[49112]: debug 2022-01-31T21:27:02.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:02.757193+0000) 2022-01-31T21:27:03.198 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:02 smithi171 conmon[41853]: debug 2022-01-31T21:27:02.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:02.903572+0000) 2022-01-31T21:27:03.477 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:03 smithi167 conmon[60316]: debug 2022-01-31T21:27:03.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:03.426311+0000) 2022-01-31T21:27:03.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:03 smithi167 conmon[54076]: debug 2022-01-31T21:27:03.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:03.423395+0000) 2022-01-31T21:27:03.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:03 smithi171 conmon[46715]: debug 2022-01-31T21:27:03.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:03.200093+0000) 2022-01-31T21:27:03.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:03 smithi171 conmon[51620]: debug 2022-01-31T21:27:03.781+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:03.781497+0000) 2022-01-31T21:27:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:03 smithi167 conmon[49112]: debug 2022-01-31T21:27:03.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:03.757404+0000) 2022-01-31T21:27:04.199 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:03 smithi171 conmon[41853]: debug 2022-01-31T21:27:03.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:03.903774+0000) 2022-01-31T21:27:04.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:04 smithi167 conmon[54076]: debug 2022-01-31T21:27:04.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.423582+0000) 2022-01-31T21:27:04.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:04 smithi167 conmon[60316]: debug 2022-01-31T21:27:04.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.426461+0000) 2022-01-31T21:27:04.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[46715]: debug 2022-01-31T21:27:04.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.200286+0000) 2022-01-31T21:27:04.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[51620]: debug 2022-01-31T21:27:04.780+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.781700+0000) 2022-01-31T21:27:05.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:04 smithi167 conmon[54076]: debug 2022-01-31T21:27:04.882+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.882697+0000) 2022-01-31T21:27:05.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:04 smithi167 conmon[60316]: debug 2022-01-31T21:27:04.882+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.882618+0000) 2022-01-31T21:27:05.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:04 smithi167 conmon[49112]: debug 2022-01-31T21:27:04.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.757659+0000) 2022-01-31T21:27:05.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:04 smithi167 conmon[49112]: debug 2022-01-31T21:27:04.880+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.881339+0000) 2022-01-31T21:27:05.199 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[35325]: debug 2022-01-31T21:27:04.894+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130741 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:05.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[41853]: debug 2022-01-31T21:27:04.880+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.881671+0000) 2022-01-31T21:27:05.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[41853]: debug 2022-01-31T21:27:04.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.903897+0000) 2022-01-31T21:27:05.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[46715]: debug 2022-01-31T21:27:04.880+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.881243+0000) 2022-01-31T21:27:05.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:04 smithi171 conmon[51620]: debug 2022-01-31T21:27:04.881+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:04.882154+0000) 2022-01-31T21:27:05.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:05 smithi167 conmon[54076]: debug 2022-01-31T21:27:05.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:05.423766+0000) 2022-01-31T21:27:05.479 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:05 smithi167 conmon[60316]: debug 2022-01-31T21:27:05.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:05.426615+0000) 2022-01-31T21:27:05.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:05 smithi171 conmon[46715]: debug 2022-01-31T21:27:05.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:05.200441+0000) 2022-01-31T21:27:05.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:05 smithi171 conmon[51620]: debug 2022-01-31T21:27:05.781+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:05.781912+0000) 2022-01-31T21:27:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:05 smithi167 conmon[49112]: debug 2022-01-31T21:27:05.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:05.757821+0000) 2022-01-31T21:27:06.199 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:05 smithi171 conmon[41853]: debug 2022-01-31T21:27:05.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:05.904017+0000) 2022-01-31T21:27:06.478 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:06 smithi167 conmon[54076]: debug 2022-01-31T21:27:06.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:06.423915+0000) 2022-01-31T21:27:06.479 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:06 smithi167 conmon[60316]: debug 2022-01-31T21:27:06.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:06.426738+0000) 2022-01-31T21:27:06.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:06 smithi171 conmon[46715]: debug 2022-01-31T21:27:06.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:06.200564+0000) 2022-01-31T21:27:06.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:06 smithi171 conmon[51620]: debug 2022-01-31T21:27:06.781+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:06.782074+0000) 2022-01-31T21:27:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:06 smithi167 conmon[49112]: debug 2022-01-31T21:27:06.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:06.757967+0000) 2022-01-31T21:27:07.199 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:06 smithi171 conmon[41853]: debug 2022-01-31T21:27:06.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:06.904133+0000) 2022-01-31T21:27:07.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:07 smithi167 conmon[60316]: debug 2022-01-31T21:27:07.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:07.426856+0000) 2022-01-31T21:27:07.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:07 smithi167 conmon[54076]: debug 2022-01-31T21:27:07.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:07.424109+0000) 2022-01-31T21:27:07.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:07 smithi171 conmon[46715]: debug 2022-01-31T21:27:07.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:07.200773+0000) 2022-01-31T21:27:07.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:07 smithi171 conmon[51620]: debug 2022-01-31T21:27:07.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:07.782294+0000) 2022-01-31T21:27:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:07 smithi167 conmon[49112]: debug 2022-01-31T21:27:07.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:07.758174+0000) 2022-01-31T21:27:08.199 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:07 smithi171 conmon[41853]: debug 2022-01-31T21:27:07.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:07.904319+0000) 2022-01-31T21:27:08.478 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:08 smithi167 conmon[60316]: debug 2022-01-31T21:27:08.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:08.427017+0000) 2022-01-31T21:27:08.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:08 smithi167 conmon[54076]: debug 2022-01-31T21:27:08.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:08.424260+0000) 2022-01-31T21:27:08.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:08 smithi171 conmon[46715]: debug 2022-01-31T21:27:08.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:08.200987+0000) 2022-01-31T21:27:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:08 smithi171 conmon[35325]: debug 2022-01-31T21:27:08.681+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:27:08.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:08 smithi171 conmon[51620]: debug 2022-01-31T21:27:08.781+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:08.782521+0000) 2022-01-31T21:27:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:08 smithi167 conmon[49112]: debug 2022-01-31T21:27:08.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:08.758317+0000) 2022-01-31T21:27:09.199 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:08 smithi171 conmon[41853]: debug 2022-01-31T21:27:08.904+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:08.904472+0000) 2022-01-31T21:27:09.479 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:09 smithi167 conmon[60316]: debug 2022-01-31T21:27:09.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.427267+0000) 2022-01-31T21:27:09.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:09 smithi167 conmon[54076]: debug 2022-01-31T21:27:09.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.424480+0000) 2022-01-31T21:27:09.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[46715]: debug 2022-01-31T21:27:09.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.201181+0000) 2022-01-31T21:27:09.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[51620]: debug 2022-01-31T21:27:09.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.782732+0000) 2022-01-31T21:27:10.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:09 smithi167 conmon[54076]: debug 2022-01-31T21:27:09.897+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.899147+0000) 2022-01-31T21:27:10.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:09 smithi167 conmon[60316]: debug 2022-01-31T21:27:09.897+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.898823+0000) 2022-01-31T21:27:10.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:09 smithi167 conmon[49112]: debug 2022-01-31T21:27:09.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.758529+0000) 2022-01-31T21:27:10.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:09 smithi167 conmon[49112]: debug 2022-01-31T21:27:09.897+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.898985+0000) 2022-01-31T21:27:10.200 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[35325]: debug 2022-01-31T21:27:09.911+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130851 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:10.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[46715]: debug 2022-01-31T21:27:09.897+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.897882+0000) 2022-01-31T21:27:10.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[51620]: debug 2022-01-31T21:27:09.898+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.898903+0000) 2022-01-31T21:27:10.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[41853]: debug 2022-01-31T21:27:09.896+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.897644+0000) 2022-01-31T21:27:10.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:09 smithi171 conmon[41853]: debug 2022-01-31T21:27:09.903+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:09.904640+0000) 2022-01-31T21:27:10.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:10 smithi167 conmon[54076]: debug 2022-01-31T21:27:10.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:10.424707+0000) 2022-01-31T21:27:10.480 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:10 smithi167 conmon[60316]: debug 2022-01-31T21:27:10.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:10.427474+0000) 2022-01-31T21:27:10.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:10 smithi171 conmon[46715]: debug 2022-01-31T21:27:10.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:10.201339+0000) 2022-01-31T21:27:10.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:10 smithi171 conmon[51620]: debug 2022-01-31T21:27:10.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:10.782934+0000) 2022-01-31T21:27:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:10 smithi167 conmon[49112]: debug 2022-01-31T21:27:10.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:10.758747+0000) 2022-01-31T21:27:11.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:10 smithi171 conmon[41853]: debug 2022-01-31T21:27:10.904+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:10.904774+0000) 2022-01-31T21:27:11.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:11 smithi167 conmon[54076]: debug 2022-01-31T21:27:11.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:11.424855+0000) 2022-01-31T21:27:11.479 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:11 smithi167 conmon[60316]: debug 2022-01-31T21:27:11.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:11.427602+0000) 2022-01-31T21:27:11.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:11 smithi171 conmon[46715]: debug 2022-01-31T21:27:11.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:11.201517+0000) 2022-01-31T21:27:11.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:11 smithi171 conmon[51620]: debug 2022-01-31T21:27:11.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:11.783077+0000) 2022-01-31T21:27:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:11 smithi167 conmon[49112]: debug 2022-01-31T21:27:11.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:11.758902+0000) 2022-01-31T21:27:12.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:11 smithi171 conmon[41853]: debug 2022-01-31T21:27:11.904+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:11.904955+0000) 2022-01-31T21:27:12.479 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:12 smithi167 conmon[60316]: debug 2022-01-31T21:27:12.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:12.427752+0000) 2022-01-31T21:27:12.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:12 smithi167 conmon[54076]: debug 2022-01-31T21:27:12.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:12.425042+0000) 2022-01-31T21:27:12.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:12 smithi171 conmon[46715]: debug 2022-01-31T21:27:12.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:12.201677+0000) 2022-01-31T21:27:12.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:12 smithi171 conmon[51620]: debug 2022-01-31T21:27:12.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:12.783270+0000) 2022-01-31T21:27:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:12 smithi167 conmon[49112]: debug 2022-01-31T21:27:12.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:12.759130+0000) 2022-01-31T21:27:13.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:12 smithi171 conmon[41853]: debug 2022-01-31T21:27:12.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:12.905154+0000) 2022-01-31T21:27:13.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:13 smithi167 conmon[54076]: debug 2022-01-31T21:27:13.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:13.425222+0000) 2022-01-31T21:27:13.480 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:13 smithi167 conmon[60316]: debug 2022-01-31T21:27:13.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:13.427912+0000) 2022-01-31T21:27:13.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:13 smithi171 conmon[46715]: debug 2022-01-31T21:27:13.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:13.201886+0000) 2022-01-31T21:27:13.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:13 smithi171 conmon[51620]: debug 2022-01-31T21:27:13.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:13.783485+0000) 2022-01-31T21:27:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:13 smithi167 conmon[49112]: debug 2022-01-31T21:27:13.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:13.759323+0000) 2022-01-31T21:27:14.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:13 smithi171 conmon[41853]: debug 2022-01-31T21:27:13.904+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:13.905332+0000) 2022-01-31T21:27:14.479 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:14 smithi167 conmon[60316]: debug 2022-01-31T21:27:14.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.428119+0000) 2022-01-31T21:27:14.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:14 smithi167 conmon[54076]: debug 2022-01-31T21:27:14.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.425433+0000) 2022-01-31T21:27:14.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[46715]: debug 2022-01-31T21:27:14.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.202062+0000) 2022-01-31T21:27:14.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[51620]: debug 2022-01-31T21:27:14.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.783686+0000) 2022-01-31T21:27:15.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:14 smithi167 conmon[54076]: debug 2022-01-31T21:27:14.913+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.914955+0000) 2022-01-31T21:27:15.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:14 smithi167 conmon[60316]: debug 2022-01-31T21:27:14.913+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.914990+0000) 2022-01-31T21:27:15.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:14 smithi167 conmon[49112]: debug 2022-01-31T21:27:14.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.759546+0000) 2022-01-31T21:27:15.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:14 smithi167 conmon[49112]: debug 2022-01-31T21:27:14.913+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.914666+0000) 2022-01-31T21:27:15.201 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[46715]: debug 2022-01-31T21:27:14.914+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.915713+0000) 2022-01-31T21:27:15.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[51620]: debug 2022-01-31T21:27:14.914+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.915189+0000) 2022-01-31T21:27:15.202 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[35325]: debug 2022-01-31T21:27:14.927+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 130960 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:15.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[41853]: debug 2022-01-31T21:27:14.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.905483+0000) 2022-01-31T21:27:15.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:14 smithi171 conmon[41853]: debug 2022-01-31T21:27:14.914+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:14.915576+0000) 2022-01-31T21:27:15.479 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:15 smithi167 conmon[54076]: debug 2022-01-31T21:27:15.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:15.425589+0000) 2022-01-31T21:27:15.480 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:15 smithi167 conmon[60316]: debug 2022-01-31T21:27:15.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:15.428321+0000) 2022-01-31T21:27:15.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:15 smithi171 conmon[46715]: debug 2022-01-31T21:27:15.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:15.202207+0000) 2022-01-31T21:27:15.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:15 smithi171 conmon[51620]: debug 2022-01-31T21:27:15.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:15.783868+0000) 2022-01-31T21:27:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:15 smithi167 conmon[49112]: debug 2022-01-31T21:27:15.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:15.759735+0000) 2022-01-31T21:27:16.201 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:15 smithi171 conmon[41853]: debug 2022-01-31T21:27:15.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:15.905687+0000) 2022-01-31T21:27:16.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:16 smithi167 conmon[54076]: debug 2022-01-31T21:27:16.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:16.425744+0000) 2022-01-31T21:27:16.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:16 smithi167 conmon[60316]: debug 2022-01-31T21:27:16.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:16.428488+0000) 2022-01-31T21:27:16.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:16 smithi171 conmon[46715]: debug 2022-01-31T21:27:16.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:16.202318+0000) 2022-01-31T21:27:16.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:16 smithi171 conmon[51620]: debug 2022-01-31T21:27:16.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:16.784035+0000) 2022-01-31T21:27:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:16 smithi167 conmon[49112]: debug 2022-01-31T21:27:16.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:16.759904+0000) 2022-01-31T21:27:17.201 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:16 smithi171 conmon[41853]: debug 2022-01-31T21:27:16.904+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:16.905849+0000) 2022-01-31T21:27:17.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:17 smithi167 conmon[54076]: debug 2022-01-31T21:27:17.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:17.425906+0000) 2022-01-31T21:27:17.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:17 smithi167 conmon[60316]: debug 2022-01-31T21:27:17.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:17.428677+0000) 2022-01-31T21:27:17.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:17 smithi171 conmon[46715]: debug 2022-01-31T21:27:17.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:17.202495+0000) 2022-01-31T21:27:17.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:17 smithi171 conmon[51620]: debug 2022-01-31T21:27:17.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:17.784235+0000) 2022-01-31T21:27:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:17 smithi167 conmon[49112]: debug 2022-01-31T21:27:17.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:17.760129+0000) 2022-01-31T21:27:18.201 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:17 smithi171 conmon[41853]: debug 2022-01-31T21:27:17.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:17.906034+0000) 2022-01-31T21:27:18.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:18 smithi167 conmon[54076]: debug 2022-01-31T21:27:18.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:18.426092+0000) 2022-01-31T21:27:18.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:18 smithi167 conmon[60316]: debug 2022-01-31T21:27:18.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:18.428866+0000) 2022-01-31T21:27:18.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:18 smithi171 conmon[46715]: debug 2022-01-31T21:27:18.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:18.202653+0000) 2022-01-31T21:27:18.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:18 smithi171 conmon[51620]: debug 2022-01-31T21:27:18.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:18.784442+0000) 2022-01-31T21:27:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:18 smithi167 conmon[49112]: debug 2022-01-31T21:27:18.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:18.760335+0000) 2022-01-31T21:27:19.201 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:18 smithi171 conmon[41853]: debug 2022-01-31T21:27:18.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:18.906228+0000) 2022-01-31T21:27:19.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:19 smithi167 conmon[54076]: debug 2022-01-31T21:27:19.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.426326+0000) 2022-01-31T21:27:19.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:19 smithi167 conmon[60316]: debug 2022-01-31T21:27:19.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.429053+0000) 2022-01-31T21:27:19.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[46715]: debug 2022-01-31T21:27:19.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.202852+0000) 2022-01-31T21:27:19.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[51620]: debug 2022-01-31T21:27:19.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.784585+0000) 2022-01-31T21:27:20.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:19 smithi167 conmon[54076]: debug 2022-01-31T21:27:19.930+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.931524+0000) 2022-01-31T21:27:20.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:19 smithi167 conmon[60316]: debug 2022-01-31T21:27:19.929+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.931015+0000) 2022-01-31T21:27:20.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:19 smithi167 conmon[49112]: debug 2022-01-31T21:27:19.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.760565+0000) 2022-01-31T21:27:20.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:19 smithi167 conmon[49112]: debug 2022-01-31T21:27:19.929+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.930841+0000) 2022-01-31T21:27:20.201 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[35325]: debug 2022-01-31T21:27:19.943+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131071 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:20.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[46715]: debug 2022-01-31T21:27:19.929+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.931426+0000) 2022-01-31T21:27:20.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[51620]: debug 2022-01-31T21:27:19.928+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.930557+0000) 2022-01-31T21:27:20.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[41853]: debug 2022-01-31T21:27:19.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.906414+0000) 2022-01-31T21:27:20.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:19 smithi171 conmon[41853]: debug 2022-01-31T21:27:19.929+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:19.931060+0000) 2022-01-31T21:27:20.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:20 smithi167 conmon[54076]: debug 2022-01-31T21:27:20.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:20.426491+0000) 2022-01-31T21:27:20.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:20 smithi167 conmon[60316]: debug 2022-01-31T21:27:20.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:20.429259+0000) 2022-01-31T21:27:20.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:20 smithi171 conmon[46715]: debug 2022-01-31T21:27:20.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:20.203018+0000) 2022-01-31T21:27:20.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:20 smithi171 conmon[51620]: debug 2022-01-31T21:27:20.782+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:20.784704+0000) 2022-01-31T21:27:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:20 smithi167 conmon[49112]: debug 2022-01-31T21:27:20.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:20.760725+0000) 2022-01-31T21:27:21.201 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:20 smithi171 conmon[41853]: debug 2022-01-31T21:27:20.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:20.906618+0000) 2022-01-31T21:27:21.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:21 smithi167 conmon[54076]: debug 2022-01-31T21:27:21.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:21.426696+0000) 2022-01-31T21:27:21.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:21 smithi167 conmon[60316]: debug 2022-01-31T21:27:21.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:21.429427+0000) 2022-01-31T21:27:21.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:21 smithi171 conmon[46715]: debug 2022-01-31T21:27:21.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:21.203173+0000) 2022-01-31T21:27:21.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:21 smithi171 conmon[51620]: debug 2022-01-31T21:27:21.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:21.784831+0000) 2022-01-31T21:27:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:21 smithi167 conmon[49112]: debug 2022-01-31T21:27:21.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:21.760921+0000) 2022-01-31T21:27:22.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:21 smithi171 conmon[41853]: debug 2022-01-31T21:27:21.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:21.906794+0000) 2022-01-31T21:27:22.481 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:22 smithi167 conmon[54076]: debug 2022-01-31T21:27:22.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:22.426910+0000) 2022-01-31T21:27:22.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:22 smithi167 conmon[60316]: debug 2022-01-31T21:27:22.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:22.429612+0000) 2022-01-31T21:27:22.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:22 smithi171 conmon[46715]: debug 2022-01-31T21:27:22.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:22.203380+0000) 2022-01-31T21:27:22.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:22 smithi171 conmon[51620]: debug 2022-01-31T21:27:22.784+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:22.784971+0000) 2022-01-31T21:27:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:22 smithi167 conmon[49112]: debug 2022-01-31T21:27:22.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:22.761137+0000) 2022-01-31T21:27:23.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:22 smithi171 conmon[41853]: debug 2022-01-31T21:27:22.905+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:22.906983+0000) 2022-01-31T21:27:23.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:23 smithi167 conmon[60316]: debug 2022-01-31T21:27:23.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:23.429770+0000) 2022-01-31T21:27:23.482 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:23 smithi167 conmon[54076]: debug 2022-01-31T21:27:23.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:23.427114+0000) 2022-01-31T21:27:23.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:23 smithi171 conmon[46715]: debug 2022-01-31T21:27:23.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:23.203543+0000) 2022-01-31T21:27:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:23 smithi171 conmon[35325]: debug 2022-01-31T21:27:23.681+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:27:23.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:23 smithi171 conmon[51620]: debug 2022-01-31T21:27:23.783+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:23.785094+0000) 2022-01-31T21:27:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:23 smithi167 conmon[49112]: debug 2022-01-31T21:27:23.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:23.761353+0000) 2022-01-31T21:27:24.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:23 smithi171 conmon[41853]: debug 2022-01-31T21:27:23.906+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:23.907169+0000) 2022-01-31T21:27:24.481 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:24 smithi167 conmon[60316]: debug 2022-01-31T21:27:24.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.429960+0000) 2022-01-31T21:27:24.482 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:24 smithi167 conmon[54076]: debug 2022-01-31T21:27:24.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.427320+0000) 2022-01-31T21:27:24.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[46715]: debug 2022-01-31T21:27:24.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.203734+0000) 2022-01-31T21:27:24.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[51620]: debug 2022-01-31T21:27:24.784+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.785243+0000) 2022-01-31T21:27:25.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:24 smithi167 conmon[54076]: debug 2022-01-31T21:27:24.946+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.948054+0000) 2022-01-31T21:27:25.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:24 smithi167 conmon[60316]: debug 2022-01-31T21:27:24.945+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.946822+0000) 2022-01-31T21:27:25.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:24 smithi167 conmon[49112]: debug 2022-01-31T21:27:24.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.761580+0000) 2022-01-31T21:27:25.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:24 smithi167 conmon[49112]: debug 2022-01-31T21:27:24.945+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.947425+0000) 2022-01-31T21:27:25.202 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[35325]: debug 2022-01-31T21:27:24.959+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131180 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:25.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[46715]: debug 2022-01-31T21:27:24.945+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.947327+0000) 2022-01-31T21:27:25.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[41853]: debug 2022-01-31T21:27:24.906+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.907372+0000) 2022-01-31T21:27:25.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[41853]: debug 2022-01-31T21:27:24.945+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.947153+0000) 2022-01-31T21:27:25.205 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:24 smithi171 conmon[51620]: debug 2022-01-31T21:27:24.945+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:24.946973+0000) 2022-01-31T21:27:25.481 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:25 smithi167 conmon[54076]: debug 2022-01-31T21:27:25.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:25.427525+0000) 2022-01-31T21:27:25.482 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:25 smithi167 conmon[60316]: debug 2022-01-31T21:27:25.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:25.430128+0000) 2022-01-31T21:27:25.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:25 smithi171 conmon[46715]: debug 2022-01-31T21:27:25.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:25.203977+0000) 2022-01-31T21:27:25.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:25 smithi171 conmon[51620]: debug 2022-01-31T21:27:25.784+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:25.785390+0000) 2022-01-31T21:27:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:25 smithi167 conmon[49112]: debug 2022-01-31T21:27:25.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:25.761794+0000) 2022-01-31T21:27:26.202 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:25 smithi171 conmon[41853]: debug 2022-01-31T21:27:25.906+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:25.907553+0000) 2022-01-31T21:27:26.482 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:26 smithi167 conmon[54076]: debug 2022-01-31T21:27:26.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:26.427675+0000) 2022-01-31T21:27:26.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:26 smithi167 conmon[60316]: debug 2022-01-31T21:27:26.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:26.430292+0000) 2022-01-31T21:27:26.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:26 smithi171 conmon[46715]: debug 2022-01-31T21:27:26.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:26.204088+0000) 2022-01-31T21:27:26.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:26 smithi171 conmon[51620]: debug 2022-01-31T21:27:26.784+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:26.785532+0000) 2022-01-31T21:27:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:26 smithi167 conmon[49112]: debug 2022-01-31T21:27:26.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:26.761971+0000) 2022-01-31T21:27:27.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:26 smithi171 conmon[41853]: debug 2022-01-31T21:27:26.906+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:26.907709+0000) 2022-01-31T21:27:27.482 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:27 smithi167 conmon[54076]: debug 2022-01-31T21:27:27.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:27.427823+0000) 2022-01-31T21:27:27.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:27 smithi167 conmon[60316]: debug 2022-01-31T21:27:27.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:27.430435+0000) 2022-01-31T21:27:27.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:27 smithi171 conmon[46715]: debug 2022-01-31T21:27:27.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:27.204258+0000) 2022-01-31T21:27:27.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:27 smithi171 conmon[51620]: debug 2022-01-31T21:27:27.784+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:27.785766+0000) 2022-01-31T21:27:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:27 smithi167 conmon[49112]: debug 2022-01-31T21:27:27.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:27.762153+0000) 2022-01-31T21:27:28.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:27 smithi171 conmon[41853]: debug 2022-01-31T21:27:27.907+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:27.907848+0000) 2022-01-31T21:27:28.482 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:28 smithi167 conmon[60316]: debug 2022-01-31T21:27:28.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:28.430601+0000) 2022-01-31T21:27:28.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:28 smithi167 conmon[54076]: debug 2022-01-31T21:27:28.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:28.427997+0000) 2022-01-31T21:27:28.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:28 smithi171 conmon[46715]: debug 2022-01-31T21:27:28.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:28.204442+0000) 2022-01-31T21:27:28.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:28 smithi171 conmon[51620]: debug 2022-01-31T21:27:28.785+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:28.785962+0000) 2022-01-31T21:27:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:28 smithi167 conmon[49112]: debug 2022-01-31T21:27:28.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:28.762331+0000) 2022-01-31T21:27:29.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:28 smithi171 conmon[41853]: debug 2022-01-31T21:27:28.907+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:28.908052+0000) 2022-01-31T21:27:29.482 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:29 smithi167 conmon[60316]: debug 2022-01-31T21:27:29.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.430763+0000) 2022-01-31T21:27:29.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:29 smithi167 conmon[54076]: debug 2022-01-31T21:27:29.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.428178+0000) 2022-01-31T21:27:29.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[46715]: debug 2022-01-31T21:27:29.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.204655+0000) 2022-01-31T21:27:29.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[51620]: debug 2022-01-31T21:27:29.785+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.786128+0000) 2022-01-31T21:27:30.085 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:29 smithi167 conmon[54076]: debug 2022-01-31T21:27:29.962+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.964369+0000) 2022-01-31T21:27:30.085 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:29 smithi167 conmon[60316]: debug 2022-01-31T21:27:29.962+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.964015+0000) 2022-01-31T21:27:30.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:29 smithi167 conmon[49112]: debug 2022-01-31T21:27:29.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.762556+0000) 2022-01-31T21:27:30.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:29 smithi167 conmon[49112]: debug 2022-01-31T21:27:29.963+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.964685+0000) 2022-01-31T21:27:30.203 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[35325]: debug 2022-01-31T21:27:29.977+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131292 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:30.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[41853]: debug 2022-01-31T21:27:29.907+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.908215+0000) 2022-01-31T21:27:30.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[41853]: debug 2022-01-31T21:27:29.963+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.964221+0000) 2022-01-31T21:27:30.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[46715]: debug 2022-01-31T21:27:29.962+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.963314+0000) 2022-01-31T21:27:30.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:29 smithi171 conmon[51620]: debug 2022-01-31T21:27:29.962+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:29.963518+0000) 2022-01-31T21:27:30.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:30 smithi167 conmon[60316]: debug 2022-01-31T21:27:30.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:30.430906+0000) 2022-01-31T21:27:30.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:30 smithi167 conmon[54076]: debug 2022-01-31T21:27:30.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:30.428391+0000) 2022-01-31T21:27:30.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:30 smithi171 conmon[46715]: debug 2022-01-31T21:27:30.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:30.204780+0000) 2022-01-31T21:27:30.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:30 smithi171 conmon[51620]: debug 2022-01-31T21:27:30.785+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:30.786342+0000) 2022-01-31T21:27:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:30 smithi167 conmon[49112]: debug 2022-01-31T21:27:30.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:30.762777+0000) 2022-01-31T21:27:31.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:30 smithi171 conmon[41853]: debug 2022-01-31T21:27:30.907+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:30.908405+0000) 2022-01-31T21:27:31.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:31 smithi167 conmon[54076]: debug 2022-01-31T21:27:31.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:31.428574+0000) 2022-01-31T21:27:31.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:31 smithi167 conmon[60316]: debug 2022-01-31T21:27:31.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:31.431050+0000) 2022-01-31T21:27:31.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:31 smithi171 conmon[46715]: debug 2022-01-31T21:27:31.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:31.204944+0000) 2022-01-31T21:27:31.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:31 smithi171 conmon[51620]: debug 2022-01-31T21:27:31.785+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:31.786479+0000) 2022-01-31T21:27:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:31 smithi167 conmon[49112]: debug 2022-01-31T21:27:31.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:31.762943+0000) 2022-01-31T21:27:32.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:31 smithi171 conmon[41853]: debug 2022-01-31T21:27:31.907+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:31.908543+0000) 2022-01-31T21:27:32.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:32 smithi167 conmon[60316]: debug 2022-01-31T21:27:32.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:32.431201+0000) 2022-01-31T21:27:32.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:32 smithi167 conmon[54076]: debug 2022-01-31T21:27:32.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:32.428713+0000) 2022-01-31T21:27:32.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:32 smithi171 conmon[46715]: debug 2022-01-31T21:27:32.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:32.205127+0000) 2022-01-31T21:27:32.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:32 smithi171 conmon[51620]: debug 2022-01-31T21:27:32.785+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:32.786695+0000) 2022-01-31T21:27:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:32 smithi167 conmon[49112]: debug 2022-01-31T21:27:32.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:32.763174+0000) 2022-01-31T21:27:33.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:32 smithi171 conmon[41853]: debug 2022-01-31T21:27:32.908+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:32.908774+0000) 2022-01-31T21:27:33.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:33 smithi167 conmon[60316]: debug 2022-01-31T21:27:33.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:33.431386+0000) 2022-01-31T21:27:33.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:33 smithi167 conmon[54076]: debug 2022-01-31T21:27:33.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:33.428864+0000) 2022-01-31T21:27:33.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:33 smithi171 conmon[46715]: debug 2022-01-31T21:27:33.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:33.205268+0000) 2022-01-31T21:27:33.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:33 smithi171 conmon[51620]: debug 2022-01-31T21:27:33.786+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:33.786852+0000) 2022-01-31T21:27:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:33 smithi167 conmon[49112]: debug 2022-01-31T21:27:33.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:33.763389+0000) 2022-01-31T21:27:34.203 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:33 smithi171 conmon[41853]: debug 2022-01-31T21:27:33.908+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:33.909000+0000) 2022-01-31T21:27:34.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:34 smithi167 conmon[60316]: debug 2022-01-31T21:27:34.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.431578+0000) 2022-01-31T21:27:34.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:34 smithi167 conmon[54076]: debug 2022-01-31T21:27:34.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.429076+0000) 2022-01-31T21:27:34.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[46715]: debug 2022-01-31T21:27:34.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.205439+0000) 2022-01-31T21:27:34.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[51620]: debug 2022-01-31T21:27:34.786+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.787054+0000) 2022-01-31T21:27:35.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:34 smithi167 conmon[54076]: debug 2022-01-31T21:27:34.980+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.981162+0000) 2022-01-31T21:27:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:34 smithi167 conmon[49112]: debug 2022-01-31T21:27:34.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.763606+0000) 2022-01-31T21:27:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:34 smithi167 conmon[49112]: debug 2022-01-31T21:27:34.981+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.981515+0000) 2022-01-31T21:27:35.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:34 smithi167 conmon[60316]: debug 2022-01-31T21:27:34.980+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.981335+0000) 2022-01-31T21:27:35.204 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[35325]: debug 2022-01-31T21:27:34.993+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131402 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:35.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[46715]: debug 2022-01-31T21:27:34.979+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.979893+0000) 2022-01-31T21:27:35.205 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[51620]: debug 2022-01-31T21:27:34.979+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.979738+0000) 2022-01-31T21:27:35.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[41853]: debug 2022-01-31T21:27:34.908+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.909116+0000) 2022-01-31T21:27:35.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:34 smithi171 conmon[41853]: debug 2022-01-31T21:27:34.980+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:34.981023+0000) 2022-01-31T21:27:35.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:35 smithi167 conmon[54076]: debug 2022-01-31T21:27:35.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:35.429262+0000) 2022-01-31T21:27:35.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:35 smithi167 conmon[60316]: debug 2022-01-31T21:27:35.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:35.431734+0000) 2022-01-31T21:27:35.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:35 smithi171 conmon[46715]: debug 2022-01-31T21:27:35.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:35.205598+0000) 2022-01-31T21:27:35.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:35 smithi171 conmon[51620]: debug 2022-01-31T21:27:35.786+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:35.787255+0000) 2022-01-31T21:27:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:35 smithi167 conmon[49112]: debug 2022-01-31T21:27:35.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:35.763827+0000) 2022-01-31T21:27:36.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:35 smithi171 conmon[41853]: debug 2022-01-31T21:27:35.908+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:35.909290+0000) 2022-01-31T21:27:36.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:36 smithi167 conmon[54076]: debug 2022-01-31T21:27:36.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:36.429411+0000) 2022-01-31T21:27:36.484 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:36 smithi167 conmon[60316]: debug 2022-01-31T21:27:36.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:36.431857+0000) 2022-01-31T21:27:36.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:36 smithi171 conmon[46715]: debug 2022-01-31T21:27:36.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:36.205715+0000) 2022-01-31T21:27:36.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:36 smithi171 conmon[51620]: debug 2022-01-31T21:27:36.786+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:36.787452+0000) 2022-01-31T21:27:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:36 smithi167 conmon[49112]: debug 2022-01-31T21:27:36.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:36.764015+0000) 2022-01-31T21:27:37.204 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:36 smithi171 conmon[41853]: debug 2022-01-31T21:27:36.908+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:36.909450+0000) 2022-01-31T21:27:37.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:37 smithi167 conmon[54076]: debug 2022-01-31T21:27:37.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:37.429607+0000) 2022-01-31T21:27:37.484 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:37 smithi167 conmon[60316]: debug 2022-01-31T21:27:37.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:37.432029+0000) 2022-01-31T21:27:37.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:37 smithi171 conmon[46715]: debug 2022-01-31T21:27:37.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:37.205945+0000) 2022-01-31T21:27:37.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:37 smithi171 conmon[51620]: debug 2022-01-31T21:27:37.786+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:37.787618+0000) 2022-01-31T21:27:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:37 smithi167 conmon[49112]: debug 2022-01-31T21:27:37.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:37.764202+0000) 2022-01-31T21:27:38.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:37 smithi171 conmon[41853]: debug 2022-01-31T21:27:37.908+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:37.909593+0000) 2022-01-31T21:27:38.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:38 smithi167 conmon[54076]: debug 2022-01-31T21:27:38.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:38.429768+0000) 2022-01-31T21:27:38.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:38 smithi167 conmon[60316]: debug 2022-01-31T21:27:38.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:38.432232+0000) 2022-01-31T21:27:38.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:38 smithi171 conmon[46715]: debug 2022-01-31T21:27:38.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:38.206131+0000) 2022-01-31T21:27:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:38 smithi171 conmon[35325]: debug 2022-01-31T21:27:38.682+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:27:38.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:38 smithi171 conmon[51620]: debug 2022-01-31T21:27:38.787+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:38.787807+0000) 2022-01-31T21:27:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:38 smithi167 conmon[49112]: debug 2022-01-31T21:27:38.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:38.764408+0000) 2022-01-31T21:27:39.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:38 smithi171 conmon[41853]: debug 2022-01-31T21:27:38.909+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:38.909793+0000) 2022-01-31T21:27:39.484 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:39 smithi167 conmon[60316]: debug 2022-01-31T21:27:39.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.432383+0000) 2022-01-31T21:27:39.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:39 smithi167 conmon[54076]: debug 2022-01-31T21:27:39.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.429958+0000) 2022-01-31T21:27:39.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:39 smithi171 conmon[46715]: debug 2022-01-31T21:27:39.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.206327+0000) 2022-01-31T21:27:39.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:39 smithi171 conmon[51620]: debug 2022-01-31T21:27:39.787+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.787996+0000) 2022-01-31T21:27:40.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:39 smithi167 conmon[54076]: debug 2022-01-31T21:27:39.997+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.998433+0000) 2022-01-31T21:27:40.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:39 smithi167 conmon[60316]: debug 2022-01-31T21:27:39.998+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.998691+0000) 2022-01-31T21:27:40.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:39 smithi167 conmon[49112]: debug 2022-01-31T21:27:39.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.764565+0000) 2022-01-31T21:27:40.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:39 smithi167 conmon[49112]: debug 2022-01-31T21:27:39.997+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.997552+0000) 2022-01-31T21:27:40.205 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:40 smithi171 conmon[35325]: debug 2022-01-31T21:27:40.010+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131514 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:40.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:39 smithi171 conmon[46715]: debug 2022-01-31T21:27:39.996+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.996786+0000) 2022-01-31T21:27:40.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:39 smithi171 conmon[51620]: debug 2022-01-31T21:27:39.996+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.996923+0000) 2022-01-31T21:27:40.207 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:39 smithi171 conmon[41853]: debug 2022-01-31T21:27:39.909+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.909918+0000) 2022-01-31T21:27:40.207 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:39 smithi171 conmon[41853]: debug 2022-01-31T21:27:39.996+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:39.997434+0000) 2022-01-31T21:27:40.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:40 smithi171 conmon[46715]: debug 2022-01-31T21:27:40.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:40.206528+0000) 2022-01-31T21:27:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:40 smithi167 conmon[54076]: debug 2022-01-31T21:27:40.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:40.430166+0000) 2022-01-31T21:27:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:40 smithi167 conmon[60316]: debug 2022-01-31T21:27:40.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:40.432564+0000) 2022-01-31T21:27:40.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:40 smithi171 conmon[51620]: debug 2022-01-31T21:27:40.787+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:40.788113+0000) 2022-01-31T21:27:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:40 smithi167 conmon[49112]: debug 2022-01-31T21:27:40.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:40.764751+0000) 2022-01-31T21:27:41.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:40 smithi171 conmon[41853]: debug 2022-01-31T21:27:40.909+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:40.910073+0000) 2022-01-31T21:27:41.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:41 smithi167 conmon[54076]: debug 2022-01-31T21:27:41.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:41.430299+0000) 2022-01-31T21:27:41.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:41 smithi167 conmon[60316]: debug 2022-01-31T21:27:41.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:41.432670+0000) 2022-01-31T21:27:41.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:41 smithi171 conmon[46715]: debug 2022-01-31T21:27:41.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:41.206635+0000) 2022-01-31T21:27:41.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:41 smithi171 conmon[51620]: debug 2022-01-31T21:27:41.787+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:41.788262+0000) 2022-01-31T21:27:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:41 smithi167 conmon[49112]: debug 2022-01-31T21:27:41.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:41.764852+0000) 2022-01-31T21:27:42.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:41 smithi171 conmon[41853]: debug 2022-01-31T21:27:41.909+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:41.910211+0000) 2022-01-31T21:27:42.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:42 smithi167 conmon[54076]: debug 2022-01-31T21:27:42.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:42.430475+0000) 2022-01-31T21:27:42.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:42 smithi167 conmon[60316]: debug 2022-01-31T21:27:42.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:42.432870+0000) 2022-01-31T21:27:42.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:42 smithi171 conmon[46715]: debug 2022-01-31T21:27:42.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:42.206797+0000) 2022-01-31T21:27:42.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:42 smithi171 conmon[51620]: debug 2022-01-31T21:27:42.787+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:42.788440+0000) 2022-01-31T21:27:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:42 smithi167 conmon[49112]: debug 2022-01-31T21:27:42.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:42.765047+0000) 2022-01-31T21:27:43.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:42 smithi171 conmon[41853]: debug 2022-01-31T21:27:42.909+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:42.910428+0000) 2022-01-31T21:27:43.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:43 smithi167 conmon[54076]: debug 2022-01-31T21:27:43.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:43.430660+0000) 2022-01-31T21:27:43.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:43 smithi167 conmon[60316]: debug 2022-01-31T21:27:43.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:43.433013+0000) 2022-01-31T21:27:43.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:43 smithi171 conmon[46715]: debug 2022-01-31T21:27:43.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:43.206957+0000) 2022-01-31T21:27:43.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:43 smithi171 conmon[51620]: debug 2022-01-31T21:27:43.787+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:43.788641+0000) 2022-01-31T21:27:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:43 smithi167 conmon[49112]: debug 2022-01-31T21:27:43.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:43.765236+0000) 2022-01-31T21:27:44.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:43 smithi171 conmon[41853]: debug 2022-01-31T21:27:43.909+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:43.910521+0000) 2022-01-31T21:27:44.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:44 smithi167 conmon[54076]: debug 2022-01-31T21:27:44.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:44.430804+0000) 2022-01-31T21:27:44.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:44 smithi167 conmon[60316]: debug 2022-01-31T21:27:44.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:44.433182+0000) 2022-01-31T21:27:44.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:44 smithi171 conmon[46715]: debug 2022-01-31T21:27:44.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:44.207155+0000) 2022-01-31T21:27:44.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:44 smithi171 conmon[51620]: debug 2022-01-31T21:27:44.788+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:44.788824+0000) 2022-01-31T21:27:45.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:44 smithi167 conmon[49112]: debug 2022-01-31T21:27:44.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:44.765452+0000) 2022-01-31T21:27:45.206 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[35325]: debug 2022-01-31T21:27:45.027+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131623 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:45.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[46715]: debug 2022-01-31T21:27:45.013+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.013744+0000) 2022-01-31T21:27:45.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[51620]: debug 2022-01-31T21:27:45.013+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.014244+0000) 2022-01-31T21:27:45.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:44 smithi171 conmon[41853]: debug 2022-01-31T21:27:44.910+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:44.910675+0000) 2022-01-31T21:27:45.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[41853]: debug 2022-01-31T21:27:45.013+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.014641+0000) 2022-01-31T21:27:45.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:45 smithi167 conmon[49112]: debug 2022-01-31T21:27:45.013+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.014140+0000) 2022-01-31T21:27:45.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:45 smithi167 conmon[54076]: debug 2022-01-31T21:27:45.014+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.015027+0000) 2022-01-31T21:27:45.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:45 smithi167 conmon[60316]: debug 2022-01-31T21:27:45.014+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.015240+0000) 2022-01-31T21:27:45.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[46715]: debug 2022-01-31T21:27:45.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.207349+0000) 2022-01-31T21:27:45.692 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:45 smithi167 conmon[54076]: debug 2022-01-31T21:27:45.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.430934+0000) 2022-01-31T21:27:45.692 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:45 smithi167 conmon[60316]: debug 2022-01-31T21:27:45.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.433361+0000) 2022-01-31T21:27:45.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[51620]: debug 2022-01-31T21:27:45.788+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.788963+0000) 2022-01-31T21:27:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:45 smithi167 conmon[49112]: debug 2022-01-31T21:27:45.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.765617+0000) 2022-01-31T21:27:46.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:45 smithi171 conmon[41853]: debug 2022-01-31T21:27:45.910+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:45.910860+0000) 2022-01-31T21:27:46.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:46 smithi167 conmon[54076]: debug 2022-01-31T21:27:46.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:46.431112+0000) 2022-01-31T21:27:46.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:46 smithi167 conmon[60316]: debug 2022-01-31T21:27:46.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:46.433480+0000) 2022-01-31T21:27:46.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:46 smithi171 conmon[46715]: debug 2022-01-31T21:27:46.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:46.207514+0000) 2022-01-31T21:27:46.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:46 smithi171 conmon[51620]: debug 2022-01-31T21:27:46.788+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:46.789103+0000) 2022-01-31T21:27:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:46 smithi167 conmon[49112]: debug 2022-01-31T21:27:46.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:46.765733+0000) 2022-01-31T21:27:47.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:46 smithi171 conmon[41853]: debug 2022-01-31T21:27:46.910+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:46.911010+0000) 2022-01-31T21:27:47.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:47 smithi167 conmon[54076]: debug 2022-01-31T21:27:47.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:47.431330+0000) 2022-01-31T21:27:47.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:47 smithi167 conmon[60316]: debug 2022-01-31T21:27:47.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:47.433683+0000) 2022-01-31T21:27:47.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:47 smithi171 conmon[46715]: debug 2022-01-31T21:27:47.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:47.207698+0000) 2022-01-31T21:27:47.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:47 smithi171 conmon[51620]: debug 2022-01-31T21:27:47.788+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:47.789288+0000) 2022-01-31T21:27:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:47 smithi167 conmon[49112]: debug 2022-01-31T21:27:47.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:47.765929+0000) 2022-01-31T21:27:48.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:47 smithi171 conmon[41853]: debug 2022-01-31T21:27:47.910+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:47.911213+0000) 2022-01-31T21:27:48.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:48 smithi167 conmon[54076]: debug 2022-01-31T21:27:48.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:48.431485+0000) 2022-01-31T21:27:48.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:48 smithi167 conmon[60316]: debug 2022-01-31T21:27:48.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:48.433863+0000) 2022-01-31T21:27:48.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:48 smithi171 conmon[46715]: debug 2022-01-31T21:27:48.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:48.207914+0000) 2022-01-31T21:27:48.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:48 smithi171 conmon[51620]: debug 2022-01-31T21:27:48.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:48.789511+0000) 2022-01-31T21:27:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:48 smithi167 conmon[49112]: debug 2022-01-31T21:27:48.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:48.766159+0000) 2022-01-31T21:27:49.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:48 smithi171 conmon[41853]: debug 2022-01-31T21:27:48.911+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:48.911404+0000) 2022-01-31T21:27:49.486 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:49 smithi167 conmon[54076]: debug 2022-01-31T21:27:49.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:49.431631+0000) 2022-01-31T21:27:49.487 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:49 smithi167 conmon[60316]: debug 2022-01-31T21:27:49.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:49.434060+0000) 2022-01-31T21:27:49.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:49 smithi171 conmon[46715]: debug 2022-01-31T21:27:49.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:49.208077+0000) 2022-01-31T21:27:49.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:49 smithi171 conmon[51620]: debug 2022-01-31T21:27:49.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:49.789717+0000) 2022-01-31T21:27:50.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:49 smithi167 conmon[49112]: debug 2022-01-31T21:27:49.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:49.766388+0000) 2022-01-31T21:27:50.207 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[35325]: debug 2022-01-31T21:27:50.044+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131735 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:50.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[46715]: debug 2022-01-31T21:27:50.031+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.031886+0000) 2022-01-31T21:27:50.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[51620]: debug 2022-01-31T21:27:50.031+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.031805+0000) 2022-01-31T21:27:50.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:49 smithi171 conmon[41853]: debug 2022-01-31T21:27:49.911+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:49.911555+0000) 2022-01-31T21:27:50.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[41853]: debug 2022-01-31T21:27:50.029+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.030512+0000) 2022-01-31T21:27:50.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:50 smithi167 conmon[49112]: debug 2022-01-31T21:27:50.030+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.030673+0000) 2022-01-31T21:27:50.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:50 smithi167 conmon[54076]: debug 2022-01-31T21:27:50.030+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.031265+0000) 2022-01-31T21:27:50.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:50 smithi167 conmon[60316]: debug 2022-01-31T21:27:50.031+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.032421+0000) 2022-01-31T21:27:50.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[46715]: debug 2022-01-31T21:27:50.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.208228+0000) 2022-01-31T21:27:50.750 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:50 smithi167 conmon[54076]: debug 2022-01-31T21:27:50.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.431777+0000) 2022-01-31T21:27:50.751 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:50 smithi167 conmon[60316]: debug 2022-01-31T21:27:50.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.434287+0000) 2022-01-31T21:27:50.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[51620]: debug 2022-01-31T21:27:50.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.789905+0000) 2022-01-31T21:27:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:50 smithi167 conmon[49112]: debug 2022-01-31T21:27:50.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.766607+0000) 2022-01-31T21:27:51.207 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:50 smithi171 conmon[41853]: debug 2022-01-31T21:27:50.910+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:50.911732+0000) 2022-01-31T21:27:51.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:51 smithi167 conmon[60316]: debug 2022-01-31T21:27:51.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:51.434475+0000) 2022-01-31T21:27:51.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:51 smithi167 conmon[54076]: debug 2022-01-31T21:27:51.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:51.431884+0000) 2022-01-31T21:27:51.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:51 smithi171 conmon[46715]: debug 2022-01-31T21:27:51.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:51.208398+0000) 2022-01-31T21:27:51.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:51 smithi171 conmon[51620]: debug 2022-01-31T21:27:51.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:51.790002+0000) 2022-01-31T21:27:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:51 smithi167 conmon[49112]: debug 2022-01-31T21:27:51.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:51.766734+0000) 2022-01-31T21:27:52.207 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:51 smithi171 conmon[41853]: debug 2022-01-31T21:27:51.911+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:51.911907+0000) 2022-01-31T21:27:52.486 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:52 smithi167 conmon[54076]: debug 2022-01-31T21:27:52.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:52.432083+0000) 2022-01-31T21:27:52.487 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:52 smithi167 conmon[60316]: debug 2022-01-31T21:27:52.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:52.434648+0000) 2022-01-31T21:27:52.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:52 smithi171 conmon[46715]: debug 2022-01-31T21:27:52.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:52.208584+0000) 2022-01-31T21:27:52.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:52 smithi171 conmon[51620]: debug 2022-01-31T21:27:52.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:52.790155+0000) 2022-01-31T21:27:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:52 smithi167 conmon[49112]: debug 2022-01-31T21:27:52.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:52.766917+0000) 2022-01-31T21:27:53.207 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:52 smithi171 conmon[41853]: debug 2022-01-31T21:27:52.911+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:52.912069+0000) 2022-01-31T21:27:53.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:53 smithi167 conmon[60316]: debug 2022-01-31T21:27:53.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:53.434838+0000) 2022-01-31T21:27:53.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:53 smithi167 conmon[54076]: debug 2022-01-31T21:27:53.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:53.432293+0000) 2022-01-31T21:27:53.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:53 smithi171 conmon[46715]: debug 2022-01-31T21:27:53.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:53.208787+0000) 2022-01-31T21:27:53.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:53 smithi171 conmon[35325]: debug 2022-01-31T21:27:53.683+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:27:53.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:53 smithi171 conmon[51620]: debug 2022-01-31T21:27:53.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:53.790360+0000) 2022-01-31T21:27:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:53 smithi167 conmon[49112]: debug 2022-01-31T21:27:53.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:53.767085+0000) 2022-01-31T21:27:54.207 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:53 smithi171 conmon[41853]: debug 2022-01-31T21:27:53.911+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:53.912226+0000) 2022-01-31T21:27:54.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:54 smithi167 conmon[54076]: debug 2022-01-31T21:27:54.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:54.432496+0000) 2022-01-31T21:27:54.487 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:54 smithi167 conmon[60316]: debug 2022-01-31T21:27:54.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:54.435021+0000) 2022-01-31T21:27:54.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:54 smithi171 conmon[46715]: debug 2022-01-31T21:27:54.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:54.208979+0000) 2022-01-31T21:27:54.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:54 smithi171 conmon[51620]: debug 2022-01-31T21:27:54.789+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:54.790566+0000) 2022-01-31T21:27:55.046 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:54 smithi167 conmon[49112]: debug 2022-01-31T21:27:54.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:54.767300+0000) 2022-01-31T21:27:55.207 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[35325]: debug 2022-01-31T21:27:55.061+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131844 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:27:55.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[46715]: debug 2022-01-31T21:27:55.047+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.048323+0000) 2022-01-31T21:27:55.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[51620]: debug 2022-01-31T21:27:55.047+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.047749+0000) 2022-01-31T21:27:55.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:54 smithi171 conmon[41853]: debug 2022-01-31T21:27:54.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:54.912435+0000) 2022-01-31T21:27:55.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[41853]: debug 2022-01-31T21:27:55.048+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.048798+0000) 2022-01-31T21:27:55.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:55 smithi167 conmon[49112]: debug 2022-01-31T21:27:55.047+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.047927+0000) 2022-01-31T21:27:55.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:55 smithi167 conmon[54076]: debug 2022-01-31T21:27:55.047+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.047652+0000) 2022-01-31T21:27:55.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:55 smithi167 conmon[60316]: debug 2022-01-31T21:27:55.048+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.048820+0000) 2022-01-31T21:27:55.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[46715]: debug 2022-01-31T21:27:55.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.209156+0000) 2022-01-31T21:27:55.751 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:55 smithi167 conmon[54076]: debug 2022-01-31T21:27:55.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.432683+0000) 2022-01-31T21:27:55.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:55 smithi167 conmon[60316]: debug 2022-01-31T21:27:55.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.435221+0000) 2022-01-31T21:27:55.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[51620]: debug 2022-01-31T21:27:55.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.790792+0000) 2022-01-31T21:27:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:55 smithi167 conmon[49112]: debug 2022-01-31T21:27:55.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.767532+0000) 2022-01-31T21:27:56.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:55 smithi171 conmon[41853]: debug 2022-01-31T21:27:55.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:55.912602+0000) 2022-01-31T21:27:56.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:56 smithi167 conmon[54076]: debug 2022-01-31T21:27:56.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:56.432846+0000) 2022-01-31T21:27:56.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:56 smithi167 conmon[60316]: debug 2022-01-31T21:27:56.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:56.435379+0000) 2022-01-31T21:27:56.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:56 smithi171 conmon[46715]: debug 2022-01-31T21:27:56.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:56.209330+0000) 2022-01-31T21:27:56.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:56 smithi171 conmon[51620]: debug 2022-01-31T21:27:56.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:56.790968+0000) 2022-01-31T21:27:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:56 smithi167 conmon[49112]: debug 2022-01-31T21:27:56.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:56.767606+0000) 2022-01-31T21:27:57.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:56 smithi171 conmon[41853]: debug 2022-01-31T21:27:56.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:56.912758+0000) 2022-01-31T21:27:57.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:57 smithi167 conmon[54076]: debug 2022-01-31T21:27:57.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:57.433046+0000) 2022-01-31T21:27:57.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:57 smithi167 conmon[60316]: debug 2022-01-31T21:27:57.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:57.435582+0000) 2022-01-31T21:27:57.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:57 smithi171 conmon[46715]: debug 2022-01-31T21:27:57.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:57.209509+0000) 2022-01-31T21:27:57.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:57 smithi171 conmon[51620]: debug 2022-01-31T21:27:57.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:57.791120+0000) 2022-01-31T21:27:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:57 smithi167 conmon[49112]: debug 2022-01-31T21:27:57.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:57.767804+0000) 2022-01-31T21:27:58.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:57 smithi171 conmon[41853]: debug 2022-01-31T21:27:57.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:57.912914+0000) 2022-01-31T21:27:58.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:58 smithi167 conmon[54076]: debug 2022-01-31T21:27:58.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:58.433216+0000) 2022-01-31T21:27:58.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:58 smithi167 conmon[60316]: debug 2022-01-31T21:27:58.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:58.435731+0000) 2022-01-31T21:27:58.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:58 smithi171 conmon[46715]: debug 2022-01-31T21:27:58.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:58.209630+0000) 2022-01-31T21:27:58.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:58 smithi171 conmon[51620]: debug 2022-01-31T21:27:58.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:58.791302+0000) 2022-01-31T21:27:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:58 smithi167 conmon[49112]: debug 2022-01-31T21:27:58.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:58.768005+0000) 2022-01-31T21:27:59.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:58 smithi171 conmon[41853]: debug 2022-01-31T21:27:58.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:58.913089+0000) 2022-01-31T21:27:59.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:27:59 smithi167 conmon[54076]: debug 2022-01-31T21:27:59.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:59.433439+0000) 2022-01-31T21:27:59.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:27:59 smithi167 conmon[60316]: debug 2022-01-31T21:27:59.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:59.435876+0000) 2022-01-31T21:27:59.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:27:59 smithi171 conmon[46715]: debug 2022-01-31T21:27:59.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:59.209792+0000) 2022-01-31T21:27:59.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:27:59 smithi171 conmon[51620]: debug 2022-01-31T21:27:59.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:59.791491+0000) 2022-01-31T21:28:00.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:27:59 smithi167 conmon[49112]: debug 2022-01-31T21:27:59.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:59.768224+0000) 2022-01-31T21:28:00.208 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[35325]: debug 2022-01-31T21:28:00.082+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 131956 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:00.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[46715]: debug 2022-01-31T21:28:00.064+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.065165+0000) 2022-01-31T21:28:00.210 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[51620]: debug 2022-01-31T21:28:00.064+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.064781+0000) 2022-01-31T21:28:00.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:27:59 smithi171 conmon[41853]: debug 2022-01-31T21:27:59.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:27:59.913279+0000) 2022-01-31T21:28:00.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[41853]: debug 2022-01-31T21:28:00.063+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.064610+0000) 2022-01-31T21:28:00.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:00 smithi167 conmon[49112]: debug 2022-01-31T21:28:00.070+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.071848+0000) 2022-01-31T21:28:00.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:00 smithi167 conmon[54076]: debug 2022-01-31T21:28:00.063+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.065004+0000) 2022-01-31T21:28:00.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:00 smithi167 conmon[60316]: debug 2022-01-31T21:28:00.070+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.072261+0000) 2022-01-31T21:28:00.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[46715]: debug 2022-01-31T21:28:00.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.209985+0000) 2022-01-31T21:28:00.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:00 smithi167 conmon[60316]: debug 2022-01-31T21:28:00.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.436081+0000) 2022-01-31T21:28:00.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:00 smithi167 conmon[54076]: debug 2022-01-31T21:28:00.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.433584+0000) 2022-01-31T21:28:00.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[51620]: debug 2022-01-31T21:28:00.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.791696+0000) 2022-01-31T21:28:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:00 smithi167 conmon[49112]: debug 2022-01-31T21:28:00.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.768399+0000) 2022-01-31T21:28:01.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:00 smithi171 conmon[41853]: debug 2022-01-31T21:28:00.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:00.913446+0000) 2022-01-31T21:28:01.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:01 smithi167 conmon[54076]: debug 2022-01-31T21:28:01.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:01.433732+0000) 2022-01-31T21:28:01.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:01 smithi167 conmon[60316]: debug 2022-01-31T21:28:01.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:01.436257+0000) 2022-01-31T21:28:01.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:01 smithi171 conmon[46715]: debug 2022-01-31T21:28:01.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:01.210105+0000) 2022-01-31T21:28:01.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:01 smithi171 conmon[51620]: debug 2022-01-31T21:28:01.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:01.791862+0000) 2022-01-31T21:28:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:01 smithi167 conmon[49112]: debug 2022-01-31T21:28:01.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:01.768580+0000) 2022-01-31T21:28:02.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:01 smithi171 conmon[41853]: debug 2022-01-31T21:28:01.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:01.913610+0000) 2022-01-31T21:28:02.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:02 smithi167 conmon[54076]: debug 2022-01-31T21:28:02.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:02.433939+0000) 2022-01-31T21:28:02.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:02 smithi167 conmon[60316]: debug 2022-01-31T21:28:02.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:02.436480+0000) 2022-01-31T21:28:02.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:02 smithi171 conmon[46715]: debug 2022-01-31T21:28:02.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:02.210295+0000) 2022-01-31T21:28:02.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:02 smithi171 conmon[51620]: debug 2022-01-31T21:28:02.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:02.791973+0000) 2022-01-31T21:28:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:02 smithi167 conmon[49112]: debug 2022-01-31T21:28:02.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:02.768765+0000) 2022-01-31T21:28:03.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:02 smithi171 conmon[41853]: debug 2022-01-31T21:28:02.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:02.913822+0000) 2022-01-31T21:28:03.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:03 smithi167 conmon[54076]: debug 2022-01-31T21:28:03.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:03.434120+0000) 2022-01-31T21:28:03.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:03 smithi167 conmon[60316]: debug 2022-01-31T21:28:03.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:03.436649+0000) 2022-01-31T21:28:03.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:03 smithi171 conmon[46715]: debug 2022-01-31T21:28:03.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:03.210463+0000) 2022-01-31T21:28:03.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:03 smithi171 conmon[51620]: debug 2022-01-31T21:28:03.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:03.792130+0000) 2022-01-31T21:28:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:03 smithi167 conmon[49112]: debug 2022-01-31T21:28:03.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:03.768948+0000) 2022-01-31T21:28:04.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:03 smithi171 conmon[41853]: debug 2022-01-31T21:28:03.912+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:03.914023+0000) 2022-01-31T21:28:04.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:04 smithi167 conmon[54076]: debug 2022-01-31T21:28:04.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:04.434333+0000) 2022-01-31T21:28:04.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:04 smithi167 conmon[60316]: debug 2022-01-31T21:28:04.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:04.436817+0000) 2022-01-31T21:28:04.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:04 smithi171 conmon[46715]: debug 2022-01-31T21:28:04.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:04.210644+0000) 2022-01-31T21:28:04.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:04 smithi171 conmon[51620]: debug 2022-01-31T21:28:04.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:04.792335+0000) 2022-01-31T21:28:05.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:04 smithi167 conmon[49112]: debug 2022-01-31T21:28:04.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:04.769112+0000) 2022-01-31T21:28:05.209 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[35325]: debug 2022-01-31T21:28:05.097+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132065 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:05.210 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[46715]: debug 2022-01-31T21:28:05.084+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.086260+0000) 2022-01-31T21:28:05.211 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[51620]: debug 2022-01-31T21:28:05.083+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.085001+0000) 2022-01-31T21:28:05.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:04 smithi171 conmon[41853]: debug 2022-01-31T21:28:04.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:04.914207+0000) 2022-01-31T21:28:05.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[41853]: debug 2022-01-31T21:28:05.083+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.085453+0000) 2022-01-31T21:28:05.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:05 smithi167 conmon[49112]: debug 2022-01-31T21:28:05.085+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.087038+0000) 2022-01-31T21:28:05.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:05 smithi167 conmon[54076]: debug 2022-01-31T21:28:05.084+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.085831+0000) 2022-01-31T21:28:05.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:05 smithi167 conmon[60316]: debug 2022-01-31T21:28:05.084+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.085984+0000) 2022-01-31T21:28:05.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[46715]: debug 2022-01-31T21:28:05.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.210802+0000) 2022-01-31T21:28:05.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:05 smithi167 conmon[54076]: debug 2022-01-31T21:28:05.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.434580+0000) 2022-01-31T21:28:05.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:05 smithi167 conmon[60316]: debug 2022-01-31T21:28:05.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.436998+0000) 2022-01-31T21:28:05.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[51620]: debug 2022-01-31T21:28:05.790+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.792561+0000) 2022-01-31T21:28:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:05 smithi167 conmon[49112]: debug 2022-01-31T21:28:05.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.769279+0000) 2022-01-31T21:28:06.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:05 smithi171 conmon[41853]: debug 2022-01-31T21:28:05.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:05.914384+0000) 2022-01-31T21:28:06.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:06 smithi167 conmon[54076]: debug 2022-01-31T21:28:06.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:06.434748+0000) 2022-01-31T21:28:06.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:06 smithi167 conmon[60316]: debug 2022-01-31T21:28:06.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:06.437173+0000) 2022-01-31T21:28:06.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:06 smithi171 conmon[46715]: debug 2022-01-31T21:28:06.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:06.210953+0000) 2022-01-31T21:28:06.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:06 smithi171 conmon[51620]: debug 2022-01-31T21:28:06.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:06.792728+0000) 2022-01-31T21:28:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:06 smithi167 conmon[49112]: debug 2022-01-31T21:28:06.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:06.769432+0000) 2022-01-31T21:28:07.209 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:06 smithi171 conmon[41853]: debug 2022-01-31T21:28:06.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:06.914526+0000) 2022-01-31T21:28:07.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:07 smithi167 conmon[54076]: debug 2022-01-31T21:28:07.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:07.434912+0000) 2022-01-31T21:28:07.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:07 smithi167 conmon[60316]: debug 2022-01-31T21:28:07.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:07.437382+0000) 2022-01-31T21:28:07.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:07 smithi171 conmon[46715]: debug 2022-01-31T21:28:07.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:07.211110+0000) 2022-01-31T21:28:07.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:07 smithi171 conmon[51620]: debug 2022-01-31T21:28:07.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:07.792854+0000) 2022-01-31T21:28:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:07 smithi167 conmon[49112]: debug 2022-01-31T21:28:07.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:07.769597+0000) 2022-01-31T21:28:08.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:07 smithi171 conmon[41853]: debug 2022-01-31T21:28:07.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:07.914729+0000) 2022-01-31T21:28:08.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:08 smithi167 conmon[54076]: debug 2022-01-31T21:28:08.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:08.435073+0000) 2022-01-31T21:28:08.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:08 smithi167 conmon[60316]: debug 2022-01-31T21:28:08.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:08.437561+0000) 2022-01-31T21:28:08.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:08 smithi171 conmon[46715]: debug 2022-01-31T21:28:08.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:08.211296+0000) 2022-01-31T21:28:08.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:08 smithi171 conmon[35325]: debug 2022-01-31T21:28:08.683+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:28:08.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:08 smithi171 conmon[51620]: debug 2022-01-31T21:28:08.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:08.792961+0000) 2022-01-31T21:28:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:08 smithi167 conmon[49112]: debug 2022-01-31T21:28:08.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:08.769715+0000) 2022-01-31T21:28:09.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:08 smithi171 conmon[41853]: debug 2022-01-31T21:28:08.913+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:08.914945+0000) 2022-01-31T21:28:09.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:09 smithi167 conmon[54076]: debug 2022-01-31T21:28:09.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:09.435293+0000) 2022-01-31T21:28:09.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:09 smithi167 conmon[60316]: debug 2022-01-31T21:28:09.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:09.437692+0000) 2022-01-31T21:28:09.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:09 smithi171 conmon[46715]: debug 2022-01-31T21:28:09.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:09.211492+0000) 2022-01-31T21:28:09.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:09 smithi171 conmon[51620]: debug 2022-01-31T21:28:09.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:09.793100+0000) 2022-01-31T21:28:10.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:09 smithi167 conmon[49112]: debug 2022-01-31T21:28:09.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:09.769928+0000) 2022-01-31T21:28:10.210 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[35325]: debug 2022-01-31T21:28:10.114+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132178 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:10.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[46715]: debug 2022-01-31T21:28:10.100+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.101811+0000) 2022-01-31T21:28:10.212 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[51620]: debug 2022-01-31T21:28:10.099+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.101373+0000) 2022-01-31T21:28:10.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:09 smithi171 conmon[41853]: debug 2022-01-31T21:28:09.914+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:09.915125+0000) 2022-01-31T21:28:10.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[41853]: debug 2022-01-31T21:28:10.100+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.101918+0000) 2022-01-31T21:28:10.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:10 smithi167 conmon[49112]: debug 2022-01-31T21:28:10.101+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.103474+0000) 2022-01-31T21:28:10.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:10 smithi167 conmon[54076]: debug 2022-01-31T21:28:10.100+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.102006+0000) 2022-01-31T21:28:10.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:10 smithi167 conmon[60316]: debug 2022-01-31T21:28:10.099+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.101275+0000) 2022-01-31T21:28:10.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[46715]: debug 2022-01-31T21:28:10.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.211669+0000) 2022-01-31T21:28:10.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:10 smithi167 conmon[60316]: debug 2022-01-31T21:28:10.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.437806+0000) 2022-01-31T21:28:10.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:10 smithi167 conmon[54076]: debug 2022-01-31T21:28:10.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.435530+0000) 2022-01-31T21:28:10.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[51620]: debug 2022-01-31T21:28:10.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.793283+0000) 2022-01-31T21:28:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:10 smithi167 conmon[49112]: debug 2022-01-31T21:28:10.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.770140+0000) 2022-01-31T21:28:11.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:10 smithi171 conmon[41853]: debug 2022-01-31T21:28:10.914+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:10.915349+0000) 2022-01-31T21:28:11.490 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:11 smithi167 conmon[54076]: debug 2022-01-31T21:28:11.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:11.435700+0000) 2022-01-31T21:28:11.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:11 smithi167 conmon[60316]: debug 2022-01-31T21:28:11.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:11.437973+0000) 2022-01-31T21:28:11.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:11 smithi171 conmon[46715]: debug 2022-01-31T21:28:11.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:11.211829+0000) 2022-01-31T21:28:11.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:11 smithi171 conmon[51620]: debug 2022-01-31T21:28:11.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:11.793397+0000) 2022-01-31T21:28:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:11 smithi167 conmon[49112]: debug 2022-01-31T21:28:11.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:11.770350+0000) 2022-01-31T21:28:12.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:11 smithi171 conmon[41853]: debug 2022-01-31T21:28:11.914+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:11.915507+0000) 2022-01-31T21:28:12.490 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:12 smithi167 conmon[54076]: debug 2022-01-31T21:28:12.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:12.435870+0000) 2022-01-31T21:28:12.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:12 smithi167 conmon[60316]: debug 2022-01-31T21:28:12.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:12.438123+0000) 2022-01-31T21:28:12.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:12 smithi171 conmon[46715]: debug 2022-01-31T21:28:12.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:12.212022+0000) 2022-01-31T21:28:12.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:12 smithi171 conmon[51620]: debug 2022-01-31T21:28:12.791+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:12.793501+0000) 2022-01-31T21:28:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:12 smithi167 conmon[49112]: debug 2022-01-31T21:28:12.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:12.770564+0000) 2022-01-31T21:28:13.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:12 smithi171 conmon[41853]: debug 2022-01-31T21:28:12.914+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:12.915709+0000) 2022-01-31T21:28:13.490 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:13 smithi167 conmon[54076]: debug 2022-01-31T21:28:13.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:13.436024+0000) 2022-01-31T21:28:13.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:13 smithi167 conmon[60316]: debug 2022-01-31T21:28:13.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:13.438323+0000) 2022-01-31T21:28:13.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:13 smithi171 conmon[46715]: debug 2022-01-31T21:28:13.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:13.212190+0000) 2022-01-31T21:28:13.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:13 smithi171 conmon[51620]: debug 2022-01-31T21:28:13.792+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:13.793836+0000) 2022-01-31T21:28:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:13 smithi167 conmon[49112]: debug 2022-01-31T21:28:13.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:13.770731+0000) 2022-01-31T21:28:14.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:13 smithi171 conmon[41853]: debug 2022-01-31T21:28:13.915+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:13.915916+0000) 2022-01-31T21:28:14.490 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:14 smithi167 conmon[54076]: debug 2022-01-31T21:28:14.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:14.436221+0000) 2022-01-31T21:28:14.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:14 smithi167 conmon[60316]: debug 2022-01-31T21:28:14.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:14.438521+0000) 2022-01-31T21:28:14.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:14 smithi171 conmon[46715]: debug 2022-01-31T21:28:14.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:14.212381+0000) 2022-01-31T21:28:14.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:14 smithi171 conmon[51620]: debug 2022-01-31T21:28:14.792+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:14.793987+0000) 2022-01-31T21:28:15.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:14 smithi167 conmon[49112]: debug 2022-01-31T21:28:14.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:14.770916+0000) 2022-01-31T21:28:15.132 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[46715]: debug 2022-01-31T21:28:15.118+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.119950+0000) 2022-01-31T21:28:15.132 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[51620]: debug 2022-01-31T21:28:15.116+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.118302+0000) 2022-01-31T21:28:15.133 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:14 smithi171 conmon[41853]: debug 2022-01-31T21:28:14.915+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:14.916099+0000) 2022-01-31T21:28:15.133 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[41853]: debug 2022-01-31T21:28:15.117+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.119436+0000) 2022-01-31T21:28:15.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:15 smithi167 conmon[49112]: debug 2022-01-31T21:28:15.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.120551+0000) 2022-01-31T21:28:15.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:15 smithi167 conmon[54076]: debug 2022-01-31T21:28:15.117+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.118880+0000) 2022-01-31T21:28:15.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:15 smithi167 conmon[60316]: debug 2022-01-31T21:28:15.118+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.119547+0000) 2022-01-31T21:28:15.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[35325]: debug 2022-01-31T21:28:15.132+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132287 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:15.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[46715]: debug 2022-01-31T21:28:15.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.212568+0000) 2022-01-31T21:28:15.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:15 smithi167 conmon[54076]: debug 2022-01-31T21:28:15.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.436428+0000) 2022-01-31T21:28:15.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:15 smithi167 conmon[60316]: debug 2022-01-31T21:28:15.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.438678+0000) 2022-01-31T21:28:15.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[51620]: debug 2022-01-31T21:28:15.793+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.794123+0000) 2022-01-31T21:28:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:15 smithi167 conmon[49112]: debug 2022-01-31T21:28:15.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.771093+0000) 2022-01-31T21:28:16.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:15 smithi171 conmon[41853]: debug 2022-01-31T21:28:15.915+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:15.916263+0000) 2022-01-31T21:28:16.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:16 smithi167 conmon[54076]: debug 2022-01-31T21:28:16.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:16.436592+0000) 2022-01-31T21:28:16.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:16 smithi167 conmon[60316]: debug 2022-01-31T21:28:16.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:16.438809+0000) 2022-01-31T21:28:16.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:16 smithi171 conmon[46715]: debug 2022-01-31T21:28:16.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:16.212705+0000) 2022-01-31T21:28:16.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:16 smithi171 conmon[51620]: debug 2022-01-31T21:28:16.793+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:16.794244+0000) 2022-01-31T21:28:17.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:16 smithi167 conmon[49112]: debug 2022-01-31T21:28:16.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:16.771275+0000) 2022-01-31T21:28:17.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:16 smithi171 conmon[41853]: debug 2022-01-31T21:28:16.915+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:16.916412+0000) 2022-01-31T21:28:17.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:17 smithi167 conmon[54076]: debug 2022-01-31T21:28:17.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:17.436743+0000) 2022-01-31T21:28:17.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:17 smithi167 conmon[60316]: debug 2022-01-31T21:28:17.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:17.438936+0000) 2022-01-31T21:28:17.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:17 smithi171 conmon[46715]: debug 2022-01-31T21:28:17.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:17.212905+0000) 2022-01-31T21:28:17.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:17 smithi171 conmon[51620]: debug 2022-01-31T21:28:17.793+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:17.794412+0000) 2022-01-31T21:28:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:17 smithi167 conmon[49112]: debug 2022-01-31T21:28:17.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:17.771465+0000) 2022-01-31T21:28:18.211 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:17 smithi171 conmon[41853]: debug 2022-01-31T21:28:17.915+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:17.916563+0000) 2022-01-31T21:28:18.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:18 smithi167 conmon[54076]: debug 2022-01-31T21:28:18.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:18.436909+0000) 2022-01-31T21:28:18.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:18 smithi167 conmon[60316]: debug 2022-01-31T21:28:18.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:18.439128+0000) 2022-01-31T21:28:18.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:18 smithi171 conmon[46715]: debug 2022-01-31T21:28:18.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:18.213130+0000) 2022-01-31T21:28:18.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:18 smithi171 conmon[51620]: debug 2022-01-31T21:28:18.793+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:18.794604+0000) 2022-01-31T21:28:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:18 smithi167 conmon[49112]: debug 2022-01-31T21:28:18.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:18.771638+0000) 2022-01-31T21:28:19.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:18 smithi171 conmon[41853]: debug 2022-01-31T21:28:18.916+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:18.916746+0000) 2022-01-31T21:28:19.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:19 smithi167 conmon[54076]: debug 2022-01-31T21:28:19.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:19.437089+0000) 2022-01-31T21:28:19.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:19 smithi167 conmon[60316]: debug 2022-01-31T21:28:19.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:19.439302+0000) 2022-01-31T21:28:19.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:19 smithi171 conmon[46715]: debug 2022-01-31T21:28:19.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:19.213302+0000) 2022-01-31T21:28:19.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:19 smithi171 conmon[51620]: debug 2022-01-31T21:28:19.794+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:19.794772+0000) 2022-01-31T21:28:20.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:19 smithi167 conmon[49112]: debug 2022-01-31T21:28:19.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:19.771847+0000) 2022-01-31T21:28:20.134 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:19 smithi171 conmon[41853]: debug 2022-01-31T21:28:19.916+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:19.916968+0000) 2022-01-31T21:28:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:20 smithi167 conmon[54076]: debug 2022-01-31T21:28:20.135+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.136531+0000) 2022-01-31T21:28:20.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:20 smithi167 conmon[60316]: debug 2022-01-31T21:28:20.134+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.136099+0000) 2022-01-31T21:28:20.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:20 smithi167 conmon[49112]: debug 2022-01-31T21:28:20.135+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.136738+0000) 2022-01-31T21:28:20.601 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[35325]: debug 2022-01-31T21:28:20.149+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132399 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:20.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[41853]: debug 2022-01-31T21:28:20.135+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.136402+0000) 2022-01-31T21:28:20.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[46715]: debug 2022-01-31T21:28:20.135+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.136327+0000) 2022-01-31T21:28:20.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[46715]: debug 2022-01-31T21:28:20.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.213458+0000) 2022-01-31T21:28:20.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[51620]: debug 2022-01-31T21:28:20.134+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.135547+0000) 2022-01-31T21:28:20.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:20 smithi167 conmon[54076]: debug 2022-01-31T21:28:20.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.437274+0000) 2022-01-31T21:28:20.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:20 smithi167 conmon[60316]: debug 2022-01-31T21:28:20.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.439548+0000) 2022-01-31T21:28:20.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[51620]: debug 2022-01-31T21:28:20.794+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.794902+0000) 2022-01-31T21:28:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:20 smithi167 conmon[49112]: debug 2022-01-31T21:28:20.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.772025+0000) 2022-01-31T21:28:21.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:20 smithi171 conmon[41853]: debug 2022-01-31T21:28:20.916+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:20.917167+0000) 2022-01-31T21:28:21.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:21 smithi167 conmon[54076]: debug 2022-01-31T21:28:21.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:21.437447+0000) 2022-01-31T21:28:21.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:21 smithi167 conmon[60316]: debug 2022-01-31T21:28:21.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:21.439632+0000) 2022-01-31T21:28:21.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:21 smithi171 conmon[46715]: debug 2022-01-31T21:28:21.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:21.213619+0000) 2022-01-31T21:28:21.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:21 smithi171 conmon[51620]: debug 2022-01-31T21:28:21.794+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:21.795050+0000) 2022-01-31T21:28:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:21 smithi167 conmon[49112]: debug 2022-01-31T21:28:21.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:21.772167+0000) 2022-01-31T21:28:22.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:21 smithi171 conmon[41853]: debug 2022-01-31T21:28:21.916+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:21.917322+0000) 2022-01-31T21:28:22.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:22 smithi167 conmon[54076]: debug 2022-01-31T21:28:22.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:22.437646+0000) 2022-01-31T21:28:22.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:22 smithi167 conmon[60316]: debug 2022-01-31T21:28:22.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:22.439760+0000) 2022-01-31T21:28:22.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:22 smithi171 conmon[46715]: debug 2022-01-31T21:28:22.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:22.213800+0000) 2022-01-31T21:28:22.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:22 smithi171 conmon[51620]: debug 2022-01-31T21:28:22.794+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:22.795212+0000) 2022-01-31T21:28:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:22 smithi167 conmon[49112]: debug 2022-01-31T21:28:22.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:22.772318+0000) 2022-01-31T21:28:23.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:22 smithi171 conmon[41853]: debug 2022-01-31T21:28:22.916+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:22.917546+0000) 2022-01-31T21:28:23.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:23 smithi167 conmon[54076]: debug 2022-01-31T21:28:23.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:23.437831+0000) 2022-01-31T21:28:23.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:23 smithi167 conmon[60316]: debug 2022-01-31T21:28:23.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:23.439941+0000) 2022-01-31T21:28:23.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:23 smithi171 conmon[46715]: debug 2022-01-31T21:28:23.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:23.214007+0000) 2022-01-31T21:28:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:23 smithi171 conmon[35325]: debug 2022-01-31T21:28:23.684+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:28:23.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:23 smithi171 conmon[51620]: debug 2022-01-31T21:28:23.794+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:23.795400+0000) 2022-01-31T21:28:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:23 smithi167 conmon[49112]: debug 2022-01-31T21:28:23.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:23.772476+0000) 2022-01-31T21:28:24.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:23 smithi171 conmon[41853]: debug 2022-01-31T21:28:23.917+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:23.917744+0000) 2022-01-31T21:28:24.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:24 smithi167 conmon[54076]: debug 2022-01-31T21:28:24.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:24.437998+0000) 2022-01-31T21:28:24.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:24 smithi167 conmon[60316]: debug 2022-01-31T21:28:24.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:24.440126+0000) 2022-01-31T21:28:24.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:24 smithi171 conmon[46715]: debug 2022-01-31T21:28:24.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:24.214174+0000) 2022-01-31T21:28:24.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:24 smithi171 conmon[51620]: debug 2022-01-31T21:28:24.794+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:24.795592+0000) 2022-01-31T21:28:25.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:24 smithi167 conmon[49112]: debug 2022-01-31T21:28:24.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:24.772634+0000) 2022-01-31T21:28:25.151 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:24 smithi171 conmon[41853]: debug 2022-01-31T21:28:24.917+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:24.917942+0000) 2022-01-31T21:28:25.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:25 smithi167 conmon[49112]: debug 2022-01-31T21:28:25.152+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.153933+0000) 2022-01-31T21:28:25.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:25 smithi167 conmon[54076]: debug 2022-01-31T21:28:25.152+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.154139+0000) 2022-01-31T21:28:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:25 smithi167 conmon[60316]: debug 2022-01-31T21:28:25.152+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.153805+0000) 2022-01-31T21:28:25.602 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[35325]: debug 2022-01-31T21:28:25.165+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132518 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:25.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[46715]: debug 2022-01-31T21:28:25.153+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.154169+0000) 2022-01-31T21:28:25.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[46715]: debug 2022-01-31T21:28:25.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.214351+0000) 2022-01-31T21:28:25.604 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[41853]: debug 2022-01-31T21:28:25.152+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.153517+0000) 2022-01-31T21:28:25.605 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[51620]: debug 2022-01-31T21:28:25.151+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.152489+0000) 2022-01-31T21:28:25.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:25 smithi167 conmon[54076]: debug 2022-01-31T21:28:25.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.438188+0000) 2022-01-31T21:28:25.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:25 smithi167 conmon[60316]: debug 2022-01-31T21:28:25.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.440274+0000) 2022-01-31T21:28:25.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[51620]: debug 2022-01-31T21:28:25.795+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.795796+0000) 2022-01-31T21:28:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:25 smithi167 conmon[49112]: debug 2022-01-31T21:28:25.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.772814+0000) 2022-01-31T21:28:26.213 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:25 smithi171 conmon[41853]: debug 2022-01-31T21:28:25.917+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:25.918138+0000) 2022-01-31T21:28:26.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:26 smithi167 conmon[54076]: debug 2022-01-31T21:28:26.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:26.438356+0000) 2022-01-31T21:28:26.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:26 smithi167 conmon[60316]: debug 2022-01-31T21:28:26.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:26.440420+0000) 2022-01-31T21:28:26.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:26 smithi171 conmon[46715]: debug 2022-01-31T21:28:26.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:26.214564+0000) 2022-01-31T21:28:26.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:26 smithi171 conmon[51620]: debug 2022-01-31T21:28:26.795+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:26.795964+0000) 2022-01-31T21:28:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:26 smithi167 conmon[49112]: debug 2022-01-31T21:28:26.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:26.772965+0000) 2022-01-31T21:28:27.213 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:26 smithi171 conmon[41853]: debug 2022-01-31T21:28:26.917+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:26.918327+0000) 2022-01-31T21:28:27.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:27 smithi167 conmon[54076]: debug 2022-01-31T21:28:27.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:27.438567+0000) 2022-01-31T21:28:27.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:27 smithi167 conmon[60316]: debug 2022-01-31T21:28:27.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:27.440599+0000) 2022-01-31T21:28:27.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:27 smithi171 conmon[46715]: debug 2022-01-31T21:28:27.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:27.214724+0000) 2022-01-31T21:28:27.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:27 smithi171 conmon[51620]: debug 2022-01-31T21:28:27.795+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:27.796179+0000) 2022-01-31T21:28:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:27 smithi167 conmon[49112]: debug 2022-01-31T21:28:27.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:27.773146+0000) 2022-01-31T21:28:28.213 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:27 smithi171 conmon[41853]: debug 2022-01-31T21:28:27.917+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:27.918537+0000) 2022-01-31T21:28:28.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:28 smithi167 conmon[54076]: debug 2022-01-31T21:28:28.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:28.438677+0000) 2022-01-31T21:28:28.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:28 smithi167 conmon[60316]: debug 2022-01-31T21:28:28.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:28.440760+0000) 2022-01-31T21:28:28.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:28 smithi171 conmon[46715]: debug 2022-01-31T21:28:28.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:28.214906+0000) 2022-01-31T21:28:28.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:28 smithi171 conmon[51620]: debug 2022-01-31T21:28:28.795+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:28.796372+0000) 2022-01-31T21:28:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:28 smithi167 conmon[49112]: debug 2022-01-31T21:28:28.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:28.773270+0000) 2022-01-31T21:28:29.213 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:28 smithi171 conmon[41853]: debug 2022-01-31T21:28:28.918+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:28.918743+0000) 2022-01-31T21:28:29.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:29 smithi167 conmon[54076]: debug 2022-01-31T21:28:29.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:29.438825+0000) 2022-01-31T21:28:29.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:29 smithi167 conmon[60316]: debug 2022-01-31T21:28:29.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:29.440967+0000) 2022-01-31T21:28:29.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:29 smithi171 conmon[46715]: debug 2022-01-31T21:28:29.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:29.215100+0000) 2022-01-31T21:28:29.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:29 smithi171 conmon[51620]: debug 2022-01-31T21:28:29.795+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:29.796564+0000) 2022-01-31T21:28:30.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:29 smithi167 conmon[49112]: debug 2022-01-31T21:28:29.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:29.773458+0000) 2022-01-31T21:28:30.168 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:29 smithi171 conmon[41853]: debug 2022-01-31T21:28:29.918+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:29.918916+0000) 2022-01-31T21:28:30.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:30 smithi167 conmon[49112]: debug 2022-01-31T21:28:30.167+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.168356+0000) 2022-01-31T21:28:30.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:30 smithi167 conmon[54076]: debug 2022-01-31T21:28:30.169+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.170218+0000) 2022-01-31T21:28:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:30 smithi167 conmon[60316]: debug 2022-01-31T21:28:30.168+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.169008+0000) 2022-01-31T21:28:30.603 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[35325]: debug 2022-01-31T21:28:30.182+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132632 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:30.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[51620]: debug 2022-01-31T21:28:30.168+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.169358+0000) 2022-01-31T21:28:30.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[41853]: debug 2022-01-31T21:28:30.168+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.169417+0000) 2022-01-31T21:28:30.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[46715]: debug 2022-01-31T21:28:30.169+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.170293+0000) 2022-01-31T21:28:30.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[46715]: debug 2022-01-31T21:28:30.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.215285+0000) 2022-01-31T21:28:30.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:30 smithi167 conmon[60316]: debug 2022-01-31T21:28:30.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.441110+0000) 2022-01-31T21:28:30.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:30 smithi167 conmon[54076]: debug 2022-01-31T21:28:30.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.439018+0000) 2022-01-31T21:28:30.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[51620]: debug 2022-01-31T21:28:30.796+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.796798+0000) 2022-01-31T21:28:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:30 smithi167 conmon[49112]: debug 2022-01-31T21:28:30.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.773607+0000) 2022-01-31T21:28:31.213 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:30 smithi171 conmon[41853]: debug 2022-01-31T21:28:30.918+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:30.919083+0000) 2022-01-31T21:28:31.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:31 smithi167 conmon[54076]: debug 2022-01-31T21:28:31.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:31.439187+0000) 2022-01-31T21:28:31.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:31 smithi167 conmon[60316]: debug 2022-01-31T21:28:31.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:31.441244+0000) 2022-01-31T21:28:31.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:31 smithi171 conmon[46715]: debug 2022-01-31T21:28:31.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:31.215428+0000) 2022-01-31T21:28:31.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:31 smithi171 conmon[51620]: debug 2022-01-31T21:28:31.796+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:31.796984+0000) 2022-01-31T21:28:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:31 smithi167 conmon[49112]: debug 2022-01-31T21:28:31.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:31.773745+0000) 2022-01-31T21:28:32.214 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:31 smithi171 conmon[41853]: debug 2022-01-31T21:28:31.918+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:31.919228+0000) 2022-01-31T21:28:32.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:32 smithi167 conmon[60316]: debug 2022-01-31T21:28:32.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:32.441408+0000) 2022-01-31T21:28:32.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:32 smithi167 conmon[54076]: debug 2022-01-31T21:28:32.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:32.439371+0000) 2022-01-31T21:28:32.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:32 smithi171 conmon[46715]: debug 2022-01-31T21:28:32.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:32.215626+0000) 2022-01-31T21:28:32.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:32 smithi171 conmon[51620]: debug 2022-01-31T21:28:32.796+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:32.797187+0000) 2022-01-31T21:28:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:32 smithi167 conmon[49112]: debug 2022-01-31T21:28:32.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:32.773931+0000) 2022-01-31T21:28:33.214 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:32 smithi171 conmon[41853]: debug 2022-01-31T21:28:32.918+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:32.919445+0000) 2022-01-31T21:28:33.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:33 smithi167 conmon[54076]: debug 2022-01-31T21:28:33.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:33.439560+0000) 2022-01-31T21:28:33.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:33 smithi167 conmon[60316]: debug 2022-01-31T21:28:33.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:33.441592+0000) 2022-01-31T21:28:33.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:33 smithi171 conmon[46715]: debug 2022-01-31T21:28:33.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:33.215845+0000) 2022-01-31T21:28:33.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:33 smithi171 conmon[51620]: debug 2022-01-31T21:28:33.796+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:33.797334+0000) 2022-01-31T21:28:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:33 smithi167 conmon[49112]: debug 2022-01-31T21:28:33.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:33.774124+0000) 2022-01-31T21:28:34.214 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:33 smithi171 conmon[41853]: debug 2022-01-31T21:28:33.919+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:33.919655+0000) 2022-01-31T21:28:34.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:34 smithi167 conmon[54076]: debug 2022-01-31T21:28:34.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:34.439684+0000) 2022-01-31T21:28:34.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:34 smithi167 conmon[60316]: debug 2022-01-31T21:28:34.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:34.441750+0000) 2022-01-31T21:28:34.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:34 smithi171 conmon[46715]: debug 2022-01-31T21:28:34.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:34.216007+0000) 2022-01-31T21:28:34.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:34 smithi171 conmon[51620]: debug 2022-01-31T21:28:34.797+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:34.797546+0000) 2022-01-31T21:28:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:34 smithi167 conmon[49112]: debug 2022-01-31T21:28:34.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:34.774335+0000) 2022-01-31T21:28:35.183 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:34 smithi171 conmon[41853]: debug 2022-01-31T21:28:34.919+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:34.919858+0000) 2022-01-31T21:28:35.439 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:35 smithi167 conmon[49112]: debug 2022-01-31T21:28:35.185+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.186265+0000) 2022-01-31T21:28:35.440 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:35 smithi167 conmon[54076]: debug 2022-01-31T21:28:35.186+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.186647+0000) 2022-01-31T21:28:35.440 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:35 smithi167 conmon[60316]: debug 2022-01-31T21:28:35.185+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.186343+0000) 2022-01-31T21:28:35.604 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[35325]: debug 2022-01-31T21:28:35.199+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132743 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[41853]: debug 2022-01-31T21:28:35.184+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.185419+0000) 2022-01-31T21:28:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[46715]: debug 2022-01-31T21:28:35.185+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.185918+0000) 2022-01-31T21:28:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[46715]: debug 2022-01-31T21:28:35.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.216167+0000) 2022-01-31T21:28:35.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[51620]: debug 2022-01-31T21:28:35.184+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.184907+0000) 2022-01-31T21:28:35.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:35 smithi167 conmon[54076]: debug 2022-01-31T21:28:35.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.439874+0000) 2022-01-31T21:28:35.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:35 smithi167 conmon[60316]: debug 2022-01-31T21:28:35.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.441879+0000) 2022-01-31T21:28:35.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[51620]: debug 2022-01-31T21:28:35.797+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.797736+0000) 2022-01-31T21:28:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:35 smithi167 conmon[49112]: debug 2022-01-31T21:28:35.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.774513+0000) 2022-01-31T21:28:36.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:35 smithi171 conmon[41853]: debug 2022-01-31T21:28:35.919+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:35.920060+0000) 2022-01-31T21:28:36.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:36 smithi167 conmon[54076]: debug 2022-01-31T21:28:36.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:36.440036+0000) 2022-01-31T21:28:36.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:36 smithi167 conmon[60316]: debug 2022-01-31T21:28:36.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:36.441986+0000) 2022-01-31T21:28:36.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:36 smithi171 conmon[46715]: debug 2022-01-31T21:28:36.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:36.216256+0000) 2022-01-31T21:28:36.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:36 smithi171 conmon[51620]: debug 2022-01-31T21:28:36.797+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:36.797908+0000) 2022-01-31T21:28:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:36 smithi167 conmon[49112]: debug 2022-01-31T21:28:36.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:36.774623+0000) 2022-01-31T21:28:37.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:36 smithi171 conmon[41853]: debug 2022-01-31T21:28:36.919+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:36.920213+0000) 2022-01-31T21:28:37.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:37 smithi167 conmon[54076]: debug 2022-01-31T21:28:37.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:37.440215+0000) 2022-01-31T21:28:37.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:37 smithi167 conmon[60316]: debug 2022-01-31T21:28:37.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:37.442191+0000) 2022-01-31T21:28:37.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:37 smithi171 conmon[46715]: debug 2022-01-31T21:28:37.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:37.216470+0000) 2022-01-31T21:28:37.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:37 smithi171 conmon[51620]: debug 2022-01-31T21:28:37.797+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:37.798096+0000) 2022-01-31T21:28:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:37 smithi167 conmon[49112]: debug 2022-01-31T21:28:37.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:37.774809+0000) 2022-01-31T21:28:38.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:37 smithi171 conmon[41853]: debug 2022-01-31T21:28:37.919+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:37.920392+0000) 2022-01-31T21:28:38.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:38 smithi167 conmon[54076]: debug 2022-01-31T21:28:38.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:38.440397+0000) 2022-01-31T21:28:38.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:38 smithi167 conmon[60316]: debug 2022-01-31T21:28:38.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:38.442410+0000) 2022-01-31T21:28:38.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:38 smithi171 conmon[46715]: debug 2022-01-31T21:28:38.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:38.216656+0000) 2022-01-31T21:28:38.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:38 smithi171 conmon[35325]: debug 2022-01-31T21:28:38.685+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:28:38.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:38 smithi171 conmon[51620]: debug 2022-01-31T21:28:38.797+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:38.798293+0000) 2022-01-31T21:28:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:38 smithi167 conmon[49112]: debug 2022-01-31T21:28:38.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:38.774996+0000) 2022-01-31T21:28:39.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:38 smithi171 conmon[41853]: debug 2022-01-31T21:28:38.919+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:38.920623+0000) 2022-01-31T21:28:39.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:39 smithi167 conmon[54076]: debug 2022-01-31T21:28:39.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:39.440573+0000) 2022-01-31T21:28:39.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:39 smithi167 conmon[60316]: debug 2022-01-31T21:28:39.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:39.442599+0000) 2022-01-31T21:28:39.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:39 smithi171 conmon[46715]: debug 2022-01-31T21:28:39.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:39.216892+0000) 2022-01-31T21:28:39.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:39 smithi171 conmon[51620]: debug 2022-01-31T21:28:39.798+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:39.798482+0000) 2022-01-31T21:28:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:39 smithi167 conmon[49112]: debug 2022-01-31T21:28:39.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:39.775164+0000) 2022-01-31T21:28:40.200 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:39 smithi171 conmon[41853]: debug 2022-01-31T21:28:39.920+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:39.920797+0000) 2022-01-31T21:28:40.495 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:40 smithi167 conmon[49112]: debug 2022-01-31T21:28:40.201+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.201954+0000) 2022-01-31T21:28:40.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:40 smithi167 conmon[54076]: debug 2022-01-31T21:28:40.203+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.203607+0000) 2022-01-31T21:28:40.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:40 smithi167 conmon[54076]: debug 2022-01-31T21:28:40.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.440731+0000) 2022-01-31T21:28:40.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:40 smithi167 conmon[60316]: debug 2022-01-31T21:28:40.203+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.203896+0000) 2022-01-31T21:28:40.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:40 smithi167 conmon[60316]: debug 2022-01-31T21:28:40.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.442805+0000) 2022-01-31T21:28:40.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[35325]: debug 2022-01-31T21:28:40.216+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132856 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[41853]: debug 2022-01-31T21:28:40.201+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.202283+0000) 2022-01-31T21:28:40.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[51620]: debug 2022-01-31T21:28:40.201+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.201928+0000) 2022-01-31T21:28:40.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[46715]: debug 2022-01-31T21:28:40.201+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.202260+0000) 2022-01-31T21:28:40.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[46715]: debug 2022-01-31T21:28:40.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.217038+0000) 2022-01-31T21:28:40.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[51620]: debug 2022-01-31T21:28:40.798+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.798660+0000) 2022-01-31T21:28:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:40 smithi167 conmon[49112]: debug 2022-01-31T21:28:40.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.775321+0000) 2022-01-31T21:28:41.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:40 smithi171 conmon[41853]: debug 2022-01-31T21:28:40.920+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:40.920927+0000) 2022-01-31T21:28:41.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:41 smithi167 conmon[54076]: debug 2022-01-31T21:28:41.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:41.440853+0000) 2022-01-31T21:28:41.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:41 smithi167 conmon[60316]: debug 2022-01-31T21:28:41.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:41.442958+0000) 2022-01-31T21:28:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:41 smithi171 conmon[46715]: debug 2022-01-31T21:28:41.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:41.217201+0000) 2022-01-31T21:28:41.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:41 smithi171 conmon[51620]: debug 2022-01-31T21:28:41.798+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:41.798835+0000) 2022-01-31T21:28:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:41 smithi167 conmon[49112]: debug 2022-01-31T21:28:41.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:41.775494+0000) 2022-01-31T21:28:42.216 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:41 smithi171 conmon[41853]: debug 2022-01-31T21:28:41.920+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:41.921067+0000) 2022-01-31T21:28:42.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:42 smithi167 conmon[54076]: debug 2022-01-31T21:28:42.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:42.441019+0000) 2022-01-31T21:28:42.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:42 smithi167 conmon[60316]: debug 2022-01-31T21:28:42.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:42.443142+0000) 2022-01-31T21:28:42.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:42 smithi171 conmon[46715]: debug 2022-01-31T21:28:42.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:42.217451+0000) 2022-01-31T21:28:42.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:42 smithi171 conmon[51620]: debug 2022-01-31T21:28:42.798+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:42.799052+0000) 2022-01-31T21:28:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:42 smithi167 conmon[49112]: debug 2022-01-31T21:28:42.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:42.775691+0000) 2022-01-31T21:28:43.216 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:42 smithi171 conmon[41853]: debug 2022-01-31T21:28:42.920+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:42.921246+0000) 2022-01-31T21:28:43.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:43 smithi167 conmon[54076]: debug 2022-01-31T21:28:43.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:43.441213+0000) 2022-01-31T21:28:43.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:43 smithi167 conmon[60316]: debug 2022-01-31T21:28:43.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:43.443351+0000) 2022-01-31T21:28:43.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:43 smithi171 conmon[46715]: debug 2022-01-31T21:28:43.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:43.217672+0000) 2022-01-31T21:28:43.878 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:43 smithi171 conmon[51620]: debug 2022-01-31T21:28:43.798+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:43.799251+0000) 2022-01-31T21:28:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:43 smithi167 conmon[49112]: debug 2022-01-31T21:28:43.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:43.775903+0000) 2022-01-31T21:28:44.216 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:43 smithi171 conmon[41853]: debug 2022-01-31T21:28:43.920+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:43.921419+0000) 2022-01-31T21:28:44.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:44 smithi167 conmon[54076]: debug 2022-01-31T21:28:44.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:44.441409+0000) 2022-01-31T21:28:44.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:44 smithi167 conmon[60316]: debug 2022-01-31T21:28:44.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:44.443585+0000) 2022-01-31T21:28:44.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:44 smithi171 conmon[46715]: debug 2022-01-31T21:28:44.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:44.217910+0000) 2022-01-31T21:28:44.879 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:44 smithi171 conmon[51620]: debug 2022-01-31T21:28:44.798+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:44.799428+0000) 2022-01-31T21:28:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:44 smithi167 conmon[49112]: debug 2022-01-31T21:28:44.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:44.776100+0000) 2022-01-31T21:28:45.217 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:44 smithi171 conmon[41853]: debug 2022-01-31T21:28:44.921+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:44.921600+0000) 2022-01-31T21:28:45.495 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:45 smithi167 conmon[49112]: debug 2022-01-31T21:28:45.219+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.220330+0000) 2022-01-31T21:28:45.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:45 smithi167 conmon[54076]: debug 2022-01-31T21:28:45.220+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.220592+0000) 2022-01-31T21:28:45.497 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:45 smithi167 conmon[54076]: debug 2022-01-31T21:28:45.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.441616+0000) 2022-01-31T21:28:45.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:45 smithi167 conmon[60316]: debug 2022-01-31T21:28:45.219+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.220389+0000) 2022-01-31T21:28:45.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:45 smithi167 conmon[60316]: debug 2022-01-31T21:28:45.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.443789+0000) 2022-01-31T21:28:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:45 smithi171 conmon[35325]: debug 2022-01-31T21:28:45.232+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 132964 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:45.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:45 smithi171 conmon[41853]: debug 2022-01-31T21:28:45.218+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.219423+0000) 2022-01-31T21:28:45.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:45 smithi171 conmon[46715]: debug 2022-01-31T21:28:45.219+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.220713+0000) 2022-01-31T21:28:45.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:45 smithi171 conmon[51620]: debug 2022-01-31T21:28:45.219+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.220287+0000) 2022-01-31T21:28:45.879 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:45 smithi171 conmon[51620]: debug 2022-01-31T21:28:45.799+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.799620+0000) 2022-01-31T21:28:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:45 smithi167 conmon[49112]: debug 2022-01-31T21:28:45.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.776307+0000) 2022-01-31T21:28:46.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:45 smithi171 conmon[41853]: debug 2022-01-31T21:28:45.921+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:45.921792+0000) 2022-01-31T21:28:46.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:46 smithi167 conmon[54076]: debug 2022-01-31T21:28:46.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:46.441726+0000) 2022-01-31T21:28:46.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:46 smithi167 conmon[60316]: debug 2022-01-31T21:28:46.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:46.443939+0000) 2022-01-31T21:28:46.879 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:46 smithi171 conmon[51620]: debug 2022-01-31T21:28:46.799+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:46.799795+0000) 2022-01-31T21:28:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:46 smithi167 conmon[49112]: debug 2022-01-31T21:28:46.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:46.776461+0000) 2022-01-31T21:28:47.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:46 smithi171 conmon[41853]: debug 2022-01-31T21:28:46.921+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:46.921924+0000) 2022-01-31T21:28:47.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:47 smithi167 conmon[54076]: debug 2022-01-31T21:28:47.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:47.441927+0000) 2022-01-31T21:28:47.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:47 smithi167 conmon[60316]: debug 2022-01-31T21:28:47.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:47.444160+0000) 2022-01-31T21:28:47.879 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:47 smithi171 conmon[51620]: debug 2022-01-31T21:28:47.799+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:47.799964+0000) 2022-01-31T21:28:48.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:47 smithi167 conmon[49112]: debug 2022-01-31T21:28:47.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:47.776681+0000) 2022-01-31T21:28:48.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:47 smithi171 conmon[41853]: debug 2022-01-31T21:28:47.921+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:47.922071+0000) 2022-01-31T21:28:48.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:48 smithi167 conmon[54076]: debug 2022-01-31T21:28:48.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:48.442156+0000) 2022-01-31T21:28:48.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:48 smithi167 conmon[60316]: debug 2022-01-31T21:28:48.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:48.444339+0000) 2022-01-31T21:28:48.879 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:48 smithi171 conmon[51620]: debug 2022-01-31T21:28:48.799+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:48.800151+0000) 2022-01-31T21:28:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:48 smithi167 conmon[49112]: debug 2022-01-31T21:28:48.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:48.776850+0000) 2022-01-31T21:28:49.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:48 smithi171 conmon[41853]: debug 2022-01-31T21:28:48.921+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:48.922206+0000) 2022-01-31T21:28:49.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:49 smithi167 conmon[54076]: debug 2022-01-31T21:28:49.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:49.442369+0000) 2022-01-31T21:28:49.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:49 smithi167 conmon[60316]: debug 2022-01-31T21:28:49.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:49.444526+0000) 2022-01-31T21:28:49.879 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:49 smithi171 conmon[51620]: debug 2022-01-31T21:28:49.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:49.800365+0000) 2022-01-31T21:28:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:49 smithi167 conmon[49112]: debug 2022-01-31T21:28:49.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:49.777048+0000) 2022-01-31T21:28:50.234 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:49 smithi171 conmon[41853]: debug 2022-01-31T21:28:49.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:49.922384+0000) 2022-01-31T21:28:50.496 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:50 smithi167 conmon[49112]: debug 2022-01-31T21:28:50.235+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.235933+0000) 2022-01-31T21:28:50.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:50 smithi167 conmon[60316]: debug 2022-01-31T21:28:50.236+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.236628+0000) 2022-01-31T21:28:50.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:50 smithi167 conmon[60316]: debug 2022-01-31T21:28:50.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.444722+0000) 2022-01-31T21:28:50.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:50 smithi167 conmon[54076]: debug 2022-01-31T21:28:50.235+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.236378+0000) 2022-01-31T21:28:50.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:50 smithi167 conmon[54076]: debug 2022-01-31T21:28:50.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.442560+0000) 2022-01-31T21:28:50.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:50 smithi171 conmon[35325]: debug 2022-01-31T21:28:50.249+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133077 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:50 smithi171 conmon[41853]: debug 2022-01-31T21:28:50.235+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.236149+0000) 2022-01-31T21:28:50.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:50 smithi171 conmon[46715]: debug 2022-01-31T21:28:50.236+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.237507+0000) 2022-01-31T21:28:50.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:50 smithi171 conmon[51620]: debug 2022-01-31T21:28:50.236+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.236807+0000) 2022-01-31T21:28:50.880 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:50 smithi171 conmon[51620]: debug 2022-01-31T21:28:50.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.800528+0000) 2022-01-31T21:28:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:50 smithi167 conmon[49112]: debug 2022-01-31T21:28:50.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.777254+0000) 2022-01-31T21:28:51.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:50 smithi171 conmon[41853]: debug 2022-01-31T21:28:50.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:50.922532+0000) 2022-01-31T21:28:51.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:51 smithi167 conmon[54076]: debug 2022-01-31T21:28:51.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:51.442698+0000) 2022-01-31T21:28:51.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:51 smithi167 conmon[60316]: debug 2022-01-31T21:28:51.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:51.444871+0000) 2022-01-31T21:28:51.880 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:51 smithi171 conmon[51620]: debug 2022-01-31T21:28:51.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:51.800688+0000) 2022-01-31T21:28:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:51 smithi167 conmon[49112]: debug 2022-01-31T21:28:51.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:51.777440+0000) 2022-01-31T21:28:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:51 smithi171 conmon[41853]: debug 2022-01-31T21:28:51.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:51.922677+0000) 2022-01-31T21:28:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:51 smithi171 conmon[46715]: debug 2022-01-31T21:28:51.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:51.968392+0000) 2022-01-31T21:28:52.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:52 smithi167 conmon[54076]: debug 2022-01-31T21:28:52.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:52.442850+0000) 2022-01-31T21:28:52.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:52 smithi167 conmon[60316]: debug 2022-01-31T21:28:52.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:52.445049+0000) 2022-01-31T21:28:52.880 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:52 smithi171 conmon[51620]: debug 2022-01-31T21:28:52.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:52.800867+0000) 2022-01-31T21:28:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:52 smithi167 conmon[49112]: debug 2022-01-31T21:28:52.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:52.777678+0000) 2022-01-31T21:28:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:52 smithi171 conmon[41853]: debug 2022-01-31T21:28:52.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:52.922843+0000) 2022-01-31T21:28:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:52 smithi171 conmon[46715]: debug 2022-01-31T21:28:52.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:52.968590+0000) 2022-01-31T21:28:53.497 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:53 smithi167 conmon[54076]: debug 2022-01-31T21:28:53.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:53.443074+0000) 2022-01-31T21:28:53.497 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:53 smithi167 conmon[60316]: debug 2022-01-31T21:28:53.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:53.445276+0000) 2022-01-31T21:28:53.880 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:53 smithi171 conmon[35325]: debug 2022-01-31T21:28:53.686+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:28:53.881 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:53 smithi171 conmon[51620]: debug 2022-01-31T21:28:53.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:53.801071+0000) 2022-01-31T21:28:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:53 smithi167 conmon[49112]: debug 2022-01-31T21:28:53.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:53.777865+0000) 2022-01-31T21:28:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:53 smithi171 conmon[41853]: debug 2022-01-31T21:28:53.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:53.923030+0000) 2022-01-31T21:28:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:53 smithi171 conmon[46715]: debug 2022-01-31T21:28:53.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:53.968763+0000) 2022-01-31T21:28:54.497 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:54 smithi167 conmon[54076]: debug 2022-01-31T21:28:54.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:54.443287+0000) 2022-01-31T21:28:54.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:54 smithi167 conmon[60316]: debug 2022-01-31T21:28:54.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:54.445487+0000) 2022-01-31T21:28:54.880 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:54 smithi171 conmon[51620]: debug 2022-01-31T21:28:54.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:54.801252+0000) 2022-01-31T21:28:55.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:54 smithi167 conmon[49112]: debug 2022-01-31T21:28:54.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:54.778060+0000) 2022-01-31T21:28:55.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:54 smithi171 conmon[46715]: debug 2022-01-31T21:28:54.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:54.968897+0000) 2022-01-31T21:28:55.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:54 smithi171 conmon[41853]: debug 2022-01-31T21:28:54.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:54.923195+0000) 2022-01-31T21:28:55.497 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:55 smithi167 conmon[49112]: debug 2022-01-31T21:28:55.253+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.253519+0000) 2022-01-31T21:28:55.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:55 smithi167 conmon[54076]: debug 2022-01-31T21:28:55.253+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.253620+0000) 2022-01-31T21:28:55.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:55 smithi167 conmon[54076]: debug 2022-01-31T21:28:55.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.443444+0000) 2022-01-31T21:28:55.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:55 smithi167 conmon[60316]: debug 2022-01-31T21:28:55.253+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.253876+0000) 2022-01-31T21:28:55.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:55 smithi167 conmon[60316]: debug 2022-01-31T21:28:55.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.445664+0000) 2022-01-31T21:28:55.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[35325]: debug 2022-01-31T21:28:55.266+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133185 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:28:55.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[41853]: debug 2022-01-31T21:28:55.251+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.252599+0000) 2022-01-31T21:28:55.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[46715]: debug 2022-01-31T21:28:55.253+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.254173+0000) 2022-01-31T21:28:55.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[51620]: debug 2022-01-31T21:28:55.251+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.252100+0000) 2022-01-31T21:28:55.880 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[51620]: debug 2022-01-31T21:28:55.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.801432+0000) 2022-01-31T21:28:56.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:55 smithi167 conmon[49112]: debug 2022-01-31T21:28:55.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.778224+0000) 2022-01-31T21:28:56.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[41853]: debug 2022-01-31T21:28:55.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.923371+0000) 2022-01-31T21:28:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:55 smithi171 conmon[46715]: debug 2022-01-31T21:28:55.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:55.969035+0000) 2022-01-31T21:28:56.497 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:56 smithi167 conmon[54076]: debug 2022-01-31T21:28:56.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:56.443588+0000) 2022-01-31T21:28:56.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:56 smithi167 conmon[60316]: debug 2022-01-31T21:28:56.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:56.445772+0000) 2022-01-31T21:28:56.881 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:56 smithi171 conmon[51620]: debug 2022-01-31T21:28:56.800+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:56.801601+0000) 2022-01-31T21:28:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:56 smithi167 conmon[49112]: debug 2022-01-31T21:28:56.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:56.778380+0000) 2022-01-31T21:28:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:56 smithi171 conmon[46715]: debug 2022-01-31T21:28:56.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:56.969181+0000) 2022-01-31T21:28:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:56 smithi171 conmon[41853]: debug 2022-01-31T21:28:56.922+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:56.923541+0000) 2022-01-31T21:28:57.497 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:57 smithi167 conmon[54076]: debug 2022-01-31T21:28:57.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:57.443721+0000) 2022-01-31T21:28:57.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:57 smithi167 conmon[60316]: debug 2022-01-31T21:28:57.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:57.445963+0000) 2022-01-31T21:28:57.881 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:57 smithi171 conmon[51620]: debug 2022-01-31T21:28:57.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:57.801770+0000) 2022-01-31T21:28:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:57 smithi167 conmon[49112]: debug 2022-01-31T21:28:57.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:57.778593+0000) 2022-01-31T21:28:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:57 smithi171 conmon[46715]: debug 2022-01-31T21:28:57.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:57.969368+0000) 2022-01-31T21:28:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:57 smithi171 conmon[41853]: debug 2022-01-31T21:28:57.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:57.923724+0000) 2022-01-31T21:28:58.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:58 smithi167 conmon[54076]: debug 2022-01-31T21:28:58.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:58.443868+0000) 2022-01-31T21:28:58.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:58 smithi167 conmon[60316]: debug 2022-01-31T21:28:58.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:58.446142+0000) 2022-01-31T21:28:58.881 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:58 smithi171 conmon[51620]: debug 2022-01-31T21:28:58.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:58.801948+0000) 2022-01-31T21:28:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:58 smithi167 conmon[49112]: debug 2022-01-31T21:28:58.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:58.778779+0000) 2022-01-31T21:28:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:58 smithi171 conmon[41853]: debug 2022-01-31T21:28:58.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:58.923900+0000) 2022-01-31T21:28:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:58 smithi171 conmon[46715]: debug 2022-01-31T21:28:58.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:58.969569+0000) 2022-01-31T21:28:59.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:28:59 smithi167 conmon[54076]: debug 2022-01-31T21:28:59.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:59.444081+0000) 2022-01-31T21:28:59.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:28:59 smithi167 conmon[60316]: debug 2022-01-31T21:28:59.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:59.446346+0000) 2022-01-31T21:28:59.881 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:28:59 smithi171 conmon[51620]: debug 2022-01-31T21:28:59.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:59.802091+0000) 2022-01-31T21:29:00.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:28:59 smithi167 conmon[49112]: debug 2022-01-31T21:28:59.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:59.778975+0000) 2022-01-31T21:29:00.267 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:28:59 smithi171 conmon[41853]: debug 2022-01-31T21:28:59.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:59.924088+0000) 2022-01-31T21:29:00.268 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[41853]: debug 2022-01-31T21:29:00.268+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.268979+0000) 2022-01-31T21:29:00.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:28:59 smithi171 conmon[46715]: debug 2022-01-31T21:28:59.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:28:59.969742+0000) 2022-01-31T21:29:00.498 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:00 smithi167 conmon[49112]: debug 2022-01-31T21:29:00.268+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.269856+0000) 2022-01-31T21:29:00.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:00 smithi167 conmon[54076]: debug 2022-01-31T21:29:00.268+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.269938+0000) 2022-01-31T21:29:00.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:00 smithi167 conmon[54076]: debug 2022-01-31T21:29:00.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.444258+0000) 2022-01-31T21:29:00.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:00 smithi167 conmon[60316]: debug 2022-01-31T21:29:00.268+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.270046+0000) 2022-01-31T21:29:00.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:00 smithi167 conmon[60316]: debug 2022-01-31T21:29:00.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.446582+0000) 2022-01-31T21:29:00.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[35325]: debug 2022-01-31T21:29:00.298+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133298 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:00.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[46715]: debug 2022-01-31T21:29:00.269+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.270734+0000) 2022-01-31T21:29:00.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[51620]: debug 2022-01-31T21:29:00.268+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.269407+0000) 2022-01-31T21:29:00.802 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[51620]: debug 2022-01-31T21:29:00.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.802283+0000) 2022-01-31T21:29:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:00 smithi167 conmon[49112]: debug 2022-01-31T21:29:00.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.779143+0000) 2022-01-31T21:29:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[46715]: debug 2022-01-31T21:29:00.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.969908+0000) 2022-01-31T21:29:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:00 smithi171 conmon[41853]: debug 2022-01-31T21:29:00.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:00.924271+0000) 2022-01-31T21:29:01.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:01 smithi167 conmon[54076]: debug 2022-01-31T21:29:01.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:01.444433+0000) 2022-01-31T21:29:01.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:01 smithi167 conmon[60316]: debug 2022-01-31T21:29:01.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:01.446697+0000) 2022-01-31T21:29:01.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:01 smithi171 conmon[51620]: debug 2022-01-31T21:29:01.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:01.802396+0000) 2022-01-31T21:29:02.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:01 smithi167 conmon[49112]: debug 2022-01-31T21:29:01.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:01.779330+0000) 2022-01-31T21:29:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:01 smithi171 conmon[41853]: debug 2022-01-31T21:29:01.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:01.924444+0000) 2022-01-31T21:29:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:01 smithi171 conmon[46715]: debug 2022-01-31T21:29:01.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:01.970060+0000) 2022-01-31T21:29:02.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:02 smithi167 conmon[60316]: debug 2022-01-31T21:29:02.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:02.446834+0000) 2022-01-31T21:29:02.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:02 smithi167 conmon[54076]: debug 2022-01-31T21:29:02.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:02.444616+0000) 2022-01-31T21:29:02.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:02 smithi171 conmon[51620]: debug 2022-01-31T21:29:02.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:02.802577+0000) 2022-01-31T21:29:03.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:02 smithi167 conmon[49112]: debug 2022-01-31T21:29:02.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:02.779516+0000) 2022-01-31T21:29:03.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:02 smithi171 conmon[41853]: debug 2022-01-31T21:29:02.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:02.924638+0000) 2022-01-31T21:29:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:02 smithi171 conmon[46715]: debug 2022-01-31T21:29:02.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:02.970204+0000) 2022-01-31T21:29:03.498 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:03 smithi167 conmon[60316]: debug 2022-01-31T21:29:03.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:03.446991+0000) 2022-01-31T21:29:03.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:03 smithi167 conmon[54076]: debug 2022-01-31T21:29:03.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:03.444778+0000) 2022-01-31T21:29:03.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:03 smithi171 conmon[51620]: debug 2022-01-31T21:29:03.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:03.802776+0000) 2022-01-31T21:29:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:03 smithi167 conmon[49112]: debug 2022-01-31T21:29:03.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:03.779694+0000) 2022-01-31T21:29:04.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:03 smithi171 conmon[41853]: debug 2022-01-31T21:29:03.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:03.924767+0000) 2022-01-31T21:29:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:03 smithi171 conmon[46715]: debug 2022-01-31T21:29:03.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:03.970368+0000) 2022-01-31T21:29:04.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:04 smithi167 conmon[54076]: debug 2022-01-31T21:29:04.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:04.444928+0000) 2022-01-31T21:29:04.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:04 smithi167 conmon[60316]: debug 2022-01-31T21:29:04.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:04.447204+0000) 2022-01-31T21:29:04.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:04 smithi171 conmon[51620]: debug 2022-01-31T21:29:04.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:04.802955+0000) 2022-01-31T21:29:05.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:04 smithi167 conmon[49112]: debug 2022-01-31T21:29:04.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:04.779900+0000) 2022-01-31T21:29:05.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:04 smithi171 conmon[46715]: debug 2022-01-31T21:29:04.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:04.970548+0000) 2022-01-31T21:29:05.300 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:04 smithi171 conmon[41853]: debug 2022-01-31T21:29:04.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:04.924891+0000) 2022-01-31T21:29:05.499 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:05 smithi167 conmon[49112]: debug 2022-01-31T21:29:05.301+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.303467+0000) 2022-01-31T21:29:05.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:05 smithi167 conmon[54076]: debug 2022-01-31T21:29:05.300+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.301980+0000) 2022-01-31T21:29:05.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:05 smithi167 conmon[54076]: debug 2022-01-31T21:29:05.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.445150+0000) 2022-01-31T21:29:05.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:05 smithi167 conmon[60316]: debug 2022-01-31T21:29:05.301+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.303102+0000) 2022-01-31T21:29:05.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:05 smithi167 conmon[60316]: debug 2022-01-31T21:29:05.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.447394+0000) 2022-01-31T21:29:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[35325]: debug 2022-01-31T21:29:05.314+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133407 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[41853]: debug 2022-01-31T21:29:05.300+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.301887+0000) 2022-01-31T21:29:05.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[46715]: debug 2022-01-31T21:29:05.301+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.302986+0000) 2022-01-31T21:29:05.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[51620]: debug 2022-01-31T21:29:05.301+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.302738+0000) 2022-01-31T21:29:05.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[51620]: debug 2022-01-31T21:29:05.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.803128+0000) 2022-01-31T21:29:06.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:05 smithi167 conmon[49112]: debug 2022-01-31T21:29:05.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.780093+0000) 2022-01-31T21:29:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[41853]: debug 2022-01-31T21:29:05.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.925060+0000) 2022-01-31T21:29:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:05 smithi171 conmon[46715]: debug 2022-01-31T21:29:05.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:05.970677+0000) 2022-01-31T21:29:06.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:06 smithi167 conmon[60316]: debug 2022-01-31T21:29:06.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:06.447577+0000) 2022-01-31T21:29:06.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:06 smithi167 conmon[54076]: debug 2022-01-31T21:29:06.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:06.445315+0000) 2022-01-31T21:29:06.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:06 smithi171 conmon[51620]: debug 2022-01-31T21:29:06.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:06.803236+0000) 2022-01-31T21:29:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:06 smithi167 conmon[49112]: debug 2022-01-31T21:29:06.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:06.780245+0000) 2022-01-31T21:29:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:06 smithi171 conmon[46715]: debug 2022-01-31T21:29:06.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:06.970789+0000) 2022-01-31T21:29:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:06 smithi171 conmon[41853]: debug 2022-01-31T21:29:06.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:06.925204+0000) 2022-01-31T21:29:07.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:07 smithi167 conmon[54076]: debug 2022-01-31T21:29:07.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:07.445535+0000) 2022-01-31T21:29:07.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:07 smithi167 conmon[60316]: debug 2022-01-31T21:29:07.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:07.447741+0000) 2022-01-31T21:29:07.882 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:07 smithi171 conmon[51620]: debug 2022-01-31T21:29:07.801+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:07.803404+0000) 2022-01-31T21:29:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:07 smithi167 conmon[49112]: debug 2022-01-31T21:29:07.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:07.780465+0000) 2022-01-31T21:29:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:07 smithi171 conmon[41853]: debug 2022-01-31T21:29:07.923+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:07.925423+0000) 2022-01-31T21:29:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:07 smithi171 conmon[46715]: debug 2022-01-31T21:29:07.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:07.970950+0000) 2022-01-31T21:29:08.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:08 smithi167 conmon[60316]: debug 2022-01-31T21:29:08.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:08.447872+0000) 2022-01-31T21:29:08.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:08 smithi167 conmon[54076]: debug 2022-01-31T21:29:08.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:08.445705+0000) 2022-01-31T21:29:08.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:08 smithi171 conmon[51620]: debug 2022-01-31T21:29:08.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:08.803594+0000) 2022-01-31T21:29:08.884 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:08 smithi171 conmon[35325]: debug 2022-01-31T21:29:08.686+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:29:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:08 smithi167 conmon[49112]: debug 2022-01-31T21:29:08.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:08.780649+0000) 2022-01-31T21:29:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:08 smithi171 conmon[41853]: debug 2022-01-31T21:29:08.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:08.925586+0000) 2022-01-31T21:29:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:08 smithi171 conmon[46715]: debug 2022-01-31T21:29:08.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:08.971108+0000) 2022-01-31T21:29:09.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:09 smithi167 conmon[54076]: debug 2022-01-31T21:29:09.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:09.445870+0000) 2022-01-31T21:29:09.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:09 smithi167 conmon[60316]: debug 2022-01-31T21:29:09.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:09.448053+0000) 2022-01-31T21:29:09.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:09 smithi171 conmon[51620]: debug 2022-01-31T21:29:09.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:09.803779+0000) 2022-01-31T21:29:10.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:09 smithi167 conmon[49112]: debug 2022-01-31T21:29:09.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:09.780831+0000) 2022-01-31T21:29:10.316 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:09 smithi171 conmon[46715]: debug 2022-01-31T21:29:09.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:09.971285+0000) 2022-01-31T21:29:10.317 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:09 smithi171 conmon[41853]: debug 2022-01-31T21:29:09.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:09.925739+0000) 2022-01-31T21:29:10.500 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:10 smithi167 conmon[49112]: debug 2022-01-31T21:29:10.317+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.318947+0000) 2022-01-31T21:29:10.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:10 smithi167 conmon[54076]: debug 2022-01-31T21:29:10.317+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.319194+0000) 2022-01-31T21:29:10.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:10 smithi167 conmon[54076]: debug 2022-01-31T21:29:10.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.446033+0000) 2022-01-31T21:29:10.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:10 smithi167 conmon[60316]: debug 2022-01-31T21:29:10.318+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.319950+0000) 2022-01-31T21:29:10.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:10 smithi167 conmon[60316]: debug 2022-01-31T21:29:10.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.448255+0000) 2022-01-31T21:29:10.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[46715]: debug 2022-01-31T21:29:10.318+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.320093+0000) 2022-01-31T21:29:10.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[51620]: debug 2022-01-31T21:29:10.317+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.319342+0000) 2022-01-31T21:29:10.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[35325]: debug 2022-01-31T21:29:10.331+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133521 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:10.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[41853]: debug 2022-01-31T21:29:10.317+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.319457+0000) 2022-01-31T21:29:10.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[51620]: debug 2022-01-31T21:29:10.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.803905+0000) 2022-01-31T21:29:11.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:10 smithi167 conmon[49112]: debug 2022-01-31T21:29:10.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.781027+0000) 2022-01-31T21:29:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[41853]: debug 2022-01-31T21:29:10.925+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.925953+0000) 2022-01-31T21:29:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:10 smithi171 conmon[46715]: debug 2022-01-31T21:29:10.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:10.971439+0000) 2022-01-31T21:29:11.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:11 smithi167 conmon[54076]: debug 2022-01-31T21:29:11.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:11.446187+0000) 2022-01-31T21:29:11.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:11 smithi167 conmon[60316]: debug 2022-01-31T21:29:11.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:11.448411+0000) 2022-01-31T21:29:11.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:11 smithi171 conmon[51620]: debug 2022-01-31T21:29:11.803+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:11.804038+0000) 2022-01-31T21:29:12.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:11 smithi167 conmon[49112]: debug 2022-01-31T21:29:11.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:11.781181+0000) 2022-01-31T21:29:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:11 smithi171 conmon[41853]: debug 2022-01-31T21:29:11.924+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:11.926088+0000) 2022-01-31T21:29:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:11 smithi171 conmon[46715]: debug 2022-01-31T21:29:11.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:11.971604+0000) 2022-01-31T21:29:12.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:12 smithi167 conmon[54076]: debug 2022-01-31T21:29:12.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:12.446410+0000) 2022-01-31T21:29:12.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:12 smithi167 conmon[60316]: debug 2022-01-31T21:29:12.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:12.448580+0000) 2022-01-31T21:29:12.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:12 smithi171 conmon[51620]: debug 2022-01-31T21:29:12.802+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:12.804208+0000) 2022-01-31T21:29:13.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:12 smithi167 conmon[49112]: debug 2022-01-31T21:29:12.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:12.781402+0000) 2022-01-31T21:29:13.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:12 smithi171 conmon[41853]: debug 2022-01-31T21:29:12.925+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:12.926271+0000) 2022-01-31T21:29:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:12 smithi171 conmon[46715]: debug 2022-01-31T21:29:12.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:12.971787+0000) 2022-01-31T21:29:13.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:13 smithi167 conmon[54076]: debug 2022-01-31T21:29:13.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:13.446588+0000) 2022-01-31T21:29:13.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:13 smithi167 conmon[60316]: debug 2022-01-31T21:29:13.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:13.448730+0000) 2022-01-31T21:29:13.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:13 smithi171 conmon[51620]: debug 2022-01-31T21:29:13.803+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:13.804391+0000) 2022-01-31T21:29:14.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:13 smithi167 conmon[49112]: debug 2022-01-31T21:29:13.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:13.781610+0000) 2022-01-31T21:29:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:13 smithi171 conmon[41853]: debug 2022-01-31T21:29:13.925+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:13.926475+0000) 2022-01-31T21:29:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:13 smithi171 conmon[46715]: debug 2022-01-31T21:29:13.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:13.971972+0000) 2022-01-31T21:29:14.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:14 smithi167 conmon[54076]: debug 2022-01-31T21:29:14.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:14.446774+0000) 2022-01-31T21:29:14.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:14 smithi167 conmon[60316]: debug 2022-01-31T21:29:14.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:14.448899+0000) 2022-01-31T21:29:14.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:14 smithi171 conmon[51620]: debug 2022-01-31T21:29:14.803+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:14.804581+0000) 2022-01-31T21:29:15.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:14 smithi167 conmon[49112]: debug 2022-01-31T21:29:14.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:14.781811+0000) 2022-01-31T21:29:15.333 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:14 smithi171 conmon[41853]: debug 2022-01-31T21:29:14.925+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:14.926656+0000) 2022-01-31T21:29:15.334 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:14 smithi171 conmon[46715]: debug 2022-01-31T21:29:14.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:14.972095+0000) 2022-01-31T21:29:15.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[35325]: debug 2022-01-31T21:29:15.348+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133630 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[41853]: debug 2022-01-31T21:29:15.334+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.335820+0000) 2022-01-31T21:29:15.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[46715]: debug 2022-01-31T21:29:15.334+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.336118+0000) 2022-01-31T21:29:15.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[51620]: debug 2022-01-31T21:29:15.333+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.335494+0000) 2022-01-31T21:29:15.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:15 smithi167 conmon[49112]: debug 2022-01-31T21:29:15.334+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.335561+0000) 2022-01-31T21:29:15.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:15 smithi167 conmon[60316]: debug 2022-01-31T21:29:15.335+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.336669+0000) 2022-01-31T21:29:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:15 smithi167 conmon[60316]: debug 2022-01-31T21:29:15.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.449090+0000) 2022-01-31T21:29:15.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:15 smithi167 conmon[54076]: debug 2022-01-31T21:29:15.335+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.337056+0000) 2022-01-31T21:29:15.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:15 smithi167 conmon[54076]: debug 2022-01-31T21:29:15.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.446981+0000) 2022-01-31T21:29:15.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[51620]: debug 2022-01-31T21:29:15.803+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.804774+0000) 2022-01-31T21:29:16.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:15 smithi167 conmon[49112]: debug 2022-01-31T21:29:15.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.782018+0000) 2022-01-31T21:29:16.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[41853]: debug 2022-01-31T21:29:15.925+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.926841+0000) 2022-01-31T21:29:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:15 smithi171 conmon[46715]: debug 2022-01-31T21:29:15.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:15.972214+0000) 2022-01-31T21:29:16.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:16 smithi167 conmon[54076]: debug 2022-01-31T21:29:16.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:16.447168+0000) 2022-01-31T21:29:16.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:16 smithi167 conmon[60316]: debug 2022-01-31T21:29:16.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:16.449251+0000) 2022-01-31T21:29:16.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:16 smithi171 conmon[51620]: debug 2022-01-31T21:29:16.804+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:16.804947+0000) 2022-01-31T21:29:17.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:16 smithi167 conmon[49112]: debug 2022-01-31T21:29:16.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:16.782181+0000) 2022-01-31T21:29:17.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:16 smithi171 conmon[41853]: debug 2022-01-31T21:29:16.925+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:16.926948+0000) 2022-01-31T21:29:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:16 smithi171 conmon[46715]: debug 2022-01-31T21:29:16.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:16.972388+0000) 2022-01-31T21:29:17.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:17 smithi167 conmon[54076]: debug 2022-01-31T21:29:17.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:17.447408+0000) 2022-01-31T21:29:17.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:17 smithi167 conmon[60316]: debug 2022-01-31T21:29:17.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:17.449397+0000) 2022-01-31T21:29:17.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:17 smithi171 conmon[51620]: debug 2022-01-31T21:29:17.804+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:17.805163+0000) 2022-01-31T21:29:18.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:17 smithi167 conmon[49112]: debug 2022-01-31T21:29:17.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:17.782384+0000) 2022-01-31T21:29:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:17 smithi171 conmon[41853]: debug 2022-01-31T21:29:17.926+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:17.927129+0000) 2022-01-31T21:29:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:17 smithi171 conmon[46715]: debug 2022-01-31T21:29:17.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:17.972590+0000) 2022-01-31T21:29:18.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:18 smithi167 conmon[54076]: debug 2022-01-31T21:29:18.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:18.447608+0000) 2022-01-31T21:29:18.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:18 smithi167 conmon[60316]: debug 2022-01-31T21:29:18.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:18.449572+0000) 2022-01-31T21:29:18.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:18 smithi171 conmon[51620]: debug 2022-01-31T21:29:18.804+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:18.805364+0000) 2022-01-31T21:29:19.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:18 smithi167 conmon[49112]: debug 2022-01-31T21:29:18.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:18.782575+0000) 2022-01-31T21:29:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:18 smithi171 conmon[41853]: debug 2022-01-31T21:29:18.926+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:18.927329+0000) 2022-01-31T21:29:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:18 smithi171 conmon[46715]: debug 2022-01-31T21:29:18.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:18.972822+0000) 2022-01-31T21:29:19.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:19 smithi167 conmon[54076]: debug 2022-01-31T21:29:19.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:19.447754+0000) 2022-01-31T21:29:19.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:19 smithi167 conmon[60316]: debug 2022-01-31T21:29:19.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:19.449723+0000) 2022-01-31T21:29:19.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:19 smithi171 conmon[51620]: debug 2022-01-31T21:29:19.803+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:19.805586+0000) 2022-01-31T21:29:20.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:19 smithi167 conmon[49112]: debug 2022-01-31T21:29:19.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:19.782695+0000) 2022-01-31T21:29:20.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:19 smithi171 conmon[46715]: debug 2022-01-31T21:29:19.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:19.973007+0000) 2022-01-31T21:29:20.351 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:19 smithi171 conmon[41853]: debug 2022-01-31T21:29:19.926+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:19.927534+0000) 2022-01-31T21:29:20.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[35325]: debug 2022-01-31T21:29:20.365+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133743 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[41853]: debug 2022-01-31T21:29:20.351+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.352324+0000) 2022-01-31T21:29:20.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[46715]: debug 2022-01-31T21:29:20.352+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.353646+0000) 2022-01-31T21:29:20.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[51620]: debug 2022-01-31T21:29:20.351+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.351888+0000) 2022-01-31T21:29:20.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:20 smithi167 conmon[49112]: debug 2022-01-31T21:29:20.352+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.354348+0000) 2022-01-31T21:29:20.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:20 smithi167 conmon[54076]: debug 2022-01-31T21:29:20.352+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.354110+0000) 2022-01-31T21:29:20.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:20 smithi167 conmon[54076]: debug 2022-01-31T21:29:20.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.447871+0000) 2022-01-31T21:29:20.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:20 smithi167 conmon[60316]: debug 2022-01-31T21:29:20.352+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.354017+0000) 2022-01-31T21:29:20.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:20 smithi167 conmon[60316]: debug 2022-01-31T21:29:20.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.449909+0000) 2022-01-31T21:29:20.885 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[51620]: debug 2022-01-31T21:29:20.805+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.805777+0000) 2022-01-31T21:29:21.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:20 smithi167 conmon[49112]: debug 2022-01-31T21:29:20.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.782815+0000) 2022-01-31T21:29:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[41853]: debug 2022-01-31T21:29:20.926+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.927688+0000) 2022-01-31T21:29:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:20 smithi171 conmon[46715]: debug 2022-01-31T21:29:20.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:20.973192+0000) 2022-01-31T21:29:21.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:21 smithi167 conmon[54076]: debug 2022-01-31T21:29:21.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:21.448031+0000) 2022-01-31T21:29:21.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:21 smithi167 conmon[60316]: debug 2022-01-31T21:29:21.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:21.450080+0000) 2022-01-31T21:29:21.885 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:21 smithi171 conmon[51620]: debug 2022-01-31T21:29:21.805+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:21.805963+0000) 2022-01-31T21:29:22.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:21 smithi167 conmon[49112]: debug 2022-01-31T21:29:21.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:21.782941+0000) 2022-01-31T21:29:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:21 smithi171 conmon[41853]: debug 2022-01-31T21:29:21.927+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:21.927844+0000) 2022-01-31T21:29:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:21 smithi171 conmon[46715]: debug 2022-01-31T21:29:21.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:21.973341+0000) 2022-01-31T21:29:22.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:22 smithi167 conmon[54076]: debug 2022-01-31T21:29:22.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:22.448223+0000) 2022-01-31T21:29:22.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:22 smithi167 conmon[60316]: debug 2022-01-31T21:29:22.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:22.450288+0000) 2022-01-31T21:29:22.885 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:22 smithi171 conmon[51620]: debug 2022-01-31T21:29:22.805+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:22.806106+0000) 2022-01-31T21:29:23.035 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:22 smithi167 conmon[49112]: debug 2022-01-31T21:29:22.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:22.783118+0000) 2022-01-31T21:29:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:22 smithi171 conmon[41853]: debug 2022-01-31T21:29:22.927+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:22.928027+0000) 2022-01-31T21:29:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:22 smithi171 conmon[46715]: debug 2022-01-31T21:29:22.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:22.973526+0000) 2022-01-31T21:29:23.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:23 smithi167 conmon[54076]: debug 2022-01-31T21:29:23.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:23.448417+0000) 2022-01-31T21:29:23.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:23 smithi167 conmon[60316]: debug 2022-01-31T21:29:23.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:23.450465+0000) 2022-01-31T21:29:23.885 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:23 smithi171 conmon[51620]: debug 2022-01-31T21:29:23.805+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:23.806336+0000) 2022-01-31T21:29:23.886 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:23 smithi171 conmon[35325]: debug 2022-01-31T21:29:23.687+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:29:24.036 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:23 smithi167 conmon[49112]: debug 2022-01-31T21:29:23.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:23.783327+0000) 2022-01-31T21:29:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:23 smithi171 conmon[41853]: debug 2022-01-31T21:29:23.927+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:23.928186+0000) 2022-01-31T21:29:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:23 smithi171 conmon[46715]: debug 2022-01-31T21:29:23.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:23.973654+0000) 2022-01-31T21:29:24.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:24 smithi167 conmon[54076]: debug 2022-01-31T21:29:24.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:24.448578+0000) 2022-01-31T21:29:24.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:24 smithi167 conmon[60316]: debug 2022-01-31T21:29:24.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:24.450595+0000) 2022-01-31T21:29:24.885 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:24 smithi171 conmon[51620]: debug 2022-01-31T21:29:24.805+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:24.806601+0000) 2022-01-31T21:29:25.036 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:24 smithi167 conmon[49112]: debug 2022-01-31T21:29:24.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:24.783611+0000) 2022-01-31T21:29:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:24 smithi171 conmon[41853]: debug 2022-01-31T21:29:24.927+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:24.928440+0000) 2022-01-31T21:29:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:24 smithi171 conmon[46715]: debug 2022-01-31T21:29:24.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:24.973880+0000) 2022-01-31T21:29:25.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:25 smithi167 conmon[49112]: debug 2022-01-31T21:29:25.368+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.370261+0000) 2022-01-31T21:29:25.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:25 smithi167 conmon[54076]: debug 2022-01-31T21:29:25.369+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.370786+0000) 2022-01-31T21:29:25.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:25 smithi167 conmon[54076]: debug 2022-01-31T21:29:25.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.448683+0000) 2022-01-31T21:29:25.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:25 smithi167 conmon[60316]: debug 2022-01-31T21:29:25.369+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.370737+0000) 2022-01-31T21:29:25.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:25 smithi167 conmon[60316]: debug 2022-01-31T21:29:25.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.450706+0000) 2022-01-31T21:29:25.805 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[35325]: debug 2022-01-31T21:29:25.383+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133851 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:25.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[41853]: debug 2022-01-31T21:29:25.368+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.369507+0000) 2022-01-31T21:29:25.806 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[46715]: debug 2022-01-31T21:29:25.369+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.369861+0000) 2022-01-31T21:29:25.807 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[51620]: debug 2022-01-31T21:29:25.367+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.368705+0000) 2022-01-31T21:29:26.036 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:25 smithi167 conmon[49112]: debug 2022-01-31T21:29:25.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.783824+0000) 2022-01-31T21:29:26.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[51620]: debug 2022-01-31T21:29:25.806+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.806776+0000) 2022-01-31T21:29:26.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[41853]: debug 2022-01-31T21:29:25.927+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.928579+0000) 2022-01-31T21:29:26.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:25 smithi171 conmon[46715]: debug 2022-01-31T21:29:25.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:25.974033+0000) 2022-01-31T21:29:26.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:26 smithi167 conmon[54076]: debug 2022-01-31T21:29:26.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:26.448860+0000) 2022-01-31T21:29:26.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:26 smithi167 conmon[60316]: debug 2022-01-31T21:29:26.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:26.450825+0000) 2022-01-31T21:29:26.886 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:26 smithi171 conmon[51620]: debug 2022-01-31T21:29:26.806+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:26.806899+0000) 2022-01-31T21:29:27.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:26 smithi167 conmon[49112]: debug 2022-01-31T21:29:26.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:26.783986+0000) 2022-01-31T21:29:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:26 smithi171 conmon[41853]: debug 2022-01-31T21:29:26.928+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:26.928720+0000) 2022-01-31T21:29:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:26 smithi171 conmon[46715]: debug 2022-01-31T21:29:26.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:26.974178+0000) 2022-01-31T21:29:27.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:27 smithi167 conmon[54076]: debug 2022-01-31T21:29:27.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:27.449041+0000) 2022-01-31T21:29:27.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:27 smithi167 conmon[60316]: debug 2022-01-31T21:29:27.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:27.450983+0000) 2022-01-31T21:29:27.886 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:27 smithi171 conmon[51620]: debug 2022-01-31T21:29:27.806+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:27.807074+0000) 2022-01-31T21:29:28.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:27 smithi167 conmon[49112]: debug 2022-01-31T21:29:27.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:27.784180+0000) 2022-01-31T21:29:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:27 smithi171 conmon[41853]: debug 2022-01-31T21:29:27.928+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:27.928923+0000) 2022-01-31T21:29:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:27 smithi171 conmon[46715]: debug 2022-01-31T21:29:27.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:27.974359+0000) 2022-01-31T21:29:28.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:28 smithi167 conmon[54076]: debug 2022-01-31T21:29:28.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:28.449221+0000) 2022-01-31T21:29:28.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:28 smithi167 conmon[60316]: debug 2022-01-31T21:29:28.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:28.451174+0000) 2022-01-31T21:29:28.886 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:28 smithi171 conmon[51620]: debug 2022-01-31T21:29:28.806+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:28.807259+0000) 2022-01-31T21:29:29.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:28 smithi167 conmon[49112]: debug 2022-01-31T21:29:28.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:28.784366+0000) 2022-01-31T21:29:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:28 smithi171 conmon[41853]: debug 2022-01-31T21:29:28.928+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:28.929094+0000) 2022-01-31T21:29:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:28 smithi171 conmon[46715]: debug 2022-01-31T21:29:28.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:28.974538+0000) 2022-01-31T21:29:29.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:29 smithi167 conmon[54076]: debug 2022-01-31T21:29:29.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:29.449404+0000) 2022-01-31T21:29:29.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:29 smithi167 conmon[60316]: debug 2022-01-31T21:29:29.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:29.451389+0000) 2022-01-31T21:29:29.886 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:29 smithi171 conmon[51620]: debug 2022-01-31T21:29:29.806+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:29.807468+0000) 2022-01-31T21:29:30.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:29 smithi167 conmon[49112]: debug 2022-01-31T21:29:29.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:29.784560+0000) 2022-01-31T21:29:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:29 smithi171 conmon[41853]: debug 2022-01-31T21:29:29.928+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:29.929274+0000) 2022-01-31T21:29:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:29 smithi171 conmon[46715]: debug 2022-01-31T21:29:29.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:29.974729+0000) 2022-01-31T21:29:30.651 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:30 smithi167 conmon[49112]: debug 2022-01-31T21:29:30.386+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.387314+0000) 2022-01-31T21:29:30.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:30 smithi167 conmon[54076]: debug 2022-01-31T21:29:30.387+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.387670+0000) 2022-01-31T21:29:30.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:30 smithi167 conmon[54076]: debug 2022-01-31T21:29:30.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.449606+0000) 2022-01-31T21:29:30.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:30 smithi167 conmon[60316]: debug 2022-01-31T21:29:30.387+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.388086+0000) 2022-01-31T21:29:30.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:30 smithi167 conmon[60316]: debug 2022-01-31T21:29:30.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.451593+0000) 2022-01-31T21:29:30.806 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[35325]: debug 2022-01-31T21:29:30.400+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 133964 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:30.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[41853]: debug 2022-01-31T21:29:30.385+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.386627+0000) 2022-01-31T21:29:30.807 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[46715]: debug 2022-01-31T21:29:30.386+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.386975+0000) 2022-01-31T21:29:30.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[51620]: debug 2022-01-31T21:29:30.386+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.386850+0000) 2022-01-31T21:29:31.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:30 smithi167 conmon[49112]: debug 2022-01-31T21:29:30.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.784791+0000) 2022-01-31T21:29:31.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[46715]: debug 2022-01-31T21:29:30.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.974911+0000) 2022-01-31T21:29:31.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[51620]: debug 2022-01-31T21:29:30.806+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.807626+0000) 2022-01-31T21:29:31.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:30 smithi171 conmon[41853]: debug 2022-01-31T21:29:30.928+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:30.929477+0000) 2022-01-31T21:29:31.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:31 smithi167 conmon[60316]: debug 2022-01-31T21:29:31.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:31.451695+0000) 2022-01-31T21:29:31.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:31 smithi167 conmon[54076]: debug 2022-01-31T21:29:31.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:31.449692+0000) 2022-01-31T21:29:31.886 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:31 smithi171 conmon[51620]: debug 2022-01-31T21:29:31.807+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:31.807788+0000) 2022-01-31T21:29:32.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:31 smithi167 conmon[49112]: debug 2022-01-31T21:29:31.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:31.784970+0000) 2022-01-31T21:29:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:31 smithi171 conmon[41853]: debug 2022-01-31T21:29:31.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:31.929661+0000) 2022-01-31T21:29:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:31 smithi171 conmon[46715]: debug 2022-01-31T21:29:31.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:31.975021+0000) 2022-01-31T21:29:32.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:32 smithi167 conmon[54076]: debug 2022-01-31T21:29:32.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:32.449847+0000) 2022-01-31T21:29:32.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:32 smithi167 conmon[60316]: debug 2022-01-31T21:29:32.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:32.451847+0000) 2022-01-31T21:29:32.887 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:32 smithi171 conmon[51620]: debug 2022-01-31T21:29:32.807+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:32.807924+0000) 2022-01-31T21:29:33.052 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:32 smithi167 conmon[49112]: debug 2022-01-31T21:29:32.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:32.785184+0000) 2022-01-31T21:29:33.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:32 smithi171 conmon[46715]: debug 2022-01-31T21:29:32.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:32.975232+0000) 2022-01-31T21:29:33.313 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:32 smithi171 conmon[41853]: debug 2022-01-31T21:29:32.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:32.929858+0000) 2022-01-31T21:29:33.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:33 smithi167 conmon[54076]: debug 2022-01-31T21:29:33.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:33.450002+0000) 2022-01-31T21:29:33.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:33 smithi167 conmon[60316]: debug 2022-01-31T21:29:33.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:33.452051+0000) 2022-01-31T21:29:33.887 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:33 smithi171 conmon[51620]: debug 2022-01-31T21:29:33.807+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:33.808085+0000) 2022-01-31T21:29:34.053 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:33 smithi167 conmon[49112]: debug 2022-01-31T21:29:33.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:33.785377+0000) 2022-01-31T21:29:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:33 smithi171 conmon[41853]: debug 2022-01-31T21:29:33.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:33.930067+0000) 2022-01-31T21:29:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:33 smithi171 conmon[46715]: debug 2022-01-31T21:29:33.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:33.975330+0000) 2022-01-31T21:29:34.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:34 smithi167 conmon[54076]: debug 2022-01-31T21:29:34.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:34.450165+0000) 2022-01-31T21:29:34.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:34 smithi167 conmon[60316]: debug 2022-01-31T21:29:34.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:34.452261+0000) 2022-01-31T21:29:34.887 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:34 smithi171 conmon[51620]: debug 2022-01-31T21:29:34.807+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:34.808278+0000) 2022-01-31T21:29:35.053 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:34 smithi167 conmon[49112]: debug 2022-01-31T21:29:34.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:34.785577+0000) 2022-01-31T21:29:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:34 smithi171 conmon[41853]: debug 2022-01-31T21:29:34.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:34.930204+0000) 2022-01-31T21:29:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:34 smithi171 conmon[46715]: debug 2022-01-31T21:29:34.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:34.975479+0000) 2022-01-31T21:29:35.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:35 smithi167 conmon[49112]: debug 2022-01-31T21:29:35.402+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.403417+0000) 2022-01-31T21:29:35.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:35 smithi167 conmon[60316]: debug 2022-01-31T21:29:35.404+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.404496+0000) 2022-01-31T21:29:35.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:35 smithi167 conmon[60316]: debug 2022-01-31T21:29:35.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.452414+0000) 2022-01-31T21:29:35.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:35 smithi167 conmon[54076]: debug 2022-01-31T21:29:35.404+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.404709+0000) 2022-01-31T21:29:35.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:35 smithi167 conmon[54076]: debug 2022-01-31T21:29:35.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.450395+0000) 2022-01-31T21:29:35.806 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[35325]: debug 2022-01-31T21:29:35.417+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134072 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:35.807 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[41853]: debug 2022-01-31T21:29:35.403+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.404431+0000) 2022-01-31T21:29:35.808 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[46715]: debug 2022-01-31T21:29:35.402+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.403710+0000) 2022-01-31T21:29:35.808 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[51620]: debug 2022-01-31T21:29:35.403+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.403875+0000) 2022-01-31T21:29:36.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:35 smithi167 conmon[49112]: debug 2022-01-31T21:29:35.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.785762+0000) 2022-01-31T21:29:36.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[41853]: debug 2022-01-31T21:29:35.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.930357+0000) 2022-01-31T21:29:36.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[46715]: debug 2022-01-31T21:29:35.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.975666+0000) 2022-01-31T21:29:36.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:35 smithi171 conmon[51620]: debug 2022-01-31T21:29:35.807+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:35.808469+0000) 2022-01-31T21:29:36.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:36 smithi167 conmon[54076]: debug 2022-01-31T21:29:36.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:36.450565+0000) 2022-01-31T21:29:36.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:36 smithi167 conmon[60316]: debug 2022-01-31T21:29:36.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:36.452578+0000) 2022-01-31T21:29:36.887 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:36 smithi171 conmon[51620]: debug 2022-01-31T21:29:36.808+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:36.808583+0000) 2022-01-31T21:29:37.057 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:36 smithi167 conmon[49112]: debug 2022-01-31T21:29:36.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:36.785929+0000) 2022-01-31T21:29:37.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:36 smithi171 conmon[41853]: debug 2022-01-31T21:29:36.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:36.930524+0000) 2022-01-31T21:29:37.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:36 smithi171 conmon[46715]: debug 2022-01-31T21:29:36.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:36.975820+0000) 2022-01-31T21:29:37.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:37 smithi167 conmon[54076]: debug 2022-01-31T21:29:37.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:37.450723+0000) 2022-01-31T21:29:37.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:37 smithi167 conmon[60316]: debug 2022-01-31T21:29:37.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:37.452731+0000) 2022-01-31T21:29:37.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:37 smithi171 conmon[51620]: debug 2022-01-31T21:29:37.808+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:37.808736+0000) 2022-01-31T21:29:38.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:37 smithi167 conmon[49112]: debug 2022-01-31T21:29:37.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:37.786134+0000) 2022-01-31T21:29:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:37 smithi171 conmon[41853]: debug 2022-01-31T21:29:37.929+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:37.930726+0000) 2022-01-31T21:29:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:37 smithi171 conmon[46715]: debug 2022-01-31T21:29:37.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:37.975974+0000) 2022-01-31T21:29:38.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:38 smithi167 conmon[54076]: debug 2022-01-31T21:29:38.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:38.450934+0000) 2022-01-31T21:29:38.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:38 smithi167 conmon[60316]: debug 2022-01-31T21:29:38.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:38.452879+0000) 2022-01-31T21:29:38.888 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:38 smithi171 conmon[35325]: debug 2022-01-31T21:29:38.688+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:29:38.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:38 smithi171 conmon[51620]: debug 2022-01-31T21:29:38.808+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:38.808934+0000) 2022-01-31T21:29:39.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:38 smithi167 conmon[49112]: debug 2022-01-31T21:29:38.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:38.786344+0000) 2022-01-31T21:29:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:38 smithi171 conmon[41853]: debug 2022-01-31T21:29:38.930+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:38.930942+0000) 2022-01-31T21:29:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:38 smithi171 conmon[46715]: debug 2022-01-31T21:29:38.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:38.976156+0000) 2022-01-31T21:29:39.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:39 smithi167 conmon[54076]: debug 2022-01-31T21:29:39.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:39.451099+0000) 2022-01-31T21:29:39.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:39 smithi167 conmon[60316]: debug 2022-01-31T21:29:39.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:39.453068+0000) 2022-01-31T21:29:39.887 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:39 smithi171 conmon[51620]: debug 2022-01-31T21:29:39.808+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:39.809096+0000) 2022-01-31T21:29:40.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:39 smithi167 conmon[49112]: debug 2022-01-31T21:29:39.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:39.786569+0000) 2022-01-31T21:29:40.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:39 smithi171 conmon[41853]: debug 2022-01-31T21:29:39.930+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:39.931090+0000) 2022-01-31T21:29:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:39 smithi171 conmon[46715]: debug 2022-01-31T21:29:39.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:39.976312+0000) 2022-01-31T21:29:40.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:40 smithi167 conmon[49112]: debug 2022-01-31T21:29:40.419+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.420416+0000) 2022-01-31T21:29:40.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:40 smithi167 conmon[54076]: debug 2022-01-31T21:29:40.420+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.421173+0000) 2022-01-31T21:29:40.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:40 smithi167 conmon[54076]: debug 2022-01-31T21:29:40.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.451248+0000) 2022-01-31T21:29:40.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:40 smithi167 conmon[60316]: debug 2022-01-31T21:29:40.420+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.421066+0000) 2022-01-31T21:29:40.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:40 smithi167 conmon[60316]: debug 2022-01-31T21:29:40.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.453292+0000) 2022-01-31T21:29:40.807 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[35325]: debug 2022-01-31T21:29:40.434+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134185 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:40.808 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[41853]: debug 2022-01-31T21:29:40.419+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.420072+0000) 2022-01-31T21:29:40.808 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[46715]: debug 2022-01-31T21:29:40.419+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.420304+0000) 2022-01-31T21:29:40.809 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[51620]: debug 2022-01-31T21:29:40.420+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.421044+0000) 2022-01-31T21:29:41.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:40 smithi167 conmon[49112]: debug 2022-01-31T21:29:40.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.786751+0000) 2022-01-31T21:29:41.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[41853]: debug 2022-01-31T21:29:40.930+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.931248+0000) 2022-01-31T21:29:41.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[46715]: debug 2022-01-31T21:29:40.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.976477+0000) 2022-01-31T21:29:41.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:40 smithi171 conmon[51620]: debug 2022-01-31T21:29:40.808+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:40.809267+0000) 2022-01-31T21:29:41.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:41 smithi167 conmon[54076]: debug 2022-01-31T21:29:41.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:41.451384+0000) 2022-01-31T21:29:41.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:41 smithi167 conmon[60316]: debug 2022-01-31T21:29:41.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:41.453372+0000) 2022-01-31T21:29:41.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:41 smithi171 conmon[51620]: debug 2022-01-31T21:29:41.809+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:41.809432+0000) 2022-01-31T21:29:42.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:41 smithi167 conmon[49112]: debug 2022-01-31T21:29:41.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:41.786917+0000) 2022-01-31T21:29:42.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:41 smithi171 conmon[41853]: debug 2022-01-31T21:29:41.930+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:41.931397+0000) 2022-01-31T21:29:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:41 smithi171 conmon[46715]: debug 2022-01-31T21:29:41.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:41.976643+0000) 2022-01-31T21:29:42.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:42 smithi167 conmon[54076]: debug 2022-01-31T21:29:42.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:42.451534+0000) 2022-01-31T21:29:42.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:42 smithi167 conmon[60316]: debug 2022-01-31T21:29:42.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:42.453554+0000) 2022-01-31T21:29:42.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:42 smithi171 conmon[51620]: debug 2022-01-31T21:29:42.808+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:42.809641+0000) 2022-01-31T21:29:43.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:42 smithi167 conmon[49112]: debug 2022-01-31T21:29:42.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:42.787070+0000) 2022-01-31T21:29:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:42 smithi171 conmon[41853]: debug 2022-01-31T21:29:42.930+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:42.931612+0000) 2022-01-31T21:29:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:42 smithi171 conmon[46715]: debug 2022-01-31T21:29:42.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:42.976848+0000) 2022-01-31T21:29:43.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:43 smithi167 conmon[54076]: debug 2022-01-31T21:29:43.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:43.451683+0000) 2022-01-31T21:29:43.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:43 smithi167 conmon[60316]: debug 2022-01-31T21:29:43.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:43.453714+0000) 2022-01-31T21:29:43.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:43 smithi171 conmon[51620]: debug 2022-01-31T21:29:43.809+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:43.809854+0000) 2022-01-31T21:29:44.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:43 smithi167 conmon[49112]: debug 2022-01-31T21:29:43.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:43.787235+0000) 2022-01-31T21:29:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:43 smithi171 conmon[41853]: debug 2022-01-31T21:29:43.931+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:43.931843+0000) 2022-01-31T21:29:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:43 smithi171 conmon[46715]: debug 2022-01-31T21:29:43.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:43.977007+0000) 2022-01-31T21:29:44.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:44 smithi167 conmon[54076]: debug 2022-01-31T21:29:44.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:44.451874+0000) 2022-01-31T21:29:44.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:44 smithi167 conmon[60316]: debug 2022-01-31T21:29:44.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:44.453891+0000) 2022-01-31T21:29:44.888 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:44 smithi171 conmon[51620]: debug 2022-01-31T21:29:44.809+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:44.810055+0000) 2022-01-31T21:29:45.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:44 smithi167 conmon[49112]: debug 2022-01-31T21:29:44.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:44.787375+0000) 2022-01-31T21:29:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:44 smithi171 conmon[41853]: debug 2022-01-31T21:29:44.931+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:44.932074+0000) 2022-01-31T21:29:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:44 smithi171 conmon[46715]: debug 2022-01-31T21:29:44.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:44.977197+0000) 2022-01-31T21:29:45.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:45 smithi167 conmon[49112]: debug 2022-01-31T21:29:45.436+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.436975+0000) 2022-01-31T21:29:45.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:45 smithi167 conmon[54076]: debug 2022-01-31T21:29:45.437+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.437495+0000) 2022-01-31T21:29:45.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:45 smithi167 conmon[54076]: debug 2022-01-31T21:29:45.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.452049+0000) 2022-01-31T21:29:45.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:45 smithi167 conmon[60316]: debug 2022-01-31T21:29:45.437+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.438358+0000) 2022-01-31T21:29:45.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:45 smithi167 conmon[60316]: debug 2022-01-31T21:29:45.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.454016+0000) 2022-01-31T21:29:45.809 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[35325]: debug 2022-01-31T21:29:45.451+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134293 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:45.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[41853]: debug 2022-01-31T21:29:45.437+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.438448+0000) 2022-01-31T21:29:45.811 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[46715]: debug 2022-01-31T21:29:45.436+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.437100+0000) 2022-01-31T21:29:45.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[51620]: debug 2022-01-31T21:29:45.438+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.438780+0000) 2022-01-31T21:29:46.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:45 smithi167 conmon[49112]: debug 2022-01-31T21:29:45.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.787540+0000) 2022-01-31T21:29:46.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[41853]: debug 2022-01-31T21:29:45.931+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.932236+0000) 2022-01-31T21:29:46.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[46715]: debug 2022-01-31T21:29:45.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.977361+0000) 2022-01-31T21:29:46.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:45 smithi171 conmon[51620]: debug 2022-01-31T21:29:45.809+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:45.810259+0000) 2022-01-31T21:29:46.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:46 smithi167 conmon[54076]: debug 2022-01-31T21:29:46.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:46.452224+0000) 2022-01-31T21:29:46.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:46 smithi167 conmon[60316]: debug 2022-01-31T21:29:46.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:46.454158+0000) 2022-01-31T21:29:46.889 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:46 smithi171 conmon[51620]: debug 2022-01-31T21:29:46.810+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:46.810395+0000) 2022-01-31T21:29:47.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:46 smithi167 conmon[49112]: debug 2022-01-31T21:29:46.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:46.787683+0000) 2022-01-31T21:29:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:46 smithi171 conmon[41853]: debug 2022-01-31T21:29:46.932+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:46.932396+0000) 2022-01-31T21:29:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:46 smithi171 conmon[46715]: debug 2022-01-31T21:29:46.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:46.977515+0000) 2022-01-31T21:29:47.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:47 smithi167 conmon[54076]: debug 2022-01-31T21:29:47.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:47.452438+0000) 2022-01-31T21:29:47.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:47 smithi167 conmon[60316]: debug 2022-01-31T21:29:47.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:47.454357+0000) 2022-01-31T21:29:47.889 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:47 smithi171 conmon[51620]: debug 2022-01-31T21:29:47.810+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:47.810573+0000) 2022-01-31T21:29:48.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:47 smithi167 conmon[49112]: debug 2022-01-31T21:29:47.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:47.787892+0000) 2022-01-31T21:29:48.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:47 smithi171 conmon[41853]: debug 2022-01-31T21:29:47.932+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:47.932602+0000) 2022-01-31T21:29:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:47 smithi171 conmon[46715]: debug 2022-01-31T21:29:47.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:47.977723+0000) 2022-01-31T21:29:48.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:48 smithi167 conmon[54076]: debug 2022-01-31T21:29:48.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:48.452611+0000) 2022-01-31T21:29:48.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:48 smithi167 conmon[60316]: debug 2022-01-31T21:29:48.454+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:48.454541+0000) 2022-01-31T21:29:48.889 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:48 smithi171 conmon[51620]: debug 2022-01-31T21:29:48.810+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:48.810793+0000) 2022-01-31T21:29:49.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:48 smithi167 conmon[49112]: debug 2022-01-31T21:29:48.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:48.788129+0000) 2022-01-31T21:29:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:48 smithi171 conmon[41853]: debug 2022-01-31T21:29:48.932+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:48.932813+0000) 2022-01-31T21:29:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:48 smithi171 conmon[46715]: debug 2022-01-31T21:29:48.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:48.977875+0000) 2022-01-31T21:29:49.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:49 smithi167 conmon[54076]: debug 2022-01-31T21:29:49.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:49.452794+0000) 2022-01-31T21:29:49.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:49 smithi167 conmon[60316]: debug 2022-01-31T21:29:49.454+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:49.454702+0000) 2022-01-31T21:29:49.889 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:49 smithi171 conmon[51620]: debug 2022-01-31T21:29:49.810+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:49.811008+0000) 2022-01-31T21:29:50.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:49 smithi167 conmon[49112]: debug 2022-01-31T21:29:49.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:49.788258+0000) 2022-01-31T21:29:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:49 smithi171 conmon[41853]: debug 2022-01-31T21:29:49.932+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:49.932992+0000) 2022-01-31T21:29:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:49 smithi171 conmon[46715]: debug 2022-01-31T21:29:49.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:49.978075+0000) 2022-01-31T21:29:50.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:50 smithi167 conmon[49112]: debug 2022-01-31T21:29:50.453+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.455262+0000) 2022-01-31T21:29:50.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:50 smithi167 conmon[54076]: debug 2022-01-31T21:29:50.453+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.455340+0000) 2022-01-31T21:29:50.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:50 smithi167 conmon[60316]: debug 2022-01-31T21:29:50.454+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.456255+0000) 2022-01-31T21:29:50.809 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[35325]: debug 2022-01-31T21:29:50.468+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134406 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:50.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[41853]: debug 2022-01-31T21:29:50.455+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.455804+0000) 2022-01-31T21:29:50.811 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[46715]: debug 2022-01-31T21:29:50.453+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.454471+0000) 2022-01-31T21:29:50.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[51620]: debug 2022-01-31T21:29:50.454+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.454983+0000) 2022-01-31T21:29:51.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:50 smithi167 conmon[49112]: debug 2022-01-31T21:29:50.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.788477+0000) 2022-01-31T21:29:51.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[41853]: debug 2022-01-31T21:29:50.932+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.933174+0000) 2022-01-31T21:29:51.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[46715]: debug 2022-01-31T21:29:50.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.978255+0000) 2022-01-31T21:29:51.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:50 smithi171 conmon[51620]: debug 2022-01-31T21:29:50.810+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:50.811201+0000) 2022-01-31T21:29:51.890 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:51 smithi171 conmon[51620]: debug 2022-01-31T21:29:51.811+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:51.811356+0000) 2022-01-31T21:29:52.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:51 smithi167 conmon[49112]: debug 2022-01-31T21:29:51.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:51.788637+0000) 2022-01-31T21:29:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:51 smithi171 conmon[46715]: debug 2022-01-31T21:29:51.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:51.978379+0000) 2022-01-31T21:29:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:51 smithi171 conmon[41853]: debug 2022-01-31T21:29:51.932+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:51.933329+0000) 2022-01-31T21:29:52.890 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:52 smithi171 conmon[51620]: debug 2022-01-31T21:29:52.810+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:52.811564+0000) 2022-01-31T21:29:53.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:52 smithi167 conmon[49112]: debug 2022-01-31T21:29:52.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:52.788834+0000) 2022-01-31T21:29:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:52 smithi171 conmon[41853]: debug 2022-01-31T21:29:52.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:52.933563+0000) 2022-01-31T21:29:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:52 smithi171 conmon[46715]: debug 2022-01-31T21:29:52.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:52.978541+0000) 2022-01-31T21:29:53.890 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:53 smithi171 conmon[35325]: debug 2022-01-31T21:29:53.689+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:29:53.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:53 smithi171 conmon[51620]: debug 2022-01-31T21:29:53.811+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:53.811796+0000) 2022-01-31T21:29:54.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:53 smithi167 conmon[49112]: debug 2022-01-31T21:29:53.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:53.789006+0000) 2022-01-31T21:29:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:53 smithi171 conmon[41853]: debug 2022-01-31T21:29:53.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:53.933667+0000) 2022-01-31T21:29:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:53 smithi171 conmon[46715]: debug 2022-01-31T21:29:53.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:53.978727+0000) 2022-01-31T21:29:54.890 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:54 smithi171 conmon[51620]: debug 2022-01-31T21:29:54.811+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:54.812000+0000) 2022-01-31T21:29:55.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:54 smithi167 conmon[49112]: debug 2022-01-31T21:29:54.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:54.789210+0000) 2022-01-31T21:29:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:54 smithi171 conmon[41853]: debug 2022-01-31T21:29:54.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:54.933821+0000) 2022-01-31T21:29:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:54 smithi171 conmon[46715]: debug 2022-01-31T21:29:54.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:54.978925+0000) 2022-01-31T21:29:55.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:55 smithi167 conmon[49112]: debug 2022-01-31T21:29:55.470+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.471850+0000) 2022-01-31T21:29:55.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:55 smithi167 conmon[54076]: debug 2022-01-31T21:29:55.469+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.470717+0000) 2022-01-31T21:29:55.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:55 smithi167 conmon[60316]: debug 2022-01-31T21:29:55.470+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.471691+0000) 2022-01-31T21:29:55.810 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[35325]: debug 2022-01-31T21:29:55.484+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134514 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:29:55.811 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[41853]: debug 2022-01-31T21:29:55.471+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.472685+0000) 2022-01-31T21:29:55.812 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[46715]: debug 2022-01-31T21:29:55.470+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.471243+0000) 2022-01-31T21:29:55.812 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[51620]: debug 2022-01-31T21:29:55.471+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.472344+0000) 2022-01-31T21:29:56.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:55 smithi167 conmon[49112]: debug 2022-01-31T21:29:55.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.789432+0000) 2022-01-31T21:29:56.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[51620]: debug 2022-01-31T21:29:55.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.812194+0000) 2022-01-31T21:29:56.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[41853]: debug 2022-01-31T21:29:55.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.934022+0000) 2022-01-31T21:29:56.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:55 smithi171 conmon[46715]: debug 2022-01-31T21:29:55.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:55.979082+0000) 2022-01-31T21:29:56.890 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:56 smithi171 conmon[51620]: debug 2022-01-31T21:29:56.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:56.812335+0000) 2022-01-31T21:29:57.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:56 smithi167 conmon[49112]: debug 2022-01-31T21:29:56.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:56.789589+0000) 2022-01-31T21:29:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:56 smithi171 conmon[41853]: debug 2022-01-31T21:29:56.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:56.934148+0000) 2022-01-31T21:29:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:56 smithi171 conmon[46715]: debug 2022-01-31T21:29:56.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:56.979196+0000) 2022-01-31T21:29:57.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:57 smithi167 conmon[54076]: debug 2022-01-31T21:29:57.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:57.203246+0000) 2022-01-31T21:29:57.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:57 smithi167 conmon[60316]: debug 2022-01-31T21:29:57.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:57.205083+0000) 2022-01-31T21:29:57.896 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:57 smithi171 conmon[51620]: debug 2022-01-31T21:29:57.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:57.812540+0000) 2022-01-31T21:29:58.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:57 smithi167 conmon[49112]: debug 2022-01-31T21:29:57.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:57.789789+0000) 2022-01-31T21:29:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:57 smithi171 conmon[41853]: debug 2022-01-31T21:29:57.934+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:57.934305+0000) 2022-01-31T21:29:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:57 smithi171 conmon[46715]: debug 2022-01-31T21:29:57.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:57.979366+0000) 2022-01-31T21:29:58.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:58 smithi167 conmon[54076]: debug 2022-01-31T21:29:58.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:58.203427+0000) 2022-01-31T21:29:58.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:58 smithi167 conmon[60316]: debug 2022-01-31T21:29:58.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:58.205254+0000) 2022-01-31T21:29:58.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:58 smithi171 conmon[51620]: debug 2022-01-31T21:29:58.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:58.812735+0000) 2022-01-31T21:29:59.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:58 smithi167 conmon[49112]: debug 2022-01-31T21:29:58.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:58.790025+0000) 2022-01-31T21:29:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:58 smithi171 conmon[46715]: debug 2022-01-31T21:29:58.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:58.979539+0000) 2022-01-31T21:29:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:58 smithi171 conmon[41853]: debug 2022-01-31T21:29:58.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:58.934450+0000) 2022-01-31T21:29:59.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:29:59 smithi167 conmon[54076]: debug 2022-01-31T21:29:59.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:59.203613+0000) 2022-01-31T21:29:59.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:29:59 smithi167 conmon[60316]: debug 2022-01-31T21:29:59.203+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:59.205471+0000) 2022-01-31T21:29:59.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:29:59 smithi171 conmon[51620]: debug 2022-01-31T21:29:59.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:59.812886+0000) 2022-01-31T21:30:00.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:29:59 smithi167 conmon[49112]: debug 2022-01-31T21:29:59.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:59.790235+0000) 2022-01-31T21:30:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:29:59 smithi171 conmon[41853]: debug 2022-01-31T21:29:59.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:59.934672+0000) 2022-01-31T21:30:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:29:59 smithi171 conmon[46715]: debug 2022-01-31T21:29:59.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:29:59.979700+0000) 2022-01-31T21:30:00.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:00 smithi167 conmon[54076]: debug 2022-01-31T21:30:00.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.203773+0000) 2022-01-31T21:30:00.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:00 smithi167 conmon[60316]: debug 2022-01-31T21:30:00.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.205641+0000) 2022-01-31T21:30:00.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:00 smithi167 conmon[49112]: debug 2022-01-31T21:30:00.485+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.487371+0000) 2022-01-31T21:30:00.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:00 smithi167 conmon[54076]: debug 2022-01-31T21:30:00.486+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.487872+0000) 2022-01-31T21:30:00.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:00 smithi167 conmon[60316]: debug 2022-01-31T21:30:00.487+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.488780+0000) 2022-01-31T21:30:00.811 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[35325]: debug 2022-01-31T21:30:00.506+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134627 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:00.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[41853]: debug 2022-01-31T21:30:00.487+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.488317+0000) 2022-01-31T21:30:00.813 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[46715]: debug 2022-01-31T21:30:00.486+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.487171+0000) 2022-01-31T21:30:00.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[51620]: debug 2022-01-31T21:30:00.487+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.488596+0000) 2022-01-31T21:30:01.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:00 smithi167 conmon[49112]: debug 2022-01-31T21:30:00.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.790438+0000) 2022-01-31T21:30:01.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[41853]: debug 2022-01-31T21:30:00.934+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.934773+0000) 2022-01-31T21:30:01.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[46715]: debug 2022-01-31T21:30:00.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.979930+0000) 2022-01-31T21:30:01.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:00 smithi171 conmon[51620]: debug 2022-01-31T21:30:00.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:00.812993+0000) 2022-01-31T21:30:01.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:01 smithi167 conmon[54076]: debug 2022-01-31T21:30:01.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:01.203870+0000) 2022-01-31T21:30:01.371 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:01 smithi167 conmon[60316]: debug 2022-01-31T21:30:01.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:01.205791+0000) 2022-01-31T21:30:01.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:01 smithi171 conmon[51620]: debug 2022-01-31T21:30:01.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:01.813142+0000) 2022-01-31T21:30:02.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:01 smithi167 conmon[49112]: debug 2022-01-31T21:30:01.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:01.790603+0000) 2022-01-31T21:30:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:01 smithi171 conmon[41853]: debug 2022-01-31T21:30:01.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:01.934911+0000) 2022-01-31T21:30:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:01 smithi171 conmon[46715]: debug 2022-01-31T21:30:01.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:01.980076+0000) 2022-01-31T21:30:02.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:02 smithi167 conmon[60316]: debug 2022-01-31T21:30:02.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:02.205931+0000) 2022-01-31T21:30:02.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:02 smithi167 conmon[54076]: debug 2022-01-31T21:30:02.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:02.204010+0000) 2022-01-31T21:30:02.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:02 smithi171 conmon[51620]: debug 2022-01-31T21:30:02.811+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:02.813290+0000) 2022-01-31T21:30:03.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:02 smithi167 conmon[49112]: debug 2022-01-31T21:30:02.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:02.790808+0000) 2022-01-31T21:30:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:02 smithi171 conmon[41853]: debug 2022-01-31T21:30:02.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:02.935109+0000) 2022-01-31T21:30:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:02 smithi171 conmon[46715]: debug 2022-01-31T21:30:02.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:02.980233+0000) 2022-01-31T21:30:03.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:03 smithi167 conmon[60316]: debug 2022-01-31T21:30:03.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:03.206102+0000) 2022-01-31T21:30:03.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:03 smithi167 conmon[54076]: debug 2022-01-31T21:30:03.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:03.204197+0000) 2022-01-31T21:30:03.892 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:03 smithi171 conmon[51620]: debug 2022-01-31T21:30:03.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:03.813473+0000) 2022-01-31T21:30:04.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:03 smithi167 conmon[49112]: debug 2022-01-31T21:30:03.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:03.790967+0000) 2022-01-31T21:30:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:03 smithi171 conmon[46715]: debug 2022-01-31T21:30:03.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:03.980387+0000) 2022-01-31T21:30:04.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:03 smithi171 conmon[41853]: debug 2022-01-31T21:30:03.933+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:03.935288+0000) 2022-01-31T21:30:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:04 smithi167 conmon[60316]: debug 2022-01-31T21:30:04.204+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:04.206302+0000) 2022-01-31T21:30:04.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:04 smithi167 conmon[54076]: debug 2022-01-31T21:30:04.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:04.204392+0000) 2022-01-31T21:30:04.892 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:04 smithi171 conmon[51620]: debug 2022-01-31T21:30:04.811+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:04.813617+0000) 2022-01-31T21:30:05.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:04 smithi167 conmon[49112]: debug 2022-01-31T21:30:04.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:04.791168+0000) 2022-01-31T21:30:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:04 smithi171 conmon[41853]: debug 2022-01-31T21:30:04.934+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:04.935479+0000) 2022-01-31T21:30:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:04 smithi171 conmon[46715]: debug 2022-01-31T21:30:04.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:04.980542+0000) 2022-01-31T21:30:05.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:05 smithi167 conmon[54076]: debug 2022-01-31T21:30:05.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.204595+0000) 2022-01-31T21:30:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:05 smithi167 conmon[60316]: debug 2022-01-31T21:30:05.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.206489+0000) 2022-01-31T21:30:05.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:05 smithi167 conmon[49112]: debug 2022-01-31T21:30:05.507+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.509372+0000) 2022-01-31T21:30:05.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:05 smithi167 conmon[54076]: debug 2022-01-31T21:30:05.508+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.509764+0000) 2022-01-31T21:30:05.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:05 smithi167 conmon[60316]: debug 2022-01-31T21:30:05.508+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.510346+0000) 2022-01-31T21:30:05.812 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[35325]: debug 2022-01-31T21:30:05.522+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134735 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:05.813 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[41853]: debug 2022-01-31T21:30:05.507+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.508989+0000) 2022-01-31T21:30:05.813 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[46715]: debug 2022-01-31T21:30:05.507+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.508785+0000) 2022-01-31T21:30:05.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[51620]: debug 2022-01-31T21:30:05.507+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.509244+0000) 2022-01-31T21:30:06.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[41853]: debug 2022-01-31T21:30:05.934+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.935668+0000) 2022-01-31T21:30:06.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[46715]: debug 2022-01-31T21:30:05.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.980701+0000) 2022-01-31T21:30:06.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:05 smithi171 conmon[51620]: debug 2022-01-31T21:30:05.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.813801+0000) 2022-01-31T21:30:06.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:05 smithi167 conmon[49112]: debug 2022-01-31T21:30:05.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:05.791392+0000) 2022-01-31T21:30:06.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:06 smithi167 conmon[54076]: debug 2022-01-31T21:30:06.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:06.204790+0000) 2022-01-31T21:30:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:06 smithi167 conmon[60316]: debug 2022-01-31T21:30:06.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:06.206631+0000) 2022-01-31T21:30:06.892 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:06 smithi171 conmon[51620]: debug 2022-01-31T21:30:06.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:06.813975+0000) 2022-01-31T21:30:07.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:06 smithi167 conmon[49112]: debug 2022-01-31T21:30:06.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:06.791557+0000) 2022-01-31T21:30:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:06 smithi171 conmon[46715]: debug 2022-01-31T21:30:06.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:06.980897+0000) 2022-01-31T21:30:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:06 smithi171 conmon[41853]: debug 2022-01-31T21:30:06.934+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:06.935812+0000) 2022-01-31T21:30:07.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:07 smithi167 conmon[54076]: debug 2022-01-31T21:30:07.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:07.204926+0000) 2022-01-31T21:30:07.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:07 smithi167 conmon[60316]: debug 2022-01-31T21:30:07.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:07.206716+0000) 2022-01-31T21:30:07.892 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:07 smithi171 conmon[51620]: debug 2022-01-31T21:30:07.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:07.814169+0000) 2022-01-31T21:30:08.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:07 smithi167 conmon[49112]: debug 2022-01-31T21:30:07.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:07.791762+0000) 2022-01-31T21:30:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:07 smithi171 conmon[41853]: debug 2022-01-31T21:30:07.935+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:07.935988+0000) 2022-01-31T21:30:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:07 smithi171 conmon[46715]: debug 2022-01-31T21:30:07.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:07.981061+0000) 2022-01-31T21:30:08.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:08 smithi167 conmon[54076]: debug 2022-01-31T21:30:08.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:08.205134+0000) 2022-01-31T21:30:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:08 smithi167 conmon[60316]: debug 2022-01-31T21:30:08.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:08.206836+0000) 2022-01-31T21:30:08.893 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:08 smithi171 conmon[51620]: debug 2022-01-31T21:30:08.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:08.814358+0000) 2022-01-31T21:30:08.893 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:08 smithi171 conmon[35325]: debug 2022-01-31T21:30:08.689+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:30:09.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:08 smithi167 conmon[49112]: debug 2022-01-31T21:30:08.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:08.791939+0000) 2022-01-31T21:30:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:08 smithi171 conmon[46715]: debug 2022-01-31T21:30:08.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:08.981244+0000) 2022-01-31T21:30:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:08 smithi171 conmon[41853]: debug 2022-01-31T21:30:08.934+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:08.936168+0000) 2022-01-31T21:30:09.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:09 smithi167 conmon[54076]: debug 2022-01-31T21:30:09.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:09.205257+0000) 2022-01-31T21:30:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:09 smithi167 conmon[60316]: debug 2022-01-31T21:30:09.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:09.207038+0000) 2022-01-31T21:30:09.893 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:09 smithi171 conmon[51620]: debug 2022-01-31T21:30:09.812+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:09.814554+0000) 2022-01-31T21:30:10.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:09 smithi167 conmon[49112]: debug 2022-01-31T21:30:09.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:09.792154+0000) 2022-01-31T21:30:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:09 smithi171 conmon[46715]: debug 2022-01-31T21:30:09.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:09.981427+0000) 2022-01-31T21:30:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:09 smithi171 conmon[41853]: debug 2022-01-31T21:30:09.935+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:09.936388+0000) 2022-01-31T21:30:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:10 smithi167 conmon[54076]: debug 2022-01-31T21:30:10.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.205439+0000) 2022-01-31T21:30:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:10 smithi167 conmon[60316]: debug 2022-01-31T21:30:10.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.207218+0000) 2022-01-31T21:30:10.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:10 smithi167 conmon[49112]: debug 2022-01-31T21:30:10.525+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.526511+0000) 2022-01-31T21:30:10.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:10 smithi167 conmon[54076]: debug 2022-01-31T21:30:10.524+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.526144+0000) 2022-01-31T21:30:10.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:10 smithi167 conmon[60316]: debug 2022-01-31T21:30:10.525+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.526825+0000) 2022-01-31T21:30:10.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[41853]: debug 2022-01-31T21:30:10.525+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.527239+0000) 2022-01-31T21:30:10.813 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[46715]: debug 2022-01-31T21:30:10.524+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.526141+0000) 2022-01-31T21:30:10.814 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[51620]: debug 2022-01-31T21:30:10.524+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.525822+0000) 2022-01-31T21:30:10.814 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[35325]: debug 2022-01-31T21:30:10.538+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134848 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:11.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[46715]: debug 2022-01-31T21:30:10.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.981630+0000) 2022-01-31T21:30:11.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[41853]: debug 2022-01-31T21:30:10.935+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.936619+0000) 2022-01-31T21:30:11.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:10 smithi171 conmon[51620]: debug 2022-01-31T21:30:10.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.814772+0000) 2022-01-31T21:30:11.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:10 smithi167 conmon[49112]: debug 2022-01-31T21:30:10.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:10.792383+0000) 2022-01-31T21:30:11.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:11 smithi167 conmon[54076]: debug 2022-01-31T21:30:11.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:11.205642+0000) 2022-01-31T21:30:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:11 smithi167 conmon[60316]: debug 2022-01-31T21:30:11.205+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:11.207386+0000) 2022-01-31T21:30:11.893 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:11 smithi171 conmon[51620]: debug 2022-01-31T21:30:11.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:11.814903+0000) 2022-01-31T21:30:12.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:11 smithi167 conmon[49112]: debug 2022-01-31T21:30:11.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:11.792570+0000) 2022-01-31T21:30:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:11 smithi171 conmon[46715]: debug 2022-01-31T21:30:11.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:11.981787+0000) 2022-01-31T21:30:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:11 smithi171 conmon[41853]: debug 2022-01-31T21:30:11.935+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:11.936794+0000) 2022-01-31T21:30:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:12 smithi167 conmon[54076]: debug 2022-01-31T21:30:12.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:12.205780+0000) 2022-01-31T21:30:12.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:12 smithi167 conmon[60316]: debug 2022-01-31T21:30:12.206+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:12.207532+0000) 2022-01-31T21:30:12.894 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:12 smithi171 conmon[51620]: debug 2022-01-31T21:30:12.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:12.815041+0000) 2022-01-31T21:30:13.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:12 smithi167 conmon[49112]: debug 2022-01-31T21:30:12.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:12.792732+0000) 2022-01-31T21:30:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:12 smithi171 conmon[41853]: debug 2022-01-31T21:30:12.936+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:12.937002+0000) 2022-01-31T21:30:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:12 smithi171 conmon[46715]: debug 2022-01-31T21:30:12.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:12.982008+0000) 2022-01-31T21:30:13.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:13 smithi167 conmon[54076]: debug 2022-01-31T21:30:13.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:13.205942+0000) 2022-01-31T21:30:13.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:13 smithi167 conmon[60316]: debug 2022-01-31T21:30:13.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:13.207711+0000) 2022-01-31T21:30:13.893 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:13 smithi171 conmon[51620]: debug 2022-01-31T21:30:13.814+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:13.815196+0000) 2022-01-31T21:30:14.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:13 smithi167 conmon[49112]: debug 2022-01-31T21:30:13.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:13.792855+0000) 2022-01-31T21:30:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:13 smithi171 conmon[46715]: debug 2022-01-31T21:30:13.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:13.982192+0000) 2022-01-31T21:30:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:13 smithi171 conmon[41853]: debug 2022-01-31T21:30:13.936+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:13.937188+0000) 2022-01-31T21:30:14.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:14 smithi167 conmon[60316]: debug 2022-01-31T21:30:14.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:14.207856+0000) 2022-01-31T21:30:14.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:14 smithi167 conmon[54076]: debug 2022-01-31T21:30:14.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:14.206140+0000) 2022-01-31T21:30:14.894 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:14 smithi171 conmon[51620]: debug 2022-01-31T21:30:14.813+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:14.815403+0000) 2022-01-31T21:30:15.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:14 smithi167 conmon[49112]: debug 2022-01-31T21:30:14.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:14.793055+0000) 2022-01-31T21:30:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:14 smithi171 conmon[46715]: debug 2022-01-31T21:30:14.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:14.982378+0000) 2022-01-31T21:30:15.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:14 smithi171 conmon[41853]: debug 2022-01-31T21:30:14.936+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:14.937354+0000) 2022-01-31T21:30:15.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:15 smithi167 conmon[60316]: debug 2022-01-31T21:30:15.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.208040+0000) 2022-01-31T21:30:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:15 smithi167 conmon[54076]: debug 2022-01-31T21:30:15.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.206326+0000) 2022-01-31T21:30:15.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:15 smithi167 conmon[49112]: debug 2022-01-31T21:30:15.542+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.542807+0000) 2022-01-31T21:30:15.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:15 smithi167 conmon[54076]: debug 2022-01-31T21:30:15.542+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.542651+0000) 2022-01-31T21:30:15.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:15 smithi167 conmon[60316]: debug 2022-01-31T21:30:15.542+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.543228+0000) 2022-01-31T21:30:15.813 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[35325]: debug 2022-01-31T21:30:15.555+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 134956 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:15.814 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[46715]: debug 2022-01-31T21:30:15.540+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.542529+0000) 2022-01-31T21:30:15.815 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[51620]: debug 2022-01-31T21:30:15.541+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.543244+0000) 2022-01-31T21:30:15.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[41853]: debug 2022-01-31T21:30:15.542+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.544224+0000) 2022-01-31T21:30:16.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[41853]: debug 2022-01-31T21:30:15.936+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.937509+0000) 2022-01-31T21:30:16.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[51620]: debug 2022-01-31T21:30:15.814+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.815591+0000) 2022-01-31T21:30:16.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:15 smithi171 conmon[46715]: debug 2022-01-31T21:30:15.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.982546+0000) 2022-01-31T21:30:16.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:15 smithi167 conmon[49112]: debug 2022-01-31T21:30:15.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:15.793250+0000) 2022-01-31T21:30:16.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:16 smithi167 conmon[54076]: debug 2022-01-31T21:30:16.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:16.206553+0000) 2022-01-31T21:30:16.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:16 smithi167 conmon[60316]: debug 2022-01-31T21:30:16.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:16.208249+0000) 2022-01-31T21:30:16.894 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:16 smithi171 conmon[51620]: debug 2022-01-31T21:30:16.815+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:16.815778+0000) 2022-01-31T21:30:17.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:16 smithi167 conmon[49112]: debug 2022-01-31T21:30:16.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:16.793405+0000) 2022-01-31T21:30:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:16 smithi171 conmon[41853]: debug 2022-01-31T21:30:16.936+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:16.937683+0000) 2022-01-31T21:30:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:16 smithi171 conmon[46715]: debug 2022-01-31T21:30:16.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:16.982718+0000) 2022-01-31T21:30:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:17 smithi167 conmon[60316]: debug 2022-01-31T21:30:17.207+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:17.208406+0000) 2022-01-31T21:30:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:17 smithi167 conmon[54076]: debug 2022-01-31T21:30:17.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:17.206712+0000) 2022-01-31T21:30:17.894 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:17 smithi171 conmon[51620]: debug 2022-01-31T21:30:17.815+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:17.815891+0000) 2022-01-31T21:30:18.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:17 smithi167 conmon[49112]: debug 2022-01-31T21:30:17.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:17.793602+0000) 2022-01-31T21:30:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:17 smithi171 conmon[46715]: debug 2022-01-31T21:30:17.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:17.982899+0000) 2022-01-31T21:30:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:17 smithi171 conmon[41853]: debug 2022-01-31T21:30:17.937+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:17.937855+0000) 2022-01-31T21:30:18.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:18 smithi167 conmon[60316]: debug 2022-01-31T21:30:18.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:18.208599+0000) 2022-01-31T21:30:18.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:18 smithi167 conmon[54076]: debug 2022-01-31T21:30:18.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:18.206928+0000) 2022-01-31T21:30:18.894 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:18 smithi171 conmon[51620]: debug 2022-01-31T21:30:18.815+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:18.816088+0000) 2022-01-31T21:30:19.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:18 smithi167 conmon[49112]: debug 2022-01-31T21:30:18.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:18.793762+0000) 2022-01-31T21:30:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:18 smithi171 conmon[46715]: debug 2022-01-31T21:30:18.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:18.983064+0000) 2022-01-31T21:30:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:18 smithi171 conmon[41853]: debug 2022-01-31T21:30:18.937+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:18.938011+0000) 2022-01-31T21:30:19.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:19 smithi167 conmon[54076]: debug 2022-01-31T21:30:19.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:19.207075+0000) 2022-01-31T21:30:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:19 smithi167 conmon[60316]: debug 2022-01-31T21:30:19.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:19.208719+0000) 2022-01-31T21:30:19.894 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:19 smithi171 conmon[51620]: debug 2022-01-31T21:30:19.815+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:19.816225+0000) 2022-01-31T21:30:20.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:19 smithi167 conmon[49112]: debug 2022-01-31T21:30:19.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:19.793907+0000) 2022-01-31T21:30:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:19 smithi171 conmon[46715]: debug 2022-01-31T21:30:19.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:19.983259+0000) 2022-01-31T21:30:20.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:19 smithi171 conmon[41853]: debug 2022-01-31T21:30:19.937+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:19.938202+0000) 2022-01-31T21:30:20.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:20 smithi167 conmon[54076]: debug 2022-01-31T21:30:20.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.207267+0000) 2022-01-31T21:30:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:20 smithi167 conmon[60316]: debug 2022-01-31T21:30:20.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.208876+0000) 2022-01-31T21:30:20.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:20 smithi167 conmon[49112]: debug 2022-01-31T21:30:20.559+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.560052+0000) 2022-01-31T21:30:20.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:20 smithi167 conmon[54076]: debug 2022-01-31T21:30:20.560+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.561074+0000) 2022-01-31T21:30:20.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:20 smithi167 conmon[60316]: debug 2022-01-31T21:30:20.559+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.560143+0000) 2022-01-31T21:30:20.814 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[35325]: debug 2022-01-31T21:30:20.573+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135069 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:20.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[41853]: debug 2022-01-31T21:30:20.559+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.560073+0000) 2022-01-31T21:30:20.816 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[46715]: debug 2022-01-31T21:30:20.559+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.560196+0000) 2022-01-31T21:30:20.816 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[51620]: debug 2022-01-31T21:30:20.558+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.559396+0000) 2022-01-31T21:30:21.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[46715]: debug 2022-01-31T21:30:20.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.983433+0000) 2022-01-31T21:30:21.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[41853]: debug 2022-01-31T21:30:20.937+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.938408+0000) 2022-01-31T21:30:21.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:20 smithi171 conmon[51620]: debug 2022-01-31T21:30:20.815+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.816421+0000) 2022-01-31T21:30:21.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:20 smithi167 conmon[49112]: debug 2022-01-31T21:30:20.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:20.794069+0000) 2022-01-31T21:30:21.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:21 smithi167 conmon[54076]: debug 2022-01-31T21:30:21.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:21.207406+0000) 2022-01-31T21:30:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:21 smithi167 conmon[60316]: debug 2022-01-31T21:30:21.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:21.209075+0000) 2022-01-31T21:30:21.895 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:21 smithi171 conmon[51620]: debug 2022-01-31T21:30:21.815+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:21.816579+0000) 2022-01-31T21:30:22.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:21 smithi167 conmon[49112]: debug 2022-01-31T21:30:21.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:21.794168+0000) 2022-01-31T21:30:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:21 smithi171 conmon[41853]: debug 2022-01-31T21:30:21.937+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:21.938573+0000) 2022-01-31T21:30:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:21 smithi171 conmon[46715]: debug 2022-01-31T21:30:21.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:21.983578+0000) 2022-01-31T21:30:22.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:22 smithi167 conmon[54076]: debug 2022-01-31T21:30:22.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:22.207540+0000) 2022-01-31T21:30:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:22 smithi167 conmon[60316]: debug 2022-01-31T21:30:22.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:22.209239+0000) 2022-01-31T21:30:22.895 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:22 smithi171 conmon[51620]: debug 2022-01-31T21:30:22.816+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:22.816797+0000) 2022-01-31T21:30:23.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:22 smithi167 conmon[49112]: debug 2022-01-31T21:30:22.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:22.794384+0000) 2022-01-31T21:30:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:22 smithi171 conmon[41853]: debug 2022-01-31T21:30:22.938+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:22.938767+0000) 2022-01-31T21:30:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:22 smithi171 conmon[46715]: debug 2022-01-31T21:30:22.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:22.983789+0000) 2022-01-31T21:30:23.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:23 smithi167 conmon[54076]: debug 2022-01-31T21:30:23.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:23.207750+0000) 2022-01-31T21:30:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:23 smithi167 conmon[60316]: debug 2022-01-31T21:30:23.208+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:23.209421+0000) 2022-01-31T21:30:23.895 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:23 smithi171 conmon[51620]: debug 2022-01-31T21:30:23.816+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:23.816996+0000) 2022-01-31T21:30:23.896 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:23 smithi171 conmon[35325]: debug 2022-01-31T21:30:23.690+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:30:24.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:23 smithi167 conmon[49112]: debug 2022-01-31T21:30:23.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:23.794590+0000) 2022-01-31T21:30:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:23 smithi171 conmon[46715]: debug 2022-01-31T21:30:23.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:23.983974+0000) 2022-01-31T21:30:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:23 smithi171 conmon[41853]: debug 2022-01-31T21:30:23.938+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:23.938972+0000) 2022-01-31T21:30:24.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:24 smithi167 conmon[54076]: debug 2022-01-31T21:30:24.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:24.207958+0000) 2022-01-31T21:30:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:24 smithi167 conmon[60316]: debug 2022-01-31T21:30:24.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:24.209554+0000) 2022-01-31T21:30:24.895 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:24 smithi171 conmon[51620]: debug 2022-01-31T21:30:24.816+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:24.817180+0000) 2022-01-31T21:30:25.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:24 smithi167 conmon[49112]: debug 2022-01-31T21:30:24.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:24.794750+0000) 2022-01-31T21:30:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:24 smithi171 conmon[41853]: debug 2022-01-31T21:30:24.938+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:24.939121+0000) 2022-01-31T21:30:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:24 smithi171 conmon[46715]: debug 2022-01-31T21:30:24.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:24.984177+0000) 2022-01-31T21:30:25.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:25 smithi167 conmon[54076]: debug 2022-01-31T21:30:25.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.208144+0000) 2022-01-31T21:30:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:25 smithi167 conmon[60316]: debug 2022-01-31T21:30:25.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.209715+0000) 2022-01-31T21:30:25.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:25 smithi167 conmon[49112]: debug 2022-01-31T21:30:25.576+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.577219+0000) 2022-01-31T21:30:25.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:25 smithi167 conmon[54076]: debug 2022-01-31T21:30:25.577+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.577746+0000) 2022-01-31T21:30:25.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:25 smithi167 conmon[60316]: debug 2022-01-31T21:30:25.576+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.577030+0000) 2022-01-31T21:30:25.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[35325]: debug 2022-01-31T21:30:25.590+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135177 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[41853]: debug 2022-01-31T21:30:25.576+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.576776+0000) 2022-01-31T21:30:25.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[46715]: debug 2022-01-31T21:30:25.575+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.576014+0000) 2022-01-31T21:30:25.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[51620]: debug 2022-01-31T21:30:25.574+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.575705+0000) 2022-01-31T21:30:25.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[51620]: debug 2022-01-31T21:30:25.816+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.817372+0000) 2022-01-31T21:30:26.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:25 smithi167 conmon[49112]: debug 2022-01-31T21:30:25.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.794946+0000) 2022-01-31T21:30:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[41853]: debug 2022-01-31T21:30:25.938+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.939293+0000) 2022-01-31T21:30:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:25 smithi171 conmon[46715]: debug 2022-01-31T21:30:25.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:25.984345+0000) 2022-01-31T21:30:26.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:26 smithi167 conmon[60316]: debug 2022-01-31T21:30:26.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:26.209906+0000) 2022-01-31T21:30:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:26 smithi167 conmon[54076]: debug 2022-01-31T21:30:26.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:26.208309+0000) 2022-01-31T21:30:26.896 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:26 smithi171 conmon[51620]: debug 2022-01-31T21:30:26.816+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:26.817530+0000) 2022-01-31T21:30:27.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:26 smithi167 conmon[49112]: debug 2022-01-31T21:30:26.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:26.795107+0000) 2022-01-31T21:30:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:26 smithi171 conmon[41853]: debug 2022-01-31T21:30:26.938+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:26.939466+0000) 2022-01-31T21:30:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:26 smithi171 conmon[46715]: debug 2022-01-31T21:30:26.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:26.984528+0000) 2022-01-31T21:30:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:27 smithi167 conmon[60316]: debug 2022-01-31T21:30:27.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:27.210067+0000) 2022-01-31T21:30:27.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:27 smithi167 conmon[54076]: debug 2022-01-31T21:30:27.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:27.208467+0000) 2022-01-31T21:30:27.896 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:27 smithi171 conmon[51620]: debug 2022-01-31T21:30:27.817+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:27.817730+0000) 2022-01-31T21:30:28.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:27 smithi167 conmon[49112]: debug 2022-01-31T21:30:27.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:27.795312+0000) 2022-01-31T21:30:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:27 smithi171 conmon[46715]: debug 2022-01-31T21:30:27.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:27.984682+0000) 2022-01-31T21:30:28.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:27 smithi171 conmon[41853]: debug 2022-01-31T21:30:27.939+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:27.939648+0000) 2022-01-31T21:30:28.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:28 smithi167 conmon[54076]: debug 2022-01-31T21:30:28.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:28.208651+0000) 2022-01-31T21:30:28.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:28 smithi167 conmon[60316]: debug 2022-01-31T21:30:28.209+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:28.210281+0000) 2022-01-31T21:30:28.896 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:28 smithi171 conmon[51620]: debug 2022-01-31T21:30:28.817+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:28.817930+0000) 2022-01-31T21:30:29.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:28 smithi167 conmon[49112]: debug 2022-01-31T21:30:28.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:28.795535+0000) 2022-01-31T21:30:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:28 smithi171 conmon[41853]: debug 2022-01-31T21:30:28.939+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:28.939856+0000) 2022-01-31T21:30:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:28 smithi171 conmon[46715]: debug 2022-01-31T21:30:28.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:28.984834+0000) 2022-01-31T21:30:29.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:29 smithi167 conmon[60316]: debug 2022-01-31T21:30:29.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:29.210520+0000) 2022-01-31T21:30:29.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:29 smithi167 conmon[54076]: debug 2022-01-31T21:30:29.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:29.208775+0000) 2022-01-31T21:30:29.896 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:29 smithi171 conmon[51620]: debug 2022-01-31T21:30:29.817+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:29.818101+0000) 2022-01-31T21:30:30.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:29 smithi167 conmon[49112]: debug 2022-01-31T21:30:29.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:29.795750+0000) 2022-01-31T21:30:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:29 smithi171 conmon[41853]: debug 2022-01-31T21:30:29.939+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:29.940047+0000) 2022-01-31T21:30:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:29 smithi171 conmon[46715]: debug 2022-01-31T21:30:29.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:29.984952+0000) 2022-01-31T21:30:30.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:30 smithi167 conmon[60316]: debug 2022-01-31T21:30:30.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.210686+0000) 2022-01-31T21:30:30.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:30 smithi167 conmon[54076]: debug 2022-01-31T21:30:30.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.208938+0000) 2022-01-31T21:30:30.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:30 smithi167 conmon[54076]: debug 2022-01-31T21:30:30.594+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.594650+0000) 2022-01-31T21:30:30.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:30 smithi167 conmon[60316]: debug 2022-01-31T21:30:30.591+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.592425+0000) 2022-01-31T21:30:30.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:30 smithi167 conmon[49112]: debug 2022-01-31T21:30:30.602+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.603251+0000) 2022-01-31T21:30:30.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[35325]: debug 2022-01-31T21:30:30.612+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135290 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[41853]: debug 2022-01-31T21:30:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[41853]: 2022-01-31T21:30:30.592+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.593040+0000) 2022-01-31T21:30:30.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[46715]: debug 2022-01-31T21:30:30.591+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.592563+0000) 2022-01-31T21:30:30.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[51620]: debug 2022-01-31T21:30:30.595+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.596528+0000) 2022-01-31T21:30:30.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[51620]: debug 2022-01-31T21:30:30.817+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.818300+0000) 2022-01-31T21:30:31.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:30 smithi167 conmon[49112]: debug 2022-01-31T21:30:30.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.795896+0000) 2022-01-31T21:30:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[41853]: debug 2022-01-31T21:30:30.939+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.940195+0000) 2022-01-31T21:30:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:30 smithi171 conmon[46715]: debug 2022-01-31T21:30:30.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:30.985121+0000) 2022-01-31T21:30:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:31 smithi167 conmon[54076]: debug 2022-01-31T21:30:31.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:31.209090+0000) 2022-01-31T21:30:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:31 smithi167 conmon[60316]: debug 2022-01-31T21:30:31.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:31.210856+0000) 2022-01-31T21:30:32.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:31 smithi171 conmon[41853]: debug 2022-01-31T21:30:31.939+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:31.940361+0000) 2022-01-31T21:30:32.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:31 smithi171 conmon[46715]: debug 2022-01-31T21:30:31.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:31.985255+0000) 2022-01-31T21:30:32.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:31 smithi171 conmon[51620]: debug 2022-01-31T21:30:31.817+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:31.818460+0000) 2022-01-31T21:30:32.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:31 smithi167 conmon[49112]: debug 2022-01-31T21:30:31.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:31.796055+0000) 2022-01-31T21:30:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:32 smithi167 conmon[54076]: debug 2022-01-31T21:30:32.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:32.209247+0000) 2022-01-31T21:30:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:32 smithi167 conmon[60316]: debug 2022-01-31T21:30:32.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:32.211011+0000) 2022-01-31T21:30:32.897 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:32 smithi171 conmon[51620]: debug 2022-01-31T21:30:32.817+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:32.818655+0000) 2022-01-31T21:30:33.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:32 smithi167 conmon[49112]: debug 2022-01-31T21:30:32.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:32.796260+0000) 2022-01-31T21:30:33.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:32 smithi171 conmon[41853]: debug 2022-01-31T21:30:32.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:32.940568+0000) 2022-01-31T21:30:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:32 smithi171 conmon[46715]: debug 2022-01-31T21:30:32.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:32.985433+0000) 2022-01-31T21:30:33.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:33 smithi167 conmon[60316]: debug 2022-01-31T21:30:33.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:33.211215+0000) 2022-01-31T21:30:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:33 smithi167 conmon[54076]: debug 2022-01-31T21:30:33.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:33.209431+0000) 2022-01-31T21:30:33.897 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:33 smithi171 conmon[51620]: debug 2022-01-31T21:30:33.818+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:33.818860+0000) 2022-01-31T21:30:34.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:33 smithi167 conmon[49112]: debug 2022-01-31T21:30:33.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:33.796426+0000) 2022-01-31T21:30:34.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:33 smithi171 conmon[41853]: debug 2022-01-31T21:30:33.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:33.940770+0000) 2022-01-31T21:30:34.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:33 smithi171 conmon[46715]: debug 2022-01-31T21:30:33.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:33.985654+0000) 2022-01-31T21:30:34.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:34 smithi167 conmon[54076]: debug 2022-01-31T21:30:34.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:34.209653+0000) 2022-01-31T21:30:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:34 smithi167 conmon[60316]: debug 2022-01-31T21:30:34.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:34.211443+0000) 2022-01-31T21:30:34.897 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:34 smithi171 conmon[51620]: debug 2022-01-31T21:30:34.818+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:34.819025+0000) 2022-01-31T21:30:35.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:34 smithi167 conmon[49112]: debug 2022-01-31T21:30:34.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:34.796643+0000) 2022-01-31T21:30:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:34 smithi171 conmon[46715]: debug 2022-01-31T21:30:34.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:34.985844+0000) 2022-01-31T21:30:35.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:34 smithi171 conmon[41853]: debug 2022-01-31T21:30:34.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:34.940954+0000) 2022-01-31T21:30:35.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:35 smithi167 conmon[54076]: debug 2022-01-31T21:30:35.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.209860+0000) 2022-01-31T21:30:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:35 smithi167 conmon[60316]: debug 2022-01-31T21:30:35.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.211609+0000) 2022-01-31T21:30:35.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:35 smithi167 conmon[49112]: debug 2022-01-31T21:30:35.616+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.616613+0000) 2022-01-31T21:30:35.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:35 smithi167 conmon[54076]: debug 2022-01-31T21:30:35.615+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.616356+0000) 2022-01-31T21:30:35.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:35 smithi167 conmon[60316]: debug 2022-01-31T21:30:35.615+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.616376+0000) 2022-01-31T21:30:35.897 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[35325]: debug 2022-01-31T21:30:35.629+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135399 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:35.898 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[41853]: debug 2022-01-31T21:30:35.615+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.615982+0000) 2022-01-31T21:30:35.899 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[46715]: debug 2022-01-31T21:30:35.614+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.615703+0000) 2022-01-31T21:30:35.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[51620]: debug 2022-01-31T21:30:35.615+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.616288+0000) 2022-01-31T21:30:35.900 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[51620]: debug 2022-01-31T21:30:35.818+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.819260+0000) 2022-01-31T21:30:36.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:35 smithi167 conmon[49112]: debug 2022-01-31T21:30:35.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.796840+0000) 2022-01-31T21:30:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[46715]: debug 2022-01-31T21:30:35.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.986001+0000) 2022-01-31T21:30:36.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:35 smithi171 conmon[41853]: debug 2022-01-31T21:30:35.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:35.941112+0000) 2022-01-31T21:30:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:36 smithi167 conmon[54076]: debug 2022-01-31T21:30:36.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:36.210049+0000) 2022-01-31T21:30:36.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:36 smithi167 conmon[60316]: debug 2022-01-31T21:30:36.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:36.211778+0000) 2022-01-31T21:30:36.897 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:36 smithi171 conmon[51620]: debug 2022-01-31T21:30:36.819+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:36.819427+0000) 2022-01-31T21:30:37.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:36 smithi167 conmon[49112]: debug 2022-01-31T21:30:36.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:36.797021+0000) 2022-01-31T21:30:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:36 smithi171 conmon[46715]: debug 2022-01-31T21:30:36.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:36.986168+0000) 2022-01-31T21:30:37.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:36 smithi171 conmon[41853]: debug 2022-01-31T21:30:36.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:36.941288+0000) 2022-01-31T21:30:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:37 smithi167 conmon[54076]: debug 2022-01-31T21:30:37.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:37.210186+0000) 2022-01-31T21:30:37.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:37 smithi167 conmon[60316]: debug 2022-01-31T21:30:37.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:37.211891+0000) 2022-01-31T21:30:37.898 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:37 smithi171 conmon[51620]: debug 2022-01-31T21:30:37.819+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:37.819609+0000) 2022-01-31T21:30:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:37 smithi167 conmon[49112]: debug 2022-01-31T21:30:37.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:37.797237+0000) 2022-01-31T21:30:38.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:37 smithi171 conmon[41853]: debug 2022-01-31T21:30:37.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:37.941488+0000) 2022-01-31T21:30:38.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:37 smithi171 conmon[46715]: debug 2022-01-31T21:30:37.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:37.986322+0000) 2022-01-31T21:30:38.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:38 smithi167 conmon[54076]: debug 2022-01-31T21:30:38.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:38.210390+0000) 2022-01-31T21:30:38.451 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:38 smithi167 conmon[60316]: debug 2022-01-31T21:30:38.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:38.212087+0000) 2022-01-31T21:30:38.898 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:38 smithi171 conmon[35325]: debug 2022-01-31T21:30:38.691+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:30:38.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:38 smithi171 conmon[51620]: debug 2022-01-31T21:30:38.819+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:38.819784+0000) 2022-01-31T21:30:39.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:38 smithi167 conmon[49112]: debug 2022-01-31T21:30:38.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:38.797452+0000) 2022-01-31T21:30:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:38 smithi171 conmon[41853]: debug 2022-01-31T21:30:38.940+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:38.941643+0000) 2022-01-31T21:30:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:38 smithi171 conmon[46715]: debug 2022-01-31T21:30:38.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:38.986470+0000) 2022-01-31T21:30:39.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:39 smithi167 conmon[54076]: debug 2022-01-31T21:30:39.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:39.210552+0000) 2022-01-31T21:30:39.452 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:39 smithi167 conmon[60316]: debug 2022-01-31T21:30:39.210+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:39.212304+0000) 2022-01-31T21:30:39.898 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:39 smithi171 conmon[51620]: debug 2022-01-31T21:30:39.819+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:39.819962+0000) 2022-01-31T21:30:40.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:39 smithi167 conmon[49112]: debug 2022-01-31T21:30:39.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:39.797701+0000) 2022-01-31T21:30:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:39 smithi171 conmon[41853]: debug 2022-01-31T21:30:39.941+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:39.941839+0000) 2022-01-31T21:30:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:39 smithi171 conmon[46715]: debug 2022-01-31T21:30:39.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:39.986675+0000) 2022-01-31T21:30:40.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:40 smithi167 conmon[54076]: debug 2022-01-31T21:30:40.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.210657+0000) 2022-01-31T21:30:40.452 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:40 smithi167 conmon[60316]: debug 2022-01-31T21:30:40.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.212518+0000) 2022-01-31T21:30:40.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:40 smithi167 conmon[49112]: debug 2022-01-31T21:30:40.631+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.633127+0000) 2022-01-31T21:30:40.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:40 smithi167 conmon[54076]: debug 2022-01-31T21:30:40.631+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.632774+0000) 2022-01-31T21:30:40.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:40 smithi167 conmon[60316]: debug 2022-01-31T21:30:40.631+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.633406+0000) 2022-01-31T21:30:40.898 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[35325]: debug 2022-01-31T21:30:40.646+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135512 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:40.899 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[41853]: debug 2022-01-31T21:30:40.631+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.631771+0000) 2022-01-31T21:30:40.900 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[46715]: debug 2022-01-31T21:30:40.631+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.632177+0000) 2022-01-31T21:30:40.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[51620]: debug 2022-01-31T21:30:40.631+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.631995+0000) 2022-01-31T21:30:40.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[51620]: debug 2022-01-31T21:30:40.819+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.820125+0000) 2022-01-31T21:30:41.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:40 smithi167 conmon[49112]: debug 2022-01-31T21:30:40.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.797841+0000) 2022-01-31T21:30:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[46715]: debug 2022-01-31T21:30:40.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.986807+0000) 2022-01-31T21:30:41.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:40 smithi171 conmon[41853]: debug 2022-01-31T21:30:40.941+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:40.942017+0000) 2022-01-31T21:30:41.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:41 smithi167 conmon[54076]: debug 2022-01-31T21:30:41.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:41.210817+0000) 2022-01-31T21:30:41.452 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:41 smithi167 conmon[60316]: debug 2022-01-31T21:30:41.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:41.212637+0000) 2022-01-31T21:30:41.898 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:41 smithi171 conmon[51620]: debug 2022-01-31T21:30:41.819+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:41.820255+0000) 2022-01-31T21:30:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:41 smithi167 conmon[49112]: debug 2022-01-31T21:30:41.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:41.798049+0000) 2022-01-31T21:30:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:41 smithi171 conmon[41853]: debug 2022-01-31T21:30:41.941+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:41.942189+0000) 2022-01-31T21:30:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:41 smithi171 conmon[46715]: debug 2022-01-31T21:30:41.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:41.986977+0000) 2022-01-31T21:30:42.451 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:42 smithi167 conmon[60316]: debug 2022-01-31T21:30:42.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:42.212756+0000) 2022-01-31T21:30:42.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:42 smithi167 conmon[54076]: debug 2022-01-31T21:30:42.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:42.210992+0000) 2022-01-31T21:30:42.898 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:42 smithi171 conmon[51620]: debug 2022-01-31T21:30:42.820+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:42.820412+0000) 2022-01-31T21:30:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:42 smithi167 conmon[49112]: debug 2022-01-31T21:30:42.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:42.798261+0000) 2022-01-31T21:30:43.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:42 smithi171 conmon[41853]: debug 2022-01-31T21:30:42.941+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:42.942422+0000) 2022-01-31T21:30:43.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:42 smithi171 conmon[46715]: debug 2022-01-31T21:30:42.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:42.987169+0000) 2022-01-31T21:30:43.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:43 smithi167 conmon[54076]: debug 2022-01-31T21:30:43.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:43.211180+0000) 2022-01-31T21:30:43.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:43 smithi167 conmon[60316]: debug 2022-01-31T21:30:43.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:43.212868+0000) 2022-01-31T21:30:43.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:43 smithi171 conmon[51620]: debug 2022-01-31T21:30:43.820+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:43.820602+0000) 2022-01-31T21:30:44.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:43 smithi167 conmon[49112]: debug 2022-01-31T21:30:43.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:43.798474+0000) 2022-01-31T21:30:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:43 smithi171 conmon[41853]: debug 2022-01-31T21:30:43.942+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:43.942580+0000) 2022-01-31T21:30:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:43 smithi171 conmon[46715]: debug 2022-01-31T21:30:43.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:43.987357+0000) 2022-01-31T21:30:44.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:44 smithi167 conmon[54076]: debug 2022-01-31T21:30:44.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:44.211369+0000) 2022-01-31T21:30:44.452 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:44 smithi167 conmon[60316]: debug 2022-01-31T21:30:44.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:44.213047+0000) 2022-01-31T21:30:44.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:44 smithi171 conmon[51620]: debug 2022-01-31T21:30:44.820+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:44.820740+0000) 2022-01-31T21:30:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:44 smithi167 conmon[49112]: debug 2022-01-31T21:30:44.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:44.798714+0000) 2022-01-31T21:30:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:44 smithi171 conmon[41853]: debug 2022-01-31T21:30:44.942+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:44.942784+0000) 2022-01-31T21:30:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:44 smithi171 conmon[46715]: debug 2022-01-31T21:30:44.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:44.987529+0000) 2022-01-31T21:30:45.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:45 smithi167 conmon[54076]: debug 2022-01-31T21:30:45.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.211572+0000) 2022-01-31T21:30:45.452 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:45 smithi167 conmon[60316]: debug 2022-01-31T21:30:45.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.213169+0000) 2022-01-31T21:30:45.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:45 smithi167 conmon[49112]: debug 2022-01-31T21:30:45.648+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.649809+0000) 2022-01-31T21:30:45.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:45 smithi167 conmon[54076]: debug 2022-01-31T21:30:45.648+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.649542+0000) 2022-01-31T21:30:45.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:45 smithi167 conmon[60316]: debug 2022-01-31T21:30:45.648+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.649704+0000) 2022-01-31T21:30:45.899 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[35325]: debug 2022-01-31T21:30:45.663+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135621 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:45.900 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[41853]: debug 2022-01-31T21:30:45.648+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.649568+0000) 2022-01-31T21:30:45.900 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[46715]: debug 2022-01-31T21:30:45.648+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.649236+0000) 2022-01-31T21:30:45.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[51620]: debug 2022-01-31T21:30:45.649+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.649862+0000) 2022-01-31T21:30:45.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[51620]: debug 2022-01-31T21:30:45.820+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.820940+0000) 2022-01-31T21:30:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:45 smithi167 conmon[49112]: debug 2022-01-31T21:30:45.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.798906+0000) 2022-01-31T21:30:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[41853]: debug 2022-01-31T21:30:45.942+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.943004+0000) 2022-01-31T21:30:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:45 smithi171 conmon[46715]: debug 2022-01-31T21:30:45.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:45.987686+0000) 2022-01-31T21:30:46.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:46 smithi167 conmon[54076]: debug 2022-01-31T21:30:46.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:46.211759+0000) 2022-01-31T21:30:46.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:46 smithi167 conmon[60316]: debug 2022-01-31T21:30:46.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:46.213344+0000) 2022-01-31T21:30:47.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:46 smithi171 conmon[41853]: debug 2022-01-31T21:30:46.942+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:46.943143+0000) 2022-01-31T21:30:47.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:46 smithi171 conmon[46715]: debug 2022-01-31T21:30:46.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:46.987818+0000) 2022-01-31T21:30:47.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:46 smithi171 conmon[51620]: debug 2022-01-31T21:30:46.820+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:46.821097+0000) 2022-01-31T21:30:47.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:46 smithi167 conmon[49112]: debug 2022-01-31T21:30:46.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:46.799070+0000) 2022-01-31T21:30:47.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:47 smithi167 conmon[54076]: debug 2022-01-31T21:30:47.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:47.211875+0000) 2022-01-31T21:30:47.451 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:47 smithi167 conmon[60316]: debug 2022-01-31T21:30:47.211+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:47.213507+0000) 2022-01-31T21:30:47.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:47 smithi171 conmon[51620]: debug 2022-01-31T21:30:47.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:47.821304+0000) 2022-01-31T21:30:48.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:47 smithi167 conmon[49112]: debug 2022-01-31T21:30:47.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:47.799270+0000) 2022-01-31T21:30:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:47 smithi171 conmon[46715]: debug 2022-01-31T21:30:47.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:47.987969+0000) 2022-01-31T21:30:48.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:47 smithi171 conmon[41853]: debug 2022-01-31T21:30:47.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:47.943311+0000) 2022-01-31T21:30:48.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:48 smithi167 conmon[54076]: debug 2022-01-31T21:30:48.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:48.212064+0000) 2022-01-31T21:30:48.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:48 smithi167 conmon[60316]: debug 2022-01-31T21:30:48.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:48.213608+0000) 2022-01-31T21:30:48.899 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:48 smithi171 conmon[51620]: debug 2022-01-31T21:30:48.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:48.821496+0000) 2022-01-31T21:30:49.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:48 smithi167 conmon[49112]: debug 2022-01-31T21:30:48.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:48.799415+0000) 2022-01-31T21:30:49.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:48 smithi171 conmon[41853]: debug 2022-01-31T21:30:48.942+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:48.943531+0000) 2022-01-31T21:30:49.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:48 smithi171 conmon[46715]: debug 2022-01-31T21:30:48.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:48.988127+0000) 2022-01-31T21:30:49.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:49 smithi167 conmon[54076]: debug 2022-01-31T21:30:49.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:49.212227+0000) 2022-01-31T21:30:49.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:49 smithi167 conmon[60316]: debug 2022-01-31T21:30:49.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:49.213732+0000) 2022-01-31T21:30:49.900 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:49 smithi171 conmon[51620]: debug 2022-01-31T21:30:49.820+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:49.821693+0000) 2022-01-31T21:30:50.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:49 smithi167 conmon[49112]: debug 2022-01-31T21:30:49.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:49.799583+0000) 2022-01-31T21:30:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:49 smithi171 conmon[41853]: debug 2022-01-31T21:30:49.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:49.943732+0000) 2022-01-31T21:30:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:49 smithi171 conmon[46715]: debug 2022-01-31T21:30:49.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:49.988310+0000) 2022-01-31T21:30:50.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:50 smithi167 conmon[54076]: debug 2022-01-31T21:30:50.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.212440+0000) 2022-01-31T21:30:50.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:50 smithi167 conmon[60316]: debug 2022-01-31T21:30:50.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.213836+0000) 2022-01-31T21:30:50.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:50 smithi167 conmon[49112]: debug 2022-01-31T21:30:50.664+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.666045+0000) 2022-01-31T21:30:50.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:50 smithi167 conmon[54076]: debug 2022-01-31T21:30:50.664+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.666385+0000) 2022-01-31T21:30:50.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:50 smithi167 conmon[60316]: debug 2022-01-31T21:30:50.665+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.667232+0000) 2022-01-31T21:30:50.900 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[35325]: debug 2022-01-31T21:30:50.680+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135729 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:50.901 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[41853]: debug 2022-01-31T21:30:50.666+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.667029+0000) 2022-01-31T21:30:50.901 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[46715]: debug 2022-01-31T21:30:50.665+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.666210+0000) 2022-01-31T21:30:50.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[51620]: debug 2022-01-31T21:30:50.665+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.665973+0000) 2022-01-31T21:30:50.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[51620]: debug 2022-01-31T21:30:50.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.821913+0000) 2022-01-31T21:30:51.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:50 smithi167 conmon[49112]: debug 2022-01-31T21:30:50.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.799758+0000) 2022-01-31T21:30:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[41853]: debug 2022-01-31T21:30:50.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.943957+0000) 2022-01-31T21:30:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:50 smithi171 conmon[46715]: debug 2022-01-31T21:30:50.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:50.988468+0000) 2022-01-31T21:30:51.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:51 smithi167 conmon[54076]: debug 2022-01-31T21:30:51.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:51.212615+0000) 2022-01-31T21:30:51.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:51 smithi167 conmon[60316]: debug 2022-01-31T21:30:51.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:51.214015+0000) 2022-01-31T21:30:51.900 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:51 smithi171 conmon[51620]: debug 2022-01-31T21:30:51.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:51.822061+0000) 2022-01-31T21:30:52.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:51 smithi167 conmon[49112]: debug 2022-01-31T21:30:51.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:51.799897+0000) 2022-01-31T21:30:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:51 smithi171 conmon[41853]: debug 2022-01-31T21:30:51.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:51.944092+0000) 2022-01-31T21:30:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:51 smithi171 conmon[46715]: debug 2022-01-31T21:30:51.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:51.988626+0000) 2022-01-31T21:30:52.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:52 smithi167 conmon[54076]: debug 2022-01-31T21:30:52.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:52.212768+0000) 2022-01-31T21:30:52.454 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:52 smithi167 conmon[60316]: debug 2022-01-31T21:30:52.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:52.214172+0000) 2022-01-31T21:30:52.900 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:52 smithi171 conmon[51620]: debug 2022-01-31T21:30:52.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:52.822260+0000) 2022-01-31T21:30:53.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:52 smithi167 conmon[49112]: debug 2022-01-31T21:30:52.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:52.800095+0000) 2022-01-31T21:30:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:52 smithi171 conmon[41853]: debug 2022-01-31T21:30:52.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:52.944288+0000) 2022-01-31T21:30:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:52 smithi171 conmon[46715]: debug 2022-01-31T21:30:52.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:52.988824+0000) 2022-01-31T21:30:53.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:53 smithi167 conmon[54076]: debug 2022-01-31T21:30:53.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:53.212892+0000) 2022-01-31T21:30:53.454 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:53 smithi167 conmon[60316]: debug 2022-01-31T21:30:53.212+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:53.214341+0000) 2022-01-31T21:30:53.900 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:53 smithi171 conmon[51620]: debug 2022-01-31T21:30:53.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:53.822466+0000) 2022-01-31T21:30:53.901 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:53 smithi171 conmon[35325]: debug 2022-01-31T21:30:53.692+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:30:54.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:53 smithi167 conmon[49112]: debug 2022-01-31T21:30:53.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:53.800296+0000) 2022-01-31T21:30:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:53 smithi171 conmon[41853]: debug 2022-01-31T21:30:53.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:53.944427+0000) 2022-01-31T21:30:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:53 smithi171 conmon[46715]: debug 2022-01-31T21:30:53.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:53.989030+0000) 2022-01-31T21:30:54.453 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:54 smithi167 conmon[60316]: debug 2022-01-31T21:30:54.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:54.214512+0000) 2022-01-31T21:30:54.454 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:54 smithi167 conmon[54076]: debug 2022-01-31T21:30:54.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:54.213060+0000) 2022-01-31T21:30:54.900 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:54 smithi171 conmon[51620]: debug 2022-01-31T21:30:54.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:54.822674+0000) 2022-01-31T21:30:55.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:54 smithi167 conmon[49112]: debug 2022-01-31T21:30:54.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:54.800521+0000) 2022-01-31T21:30:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:54 smithi171 conmon[41853]: debug 2022-01-31T21:30:54.944+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:54.944619+0000) 2022-01-31T21:30:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:54 smithi171 conmon[46715]: debug 2022-01-31T21:30:54.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:54.989210+0000) 2022-01-31T21:30:55.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:55 smithi167 conmon[54076]: debug 2022-01-31T21:30:55.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.213270+0000) 2022-01-31T21:30:55.454 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:55 smithi167 conmon[60316]: debug 2022-01-31T21:30:55.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.214640+0000) 2022-01-31T21:30:55.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:55 smithi167 conmon[49112]: debug 2022-01-31T21:30:55.683+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.684829+0000) 2022-01-31T21:30:55.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:55 smithi167 conmon[54076]: debug 2022-01-31T21:30:55.683+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.684738+0000) 2022-01-31T21:30:55.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:55 smithi167 conmon[60316]: debug 2022-01-31T21:30:55.682+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.684246+0000) 2022-01-31T21:30:55.901 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[35325]: debug 2022-01-31T21:30:55.697+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135838 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:30:55.901 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[41853]: debug 2022-01-31T21:30:55.683+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.684136+0000) 2022-01-31T21:30:55.902 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[46715]: debug 2022-01-31T21:30:55.683+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.684297+0000) 2022-01-31T21:30:55.903 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[51620]: debug 2022-01-31T21:30:55.683+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.683982+0000) 2022-01-31T21:30:55.903 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[51620]: debug 2022-01-31T21:30:55.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.822841+0000) 2022-01-31T21:30:56.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:55 smithi167 conmon[49112]: debug 2022-01-31T21:30:55.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.800668+0000) 2022-01-31T21:30:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[46715]: debug 2022-01-31T21:30:55.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.989382+0000) 2022-01-31T21:30:56.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:55 smithi171 conmon[41853]: debug 2022-01-31T21:30:55.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:55.944811+0000) 2022-01-31T21:30:56.454 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:56 smithi167 conmon[54076]: debug 2022-01-31T21:30:56.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:56.213422+0000) 2022-01-31T21:30:56.455 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:56 smithi167 conmon[60316]: debug 2022-01-31T21:30:56.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:56.214758+0000) 2022-01-31T21:30:56.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:56 smithi171 conmon[51620]: debug 2022-01-31T21:30:56.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:56.822993+0000) 2022-01-31T21:30:57.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:56 smithi167 conmon[49112]: debug 2022-01-31T21:30:56.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:56.800828+0000) 2022-01-31T21:30:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:56 smithi171 conmon[46715]: debug 2022-01-31T21:30:56.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:56.989551+0000) 2022-01-31T21:30:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:56 smithi171 conmon[41853]: debug 2022-01-31T21:30:56.943+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:56.944981+0000) 2022-01-31T21:30:57.454 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:57 smithi167 conmon[54076]: debug 2022-01-31T21:30:57.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:57.213592+0000) 2022-01-31T21:30:57.455 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:57 smithi167 conmon[60316]: debug 2022-01-31T21:30:57.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:57.214868+0000) 2022-01-31T21:30:57.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:57 smithi171 conmon[51620]: debug 2022-01-31T21:30:57.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:57.823181+0000) 2022-01-31T21:30:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:57 smithi167 conmon[49112]: debug 2022-01-31T21:30:57.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:57.801033+0000) 2022-01-31T21:30:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:57 smithi171 conmon[41853]: debug 2022-01-31T21:30:57.944+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:57.945184+0000) 2022-01-31T21:30:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:57 smithi171 conmon[46715]: debug 2022-01-31T21:30:57.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:57.989705+0000) 2022-01-31T21:30:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:58 smithi167 conmon[54076]: debug 2022-01-31T21:30:58.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:58.213715+0000) 2022-01-31T21:30:58.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:58 smithi167 conmon[60316]: debug 2022-01-31T21:30:58.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:58.215053+0000) 2022-01-31T21:30:58.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:58 smithi171 conmon[51620]: debug 2022-01-31T21:30:58.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:58.823352+0000) 2022-01-31T21:30:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:58 smithi167 conmon[49112]: debug 2022-01-31T21:30:58.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:58.801258+0000) 2022-01-31T21:30:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:58 smithi171 conmon[41853]: debug 2022-01-31T21:30:58.944+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:58.945377+0000) 2022-01-31T21:30:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:58 smithi171 conmon[46715]: debug 2022-01-31T21:30:58.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:58.989854+0000) 2022-01-31T21:30:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:30:59 smithi167 conmon[54076]: debug 2022-01-31T21:30:59.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:59.213907+0000) 2022-01-31T21:30:59.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:30:59 smithi167 conmon[60316]: debug 2022-01-31T21:30:59.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:59.215180+0000) 2022-01-31T21:30:59.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:30:59 smithi171 conmon[51620]: debug 2022-01-31T21:30:59.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:59.823507+0000) 2022-01-31T21:31:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:30:59 smithi167 conmon[49112]: debug 2022-01-31T21:30:59.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:59.801469+0000) 2022-01-31T21:31:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:30:59 smithi171 conmon[41853]: debug 2022-01-31T21:30:59.944+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:59.945603+0000) 2022-01-31T21:31:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:30:59 smithi171 conmon[46715]: debug 2022-01-31T21:30:59.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:30:59.990026+0000) 2022-01-31T21:31:00.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:00 smithi167 conmon[54076]: debug 2022-01-31T21:31:00.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.214125+0000) 2022-01-31T21:31:00.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:00 smithi167 conmon[60316]: debug 2022-01-31T21:31:00.213+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.215345+0000) 2022-01-31T21:31:00.902 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[35325]: debug 2022-01-31T21:31:00.713+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 135950 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:00.903 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[41853]: debug 2022-01-31T21:31:00.699+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.701433+0000) 2022-01-31T21:31:00.904 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[46715]: debug 2022-01-31T21:31:00.698+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.700515+0000) 2022-01-31T21:31:00.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[51620]: debug 2022-01-31T21:31:00.699+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.701144+0000) 2022-01-31T21:31:00.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[51620]: debug 2022-01-31T21:31:00.821+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.823696+0000) 2022-01-31T21:31:01.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:00 smithi167 conmon[54076]: debug 2022-01-31T21:31:00.699+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.701137+0000) 2022-01-31T21:31:01.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:00 smithi167 conmon[60316]: debug 2022-01-31T21:31:00.699+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.701397+0000) 2022-01-31T21:31:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:00 smithi167 conmon[49112]: debug 2022-01-31T21:31:00.699+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.701257+0000) 2022-01-31T21:31:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:00 smithi167 conmon[49112]: debug 2022-01-31T21:31:00.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.801728+0000) 2022-01-31T21:31:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[46715]: debug 2022-01-31T21:31:00.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.990233+0000) 2022-01-31T21:31:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:00 smithi171 conmon[41853]: debug 2022-01-31T21:31:00.944+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:00.945787+0000) 2022-01-31T21:31:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:01 smithi167 conmon[54076]: debug 2022-01-31T21:31:01.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:01.214323+0000) 2022-01-31T21:31:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:01 smithi167 conmon[60316]: debug 2022-01-31T21:31:01.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:01.215558+0000) 2022-01-31T21:31:01.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:01 smithi171 conmon[51620]: debug 2022-01-31T21:31:01.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:01.823861+0000) 2022-01-31T21:31:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:01 smithi167 conmon[49112]: debug 2022-01-31T21:31:01.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:01.801884+0000) 2022-01-31T21:31:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:01 smithi171 conmon[41853]: debug 2022-01-31T21:31:01.945+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:01.945950+0000) 2022-01-31T21:31:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:01 smithi171 conmon[46715]: debug 2022-01-31T21:31:01.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:01.990341+0000) 2022-01-31T21:31:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:02 smithi167 conmon[54076]: debug 2022-01-31T21:31:02.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:02.214452+0000) 2022-01-31T21:31:02.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:02 smithi167 conmon[60316]: debug 2022-01-31T21:31:02.214+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:02.215644+0000) 2022-01-31T21:31:02.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:02 smithi171 conmon[51620]: debug 2022-01-31T21:31:02.823+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:02.824082+0000) 2022-01-31T21:31:03.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:02 smithi167 conmon[49112]: debug 2022-01-31T21:31:02.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:02.802046+0000) 2022-01-31T21:31:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:02 smithi171 conmon[41853]: debug 2022-01-31T21:31:02.945+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:02.946131+0000) 2022-01-31T21:31:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:02 smithi171 conmon[46715]: debug 2022-01-31T21:31:02.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:02.990523+0000) 2022-01-31T21:31:03.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:03 smithi167 conmon[54076]: debug 2022-01-31T21:31:03.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:03.214579+0000) 2022-01-31T21:31:03.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:03 smithi167 conmon[60316]: debug 2022-01-31T21:31:03.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:03.215749+0000) 2022-01-31T21:31:03.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:03 smithi171 conmon[51620]: debug 2022-01-31T21:31:03.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:03.824312+0000) 2022-01-31T21:31:04.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:03 smithi167 conmon[49112]: debug 2022-01-31T21:31:03.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:03.802236+0000) 2022-01-31T21:31:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:03 smithi171 conmon[41853]: debug 2022-01-31T21:31:03.945+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:03.946339+0000) 2022-01-31T21:31:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:03 smithi171 conmon[46715]: debug 2022-01-31T21:31:03.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:03.990707+0000) 2022-01-31T21:31:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:04 smithi167 conmon[54076]: debug 2022-01-31T21:31:04.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:04.214739+0000) 2022-01-31T21:31:04.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:04 smithi167 conmon[60316]: debug 2022-01-31T21:31:04.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:04.215873+0000) 2022-01-31T21:31:04.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:04 smithi171 conmon[51620]: debug 2022-01-31T21:31:04.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:04.824498+0000) 2022-01-31T21:31:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:04 smithi167 conmon[49112]: debug 2022-01-31T21:31:04.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:04.802444+0000) 2022-01-31T21:31:05.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:04 smithi171 conmon[46715]: debug 2022-01-31T21:31:04.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:04.990927+0000) 2022-01-31T21:31:05.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:04 smithi171 conmon[41853]: debug 2022-01-31T21:31:04.945+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:04.946526+0000) 2022-01-31T21:31:05.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:05 smithi167 conmon[54076]: debug 2022-01-31T21:31:05.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.214942+0000) 2022-01-31T21:31:05.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:05 smithi167 conmon[60316]: debug 2022-01-31T21:31:05.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.216025+0000) 2022-01-31T21:31:05.902 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[35325]: debug 2022-01-31T21:31:05.735+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136059 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:05.903 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[41853]: debug 2022-01-31T21:31:05.716+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.718547+0000) 2022-01-31T21:31:05.904 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[46715]: debug 2022-01-31T21:31:05.715+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.717675+0000) 2022-01-31T21:31:05.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[51620]: debug 2022-01-31T21:31:05.716+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.717799+0000) 2022-01-31T21:31:05.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[51620]: debug 2022-01-31T21:31:05.822+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.824645+0000) 2022-01-31T21:31:06.167 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:05 smithi167 conmon[54076]: debug 2022-01-31T21:31:05.717+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.717913+0000) 2022-01-31T21:31:06.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:05 smithi167 conmon[60316]: debug 2022-01-31T21:31:05.717+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.718418+0000) 2022-01-31T21:31:06.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:05 smithi167 conmon[49112]: debug 2022-01-31T21:31:05.724+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.724893+0000) 2022-01-31T21:31:06.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:05 smithi167 conmon[49112]: debug 2022-01-31T21:31:05.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.802600+0000) 2022-01-31T21:31:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[41853]: debug 2022-01-31T21:31:05.945+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.946772+0000) 2022-01-31T21:31:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:05 smithi171 conmon[46715]: debug 2022-01-31T21:31:05.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:05.991083+0000) 2022-01-31T21:31:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:06 smithi167 conmon[54076]: debug 2022-01-31T21:31:06.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:06.215091+0000) 2022-01-31T21:31:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:06 smithi167 conmon[60316]: debug 2022-01-31T21:31:06.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:06.216216+0000) 2022-01-31T21:31:06.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:06 smithi171 conmon[51620]: debug 2022-01-31T21:31:06.823+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:06.824833+0000) 2022-01-31T21:31:07.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:06 smithi167 conmon[49112]: debug 2022-01-31T21:31:06.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:06.802754+0000) 2022-01-31T21:31:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:06 smithi171 conmon[41853]: debug 2022-01-31T21:31:06.946+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:06.946932+0000) 2022-01-31T21:31:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:06 smithi171 conmon[46715]: debug 2022-01-31T21:31:06.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:06.991248+0000) 2022-01-31T21:31:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:07 smithi167 conmon[54076]: debug 2022-01-31T21:31:07.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:07.215251+0000) 2022-01-31T21:31:07.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:07 smithi167 conmon[60316]: debug 2022-01-31T21:31:07.215+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:07.216382+0000) 2022-01-31T21:31:07.902 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:07 smithi171 conmon[51620]: debug 2022-01-31T21:31:07.824+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:07.825046+0000) 2022-01-31T21:31:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:07 smithi167 conmon[49112]: debug 2022-01-31T21:31:07.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:07.802950+0000) 2022-01-31T21:31:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:07 smithi171 conmon[41853]: debug 2022-01-31T21:31:07.946+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:07.947097+0000) 2022-01-31T21:31:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:07 smithi171 conmon[46715]: debug 2022-01-31T21:31:07.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:07.991438+0000) 2022-01-31T21:31:08.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:08 smithi167 conmon[54076]: debug 2022-01-31T21:31:08.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:08.215444+0000) 2022-01-31T21:31:08.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:08 smithi167 conmon[60316]: debug 2022-01-31T21:31:08.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:08.216557+0000) 2022-01-31T21:31:08.903 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:08 smithi171 conmon[35325]: debug 2022-01-31T21:31:08.692+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:31:08.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:08 smithi171 conmon[51620]: debug 2022-01-31T21:31:08.824+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:08.825246+0000) 2022-01-31T21:31:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:08 smithi167 conmon[49112]: debug 2022-01-31T21:31:08.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:08.803101+0000) 2022-01-31T21:31:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:08 smithi171 conmon[41853]: debug 2022-01-31T21:31:08.946+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:08.947288+0000) 2022-01-31T21:31:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:08 smithi171 conmon[46715]: debug 2022-01-31T21:31:08.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:08.991628+0000) 2022-01-31T21:31:09.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:09 smithi167 conmon[54076]: debug 2022-01-31T21:31:09.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:09.215573+0000) 2022-01-31T21:31:09.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:09 smithi167 conmon[60316]: debug 2022-01-31T21:31:09.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:09.216714+0000) 2022-01-31T21:31:09.903 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:09 smithi171 conmon[51620]: debug 2022-01-31T21:31:09.824+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:09.825414+0000) 2022-01-31T21:31:10.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:09 smithi167 conmon[49112]: debug 2022-01-31T21:31:09.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:09.803267+0000) 2022-01-31T21:31:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:09 smithi171 conmon[41853]: debug 2022-01-31T21:31:09.946+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:09.947463+0000) 2022-01-31T21:31:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:09 smithi171 conmon[46715]: debug 2022-01-31T21:31:09.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:09.991835+0000) 2022-01-31T21:31:10.671 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:10 smithi167 conmon[54076]: debug 2022-01-31T21:31:10.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.215787+0000) 2022-01-31T21:31:10.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:10 smithi167 conmon[60316]: debug 2022-01-31T21:31:10.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.216861+0000) 2022-01-31T21:31:10.903 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[35325]: debug 2022-01-31T21:31:10.753+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136171 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:10.904 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[41853]: debug 2022-01-31T21:31:10.739+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.740440+0000) 2022-01-31T21:31:10.904 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[46715]: debug 2022-01-31T21:31:10.738+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.739641+0000) 2022-01-31T21:31:10.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[51620]: debug 2022-01-31T21:31:10.738+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.739194+0000) 2022-01-31T21:31:10.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[51620]: debug 2022-01-31T21:31:10.824+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.825636+0000) 2022-01-31T21:31:11.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:10 smithi167 conmon[49112]: debug 2022-01-31T21:31:10.739+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.740057+0000) 2022-01-31T21:31:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:10 smithi167 conmon[49112]: debug 2022-01-31T21:31:10.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.803468+0000) 2022-01-31T21:31:11.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:10 smithi167 conmon[54076]: debug 2022-01-31T21:31:10.738+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.738963+0000) 2022-01-31T21:31:11.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:10 smithi167 conmon[60316]: debug 2022-01-31T21:31:10.739+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.739714+0000) 2022-01-31T21:31:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[46715]: debug 2022-01-31T21:31:10.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.992015+0000) 2022-01-31T21:31:11.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:10 smithi171 conmon[41853]: debug 2022-01-31T21:31:10.946+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:10.947579+0000) 2022-01-31T21:31:11.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:11 smithi167 conmon[54076]: debug 2022-01-31T21:31:11.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:11.215966+0000) 2022-01-31T21:31:11.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:11 smithi167 conmon[60316]: debug 2022-01-31T21:31:11.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:11.217046+0000) 2022-01-31T21:31:11.903 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:11 smithi171 conmon[51620]: debug 2022-01-31T21:31:11.825+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:11.825829+0000) 2022-01-31T21:31:12.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:11 smithi167 conmon[49112]: debug 2022-01-31T21:31:11.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:11.803630+0000) 2022-01-31T21:31:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:11 smithi171 conmon[46715]: debug 2022-01-31T21:31:11.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:11.992122+0000) 2022-01-31T21:31:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:11 smithi171 conmon[41853]: debug 2022-01-31T21:31:11.946+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:11.947737+0000) 2022-01-31T21:31:12.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:12 smithi167 conmon[54076]: debug 2022-01-31T21:31:12.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:12.216117+0000) 2022-01-31T21:31:12.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:12 smithi167 conmon[60316]: debug 2022-01-31T21:31:12.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:12.217197+0000) 2022-01-31T21:31:12.903 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:12 smithi171 conmon[51620]: debug 2022-01-31T21:31:12.825+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:12.826014+0000) 2022-01-31T21:31:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:12 smithi167 conmon[49112]: debug 2022-01-31T21:31:12.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:12.803777+0000) 2022-01-31T21:31:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:12 smithi171 conmon[41853]: debug 2022-01-31T21:31:12.947+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:12.947939+0000) 2022-01-31T21:31:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:12 smithi171 conmon[46715]: debug 2022-01-31T21:31:12.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:12.992306+0000) 2022-01-31T21:31:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:13 smithi167 conmon[54076]: debug 2022-01-31T21:31:13.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:13.216318+0000) 2022-01-31T21:31:13.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:13 smithi167 conmon[60316]: debug 2022-01-31T21:31:13.216+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:13.217387+0000) 2022-01-31T21:31:13.903 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:13 smithi171 conmon[51620]: debug 2022-01-31T21:31:13.825+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:13.826202+0000) 2022-01-31T21:31:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:13 smithi167 conmon[49112]: debug 2022-01-31T21:31:13.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:13.803917+0000) 2022-01-31T21:31:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:13 smithi171 conmon[46715]: debug 2022-01-31T21:31:13.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:13.992479+0000) 2022-01-31T21:31:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:13 smithi171 conmon[41853]: debug 2022-01-31T21:31:13.947+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:13.948151+0000) 2022-01-31T21:31:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:14 smithi167 conmon[54076]: debug 2022-01-31T21:31:14.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:14.216509+0000) 2022-01-31T21:31:14.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:14 smithi167 conmon[60316]: debug 2022-01-31T21:31:14.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:14.217550+0000) 2022-01-31T21:31:14.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:14 smithi171 conmon[51620]: debug 2022-01-31T21:31:14.825+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:14.826412+0000) 2022-01-31T21:31:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:14 smithi167 conmon[49112]: debug 2022-01-31T21:31:14.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:14.804065+0000) 2022-01-31T21:31:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:14 smithi171 conmon[41853]: debug 2022-01-31T21:31:14.947+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:14.948328+0000) 2022-01-31T21:31:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:14 smithi171 conmon[46715]: debug 2022-01-31T21:31:14.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:14.992680+0000) 2022-01-31T21:31:15.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:15 smithi167 conmon[54076]: debug 2022-01-31T21:31:15.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.216680+0000) 2022-01-31T21:31:15.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:15 smithi167 conmon[60316]: debug 2022-01-31T21:31:15.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.217677+0000) 2022-01-31T21:31:15.904 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[46715]: debug 2022-01-31T21:31:15.755+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.756308+0000) 2022-01-31T21:31:15.905 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[35325]: debug 2022-01-31T21:31:15.770+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136280 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:15.905 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[41853]: debug 2022-01-31T21:31:15.755+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.756138+0000) 2022-01-31T21:31:15.906 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[51620]: debug 2022-01-31T21:31:15.755+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.756447+0000) 2022-01-31T21:31:15.906 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[51620]: debug 2022-01-31T21:31:15.825+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.826602+0000) 2022-01-31T21:31:16.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:15 smithi167 conmon[54076]: debug 2022-01-31T21:31:15.755+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.755489+0000) 2022-01-31T21:31:16.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:15 smithi167 conmon[60316]: debug 2022-01-31T21:31:15.755+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.755697+0000) 2022-01-31T21:31:16.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:15 smithi167 conmon[49112]: debug 2022-01-31T21:31:15.756+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.756904+0000) 2022-01-31T21:31:16.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:15 smithi167 conmon[49112]: debug 2022-01-31T21:31:15.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.804201+0000) 2022-01-31T21:31:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[41853]: debug 2022-01-31T21:31:15.947+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.948513+0000) 2022-01-31T21:31:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:15 smithi171 conmon[46715]: debug 2022-01-31T21:31:15.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:15.992838+0000) 2022-01-31T21:31:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:16 smithi167 conmon[54076]: debug 2022-01-31T21:31:16.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:16.216778+0000) 2022-01-31T21:31:16.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:16 smithi167 conmon[60316]: debug 2022-01-31T21:31:16.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:16.217844+0000) 2022-01-31T21:31:16.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:16 smithi171 conmon[51620]: debug 2022-01-31T21:31:16.826+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:16.826778+0000) 2022-01-31T21:31:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:16 smithi167 conmon[49112]: debug 2022-01-31T21:31:16.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:16.804308+0000) 2022-01-31T21:31:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:16 smithi171 conmon[41853]: debug 2022-01-31T21:31:16.947+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:16.948672+0000) 2022-01-31T21:31:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:16 smithi171 conmon[46715]: debug 2022-01-31T21:31:16.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:16.992980+0000) 2022-01-31T21:31:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:17 smithi167 conmon[54076]: debug 2022-01-31T21:31:17.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:17.216938+0000) 2022-01-31T21:31:17.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:17 smithi167 conmon[60316]: debug 2022-01-31T21:31:17.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:17.217982+0000) 2022-01-31T21:31:17.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:17 smithi171 conmon[51620]: debug 2022-01-31T21:31:17.826+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:17.826987+0000) 2022-01-31T21:31:18.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:17 smithi167 conmon[49112]: debug 2022-01-31T21:31:17.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:17.804466+0000) 2022-01-31T21:31:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:17 smithi171 conmon[41853]: debug 2022-01-31T21:31:17.948+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:17.948859+0000) 2022-01-31T21:31:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:17 smithi171 conmon[46715]: debug 2022-01-31T21:31:17.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:17.993161+0000) 2022-01-31T21:31:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:18 smithi167 conmon[54076]: debug 2022-01-31T21:31:18.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:18.217126+0000) 2022-01-31T21:31:18.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:18 smithi167 conmon[60316]: debug 2022-01-31T21:31:18.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:18.218158+0000) 2022-01-31T21:31:18.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:18 smithi171 conmon[51620]: debug 2022-01-31T21:31:18.826+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:18.827193+0000) 2022-01-31T21:31:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:18 smithi167 conmon[49112]: debug 2022-01-31T21:31:18.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:18.804620+0000) 2022-01-31T21:31:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:18 smithi171 conmon[41853]: debug 2022-01-31T21:31:18.948+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:18.949070+0000) 2022-01-31T21:31:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:18 smithi171 conmon[46715]: debug 2022-01-31T21:31:18.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:18.993343+0000) 2022-01-31T21:31:19.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:19 smithi167 conmon[54076]: debug 2022-01-31T21:31:19.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:19.217289+0000) 2022-01-31T21:31:19.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:19 smithi167 conmon[60316]: debug 2022-01-31T21:31:19.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:19.218283+0000) 2022-01-31T21:31:19.904 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:19 smithi171 conmon[51620]: debug 2022-01-31T21:31:19.826+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:19.827404+0000) 2022-01-31T21:31:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:19 smithi167 conmon[49112]: debug 2022-01-31T21:31:19.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:19.804794+0000) 2022-01-31T21:31:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:19 smithi171 conmon[46715]: debug 2022-01-31T21:31:19.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:19.993530+0000) 2022-01-31T21:31:20.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:19 smithi171 conmon[41853]: debug 2022-01-31T21:31:19.948+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:19.949307+0000) 2022-01-31T21:31:20.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:20 smithi167 conmon[54076]: debug 2022-01-31T21:31:20.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.217491+0000) 2022-01-31T21:31:20.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:20 smithi167 conmon[60316]: debug 2022-01-31T21:31:20.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.218436+0000) 2022-01-31T21:31:20.905 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[46715]: debug 2022-01-31T21:31:20.772+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.773411+0000) 2022-01-31T21:31:20.905 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[35325]: debug 2022-01-31T21:31:20.787+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136392 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:20.906 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[41853]: debug 2022-01-31T21:31:20.772+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.773563+0000) 2022-01-31T21:31:20.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[51620]: debug 2022-01-31T21:31:20.773+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.774086+0000) 2022-01-31T21:31:20.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[51620]: debug 2022-01-31T21:31:20.826+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.827576+0000) 2022-01-31T21:31:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:20 smithi167 conmon[49112]: debug 2022-01-31T21:31:20.773+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.773676+0000) 2022-01-31T21:31:21.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:20 smithi167 conmon[49112]: debug 2022-01-31T21:31:20.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.804981+0000) 2022-01-31T21:31:21.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:20 smithi167 conmon[54076]: debug 2022-01-31T21:31:20.771+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.772281+0000) 2022-01-31T21:31:21.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:20 smithi167 conmon[60316]: debug 2022-01-31T21:31:20.772+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.772984+0000) 2022-01-31T21:31:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[41853]: debug 2022-01-31T21:31:20.948+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.949456+0000) 2022-01-31T21:31:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:20 smithi171 conmon[46715]: debug 2022-01-31T21:31:20.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:20.993706+0000) 2022-01-31T21:31:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:21 smithi167 conmon[54076]: debug 2022-01-31T21:31:21.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:21.217661+0000) 2022-01-31T21:31:21.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:21 smithi167 conmon[60316]: debug 2022-01-31T21:31:21.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:21.218575+0000) 2022-01-31T21:31:21.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:21 smithi171 conmon[51620]: debug 2022-01-31T21:31:21.827+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:21.827721+0000) 2022-01-31T21:31:22.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:21 smithi167 conmon[49112]: debug 2022-01-31T21:31:21.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:21.805125+0000) 2022-01-31T21:31:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:21 smithi171 conmon[41853]: debug 2022-01-31T21:31:21.948+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:21.949565+0000) 2022-01-31T21:31:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:21 smithi171 conmon[46715]: debug 2022-01-31T21:31:21.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:21.993854+0000) 2022-01-31T21:31:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:22 smithi167 conmon[54076]: debug 2022-01-31T21:31:22.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:22.217817+0000) 2022-01-31T21:31:22.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:22 smithi167 conmon[60316]: debug 2022-01-31T21:31:22.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:22.218704+0000) 2022-01-31T21:31:22.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:22 smithi171 conmon[51620]: debug 2022-01-31T21:31:22.827+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:22.827876+0000) 2022-01-31T21:31:23.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:22 smithi167 conmon[49112]: debug 2022-01-31T21:31:22.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:22.805284+0000) 2022-01-31T21:31:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:22 smithi171 conmon[41853]: debug 2022-01-31T21:31:22.949+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:22.949768+0000) 2022-01-31T21:31:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:22 smithi171 conmon[46715]: debug 2022-01-31T21:31:22.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:22.994021+0000) 2022-01-31T21:31:23.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:23 smithi167 conmon[54076]: debug 2022-01-31T21:31:23.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:23.217999+0000) 2022-01-31T21:31:23.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:23 smithi167 conmon[60316]: debug 2022-01-31T21:31:23.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:23.218860+0000) 2022-01-31T21:31:23.905 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:23 smithi171 conmon[35325]: debug 2022-01-31T21:31:23.693+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:31:23.906 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:23 smithi171 conmon[51620]: debug 2022-01-31T21:31:23.827+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:23.828036+0000) 2022-01-31T21:31:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:23 smithi167 conmon[49112]: debug 2022-01-31T21:31:23.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:23.805472+0000) 2022-01-31T21:31:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:23 smithi171 conmon[41853]: debug 2022-01-31T21:31:23.949+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:23.949953+0000) 2022-01-31T21:31:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:23 smithi171 conmon[46715]: debug 2022-01-31T21:31:23.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:23.994160+0000) 2022-01-31T21:31:24.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:24 smithi167 conmon[54076]: debug 2022-01-31T21:31:24.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:24.218180+0000) 2022-01-31T21:31:24.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:24 smithi167 conmon[60316]: debug 2022-01-31T21:31:24.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:24.219020+0000) 2022-01-31T21:31:24.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:24 smithi171 conmon[51620]: debug 2022-01-31T21:31:24.827+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:24.828181+0000) 2022-01-31T21:31:25.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:24 smithi167 conmon[49112]: debug 2022-01-31T21:31:24.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:24.805653+0000) 2022-01-31T21:31:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:24 smithi171 conmon[41853]: debug 2022-01-31T21:31:24.949+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:24.950160+0000) 2022-01-31T21:31:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:24 smithi171 conmon[46715]: debug 2022-01-31T21:31:24.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:24.994332+0000) 2022-01-31T21:31:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:25 smithi167 conmon[54076]: debug 2022-01-31T21:31:25.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.218307+0000) 2022-01-31T21:31:25.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:25 smithi167 conmon[60316]: debug 2022-01-31T21:31:25.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.219197+0000) 2022-01-31T21:31:25.905 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[35325]: debug 2022-01-31T21:31:25.804+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136512 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:25.906 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[41853]: debug 2022-01-31T21:31:25.789+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.790490+0000) 2022-01-31T21:31:25.907 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[46715]: debug 2022-01-31T21:31:25.790+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.791637+0000) 2022-01-31T21:31:25.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[51620]: debug 2022-01-31T21:31:25.789+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.790728+0000) 2022-01-31T21:31:25.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[51620]: debug 2022-01-31T21:31:25.827+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.828408+0000) 2022-01-31T21:31:26.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:25 smithi167 conmon[54076]: debug 2022-01-31T21:31:25.789+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.789619+0000) 2022-01-31T21:31:26.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:25 smithi167 conmon[60316]: debug 2022-01-31T21:31:25.790+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.790863+0000) 2022-01-31T21:31:26.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:25 smithi167 conmon[49112]: debug 2022-01-31T21:31:25.791+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.791521+0000) 2022-01-31T21:31:26.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:25 smithi167 conmon[49112]: debug 2022-01-31T21:31:25.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.805794+0000) 2022-01-31T21:31:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[41853]: debug 2022-01-31T21:31:25.949+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.950343+0000) 2022-01-31T21:31:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:25 smithi171 conmon[46715]: debug 2022-01-31T21:31:25.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:25.994491+0000) 2022-01-31T21:31:26.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:26 smithi167 conmon[54076]: debug 2022-01-31T21:31:26.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:26.218510+0000) 2022-01-31T21:31:26.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:26 smithi167 conmon[60316]: debug 2022-01-31T21:31:26.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:26.219297+0000) 2022-01-31T21:31:26.905 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:26 smithi171 conmon[51620]: debug 2022-01-31T21:31:26.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:26.828574+0000) 2022-01-31T21:31:27.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:26 smithi167 conmon[49112]: debug 2022-01-31T21:31:26.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:26.805912+0000) 2022-01-31T21:31:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:26 smithi171 conmon[46715]: debug 2022-01-31T21:31:26.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:26.994628+0000) 2022-01-31T21:31:27.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:26 smithi171 conmon[41853]: debug 2022-01-31T21:31:26.950+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:26.950515+0000) 2022-01-31T21:31:27.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:27 smithi167 conmon[54076]: debug 2022-01-31T21:31:27.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:27.218664+0000) 2022-01-31T21:31:27.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:27 smithi167 conmon[60316]: debug 2022-01-31T21:31:27.217+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:27.219398+0000) 2022-01-31T21:31:27.906 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:27 smithi171 conmon[51620]: debug 2022-01-31T21:31:27.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:27.828780+0000) 2022-01-31T21:31:28.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:27 smithi167 conmon[49112]: debug 2022-01-31T21:31:27.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:27.806083+0000) 2022-01-31T21:31:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:27 smithi171 conmon[46715]: debug 2022-01-31T21:31:27.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:27.994791+0000) 2022-01-31T21:31:28.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:27 smithi171 conmon[41853]: debug 2022-01-31T21:31:27.950+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:27.950748+0000) 2022-01-31T21:31:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:28 smithi167 conmon[54076]: debug 2022-01-31T21:31:28.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:28.218839+0000) 2022-01-31T21:31:28.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:28 smithi167 conmon[60316]: debug 2022-01-31T21:31:28.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:28.219527+0000) 2022-01-31T21:31:28.906 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:28 smithi171 conmon[51620]: debug 2022-01-31T21:31:28.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:28.828958+0000) 2022-01-31T21:31:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:28 smithi167 conmon[49112]: debug 2022-01-31T21:31:28.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:28.806253+0000) 2022-01-31T21:31:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:28 smithi171 conmon[46715]: debug 2022-01-31T21:31:28.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:28.994935+0000) 2022-01-31T21:31:29.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:28 smithi171 conmon[41853]: debug 2022-01-31T21:31:28.950+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:28.950941+0000) 2022-01-31T21:31:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:29 smithi167 conmon[54076]: debug 2022-01-31T21:31:29.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:29.219004+0000) 2022-01-31T21:31:29.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:29 smithi167 conmon[60316]: debug 2022-01-31T21:31:29.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:29.219668+0000) 2022-01-31T21:31:29.906 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:29 smithi171 conmon[51620]: debug 2022-01-31T21:31:29.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:29.829098+0000) 2022-01-31T21:31:30.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:29 smithi167 conmon[49112]: debug 2022-01-31T21:31:29.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:29.806407+0000) 2022-01-31T21:31:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:29 smithi171 conmon[46715]: debug 2022-01-31T21:31:29.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:29.995107+0000) 2022-01-31T21:31:30.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:29 smithi171 conmon[41853]: debug 2022-01-31T21:31:29.950+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:29.951143+0000) 2022-01-31T21:31:30.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:30 smithi167 conmon[54076]: debug 2022-01-31T21:31:30.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.219163+0000) 2022-01-31T21:31:30.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:30 smithi167 conmon[60316]: debug 2022-01-31T21:31:30.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.219823+0000) 2022-01-31T21:31:30.906 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[35325]: debug 2022-01-31T21:31:30.821+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136624 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:30.907 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[41853]: debug 2022-01-31T21:31:30.806+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.807443+0000) 2022-01-31T21:31:30.908 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[46715]: debug 2022-01-31T21:31:30.808+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.809104+0000) 2022-01-31T21:31:30.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[51620]: debug 2022-01-31T21:31:30.806+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.807230+0000) 2022-01-31T21:31:30.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[51620]: debug 2022-01-31T21:31:30.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.829253+0000) 2022-01-31T21:31:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:30 smithi167 conmon[49112]: debug 2022-01-31T21:31:30.807+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.808770+0000) 2022-01-31T21:31:31.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:30 smithi167 conmon[54076]: debug 2022-01-31T21:31:30.805+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.806722+0000) 2022-01-31T21:31:31.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:30 smithi167 conmon[60316]: debug 2022-01-31T21:31:30.806+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.808165+0000) 2022-01-31T21:31:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[46715]: debug 2022-01-31T21:31:30.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.995278+0000) 2022-01-31T21:31:31.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:30 smithi171 conmon[41853]: debug 2022-01-31T21:31:30.950+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:30.951332+0000) 2022-01-31T21:31:31.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:31 smithi167 conmon[54076]: debug 2022-01-31T21:31:31.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:31.219354+0000) 2022-01-31T21:31:31.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:31 smithi167 conmon[60316]: debug 2022-01-31T21:31:31.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:31.219956+0000) 2022-01-31T21:31:32.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:31 smithi171 conmon[46715]: debug 2022-01-31T21:31:31.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:31.995428+0000) 2022-01-31T21:31:32.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:31 smithi171 conmon[51620]: debug 2022-01-31T21:31:31.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:31.829390+0000) 2022-01-31T21:31:32.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:31 smithi171 conmon[41853]: debug 2022-01-31T21:31:31.951+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:31.951503+0000) 2022-01-31T21:31:32.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:32 smithi167 conmon[54076]: debug 2022-01-31T21:31:32.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:32.219514+0000) 2022-01-31T21:31:32.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:32 smithi167 conmon[60316]: debug 2022-01-31T21:31:32.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:32.220042+0000) 2022-01-31T21:31:32.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:32 smithi171 conmon[51620]: debug 2022-01-31T21:31:32.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:32.829579+0000) 2022-01-31T21:31:33.311 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:32 smithi171 conmon[41853]: debug 2022-01-31T21:31:32.950+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:32.951671+0000) 2022-01-31T21:31:33.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:32 smithi171 conmon[46715]: debug 2022-01-31T21:31:32.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:32.995625+0000) 2022-01-31T21:31:33.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:33 smithi167 conmon[54076]: debug 2022-01-31T21:31:33.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:33.219686+0000) 2022-01-31T21:31:33.531 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:33 smithi167 conmon[60316]: debug 2022-01-31T21:31:33.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:33.220152+0000) 2022-01-31T21:31:33.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:33 smithi171 conmon[51620]: debug 2022-01-31T21:31:33.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:33.829768+0000) 2022-01-31T21:31:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:33 smithi171 conmon[41853]: debug 2022-01-31T21:31:33.951+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:33.951881+0000) 2022-01-31T21:31:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:33 smithi171 conmon[46715]: debug 2022-01-31T21:31:33.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:33.995776+0000) 2022-01-31T21:31:34.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:34 smithi167 conmon[54076]: debug 2022-01-31T21:31:34.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:34.219847+0000) 2022-01-31T21:31:34.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:34 smithi167 conmon[60316]: debug 2022-01-31T21:31:34.218+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:34.220289+0000) 2022-01-31T21:31:34.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:34 smithi171 conmon[51620]: debug 2022-01-31T21:31:34.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:34.829915+0000) 2022-01-31T21:31:35.295 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:34 smithi171 conmon[41853]: debug 2022-01-31T21:31:34.951+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:34.952086+0000) 2022-01-31T21:31:35.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:34 smithi171 conmon[46715]: debug 2022-01-31T21:31:34.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:34.995905+0000) 2022-01-31T21:31:35.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:35 smithi167 conmon[54076]: debug 2022-01-31T21:31:35.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.220054+0000) 2022-01-31T21:31:35.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:35 smithi167 conmon[60316]: debug 2022-01-31T21:31:35.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.220396+0000) 2022-01-31T21:31:35.907 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[35325]: debug 2022-01-31T21:31:35.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136733 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:35.908 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[41853]: debug 2022-01-31T21:31:35.824+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.824999+0000) 2022-01-31T21:31:35.908 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[46715]: debug 2022-01-31T21:31:35.825+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.826017+0000) 2022-01-31T21:31:35.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[51620]: debug 2022-01-31T21:31:35.825+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.826088+0000) 2022-01-31T21:31:35.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[51620]: debug 2022-01-31T21:31:35.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.830048+0000) 2022-01-31T21:31:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:35 smithi167 conmon[49112]: debug 2022-01-31T21:31:35.824+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.826276+0000) 2022-01-31T21:31:36.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:35 smithi167 conmon[54076]: debug 2022-01-31T21:31:35.822+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.823954+0000) 2022-01-31T21:31:36.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:35 smithi167 conmon[60316]: debug 2022-01-31T21:31:35.824+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.825688+0000) 2022-01-31T21:31:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[41853]: debug 2022-01-31T21:31:35.952+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.952259+0000) 2022-01-31T21:31:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:35 smithi171 conmon[46715]: debug 2022-01-31T21:31:35.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:35.996057+0000) 2022-01-31T21:31:36.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:36 smithi167 conmon[54076]: debug 2022-01-31T21:31:36.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:36.220245+0000) 2022-01-31T21:31:36.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:36 smithi167 conmon[60316]: debug 2022-01-31T21:31:36.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:36.220496+0000) 2022-01-31T21:31:36.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:36 smithi171 conmon[51620]: debug 2022-01-31T21:31:36.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:36.830197+0000) 2022-01-31T21:31:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:36 smithi171 conmon[41853]: debug 2022-01-31T21:31:36.951+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:36.952384+0000) 2022-01-31T21:31:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:36 smithi171 conmon[46715]: debug 2022-01-31T21:31:36.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:36.996193+0000) 2022-01-31T21:31:37.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:37 smithi167 conmon[54076]: debug 2022-01-31T21:31:37.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:37.220383+0000) 2022-01-31T21:31:37.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:37 smithi167 conmon[60316]: debug 2022-01-31T21:31:37.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:37.220607+0000) 2022-01-31T21:31:37.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:37 smithi171 conmon[51620]: debug 2022-01-31T21:31:37.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:37.830423+0000) 2022-01-31T21:31:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:37 smithi167 conmon[49112]: debug 2022-01-31T21:31:37.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:37.556796+0000) 2022-01-31T21:31:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:37 smithi171 conmon[41853]: debug 2022-01-31T21:31:37.951+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:37.952567+0000) 2022-01-31T21:31:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:37 smithi171 conmon[46715]: debug 2022-01-31T21:31:37.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:37.996371+0000) 2022-01-31T21:31:38.538 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:38 smithi167 conmon[60316]: debug 2022-01-31T21:31:38.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:38.220704+0000) 2022-01-31T21:31:38.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:38 smithi167 conmon[54076]: debug 2022-01-31T21:31:38.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:38.220547+0000) 2022-01-31T21:31:38.908 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:38 smithi171 conmon[35325]: debug 2022-01-31T21:31:38.694+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:31:38.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:38 smithi171 conmon[51620]: debug 2022-01-31T21:31:38.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:38.830662+0000) 2022-01-31T21:31:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:38 smithi167 conmon[49112]: debug 2022-01-31T21:31:38.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:38.556947+0000) 2022-01-31T21:31:39.326 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:38 smithi171 conmon[41853]: debug 2022-01-31T21:31:38.952+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:38.952783+0000) 2022-01-31T21:31:39.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:38 smithi171 conmon[46715]: debug 2022-01-31T21:31:38.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:38.996561+0000) 2022-01-31T21:31:39.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:39 smithi167 conmon[54076]: debug 2022-01-31T21:31:39.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:39.220697+0000) 2022-01-31T21:31:39.539 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:39 smithi167 conmon[60316]: debug 2022-01-31T21:31:39.219+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:39.220864+0000) 2022-01-31T21:31:39.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:39 smithi171 conmon[51620]: debug 2022-01-31T21:31:39.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:39.830845+0000) 2022-01-31T21:31:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:39 smithi167 conmon[49112]: debug 2022-01-31T21:31:39.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:39.557106+0000) 2022-01-31T21:31:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:39 smithi171 conmon[46715]: debug 2022-01-31T21:31:39.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:39.996720+0000) 2022-01-31T21:31:40.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:39 smithi171 conmon[41853]: debug 2022-01-31T21:31:39.952+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:39.952989+0000) 2022-01-31T21:31:40.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:40 smithi167 conmon[54076]: debug 2022-01-31T21:31:40.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.220863+0000) 2022-01-31T21:31:40.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:40 smithi167 conmon[60316]: debug 2022-01-31T21:31:40.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.221010+0000) 2022-01-31T21:31:40.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:40 smithi167 conmon[49112]: debug 2022-01-31T21:31:40.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.557270+0000) 2022-01-31T21:31:40.908 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[35325]: debug 2022-01-31T21:31:40.856+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136845 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:40.909 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[41853]: debug 2022-01-31T21:31:40.841+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.842257+0000) 2022-01-31T21:31:40.909 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[46715]: debug 2022-01-31T21:31:40.841+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.842422+0000) 2022-01-31T21:31:40.910 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[51620]: debug 2022-01-31T21:31:40.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.831005+0000) 2022-01-31T21:31:40.910 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[51620]: debug 2022-01-31T21:31:40.841+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.842581+0000) 2022-01-31T21:31:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:40 smithi167 conmon[49112]: debug 2022-01-31T21:31:40.841+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.842590+0000) 2022-01-31T21:31:41.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:40 smithi167 conmon[54076]: debug 2022-01-31T21:31:40.839+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.840896+0000) 2022-01-31T21:31:41.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:40 smithi167 conmon[60316]: debug 2022-01-31T21:31:40.840+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.842108+0000) 2022-01-31T21:31:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[41853]: debug 2022-01-31T21:31:40.952+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.953145+0000) 2022-01-31T21:31:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:40 smithi171 conmon[46715]: debug 2022-01-31T21:31:40.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:40.996858+0000) 2022-01-31T21:31:41.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:41 smithi167 conmon[54076]: debug 2022-01-31T21:31:41.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:41.220996+0000) 2022-01-31T21:31:41.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:41 smithi167 conmon[60316]: debug 2022-01-31T21:31:41.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:41.221121+0000) 2022-01-31T21:31:41.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:41 smithi171 conmon[51620]: debug 2022-01-31T21:31:41.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:41.831172+0000) 2022-01-31T21:31:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:41 smithi167 conmon[49112]: debug 2022-01-31T21:31:41.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:41.557455+0000) 2022-01-31T21:31:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:41 smithi171 conmon[41853]: debug 2022-01-31T21:31:41.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:41.953321+0000) 2022-01-31T21:31:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:41 smithi171 conmon[46715]: debug 2022-01-31T21:31:41.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:41.997016+0000) 2022-01-31T21:31:42.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:42 smithi167 conmon[54076]: debug 2022-01-31T21:31:42.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:42.221136+0000) 2022-01-31T21:31:42.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:42 smithi167 conmon[60316]: debug 2022-01-31T21:31:42.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:42.221243+0000) 2022-01-31T21:31:42.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:42 smithi171 conmon[51620]: debug 2022-01-31T21:31:42.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:42.831343+0000) 2022-01-31T21:31:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:42 smithi167 conmon[49112]: debug 2022-01-31T21:31:42.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:42.557603+0000) 2022-01-31T21:31:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:42 smithi171 conmon[41853]: debug 2022-01-31T21:31:42.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:42.953570+0000) 2022-01-31T21:31:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:42 smithi171 conmon[46715]: debug 2022-01-31T21:31:42.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:42.997226+0000) 2022-01-31T21:31:43.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:43 smithi167 conmon[54076]: debug 2022-01-31T21:31:43.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:43.221298+0000) 2022-01-31T21:31:43.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:43 smithi167 conmon[60316]: debug 2022-01-31T21:31:43.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:43.221390+0000) 2022-01-31T21:31:43.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:43 smithi171 conmon[51620]: debug 2022-01-31T21:31:43.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:43.831533+0000) 2022-01-31T21:31:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:43 smithi167 conmon[49112]: debug 2022-01-31T21:31:43.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:43.557785+0000) 2022-01-31T21:31:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:43 smithi171 conmon[41853]: debug 2022-01-31T21:31:43.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:43.953792+0000) 2022-01-31T21:31:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:43 smithi171 conmon[46715]: debug 2022-01-31T21:31:43.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:43.997399+0000) 2022-01-31T21:31:44.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:44 smithi167 conmon[54076]: debug 2022-01-31T21:31:44.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:44.221525+0000) 2022-01-31T21:31:44.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:44 smithi167 conmon[60316]: debug 2022-01-31T21:31:44.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:44.221542+0000) 2022-01-31T21:31:44.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:44 smithi171 conmon[51620]: debug 2022-01-31T21:31:44.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:44.831738+0000) 2022-01-31T21:31:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:44 smithi167 conmon[49112]: debug 2022-01-31T21:31:44.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:44.557992+0000) 2022-01-31T21:31:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:44 smithi171 conmon[41853]: debug 2022-01-31T21:31:44.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:44.953995+0000) 2022-01-31T21:31:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:44 smithi171 conmon[46715]: debug 2022-01-31T21:31:44.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:44.997588+0000) 2022-01-31T21:31:45.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:45 smithi167 conmon[60316]: debug 2022-01-31T21:31:45.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.221703+0000) 2022-01-31T21:31:45.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:45 smithi167 conmon[54076]: debug 2022-01-31T21:31:45.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.221683+0000) 2022-01-31T21:31:45.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:45 smithi167 conmon[49112]: debug 2022-01-31T21:31:45.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.558239+0000) 2022-01-31T21:31:45.909 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[35325]: debug 2022-01-31T21:31:45.873+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 136954 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:45.910 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[41853]: debug 2022-01-31T21:31:45.858+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.859330+0000) 2022-01-31T21:31:45.910 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[46715]: debug 2022-01-31T21:31:45.858+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.858918+0000) 2022-01-31T21:31:45.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[51620]: debug 2022-01-31T21:31:45.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.831957+0000) 2022-01-31T21:31:45.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[51620]: debug 2022-01-31T21:31:45.858+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.859682+0000) 2022-01-31T21:31:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:45 smithi167 conmon[49112]: debug 2022-01-31T21:31:45.858+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.860375+0000) 2022-01-31T21:31:46.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:45 smithi167 conmon[54076]: debug 2022-01-31T21:31:45.858+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.859933+0000) 2022-01-31T21:31:46.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:45 smithi167 conmon[60316]: debug 2022-01-31T21:31:45.857+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.859470+0000) 2022-01-31T21:31:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[46715]: debug 2022-01-31T21:31:45.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.997741+0000) 2022-01-31T21:31:46.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:45 smithi171 conmon[41853]: debug 2022-01-31T21:31:45.954+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:45.954175+0000) 2022-01-31T21:31:46.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:46 smithi167 conmon[54076]: debug 2022-01-31T21:31:46.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:46.221871+0000) 2022-01-31T21:31:46.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:46 smithi167 conmon[60316]: debug 2022-01-31T21:31:46.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:46.221887+0000) 2022-01-31T21:31:46.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:46 smithi171 conmon[51620]: debug 2022-01-31T21:31:46.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:46.832167+0000) 2022-01-31T21:31:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:46 smithi167 conmon[49112]: debug 2022-01-31T21:31:46.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:46.558391+0000) 2022-01-31T21:31:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:46 smithi171 conmon[41853]: debug 2022-01-31T21:31:46.954+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:46.954334+0000) 2022-01-31T21:31:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:46 smithi171 conmon[46715]: debug 2022-01-31T21:31:46.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:46.997907+0000) 2022-01-31T21:31:47.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:47 smithi167 conmon[54076]: debug 2022-01-31T21:31:47.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:47.222045+0000) 2022-01-31T21:31:47.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:47 smithi167 conmon[60316]: debug 2022-01-31T21:31:47.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:47.221996+0000) 2022-01-31T21:31:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:47 smithi167 conmon[49112]: debug 2022-01-31T21:31:47.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:47.558583+0000) 2022-01-31T21:31:48.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:47 smithi171 conmon[41853]: debug 2022-01-31T21:31:47.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:47.954491+0000) 2022-01-31T21:31:48.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:47 smithi171 conmon[46715]: debug 2022-01-31T21:31:47.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:47.998090+0000) 2022-01-31T21:31:48.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:47 smithi171 conmon[51620]: debug 2022-01-31T21:31:47.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:47.832334+0000) 2022-01-31T21:31:48.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:48 smithi167 conmon[54076]: debug 2022-01-31T21:31:48.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:48.222249+0000) 2022-01-31T21:31:48.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:48 smithi167 conmon[60316]: debug 2022-01-31T21:31:48.220+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:48.222153+0000) 2022-01-31T21:31:48.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:48 smithi171 conmon[51620]: debug 2022-01-31T21:31:48.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:48.832526+0000) 2022-01-31T21:31:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:48 smithi167 conmon[49112]: debug 2022-01-31T21:31:48.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:48.558817+0000) 2022-01-31T21:31:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:48 smithi171 conmon[46715]: debug 2022-01-31T21:31:48.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:48.998286+0000) 2022-01-31T21:31:49.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:48 smithi171 conmon[41853]: debug 2022-01-31T21:31:48.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:48.954652+0000) 2022-01-31T21:31:49.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:49 smithi167 conmon[54076]: debug 2022-01-31T21:31:49.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:49.222405+0000) 2022-01-31T21:31:49.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:49 smithi167 conmon[60316]: debug 2022-01-31T21:31:49.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:49.222331+0000) 2022-01-31T21:31:49.910 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:49 smithi171 conmon[51620]: debug 2022-01-31T21:31:49.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:49.832712+0000) 2022-01-31T21:31:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:49 smithi167 conmon[49112]: debug 2022-01-31T21:31:49.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:49.559048+0000) 2022-01-31T21:31:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:49 smithi171 conmon[46715]: debug 2022-01-31T21:31:49.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:49.998409+0000) 2022-01-31T21:31:50.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:49 smithi171 conmon[41853]: debug 2022-01-31T21:31:49.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:49.954804+0000) 2022-01-31T21:31:50.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:50 smithi167 conmon[54076]: debug 2022-01-31T21:31:50.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.222629+0000) 2022-01-31T21:31:50.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:50 smithi167 conmon[60316]: debug 2022-01-31T21:31:50.221+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.222561+0000) 2022-01-31T21:31:50.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:50 smithi167 conmon[49112]: debug 2022-01-31T21:31:50.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.559260+0000) 2022-01-31T21:31:50.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[35325]: debug 2022-01-31T21:31:50.889+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137067 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:50.911 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[41853]: debug 2022-01-31T21:31:50.874+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.876277+0000) 2022-01-31T21:31:50.911 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[46715]: debug 2022-01-31T21:31:50.874+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.876597+0000) 2022-01-31T21:31:50.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[51620]: debug 2022-01-31T21:31:50.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.832887+0000) 2022-01-31T21:31:50.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[51620]: debug 2022-01-31T21:31:50.876+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.877819+0000) 2022-01-31T21:31:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:50 smithi167 conmon[49112]: debug 2022-01-31T21:31:50.875+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.876726+0000) 2022-01-31T21:31:51.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:50 smithi167 conmon[54076]: debug 2022-01-31T21:31:50.876+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.877637+0000) 2022-01-31T21:31:51.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:50 smithi167 conmon[60316]: debug 2022-01-31T21:31:50.874+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.876278+0000) 2022-01-31T21:31:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[41853]: debug 2022-01-31T21:31:50.954+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.954992+0000) 2022-01-31T21:31:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:50 smithi171 conmon[46715]: debug 2022-01-31T21:31:50.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:50.998543+0000) 2022-01-31T21:31:51.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:51 smithi167 conmon[54076]: debug 2022-01-31T21:31:51.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:51.222826+0000) 2022-01-31T21:31:51.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:51 smithi167 conmon[60316]: debug 2022-01-31T21:31:51.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:51.222773+0000) 2022-01-31T21:31:51.910 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:51 smithi171 conmon[51620]: debug 2022-01-31T21:31:51.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:51.833034+0000) 2022-01-31T21:31:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:51 smithi167 conmon[49112]: debug 2022-01-31T21:31:51.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:51.559487+0000) 2022-01-31T21:31:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:51 smithi171 conmon[46715]: debug 2022-01-31T21:31:51.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:51.998689+0000) 2022-01-31T21:31:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:51 smithi171 conmon[41853]: debug 2022-01-31T21:31:51.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:51.955167+0000) 2022-01-31T21:31:52.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:52 smithi167 conmon[54076]: debug 2022-01-31T21:31:52.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:52.222977+0000) 2022-01-31T21:31:52.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:52 smithi167 conmon[60316]: debug 2022-01-31T21:31:52.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:52.222933+0000) 2022-01-31T21:31:52.910 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:52 smithi171 conmon[51620]: debug 2022-01-31T21:31:52.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:52.833182+0000) 2022-01-31T21:31:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:52 smithi167 conmon[49112]: debug 2022-01-31T21:31:52.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:52.559674+0000) 2022-01-31T21:31:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:52 smithi171 conmon[41853]: debug 2022-01-31T21:31:52.953+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:52.955383+0000) 2022-01-31T21:31:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:52 smithi171 conmon[46715]: debug 2022-01-31T21:31:52.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:52.998909+0000) 2022-01-31T21:31:53.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:53 smithi167 conmon[54076]: debug 2022-01-31T21:31:53.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:53.223141+0000) 2022-01-31T21:31:53.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:53 smithi167 conmon[60316]: debug 2022-01-31T21:31:53.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:53.223140+0000) 2022-01-31T21:31:53.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:53 smithi171 conmon[35325]: debug 2022-01-31T21:31:53.694+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:31:53.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:53 smithi171 conmon[51620]: debug 2022-01-31T21:31:53.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:53.833337+0000) 2022-01-31T21:31:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:53 smithi167 conmon[49112]: debug 2022-01-31T21:31:53.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:53.559832+0000) 2022-01-31T21:31:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:53 smithi171 conmon[46715]: debug 2022-01-31T21:31:53.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:53.999101+0000) 2022-01-31T21:31:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:53 smithi171 conmon[41853]: debug 2022-01-31T21:31:53.954+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:53.955564+0000) 2022-01-31T21:31:54.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:54 smithi167 conmon[54076]: debug 2022-01-31T21:31:54.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:54.223347+0000) 2022-01-31T21:31:54.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:54 smithi167 conmon[60316]: debug 2022-01-31T21:31:54.222+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:54.223328+0000) 2022-01-31T21:31:54.910 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:54 smithi171 conmon[51620]: debug 2022-01-31T21:31:54.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:54.833548+0000) 2022-01-31T21:31:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:54 smithi167 conmon[49112]: debug 2022-01-31T21:31:54.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:54.560041+0000) 2022-01-31T21:31:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:54 smithi171 conmon[46715]: debug 2022-01-31T21:31:54.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:54.999314+0000) 2022-01-31T21:31:55.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:54 smithi171 conmon[41853]: debug 2022-01-31T21:31:54.954+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:54.955684+0000) 2022-01-31T21:31:55.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:55 smithi167 conmon[54076]: debug 2022-01-31T21:31:55.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.223572+0000) 2022-01-31T21:31:55.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:55 smithi167 conmon[60316]: debug 2022-01-31T21:31:55.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.223563+0000) 2022-01-31T21:31:55.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:55 smithi167 conmon[49112]: debug 2022-01-31T21:31:55.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.560268+0000) 2022-01-31T21:31:55.906 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[41853]: debug 2022-01-31T21:31:55.891+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.892955+0000) 2022-01-31T21:31:55.907 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[46715]: debug 2022-01-31T21:31:55.892+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.893943+0000) 2022-01-31T21:31:55.907 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[51620]: debug 2022-01-31T21:31:55.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.833781+0000) 2022-01-31T21:31:55.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[51620]: debug 2022-01-31T21:31:55.891+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.893341+0000) 2022-01-31T21:31:56.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:55 smithi167 conmon[60316]: debug 2022-01-31T21:31:55.892+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.892968+0000) 2022-01-31T21:31:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:55 smithi167 conmon[49112]: debug 2022-01-31T21:31:55.893+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.894065+0000) 2022-01-31T21:31:56.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:55 smithi167 conmon[54076]: debug 2022-01-31T21:31:55.893+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.893815+0000) 2022-01-31T21:31:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[41853]: debug 2022-01-31T21:31:55.954+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.955846+0000) 2022-01-31T21:31:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[46715]: debug 2022-01-31T21:31:55.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:55.999497+0000) 2022-01-31T21:31:56.356 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:31:55 smithi171 conmon[35325]: debug 2022-01-31T21:31:55.906+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137168 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:31:56.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:56 smithi167 conmon[54076]: debug 2022-01-31T21:31:56.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:56.223738+0000) 2022-01-31T21:31:56.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:56 smithi167 conmon[60316]: debug 2022-01-31T21:31:56.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:56.223758+0000) 2022-01-31T21:31:56.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:56 smithi171 conmon[51620]: debug 2022-01-31T21:31:56.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:56.833979+0000) 2022-01-31T21:31:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:56 smithi167 conmon[49112]: debug 2022-01-31T21:31:56.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:56.560519+0000) 2022-01-31T21:31:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:56 smithi171 conmon[41853]: debug 2022-01-31T21:31:56.955+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:56.956017+0000) 2022-01-31T21:31:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:56 smithi171 conmon[46715]: debug 2022-01-31T21:31:56.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:56.999647+0000) 2022-01-31T21:31:57.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:57 smithi167 conmon[54076]: debug 2022-01-31T21:31:57.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:57.223891+0000) 2022-01-31T21:31:57.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:57 smithi167 conmon[60316]: debug 2022-01-31T21:31:57.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:57.223893+0000) 2022-01-31T21:31:57.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:57 smithi171 conmon[51620]: debug 2022-01-31T21:31:57.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:57.834113+0000) 2022-01-31T21:31:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:57 smithi167 conmon[49112]: debug 2022-01-31T21:31:57.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:57.560649+0000) 2022-01-31T21:31:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:57 smithi171 conmon[41853]: debug 2022-01-31T21:31:57.955+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:57.956230+0000) 2022-01-31T21:31:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:57 smithi171 conmon[46715]: debug 2022-01-31T21:31:57.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:57.999808+0000) 2022-01-31T21:31:58.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:58 smithi167 conmon[54076]: debug 2022-01-31T21:31:58.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:58.224116+0000) 2022-01-31T21:31:58.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:58 smithi167 conmon[60316]: debug 2022-01-31T21:31:58.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:58.224119+0000) 2022-01-31T21:31:58.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:58 smithi171 conmon[51620]: debug 2022-01-31T21:31:58.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:58.834319+0000) 2022-01-31T21:31:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:58 smithi167 conmon[49112]: debug 2022-01-31T21:31:58.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:58.560890+0000) 2022-01-31T21:31:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:58 smithi171 conmon[41853]: debug 2022-01-31T21:31:58.955+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:58.956442+0000) 2022-01-31T21:31:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:31:59 smithi171 conmon[46715]: debug 2022-01-31T21:31:58.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:59.000005+0000) 2022-01-31T21:31:59.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:31:59 smithi167 conmon[54076]: debug 2022-01-31T21:31:59.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:59.224259+0000) 2022-01-31T21:31:59.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:31:59 smithi167 conmon[60316]: debug 2022-01-31T21:31:59.223+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:59.224292+0000) 2022-01-31T21:31:59.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:31:59 smithi171 conmon[51620]: debug 2022-01-31T21:31:59.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:59.834526+0000) 2022-01-31T21:31:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:31:59 smithi167 conmon[49112]: debug 2022-01-31T21:31:59.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:59.561136+0000) 2022-01-31T21:32:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:31:59 smithi171 conmon[41853]: debug 2022-01-31T21:31:59.955+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:31:59.956640+0000) 2022-01-31T21:32:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[46715]: debug 2022-01-31T21:31:59.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.000186+0000) 2022-01-31T21:32:00.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:00 smithi167 conmon[54076]: debug 2022-01-31T21:32:00.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.224472+0000) 2022-01-31T21:32:00.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:00 smithi167 conmon[60316]: debug 2022-01-31T21:32:00.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.224489+0000) 2022-01-31T21:32:00.908 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[51620]: debug 2022-01-31T21:32:00.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.834726+0000) 2022-01-31T21:32:00.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:00 smithi167 conmon[49112]: debug 2022-01-31T21:32:00.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.561364+0000) 2022-01-31T21:32:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:00 smithi167 conmon[49112]: debug 2022-01-31T21:32:00.910+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.911290+0000) 2022-01-31T21:32:01.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:00 smithi167 conmon[54076]: debug 2022-01-31T21:32:00.910+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.911238+0000) 2022-01-31T21:32:01.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:00 smithi167 conmon[60316]: debug 2022-01-31T21:32:00.910+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.911109+0000) 2022-01-31T21:32:01.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[35325]: debug 2022-01-31T21:32:00.922+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137280 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:01.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[51620]: debug 2022-01-31T21:32:00.909+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.910198+0000) 2022-01-31T21:32:01.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[41853]: debug 2022-01-31T21:32:00.909+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.910436+0000) 2022-01-31T21:32:01.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[41853]: debug 2022-01-31T21:32:00.956+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.956766+0000) 2022-01-31T21:32:01.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:00 smithi171 conmon[46715]: debug 2022-01-31T21:32:00.910+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:00.911224+0000) 2022-01-31T21:32:01.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:01 smithi171 conmon[46715]: debug 2022-01-31T21:32:00.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:01.000405+0000) 2022-01-31T21:32:01.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:01 smithi167 conmon[54076]: debug 2022-01-31T21:32:01.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:01.224646+0000) 2022-01-31T21:32:01.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:01 smithi167 conmon[60316]: debug 2022-01-31T21:32:01.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:01.224663+0000) 2022-01-31T21:32:01.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:01 smithi171 conmon[51620]: debug 2022-01-31T21:32:01.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:01.834954+0000) 2022-01-31T21:32:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:01 smithi167 conmon[49112]: debug 2022-01-31T21:32:01.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:01.561577+0000) 2022-01-31T21:32:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:01 smithi171 conmon[41853]: debug 2022-01-31T21:32:01.956+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:01.956904+0000) 2022-01-31T21:32:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:02 smithi171 conmon[46715]: debug 2022-01-31T21:32:01.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:02.000577+0000) 2022-01-31T21:32:02.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:02 smithi167 conmon[54076]: debug 2022-01-31T21:32:02.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:02.224801+0000) 2022-01-31T21:32:02.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:02 smithi167 conmon[60316]: debug 2022-01-31T21:32:02.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:02.224807+0000) 2022-01-31T21:32:02.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:02 smithi171 conmon[51620]: debug 2022-01-31T21:32:02.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:02.835123+0000) 2022-01-31T21:32:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:02 smithi167 conmon[49112]: debug 2022-01-31T21:32:02.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:02.561685+0000) 2022-01-31T21:32:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:02 smithi171 conmon[41853]: debug 2022-01-31T21:32:02.956+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:02.957106+0000) 2022-01-31T21:32:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:03 smithi171 conmon[46715]: debug 2022-01-31T21:32:03.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:03.000795+0000) 2022-01-31T21:32:03.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:03 smithi167 conmon[54076]: debug 2022-01-31T21:32:03.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:03.225014+0000) 2022-01-31T21:32:03.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:03 smithi167 conmon[60316]: debug 2022-01-31T21:32:03.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:03.225003+0000) 2022-01-31T21:32:03.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:03 smithi171 conmon[51620]: debug 2022-01-31T21:32:03.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:03.835284+0000) 2022-01-31T21:32:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:03 smithi167 conmon[49112]: debug 2022-01-31T21:32:03.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:03.561854+0000) 2022-01-31T21:32:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:03 smithi171 conmon[41853]: debug 2022-01-31T21:32:03.956+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:03.957291+0000) 2022-01-31T21:32:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:04 smithi171 conmon[46715]: debug 2022-01-31T21:32:04.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:04.000983+0000) 2022-01-31T21:32:04.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:04 smithi167 conmon[54076]: debug 2022-01-31T21:32:04.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:04.225235+0000) 2022-01-31T21:32:04.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:04 smithi167 conmon[60316]: debug 2022-01-31T21:32:04.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:04.225232+0000) 2022-01-31T21:32:04.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:04 smithi171 conmon[51620]: debug 2022-01-31T21:32:04.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:04.835460+0000) 2022-01-31T21:32:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:04 smithi167 conmon[49112]: debug 2022-01-31T21:32:04.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:04.562015+0000) 2022-01-31T21:32:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:04 smithi171 conmon[41853]: debug 2022-01-31T21:32:04.956+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:04.957494+0000) 2022-01-31T21:32:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[46715]: debug 2022-01-31T21:32:05.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.001169+0000) 2022-01-31T21:32:05.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:05 smithi167 conmon[54076]: debug 2022-01-31T21:32:05.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.225421+0000) 2022-01-31T21:32:05.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:05 smithi167 conmon[60316]: debug 2022-01-31T21:32:05.224+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.225419+0000) 2022-01-31T21:32:05.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[51620]: debug 2022-01-31T21:32:05.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.835618+0000) 2022-01-31T21:32:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:05 smithi167 conmon[49112]: debug 2022-01-31T21:32:05.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.562233+0000) 2022-01-31T21:32:06.224 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:05 smithi167 conmon[49112]: debug 2022-01-31T21:32:05.932+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.933039+0000) 2022-01-31T21:32:06.224 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:05 smithi167 conmon[54076]: debug 2022-01-31T21:32:05.932+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.932690+0000) 2022-01-31T21:32:06.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:05 smithi167 conmon[60316]: debug 2022-01-31T21:32:05.932+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.933127+0000) 2022-01-31T21:32:06.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[35325]: debug 2022-01-31T21:32:05.945+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137390 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:06.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[41853]: debug 2022-01-31T21:32:05.925+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.925815+0000) 2022-01-31T21:32:06.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[41853]: debug 2022-01-31T21:32:05.956+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.957621+0000) 2022-01-31T21:32:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[46715]: debug 2022-01-31T21:32:05.925+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.926638+0000) 2022-01-31T21:32:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:06 smithi171 conmon[46715]: debug 2022-01-31T21:32:06.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:06.001296+0000) 2022-01-31T21:32:06.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:05 smithi171 conmon[51620]: debug 2022-01-31T21:32:05.925+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:05.926346+0000) 2022-01-31T21:32:06.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:06 smithi167 conmon[60316]: debug 2022-01-31T21:32:06.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:06.225606+0000) 2022-01-31T21:32:06.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:06 smithi167 conmon[54076]: debug 2022-01-31T21:32:06.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:06.225606+0000) 2022-01-31T21:32:06.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:06 smithi171 conmon[51620]: debug 2022-01-31T21:32:06.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:06.835810+0000) 2022-01-31T21:32:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:06 smithi167 conmon[49112]: debug 2022-01-31T21:32:06.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:06.562469+0000) 2022-01-31T21:32:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:06 smithi171 conmon[41853]: debug 2022-01-31T21:32:06.957+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:06.957779+0000) 2022-01-31T21:32:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:07 smithi171 conmon[46715]: debug 2022-01-31T21:32:07.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:07.001469+0000) 2022-01-31T21:32:07.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:07 smithi167 conmon[54076]: debug 2022-01-31T21:32:07.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:07.225766+0000) 2022-01-31T21:32:07.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:07 smithi167 conmon[60316]: debug 2022-01-31T21:32:07.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:07.225765+0000) 2022-01-31T21:32:07.912 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:07 smithi171 conmon[51620]: debug 2022-01-31T21:32:07.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:07.835930+0000) 2022-01-31T21:32:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:07 smithi167 conmon[49112]: debug 2022-01-31T21:32:07.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:07.562633+0000) 2022-01-31T21:32:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:08 smithi171 conmon[46715]: debug 2022-01-31T21:32:08.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:08.001637+0000) 2022-01-31T21:32:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:07 smithi171 conmon[41853]: debug 2022-01-31T21:32:07.957+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:07.957938+0000) 2022-01-31T21:32:08.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:08 smithi167 conmon[54076]: debug 2022-01-31T21:32:08.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:08.225962+0000) 2022-01-31T21:32:08.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:08 smithi167 conmon[60316]: debug 2022-01-31T21:32:08.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:08.225974+0000) 2022-01-31T21:32:08.913 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:08 smithi171 conmon[35325]: debug 2022-01-31T21:32:08.695+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:32:08.913 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:08 smithi171 conmon[51620]: debug 2022-01-31T21:32:08.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:08.836091+0000) 2022-01-31T21:32:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:08 smithi167 conmon[49112]: debug 2022-01-31T21:32:08.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:08.562864+0000) 2022-01-31T21:32:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:09 smithi171 conmon[46715]: debug 2022-01-31T21:32:09.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:09.001818+0000) 2022-01-31T21:32:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:08 smithi171 conmon[41853]: debug 2022-01-31T21:32:08.957+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:08.958145+0000) 2022-01-31T21:32:09.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:09 smithi167 conmon[54076]: debug 2022-01-31T21:32:09.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:09.226144+0000) 2022-01-31T21:32:09.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:09 smithi167 conmon[60316]: debug 2022-01-31T21:32:09.225+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:09.226165+0000) 2022-01-31T21:32:09.913 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:09 smithi171 conmon[51620]: debug 2022-01-31T21:32:09.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:09.836292+0000) 2022-01-31T21:32:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:09 smithi167 conmon[49112]: debug 2022-01-31T21:32:09.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:09.563049+0000) 2022-01-31T21:32:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[46715]: debug 2022-01-31T21:32:10.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.002002+0000) 2022-01-31T21:32:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:09 smithi171 conmon[41853]: debug 2022-01-31T21:32:09.957+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:09.958350+0000) 2022-01-31T21:32:10.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:10 smithi167 conmon[54076]: debug 2022-01-31T21:32:10.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.226352+0000) 2022-01-31T21:32:10.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:10 smithi167 conmon[60316]: debug 2022-01-31T21:32:10.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.226353+0000) 2022-01-31T21:32:10.913 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[51620]: debug 2022-01-31T21:32:10.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.836467+0000) 2022-01-31T21:32:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:10 smithi167 conmon[49112]: debug 2022-01-31T21:32:10.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.563244+0000) 2022-01-31T21:32:11.225 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:10 smithi167 conmon[49112]: debug 2022-01-31T21:32:10.948+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.948619+0000) 2022-01-31T21:32:11.225 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:10 smithi167 conmon[54076]: debug 2022-01-31T21:32:10.947+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.948275+0000) 2022-01-31T21:32:11.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:10 smithi167 conmon[60316]: debug 2022-01-31T21:32:10.948+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.948787+0000) 2022-01-31T21:32:11.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[35325]: debug 2022-01-31T21:32:10.962+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137497 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[46715]: debug 2022-01-31T21:32:10.947+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.948306+0000) 2022-01-31T21:32:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:11 smithi171 conmon[46715]: debug 2022-01-31T21:32:11.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:11.002191+0000) 2022-01-31T21:32:11.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[51620]: debug 2022-01-31T21:32:10.947+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.947973+0000) 2022-01-31T21:32:11.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[41853]: debug 2022-01-31T21:32:10.947+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.948661+0000) 2022-01-31T21:32:11.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:10 smithi171 conmon[41853]: debug 2022-01-31T21:32:10.957+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:10.958560+0000) 2022-01-31T21:32:11.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:11 smithi167 conmon[54076]: debug 2022-01-31T21:32:11.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:11.226545+0000) 2022-01-31T21:32:11.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:11 smithi167 conmon[60316]: debug 2022-01-31T21:32:11.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:11.226547+0000) 2022-01-31T21:32:11.913 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:11 smithi171 conmon[51620]: debug 2022-01-31T21:32:11.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:11.836651+0000) 2022-01-31T21:32:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:11 smithi167 conmon[49112]: debug 2022-01-31T21:32:11.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:11.563433+0000) 2022-01-31T21:32:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:12 smithi171 conmon[46715]: debug 2022-01-31T21:32:12.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:12.002349+0000) 2022-01-31T21:32:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:11 smithi171 conmon[41853]: debug 2022-01-31T21:32:11.958+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:11.958687+0000) 2022-01-31T21:32:12.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:12 smithi167 conmon[54076]: debug 2022-01-31T21:32:12.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:12.226705+0000) 2022-01-31T21:32:12.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:12 smithi167 conmon[60316]: debug 2022-01-31T21:32:12.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:12.226700+0000) 2022-01-31T21:32:12.913 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:12 smithi171 conmon[51620]: debug 2022-01-31T21:32:12.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:12.836815+0000) 2022-01-31T21:32:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:12 smithi167 conmon[49112]: debug 2022-01-31T21:32:12.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:12.563568+0000) 2022-01-31T21:32:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:12 smithi171 conmon[41853]: debug 2022-01-31T21:32:12.958+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:12.958843+0000) 2022-01-31T21:32:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:13 smithi171 conmon[46715]: debug 2022-01-31T21:32:13.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:13.002528+0000) 2022-01-31T21:32:13.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:13 smithi167 conmon[54076]: debug 2022-01-31T21:32:13.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:13.226913+0000) 2022-01-31T21:32:13.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:13 smithi167 conmon[60316]: debug 2022-01-31T21:32:13.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:13.226912+0000) 2022-01-31T21:32:13.913 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:13 smithi171 conmon[51620]: debug 2022-01-31T21:32:13.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:13.837008+0000) 2022-01-31T21:32:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:13 smithi167 conmon[49112]: debug 2022-01-31T21:32:13.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:13.563741+0000) 2022-01-31T21:32:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:13 smithi171 conmon[41853]: debug 2022-01-31T21:32:13.958+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:13.958986+0000) 2022-01-31T21:32:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:14 smithi171 conmon[46715]: debug 2022-01-31T21:32:14.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:14.002714+0000) 2022-01-31T21:32:14.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:14 smithi167 conmon[54076]: debug 2022-01-31T21:32:14.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:14.227121+0000) 2022-01-31T21:32:14.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:14 smithi167 conmon[60316]: debug 2022-01-31T21:32:14.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:14.227125+0000) 2022-01-31T21:32:14.914 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:14 smithi171 conmon[51620]: debug 2022-01-31T21:32:14.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:14.837187+0000) 2022-01-31T21:32:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:14 smithi167 conmon[49112]: debug 2022-01-31T21:32:14.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:14.563887+0000) 2022-01-31T21:32:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:14 smithi171 conmon[41853]: debug 2022-01-31T21:32:14.958+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:14.959169+0000) 2022-01-31T21:32:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[46715]: debug 2022-01-31T21:32:15.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.002950+0000) 2022-01-31T21:32:15.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:15 smithi167 conmon[54076]: debug 2022-01-31T21:32:15.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.227311+0000) 2022-01-31T21:32:15.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:15 smithi167 conmon[60316]: debug 2022-01-31T21:32:15.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.227237+0000) 2022-01-31T21:32:15.914 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[51620]: debug 2022-01-31T21:32:15.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.837389+0000) 2022-01-31T21:32:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:15 smithi167 conmon[49112]: debug 2022-01-31T21:32:15.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.564097+0000) 2022-01-31T21:32:16.225 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:15 smithi167 conmon[49112]: debug 2022-01-31T21:32:15.963+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.965477+0000) 2022-01-31T21:32:16.226 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:15 smithi167 conmon[54076]: debug 2022-01-31T21:32:15.964+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.965841+0000) 2022-01-31T21:32:16.227 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:15 smithi167 conmon[60316]: debug 2022-01-31T21:32:15.963+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.965045+0000) 2022-01-31T21:32:16.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[35325]: debug 2022-01-31T21:32:15.979+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137607 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[46715]: debug 2022-01-31T21:32:15.965+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.966400+0000) 2022-01-31T21:32:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:16 smithi171 conmon[46715]: debug 2022-01-31T21:32:16.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:16.003080+0000) 2022-01-31T21:32:16.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[51620]: debug 2022-01-31T21:32:15.965+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.966005+0000) 2022-01-31T21:32:16.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[41853]: debug 2022-01-31T21:32:15.958+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.959355+0000) 2022-01-31T21:32:16.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:15 smithi171 conmon[41853]: debug 2022-01-31T21:32:15.964+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:15.965426+0000) 2022-01-31T21:32:16.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:16 smithi167 conmon[54076]: debug 2022-01-31T21:32:16.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:16.227463+0000) 2022-01-31T21:32:16.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:16 smithi167 conmon[60316]: debug 2022-01-31T21:32:16.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:16.227382+0000) 2022-01-31T21:32:16.914 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:16 smithi171 conmon[51620]: debug 2022-01-31T21:32:16.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:16.837611+0000) 2022-01-31T21:32:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:16 smithi167 conmon[49112]: debug 2022-01-31T21:32:16.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:16.564320+0000) 2022-01-31T21:32:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:17 smithi171 conmon[46715]: debug 2022-01-31T21:32:17.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:17.003231+0000) 2022-01-31T21:32:17.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:16 smithi171 conmon[41853]: debug 2022-01-31T21:32:16.958+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:16.959487+0000) 2022-01-31T21:32:17.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:17 smithi167 conmon[54076]: debug 2022-01-31T21:32:17.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:17.227611+0000) 2022-01-31T21:32:17.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:17 smithi167 conmon[60316]: debug 2022-01-31T21:32:17.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:17.227539+0000) 2022-01-31T21:32:17.914 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:17 smithi171 conmon[51620]: debug 2022-01-31T21:32:17.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:17.837802+0000) 2022-01-31T21:32:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:17 smithi167 conmon[49112]: debug 2022-01-31T21:32:17.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:17.564493+0000) 2022-01-31T21:32:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:18 smithi171 conmon[46715]: debug 2022-01-31T21:32:18.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:18.003436+0000) 2022-01-31T21:32:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:17 smithi171 conmon[41853]: debug 2022-01-31T21:32:17.959+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:17.959645+0000) 2022-01-31T21:32:18.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:18 smithi167 conmon[54076]: debug 2022-01-31T21:32:18.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:18.227821+0000) 2022-01-31T21:32:18.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:18 smithi167 conmon[60316]: debug 2022-01-31T21:32:18.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:18.227724+0000) 2022-01-31T21:32:18.915 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:18 smithi171 conmon[51620]: debug 2022-01-31T21:32:18.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:18.837985+0000) 2022-01-31T21:32:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:18 smithi167 conmon[49112]: debug 2022-01-31T21:32:18.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:18.564714+0000) 2022-01-31T21:32:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:19 smithi171 conmon[46715]: debug 2022-01-31T21:32:19.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:19.003630+0000) 2022-01-31T21:32:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:18 smithi171 conmon[41853]: debug 2022-01-31T21:32:18.959+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:18.959830+0000) 2022-01-31T21:32:19.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:19 smithi167 conmon[54076]: debug 2022-01-31T21:32:19.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:19.227966+0000) 2022-01-31T21:32:19.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:19 smithi167 conmon[60316]: debug 2022-01-31T21:32:19.226+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:19.227889+0000) 2022-01-31T21:32:19.915 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:19 smithi171 conmon[51620]: debug 2022-01-31T21:32:19.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:19.838182+0000) 2022-01-31T21:32:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:19 smithi167 conmon[49112]: debug 2022-01-31T21:32:19.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:19.564885+0000) 2022-01-31T21:32:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:19 smithi171 conmon[41853]: debug 2022-01-31T21:32:19.959+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:19.960025+0000) 2022-01-31T21:32:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[46715]: debug 2022-01-31T21:32:20.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.003781+0000) 2022-01-31T21:32:20.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:20 smithi167 conmon[54076]: debug 2022-01-31T21:32:20.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.228083+0000) 2022-01-31T21:32:20.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:20 smithi167 conmon[60316]: debug 2022-01-31T21:32:20.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.228042+0000) 2022-01-31T21:32:20.915 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[51620]: debug 2022-01-31T21:32:20.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.838280+0000) 2022-01-31T21:32:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:20 smithi167 conmon[49112]: debug 2022-01-31T21:32:20.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.565090+0000) 2022-01-31T21:32:21.234 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:20 smithi167 conmon[49112]: debug 2022-01-31T21:32:20.981+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.982860+0000) 2022-01-31T21:32:21.235 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:20 smithi167 conmon[54076]: debug 2022-01-31T21:32:20.982+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.983698+0000) 2022-01-31T21:32:21.236 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:21 smithi167 conmon[54076]: debug 2022-01-31T21:32:21.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:21.228240+0000) 2022-01-31T21:32:21.236 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:20 smithi167 conmon[60316]: debug 2022-01-31T21:32:20.981+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.983068+0000) 2022-01-31T21:32:21.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:21 smithi167 conmon[60316]: debug 2022-01-31T21:32:21.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:21.228241+0000) 2022-01-31T21:32:21.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[35325]: debug 2022-01-31T21:32:20.997+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137718 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:21.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[51620]: debug 2022-01-31T21:32:20.983+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.983846+0000) 2022-01-31T21:32:21.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[41853]: debug 2022-01-31T21:32:20.959+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.960209+0000) 2022-01-31T21:32:21.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[41853]: debug 2022-01-31T21:32:20.981+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.982522+0000) 2022-01-31T21:32:21.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:20 smithi171 conmon[46715]: debug 2022-01-31T21:32:20.982+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:20.983136+0000) 2022-01-31T21:32:21.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:21 smithi171 conmon[46715]: debug 2022-01-31T21:32:21.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:21.003937+0000) 2022-01-31T21:32:21.915 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:21 smithi171 conmon[51620]: debug 2022-01-31T21:32:21.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:21.838464+0000) 2022-01-31T21:32:21.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:21 smithi167 conmon[49112]: debug 2022-01-31T21:32:21.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:21.565256+0000) 2022-01-31T21:32:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:21 smithi171 conmon[41853]: debug 2022-01-31T21:32:21.960+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:21.960389+0000) 2022-01-31T21:32:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:22 smithi171 conmon[46715]: debug 2022-01-31T21:32:22.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:22.004095+0000) 2022-01-31T21:32:22.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:22 smithi167 conmon[54076]: debug 2022-01-31T21:32:22.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:22.228397+0000) 2022-01-31T21:32:22.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:22 smithi167 conmon[60316]: debug 2022-01-31T21:32:22.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:22.228402+0000) 2022-01-31T21:32:22.915 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:22 smithi171 conmon[51620]: debug 2022-01-31T21:32:22.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:22.838655+0000) 2022-01-31T21:32:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:22 smithi167 conmon[49112]: debug 2022-01-31T21:32:22.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:22.565373+0000) 2022-01-31T21:32:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:22 smithi171 conmon[41853]: debug 2022-01-31T21:32:22.959+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:22.960557+0000) 2022-01-31T21:32:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:23 smithi171 conmon[46715]: debug 2022-01-31T21:32:23.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:23.004295+0000) 2022-01-31T21:32:23.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:23 smithi167 conmon[54076]: debug 2022-01-31T21:32:23.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:23.228584+0000) 2022-01-31T21:32:23.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:23 smithi167 conmon[60316]: debug 2022-01-31T21:32:23.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:23.228568+0000) 2022-01-31T21:32:23.916 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:23 smithi171 conmon[35325]: debug 2022-01-31T21:32:23.696+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:32:23.916 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:23 smithi171 conmon[51620]: debug 2022-01-31T21:32:23.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:23.838799+0000) 2022-01-31T21:32:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:23 smithi167 conmon[49112]: debug 2022-01-31T21:32:23.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:23.565543+0000) 2022-01-31T21:32:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:24 smithi171 conmon[46715]: debug 2022-01-31T21:32:24.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:24.004496+0000) 2022-01-31T21:32:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:23 smithi171 conmon[41853]: debug 2022-01-31T21:32:23.960+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:23.960790+0000) 2022-01-31T21:32:24.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:24 smithi167 conmon[54076]: debug 2022-01-31T21:32:24.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:24.228781+0000) 2022-01-31T21:32:24.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:24 smithi167 conmon[60316]: debug 2022-01-31T21:32:24.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:24.228779+0000) 2022-01-31T21:32:24.916 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:24 smithi171 conmon[51620]: debug 2022-01-31T21:32:24.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:24.838982+0000) 2022-01-31T21:32:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:24 smithi167 conmon[49112]: debug 2022-01-31T21:32:24.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:24.565683+0000) 2022-01-31T21:32:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:25 smithi171 conmon[46715]: debug 2022-01-31T21:32:25.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.004652+0000) 2022-01-31T21:32:25.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:24 smithi171 conmon[41853]: debug 2022-01-31T21:32:24.960+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:24.960993+0000) 2022-01-31T21:32:25.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:25 smithi167 conmon[54076]: debug 2022-01-31T21:32:25.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.228995+0000) 2022-01-31T21:32:25.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:25 smithi167 conmon[60316]: debug 2022-01-31T21:32:25.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.228990+0000) 2022-01-31T21:32:25.916 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:25 smithi171 conmon[51620]: debug 2022-01-31T21:32:25.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.839223+0000) 2022-01-31T21:32:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:25 smithi167 conmon[49112]: debug 2022-01-31T21:32:25.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.565961+0000) 2022-01-31T21:32:26.284 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:25 smithi167 conmon[49112]: debug 2022-01-31T21:32:25.998+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.999840+0000) 2022-01-31T21:32:26.284 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:26 smithi167 conmon[60316]: debug 2022-01-31T21:32:25.999+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.000828+0000) 2022-01-31T21:32:26.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:26 smithi167 conmon[60316]: debug 2022-01-31T21:32:26.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.229159+0000) 2022-01-31T21:32:26.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:26 smithi167 conmon[54076]: debug 2022-01-31T21:32:25.998+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.000447+0000) 2022-01-31T21:32:26.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:26 smithi167 conmon[54076]: debug 2022-01-31T21:32:26.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.229165+0000) 2022-01-31T21:32:26.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[35325]: debug 2022-01-31T21:32:26.014+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137828 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:26.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[51620]: debug 2022-01-31T21:32:26.000+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.001703+0000) 2022-01-31T21:32:26.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:25 smithi171 conmon[41853]: debug 2022-01-31T21:32:25.960+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.961259+0000) 2022-01-31T21:32:26.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[41853]: debug 2022-01-31T21:32:25.999+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:25.999912+0000) 2022-01-31T21:32:26.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[46715]: debug 2022-01-31T21:32:25.999+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.000425+0000) 2022-01-31T21:32:26.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[46715]: debug 2022-01-31T21:32:26.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.004790+0000) 2022-01-31T21:32:26.916 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[51620]: debug 2022-01-31T21:32:26.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.839380+0000) 2022-01-31T21:32:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:26 smithi167 conmon[49112]: debug 2022-01-31T21:32:26.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.566132+0000) 2022-01-31T21:32:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:26 smithi171 conmon[41853]: debug 2022-01-31T21:32:26.961+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:26.961442+0000) 2022-01-31T21:32:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:27 smithi171 conmon[46715]: debug 2022-01-31T21:32:27.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:27.004924+0000) 2022-01-31T21:32:27.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:27 smithi167 conmon[54076]: debug 2022-01-31T21:32:27.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:27.229296+0000) 2022-01-31T21:32:27.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:27 smithi167 conmon[60316]: debug 2022-01-31T21:32:27.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:27.229288+0000) 2022-01-31T21:32:27.916 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:27 smithi171 conmon[51620]: debug 2022-01-31T21:32:27.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:27.839525+0000) 2022-01-31T21:32:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:27 smithi167 conmon[49112]: debug 2022-01-31T21:32:27.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:27.566292+0000) 2022-01-31T21:32:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:27 smithi171 conmon[41853]: debug 2022-01-31T21:32:27.960+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:27.961633+0000) 2022-01-31T21:32:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:28 smithi171 conmon[46715]: debug 2022-01-31T21:32:28.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:28.005112+0000) 2022-01-31T21:32:28.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:28 smithi167 conmon[54076]: debug 2022-01-31T21:32:28.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:28.229479+0000) 2022-01-31T21:32:28.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:28 smithi167 conmon[60316]: debug 2022-01-31T21:32:28.227+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:28.229471+0000) 2022-01-31T21:32:28.916 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:28 smithi171 conmon[51620]: debug 2022-01-31T21:32:28.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:28.839719+0000) 2022-01-31T21:32:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:28 smithi167 conmon[49112]: debug 2022-01-31T21:32:28.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:28.566465+0000) 2022-01-31T21:32:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:28 smithi171 conmon[41853]: debug 2022-01-31T21:32:28.961+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:28.961843+0000) 2022-01-31T21:32:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:29 smithi171 conmon[46715]: debug 2022-01-31T21:32:29.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:29.005304+0000) 2022-01-31T21:32:29.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:29 smithi167 conmon[54076]: debug 2022-01-31T21:32:29.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:29.229645+0000) 2022-01-31T21:32:29.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:29 smithi167 conmon[60316]: debug 2022-01-31T21:32:29.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:29.229681+0000) 2022-01-31T21:32:29.917 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:29 smithi171 conmon[51620]: debug 2022-01-31T21:32:29.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:29.839964+0000) 2022-01-31T21:32:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:29 smithi167 conmon[49112]: debug 2022-01-31T21:32:29.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:29.566662+0000) 2022-01-31T21:32:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:29 smithi171 conmon[41853]: debug 2022-01-31T21:32:29.961+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:29.961999+0000) 2022-01-31T21:32:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:30 smithi171 conmon[46715]: debug 2022-01-31T21:32:30.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:30.005535+0000) 2022-01-31T21:32:30.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:30 smithi167 conmon[54076]: debug 2022-01-31T21:32:30.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:30.229790+0000) 2022-01-31T21:32:30.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:30 smithi167 conmon[60316]: debug 2022-01-31T21:32:30.228+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:30.229855+0000) 2022-01-31T21:32:30.917 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:30 smithi171 conmon[51620]: debug 2022-01-31T21:32:30.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:30.840140+0000) 2022-01-31T21:32:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:30 smithi167 conmon[49112]: debug 2022-01-31T21:32:30.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:30.566827+0000) 2022-01-31T21:32:31.284 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:31 smithi167 conmon[49112]: debug 2022-01-31T21:32:31.015+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.017037+0000) 2022-01-31T21:32:31.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:31 smithi167 conmon[54076]: debug 2022-01-31T21:32:31.015+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.017335+0000) 2022-01-31T21:32:31.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:31 smithi167 conmon[54076]: debug 2022-01-31T21:32:31.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.229946+0000) 2022-01-31T21:32:31.286 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:31 smithi167 conmon[60316]: debug 2022-01-31T21:32:31.017+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.018569+0000) 2022-01-31T21:32:31.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:31 smithi167 conmon[60316]: debug 2022-01-31T21:32:31.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.230018+0000) 2022-01-31T21:32:31.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[35325]: debug 2022-01-31T21:32:31.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 137939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:31.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[51620]: debug 2022-01-31T21:32:31.018+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.018999+0000) 2022-01-31T21:32:31.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:30 smithi171 conmon[41853]: debug 2022-01-31T21:32:30.961+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:30.962162+0000) 2022-01-31T21:32:31.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[41853]: debug 2022-01-31T21:32:31.017+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.018350+0000) 2022-01-31T21:32:31.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[46715]: debug 2022-01-31T21:32:31.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.005746+0000) 2022-01-31T21:32:31.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[46715]: debug 2022-01-31T21:32:31.017+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.018235+0000) 2022-01-31T21:32:31.917 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[51620]: debug 2022-01-31T21:32:31.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.840318+0000) 2022-01-31T21:32:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:31 smithi167 conmon[49112]: debug 2022-01-31T21:32:31.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.566986+0000) 2022-01-31T21:32:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:31 smithi171 conmon[41853]: debug 2022-01-31T21:32:31.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:31.962320+0000) 2022-01-31T21:32:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:32 smithi171 conmon[46715]: debug 2022-01-31T21:32:32.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:32.005914+0000) 2022-01-31T21:32:32.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:32 smithi167 conmon[54076]: debug 2022-01-31T21:32:32.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:32.230101+0000) 2022-01-31T21:32:32.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:32 smithi167 conmon[60316]: debug 2022-01-31T21:32:32.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:32.230160+0000) 2022-01-31T21:32:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:32 smithi167 conmon[49112]: debug 2022-01-31T21:32:32.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:32.567128+0000) 2022-01-31T21:32:33.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:32 smithi171 conmon[41853]: debug 2022-01-31T21:32:32.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:32.962486+0000) 2022-01-31T21:32:33.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:33 smithi171 conmon[46715]: debug 2022-01-31T21:32:33.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:33.006065+0000) 2022-01-31T21:32:33.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:32 smithi171 conmon[51620]: debug 2022-01-31T21:32:32.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:32.840479+0000) 2022-01-31T21:32:33.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:33 smithi167 conmon[54076]: debug 2022-01-31T21:32:33.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:33.230286+0000) 2022-01-31T21:32:33.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:33 smithi167 conmon[60316]: debug 2022-01-31T21:32:33.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:33.230316+0000) 2022-01-31T21:32:33.917 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:33 smithi171 conmon[51620]: debug 2022-01-31T21:32:33.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:33.840660+0000) 2022-01-31T21:32:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:33 smithi167 conmon[49112]: debug 2022-01-31T21:32:33.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:33.567329+0000) 2022-01-31T21:32:34.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:34 smithi171 conmon[46715]: debug 2022-01-31T21:32:34.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:34.006218+0000) 2022-01-31T21:32:34.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:33 smithi171 conmon[41853]: debug 2022-01-31T21:32:33.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:33.962681+0000) 2022-01-31T21:32:34.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:34 smithi167 conmon[54076]: debug 2022-01-31T21:32:34.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:34.230519+0000) 2022-01-31T21:32:34.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:34 smithi167 conmon[60316]: debug 2022-01-31T21:32:34.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:34.230519+0000) 2022-01-31T21:32:34.917 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:34 smithi171 conmon[51620]: debug 2022-01-31T21:32:34.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:34.840851+0000) 2022-01-31T21:32:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:34 smithi167 conmon[49112]: debug 2022-01-31T21:32:34.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:34.567537+0000) 2022-01-31T21:32:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:35 smithi171 conmon[46715]: debug 2022-01-31T21:32:35.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:35.006405+0000) 2022-01-31T21:32:35.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:34 smithi171 conmon[41853]: debug 2022-01-31T21:32:34.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:34.962895+0000) 2022-01-31T21:32:35.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:35 smithi167 conmon[54076]: debug 2022-01-31T21:32:35.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:35.230742+0000) 2022-01-31T21:32:35.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:35 smithi167 conmon[60316]: debug 2022-01-31T21:32:35.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:35.230649+0000) 2022-01-31T21:32:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:35 smithi167 conmon[49112]: debug 2022-01-31T21:32:35.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:35.567701+0000) 2022-01-31T21:32:35.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:35 smithi171 conmon[51620]: debug 2022-01-31T21:32:35.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:35.841009+0000) 2022-01-31T21:32:36.286 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:36 smithi167 conmon[49112]: debug 2022-01-31T21:32:36.032+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.034175+0000) 2022-01-31T21:32:36.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:36 smithi167 conmon[54076]: debug 2022-01-31T21:32:36.033+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.035395+0000) 2022-01-31T21:32:36.287 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:36 smithi167 conmon[54076]: debug 2022-01-31T21:32:36.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.230877+0000) 2022-01-31T21:32:36.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:36 smithi167 conmon[60316]: debug 2022-01-31T21:32:36.033+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.035168+0000) 2022-01-31T21:32:36.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:36 smithi167 conmon[60316]: debug 2022-01-31T21:32:36.229+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.230773+0000) 2022-01-31T21:32:36.352 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[35325]: debug 2022-01-31T21:32:36.048+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138049 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:36.353 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:35 smithi171 conmon[41853]: debug 2022-01-31T21:32:35.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:35.963078+0000) 2022-01-31T21:32:36.354 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[41853]: debug 2022-01-31T21:32:36.034+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.035206+0000) 2022-01-31T21:32:36.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[46715]: debug 2022-01-31T21:32:36.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.006590+0000) 2022-01-31T21:32:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[46715]: debug 2022-01-31T21:32:36.033+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.034686+0000) 2022-01-31T21:32:36.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[51620]: debug 2022-01-31T21:32:36.034+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.035570+0000) 2022-01-31T21:32:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:36 smithi167 conmon[49112]: debug 2022-01-31T21:32:36.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.567828+0000) 2022-01-31T21:32:36.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[51620]: debug 2022-01-31T21:32:36.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.841176+0000) 2022-01-31T21:32:37.360 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:36 smithi171 conmon[41853]: debug 2022-01-31T21:32:36.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:36.963261+0000) 2022-01-31T21:32:37.361 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:37 smithi171 conmon[46715]: debug 2022-01-31T21:32:37.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:37.006747+0000) 2022-01-31T21:32:37.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:37 smithi167 conmon[54076]: debug 2022-01-31T21:32:37.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:37.231046+0000) 2022-01-31T21:32:37.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:37 smithi167 conmon[60316]: debug 2022-01-31T21:32:37.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:37.230928+0000) 2022-01-31T21:32:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:37 smithi167 conmon[49112]: debug 2022-01-31T21:32:37.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:37.567960+0000) 2022-01-31T21:32:37.918 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:37 smithi171 conmon[51620]: debug 2022-01-31T21:32:37.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:37.841334+0000) 2022-01-31T21:32:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:37 smithi171 conmon[41853]: debug 2022-01-31T21:32:37.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:37.963424+0000) 2022-01-31T21:32:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:38 smithi171 conmon[46715]: debug 2022-01-31T21:32:38.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:38.006913+0000) 2022-01-31T21:32:38.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:38 smithi167 conmon[54076]: debug 2022-01-31T21:32:38.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:38.231227+0000) 2022-01-31T21:32:38.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:38 smithi167 conmon[60316]: debug 2022-01-31T21:32:38.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:38.231118+0000) 2022-01-31T21:32:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:38 smithi167 conmon[49112]: debug 2022-01-31T21:32:38.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:38.568199+0000) 2022-01-31T21:32:38.921 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:38 smithi171 conmon[35325]: debug 2022-01-31T21:32:38.696+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:32:38.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:38 smithi171 conmon[51620]: debug 2022-01-31T21:32:38.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:38.841553+0000) 2022-01-31T21:32:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:38 smithi171 conmon[41853]: debug 2022-01-31T21:32:38.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:38.963555+0000) 2022-01-31T21:32:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:39 smithi171 conmon[46715]: debug 2022-01-31T21:32:39.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:39.007092+0000) 2022-01-31T21:32:39.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:39 smithi167 conmon[54076]: debug 2022-01-31T21:32:39.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:39.231388+0000) 2022-01-31T21:32:39.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:39 smithi167 conmon[60316]: debug 2022-01-31T21:32:39.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:39.231278+0000) 2022-01-31T21:32:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:39 smithi167 conmon[49112]: debug 2022-01-31T21:32:39.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:39.568362+0000) 2022-01-31T21:32:39.918 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:39 smithi171 conmon[51620]: debug 2022-01-31T21:32:39.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:39.841689+0000) 2022-01-31T21:32:40.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:39 smithi171 conmon[41853]: debug 2022-01-31T21:32:39.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:39.963715+0000) 2022-01-31T21:32:40.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:40 smithi171 conmon[46715]: debug 2022-01-31T21:32:40.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:40.007273+0000) 2022-01-31T21:32:40.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:40 smithi167 conmon[54076]: debug 2022-01-31T21:32:40.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:40.231520+0000) 2022-01-31T21:32:40.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:40 smithi167 conmon[60316]: debug 2022-01-31T21:32:40.230+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:40.231442+0000) 2022-01-31T21:32:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:40 smithi167 conmon[49112]: debug 2022-01-31T21:32:40.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:40.568576+0000) 2022-01-31T21:32:40.918 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:40 smithi171 conmon[51620]: debug 2022-01-31T21:32:40.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:40.841897+0000) 2022-01-31T21:32:41.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[35325]: debug 2022-01-31T21:32:41.065+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138160 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:41.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:40 smithi171 conmon[41853]: debug 2022-01-31T21:32:40.962+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:40.963912+0000) 2022-01-31T21:32:41.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[41853]: debug 2022-01-31T21:32:41.050+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.052137+0000) 2022-01-31T21:32:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[46715]: debug 2022-01-31T21:32:41.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.007456+0000) 2022-01-31T21:32:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[46715]: debug 2022-01-31T21:32:41.050+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.052628+0000) 2022-01-31T21:32:41.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[51620]: debug 2022-01-31T21:32:41.051+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.052807+0000) 2022-01-31T21:32:41.392 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:41 smithi167 conmon[49112]: debug 2022-01-31T21:32:41.051+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.051709+0000) 2022-01-31T21:32:41.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:41 smithi167 conmon[54076]: debug 2022-01-31T21:32:41.051+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.051588+0000) 2022-01-31T21:32:41.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:41 smithi167 conmon[54076]: debug 2022-01-31T21:32:41.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.231640+0000) 2022-01-31T21:32:41.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:41 smithi167 conmon[60316]: debug 2022-01-31T21:32:41.051+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.051884+0000) 2022-01-31T21:32:41.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:41 smithi167 conmon[60316]: debug 2022-01-31T21:32:41.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.231640+0000) 2022-01-31T21:32:41.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:41 smithi167 conmon[49112]: debug 2022-01-31T21:32:41.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.568770+0000) 2022-01-31T21:32:41.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[51620]: debug 2022-01-31T21:32:41.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.842102+0000) 2022-01-31T21:32:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:41 smithi171 conmon[41853]: debug 2022-01-31T21:32:41.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:41.964074+0000) 2022-01-31T21:32:42.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:42 smithi171 conmon[46715]: debug 2022-01-31T21:32:42.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:42.007643+0000) 2022-01-31T21:32:42.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:42 smithi167 conmon[54076]: debug 2022-01-31T21:32:42.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:42.231773+0000) 2022-01-31T21:32:42.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:42 smithi167 conmon[60316]: debug 2022-01-31T21:32:42.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:42.231773+0000) 2022-01-31T21:32:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:42 smithi167 conmon[49112]: debug 2022-01-31T21:32:42.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:42.568924+0000) 2022-01-31T21:32:42.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:42 smithi171 conmon[51620]: debug 2022-01-31T21:32:42.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:42.842254+0000) 2022-01-31T21:32:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:42 smithi171 conmon[41853]: debug 2022-01-31T21:32:42.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:42.964227+0000) 2022-01-31T21:32:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:43 smithi171 conmon[46715]: debug 2022-01-31T21:32:43.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:43.007811+0000) 2022-01-31T21:32:43.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:43 smithi167 conmon[54076]: debug 2022-01-31T21:32:43.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:43.231975+0000) 2022-01-31T21:32:43.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:43 smithi167 conmon[60316]: debug 2022-01-31T21:32:43.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:43.231975+0000) 2022-01-31T21:32:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:43 smithi167 conmon[49112]: debug 2022-01-31T21:32:43.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:43.569125+0000) 2022-01-31T21:32:43.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:43 smithi171 conmon[51620]: debug 2022-01-31T21:32:43.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:43.842435+0000) 2022-01-31T21:32:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:43 smithi171 conmon[41853]: debug 2022-01-31T21:32:43.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:43.964382+0000) 2022-01-31T21:32:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:44 smithi171 conmon[46715]: debug 2022-01-31T21:32:44.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:44.007962+0000) 2022-01-31T21:32:44.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:44 smithi167 conmon[54076]: debug 2022-01-31T21:32:44.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:44.232175+0000) 2022-01-31T21:32:44.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:44 smithi167 conmon[60316]: debug 2022-01-31T21:32:44.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:44.232175+0000) 2022-01-31T21:32:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:44 smithi167 conmon[49112]: debug 2022-01-31T21:32:44.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:44.569351+0000) 2022-01-31T21:32:44.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:44 smithi171 conmon[51620]: debug 2022-01-31T21:32:44.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:44.842644+0000) 2022-01-31T21:32:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:44 smithi171 conmon[41853]: debug 2022-01-31T21:32:44.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:44.964563+0000) 2022-01-31T21:32:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:45 smithi171 conmon[46715]: debug 2022-01-31T21:32:45.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:45.008148+0000) 2022-01-31T21:32:45.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:45 smithi167 conmon[54076]: debug 2022-01-31T21:32:45.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:45.232364+0000) 2022-01-31T21:32:45.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:45 smithi167 conmon[60316]: debug 2022-01-31T21:32:45.231+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:45.232364+0000) 2022-01-31T21:32:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:45 smithi167 conmon[49112]: debug 2022-01-31T21:32:45.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:45.569580+0000) 2022-01-31T21:32:45.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:45 smithi171 conmon[51620]: debug 2022-01-31T21:32:45.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:45.842853+0000) 2022-01-31T21:32:46.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[35325]: debug 2022-01-31T21:32:46.082+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138270 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:46.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[51620]: debug 2022-01-31T21:32:46.067+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.068874+0000) 2022-01-31T21:32:46.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:45 smithi171 conmon[41853]: debug 2022-01-31T21:32:45.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:45.964749+0000) 2022-01-31T21:32:46.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[41853]: debug 2022-01-31T21:32:46.067+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.069557+0000) 2022-01-31T21:32:46.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[46715]: debug 2022-01-31T21:32:46.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.008327+0000) 2022-01-31T21:32:46.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[46715]: debug 2022-01-31T21:32:46.067+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.069342+0000) 2022-01-31T21:32:46.396 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:46 smithi167 conmon[49112]: debug 2022-01-31T21:32:46.068+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.068712+0000) 2022-01-31T21:32:46.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:46 smithi167 conmon[54076]: debug 2022-01-31T21:32:46.069+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.069586+0000) 2022-01-31T21:32:46.397 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:46 smithi167 conmon[54076]: debug 2022-01-31T21:32:46.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.232517+0000) 2022-01-31T21:32:46.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:46 smithi167 conmon[60316]: debug 2022-01-31T21:32:46.069+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.069796+0000) 2022-01-31T21:32:46.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:46 smithi167 conmon[60316]: debug 2022-01-31T21:32:46.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.232517+0000) 2022-01-31T21:32:46.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:46 smithi167 conmon[49112]: debug 2022-01-31T21:32:46.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.569764+0000) 2022-01-31T21:32:46.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[51620]: debug 2022-01-31T21:32:46.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.843022+0000) 2022-01-31T21:32:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:47 smithi171 conmon[46715]: debug 2022-01-31T21:32:47.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:47.008503+0000) 2022-01-31T21:32:47.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:46 smithi171 conmon[41853]: debug 2022-01-31T21:32:46.964+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:46.964938+0000) 2022-01-31T21:32:47.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:47 smithi167 conmon[54076]: debug 2022-01-31T21:32:47.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:47.232671+0000) 2022-01-31T21:32:47.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:47 smithi167 conmon[60316]: debug 2022-01-31T21:32:47.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:47.232671+0000) 2022-01-31T21:32:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:47 smithi167 conmon[49112]: debug 2022-01-31T21:32:47.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:47.569877+0000) 2022-01-31T21:32:47.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:47 smithi171 conmon[51620]: debug 2022-01-31T21:32:47.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:47.843159+0000) 2022-01-31T21:32:48.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:47 smithi171 conmon[41853]: debug 2022-01-31T21:32:47.964+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:47.965094+0000) 2022-01-31T21:32:48.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:48 smithi171 conmon[46715]: debug 2022-01-31T21:32:48.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:48.008678+0000) 2022-01-31T21:32:48.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:48 smithi167 conmon[54076]: debug 2022-01-31T21:32:48.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:48.232826+0000) 2022-01-31T21:32:48.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:48 smithi167 conmon[60316]: debug 2022-01-31T21:32:48.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:48.232824+0000) 2022-01-31T21:32:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:48 smithi167 conmon[49112]: debug 2022-01-31T21:32:48.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:48.570103+0000) 2022-01-31T21:32:48.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:48 smithi171 conmon[51620]: debug 2022-01-31T21:32:48.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:48.843338+0000) 2022-01-31T21:32:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:48 smithi171 conmon[41853]: debug 2022-01-31T21:32:48.963+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:48.965258+0000) 2022-01-31T21:32:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:49 smithi171 conmon[46715]: debug 2022-01-31T21:32:49.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:49.008893+0000) 2022-01-31T21:32:49.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:49 smithi167 conmon[54076]: debug 2022-01-31T21:32:49.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:49.233039+0000) 2022-01-31T21:32:49.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:49 smithi167 conmon[60316]: debug 2022-01-31T21:32:49.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:49.233040+0000) 2022-01-31T21:32:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:49 smithi167 conmon[49112]: debug 2022-01-31T21:32:49.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:49.570314+0000) 2022-01-31T21:32:49.920 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:49 smithi171 conmon[51620]: debug 2022-01-31T21:32:49.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:49.843524+0000) 2022-01-31T21:32:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:49 smithi171 conmon[41853]: debug 2022-01-31T21:32:49.964+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:49.965436+0000) 2022-01-31T21:32:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:50 smithi171 conmon[46715]: debug 2022-01-31T21:32:50.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:50.009100+0000) 2022-01-31T21:32:50.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:50 smithi167 conmon[54076]: debug 2022-01-31T21:32:50.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:50.233237+0000) 2022-01-31T21:32:50.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:50 smithi167 conmon[60316]: debug 2022-01-31T21:32:50.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:50.233237+0000) 2022-01-31T21:32:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:50 smithi167 conmon[49112]: debug 2022-01-31T21:32:50.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:50.570543+0000) 2022-01-31T21:32:50.920 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:50 smithi171 conmon[51620]: debug 2022-01-31T21:32:50.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:50.843723+0000) 2022-01-31T21:32:51.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[35325]: debug 2022-01-31T21:32:51.100+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138381 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:51.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[51620]: debug 2022-01-31T21:32:51.085+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.086239+0000) 2022-01-31T21:32:51.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:50 smithi171 conmon[41853]: debug 2022-01-31T21:32:50.964+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:50.965619+0000) 2022-01-31T21:32:51.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[41853]: debug 2022-01-31T21:32:51.085+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.086670+0000) 2022-01-31T21:32:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[46715]: debug 2022-01-31T21:32:51.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.009284+0000) 2022-01-31T21:32:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[46715]: debug 2022-01-31T21:32:51.085+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.085909+0000) 2022-01-31T21:32:51.404 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:51 smithi167 conmon[49112]: debug 2022-01-31T21:32:51.085+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.085493+0000) 2022-01-31T21:32:51.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:51 smithi167 conmon[60316]: debug 2022-01-31T21:32:51.085+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.086346+0000) 2022-01-31T21:32:51.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:51 smithi167 conmon[60316]: debug 2022-01-31T21:32:51.232+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.233455+0000) 2022-01-31T21:32:51.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:51 smithi167 conmon[54076]: debug 2022-01-31T21:32:51.085+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.085690+0000) 2022-01-31T21:32:51.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:51 smithi167 conmon[54076]: debug 2022-01-31T21:32:51.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.233455+0000) 2022-01-31T21:32:51.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:51 smithi167 conmon[49112]: debug 2022-01-31T21:32:51.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.570762+0000) 2022-01-31T21:32:51.920 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[51620]: debug 2022-01-31T21:32:51.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.843890+0000) 2022-01-31T21:32:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:52 smithi171 conmon[46715]: debug 2022-01-31T21:32:52.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:52.009471+0000) 2022-01-31T21:32:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:51 smithi171 conmon[41853]: debug 2022-01-31T21:32:51.965+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:51.965832+0000) 2022-01-31T21:32:52.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:52 smithi167 conmon[54076]: debug 2022-01-31T21:32:52.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:52.233627+0000) 2022-01-31T21:32:52.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:52 smithi167 conmon[60316]: debug 2022-01-31T21:32:52.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:52.233627+0000) 2022-01-31T21:32:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:52 smithi167 conmon[49112]: debug 2022-01-31T21:32:52.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:52.570870+0000) 2022-01-31T21:32:52.920 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:52 smithi171 conmon[51620]: debug 2022-01-31T21:32:52.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:52.844056+0000) 2022-01-31T21:32:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:52 smithi171 conmon[41853]: debug 2022-01-31T21:32:52.965+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:52.966028+0000) 2022-01-31T21:32:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:53 smithi171 conmon[46715]: debug 2022-01-31T21:32:53.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:53.009677+0000) 2022-01-31T21:32:53.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:53 smithi167 conmon[54076]: debug 2022-01-31T21:32:53.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:53.233757+0000) 2022-01-31T21:32:53.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:53 smithi167 conmon[60316]: debug 2022-01-31T21:32:53.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:53.233757+0000) 2022-01-31T21:32:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:53 smithi167 conmon[49112]: debug 2022-01-31T21:32:53.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:53.571050+0000) 2022-01-31T21:32:53.920 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:53 smithi171 conmon[35325]: debug 2022-01-31T21:32:53.697+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:32:53.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:53 smithi171 conmon[51620]: debug 2022-01-31T21:32:53.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:53.844228+0000) 2022-01-31T21:32:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:54 smithi171 conmon[46715]: debug 2022-01-31T21:32:54.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:54.009820+0000) 2022-01-31T21:32:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:53 smithi171 conmon[41853]: debug 2022-01-31T21:32:53.965+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:53.966226+0000) 2022-01-31T21:32:54.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:54 smithi167 conmon[54076]: debug 2022-01-31T21:32:54.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:54.233937+0000) 2022-01-31T21:32:54.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:54 smithi167 conmon[60316]: debug 2022-01-31T21:32:54.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:54.233937+0000) 2022-01-31T21:32:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:54 smithi167 conmon[49112]: debug 2022-01-31T21:32:54.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:54.571264+0000) 2022-01-31T21:32:54.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:54 smithi171 conmon[51620]: debug 2022-01-31T21:32:54.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:54.844461+0000) 2022-01-31T21:32:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:54 smithi171 conmon[41853]: debug 2022-01-31T21:32:54.965+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:54.966408+0000) 2022-01-31T21:32:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:55 smithi171 conmon[46715]: debug 2022-01-31T21:32:55.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:55.010004+0000) 2022-01-31T21:32:55.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:55 smithi167 conmon[54076]: debug 2022-01-31T21:32:55.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:55.234135+0000) 2022-01-31T21:32:55.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:55 smithi167 conmon[60316]: debug 2022-01-31T21:32:55.233+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:55.234135+0000) 2022-01-31T21:32:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:55 smithi167 conmon[49112]: debug 2022-01-31T21:32:55.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:55.571485+0000) 2022-01-31T21:32:55.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:55 smithi171 conmon[51620]: debug 2022-01-31T21:32:55.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:55.844624+0000) 2022-01-31T21:32:56.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[35325]: debug 2022-01-31T21:32:56.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138491 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:32:56.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[51620]: debug 2022-01-31T21:32:56.102+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.103337+0000) 2022-01-31T21:32:56.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:55 smithi171 conmon[41853]: debug 2022-01-31T21:32:55.965+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:55.966612+0000) 2022-01-31T21:32:56.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[41853]: debug 2022-01-31T21:32:56.102+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.103436+0000) 2022-01-31T21:32:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[46715]: debug 2022-01-31T21:32:56.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.010219+0000) 2022-01-31T21:32:56.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[46715]: debug 2022-01-31T21:32:56.102+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.102917+0000) 2022-01-31T21:32:56.413 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:56 smithi167 conmon[49112]: debug 2022-01-31T21:32:56.103+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.103614+0000) 2022-01-31T21:32:56.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:56 smithi167 conmon[54076]: debug 2022-01-31T21:32:56.103+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.103803+0000) 2022-01-31T21:32:56.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:56 smithi167 conmon[54076]: debug 2022-01-31T21:32:56.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.234348+0000) 2022-01-31T21:32:56.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:56 smithi167 conmon[60316]: debug 2022-01-31T21:32:56.102+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.102980+0000) 2022-01-31T21:32:56.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:56 smithi167 conmon[60316]: debug 2022-01-31T21:32:56.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.234348+0000) 2022-01-31T21:32:56.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:56 smithi167 conmon[49112]: debug 2022-01-31T21:32:56.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.571666+0000) 2022-01-31T21:32:56.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[51620]: debug 2022-01-31T21:32:56.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.844799+0000) 2022-01-31T21:32:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:57 smithi171 conmon[46715]: debug 2022-01-31T21:32:57.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:57.010369+0000) 2022-01-31T21:32:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:56 smithi171 conmon[41853]: debug 2022-01-31T21:32:56.966+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:56.966794+0000) 2022-01-31T21:32:57.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:57 smithi167 conmon[54076]: debug 2022-01-31T21:32:57.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:57.234515+0000) 2022-01-31T21:32:57.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:57 smithi167 conmon[60316]: debug 2022-01-31T21:32:57.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:57.234515+0000) 2022-01-31T21:32:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:57 smithi167 conmon[49112]: debug 2022-01-31T21:32:57.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:57.571823+0000) 2022-01-31T21:32:57.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:57 smithi171 conmon[51620]: debug 2022-01-31T21:32:57.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:57.844954+0000) 2022-01-31T21:32:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:58 smithi171 conmon[46715]: debug 2022-01-31T21:32:58.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:58.010499+0000) 2022-01-31T21:32:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:57 smithi171 conmon[41853]: debug 2022-01-31T21:32:57.966+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:57.966950+0000) 2022-01-31T21:32:58.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:58 smithi167 conmon[54076]: debug 2022-01-31T21:32:58.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:58.234652+0000) 2022-01-31T21:32:58.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:58 smithi167 conmon[60316]: debug 2022-01-31T21:32:58.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:58.234652+0000) 2022-01-31T21:32:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:58 smithi167 conmon[49112]: debug 2022-01-31T21:32:58.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:58.572063+0000) 2022-01-31T21:32:58.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:58 smithi171 conmon[51620]: debug 2022-01-31T21:32:58.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:58.845155+0000) 2022-01-31T21:32:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:58 smithi171 conmon[41853]: debug 2022-01-31T21:32:58.966+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:58.967128+0000) 2022-01-31T21:32:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:32:59 smithi171 conmon[46715]: debug 2022-01-31T21:32:59.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:59.010697+0000) 2022-01-31T21:32:59.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:32:59 smithi167 conmon[54076]: debug 2022-01-31T21:32:59.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:59.234834+0000) 2022-01-31T21:32:59.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:32:59 smithi167 conmon[60316]: debug 2022-01-31T21:32:59.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:59.234834+0000) 2022-01-31T21:32:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:32:59 smithi167 conmon[49112]: debug 2022-01-31T21:32:59.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:59.572243+0000) 2022-01-31T21:32:59.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:32:59 smithi171 conmon[51620]: debug 2022-01-31T21:32:59.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:59.845353+0000) 2022-01-31T21:33:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:32:59 smithi171 conmon[41853]: debug 2022-01-31T21:32:59.966+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:32:59.967330+0000) 2022-01-31T21:33:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:00 smithi171 conmon[46715]: debug 2022-01-31T21:33:00.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:00.010907+0000) 2022-01-31T21:33:00.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:00 smithi167 conmon[54076]: debug 2022-01-31T21:33:00.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:00.235016+0000) 2022-01-31T21:33:00.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:00 smithi167 conmon[60316]: debug 2022-01-31T21:33:00.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:00.235016+0000) 2022-01-31T21:33:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:00 smithi167 conmon[49112]: debug 2022-01-31T21:33:00.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:00.572436+0000) 2022-01-31T21:33:00.921 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:00 smithi171 conmon[51620]: debug 2022-01-31T21:33:00.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:00.845557+0000) 2022-01-31T21:33:01.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[35325]: debug 2022-01-31T21:33:01.134+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138602 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:01.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[51620]: debug 2022-01-31T21:33:01.119+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.119829+0000) 2022-01-31T21:33:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:00 smithi171 conmon[41853]: debug 2022-01-31T21:33:00.966+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:00.967534+0000) 2022-01-31T21:33:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[41853]: debug 2022-01-31T21:33:01.121+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.121835+0000) 2022-01-31T21:33:01.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[46715]: debug 2022-01-31T21:33:01.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.011097+0000) 2022-01-31T21:33:01.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[46715]: debug 2022-01-31T21:33:01.119+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.120302+0000) 2022-01-31T21:33:01.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:01 smithi167 conmon[49112]: debug 2022-01-31T21:33:01.120+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.120674+0000) 2022-01-31T21:33:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:01 smithi167 conmon[54076]: debug 2022-01-31T21:33:01.119+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.119569+0000) 2022-01-31T21:33:01.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:01 smithi167 conmon[54076]: debug 2022-01-31T21:33:01.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.235207+0000) 2022-01-31T21:33:01.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:01 smithi167 conmon[60316]: debug 2022-01-31T21:33:01.119+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.120012+0000) 2022-01-31T21:33:01.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:01 smithi167 conmon[60316]: debug 2022-01-31T21:33:01.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.235207+0000) 2022-01-31T21:33:01.703 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:01 smithi167 conmon[49112]: debug 2022-01-31T21:33:01.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.572642+0000) 2022-01-31T21:33:01.922 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[51620]: debug 2022-01-31T21:33:01.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.845741+0000) 2022-01-31T21:33:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:01 smithi171 conmon[41853]: debug 2022-01-31T21:33:01.967+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:01.967711+0000) 2022-01-31T21:33:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:02 smithi171 conmon[46715]: debug 2022-01-31T21:33:02.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:02.011284+0000) 2022-01-31T21:33:02.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:02 smithi167 conmon[54076]: debug 2022-01-31T21:33:02.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:02.235369+0000) 2022-01-31T21:33:02.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:02 smithi167 conmon[60316]: debug 2022-01-31T21:33:02.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:02.235369+0000) 2022-01-31T21:33:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:02 smithi167 conmon[49112]: debug 2022-01-31T21:33:02.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:02.572769+0000) 2022-01-31T21:33:02.922 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:02 smithi171 conmon[51620]: debug 2022-01-31T21:33:02.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:02.845902+0000) 2022-01-31T21:33:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:02 smithi171 conmon[41853]: debug 2022-01-31T21:33:02.967+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:02.967890+0000) 2022-01-31T21:33:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:03 smithi171 conmon[46715]: debug 2022-01-31T21:33:03.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:03.011469+0000) 2022-01-31T21:33:03.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:03 smithi167 conmon[54076]: debug 2022-01-31T21:33:03.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:03.235590+0000) 2022-01-31T21:33:03.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:03 smithi167 conmon[60316]: debug 2022-01-31T21:33:03.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:03.235590+0000) 2022-01-31T21:33:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:03 smithi167 conmon[49112]: debug 2022-01-31T21:33:03.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:03.572967+0000) 2022-01-31T21:33:03.922 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:03 smithi171 conmon[51620]: debug 2022-01-31T21:33:03.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:03.846058+0000) 2022-01-31T21:33:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:04 smithi171 conmon[46715]: debug 2022-01-31T21:33:04.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:04.011686+0000) 2022-01-31T21:33:04.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:03 smithi171 conmon[41853]: debug 2022-01-31T21:33:03.967+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:03.968045+0000) 2022-01-31T21:33:04.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:04 smithi167 conmon[54076]: debug 2022-01-31T21:33:04.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:04.235775+0000) 2022-01-31T21:33:04.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:04 smithi167 conmon[60316]: debug 2022-01-31T21:33:04.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:04.235775+0000) 2022-01-31T21:33:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:04 smithi167 conmon[49112]: debug 2022-01-31T21:33:04.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:04.573189+0000) 2022-01-31T21:33:04.922 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:04 smithi171 conmon[51620]: debug 2022-01-31T21:33:04.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:04.846213+0000) 2022-01-31T21:33:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:04 smithi171 conmon[41853]: debug 2022-01-31T21:33:04.967+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:04.968223+0000) 2022-01-31T21:33:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:05 smithi171 conmon[46715]: debug 2022-01-31T21:33:05.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:05.011837+0000) 2022-01-31T21:33:05.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:05 smithi167 conmon[54076]: debug 2022-01-31T21:33:05.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:05.235984+0000) 2022-01-31T21:33:05.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:05 smithi167 conmon[60316]: debug 2022-01-31T21:33:05.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:05.235984+0000) 2022-01-31T21:33:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:05 smithi167 conmon[49112]: debug 2022-01-31T21:33:05.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:05.573390+0000) 2022-01-31T21:33:05.923 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:05 smithi171 conmon[51620]: debug 2022-01-31T21:33:05.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:05.846396+0000) 2022-01-31T21:33:06.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[35325]: debug 2022-01-31T21:33:06.156+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138712 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:06.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[51620]: debug 2022-01-31T21:33:06.136+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.137257+0000) 2022-01-31T21:33:06.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:05 smithi171 conmon[41853]: debug 2022-01-31T21:33:05.967+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:05.968391+0000) 2022-01-31T21:33:06.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[41853]: debug 2022-01-31T21:33:06.137+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.139400+0000) 2022-01-31T21:33:06.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[46715]: debug 2022-01-31T21:33:06.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.011965+0000) 2022-01-31T21:33:06.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[46715]: debug 2022-01-31T21:33:06.137+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.138918+0000) 2022-01-31T21:33:06.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:06 smithi167 conmon[49112]: debug 2022-01-31T21:33:06.136+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.138310+0000) 2022-01-31T21:33:06.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:06 smithi167 conmon[54076]: debug 2022-01-31T21:33:06.135+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.137406+0000) 2022-01-31T21:33:06.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:06 smithi167 conmon[54076]: debug 2022-01-31T21:33:06.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.236197+0000) 2022-01-31T21:33:06.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:06 smithi167 conmon[60316]: debug 2022-01-31T21:33:06.136+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.138406+0000) 2022-01-31T21:33:06.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:06 smithi167 conmon[60316]: debug 2022-01-31T21:33:06.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.236197+0000) 2022-01-31T21:33:06.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:06 smithi167 conmon[49112]: debug 2022-01-31T21:33:06.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.573589+0000) 2022-01-31T21:33:06.923 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[51620]: debug 2022-01-31T21:33:06.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.846583+0000) 2022-01-31T21:33:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:07 smithi171 conmon[46715]: debug 2022-01-31T21:33:07.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:07.012159+0000) 2022-01-31T21:33:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:06 smithi171 conmon[41853]: debug 2022-01-31T21:33:06.968+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:06.968583+0000) 2022-01-31T21:33:07.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:07 smithi167 conmon[54076]: debug 2022-01-31T21:33:07.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:07.236385+0000) 2022-01-31T21:33:07.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:07 smithi167 conmon[60316]: debug 2022-01-31T21:33:07.234+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:07.236385+0000) 2022-01-31T21:33:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:07 smithi167 conmon[49112]: debug 2022-01-31T21:33:07.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:07.573725+0000) 2022-01-31T21:33:07.923 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:07 smithi171 conmon[51620]: debug 2022-01-31T21:33:07.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:07.846775+0000) 2022-01-31T21:33:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:08 smithi171 conmon[46715]: debug 2022-01-31T21:33:08.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:08.012323+0000) 2022-01-31T21:33:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:07 smithi171 conmon[41853]: debug 2022-01-31T21:33:07.968+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:07.968736+0000) 2022-01-31T21:33:08.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:08 smithi167 conmon[54076]: debug 2022-01-31T21:33:08.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:08.236617+0000) 2022-01-31T21:33:08.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:08 smithi167 conmon[60316]: debug 2022-01-31T21:33:08.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:08.236617+0000) 2022-01-31T21:33:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:08 smithi167 conmon[49112]: debug 2022-01-31T21:33:08.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:08.573877+0000) 2022-01-31T21:33:08.923 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:08 smithi171 conmon[35325]: debug 2022-01-31T21:33:08.698+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:33:08.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:08 smithi171 conmon[51620]: debug 2022-01-31T21:33:08.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:08.846966+0000) 2022-01-31T21:33:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:08 smithi171 conmon[41853]: debug 2022-01-31T21:33:08.968+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:08.968942+0000) 2022-01-31T21:33:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:09 smithi171 conmon[46715]: debug 2022-01-31T21:33:09.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:09.012470+0000) 2022-01-31T21:33:09.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:09 smithi167 conmon[54076]: debug 2022-01-31T21:33:09.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:09.236856+0000) 2022-01-31T21:33:09.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:09 smithi167 conmon[60316]: debug 2022-01-31T21:33:09.235+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:09.236856+0000) 2022-01-31T21:33:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:09 smithi167 conmon[49112]: debug 2022-01-31T21:33:09.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:09.574077+0000) 2022-01-31T21:33:09.923 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:09 smithi171 conmon[51620]: debug 2022-01-31T21:33:09.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:09.847149+0000) 2022-01-31T21:33:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:09 smithi171 conmon[41853]: debug 2022-01-31T21:33:09.968+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:09.969132+0000) 2022-01-31T21:33:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:10 smithi171 conmon[46715]: debug 2022-01-31T21:33:10.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:10.012650+0000) 2022-01-31T21:33:10.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:10 smithi167 conmon[54076]: debug 2022-01-31T21:33:10.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:10.237054+0000) 2022-01-31T21:33:10.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:10 smithi167 conmon[60316]: debug 2022-01-31T21:33:10.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:10.237053+0000) 2022-01-31T21:33:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:10 smithi167 conmon[49112]: debug 2022-01-31T21:33:10.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:10.574244+0000) 2022-01-31T21:33:10.923 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:10 smithi171 conmon[51620]: debug 2022-01-31T21:33:10.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:10.847335+0000) 2022-01-31T21:33:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[46715]: debug 2022-01-31T21:33:11.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.012795+0000) 2022-01-31T21:33:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[46715]: debug 2022-01-31T21:33:11.159+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.160172+0000) 2022-01-31T21:33:11.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[51620]: debug 2022-01-31T21:33:11.158+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.159494+0000) 2022-01-31T21:33:11.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[35325]: debug 2022-01-31T21:33:11.173+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138823 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:11.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:10 smithi171 conmon[41853]: debug 2022-01-31T21:33:10.968+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:10.969315+0000) 2022-01-31T21:33:11.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[41853]: debug 2022-01-31T21:33:11.159+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.160172+0000) 2022-01-31T21:33:11.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:11 smithi167 conmon[49112]: debug 2022-01-31T21:33:11.158+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.159953+0000) 2022-01-31T21:33:11.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:11 smithi167 conmon[54076]: debug 2022-01-31T21:33:11.158+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.159796+0000) 2022-01-31T21:33:11.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:11 smithi167 conmon[54076]: debug 2022-01-31T21:33:11.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.237265+0000) 2022-01-31T21:33:11.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:11 smithi167 conmon[60316]: debug 2022-01-31T21:33:11.157+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.158932+0000) 2022-01-31T21:33:11.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:11 smithi167 conmon[60316]: debug 2022-01-31T21:33:11.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.237266+0000) 2022-01-31T21:33:11.704 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:11 smithi167 conmon[49112]: debug 2022-01-31T21:33:11.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.574407+0000) 2022-01-31T21:33:11.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[51620]: debug 2022-01-31T21:33:11.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.847499+0000) 2022-01-31T21:33:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:12 smithi171 conmon[46715]: debug 2022-01-31T21:33:12.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:12.012950+0000) 2022-01-31T21:33:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:11 smithi171 conmon[41853]: debug 2022-01-31T21:33:11.969+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:11.969454+0000) 2022-01-31T21:33:12.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:12 smithi167 conmon[54076]: debug 2022-01-31T21:33:12.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:12.237451+0000) 2022-01-31T21:33:12.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:12 smithi167 conmon[60316]: debug 2022-01-31T21:33:12.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:12.237451+0000) 2022-01-31T21:33:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:12 smithi167 conmon[49112]: debug 2022-01-31T21:33:12.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:12.574573+0000) 2022-01-31T21:33:12.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:12 smithi171 conmon[51620]: debug 2022-01-31T21:33:12.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:12.847594+0000) 2022-01-31T21:33:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:12 smithi171 conmon[41853]: debug 2022-01-31T21:33:12.968+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:12.969606+0000) 2022-01-31T21:33:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:13 smithi171 conmon[46715]: debug 2022-01-31T21:33:13.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:13.013123+0000) 2022-01-31T21:33:13.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:13 smithi167 conmon[54076]: debug 2022-01-31T21:33:13.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:13.237661+0000) 2022-01-31T21:33:13.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:13 smithi167 conmon[60316]: debug 2022-01-31T21:33:13.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:13.237663+0000) 2022-01-31T21:33:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:13 smithi167 conmon[49112]: debug 2022-01-31T21:33:13.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:13.574772+0000) 2022-01-31T21:33:13.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:13 smithi171 conmon[51620]: debug 2022-01-31T21:33:13.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:13.847729+0000) 2022-01-31T21:33:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:14 smithi171 conmon[46715]: debug 2022-01-31T21:33:14.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:14.013279+0000) 2022-01-31T21:33:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:13 smithi171 conmon[41853]: debug 2022-01-31T21:33:13.969+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:13.969774+0000) 2022-01-31T21:33:14.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:14 smithi167 conmon[54076]: debug 2022-01-31T21:33:14.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:14.237869+0000) 2022-01-31T21:33:14.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:14 smithi167 conmon[60316]: debug 2022-01-31T21:33:14.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:14.237869+0000) 2022-01-31T21:33:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:14 smithi167 conmon[49112]: debug 2022-01-31T21:33:14.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:14.574981+0000) 2022-01-31T21:33:14.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:14 smithi171 conmon[51620]: debug 2022-01-31T21:33:14.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:14.847886+0000) 2022-01-31T21:33:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:15 smithi171 conmon[46715]: debug 2022-01-31T21:33:15.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:15.013471+0000) 2022-01-31T21:33:15.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:14 smithi171 conmon[41853]: debug 2022-01-31T21:33:14.969+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:14.969962+0000) 2022-01-31T21:33:15.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:15 smithi167 conmon[54076]: debug 2022-01-31T21:33:15.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:15.238092+0000) 2022-01-31T21:33:15.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:15 smithi167 conmon[60316]: debug 2022-01-31T21:33:15.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:15.238093+0000) 2022-01-31T21:33:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:15 smithi167 conmon[49112]: debug 2022-01-31T21:33:15.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:15.575186+0000) 2022-01-31T21:33:15.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:15 smithi171 conmon[51620]: debug 2022-01-31T21:33:15.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:15.848088+0000) 2022-01-31T21:33:16.175 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:15 smithi171 conmon[41853]: debug 2022-01-31T21:33:15.969+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:15.970074+0000) 2022-01-31T21:33:16.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[51620]: debug 2022-01-31T21:33:16.176+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.176818+0000) 2022-01-31T21:33:16.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[46715]: debug 2022-01-31T21:33:16.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.013623+0000) 2022-01-31T21:33:16.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[46715]: debug 2022-01-31T21:33:16.176+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.177706+0000) 2022-01-31T21:33:16.449 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:16 smithi167 conmon[49112]: debug 2022-01-31T21:33:16.177+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.178772+0000) 2022-01-31T21:33:16.449 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:16 smithi167 conmon[54076]: debug 2022-01-31T21:33:16.176+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.177694+0000) 2022-01-31T21:33:16.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:16 smithi167 conmon[54076]: debug 2022-01-31T21:33:16.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.238328+0000) 2022-01-31T21:33:16.450 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:16 smithi167 conmon[60316]: debug 2022-01-31T21:33:16.176+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.177846+0000) 2022-01-31T21:33:16.451 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:16 smithi167 conmon[60316]: debug 2022-01-31T21:33:16.236+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.238328+0000) 2022-01-31T21:33:16.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[35325]: debug 2022-01-31T21:33:16.191+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 138933 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[41853]: debug 2022-01-31T21:33:16.176+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.177553+0000) 2022-01-31T21:33:16.705 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:16 smithi167 conmon[49112]: debug 2022-01-31T21:33:16.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.575423+0000) 2022-01-31T21:33:16.924 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[51620]: debug 2022-01-31T21:33:16.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.848236+0000) 2022-01-31T21:33:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:16 smithi171 conmon[41853]: debug 2022-01-31T21:33:16.970+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:16.970241+0000) 2022-01-31T21:33:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:17 smithi171 conmon[46715]: debug 2022-01-31T21:33:17.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:17.013807+0000) 2022-01-31T21:33:17.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:17 smithi167 conmon[54076]: debug 2022-01-31T21:33:17.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:17.238519+0000) 2022-01-31T21:33:17.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:17 smithi167 conmon[60316]: debug 2022-01-31T21:33:17.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:17.238518+0000) 2022-01-31T21:33:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:17 smithi167 conmon[49112]: debug 2022-01-31T21:33:17.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:17.575561+0000) 2022-01-31T21:33:17.925 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:17 smithi171 conmon[51620]: debug 2022-01-31T21:33:17.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:17.848384+0000) 2022-01-31T21:33:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:18 smithi171 conmon[46715]: debug 2022-01-31T21:33:18.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:18.013950+0000) 2022-01-31T21:33:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:17 smithi171 conmon[41853]: debug 2022-01-31T21:33:17.970+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:17.970392+0000) 2022-01-31T21:33:18.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:18 smithi167 conmon[54076]: debug 2022-01-31T21:33:18.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:18.238721+0000) 2022-01-31T21:33:18.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:18 smithi167 conmon[60316]: debug 2022-01-31T21:33:18.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:18.238720+0000) 2022-01-31T21:33:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:18 smithi167 conmon[49112]: debug 2022-01-31T21:33:18.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:18.575688+0000) 2022-01-31T21:33:18.925 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:18 smithi171 conmon[51620]: debug 2022-01-31T21:33:18.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:18.848554+0000) 2022-01-31T21:33:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:18 smithi171 conmon[41853]: debug 2022-01-31T21:33:18.970+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:18.970542+0000) 2022-01-31T21:33:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:19 smithi171 conmon[46715]: debug 2022-01-31T21:33:19.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:19.014083+0000) 2022-01-31T21:33:19.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:19 smithi167 conmon[54076]: debug 2022-01-31T21:33:19.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:19.238929+0000) 2022-01-31T21:33:19.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:19 smithi167 conmon[60316]: debug 2022-01-31T21:33:19.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:19.238929+0000) 2022-01-31T21:33:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:19 smithi167 conmon[49112]: debug 2022-01-31T21:33:19.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:19.575811+0000) 2022-01-31T21:33:19.925 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:19 smithi171 conmon[51620]: debug 2022-01-31T21:33:19.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:19.848725+0000) 2022-01-31T21:33:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:19 smithi171 conmon[41853]: debug 2022-01-31T21:33:19.970+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:19.970706+0000) 2022-01-31T21:33:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:20 smithi171 conmon[46715]: debug 2022-01-31T21:33:20.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:20.014238+0000) 2022-01-31T21:33:20.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:20 smithi167 conmon[60316]: debug 2022-01-31T21:33:20.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:20.239136+0000) 2022-01-31T21:33:20.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:20 smithi167 conmon[54076]: debug 2022-01-31T21:33:20.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:20.239136+0000) 2022-01-31T21:33:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:20 smithi167 conmon[49112]: debug 2022-01-31T21:33:20.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:20.575975+0000) 2022-01-31T21:33:20.925 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:20 smithi171 conmon[51620]: debug 2022-01-31T21:33:20.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:20.848920+0000) 2022-01-31T21:33:21.192 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:20 smithi171 conmon[41853]: debug 2022-01-31T21:33:20.970+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:20.970901+0000) 2022-01-31T21:33:21.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[46715]: debug 2022-01-31T21:33:21.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.014415+0000) 2022-01-31T21:33:21.458 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:21 smithi167 conmon[49112]: debug 2022-01-31T21:33:21.192+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.194171+0000) 2022-01-31T21:33:21.459 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:21 smithi167 conmon[54076]: debug 2022-01-31T21:33:21.193+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.194527+0000) 2022-01-31T21:33:21.459 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:21 smithi167 conmon[54076]: debug 2022-01-31T21:33:21.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.239348+0000) 2022-01-31T21:33:21.460 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:21 smithi167 conmon[60316]: debug 2022-01-31T21:33:21.193+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.194601+0000) 2022-01-31T21:33:21.460 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:21 smithi167 conmon[60316]: debug 2022-01-31T21:33:21.237+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.239348+0000) 2022-01-31T21:33:21.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[35325]: debug 2022-01-31T21:33:21.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139044 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[41853]: debug 2022-01-31T21:33:21.193+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.194399+0000) 2022-01-31T21:33:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[46715]: debug 2022-01-31T21:33:21.194+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.195329+0000) 2022-01-31T21:33:21.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[51620]: debug 2022-01-31T21:33:21.195+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.195939+0000) 2022-01-31T21:33:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:21 smithi167 conmon[49112]: debug 2022-01-31T21:33:21.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.576131+0000) 2022-01-31T21:33:21.925 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[51620]: debug 2022-01-31T21:33:21.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.849104+0000) 2022-01-31T21:33:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:21 smithi171 conmon[41853]: debug 2022-01-31T21:33:21.970+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:21.971081+0000) 2022-01-31T21:33:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:22 smithi171 conmon[46715]: debug 2022-01-31T21:33:22.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:22.014574+0000) 2022-01-31T21:33:22.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:22 smithi167 conmon[60316]: debug 2022-01-31T21:33:22.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:22.239481+0000) 2022-01-31T21:33:22.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:22 smithi167 conmon[54076]: debug 2022-01-31T21:33:22.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:22.239481+0000) 2022-01-31T21:33:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:22 smithi167 conmon[49112]: debug 2022-01-31T21:33:22.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:22.576290+0000) 2022-01-31T21:33:22.925 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:22 smithi171 conmon[51620]: debug 2022-01-31T21:33:22.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:22.849220+0000) 2022-01-31T21:33:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:22 smithi171 conmon[41853]: debug 2022-01-31T21:33:22.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:22.971249+0000) 2022-01-31T21:33:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:23 smithi171 conmon[46715]: debug 2022-01-31T21:33:23.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:23.014711+0000) 2022-01-31T21:33:23.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:23 smithi167 conmon[54076]: debug 2022-01-31T21:33:23.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:23.239684+0000) 2022-01-31T21:33:23.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:23 smithi167 conmon[60316]: debug 2022-01-31T21:33:23.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:23.239685+0000) 2022-01-31T21:33:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:23 smithi167 conmon[49112]: debug 2022-01-31T21:33:23.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:23.576519+0000) 2022-01-31T21:33:23.926 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:23 smithi171 conmon[35325]: debug 2022-01-31T21:33:23.699+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:33:23.926 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:23 smithi171 conmon[51620]: debug 2022-01-31T21:33:23.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:23.849421+0000) 2022-01-31T21:33:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:23 smithi171 conmon[41853]: debug 2022-01-31T21:33:23.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:23.971398+0000) 2022-01-31T21:33:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:24 smithi171 conmon[46715]: debug 2022-01-31T21:33:24.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:24.014871+0000) 2022-01-31T21:33:24.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:24 smithi167 conmon[60316]: debug 2022-01-31T21:33:24.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:24.239888+0000) 2022-01-31T21:33:24.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:24 smithi167 conmon[54076]: debug 2022-01-31T21:33:24.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:24.239888+0000) 2022-01-31T21:33:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:24 smithi167 conmon[49112]: debug 2022-01-31T21:33:24.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:24.576699+0000) 2022-01-31T21:33:24.926 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:24 smithi171 conmon[51620]: debug 2022-01-31T21:33:24.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:24.849642+0000) 2022-01-31T21:33:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:24 smithi171 conmon[41853]: debug 2022-01-31T21:33:24.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:24.971598+0000) 2022-01-31T21:33:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:25 smithi171 conmon[46715]: debug 2022-01-31T21:33:25.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:25.015032+0000) 2022-01-31T21:33:25.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:25 smithi167 conmon[54076]: debug 2022-01-31T21:33:25.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:25.240046+0000) 2022-01-31T21:33:25.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:25 smithi167 conmon[60316]: debug 2022-01-31T21:33:25.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:25.240046+0000) 2022-01-31T21:33:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:25 smithi167 conmon[49112]: debug 2022-01-31T21:33:25.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:25.576857+0000) 2022-01-31T21:33:25.926 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:25 smithi171 conmon[51620]: debug 2022-01-31T21:33:25.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:25.849839+0000) 2022-01-31T21:33:26.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[46715]: debug 2022-01-31T21:33:26.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.015217+0000) 2022-01-31T21:33:26.210 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:25 smithi171 conmon[41853]: debug 2022-01-31T21:33:25.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:25.971748+0000) 2022-01-31T21:33:26.467 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:26 smithi167 conmon[49112]: debug 2022-01-31T21:33:26.211+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.212563+0000) 2022-01-31T21:33:26.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:26 smithi167 conmon[54076]: debug 2022-01-31T21:33:26.209+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.211402+0000) 2022-01-31T21:33:26.469 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:26 smithi167 conmon[54076]: debug 2022-01-31T21:33:26.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.240203+0000) 2022-01-31T21:33:26.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:26 smithi167 conmon[60316]: debug 2022-01-31T21:33:26.210+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.211791+0000) 2022-01-31T21:33:26.469 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:26 smithi167 conmon[60316]: debug 2022-01-31T21:33:26.238+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.240204+0000) 2022-01-31T21:33:26.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[35325]: debug 2022-01-31T21:33:26.225+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139155 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[41853]: debug 2022-01-31T21:33:26.211+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.212345+0000) 2022-01-31T21:33:26.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[46715]: debug 2022-01-31T21:33:26.211+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.212182+0000) 2022-01-31T21:33:26.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[51620]: debug 2022-01-31T21:33:26.211+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.212627+0000) 2022-01-31T21:33:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:26 smithi167 conmon[49112]: debug 2022-01-31T21:33:26.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.577028+0000) 2022-01-31T21:33:26.926 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[51620]: debug 2022-01-31T21:33:26.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.849985+0000) 2022-01-31T21:33:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:26 smithi171 conmon[41853]: debug 2022-01-31T21:33:26.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:26.971910+0000) 2022-01-31T21:33:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:27 smithi171 conmon[46715]: debug 2022-01-31T21:33:27.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:27.015405+0000) 2022-01-31T21:33:27.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:27 smithi167 conmon[54076]: debug 2022-01-31T21:33:27.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:27.240379+0000) 2022-01-31T21:33:27.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:27 smithi167 conmon[60316]: debug 2022-01-31T21:33:27.239+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:27.240380+0000) 2022-01-31T21:33:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:27 smithi167 conmon[49112]: debug 2022-01-31T21:33:27.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:27.577168+0000) 2022-01-31T21:33:27.926 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:27 smithi171 conmon[51620]: debug 2022-01-31T21:33:27.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:27.850137+0000) 2022-01-31T21:33:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:27 smithi171 conmon[41853]: debug 2022-01-31T21:33:27.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:27.972041+0000) 2022-01-31T21:33:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:28 smithi171 conmon[46715]: debug 2022-01-31T21:33:28.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:28.015584+0000) 2022-01-31T21:33:28.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:28 smithi167 conmon[54076]: debug 2022-01-31T21:33:28.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:28.240563+0000) 2022-01-31T21:33:28.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:28 smithi167 conmon[60316]: debug 2022-01-31T21:33:28.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:28.240563+0000) 2022-01-31T21:33:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:28 smithi167 conmon[49112]: debug 2022-01-31T21:33:28.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:28.577394+0000) 2022-01-31T21:33:28.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:28 smithi171 conmon[51620]: debug 2022-01-31T21:33:28.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:28.850360+0000) 2022-01-31T21:33:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:28 smithi171 conmon[41853]: debug 2022-01-31T21:33:28.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:28.972195+0000) 2022-01-31T21:33:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:29 smithi171 conmon[46715]: debug 2022-01-31T21:33:29.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:29.015804+0000) 2022-01-31T21:33:29.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:29 smithi167 conmon[54076]: debug 2022-01-31T21:33:29.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:29.240737+0000) 2022-01-31T21:33:29.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:29 smithi167 conmon[60316]: debug 2022-01-31T21:33:29.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:29.240735+0000) 2022-01-31T21:33:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:29 smithi167 conmon[49112]: debug 2022-01-31T21:33:29.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:29.577581+0000) 2022-01-31T21:33:29.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:29 smithi171 conmon[51620]: debug 2022-01-31T21:33:29.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:29.850594+0000) 2022-01-31T21:33:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:29 smithi171 conmon[41853]: debug 2022-01-31T21:33:29.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:29.972375+0000) 2022-01-31T21:33:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:30 smithi171 conmon[46715]: debug 2022-01-31T21:33:30.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:30.015919+0000) 2022-01-31T21:33:30.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:30 smithi167 conmon[54076]: debug 2022-01-31T21:33:30.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:30.240940+0000) 2022-01-31T21:33:30.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:30 smithi167 conmon[60316]: debug 2022-01-31T21:33:30.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:30.240938+0000) 2022-01-31T21:33:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:30 smithi167 conmon[49112]: debug 2022-01-31T21:33:30.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:30.577727+0000) 2022-01-31T21:33:30.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:30 smithi171 conmon[51620]: debug 2022-01-31T21:33:30.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:30.850799+0000) 2022-01-31T21:33:31.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[46715]: debug 2022-01-31T21:33:31.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.016108+0000) 2022-01-31T21:33:31.226 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:30 smithi171 conmon[41853]: debug 2022-01-31T21:33:30.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:30.972554+0000) 2022-01-31T21:33:31.482 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:31 smithi167 conmon[49112]: debug 2022-01-31T21:33:31.228+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.229339+0000) 2022-01-31T21:33:31.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:31 smithi167 conmon[54076]: debug 2022-01-31T21:33:31.228+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.228528+0000) 2022-01-31T21:33:31.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:31 smithi167 conmon[54076]: debug 2022-01-31T21:33:31.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.241157+0000) 2022-01-31T21:33:31.484 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:31 smithi167 conmon[60316]: debug 2022-01-31T21:33:31.228+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.228965+0000) 2022-01-31T21:33:31.484 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:31 smithi167 conmon[60316]: debug 2022-01-31T21:33:31.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.241157+0000) 2022-01-31T21:33:31.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[35325]: debug 2022-01-31T21:33:31.242+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139261 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[41853]: debug 2022-01-31T21:33:31.228+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.229875+0000) 2022-01-31T21:33:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[46715]: debug 2022-01-31T21:33:31.226+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.228642+0000) 2022-01-31T21:33:31.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[51620]: debug 2022-01-31T21:33:31.227+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.229610+0000) 2022-01-31T21:33:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:31 smithi167 conmon[49112]: debug 2022-01-31T21:33:31.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.577889+0000) 2022-01-31T21:33:31.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[51620]: debug 2022-01-31T21:33:31.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.850978+0000) 2022-01-31T21:33:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:31 smithi171 conmon[41853]: debug 2022-01-31T21:33:31.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:31.972731+0000) 2022-01-31T21:33:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:32 smithi171 conmon[46715]: debug 2022-01-31T21:33:32.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:32.016295+0000) 2022-01-31T21:33:32.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:32 smithi167 conmon[54076]: debug 2022-01-31T21:33:32.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:32.241335+0000) 2022-01-31T21:33:32.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:32 smithi167 conmon[60316]: debug 2022-01-31T21:33:32.240+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:32.241335+0000) 2022-01-31T21:33:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:32 smithi167 conmon[49112]: debug 2022-01-31T21:33:32.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:32.578076+0000) 2022-01-31T21:33:32.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:32 smithi171 conmon[51620]: debug 2022-01-31T21:33:32.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:32.851143+0000) 2022-01-31T21:33:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:32 smithi171 conmon[41853]: debug 2022-01-31T21:33:32.971+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:32.972910+0000) 2022-01-31T21:33:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:33 smithi171 conmon[46715]: debug 2022-01-31T21:33:33.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:33.016480+0000) 2022-01-31T21:33:33.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:33 smithi167 conmon[54076]: debug 2022-01-31T21:33:33.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:33.241556+0000) 2022-01-31T21:33:33.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:33 smithi167 conmon[60316]: debug 2022-01-31T21:33:33.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:33.241556+0000) 2022-01-31T21:33:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:33 smithi167 conmon[49112]: debug 2022-01-31T21:33:33.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:33.578241+0000) 2022-01-31T21:33:33.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:33 smithi171 conmon[51620]: debug 2022-01-31T21:33:33.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:33.851347+0000) 2022-01-31T21:33:34.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:34 smithi171 conmon[46715]: debug 2022-01-31T21:33:34.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:34.016661+0000) 2022-01-31T21:33:34.184 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:33 smithi171 conmon[41853]: debug 2022-01-31T21:33:33.972+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:33.973097+0000) 2022-01-31T21:33:34.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:34 smithi167 conmon[54076]: debug 2022-01-31T21:33:34.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:34.241771+0000) 2022-01-31T21:33:34.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:34 smithi167 conmon[60316]: debug 2022-01-31T21:33:34.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:34.241771+0000) 2022-01-31T21:33:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:34 smithi167 conmon[49112]: debug 2022-01-31T21:33:34.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:34.578441+0000) 2022-01-31T21:33:34.928 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:34 smithi171 conmon[51620]: debug 2022-01-31T21:33:34.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:34.851536+0000) 2022-01-31T21:33:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:34 smithi171 conmon[41853]: debug 2022-01-31T21:33:34.972+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:34.973261+0000) 2022-01-31T21:33:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:35 smithi171 conmon[46715]: debug 2022-01-31T21:33:35.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:35.016822+0000) 2022-01-31T21:33:35.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:35 smithi167 conmon[54076]: debug 2022-01-31T21:33:35.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:35.241987+0000) 2022-01-31T21:33:35.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:35 smithi167 conmon[60316]: debug 2022-01-31T21:33:35.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:35.241987+0000) 2022-01-31T21:33:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:35 smithi167 conmon[49112]: debug 2022-01-31T21:33:35.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:35.578654+0000) 2022-01-31T21:33:35.928 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:35 smithi171 conmon[51620]: debug 2022-01-31T21:33:35.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:35.851685+0000) 2022-01-31T21:33:36.243 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:35 smithi171 conmon[41853]: debug 2022-01-31T21:33:35.972+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:35.973418+0000) 2022-01-31T21:33:36.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[46715]: debug 2022-01-31T21:33:36.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.017006+0000) 2022-01-31T21:33:36.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:36 smithi167 conmon[49112]: debug 2022-01-31T21:33:36.245+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.246379+0000) 2022-01-31T21:33:36.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:36 smithi167 conmon[54076]: debug 2022-01-31T21:33:36.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.242204+0000) 2022-01-31T21:33:36.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:36 smithi167 conmon[54076]: debug 2022-01-31T21:33:36.245+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.246224+0000) 2022-01-31T21:33:36.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:36 smithi167 conmon[60316]: debug 2022-01-31T21:33:36.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.242204+0000) 2022-01-31T21:33:36.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:36 smithi167 conmon[60316]: debug 2022-01-31T21:33:36.245+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.245938+0000) 2022-01-31T21:33:36.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[35325]: debug 2022-01-31T21:33:36.259+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139364 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:36.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[41853]: debug 2022-01-31T21:33:36.244+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.246027+0000) 2022-01-31T21:33:36.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[46715]: debug 2022-01-31T21:33:36.244+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.246554+0000) 2022-01-31T21:33:36.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[51620]: debug 2022-01-31T21:33:36.244+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.246645+0000) 2022-01-31T21:33:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:36 smithi167 conmon[49112]: debug 2022-01-31T21:33:36.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.578823+0000) 2022-01-31T21:33:36.928 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[51620]: debug 2022-01-31T21:33:36.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.851905+0000) 2022-01-31T21:33:37.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:36 smithi171 conmon[41853]: debug 2022-01-31T21:33:36.972+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:36.973621+0000) 2022-01-31T21:33:37.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:37 smithi171 conmon[46715]: debug 2022-01-31T21:33:37.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:37.017184+0000) 2022-01-31T21:33:37.560 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:37 smithi167 conmon[54076]: debug 2022-01-31T21:33:37.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:37.242435+0000) 2022-01-31T21:33:37.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:37 smithi167 conmon[60316]: debug 2022-01-31T21:33:37.241+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:37.242435+0000) 2022-01-31T21:33:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:37 smithi167 conmon[49112]: debug 2022-01-31T21:33:37.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:37.578965+0000) 2022-01-31T21:33:37.928 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:37 smithi171 conmon[51620]: debug 2022-01-31T21:33:37.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:37.852072+0000) 2022-01-31T21:33:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:37 smithi171 conmon[41853]: debug 2022-01-31T21:33:37.972+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:37.973800+0000) 2022-01-31T21:33:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:38 smithi171 conmon[46715]: debug 2022-01-31T21:33:38.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:38.017326+0000) 2022-01-31T21:33:38.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:38 smithi167 conmon[54076]: debug 2022-01-31T21:33:38.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:38.242572+0000) 2022-01-31T21:33:38.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:38 smithi167 conmon[60316]: debug 2022-01-31T21:33:38.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:38.242572+0000) 2022-01-31T21:33:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:38 smithi167 conmon[49112]: debug 2022-01-31T21:33:38.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:38.579163+0000) 2022-01-31T21:33:38.929 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:38 smithi171 conmon[35325]: debug 2022-01-31T21:33:38.699+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:33:38.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:38 smithi171 conmon[51620]: debug 2022-01-31T21:33:38.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:38.852291+0000) 2022-01-31T21:33:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:38 smithi171 conmon[41853]: debug 2022-01-31T21:33:38.973+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:38.973954+0000) 2022-01-31T21:33:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:39 smithi171 conmon[46715]: debug 2022-01-31T21:33:39.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:39.017506+0000) 2022-01-31T21:33:39.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:39 smithi167 conmon[54076]: debug 2022-01-31T21:33:39.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:39.242767+0000) 2022-01-31T21:33:39.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:39 smithi167 conmon[60316]: debug 2022-01-31T21:33:39.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:39.242767+0000) 2022-01-31T21:33:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:39 smithi167 conmon[49112]: debug 2022-01-31T21:33:39.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:39.579350+0000) 2022-01-31T21:33:39.928 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:39 smithi171 conmon[51620]: debug 2022-01-31T21:33:39.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:39.852497+0000) 2022-01-31T21:33:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:39 smithi171 conmon[41853]: debug 2022-01-31T21:33:39.973+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:39.974121+0000) 2022-01-31T21:33:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:40 smithi171 conmon[46715]: debug 2022-01-31T21:33:40.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:40.017687+0000) 2022-01-31T21:33:40.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:40 smithi167 conmon[54076]: debug 2022-01-31T21:33:40.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:40.242960+0000) 2022-01-31T21:33:40.495 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:40 smithi167 conmon[60316]: debug 2022-01-31T21:33:40.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:40.242960+0000) 2022-01-31T21:33:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:40 smithi167 conmon[49112]: debug 2022-01-31T21:33:40.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:40.579540+0000) 2022-01-31T21:33:40.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:40 smithi171 conmon[51620]: debug 2022-01-31T21:33:40.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:40.852705+0000) 2022-01-31T21:33:41.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[46715]: debug 2022-01-31T21:33:41.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.017867+0000) 2022-01-31T21:33:41.261 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:40 smithi171 conmon[41853]: debug 2022-01-31T21:33:40.973+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:40.974321+0000) 2022-01-31T21:33:41.494 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:41 smithi167 conmon[49112]: debug 2022-01-31T21:33:41.263+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.264405+0000) 2022-01-31T21:33:41.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:41 smithi167 conmon[54076]: debug 2022-01-31T21:33:41.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.243100+0000) 2022-01-31T21:33:41.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:41 smithi167 conmon[54076]: debug 2022-01-31T21:33:41.264+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.264865+0000) 2022-01-31T21:33:41.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:41 smithi167 conmon[60316]: debug 2022-01-31T21:33:41.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.243100+0000) 2022-01-31T21:33:41.496 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:41 smithi167 conmon[60316]: debug 2022-01-31T21:33:41.264+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.264605+0000) 2022-01-31T21:33:41.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[35325]: debug 2022-01-31T21:33:41.278+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139476 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[41853]: debug 2022-01-31T21:33:41.263+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.264533+0000) 2022-01-31T21:33:41.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[46715]: debug 2022-01-31T21:33:41.263+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.263811+0000) 2022-01-31T21:33:41.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[51620]: debug 2022-01-31T21:33:41.263+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.264100+0000) 2022-01-31T21:33:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:41 smithi167 conmon[49112]: debug 2022-01-31T21:33:41.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.579744+0000) 2022-01-31T21:33:41.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[51620]: debug 2022-01-31T21:33:41.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.852919+0000) 2022-01-31T21:33:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:41 smithi171 conmon[41853]: debug 2022-01-31T21:33:41.973+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:41.974486+0000) 2022-01-31T21:33:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:42 smithi171 conmon[46715]: debug 2022-01-31T21:33:42.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:42.018020+0000) 2022-01-31T21:33:42.497 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:42 smithi167 conmon[54076]: debug 2022-01-31T21:33:42.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:42.243299+0000) 2022-01-31T21:33:42.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:42 smithi167 conmon[60316]: debug 2022-01-31T21:33:42.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:42.243299+0000) 2022-01-31T21:33:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:42 smithi167 conmon[49112]: debug 2022-01-31T21:33:42.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:42.579898+0000) 2022-01-31T21:33:42.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:42 smithi171 conmon[51620]: debug 2022-01-31T21:33:42.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:42.853103+0000) 2022-01-31T21:33:43.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:43 smithi167 conmon[54076]: debug 2022-01-31T21:33:43.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:43.243427+0000) 2022-01-31T21:33:43.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:43 smithi167 conmon[60316]: debug 2022-01-31T21:33:43.242+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:43.243427+0000) 2022-01-31T21:33:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:42 smithi171 conmon[41853]: debug 2022-01-31T21:33:42.973+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:42.974664+0000) 2022-01-31T21:33:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:43 smithi171 conmon[46715]: debug 2022-01-31T21:33:43.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:43.018208+0000) 2022-01-31T21:33:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:43 smithi167 conmon[49112]: debug 2022-01-31T21:33:43.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:43.580100+0000) 2022-01-31T21:33:43.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:43 smithi171 conmon[51620]: debug 2022-01-31T21:33:43.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:43.853253+0000) 2022-01-31T21:33:44.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:44 smithi167 conmon[54076]: debug 2022-01-31T21:33:44.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:44.243597+0000) 2022-01-31T21:33:44.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:44 smithi167 conmon[60316]: debug 2022-01-31T21:33:44.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:44.243597+0000) 2022-01-31T21:33:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:44 smithi171 conmon[46715]: debug 2022-01-31T21:33:44.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:44.018382+0000) 2022-01-31T21:33:44.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:43 smithi171 conmon[41853]: debug 2022-01-31T21:33:43.974+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:43.974812+0000) 2022-01-31T21:33:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:44 smithi167 conmon[49112]: debug 2022-01-31T21:33:44.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:44.580318+0000) 2022-01-31T21:33:44.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:44 smithi171 conmon[51620]: debug 2022-01-31T21:33:44.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:44.853419+0000) 2022-01-31T21:33:45.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:45 smithi167 conmon[60316]: debug 2022-01-31T21:33:45.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:45.243884+0000) 2022-01-31T21:33:45.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:45 smithi167 conmon[54076]: debug 2022-01-31T21:33:45.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:45.243829+0000) 2022-01-31T21:33:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:45 smithi171 conmon[46715]: debug 2022-01-31T21:33:45.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:45.018566+0000) 2022-01-31T21:33:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:44 smithi171 conmon[41853]: debug 2022-01-31T21:33:44.974+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:44.974997+0000) 2022-01-31T21:33:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:45 smithi167 conmon[49112]: debug 2022-01-31T21:33:45.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:45.580537+0000) 2022-01-31T21:33:45.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:45 smithi171 conmon[51620]: debug 2022-01-31T21:33:45.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:45.853587+0000) 2022-01-31T21:33:46.279 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:46 smithi167 conmon[54076]: debug 2022-01-31T21:33:46.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.244064+0000) 2022-01-31T21:33:46.279 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:46 smithi167 conmon[60316]: debug 2022-01-31T21:33:46.243+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.244064+0000) 2022-01-31T21:33:46.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[46715]: debug 2022-01-31T21:33:46.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.018747+0000) 2022-01-31T21:33:46.281 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:45 smithi171 conmon[41853]: debug 2022-01-31T21:33:45.974+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:45.975182+0000) 2022-01-31T21:33:46.561 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:46 smithi167 conmon[54076]: debug 2022-01-31T21:33:46.281+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.282254+0000) 2022-01-31T21:33:46.562 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:46 smithi167 conmon[60316]: debug 2022-01-31T21:33:46.280+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.280876+0000) 2022-01-31T21:33:46.562 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:46 smithi167 conmon[49112]: debug 2022-01-31T21:33:46.280+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.280694+0000) 2022-01-31T21:33:46.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[35325]: debug 2022-01-31T21:33:46.295+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139587 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:46.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[41853]: debug 2022-01-31T21:33:46.281+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.282566+0000) 2022-01-31T21:33:46.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[46715]: debug 2022-01-31T21:33:46.281+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.282628+0000) 2022-01-31T21:33:46.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[51620]: debug 2022-01-31T21:33:46.281+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.281793+0000) 2022-01-31T21:33:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:46 smithi167 conmon[49112]: debug 2022-01-31T21:33:46.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.580715+0000) 2022-01-31T21:33:46.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[51620]: debug 2022-01-31T21:33:46.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.853790+0000) 2022-01-31T21:33:47.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:47 smithi167 conmon[54076]: debug 2022-01-31T21:33:47.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:47.244300+0000) 2022-01-31T21:33:47.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:47 smithi167 conmon[60316]: debug 2022-01-31T21:33:47.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:47.244300+0000) 2022-01-31T21:33:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:46 smithi171 conmon[41853]: debug 2022-01-31T21:33:46.974+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:46.975367+0000) 2022-01-31T21:33:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:47 smithi171 conmon[46715]: debug 2022-01-31T21:33:47.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:47.018963+0000) 2022-01-31T21:33:47.923 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:47 smithi167 conmon[49112]: debug 2022-01-31T21:33:47.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:47.580866+0000) 2022-01-31T21:33:47.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:47 smithi171 conmon[51620]: debug 2022-01-31T21:33:47.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:47.853960+0000) 2022-01-31T21:33:48.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:48 smithi167 conmon[60316]: debug 2022-01-31T21:33:48.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:48.244491+0000) 2022-01-31T21:33:48.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:48 smithi167 conmon[54076]: debug 2022-01-31T21:33:48.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:48.244491+0000) 2022-01-31T21:33:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:48 smithi171 conmon[46715]: debug 2022-01-31T21:33:48.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:48.019085+0000) 2022-01-31T21:33:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:47 smithi171 conmon[41853]: debug 2022-01-31T21:33:47.974+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:47.975548+0000) 2022-01-31T21:33:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:48 smithi167 conmon[49112]: debug 2022-01-31T21:33:48.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:48.581078+0000) 2022-01-31T21:33:48.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:48 smithi171 conmon[51620]: debug 2022-01-31T21:33:48.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:48.854160+0000) 2022-01-31T21:33:49.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:49 smithi167 conmon[54076]: debug 2022-01-31T21:33:49.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:49.244692+0000) 2022-01-31T21:33:49.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:49 smithi167 conmon[60316]: debug 2022-01-31T21:33:49.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:49.244692+0000) 2022-01-31T21:33:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:49 smithi171 conmon[46715]: debug 2022-01-31T21:33:49.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:49.019236+0000) 2022-01-31T21:33:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:48 smithi171 conmon[41853]: debug 2022-01-31T21:33:48.975+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:48.975739+0000) 2022-01-31T21:33:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:49 smithi167 conmon[49112]: debug 2022-01-31T21:33:49.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:49.581267+0000) 2022-01-31T21:33:49.930 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:49 smithi171 conmon[51620]: debug 2022-01-31T21:33:49.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:49.854375+0000) 2022-01-31T21:33:50.299 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:50 smithi167 conmon[54076]: debug 2022-01-31T21:33:50.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:50.244905+0000) 2022-01-31T21:33:50.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:50 smithi167 conmon[60316]: debug 2022-01-31T21:33:50.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:50.244906+0000) 2022-01-31T21:33:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:49 smithi171 conmon[41853]: debug 2022-01-31T21:33:49.975+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:49.975950+0000) 2022-01-31T21:33:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:50 smithi171 conmon[46715]: debug 2022-01-31T21:33:50.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:50.019413+0000) 2022-01-31T21:33:50.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:50 smithi167 conmon[49112]: debug 2022-01-31T21:33:50.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:50.581428+0000) 2022-01-31T21:33:50.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:50 smithi171 conmon[51620]: debug 2022-01-31T21:33:50.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:50.854562+0000) 2022-01-31T21:33:51.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[46715]: debug 2022-01-31T21:33:51.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.019642+0000) 2022-01-31T21:33:51.296 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:50 smithi171 conmon[41853]: debug 2022-01-31T21:33:50.975+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:50.976137+0000) 2022-01-31T21:33:51.297 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[41853]: debug 2022-01-31T21:33:51.297+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.298462+0000) 2022-01-31T21:33:51.298 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:51 smithi167 conmon[60316]: debug 2022-01-31T21:33:51.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.245111+0000) 2022-01-31T21:33:51.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:51 smithi167 conmon[54076]: debug 2022-01-31T21:33:51.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.245111+0000) 2022-01-31T21:33:51.562 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:51 smithi167 conmon[54076]: debug 2022-01-31T21:33:51.298+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.298750+0000) 2022-01-31T21:33:51.563 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:51 smithi167 conmon[60316]: debug 2022-01-31T21:33:51.297+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.298435+0000) 2022-01-31T21:33:51.563 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:51 smithi167 conmon[49112]: debug 2022-01-31T21:33:51.298+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.298649+0000) 2022-01-31T21:33:51.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[35325]: debug 2022-01-31T21:33:51.312+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139697 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:51.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[46715]: debug 2022-01-31T21:33:51.299+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.299742+0000) 2022-01-31T21:33:51.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[51620]: debug 2022-01-31T21:33:51.297+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.298198+0000) 2022-01-31T21:33:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:51 smithi167 conmon[49112]: debug 2022-01-31T21:33:51.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.581601+0000) 2022-01-31T21:33:51.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[51620]: debug 2022-01-31T21:33:51.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.854691+0000) 2022-01-31T21:33:52.299 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:52 smithi167 conmon[60316]: debug 2022-01-31T21:33:52.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:52.245247+0000) 2022-01-31T21:33:52.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:52 smithi167 conmon[54076]: debug 2022-01-31T21:33:52.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:52.245247+0000) 2022-01-31T21:33:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:51 smithi171 conmon[41853]: debug 2022-01-31T21:33:51.975+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:51.976323+0000) 2022-01-31T21:33:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:52 smithi171 conmon[46715]: debug 2022-01-31T21:33:52.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:52.019831+0000) 2022-01-31T21:33:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:52 smithi167 conmon[49112]: debug 2022-01-31T21:33:52.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:52.581709+0000) 2022-01-31T21:33:52.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:52 smithi171 conmon[51620]: debug 2022-01-31T21:33:52.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:52.854830+0000) 2022-01-31T21:33:53.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:53 smithi167 conmon[54076]: debug 2022-01-31T21:33:53.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:53.245408+0000) 2022-01-31T21:33:53.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:53 smithi167 conmon[60316]: debug 2022-01-31T21:33:53.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:53.245408+0000) 2022-01-31T21:33:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:52 smithi171 conmon[41853]: debug 2022-01-31T21:33:52.975+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:52.976488+0000) 2022-01-31T21:33:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:53 smithi171 conmon[46715]: debug 2022-01-31T21:33:53.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:53.020005+0000) 2022-01-31T21:33:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:53 smithi167 conmon[49112]: debug 2022-01-31T21:33:53.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:53.581870+0000) 2022-01-31T21:33:53.931 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:53 smithi171 conmon[35325]: debug 2022-01-31T21:33:53.700+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:33:53.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:53 smithi171 conmon[51620]: debug 2022-01-31T21:33:53.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:53.854967+0000) 2022-01-31T21:33:54.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:54 smithi167 conmon[54076]: debug 2022-01-31T21:33:54.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:54.245605+0000) 2022-01-31T21:33:54.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:54 smithi167 conmon[60316]: debug 2022-01-31T21:33:54.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:54.245605+0000) 2022-01-31T21:33:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:53 smithi171 conmon[41853]: debug 2022-01-31T21:33:53.976+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:53.976649+0000) 2022-01-31T21:33:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:54 smithi171 conmon[46715]: debug 2022-01-31T21:33:54.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:54.020206+0000) 2022-01-31T21:33:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:54 smithi167 conmon[49112]: debug 2022-01-31T21:33:54.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:54.582063+0000) 2022-01-31T21:33:54.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:54 smithi171 conmon[51620]: debug 2022-01-31T21:33:54.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:54.855166+0000) 2022-01-31T21:33:55.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:55 smithi167 conmon[54076]: debug 2022-01-31T21:33:55.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:55.245819+0000) 2022-01-31T21:33:55.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:55 smithi167 conmon[60316]: debug 2022-01-31T21:33:55.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:55.245817+0000) 2022-01-31T21:33:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:54 smithi171 conmon[41853]: debug 2022-01-31T21:33:54.976+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:54.976854+0000) 2022-01-31T21:33:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:55 smithi171 conmon[46715]: debug 2022-01-31T21:33:55.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:55.020396+0000) 2022-01-31T21:33:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:55 smithi167 conmon[49112]: debug 2022-01-31T21:33:55.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:55.582237+0000) 2022-01-31T21:33:55.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:55 smithi171 conmon[51620]: debug 2022-01-31T21:33:55.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:55.855370+0000) 2022-01-31T21:33:56.300 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:56 smithi167 conmon[60316]: debug 2022-01-31T21:33:56.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.246023+0000) 2022-01-31T21:33:56.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:56 smithi167 conmon[54076]: debug 2022-01-31T21:33:56.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.246022+0000) 2022-01-31T21:33:56.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[46715]: debug 2022-01-31T21:33:56.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.020574+0000) 2022-01-31T21:33:56.313 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:55 smithi171 conmon[41853]: debug 2022-01-31T21:33:55.976+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:55.976984+0000) 2022-01-31T21:33:56.563 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:56 smithi167 conmon[49112]: debug 2022-01-31T21:33:56.313+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.315012+0000) 2022-01-31T21:33:56.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:56 smithi167 conmon[54076]: debug 2022-01-31T21:33:56.313+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.315329+0000) 2022-01-31T21:33:56.564 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:56 smithi167 conmon[60316]: debug 2022-01-31T21:33:56.313+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.314636+0000) 2022-01-31T21:33:56.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[35325]: debug 2022-01-31T21:33:56.330+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:33:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[41853]: debug 2022-01-31T21:33:56.316+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.316801+0000) 2022-01-31T21:33:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[46715]: debug 2022-01-31T21:33:56.316+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.317099+0000) 2022-01-31T21:33:56.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[51620]: debug 2022-01-31T21:33:56.314+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.315526+0000) 2022-01-31T21:33:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:56 smithi167 conmon[49112]: debug 2022-01-31T21:33:56.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.582429+0000) 2022-01-31T21:33:56.931 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[51620]: debug 2022-01-31T21:33:56.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.855594+0000) 2022-01-31T21:33:57.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:57 smithi167 conmon[60316]: debug 2022-01-31T21:33:57.244+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:57.246260+0000) 2022-01-31T21:33:57.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:57 smithi167 conmon[54076]: debug 2022-01-31T21:33:57.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:57.246260+0000) 2022-01-31T21:33:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:56 smithi171 conmon[41853]: debug 2022-01-31T21:33:56.976+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:56.977137+0000) 2022-01-31T21:33:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:57 smithi171 conmon[46715]: debug 2022-01-31T21:33:57.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:57.020770+0000) 2022-01-31T21:33:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:57 smithi167 conmon[49112]: debug 2022-01-31T21:33:57.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:57.582595+0000) 2022-01-31T21:33:57.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:57 smithi171 conmon[51620]: debug 2022-01-31T21:33:57.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:57.855792+0000) 2022-01-31T21:33:58.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:58 smithi167 conmon[54076]: debug 2022-01-31T21:33:58.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:58.246433+0000) 2022-01-31T21:33:58.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:58 smithi167 conmon[60316]: debug 2022-01-31T21:33:58.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:58.246433+0000) 2022-01-31T21:33:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:57 smithi171 conmon[41853]: debug 2022-01-31T21:33:57.976+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:57.977284+0000) 2022-01-31T21:33:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:58 smithi171 conmon[46715]: debug 2022-01-31T21:33:58.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:58.020904+0000) 2022-01-31T21:33:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:58 smithi167 conmon[49112]: debug 2022-01-31T21:33:58.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:58.582798+0000) 2022-01-31T21:33:58.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:58 smithi171 conmon[51620]: debug 2022-01-31T21:33:58.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:58.855992+0000) 2022-01-31T21:33:59.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:33:59 smithi167 conmon[60316]: debug 2022-01-31T21:33:59.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:59.246638+0000) 2022-01-31T21:33:59.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:33:59 smithi167 conmon[54076]: debug 2022-01-31T21:33:59.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:59.246638+0000) 2022-01-31T21:33:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:58 smithi171 conmon[41853]: debug 2022-01-31T21:33:58.977+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:58.977447+0000) 2022-01-31T21:33:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:33:59 smithi171 conmon[46715]: debug 2022-01-31T21:33:59.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:59.021118+0000) 2022-01-31T21:33:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:33:59 smithi167 conmon[49112]: debug 2022-01-31T21:33:59.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:59.583014+0000) 2022-01-31T21:33:59.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:33:59 smithi171 conmon[51620]: debug 2022-01-31T21:33:59.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:59.856177+0000) 2022-01-31T21:34:00.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:00 smithi167 conmon[60316]: debug 2022-01-31T21:34:00.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:00.246822+0000) 2022-01-31T21:34:00.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:00 smithi167 conmon[54076]: debug 2022-01-31T21:34:00.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:00.246822+0000) 2022-01-31T21:34:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:00 smithi171 conmon[46715]: debug 2022-01-31T21:34:00.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:00.021313+0000) 2022-01-31T21:34:00.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:33:59 smithi171 conmon[41853]: debug 2022-01-31T21:33:59.977+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:33:59.977650+0000) 2022-01-31T21:34:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:00 smithi167 conmon[49112]: debug 2022-01-31T21:34:00.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:00.583213+0000) 2022-01-31T21:34:00.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:00 smithi171 conmon[51620]: debug 2022-01-31T21:34:00.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:00.856361+0000) 2022-01-31T21:34:01.301 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:01 smithi167 conmon[60316]: debug 2022-01-31T21:34:01.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.247033+0000) 2022-01-31T21:34:01.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:01 smithi167 conmon[54076]: debug 2022-01-31T21:34:01.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.247033+0000) 2022-01-31T21:34:01.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[46715]: debug 2022-01-31T21:34:01.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.021494+0000) 2022-01-31T21:34:01.331 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:00 smithi171 conmon[41853]: debug 2022-01-31T21:34:00.977+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:00.977833+0000) 2022-01-31T21:34:01.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:01 smithi167 conmon[54076]: debug 2022-01-31T21:34:01.332+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.333979+0000) 2022-01-31T21:34:01.564 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:01 smithi167 conmon[49112]: debug 2022-01-31T21:34:01.331+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.333305+0000) 2022-01-31T21:34:01.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:01 smithi167 conmon[60316]: debug 2022-01-31T21:34:01.331+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.333306+0000) 2022-01-31T21:34:01.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[35325]: debug 2022-01-31T21:34:01.347+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 139919 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[41853]: debug 2022-01-31T21:34:01.333+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.334138+0000) 2022-01-31T21:34:01.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[46715]: debug 2022-01-31T21:34:01.332+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.332920+0000) 2022-01-31T21:34:01.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[51620]: debug 2022-01-31T21:34:01.332+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.333707+0000) 2022-01-31T21:34:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:01 smithi167 conmon[49112]: debug 2022-01-31T21:34:01.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.583391+0000) 2022-01-31T21:34:01.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[51620]: debug 2022-01-31T21:34:01.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.856560+0000) 2022-01-31T21:34:02.301 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:02 smithi167 conmon[54076]: debug 2022-01-31T21:34:02.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:02.247241+0000) 2022-01-31T21:34:02.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:02 smithi167 conmon[60316]: debug 2022-01-31T21:34:02.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:02.247241+0000) 2022-01-31T21:34:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:01 smithi171 conmon[41853]: debug 2022-01-31T21:34:01.977+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:01.978015+0000) 2022-01-31T21:34:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:02 smithi171 conmon[46715]: debug 2022-01-31T21:34:02.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:02.021680+0000) 2022-01-31T21:34:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:02 smithi167 conmon[49112]: debug 2022-01-31T21:34:02.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:02.583556+0000) 2022-01-31T21:34:02.932 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:02 smithi171 conmon[51620]: debug 2022-01-31T21:34:02.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:02.856723+0000) 2022-01-31T21:34:03.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:03 smithi167 conmon[54076]: debug 2022-01-31T21:34:03.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:03.247425+0000) 2022-01-31T21:34:03.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:03 smithi167 conmon[60316]: debug 2022-01-31T21:34:03.245+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:03.247425+0000) 2022-01-31T21:34:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:02 smithi171 conmon[41853]: debug 2022-01-31T21:34:02.977+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:02.978191+0000) 2022-01-31T21:34:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:03 smithi171 conmon[46715]: debug 2022-01-31T21:34:03.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:03.021862+0000) 2022-01-31T21:34:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:03 smithi167 conmon[49112]: debug 2022-01-31T21:34:03.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:03.583777+0000) 2022-01-31T21:34:03.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:03 smithi171 conmon[51620]: debug 2022-01-31T21:34:03.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:03.856932+0000) 2022-01-31T21:34:04.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:04 smithi167 conmon[54076]: debug 2022-01-31T21:34:04.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:04.247663+0000) 2022-01-31T21:34:04.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:04 smithi167 conmon[60316]: debug 2022-01-31T21:34:04.246+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:04.247663+0000) 2022-01-31T21:34:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:03 smithi171 conmon[41853]: debug 2022-01-31T21:34:03.977+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:03.978318+0000) 2022-01-31T21:34:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:04 smithi171 conmon[46715]: debug 2022-01-31T21:34:04.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:04.022042+0000) 2022-01-31T21:34:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:04 smithi167 conmon[49112]: debug 2022-01-31T21:34:04.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:04.583966+0000) 2022-01-31T21:34:04.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:04 smithi171 conmon[51620]: debug 2022-01-31T21:34:04.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:04.857129+0000) 2022-01-31T21:34:05.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:05 smithi167 conmon[54076]: debug 2022-01-31T21:34:05.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:05.247891+0000) 2022-01-31T21:34:05.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:05 smithi167 conmon[60316]: debug 2022-01-31T21:34:05.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:05.247891+0000) 2022-01-31T21:34:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:04 smithi171 conmon[41853]: debug 2022-01-31T21:34:04.978+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:04.978474+0000) 2022-01-31T21:34:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:05 smithi171 conmon[46715]: debug 2022-01-31T21:34:05.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:05.022251+0000) 2022-01-31T21:34:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:05 smithi167 conmon[49112]: debug 2022-01-31T21:34:05.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:05.584175+0000) 2022-01-31T21:34:05.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:05 smithi171 conmon[51620]: debug 2022-01-31T21:34:05.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:05.857305+0000) 2022-01-31T21:34:06.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[54076]: debug 2022-01-31T21:34:06.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.248076+0000) 2022-01-31T21:34:06.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[60316]: debug 2022-01-31T21:34:06.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.248078+0000) 2022-01-31T21:34:06.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[46715]: debug 2022-01-31T21:34:06.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.022434+0000) 2022-01-31T21:34:06.349 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:05 smithi171 conmon[41853]: debug 2022-01-31T21:34:05.978+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:05.978656+0000) 2022-01-31T21:34:06.565 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[49112]: debug 2022-01-31T21:34:06.348+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.350459+0000) 2022-01-31T21:34:06.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[54076]: debug 2022-01-31T21:34:06.348+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.350389+0000) 2022-01-31T21:34:06.566 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[60316]: debug 2022-01-31T21:34:06.567 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[60316]: 2022-01-31T21:34:06.355+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.357045+0000) 2022-01-31T21:34:06.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[35325]: debug 2022-01-31T21:34:06.370+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140031 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[41853]: debug 2022-01-31T21:34:06.349+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.350225+0000) 2022-01-31T21:34:06.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[46715]: debug 2022-01-31T21:34:06.350+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.350774+0000) 2022-01-31T21:34:06.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[51620]: debug 2022-01-31T21:34:06.350+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.351051+0000) 2022-01-31T21:34:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:06 smithi167 conmon[49112]: debug 2022-01-31T21:34:06.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.584385+0000) 2022-01-31T21:34:06.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[51620]: debug 2022-01-31T21:34:06.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.857465+0000) 2022-01-31T21:34:07.302 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:07 smithi167 conmon[60316]: debug 2022-01-31T21:34:07.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:07.248290+0000) 2022-01-31T21:34:07.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:07 smithi167 conmon[54076]: debug 2022-01-31T21:34:07.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:07.248290+0000) 2022-01-31T21:34:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:07 smithi171 conmon[46715]: debug 2022-01-31T21:34:07.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:07.022608+0000) 2022-01-31T21:34:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:06 smithi171 conmon[41853]: debug 2022-01-31T21:34:06.978+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:06.978814+0000) 2022-01-31T21:34:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:07 smithi167 conmon[49112]: debug 2022-01-31T21:34:07.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:07.584585+0000) 2022-01-31T21:34:07.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:07 smithi171 conmon[51620]: debug 2022-01-31T21:34:07.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:07.857618+0000) 2022-01-31T21:34:08.302 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:08 smithi167 conmon[54076]: debug 2022-01-31T21:34:08.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:08.248459+0000) 2022-01-31T21:34:08.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:08 smithi167 conmon[60316]: debug 2022-01-31T21:34:08.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:08.248459+0000) 2022-01-31T21:34:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:07 smithi171 conmon[41853]: debug 2022-01-31T21:34:07.978+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:07.978966+0000) 2022-01-31T21:34:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:08 smithi171 conmon[46715]: debug 2022-01-31T21:34:08.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:08.022778+0000) 2022-01-31T21:34:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:08 smithi167 conmon[49112]: debug 2022-01-31T21:34:08.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:08.584745+0000) 2022-01-31T21:34:08.933 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:08 smithi171 conmon[35325]: debug 2022-01-31T21:34:08.701+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:34:08.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:08 smithi171 conmon[51620]: debug 2022-01-31T21:34:08.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:08.857796+0000) 2022-01-31T21:34:09.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:09 smithi167 conmon[54076]: debug 2022-01-31T21:34:09.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:09.248661+0000) 2022-01-31T21:34:09.303 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:09 smithi167 conmon[60316]: debug 2022-01-31T21:34:09.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:09.248661+0000) 2022-01-31T21:34:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:08 smithi171 conmon[41853]: debug 2022-01-31T21:34:08.978+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:08.979150+0000) 2022-01-31T21:34:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:09 smithi171 conmon[46715]: debug 2022-01-31T21:34:09.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:09.022985+0000) 2022-01-31T21:34:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:09 smithi167 conmon[49112]: debug 2022-01-31T21:34:09.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:09.584928+0000) 2022-01-31T21:34:09.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:09 smithi171 conmon[51620]: debug 2022-01-31T21:34:09.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:09.857978+0000) 2022-01-31T21:34:10.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:10 smithi167 conmon[54076]: debug 2022-01-31T21:34:10.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:10.248850+0000) 2022-01-31T21:34:10.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:10 smithi167 conmon[60316]: debug 2022-01-31T21:34:10.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:10.248850+0000) 2022-01-31T21:34:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:09 smithi171 conmon[41853]: debug 2022-01-31T21:34:09.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:09.979347+0000) 2022-01-31T21:34:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:10 smithi171 conmon[46715]: debug 2022-01-31T21:34:10.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:10.023153+0000) 2022-01-31T21:34:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:10 smithi167 conmon[49112]: debug 2022-01-31T21:34:10.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:10.585133+0000) 2022-01-31T21:34:10.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:10 smithi171 conmon[51620]: debug 2022-01-31T21:34:10.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:10.858166+0000) 2022-01-31T21:34:11.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:11 smithi167 conmon[54076]: debug 2022-01-31T21:34:11.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.249062+0000) 2022-01-31T21:34:11.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:11 smithi167 conmon[60316]: debug 2022-01-31T21:34:11.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.249062+0000) 2022-01-31T21:34:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:10 smithi171 conmon[41853]: debug 2022-01-31T21:34:10.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:10.979528+0000) 2022-01-31T21:34:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[46715]: debug 2022-01-31T21:34:11.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.023340+0000) 2022-01-31T21:34:11.565 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:11 smithi167 conmon[49112]: debug 2022-01-31T21:34:11.372+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.374147+0000) 2022-01-31T21:34:11.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:11 smithi167 conmon[54076]: debug 2022-01-31T21:34:11.371+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.372852+0000) 2022-01-31T21:34:11.567 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:11 smithi167 conmon[60316]: debug 2022-01-31T21:34:11.372+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.373809+0000) 2022-01-31T21:34:11.669 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[35325]: debug 2022-01-31T21:34:11.387+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140141 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:11.670 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[41853]: debug 2022-01-31T21:34:11.372+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.373011+0000) 2022-01-31T21:34:11.670 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[46715]: debug 2022-01-31T21:34:11.372+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.373222+0000) 2022-01-31T21:34:11.671 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[51620]: debug 2022-01-31T21:34:11.373+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.374439+0000) 2022-01-31T21:34:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:11 smithi167 conmon[49112]: debug 2022-01-31T21:34:11.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.585335+0000) 2022-01-31T21:34:11.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[51620]: debug 2022-01-31T21:34:11.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.858352+0000) 2022-01-31T21:34:12.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:12 smithi167 conmon[54076]: debug 2022-01-31T21:34:12.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:12.249296+0000) 2022-01-31T21:34:12.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:12 smithi167 conmon[60316]: debug 2022-01-31T21:34:12.247+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:12.249296+0000) 2022-01-31T21:34:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:11 smithi171 conmon[41853]: debug 2022-01-31T21:34:11.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:11.979720+0000) 2022-01-31T21:34:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:12 smithi171 conmon[46715]: debug 2022-01-31T21:34:12.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:12.023544+0000) 2022-01-31T21:34:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:12 smithi167 conmon[49112]: debug 2022-01-31T21:34:12.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:12.585481+0000) 2022-01-31T21:34:12.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:12 smithi171 conmon[51620]: debug 2022-01-31T21:34:12.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:12.858508+0000) 2022-01-31T21:34:13.303 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:13 smithi167 conmon[54076]: debug 2022-01-31T21:34:13.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:13.249465+0000) 2022-01-31T21:34:13.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:13 smithi167 conmon[60316]: debug 2022-01-31T21:34:13.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:13.249465+0000) 2022-01-31T21:34:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:12 smithi171 conmon[41853]: debug 2022-01-31T21:34:12.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:12.979859+0000) 2022-01-31T21:34:13.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:13 smithi171 conmon[46715]: debug 2022-01-31T21:34:13.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:13.023718+0000) 2022-01-31T21:34:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:13 smithi167 conmon[49112]: debug 2022-01-31T21:34:13.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:13.585710+0000) 2022-01-31T21:34:13.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:13 smithi171 conmon[51620]: debug 2022-01-31T21:34:13.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:13.858679+0000) 2022-01-31T21:34:14.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:14 smithi167 conmon[54076]: debug 2022-01-31T21:34:14.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:14.249738+0000) 2022-01-31T21:34:14.304 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:14 smithi167 conmon[60316]: debug 2022-01-31T21:34:14.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:14.249738+0000) 2022-01-31T21:34:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:13 smithi171 conmon[41853]: debug 2022-01-31T21:34:13.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:13.979972+0000) 2022-01-31T21:34:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:14 smithi171 conmon[46715]: debug 2022-01-31T21:34:14.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:14.023913+0000) 2022-01-31T21:34:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:14 smithi167 conmon[49112]: debug 2022-01-31T21:34:14.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:14.585914+0000) 2022-01-31T21:34:14.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:14 smithi171 conmon[51620]: debug 2022-01-31T21:34:14.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:14.858892+0000) 2022-01-31T21:34:15.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:15 smithi167 conmon[54076]: debug 2022-01-31T21:34:15.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:15.249902+0000) 2022-01-31T21:34:15.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:15 smithi167 conmon[60316]: debug 2022-01-31T21:34:15.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:15.249902+0000) 2022-01-31T21:34:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:14 smithi171 conmon[41853]: debug 2022-01-31T21:34:14.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:14.980156+0000) 2022-01-31T21:34:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:15 smithi171 conmon[46715]: debug 2022-01-31T21:34:15.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:15.024074+0000) 2022-01-31T21:34:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:15 smithi167 conmon[49112]: debug 2022-01-31T21:34:15.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:15.586110+0000) 2022-01-31T21:34:15.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:15 smithi171 conmon[51620]: debug 2022-01-31T21:34:15.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:15.859070+0000) 2022-01-31T21:34:16.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:16 smithi167 conmon[54076]: debug 2022-01-31T21:34:16.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.250076+0000) 2022-01-31T21:34:16.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:16 smithi167 conmon[60316]: debug 2022-01-31T21:34:16.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.250077+0000) 2022-01-31T21:34:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:15 smithi171 conmon[41853]: debug 2022-01-31T21:34:15.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:15.980356+0000) 2022-01-31T21:34:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[46715]: debug 2022-01-31T21:34:16.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.024256+0000) 2022-01-31T21:34:16.558 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:16 smithi167 conmon[49112]: debug 2022-01-31T21:34:16.389+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.390662+0000) 2022-01-31T21:34:16.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:16 smithi167 conmon[54076]: debug 2022-01-31T21:34:16.389+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.391307+0000) 2022-01-31T21:34:16.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:16 smithi167 conmon[60316]: debug 2022-01-31T21:34:16.388+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.390335+0000) 2022-01-31T21:34:16.670 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[35325]: debug 2022-01-31T21:34:16.404+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140249 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:16.671 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[41853]: debug 2022-01-31T21:34:16.389+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.391395+0000) 2022-01-31T21:34:16.672 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[46715]: debug 2022-01-31T21:34:16.388+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.390533+0000) 2022-01-31T21:34:16.672 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[51620]: debug 2022-01-31T21:34:16.389+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.390980+0000) 2022-01-31T21:34:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:16 smithi167 conmon[49112]: debug 2022-01-31T21:34:16.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.586310+0000) 2022-01-31T21:34:16.935 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[51620]: debug 2022-01-31T21:34:16.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.859258+0000) 2022-01-31T21:34:17.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:17 smithi167 conmon[54076]: debug 2022-01-31T21:34:17.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:17.250235+0000) 2022-01-31T21:34:17.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:17 smithi167 conmon[60316]: debug 2022-01-31T21:34:17.248+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:17.250235+0000) 2022-01-31T21:34:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:16 smithi171 conmon[41853]: debug 2022-01-31T21:34:16.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:16.980511+0000) 2022-01-31T21:34:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:17 smithi171 conmon[46715]: debug 2022-01-31T21:34:17.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:17.024411+0000) 2022-01-31T21:34:17.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:17 smithi167 conmon[49112]: debug 2022-01-31T21:34:17.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:17.586440+0000) 2022-01-31T21:34:17.935 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:17 smithi171 conmon[51620]: debug 2022-01-31T21:34:17.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:17.859438+0000) 2022-01-31T21:34:18.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:18 smithi167 conmon[54076]: debug 2022-01-31T21:34:18.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:18.250343+0000) 2022-01-31T21:34:18.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:18 smithi167 conmon[60316]: debug 2022-01-31T21:34:18.249+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:18.250344+0000) 2022-01-31T21:34:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:17 smithi171 conmon[41853]: debug 2022-01-31T21:34:17.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:17.980656+0000) 2022-01-31T21:34:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:18 smithi171 conmon[46715]: debug 2022-01-31T21:34:18.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:18.024565+0000) 2022-01-31T21:34:18.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:18 smithi167 conmon[49112]: debug 2022-01-31T21:34:18.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:18.586582+0000) 2022-01-31T21:34:18.935 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:18 smithi171 conmon[51620]: debug 2022-01-31T21:34:18.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:18.859648+0000) 2022-01-31T21:34:19.304 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:19 smithi167 conmon[54076]: debug 2022-01-31T21:34:19.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:19.250529+0000) 2022-01-31T21:34:19.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:19 smithi167 conmon[60316]: debug 2022-01-31T21:34:19.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:19.250527+0000) 2022-01-31T21:34:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:18 smithi171 conmon[41853]: debug 2022-01-31T21:34:18.979+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:18.980814+0000) 2022-01-31T21:34:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:19 smithi171 conmon[46715]: debug 2022-01-31T21:34:19.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:19.024746+0000) 2022-01-31T21:34:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:19 smithi167 conmon[49112]: debug 2022-01-31T21:34:19.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:19.586735+0000) 2022-01-31T21:34:19.935 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:19 smithi171 conmon[51620]: debug 2022-01-31T21:34:19.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:19.859866+0000) 2022-01-31T21:34:20.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:20 smithi167 conmon[54076]: debug 2022-01-31T21:34:20.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:20.250692+0000) 2022-01-31T21:34:20.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:20 smithi167 conmon[60316]: debug 2022-01-31T21:34:20.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:20.250692+0000) 2022-01-31T21:34:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:19 smithi171 conmon[41853]: debug 2022-01-31T21:34:19.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:19.981005+0000) 2022-01-31T21:34:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:20 smithi171 conmon[46715]: debug 2022-01-31T21:34:20.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:20.024919+0000) 2022-01-31T21:34:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:20 smithi167 conmon[49112]: debug 2022-01-31T21:34:20.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:20.586923+0000) 2022-01-31T21:34:20.936 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:20 smithi171 conmon[51620]: debug 2022-01-31T21:34:20.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:20.860075+0000) 2022-01-31T21:34:21.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:21 smithi167 conmon[54076]: debug 2022-01-31T21:34:21.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.250891+0000) 2022-01-31T21:34:21.305 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:21 smithi167 conmon[60316]: debug 2022-01-31T21:34:21.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.250890+0000) 2022-01-31T21:34:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:20 smithi171 conmon[41853]: debug 2022-01-31T21:34:20.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:20.981158+0000) 2022-01-31T21:34:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[46715]: debug 2022-01-31T21:34:21.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.025095+0000) 2022-01-31T21:34:21.567 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:21 smithi167 conmon[54076]: debug 2022-01-31T21:34:21.408+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.409412+0000) 2022-01-31T21:34:21.568 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:21 smithi167 conmon[60316]: debug 2022-01-31T21:34:21.407+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.407710+0000) 2022-01-31T21:34:21.568 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:21 smithi167 conmon[49112]: debug 2022-01-31T21:34:21.408+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.408940+0000) 2022-01-31T21:34:21.672 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[35325]: debug 2022-01-31T21:34:21.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140359 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:21.672 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[41853]: debug 2022-01-31T21:34:21.407+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.408951+0000) 2022-01-31T21:34:21.673 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[46715]: debug 2022-01-31T21:34:21.407+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.408813+0000) 2022-01-31T21:34:21.673 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[51620]: debug 2022-01-31T21:34:21.407+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.409414+0000) 2022-01-31T21:34:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:21 smithi167 conmon[49112]: debug 2022-01-31T21:34:21.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.587128+0000) 2022-01-31T21:34:21.936 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[51620]: debug 2022-01-31T21:34:21.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.860249+0000) 2022-01-31T21:34:22.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:22 smithi167 conmon[54076]: debug 2022-01-31T21:34:22.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:22.251125+0000) 2022-01-31T21:34:22.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:22 smithi167 conmon[60316]: debug 2022-01-31T21:34:22.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:22.251125+0000) 2022-01-31T21:34:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:21 smithi171 conmon[41853]: debug 2022-01-31T21:34:21.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:21.981365+0000) 2022-01-31T21:34:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:22 smithi171 conmon[46715]: debug 2022-01-31T21:34:22.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:22.025269+0000) 2022-01-31T21:34:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:22 smithi167 conmon[49112]: debug 2022-01-31T21:34:22.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:22.587300+0000) 2022-01-31T21:34:22.936 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:22 smithi171 conmon[51620]: debug 2022-01-31T21:34:22.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:22.860436+0000) 2022-01-31T21:34:23.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:23 smithi167 conmon[54076]: debug 2022-01-31T21:34:23.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:23.251254+0000) 2022-01-31T21:34:23.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:23 smithi167 conmon[60316]: debug 2022-01-31T21:34:23.250+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:23.251254+0000) 2022-01-31T21:34:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:22 smithi171 conmon[41853]: debug 2022-01-31T21:34:22.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:22.981546+0000) 2022-01-31T21:34:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:23 smithi171 conmon[46715]: debug 2022-01-31T21:34:23.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:23.025411+0000) 2022-01-31T21:34:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:23 smithi167 conmon[49112]: debug 2022-01-31T21:34:23.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:23.587484+0000) 2022-01-31T21:34:23.936 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:23 smithi171 conmon[35325]: debug 2022-01-31T21:34:23.701+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:34:23.936 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:23 smithi171 conmon[51620]: debug 2022-01-31T21:34:23.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:23.860604+0000) 2022-01-31T21:34:24.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:24 smithi167 conmon[54076]: debug 2022-01-31T21:34:24.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:24.251423+0000) 2022-01-31T21:34:24.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:24 smithi167 conmon[60316]: debug 2022-01-31T21:34:24.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:24.251423+0000) 2022-01-31T21:34:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:23 smithi171 conmon[41853]: debug 2022-01-31T21:34:23.980+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:23.981768+0000) 2022-01-31T21:34:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:24 smithi171 conmon[46715]: debug 2022-01-31T21:34:24.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:24.025567+0000) 2022-01-31T21:34:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:24 smithi167 conmon[49112]: debug 2022-01-31T21:34:24.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:24.587674+0000) 2022-01-31T21:34:24.936 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:24 smithi171 conmon[51620]: debug 2022-01-31T21:34:24.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:24.860778+0000) 2022-01-31T21:34:25.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:25 smithi167 conmon[54076]: debug 2022-01-31T21:34:25.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:25.251616+0000) 2022-01-31T21:34:25.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:25 smithi167 conmon[60316]: debug 2022-01-31T21:34:25.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:25.251616+0000) 2022-01-31T21:34:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:24 smithi171 conmon[41853]: debug 2022-01-31T21:34:24.981+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:24.981950+0000) 2022-01-31T21:34:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:25 smithi171 conmon[46715]: debug 2022-01-31T21:34:25.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:25.025723+0000) 2022-01-31T21:34:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:25 smithi167 conmon[49112]: debug 2022-01-31T21:34:25.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:25.587923+0000) 2022-01-31T21:34:25.936 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:25 smithi171 conmon[51620]: debug 2022-01-31T21:34:25.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:25.860967+0000) 2022-01-31T21:34:26.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:26 smithi167 conmon[54076]: debug 2022-01-31T21:34:26.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.251767+0000) 2022-01-31T21:34:26.306 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:26 smithi167 conmon[60316]: debug 2022-01-31T21:34:26.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.251767+0000) 2022-01-31T21:34:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[46715]: debug 2022-01-31T21:34:26.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.025892+0000) 2022-01-31T21:34:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:25 smithi171 conmon[41853]: debug 2022-01-31T21:34:25.981+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:25.982138+0000) 2022-01-31T21:34:26.568 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:26 smithi167 conmon[49112]: debug 2022-01-31T21:34:26.426+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.426565+0000) 2022-01-31T21:34:26.569 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:26 smithi167 conmon[54076]: debug 2022-01-31T21:34:26.425+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.426079+0000) 2022-01-31T21:34:26.569 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:26 smithi167 conmon[60316]: debug 2022-01-31T21:34:26.424+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.425404+0000) 2022-01-31T21:34:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[41853]: debug 2022-01-31T21:34:26.424+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.426035+0000) 2022-01-31T21:34:26.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[46715]: debug 2022-01-31T21:34:26.424+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.426657+0000) 2022-01-31T21:34:26.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[51620]: debug 2022-01-31T21:34:26.426+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.427935+0000) 2022-01-31T21:34:26.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[35325]: debug 2022-01-31T21:34:26.440+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140480 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:26 smithi167 conmon[49112]: debug 2022-01-31T21:34:26.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.588096+0000) 2022-01-31T21:34:27.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:27 smithi167 conmon[54076]: debug 2022-01-31T21:34:27.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:27.251948+0000) 2022-01-31T21:34:27.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:27 smithi167 conmon[60316]: debug 2022-01-31T21:34:27.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:27.251948+0000) 2022-01-31T21:34:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[41853]: debug 2022-01-31T21:34:26.981+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.982326+0000) 2022-01-31T21:34:27.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:26 smithi171 conmon[51620]: debug 2022-01-31T21:34:26.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:26.861082+0000) 2022-01-31T21:34:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:27 smithi171 conmon[46715]: debug 2022-01-31T21:34:27.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:27.026051+0000) 2022-01-31T21:34:27.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:27 smithi167 conmon[49112]: debug 2022-01-31T21:34:27.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:27.588251+0000) 2022-01-31T21:34:27.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:27 smithi171 conmon[51620]: debug 2022-01-31T21:34:27.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:27.861245+0000) 2022-01-31T21:34:28.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:28 smithi167 conmon[54076]: debug 2022-01-31T21:34:28.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:28.252110+0000) 2022-01-31T21:34:28.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:28 smithi167 conmon[60316]: debug 2022-01-31T21:34:28.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:28.252110+0000) 2022-01-31T21:34:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:27 smithi171 conmon[41853]: debug 2022-01-31T21:34:27.981+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:27.982477+0000) 2022-01-31T21:34:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:28 smithi171 conmon[46715]: debug 2022-01-31T21:34:28.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:28.026179+0000) 2022-01-31T21:34:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:28 smithi167 conmon[49112]: debug 2022-01-31T21:34:28.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:28.588427+0000) 2022-01-31T21:34:28.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:28 smithi171 conmon[51620]: debug 2022-01-31T21:34:28.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:28.861452+0000) 2022-01-31T21:34:29.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:29 smithi167 conmon[54076]: debug 2022-01-31T21:34:29.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:29.252318+0000) 2022-01-31T21:34:29.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:29 smithi167 conmon[60316]: debug 2022-01-31T21:34:29.251+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:29.252318+0000) 2022-01-31T21:34:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:29 smithi171 conmon[46715]: debug 2022-01-31T21:34:29.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:29.026334+0000) 2022-01-31T21:34:29.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:28 smithi171 conmon[41853]: debug 2022-01-31T21:34:28.981+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:28.982681+0000) 2022-01-31T21:34:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:29 smithi167 conmon[49112]: debug 2022-01-31T21:34:29.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:29.588614+0000) 2022-01-31T21:34:29.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:29 smithi171 conmon[51620]: debug 2022-01-31T21:34:29.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:29.861659+0000) 2022-01-31T21:34:30.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:30 smithi167 conmon[54076]: debug 2022-01-31T21:34:30.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:30.252471+0000) 2022-01-31T21:34:30.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:30 smithi167 conmon[60316]: debug 2022-01-31T21:34:30.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:30.252464+0000) 2022-01-31T21:34:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:29 smithi171 conmon[41853]: debug 2022-01-31T21:34:29.982+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:29.982867+0000) 2022-01-31T21:34:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:30 smithi171 conmon[46715]: debug 2022-01-31T21:34:30.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:30.026517+0000) 2022-01-31T21:34:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:30 smithi167 conmon[49112]: debug 2022-01-31T21:34:30.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:30.588787+0000) 2022-01-31T21:34:30.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:30 smithi171 conmon[51620]: debug 2022-01-31T21:34:30.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:30.861844+0000) 2022-01-31T21:34:31.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:31 smithi167 conmon[54076]: debug 2022-01-31T21:34:31.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.252684+0000) 2022-01-31T21:34:31.307 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:31 smithi167 conmon[60316]: debug 2022-01-31T21:34:31.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.252684+0000) 2022-01-31T21:34:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:30 smithi171 conmon[41853]: debug 2022-01-31T21:34:30.982+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:30.983018+0000) 2022-01-31T21:34:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[46715]: debug 2022-01-31T21:34:31.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.026697+0000) 2022-01-31T21:34:31.569 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:31 smithi167 conmon[49112]: debug 2022-01-31T21:34:31.443+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.443558+0000) 2022-01-31T21:34:31.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:31 smithi167 conmon[54076]: debug 2022-01-31T21:34:31.444+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.445053+0000) 2022-01-31T21:34:31.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:31 smithi167 conmon[60316]: debug 2022-01-31T21:34:31.442+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.443162+0000) 2022-01-31T21:34:31.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[35325]: debug 2022-01-31T21:34:31.458+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140591 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[41853]: debug 2022-01-31T21:34:31.443+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.444551+0000) 2022-01-31T21:34:31.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[46715]: debug 2022-01-31T21:34:31.444+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.445272+0000) 2022-01-31T21:34:31.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[51620]: debug 2022-01-31T21:34:31.443+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.444121+0000) 2022-01-31T21:34:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:31 smithi167 conmon[49112]: debug 2022-01-31T21:34:31.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.588924+0000) 2022-01-31T21:34:32.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:32 smithi167 conmon[54076]: debug 2022-01-31T21:34:32.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:32.252887+0000) 2022-01-31T21:34:32.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:32 smithi167 conmon[60316]: debug 2022-01-31T21:34:32.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:32.252888+0000) 2022-01-31T21:34:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[41853]: debug 2022-01-31T21:34:31.982+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.983193+0000) 2022-01-31T21:34:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:32 smithi171 conmon[46715]: debug 2022-01-31T21:34:32.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:32.026877+0000) 2022-01-31T21:34:32.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:31 smithi171 conmon[51620]: debug 2022-01-31T21:34:31.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:31.862012+0000) 2022-01-31T21:34:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:32 smithi167 conmon[49112]: debug 2022-01-31T21:34:32.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:32.589065+0000) 2022-01-31T21:34:32.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:32 smithi171 conmon[51620]: debug 2022-01-31T21:34:32.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:32.862156+0000) 2022-01-31T21:34:33.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:33 smithi167 conmon[54076]: debug 2022-01-31T21:34:33.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:33.253057+0000) 2022-01-31T21:34:33.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:33 smithi167 conmon[60316]: debug 2022-01-31T21:34:33.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:33.253057+0000) 2022-01-31T21:34:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:32 smithi171 conmon[41853]: debug 2022-01-31T21:34:32.982+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:32.983406+0000) 2022-01-31T21:34:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:33 smithi171 conmon[46715]: debug 2022-01-31T21:34:33.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:33.027063+0000) 2022-01-31T21:34:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:33 smithi167 conmon[49112]: debug 2022-01-31T21:34:33.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:33.589262+0000) 2022-01-31T21:34:33.938 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:33 smithi171 conmon[51620]: debug 2022-01-31T21:34:33.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:33.862354+0000) 2022-01-31T21:34:34.303 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:33 smithi171 conmon[41853]: debug 2022-01-31T21:34:33.982+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:33.983573+0000) 2022-01-31T21:34:34.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:34 smithi171 conmon[46715]: debug 2022-01-31T21:34:34.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:34.027189+0000) 2022-01-31T21:34:34.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:34 smithi167 conmon[54076]: debug 2022-01-31T21:34:34.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:34.253239+0000) 2022-01-31T21:34:34.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:34 smithi167 conmon[60316]: debug 2022-01-31T21:34:34.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:34.253239+0000) 2022-01-31T21:34:34.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:34 smithi167 conmon[49112]: debug 2022-01-31T21:34:34.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:34.589484+0000) 2022-01-31T21:34:35.111 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:34 smithi171 conmon[41853]: debug 2022-01-31T21:34:34.983+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:34.983738+0000) 2022-01-31T21:34:35.112 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:35 smithi171 conmon[46715]: debug 2022-01-31T21:34:35.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:35.027340+0000) 2022-01-31T21:34:35.112 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:34 smithi171 conmon[51620]: debug 2022-01-31T21:34:34.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:34.862563+0000) 2022-01-31T21:34:35.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:35 smithi167 conmon[54076]: debug 2022-01-31T21:34:35.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:35.253444+0000) 2022-01-31T21:34:35.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:35 smithi167 conmon[60316]: debug 2022-01-31T21:34:35.252+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:35.253441+0000) 2022-01-31T21:34:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:35 smithi167 conmon[49112]: debug 2022-01-31T21:34:35.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:35.589716+0000) 2022-01-31T21:34:35.938 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:35 smithi171 conmon[51620]: debug 2022-01-31T21:34:35.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:35.862769+0000) 2022-01-31T21:34:36.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:36 smithi167 conmon[54076]: debug 2022-01-31T21:34:36.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.253698+0000) 2022-01-31T21:34:36.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:36 smithi167 conmon[60316]: debug 2022-01-31T21:34:36.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.253698+0000) 2022-01-31T21:34:36.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[46715]: debug 2022-01-31T21:34:36.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.027549+0000) 2022-01-31T21:34:36.352 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:35 smithi171 conmon[41853]: debug 2022-01-31T21:34:35.983+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:35.983929+0000) 2022-01-31T21:34:36.570 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:36 smithi167 conmon[49112]: debug 2022-01-31T21:34:36.461+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.461892+0000) 2022-01-31T21:34:36.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:36 smithi167 conmon[54076]: debug 2022-01-31T21:34:36.462+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.463026+0000) 2022-01-31T21:34:36.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:36 smithi167 conmon[60316]: debug 2022-01-31T21:34:36.461+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.461603+0000) 2022-01-31T21:34:36.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[35325]: debug 2022-01-31T21:34:36.476+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140702 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[41853]: debug 2022-01-31T21:34:36.461+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.462604+0000) 2022-01-31T21:34:36.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[46715]: debug 2022-01-31T21:34:36.462+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.462815+0000) 2022-01-31T21:34:36.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[51620]: debug 2022-01-31T21:34:36.460+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.461151+0000) 2022-01-31T21:34:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:36 smithi167 conmon[49112]: debug 2022-01-31T21:34:36.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.590091+0000) 2022-01-31T21:34:37.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:37 smithi167 conmon[54076]: debug 2022-01-31T21:34:37.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:37.253887+0000) 2022-01-31T21:34:37.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:37 smithi167 conmon[60316]: debug 2022-01-31T21:34:37.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:37.253888+0000) 2022-01-31T21:34:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:37 smithi171 conmon[46715]: debug 2022-01-31T21:34:37.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:37.027770+0000) 2022-01-31T21:34:37.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[51620]: debug 2022-01-31T21:34:36.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.862894+0000) 2022-01-31T21:34:37.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:36 smithi171 conmon[41853]: debug 2022-01-31T21:34:36.983+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:36.984114+0000) 2022-01-31T21:34:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:37 smithi167 conmon[49112]: debug 2022-01-31T21:34:37.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:37.590491+0000) 2022-01-31T21:34:37.938 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:37 smithi171 conmon[51620]: debug 2022-01-31T21:34:37.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:37.863019+0000) 2022-01-31T21:34:38.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:38 smithi167 conmon[60316]: debug 2022-01-31T21:34:38.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:38.254036+0000) 2022-01-31T21:34:38.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:38 smithi167 conmon[54076]: debug 2022-01-31T21:34:38.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:38.254036+0000) 2022-01-31T21:34:38.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:38 smithi171 conmon[46715]: debug 2022-01-31T21:34:38.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:38.027930+0000) 2022-01-31T21:34:38.336 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:37 smithi171 conmon[41853]: debug 2022-01-31T21:34:37.983+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:37.984250+0000) 2022-01-31T21:34:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:38 smithi167 conmon[49112]: debug 2022-01-31T21:34:38.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:38.590619+0000) 2022-01-31T21:34:38.939 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:38 smithi171 conmon[35325]: debug 2022-01-31T21:34:38.702+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:34:38.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:38 smithi171 conmon[51620]: debug 2022-01-31T21:34:38.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:38.863214+0000) 2022-01-31T21:34:39.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:39 smithi167 conmon[54076]: debug 2022-01-31T21:34:39.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:39.254237+0000) 2022-01-31T21:34:39.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:39 smithi167 conmon[60316]: debug 2022-01-31T21:34:39.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:39.254237+0000) 2022-01-31T21:34:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:39 smithi171 conmon[46715]: debug 2022-01-31T21:34:39.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:39.028108+0000) 2022-01-31T21:34:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:38 smithi171 conmon[41853]: debug 2022-01-31T21:34:38.983+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:38.984397+0000) 2022-01-31T21:34:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:39 smithi167 conmon[49112]: debug 2022-01-31T21:34:39.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:39.590725+0000) 2022-01-31T21:34:39.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:39 smithi171 conmon[51620]: debug 2022-01-31T21:34:39.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:39.863409+0000) 2022-01-31T21:34:40.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:40 smithi167 conmon[60316]: debug 2022-01-31T21:34:40.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:40.254402+0000) 2022-01-31T21:34:40.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:40 smithi167 conmon[54076]: debug 2022-01-31T21:34:40.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:40.254402+0000) 2022-01-31T21:34:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:39 smithi171 conmon[41853]: debug 2022-01-31T21:34:39.983+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:39.984553+0000) 2022-01-31T21:34:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:40 smithi171 conmon[46715]: debug 2022-01-31T21:34:40.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:40.028295+0000) 2022-01-31T21:34:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:40 smithi167 conmon[49112]: debug 2022-01-31T21:34:40.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:40.590833+0000) 2022-01-31T21:34:40.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:40 smithi171 conmon[51620]: debug 2022-01-31T21:34:40.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:40.863622+0000) 2022-01-31T21:34:41.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:41 smithi167 conmon[54076]: debug 2022-01-31T21:34:41.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.254629+0000) 2022-01-31T21:34:41.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:41 smithi167 conmon[60316]: debug 2022-01-31T21:34:41.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.254629+0000) 2022-01-31T21:34:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:40 smithi171 conmon[41853]: debug 2022-01-31T21:34:40.984+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:40.984779+0000) 2022-01-31T21:34:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[46715]: debug 2022-01-31T21:34:41.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.028443+0000) 2022-01-31T21:34:41.571 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:41 smithi167 conmon[49112]: debug 2022-01-31T21:34:41.478+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.478731+0000) 2022-01-31T21:34:41.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:41 smithi167 conmon[54076]: debug 2022-01-31T21:34:41.479+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.479738+0000) 2022-01-31T21:34:41.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:41 smithi167 conmon[60316]: debug 2022-01-31T21:34:41.478+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.478588+0000) 2022-01-31T21:34:41.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[35325]: debug 2022-01-31T21:34:41.493+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140811 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:41.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[51620]: debug 2022-01-31T21:34:41.478+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.479104+0000) 2022-01-31T21:34:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[41853]: debug 2022-01-31T21:34:41.479+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.480581+0000) 2022-01-31T21:34:41.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[46715]: debug 2022-01-31T21:34:41.479+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.479801+0000) 2022-01-31T21:34:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:41 smithi167 conmon[49112]: debug 2022-01-31T21:34:41.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.590997+0000) 2022-01-31T21:34:42.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:42 smithi167 conmon[54076]: debug 2022-01-31T21:34:42.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:42.254858+0000) 2022-01-31T21:34:42.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:42 smithi167 conmon[60316]: debug 2022-01-31T21:34:42.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:42.254858+0000) 2022-01-31T21:34:42.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[51620]: debug 2022-01-31T21:34:41.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.863833+0000) 2022-01-31T21:34:42.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:41 smithi171 conmon[41853]: debug 2022-01-31T21:34:41.984+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:41.984972+0000) 2022-01-31T21:34:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:42 smithi171 conmon[46715]: debug 2022-01-31T21:34:42.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:42.028625+0000) 2022-01-31T21:34:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:42 smithi167 conmon[49112]: debug 2022-01-31T21:34:42.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:42.591085+0000) 2022-01-31T21:34:42.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:42 smithi171 conmon[51620]: debug 2022-01-31T21:34:42.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:42.864006+0000) 2022-01-31T21:34:43.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:43 smithi167 conmon[54076]: debug 2022-01-31T21:34:43.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:43.255020+0000) 2022-01-31T21:34:43.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:43 smithi167 conmon[60316]: debug 2022-01-31T21:34:43.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:43.255020+0000) 2022-01-31T21:34:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:42 smithi171 conmon[41853]: debug 2022-01-31T21:34:42.984+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:42.985165+0000) 2022-01-31T21:34:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:43 smithi171 conmon[46715]: debug 2022-01-31T21:34:43.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:43.028837+0000) 2022-01-31T21:34:43.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:43 smithi167 conmon[49112]: debug 2022-01-31T21:34:43.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:43.591302+0000) 2022-01-31T21:34:43.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:43 smithi171 conmon[51620]: debug 2022-01-31T21:34:43.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:43.864169+0000) 2022-01-31T21:34:44.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:44 smithi167 conmon[54076]: debug 2022-01-31T21:34:44.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:44.255233+0000) 2022-01-31T21:34:44.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:44 smithi167 conmon[60316]: debug 2022-01-31T21:34:44.253+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:44.255233+0000) 2022-01-31T21:34:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:43 smithi171 conmon[41853]: debug 2022-01-31T21:34:43.984+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:43.985273+0000) 2022-01-31T21:34:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:44 smithi171 conmon[46715]: debug 2022-01-31T21:34:44.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:44.029013+0000) 2022-01-31T21:34:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:44 smithi167 conmon[49112]: debug 2022-01-31T21:34:44.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:44.591461+0000) 2022-01-31T21:34:44.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:44 smithi171 conmon[51620]: debug 2022-01-31T21:34:44.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:44.864331+0000) 2022-01-31T21:34:45.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:45 smithi167 conmon[54076]: debug 2022-01-31T21:34:45.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:45.255458+0000) 2022-01-31T21:34:45.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:45 smithi167 conmon[60316]: debug 2022-01-31T21:34:45.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:45.255458+0000) 2022-01-31T21:34:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:44 smithi171 conmon[41853]: debug 2022-01-31T21:34:44.985+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:44.985455+0000) 2022-01-31T21:34:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:45 smithi171 conmon[46715]: debug 2022-01-31T21:34:45.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:45.029198+0000) 2022-01-31T21:34:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:45 smithi167 conmon[49112]: debug 2022-01-31T21:34:45.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:45.591622+0000) 2022-01-31T21:34:45.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:45 smithi171 conmon[51620]: debug 2022-01-31T21:34:45.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:45.864509+0000) 2022-01-31T21:34:46.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:46 smithi167 conmon[54076]: debug 2022-01-31T21:34:46.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.255607+0000) 2022-01-31T21:34:46.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:46 smithi167 conmon[60316]: debug 2022-01-31T21:34:46.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.255610+0000) 2022-01-31T21:34:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:45 smithi171 conmon[41853]: debug 2022-01-31T21:34:45.984+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:45.985625+0000) 2022-01-31T21:34:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[46715]: debug 2022-01-31T21:34:46.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.029400+0000) 2022-01-31T21:34:46.571 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:46 smithi167 conmon[49112]: debug 2022-01-31T21:34:46.494+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.496246+0000) 2022-01-31T21:34:46.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:46 smithi167 conmon[54076]: debug 2022-01-31T21:34:46.495+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.496621+0000) 2022-01-31T21:34:46.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:46 smithi167 conmon[60316]: debug 2022-01-31T21:34:46.494+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.495863+0000) 2022-01-31T21:34:46.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[35325]: debug 2022-01-31T21:34:46.510+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 140923 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[41853]: debug 2022-01-31T21:34:46.496+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.497605+0000) 2022-01-31T21:34:46.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[46715]: debug 2022-01-31T21:34:46.496+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.497463+0000) 2022-01-31T21:34:46.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[51620]: debug 2022-01-31T21:34:46.496+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.497212+0000) 2022-01-31T21:34:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:46 smithi167 conmon[49112]: debug 2022-01-31T21:34:46.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.591762+0000) 2022-01-31T21:34:47.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:47 smithi167 conmon[54076]: debug 2022-01-31T21:34:47.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:47.255750+0000) 2022-01-31T21:34:47.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:47 smithi167 conmon[60316]: debug 2022-01-31T21:34:47.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:47.255752+0000) 2022-01-31T21:34:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[41853]: debug 2022-01-31T21:34:46.985+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.985810+0000) 2022-01-31T21:34:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:47 smithi171 conmon[46715]: debug 2022-01-31T21:34:47.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:47.029565+0000) 2022-01-31T21:34:47.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:46 smithi171 conmon[51620]: debug 2022-01-31T21:34:46.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:46.864660+0000) 2022-01-31T21:34:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:47 smithi167 conmon[49112]: debug 2022-01-31T21:34:47.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:47.591908+0000) 2022-01-31T21:34:47.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:47 smithi171 conmon[51620]: debug 2022-01-31T21:34:47.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:47.864821+0000) 2022-01-31T21:34:48.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:48 smithi167 conmon[60316]: debug 2022-01-31T21:34:48.254+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:48.255917+0000) 2022-01-31T21:34:48.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:48 smithi167 conmon[54076]: debug 2022-01-31T21:34:48.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:48.255917+0000) 2022-01-31T21:34:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:48 smithi171 conmon[46715]: debug 2022-01-31T21:34:48.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:48.029670+0000) 2022-01-31T21:34:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:47 smithi171 conmon[41853]: debug 2022-01-31T21:34:47.985+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:47.985967+0000) 2022-01-31T21:34:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:48 smithi167 conmon[49112]: debug 2022-01-31T21:34:48.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:48.592129+0000) 2022-01-31T21:34:48.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:48 smithi171 conmon[51620]: debug 2022-01-31T21:34:48.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:48.865012+0000) 2022-01-31T21:34:49.310 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:49 smithi167 conmon[60316]: debug 2022-01-31T21:34:49.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:49.256127+0000) 2022-01-31T21:34:49.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:49 smithi167 conmon[54076]: debug 2022-01-31T21:34:49.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:49.256127+0000) 2022-01-31T21:34:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:48 smithi171 conmon[41853]: debug 2022-01-31T21:34:48.985+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:48.986131+0000) 2022-01-31T21:34:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:49 smithi171 conmon[46715]: debug 2022-01-31T21:34:49.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:49.029809+0000) 2022-01-31T21:34:49.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:49 smithi167 conmon[49112]: debug 2022-01-31T21:34:49.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:49.592336+0000) 2022-01-31T21:34:49.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:49 smithi171 conmon[51620]: debug 2022-01-31T21:34:49.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:49.865191+0000) 2022-01-31T21:34:50.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:50 smithi167 conmon[54076]: debug 2022-01-31T21:34:50.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:50.256331+0000) 2022-01-31T21:34:50.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:50 smithi167 conmon[60316]: debug 2022-01-31T21:34:50.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:50.256331+0000) 2022-01-31T21:34:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:49 smithi171 conmon[41853]: debug 2022-01-31T21:34:49.985+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:49.986284+0000) 2022-01-31T21:34:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:50 smithi171 conmon[46715]: debug 2022-01-31T21:34:50.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:50.029963+0000) 2022-01-31T21:34:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:50 smithi167 conmon[49112]: debug 2022-01-31T21:34:50.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:50.592546+0000) 2022-01-31T21:34:50.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:50 smithi171 conmon[51620]: debug 2022-01-31T21:34:50.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:50.865368+0000) 2022-01-31T21:34:51.310 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:51 smithi167 conmon[54076]: debug 2022-01-31T21:34:51.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.256558+0000) 2022-01-31T21:34:51.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:51 smithi167 conmon[60316]: debug 2022-01-31T21:34:51.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.256558+0000) 2022-01-31T21:34:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:50 smithi171 conmon[41853]: debug 2022-01-31T21:34:50.986+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:50.986466+0000) 2022-01-31T21:34:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[46715]: debug 2022-01-31T21:34:51.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.030168+0000) 2022-01-31T21:34:51.572 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:51 smithi167 conmon[49112]: debug 2022-01-31T21:34:51.510+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.512366+0000) 2022-01-31T21:34:51.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:51 smithi167 conmon[54076]: debug 2022-01-31T21:34:51.511+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.513193+0000) 2022-01-31T21:34:51.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:51 smithi167 conmon[60316]: debug 2022-01-31T21:34:51.511+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.513278+0000) 2022-01-31T21:34:51.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[35325]: debug 2022-01-31T21:34:51.527+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141032 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[41853]: debug 2022-01-31T21:34:51.513+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.514692+0000) 2022-01-31T21:34:51.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[46715]: debug 2022-01-31T21:34:51.514+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.514771+0000) 2022-01-31T21:34:51.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[51620]: debug 2022-01-31T21:34:51.513+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.514264+0000) 2022-01-31T21:34:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:51 smithi167 conmon[49112]: debug 2022-01-31T21:34:51.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.592738+0000) 2022-01-31T21:34:52.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:52 smithi167 conmon[54076]: debug 2022-01-31T21:34:52.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:52.256740+0000) 2022-01-31T21:34:52.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:52 smithi167 conmon[60316]: debug 2022-01-31T21:34:52.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:52.256740+0000) 2022-01-31T21:34:52.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[51620]: debug 2022-01-31T21:34:51.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.865574+0000) 2022-01-31T21:34:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:51 smithi171 conmon[41853]: debug 2022-01-31T21:34:51.985+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:51.986673+0000) 2022-01-31T21:34:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:52 smithi171 conmon[46715]: debug 2022-01-31T21:34:52.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:52.030332+0000) 2022-01-31T21:34:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:52 smithi167 conmon[49112]: debug 2022-01-31T21:34:52.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:52.592857+0000) 2022-01-31T21:34:52.941 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:52 smithi171 conmon[51620]: debug 2022-01-31T21:34:52.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:52.865740+0000) 2022-01-31T21:34:53.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:53 smithi167 conmon[54076]: debug 2022-01-31T21:34:53.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:53.256880+0000) 2022-01-31T21:34:53.311 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:53 smithi167 conmon[60316]: debug 2022-01-31T21:34:53.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:53.256880+0000) 2022-01-31T21:34:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:52 smithi171 conmon[41853]: debug 2022-01-31T21:34:52.986+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:52.986847+0000) 2022-01-31T21:34:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:53 smithi171 conmon[46715]: debug 2022-01-31T21:34:53.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:53.030518+0000) 2022-01-31T21:34:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:53 smithi167 conmon[49112]: debug 2022-01-31T21:34:53.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:53.593031+0000) 2022-01-31T21:34:53.941 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:53 smithi171 conmon[35325]: debug 2022-01-31T21:34:53.703+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:34:53.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:53 smithi171 conmon[51620]: debug 2022-01-31T21:34:53.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:53.865880+0000) 2022-01-31T21:34:54.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:54 smithi167 conmon[54076]: debug 2022-01-31T21:34:54.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:54.257082+0000) 2022-01-31T21:34:54.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:54 smithi167 conmon[60316]: debug 2022-01-31T21:34:54.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:54.257062+0000) 2022-01-31T21:34:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:53 smithi171 conmon[41853]: debug 2022-01-31T21:34:53.986+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:53.987030+0000) 2022-01-31T21:34:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:54 smithi171 conmon[46715]: debug 2022-01-31T21:34:54.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:54.030708+0000) 2022-01-31T21:34:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:54 smithi167 conmon[49112]: debug 2022-01-31T21:34:54.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:54.593242+0000) 2022-01-31T21:34:54.941 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:54 smithi171 conmon[51620]: debug 2022-01-31T21:34:54.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:54.866026+0000) 2022-01-31T21:34:55.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:55 smithi167 conmon[54076]: debug 2022-01-31T21:34:55.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:55.257261+0000) 2022-01-31T21:34:55.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:55 smithi167 conmon[60316]: debug 2022-01-31T21:34:55.255+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:55.257264+0000) 2022-01-31T21:34:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:54 smithi171 conmon[41853]: debug 2022-01-31T21:34:54.986+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:54.987208+0000) 2022-01-31T21:34:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:55 smithi171 conmon[46715]: debug 2022-01-31T21:34:55.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:55.030900+0000) 2022-01-31T21:34:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:55 smithi167 conmon[49112]: debug 2022-01-31T21:34:55.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:55.593465+0000) 2022-01-31T21:34:55.941 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:55 smithi171 conmon[51620]: debug 2022-01-31T21:34:55.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:55.866238+0000) 2022-01-31T21:34:56.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:56 smithi167 conmon[54076]: debug 2022-01-31T21:34:56.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.257466+0000) 2022-01-31T21:34:56.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:56 smithi167 conmon[60316]: debug 2022-01-31T21:34:56.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.257476+0000) 2022-01-31T21:34:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:55 smithi171 conmon[41853]: debug 2022-01-31T21:34:55.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:55.987359+0000) 2022-01-31T21:34:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[46715]: debug 2022-01-31T21:34:56.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.031082+0000) 2022-01-31T21:34:56.573 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:56 smithi167 conmon[49112]: debug 2022-01-31T21:34:56.528+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.529997+0000) 2022-01-31T21:34:56.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:56 smithi167 conmon[54076]: debug 2022-01-31T21:34:56.529+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.531037+0000) 2022-01-31T21:34:56.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:56 smithi167 conmon[60316]: debug 2022-01-31T21:34:56.528+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.529931+0000) 2022-01-31T21:34:56.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[35325]: debug 2022-01-31T21:34:56.545+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141144 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:34:56.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[46715]: debug 2022-01-31T21:34:56.530+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.531022+0000) 2022-01-31T21:34:56.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[51620]: debug 2022-01-31T21:34:56.530+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.531365+0000) 2022-01-31T21:34:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[41853]: debug 2022-01-31T21:34:56.531+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.532370+0000) 2022-01-31T21:34:56.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:56 smithi167 conmon[49112]: debug 2022-01-31T21:34:56.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.593698+0000) 2022-01-31T21:34:57.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:57 smithi167 conmon[54076]: debug 2022-01-31T21:34:57.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:57.257698+0000) 2022-01-31T21:34:57.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:57 smithi167 conmon[60316]: debug 2022-01-31T21:34:57.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:57.257721+0000) 2022-01-31T21:34:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[41853]: debug 2022-01-31T21:34:56.986+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.987530+0000) 2022-01-31T21:34:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:57 smithi171 conmon[46715]: debug 2022-01-31T21:34:57.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:57.031245+0000) 2022-01-31T21:34:57.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:56 smithi171 conmon[51620]: debug 2022-01-31T21:34:56.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:56.866420+0000) 2022-01-31T21:34:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:57 smithi167 conmon[49112]: debug 2022-01-31T21:34:57.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:57.593857+0000) 2022-01-31T21:34:57.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:57 smithi171 conmon[51620]: debug 2022-01-31T21:34:57.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:57.866575+0000) 2022-01-31T21:34:58.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:58 smithi167 conmon[54076]: debug 2022-01-31T21:34:58.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:58.257868+0000) 2022-01-31T21:34:58.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:58 smithi167 conmon[60316]: debug 2022-01-31T21:34:58.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:58.257865+0000) 2022-01-31T21:34:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:57 smithi171 conmon[41853]: debug 2022-01-31T21:34:57.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:57.987686+0000) 2022-01-31T21:34:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:58 smithi171 conmon[46715]: debug 2022-01-31T21:34:58.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:58.031401+0000) 2022-01-31T21:34:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:58 smithi167 conmon[49112]: debug 2022-01-31T21:34:58.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:58.594081+0000) 2022-01-31T21:34:58.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:58 smithi171 conmon[51620]: debug 2022-01-31T21:34:58.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:58.866790+0000) 2022-01-31T21:34:59.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:34:59 smithi167 conmon[54076]: debug 2022-01-31T21:34:59.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:59.258036+0000) 2022-01-31T21:34:59.312 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:34:59 smithi167 conmon[60316]: debug 2022-01-31T21:34:59.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:59.258025+0000) 2022-01-31T21:34:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:58 smithi171 conmon[41853]: debug 2022-01-31T21:34:58.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:58.987832+0000) 2022-01-31T21:34:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:34:59 smithi171 conmon[46715]: debug 2022-01-31T21:34:59.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:59.031593+0000) 2022-01-31T21:34:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:34:59 smithi167 conmon[49112]: debug 2022-01-31T21:34:59.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:59.594265+0000) 2022-01-31T21:34:59.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:34:59 smithi171 conmon[51620]: debug 2022-01-31T21:34:59.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:59.867002+0000) 2022-01-31T21:35:00.312 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:00 smithi167 conmon[54076]: debug 2022-01-31T21:35:00.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:00.258217+0000) 2022-01-31T21:35:00.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:00 smithi167 conmon[60316]: debug 2022-01-31T21:35:00.256+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:00.258214+0000) 2022-01-31T21:35:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:00 smithi171 conmon[46715]: debug 2022-01-31T21:35:00.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:00.031776+0000) 2022-01-31T21:35:00.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:34:59 smithi171 conmon[41853]: debug 2022-01-31T21:34:59.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:34:59.987988+0000) 2022-01-31T21:35:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:00 smithi167 conmon[49112]: debug 2022-01-31T21:35:00.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:00.594410+0000) 2022-01-31T21:35:00.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:00 smithi171 conmon[51620]: debug 2022-01-31T21:35:00.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:00.867194+0000) 2022-01-31T21:35:01.312 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:01 smithi167 conmon[54076]: debug 2022-01-31T21:35:01.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.258397+0000) 2022-01-31T21:35:01.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:01 smithi167 conmon[60316]: debug 2022-01-31T21:35:01.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.258400+0000) 2022-01-31T21:35:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:00 smithi171 conmon[41853]: debug 2022-01-31T21:35:00.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:00.988173+0000) 2022-01-31T21:35:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[46715]: debug 2022-01-31T21:35:01.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.031911+0000) 2022-01-31T21:35:01.574 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:01 smithi167 conmon[49112]: debug 2022-01-31T21:35:01.546+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.547980+0000) 2022-01-31T21:35:01.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:01 smithi167 conmon[54076]: debug 2022-01-31T21:35:01.547+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.548840+0000) 2022-01-31T21:35:01.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:01 smithi167 conmon[60316]: debug 2022-01-31T21:35:01.545+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.547445+0000) 2022-01-31T21:35:01.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[46715]: debug 2022-01-31T21:35:01.546+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.547622+0000) 2022-01-31T21:35:01.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[35325]: debug 2022-01-31T21:35:01.562+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141253 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[41853]: debug 2022-01-31T21:35:01.547+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.548459+0000) 2022-01-31T21:35:01.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[51620]: debug 2022-01-31T21:35:01.548+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.549724+0000) 2022-01-31T21:35:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:01 smithi167 conmon[49112]: debug 2022-01-31T21:35:01.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.594570+0000) 2022-01-31T21:35:02.312 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:02 smithi167 conmon[54076]: debug 2022-01-31T21:35:02.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:02.258530+0000) 2022-01-31T21:35:02.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:02 smithi167 conmon[60316]: debug 2022-01-31T21:35:02.257+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:02.258545+0000) 2022-01-31T21:35:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:02 smithi171 conmon[46715]: debug 2022-01-31T21:35:02.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:02.032040+0000) 2022-01-31T21:35:02.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[51620]: debug 2022-01-31T21:35:01.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.867371+0000) 2022-01-31T21:35:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:01 smithi171 conmon[41853]: debug 2022-01-31T21:35:01.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:01.988356+0000) 2022-01-31T21:35:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:02 smithi167 conmon[49112]: debug 2022-01-31T21:35:02.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:02.594711+0000) 2022-01-31T21:35:02.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:02 smithi171 conmon[51620]: debug 2022-01-31T21:35:02.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:02.867486+0000) 2022-01-31T21:35:03.312 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:03 smithi167 conmon[54076]: debug 2022-01-31T21:35:03.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:03.258703+0000) 2022-01-31T21:35:03.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:03 smithi167 conmon[60316]: debug 2022-01-31T21:35:03.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:03.258700+0000) 2022-01-31T21:35:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:02 smithi171 conmon[41853]: debug 2022-01-31T21:35:02.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:02.988548+0000) 2022-01-31T21:35:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:03 smithi171 conmon[46715]: debug 2022-01-31T21:35:03.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:03.032241+0000) 2022-01-31T21:35:03.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:03 smithi167 conmon[49112]: debug 2022-01-31T21:35:03.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:03.594891+0000) 2022-01-31T21:35:03.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:03 smithi171 conmon[51620]: debug 2022-01-31T21:35:03.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:03.867690+0000) 2022-01-31T21:35:04.312 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:04 smithi167 conmon[54076]: debug 2022-01-31T21:35:04.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:04.258903+0000) 2022-01-31T21:35:04.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:04 smithi167 conmon[60316]: debug 2022-01-31T21:35:04.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:04.258903+0000) 2022-01-31T21:35:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:04 smithi171 conmon[46715]: debug 2022-01-31T21:35:04.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:04.032445+0000) 2022-01-31T21:35:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:03 smithi171 conmon[41853]: debug 2022-01-31T21:35:03.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:03.988731+0000) 2022-01-31T21:35:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:04 smithi167 conmon[49112]: debug 2022-01-31T21:35:04.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:04.595075+0000) 2022-01-31T21:35:04.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:04 smithi171 conmon[51620]: debug 2022-01-31T21:35:04.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:04.867833+0000) 2022-01-31T21:35:05.313 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:05 smithi167 conmon[54076]: debug 2022-01-31T21:35:05.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:05.259061+0000) 2022-01-31T21:35:05.313 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:05 smithi167 conmon[60316]: debug 2022-01-31T21:35:05.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:05.259069+0000) 2022-01-31T21:35:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:05 smithi171 conmon[46715]: debug 2022-01-31T21:35:05.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:05.032632+0000) 2022-01-31T21:35:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:04 smithi171 conmon[41853]: debug 2022-01-31T21:35:04.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:04.988934+0000) 2022-01-31T21:35:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:05 smithi167 conmon[49112]: debug 2022-01-31T21:35:05.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:05.595252+0000) 2022-01-31T21:35:05.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:05 smithi171 conmon[51620]: debug 2022-01-31T21:35:05.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:05.868004+0000) 2022-01-31T21:35:06.313 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:06 smithi167 conmon[54076]: debug 2022-01-31T21:35:06.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.259231+0000) 2022-01-31T21:35:06.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:06 smithi167 conmon[60316]: debug 2022-01-31T21:35:06.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.259237+0000) 2022-01-31T21:35:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[46715]: debug 2022-01-31T21:35:06.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.032829+0000) 2022-01-31T21:35:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:05 smithi171 conmon[41853]: debug 2022-01-31T21:35:05.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:05.989123+0000) 2022-01-31T21:35:06.563 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:06 smithi167 conmon[60316]: debug 2022-01-31T21:35:06.563+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.564166+0000) 2022-01-31T21:35:06.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[35325]: debug 2022-01-31T21:35:06.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[35325]: 2022-01-31T21:35:06.583+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141365 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[41853]: debug 2022-01-31T21:35:06.563+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.565087+0000) 2022-01-31T21:35:06.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[46715]: debug 2022-01-31T21:35:06.564+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.566278+0000) 2022-01-31T21:35:06.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[51620]: debug 2022-01-31T21:35:06.564+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.566649+0000) 2022-01-31T21:35:06.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:06 smithi167 conmon[54076]: debug 2022-01-31T21:35:06.565+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.566205+0000) 2022-01-31T21:35:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:06 smithi167 conmon[49112]: debug 2022-01-31T21:35:06.565+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.565714+0000) 2022-01-31T21:35:06.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:06 smithi167 conmon[49112]: debug 2022-01-31T21:35:06.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.595413+0000) 2022-01-31T21:35:07.313 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:07 smithi167 conmon[54076]: debug 2022-01-31T21:35:07.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:07.259432+0000) 2022-01-31T21:35:07.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:07 smithi167 conmon[60316]: debug 2022-01-31T21:35:07.258+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:07.259453+0000) 2022-01-31T21:35:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:07 smithi171 conmon[46715]: debug 2022-01-31T21:35:07.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:07.032999+0000) 2022-01-31T21:35:07.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[51620]: debug 2022-01-31T21:35:06.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.868171+0000) 2022-01-31T21:35:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:06 smithi171 conmon[41853]: debug 2022-01-31T21:35:06.987+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:06.989307+0000) 2022-01-31T21:35:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:07 smithi167 conmon[49112]: debug 2022-01-31T21:35:07.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:07.595584+0000) 2022-01-31T21:35:07.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:07 smithi171 conmon[51620]: debug 2022-01-31T21:35:07.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:07.868340+0000) 2022-01-31T21:35:08.313 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:08 smithi167 conmon[54076]: debug 2022-01-31T21:35:08.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:08.259608+0000) 2022-01-31T21:35:08.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:08 smithi167 conmon[60316]: debug 2022-01-31T21:35:08.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:08.259609+0000) 2022-01-31T21:35:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:07 smithi171 conmon[41853]: debug 2022-01-31T21:35:07.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:07.989458+0000) 2022-01-31T21:35:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:08 smithi171 conmon[46715]: debug 2022-01-31T21:35:08.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:08.033096+0000) 2022-01-31T21:35:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:08 smithi167 conmon[49112]: debug 2022-01-31T21:35:08.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:08.595813+0000) 2022-01-31T21:35:08.944 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:08 smithi171 conmon[35325]: debug 2022-01-31T21:35:08.703+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:35:08.944 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:08 smithi171 conmon[51620]: debug 2022-01-31T21:35:08.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:08.868479+0000) 2022-01-31T21:35:09.313 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:09 smithi167 conmon[54076]: debug 2022-01-31T21:35:09.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:09.259791+0000) 2022-01-31T21:35:09.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:09 smithi167 conmon[60316]: debug 2022-01-31T21:35:09.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:09.259803+0000) 2022-01-31T21:35:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:08 smithi171 conmon[41853]: debug 2022-01-31T21:35:08.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:08.989640+0000) 2022-01-31T21:35:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:09 smithi171 conmon[46715]: debug 2022-01-31T21:35:09.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:09.033259+0000) 2022-01-31T21:35:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:09 smithi167 conmon[49112]: debug 2022-01-31T21:35:09.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:09.595972+0000) 2022-01-31T21:35:09.944 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:09 smithi171 conmon[51620]: debug 2022-01-31T21:35:09.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:09.868604+0000) 2022-01-31T21:35:10.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:10 smithi167 conmon[54076]: debug 2022-01-31T21:35:10.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:10.259992+0000) 2022-01-31T21:35:10.314 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:10 smithi167 conmon[60316]: debug 2022-01-31T21:35:10.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:10.260000+0000) 2022-01-31T21:35:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:09 smithi171 conmon[41853]: debug 2022-01-31T21:35:09.988+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:09.989844+0000) 2022-01-31T21:35:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:10 smithi171 conmon[46715]: debug 2022-01-31T21:35:10.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:10.033469+0000) 2022-01-31T21:35:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:10 smithi167 conmon[49112]: debug 2022-01-31T21:35:10.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:10.596175+0000) 2022-01-31T21:35:10.944 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:10 smithi171 conmon[51620]: debug 2022-01-31T21:35:10.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:10.868813+0000) 2022-01-31T21:35:11.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:11 smithi167 conmon[54076]: debug 2022-01-31T21:35:11.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.260192+0000) 2022-01-31T21:35:11.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:11 smithi167 conmon[60316]: debug 2022-01-31T21:35:11.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.260223+0000) 2022-01-31T21:35:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:10 smithi171 conmon[41853]: debug 2022-01-31T21:35:10.989+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:10.990042+0000) 2022-01-31T21:35:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[46715]: debug 2022-01-31T21:35:11.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.033667+0000) 2022-01-31T21:35:11.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[35325]: debug 2022-01-31T21:35:11.600+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141474 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:11.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[46715]: debug 2022-01-31T21:35:11.587+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.589454+0000) 2022-01-31T21:35:11.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[51620]: debug 2022-01-31T21:35:11.586+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.588274+0000) 2022-01-31T21:35:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[41853]: debug 2022-01-31T21:35:11.586+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.588118+0000) 2022-01-31T21:35:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:11 smithi167 conmon[49112]: debug 2022-01-31T21:35:11.587+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.588042+0000) 2022-01-31T21:35:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:11 smithi167 conmon[49112]: debug 2022-01-31T21:35:11.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.596406+0000) 2022-01-31T21:35:11.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:11 smithi167 conmon[54076]: debug 2022-01-31T21:35:11.587+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.588297+0000) 2022-01-31T21:35:11.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:11 smithi167 conmon[60316]: debug 2022-01-31T21:35:11.586+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.587327+0000) 2022-01-31T21:35:12.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:12 smithi167 conmon[54076]: debug 2022-01-31T21:35:12.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:12.260376+0000) 2022-01-31T21:35:12.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:12 smithi167 conmon[60316]: debug 2022-01-31T21:35:12.259+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:12.260387+0000) 2022-01-31T21:35:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:12 smithi171 conmon[46715]: debug 2022-01-31T21:35:12.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:12.033846+0000) 2022-01-31T21:35:12.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[51620]: debug 2022-01-31T21:35:11.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.868926+0000) 2022-01-31T21:35:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:11 smithi171 conmon[41853]: debug 2022-01-31T21:35:11.989+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:11.990200+0000) 2022-01-31T21:35:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:12 smithi167 conmon[49112]: debug 2022-01-31T21:35:12.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:12.596567+0000) 2022-01-31T21:35:12.944 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:12 smithi171 conmon[51620]: debug 2022-01-31T21:35:12.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:12.869013+0000) 2022-01-31T21:35:13.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:13 smithi167 conmon[54076]: debug 2022-01-31T21:35:13.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:13.260534+0000) 2022-01-31T21:35:13.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:13 smithi167 conmon[60316]: debug 2022-01-31T21:35:13.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:13.260541+0000) 2022-01-31T21:35:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:12 smithi171 conmon[41853]: debug 2022-01-31T21:35:12.989+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:12.990374+0000) 2022-01-31T21:35:13.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:13 smithi171 conmon[46715]: debug 2022-01-31T21:35:13.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:13.033997+0000) 2022-01-31T21:35:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:13 smithi167 conmon[49112]: debug 2022-01-31T21:35:13.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:13.596701+0000) 2022-01-31T21:35:13.944 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:13 smithi171 conmon[51620]: debug 2022-01-31T21:35:13.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:13.869177+0000) 2022-01-31T21:35:14.314 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:14 smithi167 conmon[54076]: debug 2022-01-31T21:35:14.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:14.260728+0000) 2022-01-31T21:35:14.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:14 smithi167 conmon[60316]: debug 2022-01-31T21:35:14.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:14.260728+0000) 2022-01-31T21:35:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:14 smithi171 conmon[46715]: debug 2022-01-31T21:35:14.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:14.034153+0000) 2022-01-31T21:35:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:13 smithi171 conmon[41853]: debug 2022-01-31T21:35:13.989+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:13.990554+0000) 2022-01-31T21:35:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:14 smithi167 conmon[49112]: debug 2022-01-31T21:35:14.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:14.596870+0000) 2022-01-31T21:35:14.945 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:14 smithi171 conmon[51620]: debug 2022-01-31T21:35:14.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:14.869358+0000) 2022-01-31T21:35:15.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:15 smithi167 conmon[54076]: debug 2022-01-31T21:35:15.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:15.260888+0000) 2022-01-31T21:35:15.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:15 smithi167 conmon[60316]: debug 2022-01-31T21:35:15.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:15.260869+0000) 2022-01-31T21:35:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:15 smithi171 conmon[46715]: debug 2022-01-31T21:35:15.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:15.034358+0000) 2022-01-31T21:35:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:14 smithi171 conmon[41853]: debug 2022-01-31T21:35:14.989+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:14.990703+0000) 2022-01-31T21:35:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:15 smithi167 conmon[49112]: debug 2022-01-31T21:35:15.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:15.597080+0000) 2022-01-31T21:35:15.945 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:15 smithi171 conmon[51620]: debug 2022-01-31T21:35:15.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:15.869555+0000) 2022-01-31T21:35:16.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:16 smithi167 conmon[54076]: debug 2022-01-31T21:35:16.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.261085+0000) 2022-01-31T21:35:16.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:16 smithi167 conmon[60316]: debug 2022-01-31T21:35:16.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.261030+0000) 2022-01-31T21:35:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[46715]: debug 2022-01-31T21:35:16.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.034565+0000) 2022-01-31T21:35:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:15 smithi171 conmon[41853]: debug 2022-01-31T21:35:15.990+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:15.990906+0000) 2022-01-31T21:35:16.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[35325]: debug 2022-01-31T21:35:16.619+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141586 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[41853]: debug 2022-01-31T21:35:16.604+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.605497+0000) 2022-01-31T21:35:16.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[46715]: debug 2022-01-31T21:35:16.605+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.606347+0000) 2022-01-31T21:35:16.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[51620]: debug 2022-01-31T21:35:16.603+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.604311+0000) 2022-01-31T21:35:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:16 smithi167 conmon[49112]: debug 2022-01-31T21:35:16.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.597239+0000) 2022-01-31T21:35:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:16 smithi167 conmon[49112]: debug 2022-01-31T21:35:16.604+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.605247+0000) 2022-01-31T21:35:16.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:16 smithi167 conmon[54076]: debug 2022-01-31T21:35:16.603+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.604366+0000) 2022-01-31T21:35:16.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:16 smithi167 conmon[60316]: debug 2022-01-31T21:35:16.604+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.605172+0000) 2022-01-31T21:35:17.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:17 smithi167 conmon[54076]: debug 2022-01-31T21:35:17.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:17.261243+0000) 2022-01-31T21:35:17.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:17 smithi167 conmon[60316]: debug 2022-01-31T21:35:17.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:17.261203+0000) 2022-01-31T21:35:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[41853]: debug 2022-01-31T21:35:16.990+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.991088+0000) 2022-01-31T21:35:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:17 smithi171 conmon[46715]: debug 2022-01-31T21:35:17.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:17.034750+0000) 2022-01-31T21:35:17.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:16 smithi171 conmon[51620]: debug 2022-01-31T21:35:16.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:16.869773+0000) 2022-01-31T21:35:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:17 smithi167 conmon[49112]: debug 2022-01-31T21:35:17.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:17.597430+0000) 2022-01-31T21:35:17.945 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:17 smithi171 conmon[51620]: debug 2022-01-31T21:35:17.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:17.869969+0000) 2022-01-31T21:35:18.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:18 smithi167 conmon[54076]: debug 2022-01-31T21:35:18.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:18.261392+0000) 2022-01-31T21:35:18.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:18 smithi167 conmon[60316]: debug 2022-01-31T21:35:18.260+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:18.261317+0000) 2022-01-31T21:35:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:17 smithi171 conmon[41853]: debug 2022-01-31T21:35:17.990+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:17.991263+0000) 2022-01-31T21:35:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:18 smithi171 conmon[46715]: debug 2022-01-31T21:35:18.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:18.034927+0000) 2022-01-31T21:35:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:18 smithi167 conmon[49112]: debug 2022-01-31T21:35:18.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:18.597606+0000) 2022-01-31T21:35:18.945 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:18 smithi171 conmon[51620]: debug 2022-01-31T21:35:18.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:18.870173+0000) 2022-01-31T21:35:19.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:19 smithi167 conmon[54076]: debug 2022-01-31T21:35:19.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:19.261545+0000) 2022-01-31T21:35:19.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:19 smithi167 conmon[60316]: debug 2022-01-31T21:35:19.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:19.261485+0000) 2022-01-31T21:35:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:19 smithi171 conmon[46715]: debug 2022-01-31T21:35:19.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:19.035114+0000) 2022-01-31T21:35:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:18 smithi171 conmon[41853]: debug 2022-01-31T21:35:18.990+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:18.991446+0000) 2022-01-31T21:35:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:19 smithi167 conmon[49112]: debug 2022-01-31T21:35:19.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:19.597773+0000) 2022-01-31T21:35:19.946 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:19 smithi171 conmon[51620]: debug 2022-01-31T21:35:19.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:19.870399+0000) 2022-01-31T21:35:20.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:20 smithi167 conmon[54076]: debug 2022-01-31T21:35:20.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:20.261744+0000) 2022-01-31T21:35:20.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:20 smithi167 conmon[60316]: debug 2022-01-31T21:35:20.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:20.261676+0000) 2022-01-31T21:35:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:19 smithi171 conmon[41853]: debug 2022-01-31T21:35:19.990+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:19.991630+0000) 2022-01-31T21:35:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:20 smithi171 conmon[46715]: debug 2022-01-31T21:35:20.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:20.035329+0000) 2022-01-31T21:35:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:20 smithi167 conmon[49112]: debug 2022-01-31T21:35:20.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:20.597938+0000) 2022-01-31T21:35:20.946 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:20 smithi171 conmon[51620]: debug 2022-01-31T21:35:20.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:20.870572+0000) 2022-01-31T21:35:21.316 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:21 smithi167 conmon[54076]: debug 2022-01-31T21:35:21.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.261924+0000) 2022-01-31T21:35:21.316 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:21 smithi167 conmon[60316]: debug 2022-01-31T21:35:21.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.261850+0000) 2022-01-31T21:35:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:20 smithi171 conmon[41853]: debug 2022-01-31T21:35:20.991+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:20.991834+0000) 2022-01-31T21:35:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[46715]: debug 2022-01-31T21:35:21.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.035482+0000) 2022-01-31T21:35:21.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:21 smithi167 conmon[54076]: debug 2022-01-31T21:35:21.620+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.621401+0000) 2022-01-31T21:35:21.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:21 smithi167 conmon[60316]: debug 2022-01-31T21:35:21.622+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.623222+0000) 2022-01-31T21:35:21.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:21 smithi167 conmon[49112]: debug 2022-01-31T21:35:21.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.598127+0000) 2022-01-31T21:35:21.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:21 smithi167 conmon[49112]: debug 2022-01-31T21:35:21.622+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.622768+0000) 2022-01-31T21:35:21.946 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[35325]: debug 2022-01-31T21:35:21.637+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141695 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:21.947 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[41853]: debug 2022-01-31T21:35:21.622+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.623019+0000) 2022-01-31T21:35:21.947 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[46715]: debug 2022-01-31T21:35:21.623+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.623864+0000) 2022-01-31T21:35:21.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[51620]: debug 2022-01-31T21:35:21.621+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.622341+0000) 2022-01-31T21:35:21.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[51620]: debug 2022-01-31T21:35:21.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.870783+0000) 2022-01-31T21:35:22.316 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:22 smithi167 conmon[54076]: debug 2022-01-31T21:35:22.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:22.262124+0000) 2022-01-31T21:35:22.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:22 smithi167 conmon[60316]: debug 2022-01-31T21:35:22.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:22.262037+0000) 2022-01-31T21:35:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:21 smithi171 conmon[41853]: debug 2022-01-31T21:35:21.991+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:21.992041+0000) 2022-01-31T21:35:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:22 smithi171 conmon[46715]: debug 2022-01-31T21:35:22.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:22.035637+0000) 2022-01-31T21:35:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:22 smithi167 conmon[49112]: debug 2022-01-31T21:35:22.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:22.598301+0000) 2022-01-31T21:35:22.946 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:22 smithi171 conmon[51620]: debug 2022-01-31T21:35:22.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:22.870933+0000) 2022-01-31T21:35:23.316 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:23 smithi167 conmon[54076]: debug 2022-01-31T21:35:23.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:23.262277+0000) 2022-01-31T21:35:23.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:23 smithi167 conmon[60316]: debug 2022-01-31T21:35:23.261+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:23.262192+0000) 2022-01-31T21:35:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:22 smithi171 conmon[41853]: debug 2022-01-31T21:35:22.991+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:22.992193+0000) 2022-01-31T21:35:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:23 smithi171 conmon[46715]: debug 2022-01-31T21:35:23.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:23.035807+0000) 2022-01-31T21:35:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:23 smithi167 conmon[49112]: debug 2022-01-31T21:35:23.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:23.598463+0000) 2022-01-31T21:35:23.946 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:23 smithi171 conmon[51620]: debug 2022-01-31T21:35:23.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:23.871069+0000) 2022-01-31T21:35:23.947 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:23 smithi171 conmon[35325]: debug 2022-01-31T21:35:23.704+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:35:24.316 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:24 smithi167 conmon[54076]: debug 2022-01-31T21:35:24.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:24.262472+0000) 2022-01-31T21:35:24.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:24 smithi167 conmon[60316]: debug 2022-01-31T21:35:24.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:24.262370+0000) 2022-01-31T21:35:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:23 smithi171 conmon[41853]: debug 2022-01-31T21:35:23.991+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:23.992409+0000) 2022-01-31T21:35:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:24 smithi171 conmon[46715]: debug 2022-01-31T21:35:24.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:24.035995+0000) 2022-01-31T21:35:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:24 smithi167 conmon[49112]: debug 2022-01-31T21:35:24.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:24.598673+0000) 2022-01-31T21:35:24.946 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:24 smithi171 conmon[51620]: debug 2022-01-31T21:35:24.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:24.871251+0000) 2022-01-31T21:35:25.316 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:25 smithi167 conmon[54076]: debug 2022-01-31T21:35:25.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:25.262656+0000) 2022-01-31T21:35:25.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:25 smithi167 conmon[60316]: debug 2022-01-31T21:35:25.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:25.262583+0000) 2022-01-31T21:35:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:24 smithi171 conmon[41853]: debug 2022-01-31T21:35:24.991+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:24.992577+0000) 2022-01-31T21:35:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:25 smithi171 conmon[46715]: debug 2022-01-31T21:35:25.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:25.036181+0000) 2022-01-31T21:35:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:25 smithi167 conmon[49112]: debug 2022-01-31T21:35:25.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:25.598848+0000) 2022-01-31T21:35:25.947 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:25 smithi171 conmon[51620]: debug 2022-01-31T21:35:25.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:25.871447+0000) 2022-01-31T21:35:26.317 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:26 smithi167 conmon[54076]: debug 2022-01-31T21:35:26.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.262919+0000) 2022-01-31T21:35:26.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:26 smithi167 conmon[60316]: debug 2022-01-31T21:35:26.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.262808+0000) 2022-01-31T21:35:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:25 smithi171 conmon[41853]: debug 2022-01-31T21:35:25.992+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:25.992768+0000) 2022-01-31T21:35:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[46715]: debug 2022-01-31T21:35:26.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.036420+0000) 2022-01-31T21:35:26.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:26 smithi167 conmon[54076]: debug 2022-01-31T21:35:26.640+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.640664+0000) 2022-01-31T21:35:26.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:26 smithi167 conmon[60316]: debug 2022-01-31T21:35:26.640+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.641124+0000) 2022-01-31T21:35:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:26 smithi167 conmon[49112]: debug 2022-01-31T21:35:26.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.599103+0000) 2022-01-31T21:35:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:26 smithi167 conmon[49112]: debug 2022-01-31T21:35:26.638+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.639441+0000) 2022-01-31T21:35:26.947 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[35325]: debug 2022-01-31T21:35:26.654+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141807 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:26.948 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[46715]: debug 2022-01-31T21:35:26.639+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.640611+0000) 2022-01-31T21:35:26.948 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[41853]: debug 2022-01-31T21:35:26.640+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.640844+0000) 2022-01-31T21:35:26.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[51620]: debug 2022-01-31T21:35:26.638+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.639412+0000) 2022-01-31T21:35:26.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[51620]: debug 2022-01-31T21:35:26.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.871645+0000) 2022-01-31T21:35:27.317 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:27 smithi167 conmon[54076]: debug 2022-01-31T21:35:27.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:27.263078+0000) 2022-01-31T21:35:27.318 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:27 smithi167 conmon[60316]: debug 2022-01-31T21:35:27.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:27.262982+0000) 2022-01-31T21:35:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:27 smithi171 conmon[46715]: debug 2022-01-31T21:35:27.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:27.036569+0000) 2022-01-31T21:35:27.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:26 smithi171 conmon[41853]: debug 2022-01-31T21:35:26.992+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:26.992917+0000) 2022-01-31T21:35:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:27 smithi167 conmon[49112]: debug 2022-01-31T21:35:27.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:27.599222+0000) 2022-01-31T21:35:27.947 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:27 smithi171 conmon[51620]: debug 2022-01-31T21:35:27.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:27.871818+0000) 2022-01-31T21:35:28.317 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:28 smithi167 conmon[54076]: debug 2022-01-31T21:35:28.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:28.263213+0000) 2022-01-31T21:35:28.318 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:28 smithi167 conmon[60316]: debug 2022-01-31T21:35:28.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:28.263161+0000) 2022-01-31T21:35:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:28 smithi171 conmon[46715]: debug 2022-01-31T21:35:28.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:28.036741+0000) 2022-01-31T21:35:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:27 smithi171 conmon[41853]: debug 2022-01-31T21:35:27.992+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:27.993021+0000) 2022-01-31T21:35:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:28 smithi167 conmon[49112]: debug 2022-01-31T21:35:28.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:28.599406+0000) 2022-01-31T21:35:28.947 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:28 smithi171 conmon[51620]: debug 2022-01-31T21:35:28.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:28.872001+0000) 2022-01-31T21:35:29.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:29 smithi167 conmon[60316]: debug 2022-01-31T21:35:29.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:29.263342+0000) 2022-01-31T21:35:29.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:29 smithi167 conmon[54076]: debug 2022-01-31T21:35:29.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:29.263392+0000) 2022-01-31T21:35:29.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:28 smithi171 conmon[41853]: debug 2022-01-31T21:35:28.992+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:28.993180+0000) 2022-01-31T21:35:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:29 smithi171 conmon[46715]: debug 2022-01-31T21:35:29.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:29.036910+0000) 2022-01-31T21:35:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:29 smithi167 conmon[49112]: debug 2022-01-31T21:35:29.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:29.599596+0000) 2022-01-31T21:35:29.947 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:29 smithi171 conmon[51620]: debug 2022-01-31T21:35:29.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:29.872204+0000) 2022-01-31T21:35:30.317 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:30 smithi167 conmon[60316]: debug 2022-01-31T21:35:30.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:30.263544+0000) 2022-01-31T21:35:30.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:30 smithi167 conmon[54076]: debug 2022-01-31T21:35:30.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:30.263604+0000) 2022-01-31T21:35:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:30 smithi171 conmon[46715]: debug 2022-01-31T21:35:30.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:30.037070+0000) 2022-01-31T21:35:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:29 smithi171 conmon[41853]: debug 2022-01-31T21:35:29.992+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:29.993363+0000) 2022-01-31T21:35:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:30 smithi167 conmon[49112]: debug 2022-01-31T21:35:30.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:30.599754+0000) 2022-01-31T21:35:30.947 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:30 smithi171 conmon[51620]: debug 2022-01-31T21:35:30.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:30.872380+0000) 2022-01-31T21:35:31.317 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:31 smithi167 conmon[54076]: debug 2022-01-31T21:35:31.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.263758+0000) 2022-01-31T21:35:31.318 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:31 smithi167 conmon[60316]: debug 2022-01-31T21:35:31.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.263738+0000) 2022-01-31T21:35:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:30 smithi171 conmon[41853]: debug 2022-01-31T21:35:30.993+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:30.993550+0000) 2022-01-31T21:35:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[46715]: debug 2022-01-31T21:35:31.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.037264+0000) 2022-01-31T21:35:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:31 smithi167 conmon[49112]: debug 2022-01-31T21:35:31.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.599949+0000) 2022-01-31T21:35:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:31 smithi167 conmon[49112]: debug 2022-01-31T21:35:31.655+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.656890+0000) 2022-01-31T21:35:31.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:31 smithi167 conmon[54076]: debug 2022-01-31T21:35:31.656+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.657973+0000) 2022-01-31T21:35:31.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:31 smithi167 conmon[60316]: debug 2022-01-31T21:35:31.656+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.657911+0000) 2022-01-31T21:35:31.948 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[46715]: debug 2022-01-31T21:35:31.657+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.657852+0000) 2022-01-31T21:35:31.948 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[35325]: debug 2022-01-31T21:35:31.672+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 141916 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:31.949 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[41853]: debug 2022-01-31T21:35:31.657+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.658271+0000) 2022-01-31T21:35:31.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[51620]: debug 2022-01-31T21:35:31.657+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.657919+0000) 2022-01-31T21:35:31.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[51620]: debug 2022-01-31T21:35:31.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.872530+0000) 2022-01-31T21:35:32.318 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:32 smithi167 conmon[60316]: debug 2022-01-31T21:35:32.262+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:32.263912+0000) 2022-01-31T21:35:32.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:32 smithi167 conmon[54076]: debug 2022-01-31T21:35:32.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:32.263932+0000) 2022-01-31T21:35:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:32 smithi171 conmon[46715]: debug 2022-01-31T21:35:32.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:32.037491+0000) 2022-01-31T21:35:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:31 smithi171 conmon[41853]: debug 2022-01-31T21:35:31.993+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:31.993740+0000) 2022-01-31T21:35:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:32 smithi167 conmon[49112]: debug 2022-01-31T21:35:32.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:32.600161+0000) 2022-01-31T21:35:32.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:32 smithi171 conmon[51620]: debug 2022-01-31T21:35:32.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:32.872714+0000) 2022-01-31T21:35:33.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:33 smithi167 conmon[54076]: debug 2022-01-31T21:35:33.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:33.264083+0000) 2022-01-31T21:35:33.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:33 smithi167 conmon[60316]: debug 2022-01-31T21:35:33.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:33.264069+0000) 2022-01-31T21:35:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:32 smithi171 conmon[41853]: debug 2022-01-31T21:35:32.993+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:32.993943+0000) 2022-01-31T21:35:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:33 smithi171 conmon[46715]: debug 2022-01-31T21:35:33.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:33.037689+0000) 2022-01-31T21:35:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:33 smithi167 conmon[49112]: debug 2022-01-31T21:35:33.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:33.600320+0000) 2022-01-31T21:35:33.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:33 smithi171 conmon[51620]: debug 2022-01-31T21:35:33.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:33.872886+0000) 2022-01-31T21:35:34.318 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:34 smithi167 conmon[60316]: debug 2022-01-31T21:35:34.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:34.264282+0000) 2022-01-31T21:35:34.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:34 smithi167 conmon[54076]: debug 2022-01-31T21:35:34.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:34.264278+0000) 2022-01-31T21:35:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:33 smithi171 conmon[41853]: debug 2022-01-31T21:35:33.993+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:33.994095+0000) 2022-01-31T21:35:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:34 smithi171 conmon[46715]: debug 2022-01-31T21:35:34.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:34.037869+0000) 2022-01-31T21:35:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:34 smithi167 conmon[49112]: debug 2022-01-31T21:35:34.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:34.600529+0000) 2022-01-31T21:35:34.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:34 smithi171 conmon[51620]: debug 2022-01-31T21:35:34.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:34.873080+0000) 2022-01-31T21:35:35.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:35 smithi167 conmon[54076]: debug 2022-01-31T21:35:35.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:35.264539+0000) 2022-01-31T21:35:35.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:35 smithi167 conmon[60316]: debug 2022-01-31T21:35:35.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:35.264480+0000) 2022-01-31T21:35:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:34 smithi171 conmon[41853]: debug 2022-01-31T21:35:34.993+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:34.994247+0000) 2022-01-31T21:35:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:35 smithi171 conmon[46715]: debug 2022-01-31T21:35:35.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:35.038024+0000) 2022-01-31T21:35:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:35 smithi167 conmon[49112]: debug 2022-01-31T21:35:35.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:35.600651+0000) 2022-01-31T21:35:36.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:36 smithi167 conmon[54076]: debug 2022-01-31T21:35:36.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.264744+0000) 2022-01-31T21:35:36.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:36 smithi167 conmon[60316]: debug 2022-01-31T21:35:36.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.264704+0000) 2022-01-31T21:35:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:35 smithi171 conmon[41853]: debug 2022-01-31T21:35:35.994+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:35.994426+0000) 2022-01-31T21:35:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[46715]: debug 2022-01-31T21:35:36.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.038217+0000) 2022-01-31T21:35:36.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:35 smithi171 conmon[51620]: debug 2022-01-31T21:35:35.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:35.873268+0000) 2022-01-31T21:35:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:36 smithi167 conmon[49112]: debug 2022-01-31T21:35:36.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.600799+0000) 2022-01-31T21:35:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:36 smithi167 conmon[49112]: debug 2022-01-31T21:35:36.672+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.674200+0000) 2022-01-31T21:35:36.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:36 smithi167 conmon[54076]: debug 2022-01-31T21:35:36.673+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.675123+0000) 2022-01-31T21:35:36.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:36 smithi167 conmon[60316]: debug 2022-01-31T21:35:36.674+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.676178+0000) 2022-01-31T21:35:36.948 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[35325]: debug 2022-01-31T21:35:36.689+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142030 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:36.949 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[46715]: debug 2022-01-31T21:35:36.674+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.674930+0000) 2022-01-31T21:35:36.950 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[41853]: debug 2022-01-31T21:35:36.674+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.675566+0000) 2022-01-31T21:35:36.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[51620]: debug 2022-01-31T21:35:36.675+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.675898+0000) 2022-01-31T21:35:36.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[51620]: debug 2022-01-31T21:35:36.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.873459+0000) 2022-01-31T21:35:37.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:36 smithi171 conmon[41853]: debug 2022-01-31T21:35:36.994+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:36.994630+0000) 2022-01-31T21:35:37.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:37 smithi171 conmon[46715]: debug 2022-01-31T21:35:37.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:37.038402+0000) 2022-01-31T21:35:37.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:37 smithi167 conmon[54076]: debug 2022-01-31T21:35:37.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:37.264902+0000) 2022-01-31T21:35:37.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:37 smithi167 conmon[60316]: debug 2022-01-31T21:35:37.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:37.264872+0000) 2022-01-31T21:35:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:37 smithi167 conmon[49112]: debug 2022-01-31T21:35:37.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:37.600997+0000) 2022-01-31T21:35:37.948 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:37 smithi171 conmon[51620]: debug 2022-01-31T21:35:37.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:37.873641+0000) 2022-01-31T21:35:38.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:38 smithi167 conmon[54076]: debug 2022-01-31T21:35:38.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:38.265028+0000) 2022-01-31T21:35:38.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:38 smithi167 conmon[60316]: debug 2022-01-31T21:35:38.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:38.265034+0000) 2022-01-31T21:35:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:37 smithi171 conmon[41853]: debug 2022-01-31T21:35:37.994+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:37.994778+0000) 2022-01-31T21:35:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:38 smithi171 conmon[46715]: debug 2022-01-31T21:35:38.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:38.038550+0000) 2022-01-31T21:35:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:38 smithi167 conmon[49112]: debug 2022-01-31T21:35:38.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:38.601196+0000) 2022-01-31T21:35:38.949 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:38 smithi171 conmon[35325]: debug 2022-01-31T21:35:38.705+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:35:38.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:38 smithi171 conmon[51620]: debug 2022-01-31T21:35:38.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:38.873864+0000) 2022-01-31T21:35:39.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:39 smithi167 conmon[54076]: debug 2022-01-31T21:35:39.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:39.265220+0000) 2022-01-31T21:35:39.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:39 smithi167 conmon[60316]: debug 2022-01-31T21:35:39.263+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:39.265244+0000) 2022-01-31T21:35:39.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:39 smithi171 conmon[46715]: debug 2022-01-31T21:35:39.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:39.038713+0000) 2022-01-31T21:35:39.327 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:38 smithi171 conmon[41853]: debug 2022-01-31T21:35:38.994+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:38.994940+0000) 2022-01-31T21:35:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:39 smithi167 conmon[49112]: debug 2022-01-31T21:35:39.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:39.601389+0000) 2022-01-31T21:35:39.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:39 smithi171 conmon[51620]: debug 2022-01-31T21:35:39.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:39.874041+0000) 2022-01-31T21:35:40.315 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:40 smithi167 conmon[60316]: debug 2022-01-31T21:35:40.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:40.265426+0000) 2022-01-31T21:35:40.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:40 smithi167 conmon[54076]: debug 2022-01-31T21:35:40.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:40.265423+0000) 2022-01-31T21:35:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:39 smithi171 conmon[41853]: debug 2022-01-31T21:35:39.994+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:39.995153+0000) 2022-01-31T21:35:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:40 smithi171 conmon[46715]: debug 2022-01-31T21:35:40.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:40.038871+0000) 2022-01-31T21:35:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:40 smithi167 conmon[49112]: debug 2022-01-31T21:35:40.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:40.601615+0000) 2022-01-31T21:35:40.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:40 smithi171 conmon[51620]: debug 2022-01-31T21:35:40.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:40.874229+0000) 2022-01-31T21:35:41.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:41 smithi167 conmon[54076]: debug 2022-01-31T21:35:41.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.265660+0000) 2022-01-31T21:35:41.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:41 smithi167 conmon[60316]: debug 2022-01-31T21:35:41.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.265664+0000) 2022-01-31T21:35:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:40 smithi171 conmon[41853]: debug 2022-01-31T21:35:40.995+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:40.995355+0000) 2022-01-31T21:35:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[46715]: debug 2022-01-31T21:35:41.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.039048+0000) 2022-01-31T21:35:41.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:41 smithi167 conmon[54076]: debug 2022-01-31T21:35:41.690+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.692295+0000) 2022-01-31T21:35:41.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:41 smithi167 conmon[60316]: debug 2022-01-31T21:35:41.691+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.692761+0000) 2022-01-31T21:35:41.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:41 smithi167 conmon[49112]: debug 2022-01-31T21:35:41.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.601767+0000) 2022-01-31T21:35:41.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:41 smithi167 conmon[49112]: debug 2022-01-31T21:35:41.691+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.692580+0000) 2022-01-31T21:35:41.949 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[46715]: debug 2022-01-31T21:35:41.691+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.692405+0000) 2022-01-31T21:35:41.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[51620]: debug 2022-01-31T21:35:41.691+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.692279+0000) 2022-01-31T21:35:41.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[51620]: debug 2022-01-31T21:35:41.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.874336+0000) 2022-01-31T21:35:41.951 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[35325]: debug 2022-01-31T21:35:41.706+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142136 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:41.952 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[41853]: debug 2022-01-31T21:35:41.692+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.692834+0000) 2022-01-31T21:35:42.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:42 smithi167 conmon[54076]: debug 2022-01-31T21:35:42.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:42.265841+0000) 2022-01-31T21:35:42.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:42 smithi167 conmon[60316]: debug 2022-01-31T21:35:42.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:42.265842+0000) 2022-01-31T21:35:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:41 smithi171 conmon[41853]: debug 2022-01-31T21:35:41.994+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:41.995520+0000) 2022-01-31T21:35:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:42 smithi171 conmon[46715]: debug 2022-01-31T21:35:42.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:42.039211+0000) 2022-01-31T21:35:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:42 smithi167 conmon[49112]: debug 2022-01-31T21:35:42.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:42.601953+0000) 2022-01-31T21:35:42.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:42 smithi171 conmon[51620]: debug 2022-01-31T21:35:42.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:42.874486+0000) 2022-01-31T21:35:43.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:43 smithi167 conmon[54076]: debug 2022-01-31T21:35:43.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:43.266000+0000) 2022-01-31T21:35:43.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:43 smithi167 conmon[60316]: debug 2022-01-31T21:35:43.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:43.265999+0000) 2022-01-31T21:35:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:42 smithi171 conmon[41853]: debug 2022-01-31T21:35:42.995+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:42.995660+0000) 2022-01-31T21:35:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:43 smithi171 conmon[46715]: debug 2022-01-31T21:35:43.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:43.039358+0000) 2022-01-31T21:35:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:43 smithi167 conmon[49112]: debug 2022-01-31T21:35:43.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:43.602084+0000) 2022-01-31T21:35:43.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:43 smithi171 conmon[51620]: debug 2022-01-31T21:35:43.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:43.874701+0000) 2022-01-31T21:35:44.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:44 smithi167 conmon[54076]: debug 2022-01-31T21:35:44.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:44.266206+0000) 2022-01-31T21:35:44.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:44 smithi167 conmon[60316]: debug 2022-01-31T21:35:44.264+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:44.266226+0000) 2022-01-31T21:35:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:43 smithi171 conmon[41853]: debug 2022-01-31T21:35:43.995+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:43.995811+0000) 2022-01-31T21:35:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:44 smithi171 conmon[46715]: debug 2022-01-31T21:35:44.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:44.039536+0000) 2022-01-31T21:35:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:44 smithi167 conmon[49112]: debug 2022-01-31T21:35:44.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:44.602288+0000) 2022-01-31T21:35:44.949 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:44 smithi171 conmon[51620]: debug 2022-01-31T21:35:44.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:44.874915+0000) 2022-01-31T21:35:45.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:45 smithi167 conmon[54076]: debug 2022-01-31T21:35:45.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:45.266372+0000) 2022-01-31T21:35:45.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:45 smithi167 conmon[60316]: debug 2022-01-31T21:35:45.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:45.266362+0000) 2022-01-31T21:35:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:45 smithi171 conmon[46715]: debug 2022-01-31T21:35:45.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:45.039724+0000) 2022-01-31T21:35:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:44 smithi171 conmon[41853]: debug 2022-01-31T21:35:44.995+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:44.995997+0000) 2022-01-31T21:35:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:45 smithi167 conmon[49112]: debug 2022-01-31T21:35:45.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:45.602530+0000) 2022-01-31T21:35:45.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:45 smithi171 conmon[51620]: debug 2022-01-31T21:35:45.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:45.875126+0000) 2022-01-31T21:35:46.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:46 smithi167 conmon[54076]: debug 2022-01-31T21:35:46.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.266549+0000) 2022-01-31T21:35:46.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:46 smithi167 conmon[60316]: debug 2022-01-31T21:35:46.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.266515+0000) 2022-01-31T21:35:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:45 smithi171 conmon[41853]: debug 2022-01-31T21:35:45.995+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:45.996191+0000) 2022-01-31T21:35:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[46715]: debug 2022-01-31T21:35:46.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.039885+0000) 2022-01-31T21:35:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:46 smithi167 conmon[49112]: debug 2022-01-31T21:35:46.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.602660+0000) 2022-01-31T21:35:46.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:46 smithi167 conmon[49112]: debug 2022-01-31T21:35:46.707+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.708999+0000) 2022-01-31T21:35:46.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:46 smithi167 conmon[54076]: debug 2022-01-31T21:35:46.708+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.709764+0000) 2022-01-31T21:35:46.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:46 smithi167 conmon[60316]: debug 2022-01-31T21:35:46.708+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.710285+0000) 2022-01-31T21:35:46.950 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[35325]: debug 2022-01-31T21:35:46.723+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142247 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:46.950 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[41853]: debug 2022-01-31T21:35:46.709+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.710177+0000) 2022-01-31T21:35:46.951 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[46715]: debug 2022-01-31T21:35:46.708+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.709085+0000) 2022-01-31T21:35:46.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[51620]: debug 2022-01-31T21:35:46.709+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.710476+0000) 2022-01-31T21:35:46.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[51620]: debug 2022-01-31T21:35:46.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.875285+0000) 2022-01-31T21:35:47.320 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:47 smithi167 conmon[60316]: debug 2022-01-31T21:35:47.265+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:47.266695+0000) 2022-01-31T21:35:47.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:47 smithi167 conmon[54076]: debug 2022-01-31T21:35:47.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:47.266720+0000) 2022-01-31T21:35:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:46 smithi171 conmon[41853]: debug 2022-01-31T21:35:46.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:46.996354+0000) 2022-01-31T21:35:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:47 smithi171 conmon[46715]: debug 2022-01-31T21:35:47.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:47.040097+0000) 2022-01-31T21:35:47.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:47 smithi167 conmon[49112]: debug 2022-01-31T21:35:47.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:47.602872+0000) 2022-01-31T21:35:47.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:47 smithi171 conmon[51620]: debug 2022-01-31T21:35:47.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:47.875430+0000) 2022-01-31T21:35:48.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:48 smithi167 conmon[60316]: debug 2022-01-31T21:35:48.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:48.266844+0000) 2022-01-31T21:35:48.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:48 smithi167 conmon[54076]: debug 2022-01-31T21:35:48.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:48.266872+0000) 2022-01-31T21:35:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:47 smithi171 conmon[41853]: debug 2022-01-31T21:35:47.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:47.996504+0000) 2022-01-31T21:35:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:48 smithi171 conmon[46715]: debug 2022-01-31T21:35:48.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:48.040229+0000) 2022-01-31T21:35:48.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:48 smithi167 conmon[49112]: debug 2022-01-31T21:35:48.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:48.603065+0000) 2022-01-31T21:35:48.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:48 smithi171 conmon[51620]: debug 2022-01-31T21:35:48.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:48.875614+0000) 2022-01-31T21:35:49.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:49 smithi167 conmon[54076]: debug 2022-01-31T21:35:49.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:49.267067+0000) 2022-01-31T21:35:49.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:49 smithi167 conmon[60316]: debug 2022-01-31T21:35:49.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:49.267051+0000) 2022-01-31T21:35:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:48 smithi171 conmon[41853]: debug 2022-01-31T21:35:48.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:48.996662+0000) 2022-01-31T21:35:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:49 smithi171 conmon[46715]: debug 2022-01-31T21:35:49.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:49.040425+0000) 2022-01-31T21:35:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:49 smithi167 conmon[49112]: debug 2022-01-31T21:35:49.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:49.603260+0000) 2022-01-31T21:35:49.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:49 smithi171 conmon[51620]: debug 2022-01-31T21:35:49.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:49.875789+0000) 2022-01-31T21:35:50.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:50 smithi167 conmon[54076]: debug 2022-01-31T21:35:50.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:50.267282+0000) 2022-01-31T21:35:50.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:50 smithi167 conmon[60316]: debug 2022-01-31T21:35:50.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:50.267228+0000) 2022-01-31T21:35:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:50 smithi171 conmon[46715]: debug 2022-01-31T21:35:50.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:50.040636+0000) 2022-01-31T21:35:50.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:49 smithi171 conmon[41853]: debug 2022-01-31T21:35:49.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:49.996843+0000) 2022-01-31T21:35:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:50 smithi167 conmon[49112]: debug 2022-01-31T21:35:50.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:50.603474+0000) 2022-01-31T21:35:50.950 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:50 smithi171 conmon[51620]: debug 2022-01-31T21:35:50.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:50.875999+0000) 2022-01-31T21:35:51.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:51 smithi167 conmon[54076]: debug 2022-01-31T21:35:51.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.267478+0000) 2022-01-31T21:35:51.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:51 smithi167 conmon[60316]: debug 2022-01-31T21:35:51.266+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.267425+0000) 2022-01-31T21:35:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:50 smithi171 conmon[41853]: debug 2022-01-31T21:35:50.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:50.996958+0000) 2022-01-31T21:35:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[46715]: debug 2022-01-31T21:35:51.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.040788+0000) 2022-01-31T21:35:51.873 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[41853]: debug 2022-01-31T21:35:51.726+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.727055+0000) 2022-01-31T21:35:51.874 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[46715]: debug 2022-01-31T21:35:51.725+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.726370+0000) 2022-01-31T21:35:51.875 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[51620]: debug 2022-01-31T21:35:51.726+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.727351+0000) 2022-01-31T21:35:51.875 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[35325]: debug 2022-01-31T21:35:51.742+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142357 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:51.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:51 smithi167 conmon[54076]: debug 2022-01-31T21:35:51.725+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.726679+0000) 2022-01-31T21:35:51.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:51 smithi167 conmon[60316]: debug 2022-01-31T21:35:51.725+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.726571+0000) 2022-01-31T21:35:51.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:51 smithi167 conmon[49112]: debug 2022-01-31T21:35:51.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.603686+0000) 2022-01-31T21:35:51.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:51 smithi167 conmon[49112]: debug 2022-01-31T21:35:51.724+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.726354+0000) 2022-01-31T21:35:52.321 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:52 smithi167 conmon[54076]: debug 2022-01-31T21:35:52.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:52.267656+0000) 2022-01-31T21:35:52.322 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:52 smithi167 conmon[60316]: debug 2022-01-31T21:35:52.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:52.267606+0000) 2022-01-31T21:35:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[41853]: debug 2022-01-31T21:35:51.997+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.997164+0000) 2022-01-31T21:35:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:52 smithi171 conmon[46715]: debug 2022-01-31T21:35:52.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:52.041005+0000) 2022-01-31T21:35:52.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:51 smithi171 conmon[51620]: debug 2022-01-31T21:35:51.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:51.876182+0000) 2022-01-31T21:35:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:52 smithi167 conmon[49112]: debug 2022-01-31T21:35:52.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:52.603842+0000) 2022-01-31T21:35:52.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:52 smithi171 conmon[51620]: debug 2022-01-31T21:35:52.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:52.876345+0000) 2022-01-31T21:35:53.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:53 smithi167 conmon[60316]: debug 2022-01-31T21:35:53.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:53.267768+0000) 2022-01-31T21:35:53.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:53 smithi167 conmon[54076]: debug 2022-01-31T21:35:53.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:53.267791+0000) 2022-01-31T21:35:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:52 smithi171 conmon[41853]: debug 2022-01-31T21:35:52.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:52.997359+0000) 2022-01-31T21:35:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:53 smithi171 conmon[46715]: debug 2022-01-31T21:35:53.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:53.041207+0000) 2022-01-31T21:35:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:53 smithi167 conmon[49112]: debug 2022-01-31T21:35:53.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:53.604038+0000) 2022-01-31T21:35:53.951 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:53 smithi171 conmon[35325]: debug 2022-01-31T21:35:53.705+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:35:53.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:53 smithi171 conmon[51620]: debug 2022-01-31T21:35:53.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:53.876540+0000) 2022-01-31T21:35:54.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:54 smithi167 conmon[54076]: debug 2022-01-31T21:35:54.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:54.267985+0000) 2022-01-31T21:35:54.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:54 smithi167 conmon[60316]: debug 2022-01-31T21:35:54.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:54.267985+0000) 2022-01-31T21:35:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:53 smithi171 conmon[41853]: debug 2022-01-31T21:35:53.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:53.997562+0000) 2022-01-31T21:35:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:54 smithi171 conmon[46715]: debug 2022-01-31T21:35:54.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:54.041416+0000) 2022-01-31T21:35:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:54 smithi167 conmon[49112]: debug 2022-01-31T21:35:54.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:54.604267+0000) 2022-01-31T21:35:54.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:54 smithi171 conmon[51620]: debug 2022-01-31T21:35:54.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:54.876694+0000) 2022-01-31T21:35:55.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:55 smithi167 conmon[54076]: debug 2022-01-31T21:35:55.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:55.268193+0000) 2022-01-31T21:35:55.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:55 smithi167 conmon[60316]: debug 2022-01-31T21:35:55.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:55.268194+0000) 2022-01-31T21:35:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:54 smithi171 conmon[41853]: debug 2022-01-31T21:35:54.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:54.997747+0000) 2022-01-31T21:35:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:55 smithi171 conmon[46715]: debug 2022-01-31T21:35:55.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:55.041574+0000) 2022-01-31T21:35:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:55 smithi167 conmon[49112]: debug 2022-01-31T21:35:55.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:55.604495+0000) 2022-01-31T21:35:55.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:55 smithi171 conmon[51620]: debug 2022-01-31T21:35:55.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:55.876909+0000) 2022-01-31T21:35:56.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:56 smithi167 conmon[54076]: debug 2022-01-31T21:35:56.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.268375+0000) 2022-01-31T21:35:56.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:56 smithi167 conmon[60316]: debug 2022-01-31T21:35:56.267+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.268401+0000) 2022-01-31T21:35:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:55 smithi171 conmon[41853]: debug 2022-01-31T21:35:55.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:55.997947+0000) 2022-01-31T21:35:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[46715]: debug 2022-01-31T21:35:56.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.041727+0000) 2022-01-31T21:35:56.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:56 smithi167 conmon[54076]: debug 2022-01-31T21:35:56.746+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.746778+0000) 2022-01-31T21:35:56.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:56 smithi167 conmon[60316]: debug 2022-01-31T21:35:56.743+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.744179+0000) 2022-01-31T21:35:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:56 smithi167 conmon[49112]: debug 2022-01-31T21:35:56.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.604635+0000) 2022-01-31T21:35:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:56 smithi167 conmon[49112]: debug 2022-01-31T21:35:56.743+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.743706+0000) 2022-01-31T21:35:56.951 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[35325]: debug 2022-01-31T21:35:56.759+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142469 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:35:56.952 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[41853]: debug 2022-01-31T21:35:56.744+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.746137+0000) 2022-01-31T21:35:56.953 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[46715]: debug 2022-01-31T21:35:56.744+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.746386+0000) 2022-01-31T21:35:56.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[51620]: debug 2022-01-31T21:35:56.742+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.744642+0000) 2022-01-31T21:35:56.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[51620]: debug 2022-01-31T21:35:56.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.877065+0000) 2022-01-31T21:35:57.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:57 smithi167 conmon[54076]: debug 2022-01-31T21:35:57.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:57.268584+0000) 2022-01-31T21:35:57.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:57 smithi167 conmon[60316]: debug 2022-01-31T21:35:57.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:57.268602+0000) 2022-01-31T21:35:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:56 smithi171 conmon[41853]: debug 2022-01-31T21:35:56.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:56.998093+0000) 2022-01-31T21:35:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:57 smithi171 conmon[46715]: debug 2022-01-31T21:35:57.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:57.041897+0000) 2022-01-31T21:35:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:57 smithi167 conmon[49112]: debug 2022-01-31T21:35:57.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:57.604778+0000) 2022-01-31T21:35:57.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:57 smithi171 conmon[51620]: debug 2022-01-31T21:35:57.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:57.877247+0000) 2022-01-31T21:35:58.322 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:58 smithi167 conmon[54076]: debug 2022-01-31T21:35:58.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:58.268756+0000) 2022-01-31T21:35:58.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:58 smithi167 conmon[60316]: debug 2022-01-31T21:35:58.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:58.268763+0000) 2022-01-31T21:35:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:57 smithi171 conmon[41853]: debug 2022-01-31T21:35:57.997+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:57.998229+0000) 2022-01-31T21:35:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:58 smithi171 conmon[46715]: debug 2022-01-31T21:35:58.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:58.042050+0000) 2022-01-31T21:35:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:58 smithi167 conmon[49112]: debug 2022-01-31T21:35:58.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:58.604909+0000) 2022-01-31T21:35:58.951 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:58 smithi171 conmon[51620]: debug 2022-01-31T21:35:58.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:58.877460+0000) 2022-01-31T21:35:59.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:35:59 smithi167 conmon[54076]: debug 2022-01-31T21:35:59.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:59.268944+0000) 2022-01-31T21:35:59.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:35:59 smithi167 conmon[60316]: debug 2022-01-31T21:35:59.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:59.268950+0000) 2022-01-31T21:35:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:58 smithi171 conmon[41853]: debug 2022-01-31T21:35:58.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:58.998405+0000) 2022-01-31T21:35:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:35:59 smithi171 conmon[46715]: debug 2022-01-31T21:35:59.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:59.042232+0000) 2022-01-31T21:35:59.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:35:59 smithi167 conmon[49112]: debug 2022-01-31T21:35:59.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:59.605099+0000) 2022-01-31T21:35:59.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:35:59 smithi171 conmon[51620]: debug 2022-01-31T21:35:59.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:59.877650+0000) 2022-01-31T21:36:00.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:00 smithi167 conmon[60316]: debug 2022-01-31T21:36:00.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:00.269136+0000) 2022-01-31T21:36:00.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:00 smithi167 conmon[54076]: debug 2022-01-31T21:36:00.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:00.269141+0000) 2022-01-31T21:36:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:35:59 smithi171 conmon[41853]: debug 2022-01-31T21:35:59.996+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:35:59.998592+0000) 2022-01-31T21:36:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:00 smithi171 conmon[46715]: debug 2022-01-31T21:36:00.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:00.042415+0000) 2022-01-31T21:36:00.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:00 smithi167 conmon[49112]: debug 2022-01-31T21:36:00.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:00.605322+0000) 2022-01-31T21:36:00.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:00 smithi171 conmon[51620]: debug 2022-01-31T21:36:00.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:00.877846+0000) 2022-01-31T21:36:01.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:01 smithi167 conmon[60316]: debug 2022-01-31T21:36:01.268+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.269331+0000) 2022-01-31T21:36:01.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:01 smithi167 conmon[54076]: debug 2022-01-31T21:36:01.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.269331+0000) 2022-01-31T21:36:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:00 smithi171 conmon[41853]: debug 2022-01-31T21:36:00.997+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:00.998742+0000) 2022-01-31T21:36:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[46715]: debug 2022-01-31T21:36:01.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.042637+0000) 2022-01-31T21:36:01.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:01 smithi167 conmon[49112]: debug 2022-01-31T21:36:01.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.605531+0000) 2022-01-31T21:36:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:01 smithi167 conmon[49112]: debug 2022-01-31T21:36:01.763+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.763726+0000) 2022-01-31T21:36:01.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:01 smithi167 conmon[54076]: debug 2022-01-31T21:36:01.763+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.763626+0000) 2022-01-31T21:36:01.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:01 smithi167 conmon[60316]: debug 2022-01-31T21:36:01.762+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.763261+0000) 2022-01-31T21:36:01.952 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[35325]: debug 2022-01-31T21:36:01.776+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142579 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:01.953 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[41853]: debug 2022-01-31T21:36:01.761+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.763522+0000) 2022-01-31T21:36:01.953 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[46715]: debug 2022-01-31T21:36:01.762+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.764356+0000) 2022-01-31T21:36:01.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[51620]: debug 2022-01-31T21:36:01.761+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.763611+0000) 2022-01-31T21:36:01.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[51620]: debug 2022-01-31T21:36:01.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.878052+0000) 2022-01-31T21:36:02.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:02 smithi167 conmon[54076]: debug 2022-01-31T21:36:02.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:02.269546+0000) 2022-01-31T21:36:02.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:02 smithi167 conmon[60316]: debug 2022-01-31T21:36:02.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:02.269551+0000) 2022-01-31T21:36:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:01 smithi171 conmon[41853]: debug 2022-01-31T21:36:01.997+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:01.998887+0000) 2022-01-31T21:36:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:02 smithi171 conmon[46715]: debug 2022-01-31T21:36:02.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:02.042839+0000) 2022-01-31T21:36:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:02 smithi167 conmon[49112]: debug 2022-01-31T21:36:02.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:02.605685+0000) 2022-01-31T21:36:02.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:02 smithi171 conmon[51620]: debug 2022-01-31T21:36:02.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:02.878168+0000) 2022-01-31T21:36:03.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:03 smithi167 conmon[54076]: debug 2022-01-31T21:36:03.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:03.269712+0000) 2022-01-31T21:36:03.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:03 smithi167 conmon[60316]: debug 2022-01-31T21:36:03.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:03.269705+0000) 2022-01-31T21:36:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:03 smithi171 conmon[46715]: debug 2022-01-31T21:36:03.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:03.043018+0000) 2022-01-31T21:36:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:02 smithi171 conmon[41853]: debug 2022-01-31T21:36:02.997+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:02.998991+0000) 2022-01-31T21:36:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:03 smithi167 conmon[49112]: debug 2022-01-31T21:36:03.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:03.605902+0000) 2022-01-31T21:36:03.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:03 smithi171 conmon[51620]: debug 2022-01-31T21:36:03.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:03.878352+0000) 2022-01-31T21:36:04.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:04 smithi167 conmon[54076]: debug 2022-01-31T21:36:04.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:04.269912+0000) 2022-01-31T21:36:04.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:04 smithi167 conmon[60316]: debug 2022-01-31T21:36:04.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:04.269832+0000) 2022-01-31T21:36:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:04 smithi171 conmon[46715]: debug 2022-01-31T21:36:04.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:04.043191+0000) 2022-01-31T21:36:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:03 smithi171 conmon[41853]: debug 2022-01-31T21:36:03.998+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:03.999134+0000) 2022-01-31T21:36:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:04 smithi167 conmon[49112]: debug 2022-01-31T21:36:04.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:04.606067+0000) 2022-01-31T21:36:04.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:04 smithi171 conmon[51620]: debug 2022-01-31T21:36:04.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:04.878517+0000) 2022-01-31T21:36:05.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:05 smithi167 conmon[54076]: debug 2022-01-31T21:36:05.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:05.270039+0000) 2022-01-31T21:36:05.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:05 smithi167 conmon[60316]: debug 2022-01-31T21:36:05.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:05.269947+0000) 2022-01-31T21:36:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:05 smithi171 conmon[46715]: debug 2022-01-31T21:36:05.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:05.043357+0000) 2022-01-31T21:36:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:04 smithi171 conmon[41853]: debug 2022-01-31T21:36:04.998+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:04.999285+0000) 2022-01-31T21:36:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:05 smithi167 conmon[49112]: debug 2022-01-31T21:36:05.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:05.606253+0000) 2022-01-31T21:36:05.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:05 smithi171 conmon[51620]: debug 2022-01-31T21:36:05.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:05.878696+0000) 2022-01-31T21:36:06.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:06 smithi167 conmon[54076]: debug 2022-01-31T21:36:06.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.270164+0000) 2022-01-31T21:36:06.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:06 smithi167 conmon[60316]: debug 2022-01-31T21:36:06.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.270104+0000) 2022-01-31T21:36:06.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[46715]: debug 2022-01-31T21:36:06.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.043485+0000) 2022-01-31T21:36:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:05 smithi171 conmon[41853]: debug 2022-01-31T21:36:05.998+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:05.999444+0000) 2022-01-31T21:36:06.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:06 smithi167 conmon[60316]: debug 2022-01-31T21:36:06.780+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.780920+0000) 2022-01-31T21:36:06.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:06 smithi167 conmon[49112]: debug 2022-01-31T21:36:06.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.606453+0000) 2022-01-31T21:36:06.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:06 smithi167 conmon[49112]: debug 2022-01-31T21:36:06.780+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.780699+0000) 2022-01-31T21:36:06.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:06 smithi167 conmon[54076]: debug 2022-01-31T21:36:06.779+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.780090+0000) 2022-01-31T21:36:06.953 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[35325]: debug 2022-01-31T21:36:06.799+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142690 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:06.954 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[46715]: debug 2022-01-31T21:36:06.779+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.780475+0000) 2022-01-31T21:36:06.955 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[41853]: debug 2022-01-31T21:36:06.779+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.780576+0000) 2022-01-31T21:36:06.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[51620]: debug 2022-01-31T21:36:06.780+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.781686+0000) 2022-01-31T21:36:06.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[51620]: debug 2022-01-31T21:36:06.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.878858+0000) 2022-01-31T21:36:07.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:07 smithi167 conmon[54076]: debug 2022-01-31T21:36:07.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:07.270274+0000) 2022-01-31T21:36:07.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:07 smithi167 conmon[60316]: debug 2022-01-31T21:36:07.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:07.270241+0000) 2022-01-31T21:36:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:07 smithi171 conmon[46715]: debug 2022-01-31T21:36:07.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:07.043643+0000) 2022-01-31T21:36:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:06 smithi171 conmon[41853]: debug 2022-01-31T21:36:06.998+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:06.999648+0000) 2022-01-31T21:36:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:07 smithi167 conmon[49112]: debug 2022-01-31T21:36:07.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:07.606610+0000) 2022-01-31T21:36:08.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:07 smithi171 conmon[51620]: debug 2022-01-31T21:36:07.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:07.879010+0000) 2022-01-31T21:36:08.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:08 smithi167 conmon[60316]: debug 2022-01-31T21:36:08.269+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:08.270413+0000) 2022-01-31T21:36:08.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:08 smithi167 conmon[54076]: debug 2022-01-31T21:36:08.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:08.270464+0000) 2022-01-31T21:36:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:07 smithi171 conmon[41853]: debug 2022-01-31T21:36:07.999+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:07.999799+0000) 2022-01-31T21:36:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:08 smithi171 conmon[46715]: debug 2022-01-31T21:36:08.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:08.043784+0000) 2022-01-31T21:36:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:08 smithi167 conmon[49112]: debug 2022-01-31T21:36:08.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:08.606781+0000) 2022-01-31T21:36:08.954 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:08 smithi171 conmon[35325]: debug 2022-01-31T21:36:08.706+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:36:08.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:08 smithi171 conmon[51620]: debug 2022-01-31T21:36:08.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:08.879219+0000) 2022-01-31T21:36:09.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:09 smithi167 conmon[54076]: debug 2022-01-31T21:36:09.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:09.270595+0000) 2022-01-31T21:36:09.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:09 smithi167 conmon[60316]: debug 2022-01-31T21:36:09.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:09.270704+0000) 2022-01-31T21:36:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:09 smithi171 conmon[46715]: debug 2022-01-31T21:36:09.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:09.043964+0000) 2022-01-31T21:36:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:09 smithi171 conmon[41853]: debug 2022-01-31T21:36:08.999+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:08.999981+0000) 2022-01-31T21:36:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:09 smithi167 conmon[49112]: debug 2022-01-31T21:36:09.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:09.606979+0000) 2022-01-31T21:36:09.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:09 smithi171 conmon[51620]: debug 2022-01-31T21:36:09.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:09.879399+0000) 2022-01-31T21:36:10.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:10 smithi167 conmon[54076]: debug 2022-01-31T21:36:10.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:10.270792+0000) 2022-01-31T21:36:10.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:10 smithi167 conmon[60316]: debug 2022-01-31T21:36:10.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:10.270886+0000) 2022-01-31T21:36:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:10 smithi171 conmon[46715]: debug 2022-01-31T21:36:10.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:10.044117+0000) 2022-01-31T21:36:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:10 smithi171 conmon[41853]: debug 2022-01-31T21:36:09.999+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:10.000151+0000) 2022-01-31T21:36:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:10 smithi167 conmon[49112]: debug 2022-01-31T21:36:10.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:10.607199+0000) 2022-01-31T21:36:10.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:10 smithi171 conmon[51620]: debug 2022-01-31T21:36:10.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:10.879589+0000) 2022-01-31T21:36:11.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[54076]: debug 2022-01-31T21:36:11.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.270982+0000) 2022-01-31T21:36:11.325 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[60316]: debug 2022-01-31T21:36:11.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.271079+0000) 2022-01-31T21:36:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[41853]: debug 2022-01-31T21:36:10.999+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.000354+0000) 2022-01-31T21:36:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[46715]: debug 2022-01-31T21:36:11.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.044247+0000) 2022-01-31T21:36:11.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[60316]: debug 2022-01-31T21:36:11.801+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.802416+0000) 2022-01-31T21:36:11.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[49112]: debug 2022-01-31T21:36:11.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.607403+0000) 2022-01-31T21:36:11.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[49112]: debug 2022-01-31T21:36:11.802+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.802767+0000) 2022-01-31T21:36:11.921 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[54076]: debug 2022-01-31T21:36:11.801+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.802045+0000) 2022-01-31T21:36:11.922 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:11 smithi167 conmon[54076]: 2022-01-31T21:36:11.954 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[35325]: debug 2022-01-31T21:36:11.817+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142800 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:11.955 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[41853]: debug 2022-01-31T21:36:11.801+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.802617+0000) 2022-01-31T21:36:11.956 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[46715]: debug 2022-01-31T21:36:11.802+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.802972+0000) 2022-01-31T21:36:11.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[51620]: debug 2022-01-31T21:36:11.801+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.802263+0000) 2022-01-31T21:36:11.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:11 smithi171 conmon[51620]: debug 2022-01-31T21:36:11.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:11.879758+0000) 2022-01-31T21:36:12.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:12 smithi167 conmon[54076]: debug 2022-01-31T21:36:12.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:12.271162+0000) 2022-01-31T21:36:12.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:12 smithi167 conmon[60316]: debug 2022-01-31T21:36:12.270+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:12.271292+0000) 2022-01-31T21:36:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:12 smithi171 conmon[41853]: debug 2022-01-31T21:36:11.999+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:12.000568+0000) 2022-01-31T21:36:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:12 smithi171 conmon[46715]: debug 2022-01-31T21:36:12.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:12.044406+0000) 2022-01-31T21:36:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:12 smithi167 conmon[49112]: debug 2022-01-31T21:36:12.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:12.607579+0000) 2022-01-31T21:36:12.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:12 smithi171 conmon[51620]: debug 2022-01-31T21:36:12.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:12.879866+0000) 2022-01-31T21:36:13.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:13 smithi167 conmon[54076]: debug 2022-01-31T21:36:13.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:13.271316+0000) 2022-01-31T21:36:13.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:13 smithi167 conmon[60316]: debug 2022-01-31T21:36:13.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:13.271425+0000) 2022-01-31T21:36:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:13 smithi171 conmon[41853]: debug 2022-01-31T21:36:13.000+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:13.000743+0000) 2022-01-31T21:36:13.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:13 smithi171 conmon[46715]: debug 2022-01-31T21:36:13.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:13.044606+0000) 2022-01-31T21:36:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:13 smithi167 conmon[49112]: debug 2022-01-31T21:36:13.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:13.607738+0000) 2022-01-31T21:36:13.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:13 smithi171 conmon[51620]: debug 2022-01-31T21:36:13.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:13.880052+0000) 2022-01-31T21:36:14.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:14 smithi167 conmon[54076]: debug 2022-01-31T21:36:14.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:14.271527+0000) 2022-01-31T21:36:14.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:14 smithi167 conmon[60316]: debug 2022-01-31T21:36:14.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:14.271579+0000) 2022-01-31T21:36:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:14 smithi171 conmon[41853]: debug 2022-01-31T21:36:14.000+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:14.000955+0000) 2022-01-31T21:36:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:14 smithi171 conmon[46715]: debug 2022-01-31T21:36:14.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:14.044815+0000) 2022-01-31T21:36:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:14 smithi167 conmon[49112]: debug 2022-01-31T21:36:14.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:14.607876+0000) 2022-01-31T21:36:14.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:14 smithi171 conmon[51620]: debug 2022-01-31T21:36:14.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:14.880254+0000) 2022-01-31T21:36:15.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:15 smithi167 conmon[54076]: debug 2022-01-31T21:36:15.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:15.271718+0000) 2022-01-31T21:36:15.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:15 smithi167 conmon[60316]: debug 2022-01-31T21:36:15.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:15.271749+0000) 2022-01-31T21:36:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:15 smithi171 conmon[41853]: debug 2022-01-31T21:36:15.000+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:15.001160+0000) 2022-01-31T21:36:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:15 smithi171 conmon[46715]: debug 2022-01-31T21:36:15.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:15.045002+0000) 2022-01-31T21:36:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:15 smithi167 conmon[49112]: debug 2022-01-31T21:36:15.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:15.608097+0000) 2022-01-31T21:36:15.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:15 smithi171 conmon[51620]: debug 2022-01-31T21:36:15.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:15.880459+0000) 2022-01-31T21:36:16.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:16 smithi167 conmon[54076]: debug 2022-01-31T21:36:16.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.271926+0000) 2022-01-31T21:36:16.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:16 smithi167 conmon[60316]: debug 2022-01-31T21:36:16.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.271946+0000) 2022-01-31T21:36:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[41853]: debug 2022-01-31T21:36:16.000+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.001303+0000) 2022-01-31T21:36:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[46715]: debug 2022-01-31T21:36:16.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.045161+0000) 2022-01-31T21:36:16.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:16 smithi167 conmon[54076]: debug 2022-01-31T21:36:16.819+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.820448+0000) 2022-01-31T21:36:16.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:16 smithi167 conmon[60316]: debug 2022-01-31T21:36:16.820+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.820811+0000) 2022-01-31T21:36:16.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:16 smithi167 conmon[49112]: debug 2022-01-31T21:36:16.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.608279+0000) 2022-01-31T21:36:16.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:16 smithi167 conmon[49112]: debug 2022-01-31T21:36:16.820+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.820977+0000) 2022-01-31T21:36:16.955 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[46715]: debug 2022-01-31T21:36:16.820+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.820749+0000) 2022-01-31T21:36:16.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[51620]: debug 2022-01-31T21:36:16.819+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.819864+0000) 2022-01-31T21:36:16.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[51620]: debug 2022-01-31T21:36:16.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.880691+0000) 2022-01-31T21:36:16.957 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[35325]: debug 2022-01-31T21:36:16.834+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 142911 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:16.957 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:16 smithi171 conmon[41853]: debug 2022-01-31T21:36:16.820+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:16.821236+0000) 2022-01-31T21:36:17.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:17 smithi167 conmon[54076]: debug 2022-01-31T21:36:17.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:17.272112+0000) 2022-01-31T21:36:17.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:17 smithi167 conmon[60316]: debug 2022-01-31T21:36:17.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:17.272118+0000) 2022-01-31T21:36:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:17 smithi171 conmon[41853]: debug 2022-01-31T21:36:17.000+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:17.001498+0000) 2022-01-31T21:36:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:17 smithi171 conmon[46715]: debug 2022-01-31T21:36:17.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:17.045322+0000) 2022-01-31T21:36:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:17 smithi167 conmon[49112]: debug 2022-01-31T21:36:17.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:17.608472+0000) 2022-01-31T21:36:17.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:17 smithi171 conmon[51620]: debug 2022-01-31T21:36:17.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:17.880838+0000) 2022-01-31T21:36:18.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:18 smithi167 conmon[54076]: debug 2022-01-31T21:36:18.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:18.272246+0000) 2022-01-31T21:36:18.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:18 smithi167 conmon[60316]: debug 2022-01-31T21:36:18.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:18.272282+0000) 2022-01-31T21:36:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:18 smithi171 conmon[41853]: debug 2022-01-31T21:36:18.001+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:18.001663+0000) 2022-01-31T21:36:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:18 smithi171 conmon[46715]: debug 2022-01-31T21:36:18.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:18.045489+0000) 2022-01-31T21:36:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:18 smithi167 conmon[49112]: debug 2022-01-31T21:36:18.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:18.608585+0000) 2022-01-31T21:36:18.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:18 smithi171 conmon[51620]: debug 2022-01-31T21:36:18.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:18.881025+0000) 2022-01-31T21:36:19.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:19 smithi167 conmon[54076]: debug 2022-01-31T21:36:19.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:19.272449+0000) 2022-01-31T21:36:19.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:19 smithi167 conmon[60316]: debug 2022-01-31T21:36:19.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:19.272442+0000) 2022-01-31T21:36:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:19 smithi171 conmon[46715]: debug 2022-01-31T21:36:19.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:19.045694+0000) 2022-01-31T21:36:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:19 smithi171 conmon[41853]: debug 2022-01-31T21:36:19.001+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:19.001824+0000) 2022-01-31T21:36:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:19 smithi167 conmon[49112]: debug 2022-01-31T21:36:19.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:19.608798+0000) 2022-01-31T21:36:19.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:19 smithi171 conmon[51620]: debug 2022-01-31T21:36:19.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:19.881217+0000) 2022-01-31T21:36:20.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:20 smithi167 conmon[60316]: debug 2022-01-31T21:36:20.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:20.272584+0000) 2022-01-31T21:36:20.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:20 smithi167 conmon[54076]: debug 2022-01-31T21:36:20.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:20.272669+0000) 2022-01-31T21:36:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:20 smithi171 conmon[46715]: debug 2022-01-31T21:36:20.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:20.045885+0000) 2022-01-31T21:36:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:20 smithi171 conmon[41853]: debug 2022-01-31T21:36:20.001+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:20.002036+0000) 2022-01-31T21:36:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:20 smithi167 conmon[49112]: debug 2022-01-31T21:36:20.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:20.608970+0000) 2022-01-31T21:36:20.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:20 smithi171 conmon[51620]: debug 2022-01-31T21:36:20.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:20.881404+0000) 2022-01-31T21:36:21.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:21 smithi167 conmon[60316]: debug 2022-01-31T21:36:21.271+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.272811+0000) 2022-01-31T21:36:21.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:21 smithi167 conmon[54076]: debug 2022-01-31T21:36:21.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.272889+0000) 2022-01-31T21:36:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[46715]: debug 2022-01-31T21:36:21.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.046022+0000) 2022-01-31T21:36:21.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[41853]: debug 2022-01-31T21:36:21.001+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.002240+0000) 2022-01-31T21:36:21.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:21 smithi167 conmon[54076]: debug 2022-01-31T21:36:21.835+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.836711+0000) 2022-01-31T21:36:21.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:21 smithi167 conmon[60316]: debug 2022-01-31T21:36:21.835+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.837074+0000) 2022-01-31T21:36:21.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:21 smithi167 conmon[49112]: debug 2022-01-31T21:36:21.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.609190+0000) 2022-01-31T21:36:21.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:21 smithi167 conmon[49112]: debug 2022-01-31T21:36:21.837+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.838665+0000) 2022-01-31T21:36:21.956 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[41853]: debug 2022-01-31T21:36:21.838+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.838777+0000) 2022-01-31T21:36:21.957 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[46715]: debug 2022-01-31T21:36:21.837+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.837912+0000) 2022-01-31T21:36:21.957 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[35325]: debug 2022-01-31T21:36:21.852+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143021 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:21.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[51620]: debug 2022-01-31T21:36:21.838+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.839238+0000) 2022-01-31T21:36:21.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:21 smithi171 conmon[51620]: debug 2022-01-31T21:36:21.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:21.881636+0000) 2022-01-31T21:36:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:22 smithi171 conmon[41853]: debug 2022-01-31T21:36:22.001+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:22.002412+0000) 2022-01-31T21:36:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:22 smithi171 conmon[46715]: debug 2022-01-31T21:36:22.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:22.046123+0000) 2022-01-31T21:36:22.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:22 smithi167 conmon[54076]: debug 2022-01-31T21:36:22.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:22.273105+0000) 2022-01-31T21:36:22.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:22 smithi167 conmon[60316]: debug 2022-01-31T21:36:22.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:22.273061+0000) 2022-01-31T21:36:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:22 smithi167 conmon[49112]: debug 2022-01-31T21:36:22.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:22.609405+0000) 2022-01-31T21:36:22.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:22 smithi171 conmon[51620]: debug 2022-01-31T21:36:22.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:22.881776+0000) 2022-01-31T21:36:23.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:23 smithi167 conmon[54076]: debug 2022-01-31T21:36:23.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:23.273240+0000) 2022-01-31T21:36:23.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:23 smithi167 conmon[60316]: debug 2022-01-31T21:36:23.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:23.273223+0000) 2022-01-31T21:36:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:23 smithi171 conmon[41853]: debug 2022-01-31T21:36:23.001+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:23.002607+0000) 2022-01-31T21:36:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:23 smithi171 conmon[46715]: debug 2022-01-31T21:36:23.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:23.046279+0000) 2022-01-31T21:36:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:23 smithi167 conmon[49112]: debug 2022-01-31T21:36:23.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:23.609616+0000) 2022-01-31T21:36:23.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:23 smithi171 conmon[51620]: debug 2022-01-31T21:36:23.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:23.881954+0000) 2022-01-31T21:36:23.957 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:23 smithi171 conmon[35325]: debug 2022-01-31T21:36:23.707+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:36:24.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:24 smithi167 conmon[54076]: debug 2022-01-31T21:36:24.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:24.273446+0000) 2022-01-31T21:36:24.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:24 smithi167 conmon[60316]: debug 2022-01-31T21:36:24.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:24.273453+0000) 2022-01-31T21:36:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:24 smithi171 conmon[41853]: debug 2022-01-31T21:36:24.002+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:24.002788+0000) 2022-01-31T21:36:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:24 smithi171 conmon[46715]: debug 2022-01-31T21:36:24.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:24.046457+0000) 2022-01-31T21:36:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:24 smithi167 conmon[49112]: debug 2022-01-31T21:36:24.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:24.609824+0000) 2022-01-31T21:36:24.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:24 smithi171 conmon[51620]: debug 2022-01-31T21:36:24.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:24.882077+0000) 2022-01-31T21:36:25.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:25 smithi167 conmon[54076]: debug 2022-01-31T21:36:25.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:25.273676+0000) 2022-01-31T21:36:25.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:25 smithi167 conmon[60316]: debug 2022-01-31T21:36:25.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:25.273687+0000) 2022-01-31T21:36:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:25 smithi171 conmon[41853]: debug 2022-01-31T21:36:25.002+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:25.002944+0000) 2022-01-31T21:36:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:25 smithi171 conmon[46715]: debug 2022-01-31T21:36:25.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:25.046610+0000) 2022-01-31T21:36:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:25 smithi167 conmon[49112]: debug 2022-01-31T21:36:25.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:25.610027+0000) 2022-01-31T21:36:25.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:25 smithi171 conmon[51620]: debug 2022-01-31T21:36:25.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:25.882227+0000) 2022-01-31T21:36:26.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:26 smithi167 conmon[54076]: debug 2022-01-31T21:36:26.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.273887+0000) 2022-01-31T21:36:26.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:26 smithi167 conmon[60316]: debug 2022-01-31T21:36:26.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.273865+0000) 2022-01-31T21:36:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[46715]: debug 2022-01-31T21:36:26.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.046799+0000) 2022-01-31T21:36:26.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[41853]: debug 2022-01-31T21:36:26.002+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.003097+0000) 2022-01-31T21:36:26.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:26 smithi167 conmon[49112]: debug 2022-01-31T21:36:26.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.610230+0000) 2022-01-31T21:36:26.957 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[35325]: debug 2022-01-31T21:36:26.870+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143132 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:26.958 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[41853]: debug 2022-01-31T21:36:26.856+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.857040+0000) 2022-01-31T21:36:26.958 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[46715]: debug 2022-01-31T21:36:26.855+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.855981+0000) 2022-01-31T21:36:26.959 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[51620]: debug 2022-01-31T21:36:26.856+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.857570+0000) 2022-01-31T21:36:26.959 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:26 smithi171 conmon[51620]: debug 2022-01-31T21:36:26.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.882427+0000) 2022-01-31T21:36:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:26 smithi167 conmon[49112]: debug 2022-01-31T21:36:26.854+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.855710+0000) 2022-01-31T21:36:27.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:26 smithi167 conmon[54076]: debug 2022-01-31T21:36:26.853+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.855330+0000) 2022-01-31T21:36:27.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:26 smithi167 conmon[60316]: debug 2022-01-31T21:36:26.853+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:26.855367+0000) 2022-01-31T21:36:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:27 smithi171 conmon[41853]: debug 2022-01-31T21:36:27.002+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:27.003279+0000) 2022-01-31T21:36:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:27 smithi171 conmon[46715]: debug 2022-01-31T21:36:27.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:27.046937+0000) 2022-01-31T21:36:27.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:27 smithi167 conmon[54076]: debug 2022-01-31T21:36:27.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:27.274063+0000) 2022-01-31T21:36:27.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:27 smithi167 conmon[60316]: debug 2022-01-31T21:36:27.272+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:27.274054+0000) 2022-01-31T21:36:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:27 smithi167 conmon[49112]: debug 2022-01-31T21:36:27.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:27.610432+0000) 2022-01-31T21:36:27.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:27 smithi171 conmon[51620]: debug 2022-01-31T21:36:27.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:27.882579+0000) 2022-01-31T21:36:28.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:28 smithi167 conmon[54076]: debug 2022-01-31T21:36:28.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:28.274222+0000) 2022-01-31T21:36:28.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:28 smithi167 conmon[60316]: debug 2022-01-31T21:36:28.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:28.274222+0000) 2022-01-31T21:36:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:28 smithi171 conmon[41853]: debug 2022-01-31T21:36:28.003+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:28.003388+0000) 2022-01-31T21:36:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:28 smithi171 conmon[46715]: debug 2022-01-31T21:36:28.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:28.047102+0000) 2022-01-31T21:36:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:28 smithi167 conmon[49112]: debug 2022-01-31T21:36:28.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:28.610607+0000) 2022-01-31T21:36:28.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:28 smithi171 conmon[51620]: debug 2022-01-31T21:36:28.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:28.882792+0000) 2022-01-31T21:36:29.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:29 smithi167 conmon[54076]: debug 2022-01-31T21:36:29.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:29.274410+0000) 2022-01-31T21:36:29.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:29 smithi167 conmon[60316]: debug 2022-01-31T21:36:29.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:29.274412+0000) 2022-01-31T21:36:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:29 smithi171 conmon[41853]: debug 2022-01-31T21:36:29.003+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:29.003520+0000) 2022-01-31T21:36:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:29 smithi171 conmon[46715]: debug 2022-01-31T21:36:29.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:29.047270+0000) 2022-01-31T21:36:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:29 smithi167 conmon[49112]: debug 2022-01-31T21:36:29.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:29.610797+0000) 2022-01-31T21:36:29.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:29 smithi171 conmon[51620]: debug 2022-01-31T21:36:29.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:29.882996+0000) 2022-01-31T21:36:30.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:30 smithi167 conmon[54076]: debug 2022-01-31T21:36:30.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:30.274605+0000) 2022-01-31T21:36:30.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:30 smithi167 conmon[60316]: debug 2022-01-31T21:36:30.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:30.274641+0000) 2022-01-31T21:36:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:30 smithi171 conmon[41853]: debug 2022-01-31T21:36:30.003+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:30.003661+0000) 2022-01-31T21:36:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:30 smithi171 conmon[46715]: debug 2022-01-31T21:36:30.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:30.047440+0000) 2022-01-31T21:36:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:30 smithi167 conmon[49112]: debug 2022-01-31T21:36:30.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:30.610987+0000) 2022-01-31T21:36:30.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:30 smithi171 conmon[51620]: debug 2022-01-31T21:36:30.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:30.883251+0000) 2022-01-31T21:36:31.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:31 smithi167 conmon[54076]: debug 2022-01-31T21:36:31.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.274797+0000) 2022-01-31T21:36:31.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:31 smithi167 conmon[60316]: debug 2022-01-31T21:36:31.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.274835+0000) 2022-01-31T21:36:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[41853]: debug 2022-01-31T21:36:31.003+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.003818+0000) 2022-01-31T21:36:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[46715]: debug 2022-01-31T21:36:31.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.047593+0000) 2022-01-31T21:36:31.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:31 smithi167 conmon[49112]: debug 2022-01-31T21:36:31.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.611210+0000) 2022-01-31T21:36:31.958 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[35325]: debug 2022-01-31T21:36:31.888+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143241 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:31.959 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[41853]: debug 2022-01-31T21:36:31.873+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.874421+0000) 2022-01-31T21:36:31.959 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[46715]: debug 2022-01-31T21:36:31.872+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.873422+0000) 2022-01-31T21:36:31.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[51620]: debug 2022-01-31T21:36:31.874+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.874794+0000) 2022-01-31T21:36:31.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:31 smithi171 conmon[51620]: debug 2022-01-31T21:36:31.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.883375+0000) 2022-01-31T21:36:32.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:31 smithi167 conmon[49112]: debug 2022-01-31T21:36:31.871+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.873371+0000) 2022-01-31T21:36:32.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:31 smithi167 conmon[54076]: debug 2022-01-31T21:36:31.871+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.872996+0000) 2022-01-31T21:36:32.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:31 smithi167 conmon[60316]: debug 2022-01-31T21:36:31.871+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:31.873184+0000) 2022-01-31T21:36:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:32 smithi171 conmon[41853]: debug 2022-01-31T21:36:32.003+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:32.003997+0000) 2022-01-31T21:36:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:32 smithi171 conmon[46715]: debug 2022-01-31T21:36:32.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:32.047747+0000) 2022-01-31T21:36:32.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:32 smithi167 conmon[54076]: debug 2022-01-31T21:36:32.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:32.274984+0000) 2022-01-31T21:36:32.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:32 smithi167 conmon[60316]: debug 2022-01-31T21:36:32.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:32.275030+0000) 2022-01-31T21:36:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:32 smithi167 conmon[49112]: debug 2022-01-31T21:36:32.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:32.611409+0000) 2022-01-31T21:36:32.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:32 smithi171 conmon[51620]: debug 2022-01-31T21:36:32.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:32.883551+0000) 2022-01-31T21:36:33.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:33 smithi167 conmon[54076]: debug 2022-01-31T21:36:33.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:33.275165+0000) 2022-01-31T21:36:33.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:33 smithi167 conmon[60316]: debug 2022-01-31T21:36:33.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:33.275143+0000) 2022-01-31T21:36:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:33 smithi171 conmon[41853]: debug 2022-01-31T21:36:33.003+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:33.004196+0000) 2022-01-31T21:36:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:33 smithi171 conmon[46715]: debug 2022-01-31T21:36:33.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:33.047946+0000) 2022-01-31T21:36:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:33 smithi167 conmon[49112]: debug 2022-01-31T21:36:33.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:33.611579+0000) 2022-01-31T21:36:33.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:33 smithi171 conmon[51620]: debug 2022-01-31T21:36:33.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:33.883736+0000) 2022-01-31T21:36:34.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:34 smithi167 conmon[54076]: debug 2022-01-31T21:36:34.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:34.275313+0000) 2022-01-31T21:36:34.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:34 smithi167 conmon[60316]: debug 2022-01-31T21:36:34.273+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:34.275314+0000) 2022-01-31T21:36:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:34 smithi171 conmon[41853]: debug 2022-01-31T21:36:34.004+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:34.004388+0000) 2022-01-31T21:36:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:34 smithi171 conmon[46715]: debug 2022-01-31T21:36:34.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:34.048128+0000) 2022-01-31T21:36:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:34 smithi167 conmon[49112]: debug 2022-01-31T21:36:34.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:34.611751+0000) 2022-01-31T21:36:34.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:34 smithi171 conmon[51620]: debug 2022-01-31T21:36:34.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:34.883893+0000) 2022-01-31T21:36:35.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:35 smithi167 conmon[54076]: debug 2022-01-31T21:36:35.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:35.275419+0000) 2022-01-31T21:36:35.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:35 smithi167 conmon[60316]: debug 2022-01-31T21:36:35.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:35.275458+0000) 2022-01-31T21:36:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:35 smithi171 conmon[41853]: debug 2022-01-31T21:36:35.004+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:35.004607+0000) 2022-01-31T21:36:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:35 smithi171 conmon[46715]: debug 2022-01-31T21:36:35.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:35.048353+0000) 2022-01-31T21:36:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:35 smithi167 conmon[49112]: debug 2022-01-31T21:36:35.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:35.611961+0000) 2022-01-31T21:36:35.926 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:35 smithi171 conmon[51620]: debug 2022-01-31T21:36:35.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:35.884098+0000) 2022-01-31T21:36:36.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:36 smithi167 conmon[54076]: debug 2022-01-31T21:36:36.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.275605+0000) 2022-01-31T21:36:36.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:36 smithi167 conmon[60316]: debug 2022-01-31T21:36:36.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.275634+0000) 2022-01-31T21:36:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[41853]: debug 2022-01-31T21:36:36.004+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.004816+0000) 2022-01-31T21:36:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[46715]: debug 2022-01-31T21:36:36.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.048519+0000) 2022-01-31T21:36:36.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:36 smithi167 conmon[49112]: debug 2022-01-31T21:36:36.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.612152+0000) 2022-01-31T21:36:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:36 smithi167 conmon[49112]: debug 2022-01-31T21:36:36.890+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.891699+0000) 2022-01-31T21:36:37.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:36 smithi167 conmon[54076]: debug 2022-01-31T21:36:36.889+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.890937+0000) 2022-01-31T21:36:37.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:36 smithi167 conmon[60316]: debug 2022-01-31T21:36:36.890+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.892097+0000) 2022-01-31T21:36:37.311 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[35325]: debug 2022-01-31T21:36:36.905+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143353 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:37.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[41853]: debug 2022-01-31T21:36:36.892+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.892819+0000) 2022-01-31T21:36:37.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:37 smithi171 conmon[41853]: debug 2022-01-31T21:36:37.004+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:37.004993+0000) 2022-01-31T21:36:37.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[46715]: debug 2022-01-31T21:36:36.890+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.891257+0000) 2022-01-31T21:36:37.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:37 smithi171 conmon[46715]: debug 2022-01-31T21:36:37.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:37.048659+0000) 2022-01-31T21:36:37.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[51620]: debug 2022-01-31T21:36:36.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.884300+0000) 2022-01-31T21:36:37.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:36 smithi171 conmon[51620]: debug 2022-01-31T21:36:36.890+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:36.891578+0000) 2022-01-31T21:36:37.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:37 smithi167 conmon[54076]: debug 2022-01-31T21:36:37.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:37.275799+0000) 2022-01-31T21:36:37.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:37 smithi167 conmon[60316]: debug 2022-01-31T21:36:37.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:37.275818+0000) 2022-01-31T21:36:37.922 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:37 smithi167 conmon[49112]: debug 2022-01-31T21:36:37.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:37.612370+0000) 2022-01-31T21:36:37.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:37 smithi171 conmon[51620]: debug 2022-01-31T21:36:37.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:37.884490+0000) 2022-01-31T21:36:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:38 smithi171 conmon[46715]: debug 2022-01-31T21:36:38.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:38.048815+0000) 2022-01-31T21:36:38.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:38 smithi171 conmon[41853]: debug 2022-01-31T21:36:38.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:38.005134+0000) 2022-01-31T21:36:38.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:38 smithi167 conmon[54076]: debug 2022-01-31T21:36:38.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:38.275963+0000) 2022-01-31T21:36:38.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:38 smithi167 conmon[60316]: debug 2022-01-31T21:36:38.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:38.275963+0000) 2022-01-31T21:36:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:38 smithi167 conmon[49112]: debug 2022-01-31T21:36:38.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:38.612567+0000) 2022-01-31T21:36:38.959 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:38 smithi171 conmon[35325]: debug 2022-01-31T21:36:38.708+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:36:38.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:38 smithi171 conmon[51620]: debug 2022-01-31T21:36:38.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:38.884691+0000) 2022-01-31T21:36:39.295 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:39 smithi171 conmon[41853]: debug 2022-01-31T21:36:39.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:39.005317+0000) 2022-01-31T21:36:39.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:39 smithi171 conmon[46715]: debug 2022-01-31T21:36:39.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:39.048962+0000) 2022-01-31T21:36:39.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:39 smithi167 conmon[54076]: debug 2022-01-31T21:36:39.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:39.276175+0000) 2022-01-31T21:36:39.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:39 smithi167 conmon[60316]: debug 2022-01-31T21:36:39.274+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:39.276175+0000) 2022-01-31T21:36:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:39 smithi167 conmon[49112]: debug 2022-01-31T21:36:39.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:39.612730+0000) 2022-01-31T21:36:39.959 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:39 smithi171 conmon[51620]: debug 2022-01-31T21:36:39.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:39.884936+0000) 2022-01-31T21:36:40.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:40 smithi167 conmon[54076]: debug 2022-01-31T21:36:40.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:40.276384+0000) 2022-01-31T21:36:40.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:40 smithi167 conmon[60316]: debug 2022-01-31T21:36:40.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:40.276313+0000) 2022-01-31T21:36:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:40 smithi171 conmon[41853]: debug 2022-01-31T21:36:40.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:40.005478+0000) 2022-01-31T21:36:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:40 smithi171 conmon[46715]: debug 2022-01-31T21:36:40.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:40.049142+0000) 2022-01-31T21:36:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:40 smithi167 conmon[49112]: debug 2022-01-31T21:36:40.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:40.612898+0000) 2022-01-31T21:36:40.959 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:40 smithi171 conmon[51620]: debug 2022-01-31T21:36:40.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:40.885143+0000) 2022-01-31T21:36:41.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:41 smithi167 conmon[54076]: debug 2022-01-31T21:36:41.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.276545+0000) 2022-01-31T21:36:41.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:41 smithi167 conmon[60316]: debug 2022-01-31T21:36:41.275+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.276544+0000) 2022-01-31T21:36:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[41853]: debug 2022-01-31T21:36:41.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.005654+0000) 2022-01-31T21:36:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[46715]: debug 2022-01-31T21:36:41.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.049336+0000) 2022-01-31T21:36:41.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:41 smithi167 conmon[49112]: debug 2022-01-31T21:36:41.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.613086+0000) 2022-01-31T21:36:41.959 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[35325]: debug 2022-01-31T21:36:41.922+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143463 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:41.960 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[41853]: debug 2022-01-31T21:36:41.907+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.909000+0000) 2022-01-31T21:36:41.961 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[46715]: debug 2022-01-31T21:36:41.906+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.907906+0000) 2022-01-31T21:36:41.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[51620]: debug 2022-01-31T21:36:41.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.885318+0000) 2022-01-31T21:36:41.963 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:41 smithi171 conmon[51620]: debug 2022-01-31T21:36:41.906+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.908177+0000) 2022-01-31T21:36:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:41 smithi167 conmon[49112]: debug 2022-01-31T21:36:41.906+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.908237+0000) 2022-01-31T21:36:42.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:41 smithi167 conmon[54076]: debug 2022-01-31T21:36:41.907+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.908489+0000) 2022-01-31T21:36:42.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:41 smithi167 conmon[60316]: debug 2022-01-31T21:36:41.906+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:41.907846+0000) 2022-01-31T21:36:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:42 smithi171 conmon[41853]: debug 2022-01-31T21:36:42.004+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:42.005806+0000) 2022-01-31T21:36:42.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:42 smithi171 conmon[46715]: debug 2022-01-31T21:36:42.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:42.049468+0000) 2022-01-31T21:36:42.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:42 smithi167 conmon[54076]: debug 2022-01-31T21:36:42.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:42.276750+0000) 2022-01-31T21:36:42.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:42 smithi167 conmon[60316]: debug 2022-01-31T21:36:42.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:42.276750+0000) 2022-01-31T21:36:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:42 smithi167 conmon[49112]: debug 2022-01-31T21:36:42.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:42.613312+0000) 2022-01-31T21:36:42.959 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:42 smithi171 conmon[51620]: debug 2022-01-31T21:36:42.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:42.885468+0000) 2022-01-31T21:36:43.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:43 smithi171 conmon[46715]: debug 2022-01-31T21:36:43.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:43.049622+0000) 2022-01-31T21:36:43.328 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:43 smithi171 conmon[41853]: debug 2022-01-31T21:36:43.004+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:43.005978+0000) 2022-01-31T21:36:43.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:43 smithi167 conmon[54076]: debug 2022-01-31T21:36:43.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:43.276925+0000) 2022-01-31T21:36:43.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:43 smithi167 conmon[60316]: debug 2022-01-31T21:36:43.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:43.276926+0000) 2022-01-31T21:36:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:43 smithi167 conmon[49112]: debug 2022-01-31T21:36:43.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:43.613491+0000) 2022-01-31T21:36:43.961 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:43 smithi171 conmon[51620]: debug 2022-01-31T21:36:43.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:43.885654+0000) 2022-01-31T21:36:44.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:44 smithi167 conmon[54076]: debug 2022-01-31T21:36:44.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:44.277075+0000) 2022-01-31T21:36:44.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:44 smithi167 conmon[60316]: debug 2022-01-31T21:36:44.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:44.277076+0000) 2022-01-31T21:36:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:44 smithi171 conmon[41853]: debug 2022-01-31T21:36:44.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:44.006192+0000) 2022-01-31T21:36:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:44 smithi171 conmon[46715]: debug 2022-01-31T21:36:44.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:44.049812+0000) 2022-01-31T21:36:44.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:44 smithi167 conmon[49112]: debug 2022-01-31T21:36:44.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:44.613675+0000) 2022-01-31T21:36:44.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:44 smithi171 conmon[51620]: debug 2022-01-31T21:36:44.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:44.885861+0000) 2022-01-31T21:36:45.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:45 smithi167 conmon[54076]: debug 2022-01-31T21:36:45.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:45.277267+0000) 2022-01-31T21:36:45.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:45 smithi167 conmon[60316]: debug 2022-01-31T21:36:45.276+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:45.277269+0000) 2022-01-31T21:36:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:45 smithi171 conmon[41853]: debug 2022-01-31T21:36:45.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:45.006390+0000) 2022-01-31T21:36:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:45 smithi171 conmon[46715]: debug 2022-01-31T21:36:45.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:45.049969+0000) 2022-01-31T21:36:45.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:45 smithi167 conmon[49112]: debug 2022-01-31T21:36:45.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:45.613873+0000) 2022-01-31T21:36:45.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:45 smithi171 conmon[51620]: debug 2022-01-31T21:36:45.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:45.886083+0000) 2022-01-31T21:36:46.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:46 smithi167 conmon[54076]: debug 2022-01-31T21:36:46.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.277495+0000) 2022-01-31T21:36:46.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:46 smithi167 conmon[60316]: debug 2022-01-31T21:36:46.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.277495+0000) 2022-01-31T21:36:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[41853]: debug 2022-01-31T21:36:46.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.006531+0000) 2022-01-31T21:36:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[46715]: debug 2022-01-31T21:36:46.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.050096+0000) 2022-01-31T21:36:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:46 smithi167 conmon[49112]: debug 2022-01-31T21:36:46.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.614062+0000) 2022-01-31T21:36:46.960 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[35325]: debug 2022-01-31T21:36:46.939+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143574 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:46.960 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[41853]: debug 2022-01-31T21:36:46.925+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.927091+0000) 2022-01-31T21:36:46.961 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[46715]: debug 2022-01-31T21:36:46.924+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.926287+0000) 2022-01-31T21:36:46.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[51620]: debug 2022-01-31T21:36:46.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.886273+0000) 2022-01-31T21:36:46.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:46 smithi171 conmon[51620]: debug 2022-01-31T21:36:46.924+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.926111+0000) 2022-01-31T21:36:47.276 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:46 smithi167 conmon[49112]: debug 2022-01-31T21:36:46.925+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.926484+0000) 2022-01-31T21:36:47.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:46 smithi167 conmon[54076]: debug 2022-01-31T21:36:46.926+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.927017+0000) 2022-01-31T21:36:47.277 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:47 smithi167 conmon[54076]: debug 2022-01-31T21:36:47.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:47.277667+0000) 2022-01-31T21:36:47.278 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:46 smithi167 conmon[60316]: debug 2022-01-31T21:36:46.925+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:46.926257+0000) 2022-01-31T21:36:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:47 smithi171 conmon[41853]: debug 2022-01-31T21:36:47.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:47.006729+0000) 2022-01-31T21:36:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:47 smithi171 conmon[46715]: debug 2022-01-31T21:36:47.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:47.050265+0000) 2022-01-31T21:36:47.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:47 smithi167 conmon[60316]: debug 2022-01-31T21:36:47.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:47.277668+0000) 2022-01-31T21:36:47.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:47 smithi167 conmon[49112]: debug 2022-01-31T21:36:47.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:47.614287+0000) 2022-01-31T21:36:47.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:47 smithi171 conmon[51620]: debug 2022-01-31T21:36:47.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:47.886453+0000) 2022-01-31T21:36:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:48 smithi171 conmon[46715]: debug 2022-01-31T21:36:48.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:48.050416+0000) 2022-01-31T21:36:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:48 smithi171 conmon[41853]: debug 2022-01-31T21:36:48.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:48.006890+0000) 2022-01-31T21:36:48.593 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:48 smithi167 conmon[54076]: debug 2022-01-31T21:36:48.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:48.277782+0000) 2022-01-31T21:36:48.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:48 smithi167 conmon[60316]: debug 2022-01-31T21:36:48.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:48.277782+0000) 2022-01-31T21:36:48.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:48 smithi167 conmon[49112]: debug 2022-01-31T21:36:48.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:48.614496+0000) 2022-01-31T21:36:48.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:48 smithi171 conmon[51620]: debug 2022-01-31T21:36:48.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:48.886659+0000) 2022-01-31T21:36:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:49 smithi171 conmon[46715]: debug 2022-01-31T21:36:49.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:49.050614+0000) 2022-01-31T21:36:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:49 smithi171 conmon[41853]: debug 2022-01-31T21:36:49.005+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:49.007035+0000) 2022-01-31T21:36:49.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:49 smithi167 conmon[54076]: debug 2022-01-31T21:36:49.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:49.277971+0000) 2022-01-31T21:36:49.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:49 smithi167 conmon[60316]: debug 2022-01-31T21:36:49.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:49.277971+0000) 2022-01-31T21:36:49.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:49 smithi167 conmon[49112]: debug 2022-01-31T21:36:49.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:49.614629+0000) 2022-01-31T21:36:49.960 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:49 smithi171 conmon[51620]: debug 2022-01-31T21:36:49.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:49.886867+0000) 2022-01-31T21:36:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:50 smithi171 conmon[46715]: debug 2022-01-31T21:36:50.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:50.050836+0000) 2022-01-31T21:36:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:50 smithi171 conmon[41853]: debug 2022-01-31T21:36:50.006+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:50.007235+0000) 2022-01-31T21:36:50.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:50 smithi167 conmon[54076]: debug 2022-01-31T21:36:50.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:50.278168+0000) 2022-01-31T21:36:50.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:50 smithi167 conmon[60316]: debug 2022-01-31T21:36:50.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:50.278168+0000) 2022-01-31T21:36:50.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:50 smithi167 conmon[49112]: debug 2022-01-31T21:36:50.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:50.614748+0000) 2022-01-31T21:36:50.961 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:50 smithi171 conmon[51620]: debug 2022-01-31T21:36:50.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:50.887069+0000) 2022-01-31T21:36:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[46715]: debug 2022-01-31T21:36:51.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.051033+0000) 2022-01-31T21:36:51.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[41853]: debug 2022-01-31T21:36:51.006+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.007391+0000) 2022-01-31T21:36:51.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:51 smithi167 conmon[54076]: debug 2022-01-31T21:36:51.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.278384+0000) 2022-01-31T21:36:51.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:51 smithi167 conmon[60316]: debug 2022-01-31T21:36:51.277+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.278384+0000) 2022-01-31T21:36:51.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:51 smithi167 conmon[49112]: debug 2022-01-31T21:36:51.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.614904+0000) 2022-01-31T21:36:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:51 smithi167 conmon[49112]: debug 2022-01-31T21:36:51.943+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.944321+0000) 2022-01-31T21:36:52.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:51 smithi167 conmon[54076]: debug 2022-01-31T21:36:51.942+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.943165+0000) 2022-01-31T21:36:52.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:51 smithi167 conmon[60316]: debug 2022-01-31T21:36:51.943+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.944070+0000) 2022-01-31T21:36:52.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[35325]: debug 2022-01-31T21:36:51.957+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[46715]: debug 2022-01-31T21:36:51.941+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.943540+0000) 2022-01-31T21:36:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:52 smithi171 conmon[46715]: debug 2022-01-31T21:36:52.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:52.051186+0000) 2022-01-31T21:36:52.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[41853]: debug 2022-01-31T21:36:51.943+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.945082+0000) 2022-01-31T21:36:52.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:52 smithi171 conmon[41853]: debug 2022-01-31T21:36:52.006+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:52.007537+0000) 2022-01-31T21:36:52.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[51620]: debug 2022-01-31T21:36:51.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.887291+0000) 2022-01-31T21:36:52.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:51 smithi171 conmon[51620]: debug 2022-01-31T21:36:51.942+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:51.943911+0000) 2022-01-31T21:36:52.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:52 smithi167 conmon[54076]: debug 2022-01-31T21:36:52.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:52.278612+0000) 2022-01-31T21:36:52.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:52 smithi167 conmon[60316]: debug 2022-01-31T21:36:52.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:52.278612+0000) 2022-01-31T21:36:52.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:52 smithi167 conmon[49112]: debug 2022-01-31T21:36:52.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:52.615052+0000) 2022-01-31T21:36:52.961 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:52 smithi171 conmon[51620]: debug 2022-01-31T21:36:52.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:52.887451+0000) 2022-01-31T21:36:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:53 smithi171 conmon[41853]: debug 2022-01-31T21:36:53.006+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:53.007681+0000) 2022-01-31T21:36:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:53 smithi171 conmon[46715]: debug 2022-01-31T21:36:53.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:53.051393+0000) 2022-01-31T21:36:53.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:53 smithi167 conmon[54076]: debug 2022-01-31T21:36:53.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:53.278794+0000) 2022-01-31T21:36:53.532 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:53 smithi167 conmon[60316]: debug 2022-01-31T21:36:53.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:53.278789+0000) 2022-01-31T21:36:53.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:53 smithi167 conmon[49112]: debug 2022-01-31T21:36:53.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:53.615174+0000) 2022-01-31T21:36:53.961 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:53 smithi171 conmon[35325]: debug 2022-01-31T21:36:53.708+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:36:53.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:53 smithi171 conmon[51620]: debug 2022-01-31T21:36:53.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:53.887571+0000) 2022-01-31T21:36:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:54 smithi171 conmon[41853]: debug 2022-01-31T21:36:54.007+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:54.007804+0000) 2022-01-31T21:36:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:54 smithi171 conmon[46715]: debug 2022-01-31T21:36:54.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:54.051548+0000) 2022-01-31T21:36:54.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:54 smithi167 conmon[54076]: debug 2022-01-31T21:36:54.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:54.279022+0000) 2022-01-31T21:36:54.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:54 smithi167 conmon[60316]: debug 2022-01-31T21:36:54.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:54.279022+0000) 2022-01-31T21:36:54.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:54 smithi167 conmon[49112]: debug 2022-01-31T21:36:54.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:54.615310+0000) 2022-01-31T21:36:54.961 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:54 smithi171 conmon[51620]: debug 2022-01-31T21:36:54.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:54.887772+0000) 2022-01-31T21:36:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:55 smithi171 conmon[41853]: debug 2022-01-31T21:36:55.007+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:55.007969+0000) 2022-01-31T21:36:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:55 smithi171 conmon[46715]: debug 2022-01-31T21:36:55.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:55.051725+0000) 2022-01-31T21:36:55.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:55 smithi167 conmon[54076]: debug 2022-01-31T21:36:55.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:55.279190+0000) 2022-01-31T21:36:55.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:55 smithi167 conmon[60316]: debug 2022-01-31T21:36:55.278+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:55.279191+0000) 2022-01-31T21:36:55.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:55 smithi167 conmon[49112]: debug 2022-01-31T21:36:55.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:55.615533+0000) 2022-01-31T21:36:55.961 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:55 smithi171 conmon[51620]: debug 2022-01-31T21:36:55.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:55.887953+0000) 2022-01-31T21:36:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[41853]: debug 2022-01-31T21:36:56.007+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.008150+0000) 2022-01-31T21:36:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[46715]: debug 2022-01-31T21:36:56.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.051935+0000) 2022-01-31T21:36:56.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:56 smithi167 conmon[54076]: debug 2022-01-31T21:36:56.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.279373+0000) 2022-01-31T21:36:56.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:56 smithi167 conmon[60316]: debug 2022-01-31T21:36:56.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.279373+0000) 2022-01-31T21:36:56.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:56 smithi167 conmon[49112]: debug 2022-01-31T21:36:56.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.615750+0000) 2022-01-31T21:36:56.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[51620]: debug 2022-01-31T21:36:56.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.888163+0000) 2022-01-31T21:36:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:56 smithi167 conmon[49112]: debug 2022-01-31T21:36:56.962+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.962526+0000) 2022-01-31T21:36:57.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:56 smithi167 conmon[54076]: debug 2022-01-31T21:36:56.961+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.961648+0000) 2022-01-31T21:36:57.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:56 smithi167 conmon[60316]: debug 2022-01-31T21:36:56.961+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.962473+0000) 2022-01-31T21:36:57.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[35325]: debug 2022-01-31T21:36:56.976+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143796 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:36:57.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[51620]: debug 2022-01-31T21:36:56.961+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.962182+0000) 2022-01-31T21:36:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[41853]: debug 2022-01-31T21:36:56.961+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.962463+0000) 2022-01-31T21:36:57.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:57 smithi171 conmon[41853]: debug 2022-01-31T21:36:57.007+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:57.008357+0000) 2022-01-31T21:36:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:56 smithi171 conmon[46715]: debug 2022-01-31T21:36:56.960+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:56.961356+0000) 2022-01-31T21:36:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:57 smithi171 conmon[46715]: debug 2022-01-31T21:36:57.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:57.052137+0000) 2022-01-31T21:36:57.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:57 smithi167 conmon[54076]: debug 2022-01-31T21:36:57.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:57.279541+0000) 2022-01-31T21:36:57.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:57 smithi167 conmon[60316]: debug 2022-01-31T21:36:57.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:57.279540+0000) 2022-01-31T21:36:57.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:57 smithi167 conmon[49112]: debug 2022-01-31T21:36:57.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:57.615946+0000) 2022-01-31T21:36:57.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:57 smithi171 conmon[51620]: debug 2022-01-31T21:36:57.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:57.888300+0000) 2022-01-31T21:36:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:58 smithi171 conmon[46715]: debug 2022-01-31T21:36:58.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:58.052286+0000) 2022-01-31T21:36:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:58 smithi171 conmon[41853]: debug 2022-01-31T21:36:58.007+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:58.008479+0000) 2022-01-31T21:36:58.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:58 smithi167 conmon[54076]: debug 2022-01-31T21:36:58.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:58.279697+0000) 2022-01-31T21:36:58.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:58 smithi167 conmon[60316]: debug 2022-01-31T21:36:58.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:58.279697+0000) 2022-01-31T21:36:58.868 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:58 smithi167 conmon[49112]: debug 2022-01-31T21:36:58.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:58.616125+0000) 2022-01-31T21:36:58.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:58 smithi171 conmon[51620]: debug 2022-01-31T21:36:58.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:58.888487+0000) 2022-01-31T21:36:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:36:59 smithi171 conmon[46715]: debug 2022-01-31T21:36:59.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:59.052515+0000) 2022-01-31T21:36:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:36:59 smithi171 conmon[41853]: debug 2022-01-31T21:36:59.007+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:59.008652+0000) 2022-01-31T21:36:59.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:36:59 smithi167 conmon[54076]: debug 2022-01-31T21:36:59.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:59.279857+0000) 2022-01-31T21:36:59.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:36:59 smithi167 conmon[60316]: debug 2022-01-31T21:36:59.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:59.279857+0000) 2022-01-31T21:36:59.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:36:59 smithi167 conmon[49112]: debug 2022-01-31T21:36:59.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:59.616322+0000) 2022-01-31T21:36:59.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:36:59 smithi171 conmon[51620]: debug 2022-01-31T21:36:59.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:36:59.888692+0000) 2022-01-31T21:37:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:00 smithi171 conmon[46715]: debug 2022-01-31T21:37:00.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:00.052702+0000) 2022-01-31T21:37:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:00 smithi171 conmon[41853]: debug 2022-01-31T21:37:00.008+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:00.008816+0000) 2022-01-31T21:37:00.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:00 smithi167 conmon[54076]: debug 2022-01-31T21:37:00.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:00.280047+0000) 2022-01-31T21:37:00.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:00 smithi167 conmon[60316]: debug 2022-01-31T21:37:00.279+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:00.280047+0000) 2022-01-31T21:37:00.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:00 smithi167 conmon[49112]: debug 2022-01-31T21:37:00.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:00.616530+0000) 2022-01-31T21:37:00.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:00 smithi171 conmon[51620]: debug 2022-01-31T21:37:00.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:00.888921+0000) 2022-01-31T21:37:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[46715]: debug 2022-01-31T21:37:01.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.052904+0000) 2022-01-31T21:37:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[41853]: debug 2022-01-31T21:37:01.008+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.009003+0000) 2022-01-31T21:37:01.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:01 smithi167 conmon[54076]: debug 2022-01-31T21:37:01.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.280271+0000) 2022-01-31T21:37:01.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:01 smithi167 conmon[60316]: debug 2022-01-31T21:37:01.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.280269+0000) 2022-01-31T21:37:01.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:01 smithi167 conmon[49112]: debug 2022-01-31T21:37:01.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.616726+0000) 2022-01-31T21:37:01.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[51620]: debug 2022-01-31T21:37:01.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.889114+0000) 2022-01-31T21:37:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:01 smithi167 conmon[49112]: debug 2022-01-31T21:37:01.979+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.980091+0000) 2022-01-31T21:37:02.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:01 smithi167 conmon[54076]: debug 2022-01-31T21:37:01.978+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.979350+0000) 2022-01-31T21:37:02.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:01 smithi167 conmon[60316]: debug 2022-01-31T21:37:01.979+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.980050+0000) 2022-01-31T21:37:02.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[35325]: debug 2022-01-31T21:37:01.994+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 143906 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:02.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[51620]: debug 2022-01-31T21:37:01.980+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.981225+0000) 2022-01-31T21:37:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[41853]: debug 2022-01-31T21:37:01.979+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.980206+0000) 2022-01-31T21:37:02.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:02 smithi171 conmon[41853]: debug 2022-01-31T21:37:02.008+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:02.009167+0000) 2022-01-31T21:37:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:01 smithi171 conmon[46715]: debug 2022-01-31T21:37:01.978+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:01.979412+0000) 2022-01-31T21:37:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:02 smithi171 conmon[46715]: debug 2022-01-31T21:37:02.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:02.053084+0000) 2022-01-31T21:37:02.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:02 smithi167 conmon[60316]: debug 2022-01-31T21:37:02.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:02.280473+0000) 2022-01-31T21:37:02.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:02 smithi167 conmon[54076]: debug 2022-01-31T21:37:02.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:02.280473+0000) 2022-01-31T21:37:02.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:02 smithi167 conmon[49112]: debug 2022-01-31T21:37:02.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:02.616914+0000) 2022-01-31T21:37:02.963 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:02 smithi171 conmon[51620]: debug 2022-01-31T21:37:02.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:02.889262+0000) 2022-01-31T21:37:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:03 smithi171 conmon[41853]: debug 2022-01-31T21:37:03.008+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:03.009338+0000) 2022-01-31T21:37:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:03 smithi171 conmon[46715]: debug 2022-01-31T21:37:03.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:03.053257+0000) 2022-01-31T21:37:03.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:03 smithi167 conmon[60316]: debug 2022-01-31T21:37:03.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:03.280595+0000) 2022-01-31T21:37:03.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:03 smithi167 conmon[54076]: debug 2022-01-31T21:37:03.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:03.280595+0000) 2022-01-31T21:37:03.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:03 smithi167 conmon[49112]: debug 2022-01-31T21:37:03.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:03.617112+0000) 2022-01-31T21:37:03.962 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:03 smithi171 conmon[51620]: debug 2022-01-31T21:37:03.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:03.889451+0000) 2022-01-31T21:37:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:04 smithi171 conmon[41853]: debug 2022-01-31T21:37:04.008+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:04.009541+0000) 2022-01-31T21:37:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:04 smithi171 conmon[46715]: debug 2022-01-31T21:37:04.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:04.053407+0000) 2022-01-31T21:37:04.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:04 smithi167 conmon[60316]: debug 2022-01-31T21:37:04.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:04.280793+0000) 2022-01-31T21:37:04.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:04 smithi167 conmon[54076]: debug 2022-01-31T21:37:04.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:04.280793+0000) 2022-01-31T21:37:04.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:04 smithi167 conmon[49112]: debug 2022-01-31T21:37:04.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:04.617293+0000) 2022-01-31T21:37:04.963 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:04 smithi171 conmon[51620]: debug 2022-01-31T21:37:04.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:04.889626+0000) 2022-01-31T21:37:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:05 smithi171 conmon[41853]: debug 2022-01-31T21:37:05.009+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:05.009725+0000) 2022-01-31T21:37:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:05 smithi171 conmon[46715]: debug 2022-01-31T21:37:05.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:05.053572+0000) 2022-01-31T21:37:05.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:05 smithi167 conmon[54076]: debug 2022-01-31T21:37:05.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:05.281009+0000) 2022-01-31T21:37:05.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:05 smithi167 conmon[60316]: debug 2022-01-31T21:37:05.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:05.281009+0000) 2022-01-31T21:37:05.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:05 smithi167 conmon[49112]: debug 2022-01-31T21:37:05.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:05.617474+0000) 2022-01-31T21:37:05.963 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:05 smithi171 conmon[51620]: debug 2022-01-31T21:37:05.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:05.889832+0000) 2022-01-31T21:37:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:06 smithi171 conmon[46715]: debug 2022-01-31T21:37:06.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.053727+0000) 2022-01-31T21:37:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:06 smithi171 conmon[41853]: debug 2022-01-31T21:37:06.009+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.009929+0000) 2022-01-31T21:37:06.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:06 smithi167 conmon[60316]: debug 2022-01-31T21:37:06.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.281176+0000) 2022-01-31T21:37:06.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:06 smithi167 conmon[54076]: debug 2022-01-31T21:37:06.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.281178+0000) 2022-01-31T21:37:06.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:06 smithi167 conmon[49112]: debug 2022-01-31T21:37:06.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.617664+0000) 2022-01-31T21:37:06.963 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:06 smithi171 conmon[51620]: debug 2022-01-31T21:37:06.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.890051+0000) 2022-01-31T21:37:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:06 smithi167 conmon[49112]: debug 2022-01-31T21:37:06.996+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.997226+0000) 2022-01-31T21:37:07.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:06 smithi167 conmon[54076]: debug 2022-01-31T21:37:06.996+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.997369+0000) 2022-01-31T21:37:07.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:06 smithi167 conmon[60316]: debug 2022-01-31T21:37:06.996+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.997030+0000) 2022-01-31T21:37:07.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:07 smithi171 conmon[35325]: debug 2022-01-31T21:37:07.028+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144017 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:07.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:06 smithi171 conmon[51620]: debug 2022-01-31T21:37:06.998+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.999309+0000) 2022-01-31T21:37:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:06 smithi171 conmon[41853]: debug 2022-01-31T21:37:06.997+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.998630+0000) 2022-01-31T21:37:07.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:07 smithi171 conmon[41853]: debug 2022-01-31T21:37:07.009+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:07.010079+0000) 2022-01-31T21:37:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:06 smithi171 conmon[46715]: debug 2022-01-31T21:37:06.996+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:06.997592+0000) 2022-01-31T21:37:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:07 smithi171 conmon[46715]: debug 2022-01-31T21:37:07.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:07.053968+0000) 2022-01-31T21:37:07.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:07 smithi167 conmon[60316]: debug 2022-01-31T21:37:07.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:07.281387+0000) 2022-01-31T21:37:07.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:07 smithi167 conmon[54076]: debug 2022-01-31T21:37:07.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:07.281387+0000) 2022-01-31T21:37:07.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:07 smithi167 conmon[49112]: debug 2022-01-31T21:37:07.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:07.617817+0000) 2022-01-31T21:37:07.963 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:07 smithi171 conmon[51620]: debug 2022-01-31T21:37:07.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:07.890223+0000) 2022-01-31T21:37:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:08 smithi171 conmon[46715]: debug 2022-01-31T21:37:08.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:08.054076+0000) 2022-01-31T21:37:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:08 smithi171 conmon[41853]: debug 2022-01-31T21:37:08.009+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:08.010233+0000) 2022-01-31T21:37:08.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:08 smithi167 conmon[54076]: debug 2022-01-31T21:37:08.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:08.281570+0000) 2022-01-31T21:37:08.598 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:08 smithi167 conmon[60316]: debug 2022-01-31T21:37:08.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:08.281570+0000) 2022-01-31T21:37:08.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:08 smithi167 conmon[49112]: debug 2022-01-31T21:37:08.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:08.617965+0000) 2022-01-31T21:37:08.963 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:08 smithi171 conmon[35325]: debug 2022-01-31T21:37:08.709+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:37:08.964 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:08 smithi171 conmon[51620]: debug 2022-01-31T21:37:08.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:08.890365+0000) 2022-01-31T21:37:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:09 smithi171 conmon[46715]: debug 2022-01-31T21:37:09.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:09.054255+0000) 2022-01-31T21:37:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:09 smithi171 conmon[41853]: debug 2022-01-31T21:37:09.009+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:09.010424+0000) 2022-01-31T21:37:09.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:09 smithi167 conmon[60316]: debug 2022-01-31T21:37:09.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:09.281723+0000) 2022-01-31T21:37:09.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:09 smithi167 conmon[54076]: debug 2022-01-31T21:37:09.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:09.281723+0000) 2022-01-31T21:37:09.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:09 smithi167 conmon[49112]: debug 2022-01-31T21:37:09.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:09.618154+0000) 2022-01-31T21:37:09.964 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:09 smithi171 conmon[51620]: debug 2022-01-31T21:37:09.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:09.890565+0000) 2022-01-31T21:37:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:10 smithi171 conmon[41853]: debug 2022-01-31T21:37:10.009+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:10.010629+0000) 2022-01-31T21:37:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:10 smithi171 conmon[46715]: debug 2022-01-31T21:37:10.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:10.054435+0000) 2022-01-31T21:37:10.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:10 smithi167 conmon[54076]: debug 2022-01-31T21:37:10.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:10.281909+0000) 2022-01-31T21:37:10.598 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:10 smithi167 conmon[60316]: debug 2022-01-31T21:37:10.280+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:10.281909+0000) 2022-01-31T21:37:10.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:10 smithi167 conmon[49112]: debug 2022-01-31T21:37:10.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:10.618341+0000) 2022-01-31T21:37:10.964 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:10 smithi171 conmon[51620]: debug 2022-01-31T21:37:10.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:10.890745+0000) 2022-01-31T21:37:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:11 smithi171 conmon[46715]: debug 2022-01-31T21:37:11.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:11.054654+0000) 2022-01-31T21:37:11.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:11 smithi171 conmon[41853]: debug 2022-01-31T21:37:11.010+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:11.010831+0000) 2022-01-31T21:37:11.598 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:11 smithi167 conmon[60316]: debug 2022-01-31T21:37:11.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:11.282104+0000) 2022-01-31T21:37:11.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:11 smithi167 conmon[54076]: debug 2022-01-31T21:37:11.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:11.282104+0000) 2022-01-31T21:37:11.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:11 smithi167 conmon[49112]: debug 2022-01-31T21:37:11.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:11.618565+0000) 2022-01-31T21:37:11.964 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:11 smithi171 conmon[51620]: debug 2022-01-31T21:37:11.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:11.890916+0000) 2022-01-31T21:37:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:12 smithi167 conmon[49112]: debug 2022-01-31T21:37:12.030+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.031888+0000) 2022-01-31T21:37:12.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:12 smithi167 conmon[60316]: debug 2022-01-31T21:37:12.030+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.032161+0000) 2022-01-31T21:37:12.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:12 smithi167 conmon[54076]: debug 2022-01-31T21:37:12.036+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.037658+0000) 2022-01-31T21:37:12.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[35325]: debug 2022-01-31T21:37:12.046+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144128 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:12.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[51620]: debug 2022-01-31T21:37:12.031+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.032298+0000) 2022-01-31T21:37:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[41853]: debug 2022-01-31T21:37:12.010+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.011005+0000) 2022-01-31T21:37:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[41853]: debug 2022-01-31T21:37:12.031+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.032601+0000) 2022-01-31T21:37:12.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[46715]: debug 2022-01-31T21:37:12.031+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.031867+0000) 2022-01-31T21:37:12.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[46715]: debug 2022-01-31T21:37:12.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.054763+0000) 2022-01-31T21:37:12.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:12 smithi167 conmon[54076]: debug 2022-01-31T21:37:12.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.282341+0000) 2022-01-31T21:37:12.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:12 smithi167 conmon[60316]: debug 2022-01-31T21:37:12.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.282341+0000) 2022-01-31T21:37:12.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:12 smithi167 conmon[49112]: debug 2022-01-31T21:37:12.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.618732+0000) 2022-01-31T21:37:12.964 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:12 smithi171 conmon[51620]: debug 2022-01-31T21:37:12.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:12.891093+0000) 2022-01-31T21:37:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:13 smithi171 conmon[41853]: debug 2022-01-31T21:37:13.010+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:13.011189+0000) 2022-01-31T21:37:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:13 smithi171 conmon[46715]: debug 2022-01-31T21:37:13.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:13.054890+0000) 2022-01-31T21:37:13.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:13 smithi167 conmon[54076]: debug 2022-01-31T21:37:13.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:13.282474+0000) 2022-01-31T21:37:13.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:13 smithi167 conmon[60316]: debug 2022-01-31T21:37:13.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:13.282595+0000) 2022-01-31T21:37:13.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:13 smithi167 conmon[49112]: debug 2022-01-31T21:37:13.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:13.618868+0000) 2022-01-31T21:37:13.965 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:13 smithi171 conmon[51620]: debug 2022-01-31T21:37:13.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:13.891244+0000) 2022-01-31T21:37:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:14 smithi171 conmon[41853]: debug 2022-01-31T21:37:14.011+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:14.011375+0000) 2022-01-31T21:37:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:14 smithi171 conmon[46715]: debug 2022-01-31T21:37:14.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:14.055065+0000) 2022-01-31T21:37:14.598 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:14 smithi167 conmon[60316]: debug 2022-01-31T21:37:14.281+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:14.282775+0000) 2022-01-31T21:37:14.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:14 smithi167 conmon[54076]: debug 2022-01-31T21:37:14.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:14.282708+0000) 2022-01-31T21:37:14.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:14 smithi167 conmon[49112]: debug 2022-01-31T21:37:14.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:14.618986+0000) 2022-01-31T21:37:14.965 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:14 smithi171 conmon[51620]: debug 2022-01-31T21:37:14.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:14.891450+0000) 2022-01-31T21:37:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:15 smithi171 conmon[41853]: debug 2022-01-31T21:37:15.011+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:15.011550+0000) 2022-01-31T21:37:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:15 smithi171 conmon[46715]: debug 2022-01-31T21:37:15.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:15.055225+0000) 2022-01-31T21:37:15.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:15 smithi167 conmon[54076]: debug 2022-01-31T21:37:15.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:15.282898+0000) 2022-01-31T21:37:15.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:15 smithi167 conmon[60316]: debug 2022-01-31T21:37:15.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:15.282970+0000) 2022-01-31T21:37:15.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:15 smithi167 conmon[49112]: debug 2022-01-31T21:37:15.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:15.619160+0000) 2022-01-31T21:37:15.965 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:15 smithi171 conmon[51620]: debug 2022-01-31T21:37:15.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:15.891650+0000) 2022-01-31T21:37:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:16 smithi171 conmon[41853]: debug 2022-01-31T21:37:16.011+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:16.011770+0000) 2022-01-31T21:37:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:16 smithi171 conmon[46715]: debug 2022-01-31T21:37:16.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:16.055378+0000) 2022-01-31T21:37:16.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:16 smithi167 conmon[54076]: debug 2022-01-31T21:37:16.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:16.283085+0000) 2022-01-31T21:37:16.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:16 smithi167 conmon[60316]: debug 2022-01-31T21:37:16.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:16.283149+0000) 2022-01-31T21:37:16.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:16 smithi167 conmon[49112]: debug 2022-01-31T21:37:16.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:16.619364+0000) 2022-01-31T21:37:16.965 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:16 smithi171 conmon[51620]: debug 2022-01-31T21:37:16.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:16.891851+0000) 2022-01-31T21:37:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:17 smithi167 conmon[49112]: debug 2022-01-31T21:37:17.047+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.048705+0000) 2022-01-31T21:37:17.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:17 smithi167 conmon[60316]: debug 2022-01-31T21:37:17.047+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.048575+0000) 2022-01-31T21:37:17.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:17 smithi167 conmon[54076]: debug 2022-01-31T21:37:17.048+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.049955+0000) 2022-01-31T21:37:17.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[35325]: debug 2022-01-31T21:37:17.064+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144241 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:17.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[51620]: debug 2022-01-31T21:37:17.049+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.050492+0000) 2022-01-31T21:37:17.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[41853]: debug 2022-01-31T21:37:17.011+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.011965+0000) 2022-01-31T21:37:17.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[41853]: debug 2022-01-31T21:37:17.049+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.050239+0000) 2022-01-31T21:37:17.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[46715]: debug 2022-01-31T21:37:17.049+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.050663+0000) 2022-01-31T21:37:17.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[46715]: debug 2022-01-31T21:37:17.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.055508+0000) 2022-01-31T21:37:17.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:17 smithi167 conmon[60316]: debug 2022-01-31T21:37:17.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.283301+0000) 2022-01-31T21:37:17.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:17 smithi167 conmon[54076]: debug 2022-01-31T21:37:17.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.283268+0000) 2022-01-31T21:37:17.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:17 smithi167 conmon[49112]: debug 2022-01-31T21:37:17.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.619566+0000) 2022-01-31T21:37:17.965 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:17 smithi171 conmon[51620]: debug 2022-01-31T21:37:17.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:17.892055+0000) 2022-01-31T21:37:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:18 smithi171 conmon[41853]: debug 2022-01-31T21:37:18.011+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:18.012153+0000) 2022-01-31T21:37:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:18 smithi171 conmon[46715]: debug 2022-01-31T21:37:18.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:18.055663+0000) 2022-01-31T21:37:18.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:18 smithi167 conmon[54076]: debug 2022-01-31T21:37:18.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:18.283401+0000) 2022-01-31T21:37:18.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:18 smithi167 conmon[60316]: debug 2022-01-31T21:37:18.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:18 smithi167 conmon[60316]: 2022-01-31T21:37:18.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:18.283460+0000) 2022-01-31T21:37:18.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:18 smithi167 conmon[49112]: debug 2022-01-31T21:37:18.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:18.619672+0000) 2022-01-31T21:37:18.965 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:18 smithi171 conmon[51620]: debug 2022-01-31T21:37:18.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:18.892209+0000) 2022-01-31T21:37:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:19 smithi171 conmon[41853]: debug 2022-01-31T21:37:19.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:19.012305+0000) 2022-01-31T21:37:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:19 smithi171 conmon[46715]: debug 2022-01-31T21:37:19.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:19.055829+0000) 2022-01-31T21:37:19.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:19 smithi167 conmon[54076]: debug 2022-01-31T21:37:19.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:19.283612+0000) 2022-01-31T21:37:19.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:19 smithi167 conmon[60316]: debug 2022-01-31T21:37:19.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:19.283584+0000) 2022-01-31T21:37:19.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:19 smithi167 conmon[49112]: debug 2022-01-31T21:37:19.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:19.619924+0000) 2022-01-31T21:37:19.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:19 smithi171 conmon[51620]: debug 2022-01-31T21:37:19.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:19.892358+0000) 2022-01-31T21:37:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:20 smithi171 conmon[41853]: debug 2022-01-31T21:37:20.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:20.012492+0000) 2022-01-31T21:37:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:20 smithi171 conmon[46715]: debug 2022-01-31T21:37:20.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:20.056039+0000) 2022-01-31T21:37:20.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:20 smithi167 conmon[54076]: debug 2022-01-31T21:37:20.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:20.283759+0000) 2022-01-31T21:37:20.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:20 smithi167 conmon[60316]: debug 2022-01-31T21:37:20.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:20.283740+0000) 2022-01-31T21:37:20.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:20 smithi167 conmon[49112]: debug 2022-01-31T21:37:20.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:20.620052+0000) 2022-01-31T21:37:20.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:20 smithi171 conmon[51620]: debug 2022-01-31T21:37:20.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:20.892515+0000) 2022-01-31T21:37:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:21 smithi171 conmon[41853]: debug 2022-01-31T21:37:21.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:21.012648+0000) 2022-01-31T21:37:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:21 smithi171 conmon[46715]: debug 2022-01-31T21:37:21.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:21.056223+0000) 2022-01-31T21:37:21.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:21 smithi167 conmon[54076]: debug 2022-01-31T21:37:21.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:21.283917+0000) 2022-01-31T21:37:21.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:21 smithi167 conmon[60316]: debug 2022-01-31T21:37:21.282+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:21.283865+0000) 2022-01-31T21:37:21.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:21 smithi167 conmon[49112]: debug 2022-01-31T21:37:21.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:21.620154+0000) 2022-01-31T21:37:21.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:21 smithi171 conmon[51620]: debug 2022-01-31T21:37:21.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:21.892702+0000) 2022-01-31T21:37:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:22 smithi167 conmon[49112]: debug 2022-01-31T21:37:22.066+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.067496+0000) 2022-01-31T21:37:22.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:22 smithi167 conmon[54076]: debug 2022-01-31T21:37:22.066+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.067931+0000) 2022-01-31T21:37:22.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:22 smithi167 conmon[60316]: debug 2022-01-31T21:37:22.066+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.068071+0000) 2022-01-31T21:37:22.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[35325]: debug 2022-01-31T21:37:22.082+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144351 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:22.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[51620]: debug 2022-01-31T21:37:22.067+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.068184+0000) 2022-01-31T21:37:22.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[41853]: debug 2022-01-31T21:37:22.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.012793+0000) 2022-01-31T21:37:22.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[41853]: debug 2022-01-31T21:37:22.067+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.068001+0000) 2022-01-31T21:37:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[46715]: debug 2022-01-31T21:37:22.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.056377+0000) 2022-01-31T21:37:22.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[46715]: debug 2022-01-31T21:37:22.067+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.068601+0000) 2022-01-31T21:37:22.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:22 smithi167 conmon[54076]: debug 2022-01-31T21:37:22.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.284079+0000) 2022-01-31T21:37:22.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:22 smithi167 conmon[60316]: debug 2022-01-31T21:37:22.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.284051+0000) 2022-01-31T21:37:22.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:22 smithi167 conmon[49112]: debug 2022-01-31T21:37:22.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.620357+0000) 2022-01-31T21:37:22.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:22 smithi171 conmon[51620]: debug 2022-01-31T21:37:22.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:22.892857+0000) 2022-01-31T21:37:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:23 smithi171 conmon[41853]: debug 2022-01-31T21:37:23.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:23.012967+0000) 2022-01-31T21:37:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:23 smithi171 conmon[46715]: debug 2022-01-31T21:37:23.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:23.056550+0000) 2022-01-31T21:37:23.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:23 smithi167 conmon[54076]: debug 2022-01-31T21:37:23.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:23.284260+0000) 2022-01-31T21:37:23.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:23 smithi167 conmon[60316]: debug 2022-01-31T21:37:23.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:23.284242+0000) 2022-01-31T21:37:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:23 smithi167 conmon[49112]: debug 2022-01-31T21:37:23.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:23.620482+0000) 2022-01-31T21:37:23.966 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:23 smithi171 conmon[35325]: debug 2022-01-31T21:37:23.710+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:37:23.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:23 smithi171 conmon[51620]: debug 2022-01-31T21:37:23.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:23.892989+0000) 2022-01-31T21:37:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:24 smithi171 conmon[41853]: debug 2022-01-31T21:37:24.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:24.013122+0000) 2022-01-31T21:37:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:24 smithi171 conmon[46715]: debug 2022-01-31T21:37:24.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:24.056689+0000) 2022-01-31T21:37:24.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:24 smithi167 conmon[54076]: debug 2022-01-31T21:37:24.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:24.284467+0000) 2022-01-31T21:37:24.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:24 smithi167 conmon[60316]: debug 2022-01-31T21:37:24.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:24.284446+0000) 2022-01-31T21:37:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:24 smithi167 conmon[49112]: debug 2022-01-31T21:37:24.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:24.620664+0000) 2022-01-31T21:37:24.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:24 smithi171 conmon[51620]: debug 2022-01-31T21:37:24.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:24.893153+0000) 2022-01-31T21:37:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:25 smithi171 conmon[41853]: debug 2022-01-31T21:37:25.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:25.013284+0000) 2022-01-31T21:37:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:25 smithi171 conmon[46715]: debug 2022-01-31T21:37:25.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:25.056845+0000) 2022-01-31T21:37:25.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:25 smithi167 conmon[54076]: debug 2022-01-31T21:37:25.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:25.284653+0000) 2022-01-31T21:37:25.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:25 smithi167 conmon[60316]: debug 2022-01-31T21:37:25.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:25.284616+0000) 2022-01-31T21:37:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:25 smithi167 conmon[49112]: debug 2022-01-31T21:37:25.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:25.620785+0000) 2022-01-31T21:37:25.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:25 smithi171 conmon[51620]: debug 2022-01-31T21:37:25.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:25.893331+0000) 2022-01-31T21:37:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:26 smithi171 conmon[46715]: debug 2022-01-31T21:37:26.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:26.057024+0000) 2022-01-31T21:37:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:26 smithi171 conmon[41853]: debug 2022-01-31T21:37:26.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:26.013500+0000) 2022-01-31T21:37:26.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:26 smithi167 conmon[60316]: debug 2022-01-31T21:37:26.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:26.284771+0000) 2022-01-31T21:37:26.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:26 smithi167 conmon[54076]: debug 2022-01-31T21:37:26.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:26.284822+0000) 2022-01-31T21:37:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:26 smithi167 conmon[49112]: debug 2022-01-31T21:37:26.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:26.620961+0000) 2022-01-31T21:37:26.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:26 smithi171 conmon[51620]: debug 2022-01-31T21:37:26.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:26.893515+0000) 2022-01-31T21:37:27.283 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:27 smithi167 conmon[49112]: debug 2022-01-31T21:37:27.083+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.085400+0000) 2022-01-31T21:37:27.284 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:27 smithi167 conmon[54076]: debug 2022-01-31T21:37:27.083+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.085486+0000) 2022-01-31T21:37:27.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:27 smithi167 conmon[60316]: debug 2022-01-31T21:37:27.084+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.085609+0000) 2022-01-31T21:37:27.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[35325]: debug 2022-01-31T21:37:27.100+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144471 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:27.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[51620]: debug 2022-01-31T21:37:27.086+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.086819+0000) 2022-01-31T21:37:27.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[41853]: debug 2022-01-31T21:37:27.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.013698+0000) 2022-01-31T21:37:27.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[41853]: debug 2022-01-31T21:37:27.084+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.085465+0000) 2022-01-31T21:37:27.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[46715]: debug 2022-01-31T21:37:27.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.057238+0000) 2022-01-31T21:37:27.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[46715]: debug 2022-01-31T21:37:27.085+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.086036+0000) 2022-01-31T21:37:27.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:27 smithi167 conmon[60316]: debug 2022-01-31T21:37:27.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.284997+0000) 2022-01-31T21:37:27.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:27 smithi167 conmon[54076]: debug 2022-01-31T21:37:27.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.284997+0000) 2022-01-31T21:37:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:27 smithi167 conmon[49112]: debug 2022-01-31T21:37:27.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.621136+0000) 2022-01-31T21:37:27.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:27 smithi171 conmon[51620]: debug 2022-01-31T21:37:27.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:27.893706+0000) 2022-01-31T21:37:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:28 smithi171 conmon[46715]: debug 2022-01-31T21:37:28.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:28.057466+0000) 2022-01-31T21:37:28.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:28 smithi171 conmon[41853]: debug 2022-01-31T21:37:28.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:28.013919+0000) 2022-01-31T21:37:28.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:28 smithi167 conmon[54076]: debug 2022-01-31T21:37:28.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:28.285196+0000) 2022-01-31T21:37:28.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:28 smithi167 conmon[60316]: debug 2022-01-31T21:37:28.283+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:28.285184+0000) 2022-01-31T21:37:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:28 smithi167 conmon[49112]: debug 2022-01-31T21:37:28.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:28.621346+0000) 2022-01-31T21:37:28.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:28 smithi171 conmon[51620]: debug 2022-01-31T21:37:28.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:28.893887+0000) 2022-01-31T21:37:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:29 smithi171 conmon[46715]: debug 2022-01-31T21:37:29.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:29.057621+0000) 2022-01-31T21:37:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:29 smithi171 conmon[41853]: debug 2022-01-31T21:37:29.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:29.014081+0000) 2022-01-31T21:37:29.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:29 smithi167 conmon[54076]: debug 2022-01-31T21:37:29.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:29.285363+0000) 2022-01-31T21:37:29.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:29 smithi167 conmon[60316]: debug 2022-01-31T21:37:29.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:29.285370+0000) 2022-01-31T21:37:29.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:29 smithi167 conmon[49112]: debug 2022-01-31T21:37:29.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:29.621547+0000) 2022-01-31T21:37:29.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:29 smithi171 conmon[51620]: debug 2022-01-31T21:37:29.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:29.894058+0000) 2022-01-31T21:37:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:30 smithi171 conmon[46715]: debug 2022-01-31T21:37:30.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:30.057804+0000) 2022-01-31T21:37:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:30 smithi171 conmon[41853]: debug 2022-01-31T21:37:30.012+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:30.014192+0000) 2022-01-31T21:37:30.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:30 smithi167 conmon[54076]: debug 2022-01-31T21:37:30.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:30.285586+0000) 2022-01-31T21:37:30.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:30 smithi167 conmon[60316]: debug 2022-01-31T21:37:30.284+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:30.285580+0000) 2022-01-31T21:37:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:30 smithi167 conmon[49112]: debug 2022-01-31T21:37:30.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:30.621695+0000) 2022-01-31T21:37:30.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:30 smithi171 conmon[51620]: debug 2022-01-31T21:37:30.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:30.894255+0000) 2022-01-31T21:37:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:31 smithi171 conmon[46715]: debug 2022-01-31T21:37:31.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:31.057945+0000) 2022-01-31T21:37:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:31 smithi171 conmon[41853]: debug 2022-01-31T21:37:31.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:31.014376+0000) 2022-01-31T21:37:31.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:31 smithi167 conmon[54076]: debug 2022-01-31T21:37:31.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:31.285818+0000) 2022-01-31T21:37:31.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:31 smithi167 conmon[60316]: debug 2022-01-31T21:37:31.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:31.285821+0000) 2022-01-31T21:37:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:31 smithi167 conmon[49112]: debug 2022-01-31T21:37:31.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:31.621842+0000) 2022-01-31T21:37:31.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:31 smithi171 conmon[51620]: debug 2022-01-31T21:37:31.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:31.894447+0000) 2022-01-31T21:37:32.284 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:32 smithi167 conmon[49112]: debug 2022-01-31T21:37:32.102+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.104020+0000) 2022-01-31T21:37:32.285 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:32 smithi167 conmon[54076]: debug 2022-01-31T21:37:32.101+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.103286+0000) 2022-01-31T21:37:32.285 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:32 smithi167 conmon[60316]: debug 2022-01-31T21:37:32.102+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.104172+0000) 2022-01-31T21:37:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[46715]: debug 2022-01-31T21:37:32.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.058122+0000) 2022-01-31T21:37:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[46715]: debug 2022-01-31T21:37:32.102+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.104089+0000) 2022-01-31T21:37:32.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[51620]: debug 2022-01-31T21:37:32.103+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.104865+0000) 2022-01-31T21:37:32.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[35325]: debug 2022-01-31T21:37:32.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144581 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:32.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[41853]: debug 2022-01-31T21:37:32.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.014563+0000) 2022-01-31T21:37:32.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[41853]: debug 2022-01-31T21:37:32.101+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.103603+0000) 2022-01-31T21:37:32.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:32 smithi167 conmon[60316]: debug 2022-01-31T21:37:32.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.286026+0000) 2022-01-31T21:37:32.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:32 smithi167 conmon[54076]: debug 2022-01-31T21:37:32.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.286019+0000) 2022-01-31T21:37:32.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:32 smithi167 conmon[49112]: debug 2022-01-31T21:37:32.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.622057+0000) 2022-01-31T21:37:32.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:32 smithi171 conmon[51620]: debug 2022-01-31T21:37:32.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:32.894577+0000) 2022-01-31T21:37:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:33 smithi171 conmon[41853]: debug 2022-01-31T21:37:33.013+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:33.014795+0000) 2022-01-31T21:37:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:33 smithi171 conmon[46715]: debug 2022-01-31T21:37:33.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:33.058326+0000) 2022-01-31T21:37:33.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:33 smithi167 conmon[54076]: debug 2022-01-31T21:37:33.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:33.286177+0000) 2022-01-31T21:37:33.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:33 smithi167 conmon[60316]: debug 2022-01-31T21:37:33.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:33.286179+0000) 2022-01-31T21:37:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:33 smithi167 conmon[49112]: debug 2022-01-31T21:37:33.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:33.622212+0000) 2022-01-31T21:37:33.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:33 smithi171 conmon[51620]: debug 2022-01-31T21:37:33.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:33.894681+0000) 2022-01-31T21:37:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:34 smithi171 conmon[41853]: debug 2022-01-31T21:37:34.014+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:34.014950+0000) 2022-01-31T21:37:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:34 smithi171 conmon[46715]: debug 2022-01-31T21:37:34.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:34.058479+0000) 2022-01-31T21:37:34.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:34 smithi167 conmon[54076]: debug 2022-01-31T21:37:34.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:34.286352+0000) 2022-01-31T21:37:34.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:34 smithi167 conmon[60316]: debug 2022-01-31T21:37:34.285+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:34.286352+0000) 2022-01-31T21:37:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:34 smithi167 conmon[49112]: debug 2022-01-31T21:37:34.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:34.622355+0000) 2022-01-31T21:37:34.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:34 smithi171 conmon[51620]: debug 2022-01-31T21:37:34.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:34.894795+0000) 2022-01-31T21:37:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:35 smithi171 conmon[41853]: debug 2022-01-31T21:37:35.014+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:35.015133+0000) 2022-01-31T21:37:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:35 smithi171 conmon[46715]: debug 2022-01-31T21:37:35.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:35.058661+0000) 2022-01-31T21:37:35.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:35 smithi167 conmon[60316]: debug 2022-01-31T21:37:35.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:35.286563+0000) 2022-01-31T21:37:35.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:35 smithi167 conmon[54076]: debug 2022-01-31T21:37:35.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:35.286565+0000) 2022-01-31T21:37:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:35 smithi167 conmon[49112]: debug 2022-01-31T21:37:35.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:35.622572+0000) 2022-01-31T21:37:35.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:35 smithi171 conmon[51620]: debug 2022-01-31T21:37:35.894+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:35.894952+0000) 2022-01-31T21:37:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:36 smithi171 conmon[41853]: debug 2022-01-31T21:37:36.014+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:36.015313+0000) 2022-01-31T21:37:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:36 smithi171 conmon[46715]: debug 2022-01-31T21:37:36.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:36.058874+0000) 2022-01-31T21:37:36.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:36 smithi167 conmon[54076]: debug 2022-01-31T21:37:36.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:36.286765+0000) 2022-01-31T21:37:36.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:36 smithi167 conmon[60316]: debug 2022-01-31T21:37:36.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:36.286766+0000) 2022-01-31T21:37:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:36 smithi167 conmon[49112]: debug 2022-01-31T21:37:36.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:36.622751+0000) 2022-01-31T21:37:37.285 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[49112]: debug 2022-01-31T21:37:37.121+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.122032+0000) 2022-01-31T21:37:37.286 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[54076]: debug 2022-01-31T21:37:37.121+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.122291+0000) 2022-01-31T21:37:37.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[60316]: debug 2022-01-31T21:37:37.287 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[60316]: 2022-01-31T21:37:37.122+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.122594+0000) 2022-01-31T21:37:37.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[35325]: debug 2022-01-31T21:37:37.135+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144692 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:37.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[41853]: debug 2022-01-31T21:37:37.014+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.015536+0000) 2022-01-31T21:37:37.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[41853]: debug 2022-01-31T21:37:37.119+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.121506+0000) 2022-01-31T21:37:37.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[46715]: debug 2022-01-31T21:37:37.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.059079+0000) 2022-01-31T21:37:37.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[46715]: debug 2022-01-31T21:37:37.120+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.122145+0000) 2022-01-31T21:37:37.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:36 smithi171 conmon[51620]: debug 2022-01-31T21:37:36.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:36.895164+0000) 2022-01-31T21:37:37.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[51620]: debug 2022-01-31T21:37:37.119+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.121739+0000) 2022-01-31T21:37:37.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[54076]: debug 2022-01-31T21:37:37.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.286958+0000) 2022-01-31T21:37:37.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[60316]: debug 2022-01-31T21:37:37.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.286982+0000) 2022-01-31T21:37:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:37 smithi167 conmon[49112]: debug 2022-01-31T21:37:37.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.622935+0000) 2022-01-31T21:37:37.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:37 smithi171 conmon[51620]: debug 2022-01-31T21:37:37.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:37.895289+0000) 2022-01-31T21:37:38.303 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:38 smithi171 conmon[41853]: debug 2022-01-31T21:37:38.014+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:38.015761+0000) 2022-01-31T21:37:38.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:38 smithi171 conmon[46715]: debug 2022-01-31T21:37:38.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:38.059261+0000) 2022-01-31T21:37:38.586 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:38 smithi167 conmon[60316]: debug 2022-01-31T21:37:38.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:38.287150+0000) 2022-01-31T21:37:38.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:38 smithi167 conmon[54076]: debug 2022-01-31T21:37:38.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:38.287140+0000) 2022-01-31T21:37:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:38 smithi167 conmon[49112]: debug 2022-01-31T21:37:38.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:38.623106+0000) 2022-01-31T21:37:38.969 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:38 smithi171 conmon[35325]: debug 2022-01-31T21:37:38.710+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:37:38.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:38 smithi171 conmon[51620]: debug 2022-01-31T21:37:38.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:38.895460+0000) 2022-01-31T21:37:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:39 smithi171 conmon[41853]: debug 2022-01-31T21:37:39.015+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:39.015918+0000) 2022-01-31T21:37:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:39 smithi171 conmon[46715]: debug 2022-01-31T21:37:39.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:39.059414+0000) 2022-01-31T21:37:39.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:39 smithi167 conmon[54076]: debug 2022-01-31T21:37:39.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:39.287334+0000) 2022-01-31T21:37:39.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:39 smithi167 conmon[60316]: debug 2022-01-31T21:37:39.286+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:39.287339+0000) 2022-01-31T21:37:39.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:39 smithi167 conmon[49112]: debug 2022-01-31T21:37:39.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:39.623309+0000) 2022-01-31T21:37:39.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:39 smithi171 conmon[51620]: debug 2022-01-31T21:37:39.894+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:39.895651+0000) 2022-01-31T21:37:40.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:40 smithi171 conmon[46715]: debug 2022-01-31T21:37:40.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:40.059601+0000) 2022-01-31T21:37:40.352 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:40 smithi171 conmon[41853]: debug 2022-01-31T21:37:40.015+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:40.016097+0000) 2022-01-31T21:37:40.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:40 smithi167 conmon[54076]: debug 2022-01-31T21:37:40.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:40.287542+0000) 2022-01-31T21:37:40.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:40 smithi167 conmon[60316]: debug 2022-01-31T21:37:40.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:40.287547+0000) 2022-01-31T21:37:40.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:40 smithi167 conmon[49112]: debug 2022-01-31T21:37:40.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:40.623483+0000) 2022-01-31T21:37:40.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:40 smithi171 conmon[51620]: debug 2022-01-31T21:37:40.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:40.895838+0000) 2022-01-31T21:37:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:41 smithi171 conmon[46715]: debug 2022-01-31T21:37:41.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:41.059825+0000) 2022-01-31T21:37:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:41 smithi171 conmon[41853]: debug 2022-01-31T21:37:41.015+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:41.016282+0000) 2022-01-31T21:37:41.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:41 smithi167 conmon[54076]: debug 2022-01-31T21:37:41.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:41.287744+0000) 2022-01-31T21:37:41.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:41 smithi167 conmon[60316]: debug 2022-01-31T21:37:41.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:41.287752+0000) 2022-01-31T21:37:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:41 smithi167 conmon[49112]: debug 2022-01-31T21:37:41.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:41.623631+0000) 2022-01-31T21:37:41.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:41 smithi171 conmon[51620]: debug 2022-01-31T21:37:41.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:41.895962+0000) 2022-01-31T21:37:42.286 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:42 smithi167 conmon[49112]: debug 2022-01-31T21:37:42.138+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.138545+0000) 2022-01-31T21:37:42.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:42 smithi167 conmon[54076]: debug 2022-01-31T21:37:42.139+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.139595+0000) 2022-01-31T21:37:42.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:42 smithi167 conmon[60316]: debug 2022-01-31T21:37:42.138+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.139147+0000) 2022-01-31T21:37:42.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[46715]: debug 2022-01-31T21:37:42.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.060011+0000) 2022-01-31T21:37:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[46715]: debug 2022-01-31T21:37:42.139+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.140607+0000) 2022-01-31T21:37:42.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[51620]: debug 2022-01-31T21:37:42.138+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.138805+0000) 2022-01-31T21:37:42.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[35325]: debug 2022-01-31T21:37:42.154+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144803 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:42.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[41853]: debug 2022-01-31T21:37:42.015+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.016490+0000) 2022-01-31T21:37:42.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[41853]: debug 2022-01-31T21:37:42.138+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.139574+0000) 2022-01-31T21:37:42.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:42 smithi167 conmon[60316]: debug 2022-01-31T21:37:42.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.287969+0000) 2022-01-31T21:37:42.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:42 smithi167 conmon[54076]: debug 2022-01-31T21:37:42.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.287930+0000) 2022-01-31T21:37:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:42 smithi167 conmon[49112]: debug 2022-01-31T21:37:42.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.623799+0000) 2022-01-31T21:37:42.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:42 smithi171 conmon[51620]: debug 2022-01-31T21:37:42.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:42.896123+0000) 2022-01-31T21:37:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:43 smithi171 conmon[41853]: debug 2022-01-31T21:37:43.015+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:43.016685+0000) 2022-01-31T21:37:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:43 smithi171 conmon[46715]: debug 2022-01-31T21:37:43.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:43.060207+0000) 2022-01-31T21:37:43.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:43 smithi167 conmon[54076]: debug 2022-01-31T21:37:43.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:43.288059+0000) 2022-01-31T21:37:43.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:43 smithi167 conmon[60316]: debug 2022-01-31T21:37:43.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:43.288108+0000) 2022-01-31T21:37:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:43 smithi167 conmon[49112]: debug 2022-01-31T21:37:43.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:43.623963+0000) 2022-01-31T21:37:43.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:43 smithi171 conmon[51620]: debug 2022-01-31T21:37:43.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:43.896284+0000) 2022-01-31T21:37:44.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:44 smithi171 conmon[41853]: debug 2022-01-31T21:37:44.016+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:44.016835+0000) 2022-01-31T21:37:44.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:44 smithi171 conmon[46715]: debug 2022-01-31T21:37:44.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:44.060367+0000) 2022-01-31T21:37:44.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:44 smithi167 conmon[60316]: debug 2022-01-31T21:37:44.287+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:44.288280+0000) 2022-01-31T21:37:44.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:44 smithi167 conmon[54076]: debug 2022-01-31T21:37:44.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:44.288255+0000) 2022-01-31T21:37:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:44 smithi167 conmon[49112]: debug 2022-01-31T21:37:44.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:44.624129+0000) 2022-01-31T21:37:44.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:44 smithi171 conmon[51620]: debug 2022-01-31T21:37:44.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:44.896498+0000) 2022-01-31T21:37:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:45 smithi171 conmon[41853]: debug 2022-01-31T21:37:45.016+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:45.017013+0000) 2022-01-31T21:37:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:45 smithi171 conmon[46715]: debug 2022-01-31T21:37:45.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:45.060521+0000) 2022-01-31T21:37:45.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:45 smithi167 conmon[54076]: debug 2022-01-31T21:37:45.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:45.288442+0000) 2022-01-31T21:37:45.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:45 smithi167 conmon[60316]: debug 2022-01-31T21:37:45.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:45.288438+0000) 2022-01-31T21:37:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:45 smithi167 conmon[49112]: debug 2022-01-31T21:37:45.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:45.624344+0000) 2022-01-31T21:37:45.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:45 smithi171 conmon[51620]: debug 2022-01-31T21:37:45.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:45.896650+0000) 2022-01-31T21:37:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:46 smithi171 conmon[41853]: debug 2022-01-31T21:37:46.016+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:46.017214+0000) 2022-01-31T21:37:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:46 smithi171 conmon[46715]: debug 2022-01-31T21:37:46.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:46.060727+0000) 2022-01-31T21:37:46.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:46 smithi167 conmon[54076]: debug 2022-01-31T21:37:46.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:46.288621+0000) 2022-01-31T21:37:46.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:46 smithi167 conmon[60316]: debug 2022-01-31T21:37:46.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:46.288649+0000) 2022-01-31T21:37:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:46 smithi167 conmon[49112]: debug 2022-01-31T21:37:46.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:46.624586+0000) 2022-01-31T21:37:46.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:46 smithi171 conmon[51620]: debug 2022-01-31T21:37:46.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:46.896842+0000) 2022-01-31T21:37:47.287 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:47 smithi167 conmon[49112]: debug 2022-01-31T21:37:47.156+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.157072+0000) 2022-01-31T21:37:47.288 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:47 smithi167 conmon[54076]: debug 2022-01-31T21:37:47.157+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.157512+0000) 2022-01-31T21:37:47.288 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:47 smithi167 conmon[60316]: debug 2022-01-31T21:37:47.157+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.157582+0000) 2022-01-31T21:37:47.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[35325]: debug 2022-01-31T21:37:47.172+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 144914 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:47.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[51620]: debug 2022-01-31T21:37:47.156+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.156969+0000) 2022-01-31T21:37:47.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[41853]: debug 2022-01-31T21:37:47.016+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.017397+0000) 2022-01-31T21:37:47.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[41853]: debug 2022-01-31T21:37:47.156+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.156987+0000) 2022-01-31T21:37:47.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[46715]: debug 2022-01-31T21:37:47.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.060944+0000) 2022-01-31T21:37:47.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[46715]: debug 2022-01-31T21:37:47.157+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.158290+0000) 2022-01-31T21:37:47.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:47 smithi167 conmon[54076]: debug 2022-01-31T21:37:47.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.288787+0000) 2022-01-31T21:37:47.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:47 smithi167 conmon[60316]: debug 2022-01-31T21:37:47.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.288782+0000) 2022-01-31T21:37:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:47 smithi167 conmon[49112]: debug 2022-01-31T21:37:47.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.624747+0000) 2022-01-31T21:37:47.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:47 smithi171 conmon[51620]: debug 2022-01-31T21:37:47.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:47.897056+0000) 2022-01-31T21:37:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:48 smithi171 conmon[41853]: debug 2022-01-31T21:37:48.016+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:48.017526+0000) 2022-01-31T21:37:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:48 smithi171 conmon[46715]: debug 2022-01-31T21:37:48.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:48.061076+0000) 2022-01-31T21:37:48.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:48 smithi167 conmon[54076]: debug 2022-01-31T21:37:48.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:48.288931+0000) 2022-01-31T21:37:48.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:48 smithi167 conmon[60316]: debug 2022-01-31T21:37:48.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:48.288932+0000) 2022-01-31T21:37:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:48 smithi167 conmon[49112]: debug 2022-01-31T21:37:48.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:48.624892+0000) 2022-01-31T21:37:48.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:48 smithi171 conmon[51620]: debug 2022-01-31T21:37:48.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:48.897218+0000) 2022-01-31T21:37:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:49 smithi171 conmon[41853]: debug 2022-01-31T21:37:49.016+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:49.017673+0000) 2022-01-31T21:37:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:49 smithi171 conmon[46715]: debug 2022-01-31T21:37:49.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:49.061221+0000) 2022-01-31T21:37:49.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:49 smithi167 conmon[54076]: debug 2022-01-31T21:37:49.288+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:49.289131+0000) 2022-01-31T21:37:49.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:49 smithi167 conmon[60316]: debug 2022-01-31T21:37:49.288+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:49.289142+0000) 2022-01-31T21:37:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:49 smithi167 conmon[49112]: debug 2022-01-31T21:37:49.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:49.625091+0000) 2022-01-31T21:37:49.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:49 smithi171 conmon[51620]: debug 2022-01-31T21:37:49.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:49.897388+0000) 2022-01-31T21:37:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:50 smithi171 conmon[41853]: debug 2022-01-31T21:37:50.017+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:50.017864+0000) 2022-01-31T21:37:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:50 smithi171 conmon[46715]: debug 2022-01-31T21:37:50.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:50.061454+0000) 2022-01-31T21:37:50.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:50 smithi167 conmon[60316]: debug 2022-01-31T21:37:50.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:50.289288+0000) 2022-01-31T21:37:50.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:50 smithi167 conmon[54076]: debug 2022-01-31T21:37:50.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:50.289286+0000) 2022-01-31T21:37:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:50 smithi167 conmon[49112]: debug 2022-01-31T21:37:50.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:50.625277+0000) 2022-01-31T21:37:50.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:50 smithi171 conmon[51620]: debug 2022-01-31T21:37:50.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:50.897556+0000) 2022-01-31T21:37:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:51 smithi171 conmon[41853]: debug 2022-01-31T21:37:51.017+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:51.018051+0000) 2022-01-31T21:37:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:51 smithi171 conmon[46715]: debug 2022-01-31T21:37:51.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:51.061641+0000) 2022-01-31T21:37:51.604 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:51 smithi167 conmon[60316]: debug 2022-01-31T21:37:51.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:51.289482+0000) 2022-01-31T21:37:51.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:51 smithi167 conmon[54076]: debug 2022-01-31T21:37:51.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:51.289474+0000) 2022-01-31T21:37:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:51 smithi167 conmon[49112]: debug 2022-01-31T21:37:51.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:51.625457+0000) 2022-01-31T21:37:51.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:51 smithi171 conmon[51620]: debug 2022-01-31T21:37:51.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:51.897749+0000) 2022-01-31T21:37:52.288 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:52 smithi167 conmon[49112]: debug 2022-01-31T21:37:52.174+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.175091+0000) 2022-01-31T21:37:52.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:52 smithi167 conmon[54076]: debug 2022-01-31T21:37:52.174+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.174833+0000) 2022-01-31T21:37:52.289 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:52 smithi167 conmon[60316]: debug 2022-01-31T21:37:52.174+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.175259+0000) 2022-01-31T21:37:52.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[35325]: debug 2022-01-31T21:37:52.189+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145024 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:52.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[51620]: debug 2022-01-31T21:37:52.174+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.174919+0000) 2022-01-31T21:37:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[46715]: debug 2022-01-31T21:37:52.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.061832+0000) 2022-01-31T21:37:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[46715]: debug 2022-01-31T21:37:52.174+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.174943+0000) 2022-01-31T21:37:52.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[41853]: debug 2022-01-31T21:37:52.017+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.018230+0000) 2022-01-31T21:37:52.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[41853]: debug 2022-01-31T21:37:52.173+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.174665+0000) 2022-01-31T21:37:52.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:52 smithi167 conmon[54076]: debug 2022-01-31T21:37:52.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.289703+0000) 2022-01-31T21:37:52.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:52 smithi167 conmon[60316]: debug 2022-01-31T21:37:52.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.289703+0000) 2022-01-31T21:37:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:52 smithi167 conmon[49112]: debug 2022-01-31T21:37:52.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.625645+0000) 2022-01-31T21:37:52.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:52 smithi171 conmon[51620]: debug 2022-01-31T21:37:52.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:52.897914+0000) 2022-01-31T21:37:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:53 smithi171 conmon[41853]: debug 2022-01-31T21:37:53.017+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:53.018452+0000) 2022-01-31T21:37:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:53 smithi171 conmon[46715]: debug 2022-01-31T21:37:53.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:53.061994+0000) 2022-01-31T21:37:53.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:53 smithi167 conmon[54076]: debug 2022-01-31T21:37:53.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:53.289891+0000) 2022-01-31T21:37:53.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:53 smithi167 conmon[60316]: debug 2022-01-31T21:37:53.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:53.289891+0000) 2022-01-31T21:37:53.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:53 smithi167 conmon[49112]: debug 2022-01-31T21:37:53.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:53.625773+0000) 2022-01-31T21:37:53.971 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:53 smithi171 conmon[35325]: debug 2022-01-31T21:37:53.711+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:37:53.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:53 smithi171 conmon[51620]: debug 2022-01-31T21:37:53.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:53.898029+0000) 2022-01-31T21:37:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:54 smithi171 conmon[41853]: debug 2022-01-31T21:37:54.017+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:54.018620+0000) 2022-01-31T21:37:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:54 smithi171 conmon[46715]: debug 2022-01-31T21:37:54.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:54.062146+0000) 2022-01-31T21:37:54.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:54 smithi167 conmon[54076]: debug 2022-01-31T21:37:54.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:54.290095+0000) 2022-01-31T21:37:54.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:54 smithi167 conmon[60316]: debug 2022-01-31T21:37:54.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:54.290092+0000) 2022-01-31T21:37:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:54 smithi167 conmon[49112]: debug 2022-01-31T21:37:54.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:54.625934+0000) 2022-01-31T21:37:54.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:54 smithi171 conmon[51620]: debug 2022-01-31T21:37:54.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:54.898220+0000) 2022-01-31T21:37:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:55 smithi171 conmon[46715]: debug 2022-01-31T21:37:55.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:55.062355+0000) 2022-01-31T21:37:55.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:55 smithi171 conmon[41853]: debug 2022-01-31T21:37:55.018+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:55.018829+0000) 2022-01-31T21:37:55.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:55 smithi167 conmon[54076]: debug 2022-01-31T21:37:55.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:55.290302+0000) 2022-01-31T21:37:55.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:55 smithi167 conmon[60316]: debug 2022-01-31T21:37:55.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:55.290302+0000) 2022-01-31T21:37:55.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:55 smithi167 conmon[49112]: debug 2022-01-31T21:37:55.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:55.626127+0000) 2022-01-31T21:37:55.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:55 smithi171 conmon[51620]: debug 2022-01-31T21:37:55.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:55.898427+0000) 2022-01-31T21:37:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:56 smithi171 conmon[46715]: debug 2022-01-31T21:37:56.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:56.062545+0000) 2022-01-31T21:37:56.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:56 smithi171 conmon[41853]: debug 2022-01-31T21:37:56.018+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:56.019036+0000) 2022-01-31T21:37:56.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:56 smithi167 conmon[54076]: debug 2022-01-31T21:37:56.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:56.290497+0000) 2022-01-31T21:37:56.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:56 smithi167 conmon[60316]: debug 2022-01-31T21:37:56.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:56.290508+0000) 2022-01-31T21:37:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:56 smithi167 conmon[49112]: debug 2022-01-31T21:37:56.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:56.626341+0000) 2022-01-31T21:37:56.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:56 smithi171 conmon[51620]: debug 2022-01-31T21:37:56.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:56.898618+0000) 2022-01-31T21:37:57.289 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:57 smithi167 conmon[49112]: debug 2022-01-31T21:37:57.191+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.193408+0000) 2022-01-31T21:37:57.290 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:57 smithi167 conmon[54076]: debug 2022-01-31T21:37:57.190+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.192168+0000) 2022-01-31T21:37:57.290 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:57 smithi167 conmon[60316]: debug 2022-01-31T21:37:57.191+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.192898+0000) 2022-01-31T21:37:57.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[35325]: debug 2022-01-31T21:37:57.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145136 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:37:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[46715]: debug 2022-01-31T21:37:57.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.062732+0000) 2022-01-31T21:37:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[46715]: debug 2022-01-31T21:37:57.192+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.193287+0000) 2022-01-31T21:37:57.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[51620]: debug 2022-01-31T21:37:57.191+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.192377+0000) 2022-01-31T21:37:57.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[41853]: debug 2022-01-31T21:37:57.018+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.019195+0000) 2022-01-31T21:37:57.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[41853]: debug 2022-01-31T21:37:57.191+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.192634+0000) 2022-01-31T21:37:57.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:57 smithi167 conmon[60316]: debug 2022-01-31T21:37:57.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.290680+0000) 2022-01-31T21:37:57.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:57 smithi167 conmon[54076]: debug 2022-01-31T21:37:57.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.290675+0000) 2022-01-31T21:37:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:57 smithi167 conmon[49112]: debug 2022-01-31T21:37:57.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.626541+0000) 2022-01-31T21:37:57.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:57 smithi171 conmon[51620]: debug 2022-01-31T21:37:57.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:57.898826+0000) 2022-01-31T21:37:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:58 smithi171 conmon[41853]: debug 2022-01-31T21:37:58.018+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:58.019352+0000) 2022-01-31T21:37:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:58 smithi171 conmon[46715]: debug 2022-01-31T21:37:58.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:58.062884+0000) 2022-01-31T21:37:58.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:58 smithi167 conmon[54076]: debug 2022-01-31T21:37:58.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:58.290805+0000) 2022-01-31T21:37:58.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:58 smithi167 conmon[60316]: debug 2022-01-31T21:37:58.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:58.290809+0000) 2022-01-31T21:37:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:58 smithi167 conmon[49112]: debug 2022-01-31T21:37:58.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:58.626668+0000) 2022-01-31T21:37:58.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:58 smithi171 conmon[51620]: debug 2022-01-31T21:37:58.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:58.899003+0000) 2022-01-31T21:37:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:37:59 smithi171 conmon[41853]: debug 2022-01-31T21:37:59.019+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:59.019501+0000) 2022-01-31T21:37:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:37:59 smithi171 conmon[46715]: debug 2022-01-31T21:37:59.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:59.062982+0000) 2022-01-31T21:37:59.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:37:59 smithi167 conmon[54076]: debug 2022-01-31T21:37:59.289+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:59.290974+0000) 2022-01-31T21:37:59.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:37:59 smithi167 conmon[60316]: debug 2022-01-31T21:37:59.289+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:59.290974+0000) 2022-01-31T21:37:59.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:37:59 smithi167 conmon[49112]: debug 2022-01-31T21:37:59.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:59.626785+0000) 2022-01-31T21:37:59.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:37:59 smithi171 conmon[51620]: debug 2022-01-31T21:37:59.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:37:59.899211+0000) 2022-01-31T21:38:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:00 smithi171 conmon[41853]: debug 2022-01-31T21:38:00.018+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:00.019700+0000) 2022-01-31T21:38:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:00 smithi171 conmon[46715]: debug 2022-01-31T21:38:00.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:00.063141+0000) 2022-01-31T21:38:00.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:00 smithi167 conmon[60316]: debug 2022-01-31T21:38:00.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:00.291178+0000) 2022-01-31T21:38:00.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:00 smithi167 conmon[54076]: debug 2022-01-31T21:38:00.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:00.291179+0000) 2022-01-31T21:38:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:00 smithi167 conmon[49112]: debug 2022-01-31T21:38:00.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:00.626962+0000) 2022-01-31T21:38:00.973 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:00 smithi171 conmon[51620]: debug 2022-01-31T21:38:00.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:00.899379+0000) 2022-01-31T21:38:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:01 smithi171 conmon[41853]: debug 2022-01-31T21:38:01.019+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:01.019876+0000) 2022-01-31T21:38:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:01 smithi171 conmon[46715]: debug 2022-01-31T21:38:01.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:01.063347+0000) 2022-01-31T21:38:01.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:01 smithi167 conmon[54076]: debug 2022-01-31T21:38:01.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:01.291447+0000) 2022-01-31T21:38:01.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:01 smithi167 conmon[60316]: debug 2022-01-31T21:38:01.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:01.291337+0000) 2022-01-31T21:38:01.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:01 smithi167 conmon[49112]: debug 2022-01-31T21:38:01.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:01.627150+0000) 2022-01-31T21:38:01.973 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:01 smithi171 conmon[51620]: debug 2022-01-31T21:38:01.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:01.899606+0000) 2022-01-31T21:38:02.224 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[51620]: debug 2022-01-31T21:38:02.210+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.211500+0000) 2022-01-31T21:38:02.224 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[41853]: debug 2022-01-31T21:38:02.019+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.020003+0000) 2022-01-31T21:38:02.225 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[41853]: debug 2022-01-31T21:38:02.209+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.210650+0000) 2022-01-31T21:38:02.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[46715]: debug 2022-01-31T21:38:02.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.063530+0000) 2022-01-31T21:38:02.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[46715]: debug 2022-01-31T21:38:02.210+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.211507+0000) 2022-01-31T21:38:02.290 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:02 smithi167 conmon[49112]: debug 2022-01-31T21:38:02.209+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.211481+0000) 2022-01-31T21:38:02.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:02 smithi167 conmon[54076]: debug 2022-01-31T21:38:02.209+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.210851+0000) 2022-01-31T21:38:02.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:02 smithi167 conmon[60316]: debug 2022-01-31T21:38:02.209+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.211092+0000) 2022-01-31T21:38:02.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[35325]: debug 2022-01-31T21:38:02.225+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145245 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:02.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:02 smithi167 conmon[60316]: debug 2022-01-31T21:38:02.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.291539+0000) 2022-01-31T21:38:02.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:02 smithi167 conmon[54076]: debug 2022-01-31T21:38:02.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.291600+0000) 2022-01-31T21:38:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:02 smithi167 conmon[49112]: debug 2022-01-31T21:38:02.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.627329+0000) 2022-01-31T21:38:02.973 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:02 smithi171 conmon[51620]: debug 2022-01-31T21:38:02.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:02.899803+0000) 2022-01-31T21:38:03.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:03 smithi171 conmon[41853]: debug 2022-01-31T21:38:03.019+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:03.020149+0000) 2022-01-31T21:38:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:03 smithi171 conmon[46715]: debug 2022-01-31T21:38:03.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:03.063741+0000) 2022-01-31T21:38:03.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:03 smithi167 conmon[54076]: debug 2022-01-31T21:38:03.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:03.291792+0000) 2022-01-31T21:38:03.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:03 smithi167 conmon[60316]: debug 2022-01-31T21:38:03.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:03.291698+0000) 2022-01-31T21:38:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:03 smithi167 conmon[49112]: debug 2022-01-31T21:38:03.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:03.627475+0000) 2022-01-31T21:38:03.973 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:03 smithi171 conmon[51620]: debug 2022-01-31T21:38:03.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:03.899979+0000) 2022-01-31T21:38:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:04 smithi171 conmon[41853]: debug 2022-01-31T21:38:04.019+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:04.020283+0000) 2022-01-31T21:38:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:04 smithi171 conmon[46715]: debug 2022-01-31T21:38:04.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:04.063867+0000) 2022-01-31T21:38:04.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:04 smithi167 conmon[54076]: debug 2022-01-31T21:38:04.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:04.291939+0000) 2022-01-31T21:38:04.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:04 smithi167 conmon[60316]: debug 2022-01-31T21:38:04.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:04.291853+0000) 2022-01-31T21:38:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:04 smithi167 conmon[49112]: debug 2022-01-31T21:38:04.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:04.627636+0000) 2022-01-31T21:38:04.973 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:04 smithi171 conmon[51620]: debug 2022-01-31T21:38:04.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:04.900209+0000) 2022-01-31T21:38:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:05 smithi171 conmon[41853]: debug 2022-01-31T21:38:05.020+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:05.020505+0000) 2022-01-31T21:38:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:05 smithi171 conmon[46715]: debug 2022-01-31T21:38:05.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:05.064021+0000) 2022-01-31T21:38:05.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:05 smithi167 conmon[54076]: debug 2022-01-31T21:38:05.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:05.292049+0000) 2022-01-31T21:38:05.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:05 smithi167 conmon[60316]: debug 2022-01-31T21:38:05.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:05.291958+0000) 2022-01-31T21:38:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:05 smithi167 conmon[49112]: debug 2022-01-31T21:38:05.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:05.627838+0000) 2022-01-31T21:38:05.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:05 smithi171 conmon[51620]: debug 2022-01-31T21:38:05.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:05.900401+0000) 2022-01-31T21:38:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:06 smithi171 conmon[41853]: debug 2022-01-31T21:38:06.020+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:06.020707+0000) 2022-01-31T21:38:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:06 smithi171 conmon[46715]: debug 2022-01-31T21:38:06.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:06.064207+0000) 2022-01-31T21:38:06.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:06 smithi167 conmon[54076]: debug 2022-01-31T21:38:06.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:06.292171+0000) 2022-01-31T21:38:06.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:06 smithi167 conmon[60316]: debug 2022-01-31T21:38:06.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:06.292081+0000) 2022-01-31T21:38:06.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:06 smithi167 conmon[49112]: debug 2022-01-31T21:38:06.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:06.628030+0000) 2022-01-31T21:38:06.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:06 smithi171 conmon[51620]: debug 2022-01-31T21:38:06.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:06.900613+0000) 2022-01-31T21:38:07.226 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[41853]: debug 2022-01-31T21:38:07.020+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.020860+0000) 2022-01-31T21:38:07.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[46715]: debug 2022-01-31T21:38:07.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.064371+0000) 2022-01-31T21:38:07.290 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:07 smithi167 conmon[49112]: debug 2022-01-31T21:38:07.227+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.229300+0000) 2022-01-31T21:38:07.291 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:07 smithi167 conmon[54076]: debug 2022-01-31T21:38:07.226+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.228068+0000) 2022-01-31T21:38:07.292 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:07 smithi167 conmon[60316]: debug 2022-01-31T21:38:07.226+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.227678+0000) 2022-01-31T21:38:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[41853]: debug 2022-01-31T21:38:07.229+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.230385+0000) 2022-01-31T21:38:07.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[46715]: debug 2022-01-31T21:38:07.227+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.228501+0000) 2022-01-31T21:38:07.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[51620]: debug 2022-01-31T21:38:07.228+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.228785+0000) 2022-01-31T21:38:07.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[35325]: debug 2022-01-31T21:38:07.244+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145356 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:07.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:07 smithi167 conmon[54076]: debug 2022-01-31T21:38:07.290+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.292334+0000) 2022-01-31T21:38:07.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:07 smithi167 conmon[60316]: debug 2022-01-31T21:38:07.290+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.292274+0000) 2022-01-31T21:38:07.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:07 smithi167 conmon[49112]: debug 2022-01-31T21:38:07.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.628207+0000) 2022-01-31T21:38:07.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:07 smithi171 conmon[51620]: debug 2022-01-31T21:38:07.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:07.900773+0000) 2022-01-31T21:38:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:08 smithi171 conmon[46715]: debug 2022-01-31T21:38:08.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:08.064583+0000) 2022-01-31T21:38:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:08 smithi171 conmon[41853]: debug 2022-01-31T21:38:08.020+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:08.021048+0000) 2022-01-31T21:38:08.608 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:08 smithi167 conmon[54076]: debug 2022-01-31T21:38:08.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:08.292545+0000) 2022-01-31T21:38:08.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:08 smithi167 conmon[60316]: debug 2022-01-31T21:38:08.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:08.292434+0000) 2022-01-31T21:38:08.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:08 smithi167 conmon[49112]: debug 2022-01-31T21:38:08.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:08.628390+0000) 2022-01-31T21:38:08.974 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:08 smithi171 conmon[35325]: debug 2022-01-31T21:38:08.712+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:38:08.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:08 smithi171 conmon[51620]: debug 2022-01-31T21:38:08.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:08.900938+0000) 2022-01-31T21:38:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:09 smithi171 conmon[41853]: debug 2022-01-31T21:38:09.020+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:09.021219+0000) 2022-01-31T21:38:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:09 smithi171 conmon[46715]: debug 2022-01-31T21:38:09.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:09.064733+0000) 2022-01-31T21:38:09.608 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:09 smithi167 conmon[54076]: debug 2022-01-31T21:38:09.291+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:09.292753+0000) 2022-01-31T21:38:09.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:09 smithi167 conmon[60316]: debug 2022-01-31T21:38:09.291+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:09.292620+0000) 2022-01-31T21:38:09.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:09 smithi167 conmon[49112]: debug 2022-01-31T21:38:09.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:09.628578+0000) 2022-01-31T21:38:09.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:09 smithi171 conmon[51620]: debug 2022-01-31T21:38:09.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:09.901127+0000) 2022-01-31T21:38:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:10 smithi171 conmon[46715]: debug 2022-01-31T21:38:10.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:10.064907+0000) 2022-01-31T21:38:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:10 smithi171 conmon[41853]: debug 2022-01-31T21:38:10.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:10.021384+0000) 2022-01-31T21:38:10.608 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:10 smithi167 conmon[54076]: debug 2022-01-31T21:38:10.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:10.292945+0000) 2022-01-31T21:38:10.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:10 smithi167 conmon[60316]: debug 2022-01-31T21:38:10.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:10.292836+0000) 2022-01-31T21:38:10.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:10 smithi167 conmon[49112]: debug 2022-01-31T21:38:10.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:10.628722+0000) 2022-01-31T21:38:10.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:10 smithi171 conmon[51620]: debug 2022-01-31T21:38:10.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:10.901322+0000) 2022-01-31T21:38:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:11 smithi171 conmon[41853]: debug 2022-01-31T21:38:11.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:11.021537+0000) 2022-01-31T21:38:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:11 smithi171 conmon[46715]: debug 2022-01-31T21:38:11.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:11.065082+0000) 2022-01-31T21:38:11.608 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:11 smithi167 conmon[60316]: debug 2022-01-31T21:38:11.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:11.293088+0000) 2022-01-31T21:38:11.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:11 smithi167 conmon[54076]: debug 2022-01-31T21:38:11.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:11.293157+0000) 2022-01-31T21:38:11.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:11 smithi167 conmon[49112]: debug 2022-01-31T21:38:11.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:11.628869+0000) 2022-01-31T21:38:11.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:11 smithi171 conmon[51620]: debug 2022-01-31T21:38:11.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:11.901506+0000) 2022-01-31T21:38:12.244 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[46715]: debug 2022-01-31T21:38:12.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.065263+0000) 2022-01-31T21:38:12.245 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[41853]: debug 2022-01-31T21:38:12.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.021724+0000) 2022-01-31T21:38:12.292 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:12 smithi167 conmon[49112]: debug 2022-01-31T21:38:12.253+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.254711+0000) 2022-01-31T21:38:12.293 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:12 smithi167 conmon[54076]: debug 2022-01-31T21:38:12.244+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.245928+0000) 2022-01-31T21:38:12.294 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:12 smithi167 conmon[60316]: debug 2022-01-31T21:38:12.251+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.252890+0000) 2022-01-31T21:38:12.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[35325]: debug 2022-01-31T21:38:12.267+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145467 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[41853]: debug 2022-01-31T21:38:12.247+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.248714+0000) 2022-01-31T21:38:12.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[46715]: debug 2022-01-31T21:38:12.245+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.246714+0000) 2022-01-31T21:38:12.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[51620]: debug 2022-01-31T21:38:12.246+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.247559+0000) 2022-01-31T21:38:12.608 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:12 smithi167 conmon[54076]: debug 2022-01-31T21:38:12.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.293344+0000) 2022-01-31T21:38:12.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:12 smithi167 conmon[60316]: debug 2022-01-31T21:38:12.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.293263+0000) 2022-01-31T21:38:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:12 smithi167 conmon[49112]: debug 2022-01-31T21:38:12.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.629073+0000) 2022-01-31T21:38:12.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:12 smithi171 conmon[51620]: debug 2022-01-31T21:38:12.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:12.901665+0000) 2022-01-31T21:38:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:13 smithi171 conmon[41853]: debug 2022-01-31T21:38:13.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:13.021912+0000) 2022-01-31T21:38:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:13 smithi171 conmon[46715]: debug 2022-01-31T21:38:13.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:13.065475+0000) 2022-01-31T21:38:13.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:13 smithi167 conmon[54076]: debug 2022-01-31T21:38:13.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:13.293469+0000) 2022-01-31T21:38:13.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:13 smithi167 conmon[60316]: debug 2022-01-31T21:38:13.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:13.293383+0000) 2022-01-31T21:38:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:13 smithi167 conmon[49112]: debug 2022-01-31T21:38:13.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:13.629237+0000) 2022-01-31T21:38:13.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:13 smithi171 conmon[51620]: debug 2022-01-31T21:38:13.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:13.901820+0000) 2022-01-31T21:38:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:14 smithi171 conmon[46715]: debug 2022-01-31T21:38:14.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:14.065625+0000) 2022-01-31T21:38:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:14 smithi171 conmon[41853]: debug 2022-01-31T21:38:14.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:14.022070+0000) 2022-01-31T21:38:14.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:14 smithi167 conmon[54076]: debug 2022-01-31T21:38:14.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:14.293642+0000) 2022-01-31T21:38:14.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:14 smithi167 conmon[60316]: debug 2022-01-31T21:38:14.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:14.293580+0000) 2022-01-31T21:38:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:14 smithi167 conmon[49112]: debug 2022-01-31T21:38:14.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:14.629456+0000) 2022-01-31T21:38:14.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:14 smithi171 conmon[51620]: debug 2022-01-31T21:38:14.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:14.902019+0000) 2022-01-31T21:38:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:15 smithi171 conmon[46715]: debug 2022-01-31T21:38:15.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:15.065793+0000) 2022-01-31T21:38:15.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:15 smithi171 conmon[41853]: debug 2022-01-31T21:38:15.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:15.022218+0000) 2022-01-31T21:38:15.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:15 smithi167 conmon[54076]: debug 2022-01-31T21:38:15.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:15.293805+0000) 2022-01-31T21:38:15.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:15 smithi167 conmon[60316]: debug 2022-01-31T21:38:15.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:15.293737+0000) 2022-01-31T21:38:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:15 smithi167 conmon[49112]: debug 2022-01-31T21:38:15.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:15.629676+0000) 2022-01-31T21:38:15.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:15 smithi171 conmon[51620]: debug 2022-01-31T21:38:15.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:15.902236+0000) 2022-01-31T21:38:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:16 smithi171 conmon[46715]: debug 2022-01-31T21:38:16.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:16.065979+0000) 2022-01-31T21:38:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:16 smithi171 conmon[41853]: debug 2022-01-31T21:38:16.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:16.022379+0000) 2022-01-31T21:38:16.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:16 smithi167 conmon[54076]: debug 2022-01-31T21:38:16.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:16.293966+0000) 2022-01-31T21:38:16.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:16 smithi167 conmon[60316]: debug 2022-01-31T21:38:16.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:16.293907+0000) 2022-01-31T21:38:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:16 smithi167 conmon[49112]: debug 2022-01-31T21:38:16.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:16.629898+0000) 2022-01-31T21:38:16.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:16 smithi171 conmon[51620]: debug 2022-01-31T21:38:16.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:16.902432+0000) 2022-01-31T21:38:17.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[46715]: debug 2022-01-31T21:38:17.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.066183+0000) 2022-01-31T21:38:17.268 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[41853]: debug 2022-01-31T21:38:17.020+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.022554+0000) 2022-01-31T21:38:17.533 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:17 smithi167 conmon[49112]: debug 2022-01-31T21:38:17.269+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.271070+0000) 2022-01-31T21:38:17.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:17 smithi167 conmon[54076]: debug 2022-01-31T21:38:17.269+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.270810+0000) 2022-01-31T21:38:17.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:17 smithi167 conmon[54076]: debug 2022-01-31T21:38:17.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.294071+0000) 2022-01-31T21:38:17.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:17 smithi167 conmon[60316]: debug 2022-01-31T21:38:17.270+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.271594+0000) 2022-01-31T21:38:17.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:17 smithi167 conmon[60316]: debug 2022-01-31T21:38:17.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.294035+0000) 2022-01-31T21:38:17.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[35325]: debug 2022-01-31T21:38:17.284+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145577 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[41853]: debug 2022-01-31T21:38:17.270+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.272501+0000) 2022-01-31T21:38:17.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[46715]: debug 2022-01-31T21:38:17.269+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.271432+0000) 2022-01-31T21:38:17.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[51620]: debug 2022-01-31T21:38:17.268+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.270701+0000) 2022-01-31T21:38:17.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:17 smithi167 conmon[49112]: debug 2022-01-31T21:38:17.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.630101+0000) 2022-01-31T21:38:17.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:17 smithi171 conmon[51620]: debug 2022-01-31T21:38:17.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:17.902632+0000) 2022-01-31T21:38:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:18 smithi171 conmon[41853]: debug 2022-01-31T21:38:18.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:18.022741+0000) 2022-01-31T21:38:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:18 smithi171 conmon[46715]: debug 2022-01-31T21:38:18.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:18.066388+0000) 2022-01-31T21:38:18.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:18 smithi167 conmon[54076]: debug 2022-01-31T21:38:18.293+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:18.294201+0000) 2022-01-31T21:38:18.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:18 smithi167 conmon[60316]: debug 2022-01-31T21:38:18.293+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:18.294199+0000) 2022-01-31T21:38:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:18 smithi167 conmon[49112]: debug 2022-01-31T21:38:18.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:18.630281+0000) 2022-01-31T21:38:18.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:18 smithi171 conmon[51620]: debug 2022-01-31T21:38:18.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:18.902799+0000) 2022-01-31T21:38:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:19 smithi171 conmon[41853]: debug 2022-01-31T21:38:19.021+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:19.022889+0000) 2022-01-31T21:38:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:19 smithi171 conmon[46715]: debug 2022-01-31T21:38:19.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:19.066570+0000) 2022-01-31T21:38:19.610 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:19 smithi167 conmon[54076]: debug 2022-01-31T21:38:19.292+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:19.294390+0000) 2022-01-31T21:38:19.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:19 smithi167 conmon[60316]: debug 2022-01-31T21:38:19.292+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:19.294390+0000) 2022-01-31T21:38:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:19 smithi167 conmon[49112]: debug 2022-01-31T21:38:19.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:19.630488+0000) 2022-01-31T21:38:19.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:19 smithi171 conmon[51620]: debug 2022-01-31T21:38:19.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:19.902957+0000) 2022-01-31T21:38:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:20 smithi171 conmon[41853]: debug 2022-01-31T21:38:20.022+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:20.023069+0000) 2022-01-31T21:38:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:20 smithi171 conmon[46715]: debug 2022-01-31T21:38:20.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:20.066777+0000) 2022-01-31T21:38:20.610 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:20 smithi167 conmon[54076]: debug 2022-01-31T21:38:20.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:20.294602+0000) 2022-01-31T21:38:20.611 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:20 smithi167 conmon[60316]: debug 2022-01-31T21:38:20.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:20.294600+0000) 2022-01-31T21:38:20.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:20 smithi167 conmon[49112]: debug 2022-01-31T21:38:20.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:20.630745+0000) 2022-01-31T21:38:20.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:20 smithi171 conmon[51620]: debug 2022-01-31T21:38:20.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:20.903168+0000) 2022-01-31T21:38:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:21 smithi171 conmon[41853]: debug 2022-01-31T21:38:21.022+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:21.023249+0000) 2022-01-31T21:38:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:21 smithi171 conmon[46715]: debug 2022-01-31T21:38:21.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:21.066935+0000) 2022-01-31T21:38:21.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:21 smithi167 conmon[60316]: debug 2022-01-31T21:38:21.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:21.294810+0000) 2022-01-31T21:38:21.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:21 smithi167 conmon[54076]: debug 2022-01-31T21:38:21.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:21.294807+0000) 2022-01-31T21:38:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:21 smithi167 conmon[49112]: debug 2022-01-31T21:38:21.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:21.630937+0000) 2022-01-31T21:38:21.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:21 smithi171 conmon[51620]: debug 2022-01-31T21:38:21.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:21.903386+0000) 2022-01-31T21:38:22.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[41853]: debug 2022-01-31T21:38:22.022+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.023428+0000) 2022-01-31T21:38:22.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[46715]: debug 2022-01-31T21:38:22.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.067122+0000) 2022-01-31T21:38:22.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[35325]: debug 2022-01-31T21:38:22.302+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145688 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[41853]: debug 2022-01-31T21:38:22.287+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.289250+0000) 2022-01-31T21:38:22.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[46715]: debug 2022-01-31T21:38:22.288+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.290020+0000) 2022-01-31T21:38:22.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[51620]: debug 2022-01-31T21:38:22.286+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.288420+0000) 2022-01-31T21:38:22.610 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:22 smithi167 conmon[49112]: debug 2022-01-31T21:38:22.287+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.287848+0000) 2022-01-31T21:38:22.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:22 smithi167 conmon[54076]: debug 2022-01-31T21:38:22.287+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.288047+0000) 2022-01-31T21:38:22.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:22 smithi167 conmon[54076]: debug 2022-01-31T21:38:22.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.294988+0000) 2022-01-31T21:38:22.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:22 smithi167 conmon[60316]: debug 2022-01-31T21:38:22.288+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.289300+0000) 2022-01-31T21:38:22.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:22 smithi167 conmon[60316]: debug 2022-01-31T21:38:22.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.294992+0000) 2022-01-31T21:38:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:22 smithi167 conmon[49112]: debug 2022-01-31T21:38:22.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.631147+0000) 2022-01-31T21:38:22.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:22 smithi171 conmon[51620]: debug 2022-01-31T21:38:22.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:22.903594+0000) 2022-01-31T21:38:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:23 smithi171 conmon[41853]: debug 2022-01-31T21:38:23.022+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:23.023580+0000) 2022-01-31T21:38:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:23 smithi171 conmon[46715]: debug 2022-01-31T21:38:23.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:23.067306+0000) 2022-01-31T21:38:23.610 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:23 smithi167 conmon[54076]: debug 2022-01-31T21:38:23.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:23.295156+0000) 2022-01-31T21:38:23.611 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:23 smithi167 conmon[60316]: debug 2022-01-31T21:38:23.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:23.295162+0000) 2022-01-31T21:38:23.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:23 smithi167 conmon[49112]: debug 2022-01-31T21:38:23.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:23.631325+0000) 2022-01-31T21:38:23.977 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:23 smithi171 conmon[35325]: debug 2022-01-31T21:38:23.712+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:38:23.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:23 smithi171 conmon[51620]: debug 2022-01-31T21:38:23.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:23.903799+0000) 2022-01-31T21:38:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:24 smithi171 conmon[41853]: debug 2022-01-31T21:38:24.022+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:24.023720+0000) 2022-01-31T21:38:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:24 smithi171 conmon[46715]: debug 2022-01-31T21:38:24.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:24.067456+0000) 2022-01-31T21:38:24.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:24 smithi167 conmon[54076]: debug 2022-01-31T21:38:24.294+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:24.295351+0000) 2022-01-31T21:38:24.611 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:24 smithi167 conmon[60316]: debug 2022-01-31T21:38:24.294+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:24.295373+0000) 2022-01-31T21:38:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:24 smithi167 conmon[49112]: debug 2022-01-31T21:38:24.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:24.631533+0000) 2022-01-31T21:38:24.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:24 smithi171 conmon[51620]: debug 2022-01-31T21:38:24.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:24.904012+0000) 2022-01-31T21:38:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:25 smithi171 conmon[41853]: debug 2022-01-31T21:38:25.023+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:25.023929+0000) 2022-01-31T21:38:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:25 smithi171 conmon[46715]: debug 2022-01-31T21:38:25.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:25.067664+0000) 2022-01-31T21:38:25.611 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:25 smithi167 conmon[60316]: debug 2022-01-31T21:38:25.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:25.295551+0000) 2022-01-31T21:38:25.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:25 smithi167 conmon[54076]: debug 2022-01-31T21:38:25.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:25.295519+0000) 2022-01-31T21:38:25.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:25 smithi167 conmon[49112]: debug 2022-01-31T21:38:25.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:25.631764+0000) 2022-01-31T21:38:25.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:25 smithi171 conmon[51620]: debug 2022-01-31T21:38:25.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:25.904230+0000) 2022-01-31T21:38:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:26 smithi171 conmon[46715]: debug 2022-01-31T21:38:26.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:26.067856+0000) 2022-01-31T21:38:26.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:26 smithi171 conmon[41853]: debug 2022-01-31T21:38:26.023+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:26.024076+0000) 2022-01-31T21:38:26.611 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:26 smithi167 conmon[60316]: debug 2022-01-31T21:38:26.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:26.295717+0000) 2022-01-31T21:38:26.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:26 smithi167 conmon[54076]: debug 2022-01-31T21:38:26.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:26.295699+0000) 2022-01-31T21:38:26.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:26 smithi167 conmon[49112]: debug 2022-01-31T21:38:26.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:26.631950+0000) 2022-01-31T21:38:26.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:26 smithi171 conmon[51620]: debug 2022-01-31T21:38:26.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:26.904535+0000) 2022-01-31T21:38:27.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[41853]: debug 2022-01-31T21:38:27.022+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.024343+0000) 2022-01-31T21:38:27.309 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[46715]: debug 2022-01-31T21:38:27.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.068093+0000) 2022-01-31T21:38:27.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[35325]: debug 2022-01-31T21:38:27.320+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145798 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[41853]: debug 2022-01-31T21:38:27.304+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.306016+0000) 2022-01-31T21:38:27.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[46715]: debug 2022-01-31T21:38:27.305+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.307073+0000) 2022-01-31T21:38:27.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[51620]: debug 2022-01-31T21:38:27.306+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.307859+0000) 2022-01-31T21:38:27.611 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:27 smithi167 conmon[49112]: debug 2022-01-31T21:38:27.305+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.306011+0000) 2022-01-31T21:38:27.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:27 smithi167 conmon[54076]: debug 2022-01-31T21:38:27.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.295973+0000) 2022-01-31T21:38:27.613 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:27 smithi167 conmon[54076]: debug 2022-01-31T21:38:27.304+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.305466+0000) 2022-01-31T21:38:27.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:27 smithi167 conmon[60316]: debug 2022-01-31T21:38:27.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.295977+0000) 2022-01-31T21:38:27.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:27 smithi167 conmon[60316]: debug 2022-01-31T21:38:27.306+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.307071+0000) 2022-01-31T21:38:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:27 smithi167 conmon[49112]: debug 2022-01-31T21:38:27.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.632151+0000) 2022-01-31T21:38:27.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:27 smithi171 conmon[51620]: debug 2022-01-31T21:38:27.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:27.904714+0000) 2022-01-31T21:38:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:28 smithi171 conmon[41853]: debug 2022-01-31T21:38:28.023+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:28.024522+0000) 2022-01-31T21:38:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:28 smithi171 conmon[46715]: debug 2022-01-31T21:38:28.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:28.068245+0000) 2022-01-31T21:38:28.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:28 smithi167 conmon[54076]: debug 2022-01-31T21:38:28.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:28.296139+0000) 2022-01-31T21:38:28.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:28 smithi167 conmon[60316]: debug 2022-01-31T21:38:28.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:28.296135+0000) 2022-01-31T21:38:28.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:28 smithi167 conmon[49112]: debug 2022-01-31T21:38:28.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:28.632345+0000) 2022-01-31T21:38:28.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:28 smithi171 conmon[51620]: debug 2022-01-31T21:38:28.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:28.904901+0000) 2022-01-31T21:38:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:29 smithi171 conmon[41853]: debug 2022-01-31T21:38:29.023+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:29.024633+0000) 2022-01-31T21:38:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:29 smithi171 conmon[46715]: debug 2022-01-31T21:38:29.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:29.068402+0000) 2022-01-31T21:38:29.611 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:29 smithi167 conmon[60316]: debug 2022-01-31T21:38:29.295+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:29.296311+0000) 2022-01-31T21:38:29.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:29 smithi167 conmon[54076]: debug 2022-01-31T21:38:29.295+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:29.296313+0000) 2022-01-31T21:38:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:29 smithi167 conmon[49112]: debug 2022-01-31T21:38:29.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:29.632558+0000) 2022-01-31T21:38:29.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:29 smithi171 conmon[51620]: debug 2022-01-31T21:38:29.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:29.905076+0000) 2022-01-31T21:38:30.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:30 smithi167 conmon[54076]: debug 2022-01-31T21:38:30.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:30.296531+0000) 2022-01-31T21:38:30.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:30 smithi167 conmon[60316]: debug 2022-01-31T21:38:30.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:30.296538+0000) 2022-01-31T21:38:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:30 smithi171 conmon[41853]: debug 2022-01-31T21:38:30.024+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:30.024820+0000) 2022-01-31T21:38:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:30 smithi171 conmon[46715]: debug 2022-01-31T21:38:30.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:30.068549+0000) 2022-01-31T21:38:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:30 smithi167 conmon[49112]: debug 2022-01-31T21:38:30.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:30.632776+0000) 2022-01-31T21:38:30.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:30 smithi171 conmon[51620]: debug 2022-01-31T21:38:30.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:30.905271+0000) 2022-01-31T21:38:31.350 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:31 smithi167 conmon[54076]: debug 2022-01-31T21:38:31.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:31.296712+0000) 2022-01-31T21:38:31.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:31 smithi167 conmon[60316]: debug 2022-01-31T21:38:31.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:31.296685+0000) 2022-01-31T21:38:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:31 smithi171 conmon[41853]: debug 2022-01-31T21:38:31.024+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:31.024952+0000) 2022-01-31T21:38:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:31 smithi171 conmon[46715]: debug 2022-01-31T21:38:31.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:31.068712+0000) 2022-01-31T21:38:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:31 smithi167 conmon[49112]: debug 2022-01-31T21:38:31.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:31.632969+0000) 2022-01-31T21:38:31.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:31 smithi171 conmon[51620]: debug 2022-01-31T21:38:31.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:31.905474+0000) 2022-01-31T21:38:32.322 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[41853]: debug 2022-01-31T21:38:32.024+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.025120+0000) 2022-01-31T21:38:32.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:32 smithi167 conmon[54076]: debug 2022-01-31T21:38:32.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.296904+0000) 2022-01-31T21:38:32.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[46715]: debug 2022-01-31T21:38:32.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.068921+0000) 2022-01-31T21:38:32.323 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:32 smithi167 conmon[60316]: debug 2022-01-31T21:38:32.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.296891+0000) 2022-01-31T21:38:32.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[35325]: debug 2022-01-31T21:38:32.339+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 145909 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[41853]: debug 2022-01-31T21:38:32.323+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.324097+0000) 2022-01-31T21:38:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[46715]: debug 2022-01-31T21:38:32.324+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.325044+0000) 2022-01-31T21:38:32.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[51620]: debug 2022-01-31T21:38:32.324+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.325584+0000) 2022-01-31T21:38:32.612 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:32 smithi167 conmon[49112]: debug 2022-01-31T21:38:32.323+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.323882+0000) 2022-01-31T21:38:32.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:32 smithi167 conmon[54076]: debug 2022-01-31T21:38:32.323+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.324391+0000) 2022-01-31T21:38:32.613 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:32 smithi167 conmon[60316]: debug 2022-01-31T21:38:32.323+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.324292+0000) 2022-01-31T21:38:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:32 smithi167 conmon[49112]: debug 2022-01-31T21:38:32.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.633165+0000) 2022-01-31T21:38:32.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:32 smithi171 conmon[51620]: debug 2022-01-31T21:38:32.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:32.905656+0000) 2022-01-31T21:38:33.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:33 smithi167 conmon[54076]: debug 2022-01-31T21:38:33.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:33.297088+0000) 2022-01-31T21:38:33.351 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:33 smithi167 conmon[60316]: debug 2022-01-31T21:38:33.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:33.297091+0000) 2022-01-31T21:38:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:33 smithi171 conmon[41853]: debug 2022-01-31T21:38:33.024+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:33.025307+0000) 2022-01-31T21:38:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:33 smithi171 conmon[46715]: debug 2022-01-31T21:38:33.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:33.069122+0000) 2022-01-31T21:38:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:33 smithi167 conmon[49112]: debug 2022-01-31T21:38:33.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:33.633339+0000) 2022-01-31T21:38:33.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:33 smithi171 conmon[51620]: debug 2022-01-31T21:38:33.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:33.905821+0000) 2022-01-31T21:38:34.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:34 smithi167 conmon[54076]: debug 2022-01-31T21:38:34.296+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:34.297258+0000) 2022-01-31T21:38:34.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:34 smithi167 conmon[60316]: debug 2022-01-31T21:38:34.296+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:34.297261+0000) 2022-01-31T21:38:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:34 smithi171 conmon[41853]: debug 2022-01-31T21:38:34.024+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:34.025452+0000) 2022-01-31T21:38:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:34 smithi171 conmon[46715]: debug 2022-01-31T21:38:34.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:34.069275+0000) 2022-01-31T21:38:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:34 smithi167 conmon[49112]: debug 2022-01-31T21:38:34.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:34.633579+0000) 2022-01-31T21:38:34.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:34 smithi171 conmon[51620]: debug 2022-01-31T21:38:34.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:34.906031+0000) 2022-01-31T21:38:35.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:35 smithi167 conmon[54076]: debug 2022-01-31T21:38:35.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:35.297468+0000) 2022-01-31T21:38:35.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:35 smithi167 conmon[60316]: debug 2022-01-31T21:38:35.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:35.297464+0000) 2022-01-31T21:38:35.354 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:35 smithi171 conmon[41853]: debug 2022-01-31T21:38:35.024+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:35.025629+0000) 2022-01-31T21:38:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:35 smithi171 conmon[46715]: debug 2022-01-31T21:38:35.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:35.069460+0000) 2022-01-31T21:38:35.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:35 smithi167 conmon[49112]: debug 2022-01-31T21:38:35.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:35.633742+0000) 2022-01-31T21:38:35.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:35 smithi171 conmon[51620]: debug 2022-01-31T21:38:35.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:35.906254+0000) 2022-01-31T21:38:36.351 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:36 smithi167 conmon[54076]: debug 2022-01-31T21:38:36.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:36.297683+0000) 2022-01-31T21:38:36.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:36 smithi167 conmon[60316]: debug 2022-01-31T21:38:36.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:36.297679+0000) 2022-01-31T21:38:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:36 smithi171 conmon[41853]: debug 2022-01-31T21:38:36.025+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:36.025830+0000) 2022-01-31T21:38:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:36 smithi171 conmon[46715]: debug 2022-01-31T21:38:36.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:36.069661+0000) 2022-01-31T21:38:36.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:36 smithi167 conmon[49112]: debug 2022-01-31T21:38:36.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:36.633943+0000) 2022-01-31T21:38:36.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:36 smithi171 conmon[51620]: debug 2022-01-31T21:38:36.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:36.906455+0000) 2022-01-31T21:38:37.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[46715]: debug 2022-01-31T21:38:37.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.069845+0000) 2022-01-31T21:38:37.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[41853]: debug 2022-01-31T21:38:37.025+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.026011+0000) 2022-01-31T21:38:37.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:37 smithi167 conmon[54076]: debug 2022-01-31T21:38:37.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.297855+0000) 2022-01-31T21:38:37.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:37 smithi167 conmon[60316]: debug 2022-01-31T21:38:37.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.297825+0000) 2022-01-31T21:38:37.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[35325]: debug 2022-01-31T21:38:37.358+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146021 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[41853]: debug 2022-01-31T21:38:37.341+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.342413+0000) 2022-01-31T21:38:37.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[46715]: debug 2022-01-31T21:38:37.341+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.342063+0000) 2022-01-31T21:38:37.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[51620]: debug 2022-01-31T21:38:37.342+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.343269+0000) 2022-01-31T21:38:37.613 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:37 smithi167 conmon[49112]: debug 2022-01-31T21:38:37.341+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.342026+0000) 2022-01-31T21:38:37.613 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:37 smithi167 conmon[54076]: debug 2022-01-31T21:38:37.341+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.341968+0000) 2022-01-31T21:38:37.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:37 smithi167 conmon[60316]: debug 2022-01-31T21:38:37.342+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.342950+0000) 2022-01-31T21:38:37.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:37 smithi167 conmon[49112]: debug 2022-01-31T21:38:37.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.634155+0000) 2022-01-31T21:38:37.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:37 smithi171 conmon[51620]: debug 2022-01-31T21:38:37.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:37.906666+0000) 2022-01-31T21:38:38.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:38 smithi167 conmon[54076]: debug 2022-01-31T21:38:38.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:38.298054+0000) 2022-01-31T21:38:38.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:38 smithi167 conmon[60316]: debug 2022-01-31T21:38:38.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:38.297998+0000) 2022-01-31T21:38:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:38 smithi171 conmon[46715]: debug 2022-01-31T21:38:38.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:38.070051+0000) 2022-01-31T21:38:38.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:38 smithi171 conmon[41853]: debug 2022-01-31T21:38:38.025+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:38.026177+0000) 2022-01-31T21:38:38.807 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:38 smithi171 conmon[35325]: debug 2022-01-31T21:38:38.713+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:38:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:38 smithi167 conmon[49112]: debug 2022-01-31T21:38:38.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:38.634327+0000) 2022-01-31T21:38:39.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:39 smithi171 conmon[46715]: debug 2022-01-31T21:38:39.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:39.070203+0000) 2022-01-31T21:38:39.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:38 smithi171 conmon[51620]: debug 2022-01-31T21:38:38.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:38.906833+0000) 2022-01-31T21:38:39.337 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:39 smithi171 conmon[41853]: debug 2022-01-31T21:38:39.025+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:39.026284+0000) 2022-01-31T21:38:39.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:39 smithi167 conmon[54076]: debug 2022-01-31T21:38:39.297+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:39.298161+0000) 2022-01-31T21:38:39.352 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:39 smithi167 conmon[60316]: debug 2022-01-31T21:38:39.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:39.298147+0000) 2022-01-31T21:38:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:39 smithi167 conmon[49112]: debug 2022-01-31T21:38:39.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:39.634544+0000) 2022-01-31T21:38:39.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:39 smithi171 conmon[51620]: debug 2022-01-31T21:38:39.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:39.907045+0000) 2022-01-31T21:38:40.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:40 smithi171 conmon[41853]: debug 2022-01-31T21:38:40.025+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:40.026450+0000) 2022-01-31T21:38:40.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:40 smithi171 conmon[46715]: debug 2022-01-31T21:38:40.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:40.070368+0000) 2022-01-31T21:38:40.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:40 smithi167 conmon[54076]: debug 2022-01-31T21:38:40.298+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:40.298991+0000) 2022-01-31T21:38:40.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:40 smithi167 conmon[60316]: debug 2022-01-31T21:38:40.297+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:40.298290+0000) 2022-01-31T21:38:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:40 smithi167 conmon[49112]: debug 2022-01-31T21:38:40.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:40.634747+0000) 2022-01-31T21:38:40.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:40 smithi171 conmon[51620]: debug 2022-01-31T21:38:40.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:40.907237+0000) 2022-01-31T21:38:41.352 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:41 smithi167 conmon[54076]: debug 2022-01-31T21:38:41.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:41.299711+0000) 2022-01-31T21:38:41.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:41 smithi167 conmon[60316]: debug 2022-01-31T21:38:41.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:41.298511+0000) 2022-01-31T21:38:41.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:41 smithi171 conmon[46715]: debug 2022-01-31T21:38:41.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:41.070534+0000) 2022-01-31T21:38:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:41 smithi171 conmon[41853]: debug 2022-01-31T21:38:41.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:41.026619+0000) 2022-01-31T21:38:41.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:41 smithi167 conmon[49112]: debug 2022-01-31T21:38:41.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:41.634971+0000) 2022-01-31T21:38:41.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:41 smithi171 conmon[51620]: debug 2022-01-31T21:38:41.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:41.907450+0000) 2022-01-31T21:38:42.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[46715]: debug 2022-01-31T21:38:42.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.070710+0000) 2022-01-31T21:38:42.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[41853]: debug 2022-01-31T21:38:42.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.026786+0000) 2022-01-31T21:38:42.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:42 smithi167 conmon[54076]: debug 2022-01-31T21:38:42.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.299866+0000) 2022-01-31T21:38:42.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:42 smithi167 conmon[60316]: debug 2022-01-31T21:38:42.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.298649+0000) 2022-01-31T21:38:42.614 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:42 smithi167 conmon[54076]: debug 2022-01-31T21:38:42.360+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.361145+0000) 2022-01-31T21:38:42.615 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:42 smithi167 conmon[60316]: debug 2022-01-31T21:38:42.360+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.360827+0000) 2022-01-31T21:38:42.615 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:42 smithi167 conmon[49112]: debug 2022-01-31T21:38:42.360+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.361295+0000) 2022-01-31T21:38:42.722 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[35325]: debug 2022-01-31T21:38:42.376+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146132 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:42.723 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[41853]: debug 2022-01-31T21:38:42.359+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.360352+0000) 2022-01-31T21:38:42.723 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[46715]: debug 2022-01-31T21:38:42.359+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.360663+0000) 2022-01-31T21:38:42.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[51620]: debug 2022-01-31T21:38:42.360+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.361586+0000) 2022-01-31T21:38:42.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:42 smithi167 conmon[49112]: debug 2022-01-31T21:38:42.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.635183+0000) 2022-01-31T21:38:42.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:42 smithi171 conmon[51620]: debug 2022-01-31T21:38:42.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:42.907673+0000) 2022-01-31T21:38:43.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:43 smithi167 conmon[54076]: debug 2022-01-31T21:38:43.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:43.299995+0000) 2022-01-31T21:38:43.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:43 smithi167 conmon[60316]: debug 2022-01-31T21:38:43.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:43.298759+0000) 2022-01-31T21:38:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:43 smithi171 conmon[41853]: debug 2022-01-31T21:38:43.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:43.026983+0000) 2022-01-31T21:38:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:43 smithi171 conmon[46715]: debug 2022-01-31T21:38:43.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:43.070871+0000) 2022-01-31T21:38:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:43 smithi167 conmon[49112]: debug 2022-01-31T21:38:43.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:43.635351+0000) 2022-01-31T21:38:43.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:43 smithi171 conmon[51620]: debug 2022-01-31T21:38:43.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:43.907820+0000) 2022-01-31T21:38:44.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:44 smithi167 conmon[54076]: debug 2022-01-31T21:38:44.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:44.300114+0000) 2022-01-31T21:38:44.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:44 smithi167 conmon[60316]: debug 2022-01-31T21:38:44.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:44.298923+0000) 2022-01-31T21:38:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:44 smithi171 conmon[46715]: debug 2022-01-31T21:38:44.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:44.070996+0000) 2022-01-31T21:38:44.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:44 smithi171 conmon[41853]: debug 2022-01-31T21:38:44.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:44.027136+0000) 2022-01-31T21:38:44.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:44 smithi167 conmon[49112]: debug 2022-01-31T21:38:44.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:44.635553+0000) 2022-01-31T21:38:44.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:44 smithi171 conmon[51620]: debug 2022-01-31T21:38:44.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:44.907985+0000) 2022-01-31T21:38:45.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:45 smithi167 conmon[54076]: debug 2022-01-31T21:38:45.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:45.300299+0000) 2022-01-31T21:38:45.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:45 smithi167 conmon[60316]: debug 2022-01-31T21:38:45.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:45.299080+0000) 2022-01-31T21:38:45.354 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:45 smithi171 conmon[41853]: debug 2022-01-31T21:38:45.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:45.027316+0000) 2022-01-31T21:38:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:45 smithi171 conmon[46715]: debug 2022-01-31T21:38:45.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:45.071154+0000) 2022-01-31T21:38:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:45 smithi167 conmon[49112]: debug 2022-01-31T21:38:45.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:45.635738+0000) 2022-01-31T21:38:45.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:45 smithi171 conmon[51620]: debug 2022-01-31T21:38:45.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:45.908188+0000) 2022-01-31T21:38:46.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:46 smithi171 conmon[41853]: debug 2022-01-31T21:38:46.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:46.027495+0000) 2022-01-31T21:38:46.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:46 smithi171 conmon[46715]: debug 2022-01-31T21:38:46.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:46.071384+0000) 2022-01-31T21:38:46.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:46 smithi167 conmon[54076]: debug 2022-01-31T21:38:46.298+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:46.300476+0000) 2022-01-31T21:38:46.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:46 smithi167 conmon[60316]: debug 2022-01-31T21:38:46.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:46.299259+0000) 2022-01-31T21:38:46.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:46 smithi167 conmon[49112]: debug 2022-01-31T21:38:46.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:46.635960+0000) 2022-01-31T21:38:46.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:46 smithi171 conmon[51620]: debug 2022-01-31T21:38:46.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:46.908375+0000) 2022-01-31T21:38:47.353 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[60316]: debug 2022-01-31T21:38:47.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.299418+0000) 2022-01-31T21:38:47.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[54076]: debug 2022-01-31T21:38:47.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.300614+0000) 2022-01-31T21:38:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[41853]: debug 2022-01-31T21:38:47.026+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.027695+0000) 2022-01-31T21:38:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[46715]: debug 2022-01-31T21:38:47.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.071603+0000) 2022-01-31T21:38:47.615 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[49112]: debug 2022-01-31T21:38:47.377+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.378569+0000) 2022-01-31T21:38:47.615 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[54076]: debug 2022-01-31T21:38:47.616 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[54076]: 2022-01-31T21:38:47.377+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.378823+0000) 2022-01-31T21:38:47.616 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[60316]: debug 2022-01-31T21:38:47.376+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.378107+0000) 2022-01-31T21:38:47.723 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[35325]: debug 2022-01-31T21:38:47.393+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146242 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:47.723 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[41853]: debug 2022-01-31T21:38:47.377+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.378628+0000) 2022-01-31T21:38:47.724 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[46715]: debug 2022-01-31T21:38:47.378+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.379265+0000) 2022-01-31T21:38:47.724 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[51620]: debug 2022-01-31T21:38:47.379+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.380377+0000) 2022-01-31T21:38:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:47 smithi167 conmon[49112]: debug 2022-01-31T21:38:47.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.636167+0000) 2022-01-31T21:38:47.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:47 smithi171 conmon[51620]: debug 2022-01-31T21:38:47.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:47.908614+0000) 2022-01-31T21:38:48.353 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:48 smithi167 conmon[54076]: debug 2022-01-31T21:38:48.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:48.300764+0000) 2022-01-31T21:38:48.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:48 smithi167 conmon[60316]: debug 2022-01-31T21:38:48.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:48.299582+0000) 2022-01-31T21:38:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:48 smithi171 conmon[41853]: debug 2022-01-31T21:38:48.027+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:48.027943+0000) 2022-01-31T21:38:48.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:48 smithi171 conmon[46715]: debug 2022-01-31T21:38:48.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:48.071791+0000) 2022-01-31T21:38:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:48 smithi167 conmon[49112]: debug 2022-01-31T21:38:48.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:48.636366+0000) 2022-01-31T21:38:48.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:48 smithi171 conmon[51620]: debug 2022-01-31T21:38:48.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:48.908796+0000) 2022-01-31T21:38:49.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:49 smithi167 conmon[54076]: debug 2022-01-31T21:38:49.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:49.300877+0000) 2022-01-31T21:38:49.354 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:49 smithi167 conmon[60316]: debug 2022-01-31T21:38:49.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:49.299709+0000) 2022-01-31T21:38:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:49 smithi171 conmon[41853]: debug 2022-01-31T21:38:49.027+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:49.028092+0000) 2022-01-31T21:38:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:49 smithi171 conmon[46715]: debug 2022-01-31T21:38:49.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:49.071904+0000) 2022-01-31T21:38:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:49 smithi167 conmon[49112]: debug 2022-01-31T21:38:49.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:49.636559+0000) 2022-01-31T21:38:49.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:49 smithi171 conmon[51620]: debug 2022-01-31T21:38:49.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:49.908980+0000) 2022-01-31T21:38:50.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:50 smithi167 conmon[54076]: debug 2022-01-31T21:38:50.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:50.301089+0000) 2022-01-31T21:38:50.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:50 smithi167 conmon[60316]: debug 2022-01-31T21:38:50.298+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:50.299889+0000) 2022-01-31T21:38:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:50 smithi171 conmon[41853]: debug 2022-01-31T21:38:50.027+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:50.028259+0000) 2022-01-31T21:38:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:50 smithi171 conmon[46715]: debug 2022-01-31T21:38:50.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:50.072042+0000) 2022-01-31T21:38:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:50 smithi167 conmon[49112]: debug 2022-01-31T21:38:50.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:50.636758+0000) 2022-01-31T21:38:50.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:50 smithi171 conmon[51620]: debug 2022-01-31T21:38:50.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:50.909168+0000) 2022-01-31T21:38:51.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:51 smithi167 conmon[54076]: debug 2022-01-31T21:38:51.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:51.301266+0000) 2022-01-31T21:38:51.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:51 smithi167 conmon[60316]: debug 2022-01-31T21:38:51.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:51.300087+0000) 2022-01-31T21:38:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:51 smithi171 conmon[41853]: debug 2022-01-31T21:38:51.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:51.028413+0000) 2022-01-31T21:38:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:51 smithi171 conmon[46715]: debug 2022-01-31T21:38:51.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:51.072239+0000) 2022-01-31T21:38:51.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:51 smithi167 conmon[49112]: debug 2022-01-31T21:38:51.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:51.636966+0000) 2022-01-31T21:38:51.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:51 smithi171 conmon[51620]: debug 2022-01-31T21:38:51.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:51.909381+0000) 2022-01-31T21:38:52.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:52 smithi167 conmon[54076]: debug 2022-01-31T21:38:52.299+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.301445+0000) 2022-01-31T21:38:52.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:52 smithi167 conmon[60316]: debug 2022-01-31T21:38:52.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.300272+0000) 2022-01-31T21:38:52.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[41853]: debug 2022-01-31T21:38:52.027+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.028566+0000) 2022-01-31T21:38:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[46715]: debug 2022-01-31T21:38:52.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.072330+0000) 2022-01-31T21:38:52.616 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:52 smithi167 conmon[49112]: debug 2022-01-31T21:38:52.394+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.396366+0000) 2022-01-31T21:38:52.617 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:52 smithi167 conmon[60316]: debug 2022-01-31T21:38:52.393+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.395400+0000) 2022-01-31T21:38:52.617 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:52 smithi167 conmon[54076]: debug 2022-01-31T21:38:52.395+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.396568+0000) 2022-01-31T21:38:52.724 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[35325]: debug 2022-01-31T21:38:52.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146353 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:52.725 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[41853]: debug 2022-01-31T21:38:52.395+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.396534+0000) 2022-01-31T21:38:52.725 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[46715]: debug 2022-01-31T21:38:52.396+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.397521+0000) 2022-01-31T21:38:52.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[51620]: debug 2022-01-31T21:38:52.397+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.398294+0000) 2022-01-31T21:38:52.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:52 smithi167 conmon[49112]: debug 2022-01-31T21:38:52.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.637143+0000) 2022-01-31T21:38:52.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:52 smithi171 conmon[51620]: debug 2022-01-31T21:38:52.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:52.909575+0000) 2022-01-31T21:38:53.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:53 smithi167 conmon[54076]: debug 2022-01-31T21:38:53.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:53.301585+0000) 2022-01-31T21:38:53.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:53 smithi167 conmon[60316]: debug 2022-01-31T21:38:53.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:53.300454+0000) 2022-01-31T21:38:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:53 smithi171 conmon[41853]: debug 2022-01-31T21:38:53.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:53.028713+0000) 2022-01-31T21:38:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:53 smithi171 conmon[46715]: debug 2022-01-31T21:38:53.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:53.072508+0000) 2022-01-31T21:38:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:53 smithi167 conmon[49112]: debug 2022-01-31T21:38:53.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:53.637303+0000) 2022-01-31T21:38:53.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:53 smithi171 conmon[35325]: debug 2022-01-31T21:38:53.714+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:38:53.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:53 smithi171 conmon[51620]: debug 2022-01-31T21:38:53.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:53.909749+0000) 2022-01-31T21:38:54.354 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:54 smithi167 conmon[54076]: debug 2022-01-31T21:38:54.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:54.301698+0000) 2022-01-31T21:38:54.355 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:54 smithi167 conmon[60316]: debug 2022-01-31T21:38:54.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:54.300607+0000) 2022-01-31T21:38:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:54 smithi171 conmon[41853]: debug 2022-01-31T21:38:54.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:54.028871+0000) 2022-01-31T21:38:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:54 smithi171 conmon[46715]: debug 2022-01-31T21:38:54.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:54.072654+0000) 2022-01-31T21:38:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:54 smithi167 conmon[49112]: debug 2022-01-31T21:38:54.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:54.637542+0000) 2022-01-31T21:38:54.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:54 smithi171 conmon[51620]: debug 2022-01-31T21:38:54.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:54.909960+0000) 2022-01-31T21:38:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:55 smithi171 conmon[46715]: debug 2022-01-31T21:38:55.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:55.072856+0000) 2022-01-31T21:38:55.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:55 smithi167 conmon[54076]: debug 2022-01-31T21:38:55.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:55.301823+0000) 2022-01-31T21:38:55.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:55 smithi171 conmon[41853]: debug 2022-01-31T21:38:55.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:55.029022+0000) 2022-01-31T21:38:55.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:55 smithi167 conmon[60316]: debug 2022-01-31T21:38:55.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:55.300792+0000) 2022-01-31T21:38:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:55 smithi167 conmon[49112]: debug 2022-01-31T21:38:55.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:55.637699+0000) 2022-01-31T21:38:55.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:55 smithi171 conmon[51620]: debug 2022-01-31T21:38:55.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:55.910182+0000) 2022-01-31T21:38:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:56 smithi171 conmon[41853]: debug 2022-01-31T21:38:56.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:56.029180+0000) 2022-01-31T21:38:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:56 smithi171 conmon[46715]: debug 2022-01-31T21:38:56.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:56.073021+0000) 2022-01-31T21:38:56.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:56 smithi167 conmon[54076]: debug 2022-01-31T21:38:56.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:56.301957+0000) 2022-01-31T21:38:56.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:56 smithi167 conmon[60316]: debug 2022-01-31T21:38:56.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:56.300977+0000) 2022-01-31T21:38:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:56 smithi167 conmon[49112]: debug 2022-01-31T21:38:56.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:56.637844+0000) 2022-01-31T21:38:56.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:56 smithi171 conmon[51620]: debug 2022-01-31T21:38:56.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:56.910414+0000) 2022-01-31T21:38:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[41853]: debug 2022-01-31T21:38:57.029+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.029362+0000) 2022-01-31T21:38:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[46715]: debug 2022-01-31T21:38:57.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.073178+0000) 2022-01-31T21:38:57.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:57 smithi167 conmon[54076]: debug 2022-01-31T21:38:57.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.302124+0000) 2022-01-31T21:38:57.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:57 smithi167 conmon[60316]: debug 2022-01-31T21:38:57.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.301174+0000) 2022-01-31T21:38:57.617 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:57 smithi167 conmon[49112]: debug 2022-01-31T21:38:57.412+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.413793+0000) 2022-01-31T21:38:57.618 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:57 smithi167 conmon[54076]: debug 2022-01-31T21:38:57.413+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.414673+0000) 2022-01-31T21:38:57.618 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:57 smithi167 conmon[60316]: debug 2022-01-31T21:38:57.411+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.413140+0000) 2022-01-31T21:38:57.725 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[35325]: debug 2022-01-31T21:38:57.429+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146463 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:38:57.725 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[41853]: debug 2022-01-31T21:38:57.413+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.413864+0000) 2022-01-31T21:38:57.726 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[46715]: debug 2022-01-31T21:38:57.415+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.415771+0000) 2022-01-31T21:38:57.726 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[51620]: debug 2022-01-31T21:38:57.414+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.415113+0000) 2022-01-31T21:38:57.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:57 smithi167 conmon[49112]: debug 2022-01-31T21:38:57.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.638052+0000) 2022-01-31T21:38:57.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:57 smithi171 conmon[51620]: debug 2022-01-31T21:38:57.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:57.910629+0000) 2022-01-31T21:38:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:58 smithi171 conmon[46715]: debug 2022-01-31T21:38:58.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:58.073392+0000) 2022-01-31T21:38:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:58 smithi171 conmon[41853]: debug 2022-01-31T21:38:58.029+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:58.029550+0000) 2022-01-31T21:38:58.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:58 smithi167 conmon[54076]: debug 2022-01-31T21:38:58.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:58.302295+0000) 2022-01-31T21:38:58.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:58 smithi167 conmon[60316]: debug 2022-01-31T21:38:58.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:58.301329+0000) 2022-01-31T21:38:58.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:58 smithi167 conmon[49112]: debug 2022-01-31T21:38:58.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:58.638209+0000) 2022-01-31T21:38:58.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:58 smithi171 conmon[51620]: debug 2022-01-31T21:38:58.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:58.910830+0000) 2022-01-31T21:38:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:38:59 smithi171 conmon[46715]: debug 2022-01-31T21:38:59.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:59.073541+0000) 2022-01-31T21:38:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:38:59 smithi171 conmon[41853]: debug 2022-01-31T21:38:59.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:59.029717+0000) 2022-01-31T21:38:59.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:38:59 smithi167 conmon[54076]: debug 2022-01-31T21:38:59.300+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:59.302451+0000) 2022-01-31T21:38:59.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:38:59 smithi167 conmon[60316]: debug 2022-01-31T21:38:59.299+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:59.301469+0000) 2022-01-31T21:38:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:38:59 smithi167 conmon[49112]: debug 2022-01-31T21:38:59.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:59.638407+0000) 2022-01-31T21:38:59.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:38:59 smithi171 conmon[51620]: debug 2022-01-31T21:38:59.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:38:59.911028+0000) 2022-01-31T21:39:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:00 smithi171 conmon[41853]: debug 2022-01-31T21:39:00.029+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:00.029874+0000) 2022-01-31T21:39:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:00 smithi171 conmon[46715]: debug 2022-01-31T21:39:00.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:00.073715+0000) 2022-01-31T21:39:00.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:00 smithi167 conmon[60316]: debug 2022-01-31T21:39:00.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:00.301669+0000) 2022-01-31T21:39:00.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:00 smithi167 conmon[54076]: debug 2022-01-31T21:39:00.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:00.302588+0000) 2022-01-31T21:39:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:00 smithi167 conmon[49112]: debug 2022-01-31T21:39:00.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:00.638564+0000) 2022-01-31T21:39:00.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:00 smithi171 conmon[51620]: debug 2022-01-31T21:39:00.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:00.911207+0000) 2022-01-31T21:39:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:01 smithi171 conmon[46715]: debug 2022-01-31T21:39:01.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:01.073915+0000) 2022-01-31T21:39:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:01 smithi171 conmon[41853]: debug 2022-01-31T21:39:01.029+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:01.030066+0000) 2022-01-31T21:39:01.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:01 smithi167 conmon[54076]: debug 2022-01-31T21:39:01.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:01.302716+0000) 2022-01-31T21:39:01.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:01 smithi167 conmon[60316]: debug 2022-01-31T21:39:01.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:01.301893+0000) 2022-01-31T21:39:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:01 smithi167 conmon[49112]: debug 2022-01-31T21:39:01.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:01.638705+0000) 2022-01-31T21:39:01.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:01 smithi171 conmon[51620]: debug 2022-01-31T21:39:01.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:01.911421+0000) 2022-01-31T21:39:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[46715]: debug 2022-01-31T21:39:02.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.074110+0000) 2022-01-31T21:39:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[41853]: debug 2022-01-31T21:39:02.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.030248+0000) 2022-01-31T21:39:02.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:02 smithi167 conmon[60316]: debug 2022-01-31T21:39:02.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.302077+0000) 2022-01-31T21:39:02.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:02 smithi167 conmon[54076]: debug 2022-01-31T21:39:02.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.302888+0000) 2022-01-31T21:39:02.617 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:02 smithi167 conmon[49112]: debug 2022-01-31T21:39:02.430+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.432373+0000) 2022-01-31T21:39:02.618 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:02 smithi167 conmon[60316]: debug 2022-01-31T21:39:02.429+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.431296+0000) 2022-01-31T21:39:02.619 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:02 smithi167 conmon[54076]: debug 2022-01-31T21:39:02.429+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.431368+0000) 2022-01-31T21:39:02.725 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[41853]: debug 2022-01-31T21:39:02.431+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.432504+0000) 2022-01-31T21:39:02.726 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[46715]: debug 2022-01-31T21:39:02.432+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.433021+0000) 2022-01-31T21:39:02.727 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[35325]: debug 2022-01-31T21:39:02.447+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146575 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:02.727 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[51620]: debug 2022-01-31T21:39:02.432+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.433281+0000) 2022-01-31T21:39:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:02 smithi167 conmon[49112]: debug 2022-01-31T21:39:02.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.638877+0000) 2022-01-31T21:39:02.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:02 smithi171 conmon[51620]: debug 2022-01-31T21:39:02.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:02.911654+0000) 2022-01-31T21:39:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:03 smithi171 conmon[41853]: debug 2022-01-31T21:39:03.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:03.030438+0000) 2022-01-31T21:39:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:03 smithi171 conmon[46715]: debug 2022-01-31T21:39:03.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:03.074257+0000) 2022-01-31T21:39:03.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:03 smithi167 conmon[60316]: debug 2022-01-31T21:39:03.300+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:03.302299+0000) 2022-01-31T21:39:03.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:03 smithi167 conmon[54076]: debug 2022-01-31T21:39:03.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:03.303086+0000) 2022-01-31T21:39:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:03 smithi167 conmon[49112]: debug 2022-01-31T21:39:03.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:03.639028+0000) 2022-01-31T21:39:03.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:03 smithi171 conmon[51620]: debug 2022-01-31T21:39:03.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:03.911842+0000) 2022-01-31T21:39:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:04 smithi171 conmon[41853]: debug 2022-01-31T21:39:04.028+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:04.030602+0000) 2022-01-31T21:39:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:04 smithi171 conmon[46715]: debug 2022-01-31T21:39:04.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:04.074430+0000) 2022-01-31T21:39:04.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:04 smithi167 conmon[54076]: debug 2022-01-31T21:39:04.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:04.303204+0000) 2022-01-31T21:39:04.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:04 smithi167 conmon[60316]: debug 2022-01-31T21:39:04.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:04.302446+0000) 2022-01-31T21:39:04.803 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:04 smithi167 conmon[49112]: debug 2022-01-31T21:39:04.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:04.639243+0000) 2022-01-31T21:39:04.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:04 smithi171 conmon[51620]: debug 2022-01-31T21:39:04.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:04.912012+0000) 2022-01-31T21:39:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:05 smithi171 conmon[41853]: debug 2022-01-31T21:39:05.029+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:05.030771+0000) 2022-01-31T21:39:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:05 smithi171 conmon[46715]: debug 2022-01-31T21:39:05.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:05.074612+0000) 2022-01-31T21:39:05.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:05 smithi167 conmon[54076]: debug 2022-01-31T21:39:05.301+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:05.303355+0000) 2022-01-31T21:39:05.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:05 smithi167 conmon[60316]: debug 2022-01-31T21:39:05.301+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:05.302617+0000) 2022-01-31T21:39:05.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:05 smithi167 conmon[49112]: debug 2022-01-31T21:39:05.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:05.639439+0000) 2022-01-31T21:39:05.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:05 smithi171 conmon[51620]: debug 2022-01-31T21:39:05.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:05.912211+0000) 2022-01-31T21:39:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:06 smithi171 conmon[41853]: debug 2022-01-31T21:39:06.029+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:06.030916+0000) 2022-01-31T21:39:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:06 smithi171 conmon[46715]: debug 2022-01-31T21:39:06.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:06.074820+0000) 2022-01-31T21:39:06.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:06 smithi167 conmon[54076]: debug 2022-01-31T21:39:06.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:06.303450+0000) 2022-01-31T21:39:06.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:06 smithi167 conmon[60316]: debug 2022-01-31T21:39:06.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:06.302782+0000) 2022-01-31T21:39:06.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:06 smithi167 conmon[49112]: debug 2022-01-31T21:39:06.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:06.639633+0000) 2022-01-31T21:39:06.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:06 smithi171 conmon[51620]: debug 2022-01-31T21:39:06.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:06.912409+0000) 2022-01-31T21:39:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[41853]: debug 2022-01-31T21:39:07.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.031099+0000) 2022-01-31T21:39:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[46715]: debug 2022-01-31T21:39:07.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.075002+0000) 2022-01-31T21:39:07.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:07 smithi167 conmon[54076]: debug 2022-01-31T21:39:07.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.303562+0000) 2022-01-31T21:39:07.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:07 smithi167 conmon[60316]: debug 2022-01-31T21:39:07.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.302963+0000) 2022-01-31T21:39:07.726 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[35325]: debug 2022-01-31T21:39:07.464+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:07.727 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[41853]: debug 2022-01-31T21:39:07.448+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.450220+0000) 2022-01-31T21:39:07.728 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[46715]: debug 2022-01-31T21:39:07.447+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.449710+0000) 2022-01-31T21:39:07.728 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[51620]: debug 2022-01-31T21:39:07.449+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.451553+0000) 2022-01-31T21:39:07.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:07 smithi167 conmon[54076]: debug 2022-01-31T21:39:07.448+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.449984+0000) 2022-01-31T21:39:07.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:07 smithi167 conmon[49112]: debug 2022-01-31T21:39:07.447+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.448802+0000) 2022-01-31T21:39:07.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:07 smithi167 conmon[49112]: debug 2022-01-31T21:39:07.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.639781+0000) 2022-01-31T21:39:07.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:07 smithi167 conmon[60316]: debug 2022-01-31T21:39:07.448+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.450071+0000) 2022-01-31T21:39:07.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:07 smithi171 conmon[51620]: debug 2022-01-31T21:39:07.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:07.912607+0000) 2022-01-31T21:39:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:08 smithi171 conmon[41853]: debug 2022-01-31T21:39:08.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:08.031282+0000) 2022-01-31T21:39:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:08 smithi171 conmon[46715]: debug 2022-01-31T21:39:08.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:08.075161+0000) 2022-01-31T21:39:08.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:08 smithi167 conmon[60316]: debug 2022-01-31T21:39:08.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:08.303114+0000) 2022-01-31T21:39:08.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:08 smithi167 conmon[54076]: debug 2022-01-31T21:39:08.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:08.303696+0000) 2022-01-31T21:39:08.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:08 smithi167 conmon[49112]: debug 2022-01-31T21:39:08.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:08.639902+0000) 2022-01-31T21:39:08.984 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:08 smithi171 conmon[35325]: debug 2022-01-31T21:39:08.714+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:39:08.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:08 smithi171 conmon[51620]: debug 2022-01-31T21:39:08.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:08.912788+0000) 2022-01-31T21:39:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:09 smithi171 conmon[41853]: debug 2022-01-31T21:39:09.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:09.031453+0000) 2022-01-31T21:39:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:09 smithi171 conmon[46715]: debug 2022-01-31T21:39:09.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:09.075299+0000) 2022-01-31T21:39:09.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:09 smithi167 conmon[54076]: debug 2022-01-31T21:39:09.302+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:09.303806+0000) 2022-01-31T21:39:09.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:09 smithi167 conmon[60316]: debug 2022-01-31T21:39:09.302+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:09.303297+0000) 2022-01-31T21:39:09.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:09 smithi167 conmon[49112]: debug 2022-01-31T21:39:09.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:09.640044+0000) 2022-01-31T21:39:09.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:09 smithi171 conmon[51620]: debug 2022-01-31T21:39:09.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:09.913005+0000) 2022-01-31T21:39:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:10 smithi171 conmon[41853]: debug 2022-01-31T21:39:10.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:10.031607+0000) 2022-01-31T21:39:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:10 smithi171 conmon[46715]: debug 2022-01-31T21:39:10.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:10.075493+0000) 2022-01-31T21:39:10.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:10 smithi167 conmon[60316]: debug 2022-01-31T21:39:10.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:10.303484+0000) 2022-01-31T21:39:10.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:10 smithi167 conmon[54076]: debug 2022-01-31T21:39:10.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:10.303943+0000) 2022-01-31T21:39:10.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:10 smithi167 conmon[49112]: debug 2022-01-31T21:39:10.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:10.640192+0000) 2022-01-31T21:39:10.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:10 smithi171 conmon[51620]: debug 2022-01-31T21:39:10.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:10.913245+0000) 2022-01-31T21:39:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:11 smithi171 conmon[46715]: debug 2022-01-31T21:39:11.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:11.075665+0000) 2022-01-31T21:39:11.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:11 smithi171 conmon[41853]: debug 2022-01-31T21:39:11.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:11.031790+0000) 2022-01-31T21:39:11.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:11 smithi167 conmon[60316]: debug 2022-01-31T21:39:11.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:11.303697+0000) 2022-01-31T21:39:11.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:11 smithi167 conmon[54076]: debug 2022-01-31T21:39:11.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:11.304063+0000) 2022-01-31T21:39:11.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:11 smithi167 conmon[49112]: debug 2022-01-31T21:39:11.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:11.640336+0000) 2022-01-31T21:39:11.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:11 smithi171 conmon[51620]: debug 2022-01-31T21:39:11.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:11.913453+0000) 2022-01-31T21:39:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[46715]: debug 2022-01-31T21:39:12.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.075876+0000) 2022-01-31T21:39:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[41853]: debug 2022-01-31T21:39:12.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.031970+0000) 2022-01-31T21:39:12.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:12 smithi167 conmon[54076]: debug 2022-01-31T21:39:12.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.304211+0000) 2022-01-31T21:39:12.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:12 smithi167 conmon[60316]: debug 2022-01-31T21:39:12.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.303884+0000) 2022-01-31T21:39:12.727 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[35325]: debug 2022-01-31T21:39:12.488+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146797 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:12.728 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[41853]: debug 2022-01-31T21:39:12.466+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.468278+0000) 2022-01-31T21:39:12.729 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[46715]: debug 2022-01-31T21:39:12.466+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.468163+0000) 2022-01-31T21:39:12.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[51620]: debug 2022-01-31T21:39:12.467+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.469367+0000) 2022-01-31T21:39:12.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:12 smithi167 conmon[54076]: debug 2022-01-31T21:39:12.478+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.478648+0000) 2022-01-31T21:39:12.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:12 smithi167 conmon[60316]: debug 2022-01-31T21:39:12.467+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.468004+0000) 2022-01-31T21:39:12.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:12 smithi167 conmon[49112]: debug 2022-01-31T21:39:12.467+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.468220+0000) 2022-01-31T21:39:12.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:12 smithi167 conmon[49112]: debug 2022-01-31T21:39:12.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.640464+0000) 2022-01-31T21:39:12.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:12 smithi171 conmon[51620]: debug 2022-01-31T21:39:12.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:12.913644+0000) 2022-01-31T21:39:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:13 smithi171 conmon[41853]: debug 2022-01-31T21:39:13.030+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:13.032152+0000) 2022-01-31T21:39:13.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:13 smithi171 conmon[46715]: debug 2022-01-31T21:39:13.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:13.076045+0000) 2022-01-31T21:39:13.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:13 smithi167 conmon[54076]: debug 2022-01-31T21:39:13.303+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:13.304374+0000) 2022-01-31T21:39:13.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:13 smithi167 conmon[60316]: debug 2022-01-31T21:39:13.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:13.304032+0000) 2022-01-31T21:39:13.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:13 smithi167 conmon[49112]: debug 2022-01-31T21:39:13.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:13.640602+0000) 2022-01-31T21:39:13.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:13 smithi171 conmon[51620]: debug 2022-01-31T21:39:13.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:13.913815+0000) 2022-01-31T21:39:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:14 smithi171 conmon[46715]: debug 2022-01-31T21:39:14.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:14.076191+0000) 2022-01-31T21:39:14.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:14 smithi171 conmon[41853]: debug 2022-01-31T21:39:14.031+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:14.032271+0000) 2022-01-31T21:39:14.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:14 smithi167 conmon[54076]: debug 2022-01-31T21:39:14.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:14.304489+0000) 2022-01-31T21:39:14.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:14 smithi167 conmon[60316]: debug 2022-01-31T21:39:14.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:14.304212+0000) 2022-01-31T21:39:14.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:14 smithi167 conmon[49112]: debug 2022-01-31T21:39:14.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:14.640785+0000) 2022-01-31T21:39:14.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:14 smithi171 conmon[51620]: debug 2022-01-31T21:39:14.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:14.914027+0000) 2022-01-31T21:39:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:15 smithi171 conmon[41853]: debug 2022-01-31T21:39:15.031+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:15.032440+0000) 2022-01-31T21:39:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:15 smithi171 conmon[46715]: debug 2022-01-31T21:39:15.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:15.076347+0000) 2022-01-31T21:39:15.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:15 smithi167 conmon[54076]: debug 2022-01-31T21:39:15.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:15.304647+0000) 2022-01-31T21:39:15.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:15 smithi167 conmon[60316]: debug 2022-01-31T21:39:15.303+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:15.304418+0000) 2022-01-31T21:39:15.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:15 smithi167 conmon[49112]: debug 2022-01-31T21:39:15.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:15.640967+0000) 2022-01-31T21:39:15.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:15 smithi171 conmon[51620]: debug 2022-01-31T21:39:15.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:15.914239+0000) 2022-01-31T21:39:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:16 smithi171 conmon[46715]: debug 2022-01-31T21:39:16.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:16.076532+0000) 2022-01-31T21:39:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:16 smithi171 conmon[41853]: debug 2022-01-31T21:39:16.031+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:16.032620+0000) 2022-01-31T21:39:16.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:16 smithi167 conmon[54076]: debug 2022-01-31T21:39:16.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:16.304783+0000) 2022-01-31T21:39:16.359 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:16 smithi167 conmon[60316]: debug 2022-01-31T21:39:16.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:16.304630+0000) 2022-01-31T21:39:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:16 smithi167 conmon[49112]: debug 2022-01-31T21:39:16.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:16.641148+0000) 2022-01-31T21:39:16.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:16 smithi171 conmon[51620]: debug 2022-01-31T21:39:16.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:16.914438+0000) 2022-01-31T21:39:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[41853]: debug 2022-01-31T21:39:17.032+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.032792+0000) 2022-01-31T21:39:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[46715]: debug 2022-01-31T21:39:17.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.076722+0000) 2022-01-31T21:39:17.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:17 smithi167 conmon[54076]: debug 2022-01-31T21:39:17.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.304924+0000) 2022-01-31T21:39:17.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:17 smithi167 conmon[60316]: debug 2022-01-31T21:39:17.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.304783+0000) 2022-01-31T21:39:17.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:17 smithi167 conmon[54076]: debug 2022-01-31T21:39:17.491+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.492287+0000) 2022-01-31T21:39:17.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:17 smithi167 conmon[60316]: debug 2022-01-31T21:39:17.491+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.491720+0000) 2022-01-31T21:39:17.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:17 smithi167 conmon[49112]: debug 2022-01-31T21:39:17.490+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.491001+0000) 2022-01-31T21:39:17.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:17 smithi167 conmon[49112]: debug 2022-01-31T21:39:17.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.641365+0000) 2022-01-31T21:39:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[41853]: debug 2022-01-31T21:39:17.492+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.493225+0000) 2022-01-31T21:39:17.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[46715]: debug 2022-01-31T21:39:17.491+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.492590+0000) 2022-01-31T21:39:17.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[35325]: debug 2022-01-31T21:39:17.507+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 146903 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:17.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[51620]: debug 2022-01-31T21:39:17.491+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.492216+0000) 2022-01-31T21:39:18.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:17 smithi171 conmon[51620]: debug 2022-01-31T21:39:17.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:17.914614+0000) 2022-01-31T21:39:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:18 smithi171 conmon[41853]: debug 2022-01-31T21:39:18.032+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:18.032952+0000) 2022-01-31T21:39:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:18 smithi171 conmon[46715]: debug 2022-01-31T21:39:18.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:18.076873+0000) 2022-01-31T21:39:18.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:18 smithi167 conmon[54076]: debug 2022-01-31T21:39:18.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:18.305131+0000) 2022-01-31T21:39:18.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:18 smithi167 conmon[60316]: debug 2022-01-31T21:39:18.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:18.304994+0000) 2022-01-31T21:39:18.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:18 smithi167 conmon[49112]: debug 2022-01-31T21:39:18.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:18.641529+0000) 2022-01-31T21:39:18.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:18 smithi171 conmon[51620]: debug 2022-01-31T21:39:18.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:18.914790+0000) 2022-01-31T21:39:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:19 smithi171 conmon[46715]: debug 2022-01-31T21:39:19.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:19.077028+0000) 2022-01-31T21:39:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:19 smithi171 conmon[41853]: debug 2022-01-31T21:39:19.032+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:19.033049+0000) 2022-01-31T21:39:19.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:19 smithi167 conmon[54076]: debug 2022-01-31T21:39:19.304+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:19.305258+0000) 2022-01-31T21:39:19.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:19 smithi167 conmon[60316]: debug 2022-01-31T21:39:19.304+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:19.305170+0000) 2022-01-31T21:39:19.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:19 smithi167 conmon[49112]: debug 2022-01-31T21:39:19.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:19.641702+0000) 2022-01-31T21:39:19.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:19 smithi171 conmon[51620]: debug 2022-01-31T21:39:19.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:19.915001+0000) 2022-01-31T21:39:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:20 smithi171 conmon[41853]: debug 2022-01-31T21:39:20.032+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:20.033168+0000) 2022-01-31T21:39:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:20 smithi171 conmon[46715]: debug 2022-01-31T21:39:20.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:20.077190+0000) 2022-01-31T21:39:20.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:20 smithi167 conmon[54076]: debug 2022-01-31T21:39:20.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:20.305434+0000) 2022-01-31T21:39:20.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:20 smithi167 conmon[60316]: debug 2022-01-31T21:39:20.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:20.305354+0000) 2022-01-31T21:39:20.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:20 smithi167 conmon[49112]: debug 2022-01-31T21:39:20.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:20.641874+0000) 2022-01-31T21:39:20.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:20 smithi171 conmon[51620]: debug 2022-01-31T21:39:20.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:20.915229+0000) 2022-01-31T21:39:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:21 smithi171 conmon[41853]: debug 2022-01-31T21:39:21.032+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:21.033352+0000) 2022-01-31T21:39:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:21 smithi171 conmon[46715]: debug 2022-01-31T21:39:21.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:21.077351+0000) 2022-01-31T21:39:21.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:21 smithi167 conmon[54076]: debug 2022-01-31T21:39:21.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:21.305604+0000) 2022-01-31T21:39:21.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:21 smithi167 conmon[60316]: debug 2022-01-31T21:39:21.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:21.305538+0000) 2022-01-31T21:39:21.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:21 smithi167 conmon[49112]: debug 2022-01-31T21:39:21.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:21.642013+0000) 2022-01-31T21:39:21.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:21 smithi171 conmon[51620]: debug 2022-01-31T21:39:21.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:21.915441+0000) 2022-01-31T21:39:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[41853]: debug 2022-01-31T21:39:22.032+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.033558+0000) 2022-01-31T21:39:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[46715]: debug 2022-01-31T21:39:22.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.077534+0000) 2022-01-31T21:39:22.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:22 smithi167 conmon[54076]: debug 2022-01-31T21:39:22.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.305745+0000) 2022-01-31T21:39:22.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:22 smithi167 conmon[60316]: debug 2022-01-31T21:39:22.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.305682+0000) 2022-01-31T21:39:22.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:22 smithi167 conmon[54076]: debug 2022-01-31T21:39:22.510+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.510777+0000) 2022-01-31T21:39:22.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:22 smithi167 conmon[60316]: debug 2022-01-31T21:39:22.509+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.509882+0000) 2022-01-31T21:39:22.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:22 smithi167 conmon[49112]: debug 2022-01-31T21:39:22.509+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.509712+0000) 2022-01-31T21:39:22.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:22 smithi167 conmon[49112]: debug 2022-01-31T21:39:22.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.642147+0000) 2022-01-31T21:39:22.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[35325]: debug 2022-01-31T21:39:22.525+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147015 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[41853]: debug 2022-01-31T21:39:22.510+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.511521+0000) 2022-01-31T21:39:22.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[46715]: debug 2022-01-31T21:39:22.509+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.510532+0000) 2022-01-31T21:39:22.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[51620]: debug 2022-01-31T21:39:22.509+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.510033+0000) 2022-01-31T21:39:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:23 smithi171 conmon[41853]: debug 2022-01-31T21:39:23.033+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:23.033794+0000) 2022-01-31T21:39:23.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:22 smithi171 conmon[51620]: debug 2022-01-31T21:39:22.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:22.915648+0000) 2022-01-31T21:39:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:23 smithi171 conmon[46715]: debug 2022-01-31T21:39:23.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:23.077722+0000) 2022-01-31T21:39:23.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:23 smithi167 conmon[60316]: debug 2022-01-31T21:39:23.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:23.305835+0000) 2022-01-31T21:39:23.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:23 smithi167 conmon[54076]: debug 2022-01-31T21:39:23.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:23.305906+0000) 2022-01-31T21:39:23.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:23 smithi167 conmon[49112]: debug 2022-01-31T21:39:23.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:23.642286+0000) 2022-01-31T21:39:23.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:23 smithi171 conmon[51620]: debug 2022-01-31T21:39:23.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:23.915840+0000) 2022-01-31T21:39:23.988 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:23 smithi171 conmon[35325]: debug 2022-01-31T21:39:23.715+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:39:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:24 smithi171 conmon[41853]: debug 2022-01-31T21:39:24.033+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:24.033988+0000) 2022-01-31T21:39:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:24 smithi171 conmon[46715]: debug 2022-01-31T21:39:24.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:24.077878+0000) 2022-01-31T21:39:24.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:24 smithi167 conmon[54076]: debug 2022-01-31T21:39:24.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:24.306028+0000) 2022-01-31T21:39:24.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:24 smithi167 conmon[60316]: debug 2022-01-31T21:39:24.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:24.305986+0000) 2022-01-31T21:39:24.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:24 smithi167 conmon[49112]: debug 2022-01-31T21:39:24.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:24.642452+0000) 2022-01-31T21:39:24.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:24 smithi171 conmon[51620]: debug 2022-01-31T21:39:24.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:24.916043+0000) 2022-01-31T21:39:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:25 smithi171 conmon[41853]: debug 2022-01-31T21:39:25.033+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:25.034195+0000) 2022-01-31T21:39:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:25 smithi171 conmon[46715]: debug 2022-01-31T21:39:25.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:25.078051+0000) 2022-01-31T21:39:25.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:25 smithi167 conmon[54076]: debug 2022-01-31T21:39:25.305+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:25.306242+0000) 2022-01-31T21:39:25.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:25 smithi167 conmon[60316]: debug 2022-01-31T21:39:25.305+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:25.306193+0000) 2022-01-31T21:39:25.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:25 smithi167 conmon[49112]: debug 2022-01-31T21:39:25.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:25.642622+0000) 2022-01-31T21:39:25.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:25 smithi171 conmon[51620]: debug 2022-01-31T21:39:25.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:25.916197+0000) 2022-01-31T21:39:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:26 smithi171 conmon[41853]: debug 2022-01-31T21:39:26.033+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:26.034386+0000) 2022-01-31T21:39:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:26 smithi171 conmon[46715]: debug 2022-01-31T21:39:26.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:26.078236+0000) 2022-01-31T21:39:26.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:26 smithi167 conmon[60316]: debug 2022-01-31T21:39:26.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:26.306355+0000) 2022-01-31T21:39:26.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:26 smithi167 conmon[54076]: debug 2022-01-31T21:39:26.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:26.306412+0000) 2022-01-31T21:39:26.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:26 smithi167 conmon[49112]: debug 2022-01-31T21:39:26.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:26.642860+0000) 2022-01-31T21:39:26.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:26 smithi171 conmon[51620]: debug 2022-01-31T21:39:26.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:26.916331+0000) 2022-01-31T21:39:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[41853]: debug 2022-01-31T21:39:27.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.034536+0000) 2022-01-31T21:39:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[46715]: debug 2022-01-31T21:39:27.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.078423+0000) 2022-01-31T21:39:27.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:27 smithi167 conmon[54076]: debug 2022-01-31T21:39:27.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.306595+0000) 2022-01-31T21:39:27.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:27 smithi167 conmon[60316]: debug 2022-01-31T21:39:27.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.306530+0000) 2022-01-31T21:39:27.807 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:27 smithi167 conmon[49112]: debug 2022-01-31T21:39:27.527+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.527682+0000) 2022-01-31T21:39:27.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:27 smithi167 conmon[49112]: debug 2022-01-31T21:39:27.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.643042+0000) 2022-01-31T21:39:27.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:27 smithi167 conmon[54076]: debug 2022-01-31T21:39:27.529+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.529698+0000) 2022-01-31T21:39:27.809 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:27 smithi167 conmon[60316]: debug 2022-01-31T21:39:27.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.528238+0000) 2022-01-31T21:39:27.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[35325]: debug 2022-01-31T21:39:27.543+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147125 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[41853]: debug 2022-01-31T21:39:27.529+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.530147+0000) 2022-01-31T21:39:27.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[46715]: debug 2022-01-31T21:39:27.528+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.528831+0000) 2022-01-31T21:39:27.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[51620]: debug 2022-01-31T21:39:27.528+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.529692+0000) 2022-01-31T21:39:28.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:27 smithi171 conmon[51620]: debug 2022-01-31T21:39:27.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:27.916530+0000) 2022-01-31T21:39:28.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:28 smithi171 conmon[41853]: debug 2022-01-31T21:39:28.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:28.034736+0000) 2022-01-31T21:39:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:28 smithi171 conmon[46715]: debug 2022-01-31T21:39:28.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:28.078551+0000) 2022-01-31T21:39:28.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:28 smithi167 conmon[54076]: debug 2022-01-31T21:39:28.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:28.306800+0000) 2022-01-31T21:39:28.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:28 smithi167 conmon[60316]: debug 2022-01-31T21:39:28.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:28.306730+0000) 2022-01-31T21:39:28.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:28 smithi167 conmon[49112]: debug 2022-01-31T21:39:28.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:28.643206+0000) 2022-01-31T21:39:28.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:28 smithi171 conmon[51620]: debug 2022-01-31T21:39:28.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:28.916669+0000) 2022-01-31T21:39:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:29 smithi171 conmon[46715]: debug 2022-01-31T21:39:29.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:29.078692+0000) 2022-01-31T21:39:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:29 smithi171 conmon[41853]: debug 2022-01-31T21:39:29.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:29.034843+0000) 2022-01-31T21:39:29.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:29 smithi167 conmon[54076]: debug 2022-01-31T21:39:29.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:29.306948+0000) 2022-01-31T21:39:29.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:29 smithi167 conmon[60316]: debug 2022-01-31T21:39:29.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:29.306879+0000) 2022-01-31T21:39:29.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:29 smithi167 conmon[49112]: debug 2022-01-31T21:39:29.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:29.643396+0000) 2022-01-31T21:39:29.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:29 smithi171 conmon[51620]: debug 2022-01-31T21:39:29.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:29.916845+0000) 2022-01-31T21:39:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:30 smithi171 conmon[41853]: debug 2022-01-31T21:39:30.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:30.034965+0000) 2022-01-31T21:39:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:30 smithi171 conmon[46715]: debug 2022-01-31T21:39:30.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:30.078894+0000) 2022-01-31T21:39:30.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:30 smithi167 conmon[54076]: debug 2022-01-31T21:39:30.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:30.307126+0000) 2022-01-31T21:39:30.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:30 smithi167 conmon[60316]: debug 2022-01-31T21:39:30.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:30.307062+0000) 2022-01-31T21:39:30.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:30 smithi167 conmon[49112]: debug 2022-01-31T21:39:30.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:30.643593+0000) 2022-01-31T21:39:30.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:30 smithi171 conmon[51620]: debug 2022-01-31T21:39:30.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:30.917048+0000) 2022-01-31T21:39:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:31 smithi171 conmon[41853]: debug 2022-01-31T21:39:31.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:31.035108+0000) 2022-01-31T21:39:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:31 smithi171 conmon[46715]: debug 2022-01-31T21:39:31.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:31.079113+0000) 2022-01-31T21:39:31.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:31 smithi167 conmon[54076]: debug 2022-01-31T21:39:31.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:31.307313+0000) 2022-01-31T21:39:31.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:31 smithi167 conmon[60316]: debug 2022-01-31T21:39:31.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:31.307240+0000) 2022-01-31T21:39:31.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:31 smithi167 conmon[49112]: debug 2022-01-31T21:39:31.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:31.643774+0000) 2022-01-31T21:39:31.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:31 smithi171 conmon[51620]: debug 2022-01-31T21:39:31.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:31.917204+0000) 2022-01-31T21:39:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[41853]: debug 2022-01-31T21:39:32.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.035258+0000) 2022-01-31T21:39:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[46715]: debug 2022-01-31T21:39:32.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.079320+0000) 2022-01-31T21:39:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:32 smithi167 conmon[54076]: debug 2022-01-31T21:39:32.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.307468+0000) 2022-01-31T21:39:32.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:32 smithi167 conmon[60316]: debug 2022-01-31T21:39:32.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.307423+0000) 2022-01-31T21:39:32.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:32 smithi167 conmon[49112]: debug 2022-01-31T21:39:32.545+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.546018+0000) 2022-01-31T21:39:32.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:32 smithi167 conmon[49112]: debug 2022-01-31T21:39:32.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.643933+0000) 2022-01-31T21:39:32.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:32 smithi167 conmon[54076]: debug 2022-01-31T21:39:32.546+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.547268+0000) 2022-01-31T21:39:32.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:32 smithi167 conmon[60316]: debug 2022-01-31T21:39:32.545+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.545856+0000) 2022-01-31T21:39:32.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[35325]: debug 2022-01-31T21:39:32.562+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147236 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[41853]: debug 2022-01-31T21:39:32.546+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.547194+0000) 2022-01-31T21:39:32.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[46715]: debug 2022-01-31T21:39:32.547+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.548076+0000) 2022-01-31T21:39:32.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[51620]: debug 2022-01-31T21:39:32.546+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.547251+0000) 2022-01-31T21:39:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:33 smithi171 conmon[41853]: debug 2022-01-31T21:39:33.034+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:33.035481+0000) 2022-01-31T21:39:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:33 smithi171 conmon[46715]: debug 2022-01-31T21:39:33.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:33.079508+0000) 2022-01-31T21:39:33.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:32 smithi171 conmon[51620]: debug 2022-01-31T21:39:32.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:32.917411+0000) 2022-01-31T21:39:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:33 smithi167 conmon[54076]: debug 2022-01-31T21:39:33.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:33.307645+0000) 2022-01-31T21:39:33.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:33 smithi167 conmon[60316]: debug 2022-01-31T21:39:33.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:33.307574+0000) 2022-01-31T21:39:33.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:33 smithi167 conmon[49112]: debug 2022-01-31T21:39:33.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:33.644143+0000) 2022-01-31T21:39:33.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:33 smithi171 conmon[51620]: debug 2022-01-31T21:39:33.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:33.917582+0000) 2022-01-31T21:39:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:34 smithi171 conmon[41853]: debug 2022-01-31T21:39:34.035+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:34.035614+0000) 2022-01-31T21:39:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:34 smithi171 conmon[46715]: debug 2022-01-31T21:39:34.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:34.079680+0000) 2022-01-31T21:39:34.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:34 smithi167 conmon[54076]: debug 2022-01-31T21:39:34.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:34.307746+0000) 2022-01-31T21:39:34.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:34 smithi167 conmon[60316]: debug 2022-01-31T21:39:34.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:34.307712+0000) 2022-01-31T21:39:34.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:34 smithi167 conmon[49112]: debug 2022-01-31T21:39:34.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:34.644351+0000) 2022-01-31T21:39:34.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:34 smithi171 conmon[51620]: debug 2022-01-31T21:39:34.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:34.917938+0000) 2022-01-31T21:39:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:35 smithi171 conmon[41853]: debug 2022-01-31T21:39:35.035+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:35.035821+0000) 2022-01-31T21:39:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:35 smithi171 conmon[46715]: debug 2022-01-31T21:39:35.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:35.079881+0000) 2022-01-31T21:39:35.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:35 smithi167 conmon[54076]: debug 2022-01-31T21:39:35.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:35.307948+0000) 2022-01-31T21:39:35.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:35 smithi167 conmon[60316]: debug 2022-01-31T21:39:35.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:35.307926+0000) 2022-01-31T21:39:35.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:35 smithi167 conmon[49112]: debug 2022-01-31T21:39:35.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:35.644523+0000) 2022-01-31T21:39:35.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:35 smithi171 conmon[51620]: debug 2022-01-31T21:39:35.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:35.918123+0000) 2022-01-31T21:39:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:36 smithi171 conmon[46715]: debug 2022-01-31T21:39:36.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:36.080054+0000) 2022-01-31T21:39:36.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:36 smithi171 conmon[41853]: debug 2022-01-31T21:39:36.035+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:36.035950+0000) 2022-01-31T21:39:36.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:36 smithi167 conmon[54076]: debug 2022-01-31T21:39:36.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:36.308106+0000) 2022-01-31T21:39:36.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:36 smithi167 conmon[60316]: debug 2022-01-31T21:39:36.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:36.308128+0000) 2022-01-31T21:39:36.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:36 smithi167 conmon[49112]: debug 2022-01-31T21:39:36.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:36.644669+0000) 2022-01-31T21:39:36.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:36 smithi171 conmon[51620]: debug 2022-01-31T21:39:36.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:36.918283+0000) 2022-01-31T21:39:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[46715]: debug 2022-01-31T21:39:37.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.080216+0000) 2022-01-31T21:39:37.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[41853]: debug 2022-01-31T21:39:37.035+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.036080+0000) 2022-01-31T21:39:37.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:37 smithi167 conmon[54076]: debug 2022-01-31T21:39:37.306+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.308321+0000) 2022-01-31T21:39:37.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:37 smithi167 conmon[60316]: debug 2022-01-31T21:39:37.306+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.308325+0000) 2022-01-31T21:39:37.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:37 smithi167 conmon[54076]: debug 2022-01-31T21:39:37.563+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.565301+0000) 2022-01-31T21:39:37.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:37 smithi167 conmon[60316]: debug 2022-01-31T21:39:37.562+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.564333+0000) 2022-01-31T21:39:37.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:37 smithi167 conmon[49112]: debug 2022-01-31T21:39:37.562+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.564248+0000) 2022-01-31T21:39:37.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:37 smithi167 conmon[49112]: debug 2022-01-31T21:39:37.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.644891+0000) 2022-01-31T21:39:37.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[46715]: debug 2022-01-31T21:39:37.566+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.566804+0000) 2022-01-31T21:39:37.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[35325]: debug 2022-01-31T21:39:37.580+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147346 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[41853]: debug 2022-01-31T21:39:37.565+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.565995+0000) 2022-01-31T21:39:37.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[51620]: debug 2022-01-31T21:39:37.565+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.566440+0000) 2022-01-31T21:39:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:38 smithi171 conmon[46715]: debug 2022-01-31T21:39:38.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:38.080426+0000) 2022-01-31T21:39:38.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:38 smithi171 conmon[41853]: debug 2022-01-31T21:39:38.035+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:38.036232+0000) 2022-01-31T21:39:38.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:37 smithi171 conmon[51620]: debug 2022-01-31T21:39:37.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:37.918423+0000) 2022-01-31T21:39:38.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:38 smithi167 conmon[54076]: debug 2022-01-31T21:39:38.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:38.308487+0000) 2022-01-31T21:39:38.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:38 smithi167 conmon[60316]: debug 2022-01-31T21:39:38.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:38.308543+0000) 2022-01-31T21:39:38.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:38 smithi167 conmon[49112]: debug 2022-01-31T21:39:38.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:38.645079+0000) 2022-01-31T21:39:38.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:38 smithi171 conmon[51620]: debug 2022-01-31T21:39:38.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:38.918579+0000) 2022-01-31T21:39:38.990 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:38 smithi171 conmon[35325]: debug 2022-01-31T21:39:38.716+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:39:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:39 smithi171 conmon[46715]: debug 2022-01-31T21:39:39.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:39.080626+0000) 2022-01-31T21:39:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:39 smithi171 conmon[41853]: debug 2022-01-31T21:39:39.035+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:39.036382+0000) 2022-01-31T21:39:39.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:39 smithi167 conmon[54076]: debug 2022-01-31T21:39:39.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:39.308677+0000) 2022-01-31T21:39:39.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:39 smithi167 conmon[60316]: debug 2022-01-31T21:39:39.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:39.308693+0000) 2022-01-31T21:39:39.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:39 smithi167 conmon[49112]: debug 2022-01-31T21:39:39.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:39.645304+0000) 2022-01-31T21:39:39.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:39 smithi171 conmon[51620]: debug 2022-01-31T21:39:39.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:39.918747+0000) 2022-01-31T21:39:40.263 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:40 smithi171 conmon[46715]: debug 2022-01-31T21:39:40.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:40.080845+0000) 2022-01-31T21:39:40.264 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:40 smithi171 conmon[41853]: debug 2022-01-31T21:39:40.036+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:40.036586+0000) 2022-01-31T21:39:40.443 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:40 smithi167 conmon[54076]: debug 2022-01-31T21:39:40.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:40.308848+0000) 2022-01-31T21:39:40.444 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:40 smithi167 conmon[60316]: debug 2022-01-31T21:39:40.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:40.308848+0000) 2022-01-31T21:39:40.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:40 smithi167 conmon[49112]: debug 2022-01-31T21:39:40.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:40.645543+0000) 2022-01-31T21:39:40.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:40 smithi171 conmon[51620]: debug 2022-01-31T21:39:40.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:40.918881+0000) 2022-01-31T21:39:41.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:41 smithi171 conmon[46715]: debug 2022-01-31T21:39:41.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:41.081021+0000) 2022-01-31T21:39:41.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:41 smithi171 conmon[41853]: debug 2022-01-31T21:39:41.036+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:41.036745+0000) 2022-01-31T21:39:41.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:41 smithi167 conmon[54076]: debug 2022-01-31T21:39:41.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:41.309035+0000) 2022-01-31T21:39:41.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:41 smithi167 conmon[60316]: debug 2022-01-31T21:39:41.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:41.309054+0000) 2022-01-31T21:39:41.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:41 smithi167 conmon[49112]: debug 2022-01-31T21:39:41.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:41.645750+0000) 2022-01-31T21:39:41.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:41 smithi171 conmon[51620]: debug 2022-01-31T21:39:41.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:41.919044+0000) 2022-01-31T21:39:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[41853]: debug 2022-01-31T21:39:42.036+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.036907+0000) 2022-01-31T21:39:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[46715]: debug 2022-01-31T21:39:42.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.081221+0000) 2022-01-31T21:39:42.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:42 smithi167 conmon[54076]: debug 2022-01-31T21:39:42.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.309251+0000) 2022-01-31T21:39:42.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:42 smithi167 conmon[60316]: debug 2022-01-31T21:39:42.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.309249+0000) 2022-01-31T21:39:42.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:42 smithi167 conmon[54076]: debug 2022-01-31T21:39:42.581+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.582993+0000) 2022-01-31T21:39:42.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:42 smithi167 conmon[49112]: debug 2022-01-31T21:39:42.582+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.584061+0000) 2022-01-31T21:39:42.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:42 smithi167 conmon[49112]: debug 2022-01-31T21:39:42.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.645938+0000) 2022-01-31T21:39:42.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:42 smithi167 conmon[60316]: debug 2022-01-31T21:39:42.581+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.583101+0000) 2022-01-31T21:39:42.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[41853]: debug 2022-01-31T21:39:42.583+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.584142+0000) 2022-01-31T21:39:42.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[46715]: debug 2022-01-31T21:39:42.584+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.584845+0000) 2022-01-31T21:39:42.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[51620]: debug 2022-01-31T21:39:42.584+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.584803+0000) 2022-01-31T21:39:42.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[35325]: debug 2022-01-31T21:39:42.599+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147457 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:43.295 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:43 smithi171 conmon[41853]: debug 2022-01-31T21:39:43.036+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:43.037061+0000) 2022-01-31T21:39:43.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:43 smithi171 conmon[46715]: debug 2022-01-31T21:39:43.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:43.081421+0000) 2022-01-31T21:39:43.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:42 smithi171 conmon[51620]: debug 2022-01-31T21:39:42.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:42.919278+0000) 2022-01-31T21:39:43.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:43 smithi167 conmon[54076]: debug 2022-01-31T21:39:43.307+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:43.309438+0000) 2022-01-31T21:39:43.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:43 smithi167 conmon[60316]: debug 2022-01-31T21:39:43.307+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:43.309444+0000) 2022-01-31T21:39:43.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:43 smithi167 conmon[49112]: debug 2022-01-31T21:39:43.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:43.646071+0000) 2022-01-31T21:39:43.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:43 smithi171 conmon[51620]: debug 2022-01-31T21:39:43.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:43.919424+0000) 2022-01-31T21:39:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:44 smithi171 conmon[41853]: debug 2022-01-31T21:39:44.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:44.037215+0000) 2022-01-31T21:39:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:44 smithi171 conmon[46715]: debug 2022-01-31T21:39:44.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:44.081574+0000) 2022-01-31T21:39:44.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:44 smithi167 conmon[54076]: debug 2022-01-31T21:39:44.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:44.309588+0000) 2022-01-31T21:39:44.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:44 smithi167 conmon[60316]: debug 2022-01-31T21:39:44.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:44.309588+0000) 2022-01-31T21:39:44.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:44 smithi167 conmon[49112]: debug 2022-01-31T21:39:44.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:44.646307+0000) 2022-01-31T21:39:44.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:44 smithi171 conmon[51620]: debug 2022-01-31T21:39:44.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:44.919543+0000) 2022-01-31T21:39:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:45 smithi171 conmon[41853]: debug 2022-01-31T21:39:45.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:45.037405+0000) 2022-01-31T21:39:45.378 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:45 smithi171 conmon[46715]: debug 2022-01-31T21:39:45.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:45.081778+0000) 2022-01-31T21:39:45.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:45 smithi167 conmon[54076]: debug 2022-01-31T21:39:45.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:45.309770+0000) 2022-01-31T21:39:45.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:45 smithi167 conmon[60316]: debug 2022-01-31T21:39:45.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:45.309768+0000) 2022-01-31T21:39:45.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:45 smithi167 conmon[49112]: debug 2022-01-31T21:39:45.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:45.646472+0000) 2022-01-31T21:39:45.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:45 smithi171 conmon[51620]: debug 2022-01-31T21:39:45.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:45.919723+0000) 2022-01-31T21:39:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:46 smithi171 conmon[41853]: debug 2022-01-31T21:39:46.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:46.037534+0000) 2022-01-31T21:39:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:46 smithi171 conmon[46715]: debug 2022-01-31T21:39:46.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:46.081964+0000) 2022-01-31T21:39:46.364 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:46 smithi167 conmon[54076]: debug 2022-01-31T21:39:46.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:46.309954+0000) 2022-01-31T21:39:46.364 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:46 smithi167 conmon[60316]: debug 2022-01-31T21:39:46.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:46.309936+0000) 2022-01-31T21:39:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:46 smithi167 conmon[49112]: debug 2022-01-31T21:39:46.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:46.646637+0000) 2022-01-31T21:39:46.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:46 smithi171 conmon[51620]: debug 2022-01-31T21:39:46.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:46.919893+0000) 2022-01-31T21:39:47.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[46715]: debug 2022-01-31T21:39:47.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.082155+0000) 2022-01-31T21:39:47.327 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[41853]: debug 2022-01-31T21:39:47.036+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.037705+0000) 2022-01-31T21:39:47.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:47 smithi167 conmon[54076]: debug 2022-01-31T21:39:47.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.310141+0000) 2022-01-31T21:39:47.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:47 smithi167 conmon[60316]: debug 2022-01-31T21:39:47.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.310109+0000) 2022-01-31T21:39:47.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:47 smithi167 conmon[54076]: debug 2022-01-31T21:39:47.599+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.600778+0000) 2022-01-31T21:39:47.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:47 smithi167 conmon[60316]: debug 2022-01-31T21:39:47.600+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.601966+0000) 2022-01-31T21:39:47.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:47 smithi167 conmon[49112]: debug 2022-01-31T21:39:47.600+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.601771+0000) 2022-01-31T21:39:47.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:47 smithi167 conmon[49112]: debug 2022-01-31T21:39:47.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.646806+0000) 2022-01-31T21:39:47.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[35325]: debug 2022-01-31T21:39:47.617+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147567 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:47.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[41853]: debug 2022-01-31T21:39:47.602+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.603151+0000) 2022-01-31T21:39:47.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[46715]: debug 2022-01-31T21:39:47.601+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.601917+0000) 2022-01-31T21:39:47.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[51620]: debug 2022-01-31T21:39:47.601+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.601931+0000) 2022-01-31T21:39:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:48 smithi171 conmon[41853]: debug 2022-01-31T21:39:48.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:48.037903+0000) 2022-01-31T21:39:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:48 smithi171 conmon[46715]: debug 2022-01-31T21:39:48.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:48.082352+0000) 2022-01-31T21:39:48.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:47 smithi171 conmon[51620]: debug 2022-01-31T21:39:47.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:47.920041+0000) 2022-01-31T21:39:48.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:48 smithi167 conmon[54076]: debug 2022-01-31T21:39:48.308+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:48.310323+0000) 2022-01-31T21:39:48.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:48 smithi167 conmon[60316]: debug 2022-01-31T21:39:48.308+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:48.310263+0000) 2022-01-31T21:39:48.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:48 smithi167 conmon[49112]: debug 2022-01-31T21:39:48.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:48.646972+0000) 2022-01-31T21:39:48.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:48 smithi171 conmon[51620]: debug 2022-01-31T21:39:48.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:48.920183+0000) 2022-01-31T21:39:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:49 smithi171 conmon[41853]: debug 2022-01-31T21:39:49.038+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:49.038067+0000) 2022-01-31T21:39:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:49 smithi171 conmon[46715]: debug 2022-01-31T21:39:49.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:49.082493+0000) 2022-01-31T21:39:49.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:49 smithi167 conmon[54076]: debug 2022-01-31T21:39:49.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:49.310464+0000) 2022-01-31T21:39:49.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:49 smithi167 conmon[60316]: debug 2022-01-31T21:39:49.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:49.310456+0000) 2022-01-31T21:39:49.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:49 smithi167 conmon[49112]: debug 2022-01-31T21:39:49.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:49.647165+0000) 2022-01-31T21:39:49.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:49 smithi171 conmon[51620]: debug 2022-01-31T21:39:49.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:49.920343+0000) 2022-01-31T21:39:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:50 smithi171 conmon[46715]: debug 2022-01-31T21:39:50.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:50.082685+0000) 2022-01-31T21:39:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:50 smithi171 conmon[41853]: debug 2022-01-31T21:39:50.036+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:50.038249+0000) 2022-01-31T21:39:50.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:50 smithi167 conmon[54076]: debug 2022-01-31T21:39:50.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:50.310702+0000) 2022-01-31T21:39:50.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:50 smithi167 conmon[60316]: debug 2022-01-31T21:39:50.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:50.310668+0000) 2022-01-31T21:39:50.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:50 smithi167 conmon[49112]: debug 2022-01-31T21:39:50.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:50.647387+0000) 2022-01-31T21:39:50.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:50 smithi171 conmon[51620]: debug 2022-01-31T21:39:50.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:50.920525+0000) 2022-01-31T21:39:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:51 smithi171 conmon[41853]: debug 2022-01-31T21:39:51.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:51.038404+0000) 2022-01-31T21:39:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:51 smithi171 conmon[46715]: debug 2022-01-31T21:39:51.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:51.082915+0000) 2022-01-31T21:39:51.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:51 smithi167 conmon[54076]: debug 2022-01-31T21:39:51.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:51.310884+0000) 2022-01-31T21:39:51.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:51 smithi167 conmon[60316]: debug 2022-01-31T21:39:51.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:51.310884+0000) 2022-01-31T21:39:51.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:51 smithi167 conmon[49112]: debug 2022-01-31T21:39:51.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:51.647576+0000) 2022-01-31T21:39:51.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:51 smithi171 conmon[51620]: debug 2022-01-31T21:39:51.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:51.920762+0000) 2022-01-31T21:39:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[41853]: debug 2022-01-31T21:39:52.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.038579+0000) 2022-01-31T21:39:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[46715]: debug 2022-01-31T21:39:52.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.083128+0000) 2022-01-31T21:39:52.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:52 smithi167 conmon[54076]: debug 2022-01-31T21:39:52.309+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.311090+0000) 2022-01-31T21:39:52.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:52 smithi167 conmon[60316]: debug 2022-01-31T21:39:52.309+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.311075+0000) 2022-01-31T21:39:52.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:52 smithi167 conmon[54076]: debug 2022-01-31T21:39:52.618+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.619730+0000) 2022-01-31T21:39:52.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:52 smithi167 conmon[60316]: debug 2022-01-31T21:39:52.618+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.620218+0000) 2022-01-31T21:39:52.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:52 smithi167 conmon[49112]: debug 2022-01-31T21:39:52.619+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.620620+0000) 2022-01-31T21:39:52.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:52 smithi167 conmon[49112]: debug 2022-01-31T21:39:52.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.647706+0000) 2022-01-31T21:39:52.918 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[35325]: debug 2022-01-31T21:39:52.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147678 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:52.919 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[41853]: debug 2022-01-31T21:39:52.620+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.621883+0000) 2022-01-31T21:39:52.920 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[46715]: debug 2022-01-31T21:39:52.619+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.620800+0000) 2022-01-31T21:39:52.920 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[51620]: debug 2022-01-31T21:39:52.619+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.621151+0000) 2022-01-31T21:39:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:53 smithi171 conmon[41853]: debug 2022-01-31T21:39:53.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:53.038792+0000) 2022-01-31T21:39:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:53 smithi171 conmon[46715]: debug 2022-01-31T21:39:53.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:53.083311+0000) 2022-01-31T21:39:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:52 smithi171 conmon[51620]: debug 2022-01-31T21:39:52.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:52.920894+0000) 2022-01-31T21:39:53.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:53 smithi167 conmon[54076]: debug 2022-01-31T21:39:53.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:53.311278+0000) 2022-01-31T21:39:53.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:53 smithi167 conmon[60316]: debug 2022-01-31T21:39:53.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:53.311277+0000) 2022-01-31T21:39:53.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:53 smithi167 conmon[49112]: debug 2022-01-31T21:39:53.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:53.647842+0000) 2022-01-31T21:39:53.992 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:53 smithi171 conmon[35325]: debug 2022-01-31T21:39:53.716+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:39:53.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:53 smithi171 conmon[51620]: debug 2022-01-31T21:39:53.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:53.921017+0000) 2022-01-31T21:39:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:54 smithi171 conmon[41853]: debug 2022-01-31T21:39:54.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:54.038940+0000) 2022-01-31T21:39:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:54 smithi171 conmon[46715]: debug 2022-01-31T21:39:54.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:54.083421+0000) 2022-01-31T21:39:54.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:54 smithi167 conmon[54076]: debug 2022-01-31T21:39:54.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:54.311446+0000) 2022-01-31T21:39:54.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:54 smithi167 conmon[60316]: debug 2022-01-31T21:39:54.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:54.311452+0000) 2022-01-31T21:39:54.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:54 smithi167 conmon[49112]: debug 2022-01-31T21:39:54.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:54.648028+0000) 2022-01-31T21:39:54.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:54 smithi171 conmon[51620]: debug 2022-01-31T21:39:54.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:54.921188+0000) 2022-01-31T21:39:55.327 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:55 smithi171 conmon[41853]: debug 2022-01-31T21:39:55.038+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:55.039093+0000) 2022-01-31T21:39:55.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:55 smithi171 conmon[46715]: debug 2022-01-31T21:39:55.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:55.083596+0000) 2022-01-31T21:39:55.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:55 smithi167 conmon[54076]: debug 2022-01-31T21:39:55.310+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:55.311588+0000) 2022-01-31T21:39:55.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:55 smithi167 conmon[60316]: debug 2022-01-31T21:39:55.310+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:55.311641+0000) 2022-01-31T21:39:55.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:55 smithi167 conmon[49112]: debug 2022-01-31T21:39:55.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:55.648125+0000) 2022-01-31T21:39:55.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:55 smithi171 conmon[51620]: debug 2022-01-31T21:39:55.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:55.921341+0000) 2022-01-31T21:39:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:56 smithi171 conmon[41853]: debug 2022-01-31T21:39:56.037+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:56.039271+0000) 2022-01-31T21:39:56.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:56 smithi171 conmon[46715]: debug 2022-01-31T21:39:56.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:56.083791+0000) 2022-01-31T21:39:56.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:56 smithi167 conmon[54076]: debug 2022-01-31T21:39:56.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:56.311723+0000) 2022-01-31T21:39:56.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:56 smithi167 conmon[60316]: debug 2022-01-31T21:39:56.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:56.311790+0000) 2022-01-31T21:39:56.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:56 smithi167 conmon[49112]: debug 2022-01-31T21:39:56.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:56.648291+0000) 2022-01-31T21:39:56.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:56 smithi171 conmon[51620]: debug 2022-01-31T21:39:56.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:56.921548+0000) 2022-01-31T21:39:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[41853]: debug 2022-01-31T21:39:57.038+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.039429+0000) 2022-01-31T21:39:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[46715]: debug 2022-01-31T21:39:57.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.083956+0000) 2022-01-31T21:39:57.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:57 smithi167 conmon[54076]: debug 2022-01-31T21:39:57.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.311928+0000) 2022-01-31T21:39:57.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:57 smithi167 conmon[60316]: debug 2022-01-31T21:39:57.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.311954+0000) 2022-01-31T21:39:57.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:57 smithi167 conmon[49112]: debug 2022-01-31T21:39:57.638+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.639548+0000) 2022-01-31T21:39:57.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:57 smithi167 conmon[49112]: debug 2022-01-31T21:39:57.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.648418+0000) 2022-01-31T21:39:57.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:57 smithi167 conmon[54076]: debug 2022-01-31T21:39:57.636+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.638091+0000) 2022-01-31T21:39:57.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:57 smithi167 conmon[60316]: debug 2022-01-31T21:39:57.636+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.638191+0000) 2022-01-31T21:39:57.919 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[35325]: debug 2022-01-31T21:39:57.653+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147788 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:39:57.922 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[41853]: debug 2022-01-31T21:39:57.637+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.639713+0000) 2022-01-31T21:39:57.923 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[46715]: debug 2022-01-31T21:39:57.638+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.639851+0000) 2022-01-31T21:39:57.923 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[51620]: debug 2022-01-31T21:39:57.637+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.638911+0000) 2022-01-31T21:39:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:58 smithi171 conmon[41853]: debug 2022-01-31T21:39:58.038+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:58.039605+0000) 2022-01-31T21:39:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:58 smithi171 conmon[46715]: debug 2022-01-31T21:39:58.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:58.084109+0000) 2022-01-31T21:39:58.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:57 smithi171 conmon[51620]: debug 2022-01-31T21:39:57.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:57.921761+0000) 2022-01-31T21:39:58.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:58 smithi167 conmon[54076]: debug 2022-01-31T21:39:58.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:58.312117+0000) 2022-01-31T21:39:58.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:58 smithi167 conmon[60316]: debug 2022-01-31T21:39:58.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:58.312154+0000) 2022-01-31T21:39:58.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:58 smithi167 conmon[49112]: debug 2022-01-31T21:39:58.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:58.648542+0000) 2022-01-31T21:39:58.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:58 smithi171 conmon[51620]: debug 2022-01-31T21:39:58.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:58.921891+0000) 2022-01-31T21:39:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:39:59 smithi171 conmon[46715]: debug 2022-01-31T21:39:59.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:59.084255+0000) 2022-01-31T21:39:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:39:59 smithi171 conmon[41853]: debug 2022-01-31T21:39:59.038+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:59.039746+0000) 2022-01-31T21:39:59.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:39:59 smithi167 conmon[54076]: debug 2022-01-31T21:39:59.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:59.312283+0000) 2022-01-31T21:39:59.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:39:59 smithi167 conmon[60316]: debug 2022-01-31T21:39:59.311+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:59.312320+0000) 2022-01-31T21:39:59.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:39:59 smithi167 conmon[49112]: debug 2022-01-31T21:39:59.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:59.648703+0000) 2022-01-31T21:39:59.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:39:59 smithi171 conmon[51620]: debug 2022-01-31T21:39:59.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:39:59.922049+0000) 2022-01-31T21:40:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:00 smithi171 conmon[46715]: debug 2022-01-31T21:40:00.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:00.084410+0000) 2022-01-31T21:40:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:00 smithi171 conmon[41853]: debug 2022-01-31T21:40:00.039+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:00.039941+0000) 2022-01-31T21:40:00.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:00 smithi167 conmon[54076]: debug 2022-01-31T21:40:00.311+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:00.312478+0000) 2022-01-31T21:40:00.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:00 smithi167 conmon[60316]: debug 2022-01-31T21:40:00.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:00.312538+0000) 2022-01-31T21:40:00.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:00 smithi167 conmon[49112]: debug 2022-01-31T21:40:00.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:00.648895+0000) 2022-01-31T21:40:00.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:00 smithi171 conmon[51620]: debug 2022-01-31T21:40:00.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:00.922219+0000) 2022-01-31T21:40:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:01 smithi171 conmon[46715]: debug 2022-01-31T21:40:01.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:01.084569+0000) 2022-01-31T21:40:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:01 smithi171 conmon[41853]: debug 2022-01-31T21:40:01.039+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:01.040125+0000) 2022-01-31T21:40:01.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:01 smithi167 conmon[54076]: debug 2022-01-31T21:40:01.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:01.312692+0000) 2022-01-31T21:40:01.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:01 smithi167 conmon[60316]: debug 2022-01-31T21:40:01.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:01.312759+0000) 2022-01-31T21:40:01.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:01 smithi167 conmon[49112]: debug 2022-01-31T21:40:01.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:01.649053+0000) 2022-01-31T21:40:01.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:01 smithi171 conmon[51620]: debug 2022-01-31T21:40:01.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:01.922378+0000) 2022-01-31T21:40:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[46715]: debug 2022-01-31T21:40:02.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.084718+0000) 2022-01-31T21:40:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[41853]: debug 2022-01-31T21:40:02.039+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.040329+0000) 2022-01-31T21:40:02.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:02 smithi167 conmon[54076]: debug 2022-01-31T21:40:02.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.312863+0000) 2022-01-31T21:40:02.551 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:02 smithi167 conmon[60316]: debug 2022-01-31T21:40:02.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.312944+0000) 2022-01-31T21:40:02.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:02 smithi167 conmon[54076]: debug 2022-01-31T21:40:02.657+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.657733+0000) 2022-01-31T21:40:02.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:02 smithi167 conmon[60316]: debug 2022-01-31T21:40:02.657+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.658171+0000) 2022-01-31T21:40:02.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:02 smithi167 conmon[49112]: debug 2022-01-31T21:40:02.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.649220+0000) 2022-01-31T21:40:02.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:02 smithi167 conmon[49112]: debug 2022-01-31T21:40:02.657+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.658028+0000) 2022-01-31T21:40:02.920 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[35325]: debug 2022-01-31T21:40:02.672+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 147899 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:02.921 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[41853]: debug 2022-01-31T21:40:02.657+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.657914+0000) 2022-01-31T21:40:02.921 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[46715]: debug 2022-01-31T21:40:02.657+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.658574+0000) 2022-01-31T21:40:02.922 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[51620]: debug 2022-01-31T21:40:02.656+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.657114+0000) 2022-01-31T21:40:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:03 smithi171 conmon[41853]: debug 2022-01-31T21:40:03.039+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:03.040503+0000) 2022-01-31T21:40:03.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:03 smithi171 conmon[46715]: debug 2022-01-31T21:40:03.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:03.084905+0000) 2022-01-31T21:40:03.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:02 smithi171 conmon[51620]: debug 2022-01-31T21:40:02.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:02.922522+0000) 2022-01-31T21:40:03.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:03 smithi167 conmon[54076]: debug 2022-01-31T21:40:03.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:03.313083+0000) 2022-01-31T21:40:03.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:03 smithi167 conmon[60316]: debug 2022-01-31T21:40:03.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:03.313108+0000) 2022-01-31T21:40:03.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:03 smithi167 conmon[49112]: debug 2022-01-31T21:40:03.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:03.649415+0000) 2022-01-31T21:40:03.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:03 smithi171 conmon[51620]: debug 2022-01-31T21:40:03.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:03.922684+0000) 2022-01-31T21:40:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:04 smithi171 conmon[41853]: debug 2022-01-31T21:40:04.039+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:04.040642+0000) 2022-01-31T21:40:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:04 smithi171 conmon[46715]: debug 2022-01-31T21:40:04.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:04.085065+0000) 2022-01-31T21:40:04.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:04 smithi167 conmon[54076]: debug 2022-01-31T21:40:04.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:04.313229+0000) 2022-01-31T21:40:04.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:04 smithi167 conmon[60316]: debug 2022-01-31T21:40:04.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:04.313263+0000) 2022-01-31T21:40:04.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:04 smithi167 conmon[49112]: debug 2022-01-31T21:40:04.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:04.649562+0000) 2022-01-31T21:40:04.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:04 smithi171 conmon[51620]: debug 2022-01-31T21:40:04.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:04.922891+0000) 2022-01-31T21:40:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:05 smithi171 conmon[46715]: debug 2022-01-31T21:40:05.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:05.085244+0000) 2022-01-31T21:40:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:05 smithi171 conmon[41853]: debug 2022-01-31T21:40:05.040+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:05.040846+0000) 2022-01-31T21:40:05.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:05 smithi167 conmon[54076]: debug 2022-01-31T21:40:05.312+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:05.313352+0000) 2022-01-31T21:40:05.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:05 smithi167 conmon[60316]: debug 2022-01-31T21:40:05.312+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:05.313371+0000) 2022-01-31T21:40:05.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:05 smithi167 conmon[49112]: debug 2022-01-31T21:40:05.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:05.649757+0000) 2022-01-31T21:40:05.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:05 smithi171 conmon[51620]: debug 2022-01-31T21:40:05.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:05.923068+0000) 2022-01-31T21:40:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:06 smithi171 conmon[41853]: debug 2022-01-31T21:40:06.040+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:06.041020+0000) 2022-01-31T21:40:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:06 smithi171 conmon[46715]: debug 2022-01-31T21:40:06.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:06.085421+0000) 2022-01-31T21:40:06.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:06 smithi167 conmon[54076]: debug 2022-01-31T21:40:06.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:06.313537+0000) 2022-01-31T21:40:06.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:06 smithi167 conmon[60316]: debug 2022-01-31T21:40:06.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:06.313493+0000) 2022-01-31T21:40:06.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:06 smithi167 conmon[49112]: debug 2022-01-31T21:40:06.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:06.649893+0000) 2022-01-31T21:40:06.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:06 smithi171 conmon[51620]: debug 2022-01-31T21:40:06.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:06.923260+0000) 2022-01-31T21:40:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[46715]: debug 2022-01-31T21:40:07.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.085584+0000) 2022-01-31T21:40:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[41853]: debug 2022-01-31T21:40:07.040+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.041203+0000) 2022-01-31T21:40:07.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:07 smithi167 conmon[54076]: debug 2022-01-31T21:40:07.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.313689+0000) 2022-01-31T21:40:07.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:07 smithi167 conmon[60316]: debug 2022-01-31T21:40:07.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.313689+0000) 2022-01-31T21:40:07.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:07 smithi167 conmon[49112]: debug 2022-01-31T21:40:07.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.650045+0000) 2022-01-31T21:40:07.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:07 smithi167 conmon[49112]: debug 2022-01-31T21:40:07.675+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.675694+0000) 2022-01-31T21:40:07.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:07 smithi167 conmon[54076]: debug 2022-01-31T21:40:07.674+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.675431+0000) 2022-01-31T21:40:07.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:07 smithi167 conmon[60316]: debug 2022-01-31T21:40:07.675+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.676049+0000) 2022-01-31T21:40:07.995 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[46715]: debug 2022-01-31T21:40:07.674+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.675154+0000) 2022-01-31T21:40:07.995 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[35325]: debug 2022-01-31T21:40:07.690+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148009 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:07.996 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[41853]: debug 2022-01-31T21:40:07.674+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.675255+0000) 2022-01-31T21:40:07.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[51620]: debug 2022-01-31T21:40:07.675+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.676322+0000) 2022-01-31T21:40:07.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:07 smithi171 conmon[51620]: debug 2022-01-31T21:40:07.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:07.923461+0000) 2022-01-31T21:40:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:08 smithi171 conmon[46715]: debug 2022-01-31T21:40:08.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:08.085785+0000) 2022-01-31T21:40:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:08 smithi171 conmon[41853]: debug 2022-01-31T21:40:08.040+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:08.041408+0000) 2022-01-31T21:40:08.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:08 smithi167 conmon[54076]: debug 2022-01-31T21:40:08.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:08.313904+0000) 2022-01-31T21:40:08.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:08 smithi167 conmon[60316]: debug 2022-01-31T21:40:08.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:08.313906+0000) 2022-01-31T21:40:08.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:08 smithi167 conmon[49112]: debug 2022-01-31T21:40:08.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:08.650247+0000) 2022-01-31T21:40:08.995 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:08 smithi171 conmon[35325]: debug 2022-01-31T21:40:08.717+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:40:08.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:08 smithi171 conmon[51620]: debug 2022-01-31T21:40:08.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:08.923662+0000) 2022-01-31T21:40:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:09 smithi171 conmon[41853]: debug 2022-01-31T21:40:09.040+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:09.041591+0000) 2022-01-31T21:40:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:09 smithi171 conmon[46715]: debug 2022-01-31T21:40:09.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:09.085962+0000) 2022-01-31T21:40:09.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:09 smithi167 conmon[54076]: debug 2022-01-31T21:40:09.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:09.314082+0000) 2022-01-31T21:40:09.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:09 smithi167 conmon[60316]: debug 2022-01-31T21:40:09.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:09.314071+0000) 2022-01-31T21:40:09.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:09 smithi167 conmon[49112]: debug 2022-01-31T21:40:09.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:09.650385+0000) 2022-01-31T21:40:09.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:09 smithi171 conmon[51620]: debug 2022-01-31T21:40:09.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:09.923870+0000) 2022-01-31T21:40:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:10 smithi171 conmon[41853]: debug 2022-01-31T21:40:10.041+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:10.041720+0000) 2022-01-31T21:40:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:10 smithi171 conmon[46715]: debug 2022-01-31T21:40:10.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:10.086133+0000) 2022-01-31T21:40:10.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:10 smithi167 conmon[54076]: debug 2022-01-31T21:40:10.313+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:10.314262+0000) 2022-01-31T21:40:10.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:10 smithi167 conmon[60316]: debug 2022-01-31T21:40:10.313+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:10.314259+0000) 2022-01-31T21:40:10.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:10 smithi167 conmon[49112]: debug 2022-01-31T21:40:10.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:10.650582+0000) 2022-01-31T21:40:10.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:10 smithi171 conmon[51620]: debug 2022-01-31T21:40:10.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:10.924073+0000) 2022-01-31T21:40:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:11 smithi171 conmon[41853]: debug 2022-01-31T21:40:11.041+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:11.041840+0000) 2022-01-31T21:40:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:11 smithi171 conmon[46715]: debug 2022-01-31T21:40:11.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:11.086283+0000) 2022-01-31T21:40:11.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:11 smithi167 conmon[54076]: debug 2022-01-31T21:40:11.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:11.314469+0000) 2022-01-31T21:40:11.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:11 smithi167 conmon[60316]: debug 2022-01-31T21:40:11.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:11.314477+0000) 2022-01-31T21:40:11.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:11 smithi167 conmon[49112]: debug 2022-01-31T21:40:11.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:11.650726+0000) 2022-01-31T21:40:11.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:11 smithi171 conmon[51620]: debug 2022-01-31T21:40:11.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:11.924248+0000) 2022-01-31T21:40:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[46715]: debug 2022-01-31T21:40:12.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.086479+0000) 2022-01-31T21:40:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[41853]: debug 2022-01-31T21:40:12.041+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.042041+0000) 2022-01-31T21:40:12.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:12 smithi167 conmon[54076]: debug 2022-01-31T21:40:12.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.314709+0000) 2022-01-31T21:40:12.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:12 smithi167 conmon[60316]: debug 2022-01-31T21:40:12.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.314713+0000) 2022-01-31T21:40:12.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:12 smithi167 conmon[49112]: debug 2022-01-31T21:40:12.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.650911+0000) 2022-01-31T21:40:12.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:12 smithi167 conmon[49112]: debug 2022-01-31T21:40:12.693+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.693828+0000) 2022-01-31T21:40:12.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:12 smithi167 conmon[54076]: debug 2022-01-31T21:40:12.710+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.711457+0000) 2022-01-31T21:40:12.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:12 smithi167 conmon[60316]: debug 2022-01-31T21:40:12.693+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.694233+0000) 2022-01-31T21:40:12.996 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[41853]: debug 2022-01-31T21:40:12.692+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.693333+0000) 2022-01-31T21:40:12.997 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[46715]: debug 2022-01-31T21:40:12.692+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.693569+0000) 2022-01-31T21:40:12.998 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[35325]: debug 2022-01-31T21:40:12.714+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148120 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:12.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[51620]: debug 2022-01-31T21:40:12.694+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.694768+0000) 2022-01-31T21:40:12.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:12 smithi171 conmon[51620]: debug 2022-01-31T21:40:12.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:12.924391+0000) 2022-01-31T21:40:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:13 smithi171 conmon[41853]: debug 2022-01-31T21:40:13.041+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:13.042253+0000) 2022-01-31T21:40:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:13 smithi171 conmon[46715]: debug 2022-01-31T21:40:13.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:13.086646+0000) 2022-01-31T21:40:13.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:13 smithi167 conmon[54076]: debug 2022-01-31T21:40:13.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:13.314869+0000) 2022-01-31T21:40:13.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:13 smithi167 conmon[60316]: debug 2022-01-31T21:40:13.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:13.314896+0000) 2022-01-31T21:40:13.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:13 smithi167 conmon[49112]: debug 2022-01-31T21:40:13.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:13.651074+0000) 2022-01-31T21:40:13.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:13 smithi171 conmon[51620]: debug 2022-01-31T21:40:13.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:13.924566+0000) 2022-01-31T21:40:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:14 smithi171 conmon[46715]: debug 2022-01-31T21:40:14.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:14.086767+0000) 2022-01-31T21:40:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:14 smithi171 conmon[41853]: debug 2022-01-31T21:40:14.041+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:14.042405+0000) 2022-01-31T21:40:14.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:14 smithi167 conmon[54076]: debug 2022-01-31T21:40:14.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:14.314999+0000) 2022-01-31T21:40:14.553 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:14 smithi167 conmon[60316]: debug 2022-01-31T21:40:14.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:14.315027+0000) 2022-01-31T21:40:14.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:14 smithi167 conmon[49112]: debug 2022-01-31T21:40:14.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:14.651213+0000) 2022-01-31T21:40:14.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:14 smithi171 conmon[51620]: debug 2022-01-31T21:40:14.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:14.924789+0000) 2022-01-31T21:40:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:15 smithi171 conmon[41853]: debug 2022-01-31T21:40:15.042+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:15.042601+0000) 2022-01-31T21:40:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:15 smithi171 conmon[46715]: debug 2022-01-31T21:40:15.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:15.086940+0000) 2022-01-31T21:40:15.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:15 smithi167 conmon[54076]: debug 2022-01-31T21:40:15.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:15.315183+0000) 2022-01-31T21:40:15.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:15 smithi167 conmon[60316]: debug 2022-01-31T21:40:15.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:15.315207+0000) 2022-01-31T21:40:15.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:15 smithi167 conmon[49112]: debug 2022-01-31T21:40:15.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:15.651372+0000) 2022-01-31T21:40:15.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:15 smithi171 conmon[51620]: debug 2022-01-31T21:40:15.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:15.924938+0000) 2022-01-31T21:40:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:16 smithi171 conmon[41853]: debug 2022-01-31T21:40:16.042+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:16.042792+0000) 2022-01-31T21:40:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:16 smithi171 conmon[46715]: debug 2022-01-31T21:40:16.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:16.087112+0000) 2022-01-31T21:40:16.369 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:16 smithi167 conmon[54076]: debug 2022-01-31T21:40:16.314+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:16.315387+0000) 2022-01-31T21:40:16.370 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:16 smithi167 conmon[60316]: debug 2022-01-31T21:40:16.314+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:16.315387+0000) 2022-01-31T21:40:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:16 smithi167 conmon[49112]: debug 2022-01-31T21:40:16.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:16.651562+0000) 2022-01-31T21:40:16.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:16 smithi171 conmon[51620]: debug 2022-01-31T21:40:16.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:16.925143+0000) 2022-01-31T21:40:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[41853]: debug 2022-01-31T21:40:17.042+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.042975+0000) 2022-01-31T21:40:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[46715]: debug 2022-01-31T21:40:17.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.087296+0000) 2022-01-31T21:40:17.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:17 smithi167 conmon[54076]: debug 2022-01-31T21:40:17.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.315579+0000) 2022-01-31T21:40:17.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:17 smithi167 conmon[60316]: debug 2022-01-31T21:40:17.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.315604+0000) 2022-01-31T21:40:17.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:17 smithi167 conmon[49112]: debug 2022-01-31T21:40:17.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.651712+0000) 2022-01-31T21:40:17.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:17 smithi167 conmon[49112]: debug 2022-01-31T21:40:17.716+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.716860+0000) 2022-01-31T21:40:17.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:17 smithi167 conmon[54076]: debug 2022-01-31T21:40:17.717+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.718008+0000) 2022-01-31T21:40:17.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:17 smithi167 conmon[60316]: debug 2022-01-31T21:40:17.717+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.717853+0000) 2022-01-31T21:40:17.996 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[35325]: debug 2022-01-31T21:40:17.733+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148230 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:17.997 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[41853]: debug 2022-01-31T21:40:17.716+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.717558+0000) 2022-01-31T21:40:17.998 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[46715]: debug 2022-01-31T21:40:17.716+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.717478+0000) 2022-01-31T21:40:17.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[51620]: debug 2022-01-31T21:40:17.717+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.718246+0000) 2022-01-31T21:40:17.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:17 smithi171 conmon[51620]: debug 2022-01-31T21:40:17.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:17.925342+0000) 2022-01-31T21:40:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:18 smithi171 conmon[46715]: debug 2022-01-31T21:40:18.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:18.087491+0000) 2022-01-31T21:40:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:18 smithi171 conmon[41853]: debug 2022-01-31T21:40:18.042+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:18.043161+0000) 2022-01-31T21:40:18.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:18 smithi167 conmon[54076]: debug 2022-01-31T21:40:18.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:18.315772+0000) 2022-01-31T21:40:18.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:18 smithi167 conmon[60316]: debug 2022-01-31T21:40:18.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:18.315807+0000) 2022-01-31T21:40:18.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:18 smithi167 conmon[49112]: debug 2022-01-31T21:40:18.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:18.651894+0000) 2022-01-31T21:40:18.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:18 smithi171 conmon[51620]: debug 2022-01-31T21:40:18.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:18.925516+0000) 2022-01-31T21:40:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:19 smithi171 conmon[46715]: debug 2022-01-31T21:40:19.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:19.087665+0000) 2022-01-31T21:40:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:19 smithi171 conmon[41853]: debug 2022-01-31T21:40:19.042+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:19.043295+0000) 2022-01-31T21:40:19.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:19 smithi167 conmon[54076]: debug 2022-01-31T21:40:19.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:19.315927+0000) 2022-01-31T21:40:19.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:19 smithi167 conmon[60316]: debug 2022-01-31T21:40:19.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:19.315958+0000) 2022-01-31T21:40:19.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:19 smithi167 conmon[49112]: debug 2022-01-31T21:40:19.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:19.652014+0000) 2022-01-31T21:40:19.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:19 smithi171 conmon[51620]: debug 2022-01-31T21:40:19.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:19.925696+0000) 2022-01-31T21:40:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:20 smithi171 conmon[46715]: debug 2022-01-31T21:40:20.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:20.087878+0000) 2022-01-31T21:40:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:20 smithi171 conmon[41853]: debug 2022-01-31T21:40:20.043+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:20.043481+0000) 2022-01-31T21:40:20.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:20 smithi167 conmon[54076]: debug 2022-01-31T21:40:20.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:20.316153+0000) 2022-01-31T21:40:20.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:20 smithi167 conmon[60316]: debug 2022-01-31T21:40:20.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:20.316153+0000) 2022-01-31T21:40:20.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:20 smithi167 conmon[49112]: debug 2022-01-31T21:40:20.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:20.652228+0000) 2022-01-31T21:40:20.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:20 smithi171 conmon[51620]: debug 2022-01-31T21:40:20.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:20.925778+0000) 2022-01-31T21:40:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:21 smithi171 conmon[46715]: debug 2022-01-31T21:40:21.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:21.088052+0000) 2022-01-31T21:40:21.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:21 smithi171 conmon[41853]: debug 2022-01-31T21:40:21.043+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:21.043663+0000) 2022-01-31T21:40:21.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:21 smithi167 conmon[54076]: debug 2022-01-31T21:40:21.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:21.316325+0000) 2022-01-31T21:40:21.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:21 smithi167 conmon[60316]: debug 2022-01-31T21:40:21.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:21.316324+0000) 2022-01-31T21:40:21.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:21 smithi167 conmon[49112]: debug 2022-01-31T21:40:21.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:21.652423+0000) 2022-01-31T21:40:22.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:21 smithi171 conmon[51620]: debug 2022-01-31T21:40:21.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:21.925932+0000) 2022-01-31T21:40:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[46715]: debug 2022-01-31T21:40:22.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.088213+0000) 2022-01-31T21:40:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[41853]: debug 2022-01-31T21:40:22.043+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.043860+0000) 2022-01-31T21:40:22.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:22 smithi167 conmon[54076]: debug 2022-01-31T21:40:22.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.316518+0000) 2022-01-31T21:40:22.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:22 smithi167 conmon[60316]: debug 2022-01-31T21:40:22.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.316530+0000) 2022-01-31T21:40:22.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:22 smithi167 conmon[54076]: debug 2022-01-31T21:40:22.736+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.737240+0000) 2022-01-31T21:40:22.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:22 smithi167 conmon[60316]: debug 2022-01-31T21:40:22.735+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.735547+0000) 2022-01-31T21:40:22.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:22 smithi167 conmon[49112]: debug 2022-01-31T21:40:22.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.652658+0000) 2022-01-31T21:40:22.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:22 smithi167 conmon[49112]: debug 2022-01-31T21:40:22.736+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.736647+0000) 2022-01-31T21:40:22.997 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[35325]: debug 2022-01-31T21:40:22.751+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148341 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:22.998 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[41853]: debug 2022-01-31T21:40:22.735+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.736439+0000) 2022-01-31T21:40:22.999 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[46715]: debug 2022-01-31T21:40:22.736+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.737181+0000) 2022-01-31T21:40:22.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[51620]: debug 2022-01-31T21:40:22.737+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.737781+0000) 2022-01-31T21:40:22.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:22 smithi171 conmon[51620]: debug 2022-01-31T21:40:22.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:22.926116+0000) 2022-01-31T21:40:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:23 smithi171 conmon[41853]: debug 2022-01-31T21:40:23.043+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:23.044073+0000) 2022-01-31T21:40:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:23 smithi171 conmon[46715]: debug 2022-01-31T21:40:23.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:23.088377+0000) 2022-01-31T21:40:23.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:23 smithi167 conmon[54076]: debug 2022-01-31T21:40:23.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:23.316694+0000) 2022-01-31T21:40:23.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:23 smithi167 conmon[60316]: debug 2022-01-31T21:40:23.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:23.316730+0000) 2022-01-31T21:40:23.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:23 smithi167 conmon[49112]: debug 2022-01-31T21:40:23.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:23.652893+0000) 2022-01-31T21:40:23.997 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:23 smithi171 conmon[35325]: debug 2022-01-31T21:40:23.718+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:40:23.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:23 smithi171 conmon[51620]: debug 2022-01-31T21:40:23.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:23.926284+0000) 2022-01-31T21:40:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:24 smithi171 conmon[41853]: debug 2022-01-31T21:40:24.043+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:24.044242+0000) 2022-01-31T21:40:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:24 smithi171 conmon[46715]: debug 2022-01-31T21:40:24.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:24.088524+0000) 2022-01-31T21:40:24.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:24 smithi167 conmon[54076]: debug 2022-01-31T21:40:24.315+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:24.316865+0000) 2022-01-31T21:40:24.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:24 smithi167 conmon[60316]: debug 2022-01-31T21:40:24.315+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:24.316871+0000) 2022-01-31T21:40:24.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:24 smithi167 conmon[49112]: debug 2022-01-31T21:40:24.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:24.653038+0000) 2022-01-31T21:40:24.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:24 smithi171 conmon[51620]: debug 2022-01-31T21:40:24.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:24.926405+0000) 2022-01-31T21:40:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:25 smithi171 conmon[46715]: debug 2022-01-31T21:40:25.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:25.088704+0000) 2022-01-31T21:40:25.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:25 smithi171 conmon[41853]: debug 2022-01-31T21:40:25.044+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:25.044396+0000) 2022-01-31T21:40:25.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:25 smithi167 conmon[54076]: debug 2022-01-31T21:40:25.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:25.317076+0000) 2022-01-31T21:40:25.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:25 smithi167 conmon[60316]: debug 2022-01-31T21:40:25.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:25.317076+0000) 2022-01-31T21:40:25.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:25 smithi167 conmon[49112]: debug 2022-01-31T21:40:25.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:25.653214+0000) 2022-01-31T21:40:25.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:25 smithi171 conmon[51620]: debug 2022-01-31T21:40:25.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:25.926568+0000) 2022-01-31T21:40:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:26 smithi171 conmon[46715]: debug 2022-01-31T21:40:26.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:26.088894+0000) 2022-01-31T21:40:26.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:26 smithi171 conmon[41853]: debug 2022-01-31T21:40:26.044+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:26.044560+0000) 2022-01-31T21:40:26.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:26 smithi167 conmon[54076]: debug 2022-01-31T21:40:26.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:26.317271+0000) 2022-01-31T21:40:26.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:26 smithi167 conmon[60316]: debug 2022-01-31T21:40:26.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:26.317271+0000) 2022-01-31T21:40:26.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:26 smithi167 conmon[49112]: debug 2022-01-31T21:40:26.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:26.653399+0000) 2022-01-31T21:40:26.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:26 smithi171 conmon[51620]: debug 2022-01-31T21:40:26.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:26.926770+0000) 2022-01-31T21:40:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[46715]: debug 2022-01-31T21:40:27.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.089080+0000) 2022-01-31T21:40:27.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[41853]: debug 2022-01-31T21:40:27.044+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.044704+0000) 2022-01-31T21:40:27.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:27 smithi167 conmon[54076]: debug 2022-01-31T21:40:27.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.317455+0000) 2022-01-31T21:40:27.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:27 smithi167 conmon[60316]: debug 2022-01-31T21:40:27.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.317453+0000) 2022-01-31T21:40:27.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:27 smithi167 conmon[54076]: debug 2022-01-31T21:40:27.754+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.755942+0000) 2022-01-31T21:40:27.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:27 smithi167 conmon[60316]: debug 2022-01-31T21:40:27.752+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.753859+0000) 2022-01-31T21:40:27.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:27 smithi167 conmon[49112]: debug 2022-01-31T21:40:27.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.653542+0000) 2022-01-31T21:40:27.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:27 smithi167 conmon[49112]: debug 2022-01-31T21:40:27.753+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.755336+0000) 2022-01-31T21:40:27.924 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[35325]: debug 2022-01-31T21:40:27.770+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148465 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:27.925 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[41853]: debug 2022-01-31T21:40:27.753+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.754682+0000) 2022-01-31T21:40:27.926 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[46715]: debug 2022-01-31T21:40:27.755+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.756257+0000) 2022-01-31T21:40:27.927 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[51620]: debug 2022-01-31T21:40:27.755+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.756098+0000) 2022-01-31T21:40:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:28 smithi171 conmon[41853]: debug 2022-01-31T21:40:28.044+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:28.044890+0000) 2022-01-31T21:40:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:28 smithi171 conmon[46715]: debug 2022-01-31T21:40:28.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:28.089230+0000) 2022-01-31T21:40:28.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:27 smithi171 conmon[51620]: debug 2022-01-31T21:40:27.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:27.926901+0000) 2022-01-31T21:40:28.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:28 smithi167 conmon[54076]: debug 2022-01-31T21:40:28.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:28.317669+0000) 2022-01-31T21:40:28.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:28 smithi167 conmon[60316]: debug 2022-01-31T21:40:28.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:28.317669+0000) 2022-01-31T21:40:28.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:28 smithi167 conmon[49112]: debug 2022-01-31T21:40:28.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:28.653703+0000) 2022-01-31T21:40:28.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:28 smithi171 conmon[51620]: debug 2022-01-31T21:40:28.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:28.927067+0000) 2022-01-31T21:40:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:29 smithi171 conmon[41853]: debug 2022-01-31T21:40:29.044+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:29.045033+0000) 2022-01-31T21:40:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:29 smithi171 conmon[46715]: debug 2022-01-31T21:40:29.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:29.089379+0000) 2022-01-31T21:40:29.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:29 smithi167 conmon[54076]: debug 2022-01-31T21:40:29.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:29.317833+0000) 2022-01-31T21:40:29.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:29 smithi167 conmon[60316]: debug 2022-01-31T21:40:29.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:29.317837+0000) 2022-01-31T21:40:29.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:29 smithi167 conmon[49112]: debug 2022-01-31T21:40:29.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:29.653850+0000) 2022-01-31T21:40:29.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:29 smithi171 conmon[51620]: debug 2022-01-31T21:40:29.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:29.927246+0000) 2022-01-31T21:40:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:30 smithi171 conmon[41853]: debug 2022-01-31T21:40:30.044+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:30.045194+0000) 2022-01-31T21:40:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:30 smithi171 conmon[46715]: debug 2022-01-31T21:40:30.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:30.089568+0000) 2022-01-31T21:40:30.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:30 smithi167 conmon[60316]: debug 2022-01-31T21:40:30.317+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:30.318030+0000) 2022-01-31T21:40:30.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:30 smithi167 conmon[54076]: debug 2022-01-31T21:40:30.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:30.318040+0000) 2022-01-31T21:40:30.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:30 smithi167 conmon[49112]: debug 2022-01-31T21:40:30.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:30.654064+0000) 2022-01-31T21:40:30.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:30 smithi171 conmon[51620]: debug 2022-01-31T21:40:30.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:30.927414+0000) 2022-01-31T21:40:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:31 smithi171 conmon[46715]: debug 2022-01-31T21:40:31.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:31.089807+0000) 2022-01-31T21:40:31.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:31 smithi171 conmon[41853]: debug 2022-01-31T21:40:31.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:31.045405+0000) 2022-01-31T21:40:31.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:31 smithi167 conmon[60316]: debug 2022-01-31T21:40:31.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:31.318179+0000) 2022-01-31T21:40:31.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:31 smithi167 conmon[54076]: debug 2022-01-31T21:40:31.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:31.318200+0000) 2022-01-31T21:40:31.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:31 smithi167 conmon[49112]: debug 2022-01-31T21:40:31.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:31.654144+0000) 2022-01-31T21:40:31.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:31 smithi171 conmon[51620]: debug 2022-01-31T21:40:31.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:31.927635+0000) 2022-01-31T21:40:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[46715]: debug 2022-01-31T21:40:32.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.089989+0000) 2022-01-31T21:40:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[41853]: debug 2022-01-31T21:40:32.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.045554+0000) 2022-01-31T21:40:32.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:32 smithi167 conmon[54076]: debug 2022-01-31T21:40:32.316+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.318382+0000) 2022-01-31T21:40:32.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:32 smithi167 conmon[60316]: debug 2022-01-31T21:40:32.316+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.318384+0000) 2022-01-31T21:40:32.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:32 smithi167 conmon[54076]: debug 2022-01-31T21:40:32.771+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.773480+0000) 2022-01-31T21:40:32.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:32 smithi167 conmon[60316]: debug 2022-01-31T21:40:32.770+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.772316+0000) 2022-01-31T21:40:32.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:32 smithi167 conmon[49112]: debug 2022-01-31T21:40:32.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.654306+0000) 2022-01-31T21:40:32.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:32 smithi167 conmon[49112]: debug 2022-01-31T21:40:32.772+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.773553+0000) 2022-01-31T21:40:32.999 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[41853]: debug 2022-01-31T21:40:32.772+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.773152+0000) 2022-01-31T21:40:33.000 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[46715]: debug 2022-01-31T21:40:32.773+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.773859+0000) 2022-01-31T21:40:33.001 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[35325]: debug 2022-01-31T21:40:32.788+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148575 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:33.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[51620]: debug 2022-01-31T21:40:32.773+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.774482+0000) 2022-01-31T21:40:33.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:32 smithi171 conmon[51620]: debug 2022-01-31T21:40:32.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:32.927827+0000) 2022-01-31T21:40:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:33 smithi171 conmon[41853]: debug 2022-01-31T21:40:33.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:33.045729+0000) 2022-01-31T21:40:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:33 smithi171 conmon[46715]: debug 2022-01-31T21:40:33.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:33.090188+0000) 2022-01-31T21:40:33.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:33 smithi167 conmon[54076]: debug 2022-01-31T21:40:33.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:33.318535+0000) 2022-01-31T21:40:33.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:33 smithi167 conmon[60316]: debug 2022-01-31T21:40:33.317+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:33.318552+0000) 2022-01-31T21:40:33.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:33 smithi167 conmon[49112]: debug 2022-01-31T21:40:33.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:33.654530+0000) 2022-01-31T21:40:33.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:33 smithi171 conmon[51620]: debug 2022-01-31T21:40:33.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:33.927947+0000) 2022-01-31T21:40:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:34 smithi171 conmon[46715]: debug 2022-01-31T21:40:34.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:34.090364+0000) 2022-01-31T21:40:34.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:34 smithi171 conmon[41853]: debug 2022-01-31T21:40:34.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:34.045869+0000) 2022-01-31T21:40:34.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:34 smithi167 conmon[54076]: debug 2022-01-31T21:40:34.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:34.318700+0000) 2022-01-31T21:40:34.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:34 smithi167 conmon[60316]: debug 2022-01-31T21:40:34.317+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:34.318698+0000) 2022-01-31T21:40:34.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:34 smithi167 conmon[49112]: debug 2022-01-31T21:40:34.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:34.654687+0000) 2022-01-31T21:40:34.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:34 smithi171 conmon[51620]: debug 2022-01-31T21:40:34.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:34.928096+0000) 2022-01-31T21:40:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:35 smithi171 conmon[41853]: debug 2022-01-31T21:40:35.046+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:35.046070+0000) 2022-01-31T21:40:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:35 smithi171 conmon[46715]: debug 2022-01-31T21:40:35.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:35.090562+0000) 2022-01-31T21:40:35.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:35 smithi167 conmon[54076]: debug 2022-01-31T21:40:35.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:35.318924+0000) 2022-01-31T21:40:35.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:35 smithi167 conmon[60316]: debug 2022-01-31T21:40:35.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:35.318901+0000) 2022-01-31T21:40:35.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:35 smithi167 conmon[49112]: debug 2022-01-31T21:40:35.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:35.654908+0000) 2022-01-31T21:40:36.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:35 smithi171 conmon[51620]: debug 2022-01-31T21:40:35.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:35.928288+0000) 2022-01-31T21:40:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:36 smithi171 conmon[41853]: debug 2022-01-31T21:40:36.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:36.046226+0000) 2022-01-31T21:40:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:36 smithi171 conmon[46715]: debug 2022-01-31T21:40:36.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:36.090743+0000) 2022-01-31T21:40:36.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:36 smithi167 conmon[54076]: debug 2022-01-31T21:40:36.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:36.319083+0000) 2022-01-31T21:40:36.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:36 smithi167 conmon[60316]: debug 2022-01-31T21:40:36.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:36.319081+0000) 2022-01-31T21:40:36.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:36 smithi167 conmon[49112]: debug 2022-01-31T21:40:36.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:36.655067+0000) 2022-01-31T21:40:37.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:36 smithi171 conmon[51620]: debug 2022-01-31T21:40:36.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:36.928444+0000) 2022-01-31T21:40:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[46715]: debug 2022-01-31T21:40:37.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.090907+0000) 2022-01-31T21:40:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[41853]: debug 2022-01-31T21:40:37.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.046382+0000) 2022-01-31T21:40:37.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:37 smithi167 conmon[54076]: debug 2022-01-31T21:40:37.317+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.319333+0000) 2022-01-31T21:40:37.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:37 smithi167 conmon[60316]: debug 2022-01-31T21:40:37.317+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.319331+0000) 2022-01-31T21:40:37.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:37 smithi167 conmon[54076]: debug 2022-01-31T21:40:37.789+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.790789+0000) 2022-01-31T21:40:37.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:37 smithi167 conmon[60316]: debug 2022-01-31T21:40:37.790+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.792202+0000) 2022-01-31T21:40:37.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:37 smithi167 conmon[49112]: debug 2022-01-31T21:40:37.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.655227+0000) 2022-01-31T21:40:37.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:37 smithi167 conmon[49112]: debug 2022-01-31T21:40:37.790+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.792044+0000) 2022-01-31T21:40:38.000 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[35325]: debug 2022-01-31T21:40:37.806+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148686 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:38.001 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[41853]: debug 2022-01-31T21:40:37.789+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.791365+0000) 2022-01-31T21:40:38.001 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[46715]: debug 2022-01-31T21:40:37.791+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.792887+0000) 2022-01-31T21:40:38.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[51620]: debug 2022-01-31T21:40:37.791+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.793234+0000) 2022-01-31T21:40:38.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:37 smithi171 conmon[51620]: debug 2022-01-31T21:40:37.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:37.928602+0000) 2022-01-31T21:40:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:38 smithi171 conmon[41853]: debug 2022-01-31T21:40:38.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:38.046609+0000) 2022-01-31T21:40:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:38 smithi171 conmon[46715]: debug 2022-01-31T21:40:38.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:38.091111+0000) 2022-01-31T21:40:38.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:38 smithi167 conmon[54076]: debug 2022-01-31T21:40:38.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:38.319541+0000) 2022-01-31T21:40:38.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:38 smithi167 conmon[60316]: debug 2022-01-31T21:40:38.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:38.319539+0000) 2022-01-31T21:40:38.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:38 smithi167 conmon[49112]: debug 2022-01-31T21:40:38.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:38.655442+0000) 2022-01-31T21:40:39.000 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:38 smithi171 conmon[35325]: debug 2022-01-31T21:40:38.718+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:40:39.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:38 smithi171 conmon[51620]: debug 2022-01-31T21:40:38.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:38.928799+0000) 2022-01-31T21:40:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:39 smithi171 conmon[41853]: debug 2022-01-31T21:40:39.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:39.046789+0000) 2022-01-31T21:40:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:39 smithi171 conmon[46715]: debug 2022-01-31T21:40:39.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:39.091235+0000) 2022-01-31T21:40:39.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:39 smithi167 conmon[54076]: debug 2022-01-31T21:40:39.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:39.319680+0000) 2022-01-31T21:40:39.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:39 smithi167 conmon[60316]: debug 2022-01-31T21:40:39.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:39.319646+0000) 2022-01-31T21:40:39.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:39 smithi167 conmon[49112]: debug 2022-01-31T21:40:39.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:39.655520+0000) 2022-01-31T21:40:40.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:39 smithi171 conmon[51620]: debug 2022-01-31T21:40:39.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:39.928965+0000) 2022-01-31T21:40:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:40 smithi171 conmon[46715]: debug 2022-01-31T21:40:40.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:40.091402+0000) 2022-01-31T21:40:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:40 smithi171 conmon[41853]: debug 2022-01-31T21:40:40.046+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:40.046987+0000) 2022-01-31T21:40:40.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:40 smithi167 conmon[54076]: debug 2022-01-31T21:40:40.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:40.319887+0000) 2022-01-31T21:40:40.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:40 smithi167 conmon[60316]: debug 2022-01-31T21:40:40.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:40.319840+0000) 2022-01-31T21:40:40.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:40 smithi167 conmon[49112]: debug 2022-01-31T21:40:40.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:40.655798+0000) 2022-01-31T21:40:41.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:40 smithi171 conmon[51620]: debug 2022-01-31T21:40:40.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:40.929121+0000) 2022-01-31T21:40:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:41 smithi171 conmon[41853]: debug 2022-01-31T21:40:41.046+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:41.047142+0000) 2022-01-31T21:40:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:41 smithi171 conmon[46715]: debug 2022-01-31T21:40:41.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:41.091554+0000) 2022-01-31T21:40:41.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:41 smithi167 conmon[54076]: debug 2022-01-31T21:40:41.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:41.320085+0000) 2022-01-31T21:40:41.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:41 smithi167 conmon[60316]: debug 2022-01-31T21:40:41.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:41.320067+0000) 2022-01-31T21:40:41.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:41 smithi167 conmon[49112]: debug 2022-01-31T21:40:41.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:41.655927+0000) 2022-01-31T21:40:42.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:41 smithi171 conmon[51620]: debug 2022-01-31T21:40:41.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:41.929321+0000) 2022-01-31T21:40:42.303 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[41853]: debug 2022-01-31T21:40:42.045+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.047301+0000) 2022-01-31T21:40:42.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[46715]: debug 2022-01-31T21:40:42.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.091737+0000) 2022-01-31T21:40:42.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:42 smithi167 conmon[54076]: debug 2022-01-31T21:40:42.318+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.320281+0000) 2022-01-31T21:40:42.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:42 smithi167 conmon[60316]: debug 2022-01-31T21:40:42.318+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.320279+0000) 2022-01-31T21:40:42.808 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:42 smithi167 conmon[49112]: debug 2022-01-31T21:40:42.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.656118+0000) 2022-01-31T21:40:42.809 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:42 smithi167 conmon[60316]: debug 2022-01-31T21:40:42.808+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.809722+0000) 2022-01-31T21:40:43.001 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[35325]: debug 2022-01-31T21:40:42.825+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148797 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:43.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[41853]: debug 2022-01-31T21:40:42.809+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.811155+0000) 2022-01-31T21:40:43.002 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[46715]: debug 2022-01-31T21:40:42.809+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.810861+0000) 2022-01-31T21:40:43.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[51620]: debug 2022-01-31T21:40:42.809+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.811013+0000) 2022-01-31T21:40:43.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:42 smithi171 conmon[51620]: debug 2022-01-31T21:40:42.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.929531+0000) 2022-01-31T21:40:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:42 smithi167 conmon[49112]: debug 2022-01-31T21:40:42.808+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.810230+0000) 2022-01-31T21:40:43.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:42 smithi167 conmon[54076]: debug 2022-01-31T21:40:42.808+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:42.810386+0000) 2022-01-31T21:40:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:43 smithi171 conmon[41853]: debug 2022-01-31T21:40:43.046+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:43.047527+0000) 2022-01-31T21:40:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:43 smithi171 conmon[46715]: debug 2022-01-31T21:40:43.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:43.091954+0000) 2022-01-31T21:40:43.558 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:43 smithi167 conmon[54076]: debug 2022-01-31T21:40:43.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:43.320423+0000) 2022-01-31T21:40:43.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:43 smithi167 conmon[60316]: debug 2022-01-31T21:40:43.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:43.320436+0000) 2022-01-31T21:40:43.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:43 smithi167 conmon[49112]: debug 2022-01-31T21:40:43.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:43.656333+0000) 2022-01-31T21:40:44.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:43 smithi171 conmon[51620]: debug 2022-01-31T21:40:43.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:43.929732+0000) 2022-01-31T21:40:44.351 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:44 smithi171 conmon[41853]: debug 2022-01-31T21:40:44.046+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:44.047701+0000) 2022-01-31T21:40:44.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:44 smithi171 conmon[46715]: debug 2022-01-31T21:40:44.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:44.092063+0000) 2022-01-31T21:40:44.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:44 smithi167 conmon[60316]: debug 2022-01-31T21:40:44.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:44.320627+0000) 2022-01-31T21:40:44.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:44 smithi167 conmon[54076]: debug 2022-01-31T21:40:44.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:44.320624+0000) 2022-01-31T21:40:44.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:44 smithi167 conmon[49112]: debug 2022-01-31T21:40:44.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:44.656488+0000) 2022-01-31T21:40:45.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:44 smithi171 conmon[51620]: debug 2022-01-31T21:40:44.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:44.929927+0000) 2022-01-31T21:40:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:45 smithi171 conmon[41853]: debug 2022-01-31T21:40:45.047+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:45.047838+0000) 2022-01-31T21:40:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:45 smithi171 conmon[46715]: debug 2022-01-31T21:40:45.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:45.092216+0000) 2022-01-31T21:40:45.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:45 smithi167 conmon[60316]: debug 2022-01-31T21:40:45.319+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:45.320784+0000) 2022-01-31T21:40:45.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:45 smithi167 conmon[54076]: debug 2022-01-31T21:40:45.319+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:45.320806+0000) 2022-01-31T21:40:45.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:45 smithi167 conmon[49112]: debug 2022-01-31T21:40:45.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:45.656681+0000) 2022-01-31T21:40:46.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:45 smithi171 conmon[51620]: debug 2022-01-31T21:40:45.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:45.930125+0000) 2022-01-31T21:40:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:46 smithi171 conmon[41853]: debug 2022-01-31T21:40:46.047+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:46.047937+0000) 2022-01-31T21:40:46.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:46 smithi171 conmon[46715]: debug 2022-01-31T21:40:46.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:46.092398+0000) 2022-01-31T21:40:46.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:46 smithi167 conmon[54076]: debug 2022-01-31T21:40:46.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:46.320984+0000) 2022-01-31T21:40:46.377 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:46 smithi167 conmon[60316]: debug 2022-01-31T21:40:46.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:46.320963+0000) 2022-01-31T21:40:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:46 smithi167 conmon[49112]: debug 2022-01-31T21:40:46.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:46.656816+0000) 2022-01-31T21:40:47.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:46 smithi171 conmon[51620]: debug 2022-01-31T21:40:46.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:46.930347+0000) 2022-01-31T21:40:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[41853]: debug 2022-01-31T21:40:47.047+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.048089+0000) 2022-01-31T21:40:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[46715]: debug 2022-01-31T21:40:47.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.092525+0000) 2022-01-31T21:40:47.636 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:47 smithi167 conmon[60316]: debug 2022-01-31T21:40:47.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.321149+0000) 2022-01-31T21:40:47.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:47 smithi167 conmon[54076]: debug 2022-01-31T21:40:47.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.321148+0000) 2022-01-31T21:40:47.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:47 smithi167 conmon[54076]: debug 2022-01-31T21:40:47.829+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.829692+0000) 2022-01-31T21:40:47.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:47 smithi167 conmon[60316]: debug 2022-01-31T21:40:47.827+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.828304+0000) 2022-01-31T21:40:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:47 smithi167 conmon[49112]: debug 2022-01-31T21:40:47.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.656936+0000) 2022-01-31T21:40:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:47 smithi167 conmon[49112]: debug 2022-01-31T21:40:47.827+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.828387+0000) 2022-01-31T21:40:48.002 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[35325]: debug 2022-01-31T21:40:47.844+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 148908 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:48.002 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[41853]: debug 2022-01-31T21:40:47.829+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.830460+0000) 2022-01-31T21:40:48.003 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[46715]: debug 2022-01-31T21:40:47.829+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.829829+0000) 2022-01-31T21:40:48.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[51620]: debug 2022-01-31T21:40:47.828+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.829250+0000) 2022-01-31T21:40:48.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:47 smithi171 conmon[51620]: debug 2022-01-31T21:40:47.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:47.930533+0000) 2022-01-31T21:40:48.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:48 smithi171 conmon[41853]: debug 2022-01-31T21:40:48.047+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:48.048246+0000) 2022-01-31T21:40:48.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:48 smithi171 conmon[46715]: debug 2022-01-31T21:40:48.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:48.092682+0000) 2022-01-31T21:40:48.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:48 smithi167 conmon[54076]: debug 2022-01-31T21:40:48.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:48.321316+0000) 2022-01-31T21:40:48.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:48 smithi167 conmon[60316]: debug 2022-01-31T21:40:48.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:48.321307+0000) 2022-01-31T21:40:48.715 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-01-31T21:40:48.728 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-01-31T21:40:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:48 smithi167 conmon[49112]: debug 2022-01-31T21:40:48.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:48.657106+0000) 2022-01-31T21:40:49.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:48 smithi171 conmon[51620]: debug 2022-01-31T21:40:48.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:48.930712+0000) 2022-01-31T21:40:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:49 smithi171 conmon[41853]: debug 2022-01-31T21:40:49.047+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:49.048428+0000) 2022-01-31T21:40:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:49 smithi171 conmon[46715]: debug 2022-01-31T21:40:49.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:49.092823+0000) 2022-01-31T21:40:49.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:49 smithi167 conmon[54076]: debug 2022-01-31T21:40:49.320+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:49.321436+0000) 2022-01-31T21:40:49.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:49 smithi167 conmon[60316]: debug 2022-01-31T21:40:49.320+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:49.321409+0000) 2022-01-31T21:40:49.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:49 smithi167 conmon[49112]: debug 2022-01-31T21:40:49.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:49.657292+0000) 2022-01-31T21:40:50.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:49 smithi171 conmon[51620]: debug 2022-01-31T21:40:49.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:49.930874+0000) 2022-01-31T21:40:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:50 smithi171 conmon[41853]: debug 2022-01-31T21:40:50.047+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:50.048610+0000) 2022-01-31T21:40:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:50 smithi171 conmon[46715]: debug 2022-01-31T21:40:50.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:50.093021+0000) 2022-01-31T21:40:50.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:50 smithi167 conmon[54076]: debug 2022-01-31T21:40:50.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:50.321608+0000) 2022-01-31T21:40:50.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:50 smithi167 conmon[60316]: debug 2022-01-31T21:40:50.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:50.321577+0000) 2022-01-31T21:40:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:50 smithi167 conmon[49112]: debug 2022-01-31T21:40:50.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:50.657467+0000) 2022-01-31T21:40:51.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:50 smithi171 conmon[51620]: debug 2022-01-31T21:40:50.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:50.931059+0000) 2022-01-31T21:40:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:51 smithi171 conmon[46715]: debug 2022-01-31T21:40:51.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:51.093203+0000) 2022-01-31T21:40:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:51 smithi171 conmon[41853]: debug 2022-01-31T21:40:51.048+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:51.048818+0000) 2022-01-31T21:40:51.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:51 smithi167 conmon[60316]: debug 2022-01-31T21:40:51.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:51.321718+0000) 2022-01-31T21:40:51.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:51 smithi167 conmon[54076]: debug 2022-01-31T21:40:51.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:51.321718+0000) 2022-01-31T21:40:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:51 smithi167 conmon[49112]: debug 2022-01-31T21:40:51.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:51.657619+0000) 2022-01-31T21:40:52.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:51 smithi171 conmon[51620]: debug 2022-01-31T21:40:51.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:51.931233+0000) 2022-01-31T21:40:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[41853]: debug 2022-01-31T21:40:52.048+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.048969+0000) 2022-01-31T21:40:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[46715]: debug 2022-01-31T21:40:52.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.093390+0000) 2022-01-31T21:40:52.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:52 smithi167 conmon[54076]: debug 2022-01-31T21:40:52.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.321851+0000) 2022-01-31T21:40:52.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:52 smithi167 conmon[60316]: debug 2022-01-31T21:40:52.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.321851+0000) 2022-01-31T21:40:52.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:52 smithi167 conmon[54076]: debug 2022-01-31T21:40:52.846+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.846936+0000) 2022-01-31T21:40:52.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:52 smithi167 conmon[60316]: debug 2022-01-31T21:40:52.845+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.846165+0000) 2022-01-31T21:40:52.921 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:52 smithi167 conmon[49112]: debug 2022-01-31T21:40:52.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.657828+0000) 2022-01-31T21:40:52.923 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:52 smithi167 conmon[49112]: debug 2022-01-31T21:40:52.846+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.847180+0000) 2022-01-31T21:40:53.003 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[35325]: debug 2022-01-31T21:40:52.862+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149018 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:53.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[41853]: debug 2022-01-31T21:40:52.847+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.848566+0000) 2022-01-31T21:40:53.005 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[46715]: debug 2022-01-31T21:40:52.847+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.848193+0000) 2022-01-31T21:40:53.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[51620]: debug 2022-01-31T21:40:52.847+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.848323+0000) 2022-01-31T21:40:53.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:52 smithi171 conmon[51620]: debug 2022-01-31T21:40:52.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:52.931423+0000) 2022-01-31T21:40:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:53 smithi171 conmon[41853]: debug 2022-01-31T21:40:53.048+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:53.049147+0000) 2022-01-31T21:40:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:53 smithi171 conmon[46715]: debug 2022-01-31T21:40:53.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:53.093574+0000) 2022-01-31T21:40:53.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:53 smithi167 conmon[54076]: debug 2022-01-31T21:40:53.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:53.322031+0000) 2022-01-31T21:40:53.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:53 smithi167 conmon[60316]: debug 2022-01-31T21:40:53.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:53.322032+0000) 2022-01-31T21:40:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:53 smithi167 conmon[49112]: debug 2022-01-31T21:40:53.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:53.658047+0000) 2022-01-31T21:40:54.002 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:53 smithi171 conmon[35325]: debug 2022-01-31T21:40:53.719+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:40:54.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:53 smithi171 conmon[51620]: debug 2022-01-31T21:40:53.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:53.931584+0000) 2022-01-31T21:40:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:54 smithi171 conmon[41853]: debug 2022-01-31T21:40:54.048+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:54.049331+0000) 2022-01-31T21:40:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:54 smithi171 conmon[46715]: debug 2022-01-31T21:40:54.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:54.093727+0000) 2022-01-31T21:40:54.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:54 smithi167 conmon[54076]: debug 2022-01-31T21:40:54.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:54.322139+0000) 2022-01-31T21:40:54.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:54 smithi167 conmon[60316]: debug 2022-01-31T21:40:54.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:54.322141+0000) 2022-01-31T21:40:54.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:54 smithi167 conmon[49112]: debug 2022-01-31T21:40:54.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:54.658149+0000) 2022-01-31T21:40:55.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:54 smithi171 conmon[51620]: debug 2022-01-31T21:40:54.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:54.931814+0000) 2022-01-31T21:40:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:55 smithi171 conmon[41853]: debug 2022-01-31T21:40:55.048+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:55.049461+0000) 2022-01-31T21:40:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:55 smithi171 conmon[46715]: debug 2022-01-31T21:40:55.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:55.093942+0000) 2022-01-31T21:40:55.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:55 smithi167 conmon[54076]: debug 2022-01-31T21:40:55.321+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:55.322352+0000) 2022-01-31T21:40:55.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:55 smithi167 conmon[60316]: debug 2022-01-31T21:40:55.321+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:55.322352+0000) 2022-01-31T21:40:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:55 smithi167 conmon[49112]: debug 2022-01-31T21:40:55.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:55.658336+0000) 2022-01-31T21:40:56.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:55 smithi171 conmon[51620]: debug 2022-01-31T21:40:55.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:55.931999+0000) 2022-01-31T21:40:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:56 smithi171 conmon[41853]: debug 2022-01-31T21:40:56.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:56.049618+0000) 2022-01-31T21:40:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:56 smithi171 conmon[46715]: debug 2022-01-31T21:40:56.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:56.094128+0000) 2022-01-31T21:40:56.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:56 smithi167 conmon[54076]: debug 2022-01-31T21:40:56.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:56.322561+0000) 2022-01-31T21:40:56.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:56 smithi167 conmon[60316]: debug 2022-01-31T21:40:56.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:56.322561+0000) 2022-01-31T21:40:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:56 smithi167 conmon[49112]: debug 2022-01-31T21:40:56.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:56.658524+0000) 2022-01-31T21:40:57.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:56 smithi171 conmon[51620]: debug 2022-01-31T21:40:56.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:56.932183+0000) 2022-01-31T21:40:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[41853]: debug 2022-01-31T21:40:57.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.049784+0000) 2022-01-31T21:40:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[46715]: debug 2022-01-31T21:40:57.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.094314+0000) 2022-01-31T21:40:57.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:57 smithi167 conmon[54076]: debug 2022-01-31T21:40:57.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.322741+0000) 2022-01-31T21:40:57.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:57 smithi167 conmon[60316]: debug 2022-01-31T21:40:57.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.322741+0000) 2022-01-31T21:40:57.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:57 smithi167 conmon[54076]: debug 2022-01-31T21:40:57.864+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.864507+0000) 2022-01-31T21:40:57.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:57 smithi167 conmon[60316]: debug 2022-01-31T21:40:57.864+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.865271+0000) 2022-01-31T21:40:57.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:57 smithi167 conmon[49112]: debug 2022-01-31T21:40:57.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.658665+0000) 2022-01-31T21:40:57.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:57 smithi167 conmon[49112]: debug 2022-01-31T21:40:57.864+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.865240+0000) 2022-01-31T21:40:58.003 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[35325]: debug 2022-01-31T21:40:57.880+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149129 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:40:58.004 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[46715]: debug 2022-01-31T21:40:57.864+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.865389+0000) 2022-01-31T21:40:58.004 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[41853]: debug 2022-01-31T21:40:57.866+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.866969+0000) 2022-01-31T21:40:58.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[51620]: debug 2022-01-31T21:40:57.865+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.866256+0000) 2022-01-31T21:40:58.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:57 smithi171 conmon[51620]: debug 2022-01-31T21:40:57.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:57.932389+0000) 2022-01-31T21:40:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:58 smithi171 conmon[41853]: debug 2022-01-31T21:40:58.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:58.049928+0000) 2022-01-31T21:40:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:58 smithi171 conmon[46715]: debug 2022-01-31T21:40:58.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:58.094513+0000) 2022-01-31T21:40:58.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:58 smithi167 conmon[54076]: debug 2022-01-31T21:40:58.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:58.322931+0000) 2022-01-31T21:40:58.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:58 smithi167 conmon[60316]: debug 2022-01-31T21:40:58.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:58.322931+0000) 2022-01-31T21:40:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:58 smithi167 conmon[49112]: debug 2022-01-31T21:40:58.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:58.658823+0000) 2022-01-31T21:40:59.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:58 smithi171 conmon[51620]: debug 2022-01-31T21:40:58.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:58.932538+0000) 2022-01-31T21:40:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:40:59 smithi171 conmon[41853]: debug 2022-01-31T21:40:59.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:59.050075+0000) 2022-01-31T21:40:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:40:59 smithi171 conmon[46715]: debug 2022-01-31T21:40:59.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:59.094656+0000) 2022-01-31T21:40:59.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:40:59 smithi167 conmon[54076]: debug 2022-01-31T21:40:59.322+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:59.323104+0000) 2022-01-31T21:40:59.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:40:59 smithi167 conmon[60316]: debug 2022-01-31T21:40:59.322+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:59.323104+0000) 2022-01-31T21:40:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:40:59 smithi167 conmon[49112]: debug 2022-01-31T21:40:59.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:59.658974+0000) 2022-01-31T21:41:00.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:40:59 smithi171 conmon[51620]: debug 2022-01-31T21:40:59.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:40:59.932650+0000) 2022-01-31T21:41:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:00 smithi171 conmon[46715]: debug 2022-01-31T21:41:00.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:00.094865+0000) 2022-01-31T21:41:00.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:00 smithi171 conmon[41853]: debug 2022-01-31T21:41:00.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:00.050229+0000) 2022-01-31T21:41:00.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:00 smithi167 conmon[54076]: debug 2022-01-31T21:41:00.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:00.323311+0000) 2022-01-31T21:41:00.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:00 smithi167 conmon[60316]: debug 2022-01-31T21:41:00.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:00.323311+0000) 2022-01-31T21:41:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:00 smithi167 conmon[49112]: debug 2022-01-31T21:41:00.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:00.659164+0000) 2022-01-31T21:41:01.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:00 smithi171 conmon[51620]: debug 2022-01-31T21:41:00.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:00.932821+0000) 2022-01-31T21:41:01.361 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:01 smithi171 conmon[41853]: debug 2022-01-31T21:41:01.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:01.050343+0000) 2022-01-31T21:41:01.362 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:01 smithi171 conmon[46715]: debug 2022-01-31T21:41:01.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:01.095078+0000) 2022-01-31T21:41:01.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:01 smithi167 conmon[54076]: debug 2022-01-31T21:41:01.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:01.323523+0000) 2022-01-31T21:41:01.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:01 smithi167 conmon[60316]: debug 2022-01-31T21:41:01.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:01.323523+0000) 2022-01-31T21:41:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:01 smithi167 conmon[49112]: debug 2022-01-31T21:41:01.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:01.659321+0000) 2022-01-31T21:41:02.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:01 smithi171 conmon[51620]: debug 2022-01-31T21:41:01.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:01.932991+0000) 2022-01-31T21:41:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[41853]: debug 2022-01-31T21:41:02.049+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.050538+0000) 2022-01-31T21:41:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[46715]: debug 2022-01-31T21:41:02.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.095263+0000) 2022-01-31T21:41:02.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:02 smithi167 conmon[54076]: debug 2022-01-31T21:41:02.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.323745+0000) 2022-01-31T21:41:02.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:02 smithi167 conmon[60316]: debug 2022-01-31T21:41:02.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.323745+0000) 2022-01-31T21:41:02.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:02 smithi167 conmon[54076]: debug 2022-01-31T21:41:02.881+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.882338+0000) 2022-01-31T21:41:02.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:02 smithi167 conmon[60316]: debug 2022-01-31T21:41:02.881+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.882453+0000) 2022-01-31T21:41:02.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:02 smithi167 conmon[49112]: debug 2022-01-31T21:41:02.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.659516+0000) 2022-01-31T21:41:02.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:02 smithi167 conmon[49112]: debug 2022-01-31T21:41:02.882+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.883173+0000) 2022-01-31T21:41:03.004 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[35325]: debug 2022-01-31T21:41:02.899+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149239 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:03.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[41853]: debug 2022-01-31T21:41:02.883+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.884684+0000) 2022-01-31T21:41:03.005 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[46715]: debug 2022-01-31T21:41:02.883+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.884309+0000) 2022-01-31T21:41:03.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[51620]: debug 2022-01-31T21:41:02.882+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.883540+0000) 2022-01-31T21:41:03.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:02 smithi171 conmon[51620]: debug 2022-01-31T21:41:02.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:02.933176+0000) 2022-01-31T21:41:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:03 smithi171 conmon[41853]: debug 2022-01-31T21:41:03.050+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:03.050733+0000) 2022-01-31T21:41:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:03 smithi171 conmon[46715]: debug 2022-01-31T21:41:03.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:03.095432+0000) 2022-01-31T21:41:03.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:03 smithi167 conmon[54076]: debug 2022-01-31T21:41:03.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:03.323918+0000) 2022-01-31T21:41:03.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:03 smithi167 conmon[60316]: debug 2022-01-31T21:41:03.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:03.323918+0000) 2022-01-31T21:41:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:03 smithi167 conmon[49112]: debug 2022-01-31T21:41:03.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:03.659674+0000) 2022-01-31T21:41:04.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:03 smithi171 conmon[51620]: debug 2022-01-31T21:41:03.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:03.933332+0000) 2022-01-31T21:41:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:04 smithi171 conmon[41853]: debug 2022-01-31T21:41:04.050+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:04.050796+0000) 2022-01-31T21:41:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:04 smithi171 conmon[46715]: debug 2022-01-31T21:41:04.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:04.095577+0000) 2022-01-31T21:41:04.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:04 smithi167 conmon[54076]: debug 2022-01-31T21:41:04.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:04.324106+0000) 2022-01-31T21:41:04.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:04 smithi167 conmon[60316]: debug 2022-01-31T21:41:04.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:04.324106+0000) 2022-01-31T21:41:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:04 smithi167 conmon[49112]: debug 2022-01-31T21:41:04.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:04.659776+0000) 2022-01-31T21:41:05.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:04 smithi171 conmon[51620]: debug 2022-01-31T21:41:04.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:04.933546+0000) 2022-01-31T21:41:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:05 smithi171 conmon[46715]: debug 2022-01-31T21:41:05.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:05.095774+0000) 2022-01-31T21:41:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:05 smithi171 conmon[41853]: debug 2022-01-31T21:41:05.050+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:05.051014+0000) 2022-01-31T21:41:05.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:05 smithi167 conmon[54076]: debug 2022-01-31T21:41:05.323+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:05.324356+0000) 2022-01-31T21:41:05.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:05 smithi167 conmon[60316]: debug 2022-01-31T21:41:05.323+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:05.324356+0000) 2022-01-31T21:41:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:05 smithi167 conmon[49112]: debug 2022-01-31T21:41:05.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:05.659961+0000) 2022-01-31T21:41:06.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:05 smithi171 conmon[51620]: debug 2022-01-31T21:41:05.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:05.933728+0000) 2022-01-31T21:41:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:06 smithi171 conmon[41853]: debug 2022-01-31T21:41:06.050+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:06.051194+0000) 2022-01-31T21:41:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:06 smithi171 conmon[46715]: debug 2022-01-31T21:41:06.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:06.095961+0000) 2022-01-31T21:41:06.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:06 smithi167 conmon[54076]: debug 2022-01-31T21:41:06.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:06.324590+0000) 2022-01-31T21:41:06.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:06 smithi167 conmon[60316]: debug 2022-01-31T21:41:06.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:06.324591+0000) 2022-01-31T21:41:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:06 smithi167 conmon[49112]: debug 2022-01-31T21:41:06.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:06.660126+0000) 2022-01-31T21:41:07.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:06 smithi171 conmon[51620]: debug 2022-01-31T21:41:06.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:06.933967+0000) 2022-01-31T21:41:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[41853]: debug 2022-01-31T21:41:07.050+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.051289+0000) 2022-01-31T21:41:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[46715]: debug 2022-01-31T21:41:07.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.096152+0000) 2022-01-31T21:41:07.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:07 smithi167 conmon[54076]: debug 2022-01-31T21:41:07.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.324792+0000) 2022-01-31T21:41:07.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:07 smithi167 conmon[60316]: debug 2022-01-31T21:41:07.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.324792+0000) 2022-01-31T21:41:07.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:07 smithi167 conmon[49112]: debug 2022-01-31T21:41:07.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.660352+0000) 2022-01-31T21:41:08.005 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[35325]: debug 2022-01-31T21:41:07.916+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149350 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:08.006 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[46715]: debug 2022-01-31T21:41:07.902+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.903711+0000) 2022-01-31T21:41:08.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[41853]: debug 2022-01-31T21:41:07.901+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.902736+0000) 2022-01-31T21:41:08.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[51620]: debug 2022-01-31T21:41:07.901+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.902211+0000) 2022-01-31T21:41:08.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:07 smithi171 conmon[51620]: debug 2022-01-31T21:41:07.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.934151+0000) 2022-01-31T21:41:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:07 smithi167 conmon[49112]: debug 2022-01-31T21:41:07.902+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.902744+0000) 2022-01-31T21:41:08.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:07 smithi167 conmon[54076]: debug 2022-01-31T21:41:07.901+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.902380+0000) 2022-01-31T21:41:08.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:07 smithi167 conmon[60316]: debug 2022-01-31T21:41:07.901+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:07.901590+0000) 2022-01-31T21:41:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:08 smithi171 conmon[41853]: debug 2022-01-31T21:41:08.051+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:08.051412+0000) 2022-01-31T21:41:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:08 smithi171 conmon[46715]: debug 2022-01-31T21:41:08.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:08.096329+0000) 2022-01-31T21:41:08.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:08 smithi167 conmon[54076]: debug 2022-01-31T21:41:08.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:08.325020+0000) 2022-01-31T21:41:08.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:08 smithi167 conmon[60316]: debug 2022-01-31T21:41:08.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:08.325020+0000) 2022-01-31T21:41:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:08 smithi167 conmon[49112]: debug 2022-01-31T21:41:08.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:08.660520+0000) 2022-01-31T21:41:09.005 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:08 smithi171 conmon[35325]: debug 2022-01-31T21:41:08.721+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:41:09.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:08 smithi171 conmon[51620]: debug 2022-01-31T21:41:08.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:08.934309+0000) 2022-01-31T21:41:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:09 smithi171 conmon[41853]: debug 2022-01-31T21:41:09.051+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:09.051524+0000) 2022-01-31T21:41:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:09 smithi171 conmon[46715]: debug 2022-01-31T21:41:09.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:09.096474+0000) 2022-01-31T21:41:09.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:09 smithi167 conmon[54076]: debug 2022-01-31T21:41:09.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:09.325168+0000) 2022-01-31T21:41:09.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:09 smithi167 conmon[60316]: debug 2022-01-31T21:41:09.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:09.325168+0000) 2022-01-31T21:41:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:09 smithi167 conmon[49112]: debug 2022-01-31T21:41:09.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:09.660692+0000) 2022-01-31T21:41:10.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:09 smithi171 conmon[51620]: debug 2022-01-31T21:41:09.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:09.934497+0000) 2022-01-31T21:41:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:10 smithi171 conmon[41853]: debug 2022-01-31T21:41:10.050+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:10.051698+0000) 2022-01-31T21:41:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:10 smithi171 conmon[46715]: debug 2022-01-31T21:41:10.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:10.096643+0000) 2022-01-31T21:41:10.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:10 smithi167 conmon[54076]: debug 2022-01-31T21:41:10.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:10.325355+0000) 2022-01-31T21:41:10.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:10 smithi167 conmon[60316]: debug 2022-01-31T21:41:10.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:10.325355+0000) 2022-01-31T21:41:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:10 smithi167 conmon[49112]: debug 2022-01-31T21:41:10.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:10.660908+0000) 2022-01-31T21:41:11.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:10 smithi171 conmon[51620]: debug 2022-01-31T21:41:10.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:10.934715+0000) 2022-01-31T21:41:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:11 smithi171 conmon[41853]: debug 2022-01-31T21:41:11.051+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:11.051913+0000) 2022-01-31T21:41:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:11 smithi171 conmon[46715]: debug 2022-01-31T21:41:11.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:11.096808+0000) 2022-01-31T21:41:11.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:11 smithi167 conmon[60316]: debug 2022-01-31T21:41:11.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:11.325541+0000) 2022-01-31T21:41:11.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:11 smithi167 conmon[54076]: debug 2022-01-31T21:41:11.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:11.325541+0000) 2022-01-31T21:41:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:11 smithi167 conmon[49112]: debug 2022-01-31T21:41:11.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:11.661106+0000) 2022-01-31T21:41:12.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:11 smithi171 conmon[51620]: debug 2022-01-31T21:41:11.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:11.934963+0000) 2022-01-31T21:41:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[41853]: debug 2022-01-31T21:41:12.051+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.052105+0000) 2022-01-31T21:41:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[46715]: debug 2022-01-31T21:41:12.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.097004+0000) 2022-01-31T21:41:12.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:12 smithi167 conmon[60316]: debug 2022-01-31T21:41:12.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.325713+0000) 2022-01-31T21:41:12.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:12 smithi167 conmon[54076]: debug 2022-01-31T21:41:12.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.325714+0000) 2022-01-31T21:41:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:12 smithi167 conmon[49112]: debug 2022-01-31T21:41:12.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.661295+0000) 2022-01-31T21:41:13.006 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[35325]: debug 2022-01-31T21:41:12.940+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149460 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:13.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[41853]: debug 2022-01-31T21:41:12.919+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.920727+0000) 2022-01-31T21:41:13.007 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[46715]: debug 2022-01-31T21:41:12.919+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.919817+0000) 2022-01-31T21:41:13.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[51620]: debug 2022-01-31T21:41:12.920+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.920878+0000) 2022-01-31T21:41:13.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:12 smithi171 conmon[51620]: debug 2022-01-31T21:41:12.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.935588+0000) 2022-01-31T21:41:13.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:12 smithi167 conmon[49112]: debug 2022-01-31T21:41:12.918+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.920040+0000) 2022-01-31T21:41:13.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:12 smithi167 conmon[54076]: debug 2022-01-31T21:41:12.919+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.920640+0000) 2022-01-31T21:41:13.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:12 smithi167 conmon[60316]: debug 2022-01-31T21:41:12.917+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:12.919067+0000) 2022-01-31T21:41:13.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:13 smithi171 conmon[41853]: debug 2022-01-31T21:41:13.051+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:13.052291+0000) 2022-01-31T21:41:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:13 smithi171 conmon[46715]: debug 2022-01-31T21:41:13.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:13.097220+0000) 2022-01-31T21:41:13.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:13 smithi167 conmon[54076]: debug 2022-01-31T21:41:13.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:13.325896+0000) 2022-01-31T21:41:13.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:13 smithi167 conmon[60316]: debug 2022-01-31T21:41:13.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:13.325896+0000) 2022-01-31T21:41:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:13 smithi167 conmon[49112]: debug 2022-01-31T21:41:13.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:13.661534+0000) 2022-01-31T21:41:14.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:13 smithi171 conmon[51620]: debug 2022-01-31T21:41:13.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:13.936143+0000) 2022-01-31T21:41:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:14 smithi171 conmon[41853]: debug 2022-01-31T21:41:14.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:14.052434+0000) 2022-01-31T21:41:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:14 smithi171 conmon[46715]: debug 2022-01-31T21:41:14.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:14.097349+0000) 2022-01-31T21:41:14.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:14 smithi167 conmon[54076]: debug 2022-01-31T21:41:14.324+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:14.325986+0000) 2022-01-31T21:41:14.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:14 smithi167 conmon[60316]: debug 2022-01-31T21:41:14.324+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:14.325986+0000) 2022-01-31T21:41:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:14 smithi167 conmon[49112]: debug 2022-01-31T21:41:14.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:14.661698+0000) 2022-01-31T21:41:15.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:14 smithi171 conmon[51620]: debug 2022-01-31T21:41:14.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:14.936261+0000) 2022-01-31T21:41:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:15 smithi171 conmon[41853]: debug 2022-01-31T21:41:15.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:15.052622+0000) 2022-01-31T21:41:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:15 smithi171 conmon[46715]: debug 2022-01-31T21:41:15.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:15.097516+0000) 2022-01-31T21:41:15.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:15 smithi167 conmon[54076]: debug 2022-01-31T21:41:15.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:15.326136+0000) 2022-01-31T21:41:15.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:15 smithi167 conmon[60316]: debug 2022-01-31T21:41:15.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:15.326137+0000) 2022-01-31T21:41:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:15 smithi167 conmon[49112]: debug 2022-01-31T21:41:15.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:15.661878+0000) 2022-01-31T21:41:16.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:15 smithi171 conmon[51620]: debug 2022-01-31T21:41:15.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:15.936378+0000) 2022-01-31T21:41:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:16 smithi171 conmon[41853]: debug 2022-01-31T21:41:16.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:16.052805+0000) 2022-01-31T21:41:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:16 smithi171 conmon[46715]: debug 2022-01-31T21:41:16.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:16.097703+0000) 2022-01-31T21:41:16.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:16 smithi167 conmon[54076]: debug 2022-01-31T21:41:16.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:16.326302+0000) 2022-01-31T21:41:16.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:16 smithi167 conmon[60316]: debug 2022-01-31T21:41:16.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:16.326304+0000) 2022-01-31T21:41:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:16 smithi167 conmon[49112]: debug 2022-01-31T21:41:16.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:16.662056+0000) 2022-01-31T21:41:17.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:16 smithi171 conmon[51620]: debug 2022-01-31T21:41:16.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:16.936509+0000) 2022-01-31T21:41:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[41853]: debug 2022-01-31T21:41:17.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.052956+0000) 2022-01-31T21:41:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[46715]: debug 2022-01-31T21:41:17.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.097899+0000) 2022-01-31T21:41:17.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:17 smithi167 conmon[54076]: debug 2022-01-31T21:41:17.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.326546+0000) 2022-01-31T21:41:17.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:17 smithi167 conmon[60316]: debug 2022-01-31T21:41:17.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.326546+0000) 2022-01-31T21:41:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:17 smithi167 conmon[49112]: debug 2022-01-31T21:41:17.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.662256+0000) 2022-01-31T21:41:18.007 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[35325]: debug 2022-01-31T21:41:17.960+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:18.007 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[41853]: debug 2022-01-31T21:41:17.943+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.944050+0000) 2022-01-31T21:41:18.008 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[46715]: debug 2022-01-31T21:41:17.942+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.943351+0000) 2022-01-31T21:41:18.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[51620]: debug 2022-01-31T21:41:17.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.936624+0000) 2022-01-31T21:41:18.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:17 smithi171 conmon[51620]: debug 2022-01-31T21:41:17.943+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.944340+0000) 2022-01-31T21:41:18.325 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:17 smithi167 conmon[49112]: debug 2022-01-31T21:41:17.940+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.942478+0000) 2022-01-31T21:41:18.326 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:17 smithi167 conmon[54076]: debug 2022-01-31T21:41:17.941+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.943006+0000) 2022-01-31T21:41:18.326 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:17 smithi167 conmon[60316]: debug 2022-01-31T21:41:17.940+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:17.941977+0000) 2022-01-31T21:41:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:18 smithi171 conmon[41853]: debug 2022-01-31T21:41:18.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:18.053098+0000) 2022-01-31T21:41:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:18 smithi171 conmon[46715]: debug 2022-01-31T21:41:18.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:18.098079+0000) 2022-01-31T21:41:18.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:18 smithi167 conmon[54076]: debug 2022-01-31T21:41:18.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:18.326753+0000) 2022-01-31T21:41:18.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:18 smithi167 conmon[60316]: debug 2022-01-31T21:41:18.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:18.326753+0000) 2022-01-31T21:41:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:18 smithi167 conmon[49112]: debug 2022-01-31T21:41:18.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:18.662473+0000) 2022-01-31T21:41:19.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:18 smithi171 conmon[51620]: debug 2022-01-31T21:41:18.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:18.936776+0000) 2022-01-31T21:41:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:19 smithi171 conmon[41853]: debug 2022-01-31T21:41:19.053+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:19.053249+0000) 2022-01-31T21:41:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:19 smithi171 conmon[46715]: debug 2022-01-31T21:41:19.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:19.098230+0000) 2022-01-31T21:41:19.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:19 smithi167 conmon[54076]: debug 2022-01-31T21:41:19.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:19.326920+0000) 2022-01-31T21:41:19.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:19 smithi167 conmon[60316]: debug 2022-01-31T21:41:19.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:19.326920+0000) 2022-01-31T21:41:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:19 smithi167 conmon[49112]: debug 2022-01-31T21:41:19.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:19.662600+0000) 2022-01-31T21:41:20.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:19 smithi171 conmon[51620]: debug 2022-01-31T21:41:19.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:19.936865+0000) 2022-01-31T21:41:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:20 smithi171 conmon[41853]: debug 2022-01-31T21:41:20.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:20.053401+0000) 2022-01-31T21:41:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:20 smithi171 conmon[46715]: debug 2022-01-31T21:41:20.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:20.098393+0000) 2022-01-31T21:41:20.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:20 smithi167 conmon[60316]: debug 2022-01-31T21:41:20.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:20.327111+0000) 2022-01-31T21:41:20.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:20 smithi167 conmon[54076]: debug 2022-01-31T21:41:20.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:20.327111+0000) 2022-01-31T21:41:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:20 smithi167 conmon[49112]: debug 2022-01-31T21:41:20.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:20.662761+0000) 2022-01-31T21:41:21.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:20 smithi171 conmon[51620]: debug 2022-01-31T21:41:20.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:20.936976+0000) 2022-01-31T21:41:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:21 smithi171 conmon[41853]: debug 2022-01-31T21:41:21.053+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:21.053556+0000) 2022-01-31T21:41:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:21 smithi171 conmon[46715]: debug 2022-01-31T21:41:21.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:21.098553+0000) 2022-01-31T21:41:21.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:21 smithi167 conmon[60316]: debug 2022-01-31T21:41:21.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:21.327296+0000) 2022-01-31T21:41:21.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:21 smithi167 conmon[54076]: debug 2022-01-31T21:41:21.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:21.327296+0000) 2022-01-31T21:41:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:21 smithi167 conmon[49112]: debug 2022-01-31T21:41:21.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:21.662944+0000) 2022-01-31T21:41:22.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:21 smithi171 conmon[51620]: debug 2022-01-31T21:41:21.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:21.937071+0000) 2022-01-31T21:41:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[41853]: debug 2022-01-31T21:41:22.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.053705+0000) 2022-01-31T21:41:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[46715]: debug 2022-01-31T21:41:22.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.098737+0000) 2022-01-31T21:41:22.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:22 smithi167 conmon[60316]: debug 2022-01-31T21:41:22.325+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.327429+0000) 2022-01-31T21:41:22.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:22 smithi167 conmon[54076]: debug 2022-01-31T21:41:22.325+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.327428+0000) 2022-01-31T21:41:22.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:22 smithi167 conmon[49112]: debug 2022-01-31T21:41:22.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.663141+0000) 2022-01-31T21:41:23.007 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[35325]: debug 2022-01-31T21:41:22.977+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149681 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:23.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[41853]: debug 2022-01-31T21:41:22.961+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.963137+0000) 2022-01-31T21:41:23.009 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[46715]: debug 2022-01-31T21:41:22.961+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.963337+0000) 2022-01-31T21:41:23.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[51620]: debug 2022-01-31T21:41:22.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.937174+0000) 2022-01-31T21:41:23.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:22 smithi171 conmon[51620]: debug 2022-01-31T21:41:22.962+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.963824+0000) 2022-01-31T21:41:23.326 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:22 smithi167 conmon[49112]: debug 2022-01-31T21:41:22.962+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.963900+0000) 2022-01-31T21:41:23.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:22 smithi167 conmon[54076]: debug 2022-01-31T21:41:22.962+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.964303+0000) 2022-01-31T21:41:23.327 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:22 smithi167 conmon[60316]: debug 2022-01-31T21:41:22.961+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:22.963483+0000) 2022-01-31T21:41:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:23 smithi171 conmon[41853]: debug 2022-01-31T21:41:23.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:23.053941+0000) 2022-01-31T21:41:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:23 smithi171 conmon[46715]: debug 2022-01-31T21:41:23.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:23.098949+0000) 2022-01-31T21:41:23.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:23 smithi167 conmon[54076]: debug 2022-01-31T21:41:23.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:23.327579+0000) 2022-01-31T21:41:23.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:23 smithi167 conmon[60316]: debug 2022-01-31T21:41:23.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:23.327579+0000) 2022-01-31T21:41:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:23 smithi167 conmon[49112]: debug 2022-01-31T21:41:23.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:23.663331+0000) 2022-01-31T21:41:24.007 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:23 smithi171 conmon[35325]: debug 2022-01-31T21:41:23.722+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:41:24.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:23 smithi171 conmon[51620]: debug 2022-01-31T21:41:23.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:23.937307+0000) 2022-01-31T21:41:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:24 smithi171 conmon[41853]: debug 2022-01-31T21:41:24.053+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:24.054080+0000) 2022-01-31T21:41:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:24 smithi171 conmon[46715]: debug 2022-01-31T21:41:24.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:24.099103+0000) 2022-01-31T21:41:24.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:24 smithi167 conmon[54076]: debug 2022-01-31T21:41:24.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:24.327723+0000) 2022-01-31T21:41:24.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:24 smithi167 conmon[60316]: debug 2022-01-31T21:41:24.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:24.327723+0000) 2022-01-31T21:41:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:24 smithi167 conmon[49112]: debug 2022-01-31T21:41:24.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:24.663523+0000) 2022-01-31T21:41:25.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:24 smithi171 conmon[51620]: debug 2022-01-31T21:41:24.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:24.937451+0000) 2022-01-31T21:41:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:25 smithi171 conmon[41853]: debug 2022-01-31T21:41:25.053+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:25.054263+0000) 2022-01-31T21:41:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:25 smithi171 conmon[46715]: debug 2022-01-31T21:41:25.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:25.099261+0000) 2022-01-31T21:41:25.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:25 smithi167 conmon[54076]: debug 2022-01-31T21:41:25.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:25.327912+0000) 2022-01-31T21:41:25.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:25 smithi167 conmon[60316]: debug 2022-01-31T21:41:25.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:25.327842+0000) 2022-01-31T21:41:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:25 smithi167 conmon[49112]: debug 2022-01-31T21:41:25.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:25.663701+0000) 2022-01-31T21:41:26.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:25 smithi171 conmon[51620]: debug 2022-01-31T21:41:25.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:25.937645+0000) 2022-01-31T21:41:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:26 smithi171 conmon[41853]: debug 2022-01-31T21:41:26.053+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:26.054438+0000) 2022-01-31T21:41:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:26 smithi171 conmon[46715]: debug 2022-01-31T21:41:26.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:26.099447+0000) 2022-01-31T21:41:26.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:26 smithi167 conmon[54076]: debug 2022-01-31T21:41:26.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:26.328044+0000) 2022-01-31T21:41:26.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:26 smithi167 conmon[60316]: debug 2022-01-31T21:41:26.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:26.328035+0000) 2022-01-31T21:41:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:26 smithi167 conmon[49112]: debug 2022-01-31T21:41:26.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:26.663821+0000) 2022-01-31T21:41:27.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:26 smithi171 conmon[51620]: debug 2022-01-31T21:41:26.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:26.937860+0000) 2022-01-31T21:41:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[41853]: debug 2022-01-31T21:41:27.052+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.054611+0000) 2022-01-31T21:41:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[46715]: debug 2022-01-31T21:41:27.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.099634+0000) 2022-01-31T21:41:27.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:27 smithi167 conmon[54076]: debug 2022-01-31T21:41:27.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.328316+0000) 2022-01-31T21:41:27.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:27 smithi167 conmon[60316]: debug 2022-01-31T21:41:27.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.328316+0000) 2022-01-31T21:41:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:27 smithi167 conmon[49112]: debug 2022-01-31T21:41:27.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.664047+0000) 2022-01-31T21:41:28.008 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[35325]: debug 2022-01-31T21:41:27.995+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149793 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:28.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[41853]: debug 2022-01-31T21:41:27.980+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.982096+0000) 2022-01-31T21:41:28.010 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[46715]: debug 2022-01-31T21:41:27.979+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.980833+0000) 2022-01-31T21:41:28.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[51620]: debug 2022-01-31T21:41:27.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.938065+0000) 2022-01-31T21:41:28.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:27 smithi171 conmon[51620]: debug 2022-01-31T21:41:27.981+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.983057+0000) 2022-01-31T21:41:28.327 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:27 smithi167 conmon[49112]: debug 2022-01-31T21:41:27.979+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.980677+0000) 2022-01-31T21:41:28.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:27 smithi167 conmon[54076]: debug 2022-01-31T21:41:27.979+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.981355+0000) 2022-01-31T21:41:28.328 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:27 smithi167 conmon[60316]: debug 2022-01-31T21:41:27.979+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:27.980783+0000) 2022-01-31T21:41:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:28 smithi171 conmon[41853]: debug 2022-01-31T21:41:28.053+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:28.054830+0000) 2022-01-31T21:41:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:28 smithi171 conmon[46715]: debug 2022-01-31T21:41:28.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:28.099850+0000) 2022-01-31T21:41:28.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:28 smithi167 conmon[54076]: debug 2022-01-31T21:41:28.326+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:28.328442+0000) 2022-01-31T21:41:28.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:28 smithi167 conmon[60316]: debug 2022-01-31T21:41:28.326+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:28.328432+0000) 2022-01-31T21:41:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:28 smithi167 conmon[49112]: debug 2022-01-31T21:41:28.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:28.664208+0000) 2022-01-31T21:41:29.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:28 smithi171 conmon[51620]: debug 2022-01-31T21:41:28.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:28.938178+0000) 2022-01-31T21:41:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:29 smithi171 conmon[46715]: debug 2022-01-31T21:41:29.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:29.100003+0000) 2022-01-31T21:41:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:29 smithi171 conmon[41853]: debug 2022-01-31T21:41:29.054+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:29.054996+0000) 2022-01-31T21:41:29.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:29 smithi167 conmon[54076]: debug 2022-01-31T21:41:29.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:29.328593+0000) 2022-01-31T21:41:29.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:29 smithi167 conmon[60316]: debug 2022-01-31T21:41:29.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:29.328591+0000) 2022-01-31T21:41:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:29 smithi167 conmon[49112]: debug 2022-01-31T21:41:29.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:29.664401+0000) 2022-01-31T21:41:30.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:29 smithi171 conmon[51620]: debug 2022-01-31T21:41:29.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:29.938291+0000) 2022-01-31T21:41:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:30 smithi171 conmon[46715]: debug 2022-01-31T21:41:30.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:30.100176+0000) 2022-01-31T21:41:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:30 smithi171 conmon[41853]: debug 2022-01-31T21:41:30.054+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:30.055213+0000) 2022-01-31T21:41:30.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:30 smithi167 conmon[54076]: debug 2022-01-31T21:41:30.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:30.328774+0000) 2022-01-31T21:41:30.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:30 smithi167 conmon[60316]: debug 2022-01-31T21:41:30.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:30.328777+0000) 2022-01-31T21:41:30.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:30 smithi167 conmon[49112]: debug 2022-01-31T21:41:30.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:30.664586+0000) 2022-01-31T21:41:31.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:30 smithi171 conmon[51620]: debug 2022-01-31T21:41:30.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:30.938492+0000) 2022-01-31T21:41:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:31 smithi171 conmon[46715]: debug 2022-01-31T21:41:31.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:31.100333+0000) 2022-01-31T21:41:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:31 smithi171 conmon[41853]: debug 2022-01-31T21:41:31.054+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:31.055418+0000) 2022-01-31T21:41:31.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:31 smithi167 conmon[54076]: debug 2022-01-31T21:41:31.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:31.328937+0000) 2022-01-31T21:41:31.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:31 smithi167 conmon[60316]: debug 2022-01-31T21:41:31.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:31.328984+0000) 2022-01-31T21:41:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:31 smithi167 conmon[49112]: debug 2022-01-31T21:41:31.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:31.664796+0000) 2022-01-31T21:41:32.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:31 smithi171 conmon[51620]: debug 2022-01-31T21:41:31.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:31.938677+0000) 2022-01-31T21:41:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:32 smithi171 conmon[46715]: debug 2022-01-31T21:41:32.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.100525+0000) 2022-01-31T21:41:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:32 smithi171 conmon[41853]: debug 2022-01-31T21:41:32.054+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.055597+0000) 2022-01-31T21:41:32.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:32 smithi167 conmon[54076]: debug 2022-01-31T21:41:32.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.329119+0000) 2022-01-31T21:41:32.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:32 smithi167 conmon[60316]: debug 2022-01-31T21:41:32.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.329141+0000) 2022-01-31T21:41:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:32 smithi167 conmon[49112]: debug 2022-01-31T21:41:32.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.664982+0000) 2022-01-31T21:41:33.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:32 smithi171 conmon[41853]: debug 2022-01-31T21:41:32.998+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.999794+0000) 2022-01-31T21:41:33.010 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:32 smithi171 conmon[46715]: debug 2022-01-31T21:41:32.997+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.999040+0000) 2022-01-31T21:41:33.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:32 smithi171 conmon[51620]: debug 2022-01-31T21:41:32.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.938776+0000) 2022-01-31T21:41:33.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:33 smithi171 conmon[51620]: debug 2022-01-31T21:41:32.999+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.000865+0000) 2022-01-31T21:41:33.328 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:32 smithi167 conmon[49112]: debug 2022-01-31T21:41:32.997+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.999111+0000) 2022-01-31T21:41:33.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:33 smithi167 conmon[54076]: debug 2022-01-31T21:41:32.998+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.000302+0000) 2022-01-31T21:41:33.329 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:32 smithi167 conmon[60316]: debug 2022-01-31T21:41:32.997+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:32.999372+0000) 2022-01-31T21:41:33.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:33 smithi171 conmon[35325]: debug 2022-01-31T21:41:33.014+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 149902 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:33 smithi171 conmon[41853]: debug 2022-01-31T21:41:33.055+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.055828+0000) 2022-01-31T21:41:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:33 smithi171 conmon[46715]: debug 2022-01-31T21:41:33.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.100703+0000) 2022-01-31T21:41:33.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:33 smithi167 conmon[54076]: debug 2022-01-31T21:41:33.328+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.329320+0000) 2022-01-31T21:41:33.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:33 smithi167 conmon[60316]: debug 2022-01-31T21:41:33.328+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.329326+0000) 2022-01-31T21:41:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:33 smithi167 conmon[49112]: debug 2022-01-31T21:41:33.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.665170+0000) 2022-01-31T21:41:34.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:33 smithi171 conmon[51620]: debug 2022-01-31T21:41:33.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:33.938940+0000) 2022-01-31T21:41:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:34 smithi171 conmon[41853]: debug 2022-01-31T21:41:34.055+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:34.055977+0000) 2022-01-31T21:41:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:34 smithi171 conmon[46715]: debug 2022-01-31T21:41:34.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:34.100868+0000) 2022-01-31T21:41:34.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:34 smithi167 conmon[54076]: debug 2022-01-31T21:41:34.327+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:34.329457+0000) 2022-01-31T21:41:34.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:34 smithi167 conmon[60316]: debug 2022-01-31T21:41:34.327+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:34.329388+0000) 2022-01-31T21:41:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:34 smithi167 conmon[49112]: debug 2022-01-31T21:41:34.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:34.665367+0000) 2022-01-31T21:41:35.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:34 smithi171 conmon[51620]: debug 2022-01-31T21:41:34.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:34.939053+0000) 2022-01-31T21:41:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:35 smithi171 conmon[41853]: debug 2022-01-31T21:41:35.055+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:35.056188+0000) 2022-01-31T21:41:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:35 smithi171 conmon[46715]: debug 2022-01-31T21:41:35.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:35.101015+0000) 2022-01-31T21:41:35.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:35 smithi167 conmon[54076]: debug 2022-01-31T21:41:35.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:35.329647+0000) 2022-01-31T21:41:35.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:35 smithi167 conmon[60316]: debug 2022-01-31T21:41:35.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:35.329559+0000) 2022-01-31T21:41:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:35 smithi167 conmon[49112]: debug 2022-01-31T21:41:35.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:35.665540+0000) 2022-01-31T21:41:36.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:35 smithi171 conmon[51620]: debug 2022-01-31T21:41:35.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:35.939218+0000) 2022-01-31T21:41:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:36 smithi171 conmon[46715]: debug 2022-01-31T21:41:36.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:36.101195+0000) 2022-01-31T21:41:36.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:36 smithi171 conmon[41853]: debug 2022-01-31T21:41:36.055+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:36.056341+0000) 2022-01-31T21:41:36.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:36 smithi167 conmon[60316]: debug 2022-01-31T21:41:36.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:36.329722+0000) 2022-01-31T21:41:36.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:36 smithi167 conmon[54076]: debug 2022-01-31T21:41:36.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:36.329830+0000) 2022-01-31T21:41:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:36 smithi167 conmon[49112]: debug 2022-01-31T21:41:36.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:36.665733+0000) 2022-01-31T21:41:37.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:36 smithi171 conmon[51620]: debug 2022-01-31T21:41:36.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:36.939373+0000) 2022-01-31T21:41:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:37 smithi171 conmon[41853]: debug 2022-01-31T21:41:37.055+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:37.056491+0000) 2022-01-31T21:41:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:37 smithi171 conmon[46715]: debug 2022-01-31T21:41:37.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:37.101382+0000) 2022-01-31T21:41:37.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:37 smithi167 conmon[54076]: debug 2022-01-31T21:41:37.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:37.329946+0000) 2022-01-31T21:41:37.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:37 smithi167 conmon[60316]: debug 2022-01-31T21:41:37.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:37.329822+0000) 2022-01-31T21:41:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:37 smithi167 conmon[49112]: debug 2022-01-31T21:41:37.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:37.665875+0000) 2022-01-31T21:41:38.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:37 smithi171 conmon[51620]: debug 2022-01-31T21:41:37.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:37.939564+0000) 2022-01-31T21:41:38.328 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:38 smithi167 conmon[49112]: debug 2022-01-31T21:41:38.017+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.017569+0000) 2022-01-31T21:41:38.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:38 smithi167 conmon[54076]: debug 2022-01-31T21:41:38.017+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.018231+0000) 2022-01-31T21:41:38.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:38 smithi167 conmon[60316]: debug 2022-01-31T21:41:38.016+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.017414+0000) 2022-01-31T21:41:38.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[35325]: debug 2022-01-31T21:41:38.033+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150015 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:38.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[51620]: debug 2022-01-31T21:41:38.018+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.019154+0000) 2022-01-31T21:41:38.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[41853]: debug 2022-01-31T21:41:38.018+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.018892+0000) 2022-01-31T21:41:38.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[41853]: debug 2022-01-31T21:41:38.055+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.056646+0000) 2022-01-31T21:41:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[46715]: debug 2022-01-31T21:41:38.016+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.017575+0000) 2022-01-31T21:41:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[46715]: debug 2022-01-31T21:41:38.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.101566+0000) 2022-01-31T21:41:38.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:38 smithi167 conmon[54076]: debug 2022-01-31T21:41:38.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.330115+0000) 2022-01-31T21:41:38.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:38 smithi167 conmon[60316]: debug 2022-01-31T21:41:38.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.330021+0000) 2022-01-31T21:41:38.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:38 smithi167 conmon[49112]: debug 2022-01-31T21:41:38.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.666036+0000) 2022-01-31T21:41:39.010 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[35325]: debug 2022-01-31T21:41:38.723+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:41:39.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:38 smithi171 conmon[51620]: debug 2022-01-31T21:41:38.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:38.939772+0000) 2022-01-31T21:41:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:39 smithi171 conmon[41853]: debug 2022-01-31T21:41:39.056+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:39.056799+0000) 2022-01-31T21:41:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:39 smithi171 conmon[46715]: debug 2022-01-31T21:41:39.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:39.101709+0000) 2022-01-31T21:41:39.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:39 smithi167 conmon[54076]: debug 2022-01-31T21:41:39.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:39.330252+0000) 2022-01-31T21:41:39.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:39 smithi167 conmon[60316]: debug 2022-01-31T21:41:39.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:39.330184+0000) 2022-01-31T21:41:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:39 smithi167 conmon[49112]: debug 2022-01-31T21:41:39.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:39.666165+0000) 2022-01-31T21:41:40.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:39 smithi171 conmon[51620]: debug 2022-01-31T21:41:39.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:39.939924+0000) 2022-01-31T21:41:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:40 smithi171 conmon[41853]: debug 2022-01-31T21:41:40.056+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:40.057003+0000) 2022-01-31T21:41:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:40 smithi171 conmon[46715]: debug 2022-01-31T21:41:40.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:40.101920+0000) 2022-01-31T21:41:40.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:40 smithi167 conmon[60316]: debug 2022-01-31T21:41:40.329+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:40.330354+0000) 2022-01-31T21:41:40.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:40 smithi167 conmon[54076]: debug 2022-01-31T21:41:40.329+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:40.330427+0000) 2022-01-31T21:41:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:40 smithi167 conmon[49112]: debug 2022-01-31T21:41:40.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:40.666367+0000) 2022-01-31T21:41:41.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:40 smithi171 conmon[51620]: debug 2022-01-31T21:41:40.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:40.940105+0000) 2022-01-31T21:41:41.311 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:41 smithi171 conmon[41853]: debug 2022-01-31T21:41:41.056+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:41.057203+0000) 2022-01-31T21:41:41.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:41 smithi171 conmon[46715]: debug 2022-01-31T21:41:41.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:41.102138+0000) 2022-01-31T21:41:41.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:41 smithi167 conmon[60316]: debug 2022-01-31T21:41:41.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:41.330565+0000) 2022-01-31T21:41:41.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:41 smithi167 conmon[54076]: debug 2022-01-31T21:41:41.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:41.330592+0000) 2022-01-31T21:41:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:41 smithi167 conmon[49112]: debug 2022-01-31T21:41:41.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:41.666580+0000) 2022-01-31T21:41:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:42 smithi171 conmon[41853]: debug 2022-01-31T21:41:42.056+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:42.057364+0000) 2022-01-31T21:41:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:42 smithi171 conmon[46715]: debug 2022-01-31T21:41:42.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:42.102368+0000) 2022-01-31T21:41:42.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:41 smithi171 conmon[51620]: debug 2022-01-31T21:41:41.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:41.940292+0000) 2022-01-31T21:41:42.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:42 smithi167 conmon[60316]: debug 2022-01-31T21:41:42.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:42.330790+0000) 2022-01-31T21:41:42.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:42 smithi167 conmon[54076]: debug 2022-01-31T21:41:42.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:42.330792+0000) 2022-01-31T21:41:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:42 smithi167 conmon[49112]: debug 2022-01-31T21:41:42.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:42.666759+0000) 2022-01-31T21:41:43.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:42 smithi171 conmon[51620]: debug 2022-01-31T21:41:42.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:42.940502+0000) 2022-01-31T21:41:43.329 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:43 smithi167 conmon[49112]: debug 2022-01-31T21:41:43.035+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.036318+0000) 2022-01-31T21:41:43.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:43 smithi167 conmon[54076]: debug 2022-01-31T21:41:43.036+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.036523+0000) 2022-01-31T21:41:43.330 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:43 smithi167 conmon[60316]: debug 2022-01-31T21:41:43.034+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.035413+0000) 2022-01-31T21:41:43.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[51620]: debug 2022-01-31T21:41:43.036+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.037440+0000) 2022-01-31T21:41:43.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[35325]: debug 2022-01-31T21:41:43.052+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150124 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:43.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[41853]: debug 2022-01-31T21:41:43.035+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.036618+0000) 2022-01-31T21:41:43.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[41853]: debug 2022-01-31T21:41:43.056+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.057510+0000) 2022-01-31T21:41:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[46715]: debug 2022-01-31T21:41:43.036+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.036884+0000) 2022-01-31T21:41:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[46715]: debug 2022-01-31T21:41:43.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.102595+0000) 2022-01-31T21:41:43.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:43 smithi167 conmon[54076]: debug 2022-01-31T21:41:43.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.331002+0000) 2022-01-31T21:41:43.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:43 smithi167 conmon[60316]: debug 2022-01-31T21:41:43.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.330947+0000) 2022-01-31T21:41:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:43 smithi167 conmon[49112]: debug 2022-01-31T21:41:43.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.666933+0000) 2022-01-31T21:41:44.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:43 smithi171 conmon[51620]: debug 2022-01-31T21:41:43.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:43.940734+0000) 2022-01-31T21:41:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:44 smithi171 conmon[41853]: debug 2022-01-31T21:41:44.056+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:44.057672+0000) 2022-01-31T21:41:44.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:44 smithi171 conmon[46715]: debug 2022-01-31T21:41:44.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:44.102767+0000) 2022-01-31T21:41:44.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:44 smithi167 conmon[54076]: debug 2022-01-31T21:41:44.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:44.331190+0000) 2022-01-31T21:41:44.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:44 smithi167 conmon[60316]: debug 2022-01-31T21:41:44.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:44.331095+0000) 2022-01-31T21:41:44.923 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:44 smithi167 conmon[49112]: debug 2022-01-31T21:41:44.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:44.667121+0000) 2022-01-31T21:41:45.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:44 smithi171 conmon[51620]: debug 2022-01-31T21:41:44.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:44.940942+0000) 2022-01-31T21:41:45.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:45 smithi171 conmon[46715]: debug 2022-01-31T21:41:45.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:45.102950+0000) 2022-01-31T21:41:45.345 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:45 smithi171 conmon[41853]: debug 2022-01-31T21:41:45.057+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:45.057828+0000) 2022-01-31T21:41:45.569 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:45 smithi167 conmon[54076]: debug 2022-01-31T21:41:45.330+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:45.331347+0000) 2022-01-31T21:41:45.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:45 smithi167 conmon[60316]: debug 2022-01-31T21:41:45.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:45.331300+0000) 2022-01-31T21:41:45.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:45 smithi167 conmon[49112]: debug 2022-01-31T21:41:45.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:45.667349+0000) 2022-01-31T21:41:46.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:45 smithi171 conmon[51620]: debug 2022-01-31T21:41:45.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:45.941136+0000) 2022-01-31T21:41:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:46 smithi171 conmon[46715]: debug 2022-01-31T21:41:46.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:46.103124+0000) 2022-01-31T21:41:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:46 smithi171 conmon[41853]: debug 2022-01-31T21:41:46.057+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:46.057984+0000) 2022-01-31T21:41:46.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:46 smithi167 conmon[54076]: debug 2022-01-31T21:41:46.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:46.331525+0000) 2022-01-31T21:41:46.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:46 smithi167 conmon[60316]: debug 2022-01-31T21:41:46.330+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:46.331426+0000) 2022-01-31T21:41:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:46 smithi167 conmon[49112]: debug 2022-01-31T21:41:46.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:46.667575+0000) 2022-01-31T21:41:47.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:46 smithi171 conmon[51620]: debug 2022-01-31T21:41:46.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:46.941322+0000) 2022-01-31T21:41:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:47 smithi171 conmon[41853]: debug 2022-01-31T21:41:47.057+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:47.058176+0000) 2022-01-31T21:41:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:47 smithi171 conmon[46715]: debug 2022-01-31T21:41:47.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:47.103237+0000) 2022-01-31T21:41:47.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:47 smithi167 conmon[54076]: debug 2022-01-31T21:41:47.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:47.331684+0000) 2022-01-31T21:41:47.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:47 smithi167 conmon[60316]: debug 2022-01-31T21:41:47.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:47.331614+0000) 2022-01-31T21:41:47.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:47 smithi167 conmon[49112]: debug 2022-01-31T21:41:47.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:47.667797+0000) 2022-01-31T21:41:48.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:47 smithi171 conmon[51620]: debug 2022-01-31T21:41:47.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:47.941513+0000) 2022-01-31T21:41:48.330 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:48 smithi167 conmon[49112]: debug 2022-01-31T21:41:48.053+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.054021+0000) 2022-01-31T21:41:48.331 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:48 smithi167 conmon[60316]: debug 2022-01-31T21:41:48.054+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.055325+0000) 2022-01-31T21:41:48.331 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:48 smithi167 conmon[54076]: debug 2022-01-31T21:41:48.054+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.055465+0000) 2022-01-31T21:41:48.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[35325]: debug 2022-01-31T21:41:48.071+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150236 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:48.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[51620]: debug 2022-01-31T21:41:48.054+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.055568+0000) 2022-01-31T21:41:48.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[41853]: debug 2022-01-31T21:41:48.053+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.054586+0000) 2022-01-31T21:41:48.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[41853]: debug 2022-01-31T21:41:48.057+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.058310+0000) 2022-01-31T21:41:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[46715]: debug 2022-01-31T21:41:48.054+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.055300+0000) 2022-01-31T21:41:48.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[46715]: debug 2022-01-31T21:41:48.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.103444+0000) 2022-01-31T21:41:48.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:48 smithi167 conmon[54076]: debug 2022-01-31T21:41:48.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.331873+0000) 2022-01-31T21:41:48.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:48 smithi167 conmon[60316]: debug 2022-01-31T21:41:48.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.331820+0000) 2022-01-31T21:41:48.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:48 smithi167 conmon[49112]: debug 2022-01-31T21:41:48.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.667959+0000) 2022-01-31T21:41:49.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:48 smithi171 conmon[51620]: debug 2022-01-31T21:41:48.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:48.941709+0000) 2022-01-31T21:41:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:49 smithi171 conmon[41853]: debug 2022-01-31T21:41:49.057+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:49.058491+0000) 2022-01-31T21:41:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:49 smithi171 conmon[46715]: debug 2022-01-31T21:41:49.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:49.103601+0000) 2022-01-31T21:41:49.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:49 smithi167 conmon[54076]: debug 2022-01-31T21:41:49.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:49.332027+0000) 2022-01-31T21:41:49.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:49 smithi167 conmon[60316]: debug 2022-01-31T21:41:49.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:49.331948+0000) 2022-01-31T21:41:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:49 smithi167 conmon[49112]: debug 2022-01-31T21:41:49.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:49.668153+0000) 2022-01-31T21:41:50.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:49 smithi171 conmon[51620]: debug 2022-01-31T21:41:49.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:49.941858+0000) 2022-01-31T21:41:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:50 smithi171 conmon[41853]: debug 2022-01-31T21:41:50.058+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:50.058677+0000) 2022-01-31T21:41:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:50 smithi171 conmon[46715]: debug 2022-01-31T21:41:50.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:50.103763+0000) 2022-01-31T21:41:50.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:50 smithi167 conmon[54076]: debug 2022-01-31T21:41:50.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:50.332172+0000) 2022-01-31T21:41:50.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:50 smithi167 conmon[60316]: debug 2022-01-31T21:41:50.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:50.332099+0000) 2022-01-31T21:41:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:50 smithi167 conmon[49112]: debug 2022-01-31T21:41:50.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:50.668361+0000) 2022-01-31T21:41:51.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:50 smithi171 conmon[51620]: debug 2022-01-31T21:41:50.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:50.941991+0000) 2022-01-31T21:41:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:51 smithi171 conmon[46715]: debug 2022-01-31T21:41:51.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:51.103921+0000) 2022-01-31T21:41:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:51 smithi171 conmon[41853]: debug 2022-01-31T21:41:51.058+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:51.058831+0000) 2022-01-31T21:41:51.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:51 smithi167 conmon[54076]: debug 2022-01-31T21:41:51.331+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:51.332378+0000) 2022-01-31T21:41:51.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:51 smithi167 conmon[60316]: debug 2022-01-31T21:41:51.331+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:51.332249+0000) 2022-01-31T21:41:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:51 smithi167 conmon[49112]: debug 2022-01-31T21:41:51.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:51.668534+0000) 2022-01-31T21:41:52.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:51 smithi171 conmon[51620]: debug 2022-01-31T21:41:51.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:51.942170+0000) 2022-01-31T21:41:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:52 smithi171 conmon[46715]: debug 2022-01-31T21:41:52.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:52.104116+0000) 2022-01-31T21:41:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:52 smithi171 conmon[41853]: debug 2022-01-31T21:41:52.058+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:52.059015+0000) 2022-01-31T21:41:52.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:52 smithi167 conmon[54076]: debug 2022-01-31T21:41:52.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:52.332768+0000) 2022-01-31T21:41:52.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:52 smithi167 conmon[60316]: debug 2022-01-31T21:41:52.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:52.332759+0000) 2022-01-31T21:41:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:52 smithi167 conmon[49112]: debug 2022-01-31T21:41:52.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:52.668756+0000) 2022-01-31T21:41:53.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:52 smithi171 conmon[51620]: debug 2022-01-31T21:41:52.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:52.942369+0000) 2022-01-31T21:41:53.331 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:53 smithi167 conmon[49112]: debug 2022-01-31T21:41:53.072+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.073442+0000) 2022-01-31T21:41:53.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:53 smithi167 conmon[54076]: debug 2022-01-31T21:41:53.074+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.074646+0000) 2022-01-31T21:41:53.333 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:53 smithi167 conmon[60316]: debug 2022-01-31T21:41:53.073+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.074429+0000) 2022-01-31T21:41:53.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[35325]: debug 2022-01-31T21:41:53.089+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150345 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:53.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[51620]: debug 2022-01-31T21:41:53.073+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.073959+0000) 2022-01-31T21:41:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[41853]: debug 2022-01-31T21:41:53.058+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.059209+0000) 2022-01-31T21:41:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[41853]: debug 2022-01-31T21:41:53.073+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.074352+0000) 2022-01-31T21:41:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[46715]: debug 2022-01-31T21:41:53.074+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.075133+0000) 2022-01-31T21:41:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[46715]: debug 2022-01-31T21:41:53.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.104264+0000) 2022-01-31T21:41:53.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:53 smithi167 conmon[60316]: debug 2022-01-31T21:41:53.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.332952+0000) 2022-01-31T21:41:53.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:53 smithi167 conmon[54076]: debug 2022-01-31T21:41:53.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.332952+0000) 2022-01-31T21:41:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:53 smithi167 conmon[49112]: debug 2022-01-31T21:41:53.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.668933+0000) 2022-01-31T21:41:54.013 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[35325]: debug 2022-01-31T21:41:53.724+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:41:54.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:53 smithi171 conmon[51620]: debug 2022-01-31T21:41:53.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:53.942508+0000) 2022-01-31T21:41:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:54 smithi171 conmon[46715]: debug 2022-01-31T21:41:54.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:54.104403+0000) 2022-01-31T21:41:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:54 smithi171 conmon[41853]: debug 2022-01-31T21:41:54.058+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:54.059387+0000) 2022-01-31T21:41:54.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:54 smithi167 conmon[54076]: debug 2022-01-31T21:41:54.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:54.333111+0000) 2022-01-31T21:41:54.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:54 smithi167 conmon[60316]: debug 2022-01-31T21:41:54.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:54.333122+0000) 2022-01-31T21:41:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:54 smithi167 conmon[49112]: debug 2022-01-31T21:41:54.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:54.669094+0000) 2022-01-31T21:41:55.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:54 smithi171 conmon[51620]: debug 2022-01-31T21:41:54.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:54.942647+0000) 2022-01-31T21:41:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:55 smithi171 conmon[46715]: debug 2022-01-31T21:41:55.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:55.104586+0000) 2022-01-31T21:41:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:55 smithi171 conmon[41853]: debug 2022-01-31T21:41:55.059+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:55.059568+0000) 2022-01-31T21:41:55.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:55 smithi167 conmon[54076]: debug 2022-01-31T21:41:55.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:55.333298+0000) 2022-01-31T21:41:55.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:55 smithi167 conmon[60316]: debug 2022-01-31T21:41:55.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:55.333317+0000) 2022-01-31T21:41:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:55 smithi167 conmon[49112]: debug 2022-01-31T21:41:55.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:55.669264+0000) 2022-01-31T21:41:56.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:55 smithi171 conmon[51620]: debug 2022-01-31T21:41:55.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:55.942858+0000) 2022-01-31T21:41:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:56 smithi171 conmon[46715]: debug 2022-01-31T21:41:56.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:56.104786+0000) 2022-01-31T21:41:56.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:56 smithi171 conmon[41853]: debug 2022-01-31T21:41:56.059+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:56.059723+0000) 2022-01-31T21:41:56.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:56 smithi167 conmon[54076]: debug 2022-01-31T21:41:56.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:56.333497+0000) 2022-01-31T21:41:56.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:56 smithi167 conmon[60316]: debug 2022-01-31T21:41:56.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:56.333541+0000) 2022-01-31T21:41:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:56 smithi167 conmon[49112]: debug 2022-01-31T21:41:56.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:56.669450+0000) 2022-01-31T21:41:57.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:56 smithi171 conmon[51620]: debug 2022-01-31T21:41:56.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:56.943006+0000) 2022-01-31T21:41:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:57 smithi171 conmon[41853]: debug 2022-01-31T21:41:57.059+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:57.059931+0000) 2022-01-31T21:41:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:57 smithi171 conmon[46715]: debug 2022-01-31T21:41:57.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:57.104929+0000) 2022-01-31T21:41:57.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:57 smithi167 conmon[54076]: debug 2022-01-31T21:41:57.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:57.333756+0000) 2022-01-31T21:41:57.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:57 smithi167 conmon[60316]: debug 2022-01-31T21:41:57.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:57.333719+0000) 2022-01-31T21:41:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:57 smithi167 conmon[49112]: debug 2022-01-31T21:41:57.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:57.669651+0000) 2022-01-31T21:41:58.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:57 smithi171 conmon[51620]: debug 2022-01-31T21:41:57.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:57.943166+0000) 2022-01-31T21:41:58.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[35325]: debug 2022-01-31T21:41:58.108+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150458 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:41:58.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[51620]: debug 2022-01-31T21:41:58.091+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.092174+0000) 2022-01-31T21:41:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[41853]: debug 2022-01-31T21:41:58.059+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.060077+0000) 2022-01-31T21:41:58.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[41853]: debug 2022-01-31T21:41:58.092+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.093318+0000) 2022-01-31T21:41:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[46715]: debug 2022-01-31T21:41:58.092+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.093661+0000) 2022-01-31T21:41:58.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[46715]: debug 2022-01-31T21:41:58.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.105122+0000) 2022-01-31T21:41:58.388 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:58 smithi167 conmon[49112]: debug 2022-01-31T21:41:58.091+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.092347+0000) 2022-01-31T21:41:58.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:58 smithi167 conmon[54076]: debug 2022-01-31T21:41:58.092+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.093144+0000) 2022-01-31T21:41:58.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:58 smithi167 conmon[54076]: debug 2022-01-31T21:41:58.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.333884+0000) 2022-01-31T21:41:58.389 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:58 smithi167 conmon[60316]: debug 2022-01-31T21:41:58.092+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.092791+0000) 2022-01-31T21:41:58.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:58 smithi167 conmon[60316]: debug 2022-01-31T21:41:58.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.333843+0000) 2022-01-31T21:41:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:58 smithi167 conmon[49112]: debug 2022-01-31T21:41:58.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.669830+0000) 2022-01-31T21:41:59.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:58 smithi171 conmon[51620]: debug 2022-01-31T21:41:58.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:58.943339+0000) 2022-01-31T21:41:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:41:59 smithi171 conmon[41853]: debug 2022-01-31T21:41:59.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:59.060270+0000) 2022-01-31T21:41:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:41:59 smithi171 conmon[46715]: debug 2022-01-31T21:41:59.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:59.105289+0000) 2022-01-31T21:41:59.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:41:59 smithi167 conmon[60316]: debug 2022-01-31T21:41:59.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:59.334002+0000) 2022-01-31T21:41:59.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:41:59 smithi167 conmon[54076]: debug 2022-01-31T21:41:59.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:59.334023+0000) 2022-01-31T21:41:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:41:59 smithi167 conmon[49112]: debug 2022-01-31T21:41:59.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:59.669992+0000) 2022-01-31T21:42:00.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:41:59 smithi171 conmon[51620]: debug 2022-01-31T21:41:59.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:41:59.943538+0000) 2022-01-31T21:42:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:00 smithi171 conmon[41853]: debug 2022-01-31T21:42:00.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:00.060419+0000) 2022-01-31T21:42:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:00 smithi171 conmon[46715]: debug 2022-01-31T21:42:00.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:00.105445+0000) 2022-01-31T21:42:00.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:00 smithi167 conmon[54076]: debug 2022-01-31T21:42:00.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:00.334209+0000) 2022-01-31T21:42:00.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:00 smithi167 conmon[60316]: debug 2022-01-31T21:42:00.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:00.334192+0000) 2022-01-31T21:42:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:00 smithi167 conmon[49112]: debug 2022-01-31T21:42:00.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:00.670202+0000) 2022-01-31T21:42:01.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:00 smithi171 conmon[51620]: debug 2022-01-31T21:42:00.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:00.943725+0000) 2022-01-31T21:42:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:01 smithi171 conmon[41853]: debug 2022-01-31T21:42:01.059+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:01.060637+0000) 2022-01-31T21:42:01.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:01 smithi171 conmon[46715]: debug 2022-01-31T21:42:01.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:01.105647+0000) 2022-01-31T21:42:01.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:01 smithi167 conmon[54076]: debug 2022-01-31T21:42:01.332+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:01.334379+0000) 2022-01-31T21:42:01.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:01 smithi167 conmon[60316]: debug 2022-01-31T21:42:01.332+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:01.334382+0000) 2022-01-31T21:42:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:01 smithi167 conmon[49112]: debug 2022-01-31T21:42:01.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:01.670364+0000) 2022-01-31T21:42:02.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:01 smithi171 conmon[51620]: debug 2022-01-31T21:42:01.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:01.943915+0000) 2022-01-31T21:42:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:02 smithi171 conmon[41853]: debug 2022-01-31T21:42:02.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:02.060820+0000) 2022-01-31T21:42:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:02 smithi171 conmon[46715]: debug 2022-01-31T21:42:02.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:02.105831+0000) 2022-01-31T21:42:02.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:02 smithi167 conmon[54076]: debug 2022-01-31T21:42:02.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:02.334569+0000) 2022-01-31T21:42:02.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:02 smithi167 conmon[60316]: debug 2022-01-31T21:42:02.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:02.334570+0000) 2022-01-31T21:42:02.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:02 smithi167 conmon[49112]: debug 2022-01-31T21:42:02.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:02.670527+0000) 2022-01-31T21:42:03.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:02 smithi171 conmon[51620]: debug 2022-01-31T21:42:02.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:02.944094+0000) 2022-01-31T21:42:03.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[35325]: debug 2022-01-31T21:42:03.126+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:03.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[41853]: debug 2022-01-31T21:42:03.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.060997+0000) 2022-01-31T21:42:03.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[41853]: debug 2022-01-31T21:42:03.110+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.111645+0000) 2022-01-31T21:42:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[46715]: debug 2022-01-31T21:42:03.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.105990+0000) 2022-01-31T21:42:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[46715]: debug 2022-01-31T21:42:03.111+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.112039+0000) 2022-01-31T21:42:03.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[51620]: debug 2022-01-31T21:42:03.111+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.112426+0000) 2022-01-31T21:42:03.389 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:03 smithi167 conmon[49112]: debug 2022-01-31T21:42:03.110+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.112058+0000) 2022-01-31T21:42:03.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:03 smithi167 conmon[54076]: debug 2022-01-31T21:42:03.110+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.111831+0000) 2022-01-31T21:42:03.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:03 smithi167 conmon[54076]: debug 2022-01-31T21:42:03.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.334784+0000) 2022-01-31T21:42:03.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:03 smithi167 conmon[60316]: debug 2022-01-31T21:42:03.109+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.110773+0000) 2022-01-31T21:42:03.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:03 smithi167 conmon[60316]: debug 2022-01-31T21:42:03.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.334783+0000) 2022-01-31T21:42:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:03 smithi167 conmon[49112]: debug 2022-01-31T21:42:03.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.670696+0000) 2022-01-31T21:42:04.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:03 smithi171 conmon[51620]: debug 2022-01-31T21:42:03.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:03.944297+0000) 2022-01-31T21:42:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:04 smithi171 conmon[41853]: debug 2022-01-31T21:42:04.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:04.061157+0000) 2022-01-31T21:42:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:04 smithi171 conmon[46715]: debug 2022-01-31T21:42:04.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:04.106174+0000) 2022-01-31T21:42:04.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:04 smithi167 conmon[54076]: debug 2022-01-31T21:42:04.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:04.334961+0000) 2022-01-31T21:42:04.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:04 smithi167 conmon[60316]: debug 2022-01-31T21:42:04.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:04.334951+0000) 2022-01-31T21:42:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:04 smithi167 conmon[49112]: debug 2022-01-31T21:42:04.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:04.670819+0000) 2022-01-31T21:42:05.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:04 smithi171 conmon[51620]: debug 2022-01-31T21:42:04.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:04.944495+0000) 2022-01-31T21:42:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:05 smithi171 conmon[41853]: debug 2022-01-31T21:42:05.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:05.061309+0000) 2022-01-31T21:42:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:05 smithi171 conmon[46715]: debug 2022-01-31T21:42:05.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:05.106352+0000) 2022-01-31T21:42:05.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:05 smithi167 conmon[54076]: debug 2022-01-31T21:42:05.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:05.335179+0000) 2022-01-31T21:42:05.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:05 smithi167 conmon[60316]: debug 2022-01-31T21:42:05.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:05.335180+0000) 2022-01-31T21:42:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:05 smithi167 conmon[49112]: debug 2022-01-31T21:42:05.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:05.671028+0000) 2022-01-31T21:42:06.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:05 smithi171 conmon[51620]: debug 2022-01-31T21:42:05.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:05.944698+0000) 2022-01-31T21:42:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:06 smithi171 conmon[41853]: debug 2022-01-31T21:42:06.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:06.061473+0000) 2022-01-31T21:42:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:06 smithi171 conmon[46715]: debug 2022-01-31T21:42:06.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:06.106519+0000) 2022-01-31T21:42:06.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:06 smithi167 conmon[60316]: debug 2022-01-31T21:42:06.333+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:06.335399+0000) 2022-01-31T21:42:06.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:06 smithi167 conmon[54076]: debug 2022-01-31T21:42:06.333+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:06.335401+0000) 2022-01-31T21:42:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:06 smithi167 conmon[49112]: debug 2022-01-31T21:42:06.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:06.671202+0000) 2022-01-31T21:42:07.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:06 smithi171 conmon[51620]: debug 2022-01-31T21:42:06.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:06.944924+0000) 2022-01-31T21:42:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:07 smithi171 conmon[41853]: debug 2022-01-31T21:42:07.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:07.061618+0000) 2022-01-31T21:42:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:07 smithi171 conmon[46715]: debug 2022-01-31T21:42:07.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:07.106630+0000) 2022-01-31T21:42:07.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:07 smithi167 conmon[54076]: debug 2022-01-31T21:42:07.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:07.335589+0000) 2022-01-31T21:42:07.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:07 smithi167 conmon[60316]: debug 2022-01-31T21:42:07.334+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:07.335555+0000) 2022-01-31T21:42:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:07 smithi167 conmon[49112]: debug 2022-01-31T21:42:07.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:07.671392+0000) 2022-01-31T21:42:08.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:07 smithi171 conmon[51620]: debug 2022-01-31T21:42:07.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:07.945111+0000) 2022-01-31T21:42:08.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[35325]: debug 2022-01-31T21:42:08.143+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150681 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:08.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[51620]: debug 2022-01-31T21:42:08.127+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.129038+0000) 2022-01-31T21:42:08.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[41853]: debug 2022-01-31T21:42:08.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.061801+0000) 2022-01-31T21:42:08.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[41853]: debug 2022-01-31T21:42:08.128+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.130417+0000) 2022-01-31T21:42:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[46715]: debug 2022-01-31T21:42:08.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.106841+0000) 2022-01-31T21:42:08.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[46715]: debug 2022-01-31T21:42:08.127+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.129395+0000) 2022-01-31T21:42:08.390 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:08 smithi167 conmon[49112]: debug 2022-01-31T21:42:08.127+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.129027+0000) 2022-01-31T21:42:08.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:08 smithi167 conmon[60316]: debug 2022-01-31T21:42:08.128+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.129648+0000) 2022-01-31T21:42:08.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:08 smithi167 conmon[60316]: debug 2022-01-31T21:42:08.334+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.335755+0000) 2022-01-31T21:42:08.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:08 smithi167 conmon[54076]: debug 2022-01-31T21:42:08.127+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.128716+0000) 2022-01-31T21:42:08.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:08 smithi167 conmon[54076]: debug 2022-01-31T21:42:08.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.335760+0000) 2022-01-31T21:42:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:08 smithi167 conmon[49112]: debug 2022-01-31T21:42:08.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.671615+0000) 2022-01-31T21:42:09.015 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[35325]: debug 2022-01-31T21:42:08.724+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:42:09.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:08 smithi171 conmon[51620]: debug 2022-01-31T21:42:08.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:08.945309+0000) 2022-01-31T21:42:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:09 smithi171 conmon[41853]: debug 2022-01-31T21:42:09.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:09.061919+0000) 2022-01-31T21:42:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:09 smithi171 conmon[46715]: debug 2022-01-31T21:42:09.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:09.107017+0000) 2022-01-31T21:42:09.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:09 smithi167 conmon[54076]: debug 2022-01-31T21:42:09.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:09.335934+0000) 2022-01-31T21:42:09.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:09 smithi167 conmon[60316]: debug 2022-01-31T21:42:09.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:09.335933+0000) 2022-01-31T21:42:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:09 smithi167 conmon[49112]: debug 2022-01-31T21:42:09.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:09.671744+0000) 2022-01-31T21:42:10.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:09 smithi171 conmon[51620]: debug 2022-01-31T21:42:09.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:09.945465+0000) 2022-01-31T21:42:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:10 smithi171 conmon[46715]: debug 2022-01-31T21:42:10.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:10.107171+0000) 2022-01-31T21:42:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:10 smithi171 conmon[41853]: debug 2022-01-31T21:42:10.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:10.062049+0000) 2022-01-31T21:42:10.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:10 smithi167 conmon[54076]: debug 2022-01-31T21:42:10.334+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:10.336156+0000) 2022-01-31T21:42:10.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:10 smithi167 conmon[60316]: debug 2022-01-31T21:42:10.334+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:10.336158+0000) 2022-01-31T21:42:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:10 smithi167 conmon[49112]: debug 2022-01-31T21:42:10.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:10.671902+0000) 2022-01-31T21:42:11.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:10 smithi171 conmon[51620]: debug 2022-01-31T21:42:10.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:10.945646+0000) 2022-01-31T21:42:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:11 smithi171 conmon[41853]: debug 2022-01-31T21:42:11.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:11.062193+0000) 2022-01-31T21:42:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:11 smithi171 conmon[46715]: debug 2022-01-31T21:42:11.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:11.107355+0000) 2022-01-31T21:42:11.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:11 smithi167 conmon[54076]: debug 2022-01-31T21:42:11.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:11.336363+0000) 2022-01-31T21:42:11.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:11 smithi167 conmon[60316]: debug 2022-01-31T21:42:11.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:11.336365+0000) 2022-01-31T21:42:11.923 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:11 smithi167 conmon[49112]: debug 2022-01-31T21:42:11.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:11.672110+0000) 2022-01-31T21:42:12.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:11 smithi171 conmon[51620]: debug 2022-01-31T21:42:11.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:11.945831+0000) 2022-01-31T21:42:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:12 smithi171 conmon[41853]: debug 2022-01-31T21:42:12.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:12.062354+0000) 2022-01-31T21:42:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:12 smithi171 conmon[46715]: debug 2022-01-31T21:42:12.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:12.107531+0000) 2022-01-31T21:42:12.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:12 smithi167 conmon[54076]: debug 2022-01-31T21:42:12.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:12.336548+0000) 2022-01-31T21:42:12.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:12 smithi167 conmon[60316]: debug 2022-01-31T21:42:12.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:12.336545+0000) 2022-01-31T21:42:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:12 smithi167 conmon[49112]: debug 2022-01-31T21:42:12.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:12.672320+0000) 2022-01-31T21:42:13.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:12 smithi171 conmon[51620]: debug 2022-01-31T21:42:12.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:12.945935+0000) 2022-01-31T21:42:13.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[35325]: debug 2022-01-31T21:42:13.165+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150790 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:13.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[51620]: debug 2022-01-31T21:42:13.148+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.150602+0000) 2022-01-31T21:42:13.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[41853]: debug 2022-01-31T21:42:13.060+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.062588+0000) 2022-01-31T21:42:13.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[41853]: debug 2022-01-31T21:42:13.147+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.149197+0000) 2022-01-31T21:42:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[46715]: debug 2022-01-31T21:42:13.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.107728+0000) 2022-01-31T21:42:13.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[46715]: debug 2022-01-31T21:42:13.150+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.151937+0000) 2022-01-31T21:42:13.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:13 smithi167 conmon[49112]: debug 2022-01-31T21:42:13.146+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.148456+0000) 2022-01-31T21:42:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:13 smithi167 conmon[54076]: debug 2022-01-31T21:42:13.146+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.147926+0000) 2022-01-31T21:42:13.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:13 smithi167 conmon[54076]: debug 2022-01-31T21:42:13.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.336751+0000) 2022-01-31T21:42:13.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:13 smithi167 conmon[60316]: debug 2022-01-31T21:42:13.146+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.148209+0000) 2022-01-31T21:42:13.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:13 smithi167 conmon[60316]: debug 2022-01-31T21:42:13.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.336746+0000) 2022-01-31T21:42:13.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:13 smithi167 conmon[49112]: debug 2022-01-31T21:42:13.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.672476+0000) 2022-01-31T21:42:14.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:13 smithi171 conmon[51620]: debug 2022-01-31T21:42:13.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:13.946043+0000) 2022-01-31T21:42:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:14 smithi171 conmon[46715]: debug 2022-01-31T21:42:14.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:14.107938+0000) 2022-01-31T21:42:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:14 smithi171 conmon[41853]: debug 2022-01-31T21:42:14.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:14.062793+0000) 2022-01-31T21:42:14.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:14 smithi167 conmon[54076]: debug 2022-01-31T21:42:14.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:14.336856+0000) 2022-01-31T21:42:14.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:14 smithi167 conmon[60316]: debug 2022-01-31T21:42:14.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:14.336889+0000) 2022-01-31T21:42:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:14 smithi167 conmon[49112]: debug 2022-01-31T21:42:14.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:14.672657+0000) 2022-01-31T21:42:15.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:14 smithi171 conmon[51620]: debug 2022-01-31T21:42:14.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:14.946220+0000) 2022-01-31T21:42:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:15 smithi171 conmon[41853]: debug 2022-01-31T21:42:15.062+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:15.062945+0000) 2022-01-31T21:42:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:15 smithi171 conmon[46715]: debug 2022-01-31T21:42:15.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:15.108088+0000) 2022-01-31T21:42:15.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:15 smithi167 conmon[60316]: debug 2022-01-31T21:42:15.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:15.337097+0000) 2022-01-31T21:42:15.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:15 smithi167 conmon[54076]: debug 2022-01-31T21:42:15.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:15.337069+0000) 2022-01-31T21:42:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:15 smithi167 conmon[49112]: debug 2022-01-31T21:42:15.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:15.672851+0000) 2022-01-31T21:42:16.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:15 smithi171 conmon[51620]: debug 2022-01-31T21:42:15.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:15.946426+0000) 2022-01-31T21:42:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:16 smithi171 conmon[41853]: debug 2022-01-31T21:42:16.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:16.063099+0000) 2022-01-31T21:42:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:16 smithi171 conmon[46715]: debug 2022-01-31T21:42:16.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:16.108257+0000) 2022-01-31T21:42:16.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:16 smithi167 conmon[54076]: debug 2022-01-31T21:42:16.335+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:16.337275+0000) 2022-01-31T21:42:16.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:16 smithi167 conmon[60316]: debug 2022-01-31T21:42:16.335+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:16.337282+0000) 2022-01-31T21:42:16.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:16 smithi167 conmon[49112]: debug 2022-01-31T21:42:16.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:16.673051+0000) 2022-01-31T21:42:17.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:16 smithi171 conmon[51620]: debug 2022-01-31T21:42:16.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:16.946595+0000) 2022-01-31T21:42:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:17 smithi171 conmon[46715]: debug 2022-01-31T21:42:17.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:17.108396+0000) 2022-01-31T21:42:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:17 smithi171 conmon[41853]: debug 2022-01-31T21:42:17.061+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:17.063222+0000) 2022-01-31T21:42:17.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:17 smithi167 conmon[60316]: debug 2022-01-31T21:42:17.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:17.337480+0000) 2022-01-31T21:42:17.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:17 smithi167 conmon[54076]: debug 2022-01-31T21:42:17.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:17.337483+0000) 2022-01-31T21:42:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:17 smithi167 conmon[49112]: debug 2022-01-31T21:42:17.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:17.673268+0000) 2022-01-31T21:42:18.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:17 smithi171 conmon[51620]: debug 2022-01-31T21:42:17.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:17.946814+0000) 2022-01-31T21:42:18.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[35325]: debug 2022-01-31T21:42:18.185+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 150902 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:18.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[51620]: debug 2022-01-31T21:42:18.168+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.170329+0000) 2022-01-31T21:42:18.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[41853]: debug 2022-01-31T21:42:18.062+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.063386+0000) 2022-01-31T21:42:18.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[41853]: debug 2022-01-31T21:42:18.169+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.170881+0000) 2022-01-31T21:42:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[46715]: debug 2022-01-31T21:42:18.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.108553+0000) 2022-01-31T21:42:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[46715]: debug 2022-01-31T21:42:18.168+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.170726+0000) 2022-01-31T21:42:18.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:18 smithi167 conmon[49112]: debug 2022-01-31T21:42:18.167+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.168997+0000) 2022-01-31T21:42:18.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:18 smithi167 conmon[54076]: debug 2022-01-31T21:42:18.168+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.170010+0000) 2022-01-31T21:42:18.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:18 smithi167 conmon[54076]: debug 2022-01-31T21:42:18.337+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.337725+0000) 2022-01-31T21:42:18.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:18 smithi167 conmon[60316]: debug 2022-01-31T21:42:18.168+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.170241+0000) 2022-01-31T21:42:18.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:18 smithi167 conmon[60316]: debug 2022-01-31T21:42:18.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.337718+0000) 2022-01-31T21:42:18.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:18 smithi167 conmon[49112]: debug 2022-01-31T21:42:18.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.673492+0000) 2022-01-31T21:42:19.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:18 smithi171 conmon[51620]: debug 2022-01-31T21:42:18.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:18.947001+0000) 2022-01-31T21:42:19.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:19 smithi171 conmon[41853]: debug 2022-01-31T21:42:19.062+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:19.063554+0000) 2022-01-31T21:42:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:19 smithi171 conmon[46715]: debug 2022-01-31T21:42:19.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:19.108726+0000) 2022-01-31T21:42:19.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:19 smithi167 conmon[54076]: debug 2022-01-31T21:42:19.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:19.337873+0000) 2022-01-31T21:42:19.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:19 smithi167 conmon[60316]: debug 2022-01-31T21:42:19.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:19.337833+0000) 2022-01-31T21:42:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:19 smithi167 conmon[49112]: debug 2022-01-31T21:42:19.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:19.673660+0000) 2022-01-31T21:42:20.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:19 smithi171 conmon[51620]: debug 2022-01-31T21:42:19.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:19.947117+0000) 2022-01-31T21:42:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:20 smithi171 conmon[41853]: debug 2022-01-31T21:42:20.062+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:20.063697+0000) 2022-01-31T21:42:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:20 smithi171 conmon[46715]: debug 2022-01-31T21:42:20.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:20.108903+0000) 2022-01-31T21:42:20.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:20 smithi167 conmon[60316]: debug 2022-01-31T21:42:20.337+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:20.338025+0000) 2022-01-31T21:42:20.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:20 smithi167 conmon[54076]: debug 2022-01-31T21:42:20.337+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:20.338049+0000) 2022-01-31T21:42:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:20 smithi167 conmon[49112]: debug 2022-01-31T21:42:20.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:20.673858+0000) 2022-01-31T21:42:21.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:20 smithi171 conmon[51620]: debug 2022-01-31T21:42:20.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:20.947317+0000) 2022-01-31T21:42:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:21 smithi171 conmon[41853]: debug 2022-01-31T21:42:21.063+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:21.063872+0000) 2022-01-31T21:42:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:21 smithi171 conmon[46715]: debug 2022-01-31T21:42:21.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:21.109052+0000) 2022-01-31T21:42:21.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:21 smithi167 conmon[54076]: debug 2022-01-31T21:42:21.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:21.338236+0000) 2022-01-31T21:42:21.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:21 smithi167 conmon[60316]: debug 2022-01-31T21:42:21.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:21.338251+0000) 2022-01-31T21:42:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:21 smithi167 conmon[49112]: debug 2022-01-31T21:42:21.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:21.674033+0000) 2022-01-31T21:42:22.018 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:21 smithi171 conmon[51620]: debug 2022-01-31T21:42:21.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:21.947498+0000) 2022-01-31T21:42:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:22 smithi171 conmon[41853]: debug 2022-01-31T21:42:22.063+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:22.063991+0000) 2022-01-31T21:42:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:22 smithi171 conmon[46715]: debug 2022-01-31T21:42:22.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:22.109269+0000) 2022-01-31T21:42:22.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:22 smithi167 conmon[54076]: debug 2022-01-31T21:42:22.336+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:22.338435+0000) 2022-01-31T21:42:22.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:22 smithi167 conmon[60316]: debug 2022-01-31T21:42:22.336+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:22.338443+0000) 2022-01-31T21:42:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:22 smithi167 conmon[49112]: debug 2022-01-31T21:42:22.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:22.674223+0000) 2022-01-31T21:42:23.018 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:22 smithi171 conmon[51620]: debug 2022-01-31T21:42:22.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:22.947659+0000) 2022-01-31T21:42:23.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[35325]: debug 2022-01-31T21:42:23.203+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151011 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:23.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[51620]: debug 2022-01-31T21:42:23.188+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.189314+0000) 2022-01-31T21:42:23.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[41853]: debug 2022-01-31T21:42:23.063+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.064217+0000) 2022-01-31T21:42:23.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[41853]: debug 2022-01-31T21:42:23.189+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.189790+0000) 2022-01-31T21:42:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[46715]: debug 2022-01-31T21:42:23.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.109409+0000) 2022-01-31T21:42:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[46715]: debug 2022-01-31T21:42:23.189+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.190458+0000) 2022-01-31T21:42:23.505 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:23 smithi167 conmon[49112]: debug 2022-01-31T21:42:23.186+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.187799+0000) 2022-01-31T21:42:23.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:23 smithi167 conmon[60316]: debug 2022-01-31T21:42:23.187+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.188659+0000) 2022-01-31T21:42:23.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:23 smithi167 conmon[60316]: debug 2022-01-31T21:42:23.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.338663+0000) 2022-01-31T21:42:23.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:23 smithi167 conmon[54076]: debug 2022-01-31T21:42:23.188+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.189693+0000) 2022-01-31T21:42:23.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:23 smithi167 conmon[54076]: debug 2022-01-31T21:42:23.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.338661+0000) 2022-01-31T21:42:23.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:23 smithi167 conmon[49112]: debug 2022-01-31T21:42:23.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.674420+0000) 2022-01-31T21:42:24.020 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[35325]: debug 2022-01-31T21:42:23.725+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:42:24.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:23 smithi171 conmon[51620]: debug 2022-01-31T21:42:23.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:23.947853+0000) 2022-01-31T21:42:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:24 smithi171 conmon[41853]: debug 2022-01-31T21:42:24.063+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:24.064401+0000) 2022-01-31T21:42:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:24 smithi171 conmon[46715]: debug 2022-01-31T21:42:24.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:24.109585+0000) 2022-01-31T21:42:24.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:24 smithi167 conmon[54076]: debug 2022-01-31T21:42:24.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:24.338826+0000) 2022-01-31T21:42:24.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:24 smithi167 conmon[60316]: debug 2022-01-31T21:42:24.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:24.338830+0000) 2022-01-31T21:42:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:24 smithi167 conmon[49112]: debug 2022-01-31T21:42:24.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:24.674586+0000) 2022-01-31T21:42:25.018 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:24 smithi171 conmon[51620]: debug 2022-01-31T21:42:24.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:24.948024+0000) 2022-01-31T21:42:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:25 smithi171 conmon[41853]: debug 2022-01-31T21:42:25.063+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:25.064551+0000) 2022-01-31T21:42:25.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:25 smithi171 conmon[46715]: debug 2022-01-31T21:42:25.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:25.109741+0000) 2022-01-31T21:42:25.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:25 smithi167 conmon[54076]: debug 2022-01-31T21:42:25.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:25.339044+0000) 2022-01-31T21:42:25.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:25 smithi167 conmon[60316]: debug 2022-01-31T21:42:25.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:25.339044+0000) 2022-01-31T21:42:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:25 smithi167 conmon[49112]: debug 2022-01-31T21:42:25.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:25.674769+0000) 2022-01-31T21:42:26.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:25 smithi171 conmon[51620]: debug 2022-01-31T21:42:25.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:25.948236+0000) 2022-01-31T21:42:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:26 smithi171 conmon[41853]: debug 2022-01-31T21:42:26.064+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:26.064708+0000) 2022-01-31T21:42:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:26 smithi171 conmon[46715]: debug 2022-01-31T21:42:26.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:26.109949+0000) 2022-01-31T21:42:26.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:26 smithi167 conmon[60316]: debug 2022-01-31T21:42:26.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:26.339246+0000) 2022-01-31T21:42:26.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:26 smithi167 conmon[54076]: debug 2022-01-31T21:42:26.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:26.339245+0000) 2022-01-31T21:42:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:26 smithi167 conmon[49112]: debug 2022-01-31T21:42:26.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:26.674932+0000) 2022-01-31T21:42:27.018 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:26 smithi171 conmon[51620]: debug 2022-01-31T21:42:26.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:26.948412+0000) 2022-01-31T21:42:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:27 smithi171 conmon[41853]: debug 2022-01-31T21:42:27.064+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:27.064854+0000) 2022-01-31T21:42:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:27 smithi171 conmon[46715]: debug 2022-01-31T21:42:27.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:27.110136+0000) 2022-01-31T21:42:27.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:27 smithi167 conmon[60316]: debug 2022-01-31T21:42:27.338+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:27.339432+0000) 2022-01-31T21:42:27.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:27 smithi167 conmon[54076]: debug 2022-01-31T21:42:27.338+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:27.339428+0000) 2022-01-31T21:42:27.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:27 smithi167 conmon[49112]: debug 2022-01-31T21:42:27.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:27.675136+0000) 2022-01-31T21:42:28.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:27 smithi171 conmon[51620]: debug 2022-01-31T21:42:27.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:27.948596+0000) 2022-01-31T21:42:28.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[35325]: debug 2022-01-31T21:42:28.221+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151124 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[46715]: debug 2022-01-31T21:42:28.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.110302+0000) 2022-01-31T21:42:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[46715]: debug 2022-01-31T21:42:28.207+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.207884+0000) 2022-01-31T21:42:28.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[51620]: debug 2022-01-31T21:42:28.205+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.206087+0000) 2022-01-31T21:42:28.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[41853]: debug 2022-01-31T21:42:28.064+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.064995+0000) 2022-01-31T21:42:28.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[41853]: debug 2022-01-31T21:42:28.206+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.207204+0000) 2022-01-31T21:42:28.514 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:28 smithi167 conmon[49112]: debug 2022-01-31T21:42:28.205+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.205686+0000) 2022-01-31T21:42:28.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:28 smithi167 conmon[60316]: debug 2022-01-31T21:42:28.205+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.205575+0000) 2022-01-31T21:42:28.515 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:28 smithi167 conmon[60316]: debug 2022-01-31T21:42:28.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.339595+0000) 2022-01-31T21:42:28.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:28 smithi167 conmon[54076]: debug 2022-01-31T21:42:28.206+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.207055+0000) 2022-01-31T21:42:28.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:28 smithi167 conmon[54076]: debug 2022-01-31T21:42:28.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.339602+0000) 2022-01-31T21:42:28.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:28 smithi167 conmon[49112]: debug 2022-01-31T21:42:28.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.675380+0000) 2022-01-31T21:42:29.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:28 smithi171 conmon[51620]: debug 2022-01-31T21:42:28.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:28.948799+0000) 2022-01-31T21:42:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:29 smithi171 conmon[41853]: debug 2022-01-31T21:42:29.064+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:29.065180+0000) 2022-01-31T21:42:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:29 smithi171 conmon[46715]: debug 2022-01-31T21:42:29.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:29.110525+0000) 2022-01-31T21:42:29.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:29 smithi167 conmon[60316]: debug 2022-01-31T21:42:29.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:29.339789+0000) 2022-01-31T21:42:29.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:29 smithi167 conmon[54076]: debug 2022-01-31T21:42:29.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:29.339789+0000) 2022-01-31T21:42:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:29 smithi167 conmon[49112]: debug 2022-01-31T21:42:29.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:29.675577+0000) 2022-01-31T21:42:30.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:29 smithi171 conmon[51620]: debug 2022-01-31T21:42:29.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:29.948920+0000) 2022-01-31T21:42:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:30 smithi171 conmon[46715]: debug 2022-01-31T21:42:30.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:30.110646+0000) 2022-01-31T21:42:30.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:30 smithi171 conmon[41853]: debug 2022-01-31T21:42:30.064+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:30.065309+0000) 2022-01-31T21:42:30.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:30 smithi167 conmon[60316]: debug 2022-01-31T21:42:30.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:30.340004+0000) 2022-01-31T21:42:30.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:30 smithi167 conmon[54076]: debug 2022-01-31T21:42:30.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:30.340003+0000) 2022-01-31T21:42:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:30 smithi167 conmon[49112]: debug 2022-01-31T21:42:30.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:30.675711+0000) 2022-01-31T21:42:31.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:30 smithi171 conmon[51620]: debug 2022-01-31T21:42:30.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:30.949134+0000) 2022-01-31T21:42:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:31 smithi171 conmon[41853]: debug 2022-01-31T21:42:31.064+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:31.065441+0000) 2022-01-31T21:42:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:31 smithi171 conmon[46715]: debug 2022-01-31T21:42:31.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:31.110796+0000) 2022-01-31T21:42:31.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:31 smithi167 conmon[60316]: debug 2022-01-31T21:42:31.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:31.340247+0000) 2022-01-31T21:42:31.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:31 smithi167 conmon[54076]: debug 2022-01-31T21:42:31.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:31.340151+0000) 2022-01-31T21:42:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:31 smithi167 conmon[49112]: debug 2022-01-31T21:42:31.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:31.675911+0000) 2022-01-31T21:42:32.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:31 smithi171 conmon[51620]: debug 2022-01-31T21:42:31.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:31.949349+0000) 2022-01-31T21:42:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:32 smithi171 conmon[41853]: debug 2022-01-31T21:42:32.065+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:32.065600+0000) 2022-01-31T21:42:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:32 smithi171 conmon[46715]: debug 2022-01-31T21:42:32.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:32.110999+0000) 2022-01-31T21:42:32.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:32 smithi167 conmon[60316]: debug 2022-01-31T21:42:32.339+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:32.340412+0000) 2022-01-31T21:42:32.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:32 smithi167 conmon[54076]: debug 2022-01-31T21:42:32.339+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:32.340341+0000) 2022-01-31T21:42:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:32 smithi167 conmon[49112]: debug 2022-01-31T21:42:32.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:32.676098+0000) 2022-01-31T21:42:33.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:32 smithi171 conmon[51620]: debug 2022-01-31T21:42:32.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:32.949537+0000) 2022-01-31T21:42:33.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[35325]: debug 2022-01-31T21:42:33.239+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151233 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:33.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[41853]: debug 2022-01-31T21:42:33.065+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.065802+0000) 2022-01-31T21:42:33.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[41853]: debug 2022-01-31T21:42:33.224+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.225223+0000) 2022-01-31T21:42:33.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[51620]: debug 2022-01-31T21:42:33.224+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.225000+0000) 2022-01-31T21:42:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[46715]: debug 2022-01-31T21:42:33.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.111128+0000) 2022-01-31T21:42:33.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[46715]: debug 2022-01-31T21:42:33.225+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.226101+0000) 2022-01-31T21:42:33.524 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:33 smithi167 conmon[49112]: debug 2022-01-31T21:42:33.223+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.224291+0000) 2022-01-31T21:42:33.525 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:33 smithi167 conmon[54076]: debug 2022-01-31T21:42:33.224+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.224811+0000) 2022-01-31T21:42:33.525 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:33 smithi167 conmon[54076]: debug 2022-01-31T21:42:33.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.340534+0000) 2022-01-31T21:42:33.526 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:33 smithi167 conmon[60316]: debug 2022-01-31T21:42:33.224+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.224574+0000) 2022-01-31T21:42:33.526 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:33 smithi167 conmon[60316]: debug 2022-01-31T21:42:33.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.340572+0000) 2022-01-31T21:42:33.803 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:33 smithi167 conmon[49112]: debug 2022-01-31T21:42:33.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.676281+0000) 2022-01-31T21:42:34.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:33 smithi171 conmon[51620]: debug 2022-01-31T21:42:33.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:33.949714+0000) 2022-01-31T21:42:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:34 smithi171 conmon[41853]: debug 2022-01-31T21:42:34.065+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:34.065988+0000) 2022-01-31T21:42:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:34 smithi171 conmon[46715]: debug 2022-01-31T21:42:34.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:34.111308+0000) 2022-01-31T21:42:34.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:34 smithi167 conmon[54076]: debug 2022-01-31T21:42:34.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:34.340673+0000) 2022-01-31T21:42:34.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:34 smithi167 conmon[60316]: debug 2022-01-31T21:42:34.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:34.340682+0000) 2022-01-31T21:42:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:34 smithi167 conmon[49112]: debug 2022-01-31T21:42:34.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:34.676425+0000) 2022-01-31T21:42:35.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:34 smithi171 conmon[51620]: debug 2022-01-31T21:42:34.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:34.949862+0000) 2022-01-31T21:42:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:35 smithi171 conmon[41853]: debug 2022-01-31T21:42:35.065+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:35.066090+0000) 2022-01-31T21:42:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:35 smithi171 conmon[46715]: debug 2022-01-31T21:42:35.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:35.111455+0000) 2022-01-31T21:42:35.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:35 smithi167 conmon[54076]: debug 2022-01-31T21:42:35.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:35.340877+0000) 2022-01-31T21:42:35.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:35 smithi167 conmon[60316]: debug 2022-01-31T21:42:35.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:35.340883+0000) 2022-01-31T21:42:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:35 smithi167 conmon[49112]: debug 2022-01-31T21:42:35.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:35.676609+0000) 2022-01-31T21:42:36.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:35 smithi171 conmon[51620]: debug 2022-01-31T21:42:35.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:35.950071+0000) 2022-01-31T21:42:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:36 smithi171 conmon[46715]: debug 2022-01-31T21:42:36.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:36.111593+0000) 2022-01-31T21:42:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:36 smithi171 conmon[41853]: debug 2022-01-31T21:42:36.065+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:36.066272+0000) 2022-01-31T21:42:36.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:36 smithi167 conmon[60316]: debug 2022-01-31T21:42:36.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:36.341087+0000) 2022-01-31T21:42:36.657 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:36 smithi167 conmon[54076]: debug 2022-01-31T21:42:36.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:36.341065+0000) 2022-01-31T21:42:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:36 smithi167 conmon[49112]: debug 2022-01-31T21:42:36.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:36.676761+0000) 2022-01-31T21:42:37.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:36 smithi171 conmon[51620]: debug 2022-01-31T21:42:36.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:36.950260+0000) 2022-01-31T21:42:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:37 smithi171 conmon[41853]: debug 2022-01-31T21:42:37.065+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:37.066507+0000) 2022-01-31T21:42:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:37 smithi171 conmon[46715]: debug 2022-01-31T21:42:37.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:37.111710+0000) 2022-01-31T21:42:37.655 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:37 smithi167 conmon[60316]: debug 2022-01-31T21:42:37.340+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:37.341264+0000) 2022-01-31T21:42:37.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:37 smithi167 conmon[54076]: debug 2022-01-31T21:42:37.340+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:37.341263+0000) 2022-01-31T21:42:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:37 smithi167 conmon[49112]: debug 2022-01-31T21:42:37.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:37.676958+0000) 2022-01-31T21:42:38.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:37 smithi171 conmon[51620]: debug 2022-01-31T21:42:37.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:37.950443+0000) 2022-01-31T21:42:38.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[35325]: debug 2022-01-31T21:42:38.257+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151347 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:38.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[51620]: debug 2022-01-31T21:42:38.241+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.242087+0000) 2022-01-31T21:42:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[46715]: debug 2022-01-31T21:42:38.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.111851+0000) 2022-01-31T21:42:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[46715]: debug 2022-01-31T21:42:38.243+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.243802+0000) 2022-01-31T21:42:38.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[41853]: debug 2022-01-31T21:42:38.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.066675+0000) 2022-01-31T21:42:38.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[41853]: debug 2022-01-31T21:42:38.242+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.243640+0000) 2022-01-31T21:42:38.533 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:38 smithi167 conmon[49112]: debug 2022-01-31T21:42:38.242+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.243029+0000) 2022-01-31T21:42:38.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:38 smithi167 conmon[54076]: debug 2022-01-31T21:42:38.241+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.241798+0000) 2022-01-31T21:42:38.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:38 smithi167 conmon[54076]: debug 2022-01-31T21:42:38.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.341457+0000) 2022-01-31T21:42:38.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:38 smithi167 conmon[60316]: debug 2022-01-31T21:42:38.241+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.241580+0000) 2022-01-31T21:42:38.536 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:38 smithi167 conmon[60316]: debug 2022-01-31T21:42:38.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.341479+0000) 2022-01-31T21:42:38.804 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:38 smithi167 conmon[49112]: debug 2022-01-31T21:42:38.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.677122+0000) 2022-01-31T21:42:39.021 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[35325]: debug 2022-01-31T21:42:38.726+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:42:39.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:38 smithi171 conmon[51620]: debug 2022-01-31T21:42:38.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:38.950649+0000) 2022-01-31T21:42:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:39 smithi171 conmon[41853]: debug 2022-01-31T21:42:39.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:39.066843+0000) 2022-01-31T21:42:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:39 smithi171 conmon[46715]: debug 2022-01-31T21:42:39.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:39.112021+0000) 2022-01-31T21:42:39.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:39 smithi167 conmon[54076]: debug 2022-01-31T21:42:39.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:39.341622+0000) 2022-01-31T21:42:39.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:39 smithi167 conmon[60316]: debug 2022-01-31T21:42:39.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:39.341624+0000) 2022-01-31T21:42:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:39 smithi167 conmon[49112]: debug 2022-01-31T21:42:39.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:39.677275+0000) 2022-01-31T21:42:40.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:39 smithi171 conmon[51620]: debug 2022-01-31T21:42:39.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:39.950804+0000) 2022-01-31T21:42:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:40 smithi171 conmon[41853]: debug 2022-01-31T21:42:40.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:40.066994+0000) 2022-01-31T21:42:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:40 smithi171 conmon[46715]: debug 2022-01-31T21:42:40.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:40.112173+0000) 2022-01-31T21:42:40.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:40 smithi167 conmon[54076]: debug 2022-01-31T21:42:40.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:40.341778+0000) 2022-01-31T21:42:40.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:40 smithi167 conmon[60316]: debug 2022-01-31T21:42:40.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:40.341784+0000) 2022-01-31T21:42:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:40 smithi167 conmon[49112]: debug 2022-01-31T21:42:40.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:40.677435+0000) 2022-01-31T21:42:41.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:40 smithi171 conmon[51620]: debug 2022-01-31T21:42:40.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:40.951005+0000) 2022-01-31T21:42:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:41 smithi171 conmon[41853]: debug 2022-01-31T21:42:41.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:41.067089+0000) 2022-01-31T21:42:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:41 smithi171 conmon[46715]: debug 2022-01-31T21:42:41.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:41.112336+0000) 2022-01-31T21:42:41.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:41 smithi167 conmon[54076]: debug 2022-01-31T21:42:41.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:41.341983+0000) 2022-01-31T21:42:41.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:41 smithi167 conmon[60316]: debug 2022-01-31T21:42:41.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:41.342008+0000) 2022-01-31T21:42:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:41 smithi167 conmon[49112]: debug 2022-01-31T21:42:41.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:41.677655+0000) 2022-01-31T21:42:42.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:41 smithi171 conmon[51620]: debug 2022-01-31T21:42:41.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:41.951205+0000) 2022-01-31T21:42:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:42 smithi171 conmon[41853]: debug 2022-01-31T21:42:42.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:42.067243+0000) 2022-01-31T21:42:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:42 smithi171 conmon[46715]: debug 2022-01-31T21:42:42.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:42.112540+0000) 2022-01-31T21:42:42.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:42 smithi167 conmon[54076]: debug 2022-01-31T21:42:42.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:42.342169+0000) 2022-01-31T21:42:42.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:42 smithi167 conmon[60316]: debug 2022-01-31T21:42:42.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:42.342188+0000) 2022-01-31T21:42:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:42 smithi167 conmon[49112]: debug 2022-01-31T21:42:42.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:42.677830+0000) 2022-01-31T21:42:43.258 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[41853]: debug 2022-01-31T21:42:43.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.067447+0000) 2022-01-31T21:42:43.259 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[46715]: debug 2022-01-31T21:42:43.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.112747+0000) 2022-01-31T21:42:43.259 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:42 smithi171 conmon[51620]: debug 2022-01-31T21:42:42.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:42.951382+0000) 2022-01-31T21:42:43.544 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:43 smithi167 conmon[49112]: debug 2022-01-31T21:42:43.260+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.261004+0000) 2022-01-31T21:42:43.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:43 smithi167 conmon[60316]: debug 2022-01-31T21:42:43.259+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.259835+0000) 2022-01-31T21:42:43.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:43 smithi167 conmon[60316]: debug 2022-01-31T21:42:43.341+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.342373+0000) 2022-01-31T21:42:43.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:43 smithi167 conmon[54076]: debug 2022-01-31T21:42:43.259+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.259743+0000) 2022-01-31T21:42:43.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:43 smithi167 conmon[54076]: debug 2022-01-31T21:42:43.341+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.342348+0000) 2022-01-31T21:42:43.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[41853]: debug 2022-01-31T21:42:43.260+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.261267+0000) 2022-01-31T21:42:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[46715]: debug 2022-01-31T21:42:43.261+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.261834+0000) 2022-01-31T21:42:43.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[51620]: debug 2022-01-31T21:42:43.260+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.261445+0000) 2022-01-31T21:42:43.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[35325]: debug 2022-01-31T21:42:43.274+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151455 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:43.805 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:43 smithi167 conmon[49112]: debug 2022-01-31T21:42:43.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.678015+0000) 2022-01-31T21:42:44.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:43 smithi171 conmon[51620]: debug 2022-01-31T21:42:43.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:43.951566+0000) 2022-01-31T21:42:44.351 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:44 smithi171 conmon[41853]: debug 2022-01-31T21:42:44.066+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:44.067579+0000) 2022-01-31T21:42:44.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:44 smithi171 conmon[46715]: debug 2022-01-31T21:42:44.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:44.112922+0000) 2022-01-31T21:42:44.657 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:44 smithi167 conmon[54076]: debug 2022-01-31T21:42:44.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:44.342553+0000) 2022-01-31T21:42:44.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:44 smithi167 conmon[60316]: debug 2022-01-31T21:42:44.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:44.342542+0000) 2022-01-31T21:42:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:44 smithi167 conmon[49112]: debug 2022-01-31T21:42:44.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:44.678195+0000) 2022-01-31T21:42:45.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:44 smithi171 conmon[51620]: debug 2022-01-31T21:42:44.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:44.951722+0000) 2022-01-31T21:42:45.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:45 smithi171 conmon[46715]: debug 2022-01-31T21:42:45.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:45.113015+0000) 2022-01-31T21:42:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:45 smithi171 conmon[41853]: debug 2022-01-31T21:42:45.067+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:45.067727+0000) 2022-01-31T21:42:45.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:45 smithi167 conmon[60316]: debug 2022-01-31T21:42:45.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:45.342718+0000) 2022-01-31T21:42:45.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:45 smithi167 conmon[54076]: debug 2022-01-31T21:42:45.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:45 smithi167 conmon[54076]: 2022-01-31T21:42:45.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:45.342737+0000) 2022-01-31T21:42:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:45 smithi167 conmon[49112]: debug 2022-01-31T21:42:45.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:45.678421+0000) 2022-01-31T21:42:46.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:45 smithi171 conmon[51620]: debug 2022-01-31T21:42:45.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:45.951934+0000) 2022-01-31T21:42:46.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:46 smithi171 conmon[41853]: debug 2022-01-31T21:42:46.067+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:46.067935+0000) 2022-01-31T21:42:46.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:46 smithi171 conmon[46715]: debug 2022-01-31T21:42:46.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:46.113118+0000) 2022-01-31T21:42:46.657 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:46 smithi167 conmon[54076]: debug 2022-01-31T21:42:46.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:46.342900+0000) 2022-01-31T21:42:46.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:46 smithi167 conmon[60316]: debug 2022-01-31T21:42:46.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:46.342901+0000) 2022-01-31T21:42:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:46 smithi167 conmon[49112]: debug 2022-01-31T21:42:46.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:46.678612+0000) 2022-01-31T21:42:47.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:46 smithi171 conmon[51620]: debug 2022-01-31T21:42:46.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:46.952135+0000) 2022-01-31T21:42:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:47 smithi171 conmon[41853]: debug 2022-01-31T21:42:47.067+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:47.068120+0000) 2022-01-31T21:42:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:47 smithi171 conmon[46715]: debug 2022-01-31T21:42:47.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:47.113268+0000) 2022-01-31T21:42:47.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:47 smithi167 conmon[54076]: debug 2022-01-31T21:42:47.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:47.343121+0000) 2022-01-31T21:42:47.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:47 smithi167 conmon[60316]: debug 2022-01-31T21:42:47.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:47.343141+0000) 2022-01-31T21:42:47.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:47 smithi167 conmon[49112]: debug 2022-01-31T21:42:47.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:47.678802+0000) 2022-01-31T21:42:48.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:47 smithi171 conmon[51620]: debug 2022-01-31T21:42:47.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:47.952302+0000) 2022-01-31T21:42:48.275 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[46715]: debug 2022-01-31T21:42:48.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.113417+0000) 2022-01-31T21:42:48.276 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[41853]: debug 2022-01-31T21:42:48.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.068271+0000) 2022-01-31T21:42:48.554 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:48 smithi167 conmon[49112]: debug 2022-01-31T21:42:48.276+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.277080+0000) 2022-01-31T21:42:48.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:48 smithi167 conmon[54076]: debug 2022-01-31T21:42:48.276+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.276671+0000) 2022-01-31T21:42:48.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:48 smithi167 conmon[54076]: debug 2022-01-31T21:42:48.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.343315+0000) 2022-01-31T21:42:48.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:48 smithi167 conmon[60316]: debug 2022-01-31T21:42:48.276+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.276748+0000) 2022-01-31T21:42:48.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:48 smithi167 conmon[60316]: debug 2022-01-31T21:42:48.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.343318+0000) 2022-01-31T21:42:48.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[35325]: debug 2022-01-31T21:42:48.293+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[41853]: debug 2022-01-31T21:42:48.279+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.279887+0000) 2022-01-31T21:42:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[46715]: debug 2022-01-31T21:42:48.277+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.278652+0000) 2022-01-31T21:42:48.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[51620]: debug 2022-01-31T21:42:48.277+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.278743+0000) 2022-01-31T21:42:48.806 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:48 smithi167 conmon[49112]: debug 2022-01-31T21:42:48.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.679015+0000) 2022-01-31T21:42:49.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:48 smithi171 conmon[51620]: debug 2022-01-31T21:42:48.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:48.952504+0000) 2022-01-31T21:42:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:49 smithi171 conmon[41853]: debug 2022-01-31T21:42:49.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:49.068425+0000) 2022-01-31T21:42:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:49 smithi171 conmon[46715]: debug 2022-01-31T21:42:49.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:49.113600+0000) 2022-01-31T21:42:49.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:49 smithi167 conmon[54076]: debug 2022-01-31T21:42:49.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:49.343486+0000) 2022-01-31T21:42:49.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:49 smithi167 conmon[60316]: debug 2022-01-31T21:42:49.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:49.343516+0000) 2022-01-31T21:42:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:49 smithi167 conmon[49112]: debug 2022-01-31T21:42:49.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:49.679198+0000) 2022-01-31T21:42:50.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:49 smithi171 conmon[51620]: debug 2022-01-31T21:42:49.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:49.952686+0000) 2022-01-31T21:42:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:50 smithi171 conmon[41853]: debug 2022-01-31T21:42:50.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:50.068578+0000) 2022-01-31T21:42:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:50 smithi171 conmon[46715]: debug 2022-01-31T21:42:50.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:50.113738+0000) 2022-01-31T21:42:50.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:50 smithi167 conmon[54076]: debug 2022-01-31T21:42:50.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:50.343710+0000) 2022-01-31T21:42:50.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:50 smithi167 conmon[60316]: debug 2022-01-31T21:42:50.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:50.343695+0000) 2022-01-31T21:42:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:50 smithi167 conmon[49112]: debug 2022-01-31T21:42:50.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:50.679366+0000) 2022-01-31T21:42:51.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:50 smithi171 conmon[51620]: debug 2022-01-31T21:42:50.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:50.952871+0000) 2022-01-31T21:42:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:51 smithi171 conmon[41853]: debug 2022-01-31T21:42:51.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:51.068781+0000) 2022-01-31T21:42:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:51 smithi171 conmon[46715]: debug 2022-01-31T21:42:51.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:51.113916+0000) 2022-01-31T21:42:51.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:51 smithi167 conmon[54076]: debug 2022-01-31T21:42:51.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:51.343846+0000) 2022-01-31T21:42:51.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:51 smithi167 conmon[60316]: debug 2022-01-31T21:42:51.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:51.343847+0000) 2022-01-31T21:42:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:51 smithi167 conmon[49112]: debug 2022-01-31T21:42:51.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:51.679574+0000) 2022-01-31T21:42:52.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:51 smithi171 conmon[51620]: debug 2022-01-31T21:42:51.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:51.953085+0000) 2022-01-31T21:42:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:52 smithi171 conmon[41853]: debug 2022-01-31T21:42:52.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:52.068933+0000) 2022-01-31T21:42:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:52 smithi171 conmon[46715]: debug 2022-01-31T21:42:52.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:52.114071+0000) 2022-01-31T21:42:52.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:52 smithi167 conmon[54076]: debug 2022-01-31T21:42:52.342+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:52.344016+0000) 2022-01-31T21:42:52.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:52 smithi167 conmon[60316]: debug 2022-01-31T21:42:52.342+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:52.344000+0000) 2022-01-31T21:42:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:52 smithi167 conmon[49112]: debug 2022-01-31T21:42:52.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:52.679799+0000) 2022-01-31T21:42:53.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:52 smithi171 conmon[51620]: debug 2022-01-31T21:42:52.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:52.953276+0000) 2022-01-31T21:42:53.294 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[41853]: debug 2022-01-31T21:42:53.069+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.069129+0000) 2022-01-31T21:42:53.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[46715]: debug 2022-01-31T21:42:53.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.114216+0000) 2022-01-31T21:42:53.563 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:53 smithi167 conmon[49112]: debug 2022-01-31T21:42:53.294+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.296137+0000) 2022-01-31T21:42:53.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:53 smithi167 conmon[54076]: debug 2022-01-31T21:42:53.294+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.296036+0000) 2022-01-31T21:42:53.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:53 smithi167 conmon[54076]: debug 2022-01-31T21:42:53.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.344242+0000) 2022-01-31T21:42:53.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:53 smithi167 conmon[60316]: debug 2022-01-31T21:42:53.295+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.296999+0000) 2022-01-31T21:42:53.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:53 smithi167 conmon[60316]: debug 2022-01-31T21:42:53.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.344244+0000) 2022-01-31T21:42:53.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[35325]: debug 2022-01-31T21:42:53.312+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151677 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[41853]: debug 2022-01-31T21:42:53.297+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.298219+0000) 2022-01-31T21:42:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[46715]: debug 2022-01-31T21:42:53.295+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.296468+0000) 2022-01-31T21:42:53.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[51620]: debug 2022-01-31T21:42:53.296+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.297555+0000) 2022-01-31T21:42:53.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:53 smithi167 conmon[49112]: debug 2022-01-31T21:42:53.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.679998+0000) 2022-01-31T21:42:54.023 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[35325]: debug 2022-01-31T21:42:53.727+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:42:54.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:53 smithi171 conmon[51620]: debug 2022-01-31T21:42:53.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:53.953462+0000) 2022-01-31T21:42:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:54 smithi171 conmon[46715]: debug 2022-01-31T21:42:54.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:54.114396+0000) 2022-01-31T21:42:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:54 smithi171 conmon[41853]: debug 2022-01-31T21:42:54.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:54.069333+0000) 2022-01-31T21:42:54.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:54 smithi167 conmon[54076]: debug 2022-01-31T21:42:54.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:54.344393+0000) 2022-01-31T21:42:54.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:54 smithi167 conmon[60316]: debug 2022-01-31T21:42:54.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:54.344416+0000) 2022-01-31T21:42:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:54 smithi167 conmon[49112]: debug 2022-01-31T21:42:54.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:54.680163+0000) 2022-01-31T21:42:55.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:54 smithi171 conmon[51620]: debug 2022-01-31T21:42:54.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:54.953644+0000) 2022-01-31T21:42:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:55 smithi171 conmon[46715]: debug 2022-01-31T21:42:55.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:55.114559+0000) 2022-01-31T21:42:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:55 smithi171 conmon[41853]: debug 2022-01-31T21:42:55.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:55.069486+0000) 2022-01-31T21:42:55.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:55 smithi167 conmon[54076]: debug 2022-01-31T21:42:55.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:55.344586+0000) 2022-01-31T21:42:55.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:55 smithi167 conmon[60316]: debug 2022-01-31T21:42:55.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:55.344624+0000) 2022-01-31T21:42:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:55 smithi167 conmon[49112]: debug 2022-01-31T21:42:55.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:55.680372+0000) 2022-01-31T21:42:56.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:55 smithi171 conmon[51620]: debug 2022-01-31T21:42:55.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:55.953799+0000) 2022-01-31T21:42:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:56 smithi171 conmon[46715]: debug 2022-01-31T21:42:56.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:56.114736+0000) 2022-01-31T21:42:56.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:56 smithi171 conmon[41853]: debug 2022-01-31T21:42:56.067+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:56.069711+0000) 2022-01-31T21:42:56.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:56 smithi167 conmon[54076]: debug 2022-01-31T21:42:56.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:56.344741+0000) 2022-01-31T21:42:56.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:56 smithi167 conmon[60316]: debug 2022-01-31T21:42:56.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:56.344818+0000) 2022-01-31T21:42:56.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:56 smithi167 conmon[49112]: debug 2022-01-31T21:42:56.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:56.680572+0000) 2022-01-31T21:42:57.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:56 smithi171 conmon[51620]: debug 2022-01-31T21:42:56.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:56.953922+0000) 2022-01-31T21:42:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:57 smithi171 conmon[41853]: debug 2022-01-31T21:42:57.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:57.069859+0000) 2022-01-31T21:42:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:57 smithi171 conmon[46715]: debug 2022-01-31T21:42:57.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:57.114933+0000) 2022-01-31T21:42:57.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:57 smithi167 conmon[54076]: debug 2022-01-31T21:42:57.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:57.344916+0000) 2022-01-31T21:42:57.660 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:57 smithi167 conmon[60316]: debug 2022-01-31T21:42:57.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:57.345011+0000) 2022-01-31T21:42:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:57 smithi167 conmon[49112]: debug 2022-01-31T21:42:57.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:57.680708+0000) 2022-01-31T21:42:58.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:57 smithi171 conmon[51620]: debug 2022-01-31T21:42:57.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:57.954110+0000) 2022-01-31T21:42:58.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[46715]: debug 2022-01-31T21:42:58.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.115066+0000) 2022-01-31T21:42:58.313 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[41853]: debug 2022-01-31T21:42:58.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.069987+0000) 2022-01-31T21:42:58.576 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:58 smithi167 conmon[49112]: debug 2022-01-31T21:42:58.315+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.316650+0000) 2022-01-31T21:42:58.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:58 smithi167 conmon[54076]: debug 2022-01-31T21:42:58.313+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.315148+0000) 2022-01-31T21:42:58.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:58 smithi167 conmon[54076]: debug 2022-01-31T21:42:58.343+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.345107+0000) 2022-01-31T21:42:58.577 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:58 smithi167 conmon[60316]: debug 2022-01-31T21:42:58.313+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.314794+0000) 2022-01-31T21:42:58.578 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:58 smithi167 conmon[60316]: debug 2022-01-31T21:42:58.343+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.345171+0000) 2022-01-31T21:42:58.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[35325]: debug 2022-01-31T21:42:58.330+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151790 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:42:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[41853]: debug 2022-01-31T21:42:58.314+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.315993+0000) 2022-01-31T21:42:58.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[46715]: debug 2022-01-31T21:42:58.313+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.314973+0000) 2022-01-31T21:42:58.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[51620]: debug 2022-01-31T21:42:58.314+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.316352+0000) 2022-01-31T21:42:58.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:58 smithi167 conmon[49112]: debug 2022-01-31T21:42:58.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.680916+0000) 2022-01-31T21:42:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:42:59 smithi171 conmon[46715]: debug 2022-01-31T21:42:59.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:59.115226+0000) 2022-01-31T21:42:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:42:59 smithi171 conmon[41853]: debug 2022-01-31T21:42:59.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:59.070142+0000) 2022-01-31T21:42:59.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:58 smithi171 conmon[51620]: debug 2022-01-31T21:42:58.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:58.954272+0000) 2022-01-31T21:42:59.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:42:59 smithi167 conmon[54076]: debug 2022-01-31T21:42:59.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:59.345269+0000) 2022-01-31T21:42:59.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:42:59 smithi167 conmon[60316]: debug 2022-01-31T21:42:59.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:59.345306+0000) 2022-01-31T21:42:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:42:59 smithi167 conmon[49112]: debug 2022-01-31T21:42:59.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:59.681101+0000) 2022-01-31T21:43:00.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:42:59 smithi171 conmon[51620]: debug 2022-01-31T21:42:59.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:42:59.954453+0000) 2022-01-31T21:43:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:00 smithi171 conmon[46715]: debug 2022-01-31T21:43:00.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:00.115360+0000) 2022-01-31T21:43:00.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:00 smithi171 conmon[41853]: debug 2022-01-31T21:43:00.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:00.070264+0000) 2022-01-31T21:43:00.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:00 smithi167 conmon[54076]: debug 2022-01-31T21:43:00.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:00.345469+0000) 2022-01-31T21:43:00.660 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:00 smithi167 conmon[60316]: debug 2022-01-31T21:43:00.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:00.345471+0000) 2022-01-31T21:43:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:00 smithi167 conmon[49112]: debug 2022-01-31T21:43:00.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:00.681332+0000) 2022-01-31T21:43:01.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:00 smithi171 conmon[51620]: debug 2022-01-31T21:43:00.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:00.954672+0000) 2022-01-31T21:43:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:01 smithi171 conmon[46715]: debug 2022-01-31T21:43:01.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:01.115589+0000) 2022-01-31T21:43:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:01 smithi171 conmon[41853]: debug 2022-01-31T21:43:01.068+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:01.070377+0000) 2022-01-31T21:43:01.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:01 smithi167 conmon[54076]: debug 2022-01-31T21:43:01.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:01.345645+0000) 2022-01-31T21:43:01.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:01 smithi167 conmon[60316]: debug 2022-01-31T21:43:01.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:01.345649+0000) 2022-01-31T21:43:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:01 smithi167 conmon[49112]: debug 2022-01-31T21:43:01.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:01.681519+0000) 2022-01-31T21:43:02.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:01 smithi171 conmon[51620]: debug 2022-01-31T21:43:01.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:01.954865+0000) 2022-01-31T21:43:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:02 smithi171 conmon[41853]: debug 2022-01-31T21:43:02.069+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:02.070530+0000) 2022-01-31T21:43:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:02 smithi171 conmon[46715]: debug 2022-01-31T21:43:02.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:02.115789+0000) 2022-01-31T21:43:02.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:02 smithi167 conmon[54076]: debug 2022-01-31T21:43:02.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:02.345801+0000) 2022-01-31T21:43:02.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:02 smithi167 conmon[60316]: debug 2022-01-31T21:43:02.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:02.345838+0000) 2022-01-31T21:43:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:02 smithi167 conmon[49112]: debug 2022-01-31T21:43:02.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:02.681703+0000) 2022-01-31T21:43:03.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:02 smithi171 conmon[51620]: debug 2022-01-31T21:43:02.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:02.955053+0000) 2022-01-31T21:43:03.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[46715]: debug 2022-01-31T21:43:03.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.115949+0000) 2022-01-31T21:43:03.332 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[41853]: debug 2022-01-31T21:43:03.069+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.070725+0000) 2022-01-31T21:43:03.582 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:03 smithi167 conmon[49112]: debug 2022-01-31T21:43:03.333+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.334610+0000) 2022-01-31T21:43:03.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:03 smithi167 conmon[54076]: debug 2022-01-31T21:43:03.332+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.333542+0000) 2022-01-31T21:43:03.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:03 smithi167 conmon[54076]: debug 2022-01-31T21:43:03.344+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.346002+0000) 2022-01-31T21:43:03.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:03 smithi167 conmon[60316]: debug 2022-01-31T21:43:03.331+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.332825+0000) 2022-01-31T21:43:03.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:03 smithi167 conmon[60316]: debug 2022-01-31T21:43:03.344+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.345991+0000) 2022-01-31T21:43:03.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[35325]: debug 2022-01-31T21:43:03.348+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 151898 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[41853]: debug 2022-01-31T21:43:03.333+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.335016+0000) 2022-01-31T21:43:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[46715]: debug 2022-01-31T21:43:03.332+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.334150+0000) 2022-01-31T21:43:03.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[51620]: debug 2022-01-31T21:43:03.334+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.335749+0000) 2022-01-31T21:43:03.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:03 smithi167 conmon[49112]: debug 2022-01-31T21:43:03.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.681893+0000) 2022-01-31T21:43:04.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:03 smithi171 conmon[51620]: debug 2022-01-31T21:43:03.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:03.955275+0000) 2022-01-31T21:43:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:04 smithi171 conmon[46715]: debug 2022-01-31T21:43:04.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:04.116136+0000) 2022-01-31T21:43:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:04 smithi171 conmon[41853]: debug 2022-01-31T21:43:04.070+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:04.070848+0000) 2022-01-31T21:43:04.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:04 smithi167 conmon[54076]: debug 2022-01-31T21:43:04.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:04.346174+0000) 2022-01-31T21:43:04.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:04 smithi167 conmon[60316]: debug 2022-01-31T21:43:04.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:04.346154+0000) 2022-01-31T21:43:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:04 smithi167 conmon[49112]: debug 2022-01-31T21:43:04.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:04.682151+0000) 2022-01-31T21:43:05.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:04 smithi171 conmon[51620]: debug 2022-01-31T21:43:04.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:04.955453+0000) 2022-01-31T21:43:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:05 smithi171 conmon[41853]: debug 2022-01-31T21:43:05.070+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:05.070987+0000) 2022-01-31T21:43:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:05 smithi171 conmon[46715]: debug 2022-01-31T21:43:05.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:05.116320+0000) 2022-01-31T21:43:05.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:05 smithi167 conmon[54076]: debug 2022-01-31T21:43:05.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:05.346343+0000) 2022-01-31T21:43:05.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:05 smithi167 conmon[60316]: debug 2022-01-31T21:43:05.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:05.346346+0000) 2022-01-31T21:43:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:05 smithi167 conmon[49112]: debug 2022-01-31T21:43:05.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:05.682331+0000) 2022-01-31T21:43:06.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:05 smithi171 conmon[51620]: debug 2022-01-31T21:43:05.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:05.955639+0000) 2022-01-31T21:43:06.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:06 smithi171 conmon[41853]: debug 2022-01-31T21:43:06.070+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:06.071144+0000) 2022-01-31T21:43:06.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:06 smithi171 conmon[46715]: debug 2022-01-31T21:43:06.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:06.116525+0000) 2022-01-31T21:43:06.661 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:06 smithi167 conmon[54076]: debug 2022-01-31T21:43:06.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:06.346555+0000) 2022-01-31T21:43:06.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:06 smithi167 conmon[60316]: debug 2022-01-31T21:43:06.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:06.346549+0000) 2022-01-31T21:43:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:06 smithi167 conmon[49112]: debug 2022-01-31T21:43:06.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:06.682479+0000) 2022-01-31T21:43:07.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:06 smithi171 conmon[51620]: debug 2022-01-31T21:43:06.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:06.955849+0000) 2022-01-31T21:43:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:07 smithi171 conmon[41853]: debug 2022-01-31T21:43:07.070+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:07.071288+0000) 2022-01-31T21:43:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:07 smithi171 conmon[46715]: debug 2022-01-31T21:43:07.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:07.116652+0000) 2022-01-31T21:43:07.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:07 smithi167 conmon[54076]: debug 2022-01-31T21:43:07.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:07.346684+0000) 2022-01-31T21:43:07.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:07 smithi167 conmon[60316]: debug 2022-01-31T21:43:07.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:07.346708+0000) 2022-01-31T21:43:07.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:07 smithi167 conmon[49112]: debug 2022-01-31T21:43:07.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:07.682683+0000) 2022-01-31T21:43:08.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:07 smithi171 conmon[51620]: debug 2022-01-31T21:43:07.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:07.956006+0000) 2022-01-31T21:43:08.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[46715]: debug 2022-01-31T21:43:08.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.116856+0000) 2022-01-31T21:43:08.351 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[41853]: debug 2022-01-31T21:43:08.070+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.071446+0000) 2022-01-31T21:43:08.595 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:08 smithi167 conmon[49112]: debug 2022-01-31T21:43:08.352+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.353604+0000) 2022-01-31T21:43:08.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:08 smithi167 conmon[54076]: debug 2022-01-31T21:43:08.346+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.346819+0000) 2022-01-31T21:43:08.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:08 smithi167 conmon[54076]: debug 2022-01-31T21:43:08.351+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.353154+0000) 2022-01-31T21:43:08.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:08 smithi167 conmon[60316]: debug 2022-01-31T21:43:08.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.346856+0000) 2022-01-31T21:43:08.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:08 smithi167 conmon[60316]: debug 2022-01-31T21:43:08.350+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.352325+0000) 2022-01-31T21:43:08.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[35325]: debug 2022-01-31T21:43:08.368+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152011 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[41853]: debug 2022-01-31T21:43:08.353+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.354280+0000) 2022-01-31T21:43:08.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[46715]: debug 2022-01-31T21:43:08.353+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.354635+0000) 2022-01-31T21:43:08.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[51620]: debug 2022-01-31T21:43:08.352+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.353031+0000) 2022-01-31T21:43:08.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:08 smithi167 conmon[49112]: debug 2022-01-31T21:43:08.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.682891+0000) 2022-01-31T21:43:09.026 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[35325]: debug 2022-01-31T21:43:08.727+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:43:09.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:08 smithi171 conmon[51620]: debug 2022-01-31T21:43:08.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:08.956192+0000) 2022-01-31T21:43:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:09 smithi171 conmon[41853]: debug 2022-01-31T21:43:09.070+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:09.071683+0000) 2022-01-31T21:43:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:09 smithi171 conmon[46715]: debug 2022-01-31T21:43:09.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:09.117032+0000) 2022-01-31T21:43:09.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:09 smithi167 conmon[54076]: debug 2022-01-31T21:43:09.346+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:09.347017+0000) 2022-01-31T21:43:09.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:09 smithi167 conmon[60316]: debug 2022-01-31T21:43:09.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:09.347014+0000) 2022-01-31T21:43:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:09 smithi167 conmon[49112]: debug 2022-01-31T21:43:09.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:09.683032+0000) 2022-01-31T21:43:10.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:09 smithi171 conmon[51620]: debug 2022-01-31T21:43:09.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:09.956341+0000) 2022-01-31T21:43:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:10 smithi171 conmon[41853]: debug 2022-01-31T21:43:10.071+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:10.071819+0000) 2022-01-31T21:43:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:10 smithi171 conmon[46715]: debug 2022-01-31T21:43:10.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:10.117146+0000) 2022-01-31T21:43:10.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:10 smithi167 conmon[54076]: debug 2022-01-31T21:43:10.345+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:10.347202+0000) 2022-01-31T21:43:10.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:10 smithi167 conmon[60316]: debug 2022-01-31T21:43:10.345+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:10.347193+0000) 2022-01-31T21:43:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:10 smithi167 conmon[49112]: debug 2022-01-31T21:43:10.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:10.683220+0000) 2022-01-31T21:43:11.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:10 smithi171 conmon[51620]: debug 2022-01-31T21:43:10.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:10.956529+0000) 2022-01-31T21:43:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:11 smithi171 conmon[41853]: debug 2022-01-31T21:43:11.071+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:11.072008+0000) 2022-01-31T21:43:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:11 smithi171 conmon[46715]: debug 2022-01-31T21:43:11.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:11.117322+0000) 2022-01-31T21:43:11.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:11 smithi167 conmon[60316]: debug 2022-01-31T21:43:11.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:11.347409+0000) 2022-01-31T21:43:11.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:11 smithi167 conmon[54076]: debug 2022-01-31T21:43:11.346+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:11.347401+0000) 2022-01-31T21:43:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:11 smithi167 conmon[49112]: debug 2022-01-31T21:43:11.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:11.683432+0000) 2022-01-31T21:43:12.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:11 smithi171 conmon[51620]: debug 2022-01-31T21:43:11.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:11.956726+0000) 2022-01-31T21:43:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:12 smithi171 conmon[41853]: debug 2022-01-31T21:43:12.071+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:12.072188+0000) 2022-01-31T21:43:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:12 smithi171 conmon[46715]: debug 2022-01-31T21:43:12.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:12.117462+0000) 2022-01-31T21:43:12.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:12 smithi167 conmon[60316]: debug 2022-01-31T21:43:12.346+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:12.347607+0000) 2022-01-31T21:43:12.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:12 smithi167 conmon[54076]: debug 2022-01-31T21:43:12.346+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:12.347553+0000) 2022-01-31T21:43:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:12 smithi167 conmon[49112]: debug 2022-01-31T21:43:12.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:12.683599+0000) 2022-01-31T21:43:13.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:12 smithi171 conmon[51620]: debug 2022-01-31T21:43:12.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:12.956973+0000) 2022-01-31T21:43:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[41853]: debug 2022-01-31T21:43:13.071+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.072415+0000) 2022-01-31T21:43:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[46715]: debug 2022-01-31T21:43:13.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.117695+0000) 2022-01-31T21:43:13.605 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:13 smithi167 conmon[49112]: debug 2022-01-31T21:43:13.370+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.371261+0000) 2022-01-31T21:43:13.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:13 smithi167 conmon[54076]: debug 2022-01-31T21:43:13.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.347744+0000) 2022-01-31T21:43:13.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:13 smithi167 conmon[54076]: debug 2022-01-31T21:43:13.377+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.377585+0000) 2022-01-31T21:43:13.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:13 smithi167 conmon[60316]: debug 2022-01-31T21:43:13.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.347812+0000) 2022-01-31T21:43:13.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:13 smithi167 conmon[60316]: debug 2022-01-31T21:43:13.370+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.371083+0000) 2022-01-31T21:43:13.773 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[35325]: debug 2022-01-31T21:43:13.392+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152120 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:13.774 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[41853]: debug 2022-01-31T21:43:13.369+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.372996+0000) 2022-01-31T21:43:13.775 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[46715]: debug 2022-01-31T21:43:13.369+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.372423+0000) 2022-01-31T21:43:13.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[51620]: debug 2022-01-31T21:43:13.369+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.372166+0000) 2022-01-31T21:43:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:13 smithi167 conmon[49112]: debug 2022-01-31T21:43:13.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.683813+0000) 2022-01-31T21:43:14.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:13 smithi171 conmon[51620]: debug 2022-01-31T21:43:13.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:13.957164+0000) 2022-01-31T21:43:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:14 smithi171 conmon[46715]: debug 2022-01-31T21:43:14.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:14.117893+0000) 2022-01-31T21:43:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:14 smithi171 conmon[41853]: debug 2022-01-31T21:43:14.071+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:14.072611+0000) 2022-01-31T21:43:14.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:14 smithi167 conmon[54076]: debug 2022-01-31T21:43:14.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:14.347912+0000) 2022-01-31T21:43:14.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:14 smithi167 conmon[60316]: debug 2022-01-31T21:43:14.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:14.347991+0000) 2022-01-31T21:43:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:14 smithi167 conmon[49112]: debug 2022-01-31T21:43:14.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:14.683979+0000) 2022-01-31T21:43:15.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:14 smithi171 conmon[51620]: debug 2022-01-31T21:43:14.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:14.957335+0000) 2022-01-31T21:43:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:15 smithi171 conmon[41853]: debug 2022-01-31T21:43:15.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:15.072764+0000) 2022-01-31T21:43:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:15 smithi171 conmon[46715]: debug 2022-01-31T21:43:15.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:15.118043+0000) 2022-01-31T21:43:15.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:15 smithi167 conmon[60316]: debug 2022-01-31T21:43:15.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:15.348138+0000) 2022-01-31T21:43:15.613 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:15 smithi167 conmon[54076]: debug 2022-01-31T21:43:15.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:15.348084+0000) 2022-01-31T21:43:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:15 smithi167 conmon[49112]: debug 2022-01-31T21:43:15.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:15.684171+0000) 2022-01-31T21:43:16.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:15 smithi171 conmon[51620]: debug 2022-01-31T21:43:15.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:15.957498+0000) 2022-01-31T21:43:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:16 smithi171 conmon[41853]: debug 2022-01-31T21:43:16.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:16.072924+0000) 2022-01-31T21:43:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:16 smithi171 conmon[46715]: debug 2022-01-31T21:43:16.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:16.118198+0000) 2022-01-31T21:43:16.614 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:16 smithi167 conmon[54076]: debug 2022-01-31T21:43:16.347+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:16.348304+0000) 2022-01-31T21:43:16.615 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:16 smithi167 conmon[60316]: debug 2022-01-31T21:43:16.347+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:16.348309+0000) 2022-01-31T21:43:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:16 smithi167 conmon[49112]: debug 2022-01-31T21:43:16.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:16.684372+0000) 2022-01-31T21:43:17.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:16 smithi171 conmon[51620]: debug 2022-01-31T21:43:16.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:16.957697+0000) 2022-01-31T21:43:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:17 smithi171 conmon[41853]: debug 2022-01-31T21:43:17.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:17.073111+0000) 2022-01-31T21:43:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:17 smithi171 conmon[46715]: debug 2022-01-31T21:43:17.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:17.118382+0000) 2022-01-31T21:43:17.615 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:17 smithi167 conmon[54076]: debug 2022-01-31T21:43:17.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:17.348491+0000) 2022-01-31T21:43:17.615 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:17 smithi167 conmon[60316]: debug 2022-01-31T21:43:17.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:17.348525+0000) 2022-01-31T21:43:17.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:17 smithi167 conmon[49112]: debug 2022-01-31T21:43:17.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:17.684546+0000) 2022-01-31T21:43:18.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:17 smithi171 conmon[51620]: debug 2022-01-31T21:43:17.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:17.957908+0000) 2022-01-31T21:43:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[41853]: debug 2022-01-31T21:43:18.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.073245+0000) 2022-01-31T21:43:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[46715]: debug 2022-01-31T21:43:18.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.118559+0000) 2022-01-31T21:43:18.615 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:18 smithi167 conmon[49112]: debug 2022-01-31T21:43:18.402+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.402898+0000) 2022-01-31T21:43:18.616 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:18 smithi167 conmon[54076]: debug 2022-01-31T21:43:18.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.348695+0000) 2022-01-31T21:43:18.616 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:18 smithi167 conmon[54076]: debug 2022-01-31T21:43:18.395+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.395907+0000) 2022-01-31T21:43:18.617 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:18 smithi167 conmon[60316]: debug 2022-01-31T21:43:18.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.348694+0000) 2022-01-31T21:43:18.617 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:18 smithi167 conmon[60316]: debug 2022-01-31T21:43:18.394+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.395387+0000) 2022-01-31T21:43:18.773 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[46715]: debug 2022-01-31T21:43:18.394+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.395494+0000) 2022-01-31T21:43:18.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[51620]: debug 2022-01-31T21:43:18.395+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.396127+0000) 2022-01-31T21:43:18.775 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[35325]: debug 2022-01-31T21:43:18.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152235 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:18.775 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[41853]: debug 2022-01-31T21:43:18.396+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.396890+0000) 2022-01-31T21:43:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:18 smithi167 conmon[49112]: debug 2022-01-31T21:43:18.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.684770+0000) 2022-01-31T21:43:19.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:18 smithi171 conmon[51620]: debug 2022-01-31T21:43:18.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:18.958075+0000) 2022-01-31T21:43:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:19 smithi171 conmon[46715]: debug 2022-01-31T21:43:19.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:19.118682+0000) 2022-01-31T21:43:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:19 smithi171 conmon[41853]: debug 2022-01-31T21:43:19.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:19.073434+0000) 2022-01-31T21:43:19.618 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:19 smithi167 conmon[54076]: debug 2022-01-31T21:43:19.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:19.348889+0000) 2022-01-31T21:43:19.619 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:19 smithi167 conmon[60316]: debug 2022-01-31T21:43:19.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:19.348887+0000) 2022-01-31T21:43:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:19 smithi167 conmon[49112]: debug 2022-01-31T21:43:19.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:19.684913+0000) 2022-01-31T21:43:20.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:19 smithi171 conmon[51620]: debug 2022-01-31T21:43:19.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:19.958237+0000) 2022-01-31T21:43:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:20 smithi171 conmon[46715]: debug 2022-01-31T21:43:20.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:20.118777+0000) 2022-01-31T21:43:20.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:20 smithi171 conmon[41853]: debug 2022-01-31T21:43:20.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:20.073553+0000) 2022-01-31T21:43:20.622 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:20 smithi167 conmon[54076]: debug 2022-01-31T21:43:20.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:20.349075+0000) 2022-01-31T21:43:20.623 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:20 smithi167 conmon[60316]: debug 2022-01-31T21:43:20.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:20.349096+0000) 2022-01-31T21:43:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:20 smithi167 conmon[49112]: debug 2022-01-31T21:43:20.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:20.685082+0000) 2022-01-31T21:43:21.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:20 smithi171 conmon[51620]: debug 2022-01-31T21:43:20.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:20.958426+0000) 2022-01-31T21:43:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:21 smithi171 conmon[41853]: debug 2022-01-31T21:43:21.072+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:21.073724+0000) 2022-01-31T21:43:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:21 smithi171 conmon[46715]: debug 2022-01-31T21:43:21.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:21.118950+0000) 2022-01-31T21:43:21.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:21 smithi167 conmon[60316]: debug 2022-01-31T21:43:21.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:21.349263+0000) 2022-01-31T21:43:21.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:21 smithi167 conmon[54076]: debug 2022-01-31T21:43:21.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:21.349227+0000) 2022-01-31T21:43:21.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:21 smithi167 conmon[49112]: debug 2022-01-31T21:43:21.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:21.685249+0000) 2022-01-31T21:43:22.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:21 smithi171 conmon[51620]: debug 2022-01-31T21:43:21.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:21.958610+0000) 2022-01-31T21:43:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:22 smithi171 conmon[46715]: debug 2022-01-31T21:43:22.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:22.119108+0000) 2022-01-31T21:43:22.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:22 smithi171 conmon[41853]: debug 2022-01-31T21:43:22.073+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:22.073931+0000) 2022-01-31T21:43:22.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:22 smithi167 conmon[54076]: debug 2022-01-31T21:43:22.348+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:22.349402+0000) 2022-01-31T21:43:22.626 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:22 smithi167 conmon[60316]: debug 2022-01-31T21:43:22.348+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:22.349444+0000) 2022-01-31T21:43:22.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:22 smithi167 conmon[49112]: debug 2022-01-31T21:43:22.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:22.685459+0000) 2022-01-31T21:43:23.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:22 smithi171 conmon[51620]: debug 2022-01-31T21:43:22.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:22.958775+0000) 2022-01-31T21:43:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[46715]: debug 2022-01-31T21:43:23.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.119286+0000) 2022-01-31T21:43:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[41853]: debug 2022-01-31T21:43:23.073+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.074163+0000) 2022-01-31T21:43:23.625 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:23 smithi167 conmon[49112]: debug 2022-01-31T21:43:23.415+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.415491+0000) 2022-01-31T21:43:23.626 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:23 smithi167 conmon[54076]: debug 2022-01-31T21:43:23.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.349580+0000) 2022-01-31T21:43:23.626 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:23 smithi167 conmon[54076]: debug 2022-01-31T21:43:23.414+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.414858+0000) 2022-01-31T21:43:23.627 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:23 smithi167 conmon[60316]: debug 2022-01-31T21:43:23.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.349601+0000) 2022-01-31T21:43:23.627 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:23 smithi167 conmon[60316]: debug 2022-01-31T21:43:23.413+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.414218+0000) 2022-01-31T21:43:23.727 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[35325]: debug 2022-01-31T21:43:23.429+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152343 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:23.727 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[41853]: debug 2022-01-31T21:43:23.414+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.414817+0000) 2022-01-31T21:43:23.728 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[46715]: debug 2022-01-31T21:43:23.413+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.414598+0000) 2022-01-31T21:43:23.729 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[51620]: debug 2022-01-31T21:43:23.413+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.414447+0000) 2022-01-31T21:43:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:23 smithi167 conmon[49112]: debug 2022-01-31T21:43:23.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.685643+0000) 2022-01-31T21:43:24.028 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[35325]: debug 2022-01-31T21:43:23.728+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:43:24.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:23 smithi171 conmon[51620]: debug 2022-01-31T21:43:23.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:23.958895+0000) 2022-01-31T21:43:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:24 smithi171 conmon[41853]: debug 2022-01-31T21:43:24.073+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:24.074339+0000) 2022-01-31T21:43:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:24 smithi171 conmon[46715]: debug 2022-01-31T21:43:24.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:24.119471+0000) 2022-01-31T21:43:24.628 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:24 smithi167 conmon[54076]: debug 2022-01-31T21:43:24.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:24.349768+0000) 2022-01-31T21:43:24.629 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:24 smithi167 conmon[60316]: debug 2022-01-31T21:43:24.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:24.349769+0000) 2022-01-31T21:43:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:24 smithi167 conmon[49112]: debug 2022-01-31T21:43:24.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:24.685801+0000) 2022-01-31T21:43:25.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:24 smithi171 conmon[51620]: debug 2022-01-31T21:43:24.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:24.959049+0000) 2022-01-31T21:43:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:25 smithi171 conmon[41853]: debug 2022-01-31T21:43:25.074+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:25.074494+0000) 2022-01-31T21:43:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:25 smithi171 conmon[46715]: debug 2022-01-31T21:43:25.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:25.119624+0000) 2022-01-31T21:43:25.634 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:25 smithi167 conmon[54076]: debug 2022-01-31T21:43:25.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:25.349933+0000) 2022-01-31T21:43:25.635 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:25 smithi167 conmon[60316]: debug 2022-01-31T21:43:25.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:25.349931+0000) 2022-01-31T21:43:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:25 smithi167 conmon[49112]: debug 2022-01-31T21:43:25.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:25.685997+0000) 2022-01-31T21:43:26.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:25 smithi171 conmon[51620]: debug 2022-01-31T21:43:25.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:25.959277+0000) 2022-01-31T21:43:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:26 smithi171 conmon[41853]: debug 2022-01-31T21:43:26.073+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:26.074686+0000) 2022-01-31T21:43:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:26 smithi171 conmon[46715]: debug 2022-01-31T21:43:26.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:26.119772+0000) 2022-01-31T21:43:26.634 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:26 smithi167 conmon[60316]: debug 2022-01-31T21:43:26.349+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:26.350120+0000) 2022-01-31T21:43:26.635 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:26 smithi167 conmon[54076]: debug 2022-01-31T21:43:26.349+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:26.350136+0000) 2022-01-31T21:43:26.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:26 smithi167 conmon[49112]: debug 2022-01-31T21:43:26.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:26.686211+0000) 2022-01-31T21:43:27.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:26 smithi171 conmon[51620]: debug 2022-01-31T21:43:26.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:26.959420+0000) 2022-01-31T21:43:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:27 smithi171 conmon[41853]: debug 2022-01-31T21:43:27.074+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:27.074875+0000) 2022-01-31T21:43:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:27 smithi171 conmon[46715]: debug 2022-01-31T21:43:27.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:27.119919+0000) 2022-01-31T21:43:27.634 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:27 smithi167 conmon[54076]: debug 2022-01-31T21:43:27.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:27.350330+0000) 2022-01-31T21:43:27.635 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:27 smithi167 conmon[60316]: debug 2022-01-31T21:43:27.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:27.350324+0000) 2022-01-31T21:43:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:27 smithi167 conmon[49112]: debug 2022-01-31T21:43:27.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:27.686395+0000) 2022-01-31T21:43:28.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:27 smithi171 conmon[51620]: debug 2022-01-31T21:43:27.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:27.959593+0000) 2022-01-31T21:43:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[41853]: debug 2022-01-31T21:43:28.074+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.075054+0000) 2022-01-31T21:43:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[46715]: debug 2022-01-31T21:43:28.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.120069+0000) 2022-01-31T21:43:28.635 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:28 smithi167 conmon[49112]: debug 2022-01-31T21:43:28.432+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.432708+0000) 2022-01-31T21:43:28.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:28 smithi167 conmon[54076]: debug 2022-01-31T21:43:28.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.350547+0000) 2022-01-31T21:43:28.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:28 smithi167 conmon[54076]: debug 2022-01-31T21:43:28.432+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.433289+0000) 2022-01-31T21:43:28.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:28 smithi167 conmon[60316]: debug 2022-01-31T21:43:28.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.350545+0000) 2022-01-31T21:43:28.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:28 smithi167 conmon[60316]: debug 2022-01-31T21:43:28.432+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.433172+0000) 2022-01-31T21:43:28.776 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[35325]: debug 2022-01-31T21:43:28.448+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152469 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:28.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[41853]: debug 2022-01-31T21:43:28.431+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.432635+0000) 2022-01-31T21:43:28.777 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[46715]: debug 2022-01-31T21:43:28.433+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.433985+0000) 2022-01-31T21:43:28.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[51620]: debug 2022-01-31T21:43:28.432+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.433065+0000) 2022-01-31T21:43:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:28 smithi167 conmon[49112]: debug 2022-01-31T21:43:28.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.686577+0000) 2022-01-31T21:43:29.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:28 smithi171 conmon[51620]: debug 2022-01-31T21:43:28.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:28.959803+0000) 2022-01-31T21:43:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:29 smithi171 conmon[46715]: debug 2022-01-31T21:43:29.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:29.120223+0000) 2022-01-31T21:43:29.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:29 smithi171 conmon[41853]: debug 2022-01-31T21:43:29.074+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:29.075256+0000) 2022-01-31T21:43:29.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:29 smithi167 conmon[54076]: debug 2022-01-31T21:43:29.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:29.350743+0000) 2022-01-31T21:43:29.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:29 smithi167 conmon[60316]: debug 2022-01-31T21:43:29.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:29.350740+0000) 2022-01-31T21:43:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:29 smithi167 conmon[49112]: debug 2022-01-31T21:43:29.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:29.686744+0000) 2022-01-31T21:43:30.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:29 smithi171 conmon[51620]: debug 2022-01-31T21:43:29.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:29.959953+0000) 2022-01-31T21:43:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:30 smithi171 conmon[41853]: debug 2022-01-31T21:43:30.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:30.075421+0000) 2022-01-31T21:43:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:30 smithi171 conmon[46715]: debug 2022-01-31T21:43:30.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:30.120411+0000) 2022-01-31T21:43:30.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:30 smithi167 conmon[54076]: debug 2022-01-31T21:43:30.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:30.350919+0000) 2022-01-31T21:43:30.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:30 smithi167 conmon[60316]: debug 2022-01-31T21:43:30.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:30.350890+0000) 2022-01-31T21:43:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:30 smithi167 conmon[49112]: debug 2022-01-31T21:43:30.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:30.686979+0000) 2022-01-31T21:43:31.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:30 smithi171 conmon[51620]: debug 2022-01-31T21:43:30.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:30.960157+0000) 2022-01-31T21:43:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:31 smithi171 conmon[41853]: debug 2022-01-31T21:43:31.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:31.075579+0000) 2022-01-31T21:43:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:31 smithi171 conmon[46715]: debug 2022-01-31T21:43:31.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:31.120621+0000) 2022-01-31T21:43:31.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:31 smithi167 conmon[54076]: debug 2022-01-31T21:43:31.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:31.351123+0000) 2022-01-31T21:43:31.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:31 smithi167 conmon[60316]: debug 2022-01-31T21:43:31.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:31.351103+0000) 2022-01-31T21:43:31.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:31 smithi167 conmon[49112]: debug 2022-01-31T21:43:31.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:31.687185+0000) 2022-01-31T21:43:32.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:31 smithi171 conmon[51620]: debug 2022-01-31T21:43:31.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:31.960358+0000) 2022-01-31T21:43:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:32 smithi171 conmon[41853]: debug 2022-01-31T21:43:32.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:32.075736+0000) 2022-01-31T21:43:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:32 smithi171 conmon[46715]: debug 2022-01-31T21:43:32.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:32.120841+0000) 2022-01-31T21:43:32.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:32 smithi167 conmon[54076]: debug 2022-01-31T21:43:32.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:32.351244+0000) 2022-01-31T21:43:32.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:32 smithi167 conmon[60316]: debug 2022-01-31T21:43:32.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:32.351267+0000) 2022-01-31T21:43:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:32 smithi167 conmon[49112]: debug 2022-01-31T21:43:32.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:32.687400+0000) 2022-01-31T21:43:33.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:32 smithi171 conmon[51620]: debug 2022-01-31T21:43:32.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:32.960566+0000) 2022-01-31T21:43:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[41853]: debug 2022-01-31T21:43:33.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.075936+0000) 2022-01-31T21:43:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[46715]: debug 2022-01-31T21:43:33.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.120985+0000) 2022-01-31T21:43:33.645 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:33 smithi167 conmon[49112]: debug 2022-01-31T21:43:33.450+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.451207+0000) 2022-01-31T21:43:33.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:33 smithi167 conmon[54076]: debug 2022-01-31T21:43:33.350+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.351441+0000) 2022-01-31T21:43:33.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:33 smithi167 conmon[54076]: debug 2022-01-31T21:43:33.450+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.451478+0000) 2022-01-31T21:43:33.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:33 smithi167 conmon[60316]: debug 2022-01-31T21:43:33.350+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.351441+0000) 2022-01-31T21:43:33.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:33 smithi167 conmon[60316]: debug 2022-01-31T21:43:33.451+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.452012+0000) 2022-01-31T21:43:33.776 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[35325]: debug 2022-01-31T21:43:33.467+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152577 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:33.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[41853]: debug 2022-01-31T21:43:33.451+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.452065+0000) 2022-01-31T21:43:33.778 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[46715]: debug 2022-01-31T21:43:33.452+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.452894+0000) 2022-01-31T21:43:33.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[51620]: debug 2022-01-31T21:43:33.452+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.453155+0000) 2022-01-31T21:43:33.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:33 smithi167 conmon[49112]: debug 2022-01-31T21:43:33.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.687581+0000) 2022-01-31T21:43:34.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:33 smithi171 conmon[51620]: debug 2022-01-31T21:43:33.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:33.960743+0000) 2022-01-31T21:43:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:34 smithi171 conmon[41853]: debug 2022-01-31T21:43:34.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:34.076110+0000) 2022-01-31T21:43:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:34 smithi171 conmon[46715]: debug 2022-01-31T21:43:34.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:34.121148+0000) 2022-01-31T21:43:34.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:34 smithi167 conmon[54076]: debug 2022-01-31T21:43:34.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:34.351685+0000) 2022-01-31T21:43:34.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:34 smithi167 conmon[60316]: debug 2022-01-31T21:43:34.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:34.351685+0000) 2022-01-31T21:43:34.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:34 smithi167 conmon[49112]: debug 2022-01-31T21:43:34.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:34.687717+0000) 2022-01-31T21:43:35.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:34 smithi171 conmon[51620]: debug 2022-01-31T21:43:34.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:34.960865+0000) 2022-01-31T21:43:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:35 smithi171 conmon[46715]: debug 2022-01-31T21:43:35.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:35.121321+0000) 2022-01-31T21:43:35.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:35 smithi171 conmon[41853]: debug 2022-01-31T21:43:35.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:35.076252+0000) 2022-01-31T21:43:35.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:35 smithi167 conmon[54076]: debug 2022-01-31T21:43:35.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:35.351853+0000) 2022-01-31T21:43:35.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:35 smithi167 conmon[60316]: debug 2022-01-31T21:43:35.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:35.351862+0000) 2022-01-31T21:43:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:35 smithi167 conmon[49112]: debug 2022-01-31T21:43:35.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:35.687898+0000) 2022-01-31T21:43:36.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:35 smithi171 conmon[51620]: debug 2022-01-31T21:43:35.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:35.961015+0000) 2022-01-31T21:43:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:36 smithi171 conmon[46715]: debug 2022-01-31T21:43:36.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:36.121507+0000) 2022-01-31T21:43:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:36 smithi171 conmon[41853]: debug 2022-01-31T21:43:36.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:36.076398+0000) 2022-01-31T21:43:36.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:36 smithi167 conmon[54076]: debug 2022-01-31T21:43:36.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:36.352077+0000) 2022-01-31T21:43:36.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:36 smithi167 conmon[60316]: debug 2022-01-31T21:43:36.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:36.352079+0000) 2022-01-31T21:43:36.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:36 smithi167 conmon[49112]: debug 2022-01-31T21:43:36.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:36.688083+0000) 2022-01-31T21:43:37.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:36 smithi171 conmon[51620]: debug 2022-01-31T21:43:36.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:36.961153+0000) 2022-01-31T21:43:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:37 smithi171 conmon[41853]: debug 2022-01-31T21:43:37.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:37.076556+0000) 2022-01-31T21:43:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:37 smithi171 conmon[46715]: debug 2022-01-31T21:43:37.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:37.121653+0000) 2022-01-31T21:43:37.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:37 smithi167 conmon[54076]: debug 2022-01-31T21:43:37.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:37.352297+0000) 2022-01-31T21:43:37.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:37 smithi167 conmon[60316]: debug 2022-01-31T21:43:37.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:37.352302+0000) 2022-01-31T21:43:37.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:37 smithi167 conmon[49112]: debug 2022-01-31T21:43:37.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:37.688270+0000) 2022-01-31T21:43:38.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:37 smithi171 conmon[51620]: debug 2022-01-31T21:43:37.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:37.961329+0000) 2022-01-31T21:43:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[46715]: debug 2022-01-31T21:43:38.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.121817+0000) 2022-01-31T21:43:38.359 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[41853]: debug 2022-01-31T21:43:38.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.076730+0000) 2022-01-31T21:43:38.656 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:38 smithi167 conmon[49112]: debug 2022-01-31T21:43:38.468+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.470094+0000) 2022-01-31T21:43:38.657 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:38 smithi167 conmon[54076]: debug 2022-01-31T21:43:38.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.352519+0000) 2022-01-31T21:43:38.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:38 smithi167 conmon[54076]: debug 2022-01-31T21:43:38.468+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.470199+0000) 2022-01-31T21:43:38.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:38 smithi167 conmon[60316]: debug 2022-01-31T21:43:38.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.352536+0000) 2022-01-31T21:43:38.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:38 smithi167 conmon[60316]: debug 2022-01-31T21:43:38.468+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.470412+0000) 2022-01-31T21:43:38.728 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[35325]: debug 2022-01-31T21:43:38.485+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152691 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:38.728 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[41853]: debug 2022-01-31T21:43:38.469+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.470359+0000) 2022-01-31T21:43:38.729 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[46715]: debug 2022-01-31T21:43:38.469+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.470674+0000) 2022-01-31T21:43:38.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[51620]: debug 2022-01-31T21:43:38.470+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.471436+0000) 2022-01-31T21:43:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:38 smithi167 conmon[49112]: debug 2022-01-31T21:43:38.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.688475+0000) 2022-01-31T21:43:39.031 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[35325]: debug 2022-01-31T21:43:38.729+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:43:39.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:38 smithi171 conmon[51620]: debug 2022-01-31T21:43:38.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:38.961528+0000) 2022-01-31T21:43:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:39 smithi171 conmon[41853]: debug 2022-01-31T21:43:39.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:39.076936+0000) 2022-01-31T21:43:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:39 smithi171 conmon[46715]: debug 2022-01-31T21:43:39.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:39.121961+0000) 2022-01-31T21:43:39.660 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:39 smithi167 conmon[60316]: debug 2022-01-31T21:43:39.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:39.352721+0000) 2022-01-31T21:43:39.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:39 smithi167 conmon[54076]: debug 2022-01-31T21:43:39.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:39.352712+0000) 2022-01-31T21:43:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:39 smithi167 conmon[49112]: debug 2022-01-31T21:43:39.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:39.688617+0000) 2022-01-31T21:43:40.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:39 smithi171 conmon[51620]: debug 2022-01-31T21:43:39.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:39.961705+0000) 2022-01-31T21:43:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:40 smithi171 conmon[41853]: debug 2022-01-31T21:43:40.077+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:40.077089+0000) 2022-01-31T21:43:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:40 smithi171 conmon[46715]: debug 2022-01-31T21:43:40.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:40.122109+0000) 2022-01-31T21:43:40.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:40 smithi167 conmon[60316]: debug 2022-01-31T21:43:40.351+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:40.352823+0000) 2022-01-31T21:43:40.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:40 smithi167 conmon[54076]: debug 2022-01-31T21:43:40.351+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:40.352929+0000) 2022-01-31T21:43:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:40 smithi167 conmon[49112]: debug 2022-01-31T21:43:40.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:40.688846+0000) 2022-01-31T21:43:41.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:40 smithi171 conmon[51620]: debug 2022-01-31T21:43:40.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:40.961912+0000) 2022-01-31T21:43:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:41 smithi171 conmon[41853]: debug 2022-01-31T21:43:41.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:41.077282+0000) 2022-01-31T21:43:41.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:41 smithi171 conmon[46715]: debug 2022-01-31T21:43:41.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:41.122230+0000) 2022-01-31T21:43:41.665 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:41 smithi167 conmon[54076]: debug 2022-01-31T21:43:41.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:41.353154+0000) 2022-01-31T21:43:41.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:41 smithi167 conmon[60316]: debug 2022-01-31T21:43:41.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:41.353003+0000) 2022-01-31T21:43:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:41 smithi167 conmon[49112]: debug 2022-01-31T21:43:41.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:41.689005+0000) 2022-01-31T21:43:42.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:41 smithi171 conmon[51620]: debug 2022-01-31T21:43:41.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:41.962082+0000) 2022-01-31T21:43:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:42 smithi171 conmon[41853]: debug 2022-01-31T21:43:42.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:42.077458+0000) 2022-01-31T21:43:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:42 smithi171 conmon[46715]: debug 2022-01-31T21:43:42.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:42.122381+0000) 2022-01-31T21:43:42.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:42 smithi167 conmon[54076]: debug 2022-01-31T21:43:42.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:42.353278+0000) 2022-01-31T21:43:42.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:42 smithi167 conmon[60316]: debug 2022-01-31T21:43:42.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:42.353187+0000) 2022-01-31T21:43:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:42 smithi167 conmon[49112]: debug 2022-01-31T21:43:42.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:42.689205+0000) 2022-01-31T21:43:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[41853]: debug 2022-01-31T21:43:43.075+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.077616+0000) 2022-01-31T21:43:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[46715]: debug 2022-01-31T21:43:43.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.122572+0000) 2022-01-31T21:43:43.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:42 smithi171 conmon[51620]: debug 2022-01-31T21:43:42.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:42.962290+0000) 2022-01-31T21:43:43.666 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[49112]: debug 2022-01-31T21:43:43.487+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.488891+0000) 2022-01-31T21:43:43.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[60316]: debug 2022-01-31T21:43:43.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.353376+0000) 2022-01-31T21:43:43.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[60316]: debug 2022-01-31T21:43:43.487+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.489013+0000) 2022-01-31T21:43:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[54076]: debug 2022-01-31T21:43:43.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.353448+0000) 2022-01-31T21:43:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[54076]: debug 2022-01-31T21:43:43.486+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.488339+0000) 2022-01-31T21:43:43.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[35325]: debug 2022-01-31T21:43:43.502+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152799 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:43.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[41853]: debug 2022-01-31T21:43:43.487+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.488836+0000) 2022-01-31T21:43:43.780 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[46715]: debug 2022-01-31T21:43:43.487+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.489089+0000) 2022-01-31T21:43:43.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[51620]: debug 2022-01-31T21:43:43.488+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.489869+0000) 2022-01-31T21:43:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[49112]: debug 2022-01-31T21:43:43.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.689425+0000) 2022-01-31T21:43:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:43 smithi167 conmon[49112]: 2022-01-31T21:43:44.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:43 smithi171 conmon[51620]: debug 2022-01-31T21:43:43.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:43.962511+0000) 2022-01-31T21:43:44.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:44 smithi171 conmon[46715]: debug 2022-01-31T21:43:44.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:44.122779+0000) 2022-01-31T21:43:44.321 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:44 smithi171 conmon[41853]: debug 2022-01-31T21:43:44.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:44.077787+0000) 2022-01-31T21:43:44.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:44 smithi167 conmon[60316]: debug 2022-01-31T21:43:44.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:44.353558+0000) 2022-01-31T21:43:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:44 smithi167 conmon[54076]: debug 2022-01-31T21:43:44.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:44.353637+0000) 2022-01-31T21:43:44.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:44 smithi167 conmon[49112]: debug 2022-01-31T21:43:44.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:44.689583+0000) 2022-01-31T21:43:45.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:44 smithi171 conmon[51620]: debug 2022-01-31T21:43:44.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:44.962691+0000) 2022-01-31T21:43:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:45 smithi171 conmon[41853]: debug 2022-01-31T21:43:45.077+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:45.077975+0000) 2022-01-31T21:43:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:45 smithi171 conmon[46715]: debug 2022-01-31T21:43:45.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:45.122939+0000) 2022-01-31T21:43:45.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:45 smithi167 conmon[60316]: debug 2022-01-31T21:43:45.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:45.353711+0000) 2022-01-31T21:43:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:45 smithi167 conmon[54076]: debug 2022-01-31T21:43:45.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:45.353731+0000) 2022-01-31T21:43:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:45 smithi167 conmon[49112]: debug 2022-01-31T21:43:45.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:45.689767+0000) 2022-01-31T21:43:46.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:45 smithi171 conmon[51620]: debug 2022-01-31T21:43:45.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:45.962940+0000) 2022-01-31T21:43:46.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:46 smithi171 conmon[46715]: debug 2022-01-31T21:43:46.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:46.123125+0000) 2022-01-31T21:43:46.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:46 smithi171 conmon[41853]: debug 2022-01-31T21:43:46.077+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:46.078182+0000) 2022-01-31T21:43:46.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:46 smithi167 conmon[54076]: debug 2022-01-31T21:43:46.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:46.353881+0000) 2022-01-31T21:43:46.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:46 smithi167 conmon[60316]: debug 2022-01-31T21:43:46.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:46.353876+0000) 2022-01-31T21:43:46.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:46 smithi167 conmon[49112]: debug 2022-01-31T21:43:46.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:46.689973+0000) 2022-01-31T21:43:47.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:46 smithi171 conmon[51620]: debug 2022-01-31T21:43:46.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:46.963150+0000) 2022-01-31T21:43:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:47 smithi171 conmon[41853]: debug 2022-01-31T21:43:47.076+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:47.078398+0000) 2022-01-31T21:43:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:47 smithi171 conmon[46715]: debug 2022-01-31T21:43:47.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:47.123323+0000) 2022-01-31T21:43:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:47 smithi167 conmon[54076]: debug 2022-01-31T21:43:47.352+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:47.354064+0000) 2022-01-31T21:43:47.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:47 smithi167 conmon[60316]: debug 2022-01-31T21:43:47.352+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:47.354064+0000) 2022-01-31T21:43:48.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:47 smithi171 conmon[51620]: debug 2022-01-31T21:43:47.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:47.963346+0000) 2022-01-31T21:43:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:47 smithi167 conmon[49112]: debug 2022-01-31T21:43:47.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:47.690167+0000) 2022-01-31T21:43:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[41853]: debug 2022-01-31T21:43:48.077+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.078624+0000) 2022-01-31T21:43:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[46715]: debug 2022-01-31T21:43:48.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.123501+0000) 2022-01-31T21:43:48.590 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:48 smithi167 conmon[49112]: debug 2022-01-31T21:43:48.504+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.505977+0000) 2022-01-31T21:43:48.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:48 smithi167 conmon[54076]: debug 2022-01-31T21:43:48.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.354226+0000) 2022-01-31T21:43:48.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:48 smithi167 conmon[54076]: debug 2022-01-31T21:43:48.504+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.506154+0000) 2022-01-31T21:43:48.592 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:48 smithi167 conmon[60316]: debug 2022-01-31T21:43:48.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.354226+0000) 2022-01-31T21:43:48.593 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:48 smithi167 conmon[60316]: debug 2022-01-31T21:43:48.504+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.506053+0000) 2022-01-31T21:43:48.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[35325]: debug 2022-01-31T21:43:48.521+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 152912 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:48.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[41853]: debug 2022-01-31T21:43:48.504+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.506330+0000) 2022-01-31T21:43:48.781 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[46715]: debug 2022-01-31T21:43:48.504+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.506429+0000) 2022-01-31T21:43:48.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[51620]: debug 2022-01-31T21:43:48.505+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.507654+0000) 2022-01-31T21:43:48.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:48 smithi167 conmon[49112]: debug 2022-01-31T21:43:48.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.690361+0000) 2022-01-31T21:43:49.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:48 smithi171 conmon[51620]: debug 2022-01-31T21:43:48.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:48.963497+0000) 2022-01-31T21:43:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:49 smithi171 conmon[41853]: debug 2022-01-31T21:43:49.077+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:49.078735+0000) 2022-01-31T21:43:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:49 smithi171 conmon[46715]: debug 2022-01-31T21:43:49.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:49.123629+0000) 2022-01-31T21:43:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:49 smithi167 conmon[54076]: debug 2022-01-31T21:43:49.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:49.354436+0000) 2022-01-31T21:43:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:49 smithi167 conmon[60316]: debug 2022-01-31T21:43:49.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:49.354437+0000) 2022-01-31T21:43:50.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:49 smithi171 conmon[51620]: debug 2022-01-31T21:43:49.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:49.963648+0000) 2022-01-31T21:43:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:49 smithi167 conmon[49112]: debug 2022-01-31T21:43:49.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:49.690547+0000) 2022-01-31T21:43:50.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:50 smithi171 conmon[41853]: debug 2022-01-31T21:43:50.077+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:50.078861+0000) 2022-01-31T21:43:50.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:50 smithi171 conmon[46715]: debug 2022-01-31T21:43:50.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:50.123716+0000) 2022-01-31T21:43:50.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:50 smithi171 conmon[46715]: 2022-01-31T21:43:50.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:50 smithi167 conmon[54076]: debug 2022-01-31T21:43:50.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:50.354587+0000) 2022-01-31T21:43:50.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:50 smithi167 conmon[60316]: debug 2022-01-31T21:43:50.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:50.354586+0000) 2022-01-31T21:43:51.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:50 smithi171 conmon[51620]: debug 2022-01-31T21:43:50.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:50.963855+0000) 2022-01-31T21:43:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:50 smithi167 conmon[49112]: debug 2022-01-31T21:43:50.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:50.690695+0000) 2022-01-31T21:43:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:51 smithi171 conmon[41853]: debug 2022-01-31T21:43:51.078+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:51.079033+0000) 2022-01-31T21:43:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:51 smithi171 conmon[46715]: debug 2022-01-31T21:43:51.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:51.123911+0000) 2022-01-31T21:43:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:51 smithi167 conmon[54076]: debug 2022-01-31T21:43:51.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:51.354809+0000) 2022-01-31T21:43:51.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:51 smithi167 conmon[60316]: debug 2022-01-31T21:43:51.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:51.354807+0000) 2022-01-31T21:43:52.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:51 smithi171 conmon[51620]: debug 2022-01-31T21:43:51.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:51.964052+0000) 2022-01-31T21:43:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:51 smithi167 conmon[49112]: debug 2022-01-31T21:43:51.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:51.690872+0000) 2022-01-31T21:43:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:52 smithi171 conmon[41853]: debug 2022-01-31T21:43:52.078+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:52.079203+0000) 2022-01-31T21:43:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:52 smithi171 conmon[46715]: debug 2022-01-31T21:43:52.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:52.124110+0000) 2022-01-31T21:43:52.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:52 smithi167 conmon[54076]: debug 2022-01-31T21:43:52.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:52.354973+0000) 2022-01-31T21:43:52.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:52 smithi167 conmon[60316]: debug 2022-01-31T21:43:52.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:52.354973+0000) 2022-01-31T21:43:53.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:52 smithi171 conmon[51620]: debug 2022-01-31T21:43:52.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:52.964198+0000) 2022-01-31T21:43:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:52 smithi167 conmon[49112]: debug 2022-01-31T21:43:52.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:52.691088+0000) 2022-01-31T21:43:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[41853]: debug 2022-01-31T21:43:53.078+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.079374+0000) 2022-01-31T21:43:53.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[46715]: debug 2022-01-31T21:43:53.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.124309+0000) 2022-01-31T21:43:53.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:53 smithi167 conmon[49112]: debug 2022-01-31T21:43:53.523+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.525186+0000) 2022-01-31T21:43:53.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:53 smithi167 conmon[54076]: debug 2022-01-31T21:43:53.353+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.355186+0000) 2022-01-31T21:43:53.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:53 smithi167 conmon[54076]: debug 2022-01-31T21:43:53.523+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.525433+0000) 2022-01-31T21:43:53.707 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:53 smithi167 conmon[60316]: debug 2022-01-31T21:43:53.353+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.355186+0000) 2022-01-31T21:43:53.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:53 smithi167 conmon[60316]: debug 2022-01-31T21:43:53.524+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.525750+0000) 2022-01-31T21:43:53.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[41853]: debug 2022-01-31T21:43:53.525+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.526068+0000) 2022-01-31T21:43:53.781 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[46715]: debug 2022-01-31T21:43:53.525+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.525829+0000) 2022-01-31T21:43:53.782 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[35325]: debug 2022-01-31T21:43:53.541+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153021 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:53.782 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[35325]: debug 2022-01-31T21:43:53.729+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:43:53.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[51620]: debug 2022-01-31T21:43:53.524+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.525615+0000) 2022-01-31T21:43:54.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:53 smithi171 conmon[51620]: debug 2022-01-31T21:43:53.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.964394+0000) 2022-01-31T21:43:54.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:53 smithi167 conmon[49112]: debug 2022-01-31T21:43:53.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:53.691203+0000) 2022-01-31T21:43:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:54 smithi171 conmon[41853]: debug 2022-01-31T21:43:54.078+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:54.079505+0000) 2022-01-31T21:43:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:54 smithi171 conmon[46715]: debug 2022-01-31T21:43:54.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:54.124515+0000) 2022-01-31T21:43:54.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:54 smithi167 conmon[54076]: debug 2022-01-31T21:43:54.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:54.355355+0000) 2022-01-31T21:43:54.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:54 smithi167 conmon[60316]: debug 2022-01-31T21:43:54.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:54.355355+0000) 2022-01-31T21:43:55.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:54 smithi171 conmon[51620]: debug 2022-01-31T21:43:54.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:54.964567+0000) 2022-01-31T21:43:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:54 smithi167 conmon[49112]: debug 2022-01-31T21:43:54.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:54.691389+0000) 2022-01-31T21:43:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:55 smithi171 conmon[41853]: debug 2022-01-31T21:43:55.078+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:55.079666+0000) 2022-01-31T21:43:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:55 smithi171 conmon[46715]: debug 2022-01-31T21:43:55.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:55.124669+0000) 2022-01-31T21:43:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:55 smithi167 conmon[54076]: debug 2022-01-31T21:43:55.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:55.355547+0000) 2022-01-31T21:43:55.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:55 smithi167 conmon[60316]: debug 2022-01-31T21:43:55.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:55.355547+0000) 2022-01-31T21:43:56.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:55 smithi171 conmon[51620]: debug 2022-01-31T21:43:55.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:55.964747+0000) 2022-01-31T21:43:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:55 smithi167 conmon[49112]: debug 2022-01-31T21:43:55.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:55.691630+0000) 2022-01-31T21:43:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:56 smithi171 conmon[41853]: debug 2022-01-31T21:43:56.079+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:56.079868+0000) 2022-01-31T21:43:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:56 smithi171 conmon[46715]: debug 2022-01-31T21:43:56.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:56.124855+0000) 2022-01-31T21:43:56.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:56 smithi167 conmon[54076]: debug 2022-01-31T21:43:56.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:56.355784+0000) 2022-01-31T21:43:56.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:56 smithi167 conmon[60316]: debug 2022-01-31T21:43:56.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:56.355784+0000) 2022-01-31T21:43:57.034 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:56 smithi171 conmon[51620]: debug 2022-01-31T21:43:56.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:56.964889+0000) 2022-01-31T21:43:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:56 smithi167 conmon[49112]: debug 2022-01-31T21:43:56.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:56.691810+0000) 2022-01-31T21:43:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:57 smithi171 conmon[41853]: debug 2022-01-31T21:43:57.079+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:57.080054+0000) 2022-01-31T21:43:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:57 smithi171 conmon[46715]: debug 2022-01-31T21:43:57.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:57.124984+0000) 2022-01-31T21:43:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:57 smithi167 conmon[54076]: debug 2022-01-31T21:43:57.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:57.355991+0000) 2022-01-31T21:43:57.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:57 smithi167 conmon[60316]: debug 2022-01-31T21:43:57.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:57.355990+0000) 2022-01-31T21:43:58.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:57 smithi171 conmon[51620]: debug 2022-01-31T21:43:57.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:57.965064+0000) 2022-01-31T21:43:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:57 smithi167 conmon[49112]: debug 2022-01-31T21:43:57.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:57.692004+0000) 2022-01-31T21:43:58.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[41853]: debug 2022-01-31T21:43:58.079+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.080251+0000) 2022-01-31T21:43:58.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[46715]: debug 2022-01-31T21:43:58.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.125124+0000) 2022-01-31T21:43:58.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:58 smithi167 conmon[49112]: debug 2022-01-31T21:43:58.544+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.546215+0000) 2022-01-31T21:43:58.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:58 smithi167 conmon[54076]: debug 2022-01-31T21:43:58.355+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.356201+0000) 2022-01-31T21:43:58.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:58 smithi167 conmon[54076]: debug 2022-01-31T21:43:58.544+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.546147+0000) 2022-01-31T21:43:58.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:58 smithi167 conmon[60316]: debug 2022-01-31T21:43:58.355+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.356200+0000) 2022-01-31T21:43:58.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:58 smithi167 conmon[60316]: debug 2022-01-31T21:43:58.544+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.546042+0000) 2022-01-31T21:43:58.781 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[35325]: debug 2022-01-31T21:43:58.562+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153134 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:43:58.782 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[41853]: debug 2022-01-31T21:43:58.545+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.546485+0000) 2022-01-31T21:43:58.783 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[46715]: debug 2022-01-31T21:43:58.546+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.547694+0000) 2022-01-31T21:43:58.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[51620]: debug 2022-01-31T21:43:58.544+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.545254+0000) 2022-01-31T21:43:59.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:58 smithi171 conmon[51620]: debug 2022-01-31T21:43:58.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.965251+0000) 2022-01-31T21:43:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:58 smithi167 conmon[49112]: debug 2022-01-31T21:43:58.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:58.692235+0000) 2022-01-31T21:43:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:43:59 smithi171 conmon[41853]: debug 2022-01-31T21:43:59.079+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:59.080439+0000) 2022-01-31T21:43:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:43:59 smithi171 conmon[46715]: debug 2022-01-31T21:43:59.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:59.125296+0000) 2022-01-31T21:43:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:43:59 smithi167 conmon[54076]: debug 2022-01-31T21:43:59.354+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:59.356423+0000) 2022-01-31T21:43:59.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:43:59 smithi167 conmon[60316]: debug 2022-01-31T21:43:59.354+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:59.356425+0000) 2022-01-31T21:44:00.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:43:59 smithi171 conmon[51620]: debug 2022-01-31T21:43:59.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:59.965411+0000) 2022-01-31T21:44:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:43:59 smithi167 conmon[49112]: debug 2022-01-31T21:43:59.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:43:59.692430+0000) 2022-01-31T21:44:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:00 smithi171 conmon[41853]: debug 2022-01-31T21:44:00.079+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:00.080576+0000) 2022-01-31T21:44:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:00 smithi171 conmon[46715]: debug 2022-01-31T21:44:00.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:00.125446+0000) 2022-01-31T21:44:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:00 smithi167 conmon[54076]: debug 2022-01-31T21:44:00.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:00.356558+0000) 2022-01-31T21:44:00.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:00 smithi167 conmon[60316]: debug 2022-01-31T21:44:00.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:00.356558+0000) 2022-01-31T21:44:01.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:00 smithi171 conmon[51620]: debug 2022-01-31T21:44:00.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:00.965611+0000) 2022-01-31T21:44:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:00 smithi167 conmon[49112]: debug 2022-01-31T21:44:00.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:00.692645+0000) 2022-01-31T21:44:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:01 smithi171 conmon[41853]: debug 2022-01-31T21:44:01.080+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:01.080716+0000) 2022-01-31T21:44:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:01 smithi171 conmon[46715]: debug 2022-01-31T21:44:01.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:01.125552+0000) 2022-01-31T21:44:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:01 smithi167 conmon[54076]: debug 2022-01-31T21:44:01.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:01.356692+0000) 2022-01-31T21:44:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:01 smithi167 conmon[60316]: debug 2022-01-31T21:44:01.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:01.356718+0000) 2022-01-31T21:44:02.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:01 smithi171 conmon[51620]: debug 2022-01-31T21:44:01.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:01.965807+0000) 2022-01-31T21:44:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:01 smithi167 conmon[49112]: debug 2022-01-31T21:44:01.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:01.692860+0000) 2022-01-31T21:44:02.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:02 smithi171 conmon[41853]: debug 2022-01-31T21:44:02.080+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:02.080907+0000) 2022-01-31T21:44:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:02 smithi171 conmon[46715]: debug 2022-01-31T21:44:02.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:02.125784+0000) 2022-01-31T21:44:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:02 smithi167 conmon[54076]: debug 2022-01-31T21:44:02.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:02.356927+0000) 2022-01-31T21:44:02.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:02 smithi167 conmon[60316]: debug 2022-01-31T21:44:02.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:02.356927+0000) 2022-01-31T21:44:03.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:02 smithi171 conmon[51620]: debug 2022-01-31T21:44:02.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:02.965978+0000) 2022-01-31T21:44:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:02 smithi167 conmon[49112]: debug 2022-01-31T21:44:02.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:02.693063+0000) 2022-01-31T21:44:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[41853]: debug 2022-01-31T21:44:03.080+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.081068+0000) 2022-01-31T21:44:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[46715]: debug 2022-01-31T21:44:03.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.125981+0000) 2022-01-31T21:44:03.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:03 smithi167 conmon[49112]: debug 2022-01-31T21:44:03.565+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.565627+0000) 2022-01-31T21:44:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:03 smithi167 conmon[54076]: debug 2022-01-31T21:44:03.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.357151+0000) 2022-01-31T21:44:03.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:03 smithi167 conmon[54076]: debug 2022-01-31T21:44:03.564+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.564912+0000) 2022-01-31T21:44:03.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:03 smithi167 conmon[60316]: debug 2022-01-31T21:44:03.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.357151+0000) 2022-01-31T21:44:03.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:03 smithi167 conmon[60316]: debug 2022-01-31T21:44:03.565+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.565769+0000) 2022-01-31T21:44:03.855 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[51620]: debug 2022-01-31T21:44:03.564+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.565315+0000) 2022-01-31T21:44:03.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[35325]: debug 2022-01-31T21:44:03.580+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153243 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[41853]: debug 2022-01-31T21:44:03.566+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.566774+0000) 2022-01-31T21:44:03.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[46715]: debug 2022-01-31T21:44:03.564+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.565453+0000) 2022-01-31T21:44:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:03 smithi167 conmon[49112]: debug 2022-01-31T21:44:03.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.693264+0000) 2022-01-31T21:44:04.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:04 smithi171 conmon[46715]: debug 2022-01-31T21:44:04.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:04.126160+0000) 2022-01-31T21:44:04.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:03 smithi171 conmon[51620]: debug 2022-01-31T21:44:03.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:03.966142+0000) 2022-01-31T21:44:04.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:04 smithi171 conmon[41853]: debug 2022-01-31T21:44:04.080+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:04.081240+0000) 2022-01-31T21:44:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:04 smithi167 conmon[54076]: debug 2022-01-31T21:44:04.356+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:04.357323+0000) 2022-01-31T21:44:04.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:04 smithi167 conmon[60316]: debug 2022-01-31T21:44:04.356+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:04.357322+0000) 2022-01-31T21:44:05.036 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:04 smithi171 conmon[51620]: debug 2022-01-31T21:44:04.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:04.966321+0000) 2022-01-31T21:44:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:04 smithi167 conmon[49112]: debug 2022-01-31T21:44:04.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:04.693440+0000) 2022-01-31T21:44:05.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:05 smithi171 conmon[41853]: debug 2022-01-31T21:44:05.080+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:05.081405+0000) 2022-01-31T21:44:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:05 smithi171 conmon[46715]: debug 2022-01-31T21:44:05.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:05.126242+0000) 2022-01-31T21:44:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:05 smithi167 conmon[54076]: debug 2022-01-31T21:44:05.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:05.357490+0000) 2022-01-31T21:44:05.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:05 smithi167 conmon[60316]: debug 2022-01-31T21:44:05.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:05.357490+0000) 2022-01-31T21:44:06.036 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:05 smithi171 conmon[51620]: debug 2022-01-31T21:44:05.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:05.966515+0000) 2022-01-31T21:44:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:05 smithi167 conmon[49112]: debug 2022-01-31T21:44:05.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:05.693675+0000) 2022-01-31T21:44:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:06 smithi171 conmon[46715]: debug 2022-01-31T21:44:06.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:06.126342+0000) 2022-01-31T21:44:06.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:06 smithi171 conmon[41853]: debug 2022-01-31T21:44:06.081+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:06.081593+0000) 2022-01-31T21:44:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:06 smithi167 conmon[54076]: debug 2022-01-31T21:44:06.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:06.357720+0000) 2022-01-31T21:44:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:06 smithi167 conmon[60316]: debug 2022-01-31T21:44:06.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:06.357719+0000) 2022-01-31T21:44:07.036 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:06 smithi171 conmon[51620]: debug 2022-01-31T21:44:06.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:06.966720+0000) 2022-01-31T21:44:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:06 smithi167 conmon[49112]: debug 2022-01-31T21:44:06.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:06.693869+0000) 2022-01-31T21:44:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:07 smithi171 conmon[46715]: debug 2022-01-31T21:44:07.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:07.126459+0000) 2022-01-31T21:44:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:07 smithi171 conmon[41853]: debug 2022-01-31T21:44:07.081+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:07.081747+0000) 2022-01-31T21:44:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:07 smithi167 conmon[54076]: debug 2022-01-31T21:44:07.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:07.357925+0000) 2022-01-31T21:44:07.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:07 smithi167 conmon[60316]: debug 2022-01-31T21:44:07.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:07.357925+0000) 2022-01-31T21:44:08.036 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:07 smithi171 conmon[51620]: debug 2022-01-31T21:44:07.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:07.966921+0000) 2022-01-31T21:44:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:07 smithi167 conmon[49112]: debug 2022-01-31T21:44:07.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:07.694102+0000) 2022-01-31T21:44:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[41853]: debug 2022-01-31T21:44:08.081+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.081912+0000) 2022-01-31T21:44:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[46715]: debug 2022-01-31T21:44:08.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.126651+0000) 2022-01-31T21:44:08.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:08 smithi167 conmon[49112]: debug 2022-01-31T21:44:08.582+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.583454+0000) 2022-01-31T21:44:08.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:08 smithi167 conmon[54076]: debug 2022-01-31T21:44:08.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.358129+0000) 2022-01-31T21:44:08.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:08 smithi167 conmon[54076]: debug 2022-01-31T21:44:08.583+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.583895+0000) 2022-01-31T21:44:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:08 smithi167 conmon[60316]: debug 2022-01-31T21:44:08.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.358129+0000) 2022-01-31T21:44:08.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:08 smithi167 conmon[60316]: debug 2022-01-31T21:44:08.583+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.583682+0000) 2022-01-31T21:44:08.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[41853]: debug 2022-01-31T21:44:08.583+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.584591+0000) 2022-01-31T21:44:08.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[46715]: debug 2022-01-31T21:44:08.583+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.583878+0000) 2022-01-31T21:44:08.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[51620]: debug 2022-01-31T21:44:08.582+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.582960+0000) 2022-01-31T21:44:08.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[35325]: debug 2022-01-31T21:44:08.599+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153356 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:08.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[35325]: debug 2022-01-31T21:44:08.730+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:44:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:08 smithi167 conmon[49112]: debug 2022-01-31T21:44:08.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.694349+0000) 2022-01-31T21:44:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:09 smithi171 conmon[41853]: debug 2022-01-31T21:44:09.081+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:09.082046+0000) 2022-01-31T21:44:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:09 smithi171 conmon[46715]: debug 2022-01-31T21:44:09.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:09.126799+0000) 2022-01-31T21:44:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:08 smithi171 conmon[51620]: debug 2022-01-31T21:44:08.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:08.967091+0000) 2022-01-31T21:44:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:09 smithi167 conmon[54076]: debug 2022-01-31T21:44:09.357+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:09.358359+0000) 2022-01-31T21:44:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:09 smithi167 conmon[60316]: debug 2022-01-31T21:44:09.357+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:09.358359+0000) 2022-01-31T21:44:10.036 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:09 smithi171 conmon[51620]: debug 2022-01-31T21:44:09.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:09.967170+0000) 2022-01-31T21:44:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:09 smithi167 conmon[49112]: debug 2022-01-31T21:44:09.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:09.694538+0000) 2022-01-31T21:44:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:10 smithi171 conmon[46715]: debug 2022-01-31T21:44:10.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:10.126921+0000) 2022-01-31T21:44:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:10 smithi171 conmon[41853]: debug 2022-01-31T21:44:10.081+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:10.082186+0000) 2022-01-31T21:44:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:10 smithi167 conmon[54076]: debug 2022-01-31T21:44:10.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:10.358513+0000) 2022-01-31T21:44:10.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:10 smithi167 conmon[60316]: debug 2022-01-31T21:44:10.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:10.358513+0000) 2022-01-31T21:44:11.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:10 smithi171 conmon[51620]: debug 2022-01-31T21:44:10.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:10.967287+0000) 2022-01-31T21:44:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:10 smithi167 conmon[49112]: debug 2022-01-31T21:44:10.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:10.694710+0000) 2022-01-31T21:44:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:11 smithi171 conmon[41853]: debug 2022-01-31T21:44:11.081+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:11.082348+0000) 2022-01-31T21:44:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:11 smithi171 conmon[46715]: debug 2022-01-31T21:44:11.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:11.127147+0000) 2022-01-31T21:44:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:11 smithi167 conmon[54076]: debug 2022-01-31T21:44:11.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:11.358712+0000) 2022-01-31T21:44:11.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:11 smithi167 conmon[60316]: debug 2022-01-31T21:44:11.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:11.358712+0000) 2022-01-31T21:44:12.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:11 smithi171 conmon[51620]: debug 2022-01-31T21:44:11.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:11.967448+0000) 2022-01-31T21:44:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:11 smithi167 conmon[49112]: debug 2022-01-31T21:44:11.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:11.694881+0000) 2022-01-31T21:44:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:12 smithi171 conmon[46715]: debug 2022-01-31T21:44:12.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:12.127301+0000) 2022-01-31T21:44:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:12 smithi171 conmon[41853]: debug 2022-01-31T21:44:12.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:12.082509+0000) 2022-01-31T21:44:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:12 smithi167 conmon[54076]: debug 2022-01-31T21:44:12.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:12.358899+0000) 2022-01-31T21:44:12.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:12 smithi167 conmon[60316]: debug 2022-01-31T21:44:12.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:12.358899+0000) 2022-01-31T21:44:13.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:12 smithi171 conmon[51620]: debug 2022-01-31T21:44:12.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:12.967596+0000) 2022-01-31T21:44:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:12 smithi167 conmon[49112]: debug 2022-01-31T21:44:12.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:12.695062+0000) 2022-01-31T21:44:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[41853]: debug 2022-01-31T21:44:13.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.082691+0000) 2022-01-31T21:44:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[46715]: debug 2022-01-31T21:44:13.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.127460+0000) 2022-01-31T21:44:13.608 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:13 smithi167 conmon[49112]: debug 2022-01-31T21:44:13.602+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.603267+0000) 2022-01-31T21:44:13.608 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:13 smithi167 conmon[54076]: debug 2022-01-31T21:44:13.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.359078+0000) 2022-01-31T21:44:13.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:13 smithi167 conmon[54076]: debug 2022-01-31T21:44:13.602+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.602558+0000) 2022-01-31T21:44:13.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:13 smithi167 conmon[60316]: debug 2022-01-31T21:44:13.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.359078+0000) 2022-01-31T21:44:13.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:13 smithi167 conmon[60316]: debug 2022-01-31T21:44:13.602+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.602705+0000) 2022-01-31T21:44:13.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[41853]: debug 2022-01-31T21:44:13.603+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.604349+0000) 2022-01-31T21:44:13.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[46715]: debug 2022-01-31T21:44:13.601+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.602467+0000) 2022-01-31T21:44:13.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[51620]: debug 2022-01-31T21:44:13.601+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.602623+0000) 2022-01-31T21:44:13.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[35325]: debug 2022-01-31T21:44:13.633+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153464 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:13.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:13 smithi167 conmon[49112]: debug 2022-01-31T21:44:13.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.695273+0000) 2022-01-31T21:44:14.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:13 smithi171 conmon[51620]: debug 2022-01-31T21:44:13.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:13.967773+0000) 2022-01-31T21:44:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:14 smithi171 conmon[41853]: debug 2022-01-31T21:44:14.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:14.082873+0000) 2022-01-31T21:44:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:14 smithi171 conmon[46715]: debug 2022-01-31T21:44:14.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:14.127636+0000) 2022-01-31T21:44:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:14 smithi167 conmon[54076]: debug 2022-01-31T21:44:14.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:14.359266+0000) 2022-01-31T21:44:14.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:14 smithi167 conmon[60316]: debug 2022-01-31T21:44:14.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:14.359267+0000) 2022-01-31T21:44:15.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:14 smithi171 conmon[51620]: debug 2022-01-31T21:44:14.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:14.967882+0000) 2022-01-31T21:44:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:14 smithi167 conmon[49112]: debug 2022-01-31T21:44:14.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:14.695450+0000) 2022-01-31T21:44:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:15 smithi171 conmon[41853]: debug 2022-01-31T21:44:15.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:15.082988+0000) 2022-01-31T21:44:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:15 smithi171 conmon[46715]: debug 2022-01-31T21:44:15.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:15.127785+0000) 2022-01-31T21:44:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:15 smithi167 conmon[54076]: debug 2022-01-31T21:44:15.358+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:15.359438+0000) 2022-01-31T21:44:15.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:15 smithi167 conmon[60316]: debug 2022-01-31T21:44:15.358+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:15.359440+0000) 2022-01-31T21:44:16.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:15 smithi171 conmon[51620]: debug 2022-01-31T21:44:15.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:15.968041+0000) 2022-01-31T21:44:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:15 smithi167 conmon[49112]: debug 2022-01-31T21:44:15.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:15.695611+0000) 2022-01-31T21:44:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:16 smithi171 conmon[41853]: debug 2022-01-31T21:44:16.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:16.083163+0000) 2022-01-31T21:44:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:16 smithi171 conmon[46715]: debug 2022-01-31T21:44:16.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:16.127975+0000) 2022-01-31T21:44:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:16 smithi167 conmon[54076]: debug 2022-01-31T21:44:16.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:16.359637+0000) 2022-01-31T21:44:16.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:16 smithi167 conmon[60316]: debug 2022-01-31T21:44:16.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:16.359636+0000) 2022-01-31T21:44:17.038 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:16 smithi171 conmon[51620]: debug 2022-01-31T21:44:16.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:16.968235+0000) 2022-01-31T21:44:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:16 smithi167 conmon[49112]: debug 2022-01-31T21:44:16.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:16.695820+0000) 2022-01-31T21:44:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:17 smithi171 conmon[41853]: debug 2022-01-31T21:44:17.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:17.083385+0000) 2022-01-31T21:44:17.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:17 smithi171 conmon[46715]: debug 2022-01-31T21:44:17.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:17.128138+0000) 2022-01-31T21:44:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:17 smithi167 conmon[54076]: debug 2022-01-31T21:44:17.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:17.359865+0000) 2022-01-31T21:44:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:17 smithi167 conmon[60316]: debug 2022-01-31T21:44:17.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:17.359863+0000) 2022-01-31T21:44:18.038 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:17 smithi171 conmon[51620]: debug 2022-01-31T21:44:17.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:17.968441+0000) 2022-01-31T21:44:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:17 smithi167 conmon[49112]: debug 2022-01-31T21:44:17.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:17.696024+0000) 2022-01-31T21:44:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[41853]: debug 2022-01-31T21:44:18.082+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.083568+0000) 2022-01-31T21:44:18.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[46715]: debug 2022-01-31T21:44:18.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.128319+0000) 2022-01-31T21:44:18.634 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[54076]: debug 2022-01-31T21:44:18.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.360060+0000) 2022-01-31T21:44:18.635 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[60316]: debug 2022-01-31T21:44:18.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.360012+0000) 2022-01-31T21:44:18.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[54076]: debug 2022-01-31T21:44:18.641+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.642481+0000) 2022-01-31T21:44:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[49112]: debug 2022-01-31T21:44:18.635+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.636362+0000) 2022-01-31T21:44:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[49112]: debug 2022-01-31T21:44:18.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.696227+0000) 2022-01-31T21:44:18.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[60316]: debug 2022-01-31T21:44:18.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:18 smithi167 conmon[60316]: 2022-01-31T21:44:18.635+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.635935+0000) 2022-01-31T21:44:18.966 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[46715]: debug 2022-01-31T21:44:18.635+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.636296+0000) 2022-01-31T21:44:18.967 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[35325]: debug 2022-01-31T21:44:18.652+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153577 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:18.967 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[41853]: debug 2022-01-31T21:44:18.636+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.637634+0000) 2022-01-31T21:44:18.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[51620]: debug 2022-01-31T21:44:18.635+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.636188+0000) 2022-01-31T21:44:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:19 smithi171 conmon[46715]: debug 2022-01-31T21:44:19.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:19.128465+0000) 2022-01-31T21:44:19.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:18 smithi171 conmon[51620]: debug 2022-01-31T21:44:18.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:18.968637+0000) 2022-01-31T21:44:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:19 smithi171 conmon[41853]: debug 2022-01-31T21:44:19.083+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:19.083788+0000) 2022-01-31T21:44:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:19 smithi167 conmon[54076]: debug 2022-01-31T21:44:19.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:19.360229+0000) 2022-01-31T21:44:19.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:19 smithi167 conmon[60316]: debug 2022-01-31T21:44:19.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:19.360210+0000) 2022-01-31T21:44:20.038 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:19 smithi171 conmon[51620]: debug 2022-01-31T21:44:19.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:19.968784+0000) 2022-01-31T21:44:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:19 smithi167 conmon[49112]: debug 2022-01-31T21:44:19.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:19.696360+0000) 2022-01-31T21:44:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:20 smithi171 conmon[41853]: debug 2022-01-31T21:44:20.083+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:20.083942+0000) 2022-01-31T21:44:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:20 smithi171 conmon[46715]: debug 2022-01-31T21:44:20.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:20.128601+0000) 2022-01-31T21:44:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:20 smithi167 conmon[54076]: debug 2022-01-31T21:44:20.359+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:20.360398+0000) 2022-01-31T21:44:20.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:20 smithi167 conmon[60316]: debug 2022-01-31T21:44:20.359+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:20.360395+0000) 2022-01-31T21:44:21.038 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:20 smithi171 conmon[51620]: debug 2022-01-31T21:44:20.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:20.968950+0000) 2022-01-31T21:44:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:20 smithi167 conmon[49112]: debug 2022-01-31T21:44:20.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:20.696570+0000) 2022-01-31T21:44:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:21 smithi171 conmon[41853]: debug 2022-01-31T21:44:21.083+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:21.084102+0000) 2022-01-31T21:44:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:21 smithi171 conmon[46715]: debug 2022-01-31T21:44:21.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:21.128770+0000) 2022-01-31T21:44:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:21 smithi167 conmon[54076]: debug 2022-01-31T21:44:21.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:21.360615+0000) 2022-01-31T21:44:21.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:21 smithi167 conmon[60316]: debug 2022-01-31T21:44:21.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:21.360618+0000) 2022-01-31T21:44:22.039 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:21 smithi171 conmon[51620]: debug 2022-01-31T21:44:21.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:21.969107+0000) 2022-01-31T21:44:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:21 smithi167 conmon[49112]: debug 2022-01-31T21:44:21.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:21.696810+0000) 2022-01-31T21:44:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:22 smithi171 conmon[41853]: debug 2022-01-31T21:44:22.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:22.084255+0000) 2022-01-31T21:44:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:22 smithi171 conmon[46715]: debug 2022-01-31T21:44:22.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:22.128956+0000) 2022-01-31T21:44:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:22 smithi167 conmon[54076]: debug 2022-01-31T21:44:22.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:22.360760+0000) 2022-01-31T21:44:22.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:22 smithi167 conmon[60316]: debug 2022-01-31T21:44:22.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:22.360792+0000) 2022-01-31T21:44:23.039 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:22 smithi171 conmon[51620]: debug 2022-01-31T21:44:22.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:22.969307+0000) 2022-01-31T21:44:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:22 smithi167 conmon[49112]: debug 2022-01-31T21:44:22.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:22.696998+0000) 2022-01-31T21:44:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[41853]: debug 2022-01-31T21:44:23.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.084425+0000) 2022-01-31T21:44:23.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[46715]: debug 2022-01-31T21:44:23.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.129069+0000) 2022-01-31T21:44:23.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:23 smithi167 conmon[54076]: debug 2022-01-31T21:44:23.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.360952+0000) 2022-01-31T21:44:23.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:23 smithi167 conmon[60316]: debug 2022-01-31T21:44:23.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.360970+0000) 2022-01-31T21:44:23.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:23 smithi167 conmon[54076]: debug 2022-01-31T21:44:23.654+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.655039+0000) 2022-01-31T21:44:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:23 smithi167 conmon[49112]: debug 2022-01-31T21:44:23.654+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.655301+0000) 2022-01-31T21:44:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:23 smithi167 conmon[49112]: debug 2022-01-31T21:44:23.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.697212+0000) 2022-01-31T21:44:23.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:23 smithi167 conmon[60316]: debug 2022-01-31T21:44:23.654+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.654900+0000) 2022-01-31T21:44:23.967 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[35325]: debug 2022-01-31T21:44:23.671+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:23.968 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[35325]: debug 2022-01-31T21:44:23.731+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:44:23.968 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[41853]: debug 2022-01-31T21:44:23.655+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.656211+0000) 2022-01-31T21:44:23.969 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[46715]: debug 2022-01-31T21:44:23.654+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.655310+0000) 2022-01-31T21:44:23.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[51620]: debug 2022-01-31T21:44:23.655+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.656114+0000) 2022-01-31T21:44:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:24 smithi171 conmon[41853]: debug 2022-01-31T21:44:24.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:24.084608+0000) 2022-01-31T21:44:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:24 smithi171 conmon[46715]: debug 2022-01-31T21:44:24.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:24.129214+0000) 2022-01-31T21:44:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:23 smithi171 conmon[51620]: debug 2022-01-31T21:44:23.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:23.969541+0000) 2022-01-31T21:44:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:24 smithi167 conmon[54076]: debug 2022-01-31T21:44:24.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:24.361169+0000) 2022-01-31T21:44:24.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:24 smithi167 conmon[60316]: debug 2022-01-31T21:44:24.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:24.361194+0000) 2022-01-31T21:44:25.039 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:24 smithi171 conmon[51620]: debug 2022-01-31T21:44:24.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:24.969703+0000) 2022-01-31T21:44:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:24 smithi167 conmon[49112]: debug 2022-01-31T21:44:24.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:24.697374+0000) 2022-01-31T21:44:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:25 smithi171 conmon[41853]: debug 2022-01-31T21:44:25.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:25.084735+0000) 2022-01-31T21:44:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:25 smithi171 conmon[46715]: debug 2022-01-31T21:44:25.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:25.129345+0000) 2022-01-31T21:44:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:25 smithi167 conmon[54076]: debug 2022-01-31T21:44:25.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:25.361341+0000) 2022-01-31T21:44:25.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:25 smithi167 conmon[60316]: debug 2022-01-31T21:44:25.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:25.361370+0000) 2022-01-31T21:44:26.039 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:25 smithi171 conmon[51620]: debug 2022-01-31T21:44:25.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:25.969901+0000) 2022-01-31T21:44:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:25 smithi167 conmon[49112]: debug 2022-01-31T21:44:25.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:25.697569+0000) 2022-01-31T21:44:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:26 smithi171 conmon[41853]: debug 2022-01-31T21:44:26.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:26.084905+0000) 2022-01-31T21:44:26.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:26 smithi171 conmon[46715]: debug 2022-01-31T21:44:26.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:26.129551+0000) 2022-01-31T21:44:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:26 smithi167 conmon[54076]: debug 2022-01-31T21:44:26.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:26.361487+0000) 2022-01-31T21:44:26.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:26 smithi167 conmon[60316]: debug 2022-01-31T21:44:26.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:26.361549+0000) 2022-01-31T21:44:27.040 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:26 smithi171 conmon[51620]: debug 2022-01-31T21:44:26.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:26.970119+0000) 2022-01-31T21:44:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:26 smithi167 conmon[49112]: debug 2022-01-31T21:44:26.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:26.697754+0000) 2022-01-31T21:44:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:27 smithi171 conmon[41853]: debug 2022-01-31T21:44:27.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:27.085096+0000) 2022-01-31T21:44:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:27 smithi171 conmon[46715]: debug 2022-01-31T21:44:27.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:27.129758+0000) 2022-01-31T21:44:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:27 smithi167 conmon[54076]: debug 2022-01-31T21:44:27.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:27.361736+0000) 2022-01-31T21:44:27.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:27 smithi167 conmon[60316]: debug 2022-01-31T21:44:27.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:27.361739+0000) 2022-01-31T21:44:28.040 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:27 smithi171 conmon[51620]: debug 2022-01-31T21:44:27.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:27.970320+0000) 2022-01-31T21:44:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:27 smithi167 conmon[49112]: debug 2022-01-31T21:44:27.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:27.697914+0000) 2022-01-31T21:44:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[41853]: debug 2022-01-31T21:44:28.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.085314+0000) 2022-01-31T21:44:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[46715]: debug 2022-01-31T21:44:28.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.129966+0000) 2022-01-31T21:44:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:28 smithi167 conmon[54076]: debug 2022-01-31T21:44:28.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.361974+0000) 2022-01-31T21:44:28.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:28 smithi167 conmon[60316]: debug 2022-01-31T21:44:28.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.361991+0000) 2022-01-31T21:44:28.968 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[35325]: debug 2022-01-31T21:44:28.689+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153798 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:28.969 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[41853]: debug 2022-01-31T21:44:28.675+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.676903+0000) 2022-01-31T21:44:28.969 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[46715]: debug 2022-01-31T21:44:28.672+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.674384+0000) 2022-01-31T21:44:28.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[51620]: debug 2022-01-31T21:44:28.673+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.675435+0000) 2022-01-31T21:44:29.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:28 smithi167 conmon[54076]: debug 2022-01-31T21:44:28.673+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.674861+0000) 2022-01-31T21:44:29.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:28 smithi167 conmon[60316]: debug 2022-01-31T21:44:28.673+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.675404+0000) 2022-01-31T21:44:29.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:28 smithi167 conmon[49112]: debug 2022-01-31T21:44:28.672+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.674301+0000) 2022-01-31T21:44:29.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:28 smithi167 conmon[49112]: debug 2022-01-31T21:44:28.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.698061+0000) 2022-01-31T21:44:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:29 smithi171 conmon[41853]: debug 2022-01-31T21:44:29.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:29.085456+0000) 2022-01-31T21:44:29.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:29 smithi171 conmon[46715]: debug 2022-01-31T21:44:29.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:29.130140+0000) 2022-01-31T21:44:29.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:28 smithi171 conmon[51620]: debug 2022-01-31T21:44:28.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:28.970514+0000) 2022-01-31T21:44:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:29 smithi167 conmon[54076]: debug 2022-01-31T21:44:29.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:29.362131+0000) 2022-01-31T21:44:29.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:29 smithi167 conmon[60316]: debug 2022-01-31T21:44:29.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:29.362150+0000) 2022-01-31T21:44:30.040 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:29 smithi171 conmon[51620]: debug 2022-01-31T21:44:29.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:29.970674+0000) 2022-01-31T21:44:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:29 smithi167 conmon[49112]: debug 2022-01-31T21:44:29.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:29.698222+0000) 2022-01-31T21:44:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:30 smithi171 conmon[46715]: debug 2022-01-31T21:44:30.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:30.130288+0000) 2022-01-31T21:44:30.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:30 smithi171 conmon[41853]: debug 2022-01-31T21:44:30.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:30.085617+0000) 2022-01-31T21:44:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:30 smithi167 conmon[54076]: debug 2022-01-31T21:44:30.360+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:30.362324+0000) 2022-01-31T21:44:30.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:30 smithi167 conmon[60316]: debug 2022-01-31T21:44:30.360+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:30.362307+0000) 2022-01-31T21:44:31.040 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:30 smithi171 conmon[51620]: debug 2022-01-31T21:44:30.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:30.970886+0000) 2022-01-31T21:44:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:30 smithi167 conmon[49112]: debug 2022-01-31T21:44:30.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:30.698442+0000) 2022-01-31T21:44:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:31 smithi171 conmon[46715]: debug 2022-01-31T21:44:31.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:31.130466+0000) 2022-01-31T21:44:31.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:31 smithi171 conmon[41853]: debug 2022-01-31T21:44:31.084+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:31.085819+0000) 2022-01-31T21:44:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:31 smithi167 conmon[54076]: debug 2022-01-31T21:44:31.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:31.362535+0000) 2022-01-31T21:44:31.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:31 smithi167 conmon[60316]: debug 2022-01-31T21:44:31.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:31.362543+0000) 2022-01-31T21:44:32.040 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:31 smithi171 conmon[51620]: debug 2022-01-31T21:44:31.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:31.971061+0000) 2022-01-31T21:44:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:31 smithi167 conmon[49112]: debug 2022-01-31T21:44:31.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:31.698602+0000) 2022-01-31T21:44:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:32 smithi171 conmon[46715]: debug 2022-01-31T21:44:32.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:32.130601+0000) 2022-01-31T21:44:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:32 smithi171 conmon[41853]: debug 2022-01-31T21:44:32.085+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:32.086002+0000) 2022-01-31T21:44:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:32 smithi167 conmon[54076]: debug 2022-01-31T21:44:32.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:32.362759+0000) 2022-01-31T21:44:32.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:32 smithi167 conmon[60316]: debug 2022-01-31T21:44:32.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:32.362768+0000) 2022-01-31T21:44:33.041 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:32 smithi171 conmon[51620]: debug 2022-01-31T21:44:32.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:32.971252+0000) 2022-01-31T21:44:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:32 smithi167 conmon[49112]: debug 2022-01-31T21:44:32.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:32.698789+0000) 2022-01-31T21:44:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[41853]: debug 2022-01-31T21:44:33.085+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.086212+0000) 2022-01-31T21:44:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[46715]: debug 2022-01-31T21:44:33.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.130801+0000) 2022-01-31T21:44:33.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:33 smithi167 conmon[54076]: debug 2022-01-31T21:44:33.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.362941+0000) 2022-01-31T21:44:33.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:33 smithi167 conmon[60316]: debug 2022-01-31T21:44:33.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.362934+0000) 2022-01-31T21:44:33.969 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[35325]: debug 2022-01-31T21:44:33.708+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 153906 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:33.970 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[41853]: debug 2022-01-31T21:44:33.693+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.694895+0000) 2022-01-31T21:44:33.971 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[46715]: debug 2022-01-31T21:44:33.691+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.692860+0000) 2022-01-31T21:44:33.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[51620]: debug 2022-01-31T21:44:33.691+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.693678+0000) 2022-01-31T21:44:34.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:33 smithi167 conmon[54076]: debug 2022-01-31T21:44:33.692+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.694116+0000) 2022-01-31T21:44:34.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:33 smithi167 conmon[60316]: debug 2022-01-31T21:44:33.693+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.694620+0000) 2022-01-31T21:44:34.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:33 smithi167 conmon[49112]: debug 2022-01-31T21:44:33.692+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.693735+0000) 2022-01-31T21:44:34.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:33 smithi167 conmon[49112]: debug 2022-01-31T21:44:33.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.698969+0000) 2022-01-31T21:44:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:34 smithi171 conmon[46715]: debug 2022-01-31T21:44:34.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:34.131008+0000) 2022-01-31T21:44:34.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:34 smithi171 conmon[41853]: debug 2022-01-31T21:44:34.085+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:34.086396+0000) 2022-01-31T21:44:34.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:33 smithi171 conmon[51620]: debug 2022-01-31T21:44:33.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:33.971466+0000) 2022-01-31T21:44:34.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:34 smithi167 conmon[60316]: debug 2022-01-31T21:44:34.362+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:34.363103+0000) 2022-01-31T21:44:34.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:34 smithi167 conmon[54076]: debug 2022-01-31T21:44:34.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:34.363102+0000) 2022-01-31T21:44:35.041 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:34 smithi171 conmon[51620]: debug 2022-01-31T21:44:34.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:34.971598+0000) 2022-01-31T21:44:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:34 smithi167 conmon[49112]: debug 2022-01-31T21:44:34.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:34.699110+0000) 2022-01-31T21:44:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:35 smithi171 conmon[41853]: debug 2022-01-31T21:44:35.085+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:35.086544+0000) 2022-01-31T21:44:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:35 smithi171 conmon[46715]: debug 2022-01-31T21:44:35.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:35.131152+0000) 2022-01-31T21:44:35.672 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:35 smithi167 conmon[54076]: debug 2022-01-31T21:44:35.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:35.363278+0000) 2022-01-31T21:44:35.673 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:35 smithi167 conmon[60316]: debug 2022-01-31T21:44:35.361+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:35.363283+0000) 2022-01-31T21:44:36.041 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:35 smithi171 conmon[51620]: debug 2022-01-31T21:44:35.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:35.971801+0000) 2022-01-31T21:44:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:35 smithi167 conmon[49112]: debug 2022-01-31T21:44:35.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:35.699281+0000) 2022-01-31T21:44:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:36 smithi171 conmon[41853]: debug 2022-01-31T21:44:36.085+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:36.086711+0000) 2022-01-31T21:44:36.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:36 smithi171 conmon[46715]: debug 2022-01-31T21:44:36.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:36.131313+0000) 2022-01-31T21:44:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:36 smithi167 conmon[54076]: debug 2022-01-31T21:44:36.361+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:36.363466+0000) 2022-01-31T21:44:36.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:36 smithi167 conmon[60316]: debug 2022-01-31T21:44:36.362+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:36.363521+0000) 2022-01-31T21:44:37.041 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:36 smithi171 conmon[51620]: debug 2022-01-31T21:44:36.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:36.971949+0000) 2022-01-31T21:44:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:36 smithi167 conmon[49112]: debug 2022-01-31T21:44:36.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:36.699492+0000) 2022-01-31T21:44:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:37 smithi171 conmon[41853]: debug 2022-01-31T21:44:37.086+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:37.086862+0000) 2022-01-31T21:44:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:37 smithi171 conmon[46715]: debug 2022-01-31T21:44:37.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:37.131487+0000) 2022-01-31T21:44:37.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:37 smithi167 conmon[54076]: debug 2022-01-31T21:44:37.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:37.363677+0000) 2022-01-31T21:44:37.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:37 smithi167 conmon[60316]: debug 2022-01-31T21:44:37.362+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:37.363730+0000) 2022-01-31T21:44:38.042 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:37 smithi171 conmon[51620]: debug 2022-01-31T21:44:37.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:37.972123+0000) 2022-01-31T21:44:38.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:37 smithi167 conmon[49112]: debug 2022-01-31T21:44:37.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:37.699747+0000) 2022-01-31T21:44:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[41853]: debug 2022-01-31T21:44:38.086+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.087015+0000) 2022-01-31T21:44:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[46715]: debug 2022-01-31T21:44:38.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.131674+0000) 2022-01-31T21:44:38.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:38 smithi167 conmon[54076]: debug 2022-01-31T21:44:38.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.363889+0000) 2022-01-31T21:44:38.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:38 smithi167 conmon[60316]: debug 2022-01-31T21:44:38.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.363935+0000) 2022-01-31T21:44:38.969 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[41853]: debug 2022-01-31T21:44:38.712+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.713013+0000) 2022-01-31T21:44:38.970 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[46715]: debug 2022-01-31T21:44:38.711+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.712119+0000) 2022-01-31T21:44:38.971 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[35325]: debug 2022-01-31T21:44:38.727+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154019 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:38.971 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[35325]: debug 2022-01-31T21:44:38.732+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:44:38.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[51620]: debug 2022-01-31T21:44:38.711+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.711967+0000) 2022-01-31T21:44:38.972 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:38 smithi171 conmon[51620]: debug 2022-01-31T21:44:38.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.972292+0000) 2022-01-31T21:44:39.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:38 smithi167 conmon[54076]: debug 2022-01-31T21:44:38.710+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.712179+0000) 2022-01-31T21:44:39.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:38 smithi167 conmon[60316]: debug 2022-01-31T21:44:38.710+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.712305+0000) 2022-01-31T21:44:39.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:38 smithi167 conmon[49112]: debug 2022-01-31T21:44:38.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.699958+0000) 2022-01-31T21:44:39.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:38 smithi167 conmon[49112]: debug 2022-01-31T21:44:38.711+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:38.713008+0000) 2022-01-31T21:44:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:39 smithi171 conmon[41853]: debug 2022-01-31T21:44:39.086+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:39.087191+0000) 2022-01-31T21:44:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:39 smithi171 conmon[46715]: debug 2022-01-31T21:44:39.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:39.131864+0000) 2022-01-31T21:44:39.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:39 smithi167 conmon[60316]: debug 2022-01-31T21:44:39.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:39.364096+0000) 2022-01-31T21:44:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:39 smithi167 conmon[54076]: debug 2022-01-31T21:44:39.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:39.364058+0000) 2022-01-31T21:44:40.042 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:39 smithi171 conmon[51620]: debug 2022-01-31T21:44:39.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:39.972430+0000) 2022-01-31T21:44:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:39 smithi167 conmon[49112]: debug 2022-01-31T21:44:39.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:39.700191+0000) 2022-01-31T21:44:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:40 smithi171 conmon[41853]: debug 2022-01-31T21:44:40.086+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:40.087347+0000) 2022-01-31T21:44:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:40 smithi171 conmon[46715]: debug 2022-01-31T21:44:40.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:40.131996+0000) 2022-01-31T21:44:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:40 smithi167 conmon[54076]: debug 2022-01-31T21:44:40.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:40.364257+0000) 2022-01-31T21:44:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:40 smithi167 conmon[60316]: debug 2022-01-31T21:44:40.362+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:40.364259+0000) 2022-01-31T21:44:41.042 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:40 smithi171 conmon[51620]: debug 2022-01-31T21:44:40.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:40.972597+0000) 2022-01-31T21:44:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:40 smithi167 conmon[49112]: debug 2022-01-31T21:44:40.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:40.700360+0000) 2022-01-31T21:44:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:41 smithi171 conmon[46715]: debug 2022-01-31T21:44:41.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:41.132203+0000) 2022-01-31T21:44:41.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:41 smithi171 conmon[41853]: debug 2022-01-31T21:44:41.086+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:41.087501+0000) 2022-01-31T21:44:41.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:41 smithi167 conmon[54076]: debug 2022-01-31T21:44:41.362+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:41.364438+0000) 2022-01-31T21:44:41.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:41 smithi167 conmon[60316]: debug 2022-01-31T21:44:41.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:41.364516+0000) 2022-01-31T21:44:42.042 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:41 smithi171 conmon[51620]: debug 2022-01-31T21:44:41.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:41.972783+0000) 2022-01-31T21:44:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:41 smithi167 conmon[49112]: debug 2022-01-31T21:44:41.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:41.700570+0000) 2022-01-31T21:44:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:42 smithi171 conmon[41853]: debug 2022-01-31T21:44:42.086+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:42.087632+0000) 2022-01-31T21:44:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:42 smithi171 conmon[46715]: debug 2022-01-31T21:44:42.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:42.132361+0000) 2022-01-31T21:44:42.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:42 smithi167 conmon[54076]: debug 2022-01-31T21:44:42.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:42.364578+0000) 2022-01-31T21:44:42.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:42 smithi167 conmon[60316]: debug 2022-01-31T21:44:42.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:42.364642+0000) 2022-01-31T21:44:43.042 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:42 smithi171 conmon[51620]: debug 2022-01-31T21:44:42.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:42.972944+0000) 2022-01-31T21:44:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:42 smithi167 conmon[49112]: debug 2022-01-31T21:44:42.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:42.700767+0000) 2022-01-31T21:44:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[41853]: debug 2022-01-31T21:44:43.087+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.087776+0000) 2022-01-31T21:44:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[46715]: debug 2022-01-31T21:44:43.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.132491+0000) 2022-01-31T21:44:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[54076]: debug 2022-01-31T21:44:43.363+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.364698+0000) 2022-01-31T21:44:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[60316]: debug 2022-01-31T21:44:43.363+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.364818+0000) 2022-01-31T21:44:43.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[35325]: debug 2022-01-31T21:44:43.746+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154127 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[41853]: debug 2022-01-31T21:44:43.730+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.730782+0000) 2022-01-31T21:44:43.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[46715]: debug 2022-01-31T21:44:43.730+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.731322+0000) 2022-01-31T21:44:43.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[51620]: debug 2022-01-31T21:44:43.729+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.730323+0000) 2022-01-31T21:44:44.130 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:43 smithi171 conmon[51620]: debug 2022-01-31T21:44:43.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.973096+0000) 2022-01-31T21:44:44.131 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:44 smithi171 conmon[41853]: debug 2022-01-31T21:44:44.087+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:44.087937+0000) 2022-01-31T21:44:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[49112]: debug 2022-01-31T21:44:43.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.700979+0000) 2022-01-31T21:44:44.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[49112]: debug 2022-01-31T21:44:44.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[49112]: 2022-01-31T21:44:43.729+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.731173+0000) 2022-01-31T21:44:44.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[54076]: debug 2022-01-31T21:44:43.728+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.730406+0000) 2022-01-31T21:44:44.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:43 smithi167 conmon[60316]: debug 2022-01-31T21:44:43.729+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:43.730598+0000) 2022-01-31T21:44:44.391 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:44 smithi171 conmon[46715]: debug 2022-01-31T21:44:44.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:44.132670+0000) 2022-01-31T21:44:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:44 smithi167 conmon[54076]: debug 2022-01-31T21:44:44.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:44.364874+0000) 2022-01-31T21:44:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:44 smithi167 conmon[60316]: debug 2022-01-31T21:44:44.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:44.364971+0000) 2022-01-31T21:44:45.043 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:44 smithi171 conmon[51620]: debug 2022-01-31T21:44:44.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:44.973243+0000) 2022-01-31T21:44:45.171 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:44 smithi167 conmon[49112]: debug 2022-01-31T21:44:44.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:44.701193+0000) 2022-01-31T21:44:45.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:45 smithi171 conmon[46715]: debug 2022-01-31T21:44:45.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:45.132826+0000) 2022-01-31T21:44:45.327 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:45 smithi171 conmon[41853]: debug 2022-01-31T21:44:45.087+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:45.088045+0000) 2022-01-31T21:44:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:45 smithi167 conmon[54076]: debug 2022-01-31T21:44:45.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:45.365064+0000) 2022-01-31T21:44:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:45 smithi167 conmon[60316]: debug 2022-01-31T21:44:45.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:45.365113+0000) 2022-01-31T21:44:46.043 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:45 smithi171 conmon[51620]: debug 2022-01-31T21:44:45.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:45.973431+0000) 2022-01-31T21:44:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:45 smithi167 conmon[49112]: debug 2022-01-31T21:44:45.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:45.701334+0000) 2022-01-31T21:44:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:46 smithi171 conmon[41853]: debug 2022-01-31T21:44:46.087+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:46.088195+0000) 2022-01-31T21:44:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:46 smithi171 conmon[46715]: debug 2022-01-31T21:44:46.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:46.133014+0000) 2022-01-31T21:44:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:46 smithi167 conmon[54076]: debug 2022-01-31T21:44:46.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:46.365254+0000) 2022-01-31T21:44:46.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:46 smithi167 conmon[60316]: debug 2022-01-31T21:44:46.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:46.365335+0000) 2022-01-31T21:44:47.043 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:46 smithi171 conmon[51620]: debug 2022-01-31T21:44:46.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:46.973619+0000) 2022-01-31T21:44:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:46 smithi167 conmon[49112]: debug 2022-01-31T21:44:46.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:46.701545+0000) 2022-01-31T21:44:47.295 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:47 smithi171 conmon[46715]: debug 2022-01-31T21:44:47.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:47.133160+0000) 2022-01-31T21:44:47.296 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:47 smithi171 conmon[41853]: debug 2022-01-31T21:44:47.087+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:47.088410+0000) 2022-01-31T21:44:47.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:47 smithi167 conmon[54076]: debug 2022-01-31T21:44:47.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:47.365452+0000) 2022-01-31T21:44:47.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:47 smithi167 conmon[60316]: debug 2022-01-31T21:44:47.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:47.365497+0000) 2022-01-31T21:44:47.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:47 smithi167 conmon[49112]: debug 2022-01-31T21:44:47.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:47.701724+0000) 2022-01-31T21:44:48.043 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:47 smithi171 conmon[51620]: debug 2022-01-31T21:44:47.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:47.973805+0000) 2022-01-31T21:44:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[41853]: debug 2022-01-31T21:44:48.088+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.088595+0000) 2022-01-31T21:44:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[46715]: debug 2022-01-31T21:44:48.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.133316+0000) 2022-01-31T21:44:48.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:48 smithi167 conmon[60316]: debug 2022-01-31T21:44:48.364+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.365620+0000) 2022-01-31T21:44:48.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:48 smithi167 conmon[54076]: debug 2022-01-31T21:44:48.364+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.365639+0000) 2022-01-31T21:44:49.043 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[35325]: debug 2022-01-31T21:44:48.765+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154240 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:49.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[41853]: debug 2022-01-31T21:44:48.749+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.749989+0000) 2022-01-31T21:44:49.045 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[46715]: debug 2022-01-31T21:44:48.748+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.749030+0000) 2022-01-31T21:44:49.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[51620]: debug 2022-01-31T21:44:48.748+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.749102+0000) 2022-01-31T21:44:49.046 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:48 smithi171 conmon[51620]: debug 2022-01-31T21:44:48.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.974011+0000) 2022-01-31T21:44:49.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:48 smithi167 conmon[54076]: debug 2022-01-31T21:44:48.747+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.748713+0000) 2022-01-31T21:44:49.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:48 smithi167 conmon[60316]: debug 2022-01-31T21:44:48.748+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.749835+0000) 2022-01-31T21:44:49.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:48 smithi167 conmon[49112]: debug 2022-01-31T21:44:48.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.701920+0000) 2022-01-31T21:44:49.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:48 smithi167 conmon[49112]: debug 2022-01-31T21:44:48.748+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:48.749755+0000) 2022-01-31T21:44:49.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:49 smithi171 conmon[46715]: debug 2022-01-31T21:44:49.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:49.133504+0000) 2022-01-31T21:44:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:49 smithi171 conmon[41853]: debug 2022-01-31T21:44:49.088+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:49.088804+0000) 2022-01-31T21:44:49.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:49 smithi167 conmon[54076]: debug 2022-01-31T21:44:49.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:49.365855+0000) 2022-01-31T21:44:49.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:49 smithi167 conmon[60316]: debug 2022-01-31T21:44:49.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:49.365855+0000) 2022-01-31T21:44:49.867 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:49 smithi167 conmon[49112]: debug 2022-01-31T21:44:49.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:49.702131+0000) 2022-01-31T21:44:50.044 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:49 smithi171 conmon[51620]: debug 2022-01-31T21:44:49.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:49.974163+0000) 2022-01-31T21:44:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:50 smithi171 conmon[41853]: debug 2022-01-31T21:44:50.088+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:50.088954+0000) 2022-01-31T21:44:50.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:50 smithi171 conmon[46715]: debug 2022-01-31T21:44:50.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:50.133690+0000) 2022-01-31T21:44:50.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:50 smithi167 conmon[54076]: debug 2022-01-31T21:44:50.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:50.366057+0000) 2022-01-31T21:44:50.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:50 smithi167 conmon[60316]: debug 2022-01-31T21:44:50.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:50.366005+0000) 2022-01-31T21:44:51.044 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:50 smithi171 conmon[51620]: debug 2022-01-31T21:44:50.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:50.974348+0000) 2022-01-31T21:44:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:50 smithi167 conmon[49112]: debug 2022-01-31T21:44:50.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:50.702298+0000) 2022-01-31T21:44:51.327 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:51 smithi171 conmon[41853]: debug 2022-01-31T21:44:51.088+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:51.089158+0000) 2022-01-31T21:44:51.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:51 smithi171 conmon[46715]: debug 2022-01-31T21:44:51.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:51.133921+0000) 2022-01-31T21:44:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:51 smithi167 conmon[54076]: debug 2022-01-31T21:44:51.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:51.366259+0000) 2022-01-31T21:44:51.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:51 smithi167 conmon[60316]: debug 2022-01-31T21:44:51.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:51.366207+0000) 2022-01-31T21:44:52.044 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:51 smithi171 conmon[51620]: debug 2022-01-31T21:44:51.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:51.974555+0000) 2022-01-31T21:44:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:51 smithi167 conmon[49112]: debug 2022-01-31T21:44:51.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:51.702536+0000) 2022-01-31T21:44:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:52 smithi171 conmon[41853]: debug 2022-01-31T21:44:52.088+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:52.089333+0000) 2022-01-31T21:44:52.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:52 smithi171 conmon[46715]: debug 2022-01-31T21:44:52.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:52.134136+0000) 2022-01-31T21:44:52.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:52 smithi167 conmon[54076]: debug 2022-01-31T21:44:52.365+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:52.366427+0000) 2022-01-31T21:44:52.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:52 smithi167 conmon[60316]: debug 2022-01-31T21:44:52.365+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:52.366384+0000) 2022-01-31T21:44:53.044 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:52 smithi171 conmon[51620]: debug 2022-01-31T21:44:52.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:52.974744+0000) 2022-01-31T21:44:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:52 smithi167 conmon[49112]: debug 2022-01-31T21:44:52.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:52.702701+0000) 2022-01-31T21:44:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[41853]: debug 2022-01-31T21:44:53.089+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.089507+0000) 2022-01-31T21:44:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[46715]: debug 2022-01-31T21:44:53.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.134315+0000) 2022-01-31T21:44:53.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:53 smithi167 conmon[54076]: debug 2022-01-31T21:44:53.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.366625+0000) 2022-01-31T21:44:53.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:53 smithi167 conmon[60316]: debug 2022-01-31T21:44:53.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.366575+0000) 2022-01-31T21:44:53.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:53 smithi167 conmon[54076]: debug 2022-01-31T21:44:53.767+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.767694+0000) 2022-01-31T21:44:53.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:53 smithi167 conmon[60316]: debug 2022-01-31T21:44:53.767+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.767979+0000) 2022-01-31T21:44:53.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:53 smithi167 conmon[49112]: debug 2022-01-31T21:44:53.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.702936+0000) 2022-01-31T21:44:53.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:53 smithi167 conmon[49112]: debug 2022-01-31T21:44:53.767+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.768325+0000) 2022-01-31T21:44:54.044 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[41853]: debug 2022-01-31T21:44:53.767+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.767995+0000) 2022-01-31T21:44:54.045 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[46715]: debug 2022-01-31T21:44:53.766+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.767604+0000) 2022-01-31T21:44:54.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[35325]: debug 2022-01-31T21:44:53.733+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:44:54.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[35325]: debug 2022-01-31T21:44:53.783+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154348 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:54.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[51620]: debug 2022-01-31T21:44:53.767+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.767839+0000) 2022-01-31T21:44:54.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:53 smithi171 conmon[51620]: debug 2022-01-31T21:44:53.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:53.974914+0000) 2022-01-31T21:44:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:54 smithi171 conmon[41853]: debug 2022-01-31T21:44:54.089+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:54.089705+0000) 2022-01-31T21:44:54.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:54 smithi171 conmon[46715]: debug 2022-01-31T21:44:54.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:54.134473+0000) 2022-01-31T21:44:54.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:54 smithi167 conmon[54076]: debug 2022-01-31T21:44:54.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:54.366788+0000) 2022-01-31T21:44:54.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:54 smithi167 conmon[60316]: debug 2022-01-31T21:44:54.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:54.366787+0000) 2022-01-31T21:44:55.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:54 smithi171 conmon[51620]: debug 2022-01-31T21:44:54.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:54.975068+0000) 2022-01-31T21:44:55.206 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:54 smithi167 conmon[49112]: debug 2022-01-31T21:44:54.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:54.703172+0000) 2022-01-31T21:44:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:55 smithi171 conmon[41853]: debug 2022-01-31T21:44:55.089+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:55.089865+0000) 2022-01-31T21:44:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:55 smithi171 conmon[46715]: debug 2022-01-31T21:44:55.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:55.134626+0000) 2022-01-31T21:44:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:55 smithi167 conmon[54076]: debug 2022-01-31T21:44:55.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:55.366967+0000) 2022-01-31T21:44:55.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:55 smithi167 conmon[60316]: debug 2022-01-31T21:44:55.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:55.366969+0000) 2022-01-31T21:44:56.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:55 smithi171 conmon[51620]: debug 2022-01-31T21:44:55.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:55.975254+0000) 2022-01-31T21:44:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:55 smithi167 conmon[49112]: debug 2022-01-31T21:44:55.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:55.703344+0000) 2022-01-31T21:44:56.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:56 smithi171 conmon[41853]: debug 2022-01-31T21:44:56.089+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:56.090015+0000) 2022-01-31T21:44:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:56 smithi171 conmon[46715]: debug 2022-01-31T21:44:56.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:56.134790+0000) 2022-01-31T21:44:56.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:56 smithi167 conmon[54076]: debug 2022-01-31T21:44:56.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:56.367173+0000) 2022-01-31T21:44:56.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:56 smithi167 conmon[60316]: debug 2022-01-31T21:44:56.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:56.367172+0000) 2022-01-31T21:44:57.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:56 smithi171 conmon[51620]: debug 2022-01-31T21:44:56.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:56.975460+0000) 2022-01-31T21:44:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:56 smithi167 conmon[49112]: debug 2022-01-31T21:44:56.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:56.703549+0000) 2022-01-31T21:44:57.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:57 smithi171 conmon[41853]: debug 2022-01-31T21:44:57.089+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:57.090218+0000) 2022-01-31T21:44:57.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:57 smithi171 conmon[46715]: debug 2022-01-31T21:44:57.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:57.134948+0000) 2022-01-31T21:44:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:57 smithi167 conmon[54076]: debug 2022-01-31T21:44:57.366+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:57.367351+0000) 2022-01-31T21:44:57.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:57 smithi167 conmon[60316]: debug 2022-01-31T21:44:57.366+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:57.367351+0000) 2022-01-31T21:44:58.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:57 smithi171 conmon[51620]: debug 2022-01-31T21:44:57.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:57.975645+0000) 2022-01-31T21:44:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:57 smithi167 conmon[49112]: debug 2022-01-31T21:44:57.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:57.703764+0000) 2022-01-31T21:44:58.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[46715]: debug 2022-01-31T21:44:58.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.135130+0000) 2022-01-31T21:44:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[41853]: debug 2022-01-31T21:44:58.089+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.090403+0000) 2022-01-31T21:44:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:58 smithi167 conmon[54076]: debug 2022-01-31T21:44:58.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.367539+0000) 2022-01-31T21:44:58.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:58 smithi167 conmon[60316]: debug 2022-01-31T21:44:58.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.367536+0000) 2022-01-31T21:44:59.045 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[35325]: debug 2022-01-31T21:44:58.802+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154460 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:44:59.046 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[41853]: debug 2022-01-31T21:44:58.786+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.787052+0000) 2022-01-31T21:44:59.047 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[46715]: debug 2022-01-31T21:44:58.785+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.786378+0000) 2022-01-31T21:44:59.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[51620]: debug 2022-01-31T21:44:58.786+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.787074+0000) 2022-01-31T21:44:59.048 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:58 smithi171 conmon[51620]: debug 2022-01-31T21:44:58.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.975851+0000) 2022-01-31T21:44:59.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:58 smithi167 conmon[54076]: debug 2022-01-31T21:44:58.786+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.786893+0000) 2022-01-31T21:44:59.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:58 smithi167 conmon[60316]: debug 2022-01-31T21:44:58.785+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.786260+0000) 2022-01-31T21:44:59.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:58 smithi167 conmon[49112]: debug 2022-01-31T21:44:58.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.703969+0000) 2022-01-31T21:44:59.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:58 smithi167 conmon[49112]: debug 2022-01-31T21:44:58.786+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:58.787051+0000) 2022-01-31T21:44:59.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:44:59 smithi171 conmon[41853]: debug 2022-01-31T21:44:59.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:59.090634+0000) 2022-01-31T21:44:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:44:59 smithi171 conmon[46715]: debug 2022-01-31T21:44:59.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:59.135312+0000) 2022-01-31T21:44:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:44:59 smithi167 conmon[54076]: debug 2022-01-31T21:44:59.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:59.367761+0000) 2022-01-31T21:44:59.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:44:59 smithi167 conmon[60316]: debug 2022-01-31T21:44:59.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:59.367752+0000) 2022-01-31T21:45:00.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:44:59 smithi171 conmon[51620]: debug 2022-01-31T21:44:59.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:59.975970+0000) 2022-01-31T21:45:00.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:44:59 smithi167 conmon[49112]: debug 2022-01-31T21:44:59.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:44:59.704142+0000) 2022-01-31T21:45:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:00 smithi171 conmon[41853]: debug 2022-01-31T21:45:00.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:00.090813+0000) 2022-01-31T21:45:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:00 smithi171 conmon[46715]: debug 2022-01-31T21:45:00.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:00.135406+0000) 2022-01-31T21:45:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:00 smithi167 conmon[54076]: debug 2022-01-31T21:45:00.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:00.367927+0000) 2022-01-31T21:45:00.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:00 smithi167 conmon[60316]: debug 2022-01-31T21:45:00.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:00.367909+0000) 2022-01-31T21:45:01.046 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:00 smithi171 conmon[51620]: debug 2022-01-31T21:45:00.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:00.976157+0000) 2022-01-31T21:45:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:00 smithi167 conmon[49112]: debug 2022-01-31T21:45:00.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:00.704310+0000) 2022-01-31T21:45:01.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:01 smithi171 conmon[41853]: debug 2022-01-31T21:45:01.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:01.090989+0000) 2022-01-31T21:45:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:01 smithi171 conmon[46715]: debug 2022-01-31T21:45:01.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:01.135560+0000) 2022-01-31T21:45:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:01 smithi167 conmon[54076]: debug 2022-01-31T21:45:01.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:01.368048+0000) 2022-01-31T21:45:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:01 smithi167 conmon[60316]: debug 2022-01-31T21:45:01.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:01.368084+0000) 2022-01-31T21:45:02.046 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:01 smithi171 conmon[51620]: debug 2022-01-31T21:45:01.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:01.976336+0000) 2022-01-31T21:45:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:01 smithi167 conmon[49112]: debug 2022-01-31T21:45:01.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:01.704520+0000) 2022-01-31T21:45:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:02 smithi171 conmon[46715]: debug 2022-01-31T21:45:02.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:02.135772+0000) 2022-01-31T21:45:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:02 smithi171 conmon[41853]: debug 2022-01-31T21:45:02.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:02.091150+0000) 2022-01-31T21:45:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:02 smithi167 conmon[54076]: debug 2022-01-31T21:45:02.367+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:02.368264+0000) 2022-01-31T21:45:02.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:02 smithi167 conmon[60316]: debug 2022-01-31T21:45:02.367+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:02.368263+0000) 2022-01-31T21:45:03.046 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:02 smithi171 conmon[51620]: debug 2022-01-31T21:45:02.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:02.976487+0000) 2022-01-31T21:45:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:02 smithi167 conmon[49112]: debug 2022-01-31T21:45:02.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:02.704674+0000) 2022-01-31T21:45:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[41853]: debug 2022-01-31T21:45:03.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.091363+0000) 2022-01-31T21:45:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[46715]: debug 2022-01-31T21:45:03.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.135953+0000) 2022-01-31T21:45:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:03 smithi167 conmon[54076]: debug 2022-01-31T21:45:03.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.368453+0000) 2022-01-31T21:45:03.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:03 smithi167 conmon[60316]: debug 2022-01-31T21:45:03.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.368441+0000) 2022-01-31T21:45:04.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[35325]: debug 2022-01-31T21:45:03.821+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:04.047 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[41853]: debug 2022-01-31T21:45:03.805+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.805991+0000) 2022-01-31T21:45:04.048 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[46715]: debug 2022-01-31T21:45:03.804+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.804998+0000) 2022-01-31T21:45:04.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[51620]: debug 2022-01-31T21:45:03.805+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.805865+0000) 2022-01-31T21:45:04.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:03 smithi171 conmon[51620]: debug 2022-01-31T21:45:03.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.976652+0000) 2022-01-31T21:45:04.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:03 smithi167 conmon[54076]: debug 2022-01-31T21:45:03.803+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.804246+0000) 2022-01-31T21:45:04.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:03 smithi167 conmon[60316]: debug 2022-01-31T21:45:03.804+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.805081+0000) 2022-01-31T21:45:04.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:03 smithi167 conmon[49112]: debug 2022-01-31T21:45:03.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.704874+0000) 2022-01-31T21:45:04.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:03 smithi167 conmon[49112]: debug 2022-01-31T21:45:03.804+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:03.804758+0000) 2022-01-31T21:45:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:04 smithi171 conmon[41853]: debug 2022-01-31T21:45:04.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:04.091539+0000) 2022-01-31T21:45:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:04 smithi171 conmon[46715]: debug 2022-01-31T21:45:04.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:04.136115+0000) 2022-01-31T21:45:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:04 smithi167 conmon[54076]: debug 2022-01-31T21:45:04.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:04.368592+0000) 2022-01-31T21:45:04.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:04 smithi167 conmon[60316]: debug 2022-01-31T21:45:04.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:04.368608+0000) 2022-01-31T21:45:05.046 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:04 smithi171 conmon[51620]: debug 2022-01-31T21:45:04.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:04.976840+0000) 2022-01-31T21:45:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:04 smithi167 conmon[49112]: debug 2022-01-31T21:45:04.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:04.705108+0000) 2022-01-31T21:45:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:05 smithi171 conmon[46715]: debug 2022-01-31T21:45:05.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:05.136255+0000) 2022-01-31T21:45:05.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:05 smithi171 conmon[41853]: debug 2022-01-31T21:45:05.090+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:05.091711+0000) 2022-01-31T21:45:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:05 smithi167 conmon[54076]: debug 2022-01-31T21:45:05.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:05.368743+0000) 2022-01-31T21:45:05.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:05 smithi167 conmon[60316]: debug 2022-01-31T21:45:05.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:05.368722+0000) 2022-01-31T21:45:06.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:05 smithi171 conmon[51620]: debug 2022-01-31T21:45:05.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:05.976989+0000) 2022-01-31T21:45:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:05 smithi167 conmon[49112]: debug 2022-01-31T21:45:05.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:05.705211+0000) 2022-01-31T21:45:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:06 smithi171 conmon[41853]: debug 2022-01-31T21:45:06.091+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:06.091936+0000) 2022-01-31T21:45:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:06 smithi171 conmon[46715]: debug 2022-01-31T21:45:06.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:06.136443+0000) 2022-01-31T21:45:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:06 smithi167 conmon[54076]: debug 2022-01-31T21:45:06.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:06.368935+0000) 2022-01-31T21:45:06.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:06 smithi167 conmon[60316]: debug 2022-01-31T21:45:06.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:06.368933+0000) 2022-01-31T21:45:07.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:06 smithi171 conmon[51620]: debug 2022-01-31T21:45:06.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:06.977137+0000) 2022-01-31T21:45:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:06 smithi167 conmon[49112]: debug 2022-01-31T21:45:06.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:06.705373+0000) 2022-01-31T21:45:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:07 smithi171 conmon[41853]: debug 2022-01-31T21:45:07.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:07.092172+0000) 2022-01-31T21:45:07.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:07 smithi171 conmon[46715]: debug 2022-01-31T21:45:07.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:07.136655+0000) 2022-01-31T21:45:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:07 smithi167 conmon[54076]: debug 2022-01-31T21:45:07.368+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:07.369080+0000) 2022-01-31T21:45:07.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:07 smithi167 conmon[60316]: debug 2022-01-31T21:45:07.368+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:07.369086+0000) 2022-01-31T21:45:08.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:07 smithi171 conmon[51620]: debug 2022-01-31T21:45:07.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:07.977298+0000) 2022-01-31T21:45:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:07 smithi167 conmon[49112]: debug 2022-01-31T21:45:07.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:07.705566+0000) 2022-01-31T21:45:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[41853]: debug 2022-01-31T21:45:08.091+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.092370+0000) 2022-01-31T21:45:08.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[46715]: debug 2022-01-31T21:45:08.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.136841+0000) 2022-01-31T21:45:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:08 smithi167 conmon[54076]: debug 2022-01-31T21:45:08.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.369248+0000) 2022-01-31T21:45:08.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:08 smithi167 conmon[60316]: debug 2022-01-31T21:45:08.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.369256+0000) 2022-01-31T21:45:09.047 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[35325]: debug 2022-01-31T21:45:08.734+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:45:09.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[35325]: debug 2022-01-31T21:45:08.840+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154683 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:09.048 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[41853]: debug 2022-01-31T21:45:08.824+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.825591+0000) 2022-01-31T21:45:09.049 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[46715]: debug 2022-01-31T21:45:08.823+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.824598+0000) 2022-01-31T21:45:09.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[51620]: debug 2022-01-31T21:45:08.824+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.825408+0000) 2022-01-31T21:45:09.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:08 smithi171 conmon[51620]: debug 2022-01-31T21:45:08.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.977469+0000) 2022-01-31T21:45:09.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:08 smithi167 conmon[54076]: debug 2022-01-31T21:45:08.822+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.822923+0000) 2022-01-31T21:45:09.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:08 smithi167 conmon[60316]: debug 2022-01-31T21:45:08.822+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.823142+0000) 2022-01-31T21:45:09.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:08 smithi167 conmon[49112]: debug 2022-01-31T21:45:08.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.705759+0000) 2022-01-31T21:45:09.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:08 smithi167 conmon[49112]: debug 2022-01-31T21:45:08.823+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:08.823886+0000) 2022-01-31T21:45:09.359 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:09 smithi171 conmon[41853]: debug 2022-01-31T21:45:09.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:09.092531+0000) 2022-01-31T21:45:09.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:09 smithi171 conmon[46715]: debug 2022-01-31T21:45:09.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:09.136941+0000) 2022-01-31T21:45:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:09 smithi167 conmon[54076]: debug 2022-01-31T21:45:09.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:09.369439+0000) 2022-01-31T21:45:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:09 smithi167 conmon[60316]: debug 2022-01-31T21:45:09.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:09.369385+0000) 2022-01-31T21:45:10.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:09 smithi171 conmon[51620]: debug 2022-01-31T21:45:09.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:09.977610+0000) 2022-01-31T21:45:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:09 smithi167 conmon[49112]: debug 2022-01-31T21:45:09.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:09.705921+0000) 2022-01-31T21:45:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:10 smithi171 conmon[41853]: debug 2022-01-31T21:45:10.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:10.092675+0000) 2022-01-31T21:45:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:10 smithi171 conmon[46715]: debug 2022-01-31T21:45:10.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:10.137102+0000) 2022-01-31T21:45:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:10 smithi167 conmon[54076]: debug 2022-01-31T21:45:10.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:10.369587+0000) 2022-01-31T21:45:10.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:10 smithi167 conmon[60316]: debug 2022-01-31T21:45:10.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:10.369571+0000) 2022-01-31T21:45:11.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:10 smithi171 conmon[51620]: debug 2022-01-31T21:45:10.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:10.977814+0000) 2022-01-31T21:45:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:10 smithi167 conmon[49112]: debug 2022-01-31T21:45:10.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:10.706075+0000) 2022-01-31T21:45:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:11 smithi171 conmon[46715]: debug 2022-01-31T21:45:11.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:11.137285+0000) 2022-01-31T21:45:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:11 smithi171 conmon[41853]: debug 2022-01-31T21:45:11.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:11.092872+0000) 2022-01-31T21:45:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:11 smithi167 conmon[54076]: debug 2022-01-31T21:45:11.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:11.369736+0000) 2022-01-31T21:45:11.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:11 smithi167 conmon[60316]: debug 2022-01-31T21:45:11.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:11.369775+0000) 2022-01-31T21:45:12.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:11 smithi171 conmon[51620]: debug 2022-01-31T21:45:11.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:11.978020+0000) 2022-01-31T21:45:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:11 smithi167 conmon[49112]: debug 2022-01-31T21:45:11.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:11.706293+0000) 2022-01-31T21:45:12.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:12 smithi171 conmon[41853]: debug 2022-01-31T21:45:12.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:12.093021+0000) 2022-01-31T21:45:12.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:12 smithi171 conmon[46715]: debug 2022-01-31T21:45:12.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:12.137472+0000) 2022-01-31T21:45:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:12 smithi167 conmon[54076]: debug 2022-01-31T21:45:12.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:12.369922+0000) 2022-01-31T21:45:12.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:12 smithi167 conmon[60316]: debug 2022-01-31T21:45:12.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:12.369960+0000) 2022-01-31T21:45:13.048 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:12 smithi171 conmon[51620]: debug 2022-01-31T21:45:12.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:12.978210+0000) 2022-01-31T21:45:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:12 smithi167 conmon[49112]: debug 2022-01-31T21:45:12.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:12.706506+0000) 2022-01-31T21:45:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[41853]: debug 2022-01-31T21:45:13.091+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.093203+0000) 2022-01-31T21:45:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[46715]: debug 2022-01-31T21:45:13.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.137666+0000) 2022-01-31T21:45:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:13 smithi167 conmon[54076]: debug 2022-01-31T21:45:13.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.370114+0000) 2022-01-31T21:45:13.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:13 smithi167 conmon[60316]: debug 2022-01-31T21:45:13.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.370148+0000) 2022-01-31T21:45:14.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[35325]: debug 2022-01-31T21:45:13.857+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154793 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:14.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[41853]: debug 2022-01-31T21:45:13.842+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.844298+0000) 2022-01-31T21:45:14.051 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[46715]: debug 2022-01-31T21:45:13.841+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.843047+0000) 2022-01-31T21:45:14.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[51620]: debug 2022-01-31T21:45:13.842+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.844329+0000) 2022-01-31T21:45:14.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:13 smithi171 conmon[51620]: debug 2022-01-31T21:45:13.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.978376+0000) 2022-01-31T21:45:14.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:13 smithi167 conmon[54076]: debug 2022-01-31T21:45:13.843+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.843637+0000) 2022-01-31T21:45:14.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:13 smithi167 conmon[60316]: debug 2022-01-31T21:45:13.842+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.842988+0000) 2022-01-31T21:45:14.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:13 smithi167 conmon[49112]: debug 2022-01-31T21:45:13.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.706668+0000) 2022-01-31T21:45:14.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:13 smithi167 conmon[49112]: debug 2022-01-31T21:45:13.842+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:13.842747+0000) 2022-01-31T21:45:14.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:14 smithi171 conmon[46715]: debug 2022-01-31T21:45:14.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:14.137858+0000) 2022-01-31T21:45:14.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:14 smithi171 conmon[41853]: debug 2022-01-31T21:45:14.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:14.093392+0000) 2022-01-31T21:45:14.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:14 smithi167 conmon[60316]: debug 2022-01-31T21:45:14.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:14.370344+0000) 2022-01-31T21:45:14.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:14 smithi167 conmon[54076]: debug 2022-01-31T21:45:14.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:14.370313+0000) 2022-01-31T21:45:15.048 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:14 smithi171 conmon[51620]: debug 2022-01-31T21:45:14.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:14.978558+0000) 2022-01-31T21:45:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:14 smithi167 conmon[49112]: debug 2022-01-31T21:45:14.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:14.706854+0000) 2022-01-31T21:45:15.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:15 smithi171 conmon[46715]: debug 2022-01-31T21:45:15.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:15.138030+0000) 2022-01-31T21:45:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:15 smithi171 conmon[41853]: debug 2022-01-31T21:45:15.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:15.093549+0000) 2022-01-31T21:45:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:15 smithi167 conmon[54076]: debug 2022-01-31T21:45:15.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:15.370471+0000) 2022-01-31T21:45:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:15 smithi167 conmon[60316]: debug 2022-01-31T21:45:15.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:15.370464+0000) 2022-01-31T21:45:16.048 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:15 smithi171 conmon[51620]: debug 2022-01-31T21:45:15.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:15.978769+0000) 2022-01-31T21:45:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:15 smithi167 conmon[49112]: debug 2022-01-31T21:45:15.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:15.707022+0000) 2022-01-31T21:45:16.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:16 smithi171 conmon[41853]: debug 2022-01-31T21:45:16.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:16.093724+0000) 2022-01-31T21:45:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:16 smithi171 conmon[46715]: debug 2022-01-31T21:45:16.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:16.138240+0000) 2022-01-31T21:45:16.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:16 smithi167 conmon[54076]: debug 2022-01-31T21:45:16.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:16.370723+0000) 2022-01-31T21:45:16.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:16 smithi167 conmon[60316]: debug 2022-01-31T21:45:16.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:16.370729+0000) 2022-01-31T21:45:16.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:16 smithi167 conmon[49112]: debug 2022-01-31T21:45:16.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:16.707237+0000) 2022-01-31T21:45:17.048 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:16 smithi171 conmon[51620]: debug 2022-01-31T21:45:16.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:16.978982+0000) 2022-01-31T21:45:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:17 smithi171 conmon[46715]: debug 2022-01-31T21:45:17.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:17.138427+0000) 2022-01-31T21:45:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:17 smithi171 conmon[41853]: debug 2022-01-31T21:45:17.092+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:17.093917+0000) 2022-01-31T21:45:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:17 smithi167 conmon[54076]: debug 2022-01-31T21:45:17.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:17.370940+0000) 2022-01-31T21:45:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:17 smithi167 conmon[60316]: debug 2022-01-31T21:45:17.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:17.370935+0000) 2022-01-31T21:45:18.048 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:17 smithi171 conmon[51620]: debug 2022-01-31T21:45:17.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:17.979190+0000) 2022-01-31T21:45:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:17 smithi167 conmon[49112]: debug 2022-01-31T21:45:17.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:17.707463+0000) 2022-01-31T21:45:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[41853]: debug 2022-01-31T21:45:18.093+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.094071+0000) 2022-01-31T21:45:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[46715]: debug 2022-01-31T21:45:18.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.138581+0000) 2022-01-31T21:45:18.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:18 smithi167 conmon[60316]: debug 2022-01-31T21:45:18.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.371112+0000) 2022-01-31T21:45:18.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:18 smithi167 conmon[54076]: debug 2022-01-31T21:45:18.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.371117+0000) 2022-01-31T21:45:19.049 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[35325]: debug 2022-01-31T21:45:18.883+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 154904 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:19.049 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[41853]: debug 2022-01-31T21:45:18.860+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.862401+0000) 2022-01-31T21:45:19.050 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[46715]: debug 2022-01-31T21:45:18.859+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.861610+0000) 2022-01-31T21:45:19.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[51620]: debug 2022-01-31T21:45:18.861+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.862772+0000) 2022-01-31T21:45:19.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:18 smithi171 conmon[51620]: debug 2022-01-31T21:45:18.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.979354+0000) 2022-01-31T21:45:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:18 smithi167 conmon[49112]: debug 2022-01-31T21:45:18.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.707726+0000) 2022-01-31T21:45:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:18 smithi167 conmon[49112]: debug 2022-01-31T21:45:18.860+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.861600+0000) 2022-01-31T21:45:19.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:18 smithi167 conmon[54076]: debug 2022-01-31T21:45:18.860+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.862107+0000) 2022-01-31T21:45:19.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:18 smithi167 conmon[60316]: debug 2022-01-31T21:45:18.859+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:18.860915+0000) 2022-01-31T21:45:19.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:19 smithi171 conmon[46715]: debug 2022-01-31T21:45:19.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:19.138796+0000) 2022-01-31T21:45:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:19 smithi171 conmon[41853]: debug 2022-01-31T21:45:19.093+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:19.094241+0000) 2022-01-31T21:45:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:19 smithi167 conmon[54076]: debug 2022-01-31T21:45:19.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:19.371217+0000) 2022-01-31T21:45:19.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:19 smithi167 conmon[60316]: debug 2022-01-31T21:45:19.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:19.371222+0000) 2022-01-31T21:45:20.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:19 smithi171 conmon[51620]: debug 2022-01-31T21:45:19.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:19.979512+0000) 2022-01-31T21:45:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:19 smithi167 conmon[49112]: debug 2022-01-31T21:45:19.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:19.707910+0000) 2022-01-31T21:45:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:20 smithi171 conmon[41853]: debug 2022-01-31T21:45:20.093+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:20.094407+0000) 2022-01-31T21:45:20.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:20 smithi171 conmon[46715]: debug 2022-01-31T21:45:20.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:20.138982+0000) 2022-01-31T21:45:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:20 smithi167 conmon[54076]: debug 2022-01-31T21:45:20.369+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:20.371399+0000) 2022-01-31T21:45:20.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:20 smithi167 conmon[60316]: debug 2022-01-31T21:45:20.369+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:20.371403+0000) 2022-01-31T21:45:21.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:20 smithi171 conmon[51620]: debug 2022-01-31T21:45:20.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:20.979727+0000) 2022-01-31T21:45:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:20 smithi167 conmon[49112]: debug 2022-01-31T21:45:20.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:20.708035+0000) 2022-01-31T21:45:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:21 smithi171 conmon[41853]: debug 2022-01-31T21:45:21.093+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:21.094604+0000) 2022-01-31T21:45:21.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:21 smithi171 conmon[46715]: debug 2022-01-31T21:45:21.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:21.139166+0000) 2022-01-31T21:45:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:21 smithi167 conmon[54076]: debug 2022-01-31T21:45:21.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:21.371664+0000) 2022-01-31T21:45:21.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:21 smithi167 conmon[60316]: debug 2022-01-31T21:45:21.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:21.371654+0000) 2022-01-31T21:45:22.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:21 smithi171 conmon[51620]: debug 2022-01-31T21:45:21.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:21.979917+0000) 2022-01-31T21:45:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:21 smithi167 conmon[49112]: debug 2022-01-31T21:45:21.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:21.708198+0000) 2022-01-31T21:45:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:22 smithi171 conmon[41853]: debug 2022-01-31T21:45:22.093+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:22.094812+0000) 2022-01-31T21:45:22.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:22 smithi171 conmon[46715]: debug 2022-01-31T21:45:22.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:22.139341+0000) 2022-01-31T21:45:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:22 smithi167 conmon[54076]: debug 2022-01-31T21:45:22.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:22.371830+0000) 2022-01-31T21:45:22.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:22 smithi167 conmon[60316]: debug 2022-01-31T21:45:22.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:22.371831+0000) 2022-01-31T21:45:23.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:22 smithi171 conmon[51620]: debug 2022-01-31T21:45:22.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:22.980048+0000) 2022-01-31T21:45:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:22 smithi167 conmon[49112]: debug 2022-01-31T21:45:22.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:22.708367+0000) 2022-01-31T21:45:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[41853]: debug 2022-01-31T21:45:23.094+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.095005+0000) 2022-01-31T21:45:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[46715]: debug 2022-01-31T21:45:23.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.139534+0000) 2022-01-31T21:45:23.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:23 smithi167 conmon[54076]: debug 2022-01-31T21:45:23.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.371991+0000) 2022-01-31T21:45:23.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:23 smithi167 conmon[60316]: debug 2022-01-31T21:45:23.371+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.371991+0000) 2022-01-31T21:45:24.049 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[46715]: debug 2022-01-31T21:45:23.886+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.886829+0000) 2022-01-31T21:45:24.050 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[35325]: debug 2022-01-31T21:45:23.734+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:45:24.051 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[35325]: debug 2022-01-31T21:45:23.902+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155015 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:24.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[41853]: debug 2022-01-31T21:45:23.887+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.888499+0000) 2022-01-31T21:45:24.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[51620]: debug 2022-01-31T21:45:23.886+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.887678+0000) 2022-01-31T21:45:24.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:23 smithi171 conmon[51620]: debug 2022-01-31T21:45:23.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.980235+0000) 2022-01-31T21:45:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:23 smithi167 conmon[49112]: debug 2022-01-31T21:45:23.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.708566+0000) 2022-01-31T21:45:24.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:23 smithi167 conmon[49112]: debug 2022-01-31T21:45:23.885+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.887303+0000) 2022-01-31T21:45:24.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:23 smithi167 conmon[54076]: debug 2022-01-31T21:45:23.885+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.887105+0000) 2022-01-31T21:45:24.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:23 smithi167 conmon[60316]: debug 2022-01-31T21:45:23.885+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:23.886792+0000) 2022-01-31T21:45:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:24 smithi171 conmon[41853]: debug 2022-01-31T21:45:24.094+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:24.095245+0000) 2022-01-31T21:45:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:24 smithi171 conmon[46715]: debug 2022-01-31T21:45:24.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:24.139694+0000) 2022-01-31T21:45:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:24 smithi167 conmon[54076]: debug 2022-01-31T21:45:24.370+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:24.372181+0000) 2022-01-31T21:45:24.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:24 smithi167 conmon[60316]: debug 2022-01-31T21:45:24.370+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:24.372189+0000) 2022-01-31T21:45:25.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:24 smithi171 conmon[51620]: debug 2022-01-31T21:45:24.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:24.980397+0000) 2022-01-31T21:45:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:24 smithi167 conmon[49112]: debug 2022-01-31T21:45:24.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:24.708761+0000) 2022-01-31T21:45:25.359 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:25 smithi171 conmon[41853]: debug 2022-01-31T21:45:25.094+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:25.095397+0000) 2022-01-31T21:45:25.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:25 smithi171 conmon[46715]: debug 2022-01-31T21:45:25.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:25.139831+0000) 2022-01-31T21:45:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:25 smithi167 conmon[54076]: debug 2022-01-31T21:45:25.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:25.372366+0000) 2022-01-31T21:45:25.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:25 smithi167 conmon[60316]: debug 2022-01-31T21:45:25.371+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:25.372368+0000) 2022-01-31T21:45:26.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:25 smithi171 conmon[51620]: debug 2022-01-31T21:45:25.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:25.980576+0000) 2022-01-31T21:45:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:25 smithi167 conmon[49112]: debug 2022-01-31T21:45:25.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:25.708920+0000) 2022-01-31T21:45:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:26 smithi171 conmon[41853]: debug 2022-01-31T21:45:26.094+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:26.095591+0000) 2022-01-31T21:45:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:26 smithi171 conmon[46715]: debug 2022-01-31T21:45:26.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:26.140018+0000) 2022-01-31T21:45:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:26 smithi167 conmon[54076]: debug 2022-01-31T21:45:26.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:26.372574+0000) 2022-01-31T21:45:26.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:26 smithi167 conmon[60316]: debug 2022-01-31T21:45:26.371+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:26.372571+0000) 2022-01-31T21:45:27.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:26 smithi171 conmon[51620]: debug 2022-01-31T21:45:26.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:26.980780+0000) 2022-01-31T21:45:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:26 smithi167 conmon[49112]: debug 2022-01-31T21:45:26.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:26.709090+0000) 2022-01-31T21:45:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:27 smithi171 conmon[41853]: debug 2022-01-31T21:45:27.095+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:27.095784+0000) 2022-01-31T21:45:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:27 smithi171 conmon[46715]: debug 2022-01-31T21:45:27.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:27.140200+0000) 2022-01-31T21:45:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:27 smithi167 conmon[54076]: debug 2022-01-31T21:45:27.371+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:27.372772+0000) 2022-01-31T21:45:27.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:27 smithi167 conmon[60316]: debug 2022-01-31T21:45:27.371+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:27.372775+0000) 2022-01-31T21:45:28.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:27 smithi171 conmon[51620]: debug 2022-01-31T21:45:27.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:27.981011+0000) 2022-01-31T21:45:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:27 smithi167 conmon[49112]: debug 2022-01-31T21:45:27.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:27.709314+0000) 2022-01-31T21:45:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[46715]: debug 2022-01-31T21:45:28.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.140381+0000) 2022-01-31T21:45:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[41853]: debug 2022-01-31T21:45:28.095+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.095912+0000) 2022-01-31T21:45:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:28 smithi167 conmon[54076]: debug 2022-01-31T21:45:28.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.372933+0000) 2022-01-31T21:45:28.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:28 smithi167 conmon[60316]: debug 2022-01-31T21:45:28.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.372953+0000) 2022-01-31T21:45:29.050 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[35325]: debug 2022-01-31T21:45:28.921+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155126 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:29.051 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[41853]: debug 2022-01-31T21:45:28.906+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.906768+0000) 2022-01-31T21:45:29.052 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[46715]: debug 2022-01-31T21:45:28.904+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.905212+0000) 2022-01-31T21:45:29.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[51620]: debug 2022-01-31T21:45:28.905+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.906584+0000) 2022-01-31T21:45:29.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:28 smithi171 conmon[51620]: debug 2022-01-31T21:45:28.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.981196+0000) 2022-01-31T21:45:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:28 smithi167 conmon[49112]: debug 2022-01-31T21:45:28.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.709551+0000) 2022-01-31T21:45:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:28 smithi167 conmon[49112]: debug 2022-01-31T21:45:28.903+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.905121+0000) 2022-01-31T21:45:29.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:28 smithi167 conmon[54076]: debug 2022-01-31T21:45:28.903+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.905366+0000) 2022-01-31T21:45:29.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:28 smithi167 conmon[60316]: debug 2022-01-31T21:45:28.903+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:28.904992+0000) 2022-01-31T21:45:29.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:29 smithi171 conmon[41853]: debug 2022-01-31T21:45:29.095+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:29.096090+0000) 2022-01-31T21:45:29.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:29 smithi171 conmon[46715]: debug 2022-01-31T21:45:29.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:29.140568+0000) 2022-01-31T21:45:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:29 smithi167 conmon[54076]: debug 2022-01-31T21:45:29.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:29.373122+0000) 2022-01-31T21:45:29.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:29 smithi167 conmon[60316]: debug 2022-01-31T21:45:29.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:29.373129+0000) 2022-01-31T21:45:30.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:29 smithi171 conmon[51620]: debug 2022-01-31T21:45:29.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:29.981371+0000) 2022-01-31T21:45:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:29 smithi167 conmon[49112]: debug 2022-01-31T21:45:29.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:29.709762+0000) 2022-01-31T21:45:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:30 smithi171 conmon[41853]: debug 2022-01-31T21:45:30.095+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:30.096242+0000) 2022-01-31T21:45:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:30 smithi171 conmon[46715]: debug 2022-01-31T21:45:30.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:30.140727+0000) 2022-01-31T21:45:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:30 smithi167 conmon[54076]: debug 2022-01-31T21:45:30.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:30.373323+0000) 2022-01-31T21:45:30.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:30 smithi167 conmon[60316]: debug 2022-01-31T21:45:30.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:30.373279+0000) 2022-01-31T21:45:31.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:30 smithi171 conmon[51620]: debug 2022-01-31T21:45:30.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:30.981583+0000) 2022-01-31T21:45:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:30 smithi167 conmon[49112]: debug 2022-01-31T21:45:30.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:30.709956+0000) 2022-01-31T21:45:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:31 smithi171 conmon[41853]: debug 2022-01-31T21:45:31.095+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:31.096428+0000) 2022-01-31T21:45:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:31 smithi171 conmon[46715]: debug 2022-01-31T21:45:31.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:31.140972+0000) 2022-01-31T21:45:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:31 smithi167 conmon[54076]: debug 2022-01-31T21:45:31.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:31.373481+0000) 2022-01-31T21:45:31.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:31 smithi167 conmon[60316]: debug 2022-01-31T21:45:31.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:31.373513+0000) 2022-01-31T21:45:32.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:31 smithi171 conmon[51620]: debug 2022-01-31T21:45:31.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:31.981799+0000) 2022-01-31T21:45:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:31 smithi167 conmon[49112]: debug 2022-01-31T21:45:31.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:31.710195+0000) 2022-01-31T21:45:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:32 smithi171 conmon[46715]: debug 2022-01-31T21:45:32.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:32.141130+0000) 2022-01-31T21:45:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:32 smithi171 conmon[41853]: debug 2022-01-31T21:45:32.095+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:32.096599+0000) 2022-01-31T21:45:32.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:32 smithi167 conmon[60316]: debug 2022-01-31T21:45:32.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:32.373643+0000) 2022-01-31T21:45:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:32 smithi167 conmon[54076]: debug 2022-01-31T21:45:32.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:32.373681+0000) 2022-01-31T21:45:33.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:32 smithi171 conmon[51620]: debug 2022-01-31T21:45:32.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:32.981990+0000) 2022-01-31T21:45:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:32 smithi167 conmon[49112]: debug 2022-01-31T21:45:32.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:32.710384+0000) 2022-01-31T21:45:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[41853]: debug 2022-01-31T21:45:33.096+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.096786+0000) 2022-01-31T21:45:33.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[46715]: debug 2022-01-31T21:45:33.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.141351+0000) 2022-01-31T21:45:33.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:33 smithi167 conmon[60316]: debug 2022-01-31T21:45:33.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.373836+0000) 2022-01-31T21:45:33.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:33 smithi167 conmon[54076]: debug 2022-01-31T21:45:33.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.373878+0000) 2022-01-31T21:45:34.051 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[46715]: debug 2022-01-31T21:45:33.923+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.924369+0000) 2022-01-31T21:45:34.052 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[35325]: debug 2022-01-31T21:45:33.941+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155237 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:34.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[41853]: debug 2022-01-31T21:45:33.925+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.925790+0000) 2022-01-31T21:45:34.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[51620]: debug 2022-01-31T21:45:33.924+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.925104+0000) 2022-01-31T21:45:34.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:33 smithi171 conmon[51620]: debug 2022-01-31T21:45:33.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.982213+0000) 2022-01-31T21:45:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:33 smithi167 conmon[49112]: debug 2022-01-31T21:45:33.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.710588+0000) 2022-01-31T21:45:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:33 smithi167 conmon[49112]: debug 2022-01-31T21:45:33.921+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.923246+0000) 2022-01-31T21:45:34.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:33 smithi167 conmon[54076]: debug 2022-01-31T21:45:33.922+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.924319+0000) 2022-01-31T21:45:34.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:33 smithi167 conmon[60316]: debug 2022-01-31T21:45:33.922+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:33.923620+0000) 2022-01-31T21:45:34.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:34 smithi171 conmon[41853]: debug 2022-01-31T21:45:34.096+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:34.096967+0000) 2022-01-31T21:45:34.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:34 smithi171 conmon[46715]: debug 2022-01-31T21:45:34.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:34.141531+0000) 2022-01-31T21:45:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:34 smithi167 conmon[54076]: debug 2022-01-31T21:45:34.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:34.374057+0000) 2022-01-31T21:45:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:34 smithi167 conmon[60316]: debug 2022-01-31T21:45:34.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:34.374041+0000) 2022-01-31T21:45:35.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:34 smithi171 conmon[51620]: debug 2022-01-31T21:45:34.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:34.982378+0000) 2022-01-31T21:45:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:34 smithi167 conmon[49112]: debug 2022-01-31T21:45:34.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:34.710732+0000) 2022-01-31T21:45:35.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:35 smithi171 conmon[41853]: debug 2022-01-31T21:45:35.096+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:35.097100+0000) 2022-01-31T21:45:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:35 smithi171 conmon[46715]: debug 2022-01-31T21:45:35.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:35.141652+0000) 2022-01-31T21:45:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:35 smithi167 conmon[54076]: debug 2022-01-31T21:45:35.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:35.374233+0000) 2022-01-31T21:45:35.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:35 smithi167 conmon[60316]: debug 2022-01-31T21:45:35.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:35.374231+0000) 2022-01-31T21:45:36.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:35 smithi171 conmon[51620]: debug 2022-01-31T21:45:35.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:35.982493+0000) 2022-01-31T21:45:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:35 smithi167 conmon[49112]: debug 2022-01-31T21:45:35.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:35.710873+0000) 2022-01-31T21:45:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:36 smithi171 conmon[41853]: debug 2022-01-31T21:45:36.096+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:36.097230+0000) 2022-01-31T21:45:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:36 smithi171 conmon[46715]: debug 2022-01-31T21:45:36.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:36.141836+0000) 2022-01-31T21:45:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:36 smithi167 conmon[54076]: debug 2022-01-31T21:45:36.372+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:36.374416+0000) 2022-01-31T21:45:36.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:36 smithi167 conmon[60316]: debug 2022-01-31T21:45:36.372+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:36.374436+0000) 2022-01-31T21:45:37.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:36 smithi171 conmon[51620]: debug 2022-01-31T21:45:36.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:36.982685+0000) 2022-01-31T21:45:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:36 smithi167 conmon[49112]: debug 2022-01-31T21:45:36.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:36.711086+0000) 2022-01-31T21:45:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:37 smithi171 conmon[41853]: debug 2022-01-31T21:45:37.096+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:37.097412+0000) 2022-01-31T21:45:37.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:37 smithi171 conmon[46715]: debug 2022-01-31T21:45:37.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:37.142042+0000) 2022-01-31T21:45:37.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:37 smithi167 conmon[54076]: debug 2022-01-31T21:45:37.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:37.374624+0000) 2022-01-31T21:45:37.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:37 smithi167 conmon[60316]: debug 2022-01-31T21:45:37.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:37.374643+0000) 2022-01-31T21:45:38.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:37 smithi171 conmon[51620]: debug 2022-01-31T21:45:37.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:37.982793+0000) 2022-01-31T21:45:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:37 smithi167 conmon[49112]: debug 2022-01-31T21:45:37.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:37.711298+0000) 2022-01-31T21:45:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[46715]: debug 2022-01-31T21:45:38.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.142199+0000) 2022-01-31T21:45:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[41853]: debug 2022-01-31T21:45:38.096+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.097539+0000) 2022-01-31T21:45:38.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:38 smithi167 conmon[54076]: debug 2022-01-31T21:45:38.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.374836+0000) 2022-01-31T21:45:38.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:38 smithi167 conmon[60316]: debug 2022-01-31T21:45:38.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.374816+0000) 2022-01-31T21:45:38.941 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:38 smithi167 conmon[49112]: debug 2022-01-31T21:45:38.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.711547+0000) 2022-01-31T21:45:39.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[41853]: debug 2022-01-31T21:45:38.944+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.945646+0000) 2022-01-31T21:45:39.053 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[46715]: debug 2022-01-31T21:45:38.943+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.944247+0000) 2022-01-31T21:45:39.053 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[35325]: debug 2022-01-31T21:45:38.735+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:45:39.054 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[35325]: debug 2022-01-31T21:45:38.960+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155347 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:39.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[51620]: debug 2022-01-31T21:45:38.944+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.944918+0000) 2022-01-31T21:45:39.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:38 smithi171 conmon[51620]: debug 2022-01-31T21:45:38.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.982955+0000) 2022-01-31T21:45:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:39 smithi171 conmon[41853]: debug 2022-01-31T21:45:39.097+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:39.097736+0000) 2022-01-31T21:45:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:39 smithi171 conmon[46715]: debug 2022-01-31T21:45:39.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:39.142375+0000) 2022-01-31T21:45:39.373 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:38 smithi167 conmon[49112]: debug 2022-01-31T21:45:38.943+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.944428+0000) 2022-01-31T21:45:39.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:38 smithi167 conmon[54076]: debug 2022-01-31T21:45:38.943+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.944336+0000) 2022-01-31T21:45:39.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:38 smithi167 conmon[60316]: debug 2022-01-31T21:45:38.943+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:38.944110+0000) 2022-01-31T21:45:39.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:39 smithi167 conmon[60316]: debug 2022-01-31T21:45:39.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:39.374996+0000) 2022-01-31T21:45:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:39 smithi167 conmon[54076]: debug 2022-01-31T21:45:39.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:39.374980+0000) 2022-01-31T21:45:40.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:39 smithi171 conmon[51620]: debug 2022-01-31T21:45:39.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:39.983075+0000) 2022-01-31T21:45:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:39 smithi167 conmon[49112]: debug 2022-01-31T21:45:39.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:39.711762+0000) 2022-01-31T21:45:40.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:40 smithi171 conmon[46715]: debug 2022-01-31T21:45:40.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:40.142530+0000) 2022-01-31T21:45:40.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:40 smithi171 conmon[41853]: debug 2022-01-31T21:45:40.097+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:40.097914+0000) 2022-01-31T21:45:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:40 smithi167 conmon[54076]: debug 2022-01-31T21:45:40.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:40.375174+0000) 2022-01-31T21:45:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:40 smithi167 conmon[60316]: debug 2022-01-31T21:45:40.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:40.375157+0000) 2022-01-31T21:45:41.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:40 smithi171 conmon[51620]: debug 2022-01-31T21:45:40.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:40.983262+0000) 2022-01-31T21:45:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:40 smithi167 conmon[49112]: debug 2022-01-31T21:45:40.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:40.711940+0000) 2022-01-31T21:45:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:41 smithi171 conmon[46715]: debug 2022-01-31T21:45:41.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:41.142743+0000) 2022-01-31T21:45:41.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:41 smithi171 conmon[41853]: debug 2022-01-31T21:45:41.097+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:41.098095+0000) 2022-01-31T21:45:41.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:41 smithi167 conmon[54076]: debug 2022-01-31T21:45:41.374+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:41.375389+0000) 2022-01-31T21:45:41.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:41 smithi167 conmon[60316]: debug 2022-01-31T21:45:41.374+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:41.375387+0000) 2022-01-31T21:45:42.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:41 smithi171 conmon[51620]: debug 2022-01-31T21:45:41.983+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:41.983473+0000) 2022-01-31T21:45:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:41 smithi167 conmon[49112]: debug 2022-01-31T21:45:41.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:41.712135+0000) 2022-01-31T21:45:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:42 smithi171 conmon[41853]: debug 2022-01-31T21:45:42.097+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:42.098278+0000) 2022-01-31T21:45:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:42 smithi171 conmon[46715]: debug 2022-01-31T21:45:42.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:42.142882+0000) 2022-01-31T21:45:42.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:42 smithi167 conmon[54076]: debug 2022-01-31T21:45:42.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:42.375601+0000) 2022-01-31T21:45:42.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:42 smithi167 conmon[60316]: debug 2022-01-31T21:45:42.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:42.375603+0000) 2022-01-31T21:45:43.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:42 smithi171 conmon[51620]: debug 2022-01-31T21:45:42.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:42.983693+0000) 2022-01-31T21:45:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:42 smithi167 conmon[49112]: debug 2022-01-31T21:45:42.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:42.712345+0000) 2022-01-31T21:45:43.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[46715]: debug 2022-01-31T21:45:43.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.143030+0000) 2022-01-31T21:45:43.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[41853]: debug 2022-01-31T21:45:43.097+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.098501+0000) 2022-01-31T21:45:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:43 smithi167 conmon[54076]: debug 2022-01-31T21:45:43.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.375819+0000) 2022-01-31T21:45:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:43 smithi167 conmon[60316]: debug 2022-01-31T21:45:43.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.375813+0000) 2022-01-31T21:45:43.961 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:43 smithi167 conmon[49112]: debug 2022-01-31T21:45:43.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.712594+0000) 2022-01-31T21:45:44.053 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[35325]: debug 2022-01-31T21:45:43.980+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155458 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:44.054 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[41853]: debug 2022-01-31T21:45:43.963+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.964154+0000) 2022-01-31T21:45:44.055 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[46715]: debug 2022-01-31T21:45:43.962+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.963090+0000) 2022-01-31T21:45:44.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[51620]: debug 2022-01-31T21:45:43.964+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.964797+0000) 2022-01-31T21:45:44.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:43 smithi171 conmon[51620]: debug 2022-01-31T21:45:43.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.984844+0000) 2022-01-31T21:45:44.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:44 smithi171 conmon[41853]: debug 2022-01-31T21:45:44.097+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:44.098720+0000) 2022-01-31T21:45:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:44 smithi171 conmon[46715]: debug 2022-01-31T21:45:44.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:44.143166+0000) 2022-01-31T21:45:44.374 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:43 smithi167 conmon[49112]: debug 2022-01-31T21:45:43.963+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.964057+0000) 2022-01-31T21:45:44.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:43 smithi167 conmon[54076]: debug 2022-01-31T21:45:43.963+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.963838+0000) 2022-01-31T21:45:44.376 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:43 smithi167 conmon[60316]: debug 2022-01-31T21:45:43.963+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:43.963661+0000) 2022-01-31T21:45:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:44 smithi167 conmon[54076]: debug 2022-01-31T21:45:44.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:44.376016+0000) 2022-01-31T21:45:44.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:44 smithi167 conmon[60316]: debug 2022-01-31T21:45:44.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:44.376019+0000) 2022-01-31T21:45:45.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:44 smithi171 conmon[51620]: debug 2022-01-31T21:45:44.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:44.984932+0000) 2022-01-31T21:45:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:44 smithi167 conmon[49112]: debug 2022-01-31T21:45:44.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:44.712790+0000) 2022-01-31T21:45:45.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:45 smithi171 conmon[41853]: debug 2022-01-31T21:45:45.098+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:45.098860+0000) 2022-01-31T21:45:45.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:45 smithi171 conmon[46715]: debug 2022-01-31T21:45:45.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:45.143299+0000) 2022-01-31T21:45:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:45 smithi167 conmon[54076]: debug 2022-01-31T21:45:45.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:45.376173+0000) 2022-01-31T21:45:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:45 smithi167 conmon[60316]: debug 2022-01-31T21:45:45.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:45.376193+0000) 2022-01-31T21:45:46.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:45 smithi171 conmon[51620]: debug 2022-01-31T21:45:45.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:45.985026+0000) 2022-01-31T21:45:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:45 smithi167 conmon[49112]: debug 2022-01-31T21:45:45.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:45.712954+0000) 2022-01-31T21:45:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:46 smithi171 conmon[41853]: debug 2022-01-31T21:45:46.098+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:46.099049+0000) 2022-01-31T21:45:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:46 smithi171 conmon[46715]: debug 2022-01-31T21:45:46.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:46.143486+0000) 2022-01-31T21:45:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:46 smithi167 conmon[54076]: debug 2022-01-31T21:45:46.375+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:46.376377+0000) 2022-01-31T21:45:46.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:46 smithi167 conmon[60316]: debug 2022-01-31T21:45:46.375+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:46.376381+0000) 2022-01-31T21:45:47.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:46 smithi171 conmon[51620]: debug 2022-01-31T21:45:46.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:46.985163+0000) 2022-01-31T21:45:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:46 smithi167 conmon[49112]: debug 2022-01-31T21:45:46.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:46.713168+0000) 2022-01-31T21:45:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:47 smithi171 conmon[41853]: debug 2022-01-31T21:45:47.098+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:47.099237+0000) 2022-01-31T21:45:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:47 smithi171 conmon[46715]: debug 2022-01-31T21:45:47.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:47.143677+0000) 2022-01-31T21:45:47.613 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:47 smithi167 conmon[54076]: debug 2022-01-31T21:45:47.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:47.376538+0000) 2022-01-31T21:45:47.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:47 smithi167 conmon[60316]: debug 2022-01-31T21:45:47.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:47.376575+0000) 2022-01-31T21:45:47.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:47 smithi167 conmon[49112]: debug 2022-01-31T21:45:47.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:47.713293+0000) 2022-01-31T21:45:48.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:47 smithi171 conmon[51620]: debug 2022-01-31T21:45:47.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:47.985307+0000) 2022-01-31T21:45:48.351 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[41853]: debug 2022-01-31T21:45:48.098+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.099418+0000) 2022-01-31T21:45:48.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[46715]: debug 2022-01-31T21:45:48.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.143894+0000) 2022-01-31T21:45:48.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:48 smithi167 conmon[54076]: debug 2022-01-31T21:45:48.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.376718+0000) 2022-01-31T21:45:48.613 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:48 smithi167 conmon[60316]: debug 2022-01-31T21:45:48.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.376728+0000) 2022-01-31T21:45:48.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:48 smithi167 conmon[49112]: debug 2022-01-31T21:45:48.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.713462+0000) 2022-01-31T21:45:49.054 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[35325]: debug 2022-01-31T21:45:48.998+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155568 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:49.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[41853]: debug 2022-01-31T21:45:48.982+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.983662+0000) 2022-01-31T21:45:49.055 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[46715]: debug 2022-01-31T21:45:48.982+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.983021+0000) 2022-01-31T21:45:49.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[51620]: debug 2022-01-31T21:45:48.983+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.984271+0000) 2022-01-31T21:45:49.057 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:48 smithi171 conmon[51620]: debug 2022-01-31T21:45:48.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.985423+0000) 2022-01-31T21:45:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:48 smithi167 conmon[49112]: debug 2022-01-31T21:45:48.982+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.982980+0000) 2022-01-31T21:45:49.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:48 smithi167 conmon[54076]: debug 2022-01-31T21:45:48.982+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.983444+0000) 2022-01-31T21:45:49.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:48 smithi167 conmon[60316]: debug 2022-01-31T21:45:48.982+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:48.982909+0000) 2022-01-31T21:45:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:49 smithi171 conmon[41853]: debug 2022-01-31T21:45:49.099+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:49.099607+0000) 2022-01-31T21:45:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:49 smithi171 conmon[46715]: debug 2022-01-31T21:45:49.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:49.144043+0000) 2022-01-31T21:45:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:49 smithi167 conmon[54076]: debug 2022-01-31T21:45:49.376+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:49.376906+0000) 2022-01-31T21:45:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:49 smithi167 conmon[60316]: debug 2022-01-31T21:45:49.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:49.376927+0000) 2022-01-31T21:45:50.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:49 smithi171 conmon[51620]: debug 2022-01-31T21:45:49.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:49.985521+0000) 2022-01-31T21:45:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:49 smithi167 conmon[49112]: debug 2022-01-31T21:45:49.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:49.713639+0000) 2022-01-31T21:45:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:50 smithi171 conmon[41853]: debug 2022-01-31T21:45:50.099+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:50.099778+0000) 2022-01-31T21:45:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:50 smithi171 conmon[46715]: debug 2022-01-31T21:45:50.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:50.144136+0000) 2022-01-31T21:45:50.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:50 smithi167 conmon[54076]: debug 2022-01-31T21:45:50.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:50.377422+0000) 2022-01-31T21:45:50.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:50 smithi167 conmon[60316]: debug 2022-01-31T21:45:50.376+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:50.377069+0000) 2022-01-31T21:45:51.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:50 smithi171 conmon[51620]: debug 2022-01-31T21:45:50.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:50.985673+0000) 2022-01-31T21:45:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:50 smithi167 conmon[49112]: debug 2022-01-31T21:45:50.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:50.713785+0000) 2022-01-31T21:45:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:51 smithi171 conmon[41853]: debug 2022-01-31T21:45:51.099+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:51.099961+0000) 2022-01-31T21:45:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:51 smithi171 conmon[46715]: debug 2022-01-31T21:45:51.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:51.144294+0000) 2022-01-31T21:45:51.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:51 smithi167 conmon[60316]: debug 2022-01-31T21:45:51.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:51.377319+0000) 2022-01-31T21:45:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:51 smithi167 conmon[54076]: debug 2022-01-31T21:45:51.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:51.378164+0000) 2022-01-31T21:45:52.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:51 smithi171 conmon[51620]: debug 2022-01-31T21:45:51.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:51.985848+0000) 2022-01-31T21:45:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:51 smithi167 conmon[49112]: debug 2022-01-31T21:45:51.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:51.714002+0000) 2022-01-31T21:45:52.320 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:52 smithi171 conmon[41853]: debug 2022-01-31T21:45:52.099+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:52.100158+0000) 2022-01-31T21:45:52.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:52 smithi171 conmon[46715]: debug 2022-01-31T21:45:52.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:52.144476+0000) 2022-01-31T21:45:52.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:52 smithi167 conmon[60316]: debug 2022-01-31T21:45:52.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:52.377541+0000) 2022-01-31T21:45:52.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:52 smithi167 conmon[54076]: debug 2022-01-31T21:45:52.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:52.378297+0000) 2022-01-31T21:45:52.939 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:52 smithi167 conmon[49112]: debug 2022-01-31T21:45:52.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:52.714220+0000) 2022-01-31T21:45:53.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:52 smithi171 conmon[51620]: debug 2022-01-31T21:45:52.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:52.985985+0000) 2022-01-31T21:45:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:53 smithi171 conmon[41853]: debug 2022-01-31T21:45:53.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:53.100335+0000) 2022-01-31T21:45:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:53 smithi171 conmon[46715]: debug 2022-01-31T21:45:53.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:53.144618+0000) 2022-01-31T21:45:53.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:53 smithi167 conmon[54076]: debug 2022-01-31T21:45:53.377+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:53.378462+0000) 2022-01-31T21:45:53.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:53 smithi167 conmon[60316]: debug 2022-01-31T21:45:53.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:53.377699+0000) 2022-01-31T21:45:53.939 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:53 smithi167 conmon[49112]: debug 2022-01-31T21:45:53.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:53.714435+0000) 2022-01-31T21:45:53.999 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:53 smithi171 conmon[35325]: debug 2022-01-31T21:45:53.736+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:45:53.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:53 smithi171 conmon[51620]: debug 2022-01-31T21:45:53.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:53.986135+0000) 2022-01-31T21:45:54.331 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:54 smithi167 conmon[49112]: debug 2022-01-31T21:45:54.000+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.000586+0000) 2022-01-31T21:45:54.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:54 smithi167 conmon[54076]: debug 2022-01-31T21:45:54.001+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.002147+0000) 2022-01-31T21:45:54.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:54 smithi167 conmon[60316]: debug 2022-01-31T21:45:54.001+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.001750+0000) 2022-01-31T21:45:54.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[35325]: debug 2022-01-31T21:45:54.017+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155679 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[51620]: debug 2022-01-31T21:45:54.002+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.003159+0000) 2022-01-31T21:45:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[41853]: debug 2022-01-31T21:45:54.002+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.002862+0000) 2022-01-31T21:45:54.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[41853]: debug 2022-01-31T21:45:54.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.100489+0000) 2022-01-31T21:45:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[46715]: debug 2022-01-31T21:45:54.001+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.001919+0000) 2022-01-31T21:45:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[46715]: debug 2022-01-31T21:45:54.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.144787+0000) 2022-01-31T21:45:54.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:54 smithi167 conmon[54076]: debug 2022-01-31T21:45:54.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.378624+0000) 2022-01-31T21:45:54.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:54 smithi167 conmon[60316]: debug 2022-01-31T21:45:54.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.377849+0000) 2022-01-31T21:45:54.939 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:54 smithi167 conmon[49112]: debug 2022-01-31T21:45:54.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.714603+0000) 2022-01-31T21:45:55.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:54 smithi171 conmon[51620]: debug 2022-01-31T21:45:54.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:54.986322+0000) 2022-01-31T21:45:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:55 smithi171 conmon[41853]: debug 2022-01-31T21:45:55.099+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:55.100649+0000) 2022-01-31T21:45:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:55 smithi171 conmon[46715]: debug 2022-01-31T21:45:55.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:55.144953+0000) 2022-01-31T21:45:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:55 smithi167 conmon[54076]: debug 2022-01-31T21:45:55.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:55.378726+0000) 2022-01-31T21:45:55.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:55 smithi167 conmon[60316]: debug 2022-01-31T21:45:55.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:55.377975+0000) 2022-01-31T21:45:55.943 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:55 smithi167 conmon[49112]: debug 2022-01-31T21:45:55.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:55.714739+0000) 2022-01-31T21:45:56.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:55 smithi171 conmon[51620]: debug 2022-01-31T21:45:55.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:55.986508+0000) 2022-01-31T21:45:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:56 smithi171 conmon[46715]: debug 2022-01-31T21:45:56.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:56.145164+0000) 2022-01-31T21:45:56.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:56 smithi171 conmon[41853]: debug 2022-01-31T21:45:56.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:56.100817+0000) 2022-01-31T21:45:56.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:56 smithi167 conmon[60316]: debug 2022-01-31T21:45:56.377+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:56.378134+0000) 2022-01-31T21:45:56.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:56 smithi167 conmon[54076]: debug 2022-01-31T21:45:56.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:56.378889+0000) 2022-01-31T21:45:56.947 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:56 smithi167 conmon[49112]: debug 2022-01-31T21:45:56.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:56.714940+0000) 2022-01-31T21:45:57.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:56 smithi171 conmon[51620]: debug 2022-01-31T21:45:56.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:56.986696+0000) 2022-01-31T21:45:57.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:57 smithi171 conmon[46715]: debug 2022-01-31T21:45:57.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:57.145309+0000) 2022-01-31T21:45:57.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:57 smithi171 conmon[41853]: debug 2022-01-31T21:45:57.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:57.100973+0000) 2022-01-31T21:45:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:57 smithi167 conmon[54076]: debug 2022-01-31T21:45:57.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:57.378992+0000) 2022-01-31T21:45:57.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:57 smithi167 conmon[60316]: debug 2022-01-31T21:45:57.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:57.378358+0000) 2022-01-31T21:45:57.950 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:57 smithi167 conmon[49112]: debug 2022-01-31T21:45:57.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:57.715156+0000) 2022-01-31T21:45:58.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:57 smithi171 conmon[51620]: debug 2022-01-31T21:45:57.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:57.986920+0000) 2022-01-31T21:45:58.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:58 smithi171 conmon[41853]: debug 2022-01-31T21:45:58.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:58.101100+0000) 2022-01-31T21:45:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:58 smithi171 conmon[46715]: debug 2022-01-31T21:45:58.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:58.145466+0000) 2022-01-31T21:45:58.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:58 smithi167 conmon[60316]: debug 2022-01-31T21:45:58.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:58.378550+0000) 2022-01-31T21:45:58.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:58 smithi167 conmon[54076]: debug 2022-01-31T21:45:58.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:58.379098+0000) 2022-01-31T21:45:58.950 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:58 smithi167 conmon[49112]: debug 2022-01-31T21:45:58.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:58.715381+0000) 2022-01-31T21:45:59.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[35325]: debug 2022-01-31T21:45:59.035+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155789 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:45:59.056 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[41853]: debug 2022-01-31T21:45:59.019+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.021386+0000) 2022-01-31T21:45:59.057 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[46715]: debug 2022-01-31T21:45:59.018+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.020611+0000) 2022-01-31T21:45:59.057 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:58 smithi171 conmon[51620]: debug 2022-01-31T21:45:58.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:58.987121+0000) 2022-01-31T21:45:59.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[51620]: debug 2022-01-31T21:45:59.019+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.021039+0000) 2022-01-31T21:45:59.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[46715]: debug 2022-01-31T21:45:59.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.145649+0000) 2022-01-31T21:45:59.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[41853]: debug 2022-01-31T21:45:59.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.101275+0000) 2022-01-31T21:45:59.377 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:59 smithi167 conmon[49112]: debug 2022-01-31T21:45:59.019+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.019917+0000) 2022-01-31T21:45:59.378 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:59 smithi167 conmon[54076]: debug 2022-01-31T21:45:59.019+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.019828+0000) 2022-01-31T21:45:59.378 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:59 smithi167 conmon[60316]: debug 2022-01-31T21:45:59.020+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.020612+0000) 2022-01-31T21:45:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:45:59 smithi167 conmon[54076]: debug 2022-01-31T21:45:59.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.379201+0000) 2022-01-31T21:45:59.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:45:59 smithi167 conmon[60316]: debug 2022-01-31T21:45:59.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.378716+0000) 2022-01-31T21:45:59.950 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:45:59 smithi167 conmon[49112]: debug 2022-01-31T21:45:59.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.715578+0000) 2022-01-31T21:46:00.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:45:59 smithi171 conmon[51620]: debug 2022-01-31T21:45:59.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:45:59.987320+0000) 2022-01-31T21:46:00.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:00 smithi171 conmon[46715]: debug 2022-01-31T21:46:00.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:00.145806+0000) 2022-01-31T21:46:00.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:00 smithi171 conmon[41853]: debug 2022-01-31T21:46:00.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:00.101431+0000) 2022-01-31T21:46:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:00 smithi167 conmon[54076]: debug 2022-01-31T21:46:00.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:00.379298+0000) 2022-01-31T21:46:00.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:00 smithi167 conmon[60316]: debug 2022-01-31T21:46:00.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:00.378859+0000) 2022-01-31T21:46:00.954 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:00 smithi167 conmon[49112]: debug 2022-01-31T21:46:00.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:00.715738+0000) 2022-01-31T21:46:01.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:00 smithi171 conmon[51620]: debug 2022-01-31T21:46:00.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:00.987494+0000) 2022-01-31T21:46:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:01 smithi171 conmon[46715]: debug 2022-01-31T21:46:01.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:01.145995+0000) 2022-01-31T21:46:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:01 smithi171 conmon[41853]: debug 2022-01-31T21:46:01.099+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:01.101632+0000) 2022-01-31T21:46:01.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:01 smithi167 conmon[60316]: debug 2022-01-31T21:46:01.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:01.379070+0000) 2022-01-31T21:46:01.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:01 smithi167 conmon[54076]: debug 2022-01-31T21:46:01.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:01.379463+0000) 2022-01-31T21:46:01.959 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:01 smithi167 conmon[49112]: debug 2022-01-31T21:46:01.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:01.715953+0000) 2022-01-31T21:46:02.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:01 smithi171 conmon[51620]: debug 2022-01-31T21:46:01.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:01.987657+0000) 2022-01-31T21:46:02.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:02 smithi171 conmon[46715]: debug 2022-01-31T21:46:02.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:02.146177+0000) 2022-01-31T21:46:02.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:02 smithi171 conmon[41853]: debug 2022-01-31T21:46:02.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:02.101813+0000) 2022-01-31T21:46:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:02 smithi167 conmon[54076]: debug 2022-01-31T21:46:02.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:02.379589+0000) 2022-01-31T21:46:02.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:02 smithi167 conmon[60316]: debug 2022-01-31T21:46:02.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:02.379289+0000) 2022-01-31T21:46:02.960 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:02 smithi167 conmon[49112]: debug 2022-01-31T21:46:02.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:02.716158+0000) 2022-01-31T21:46:03.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:02 smithi171 conmon[51620]: debug 2022-01-31T21:46:02.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:02.987842+0000) 2022-01-31T21:46:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:03 smithi171 conmon[41853]: debug 2022-01-31T21:46:03.101+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:03.102045+0000) 2022-01-31T21:46:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:03 smithi171 conmon[46715]: debug 2022-01-31T21:46:03.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:03.146324+0000) 2022-01-31T21:46:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:03 smithi167 conmon[54076]: debug 2022-01-31T21:46:03.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:03.379731+0000) 2022-01-31T21:46:03.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:03 smithi167 conmon[60316]: debug 2022-01-31T21:46:03.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:03.379523+0000) 2022-01-31T21:46:03.960 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:03 smithi167 conmon[49112]: debug 2022-01-31T21:46:03.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:03.716374+0000) 2022-01-31T21:46:04.057 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[46715]: debug 2022-01-31T21:46:04.037+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.039341+0000) 2022-01-31T21:46:04.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:03 smithi171 conmon[51620]: debug 2022-01-31T21:46:03.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:03.988013+0000) 2022-01-31T21:46:04.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[51620]: debug 2022-01-31T21:46:04.038+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.039944+0000) 2022-01-31T21:46:04.059 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[41853]: debug 2022-01-31T21:46:04.038+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.040142+0000) 2022-01-31T21:46:04.059 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[35325]: debug 2022-01-31T21:46:04.054+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 155900 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:04.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[41853]: debug 2022-01-31T21:46:04.101+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.102215+0000) 2022-01-31T21:46:04.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[46715]: debug 2022-01-31T21:46:04.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.146491+0000) 2022-01-31T21:46:04.378 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:04 smithi167 conmon[49112]: debug 2022-01-31T21:46:04.037+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.038570+0000) 2022-01-31T21:46:04.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:04 smithi167 conmon[54076]: debug 2022-01-31T21:46:04.036+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.038093+0000) 2022-01-31T21:46:04.379 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:04 smithi167 conmon[60316]: debug 2022-01-31T21:46:04.037+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.039035+0000) 2022-01-31T21:46:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:04 smithi167 conmon[54076]: debug 2022-01-31T21:46:04.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.379881+0000) 2022-01-31T21:46:04.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:04 smithi167 conmon[60316]: debug 2022-01-31T21:46:04.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.379715+0000) 2022-01-31T21:46:04.960 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:04 smithi167 conmon[49112]: debug 2022-01-31T21:46:04.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.716576+0000) 2022-01-31T21:46:05.057 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:04 smithi171 conmon[51620]: debug 2022-01-31T21:46:04.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:04.988218+0000) 2022-01-31T21:46:05.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:05 smithi171 conmon[46715]: debug 2022-01-31T21:46:05.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:05.146676+0000) 2022-01-31T21:46:05.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:05 smithi171 conmon[41853]: debug 2022-01-31T21:46:05.101+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:05.102331+0000) 2022-01-31T21:46:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:05 smithi167 conmon[54076]: debug 2022-01-31T21:46:05.378+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:05.379986+0000) 2022-01-31T21:46:05.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:05 smithi167 conmon[60316]: debug 2022-01-31T21:46:05.378+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:05.379861+0000) 2022-01-31T21:46:05.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:05 smithi167 conmon[49112]: debug 2022-01-31T21:46:05.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:05.716728+0000) 2022-01-31T21:46:06.057 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:05 smithi171 conmon[51620]: debug 2022-01-31T21:46:05.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:05.988395+0000) 2022-01-31T21:46:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:06 smithi171 conmon[41853]: debug 2022-01-31T21:46:06.100+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:06.102482+0000) 2022-01-31T21:46:06.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:06 smithi171 conmon[46715]: debug 2022-01-31T21:46:06.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:06.146864+0000) 2022-01-31T21:46:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:06 smithi167 conmon[54076]: debug 2022-01-31T21:46:06.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:06.380129+0000) 2022-01-31T21:46:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:06 smithi167 conmon[60316]: debug 2022-01-31T21:46:06.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:06.380025+0000) 2022-01-31T21:46:06.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:06 smithi167 conmon[49112]: debug 2022-01-31T21:46:06.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:06.716939+0000) 2022-01-31T21:46:07.057 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:06 smithi171 conmon[51620]: debug 2022-01-31T21:46:06.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:06.988599+0000) 2022-01-31T21:46:07.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:07 smithi171 conmon[41853]: debug 2022-01-31T21:46:07.101+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:07.102628+0000) 2022-01-31T21:46:07.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:07 smithi171 conmon[46715]: debug 2022-01-31T21:46:07.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:07.147071+0000) 2022-01-31T21:46:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:07 smithi167 conmon[54076]: debug 2022-01-31T21:46:07.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:07.380278+0000) 2022-01-31T21:46:07.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:07 smithi167 conmon[60316]: debug 2022-01-31T21:46:07.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:07.380164+0000) 2022-01-31T21:46:07.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:07 smithi167 conmon[49112]: debug 2022-01-31T21:46:07.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:07.717154+0000) 2022-01-31T21:46:08.057 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:07 smithi171 conmon[51620]: debug 2022-01-31T21:46:07.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:07.988817+0000) 2022-01-31T21:46:08.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:08 smithi171 conmon[41853]: debug 2022-01-31T21:46:08.101+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:08.102781+0000) 2022-01-31T21:46:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:08 smithi171 conmon[46715]: debug 2022-01-31T21:46:08.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:08.147280+0000) 2022-01-31T21:46:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:08 smithi167 conmon[54076]: debug 2022-01-31T21:46:08.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:08.380424+0000) 2022-01-31T21:46:08.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:08 smithi167 conmon[60316]: debug 2022-01-31T21:46:08.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:08.380339+0000) 2022-01-31T21:46:08.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:08 smithi167 conmon[49112]: debug 2022-01-31T21:46:08.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:08.717370+0000) 2022-01-31T21:46:08.986 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:08 smithi171 conmon[35325]: debug 2022-01-31T21:46:08.736+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:46:09.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[35325]: debug 2022-01-31T21:46:09.073+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156010 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[46715]: debug 2022-01-31T21:46:09.056+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.058473+0000) 2022-01-31T21:46:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[46715]: debug 2022-01-31T21:46:09.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.147457+0000) 2022-01-31T21:46:09.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:08 smithi171 conmon[51620]: debug 2022-01-31T21:46:08.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:08.988986+0000) 2022-01-31T21:46:09.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[51620]: debug 2022-01-31T21:46:09.056+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.058657+0000) 2022-01-31T21:46:09.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[41853]: debug 2022-01-31T21:46:09.057+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.059013+0000) 2022-01-31T21:46:09.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[41853]: debug 2022-01-31T21:46:09.102+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.102971+0000) 2022-01-31T21:46:09.379 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:09 smithi167 conmon[49112]: debug 2022-01-31T21:46:09.055+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.057170+0000) 2022-01-31T21:46:09.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:09 smithi167 conmon[54076]: debug 2022-01-31T21:46:09.055+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.056992+0000) 2022-01-31T21:46:09.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:09 smithi167 conmon[60316]: debug 2022-01-31T21:46:09.056+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.057837+0000) 2022-01-31T21:46:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:09 smithi167 conmon[54076]: debug 2022-01-31T21:46:09.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.380588+0000) 2022-01-31T21:46:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:09 smithi167 conmon[60316]: debug 2022-01-31T21:46:09.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.380542+0000) 2022-01-31T21:46:09.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:09 smithi167 conmon[49112]: debug 2022-01-31T21:46:09.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.717589+0000) 2022-01-31T21:46:10.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:09 smithi171 conmon[51620]: debug 2022-01-31T21:46:09.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:09.989150+0000) 2022-01-31T21:46:10.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:10 smithi171 conmon[46715]: debug 2022-01-31T21:46:10.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:10.147607+0000) 2022-01-31T21:46:10.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:10 smithi171 conmon[41853]: debug 2022-01-31T21:46:10.102+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:10.103152+0000) 2022-01-31T21:46:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:10 smithi167 conmon[54076]: debug 2022-01-31T21:46:10.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:10.380772+0000) 2022-01-31T21:46:10.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:10 smithi167 conmon[60316]: debug 2022-01-31T21:46:10.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:10.380703+0000) 2022-01-31T21:46:10.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:10 smithi167 conmon[49112]: debug 2022-01-31T21:46:10.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:10.717758+0000) 2022-01-31T21:46:11.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:10 smithi171 conmon[51620]: debug 2022-01-31T21:46:10.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:10.989351+0000) 2022-01-31T21:46:11.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:11 smithi171 conmon[46715]: debug 2022-01-31T21:46:11.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:11.147814+0000) 2022-01-31T21:46:11.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:11 smithi171 conmon[41853]: debug 2022-01-31T21:46:11.102+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:11.103305+0000) 2022-01-31T21:46:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:11 smithi167 conmon[54076]: debug 2022-01-31T21:46:11.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:11.380936+0000) 2022-01-31T21:46:11.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:11 smithi167 conmon[60316]: debug 2022-01-31T21:46:11.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:11.380873+0000) 2022-01-31T21:46:11.980 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:11 smithi167 conmon[49112]: debug 2022-01-31T21:46:11.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:11.717978+0000) 2022-01-31T21:46:12.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:11 smithi171 conmon[51620]: debug 2022-01-31T21:46:11.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:11.989517+0000) 2022-01-31T21:46:12.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:12 smithi171 conmon[46715]: debug 2022-01-31T21:46:12.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:12.147996+0000) 2022-01-31T21:46:12.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:12 smithi171 conmon[41853]: debug 2022-01-31T21:46:12.102+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:12.103433+0000) 2022-01-31T21:46:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:12 smithi167 conmon[54076]: debug 2022-01-31T21:46:12.379+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:12.381077+0000) 2022-01-31T21:46:12.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:12 smithi167 conmon[60316]: debug 2022-01-31T21:46:12.379+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:12.381044+0000) 2022-01-31T21:46:12.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:12 smithi167 conmon[49112]: debug 2022-01-31T21:46:12.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:12.718186+0000) 2022-01-31T21:46:13.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:12 smithi171 conmon[51620]: debug 2022-01-31T21:46:12.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:12.989716+0000) 2022-01-31T21:46:13.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:13 smithi171 conmon[41853]: debug 2022-01-31T21:46:13.102+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:13.103630+0000) 2022-01-31T21:46:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:13 smithi171 conmon[46715]: debug 2022-01-31T21:46:13.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:13.148142+0000) 2022-01-31T21:46:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:13 smithi167 conmon[54076]: debug 2022-01-31T21:46:13.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:13.381279+0000) 2022-01-31T21:46:13.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:13 smithi167 conmon[60316]: debug 2022-01-31T21:46:13.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:13.381255+0000) 2022-01-31T21:46:13.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:13 smithi167 conmon[49112]: debug 2022-01-31T21:46:13.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:13.718414+0000) 2022-01-31T21:46:14.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:13 smithi171 conmon[51620]: debug 2022-01-31T21:46:13.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:13.989939+0000) 2022-01-31T21:46:14.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[35325]: debug 2022-01-31T21:46:14.093+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[46715]: debug 2022-01-31T21:46:14.077+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.077955+0000) 2022-01-31T21:46:14.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[46715]: debug 2022-01-31T21:46:14.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.148336+0000) 2022-01-31T21:46:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[51620]: debug 2022-01-31T21:46:14.077+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.078130+0000) 2022-01-31T21:46:14.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[41853]: debug 2022-01-31T21:46:14.076+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.077716+0000) 2022-01-31T21:46:14.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[41853]: debug 2022-01-31T21:46:14.103+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.103801+0000) 2022-01-31T21:46:14.380 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:14 smithi167 conmon[60316]: debug 2022-01-31T21:46:14.075+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.077297+0000) 2022-01-31T21:46:14.380 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:14 smithi167 conmon[49112]: debug 2022-01-31T21:46:14.076+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.077523+0000) 2022-01-31T21:46:14.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:14 smithi167 conmon[54076]: debug 2022-01-31T21:46:14.075+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.077164+0000) 2022-01-31T21:46:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:14 smithi167 conmon[54076]: debug 2022-01-31T21:46:14.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.381464+0000) 2022-01-31T21:46:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:14 smithi167 conmon[60316]: debug 2022-01-31T21:46:14.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.381464+0000) 2022-01-31T21:46:14.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:14 smithi167 conmon[49112]: debug 2022-01-31T21:46:14.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.718607+0000) 2022-01-31T21:46:15.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:14 smithi171 conmon[51620]: debug 2022-01-31T21:46:14.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:14.990132+0000) 2022-01-31T21:46:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:15 smithi171 conmon[41853]: debug 2022-01-31T21:46:15.103+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:15.103946+0000) 2022-01-31T21:46:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:15 smithi171 conmon[46715]: debug 2022-01-31T21:46:15.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:15.148486+0000) 2022-01-31T21:46:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:15 smithi167 conmon[54076]: debug 2022-01-31T21:46:15.380+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:15.381677+0000) 2022-01-31T21:46:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:15 smithi167 conmon[60316]: debug 2022-01-31T21:46:15.380+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:15.381652+0000) 2022-01-31T21:46:15.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:15 smithi167 conmon[49112]: debug 2022-01-31T21:46:15.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:15.718735+0000) 2022-01-31T21:46:16.059 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:15 smithi171 conmon[51620]: debug 2022-01-31T21:46:15.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:15.990214+0000) 2022-01-31T21:46:16.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:16 smithi171 conmon[46715]: debug 2022-01-31T21:46:16.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:16.148644+0000) 2022-01-31T21:46:16.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:16 smithi171 conmon[41853]: debug 2022-01-31T21:46:16.103+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:16.104106+0000) 2022-01-31T21:46:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:16 smithi167 conmon[54076]: debug 2022-01-31T21:46:16.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:16.381838+0000) 2022-01-31T21:46:16.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:16 smithi167 conmon[60316]: debug 2022-01-31T21:46:16.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:16.381839+0000) 2022-01-31T21:46:16.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:16 smithi167 conmon[49112]: debug 2022-01-31T21:46:16.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:16.718949+0000) 2022-01-31T21:46:17.059 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:16 smithi171 conmon[51620]: debug 2022-01-31T21:46:16.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:16.990386+0000) 2022-01-31T21:46:17.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:17 smithi171 conmon[46715]: debug 2022-01-31T21:46:17.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:17.148856+0000) 2022-01-31T21:46:17.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:17 smithi171 conmon[41853]: debug 2022-01-31T21:46:17.103+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:17.104262+0000) 2022-01-31T21:46:17.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:17 smithi167 conmon[60316]: debug 2022-01-31T21:46:17.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:17.382049+0000) 2022-01-31T21:46:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:17 smithi167 conmon[54076]: debug 2022-01-31T21:46:17.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:17.382047+0000) 2022-01-31T21:46:17.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:17 smithi167 conmon[49112]: debug 2022-01-31T21:46:17.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:17.719114+0000) 2022-01-31T21:46:18.059 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:17 smithi171 conmon[51620]: debug 2022-01-31T21:46:17.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:17.990573+0000) 2022-01-31T21:46:18.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:18 smithi171 conmon[41853]: debug 2022-01-31T21:46:18.103+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:18.104419+0000) 2022-01-31T21:46:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:18 smithi171 conmon[46715]: debug 2022-01-31T21:46:18.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:18.149011+0000) 2022-01-31T21:46:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:18 smithi167 conmon[54076]: debug 2022-01-31T21:46:18.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:18.382245+0000) 2022-01-31T21:46:18.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:18 smithi167 conmon[60316]: debug 2022-01-31T21:46:18.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:18.382253+0000) 2022-01-31T21:46:18.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:18 smithi167 conmon[49112]: debug 2022-01-31T21:46:18.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:18.719264+0000) 2022-01-31T21:46:19.059 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:18 smithi171 conmon[51620]: debug 2022-01-31T21:46:18.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:18.990795+0000) 2022-01-31T21:46:19.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[35325]: debug 2022-01-31T21:46:19.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156231 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:19.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[51620]: debug 2022-01-31T21:46:19.095+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.096657+0000) 2022-01-31T21:46:19.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[41853]: debug 2022-01-31T21:46:19.095+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.096550+0000) 2022-01-31T21:46:19.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[41853]: debug 2022-01-31T21:46:19.103+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.104570+0000) 2022-01-31T21:46:19.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[46715]: debug 2022-01-31T21:46:19.096+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.097288+0000) 2022-01-31T21:46:19.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[46715]: debug 2022-01-31T21:46:19.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.149190+0000) 2022-01-31T21:46:19.381 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:19 smithi167 conmon[49112]: debug 2022-01-31T21:46:19.101+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.103262+0000) 2022-01-31T21:46:19.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:19 smithi167 conmon[54076]: debug 2022-01-31T21:46:19.094+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.096226+0000) 2022-01-31T21:46:19.382 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:19 smithi167 conmon[60316]: debug 2022-01-31T21:46:19.094+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.095792+0000) 2022-01-31T21:46:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:19 smithi167 conmon[54076]: debug 2022-01-31T21:46:19.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.382472+0000) 2022-01-31T21:46:19.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:19 smithi167 conmon[60316]: debug 2022-01-31T21:46:19.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.382473+0000) 2022-01-31T21:46:19.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:19 smithi167 conmon[49112]: debug 2022-01-31T21:46:19.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.719481+0000) 2022-01-31T21:46:20.059 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:19 smithi171 conmon[51620]: debug 2022-01-31T21:46:19.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:19.990929+0000) 2022-01-31T21:46:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:20 smithi171 conmon[41853]: debug 2022-01-31T21:46:20.104+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:20.104745+0000) 2022-01-31T21:46:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:20 smithi171 conmon[46715]: debug 2022-01-31T21:46:20.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:20.149339+0000) 2022-01-31T21:46:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:20 smithi167 conmon[54076]: debug 2022-01-31T21:46:20.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:20.382687+0000) 2022-01-31T21:46:20.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:20 smithi167 conmon[60316]: debug 2022-01-31T21:46:20.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:20.382646+0000) 2022-01-31T21:46:20.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:20 smithi167 conmon[49112]: debug 2022-01-31T21:46:20.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:20.719611+0000) 2022-01-31T21:46:21.060 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:20 smithi171 conmon[51620]: debug 2022-01-31T21:46:20.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:20.991077+0000) 2022-01-31T21:46:21.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:21 smithi171 conmon[41853]: debug 2022-01-31T21:46:21.104+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:21.104963+0000) 2022-01-31T21:46:21.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:21 smithi171 conmon[46715]: debug 2022-01-31T21:46:21.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:21.149546+0000) 2022-01-31T21:46:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:21 smithi167 conmon[54076]: debug 2022-01-31T21:46:21.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:21.382895+0000) 2022-01-31T21:46:21.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:21 smithi167 conmon[60316]: debug 2022-01-31T21:46:21.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:21.382873+0000) 2022-01-31T21:46:22.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:21 smithi167 conmon[49112]: debug 2022-01-31T21:46:21.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:21.719807+0000) 2022-01-31T21:46:22.060 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:21 smithi171 conmon[51620]: debug 2022-01-31T21:46:21.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:21.991287+0000) 2022-01-31T21:46:22.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:22 smithi171 conmon[41853]: debug 2022-01-31T21:46:22.104+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:22.105122+0000) 2022-01-31T21:46:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:22 smithi171 conmon[46715]: debug 2022-01-31T21:46:22.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:22.149786+0000) 2022-01-31T21:46:22.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:22 smithi167 conmon[60316]: debug 2022-01-31T21:46:22.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:22.383056+0000) 2022-01-31T21:46:22.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:22 smithi167 conmon[54076]: debug 2022-01-31T21:46:22.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:22.383076+0000) 2022-01-31T21:46:23.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:22 smithi167 conmon[49112]: debug 2022-01-31T21:46:22.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:22.720022+0000) 2022-01-31T21:46:23.060 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:22 smithi171 conmon[51620]: debug 2022-01-31T21:46:22.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:22.991492+0000) 2022-01-31T21:46:23.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:23 smithi171 conmon[41853]: debug 2022-01-31T21:46:23.104+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:23.105331+0000) 2022-01-31T21:46:23.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:23 smithi171 conmon[46715]: debug 2022-01-31T21:46:23.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:23.149932+0000) 2022-01-31T21:46:23.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:23 smithi167 conmon[54076]: debug 2022-01-31T21:46:23.382+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:23.383252+0000) 2022-01-31T21:46:23.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:23 smithi167 conmon[60316]: debug 2022-01-31T21:46:23.382+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:23.383235+0000) 2022-01-31T21:46:23.989 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:23 smithi171 conmon[35325]: debug 2022-01-31T21:46:23.737+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:46:24.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:23 smithi167 conmon[49112]: debug 2022-01-31T21:46:23.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:23.720226+0000) 2022-01-31T21:46:24.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[35325]: debug 2022-01-31T21:46:24.135+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156343 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[41853]: debug 2022-01-31T21:46:24.104+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.105530+0000) 2022-01-31T21:46:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[41853]: debug 2022-01-31T21:46:24.119+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.120512+0000) 2022-01-31T21:46:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[46715]: debug 2022-01-31T21:46:24.121+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.121908+0000) 2022-01-31T21:46:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[46715]: debug 2022-01-31T21:46:24.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.150061+0000) 2022-01-31T21:46:24.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:23 smithi171 conmon[51620]: debug 2022-01-31T21:46:23.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:23.991712+0000) 2022-01-31T21:46:24.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[51620]: debug 2022-01-31T21:46:24.119+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.120432+0000) 2022-01-31T21:46:24.382 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:24 smithi167 conmon[49112]: debug 2022-01-31T21:46:24.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.121107+0000) 2022-01-31T21:46:24.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:24 smithi167 conmon[54076]: debug 2022-01-31T21:46:24.119+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.120492+0000) 2022-01-31T21:46:24.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:24 smithi167 conmon[60316]: debug 2022-01-31T21:46:24.118+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.120094+0000) 2022-01-31T21:46:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:24 smithi167 conmon[54076]: debug 2022-01-31T21:46:24.381+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.383435+0000) 2022-01-31T21:46:24.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:24 smithi167 conmon[60316]: debug 2022-01-31T21:46:24.381+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.383435+0000) 2022-01-31T21:46:25.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:24 smithi167 conmon[49112]: debug 2022-01-31T21:46:24.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.720440+0000) 2022-01-31T21:46:25.060 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:24 smithi171 conmon[51620]: debug 2022-01-31T21:46:24.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:24.991959+0000) 2022-01-31T21:46:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:25 smithi171 conmon[41853]: debug 2022-01-31T21:46:25.104+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:25.105739+0000) 2022-01-31T21:46:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:25 smithi171 conmon[46715]: debug 2022-01-31T21:46:25.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:25.150237+0000) 2022-01-31T21:46:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:25 smithi167 conmon[54076]: debug 2022-01-31T21:46:25.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:25.383652+0000) 2022-01-31T21:46:25.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:25 smithi167 conmon[60316]: debug 2022-01-31T21:46:25.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:25.383585+0000) 2022-01-31T21:46:26.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:25 smithi167 conmon[49112]: debug 2022-01-31T21:46:25.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:25.720553+0000) 2022-01-31T21:46:26.060 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:25 smithi171 conmon[51620]: debug 2022-01-31T21:46:25.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:25.992120+0000) 2022-01-31T21:46:26.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:26 smithi171 conmon[41853]: debug 2022-01-31T21:46:26.105+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:26.105858+0000) 2022-01-31T21:46:26.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:26 smithi171 conmon[46715]: debug 2022-01-31T21:46:26.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:26.150417+0000) 2022-01-31T21:46:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:26 smithi167 conmon[54076]: debug 2022-01-31T21:46:26.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:26.383846+0000) 2022-01-31T21:46:26.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:26 smithi167 conmon[60316]: debug 2022-01-31T21:46:26.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:26.383748+0000) 2022-01-31T21:46:27.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:26 smithi167 conmon[49112]: debug 2022-01-31T21:46:26.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:26.720718+0000) 2022-01-31T21:46:27.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:26 smithi171 conmon[51620]: debug 2022-01-31T21:46:26.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:26.992286+0000) 2022-01-31T21:46:27.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:27 smithi171 conmon[41853]: debug 2022-01-31T21:46:27.105+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:27.106021+0000) 2022-01-31T21:46:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:27 smithi171 conmon[46715]: debug 2022-01-31T21:46:27.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:27.150596+0000) 2022-01-31T21:46:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:27 smithi167 conmon[54076]: debug 2022-01-31T21:46:27.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:27.383998+0000) 2022-01-31T21:46:27.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:27 smithi167 conmon[60316]: debug 2022-01-31T21:46:27.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:27.383903+0000) 2022-01-31T21:46:28.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:27 smithi167 conmon[49112]: debug 2022-01-31T21:46:27.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:27.720892+0000) 2022-01-31T21:46:28.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:27 smithi171 conmon[51620]: debug 2022-01-31T21:46:27.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:27.992496+0000) 2022-01-31T21:46:28.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:28 smithi171 conmon[41853]: debug 2022-01-31T21:46:28.105+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:28.106265+0000) 2022-01-31T21:46:28.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:28 smithi171 conmon[46715]: debug 2022-01-31T21:46:28.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:28.150785+0000) 2022-01-31T21:46:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:28 smithi167 conmon[54076]: debug 2022-01-31T21:46:28.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:28.384163+0000) 2022-01-31T21:46:28.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:28 smithi167 conmon[60316]: debug 2022-01-31T21:46:28.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:28.384057+0000) 2022-01-31T21:46:29.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:28 smithi167 conmon[49112]: debug 2022-01-31T21:46:28.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:28.721091+0000) 2022-01-31T21:46:29.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:28 smithi171 conmon[51620]: debug 2022-01-31T21:46:28.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:28.992713+0000) 2022-01-31T21:46:29.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[35325]: debug 2022-01-31T21:46:29.154+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156459 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:29.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[41853]: debug 2022-01-31T21:46:29.106+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.106424+0000) 2022-01-31T21:46:29.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[41853]: debug 2022-01-31T21:46:29.137+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.138318+0000) 2022-01-31T21:46:29.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[51620]: debug 2022-01-31T21:46:29.137+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.138555+0000) 2022-01-31T21:46:29.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[46715]: debug 2022-01-31T21:46:29.137+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.138665+0000) 2022-01-31T21:46:29.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[46715]: debug 2022-01-31T21:46:29.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.150959+0000) 2022-01-31T21:46:29.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:29 smithi167 conmon[54076]: debug 2022-01-31T21:46:29.137+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.138467+0000) 2022-01-31T21:46:29.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:29 smithi167 conmon[60316]: debug 2022-01-31T21:46:29.138+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.138495+0000) 2022-01-31T21:46:29.384 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:29 smithi167 conmon[49112]: debug 2022-01-31T21:46:29.138+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.138729+0000) 2022-01-31T21:46:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:29 smithi167 conmon[54076]: debug 2022-01-31T21:46:29.383+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.384346+0000) 2022-01-31T21:46:29.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:29 smithi167 conmon[60316]: debug 2022-01-31T21:46:29.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.384255+0000) 2022-01-31T21:46:30.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:29 smithi167 conmon[49112]: debug 2022-01-31T21:46:29.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.721298+0000) 2022-01-31T21:46:30.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:29 smithi171 conmon[51620]: debug 2022-01-31T21:46:29.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:29.992948+0000) 2022-01-31T21:46:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:30 smithi171 conmon[41853]: debug 2022-01-31T21:46:30.106+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:30.106611+0000) 2022-01-31T21:46:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:30 smithi171 conmon[46715]: debug 2022-01-31T21:46:30.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:30.151104+0000) 2022-01-31T21:46:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:30 smithi167 conmon[54076]: debug 2022-01-31T21:46:30.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:30.384556+0000) 2022-01-31T21:46:30.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:30 smithi167 conmon[60316]: debug 2022-01-31T21:46:30.383+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:30.384414+0000) 2022-01-31T21:46:31.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:30 smithi167 conmon[49112]: debug 2022-01-31T21:46:30.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:30.721448+0000) 2022-01-31T21:46:31.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:30 smithi171 conmon[51620]: debug 2022-01-31T21:46:30.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:30.993123+0000) 2022-01-31T21:46:31.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:31 smithi171 conmon[46715]: debug 2022-01-31T21:46:31.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:31.151265+0000) 2022-01-31T21:46:31.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:31 smithi171 conmon[41853]: debug 2022-01-31T21:46:31.106+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:31.106784+0000) 2022-01-31T21:46:31.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:31 smithi167 conmon[60316]: debug 2022-01-31T21:46:31.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:31.384626+0000) 2022-01-31T21:46:31.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:31 smithi167 conmon[54076]: debug 2022-01-31T21:46:31.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:31.384743+0000) 2022-01-31T21:46:32.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:31 smithi167 conmon[49112]: debug 2022-01-31T21:46:31.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:31.721678+0000) 2022-01-31T21:46:32.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:31 smithi171 conmon[51620]: debug 2022-01-31T21:46:31.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:31.993349+0000) 2022-01-31T21:46:32.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:32 smithi171 conmon[41853]: debug 2022-01-31T21:46:32.106+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:32.106991+0000) 2022-01-31T21:46:32.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:32 smithi171 conmon[46715]: debug 2022-01-31T21:46:32.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:32.151446+0000) 2022-01-31T21:46:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:32 smithi167 conmon[54076]: debug 2022-01-31T21:46:32.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:32.384874+0000) 2022-01-31T21:46:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:32 smithi167 conmon[60316]: debug 2022-01-31T21:46:32.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:32.384798+0000) 2022-01-31T21:46:33.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:32 smithi167 conmon[49112]: debug 2022-01-31T21:46:32.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:32.721899+0000) 2022-01-31T21:46:33.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:32 smithi171 conmon[51620]: debug 2022-01-31T21:46:32.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:32.993526+0000) 2022-01-31T21:46:33.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:33 smithi171 conmon[41853]: debug 2022-01-31T21:46:33.106+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:33.107188+0000) 2022-01-31T21:46:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:33 smithi171 conmon[46715]: debug 2022-01-31T21:46:33.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:33.151586+0000) 2022-01-31T21:46:33.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:33 smithi167 conmon[54076]: debug 2022-01-31T21:46:33.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:33.384996+0000) 2022-01-31T21:46:33.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:33 smithi167 conmon[60316]: debug 2022-01-31T21:46:33.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:33.384950+0000) 2022-01-31T21:46:34.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:33 smithi167 conmon[49112]: debug 2022-01-31T21:46:33.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:33.722081+0000) 2022-01-31T21:46:34.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:33 smithi171 conmon[51620]: debug 2022-01-31T21:46:33.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:33.993708+0000) 2022-01-31T21:46:34.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[35325]: debug 2022-01-31T21:46:34.173+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:34.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[51620]: debug 2022-01-31T21:46:34.156+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.156802+0000) 2022-01-31T21:46:34.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[41853]: debug 2022-01-31T21:46:34.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.107373+0000) 2022-01-31T21:46:34.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[41853]: debug 2022-01-31T21:46:34.157+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.157798+0000) 2022-01-31T21:46:34.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[46715]: debug 2022-01-31T21:46:34.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.151768+0000) 2022-01-31T21:46:34.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[46715]: debug 2022-01-31T21:46:34.156+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.157553+0000) 2022-01-31T21:46:34.383 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:34 smithi167 conmon[49112]: debug 2022-01-31T21:46:34.157+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.157636+0000) 2022-01-31T21:46:34.384 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:34 smithi167 conmon[54076]: debug 2022-01-31T21:46:34.156+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.157448+0000) 2022-01-31T21:46:34.384 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:34 smithi167 conmon[60316]: debug 2022-01-31T21:46:34.156+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.157261+0000) 2022-01-31T21:46:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:34 smithi167 conmon[54076]: debug 2022-01-31T21:46:34.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.385158+0000) 2022-01-31T21:46:34.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:34 smithi167 conmon[60316]: debug 2022-01-31T21:46:34.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.385122+0000) 2022-01-31T21:46:35.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:34 smithi167 conmon[49112]: debug 2022-01-31T21:46:34.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.722257+0000) 2022-01-31T21:46:35.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:34 smithi171 conmon[51620]: debug 2022-01-31T21:46:34.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:34.993858+0000) 2022-01-31T21:46:35.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:35 smithi171 conmon[46715]: debug 2022-01-31T21:46:35.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:35.151945+0000) 2022-01-31T21:46:35.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:35 smithi171 conmon[41853]: debug 2022-01-31T21:46:35.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:35.107555+0000) 2022-01-31T21:46:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:35 smithi167 conmon[54076]: debug 2022-01-31T21:46:35.384+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:35.385293+0000) 2022-01-31T21:46:35.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:35 smithi167 conmon[60316]: debug 2022-01-31T21:46:35.384+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:35.385293+0000) 2022-01-31T21:46:36.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:35 smithi167 conmon[49112]: debug 2022-01-31T21:46:35.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:35.722388+0000) 2022-01-31T21:46:36.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:35 smithi171 conmon[51620]: debug 2022-01-31T21:46:35.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:35.994010+0000) 2022-01-31T21:46:36.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:36 smithi171 conmon[41853]: debug 2022-01-31T21:46:36.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:36.107708+0000) 2022-01-31T21:46:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:36 smithi171 conmon[46715]: debug 2022-01-31T21:46:36.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:36.152086+0000) 2022-01-31T21:46:36.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:36 smithi167 conmon[60316]: debug 2022-01-31T21:46:36.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:36.385535+0000) 2022-01-31T21:46:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:36 smithi167 conmon[54076]: debug 2022-01-31T21:46:36.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:36.385535+0000) 2022-01-31T21:46:37.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:36 smithi167 conmon[49112]: debug 2022-01-31T21:46:36.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:36.722538+0000) 2022-01-31T21:46:37.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:36 smithi171 conmon[51620]: debug 2022-01-31T21:46:36.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:36.994180+0000) 2022-01-31T21:46:37.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:37 smithi171 conmon[41853]: debug 2022-01-31T21:46:37.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:37.107866+0000) 2022-01-31T21:46:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:37 smithi171 conmon[46715]: debug 2022-01-31T21:46:37.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:37.152294+0000) 2022-01-31T21:46:37.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:37 smithi167 conmon[54076]: debug 2022-01-31T21:46:37.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:37.385802+0000) 2022-01-31T21:46:37.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:37 smithi167 conmon[60316]: debug 2022-01-31T21:46:37.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:37.385752+0000) 2022-01-31T21:46:38.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:37 smithi167 conmon[49112]: debug 2022-01-31T21:46:37.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:37.722750+0000) 2022-01-31T21:46:38.063 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:37 smithi171 conmon[51620]: debug 2022-01-31T21:46:37.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:37.994373+0000) 2022-01-31T21:46:38.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:38 smithi171 conmon[41853]: debug 2022-01-31T21:46:38.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:38.108055+0000) 2022-01-31T21:46:38.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:38 smithi171 conmon[46715]: debug 2022-01-31T21:46:38.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:38.152514+0000) 2022-01-31T21:46:38.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:38 smithi167 conmon[60316]: debug 2022-01-31T21:46:38.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:38.385972+0000) 2022-01-31T21:46:38.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:38 smithi167 conmon[54076]: debug 2022-01-31T21:46:38.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:38.385972+0000) 2022-01-31T21:46:38.992 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:38 smithi171 conmon[35325]: debug 2022-01-31T21:46:38.738+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:46:39.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:38 smithi167 conmon[49112]: debug 2022-01-31T21:46:38.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:38.722992+0000) 2022-01-31T21:46:39.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[35325]: debug 2022-01-31T21:46:39.192+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156683 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[41853]: debug 2022-01-31T21:46:39.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.108230+0000) 2022-01-31T21:46:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[41853]: debug 2022-01-31T21:46:39.176+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.176907+0000) 2022-01-31T21:46:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[46715]: debug 2022-01-31T21:46:39.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.152718+0000) 2022-01-31T21:46:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[46715]: debug 2022-01-31T21:46:39.175+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.176441+0000) 2022-01-31T21:46:39.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:38 smithi171 conmon[51620]: debug 2022-01-31T21:46:38.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:38.994553+0000) 2022-01-31T21:46:39.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[51620]: debug 2022-01-31T21:46:39.176+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.177181+0000) 2022-01-31T21:46:39.385 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:39 smithi167 conmon[49112]: debug 2022-01-31T21:46:39.175+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.176449+0000) 2022-01-31T21:46:39.385 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:39 smithi167 conmon[54076]: debug 2022-01-31T21:46:39.175+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.176308+0000) 2022-01-31T21:46:39.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:39 smithi167 conmon[60316]: debug 2022-01-31T21:46:39.176+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.176977+0000) 2022-01-31T21:46:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:39 smithi167 conmon[54076]: debug 2022-01-31T21:46:39.385+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.386187+0000) 2022-01-31T21:46:39.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:39 smithi167 conmon[60316]: debug 2022-01-31T21:46:39.385+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.386186+0000) 2022-01-31T21:46:40.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:39 smithi167 conmon[49112]: debug 2022-01-31T21:46:39.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.723178+0000) 2022-01-31T21:46:40.063 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:39 smithi171 conmon[51620]: debug 2022-01-31T21:46:39.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:39.994730+0000) 2022-01-31T21:46:40.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:40 smithi171 conmon[41853]: debug 2022-01-31T21:46:40.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:40.108361+0000) 2022-01-31T21:46:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:40 smithi171 conmon[46715]: debug 2022-01-31T21:46:40.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:40.152911+0000) 2022-01-31T21:46:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:40 smithi167 conmon[54076]: debug 2022-01-31T21:46:40.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:40.386354+0000) 2022-01-31T21:46:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:40 smithi167 conmon[60316]: debug 2022-01-31T21:46:40.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:40.386354+0000) 2022-01-31T21:46:41.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:40 smithi167 conmon[49112]: debug 2022-01-31T21:46:40.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:40.723323+0000) 2022-01-31T21:46:41.063 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:40 smithi171 conmon[51620]: debug 2022-01-31T21:46:40.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:40.994858+0000) 2022-01-31T21:46:41.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:41 smithi171 conmon[41853]: debug 2022-01-31T21:46:41.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:41.108495+0000) 2022-01-31T21:46:41.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:41 smithi171 conmon[46715]: debug 2022-01-31T21:46:41.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:41.153044+0000) 2022-01-31T21:46:41.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:41 smithi167 conmon[60316]: debug 2022-01-31T21:46:41.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:41.386563+0000) 2022-01-31T21:46:41.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:41 smithi167 conmon[54076]: debug 2022-01-31T21:46:41.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:41.386561+0000) 2022-01-31T21:46:42.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:41 smithi167 conmon[49112]: debug 2022-01-31T21:46:41.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:41.723536+0000) 2022-01-31T21:46:42.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:41 smithi171 conmon[51620]: debug 2022-01-31T21:46:41.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:41.995047+0000) 2022-01-31T21:46:42.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:42 smithi171 conmon[41853]: debug 2022-01-31T21:46:42.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:42.108676+0000) 2022-01-31T21:46:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:42 smithi171 conmon[46715]: debug 2022-01-31T21:46:42.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:42.153228+0000) 2022-01-31T21:46:42.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:42 smithi167 conmon[54076]: debug 2022-01-31T21:46:42.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:42.386768+0000) 2022-01-31T21:46:42.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:42 smithi167 conmon[60316]: debug 2022-01-31T21:46:42.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:42.386768+0000) 2022-01-31T21:46:43.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:42 smithi167 conmon[49112]: debug 2022-01-31T21:46:42.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:42.723752+0000) 2022-01-31T21:46:43.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:42 smithi171 conmon[51620]: debug 2022-01-31T21:46:42.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:42.995240+0000) 2022-01-31T21:46:43.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:43 smithi171 conmon[41853]: debug 2022-01-31T21:46:43.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:43.108868+0000) 2022-01-31T21:46:43.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:43 smithi171 conmon[46715]: debug 2022-01-31T21:46:43.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:43.153449+0000) 2022-01-31T21:46:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:43 smithi167 conmon[54076]: debug 2022-01-31T21:46:43.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:43.386945+0000) 2022-01-31T21:46:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:43 smithi167 conmon[60316]: debug 2022-01-31T21:46:43.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:43.386946+0000) 2022-01-31T21:46:44.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:43 smithi167 conmon[49112]: debug 2022-01-31T21:46:43.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:43.723936+0000) 2022-01-31T21:46:44.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:43 smithi171 conmon[51620]: debug 2022-01-31T21:46:43.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:43.995446+0000) 2022-01-31T21:46:44.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[35325]: debug 2022-01-31T21:46:44.211+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156795 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:44.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[51620]: debug 2022-01-31T21:46:44.194+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.195374+0000) 2022-01-31T21:46:44.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[41853]: debug 2022-01-31T21:46:44.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.109022+0000) 2022-01-31T21:46:44.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[41853]: debug 2022-01-31T21:46:44.194+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.195592+0000) 2022-01-31T21:46:44.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[46715]: debug 2022-01-31T21:46:44.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.153622+0000) 2022-01-31T21:46:44.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[46715]: debug 2022-01-31T21:46:44.195+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.196152+0000) 2022-01-31T21:46:44.385 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:44 smithi167 conmon[49112]: debug 2022-01-31T21:46:44.194+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.195080+0000) 2022-01-31T21:46:44.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:44 smithi167 conmon[54076]: debug 2022-01-31T21:46:44.194+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.194806+0000) 2022-01-31T21:46:44.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:44 smithi167 conmon[60316]: debug 2022-01-31T21:46:44.195+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.195964+0000) 2022-01-31T21:46:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:44 smithi167 conmon[54076]: debug 2022-01-31T21:46:44.386+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.387115+0000) 2022-01-31T21:46:44.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:44 smithi167 conmon[60316]: debug 2022-01-31T21:46:44.386+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.387115+0000) 2022-01-31T21:46:45.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:44 smithi167 conmon[49112]: debug 2022-01-31T21:46:44.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.724129+0000) 2022-01-31T21:46:45.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:44 smithi171 conmon[51620]: debug 2022-01-31T21:46:44.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:44.995659+0000) 2022-01-31T21:46:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:45 smithi171 conmon[41853]: debug 2022-01-31T21:46:45.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:45.109208+0000) 2022-01-31T21:46:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:45 smithi171 conmon[46715]: debug 2022-01-31T21:46:45.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:45.153804+0000) 2022-01-31T21:46:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:45 smithi167 conmon[54076]: debug 2022-01-31T21:46:45.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:45.387286+0000) 2022-01-31T21:46:45.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:45 smithi167 conmon[60316]: debug 2022-01-31T21:46:45.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:45.387286+0000) 2022-01-31T21:46:46.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:45 smithi167 conmon[49112]: debug 2022-01-31T21:46:45.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:45.724283+0000) 2022-01-31T21:46:46.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:45 smithi171 conmon[51620]: debug 2022-01-31T21:46:45.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:45.995808+0000) 2022-01-31T21:46:46.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:46 smithi171 conmon[41853]: debug 2022-01-31T21:46:46.107+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:46.109368+0000) 2022-01-31T21:46:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:46 smithi171 conmon[46715]: debug 2022-01-31T21:46:46.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:46.153973+0000) 2022-01-31T21:46:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:46 smithi167 conmon[54076]: debug 2022-01-31T21:46:46.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:46.387527+0000) 2022-01-31T21:46:46.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:46 smithi167 conmon[60316]: debug 2022-01-31T21:46:46.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:46.387527+0000) 2022-01-31T21:46:47.051 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:46 smithi167 conmon[49112]: debug 2022-01-31T21:46:46.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:46.724492+0000) 2022-01-31T21:46:47.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:46 smithi171 conmon[51620]: debug 2022-01-31T21:46:46.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:46.995969+0000) 2022-01-31T21:46:47.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:47 smithi171 conmon[41853]: debug 2022-01-31T21:46:47.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:47.109558+0000) 2022-01-31T21:46:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:47 smithi171 conmon[46715]: debug 2022-01-31T21:46:47.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:47.154149+0000) 2022-01-31T21:46:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:47 smithi167 conmon[54076]: debug 2022-01-31T21:46:47.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:47.387734+0000) 2022-01-31T21:46:47.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:47 smithi167 conmon[60316]: debug 2022-01-31T21:46:47.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:47.387734+0000) 2022-01-31T21:46:48.054 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:47 smithi167 conmon[49112]: debug 2022-01-31T21:46:47.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:47.724693+0000) 2022-01-31T21:46:48.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:47 smithi171 conmon[51620]: debug 2022-01-31T21:46:47.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:47.996157+0000) 2022-01-31T21:46:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:48 smithi171 conmon[41853]: debug 2022-01-31T21:46:48.108+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:48.109786+0000) 2022-01-31T21:46:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:48 smithi171 conmon[46715]: debug 2022-01-31T21:46:48.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:48.154341+0000) 2022-01-31T21:46:48.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:48 smithi167 conmon[54076]: debug 2022-01-31T21:46:48.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:48.387926+0000) 2022-01-31T21:46:48.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:48 smithi167 conmon[60316]: debug 2022-01-31T21:46:48.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:48.387926+0000) 2022-01-31T21:46:49.054 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:48 smithi167 conmon[49112]: debug 2022-01-31T21:46:48.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:48.724906+0000) 2022-01-31T21:46:49.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:48 smithi171 conmon[51620]: debug 2022-01-31T21:46:48.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:48.996364+0000) 2022-01-31T21:46:49.307 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:49 smithi167 conmon[49112]: debug 2022-01-31T21:46:49.214+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.214612+0000) 2022-01-31T21:46:49.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:49 smithi167 conmon[54076]: debug 2022-01-31T21:46:49.213+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.214294+0000) 2022-01-31T21:46:49.308 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:49 smithi167 conmon[60316]: debug 2022-01-31T21:46:49.214+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.214536+0000) 2022-01-31T21:46:49.343 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[35325]: debug 2022-01-31T21:46:49.230+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 156904 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:49.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[51620]: debug 2022-01-31T21:46:49.213+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.214764+0000) 2022-01-31T21:46:49.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[46715]: debug 2022-01-31T21:46:49.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.154524+0000) 2022-01-31T21:46:49.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[46715]: debug 2022-01-31T21:46:49.213+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.215157+0000) 2022-01-31T21:46:49.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[41853]: debug 2022-01-31T21:46:49.109+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.109966+0000) 2022-01-31T21:46:49.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[41853]: debug 2022-01-31T21:46:49.213+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.215268+0000) 2022-01-31T21:46:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:49 smithi167 conmon[60316]: debug 2022-01-31T21:46:49.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.388147+0000) 2022-01-31T21:46:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:49 smithi167 conmon[54076]: debug 2022-01-31T21:46:49.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.388146+0000) 2022-01-31T21:46:50.054 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:49 smithi167 conmon[49112]: debug 2022-01-31T21:46:49.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.725122+0000) 2022-01-31T21:46:50.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:49 smithi171 conmon[51620]: debug 2022-01-31T21:46:49.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:49.996567+0000) 2022-01-31T21:46:50.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:50 smithi171 conmon[41853]: debug 2022-01-31T21:46:50.109+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:50.110147+0000) 2022-01-31T21:46:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:50 smithi171 conmon[46715]: debug 2022-01-31T21:46:50.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:50.154681+0000) 2022-01-31T21:46:50.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:50 smithi167 conmon[54076]: debug 2022-01-31T21:46:50.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:50.388309+0000) 2022-01-31T21:46:50.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:50 smithi167 conmon[60316]: debug 2022-01-31T21:46:50.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:50.388291+0000) 2022-01-31T21:46:51.058 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:50 smithi167 conmon[49112]: debug 2022-01-31T21:46:50.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:50.725284+0000) 2022-01-31T21:46:51.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:50 smithi171 conmon[51620]: debug 2022-01-31T21:46:50.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:50.996716+0000) 2022-01-31T21:46:51.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:51 smithi171 conmon[46715]: debug 2022-01-31T21:46:51.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:51.154851+0000) 2022-01-31T21:46:51.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:51 smithi171 conmon[41853]: debug 2022-01-31T21:46:51.109+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:51.110297+0000) 2022-01-31T21:46:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:51 smithi167 conmon[54076]: debug 2022-01-31T21:46:51.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:51.388541+0000) 2022-01-31T21:46:51.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:51 smithi167 conmon[60316]: debug 2022-01-31T21:46:51.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:51.388542+0000) 2022-01-31T21:46:52.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:51 smithi167 conmon[49112]: debug 2022-01-31T21:46:51.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:51.725491+0000) 2022-01-31T21:46:52.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:51 smithi171 conmon[51620]: debug 2022-01-31T21:46:51.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:51.996890+0000) 2022-01-31T21:46:52.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:52 smithi171 conmon[41853]: debug 2022-01-31T21:46:52.109+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:52.110456+0000) 2022-01-31T21:46:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:52 smithi171 conmon[46715]: debug 2022-01-31T21:46:52.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:52.155042+0000) 2022-01-31T21:46:52.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:52 smithi167 conmon[54076]: debug 2022-01-31T21:46:52.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:52.388752+0000) 2022-01-31T21:46:52.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:52 smithi167 conmon[60316]: debug 2022-01-31T21:46:52.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:52.388752+0000) 2022-01-31T21:46:53.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:52 smithi167 conmon[49112]: debug 2022-01-31T21:46:52.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:52.725637+0000) 2022-01-31T21:46:53.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:52 smithi171 conmon[51620]: debug 2022-01-31T21:46:52.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:52.997049+0000) 2022-01-31T21:46:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:53 smithi171 conmon[41853]: debug 2022-01-31T21:46:53.109+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:53.110624+0000) 2022-01-31T21:46:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:53 smithi171 conmon[46715]: debug 2022-01-31T21:46:53.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:53.155180+0000) 2022-01-31T21:46:53.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:53 smithi167 conmon[54076]: debug 2022-01-31T21:46:53.387+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:53.388960+0000) 2022-01-31T21:46:53.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:53 smithi167 conmon[60316]: debug 2022-01-31T21:46:53.387+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:53.388960+0000) 2022-01-31T21:46:53.995 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:53 smithi171 conmon[35325]: debug 2022-01-31T21:46:53.738+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:46:54.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:53 smithi167 conmon[49112]: debug 2022-01-31T21:46:53.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:53.725760+0000) 2022-01-31T21:46:54.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[46715]: debug 2022-01-31T21:46:54.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.155364+0000) 2022-01-31T21:46:54.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[46715]: debug 2022-01-31T21:46:54.232+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.233844+0000) 2022-01-31T21:46:54.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:53 smithi171 conmon[51620]: debug 2022-01-31T21:46:53.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:53.997206+0000) 2022-01-31T21:46:54.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[51620]: debug 2022-01-31T21:46:54.231+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.233694+0000) 2022-01-31T21:46:54.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[41853]: debug 2022-01-31T21:46:54.109+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.110806+0000) 2022-01-31T21:46:54.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[41853]: debug 2022-01-31T21:46:54.232+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.233971+0000) 2022-01-31T21:46:54.387 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:54 smithi167 conmon[49112]: debug 2022-01-31T21:46:54.233+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.234837+0000) 2022-01-31T21:46:54.388 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:54 smithi167 conmon[54076]: debug 2022-01-31T21:46:54.232+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.233550+0000) 2022-01-31T21:46:54.389 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:54 smithi167 conmon[60316]: debug 2022-01-31T21:46:54.231+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.233298+0000) 2022-01-31T21:46:54.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[35325]: debug 2022-01-31T21:46:54.249+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157016 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:54.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:54 smithi167 conmon[60316]: debug 2022-01-31T21:46:54.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.389172+0000) 2022-01-31T21:46:54.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:54 smithi167 conmon[54076]: debug 2022-01-31T21:46:54.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.389172+0000) 2022-01-31T21:46:55.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:54 smithi167 conmon[49112]: debug 2022-01-31T21:46:54.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.725951+0000) 2022-01-31T21:46:55.066 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:54 smithi171 conmon[51620]: debug 2022-01-31T21:46:54.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:54.997391+0000) 2022-01-31T21:46:55.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:55 smithi171 conmon[46715]: debug 2022-01-31T21:46:55.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:55.155523+0000) 2022-01-31T21:46:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:55 smithi171 conmon[41853]: debug 2022-01-31T21:46:55.110+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:55.110945+0000) 2022-01-31T21:46:55.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:55 smithi167 conmon[60316]: debug 2022-01-31T21:46:55.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:55.389328+0000) 2022-01-31T21:46:55.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:55 smithi167 conmon[54076]: debug 2022-01-31T21:46:55.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:55.389329+0000) 2022-01-31T21:46:56.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:55 smithi167 conmon[49112]: debug 2022-01-31T21:46:55.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:55.726125+0000) 2022-01-31T21:46:56.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:55 smithi171 conmon[51620]: debug 2022-01-31T21:46:55.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:55.997533+0000) 2022-01-31T21:46:56.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:56 smithi171 conmon[41853]: debug 2022-01-31T21:46:56.110+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:56.111030+0000) 2022-01-31T21:46:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:56 smithi171 conmon[46715]: debug 2022-01-31T21:46:56.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:56.155662+0000) 2022-01-31T21:46:56.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:56 smithi167 conmon[60316]: debug 2022-01-31T21:46:56.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:56.389548+0000) 2022-01-31T21:46:56.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:56 smithi167 conmon[54076]: debug 2022-01-31T21:46:56.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:56.389549+0000) 2022-01-31T21:46:57.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:56 smithi167 conmon[49112]: debug 2022-01-31T21:46:56.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:56.726341+0000) 2022-01-31T21:46:57.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:56 smithi171 conmon[51620]: debug 2022-01-31T21:46:56.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:56.997729+0000) 2022-01-31T21:46:57.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:57 smithi171 conmon[41853]: debug 2022-01-31T21:46:57.110+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:57.111189+0000) 2022-01-31T21:46:57.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:57 smithi171 conmon[46715]: debug 2022-01-31T21:46:57.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:57.155851+0000) 2022-01-31T21:46:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:57 smithi167 conmon[54076]: debug 2022-01-31T21:46:57.388+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:57.389759+0000) 2022-01-31T21:46:57.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:57 smithi167 conmon[60316]: debug 2022-01-31T21:46:57.388+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:57.389758+0000) 2022-01-31T21:46:58.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:57 smithi167 conmon[49112]: debug 2022-01-31T21:46:57.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:57.726562+0000) 2022-01-31T21:46:58.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:57 smithi171 conmon[51620]: debug 2022-01-31T21:46:57.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:57.997871+0000) 2022-01-31T21:46:58.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:58 smithi171 conmon[46715]: debug 2022-01-31T21:46:58.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:58.156036+0000) 2022-01-31T21:46:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:58 smithi171 conmon[41853]: debug 2022-01-31T21:46:58.110+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:58.111382+0000) 2022-01-31T21:46:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:58 smithi167 conmon[54076]: debug 2022-01-31T21:46:58.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:58.389964+0000) 2022-01-31T21:46:58.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:58 smithi167 conmon[60316]: debug 2022-01-31T21:46:58.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:58.389964+0000) 2022-01-31T21:46:59.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:58 smithi167 conmon[49112]: debug 2022-01-31T21:46:58.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:58.726749+0000) 2022-01-31T21:46:59.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:58 smithi171 conmon[51620]: debug 2022-01-31T21:46:58.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:58.998031+0000) 2022-01-31T21:46:59.389 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:59 smithi167 conmon[49112]: debug 2022-01-31T21:46:59.251+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.253085+0000) 2022-01-31T21:46:59.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:59 smithi167 conmon[54076]: debug 2022-01-31T21:46:59.252+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.254134+0000) 2022-01-31T21:46:59.390 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:59 smithi167 conmon[60316]: debug 2022-01-31T21:46:59.251+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.253446+0000) 2022-01-31T21:46:59.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[35325]: debug 2022-01-31T21:46:59.269+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157125 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:46:59.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[46715]: debug 2022-01-31T21:46:59.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.156191+0000) 2022-01-31T21:46:59.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[46715]: debug 2022-01-31T21:46:59.252+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.253219+0000) 2022-01-31T21:46:59.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[51620]: debug 2022-01-31T21:46:59.253+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.253841+0000) 2022-01-31T21:46:59.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[41853]: debug 2022-01-31T21:46:59.110+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.111564+0000) 2022-01-31T21:46:59.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[41853]: debug 2022-01-31T21:46:59.252+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.253352+0000) 2022-01-31T21:46:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:46:59 smithi167 conmon[54076]: debug 2022-01-31T21:46:59.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.390175+0000) 2022-01-31T21:46:59.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:46:59 smithi167 conmon[60316]: debug 2022-01-31T21:46:59.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.390175+0000) 2022-01-31T21:47:00.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:46:59 smithi167 conmon[49112]: debug 2022-01-31T21:46:59.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.726967+0000) 2022-01-31T21:47:00.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:46:59 smithi171 conmon[51620]: debug 2022-01-31T21:46:59.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:46:59.998148+0000) 2022-01-31T21:47:00.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:00 smithi171 conmon[46715]: debug 2022-01-31T21:47:00.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:00.156372+0000) 2022-01-31T21:47:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:00 smithi171 conmon[41853]: debug 2022-01-31T21:47:00.111+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:00.111796+0000) 2022-01-31T21:47:00.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:00 smithi167 conmon[60316]: debug 2022-01-31T21:47:00.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:00.390349+0000) 2022-01-31T21:47:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:00 smithi167 conmon[54076]: debug 2022-01-31T21:47:00.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:00.390348+0000) 2022-01-31T21:47:01.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:00 smithi167 conmon[49112]: debug 2022-01-31T21:47:00.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:00.727127+0000) 2022-01-31T21:47:01.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:00 smithi171 conmon[51620]: debug 2022-01-31T21:47:00.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:00.998309+0000) 2022-01-31T21:47:01.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:01 smithi171 conmon[41853]: debug 2022-01-31T21:47:01.111+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:01.111901+0000) 2022-01-31T21:47:01.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:01 smithi171 conmon[46715]: debug 2022-01-31T21:47:01.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:01.156526+0000) 2022-01-31T21:47:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:01 smithi167 conmon[54076]: debug 2022-01-31T21:47:01.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:01.390560+0000) 2022-01-31T21:47:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:01 smithi167 conmon[60316]: debug 2022-01-31T21:47:01.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:01.390559+0000) 2022-01-31T21:47:02.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:01 smithi167 conmon[49112]: debug 2022-01-31T21:47:01.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:01.727354+0000) 2022-01-31T21:47:02.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:01 smithi171 conmon[51620]: debug 2022-01-31T21:47:01.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:01.998488+0000) 2022-01-31T21:47:02.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:02 smithi171 conmon[41853]: debug 2022-01-31T21:47:02.111+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:02.112064+0000) 2022-01-31T21:47:02.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:02 smithi171 conmon[46715]: debug 2022-01-31T21:47:02.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:02.156710+0000) 2022-01-31T21:47:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:02 smithi167 conmon[54076]: debug 2022-01-31T21:47:02.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:02.390767+0000) 2022-01-31T21:47:02.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:02 smithi167 conmon[60316]: debug 2022-01-31T21:47:02.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:02.390767+0000) 2022-01-31T21:47:03.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:02 smithi167 conmon[49112]: debug 2022-01-31T21:47:02.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:02.727519+0000) 2022-01-31T21:47:03.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:02 smithi171 conmon[51620]: debug 2022-01-31T21:47:02.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:02.998666+0000) 2022-01-31T21:47:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:03 smithi171 conmon[41853]: debug 2022-01-31T21:47:03.111+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:03.112256+0000) 2022-01-31T21:47:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:03 smithi171 conmon[46715]: debug 2022-01-31T21:47:03.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:03.156876+0000) 2022-01-31T21:47:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:03 smithi167 conmon[54076]: debug 2022-01-31T21:47:03.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:03.390979+0000) 2022-01-31T21:47:03.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:03 smithi167 conmon[60316]: debug 2022-01-31T21:47:03.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:03.390980+0000) 2022-01-31T21:47:04.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:03 smithi167 conmon[49112]: debug 2022-01-31T21:47:03.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:03.727680+0000) 2022-01-31T21:47:04.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:03 smithi171 conmon[51620]: debug 2022-01-31T21:47:03.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:03.998872+0000) 2022-01-31T21:47:04.390 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:04 smithi167 conmon[49112]: debug 2022-01-31T21:47:04.270+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.271534+0000) 2022-01-31T21:47:04.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:04 smithi167 conmon[54076]: debug 2022-01-31T21:47:04.271+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.273300+0000) 2022-01-31T21:47:04.391 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:04 smithi167 conmon[60316]: debug 2022-01-31T21:47:04.270+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.271725+0000) 2022-01-31T21:47:04.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[35325]: debug 2022-01-31T21:47:04.288+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157239 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:04.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[46715]: debug 2022-01-31T21:47:04.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.157047+0000) 2022-01-31T21:47:04.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[46715]: debug 2022-01-31T21:47:04.271+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.272291+0000) 2022-01-31T21:47:04.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[51620]: debug 2022-01-31T21:47:04.272+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.272998+0000) 2022-01-31T21:47:04.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[41853]: debug 2022-01-31T21:47:04.111+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.112466+0000) 2022-01-31T21:47:04.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[41853]: debug 2022-01-31T21:47:04.271+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.272354+0000) 2022-01-31T21:47:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:04 smithi167 conmon[54076]: debug 2022-01-31T21:47:04.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.391200+0000) 2022-01-31T21:47:04.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:04 smithi167 conmon[60316]: debug 2022-01-31T21:47:04.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.391200+0000) 2022-01-31T21:47:05.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:04 smithi167 conmon[49112]: debug 2022-01-31T21:47:04.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.727814+0000) 2022-01-31T21:47:05.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:04 smithi171 conmon[51620]: debug 2022-01-31T21:47:04.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:04.999038+0000) 2022-01-31T21:47:05.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:05 smithi171 conmon[41853]: debug 2022-01-31T21:47:05.111+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:05.112621+0000) 2022-01-31T21:47:05.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:05 smithi171 conmon[46715]: debug 2022-01-31T21:47:05.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:05.157227+0000) 2022-01-31T21:47:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:05 smithi167 conmon[54076]: debug 2022-01-31T21:47:05.389+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:05.391394+0000) 2022-01-31T21:47:05.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:05 smithi167 conmon[60316]: debug 2022-01-31T21:47:05.389+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:05.391394+0000) 2022-01-31T21:47:06.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:05 smithi167 conmon[49112]: debug 2022-01-31T21:47:05.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:05.727981+0000) 2022-01-31T21:47:06.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:05 smithi171 conmon[51620]: debug 2022-01-31T21:47:05.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:05.999179+0000) 2022-01-31T21:47:06.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:06 smithi171 conmon[41853]: debug 2022-01-31T21:47:06.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:06.112804+0000) 2022-01-31T21:47:06.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:06 smithi171 conmon[46715]: debug 2022-01-31T21:47:06.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:06.157392+0000) 2022-01-31T21:47:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:06 smithi167 conmon[54076]: debug 2022-01-31T21:47:06.390+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:06.391609+0000) 2022-01-31T21:47:06.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:06 smithi167 conmon[60316]: debug 2022-01-31T21:47:06.390+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:06.391609+0000) 2022-01-31T21:47:07.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:06 smithi167 conmon[49112]: debug 2022-01-31T21:47:06.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:06.728209+0000) 2022-01-31T21:47:07.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:06 smithi171 conmon[51620]: debug 2022-01-31T21:47:06.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:06.999361+0000) 2022-01-31T21:47:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:07 smithi171 conmon[41853]: debug 2022-01-31T21:47:07.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:07.112919+0000) 2022-01-31T21:47:07.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:07 smithi171 conmon[46715]: debug 2022-01-31T21:47:07.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:07.157554+0000) 2022-01-31T21:47:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:07 smithi167 conmon[54076]: debug 2022-01-31T21:47:07.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:07.391832+0000) 2022-01-31T21:47:07.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:07 smithi167 conmon[60316]: debug 2022-01-31T21:47:07.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:07.391832+0000) 2022-01-31T21:47:08.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:07 smithi167 conmon[49112]: debug 2022-01-31T21:47:07.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:07.728421+0000) 2022-01-31T21:47:08.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:07 smithi171 conmon[51620]: debug 2022-01-31T21:47:07.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:07.999537+0000) 2022-01-31T21:47:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:08 smithi171 conmon[41853]: debug 2022-01-31T21:47:08.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:08.113023+0000) 2022-01-31T21:47:08.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:08 smithi171 conmon[46715]: debug 2022-01-31T21:47:08.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:08.157727+0000) 2022-01-31T21:47:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:08 smithi167 conmon[54076]: debug 2022-01-31T21:47:08.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:08.392025+0000) 2022-01-31T21:47:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:08 smithi167 conmon[60316]: debug 2022-01-31T21:47:08.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:08.392026+0000) 2022-01-31T21:47:08.997 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:08 smithi171 conmon[35325]: debug 2022-01-31T21:47:08.739+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:47:09.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:08 smithi167 conmon[49112]: debug 2022-01-31T21:47:08.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:08.728662+0000) 2022-01-31T21:47:09.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:09 smithi171 conmon[46715]: debug 2022-01-31T21:47:09.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.157876+0000) 2022-01-31T21:47:09.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:08 smithi171 conmon[51620]: debug 2022-01-31T21:47:08.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:08.999779+0000) 2022-01-31T21:47:09.290 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:09 smithi171 conmon[41853]: debug 2022-01-31T21:47:09.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.113170+0000) 2022-01-31T21:47:09.391 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:09 smithi167 conmon[49112]: debug 2022-01-31T21:47:09.290+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.291791+0000) 2022-01-31T21:47:09.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:09 smithi167 conmon[54076]: debug 2022-01-31T21:47:09.290+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.291955+0000) 2022-01-31T21:47:09.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:09 smithi167 conmon[60316]: debug 2022-01-31T21:47:09.289+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.290561+0000) 2022-01-31T21:47:09.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:09 smithi171 conmon[35325]: debug 2022-01-31T21:47:09.308+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157347 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:09 smithi171 conmon[41853]: debug 2022-01-31T21:47:09.290+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.291698+0000) 2022-01-31T21:47:09.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:09 smithi171 conmon[46715]: debug 2022-01-31T21:47:09.290+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.290992+0000) 2022-01-31T21:47:09.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:09 smithi171 conmon[51620]: debug 2022-01-31T21:47:09.291+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.292158+0000) 2022-01-31T21:47:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:09 smithi167 conmon[54076]: debug 2022-01-31T21:47:09.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.392263+0000) 2022-01-31T21:47:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:09 smithi167 conmon[60316]: debug 2022-01-31T21:47:09.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.392265+0000) 2022-01-31T21:47:10.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:09 smithi167 conmon[49112]: debug 2022-01-31T21:47:09.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.728876+0000) 2022-01-31T21:47:10.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:10 smithi171 conmon[51620]: debug 2022-01-31T21:47:09.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:09.999997+0000) 2022-01-31T21:47:10.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:10 smithi171 conmon[41853]: debug 2022-01-31T21:47:10.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:10.113314+0000) 2022-01-31T21:47:10.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:10 smithi171 conmon[46715]: debug 2022-01-31T21:47:10.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:10.158034+0000) 2022-01-31T21:47:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:10 smithi167 conmon[54076]: debug 2022-01-31T21:47:10.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:10.392391+0000) 2022-01-31T21:47:10.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:10 smithi167 conmon[60316]: debug 2022-01-31T21:47:10.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:10.392375+0000) 2022-01-31T21:47:11.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:10 smithi167 conmon[49112]: debug 2022-01-31T21:47:10.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:10.729008+0000) 2022-01-31T21:47:11.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:11 smithi171 conmon[51620]: debug 2022-01-31T21:47:10.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:11.000163+0000) 2022-01-31T21:47:11.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:11 smithi171 conmon[41853]: debug 2022-01-31T21:47:11.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:11.113431+0000) 2022-01-31T21:47:11.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:11 smithi171 conmon[46715]: debug 2022-01-31T21:47:11.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:11.158187+0000) 2022-01-31T21:47:11.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:11 smithi167 conmon[60316]: debug 2022-01-31T21:47:11.391+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:11.392579+0000) 2022-01-31T21:47:11.672 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:11 smithi167 conmon[54076]: debug 2022-01-31T21:47:11.391+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:11.392579+0000) 2022-01-31T21:47:12.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:11 smithi167 conmon[49112]: debug 2022-01-31T21:47:11.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:11.729206+0000) 2022-01-31T21:47:12.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:12 smithi171 conmon[51620]: debug 2022-01-31T21:47:11.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:12.000378+0000) 2022-01-31T21:47:12.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:12 smithi171 conmon[41853]: debug 2022-01-31T21:47:12.112+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:12.113594+0000) 2022-01-31T21:47:12.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:12 smithi171 conmon[46715]: debug 2022-01-31T21:47:12.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:12.158412+0000) 2022-01-31T21:47:12.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:12 smithi167 conmon[60316]: debug 2022-01-31T21:47:12.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:12.392774+0000) 2022-01-31T21:47:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:12 smithi167 conmon[54076]: debug 2022-01-31T21:47:12.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:12.392780+0000) 2022-01-31T21:47:13.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:13 smithi171 conmon[51620]: debug 2022-01-31T21:47:12.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:13.000573+0000) 2022-01-31T21:47:13.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:12 smithi167 conmon[49112]: debug 2022-01-31T21:47:12.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:12.729418+0000) 2022-01-31T21:47:13.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:13 smithi171 conmon[41853]: debug 2022-01-31T21:47:13.113+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:13.113796+0000) 2022-01-31T21:47:13.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:13 smithi171 conmon[46715]: debug 2022-01-31T21:47:13.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:13.158555+0000) 2022-01-31T21:47:13.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:13 smithi167 conmon[60316]: debug 2022-01-31T21:47:13.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:13.392988+0000) 2022-01-31T21:47:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:13 smithi167 conmon[54076]: debug 2022-01-31T21:47:13.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:13.392987+0000) 2022-01-31T21:47:14.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[51620]: debug 2022-01-31T21:47:14.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.000781+0000) 2022-01-31T21:47:14.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:13 smithi167 conmon[49112]: debug 2022-01-31T21:47:13.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:13.729631+0000) 2022-01-31T21:47:14.392 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:14 smithi167 conmon[49112]: debug 2022-01-31T21:47:14.309+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.311413+0000) 2022-01-31T21:47:14.392 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:14 smithi167 conmon[54076]: debug 2022-01-31T21:47:14.309+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.311308+0000) 2022-01-31T21:47:14.393 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:14 smithi167 conmon[60316]: debug 2022-01-31T21:47:14.309+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.311255+0000) 2022-01-31T21:47:14.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[35325]: debug 2022-01-31T21:47:14.327+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:14.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[46715]: debug 2022-01-31T21:47:14.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.158749+0000) 2022-01-31T21:47:14.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[46715]: debug 2022-01-31T21:47:14.310+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.311519+0000) 2022-01-31T21:47:14.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[41853]: debug 2022-01-31T21:47:14.113+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.113947+0000) 2022-01-31T21:47:14.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[41853]: debug 2022-01-31T21:47:14.311+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.312599+0000) 2022-01-31T21:47:14.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:14 smithi171 conmon[51620]: debug 2022-01-31T21:47:14.311+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.311858+0000) 2022-01-31T21:47:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:14 smithi167 conmon[54076]: debug 2022-01-31T21:47:14.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.393153+0000) 2022-01-31T21:47:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:14 smithi167 conmon[60316]: debug 2022-01-31T21:47:14.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.393153+0000) 2022-01-31T21:47:15.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:15 smithi171 conmon[51620]: debug 2022-01-31T21:47:15.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:15.000974+0000) 2022-01-31T21:47:15.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:14 smithi167 conmon[49112]: debug 2022-01-31T21:47:14.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:14.729836+0000) 2022-01-31T21:47:15.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:15 smithi171 conmon[41853]: debug 2022-01-31T21:47:15.113+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:15.114115+0000) 2022-01-31T21:47:15.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:15 smithi171 conmon[46715]: debug 2022-01-31T21:47:15.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:15.158947+0000) 2022-01-31T21:47:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:15 smithi167 conmon[54076]: debug 2022-01-31T21:47:15.392+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:15.393322+0000) 2022-01-31T21:47:15.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:15 smithi167 conmon[60316]: debug 2022-01-31T21:47:15.392+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:15.393322+0000) 2022-01-31T21:47:16.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:16 smithi171 conmon[51620]: debug 2022-01-31T21:47:16.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:16.001131+0000) 2022-01-31T21:47:16.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:15 smithi167 conmon[49112]: debug 2022-01-31T21:47:15.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:15.729996+0000) 2022-01-31T21:47:16.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:16 smithi171 conmon[41853]: debug 2022-01-31T21:47:16.113+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:16.114272+0000) 2022-01-31T21:47:16.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:16 smithi171 conmon[46715]: debug 2022-01-31T21:47:16.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:16.159077+0000) 2022-01-31T21:47:16.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:16 smithi167 conmon[54076]: debug 2022-01-31T21:47:16.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:16.393528+0000) 2022-01-31T21:47:16.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:16 smithi167 conmon[60316]: debug 2022-01-31T21:47:16.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:16.393528+0000) 2022-01-31T21:47:17.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:17 smithi171 conmon[51620]: debug 2022-01-31T21:47:17.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:17.001325+0000) 2022-01-31T21:47:17.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:16 smithi167 conmon[49112]: debug 2022-01-31T21:47:16.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:16.730202+0000) 2022-01-31T21:47:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:17 smithi171 conmon[41853]: debug 2022-01-31T21:47:17.113+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:17.114447+0000) 2022-01-31T21:47:17.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:17 smithi171 conmon[46715]: debug 2022-01-31T21:47:17.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:17.159174+0000) 2022-01-31T21:47:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:17 smithi167 conmon[54076]: debug 2022-01-31T21:47:17.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:17.393713+0000) 2022-01-31T21:47:17.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:17 smithi167 conmon[60316]: debug 2022-01-31T21:47:17.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:17.393713+0000) 2022-01-31T21:47:18.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:18 smithi171 conmon[51620]: debug 2022-01-31T21:47:18.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:18.001514+0000) 2022-01-31T21:47:18.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:17 smithi167 conmon[49112]: debug 2022-01-31T21:47:17.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:17.730405+0000) 2022-01-31T21:47:18.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:18 smithi171 conmon[46715]: debug 2022-01-31T21:47:18.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:18.159347+0000) 2022-01-31T21:47:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:18 smithi171 conmon[41853]: debug 2022-01-31T21:47:18.113+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:18.114643+0000) 2022-01-31T21:47:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:18 smithi167 conmon[54076]: debug 2022-01-31T21:47:18.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:18.393892+0000) 2022-01-31T21:47:18.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:18 smithi167 conmon[60316]: debug 2022-01-31T21:47:18.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:18.393892+0000) 2022-01-31T21:47:19.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[51620]: debug 2022-01-31T21:47:19.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.001736+0000) 2022-01-31T21:47:19.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:18 smithi167 conmon[49112]: debug 2022-01-31T21:47:18.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:18.730656+0000) 2022-01-31T21:47:19.392 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:19 smithi167 conmon[49112]: debug 2022-01-31T21:47:19.336+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.336952+0000) 2022-01-31T21:47:19.393 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:19 smithi167 conmon[54076]: debug 2022-01-31T21:47:19.330+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.330836+0000) 2022-01-31T21:47:19.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:19 smithi167 conmon[60316]: debug 2022-01-31T21:47:19.329+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.330252+0000) 2022-01-31T21:47:19.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[35325]: debug 2022-01-31T21:47:19.361+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:19.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[51620]: debug 2022-01-31T21:47:19.329+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.330484+0000) 2022-01-31T21:47:19.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[41853]: debug 2022-01-31T21:47:19.114+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.114822+0000) 2022-01-31T21:47:19.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[41853]: debug 2022-01-31T21:47:19.330+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.330945+0000) 2022-01-31T21:47:19.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[46715]: debug 2022-01-31T21:47:19.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.159555+0000) 2022-01-31T21:47:19.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:19 smithi171 conmon[46715]: debug 2022-01-31T21:47:19.329+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.330417+0000) 2022-01-31T21:47:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:19 smithi167 conmon[54076]: debug 2022-01-31T21:47:19.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.394018+0000) 2022-01-31T21:47:19.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:19 smithi167 conmon[60316]: debug 2022-01-31T21:47:19.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.394018+0000) 2022-01-31T21:47:20.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:20 smithi171 conmon[51620]: debug 2022-01-31T21:47:20.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:20.001923+0000) 2022-01-31T21:47:20.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:19 smithi167 conmon[49112]: debug 2022-01-31T21:47:19.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:19.730872+0000) 2022-01-31T21:47:20.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:20 smithi171 conmon[41853]: debug 2022-01-31T21:47:20.114+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:20.114944+0000) 2022-01-31T21:47:20.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:20 smithi171 conmon[46715]: debug 2022-01-31T21:47:20.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:20.159781+0000) 2022-01-31T21:47:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:20 smithi167 conmon[54076]: debug 2022-01-31T21:47:20.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:20.394201+0000) 2022-01-31T21:47:20.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:20 smithi167 conmon[60316]: debug 2022-01-31T21:47:20.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:20.394201+0000) 2022-01-31T21:47:21.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:21 smithi171 conmon[51620]: debug 2022-01-31T21:47:21.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:21.002103+0000) 2022-01-31T21:47:21.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:20 smithi167 conmon[49112]: debug 2022-01-31T21:47:20.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:20.731036+0000) 2022-01-31T21:47:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:21 smithi171 conmon[41853]: debug 2022-01-31T21:47:21.114+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:21.115063+0000) 2022-01-31T21:47:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:21 smithi171 conmon[46715]: debug 2022-01-31T21:47:21.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:21.159922+0000) 2022-01-31T21:47:21.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:21 smithi167 conmon[54076]: debug 2022-01-31T21:47:21.393+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:21.394420+0000) 2022-01-31T21:47:21.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:21 smithi167 conmon[60316]: debug 2022-01-31T21:47:21.393+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:21.394420+0000) 2022-01-31T21:47:22.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:22 smithi171 conmon[51620]: debug 2022-01-31T21:47:22.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:22.002291+0000) 2022-01-31T21:47:22.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:21 smithi167 conmon[49112]: debug 2022-01-31T21:47:21.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:21.731221+0000) 2022-01-31T21:47:22.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:22 smithi171 conmon[41853]: debug 2022-01-31T21:47:22.114+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:22.115269+0000) 2022-01-31T21:47:22.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:22 smithi171 conmon[46715]: debug 2022-01-31T21:47:22.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:22.160094+0000) 2022-01-31T21:47:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:22 smithi167 conmon[54076]: debug 2022-01-31T21:47:22.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:22.394595+0000) 2022-01-31T21:47:22.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:22 smithi167 conmon[60316]: debug 2022-01-31T21:47:22.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:22.394662+0000) 2022-01-31T21:47:23.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:23 smithi171 conmon[51620]: debug 2022-01-31T21:47:23.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:23.002510+0000) 2022-01-31T21:47:23.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:22 smithi167 conmon[49112]: debug 2022-01-31T21:47:22.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:22.731415+0000) 2022-01-31T21:47:23.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:23 smithi171 conmon[41853]: debug 2022-01-31T21:47:23.114+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:23.115496+0000) 2022-01-31T21:47:23.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:23 smithi171 conmon[46715]: debug 2022-01-31T21:47:23.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:23.160261+0000) 2022-01-31T21:47:23.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:23 smithi167 conmon[60316]: debug 2022-01-31T21:47:23.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:23.394829+0000) 2022-01-31T21:47:23.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:23 smithi167 conmon[54076]: debug 2022-01-31T21:47:23.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:23.394798+0000) 2022-01-31T21:47:24.000 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:23 smithi171 conmon[35325]: debug 2022-01-31T21:47:23.740+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:47:24.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:23 smithi167 conmon[49112]: debug 2022-01-31T21:47:23.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:23.731630+0000) 2022-01-31T21:47:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[41853]: debug 2022-01-31T21:47:24.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.115676+0000) 2022-01-31T21:47:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[46715]: debug 2022-01-31T21:47:24.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.160407+0000) 2022-01-31T21:47:24.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[51620]: debug 2022-01-31T21:47:24.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.002712+0000) 2022-01-31T21:47:24.393 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:24 smithi167 conmon[49112]: debug 2022-01-31T21:47:24.364+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.365362+0000) 2022-01-31T21:47:24.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:24 smithi167 conmon[54076]: debug 2022-01-31T21:47:24.364+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.364624+0000) 2022-01-31T21:47:24.394 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:24 smithi167 conmon[60316]: debug 2022-01-31T21:47:24.363+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.364289+0000) 2022-01-31T21:47:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:24 smithi167 conmon[54076]: debug 2022-01-31T21:47:24.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.394966+0000) 2022-01-31T21:47:24.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:24 smithi167 conmon[60316]: debug 2022-01-31T21:47:24.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.395011+0000) 2022-01-31T21:47:24.821 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[35325]: debug 2022-01-31T21:47:24.380+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157684 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:24.822 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[41853]: debug 2022-01-31T21:47:24.365+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.365746+0000) 2022-01-31T21:47:24.823 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[46715]: debug 2022-01-31T21:47:24.364+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.364836+0000) 2022-01-31T21:47:24.824 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:24 smithi171 conmon[51620]: debug 2022-01-31T21:47:24.363+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.364448+0000) 2022-01-31T21:47:25.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:25 smithi171 conmon[51620]: debug 2022-01-31T21:47:25.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:25.002938+0000) 2022-01-31T21:47:25.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:24 smithi167 conmon[49112]: debug 2022-01-31T21:47:24.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:24.731803+0000) 2022-01-31T21:47:25.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:25 smithi171 conmon[41853]: debug 2022-01-31T21:47:25.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:25.115806+0000) 2022-01-31T21:47:25.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:25 smithi171 conmon[46715]: debug 2022-01-31T21:47:25.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:25.160542+0000) 2022-01-31T21:47:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:25 smithi167 conmon[54076]: debug 2022-01-31T21:47:25.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:25.395149+0000) 2022-01-31T21:47:25.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:25 smithi167 conmon[60316]: debug 2022-01-31T21:47:25.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:25.395150+0000) 2022-01-31T21:47:26.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:26 smithi171 conmon[51620]: debug 2022-01-31T21:47:26.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:26.003099+0000) 2022-01-31T21:47:26.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:25 smithi167 conmon[49112]: debug 2022-01-31T21:47:25.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:25.731948+0000) 2022-01-31T21:47:26.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:26 smithi171 conmon[41853]: debug 2022-01-31T21:47:26.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:26.115911+0000) 2022-01-31T21:47:26.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:26 smithi171 conmon[46715]: debug 2022-01-31T21:47:26.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:26.160665+0000) 2022-01-31T21:47:26.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:26 smithi167 conmon[60316]: debug 2022-01-31T21:47:26.394+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:26.395351+0000) 2022-01-31T21:47:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:26 smithi167 conmon[54076]: debug 2022-01-31T21:47:26.394+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:26.395350+0000) 2022-01-31T21:47:27.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:27 smithi171 conmon[51620]: debug 2022-01-31T21:47:27.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:27.003301+0000) 2022-01-31T21:47:27.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:26 smithi167 conmon[49112]: debug 2022-01-31T21:47:26.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:26.732178+0000) 2022-01-31T21:47:27.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:27 smithi171 conmon[41853]: debug 2022-01-31T21:47:27.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:27.116106+0000) 2022-01-31T21:47:27.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:27 smithi171 conmon[46715]: debug 2022-01-31T21:47:27.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:27.160770+0000) 2022-01-31T21:47:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:27 smithi167 conmon[54076]: debug 2022-01-31T21:47:27.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:27.395548+0000) 2022-01-31T21:47:27.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:27 smithi167 conmon[60316]: debug 2022-01-31T21:47:27.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:27.395548+0000) 2022-01-31T21:47:28.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:28 smithi171 conmon[51620]: debug 2022-01-31T21:47:28.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:28.003504+0000) 2022-01-31T21:47:28.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:27 smithi167 conmon[49112]: debug 2022-01-31T21:47:27.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:27.732347+0000) 2022-01-31T21:47:28.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:28 smithi171 conmon[41853]: debug 2022-01-31T21:47:28.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:28.116295+0000) 2022-01-31T21:47:28.692 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:28 smithi171 conmon[46715]: debug 2022-01-31T21:47:28.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:28.160932+0000) 2022-01-31T21:47:28.693 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:28 smithi167 conmon[54076]: debug 2022-01-31T21:47:28.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:28.395675+0000) 2022-01-31T21:47:28.693 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:28 smithi167 conmon[60316]: debug 2022-01-31T21:47:28.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:28.395720+0000) 2022-01-31T21:47:29.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[51620]: debug 2022-01-31T21:47:29.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.003706+0000) 2022-01-31T21:47:29.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:28 smithi167 conmon[49112]: debug 2022-01-31T21:47:28.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:28.732658+0000) 2022-01-31T21:47:29.381 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[41853]: debug 2022-01-31T21:47:29.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.116530+0000) 2022-01-31T21:47:29.382 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[46715]: debug 2022-01-31T21:47:29.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.161184+0000) 2022-01-31T21:47:29.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:29 smithi167 conmon[49112]: debug 2022-01-31T21:47:29.384+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.384616+0000) 2022-01-31T21:47:29.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:29 smithi167 conmon[54076]: debug 2022-01-31T21:47:29.383+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.384033+0000) 2022-01-31T21:47:29.395 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:29 smithi167 conmon[60316]: debug 2022-01-31T21:47:29.382+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.383442+0000) 2022-01-31T21:47:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:29 smithi167 conmon[54076]: debug 2022-01-31T21:47:29.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.395824+0000) 2022-01-31T21:47:29.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:29 smithi167 conmon[60316]: debug 2022-01-31T21:47:29.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.395933+0000) 2022-01-31T21:47:29.823 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[35325]: debug 2022-01-31T21:47:29.399+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157792 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:29.823 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[41853]: debug 2022-01-31T21:47:29.383+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.384634+0000) 2022-01-31T21:47:29.824 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[46715]: debug 2022-01-31T21:47:29.383+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.384353+0000) 2022-01-31T21:47:29.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:29 smithi171 conmon[51620]: debug 2022-01-31T21:47:29.383+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.383927+0000) 2022-01-31T21:47:30.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:30 smithi171 conmon[51620]: debug 2022-01-31T21:47:30.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:30.003853+0000) 2022-01-31T21:47:30.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:29 smithi167 conmon[49112]: debug 2022-01-31T21:47:29.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:29.732838+0000) 2022-01-31T21:47:30.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:30 smithi171 conmon[41853]: debug 2022-01-31T21:47:30.116+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:30.116699+0000) 2022-01-31T21:47:30.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:30 smithi171 conmon[46715]: debug 2022-01-31T21:47:30.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:30.161359+0000) 2022-01-31T21:47:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:30 smithi167 conmon[54076]: debug 2022-01-31T21:47:30.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:30.396072+0000) 2022-01-31T21:47:30.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:30 smithi167 conmon[60316]: debug 2022-01-31T21:47:30.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:30.396077+0000) 2022-01-31T21:47:31.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:31 smithi171 conmon[51620]: debug 2022-01-31T21:47:31.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:31.004026+0000) 2022-01-31T21:47:31.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:30 smithi167 conmon[49112]: debug 2022-01-31T21:47:30.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:30.732996+0000) 2022-01-31T21:47:31.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:31 smithi171 conmon[41853]: debug 2022-01-31T21:47:31.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:31.116833+0000) 2022-01-31T21:47:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:31 smithi171 conmon[46715]: debug 2022-01-31T21:47:31.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:31.161545+0000) 2022-01-31T21:47:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:31 smithi167 conmon[54076]: debug 2022-01-31T21:47:31.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:31.396254+0000) 2022-01-31T21:47:31.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:31 smithi167 conmon[60316]: debug 2022-01-31T21:47:31.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:31.396294+0000) 2022-01-31T21:47:32.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:32 smithi171 conmon[51620]: debug 2022-01-31T21:47:32.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:32.004236+0000) 2022-01-31T21:47:32.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:31 smithi167 conmon[49112]: debug 2022-01-31T21:47:31.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:31.733166+0000) 2022-01-31T21:47:32.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:32 smithi171 conmon[46715]: debug 2022-01-31T21:47:32.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:32.161744+0000) 2022-01-31T21:47:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:32 smithi171 conmon[41853]: debug 2022-01-31T21:47:32.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:32.116954+0000) 2022-01-31T21:47:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:32 smithi167 conmon[54076]: debug 2022-01-31T21:47:32.395+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:32.396450+0000) 2022-01-31T21:47:32.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:32 smithi167 conmon[60316]: debug 2022-01-31T21:47:32.395+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:32.396452+0000) 2022-01-31T21:47:33.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:33 smithi171 conmon[51620]: debug 2022-01-31T21:47:33.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:33.004467+0000) 2022-01-31T21:47:33.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:32 smithi167 conmon[49112]: debug 2022-01-31T21:47:32.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:32.733385+0000) 2022-01-31T21:47:33.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:33 smithi167 conmon[60316]: debug 2022-01-31T21:47:33.396+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:33.396649+0000) 2022-01-31T21:47:33.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:33 smithi167 conmon[54076]: debug 2022-01-31T21:47:33.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:33.396648+0000) 2022-01-31T21:47:33.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:33 smithi171 conmon[41853]: debug 2022-01-31T21:47:33.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:33.117157+0000) 2022-01-31T21:47:33.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:33 smithi171 conmon[46715]: debug 2022-01-31T21:47:33.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:33.161905+0000) 2022-01-31T21:47:34.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[51620]: debug 2022-01-31T21:47:34.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.004681+0000) 2022-01-31T21:47:34.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:33 smithi167 conmon[49112]: debug 2022-01-31T21:47:33.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:33.733602+0000) 2022-01-31T21:47:34.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:34 smithi167 conmon[54076]: debug 2022-01-31T21:47:34.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.396860+0000) 2022-01-31T21:47:34.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:34 smithi167 conmon[60316]: debug 2022-01-31T21:47:34.396+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.396850+0000) 2022-01-31T21:47:34.401 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[46715]: debug 2022-01-31T21:47:34.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.162081+0000) 2022-01-31T21:47:34.402 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[41853]: debug 2022-01-31T21:47:34.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.117364+0000) 2022-01-31T21:47:34.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:34 smithi167 conmon[49112]: debug 2022-01-31T21:47:34.402+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.403404+0000) 2022-01-31T21:47:34.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:34 smithi167 conmon[54076]: debug 2022-01-31T21:47:34.401+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.402422+0000) 2022-01-31T21:47:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:34 smithi167 conmon[60316]: debug 2022-01-31T21:47:34.401+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.401853+0000) 2022-01-31T21:47:34.824 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[35325]: debug 2022-01-31T21:47:34.417+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 157905 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:34.824 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[41853]: debug 2022-01-31T21:47:34.401+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.402992+0000) 2022-01-31T21:47:34.825 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[46715]: debug 2022-01-31T21:47:34.402+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.404635+0000) 2022-01-31T21:47:34.826 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:34 smithi171 conmon[51620]: debug 2022-01-31T21:47:34.401+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.403307+0000) 2022-01-31T21:47:35.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:35 smithi171 conmon[51620]: debug 2022-01-31T21:47:35.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:35.004901+0000) 2022-01-31T21:47:35.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:34 smithi167 conmon[49112]: debug 2022-01-31T21:47:34.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:34.733816+0000) 2022-01-31T21:47:35.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:35 smithi167 conmon[60316]: debug 2022-01-31T21:47:35.396+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:35.397056+0000) 2022-01-31T21:47:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:35 smithi167 conmon[54076]: debug 2022-01-31T21:47:35.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:35.397056+0000) 2022-01-31T21:47:35.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:35 smithi171 conmon[46715]: debug 2022-01-31T21:47:35.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:35.162269+0000) 2022-01-31T21:47:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:35 smithi171 conmon[41853]: debug 2022-01-31T21:47:35.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:35.117553+0000) 2022-01-31T21:47:36.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:36 smithi171 conmon[51620]: debug 2022-01-31T21:47:36.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:36.005066+0000) 2022-01-31T21:47:36.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:35 smithi167 conmon[49112]: debug 2022-01-31T21:47:35.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:35.733959+0000) 2022-01-31T21:47:36.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:36 smithi167 conmon[54076]: debug 2022-01-31T21:47:36.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:36.397222+0000) 2022-01-31T21:47:36.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:36 smithi167 conmon[60316]: debug 2022-01-31T21:47:36.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:36.397223+0000) 2022-01-31T21:47:36.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:36 smithi171 conmon[41853]: debug 2022-01-31T21:47:36.115+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:36.117708+0000) 2022-01-31T21:47:36.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:36 smithi171 conmon[46715]: debug 2022-01-31T21:47:36.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:36.162413+0000) 2022-01-31T21:47:37.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:37 smithi171 conmon[51620]: debug 2022-01-31T21:47:37.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:37.005274+0000) 2022-01-31T21:47:37.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:36 smithi167 conmon[49112]: debug 2022-01-31T21:47:36.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:36.734171+0000) 2022-01-31T21:47:37.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:37 smithi167 conmon[54076]: debug 2022-01-31T21:47:37.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:37.397379+0000) 2022-01-31T21:47:37.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:37 smithi167 conmon[60316]: debug 2022-01-31T21:47:37.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:37.397426+0000) 2022-01-31T21:47:37.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:37 smithi171 conmon[41853]: debug 2022-01-31T21:47:37.116+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:37.117905+0000) 2022-01-31T21:47:37.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:37 smithi171 conmon[46715]: debug 2022-01-31T21:47:37.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:37.162630+0000) 2022-01-31T21:47:38.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:38 smithi171 conmon[51620]: debug 2022-01-31T21:47:38.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:38.005453+0000) 2022-01-31T21:47:38.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:37 smithi167 conmon[49112]: debug 2022-01-31T21:47:37.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:37.734381+0000) 2022-01-31T21:47:38.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:38 smithi167 conmon[54076]: debug 2022-01-31T21:47:38.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:38.397544+0000) 2022-01-31T21:47:38.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:38 smithi167 conmon[60316]: debug 2022-01-31T21:47:38.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:38.397543+0000) 2022-01-31T21:47:38.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:38 smithi171 conmon[41853]: debug 2022-01-31T21:47:38.116+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:38.118053+0000) 2022-01-31T21:47:38.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:38 smithi171 conmon[46715]: debug 2022-01-31T21:47:38.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:38.162796+0000) 2022-01-31T21:47:39.003 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:38 smithi171 conmon[35325]: debug 2022-01-31T21:47:38.740+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:47:39.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:38 smithi167 conmon[49112]: debug 2022-01-31T21:47:38.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:38.734612+0000) 2022-01-31T21:47:39.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[46715]: debug 2022-01-31T21:47:39.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.162926+0000) 2022-01-31T21:47:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[51620]: debug 2022-01-31T21:47:39.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.005634+0000) 2022-01-31T21:47:39.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[41853]: debug 2022-01-31T21:47:39.117+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.118234+0000) 2022-01-31T21:47:39.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:39 smithi167 conmon[54076]: debug 2022-01-31T21:47:39.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.397721+0000) 2022-01-31T21:47:39.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:39 smithi167 conmon[60316]: debug 2022-01-31T21:47:39.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.397719+0000) 2022-01-31T21:47:39.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:39 smithi167 conmon[49112]: debug 2022-01-31T21:47:39.421+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.422117+0000) 2022-01-31T21:47:39.709 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:39 smithi167 conmon[60316]: debug 2022-01-31T21:47:39.421+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.422204+0000) 2022-01-31T21:47:39.709 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:39 smithi167 conmon[54076]: debug 2022-01-31T21:47:39.421+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.421660+0000) 2022-01-31T21:47:39.825 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[35325]: debug 2022-01-31T21:47:39.437+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158013 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:39.826 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[41853]: debug 2022-01-31T21:47:39.420+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.421841+0000) 2022-01-31T21:47:39.826 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[46715]: debug 2022-01-31T21:47:39.420+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.422540+0000) 2022-01-31T21:47:39.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:39 smithi171 conmon[51620]: debug 2022-01-31T21:47:39.419+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.421362+0000) 2022-01-31T21:47:40.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:40 smithi171 conmon[51620]: debug 2022-01-31T21:47:40.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:40.005830+0000) 2022-01-31T21:47:40.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:39 smithi167 conmon[49112]: debug 2022-01-31T21:47:39.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:39.734788+0000) 2022-01-31T21:47:40.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:40 smithi167 conmon[54076]: debug 2022-01-31T21:47:40.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:40.397919+0000) 2022-01-31T21:47:40.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:40 smithi167 conmon[60316]: debug 2022-01-31T21:47:40.396+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:40.397885+0000) 2022-01-31T21:47:40.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:40 smithi171 conmon[41853]: debug 2022-01-31T21:47:40.117+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:40.118418+0000) 2022-01-31T21:47:40.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:40 smithi171 conmon[46715]: debug 2022-01-31T21:47:40.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:40.163081+0000) 2022-01-31T21:47:41.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:41 smithi171 conmon[51620]: debug 2022-01-31T21:47:41.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:41.005975+0000) 2022-01-31T21:47:41.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:40 smithi167 conmon[49112]: debug 2022-01-31T21:47:40.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:40.734918+0000) 2022-01-31T21:47:41.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:41 smithi167 conmon[60316]: debug 2022-01-31T21:47:41.396+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:41.398075+0000) 2022-01-31T21:47:41.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:41 smithi167 conmon[54076]: debug 2022-01-31T21:47:41.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:41.398075+0000) 2022-01-31T21:47:41.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:41 smithi171 conmon[41853]: debug 2022-01-31T21:47:41.117+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:41.118585+0000) 2022-01-31T21:47:41.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:41 smithi171 conmon[46715]: debug 2022-01-31T21:47:41.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:41.163265+0000) 2022-01-31T21:47:42.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:42 smithi171 conmon[51620]: debug 2022-01-31T21:47:42.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:42.006148+0000) 2022-01-31T21:47:42.165 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:41 smithi167 conmon[49112]: debug 2022-01-31T21:47:41.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:41.735111+0000) 2022-01-31T21:47:42.418 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:42 smithi167 conmon[60316]: debug 2022-01-31T21:47:42.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:42.398248+0000) 2022-01-31T21:47:42.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:42 smithi167 conmon[54076]: debug 2022-01-31T21:47:42.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:42.398287+0000) 2022-01-31T21:47:42.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:42 smithi171 conmon[41853]: debug 2022-01-31T21:47:42.118+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:42.118783+0000) 2022-01-31T21:47:42.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:42 smithi171 conmon[46715]: debug 2022-01-31T21:47:42.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:42.163467+0000) 2022-01-31T21:47:43.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:43 smithi171 conmon[51620]: debug 2022-01-31T21:47:43.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:43.006302+0000) 2022-01-31T21:47:43.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:42 smithi167 conmon[49112]: debug 2022-01-31T21:47:42.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:42.735319+0000) 2022-01-31T21:47:43.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:43 smithi167 conmon[54076]: debug 2022-01-31T21:47:43.396+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:43.398455+0000) 2022-01-31T21:47:43.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:43 smithi167 conmon[60316]: debug 2022-01-31T21:47:43.396+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:43.398455+0000) 2022-01-31T21:47:43.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:43 smithi171 conmon[41853]: debug 2022-01-31T21:47:43.118+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:43.118977+0000) 2022-01-31T21:47:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:43 smithi171 conmon[46715]: debug 2022-01-31T21:47:43.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:43.163659+0000) 2022-01-31T21:47:44.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[51620]: debug 2022-01-31T21:47:44.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.006491+0000) 2022-01-31T21:47:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:43 smithi167 conmon[49112]: debug 2022-01-31T21:47:43.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:43.735562+0000) 2022-01-31T21:47:44.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[46715]: debug 2022-01-31T21:47:44.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.163818+0000) 2022-01-31T21:47:44.439 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[41853]: debug 2022-01-31T21:47:44.118+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.119189+0000) 2022-01-31T21:47:44.439 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[41853]: debug 2022-01-31T21:47:44.440+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.441077+0000) 2022-01-31T21:47:44.440 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:44 smithi167 conmon[54076]: debug 2022-01-31T21:47:44.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.398677+0000) 2022-01-31T21:47:44.441 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:44 smithi167 conmon[60316]: debug 2022-01-31T21:47:44.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.398673+0000) 2022-01-31T21:47:44.709 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:44 smithi167 conmon[49112]: debug 2022-01-31T21:47:44.439+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.441211+0000) 2022-01-31T21:47:44.710 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:44 smithi167 conmon[54076]: debug 2022-01-31T21:47:44.439+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.441080+0000) 2022-01-31T21:47:44.710 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:44 smithi167 conmon[60316]: debug 2022-01-31T21:47:44.440+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.441771+0000) 2022-01-31T21:47:44.826 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[35325]: debug 2022-01-31T21:47:44.457+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158126 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:44.827 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[46715]: debug 2022-01-31T21:47:44.440+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.441572+0000) 2022-01-31T21:47:44.827 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:44 smithi171 conmon[51620]: debug 2022-01-31T21:47:44.440+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.441547+0000) 2022-01-31T21:47:45.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:45 smithi171 conmon[51620]: debug 2022-01-31T21:47:45.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:45.006655+0000) 2022-01-31T21:47:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:44 smithi167 conmon[49112]: debug 2022-01-31T21:47:44.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:44.735742+0000) 2022-01-31T21:47:45.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:45 smithi167 conmon[54076]: debug 2022-01-31T21:47:45.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:45.398894+0000) 2022-01-31T21:47:45.455 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:45 smithi167 conmon[60316]: debug 2022-01-31T21:47:45.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:45.398874+0000) 2022-01-31T21:47:45.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:45 smithi171 conmon[46715]: debug 2022-01-31T21:47:45.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:45.163971+0000) 2022-01-31T21:47:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:45 smithi171 conmon[41853]: debug 2022-01-31T21:47:45.118+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:45.119404+0000) 2022-01-31T21:47:46.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:46 smithi171 conmon[51620]: debug 2022-01-31T21:47:46.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:46.006819+0000) 2022-01-31T21:47:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:45 smithi167 conmon[49112]: debug 2022-01-31T21:47:45.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:45.735906+0000) 2022-01-31T21:47:46.367 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:46 smithi171 conmon[41853]: debug 2022-01-31T21:47:46.118+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:46.119562+0000) 2022-01-31T21:47:46.368 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:46 smithi171 conmon[46715]: debug 2022-01-31T21:47:46.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:46.164104+0000) 2022-01-31T21:47:46.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:46 smithi167 conmon[54076]: debug 2022-01-31T21:47:46.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:46.399033+0000) 2022-01-31T21:47:46.456 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:46 smithi167 conmon[60316]: debug 2022-01-31T21:47:46.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:46.399032+0000) 2022-01-31T21:47:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:46 smithi167 conmon[49112]: debug 2022-01-31T21:47:46.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:46.736074+0000) 2022-01-31T21:47:47.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:47 smithi171 conmon[46715]: debug 2022-01-31T21:47:47.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:47.164290+0000) 2022-01-31T21:47:47.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:47 smithi171 conmon[41853]: debug 2022-01-31T21:47:47.119+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:47.119725+0000) 2022-01-31T21:47:47.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:47 smithi171 conmon[51620]: debug 2022-01-31T21:47:47.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:47.007009+0000) 2022-01-31T21:47:47.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:47 smithi167 conmon[54076]: debug 2022-01-31T21:47:47.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:47.399223+0000) 2022-01-31T21:47:47.456 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:47 smithi167 conmon[60316]: debug 2022-01-31T21:47:47.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:47.399222+0000) 2022-01-31T21:47:48.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:48 smithi171 conmon[51620]: debug 2022-01-31T21:47:48.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:48.007233+0000) 2022-01-31T21:47:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:47 smithi167 conmon[49112]: debug 2022-01-31T21:47:47.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:47.736305+0000) 2022-01-31T21:47:48.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:48 smithi167 conmon[54076]: debug 2022-01-31T21:47:48.397+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:48.399428+0000) 2022-01-31T21:47:48.456 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:48 smithi167 conmon[60316]: debug 2022-01-31T21:47:48.397+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:48.399426+0000) 2022-01-31T21:47:48.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:48 smithi171 conmon[41853]: debug 2022-01-31T21:47:48.119+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:48.119932+0000) 2022-01-31T21:47:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:48 smithi171 conmon[46715]: debug 2022-01-31T21:47:48.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:48.164479+0000) 2022-01-31T21:47:49.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[51620]: debug 2022-01-31T21:47:49.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.007397+0000) 2022-01-31T21:47:49.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:48 smithi167 conmon[49112]: debug 2022-01-31T21:47:48.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:48.736564+0000) 2022-01-31T21:47:49.456 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:49 smithi167 conmon[54076]: debug 2022-01-31T21:47:49.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.399569+0000) 2022-01-31T21:47:49.456 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:49 smithi167 conmon[60316]: debug 2022-01-31T21:47:49.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.399593+0000) 2022-01-31T21:47:49.457 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[41853]: debug 2022-01-31T21:47:49.119+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.120087+0000) 2022-01-31T21:47:49.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[51620]: debug 2022-01-31T21:47:49.459+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.460001+0000) 2022-01-31T21:47:49.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[46715]: debug 2022-01-31T21:47:49.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.164673+0000) 2022-01-31T21:47:49.710 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:49 smithi167 conmon[49112]: debug 2022-01-31T21:47:49.458+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.460072+0000) 2022-01-31T21:47:49.711 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:49 smithi167 conmon[54076]: debug 2022-01-31T21:47:49.459+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.460743+0000) 2022-01-31T21:47:49.711 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:49 smithi167 conmon[60316]: debug 2022-01-31T21:47:49.459+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.460912+0000) 2022-01-31T21:47:49.827 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[35325]: debug 2022-01-31T21:47:49.476+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158234 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:49.828 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[41853]: debug 2022-01-31T21:47:49.459+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.460370+0000) 2022-01-31T21:47:49.828 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:49 smithi171 conmon[46715]: debug 2022-01-31T21:47:49.460+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.461359+0000) 2022-01-31T21:47:50.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:50 smithi171 conmon[51620]: debug 2022-01-31T21:47:50.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:50.007580+0000) 2022-01-31T21:47:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:49 smithi167 conmon[49112]: debug 2022-01-31T21:47:49.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:49.736799+0000) 2022-01-31T21:47:50.456 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:50 smithi167 conmon[54076]: debug 2022-01-31T21:47:50.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:50.399743+0000) 2022-01-31T21:47:50.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:50 smithi167 conmon[60316]: debug 2022-01-31T21:47:50.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:50.399744+0000) 2022-01-31T21:47:50.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:50 smithi171 conmon[41853]: debug 2022-01-31T21:47:50.119+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:50.120296+0000) 2022-01-31T21:47:50.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:50 smithi171 conmon[46715]: debug 2022-01-31T21:47:50.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:50.164856+0000) 2022-01-31T21:47:51.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:51 smithi171 conmon[51620]: debug 2022-01-31T21:47:51.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:51.007707+0000) 2022-01-31T21:47:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:50 smithi167 conmon[49112]: debug 2022-01-31T21:47:50.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:50.736932+0000) 2022-01-31T21:47:51.456 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:51 smithi167 conmon[54076]: debug 2022-01-31T21:47:51.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:51.399859+0000) 2022-01-31T21:47:51.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:51 smithi167 conmon[60316]: debug 2022-01-31T21:47:51.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:51.399870+0000) 2022-01-31T21:47:51.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:51 smithi171 conmon[46715]: debug 2022-01-31T21:47:51.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:51.165007+0000) 2022-01-31T21:47:51.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:51 smithi171 conmon[41853]: debug 2022-01-31T21:47:51.119+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:51.120468+0000) 2022-01-31T21:47:52.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:52 smithi171 conmon[51620]: debug 2022-01-31T21:47:52.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:52.007924+0000) 2022-01-31T21:47:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:51 smithi167 conmon[49112]: debug 2022-01-31T21:47:51.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:51.737130+0000) 2022-01-31T21:47:52.456 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:52 smithi167 conmon[54076]: debug 2022-01-31T21:47:52.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:52.400074+0000) 2022-01-31T21:47:52.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:52 smithi167 conmon[60316]: debug 2022-01-31T21:47:52.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:52.400075+0000) 2022-01-31T21:47:52.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:52 smithi171 conmon[46715]: debug 2022-01-31T21:47:52.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:52.165165+0000) 2022-01-31T21:47:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:52 smithi171 conmon[41853]: debug 2022-01-31T21:47:52.119+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:52.120644+0000) 2022-01-31T21:47:53.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:53 smithi171 conmon[51620]: debug 2022-01-31T21:47:53.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:53.008055+0000) 2022-01-31T21:47:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:52 smithi167 conmon[49112]: debug 2022-01-31T21:47:52.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:52.737327+0000) 2022-01-31T21:47:53.457 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:53 smithi167 conmon[54076]: debug 2022-01-31T21:47:53.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:53.400274+0000) 2022-01-31T21:47:53.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:53 smithi167 conmon[60316]: debug 2022-01-31T21:47:53.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:53.400252+0000) 2022-01-31T21:47:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:53 smithi171 conmon[41853]: debug 2022-01-31T21:47:53.120+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:53.120807+0000) 2022-01-31T21:47:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:53 smithi171 conmon[46715]: debug 2022-01-31T21:47:53.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:53.165334+0000) 2022-01-31T21:47:54.006 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:53 smithi171 conmon[35325]: debug 2022-01-31T21:47:53.741+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:47:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:53 smithi167 conmon[49112]: debug 2022-01-31T21:47:53.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:53.737423+0000) 2022-01-31T21:47:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[41853]: debug 2022-01-31T21:47:54.120+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.120965+0000) 2022-01-31T21:47:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[46715]: debug 2022-01-31T21:47:54.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.165547+0000) 2022-01-31T21:47:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[51620]: debug 2022-01-31T21:47:54.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.008230+0000) 2022-01-31T21:47:54.456 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:54 smithi167 conmon[54076]: debug 2022-01-31T21:47:54.398+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.400462+0000) 2022-01-31T21:47:54.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:54 smithi167 conmon[60316]: debug 2022-01-31T21:47:54.398+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.400470+0000) 2022-01-31T21:47:54.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:54 smithi167 conmon[49112]: debug 2022-01-31T21:47:54.477+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.479413+0000) 2022-01-31T21:47:54.711 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:54 smithi167 conmon[54076]: debug 2022-01-31T21:47:54.478+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.479682+0000) 2022-01-31T21:47:54.712 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:54 smithi167 conmon[60316]: debug 2022-01-31T21:47:54.478+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.480169+0000) 2022-01-31T21:47:54.828 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[35325]: debug 2022-01-31T21:47:54.495+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158349 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:54.828 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[41853]: debug 2022-01-31T21:47:54.478+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.479389+0000) 2022-01-31T21:47:54.829 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[46715]: debug 2022-01-31T21:47:54.479+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.480302+0000) 2022-01-31T21:47:54.830 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:54 smithi171 conmon[51620]: debug 2022-01-31T21:47:54.478+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.478830+0000) 2022-01-31T21:47:55.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:55 smithi171 conmon[51620]: debug 2022-01-31T21:47:55.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:55.008446+0000) 2022-01-31T21:47:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:54 smithi167 conmon[49112]: debug 2022-01-31T21:47:54.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:54.737603+0000) 2022-01-31T21:47:55.457 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:55 smithi167 conmon[54076]: debug 2022-01-31T21:47:55.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:55.400651+0000) 2022-01-31T21:47:55.457 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:55 smithi167 conmon[60316]: debug 2022-01-31T21:47:55.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:55.400645+0000) 2022-01-31T21:47:55.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:55 smithi171 conmon[41853]: debug 2022-01-31T21:47:55.120+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:55.121093+0000) 2022-01-31T21:47:55.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:55 smithi171 conmon[46715]: debug 2022-01-31T21:47:55.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:55.165713+0000) 2022-01-31T21:47:56.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:56 smithi171 conmon[51620]: debug 2022-01-31T21:47:56.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:56.008597+0000) 2022-01-31T21:47:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:55 smithi167 conmon[49112]: debug 2022-01-31T21:47:55.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:55.737774+0000) 2022-01-31T21:47:56.457 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:56 smithi167 conmon[54076]: debug 2022-01-31T21:47:56.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:56.400830+0000) 2022-01-31T21:47:56.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:56 smithi167 conmon[60316]: debug 2022-01-31T21:47:56.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:56.400834+0000) 2022-01-31T21:47:56.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:56 smithi171 conmon[46715]: debug 2022-01-31T21:47:56.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:56.165864+0000) 2022-01-31T21:47:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:56 smithi171 conmon[41853]: debug 2022-01-31T21:47:56.120+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:56.121248+0000) 2022-01-31T21:47:57.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:57 smithi171 conmon[51620]: debug 2022-01-31T21:47:57.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:57.008817+0000) 2022-01-31T21:47:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:56 smithi167 conmon[49112]: debug 2022-01-31T21:47:56.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:56.737991+0000) 2022-01-31T21:47:57.457 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:57 smithi167 conmon[54076]: debug 2022-01-31T21:47:57.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:57.401041+0000) 2022-01-31T21:47:57.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:57 smithi167 conmon[60316]: debug 2022-01-31T21:47:57.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:57.401062+0000) 2022-01-31T21:47:57.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:57 smithi171 conmon[46715]: debug 2022-01-31T21:47:57.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:57.166038+0000) 2022-01-31T21:47:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:57 smithi171 conmon[41853]: debug 2022-01-31T21:47:57.120+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:57.121455+0000) 2022-01-31T21:47:58.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:58 smithi171 conmon[51620]: debug 2022-01-31T21:47:58.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:58.008962+0000) 2022-01-31T21:47:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:57 smithi167 conmon[49112]: debug 2022-01-31T21:47:57.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:57.738201+0000) 2022-01-31T21:47:58.457 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:58 smithi167 conmon[54076]: debug 2022-01-31T21:47:58.399+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:58.401240+0000) 2022-01-31T21:47:58.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:58 smithi167 conmon[60316]: debug 2022-01-31T21:47:58.399+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:58.401240+0000) 2022-01-31T21:47:58.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:58 smithi171 conmon[46715]: debug 2022-01-31T21:47:58.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:58.166229+0000) 2022-01-31T21:47:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:58 smithi171 conmon[41853]: debug 2022-01-31T21:47:58.120+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:58.121643+0000) 2022-01-31T21:47:59.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[51620]: debug 2022-01-31T21:47:59.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.009146+0000) 2022-01-31T21:47:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:58 smithi167 conmon[49112]: debug 2022-01-31T21:47:58.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:58.738452+0000) 2022-01-31T21:47:59.458 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:59 smithi167 conmon[54076]: debug 2022-01-31T21:47:59.400+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.401467+0000) 2022-01-31T21:47:59.470 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:59 smithi167 conmon[60316]: debug 2022-01-31T21:47:59.400+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.401467+0000) 2022-01-31T21:47:59.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[46715]: debug 2022-01-31T21:47:59.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.166407+0000) 2022-01-31T21:47:59.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[41853]: debug 2022-01-31T21:47:59.121+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.121797+0000) 2022-01-31T21:47:59.711 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:59 smithi167 conmon[49112]: debug 2022-01-31T21:47:59.497+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.499148+0000) 2022-01-31T21:47:59.712 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:47:59 smithi167 conmon[54076]: debug 2022-01-31T21:47:59.497+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.499427+0000) 2022-01-31T21:47:59.713 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:47:59 smithi167 conmon[60316]: debug 2022-01-31T21:47:59.498+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.500368+0000) 2022-01-31T21:47:59.829 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[35325]: debug 2022-01-31T21:47:59.515+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158457 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:47:59.829 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[41853]: debug 2022-01-31T21:47:59.497+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.498293+0000) 2022-01-31T21:47:59.830 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[46715]: debug 2022-01-31T21:47:59.497+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.498726+0000) 2022-01-31T21:47:59.831 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:47:59 smithi171 conmon[51620]: debug 2022-01-31T21:47:59.497+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.497974+0000) 2022-01-31T21:48:00.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:00 smithi171 conmon[51620]: debug 2022-01-31T21:48:00.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:00.009318+0000) 2022-01-31T21:48:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:47:59 smithi167 conmon[49112]: debug 2022-01-31T21:47:59.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:47:59.738599+0000) 2022-01-31T21:48:00.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:00 smithi167 conmon[60316]: debug 2022-01-31T21:48:00.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:00.401697+0000) 2022-01-31T21:48:00.458 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:00 smithi167 conmon[54076]: debug 2022-01-31T21:48:00.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:00.401700+0000) 2022-01-31T21:48:00.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:00 smithi171 conmon[41853]: debug 2022-01-31T21:48:00.121+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:00.121976+0000) 2022-01-31T21:48:00.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:00 smithi171 conmon[46715]: debug 2022-01-31T21:48:00.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:00.166552+0000) 2022-01-31T21:48:01.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:01 smithi171 conmon[51620]: debug 2022-01-31T21:48:01.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:01.009457+0000) 2022-01-31T21:48:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:00 smithi167 conmon[49112]: debug 2022-01-31T21:48:00.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:00.738717+0000) 2022-01-31T21:48:01.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:01 smithi167 conmon[60316]: debug 2022-01-31T21:48:01.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:01.401878+0000) 2022-01-31T21:48:01.459 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:01 smithi167 conmon[54076]: debug 2022-01-31T21:48:01.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:01.401875+0000) 2022-01-31T21:48:01.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:01 smithi171 conmon[41853]: debug 2022-01-31T21:48:01.121+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:01.122124+0000) 2022-01-31T21:48:01.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:01 smithi171 conmon[46715]: debug 2022-01-31T21:48:01.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:01.166723+0000) 2022-01-31T21:48:02.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:02 smithi171 conmon[51620]: debug 2022-01-31T21:48:02.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:02.009617+0000) 2022-01-31T21:48:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:01 smithi167 conmon[49112]: debug 2022-01-31T21:48:01.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:01.738921+0000) 2022-01-31T21:48:02.458 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:02 smithi167 conmon[60316]: debug 2022-01-31T21:48:02.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:02.402085+0000) 2022-01-31T21:48:02.459 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:02 smithi167 conmon[54076]: debug 2022-01-31T21:48:02.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:02.402084+0000) 2022-01-31T21:48:02.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:02 smithi171 conmon[46715]: debug 2022-01-31T21:48:02.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:02.166956+0000) 2022-01-31T21:48:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:02 smithi171 conmon[41853]: debug 2022-01-31T21:48:02.121+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:02.122311+0000) 2022-01-31T21:48:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:02 smithi167 conmon[49112]: debug 2022-01-31T21:48:02.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:02.739108+0000) 2022-01-31T21:48:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:03 smithi171 conmon[41853]: debug 2022-01-31T21:48:03.122+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:03.122471+0000) 2022-01-31T21:48:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:03 smithi171 conmon[46715]: debug 2022-01-31T21:48:03.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:03.167136+0000) 2022-01-31T21:48:03.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:03 smithi171 conmon[51620]: debug 2022-01-31T21:48:03.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:03.009771+0000) 2022-01-31T21:48:03.636 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:03 smithi167 conmon[60316]: debug 2022-01-31T21:48:03.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:03.402287+0000) 2022-01-31T21:48:03.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:03 smithi167 conmon[54076]: debug 2022-01-31T21:48:03.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:03.402282+0000) 2022-01-31T21:48:03.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:03 smithi167 conmon[49112]: debug 2022-01-31T21:48:03.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:03.739327+0000) 2022-01-31T21:48:04.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[51620]: debug 2022-01-31T21:48:04.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.009968+0000) 2022-01-31T21:48:04.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:04 smithi167 conmon[54076]: debug 2022-01-31T21:48:04.401+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.402447+0000) 2022-01-31T21:48:04.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[46715]: debug 2022-01-31T21:48:04.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.167299+0000) 2022-01-31T21:48:04.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:04 smithi167 conmon[60316]: debug 2022-01-31T21:48:04.401+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.402463+0000) 2022-01-31T21:48:04.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[41853]: debug 2022-01-31T21:48:04.121+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.122676+0000) 2022-01-31T21:48:04.830 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[35325]: debug 2022-01-31T21:48:04.534+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158570 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:04.830 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[41853]: debug 2022-01-31T21:48:04.517+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.517942+0000) 2022-01-31T21:48:04.831 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[46715]: debug 2022-01-31T21:48:04.517+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.518171+0000) 2022-01-31T21:48:04.832 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:04 smithi171 conmon[51620]: debug 2022-01-31T21:48:04.517+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.518494+0000) 2022-01-31T21:48:04.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:04 smithi167 conmon[49112]: debug 2022-01-31T21:48:04.517+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.517654+0000) 2022-01-31T21:48:04.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:04 smithi167 conmon[49112]: debug 2022-01-31T21:48:04.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.739538+0000) 2022-01-31T21:48:04.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:04 smithi167 conmon[54076]: debug 2022-01-31T21:48:04.517+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.518253+0000) 2022-01-31T21:48:04.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:04 smithi167 conmon[60316]: debug 2022-01-31T21:48:04.518+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:04.518997+0000) 2022-01-31T21:48:05.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:05 smithi171 conmon[51620]: debug 2022-01-31T21:48:05.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:05.010116+0000) 2022-01-31T21:48:05.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:05 smithi171 conmon[41853]: debug 2022-01-31T21:48:05.122+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:05.122870+0000) 2022-01-31T21:48:05.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:05 smithi171 conmon[46715]: debug 2022-01-31T21:48:05.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:05.167512+0000) 2022-01-31T21:48:05.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:05 smithi167 conmon[54076]: debug 2022-01-31T21:48:05.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:05.402659+0000) 2022-01-31T21:48:05.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:05 smithi167 conmon[60316]: debug 2022-01-31T21:48:05.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:05.402663+0000) 2022-01-31T21:48:05.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:05 smithi167 conmon[49112]: debug 2022-01-31T21:48:05.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:05.739703+0000) 2022-01-31T21:48:06.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:06 smithi171 conmon[51620]: debug 2022-01-31T21:48:06.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:06.010265+0000) 2022-01-31T21:48:06.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:06 smithi171 conmon[41853]: debug 2022-01-31T21:48:06.122+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:06.122958+0000) 2022-01-31T21:48:06.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:06 smithi171 conmon[46715]: debug 2022-01-31T21:48:06.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:06.167688+0000) 2022-01-31T21:48:06.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:06 smithi167 conmon[54076]: debug 2022-01-31T21:48:06.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:06.402837+0000) 2022-01-31T21:48:06.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:06 smithi167 conmon[60316]: debug 2022-01-31T21:48:06.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:06.402815+0000) 2022-01-31T21:48:06.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:06 smithi167 conmon[49112]: debug 2022-01-31T21:48:06.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:06.739895+0000) 2022-01-31T21:48:07.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:07 smithi171 conmon[51620]: debug 2022-01-31T21:48:07.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:07.010420+0000) 2022-01-31T21:48:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:07 smithi171 conmon[41853]: debug 2022-01-31T21:48:07.122+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:07.123141+0000) 2022-01-31T21:48:07.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:07 smithi171 conmon[46715]: debug 2022-01-31T21:48:07.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:07.167847+0000) 2022-01-31T21:48:07.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:07 smithi167 conmon[54076]: debug 2022-01-31T21:48:07.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:07.402996+0000) 2022-01-31T21:48:07.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:07 smithi167 conmon[60316]: debug 2022-01-31T21:48:07.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:07.403036+0000) 2022-01-31T21:48:07.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:07 smithi167 conmon[49112]: debug 2022-01-31T21:48:07.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:07.740091+0000) 2022-01-31T21:48:08.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:08 smithi171 conmon[51620]: debug 2022-01-31T21:48:08.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:08.010611+0000) 2022-01-31T21:48:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:08 smithi171 conmon[41853]: debug 2022-01-31T21:48:08.122+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:08.123269+0000) 2022-01-31T21:48:08.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:08 smithi171 conmon[46715]: debug 2022-01-31T21:48:08.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:08.168039+0000) 2022-01-31T21:48:08.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:08 smithi167 conmon[54076]: debug 2022-01-31T21:48:08.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:08.403185+0000) 2022-01-31T21:48:08.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:08 smithi167 conmon[60316]: debug 2022-01-31T21:48:08.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:08.403195+0000) 2022-01-31T21:48:08.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:08 smithi167 conmon[49112]: debug 2022-01-31T21:48:08.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:08.740283+0000) 2022-01-31T21:48:09.008 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:08 smithi171 conmon[35325]: debug 2022-01-31T21:48:08.742+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:48:09.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[46715]: debug 2022-01-31T21:48:09.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.168195+0000) 2022-01-31T21:48:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[51620]: debug 2022-01-31T21:48:09.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.010768+0000) 2022-01-31T21:48:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[41853]: debug 2022-01-31T21:48:09.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.123422+0000) 2022-01-31T21:48:09.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:09 smithi167 conmon[54076]: debug 2022-01-31T21:48:09.402+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.403301+0000) 2022-01-31T21:48:09.535 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:09 smithi167 conmon[60316]: debug 2022-01-31T21:48:09.402+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.403375+0000) 2022-01-31T21:48:09.831 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[35325]: debug 2022-01-31T21:48:09.553+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158678 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:09.832 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[41853]: debug 2022-01-31T21:48:09.536+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.536936+0000) 2022-01-31T21:48:09.832 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[46715]: debug 2022-01-31T21:48:09.536+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.537110+0000) 2022-01-31T21:48:09.833 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:09 smithi171 conmon[51620]: debug 2022-01-31T21:48:09.535+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.536641+0000) 2022-01-31T21:48:09.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:09 smithi167 conmon[49112]: debug 2022-01-31T21:48:09.536+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.536940+0000) 2022-01-31T21:48:09.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:09 smithi167 conmon[49112]: debug 2022-01-31T21:48:09.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.740477+0000) 2022-01-31T21:48:09.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:09 smithi167 conmon[54076]: debug 2022-01-31T21:48:09.535+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.536409+0000) 2022-01-31T21:48:09.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:09 smithi167 conmon[60316]: debug 2022-01-31T21:48:09.536+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:09.537223+0000) 2022-01-31T21:48:10.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:10 smithi171 conmon[51620]: debug 2022-01-31T21:48:10.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:10.010918+0000) 2022-01-31T21:48:10.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:10 smithi171 conmon[41853]: debug 2022-01-31T21:48:10.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:10.123607+0000) 2022-01-31T21:48:10.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:10 smithi171 conmon[46715]: debug 2022-01-31T21:48:10.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:10.168378+0000) 2022-01-31T21:48:10.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:10 smithi167 conmon[54076]: debug 2022-01-31T21:48:10.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:10.403535+0000) 2022-01-31T21:48:10.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:10 smithi167 conmon[60316]: debug 2022-01-31T21:48:10.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:10.403538+0000) 2022-01-31T21:48:10.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:10 smithi167 conmon[49112]: debug 2022-01-31T21:48:10.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:10.740662+0000) 2022-01-31T21:48:11.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:11 smithi171 conmon[51620]: debug 2022-01-31T21:48:11.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:11.011170+0000) 2022-01-31T21:48:11.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:11 smithi171 conmon[46715]: debug 2022-01-31T21:48:11.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:11.168526+0000) 2022-01-31T21:48:11.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:11 smithi171 conmon[41853]: debug 2022-01-31T21:48:11.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:11.123786+0000) 2022-01-31T21:48:11.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:11 smithi167 conmon[54076]: debug 2022-01-31T21:48:11.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:11.403697+0000) 2022-01-31T21:48:11.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:11 smithi167 conmon[60316]: debug 2022-01-31T21:48:11.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:11.403742+0000) 2022-01-31T21:48:11.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:11 smithi167 conmon[49112]: debug 2022-01-31T21:48:11.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:11.740830+0000) 2022-01-31T21:48:12.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:12 smithi171 conmon[51620]: debug 2022-01-31T21:48:12.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:12.011322+0000) 2022-01-31T21:48:12.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:12 smithi171 conmon[46715]: debug 2022-01-31T21:48:12.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:12.168712+0000) 2022-01-31T21:48:12.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:12 smithi171 conmon[41853]: debug 2022-01-31T21:48:12.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:12.124015+0000) 2022-01-31T21:48:12.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:12 smithi167 conmon[54076]: debug 2022-01-31T21:48:12.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:12.403847+0000) 2022-01-31T21:48:12.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:12 smithi167 conmon[60316]: debug 2022-01-31T21:48:12.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:12.403893+0000) 2022-01-31T21:48:12.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:12 smithi167 conmon[49112]: debug 2022-01-31T21:48:12.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:12.740979+0000) 2022-01-31T21:48:13.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:13 smithi171 conmon[51620]: debug 2022-01-31T21:48:13.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:13.011484+0000) 2022-01-31T21:48:13.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:13 smithi171 conmon[41853]: debug 2022-01-31T21:48:13.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:13.124140+0000) 2022-01-31T21:48:13.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:13 smithi171 conmon[46715]: debug 2022-01-31T21:48:13.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:13.168853+0000) 2022-01-31T21:48:13.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:13 smithi167 conmon[54076]: debug 2022-01-31T21:48:13.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:13.404023+0000) 2022-01-31T21:48:13.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:13 smithi167 conmon[60316]: debug 2022-01-31T21:48:13.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:13.404055+0000) 2022-01-31T21:48:13.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:13 smithi167 conmon[49112]: debug 2022-01-31T21:48:13.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:13.741189+0000) 2022-01-31T21:48:14.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[51620]: debug 2022-01-31T21:48:14.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.011699+0000) 2022-01-31T21:48:14.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[41853]: debug 2022-01-31T21:48:14.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.124317+0000) 2022-01-31T21:48:14.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[46715]: debug 2022-01-31T21:48:14.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.169024+0000) 2022-01-31T21:48:14.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:14 smithi167 conmon[54076]: debug 2022-01-31T21:48:14.403+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.404205+0000) 2022-01-31T21:48:14.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:14 smithi167 conmon[60316]: debug 2022-01-31T21:48:14.403+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.404255+0000) 2022-01-31T21:48:14.832 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[35325]: debug 2022-01-31T21:48:14.573+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158792 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:14.832 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[41853]: debug 2022-01-31T21:48:14.556+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.557520+0000) 2022-01-31T21:48:14.833 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[46715]: debug 2022-01-31T21:48:14.554+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.555738+0000) 2022-01-31T21:48:14.834 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:14 smithi171 conmon[51620]: debug 2022-01-31T21:48:14.555+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.556039+0000) 2022-01-31T21:48:14.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:14 smithi167 conmon[49112]: debug 2022-01-31T21:48:14.554+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.555241+0000) 2022-01-31T21:48:14.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:14 smithi167 conmon[49112]: debug 2022-01-31T21:48:14.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.741412+0000) 2022-01-31T21:48:14.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:14 smithi167 conmon[54076]: debug 2022-01-31T21:48:14.554+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.555438+0000) 2022-01-31T21:48:14.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:14 smithi167 conmon[60316]: debug 2022-01-31T21:48:14.555+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:14.555851+0000) 2022-01-31T21:48:15.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:15 smithi171 conmon[51620]: debug 2022-01-31T21:48:15.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:15.011906+0000) 2022-01-31T21:48:15.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:15 smithi171 conmon[41853]: debug 2022-01-31T21:48:15.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:15.124497+0000) 2022-01-31T21:48:15.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:15 smithi171 conmon[46715]: debug 2022-01-31T21:48:15.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:15.169175+0000) 2022-01-31T21:48:15.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:15 smithi167 conmon[54076]: debug 2022-01-31T21:48:15.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:15.404413+0000) 2022-01-31T21:48:15.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:15 smithi167 conmon[60316]: debug 2022-01-31T21:48:15.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:15.404414+0000) 2022-01-31T21:48:15.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:15 smithi167 conmon[49112]: debug 2022-01-31T21:48:15.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:15.741580+0000) 2022-01-31T21:48:16.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:16 smithi171 conmon[51620]: debug 2022-01-31T21:48:16.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:16.012064+0000) 2022-01-31T21:48:16.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:16 smithi171 conmon[41853]: debug 2022-01-31T21:48:16.124+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:16.124648+0000) 2022-01-31T21:48:16.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:16 smithi171 conmon[46715]: debug 2022-01-31T21:48:16.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:16.169358+0000) 2022-01-31T21:48:16.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:16 smithi167 conmon[54076]: debug 2022-01-31T21:48:16.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:16.404575+0000) 2022-01-31T21:48:16.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:16 smithi167 conmon[60316]: debug 2022-01-31T21:48:16.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:16.404575+0000) 2022-01-31T21:48:16.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:16 smithi167 conmon[49112]: debug 2022-01-31T21:48:16.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:16.741759+0000) 2022-01-31T21:48:17.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:17 smithi171 conmon[51620]: debug 2022-01-31T21:48:17.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:17.012248+0000) 2022-01-31T21:48:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:17 smithi171 conmon[41853]: debug 2022-01-31T21:48:17.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:17.124838+0000) 2022-01-31T21:48:17.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:17 smithi171 conmon[46715]: debug 2022-01-31T21:48:17.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:17.169560+0000) 2022-01-31T21:48:17.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:17 smithi167 conmon[54076]: debug 2022-01-31T21:48:17.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:17.404792+0000) 2022-01-31T21:48:17.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:17 smithi167 conmon[60316]: debug 2022-01-31T21:48:17.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:17.404791+0000) 2022-01-31T21:48:17.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:17 smithi167 conmon[49112]: debug 2022-01-31T21:48:17.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:17.741973+0000) 2022-01-31T21:48:18.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:18 smithi171 conmon[51620]: debug 2022-01-31T21:48:18.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:18.012448+0000) 2022-01-31T21:48:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:18 smithi171 conmon[41853]: debug 2022-01-31T21:48:18.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:18.125019+0000) 2022-01-31T21:48:18.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:18 smithi171 conmon[46715]: debug 2022-01-31T21:48:18.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:18.169743+0000) 2022-01-31T21:48:18.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:18 smithi167 conmon[54076]: debug 2022-01-31T21:48:18.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:18.405025+0000) 2022-01-31T21:48:18.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:18 smithi167 conmon[60316]: debug 2022-01-31T21:48:18.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:18.405019+0000) 2022-01-31T21:48:18.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:18 smithi167 conmon[49112]: debug 2022-01-31T21:48:18.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:18.742200+0000) 2022-01-31T21:48:19.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[51620]: debug 2022-01-31T21:48:19.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.012608+0000) 2022-01-31T21:48:19.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[41853]: debug 2022-01-31T21:48:19.124+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.125232+0000) 2022-01-31T21:48:19.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:19 smithi167 conmon[54076]: debug 2022-01-31T21:48:19.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.405209+0000) 2022-01-31T21:48:19.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[46715]: debug 2022-01-31T21:48:19.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.169910+0000) 2022-01-31T21:48:19.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:19 smithi167 conmon[60316]: debug 2022-01-31T21:48:19.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.405196+0000) 2022-01-31T21:48:19.833 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[35325]: debug 2022-01-31T21:48:19.598+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 158900 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:19.833 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[41853]: debug 2022-01-31T21:48:19.575+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.576980+0000) 2022-01-31T21:48:19.834 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[46715]: debug 2022-01-31T21:48:19.574+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.575765+0000) 2022-01-31T21:48:19.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:19 smithi171 conmon[51620]: debug 2022-01-31T21:48:19.574+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.576321+0000) 2022-01-31T21:48:19.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:19 smithi167 conmon[49112]: debug 2022-01-31T21:48:19.574+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.574785+0000) 2022-01-31T21:48:19.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:19 smithi167 conmon[49112]: debug 2022-01-31T21:48:19.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.742414+0000) 2022-01-31T21:48:19.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:19 smithi167 conmon[54076]: debug 2022-01-31T21:48:19.575+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.576161+0000) 2022-01-31T21:48:19.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:19 smithi167 conmon[60316]: debug 2022-01-31T21:48:19.574+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:19.574942+0000) 2022-01-31T21:48:20.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:20 smithi171 conmon[51620]: debug 2022-01-31T21:48:20.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:20.012806+0000) 2022-01-31T21:48:20.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:20 smithi171 conmon[41853]: debug 2022-01-31T21:48:20.124+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:20.125472+0000) 2022-01-31T21:48:20.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:20 smithi171 conmon[46715]: debug 2022-01-31T21:48:20.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:20.170089+0000) 2022-01-31T21:48:20.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:20 smithi167 conmon[60316]: debug 2022-01-31T21:48:20.404+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:20.405423+0000) 2022-01-31T21:48:20.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:20 smithi167 conmon[60316]: 2022-01-31T21:48:20.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:20 smithi167 conmon[54076]: debug 2022-01-31T21:48:20.404+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:20.405417+0000) 2022-01-31T21:48:20.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:20 smithi167 conmon[49112]: debug 2022-01-31T21:48:20.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:20.742591+0000) 2022-01-31T21:48:21.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:21 smithi171 conmon[51620]: debug 2022-01-31T21:48:21.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:21.012950+0000) 2022-01-31T21:48:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:21 smithi171 conmon[41853]: debug 2022-01-31T21:48:21.123+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:21.125589+0000) 2022-01-31T21:48:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:21 smithi171 conmon[46715]: debug 2022-01-31T21:48:21.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:21.170246+0000) 2022-01-31T21:48:21.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:21 smithi167 conmon[60316]: debug 2022-01-31T21:48:21.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:21.405595+0000) 2022-01-31T21:48:21.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:21 smithi167 conmon[54076]: debug 2022-01-31T21:48:21.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:21.405588+0000) 2022-01-31T21:48:21.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:21 smithi167 conmon[49112]: debug 2022-01-31T21:48:21.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:21.742738+0000) 2022-01-31T21:48:22.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:22 smithi171 conmon[51620]: debug 2022-01-31T21:48:22.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:22.013145+0000) 2022-01-31T21:48:22.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:22 smithi171 conmon[46715]: debug 2022-01-31T21:48:22.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:22.170424+0000) 2022-01-31T21:48:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:22 smithi171 conmon[41853]: debug 2022-01-31T21:48:22.124+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:22.125784+0000) 2022-01-31T21:48:22.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:22 smithi167 conmon[54076]: debug 2022-01-31T21:48:22.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:22.405825+0000) 2022-01-31T21:48:22.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:22 smithi167 conmon[60316]: debug 2022-01-31T21:48:22.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:22.405825+0000) 2022-01-31T21:48:22.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:22 smithi167 conmon[49112]: debug 2022-01-31T21:48:22.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:22.742929+0000) 2022-01-31T21:48:23.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:23 smithi171 conmon[51620]: debug 2022-01-31T21:48:23.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:23.013270+0000) 2022-01-31T21:48:23.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:23 smithi171 conmon[41853]: debug 2022-01-31T21:48:23.124+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:23.125938+0000) 2022-01-31T21:48:23.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:23 smithi171 conmon[46715]: debug 2022-01-31T21:48:23.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:23.170647+0000) 2022-01-31T21:48:23.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:23 smithi167 conmon[54076]: debug 2022-01-31T21:48:23.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:23.406007+0000) 2022-01-31T21:48:23.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:23 smithi167 conmon[60316]: debug 2022-01-31T21:48:23.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:23.406015+0000) 2022-01-31T21:48:23.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:23 smithi167 conmon[49112]: debug 2022-01-31T21:48:23.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:23.743090+0000) 2022-01-31T21:48:24.011 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:23 smithi171 conmon[35325]: debug 2022-01-31T21:48:23.742+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:48:24.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[51620]: debug 2022-01-31T21:48:24.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.013421+0000) 2022-01-31T21:48:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[41853]: debug 2022-01-31T21:48:24.125+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.126147+0000) 2022-01-31T21:48:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[46715]: debug 2022-01-31T21:48:24.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.170847+0000) 2022-01-31T21:48:24.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:24 smithi167 conmon[54076]: debug 2022-01-31T21:48:24.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.406207+0000) 2022-01-31T21:48:24.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:24 smithi167 conmon[60316]: debug 2022-01-31T21:48:24.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.406215+0000) 2022-01-31T21:48:24.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[35325]: debug 2022-01-31T21:48:24.617+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159014 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[41853]: debug 2022-01-31T21:48:24.601+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.603158+0000) 2022-01-31T21:48:24.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[46715]: debug 2022-01-31T21:48:24.600+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.602386+0000) 2022-01-31T21:48:24.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:24 smithi171 conmon[51620]: debug 2022-01-31T21:48:24.601+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.602777+0000) 2022-01-31T21:48:24.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:24 smithi167 conmon[54076]: debug 2022-01-31T21:48:24.601+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.602297+0000) 2022-01-31T21:48:24.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:24 smithi167 conmon[60316]: debug 2022-01-31T21:48:24.600+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.601452+0000) 2022-01-31T21:48:24.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:24 smithi167 conmon[49112]: debug 2022-01-31T21:48:24.601+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.602054+0000) 2022-01-31T21:48:24.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:24 smithi167 conmon[49112]: debug 2022-01-31T21:48:24.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:24.743268+0000) 2022-01-31T21:48:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:25 smithi171 conmon[41853]: debug 2022-01-31T21:48:25.125+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:25.126324+0000) 2022-01-31T21:48:25.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:25 smithi171 conmon[51620]: debug 2022-01-31T21:48:25.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:25.013600+0000) 2022-01-31T21:48:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:25 smithi171 conmon[46715]: debug 2022-01-31T21:48:25.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:25.171037+0000) 2022-01-31T21:48:25.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:25 smithi167 conmon[54076]: debug 2022-01-31T21:48:25.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:25.406409+0000) 2022-01-31T21:48:25.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:25 smithi167 conmon[60316]: debug 2022-01-31T21:48:25.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:25.406409+0000) 2022-01-31T21:48:25.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:25 smithi167 conmon[49112]: debug 2022-01-31T21:48:25.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:25.743431+0000) 2022-01-31T21:48:26.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:26 smithi171 conmon[51620]: debug 2022-01-31T21:48:26.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:26.013741+0000) 2022-01-31T21:48:26.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:26 smithi171 conmon[41853]: debug 2022-01-31T21:48:26.125+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:26.126502+0000) 2022-01-31T21:48:26.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:26 smithi171 conmon[46715]: debug 2022-01-31T21:48:26.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:26.171190+0000) 2022-01-31T21:48:26.639 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:26 smithi167 conmon[54076]: debug 2022-01-31T21:48:26.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:26.406560+0000) 2022-01-31T21:48:26.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:26 smithi167 conmon[60316]: debug 2022-01-31T21:48:26.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:26.406534+0000) 2022-01-31T21:48:26.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:26 smithi167 conmon[49112]: debug 2022-01-31T21:48:26.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:26.743656+0000) 2022-01-31T21:48:27.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:27 smithi171 conmon[51620]: debug 2022-01-31T21:48:27.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:27.013924+0000) 2022-01-31T21:48:27.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:27 smithi171 conmon[41853]: debug 2022-01-31T21:48:27.125+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:27.126608+0000) 2022-01-31T21:48:27.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:27 smithi171 conmon[46715]: debug 2022-01-31T21:48:27.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:27.171415+0000) 2022-01-31T21:48:27.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:27 smithi167 conmon[54076]: debug 2022-01-31T21:48:27.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:27.406754+0000) 2022-01-31T21:48:27.640 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:27 smithi167 conmon[60316]: debug 2022-01-31T21:48:27.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:27.406735+0000) 2022-01-31T21:48:27.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:27 smithi167 conmon[49112]: debug 2022-01-31T21:48:27.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:27.743766+0000) 2022-01-31T21:48:28.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:28 smithi171 conmon[51620]: debug 2022-01-31T21:48:28.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:28.014126+0000) 2022-01-31T21:48:28.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:28 smithi171 conmon[41853]: debug 2022-01-31T21:48:28.126+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:28.126831+0000) 2022-01-31T21:48:28.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:28 smithi171 conmon[46715]: debug 2022-01-31T21:48:28.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:28.171624+0000) 2022-01-31T21:48:28.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:28 smithi167 conmon[54076]: debug 2022-01-31T21:48:28.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:28.406932+0000) 2022-01-31T21:48:28.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:28 smithi167 conmon[60316]: debug 2022-01-31T21:48:28.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:28.406905+0000) 2022-01-31T21:48:28.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:28 smithi167 conmon[49112]: debug 2022-01-31T21:48:28.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:28.744075+0000) 2022-01-31T21:48:29.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[51620]: debug 2022-01-31T21:48:29.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.014306+0000) 2022-01-31T21:48:29.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[46715]: debug 2022-01-31T21:48:29.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.171823+0000) 2022-01-31T21:48:29.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[41853]: debug 2022-01-31T21:48:29.126+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.127137+0000) 2022-01-31T21:48:29.619 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:29 smithi167 conmon[54076]: debug 2022-01-31T21:48:29.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.407041+0000) 2022-01-31T21:48:29.620 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:29 smithi167 conmon[60316]: debug 2022-01-31T21:48:29.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.407037+0000) 2022-01-31T21:48:29.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:29 smithi167 conmon[54076]: debug 2022-01-31T21:48:29.620+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.622260+0000) 2022-01-31T21:48:29.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:29 smithi167 conmon[60316]: debug 2022-01-31T21:48:29.620+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.622009+0000) 2022-01-31T21:48:29.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:29 smithi167 conmon[49112]: debug 2022-01-31T21:48:29.619+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.621207+0000) 2022-01-31T21:48:29.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:29 smithi167 conmon[49112]: debug 2022-01-31T21:48:29.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.744279+0000) 2022-01-31T21:48:30.012 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[35325]: debug 2022-01-31T21:48:29.637+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:30.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[41853]: debug 2022-01-31T21:48:29.620+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.621708+0000) 2022-01-31T21:48:30.013 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[46715]: debug 2022-01-31T21:48:29.620+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.621352+0000) 2022-01-31T21:48:30.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:29 smithi171 conmon[51620]: debug 2022-01-31T21:48:29.621+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:29.622341+0000) 2022-01-31T21:48:30.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:30 smithi171 conmon[41853]: debug 2022-01-31T21:48:30.126+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:30.127272+0000) 2022-01-31T21:48:30.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:30 smithi171 conmon[46715]: debug 2022-01-31T21:48:30.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:30.172020+0000) 2022-01-31T21:48:30.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:30 smithi171 conmon[51620]: debug 2022-01-31T21:48:30.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:30.014507+0000) 2022-01-31T21:48:30.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:30 smithi167 conmon[54076]: debug 2022-01-31T21:48:30.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:30.407210+0000) 2022-01-31T21:48:30.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:30 smithi167 conmon[60316]: debug 2022-01-31T21:48:30.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:30.407204+0000) 2022-01-31T21:48:30.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:30 smithi167 conmon[49112]: debug 2022-01-31T21:48:30.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:30.744383+0000) 2022-01-31T21:48:31.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:31 smithi171 conmon[51620]: debug 2022-01-31T21:48:31.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:31.014618+0000) 2022-01-31T21:48:31.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:31 smithi171 conmon[41853]: debug 2022-01-31T21:48:31.126+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:31.127430+0000) 2022-01-31T21:48:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:31 smithi171 conmon[46715]: debug 2022-01-31T21:48:31.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:31.172167+0000) 2022-01-31T21:48:31.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:31 smithi167 conmon[54076]: debug 2022-01-31T21:48:31.405+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:31.407396+0000) 2022-01-31T21:48:31.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:31 smithi167 conmon[60316]: debug 2022-01-31T21:48:31.405+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:31.407395+0000) 2022-01-31T21:48:31.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:31 smithi167 conmon[49112]: debug 2022-01-31T21:48:31.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:31.744600+0000) 2022-01-31T21:48:32.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:32 smithi171 conmon[51620]: debug 2022-01-31T21:48:32.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:32.014736+0000) 2022-01-31T21:48:32.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:32 smithi171 conmon[41853]: debug 2022-01-31T21:48:32.126+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:32.127613+0000) 2022-01-31T21:48:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:32 smithi171 conmon[46715]: debug 2022-01-31T21:48:32.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:32.172324+0000) 2022-01-31T21:48:32.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:32 smithi167 conmon[54076]: debug 2022-01-31T21:48:32.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:32.407556+0000) 2022-01-31T21:48:32.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:32 smithi167 conmon[60316]: debug 2022-01-31T21:48:32.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:32.407555+0000) 2022-01-31T21:48:32.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:32 smithi167 conmon[49112]: debug 2022-01-31T21:48:32.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:32.744795+0000) 2022-01-31T21:48:33.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:33 smithi171 conmon[51620]: debug 2022-01-31T21:48:33.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:33.014955+0000) 2022-01-31T21:48:33.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:33 smithi171 conmon[41853]: debug 2022-01-31T21:48:33.127+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:33.127806+0000) 2022-01-31T21:48:33.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:33 smithi171 conmon[46715]: debug 2022-01-31T21:48:33.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:33.172538+0000) 2022-01-31T21:48:33.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:33 smithi167 conmon[54076]: debug 2022-01-31T21:48:33.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:33.407739+0000) 2022-01-31T21:48:33.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:33 smithi167 conmon[60316]: debug 2022-01-31T21:48:33.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:33.407744+0000) 2022-01-31T21:48:33.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:33 smithi167 conmon[49112]: debug 2022-01-31T21:48:33.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:33.744953+0000) 2022-01-31T21:48:34.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[51620]: debug 2022-01-31T21:48:34.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.015116+0000) 2022-01-31T21:48:34.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[41853]: debug 2022-01-31T21:48:34.127+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.127996+0000) 2022-01-31T21:48:34.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[46715]: debug 2022-01-31T21:48:34.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.172732+0000) 2022-01-31T21:48:34.638 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:34 smithi167 conmon[54076]: debug 2022-01-31T21:48:34.406+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.407907+0000) 2022-01-31T21:48:34.639 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:34 smithi167 conmon[60316]: debug 2022-01-31T21:48:34.406+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.407939+0000) 2022-01-31T21:48:34.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:34 smithi167 conmon[54076]: debug 2022-01-31T21:48:34.639+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.640637+0000) 2022-01-31T21:48:34.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:34 smithi167 conmon[49112]: debug 2022-01-31T21:48:34.638+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.640283+0000) 2022-01-31T21:48:34.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:34 smithi167 conmon[49112]: debug 2022-01-31T21:48:34.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.745181+0000) 2022-01-31T21:48:34.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:34 smithi167 conmon[60316]: debug 2022-01-31T21:48:34.639+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.641005+0000) 2022-01-31T21:48:35.013 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[35325]: debug 2022-01-31T21:48:34.656+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159236 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:35.013 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[41853]: debug 2022-01-31T21:48:34.640+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.640882+0000) 2022-01-31T21:48:35.014 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[46715]: debug 2022-01-31T21:48:34.640+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.641003+0000) 2022-01-31T21:48:35.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:34 smithi171 conmon[51620]: debug 2022-01-31T21:48:34.641+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:34.642095+0000) 2022-01-31T21:48:35.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:35 smithi171 conmon[51620]: debug 2022-01-31T21:48:35.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:35.015265+0000) 2022-01-31T21:48:35.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:35 smithi171 conmon[41853]: debug 2022-01-31T21:48:35.127+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:35.128183+0000) 2022-01-31T21:48:35.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:35 smithi171 conmon[46715]: debug 2022-01-31T21:48:35.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:35.172934+0000) 2022-01-31T21:48:35.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:35 smithi167 conmon[54076]: debug 2022-01-31T21:48:35.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:35.408102+0000) 2022-01-31T21:48:35.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:35 smithi167 conmon[60316]: debug 2022-01-31T21:48:35.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:35.408110+0000) 2022-01-31T21:48:35.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:35 smithi167 conmon[49112]: debug 2022-01-31T21:48:35.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:35.745387+0000) 2022-01-31T21:48:36.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:36 smithi171 conmon[51620]: debug 2022-01-31T21:48:36.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:36.015444+0000) 2022-01-31T21:48:36.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:36 smithi171 conmon[41853]: debug 2022-01-31T21:48:36.127+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:36.128304+0000) 2022-01-31T21:48:36.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:36 smithi171 conmon[46715]: debug 2022-01-31T21:48:36.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:36.173092+0000) 2022-01-31T21:48:36.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:36 smithi167 conmon[54076]: debug 2022-01-31T21:48:36.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:36.408259+0000) 2022-01-31T21:48:36.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:36 smithi167 conmon[60316]: debug 2022-01-31T21:48:36.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:36.408279+0000) 2022-01-31T21:48:36.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:36 smithi167 conmon[49112]: debug 2022-01-31T21:48:36.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:36.745575+0000) 2022-01-31T21:48:37.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:37 smithi171 conmon[51620]: debug 2022-01-31T21:48:37.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:37.015644+0000) 2022-01-31T21:48:37.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:37 smithi171 conmon[41853]: debug 2022-01-31T21:48:37.127+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:37.128400+0000) 2022-01-31T21:48:37.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:37 smithi171 conmon[46715]: debug 2022-01-31T21:48:37.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:37.173242+0000) 2022-01-31T21:48:37.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:37 smithi167 conmon[54076]: debug 2022-01-31T21:48:37.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:37.408520+0000) 2022-01-31T21:48:37.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:37 smithi167 conmon[60316]: debug 2022-01-31T21:48:37.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:37.408523+0000) 2022-01-31T21:48:37.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:37 smithi167 conmon[49112]: debug 2022-01-31T21:48:37.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:37.745769+0000) 2022-01-31T21:48:38.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:38 smithi171 conmon[51620]: debug 2022-01-31T21:48:38.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:38.015817+0000) 2022-01-31T21:48:38.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:38 smithi171 conmon[41853]: debug 2022-01-31T21:48:38.127+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:38.128574+0000) 2022-01-31T21:48:38.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:38 smithi171 conmon[46715]: debug 2022-01-31T21:48:38.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:38.173399+0000) 2022-01-31T21:48:38.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:38 smithi167 conmon[54076]: debug 2022-01-31T21:48:38.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:38.408729+0000) 2022-01-31T21:48:38.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:38 smithi167 conmon[60316]: debug 2022-01-31T21:48:38.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:38.408732+0000) 2022-01-31T21:48:38.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:38 smithi167 conmon[49112]: debug 2022-01-31T21:48:38.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:38.745939+0000) 2022-01-31T21:48:39.013 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:38 smithi171 conmon[35325]: debug 2022-01-31T21:48:38.743+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:48:39.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[51620]: debug 2022-01-31T21:48:39.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.016018+0000) 2022-01-31T21:48:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[41853]: debug 2022-01-31T21:48:39.128+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.128791+0000) 2022-01-31T21:48:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[46715]: debug 2022-01-31T21:48:39.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.173585+0000) 2022-01-31T21:48:39.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:39 smithi167 conmon[54076]: debug 2022-01-31T21:48:39.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.408937+0000) 2022-01-31T21:48:39.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:39 smithi167 conmon[60316]: debug 2022-01-31T21:48:39.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.408929+0000) 2022-01-31T21:48:39.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:39 smithi167 conmon[54076]: debug 2022-01-31T21:48:39.658+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.660368+0000) 2022-01-31T21:48:39.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:39 smithi167 conmon[60316]: debug 2022-01-31T21:48:39.659+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.660633+0000) 2022-01-31T21:48:39.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:39 smithi167 conmon[49112]: debug 2022-01-31T21:48:39.657+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.659428+0000) 2022-01-31T21:48:39.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:39 smithi167 conmon[49112]: debug 2022-01-31T21:48:39.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.746097+0000) 2022-01-31T21:48:40.014 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[35325]: debug 2022-01-31T21:48:39.677+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159344 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:40.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[41853]: debug 2022-01-31T21:48:39.659+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.660142+0000) 2022-01-31T21:48:40.015 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[46715]: debug 2022-01-31T21:48:39.659+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.660084+0000) 2022-01-31T21:48:40.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:39 smithi171 conmon[51620]: debug 2022-01-31T21:48:39.659+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:39.660554+0000) 2022-01-31T21:48:40.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:40 smithi171 conmon[51620]: debug 2022-01-31T21:48:40.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:40.016229+0000) 2022-01-31T21:48:40.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:40 smithi171 conmon[41853]: debug 2022-01-31T21:48:40.128+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:40.128938+0000) 2022-01-31T21:48:40.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:40 smithi171 conmon[46715]: debug 2022-01-31T21:48:40.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:40.173770+0000) 2022-01-31T21:48:40.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:40 smithi167 conmon[54076]: debug 2022-01-31T21:48:40.407+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:40.409151+0000) 2022-01-31T21:48:40.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:40 smithi167 conmon[60316]: debug 2022-01-31T21:48:40.407+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:40.409138+0000) 2022-01-31T21:48:40.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:40 smithi167 conmon[49112]: debug 2022-01-31T21:48:40.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:40.746245+0000) 2022-01-31T21:48:41.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:41 smithi171 conmon[51620]: debug 2022-01-31T21:48:41.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:41.016384+0000) 2022-01-31T21:48:41.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:41 smithi171 conmon[41853]: debug 2022-01-31T21:48:41.128+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:41.129078+0000) 2022-01-31T21:48:41.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:41 smithi171 conmon[46715]: debug 2022-01-31T21:48:41.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:41.173914+0000) 2022-01-31T21:48:41.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:41 smithi167 conmon[54076]: debug 2022-01-31T21:48:41.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:41.409313+0000) 2022-01-31T21:48:41.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:41 smithi167 conmon[60316]: debug 2022-01-31T21:48:41.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:41.409289+0000) 2022-01-31T21:48:41.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:41 smithi167 conmon[49112]: debug 2022-01-31T21:48:41.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:41.746419+0000) 2022-01-31T21:48:42.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:42 smithi171 conmon[51620]: debug 2022-01-31T21:48:42.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:42.016565+0000) 2022-01-31T21:48:42.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:42 smithi171 conmon[41853]: debug 2022-01-31T21:48:42.128+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:42.129284+0000) 2022-01-31T21:48:42.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:42 smithi171 conmon[46715]: debug 2022-01-31T21:48:42.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:42.174123+0000) 2022-01-31T21:48:42.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:42 smithi167 conmon[54076]: debug 2022-01-31T21:48:42.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:42.409536+0000) 2022-01-31T21:48:42.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:42 smithi167 conmon[60316]: debug 2022-01-31T21:48:42.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:42.409525+0000) 2022-01-31T21:48:42.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:42 smithi167 conmon[49112]: debug 2022-01-31T21:48:42.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:42.746602+0000) 2022-01-31T21:48:43.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:43 smithi171 conmon[51620]: debug 2022-01-31T21:48:43.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:43.016745+0000) 2022-01-31T21:48:43.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:43 smithi171 conmon[41853]: debug 2022-01-31T21:48:43.128+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:43.129459+0000) 2022-01-31T21:48:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:43 smithi171 conmon[46715]: debug 2022-01-31T21:48:43.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:43.174338+0000) 2022-01-31T21:48:43.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:43 smithi167 conmon[54076]: debug 2022-01-31T21:48:43.408+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:43.409726+0000) 2022-01-31T21:48:43.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:43 smithi167 conmon[60316]: debug 2022-01-31T21:48:43.408+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:43.409725+0000) 2022-01-31T21:48:43.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:43 smithi167 conmon[49112]: debug 2022-01-31T21:48:43.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:43.746802+0000) 2022-01-31T21:48:44.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[51620]: debug 2022-01-31T21:48:44.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.016921+0000) 2022-01-31T21:48:44.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[41853]: debug 2022-01-31T21:48:44.128+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.129631+0000) 2022-01-31T21:48:44.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[46715]: debug 2022-01-31T21:48:44.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.174534+0000) 2022-01-31T21:48:44.642 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:44 smithi167 conmon[54076]: debug 2022-01-31T21:48:44.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.409900+0000) 2022-01-31T21:48:44.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:44 smithi167 conmon[60316]: debug 2022-01-31T21:48:44.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.409906+0000) 2022-01-31T21:48:44.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:44 smithi167 conmon[60316]: debug 2022-01-31T21:48:44.678+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.680409+0000) 2022-01-31T21:48:44.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:44 smithi167 conmon[49112]: debug 2022-01-31T21:48:44.678+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.680268+0000) 2022-01-31T21:48:44.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:44 smithi167 conmon[49112]: debug 2022-01-31T21:48:44.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.746985+0000) 2022-01-31T21:48:44.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:44 smithi167 conmon[54076]: debug 2022-01-31T21:48:44.679+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.680864+0000) 2022-01-31T21:48:45.014 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[46715]: debug 2022-01-31T21:48:44.679+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.680539+0000) 2022-01-31T21:48:45.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[51620]: debug 2022-01-31T21:48:44.679+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.680356+0000) 2022-01-31T21:48:45.016 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[35325]: debug 2022-01-31T21:48:44.696+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159457 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:45.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:44 smithi171 conmon[41853]: debug 2022-01-31T21:48:44.680+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:44.681207+0000) 2022-01-31T21:48:45.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:45 smithi171 conmon[41853]: debug 2022-01-31T21:48:45.129+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:45.129766+0000) 2022-01-31T21:48:45.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:45 smithi171 conmon[46715]: debug 2022-01-31T21:48:45.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:45.174732+0000) 2022-01-31T21:48:45.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:45 smithi171 conmon[51620]: debug 2022-01-31T21:48:45.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:45.017116+0000) 2022-01-31T21:48:45.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:45 smithi167 conmon[54076]: debug 2022-01-31T21:48:45.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:45.410083+0000) 2022-01-31T21:48:45.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:45 smithi167 conmon[60316]: debug 2022-01-31T21:48:45.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:45.410082+0000) 2022-01-31T21:48:45.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:45 smithi167 conmon[49112]: debug 2022-01-31T21:48:45.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:45.747137+0000) 2022-01-31T21:48:46.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:46 smithi171 conmon[51620]: debug 2022-01-31T21:48:46.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:46.017221+0000) 2022-01-31T21:48:46.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:46 smithi171 conmon[41853]: debug 2022-01-31T21:48:46.129+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:46.129914+0000) 2022-01-31T21:48:46.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:46 smithi171 conmon[46715]: debug 2022-01-31T21:48:46.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:46.174914+0000) 2022-01-31T21:48:46.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:46 smithi167 conmon[54076]: debug 2022-01-31T21:48:46.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:46.410283+0000) 2022-01-31T21:48:46.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:46 smithi167 conmon[60316]: debug 2022-01-31T21:48:46.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:46.410283+0000) 2022-01-31T21:48:46.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:46 smithi167 conmon[49112]: debug 2022-01-31T21:48:46.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:46.747323+0000) 2022-01-31T21:48:47.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:47 smithi171 conmon[51620]: debug 2022-01-31T21:48:47.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:47.017422+0000) 2022-01-31T21:48:47.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:47 smithi171 conmon[41853]: debug 2022-01-31T21:48:47.129+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:47.130015+0000) 2022-01-31T21:48:47.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:47 smithi171 conmon[46715]: debug 2022-01-31T21:48:47.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:47.175100+0000) 2022-01-31T21:48:47.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:47 smithi167 conmon[54076]: debug 2022-01-31T21:48:47.409+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:47.410536+0000) 2022-01-31T21:48:47.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:47 smithi167 conmon[60316]: debug 2022-01-31T21:48:47.409+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:47.410542+0000) 2022-01-31T21:48:47.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:47 smithi167 conmon[49112]: debug 2022-01-31T21:48:47.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:47.747535+0000) 2022-01-31T21:48:48.085 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:48 smithi171 conmon[51620]: debug 2022-01-31T21:48:48.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:48.017589+0000) 2022-01-31T21:48:48.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:48 smithi171 conmon[41853]: debug 2022-01-31T21:48:48.129+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:48.130203+0000) 2022-01-31T21:48:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:48 smithi171 conmon[46715]: debug 2022-01-31T21:48:48.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:48.175278+0000) 2022-01-31T21:48:48.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:48 smithi167 conmon[54076]: debug 2022-01-31T21:48:48.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:48.410726+0000) 2022-01-31T21:48:48.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:48 smithi167 conmon[60316]: debug 2022-01-31T21:48:48.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:48.410732+0000) 2022-01-31T21:48:48.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:48 smithi167 conmon[49112]: debug 2022-01-31T21:48:48.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:48.747695+0000) 2022-01-31T21:48:49.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[51620]: debug 2022-01-31T21:48:49.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.017788+0000) 2022-01-31T21:48:49.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[41853]: debug 2022-01-31T21:48:49.130+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.130390+0000) 2022-01-31T21:48:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[46715]: debug 2022-01-31T21:48:49.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.175503+0000) 2022-01-31T21:48:49.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:49 smithi167 conmon[54076]: debug 2022-01-31T21:48:49.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.410938+0000) 2022-01-31T21:48:49.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:49 smithi167 conmon[60316]: debug 2022-01-31T21:48:49.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.410942+0000) 2022-01-31T21:48:49.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:49 smithi167 conmon[54076]: debug 2022-01-31T21:48:49.697+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.699304+0000) 2022-01-31T21:48:49.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:49 smithi167 conmon[60316]: debug 2022-01-31T21:48:49.698+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.699899+0000) 2022-01-31T21:48:49.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:49 smithi167 conmon[49112]: debug 2022-01-31T21:48:49.697+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.698908+0000) 2022-01-31T21:48:49.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:49 smithi167 conmon[49112]: debug 2022-01-31T21:48:49.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.747838+0000) 2022-01-31T21:48:50.015 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[35325]: debug 2022-01-31T21:48:49.715+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159565 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:50.016 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[41853]: debug 2022-01-31T21:48:49.700+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.701621+0000) 2022-01-31T21:48:50.017 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[46715]: debug 2022-01-31T21:48:49.699+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.699795+0000) 2022-01-31T21:48:50.017 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:49 smithi171 conmon[51620]: debug 2022-01-31T21:48:49.699+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:49.700575+0000) 2022-01-31T21:48:50.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:50 smithi171 conmon[46715]: debug 2022-01-31T21:48:50.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:50.175667+0000) 2022-01-31T21:48:50.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:50 smithi171 conmon[51620]: debug 2022-01-31T21:48:50.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:50.017948+0000) 2022-01-31T21:48:50.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:50 smithi171 conmon[41853]: debug 2022-01-31T21:48:50.130+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:50.130529+0000) 2022-01-31T21:48:50.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:50 smithi167 conmon[54076]: debug 2022-01-31T21:48:50.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:50.411122+0000) 2022-01-31T21:48:50.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:50 smithi167 conmon[60316]: debug 2022-01-31T21:48:50.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:50.411104+0000) 2022-01-31T21:48:50.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:50 smithi167 conmon[49112]: debug 2022-01-31T21:48:50.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:50.747986+0000) 2022-01-31T21:48:51.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:51 smithi171 conmon[51620]: debug 2022-01-31T21:48:51.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:51.018082+0000) 2022-01-31T21:48:51.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:51 smithi171 conmon[41853]: debug 2022-01-31T21:48:51.129+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:51.130689+0000) 2022-01-31T21:48:51.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:51 smithi171 conmon[46715]: debug 2022-01-31T21:48:51.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:51.175809+0000) 2022-01-31T21:48:51.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:51 smithi167 conmon[54076]: debug 2022-01-31T21:48:51.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:51.411240+0000) 2022-01-31T21:48:51.644 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:51 smithi167 conmon[60316]: debug 2022-01-31T21:48:51.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:51.411260+0000) 2022-01-31T21:48:51.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:51 smithi167 conmon[49112]: debug 2022-01-31T21:48:51.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:51.748094+0000) 2022-01-31T21:48:52.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:52 smithi171 conmon[51620]: debug 2022-01-31T21:48:52.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:52.018213+0000) 2022-01-31T21:48:52.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:52 smithi171 conmon[41853]: debug 2022-01-31T21:48:52.130+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:52.130850+0000) 2022-01-31T21:48:52.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:52 smithi171 conmon[46715]: debug 2022-01-31T21:48:52.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:52.176024+0000) 2022-01-31T21:48:52.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:52 smithi167 conmon[54076]: debug 2022-01-31T21:48:52.410+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:52.411382+0000) 2022-01-31T21:48:52.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:52 smithi167 conmon[60316]: debug 2022-01-31T21:48:52.410+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:52.411417+0000) 2022-01-31T21:48:52.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:52 smithi167 conmon[49112]: debug 2022-01-31T21:48:52.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:52.748305+0000) 2022-01-31T21:48:53.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:53 smithi171 conmon[51620]: debug 2022-01-31T21:48:53.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:53.018409+0000) 2022-01-31T21:48:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:53 smithi171 conmon[41853]: debug 2022-01-31T21:48:53.130+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:53.131069+0000) 2022-01-31T21:48:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:53 smithi171 conmon[46715]: debug 2022-01-31T21:48:53.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:53.176186+0000) 2022-01-31T21:48:53.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:53 smithi167 conmon[54076]: debug 2022-01-31T21:48:53.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:53.411558+0000) 2022-01-31T21:48:53.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:53 smithi167 conmon[60316]: debug 2022-01-31T21:48:53.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:53.411562+0000) 2022-01-31T21:48:53.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:53 smithi167 conmon[49112]: debug 2022-01-31T21:48:53.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:53.748491+0000) 2022-01-31T21:48:54.016 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:53 smithi171 conmon[35325]: debug 2022-01-31T21:48:53.744+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:48:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[41853]: debug 2022-01-31T21:48:54.131+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.131302+0000) 2022-01-31T21:48:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[46715]: debug 2022-01-31T21:48:54.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.176347+0000) 2022-01-31T21:48:54.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[51620]: debug 2022-01-31T21:48:54.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.018609+0000) 2022-01-31T21:48:54.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:54 smithi167 conmon[54076]: debug 2022-01-31T21:48:54.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.411736+0000) 2022-01-31T21:48:54.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:54 smithi167 conmon[60316]: debug 2022-01-31T21:48:54.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.411737+0000) 2022-01-31T21:48:54.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:54 smithi167 conmon[49112]: debug 2022-01-31T21:48:54.718+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.719259+0000) 2022-01-31T21:48:54.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:54 smithi167 conmon[49112]: debug 2022-01-31T21:48:54.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.748645+0000) 2022-01-31T21:48:54.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:54 smithi167 conmon[54076]: debug 2022-01-31T21:48:54.718+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.718557+0000) 2022-01-31T21:48:54.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:54 smithi167 conmon[60316]: debug 2022-01-31T21:48:54.718+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.719365+0000) 2022-01-31T21:48:55.016 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[35325]: debug 2022-01-31T21:48:54.735+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159678 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:48:55.017 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[41853]: debug 2022-01-31T21:48:54.719+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.720347+0000) 2022-01-31T21:48:55.018 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[46715]: debug 2022-01-31T21:48:54.718+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.718825+0000) 2022-01-31T21:48:55.018 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:54 smithi171 conmon[51620]: debug 2022-01-31T21:48:54.718+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:54.719238+0000) 2022-01-31T21:48:55.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:55 smithi171 conmon[41853]: debug 2022-01-31T21:48:55.130+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:55.131498+0000) 2022-01-31T21:48:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:55 smithi171 conmon[46715]: debug 2022-01-31T21:48:55.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:55.176504+0000) 2022-01-31T21:48:55.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:55 smithi171 conmon[51620]: debug 2022-01-31T21:48:55.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:55.018783+0000) 2022-01-31T21:48:55.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:55 smithi167 conmon[54076]: debug 2022-01-31T21:48:55.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:55.411954+0000) 2022-01-31T21:48:55.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:55 smithi167 conmon[60316]: debug 2022-01-31T21:48:55.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:55.411944+0000) 2022-01-31T21:48:55.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:55 smithi167 conmon[49112]: debug 2022-01-31T21:48:55.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:55.748735+0000) 2022-01-31T21:48:56.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:56 smithi171 conmon[51620]: debug 2022-01-31T21:48:56.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:56.018966+0000) 2022-01-31T21:48:56.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:56 smithi171 conmon[41853]: debug 2022-01-31T21:48:56.131+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:56.131634+0000) 2022-01-31T21:48:56.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:56 smithi171 conmon[46715]: debug 2022-01-31T21:48:56.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:56.176657+0000) 2022-01-31T21:48:56.644 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:56 smithi167 conmon[54076]: debug 2022-01-31T21:48:56.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:56.412127+0000) 2022-01-31T21:48:56.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:56 smithi167 conmon[60316]: debug 2022-01-31T21:48:56.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:56.412129+0000) 2022-01-31T21:48:56.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:56 smithi167 conmon[49112]: debug 2022-01-31T21:48:56.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:56.748835+0000) 2022-01-31T21:48:57.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:57 smithi171 conmon[51620]: debug 2022-01-31T21:48:57.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:57.019163+0000) 2022-01-31T21:48:57.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:57 smithi171 conmon[41853]: debug 2022-01-31T21:48:57.131+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:57.131841+0000) 2022-01-31T21:48:57.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:57 smithi171 conmon[46715]: debug 2022-01-31T21:48:57.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:57.176860+0000) 2022-01-31T21:48:57.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:57 smithi167 conmon[54076]: debug 2022-01-31T21:48:57.411+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:57.412333+0000) 2022-01-31T21:48:57.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:57 smithi167 conmon[60316]: debug 2022-01-31T21:48:57.411+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:57.412338+0000) 2022-01-31T21:48:57.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:57 smithi167 conmon[49112]: debug 2022-01-31T21:48:57.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:57.748964+0000) 2022-01-31T21:48:58.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:58 smithi171 conmon[51620]: debug 2022-01-31T21:48:58.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:58.019349+0000) 2022-01-31T21:48:58.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:58 smithi171 conmon[41853]: debug 2022-01-31T21:48:58.131+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:58.131996+0000) 2022-01-31T21:48:58.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:58 smithi171 conmon[46715]: debug 2022-01-31T21:48:58.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:58.177016+0000) 2022-01-31T21:48:58.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:58 smithi167 conmon[54076]: debug 2022-01-31T21:48:58.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:58.412533+0000) 2022-01-31T21:48:58.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:58 smithi167 conmon[60316]: debug 2022-01-31T21:48:58.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:58.412536+0000) 2022-01-31T21:48:58.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:58 smithi167 conmon[49112]: debug 2022-01-31T21:48:58.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:58.749131+0000) 2022-01-31T21:48:59.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[51620]: debug 2022-01-31T21:48:59.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.019514+0000) 2022-01-31T21:48:59.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[41853]: debug 2022-01-31T21:48:59.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.132164+0000) 2022-01-31T21:48:59.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[46715]: debug 2022-01-31T21:48:59.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.177200+0000) 2022-01-31T21:48:59.645 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[60316]: debug 2022-01-31T21:48:59.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.412763+0000) 2022-01-31T21:48:59.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[54076]: debug 2022-01-31T21:48:59.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.412763+0000) 2022-01-31T21:48:59.911 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[54076]: debug 2022-01-31T21:48:59.736+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.737244+0000) 2022-01-31T21:48:59.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[60316]: debug 2022-01-31T21:48:59.738+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.738534+0000) 2022-01-31T21:48:59.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[49112]: debug 2022-01-31T21:48:59.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[49112]: 2022-01-31T21:48:59.737+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.737941+0000) 2022-01-31T21:48:59.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:48:59 smithi167 conmon[49112]: debug 2022-01-31T21:48:59.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.749334+0000) 2022-01-31T21:49:00.017 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[35325]: debug 2022-01-31T21:48:59.754+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159786 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:00.018 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[41853]: debug 2022-01-31T21:48:59.737+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.738521+0000) 2022-01-31T21:49:00.019 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[46715]: debug 2022-01-31T21:48:59.738+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.738857+0000) 2022-01-31T21:49:00.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:48:59 smithi171 conmon[51620]: debug 2022-01-31T21:48:59.737+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:48:59.738012+0000) 2022-01-31T21:49:00.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:00 smithi171 conmon[51620]: debug 2022-01-31T21:49:00.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:00.019722+0000) 2022-01-31T21:49:00.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:00 smithi171 conmon[41853]: debug 2022-01-31T21:49:00.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:00.132354+0000) 2022-01-31T21:49:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:00 smithi171 conmon[46715]: debug 2022-01-31T21:49:00.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:00.177358+0000) 2022-01-31T21:49:00.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:00 smithi167 conmon[54076]: debug 2022-01-31T21:49:00.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:00.412965+0000) 2022-01-31T21:49:00.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:00 smithi167 conmon[60316]: debug 2022-01-31T21:49:00.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:00.412965+0000) 2022-01-31T21:49:00.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:00 smithi167 conmon[49112]: debug 2022-01-31T21:49:00.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:00.749467+0000) 2022-01-31T21:49:01.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:01 smithi171 conmon[51620]: debug 2022-01-31T21:49:01.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:01.019902+0000) 2022-01-31T21:49:01.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:01 smithi171 conmon[46715]: debug 2022-01-31T21:49:01.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:01.177544+0000) 2022-01-31T21:49:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:01 smithi171 conmon[41853]: debug 2022-01-31T21:49:01.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:01.132538+0000) 2022-01-31T21:49:01.645 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:01 smithi167 conmon[54076]: debug 2022-01-31T21:49:01.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:01.413153+0000) 2022-01-31T21:49:01.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:01 smithi167 conmon[60316]: debug 2022-01-31T21:49:01.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:01.413151+0000) 2022-01-31T21:49:01.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:01 smithi167 conmon[49112]: debug 2022-01-31T21:49:01.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:01.749654+0000) 2022-01-31T21:49:02.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:02 smithi171 conmon[51620]: debug 2022-01-31T21:49:02.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:02.020042+0000) 2022-01-31T21:49:02.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:02 smithi171 conmon[41853]: debug 2022-01-31T21:49:02.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:02.132739+0000) 2022-01-31T21:49:02.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:02 smithi171 conmon[46715]: debug 2022-01-31T21:49:02.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:02.177670+0000) 2022-01-31T21:49:02.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:02 smithi167 conmon[54076]: debug 2022-01-31T21:49:02.412+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:02.413353+0000) 2022-01-31T21:49:02.646 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:02 smithi167 conmon[60316]: debug 2022-01-31T21:49:02.412+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:02.413360+0000) 2022-01-31T21:49:02.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:02 smithi167 conmon[49112]: debug 2022-01-31T21:49:02.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:02.749841+0000) 2022-01-31T21:49:03.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:03 smithi171 conmon[51620]: debug 2022-01-31T21:49:03.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:03.020222+0000) 2022-01-31T21:49:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:03 smithi171 conmon[41853]: debug 2022-01-31T21:49:03.131+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:03.132953+0000) 2022-01-31T21:49:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:03 smithi171 conmon[46715]: debug 2022-01-31T21:49:03.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:03.177858+0000) 2022-01-31T21:49:03.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:03 smithi167 conmon[54076]: debug 2022-01-31T21:49:03.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:03.413544+0000) 2022-01-31T21:49:03.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:03 smithi167 conmon[60316]: debug 2022-01-31T21:49:03.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:03.413554+0000) 2022-01-31T21:49:03.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:03 smithi167 conmon[49112]: debug 2022-01-31T21:49:03.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:03.750003+0000) 2022-01-31T21:49:04.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[51620]: debug 2022-01-31T21:49:04.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.020403+0000) 2022-01-31T21:49:04.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[46715]: debug 2022-01-31T21:49:04.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.178024+0000) 2022-01-31T21:49:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[41853]: debug 2022-01-31T21:49:04.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.133140+0000) 2022-01-31T21:49:04.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:04 smithi167 conmon[54076]: debug 2022-01-31T21:49:04.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.413735+0000) 2022-01-31T21:49:04.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:04 smithi167 conmon[60316]: debug 2022-01-31T21:49:04.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.413742+0000) 2022-01-31T21:49:04.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:04 smithi167 conmon[49112]: debug 2022-01-31T21:49:04.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.750178+0000) 2022-01-31T21:49:04.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:04 smithi167 conmon[49112]: debug 2022-01-31T21:49:04.756+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.756958+0000) 2022-01-31T21:49:04.913 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:04 smithi167 conmon[54076]: debug 2022-01-31T21:49:04.756+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.756520+0000) 2022-01-31T21:49:04.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:04 smithi167 conmon[60316]: debug 2022-01-31T21:49:04.757+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.757613+0000) 2022-01-31T21:49:05.018 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[35325]: debug 2022-01-31T21:49:04.772+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 159899 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:05.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[41853]: debug 2022-01-31T21:49:04.755+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.757431+0000) 2022-01-31T21:49:05.019 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[46715]: debug 2022-01-31T21:49:04.756+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.758063+0000) 2022-01-31T21:49:05.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:04 smithi171 conmon[51620]: debug 2022-01-31T21:49:04.755+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:04.757211+0000) 2022-01-31T21:49:05.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:05 smithi171 conmon[51620]: debug 2022-01-31T21:49:05.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:05.020588+0000) 2022-01-31T21:49:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:05 smithi171 conmon[46715]: debug 2022-01-31T21:49:05.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:05.178198+0000) 2022-01-31T21:49:05.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:05 smithi171 conmon[41853]: debug 2022-01-31T21:49:05.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:05.133347+0000) 2022-01-31T21:49:05.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:05 smithi167 conmon[54076]: debug 2022-01-31T21:49:05.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:05.413951+0000) 2022-01-31T21:49:05.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:05 smithi167 conmon[60316]: debug 2022-01-31T21:49:05.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:05.413929+0000) 2022-01-31T21:49:05.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:05 smithi167 conmon[49112]: debug 2022-01-31T21:49:05.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:05.750336+0000) 2022-01-31T21:49:06.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:06 smithi171 conmon[51620]: debug 2022-01-31T21:49:06.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:06.020733+0000) 2022-01-31T21:49:06.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:06 smithi171 conmon[46715]: debug 2022-01-31T21:49:06.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:06.178373+0000) 2022-01-31T21:49:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:06 smithi171 conmon[41853]: debug 2022-01-31T21:49:06.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:06.133505+0000) 2022-01-31T21:49:06.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:06 smithi167 conmon[54076]: debug 2022-01-31T21:49:06.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:06.414092+0000) 2022-01-31T21:49:06.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:06 smithi167 conmon[60316]: debug 2022-01-31T21:49:06.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:06.414094+0000) 2022-01-31T21:49:06.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:06 smithi167 conmon[49112]: debug 2022-01-31T21:49:06.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:06.750494+0000) 2022-01-31T21:49:07.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:07 smithi171 conmon[51620]: debug 2022-01-31T21:49:07.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:07.020919+0000) 2022-01-31T21:49:07.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:07 smithi171 conmon[46715]: debug 2022-01-31T21:49:07.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:07.178538+0000) 2022-01-31T21:49:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:07 smithi171 conmon[41853]: debug 2022-01-31T21:49:07.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:07.133690+0000) 2022-01-31T21:49:07.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:07 smithi167 conmon[54076]: debug 2022-01-31T21:49:07.413+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:07.414301+0000) 2022-01-31T21:49:07.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:07 smithi167 conmon[60316]: debug 2022-01-31T21:49:07.413+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:07.414302+0000) 2022-01-31T21:49:07.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:07 smithi167 conmon[49112]: debug 2022-01-31T21:49:07.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:07.750708+0000) 2022-01-31T21:49:08.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:08 smithi171 conmon[51620]: debug 2022-01-31T21:49:08.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:08.021139+0000) 2022-01-31T21:49:08.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:08 smithi171 conmon[46715]: debug 2022-01-31T21:49:08.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:08.178723+0000) 2022-01-31T21:49:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:08 smithi171 conmon[41853]: debug 2022-01-31T21:49:08.132+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:08.133827+0000) 2022-01-31T21:49:08.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:08 smithi167 conmon[54076]: debug 2022-01-31T21:49:08.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:08.414487+0000) 2022-01-31T21:49:08.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:08 smithi167 conmon[60316]: debug 2022-01-31T21:49:08.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:08.414486+0000) 2022-01-31T21:49:08.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:08 smithi167 conmon[49112]: debug 2022-01-31T21:49:08.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:08.750899+0000) 2022-01-31T21:49:09.019 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:08 smithi171 conmon[35325]: debug 2022-01-31T21:49:08.744+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:49:09.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[51620]: debug 2022-01-31T21:49:09.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.021350+0000) 2022-01-31T21:49:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[41853]: debug 2022-01-31T21:49:09.133+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.133988+0000) 2022-01-31T21:49:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[46715]: debug 2022-01-31T21:49:09.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.178922+0000) 2022-01-31T21:49:09.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:09 smithi167 conmon[54076]: debug 2022-01-31T21:49:09.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.414654+0000) 2022-01-31T21:49:09.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:09 smithi167 conmon[60316]: debug 2022-01-31T21:49:09.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.414784+0000) 2022-01-31T21:49:09.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:09 smithi167 conmon[49112]: debug 2022-01-31T21:49:09.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.751074+0000) 2022-01-31T21:49:09.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:09 smithi167 conmon[49112]: debug 2022-01-31T21:49:09.775+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.775771+0000) 2022-01-31T21:49:09.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:09 smithi167 conmon[54076]: debug 2022-01-31T21:49:09.776+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.776782+0000) 2022-01-31T21:49:09.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:09 smithi167 conmon[60316]: debug 2022-01-31T21:49:09.776+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.777005+0000) 2022-01-31T21:49:10.088 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[35325]: debug 2022-01-31T21:49:09.792+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160007 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:10.089 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[41853]: debug 2022-01-31T21:49:09.774+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.776568+0000) 2022-01-31T21:49:10.090 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[46715]: debug 2022-01-31T21:49:09.775+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.777716+0000) 2022-01-31T21:49:10.090 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:09 smithi171 conmon[51620]: debug 2022-01-31T21:49:09.775+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:09.777402+0000) 2022-01-31T21:49:10.091 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:10 smithi171 conmon[51620]: debug 2022-01-31T21:49:10.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:10.021538+0000) 2022-01-31T21:49:10.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:10 smithi171 conmon[41853]: debug 2022-01-31T21:49:10.133+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:10.134200+0000) 2022-01-31T21:49:10.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:10 smithi171 conmon[46715]: debug 2022-01-31T21:49:10.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:10.179056+0000) 2022-01-31T21:49:10.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:10 smithi167 conmon[54076]: debug 2022-01-31T21:49:10.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:10.414884+0000) 2022-01-31T21:49:10.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:10 smithi167 conmon[60316]: debug 2022-01-31T21:49:10.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:10.414949+0000) 2022-01-31T21:49:10.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:10 smithi167 conmon[49112]: debug 2022-01-31T21:49:10.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:10.751189+0000) 2022-01-31T21:49:11.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:11 smithi171 conmon[51620]: debug 2022-01-31T21:49:11.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:11.021722+0000) 2022-01-31T21:49:11.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:11 smithi171 conmon[41853]: debug 2022-01-31T21:49:11.133+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:11.134361+0000) 2022-01-31T21:49:11.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:11 smithi171 conmon[46715]: debug 2022-01-31T21:49:11.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:11.179240+0000) 2022-01-31T21:49:11.647 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:11 smithi167 conmon[54076]: debug 2022-01-31T21:49:11.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:11.415007+0000) 2022-01-31T21:49:11.648 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:11 smithi167 conmon[60316]: debug 2022-01-31T21:49:11.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:11.415121+0000) 2022-01-31T21:49:11.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:11 smithi167 conmon[49112]: debug 2022-01-31T21:49:11.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:11.751366+0000) 2022-01-31T21:49:12.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:12 smithi171 conmon[51620]: debug 2022-01-31T21:49:12.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:12.021936+0000) 2022-01-31T21:49:12.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:12 smithi171 conmon[41853]: debug 2022-01-31T21:49:12.133+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:12.134547+0000) 2022-01-31T21:49:12.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:12 smithi171 conmon[46715]: debug 2022-01-31T21:49:12.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:12.179421+0000) 2022-01-31T21:49:12.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:12 smithi167 conmon[60316]: debug 2022-01-31T21:49:12.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:12.415268+0000) 2022-01-31T21:49:12.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:12 smithi167 conmon[54076]: debug 2022-01-31T21:49:12.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:12.415171+0000) 2022-01-31T21:49:12.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:12 smithi167 conmon[49112]: debug 2022-01-31T21:49:12.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:12.751546+0000) 2022-01-31T21:49:13.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:13 smithi171 conmon[51620]: debug 2022-01-31T21:49:13.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:13.022080+0000) 2022-01-31T21:49:13.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:13 smithi171 conmon[41853]: debug 2022-01-31T21:49:13.133+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:13.134721+0000) 2022-01-31T21:49:13.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:13 smithi171 conmon[46715]: debug 2022-01-31T21:49:13.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:13.179618+0000) 2022-01-31T21:49:13.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:13 smithi167 conmon[60316]: debug 2022-01-31T21:49:13.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:13.415431+0000) 2022-01-31T21:49:13.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:13 smithi167 conmon[54076]: debug 2022-01-31T21:49:13.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:13.415327+0000) 2022-01-31T21:49:13.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:13 smithi167 conmon[49112]: debug 2022-01-31T21:49:13.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:13.751729+0000) 2022-01-31T21:49:14.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[51620]: debug 2022-01-31T21:49:14.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.022284+0000) 2022-01-31T21:49:14.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[41853]: debug 2022-01-31T21:49:14.134+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.134923+0000) 2022-01-31T21:49:14.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[46715]: debug 2022-01-31T21:49:14.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.179866+0000) 2022-01-31T21:49:14.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:14 smithi167 conmon[60316]: debug 2022-01-31T21:49:14.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.415613+0000) 2022-01-31T21:49:14.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:14 smithi167 conmon[54076]: debug 2022-01-31T21:49:14.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.415516+0000) 2022-01-31T21:49:14.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:14 smithi167 conmon[54076]: debug 2022-01-31T21:49:14.795+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.796453+0000) 2022-01-31T21:49:14.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:14 smithi167 conmon[60316]: debug 2022-01-31T21:49:14.795+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.796189+0000) 2022-01-31T21:49:14.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:14 smithi167 conmon[49112]: debug 2022-01-31T21:49:14.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.751939+0000) 2022-01-31T21:49:14.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:14 smithi167 conmon[49112]: debug 2022-01-31T21:49:14.795+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.795825+0000) 2022-01-31T21:49:15.089 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[35325]: debug 2022-01-31T21:49:14.813+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160120 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:15.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[41853]: debug 2022-01-31T21:49:14.795+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.795977+0000) 2022-01-31T21:49:15.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[46715]: debug 2022-01-31T21:49:14.795+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.796676+0000) 2022-01-31T21:49:15.091 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:14 smithi171 conmon[51620]: debug 2022-01-31T21:49:14.797+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:14.797942+0000) 2022-01-31T21:49:15.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:15 smithi171 conmon[51620]: debug 2022-01-31T21:49:15.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:15.022411+0000) 2022-01-31T21:49:15.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:15 smithi171 conmon[41853]: debug 2022-01-31T21:49:15.134+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:15.135100+0000) 2022-01-31T21:49:15.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:15 smithi171 conmon[46715]: debug 2022-01-31T21:49:15.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:15.180011+0000) 2022-01-31T21:49:15.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:15 smithi167 conmon[54076]: debug 2022-01-31T21:49:15.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:15.415650+0000) 2022-01-31T21:49:15.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:15 smithi167 conmon[60316]: debug 2022-01-31T21:49:15.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:15.415715+0000) 2022-01-31T21:49:15.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:15 smithi167 conmon[49112]: debug 2022-01-31T21:49:15.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:15.752137+0000) 2022-01-31T21:49:16.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:16 smithi171 conmon[51620]: debug 2022-01-31T21:49:16.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:16.022522+0000) 2022-01-31T21:49:16.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:16 smithi171 conmon[46715]: debug 2022-01-31T21:49:16.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:16.180164+0000) 2022-01-31T21:49:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:16 smithi171 conmon[41853]: debug 2022-01-31T21:49:16.134+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:16.135229+0000) 2022-01-31T21:49:16.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:16 smithi167 conmon[54076]: debug 2022-01-31T21:49:16.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:16.415785+0000) 2022-01-31T21:49:16.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:16 smithi167 conmon[60316]: debug 2022-01-31T21:49:16.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:16.415835+0000) 2022-01-31T21:49:16.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:16 smithi167 conmon[49112]: debug 2022-01-31T21:49:16.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:16.752283+0000) 2022-01-31T21:49:17.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:17 smithi171 conmon[51620]: debug 2022-01-31T21:49:17.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:17.022678+0000) 2022-01-31T21:49:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:17 smithi171 conmon[41853]: debug 2022-01-31T21:49:17.134+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:17.135376+0000) 2022-01-31T21:49:17.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:17 smithi171 conmon[46715]: debug 2022-01-31T21:49:17.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:17.180347+0000) 2022-01-31T21:49:17.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:17 smithi167 conmon[54076]: debug 2022-01-31T21:49:17.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:17.416029+0000) 2022-01-31T21:49:17.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:17 smithi167 conmon[60316]: debug 2022-01-31T21:49:17.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:17.416029+0000) 2022-01-31T21:49:17.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:17 smithi167 conmon[49112]: debug 2022-01-31T21:49:17.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:17.752534+0000) 2022-01-31T21:49:18.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:18 smithi171 conmon[51620]: debug 2022-01-31T21:49:18.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:18.022848+0000) 2022-01-31T21:49:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:18 smithi171 conmon[41853]: debug 2022-01-31T21:49:18.134+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:18.135553+0000) 2022-01-31T21:49:18.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:18 smithi171 conmon[46715]: debug 2022-01-31T21:49:18.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:18.180492+0000) 2022-01-31T21:49:18.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:18 smithi167 conmon[54076]: debug 2022-01-31T21:49:18.414+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:18.416290+0000) 2022-01-31T21:49:18.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:18 smithi167 conmon[60316]: debug 2022-01-31T21:49:18.414+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:18.416290+0000) 2022-01-31T21:49:18.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:18 smithi167 conmon[49112]: debug 2022-01-31T21:49:18.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:18.752752+0000) 2022-01-31T21:49:19.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[51620]: debug 2022-01-31T21:49:19.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.023079+0000) 2022-01-31T21:49:19.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[41853]: debug 2022-01-31T21:49:19.135+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.135742+0000) 2022-01-31T21:49:19.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[46715]: debug 2022-01-31T21:49:19.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.180645+0000) 2022-01-31T21:49:19.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:19 smithi167 conmon[54076]: debug 2022-01-31T21:49:19.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.416483+0000) 2022-01-31T21:49:19.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:19 smithi167 conmon[60316]: debug 2022-01-31T21:49:19.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.416483+0000) 2022-01-31T21:49:19.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:19 smithi167 conmon[54076]: debug 2022-01-31T21:49:19.813+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.814992+0000) 2022-01-31T21:49:19.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:19 smithi167 conmon[60316]: debug 2022-01-31T21:49:19.813+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.815076+0000) 2022-01-31T21:49:19.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:19 smithi167 conmon[49112]: debug 2022-01-31T21:49:19.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.752953+0000) 2022-01-31T21:49:19.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:19 smithi167 conmon[49112]: debug 2022-01-31T21:49:19.813+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.815346+0000) 2022-01-31T21:49:20.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[35325]: debug 2022-01-31T21:49:19.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160228 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:20.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[41853]: debug 2022-01-31T21:49:19.814+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.815527+0000) 2022-01-31T21:49:20.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[46715]: debug 2022-01-31T21:49:19.815+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.816373+0000) 2022-01-31T21:49:20.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:19 smithi171 conmon[51620]: debug 2022-01-31T21:49:19.815+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:19.816581+0000) 2022-01-31T21:49:20.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:20 smithi171 conmon[51620]: debug 2022-01-31T21:49:20.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:20.023264+0000) 2022-01-31T21:49:20.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:20 smithi171 conmon[41853]: debug 2022-01-31T21:49:20.135+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:20.135946+0000) 2022-01-31T21:49:20.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:20 smithi171 conmon[46715]: debug 2022-01-31T21:49:20.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:20.180846+0000) 2022-01-31T21:49:20.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:20 smithi167 conmon[54076]: debug 2022-01-31T21:49:20.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:20.416719+0000) 2022-01-31T21:49:20.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:20 smithi167 conmon[60316]: debug 2022-01-31T21:49:20.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:20.416719+0000) 2022-01-31T21:49:20.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:20 smithi167 conmon[49112]: debug 2022-01-31T21:49:20.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:20.753154+0000) 2022-01-31T21:49:21.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:21 smithi171 conmon[51620]: debug 2022-01-31T21:49:21.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:21.023435+0000) 2022-01-31T21:49:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:21 smithi171 conmon[41853]: debug 2022-01-31T21:49:21.135+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:21.136115+0000) 2022-01-31T21:49:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:21 smithi171 conmon[46715]: debug 2022-01-31T21:49:21.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:21.180985+0000) 2022-01-31T21:49:21.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:21 smithi167 conmon[54076]: debug 2022-01-31T21:49:21.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:21.416862+0000) 2022-01-31T21:49:21.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:21 smithi167 conmon[60316]: debug 2022-01-31T21:49:21.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:21.416862+0000) 2022-01-31T21:49:21.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:21 smithi167 conmon[49112]: debug 2022-01-31T21:49:21.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:21.753272+0000) 2022-01-31T21:49:22.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:22 smithi171 conmon[51620]: debug 2022-01-31T21:49:22.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:22.023629+0000) 2022-01-31T21:49:22.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:22 smithi171 conmon[41853]: debug 2022-01-31T21:49:22.135+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:22.136320+0000) 2022-01-31T21:49:22.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:22 smithi171 conmon[46715]: debug 2022-01-31T21:49:22.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:22.181166+0000) 2022-01-31T21:49:22.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:22 smithi167 conmon[54076]: debug 2022-01-31T21:49:22.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:22.417049+0000) 2022-01-31T21:49:22.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:22 smithi167 conmon[60316]: debug 2022-01-31T21:49:22.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:22.417048+0000) 2022-01-31T21:49:22.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:22 smithi167 conmon[49112]: debug 2022-01-31T21:49:22.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:22.753454+0000) 2022-01-31T21:49:23.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:23 smithi171 conmon[51620]: debug 2022-01-31T21:49:23.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:23.023826+0000) 2022-01-31T21:49:23.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:23 smithi171 conmon[41853]: debug 2022-01-31T21:49:23.135+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:23.136520+0000) 2022-01-31T21:49:23.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:23 smithi171 conmon[46715]: debug 2022-01-31T21:49:23.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:23.181382+0000) 2022-01-31T21:49:23.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:23 smithi167 conmon[54076]: debug 2022-01-31T21:49:23.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:23.417244+0000) 2022-01-31T21:49:23.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:23 smithi167 conmon[60316]: debug 2022-01-31T21:49:23.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:23.417244+0000) 2022-01-31T21:49:23.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:23 smithi167 conmon[49112]: debug 2022-01-31T21:49:23.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:23.753608+0000) 2022-01-31T21:49:24.021 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:23 smithi171 conmon[35325]: debug 2022-01-31T21:49:23.745+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:49:24.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[46715]: debug 2022-01-31T21:49:24.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.181567+0000) 2022-01-31T21:49:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[41853]: debug 2022-01-31T21:49:24.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.136702+0000) 2022-01-31T21:49:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[51620]: debug 2022-01-31T21:49:24.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.024001+0000) 2022-01-31T21:49:24.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:24 smithi167 conmon[54076]: debug 2022-01-31T21:49:24.415+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.417384+0000) 2022-01-31T21:49:24.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:24 smithi167 conmon[60316]: debug 2022-01-31T21:49:24.415+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.417384+0000) 2022-01-31T21:49:24.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:24 smithi167 conmon[54076]: debug 2022-01-31T21:49:24.838+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.840469+0000) 2022-01-31T21:49:24.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:24 smithi167 conmon[60316]: debug 2022-01-31T21:49:24.839+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.841111+0000) 2022-01-31T21:49:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:24 smithi167 conmon[49112]: debug 2022-01-31T21:49:24.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.753773+0000) 2022-01-31T21:49:24.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:24 smithi167 conmon[49112]: debug 2022-01-31T21:49:24.839+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.840791+0000) 2022-01-31T21:49:25.091 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[35325]: debug 2022-01-31T21:49:24.858+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160341 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:25.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[41853]: debug 2022-01-31T21:49:24.840+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.841550+0000) 2022-01-31T21:49:25.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[46715]: debug 2022-01-31T21:49:24.840+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.841675+0000) 2022-01-31T21:49:25.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:24 smithi171 conmon[51620]: debug 2022-01-31T21:49:24.841+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:24.842553+0000) 2022-01-31T21:49:25.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:25 smithi171 conmon[51620]: debug 2022-01-31T21:49:25.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:25.024190+0000) 2022-01-31T21:49:25.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:25 smithi171 conmon[41853]: debug 2022-01-31T21:49:25.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:25.136778+0000) 2022-01-31T21:49:25.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:25 smithi171 conmon[46715]: debug 2022-01-31T21:49:25.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:25.181768+0000) 2022-01-31T21:49:25.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:25 smithi167 conmon[54076]: debug 2022-01-31T21:49:25.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:25.417527+0000) 2022-01-31T21:49:25.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:25 smithi167 conmon[60316]: debug 2022-01-31T21:49:25.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:25.417525+0000) 2022-01-31T21:49:25.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:25 smithi167 conmon[49112]: debug 2022-01-31T21:49:25.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:25.753972+0000) 2022-01-31T21:49:26.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:26 smithi171 conmon[51620]: debug 2022-01-31T21:49:26.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:26.024373+0000) 2022-01-31T21:49:26.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:26 smithi171 conmon[46715]: debug 2022-01-31T21:49:26.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:26.181871+0000) 2022-01-31T21:49:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:26 smithi171 conmon[41853]: debug 2022-01-31T21:49:26.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:26.136879+0000) 2022-01-31T21:49:26.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:26 smithi167 conmon[54076]: debug 2022-01-31T21:49:26.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:26.417638+0000) 2022-01-31T21:49:26.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:26 smithi167 conmon[60316]: debug 2022-01-31T21:49:26.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:26.417638+0000) 2022-01-31T21:49:26.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:26 smithi167 conmon[49112]: debug 2022-01-31T21:49:26.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:26.754148+0000) 2022-01-31T21:49:27.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:27 smithi171 conmon[51620]: debug 2022-01-31T21:49:27.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:27.024538+0000) 2022-01-31T21:49:27.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:27 smithi171 conmon[41853]: debug 2022-01-31T21:49:27.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:27.137040+0000) 2022-01-31T21:49:27.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:27 smithi171 conmon[46715]: debug 2022-01-31T21:49:27.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:27.182023+0000) 2022-01-31T21:49:27.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:27 smithi167 conmon[54076]: debug 2022-01-31T21:49:27.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:27.417867+0000) 2022-01-31T21:49:27.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:27 smithi167 conmon[60316]: debug 2022-01-31T21:49:27.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:27.417868+0000) 2022-01-31T21:49:27.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:27 smithi167 conmon[49112]: debug 2022-01-31T21:49:27.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:27.754357+0000) 2022-01-31T21:49:28.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:28 smithi171 conmon[51620]: debug 2022-01-31T21:49:28.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:28.024725+0000) 2022-01-31T21:49:28.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:28 smithi171 conmon[46715]: debug 2022-01-31T21:49:28.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:28.182238+0000) 2022-01-31T21:49:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:28 smithi171 conmon[41853]: debug 2022-01-31T21:49:28.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:28.137139+0000) 2022-01-31T21:49:28.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:28 smithi167 conmon[54076]: debug 2022-01-31T21:49:28.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:28.418082+0000) 2022-01-31T21:49:28.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:28 smithi167 conmon[60316]: debug 2022-01-31T21:49:28.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:28.418082+0000) 2022-01-31T21:49:28.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:28 smithi167 conmon[49112]: debug 2022-01-31T21:49:28.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:28.754561+0000) 2022-01-31T21:49:29.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[51620]: debug 2022-01-31T21:49:29.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.024959+0000) 2022-01-31T21:49:29.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[46715]: debug 2022-01-31T21:49:29.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.182417+0000) 2022-01-31T21:49:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[41853]: debug 2022-01-31T21:49:29.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.137293+0000) 2022-01-31T21:49:29.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:29 smithi167 conmon[54076]: debug 2022-01-31T21:49:29.416+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.418277+0000) 2022-01-31T21:49:29.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:29 smithi167 conmon[60316]: debug 2022-01-31T21:49:29.416+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.418277+0000) 2022-01-31T21:49:29.916 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:29 smithi167 conmon[54076]: debug 2022-01-31T21:49:29.859+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.860746+0000) 2022-01-31T21:49:29.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:29 smithi167 conmon[60316]: debug 2022-01-31T21:49:29.858+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.859905+0000) 2022-01-31T21:49:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:29 smithi167 conmon[49112]: debug 2022-01-31T21:49:29.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.754701+0000) 2022-01-31T21:49:29.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:29 smithi167 conmon[49112]: debug 2022-01-31T21:49:29.858+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.860204+0000) 2022-01-31T21:49:30.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[46715]: debug 2022-01-31T21:49:29.860+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.861571+0000) 2022-01-31T21:49:30.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[51620]: debug 2022-01-31T21:49:29.861+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.862101+0000) 2022-01-31T21:49:30.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:30 smithi171 conmon[51620]: debug 2022-01-31T21:49:30.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:30.025144+0000) 2022-01-31T21:49:30.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[35325]: debug 2022-01-31T21:49:29.877+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:30.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:29 smithi171 conmon[41853]: debug 2022-01-31T21:49:29.860+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:29.861705+0000) 2022-01-31T21:49:30.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:30 smithi171 conmon[41853]: debug 2022-01-31T21:49:30.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:30.137504+0000) 2022-01-31T21:49:30.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:30 smithi171 conmon[46715]: debug 2022-01-31T21:49:30.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:30.182527+0000) 2022-01-31T21:49:30.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:30 smithi167 conmon[54076]: debug 2022-01-31T21:49:30.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:30.418468+0000) 2022-01-31T21:49:30.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:30 smithi167 conmon[60316]: debug 2022-01-31T21:49:30.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:30.418468+0000) 2022-01-31T21:49:30.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:30 smithi167 conmon[49112]: debug 2022-01-31T21:49:30.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:30.754848+0000) 2022-01-31T21:49:31.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:31 smithi171 conmon[51620]: debug 2022-01-31T21:49:31.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:31.025298+0000) 2022-01-31T21:49:31.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:31 smithi171 conmon[41853]: debug 2022-01-31T21:49:31.136+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:31.137650+0000) 2022-01-31T21:49:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:31 smithi171 conmon[46715]: debug 2022-01-31T21:49:31.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:31.182683+0000) 2022-01-31T21:49:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:31 smithi167 conmon[54076]: debug 2022-01-31T21:49:31.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:31.418623+0000) 2022-01-31T21:49:31.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:31 smithi167 conmon[60316]: debug 2022-01-31T21:49:31.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:31.418625+0000) 2022-01-31T21:49:32.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:32 smithi171 conmon[51620]: debug 2022-01-31T21:49:32.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:32.025384+0000) 2022-01-31T21:49:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:31 smithi167 conmon[49112]: debug 2022-01-31T21:49:31.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:31.754938+0000) 2022-01-31T21:49:32.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:32 smithi171 conmon[41853]: debug 2022-01-31T21:49:32.137+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:32.137799+0000) 2022-01-31T21:49:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:32 smithi171 conmon[46715]: debug 2022-01-31T21:49:32.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:32.182902+0000) 2022-01-31T21:49:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:32 smithi167 conmon[54076]: debug 2022-01-31T21:49:32.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:32.418825+0000) 2022-01-31T21:49:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:32 smithi167 conmon[60316]: debug 2022-01-31T21:49:32.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:32.418825+0000) 2022-01-31T21:49:33.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:33 smithi171 conmon[51620]: debug 2022-01-31T21:49:33.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:33.025543+0000) 2022-01-31T21:49:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:32 smithi167 conmon[49112]: debug 2022-01-31T21:49:32.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:32.755062+0000) 2022-01-31T21:49:33.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:33 smithi171 conmon[46715]: debug 2022-01-31T21:49:33.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:33.183097+0000) 2022-01-31T21:49:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:33 smithi171 conmon[41853]: debug 2022-01-31T21:49:33.137+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:33.137937+0000) 2022-01-31T21:49:33.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:33 smithi167 conmon[54076]: debug 2022-01-31T21:49:33.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:33.419070+0000) 2022-01-31T21:49:33.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:33 smithi167 conmon[60316]: debug 2022-01-31T21:49:33.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:33.419070+0000) 2022-01-31T21:49:34.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[51620]: debug 2022-01-31T21:49:34.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.025703+0000) 2022-01-31T21:49:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:33 smithi167 conmon[49112]: debug 2022-01-31T21:49:33.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:33.755191+0000) 2022-01-31T21:49:34.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[41853]: debug 2022-01-31T21:49:34.137+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.138049+0000) 2022-01-31T21:49:34.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[46715]: debug 2022-01-31T21:49:34.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.183243+0000) 2022-01-31T21:49:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:34 smithi167 conmon[54076]: debug 2022-01-31T21:49:34.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.419319+0000) 2022-01-31T21:49:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:34 smithi167 conmon[60316]: debug 2022-01-31T21:49:34.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.419320+0000) 2022-01-31T21:49:35.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[35325]: debug 2022-01-31T21:49:34.896+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160576 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:35.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[46715]: debug 2022-01-31T21:49:34.879+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.879957+0000) 2022-01-31T21:49:35.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[41853]: debug 2022-01-31T21:49:34.879+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.880299+0000) 2022-01-31T21:49:35.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:34 smithi171 conmon[51620]: debug 2022-01-31T21:49:34.880+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.880981+0000) 2022-01-31T21:49:35.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:35 smithi171 conmon[51620]: debug 2022-01-31T21:49:35.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:35.025908+0000) 2022-01-31T21:49:35.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:34 smithi167 conmon[54076]: debug 2022-01-31T21:49:34.877+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.879189+0000) 2022-01-31T21:49:35.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:34 smithi167 conmon[60316]: debug 2022-01-31T21:49:34.878+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.879702+0000) 2022-01-31T21:49:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:34 smithi167 conmon[49112]: debug 2022-01-31T21:49:34.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.755340+0000) 2022-01-31T21:49:35.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:34 smithi167 conmon[49112]: debug 2022-01-31T21:49:34.878+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:34.880236+0000) 2022-01-31T21:49:35.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:35 smithi171 conmon[46715]: debug 2022-01-31T21:49:35.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:35.183423+0000) 2022-01-31T21:49:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:35 smithi171 conmon[41853]: debug 2022-01-31T21:49:35.137+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:35.138242+0000) 2022-01-31T21:49:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:35 smithi167 conmon[54076]: debug 2022-01-31T21:49:35.417+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:35.419472+0000) 2022-01-31T21:49:35.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:35 smithi167 conmon[60316]: debug 2022-01-31T21:49:35.417+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:35.419476+0000) 2022-01-31T21:49:36.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:36 smithi171 conmon[51620]: debug 2022-01-31T21:49:36.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:36.026032+0000) 2022-01-31T21:49:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:35 smithi167 conmon[49112]: debug 2022-01-31T21:49:35.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:35.755472+0000) 2022-01-31T21:49:36.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:36 smithi171 conmon[41853]: debug 2022-01-31T21:49:36.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:36.138401+0000) 2022-01-31T21:49:36.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:36 smithi171 conmon[46715]: debug 2022-01-31T21:49:36.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:36.183568+0000) 2022-01-31T21:49:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:36 smithi167 conmon[54076]: debug 2022-01-31T21:49:36.418+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:36.419551+0000) 2022-01-31T21:49:36.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:36 smithi167 conmon[60316]: debug 2022-01-31T21:49:36.418+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:36.419547+0000) 2022-01-31T21:49:37.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:37 smithi171 conmon[51620]: debug 2022-01-31T21:49:37.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:37.026191+0000) 2022-01-31T21:49:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:36 smithi167 conmon[49112]: debug 2022-01-31T21:49:36.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:36.755650+0000) 2022-01-31T21:49:37.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:37 smithi171 conmon[41853]: debug 2022-01-31T21:49:37.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:37.138602+0000) 2022-01-31T21:49:37.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:37 smithi171 conmon[46715]: debug 2022-01-31T21:49:37.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:37.183777+0000) 2022-01-31T21:49:37.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:37 smithi167 conmon[54076]: debug 2022-01-31T21:49:37.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:37.419721+0000) 2022-01-31T21:49:37.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:37 smithi167 conmon[60316]: debug 2022-01-31T21:49:37.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:37.419718+0000) 2022-01-31T21:49:38.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:38 smithi171 conmon[51620]: debug 2022-01-31T21:49:38.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:38.026384+0000) 2022-01-31T21:49:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:37 smithi167 conmon[49112]: debug 2022-01-31T21:49:37.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:37.755827+0000) 2022-01-31T21:49:38.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:38 smithi171 conmon[41853]: debug 2022-01-31T21:49:38.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:38.138780+0000) 2022-01-31T21:49:38.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:38 smithi171 conmon[46715]: debug 2022-01-31T21:49:38.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:38.183957+0000) 2022-01-31T21:49:38.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:38 smithi167 conmon[54076]: debug 2022-01-31T21:49:38.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:38.419912+0000) 2022-01-31T21:49:38.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:38 smithi167 conmon[60316]: debug 2022-01-31T21:49:38.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:38.419914+0000) 2022-01-31T21:49:39.024 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:38 smithi171 conmon[35325]: debug 2022-01-31T21:49:38.746+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:49:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:38 smithi167 conmon[49112]: debug 2022-01-31T21:49:38.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:38.755999+0000) 2022-01-31T21:49:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[41853]: debug 2022-01-31T21:49:39.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.138939+0000) 2022-01-31T21:49:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[46715]: debug 2022-01-31T21:49:39.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.184108+0000) 2022-01-31T21:49:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[51620]: debug 2022-01-31T21:49:39.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.026587+0000) 2022-01-31T21:49:39.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:39 smithi167 conmon[54076]: debug 2022-01-31T21:49:39.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.420116+0000) 2022-01-31T21:49:39.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:39 smithi167 conmon[60316]: debug 2022-01-31T21:49:39.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.420119+0000) 2022-01-31T21:49:40.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[41853]: debug 2022-01-31T21:49:39.898+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.899678+0000) 2022-01-31T21:49:40.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[46715]: debug 2022-01-31T21:49:39.898+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.899509+0000) 2022-01-31T21:49:40.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[35325]: debug 2022-01-31T21:49:39.916+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:40.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:39 smithi171 conmon[51620]: debug 2022-01-31T21:49:39.899+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.900610+0000) 2022-01-31T21:49:40.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:40 smithi171 conmon[51620]: debug 2022-01-31T21:49:40.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:40.026793+0000) 2022-01-31T21:49:40.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:39 smithi167 conmon[54076]: debug 2022-01-31T21:49:39.896+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.898417+0000) 2022-01-31T21:49:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:39 smithi167 conmon[49112]: debug 2022-01-31T21:49:39.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.756190+0000) 2022-01-31T21:49:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:39 smithi167 conmon[49112]: debug 2022-01-31T21:49:39.898+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.900138+0000) 2022-01-31T21:49:40.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:39 smithi167 conmon[60316]: debug 2022-01-31T21:49:39.897+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:39.898820+0000) 2022-01-31T21:49:40.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:40 smithi171 conmon[41853]: debug 2022-01-31T21:49:40.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:40.139145+0000) 2022-01-31T21:49:40.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:40 smithi171 conmon[46715]: debug 2022-01-31T21:49:40.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:40.184286+0000) 2022-01-31T21:49:40.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:40 smithi167 conmon[54076]: debug 2022-01-31T21:49:40.419+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:40.420347+0000) 2022-01-31T21:49:40.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:40 smithi167 conmon[60316]: debug 2022-01-31T21:49:40.419+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:40.420347+0000) 2022-01-31T21:49:41.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:41 smithi171 conmon[51620]: debug 2022-01-31T21:49:41.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:41.026945+0000) 2022-01-31T21:49:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:40 smithi167 conmon[49112]: debug 2022-01-31T21:49:40.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:40.756404+0000) 2022-01-31T21:49:41.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:41 smithi171 conmon[41853]: debug 2022-01-31T21:49:41.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:41.139239+0000) 2022-01-31T21:49:41.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:41 smithi171 conmon[46715]: debug 2022-01-31T21:49:41.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:41.184474+0000) 2022-01-31T21:49:41.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:41 smithi167 conmon[54076]: debug 2022-01-31T21:49:41.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:41.420553+0000) 2022-01-31T21:49:41.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:41 smithi167 conmon[60316]: debug 2022-01-31T21:49:41.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:41.420552+0000) 2022-01-31T21:49:42.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:42 smithi171 conmon[51620]: debug 2022-01-31T21:49:42.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:42.027102+0000) 2022-01-31T21:49:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:41 smithi167 conmon[49112]: debug 2022-01-31T21:49:41.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:41.756589+0000) 2022-01-31T21:49:42.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:42 smithi171 conmon[41853]: debug 2022-01-31T21:49:42.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:42.139339+0000) 2022-01-31T21:49:42.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:42 smithi171 conmon[46715]: debug 2022-01-31T21:49:42.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:42.184652+0000) 2022-01-31T21:49:42.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:42 smithi167 conmon[54076]: debug 2022-01-31T21:49:42.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:42.420742+0000) 2022-01-31T21:49:42.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:42 smithi167 conmon[60316]: debug 2022-01-31T21:49:42.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:42.420742+0000) 2022-01-31T21:49:43.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:43 smithi171 conmon[51620]: debug 2022-01-31T21:49:43.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:43.027269+0000) 2022-01-31T21:49:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:42 smithi167 conmon[49112]: debug 2022-01-31T21:49:42.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:42.756801+0000) 2022-01-31T21:49:43.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:43 smithi171 conmon[41853]: debug 2022-01-31T21:49:43.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:43.139506+0000) 2022-01-31T21:49:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:43 smithi171 conmon[46715]: debug 2022-01-31T21:49:43.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:43.184849+0000) 2022-01-31T21:49:43.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:43 smithi167 conmon[54076]: debug 2022-01-31T21:49:43.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:43.420945+0000) 2022-01-31T21:49:43.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:43 smithi167 conmon[60316]: debug 2022-01-31T21:49:43.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:43.420950+0000) 2022-01-31T21:49:44.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[51620]: debug 2022-01-31T21:49:44.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.027430+0000) 2022-01-31T21:49:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:43 smithi167 conmon[49112]: debug 2022-01-31T21:49:43.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:43.756953+0000) 2022-01-31T21:49:44.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[41853]: debug 2022-01-31T21:49:44.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.139697+0000) 2022-01-31T21:49:44.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[46715]: debug 2022-01-31T21:49:44.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.185071+0000) 2022-01-31T21:49:44.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:44 smithi167 conmon[54076]: debug 2022-01-31T21:49:44.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.421175+0000) 2022-01-31T21:49:44.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:44 smithi167 conmon[60316]: debug 2022-01-31T21:49:44.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.421176+0000) 2022-01-31T21:49:45.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[35325]: debug 2022-01-31T21:49:44.935+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160797 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:45.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[41853]: debug 2022-01-31T21:49:44.919+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.920466+0000) 2022-01-31T21:49:45.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[46715]: debug 2022-01-31T21:49:44.918+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.918849+0000) 2022-01-31T21:49:45.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:44 smithi171 conmon[51620]: debug 2022-01-31T21:49:44.919+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.919800+0000) 2022-01-31T21:49:45.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:45 smithi171 conmon[51620]: debug 2022-01-31T21:49:45.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:45.027607+0000) 2022-01-31T21:49:45.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:44 smithi167 conmon[54076]: debug 2022-01-31T21:49:44.917+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.917651+0000) 2022-01-31T21:49:45.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:44 smithi167 conmon[60316]: debug 2022-01-31T21:49:44.918+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.918524+0000) 2022-01-31T21:49:45.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:44 smithi167 conmon[49112]: debug 2022-01-31T21:49:44.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.757163+0000) 2022-01-31T21:49:45.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:44 smithi167 conmon[49112]: debug 2022-01-31T21:49:44.918+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:44.918853+0000) 2022-01-31T21:49:45.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:45 smithi171 conmon[41853]: debug 2022-01-31T21:49:45.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:45.139920+0000) 2022-01-31T21:49:45.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:45 smithi171 conmon[46715]: debug 2022-01-31T21:49:45.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:45.185276+0000) 2022-01-31T21:49:45.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:45 smithi167 conmon[54076]: debug 2022-01-31T21:49:45.420+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:45.421323+0000) 2022-01-31T21:49:45.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:45 smithi167 conmon[60316]: debug 2022-01-31T21:49:45.420+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:45.421370+0000) 2022-01-31T21:49:46.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:46 smithi171 conmon[51620]: debug 2022-01-31T21:49:46.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:46.027748+0000) 2022-01-31T21:49:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:45 smithi167 conmon[49112]: debug 2022-01-31T21:49:45.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:45.757351+0000) 2022-01-31T21:49:46.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:46 smithi171 conmon[41853]: debug 2022-01-31T21:49:46.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:46.140073+0000) 2022-01-31T21:49:46.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:46 smithi171 conmon[46715]: debug 2022-01-31T21:49:46.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:46.185441+0000) 2022-01-31T21:49:46.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:46 smithi167 conmon[54076]: debug 2022-01-31T21:49:46.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:46.421496+0000) 2022-01-31T21:49:46.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:46 smithi167 conmon[60316]: debug 2022-01-31T21:49:46.421+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:46.421535+0000) 2022-01-31T21:49:47.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:47 smithi171 conmon[51620]: debug 2022-01-31T21:49:47.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:47.027923+0000) 2022-01-31T21:49:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:46 smithi167 conmon[49112]: debug 2022-01-31T21:49:46.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:46.757528+0000) 2022-01-31T21:49:47.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:47 smithi171 conmon[41853]: debug 2022-01-31T21:49:47.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:47.140275+0000) 2022-01-31T21:49:47.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:47 smithi171 conmon[46715]: debug 2022-01-31T21:49:47.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:47.185599+0000) 2022-01-31T21:49:47.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:47 smithi167 conmon[54076]: debug 2022-01-31T21:49:47.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:47.421698+0000) 2022-01-31T21:49:47.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:47 smithi167 conmon[60316]: debug 2022-01-31T21:49:47.421+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:47.421697+0000) 2022-01-31T21:49:48.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:48 smithi171 conmon[51620]: debug 2022-01-31T21:49:48.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:48.028070+0000) 2022-01-31T21:49:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:47 smithi167 conmon[49112]: debug 2022-01-31T21:49:47.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:47.757720+0000) 2022-01-31T21:49:48.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:48 smithi171 conmon[41853]: debug 2022-01-31T21:49:48.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:48.140462+0000) 2022-01-31T21:49:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:48 smithi171 conmon[46715]: debug 2022-01-31T21:49:48.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:48.185796+0000) 2022-01-31T21:49:48.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:48 smithi167 conmon[54076]: debug 2022-01-31T21:49:48.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:48.421867+0000) 2022-01-31T21:49:48.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:48 smithi167 conmon[60316]: debug 2022-01-31T21:49:48.421+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:48.421874+0000) 2022-01-31T21:49:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:48 smithi167 conmon[49112]: debug 2022-01-31T21:49:48.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:48.757851+0000) 2022-01-31T21:49:49.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[41853]: debug 2022-01-31T21:49:49.138+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.140663+0000) 2022-01-31T21:49:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[46715]: debug 2022-01-31T21:49:49.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.185985+0000) 2022-01-31T21:49:49.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[51620]: debug 2022-01-31T21:49:49.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.028560+0000) 2022-01-31T21:49:49.724 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:49 smithi167 conmon[54076]: debug 2022-01-31T21:49:49.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.422047+0000) 2022-01-31T21:49:49.724 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:49 smithi167 conmon[60316]: debug 2022-01-31T21:49:49.421+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.422055+0000) 2022-01-31T21:49:49.988 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:49 smithi167 conmon[54076]: debug 2022-01-31T21:49:49.936+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.937463+0000) 2022-01-31T21:49:49.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:49 smithi167 conmon[60316]: debug 2022-01-31T21:49:49.937+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.938246+0000) 2022-01-31T21:49:49.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:49 smithi167 conmon[49112]: debug 2022-01-31T21:49:49.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.758015+0000) 2022-01-31T21:49:49.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:49 smithi167 conmon[49112]: debug 2022-01-31T21:49:49.937+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.938368+0000) 2022-01-31T21:49:50.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[35325]: debug 2022-01-31T21:49:49.953+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 160907 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:50.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[41853]: debug 2022-01-31T21:49:49.937+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.939621+0000) 2022-01-31T21:49:50.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[46715]: debug 2022-01-31T21:49:49.937+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.939029+0000) 2022-01-31T21:49:50.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:49 smithi171 conmon[51620]: debug 2022-01-31T21:49:49.938+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:49.939770+0000) 2022-01-31T21:49:50.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:50 smithi171 conmon[51620]: debug 2022-01-31T21:49:50.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:50.028712+0000) 2022-01-31T21:49:50.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:50 smithi171 conmon[41853]: debug 2022-01-31T21:49:50.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:50.140763+0000) 2022-01-31T21:49:50.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:50 smithi171 conmon[46715]: debug 2022-01-31T21:49:50.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:50.186134+0000) 2022-01-31T21:49:50.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:50 smithi167 conmon[54076]: debug 2022-01-31T21:49:50.421+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:50.422230+0000) 2022-01-31T21:49:50.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:50 smithi167 conmon[60316]: debug 2022-01-31T21:49:50.421+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:50.422230+0000) 2022-01-31T21:49:51.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:51 smithi171 conmon[51620]: debug 2022-01-31T21:49:51.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:51.028897+0000) 2022-01-31T21:49:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:50 smithi167 conmon[49112]: debug 2022-01-31T21:49:50.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:50.758200+0000) 2022-01-31T21:49:51.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:51 smithi171 conmon[41853]: debug 2022-01-31T21:49:51.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:51.140909+0000) 2022-01-31T21:49:51.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:51 smithi171 conmon[46715]: debug 2022-01-31T21:49:51.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:51.186274+0000) 2022-01-31T21:49:51.629 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:51 smithi167 conmon[54076]: debug 2022-01-31T21:49:51.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:51.422402+0000) 2022-01-31T21:49:51.630 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:51 smithi167 conmon[60316]: debug 2022-01-31T21:49:51.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:51.422405+0000) 2022-01-31T21:49:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:51 smithi167 conmon[49112]: debug 2022-01-31T21:49:51.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:51.758344+0000) 2022-01-31T21:49:52.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:52 smithi171 conmon[51620]: debug 2022-01-31T21:49:52.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:52.029123+0000) 2022-01-31T21:49:52.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:52 smithi171 conmon[41853]: debug 2022-01-31T21:49:52.139+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:52.141057+0000) 2022-01-31T21:49:52.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:52 smithi171 conmon[46715]: debug 2022-01-31T21:49:52.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:52.186486+0000) 2022-01-31T21:49:52.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:52 smithi167 conmon[54076]: debug 2022-01-31T21:49:52.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:52.422600+0000) 2022-01-31T21:49:52.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:52 smithi167 conmon[60316]: debug 2022-01-31T21:49:52.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:52.422599+0000) 2022-01-31T21:49:53.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:53 smithi171 conmon[51620]: debug 2022-01-31T21:49:53.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:53.029284+0000) 2022-01-31T21:49:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:52 smithi167 conmon[49112]: debug 2022-01-31T21:49:52.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:52.758547+0000) 2022-01-31T21:49:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:53 smithi171 conmon[41853]: debug 2022-01-31T21:49:53.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:53.141243+0000) 2022-01-31T21:49:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:53 smithi171 conmon[46715]: debug 2022-01-31T21:49:53.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:53.186688+0000) 2022-01-31T21:49:53.630 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:53 smithi167 conmon[54076]: debug 2022-01-31T21:49:53.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:53.422797+0000) 2022-01-31T21:49:53.630 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:53 smithi167 conmon[60316]: debug 2022-01-31T21:49:53.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:53.422797+0000) 2022-01-31T21:49:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:53 smithi167 conmon[49112]: debug 2022-01-31T21:49:53.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:53.758699+0000) 2022-01-31T21:49:54.027 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:53 smithi171 conmon[35325]: debug 2022-01-31T21:49:53.746+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:49:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[41853]: debug 2022-01-31T21:49:54.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.141422+0000) 2022-01-31T21:49:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[46715]: debug 2022-01-31T21:49:54.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.186906+0000) 2022-01-31T21:49:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[51620]: debug 2022-01-31T21:49:54.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.029461+0000) 2022-01-31T21:49:54.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:54 smithi167 conmon[54076]: debug 2022-01-31T21:49:54.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.423034+0000) 2022-01-31T21:49:54.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:54 smithi167 conmon[60316]: debug 2022-01-31T21:49:54.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.423039+0000) 2022-01-31T21:49:55.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[35325]: debug 2022-01-31T21:49:54.973+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161018 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:49:55.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[41853]: debug 2022-01-31T21:49:54.956+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.958646+0000) 2022-01-31T21:49:55.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[46715]: debug 2022-01-31T21:49:54.955+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.957233+0000) 2022-01-31T21:49:55.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:54 smithi171 conmon[51620]: debug 2022-01-31T21:49:54.957+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.959059+0000) 2022-01-31T21:49:55.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:55 smithi171 conmon[51620]: debug 2022-01-31T21:49:55.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:55.029657+0000) 2022-01-31T21:49:55.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:54 smithi167 conmon[54076]: debug 2022-01-31T21:49:54.955+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.956459+0000) 2022-01-31T21:49:55.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:54 smithi167 conmon[60316]: debug 2022-01-31T21:49:54.956+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.957383+0000) 2022-01-31T21:49:55.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:54 smithi167 conmon[49112]: debug 2022-01-31T21:49:54.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.758902+0000) 2022-01-31T21:49:55.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:54 smithi167 conmon[49112]: debug 2022-01-31T21:49:54.956+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:54.957274+0000) 2022-01-31T21:49:55.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:55 smithi171 conmon[41853]: debug 2022-01-31T21:49:55.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:55.141608+0000) 2022-01-31T21:49:55.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:55 smithi171 conmon[46715]: debug 2022-01-31T21:49:55.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:55.187088+0000) 2022-01-31T21:49:55.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:55 smithi167 conmon[54076]: debug 2022-01-31T21:49:55.422+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:55.423231+0000) 2022-01-31T21:49:55.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:55 smithi167 conmon[60316]: debug 2022-01-31T21:49:55.422+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:55.423234+0000) 2022-01-31T21:49:56.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:56 smithi171 conmon[51620]: debug 2022-01-31T21:49:56.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:56.029819+0000) 2022-01-31T21:49:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:55 smithi167 conmon[49112]: debug 2022-01-31T21:49:55.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:55.759090+0000) 2022-01-31T21:49:56.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:56 smithi171 conmon[46715]: debug 2022-01-31T21:49:56.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:56.187247+0000) 2022-01-31T21:49:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:56 smithi171 conmon[41853]: debug 2022-01-31T21:49:56.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:56.141819+0000) 2022-01-31T21:49:56.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:56 smithi167 conmon[54076]: debug 2022-01-31T21:49:56.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:56.423416+0000) 2022-01-31T21:49:56.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:56 smithi167 conmon[60316]: debug 2022-01-31T21:49:56.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:56.423416+0000) 2022-01-31T21:49:57.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:57 smithi171 conmon[51620]: debug 2022-01-31T21:49:57.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:57.030032+0000) 2022-01-31T21:49:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:56 smithi167 conmon[49112]: debug 2022-01-31T21:49:56.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:56.759309+0000) 2022-01-31T21:49:57.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:57 smithi171 conmon[46715]: debug 2022-01-31T21:49:57.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:57.187384+0000) 2022-01-31T21:49:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:57 smithi171 conmon[41853]: debug 2022-01-31T21:49:57.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:57.141957+0000) 2022-01-31T21:49:57.630 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:57 smithi167 conmon[54076]: debug 2022-01-31T21:49:57.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:57.423617+0000) 2022-01-31T21:49:57.631 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:57 smithi167 conmon[60316]: debug 2022-01-31T21:49:57.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:57.423620+0000) 2022-01-31T21:49:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:57 smithi167 conmon[49112]: debug 2022-01-31T21:49:57.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:57.759485+0000) 2022-01-31T21:49:58.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:58 smithi171 conmon[51620]: debug 2022-01-31T21:49:58.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:58.030220+0000) 2022-01-31T21:49:58.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:58 smithi171 conmon[46715]: debug 2022-01-31T21:49:58.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:58.187522+0000) 2022-01-31T21:49:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:58 smithi171 conmon[41853]: debug 2022-01-31T21:49:58.140+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:58.142076+0000) 2022-01-31T21:49:58.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:58 smithi167 conmon[54076]: debug 2022-01-31T21:49:58.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:58.423756+0000) 2022-01-31T21:49:58.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:58 smithi167 conmon[60316]: debug 2022-01-31T21:49:58.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:58.423752+0000) 2022-01-31T21:49:59.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[51620]: debug 2022-01-31T21:49:59.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.030412+0000) 2022-01-31T21:49:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:58 smithi167 conmon[49112]: debug 2022-01-31T21:49:58.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:58.759826+0000) 2022-01-31T21:49:59.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[46715]: debug 2022-01-31T21:49:59.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.187655+0000) 2022-01-31T21:49:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[41853]: debug 2022-01-31T21:49:59.141+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.142282+0000) 2022-01-31T21:49:59.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:59 smithi167 conmon[54076]: debug 2022-01-31T21:49:59.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.423907+0000) 2022-01-31T21:49:59.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:59 smithi167 conmon[60316]: debug 2022-01-31T21:49:59.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.423907+0000) 2022-01-31T21:50:00.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[35325]: debug 2022-01-31T21:49:59.993+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161128 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:00.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[46715]: debug 2022-01-31T21:49:59.975+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.976814+0000) 2022-01-31T21:50:00.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[41853]: debug 2022-01-31T21:49:59.977+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.979682+0000) 2022-01-31T21:50:00.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:49:59 smithi171 conmon[51620]: debug 2022-01-31T21:49:59.977+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.979415+0000) 2022-01-31T21:50:00.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:00 smithi171 conmon[51620]: debug 2022-01-31T21:50:00.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:00.030602+0000) 2022-01-31T21:50:00.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:49:59 smithi167 conmon[54076]: debug 2022-01-31T21:49:59.977+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.978305+0000) 2022-01-31T21:50:00.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:49:59 smithi167 conmon[60316]: debug 2022-01-31T21:49:59.977+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.978114+0000) 2022-01-31T21:50:00.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:59 smithi167 conmon[49112]: debug 2022-01-31T21:49:59.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.760017+0000) 2022-01-31T21:50:00.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:49:59 smithi167 conmon[49112]: debug 2022-01-31T21:49:59.976+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:49:59.977065+0000) 2022-01-31T21:50:00.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:00 smithi171 conmon[46715]: debug 2022-01-31T21:50:00.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:00.187812+0000) 2022-01-31T21:50:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:00 smithi171 conmon[41853]: debug 2022-01-31T21:50:00.141+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:00.142483+0000) 2022-01-31T21:50:00.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:00 smithi167 conmon[54076]: debug 2022-01-31T21:50:00.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:00.424118+0000) 2022-01-31T21:50:00.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:00 smithi167 conmon[60316]: debug 2022-01-31T21:50:00.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:00.424098+0000) 2022-01-31T21:50:01.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:01 smithi171 conmon[51620]: debug 2022-01-31T21:50:01.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:01.030772+0000) 2022-01-31T21:50:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:00 smithi167 conmon[49112]: debug 2022-01-31T21:50:00.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:00.760220+0000) 2022-01-31T21:50:01.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:01 smithi171 conmon[46715]: debug 2022-01-31T21:50:01.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:01.187965+0000) 2022-01-31T21:50:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:01 smithi171 conmon[41853]: debug 2022-01-31T21:50:01.141+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:01.142671+0000) 2022-01-31T21:50:01.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:01 smithi167 conmon[54076]: debug 2022-01-31T21:50:01.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:01.424270+0000) 2022-01-31T21:50:01.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:01 smithi167 conmon[60316]: debug 2022-01-31T21:50:01.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:01.424277+0000) 2022-01-31T21:50:02.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:02 smithi171 conmon[51620]: debug 2022-01-31T21:50:02.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:02.030979+0000) 2022-01-31T21:50:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:01 smithi167 conmon[49112]: debug 2022-01-31T21:50:01.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:01.760377+0000) 2022-01-31T21:50:02.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:02 smithi171 conmon[41853]: debug 2022-01-31T21:50:02.142+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:02.142863+0000) 2022-01-31T21:50:02.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:02 smithi171 conmon[46715]: debug 2022-01-31T21:50:02.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:02.188170+0000) 2022-01-31T21:50:02.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:02 smithi167 conmon[54076]: debug 2022-01-31T21:50:02.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:02.424477+0000) 2022-01-31T21:50:02.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:02 smithi167 conmon[60316]: debug 2022-01-31T21:50:02.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:02.424485+0000) 2022-01-31T21:50:03.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:03 smithi171 conmon[51620]: debug 2022-01-31T21:50:03.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:03.031203+0000) 2022-01-31T21:50:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:02 smithi167 conmon[49112]: debug 2022-01-31T21:50:02.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:02.760508+0000) 2022-01-31T21:50:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:03 smithi171 conmon[41853]: debug 2022-01-31T21:50:03.142+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:03.143005+0000) 2022-01-31T21:50:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:03 smithi171 conmon[46715]: debug 2022-01-31T21:50:03.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:03.188351+0000) 2022-01-31T21:50:03.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:03 smithi167 conmon[54076]: debug 2022-01-31T21:50:03.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:03.424702+0000) 2022-01-31T21:50:03.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:03 smithi167 conmon[60316]: debug 2022-01-31T21:50:03.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:03.424689+0000) 2022-01-31T21:50:04.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:04 smithi171 conmon[51620]: debug 2022-01-31T21:50:04.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.031367+0000) 2022-01-31T21:50:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:03 smithi167 conmon[49112]: debug 2022-01-31T21:50:03.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:03.760649+0000) 2022-01-31T21:50:04.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:04 smithi171 conmon[41853]: debug 2022-01-31T21:50:04.142+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.143127+0000) 2022-01-31T21:50:04.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:04 smithi171 conmon[46715]: debug 2022-01-31T21:50:04.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.188549+0000) 2022-01-31T21:50:04.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:04 smithi167 conmon[54076]: debug 2022-01-31T21:50:04.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.424899+0000) 2022-01-31T21:50:04.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:04 smithi167 conmon[60316]: debug 2022-01-31T21:50:04.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.424895+0000) 2022-01-31T21:50:04.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:04 smithi167 conmon[49112]: debug 2022-01-31T21:50:04.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.760836+0000) 2022-01-31T21:50:05.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:04 smithi171 conmon[41853]: debug 2022-01-31T21:50:04.997+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.997920+0000) 2022-01-31T21:50:05.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:04 smithi171 conmon[46715]: debug 2022-01-31T21:50:04.996+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.997383+0000) 2022-01-31T21:50:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:05 smithi171 conmon[35325]: debug 2022-01-31T21:50:05.013+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161239 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:05.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:04 smithi171 conmon[51620]: debug 2022-01-31T21:50:04.996+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.997645+0000) 2022-01-31T21:50:05.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:05 smithi171 conmon[51620]: debug 2022-01-31T21:50:05.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:05.031578+0000) 2022-01-31T21:50:05.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:04 smithi167 conmon[49112]: debug 2022-01-31T21:50:04.994+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.995796+0000) 2022-01-31T21:50:05.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:04 smithi167 conmon[54076]: debug 2022-01-31T21:50:04.995+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.997335+0000) 2022-01-31T21:50:05.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:04 smithi167 conmon[60316]: debug 2022-01-31T21:50:04.995+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:04.996854+0000) 2022-01-31T21:50:05.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:05 smithi171 conmon[41853]: debug 2022-01-31T21:50:05.142+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:05.143289+0000) 2022-01-31T21:50:05.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:05 smithi171 conmon[46715]: debug 2022-01-31T21:50:05.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:05.188721+0000) 2022-01-31T21:50:05.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:05 smithi167 conmon[54076]: debug 2022-01-31T21:50:05.423+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:05.425107+0000) 2022-01-31T21:50:05.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:05 smithi167 conmon[60316]: debug 2022-01-31T21:50:05.423+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:05.425104+0000) 2022-01-31T21:50:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:05 smithi167 conmon[49112]: debug 2022-01-31T21:50:05.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:05.760998+0000) 2022-01-31T21:50:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:06 smithi171 conmon[41853]: debug 2022-01-31T21:50:06.142+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:06.143471+0000) 2022-01-31T21:50:06.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:06 smithi171 conmon[46715]: debug 2022-01-31T21:50:06.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:06.188816+0000) 2022-01-31T21:50:06.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:06 smithi171 conmon[51620]: debug 2022-01-31T21:50:06.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:06.031717+0000) 2022-01-31T21:50:06.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:06 smithi167 conmon[54076]: debug 2022-01-31T21:50:06.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:06.425282+0000) 2022-01-31T21:50:06.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:06 smithi167 conmon[60316]: debug 2022-01-31T21:50:06.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:06.425274+0000) 2022-01-31T21:50:07.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:07 smithi171 conmon[51620]: debug 2022-01-31T21:50:07.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:07.031915+0000) 2022-01-31T21:50:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:06 smithi167 conmon[49112]: debug 2022-01-31T21:50:06.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:06.761106+0000) 2022-01-31T21:50:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:07 smithi171 conmon[41853]: debug 2022-01-31T21:50:07.142+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:07.143614+0000) 2022-01-31T21:50:07.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:07 smithi171 conmon[46715]: debug 2022-01-31T21:50:07.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:07.188929+0000) 2022-01-31T21:50:07.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:07 smithi167 conmon[54076]: debug 2022-01-31T21:50:07.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:07.425464+0000) 2022-01-31T21:50:07.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:07 smithi167 conmon[60316]: debug 2022-01-31T21:50:07.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:07.425465+0000) 2022-01-31T21:50:08.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:08 smithi171 conmon[51620]: debug 2022-01-31T21:50:08.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:08.032100+0000) 2022-01-31T21:50:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:07 smithi167 conmon[49112]: debug 2022-01-31T21:50:07.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:07.761269+0000) 2022-01-31T21:50:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:08 smithi171 conmon[41853]: debug 2022-01-31T21:50:08.143+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:08.143819+0000) 2022-01-31T21:50:08.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:08 smithi171 conmon[46715]: debug 2022-01-31T21:50:08.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:08.189171+0000) 2022-01-31T21:50:08.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:08 smithi167 conmon[54076]: debug 2022-01-31T21:50:08.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:08.425657+0000) 2022-01-31T21:50:08.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:08 smithi167 conmon[60316]: debug 2022-01-31T21:50:08.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:08.425657+0000) 2022-01-31T21:50:09.030 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:08 smithi171 conmon[35325]: debug 2022-01-31T21:50:08.747+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:50:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:08 smithi167 conmon[49112]: debug 2022-01-31T21:50:08.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:08.761457+0000) 2022-01-31T21:50:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:09 smithi171 conmon[41853]: debug 2022-01-31T21:50:09.143+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:09.144004+0000) 2022-01-31T21:50:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:09 smithi171 conmon[46715]: debug 2022-01-31T21:50:09.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:09.189328+0000) 2022-01-31T21:50:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:09 smithi171 conmon[51620]: debug 2022-01-31T21:50:09.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:09.032255+0000) 2022-01-31T21:50:09.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:09 smithi167 conmon[54076]: debug 2022-01-31T21:50:09.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:09.425847+0000) 2022-01-31T21:50:09.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:09 smithi167 conmon[60316]: debug 2022-01-31T21:50:09.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:09.425847+0000) 2022-01-31T21:50:10.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:09 smithi167 conmon[49112]: debug 2022-01-31T21:50:09.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:09.761632+0000) 2022-01-31T21:50:10.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[46715]: debug 2022-01-31T21:50:10.015+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.016534+0000) 2022-01-31T21:50:10.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[35325]: debug 2022-01-31T21:50:10.033+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161349 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:10.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[41853]: debug 2022-01-31T21:50:10.016+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.017001+0000) 2022-01-31T21:50:10.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[51620]: debug 2022-01-31T21:50:10.016+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.017583+0000) 2022-01-31T21:50:10.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[51620]: debug 2022-01-31T21:50:10.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.032462+0000) 2022-01-31T21:50:10.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:10 smithi167 conmon[49112]: debug 2022-01-31T21:50:10.014+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.015922+0000) 2022-01-31T21:50:10.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:10 smithi167 conmon[54076]: debug 2022-01-31T21:50:10.015+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.017185+0000) 2022-01-31T21:50:10.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:10 smithi167 conmon[60316]: debug 2022-01-31T21:50:10.014+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.015747+0000) 2022-01-31T21:50:10.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[46715]: debug 2022-01-31T21:50:10.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.189483+0000) 2022-01-31T21:50:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:10 smithi171 conmon[41853]: debug 2022-01-31T21:50:10.143+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.144190+0000) 2022-01-31T21:50:10.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:10 smithi167 conmon[54076]: debug 2022-01-31T21:50:10.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.426051+0000) 2022-01-31T21:50:10.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:10 smithi167 conmon[60316]: debug 2022-01-31T21:50:10.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.426003+0000) 2022-01-31T21:50:11.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:11 smithi171 conmon[51620]: debug 2022-01-31T21:50:11.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:11.032618+0000) 2022-01-31T21:50:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:10 smithi167 conmon[49112]: debug 2022-01-31T21:50:10.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:10.761756+0000) 2022-01-31T21:50:11.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:11 smithi171 conmon[46715]: debug 2022-01-31T21:50:11.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:11.189686+0000) 2022-01-31T21:50:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:11 smithi171 conmon[41853]: debug 2022-01-31T21:50:11.143+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:11.144348+0000) 2022-01-31T21:50:11.634 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:11 smithi167 conmon[60316]: debug 2022-01-31T21:50:11.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:11.426199+0000) 2022-01-31T21:50:11.635 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:11 smithi167 conmon[54076]: debug 2022-01-31T21:50:11.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:11.426210+0000) 2022-01-31T21:50:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:11 smithi167 conmon[49112]: debug 2022-01-31T21:50:11.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:11.761914+0000) 2022-01-31T21:50:12.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:12 smithi171 conmon[51620]: debug 2022-01-31T21:50:12.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:12.032837+0000) 2022-01-31T21:50:12.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:12 smithi171 conmon[46715]: debug 2022-01-31T21:50:12.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:12.189878+0000) 2022-01-31T21:50:12.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:12 smithi171 conmon[41853]: debug 2022-01-31T21:50:12.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:12.144536+0000) 2022-01-31T21:50:12.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:12 smithi167 conmon[60316]: debug 2022-01-31T21:50:12.424+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:12.426396+0000) 2022-01-31T21:50:12.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:12 smithi167 conmon[54076]: debug 2022-01-31T21:50:12.424+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:12.426396+0000) 2022-01-31T21:50:13.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:13 smithi171 conmon[51620]: debug 2022-01-31T21:50:13.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:13.033036+0000) 2022-01-31T21:50:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:12 smithi167 conmon[49112]: debug 2022-01-31T21:50:12.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:12.762107+0000) 2022-01-31T21:50:13.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:13 smithi171 conmon[41853]: debug 2022-01-31T21:50:13.143+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:13.144668+0000) 2022-01-31T21:50:13.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:13 smithi171 conmon[46715]: debug 2022-01-31T21:50:13.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:13.190032+0000) 2022-01-31T21:50:13.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:13 smithi167 conmon[54076]: debug 2022-01-31T21:50:13.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:13.426599+0000) 2022-01-31T21:50:13.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:13 smithi167 conmon[60316]: debug 2022-01-31T21:50:13.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:13.426597+0000) 2022-01-31T21:50:14.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:14 smithi171 conmon[51620]: debug 2022-01-31T21:50:14.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:14.033234+0000) 2022-01-31T21:50:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:13 smithi167 conmon[49112]: debug 2022-01-31T21:50:13.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:13.762307+0000) 2022-01-31T21:50:14.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:14 smithi171 conmon[41853]: debug 2022-01-31T21:50:14.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:14.144772+0000) 2022-01-31T21:50:14.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:14 smithi171 conmon[46715]: debug 2022-01-31T21:50:14.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:14.190191+0000) 2022-01-31T21:50:14.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:14 smithi167 conmon[54076]: debug 2022-01-31T21:50:14.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:14.426820+0000) 2022-01-31T21:50:14.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:14 smithi167 conmon[60316]: debug 2022-01-31T21:50:14.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:14.426816+0000) 2022-01-31T21:50:15.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:14 smithi167 conmon[49112]: debug 2022-01-31T21:50:14.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:14.762492+0000) 2022-01-31T21:50:15.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[35325]: debug 2022-01-31T21:50:15.052+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161460 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:15.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[46715]: debug 2022-01-31T21:50:15.035+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.036455+0000) 2022-01-31T21:50:15.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[41853]: debug 2022-01-31T21:50:15.036+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.037388+0000) 2022-01-31T21:50:15.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[51620]: debug 2022-01-31T21:50:15.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.033443+0000) 2022-01-31T21:50:15.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[51620]: debug 2022-01-31T21:50:15.036+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.036819+0000) 2022-01-31T21:50:15.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[49112]: debug 2022-01-31T21:50:15.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[49112]: 2022-01-31T21:50:15.033+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.035103+0000) 2022-01-31T21:50:15.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[54076]: debug 2022-01-31T21:50:15.034+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.036155+0000) 2022-01-31T21:50:15.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[60316]: debug 2022-01-31T21:50:15.034+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.036256+0000) 2022-01-31T21:50:15.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[46715]: debug 2022-01-31T21:50:15.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.190357+0000) 2022-01-31T21:50:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:15 smithi171 conmon[41853]: debug 2022-01-31T21:50:15.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.144954+0000) 2022-01-31T21:50:15.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[54076]: debug 2022-01-31T21:50:15.425+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.427011+0000) 2022-01-31T21:50:15.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[60316]: debug 2022-01-31T21:50:15.425+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.427032+0000) 2022-01-31T21:50:16.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:16 smithi171 conmon[51620]: debug 2022-01-31T21:50:16.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:16.033583+0000) 2022-01-31T21:50:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:15 smithi167 conmon[49112]: debug 2022-01-31T21:50:15.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:15.762689+0000) 2022-01-31T21:50:16.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:16 smithi171 conmon[46715]: debug 2022-01-31T21:50:16.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:16.190471+0000) 2022-01-31T21:50:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:16 smithi171 conmon[41853]: debug 2022-01-31T21:50:16.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:16.145086+0000) 2022-01-31T21:50:16.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:16 smithi167 conmon[54076]: debug 2022-01-31T21:50:16.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:16.427176+0000) 2022-01-31T21:50:16.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:16 smithi167 conmon[60316]: debug 2022-01-31T21:50:16.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:16.427175+0000) 2022-01-31T21:50:17.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:17 smithi171 conmon[51620]: debug 2022-01-31T21:50:17.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:17.033712+0000) 2022-01-31T21:50:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:16 smithi167 conmon[49112]: debug 2022-01-31T21:50:16.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:16.762835+0000) 2022-01-31T21:50:17.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:17 smithi171 conmon[46715]: debug 2022-01-31T21:50:17.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:17.190632+0000) 2022-01-31T21:50:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:17 smithi171 conmon[41853]: debug 2022-01-31T21:50:17.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:17.145272+0000) 2022-01-31T21:50:17.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:17 smithi167 conmon[54076]: debug 2022-01-31T21:50:17.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:17.427379+0000) 2022-01-31T21:50:17.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:17 smithi167 conmon[60316]: debug 2022-01-31T21:50:17.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:17.427376+0000) 2022-01-31T21:50:18.143 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:18 smithi171 conmon[51620]: debug 2022-01-31T21:50:18.033+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:18.033890+0000) 2022-01-31T21:50:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:17 smithi167 conmon[49112]: debug 2022-01-31T21:50:17.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:17.763022+0000) 2022-01-31T21:50:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:18 smithi171 conmon[41853]: debug 2022-01-31T21:50:18.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:18.145477+0000) 2022-01-31T21:50:18.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:18 smithi171 conmon[46715]: debug 2022-01-31T21:50:18.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:18.190831+0000) 2022-01-31T21:50:18.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:18 smithi167 conmon[60316]: debug 2022-01-31T21:50:18.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:18.427599+0000) 2022-01-31T21:50:18.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:18 smithi167 conmon[54076]: debug 2022-01-31T21:50:18.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:18.427577+0000) 2022-01-31T21:50:19.143 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:19 smithi171 conmon[51620]: debug 2022-01-31T21:50:19.033+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:19.034070+0000) 2022-01-31T21:50:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:18 smithi167 conmon[49112]: debug 2022-01-31T21:50:18.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:18.763203+0000) 2022-01-31T21:50:19.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:19 smithi171 conmon[41853]: debug 2022-01-31T21:50:19.144+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:19.145643+0000) 2022-01-31T21:50:19.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:19 smithi171 conmon[46715]: debug 2022-01-31T21:50:19.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:19.190980+0000) 2022-01-31T21:50:19.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:19 smithi167 conmon[60316]: debug 2022-01-31T21:50:19.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:19.427776+0000) 2022-01-31T21:50:19.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:19 smithi167 conmon[54076]: debug 2022-01-31T21:50:19.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:19.427776+0000) 2022-01-31T21:50:20.053 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:19 smithi167 conmon[49112]: debug 2022-01-31T21:50:19.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:19.763408+0000) 2022-01-31T21:50:20.143 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[35325]: debug 2022-01-31T21:50:20.078+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161570 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:20.144 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[41853]: debug 2022-01-31T21:50:20.054+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.055384+0000) 2022-01-31T21:50:20.145 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[46715]: debug 2022-01-31T21:50:20.054+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.055644+0000) 2022-01-31T21:50:20.145 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[51620]: debug 2022-01-31T21:50:20.033+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.034175+0000) 2022-01-31T21:50:20.146 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[51620]: debug 2022-01-31T21:50:20.054+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.055540+0000) 2022-01-31T21:50:20.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:20 smithi167 conmon[49112]: debug 2022-01-31T21:50:20.054+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.055711+0000) 2022-01-31T21:50:20.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:20 smithi167 conmon[54076]: debug 2022-01-31T21:50:20.053+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.054720+0000) 2022-01-31T21:50:20.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:20 smithi167 conmon[60316]: debug 2022-01-31T21:50:20.053+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.054880+0000) 2022-01-31T21:50:20.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[41853]: debug 2022-01-31T21:50:20.145+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.145824+0000) 2022-01-31T21:50:20.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:20 smithi171 conmon[46715]: debug 2022-01-31T21:50:20.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.191131+0000) 2022-01-31T21:50:20.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:20 smithi167 conmon[54076]: debug 2022-01-31T21:50:20.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.427979+0000) 2022-01-31T21:50:20.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:20 smithi167 conmon[60316]: debug 2022-01-31T21:50:20.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.427982+0000) 2022-01-31T21:50:21.143 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:21 smithi171 conmon[51620]: debug 2022-01-31T21:50:21.033+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:21.034361+0000) 2022-01-31T21:50:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:20 smithi167 conmon[49112]: debug 2022-01-31T21:50:20.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:20.763543+0000) 2022-01-31T21:50:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:21 smithi171 conmon[41853]: debug 2022-01-31T21:50:21.145+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:21.146014+0000) 2022-01-31T21:50:21.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:21 smithi171 conmon[46715]: debug 2022-01-31T21:50:21.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:21.191285+0000) 2022-01-31T21:50:21.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:21 smithi167 conmon[54076]: debug 2022-01-31T21:50:21.426+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:21.428154+0000) 2022-01-31T21:50:21.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:21 smithi167 conmon[60316]: debug 2022-01-31T21:50:21.426+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:21.428181+0000) 2022-01-31T21:50:22.144 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:22 smithi171 conmon[51620]: debug 2022-01-31T21:50:22.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:22.034558+0000) 2022-01-31T21:50:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:21 smithi167 conmon[49112]: debug 2022-01-31T21:50:21.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:21.763682+0000) 2022-01-31T21:50:22.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:22 smithi171 conmon[46715]: debug 2022-01-31T21:50:22.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:22.191456+0000) 2022-01-31T21:50:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:22 smithi171 conmon[41853]: debug 2022-01-31T21:50:22.145+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:22.146198+0000) 2022-01-31T21:50:22.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:22 smithi167 conmon[54076]: debug 2022-01-31T21:50:22.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:22.428332+0000) 2022-01-31T21:50:22.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:22 smithi167 conmon[60316]: debug 2022-01-31T21:50:22.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:22.428346+0000) 2022-01-31T21:50:23.144 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:23 smithi171 conmon[51620]: debug 2022-01-31T21:50:23.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:23.034748+0000) 2022-01-31T21:50:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:22 smithi167 conmon[49112]: debug 2022-01-31T21:50:22.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:22.763841+0000) 2022-01-31T21:50:23.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:23 smithi171 conmon[41853]: debug 2022-01-31T21:50:23.145+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:23.146356+0000) 2022-01-31T21:50:23.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:23 smithi171 conmon[46715]: debug 2022-01-31T21:50:23.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:23.191605+0000) 2022-01-31T21:50:23.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:23 smithi167 conmon[54076]: debug 2022-01-31T21:50:23.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:23.428538+0000) 2022-01-31T21:50:23.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:23 smithi167 conmon[60316]: debug 2022-01-31T21:50:23.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:23.428545+0000) 2022-01-31T21:50:24.032 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:23 smithi171 conmon[35325]: debug 2022-01-31T21:50:23.748+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:50:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:23 smithi167 conmon[49112]: debug 2022-01-31T21:50:23.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:23.764049+0000) 2022-01-31T21:50:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:24 smithi171 conmon[41853]: debug 2022-01-31T21:50:24.146+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:24.146535+0000) 2022-01-31T21:50:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:24 smithi171 conmon[46715]: debug 2022-01-31T21:50:24.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:24.191782+0000) 2022-01-31T21:50:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:24 smithi171 conmon[51620]: debug 2022-01-31T21:50:24.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:24.034972+0000) 2022-01-31T21:50:24.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:24 smithi167 conmon[54076]: debug 2022-01-31T21:50:24.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:24.428762+0000) 2022-01-31T21:50:24.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:24 smithi167 conmon[60316]: debug 2022-01-31T21:50:24.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:24.428764+0000) 2022-01-31T21:50:25.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:24 smithi167 conmon[49112]: debug 2022-01-31T21:50:24.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:24.764265+0000) 2022-01-31T21:50:25.144 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[35325]: debug 2022-01-31T21:50:25.097+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161681 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:25.145 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[41853]: debug 2022-01-31T21:50:25.080+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.081351+0000) 2022-01-31T21:50:25.146 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[46715]: debug 2022-01-31T21:50:25.080+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.081016+0000) 2022-01-31T21:50:25.146 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[51620]: debug 2022-01-31T21:50:25.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.035166+0000) 2022-01-31T21:50:25.147 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[51620]: debug 2022-01-31T21:50:25.080+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.081575+0000) 2022-01-31T21:50:25.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:25 smithi167 conmon[49112]: debug 2022-01-31T21:50:25.079+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.081254+0000) 2022-01-31T21:50:25.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:25 smithi167 conmon[54076]: debug 2022-01-31T21:50:25.078+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.080099+0000) 2022-01-31T21:50:25.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:25 smithi167 conmon[60316]: debug 2022-01-31T21:50:25.079+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.080603+0000) 2022-01-31T21:50:25.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[41853]: debug 2022-01-31T21:50:25.146+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.146736+0000) 2022-01-31T21:50:25.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:25 smithi171 conmon[46715]: debug 2022-01-31T21:50:25.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.191937+0000) 2022-01-31T21:50:25.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:25 smithi167 conmon[54076]: debug 2022-01-31T21:50:25.427+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.428962+0000) 2022-01-31T21:50:25.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:25 smithi167 conmon[60316]: debug 2022-01-31T21:50:25.427+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.428963+0000) 2022-01-31T21:50:26.144 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:26 smithi171 conmon[51620]: debug 2022-01-31T21:50:26.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:26.035353+0000) 2022-01-31T21:50:26.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:25 smithi167 conmon[49112]: debug 2022-01-31T21:50:25.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:25.764409+0000) 2022-01-31T21:50:26.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:26 smithi171 conmon[46715]: debug 2022-01-31T21:50:26.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:26.192070+0000) 2022-01-31T21:50:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:26 smithi171 conmon[41853]: debug 2022-01-31T21:50:26.146+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:26.146877+0000) 2022-01-31T21:50:26.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:26 smithi167 conmon[54076]: debug 2022-01-31T21:50:26.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:26.429124+0000) 2022-01-31T21:50:26.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:26 smithi167 conmon[60316]: debug 2022-01-31T21:50:26.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:26.429120+0000) 2022-01-31T21:50:27.144 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:27 smithi171 conmon[51620]: debug 2022-01-31T21:50:27.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:27.035538+0000) 2022-01-31T21:50:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:26 smithi167 conmon[49112]: debug 2022-01-31T21:50:26.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:26.764566+0000) 2022-01-31T21:50:27.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:27 smithi171 conmon[46715]: debug 2022-01-31T21:50:27.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:27.192268+0000) 2022-01-31T21:50:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:27 smithi171 conmon[41853]: debug 2022-01-31T21:50:27.146+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:27.147063+0000) 2022-01-31T21:50:27.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:27 smithi167 conmon[54076]: debug 2022-01-31T21:50:27.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:27.429311+0000) 2022-01-31T21:50:27.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:27 smithi167 conmon[60316]: debug 2022-01-31T21:50:27.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:27.429300+0000) 2022-01-31T21:50:28.145 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:28 smithi171 conmon[51620]: debug 2022-01-31T21:50:28.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:28.035726+0000) 2022-01-31T21:50:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:27 smithi167 conmon[49112]: debug 2022-01-31T21:50:27.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:27.764719+0000) 2022-01-31T21:50:28.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:28 smithi171 conmon[46715]: debug 2022-01-31T21:50:28.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:28.192449+0000) 2022-01-31T21:50:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:28 smithi171 conmon[41853]: debug 2022-01-31T21:50:28.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:28.147246+0000) 2022-01-31T21:50:28.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:28 smithi167 conmon[54076]: debug 2022-01-31T21:50:28.428+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:28.429478+0000) 2022-01-31T21:50:28.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:28 smithi167 conmon[60316]: debug 2022-01-31T21:50:28.428+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:28.429478+0000) 2022-01-31T21:50:29.145 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:29 smithi171 conmon[51620]: debug 2022-01-31T21:50:29.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:29.035899+0000) 2022-01-31T21:50:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:28 smithi167 conmon[49112]: debug 2022-01-31T21:50:28.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:28.764948+0000) 2022-01-31T21:50:29.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:29 smithi171 conmon[41853]: debug 2022-01-31T21:50:29.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:29.147433+0000) 2022-01-31T21:50:29.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:29 smithi171 conmon[46715]: debug 2022-01-31T21:50:29.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:29.192686+0000) 2022-01-31T21:50:29.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:29 smithi167 conmon[54076]: debug 2022-01-31T21:50:29.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:29.429710+0000) 2022-01-31T21:50:29.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:29 smithi167 conmon[60316]: debug 2022-01-31T21:50:29.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:29.429695+0000) 2022-01-31T21:50:30.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:29 smithi167 conmon[49112]: debug 2022-01-31T21:50:29.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:29.765197+0000) 2022-01-31T21:50:30.116 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[41853]: debug 2022-01-31T21:50:30.100+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.100955+0000) 2022-01-31T21:50:30.116 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[46715]: debug 2022-01-31T21:50:30.101+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.102149+0000) 2022-01-31T21:50:30.117 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[51620]: debug 2022-01-31T21:50:30.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.036075+0000) 2022-01-31T21:50:30.117 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[51620]: debug 2022-01-31T21:50:30.099+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.100539+0000) 2022-01-31T21:50:30.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:30 smithi167 conmon[49112]: debug 2022-01-31T21:50:30.101+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.101908+0000) 2022-01-31T21:50:30.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:30 smithi167 conmon[54076]: debug 2022-01-31T21:50:30.100+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.101367+0000) 2022-01-31T21:50:30.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:30 smithi167 conmon[60316]: debug 2022-01-31T21:50:30.101+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.101701+0000) 2022-01-31T21:50:30.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[35325]: debug 2022-01-31T21:50:30.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161791 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[41853]: debug 2022-01-31T21:50:30.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.147640+0000) 2022-01-31T21:50:30.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:30 smithi171 conmon[46715]: debug 2022-01-31T21:50:30.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.192901+0000) 2022-01-31T21:50:30.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:30 smithi167 conmon[54076]: debug 2022-01-31T21:50:30.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.429826+0000) 2022-01-31T21:50:30.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:30 smithi167 conmon[60316]: debug 2022-01-31T21:50:30.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.429784+0000) 2022-01-31T21:50:31.145 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:31 smithi171 conmon[51620]: debug 2022-01-31T21:50:31.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:31.036229+0000) 2022-01-31T21:50:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:30 smithi167 conmon[49112]: debug 2022-01-31T21:50:30.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:30.765341+0000) 2022-01-31T21:50:31.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:31 smithi171 conmon[41853]: debug 2022-01-31T21:50:31.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:31.147823+0000) 2022-01-31T21:50:31.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:31 smithi171 conmon[46715]: debug 2022-01-31T21:50:31.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:31.193059+0000) 2022-01-31T21:50:31.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:31 smithi167 conmon[54076]: debug 2022-01-31T21:50:31.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:31.429951+0000) 2022-01-31T21:50:31.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:31 smithi167 conmon[60316]: debug 2022-01-31T21:50:31.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:31.429930+0000) 2022-01-31T21:50:32.145 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:32 smithi171 conmon[51620]: debug 2022-01-31T21:50:32.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:32.036383+0000) 2022-01-31T21:50:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:31 smithi167 conmon[49112]: debug 2022-01-31T21:50:31.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:31.765533+0000) 2022-01-31T21:50:32.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:32 smithi171 conmon[41853]: debug 2022-01-31T21:50:32.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:32.148012+0000) 2022-01-31T21:50:32.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:32 smithi171 conmon[46715]: debug 2022-01-31T21:50:32.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:32.193202+0000) 2022-01-31T21:50:32.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:32 smithi167 conmon[54076]: debug 2022-01-31T21:50:32.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:32.430140+0000) 2022-01-31T21:50:32.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:32 smithi167 conmon[60316]: debug 2022-01-31T21:50:32.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:32.430146+0000) 2022-01-31T21:50:33.145 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:33 smithi171 conmon[51620]: debug 2022-01-31T21:50:33.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:33.036562+0000) 2022-01-31T21:50:33.146 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:33 smithi171 conmon[41853]: debug 2022-01-31T21:50:33.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:33.148171+0000) 2022-01-31T21:50:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:32 smithi167 conmon[49112]: debug 2022-01-31T21:50:32.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:32.765721+0000) 2022-01-31T21:50:33.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:33 smithi171 conmon[46715]: debug 2022-01-31T21:50:33.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:33.193397+0000) 2022-01-31T21:50:33.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:33 smithi167 conmon[54076]: debug 2022-01-31T21:50:33.429+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:33.430320+0000) 2022-01-31T21:50:33.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:33 smithi167 conmon[60316]: debug 2022-01-31T21:50:33.429+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:33.430334+0000) 2022-01-31T21:50:34.146 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:34 smithi171 conmon[51620]: debug 2022-01-31T21:50:34.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:34.036735+0000) 2022-01-31T21:50:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:33 smithi167 conmon[49112]: debug 2022-01-31T21:50:33.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:33.765916+0000) 2022-01-31T21:50:34.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:34 smithi171 conmon[41853]: debug 2022-01-31T21:50:34.148+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:34.148354+0000) 2022-01-31T21:50:34.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:34 smithi171 conmon[46715]: debug 2022-01-31T21:50:34.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:34.193588+0000) 2022-01-31T21:50:34.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:34 smithi167 conmon[54076]: debug 2022-01-31T21:50:34.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:34.430434+0000) 2022-01-31T21:50:34.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:34 smithi167 conmon[60316]: debug 2022-01-31T21:50:34.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:34.430521+0000) 2022-01-31T21:50:35.118 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[51620]: debug 2022-01-31T21:50:35.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.036905+0000) 2022-01-31T21:50:35.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:34 smithi167 conmon[49112]: debug 2022-01-31T21:50:34.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:34.766119+0000) 2022-01-31T21:50:35.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:35 smithi167 conmon[49112]: debug 2022-01-31T21:50:35.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.119897+0000) 2022-01-31T21:50:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:35 smithi167 conmon[54076]: debug 2022-01-31T21:50:35.120+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.121170+0000) 2022-01-31T21:50:35.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:35 smithi167 conmon[60316]: debug 2022-01-31T21:50:35.120+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.121336+0000) 2022-01-31T21:50:35.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[35325]: debug 2022-01-31T21:50:35.136+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 161902 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:35.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[51620]: debug 2022-01-31T21:50:35.119+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.121512+0000) 2022-01-31T21:50:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[41853]: debug 2022-01-31T21:50:35.118+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.120570+0000) 2022-01-31T21:50:35.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[41853]: debug 2022-01-31T21:50:35.146+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.148526+0000) 2022-01-31T21:50:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[46715]: debug 2022-01-31T21:50:35.120+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.122134+0000) 2022-01-31T21:50:35.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:35 smithi171 conmon[46715]: debug 2022-01-31T21:50:35.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.193742+0000) 2022-01-31T21:50:35.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:35 smithi167 conmon[54076]: debug 2022-01-31T21:50:35.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.430616+0000) 2022-01-31T21:50:35.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:35 smithi167 conmon[60316]: debug 2022-01-31T21:50:35.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.430643+0000) 2022-01-31T21:50:36.146 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:36 smithi171 conmon[51620]: debug 2022-01-31T21:50:36.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:36.037112+0000) 2022-01-31T21:50:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:35 smithi167 conmon[49112]: debug 2022-01-31T21:50:35.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:35.766291+0000) 2022-01-31T21:50:36.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:36 smithi171 conmon[46715]: debug 2022-01-31T21:50:36.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:36.193901+0000) 2022-01-31T21:50:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:36 smithi171 conmon[41853]: debug 2022-01-31T21:50:36.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:36.148666+0000) 2022-01-31T21:50:36.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:36 smithi167 conmon[54076]: debug 2022-01-31T21:50:36.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:36.430755+0000) 2022-01-31T21:50:36.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:36 smithi167 conmon[60316]: debug 2022-01-31T21:50:36.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:36.430751+0000) 2022-01-31T21:50:37.146 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:37 smithi171 conmon[51620]: debug 2022-01-31T21:50:37.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:37.037250+0000) 2022-01-31T21:50:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:36 smithi167 conmon[49112]: debug 2022-01-31T21:50:36.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:36.766448+0000) 2022-01-31T21:50:37.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:37 smithi171 conmon[46715]: debug 2022-01-31T21:50:37.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:37.194104+0000) 2022-01-31T21:50:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:37 smithi171 conmon[41853]: debug 2022-01-31T21:50:37.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:37.148859+0000) 2022-01-31T21:50:37.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:37 smithi167 conmon[54076]: debug 2022-01-31T21:50:37.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:37.430941+0000) 2022-01-31T21:50:37.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:37 smithi167 conmon[60316]: debug 2022-01-31T21:50:37.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:37.430939+0000) 2022-01-31T21:50:38.146 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:38 smithi171 conmon[51620]: debug 2022-01-31T21:50:38.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:38.037440+0000) 2022-01-31T21:50:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:37 smithi167 conmon[49112]: debug 2022-01-31T21:50:37.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:37.766675+0000) 2022-01-31T21:50:38.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:38 smithi171 conmon[41853]: debug 2022-01-31T21:50:38.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:38.149045+0000) 2022-01-31T21:50:38.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:38 smithi171 conmon[46715]: debug 2022-01-31T21:50:38.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:38.194256+0000) 2022-01-31T21:50:38.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:38 smithi167 conmon[54076]: debug 2022-01-31T21:50:38.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:38.431132+0000) 2022-01-31T21:50:38.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:38 smithi167 conmon[60316]: debug 2022-01-31T21:50:38.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:38.431132+0000) 2022-01-31T21:50:39.035 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:38 smithi171 conmon[35325]: debug 2022-01-31T21:50:38.748+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:50:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:38 smithi167 conmon[49112]: debug 2022-01-31T21:50:38.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:38.766843+0000) 2022-01-31T21:50:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:39 smithi171 conmon[41853]: debug 2022-01-31T21:50:39.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:39.149281+0000) 2022-01-31T21:50:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:39 smithi171 conmon[46715]: debug 2022-01-31T21:50:39.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:39.194427+0000) 2022-01-31T21:50:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:39 smithi171 conmon[51620]: debug 2022-01-31T21:50:39.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:39.037633+0000) 2022-01-31T21:50:39.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:39 smithi167 conmon[54076]: debug 2022-01-31T21:50:39.430+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:39.431325+0000) 2022-01-31T21:50:39.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:39 smithi167 conmon[60316]: debug 2022-01-31T21:50:39.430+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:39.431320+0000) 2022-01-31T21:50:40.137 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[51620]: debug 2022-01-31T21:50:40.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.037787+0000) 2022-01-31T21:50:40.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:39 smithi167 conmon[49112]: debug 2022-01-31T21:50:39.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:39.767029+0000) 2022-01-31T21:50:40.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[49112]: debug 2022-01-31T21:50:40.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.140114+0000) 2022-01-31T21:50:40.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[54076]: debug 2022-01-31T21:50:40.139+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.140402+0000) 2022-01-31T21:50:40.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[60316]: debug 2022-01-31T21:50:40.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[60316]: 2022-01-31T21:50:40.140+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.140494+0000) 2022-01-31T21:50:40.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[35325]: debug 2022-01-31T21:50:40.155+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162013 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[41853]: debug 2022-01-31T21:50:40.137+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.139475+0000) 2022-01-31T21:50:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[41853]: debug 2022-01-31T21:50:40.147+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.149447+0000) 2022-01-31T21:50:40.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[46715]: debug 2022-01-31T21:50:40.139+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.141167+0000) 2022-01-31T21:50:40.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[46715]: debug 2022-01-31T21:50:40.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.194602+0000) 2022-01-31T21:50:40.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:40 smithi171 conmon[51620]: debug 2022-01-31T21:50:40.139+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.140830+0000) 2022-01-31T21:50:40.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[54076]: debug 2022-01-31T21:50:40.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.431540+0000) 2022-01-31T21:50:40.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[60316]: debug 2022-01-31T21:50:40.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.431536+0000) 2022-01-31T21:50:41.147 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:41 smithi171 conmon[51620]: debug 2022-01-31T21:50:41.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:41.037976+0000) 2022-01-31T21:50:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:40 smithi167 conmon[49112]: debug 2022-01-31T21:50:40.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:40.767198+0000) 2022-01-31T21:50:41.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:41 smithi171 conmon[46715]: debug 2022-01-31T21:50:41.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:41.194767+0000) 2022-01-31T21:50:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:41 smithi171 conmon[41853]: debug 2022-01-31T21:50:41.148+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:41.149601+0000) 2022-01-31T21:50:41.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:41 smithi167 conmon[54076]: debug 2022-01-31T21:50:41.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:41.431695+0000) 2022-01-31T21:50:41.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:41 smithi167 conmon[60316]: debug 2022-01-31T21:50:41.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:41.431720+0000) 2022-01-31T21:50:42.147 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:42 smithi171 conmon[51620]: debug 2022-01-31T21:50:42.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:42.038117+0000) 2022-01-31T21:50:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:41 smithi167 conmon[49112]: debug 2022-01-31T21:50:41.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:41.767370+0000) 2022-01-31T21:50:42.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:42 smithi171 conmon[46715]: debug 2022-01-31T21:50:42.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:42.194965+0000) 2022-01-31T21:50:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:42 smithi171 conmon[41853]: debug 2022-01-31T21:50:42.148+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:42.149810+0000) 2022-01-31T21:50:42.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:42 smithi167 conmon[54076]: debug 2022-01-31T21:50:42.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:42.431878+0000) 2022-01-31T21:50:42.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:42 smithi167 conmon[60316]: debug 2022-01-31T21:50:42.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:42.431905+0000) 2022-01-31T21:50:43.147 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:43 smithi171 conmon[51620]: debug 2022-01-31T21:50:43.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:43.038276+0000) 2022-01-31T21:50:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:42 smithi167 conmon[49112]: debug 2022-01-31T21:50:42.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:42.767540+0000) 2022-01-31T21:50:43.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:43 smithi171 conmon[46715]: debug 2022-01-31T21:50:43.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:43.195192+0000) 2022-01-31T21:50:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:43 smithi171 conmon[41853]: debug 2022-01-31T21:50:43.148+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:43.149981+0000) 2022-01-31T21:50:43.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:43 smithi167 conmon[54076]: debug 2022-01-31T21:50:43.431+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:43.432093+0000) 2022-01-31T21:50:43.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:43 smithi167 conmon[60316]: debug 2022-01-31T21:50:43.431+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:43.432097+0000) 2022-01-31T21:50:44.148 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:44 smithi171 conmon[51620]: debug 2022-01-31T21:50:44.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:44.038461+0000) 2022-01-31T21:50:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:43 smithi167 conmon[49112]: debug 2022-01-31T21:50:43.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:43.767690+0000) 2022-01-31T21:50:44.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:44 smithi171 conmon[46715]: debug 2022-01-31T21:50:44.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:44.195400+0000) 2022-01-31T21:50:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:44 smithi171 conmon[41853]: debug 2022-01-31T21:50:44.149+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:44.150164+0000) 2022-01-31T21:50:44.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:44 smithi167 conmon[54076]: debug 2022-01-31T21:50:44.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:44.432306+0000) 2022-01-31T21:50:44.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:44 smithi167 conmon[60316]: debug 2022-01-31T21:50:44.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:44.432303+0000) 2022-01-31T21:50:45.148 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[51620]: debug 2022-01-31T21:50:45.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.038662+0000) 2022-01-31T21:50:45.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:44 smithi167 conmon[49112]: debug 2022-01-31T21:50:44.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:44.767885+0000) 2022-01-31T21:50:45.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:45 smithi167 conmon[49112]: debug 2022-01-31T21:50:45.157+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.158042+0000) 2022-01-31T21:50:45.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:45 smithi167 conmon[54076]: debug 2022-01-31T21:50:45.158+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.159404+0000) 2022-01-31T21:50:45.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:45 smithi167 conmon[60316]: debug 2022-01-31T21:50:45.159+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.159758+0000) 2022-01-31T21:50:45.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[35325]: debug 2022-01-31T21:50:45.175+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162124 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:45.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[46715]: debug 2022-01-31T21:50:45.158+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.160565+0000) 2022-01-31T21:50:45.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[46715]: debug 2022-01-31T21:50:45.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.195613+0000) 2022-01-31T21:50:45.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[51620]: debug 2022-01-31T21:50:45.158+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.160417+0000) 2022-01-31T21:50:45.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[41853]: debug 2022-01-31T21:50:45.149+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.150363+0000) 2022-01-31T21:50:45.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:45 smithi171 conmon[41853]: debug 2022-01-31T21:50:45.157+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.159578+0000) 2022-01-31T21:50:45.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:45 smithi167 conmon[54076]: debug 2022-01-31T21:50:45.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.432536+0000) 2022-01-31T21:50:45.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:45 smithi167 conmon[60316]: debug 2022-01-31T21:50:45.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.432533+0000) 2022-01-31T21:50:46.148 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:46 smithi171 conmon[51620]: debug 2022-01-31T21:50:46.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:46.038829+0000) 2022-01-31T21:50:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:45 smithi167 conmon[49112]: debug 2022-01-31T21:50:45.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:45.768057+0000) 2022-01-31T21:50:46.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:46 smithi171 conmon[41853]: debug 2022-01-31T21:50:46.149+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:46.150508+0000) 2022-01-31T21:50:46.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:46 smithi171 conmon[46715]: debug 2022-01-31T21:50:46.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:46.195803+0000) 2022-01-31T21:50:46.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:46 smithi167 conmon[54076]: debug 2022-01-31T21:50:46.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:46.432692+0000) 2022-01-31T21:50:46.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:46 smithi167 conmon[60316]: debug 2022-01-31T21:50:46.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:46.432692+0000) 2022-01-31T21:50:47.148 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:47 smithi171 conmon[51620]: debug 2022-01-31T21:50:47.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:47.038996+0000) 2022-01-31T21:50:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:46 smithi167 conmon[49112]: debug 2022-01-31T21:50:46.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:46.768254+0000) 2022-01-31T21:50:47.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:47 smithi171 conmon[41853]: debug 2022-01-31T21:50:47.149+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:47.150646+0000) 2022-01-31T21:50:47.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:47 smithi171 conmon[46715]: debug 2022-01-31T21:50:47.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:47.195932+0000) 2022-01-31T21:50:47.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:47 smithi167 conmon[54076]: debug 2022-01-31T21:50:47.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:47.432904+0000) 2022-01-31T21:50:47.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:47 smithi167 conmon[60316]: debug 2022-01-31T21:50:47.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:47.432897+0000) 2022-01-31T21:50:48.148 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:48 smithi171 conmon[51620]: debug 2022-01-31T21:50:48.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:48.039135+0000) 2022-01-31T21:50:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:47 smithi167 conmon[49112]: debug 2022-01-31T21:50:47.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:47.768459+0000) 2022-01-31T21:50:48.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:48 smithi171 conmon[41853]: debug 2022-01-31T21:50:48.150+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:48.150831+0000) 2022-01-31T21:50:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:48 smithi171 conmon[46715]: debug 2022-01-31T21:50:48.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:48.196086+0000) 2022-01-31T21:50:48.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:48 smithi167 conmon[60316]: debug 2022-01-31T21:50:48.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:48.433096+0000) 2022-01-31T21:50:48.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:48 smithi167 conmon[54076]: debug 2022-01-31T21:50:48.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:48.433100+0000) 2022-01-31T21:50:49.148 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:49 smithi171 conmon[51620]: debug 2022-01-31T21:50:49.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:49.039374+0000) 2022-01-31T21:50:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:48 smithi167 conmon[49112]: debug 2022-01-31T21:50:48.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:48.768670+0000) 2022-01-31T21:50:49.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:49 smithi171 conmon[41853]: debug 2022-01-31T21:50:49.150+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:49.151060+0000) 2022-01-31T21:50:49.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:49 smithi171 conmon[46715]: debug 2022-01-31T21:50:49.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:49.196267+0000) 2022-01-31T21:50:49.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:49 smithi167 conmon[54076]: debug 2022-01-31T21:50:49.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:49.433300+0000) 2022-01-31T21:50:49.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:49 smithi167 conmon[60316]: debug 2022-01-31T21:50:49.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:49.433296+0000) 2022-01-31T21:50:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:49 smithi167 conmon[49112]: debug 2022-01-31T21:50:49.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:49.768848+0000) 2022-01-31T21:50:50.355 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[35325]: debug 2022-01-31T21:50:50.195+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162234 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:50.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[41853]: debug 2022-01-31T21:50:50.150+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.151245+0000) 2022-01-31T21:50:50.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[41853]: debug 2022-01-31T21:50:50.178+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.179074+0000) 2022-01-31T21:50:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[46715]: debug 2022-01-31T21:50:50.179+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.180158+0000) 2022-01-31T21:50:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[46715]: debug 2022-01-31T21:50:50.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.196371+0000) 2022-01-31T21:50:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[51620]: debug 2022-01-31T21:50:50.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.039535+0000) 2022-01-31T21:50:50.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:50 smithi171 conmon[51620]: debug 2022-01-31T21:50:50.178+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.179667+0000) 2022-01-31T21:50:50.432 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:50 smithi167 conmon[49112]: debug 2022-01-31T21:50:50.178+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.178929+0000) 2022-01-31T21:50:50.433 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:50 smithi167 conmon[54076]: debug 2022-01-31T21:50:50.178+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.179392+0000) 2022-01-31T21:50:50.433 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:50 smithi167 conmon[60316]: debug 2022-01-31T21:50:50.179+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.179813+0000) 2022-01-31T21:50:50.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:50 smithi167 conmon[54076]: debug 2022-01-31T21:50:50.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.433535+0000) 2022-01-31T21:50:50.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:50 smithi167 conmon[60316]: debug 2022-01-31T21:50:50.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.433490+0000) 2022-01-31T21:50:51.149 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:51 smithi171 conmon[51620]: debug 2022-01-31T21:50:51.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:51.039720+0000) 2022-01-31T21:50:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:50 smithi167 conmon[49112]: debug 2022-01-31T21:50:50.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:50.769018+0000) 2022-01-31T21:50:51.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:51 smithi171 conmon[41853]: debug 2022-01-31T21:50:51.150+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:51.151467+0000) 2022-01-31T21:50:51.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:51 smithi171 conmon[46715]: debug 2022-01-31T21:50:51.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:51.196510+0000) 2022-01-31T21:50:51.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:51 smithi167 conmon[60316]: debug 2022-01-31T21:50:51.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:51.433675+0000) 2022-01-31T21:50:51.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:51 smithi167 conmon[54076]: debug 2022-01-31T21:50:51.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:51 smithi167 conmon[54076]: 2022-01-31T21:50:51.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:51.433662+0000) 2022-01-31T21:50:51.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:51 smithi167 conmon[54076]: 2022-01-31T21:50:52.149 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:52 smithi171 conmon[51620]: debug 2022-01-31T21:50:52.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:52.039834+0000) 2022-01-31T21:50:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:51 smithi167 conmon[49112]: debug 2022-01-31T21:50:51.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:51.769198+0000) 2022-01-31T21:50:52.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:52 smithi167 conmon[54076]: debug 2022-01-31T21:50:52.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:52.433843+0000) 2022-01-31T21:50:52.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:52 smithi167 conmon[60316]: debug 2022-01-31T21:50:52.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:52.433844+0000) 2022-01-31T21:50:52.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:52 smithi171 conmon[41853]: debug 2022-01-31T21:50:52.150+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:52.151638+0000) 2022-01-31T21:50:52.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:52 smithi171 conmon[46715]: debug 2022-01-31T21:50:52.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:52.196693+0000) 2022-01-31T21:50:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:52 smithi167 conmon[49112]: debug 2022-01-31T21:50:52.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:52.769401+0000) 2022-01-31T21:50:53.149 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:53 smithi171 conmon[51620]: debug 2022-01-31T21:50:53.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:53.040012+0000) 2022-01-31T21:50:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:53 smithi171 conmon[41853]: debug 2022-01-31T21:50:53.151+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:53.151774+0000) 2022-01-31T21:50:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:53 smithi171 conmon[46715]: debug 2022-01-31T21:50:53.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:53.196934+0000) 2022-01-31T21:50:53.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:53 smithi167 conmon[54076]: debug 2022-01-31T21:50:53.432+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:53.434022+0000) 2022-01-31T21:50:53.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:53 smithi167 conmon[60316]: debug 2022-01-31T21:50:53.432+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:53.434063+0000) 2022-01-31T21:50:54.037 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:53 smithi171 conmon[35325]: debug 2022-01-31T21:50:53.749+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:50:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:53 smithi167 conmon[49112]: debug 2022-01-31T21:50:53.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:53.769555+0000) 2022-01-31T21:50:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:54 smithi171 conmon[41853]: debug 2022-01-31T21:50:54.151+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:54.151920+0000) 2022-01-31T21:50:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:54 smithi171 conmon[46715]: debug 2022-01-31T21:50:54.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:54.197117+0000) 2022-01-31T21:50:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:54 smithi171 conmon[51620]: debug 2022-01-31T21:50:54.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:54.040209+0000) 2022-01-31T21:50:54.641 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:54 smithi167 conmon[54076]: debug 2022-01-31T21:50:54.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:54.434223+0000) 2022-01-31T21:50:54.642 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:54 smithi167 conmon[60316]: debug 2022-01-31T21:50:54.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:54.434245+0000) 2022-01-31T21:50:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:54 smithi167 conmon[49112]: debug 2022-01-31T21:50:54.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:54.769721+0000) 2022-01-31T21:50:55.149 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:55 smithi171 conmon[51620]: debug 2022-01-31T21:50:55.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.040431+0000) 2022-01-31T21:50:55.490 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:55 smithi167 conmon[49112]: debug 2022-01-31T21:50:55.196+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.198244+0000) 2022-01-31T21:50:55.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:55 smithi167 conmon[54076]: debug 2022-01-31T21:50:55.196+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.198143+0000) 2022-01-31T21:50:55.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:55 smithi167 conmon[54076]: debug 2022-01-31T21:50:55.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.434420+0000) 2022-01-31T21:50:55.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:55 smithi167 conmon[60316]: debug 2022-01-31T21:50:55.197+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.198842+0000) 2022-01-31T21:50:55.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:55 smithi167 conmon[60316]: debug 2022-01-31T21:50:55.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.434419+0000) 2022-01-31T21:50:55.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:50:55 smithi171 conmon[35325]: debug 2022-01-31T21:50:55.214+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162345 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:50:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:55 smithi171 conmon[41853]: debug 2022-01-31T21:50:55.151+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.152099+0000) 2022-01-31T21:50:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:55 smithi171 conmon[41853]: debug 2022-01-31T21:50:55.198+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.199037+0000) 2022-01-31T21:50:55.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:50:55 smithi171 conmon[46715]: debug 2022-01-31T21:50:55.199+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.199775+0000) 2022-01-31T21:50:55.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:55 smithi171 conmon[51620]: debug 2022-01-31T21:50:55.197+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.198376+0000) 2022-01-31T21:50:56.150 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:56 smithi171 conmon[51620]: debug 2022-01-31T21:50:56.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:56.040630+0000) 2022-01-31T21:50:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:55 smithi167 conmon[49112]: debug 2022-01-31T21:50:55.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:55.769893+0000) 2022-01-31T21:50:56.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:56 smithi171 conmon[41853]: debug 2022-01-31T21:50:56.151+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:56.152282+0000) 2022-01-31T21:50:56.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:56 smithi167 conmon[60316]: debug 2022-01-31T21:50:56.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:56.434585+0000) 2022-01-31T21:50:56.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:56 smithi167 conmon[54076]: debug 2022-01-31T21:50:56.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:56.434608+0000) 2022-01-31T21:50:57.150 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:57 smithi171 conmon[51620]: debug 2022-01-31T21:50:57.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:57.040778+0000) 2022-01-31T21:50:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:56 smithi167 conmon[49112]: debug 2022-01-31T21:50:56.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:56.770069+0000) 2022-01-31T21:50:57.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:57 smithi171 conmon[41853]: debug 2022-01-31T21:50:57.151+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:57.152439+0000) 2022-01-31T21:50:57.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:57 smithi167 conmon[54076]: debug 2022-01-31T21:50:57.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:57.434807+0000) 2022-01-31T21:50:57.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:57 smithi167 conmon[60316]: debug 2022-01-31T21:50:57.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:57.434780+0000) 2022-01-31T21:50:58.150 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:58 smithi171 conmon[51620]: debug 2022-01-31T21:50:58.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:58.040933+0000) 2022-01-31T21:50:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:57 smithi167 conmon[49112]: debug 2022-01-31T21:50:57.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:57.770284+0000) 2022-01-31T21:50:58.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:58 smithi167 conmon[54076]: debug 2022-01-31T21:50:58.433+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:58.435006+0000) 2022-01-31T21:50:58.602 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:58 smithi167 conmon[60316]: debug 2022-01-31T21:50:58.433+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:58.434973+0000) 2022-01-31T21:50:58.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:58 smithi171 conmon[41853]: debug 2022-01-31T21:50:58.152+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:58.152572+0000) 2022-01-31T21:50:58.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:58 smithi167 conmon[49112]: debug 2022-01-31T21:50:58.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:58.770524+0000) 2022-01-31T21:50:59.150 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:50:59 smithi171 conmon[51620]: debug 2022-01-31T21:50:59.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:59.041130+0000) 2022-01-31T21:50:59.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:50:59 smithi171 conmon[41853]: debug 2022-01-31T21:50:59.152+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:59.152739+0000) 2022-01-31T21:50:59.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:50:59 smithi167 conmon[54076]: debug 2022-01-31T21:50:59.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:59.435205+0000) 2022-01-31T21:50:59.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:50:59 smithi167 conmon[60316]: debug 2022-01-31T21:50:59.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:59.435200+0000) 2022-01-31T21:51:00.150 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:00 smithi171 conmon[51620]: debug 2022-01-31T21:51:00.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.041321+0000) 2022-01-31T21:51:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:50:59 smithi167 conmon[49112]: debug 2022-01-31T21:50:59.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:50:59.770741+0000) 2022-01-31T21:51:00.491 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:00 smithi167 conmon[49112]: debug 2022-01-31T21:51:00.216+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.217485+0000) 2022-01-31T21:51:00.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:00 smithi167 conmon[54076]: debug 2022-01-31T21:51:00.216+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.218331+0000) 2022-01-31T21:51:00.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:00 smithi167 conmon[54076]: debug 2022-01-31T21:51:00.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.435434+0000) 2022-01-31T21:51:00.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:00 smithi167 conmon[60316]: debug 2022-01-31T21:51:00.215+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.216923+0000) 2022-01-31T21:51:00.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:00 smithi167 conmon[60316]: debug 2022-01-31T21:51:00.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.435390+0000) 2022-01-31T21:51:00.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:00 smithi171 conmon[35325]: debug 2022-01-31T21:51:00.233+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162456 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:00 smithi171 conmon[41853]: debug 2022-01-31T21:51:00.152+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.152985+0000) 2022-01-31T21:51:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:00 smithi171 conmon[41853]: debug 2022-01-31T21:51:00.217+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.218213+0000) 2022-01-31T21:51:00.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:00 smithi171 conmon[46715]: debug 2022-01-31T21:51:00.218+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.219039+0000) 2022-01-31T21:51:00.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:00 smithi171 conmon[51620]: debug 2022-01-31T21:51:00.217+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.218096+0000) 2022-01-31T21:51:01.151 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:01 smithi171 conmon[51620]: debug 2022-01-31T21:51:01.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:01.041507+0000) 2022-01-31T21:51:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:00 smithi167 conmon[49112]: debug 2022-01-31T21:51:00.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:00.770889+0000) 2022-01-31T21:51:01.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:01 smithi171 conmon[41853]: debug 2022-01-31T21:51:01.152+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:01.153186+0000) 2022-01-31T21:51:01.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:01 smithi167 conmon[60316]: debug 2022-01-31T21:51:01.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:01.435574+0000) 2022-01-31T21:51:01.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:01 smithi167 conmon[54076]: debug 2022-01-31T21:51:01.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:01.435580+0000) 2022-01-31T21:51:02.151 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:01 smithi171 conmon[46715]: debug 2022-01-31T21:51:01.947+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:01.947855+0000) 2022-01-31T21:51:02.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:02 smithi171 conmon[51620]: debug 2022-01-31T21:51:02.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:02.041647+0000) 2022-01-31T21:51:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:01 smithi167 conmon[49112]: debug 2022-01-31T21:51:01.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:01.771036+0000) 2022-01-31T21:51:02.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:02 smithi171 conmon[41853]: debug 2022-01-31T21:51:02.152+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:02.153343+0000) 2022-01-31T21:51:02.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:02 smithi167 conmon[60316]: debug 2022-01-31T21:51:02.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:02.435779+0000) 2022-01-31T21:51:02.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:02 smithi167 conmon[54076]: debug 2022-01-31T21:51:02.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:02.435782+0000) 2022-01-31T21:51:03.151 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:02 smithi171 conmon[46715]: debug 2022-01-31T21:51:02.947+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:02.948034+0000) 2022-01-31T21:51:03.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:03 smithi171 conmon[51620]: debug 2022-01-31T21:51:03.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:03.041835+0000) 2022-01-31T21:51:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:02 smithi167 conmon[49112]: debug 2022-01-31T21:51:02.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:02.771171+0000) 2022-01-31T21:51:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:03 smithi171 conmon[41853]: debug 2022-01-31T21:51:03.152+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:03.153501+0000) 2022-01-31T21:51:03.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:03 smithi167 conmon[54076]: debug 2022-01-31T21:51:03.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:03.435988+0000) 2022-01-31T21:51:03.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:03 smithi167 conmon[60316]: debug 2022-01-31T21:51:03.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:03.435989+0000) 2022-01-31T21:51:04.151 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:03 smithi171 conmon[46715]: debug 2022-01-31T21:51:03.947+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:03.948232+0000) 2022-01-31T21:51:04.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:04 smithi171 conmon[51620]: debug 2022-01-31T21:51:04.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:04.041956+0000) 2022-01-31T21:51:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:03 smithi167 conmon[49112]: debug 2022-01-31T21:51:03.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:03.771353+0000) 2022-01-31T21:51:04.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:04 smithi171 conmon[41853]: debug 2022-01-31T21:51:04.153+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:04.153607+0000) 2022-01-31T21:51:04.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:04 smithi167 conmon[54076]: debug 2022-01-31T21:51:04.434+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:04.436186+0000) 2022-01-31T21:51:04.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:04 smithi167 conmon[60316]: debug 2022-01-31T21:51:04.434+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:04.436186+0000) 2022-01-31T21:51:05.151 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:04 smithi171 conmon[46715]: debug 2022-01-31T21:51:04.948+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:04.948367+0000) 2022-01-31T21:51:05.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[51620]: debug 2022-01-31T21:51:05.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.042167+0000) 2022-01-31T21:51:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:04 smithi167 conmon[49112]: debug 2022-01-31T21:51:04.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:04.771632+0000) 2022-01-31T21:51:05.492 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:05 smithi167 conmon[49112]: debug 2022-01-31T21:51:05.234+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.236230+0000) 2022-01-31T21:51:05.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:05 smithi167 conmon[54076]: debug 2022-01-31T21:51:05.235+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.236702+0000) 2022-01-31T21:51:05.493 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:05 smithi167 conmon[54076]: debug 2022-01-31T21:51:05.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.436372+0000) 2022-01-31T21:51:05.493 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:05 smithi167 conmon[60316]: debug 2022-01-31T21:51:05.234+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.236102+0000) 2022-01-31T21:51:05.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:05 smithi167 conmon[60316]: debug 2022-01-31T21:51:05.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.436372+0000) 2022-01-31T21:51:05.535 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[35325]: debug 2022-01-31T21:51:05.254+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:05.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[51620]: debug 2022-01-31T21:51:05.236+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.237423+0000) 2022-01-31T21:51:05.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[41853]: debug 2022-01-31T21:51:05.153+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.153799+0000) 2022-01-31T21:51:05.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[41853]: debug 2022-01-31T21:51:05.236+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.237536+0000) 2022-01-31T21:51:05.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[46715]: debug 2022-01-31T21:51:05.237+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.237739+0000) 2022-01-31T21:51:06.151 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:05 smithi171 conmon[46715]: debug 2022-01-31T21:51:05.947+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.948479+0000) 2022-01-31T21:51:06.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:06 smithi171 conmon[51620]: debug 2022-01-31T21:51:06.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:06.042372+0000) 2022-01-31T21:51:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:05 smithi167 conmon[49112]: debug 2022-01-31T21:51:05.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:05.771781+0000) 2022-01-31T21:51:06.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:06 smithi171 conmon[41853]: debug 2022-01-31T21:51:06.153+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:06.153955+0000) 2022-01-31T21:51:06.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:06 smithi167 conmon[54076]: debug 2022-01-31T21:51:06.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:06.436535+0000) 2022-01-31T21:51:06.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:06 smithi167 conmon[60316]: debug 2022-01-31T21:51:06.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:06.436535+0000) 2022-01-31T21:51:07.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:06 smithi171 conmon[46715]: debug 2022-01-31T21:51:06.948+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:06.948636+0000) 2022-01-31T21:51:07.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:07 smithi171 conmon[51620]: debug 2022-01-31T21:51:07.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:07.042534+0000) 2022-01-31T21:51:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:06 smithi167 conmon[49112]: debug 2022-01-31T21:51:06.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:06.771926+0000) 2022-01-31T21:51:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:07 smithi171 conmon[41853]: debug 2022-01-31T21:51:07.153+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:07.154063+0000) 2022-01-31T21:51:07.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:07 smithi167 conmon[54076]: debug 2022-01-31T21:51:07.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:07.436699+0000) 2022-01-31T21:51:07.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:07 smithi167 conmon[60316]: debug 2022-01-31T21:51:07.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:07.436709+0000) 2022-01-31T21:51:08.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:07 smithi171 conmon[46715]: debug 2022-01-31T21:51:07.948+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:07.948827+0000) 2022-01-31T21:51:08.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:08 smithi171 conmon[51620]: debug 2022-01-31T21:51:08.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:08.042737+0000) 2022-01-31T21:51:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:07 smithi167 conmon[49112]: debug 2022-01-31T21:51:07.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:07.772137+0000) 2022-01-31T21:51:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:08 smithi171 conmon[41853]: debug 2022-01-31T21:51:08.153+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:08.154223+0000) 2022-01-31T21:51:08.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:08 smithi167 conmon[54076]: debug 2022-01-31T21:51:08.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:08.436900+0000) 2022-01-31T21:51:08.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:08 smithi167 conmon[60316]: debug 2022-01-31T21:51:08.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:08.436874+0000) 2022-01-31T21:51:09.040 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:08 smithi171 conmon[35325]: debug 2022-01-31T21:51:08.750+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:51:09.041 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:08 smithi171 conmon[46715]: debug 2022-01-31T21:51:08.948+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:08.948982+0000) 2022-01-31T21:51:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:08 smithi167 conmon[49112]: debug 2022-01-31T21:51:08.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:08.772309+0000) 2022-01-31T21:51:09.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:09 smithi171 conmon[41853]: debug 2022-01-31T21:51:09.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:09.154380+0000) 2022-01-31T21:51:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:09 smithi171 conmon[51620]: debug 2022-01-31T21:51:09.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:09.042888+0000) 2022-01-31T21:51:09.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:09 smithi167 conmon[54076]: debug 2022-01-31T21:51:09.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:09.437055+0000) 2022-01-31T21:51:09.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:09 smithi167 conmon[60316]: debug 2022-01-31T21:51:09.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:09.437055+0000) 2022-01-31T21:51:10.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:09 smithi171 conmon[46715]: debug 2022-01-31T21:51:09.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:09.949204+0000) 2022-01-31T21:51:10.153 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[51620]: debug 2022-01-31T21:51:10.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.043071+0000) 2022-01-31T21:51:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:09 smithi167 conmon[49112]: debug 2022-01-31T21:51:09.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:09.772451+0000) 2022-01-31T21:51:10.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[35325]: debug 2022-01-31T21:51:10.273+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162680 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[41853]: debug 2022-01-31T21:51:10.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.154568+0000) 2022-01-31T21:51:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[41853]: debug 2022-01-31T21:51:10.257+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.257875+0000) 2022-01-31T21:51:10.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[46715]: debug 2022-01-31T21:51:10.257+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.257791+0000) 2022-01-31T21:51:10.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[51620]: debug 2022-01-31T21:51:10.256+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.257495+0000) 2022-01-31T21:51:10.624 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:10 smithi167 conmon[49112]: debug 2022-01-31T21:51:10.254+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.255833+0000) 2022-01-31T21:51:10.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:10 smithi167 conmon[54076]: debug 2022-01-31T21:51:10.255+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.256645+0000) 2022-01-31T21:51:10.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:10 smithi167 conmon[54076]: debug 2022-01-31T21:51:10.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.437267+0000) 2022-01-31T21:51:10.625 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:10 smithi167 conmon[60316]: debug 2022-01-31T21:51:10.255+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.256991+0000) 2022-01-31T21:51:10.626 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:10 smithi167 conmon[60316]: debug 2022-01-31T21:51:10.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.437267+0000) 2022-01-31T21:51:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:10 smithi167 conmon[49112]: debug 2022-01-31T21:51:10.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.772572+0000) 2022-01-31T21:51:11.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:10 smithi171 conmon[46715]: debug 2022-01-31T21:51:10.948+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:10.949403+0000) 2022-01-31T21:51:11.153 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:11 smithi171 conmon[51620]: debug 2022-01-31T21:51:11.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:11.043250+0000) 2022-01-31T21:51:11.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:11 smithi171 conmon[41853]: debug 2022-01-31T21:51:11.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:11.154785+0000) 2022-01-31T21:51:11.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:11 smithi167 conmon[54076]: debug 2022-01-31T21:51:11.435+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:11.437430+0000) 2022-01-31T21:51:11.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:11 smithi167 conmon[60316]: debug 2022-01-31T21:51:11.435+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:11.437427+0000) 2022-01-31T21:51:12.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:11 smithi171 conmon[46715]: debug 2022-01-31T21:51:11.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:11.949573+0000) 2022-01-31T21:51:12.153 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:12 smithi171 conmon[51620]: debug 2022-01-31T21:51:12.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:12.043371+0000) 2022-01-31T21:51:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:11 smithi167 conmon[49112]: debug 2022-01-31T21:51:11.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:11.772699+0000) 2022-01-31T21:51:12.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:12 smithi171 conmon[41853]: debug 2022-01-31T21:51:12.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:12.154958+0000) 2022-01-31T21:51:12.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:12 smithi167 conmon[54076]: debug 2022-01-31T21:51:12.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:12.437645+0000) 2022-01-31T21:51:12.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:12 smithi167 conmon[60316]: debug 2022-01-31T21:51:12.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:12.437643+0000) 2022-01-31T21:51:13.153 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:12 smithi171 conmon[46715]: debug 2022-01-31T21:51:12.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:12.949744+0000) 2022-01-31T21:51:13.153 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:13 smithi171 conmon[51620]: debug 2022-01-31T21:51:13.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:13.043551+0000) 2022-01-31T21:51:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:12 smithi167 conmon[49112]: debug 2022-01-31T21:51:12.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:12.772857+0000) 2022-01-31T21:51:13.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:13 smithi171 conmon[41853]: debug 2022-01-31T21:51:13.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:13.155118+0000) 2022-01-31T21:51:13.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:13 smithi167 conmon[54076]: debug 2022-01-31T21:51:13.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:13.437801+0000) 2022-01-31T21:51:13.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:13 smithi167 conmon[60316]: debug 2022-01-31T21:51:13.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:13.437783+0000) 2022-01-31T21:51:14.153 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:13 smithi171 conmon[46715]: debug 2022-01-31T21:51:13.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:13.949959+0000) 2022-01-31T21:51:14.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:14 smithi171 conmon[51620]: debug 2022-01-31T21:51:14.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:14.043727+0000) 2022-01-31T21:51:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:13 smithi167 conmon[49112]: debug 2022-01-31T21:51:13.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:13.773031+0000) 2022-01-31T21:51:14.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:14 smithi171 conmon[41853]: debug 2022-01-31T21:51:14.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:14.155336+0000) 2022-01-31T21:51:14.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:14 smithi167 conmon[54076]: debug 2022-01-31T21:51:14.436+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:14.437991+0000) 2022-01-31T21:51:14.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:14 smithi167 conmon[60316]: debug 2022-01-31T21:51:14.436+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:14.437959+0000) 2022-01-31T21:51:15.153 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:14 smithi171 conmon[46715]: debug 2022-01-31T21:51:14.950+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:14.950145+0000) 2022-01-31T21:51:15.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[51620]: debug 2022-01-31T21:51:15.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.043916+0000) 2022-01-31T21:51:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:14 smithi167 conmon[49112]: debug 2022-01-31T21:51:14.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:14.773247+0000) 2022-01-31T21:51:15.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[35325]: debug 2022-01-31T21:51:15.293+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162790 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:15.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[46715]: debug 2022-01-31T21:51:15.275+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.276675+0000) 2022-01-31T21:51:15.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[51620]: debug 2022-01-31T21:51:15.276+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.277071+0000) 2022-01-31T21:51:15.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[41853]: debug 2022-01-31T21:51:15.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.155555+0000) 2022-01-31T21:51:15.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[41853]: debug 2022-01-31T21:51:15.276+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.277630+0000) 2022-01-31T21:51:15.636 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:15 smithi167 conmon[49112]: debug 2022-01-31T21:51:15.274+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.275805+0000) 2022-01-31T21:51:15.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:15 smithi167 conmon[54076]: debug 2022-01-31T21:51:15.274+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.276304+0000) 2022-01-31T21:51:15.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:15 smithi167 conmon[54076]: debug 2022-01-31T21:51:15.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.438155+0000) 2022-01-31T21:51:15.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:15 smithi167 conmon[60316]: debug 2022-01-31T21:51:15.274+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.276433+0000) 2022-01-31T21:51:15.638 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:15 smithi167 conmon[60316]: debug 2022-01-31T21:51:15.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.438134+0000) 2022-01-31T21:51:15.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:15 smithi167 conmon[49112]: debug 2022-01-31T21:51:15.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.773443+0000) 2022-01-31T21:51:16.153 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:15 smithi171 conmon[46715]: debug 2022-01-31T21:51:15.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:15.950315+0000) 2022-01-31T21:51:16.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:16 smithi171 conmon[51620]: debug 2022-01-31T21:51:16.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:16.044092+0000) 2022-01-31T21:51:16.494 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:16 smithi167 conmon[54076]: debug 2022-01-31T21:51:16.437+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:16.438325+0000) 2022-01-31T21:51:16.494 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:16 smithi167 conmon[60316]: debug 2022-01-31T21:51:16.437+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:16.438355+0000) 2022-01-31T21:51:16.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:16 smithi171 conmon[41853]: debug 2022-01-31T21:51:16.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:16.155738+0000) 2022-01-31T21:51:17.154 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:16 smithi171 conmon[46715]: debug 2022-01-31T21:51:16.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:16.950486+0000) 2022-01-31T21:51:17.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:17 smithi171 conmon[51620]: debug 2022-01-31T21:51:17.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:17.044266+0000) 2022-01-31T21:51:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:16 smithi167 conmon[49112]: debug 2022-01-31T21:51:16.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:16.773590+0000) 2022-01-31T21:51:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:17 smithi171 conmon[41853]: debug 2022-01-31T21:51:17.154+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:17.155907+0000) 2022-01-31T21:51:17.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:17 smithi167 conmon[54076]: debug 2022-01-31T21:51:17.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:17.438521+0000) 2022-01-31T21:51:17.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:17 smithi167 conmon[60316]: debug 2022-01-31T21:51:17.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:17.438528+0000) 2022-01-31T21:51:18.154 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:17 smithi171 conmon[46715]: debug 2022-01-31T21:51:17.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:17.950678+0000) 2022-01-31T21:51:18.155 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:18 smithi171 conmon[51620]: debug 2022-01-31T21:51:18.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:18.044489+0000) 2022-01-31T21:51:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:17 smithi167 conmon[49112]: debug 2022-01-31T21:51:17.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:17.773742+0000) 2022-01-31T21:51:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:18 smithi171 conmon[41853]: debug 2022-01-31T21:51:18.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:18.156132+0000) 2022-01-31T21:51:18.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:18 smithi167 conmon[54076]: debug 2022-01-31T21:51:18.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:18.438639+0000) 2022-01-31T21:51:18.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:18 smithi167 conmon[60316]: debug 2022-01-31T21:51:18.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:18.438646+0000) 2022-01-31T21:51:19.154 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:18 smithi171 conmon[46715]: debug 2022-01-31T21:51:18.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:18.950873+0000) 2022-01-31T21:51:19.155 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:19 smithi171 conmon[51620]: debug 2022-01-31T21:51:19.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:19.044687+0000) 2022-01-31T21:51:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:18 smithi167 conmon[49112]: debug 2022-01-31T21:51:18.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:18.773901+0000) 2022-01-31T21:51:19.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:19 smithi171 conmon[41853]: debug 2022-01-31T21:51:19.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:19.156312+0000) 2022-01-31T21:51:19.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:19 smithi167 conmon[54076]: debug 2022-01-31T21:51:19.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:19.438798+0000) 2022-01-31T21:51:19.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:19 smithi167 conmon[60316]: debug 2022-01-31T21:51:19.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:19.438773+0000) 2022-01-31T21:51:20.154 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:19 smithi171 conmon[46715]: debug 2022-01-31T21:51:19.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:19.951090+0000) 2022-01-31T21:51:20.155 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[51620]: debug 2022-01-31T21:51:20.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.044843+0000) 2022-01-31T21:51:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:19 smithi167 conmon[49112]: debug 2022-01-31T21:51:19.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:19.774042+0000) 2022-01-31T21:51:20.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[35325]: debug 2022-01-31T21:51:20.311+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 162901 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:20.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[46715]: debug 2022-01-31T21:51:20.294+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.295767+0000) 2022-01-31T21:51:20.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[51620]: debug 2022-01-31T21:51:20.294+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.295926+0000) 2022-01-31T21:51:20.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[41853]: debug 2022-01-31T21:51:20.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.156496+0000) 2022-01-31T21:51:20.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[41853]: debug 2022-01-31T21:51:20.295+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.296854+0000) 2022-01-31T21:51:20.645 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:20 smithi167 conmon[49112]: debug 2022-01-31T21:51:20.295+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.295766+0000) 2022-01-31T21:51:20.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:20 smithi167 conmon[54076]: debug 2022-01-31T21:51:20.296+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.297365+0000) 2022-01-31T21:51:20.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:20 smithi167 conmon[54076]: debug 2022-01-31T21:51:20.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.438969+0000) 2022-01-31T21:51:20.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:20 smithi167 conmon[60316]: debug 2022-01-31T21:51:20.296+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.296549+0000) 2022-01-31T21:51:20.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:20 smithi167 conmon[60316]: debug 2022-01-31T21:51:20.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.438901+0000) 2022-01-31T21:51:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:20 smithi167 conmon[49112]: debug 2022-01-31T21:51:20.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.774198+0000) 2022-01-31T21:51:21.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:21 smithi171 conmon[51620]: debug 2022-01-31T21:51:21.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:21.045035+0000) 2022-01-31T21:51:21.155 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:20 smithi171 conmon[46715]: debug 2022-01-31T21:51:20.950+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:20.951234+0000) 2022-01-31T21:51:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:21 smithi171 conmon[41853]: debug 2022-01-31T21:51:21.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:21.156684+0000) 2022-01-31T21:51:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:21 smithi171 conmon[41853]: 2022-01-31T21:51:21.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:21 smithi167 conmon[54076]: debug 2022-01-31T21:51:21.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:21.439184+0000) 2022-01-31T21:51:21.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:21 smithi167 conmon[60316]: debug 2022-01-31T21:51:21.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:21.439057+0000) 2022-01-31T21:51:22.154 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:21 smithi171 conmon[46715]: debug 2022-01-31T21:51:21.950+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:21.951381+0000) 2022-01-31T21:51:22.155 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:22 smithi171 conmon[51620]: debug 2022-01-31T21:51:22.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:22.045210+0000) 2022-01-31T21:51:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:21 smithi167 conmon[49112]: debug 2022-01-31T21:51:21.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:21.774351+0000) 2022-01-31T21:51:22.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:22 smithi171 conmon[41853]: debug 2022-01-31T21:51:22.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:22.156838+0000) 2022-01-31T21:51:22.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:22 smithi167 conmon[54076]: debug 2022-01-31T21:51:22.438+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:22.439326+0000) 2022-01-31T21:51:22.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:22 smithi167 conmon[60316]: debug 2022-01-31T21:51:22.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:22.439243+0000) 2022-01-31T21:51:23.155 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:22 smithi171 conmon[46715]: debug 2022-01-31T21:51:22.949+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:22.951587+0000) 2022-01-31T21:51:23.155 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:23 smithi171 conmon[51620]: debug 2022-01-31T21:51:23.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:23.045408+0000) 2022-01-31T21:51:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:22 smithi167 conmon[49112]: debug 2022-01-31T21:51:22.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:22.774543+0000) 2022-01-31T21:51:23.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:23 smithi171 conmon[41853]: debug 2022-01-31T21:51:23.155+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:23.156999+0000) 2022-01-31T21:51:23.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:23 smithi167 conmon[60316]: debug 2022-01-31T21:51:23.438+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:23.439477+0000) 2022-01-31T21:51:23.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:23 smithi167 conmon[54076]: debug 2022-01-31T21:51:23.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:23.439477+0000) 2022-01-31T21:51:24.048 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:23 smithi171 conmon[46715]: debug 2022-01-31T21:51:23.950+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:23.951783+0000) 2022-01-31T21:51:24.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:23 smithi171 conmon[35325]: debug 2022-01-31T21:51:23.750+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:51:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:23 smithi167 conmon[49112]: debug 2022-01-31T21:51:23.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:23.774716+0000) 2022-01-31T21:51:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:24 smithi171 conmon[41853]: debug 2022-01-31T21:51:24.156+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:24.157219+0000) 2022-01-31T21:51:24.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:24 smithi171 conmon[51620]: debug 2022-01-31T21:51:24.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:24.045594+0000) 2022-01-31T21:51:24.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:24 smithi167 conmon[54076]: debug 2022-01-31T21:51:24.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:24.439676+0000) 2022-01-31T21:51:24.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:24 smithi167 conmon[60316]: debug 2022-01-31T21:51:24.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:24.439676+0000) 2022-01-31T21:51:25.155 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:24 smithi171 conmon[46715]: debug 2022-01-31T21:51:24.951+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:24.951958+0000) 2022-01-31T21:51:25.156 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[51620]: debug 2022-01-31T21:51:25.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.045789+0000) 2022-01-31T21:51:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:24 smithi167 conmon[49112]: debug 2022-01-31T21:51:24.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:24.774885+0000) 2022-01-31T21:51:25.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[35325]: debug 2022-01-31T21:51:25.338+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163011 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[41853]: debug 2022-01-31T21:51:25.156+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.157375+0000) 2022-01-31T21:51:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[41853]: debug 2022-01-31T21:51:25.316+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.317076+0000) 2022-01-31T21:51:25.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[46715]: debug 2022-01-31T21:51:25.314+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.315393+0000) 2022-01-31T21:51:25.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[51620]: debug 2022-01-31T21:51:25.315+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.316525+0000) 2022-01-31T21:51:25.646 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:25 smithi167 conmon[49112]: debug 2022-01-31T21:51:25.313+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.315551+0000) 2022-01-31T21:51:25.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:25 smithi167 conmon[60316]: debug 2022-01-31T21:51:25.313+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.315278+0000) 2022-01-31T21:51:25.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:25 smithi167 conmon[60316]: debug 2022-01-31T21:51:25.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.439814+0000) 2022-01-31T21:51:25.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:25 smithi167 conmon[54076]: debug 2022-01-31T21:51:25.313+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.316284+0000) 2022-01-31T21:51:25.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:25 smithi167 conmon[54076]: debug 2022-01-31T21:51:25.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.439814+0000) 2022-01-31T21:51:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:25 smithi167 conmon[49112]: debug 2022-01-31T21:51:25.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.775189+0000) 2022-01-31T21:51:26.155 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:25 smithi171 conmon[46715]: debug 2022-01-31T21:51:25.951+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:25.952141+0000) 2022-01-31T21:51:26.156 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:26 smithi171 conmon[51620]: debug 2022-01-31T21:51:26.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:26.045982+0000) 2022-01-31T21:51:26.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:26 smithi171 conmon[41853]: debug 2022-01-31T21:51:26.156+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:26.157553+0000) 2022-01-31T21:51:26.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:26 smithi167 conmon[54076]: debug 2022-01-31T21:51:26.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:26.439983+0000) 2022-01-31T21:51:26.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:26 smithi167 conmon[60316]: debug 2022-01-31T21:51:26.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:26.439983+0000) 2022-01-31T21:51:27.155 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:26 smithi171 conmon[46715]: debug 2022-01-31T21:51:26.951+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:26.952315+0000) 2022-01-31T21:51:27.156 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:27 smithi171 conmon[51620]: debug 2022-01-31T21:51:27.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:27.046169+0000) 2022-01-31T21:51:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:26 smithi167 conmon[49112]: debug 2022-01-31T21:51:26.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:26.775348+0000) 2022-01-31T21:51:27.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:27 smithi171 conmon[41853]: debug 2022-01-31T21:51:27.156+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:27.157705+0000) 2022-01-31T21:51:27.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:27 smithi167 conmon[54076]: debug 2022-01-31T21:51:27.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:27.440190+0000) 2022-01-31T21:51:27.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:27 smithi167 conmon[60316]: debug 2022-01-31T21:51:27.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:27.440190+0000) 2022-01-31T21:51:28.156 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:28 smithi171 conmon[51620]: debug 2022-01-31T21:51:28.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:28.046382+0000) 2022-01-31T21:51:28.157 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:27 smithi171 conmon[46715]: debug 2022-01-31T21:51:27.951+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:27.952509+0000) 2022-01-31T21:51:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:27 smithi167 conmon[49112]: debug 2022-01-31T21:51:27.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:27.775536+0000) 2022-01-31T21:51:28.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:28 smithi171 conmon[41853]: debug 2022-01-31T21:51:28.157+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:28.157868+0000) 2022-01-31T21:51:28.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:28 smithi167 conmon[54076]: debug 2022-01-31T21:51:28.439+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:28.440389+0000) 2022-01-31T21:51:28.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:28 smithi167 conmon[60316]: debug 2022-01-31T21:51:28.439+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:28.440389+0000) 2022-01-31T21:51:29.156 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:28 smithi171 conmon[46715]: debug 2022-01-31T21:51:28.952+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:28.952699+0000) 2022-01-31T21:51:29.156 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:29 smithi171 conmon[51620]: debug 2022-01-31T21:51:29.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:29.046604+0000) 2022-01-31T21:51:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:28 smithi167 conmon[49112]: debug 2022-01-31T21:51:28.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:28.775705+0000) 2022-01-31T21:51:29.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:29 smithi171 conmon[41853]: debug 2022-01-31T21:51:29.157+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:29.158026+0000) 2022-01-31T21:51:29.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:29 smithi167 conmon[54076]: debug 2022-01-31T21:51:29.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:29.440594+0000) 2022-01-31T21:51:29.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:29 smithi167 conmon[60316]: debug 2022-01-31T21:51:29.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:29.440594+0000) 2022-01-31T21:51:30.156 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:29 smithi171 conmon[46715]: debug 2022-01-31T21:51:29.952+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:29.952861+0000) 2022-01-31T21:51:30.156 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[51620]: debug 2022-01-31T21:51:30.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.046819+0000) 2022-01-31T21:51:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:29 smithi167 conmon[49112]: debug 2022-01-31T21:51:29.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:29.775889+0000) 2022-01-31T21:51:30.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[35325]: debug 2022-01-31T21:51:30.357+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163123 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:30.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[46715]: debug 2022-01-31T21:51:30.340+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.341195+0000) 2022-01-31T21:51:30.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[51620]: debug 2022-01-31T21:51:30.340+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.341368+0000) 2022-01-31T21:51:30.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[41853]: debug 2022-01-31T21:51:30.157+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.158185+0000) 2022-01-31T21:51:30.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[41853]: debug 2022-01-31T21:51:30.342+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.342938+0000) 2022-01-31T21:51:30.647 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:30 smithi167 conmon[49112]: debug 2022-01-31T21:51:30.341+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.342359+0000) 2022-01-31T21:51:30.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:30 smithi167 conmon[54076]: debug 2022-01-31T21:51:30.341+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.342023+0000) 2022-01-31T21:51:30.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:30 smithi167 conmon[54076]: debug 2022-01-31T21:51:30.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.440755+0000) 2022-01-31T21:51:30.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:30 smithi167 conmon[60316]: debug 2022-01-31T21:51:30.340+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.341089+0000) 2022-01-31T21:51:30.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:30 smithi167 conmon[60316]: debug 2022-01-31T21:51:30.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.440755+0000) 2022-01-31T21:51:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:30 smithi167 conmon[49112]: debug 2022-01-31T21:51:30.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.776037+0000) 2022-01-31T21:51:31.156 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:30 smithi171 conmon[46715]: debug 2022-01-31T21:51:30.952+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:30.953052+0000) 2022-01-31T21:51:31.157 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:31 smithi171 conmon[51620]: debug 2022-01-31T21:51:31.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:31.046978+0000) 2022-01-31T21:51:31.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:31 smithi171 conmon[41853]: debug 2022-01-31T21:51:31.157+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:31.158351+0000) 2022-01-31T21:51:31.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:31 smithi167 conmon[54076]: debug 2022-01-31T21:51:31.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:31.440864+0000) 2022-01-31T21:51:31.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:31 smithi167 conmon[60316]: debug 2022-01-31T21:51:31.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:31.440909+0000) 2022-01-31T21:51:32.156 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:31 smithi171 conmon[46715]: debug 2022-01-31T21:51:31.952+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:31.953217+0000) 2022-01-31T21:51:32.157 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:32 smithi171 conmon[51620]: debug 2022-01-31T21:51:32.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:32.047227+0000) 2022-01-31T21:51:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:31 smithi167 conmon[49112]: debug 2022-01-31T21:51:31.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:31.776160+0000) 2022-01-31T21:51:32.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:32 smithi171 conmon[41853]: debug 2022-01-31T21:51:32.157+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:32.158517+0000) 2022-01-31T21:51:32.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:32 smithi167 conmon[54076]: debug 2022-01-31T21:51:32.440+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:32.441090+0000) 2022-01-31T21:51:32.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:32 smithi167 conmon[60316]: debug 2022-01-31T21:51:32.440+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:32.441090+0000) 2022-01-31T21:51:33.156 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:32 smithi171 conmon[46715]: debug 2022-01-31T21:51:32.952+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:32.953404+0000) 2022-01-31T21:51:33.157 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:33 smithi171 conmon[51620]: debug 2022-01-31T21:51:33.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:33.047357+0000) 2022-01-31T21:51:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:32 smithi167 conmon[49112]: debug 2022-01-31T21:51:32.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:32.776354+0000) 2022-01-31T21:51:33.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:33 smithi171 conmon[41853]: debug 2022-01-31T21:51:33.157+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:33.158659+0000) 2022-01-31T21:51:33.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:33 smithi167 conmon[54076]: debug 2022-01-31T21:51:33.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:33.441319+0000) 2022-01-31T21:51:33.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:33 smithi167 conmon[60316]: debug 2022-01-31T21:51:33.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:33.441319+0000) 2022-01-31T21:51:34.156 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:33 smithi171 conmon[46715]: debug 2022-01-31T21:51:33.952+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:33.953585+0000) 2022-01-31T21:51:34.157 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:34 smithi171 conmon[51620]: debug 2022-01-31T21:51:34.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:34.047535+0000) 2022-01-31T21:51:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:33 smithi167 conmon[49112]: debug 2022-01-31T21:51:33.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:33.776528+0000) 2022-01-31T21:51:34.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:34 smithi171 conmon[41853]: debug 2022-01-31T21:51:34.158+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:34.158847+0000) 2022-01-31T21:51:34.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:34 smithi167 conmon[54076]: debug 2022-01-31T21:51:34.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:34.441493+0000) 2022-01-31T21:51:34.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:34 smithi167 conmon[60316]: debug 2022-01-31T21:51:34.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:34.441493+0000) 2022-01-31T21:51:35.157 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:34 smithi171 conmon[46715]: debug 2022-01-31T21:51:34.953+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:34.953748+0000) 2022-01-31T21:51:35.157 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[51620]: debug 2022-01-31T21:51:35.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.047642+0000) 2022-01-31T21:51:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:34 smithi167 conmon[49112]: debug 2022-01-31T21:51:34.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:34.776659+0000) 2022-01-31T21:51:35.605 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[35325]: debug 2022-01-31T21:51:35.377+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163233 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:35.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[46715]: debug 2022-01-31T21:51:35.359+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.360172+0000) 2022-01-31T21:51:35.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[51620]: debug 2022-01-31T21:51:35.360+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.360992+0000) 2022-01-31T21:51:35.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[41853]: debug 2022-01-31T21:51:35.158+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.158987+0000) 2022-01-31T21:51:35.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[41853]: debug 2022-01-31T21:51:35.360+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.361429+0000) 2022-01-31T21:51:35.648 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:35 smithi167 conmon[49112]: debug 2022-01-31T21:51:35.361+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.361765+0000) 2022-01-31T21:51:35.648 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:35 smithi167 conmon[54076]: debug 2022-01-31T21:51:35.360+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.360959+0000) 2022-01-31T21:51:35.649 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:35 smithi167 conmon[54076]: debug 2022-01-31T21:51:35.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.441693+0000) 2022-01-31T21:51:35.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:35 smithi167 conmon[60316]: debug 2022-01-31T21:51:35.360+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.360857+0000) 2022-01-31T21:51:35.650 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:35 smithi167 conmon[60316]: debug 2022-01-31T21:51:35.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.441693+0000) 2022-01-31T21:51:35.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:35 smithi167 conmon[49112]: debug 2022-01-31T21:51:35.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.776846+0000) 2022-01-31T21:51:36.157 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:35 smithi171 conmon[46715]: debug 2022-01-31T21:51:35.953+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:35.953922+0000) 2022-01-31T21:51:36.157 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:36 smithi171 conmon[51620]: debug 2022-01-31T21:51:36.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:36.047812+0000) 2022-01-31T21:51:36.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:36 smithi171 conmon[41853]: debug 2022-01-31T21:51:36.158+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:36.159164+0000) 2022-01-31T21:51:36.750 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:36 smithi167 conmon[54076]: debug 2022-01-31T21:51:36.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:36.441858+0000) 2022-01-31T21:51:36.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:36 smithi167 conmon[60316]: debug 2022-01-31T21:51:36.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:36.441858+0000) 2022-01-31T21:51:37.157 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:36 smithi171 conmon[46715]: debug 2022-01-31T21:51:36.953+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:36.954005+0000) 2022-01-31T21:51:37.158 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:37 smithi171 conmon[51620]: debug 2022-01-31T21:51:37.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:37.047961+0000) 2022-01-31T21:51:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:36 smithi167 conmon[49112]: debug 2022-01-31T21:51:36.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:36.776968+0000) 2022-01-31T21:51:37.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:37 smithi171 conmon[41853]: debug 2022-01-31T21:51:37.158+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:37.159320+0000) 2022-01-31T21:51:37.691 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:37 smithi167 conmon[54076]: debug 2022-01-31T21:51:37.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:37.442013+0000) 2022-01-31T21:51:37.692 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:37 smithi167 conmon[60316]: debug 2022-01-31T21:51:37.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:37.442013+0000) 2022-01-31T21:51:38.157 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:37 smithi171 conmon[46715]: debug 2022-01-31T21:51:37.953+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:37.954190+0000) 2022-01-31T21:51:38.158 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:38 smithi171 conmon[51620]: debug 2022-01-31T21:51:38.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:38.048166+0000) 2022-01-31T21:51:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:37 smithi167 conmon[49112]: debug 2022-01-31T21:51:37.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:37.777152+0000) 2022-01-31T21:51:38.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:38 smithi171 conmon[41853]: debug 2022-01-31T21:51:38.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:38.159502+0000) 2022-01-31T21:51:38.694 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:38 smithi167 conmon[54076]: debug 2022-01-31T21:51:38.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:38.442214+0000) 2022-01-31T21:51:38.695 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:38 smithi167 conmon[60316]: debug 2022-01-31T21:51:38.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:38.442214+0000) 2022-01-31T21:51:39.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:38 smithi171 conmon[35325]: debug 2022-01-31T21:51:38.751+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:51:39.047 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:38 smithi171 conmon[46715]: debug 2022-01-31T21:51:38.953+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:38.954395+0000) 2022-01-31T21:51:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:38 smithi167 conmon[49112]: debug 2022-01-31T21:51:38.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:38.777422+0000) 2022-01-31T21:51:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:39 smithi171 conmon[41853]: debug 2022-01-31T21:51:39.158+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:39.159676+0000) 2022-01-31T21:51:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:39 smithi171 conmon[51620]: debug 2022-01-31T21:51:39.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:39.048360+0000) 2022-01-31T21:51:39.695 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:39 smithi167 conmon[54076]: debug 2022-01-31T21:51:39.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:39.442387+0000) 2022-01-31T21:51:39.695 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:39 smithi167 conmon[60316]: debug 2022-01-31T21:51:39.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:39.442387+0000) 2022-01-31T21:51:40.157 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:39 smithi171 conmon[46715]: debug 2022-01-31T21:51:39.954+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:39.954584+0000) 2022-01-31T21:51:40.158 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[51620]: debug 2022-01-31T21:51:40.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.048574+0000) 2022-01-31T21:51:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:39 smithi167 conmon[49112]: debug 2022-01-31T21:51:39.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:39.777613+0000) 2022-01-31T21:51:40.605 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[46715]: debug 2022-01-31T21:51:40.379+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.380097+0000) 2022-01-31T21:51:40.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[51620]: debug 2022-01-31T21:51:40.380+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.381464+0000) 2022-01-31T21:51:40.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[35325]: debug 2022-01-31T21:51:40.397+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163340 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:40.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[41853]: debug 2022-01-31T21:51:40.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.159838+0000) 2022-01-31T21:51:40.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[41853]: debug 2022-01-31T21:51:40.380+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.380862+0000) 2022-01-31T21:51:40.648 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:40 smithi167 conmon[49112]: debug 2022-01-31T21:51:40.380+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.381255+0000) 2022-01-31T21:51:40.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:40 smithi167 conmon[60316]: debug 2022-01-31T21:51:40.380+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.381388+0000) 2022-01-31T21:51:40.649 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:40 smithi167 conmon[60316]: debug 2022-01-31T21:51:40.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.442581+0000) 2022-01-31T21:51:40.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:40 smithi167 conmon[54076]: debug 2022-01-31T21:51:40.380+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.380589+0000) 2022-01-31T21:51:40.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:40 smithi167 conmon[54076]: debug 2022-01-31T21:51:40.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.442581+0000) 2022-01-31T21:51:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:40 smithi167 conmon[49112]: debug 2022-01-31T21:51:40.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.777798+0000) 2022-01-31T21:51:41.158 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:41 smithi171 conmon[51620]: debug 2022-01-31T21:51:41.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:41.048776+0000) 2022-01-31T21:51:41.158 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:40 smithi171 conmon[46715]: debug 2022-01-31T21:51:40.954+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:40.954786+0000) 2022-01-31T21:51:41.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:41 smithi171 conmon[41853]: debug 2022-01-31T21:51:41.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:41.159955+0000) 2022-01-31T21:51:41.699 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:41 smithi167 conmon[54076]: debug 2022-01-31T21:51:41.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:41.442731+0000) 2022-01-31T21:51:41.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:41 smithi167 conmon[60316]: debug 2022-01-31T21:51:41.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:41.442748+0000) 2022-01-31T21:51:42.158 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:41 smithi171 conmon[46715]: debug 2022-01-31T21:51:41.954+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:41.954934+0000) 2022-01-31T21:51:42.158 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:42 smithi171 conmon[51620]: debug 2022-01-31T21:51:42.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:42.048928+0000) 2022-01-31T21:51:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:41 smithi167 conmon[49112]: debug 2022-01-31T21:51:41.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:41.777917+0000) 2022-01-31T21:51:42.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:42 smithi171 conmon[41853]: debug 2022-01-31T21:51:42.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:42.160072+0000) 2022-01-31T21:51:42.702 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:42 smithi167 conmon[54076]: debug 2022-01-31T21:51:42.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:42.442946+0000) 2022-01-31T21:51:42.703 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:42 smithi167 conmon[60316]: debug 2022-01-31T21:51:42.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:42.442947+0000) 2022-01-31T21:51:43.158 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:42 smithi171 conmon[46715]: debug 2022-01-31T21:51:42.954+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:42.955117+0000) 2022-01-31T21:51:43.159 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:43 smithi171 conmon[51620]: debug 2022-01-31T21:51:43.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:43.049140+0000) 2022-01-31T21:51:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:42 smithi167 conmon[49112]: debug 2022-01-31T21:51:42.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:42.778102+0000) 2022-01-31T21:51:43.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:43 smithi171 conmon[41853]: debug 2022-01-31T21:51:43.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:43.160245+0000) 2022-01-31T21:51:43.705 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:43 smithi167 conmon[54076]: debug 2022-01-31T21:51:43.441+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:43.443160+0000) 2022-01-31T21:51:43.706 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:43 smithi167 conmon[60316]: debug 2022-01-31T21:51:43.441+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:43.443161+0000) 2022-01-31T21:51:44.158 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:43 smithi171 conmon[46715]: debug 2022-01-31T21:51:43.954+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:43.955306+0000) 2022-01-31T21:51:44.159 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:44 smithi171 conmon[51620]: debug 2022-01-31T21:51:44.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:44.049317+0000) 2022-01-31T21:51:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:43 smithi167 conmon[49112]: debug 2022-01-31T21:51:43.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:43.778300+0000) 2022-01-31T21:51:44.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:44 smithi171 conmon[41853]: debug 2022-01-31T21:51:44.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:44.160431+0000) 2022-01-31T21:51:44.705 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:44 smithi167 conmon[60316]: debug 2022-01-31T21:51:44.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:44.443383+0000) 2022-01-31T21:51:44.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:44 smithi167 conmon[54076]: debug 2022-01-31T21:51:44.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:44.443383+0000) 2022-01-31T21:51:45.158 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:44 smithi171 conmon[46715]: debug 2022-01-31T21:51:44.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:44.955488+0000) 2022-01-31T21:51:45.159 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[51620]: debug 2022-01-31T21:51:45.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.049474+0000) 2022-01-31T21:51:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:44 smithi167 conmon[49112]: debug 2022-01-31T21:51:44.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:44.778511+0000) 2022-01-31T21:51:45.415 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[46715]: debug 2022-01-31T21:51:45.400+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.401518+0000) 2022-01-31T21:51:45.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[51620]: debug 2022-01-31T21:51:45.400+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.401382+0000) 2022-01-31T21:51:45.417 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[41853]: debug 2022-01-31T21:51:45.159+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.160621+0000) 2022-01-31T21:51:45.417 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[41853]: debug 2022-01-31T21:51:45.399+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.400066+0000) 2022-01-31T21:51:45.650 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:45 smithi167 conmon[49112]: debug 2022-01-31T21:51:45.399+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.400829+0000) 2022-01-31T21:51:45.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:45 smithi167 conmon[54076]: debug 2022-01-31T21:51:45.399+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.400490+0000) 2022-01-31T21:51:45.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:45 smithi167 conmon[54076]: debug 2022-01-31T21:51:45.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.443626+0000) 2022-01-31T21:51:45.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:45 smithi167 conmon[60316]: debug 2022-01-31T21:51:45.398+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.400068+0000) 2022-01-31T21:51:45.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:45 smithi167 conmon[60316]: debug 2022-01-31T21:51:45.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.443585+0000) 2022-01-31T21:51:45.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[35325]: debug 2022-01-31T21:51:45.416+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163450 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:45 smithi167 conmon[49112]: debug 2022-01-31T21:51:45.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.778672+0000) 2022-01-31T21:51:46.158 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:45 smithi171 conmon[46715]: debug 2022-01-31T21:51:45.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:45.955670+0000) 2022-01-31T21:51:46.159 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:46 smithi171 conmon[51620]: debug 2022-01-31T21:51:46.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:46.049646+0000) 2022-01-31T21:51:46.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:46 smithi167 conmon[54076]: debug 2022-01-31T21:51:46.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:46.443732+0000) 2022-01-31T21:51:46.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:46 smithi167 conmon[60316]: debug 2022-01-31T21:51:46.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:46.443692+0000) 2022-01-31T21:51:46.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:46 smithi171 conmon[41853]: debug 2022-01-31T21:51:46.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:46.160799+0000) 2022-01-31T21:51:47.159 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:46 smithi171 conmon[46715]: debug 2022-01-31T21:51:46.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:46.955817+0000) 2022-01-31T21:51:47.159 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:47 smithi171 conmon[51620]: debug 2022-01-31T21:51:47.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:47.049810+0000) 2022-01-31T21:51:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:46 smithi167 conmon[49112]: debug 2022-01-31T21:51:46.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:46.778767+0000) 2022-01-31T21:51:47.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:47 smithi171 conmon[41853]: debug 2022-01-31T21:51:47.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:47.160944+0000) 2022-01-31T21:51:47.713 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:47 smithi167 conmon[54076]: debug 2022-01-31T21:51:47.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:47.443838+0000) 2022-01-31T21:51:47.714 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:47 smithi167 conmon[60316]: debug 2022-01-31T21:51:47.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:47.443839+0000) 2022-01-31T21:51:48.159 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:47 smithi171 conmon[46715]: debug 2022-01-31T21:51:47.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:47.955982+0000) 2022-01-31T21:51:48.160 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:48 smithi171 conmon[51620]: debug 2022-01-31T21:51:48.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:48.050041+0000) 2022-01-31T21:51:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:47 smithi167 conmon[49112]: debug 2022-01-31T21:51:47.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:47.778979+0000) 2022-01-31T21:51:48.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:48 smithi171 conmon[41853]: debug 2022-01-31T21:51:48.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:48.161074+0000) 2022-01-31T21:51:48.716 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:48 smithi167 conmon[54076]: debug 2022-01-31T21:51:48.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:48.444001+0000) 2022-01-31T21:51:48.717 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:48 smithi167 conmon[60316]: debug 2022-01-31T21:51:48.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:48.444001+0000) 2022-01-31T21:51:49.159 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:48 smithi171 conmon[46715]: debug 2022-01-31T21:51:48.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:48.956136+0000) 2022-01-31T21:51:49.160 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:49 smithi171 conmon[51620]: debug 2022-01-31T21:51:49.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:49.050246+0000) 2022-01-31T21:51:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:48 smithi167 conmon[49112]: debug 2022-01-31T21:51:48.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:48.779179+0000) 2022-01-31T21:51:49.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:49 smithi171 conmon[41853]: debug 2022-01-31T21:51:49.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:49.161237+0000) 2022-01-31T21:51:49.717 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:49 smithi167 conmon[54076]: debug 2022-01-31T21:51:49.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:49.444218+0000) 2022-01-31T21:51:49.718 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:49 smithi167 conmon[60316]: debug 2022-01-31T21:51:49.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:49.444218+0000) 2022-01-31T21:51:50.159 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:49 smithi171 conmon[46715]: debug 2022-01-31T21:51:49.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:49.956356+0000) 2022-01-31T21:51:50.160 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[51620]: debug 2022-01-31T21:51:50.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.050401+0000) 2022-01-31T21:51:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:49 smithi167 conmon[49112]: debug 2022-01-31T21:51:49.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:49.779347+0000) 2022-01-31T21:51:50.418 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[41853]: debug 2022-01-31T21:51:50.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.161395+0000) 2022-01-31T21:51:50.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:50 smithi167 conmon[49112]: debug 2022-01-31T21:51:50.417+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.419416+0000) 2022-01-31T21:51:50.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:50 smithi167 conmon[60316]: debug 2022-01-31T21:51:50.418+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.420450+0000) 2022-01-31T21:51:50.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:50 smithi167 conmon[60316]: debug 2022-01-31T21:51:50.442+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.444380+0000) 2022-01-31T21:51:50.671 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:50 smithi167 conmon[54076]: debug 2022-01-31T21:51:50.418+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.419520+0000) 2022-01-31T21:51:50.671 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:50 smithi167 conmon[54076]: debug 2022-01-31T21:51:50.442+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.444391+0000) 2022-01-31T21:51:50.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[35325]: debug 2022-01-31T21:51:50.435+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163562 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[41853]: debug 2022-01-31T21:51:50.419+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.419745+0000) 2022-01-31T21:51:50.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[46715]: debug 2022-01-31T21:51:50.419+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.419880+0000) 2022-01-31T21:51:50.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[51620]: debug 2022-01-31T21:51:50.419+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.419766+0000) 2022-01-31T21:51:50.942 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:50 smithi167 conmon[49112]: debug 2022-01-31T21:51:50.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.779562+0000) 2022-01-31T21:51:51.159 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:50 smithi171 conmon[46715]: debug 2022-01-31T21:51:50.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:50.956506+0000) 2022-01-31T21:51:51.160 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:51 smithi171 conmon[51620]: debug 2022-01-31T21:51:51.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:51.050601+0000) 2022-01-31T21:51:51.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:51 smithi171 conmon[41853]: debug 2022-01-31T21:51:51.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:51.161549+0000) 2022-01-31T21:51:51.721 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:51 smithi167 conmon[54076]: debug 2022-01-31T21:51:51.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:51.444538+0000) 2022-01-31T21:51:51.721 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:51 smithi167 conmon[60316]: debug 2022-01-31T21:51:51.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:51.444543+0000) 2022-01-31T21:51:52.159 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:51 smithi171 conmon[46715]: debug 2022-01-31T21:51:51.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:51.956692+0000) 2022-01-31T21:51:52.160 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:52 smithi171 conmon[51620]: debug 2022-01-31T21:51:52.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:52.050726+0000) 2022-01-31T21:51:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:51 smithi167 conmon[49112]: debug 2022-01-31T21:51:51.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:51.779696+0000) 2022-01-31T21:51:52.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:52 smithi171 conmon[41853]: debug 2022-01-31T21:51:52.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:52.161714+0000) 2022-01-31T21:51:52.651 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:52 smithi167 conmon[54076]: debug 2022-01-31T21:51:52.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:52.444711+0000) 2022-01-31T21:51:52.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:52 smithi167 conmon[60316]: debug 2022-01-31T21:51:52.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:52.444709+0000) 2022-01-31T21:51:52.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:52 smithi167 conmon[49112]: debug 2022-01-31T21:51:52.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:52.779904+0000) 2022-01-31T21:51:53.160 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:52 smithi171 conmon[46715]: debug 2022-01-31T21:51:52.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:52.956934+0000) 2022-01-31T21:51:53.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:53 smithi171 conmon[51620]: debug 2022-01-31T21:51:53.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:53.050828+0000) 2022-01-31T21:51:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:53 smithi171 conmon[41853]: debug 2022-01-31T21:51:53.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:53.161878+0000) 2022-01-31T21:51:53.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:53 smithi167 conmon[54076]: debug 2022-01-31T21:51:53.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:53.444893+0000) 2022-01-31T21:51:53.729 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:53 smithi167 conmon[60316]: debug 2022-01-31T21:51:53.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:53.444898+0000) 2022-01-31T21:51:54.049 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:53 smithi171 conmon[35325]: debug 2022-01-31T21:51:53.752+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:51:54.049 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:53 smithi171 conmon[46715]: debug 2022-01-31T21:51:53.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:53.957115+0000) 2022-01-31T21:51:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:53 smithi167 conmon[49112]: debug 2022-01-31T21:51:53.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:53.780080+0000) 2022-01-31T21:51:54.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:54 smithi171 conmon[51620]: debug 2022-01-31T21:51:54.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:54.050978+0000) 2022-01-31T21:51:54.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:54 smithi171 conmon[41853]: debug 2022-01-31T21:51:54.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:54.162021+0000) 2022-01-31T21:51:54.728 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:54 smithi167 conmon[60316]: debug 2022-01-31T21:51:54.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:54.445104+0000) 2022-01-31T21:51:54.728 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:54 smithi167 conmon[54076]: debug 2022-01-31T21:51:54.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:54.445097+0000) 2022-01-31T21:51:55.160 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:54 smithi171 conmon[46715]: debug 2022-01-31T21:51:54.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:54.957274+0000) 2022-01-31T21:51:55.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[51620]: debug 2022-01-31T21:51:55.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.051187+0000) 2022-01-31T21:51:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:54 smithi167 conmon[49112]: debug 2022-01-31T21:51:54.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:54.780291+0000) 2022-01-31T21:51:55.436 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[41853]: debug 2022-01-31T21:51:55.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.162186+0000) 2022-01-31T21:51:55.651 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:55 smithi167 conmon[49112]: debug 2022-01-31T21:51:55.437+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.438707+0000) 2022-01-31T21:51:55.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:55 smithi167 conmon[54076]: debug 2022-01-31T21:51:55.436+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.438294+0000) 2022-01-31T21:51:55.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:55 smithi167 conmon[54076]: debug 2022-01-31T21:51:55.443+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.445260+0000) 2022-01-31T21:51:55.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:55 smithi167 conmon[60316]: debug 2022-01-31T21:51:55.438+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.439727+0000) 2022-01-31T21:51:55.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:55 smithi167 conmon[60316]: debug 2022-01-31T21:51:55.443+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.445262+0000) 2022-01-31T21:51:55.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[35325]: debug 2022-01-31T21:51:55.454+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163672 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:51:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[41853]: debug 2022-01-31T21:51:55.437+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.439337+0000) 2022-01-31T21:51:55.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[46715]: debug 2022-01-31T21:51:55.438+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.439975+0000) 2022-01-31T21:51:55.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[51620]: debug 2022-01-31T21:51:55.436+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.438465+0000) 2022-01-31T21:51:55.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:55 smithi167 conmon[49112]: debug 2022-01-31T21:51:55.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.780422+0000) 2022-01-31T21:51:56.160 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:55 smithi171 conmon[46715]: debug 2022-01-31T21:51:55.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:55.957452+0000) 2022-01-31T21:51:56.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:56 smithi171 conmon[51620]: debug 2022-01-31T21:51:56.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:56.051332+0000) 2022-01-31T21:51:56.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:56 smithi171 conmon[41853]: debug 2022-01-31T21:51:56.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:56.162357+0000) 2022-01-31T21:51:56.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:56 smithi167 conmon[60316]: debug 2022-01-31T21:51:56.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:56.445428+0000) 2022-01-31T21:51:56.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:56 smithi167 conmon[54076]: debug 2022-01-31T21:51:56.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:56.445448+0000) 2022-01-31T21:51:57.160 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:56 smithi171 conmon[46715]: debug 2022-01-31T21:51:56.955+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:56.957619+0000) 2022-01-31T21:51:57.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:57 smithi171 conmon[51620]: debug 2022-01-31T21:51:57.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:57.051427+0000) 2022-01-31T21:51:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:56 smithi167 conmon[49112]: debug 2022-01-31T21:51:56.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:56.780598+0000) 2022-01-31T21:51:57.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:57 smithi171 conmon[41853]: debug 2022-01-31T21:51:57.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:57.162532+0000) 2022-01-31T21:51:57.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:57 smithi167 conmon[54076]: debug 2022-01-31T21:51:57.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:57.445608+0000) 2022-01-31T21:51:57.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:57 smithi167 conmon[60316]: debug 2022-01-31T21:51:57.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:57.445602+0000) 2022-01-31T21:51:58.160 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:57 smithi171 conmon[46715]: debug 2022-01-31T21:51:57.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:57.957814+0000) 2022-01-31T21:51:58.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:58 smithi171 conmon[51620]: debug 2022-01-31T21:51:58.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:58.051588+0000) 2022-01-31T21:51:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:57 smithi167 conmon[49112]: debug 2022-01-31T21:51:57.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:57.780808+0000) 2022-01-31T21:51:58.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:58 smithi171 conmon[41853]: debug 2022-01-31T21:51:58.160+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:58.162712+0000) 2022-01-31T21:51:58.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:58 smithi167 conmon[54076]: debug 2022-01-31T21:51:58.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:58.445807+0000) 2022-01-31T21:51:58.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:58 smithi167 conmon[60316]: debug 2022-01-31T21:51:58.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:58.445802+0000) 2022-01-31T21:51:59.161 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:58 smithi171 conmon[46715]: debug 2022-01-31T21:51:58.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:58.958000+0000) 2022-01-31T21:51:59.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:51:59 smithi171 conmon[51620]: debug 2022-01-31T21:51:59.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:59.051804+0000) 2022-01-31T21:51:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:58 smithi167 conmon[49112]: debug 2022-01-31T21:51:58.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:58.781026+0000) 2022-01-31T21:51:59.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:51:59 smithi171 conmon[41853]: debug 2022-01-31T21:51:59.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:59.162934+0000) 2022-01-31T21:51:59.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:51:59 smithi167 conmon[54076]: debug 2022-01-31T21:51:59.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:59.446003+0000) 2022-01-31T21:51:59.653 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:51:59 smithi167 conmon[60316]: debug 2022-01-31T21:51:59.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:59.445999+0000) 2022-01-31T21:51:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:51:59 smithi167 conmon[49112]: debug 2022-01-31T21:51:59.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:59.781239+0000) 2022-01-31T21:52:00.161 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:51:59 smithi171 conmon[46715]: debug 2022-01-31T21:51:59.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:51:59.958177+0000) 2022-01-31T21:52:00.162 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[51620]: debug 2022-01-31T21:52:00.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.051970+0000) 2022-01-31T21:52:00.456 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[41853]: debug 2022-01-31T21:52:00.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.163104+0000) 2022-01-31T21:52:00.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:00 smithi167 conmon[49112]: debug 2022-01-31T21:52:00.456+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.457509+0000) 2022-01-31T21:52:00.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:00 smithi167 conmon[54076]: debug 2022-01-31T21:52:00.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.446208+0000) 2022-01-31T21:52:00.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:00 smithi167 conmon[54076]: debug 2022-01-31T21:52:00.456+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.458049+0000) 2022-01-31T21:52:00.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:00 smithi167 conmon[60316]: debug 2022-01-31T21:52:00.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.446183+0000) 2022-01-31T21:52:00.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:00 smithi167 conmon[60316]: debug 2022-01-31T21:52:00.456+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.457749+0000) 2022-01-31T21:52:00.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[35325]: debug 2022-01-31T21:52:00.473+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163782 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[41853]: debug 2022-01-31T21:52:00.457+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.459153+0000) 2022-01-31T21:52:00.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[46715]: debug 2022-01-31T21:52:00.456+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.458187+0000) 2022-01-31T21:52:00.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[51620]: debug 2022-01-31T21:52:00.456+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.457857+0000) 2022-01-31T21:52:01.161 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:01 smithi171 conmon[51620]: debug 2022-01-31T21:52:01.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:01.052175+0000) 2022-01-31T21:52:01.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:00 smithi171 conmon[46715]: debug 2022-01-31T21:52:00.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.958358+0000) 2022-01-31T21:52:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:00 smithi167 conmon[49112]: debug 2022-01-31T21:52:00.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:00.781410+0000) 2022-01-31T21:52:01.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:01 smithi171 conmon[41853]: debug 2022-01-31T21:52:01.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:01.163309+0000) 2022-01-31T21:52:01.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:01 smithi167 conmon[54076]: debug 2022-01-31T21:52:01.444+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:01.446387+0000) 2022-01-31T21:52:01.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:01 smithi167 conmon[60316]: debug 2022-01-31T21:52:01.444+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:01.446326+0000) 2022-01-31T21:52:02.161 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:01 smithi171 conmon[46715]: debug 2022-01-31T21:52:01.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:01.958508+0000) 2022-01-31T21:52:02.162 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:02 smithi171 conmon[51620]: debug 2022-01-31T21:52:02.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:02.052322+0000) 2022-01-31T21:52:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:01 smithi167 conmon[49112]: debug 2022-01-31T21:52:01.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:01.781554+0000) 2022-01-31T21:52:02.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:02 smithi171 conmon[41853]: debug 2022-01-31T21:52:02.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:02.163470+0000) 2022-01-31T21:52:02.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:02 smithi167 conmon[54076]: debug 2022-01-31T21:52:02.445+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:02.446540+0000) 2022-01-31T21:52:02.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:02 smithi167 conmon[60316]: debug 2022-01-31T21:52:02.445+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:02.446521+0000) 2022-01-31T21:52:03.161 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:02 smithi171 conmon[46715]: debug 2022-01-31T21:52:02.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:02.958666+0000) 2022-01-31T21:52:03.162 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:03 smithi171 conmon[51620]: debug 2022-01-31T21:52:03.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:03.052531+0000) 2022-01-31T21:52:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:02 smithi167 conmon[49112]: debug 2022-01-31T21:52:02.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:02.781740+0000) 2022-01-31T21:52:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:03 smithi171 conmon[41853]: debug 2022-01-31T21:52:03.161+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:03.163655+0000) 2022-01-31T21:52:03.751 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:03 smithi167 conmon[60316]: debug 2022-01-31T21:52:03.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:03.446775+0000) 2022-01-31T21:52:03.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:03 smithi167 conmon[54076]: debug 2022-01-31T21:52:03.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:03.446777+0000) 2022-01-31T21:52:04.161 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:03 smithi171 conmon[46715]: debug 2022-01-31T21:52:03.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:03.958860+0000) 2022-01-31T21:52:04.162 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:04 smithi171 conmon[51620]: debug 2022-01-31T21:52:04.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:04.052726+0000) 2022-01-31T21:52:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:03 smithi167 conmon[49112]: debug 2022-01-31T21:52:03.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:03.781941+0000) 2022-01-31T21:52:04.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:04 smithi171 conmon[41853]: debug 2022-01-31T21:52:04.163+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:04.163831+0000) 2022-01-31T21:52:04.751 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:04 smithi167 conmon[54076]: debug 2022-01-31T21:52:04.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:04.446961+0000) 2022-01-31T21:52:04.751 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:04 smithi167 conmon[60316]: debug 2022-01-31T21:52:04.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:04.446916+0000) 2022-01-31T21:52:05.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:04 smithi171 conmon[46715]: debug 2022-01-31T21:52:04.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:04.959015+0000) 2022-01-31T21:52:05.379 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[51620]: debug 2022-01-31T21:52:05.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.052952+0000) 2022-01-31T21:52:05.380 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:04 smithi167 conmon[49112]: debug 2022-01-31T21:52:04.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:04.782169+0000) 2022-01-31T21:52:05.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[41853]: debug 2022-01-31T21:52:05.163+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.163931+0000) 2022-01-31T21:52:05.751 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:05 smithi167 conmon[49112]: debug 2022-01-31T21:52:05.474+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.476350+0000) 2022-01-31T21:52:05.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:05 smithi167 conmon[54076]: debug 2022-01-31T21:52:05.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.447161+0000) 2022-01-31T21:52:05.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:05 smithi167 conmon[54076]: debug 2022-01-31T21:52:05.476+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.477605+0000) 2022-01-31T21:52:05.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:05 smithi167 conmon[60316]: debug 2022-01-31T21:52:05.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.447113+0000) 2022-01-31T21:52:05.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:05 smithi167 conmon[60316]: debug 2022-01-31T21:52:05.475+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.476641+0000) 2022-01-31T21:52:05.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[35325]: debug 2022-01-31T21:52:05.494+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 163892 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[41853]: debug 2022-01-31T21:52:05.478+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.478924+0000) 2022-01-31T21:52:05.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[46715]: debug 2022-01-31T21:52:05.476+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.477226+0000) 2022-01-31T21:52:05.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[51620]: debug 2022-01-31T21:52:05.477+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.477860+0000) 2022-01-31T21:52:06.162 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:06 smithi171 conmon[51620]: debug 2022-01-31T21:52:06.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:06.053168+0000) 2022-01-31T21:52:06.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:05 smithi171 conmon[46715]: debug 2022-01-31T21:52:05.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.959178+0000) 2022-01-31T21:52:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:05 smithi167 conmon[49112]: debug 2022-01-31T21:52:05.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:05.782335+0000) 2022-01-31T21:52:06.464 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:06 smithi171 conmon[41853]: debug 2022-01-31T21:52:06.163+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:06.164068+0000) 2022-01-31T21:52:06.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:06 smithi167 conmon[54076]: debug 2022-01-31T21:52:06.446+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:06.447365+0000) 2022-01-31T21:52:06.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:06 smithi167 conmon[60316]: debug 2022-01-31T21:52:06.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:06.447321+0000) 2022-01-31T21:52:07.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:06 smithi171 conmon[46715]: debug 2022-01-31T21:52:06.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:06.959297+0000) 2022-01-31T21:52:07.163 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:07 smithi171 conmon[51620]: debug 2022-01-31T21:52:07.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:07.053323+0000) 2022-01-31T21:52:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:06 smithi167 conmon[49112]: debug 2022-01-31T21:52:06.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:06.782574+0000) 2022-01-31T21:52:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:07 smithi171 conmon[41853]: debug 2022-01-31T21:52:07.163+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:07.164215+0000) 2022-01-31T21:52:07.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:07 smithi167 conmon[60316]: debug 2022-01-31T21:52:07.446+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:07.447427+0000) 2022-01-31T21:52:07.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:07 smithi167 conmon[54076]: debug 2022-01-31T21:52:07.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:07.447489+0000) 2022-01-31T21:52:08.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:07 smithi171 conmon[46715]: debug 2022-01-31T21:52:07.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:07.959433+0000) 2022-01-31T21:52:08.163 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:08 smithi171 conmon[51620]: debug 2022-01-31T21:52:08.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:08.053523+0000) 2022-01-31T21:52:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:07 smithi167 conmon[49112]: debug 2022-01-31T21:52:07.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:07.782723+0000) 2022-01-31T21:52:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:08 smithi171 conmon[41853]: debug 2022-01-31T21:52:08.163+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:08.164402+0000) 2022-01-31T21:52:08.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:08 smithi167 conmon[60316]: debug 2022-01-31T21:52:08.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:08.447623+0000) 2022-01-31T21:52:08.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:08 smithi167 conmon[54076]: debug 2022-01-31T21:52:08.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:08.447668+0000) 2022-01-31T21:52:09.052 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:08 smithi171 conmon[46715]: debug 2022-01-31T21:52:08.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:08.959611+0000) 2022-01-31T21:52:09.052 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:08 smithi171 conmon[35325]: debug 2022-01-31T21:52:08.752+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:52:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:08 smithi167 conmon[49112]: debug 2022-01-31T21:52:08.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:08.782902+0000) 2022-01-31T21:52:09.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:09 smithi171 conmon[51620]: debug 2022-01-31T21:52:09.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:09.053695+0000) 2022-01-31T21:52:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:09 smithi171 conmon[41853]: debug 2022-01-31T21:52:09.163+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:09.164579+0000) 2022-01-31T21:52:09.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:09 smithi167 conmon[54076]: debug 2022-01-31T21:52:09.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:09.447880+0000) 2022-01-31T21:52:09.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:09 smithi167 conmon[60316]: debug 2022-01-31T21:52:09.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:09.447813+0000) 2022-01-31T21:52:10.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:09 smithi171 conmon[46715]: debug 2022-01-31T21:52:09.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:09.959819+0000) 2022-01-31T21:52:10.163 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[51620]: debug 2022-01-31T21:52:10.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.053920+0000) 2022-01-31T21:52:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:09 smithi167 conmon[49112]: debug 2022-01-31T21:52:09.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:09.783155+0000) 2022-01-31T21:52:10.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[41853]: debug 2022-01-31T21:52:10.164+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.164719+0000) 2022-01-31T21:52:10.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:10 smithi167 conmon[49112]: debug 2022-01-31T21:52:10.496+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.496912+0000) 2022-01-31T21:52:10.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:10 smithi167 conmon[54076]: debug 2022-01-31T21:52:10.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.448052+0000) 2022-01-31T21:52:10.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:10 smithi167 conmon[54076]: debug 2022-01-31T21:52:10.496+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.497282+0000) 2022-01-31T21:52:10.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:10 smithi167 conmon[60316]: debug 2022-01-31T21:52:10.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.448006+0000) 2022-01-31T21:52:10.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:10 smithi167 conmon[60316]: debug 2022-01-31T21:52:10.496+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.497159+0000) 2022-01-31T21:52:10.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[35325]: debug 2022-01-31T21:52:10.514+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164004 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[41853]: debug 2022-01-31T21:52:10.497+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.498448+0000) 2022-01-31T21:52:10.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[46715]: debug 2022-01-31T21:52:10.496+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.497442+0000) 2022-01-31T21:52:10.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[51620]: debug 2022-01-31T21:52:10.497+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.498508+0000) 2022-01-31T21:52:11.163 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:10 smithi171 conmon[46715]: debug 2022-01-31T21:52:10.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.959989+0000) 2022-01-31T21:52:11.163 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:11 smithi171 conmon[51620]: debug 2022-01-31T21:52:11.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:11.054124+0000) 2022-01-31T21:52:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:10 smithi167 conmon[49112]: debug 2022-01-31T21:52:10.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:10.783324+0000) 2022-01-31T21:52:11.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:11 smithi171 conmon[41853]: debug 2022-01-31T21:52:11.164+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:11.164887+0000) 2022-01-31T21:52:11.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:11 smithi167 conmon[54076]: debug 2022-01-31T21:52:11.447+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:11.448230+0000) 2022-01-31T21:52:11.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:11 smithi167 conmon[60316]: debug 2022-01-31T21:52:11.447+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:11.448224+0000) 2022-01-31T21:52:12.163 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:11 smithi171 conmon[46715]: debug 2022-01-31T21:52:11.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:11.960133+0000) 2022-01-31T21:52:12.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:12 smithi171 conmon[51620]: debug 2022-01-31T21:52:12.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:12.054282+0000) 2022-01-31T21:52:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:11 smithi167 conmon[49112]: debug 2022-01-31T21:52:11.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:11.783443+0000) 2022-01-31T21:52:12.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:12 smithi171 conmon[41853]: debug 2022-01-31T21:52:12.164+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:12.165061+0000) 2022-01-31T21:52:12.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:12 smithi167 conmon[54076]: debug 2022-01-31T21:52:12.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:12.448411+0000) 2022-01-31T21:52:12.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:12 smithi167 conmon[60316]: debug 2022-01-31T21:52:12.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:12.448413+0000) 2022-01-31T21:52:13.163 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:12 smithi171 conmon[46715]: debug 2022-01-31T21:52:12.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:12.960319+0000) 2022-01-31T21:52:13.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:13 smithi171 conmon[51620]: debug 2022-01-31T21:52:13.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:13.054508+0000) 2022-01-31T21:52:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:12 smithi167 conmon[49112]: debug 2022-01-31T21:52:12.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:12.783724+0000) 2022-01-31T21:52:13.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:13 smithi171 conmon[41853]: debug 2022-01-31T21:52:13.164+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:13.165229+0000) 2022-01-31T21:52:13.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:13 smithi167 conmon[54076]: debug 2022-01-31T21:52:13.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:13.448598+0000) 2022-01-31T21:52:13.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:13 smithi167 conmon[60316]: debug 2022-01-31T21:52:13.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:13.448602+0000) 2022-01-31T21:52:14.163 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:13 smithi171 conmon[46715]: debug 2022-01-31T21:52:13.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:13.960482+0000) 2022-01-31T21:52:14.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:14 smithi171 conmon[51620]: debug 2022-01-31T21:52:14.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:14.054698+0000) 2022-01-31T21:52:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:13 smithi167 conmon[49112]: debug 2022-01-31T21:52:13.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:13.783867+0000) 2022-01-31T21:52:14.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:14 smithi171 conmon[41853]: debug 2022-01-31T21:52:14.164+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:14.165381+0000) 2022-01-31T21:52:14.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:14 smithi167 conmon[54076]: debug 2022-01-31T21:52:14.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:14.448813+0000) 2022-01-31T21:52:14.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:14 smithi167 conmon[60316]: debug 2022-01-31T21:52:14.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:14.448814+0000) 2022-01-31T21:52:15.163 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:14 smithi171 conmon[46715]: debug 2022-01-31T21:52:14.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:14.960669+0000) 2022-01-31T21:52:15.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[51620]: debug 2022-01-31T21:52:15.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.054924+0000) 2022-01-31T21:52:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:14 smithi167 conmon[49112]: debug 2022-01-31T21:52:14.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:14.784010+0000) 2022-01-31T21:52:15.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[41853]: debug 2022-01-31T21:52:15.165+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.165590+0000) 2022-01-31T21:52:15.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:15 smithi167 conmon[49112]: debug 2022-01-31T21:52:15.516+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.516644+0000) 2022-01-31T21:52:15.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:15 smithi167 conmon[54076]: debug 2022-01-31T21:52:15.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.448996+0000) 2022-01-31T21:52:15.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:15 smithi167 conmon[54076]: debug 2022-01-31T21:52:15.515+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.516121+0000) 2022-01-31T21:52:15.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:15 smithi167 conmon[60316]: debug 2022-01-31T21:52:15.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.449018+0000) 2022-01-31T21:52:15.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:15 smithi167 conmon[60316]: debug 2022-01-31T21:52:15.517+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.517527+0000) 2022-01-31T21:52:15.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[35325]: debug 2022-01-31T21:52:15.533+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164116 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[41853]: debug 2022-01-31T21:52:15.517+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.517821+0000) 2022-01-31T21:52:15.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[46715]: debug 2022-01-31T21:52:15.516+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.517599+0000) 2022-01-31T21:52:15.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[51620]: debug 2022-01-31T21:52:15.516+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.517645+0000) 2022-01-31T21:52:16.164 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:15 smithi171 conmon[46715]: debug 2022-01-31T21:52:15.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.960847+0000) 2022-01-31T21:52:16.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:16 smithi171 conmon[51620]: debug 2022-01-31T21:52:16.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:16.055090+0000) 2022-01-31T21:52:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:15 smithi167 conmon[49112]: debug 2022-01-31T21:52:15.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:15.784165+0000) 2022-01-31T21:52:16.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:16 smithi167 conmon[54076]: debug 2022-01-31T21:52:16.448+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:16.449174+0000) 2022-01-31T21:52:16.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:16 smithi167 conmon[60316]: debug 2022-01-31T21:52:16.448+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:16.449173+0000) 2022-01-31T21:52:16.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:16 smithi171 conmon[41853]: debug 2022-01-31T21:52:16.165+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:16.165738+0000) 2022-01-31T21:52:17.164 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:16 smithi171 conmon[46715]: debug 2022-01-31T21:52:16.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:16.961030+0000) 2022-01-31T21:52:17.165 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:17 smithi171 conmon[51620]: debug 2022-01-31T21:52:17.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:17.055231+0000) 2022-01-31T21:52:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:16 smithi167 conmon[49112]: debug 2022-01-31T21:52:16.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:16.784330+0000) 2022-01-31T21:52:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:17 smithi171 conmon[41853]: debug 2022-01-31T21:52:17.165+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:17.165925+0000) 2022-01-31T21:52:17.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:17 smithi167 conmon[54076]: debug 2022-01-31T21:52:17.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:17.449355+0000) 2022-01-31T21:52:17.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:17 smithi167 conmon[60316]: debug 2022-01-31T21:52:17.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:17.449334+0000) 2022-01-31T21:52:18.164 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:17 smithi171 conmon[46715]: debug 2022-01-31T21:52:17.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:17.961256+0000) 2022-01-31T21:52:18.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:18 smithi171 conmon[51620]: debug 2022-01-31T21:52:18.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:18.055397+0000) 2022-01-31T21:52:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:17 smithi167 conmon[49112]: debug 2022-01-31T21:52:17.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:17.784511+0000) 2022-01-31T21:52:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:18 smithi171 conmon[41853]: debug 2022-01-31T21:52:18.165+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:18.166083+0000) 2022-01-31T21:52:18.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:18 smithi167 conmon[54076]: debug 2022-01-31T21:52:18.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:18.449537+0000) 2022-01-31T21:52:18.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:18 smithi167 conmon[60316]: debug 2022-01-31T21:52:18.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:18.449518+0000) 2022-01-31T21:52:19.165 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:18 smithi171 conmon[46715]: debug 2022-01-31T21:52:18.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:18.961448+0000) 2022-01-31T21:52:19.165 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:19 smithi171 conmon[51620]: debug 2022-01-31T21:52:19.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:19.055608+0000) 2022-01-31T21:52:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:18 smithi167 conmon[49112]: debug 2022-01-31T21:52:18.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:18.784695+0000) 2022-01-31T21:52:19.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:19 smithi171 conmon[41853]: debug 2022-01-31T21:52:19.165+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:19.166270+0000) 2022-01-31T21:52:19.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:19 smithi167 conmon[54076]: debug 2022-01-31T21:52:19.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:19.449710+0000) 2022-01-31T21:52:19.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:19 smithi167 conmon[60316]: debug 2022-01-31T21:52:19.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:19.449708+0000) 2022-01-31T21:52:20.164 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:19 smithi171 conmon[46715]: debug 2022-01-31T21:52:19.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:19.961601+0000) 2022-01-31T21:52:20.165 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[51620]: debug 2022-01-31T21:52:20.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.055787+0000) 2022-01-31T21:52:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:19 smithi167 conmon[49112]: debug 2022-01-31T21:52:19.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:19.784904+0000) 2022-01-31T21:52:20.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[41853]: debug 2022-01-31T21:52:20.166+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.166442+0000) 2022-01-31T21:52:20.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:20 smithi167 conmon[49112]: debug 2022-01-31T21:52:20.535+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.536064+0000) 2022-01-31T21:52:20.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:20 smithi167 conmon[54076]: debug 2022-01-31T21:52:20.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.449881+0000) 2022-01-31T21:52:20.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:20 smithi167 conmon[54076]: debug 2022-01-31T21:52:20.535+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.535930+0000) 2022-01-31T21:52:20.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:20 smithi167 conmon[60316]: debug 2022-01-31T21:52:20.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.449857+0000) 2022-01-31T21:52:20.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:20 smithi167 conmon[60316]: debug 2022-01-31T21:52:20.536+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.536652+0000) 2022-01-31T21:52:20.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[35325]: debug 2022-01-31T21:52:20.552+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164227 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:20.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[51620]: debug 2022-01-31T21:52:20.536+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.536893+0000) 2022-01-31T21:52:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[41853]: debug 2022-01-31T21:52:20.535+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.536067+0000) 2022-01-31T21:52:20.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[46715]: debug 2022-01-31T21:52:20.536+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.537661+0000) 2022-01-31T21:52:21.165 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:20 smithi171 conmon[46715]: debug 2022-01-31T21:52:20.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.961779+0000) 2022-01-31T21:52:21.165 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:21 smithi171 conmon[51620]: debug 2022-01-31T21:52:21.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:21.055985+0000) 2022-01-31T21:52:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:20 smithi167 conmon[49112]: debug 2022-01-31T21:52:20.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:20.785120+0000) 2022-01-31T21:52:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:21 smithi171 conmon[41853]: debug 2022-01-31T21:52:21.166+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:21.166630+0000) 2022-01-31T21:52:21.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:21 smithi167 conmon[54076]: debug 2022-01-31T21:52:21.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:21.450059+0000) 2022-01-31T21:52:21.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:21 smithi167 conmon[60316]: debug 2022-01-31T21:52:21.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:21.450055+0000) 2022-01-31T21:52:22.165 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:21 smithi171 conmon[46715]: debug 2022-01-31T21:52:21.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:21.961938+0000) 2022-01-31T21:52:22.165 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:22 smithi171 conmon[51620]: debug 2022-01-31T21:52:22.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:22.056154+0000) 2022-01-31T21:52:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:21 smithi167 conmon[49112]: debug 2022-01-31T21:52:21.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:21.785259+0000) 2022-01-31T21:52:22.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:22 smithi171 conmon[41853]: debug 2022-01-31T21:52:22.166+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:22.166799+0000) 2022-01-31T21:52:22.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:22 smithi167 conmon[54076]: debug 2022-01-31T21:52:22.449+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:22.450186+0000) 2022-01-31T21:52:22.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:22 smithi167 conmon[60316]: debug 2022-01-31T21:52:22.449+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:22.450187+0000) 2022-01-31T21:52:23.165 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:22 smithi171 conmon[46715]: debug 2022-01-31T21:52:22.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:22.962125+0000) 2022-01-31T21:52:23.166 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:23 smithi171 conmon[51620]: debug 2022-01-31T21:52:23.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:23.056354+0000) 2022-01-31T21:52:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:22 smithi167 conmon[49112]: debug 2022-01-31T21:52:22.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:22.785458+0000) 2022-01-31T21:52:23.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:23 smithi171 conmon[41853]: debug 2022-01-31T21:52:23.166+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:23.166969+0000) 2022-01-31T21:52:23.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:23 smithi167 conmon[54076]: debug 2022-01-31T21:52:23.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:23.450374+0000) 2022-01-31T21:52:23.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:23 smithi167 conmon[60316]: debug 2022-01-31T21:52:23.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:23.450375+0000) 2022-01-31T21:52:24.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:23 smithi171 conmon[35325]: debug 2022-01-31T21:52:23.753+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:52:24.056 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:23 smithi171 conmon[46715]: debug 2022-01-31T21:52:23.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:23.962319+0000) 2022-01-31T21:52:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:23 smithi167 conmon[49112]: debug 2022-01-31T21:52:23.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:23.785646+0000) 2022-01-31T21:52:24.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:24 smithi171 conmon[41853]: debug 2022-01-31T21:52:24.166+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:24.167116+0000) 2022-01-31T21:52:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:24 smithi171 conmon[51620]: debug 2022-01-31T21:52:24.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:24.056541+0000) 2022-01-31T21:52:24.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:24 smithi167 conmon[54076]: debug 2022-01-31T21:52:24.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:24.450566+0000) 2022-01-31T21:52:24.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:24 smithi167 conmon[60316]: debug 2022-01-31T21:52:24.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:24.450578+0000) 2022-01-31T21:52:25.165 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:24 smithi171 conmon[46715]: debug 2022-01-31T21:52:24.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:24.962502+0000) 2022-01-31T21:52:25.166 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[51620]: debug 2022-01-31T21:52:25.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.056747+0000) 2022-01-31T21:52:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:24 smithi167 conmon[49112]: debug 2022-01-31T21:52:24.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:24.785817+0000) 2022-01-31T21:52:25.553 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[41853]: debug 2022-01-31T21:52:25.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.167274+0000) 2022-01-31T21:52:25.758 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:25 smithi167 conmon[49112]: debug 2022-01-31T21:52:25.554+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.555139+0000) 2022-01-31T21:52:25.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:25 smithi167 conmon[60316]: debug 2022-01-31T21:52:25.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.450801+0000) 2022-01-31T21:52:25.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:25 smithi167 conmon[60316]: debug 2022-01-31T21:52:25.554+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.555425+0000) 2022-01-31T21:52:25.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:25 smithi167 conmon[54076]: debug 2022-01-31T21:52:25.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.450797+0000) 2022-01-31T21:52:25.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:25 smithi167 conmon[54076]: debug 2022-01-31T21:52:25.562+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.562547+0000) 2022-01-31T21:52:25.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[35325]: debug 2022-01-31T21:52:25.588+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164337 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[41853]: debug 2022-01-31T21:52:25.556+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.557003+0000) 2022-01-31T21:52:25.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[46715]: debug 2022-01-31T21:52:25.555+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.556053+0000) 2022-01-31T21:52:25.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[51620]: debug 2022-01-31T21:52:25.556+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.556844+0000) 2022-01-31T21:52:26.165 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:25 smithi171 conmon[46715]: debug 2022-01-31T21:52:25.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.962666+0000) 2022-01-31T21:52:26.166 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:26 smithi171 conmon[51620]: debug 2022-01-31T21:52:26.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:26.056971+0000) 2022-01-31T21:52:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:25 smithi167 conmon[49112]: debug 2022-01-31T21:52:25.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:25.786012+0000) 2022-01-31T21:52:26.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:26 smithi171 conmon[41853]: debug 2022-01-31T21:52:26.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:26.167469+0000) 2022-01-31T21:52:26.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:26 smithi167 conmon[54076]: debug 2022-01-31T21:52:26.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:26.451023+0000) 2022-01-31T21:52:26.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:26 smithi167 conmon[60316]: debug 2022-01-31T21:52:26.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:26.451026+0000) 2022-01-31T21:52:27.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:26 smithi171 conmon[46715]: debug 2022-01-31T21:52:26.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:26.962805+0000) 2022-01-31T21:52:27.166 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:27 smithi171 conmon[51620]: debug 2022-01-31T21:52:27.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:27.057110+0000) 2022-01-31T21:52:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:26 smithi167 conmon[49112]: debug 2022-01-31T21:52:26.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:26.786175+0000) 2022-01-31T21:52:27.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:27 smithi171 conmon[41853]: debug 2022-01-31T21:52:27.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:27.167605+0000) 2022-01-31T21:52:27.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:27 smithi167 conmon[54076]: debug 2022-01-31T21:52:27.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:27.451164+0000) 2022-01-31T21:52:27.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:27 smithi167 conmon[60316]: debug 2022-01-31T21:52:27.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:27.451165+0000) 2022-01-31T21:52:28.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:27 smithi171 conmon[46715]: debug 2022-01-31T21:52:27.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:27.962972+0000) 2022-01-31T21:52:28.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:28 smithi171 conmon[51620]: debug 2022-01-31T21:52:28.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:28.057297+0000) 2022-01-31T21:52:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:27 smithi167 conmon[49112]: debug 2022-01-31T21:52:27.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:27.786376+0000) 2022-01-31T21:52:28.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:28 smithi171 conmon[41853]: debug 2022-01-31T21:52:28.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:28.167788+0000) 2022-01-31T21:52:28.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:28 smithi167 conmon[54076]: debug 2022-01-31T21:52:28.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:28.451291+0000) 2022-01-31T21:52:28.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:28 smithi167 conmon[60316]: debug 2022-01-31T21:52:28.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:28.451337+0000) 2022-01-31T21:52:29.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:28 smithi171 conmon[46715]: debug 2022-01-31T21:52:28.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:28.963164+0000) 2022-01-31T21:52:29.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:29 smithi171 conmon[51620]: debug 2022-01-31T21:52:29.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:29.057464+0000) 2022-01-31T21:52:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:28 smithi167 conmon[49112]: debug 2022-01-31T21:52:28.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:28.786585+0000) 2022-01-31T21:52:29.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:29 smithi171 conmon[41853]: debug 2022-01-31T21:52:29.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:29.167973+0000) 2022-01-31T21:52:29.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:29 smithi167 conmon[54076]: debug 2022-01-31T21:52:29.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:29.451487+0000) 2022-01-31T21:52:29.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:29 smithi167 conmon[60316]: debug 2022-01-31T21:52:29.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:29.451536+0000) 2022-01-31T21:52:30.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:29 smithi171 conmon[46715]: debug 2022-01-31T21:52:29.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:29.963352+0000) 2022-01-31T21:52:30.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[51620]: debug 2022-01-31T21:52:30.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.057665+0000) 2022-01-31T21:52:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:29 smithi167 conmon[49112]: debug 2022-01-31T21:52:29.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:29.786797+0000) 2022-01-31T21:52:30.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[41853]: debug 2022-01-31T21:52:30.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.168151+0000) 2022-01-31T21:52:30.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[41853]: debug 2022-01-31T21:52:30.589+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.590606+0000) 2022-01-31T21:52:30.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:30 smithi167 conmon[49112]: debug 2022-01-31T21:52:30.589+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.591332+0000) 2022-01-31T21:52:30.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:30 smithi167 conmon[54076]: debug 2022-01-31T21:52:30.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.451689+0000) 2022-01-31T21:52:30.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:30 smithi167 conmon[54076]: debug 2022-01-31T21:52:30.590+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.591932+0000) 2022-01-31T21:52:30.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:30 smithi167 conmon[60316]: debug 2022-01-31T21:52:30.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.451706+0000) 2022-01-31T21:52:30.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:30 smithi167 conmon[60316]: debug 2022-01-31T21:52:30.589+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.590912+0000) 2022-01-31T21:52:30.855 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[35325]: debug 2022-01-31T21:52:30.607+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:30.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[46715]: debug 2022-01-31T21:52:30.590+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.591222+0000) 2022-01-31T21:52:30.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[51620]: debug 2022-01-31T21:52:30.590+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.591649+0000) 2022-01-31T21:52:31.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:30 smithi171 conmon[46715]: debug 2022-01-31T21:52:30.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.963560+0000) 2022-01-31T21:52:31.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:31 smithi171 conmon[51620]: debug 2022-01-31T21:52:31.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:31.057858+0000) 2022-01-31T21:52:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:30 smithi167 conmon[49112]: debug 2022-01-31T21:52:30.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:30.786987+0000) 2022-01-31T21:52:31.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:31 smithi171 conmon[41853]: debug 2022-01-31T21:52:31.168+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:31.168337+0000) 2022-01-31T21:52:31.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:31 smithi167 conmon[54076]: debug 2022-01-31T21:52:31.450+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:31.451890+0000) 2022-01-31T21:52:31.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:31 smithi167 conmon[60316]: debug 2022-01-31T21:52:31.450+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:31.451900+0000) 2022-01-31T21:52:32.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:31 smithi171 conmon[46715]: debug 2022-01-31T21:52:31.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:31.963714+0000) 2022-01-31T21:52:32.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:32 smithi171 conmon[51620]: debug 2022-01-31T21:52:32.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:32.058003+0000) 2022-01-31T21:52:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:31 smithi167 conmon[49112]: debug 2022-01-31T21:52:31.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:31.787144+0000) 2022-01-31T21:52:32.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:32 smithi171 conmon[41853]: debug 2022-01-31T21:52:32.168+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:32.168488+0000) 2022-01-31T21:52:32.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:32 smithi167 conmon[60316]: debug 2022-01-31T21:52:32.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:32.452091+0000) 2022-01-31T21:52:32.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:32 smithi167 conmon[54076]: debug 2022-01-31T21:52:32.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:32.452090+0000) 2022-01-31T21:52:33.167 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:32 smithi171 conmon[46715]: debug 2022-01-31T21:52:32.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:32.963917+0000) 2022-01-31T21:52:33.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:33 smithi171 conmon[51620]: debug 2022-01-31T21:52:33.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:33.058170+0000) 2022-01-31T21:52:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:32 smithi167 conmon[49112]: debug 2022-01-31T21:52:32.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:32.787317+0000) 2022-01-31T21:52:33.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:33 smithi171 conmon[41853]: debug 2022-01-31T21:52:33.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:33.168687+0000) 2022-01-31T21:52:33.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:33 smithi167 conmon[54076]: debug 2022-01-31T21:52:33.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:33.452289+0000) 2022-01-31T21:52:33.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:33 smithi167 conmon[60316]: debug 2022-01-31T21:52:33.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:33.452291+0000) 2022-01-31T21:52:34.167 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:33 smithi171 conmon[46715]: debug 2022-01-31T21:52:33.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:33.964116+0000) 2022-01-31T21:52:34.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:34 smithi171 conmon[51620]: debug 2022-01-31T21:52:34.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:34.058326+0000) 2022-01-31T21:52:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:33 smithi167 conmon[49112]: debug 2022-01-31T21:52:33.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:33.787507+0000) 2022-01-31T21:52:34.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:34 smithi171 conmon[41853]: debug 2022-01-31T21:52:34.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:34.168895+0000) 2022-01-31T21:52:34.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:34 smithi167 conmon[54076]: debug 2022-01-31T21:52:34.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:34.452518+0000) 2022-01-31T21:52:34.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:34 smithi167 conmon[60316]: debug 2022-01-31T21:52:34.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:34.452527+0000) 2022-01-31T21:52:35.167 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:34 smithi171 conmon[46715]: debug 2022-01-31T21:52:34.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:34.964302+0000) 2022-01-31T21:52:35.168 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[51620]: debug 2022-01-31T21:52:35.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.058472+0000) 2022-01-31T21:52:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:34 smithi167 conmon[49112]: debug 2022-01-31T21:52:34.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:34.787689+0000) 2022-01-31T21:52:35.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[41853]: debug 2022-01-31T21:52:35.168+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.169098+0000) 2022-01-31T21:52:35.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:35 smithi167 conmon[49112]: debug 2022-01-31T21:52:35.609+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.610533+0000) 2022-01-31T21:52:35.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:35 smithi167 conmon[54076]: debug 2022-01-31T21:52:35.451+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.452737+0000) 2022-01-31T21:52:35.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:35 smithi167 conmon[54076]: debug 2022-01-31T21:52:35.609+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.611255+0000) 2022-01-31T21:52:35.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:35 smithi167 conmon[60316]: debug 2022-01-31T21:52:35.451+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.452734+0000) 2022-01-31T21:52:35.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:35 smithi167 conmon[60316]: debug 2022-01-31T21:52:35.609+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.610765+0000) 2022-01-31T21:52:35.882 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[35325]: debug 2022-01-31T21:52:35.625+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164572 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:35.900 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[41853]: debug 2022-01-31T21:52:35.609+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.611087+0000) 2022-01-31T21:52:35.901 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[46715]: debug 2022-01-31T21:52:35.609+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.611721+0000) 2022-01-31T21:52:35.901 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[51620]: debug 2022-01-31T21:52:35.608+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.610471+0000) 2022-01-31T21:52:36.167 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:35 smithi171 conmon[46715]: debug 2022-01-31T21:52:35.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.964440+0000) 2022-01-31T21:52:36.168 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:36 smithi171 conmon[51620]: debug 2022-01-31T21:52:36.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:36.058667+0000) 2022-01-31T21:52:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:35 smithi167 conmon[49112]: debug 2022-01-31T21:52:35.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:35.787975+0000) 2022-01-31T21:52:36.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:36 smithi171 conmon[41853]: debug 2022-01-31T21:52:36.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:36.169214+0000) 2022-01-31T21:52:36.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:36 smithi167 conmon[54076]: debug 2022-01-31T21:52:36.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:36.452924+0000) 2022-01-31T21:52:36.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:36 smithi167 conmon[60316]: debug 2022-01-31T21:52:36.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:36.452919+0000) 2022-01-31T21:52:37.167 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:36 smithi171 conmon[46715]: debug 2022-01-31T21:52:36.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:36.964580+0000) 2022-01-31T21:52:37.168 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:37 smithi171 conmon[51620]: debug 2022-01-31T21:52:37.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:37.058789+0000) 2022-01-31T21:52:37.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:36 smithi167 conmon[49112]: debug 2022-01-31T21:52:36.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:36.788117+0000) 2022-01-31T21:52:37.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:37 smithi171 conmon[41853]: debug 2022-01-31T21:52:37.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:37.169375+0000) 2022-01-31T21:52:37.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:37 smithi167 conmon[54076]: debug 2022-01-31T21:52:37.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:37.453098+0000) 2022-01-31T21:52:37.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:37 smithi167 conmon[60316]: debug 2022-01-31T21:52:37.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:37.453096+0000) 2022-01-31T21:52:38.167 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:37 smithi171 conmon[46715]: debug 2022-01-31T21:52:37.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:37.964791+0000) 2022-01-31T21:52:38.168 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:38 smithi171 conmon[51620]: debug 2022-01-31T21:52:38.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:38.058938+0000) 2022-01-31T21:52:38.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:37 smithi167 conmon[49112]: debug 2022-01-31T21:52:37.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:37.788228+0000) 2022-01-31T21:52:38.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:38 smithi171 conmon[41853]: debug 2022-01-31T21:52:38.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:38.169531+0000) 2022-01-31T21:52:38.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:38 smithi167 conmon[54076]: debug 2022-01-31T21:52:38.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:38.453317+0000) 2022-01-31T21:52:38.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:38 smithi167 conmon[60316]: debug 2022-01-31T21:52:38.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:38.453315+0000) 2022-01-31T21:52:39.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:38 smithi171 conmon[35325]: debug 2022-01-31T21:52:38.753+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:52:39.058 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:38 smithi171 conmon[46715]: debug 2022-01-31T21:52:38.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:38.964989+0000) 2022-01-31T21:52:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:38 smithi167 conmon[49112]: debug 2022-01-31T21:52:38.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:38.788368+0000) 2022-01-31T21:52:39.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:39 smithi171 conmon[41853]: debug 2022-01-31T21:52:39.167+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:39.169720+0000) 2022-01-31T21:52:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:39 smithi171 conmon[51620]: debug 2022-01-31T21:52:39.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:39.059106+0000) 2022-01-31T21:52:39.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:39 smithi167 conmon[54076]: debug 2022-01-31T21:52:39.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:39.453471+0000) 2022-01-31T21:52:39.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:39 smithi167 conmon[60316]: debug 2022-01-31T21:52:39.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:39.453529+0000) 2022-01-31T21:52:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:39 smithi167 conmon[49112]: debug 2022-01-31T21:52:39.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:39.788597+0000) 2022-01-31T21:52:40.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:39 smithi171 conmon[46715]: debug 2022-01-31T21:52:39.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:39.965158+0000) 2022-01-31T21:52:40.170 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[51620]: debug 2022-01-31T21:52:40.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.059303+0000) 2022-01-31T21:52:40.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[41853]: debug 2022-01-31T21:52:40.169+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.169927+0000) 2022-01-31T21:52:40.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:40 smithi167 conmon[49112]: debug 2022-01-31T21:52:40.628+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.630440+0000) 2022-01-31T21:52:40.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:40 smithi167 conmon[54076]: debug 2022-01-31T21:52:40.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.453636+0000) 2022-01-31T21:52:40.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:40 smithi167 conmon[54076]: debug 2022-01-31T21:52:40.628+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.629885+0000) 2022-01-31T21:52:40.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:40 smithi167 conmon[60316]: debug 2022-01-31T21:52:40.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.453725+0000) 2022-01-31T21:52:40.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:40 smithi167 conmon[60316]: debug 2022-01-31T21:52:40.629+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.631056+0000) 2022-01-31T21:52:40.884 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[35325]: debug 2022-01-31T21:52:40.645+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164683 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:40.885 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[41853]: debug 2022-01-31T21:52:40.629+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.630933+0000) 2022-01-31T21:52:40.885 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[46715]: debug 2022-01-31T21:52:40.628+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.630682+0000) 2022-01-31T21:52:40.886 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[51620]: debug 2022-01-31T21:52:40.628+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.629968+0000) 2022-01-31T21:52:41.168 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:40 smithi171 conmon[46715]: debug 2022-01-31T21:52:40.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.965306+0000) 2022-01-31T21:52:41.169 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:41 smithi171 conmon[51620]: debug 2022-01-31T21:52:41.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:41.059479+0000) 2022-01-31T21:52:41.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:40 smithi167 conmon[49112]: debug 2022-01-31T21:52:40.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:40.788750+0000) 2022-01-31T21:52:41.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:41 smithi171 conmon[41853]: debug 2022-01-31T21:52:41.169+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:41.170117+0000) 2022-01-31T21:52:41.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:41 smithi167 conmon[54076]: debug 2022-01-31T21:52:41.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:41.453826+0000) 2022-01-31T21:52:41.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:41 smithi167 conmon[60316]: debug 2022-01-31T21:52:41.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:41.453899+0000) 2022-01-31T21:52:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:41 smithi167 conmon[49112]: debug 2022-01-31T21:52:41.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:41.788881+0000) 2022-01-31T21:52:42.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:41 smithi171 conmon[46715]: debug 2022-01-31T21:52:41.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:41.965456+0000) 2022-01-31T21:52:42.170 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:42 smithi171 conmon[51620]: debug 2022-01-31T21:52:42.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:42.059624+0000) 2022-01-31T21:52:42.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:42 smithi171 conmon[41853]: debug 2022-01-31T21:52:42.169+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:42.170271+0000) 2022-01-31T21:52:42.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:42 smithi167 conmon[54076]: debug 2022-01-31T21:52:42.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:42.454000+0000) 2022-01-31T21:52:42.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:42 smithi167 conmon[60316]: debug 2022-01-31T21:52:42.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:42.454051+0000) 2022-01-31T21:52:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:42 smithi167 conmon[49112]: debug 2022-01-31T21:52:42.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:42.789049+0000) 2022-01-31T21:52:43.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:42 smithi171 conmon[46715]: debug 2022-01-31T21:52:42.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:42.965642+0000) 2022-01-31T21:52:43.169 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:43 smithi171 conmon[51620]: debug 2022-01-31T21:52:43.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:43.059800+0000) 2022-01-31T21:52:43.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:43 smithi171 conmon[41853]: debug 2022-01-31T21:52:43.169+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:43.170496+0000) 2022-01-31T21:52:43.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:43 smithi167 conmon[54076]: debug 2022-01-31T21:52:43.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:43.454220+0000) 2022-01-31T21:52:43.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:43 smithi167 conmon[60316]: debug 2022-01-31T21:52:43.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:43.454257+0000) 2022-01-31T21:52:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:43 smithi167 conmon[49112]: debug 2022-01-31T21:52:43.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:43.789240+0000) 2022-01-31T21:52:44.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:43 smithi171 conmon[46715]: debug 2022-01-31T21:52:43.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:43.965862+0000) 2022-01-31T21:52:44.169 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:44 smithi171 conmon[51620]: debug 2022-01-31T21:52:44.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:44.059949+0000) 2022-01-31T21:52:44.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:44 smithi171 conmon[41853]: debug 2022-01-31T21:52:44.169+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:44.170654+0000) 2022-01-31T21:52:44.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:44 smithi167 conmon[54076]: debug 2022-01-31T21:52:44.452+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:44.454353+0000) 2022-01-31T21:52:44.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:44 smithi167 conmon[60316]: debug 2022-01-31T21:52:44.452+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:44.454427+0000) 2022-01-31T21:52:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:44 smithi167 conmon[49112]: debug 2022-01-31T21:52:44.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:44.789439+0000) 2022-01-31T21:52:45.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:44 smithi171 conmon[46715]: debug 2022-01-31T21:52:44.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:44.966074+0000) 2022-01-31T21:52:45.169 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[51620]: debug 2022-01-31T21:52:45.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.060099+0000) 2022-01-31T21:52:45.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[41853]: debug 2022-01-31T21:52:45.170+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.170815+0000) 2022-01-31T21:52:45.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:45 smithi167 conmon[49112]: debug 2022-01-31T21:52:45.648+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.649986+0000) 2022-01-31T21:52:45.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:45 smithi167 conmon[54076]: debug 2022-01-31T21:52:45.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.454535+0000) 2022-01-31T21:52:45.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:45 smithi167 conmon[54076]: debug 2022-01-31T21:52:45.647+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.648964+0000) 2022-01-31T21:52:45.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:45 smithi167 conmon[60316]: debug 2022-01-31T21:52:45.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.454578+0000) 2022-01-31T21:52:45.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:45 smithi167 conmon[60316]: debug 2022-01-31T21:52:45.649+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.650612+0000) 2022-01-31T21:52:45.964 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[35325]: debug 2022-01-31T21:52:45.666+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164794 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:45.965 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[41853]: debug 2022-01-31T21:52:45.648+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.649711+0000) 2022-01-31T21:52:45.966 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[46715]: debug 2022-01-31T21:52:45.648+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.649629+0000) 2022-01-31T21:52:45.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[51620]: debug 2022-01-31T21:52:45.649+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.650174+0000) 2022-01-31T21:52:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:45 smithi167 conmon[49112]: debug 2022-01-31T21:52:45.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.789611+0000) 2022-01-31T21:52:46.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:45 smithi171 conmon[46715]: debug 2022-01-31T21:52:45.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:45.966230+0000) 2022-01-31T21:52:46.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:46 smithi171 conmon[41853]: debug 2022-01-31T21:52:46.170+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:46.171023+0000) 2022-01-31T21:52:46.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:46 smithi171 conmon[51620]: debug 2022-01-31T21:52:46.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:46.060249+0000) 2022-01-31T21:52:46.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:46 smithi167 conmon[54076]: debug 2022-01-31T21:52:46.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:46.454664+0000) 2022-01-31T21:52:46.512 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:46 smithi167 conmon[60316]: debug 2022-01-31T21:52:46.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:46.454754+0000) 2022-01-31T21:52:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:46 smithi167 conmon[49112]: debug 2022-01-31T21:52:46.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:46.789729+0000) 2022-01-31T21:52:47.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:46 smithi171 conmon[46715]: debug 2022-01-31T21:52:46.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:46.966378+0000) 2022-01-31T21:52:47.169 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:47 smithi171 conmon[51620]: debug 2022-01-31T21:52:47.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:47.060384+0000) 2022-01-31T21:52:47.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:47 smithi171 conmon[41853]: debug 2022-01-31T21:52:47.170+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:47.171186+0000) 2022-01-31T21:52:47.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:47 smithi167 conmon[60316]: debug 2022-01-31T21:52:47.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:47.454902+0000) 2022-01-31T21:52:47.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:47 smithi167 conmon[54076]: debug 2022-01-31T21:52:47.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:47.454836+0000) 2022-01-31T21:52:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:47 smithi167 conmon[49112]: debug 2022-01-31T21:52:47.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:47.789946+0000) 2022-01-31T21:52:48.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:47 smithi171 conmon[46715]: debug 2022-01-31T21:52:47.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:47.966537+0000) 2022-01-31T21:52:48.170 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:48 smithi171 conmon[51620]: debug 2022-01-31T21:52:48.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:48.060554+0000) 2022-01-31T21:52:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:48 smithi171 conmon[41853]: debug 2022-01-31T21:52:48.170+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:48.171354+0000) 2022-01-31T21:52:48.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:48 smithi167 conmon[60316]: debug 2022-01-31T21:52:48.454+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:48.455090+0000) 2022-01-31T21:52:48.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:48 smithi167 conmon[54076]: debug 2022-01-31T21:52:48.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:48.455040+0000) 2022-01-31T21:52:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:48 smithi167 conmon[49112]: debug 2022-01-31T21:52:48.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:48.790139+0000) 2022-01-31T21:52:49.169 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:48 smithi171 conmon[46715]: debug 2022-01-31T21:52:48.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:48.966727+0000) 2022-01-31T21:52:49.170 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:49 smithi171 conmon[51620]: debug 2022-01-31T21:52:49.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:49.060787+0000) 2022-01-31T21:52:49.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:49 smithi171 conmon[41853]: debug 2022-01-31T21:52:49.170+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:49.171508+0000) 2022-01-31T21:52:49.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:49 smithi167 conmon[54076]: debug 2022-01-31T21:52:49.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:49.455200+0000) 2022-01-31T21:52:49.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:49 smithi167 conmon[60316]: debug 2022-01-31T21:52:49.454+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:49.455276+0000) 2022-01-31T21:52:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:49 smithi167 conmon[49112]: debug 2022-01-31T21:52:49.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:49.790329+0000) 2022-01-31T21:52:50.170 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:49 smithi171 conmon[46715]: debug 2022-01-31T21:52:49.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:49.966929+0000) 2022-01-31T21:52:50.170 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[51620]: debug 2022-01-31T21:52:50.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.060970+0000) 2022-01-31T21:52:50.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[41853]: debug 2022-01-31T21:52:50.170+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.171698+0000) 2022-01-31T21:52:50.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:50 smithi167 conmon[49112]: debug 2022-01-31T21:52:50.667+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.669222+0000) 2022-01-31T21:52:50.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:50 smithi167 conmon[54076]: debug 2022-01-31T21:52:50.453+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.455414+0000) 2022-01-31T21:52:50.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:50 smithi167 conmon[54076]: debug 2022-01-31T21:52:50.666+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.668435+0000) 2022-01-31T21:52:50.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:50 smithi167 conmon[60316]: debug 2022-01-31T21:52:50.453+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.455437+0000) 2022-01-31T21:52:50.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:50 smithi167 conmon[60316]: debug 2022-01-31T21:52:50.667+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.669019+0000) 2022-01-31T21:52:50.965 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[35325]: debug 2022-01-31T21:52:50.685+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 164905 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:50.966 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[41853]: debug 2022-01-31T21:52:50.670+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.670816+0000) 2022-01-31T21:52:50.967 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[46715]: debug 2022-01-31T21:52:50.668+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.668878+0000) 2022-01-31T21:52:50.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[51620]: debug 2022-01-31T21:52:50.668+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.669459+0000) 2022-01-31T21:52:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:50 smithi167 conmon[49112]: debug 2022-01-31T21:52:50.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.790540+0000) 2022-01-31T21:52:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:50 smithi171 conmon[46715]: debug 2022-01-31T21:52:50.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:50.967133+0000) 2022-01-31T21:52:51.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:51 smithi171 conmon[51620]: debug 2022-01-31T21:52:51.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:51.061129+0000) 2022-01-31T21:52:51.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:51 smithi171 conmon[41853]: debug 2022-01-31T21:52:51.171+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:51.171893+0000) 2022-01-31T21:52:51.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:51 smithi167 conmon[60316]: debug 2022-01-31T21:52:51.454+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:51.455664+0000) 2022-01-31T21:52:51.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:51 smithi167 conmon[54076]: debug 2022-01-31T21:52:51.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:51.455597+0000) 2022-01-31T21:52:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:51 smithi167 conmon[49112]: debug 2022-01-31T21:52:51.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:51.790687+0000) 2022-01-31T21:52:52.170 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:51 smithi171 conmon[46715]: debug 2022-01-31T21:52:51.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:51.967273+0000) 2022-01-31T21:52:52.170 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:52 smithi171 conmon[51620]: debug 2022-01-31T21:52:52.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:52.061264+0000) 2022-01-31T21:52:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:52 smithi171 conmon[41853]: debug 2022-01-31T21:52:52.171+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:52.172050+0000) 2022-01-31T21:52:52.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:52 smithi167 conmon[54076]: debug 2022-01-31T21:52:52.454+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:52.455781+0000) 2022-01-31T21:52:52.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:52 smithi167 conmon[60316]: debug 2022-01-31T21:52:52.454+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:52.455795+0000) 2022-01-31T21:52:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:52 smithi167 conmon[49112]: debug 2022-01-31T21:52:52.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:52.790894+0000) 2022-01-31T21:52:53.170 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:52 smithi171 conmon[46715]: debug 2022-01-31T21:52:52.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:52.967496+0000) 2022-01-31T21:52:53.171 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:53 smithi171 conmon[51620]: debug 2022-01-31T21:52:53.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:53.061419+0000) 2022-01-31T21:52:53.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:53 smithi171 conmon[41853]: debug 2022-01-31T21:52:53.171+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:53.172191+0000) 2022-01-31T21:52:53.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:53 smithi167 conmon[54076]: debug 2022-01-31T21:52:53.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:53.455949+0000) 2022-01-31T21:52:53.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:53 smithi167 conmon[60316]: debug 2022-01-31T21:52:53.455+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:53.455966+0000) 2022-01-31T21:52:54.060 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:53 smithi171 conmon[35325]: debug 2022-01-31T21:52:53.754+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:52:54.060 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:53 smithi171 conmon[46715]: debug 2022-01-31T21:52:53.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:53.967692+0000) 2022-01-31T21:52:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:53 smithi167 conmon[49112]: debug 2022-01-31T21:52:53.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:53.791056+0000) 2022-01-31T21:52:54.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:54 smithi171 conmon[41853]: debug 2022-01-31T21:52:54.171+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:54.172349+0000) 2022-01-31T21:52:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:54 smithi171 conmon[51620]: debug 2022-01-31T21:52:54.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:54.061619+0000) 2022-01-31T21:52:54.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:54 smithi167 conmon[54076]: debug 2022-01-31T21:52:54.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:54.456202+0000) 2022-01-31T21:52:54.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:54 smithi167 conmon[60316]: debug 2022-01-31T21:52:54.455+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:54.456183+0000) 2022-01-31T21:52:54.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:54 smithi167 conmon[49112]: debug 2022-01-31T21:52:54.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:54.791253+0000) 2022-01-31T21:52:55.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:54 smithi171 conmon[46715]: debug 2022-01-31T21:52:54.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:54.967879+0000) 2022-01-31T21:52:55.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[51620]: debug 2022-01-31T21:52:55.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.061794+0000) 2022-01-31T21:52:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[41853]: debug 2022-01-31T21:52:55.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.172530+0000) 2022-01-31T21:52:55.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[49112]: debug 2022-01-31T21:52:55.689+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.689708+0000) 2022-01-31T21:52:55.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[54076]: debug 2022-01-31T21:52:55.455+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.456415+0000) 2022-01-31T21:52:55.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[54076]: debug 2022-01-31T21:52:55.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[54076]: 2022-01-31T21:52:55.688+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.688598+0000) 2022-01-31T21:52:55.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[60316]: debug 2022-01-31T21:52:55.455+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.456416+0000) 2022-01-31T21:52:55.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[60316]: debug 2022-01-31T21:52:55.688+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.688720+0000) 2022-01-31T21:52:55.966 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[35325]: debug 2022-01-31T21:52:55.705+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165015 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:52:55.967 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[41853]: debug 2022-01-31T21:52:55.688+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.689637+0000) 2022-01-31T21:52:55.968 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[46715]: debug 2022-01-31T21:52:55.687+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.688240+0000) 2022-01-31T21:52:55.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[51620]: debug 2022-01-31T21:52:55.688+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.689110+0000) 2022-01-31T21:52:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:55 smithi167 conmon[49112]: debug 2022-01-31T21:52:55.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.791487+0000) 2022-01-31T21:52:56.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:55 smithi171 conmon[46715]: debug 2022-01-31T21:52:55.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:55.968082+0000) 2022-01-31T21:52:56.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:56 smithi171 conmon[51620]: debug 2022-01-31T21:52:56.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:56.061946+0000) 2022-01-31T21:52:56.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:56 smithi171 conmon[41853]: debug 2022-01-31T21:52:56.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:56.172722+0000) 2022-01-31T21:52:56.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:56 smithi167 conmon[54076]: debug 2022-01-31T21:52:56.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:56.456596+0000) 2022-01-31T21:52:56.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:56 smithi167 conmon[60316]: debug 2022-01-31T21:52:56.456+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:56.456604+0000) 2022-01-31T21:52:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:56 smithi167 conmon[49112]: debug 2022-01-31T21:52:56.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:56.791611+0000) 2022-01-31T21:52:57.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:56 smithi171 conmon[46715]: debug 2022-01-31T21:52:56.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:56.968256+0000) 2022-01-31T21:52:57.171 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:57 smithi171 conmon[51620]: debug 2022-01-31T21:52:57.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:57.062099+0000) 2022-01-31T21:52:57.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:57 smithi171 conmon[41853]: debug 2022-01-31T21:52:57.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:57.172916+0000) 2022-01-31T21:52:57.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:57 smithi167 conmon[54076]: debug 2022-01-31T21:52:57.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:57.456755+0000) 2022-01-31T21:52:57.662 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:57 smithi167 conmon[60316]: debug 2022-01-31T21:52:57.456+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:57.456779+0000) 2022-01-31T21:52:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:57 smithi167 conmon[49112]: debug 2022-01-31T21:52:57.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:57.791827+0000) 2022-01-31T21:52:58.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:57 smithi171 conmon[46715]: debug 2022-01-31T21:52:57.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:57.968468+0000) 2022-01-31T21:52:58.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:58 smithi171 conmon[51620]: debug 2022-01-31T21:52:58.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:58.062278+0000) 2022-01-31T21:52:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:58 smithi171 conmon[41853]: debug 2022-01-31T21:52:58.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:58.173108+0000) 2022-01-31T21:52:58.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:58 smithi167 conmon[54076]: debug 2022-01-31T21:52:58.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:58.456942+0000) 2022-01-31T21:52:58.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:58 smithi167 conmon[60316]: debug 2022-01-31T21:52:58.456+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:58.456941+0000) 2022-01-31T21:52:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:58 smithi167 conmon[49112]: debug 2022-01-31T21:52:58.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:58.792026+0000) 2022-01-31T21:52:59.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:58 smithi171 conmon[46715]: debug 2022-01-31T21:52:58.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:58.968631+0000) 2022-01-31T21:52:59.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:52:59 smithi171 conmon[51620]: debug 2022-01-31T21:52:59.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:59.062472+0000) 2022-01-31T21:52:59.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:52:59 smithi171 conmon[41853]: debug 2022-01-31T21:52:59.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:59.173270+0000) 2022-01-31T21:52:59.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:52:59 smithi167 conmon[54076]: debug 2022-01-31T21:52:59.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:59.457139+0000) 2022-01-31T21:52:59.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:52:59 smithi167 conmon[60316]: debug 2022-01-31T21:52:59.456+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:59.457135+0000) 2022-01-31T21:53:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:52:59 smithi167 conmon[49112]: debug 2022-01-31T21:52:59.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:59.792214+0000) 2022-01-31T21:53:00.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:52:59 smithi171 conmon[46715]: debug 2022-01-31T21:52:59.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:52:59.968791+0000) 2022-01-31T21:53:00.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[51620]: debug 2022-01-31T21:53:00.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.062643+0000) 2022-01-31T21:53:00.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[41853]: debug 2022-01-31T21:53:00.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.173429+0000) 2022-01-31T21:53:00.706 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:00 smithi167 conmon[60316]: debug 2022-01-31T21:53:00.456+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.457337+0000) 2022-01-31T21:53:00.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:00 smithi167 conmon[54076]: debug 2022-01-31T21:53:00.456+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.457337+0000) 2022-01-31T21:53:00.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:00 smithi167 conmon[54076]: debug 2022-01-31T21:53:00.706+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.707156+0000) 2022-01-31T21:53:00.967 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[35325]: debug 2022-01-31T21:53:00.725+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:00.968 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[41853]: debug 2022-01-31T21:53:00.708+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.709051+0000) 2022-01-31T21:53:00.968 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[46715]: debug 2022-01-31T21:53:00.707+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.708593+0000) 2022-01-31T21:53:00.969 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[51620]: debug 2022-01-31T21:53:00.707+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.708187+0000) 2022-01-31T21:53:01.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:00 smithi167 conmon[60316]: debug 2022-01-31T21:53:00.707+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.707601+0000) 2022-01-31T21:53:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:00 smithi167 conmon[49112]: debug 2022-01-31T21:53:00.708+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.708813+0000) 2022-01-31T21:53:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:00 smithi167 conmon[49112]: debug 2022-01-31T21:53:00.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.792425+0000) 2022-01-31T21:53:01.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:00 smithi171 conmon[46715]: debug 2022-01-31T21:53:00.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:00.969000+0000) 2022-01-31T21:53:01.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:01 smithi171 conmon[41853]: debug 2022-01-31T21:53:01.172+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:01.173555+0000) 2022-01-31T21:53:01.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:01 smithi171 conmon[51620]: debug 2022-01-31T21:53:01.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:01.062826+0000) 2022-01-31T21:53:01.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:01 smithi167 conmon[60316]: debug 2022-01-31T21:53:01.457+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:01.457535+0000) 2022-01-31T21:53:01.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:01 smithi167 conmon[54076]: debug 2022-01-31T21:53:01.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:01.457536+0000) 2022-01-31T21:53:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:01 smithi167 conmon[49112]: debug 2022-01-31T21:53:01.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:01.792622+0000) 2022-01-31T21:53:02.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:01 smithi171 conmon[46715]: debug 2022-01-31T21:53:01.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:01.969171+0000) 2022-01-31T21:53:02.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:02 smithi171 conmon[51620]: debug 2022-01-31T21:53:02.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:02.062979+0000) 2022-01-31T21:53:02.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:02 smithi171 conmon[41853]: debug 2022-01-31T21:53:02.173+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:02.173682+0000) 2022-01-31T21:53:02.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:02 smithi167 conmon[54076]: debug 2022-01-31T21:53:02.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:02.457716+0000) 2022-01-31T21:53:02.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:02 smithi167 conmon[60316]: debug 2022-01-31T21:53:02.457+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:02.457700+0000) 2022-01-31T21:53:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:02 smithi167 conmon[49112]: debug 2022-01-31T21:53:02.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:02.792847+0000) 2022-01-31T21:53:03.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:02 smithi171 conmon[46715]: debug 2022-01-31T21:53:02.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:02.969335+0000) 2022-01-31T21:53:03.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:03 smithi171 conmon[51620]: debug 2022-01-31T21:53:03.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:03.063170+0000) 2022-01-31T21:53:03.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:03 smithi171 conmon[41853]: debug 2022-01-31T21:53:03.173+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:03.173836+0000) 2022-01-31T21:53:03.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:03 smithi167 conmon[54076]: debug 2022-01-31T21:53:03.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:03.457914+0000) 2022-01-31T21:53:03.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:03 smithi167 conmon[60316]: debug 2022-01-31T21:53:03.457+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:03.457913+0000) 2022-01-31T21:53:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:03 smithi167 conmon[49112]: debug 2022-01-31T21:53:03.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:03.793065+0000) 2022-01-31T21:53:04.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:03 smithi171 conmon[46715]: debug 2022-01-31T21:53:03.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:03.969520+0000) 2022-01-31T21:53:04.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:04 smithi171 conmon[51620]: debug 2022-01-31T21:53:04.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:04.063350+0000) 2022-01-31T21:53:04.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:04 smithi171 conmon[41853]: debug 2022-01-31T21:53:04.173+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:04.173942+0000) 2022-01-31T21:53:04.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:04 smithi167 conmon[54076]: debug 2022-01-31T21:53:04.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:04.458131+0000) 2022-01-31T21:53:04.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:04 smithi167 conmon[60316]: debug 2022-01-31T21:53:04.457+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:04.458127+0000) 2022-01-31T21:53:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:04 smithi167 conmon[49112]: debug 2022-01-31T21:53:04.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:04.793282+0000) 2022-01-31T21:53:05.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:04 smithi171 conmon[46715]: debug 2022-01-31T21:53:04.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:04.969716+0000) 2022-01-31T21:53:05.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[51620]: debug 2022-01-31T21:53:05.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.063536+0000) 2022-01-31T21:53:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[41853]: debug 2022-01-31T21:53:05.173+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.174068+0000) 2022-01-31T21:53:05.726 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:05 smithi167 conmon[60316]: debug 2022-01-31T21:53:05.457+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.458260+0000) 2022-01-31T21:53:05.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:05 smithi167 conmon[54076]: debug 2022-01-31T21:53:05.457+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.458311+0000) 2022-01-31T21:53:06.062 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[35325]: debug 2022-01-31T21:53:05.745+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165232 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:06.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[41853]: debug 2022-01-31T21:53:05.728+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.729478+0000) 2022-01-31T21:53:06.063 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[51620]: debug 2022-01-31T21:53:05.727+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.728615+0000) 2022-01-31T21:53:06.064 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[46715]: debug 2022-01-31T21:53:05.728+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.728837+0000) 2022-01-31T21:53:06.064 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:05 smithi171 conmon[46715]: debug 2022-01-31T21:53:05.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.969952+0000) 2022-01-31T21:53:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:05 smithi167 conmon[49112]: debug 2022-01-31T21:53:05.728+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.728841+0000) 2022-01-31T21:53:06.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:05 smithi167 conmon[49112]: debug 2022-01-31T21:53:05.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.793485+0000) 2022-01-31T21:53:06.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:05 smithi167 conmon[54076]: debug 2022-01-31T21:53:05.727+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.727570+0000) 2022-01-31T21:53:06.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:05 smithi167 conmon[60316]: debug 2022-01-31T21:53:05.726+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:05.727464+0000) 2022-01-31T21:53:06.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:06 smithi171 conmon[41853]: debug 2022-01-31T21:53:06.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:06.174265+0000) 2022-01-31T21:53:06.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:06 smithi171 conmon[51620]: debug 2022-01-31T21:53:06.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:06.063723+0000) 2022-01-31T21:53:06.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:06 smithi167 conmon[60316]: debug 2022-01-31T21:53:06.458+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:06.458459+0000) 2022-01-31T21:53:06.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:06 smithi167 conmon[54076]: debug 2022-01-31T21:53:06.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:06.458464+0000) 2022-01-31T21:53:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:06 smithi167 conmon[49112]: debug 2022-01-31T21:53:06.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:06.793668+0000) 2022-01-31T21:53:07.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:06 smithi171 conmon[46715]: debug 2022-01-31T21:53:06.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:06.970112+0000) 2022-01-31T21:53:07.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:07 smithi171 conmon[51620]: debug 2022-01-31T21:53:07.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:07.063903+0000) 2022-01-31T21:53:07.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:07 smithi171 conmon[41853]: debug 2022-01-31T21:53:07.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:07.174413+0000) 2022-01-31T21:53:07.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:07 smithi167 conmon[54076]: debug 2022-01-31T21:53:07.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:07.458613+0000) 2022-01-31T21:53:07.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:07 smithi167 conmon[60316]: debug 2022-01-31T21:53:07.458+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:07.458576+0000) 2022-01-31T21:53:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:07 smithi167 conmon[49112]: debug 2022-01-31T21:53:07.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:07.793850+0000) 2022-01-31T21:53:08.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:07 smithi171 conmon[46715]: debug 2022-01-31T21:53:07.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:07.970281+0000) 2022-01-31T21:53:08.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:08 smithi171 conmon[51620]: debug 2022-01-31T21:53:08.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:08.064071+0000) 2022-01-31T21:53:08.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:08 smithi171 conmon[41853]: debug 2022-01-31T21:53:08.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:08.174594+0000) 2022-01-31T21:53:08.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:08 smithi167 conmon[54076]: debug 2022-01-31T21:53:08.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:08.458809+0000) 2022-01-31T21:53:08.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:08 smithi167 conmon[60316]: debug 2022-01-31T21:53:08.458+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:08.458791+0000) 2022-01-31T21:53:09.062 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:08 smithi171 conmon[35325]: debug 2022-01-31T21:53:08.755+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:53:09.063 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:08 smithi171 conmon[46715]: debug 2022-01-31T21:53:08.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:08.970458+0000) 2022-01-31T21:53:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:08 smithi167 conmon[49112]: debug 2022-01-31T21:53:08.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:08.793990+0000) 2022-01-31T21:53:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:09 smithi171 conmon[41853]: debug 2022-01-31T21:53:09.173+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:09.174691+0000) 2022-01-31T21:53:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:09 smithi171 conmon[51620]: debug 2022-01-31T21:53:09.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:09.064256+0000) 2022-01-31T21:53:09.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:09 smithi167 conmon[54076]: debug 2022-01-31T21:53:09.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:09.459003+0000) 2022-01-31T21:53:09.665 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:09 smithi167 conmon[60316]: debug 2022-01-31T21:53:09.458+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:09.458995+0000) 2022-01-31T21:53:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:09 smithi167 conmon[49112]: debug 2022-01-31T21:53:09.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:09.794175+0000) 2022-01-31T21:53:10.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:09 smithi171 conmon[46715]: debug 2022-01-31T21:53:09.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:09.970648+0000) 2022-01-31T21:53:10.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[51620]: debug 2022-01-31T21:53:10.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.064448+0000) 2022-01-31T21:53:10.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[41853]: debug 2022-01-31T21:53:10.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.174857+0000) 2022-01-31T21:53:10.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:10 smithi167 conmon[54076]: debug 2022-01-31T21:53:10.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.459222+0000) 2022-01-31T21:53:10.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:10 smithi167 conmon[60316]: debug 2022-01-31T21:53:10.458+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.459203+0000) 2022-01-31T21:53:11.063 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[35325]: debug 2022-01-31T21:53:10.765+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165343 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:11.064 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[41853]: debug 2022-01-31T21:53:10.747+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.748404+0000) 2022-01-31T21:53:11.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[51620]: debug 2022-01-31T21:53:10.747+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.748101+0000) 2022-01-31T21:53:11.065 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[46715]: debug 2022-01-31T21:53:10.748+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.749242+0000) 2022-01-31T21:53:11.065 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:10 smithi171 conmon[46715]: debug 2022-01-31T21:53:10.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.970858+0000) 2022-01-31T21:53:11.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:10 smithi167 conmon[54076]: debug 2022-01-31T21:53:10.748+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.749028+0000) 2022-01-31T21:53:11.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:10 smithi167 conmon[60316]: debug 2022-01-31T21:53:10.747+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.748034+0000) 2022-01-31T21:53:11.171 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:10 smithi167 conmon[49112]: debug 2022-01-31T21:53:10.748+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.748550+0000) 2022-01-31T21:53:11.171 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:10 smithi167 conmon[49112]: debug 2022-01-31T21:53:10.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:10.794353+0000) 2022-01-31T21:53:11.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:11 smithi171 conmon[51620]: debug 2022-01-31T21:53:11.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:11.064643+0000) 2022-01-31T21:53:11.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:11 smithi171 conmon[41853]: debug 2022-01-31T21:53:11.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:11.175040+0000) 2022-01-31T21:53:11.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:11 smithi167 conmon[60316]: debug 2022-01-31T21:53:11.458+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:11.459440+0000) 2022-01-31T21:53:11.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:11 smithi167 conmon[54076]: debug 2022-01-31T21:53:11.458+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:11.459438+0000) 2022-01-31T21:53:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:11 smithi167 conmon[49112]: debug 2022-01-31T21:53:11.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:11.794488+0000) 2022-01-31T21:53:12.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:11 smithi171 conmon[46715]: debug 2022-01-31T21:53:11.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:11.971035+0000) 2022-01-31T21:53:12.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:12 smithi171 conmon[51620]: debug 2022-01-31T21:53:12.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:12.064739+0000) 2022-01-31T21:53:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:12 smithi171 conmon[41853]: debug 2022-01-31T21:53:12.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:12.175134+0000) 2022-01-31T21:53:12.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:12 smithi167 conmon[60316]: debug 2022-01-31T21:53:12.459+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:12.459622+0000) 2022-01-31T21:53:12.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:12 smithi167 conmon[54076]: debug 2022-01-31T21:53:12.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:12.459626+0000) 2022-01-31T21:53:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:12 smithi167 conmon[49112]: debug 2022-01-31T21:53:12.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:12.794690+0000) 2022-01-31T21:53:13.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:12 smithi171 conmon[46715]: debug 2022-01-31T21:53:12.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:12.971223+0000) 2022-01-31T21:53:13.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:13 smithi171 conmon[51620]: debug 2022-01-31T21:53:13.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:13.064950+0000) 2022-01-31T21:53:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:13 smithi171 conmon[41853]: debug 2022-01-31T21:53:13.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:13.175312+0000) 2022-01-31T21:53:13.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:13 smithi167 conmon[60316]: debug 2022-01-31T21:53:13.459+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:13.459815+0000) 2022-01-31T21:53:13.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:13 smithi167 conmon[54076]: debug 2022-01-31T21:53:13.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:13.459834+0000) 2022-01-31T21:53:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:13 smithi167 conmon[49112]: debug 2022-01-31T21:53:13.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:13.794873+0000) 2022-01-31T21:53:14.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:13 smithi171 conmon[46715]: debug 2022-01-31T21:53:13.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:13.971432+0000) 2022-01-31T21:53:14.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:14 smithi171 conmon[51620]: debug 2022-01-31T21:53:14.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:14.065136+0000) 2022-01-31T21:53:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:14 smithi171 conmon[41853]: debug 2022-01-31T21:53:14.175+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:14.175507+0000) 2022-01-31T21:53:14.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:14 smithi167 conmon[60316]: debug 2022-01-31T21:53:14.459+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:14.460003+0000) 2022-01-31T21:53:14.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:14 smithi167 conmon[54076]: debug 2022-01-31T21:53:14.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:14.460003+0000) 2022-01-31T21:53:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:14 smithi167 conmon[49112]: debug 2022-01-31T21:53:14.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:14.795083+0000) 2022-01-31T21:53:15.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:14 smithi171 conmon[46715]: debug 2022-01-31T21:53:14.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:14.971620+0000) 2022-01-31T21:53:15.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[51620]: debug 2022-01-31T21:53:15.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.065316+0000) 2022-01-31T21:53:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[41853]: debug 2022-01-31T21:53:15.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.175709+0000) 2022-01-31T21:53:15.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:15 smithi167 conmon[54076]: debug 2022-01-31T21:53:15.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.460219+0000) 2022-01-31T21:53:15.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:15 smithi167 conmon[60316]: debug 2022-01-31T21:53:15.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.460221+0000) 2022-01-31T21:53:16.064 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[35325]: debug 2022-01-31T21:53:15.784+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165453 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:16.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[41853]: debug 2022-01-31T21:53:15.766+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.768415+0000) 2022-01-31T21:53:16.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[51620]: debug 2022-01-31T21:53:15.767+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.769336+0000) 2022-01-31T21:53:16.066 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[46715]: debug 2022-01-31T21:53:15.766+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.767742+0000) 2022-01-31T21:53:16.066 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:15 smithi171 conmon[46715]: debug 2022-01-31T21:53:15.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.971804+0000) 2022-01-31T21:53:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:15 smithi167 conmon[49112]: debug 2022-01-31T21:53:15.767+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.767774+0000) 2022-01-31T21:53:16.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:15 smithi167 conmon[49112]: debug 2022-01-31T21:53:15.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.795265+0000) 2022-01-31T21:53:16.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:15 smithi167 conmon[54076]: debug 2022-01-31T21:53:15.768+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.768652+0000) 2022-01-31T21:53:16.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:15 smithi167 conmon[60316]: debug 2022-01-31T21:53:15.767+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:15.767633+0000) 2022-01-31T21:53:16.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:16 smithi171 conmon[51620]: debug 2022-01-31T21:53:16.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:16.065487+0000) 2022-01-31T21:53:16.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:16 smithi171 conmon[41853]: debug 2022-01-31T21:53:16.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:16.175952+0000) 2022-01-31T21:53:16.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:16 smithi167 conmon[54076]: debug 2022-01-31T21:53:16.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:16.460367+0000) 2022-01-31T21:53:16.517 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:16 smithi167 conmon[60316]: debug 2022-01-31T21:53:16.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:16.460369+0000) 2022-01-31T21:53:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:16 smithi167 conmon[49112]: debug 2022-01-31T21:53:16.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:16.795405+0000) 2022-01-31T21:53:17.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:16 smithi171 conmon[46715]: debug 2022-01-31T21:53:16.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:16.971955+0000) 2022-01-31T21:53:17.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:17 smithi171 conmon[51620]: debug 2022-01-31T21:53:17.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:17.065638+0000) 2022-01-31T21:53:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:17 smithi171 conmon[41853]: debug 2022-01-31T21:53:17.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:17.176054+0000) 2022-01-31T21:53:17.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:17 smithi167 conmon[60316]: debug 2022-01-31T21:53:17.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:17.460541+0000) 2022-01-31T21:53:17.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:17 smithi167 conmon[54076]: debug 2022-01-31T21:53:17.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:17.460545+0000) 2022-01-31T21:53:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:17 smithi167 conmon[49112]: debug 2022-01-31T21:53:17.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:17.795604+0000) 2022-01-31T21:53:18.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:17 smithi171 conmon[46715]: debug 2022-01-31T21:53:17.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:17.972109+0000) 2022-01-31T21:53:18.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:18 smithi171 conmon[51620]: debug 2022-01-31T21:53:18.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:18.065819+0000) 2022-01-31T21:53:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:18 smithi171 conmon[41853]: debug 2022-01-31T21:53:18.174+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:18.176223+0000) 2022-01-31T21:53:18.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:18 smithi167 conmon[54076]: debug 2022-01-31T21:53:18.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:18.460748+0000) 2022-01-31T21:53:18.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:18 smithi167 conmon[60316]: debug 2022-01-31T21:53:18.459+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:18.460744+0000) 2022-01-31T21:53:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:18 smithi167 conmon[49112]: debug 2022-01-31T21:53:18.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:18.795775+0000) 2022-01-31T21:53:19.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:18 smithi171 conmon[46715]: debug 2022-01-31T21:53:18.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:18.972315+0000) 2022-01-31T21:53:19.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:19 smithi171 conmon[51620]: debug 2022-01-31T21:53:19.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:19.065987+0000) 2022-01-31T21:53:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:19 smithi171 conmon[41853]: debug 2022-01-31T21:53:19.175+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:19.176406+0000) 2022-01-31T21:53:19.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:19 smithi167 conmon[54076]: debug 2022-01-31T21:53:19.459+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:19.460949+0000) 2022-01-31T21:53:19.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:19 smithi167 conmon[60316]: debug 2022-01-31T21:53:19.459+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:19.460949+0000) 2022-01-31T21:53:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:19 smithi167 conmon[49112]: debug 2022-01-31T21:53:19.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:19.795979+0000) 2022-01-31T21:53:20.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:19 smithi171 conmon[46715]: debug 2022-01-31T21:53:19.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:19.972500+0000) 2022-01-31T21:53:20.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[51620]: debug 2022-01-31T21:53:20.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.066141+0000) 2022-01-31T21:53:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[41853]: debug 2022-01-31T21:53:20.175+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.176594+0000) 2022-01-31T21:53:20.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:20 smithi167 conmon[54076]: debug 2022-01-31T21:53:20.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.461150+0000) 2022-01-31T21:53:20.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:20 smithi167 conmon[60316]: debug 2022-01-31T21:53:20.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.461150+0000) 2022-01-31T21:53:21.064 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[35325]: debug 2022-01-31T21:53:20.804+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165564 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:21.066 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[41853]: debug 2022-01-31T21:53:20.787+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.788973+0000) 2022-01-31T21:53:21.066 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[51620]: debug 2022-01-31T21:53:20.787+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.789058+0000) 2022-01-31T21:53:21.067 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[46715]: debug 2022-01-31T21:53:20.786+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.787921+0000) 2022-01-31T21:53:21.068 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:20 smithi171 conmon[46715]: debug 2022-01-31T21:53:20.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.972687+0000) 2022-01-31T21:53:21.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:20 smithi167 conmon[54076]: debug 2022-01-31T21:53:20.787+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.789061+0000) 2022-01-31T21:53:21.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:20 smithi167 conmon[60316]: debug 2022-01-31T21:53:20.787+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.788807+0000) 2022-01-31T21:53:21.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:20 smithi167 conmon[49112]: debug 2022-01-31T21:53:20.786+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.787958+0000) 2022-01-31T21:53:21.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:20 smithi167 conmon[49112]: debug 2022-01-31T21:53:20.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:20.796170+0000) 2022-01-31T21:53:21.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:21 smithi171 conmon[41853]: debug 2022-01-31T21:53:21.175+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:21.176799+0000) 2022-01-31T21:53:21.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:21 smithi171 conmon[51620]: debug 2022-01-31T21:53:21.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:21.066323+0000) 2022-01-31T21:53:21.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:21 smithi167 conmon[54076]: debug 2022-01-31T21:53:21.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:21.461334+0000) 2022-01-31T21:53:21.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:21 smithi167 conmon[60316]: debug 2022-01-31T21:53:21.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:21.461333+0000) 2022-01-31T21:53:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:21 smithi167 conmon[49112]: debug 2022-01-31T21:53:21.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:21.796338+0000) 2022-01-31T21:53:22.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:21 smithi171 conmon[46715]: debug 2022-01-31T21:53:21.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:21.972836+0000) 2022-01-31T21:53:22.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:22 smithi171 conmon[51620]: debug 2022-01-31T21:53:22.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:22.066471+0000) 2022-01-31T21:53:22.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:22 smithi171 conmon[41853]: debug 2022-01-31T21:53:22.175+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:22.176913+0000) 2022-01-31T21:53:22.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:22 smithi167 conmon[54076]: debug 2022-01-31T21:53:22.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:22.461487+0000) 2022-01-31T21:53:22.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:22 smithi167 conmon[60316]: debug 2022-01-31T21:53:22.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:22.461491+0000) 2022-01-31T21:53:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:22 smithi167 conmon[49112]: debug 2022-01-31T21:53:22.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:22.796564+0000) 2022-01-31T21:53:23.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:22 smithi171 conmon[46715]: debug 2022-01-31T21:53:22.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:22.972996+0000) 2022-01-31T21:53:23.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:23 smithi171 conmon[51620]: debug 2022-01-31T21:53:23.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:23.066650+0000) 2022-01-31T21:53:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:23 smithi171 conmon[41853]: debug 2022-01-31T21:53:23.176+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:23.177115+0000) 2022-01-31T21:53:23.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:23 smithi167 conmon[60316]: debug 2022-01-31T21:53:23.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:23.461723+0000) 2022-01-31T21:53:23.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:23 smithi167 conmon[54076]: debug 2022-01-31T21:53:23.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:23.461705+0000) 2022-01-31T21:53:24.065 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:23 smithi171 conmon[46715]: debug 2022-01-31T21:53:23.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:23.973174+0000) 2022-01-31T21:53:24.066 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:23 smithi171 conmon[35325]: debug 2022-01-31T21:53:23.755+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:53:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:23 smithi167 conmon[49112]: debug 2022-01-31T21:53:23.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:23.796698+0000) 2022-01-31T21:53:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:24 smithi171 conmon[51620]: debug 2022-01-31T21:53:24.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:24.066812+0000) 2022-01-31T21:53:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:24 smithi171 conmon[41853]: debug 2022-01-31T21:53:24.176+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:24.177333+0000) 2022-01-31T21:53:24.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:24 smithi167 conmon[54076]: debug 2022-01-31T21:53:24.460+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:24.461887+0000) 2022-01-31T21:53:24.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:24 smithi167 conmon[60316]: debug 2022-01-31T21:53:24.460+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:24.461901+0000) 2022-01-31T21:53:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:24 smithi167 conmon[49112]: debug 2022-01-31T21:53:24.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:24.796838+0000) 2022-01-31T21:53:25.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:24 smithi171 conmon[46715]: debug 2022-01-31T21:53:24.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:24.973380+0000) 2022-01-31T21:53:25.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[51620]: debug 2022-01-31T21:53:25.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.066969+0000) 2022-01-31T21:53:25.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[41853]: debug 2022-01-31T21:53:25.176+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.177541+0000) 2022-01-31T21:53:25.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:25 smithi167 conmon[54076]: debug 2022-01-31T21:53:25.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.462090+0000) 2022-01-31T21:53:25.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:25 smithi167 conmon[60316]: debug 2022-01-31T21:53:25.461+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.462100+0000) 2022-01-31T21:53:26.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[41853]: debug 2022-01-31T21:53:25.808+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.809104+0000) 2022-01-31T21:53:26.066 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[51620]: debug 2022-01-31T21:53:25.807+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.808736+0000) 2022-01-31T21:53:26.067 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[35325]: debug 2022-01-31T21:53:25.832+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165674 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:26.067 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[46715]: debug 2022-01-31T21:53:25.806+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.807729+0000) 2022-01-31T21:53:26.068 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:25 smithi171 conmon[46715]: debug 2022-01-31T21:53:25.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.973510+0000) 2022-01-31T21:53:26.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:25 smithi167 conmon[54076]: debug 2022-01-31T21:53:25.807+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.808513+0000) 2022-01-31T21:53:26.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:25 smithi167 conmon[60316]: debug 2022-01-31T21:53:25.813+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.814817+0000) 2022-01-31T21:53:26.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:25 smithi167 conmon[49112]: debug 2022-01-31T21:53:25.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.797054+0000) 2022-01-31T21:53:26.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:25 smithi167 conmon[49112]: debug 2022-01-31T21:53:25.805+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:25.807262+0000) 2022-01-31T21:53:26.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:26 smithi171 conmon[51620]: debug 2022-01-31T21:53:26.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:26.067183+0000) 2022-01-31T21:53:26.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:26 smithi171 conmon[41853]: debug 2022-01-31T21:53:26.176+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:26.177713+0000) 2022-01-31T21:53:26.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:26 smithi167 conmon[60316]: debug 2022-01-31T21:53:26.461+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:26.462300+0000) 2022-01-31T21:53:26.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:26 smithi167 conmon[54076]: debug 2022-01-31T21:53:26.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:26.462301+0000) 2022-01-31T21:53:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:26 smithi167 conmon[49112]: debug 2022-01-31T21:53:26.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:26.797267+0000) 2022-01-31T21:53:27.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:26 smithi171 conmon[46715]: debug 2022-01-31T21:53:26.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:26.973668+0000) 2022-01-31T21:53:27.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:27 smithi171 conmon[51620]: debug 2022-01-31T21:53:27.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:27.067337+0000) 2022-01-31T21:53:27.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:27 smithi171 conmon[41853]: debug 2022-01-31T21:53:27.177+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:27.177921+0000) 2022-01-31T21:53:27.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:27 smithi167 conmon[54076]: debug 2022-01-31T21:53:27.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:27.462455+0000) 2022-01-31T21:53:27.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:27 smithi167 conmon[60316]: debug 2022-01-31T21:53:27.461+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:27.462455+0000) 2022-01-31T21:53:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:27 smithi167 conmon[49112]: debug 2022-01-31T21:53:27.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:27.797467+0000) 2022-01-31T21:53:28.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:27 smithi171 conmon[46715]: debug 2022-01-31T21:53:27.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:27.973800+0000) 2022-01-31T21:53:28.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:28 smithi171 conmon[51620]: debug 2022-01-31T21:53:28.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:28.067546+0000) 2022-01-31T21:53:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:28 smithi171 conmon[41853]: debug 2022-01-31T21:53:28.177+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:28.178107+0000) 2022-01-31T21:53:28.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:28 smithi167 conmon[54076]: debug 2022-01-31T21:53:28.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:28.462635+0000) 2022-01-31T21:53:28.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:28 smithi167 conmon[60316]: debug 2022-01-31T21:53:28.461+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:28.462635+0000) 2022-01-31T21:53:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:28 smithi167 conmon[49112]: debug 2022-01-31T21:53:28.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:28.797678+0000) 2022-01-31T21:53:29.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:28 smithi171 conmon[46715]: debug 2022-01-31T21:53:28.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:28.973954+0000) 2022-01-31T21:53:29.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:29 smithi171 conmon[51620]: debug 2022-01-31T21:53:29.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:29.067745+0000) 2022-01-31T21:53:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:29 smithi171 conmon[41853]: debug 2022-01-31T21:53:29.177+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:29.178289+0000) 2022-01-31T21:53:29.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:29 smithi167 conmon[54076]: debug 2022-01-31T21:53:29.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:29.462834+0000) 2022-01-31T21:53:29.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:29 smithi167 conmon[60316]: debug 2022-01-31T21:53:29.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:29.462832+0000) 2022-01-31T21:53:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:29 smithi167 conmon[49112]: debug 2022-01-31T21:53:29.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:29.797837+0000) 2022-01-31T21:53:30.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:29 smithi171 conmon[46715]: debug 2022-01-31T21:53:29.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:29.974119+0000) 2022-01-31T21:53:30.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[51620]: debug 2022-01-31T21:53:30.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.067960+0000) 2022-01-31T21:53:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[41853]: debug 2022-01-31T21:53:30.177+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.178482+0000) 2022-01-31T21:53:30.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:30 smithi167 conmon[54076]: debug 2022-01-31T21:53:30.461+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.463093+0000) 2022-01-31T21:53:30.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:30 smithi167 conmon[60316]: debug 2022-01-31T21:53:30.461+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.463110+0000) 2022-01-31T21:53:31.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[35325]: debug 2022-01-31T21:53:30.851+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165785 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:31.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[41853]: debug 2022-01-31T21:53:30.835+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.836282+0000) 2022-01-31T21:53:31.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[46715]: debug 2022-01-31T21:53:30.834+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.835656+0000) 2022-01-31T21:53:31.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[46715]: debug 2022-01-31T21:53:30.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.974296+0000) 2022-01-31T21:53:31.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:30 smithi171 conmon[51620]: debug 2022-01-31T21:53:30.835+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.835930+0000) 2022-01-31T21:53:31.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:31 smithi171 conmon[51620]: debug 2022-01-31T21:53:31.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:31.068137+0000) 2022-01-31T21:53:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:30 smithi167 conmon[49112]: debug 2022-01-31T21:53:30.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.798031+0000) 2022-01-31T21:53:31.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:30 smithi167 conmon[49112]: debug 2022-01-31T21:53:30.834+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.835647+0000) 2022-01-31T21:53:31.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:30 smithi167 conmon[54076]: debug 2022-01-31T21:53:30.832+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.834473+0000) 2022-01-31T21:53:31.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:30 smithi167 conmon[60316]: debug 2022-01-31T21:53:30.834+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:30.835963+0000) 2022-01-31T21:53:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:31 smithi171 conmon[41853]: debug 2022-01-31T21:53:31.177+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:31.178628+0000) 2022-01-31T21:53:31.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:31 smithi167 conmon[54076]: debug 2022-01-31T21:53:31.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:31.463257+0000) 2022-01-31T21:53:31.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:31 smithi167 conmon[60316]: debug 2022-01-31T21:53:31.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:31.463303+0000) 2022-01-31T21:53:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:31 smithi167 conmon[49112]: debug 2022-01-31T21:53:31.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:31.798179+0000) 2022-01-31T21:53:32.409 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:31 smithi171 conmon[46715]: debug 2022-01-31T21:53:31.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:31.974454+0000) 2022-01-31T21:53:32.409 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:32 smithi171 conmon[51620]: debug 2022-01-31T21:53:32.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:32.068263+0000) 2022-01-31T21:53:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:32 smithi171 conmon[41853]: debug 2022-01-31T21:53:32.178+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:32.178801+0000) 2022-01-31T21:53:32.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:32 smithi167 conmon[54076]: debug 2022-01-31T21:53:32.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:32.463450+0000) 2022-01-31T21:53:32.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:32 smithi167 conmon[60316]: debug 2022-01-31T21:53:32.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:32.463459+0000) 2022-01-31T21:53:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:32 smithi167 conmon[49112]: debug 2022-01-31T21:53:32.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:32.798345+0000) 2022-01-31T21:53:33.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:32 smithi171 conmon[46715]: debug 2022-01-31T21:53:32.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:32.974668+0000) 2022-01-31T21:53:33.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:33 smithi171 conmon[51620]: debug 2022-01-31T21:53:33.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:33.068447+0000) 2022-01-31T21:53:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:33 smithi171 conmon[41853]: debug 2022-01-31T21:53:33.178+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:33.178940+0000) 2022-01-31T21:53:33.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:33 smithi167 conmon[54076]: debug 2022-01-31T21:53:33.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:33.463682+0000) 2022-01-31T21:53:33.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:33 smithi167 conmon[60316]: debug 2022-01-31T21:53:33.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:33.463677+0000) 2022-01-31T21:53:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:33 smithi167 conmon[49112]: debug 2022-01-31T21:53:33.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:33.798578+0000) 2022-01-31T21:53:34.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:33 smithi171 conmon[46715]: debug 2022-01-31T21:53:33.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:33.974871+0000) 2022-01-31T21:53:34.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:34 smithi171 conmon[51620]: debug 2022-01-31T21:53:34.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:34.068592+0000) 2022-01-31T21:53:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:34 smithi171 conmon[41853]: debug 2022-01-31T21:53:34.178+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:34.179113+0000) 2022-01-31T21:53:34.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:34 smithi167 conmon[54076]: debug 2022-01-31T21:53:34.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:34.463809+0000) 2022-01-31T21:53:34.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:34 smithi167 conmon[60316]: debug 2022-01-31T21:53:34.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:34.463812+0000) 2022-01-31T21:53:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:34 smithi167 conmon[49112]: debug 2022-01-31T21:53:34.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:34.798793+0000) 2022-01-31T21:53:35.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:34 smithi171 conmon[46715]: debug 2022-01-31T21:53:34.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:34.975062+0000) 2022-01-31T21:53:35.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[51620]: debug 2022-01-31T21:53:35.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.068722+0000) 2022-01-31T21:53:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[41853]: debug 2022-01-31T21:53:35.178+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.179302+0000) 2022-01-31T21:53:35.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:35 smithi167 conmon[54076]: debug 2022-01-31T21:53:35.462+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.463975+0000) 2022-01-31T21:53:35.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:35 smithi167 conmon[60316]: debug 2022-01-31T21:53:35.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.463973+0000) 2022-01-31T21:53:36.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[35325]: debug 2022-01-31T21:53:35.871+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 165895 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:36.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[41853]: debug 2022-01-31T21:53:35.854+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.854891+0000) 2022-01-31T21:53:36.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[46715]: debug 2022-01-31T21:53:35.853+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.854423+0000) 2022-01-31T21:53:36.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[46715]: debug 2022-01-31T21:53:35.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.975220+0000) 2022-01-31T21:53:36.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:35 smithi171 conmon[51620]: debug 2022-01-31T21:53:35.854+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.854781+0000) 2022-01-31T21:53:36.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:36 smithi171 conmon[51620]: debug 2022-01-31T21:53:36.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:36.068930+0000) 2022-01-31T21:53:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:35 smithi167 conmon[49112]: debug 2022-01-31T21:53:35.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.799016+0000) 2022-01-31T21:53:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:35 smithi167 conmon[49112]: debug 2022-01-31T21:53:35.853+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.855016+0000) 2022-01-31T21:53:36.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:35 smithi167 conmon[54076]: debug 2022-01-31T21:53:35.852+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.853726+0000) 2022-01-31T21:53:36.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:35 smithi167 conmon[60316]: debug 2022-01-31T21:53:35.853+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:35.855125+0000) 2022-01-31T21:53:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:36 smithi171 conmon[41853]: debug 2022-01-31T21:53:36.178+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:36.179518+0000) 2022-01-31T21:53:36.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:36 smithi167 conmon[54076]: debug 2022-01-31T21:53:36.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:36.464163+0000) 2022-01-31T21:53:36.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:36 smithi167 conmon[60316]: debug 2022-01-31T21:53:36.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:36.464166+0000) 2022-01-31T21:53:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:36 smithi167 conmon[49112]: debug 2022-01-31T21:53:36.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:36.799229+0000) 2022-01-31T21:53:37.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:36 smithi171 conmon[46715]: debug 2022-01-31T21:53:36.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:36.975373+0000) 2022-01-31T21:53:37.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:37 smithi171 conmon[51620]: debug 2022-01-31T21:53:37.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:37.069082+0000) 2022-01-31T21:53:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:37 smithi171 conmon[41853]: debug 2022-01-31T21:53:37.179+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:37.179658+0000) 2022-01-31T21:53:37.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:37 smithi167 conmon[54076]: debug 2022-01-31T21:53:37.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:37.464332+0000) 2022-01-31T21:53:37.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:37 smithi167 conmon[60316]: debug 2022-01-31T21:53:37.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:37.464332+0000) 2022-01-31T21:53:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:37 smithi167 conmon[49112]: debug 2022-01-31T21:53:37.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:37.799384+0000) 2022-01-31T21:53:38.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:37 smithi171 conmon[46715]: debug 2022-01-31T21:53:37.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:37.975524+0000) 2022-01-31T21:53:38.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:38 smithi171 conmon[51620]: debug 2022-01-31T21:53:38.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:38.069262+0000) 2022-01-31T21:53:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:38 smithi171 conmon[41853]: debug 2022-01-31T21:53:38.179+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:38.179845+0000) 2022-01-31T21:53:38.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:38 smithi167 conmon[54076]: debug 2022-01-31T21:53:38.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:38.464551+0000) 2022-01-31T21:53:38.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:38 smithi167 conmon[60316]: debug 2022-01-31T21:53:38.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:38.464551+0000) 2022-01-31T21:53:39.068 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:38 smithi171 conmon[46715]: debug 2022-01-31T21:53:38.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:38.975673+0000) 2022-01-31T21:53:39.068 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:38 smithi171 conmon[35325]: debug 2022-01-31T21:53:38.756+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:53:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:38 smithi167 conmon[49112]: debug 2022-01-31T21:53:38.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:38.799536+0000) 2022-01-31T21:53:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:39 smithi171 conmon[41853]: debug 2022-01-31T21:53:39.179+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:39.180010+0000) 2022-01-31T21:53:39.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:39 smithi171 conmon[51620]: debug 2022-01-31T21:53:39.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:39.069464+0000) 2022-01-31T21:53:39.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:39 smithi167 conmon[54076]: debug 2022-01-31T21:53:39.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:39.464681+0000) 2022-01-31T21:53:39.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:39 smithi167 conmon[60316]: debug 2022-01-31T21:53:39.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:39.464698+0000) 2022-01-31T21:53:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:39 smithi167 conmon[49112]: debug 2022-01-31T21:53:39.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:39.799709+0000) 2022-01-31T21:53:40.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:39 smithi171 conmon[46715]: debug 2022-01-31T21:53:39.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:39.975832+0000) 2022-01-31T21:53:40.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[51620]: debug 2022-01-31T21:53:40.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.069658+0000) 2022-01-31T21:53:40.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[41853]: debug 2022-01-31T21:53:40.179+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.180132+0000) 2022-01-31T21:53:40.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:40 smithi167 conmon[54076]: debug 2022-01-31T21:53:40.463+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.464866+0000) 2022-01-31T21:53:40.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:40 smithi167 conmon[60316]: debug 2022-01-31T21:53:40.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.464871+0000) 2022-01-31T21:53:41.135 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[35325]: debug 2022-01-31T21:53:40.891+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166006 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:41.136 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[41853]: debug 2022-01-31T21:53:40.875+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.876432+0000) 2022-01-31T21:53:41.137 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[46715]: debug 2022-01-31T21:53:40.873+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.874301+0000) 2022-01-31T21:53:41.137 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[46715]: debug 2022-01-31T21:53:40.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.976026+0000) 2022-01-31T21:53:41.138 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:40 smithi171 conmon[51620]: debug 2022-01-31T21:53:40.875+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.876203+0000) 2022-01-31T21:53:41.138 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:41 smithi171 conmon[51620]: debug 2022-01-31T21:53:41.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:41.069823+0000) 2022-01-31T21:53:41.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:40 smithi167 conmon[54076]: debug 2022-01-31T21:53:40.872+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.874473+0000) 2022-01-31T21:53:41.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:40 smithi167 conmon[60316]: debug 2022-01-31T21:53:40.874+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.875496+0000) 2022-01-31T21:53:41.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:40 smithi167 conmon[49112]: debug 2022-01-31T21:53:40.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.799916+0000) 2022-01-31T21:53:41.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:40 smithi167 conmon[49112]: debug 2022-01-31T21:53:40.873+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:40.875383+0000) 2022-01-31T21:53:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:41 smithi171 conmon[41853]: debug 2022-01-31T21:53:41.179+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:41.180328+0000) 2022-01-31T21:53:41.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:41 smithi167 conmon[54076]: debug 2022-01-31T21:53:41.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:41.465104+0000) 2022-01-31T21:53:41.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:41 smithi167 conmon[60316]: debug 2022-01-31T21:53:41.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:41.465104+0000) 2022-01-31T21:53:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:41 smithi167 conmon[49112]: debug 2022-01-31T21:53:41.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:41.800105+0000) 2022-01-31T21:53:42.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:41 smithi171 conmon[46715]: debug 2022-01-31T21:53:41.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:41.976146+0000) 2022-01-31T21:53:42.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:42 smithi171 conmon[51620]: debug 2022-01-31T21:53:42.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:42.069959+0000) 2022-01-31T21:53:42.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:42 smithi171 conmon[41853]: debug 2022-01-31T21:53:42.180+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:42.180492+0000) 2022-01-31T21:53:42.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:42 smithi167 conmon[54076]: debug 2022-01-31T21:53:42.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:42.465271+0000) 2022-01-31T21:53:42.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:42 smithi167 conmon[60316]: debug 2022-01-31T21:53:42.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:42.465273+0000) 2022-01-31T21:53:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:42 smithi167 conmon[49112]: debug 2022-01-31T21:53:42.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:42.800252+0000) 2022-01-31T21:53:43.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:42 smithi171 conmon[46715]: debug 2022-01-31T21:53:42.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:42.976325+0000) 2022-01-31T21:53:43.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:43 smithi171 conmon[51620]: debug 2022-01-31T21:53:43.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:43.070089+0000) 2022-01-31T21:53:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:43 smithi171 conmon[41853]: debug 2022-01-31T21:53:43.180+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:43.180748+0000) 2022-01-31T21:53:43.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:43 smithi167 conmon[54076]: debug 2022-01-31T21:53:43.464+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:43.465465+0000) 2022-01-31T21:53:43.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:43 smithi167 conmon[60316]: debug 2022-01-31T21:53:43.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:43.465490+0000) 2022-01-31T21:53:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:43 smithi167 conmon[49112]: debug 2022-01-31T21:53:43.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:43.800473+0000) 2022-01-31T21:53:44.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:43 smithi171 conmon[46715]: debug 2022-01-31T21:53:43.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:43.976525+0000) 2022-01-31T21:53:44.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:44 smithi171 conmon[51620]: debug 2022-01-31T21:53:44.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:44.070300+0000) 2022-01-31T21:53:44.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:44 smithi171 conmon[41853]: debug 2022-01-31T21:53:44.180+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:44.180962+0000) 2022-01-31T21:53:44.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:44 smithi167 conmon[54076]: debug 2022-01-31T21:53:44.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:44.465672+0000) 2022-01-31T21:53:44.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:44 smithi167 conmon[60316]: debug 2022-01-31T21:53:44.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:44.465672+0000) 2022-01-31T21:53:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:44 smithi167 conmon[49112]: debug 2022-01-31T21:53:44.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:44.800648+0000) 2022-01-31T21:53:45.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:44 smithi171 conmon[46715]: debug 2022-01-31T21:53:44.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:44.976713+0000) 2022-01-31T21:53:45.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[51620]: debug 2022-01-31T21:53:45.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.070474+0000) 2022-01-31T21:53:45.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[41853]: debug 2022-01-31T21:53:45.180+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.181151+0000) 2022-01-31T21:53:45.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:45 smithi167 conmon[54076]: debug 2022-01-31T21:53:45.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.465861+0000) 2022-01-31T21:53:45.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:45 smithi167 conmon[60316]: debug 2022-01-31T21:53:45.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.465869+0000) 2022-01-31T21:53:46.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:45 smithi167 conmon[54076]: debug 2022-01-31T21:53:45.893+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.894275+0000) 2022-01-31T21:53:46.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:45 smithi167 conmon[60316]: debug 2022-01-31T21:53:45.893+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.894422+0000) 2022-01-31T21:53:46.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:45 smithi167 conmon[49112]: debug 2022-01-31T21:53:45.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.800809+0000) 2022-01-31T21:53:46.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:45 smithi167 conmon[49112]: debug 2022-01-31T21:53:45.893+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.894123+0000) 2022-01-31T21:53:46.179 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[41853]: debug 2022-01-31T21:53:45.895+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.895748+0000) 2022-01-31T21:53:46.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[46715]: debug 2022-01-31T21:53:45.893+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.894550+0000) 2022-01-31T21:53:46.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[46715]: debug 2022-01-31T21:53:45.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.976899+0000) 2022-01-31T21:53:46.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[35325]: debug 2022-01-31T21:53:45.911+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166116 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:46.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:45 smithi171 conmon[51620]: debug 2022-01-31T21:53:45.894+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:45.895385+0000) 2022-01-31T21:53:46.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:46 smithi171 conmon[51620]: debug 2022-01-31T21:53:46.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:46.070625+0000) 2022-01-31T21:53:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:46 smithi171 conmon[41853]: debug 2022-01-31T21:53:46.181+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:46.181333+0000) 2022-01-31T21:53:46.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:46 smithi167 conmon[54076]: debug 2022-01-31T21:53:46.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:46.466070+0000) 2022-01-31T21:53:46.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:46 smithi167 conmon[60316]: debug 2022-01-31T21:53:46.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:46.466070+0000) 2022-01-31T21:53:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:46 smithi167 conmon[49112]: debug 2022-01-31T21:53:46.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:46.800993+0000) 2022-01-31T21:53:47.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:46 smithi171 conmon[46715]: debug 2022-01-31T21:53:46.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:46.976988+0000) 2022-01-31T21:53:47.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:47 smithi171 conmon[51620]: debug 2022-01-31T21:53:47.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:47.070787+0000) 2022-01-31T21:53:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:47 smithi171 conmon[41853]: debug 2022-01-31T21:53:47.180+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:47.181522+0000) 2022-01-31T21:53:47.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:47 smithi167 conmon[54076]: debug 2022-01-31T21:53:47.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:47.466246+0000) 2022-01-31T21:53:47.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:47 smithi167 conmon[60316]: debug 2022-01-31T21:53:47.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:47.466208+0000) 2022-01-31T21:53:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:47 smithi167 conmon[49112]: debug 2022-01-31T21:53:47.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:47.801160+0000) 2022-01-31T21:53:48.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:47 smithi171 conmon[46715]: debug 2022-01-31T21:53:47.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:47.977168+0000) 2022-01-31T21:53:48.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:48 smithi171 conmon[51620]: debug 2022-01-31T21:53:48.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:48.070946+0000) 2022-01-31T21:53:48.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:48 smithi171 conmon[41853]: debug 2022-01-31T21:53:48.181+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:48.181733+0000) 2022-01-31T21:53:48.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:48 smithi167 conmon[54076]: debug 2022-01-31T21:53:48.465+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:48.466430+0000) 2022-01-31T21:53:48.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:48 smithi167 conmon[60316]: debug 2022-01-31T21:53:48.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:48.466427+0000) 2022-01-31T21:53:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:48 smithi167 conmon[49112]: debug 2022-01-31T21:53:48.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:48.801382+0000) 2022-01-31T21:53:49.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:48 smithi171 conmon[46715]: debug 2022-01-31T21:53:48.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:48.977354+0000) 2022-01-31T21:53:49.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:49 smithi171 conmon[51620]: debug 2022-01-31T21:53:49.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:49.071176+0000) 2022-01-31T21:53:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:49 smithi171 conmon[41853]: debug 2022-01-31T21:53:49.181+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:49.181944+0000) 2022-01-31T21:53:49.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:49 smithi167 conmon[54076]: debug 2022-01-31T21:53:49.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:49.466594+0000) 2022-01-31T21:53:49.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:49 smithi167 conmon[60316]: debug 2022-01-31T21:53:49.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:49.466526+0000) 2022-01-31T21:53:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:49 smithi167 conmon[49112]: debug 2022-01-31T21:53:49.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:49.801556+0000) 2022-01-31T21:53:50.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:49 smithi171 conmon[46715]: debug 2022-01-31T21:53:49.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:49.977563+0000) 2022-01-31T21:53:50.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[51620]: debug 2022-01-31T21:53:50.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.071353+0000) 2022-01-31T21:53:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[41853]: debug 2022-01-31T21:53:50.181+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.182128+0000) 2022-01-31T21:53:50.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[54076]: debug 2022-01-31T21:53:50.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.466734+0000) 2022-01-31T21:53:50.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[60316]: debug 2022-01-31T21:53:50.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.466704+0000) 2022-01-31T21:53:51.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[54076]: debug 2022-01-31T21:53:50.914+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.914663+0000) 2022-01-31T21:53:51.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[60316]: debug 2022-01-31T21:53:50.914+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.914951+0000) 2022-01-31T21:53:51.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[49112]: debug 2022-01-31T21:53:50.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.801736+0000) 2022-01-31T21:53:51.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[49112]: debug 2022-01-31T21:53:51.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:50 smithi167 conmon[49112]: 2022-01-31T21:53:50.913+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.913713+0000) 2022-01-31T21:53:51.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[35325]: debug 2022-01-31T21:53:50.931+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166227 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:51.181 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[41853]: debug 2022-01-31T21:53:50.915+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.915908+0000) 2022-01-31T21:53:51.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[46715]: debug 2022-01-31T21:53:50.913+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.913962+0000) 2022-01-31T21:53:51.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[46715]: debug 2022-01-31T21:53:50.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.977766+0000) 2022-01-31T21:53:51.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:50 smithi171 conmon[51620]: debug 2022-01-31T21:53:50.914+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:50.915589+0000) 2022-01-31T21:53:51.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:51 smithi171 conmon[51620]: debug 2022-01-31T21:53:51.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:51.071529+0000) 2022-01-31T21:53:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:51 smithi171 conmon[41853]: debug 2022-01-31T21:53:51.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:51.182319+0000) 2022-01-31T21:53:51.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:51 smithi167 conmon[54076]: debug 2022-01-31T21:53:51.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:51.466931+0000) 2022-01-31T21:53:51.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:51 smithi167 conmon[60316]: debug 2022-01-31T21:53:51.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:51.466924+0000) 2022-01-31T21:53:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:51 smithi167 conmon[49112]: debug 2022-01-31T21:53:51.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:51.801960+0000) 2022-01-31T21:53:52.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:51 smithi171 conmon[46715]: debug 2022-01-31T21:53:51.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:51.977923+0000) 2022-01-31T21:53:52.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:52 smithi171 conmon[51620]: debug 2022-01-31T21:53:52.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:52.071703+0000) 2022-01-31T21:53:52.448 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:52 smithi171 conmon[41853]: debug 2022-01-31T21:53:52.181+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:52.182465+0000) 2022-01-31T21:53:52.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:52 smithi167 conmon[54076]: debug 2022-01-31T21:53:52.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:52.467103+0000) 2022-01-31T21:53:52.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:52 smithi167 conmon[60316]: debug 2022-01-31T21:53:52.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:52.467114+0000) 2022-01-31T21:53:53.070 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:52 smithi171 conmon[46715]: debug 2022-01-31T21:53:52.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:52.978132+0000) 2022-01-31T21:53:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:52 smithi167 conmon[49112]: debug 2022-01-31T21:53:52.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:52.802133+0000) 2022-01-31T21:53:53.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:53 smithi171 conmon[41853]: debug 2022-01-31T21:53:53.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:53.182643+0000) 2022-01-31T21:53:53.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:53 smithi171 conmon[51620]: debug 2022-01-31T21:53:53.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:53.071923+0000) 2022-01-31T21:53:53.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:53 smithi167 conmon[54076]: debug 2022-01-31T21:53:53.466+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:53.467331+0000) 2022-01-31T21:53:53.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:53 smithi167 conmon[60316]: debug 2022-01-31T21:53:53.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:53.467323+0000) 2022-01-31T21:53:54.070 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:53 smithi171 conmon[35325]: debug 2022-01-31T21:53:53.757+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:53:54.071 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:53 smithi171 conmon[46715]: debug 2022-01-31T21:53:53.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:53.978304+0000) 2022-01-31T21:53:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:53 smithi167 conmon[49112]: debug 2022-01-31T21:53:53.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:53.802377+0000) 2022-01-31T21:53:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:54 smithi171 conmon[41853]: debug 2022-01-31T21:53:54.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:54.182852+0000) 2022-01-31T21:53:54.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:54 smithi171 conmon[51620]: debug 2022-01-31T21:53:54.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:54.072117+0000) 2022-01-31T21:53:54.723 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:54 smithi167 conmon[54076]: debug 2022-01-31T21:53:54.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:54.467533+0000) 2022-01-31T21:53:54.724 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:54 smithi167 conmon[60316]: debug 2022-01-31T21:53:54.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:54.467594+0000) 2022-01-31T21:53:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:54 smithi167 conmon[49112]: debug 2022-01-31T21:53:54.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:54.802600+0000) 2022-01-31T21:53:55.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:54 smithi171 conmon[46715]: debug 2022-01-31T21:53:54.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:54.978486+0000) 2022-01-31T21:53:55.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[51620]: debug 2022-01-31T21:53:55.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.072281+0000) 2022-01-31T21:53:55.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:55 smithi167 conmon[54076]: debug 2022-01-31T21:53:55.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.467711+0000) 2022-01-31T21:53:55.524 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:55 smithi167 conmon[60316]: debug 2022-01-31T21:53:55.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.467755+0000) 2022-01-31T21:53:55.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[41853]: debug 2022-01-31T21:53:55.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.183040+0000) 2022-01-31T21:53:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:55 smithi167 conmon[49112]: debug 2022-01-31T21:53:55.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.802736+0000) 2022-01-31T21:53:56.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:55 smithi167 conmon[49112]: debug 2022-01-31T21:53:55.934+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.935254+0000) 2022-01-31T21:53:56.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:55 smithi167 conmon[54076]: debug 2022-01-31T21:53:55.933+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.933659+0000) 2022-01-31T21:53:56.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:55 smithi167 conmon[60316]: debug 2022-01-31T21:53:55.934+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.934520+0000) 2022-01-31T21:53:56.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[35325]: debug 2022-01-31T21:53:55.951+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166337 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:53:56.182 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[41853]: debug 2022-01-31T21:53:55.935+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.935786+0000) 2022-01-31T21:53:56.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[46715]: debug 2022-01-31T21:53:55.934+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.934817+0000) 2022-01-31T21:53:56.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[46715]: debug 2022-01-31T21:53:55.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.978640+0000) 2022-01-31T21:53:56.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:55 smithi171 conmon[51620]: debug 2022-01-31T21:53:55.934+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:55.935116+0000) 2022-01-31T21:53:56.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:56 smithi171 conmon[51620]: debug 2022-01-31T21:53:56.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:56.072452+0000) 2022-01-31T21:53:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:56 smithi171 conmon[41853]: debug 2022-01-31T21:53:56.183+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:56.183246+0000) 2022-01-31T21:53:56.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:56 smithi167 conmon[60316]: debug 2022-01-31T21:53:56.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:56.467949+0000) 2022-01-31T21:53:56.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:56 smithi167 conmon[54076]: debug 2022-01-31T21:53:56.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:56.467905+0000) 2022-01-31T21:53:56.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:56 smithi167 conmon[49112]: debug 2022-01-31T21:53:56.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:56.802947+0000) 2022-01-31T21:53:57.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:56 smithi171 conmon[46715]: debug 2022-01-31T21:53:56.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:56.978814+0000) 2022-01-31T21:53:57.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:57 smithi171 conmon[51620]: debug 2022-01-31T21:53:57.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:57.072627+0000) 2022-01-31T21:53:57.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:57 smithi171 conmon[41853]: debug 2022-01-31T21:53:57.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:57.183417+0000) 2022-01-31T21:53:57.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:57 smithi167 conmon[54076]: debug 2022-01-31T21:53:57.467+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:57.468088+0000) 2022-01-31T21:53:57.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:57 smithi167 conmon[60316]: debug 2022-01-31T21:53:57.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:57.468088+0000) 2022-01-31T21:53:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:57 smithi167 conmon[49112]: debug 2022-01-31T21:53:57.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:57.803147+0000) 2022-01-31T21:53:58.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:57 smithi171 conmon[46715]: debug 2022-01-31T21:53:57.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:57.979006+0000) 2022-01-31T21:53:58.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:58 smithi171 conmon[51620]: debug 2022-01-31T21:53:58.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:58.072865+0000) 2022-01-31T21:53:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:58 smithi171 conmon[41853]: debug 2022-01-31T21:53:58.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:58.183583+0000) 2022-01-31T21:53:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:58 smithi167 conmon[54076]: debug 2022-01-31T21:53:58.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:58.468298+0000) 2022-01-31T21:53:58.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:58 smithi167 conmon[60316]: debug 2022-01-31T21:53:58.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:58.468296+0000) 2022-01-31T21:53:58.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:58 smithi167 conmon[49112]: debug 2022-01-31T21:53:58.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:58.803394+0000) 2022-01-31T21:53:59.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:58 smithi171 conmon[46715]: debug 2022-01-31T21:53:58.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:58.979188+0000) 2022-01-31T21:53:59.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:53:59 smithi171 conmon[51620]: debug 2022-01-31T21:53:59.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:59.073082+0000) 2022-01-31T21:53:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:53:59 smithi171 conmon[41853]: debug 2022-01-31T21:53:59.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:59.183747+0000) 2022-01-31T21:53:59.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:53:59 smithi167 conmon[54076]: debug 2022-01-31T21:53:59.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:59.468543+0000) 2022-01-31T21:53:59.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:53:59 smithi167 conmon[60316]: debug 2022-01-31T21:53:59.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:59.468538+0000) 2022-01-31T21:54:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:53:59 smithi167 conmon[49112]: debug 2022-01-31T21:53:59.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:59.803582+0000) 2022-01-31T21:54:00.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:53:59 smithi171 conmon[46715]: debug 2022-01-31T21:53:59.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:53:59.979371+0000) 2022-01-31T21:54:00.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[51620]: debug 2022-01-31T21:54:00.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.073263+0000) 2022-01-31T21:54:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[41853]: debug 2022-01-31T21:54:00.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.183988+0000) 2022-01-31T21:54:00.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[54076]: debug 2022-01-31T21:54:00.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.468739+0000) 2022-01-31T21:54:00.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[60316]: debug 2022-01-31T21:54:00.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.468723+0000) 2022-01-31T21:54:01.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[54076]: debug 2022-01-31T21:54:00.954+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.954571+0000) 2022-01-31T21:54:01.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[60316]: debug 2022-01-31T21:54:00.954+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.955167+0000) 2022-01-31T21:54:01.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[60316]: 2022-01-31T21:54:01.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[49112]: debug 2022-01-31T21:54:00.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.803729+0000) 2022-01-31T21:54:01.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:00 smithi167 conmon[49112]: debug 2022-01-31T21:54:00.953+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.954363+0000) 2022-01-31T21:54:01.182 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[35325]: debug 2022-01-31T21:54:00.970+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166450 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:01.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[46715]: debug 2022-01-31T21:54:00.953+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.954843+0000) 2022-01-31T21:54:01.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[46715]: debug 2022-01-31T21:54:00.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.979515+0000) 2022-01-31T21:54:01.184 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[41853]: debug 2022-01-31T21:54:00.952+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.954519+0000) 2022-01-31T21:54:01.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:00 smithi171 conmon[51620]: debug 2022-01-31T21:54:00.952+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:00.954202+0000) 2022-01-31T21:54:01.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:01 smithi171 conmon[51620]: debug 2022-01-31T21:54:01.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:01.073449+0000) 2022-01-31T21:54:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:01 smithi171 conmon[41853]: debug 2022-01-31T21:54:01.182+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:01.184192+0000) 2022-01-31T21:54:01.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:01 smithi167 conmon[54076]: debug 2022-01-31T21:54:01.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:01.468927+0000) 2022-01-31T21:54:01.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:01 smithi167 conmon[60316]: debug 2022-01-31T21:54:01.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:01.468928+0000) 2022-01-31T21:54:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:01 smithi167 conmon[49112]: debug 2022-01-31T21:54:01.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:01.803912+0000) 2022-01-31T21:54:02.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:01 smithi171 conmon[46715]: debug 2022-01-31T21:54:01.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:01.979646+0000) 2022-01-31T21:54:02.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:02 smithi171 conmon[51620]: debug 2022-01-31T21:54:02.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:02.073619+0000) 2022-01-31T21:54:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:02 smithi171 conmon[41853]: debug 2022-01-31T21:54:02.183+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:02.184345+0000) 2022-01-31T21:54:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:02 smithi167 conmon[54076]: debug 2022-01-31T21:54:02.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:02.469074+0000) 2022-01-31T21:54:02.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:02 smithi167 conmon[60316]: debug 2022-01-31T21:54:02.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:02.469094+0000) 2022-01-31T21:54:02.966 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:02 smithi167 conmon[49112]: debug 2022-01-31T21:54:02.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:02.804102+0000) 2022-01-31T21:54:03.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:02 smithi171 conmon[46715]: debug 2022-01-31T21:54:02.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:02.979801+0000) 2022-01-31T21:54:03.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:03 smithi171 conmon[51620]: debug 2022-01-31T21:54:03.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:03.073804+0000) 2022-01-31T21:54:03.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:03 smithi171 conmon[41853]: debug 2022-01-31T21:54:03.183+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:03.184535+0000) 2022-01-31T21:54:03.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:03 smithi167 conmon[54076]: debug 2022-01-31T21:54:03.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:03.469280+0000) 2022-01-31T21:54:03.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:03 smithi167 conmon[60316]: debug 2022-01-31T21:54:03.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:03.469287+0000) 2022-01-31T21:54:04.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:03 smithi167 conmon[49112]: debug 2022-01-31T21:54:03.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:03.804286+0000) 2022-01-31T21:54:04.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:03 smithi171 conmon[46715]: debug 2022-01-31T21:54:03.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:03.980015+0000) 2022-01-31T21:54:04.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:04 smithi171 conmon[51620]: debug 2022-01-31T21:54:04.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:04.073986+0000) 2022-01-31T21:54:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:04 smithi171 conmon[41853]: debug 2022-01-31T21:54:04.183+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:04.184689+0000) 2022-01-31T21:54:04.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:04 smithi167 conmon[54076]: debug 2022-01-31T21:54:04.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:04.469485+0000) 2022-01-31T21:54:04.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:04 smithi167 conmon[60316]: debug 2022-01-31T21:54:04.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:04.469484+0000) 2022-01-31T21:54:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:04 smithi167 conmon[49112]: debug 2022-01-31T21:54:04.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:04.804528+0000) 2022-01-31T21:54:05.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:04 smithi171 conmon[46715]: debug 2022-01-31T21:54:04.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:04.980174+0000) 2022-01-31T21:54:05.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[51620]: debug 2022-01-31T21:54:05.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.074181+0000) 2022-01-31T21:54:05.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[41853]: debug 2022-01-31T21:54:05.184+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.184906+0000) 2022-01-31T21:54:05.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:05 smithi167 conmon[54076]: debug 2022-01-31T21:54:05.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.469698+0000) 2022-01-31T21:54:05.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:05 smithi167 conmon[60316]: debug 2022-01-31T21:54:05.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.469701+0000) 2022-01-31T21:54:06.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:05 smithi167 conmon[54076]: debug 2022-01-31T21:54:05.973+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.974260+0000) 2022-01-31T21:54:06.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:05 smithi167 conmon[60316]: debug 2022-01-31T21:54:05.973+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.974289+0000) 2022-01-31T21:54:06.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:05 smithi167 conmon[49112]: debug 2022-01-31T21:54:05.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.804707+0000) 2022-01-31T21:54:06.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:05 smithi167 conmon[49112]: debug 2022-01-31T21:54:05.974+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.975414+0000) 2022-01-31T21:54:06.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[35325]: debug 2022-01-31T21:54:05.990+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166562 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:06.184 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[41853]: debug 2022-01-31T21:54:05.972+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.974053+0000) 2022-01-31T21:54:06.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[46715]: debug 2022-01-31T21:54:05.973+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.975528+0000) 2022-01-31T21:54:06.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[46715]: debug 2022-01-31T21:54:05.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.980338+0000) 2022-01-31T21:54:06.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:05 smithi171 conmon[51620]: debug 2022-01-31T21:54:05.973+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:05.974947+0000) 2022-01-31T21:54:06.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:06 smithi171 conmon[51620]: debug 2022-01-31T21:54:06.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:06.074355+0000) 2022-01-31T21:54:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:06 smithi171 conmon[41853]: debug 2022-01-31T21:54:06.183+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:06.185104+0000) 2022-01-31T21:54:06.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:06 smithi167 conmon[54076]: debug 2022-01-31T21:54:06.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:06.469867+0000) 2022-01-31T21:54:06.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:06 smithi167 conmon[60316]: debug 2022-01-31T21:54:06.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:06.469867+0000) 2022-01-31T21:54:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:06 smithi167 conmon[49112]: debug 2022-01-31T21:54:06.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:06.804898+0000) 2022-01-31T21:54:07.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:06 smithi171 conmon[46715]: debug 2022-01-31T21:54:06.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:06.980459+0000) 2022-01-31T21:54:07.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:07 smithi171 conmon[51620]: debug 2022-01-31T21:54:07.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:07.074514+0000) 2022-01-31T21:54:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:07 smithi171 conmon[41853]: debug 2022-01-31T21:54:07.184+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:07.185253+0000) 2022-01-31T21:54:07.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:07 smithi167 conmon[54076]: debug 2022-01-31T21:54:07.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:07.469991+0000) 2022-01-31T21:54:07.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:07 smithi167 conmon[60316]: debug 2022-01-31T21:54:07.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:07.470028+0000) 2022-01-31T21:54:08.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:07 smithi167 conmon[49112]: debug 2022-01-31T21:54:07.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:07.805045+0000) 2022-01-31T21:54:08.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:07 smithi171 conmon[46715]: debug 2022-01-31T21:54:07.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:07.980528+0000) 2022-01-31T21:54:08.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:08 smithi171 conmon[51620]: debug 2022-01-31T21:54:08.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:08.074662+0000) 2022-01-31T21:54:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:08 smithi171 conmon[41853]: debug 2022-01-31T21:54:08.184+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:08.185431+0000) 2022-01-31T21:54:08.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:08 smithi167 conmon[54076]: debug 2022-01-31T21:54:08.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:08.470227+0000) 2022-01-31T21:54:08.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:08 smithi167 conmon[60316]: debug 2022-01-31T21:54:08.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:08.470257+0000) 2022-01-31T21:54:09.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:08 smithi167 conmon[49112]: debug 2022-01-31T21:54:08.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:08.805217+0000) 2022-01-31T21:54:09.073 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:08 smithi171 conmon[46715]: debug 2022-01-31T21:54:08.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:08.980683+0000) 2022-01-31T21:54:09.074 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:08 smithi171 conmon[35325]: debug 2022-01-31T21:54:08.757+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:54:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:09 smithi171 conmon[41853]: debug 2022-01-31T21:54:09.184+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:09.185624+0000) 2022-01-31T21:54:09.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:09 smithi171 conmon[51620]: debug 2022-01-31T21:54:09.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:09.074859+0000) 2022-01-31T21:54:09.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:09 smithi167 conmon[54076]: debug 2022-01-31T21:54:09.468+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:09.470377+0000) 2022-01-31T21:54:09.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:09 smithi167 conmon[60316]: debug 2022-01-31T21:54:09.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:09.470382+0000) 2022-01-31T21:54:10.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:09 smithi167 conmon[49112]: debug 2022-01-31T21:54:09.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:09.805426+0000) 2022-01-31T21:54:10.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:09 smithi171 conmon[46715]: debug 2022-01-31T21:54:09.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:09.980864+0000) 2022-01-31T21:54:10.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:10 smithi171 conmon[51620]: debug 2022-01-31T21:54:10.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.075032+0000) 2022-01-31T21:54:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:10 smithi171 conmon[41853]: debug 2022-01-31T21:54:10.185+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.185812+0000) 2022-01-31T21:54:10.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:10 smithi167 conmon[54076]: debug 2022-01-31T21:54:10.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.470564+0000) 2022-01-31T21:54:10.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:10 smithi167 conmon[60316]: debug 2022-01-31T21:54:10.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.470564+0000) 2022-01-31T21:54:11.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:10 smithi167 conmon[54076]: debug 2022-01-31T21:54:10.992+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.994392+0000) 2022-01-31T21:54:11.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:10 smithi167 conmon[60316]: debug 2022-01-31T21:54:10.993+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.995185+0000) 2022-01-31T21:54:11.036 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:10 smithi167 conmon[49112]: debug 2022-01-31T21:54:10.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.805635+0000) 2022-01-31T21:54:11.036 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:10 smithi167 conmon[49112]: debug 2022-01-31T21:54:10.993+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.995478+0000) 2022-01-31T21:54:11.184 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:11 smithi171 conmon[35325]: debug 2022-01-31T21:54:11.011+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166671 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:11.185 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:10 smithi171 conmon[41853]: debug 2022-01-31T21:54:10.992+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.993733+0000) 2022-01-31T21:54:11.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:10 smithi171 conmon[46715]: debug 2022-01-31T21:54:10.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.981040+0000) 2022-01-31T21:54:11.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:10 smithi171 conmon[46715]: debug 2022-01-31T21:54:10.995+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.996088+0000) 2022-01-31T21:54:11.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:10 smithi171 conmon[51620]: debug 2022-01-31T21:54:10.994+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:10.995196+0000) 2022-01-31T21:54:11.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:11 smithi171 conmon[51620]: debug 2022-01-31T21:54:11.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:11.075220+0000) 2022-01-31T21:54:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:11 smithi171 conmon[41853]: debug 2022-01-31T21:54:11.185+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:11.185926+0000) 2022-01-31T21:54:11.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:11 smithi167 conmon[54076]: debug 2022-01-31T21:54:11.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:11.470743+0000) 2022-01-31T21:54:11.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:11 smithi167 conmon[60316]: debug 2022-01-31T21:54:11.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:11.470751+0000) 2022-01-31T21:54:12.036 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:11 smithi167 conmon[49112]: debug 2022-01-31T21:54:11.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:11.805806+0000) 2022-01-31T21:54:12.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:12 smithi171 conmon[51620]: debug 2022-01-31T21:54:12.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:12.075388+0000) 2022-01-31T21:54:12.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:11 smithi171 conmon[46715]: debug 2022-01-31T21:54:11.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:11.981245+0000) 2022-01-31T21:54:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:12 smithi171 conmon[41853]: debug 2022-01-31T21:54:12.185+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:12.186016+0000) 2022-01-31T21:54:12.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:12 smithi167 conmon[54076]: debug 2022-01-31T21:54:12.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:12.470844+0000) 2022-01-31T21:54:12.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:12 smithi167 conmon[60316]: debug 2022-01-31T21:54:12.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:12.470883+0000) 2022-01-31T21:54:13.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:12 smithi167 conmon[49112]: debug 2022-01-31T21:54:12.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:12.805966+0000) 2022-01-31T21:54:13.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:12 smithi171 conmon[46715]: debug 2022-01-31T21:54:12.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:12.981398+0000) 2022-01-31T21:54:13.237 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:13 smithi171 conmon[51620]: debug 2022-01-31T21:54:13.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:13.075539+0000) 2022-01-31T21:54:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:13 smithi171 conmon[41853]: debug 2022-01-31T21:54:13.185+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:13.186211+0000) 2022-01-31T21:54:13.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:13 smithi167 conmon[54076]: debug 2022-01-31T21:54:13.469+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:13.471052+0000) 2022-01-31T21:54:13.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:13 smithi167 conmon[60316]: debug 2022-01-31T21:54:13.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:13.471042+0000) 2022-01-31T21:54:14.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:13 smithi167 conmon[49112]: debug 2022-01-31T21:54:13.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:13.806207+0000) 2022-01-31T21:54:14.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:13 smithi171 conmon[46715]: debug 2022-01-31T21:54:13.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:13.981600+0000) 2022-01-31T21:54:14.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:14 smithi171 conmon[51620]: debug 2022-01-31T21:54:14.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:14.075656+0000) 2022-01-31T21:54:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:14 smithi171 conmon[41853]: debug 2022-01-31T21:54:14.185+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:14.186392+0000) 2022-01-31T21:54:14.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:14 smithi167 conmon[60316]: debug 2022-01-31T21:54:14.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:14.471185+0000) 2022-01-31T21:54:14.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:14 smithi167 conmon[54076]: debug 2022-01-31T21:54:14.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:14.471220+0000) 2022-01-31T21:54:15.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:14 smithi167 conmon[49112]: debug 2022-01-31T21:54:14.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:14.806429+0000) 2022-01-31T21:54:15.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:14 smithi171 conmon[46715]: debug 2022-01-31T21:54:14.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:14.981819+0000) 2022-01-31T21:54:15.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:15 smithi171 conmon[51620]: debug 2022-01-31T21:54:15.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:15.075812+0000) 2022-01-31T21:54:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:15 smithi171 conmon[41853]: debug 2022-01-31T21:54:15.185+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:15.186571+0000) 2022-01-31T21:54:15.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:15 smithi167 conmon[54076]: debug 2022-01-31T21:54:15.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:15.471357+0000) 2022-01-31T21:54:15.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:15 smithi167 conmon[60316]: debug 2022-01-31T21:54:15.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:15.471342+0000) 2022-01-31T21:54:16.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:16 smithi167 conmon[54076]: debug 2022-01-31T21:54:16.012+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.014265+0000) 2022-01-31T21:54:16.045 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:16 smithi167 conmon[60316]: debug 2022-01-31T21:54:16.012+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.013966+0000) 2022-01-31T21:54:16.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:15 smithi167 conmon[49112]: debug 2022-01-31T21:54:15.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:15.806622+0000) 2022-01-31T21:54:16.046 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:16 smithi167 conmon[49112]: debug 2022-01-31T21:54:16.013+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.015143+0000) 2022-01-31T21:54:16.185 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[35325]: debug 2022-01-31T21:54:16.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166784 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:16.186 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[41853]: debug 2022-01-31T21:54:16.013+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.014493+0000) 2022-01-31T21:54:16.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:15 smithi171 conmon[46715]: debug 2022-01-31T21:54:15.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:15.981987+0000) 2022-01-31T21:54:16.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[46715]: debug 2022-01-31T21:54:16.014+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.014876+0000) 2022-01-31T21:54:16.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[51620]: debug 2022-01-31T21:54:16.014+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.015173+0000) 2022-01-31T21:54:16.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[51620]: debug 2022-01-31T21:54:16.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.075956+0000) 2022-01-31T21:54:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[41853]: debug 2022-01-31T21:54:16.186+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.186789+0000) 2022-01-31T21:54:16.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:16 smithi167 conmon[54076]: debug 2022-01-31T21:54:16.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.471534+0000) 2022-01-31T21:54:16.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:16 smithi167 conmon[60316]: debug 2022-01-31T21:54:16.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.471547+0000) 2022-01-31T21:54:17.047 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:16 smithi167 conmon[49112]: debug 2022-01-31T21:54:16.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.806800+0000) 2022-01-31T21:54:17.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:16 smithi171 conmon[46715]: debug 2022-01-31T21:54:16.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:16.982137+0000) 2022-01-31T21:54:17.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:17 smithi171 conmon[51620]: debug 2022-01-31T21:54:17.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:17.076095+0000) 2022-01-31T21:54:17.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:17 smithi171 conmon[41853]: debug 2022-01-31T21:54:17.186+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:17.186896+0000) 2022-01-31T21:54:17.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:17 smithi167 conmon[54076]: debug 2022-01-31T21:54:17.470+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:17.471667+0000) 2022-01-31T21:54:17.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:17 smithi167 conmon[60316]: debug 2022-01-31T21:54:17.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:17.471703+0000) 2022-01-31T21:54:18.051 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:17 smithi167 conmon[49112]: debug 2022-01-31T21:54:17.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:17.806920+0000) 2022-01-31T21:54:18.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:18 smithi171 conmon[51620]: debug 2022-01-31T21:54:18.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:18.076278+0000) 2022-01-31T21:54:18.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:17 smithi171 conmon[46715]: debug 2022-01-31T21:54:17.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:17.982295+0000) 2022-01-31T21:54:18.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:18 smithi171 conmon[41853]: debug 2022-01-31T21:54:18.186+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:18.187013+0000) 2022-01-31T21:54:18.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:18 smithi167 conmon[54076]: debug 2022-01-31T21:54:18.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:18.471870+0000) 2022-01-31T21:54:18.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:18 smithi167 conmon[60316]: debug 2022-01-31T21:54:18.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:18.471873+0000) 2022-01-31T21:54:19.054 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:18 smithi167 conmon[49112]: debug 2022-01-31T21:54:18.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:18.807109+0000) 2022-01-31T21:54:19.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:18 smithi171 conmon[46715]: debug 2022-01-31T21:54:18.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:18.982508+0000) 2022-01-31T21:54:19.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:19 smithi171 conmon[51620]: debug 2022-01-31T21:54:19.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:19.076489+0000) 2022-01-31T21:54:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:19 smithi171 conmon[41853]: debug 2022-01-31T21:54:19.186+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:19.187235+0000) 2022-01-31T21:54:19.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:19 smithi167 conmon[54076]: debug 2022-01-31T21:54:19.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:19.472093+0000) 2022-01-31T21:54:19.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:19 smithi167 conmon[60316]: debug 2022-01-31T21:54:19.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:19.472069+0000) 2022-01-31T21:54:20.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:19 smithi167 conmon[49112]: debug 2022-01-31T21:54:19.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:19.807321+0000) 2022-01-31T21:54:20.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:19 smithi171 conmon[46715]: debug 2022-01-31T21:54:19.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:19.982716+0000) 2022-01-31T21:54:20.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:20 smithi171 conmon[51620]: debug 2022-01-31T21:54:20.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:20.076670+0000) 2022-01-31T21:54:20.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:20 smithi171 conmon[41853]: debug 2022-01-31T21:54:20.186+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:20.187426+0000) 2022-01-31T21:54:20.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:20 smithi167 conmon[54076]: debug 2022-01-31T21:54:20.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:20.472288+0000) 2022-01-31T21:54:20.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:20 smithi167 conmon[60316]: debug 2022-01-31T21:54:20.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:20.472266+0000) 2022-01-31T21:54:21.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:20 smithi167 conmon[49112]: debug 2022-01-31T21:54:20.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:20.807555+0000) 2022-01-31T21:54:21.186 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[35325]: debug 2022-01-31T21:54:21.051+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 166893 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:21.187 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[41853]: debug 2022-01-31T21:54:21.033+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.034710+0000) 2022-01-31T21:54:21.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[51620]: debug 2022-01-31T21:54:21.034+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.035315+0000) 2022-01-31T21:54:21.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[51620]: debug 2022-01-31T21:54:21.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.076795+0000) 2022-01-31T21:54:21.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:20 smithi171 conmon[46715]: debug 2022-01-31T21:54:20.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:20.982917+0000) 2022-01-31T21:54:21.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[46715]: debug 2022-01-31T21:54:21.033+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.034323+0000) 2022-01-31T21:54:21.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:21 smithi167 conmon[49112]: debug 2022-01-31T21:54:21.034+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.035521+0000) 2022-01-31T21:54:21.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:21 smithi167 conmon[54076]: debug 2022-01-31T21:54:21.033+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.034981+0000) 2022-01-31T21:54:21.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:21 smithi167 conmon[60316]: debug 2022-01-31T21:54:21.032+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.034359+0000) 2022-01-31T21:54:21.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[41853]: debug 2022-01-31T21:54:21.186+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.187573+0000) 2022-01-31T21:54:21.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:21 smithi167 conmon[60316]: debug 2022-01-31T21:54:21.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.472455+0000) 2022-01-31T21:54:21.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:21 smithi167 conmon[54076]: debug 2022-01-31T21:54:21.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.472455+0000) 2022-01-31T21:54:22.059 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:21 smithi167 conmon[49112]: debug 2022-01-31T21:54:21.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.807767+0000) 2022-01-31T21:54:22.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:21 smithi171 conmon[46715]: debug 2022-01-31T21:54:21.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:21.983091+0000) 2022-01-31T21:54:22.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:22 smithi171 conmon[51620]: debug 2022-01-31T21:54:22.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:22.076941+0000) 2022-01-31T21:54:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:22 smithi171 conmon[41853]: debug 2022-01-31T21:54:22.187+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:22.187733+0000) 2022-01-31T21:54:22.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:22 smithi167 conmon[54076]: debug 2022-01-31T21:54:22.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:22.472626+0000) 2022-01-31T21:54:22.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:22 smithi167 conmon[60316]: debug 2022-01-31T21:54:22.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:22.472666+0000) 2022-01-31T21:54:23.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:22 smithi167 conmon[49112]: debug 2022-01-31T21:54:22.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:22.807940+0000) 2022-01-31T21:54:23.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:22 smithi171 conmon[46715]: debug 2022-01-31T21:54:22.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:22.983244+0000) 2022-01-31T21:54:23.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:23 smithi171 conmon[51620]: debug 2022-01-31T21:54:23.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:23.077142+0000) 2022-01-31T21:54:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:23 smithi171 conmon[41853]: debug 2022-01-31T21:54:23.187+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:23.187981+0000) 2022-01-31T21:54:23.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:23 smithi167 conmon[54076]: debug 2022-01-31T21:54:23.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:23.472766+0000) 2022-01-31T21:54:23.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:23 smithi167 conmon[60316]: debug 2022-01-31T21:54:23.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:23.472847+0000) 2022-01-31T21:54:24.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:23 smithi167 conmon[49112]: debug 2022-01-31T21:54:23.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:23.808132+0000) 2022-01-31T21:54:24.075 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:23 smithi171 conmon[46715]: debug 2022-01-31T21:54:23.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:23.983504+0000) 2022-01-31T21:54:24.076 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:23 smithi171 conmon[35325]: debug 2022-01-31T21:54:23.758+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:54:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:24 smithi171 conmon[41853]: debug 2022-01-31T21:54:24.187+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:24.188211+0000) 2022-01-31T21:54:24.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:24 smithi171 conmon[51620]: debug 2022-01-31T21:54:24.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:24.077313+0000) 2022-01-31T21:54:24.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:24 smithi167 conmon[54076]: debug 2022-01-31T21:54:24.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:24.472928+0000) 2022-01-31T21:54:24.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:24 smithi167 conmon[60316]: debug 2022-01-31T21:54:24.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:24.473053+0000) 2022-01-31T21:54:25.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:24 smithi167 conmon[49112]: debug 2022-01-31T21:54:24.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:24.808334+0000) 2022-01-31T21:54:25.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:24 smithi171 conmon[46715]: debug 2022-01-31T21:54:24.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:24.983696+0000) 2022-01-31T21:54:25.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:25 smithi171 conmon[51620]: debug 2022-01-31T21:54:25.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:25.077511+0000) 2022-01-31T21:54:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:25 smithi171 conmon[41853]: debug 2022-01-31T21:54:25.187+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:25.188371+0000) 2022-01-31T21:54:25.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:25 smithi167 conmon[54076]: debug 2022-01-31T21:54:25.471+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:25.473071+0000) 2022-01-31T21:54:25.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:25 smithi167 conmon[60316]: debug 2022-01-31T21:54:25.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:25.473237+0000) 2022-01-31T21:54:26.053 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:25 smithi167 conmon[49112]: debug 2022-01-31T21:54:25.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:25.808539+0000) 2022-01-31T21:54:26.187 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[35325]: debug 2022-01-31T21:54:26.076+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167005 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:26.188 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[41853]: debug 2022-01-31T21:54:26.053+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.053945+0000) 2022-01-31T21:54:26.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[51620]: debug 2022-01-31T21:54:26.053+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.054722+0000) 2022-01-31T21:54:26.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[51620]: debug 2022-01-31T21:54:26.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.077693+0000) 2022-01-31T21:54:26.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:25 smithi171 conmon[46715]: debug 2022-01-31T21:54:25.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:25.983904+0000) 2022-01-31T21:54:26.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[46715]: debug 2022-01-31T21:54:26.053+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.053816+0000) 2022-01-31T21:54:26.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:26 smithi167 conmon[49112]: debug 2022-01-31T21:54:26.053+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.055064+0000) 2022-01-31T21:54:26.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:26 smithi167 conmon[54076]: debug 2022-01-31T21:54:26.053+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.054885+0000) 2022-01-31T21:54:26.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:26 smithi167 conmon[60316]: debug 2022-01-31T21:54:26.060+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.062380+0000) 2022-01-31T21:54:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[41853]: debug 2022-01-31T21:54:26.188+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.188572+0000) 2022-01-31T21:54:26.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:26 smithi167 conmon[54076]: debug 2022-01-31T21:54:26.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.473271+0000) 2022-01-31T21:54:26.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:26 smithi167 conmon[60316]: debug 2022-01-31T21:54:26.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.473374+0000) 2022-01-31T21:54:27.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:26 smithi167 conmon[49112]: debug 2022-01-31T21:54:26.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.808743+0000) 2022-01-31T21:54:27.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:26 smithi171 conmon[46715]: debug 2022-01-31T21:54:26.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:26.984085+0000) 2022-01-31T21:54:27.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:27 smithi171 conmon[51620]: debug 2022-01-31T21:54:27.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:27.077806+0000) 2022-01-31T21:54:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:27 smithi171 conmon[41853]: debug 2022-01-31T21:54:27.187+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:27.188686+0000) 2022-01-31T21:54:27.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:27 smithi167 conmon[54076]: debug 2022-01-31T21:54:27.472+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:27.473408+0000) 2022-01-31T21:54:27.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:27 smithi167 conmon[60316]: debug 2022-01-31T21:54:27.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:27 smithi167 conmon[60316]: 2022-01-31T21:54:27.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:27.473480+0000) 2022-01-31T21:54:28.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:27 smithi167 conmon[49112]: debug 2022-01-31T21:54:27.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:27.808909+0000) 2022-01-31T21:54:28.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:27 smithi171 conmon[46715]: debug 2022-01-31T21:54:27.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:27.984245+0000) 2022-01-31T21:54:28.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:28 smithi171 conmon[51620]: debug 2022-01-31T21:54:28.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:28.077939+0000) 2022-01-31T21:54:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:28 smithi171 conmon[41853]: debug 2022-01-31T21:54:28.188+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:28.188847+0000) 2022-01-31T21:54:28.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:28 smithi167 conmon[54076]: debug 2022-01-31T21:54:28.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:28.473616+0000) 2022-01-31T21:54:28.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:28 smithi167 conmon[60316]: debug 2022-01-31T21:54:28.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:28.473682+0000) 2022-01-31T21:54:29.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:28 smithi167 conmon[49112]: debug 2022-01-31T21:54:28.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:28.809153+0000) 2022-01-31T21:54:29.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:28 smithi171 conmon[46715]: debug 2022-01-31T21:54:28.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:28.984431+0000) 2022-01-31T21:54:29.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:29 smithi171 conmon[51620]: debug 2022-01-31T21:54:29.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:29.078110+0000) 2022-01-31T21:54:29.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:29 smithi171 conmon[41853]: debug 2022-01-31T21:54:29.188+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:29.188946+0000) 2022-01-31T21:54:29.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:29 smithi167 conmon[54076]: debug 2022-01-31T21:54:29.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:29.473740+0000) 2022-01-31T21:54:29.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:29 smithi167 conmon[60316]: debug 2022-01-31T21:54:29.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:29.473808+0000) 2022-01-31T21:54:30.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:29 smithi167 conmon[49112]: debug 2022-01-31T21:54:29.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:29.809365+0000) 2022-01-31T21:54:30.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:29 smithi171 conmon[46715]: debug 2022-01-31T21:54:29.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:29.984589+0000) 2022-01-31T21:54:30.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:30 smithi171 conmon[51620]: debug 2022-01-31T21:54:30.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:30.078312+0000) 2022-01-31T21:54:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:30 smithi171 conmon[41853]: debug 2022-01-31T21:54:30.188+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:30.189101+0000) 2022-01-31T21:54:30.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:30 smithi167 conmon[54076]: debug 2022-01-31T21:54:30.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:30.473968+0000) 2022-01-31T21:54:30.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:30 smithi167 conmon[60316]: debug 2022-01-31T21:54:30.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:30.474010+0000) 2022-01-31T21:54:31.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:30 smithi167 conmon[49112]: debug 2022-01-31T21:54:30.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:30.809567+0000) 2022-01-31T21:54:31.187 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:31 smithi171 conmon[35325]: debug 2022-01-31T21:54:31.097+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167114 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:31.188 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:31 smithi171 conmon[41853]: debug 2022-01-31T21:54:31.078+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.079303+0000) 2022-01-31T21:54:31.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:31 smithi171 conmon[51620]: debug 2022-01-31T21:54:31.079+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.080519+0000) 2022-01-31T21:54:31.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:30 smithi171 conmon[46715]: debug 2022-01-31T21:54:30.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:30.984739+0000) 2022-01-31T21:54:31.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:31 smithi171 conmon[46715]: debug 2022-01-31T21:54:31.078+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.078853+0000) 2022-01-31T21:54:31.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:31 smithi167 conmon[49112]: debug 2022-01-31T21:54:31.078+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.080075+0000) 2022-01-31T21:54:31.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:31 smithi167 conmon[54076]: debug 2022-01-31T21:54:31.078+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.079897+0000) 2022-01-31T21:54:31.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:31 smithi167 conmon[60316]: debug 2022-01-31T21:54:31.079+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.080802+0000) 2022-01-31T21:54:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:31 smithi171 conmon[41853]: debug 2022-01-31T21:54:31.188+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.189275+0000) 2022-01-31T21:54:31.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:31 smithi167 conmon[60316]: debug 2022-01-31T21:54:31.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.474180+0000) 2022-01-31T21:54:31.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:31 smithi167 conmon[54076]: debug 2022-01-31T21:54:31.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.474155+0000) 2022-01-31T21:54:32.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:31 smithi167 conmon[49112]: debug 2022-01-31T21:54:31.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.809755+0000) 2022-01-31T21:54:32.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:31 smithi171 conmon[46715]: debug 2022-01-31T21:54:31.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:31.984916+0000) 2022-01-31T21:54:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:32 smithi171 conmon[41853]: debug 2022-01-31T21:54:32.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:32.189425+0000) 2022-01-31T21:54:32.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:32 smithi167 conmon[54076]: debug 2022-01-31T21:54:32.473+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:32.474344+0000) 2022-01-31T21:54:32.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:32 smithi167 conmon[60316]: debug 2022-01-31T21:54:32.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:32.474342+0000) 2022-01-31T21:54:33.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:32 smithi167 conmon[49112]: debug 2022-01-31T21:54:32.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:32.809897+0000) 2022-01-31T21:54:33.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:32 smithi171 conmon[46715]: debug 2022-01-31T21:54:32.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:32.985045+0000) 2022-01-31T21:54:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:33 smithi171 conmon[41853]: debug 2022-01-31T21:54:33.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:33.189631+0000) 2022-01-31T21:54:33.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:33 smithi167 conmon[54076]: debug 2022-01-31T21:54:33.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:33.474542+0000) 2022-01-31T21:54:33.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:33 smithi167 conmon[60316]: debug 2022-01-31T21:54:33.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:33.474533+0000) 2022-01-31T21:54:34.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:33 smithi167 conmon[49112]: debug 2022-01-31T21:54:33.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:33.810107+0000) 2022-01-31T21:54:34.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:33 smithi171 conmon[46715]: debug 2022-01-31T21:54:33.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:33.985207+0000) 2022-01-31T21:54:34.611 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:34 smithi171 conmon[41853]: debug 2022-01-31T21:54:34.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:34.189827+0000) 2022-01-31T21:54:34.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:34 smithi167 conmon[54076]: debug 2022-01-31T21:54:34.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:34.474760+0000) 2022-01-31T21:54:34.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:34 smithi167 conmon[60316]: debug 2022-01-31T21:54:34.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:34.474760+0000) 2022-01-31T21:54:35.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:34 smithi167 conmon[49112]: debug 2022-01-31T21:54:34.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:34.810318+0000) 2022-01-31T21:54:35.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:34 smithi171 conmon[46715]: debug 2022-01-31T21:54:34.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:34.985360+0000) 2022-01-31T21:54:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:35 smithi171 conmon[41853]: debug 2022-01-31T21:54:35.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:35.190028+0000) 2022-01-31T21:54:35.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:35 smithi167 conmon[54076]: debug 2022-01-31T21:54:35.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:35.474945+0000) 2022-01-31T21:54:35.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:35 smithi167 conmon[60316]: debug 2022-01-31T21:54:35.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:35.474949+0000) 2022-01-31T21:54:36.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:35 smithi167 conmon[49112]: debug 2022-01-31T21:54:35.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:35.810547+0000) 2022-01-31T21:54:36.188 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:36 smithi171 conmon[35325]: debug 2022-01-31T21:54:36.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167226 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:36.189 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:36 smithi171 conmon[41853]: debug 2022-01-31T21:54:36.099+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.100503+0000) 2022-01-31T21:54:36.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:35 smithi171 conmon[46715]: debug 2022-01-31T21:54:35.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:35.985557+0000) 2022-01-31T21:54:36.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:36 smithi171 conmon[46715]: debug 2022-01-31T21:54:36.099+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.100194+0000) 2022-01-31T21:54:36.191 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:36 smithi171 conmon[51620]: debug 2022-01-31T21:54:36.099+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.099888+0000) 2022-01-31T21:54:36.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:36 smithi167 conmon[49112]: debug 2022-01-31T21:54:36.099+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.100305+0000) 2022-01-31T21:54:36.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:36 smithi167 conmon[54076]: debug 2022-01-31T21:54:36.098+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.099267+0000) 2022-01-31T21:54:36.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:36 smithi167 conmon[60316]: debug 2022-01-31T21:54:36.099+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.100469+0000) 2022-01-31T21:54:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:36 smithi171 conmon[41853]: debug 2022-01-31T21:54:36.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.190222+0000) 2022-01-31T21:54:36.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:36 smithi167 conmon[54076]: debug 2022-01-31T21:54:36.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.475138+0000) 2022-01-31T21:54:36.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:36 smithi167 conmon[60316]: debug 2022-01-31T21:54:36.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.475142+0000) 2022-01-31T21:54:37.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:36 smithi167 conmon[49112]: debug 2022-01-31T21:54:36.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.810712+0000) 2022-01-31T21:54:37.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:36 smithi171 conmon[46715]: debug 2022-01-31T21:54:36.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:36.985776+0000) 2022-01-31T21:54:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:37 smithi171 conmon[41853]: debug 2022-01-31T21:54:37.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:37.190331+0000) 2022-01-31T21:54:37.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:37 smithi167 conmon[54076]: debug 2022-01-31T21:54:37.474+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:37.475293+0000) 2022-01-31T21:54:37.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:37 smithi167 conmon[60316]: debug 2022-01-31T21:54:37.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:37.475293+0000) 2022-01-31T21:54:38.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:37 smithi167 conmon[49112]: debug 2022-01-31T21:54:37.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:37.810878+0000) 2022-01-31T21:54:38.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:37 smithi171 conmon[46715]: debug 2022-01-31T21:54:37.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:37.985918+0000) 2022-01-31T21:54:38.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:37 smithi171 conmon[51620]: debug 2022-01-31T21:54:37.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:37.828733+0000) 2022-01-31T21:54:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:38 smithi171 conmon[41853]: debug 2022-01-31T21:54:38.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:38.190530+0000) 2022-01-31T21:54:38.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:38 smithi167 conmon[54076]: debug 2022-01-31T21:54:38.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:38.475532+0000) 2022-01-31T21:54:38.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:38 smithi167 conmon[60316]: debug 2022-01-31T21:54:38.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:38.475532+0000) 2022-01-31T21:54:39.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:38 smithi167 conmon[49112]: debug 2022-01-31T21:54:38.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:38.811086+0000) 2022-01-31T21:54:39.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:38 smithi171 conmon[35325]: debug 2022-01-31T21:54:38.759+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:54:39.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:38 smithi171 conmon[46715]: debug 2022-01-31T21:54:38.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:38.986123+0000) 2022-01-31T21:54:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:38 smithi171 conmon[51620]: debug 2022-01-31T21:54:38.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:38.828904+0000) 2022-01-31T21:54:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:39 smithi171 conmon[41853]: debug 2022-01-31T21:54:39.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:39.190715+0000) 2022-01-31T21:54:39.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:39 smithi167 conmon[54076]: debug 2022-01-31T21:54:39.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:39.475696+0000) 2022-01-31T21:54:39.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:39 smithi167 conmon[60316]: debug 2022-01-31T21:54:39.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:39.475681+0000) 2022-01-31T21:54:40.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:39 smithi167 conmon[49112]: debug 2022-01-31T21:54:39.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:39.811302+0000) 2022-01-31T21:54:40.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:39 smithi171 conmon[46715]: debug 2022-01-31T21:54:39.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:39.986323+0000) 2022-01-31T21:54:40.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:39 smithi171 conmon[51620]: debug 2022-01-31T21:54:39.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:39.829075+0000) 2022-01-31T21:54:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:40 smithi171 conmon[41853]: debug 2022-01-31T21:54:40.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:40.190912+0000) 2022-01-31T21:54:40.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:40 smithi167 conmon[54076]: debug 2022-01-31T21:54:40.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:40.475913+0000) 2022-01-31T21:54:40.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:40 smithi167 conmon[60316]: debug 2022-01-31T21:54:40.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:40.475865+0000) 2022-01-31T21:54:41.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:40 smithi167 conmon[49112]: debug 2022-01-31T21:54:40.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:40.811482+0000) 2022-01-31T21:54:41.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:40 smithi171 conmon[46715]: debug 2022-01-31T21:54:40.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:40.986529+0000) 2022-01-31T21:54:41.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:40 smithi171 conmon[51620]: debug 2022-01-31T21:54:40.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:40.829232+0000) 2022-01-31T21:54:41.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:41 smithi167 conmon[49112]: debug 2022-01-31T21:54:41.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.120138+0000) 2022-01-31T21:54:41.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:41 smithi167 conmon[54076]: debug 2022-01-31T21:54:41.120+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.120586+0000) 2022-01-31T21:54:41.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:41 smithi167 conmon[60316]: debug 2022-01-31T21:54:41.119+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.120292+0000) 2022-01-31T21:54:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[41853]: debug 2022-01-31T21:54:41.120+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.121332+0000) 2022-01-31T21:54:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[41853]: debug 2022-01-31T21:54:41.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.191001+0000) 2022-01-31T21:54:41.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[46715]: debug 2022-01-31T21:54:41.119+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.120438+0000) 2022-01-31T21:54:41.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[51620]: debug 2022-01-31T21:54:41.119+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.120475+0000) 2022-01-31T21:54:41.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[35325]: debug 2022-01-31T21:54:41.137+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167335 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:41.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:41 smithi167 conmon[54076]: debug 2022-01-31T21:54:41.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.476097+0000) 2022-01-31T21:54:41.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:41 smithi167 conmon[60316]: debug 2022-01-31T21:54:41.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.476087+0000) 2022-01-31T21:54:42.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:41 smithi167 conmon[49112]: debug 2022-01-31T21:54:41.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.811679+0000) 2022-01-31T21:54:42.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[46715]: debug 2022-01-31T21:54:41.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.986731+0000) 2022-01-31T21:54:42.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:41 smithi171 conmon[51620]: debug 2022-01-31T21:54:41.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:41.829372+0000) 2022-01-31T21:54:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:42 smithi171 conmon[41853]: debug 2022-01-31T21:54:42.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:42.191142+0000) 2022-01-31T21:54:42.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:42 smithi167 conmon[54076]: debug 2022-01-31T21:54:42.475+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:42.476271+0000) 2022-01-31T21:54:42.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:42 smithi167 conmon[60316]: debug 2022-01-31T21:54:42.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:42.476271+0000) 2022-01-31T21:54:43.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:42 smithi171 conmon[46715]: debug 2022-01-31T21:54:42.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:42.986902+0000) 2022-01-31T21:54:43.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:42 smithi171 conmon[51620]: debug 2022-01-31T21:54:42.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:42.829516+0000) 2022-01-31T21:54:43.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:42 smithi167 conmon[49112]: debug 2022-01-31T21:54:42.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:42.811853+0000) 2022-01-31T21:54:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:43 smithi171 conmon[41853]: debug 2022-01-31T21:54:43.189+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:43.191354+0000) 2022-01-31T21:54:43.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:43 smithi167 conmon[54076]: debug 2022-01-31T21:54:43.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:43.476531+0000) 2022-01-31T21:54:43.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:43 smithi167 conmon[60316]: debug 2022-01-31T21:54:43.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:43.476522+0000) 2022-01-31T21:54:44.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:43 smithi171 conmon[46715]: debug 2022-01-31T21:54:43.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:43.987077+0000) 2022-01-31T21:54:44.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:43 smithi171 conmon[51620]: debug 2022-01-31T21:54:43.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:43.829716+0000) 2022-01-31T21:54:44.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:43 smithi167 conmon[49112]: debug 2022-01-31T21:54:43.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:43.812067+0000) 2022-01-31T21:54:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:44 smithi171 conmon[41853]: debug 2022-01-31T21:54:44.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:44.191550+0000) 2022-01-31T21:54:44.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:44 smithi167 conmon[54076]: debug 2022-01-31T21:54:44.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:44.476667+0000) 2022-01-31T21:54:44.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:44 smithi167 conmon[60316]: debug 2022-01-31T21:54:44.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:44.476696+0000) 2022-01-31T21:54:45.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:44 smithi171 conmon[51620]: debug 2022-01-31T21:54:44.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:44.829977+0000) 2022-01-31T21:54:45.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:44 smithi171 conmon[46715]: debug 2022-01-31T21:54:44.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:44.987236+0000) 2022-01-31T21:54:45.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:44 smithi167 conmon[49112]: debug 2022-01-31T21:54:44.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:44.812287+0000) 2022-01-31T21:54:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:45 smithi171 conmon[41853]: debug 2022-01-31T21:54:45.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:45.191780+0000) 2022-01-31T21:54:45.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:45 smithi167 conmon[54076]: debug 2022-01-31T21:54:45.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:45.476864+0000) 2022-01-31T21:54:45.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:45 smithi167 conmon[60316]: debug 2022-01-31T21:54:45.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:45.476870+0000) 2022-01-31T21:54:46.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:45 smithi171 conmon[46715]: debug 2022-01-31T21:54:45.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:45.987425+0000) 2022-01-31T21:54:46.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:45 smithi171 conmon[51620]: debug 2022-01-31T21:54:45.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:45.830168+0000) 2022-01-31T21:54:46.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:45 smithi167 conmon[49112]: debug 2022-01-31T21:54:45.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:45.812524+0000) 2022-01-31T21:54:46.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:46 smithi167 conmon[49112]: debug 2022-01-31T21:54:46.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.140053+0000) 2022-01-31T21:54:46.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:46 smithi167 conmon[54076]: debug 2022-01-31T21:54:46.140+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.141089+0000) 2022-01-31T21:54:46.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:46 smithi167 conmon[60316]: debug 2022-01-31T21:54:46.140+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.141068+0000) 2022-01-31T21:54:46.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[35325]: debug 2022-01-31T21:54:46.156+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167447 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:46.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[46715]: debug 2022-01-31T21:54:46.139+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.140776+0000) 2022-01-31T21:54:46.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[51620]: debug 2022-01-31T21:54:46.138+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.139946+0000) 2022-01-31T21:54:46.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[41853]: debug 2022-01-31T21:54:46.139+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.140913+0000) 2022-01-31T21:54:46.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[41853]: debug 2022-01-31T21:54:46.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.191903+0000) 2022-01-31T21:54:46.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:46 smithi167 conmon[54076]: debug 2022-01-31T21:54:46.476+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.477073+0000) 2022-01-31T21:54:46.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:46 smithi167 conmon[60316]: debug 2022-01-31T21:54:46.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.477086+0000) 2022-01-31T21:54:47.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[46715]: debug 2022-01-31T21:54:46.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.987587+0000) 2022-01-31T21:54:47.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:46 smithi171 conmon[51620]: debug 2022-01-31T21:54:46.828+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.830374+0000) 2022-01-31T21:54:47.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:46 smithi167 conmon[49112]: debug 2022-01-31T21:54:46.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:46.812723+0000) 2022-01-31T21:54:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:47 smithi171 conmon[41853]: debug 2022-01-31T21:54:47.191+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:47.192017+0000) 2022-01-31T21:54:47.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:47 smithi167 conmon[54076]: debug 2022-01-31T21:54:47.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:47.477204+0000) 2022-01-31T21:54:47.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:47 smithi167 conmon[60316]: debug 2022-01-31T21:54:47.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:47.477236+0000) 2022-01-31T21:54:48.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:47 smithi171 conmon[46715]: debug 2022-01-31T21:54:47.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:47.987712+0000) 2022-01-31T21:54:48.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:47 smithi171 conmon[51620]: debug 2022-01-31T21:54:47.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:47.830548+0000) 2022-01-31T21:54:48.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:47 smithi167 conmon[49112]: debug 2022-01-31T21:54:47.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:47.812851+0000) 2022-01-31T21:54:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:48 smithi171 conmon[41853]: debug 2022-01-31T21:54:48.191+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:48.192173+0000) 2022-01-31T21:54:48.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:48 smithi167 conmon[54076]: debug 2022-01-31T21:54:48.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:48.477396+0000) 2022-01-31T21:54:48.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:48 smithi167 conmon[60316]: debug 2022-01-31T21:54:48.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:48.477400+0000) 2022-01-31T21:54:49.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:48 smithi171 conmon[46715]: debug 2022-01-31T21:54:48.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:48.987875+0000) 2022-01-31T21:54:49.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:48 smithi171 conmon[51620]: debug 2022-01-31T21:54:48.829+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:48.830783+0000) 2022-01-31T21:54:49.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:48 smithi167 conmon[49112]: debug 2022-01-31T21:54:48.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:48.813067+0000) 2022-01-31T21:54:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:49 smithi171 conmon[41853]: debug 2022-01-31T21:54:49.190+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:49.192298+0000) 2022-01-31T21:54:49.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:49 smithi167 conmon[54076]: debug 2022-01-31T21:54:49.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:49.477525+0000) 2022-01-31T21:54:49.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:49 smithi167 conmon[60316]: debug 2022-01-31T21:54:49.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:49.477526+0000) 2022-01-31T21:54:50.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:49 smithi171 conmon[46715]: debug 2022-01-31T21:54:49.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:49.988087+0000) 2022-01-31T21:54:50.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:49 smithi171 conmon[51620]: debug 2022-01-31T21:54:49.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:49.831012+0000) 2022-01-31T21:54:50.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:49 smithi167 conmon[49112]: debug 2022-01-31T21:54:49.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:49.813296+0000) 2022-01-31T21:54:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:50 smithi171 conmon[41853]: debug 2022-01-31T21:54:50.191+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:50.192456+0000) 2022-01-31T21:54:50.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:50 smithi167 conmon[54076]: debug 2022-01-31T21:54:50.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:50.477699+0000) 2022-01-31T21:54:50.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:50 smithi167 conmon[60316]: debug 2022-01-31T21:54:50.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:50.477696+0000) 2022-01-31T21:54:51.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:50 smithi171 conmon[51620]: debug 2022-01-31T21:54:50.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:50.831224+0000) 2022-01-31T21:54:51.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:50 smithi171 conmon[46715]: debug 2022-01-31T21:54:50.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:50.988272+0000) 2022-01-31T21:54:51.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:50 smithi167 conmon[49112]: debug 2022-01-31T21:54:50.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:50.813533+0000) 2022-01-31T21:54:51.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:51 smithi167 conmon[49112]: debug 2022-01-31T21:54:51.159+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.159903+0000) 2022-01-31T21:54:51.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:51 smithi167 conmon[54076]: debug 2022-01-31T21:54:51.160+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.160998+0000) 2022-01-31T21:54:51.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:51 smithi167 conmon[60316]: debug 2022-01-31T21:54:51.160+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.160804+0000) 2022-01-31T21:54:51.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[35325]: debug 2022-01-31T21:54:51.176+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167557 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:51.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[46715]: debug 2022-01-31T21:54:51.158+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.160534+0000) 2022-01-31T21:54:51.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[51620]: debug 2022-01-31T21:54:51.158+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.160186+0000) 2022-01-31T21:54:51.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[41853]: debug 2022-01-31T21:54:51.160+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.162079+0000) 2022-01-31T21:54:51.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[41853]: debug 2022-01-31T21:54:51.191+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.192657+0000) 2022-01-31T21:54:51.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:51 smithi167 conmon[54076]: debug 2022-01-31T21:54:51.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.477886+0000) 2022-01-31T21:54:51.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:51 smithi167 conmon[60316]: debug 2022-01-31T21:54:51.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.477883+0000) 2022-01-31T21:54:52.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[46715]: debug 2022-01-31T21:54:51.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.988402+0000) 2022-01-31T21:54:52.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:51 smithi171 conmon[51620]: debug 2022-01-31T21:54:51.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.831453+0000) 2022-01-31T21:54:52.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:51 smithi167 conmon[49112]: debug 2022-01-31T21:54:51.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:51.813715+0000) 2022-01-31T21:54:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:52 smithi171 conmon[41853]: debug 2022-01-31T21:54:52.191+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:52.192809+0000) 2022-01-31T21:54:52.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:52 smithi167 conmon[54076]: debug 2022-01-31T21:54:52.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:52.478049+0000) 2022-01-31T21:54:52.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:52 smithi167 conmon[60316]: debug 2022-01-31T21:54:52.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:52.478058+0000) 2022-01-31T21:54:53.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:52 smithi171 conmon[46715]: debug 2022-01-31T21:54:52.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:52.988554+0000) 2022-01-31T21:54:53.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:52 smithi171 conmon[51620]: debug 2022-01-31T21:54:52.830+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:52.831615+0000) 2022-01-31T21:54:53.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:52 smithi167 conmon[49112]: debug 2022-01-31T21:54:52.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:52.813805+0000) 2022-01-31T21:54:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:53 smithi171 conmon[41853]: debug 2022-01-31T21:54:53.192+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:53.192970+0000) 2022-01-31T21:54:53.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:53 smithi167 conmon[54076]: debug 2022-01-31T21:54:53.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:53.478267+0000) 2022-01-31T21:54:53.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:53 smithi167 conmon[60316]: debug 2022-01-31T21:54:53.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:53.478258+0000) 2022-01-31T21:54:53.910 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:53 smithi171 conmon[35325]: debug 2022-01-31T21:54:53.759+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:54:53.911 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:53 smithi171 conmon[51620]: debug 2022-01-31T21:54:53.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:53.831779+0000) 2022-01-31T21:54:54.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:53 smithi167 conmon[49112]: debug 2022-01-31T21:54:53.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:53.813992+0000) 2022-01-31T21:54:54.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:53 smithi171 conmon[46715]: debug 2022-01-31T21:54:53.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:53.988801+0000) 2022-01-31T21:54:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:54 smithi171 conmon[41853]: debug 2022-01-31T21:54:54.192+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:54.193135+0000) 2022-01-31T21:54:54.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:54 smithi167 conmon[54076]: debug 2022-01-31T21:54:54.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:54.478470+0000) 2022-01-31T21:54:54.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:54 smithi167 conmon[60316]: debug 2022-01-31T21:54:54.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:54.478468+0000) 2022-01-31T21:54:55.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:54 smithi171 conmon[46715]: debug 2022-01-31T21:54:54.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:54.988991+0000) 2022-01-31T21:54:55.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:54 smithi171 conmon[51620]: debug 2022-01-31T21:54:54.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:54.831953+0000) 2022-01-31T21:54:55.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:54 smithi167 conmon[49112]: debug 2022-01-31T21:54:54.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:54.814187+0000) 2022-01-31T21:54:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:55 smithi171 conmon[41853]: debug 2022-01-31T21:54:55.192+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:55.193315+0000) 2022-01-31T21:54:55.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:55 smithi167 conmon[54076]: debug 2022-01-31T21:54:55.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:55.478706+0000) 2022-01-31T21:54:55.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:55 smithi167 conmon[60316]: debug 2022-01-31T21:54:55.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:55.478706+0000) 2022-01-31T21:54:56.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:55 smithi171 conmon[46715]: debug 2022-01-31T21:54:55.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:55.989147+0000) 2022-01-31T21:54:56.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:55 smithi171 conmon[51620]: debug 2022-01-31T21:54:55.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:55.832165+0000) 2022-01-31T21:54:56.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:55 smithi167 conmon[49112]: debug 2022-01-31T21:54:55.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:55.814414+0000) 2022-01-31T21:54:56.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:56 smithi167 conmon[49112]: debug 2022-01-31T21:54:56.179+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.181197+0000) 2022-01-31T21:54:56.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:56 smithi167 conmon[54076]: debug 2022-01-31T21:54:56.180+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.181857+0000) 2022-01-31T21:54:56.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:56 smithi167 conmon[60316]: debug 2022-01-31T21:54:56.179+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.181001+0000) 2022-01-31T21:54:56.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[35325]: debug 2022-01-31T21:54:56.197+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167669 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:54:56.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[46715]: debug 2022-01-31T21:54:56.179+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.179789+0000) 2022-01-31T21:54:56.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[51620]: debug 2022-01-31T21:54:56.179+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.179809+0000) 2022-01-31T21:54:56.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[41853]: debug 2022-01-31T21:54:56.180+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.180913+0000) 2022-01-31T21:54:56.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[41853]: debug 2022-01-31T21:54:56.192+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.193462+0000) 2022-01-31T21:54:56.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:56 smithi167 conmon[54076]: debug 2022-01-31T21:54:56.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.478891+0000) 2022-01-31T21:54:56.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:56 smithi167 conmon[60316]: debug 2022-01-31T21:54:56.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.478868+0000) 2022-01-31T21:54:57.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[46715]: debug 2022-01-31T21:54:56.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.989319+0000) 2022-01-31T21:54:57.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:56 smithi171 conmon[51620]: debug 2022-01-31T21:54:56.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.832395+0000) 2022-01-31T21:54:57.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:56 smithi167 conmon[49112]: debug 2022-01-31T21:54:56.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:56.814627+0000) 2022-01-31T21:54:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:57 smithi171 conmon[41853]: debug 2022-01-31T21:54:57.192+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:57.193635+0000) 2022-01-31T21:54:57.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:57 smithi167 conmon[54076]: debug 2022-01-31T21:54:57.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:57.479033+0000) 2022-01-31T21:54:57.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:57 smithi167 conmon[60316]: debug 2022-01-31T21:54:57.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:57.479038+0000) 2022-01-31T21:54:58.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:57 smithi171 conmon[46715]: debug 2022-01-31T21:54:57.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:57.989468+0000) 2022-01-31T21:54:58.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:57 smithi171 conmon[51620]: debug 2022-01-31T21:54:57.831+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:57.832564+0000) 2022-01-31T21:54:58.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:57 smithi167 conmon[49112]: debug 2022-01-31T21:54:57.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:57.814768+0000) 2022-01-31T21:54:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:58 smithi171 conmon[41853]: debug 2022-01-31T21:54:58.193+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:58.193828+0000) 2022-01-31T21:54:58.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:58 smithi167 conmon[54076]: debug 2022-01-31T21:54:58.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:58.479264+0000) 2022-01-31T21:54:58.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:58 smithi167 conmon[60316]: debug 2022-01-31T21:54:58.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:58.479264+0000) 2022-01-31T21:54:59.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:58 smithi171 conmon[46715]: debug 2022-01-31T21:54:58.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:58.989615+0000) 2022-01-31T21:54:59.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:58 smithi171 conmon[51620]: debug 2022-01-31T21:54:58.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:58.832797+0000) 2022-01-31T21:54:59.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:58 smithi167 conmon[49112]: debug 2022-01-31T21:54:58.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:58.814950+0000) 2022-01-31T21:54:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:54:59 smithi171 conmon[41853]: debug 2022-01-31T21:54:59.193+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:59.193995+0000) 2022-01-31T21:54:59.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:54:59 smithi167 conmon[54076]: debug 2022-01-31T21:54:59.477+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:59.479449+0000) 2022-01-31T21:54:59.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:54:59 smithi167 conmon[60316]: debug 2022-01-31T21:54:59.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:59.479415+0000) 2022-01-31T21:55:00.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:54:59 smithi171 conmon[51620]: debug 2022-01-31T21:54:59.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:59.832988+0000) 2022-01-31T21:55:00.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:54:59 smithi171 conmon[46715]: debug 2022-01-31T21:54:59.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:59.989726+0000) 2022-01-31T21:55:00.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:54:59 smithi167 conmon[49112]: debug 2022-01-31T21:54:59.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:54:59.815132+0000) 2022-01-31T21:55:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:00 smithi171 conmon[41853]: debug 2022-01-31T21:55:00.193+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:00.194156+0000) 2022-01-31T21:55:00.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:00 smithi167 conmon[54076]: debug 2022-01-31T21:55:00.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:00.479651+0000) 2022-01-31T21:55:00.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:00 smithi167 conmon[60316]: debug 2022-01-31T21:55:00.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:00.479607+0000) 2022-01-31T21:55:01.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:00 smithi171 conmon[46715]: debug 2022-01-31T21:55:00.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:00.989899+0000) 2022-01-31T21:55:01.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:00 smithi171 conmon[51620]: debug 2022-01-31T21:55:00.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:00.833197+0000) 2022-01-31T21:55:01.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:00 smithi167 conmon[49112]: debug 2022-01-31T21:55:00.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:00.815370+0000) 2022-01-31T21:55:01.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:01 smithi167 conmon[49112]: debug 2022-01-31T21:55:01.200+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.201905+0000) 2022-01-31T21:55:01.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:01 smithi167 conmon[54076]: debug 2022-01-31T21:55:01.199+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.200924+0000) 2022-01-31T21:55:01.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:01 smithi167 conmon[60316]: debug 2022-01-31T21:55:01.198+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.200280+0000) 2022-01-31T21:55:01.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[35325]: debug 2022-01-31T21:55:01.217+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167774 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:01.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[46715]: debug 2022-01-31T21:55:01.200+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.200881+0000) 2022-01-31T21:55:01.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[51620]: debug 2022-01-31T21:55:01.198+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.199344+0000) 2022-01-31T21:55:01.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[41853]: debug 2022-01-31T21:55:01.193+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.194320+0000) 2022-01-31T21:55:01.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[41853]: debug 2022-01-31T21:55:01.199+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.200566+0000) 2022-01-31T21:55:01.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:01 smithi167 conmon[60316]: debug 2022-01-31T21:55:01.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.479748+0000) 2022-01-31T21:55:01.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:01 smithi167 conmon[54076]: debug 2022-01-31T21:55:01.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.479759+0000) 2022-01-31T21:55:02.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[46715]: debug 2022-01-31T21:55:01.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.990075+0000) 2022-01-31T21:55:02.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:01 smithi171 conmon[51620]: debug 2022-01-31T21:55:01.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.833389+0000) 2022-01-31T21:55:02.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:01 smithi167 conmon[49112]: debug 2022-01-31T21:55:01.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:01.815586+0000) 2022-01-31T21:55:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:02 smithi171 conmon[41853]: debug 2022-01-31T21:55:02.193+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:02.194446+0000) 2022-01-31T21:55:02.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:02 smithi167 conmon[54076]: debug 2022-01-31T21:55:02.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:02.479916+0000) 2022-01-31T21:55:02.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:02 smithi167 conmon[60316]: debug 2022-01-31T21:55:02.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:02.479909+0000) 2022-01-31T21:55:03.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:02 smithi171 conmon[46715]: debug 2022-01-31T21:55:02.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:02.990217+0000) 2022-01-31T21:55:03.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:02 smithi171 conmon[51620]: debug 2022-01-31T21:55:02.832+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:02.833555+0000) 2022-01-31T21:55:03.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:02 smithi167 conmon[49112]: debug 2022-01-31T21:55:02.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:02.815759+0000) 2022-01-31T21:55:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:03 smithi171 conmon[41853]: debug 2022-01-31T21:55:03.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:03.194620+0000) 2022-01-31T21:55:03.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:03 smithi167 conmon[54076]: debug 2022-01-31T21:55:03.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:03.480142+0000) 2022-01-31T21:55:03.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:03 smithi167 conmon[60316]: debug 2022-01-31T21:55:03.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:03.480140+0000) 2022-01-31T21:55:04.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:03 smithi171 conmon[46715]: debug 2022-01-31T21:55:03.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:03.990389+0000) 2022-01-31T21:55:04.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:03 smithi171 conmon[51620]: debug 2022-01-31T21:55:03.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:03.833734+0000) 2022-01-31T21:55:04.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:03 smithi167 conmon[49112]: debug 2022-01-31T21:55:03.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:03.815969+0000) 2022-01-31T21:55:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:04 smithi171 conmon[41853]: debug 2022-01-31T21:55:04.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:04.194808+0000) 2022-01-31T21:55:04.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:04 smithi167 conmon[54076]: debug 2022-01-31T21:55:04.478+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:04.480347+0000) 2022-01-31T21:55:04.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:04 smithi167 conmon[60316]: debug 2022-01-31T21:55:04.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:04.480344+0000) 2022-01-31T21:55:05.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:04 smithi171 conmon[46715]: debug 2022-01-31T21:55:04.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:04.990573+0000) 2022-01-31T21:55:05.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:04 smithi171 conmon[51620]: debug 2022-01-31T21:55:04.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:04.833933+0000) 2022-01-31T21:55:05.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:04 smithi167 conmon[49112]: debug 2022-01-31T21:55:04.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:04.816183+0000) 2022-01-31T21:55:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:05 smithi171 conmon[41853]: debug 2022-01-31T21:55:05.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:05.194986+0000) 2022-01-31T21:55:05.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:05 smithi167 conmon[54076]: debug 2022-01-31T21:55:05.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:05.480559+0000) 2022-01-31T21:55:05.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:05 smithi167 conmon[60316]: debug 2022-01-31T21:55:05.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:05.480556+0000) 2022-01-31T21:55:06.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:05 smithi171 conmon[46715]: debug 2022-01-31T21:55:05.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:05.990771+0000) 2022-01-31T21:55:06.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:05 smithi171 conmon[51620]: debug 2022-01-31T21:55:05.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:05.834132+0000) 2022-01-31T21:55:06.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:05 smithi167 conmon[49112]: debug 2022-01-31T21:55:05.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:05.816423+0000) 2022-01-31T21:55:06.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:06 smithi167 conmon[49112]: debug 2022-01-31T21:55:06.220+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.221816+0000) 2022-01-31T21:55:06.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:06 smithi167 conmon[54076]: debug 2022-01-31T21:55:06.220+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.221506+0000) 2022-01-31T21:55:06.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:06 smithi167 conmon[60316]: debug 2022-01-31T21:55:06.218+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.220271+0000) 2022-01-31T21:55:06.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[35325]: debug 2022-01-31T21:55:06.237+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:06.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[46715]: debug 2022-01-31T21:55:06.221+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.221788+0000) 2022-01-31T21:55:06.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[51620]: debug 2022-01-31T21:55:06.220+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.220834+0000) 2022-01-31T21:55:06.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[41853]: debug 2022-01-31T21:55:06.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.195177+0000) 2022-01-31T21:55:06.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[41853]: debug 2022-01-31T21:55:06.220+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.220783+0000) 2022-01-31T21:55:06.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:06 smithi167 conmon[60316]: debug 2022-01-31T21:55:06.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.480685+0000) 2022-01-31T21:55:06.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:06 smithi167 conmon[54076]: debug 2022-01-31T21:55:06.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.480733+0000) 2022-01-31T21:55:07.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[46715]: debug 2022-01-31T21:55:06.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.990948+0000) 2022-01-31T21:55:07.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:06 smithi171 conmon[51620]: debug 2022-01-31T21:55:06.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.834307+0000) 2022-01-31T21:55:07.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:06 smithi167 conmon[49112]: debug 2022-01-31T21:55:06.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:06.816647+0000) 2022-01-31T21:55:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:07 smithi171 conmon[41853]: debug 2022-01-31T21:55:07.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:07.195359+0000) 2022-01-31T21:55:07.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:07 smithi167 conmon[54076]: debug 2022-01-31T21:55:07.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:07.480886+0000) 2022-01-31T21:55:07.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:07 smithi167 conmon[60316]: debug 2022-01-31T21:55:07.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:07.480881+0000) 2022-01-31T21:55:08.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:07 smithi171 conmon[46715]: debug 2022-01-31T21:55:07.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:07.991080+0000) 2022-01-31T21:55:08.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:07 smithi171 conmon[51620]: debug 2022-01-31T21:55:07.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:07.834446+0000) 2022-01-31T21:55:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:07 smithi167 conmon[49112]: debug 2022-01-31T21:55:07.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:07.816778+0000) 2022-01-31T21:55:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:08 smithi171 conmon[41853]: debug 2022-01-31T21:55:08.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:08.195505+0000) 2022-01-31T21:55:08.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:08 smithi167 conmon[54076]: debug 2022-01-31T21:55:08.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:08.481074+0000) 2022-01-31T21:55:08.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:08 smithi167 conmon[60316]: debug 2022-01-31T21:55:08.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:08.481073+0000) 2022-01-31T21:55:09.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:08 smithi171 conmon[51620]: debug 2022-01-31T21:55:08.833+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:08.834586+0000) 2022-01-31T21:55:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:08 smithi171 conmon[35325]: debug 2022-01-31T21:55:08.760+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:55:09.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:08 smithi171 conmon[46715]: debug 2022-01-31T21:55:08.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:08.991252+0000) 2022-01-31T21:55:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:08 smithi167 conmon[49112]: debug 2022-01-31T21:55:08.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:08.816981+0000) 2022-01-31T21:55:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:09 smithi171 conmon[41853]: debug 2022-01-31T21:55:09.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:09.195675+0000) 2022-01-31T21:55:09.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:09 smithi167 conmon[60316]: debug 2022-01-31T21:55:09.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:09.481201+0000) 2022-01-31T21:55:09.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:09 smithi167 conmon[54076]: debug 2022-01-31T21:55:09.479+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:09.481241+0000) 2022-01-31T21:55:10.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:09 smithi171 conmon[46715]: debug 2022-01-31T21:55:09.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:09.991412+0000) 2022-01-31T21:55:10.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:09 smithi171 conmon[51620]: debug 2022-01-31T21:55:09.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:09.834748+0000) 2022-01-31T21:55:10.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:09 smithi167 conmon[49112]: debug 2022-01-31T21:55:09.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:09.817209+0000) 2022-01-31T21:55:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:10 smithi171 conmon[41853]: debug 2022-01-31T21:55:10.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:10.195850+0000) 2022-01-31T21:55:10.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:10 smithi167 conmon[54076]: debug 2022-01-31T21:55:10.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:10.481417+0000) 2022-01-31T21:55:10.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:10 smithi167 conmon[60316]: debug 2022-01-31T21:55:10.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:10.481374+0000) 2022-01-31T21:55:11.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:10 smithi171 conmon[46715]: debug 2022-01-31T21:55:10.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:10.991587+0000) 2022-01-31T21:55:11.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:10 smithi171 conmon[51620]: debug 2022-01-31T21:55:10.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:10.834952+0000) 2022-01-31T21:55:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:10 smithi167 conmon[49112]: debug 2022-01-31T21:55:10.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:10.817413+0000) 2022-01-31T21:55:11.462 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:11 smithi167 conmon[49112]: debug 2022-01-31T21:55:11.239+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.240924+0000) 2022-01-31T21:55:11.463 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:11 smithi167 conmon[54076]: debug 2022-01-31T21:55:11.238+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.240321+0000) 2022-01-31T21:55:11.463 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:11 smithi167 conmon[60316]: debug 2022-01-31T21:55:11.237+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.239315+0000) 2022-01-31T21:55:11.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[51620]: debug 2022-01-31T21:55:11.239+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.240073+0000) 2022-01-31T21:55:11.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[35325]: debug 2022-01-31T21:55:11.257+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 167995 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:11.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[41853]: debug 2022-01-31T21:55:11.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.196038+0000) 2022-01-31T21:55:11.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[41853]: debug 2022-01-31T21:55:11.239+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.239892+0000) 2022-01-31T21:55:11.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[46715]: debug 2022-01-31T21:55:11.240+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.241465+0000) 2022-01-31T21:55:11.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:11 smithi167 conmon[60316]: debug 2022-01-31T21:55:11.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.481581+0000) 2022-01-31T21:55:11.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:11 smithi167 conmon[54076]: debug 2022-01-31T21:55:11.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.481587+0000) 2022-01-31T21:55:12.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[46715]: debug 2022-01-31T21:55:11.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.991773+0000) 2022-01-31T21:55:12.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:11 smithi171 conmon[51620]: debug 2022-01-31T21:55:11.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.835074+0000) 2022-01-31T21:55:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:11 smithi167 conmon[49112]: debug 2022-01-31T21:55:11.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:11.817579+0000) 2022-01-31T21:55:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:12 smithi171 conmon[41853]: debug 2022-01-31T21:55:12.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:12.196222+0000) 2022-01-31T21:55:12.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:12 smithi167 conmon[60316]: debug 2022-01-31T21:55:12.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:12.481739+0000) 2022-01-31T21:55:12.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:12 smithi167 conmon[54076]: debug 2022-01-31T21:55:12.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:12.481714+0000) 2022-01-31T21:55:13.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:12 smithi171 conmon[46715]: debug 2022-01-31T21:55:12.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:12.991916+0000) 2022-01-31T21:55:13.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:12 smithi171 conmon[51620]: debug 2022-01-31T21:55:12.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:12.835247+0000) 2022-01-31T21:55:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:12 smithi167 conmon[49112]: debug 2022-01-31T21:55:12.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:12.817760+0000) 2022-01-31T21:55:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:13 smithi171 conmon[41853]: debug 2022-01-31T21:55:13.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:13.196434+0000) 2022-01-31T21:55:13.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:13 smithi167 conmon[54076]: debug 2022-01-31T21:55:13.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:13.481866+0000) 2022-01-31T21:55:13.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:13 smithi167 conmon[60316]: debug 2022-01-31T21:55:13.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:13.481911+0000) 2022-01-31T21:55:14.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:13 smithi171 conmon[46715]: debug 2022-01-31T21:55:13.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:13.992079+0000) 2022-01-31T21:55:14.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:13 smithi171 conmon[51620]: debug 2022-01-31T21:55:13.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:13.835485+0000) 2022-01-31T21:55:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:13 smithi167 conmon[49112]: debug 2022-01-31T21:55:13.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:13.817955+0000) 2022-01-31T21:55:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:14 smithi171 conmon[41853]: debug 2022-01-31T21:55:14.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:14.196587+0000) 2022-01-31T21:55:14.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:14 smithi167 conmon[60316]: debug 2022-01-31T21:55:14.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:14.482050+0000) 2022-01-31T21:55:14.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:14 smithi167 conmon[54076]: debug 2022-01-31T21:55:14.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:14.482004+0000) 2022-01-31T21:55:15.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:14 smithi171 conmon[46715]: debug 2022-01-31T21:55:14.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:14.992261+0000) 2022-01-31T21:55:15.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:14 smithi171 conmon[51620]: debug 2022-01-31T21:55:14.834+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:14.835698+0000) 2022-01-31T21:55:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:14 smithi167 conmon[49112]: debug 2022-01-31T21:55:14.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:14.818166+0000) 2022-01-31T21:55:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:15 smithi171 conmon[41853]: debug 2022-01-31T21:55:15.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:15.196713+0000) 2022-01-31T21:55:15.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:15 smithi167 conmon[60316]: debug 2022-01-31T21:55:15.481+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:15.482238+0000) 2022-01-31T21:55:15.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:15 smithi167 conmon[54076]: debug 2022-01-31T21:55:15.481+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:15.482166+0000) 2022-01-31T21:55:16.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:15 smithi171 conmon[46715]: debug 2022-01-31T21:55:15.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:15.992438+0000) 2022-01-31T21:55:16.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:15 smithi171 conmon[51620]: debug 2022-01-31T21:55:15.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:15.835918+0000) 2022-01-31T21:55:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:15 smithi167 conmon[49112]: debug 2022-01-31T21:55:15.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:15.818380+0000) 2022-01-31T21:55:16.463 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:16 smithi167 conmon[49112]: debug 2022-01-31T21:55:16.259+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.260822+0000) 2022-01-31T21:55:16.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:16 smithi167 conmon[54076]: debug 2022-01-31T21:55:16.259+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.260988+0000) 2022-01-31T21:55:16.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:16 smithi167 conmon[60316]: debug 2022-01-31T21:55:16.258+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.259859+0000) 2022-01-31T21:55:16.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[35325]: debug 2022-01-31T21:55:16.277+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168108 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:16.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[41853]: debug 2022-01-31T21:55:16.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.196797+0000) 2022-01-31T21:55:16.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[41853]: debug 2022-01-31T21:55:16.259+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.259891+0000) 2022-01-31T21:55:16.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[46715]: debug 2022-01-31T21:55:16.259+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.260076+0000) 2022-01-31T21:55:16.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[51620]: debug 2022-01-31T21:55:16.259+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.260380+0000) 2022-01-31T21:55:16.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:16 smithi167 conmon[54076]: debug 2022-01-31T21:55:16.481+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.482359+0000) 2022-01-31T21:55:16.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:16 smithi167 conmon[60316]: debug 2022-01-31T21:55:16.481+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.482437+0000) 2022-01-31T21:55:17.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[46715]: debug 2022-01-31T21:55:16.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.992607+0000) 2022-01-31T21:55:17.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:16 smithi171 conmon[51620]: debug 2022-01-31T21:55:16.835+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.836129+0000) 2022-01-31T21:55:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:16 smithi167 conmon[49112]: debug 2022-01-31T21:55:16.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:16.818616+0000) 2022-01-31T21:55:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:17 smithi171 conmon[41853]: debug 2022-01-31T21:55:17.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:17.196946+0000) 2022-01-31T21:55:17.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:17 smithi167 conmon[60316]: debug 2022-01-31T21:55:17.481+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:17.482549+0000) 2022-01-31T21:55:17.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:17 smithi167 conmon[54076]: debug 2022-01-31T21:55:17.480+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:17.482457+0000) 2022-01-31T21:55:18.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:17 smithi171 conmon[46715]: debug 2022-01-31T21:55:17.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:17.992777+0000) 2022-01-31T21:55:18.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:17 smithi171 conmon[51620]: debug 2022-01-31T21:55:17.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:17.836318+0000) 2022-01-31T21:55:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:17 smithi167 conmon[49112]: debug 2022-01-31T21:55:17.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:17.818796+0000) 2022-01-31T21:55:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:18 smithi171 conmon[41853]: debug 2022-01-31T21:55:18.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:18.197045+0000) 2022-01-31T21:55:18.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:18 smithi167 conmon[54076]: debug 2022-01-31T21:55:18.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:18.482644+0000) 2022-01-31T21:55:18.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:18 smithi167 conmon[60316]: debug 2022-01-31T21:55:18.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:18.482714+0000) 2022-01-31T21:55:19.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:18 smithi171 conmon[51620]: debug 2022-01-31T21:55:18.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:18.836524+0000) 2022-01-31T21:55:19.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:18 smithi171 conmon[46715]: debug 2022-01-31T21:55:18.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:18.992965+0000) 2022-01-31T21:55:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:18 smithi167 conmon[49112]: debug 2022-01-31T21:55:18.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:18.818996+0000) 2022-01-31T21:55:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:19 smithi171 conmon[41853]: debug 2022-01-31T21:55:19.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:19.197224+0000) 2022-01-31T21:55:19.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:19 smithi167 conmon[54076]: debug 2022-01-31T21:55:19.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:19.482852+0000) 2022-01-31T21:55:19.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:19 smithi167 conmon[60316]: debug 2022-01-31T21:55:19.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:19.482930+0000) 2022-01-31T21:55:20.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:19 smithi171 conmon[46715]: debug 2022-01-31T21:55:19.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:19.993113+0000) 2022-01-31T21:55:20.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:19 smithi171 conmon[51620]: debug 2022-01-31T21:55:19.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:19.836774+0000) 2022-01-31T21:55:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:19 smithi167 conmon[49112]: debug 2022-01-31T21:55:19.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:19.819186+0000) 2022-01-31T21:55:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:20 smithi171 conmon[41853]: debug 2022-01-31T21:55:20.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:20.197430+0000) 2022-01-31T21:55:20.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:20 smithi167 conmon[54076]: debug 2022-01-31T21:55:20.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:20.483100+0000) 2022-01-31T21:55:20.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:20 smithi167 conmon[60316]: debug 2022-01-31T21:55:20.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:20.483100+0000) 2022-01-31T21:55:21.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:20 smithi171 conmon[46715]: debug 2022-01-31T21:55:20.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:20.993270+0000) 2022-01-31T21:55:21.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:20 smithi171 conmon[51620]: debug 2022-01-31T21:55:20.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:20.836926+0000) 2022-01-31T21:55:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:20 smithi167 conmon[49112]: debug 2022-01-31T21:55:20.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:20.819425+0000) 2022-01-31T21:55:21.464 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:21 smithi167 conmon[49112]: debug 2022-01-31T21:55:21.279+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.280180+0000) 2022-01-31T21:55:21.464 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:21 smithi167 conmon[54076]: debug 2022-01-31T21:55:21.279+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.280366+0000) 2022-01-31T21:55:21.465 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:21 smithi167 conmon[60316]: debug 2022-01-31T21:55:21.281+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.282282+0000) 2022-01-31T21:55:21.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[35325]: debug 2022-01-31T21:55:21.297+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168217 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:21.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[46715]: debug 2022-01-31T21:55:21.280+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.280922+0000) 2022-01-31T21:55:21.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[51620]: debug 2022-01-31T21:55:21.280+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.281514+0000) 2022-01-31T21:55:21.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[41853]: debug 2022-01-31T21:55:21.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.197591+0000) 2022-01-31T21:55:21.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[41853]: debug 2022-01-31T21:55:21.281+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.282364+0000) 2022-01-31T21:55:21.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:21 smithi167 conmon[54076]: debug 2022-01-31T21:55:21.482+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.483302+0000) 2022-01-31T21:55:21.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:21 smithi167 conmon[60316]: debug 2022-01-31T21:55:21.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.483310+0000) 2022-01-31T21:55:22.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[51620]: debug 2022-01-31T21:55:21.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.837110+0000) 2022-01-31T21:55:22.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:21 smithi171 conmon[46715]: debug 2022-01-31T21:55:21.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.993451+0000) 2022-01-31T21:55:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:21 smithi167 conmon[49112]: debug 2022-01-31T21:55:21.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:21.819554+0000) 2022-01-31T21:55:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:22 smithi171 conmon[41853]: debug 2022-01-31T21:55:22.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:22.197746+0000) 2022-01-31T21:55:22.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:22 smithi167 conmon[54076]: debug 2022-01-31T21:55:22.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:22.483483+0000) 2022-01-31T21:55:22.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:22 smithi167 conmon[60316]: debug 2022-01-31T21:55:22.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:22.483484+0000) 2022-01-31T21:55:23.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:22 smithi171 conmon[46715]: debug 2022-01-31T21:55:22.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:22.993623+0000) 2022-01-31T21:55:23.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:22 smithi171 conmon[51620]: debug 2022-01-31T21:55:22.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:22.837309+0000) 2022-01-31T21:55:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:22 smithi167 conmon[49112]: debug 2022-01-31T21:55:22.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:22.819698+0000) 2022-01-31T21:55:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:23 smithi171 conmon[41853]: debug 2022-01-31T21:55:23.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:23.197884+0000) 2022-01-31T21:55:23.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:23 smithi167 conmon[54076]: debug 2022-01-31T21:55:23.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:23.483605+0000) 2022-01-31T21:55:23.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:23 smithi167 conmon[60316]: debug 2022-01-31T21:55:23.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:23.483608+0000) 2022-01-31T21:55:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:23 smithi171 conmon[35325]: debug 2022-01-31T21:55:23.761+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:55:24.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:23 smithi171 conmon[46715]: debug 2022-01-31T21:55:23.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:23.993819+0000) 2022-01-31T21:55:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:23 smithi171 conmon[51620]: debug 2022-01-31T21:55:23.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:23.837519+0000) 2022-01-31T21:55:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:23 smithi167 conmon[49112]: debug 2022-01-31T21:55:23.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:23.819928+0000) 2022-01-31T21:55:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:24 smithi171 conmon[41853]: debug 2022-01-31T21:55:24.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:24.198075+0000) 2022-01-31T21:55:24.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:24 smithi167 conmon[54076]: debug 2022-01-31T21:55:24.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:24.483812+0000) 2022-01-31T21:55:24.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:24 smithi167 conmon[60316]: debug 2022-01-31T21:55:24.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:24.483759+0000) 2022-01-31T21:55:25.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:24 smithi171 conmon[46715]: debug 2022-01-31T21:55:24.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:24.994004+0000) 2022-01-31T21:55:25.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:24 smithi171 conmon[51620]: debug 2022-01-31T21:55:24.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:24.837726+0000) 2022-01-31T21:55:25.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:24 smithi167 conmon[49112]: debug 2022-01-31T21:55:24.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:24.820157+0000) 2022-01-31T21:55:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:25 smithi171 conmon[41853]: debug 2022-01-31T21:55:25.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:25.198229+0000) 2022-01-31T21:55:25.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:25 smithi167 conmon[54076]: debug 2022-01-31T21:55:25.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:25.484062+0000) 2022-01-31T21:55:25.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:25 smithi167 conmon[60316]: debug 2022-01-31T21:55:25.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:25.483955+0000) 2022-01-31T21:55:26.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:25 smithi171 conmon[51620]: debug 2022-01-31T21:55:25.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:25.837925+0000) 2022-01-31T21:55:26.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:25 smithi171 conmon[46715]: debug 2022-01-31T21:55:25.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:25.994186+0000) 2022-01-31T21:55:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:25 smithi167 conmon[49112]: debug 2022-01-31T21:55:25.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:25.820372+0000) 2022-01-31T21:55:26.465 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[49112]: debug 2022-01-31T21:55:26.299+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.300132+0000) 2022-01-31T21:55:26.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[54076]: debug 2022-01-31T21:55:26.300+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.300764+0000) 2022-01-31T21:55:26.466 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[60316]: debug 2022-01-31T21:55:26.467 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[60316]: 2022-01-31T21:55:26.307+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.307563+0000) 2022-01-31T21:55:26.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[35325]: debug 2022-01-31T21:55:26.322+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168332 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:26.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[46715]: debug 2022-01-31T21:55:26.300+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.302029+0000) 2022-01-31T21:55:26.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[51620]: debug 2022-01-31T21:55:26.299+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.300933+0000) 2022-01-31T21:55:26.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[41853]: debug 2022-01-31T21:55:26.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.198411+0000) 2022-01-31T21:55:26.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[41853]: debug 2022-01-31T21:55:26.299+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.301281+0000) 2022-01-31T21:55:26.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[54076]: debug 2022-01-31T21:55:26.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.484242+0000) 2022-01-31T21:55:26.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[60316]: debug 2022-01-31T21:55:26.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.484140+0000) 2022-01-31T21:55:27.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[51620]: debug 2022-01-31T21:55:26.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.838092+0000) 2022-01-31T21:55:27.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:26 smithi171 conmon[46715]: debug 2022-01-31T21:55:26.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.994356+0000) 2022-01-31T21:55:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:26 smithi167 conmon[49112]: debug 2022-01-31T21:55:26.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:26.820605+0000) 2022-01-31T21:55:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:27 smithi171 conmon[41853]: debug 2022-01-31T21:55:27.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:27.198590+0000) 2022-01-31T21:55:27.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:27 smithi167 conmon[54076]: debug 2022-01-31T21:55:27.483+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:27.484388+0000) 2022-01-31T21:55:27.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:27 smithi167 conmon[60316]: debug 2022-01-31T21:55:27.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:27.484335+0000) 2022-01-31T21:55:28.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:27 smithi171 conmon[51620]: debug 2022-01-31T21:55:27.836+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:27.838262+0000) 2022-01-31T21:55:28.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:27 smithi171 conmon[46715]: debug 2022-01-31T21:55:27.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:27.994511+0000) 2022-01-31T21:55:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:27 smithi167 conmon[49112]: debug 2022-01-31T21:55:27.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:27.820751+0000) 2022-01-31T21:55:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:28 smithi171 conmon[41853]: debug 2022-01-31T21:55:28.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:28.198742+0000) 2022-01-31T21:55:28.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:28 smithi167 conmon[54076]: debug 2022-01-31T21:55:28.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:28.484587+0000) 2022-01-31T21:55:28.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:28 smithi167 conmon[60316]: debug 2022-01-31T21:55:28.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:28.484582+0000) 2022-01-31T21:55:29.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:28 smithi171 conmon[46715]: debug 2022-01-31T21:55:28.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:28.994675+0000) 2022-01-31T21:55:29.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:28 smithi171 conmon[51620]: debug 2022-01-31T21:55:28.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:28.838485+0000) 2022-01-31T21:55:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:28 smithi167 conmon[49112]: debug 2022-01-31T21:55:28.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:28.820973+0000) 2022-01-31T21:55:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:29 smithi171 conmon[41853]: debug 2022-01-31T21:55:29.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:29.198960+0000) 2022-01-31T21:55:29.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:29 smithi167 conmon[54076]: debug 2022-01-31T21:55:29.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:29.484797+0000) 2022-01-31T21:55:29.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:29 smithi167 conmon[60316]: debug 2022-01-31T21:55:29.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:29.484796+0000) 2022-01-31T21:55:30.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:29 smithi171 conmon[46715]: debug 2022-01-31T21:55:29.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:29.994863+0000) 2022-01-31T21:55:30.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:29 smithi171 conmon[51620]: debug 2022-01-31T21:55:29.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:29.838683+0000) 2022-01-31T21:55:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:29 smithi167 conmon[49112]: debug 2022-01-31T21:55:29.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:29.821189+0000) 2022-01-31T21:55:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:30 smithi171 conmon[41853]: debug 2022-01-31T21:55:30.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:30.199154+0000) 2022-01-31T21:55:30.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:30 smithi167 conmon[54076]: debug 2022-01-31T21:55:30.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:30.485005+0000) 2022-01-31T21:55:30.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:30 smithi167 conmon[60316]: debug 2022-01-31T21:55:30.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:30.484993+0000) 2022-01-31T21:55:31.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:30 smithi171 conmon[46715]: debug 2022-01-31T21:55:30.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:30.995047+0000) 2022-01-31T21:55:31.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:30 smithi171 conmon[51620]: debug 2022-01-31T21:55:30.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:30.838908+0000) 2022-01-31T21:55:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:30 smithi167 conmon[49112]: debug 2022-01-31T21:55:30.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:30.821408+0000) 2022-01-31T21:55:31.484 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:31 smithi167 conmon[49112]: debug 2022-01-31T21:55:31.326+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.326795+0000) 2022-01-31T21:55:31.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:31 smithi167 conmon[54076]: debug 2022-01-31T21:55:31.326+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.326960+0000) 2022-01-31T21:55:31.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:31 smithi167 conmon[60316]: debug 2022-01-31T21:55:31.327+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.327838+0000) 2022-01-31T21:55:31.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[35325]: debug 2022-01-31T21:55:31.342+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168450 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:31.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[46715]: debug 2022-01-31T21:55:31.325+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.327536+0000) 2022-01-31T21:55:31.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[51620]: debug 2022-01-31T21:55:31.325+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.327630+0000) 2022-01-31T21:55:31.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[41853]: debug 2022-01-31T21:55:31.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.199329+0000) 2022-01-31T21:55:31.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[41853]: debug 2022-01-31T21:55:31.324+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.326473+0000) 2022-01-31T21:55:31.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:31 smithi167 conmon[54076]: debug 2022-01-31T21:55:31.484+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.485195+0000) 2022-01-31T21:55:31.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:31 smithi167 conmon[60316]: debug 2022-01-31T21:55:31.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.485213+0000) 2022-01-31T21:55:32.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[46715]: debug 2022-01-31T21:55:31.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.995197+0000) 2022-01-31T21:55:32.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:31 smithi171 conmon[51620]: debug 2022-01-31T21:55:31.837+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.839024+0000) 2022-01-31T21:55:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:31 smithi167 conmon[49112]: debug 2022-01-31T21:55:31.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:31.821597+0000) 2022-01-31T21:55:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:32 smithi171 conmon[41853]: debug 2022-01-31T21:55:32.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:32.199527+0000) 2022-01-31T21:55:32.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:32 smithi167 conmon[54076]: debug 2022-01-31T21:55:32.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:32.485370+0000) 2022-01-31T21:55:32.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:32 smithi167 conmon[60316]: debug 2022-01-31T21:55:32.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:32.485397+0000) 2022-01-31T21:55:33.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:32 smithi171 conmon[51620]: debug 2022-01-31T21:55:32.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:32.839194+0000) 2022-01-31T21:55:33.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:32 smithi171 conmon[46715]: debug 2022-01-31T21:55:32.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:32.995348+0000) 2022-01-31T21:55:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:32 smithi167 conmon[49112]: debug 2022-01-31T21:55:32.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:32.821744+0000) 2022-01-31T21:55:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:33 smithi171 conmon[41853]: debug 2022-01-31T21:55:33.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:33.199712+0000) 2022-01-31T21:55:33.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:33 smithi167 conmon[60316]: debug 2022-01-31T21:55:33.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:33.485601+0000) 2022-01-31T21:55:33.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:33 smithi167 conmon[54076]: debug 2022-01-31T21:55:33.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:33.485590+0000) 2022-01-31T21:55:34.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:33 smithi171 conmon[51620]: debug 2022-01-31T21:55:33.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:33.839411+0000) 2022-01-31T21:55:34.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:33 smithi171 conmon[46715]: debug 2022-01-31T21:55:33.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:33.995491+0000) 2022-01-31T21:55:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:33 smithi167 conmon[49112]: debug 2022-01-31T21:55:33.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:33.821916+0000) 2022-01-31T21:55:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:34 smithi171 conmon[41853]: debug 2022-01-31T21:55:34.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:34.199944+0000) 2022-01-31T21:55:34.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:34 smithi167 conmon[60316]: debug 2022-01-31T21:55:34.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:34.485846+0000) 2022-01-31T21:55:34.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:34 smithi167 conmon[54076]: debug 2022-01-31T21:55:34.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:34.485824+0000) 2022-01-31T21:55:35.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:34 smithi171 conmon[51620]: debug 2022-01-31T21:55:34.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:34.839626+0000) 2022-01-31T21:55:35.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:34 smithi171 conmon[46715]: debug 2022-01-31T21:55:34.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:34.995603+0000) 2022-01-31T21:55:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:34 smithi167 conmon[49112]: debug 2022-01-31T21:55:34.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:34.822092+0000) 2022-01-31T21:55:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:35 smithi171 conmon[41853]: debug 2022-01-31T21:55:35.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:35.200125+0000) 2022-01-31T21:55:35.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:35 smithi167 conmon[54076]: debug 2022-01-31T21:55:35.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:35.486013+0000) 2022-01-31T21:55:35.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:35 smithi167 conmon[60316]: debug 2022-01-31T21:55:35.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:35.486015+0000) 2022-01-31T21:55:36.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:35 smithi171 conmon[51620]: debug 2022-01-31T21:55:35.838+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:35.839803+0000) 2022-01-31T21:55:36.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:35 smithi171 conmon[46715]: debug 2022-01-31T21:55:35.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:35.995767+0000) 2022-01-31T21:55:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:35 smithi167 conmon[49112]: debug 2022-01-31T21:55:35.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:35.822303+0000) 2022-01-31T21:55:36.485 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:36 smithi167 conmon[49112]: debug 2022-01-31T21:55:36.346+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.347189+0000) 2022-01-31T21:55:36.486 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:36 smithi167 conmon[54076]: debug 2022-01-31T21:55:36.346+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.347387+0000) 2022-01-31T21:55:36.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:36 smithi167 conmon[60316]: debug 2022-01-31T21:55:36.347+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.347544+0000) 2022-01-31T21:55:36.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[35325]: debug 2022-01-31T21:55:36.363+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168563 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:36.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[41853]: debug 2022-01-31T21:55:36.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.200252+0000) 2022-01-31T21:55:36.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[41853]: debug 2022-01-31T21:55:36.344+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.346268+0000) 2022-01-31T21:55:36.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[46715]: debug 2022-01-31T21:55:36.345+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.347041+0000) 2022-01-31T21:55:36.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[51620]: debug 2022-01-31T21:55:36.345+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.347433+0000) 2022-01-31T21:55:36.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:36 smithi167 conmon[54076]: debug 2022-01-31T21:55:36.485+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.486224+0000) 2022-01-31T21:55:36.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:36 smithi167 conmon[60316]: debug 2022-01-31T21:55:36.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.486221+0000) 2022-01-31T21:55:37.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[46715]: debug 2022-01-31T21:55:36.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.995926+0000) 2022-01-31T21:55:37.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:36 smithi171 conmon[51620]: debug 2022-01-31T21:55:36.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.839961+0000) 2022-01-31T21:55:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:36 smithi167 conmon[49112]: debug 2022-01-31T21:55:36.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:36.822546+0000) 2022-01-31T21:55:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:37 smithi171 conmon[41853]: debug 2022-01-31T21:55:37.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:37.200398+0000) 2022-01-31T21:55:37.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:37 smithi167 conmon[54076]: debug 2022-01-31T21:55:37.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:37.486391+0000) 2022-01-31T21:55:37.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:37 smithi167 conmon[60316]: debug 2022-01-31T21:55:37.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:37.486384+0000) 2022-01-31T21:55:38.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:37 smithi171 conmon[51620]: debug 2022-01-31T21:55:37.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:37.840127+0000) 2022-01-31T21:55:38.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:37 smithi171 conmon[46715]: debug 2022-01-31T21:55:37.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:37.996129+0000) 2022-01-31T21:55:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:37 smithi167 conmon[49112]: debug 2022-01-31T21:55:37.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:37.822718+0000) 2022-01-31T21:55:38.541 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:38 smithi167 conmon[60316]: debug 2022-01-31T21:55:38.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:38.486600+0000) 2022-01-31T21:55:38.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:38 smithi167 conmon[54076]: debug 2022-01-31T21:55:38.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:38.486600+0000) 2022-01-31T21:55:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:38 smithi171 conmon[41853]: debug 2022-01-31T21:55:38.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:38.200574+0000) 2022-01-31T21:55:39.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:38 smithi171 conmon[35325]: debug 2022-01-31T21:55:38.761+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:55:39.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:38 smithi171 conmon[46715]: debug 2022-01-31T21:55:38.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:38.996287+0000) 2022-01-31T21:55:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:38 smithi171 conmon[51620]: debug 2022-01-31T21:55:38.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:38.840334+0000) 2022-01-31T21:55:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:38 smithi167 conmon[49112]: debug 2022-01-31T21:55:38.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:38.822901+0000) 2022-01-31T21:55:39.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:39 smithi167 conmon[54076]: debug 2022-01-31T21:55:39.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:39.486758+0000) 2022-01-31T21:55:39.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:39 smithi167 conmon[60316]: debug 2022-01-31T21:55:39.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:39.486794+0000) 2022-01-31T21:55:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:39 smithi171 conmon[41853]: debug 2022-01-31T21:55:39.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:39.200778+0000) 2022-01-31T21:55:40.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:39 smithi171 conmon[46715]: debug 2022-01-31T21:55:39.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:39.996436+0000) 2022-01-31T21:55:40.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:39 smithi171 conmon[51620]: debug 2022-01-31T21:55:39.839+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:39.840519+0000) 2022-01-31T21:55:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:39 smithi167 conmon[49112]: debug 2022-01-31T21:55:39.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:39.823082+0000) 2022-01-31T21:55:40.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:40 smithi167 conmon[54076]: debug 2022-01-31T21:55:40.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:40.486958+0000) 2022-01-31T21:55:40.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:40 smithi167 conmon[60316]: debug 2022-01-31T21:55:40.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:40.486962+0000) 2022-01-31T21:55:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:40 smithi171 conmon[41853]: debug 2022-01-31T21:55:40.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:40.200900+0000) 2022-01-31T21:55:41.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:40 smithi171 conmon[51620]: debug 2022-01-31T21:55:40.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:40.840701+0000) 2022-01-31T21:55:41.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:40 smithi171 conmon[46715]: debug 2022-01-31T21:55:40.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:40.996643+0000) 2022-01-31T21:55:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:40 smithi167 conmon[49112]: debug 2022-01-31T21:55:40.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:40.823268+0000) 2022-01-31T21:55:41.542 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:41 smithi167 conmon[49112]: debug 2022-01-31T21:55:41.366+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.367268+0000) 2022-01-31T21:55:41.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:41 smithi167 conmon[60316]: debug 2022-01-31T21:55:41.366+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.367222+0000) 2022-01-31T21:55:41.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:41 smithi167 conmon[60316]: debug 2022-01-31T21:55:41.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.487193+0000) 2022-01-31T21:55:41.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:41 smithi167 conmon[54076]: debug 2022-01-31T21:55:41.367+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.367977+0000) 2022-01-31T21:55:41.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:41 smithi167 conmon[54076]: debug 2022-01-31T21:55:41.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.487155+0000) 2022-01-31T21:55:41.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[35325]: debug 2022-01-31T21:55:41.384+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168671 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:41.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[46715]: debug 2022-01-31T21:55:41.367+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.367943+0000) 2022-01-31T21:55:41.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[51620]: debug 2022-01-31T21:55:41.366+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.367152+0000) 2022-01-31T21:55:41.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[41853]: debug 2022-01-31T21:55:41.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.201025+0000) 2022-01-31T21:55:41.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[41853]: debug 2022-01-31T21:55:41.366+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.367003+0000) 2022-01-31T21:55:42.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[46715]: debug 2022-01-31T21:55:41.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.996831+0000) 2022-01-31T21:55:42.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:41 smithi171 conmon[51620]: debug 2022-01-31T21:55:41.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.840908+0000) 2022-01-31T21:55:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:41 smithi167 conmon[49112]: debug 2022-01-31T21:55:41.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:41.823484+0000) 2022-01-31T21:55:42.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:42 smithi167 conmon[54076]: debug 2022-01-31T21:55:42.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:42.487291+0000) 2022-01-31T21:55:42.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:42 smithi167 conmon[60316]: debug 2022-01-31T21:55:42.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:42.487326+0000) 2022-01-31T21:55:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:42 smithi171 conmon[41853]: debug 2022-01-31T21:55:42.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:42.201182+0000) 2022-01-31T21:55:43.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:42 smithi171 conmon[46715]: debug 2022-01-31T21:55:42.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:42.997020+0000) 2022-01-31T21:55:43.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:42 smithi171 conmon[51620]: debug 2022-01-31T21:55:42.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:42.841035+0000) 2022-01-31T21:55:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:42 smithi167 conmon[49112]: debug 2022-01-31T21:55:42.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:42.823693+0000) 2022-01-31T21:55:43.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:43 smithi167 conmon[54076]: debug 2022-01-31T21:55:43.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:43.487483+0000) 2022-01-31T21:55:43.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:43 smithi167 conmon[60316]: debug 2022-01-31T21:55:43.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:43.487487+0000) 2022-01-31T21:55:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:43 smithi171 conmon[41853]: debug 2022-01-31T21:55:43.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:43.201397+0000) 2022-01-31T21:55:44.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:43 smithi171 conmon[46715]: debug 2022-01-31T21:55:43.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:43.997215+0000) 2022-01-31T21:55:44.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:43 smithi171 conmon[51620]: debug 2022-01-31T21:55:43.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:43.841274+0000) 2022-01-31T21:55:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:43 smithi167 conmon[49112]: debug 2022-01-31T21:55:43.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:43.823881+0000) 2022-01-31T21:55:44.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:44 smithi167 conmon[54076]: debug 2022-01-31T21:55:44.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:44.487723+0000) 2022-01-31T21:55:44.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:44 smithi167 conmon[60316]: debug 2022-01-31T21:55:44.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:44.487720+0000) 2022-01-31T21:55:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:44 smithi171 conmon[41853]: debug 2022-01-31T21:55:44.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:44.201608+0000) 2022-01-31T21:55:45.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:44 smithi171 conmon[51620]: debug 2022-01-31T21:55:44.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:44.841465+0000) 2022-01-31T21:55:45.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:44 smithi171 conmon[46715]: debug 2022-01-31T21:55:44.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:44.997402+0000) 2022-01-31T21:55:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:44 smithi167 conmon[49112]: debug 2022-01-31T21:55:44.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:44.824066+0000) 2022-01-31T21:55:45.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:45 smithi167 conmon[54076]: debug 2022-01-31T21:55:45.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:45.487929+0000) 2022-01-31T21:55:45.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:45 smithi167 conmon[60316]: debug 2022-01-31T21:55:45.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:45.487929+0000) 2022-01-31T21:55:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:45 smithi171 conmon[41853]: debug 2022-01-31T21:55:45.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:45.201792+0000) 2022-01-31T21:55:46.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:45 smithi171 conmon[51620]: debug 2022-01-31T21:55:45.840+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:45.841612+0000) 2022-01-31T21:55:46.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:45 smithi171 conmon[46715]: debug 2022-01-31T21:55:45.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:45.997596+0000) 2022-01-31T21:55:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:45 smithi167 conmon[49112]: debug 2022-01-31T21:55:45.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:45.824240+0000) 2022-01-31T21:55:46.487 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:46 smithi167 conmon[49112]: debug 2022-01-31T21:55:46.385+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.387109+0000) 2022-01-31T21:55:46.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:46 smithi167 conmon[54076]: debug 2022-01-31T21:55:46.386+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.388303+0000) 2022-01-31T21:55:46.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:46 smithi167 conmon[60316]: debug 2022-01-31T21:55:46.385+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.386722+0000) 2022-01-31T21:55:46.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[35325]: debug 2022-01-31T21:55:46.404+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168784 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:46.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[46715]: debug 2022-01-31T21:55:46.387+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.387916+0000) 2022-01-31T21:55:46.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[51620]: debug 2022-01-31T21:55:46.387+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.388517+0000) 2022-01-31T21:55:46.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[41853]: debug 2022-01-31T21:55:46.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.201973+0000) 2022-01-31T21:55:46.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[41853]: debug 2022-01-31T21:55:46.386+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.387728+0000) 2022-01-31T21:55:46.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:46 smithi167 conmon[60316]: debug 2022-01-31T21:55:46.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.488128+0000) 2022-01-31T21:55:46.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:46 smithi167 conmon[54076]: debug 2022-01-31T21:55:46.486+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.488130+0000) 2022-01-31T21:55:47.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[46715]: debug 2022-01-31T21:55:46.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.997813+0000) 2022-01-31T21:55:47.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:46 smithi171 conmon[51620]: debug 2022-01-31T21:55:46.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.841800+0000) 2022-01-31T21:55:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:46 smithi167 conmon[49112]: debug 2022-01-31T21:55:46.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:46.824423+0000) 2022-01-31T21:55:47.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:47 smithi167 conmon[54076]: debug 2022-01-31T21:55:47.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:47.488286+0000) 2022-01-31T21:55:47.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:47 smithi167 conmon[60316]: debug 2022-01-31T21:55:47.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:47.488303+0000) 2022-01-31T21:55:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:47 smithi171 conmon[41853]: debug 2022-01-31T21:55:47.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:47.202130+0000) 2022-01-31T21:55:48.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:47 smithi171 conmon[46715]: debug 2022-01-31T21:55:47.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:47.997966+0000) 2022-01-31T21:55:48.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:47 smithi171 conmon[51620]: debug 2022-01-31T21:55:47.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:47.841925+0000) 2022-01-31T21:55:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:47 smithi167 conmon[49112]: debug 2022-01-31T21:55:47.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:47.824554+0000) 2022-01-31T21:55:48.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:48 smithi167 conmon[54076]: debug 2022-01-31T21:55:48.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:48.488461+0000) 2022-01-31T21:55:48.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:48 smithi167 conmon[60316]: debug 2022-01-31T21:55:48.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:48.488492+0000) 2022-01-31T21:55:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:48 smithi171 conmon[41853]: debug 2022-01-31T21:55:48.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:48.202286+0000) 2022-01-31T21:55:49.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:48 smithi171 conmon[46715]: debug 2022-01-31T21:55:48.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:48.998176+0000) 2022-01-31T21:55:49.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:48 smithi171 conmon[51620]: debug 2022-01-31T21:55:48.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:48.842157+0000) 2022-01-31T21:55:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:48 smithi167 conmon[49112]: debug 2022-01-31T21:55:48.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:48.824768+0000) 2022-01-31T21:55:49.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:49 smithi167 conmon[54076]: debug 2022-01-31T21:55:49.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:49.488639+0000) 2022-01-31T21:55:49.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:49 smithi167 conmon[60316]: debug 2022-01-31T21:55:49.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:49.488670+0000) 2022-01-31T21:55:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:49 smithi171 conmon[41853]: debug 2022-01-31T21:55:49.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:49.202504+0000) 2022-01-31T21:55:50.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:49 smithi171 conmon[46715]: debug 2022-01-31T21:55:49.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:49.998362+0000) 2022-01-31T21:55:50.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:49 smithi171 conmon[51620]: debug 2022-01-31T21:55:49.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:49.842371+0000) 2022-01-31T21:55:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:49 smithi167 conmon[49112]: debug 2022-01-31T21:55:49.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:49.824952+0000) 2022-01-31T21:55:50.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:50 smithi167 conmon[54076]: debug 2022-01-31T21:55:50.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:50.488836+0000) 2022-01-31T21:55:50.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:50 smithi167 conmon[60316]: debug 2022-01-31T21:55:50.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:50.488793+0000) 2022-01-31T21:55:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:50 smithi171 conmon[41853]: debug 2022-01-31T21:55:50.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:50.202724+0000) 2022-01-31T21:55:51.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:50 smithi171 conmon[46715]: debug 2022-01-31T21:55:50.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:50.998551+0000) 2022-01-31T21:55:51.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:50 smithi171 conmon[51620]: debug 2022-01-31T21:55:50.841+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:50.842601+0000) 2022-01-31T21:55:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:50 smithi167 conmon[49112]: debug 2022-01-31T21:55:50.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:50.825161+0000) 2022-01-31T21:55:51.424 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[46715]: debug 2022-01-31T21:55:51.406+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.407172+0000) 2022-01-31T21:55:51.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[51620]: debug 2022-01-31T21:55:51.407+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.408054+0000) 2022-01-31T21:55:51.425 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[41853]: debug 2022-01-31T21:55:51.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.202932+0000) 2022-01-31T21:55:51.425 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[41853]: debug 2022-01-31T21:55:51.407+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.408254+0000) 2022-01-31T21:55:51.543 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:51 smithi167 conmon[49112]: debug 2022-01-31T21:55:51.406+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.407945+0000) 2022-01-31T21:55:51.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:51 smithi167 conmon[54076]: debug 2022-01-31T21:55:51.406+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.408367+0000) 2022-01-31T21:55:51.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:51 smithi167 conmon[54076]: debug 2022-01-31T21:55:51.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.489044+0000) 2022-01-31T21:55:51.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:51 smithi167 conmon[60316]: debug 2022-01-31T21:55:51.405+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.407265+0000) 2022-01-31T21:55:51.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:51 smithi167 conmon[60316]: debug 2022-01-31T21:55:51.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.489019+0000) 2022-01-31T21:55:51.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[35325]: debug 2022-01-31T21:55:51.424+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 168893 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:52.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[51620]: debug 2022-01-31T21:55:51.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.842797+0000) 2022-01-31T21:55:52.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:51 smithi171 conmon[46715]: debug 2022-01-31T21:55:51.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.998737+0000) 2022-01-31T21:55:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:51 smithi167 conmon[49112]: debug 2022-01-31T21:55:51.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:51.825368+0000) 2022-01-31T21:55:52.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:52 smithi167 conmon[54076]: debug 2022-01-31T21:55:52.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:52.489200+0000) 2022-01-31T21:55:52.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:52 smithi167 conmon[60316]: debug 2022-01-31T21:55:52.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:52.489204+0000) 2022-01-31T21:55:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:52 smithi171 conmon[41853]: debug 2022-01-31T21:55:52.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:52.203141+0000) 2022-01-31T21:55:53.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:52 smithi171 conmon[46715]: debug 2022-01-31T21:55:52.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:52.998884+0000) 2022-01-31T21:55:53.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:52 smithi171 conmon[51620]: debug 2022-01-31T21:55:52.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:52.842956+0000) 2022-01-31T21:55:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:52 smithi167 conmon[49112]: debug 2022-01-31T21:55:52.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:52.825530+0000) 2022-01-31T21:55:53.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:53 smithi167 conmon[54076]: debug 2022-01-31T21:55:53.487+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:53.489396+0000) 2022-01-31T21:55:53.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:53 smithi167 conmon[60316]: debug 2022-01-31T21:55:53.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:53.489399+0000) 2022-01-31T21:55:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:53 smithi171 conmon[41853]: debug 2022-01-31T21:55:53.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:53.203315+0000) 2022-01-31T21:55:54.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:53 smithi171 conmon[51620]: debug 2022-01-31T21:55:53.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:53.843116+0000) 2022-01-31T21:55:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:53 smithi171 conmon[35325]: debug 2022-01-31T21:55:53.762+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:55:54.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:53 smithi171 conmon[46715]: debug 2022-01-31T21:55:53.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:53.999030+0000) 2022-01-31T21:55:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:53 smithi167 conmon[49112]: debug 2022-01-31T21:55:53.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:53.825652+0000) 2022-01-31T21:55:54.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:54 smithi167 conmon[54076]: debug 2022-01-31T21:55:54.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:54.489596+0000) 2022-01-31T21:55:54.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:54 smithi167 conmon[60316]: debug 2022-01-31T21:55:54.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:54.489642+0000) 2022-01-31T21:55:54.584 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:54 smithi171 conmon[41853]: debug 2022-01-31T21:55:54.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:54.203505+0000) 2022-01-31T21:55:55.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:54 smithi171 conmon[46715]: debug 2022-01-31T21:55:54.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:54.999236+0000) 2022-01-31T21:55:55.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:54 smithi171 conmon[51620]: debug 2022-01-31T21:55:54.842+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:54.843327+0000) 2022-01-31T21:55:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:54 smithi167 conmon[49112]: debug 2022-01-31T21:55:54.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:54.825814+0000) 2022-01-31T21:55:55.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:55 smithi167 conmon[54076]: debug 2022-01-31T21:55:55.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:55.489787+0000) 2022-01-31T21:55:55.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:55 smithi167 conmon[60316]: debug 2022-01-31T21:55:55.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:55.489829+0000) 2022-01-31T21:55:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:55 smithi171 conmon[41853]: debug 2022-01-31T21:55:55.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:55.203684+0000) 2022-01-31T21:55:56.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:55 smithi171 conmon[51620]: debug 2022-01-31T21:55:55.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:55.843543+0000) 2022-01-31T21:55:56.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:55 smithi171 conmon[46715]: debug 2022-01-31T21:55:55.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:55.999442+0000) 2022-01-31T21:55:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:55 smithi167 conmon[49112]: debug 2022-01-31T21:55:55.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:55.826016+0000) 2022-01-31T21:55:56.426 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[41853]: debug 2022-01-31T21:55:56.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.203860+0000) 2022-01-31T21:55:56.544 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:56 smithi167 conmon[49112]: debug 2022-01-31T21:55:56.427+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.428526+0000) 2022-01-31T21:55:56.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:56 smithi167 conmon[54076]: debug 2022-01-31T21:55:56.427+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.428747+0000) 2022-01-31T21:55:56.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:56 smithi167 conmon[54076]: debug 2022-01-31T21:55:56.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.489938+0000) 2022-01-31T21:55:56.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:56 smithi167 conmon[60316]: debug 2022-01-31T21:55:56.426+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.427904+0000) 2022-01-31T21:55:56.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:56 smithi167 conmon[60316]: debug 2022-01-31T21:55:56.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.489975+0000) 2022-01-31T21:55:56.842 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[35325]: debug 2022-01-31T21:55:56.445+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169006 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:55:56.843 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[41853]: debug 2022-01-31T21:55:56.427+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.428480+0000) 2022-01-31T21:55:56.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[46715]: debug 2022-01-31T21:55:56.426+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.427404+0000) 2022-01-31T21:55:56.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[51620]: debug 2022-01-31T21:55:56.427+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.427948+0000) 2022-01-31T21:55:57.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:56 smithi167 conmon[49112]: debug 2022-01-31T21:55:56.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.826197+0000) 2022-01-31T21:55:57.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[46715]: debug 2022-01-31T21:55:56.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.999684+0000) 2022-01-31T21:55:57.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:56 smithi171 conmon[51620]: debug 2022-01-31T21:55:56.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:56.843800+0000) 2022-01-31T21:55:57.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:57 smithi167 conmon[54076]: debug 2022-01-31T21:55:57.488+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:57.490082+0000) 2022-01-31T21:55:57.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:57 smithi167 conmon[60316]: debug 2022-01-31T21:55:57.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:57.490111+0000) 2022-01-31T21:55:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:57 smithi171 conmon[41853]: debug 2022-01-31T21:55:57.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:57.204048+0000) 2022-01-31T21:55:58.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:57 smithi171 conmon[46715]: debug 2022-01-31T21:55:57.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:57.999804+0000) 2022-01-31T21:55:58.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:57 smithi171 conmon[51620]: debug 2022-01-31T21:55:57.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:57.843981+0000) 2022-01-31T21:55:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:57 smithi167 conmon[49112]: debug 2022-01-31T21:55:57.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:57.826338+0000) 2022-01-31T21:55:58.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:58 smithi167 conmon[54076]: debug 2022-01-31T21:55:58.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:58.490304+0000) 2022-01-31T21:55:58.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:58 smithi167 conmon[60316]: debug 2022-01-31T21:55:58.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:58.490321+0000) 2022-01-31T21:55:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:58 smithi171 conmon[41853]: debug 2022-01-31T21:55:58.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:58.204256+0000) 2022-01-31T21:55:59.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:55:59 smithi171 conmon[46715]: debug 2022-01-31T21:55:58.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:58.999986+0000) 2022-01-31T21:55:59.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:58 smithi171 conmon[51620]: debug 2022-01-31T21:55:58.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:58.844162+0000) 2022-01-31T21:55:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:58 smithi167 conmon[49112]: debug 2022-01-31T21:55:58.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:58.826539+0000) 2022-01-31T21:55:59.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:55:59 smithi167 conmon[54076]: debug 2022-01-31T21:55:59.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:59.490542+0000) 2022-01-31T21:55:59.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:55:59 smithi167 conmon[60316]: debug 2022-01-31T21:55:59.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:59.490534+0000) 2022-01-31T21:55:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:55:59 smithi171 conmon[41853]: debug 2022-01-31T21:55:59.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:59.204455+0000) 2022-01-31T21:56:00.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:00 smithi171 conmon[46715]: debug 2022-01-31T21:55:59.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:00.000169+0000) 2022-01-31T21:56:00.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:55:59 smithi171 conmon[51620]: debug 2022-01-31T21:55:59.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:59.844350+0000) 2022-01-31T21:56:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:55:59 smithi167 conmon[49112]: debug 2022-01-31T21:55:59.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:55:59.826759+0000) 2022-01-31T21:56:00.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:00 smithi167 conmon[54076]: debug 2022-01-31T21:56:00.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:00.490707+0000) 2022-01-31T21:56:00.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:00 smithi167 conmon[60316]: debug 2022-01-31T21:56:00.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:00.490737+0000) 2022-01-31T21:56:00.605 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:00 smithi171 conmon[41853]: debug 2022-01-31T21:56:00.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:00.204628+0000) 2022-01-31T21:56:01.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[46715]: debug 2022-01-31T21:56:00.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.000359+0000) 2022-01-31T21:56:01.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:00 smithi171 conmon[51620]: debug 2022-01-31T21:56:00.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:00.844513+0000) 2022-01-31T21:56:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:00 smithi167 conmon[49112]: debug 2022-01-31T21:56:00.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:00.826917+0000) 2022-01-31T21:56:01.446 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[41853]: debug 2022-01-31T21:56:01.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.204827+0000) 2022-01-31T21:56:01.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[51620]: debug 2022-01-31T21:56:01.446+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.447714+0000) 2022-01-31T21:56:01.545 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:01 smithi167 conmon[49112]: debug 2022-01-31T21:56:01.446+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.448363+0000) 2022-01-31T21:56:01.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:01 smithi167 conmon[54076]: debug 2022-01-31T21:56:01.447+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.448618+0000) 2022-01-31T21:56:01.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:01 smithi167 conmon[54076]: debug 2022-01-31T21:56:01.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.490863+0000) 2022-01-31T21:56:01.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:01 smithi167 conmon[60316]: debug 2022-01-31T21:56:01.445+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.447307+0000) 2022-01-31T21:56:01.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:01 smithi167 conmon[60316]: debug 2022-01-31T21:56:01.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.490925+0000) 2022-01-31T21:56:01.843 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[35325]: debug 2022-01-31T21:56:01.464+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169114 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:01.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[46715]: debug 2022-01-31T21:56:01.447+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.447886+0000) 2022-01-31T21:56:01.844 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[41853]: debug 2022-01-31T21:56:01.448+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.449160+0000) 2022-01-31T21:56:02.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:02 smithi171 conmon[46715]: debug 2022-01-31T21:56:02.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:02.000525+0000) 2022-01-31T21:56:02.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:01 smithi171 conmon[51620]: debug 2022-01-31T21:56:01.843+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.844684+0000) 2022-01-31T21:56:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:01 smithi167 conmon[49112]: debug 2022-01-31T21:56:01.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:01.827120+0000) 2022-01-31T21:56:02.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:02 smithi167 conmon[60316]: debug 2022-01-31T21:56:02.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:02.491042+0000) 2022-01-31T21:56:02.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:02 smithi167 conmon[54076]: debug 2022-01-31T21:56:02.489+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:02.490976+0000) 2022-01-31T21:56:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:02 smithi171 conmon[41853]: debug 2022-01-31T21:56:02.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:02.204986+0000) 2022-01-31T21:56:03.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:03 smithi171 conmon[46715]: debug 2022-01-31T21:56:03.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:03.000659+0000) 2022-01-31T21:56:03.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:02 smithi171 conmon[51620]: debug 2022-01-31T21:56:02.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:02.844854+0000) 2022-01-31T21:56:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:02 smithi167 conmon[49112]: debug 2022-01-31T21:56:02.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:02.827300+0000) 2022-01-31T21:56:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:03 smithi171 conmon[41853]: debug 2022-01-31T21:56:03.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:03.205187+0000) 2022-01-31T21:56:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:03 smithi167 conmon[54076]: debug 2022-01-31T21:56:03.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:03.491135+0000) 2022-01-31T21:56:03.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:03 smithi167 conmon[60316]: debug 2022-01-31T21:56:03.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:03.491196+0000) 2022-01-31T21:56:03.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:03 smithi167 conmon[49112]: debug 2022-01-31T21:56:03.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:03.827522+0000) 2022-01-31T21:56:04.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:04 smithi171 conmon[46715]: debug 2022-01-31T21:56:04.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:04.000831+0000) 2022-01-31T21:56:04.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:03 smithi171 conmon[51620]: debug 2022-01-31T21:56:03.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:03.845088+0000) 2022-01-31T21:56:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:04 smithi171 conmon[41853]: debug 2022-01-31T21:56:04.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:04.205373+0000) 2022-01-31T21:56:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:04 smithi167 conmon[54076]: debug 2022-01-31T21:56:04.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:04.491358+0000) 2022-01-31T21:56:04.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:04 smithi167 conmon[60316]: debug 2022-01-31T21:56:04.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:04.491361+0000) 2022-01-31T21:56:04.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:04 smithi167 conmon[49112]: debug 2022-01-31T21:56:04.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:04.827677+0000) 2022-01-31T21:56:05.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:05 smithi171 conmon[46715]: debug 2022-01-31T21:56:05.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:05.000958+0000) 2022-01-31T21:56:05.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:04 smithi171 conmon[51620]: debug 2022-01-31T21:56:04.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:04.845300+0000) 2022-01-31T21:56:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:05 smithi171 conmon[41853]: debug 2022-01-31T21:56:05.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:05.205561+0000) 2022-01-31T21:56:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:05 smithi167 conmon[54076]: debug 2022-01-31T21:56:05.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:05.491552+0000) 2022-01-31T21:56:05.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:05 smithi167 conmon[60316]: debug 2022-01-31T21:56:05.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:05.491552+0000) 2022-01-31T21:56:05.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:05 smithi167 conmon[49112]: debug 2022-01-31T21:56:05.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:05.827852+0000) 2022-01-31T21:56:06.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[46715]: debug 2022-01-31T21:56:06.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.001079+0000) 2022-01-31T21:56:06.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:05 smithi171 conmon[51620]: debug 2022-01-31T21:56:05.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:05.845531+0000) 2022-01-31T21:56:06.466 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[41853]: debug 2022-01-31T21:56:06.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.205788+0000) 2022-01-31T21:56:06.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:06 smithi167 conmon[49112]: debug 2022-01-31T21:56:06.466+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.467689+0000) 2022-01-31T21:56:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:06 smithi167 conmon[60316]: debug 2022-01-31T21:56:06.465+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.467382+0000) 2022-01-31T21:56:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:06 smithi167 conmon[60316]: debug 2022-01-31T21:56:06.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.491674+0000) 2022-01-31T21:56:06.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:06 smithi167 conmon[54076]: debug 2022-01-31T21:56:06.466+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.468290+0000) 2022-01-31T21:56:06.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:06 smithi167 conmon[54076]: debug 2022-01-31T21:56:06.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.491732+0000) 2022-01-31T21:56:06.844 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[35325]: debug 2022-01-31T21:56:06.485+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169228 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:06.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[41853]: debug 2022-01-31T21:56:06.468+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.469199+0000) 2022-01-31T21:56:06.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[46715]: debug 2022-01-31T21:56:06.466+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.467498+0000) 2022-01-31T21:56:06.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[51620]: debug 2022-01-31T21:56:06.466+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.467420+0000) 2022-01-31T21:56:06.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:06 smithi167 conmon[49112]: debug 2022-01-31T21:56:06.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.828036+0000) 2022-01-31T21:56:07.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:07 smithi171 conmon[46715]: debug 2022-01-31T21:56:07.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:07.001271+0000) 2022-01-31T21:56:07.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:06 smithi171 conmon[51620]: debug 2022-01-31T21:56:06.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:06.845743+0000) 2022-01-31T21:56:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:07 smithi171 conmon[41853]: debug 2022-01-31T21:56:07.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:07.205975+0000) 2022-01-31T21:56:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:07 smithi167 conmon[54076]: debug 2022-01-31T21:56:07.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:07.491943+0000) 2022-01-31T21:56:07.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:07 smithi167 conmon[60316]: debug 2022-01-31T21:56:07.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:07.491836+0000) 2022-01-31T21:56:07.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:07 smithi167 conmon[49112]: debug 2022-01-31T21:56:07.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:07.828201+0000) 2022-01-31T21:56:08.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:08 smithi171 conmon[46715]: debug 2022-01-31T21:56:08.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:08.001442+0000) 2022-01-31T21:56:08.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:07 smithi171 conmon[51620]: debug 2022-01-31T21:56:07.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:07.845916+0000) 2022-01-31T21:56:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:08 smithi171 conmon[41853]: debug 2022-01-31T21:56:08.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:08.206126+0000) 2022-01-31T21:56:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:08 smithi167 conmon[54076]: debug 2022-01-31T21:56:08.490+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:08.492159+0000) 2022-01-31T21:56:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:08 smithi167 conmon[60316]: debug 2022-01-31T21:56:08.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:08.492065+0000) 2022-01-31T21:56:08.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:08 smithi167 conmon[49112]: debug 2022-01-31T21:56:08.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:08.828400+0000) 2022-01-31T21:56:09.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:08 smithi171 conmon[51620]: debug 2022-01-31T21:56:08.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:08.846093+0000) 2022-01-31T21:56:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:08 smithi171 conmon[35325]: debug 2022-01-31T21:56:08.763+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:56:09.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:09 smithi171 conmon[46715]: debug 2022-01-31T21:56:09.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:09.001630+0000) 2022-01-31T21:56:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:09 smithi171 conmon[41853]: debug 2022-01-31T21:56:09.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:09.206312+0000) 2022-01-31T21:56:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:09 smithi167 conmon[54076]: debug 2022-01-31T21:56:09.491+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:09.492303+0000) 2022-01-31T21:56:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:09 smithi167 conmon[60316]: debug 2022-01-31T21:56:09.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:09.492220+0000) 2022-01-31T21:56:09.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:09 smithi167 conmon[49112]: debug 2022-01-31T21:56:09.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:09.828572+0000) 2022-01-31T21:56:10.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:10 smithi171 conmon[46715]: debug 2022-01-31T21:56:10.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:10.001820+0000) 2022-01-31T21:56:10.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:09 smithi171 conmon[51620]: debug 2022-01-31T21:56:09.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:09.846306+0000) 2022-01-31T21:56:10.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:10 smithi171 conmon[41853]: debug 2022-01-31T21:56:10.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:10.206490+0000) 2022-01-31T21:56:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:10 smithi167 conmon[54076]: debug 2022-01-31T21:56:10.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:10.492563+0000) 2022-01-31T21:56:10.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:10 smithi167 conmon[60316]: debug 2022-01-31T21:56:10.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:10.492427+0000) 2022-01-31T21:56:10.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:10 smithi167 conmon[49112]: debug 2022-01-31T21:56:10.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:10.828756+0000) 2022-01-31T21:56:11.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:10 smithi171 conmon[51620]: debug 2022-01-31T21:56:10.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:10.846489+0000) 2022-01-31T21:56:11.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[41853]: debug 2022-01-31T21:56:11.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.206682+0000) 2022-01-31T21:56:11.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[46715]: debug 2022-01-31T21:56:11.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.002016+0000) 2022-01-31T21:56:11.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:11 smithi167 conmon[49112]: debug 2022-01-31T21:56:11.488+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.489144+0000) 2022-01-31T21:56:11.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:11 smithi167 conmon[54076]: debug 2022-01-31T21:56:11.488+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.488581+0000) 2022-01-31T21:56:11.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:11 smithi167 conmon[54076]: debug 2022-01-31T21:56:11.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.492963+0000) 2022-01-31T21:56:11.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:11 smithi167 conmon[60316]: debug 2022-01-31T21:56:11.487+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.488348+0000) 2022-01-31T21:56:11.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:11 smithi167 conmon[60316]: debug 2022-01-31T21:56:11.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.492596+0000) 2022-01-31T21:56:11.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[35325]: debug 2022-01-31T21:56:11.504+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169337 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:11.846 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[41853]: debug 2022-01-31T21:56:11.487+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.489693+0000) 2022-01-31T21:56:11.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[46715]: debug 2022-01-31T21:56:11.486+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.488326+0000) 2022-01-31T21:56:11.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[51620]: debug 2022-01-31T21:56:11.486+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.488262+0000) 2022-01-31T21:56:11.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:11 smithi167 conmon[49112]: debug 2022-01-31T21:56:11.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.828970+0000) 2022-01-31T21:56:12.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:12 smithi171 conmon[46715]: debug 2022-01-31T21:56:12.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:12.002219+0000) 2022-01-31T21:56:12.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:11 smithi171 conmon[51620]: debug 2022-01-31T21:56:11.844+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:11.846688+0000) 2022-01-31T21:56:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:12 smithi171 conmon[41853]: debug 2022-01-31T21:56:12.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:12.206874+0000) 2022-01-31T21:56:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:12 smithi167 conmon[54076]: debug 2022-01-31T21:56:12.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:12.493077+0000) 2022-01-31T21:56:12.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:12 smithi167 conmon[60316]: debug 2022-01-31T21:56:12.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:12.492736+0000) 2022-01-31T21:56:12.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:12 smithi167 conmon[49112]: debug 2022-01-31T21:56:12.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:12.829132+0000) 2022-01-31T21:56:13.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:13 smithi171 conmon[46715]: debug 2022-01-31T21:56:13.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:13.002390+0000) 2022-01-31T21:56:13.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:12 smithi171 conmon[51620]: debug 2022-01-31T21:56:12.845+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:12.846827+0000) 2022-01-31T21:56:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:13 smithi171 conmon[41853]: debug 2022-01-31T21:56:13.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:13.207076+0000) 2022-01-31T21:56:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:13 smithi167 conmon[54076]: debug 2022-01-31T21:56:13.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:13.493215+0000) 2022-01-31T21:56:13.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:13 smithi167 conmon[60316]: debug 2022-01-31T21:56:13.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:13.492894+0000) 2022-01-31T21:56:13.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:13 smithi167 conmon[49112]: debug 2022-01-31T21:56:13.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:13.829337+0000) 2022-01-31T21:56:14.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:14 smithi171 conmon[46715]: debug 2022-01-31T21:56:14.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:14.002564+0000) 2022-01-31T21:56:14.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:13 smithi171 conmon[51620]: debug 2022-01-31T21:56:13.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:13.847013+0000) 2022-01-31T21:56:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:14 smithi171 conmon[41853]: debug 2022-01-31T21:56:14.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:14.207253+0000) 2022-01-31T21:56:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:14 smithi167 conmon[54076]: debug 2022-01-31T21:56:14.492+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:14.493374+0000) 2022-01-31T21:56:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:14 smithi167 conmon[60316]: debug 2022-01-31T21:56:14.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:14.493076+0000) 2022-01-31T21:56:14.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:14 smithi167 conmon[49112]: debug 2022-01-31T21:56:14.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:14.829549+0000) 2022-01-31T21:56:15.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:15 smithi171 conmon[46715]: debug 2022-01-31T21:56:15.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:15.002728+0000) 2022-01-31T21:56:15.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:14 smithi171 conmon[51620]: debug 2022-01-31T21:56:14.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:14.847209+0000) 2022-01-31T21:56:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:15 smithi171 conmon[41853]: debug 2022-01-31T21:56:15.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:15.207458+0000) 2022-01-31T21:56:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:15 smithi167 conmon[54076]: debug 2022-01-31T21:56:15.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:15.493536+0000) 2022-01-31T21:56:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:15 smithi167 conmon[60316]: debug 2022-01-31T21:56:15.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:15.493272+0000) 2022-01-31T21:56:15.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:15 smithi167 conmon[49112]: debug 2022-01-31T21:56:15.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:15.829758+0000) 2022-01-31T21:56:16.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[46715]: debug 2022-01-31T21:56:16.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.002923+0000) 2022-01-31T21:56:16.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:15 smithi171 conmon[51620]: debug 2022-01-31T21:56:15.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:15.847420+0000) 2022-01-31T21:56:16.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[41853]: debug 2022-01-31T21:56:16.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.207640+0000) 2022-01-31T21:56:16.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[41853]: debug 2022-01-31T21:56:16.506+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.507949+0000) 2022-01-31T21:56:16.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:16 smithi167 conmon[49112]: debug 2022-01-31T21:56:16.508+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.508700+0000) 2022-01-31T21:56:16.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:16 smithi167 conmon[54076]: debug 2022-01-31T21:56:16.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.493664+0000) 2022-01-31T21:56:16.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:16 smithi167 conmon[54076]: debug 2022-01-31T21:56:16.507+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.507745+0000) 2022-01-31T21:56:16.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:16 smithi167 conmon[60316]: debug 2022-01-31T21:56:16.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.493472+0000) 2022-01-31T21:56:16.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:16 smithi167 conmon[60316]: debug 2022-01-31T21:56:16.507+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.507667+0000) 2022-01-31T21:56:16.846 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[35325]: debug 2022-01-31T21:56:16.523+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169450 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:16.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[46715]: debug 2022-01-31T21:56:16.506+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.508704+0000) 2022-01-31T21:56:16.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[51620]: debug 2022-01-31T21:56:16.506+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.508219+0000) 2022-01-31T21:56:16.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:16 smithi167 conmon[49112]: debug 2022-01-31T21:56:16.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.829945+0000) 2022-01-31T21:56:17.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:17 smithi171 conmon[46715]: debug 2022-01-31T21:56:17.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:17.003085+0000) 2022-01-31T21:56:17.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:16 smithi171 conmon[51620]: debug 2022-01-31T21:56:16.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:16.847625+0000) 2022-01-31T21:56:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:17 smithi171 conmon[41853]: debug 2022-01-31T21:56:17.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:17.207846+0000) 2022-01-31T21:56:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:17 smithi167 conmon[54076]: debug 2022-01-31T21:56:17.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:17.493816+0000) 2022-01-31T21:56:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:17 smithi167 conmon[60316]: debug 2022-01-31T21:56:17.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:17.493637+0000) 2022-01-31T21:56:17.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:17 smithi167 conmon[49112]: debug 2022-01-31T21:56:17.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:17.830105+0000) 2022-01-31T21:56:18.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:18 smithi171 conmon[46715]: debug 2022-01-31T21:56:18.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:18.003226+0000) 2022-01-31T21:56:18.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:17 smithi171 conmon[51620]: debug 2022-01-31T21:56:17.846+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:17.847792+0000) 2022-01-31T21:56:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:18 smithi171 conmon[41853]: debug 2022-01-31T21:56:18.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:18.207984+0000) 2022-01-31T21:56:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:18 smithi167 conmon[54076]: debug 2022-01-31T21:56:18.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:18.493932+0000) 2022-01-31T21:56:18.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:18 smithi167 conmon[60316]: debug 2022-01-31T21:56:18.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:18.493786+0000) 2022-01-31T21:56:18.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:18 smithi167 conmon[49112]: debug 2022-01-31T21:56:18.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:18.830304+0000) 2022-01-31T21:56:19.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:19 smithi171 conmon[46715]: debug 2022-01-31T21:56:19.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:19.003408+0000) 2022-01-31T21:56:19.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:18 smithi171 conmon[51620]: debug 2022-01-31T21:56:18.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:18.848006+0000) 2022-01-31T21:56:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:19 smithi171 conmon[41853]: debug 2022-01-31T21:56:19.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:19.208135+0000) 2022-01-31T21:56:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:19 smithi167 conmon[54076]: debug 2022-01-31T21:56:19.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:19.494098+0000) 2022-01-31T21:56:19.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:19 smithi167 conmon[60316]: debug 2022-01-31T21:56:19.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:19.493981+0000) 2022-01-31T21:56:19.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:19 smithi167 conmon[49112]: debug 2022-01-31T21:56:19.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:19.830542+0000) 2022-01-31T21:56:20.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:20 smithi171 conmon[46715]: debug 2022-01-31T21:56:20.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:20.003599+0000) 2022-01-31T21:56:20.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:19 smithi171 conmon[51620]: debug 2022-01-31T21:56:19.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:19.848223+0000) 2022-01-31T21:56:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:20 smithi171 conmon[41853]: debug 2022-01-31T21:56:20.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:20.208263+0000) 2022-01-31T21:56:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:20 smithi167 conmon[54076]: debug 2022-01-31T21:56:20.493+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:20.494305+0000) 2022-01-31T21:56:20.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:20 smithi167 conmon[60316]: debug 2022-01-31T21:56:20.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:20.494190+0000) 2022-01-31T21:56:20.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:20 smithi167 conmon[49112]: debug 2022-01-31T21:56:20.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:20.830714+0000) 2022-01-31T21:56:21.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[46715]: debug 2022-01-31T21:56:21.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.003803+0000) 2022-01-31T21:56:21.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:20 smithi171 conmon[51620]: debug 2022-01-31T21:56:20.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:20.848457+0000) 2022-01-31T21:56:21.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[41853]: debug 2022-01-31T21:56:21.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.208417+0000) 2022-01-31T21:56:21.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:21 smithi167 conmon[49112]: debug 2022-01-31T21:56:21.527+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.528311+0000) 2022-01-31T21:56:21.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:21 smithi167 conmon[54076]: debug 2022-01-31T21:56:21.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.494493+0000) 2022-01-31T21:56:21.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:21 smithi167 conmon[54076]: debug 2022-01-31T21:56:21.526+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.527404+0000) 2022-01-31T21:56:21.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:21 smithi167 conmon[60316]: debug 2022-01-31T21:56:21.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.494423+0000) 2022-01-31T21:56:21.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:21 smithi167 conmon[60316]: debug 2022-01-31T21:56:21.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.528014+0000) 2022-01-31T21:56:21.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[35325]: debug 2022-01-31T21:56:21.544+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169558 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:21.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[41853]: debug 2022-01-31T21:56:21.525+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.527631+0000) 2022-01-31T21:56:21.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[46715]: debug 2022-01-31T21:56:21.526+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.527949+0000) 2022-01-31T21:56:21.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[51620]: debug 2022-01-31T21:56:21.526+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.528514+0000) 2022-01-31T21:56:21.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:21 smithi167 conmon[49112]: debug 2022-01-31T21:56:21.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.830886+0000) 2022-01-31T21:56:22.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:22 smithi171 conmon[46715]: debug 2022-01-31T21:56:22.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:22.004001+0000) 2022-01-31T21:56:22.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:21 smithi171 conmon[51620]: debug 2022-01-31T21:56:21.847+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:21.848634+0000) 2022-01-31T21:56:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:22 smithi171 conmon[41853]: debug 2022-01-31T21:56:22.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:22.208616+0000) 2022-01-31T21:56:22.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:22 smithi167 conmon[60316]: debug 2022-01-31T21:56:22.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:22.494609+0000) 2022-01-31T21:56:22.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:22 smithi167 conmon[54076]: debug 2022-01-31T21:56:22.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:22.494687+0000) 2022-01-31T21:56:22.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:22 smithi167 conmon[49112]: debug 2022-01-31T21:56:22.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:22.831026+0000) 2022-01-31T21:56:23.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:23 smithi171 conmon[46715]: debug 2022-01-31T21:56:23.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:23.004124+0000) 2022-01-31T21:56:23.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:22 smithi171 conmon[51620]: debug 2022-01-31T21:56:22.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:22.848782+0000) 2022-01-31T21:56:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:23 smithi171 conmon[41853]: debug 2022-01-31T21:56:23.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:23.208826+0000) 2022-01-31T21:56:23.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:23 smithi167 conmon[54076]: debug 2022-01-31T21:56:23.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:23.494841+0000) 2022-01-31T21:56:23.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:23 smithi167 conmon[60316]: debug 2022-01-31T21:56:23.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:23.494762+0000) 2022-01-31T21:56:23.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:23 smithi167 conmon[49112]: debug 2022-01-31T21:56:23.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:23.831223+0000) 2022-01-31T21:56:24.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:24 smithi171 conmon[46715]: debug 2022-01-31T21:56:24.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:24.004326+0000) 2022-01-31T21:56:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:23 smithi171 conmon[35325]: debug 2022-01-31T21:56:23.763+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:56:24.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:23 smithi171 conmon[51620]: debug 2022-01-31T21:56:23.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:23.848917+0000) 2022-01-31T21:56:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:24 smithi171 conmon[41853]: debug 2022-01-31T21:56:24.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:24.208960+0000) 2022-01-31T21:56:24.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:24 smithi167 conmon[54076]: debug 2022-01-31T21:56:24.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:24.495022+0000) 2022-01-31T21:56:24.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:24 smithi167 conmon[60316]: debug 2022-01-31T21:56:24.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:24.494944+0000) 2022-01-31T21:56:24.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:24 smithi167 conmon[49112]: debug 2022-01-31T21:56:24.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:24.831376+0000) 2022-01-31T21:56:25.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:25 smithi171 conmon[46715]: debug 2022-01-31T21:56:25.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:25.004532+0000) 2022-01-31T21:56:25.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:24 smithi171 conmon[51620]: debug 2022-01-31T21:56:24.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:24.849088+0000) 2022-01-31T21:56:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:25 smithi171 conmon[41853]: debug 2022-01-31T21:56:25.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:25.209164+0000) 2022-01-31T21:56:25.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:25 smithi167 conmon[60316]: debug 2022-01-31T21:56:25.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:25.495119+0000) 2022-01-31T21:56:25.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:25 smithi167 conmon[54076]: debug 2022-01-31T21:56:25.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:25.495167+0000) 2022-01-31T21:56:25.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:25 smithi167 conmon[49112]: debug 2022-01-31T21:56:25.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:25.831550+0000) 2022-01-31T21:56:26.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[46715]: debug 2022-01-31T21:56:26.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.004715+0000) 2022-01-31T21:56:26.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:25 smithi171 conmon[51620]: debug 2022-01-31T21:56:25.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:25.849266+0000) 2022-01-31T21:56:26.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[41853]: debug 2022-01-31T21:56:26.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.209317+0000) 2022-01-31T21:56:26.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[41853]: debug 2022-01-31T21:56:26.547+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.548399+0000) 2022-01-31T21:56:26.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[46715]: debug 2022-01-31T21:56:26.547+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.548531+0000) 2022-01-31T21:56:26.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:26 smithi167 conmon[49112]: debug 2022-01-31T21:56:26.555+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.555517+0000) 2022-01-31T21:56:26.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:26 smithi167 conmon[54076]: debug 2022-01-31T21:56:26.494+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.495346+0000) 2022-01-31T21:56:26.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:26 smithi167 conmon[54076]: debug 2022-01-31T21:56:26.554+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.555064+0000) 2022-01-31T21:56:26.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:26 smithi167 conmon[60316]: debug 2022-01-31T21:56:26.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.495312+0000) 2022-01-31T21:56:26.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:26 smithi167 conmon[60316]: debug 2022-01-31T21:56:26.547+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.548228+0000) 2022-01-31T21:56:26.848 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[35325]: debug 2022-01-31T21:56:26.581+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169671 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:26.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[51620]: debug 2022-01-31T21:56:26.549+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.549793+0000) 2022-01-31T21:56:26.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:26 smithi167 conmon[49112]: debug 2022-01-31T21:56:26.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.831754+0000) 2022-01-31T21:56:27.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:27 smithi171 conmon[46715]: debug 2022-01-31T21:56:27.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:27.004925+0000) 2022-01-31T21:56:27.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:26 smithi171 conmon[51620]: debug 2022-01-31T21:56:26.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:26.849481+0000) 2022-01-31T21:56:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:27 smithi171 conmon[41853]: debug 2022-01-31T21:56:27.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:27.209449+0000) 2022-01-31T21:56:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:27 smithi167 conmon[54076]: debug 2022-01-31T21:56:27.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:27.495497+0000) 2022-01-31T21:56:27.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:27 smithi167 conmon[60316]: debug 2022-01-31T21:56:27.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:27.495492+0000) 2022-01-31T21:56:27.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:27 smithi167 conmon[49112]: debug 2022-01-31T21:56:27.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:27.831895+0000) 2022-01-31T21:56:28.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:28 smithi171 conmon[46715]: debug 2022-01-31T21:56:28.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:28.005084+0000) 2022-01-31T21:56:28.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:27 smithi171 conmon[51620]: debug 2022-01-31T21:56:27.848+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:27.849666+0000) 2022-01-31T21:56:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:28 smithi171 conmon[41853]: debug 2022-01-31T21:56:28.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:28.209607+0000) 2022-01-31T21:56:28.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:28 smithi167 conmon[60316]: debug 2022-01-31T21:56:28.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:28.495678+0000) 2022-01-31T21:56:28.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:28 smithi167 conmon[54076]: debug 2022-01-31T21:56:28.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:28.495679+0000) 2022-01-31T21:56:28.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:28 smithi167 conmon[49112]: debug 2022-01-31T21:56:28.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:28.832098+0000) 2022-01-31T21:56:29.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:29 smithi171 conmon[46715]: debug 2022-01-31T21:56:29.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:29.005265+0000) 2022-01-31T21:56:29.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:28 smithi171 conmon[51620]: debug 2022-01-31T21:56:28.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:28.849866+0000) 2022-01-31T21:56:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:29 smithi171 conmon[41853]: debug 2022-01-31T21:56:29.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:29.209800+0000) 2022-01-31T21:56:29.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:29 smithi167 conmon[60316]: debug 2022-01-31T21:56:29.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:29.495877+0000) 2022-01-31T21:56:29.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:29 smithi167 conmon[54076]: debug 2022-01-31T21:56:29.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:29.495879+0000) 2022-01-31T21:56:29.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:29 smithi167 conmon[49112]: debug 2022-01-31T21:56:29.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:29.832297+0000) 2022-01-31T21:56:30.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:30 smithi171 conmon[46715]: debug 2022-01-31T21:56:30.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:30.005415+0000) 2022-01-31T21:56:30.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:29 smithi171 conmon[51620]: debug 2022-01-31T21:56:29.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:29.850093+0000) 2022-01-31T21:56:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:30 smithi171 conmon[41853]: debug 2022-01-31T21:56:30.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:30.209955+0000) 2022-01-31T21:56:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:30 smithi167 conmon[54076]: debug 2022-01-31T21:56:30.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:30.496124+0000) 2022-01-31T21:56:30.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:30 smithi167 conmon[60316]: debug 2022-01-31T21:56:30.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:30.496102+0000) 2022-01-31T21:56:30.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:30 smithi167 conmon[49112]: debug 2022-01-31T21:56:30.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:30.832535+0000) 2022-01-31T21:56:31.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[46715]: debug 2022-01-31T21:56:31.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.005652+0000) 2022-01-31T21:56:31.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:30 smithi171 conmon[51620]: debug 2022-01-31T21:56:30.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:30.850364+0000) 2022-01-31T21:56:31.582 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[41853]: debug 2022-01-31T21:56:31.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.210113+0000) 2022-01-31T21:56:31.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:31 smithi167 conmon[49112]: debug 2022-01-31T21:56:31.585+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.586438+0000) 2022-01-31T21:56:31.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:31 smithi167 conmon[54076]: debug 2022-01-31T21:56:31.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.496361+0000) 2022-01-31T21:56:31.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:31 smithi167 conmon[54076]: debug 2022-01-31T21:56:31.585+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.585563+0000) 2022-01-31T21:56:31.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:31 smithi167 conmon[60316]: debug 2022-01-31T21:56:31.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.496366+0000) 2022-01-31T21:56:31.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:31 smithi167 conmon[60316]: debug 2022-01-31T21:56:31.583+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.584103+0000) 2022-01-31T21:56:31.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[35325]: debug 2022-01-31T21:56:31.600+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169779 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:31.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[41853]: debug 2022-01-31T21:56:31.584+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.584851+0000) 2022-01-31T21:56:31.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[46715]: debug 2022-01-31T21:56:31.584+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.585191+0000) 2022-01-31T21:56:31.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[51620]: debug 2022-01-31T21:56:31.584+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.585522+0000) 2022-01-31T21:56:31.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:31 smithi167 conmon[49112]: debug 2022-01-31T21:56:31.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.832753+0000) 2022-01-31T21:56:32.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:32 smithi171 conmon[46715]: debug 2022-01-31T21:56:32.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:32.005814+0000) 2022-01-31T21:56:32.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:31 smithi171 conmon[51620]: debug 2022-01-31T21:56:31.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:31.850482+0000) 2022-01-31T21:56:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:32 smithi171 conmon[41853]: debug 2022-01-31T21:56:32.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:32.210284+0000) 2022-01-31T21:56:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:32 smithi167 conmon[54076]: debug 2022-01-31T21:56:32.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:32.496544+0000) 2022-01-31T21:56:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:32 smithi167 conmon[60316]: debug 2022-01-31T21:56:32.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:32.496510+0000) 2022-01-31T21:56:32.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:32 smithi167 conmon[49112]: debug 2022-01-31T21:56:32.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:32.832900+0000) 2022-01-31T21:56:33.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:33 smithi171 conmon[46715]: debug 2022-01-31T21:56:33.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:33.005996+0000) 2022-01-31T21:56:33.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:32 smithi171 conmon[51620]: debug 2022-01-31T21:56:32.849+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:32.850643+0000) 2022-01-31T21:56:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:33 smithi171 conmon[41853]: debug 2022-01-31T21:56:33.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:33.210446+0000) 2022-01-31T21:56:33.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:33 smithi167 conmon[54076]: debug 2022-01-31T21:56:33.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:33.496681+0000) 2022-01-31T21:56:33.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:33 smithi167 conmon[60316]: debug 2022-01-31T21:56:33.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:33.496659+0000) 2022-01-31T21:56:33.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:33 smithi167 conmon[49112]: debug 2022-01-31T21:56:33.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:33.833207+0000) 2022-01-31T21:56:34.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:34 smithi171 conmon[46715]: debug 2022-01-31T21:56:34.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:34.006221+0000) 2022-01-31T21:56:34.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:33 smithi171 conmon[51620]: debug 2022-01-31T21:56:33.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:33.850879+0000) 2022-01-31T21:56:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:34 smithi171 conmon[41853]: debug 2022-01-31T21:56:34.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:34.210610+0000) 2022-01-31T21:56:34.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:34 smithi167 conmon[60316]: debug 2022-01-31T21:56:34.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:34.496827+0000) 2022-01-31T21:56:34.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:34 smithi167 conmon[54076]: debug 2022-01-31T21:56:34.495+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:34.496867+0000) 2022-01-31T21:56:34.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:34 smithi167 conmon[49112]: debug 2022-01-31T21:56:34.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:34.833404+0000) 2022-01-31T21:56:35.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:35 smithi171 conmon[46715]: debug 2022-01-31T21:56:35.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:35.006421+0000) 2022-01-31T21:56:35.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:34 smithi171 conmon[51620]: debug 2022-01-31T21:56:34.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:34.851054+0000) 2022-01-31T21:56:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:35 smithi171 conmon[41853]: debug 2022-01-31T21:56:35.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:35.210773+0000) 2022-01-31T21:56:35.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:35 smithi167 conmon[60316]: debug 2022-01-31T21:56:35.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:35.497026+0000) 2022-01-31T21:56:35.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:35 smithi167 conmon[54076]: debug 2022-01-31T21:56:35.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:35.497055+0000) 2022-01-31T21:56:35.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:35 smithi167 conmon[49112]: debug 2022-01-31T21:56:35.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:35.833619+0000) 2022-01-31T21:56:36.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[46715]: debug 2022-01-31T21:56:36.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.006602+0000) 2022-01-31T21:56:36.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:35 smithi171 conmon[51620]: debug 2022-01-31T21:56:35.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:35.851219+0000) 2022-01-31T21:56:36.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[41853]: debug 2022-01-31T21:56:36.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.210931+0000) 2022-01-31T21:56:36.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:36 smithi167 conmon[49112]: debug 2022-01-31T21:56:36.602+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.604302+0000) 2022-01-31T21:56:36.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:36 smithi167 conmon[54076]: debug 2022-01-31T21:56:36.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.497261+0000) 2022-01-31T21:56:36.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:36 smithi167 conmon[54076]: debug 2022-01-31T21:56:36.602+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.604200+0000) 2022-01-31T21:56:36.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:36 smithi167 conmon[60316]: debug 2022-01-31T21:56:36.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.497260+0000) 2022-01-31T21:56:36.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:36 smithi167 conmon[60316]: debug 2022-01-31T21:56:36.601+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.603286+0000) 2022-01-31T21:56:36.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[35325]: debug 2022-01-31T21:56:36.621+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169885 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:36.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[46715]: debug 2022-01-31T21:56:36.604+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.605074+0000) 2022-01-31T21:56:36.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[51620]: debug 2022-01-31T21:56:36.603+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.604421+0000) 2022-01-31T21:56:36.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[51620]: debug 2022-01-31T21:56:36.850+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.851425+0000) 2022-01-31T21:56:36.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:36 smithi171 conmon[41853]: debug 2022-01-31T21:56:36.602+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.603589+0000) 2022-01-31T21:56:36.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:36 smithi167 conmon[49112]: debug 2022-01-31T21:56:36.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:36.833793+0000) 2022-01-31T21:56:37.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:37 smithi171 conmon[46715]: debug 2022-01-31T21:56:37.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:37.006809+0000) 2022-01-31T21:56:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:37 smithi171 conmon[41853]: debug 2022-01-31T21:56:37.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:37.211117+0000) 2022-01-31T21:56:37.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:37 smithi167 conmon[54076]: debug 2022-01-31T21:56:37.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:37.497480+0000) 2022-01-31T21:56:37.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:37 smithi167 conmon[60316]: debug 2022-01-31T21:56:37.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:37.497479+0000) 2022-01-31T21:56:37.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:37 smithi167 conmon[49112]: debug 2022-01-31T21:56:37.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:37.834032+0000) 2022-01-31T21:56:38.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:38 smithi171 conmon[46715]: debug 2022-01-31T21:56:38.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:38.006985+0000) 2022-01-31T21:56:38.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:37 smithi171 conmon[51620]: debug 2022-01-31T21:56:37.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:37.851619+0000) 2022-01-31T21:56:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:38 smithi171 conmon[41853]: debug 2022-01-31T21:56:38.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:38.211267+0000) 2022-01-31T21:56:38.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:38 smithi167 conmon[54076]: debug 2022-01-31T21:56:38.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:38.497650+0000) 2022-01-31T21:56:38.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:38 smithi167 conmon[60316]: debug 2022-01-31T21:56:38.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:38.497650+0000) 2022-01-31T21:56:38.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:38 smithi167 conmon[49112]: debug 2022-01-31T21:56:38.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:38.834177+0000) 2022-01-31T21:56:39.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:39 smithi171 conmon[46715]: debug 2022-01-31T21:56:39.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:39.007190+0000) 2022-01-31T21:56:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:38 smithi171 conmon[51620]: debug 2022-01-31T21:56:38.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:38.851836+0000) 2022-01-31T21:56:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:38 smithi171 conmon[35325]: debug 2022-01-31T21:56:38.764+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:56:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:39 smithi171 conmon[41853]: debug 2022-01-31T21:56:39.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:39.211470+0000) 2022-01-31T21:56:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:39 smithi167 conmon[54076]: debug 2022-01-31T21:56:39.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:39.497850+0000) 2022-01-31T21:56:39.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:39 smithi167 conmon[60316]: debug 2022-01-31T21:56:39.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:39.497827+0000) 2022-01-31T21:56:39.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:39 smithi167 conmon[49112]: debug 2022-01-31T21:56:39.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:39.834363+0000) 2022-01-31T21:56:40.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:39 smithi171 conmon[51620]: debug 2022-01-31T21:56:39.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:39.852042+0000) 2022-01-31T21:56:40.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:40 smithi171 conmon[46715]: debug 2022-01-31T21:56:40.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:40.007373+0000) 2022-01-31T21:56:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:40 smithi171 conmon[41853]: debug 2022-01-31T21:56:40.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:40.211651+0000) 2022-01-31T21:56:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:40 smithi167 conmon[60316]: debug 2022-01-31T21:56:40.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:40.498015+0000) 2022-01-31T21:56:40.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:40 smithi167 conmon[54076]: debug 2022-01-31T21:56:40.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:40.498021+0000) 2022-01-31T21:56:40.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:40 smithi167 conmon[49112]: debug 2022-01-31T21:56:40.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:40.834536+0000) 2022-01-31T21:56:41.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:40 smithi171 conmon[51620]: debug 2022-01-31T21:56:40.851+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:40.852238+0000) 2022-01-31T21:56:41.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[46715]: debug 2022-01-31T21:56:41.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.007558+0000) 2022-01-31T21:56:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[41853]: debug 2022-01-31T21:56:41.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.211821+0000) 2022-01-31T21:56:41.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:41 smithi167 conmon[49112]: debug 2022-01-31T21:56:41.623+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.625379+0000) 2022-01-31T21:56:41.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:41 smithi167 conmon[54076]: debug 2022-01-31T21:56:41.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.498212+0000) 2022-01-31T21:56:41.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:41 smithi167 conmon[54076]: debug 2022-01-31T21:56:41.622+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.624286+0000) 2022-01-31T21:56:41.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:41 smithi167 conmon[60316]: debug 2022-01-31T21:56:41.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.498190+0000) 2022-01-31T21:56:41.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:41 smithi167 conmon[60316]: debug 2022-01-31T21:56:41.623+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.625014+0000) 2022-01-31T21:56:41.931 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[41853]: debug 2022-01-31T21:56:41.622+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.623713+0000) 2022-01-31T21:56:41.932 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[46715]: debug 2022-01-31T21:56:41.624+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.624838+0000) 2022-01-31T21:56:41.933 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[35325]: debug 2022-01-31T21:56:41.641+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 169993 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:41.933 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[51620]: debug 2022-01-31T21:56:41.624+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.624985+0000) 2022-01-31T21:56:41.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:41 smithi171 conmon[51620]: debug 2022-01-31T21:56:41.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.852428+0000) 2022-01-31T21:56:41.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:41 smithi167 conmon[49112]: debug 2022-01-31T21:56:41.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:41.834660+0000) 2022-01-31T21:56:42.210 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:42 smithi171 conmon[46715]: debug 2022-01-31T21:56:42.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:42.007743+0000) 2022-01-31T21:56:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:42 smithi171 conmon[41853]: debug 2022-01-31T21:56:42.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:42.211969+0000) 2022-01-31T21:56:42.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:42 smithi167 conmon[54076]: debug 2022-01-31T21:56:42.496+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:42.498394+0000) 2022-01-31T21:56:42.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:42 smithi167 conmon[60316]: debug 2022-01-31T21:56:42.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:42.498392+0000) 2022-01-31T21:56:42.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:42 smithi167 conmon[49112]: debug 2022-01-31T21:56:42.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:42.834807+0000) 2022-01-31T21:56:43.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:42 smithi171 conmon[51620]: debug 2022-01-31T21:56:42.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:42.852599+0000) 2022-01-31T21:56:43.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:43 smithi171 conmon[46715]: debug 2022-01-31T21:56:43.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:43.007935+0000) 2022-01-31T21:56:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:43 smithi171 conmon[41853]: debug 2022-01-31T21:56:43.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:43.212125+0000) 2022-01-31T21:56:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:43 smithi167 conmon[54076]: debug 2022-01-31T21:56:43.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:43.498568+0000) 2022-01-31T21:56:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:43 smithi167 conmon[60316]: debug 2022-01-31T21:56:43.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:43.498528+0000) 2022-01-31T21:56:43.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:43 smithi167 conmon[49112]: debug 2022-01-31T21:56:43.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:43.835007+0000) 2022-01-31T21:56:44.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:44 smithi171 conmon[46715]: debug 2022-01-31T21:56:44.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:44.008160+0000) 2022-01-31T21:56:44.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:43 smithi171 conmon[51620]: debug 2022-01-31T21:56:43.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:43.852791+0000) 2022-01-31T21:56:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:44 smithi171 conmon[41853]: debug 2022-01-31T21:56:44.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:44.212285+0000) 2022-01-31T21:56:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:44 smithi167 conmon[54076]: debug 2022-01-31T21:56:44.497+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:44.498725+0000) 2022-01-31T21:56:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:44 smithi167 conmon[60316]: debug 2022-01-31T21:56:44.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:44.498700+0000) 2022-01-31T21:56:44.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:44 smithi167 conmon[49112]: debug 2022-01-31T21:56:44.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:44.835216+0000) 2022-01-31T21:56:45.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:45 smithi171 conmon[46715]: debug 2022-01-31T21:56:45.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:45.008366+0000) 2022-01-31T21:56:45.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:44 smithi171 conmon[51620]: debug 2022-01-31T21:56:44.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:44.852988+0000) 2022-01-31T21:56:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:45 smithi171 conmon[41853]: debug 2022-01-31T21:56:45.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:45.212465+0000) 2022-01-31T21:56:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:45 smithi167 conmon[54076]: debug 2022-01-31T21:56:45.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:45.498885+0000) 2022-01-31T21:56:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:45 smithi167 conmon[60316]: debug 2022-01-31T21:56:45.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:45.498862+0000) 2022-01-31T21:56:45.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:45 smithi167 conmon[49112]: debug 2022-01-31T21:56:45.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:45.835414+0000) 2022-01-31T21:56:46.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[46715]: debug 2022-01-31T21:56:46.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.008546+0000) 2022-01-31T21:56:46.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:45 smithi171 conmon[51620]: debug 2022-01-31T21:56:45.852+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:45.853203+0000) 2022-01-31T21:56:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[41853]: debug 2022-01-31T21:56:46.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.212622+0000) 2022-01-31T21:56:46.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:46 smithi167 conmon[49112]: debug 2022-01-31T21:56:46.643+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.644972+0000) 2022-01-31T21:56:46.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:46 smithi167 conmon[54076]: debug 2022-01-31T21:56:46.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.499069+0000) 2022-01-31T21:56:46.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:46 smithi167 conmon[54076]: debug 2022-01-31T21:56:46.643+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.644872+0000) 2022-01-31T21:56:46.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:46 smithi167 conmon[60316]: debug 2022-01-31T21:56:46.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.499043+0000) 2022-01-31T21:56:46.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:46 smithi167 conmon[60316]: debug 2022-01-31T21:56:46.643+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.644616+0000) 2022-01-31T21:56:46.932 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[35325]: debug 2022-01-31T21:56:46.661+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170106 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:46.933 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[41853]: debug 2022-01-31T21:56:46.643+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.644257+0000) 2022-01-31T21:56:46.934 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[46715]: debug 2022-01-31T21:56:46.644+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.645087+0000) 2022-01-31T21:56:46.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[51620]: debug 2022-01-31T21:56:46.644+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.645468+0000) 2022-01-31T21:56:46.935 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:46 smithi171 conmon[51620]: debug 2022-01-31T21:56:46.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.853386+0000) 2022-01-31T21:56:46.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:46 smithi167 conmon[49112]: debug 2022-01-31T21:56:46.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:46.835600+0000) 2022-01-31T21:56:47.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:47 smithi171 conmon[46715]: debug 2022-01-31T21:56:47.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:47.008704+0000) 2022-01-31T21:56:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:47 smithi171 conmon[41853]: debug 2022-01-31T21:56:47.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:47.212803+0000) 2022-01-31T21:56:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:47 smithi167 conmon[54076]: debug 2022-01-31T21:56:47.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:47.499324+0000) 2022-01-31T21:56:47.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:47 smithi167 conmon[60316]: debug 2022-01-31T21:56:47.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:47.499238+0000) 2022-01-31T21:56:47.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:47 smithi167 conmon[49112]: debug 2022-01-31T21:56:47.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:47.835714+0000) 2022-01-31T21:56:48.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:47 smithi171 conmon[51620]: debug 2022-01-31T21:56:47.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:47.853535+0000) 2022-01-31T21:56:48.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:48 smithi171 conmon[46715]: debug 2022-01-31T21:56:48.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:48.008875+0000) 2022-01-31T21:56:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:48 smithi171 conmon[41853]: debug 2022-01-31T21:56:48.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:48.212952+0000) 2022-01-31T21:56:48.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:48 smithi167 conmon[54076]: debug 2022-01-31T21:56:48.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:48.499415+0000) 2022-01-31T21:56:48.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:48 smithi167 conmon[60316]: debug 2022-01-31T21:56:48.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:48.499537+0000) 2022-01-31T21:56:48.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:48 smithi167 conmon[49112]: debug 2022-01-31T21:56:48.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:48.835880+0000) 2022-01-31T21:56:49.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:49 smithi171 conmon[46715]: debug 2022-01-31T21:56:49.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:49.009035+0000) 2022-01-31T21:56:49.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:48 smithi171 conmon[51620]: debug 2022-01-31T21:56:48.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:48.853775+0000) 2022-01-31T21:56:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:49 smithi171 conmon[41853]: debug 2022-01-31T21:56:49.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:49.213128+0000) 2022-01-31T21:56:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:49 smithi167 conmon[54076]: debug 2022-01-31T21:56:49.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:49.499579+0000) 2022-01-31T21:56:49.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:49 smithi167 conmon[60316]: debug 2022-01-31T21:56:49.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:49.499714+0000) 2022-01-31T21:56:49.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:49 smithi167 conmon[49112]: debug 2022-01-31T21:56:49.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:49.836100+0000) 2022-01-31T21:56:50.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:49 smithi171 conmon[51620]: debug 2022-01-31T21:56:49.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:49.853969+0000) 2022-01-31T21:56:50.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:50 smithi171 conmon[46715]: debug 2022-01-31T21:56:50.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:50.009216+0000) 2022-01-31T21:56:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:50 smithi171 conmon[41853]: debug 2022-01-31T21:56:50.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:50.213316+0000) 2022-01-31T21:56:50.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:50 smithi167 conmon[54076]: debug 2022-01-31T21:56:50.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:50.499728+0000) 2022-01-31T21:56:50.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:50 smithi167 conmon[60316]: debug 2022-01-31T21:56:50.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:50.499869+0000) 2022-01-31T21:56:50.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:50 smithi167 conmon[49112]: debug 2022-01-31T21:56:50.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:50.836292+0000) 2022-01-31T21:56:51.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[46715]: debug 2022-01-31T21:56:51.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.009347+0000) 2022-01-31T21:56:51.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:50 smithi171 conmon[51620]: debug 2022-01-31T21:56:50.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:50.854130+0000) 2022-01-31T21:56:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[41853]: debug 2022-01-31T21:56:51.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.213517+0000) 2022-01-31T21:56:51.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:51 smithi167 conmon[54076]: debug 2022-01-31T21:56:51.498+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.499955+0000) 2022-01-31T21:56:51.664 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:51 smithi167 conmon[60316]: debug 2022-01-31T21:56:51.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.500022+0000) 2022-01-31T21:56:51.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:51 smithi167 conmon[60316]: debug 2022-01-31T21:56:51.663+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.664520+0000) 2022-01-31T21:56:51.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:51 smithi167 conmon[54076]: debug 2022-01-31T21:56:51.663+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.665365+0000) 2022-01-31T21:56:51.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:51 smithi167 conmon[49112]: debug 2022-01-31T21:56:51.663+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.664708+0000) 2022-01-31T21:56:51.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:51 smithi167 conmon[49112]: debug 2022-01-31T21:56:51.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.836519+0000) 2022-01-31T21:56:51.933 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[46715]: debug 2022-01-31T21:56:51.663+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.664668+0000) 2022-01-31T21:56:51.934 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[51620]: debug 2022-01-31T21:56:51.664+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.665292+0000) 2022-01-31T21:56:51.935 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[51620]: debug 2022-01-31T21:56:51.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.854321+0000) 2022-01-31T21:56:51.935 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[35325]: debug 2022-01-31T21:56:51.682+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170215 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:51.936 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:51 smithi171 conmon[41853]: debug 2022-01-31T21:56:51.664+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:51.665541+0000) 2022-01-31T21:56:52.212 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:52 smithi171 conmon[46715]: debug 2022-01-31T21:56:52.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:52.009478+0000) 2022-01-31T21:56:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:52 smithi171 conmon[41853]: debug 2022-01-31T21:56:52.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:52.213694+0000) 2022-01-31T21:56:52.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:52 smithi167 conmon[54076]: debug 2022-01-31T21:56:52.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:52.500124+0000) 2022-01-31T21:56:52.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:52 smithi167 conmon[60316]: debug 2022-01-31T21:56:52.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:52.500207+0000) 2022-01-31T21:56:52.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:52 smithi167 conmon[49112]: debug 2022-01-31T21:56:52.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:52.836668+0000) 2022-01-31T21:56:53.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:53 smithi171 conmon[46715]: debug 2022-01-31T21:56:53.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:53.009612+0000) 2022-01-31T21:56:53.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:52 smithi171 conmon[51620]: debug 2022-01-31T21:56:52.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:52.854491+0000) 2022-01-31T21:56:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:53 smithi171 conmon[41853]: debug 2022-01-31T21:56:53.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:53.213848+0000) 2022-01-31T21:56:53.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:53 smithi167 conmon[54076]: debug 2022-01-31T21:56:53.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:53.500259+0000) 2022-01-31T21:56:53.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:53 smithi167 conmon[60316]: debug 2022-01-31T21:56:53.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:53.500345+0000) 2022-01-31T21:56:53.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:53 smithi167 conmon[49112]: debug 2022-01-31T21:56:53.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:53.836863+0000) 2022-01-31T21:56:54.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:54 smithi171 conmon[46715]: debug 2022-01-31T21:56:54.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:54.009810+0000) 2022-01-31T21:56:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:53 smithi171 conmon[51620]: debug 2022-01-31T21:56:53.853+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:53.854711+0000) 2022-01-31T21:56:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:53 smithi171 conmon[35325]: debug 2022-01-31T21:56:53.765+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:56:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:54 smithi171 conmon[41853]: debug 2022-01-31T21:56:54.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:54.214013+0000) 2022-01-31T21:56:54.705 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:54 smithi167 conmon[60316]: debug 2022-01-31T21:56:54.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:54.500546+0000) 2022-01-31T21:56:54.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:54 smithi167 conmon[54076]: debug 2022-01-31T21:56:54.499+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:54.500446+0000) 2022-01-31T21:56:54.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:54 smithi167 conmon[49112]: debug 2022-01-31T21:56:54.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:54.837047+0000) 2022-01-31T21:56:55.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:54 smithi171 conmon[51620]: debug 2022-01-31T21:56:54.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:54.854947+0000) 2022-01-31T21:56:55.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:55 smithi171 conmon[41853]: debug 2022-01-31T21:56:55.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:55.214169+0000) 2022-01-31T21:56:55.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:55 smithi171 conmon[46715]: debug 2022-01-31T21:56:55.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:55.009957+0000) 2022-01-31T21:56:55.705 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:55 smithi167 conmon[60316]: debug 2022-01-31T21:56:55.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:55.500720+0000) 2022-01-31T21:56:55.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:55 smithi167 conmon[54076]: debug 2022-01-31T21:56:55.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:55.500660+0000) 2022-01-31T21:56:55.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:55 smithi167 conmon[49112]: debug 2022-01-31T21:56:55.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:55.837208+0000) 2022-01-31T21:56:56.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[46715]: debug 2022-01-31T21:56:56.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.010133+0000) 2022-01-31T21:56:56.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:55 smithi171 conmon[51620]: debug 2022-01-31T21:56:55.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:55.855172+0000) 2022-01-31T21:56:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[41853]: debug 2022-01-31T21:56:56.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.214333+0000) 2022-01-31T21:56:56.683 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:56 smithi167 conmon[54076]: debug 2022-01-31T21:56:56.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.500819+0000) 2022-01-31T21:56:56.684 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:56 smithi167 conmon[60316]: debug 2022-01-31T21:56:56.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.500874+0000) 2022-01-31T21:56:56.935 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[35325]: debug 2022-01-31T21:56:56.701+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170328 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:56:56.935 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[41853]: debug 2022-01-31T21:56:56.683+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.685447+0000) 2022-01-31T21:56:56.936 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[46715]: debug 2022-01-31T21:56:56.684+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.686008+0000) 2022-01-31T21:56:56.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[51620]: debug 2022-01-31T21:56:56.683+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.685220+0000) 2022-01-31T21:56:56.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:56 smithi171 conmon[51620]: debug 2022-01-31T21:56:56.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.855370+0000) 2022-01-31T21:56:56.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:56 smithi167 conmon[54076]: debug 2022-01-31T21:56:56.684+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.686481+0000) 2022-01-31T21:56:56.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:56 smithi167 conmon[60316]: debug 2022-01-31T21:56:56.683+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.684842+0000) 2022-01-31T21:56:56.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:56 smithi167 conmon[49112]: debug 2022-01-31T21:56:56.683+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.684681+0000) 2022-01-31T21:56:56.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:56 smithi167 conmon[49112]: debug 2022-01-31T21:56:56.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:56.837409+0000) 2022-01-31T21:56:57.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:57 smithi171 conmon[46715]: debug 2022-01-31T21:56:57.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:57.010324+0000) 2022-01-31T21:56:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:57 smithi171 conmon[41853]: debug 2022-01-31T21:56:57.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:57.214492+0000) 2022-01-31T21:56:57.705 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:57 smithi167 conmon[54076]: debug 2022-01-31T21:56:57.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:57.501012+0000) 2022-01-31T21:56:57.706 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:57 smithi167 conmon[60316]: debug 2022-01-31T21:56:57.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:57.501118+0000) 2022-01-31T21:56:57.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:57 smithi167 conmon[49112]: debug 2022-01-31T21:56:57.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:57.837580+0000) 2022-01-31T21:56:58.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:58 smithi171 conmon[46715]: debug 2022-01-31T21:56:58.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:58.010424+0000) 2022-01-31T21:56:58.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:57 smithi171 conmon[51620]: debug 2022-01-31T21:56:57.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:57.855532+0000) 2022-01-31T21:56:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:58 smithi171 conmon[41853]: debug 2022-01-31T21:56:58.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:58.214613+0000) 2022-01-31T21:56:58.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:58 smithi167 conmon[54076]: debug 2022-01-31T21:56:58.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:58.501164+0000) 2022-01-31T21:56:58.706 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:58 smithi167 conmon[60316]: debug 2022-01-31T21:56:58.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:58.501257+0000) 2022-01-31T21:56:58.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:58 smithi167 conmon[49112]: debug 2022-01-31T21:56:58.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:58.837812+0000) 2022-01-31T21:56:59.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:56:59 smithi171 conmon[46715]: debug 2022-01-31T21:56:59.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:59.010580+0000) 2022-01-31T21:56:59.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:58 smithi171 conmon[51620]: debug 2022-01-31T21:56:58.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:58.855742+0000) 2022-01-31T21:56:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:56:59 smithi171 conmon[41853]: debug 2022-01-31T21:56:59.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:59.214729+0000) 2022-01-31T21:56:59.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:56:59 smithi167 conmon[54076]: debug 2022-01-31T21:56:59.500+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:59.501353+0000) 2022-01-31T21:56:59.706 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:56:59 smithi167 conmon[60316]: debug 2022-01-31T21:56:59.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:59.501446+0000) 2022-01-31T21:56:59.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:56:59 smithi167 conmon[49112]: debug 2022-01-31T21:56:59.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:59.838007+0000) 2022-01-31T21:57:00.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:00 smithi171 conmon[46715]: debug 2022-01-31T21:57:00.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:00.010786+0000) 2022-01-31T21:57:00.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:56:59 smithi171 conmon[51620]: debug 2022-01-31T21:56:59.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:56:59.855989+0000) 2022-01-31T21:57:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:00 smithi171 conmon[41853]: debug 2022-01-31T21:57:00.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:00.214918+0000) 2022-01-31T21:57:00.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:00 smithi167 conmon[54076]: debug 2022-01-31T21:57:00.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:00.501552+0000) 2022-01-31T21:57:00.706 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:00 smithi167 conmon[60316]: debug 2022-01-31T21:57:00.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:00.501602+0000) 2022-01-31T21:57:00.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:00 smithi167 conmon[49112]: debug 2022-01-31T21:57:00.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:00.838163+0000) 2022-01-31T21:57:01.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[46715]: debug 2022-01-31T21:57:01.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.010975+0000) 2022-01-31T21:57:01.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:00 smithi171 conmon[51620]: debug 2022-01-31T21:57:00.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:00.856206+0000) 2022-01-31T21:57:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[41853]: debug 2022-01-31T21:57:01.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.215096+0000) 2022-01-31T21:57:01.704 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:01 smithi167 conmon[54076]: debug 2022-01-31T21:57:01.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.501741+0000) 2022-01-31T21:57:01.705 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:01 smithi167 conmon[60316]: debug 2022-01-31T21:57:01.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.501806+0000) 2022-01-31T21:57:01.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:01 smithi167 conmon[49112]: debug 2022-01-31T21:57:01.705+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.705863+0000) 2022-01-31T21:57:01.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:01 smithi167 conmon[49112]: debug 2022-01-31T21:57:01.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.838356+0000) 2022-01-31T21:57:01.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:01 smithi167 conmon[54076]: debug 2022-01-31T21:57:01.705+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.705645+0000) 2022-01-31T21:57:01.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:01 smithi167 conmon[60316]: debug 2022-01-31T21:57:01.706+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.706516+0000) 2022-01-31T21:57:02.009 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[35325]: debug 2022-01-31T21:57:01.721+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170437 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:02.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[41853]: debug 2022-01-31T21:57:01.703+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.705533+0000) 2022-01-31T21:57:02.010 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[46715]: debug 2022-01-31T21:57:01.705+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.706878+0000) 2022-01-31T21:57:02.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[51620]: debug 2022-01-31T21:57:01.703+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.705643+0000) 2022-01-31T21:57:02.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:01 smithi171 conmon[51620]: debug 2022-01-31T21:57:01.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:01.856425+0000) 2022-01-31T21:57:02.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:02 smithi171 conmon[46715]: debug 2022-01-31T21:57:02.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:02.011163+0000) 2022-01-31T21:57:02.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:02 smithi171 conmon[41853]: debug 2022-01-31T21:57:02.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:02.215277+0000) 2022-01-31T21:57:02.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:02 smithi167 conmon[54076]: debug 2022-01-31T21:57:02.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:02.501944+0000) 2022-01-31T21:57:02.707 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:02 smithi167 conmon[60316]: debug 2022-01-31T21:57:02.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:02.502011+0000) 2022-01-31T21:57:02.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:02 smithi167 conmon[49112]: debug 2022-01-31T21:57:02.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:02.838532+0000) 2022-01-31T21:57:03.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:03 smithi171 conmon[46715]: debug 2022-01-31T21:57:03.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:03.011323+0000) 2022-01-31T21:57:03.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:02 smithi171 conmon[51620]: debug 2022-01-31T21:57:02.854+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:02.856621+0000) 2022-01-31T21:57:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:03 smithi171 conmon[41853]: debug 2022-01-31T21:57:03.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:03.215429+0000) 2022-01-31T21:57:03.706 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:03 smithi167 conmon[54076]: debug 2022-01-31T21:57:03.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:03.502095+0000) 2022-01-31T21:57:03.707 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:03 smithi167 conmon[60316]: debug 2022-01-31T21:57:03.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:03.502158+0000) 2022-01-31T21:57:03.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:03 smithi167 conmon[49112]: debug 2022-01-31T21:57:03.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:03.838663+0000) 2022-01-31T21:57:04.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:04 smithi171 conmon[46715]: debug 2022-01-31T21:57:04.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:04.011519+0000) 2022-01-31T21:57:04.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:03 smithi171 conmon[51620]: debug 2022-01-31T21:57:03.855+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:03.856842+0000) 2022-01-31T21:57:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:04 smithi171 conmon[41853]: debug 2022-01-31T21:57:04.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:04.215611+0000) 2022-01-31T21:57:04.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:04 smithi167 conmon[54076]: debug 2022-01-31T21:57:04.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:04.502284+0000) 2022-01-31T21:57:04.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:04 smithi167 conmon[60316]: debug 2022-01-31T21:57:04.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:04.502367+0000) 2022-01-31T21:57:04.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:04 smithi167 conmon[49112]: debug 2022-01-31T21:57:04.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:04.838770+0000) 2022-01-31T21:57:05.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:05 smithi171 conmon[46715]: debug 2022-01-31T21:57:05.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:05.011678+0000) 2022-01-31T21:57:05.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:04 smithi171 conmon[51620]: debug 2022-01-31T21:57:04.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:04.857032+0000) 2022-01-31T21:57:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:05 smithi171 conmon[41853]: debug 2022-01-31T21:57:05.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:05.215781+0000) 2022-01-31T21:57:05.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:05 smithi167 conmon[54076]: debug 2022-01-31T21:57:05.501+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:05.502446+0000) 2022-01-31T21:57:05.707 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:05 smithi167 conmon[60316]: debug 2022-01-31T21:57:05.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:05.502544+0000) 2022-01-31T21:57:05.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:05 smithi167 conmon[49112]: debug 2022-01-31T21:57:05.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:05.838925+0000) 2022-01-31T21:57:06.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[46715]: debug 2022-01-31T21:57:06.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.011833+0000) 2022-01-31T21:57:06.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:05 smithi171 conmon[51620]: debug 2022-01-31T21:57:05.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:05.857249+0000) 2022-01-31T21:57:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[41853]: debug 2022-01-31T21:57:06.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.215982+0000) 2022-01-31T21:57:06.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:06 smithi167 conmon[54076]: debug 2022-01-31T21:57:06.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.502623+0000) 2022-01-31T21:57:06.707 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:06 smithi167 conmon[60316]: debug 2022-01-31T21:57:06.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.502711+0000) 2022-01-31T21:57:06.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:06 smithi167 conmon[54076]: debug 2022-01-31T21:57:06.725+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.726307+0000) 2022-01-31T21:57:06.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:06 smithi167 conmon[60316]: debug 2022-01-31T21:57:06.726+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.727095+0000) 2022-01-31T21:57:06.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:06 smithi167 conmon[49112]: debug 2022-01-31T21:57:06.725+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.725977+0000) 2022-01-31T21:57:06.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:06 smithi167 conmon[49112]: debug 2022-01-31T21:57:06.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.839146+0000) 2022-01-31T21:57:07.010 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[35325]: debug 2022-01-31T21:57:06.743+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:07.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[41853]: debug 2022-01-31T21:57:06.725+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.726128+0000) 2022-01-31T21:57:07.011 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[46715]: debug 2022-01-31T21:57:06.726+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.726937+0000) 2022-01-31T21:57:07.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[51620]: debug 2022-01-31T21:57:06.724+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.725465+0000) 2022-01-31T21:57:07.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:06 smithi171 conmon[51620]: debug 2022-01-31T21:57:06.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:06.857459+0000) 2022-01-31T21:57:07.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:07 smithi171 conmon[41853]: debug 2022-01-31T21:57:07.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:07.216148+0000) 2022-01-31T21:57:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:07 smithi171 conmon[46715]: debug 2022-01-31T21:57:07.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:07.011990+0000) 2022-01-31T21:57:07.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:07 smithi167 conmon[54076]: debug 2022-01-31T21:57:07.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:07.502831+0000) 2022-01-31T21:57:07.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:07 smithi167 conmon[60316]: debug 2022-01-31T21:57:07.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:07.502916+0000) 2022-01-31T21:57:07.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:07 smithi167 conmon[49112]: debug 2022-01-31T21:57:07.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:07.839335+0000) 2022-01-31T21:57:08.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:08 smithi171 conmon[46715]: debug 2022-01-31T21:57:08.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:08.012129+0000) 2022-01-31T21:57:08.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:07 smithi171 conmon[51620]: debug 2022-01-31T21:57:07.856+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:07.857658+0000) 2022-01-31T21:57:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:08 smithi171 conmon[41853]: debug 2022-01-31T21:57:08.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:08.216299+0000) 2022-01-31T21:57:08.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:08 smithi167 conmon[54076]: debug 2022-01-31T21:57:08.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:08.503000+0000) 2022-01-31T21:57:08.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:08 smithi167 conmon[60316]: debug 2022-01-31T21:57:08.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:08.503066+0000) 2022-01-31T21:57:08.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:08 smithi167 conmon[49112]: debug 2022-01-31T21:57:08.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:08.839558+0000) 2022-01-31T21:57:09.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:08 smithi171 conmon[51620]: debug 2022-01-31T21:57:08.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:08.857832+0000) 2022-01-31T21:57:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:08 smithi171 conmon[35325]: debug 2022-01-31T21:57:08.765+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:57:09.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:09 smithi171 conmon[46715]: debug 2022-01-31T21:57:09.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:09.012328+0000) 2022-01-31T21:57:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:09 smithi171 conmon[41853]: debug 2022-01-31T21:57:09.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:09.216518+0000) 2022-01-31T21:57:09.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:09 smithi167 conmon[54076]: debug 2022-01-31T21:57:09.502+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:09.503212+0000) 2022-01-31T21:57:09.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:09 smithi167 conmon[60316]: debug 2022-01-31T21:57:09.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:09.503250+0000) 2022-01-31T21:57:09.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:09 smithi167 conmon[49112]: debug 2022-01-31T21:57:09.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:09.839695+0000) 2022-01-31T21:57:10.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:10 smithi171 conmon[46715]: debug 2022-01-31T21:57:10.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:10.012519+0000) 2022-01-31T21:57:10.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:09 smithi171 conmon[51620]: debug 2022-01-31T21:57:09.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:09.858024+0000) 2022-01-31T21:57:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:10 smithi171 conmon[41853]: debug 2022-01-31T21:57:10.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:10.216722+0000) 2022-01-31T21:57:10.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:10 smithi167 conmon[54076]: debug 2022-01-31T21:57:10.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:10.503413+0000) 2022-01-31T21:57:10.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:10 smithi167 conmon[60316]: debug 2022-01-31T21:57:10.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:10.503451+0000) 2022-01-31T21:57:10.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:10 smithi167 conmon[49112]: debug 2022-01-31T21:57:10.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:10.839840+0000) 2022-01-31T21:57:11.171 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[46715]: debug 2022-01-31T21:57:11.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.012680+0000) 2022-01-31T21:57:11.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:10 smithi171 conmon[51620]: debug 2022-01-31T21:57:10.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:10.858234+0000) 2022-01-31T21:57:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[41853]: debug 2022-01-31T21:57:11.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.216872+0000) 2022-01-31T21:57:11.708 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:11 smithi167 conmon[54076]: debug 2022-01-31T21:57:11.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.503603+0000) 2022-01-31T21:57:11.709 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:11 smithi167 conmon[60316]: debug 2022-01-31T21:57:11.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.503669+0000) 2022-01-31T21:57:11.937 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[35325]: debug 2022-01-31T21:57:11.763+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170658 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:11.938 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[41853]: debug 2022-01-31T21:57:11.746+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.746770+0000) 2022-01-31T21:57:11.939 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[46715]: debug 2022-01-31T21:57:11.746+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.747438+0000) 2022-01-31T21:57:11.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[51620]: debug 2022-01-31T21:57:11.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[51620]: 2022-01-31T21:57:11.744+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.745579+0000) 2022-01-31T21:57:11.940 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:11 smithi171 conmon[51620]: debug 2022-01-31T21:57:11.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.858438+0000) 2022-01-31T21:57:11.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:11 smithi167 conmon[54076]: debug 2022-01-31T21:57:11.746+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.747359+0000) 2022-01-31T21:57:11.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:11 smithi167 conmon[60316]: debug 2022-01-31T21:57:11.746+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.746955+0000) 2022-01-31T21:57:11.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:11 smithi167 conmon[49112]: debug 2022-01-31T21:57:11.746+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.747005+0000) 2022-01-31T21:57:12.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:11 smithi167 conmon[49112]: debug 2022-01-31T21:57:11.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:11.840061+0000) 2022-01-31T21:57:12.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:12 smithi171 conmon[46715]: debug 2022-01-31T21:57:12.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:12.012861+0000) 2022-01-31T21:57:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:12 smithi171 conmon[41853]: debug 2022-01-31T21:57:12.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:12.217026+0000) 2022-01-31T21:57:12.708 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:12 smithi167 conmon[54076]: debug 2022-01-31T21:57:12.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:12.503762+0000) 2022-01-31T21:57:12.709 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:12 smithi167 conmon[60316]: debug 2022-01-31T21:57:12.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:12.503865+0000) 2022-01-31T21:57:12.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:12 smithi167 conmon[49112]: debug 2022-01-31T21:57:12.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:12.840220+0000) 2022-01-31T21:57:13.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:13 smithi171 conmon[46715]: debug 2022-01-31T21:57:13.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:13.012939+0000) 2022-01-31T21:57:13.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:12 smithi171 conmon[51620]: debug 2022-01-31T21:57:12.857+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:12.858607+0000) 2022-01-31T21:57:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:13 smithi171 conmon[41853]: debug 2022-01-31T21:57:13.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:13.217219+0000) 2022-01-31T21:57:13.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:13 smithi167 conmon[54076]: debug 2022-01-31T21:57:13.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:13.503889+0000) 2022-01-31T21:57:13.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:13 smithi167 conmon[60316]: debug 2022-01-31T21:57:13.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:13.503964+0000) 2022-01-31T21:57:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:13 smithi167 conmon[49112]: debug 2022-01-31T21:57:13.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:13.840438+0000) 2022-01-31T21:57:14.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:14 smithi171 conmon[46715]: debug 2022-01-31T21:57:14.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:14.013122+0000) 2022-01-31T21:57:14.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:13 smithi171 conmon[51620]: debug 2022-01-31T21:57:13.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:13.858843+0000) 2022-01-31T21:57:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:14 smithi171 conmon[41853]: debug 2022-01-31T21:57:14.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:14.217408+0000) 2022-01-31T21:57:14.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:14 smithi167 conmon[54076]: debug 2022-01-31T21:57:14.503+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:14.504065+0000) 2022-01-31T21:57:14.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:14 smithi167 conmon[60316]: debug 2022-01-31T21:57:14.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:14.504141+0000) 2022-01-31T21:57:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:14 smithi167 conmon[49112]: debug 2022-01-31T21:57:14.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:14.840652+0000) 2022-01-31T21:57:15.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:15 smithi171 conmon[46715]: debug 2022-01-31T21:57:15.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:15.013306+0000) 2022-01-31T21:57:15.172 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:14 smithi171 conmon[51620]: debug 2022-01-31T21:57:14.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:14.859052+0000) 2022-01-31T21:57:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:15 smithi171 conmon[41853]: debug 2022-01-31T21:57:15.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:15.217594+0000) 2022-01-31T21:57:15.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:15 smithi167 conmon[54076]: debug 2022-01-31T21:57:15.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:15.504254+0000) 2022-01-31T21:57:15.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:15 smithi167 conmon[60316]: debug 2022-01-31T21:57:15.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:15.504322+0000) 2022-01-31T21:57:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:15 smithi167 conmon[49112]: debug 2022-01-31T21:57:15.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:15.840825+0000) 2022-01-31T21:57:16.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[46715]: debug 2022-01-31T21:57:16.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.013438+0000) 2022-01-31T21:57:16.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:15 smithi171 conmon[51620]: debug 2022-01-31T21:57:15.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:15.859244+0000) 2022-01-31T21:57:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[41853]: debug 2022-01-31T21:57:16.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.217783+0000) 2022-01-31T21:57:16.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:16 smithi167 conmon[54076]: debug 2022-01-31T21:57:16.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.504439+0000) 2022-01-31T21:57:16.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:16 smithi167 conmon[60316]: debug 2022-01-31T21:57:16.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.504475+0000) 2022-01-31T21:57:17.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[35325]: debug 2022-01-31T21:57:16.784+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170770 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:17.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[41853]: debug 2022-01-31T21:57:16.766+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.767342+0000) 2022-01-31T21:57:17.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[46715]: debug 2022-01-31T21:57:16.767+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.768230+0000) 2022-01-31T21:57:17.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:17 smithi171 conmon[46715]: debug 2022-01-31T21:57:17.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:17.013593+0000) 2022-01-31T21:57:17.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[51620]: debug 2022-01-31T21:57:16.766+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.766915+0000) 2022-01-31T21:57:17.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:16 smithi171 conmon[51620]: debug 2022-01-31T21:57:16.858+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.859430+0000) 2022-01-31T21:57:17.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:16 smithi167 conmon[54076]: debug 2022-01-31T21:57:16.767+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.767887+0000) 2022-01-31T21:57:17.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:16 smithi167 conmon[60316]: debug 2022-01-31T21:57:16.767+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.767547+0000) 2022-01-31T21:57:17.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:16 smithi167 conmon[49112]: debug 2022-01-31T21:57:16.767+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.767694+0000) 2022-01-31T21:57:17.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:16 smithi167 conmon[49112]: debug 2022-01-31T21:57:16.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:16.841034+0000) 2022-01-31T21:57:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:17 smithi171 conmon[41853]: debug 2022-01-31T21:57:17.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:17.217937+0000) 2022-01-31T21:57:17.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:17 smithi167 conmon[54076]: debug 2022-01-31T21:57:17.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:17.504637+0000) 2022-01-31T21:57:17.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:17 smithi167 conmon[60316]: debug 2022-01-31T21:57:17.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:17.504638+0000) 2022-01-31T21:57:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:17 smithi167 conmon[49112]: debug 2022-01-31T21:57:17.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:17.841197+0000) 2022-01-31T21:57:18.172 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:18 smithi171 conmon[46715]: debug 2022-01-31T21:57:18.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:18.013779+0000) 2022-01-31T21:57:18.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:17 smithi171 conmon[51620]: debug 2022-01-31T21:57:17.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:17.859607+0000) 2022-01-31T21:57:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:18 smithi171 conmon[41853]: debug 2022-01-31T21:57:18.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:18.218073+0000) 2022-01-31T21:57:18.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:18 smithi167 conmon[54076]: debug 2022-01-31T21:57:18.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:18.504800+0000) 2022-01-31T21:57:18.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:18 smithi167 conmon[60316]: debug 2022-01-31T21:57:18.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:18.504826+0000) 2022-01-31T21:57:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:18 smithi167 conmon[49112]: debug 2022-01-31T21:57:18.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:18.841397+0000) 2022-01-31T21:57:19.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:19 smithi171 conmon[46715]: debug 2022-01-31T21:57:19.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:19.013944+0000) 2022-01-31T21:57:19.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:18 smithi171 conmon[51620]: debug 2022-01-31T21:57:18.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:18.859821+0000) 2022-01-31T21:57:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:19 smithi171 conmon[41853]: debug 2022-01-31T21:57:19.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:19.218257+0000) 2022-01-31T21:57:19.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:19 smithi167 conmon[54076]: debug 2022-01-31T21:57:19.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:19.504992+0000) 2022-01-31T21:57:19.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:19 smithi167 conmon[60316]: debug 2022-01-31T21:57:19.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:19.504993+0000) 2022-01-31T21:57:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:19 smithi167 conmon[49112]: debug 2022-01-31T21:57:19.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:19.841578+0000) 2022-01-31T21:57:20.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:20 smithi171 conmon[46715]: debug 2022-01-31T21:57:20.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:20.014129+0000) 2022-01-31T21:57:20.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:19 smithi171 conmon[51620]: debug 2022-01-31T21:57:19.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:19.860010+0000) 2022-01-31T21:57:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:20 smithi171 conmon[41853]: debug 2022-01-31T21:57:20.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:20.218439+0000) 2022-01-31T21:57:20.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:20 smithi167 conmon[54076]: debug 2022-01-31T21:57:20.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:20.505193+0000) 2022-01-31T21:57:20.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:20 smithi167 conmon[60316]: debug 2022-01-31T21:57:20.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:20.505192+0000) 2022-01-31T21:57:21.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:20 smithi167 conmon[49112]: debug 2022-01-31T21:57:20.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:20.841762+0000) 2022-01-31T21:57:21.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[46715]: debug 2022-01-31T21:57:21.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.014253+0000) 2022-01-31T21:57:21.173 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:20 smithi171 conmon[51620]: debug 2022-01-31T21:57:20.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:20.860192+0000) 2022-01-31T21:57:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[41853]: debug 2022-01-31T21:57:21.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.218615+0000) 2022-01-31T21:57:21.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:21 smithi167 conmon[60316]: debug 2022-01-31T21:57:21.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.505390+0000) 2022-01-31T21:57:21.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:21 smithi167 conmon[54076]: debug 2022-01-31T21:57:21.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.505393+0000) 2022-01-31T21:57:22.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[35325]: debug 2022-01-31T21:57:21.804+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170879 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:22.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[41853]: debug 2022-01-31T21:57:21.786+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.787613+0000) 2022-01-31T21:57:22.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[46715]: debug 2022-01-31T21:57:21.787+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.787995+0000) 2022-01-31T21:57:22.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:22 smithi171 conmon[46715]: debug 2022-01-31T21:57:22.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:22.014420+0000) 2022-01-31T21:57:22.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[51620]: debug 2022-01-31T21:57:21.786+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.786962+0000) 2022-01-31T21:57:22.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:21 smithi171 conmon[51620]: debug 2022-01-31T21:57:21.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.860381+0000) 2022-01-31T21:57:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:21 smithi167 conmon[49112]: debug 2022-01-31T21:57:21.787+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.788260+0000) 2022-01-31T21:57:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:21 smithi167 conmon[49112]: debug 2022-01-31T21:57:21.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.841928+0000) 2022-01-31T21:57:22.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:21 smithi167 conmon[54076]: debug 2022-01-31T21:57:21.787+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.787566+0000) 2022-01-31T21:57:22.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:21 smithi167 conmon[60316]: debug 2022-01-31T21:57:21.787+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:21.788455+0000) 2022-01-31T21:57:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:22 smithi171 conmon[41853]: debug 2022-01-31T21:57:22.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:22.218790+0000) 2022-01-31T21:57:22.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:22 smithi167 conmon[54076]: debug 2022-01-31T21:57:22.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:22.505580+0000) 2022-01-31T21:57:22.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:22 smithi167 conmon[60316]: debug 2022-01-31T21:57:22.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:22.505532+0000) 2022-01-31T21:57:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:22 smithi167 conmon[49112]: debug 2022-01-31T21:57:22.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:22.842098+0000) 2022-01-31T21:57:23.173 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:23 smithi171 conmon[46715]: debug 2022-01-31T21:57:23.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:23.014559+0000) 2022-01-31T21:57:23.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:22 smithi171 conmon[51620]: debug 2022-01-31T21:57:22.859+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:22.860570+0000) 2022-01-31T21:57:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:23 smithi171 conmon[41853]: debug 2022-01-31T21:57:23.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:23.218989+0000) 2022-01-31T21:57:23.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:23 smithi167 conmon[54076]: debug 2022-01-31T21:57:23.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:23.505680+0000) 2022-01-31T21:57:23.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:23 smithi167 conmon[60316]: debug 2022-01-31T21:57:23.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:23.505659+0000) 2022-01-31T21:57:24.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:24 smithi171 conmon[46715]: debug 2022-01-31T21:57:24.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:24.014779+0000) 2022-01-31T21:57:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:23 smithi171 conmon[51620]: debug 2022-01-31T21:57:23.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:23.860817+0000) 2022-01-31T21:57:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:23 smithi171 conmon[35325]: debug 2022-01-31T21:57:23.766+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:57:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:23 smithi167 conmon[49112]: debug 2022-01-31T21:57:23.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:23.842341+0000) 2022-01-31T21:57:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:24 smithi171 conmon[41853]: debug 2022-01-31T21:57:24.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:24.219187+0000) 2022-01-31T21:57:24.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:24 smithi167 conmon[54076]: debug 2022-01-31T21:57:24.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:24.505837+0000) 2022-01-31T21:57:24.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:24 smithi167 conmon[60316]: debug 2022-01-31T21:57:24.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:24.505789+0000) 2022-01-31T21:57:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:24 smithi167 conmon[49112]: debug 2022-01-31T21:57:24.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:24.842533+0000) 2022-01-31T21:57:25.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:25 smithi171 conmon[46715]: debug 2022-01-31T21:57:25.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:25.014964+0000) 2022-01-31T21:57:25.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:24 smithi171 conmon[51620]: debug 2022-01-31T21:57:24.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:24.861025+0000) 2022-01-31T21:57:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:25 smithi171 conmon[41853]: debug 2022-01-31T21:57:25.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:25.219366+0000) 2022-01-31T21:57:25.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:25 smithi167 conmon[54076]: debug 2022-01-31T21:57:25.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:25.506015+0000) 2022-01-31T21:57:25.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:25 smithi167 conmon[60316]: debug 2022-01-31T21:57:25.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:25.505976+0000) 2022-01-31T21:57:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:25 smithi167 conmon[49112]: debug 2022-01-31T21:57:25.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:25.842749+0000) 2022-01-31T21:57:26.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[46715]: debug 2022-01-31T21:57:26.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.015125+0000) 2022-01-31T21:57:26.174 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:25 smithi171 conmon[51620]: debug 2022-01-31T21:57:25.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:25.861234+0000) 2022-01-31T21:57:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[41853]: debug 2022-01-31T21:57:26.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.219541+0000) 2022-01-31T21:57:26.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:26 smithi167 conmon[54076]: debug 2022-01-31T21:57:26.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.506188+0000) 2022-01-31T21:57:26.807 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:26 smithi167 conmon[60316]: debug 2022-01-31T21:57:26.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.506187+0000) 2022-01-31T21:57:27.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[35325]: debug 2022-01-31T21:57:26.831+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 170991 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:27.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[41853]: debug 2022-01-31T21:57:26.808+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.808841+0000) 2022-01-31T21:57:27.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[46715]: debug 2022-01-31T21:57:26.807+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.808544+0000) 2022-01-31T21:57:27.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:27 smithi171 conmon[46715]: debug 2022-01-31T21:57:27.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:27.015286+0000) 2022-01-31T21:57:27.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[51620]: debug 2022-01-31T21:57:26.807+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.807949+0000) 2022-01-31T21:57:27.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:26 smithi171 conmon[51620]: debug 2022-01-31T21:57:26.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.861423+0000) 2022-01-31T21:57:27.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:26 smithi167 conmon[54076]: debug 2022-01-31T21:57:26.807+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.809022+0000) 2022-01-31T21:57:27.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:26 smithi167 conmon[60316]: debug 2022-01-31T21:57:26.806+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.808235+0000) 2022-01-31T21:57:27.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:26 smithi167 conmon[49112]: debug 2022-01-31T21:57:26.807+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.808976+0000) 2022-01-31T21:57:27.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:26 smithi167 conmon[49112]: debug 2022-01-31T21:57:26.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:26.842855+0000) 2022-01-31T21:57:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:27 smithi171 conmon[41853]: debug 2022-01-31T21:57:27.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:27.219728+0000) 2022-01-31T21:57:27.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:27 smithi167 conmon[54076]: debug 2022-01-31T21:57:27.504+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:27.506367+0000) 2022-01-31T21:57:27.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:27 smithi167 conmon[60316]: debug 2022-01-31T21:57:27.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:27.506317+0000) 2022-01-31T21:57:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:27 smithi167 conmon[49112]: debug 2022-01-31T21:57:27.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:27.843038+0000) 2022-01-31T21:57:28.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:28 smithi171 conmon[46715]: debug 2022-01-31T21:57:28.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:28.015435+0000) 2022-01-31T21:57:28.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:27 smithi171 conmon[51620]: debug 2022-01-31T21:57:27.860+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:27.861562+0000) 2022-01-31T21:57:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:28 smithi171 conmon[41853]: debug 2022-01-31T21:57:28.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:28.219886+0000) 2022-01-31T21:57:28.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:28 smithi167 conmon[54076]: debug 2022-01-31T21:57:28.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:28.506491+0000) 2022-01-31T21:57:28.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:28 smithi167 conmon[60316]: debug 2022-01-31T21:57:28.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:28.506473+0000) 2022-01-31T21:57:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:28 smithi167 conmon[49112]: debug 2022-01-31T21:57:28.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:28.843258+0000) 2022-01-31T21:57:29.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:29 smithi171 conmon[46715]: debug 2022-01-31T21:57:29.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:29.015578+0000) 2022-01-31T21:57:29.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:28 smithi171 conmon[51620]: debug 2022-01-31T21:57:28.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:28.861792+0000) 2022-01-31T21:57:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:29 smithi171 conmon[41853]: debug 2022-01-31T21:57:29.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:29.220077+0000) 2022-01-31T21:57:29.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:29 smithi167 conmon[54076]: debug 2022-01-31T21:57:29.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:29.506619+0000) 2022-01-31T21:57:29.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:29 smithi167 conmon[60316]: debug 2022-01-31T21:57:29.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:29.506603+0000) 2022-01-31T21:57:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:29 smithi167 conmon[49112]: debug 2022-01-31T21:57:29.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:29.843455+0000) 2022-01-31T21:57:30.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:30 smithi171 conmon[46715]: debug 2022-01-31T21:57:30.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:30.015788+0000) 2022-01-31T21:57:30.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:29 smithi171 conmon[51620]: debug 2022-01-31T21:57:29.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:29.862003+0000) 2022-01-31T21:57:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:30 smithi171 conmon[41853]: debug 2022-01-31T21:57:30.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:30.220214+0000) 2022-01-31T21:57:30.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:30 smithi167 conmon[54076]: debug 2022-01-31T21:57:30.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:30.506796+0000) 2022-01-31T21:57:30.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:30 smithi167 conmon[60316]: debug 2022-01-31T21:57:30.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:30.506766+0000) 2022-01-31T21:57:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:30 smithi167 conmon[49112]: debug 2022-01-31T21:57:30.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:30.843665+0000) 2022-01-31T21:57:31.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[46715]: debug 2022-01-31T21:57:31.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.015975+0000) 2022-01-31T21:57:31.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:30 smithi171 conmon[51620]: debug 2022-01-31T21:57:30.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:30.862220+0000) 2022-01-31T21:57:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[41853]: debug 2022-01-31T21:57:31.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.220421+0000) 2022-01-31T21:57:31.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:31 smithi167 conmon[54076]: debug 2022-01-31T21:57:31.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.507012+0000) 2022-01-31T21:57:31.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:31 smithi167 conmon[60316]: debug 2022-01-31T21:57:31.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.507003+0000) 2022-01-31T21:57:32.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[35325]: debug 2022-01-31T21:57:31.851+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171100 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:32.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[41853]: debug 2022-01-31T21:57:31.833+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.834564+0000) 2022-01-31T21:57:32.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[46715]: debug 2022-01-31T21:57:31.834+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.835212+0000) 2022-01-31T21:57:32.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:32 smithi171 conmon[46715]: debug 2022-01-31T21:57:32.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:32.016148+0000) 2022-01-31T21:57:32.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[51620]: debug 2022-01-31T21:57:31.833+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.834469+0000) 2022-01-31T21:57:32.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:31 smithi171 conmon[51620]: debug 2022-01-31T21:57:31.861+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.862374+0000) 2022-01-31T21:57:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:31 smithi167 conmon[49112]: debug 2022-01-31T21:57:31.832+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.835344+0000) 2022-01-31T21:57:32.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:31 smithi167 conmon[49112]: debug 2022-01-31T21:57:31.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.843776+0000) 2022-01-31T21:57:32.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:31 smithi167 conmon[54076]: debug 2022-01-31T21:57:31.832+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.835551+0000) 2022-01-31T21:57:32.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:31 smithi167 conmon[60316]: debug 2022-01-31T21:57:31.832+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:31.834363+0000) 2022-01-31T21:57:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:32 smithi171 conmon[41853]: debug 2022-01-31T21:57:32.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:32.220608+0000) 2022-01-31T21:57:32.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:32 smithi167 conmon[54076]: debug 2022-01-31T21:57:32.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:32.507232+0000) 2022-01-31T21:57:32.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:32 smithi167 conmon[60316]: debug 2022-01-31T21:57:32.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:32.507234+0000) 2022-01-31T21:57:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:32 smithi167 conmon[49112]: debug 2022-01-31T21:57:32.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:32.843944+0000) 2022-01-31T21:57:33.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:33 smithi171 conmon[46715]: debug 2022-01-31T21:57:33.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:33.016271+0000) 2022-01-31T21:57:33.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:32 smithi171 conmon[51620]: debug 2022-01-31T21:57:32.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:32.862543+0000) 2022-01-31T21:57:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:33 smithi171 conmon[41853]: debug 2022-01-31T21:57:33.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:33.220794+0000) 2022-01-31T21:57:33.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:33 smithi167 conmon[54076]: debug 2022-01-31T21:57:33.505+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:33.507387+0000) 2022-01-31T21:57:33.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:33 smithi167 conmon[60316]: debug 2022-01-31T21:57:33.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:33.507408+0000) 2022-01-31T21:57:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:33 smithi167 conmon[49112]: debug 2022-01-31T21:57:33.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:33.844122+0000) 2022-01-31T21:57:34.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:34 smithi171 conmon[46715]: debug 2022-01-31T21:57:34.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:34.016445+0000) 2022-01-31T21:57:34.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:33 smithi171 conmon[51620]: debug 2022-01-31T21:57:33.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:33.862770+0000) 2022-01-31T21:57:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:34 smithi171 conmon[41853]: debug 2022-01-31T21:57:34.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:34.221002+0000) 2022-01-31T21:57:34.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:34 smithi167 conmon[54076]: debug 2022-01-31T21:57:34.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:34.507602+0000) 2022-01-31T21:57:34.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:34 smithi167 conmon[60316]: debug 2022-01-31T21:57:34.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:34.507602+0000) 2022-01-31T21:57:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:34 smithi167 conmon[49112]: debug 2022-01-31T21:57:34.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:34.844308+0000) 2022-01-31T21:57:35.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:35 smithi171 conmon[46715]: debug 2022-01-31T21:57:35.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:35.016634+0000) 2022-01-31T21:57:35.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:34 smithi171 conmon[51620]: debug 2022-01-31T21:57:34.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:34.862997+0000) 2022-01-31T21:57:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:35 smithi171 conmon[41853]: debug 2022-01-31T21:57:35.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:35.221205+0000) 2022-01-31T21:57:35.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:35 smithi167 conmon[54076]: debug 2022-01-31T21:57:35.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:35.507818+0000) 2022-01-31T21:57:35.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:35 smithi167 conmon[60316]: debug 2022-01-31T21:57:35.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:35.507802+0000) 2022-01-31T21:57:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:35 smithi167 conmon[49112]: debug 2022-01-31T21:57:35.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:35.844531+0000) 2022-01-31T21:57:36.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:35 smithi171 conmon[51620]: debug 2022-01-31T21:57:35.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:35.863208+0000) 2022-01-31T21:57:36.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[46715]: debug 2022-01-31T21:57:36.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.016784+0000) 2022-01-31T21:57:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[41853]: debug 2022-01-31T21:57:36.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.221409+0000) 2022-01-31T21:57:36.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:36 smithi167 conmon[54076]: debug 2022-01-31T21:57:36.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.508002+0000) 2022-01-31T21:57:36.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:36 smithi167 conmon[60316]: debug 2022-01-31T21:57:36.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.507986+0000) 2022-01-31T21:57:37.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[35325]: debug 2022-01-31T21:57:36.872+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171212 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:37.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[41853]: debug 2022-01-31T21:57:36.853+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.854722+0000) 2022-01-31T21:57:37.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[46715]: debug 2022-01-31T21:57:36.854+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.855139+0000) 2022-01-31T21:57:37.109 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:37 smithi171 conmon[46715]: debug 2022-01-31T21:57:37.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:37.016901+0000) 2022-01-31T21:57:37.110 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[51620]: debug 2022-01-31T21:57:36.853+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.854322+0000) 2022-01-31T21:57:37.111 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:36 smithi171 conmon[51620]: debug 2022-01-31T21:57:36.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.863370+0000) 2022-01-31T21:57:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:36 smithi167 conmon[49112]: debug 2022-01-31T21:57:36.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.844629+0000) 2022-01-31T21:57:37.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:36 smithi167 conmon[49112]: debug 2022-01-31T21:57:36.854+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.856199+0000) 2022-01-31T21:57:37.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:36 smithi167 conmon[54076]: debug 2022-01-31T21:57:36.854+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.855536+0000) 2022-01-31T21:57:37.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:36 smithi167 conmon[60316]: debug 2022-01-31T21:57:36.854+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:36.855616+0000) 2022-01-31T21:57:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:37 smithi171 conmon[41853]: debug 2022-01-31T21:57:37.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:37.221616+0000) 2022-01-31T21:57:37.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:37 smithi167 conmon[54076]: debug 2022-01-31T21:57:37.506+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:37.508218+0000) 2022-01-31T21:57:37.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:37 smithi167 conmon[60316]: debug 2022-01-31T21:57:37.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:37.508217+0000) 2022-01-31T21:57:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:37 smithi167 conmon[49112]: debug 2022-01-31T21:57:37.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:37.844803+0000) 2022-01-31T21:57:38.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:38 smithi171 conmon[46715]: debug 2022-01-31T21:57:38.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:38.017069+0000) 2022-01-31T21:57:38.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:37 smithi171 conmon[51620]: debug 2022-01-31T21:57:37.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:37.863556+0000) 2022-01-31T21:57:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:38 smithi171 conmon[41853]: debug 2022-01-31T21:57:38.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:38.221776+0000) 2022-01-31T21:57:38.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:38 smithi167 conmon[54076]: debug 2022-01-31T21:57:38.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:38.508367+0000) 2022-01-31T21:57:38.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:38 smithi167 conmon[60316]: debug 2022-01-31T21:57:38.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:38.508368+0000) 2022-01-31T21:57:39.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:38 smithi171 conmon[35325]: debug 2022-01-31T21:57:38.767+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:57:39.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:39 smithi171 conmon[46715]: debug 2022-01-31T21:57:39.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:39.017226+0000) 2022-01-31T21:57:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:38 smithi171 conmon[51620]: debug 2022-01-31T21:57:38.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:38.863764+0000) 2022-01-31T21:57:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:38 smithi167 conmon[49112]: debug 2022-01-31T21:57:38.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:38.844986+0000) 2022-01-31T21:57:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:39 smithi171 conmon[41853]: debug 2022-01-31T21:57:39.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:39.221933+0000) 2022-01-31T21:57:39.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:39 smithi167 conmon[54076]: debug 2022-01-31T21:57:39.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:39.508548+0000) 2022-01-31T21:57:39.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:39 smithi167 conmon[60316]: debug 2022-01-31T21:57:39.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:39.508549+0000) 2022-01-31T21:57:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:39 smithi167 conmon[49112]: debug 2022-01-31T21:57:39.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:39.845180+0000) 2022-01-31T21:57:40.176 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:40 smithi171 conmon[46715]: debug 2022-01-31T21:57:40.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:40.017382+0000) 2022-01-31T21:57:40.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:39 smithi171 conmon[51620]: debug 2022-01-31T21:57:39.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:39.863969+0000) 2022-01-31T21:57:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:40 smithi171 conmon[41853]: debug 2022-01-31T21:57:40.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:40.222089+0000) 2022-01-31T21:57:40.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:40 smithi167 conmon[54076]: debug 2022-01-31T21:57:40.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:40.508768+0000) 2022-01-31T21:57:40.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:40 smithi167 conmon[60316]: debug 2022-01-31T21:57:40.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:40.508790+0000) 2022-01-31T21:57:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:40 smithi167 conmon[49112]: debug 2022-01-31T21:57:40.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:40.845368+0000) 2022-01-31T21:57:41.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[46715]: debug 2022-01-31T21:57:41.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.017571+0000) 2022-01-31T21:57:41.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:40 smithi171 conmon[51620]: debug 2022-01-31T21:57:40.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:40.864158+0000) 2022-01-31T21:57:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[41853]: debug 2022-01-31T21:57:41.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.222239+0000) 2022-01-31T21:57:41.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:41 smithi167 conmon[54076]: debug 2022-01-31T21:57:41.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.508935+0000) 2022-01-31T21:57:41.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:41 smithi167 conmon[60316]: debug 2022-01-31T21:57:41.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.508935+0000) 2022-01-31T21:57:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:41 smithi167 conmon[49112]: debug 2022-01-31T21:57:41.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.845580+0000) 2022-01-31T21:57:42.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:41 smithi167 conmon[49112]: debug 2022-01-31T21:57:41.875+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.876552+0000) 2022-01-31T21:57:42.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:41 smithi167 conmon[54076]: debug 2022-01-31T21:57:41.874+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.876290+0000) 2022-01-31T21:57:42.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:41 smithi167 conmon[60316]: debug 2022-01-31T21:57:41.875+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.877311+0000) 2022-01-31T21:57:42.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[35325]: debug 2022-01-31T21:57:41.892+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171321 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:42.177 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[41853]: debug 2022-01-31T21:57:41.875+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.876981+0000) 2022-01-31T21:57:42.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[46715]: debug 2022-01-31T21:57:41.873+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.875437+0000) 2022-01-31T21:57:42.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:42 smithi171 conmon[46715]: debug 2022-01-31T21:57:42.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:42.017764+0000) 2022-01-31T21:57:42.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[51620]: debug 2022-01-31T21:57:41.862+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.864371+0000) 2022-01-31T21:57:42.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:41 smithi171 conmon[51620]: debug 2022-01-31T21:57:41.874+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:41.876262+0000) 2022-01-31T21:57:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:42 smithi171 conmon[41853]: debug 2022-01-31T21:57:42.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:42.222392+0000) 2022-01-31T21:57:42.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:42 smithi167 conmon[54076]: debug 2022-01-31T21:57:42.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:42.509111+0000) 2022-01-31T21:57:42.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:42 smithi167 conmon[60316]: debug 2022-01-31T21:57:42.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:42.509111+0000) 2022-01-31T21:57:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:42 smithi167 conmon[49112]: debug 2022-01-31T21:57:42.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:42.845747+0000) 2022-01-31T21:57:43.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:43 smithi171 conmon[46715]: debug 2022-01-31T21:57:43.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:43.017921+0000) 2022-01-31T21:57:43.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:42 smithi171 conmon[51620]: debug 2022-01-31T21:57:42.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:42.864588+0000) 2022-01-31T21:57:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:43 smithi171 conmon[41853]: debug 2022-01-31T21:57:43.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:43.222558+0000) 2022-01-31T21:57:43.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:43 smithi167 conmon[54076]: debug 2022-01-31T21:57:43.508+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:43.509281+0000) 2022-01-31T21:57:43.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:43 smithi167 conmon[60316]: debug 2022-01-31T21:57:43.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:43.509277+0000) 2022-01-31T21:57:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:43 smithi167 conmon[49112]: debug 2022-01-31T21:57:43.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:43.845919+0000) 2022-01-31T21:57:44.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:44 smithi171 conmon[46715]: debug 2022-01-31T21:57:44.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:44.018143+0000) 2022-01-31T21:57:44.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:43 smithi171 conmon[51620]: debug 2022-01-31T21:57:43.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:43.864742+0000) 2022-01-31T21:57:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:44 smithi171 conmon[41853]: debug 2022-01-31T21:57:44.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:44.222720+0000) 2022-01-31T21:57:44.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:44 smithi167 conmon[54076]: debug 2022-01-31T21:57:44.507+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:44.509466+0000) 2022-01-31T21:57:44.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:44 smithi167 conmon[60316]: debug 2022-01-31T21:57:44.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:44.509476+0000) 2022-01-31T21:57:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:44 smithi167 conmon[49112]: debug 2022-01-31T21:57:44.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:44.846134+0000) 2022-01-31T21:57:45.177 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:44 smithi171 conmon[51620]: debug 2022-01-31T21:57:44.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:44.864965+0000) 2022-01-31T21:57:45.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:45 smithi171 conmon[46715]: debug 2022-01-31T21:57:45.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:45.018327+0000) 2022-01-31T21:57:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:45 smithi171 conmon[41853]: debug 2022-01-31T21:57:45.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:45.222942+0000) 2022-01-31T21:57:45.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:45 smithi167 conmon[54076]: debug 2022-01-31T21:57:45.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:45.509600+0000) 2022-01-31T21:57:45.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:45 smithi167 conmon[60316]: debug 2022-01-31T21:57:45.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:45.509631+0000) 2022-01-31T21:57:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:45 smithi167 conmon[49112]: debug 2022-01-31T21:57:45.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:45.846341+0000) 2022-01-31T21:57:46.177 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[46715]: debug 2022-01-31T21:57:46.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.018541+0000) 2022-01-31T21:57:46.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:45 smithi171 conmon[51620]: debug 2022-01-31T21:57:45.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:45.865168+0000) 2022-01-31T21:57:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[41853]: debug 2022-01-31T21:57:46.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.223164+0000) 2022-01-31T21:57:46.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:46 smithi167 conmon[54076]: debug 2022-01-31T21:57:46.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.509790+0000) 2022-01-31T21:57:46.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:46 smithi167 conmon[60316]: debug 2022-01-31T21:57:46.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.509793+0000) 2022-01-31T21:57:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:46 smithi167 conmon[49112]: debug 2022-01-31T21:57:46.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.846534+0000) 2022-01-31T21:57:47.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:46 smithi167 conmon[49112]: debug 2022-01-31T21:57:46.895+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.896925+0000) 2022-01-31T21:57:47.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:46 smithi167 conmon[54076]: debug 2022-01-31T21:57:46.894+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.896200+0000) 2022-01-31T21:57:47.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:46 smithi167 conmon[60316]: debug 2022-01-31T21:57:46.895+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.897437+0000) 2022-01-31T21:57:47.178 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[35325]: debug 2022-01-31T21:57:46.912+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171433 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:47.178 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[41853]: debug 2022-01-31T21:57:46.895+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.897500+0000) 2022-01-31T21:57:47.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[51620]: debug 2022-01-31T21:57:46.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.865325+0000) 2022-01-31T21:57:47.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[51620]: debug 2022-01-31T21:57:46.894+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.895997+0000) 2022-01-31T21:57:47.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:46 smithi171 conmon[46715]: debug 2022-01-31T21:57:46.895+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:46.897339+0000) 2022-01-31T21:57:47.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:47 smithi171 conmon[46715]: debug 2022-01-31T21:57:47.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:47.018697+0000) 2022-01-31T21:57:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:47 smithi171 conmon[41853]: debug 2022-01-31T21:57:47.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:47.223345+0000) 2022-01-31T21:57:47.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:47 smithi167 conmon[54076]: debug 2022-01-31T21:57:47.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:47.510000+0000) 2022-01-31T21:57:47.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:47 smithi167 conmon[60316]: debug 2022-01-31T21:57:47.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:47.509996+0000) 2022-01-31T21:57:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:47 smithi167 conmon[49112]: debug 2022-01-31T21:57:47.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:47.846660+0000) 2022-01-31T21:57:48.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:48 smithi171 conmon[46715]: debug 2022-01-31T21:57:48.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:48.018813+0000) 2022-01-31T21:57:48.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:47 smithi171 conmon[51620]: debug 2022-01-31T21:57:47.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:47.865493+0000) 2022-01-31T21:57:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:48 smithi171 conmon[41853]: debug 2022-01-31T21:57:48.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:48.223501+0000) 2022-01-31T21:57:48.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:48 smithi167 conmon[54076]: debug 2022-01-31T21:57:48.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:48.510163+0000) 2022-01-31T21:57:48.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:48 smithi167 conmon[60316]: debug 2022-01-31T21:57:48.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:48.510168+0000) 2022-01-31T21:57:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:48 smithi167 conmon[49112]: debug 2022-01-31T21:57:48.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:48.846836+0000) 2022-01-31T21:57:49.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:48 smithi171 conmon[51620]: debug 2022-01-31T21:57:48.863+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:48.865632+0000) 2022-01-31T21:57:49.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:49 smithi171 conmon[46715]: debug 2022-01-31T21:57:49.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:49.018984+0000) 2022-01-31T21:57:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:49 smithi171 conmon[41853]: debug 2022-01-31T21:57:49.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:49.223705+0000) 2022-01-31T21:57:49.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:49 smithi167 conmon[54076]: debug 2022-01-31T21:57:49.509+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:49.510356+0000) 2022-01-31T21:57:49.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:49 smithi167 conmon[60316]: debug 2022-01-31T21:57:49.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:49.510375+0000) 2022-01-31T21:57:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:49 smithi167 conmon[49112]: debug 2022-01-31T21:57:49.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:49.847044+0000) 2022-01-31T21:57:50.178 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:50 smithi171 conmon[46715]: debug 2022-01-31T21:57:50.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:50.019170+0000) 2022-01-31T21:57:50.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:49 smithi171 conmon[51620]: debug 2022-01-31T21:57:49.864+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:49.865833+0000) 2022-01-31T21:57:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:50 smithi171 conmon[41853]: debug 2022-01-31T21:57:50.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:50.223957+0000) 2022-01-31T21:57:50.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:50 smithi167 conmon[54076]: debug 2022-01-31T21:57:50.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:50.510563+0000) 2022-01-31T21:57:50.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:50 smithi167 conmon[60316]: debug 2022-01-31T21:57:50.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:50.510563+0000) 2022-01-31T21:57:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:50 smithi167 conmon[49112]: debug 2022-01-31T21:57:50.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:50.847226+0000) 2022-01-31T21:57:51.178 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:50 smithi171 conmon[51620]: debug 2022-01-31T21:57:50.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:50.865971+0000) 2022-01-31T21:57:51.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[46715]: debug 2022-01-31T21:57:51.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.019354+0000) 2022-01-31T21:57:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[41853]: debug 2022-01-31T21:57:51.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.224133+0000) 2022-01-31T21:57:51.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[54076]: debug 2022-01-31T21:57:51.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.510777+0000) 2022-01-31T21:57:51.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[60316]: debug 2022-01-31T21:57:51.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.510779+0000) 2022-01-31T21:57:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[49112]: debug 2022-01-31T21:57:51.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.847411+0000) 2022-01-31T21:57:52.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[49112]: debug 2022-01-31T21:57:51.916+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.917023+0000) 2022-01-31T21:57:52.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[54076]: debug 2022-01-31T21:57:51.915+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.916401+0000) 2022-01-31T21:57:52.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[60316]: debug 2022-01-31T21:57:51.917+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.917875+0000) 2022-01-31T21:57:52.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:51 smithi167 conmon[60316]: 2022-01-31T21:57:52.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[35325]: debug 2022-01-31T21:57:51.933+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171542 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:52.179 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[41853]: debug 2022-01-31T21:57:51.917+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.918212+0000) 2022-01-31T21:57:52.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[46715]: debug 2022-01-31T21:57:51.916+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.917571+0000) 2022-01-31T21:57:52.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:52 smithi171 conmon[46715]: debug 2022-01-31T21:57:52.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:52.019486+0000) 2022-01-31T21:57:52.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[51620]: debug 2022-01-31T21:57:51.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.866162+0000) 2022-01-31T21:57:52.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:51 smithi171 conmon[51620]: debug 2022-01-31T21:57:51.915+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:51.916033+0000) 2022-01-31T21:57:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:52 smithi171 conmon[41853]: debug 2022-01-31T21:57:52.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:52.224322+0000) 2022-01-31T21:57:52.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:52 smithi167 conmon[54076]: debug 2022-01-31T21:57:52.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:52.510972+0000) 2022-01-31T21:57:52.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:52 smithi167 conmon[60316]: debug 2022-01-31T21:57:52.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:52.510973+0000) 2022-01-31T21:57:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:52 smithi167 conmon[49112]: debug 2022-01-31T21:57:52.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:52.847651+0000) 2022-01-31T21:57:53.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:53 smithi171 conmon[46715]: debug 2022-01-31T21:57:53.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:53.019624+0000) 2022-01-31T21:57:53.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:52 smithi171 conmon[51620]: debug 2022-01-31T21:57:52.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:52.866385+0000) 2022-01-31T21:57:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:53 smithi171 conmon[41853]: debug 2022-01-31T21:57:53.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:53.224497+0000) 2022-01-31T21:57:53.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:53 smithi167 conmon[54076]: debug 2022-01-31T21:57:53.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:53.511106+0000) 2022-01-31T21:57:53.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:53 smithi167 conmon[60316]: debug 2022-01-31T21:57:53.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:53.511126+0000) 2022-01-31T21:57:54.018 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:53 smithi171 conmon[35325]: debug 2022-01-31T21:57:53.767+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:57:54.019 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:53 smithi171 conmon[51620]: debug 2022-01-31T21:57:53.865+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:53.866566+0000) 2022-01-31T21:57:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:53 smithi167 conmon[49112]: debug 2022-01-31T21:57:53.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:53.847779+0000) 2022-01-31T21:57:54.360 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:54 smithi171 conmon[41853]: debug 2022-01-31T21:57:54.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:54.224651+0000) 2022-01-31T21:57:54.361 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:54 smithi171 conmon[46715]: debug 2022-01-31T21:57:54.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:54.019832+0000) 2022-01-31T21:57:54.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:54 smithi167 conmon[54076]: debug 2022-01-31T21:57:54.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:54.511268+0000) 2022-01-31T21:57:54.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:54 smithi167 conmon[60316]: debug 2022-01-31T21:57:54.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:54.511299+0000) 2022-01-31T21:57:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:54 smithi167 conmon[49112]: debug 2022-01-31T21:57:54.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:54.847967+0000) 2022-01-31T21:57:55.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:55 smithi171 conmon[46715]: debug 2022-01-31T21:57:55.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:55.019975+0000) 2022-01-31T21:57:55.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:54 smithi171 conmon[51620]: debug 2022-01-31T21:57:54.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:54.866746+0000) 2022-01-31T21:57:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:55 smithi171 conmon[41853]: debug 2022-01-31T21:57:55.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:55.224834+0000) 2022-01-31T21:57:55.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:55 smithi167 conmon[54076]: debug 2022-01-31T21:57:55.510+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:55.511452+0000) 2022-01-31T21:57:55.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:55 smithi167 conmon[60316]: debug 2022-01-31T21:57:55.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:55.511449+0000) 2022-01-31T21:57:55.929 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:55 smithi171 conmon[51620]: debug 2022-01-31T21:57:55.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:55.866961+0000) 2022-01-31T21:57:56.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:55 smithi167 conmon[49112]: debug 2022-01-31T21:57:55.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:55.848179+0000) 2022-01-31T21:57:57.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[46715]: debug 2022-01-31T21:57:56.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.020121+0000) 2022-01-31T21:57:57.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:56 smithi167 conmon[60316]: debug 2022-01-31T21:57:56.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.511660+0000) 2022-01-31T21:57:57.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[41853]: debug 2022-01-31T21:57:56.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.224988+0000) 2022-01-31T21:57:57.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:56 smithi167 conmon[54076]: debug 2022-01-31T21:57:56.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.511663+0000) 2022-01-31T21:57:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:56 smithi167 conmon[49112]: debug 2022-01-31T21:57:56.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.848336+0000) 2022-01-31T21:57:57.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:56 smithi167 conmon[49112]: debug 2022-01-31T21:57:56.936+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.936556+0000) 2022-01-31T21:57:57.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:56 smithi167 conmon[54076]: debug 2022-01-31T21:57:56.935+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.936413+0000) 2022-01-31T21:57:57.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:56 smithi167 conmon[60316]: debug 2022-01-31T21:57:56.937+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.937749+0000) 2022-01-31T21:57:57.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[35325]: debug 2022-01-31T21:57:56.954+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171654 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:57:57.180 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[41853]: debug 2022-01-31T21:57:56.936+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.937263+0000) 2022-01-31T21:57:57.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[51620]: debug 2022-01-31T21:57:56.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.867129+0000) 2022-01-31T21:57:57.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[51620]: debug 2022-01-31T21:57:56.936+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.936794+0000) 2022-01-31T21:57:57.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:56 smithi171 conmon[46715]: debug 2022-01-31T21:57:56.937+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:56.937990+0000) 2022-01-31T21:57:57.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:57 smithi171 conmon[46715]: debug 2022-01-31T21:57:57.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:57.020307+0000) 2022-01-31T21:57:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:57 smithi171 conmon[41853]: debug 2022-01-31T21:57:57.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:57.225104+0000) 2022-01-31T21:57:57.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:57 smithi167 conmon[60316]: debug 2022-01-31T21:57:57.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:57.511873+0000) 2022-01-31T21:57:57.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:57 smithi167 conmon[54076]: debug 2022-01-31T21:57:57.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:57.511873+0000) 2022-01-31T21:57:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:57 smithi167 conmon[49112]: debug 2022-01-31T21:57:57.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:57.848560+0000) 2022-01-31T21:57:58.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:58 smithi171 conmon[46715]: debug 2022-01-31T21:57:58.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:58.020467+0000) 2022-01-31T21:57:58.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:57 smithi171 conmon[51620]: debug 2022-01-31T21:57:57.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:57.867322+0000) 2022-01-31T21:57:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:58 smithi171 conmon[41853]: debug 2022-01-31T21:57:58.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:58.225251+0000) 2022-01-31T21:57:58.716 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:58 smithi167 conmon[54076]: debug 2022-01-31T21:57:58.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:58.512008+0000) 2022-01-31T21:57:58.717 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:58 smithi167 conmon[60316]: debug 2022-01-31T21:57:58.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:58.512008+0000) 2022-01-31T21:57:59.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:58 smithi167 conmon[49112]: debug 2022-01-31T21:57:58.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:58.848699+0000) 2022-01-31T21:57:59.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:57:59 smithi171 conmon[46715]: debug 2022-01-31T21:57:59.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:59.020626+0000) 2022-01-31T21:57:59.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:58 smithi171 conmon[51620]: debug 2022-01-31T21:57:58.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:58.867521+0000) 2022-01-31T21:57:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:57:59 smithi171 conmon[41853]: debug 2022-01-31T21:57:59.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:59.225428+0000) 2022-01-31T21:57:59.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:57:59 smithi167 conmon[60316]: debug 2022-01-31T21:57:59.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:59.512227+0000) 2022-01-31T21:57:59.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:57:59 smithi167 conmon[54076]: debug 2022-01-31T21:57:59.511+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:59.512226+0000) 2022-01-31T21:58:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:57:59 smithi167 conmon[49112]: debug 2022-01-31T21:57:59.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:59.848848+0000) 2022-01-31T21:58:00.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:00 smithi171 conmon[46715]: debug 2022-01-31T21:58:00.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:00.020847+0000) 2022-01-31T21:58:00.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:57:59 smithi171 conmon[51620]: debug 2022-01-31T21:57:59.866+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:57:59.867706+0000) 2022-01-31T21:58:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:00 smithi171 conmon[41853]: debug 2022-01-31T21:58:00.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:00.225618+0000) 2022-01-31T21:58:00.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:00 smithi167 conmon[60316]: debug 2022-01-31T21:58:00.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:00.512448+0000) 2022-01-31T21:58:00.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:00 smithi167 conmon[54076]: debug 2022-01-31T21:58:00.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:00.512443+0000) 2022-01-31T21:58:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:00 smithi167 conmon[49112]: debug 2022-01-31T21:58:00.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:00.849067+0000) 2022-01-31T21:58:01.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[46715]: debug 2022-01-31T21:58:01.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.020979+0000) 2022-01-31T21:58:01.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:00 smithi171 conmon[51620]: debug 2022-01-31T21:58:00.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:00.867842+0000) 2022-01-31T21:58:01.567 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:01 smithi167 conmon[60316]: debug 2022-01-31T21:58:01.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.512705+0000) 2022-01-31T21:58:01.567 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:01 smithi167 conmon[54076]: debug 2022-01-31T21:58:01.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.512703+0000) 2022-01-31T21:58:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[41853]: debug 2022-01-31T21:58:01.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.225793+0000) 2022-01-31T21:58:02.168 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:01 smithi167 conmon[60316]: debug 2022-01-31T21:58:01.957+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.957582+0000) 2022-01-31T21:58:02.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:01 smithi167 conmon[49112]: debug 2022-01-31T21:58:01.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.849241+0000) 2022-01-31T21:58:02.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:01 smithi167 conmon[49112]: debug 2022-01-31T21:58:01.956+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.957242+0000) 2022-01-31T21:58:02.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:01 smithi167 conmon[54076]: debug 2022-01-31T21:58:01.956+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.957403+0000) 2022-01-31T21:58:02.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[35325]: debug 2022-01-31T21:58:01.974+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171764 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:02.181 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[41853]: debug 2022-01-31T21:58:01.956+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.957457+0000) 2022-01-31T21:58:02.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[46715]: debug 2022-01-31T21:58:01.958+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.959052+0000) 2022-01-31T21:58:02.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:02 smithi171 conmon[46715]: debug 2022-01-31T21:58:02.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:02.021075+0000) 2022-01-31T21:58:02.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[51620]: debug 2022-01-31T21:58:01.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.868048+0000) 2022-01-31T21:58:02.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:01 smithi171 conmon[51620]: debug 2022-01-31T21:58:01.956+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:01.957140+0000) 2022-01-31T21:58:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:02 smithi171 conmon[41853]: debug 2022-01-31T21:58:02.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:02.225945+0000) 2022-01-31T21:58:02.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:02 smithi167 conmon[60316]: debug 2022-01-31T21:58:02.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:02.512869+0000) 2022-01-31T21:58:02.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:02 smithi167 conmon[54076]: debug 2022-01-31T21:58:02.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:02.512891+0000) 2022-01-31T21:58:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:02 smithi167 conmon[49112]: debug 2022-01-31T21:58:02.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:02.849443+0000) 2022-01-31T21:58:03.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:03 smithi171 conmon[46715]: debug 2022-01-31T21:58:03.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:03.021162+0000) 2022-01-31T21:58:03.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:02 smithi171 conmon[51620]: debug 2022-01-31T21:58:02.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:02.868232+0000) 2022-01-31T21:58:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:03 smithi171 conmon[41853]: debug 2022-01-31T21:58:03.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:03.226138+0000) 2022-01-31T21:58:03.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:03 smithi167 conmon[54076]: debug 2022-01-31T21:58:03.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:03.513033+0000) 2022-01-31T21:58:03.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:03 smithi167 conmon[60316]: debug 2022-01-31T21:58:03.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:03.513033+0000) 2022-01-31T21:58:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:03 smithi167 conmon[49112]: debug 2022-01-31T21:58:03.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:03.849712+0000) 2022-01-31T21:58:04.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:04 smithi171 conmon[46715]: debug 2022-01-31T21:58:04.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:04.021261+0000) 2022-01-31T21:58:04.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:03 smithi171 conmon[51620]: debug 2022-01-31T21:58:03.867+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:03.868418+0000) 2022-01-31T21:58:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:04 smithi171 conmon[41853]: debug 2022-01-31T21:58:04.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:04.226327+0000) 2022-01-31T21:58:04.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:04 smithi167 conmon[54076]: debug 2022-01-31T21:58:04.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:04.513242+0000) 2022-01-31T21:58:04.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:04 smithi167 conmon[60316]: debug 2022-01-31T21:58:04.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:04.513245+0000) 2022-01-31T21:58:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:04 smithi167 conmon[49112]: debug 2022-01-31T21:58:04.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:04.849891+0000) 2022-01-31T21:58:05.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:05 smithi171 conmon[46715]: debug 2022-01-31T21:58:05.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:05.021433+0000) 2022-01-31T21:58:05.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:04 smithi171 conmon[51620]: debug 2022-01-31T21:58:04.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:04.868595+0000) 2022-01-31T21:58:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:05 smithi171 conmon[41853]: debug 2022-01-31T21:58:05.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:05.226514+0000) 2022-01-31T21:58:05.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:05 smithi167 conmon[60316]: debug 2022-01-31T21:58:05.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:05.513480+0000) 2022-01-31T21:58:05.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:05 smithi167 conmon[54076]: debug 2022-01-31T21:58:05.512+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:05.513482+0000) 2022-01-31T21:58:06.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:05 smithi167 conmon[49112]: debug 2022-01-31T21:58:05.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:05.850064+0000) 2022-01-31T21:58:06.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[46715]: debug 2022-01-31T21:58:06.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.021548+0000) 2022-01-31T21:58:06.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:05 smithi171 conmon[51620]: debug 2022-01-31T21:58:05.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:05.868767+0000) 2022-01-31T21:58:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[41853]: debug 2022-01-31T21:58:06.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.226692+0000) 2022-01-31T21:58:06.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:06 smithi167 conmon[54076]: debug 2022-01-31T21:58:06.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.513690+0000) 2022-01-31T21:58:06.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:06 smithi167 conmon[60316]: debug 2022-01-31T21:58:06.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.513687+0000) 2022-01-31T21:58:07.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:06 smithi167 conmon[54076]: debug 2022-01-31T21:58:06.977+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.978089+0000) 2022-01-31T21:58:07.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:06 smithi167 conmon[60316]: debug 2022-01-31T21:58:06.977+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.977894+0000) 2022-01-31T21:58:07.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:06 smithi167 conmon[49112]: debug 2022-01-31T21:58:06.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.850251+0000) 2022-01-31T21:58:07.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:06 smithi167 conmon[49112]: debug 2022-01-31T21:58:06.978+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.978895+0000) 2022-01-31T21:58:07.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[35325]: debug 2022-01-31T21:58:06.995+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171875 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:07.182 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[41853]: debug 2022-01-31T21:58:06.977+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.978642+0000) 2022-01-31T21:58:07.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[46715]: debug 2022-01-31T21:58:06.978+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.979328+0000) 2022-01-31T21:58:07.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:07 smithi171 conmon[46715]: debug 2022-01-31T21:58:07.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:07.021712+0000) 2022-01-31T21:58:07.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[51620]: debug 2022-01-31T21:58:06.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.868926+0000) 2022-01-31T21:58:07.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:06 smithi171 conmon[51620]: debug 2022-01-31T21:58:06.976+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:06.977337+0000) 2022-01-31T21:58:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:07 smithi171 conmon[41853]: debug 2022-01-31T21:58:07.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:07.226844+0000) 2022-01-31T21:58:07.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:07 smithi167 conmon[60316]: debug 2022-01-31T21:58:07.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:07.513892+0000) 2022-01-31T21:58:07.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:07 smithi167 conmon[54076]: debug 2022-01-31T21:58:07.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:07.513890+0000) 2022-01-31T21:58:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:07 smithi167 conmon[49112]: debug 2022-01-31T21:58:07.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:07.850427+0000) 2022-01-31T21:58:08.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:08 smithi171 conmon[46715]: debug 2022-01-31T21:58:08.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:08.021831+0000) 2022-01-31T21:58:08.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:07 smithi171 conmon[51620]: debug 2022-01-31T21:58:07.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:07.869122+0000) 2022-01-31T21:58:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:08 smithi171 conmon[41853]: debug 2022-01-31T21:58:08.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:08.226995+0000) 2022-01-31T21:58:08.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:08 smithi167 conmon[54076]: debug 2022-01-31T21:58:08.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:08.514057+0000) 2022-01-31T21:58:08.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:08 smithi167 conmon[60316]: debug 2022-01-31T21:58:08.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:08.514057+0000) 2022-01-31T21:58:09.020 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:08 smithi171 conmon[35325]: debug 2022-01-31T21:58:08.768+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:58:09.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:08 smithi171 conmon[51620]: debug 2022-01-31T21:58:08.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:08.869320+0000) 2022-01-31T21:58:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:08 smithi167 conmon[49112]: debug 2022-01-31T21:58:08.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:08.850591+0000) 2022-01-31T21:58:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:09 smithi171 conmon[41853]: debug 2022-01-31T21:58:09.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:09.227177+0000) 2022-01-31T21:58:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:09 smithi171 conmon[46715]: debug 2022-01-31T21:58:09.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:09.022022+0000) 2022-01-31T21:58:09.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:09 smithi167 conmon[60316]: debug 2022-01-31T21:58:09.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:09.514244+0000) 2022-01-31T21:58:09.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:09 smithi167 conmon[54076]: debug 2022-01-31T21:58:09.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:09.514242+0000) 2022-01-31T21:58:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:09 smithi167 conmon[49112]: debug 2022-01-31T21:58:09.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:09.850745+0000) 2022-01-31T21:58:10.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:10 smithi171 conmon[46715]: debug 2022-01-31T21:58:10.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:10.022194+0000) 2022-01-31T21:58:10.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:09 smithi171 conmon[51620]: debug 2022-01-31T21:58:09.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:09.869510+0000) 2022-01-31T21:58:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:10 smithi171 conmon[41853]: debug 2022-01-31T21:58:10.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:10.227346+0000) 2022-01-31T21:58:10.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:10 smithi167 conmon[54076]: debug 2022-01-31T21:58:10.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:10.514410+0000) 2022-01-31T21:58:10.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:10 smithi167 conmon[60316]: debug 2022-01-31T21:58:10.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:10.514418+0000) 2022-01-31T21:58:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:10 smithi167 conmon[49112]: debug 2022-01-31T21:58:10.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:10.850937+0000) 2022-01-31T21:58:11.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:11 smithi171 conmon[46715]: debug 2022-01-31T21:58:11.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.022369+0000) 2022-01-31T21:58:11.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:10 smithi171 conmon[51620]: debug 2022-01-31T21:58:10.868+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:10.869682+0000) 2022-01-31T21:58:11.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:11 smithi171 conmon[41853]: debug 2022-01-31T21:58:11.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.227505+0000) 2022-01-31T21:58:11.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:11 smithi167 conmon[60316]: debug 2022-01-31T21:58:11.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.514630+0000) 2022-01-31T21:58:11.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:11 smithi167 conmon[54076]: debug 2022-01-31T21:58:11.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.514625+0000) 2022-01-31T21:58:12.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:11 smithi167 conmon[54076]: debug 2022-01-31T21:58:11.996+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.998106+0000) 2022-01-31T21:58:12.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:11 smithi167 conmon[60316]: debug 2022-01-31T21:58:11.996+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.998368+0000) 2022-01-31T21:58:12.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:11 smithi167 conmon[49112]: debug 2022-01-31T21:58:11.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.851141+0000) 2022-01-31T21:58:12.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:11 smithi167 conmon[49112]: debug 2022-01-31T21:58:11.997+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.998724+0000) 2022-01-31T21:58:12.182 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:12 smithi171 conmon[35325]: debug 2022-01-31T21:58:12.015+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 171987 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:12.183 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:11 smithi171 conmon[41853]: debug 2022-01-31T21:58:11.999+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.999832+0000) 2022-01-31T21:58:12.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:11 smithi171 conmon[51620]: debug 2022-01-31T21:58:11.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.869864+0000) 2022-01-31T21:58:12.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:11 smithi171 conmon[51620]: debug 2022-01-31T21:58:11.997+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.997920+0000) 2022-01-31T21:58:12.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:11 smithi171 conmon[46715]: debug 2022-01-31T21:58:11.998+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:11.999353+0000) 2022-01-31T21:58:12.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:12 smithi171 conmon[46715]: debug 2022-01-31T21:58:12.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:12.022501+0000) 2022-01-31T21:58:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:12 smithi171 conmon[41853]: debug 2022-01-31T21:58:12.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:12.227693+0000) 2022-01-31T21:58:12.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:12 smithi167 conmon[54076]: debug 2022-01-31T21:58:12.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:12.514828+0000) 2022-01-31T21:58:12.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:12 smithi167 conmon[60316]: debug 2022-01-31T21:58:12.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:12.514829+0000) 2022-01-31T21:58:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:12 smithi167 conmon[49112]: debug 2022-01-31T21:58:12.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:12.851347+0000) 2022-01-31T21:58:13.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:13 smithi171 conmon[46715]: debug 2022-01-31T21:58:13.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:13.022652+0000) 2022-01-31T21:58:13.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:12 smithi171 conmon[51620]: debug 2022-01-31T21:58:12.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:12.870070+0000) 2022-01-31T21:58:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:13 smithi171 conmon[41853]: debug 2022-01-31T21:58:13.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:13.227915+0000) 2022-01-31T21:58:13.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:13 smithi167 conmon[54076]: debug 2022-01-31T21:58:13.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:13.514979+0000) 2022-01-31T21:58:13.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:13 smithi167 conmon[60316]: debug 2022-01-31T21:58:13.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:13.514980+0000) 2022-01-31T21:58:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:13 smithi167 conmon[49112]: debug 2022-01-31T21:58:13.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:13.851535+0000) 2022-01-31T21:58:14.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:14 smithi171 conmon[46715]: debug 2022-01-31T21:58:14.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:14.022836+0000) 2022-01-31T21:58:14.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:13 smithi171 conmon[51620]: debug 2022-01-31T21:58:13.869+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:13.870238+0000) 2022-01-31T21:58:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:14 smithi171 conmon[41853]: debug 2022-01-31T21:58:14.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:14.228095+0000) 2022-01-31T21:58:14.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:14 smithi167 conmon[54076]: debug 2022-01-31T21:58:14.513+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:14.515182+0000) 2022-01-31T21:58:14.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:14 smithi167 conmon[60316]: debug 2022-01-31T21:58:14.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:14.515194+0000) 2022-01-31T21:58:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:14 smithi167 conmon[49112]: debug 2022-01-31T21:58:14.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:14.851751+0000) 2022-01-31T21:58:15.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:14 smithi171 conmon[51620]: debug 2022-01-31T21:58:14.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:14.870433+0000) 2022-01-31T21:58:15.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:15 smithi171 conmon[46715]: debug 2022-01-31T21:58:15.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:15.023031+0000) 2022-01-31T21:58:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:15 smithi171 conmon[41853]: debug 2022-01-31T21:58:15.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:15.228280+0000) 2022-01-31T21:58:15.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:15 smithi167 conmon[54076]: debug 2022-01-31T21:58:15.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:15.515378+0000) 2022-01-31T21:58:15.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:15 smithi167 conmon[60316]: debug 2022-01-31T21:58:15.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:15.515379+0000) 2022-01-31T21:58:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:15 smithi167 conmon[49112]: debug 2022-01-31T21:58:15.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:15.851935+0000) 2022-01-31T21:58:16.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:16 smithi171 conmon[46715]: debug 2022-01-31T21:58:16.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:16.023211+0000) 2022-01-31T21:58:16.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:15 smithi171 conmon[51620]: debug 2022-01-31T21:58:15.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:15.870623+0000) 2022-01-31T21:58:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:16 smithi171 conmon[41853]: debug 2022-01-31T21:58:16.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:16.228455+0000) 2022-01-31T21:58:16.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:16 smithi167 conmon[54076]: debug 2022-01-31T21:58:16.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:16.515586+0000) 2022-01-31T21:58:16.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:16 smithi167 conmon[60316]: debug 2022-01-31T21:58:16.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:16.515584+0000) 2022-01-31T21:58:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:16 smithi167 conmon[49112]: debug 2022-01-31T21:58:16.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:16.852124+0000) 2022-01-31T21:58:17.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:17 smithi167 conmon[49112]: debug 2022-01-31T21:58:17.018+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.019631+0000) 2022-01-31T21:58:17.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:17 smithi167 conmon[54076]: debug 2022-01-31T21:58:17.016+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.018485+0000) 2022-01-31T21:58:17.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:17 smithi167 conmon[60316]: debug 2022-01-31T21:58:17.017+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.019045+0000) 2022-01-31T21:58:17.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[35325]: debug 2022-01-31T21:58:17.035+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172099 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:17.183 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[41853]: debug 2022-01-31T21:58:17.018+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.019442+0000) 2022-01-31T21:58:17.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[46715]: debug 2022-01-31T21:58:17.019+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.020253+0000) 2022-01-31T21:58:17.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[46715]: debug 2022-01-31T21:58:17.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.023306+0000) 2022-01-31T21:58:17.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:16 smithi171 conmon[51620]: debug 2022-01-31T21:58:16.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:16.870815+0000) 2022-01-31T21:58:17.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[51620]: debug 2022-01-31T21:58:17.018+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.019531+0000) 2022-01-31T21:58:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[41853]: debug 2022-01-31T21:58:17.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.228584+0000) 2022-01-31T21:58:17.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:17 smithi167 conmon[54076]: debug 2022-01-31T21:58:17.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.515749+0000) 2022-01-31T21:58:17.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:17 smithi167 conmon[60316]: debug 2022-01-31T21:58:17.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.515787+0000) 2022-01-31T21:58:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:17 smithi167 conmon[49112]: debug 2022-01-31T21:58:17.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.852317+0000) 2022-01-31T21:58:18.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:18 smithi171 conmon[46715]: debug 2022-01-31T21:58:18.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:18.023470+0000) 2022-01-31T21:58:18.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:17 smithi171 conmon[51620]: debug 2022-01-31T21:58:17.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:17.870997+0000) 2022-01-31T21:58:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:18 smithi171 conmon[41853]: debug 2022-01-31T21:58:18.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:18.228777+0000) 2022-01-31T21:58:18.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:18 smithi167 conmon[54076]: debug 2022-01-31T21:58:18.514+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:18.515904+0000) 2022-01-31T21:58:18.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:18 smithi167 conmon[60316]: debug 2022-01-31T21:58:18.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:18.515927+0000) 2022-01-31T21:58:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:18 smithi167 conmon[49112]: debug 2022-01-31T21:58:18.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:18.852496+0000) 2022-01-31T21:58:19.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:19 smithi171 conmon[46715]: debug 2022-01-31T21:58:19.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:19.023626+0000) 2022-01-31T21:58:19.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:18 smithi171 conmon[51620]: debug 2022-01-31T21:58:18.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:18.871168+0000) 2022-01-31T21:58:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:19 smithi171 conmon[41853]: debug 2022-01-31T21:58:19.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:19.228979+0000) 2022-01-31T21:58:19.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:19 smithi167 conmon[54076]: debug 2022-01-31T21:58:19.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:19.516114+0000) 2022-01-31T21:58:19.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:19 smithi167 conmon[60316]: debug 2022-01-31T21:58:19.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:19.516119+0000) 2022-01-31T21:58:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:19 smithi167 conmon[49112]: debug 2022-01-31T21:58:19.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:19.852678+0000) 2022-01-31T21:58:20.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:20 smithi171 conmon[46715]: debug 2022-01-31T21:58:20.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:20.023792+0000) 2022-01-31T21:58:20.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:19 smithi171 conmon[51620]: debug 2022-01-31T21:58:19.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:19.871381+0000) 2022-01-31T21:58:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:20 smithi171 conmon[41853]: debug 2022-01-31T21:58:20.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:20.229140+0000) 2022-01-31T21:58:20.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:20 smithi167 conmon[54076]: debug 2022-01-31T21:58:20.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:20.516301+0000) 2022-01-31T21:58:20.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:20 smithi167 conmon[60316]: debug 2022-01-31T21:58:20.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:20.516238+0000) 2022-01-31T21:58:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:20 smithi167 conmon[49112]: debug 2022-01-31T21:58:20.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:20.852834+0000) 2022-01-31T21:58:21.183 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:21 smithi171 conmon[46715]: debug 2022-01-31T21:58:21.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:21.023984+0000) 2022-01-31T21:58:21.184 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:20 smithi171 conmon[51620]: debug 2022-01-31T21:58:20.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:20.871555+0000) 2022-01-31T21:58:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:21 smithi171 conmon[41853]: debug 2022-01-31T21:58:21.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:21.229304+0000) 2022-01-31T21:58:21.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:21 smithi167 conmon[54076]: debug 2022-01-31T21:58:21.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:21.516465+0000) 2022-01-31T21:58:21.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:21 smithi167 conmon[60316]: debug 2022-01-31T21:58:21.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:21.516428+0000) 2022-01-31T21:58:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:21 smithi167 conmon[49112]: debug 2022-01-31T21:58:21.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:21.853008+0000) 2022-01-31T21:58:22.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:22 smithi167 conmon[49112]: debug 2022-01-31T21:58:22.037+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.038993+0000) 2022-01-31T21:58:22.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:22 smithi167 conmon[54076]: debug 2022-01-31T21:58:22.036+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.038422+0000) 2022-01-31T21:58:22.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:22 smithi167 conmon[60316]: debug 2022-01-31T21:58:22.037+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.038550+0000) 2022-01-31T21:58:22.184 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[35325]: debug 2022-01-31T21:58:22.055+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172210 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:22.185 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[41853]: debug 2022-01-31T21:58:22.038+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.039168+0000) 2022-01-31T21:58:22.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[46715]: debug 2022-01-31T21:58:22.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.024176+0000) 2022-01-31T21:58:22.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[46715]: debug 2022-01-31T21:58:22.038+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.039026+0000) 2022-01-31T21:58:22.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:21 smithi171 conmon[51620]: debug 2022-01-31T21:58:21.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:21.871713+0000) 2022-01-31T21:58:22.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[51620]: debug 2022-01-31T21:58:22.038+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.038747+0000) 2022-01-31T21:58:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[41853]: debug 2022-01-31T21:58:22.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.229461+0000) 2022-01-31T21:58:22.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:22 smithi167 conmon[54076]: debug 2022-01-31T21:58:22.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.516689+0000) 2022-01-31T21:58:22.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:22 smithi167 conmon[60316]: debug 2022-01-31T21:58:22.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.516616+0000) 2022-01-31T21:58:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:22 smithi167 conmon[49112]: debug 2022-01-31T21:58:22.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.853169+0000) 2022-01-31T21:58:23.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:23 smithi171 conmon[46715]: debug 2022-01-31T21:58:23.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:23.024358+0000) 2022-01-31T21:58:23.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:22 smithi171 conmon[51620]: debug 2022-01-31T21:58:22.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:22.871935+0000) 2022-01-31T21:58:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:23 smithi171 conmon[41853]: debug 2022-01-31T21:58:23.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:23.229643+0000) 2022-01-31T21:58:23.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:23 smithi167 conmon[54076]: debug 2022-01-31T21:58:23.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:23.516835+0000) 2022-01-31T21:58:23.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:23 smithi167 conmon[60316]: debug 2022-01-31T21:58:23.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:23.516794+0000) 2022-01-31T21:58:24.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:23 smithi171 conmon[51620]: debug 2022-01-31T21:58:23.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:23.872036+0000) 2022-01-31T21:58:24.024 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:23 smithi171 conmon[35325]: debug 2022-01-31T21:58:23.769+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:58:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:23 smithi167 conmon[49112]: debug 2022-01-31T21:58:23.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:23.853322+0000) 2022-01-31T21:58:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:24 smithi171 conmon[41853]: debug 2022-01-31T21:58:24.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:24.229839+0000) 2022-01-31T21:58:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:24 smithi171 conmon[46715]: debug 2022-01-31T21:58:24.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:24.024534+0000) 2022-01-31T21:58:24.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:24 smithi167 conmon[54076]: debug 2022-01-31T21:58:24.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:24.517010+0000) 2022-01-31T21:58:24.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:24 smithi167 conmon[60316]: debug 2022-01-31T21:58:24.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:24.516989+0000) 2022-01-31T21:58:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:24 smithi167 conmon[49112]: debug 2022-01-31T21:58:24.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:24.853524+0000) 2022-01-31T21:58:25.184 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:25 smithi171 conmon[46715]: debug 2022-01-31T21:58:25.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:25.024687+0000) 2022-01-31T21:58:25.185 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:24 smithi171 conmon[51620]: debug 2022-01-31T21:58:24.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:24.872178+0000) 2022-01-31T21:58:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:25 smithi171 conmon[41853]: debug 2022-01-31T21:58:25.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:25.230020+0000) 2022-01-31T21:58:25.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:25 smithi167 conmon[54076]: debug 2022-01-31T21:58:25.515+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:25.517190+0000) 2022-01-31T21:58:25.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:25 smithi167 conmon[60316]: debug 2022-01-31T21:58:25.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:25.517146+0000) 2022-01-31T21:58:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:25 smithi167 conmon[49112]: debug 2022-01-31T21:58:25.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:25.853736+0000) 2022-01-31T21:58:26.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:26 smithi171 conmon[46715]: debug 2022-01-31T21:58:26.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:26.024872+0000) 2022-01-31T21:58:26.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:25 smithi171 conmon[51620]: debug 2022-01-31T21:58:25.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:25.872372+0000) 2022-01-31T21:58:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:26 smithi171 conmon[41853]: debug 2022-01-31T21:58:26.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:26.230210+0000) 2022-01-31T21:58:26.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:26 smithi167 conmon[60316]: debug 2022-01-31T21:58:26.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:26.517333+0000) 2022-01-31T21:58:26.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:26 smithi167 conmon[54076]: debug 2022-01-31T21:58:26.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:26.517379+0000) 2022-01-31T21:58:27.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:27 smithi167 conmon[54076]: debug 2022-01-31T21:58:27.064+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.065756+0000) 2022-01-31T21:58:27.436 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:27 smithi167 conmon[60316]: debug 2022-01-31T21:58:27.064+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.065939+0000) 2022-01-31T21:58:27.437 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:26 smithi167 conmon[49112]: debug 2022-01-31T21:58:26.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:26.853933+0000) 2022-01-31T21:58:27.437 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:27 smithi167 conmon[49112]: debug 2022-01-31T21:58:27.057+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.059002+0000) 2022-01-31T21:58:27.439 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[35325]: debug 2022-01-31T21:58:27.081+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172320 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:27.440 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[41853]: debug 2022-01-31T21:58:27.057+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.059067+0000) 2022-01-31T21:58:27.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[46715]: debug 2022-01-31T21:58:27.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.024993+0000) 2022-01-31T21:58:27.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[46715]: debug 2022-01-31T21:58:27.056+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.058502+0000) 2022-01-31T21:58:27.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:26 smithi171 conmon[51620]: debug 2022-01-31T21:58:26.870+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:26.872553+0000) 2022-01-31T21:58:27.443 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[51620]: debug 2022-01-31T21:58:27.056+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.058376+0000) 2022-01-31T21:58:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[41853]: debug 2022-01-31T21:58:27.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.230411+0000) 2022-01-31T21:58:27.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:27 smithi167 conmon[60316]: debug 2022-01-31T21:58:27.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.517551+0000) 2022-01-31T21:58:27.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:27 smithi167 conmon[54076]: debug 2022-01-31T21:58:27.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.517597+0000) 2022-01-31T21:58:28.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:27 smithi167 conmon[49112]: debug 2022-01-31T21:58:27.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.854090+0000) 2022-01-31T21:58:28.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:28 smithi171 conmon[46715]: debug 2022-01-31T21:58:28.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:28.025173+0000) 2022-01-31T21:58:28.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:27 smithi171 conmon[51620]: debug 2022-01-31T21:58:27.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:27.872785+0000) 2022-01-31T21:58:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:28 smithi171 conmon[41853]: debug 2022-01-31T21:58:28.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:28.230571+0000) 2022-01-31T21:58:28.640 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:28 smithi167 conmon[54076]: debug 2022-01-31T21:58:28.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:28.517707+0000) 2022-01-31T21:58:28.641 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:28 smithi167 conmon[60316]: debug 2022-01-31T21:58:28.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:28.517680+0000) 2022-01-31T21:58:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:28 smithi167 conmon[49112]: debug 2022-01-31T21:58:28.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:28.854269+0000) 2022-01-31T21:58:29.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:29 smithi171 conmon[46715]: debug 2022-01-31T21:58:29.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:29.025342+0000) 2022-01-31T21:58:29.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:28 smithi171 conmon[51620]: debug 2022-01-31T21:58:28.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:28.872897+0000) 2022-01-31T21:58:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:29 smithi171 conmon[41853]: debug 2022-01-31T21:58:29.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:29.230770+0000) 2022-01-31T21:58:29.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:29 smithi167 conmon[54076]: debug 2022-01-31T21:58:29.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:29.517877+0000) 2022-01-31T21:58:29.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:29 smithi167 conmon[60316]: debug 2022-01-31T21:58:29.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:29.517867+0000) 2022-01-31T21:58:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:29 smithi167 conmon[49112]: debug 2022-01-31T21:58:29.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:29.854454+0000) 2022-01-31T21:58:30.185 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:30 smithi171 conmon[46715]: debug 2022-01-31T21:58:30.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:30.025532+0000) 2022-01-31T21:58:30.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:29 smithi171 conmon[51620]: debug 2022-01-31T21:58:29.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:29.873092+0000) 2022-01-31T21:58:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:30 smithi171 conmon[41853]: debug 2022-01-31T21:58:30.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:30.230952+0000) 2022-01-31T21:58:30.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:30 smithi167 conmon[54076]: debug 2022-01-31T21:58:30.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:30.518067+0000) 2022-01-31T21:58:30.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:30 smithi167 conmon[60316]: debug 2022-01-31T21:58:30.517+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:30.518059+0000) 2022-01-31T21:58:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:30 smithi167 conmon[49112]: debug 2022-01-31T21:58:30.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:30.854656+0000) 2022-01-31T21:58:31.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:31 smithi171 conmon[46715]: debug 2022-01-31T21:58:31.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:31.025698+0000) 2022-01-31T21:58:31.186 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:30 smithi171 conmon[51620]: debug 2022-01-31T21:58:30.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:30.873320+0000) 2022-01-31T21:58:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:31 smithi171 conmon[41853]: debug 2022-01-31T21:58:31.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:31.231135+0000) 2022-01-31T21:58:31.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:31 smithi167 conmon[54076]: debug 2022-01-31T21:58:31.516+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:31.518282+0000) 2022-01-31T21:58:31.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:31 smithi167 conmon[60316]: debug 2022-01-31T21:58:31.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:31.518278+0000) 2022-01-31T21:58:32.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:31 smithi167 conmon[49112]: debug 2022-01-31T21:58:31.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:31.854878+0000) 2022-01-31T21:58:32.186 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[35325]: debug 2022-01-31T21:58:32.102+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172444 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:32.187 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[41853]: debug 2022-01-31T21:58:32.085+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.086973+0000) 2022-01-31T21:58:32.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[46715]: debug 2022-01-31T21:58:32.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.025846+0000) 2022-01-31T21:58:32.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[46715]: debug 2022-01-31T21:58:32.084+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.086044+0000) 2022-01-31T21:58:32.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:31 smithi171 conmon[51620]: debug 2022-01-31T21:58:31.871+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:31.873479+0000) 2022-01-31T21:58:32.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[51620]: debug 2022-01-31T21:58:32.083+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.084959+0000) 2022-01-31T21:58:32.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:32 smithi167 conmon[54076]: debug 2022-01-31T21:58:32.084+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.086237+0000) 2022-01-31T21:58:32.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:32 smithi167 conmon[49112]: debug 2022-01-31T21:58:32.084+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.086470+0000) 2022-01-31T21:58:32.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:32 smithi167 conmon[60316]: debug 2022-01-31T21:58:32.083+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.085221+0000) 2022-01-31T21:58:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[41853]: debug 2022-01-31T21:58:32.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.231317+0000) 2022-01-31T21:58:32.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:32 smithi167 conmon[60316]: debug 2022-01-31T21:58:32.517+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.518465+0000) 2022-01-31T21:58:32.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:32 smithi167 conmon[54076]: debug 2022-01-31T21:58:32.517+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.518488+0000) 2022-01-31T21:58:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:32 smithi167 conmon[49112]: debug 2022-01-31T21:58:32.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.855054+0000) 2022-01-31T21:58:33.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:33 smithi171 conmon[46715]: debug 2022-01-31T21:58:33.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:33.026036+0000) 2022-01-31T21:58:33.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:32 smithi171 conmon[51620]: debug 2022-01-31T21:58:32.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:32.873678+0000) 2022-01-31T21:58:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:33 smithi171 conmon[41853]: debug 2022-01-31T21:58:33.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:33.231490+0000) 2022-01-31T21:58:33.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:33 smithi167 conmon[54076]: debug 2022-01-31T21:58:33.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:33.518664+0000) 2022-01-31T21:58:33.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:33 smithi167 conmon[60316]: debug 2022-01-31T21:58:33.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:33.518598+0000) 2022-01-31T21:58:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:33 smithi167 conmon[49112]: debug 2022-01-31T21:58:33.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:33.855192+0000) 2022-01-31T21:58:34.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:34 smithi171 conmon[46715]: debug 2022-01-31T21:58:34.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:34.026221+0000) 2022-01-31T21:58:34.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:33 smithi171 conmon[51620]: debug 2022-01-31T21:58:33.872+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:33.873819+0000) 2022-01-31T21:58:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:34 smithi171 conmon[41853]: debug 2022-01-31T21:58:34.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:34.231679+0000) 2022-01-31T21:58:34.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:34 smithi167 conmon[54076]: debug 2022-01-31T21:58:34.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:34.518829+0000) 2022-01-31T21:58:34.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:34 smithi167 conmon[60316]: debug 2022-01-31T21:58:34.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:34.518792+0000) 2022-01-31T21:58:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:34 smithi167 conmon[49112]: debug 2022-01-31T21:58:34.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:34.855379+0000) 2022-01-31T21:58:35.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:35 smithi171 conmon[46715]: debug 2022-01-31T21:58:35.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:35.026377+0000) 2022-01-31T21:58:35.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:34 smithi171 conmon[51620]: debug 2022-01-31T21:58:34.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:34.873950+0000) 2022-01-31T21:58:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:35 smithi171 conmon[41853]: debug 2022-01-31T21:58:35.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:35.231860+0000) 2022-01-31T21:58:35.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:35 smithi167 conmon[60316]: debug 2022-01-31T21:58:35.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:35.518998+0000) 2022-01-31T21:58:35.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:35 smithi167 conmon[54076]: debug 2022-01-31T21:58:35.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:35.519030+0000) 2022-01-31T21:58:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:35 smithi167 conmon[49112]: debug 2022-01-31T21:58:35.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:35.855587+0000) 2022-01-31T21:58:36.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:36 smithi171 conmon[46715]: debug 2022-01-31T21:58:36.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:36.026569+0000) 2022-01-31T21:58:36.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:35 smithi171 conmon[51620]: debug 2022-01-31T21:58:35.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:35.874166+0000) 2022-01-31T21:58:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:36 smithi171 conmon[41853]: debug 2022-01-31T21:58:36.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:36.232046+0000) 2022-01-31T21:58:36.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:36 smithi167 conmon[60316]: debug 2022-01-31T21:58:36.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:36.519213+0000) 2022-01-31T21:58:36.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:36 smithi167 conmon[54076]: debug 2022-01-31T21:58:36.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:36.519245+0000) 2022-01-31T21:58:37.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:36 smithi167 conmon[49112]: debug 2022-01-31T21:58:36.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:36.855746+0000) 2022-01-31T21:58:37.123 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[41853]: debug 2022-01-31T21:58:37.105+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.106226+0000) 2022-01-31T21:58:37.124 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[46715]: debug 2022-01-31T21:58:37.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.026770+0000) 2022-01-31T21:58:37.124 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[46715]: debug 2022-01-31T21:58:37.105+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.106569+0000) 2022-01-31T21:58:37.125 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:36 smithi171 conmon[51620]: debug 2022-01-31T21:58:36.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:36.874375+0000) 2022-01-31T21:58:37.125 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[51620]: debug 2022-01-31T21:58:37.105+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.105932+0000) 2022-01-31T21:58:37.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:37 smithi167 conmon[49112]: debug 2022-01-31T21:58:37.105+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.106034+0000) 2022-01-31T21:58:37.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:37 smithi167 conmon[54076]: debug 2022-01-31T21:58:37.106+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.107047+0000) 2022-01-31T21:58:37.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:37 smithi167 conmon[60316]: debug 2022-01-31T21:58:37.106+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.106642+0000) 2022-01-31T21:58:37.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[35325]: debug 2022-01-31T21:58:37.123+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172554 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:37.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[41853]: debug 2022-01-31T21:58:37.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.232234+0000) 2022-01-31T21:58:37.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:37 smithi167 conmon[60316]: debug 2022-01-31T21:58:37.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.519402+0000) 2022-01-31T21:58:37.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:37 smithi167 conmon[54076]: debug 2022-01-31T21:58:37.518+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.519408+0000) 2022-01-31T21:58:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:37 smithi167 conmon[49112]: debug 2022-01-31T21:58:37.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.855909+0000) 2022-01-31T21:58:38.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:38 smithi171 conmon[46715]: debug 2022-01-31T21:58:38.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:38.026954+0000) 2022-01-31T21:58:38.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:37 smithi171 conmon[51620]: debug 2022-01-31T21:58:37.873+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:37.874580+0000) 2022-01-31T21:58:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:38 smithi171 conmon[41853]: debug 2022-01-31T21:58:38.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:38.232389+0000) 2022-01-31T21:58:38.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:38 smithi167 conmon[54076]: debug 2022-01-31T21:58:38.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:38.519589+0000) 2022-01-31T21:58:38.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:38 smithi167 conmon[60316]: debug 2022-01-31T21:58:38.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:38.519589+0000) 2022-01-31T21:58:39.026 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:38 smithi171 conmon[35325]: debug 2022-01-31T21:58:38.769+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:58:39.027 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:38 smithi171 conmon[51620]: debug 2022-01-31T21:58:38.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:38.874774+0000) 2022-01-31T21:58:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:38 smithi167 conmon[49112]: debug 2022-01-31T21:58:38.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:38.856102+0000) 2022-01-31T21:58:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:39 smithi171 conmon[41853]: debug 2022-01-31T21:58:39.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:39.232593+0000) 2022-01-31T21:58:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:39 smithi171 conmon[46715]: debug 2022-01-31T21:58:39.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:39.027138+0000) 2022-01-31T21:58:39.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:39 smithi167 conmon[54076]: debug 2022-01-31T21:58:39.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:39.519766+0000) 2022-01-31T21:58:39.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:39 smithi167 conmon[60316]: debug 2022-01-31T21:58:39.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:39.519764+0000) 2022-01-31T21:58:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:39 smithi167 conmon[49112]: debug 2022-01-31T21:58:39.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:39.856282+0000) 2022-01-31T21:58:40.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:40 smithi171 conmon[46715]: debug 2022-01-31T21:58:40.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:40.027289+0000) 2022-01-31T21:58:40.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:39 smithi171 conmon[51620]: debug 2022-01-31T21:58:39.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:39.874936+0000) 2022-01-31T21:58:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:40 smithi171 conmon[41853]: debug 2022-01-31T21:58:40.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:40.232763+0000) 2022-01-31T21:58:40.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:40 smithi167 conmon[54076]: debug 2022-01-31T21:58:40.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:40.519927+0000) 2022-01-31T21:58:40.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:40 smithi167 conmon[60316]: debug 2022-01-31T21:58:40.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:40.519879+0000) 2022-01-31T21:58:40.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:40 smithi171 conmon[51620]: debug 2022-01-31T21:58:40.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:40.875118+0000) 2022-01-31T21:58:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:40 smithi167 conmon[49112]: debug 2022-01-31T21:58:40.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:40.856441+0000) 2022-01-31T21:58:41.187 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:41 smithi171 conmon[46715]: debug 2022-01-31T21:58:41.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:41.027478+0000) 2022-01-31T21:58:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:41 smithi171 conmon[41853]: debug 2022-01-31T21:58:41.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:41.232954+0000) 2022-01-31T21:58:41.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:41 smithi167 conmon[54076]: debug 2022-01-31T21:58:41.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:41.520121+0000) 2022-01-31T21:58:41.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:41 smithi167 conmon[60316]: debug 2022-01-31T21:58:41.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:41.520114+0000) 2022-01-31T21:58:42.124 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[46715]: debug 2022-01-31T21:58:42.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.027667+0000) 2022-01-31T21:58:42.125 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:41 smithi171 conmon[51620]: debug 2022-01-31T21:58:41.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:41.875296+0000) 2022-01-31T21:58:42.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:41 smithi167 conmon[49112]: debug 2022-01-31T21:58:41.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:41.856608+0000) 2022-01-31T21:58:42.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:42 smithi167 conmon[49112]: debug 2022-01-31T21:58:42.126+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.127340+0000) 2022-01-31T21:58:42.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:42 smithi167 conmon[54076]: debug 2022-01-31T21:58:42.126+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.127246+0000) 2022-01-31T21:58:42.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:42 smithi167 conmon[60316]: debug 2022-01-31T21:58:42.126+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.126888+0000) 2022-01-31T21:58:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[41853]: debug 2022-01-31T21:58:42.126+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.126774+0000) 2022-01-31T21:58:42.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[41853]: debug 2022-01-31T21:58:42.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.233170+0000) 2022-01-31T21:58:42.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[46715]: debug 2022-01-31T21:58:42.125+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.126347+0000) 2022-01-31T21:58:42.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[51620]: debug 2022-01-31T21:58:42.126+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.127089+0000) 2022-01-31T21:58:42.610 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[35325]: debug 2022-01-31T21:58:42.144+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172665 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:42.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:42 smithi167 conmon[54076]: debug 2022-01-31T21:58:42.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.520305+0000) 2022-01-31T21:58:42.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:42 smithi167 conmon[60316]: debug 2022-01-31T21:58:42.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.520301+0000) 2022-01-31T21:58:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:42 smithi167 conmon[49112]: debug 2022-01-31T21:58:42.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.856737+0000) 2022-01-31T21:58:43.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:43 smithi171 conmon[46715]: debug 2022-01-31T21:58:43.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:43.027853+0000) 2022-01-31T21:58:43.188 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:42 smithi171 conmon[51620]: debug 2022-01-31T21:58:42.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:42.875497+0000) 2022-01-31T21:58:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:43 smithi171 conmon[41853]: debug 2022-01-31T21:58:43.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:43.233368+0000) 2022-01-31T21:58:43.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:43 smithi167 conmon[54076]: debug 2022-01-31T21:58:43.519+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:43.520408+0000) 2022-01-31T21:58:43.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:43 smithi167 conmon[60316]: debug 2022-01-31T21:58:43.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:43.520457+0000) 2022-01-31T21:58:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:43 smithi167 conmon[49112]: debug 2022-01-31T21:58:43.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:43.856873+0000) 2022-01-31T21:58:44.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:44 smithi171 conmon[46715]: debug 2022-01-31T21:58:44.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:44.028025+0000) 2022-01-31T21:58:44.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:43 smithi171 conmon[51620]: debug 2022-01-31T21:58:43.874+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:43.875671+0000) 2022-01-31T21:58:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:44 smithi171 conmon[41853]: debug 2022-01-31T21:58:44.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:44.233549+0000) 2022-01-31T21:58:44.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:44 smithi167 conmon[54076]: debug 2022-01-31T21:58:44.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:44.520598+0000) 2022-01-31T21:58:44.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:44 smithi167 conmon[60316]: debug 2022-01-31T21:58:44.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:44.520637+0000) 2022-01-31T21:58:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:44 smithi167 conmon[49112]: debug 2022-01-31T21:58:44.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:44.857048+0000) 2022-01-31T21:58:45.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:45 smithi171 conmon[46715]: debug 2022-01-31T21:58:45.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:45.028203+0000) 2022-01-31T21:58:45.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:44 smithi171 conmon[51620]: debug 2022-01-31T21:58:44.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:44.875868+0000) 2022-01-31T21:58:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:45 smithi171 conmon[41853]: debug 2022-01-31T21:58:45.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:45.233738+0000) 2022-01-31T21:58:45.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:45 smithi167 conmon[54076]: debug 2022-01-31T21:58:45.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:45.520808+0000) 2022-01-31T21:58:45.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:45 smithi167 conmon[60316]: debug 2022-01-31T21:58:45.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:45.520805+0000) 2022-01-31T21:58:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:45 smithi167 conmon[49112]: debug 2022-01-31T21:58:45.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:45.857247+0000) 2022-01-31T21:58:46.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:46 smithi171 conmon[46715]: debug 2022-01-31T21:58:46.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:46.028377+0000) 2022-01-31T21:58:46.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:45 smithi171 conmon[51620]: debug 2022-01-31T21:58:45.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:45.876048+0000) 2022-01-31T21:58:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:46 smithi171 conmon[41853]: debug 2022-01-31T21:58:46.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:46.233935+0000) 2022-01-31T21:58:46.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:46 smithi167 conmon[54076]: debug 2022-01-31T21:58:46.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:46.521030+0000) 2022-01-31T21:58:46.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:46 smithi167 conmon[60316]: debug 2022-01-31T21:58:46.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:46.521010+0000) 2022-01-31T21:58:47.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:46 smithi167 conmon[49112]: debug 2022-01-31T21:58:46.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:46.857401+0000) 2022-01-31T21:58:47.147 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[46715]: debug 2022-01-31T21:58:47.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.028552+0000) 2022-01-31T21:58:47.147 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:46 smithi171 conmon[51620]: debug 2022-01-31T21:58:46.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:46.876214+0000) 2022-01-31T21:58:47.148 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[41853]: debug 2022-01-31T21:58:47.146+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.147390+0000) 2022-01-31T21:58:47.418 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:47 smithi167 conmon[54076]: debug 2022-01-31T21:58:47.148+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.148642+0000) 2022-01-31T21:58:47.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:47 smithi167 conmon[60316]: debug 2022-01-31T21:58:47.147+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.148432+0000) 2022-01-31T21:58:47.420 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:47 smithi167 conmon[49112]: debug 2022-01-31T21:58:47.148+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.148930+0000) 2022-01-31T21:58:47.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[35325]: debug 2022-01-31T21:58:47.165+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172775 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:47.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[41853]: debug 2022-01-31T21:58:47.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.234118+0000) 2022-01-31T21:58:47.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[46715]: debug 2022-01-31T21:58:47.147+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.148082+0000) 2022-01-31T21:58:47.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[51620]: debug 2022-01-31T21:58:47.147+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.147841+0000) 2022-01-31T21:58:47.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:47 smithi167 conmon[60316]: debug 2022-01-31T21:58:47.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.521181+0000) 2022-01-31T21:58:47.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:47 smithi167 conmon[54076]: debug 2022-01-31T21:58:47.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.521215+0000) 2022-01-31T21:58:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:47 smithi167 conmon[49112]: debug 2022-01-31T21:58:47.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.857578+0000) 2022-01-31T21:58:48.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:48 smithi171 conmon[46715]: debug 2022-01-31T21:58:48.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:48.028716+0000) 2022-01-31T21:58:48.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:47 smithi171 conmon[51620]: debug 2022-01-31T21:58:47.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:47.876417+0000) 2022-01-31T21:58:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:48 smithi171 conmon[41853]: debug 2022-01-31T21:58:48.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:48.234276+0000) 2022-01-31T21:58:48.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:48 smithi167 conmon[60316]: debug 2022-01-31T21:58:48.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:48.521358+0000) 2022-01-31T21:58:48.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:48 smithi167 conmon[54076]: debug 2022-01-31T21:58:48.520+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:48.521361+0000) 2022-01-31T21:58:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:48 smithi167 conmon[49112]: debug 2022-01-31T21:58:48.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:48.857720+0000) 2022-01-31T21:58:49.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:49 smithi171 conmon[46715]: debug 2022-01-31T21:58:49.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:49.028836+0000) 2022-01-31T21:58:49.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:48 smithi171 conmon[51620]: debug 2022-01-31T21:58:48.875+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:48.876595+0000) 2022-01-31T21:58:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:49 smithi171 conmon[41853]: debug 2022-01-31T21:58:49.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:49.234463+0000) 2022-01-31T21:58:49.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:49 smithi167 conmon[54076]: debug 2022-01-31T21:58:49.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:49.521578+0000) 2022-01-31T21:58:49.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:49 smithi167 conmon[60316]: debug 2022-01-31T21:58:49.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:49.521578+0000) 2022-01-31T21:58:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:49 smithi167 conmon[49112]: debug 2022-01-31T21:58:49.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:49.857903+0000) 2022-01-31T21:58:50.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:50 smithi171 conmon[46715]: debug 2022-01-31T21:58:50.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:50.028941+0000) 2022-01-31T21:58:50.189 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:49 smithi171 conmon[51620]: debug 2022-01-31T21:58:49.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:49.876800+0000) 2022-01-31T21:58:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:50 smithi171 conmon[41853]: debug 2022-01-31T21:58:50.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:50.234645+0000) 2022-01-31T21:58:50.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:50 smithi167 conmon[54076]: debug 2022-01-31T21:58:50.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:50.521776+0000) 2022-01-31T21:58:50.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:50 smithi167 conmon[60316]: debug 2022-01-31T21:58:50.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:50.521770+0000) 2022-01-31T21:58:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:50 smithi167 conmon[49112]: debug 2022-01-31T21:58:50.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:50.858084+0000) 2022-01-31T21:58:51.189 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:51 smithi171 conmon[46715]: debug 2022-01-31T21:58:51.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:51.029099+0000) 2022-01-31T21:58:51.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:50 smithi171 conmon[51620]: debug 2022-01-31T21:58:50.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:50.876990+0000) 2022-01-31T21:58:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:51 smithi171 conmon[41853]: debug 2022-01-31T21:58:51.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:51.234825+0000) 2022-01-31T21:58:51.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:51 smithi167 conmon[54076]: debug 2022-01-31T21:58:51.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:51.521935+0000) 2022-01-31T21:58:51.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:51 smithi167 conmon[60316]: debug 2022-01-31T21:58:51.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:51.521935+0000) 2022-01-31T21:58:52.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[46715]: debug 2022-01-31T21:58:52.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.029280+0000) 2022-01-31T21:58:52.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:51 smithi171 conmon[51620]: debug 2022-01-31T21:58:51.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:51.877158+0000) 2022-01-31T21:58:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:51 smithi167 conmon[49112]: debug 2022-01-31T21:58:51.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:51.858268+0000) 2022-01-31T21:58:52.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:52 smithi167 conmon[49112]: debug 2022-01-31T21:58:52.168+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.169020+0000) 2022-01-31T21:58:52.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:52 smithi167 conmon[54076]: debug 2022-01-31T21:58:52.168+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.168916+0000) 2022-01-31T21:58:52.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:52 smithi167 conmon[60316]: debug 2022-01-31T21:58:52.168+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.168679+0000) 2022-01-31T21:58:52.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[35325]: debug 2022-01-31T21:58:52.185+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172886 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:52.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[46715]: debug 2022-01-31T21:58:52.167+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.168533+0000) 2022-01-31T21:58:52.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[51620]: debug 2022-01-31T21:58:52.168+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.169071+0000) 2022-01-31T21:58:52.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[41853]: debug 2022-01-31T21:58:52.167+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.167941+0000) 2022-01-31T21:58:52.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[41853]: debug 2022-01-31T21:58:52.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.234985+0000) 2022-01-31T21:58:52.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:52 smithi167 conmon[60316]: debug 2022-01-31T21:58:52.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.522141+0000) 2022-01-31T21:58:52.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:52 smithi167 conmon[54076]: debug 2022-01-31T21:58:52.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.522148+0000) 2022-01-31T21:58:53.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:52 smithi167 conmon[49112]: debug 2022-01-31T21:58:52.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.858455+0000) 2022-01-31T21:58:53.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:53 smithi171 conmon[46715]: debug 2022-01-31T21:58:53.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:53.029459+0000) 2022-01-31T21:58:53.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:52 smithi171 conmon[51620]: debug 2022-01-31T21:58:52.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:52.877363+0000) 2022-01-31T21:58:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:53 smithi171 conmon[41853]: debug 2022-01-31T21:58:53.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:53.235158+0000) 2022-01-31T21:58:53.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:53 smithi167 conmon[54076]: debug 2022-01-31T21:58:53.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:53.522315+0000) 2022-01-31T21:58:53.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:53 smithi167 conmon[60316]: debug 2022-01-31T21:58:53.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:53.522296+0000) 2022-01-31T21:58:54.028 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:53 smithi171 conmon[51620]: debug 2022-01-31T21:58:53.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:53.877535+0000) 2022-01-31T21:58:54.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:53 smithi171 conmon[35325]: debug 2022-01-31T21:58:53.770+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:58:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:53 smithi167 conmon[49112]: debug 2022-01-31T21:58:53.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:53.858555+0000) 2022-01-31T21:58:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:54 smithi171 conmon[41853]: debug 2022-01-31T21:58:54.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:54.235351+0000) 2022-01-31T21:58:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:54 smithi171 conmon[46715]: debug 2022-01-31T21:58:54.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:54.029674+0000) 2022-01-31T21:58:54.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:54 smithi167 conmon[60316]: debug 2022-01-31T21:58:54.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:54.522485+0000) 2022-01-31T21:58:54.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:54 smithi167 conmon[54076]: debug 2022-01-31T21:58:54.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:54.522516+0000) 2022-01-31T21:58:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:54 smithi167 conmon[49112]: debug 2022-01-31T21:58:54.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:54.858731+0000) 2022-01-31T21:58:55.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:55 smithi171 conmon[46715]: debug 2022-01-31T21:58:55.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:55.029866+0000) 2022-01-31T21:58:55.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:54 smithi171 conmon[51620]: debug 2022-01-31T21:58:54.876+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:54.877726+0000) 2022-01-31T21:58:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:55 smithi171 conmon[41853]: debug 2022-01-31T21:58:55.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:55.235512+0000) 2022-01-31T21:58:55.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:55 smithi167 conmon[60316]: debug 2022-01-31T21:58:55.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:55.522641+0000) 2022-01-31T21:58:55.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:55 smithi167 conmon[54076]: debug 2022-01-31T21:58:55.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:55.522663+0000) 2022-01-31T21:58:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:55 smithi167 conmon[49112]: debug 2022-01-31T21:58:55.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:55.858942+0000) 2022-01-31T21:58:56.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:56 smithi171 conmon[46715]: debug 2022-01-31T21:58:56.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:56.030052+0000) 2022-01-31T21:58:56.191 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:55 smithi171 conmon[51620]: debug 2022-01-31T21:58:55.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:55.877883+0000) 2022-01-31T21:58:56.576 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:56 smithi171 conmon[41853]: debug 2022-01-31T21:58:56.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:56.235698+0000) 2022-01-31T21:58:56.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:56 smithi167 conmon[60316]: debug 2022-01-31T21:58:56.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:56.522780+0000) 2022-01-31T21:58:56.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:56 smithi167 conmon[54076]: debug 2022-01-31T21:58:56.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:56.522851+0000) 2022-01-31T21:58:57.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:56 smithi167 conmon[49112]: debug 2022-01-31T21:58:56.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:56.859096+0000) 2022-01-31T21:58:57.186 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[46715]: debug 2022-01-31T21:58:57.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.030193+0000) 2022-01-31T21:58:57.187 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:56 smithi171 conmon[51620]: debug 2022-01-31T21:58:56.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:56.878041+0000) 2022-01-31T21:58:57.502 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:57 smithi167 conmon[49112]: debug 2022-01-31T21:58:57.188+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.189430+0000) 2022-01-31T21:58:57.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:57 smithi167 conmon[54076]: debug 2022-01-31T21:58:57.188+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.189293+0000) 2022-01-31T21:58:57.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:57 smithi167 conmon[60316]: debug 2022-01-31T21:58:57.188+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.188813+0000) 2022-01-31T21:58:57.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[46715]: debug 2022-01-31T21:58:57.188+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.188829+0000) 2022-01-31T21:58:57.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[51620]: debug 2022-01-31T21:58:57.187+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.188331+0000) 2022-01-31T21:58:57.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[35325]: debug 2022-01-31T21:58:57.206+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 172996 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:58:57.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[41853]: debug 2022-01-31T21:58:57.189+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.189847+0000) 2022-01-31T21:58:57.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[41853]: debug 2022-01-31T21:58:57.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.235897+0000) 2022-01-31T21:58:57.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:57 smithi167 conmon[54076]: debug 2022-01-31T21:58:57.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.523002+0000) 2022-01-31T21:58:57.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:57 smithi167 conmon[60316]: debug 2022-01-31T21:58:57.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.522912+0000) 2022-01-31T21:58:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:57 smithi167 conmon[49112]: debug 2022-01-31T21:58:57.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.859284+0000) 2022-01-31T21:58:58.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:58 smithi171 conmon[46715]: debug 2022-01-31T21:58:58.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:58.030357+0000) 2022-01-31T21:58:58.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:57 smithi171 conmon[51620]: debug 2022-01-31T21:58:57.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:57.878200+0000) 2022-01-31T21:58:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:58 smithi171 conmon[41853]: debug 2022-01-31T21:58:58.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:58.236048+0000) 2022-01-31T21:58:58.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:58 smithi167 conmon[54076]: debug 2022-01-31T21:58:58.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:58.523169+0000) 2022-01-31T21:58:58.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:58 smithi167 conmon[60316]: debug 2022-01-31T21:58:58.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:58.523093+0000) 2022-01-31T21:58:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:58 smithi167 conmon[49112]: debug 2022-01-31T21:58:58.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:58.859471+0000) 2022-01-31T21:58:59.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:58:59 smithi171 conmon[46715]: debug 2022-01-31T21:58:59.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:59.030492+0000) 2022-01-31T21:58:59.191 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:58 smithi171 conmon[51620]: debug 2022-01-31T21:58:58.877+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:58.878357+0000) 2022-01-31T21:58:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:58:59 smithi171 conmon[41853]: debug 2022-01-31T21:58:59.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:59.236206+0000) 2022-01-31T21:58:59.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:58:59 smithi167 conmon[60316]: debug 2022-01-31T21:58:59.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:59.523302+0000) 2022-01-31T21:58:59.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:58:59 smithi167 conmon[54076]: debug 2022-01-31T21:58:59.521+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:59.523367+0000) 2022-01-31T21:59:00.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:58:59 smithi167 conmon[49112]: debug 2022-01-31T21:58:59.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:59.859702+0000) 2022-01-31T21:59:00.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:00 smithi171 conmon[46715]: debug 2022-01-31T21:59:00.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:00.030648+0000) 2022-01-31T21:59:00.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:58:59 smithi171 conmon[51620]: debug 2022-01-31T21:58:59.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:58:59.878552+0000) 2022-01-31T21:59:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:00 smithi171 conmon[41853]: debug 2022-01-31T21:59:00.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:00.236366+0000) 2022-01-31T21:59:00.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:00 smithi167 conmon[54076]: debug 2022-01-31T21:59:00.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:00.523523+0000) 2022-01-31T21:59:00.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:00 smithi167 conmon[60316]: debug 2022-01-31T21:59:00.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:00.523433+0000) 2022-01-31T21:59:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:00 smithi167 conmon[49112]: debug 2022-01-31T21:59:00.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:00.859954+0000) 2022-01-31T21:59:01.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:01 smithi171 conmon[46715]: debug 2022-01-31T21:59:01.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:01.030857+0000) 2022-01-31T21:59:01.191 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:00 smithi171 conmon[51620]: debug 2022-01-31T21:59:00.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:00.878734+0000) 2022-01-31T21:59:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:01 smithi171 conmon[41853]: debug 2022-01-31T21:59:01.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:01.236548+0000) 2022-01-31T21:59:01.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:01 smithi167 conmon[60316]: debug 2022-01-31T21:59:01.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:01.523547+0000) 2022-01-31T21:59:01.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:01 smithi167 conmon[54076]: debug 2022-01-31T21:59:01.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:01.523684+0000) 2022-01-31T21:59:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:01 smithi167 conmon[49112]: debug 2022-01-31T21:59:01.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:01.860131+0000) 2022-01-31T21:59:02.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[46715]: debug 2022-01-31T21:59:02.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.031042+0000) 2022-01-31T21:59:02.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:01 smithi171 conmon[51620]: debug 2022-01-31T21:59:01.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:01.878950+0000) 2022-01-31T21:59:02.503 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:02 smithi167 conmon[49112]: debug 2022-01-31T21:59:02.208+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.210102+0000) 2022-01-31T21:59:02.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:02 smithi167 conmon[54076]: debug 2022-01-31T21:59:02.207+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.209341+0000) 2022-01-31T21:59:02.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:02 smithi167 conmon[60316]: debug 2022-01-31T21:59:02.208+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.209906+0000) 2022-01-31T21:59:02.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[35325]: debug 2022-01-31T21:59:02.226+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173107 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:02.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[46715]: debug 2022-01-31T21:59:02.209+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.210563+0000) 2022-01-31T21:59:02.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[51620]: debug 2022-01-31T21:59:02.208+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.208911+0000) 2022-01-31T21:59:02.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[41853]: debug 2022-01-31T21:59:02.209+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.210167+0000) 2022-01-31T21:59:02.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[41853]: debug 2022-01-31T21:59:02.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.236663+0000) 2022-01-31T21:59:02.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:02 smithi167 conmon[54076]: debug 2022-01-31T21:59:02.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.523826+0000) 2022-01-31T21:59:02.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:02 smithi167 conmon[60316]: debug 2022-01-31T21:59:02.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.523727+0000) 2022-01-31T21:59:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:02 smithi167 conmon[49112]: debug 2022-01-31T21:59:02.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.860328+0000) 2022-01-31T21:59:03.191 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:02 smithi171 conmon[51620]: debug 2022-01-31T21:59:02.878+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:02.879106+0000) 2022-01-31T21:59:03.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:03 smithi171 conmon[46715]: debug 2022-01-31T21:59:03.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:03.031225+0000) 2022-01-31T21:59:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:03 smithi171 conmon[41853]: debug 2022-01-31T21:59:03.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:03.236833+0000) 2022-01-31T21:59:03.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:03 smithi167 conmon[54076]: debug 2022-01-31T21:59:03.522+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:03.523977+0000) 2022-01-31T21:59:03.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:03 smithi167 conmon[60316]: debug 2022-01-31T21:59:03.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:03.523879+0000) 2022-01-31T21:59:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:03 smithi167 conmon[49112]: debug 2022-01-31T21:59:03.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:03.860537+0000) 2022-01-31T21:59:04.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:04 smithi171 conmon[46715]: debug 2022-01-31T21:59:04.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:04.031420+0000) 2022-01-31T21:59:04.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:03 smithi171 conmon[51620]: debug 2022-01-31T21:59:03.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:03.879270+0000) 2022-01-31T21:59:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:04 smithi171 conmon[41853]: debug 2022-01-31T21:59:04.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:04.236990+0000) 2022-01-31T21:59:04.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:04 smithi167 conmon[54076]: debug 2022-01-31T21:59:04.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:04.524183+0000) 2022-01-31T21:59:04.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:04 smithi167 conmon[60316]: debug 2022-01-31T21:59:04.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:04.524062+0000) 2022-01-31T21:59:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:04 smithi167 conmon[49112]: debug 2022-01-31T21:59:04.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:04.860685+0000) 2022-01-31T21:59:05.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:05 smithi171 conmon[46715]: debug 2022-01-31T21:59:05.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:05.031524+0000) 2022-01-31T21:59:05.193 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:04 smithi171 conmon[51620]: debug 2022-01-31T21:59:04.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:04.879454+0000) 2022-01-31T21:59:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:05 smithi171 conmon[41853]: debug 2022-01-31T21:59:05.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:05.237153+0000) 2022-01-31T21:59:05.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:05 smithi167 conmon[54076]: debug 2022-01-31T21:59:05.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:05.524299+0000) 2022-01-31T21:59:05.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:05 smithi167 conmon[60316]: debug 2022-01-31T21:59:05.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:05.524222+0000) 2022-01-31T21:59:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:05 smithi167 conmon[49112]: debug 2022-01-31T21:59:05.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:05.860897+0000) 2022-01-31T21:59:06.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:06 smithi171 conmon[46715]: debug 2022-01-31T21:59:06.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:06.031681+0000) 2022-01-31T21:59:06.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:05 smithi171 conmon[51620]: debug 2022-01-31T21:59:05.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:05.879637+0000) 2022-01-31T21:59:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:06 smithi171 conmon[41853]: debug 2022-01-31T21:59:06.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:06.237281+0000) 2022-01-31T21:59:06.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:06 smithi167 conmon[60316]: debug 2022-01-31T21:59:06.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:06.524376+0000) 2022-01-31T21:59:06.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:06 smithi167 conmon[54076]: debug 2022-01-31T21:59:06.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:06.524440+0000) 2022-01-31T21:59:07.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:06 smithi167 conmon[49112]: debug 2022-01-31T21:59:06.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:06.861119+0000) 2022-01-31T21:59:07.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[46715]: debug 2022-01-31T21:59:07.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.031869+0000) 2022-01-31T21:59:07.193 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:06 smithi171 conmon[51620]: debug 2022-01-31T21:59:06.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:06.879823+0000) 2022-01-31T21:59:07.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:07 smithi167 conmon[49112]: debug 2022-01-31T21:59:07.228+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.230267+0000) 2022-01-31T21:59:07.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:07 smithi167 conmon[54076]: debug 2022-01-31T21:59:07.228+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.229958+0000) 2022-01-31T21:59:07.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:07 smithi167 conmon[60316]: debug 2022-01-31T21:59:07.228+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.230135+0000) 2022-01-31T21:59:07.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[35325]: debug 2022-01-31T21:59:07.246+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173217 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:07.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[46715]: debug 2022-01-31T21:59:07.230+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.231093+0000) 2022-01-31T21:59:07.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[41853]: debug 2022-01-31T21:59:07.228+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.229566+0000) 2022-01-31T21:59:07.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[41853]: debug 2022-01-31T21:59:07.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.237392+0000) 2022-01-31T21:59:07.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[51620]: debug 2022-01-31T21:59:07.229+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.229842+0000) 2022-01-31T21:59:07.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:07 smithi167 conmon[54076]: debug 2022-01-31T21:59:07.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.524585+0000) 2022-01-31T21:59:07.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:07 smithi167 conmon[60316]: debug 2022-01-31T21:59:07.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.524537+0000) 2022-01-31T21:59:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:07 smithi167 conmon[49112]: debug 2022-01-31T21:59:07.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.861332+0000) 2022-01-31T21:59:08.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:08 smithi171 conmon[46715]: debug 2022-01-31T21:59:08.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:08.032038+0000) 2022-01-31T21:59:08.193 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:07 smithi171 conmon[51620]: debug 2022-01-31T21:59:07.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:07.879951+0000) 2022-01-31T21:59:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:08 smithi171 conmon[41853]: debug 2022-01-31T21:59:08.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:08.237533+0000) 2022-01-31T21:59:08.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:08 smithi167 conmon[54076]: debug 2022-01-31T21:59:08.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:08.524718+0000) 2022-01-31T21:59:08.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:08 smithi167 conmon[60316]: debug 2022-01-31T21:59:08.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:08.524702+0000) 2022-01-31T21:59:09.031 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:08 smithi171 conmon[35325]: debug 2022-01-31T21:59:08.771+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:59:09.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:08 smithi171 conmon[51620]: debug 2022-01-31T21:59:08.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:08.880083+0000) 2022-01-31T21:59:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:08 smithi167 conmon[49112]: debug 2022-01-31T21:59:08.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:08.861521+0000) 2022-01-31T21:59:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:09 smithi171 conmon[41853]: debug 2022-01-31T21:59:09.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:09.237696+0000) 2022-01-31T21:59:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:09 smithi171 conmon[46715]: debug 2022-01-31T21:59:09.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:09.032186+0000) 2022-01-31T21:59:09.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:09 smithi167 conmon[54076]: debug 2022-01-31T21:59:09.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:09.524916+0000) 2022-01-31T21:59:09.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:09 smithi167 conmon[60316]: debug 2022-01-31T21:59:09.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:09.524897+0000) 2022-01-31T21:59:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:09 smithi167 conmon[49112]: debug 2022-01-31T21:59:09.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:09.861686+0000) 2022-01-31T21:59:10.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:10 smithi171 conmon[46715]: debug 2022-01-31T21:59:10.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:10.032396+0000) 2022-01-31T21:59:10.194 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:09 smithi171 conmon[51620]: debug 2022-01-31T21:59:09.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:09.880318+0000) 2022-01-31T21:59:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:10 smithi171 conmon[41853]: debug 2022-01-31T21:59:10.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:10.237894+0000) 2022-01-31T21:59:10.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:10 smithi167 conmon[54076]: debug 2022-01-31T21:59:10.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:10.525112+0000) 2022-01-31T21:59:10.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:10 smithi167 conmon[60316]: debug 2022-01-31T21:59:10.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:10.525111+0000) 2022-01-31T21:59:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:10 smithi167 conmon[49112]: debug 2022-01-31T21:59:10.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:10.861884+0000) 2022-01-31T21:59:11.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:11 smithi171 conmon[46715]: debug 2022-01-31T21:59:11.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:11.032594+0000) 2022-01-31T21:59:11.193 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:10 smithi171 conmon[51620]: debug 2022-01-31T21:59:10.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:10.880546+0000) 2022-01-31T21:59:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:11 smithi171 conmon[41853]: debug 2022-01-31T21:59:11.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:11.238074+0000) 2022-01-31T21:59:11.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:11 smithi167 conmon[60316]: debug 2022-01-31T21:59:11.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:11.525324+0000) 2022-01-31T21:59:11.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:11 smithi167 conmon[54076]: debug 2022-01-31T21:59:11.523+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:11.525320+0000) 2022-01-31T21:59:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:11 smithi167 conmon[49112]: debug 2022-01-31T21:59:11.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:11.862076+0000) 2022-01-31T21:59:12.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[46715]: debug 2022-01-31T21:59:12.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.032813+0000) 2022-01-31T21:59:12.194 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:11 smithi171 conmon[51620]: debug 2022-01-31T21:59:11.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:11.880749+0000) 2022-01-31T21:59:12.449 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[46715]: debug 2022-01-31T21:59:12.248+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.249938+0000) 2022-01-31T21:59:12.449 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[51620]: debug 2022-01-31T21:59:12.247+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.249622+0000) 2022-01-31T21:59:12.450 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[41853]: debug 2022-01-31T21:59:12.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.238210+0000) 2022-01-31T21:59:12.450 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[41853]: debug 2022-01-31T21:59:12.247+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.249330+0000) 2022-01-31T21:59:12.451 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[35325]: debug 2022-01-31T21:59:12.266+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173329 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:12.505 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:12 smithi167 conmon[49112]: debug 2022-01-31T21:59:12.249+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.250744+0000) 2022-01-31T21:59:12.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:12 smithi167 conmon[54076]: debug 2022-01-31T21:59:12.248+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.250285+0000) 2022-01-31T21:59:12.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:12 smithi167 conmon[60316]: debug 2022-01-31T21:59:12.248+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.249850+0000) 2022-01-31T21:59:12.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:12 smithi167 conmon[54076]: debug 2022-01-31T21:59:12.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.525521+0000) 2022-01-31T21:59:12.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:12 smithi167 conmon[60316]: debug 2022-01-31T21:59:12.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.525546+0000) 2022-01-31T21:59:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:12 smithi167 conmon[49112]: debug 2022-01-31T21:59:12.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.862273+0000) 2022-01-31T21:59:13.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:13 smithi171 conmon[46715]: debug 2022-01-31T21:59:13.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:13.032975+0000) 2022-01-31T21:59:13.194 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:12 smithi171 conmon[51620]: debug 2022-01-31T21:59:12.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:12.880966+0000) 2022-01-31T21:59:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:13 smithi171 conmon[41853]: debug 2022-01-31T21:59:13.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:13.238390+0000) 2022-01-31T21:59:13.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:13 smithi167 conmon[54076]: debug 2022-01-31T21:59:13.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:13.525597+0000) 2022-01-31T21:59:13.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:13 smithi167 conmon[60316]: debug 2022-01-31T21:59:13.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:13.525649+0000) 2022-01-31T21:59:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:13 smithi167 conmon[49112]: debug 2022-01-31T21:59:13.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:13.862468+0000) 2022-01-31T21:59:14.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:14 smithi171 conmon[46715]: debug 2022-01-31T21:59:14.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:14.033167+0000) 2022-01-31T21:59:14.194 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:13 smithi171 conmon[51620]: debug 2022-01-31T21:59:13.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:13.881136+0000) 2022-01-31T21:59:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:14 smithi171 conmon[41853]: debug 2022-01-31T21:59:14.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:14.238523+0000) 2022-01-31T21:59:14.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:14 smithi167 conmon[54076]: debug 2022-01-31T21:59:14.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:14.525777+0000) 2022-01-31T21:59:14.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:14 smithi167 conmon[60316]: debug 2022-01-31T21:59:14.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:14.525820+0000) 2022-01-31T21:59:15.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:14 smithi167 conmon[49112]: debug 2022-01-31T21:59:14.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:14.862662+0000) 2022-01-31T21:59:15.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:15 smithi171 conmon[46715]: debug 2022-01-31T21:59:15.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:15.033318+0000) 2022-01-31T21:59:15.194 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:14 smithi171 conmon[51620]: debug 2022-01-31T21:59:14.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:14.881307+0000) 2022-01-31T21:59:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:15 smithi171 conmon[41853]: debug 2022-01-31T21:59:15.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:15.238740+0000) 2022-01-31T21:59:15.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:15 smithi167 conmon[54076]: debug 2022-01-31T21:59:15.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:15.525971+0000) 2022-01-31T21:59:15.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:15 smithi167 conmon[60316]: debug 2022-01-31T21:59:15.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:15.525993+0000) 2022-01-31T21:59:16.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:15 smithi167 conmon[49112]: debug 2022-01-31T21:59:15.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:15.862875+0000) 2022-01-31T21:59:16.194 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:16 smithi171 conmon[46715]: debug 2022-01-31T21:59:16.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:16.033469+0000) 2022-01-31T21:59:16.195 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:15 smithi171 conmon[51620]: debug 2022-01-31T21:59:15.879+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:15.881518+0000) 2022-01-31T21:59:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:16 smithi171 conmon[41853]: debug 2022-01-31T21:59:16.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:16.238941+0000) 2022-01-31T21:59:16.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:16 smithi167 conmon[60316]: debug 2022-01-31T21:59:16.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:16.526169+0000) 2022-01-31T21:59:16.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:16 smithi167 conmon[54076]: debug 2022-01-31T21:59:16.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:16.526173+0000) 2022-01-31T21:59:17.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:16 smithi167 conmon[49112]: debug 2022-01-31T21:59:16.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:16.863063+0000) 2022-01-31T21:59:17.194 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[46715]: debug 2022-01-31T21:59:17.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.033615+0000) 2022-01-31T21:59:17.195 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:16 smithi171 conmon[51620]: debug 2022-01-31T21:59:16.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:16.881714+0000) 2022-01-31T21:59:17.270 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:17 smithi167 conmon[60316]: debug 2022-01-31T21:59:17.268+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.270215+0000) 2022-01-31T21:59:17.525 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:17 smithi167 conmon[49112]: debug 2022-01-31T21:59:17.269+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.271200+0000) 2022-01-31T21:59:17.526 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:17 smithi167 conmon[54076]: debug 2022-01-31T21:59:17.270+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.271965+0000) 2022-01-31T21:59:17.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[35325]: debug 2022-01-31T21:59:17.287+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173438 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:17.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[46715]: debug 2022-01-31T21:59:17.269+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.271594+0000) 2022-01-31T21:59:17.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[51620]: debug 2022-01-31T21:59:17.268+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.270287+0000) 2022-01-31T21:59:17.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[41853]: debug 2022-01-31T21:59:17.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.239083+0000) 2022-01-31T21:59:17.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[41853]: debug 2022-01-31T21:59:17.269+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.271253+0000) 2022-01-31T21:59:17.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:17 smithi167 conmon[54076]: debug 2022-01-31T21:59:17.524+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.526370+0000) 2022-01-31T21:59:17.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:17 smithi167 conmon[60316]: debug 2022-01-31T21:59:17.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.526384+0000) 2022-01-31T21:59:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:17 smithi167 conmon[49112]: debug 2022-01-31T21:59:17.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.863192+0000) 2022-01-31T21:59:18.194 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:18 smithi171 conmon[46715]: debug 2022-01-31T21:59:18.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:18.033806+0000) 2022-01-31T21:59:18.194 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:17 smithi171 conmon[51620]: debug 2022-01-31T21:59:17.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:17.881948+0000) 2022-01-31T21:59:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:18 smithi171 conmon[41853]: debug 2022-01-31T21:59:18.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:18.239276+0000) 2022-01-31T21:59:18.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:18 smithi167 conmon[54076]: debug 2022-01-31T21:59:18.525+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:18.526539+0000) 2022-01-31T21:59:18.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:18 smithi167 conmon[60316]: debug 2022-01-31T21:59:18.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:18.526539+0000) 2022-01-31T21:59:19.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:18 smithi167 conmon[49112]: debug 2022-01-31T21:59:18.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:18.863374+0000) 2022-01-31T21:59:19.194 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:19 smithi171 conmon[46715]: debug 2022-01-31T21:59:19.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:19.033949+0000) 2022-01-31T21:59:19.195 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:18 smithi171 conmon[51620]: debug 2022-01-31T21:59:18.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:18.882114+0000) 2022-01-31T21:59:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:19 smithi171 conmon[41853]: debug 2022-01-31T21:59:19.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:19.239462+0000) 2022-01-31T21:59:19.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:19 smithi167 conmon[54076]: debug 2022-01-31T21:59:19.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:19.526747+0000) 2022-01-31T21:59:19.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:19 smithi167 conmon[60316]: debug 2022-01-31T21:59:19.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:19.526746+0000) 2022-01-31T21:59:20.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:19 smithi167 conmon[49112]: debug 2022-01-31T21:59:19.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:19.863563+0000) 2022-01-31T21:59:20.194 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:20 smithi171 conmon[46715]: debug 2022-01-31T21:59:20.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:20.034050+0000) 2022-01-31T21:59:20.195 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:19 smithi171 conmon[51620]: debug 2022-01-31T21:59:19.880+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:19.882310+0000) 2022-01-31T21:59:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:20 smithi171 conmon[41853]: debug 2022-01-31T21:59:20.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:20.239664+0000) 2022-01-31T21:59:20.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:20 smithi167 conmon[54076]: debug 2022-01-31T21:59:20.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:20.526967+0000) 2022-01-31T21:59:20.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:20 smithi167 conmon[60316]: debug 2022-01-31T21:59:20.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:20.526945+0000) 2022-01-31T21:59:21.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:20 smithi167 conmon[49112]: debug 2022-01-31T21:59:20.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:20.863746+0000) 2022-01-31T21:59:21.194 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:21 smithi171 conmon[46715]: debug 2022-01-31T21:59:21.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:21.034176+0000) 2022-01-31T21:59:21.195 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:20 smithi171 conmon[51620]: debug 2022-01-31T21:59:20.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:20.882515+0000) 2022-01-31T21:59:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:21 smithi171 conmon[41853]: debug 2022-01-31T21:59:21.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:21.239845+0000) 2022-01-31T21:59:21.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:21 smithi167 conmon[54076]: debug 2022-01-31T21:59:21.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:21.527156+0000) 2022-01-31T21:59:21.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:21 smithi167 conmon[60316]: debug 2022-01-31T21:59:21.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:21.527153+0000) 2022-01-31T21:59:22.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:21 smithi167 conmon[49112]: debug 2022-01-31T21:59:21.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:21.863937+0000) 2022-01-31T21:59:22.195 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[46715]: debug 2022-01-31T21:59:22.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.034323+0000) 2022-01-31T21:59:22.195 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:21 smithi171 conmon[51620]: debug 2022-01-31T21:59:21.881+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:21.882719+0000) 2022-01-31T21:59:22.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:22 smithi167 conmon[49112]: debug 2022-01-31T21:59:22.291+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.292520+0000) 2022-01-31T21:59:22.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:22 smithi167 conmon[54076]: debug 2022-01-31T21:59:22.291+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.292664+0000) 2022-01-31T21:59:22.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:22 smithi167 conmon[54076]: debug 2022-01-31T21:59:22.526+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.527355+0000) 2022-01-31T21:59:22.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:22 smithi167 conmon[60316]: debug 2022-01-31T21:59:22.291+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.292696+0000) 2022-01-31T21:59:22.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:22 smithi167 conmon[60316]: debug 2022-01-31T21:59:22.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.527345+0000) 2022-01-31T21:59:22.606 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[51620]: debug 2022-01-31T21:59:22.290+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.291216+0000) 2022-01-31T21:59:22.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[35325]: debug 2022-01-31T21:59:22.308+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173551 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:22.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[46715]: debug 2022-01-31T21:59:22.291+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.291795+0000) 2022-01-31T21:59:22.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[41853]: debug 2022-01-31T21:59:22.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.239995+0000) 2022-01-31T21:59:22.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[41853]: debug 2022-01-31T21:59:22.290+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.291316+0000) 2022-01-31T21:59:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:22 smithi167 conmon[49112]: debug 2022-01-31T21:59:22.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.864149+0000) 2022-01-31T21:59:23.195 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:23 smithi171 conmon[46715]: debug 2022-01-31T21:59:23.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:23.034508+0000) 2022-01-31T21:59:23.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:22 smithi171 conmon[51620]: debug 2022-01-31T21:59:22.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:22.882935+0000) 2022-01-31T21:59:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:23 smithi171 conmon[41853]: debug 2022-01-31T21:59:23.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:23.240207+0000) 2022-01-31T21:59:23.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:23 smithi167 conmon[60316]: debug 2022-01-31T21:59:23.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:23.527532+0000) 2022-01-31T21:59:23.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:23 smithi167 conmon[54076]: debug 2022-01-31T21:59:23.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:23.527537+0000) 2022-01-31T21:59:24.034 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:23 smithi171 conmon[35325]: debug 2022-01-31T21:59:23.771+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:59:24.035 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:23 smithi171 conmon[51620]: debug 2022-01-31T21:59:23.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:23.883041+0000) 2022-01-31T21:59:24.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:23 smithi167 conmon[49112]: debug 2022-01-31T21:59:23.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:23.864305+0000) 2022-01-31T21:59:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:24 smithi171 conmon[41853]: debug 2022-01-31T21:59:24.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:24.240382+0000) 2022-01-31T21:59:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:24 smithi171 conmon[46715]: debug 2022-01-31T21:59:24.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:24.034713+0000) 2022-01-31T21:59:24.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:24 smithi167 conmon[54076]: debug 2022-01-31T21:59:24.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:24.527710+0000) 2022-01-31T21:59:24.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:24 smithi167 conmon[60316]: debug 2022-01-31T21:59:24.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:24.527703+0000) 2022-01-31T21:59:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:24 smithi167 conmon[49112]: debug 2022-01-31T21:59:24.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:24.864520+0000) 2022-01-31T21:59:25.195 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:25 smithi171 conmon[46715]: debug 2022-01-31T21:59:25.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:25.034974+0000) 2022-01-31T21:59:25.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:24 smithi171 conmon[51620]: debug 2022-01-31T21:59:24.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:24.883203+0000) 2022-01-31T21:59:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:25 smithi171 conmon[41853]: debug 2022-01-31T21:59:25.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:25.240568+0000) 2022-01-31T21:59:25.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:25 smithi167 conmon[54076]: debug 2022-01-31T21:59:25.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:25.527891+0000) 2022-01-31T21:59:25.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:25 smithi167 conmon[60316]: debug 2022-01-31T21:59:25.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:25.527891+0000) 2022-01-31T21:59:26.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:25 smithi167 conmon[49112]: debug 2022-01-31T21:59:25.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:25.864729+0000) 2022-01-31T21:59:26.195 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:26 smithi171 conmon[46715]: debug 2022-01-31T21:59:26.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:26.035156+0000) 2022-01-31T21:59:26.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:25 smithi171 conmon[51620]: debug 2022-01-31T21:59:25.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:25.883392+0000) 2022-01-31T21:59:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:26 smithi171 conmon[41853]: debug 2022-01-31T21:59:26.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:26.240787+0000) 2022-01-31T21:59:26.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:26 smithi167 conmon[54076]: debug 2022-01-31T21:59:26.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:26.528061+0000) 2022-01-31T21:59:26.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:26 smithi167 conmon[60316]: debug 2022-01-31T21:59:26.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:26.528085+0000) 2022-01-31T21:59:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:26 smithi167 conmon[49112]: debug 2022-01-31T21:59:26.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:26.864912+0000) 2022-01-31T21:59:27.195 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[46715]: debug 2022-01-31T21:59:27.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.035339+0000) 2022-01-31T21:59:27.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:26 smithi171 conmon[51620]: debug 2022-01-31T21:59:26.882+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:26.883595+0000) 2022-01-31T21:59:27.582 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:27 smithi167 conmon[49112]: debug 2022-01-31T21:59:27.312+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.312529+0000) 2022-01-31T21:59:27.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:27 smithi167 conmon[54076]: debug 2022-01-31T21:59:27.312+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.313202+0000) 2022-01-31T21:59:27.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:27 smithi167 conmon[54076]: debug 2022-01-31T21:59:27.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.528251+0000) 2022-01-31T21:59:27.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:27 smithi167 conmon[60316]: debug 2022-01-31T21:59:27.311+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.311837+0000) 2022-01-31T21:59:27.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:27 smithi167 conmon[60316]: debug 2022-01-31T21:59:27.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.528254+0000) 2022-01-31T21:59:27.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[35325]: debug 2022-01-31T21:59:27.329+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173659 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:27.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[46715]: debug 2022-01-31T21:59:27.310+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.311340+0000) 2022-01-31T21:59:27.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[51620]: debug 2022-01-31T21:59:27.311+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.311801+0000) 2022-01-31T21:59:27.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[41853]: debug 2022-01-31T21:59:27.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.240941+0000) 2022-01-31T21:59:27.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[41853]: debug 2022-01-31T21:59:27.310+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.311481+0000) 2022-01-31T21:59:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:27 smithi167 conmon[49112]: debug 2022-01-31T21:59:27.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.865043+0000) 2022-01-31T21:59:28.196 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:28 smithi171 conmon[46715]: debug 2022-01-31T21:59:28.034+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:28.035514+0000) 2022-01-31T21:59:28.197 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:27 smithi171 conmon[51620]: debug 2022-01-31T21:59:27.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:27.883773+0000) 2022-01-31T21:59:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:28 smithi171 conmon[41853]: debug 2022-01-31T21:59:28.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:28.241114+0000) 2022-01-31T21:59:28.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:28 smithi167 conmon[54076]: debug 2022-01-31T21:59:28.527+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:28.528408+0000) 2022-01-31T21:59:28.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:28 smithi167 conmon[60316]: debug 2022-01-31T21:59:28.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:28.528408+0000) 2022-01-31T21:59:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:28 smithi167 conmon[49112]: debug 2022-01-31T21:59:28.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:28.865248+0000) 2022-01-31T21:59:29.196 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:29 smithi171 conmon[46715]: debug 2022-01-31T21:59:29.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:29.035659+0000) 2022-01-31T21:59:29.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:28 smithi171 conmon[51620]: debug 2022-01-31T21:59:28.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:28.883901+0000) 2022-01-31T21:59:29.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:29 smithi167 conmon[54076]: debug 2022-01-31T21:59:29.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:29.528622+0000) 2022-01-31T21:59:29.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:29 smithi167 conmon[60316]: debug 2022-01-31T21:59:29.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:29.528643+0000) 2022-01-31T21:59:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:29 smithi171 conmon[41853]: debug 2022-01-31T21:59:29.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:29.241270+0000) 2022-01-31T21:59:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:29 smithi167 conmon[49112]: debug 2022-01-31T21:59:29.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:29.865438+0000) 2022-01-31T21:59:30.196 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:30 smithi171 conmon[46715]: debug 2022-01-31T21:59:30.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:30.035809+0000) 2022-01-31T21:59:30.197 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:29 smithi171 conmon[51620]: debug 2022-01-31T21:59:29.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:29.884158+0000) 2022-01-31T21:59:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:30 smithi171 conmon[41853]: debug 2022-01-31T21:59:30.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:30.241455+0000) 2022-01-31T21:59:30.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:30 smithi167 conmon[54076]: debug 2022-01-31T21:59:30.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:30.528811+0000) 2022-01-31T21:59:30.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:30 smithi167 conmon[60316]: debug 2022-01-31T21:59:30.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:30.528809+0000) 2022-01-31T21:59:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:30 smithi167 conmon[49112]: debug 2022-01-31T21:59:30.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:30.865633+0000) 2022-01-31T21:59:31.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:30 smithi171 conmon[51620]: debug 2022-01-31T21:59:30.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:30.884348+0000) 2022-01-31T21:59:31.197 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:31 smithi171 conmon[46715]: debug 2022-01-31T21:59:31.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:31.035970+0000) 2022-01-31T21:59:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:31 smithi171 conmon[41853]: debug 2022-01-31T21:59:31.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:31.241663+0000) 2022-01-31T21:59:31.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:31 smithi167 conmon[54076]: debug 2022-01-31T21:59:31.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:31.529018+0000) 2022-01-31T21:59:31.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:31 smithi167 conmon[60316]: debug 2022-01-31T21:59:31.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:31.529033+0000) 2022-01-31T21:59:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:31 smithi167 conmon[49112]: debug 2022-01-31T21:59:31.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:31.865925+0000) 2022-01-31T21:59:32.196 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:31 smithi171 conmon[51620]: debug 2022-01-31T21:59:31.883+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:31.884623+0000) 2022-01-31T21:59:32.197 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[46715]: debug 2022-01-31T21:59:32.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.036218+0000) 2022-01-31T21:59:32.583 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:32 smithi167 conmon[49112]: debug 2022-01-31T21:59:32.332+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.333288+0000) 2022-01-31T21:59:32.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:32 smithi167 conmon[54076]: debug 2022-01-31T21:59:32.332+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.332907+0000) 2022-01-31T21:59:32.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:32 smithi167 conmon[54076]: debug 2022-01-31T21:59:32.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.529227+0000) 2022-01-31T21:59:32.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:32 smithi167 conmon[60316]: debug 2022-01-31T21:59:32.339+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.340187+0000) 2022-01-31T21:59:32.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:32 smithi167 conmon[60316]: debug 2022-01-31T21:59:32.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.529227+0000) 2022-01-31T21:59:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[41853]: debug 2022-01-31T21:59:32.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.241852+0000) 2022-01-31T21:59:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[41853]: debug 2022-01-31T21:59:32.332+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.333123+0000) 2022-01-31T21:59:32.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[46715]: debug 2022-01-31T21:59:32.332+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.332786+0000) 2022-01-31T21:59:32.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[35325]: debug 2022-01-31T21:59:32.356+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173773 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:32.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[51620]: debug 2022-01-31T21:59:32.332+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.333510+0000) 2022-01-31T21:59:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:32 smithi167 conmon[49112]: debug 2022-01-31T21:59:32.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.866072+0000) 2022-01-31T21:59:33.197 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:33 smithi171 conmon[46715]: debug 2022-01-31T21:59:33.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:33.036406+0000) 2022-01-31T21:59:33.197 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:32 smithi171 conmon[51620]: debug 2022-01-31T21:59:32.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:32.884799+0000) 2022-01-31T21:59:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:33 smithi171 conmon[41853]: debug 2022-01-31T21:59:33.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:33.242021+0000) 2022-01-31T21:59:33.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:33 smithi167 conmon[54076]: debug 2022-01-31T21:59:33.528+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:33.529369+0000) 2022-01-31T21:59:33.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:33 smithi167 conmon[60316]: debug 2022-01-31T21:59:33.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:33.529325+0000) 2022-01-31T21:59:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:33 smithi167 conmon[49112]: debug 2022-01-31T21:59:33.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:33.866231+0000) 2022-01-31T21:59:34.197 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:34 smithi171 conmon[46715]: debug 2022-01-31T21:59:34.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:34.036602+0000) 2022-01-31T21:59:34.197 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:33 smithi171 conmon[51620]: debug 2022-01-31T21:59:33.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:33.884958+0000) 2022-01-31T21:59:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:34 smithi171 conmon[41853]: debug 2022-01-31T21:59:34.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:34.242158+0000) 2022-01-31T21:59:34.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:34 smithi167 conmon[54076]: debug 2022-01-31T21:59:34.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:34.529522+0000) 2022-01-31T21:59:34.804 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:34 smithi167 conmon[60316]: debug 2022-01-31T21:59:34.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:34.529518+0000) 2022-01-31T21:59:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:34 smithi167 conmon[49112]: debug 2022-01-31T21:59:34.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:34.866440+0000) 2022-01-31T21:59:35.197 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:35 smithi171 conmon[46715]: debug 2022-01-31T21:59:35.035+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:35.036731+0000) 2022-01-31T21:59:35.198 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:34 smithi171 conmon[51620]: debug 2022-01-31T21:59:34.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:34.885155+0000) 2022-01-31T21:59:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:35 smithi171 conmon[41853]: debug 2022-01-31T21:59:35.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:35.242359+0000) 2022-01-31T21:59:35.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:35 smithi167 conmon[54076]: debug 2022-01-31T21:59:35.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:35.529681+0000) 2022-01-31T21:59:35.807 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:35 smithi167 conmon[60316]: debug 2022-01-31T21:59:35.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:35.529717+0000) 2022-01-31T21:59:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:35 smithi167 conmon[49112]: debug 2022-01-31T21:59:35.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:35.866674+0000) 2022-01-31T21:59:36.197 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:36 smithi171 conmon[46715]: debug 2022-01-31T21:59:36.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:36.036913+0000) 2022-01-31T21:59:36.198 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:35 smithi171 conmon[51620]: debug 2022-01-31T21:59:35.884+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:35.885353+0000) 2022-01-31T21:59:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:36 smithi171 conmon[41853]: debug 2022-01-31T21:59:36.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:36.242512+0000) 2022-01-31T21:59:36.806 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:36 smithi167 conmon[54076]: debug 2022-01-31T21:59:36.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:36.529885+0000) 2022-01-31T21:59:36.807 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:36 smithi167 conmon[60316]: debug 2022-01-31T21:59:36.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:36.529864+0000) 2022-01-31T21:59:37.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:36 smithi167 conmon[49112]: debug 2022-01-31T21:59:36.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:36.866854+0000) 2022-01-31T21:59:37.198 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[46715]: debug 2022-01-31T21:59:37.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.037094+0000) 2022-01-31T21:59:37.199 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:36 smithi171 conmon[51620]: debug 2022-01-31T21:59:36.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:36.885539+0000) 2022-01-31T21:59:37.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[35325]: debug 2022-01-31T21:59:37.377+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173883 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:37.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[41853]: debug 2022-01-31T21:59:37.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.242686+0000) 2022-01-31T21:59:37.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[41853]: debug 2022-01-31T21:59:37.359+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.359889+0000) 2022-01-31T21:59:37.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[46715]: debug 2022-01-31T21:59:37.358+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.359589+0000) 2022-01-31T21:59:37.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[51620]: debug 2022-01-31T21:59:37.359+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.360322+0000) 2022-01-31T21:59:37.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:37 smithi167 conmon[49112]: debug 2022-01-31T21:59:37.359+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.359750+0000) 2022-01-31T21:59:37.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:37 smithi167 conmon[54076]: debug 2022-01-31T21:59:37.359+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.360023+0000) 2022-01-31T21:59:37.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:37 smithi167 conmon[54076]: debug 2022-01-31T21:59:37.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.530097+0000) 2022-01-31T21:59:37.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:37 smithi167 conmon[60316]: debug 2022-01-31T21:59:37.359+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.360374+0000) 2022-01-31T21:59:37.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:37 smithi167 conmon[60316]: debug 2022-01-31T21:59:37.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.530049+0000) 2022-01-31T21:59:38.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:37 smithi167 conmon[49112]: debug 2022-01-31T21:59:37.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.866985+0000) 2022-01-31T21:59:38.199 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:37 smithi171 conmon[51620]: debug 2022-01-31T21:59:37.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:37.885736+0000) 2022-01-31T21:59:38.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:38 smithi171 conmon[46715]: debug 2022-01-31T21:59:38.036+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:38.037301+0000) 2022-01-31T21:59:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:38 smithi171 conmon[41853]: debug 2022-01-31T21:59:38.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:38.242843+0000) 2022-01-31T21:59:38.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:38 smithi167 conmon[54076]: debug 2022-01-31T21:59:38.529+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:38.530251+0000) 2022-01-31T21:59:38.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:38 smithi167 conmon[60316]: debug 2022-01-31T21:59:38.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:38.530214+0000) 2022-01-31T21:59:39.036 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:38 smithi171 conmon[35325]: debug 2022-01-31T21:59:38.772+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:59:39.037 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:38 smithi171 conmon[51620]: debug 2022-01-31T21:59:38.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:38.885885+0000) 2022-01-31T21:59:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:38 smithi167 conmon[49112]: debug 2022-01-31T21:59:38.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:38.867163+0000) 2022-01-31T21:59:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:39 smithi171 conmon[46715]: debug 2022-01-31T21:59:39.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:39.037438+0000) 2022-01-31T21:59:39.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:39 smithi171 conmon[41853]: debug 2022-01-31T21:59:39.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:39.243017+0000) 2022-01-31T21:59:39.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:39 smithi167 conmon[60316]: debug 2022-01-31T21:59:39.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:39.530412+0000) 2022-01-31T21:59:39.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:39 smithi167 conmon[54076]: debug 2022-01-31T21:59:39.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:39.530444+0000) 2022-01-31T21:59:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:39 smithi167 conmon[49112]: debug 2022-01-31T21:59:39.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:39.867375+0000) 2022-01-31T21:59:40.198 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:40 smithi171 conmon[46715]: debug 2022-01-31T21:59:40.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:40.037646+0000) 2022-01-31T21:59:40.199 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:39 smithi171 conmon[51620]: debug 2022-01-31T21:59:39.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:39.886057+0000) 2022-01-31T21:59:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:40 smithi171 conmon[41853]: debug 2022-01-31T21:59:40.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:40.243172+0000) 2022-01-31T21:59:40.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:40 smithi167 conmon[54076]: debug 2022-01-31T21:59:40.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:40.530611+0000) 2022-01-31T21:59:40.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:40 smithi167 conmon[60316]: debug 2022-01-31T21:59:40.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:40.530589+0000) 2022-01-31T21:59:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:40 smithi167 conmon[49112]: debug 2022-01-31T21:59:40.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:40.867640+0000) 2022-01-31T21:59:41.198 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:41 smithi171 conmon[46715]: debug 2022-01-31T21:59:41.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:41.037836+0000) 2022-01-31T21:59:41.199 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:40 smithi171 conmon[51620]: debug 2022-01-31T21:59:40.885+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:40.886245+0000) 2022-01-31T21:59:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:41 smithi171 conmon[41853]: debug 2022-01-31T21:59:41.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:41.243328+0000) 2022-01-31T21:59:41.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:41 smithi167 conmon[54076]: debug 2022-01-31T21:59:41.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:41.530816+0000) 2022-01-31T21:59:41.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:41 smithi167 conmon[60316]: debug 2022-01-31T21:59:41.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:41.530813+0000) 2022-01-31T21:59:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:41 smithi167 conmon[49112]: debug 2022-01-31T21:59:41.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:41.867777+0000) 2022-01-31T21:59:42.198 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[46715]: debug 2022-01-31T21:59:42.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.038028+0000) 2022-01-31T21:59:42.199 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:41 smithi171 conmon[51620]: debug 2022-01-31T21:59:41.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:41.886435+0000) 2022-01-31T21:59:42.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[35325]: debug 2022-01-31T21:59:42.398+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 173998 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:42.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[46715]: debug 2022-01-31T21:59:42.379+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.380252+0000) 2022-01-31T21:59:42.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[51620]: debug 2022-01-31T21:59:42.380+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.381464+0000) 2022-01-31T21:59:42.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[41853]: debug 2022-01-31T21:59:42.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.243513+0000) 2022-01-31T21:59:42.610 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[41853]: debug 2022-01-31T21:59:42.379+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.379745+0000) 2022-01-31T21:59:42.669 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:42 smithi167 conmon[49112]: debug 2022-01-31T21:59:42.380+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.381434+0000) 2022-01-31T21:59:42.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:42 smithi167 conmon[54076]: debug 2022-01-31T21:59:42.380+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.381088+0000) 2022-01-31T21:59:42.671 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:42 smithi167 conmon[54076]: debug 2022-01-31T21:59:42.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.531008+0000) 2022-01-31T21:59:42.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:42 smithi167 conmon[60316]: debug 2022-01-31T21:59:42.380+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.381307+0000) 2022-01-31T21:59:42.672 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:42 smithi167 conmon[60316]: debug 2022-01-31T21:59:42.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.531009+0000) 2022-01-31T21:59:43.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:42 smithi167 conmon[49112]: debug 2022-01-31T21:59:42.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.867933+0000) 2022-01-31T21:59:43.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:43 smithi171 conmon[46715]: debug 2022-01-31T21:59:43.037+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:43.038206+0000) 2022-01-31T21:59:43.200 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:42 smithi171 conmon[51620]: debug 2022-01-31T21:59:42.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:42.886617+0000) 2022-01-31T21:59:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:43 smithi171 conmon[41853]: debug 2022-01-31T21:59:43.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:43.243694+0000) 2022-01-31T21:59:43.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:43 smithi167 conmon[54076]: debug 2022-01-31T21:59:43.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:43.531133+0000) 2022-01-31T21:59:43.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:43 smithi167 conmon[60316]: debug 2022-01-31T21:59:43.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:43.531133+0000) 2022-01-31T21:59:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:43 smithi167 conmon[49112]: debug 2022-01-31T21:59:43.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:43.868061+0000) 2022-01-31T21:59:44.199 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:43 smithi171 conmon[51620]: debug 2022-01-31T21:59:43.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:43.886794+0000) 2022-01-31T21:59:44.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:44 smithi171 conmon[46715]: debug 2022-01-31T21:59:44.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:44.038386+0000) 2022-01-31T21:59:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:44 smithi171 conmon[41853]: debug 2022-01-31T21:59:44.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:44.243845+0000) 2022-01-31T21:59:44.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:44 smithi167 conmon[60316]: debug 2022-01-31T21:59:44.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:44.531321+0000) 2022-01-31T21:59:44.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:44 smithi167 conmon[54076]: debug 2022-01-31T21:59:44.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:44.531321+0000) 2022-01-31T21:59:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:44 smithi167 conmon[49112]: debug 2022-01-31T21:59:44.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:44.868213+0000) 2022-01-31T21:59:45.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:45 smithi171 conmon[46715]: debug 2022-01-31T21:59:45.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:45.038547+0000) 2022-01-31T21:59:45.200 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:44 smithi171 conmon[51620]: debug 2022-01-31T21:59:44.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:44.886984+0000) 2022-01-31T21:59:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:45 smithi171 conmon[41853]: debug 2022-01-31T21:59:45.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:45.244013+0000) 2022-01-31T21:59:45.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:45 smithi167 conmon[54076]: debug 2022-01-31T21:59:45.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:45.531539+0000) 2022-01-31T21:59:45.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:45 smithi167 conmon[60316]: debug 2022-01-31T21:59:45.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:45.531530+0000) 2022-01-31T21:59:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:45 smithi167 conmon[49112]: debug 2022-01-31T21:59:45.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:45.868389+0000) 2022-01-31T21:59:46.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:46 smithi171 conmon[46715]: debug 2022-01-31T21:59:46.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:46.038729+0000) 2022-01-31T21:59:46.200 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:45 smithi171 conmon[51620]: debug 2022-01-31T21:59:45.886+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:45.887176+0000) 2022-01-31T21:59:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:46 smithi171 conmon[41853]: debug 2022-01-31T21:59:46.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:46.244190+0000) 2022-01-31T21:59:46.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:46 smithi167 conmon[54076]: debug 2022-01-31T21:59:46.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:46.531761+0000) 2022-01-31T21:59:46.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:46 smithi167 conmon[60316]: debug 2022-01-31T21:59:46.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:46.531755+0000) 2022-01-31T21:59:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:46 smithi167 conmon[49112]: debug 2022-01-31T21:59:46.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:46.868548+0000) 2022-01-31T21:59:47.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[46715]: debug 2022-01-31T21:59:47.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.038885+0000) 2022-01-31T21:59:47.200 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:46 smithi171 conmon[51620]: debug 2022-01-31T21:59:46.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:46.887371+0000) 2022-01-31T21:59:47.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[35325]: debug 2022-01-31T21:59:47.419+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174106 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:47.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[46715]: debug 2022-01-31T21:59:47.400+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.401728+0000) 2022-01-31T21:59:47.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[51620]: debug 2022-01-31T21:59:47.402+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.402992+0000) 2022-01-31T21:59:47.610 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[41853]: debug 2022-01-31T21:59:47.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.244335+0000) 2022-01-31T21:59:47.611 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[41853]: debug 2022-01-31T21:59:47.400+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.401148+0000) 2022-01-31T21:59:47.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[49112]: debug 2022-01-31T21:59:47.400+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.402253+0000) 2022-01-31T21:59:47.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[54076]: debug 2022-01-31T21:59:47.671 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[54076]: 2022-01-31T21:59:47.400+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.402140+0000) 2022-01-31T21:59:47.672 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[54076]: debug 2022-01-31T21:59:47.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.531966+0000) 2022-01-31T21:59:47.673 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[60316]: debug 2022-01-31T21:59:47.400+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.401860+0000) 2022-01-31T21:59:47.673 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[60316]: debug 2022-01-31T21:59:47.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.531963+0000) 2022-01-31T21:59:48.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:47 smithi167 conmon[49112]: debug 2022-01-31T21:59:47.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.868731+0000) 2022-01-31T21:59:48.199 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:48 smithi171 conmon[46715]: debug 2022-01-31T21:59:48.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:48.039014+0000) 2022-01-31T21:59:48.200 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:47 smithi171 conmon[51620]: debug 2022-01-31T21:59:47.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:47.887550+0000) 2022-01-31T21:59:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:48 smithi171 conmon[41853]: debug 2022-01-31T21:59:48.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:48.244537+0000) 2022-01-31T21:59:48.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:48 smithi167 conmon[54076]: debug 2022-01-31T21:59:48.530+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:48.532140+0000) 2022-01-31T21:59:48.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:48 smithi167 conmon[60316]: debug 2022-01-31T21:59:48.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:48.532143+0000) 2022-01-31T21:59:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:48 smithi167 conmon[49112]: debug 2022-01-31T21:59:48.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:48.868912+0000) 2022-01-31T21:59:49.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:49 smithi171 conmon[46715]: debug 2022-01-31T21:59:49.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:49.039128+0000) 2022-01-31T21:59:49.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:48 smithi171 conmon[51620]: debug 2022-01-31T21:59:48.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:48.887733+0000) 2022-01-31T21:59:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:49 smithi171 conmon[41853]: debug 2022-01-31T21:59:49.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:49.244676+0000) 2022-01-31T21:59:49.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:49 smithi167 conmon[54076]: debug 2022-01-31T21:59:49.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:49.532355+0000) 2022-01-31T21:59:49.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:49 smithi167 conmon[60316]: debug 2022-01-31T21:59:49.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:49.532334+0000) 2022-01-31T21:59:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:49 smithi167 conmon[49112]: debug 2022-01-31T21:59:49.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:49.869050+0000) 2022-01-31T21:59:50.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:50 smithi171 conmon[46715]: debug 2022-01-31T21:59:50.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:50.039287+0000) 2022-01-31T21:59:50.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:49 smithi171 conmon[51620]: debug 2022-01-31T21:59:49.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:49.887989+0000) 2022-01-31T21:59:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:50 smithi171 conmon[41853]: debug 2022-01-31T21:59:50.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:50.244885+0000) 2022-01-31T21:59:50.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:50 smithi167 conmon[54076]: debug 2022-01-31T21:59:50.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:50.532546+0000) 2022-01-31T21:59:50.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:50 smithi167 conmon[60316]: debug 2022-01-31T21:59:50.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:50.532524+0000) 2022-01-31T21:59:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:50 smithi167 conmon[49112]: debug 2022-01-31T21:59:50.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:50.869205+0000) 2022-01-31T21:59:51.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:51 smithi171 conmon[46715]: debug 2022-01-31T21:59:51.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:51.039504+0000) 2022-01-31T21:59:51.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:50 smithi171 conmon[51620]: debug 2022-01-31T21:59:50.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:50.888161+0000) 2022-01-31T21:59:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:51 smithi171 conmon[41853]: debug 2022-01-31T21:59:51.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:51.245011+0000) 2022-01-31T21:59:51.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:51 smithi167 conmon[54076]: debug 2022-01-31T21:59:51.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:51.532713+0000) 2022-01-31T21:59:51.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:51 smithi167 conmon[60316]: debug 2022-01-31T21:59:51.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:51.532724+0000) 2022-01-31T21:59:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:51 smithi167 conmon[49112]: debug 2022-01-31T21:59:51.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:51.869340+0000) 2022-01-31T21:59:52.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[46715]: debug 2022-01-31T21:59:52.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.039625+0000) 2022-01-31T21:59:52.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:51 smithi171 conmon[51620]: debug 2022-01-31T21:59:51.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:51.888357+0000) 2022-01-31T21:59:52.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[35325]: debug 2022-01-31T21:59:52.440+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174219 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:52.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[46715]: debug 2022-01-31T21:59:52.422+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.422937+0000) 2022-01-31T21:59:52.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[51620]: debug 2022-01-31T21:59:52.422+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.423623+0000) 2022-01-31T21:59:52.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[41853]: debug 2022-01-31T21:59:52.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.245165+0000) 2022-01-31T21:59:52.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[41853]: debug 2022-01-31T21:59:52.421+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.422287+0000) 2022-01-31T21:59:52.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:52 smithi167 conmon[49112]: debug 2022-01-31T21:59:52.422+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.423848+0000) 2022-01-31T21:59:52.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:52 smithi167 conmon[54076]: debug 2022-01-31T21:59:52.421+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.423118+0000) 2022-01-31T21:59:52.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:52 smithi167 conmon[54076]: debug 2022-01-31T21:59:52.531+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.532915+0000) 2022-01-31T21:59:52.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:52 smithi167 conmon[60316]: debug 2022-01-31T21:59:52.421+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.422697+0000) 2022-01-31T21:59:52.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:52 smithi167 conmon[60316]: debug 2022-01-31T21:59:52.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.532916+0000) 2022-01-31T21:59:53.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:52 smithi167 conmon[49112]: debug 2022-01-31T21:59:52.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.869485+0000) 2022-01-31T21:59:53.200 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:53 smithi171 conmon[46715]: debug 2022-01-31T21:59:53.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:53.039833+0000) 2022-01-31T21:59:53.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:52 smithi171 conmon[51620]: debug 2022-01-31T21:59:52.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:52.888550+0000) 2022-01-31T21:59:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:53 smithi171 conmon[41853]: debug 2022-01-31T21:59:53.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:53.245378+0000) 2022-01-31T21:59:53.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:53 smithi167 conmon[54076]: debug 2022-01-31T21:59:53.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:53.533060+0000) 2022-01-31T21:59:53.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:53 smithi167 conmon[60316]: debug 2022-01-31T21:59:53.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:53.533074+0000) 2022-01-31T21:59:54.039 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:53 smithi171 conmon[35325]: debug 2022-01-31T21:59:53.773+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T21:59:54.040 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:53 smithi171 conmon[51620]: debug 2022-01-31T21:59:53.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:53.888711+0000) 2022-01-31T21:59:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:53 smithi167 conmon[49112]: debug 2022-01-31T21:59:53.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:53.869646+0000) 2022-01-31T21:59:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:54 smithi171 conmon[41853]: debug 2022-01-31T21:59:54.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:54.245524+0000) 2022-01-31T21:59:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:54 smithi171 conmon[46715]: debug 2022-01-31T21:59:54.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:54.039960+0000) 2022-01-31T21:59:54.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:54 smithi167 conmon[54076]: debug 2022-01-31T21:59:54.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:54.533273+0000) 2022-01-31T21:59:54.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:54 smithi167 conmon[60316]: debug 2022-01-31T21:59:54.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:54.533273+0000) 2022-01-31T21:59:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:54 smithi167 conmon[49112]: debug 2022-01-31T21:59:54.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:54.869820+0000) 2022-01-31T21:59:55.201 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:55 smithi171 conmon[46715]: debug 2022-01-31T21:59:55.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:55.040090+0000) 2022-01-31T21:59:55.201 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:54 smithi171 conmon[51620]: debug 2022-01-31T21:59:54.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:54.888909+0000) 2022-01-31T21:59:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:55 smithi171 conmon[41853]: debug 2022-01-31T21:59:55.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:55.245745+0000) 2022-01-31T21:59:55.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:55 smithi167 conmon[54076]: debug 2022-01-31T21:59:55.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:55.533490+0000) 2022-01-31T21:59:55.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:55 smithi167 conmon[60316]: debug 2022-01-31T21:59:55.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:55.533491+0000) 2022-01-31T21:59:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:55 smithi167 conmon[49112]: debug 2022-01-31T21:59:55.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:55.870022+0000) 2022-01-31T21:59:56.201 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:56 smithi171 conmon[46715]: debug 2022-01-31T21:59:56.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:56.040223+0000) 2022-01-31T21:59:56.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:55 smithi171 conmon[51620]: debug 2022-01-31T21:59:55.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:55.889060+0000) 2022-01-31T21:59:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:56 smithi171 conmon[41853]: debug 2022-01-31T21:59:56.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:56.245932+0000) 2022-01-31T21:59:56.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:56 smithi167 conmon[54076]: debug 2022-01-31T21:59:56.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:56.533714+0000) 2022-01-31T21:59:56.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:56 smithi167 conmon[60316]: debug 2022-01-31T21:59:56.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:56.533715+0000) 2022-01-31T21:59:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:56 smithi167 conmon[49112]: debug 2022-01-31T21:59:56.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:56.870209+0000) 2022-01-31T21:59:57.201 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[46715]: debug 2022-01-31T21:59:57.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.040363+0000) 2022-01-31T21:59:57.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:56 smithi171 conmon[51620]: debug 2022-01-31T21:59:56.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:56.889256+0000) 2022-01-31T21:59:57.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[46715]: debug 2022-01-31T21:59:57.442+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.444279+0000) 2022-01-31T21:59:57.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[51620]: debug 2022-01-31T21:59:57.443+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.445169+0000) 2022-01-31T21:59:57.463 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[41853]: debug 2022-01-31T21:59:57.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.246077+0000) 2022-01-31T21:59:57.463 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[41853]: debug 2022-01-31T21:59:57.441+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.443701+0000) 2022-01-31T21:59:57.738 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:57 smithi167 conmon[49112]: debug 2022-01-31T21:59:57.444+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.445757+0000) 2022-01-31T21:59:57.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:57 smithi167 conmon[54076]: debug 2022-01-31T21:59:57.442+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.444272+0000) 2022-01-31T21:59:57.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:57 smithi167 conmon[54076]: debug 2022-01-31T21:59:57.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.533904+0000) 2022-01-31T21:59:57.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:57 smithi167 conmon[60316]: debug 2022-01-31T21:59:57.442+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.444147+0000) 2022-01-31T21:59:57.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:57 smithi167 conmon[60316]: debug 2022-01-31T21:59:57.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.533904+0000) 2022-01-31T21:59:57.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[35325]: debug 2022-01-31T21:59:57.460+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174327 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T21:59:58.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:57 smithi167 conmon[49112]: debug 2022-01-31T21:59:57.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.870341+0000) 2022-01-31T21:59:58.201 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:58 smithi171 conmon[46715]: debug 2022-01-31T21:59:58.038+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:58.040562+0000) 2022-01-31T21:59:58.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:57 smithi171 conmon[51620]: debug 2022-01-31T21:59:57.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:57.889480+0000) 2022-01-31T21:59:58.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:58 smithi171 conmon[41853]: debug 2022-01-31T21:59:58.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:58.246229+0000) 2022-01-31T21:59:58.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:58 smithi167 conmon[54076]: debug 2022-01-31T21:59:58.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:58.534085+0000) 2022-01-31T21:59:58.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:58 smithi167 conmon[60316]: debug 2022-01-31T21:59:58.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:58.534093+0000) 2022-01-31T21:59:59.106 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 21:59:59 smithi171 conmon[46715]: debug 2022-01-31T21:59:59.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:59.040737+0000) 2022-01-31T21:59:59.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:58 smithi171 conmon[51620]: debug 2022-01-31T21:59:58.887+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:58.889658+0000) 2022-01-31T21:59:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:58 smithi167 conmon[49112]: debug 2022-01-31T21:59:58.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:58.870538+0000) 2022-01-31T21:59:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 21:59:59 smithi171 conmon[41853]: debug 2022-01-31T21:59:59.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:59.246403+0000) 2022-01-31T21:59:59.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 21:59:59 smithi167 conmon[54076]: debug 2022-01-31T21:59:59.532+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:59.534285+0000) 2022-01-31T21:59:59.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 21:59:59 smithi167 conmon[60316]: debug 2022-01-31T21:59:59.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:59.534307+0000) 2022-01-31T22:00:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 21:59:59 smithi167 conmon[49112]: debug 2022-01-31T21:59:59.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:59.870694+0000) 2022-01-31T22:00:00.201 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:00 smithi171 conmon[46715]: debug 2022-01-31T22:00:00.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:00.040933+0000) 2022-01-31T22:00:00.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 21:59:59 smithi171 conmon[51620]: debug 2022-01-31T21:59:59.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T20:59:59.889837+0000) 2022-01-31T22:00:00.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:00 smithi167 conmon[54076]: debug 2022-01-31T22:00:00.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:00.534491+0000) 2022-01-31T22:00:00.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:00 smithi167 conmon[60316]: debug 2022-01-31T22:00:00.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:00.534490+0000) 2022-01-31T22:00:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:00 smithi171 conmon[41853]: debug 2022-01-31T22:00:00.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:00.246606+0000) 2022-01-31T22:00:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:00 smithi167 conmon[49112]: debug 2022-01-31T22:00:00.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:00.870844+0000) 2022-01-31T22:00:01.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:01 smithi171 conmon[46715]: debug 2022-01-31T22:00:01.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:01.041057+0000) 2022-01-31T22:00:01.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:00 smithi171 conmon[51620]: debug 2022-01-31T22:00:00.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:00.890036+0000) 2022-01-31T22:00:01.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:01 smithi167 conmon[54076]: debug 2022-01-31T22:00:01.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:01.534706+0000) 2022-01-31T22:00:01.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:01 smithi167 conmon[60316]: debug 2022-01-31T22:00:01.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:01.534727+0000) 2022-01-31T22:00:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:01 smithi171 conmon[41853]: debug 2022-01-31T22:00:01.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:01.246794+0000) 2022-01-31T22:00:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:01 smithi167 conmon[49112]: debug 2022-01-31T22:00:01.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:01.871005+0000) 2022-01-31T22:00:02.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[46715]: debug 2022-01-31T22:00:02.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.041233+0000) 2022-01-31T22:00:02.202 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:01 smithi171 conmon[51620]: debug 2022-01-31T22:00:01.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:01.890238+0000) 2022-01-31T22:00:02.463 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[41853]: debug 2022-01-31T22:00:02.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.246952+0000) 2022-01-31T22:00:02.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:02 smithi167 conmon[49112]: debug 2022-01-31T22:00:02.464+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.466025+0000) 2022-01-31T22:00:02.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:02 smithi167 conmon[54076]: debug 2022-01-31T22:00:02.463+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.465452+0000) 2022-01-31T22:00:02.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:02 smithi167 conmon[54076]: debug 2022-01-31T22:00:02.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.534905+0000) 2022-01-31T22:00:02.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:02 smithi167 conmon[60316]: debug 2022-01-31T22:00:02.463+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.465039+0000) 2022-01-31T22:00:02.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:02 smithi167 conmon[60316]: debug 2022-01-31T22:00:02.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.534859+0000) 2022-01-31T22:00:02.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[35325]: debug 2022-01-31T22:00:02.481+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174441 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[41853]: debug 2022-01-31T22:00:02.463+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.464973+0000) 2022-01-31T22:00:02.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[46715]: debug 2022-01-31T22:00:02.463+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.465226+0000) 2022-01-31T22:00:02.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[51620]: debug 2022-01-31T22:00:02.464+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.466344+0000) 2022-01-31T22:00:03.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:02 smithi167 conmon[49112]: debug 2022-01-31T22:00:02.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.871141+0000) 2022-01-31T22:00:03.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:03 smithi171 conmon[46715]: debug 2022-01-31T22:00:03.039+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:03.041452+0000) 2022-01-31T22:00:03.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:02 smithi171 conmon[51620]: debug 2022-01-31T22:00:02.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:02.890398+0000) 2022-01-31T22:00:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:03 smithi171 conmon[41853]: debug 2022-01-31T22:00:03.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:03.247139+0000) 2022-01-31T22:00:03.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:03 smithi167 conmon[54076]: debug 2022-01-31T22:00:03.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:03.535072+0000) 2022-01-31T22:00:03.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:03 smithi167 conmon[60316]: debug 2022-01-31T22:00:03.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:03.534991+0000) 2022-01-31T22:00:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:03 smithi167 conmon[49112]: debug 2022-01-31T22:00:03.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:03.871249+0000) 2022-01-31T22:00:04.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:04 smithi171 conmon[46715]: debug 2022-01-31T22:00:04.040+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:04.041625+0000) 2022-01-31T22:00:04.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:03 smithi171 conmon[51620]: debug 2022-01-31T22:00:03.888+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:03.890508+0000) 2022-01-31T22:00:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:04 smithi171 conmon[41853]: debug 2022-01-31T22:00:04.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:04.247313+0000) 2022-01-31T22:00:04.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:04 smithi167 conmon[54076]: debug 2022-01-31T22:00:04.533+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:04.535341+0000) 2022-01-31T22:00:04.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:04 smithi167 conmon[60316]: debug 2022-01-31T22:00:04.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:04.535169+0000) 2022-01-31T22:00:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:04 smithi167 conmon[49112]: debug 2022-01-31T22:00:04.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:04.871398+0000) 2022-01-31T22:00:05.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:05 smithi171 conmon[46715]: debug 2022-01-31T22:00:05.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:05.041774+0000) 2022-01-31T22:00:05.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:04 smithi171 conmon[51620]: debug 2022-01-31T22:00:04.889+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:04.890630+0000) 2022-01-31T22:00:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:05 smithi171 conmon[41853]: debug 2022-01-31T22:00:05.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:05.247470+0000) 2022-01-31T22:00:05.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:05 smithi167 conmon[54076]: debug 2022-01-31T22:00:05.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:05.535542+0000) 2022-01-31T22:00:05.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:05 smithi167 conmon[60316]: debug 2022-01-31T22:00:05.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:05.535363+0000) 2022-01-31T22:00:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:05 smithi167 conmon[49112]: debug 2022-01-31T22:00:05.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:05.871515+0000) 2022-01-31T22:00:06.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:06 smithi171 conmon[46715]: debug 2022-01-31T22:00:06.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:06.041934+0000) 2022-01-31T22:00:06.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:05 smithi171 conmon[51620]: debug 2022-01-31T22:00:05.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:05.890864+0000) 2022-01-31T22:00:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:06 smithi171 conmon[41853]: debug 2022-01-31T22:00:06.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:06.247655+0000) 2022-01-31T22:00:06.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:06 smithi167 conmon[54076]: debug 2022-01-31T22:00:06.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:06.535735+0000) 2022-01-31T22:00:06.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:06 smithi167 conmon[60316]: debug 2022-01-31T22:00:06.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:06.535578+0000) 2022-01-31T22:00:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:06 smithi167 conmon[49112]: debug 2022-01-31T22:00:06.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:06.871682+0000) 2022-01-31T22:00:07.202 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[46715]: debug 2022-01-31T22:00:07.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.042053+0000) 2022-01-31T22:00:07.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:06 smithi171 conmon[51620]: debug 2022-01-31T22:00:06.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:06.891065+0000) 2022-01-31T22:00:07.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[41853]: debug 2022-01-31T22:00:07.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.247837+0000) 2022-01-31T22:00:07.589 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:07 smithi167 conmon[49112]: debug 2022-01-31T22:00:07.484+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.486128+0000) 2022-01-31T22:00:07.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:07 smithi167 conmon[60316]: debug 2022-01-31T22:00:07.484+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.485723+0000) 2022-01-31T22:00:07.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:07 smithi167 conmon[60316]: debug 2022-01-31T22:00:07.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.535724+0000) 2022-01-31T22:00:07.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:07 smithi167 conmon[54076]: debug 2022-01-31T22:00:07.484+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.486467+0000) 2022-01-31T22:00:07.592 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:07 smithi167 conmon[54076]: debug 2022-01-31T22:00:07.534+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.535861+0000) 2022-01-31T22:00:07.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[35325]: debug 2022-01-31T22:00:07.503+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[41853]: debug 2022-01-31T22:00:07.484+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.485546+0000) 2022-01-31T22:00:07.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[46715]: debug 2022-01-31T22:00:07.485+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.485904+0000) 2022-01-31T22:00:07.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[51620]: debug 2022-01-31T22:00:07.486+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.487267+0000) 2022-01-31T22:00:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:07 smithi167 conmon[49112]: debug 2022-01-31T22:00:07.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.871859+0000) 2022-01-31T22:00:08.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:08 smithi171 conmon[46715]: debug 2022-01-31T22:00:08.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:08.042217+0000) 2022-01-31T22:00:08.203 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:07 smithi171 conmon[51620]: debug 2022-01-31T22:00:07.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:07.891261+0000) 2022-01-31T22:00:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:08 smithi171 conmon[41853]: debug 2022-01-31T22:00:08.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:08.247930+0000) 2022-01-31T22:00:08.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:08 smithi167 conmon[54076]: debug 2022-01-31T22:00:08.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:08.535993+0000) 2022-01-31T22:00:08.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:08 smithi167 conmon[60316]: debug 2022-01-31T22:00:08.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:08.535876+0000) 2022-01-31T22:00:09.041 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:08 smithi171 conmon[35325]: debug 2022-01-31T22:00:08.773+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:00:09.042 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:08 smithi171 conmon[51620]: debug 2022-01-31T22:00:08.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:08.891419+0000) 2022-01-31T22:00:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:08 smithi167 conmon[49112]: debug 2022-01-31T22:00:08.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:08.872053+0000) 2022-01-31T22:00:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:09 smithi171 conmon[46715]: debug 2022-01-31T22:00:09.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:09.042344+0000) 2022-01-31T22:00:09.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:09 smithi171 conmon[41853]: debug 2022-01-31T22:00:09.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:09.248067+0000) 2022-01-31T22:00:09.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:09 smithi167 conmon[54076]: debug 2022-01-31T22:00:09.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:09.536206+0000) 2022-01-31T22:00:09.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:09 smithi167 conmon[60316]: debug 2022-01-31T22:00:09.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:09.536066+0000) 2022-01-31T22:00:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:09 smithi167 conmon[49112]: debug 2022-01-31T22:00:09.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:09.872250+0000) 2022-01-31T22:00:10.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:10 smithi171 conmon[46715]: debug 2022-01-31T22:00:10.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:10.042435+0000) 2022-01-31T22:00:10.204 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:09 smithi171 conmon[51620]: debug 2022-01-31T22:00:09.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:09.891611+0000) 2022-01-31T22:00:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:10 smithi171 conmon[41853]: debug 2022-01-31T22:00:10.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:10.248220+0000) 2022-01-31T22:00:10.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:10 smithi167 conmon[54076]: debug 2022-01-31T22:00:10.535+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:10.536409+0000) 2022-01-31T22:00:10.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:10 smithi167 conmon[60316]: debug 2022-01-31T22:00:10.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:10.536259+0000) 2022-01-31T22:00:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:10 smithi167 conmon[49112]: debug 2022-01-31T22:00:10.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:10.872401+0000) 2022-01-31T22:00:11.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:11 smithi171 conmon[46715]: debug 2022-01-31T22:00:11.041+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:11.042590+0000) 2022-01-31T22:00:11.204 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:10 smithi171 conmon[51620]: debug 2022-01-31T22:00:10.890+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:10.891716+0000) 2022-01-31T22:00:11.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:11 smithi167 conmon[54076]: debug 2022-01-31T22:00:11.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:11.536550+0000) 2022-01-31T22:00:11.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:11 smithi167 conmon[60316]: debug 2022-01-31T22:00:11.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:11.536465+0000) 2022-01-31T22:00:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:11 smithi171 conmon[41853]: debug 2022-01-31T22:00:11.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:11.248427+0000) 2022-01-31T22:00:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:11 smithi167 conmon[49112]: debug 2022-01-31T22:00:11.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:11.872582+0000) 2022-01-31T22:00:12.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[46715]: debug 2022-01-31T22:00:12.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.042798+0000) 2022-01-31T22:00:12.204 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:11 smithi171 conmon[51620]: debug 2022-01-31T22:00:11.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:11.891866+0000) 2022-01-31T22:00:12.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[41853]: debug 2022-01-31T22:00:12.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.248622+0000) 2022-01-31T22:00:12.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[41853]: debug 2022-01-31T22:00:12.505+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.506722+0000) 2022-01-31T22:00:12.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:12 smithi167 conmon[49112]: debug 2022-01-31T22:00:12.507+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.507710+0000) 2022-01-31T22:00:12.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:12 smithi167 conmon[54076]: debug 2022-01-31T22:00:12.505+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.506110+0000) 2022-01-31T22:00:12.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:12 smithi167 conmon[54076]: debug 2022-01-31T22:00:12.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.536660+0000) 2022-01-31T22:00:12.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:12 smithi167 conmon[60316]: debug 2022-01-31T22:00:12.506+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.507050+0000) 2022-01-31T22:00:12.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:12 smithi167 conmon[60316]: debug 2022-01-31T22:00:12.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.536529+0000) 2022-01-31T22:00:12.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[46715]: debug 2022-01-31T22:00:12.506+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.507137+0000) 2022-01-31T22:00:12.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[51620]: debug 2022-01-31T22:00:12.507+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.508093+0000) 2022-01-31T22:00:12.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[35325]: debug 2022-01-31T22:00:12.524+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174662 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:12 smithi167 conmon[49112]: debug 2022-01-31T22:00:12.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.872759+0000) 2022-01-31T22:00:13.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:13 smithi171 conmon[46715]: debug 2022-01-31T22:00:13.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:13.042952+0000) 2022-01-31T22:00:13.204 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:12 smithi171 conmon[51620]: debug 2022-01-31T22:00:12.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:12.892058+0000) 2022-01-31T22:00:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:13 smithi171 conmon[41853]: debug 2022-01-31T22:00:13.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:13.248789+0000) 2022-01-31T22:00:13.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:13 smithi167 conmon[54076]: debug 2022-01-31T22:00:13.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:13.536759+0000) 2022-01-31T22:00:13.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:13 smithi167 conmon[60316]: debug 2022-01-31T22:00:13.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:13.536677+0000) 2022-01-31T22:00:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:13 smithi167 conmon[49112]: debug 2022-01-31T22:00:13.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:13.872920+0000) 2022-01-31T22:00:14.203 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:14 smithi171 conmon[46715]: debug 2022-01-31T22:00:14.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:14.043108+0000) 2022-01-31T22:00:14.204 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:13 smithi171 conmon[51620]: debug 2022-01-31T22:00:13.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:13.892211+0000) 2022-01-31T22:00:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:14 smithi171 conmon[41853]: debug 2022-01-31T22:00:14.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:14.248924+0000) 2022-01-31T22:00:14.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:14 smithi167 conmon[54076]: debug 2022-01-31T22:00:14.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:14.536895+0000) 2022-01-31T22:00:14.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:14 smithi167 conmon[60316]: debug 2022-01-31T22:00:14.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:14.536813+0000) 2022-01-31T22:00:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:14 smithi167 conmon[49112]: debug 2022-01-31T22:00:14.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:14.873143+0000) 2022-01-31T22:00:15.204 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:15 smithi171 conmon[46715]: debug 2022-01-31T22:00:15.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:15.043268+0000) 2022-01-31T22:00:15.205 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:14 smithi171 conmon[51620]: debug 2022-01-31T22:00:14.891+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:14.892403+0000) 2022-01-31T22:00:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:15 smithi171 conmon[41853]: debug 2022-01-31T22:00:15.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:15.249136+0000) 2022-01-31T22:00:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:15 smithi167 conmon[54076]: debug 2022-01-31T22:00:15.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:15.537048+0000) 2022-01-31T22:00:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:15 smithi167 conmon[60316]: debug 2022-01-31T22:00:15.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:15.536947+0000) 2022-01-31T22:00:16.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:15 smithi167 conmon[49112]: debug 2022-01-31T22:00:15.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:15.873346+0000) 2022-01-31T22:00:16.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:16 smithi171 conmon[46715]: debug 2022-01-31T22:00:16.042+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:16.043441+0000) 2022-01-31T22:00:16.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:15 smithi171 conmon[51620]: debug 2022-01-31T22:00:15.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:15.892585+0000) 2022-01-31T22:00:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:16 smithi171 conmon[41853]: debug 2022-01-31T22:00:16.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:16.249288+0000) 2022-01-31T22:00:16.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:16 smithi167 conmon[54076]: debug 2022-01-31T22:00:16.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:16.537258+0000) 2022-01-31T22:00:16.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:16 smithi167 conmon[60316]: debug 2022-01-31T22:00:16.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:16.537089+0000) 2022-01-31T22:00:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:16 smithi167 conmon[49112]: debug 2022-01-31T22:00:16.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:16.873537+0000) 2022-01-31T22:00:17.204 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[46715]: debug 2022-01-31T22:00:17.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.043631+0000) 2022-01-31T22:00:17.205 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:16 smithi171 conmon[51620]: debug 2022-01-31T22:00:16.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:16.892784+0000) 2022-01-31T22:00:17.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[41853]: debug 2022-01-31T22:00:17.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.249489+0000) 2022-01-31T22:00:17.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:17 smithi167 conmon[49112]: debug 2022-01-31T22:00:17.528+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.528720+0000) 2022-01-31T22:00:17.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:17 smithi167 conmon[54076]: debug 2022-01-31T22:00:17.527+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.527748+0000) 2022-01-31T22:00:17.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:17 smithi167 conmon[54076]: debug 2022-01-31T22:00:17.536+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.537452+0000) 2022-01-31T22:00:17.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:17 smithi167 conmon[60316]: debug 2022-01-31T22:00:17.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.528461+0000) 2022-01-31T22:00:17.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:17 smithi167 conmon[60316]: debug 2022-01-31T22:00:17.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.537304+0000) 2022-01-31T22:00:17.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[35325]: debug 2022-01-31T22:00:17.545+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174771 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[41853]: debug 2022-01-31T22:00:17.526+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.527471+0000) 2022-01-31T22:00:17.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[46715]: debug 2022-01-31T22:00:17.527+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.528337+0000) 2022-01-31T22:00:17.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[51620]: debug 2022-01-31T22:00:17.527+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.528506+0000) 2022-01-31T22:00:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:17 smithi167 conmon[49112]: debug 2022-01-31T22:00:17.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.873671+0000) 2022-01-31T22:00:18.204 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:18 smithi171 conmon[46715]: debug 2022-01-31T22:00:18.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:18.043791+0000) 2022-01-31T22:00:18.205 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:17 smithi171 conmon[51620]: debug 2022-01-31T22:00:17.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:17.892937+0000) 2022-01-31T22:00:18.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:18 smithi171 conmon[41853]: debug 2022-01-31T22:00:18.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:18.249665+0000) 2022-01-31T22:00:18.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:18 smithi167 conmon[54076]: debug 2022-01-31T22:00:18.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:18.537616+0000) 2022-01-31T22:00:18.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:18 smithi167 conmon[60316]: debug 2022-01-31T22:00:18.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:18.537492+0000) 2022-01-31T22:00:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:18 smithi167 conmon[49112]: debug 2022-01-31T22:00:18.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:18.873888+0000) 2022-01-31T22:00:19.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:19 smithi171 conmon[46715]: debug 2022-01-31T22:00:19.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:19.043931+0000) 2022-01-31T22:00:19.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:18 smithi171 conmon[51620]: debug 2022-01-31T22:00:18.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:18.893089+0000) 2022-01-31T22:00:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:19 smithi171 conmon[41853]: debug 2022-01-31T22:00:19.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:19.249817+0000) 2022-01-31T22:00:19.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:19 smithi167 conmon[54076]: debug 2022-01-31T22:00:19.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:19.537740+0000) 2022-01-31T22:00:19.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:19 smithi167 conmon[60316]: debug 2022-01-31T22:00:19.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:19.537656+0000) 2022-01-31T22:00:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:19 smithi167 conmon[49112]: debug 2022-01-31T22:00:19.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:19.874054+0000) 2022-01-31T22:00:20.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:20 smithi171 conmon[46715]: debug 2022-01-31T22:00:20.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:20.044074+0000) 2022-01-31T22:00:20.205 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:19 smithi171 conmon[51620]: debug 2022-01-31T22:00:19.892+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:19.893284+0000) 2022-01-31T22:00:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:20 smithi171 conmon[41853]: debug 2022-01-31T22:00:20.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:20.250000+0000) 2022-01-31T22:00:20.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:20 smithi167 conmon[54076]: debug 2022-01-31T22:00:20.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:20.537872+0000) 2022-01-31T22:00:20.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:20 smithi167 conmon[60316]: debug 2022-01-31T22:00:20.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:20.537793+0000) 2022-01-31T22:00:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:20 smithi167 conmon[49112]: debug 2022-01-31T22:00:20.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:20.874241+0000) 2022-01-31T22:00:21.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:21 smithi171 conmon[46715]: debug 2022-01-31T22:00:21.043+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:21.044256+0000) 2022-01-31T22:00:21.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:20 smithi171 conmon[51620]: debug 2022-01-31T22:00:20.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:20.893465+0000) 2022-01-31T22:00:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:21 smithi171 conmon[41853]: debug 2022-01-31T22:00:21.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:21.250188+0000) 2022-01-31T22:00:21.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:21 smithi167 conmon[60316]: debug 2022-01-31T22:00:21.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:21.537978+0000) 2022-01-31T22:00:21.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:21 smithi167 conmon[54076]: debug 2022-01-31T22:00:21.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:21.538034+0000) 2022-01-31T22:00:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:21 smithi167 conmon[49112]: debug 2022-01-31T22:00:21.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:21.874477+0000) 2022-01-31T22:00:22.205 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[46715]: debug 2022-01-31T22:00:22.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.044443+0000) 2022-01-31T22:00:22.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:21 smithi171 conmon[51620]: debug 2022-01-31T22:00:21.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:21.893640+0000) 2022-01-31T22:00:22.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[41853]: debug 2022-01-31T22:00:22.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.250354+0000) 2022-01-31T22:00:22.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:22 smithi167 conmon[49112]: debug 2022-01-31T22:00:22.548+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.548650+0000) 2022-01-31T22:00:22.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:22 smithi167 conmon[60316]: debug 2022-01-31T22:00:22.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.538164+0000) 2022-01-31T22:00:22.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:22 smithi167 conmon[60316]: debug 2022-01-31T22:00:22.547+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.548223+0000) 2022-01-31T22:00:22.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:22 smithi167 conmon[54076]: debug 2022-01-31T22:00:22.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.538233+0000) 2022-01-31T22:00:22.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:22 smithi167 conmon[54076]: debug 2022-01-31T22:00:22.547+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.547991+0000) 2022-01-31T22:00:22.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[35325]: debug 2022-01-31T22:00:22.565+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174885 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[41853]: debug 2022-01-31T22:00:22.547+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.548037+0000) 2022-01-31T22:00:22.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[51620]: debug 2022-01-31T22:00:22.548+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.549176+0000) 2022-01-31T22:00:22.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[46715]: debug 2022-01-31T22:00:22.548+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.548854+0000) 2022-01-31T22:00:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:22 smithi167 conmon[49112]: debug 2022-01-31T22:00:22.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.874689+0000) 2022-01-31T22:00:23.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:23 smithi171 conmon[46715]: debug 2022-01-31T22:00:23.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:23.044629+0000) 2022-01-31T22:00:23.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:22 smithi171 conmon[51620]: debug 2022-01-31T22:00:22.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:22.893829+0000) 2022-01-31T22:00:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:23 smithi171 conmon[41853]: debug 2022-01-31T22:00:23.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:23.250550+0000) 2022-01-31T22:00:23.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:23 smithi167 conmon[60316]: debug 2022-01-31T22:00:23.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:23.538296+0000) 2022-01-31T22:00:23.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:23 smithi167 conmon[54076]: debug 2022-01-31T22:00:23.537+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:23.538421+0000) 2022-01-31T22:00:24.043 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:23 smithi171 conmon[35325]: debug 2022-01-31T22:00:23.774+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:00:24.045 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:23 smithi171 conmon[51620]: debug 2022-01-31T22:00:23.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:23.893946+0000) 2022-01-31T22:00:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:23 smithi167 conmon[49112]: debug 2022-01-31T22:00:23.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:23.874788+0000) 2022-01-31T22:00:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:24 smithi171 conmon[46715]: debug 2022-01-31T22:00:24.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:24.044834+0000) 2022-01-31T22:00:24.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:24 smithi171 conmon[41853]: debug 2022-01-31T22:00:24.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:24.250726+0000) 2022-01-31T22:00:24.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:24 smithi167 conmon[60316]: debug 2022-01-31T22:00:24.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:24.538439+0000) 2022-01-31T22:00:24.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:24 smithi167 conmon[54076]: debug 2022-01-31T22:00:24.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:24.538529+0000) 2022-01-31T22:00:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:24 smithi167 conmon[49112]: debug 2022-01-31T22:00:24.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:24.874995+0000) 2022-01-31T22:00:25.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:25 smithi171 conmon[46715]: debug 2022-01-31T22:00:25.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:25.045017+0000) 2022-01-31T22:00:25.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:24 smithi171 conmon[51620]: debug 2022-01-31T22:00:24.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:24.894127+0000) 2022-01-31T22:00:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:25 smithi171 conmon[41853]: debug 2022-01-31T22:00:25.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:25.250926+0000) 2022-01-31T22:00:25.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:25 smithi167 conmon[54076]: debug 2022-01-31T22:00:25.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:25.538713+0000) 2022-01-31T22:00:25.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:25 smithi167 conmon[60316]: debug 2022-01-31T22:00:25.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:25.538641+0000) 2022-01-31T22:00:26.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:25 smithi167 conmon[49112]: debug 2022-01-31T22:00:25.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:25.875199+0000) 2022-01-31T22:00:26.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:26 smithi171 conmon[46715]: debug 2022-01-31T22:00:26.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:26.045212+0000) 2022-01-31T22:00:26.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:25 smithi171 conmon[51620]: debug 2022-01-31T22:00:25.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:25.894307+0000) 2022-01-31T22:00:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:26 smithi171 conmon[41853]: debug 2022-01-31T22:00:26.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:26.251070+0000) 2022-01-31T22:00:26.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:26 smithi167 conmon[54076]: debug 2022-01-31T22:00:26.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:26.538916+0000) 2022-01-31T22:00:26.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:26 smithi167 conmon[60316]: debug 2022-01-31T22:00:26.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:26.538813+0000) 2022-01-31T22:00:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:26 smithi167 conmon[49112]: debug 2022-01-31T22:00:26.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:26.875418+0000) 2022-01-31T22:00:27.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[46715]: debug 2022-01-31T22:00:27.044+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.045406+0000) 2022-01-31T22:00:27.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:26 smithi171 conmon[51620]: debug 2022-01-31T22:00:26.893+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:26.894494+0000) 2022-01-31T22:00:27.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[41853]: debug 2022-01-31T22:00:27.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.251219+0000) 2022-01-31T22:00:27.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:27 smithi167 conmon[49112]: debug 2022-01-31T22:00:27.568+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.569187+0000) 2022-01-31T22:00:27.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:27 smithi167 conmon[60316]: debug 2022-01-31T22:00:27.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.538958+0000) 2022-01-31T22:00:27.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:27 smithi167 conmon[60316]: debug 2022-01-31T22:00:27.568+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.568905+0000) 2022-01-31T22:00:27.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:27 smithi167 conmon[54076]: debug 2022-01-31T22:00:27.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.539074+0000) 2022-01-31T22:00:27.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:27 smithi167 conmon[54076]: debug 2022-01-31T22:00:27.569+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.569738+0000) 2022-01-31T22:00:27.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[35325]: debug 2022-01-31T22:00:27.586+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 174993 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:27.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[51620]: debug 2022-01-31T22:00:27.569+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.570337+0000) 2022-01-31T22:00:27.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[41853]: debug 2022-01-31T22:00:27.567+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.568391+0000) 2022-01-31T22:00:27.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[46715]: debug 2022-01-31T22:00:27.569+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.570155+0000) 2022-01-31T22:00:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:27 smithi167 conmon[49112]: debug 2022-01-31T22:00:27.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.875531+0000) 2022-01-31T22:00:28.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:28 smithi171 conmon[46715]: debug 2022-01-31T22:00:28.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:28.045586+0000) 2022-01-31T22:00:28.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:27 smithi171 conmon[51620]: debug 2022-01-31T22:00:27.894+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:27.894699+0000) 2022-01-31T22:00:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:28 smithi171 conmon[41853]: debug 2022-01-31T22:00:28.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:28.251381+0000) 2022-01-31T22:00:28.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:28 smithi167 conmon[54076]: debug 2022-01-31T22:00:28.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:28.539218+0000) 2022-01-31T22:00:28.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:28 smithi167 conmon[60316]: debug 2022-01-31T22:00:28.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:28.539111+0000) 2022-01-31T22:00:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:28 smithi167 conmon[49112]: debug 2022-01-31T22:00:28.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:28.875688+0000) 2022-01-31T22:00:29.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:29 smithi171 conmon[46715]: debug 2022-01-31T22:00:29.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:29.045730+0000) 2022-01-31T22:00:29.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:28 smithi171 conmon[51620]: debug 2022-01-31T22:00:28.894+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:28.894888+0000) 2022-01-31T22:00:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:29 smithi171 conmon[41853]: debug 2022-01-31T22:00:29.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:29.251537+0000) 2022-01-31T22:00:29.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:29 smithi167 conmon[54076]: debug 2022-01-31T22:00:29.538+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:29.539344+0000) 2022-01-31T22:00:29.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:29 smithi167 conmon[60316]: debug 2022-01-31T22:00:29.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:29.539274+0000) 2022-01-31T22:00:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:29 smithi167 conmon[49112]: debug 2022-01-31T22:00:29.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:29.875843+0000) 2022-01-31T22:00:30.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:30 smithi171 conmon[46715]: debug 2022-01-31T22:00:30.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:30.045942+0000) 2022-01-31T22:00:30.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:29 smithi171 conmon[51620]: debug 2022-01-31T22:00:29.894+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:29.895113+0000) 2022-01-31T22:00:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:30 smithi171 conmon[41853]: debug 2022-01-31T22:00:30.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:30.251720+0000) 2022-01-31T22:00:30.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:30 smithi167 conmon[54076]: debug 2022-01-31T22:00:30.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:30.539526+0000) 2022-01-31T22:00:30.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:30 smithi167 conmon[60316]: debug 2022-01-31T22:00:30.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:30.539475+0000) 2022-01-31T22:00:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:30 smithi167 conmon[49112]: debug 2022-01-31T22:00:30.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:30.876010+0000) 2022-01-31T22:00:31.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:31 smithi171 conmon[46715]: debug 2022-01-31T22:00:31.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:31.046129+0000) 2022-01-31T22:00:31.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:30 smithi171 conmon[51620]: debug 2022-01-31T22:00:30.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:30.895303+0000) 2022-01-31T22:00:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:31 smithi171 conmon[41853]: debug 2022-01-31T22:00:31.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:31.251884+0000) 2022-01-31T22:00:31.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:31 smithi167 conmon[54076]: debug 2022-01-31T22:00:31.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:31.539733+0000) 2022-01-31T22:00:31.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:31 smithi167 conmon[60316]: debug 2022-01-31T22:00:31.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:31.539716+0000) 2022-01-31T22:00:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:31 smithi167 conmon[49112]: debug 2022-01-31T22:00:31.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:31.876206+0000) 2022-01-31T22:00:32.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[46715]: debug 2022-01-31T22:00:32.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.046285+0000) 2022-01-31T22:00:32.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:31 smithi171 conmon[51620]: debug 2022-01-31T22:00:31.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:31.895489+0000) 2022-01-31T22:00:32.588 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[41853]: debug 2022-01-31T22:00:32.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.252003+0000) 2022-01-31T22:00:32.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[49112]: debug 2022-01-31T22:00:32.589+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.590214+0000) 2022-01-31T22:00:32.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[54076]: debug 2022-01-31T22:00:32.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.539924+0000) 2022-01-31T22:00:32.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[54076]: debug 2022-01-31T22:00:32.598+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.598643+0000) 2022-01-31T22:00:32.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[60316]: debug 2022-01-31T22:00:32.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.539919+0000) 2022-01-31T22:00:32.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[60316]: debug 2022-01-31T22:00:32.588+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.589462+0000) 2022-01-31T22:00:32.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[60316]: 2022-01-31T22:00:32.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[35325]: debug 2022-01-31T22:00:32.613+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175106 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[41853]: debug 2022-01-31T22:00:32.588+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.589428+0000) 2022-01-31T22:00:32.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[46715]: debug 2022-01-31T22:00:32.590+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.590831+0000) 2022-01-31T22:00:32.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[51620]: debug 2022-01-31T22:00:32.589+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.590503+0000) 2022-01-31T22:00:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:32 smithi167 conmon[49112]: debug 2022-01-31T22:00:32.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.876367+0000) 2022-01-31T22:00:33.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:33 smithi171 conmon[46715]: debug 2022-01-31T22:00:33.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:33.046446+0000) 2022-01-31T22:00:33.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:32 smithi171 conmon[51620]: debug 2022-01-31T22:00:32.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:32.895678+0000) 2022-01-31T22:00:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:33 smithi171 conmon[41853]: debug 2022-01-31T22:00:33.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:33.252188+0000) 2022-01-31T22:00:33.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:33 smithi167 conmon[54076]: debug 2022-01-31T22:00:33.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:33.540106+0000) 2022-01-31T22:00:33.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:33 smithi167 conmon[60316]: debug 2022-01-31T22:00:33.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:33.540131+0000) 2022-01-31T22:00:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:33 smithi167 conmon[49112]: debug 2022-01-31T22:00:33.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:33.876542+0000) 2022-01-31T22:00:34.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:34 smithi171 conmon[46715]: debug 2022-01-31T22:00:34.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:34.046639+0000) 2022-01-31T22:00:34.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:33 smithi171 conmon[51620]: debug 2022-01-31T22:00:33.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:33.895837+0000) 2022-01-31T22:00:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:34 smithi171 conmon[41853]: debug 2022-01-31T22:00:34.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:34.252357+0000) 2022-01-31T22:00:34.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:34 smithi167 conmon[54076]: debug 2022-01-31T22:00:34.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:34.540275+0000) 2022-01-31T22:00:34.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:34 smithi167 conmon[60316]: debug 2022-01-31T22:00:34.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:34.540271+0000) 2022-01-31T22:00:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:34 smithi167 conmon[49112]: debug 2022-01-31T22:00:34.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:34.876726+0000) 2022-01-31T22:00:35.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:35 smithi171 conmon[46715]: debug 2022-01-31T22:00:35.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:35.046805+0000) 2022-01-31T22:00:35.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:34 smithi171 conmon[51620]: debug 2022-01-31T22:00:34.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:34.896025+0000) 2022-01-31T22:00:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:35 smithi171 conmon[41853]: debug 2022-01-31T22:00:35.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:35.252506+0000) 2022-01-31T22:00:35.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:35 smithi167 conmon[54076]: debug 2022-01-31T22:00:35.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:35.540465+0000) 2022-01-31T22:00:35.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:35 smithi167 conmon[60316]: debug 2022-01-31T22:00:35.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:35.540464+0000) 2022-01-31T22:00:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:35 smithi167 conmon[49112]: debug 2022-01-31T22:00:35.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:35.876878+0000) 2022-01-31T22:00:36.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:36 smithi171 conmon[46715]: debug 2022-01-31T22:00:36.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:36.047017+0000) 2022-01-31T22:00:36.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:35 smithi171 conmon[51620]: debug 2022-01-31T22:00:35.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:35.896213+0000) 2022-01-31T22:00:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:36 smithi171 conmon[41853]: debug 2022-01-31T22:00:36.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:36.252687+0000) 2022-01-31T22:00:36.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:36 smithi167 conmon[54076]: debug 2022-01-31T22:00:36.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:36.540645+0000) 2022-01-31T22:00:36.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:36 smithi167 conmon[60316]: debug 2022-01-31T22:00:36.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:36.540644+0000) 2022-01-31T22:00:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:36 smithi167 conmon[49112]: debug 2022-01-31T22:00:36.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:36.877024+0000) 2022-01-31T22:00:37.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[46715]: debug 2022-01-31T22:00:37.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.047179+0000) 2022-01-31T22:00:37.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:36 smithi171 conmon[51620]: debug 2022-01-31T22:00:36.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:36.896397+0000) 2022-01-31T22:00:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[41853]: debug 2022-01-31T22:00:37.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.252873+0000) 2022-01-31T22:00:37.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:37 smithi167 conmon[49112]: debug 2022-01-31T22:00:37.615+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.616893+0000) 2022-01-31T22:00:37.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:37 smithi167 conmon[60316]: debug 2022-01-31T22:00:37.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.540764+0000) 2022-01-31T22:00:37.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:37 smithi167 conmon[60316]: debug 2022-01-31T22:00:37.614+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.616356+0000) 2022-01-31T22:00:37.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:37 smithi167 conmon[54076]: debug 2022-01-31T22:00:37.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.540809+0000) 2022-01-31T22:00:37.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:37 smithi167 conmon[54076]: debug 2022-01-31T22:00:37.616+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.617545+0000) 2022-01-31T22:00:37.895 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[35325]: debug 2022-01-31T22:00:37.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175214 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:37.896 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[41853]: debug 2022-01-31T22:00:37.616+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.617287+0000) 2022-01-31T22:00:37.896 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[46715]: debug 2022-01-31T22:00:37.616+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.616819+0000) 2022-01-31T22:00:37.897 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[51620]: debug 2022-01-31T22:00:37.615+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.616481+0000) 2022-01-31T22:00:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:37 smithi167 conmon[49112]: debug 2022-01-31T22:00:37.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.877187+0000) 2022-01-31T22:00:38.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:38 smithi171 conmon[46715]: debug 2022-01-31T22:00:38.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:38.047362+0000) 2022-01-31T22:00:38.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:37 smithi171 conmon[51620]: debug 2022-01-31T22:00:37.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:37.896578+0000) 2022-01-31T22:00:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:38 smithi171 conmon[41853]: debug 2022-01-31T22:00:38.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:38.253076+0000) 2022-01-31T22:00:38.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:38 smithi167 conmon[60316]: debug 2022-01-31T22:00:38.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:38.540922+0000) 2022-01-31T22:00:38.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:38 smithi167 conmon[54076]: debug 2022-01-31T22:00:38.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:38.540949+0000) 2022-01-31T22:00:39.046 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:38 smithi171 conmon[35325]: debug 2022-01-31T22:00:38.774+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:00:39.047 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:38 smithi171 conmon[51620]: debug 2022-01-31T22:00:38.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:38.896745+0000) 2022-01-31T22:00:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:38 smithi167 conmon[49112]: debug 2022-01-31T22:00:38.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:38.877368+0000) 2022-01-31T22:00:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:39 smithi171 conmon[46715]: debug 2022-01-31T22:00:39.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:39.047532+0000) 2022-01-31T22:00:39.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:39 smithi171 conmon[41853]: debug 2022-01-31T22:00:39.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:39.253248+0000) 2022-01-31T22:00:39.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:39 smithi167 conmon[54076]: debug 2022-01-31T22:00:39.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:39.541111+0000) 2022-01-31T22:00:39.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:39 smithi167 conmon[60316]: debug 2022-01-31T22:00:39.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:39.541084+0000) 2022-01-31T22:00:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:39 smithi167 conmon[49112]: debug 2022-01-31T22:00:39.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:39.877459+0000) 2022-01-31T22:00:40.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:40 smithi171 conmon[46715]: debug 2022-01-31T22:00:40.045+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:40.047729+0000) 2022-01-31T22:00:40.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:39 smithi171 conmon[51620]: debug 2022-01-31T22:00:39.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:39.896951+0000) 2022-01-31T22:00:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:40 smithi171 conmon[41853]: debug 2022-01-31T22:00:40.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:40.253457+0000) 2022-01-31T22:00:40.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:40 smithi167 conmon[54076]: debug 2022-01-31T22:00:40.539+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:40.541255+0000) 2022-01-31T22:00:40.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:40 smithi167 conmon[60316]: debug 2022-01-31T22:00:40.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:40.541242+0000) 2022-01-31T22:00:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:40 smithi167 conmon[49112]: debug 2022-01-31T22:00:40.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:40.877605+0000) 2022-01-31T22:00:41.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:41 smithi171 conmon[46715]: debug 2022-01-31T22:00:41.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:41.047928+0000) 2022-01-31T22:00:41.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:40 smithi171 conmon[51620]: debug 2022-01-31T22:00:40.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:40.897134+0000) 2022-01-31T22:00:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:41 smithi171 conmon[41853]: debug 2022-01-31T22:00:41.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:41.253644+0000) 2022-01-31T22:00:41.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:41 smithi167 conmon[54076]: debug 2022-01-31T22:00:41.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:41.541434+0000) 2022-01-31T22:00:41.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:41 smithi167 conmon[60316]: debug 2022-01-31T22:00:41.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:41.541435+0000) 2022-01-31T22:00:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:41 smithi167 conmon[49112]: debug 2022-01-31T22:00:41.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:41.877814+0000) 2022-01-31T22:00:42.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[46715]: debug 2022-01-31T22:00:42.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.048111+0000) 2022-01-31T22:00:42.210 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:41 smithi171 conmon[51620]: debug 2022-01-31T22:00:41.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:41.897338+0000) 2022-01-31T22:00:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[41853]: debug 2022-01-31T22:00:42.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.253798+0000) 2022-01-31T22:00:42.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:42 smithi167 conmon[49112]: debug 2022-01-31T22:00:42.637+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.638616+0000) 2022-01-31T22:00:42.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:42 smithi167 conmon[54076]: debug 2022-01-31T22:00:42.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.541615+0000) 2022-01-31T22:00:42.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:42 smithi167 conmon[54076]: debug 2022-01-31T22:00:42.637+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.638900+0000) 2022-01-31T22:00:42.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:42 smithi167 conmon[60316]: debug 2022-01-31T22:00:42.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.541612+0000) 2022-01-31T22:00:42.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:42 smithi167 conmon[60316]: debug 2022-01-31T22:00:42.637+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.639107+0000) 2022-01-31T22:00:42.896 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[35325]: debug 2022-01-31T22:00:42.654+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175327 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:42.897 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[41853]: debug 2022-01-31T22:00:42.637+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.638859+0000) 2022-01-31T22:00:42.897 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[46715]: debug 2022-01-31T22:00:42.635+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.637089+0000) 2022-01-31T22:00:42.898 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[51620]: debug 2022-01-31T22:00:42.636+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.638548+0000) 2022-01-31T22:00:43.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:42 smithi167 conmon[49112]: debug 2022-01-31T22:00:42.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.877999+0000) 2022-01-31T22:00:43.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:43 smithi171 conmon[46715]: debug 2022-01-31T22:00:43.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:43.048297+0000) 2022-01-31T22:00:43.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:42 smithi171 conmon[51620]: debug 2022-01-31T22:00:42.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:42.897500+0000) 2022-01-31T22:00:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:43 smithi171 conmon[41853]: debug 2022-01-31T22:00:43.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:43.253993+0000) 2022-01-31T22:00:43.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:43 smithi167 conmon[54076]: debug 2022-01-31T22:00:43.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:43.541812+0000) 2022-01-31T22:00:43.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:43 smithi167 conmon[60316]: debug 2022-01-31T22:00:43.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:43.541807+0000) 2022-01-31T22:00:44.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:43 smithi167 conmon[49112]: debug 2022-01-31T22:00:43.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:43.878130+0000) 2022-01-31T22:00:44.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:44 smithi171 conmon[46715]: debug 2022-01-31T22:00:44.046+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:44.048510+0000) 2022-01-31T22:00:44.210 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:43 smithi171 conmon[51620]: debug 2022-01-31T22:00:43.895+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:43.897686+0000) 2022-01-31T22:00:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:44 smithi171 conmon[41853]: debug 2022-01-31T22:00:44.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:44.254147+0000) 2022-01-31T22:00:44.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:44 smithi167 conmon[54076]: debug 2022-01-31T22:00:44.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:44.541967+0000) 2022-01-31T22:00:44.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:44 smithi167 conmon[60316]: debug 2022-01-31T22:00:44.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:44.541966+0000) 2022-01-31T22:00:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:44 smithi167 conmon[49112]: debug 2022-01-31T22:00:44.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:44.878288+0000) 2022-01-31T22:00:45.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:45 smithi171 conmon[46715]: debug 2022-01-31T22:00:45.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:45.048699+0000) 2022-01-31T22:00:45.210 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:44 smithi171 conmon[51620]: debug 2022-01-31T22:00:44.896+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:44.897875+0000) 2022-01-31T22:00:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:45 smithi171 conmon[41853]: debug 2022-01-31T22:00:45.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:45.254329+0000) 2022-01-31T22:00:45.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:45 smithi167 conmon[54076]: debug 2022-01-31T22:00:45.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:45.542180+0000) 2022-01-31T22:00:45.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:45 smithi167 conmon[60316]: debug 2022-01-31T22:00:45.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:45.542177+0000) 2022-01-31T22:00:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:45 smithi167 conmon[49112]: debug 2022-01-31T22:00:45.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:45.878460+0000) 2022-01-31T22:00:46.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:46 smithi171 conmon[46715]: debug 2022-01-31T22:00:46.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:46.048909+0000) 2022-01-31T22:00:46.210 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:45 smithi171 conmon[51620]: debug 2022-01-31T22:00:45.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:45.898065+0000) 2022-01-31T22:00:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:46 smithi171 conmon[41853]: debug 2022-01-31T22:00:46.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:46.254507+0000) 2022-01-31T22:00:46.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:46 smithi167 conmon[54076]: debug 2022-01-31T22:00:46.540+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:46.542401+0000) 2022-01-31T22:00:46.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:46 smithi167 conmon[60316]: debug 2022-01-31T22:00:46.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:46.542399+0000) 2022-01-31T22:00:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:46 smithi167 conmon[49112]: debug 2022-01-31T22:00:46.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:46.878643+0000) 2022-01-31T22:00:47.210 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[46715]: debug 2022-01-31T22:00:47.047+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.049051+0000) 2022-01-31T22:00:47.210 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:46 smithi171 conmon[51620]: debug 2022-01-31T22:00:46.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:46.898254+0000) 2022-01-31T22:00:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[41853]: debug 2022-01-31T22:00:47.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.254686+0000) 2022-01-31T22:00:47.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:47 smithi167 conmon[49112]: debug 2022-01-31T22:00:47.657+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.658739+0000) 2022-01-31T22:00:47.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:47 smithi167 conmon[54076]: debug 2022-01-31T22:00:47.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.542637+0000) 2022-01-31T22:00:47.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:47 smithi167 conmon[54076]: debug 2022-01-31T22:00:47.657+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.658854+0000) 2022-01-31T22:00:47.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:47 smithi167 conmon[60316]: debug 2022-01-31T22:00:47.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.542613+0000) 2022-01-31T22:00:47.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:47 smithi167 conmon[60316]: debug 2022-01-31T22:00:47.657+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.659290+0000) 2022-01-31T22:00:47.980 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[35325]: debug 2022-01-31T22:00:47.675+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175436 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:47.981 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[41853]: debug 2022-01-31T22:00:47.657+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.659115+0000) 2022-01-31T22:00:47.981 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[46715]: debug 2022-01-31T22:00:47.655+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.657645+0000) 2022-01-31T22:00:47.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[51620]: debug 2022-01-31T22:00:47.657+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.658788+0000) 2022-01-31T22:00:47.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:47 smithi171 conmon[51620]: debug 2022-01-31T22:00:47.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.898503+0000) 2022-01-31T22:00:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:47 smithi167 conmon[49112]: debug 2022-01-31T22:00:47.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:47.878793+0000) 2022-01-31T22:00:48.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:48 smithi171 conmon[46715]: debug 2022-01-31T22:00:48.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:48.049229+0000) 2022-01-31T22:00:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:48 smithi171 conmon[41853]: debug 2022-01-31T22:00:48.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:48.254814+0000) 2022-01-31T22:00:48.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:48 smithi167 conmon[54076]: debug 2022-01-31T22:00:48.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:48.542800+0000) 2022-01-31T22:00:48.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:48 smithi167 conmon[60316]: debug 2022-01-31T22:00:48.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:48.542800+0000) 2022-01-31T22:00:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:48 smithi167 conmon[49112]: debug 2022-01-31T22:00:48.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:48.878966+0000) 2022-01-31T22:00:49.210 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:49 smithi171 conmon[46715]: debug 2022-01-31T22:00:49.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:49.049381+0000) 2022-01-31T22:00:49.211 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:48 smithi171 conmon[51620]: debug 2022-01-31T22:00:48.897+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:48.898669+0000) 2022-01-31T22:00:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:49 smithi171 conmon[41853]: debug 2022-01-31T22:00:49.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:49.254977+0000) 2022-01-31T22:00:49.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:49 smithi167 conmon[54076]: debug 2022-01-31T22:00:49.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:49.542963+0000) 2022-01-31T22:00:49.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:49 smithi167 conmon[60316]: debug 2022-01-31T22:00:49.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:49.542968+0000) 2022-01-31T22:00:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:49 smithi167 conmon[49112]: debug 2022-01-31T22:00:49.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:49.879157+0000) 2022-01-31T22:00:50.210 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:50 smithi171 conmon[46715]: debug 2022-01-31T22:00:50.048+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:50.049587+0000) 2022-01-31T22:00:50.211 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:49 smithi171 conmon[51620]: debug 2022-01-31T22:00:49.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:49.898864+0000) 2022-01-31T22:00:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:50 smithi171 conmon[41853]: debug 2022-01-31T22:00:50.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:50.255159+0000) 2022-01-31T22:00:50.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:50 smithi167 conmon[54076]: debug 2022-01-31T22:00:50.541+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:50.543196+0000) 2022-01-31T22:00:50.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:50 smithi167 conmon[60316]: debug 2022-01-31T22:00:50.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:50.543195+0000) 2022-01-31T22:00:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:50 smithi167 conmon[49112]: debug 2022-01-31T22:00:50.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:50.879305+0000) 2022-01-31T22:00:51.210 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:51 smithi171 conmon[46715]: debug 2022-01-31T22:00:51.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:51.049789+0000) 2022-01-31T22:00:51.211 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:50 smithi171 conmon[51620]: debug 2022-01-31T22:00:50.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:50.899042+0000) 2022-01-31T22:00:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:51 smithi171 conmon[41853]: debug 2022-01-31T22:00:51.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:51.255334+0000) 2022-01-31T22:00:51.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:51 smithi167 conmon[54076]: debug 2022-01-31T22:00:51.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:51.543409+0000) 2022-01-31T22:00:51.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:51 smithi167 conmon[60316]: debug 2022-01-31T22:00:51.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:51.543438+0000) 2022-01-31T22:00:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:51 smithi167 conmon[49112]: debug 2022-01-31T22:00:51.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:51.879462+0000) 2022-01-31T22:00:52.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[46715]: debug 2022-01-31T22:00:52.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.049938+0000) 2022-01-31T22:00:52.212 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:51 smithi171 conmon[51620]: debug 2022-01-31T22:00:51.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:51.899239+0000) 2022-01-31T22:00:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[41853]: debug 2022-01-31T22:00:52.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.255515+0000) 2022-01-31T22:00:52.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:52 smithi167 conmon[49112]: debug 2022-01-31T22:00:52.677+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.679383+0000) 2022-01-31T22:00:52.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:52 smithi167 conmon[54076]: debug 2022-01-31T22:00:52.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.543599+0000) 2022-01-31T22:00:52.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:52 smithi167 conmon[54076]: debug 2022-01-31T22:00:52.678+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.679596+0000) 2022-01-31T22:00:52.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:52 smithi167 conmon[60316]: debug 2022-01-31T22:00:52.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.543610+0000) 2022-01-31T22:00:52.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:52 smithi167 conmon[60316]: debug 2022-01-31T22:00:52.678+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.679656+0000) 2022-01-31T22:00:52.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[35325]: debug 2022-01-31T22:00:52.696+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:52.982 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[41853]: debug 2022-01-31T22:00:52.679+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.679918+0000) 2022-01-31T22:00:52.983 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[46715]: debug 2022-01-31T22:00:52.678+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.678894+0000) 2022-01-31T22:00:52.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[51620]: debug 2022-01-31T22:00:52.678+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.679340+0000) 2022-01-31T22:00:52.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:52 smithi171 conmon[51620]: debug 2022-01-31T22:00:52.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.899428+0000) 2022-01-31T22:00:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:52 smithi167 conmon[49112]: debug 2022-01-31T22:00:52.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:52.879610+0000) 2022-01-31T22:00:53.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:53 smithi171 conmon[46715]: debug 2022-01-31T22:00:53.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:53.050146+0000) 2022-01-31T22:00:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:53 smithi171 conmon[41853]: debug 2022-01-31T22:00:53.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:53.255714+0000) 2022-01-31T22:00:53.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:53 smithi167 conmon[54076]: debug 2022-01-31T22:00:53.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:53.543812+0000) 2022-01-31T22:00:53.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:53 smithi167 conmon[60316]: debug 2022-01-31T22:00:53.543+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:53.543820+0000) 2022-01-31T22:00:54.049 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:53 smithi171 conmon[35325]: debug 2022-01-31T22:00:53.775+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:00:54.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:53 smithi171 conmon[51620]: debug 2022-01-31T22:00:53.898+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:53.899575+0000) 2022-01-31T22:00:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:53 smithi167 conmon[49112]: debug 2022-01-31T22:00:53.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:53.879770+0000) 2022-01-31T22:00:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:54 smithi171 conmon[46715]: debug 2022-01-31T22:00:54.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:54.050300+0000) 2022-01-31T22:00:54.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:54 smithi171 conmon[41853]: debug 2022-01-31T22:00:54.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:54.255836+0000) 2022-01-31T22:00:54.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:54 smithi167 conmon[54076]: debug 2022-01-31T22:00:54.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:54.543975+0000) 2022-01-31T22:00:54.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:54 smithi167 conmon[60316]: debug 2022-01-31T22:00:54.543+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:54.543993+0000) 2022-01-31T22:00:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:54 smithi167 conmon[49112]: debug 2022-01-31T22:00:54.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:54.879955+0000) 2022-01-31T22:00:55.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:55 smithi171 conmon[46715]: debug 2022-01-31T22:00:55.049+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:55.050460+0000) 2022-01-31T22:00:55.212 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:54 smithi171 conmon[51620]: debug 2022-01-31T22:00:54.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:54.899775+0000) 2022-01-31T22:00:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:55 smithi171 conmon[41853]: debug 2022-01-31T22:00:55.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:55.255992+0000) 2022-01-31T22:00:55.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:55 smithi167 conmon[54076]: debug 2022-01-31T22:00:55.542+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:55.544162+0000) 2022-01-31T22:00:55.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:55 smithi167 conmon[60316]: debug 2022-01-31T22:00:55.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:55.544164+0000) 2022-01-31T22:00:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:55 smithi167 conmon[49112]: debug 2022-01-31T22:00:55.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:55.880165+0000) 2022-01-31T22:00:56.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:56 smithi171 conmon[46715]: debug 2022-01-31T22:00:56.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:56.050652+0000) 2022-01-31T22:00:56.212 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:55 smithi171 conmon[51620]: debug 2022-01-31T22:00:55.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:55.899977+0000) 2022-01-31T22:00:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:56 smithi171 conmon[41853]: debug 2022-01-31T22:00:56.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:56.256177+0000) 2022-01-31T22:00:56.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:56 smithi167 conmon[54076]: debug 2022-01-31T22:00:56.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:56.544377+0000) 2022-01-31T22:00:56.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:56 smithi167 conmon[60316]: debug 2022-01-31T22:00:56.543+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:56.544368+0000) 2022-01-31T22:00:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:56 smithi167 conmon[49112]: debug 2022-01-31T22:00:56.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:56.880333+0000) 2022-01-31T22:00:57.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[46715]: debug 2022-01-31T22:00:57.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.050804+0000) 2022-01-31T22:00:57.212 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:56 smithi171 conmon[51620]: debug 2022-01-31T22:00:56.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:56.900166+0000) 2022-01-31T22:00:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[41853]: debug 2022-01-31T22:00:57.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.256384+0000) 2022-01-31T22:00:57.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[49112]: debug 2022-01-31T22:00:57.698+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.700301+0000) 2022-01-31T22:00:57.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[54076]: debug 2022-01-31T22:00:57.543+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.544582+0000) 2022-01-31T22:00:57.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[54076]: debug 2022-01-31T22:00:57.699+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.701074+0000) 2022-01-31T22:00:57.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[60316]: debug 2022-01-31T22:00:57.543+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.544559+0000) 2022-01-31T22:00:57.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[60316]: debug 2022-01-31T22:00:57.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[60316]: 2022-01-31T22:00:57.699+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.700810+0000) 2022-01-31T22:00:57.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[35325]: debug 2022-01-31T22:00:57.716+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175658 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:00:57.983 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[41853]: debug 2022-01-31T22:00:57.699+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.699906+0000) 2022-01-31T22:00:57.983 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[46715]: debug 2022-01-31T22:00:57.698+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.699462+0000) 2022-01-31T22:00:57.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[51620]: debug 2022-01-31T22:00:57.700+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.700875+0000) 2022-01-31T22:00:57.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:57 smithi171 conmon[51620]: debug 2022-01-31T22:00:57.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.900355+0000) 2022-01-31T22:00:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:57 smithi167 conmon[49112]: debug 2022-01-31T22:00:57.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:57.880542+0000) 2022-01-31T22:00:58.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:58 smithi171 conmon[46715]: debug 2022-01-31T22:00:58.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:58.050998+0000) 2022-01-31T22:00:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:58 smithi171 conmon[41853]: debug 2022-01-31T22:00:58.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:58.256565+0000) 2022-01-31T22:00:58.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:58 smithi167 conmon[54076]: debug 2022-01-31T22:00:58.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:58.544782+0000) 2022-01-31T22:00:58.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:58 smithi167 conmon[60316]: debug 2022-01-31T22:00:58.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:58.544770+0000) 2022-01-31T22:00:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:58 smithi167 conmon[49112]: debug 2022-01-31T22:00:58.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:58.880732+0000) 2022-01-31T22:00:59.212 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:00:59 smithi171 conmon[46715]: debug 2022-01-31T22:00:59.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:59.051151+0000) 2022-01-31T22:00:59.212 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:58 smithi171 conmon[51620]: debug 2022-01-31T22:00:58.899+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:58.900483+0000) 2022-01-31T22:00:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:00:59 smithi171 conmon[41853]: debug 2022-01-31T22:00:59.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:59.256708+0000) 2022-01-31T22:00:59.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:00:59 smithi167 conmon[54076]: debug 2022-01-31T22:00:59.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:59.544960+0000) 2022-01-31T22:00:59.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:00:59 smithi167 conmon[60316]: debug 2022-01-31T22:00:59.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:59.544956+0000) 2022-01-31T22:01:00.050 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:00:59 smithi171 conmon[51620]: debug 2022-01-31T22:00:59.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:59.900629+0000) 2022-01-31T22:01:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:00:59 smithi167 conmon[49112]: debug 2022-01-31T22:00:59.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:00:59.880943+0000) 2022-01-31T22:01:00.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:00 smithi171 conmon[46715]: debug 2022-01-31T22:01:00.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:00.051316+0000) 2022-01-31T22:01:00.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:00 smithi171 conmon[41853]: debug 2022-01-31T22:01:00.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:00 smithi171 conmon[41853]: 2022-01-31T22:01:00.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:00.256932+0000) 2022-01-31T22:01:00.707 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:00 smithi167 conmon[54076]: debug 2022-01-31T22:01:00.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:00.545162+0000) 2022-01-31T22:01:00.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:00 smithi167 conmon[60316]: debug 2022-01-31T22:01:00.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:00.545162+0000) 2022-01-31T22:01:01.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:00 smithi167 conmon[49112]: debug 2022-01-31T22:01:00.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:00.881132+0000) 2022-01-31T22:01:01.212 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:01 smithi171 conmon[46715]: debug 2022-01-31T22:01:01.050+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:01.051506+0000) 2022-01-31T22:01:01.213 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:00 smithi171 conmon[51620]: debug 2022-01-31T22:01:00.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:00.900818+0000) 2022-01-31T22:01:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:01 smithi171 conmon[41853]: debug 2022-01-31T22:01:01.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:01.257089+0000) 2022-01-31T22:01:01.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:01 smithi167 conmon[54076]: debug 2022-01-31T22:01:01.544+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:01.545379+0000) 2022-01-31T22:01:01.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:01 smithi167 conmon[60316]: debug 2022-01-31T22:01:01.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:01.545382+0000) 2022-01-31T22:01:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:01 smithi167 conmon[49112]: debug 2022-01-31T22:01:01.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:01.881336+0000) 2022-01-31T22:01:02.212 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[46715]: debug 2022-01-31T22:01:02.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.051700+0000) 2022-01-31T22:01:02.213 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:01 smithi171 conmon[51620]: debug 2022-01-31T22:01:01.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:01.901003+0000) 2022-01-31T22:01:02.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[41853]: debug 2022-01-31T22:01:02.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.257328+0000) 2022-01-31T22:01:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:02 smithi167 conmon[54076]: debug 2022-01-31T22:01:02.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.545569+0000) 2022-01-31T22:01:02.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:02 smithi167 conmon[60316]: debug 2022-01-31T22:01:02.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.545564+0000) 2022-01-31T22:01:02.983 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[41853]: debug 2022-01-31T22:01:02.719+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.719851+0000) 2022-01-31T22:01:02.984 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[46715]: debug 2022-01-31T22:01:02.718+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.719332+0000) 2022-01-31T22:01:02.984 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[35325]: debug 2022-01-31T22:01:02.738+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175772 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:02.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[51620]: debug 2022-01-31T22:01:02.720+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.721244+0000) 2022-01-31T22:01:02.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:02 smithi171 conmon[51620]: debug 2022-01-31T22:01:02.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.901190+0000) 2022-01-31T22:01:03.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:02 smithi167 conmon[54076]: debug 2022-01-31T22:01:02.720+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.720762+0000) 2022-01-31T22:01:03.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:02 smithi167 conmon[60316]: debug 2022-01-31T22:01:02.719+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.719538+0000) 2022-01-31T22:01:03.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:02 smithi167 conmon[49112]: debug 2022-01-31T22:01:02.719+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.720157+0000) 2022-01-31T22:01:03.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:02 smithi167 conmon[49112]: debug 2022-01-31T22:01:02.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:02.881528+0000) 2022-01-31T22:01:03.256 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:03 smithi171 conmon[46715]: debug 2022-01-31T22:01:03.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:03.051905+0000) 2022-01-31T22:01:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:03 smithi171 conmon[41853]: debug 2022-01-31T22:01:03.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:03.257522+0000) 2022-01-31T22:01:03.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:03 smithi167 conmon[54076]: debug 2022-01-31T22:01:03.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:03.545700+0000) 2022-01-31T22:01:03.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:03 smithi167 conmon[60316]: debug 2022-01-31T22:01:03.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:03.545764+0000) 2022-01-31T22:01:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:03 smithi167 conmon[49112]: debug 2022-01-31T22:01:03.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:03.881617+0000) 2022-01-31T22:01:04.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:04 smithi171 conmon[46715]: debug 2022-01-31T22:01:04.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:04.052103+0000) 2022-01-31T22:01:04.213 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:03 smithi171 conmon[51620]: debug 2022-01-31T22:01:03.900+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:03.901329+0000) 2022-01-31T22:01:04.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:04 smithi167 conmon[54076]: debug 2022-01-31T22:01:04.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:04.545856+0000) 2022-01-31T22:01:04.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:04 smithi167 conmon[60316]: debug 2022-01-31T22:01:04.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:04.545880+0000) 2022-01-31T22:01:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:04 smithi171 conmon[41853]: debug 2022-01-31T22:01:04.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:04.257697+0000) 2022-01-31T22:01:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:04 smithi167 conmon[49112]: debug 2022-01-31T22:01:04.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:04.881784+0000) 2022-01-31T22:01:05.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:05 smithi171 conmon[46715]: debug 2022-01-31T22:01:05.051+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:05.052310+0000) 2022-01-31T22:01:05.213 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:04 smithi171 conmon[51620]: debug 2022-01-31T22:01:04.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:04.901508+0000) 2022-01-31T22:01:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:05 smithi171 conmon[41853]: debug 2022-01-31T22:01:05.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:05.257851+0000) 2022-01-31T22:01:05.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:05 smithi167 conmon[54076]: debug 2022-01-31T22:01:05.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:05.546059+0000) 2022-01-31T22:01:05.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:05 smithi167 conmon[60316]: debug 2022-01-31T22:01:05.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:05.546060+0000) 2022-01-31T22:01:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:05 smithi167 conmon[49112]: debug 2022-01-31T22:01:05.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:05.881978+0000) 2022-01-31T22:01:06.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:06 smithi171 conmon[46715]: debug 2022-01-31T22:01:06.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:06.052502+0000) 2022-01-31T22:01:06.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:05 smithi171 conmon[51620]: debug 2022-01-31T22:01:05.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:05.901692+0000) 2022-01-31T22:01:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:06 smithi171 conmon[41853]: debug 2022-01-31T22:01:06.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:06.258047+0000) 2022-01-31T22:01:06.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:06 smithi167 conmon[54076]: debug 2022-01-31T22:01:06.545+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:06.546248+0000) 2022-01-31T22:01:06.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:06 smithi167 conmon[60316]: debug 2022-01-31T22:01:06.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:06.546243+0000) 2022-01-31T22:01:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:06 smithi167 conmon[49112]: debug 2022-01-31T22:01:06.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:06.882179+0000) 2022-01-31T22:01:07.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[46715]: debug 2022-01-31T22:01:07.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.052662+0000) 2022-01-31T22:01:07.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:06 smithi171 conmon[51620]: debug 2022-01-31T22:01:06.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:06.901867+0000) 2022-01-31T22:01:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[41853]: debug 2022-01-31T22:01:07.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.258201+0000) 2022-01-31T22:01:07.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:07 smithi167 conmon[49112]: debug 2022-01-31T22:01:07.740+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.741354+0000) 2022-01-31T22:01:07.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:07 smithi167 conmon[54076]: debug 2022-01-31T22:01:07.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.546462+0000) 2022-01-31T22:01:07.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:07 smithi167 conmon[54076]: debug 2022-01-31T22:01:07.741+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.741810+0000) 2022-01-31T22:01:07.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:07 smithi167 conmon[60316]: debug 2022-01-31T22:01:07.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.546431+0000) 2022-01-31T22:01:07.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:07 smithi167 conmon[60316]: debug 2022-01-31T22:01:07.741+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.741583+0000) 2022-01-31T22:01:08.051 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[35325]: debug 2022-01-31T22:01:07.759+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175881 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:08.052 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[41853]: debug 2022-01-31T22:01:07.740+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.740856+0000) 2022-01-31T22:01:08.053 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[46715]: debug 2022-01-31T22:01:07.740+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.741681+0000) 2022-01-31T22:01:08.053 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[51620]: debug 2022-01-31T22:01:07.741+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.742303+0000) 2022-01-31T22:01:08.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:07 smithi171 conmon[51620]: debug 2022-01-31T22:01:07.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.902087+0000) 2022-01-31T22:01:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:07 smithi167 conmon[49112]: debug 2022-01-31T22:01:07.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:07.882367+0000) 2022-01-31T22:01:08.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:08 smithi171 conmon[46715]: debug 2022-01-31T22:01:08.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:08.052765+0000) 2022-01-31T22:01:08.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:08 smithi171 conmon[41853]: debug 2022-01-31T22:01:08.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:08.258398+0000) 2022-01-31T22:01:08.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:08 smithi167 conmon[54076]: debug 2022-01-31T22:01:08.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:08.546707+0000) 2022-01-31T22:01:08.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:08 smithi167 conmon[60316]: debug 2022-01-31T22:01:08.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:08.546706+0000) 2022-01-31T22:01:09.052 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:08 smithi171 conmon[35325]: debug 2022-01-31T22:01:08.776+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:01:09.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:08 smithi171 conmon[51620]: debug 2022-01-31T22:01:08.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:08.902181+0000) 2022-01-31T22:01:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:08 smithi167 conmon[49112]: debug 2022-01-31T22:01:08.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:08.882549+0000) 2022-01-31T22:01:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:09 smithi171 conmon[41853]: debug 2022-01-31T22:01:09.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:09.258551+0000) 2022-01-31T22:01:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:09 smithi171 conmon[46715]: debug 2022-01-31T22:01:09.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:09.052898+0000) 2022-01-31T22:01:09.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:09 smithi167 conmon[54076]: debug 2022-01-31T22:01:09.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:09.546809+0000) 2022-01-31T22:01:09.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:09 smithi167 conmon[60316]: debug 2022-01-31T22:01:09.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:09.546807+0000) 2022-01-31T22:01:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:09 smithi167 conmon[49112]: debug 2022-01-31T22:01:09.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:09.882745+0000) 2022-01-31T22:01:10.214 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:10 smithi171 conmon[46715]: debug 2022-01-31T22:01:10.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:10.053108+0000) 2022-01-31T22:01:10.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:09 smithi171 conmon[51620]: debug 2022-01-31T22:01:09.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:09.902371+0000) 2022-01-31T22:01:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:10 smithi171 conmon[41853]: debug 2022-01-31T22:01:10.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:10.258787+0000) 2022-01-31T22:01:10.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:10 smithi167 conmon[54076]: debug 2022-01-31T22:01:10.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:10.546990+0000) 2022-01-31T22:01:10.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:10 smithi167 conmon[60316]: debug 2022-01-31T22:01:10.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:10.546982+0000) 2022-01-31T22:01:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:10 smithi167 conmon[49112]: debug 2022-01-31T22:01:10.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:10.882958+0000) 2022-01-31T22:01:11.214 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:11 smithi171 conmon[46715]: debug 2022-01-31T22:01:11.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:11.053291+0000) 2022-01-31T22:01:11.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:10 smithi171 conmon[51620]: debug 2022-01-31T22:01:10.901+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:10.902573+0000) 2022-01-31T22:01:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:11 smithi171 conmon[41853]: debug 2022-01-31T22:01:11.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:11.258969+0000) 2022-01-31T22:01:11.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:11 smithi167 conmon[54076]: debug 2022-01-31T22:01:11.546+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:11.547182+0000) 2022-01-31T22:01:11.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:11 smithi167 conmon[60316]: debug 2022-01-31T22:01:11.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:11.547186+0000) 2022-01-31T22:01:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:11 smithi167 conmon[49112]: debug 2022-01-31T22:01:11.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:11.883126+0000) 2022-01-31T22:01:12.214 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[46715]: debug 2022-01-31T22:01:12.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.053475+0000) 2022-01-31T22:01:12.215 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:11 smithi171 conmon[51620]: debug 2022-01-31T22:01:11.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:11.902779+0000) 2022-01-31T22:01:12.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[41853]: debug 2022-01-31T22:01:12.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.259156+0000) 2022-01-31T22:01:12.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:12 smithi167 conmon[49112]: debug 2022-01-31T22:01:12.762+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.763087+0000) 2022-01-31T22:01:12.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:12 smithi167 conmon[54076]: debug 2022-01-31T22:01:12.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.547378+0000) 2022-01-31T22:01:12.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:12 smithi167 conmon[54076]: debug 2022-01-31T22:01:12.762+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.763268+0000) 2022-01-31T22:01:12.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:12 smithi167 conmon[60316]: debug 2022-01-31T22:01:12.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.547377+0000) 2022-01-31T22:01:12.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:12 smithi167 conmon[60316]: debug 2022-01-31T22:01:12.762+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.762735+0000) 2022-01-31T22:01:13.052 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[35325]: debug 2022-01-31T22:01:12.780+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 175996 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:13.053 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[41853]: debug 2022-01-31T22:01:12.761+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.762211+0000) 2022-01-31T22:01:13.054 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[46715]: debug 2022-01-31T22:01:12.762+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.763364+0000) 2022-01-31T22:01:13.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[51620]: debug 2022-01-31T22:01:12.762+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.763698+0000) 2022-01-31T22:01:13.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:12 smithi171 conmon[51620]: debug 2022-01-31T22:01:12.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.902903+0000) 2022-01-31T22:01:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:12 smithi167 conmon[49112]: debug 2022-01-31T22:01:12.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:12.883297+0000) 2022-01-31T22:01:13.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:13 smithi171 conmon[41853]: debug 2022-01-31T22:01:13.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:13.259352+0000) 2022-01-31T22:01:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:13 smithi171 conmon[46715]: debug 2022-01-31T22:01:13.052+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:13.053627+0000) 2022-01-31T22:01:13.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:13 smithi167 conmon[54076]: debug 2022-01-31T22:01:13.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:13.547584+0000) 2022-01-31T22:01:13.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:13 smithi167 conmon[60316]: debug 2022-01-31T22:01:13.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:13.547586+0000) 2022-01-31T22:01:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:13 smithi167 conmon[49112]: debug 2022-01-31T22:01:13.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:13.883462+0000) 2022-01-31T22:01:14.214 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:14 smithi171 conmon[46715]: debug 2022-01-31T22:01:14.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:14.053799+0000) 2022-01-31T22:01:14.215 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:13 smithi171 conmon[51620]: debug 2022-01-31T22:01:13.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:13.902990+0000) 2022-01-31T22:01:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:14 smithi171 conmon[41853]: debug 2022-01-31T22:01:14.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:14.259550+0000) 2022-01-31T22:01:14.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:14 smithi167 conmon[54076]: debug 2022-01-31T22:01:14.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:14.547744+0000) 2022-01-31T22:01:14.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:14 smithi167 conmon[60316]: debug 2022-01-31T22:01:14.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:14.547708+0000) 2022-01-31T22:01:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:14 smithi167 conmon[49112]: debug 2022-01-31T22:01:14.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:14.883704+0000) 2022-01-31T22:01:15.214 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:15 smithi171 conmon[46715]: debug 2022-01-31T22:01:15.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:15.053953+0000) 2022-01-31T22:01:15.215 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:14 smithi171 conmon[51620]: debug 2022-01-31T22:01:14.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:14.903148+0000) 2022-01-31T22:01:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:15 smithi171 conmon[41853]: debug 2022-01-31T22:01:15.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:15.259761+0000) 2022-01-31T22:01:15.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:15 smithi167 conmon[54076]: debug 2022-01-31T22:01:15.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:15.547955+0000) 2022-01-31T22:01:15.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:15 smithi167 conmon[60316]: debug 2022-01-31T22:01:15.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:15.547932+0000) 2022-01-31T22:01:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:15 smithi167 conmon[49112]: debug 2022-01-31T22:01:15.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:15.883888+0000) 2022-01-31T22:01:16.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:16 smithi171 conmon[46715]: debug 2022-01-31T22:01:16.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:16.054165+0000) 2022-01-31T22:01:16.215 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:15 smithi171 conmon[51620]: debug 2022-01-31T22:01:15.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:15.903298+0000) 2022-01-31T22:01:16.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:16 smithi167 conmon[54076]: debug 2022-01-31T22:01:16.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:16.548161+0000) 2022-01-31T22:01:16.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:16 smithi167 conmon[60316]: debug 2022-01-31T22:01:16.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:16.548104+0000) 2022-01-31T22:01:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:16 smithi171 conmon[41853]: debug 2022-01-31T22:01:16.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:16.259908+0000) 2022-01-31T22:01:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:16 smithi167 conmon[49112]: debug 2022-01-31T22:01:16.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:16.884024+0000) 2022-01-31T22:01:17.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[46715]: debug 2022-01-31T22:01:17.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.054352+0000) 2022-01-31T22:01:17.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:16 smithi171 conmon[51620]: debug 2022-01-31T22:01:16.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:16.903445+0000) 2022-01-31T22:01:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[41853]: debug 2022-01-31T22:01:17.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.260080+0000) 2022-01-31T22:01:17.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:17 smithi167 conmon[54076]: debug 2022-01-31T22:01:17.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.548328+0000) 2022-01-31T22:01:17.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:17 smithi167 conmon[60316]: debug 2022-01-31T22:01:17.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.548293+0000) 2022-01-31T22:01:18.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:17 smithi167 conmon[54076]: debug 2022-01-31T22:01:17.784+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.784992+0000) 2022-01-31T22:01:18.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:17 smithi167 conmon[60316]: debug 2022-01-31T22:01:17.783+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.784233+0000) 2022-01-31T22:01:18.042 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:17 smithi167 conmon[49112]: debug 2022-01-31T22:01:17.784+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.785035+0000) 2022-01-31T22:01:18.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:17 smithi167 conmon[49112]: debug 2022-01-31T22:01:17.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.884206+0000) 2022-01-31T22:01:18.053 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[41853]: debug 2022-01-31T22:01:17.783+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.784548+0000) 2022-01-31T22:01:18.053 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[35325]: debug 2022-01-31T22:01:17.801+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176105 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:18.054 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[46715]: debug 2022-01-31T22:01:17.782+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.783695+0000) 2022-01-31T22:01:18.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[51620]: debug 2022-01-31T22:01:17.783+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.783974+0000) 2022-01-31T22:01:18.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:17 smithi171 conmon[51620]: debug 2022-01-31T22:01:17.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:17.903620+0000) 2022-01-31T22:01:18.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:18 smithi171 conmon[46715]: debug 2022-01-31T22:01:18.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:18.054479+0000) 2022-01-31T22:01:18.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:18 smithi171 conmon[41853]: debug 2022-01-31T22:01:18.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:18.260238+0000) 2022-01-31T22:01:18.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:18 smithi167 conmon[54076]: debug 2022-01-31T22:01:18.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:18.548529+0000) 2022-01-31T22:01:18.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:18 smithi167 conmon[60316]: debug 2022-01-31T22:01:18.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:18.548480+0000) 2022-01-31T22:01:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:18 smithi167 conmon[49112]: debug 2022-01-31T22:01:18.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:18.884329+0000) 2022-01-31T22:01:19.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:19 smithi171 conmon[46715]: debug 2022-01-31T22:01:19.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:19.054643+0000) 2022-01-31T22:01:19.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:18 smithi171 conmon[51620]: debug 2022-01-31T22:01:18.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:18.903699+0000) 2022-01-31T22:01:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:19 smithi171 conmon[41853]: debug 2022-01-31T22:01:19.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:19.260385+0000) 2022-01-31T22:01:19.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:19 smithi167 conmon[54076]: debug 2022-01-31T22:01:19.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:19.548659+0000) 2022-01-31T22:01:19.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:19 smithi167 conmon[60316]: debug 2022-01-31T22:01:19.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:19.548660+0000) 2022-01-31T22:01:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:19 smithi167 conmon[49112]: debug 2022-01-31T22:01:19.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:19.884489+0000) 2022-01-31T22:01:20.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:20 smithi171 conmon[46715]: debug 2022-01-31T22:01:20.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:20.054836+0000) 2022-01-31T22:01:20.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:19 smithi171 conmon[51620]: debug 2022-01-31T22:01:19.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:19.903836+0000) 2022-01-31T22:01:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:20 smithi171 conmon[41853]: debug 2022-01-31T22:01:20.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:20.260568+0000) 2022-01-31T22:01:20.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:20 smithi167 conmon[54076]: debug 2022-01-31T22:01:20.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:20.548885+0000) 2022-01-31T22:01:20.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:20 smithi167 conmon[60316]: debug 2022-01-31T22:01:20.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:20.548863+0000) 2022-01-31T22:01:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:20 smithi167 conmon[49112]: debug 2022-01-31T22:01:20.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:20.884681+0000) 2022-01-31T22:01:21.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:21 smithi171 conmon[46715]: debug 2022-01-31T22:01:21.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:21.055002+0000) 2022-01-31T22:01:21.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:20 smithi171 conmon[51620]: debug 2022-01-31T22:01:20.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:20.903974+0000) 2022-01-31T22:01:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:21 smithi171 conmon[41853]: debug 2022-01-31T22:01:21.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:21.260725+0000) 2022-01-31T22:01:21.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:21 smithi167 conmon[60316]: debug 2022-01-31T22:01:21.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:21.549084+0000) 2022-01-31T22:01:21.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:21 smithi167 conmon[54076]: debug 2022-01-31T22:01:21.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:21.549059+0000) 2022-01-31T22:01:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:21 smithi167 conmon[49112]: debug 2022-01-31T22:01:21.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:21.884890+0000) 2022-01-31T22:01:22.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:21 smithi171 conmon[51620]: debug 2022-01-31T22:01:21.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:21.904136+0000) 2022-01-31T22:01:22.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[46715]: debug 2022-01-31T22:01:22.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.055177+0000) 2022-01-31T22:01:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[41853]: debug 2022-01-31T22:01:22.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.260941+0000) 2022-01-31T22:01:22.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:22 smithi167 conmon[54076]: debug 2022-01-31T22:01:22.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.549239+0000) 2022-01-31T22:01:22.804 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:22 smithi167 conmon[60316]: debug 2022-01-31T22:01:22.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.549240+0000) 2022-01-31T22:01:23.054 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[35325]: debug 2022-01-31T22:01:22.821+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176218 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:23.055 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[41853]: debug 2022-01-31T22:01:22.804+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.806073+0000) 2022-01-31T22:01:23.055 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[46715]: debug 2022-01-31T22:01:22.802+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.804112+0000) 2022-01-31T22:01:23.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[51620]: debug 2022-01-31T22:01:22.803+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.804782+0000) 2022-01-31T22:01:23.056 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:22 smithi171 conmon[51620]: debug 2022-01-31T22:01:22.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.904315+0000) 2022-01-31T22:01:23.059 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:22 smithi167 conmon[54076]: debug 2022-01-31T22:01:22.804+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.805047+0000) 2022-01-31T22:01:23.060 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:22 smithi167 conmon[60316]: debug 2022-01-31T22:01:22.805+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.806064+0000) 2022-01-31T22:01:23.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:22 smithi167 conmon[49112]: debug 2022-01-31T22:01:22.805+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.805774+0000) 2022-01-31T22:01:23.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:22 smithi167 conmon[49112]: debug 2022-01-31T22:01:22.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:22.885064+0000) 2022-01-31T22:01:23.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:23 smithi171 conmon[41853]: debug 2022-01-31T22:01:23.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:23.261136+0000) 2022-01-31T22:01:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:23 smithi171 conmon[46715]: debug 2022-01-31T22:01:23.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:23.055328+0000) 2022-01-31T22:01:23.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:23 smithi167 conmon[54076]: debug 2022-01-31T22:01:23.547+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:23.549406+0000) 2022-01-31T22:01:23.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:23 smithi167 conmon[60316]: debug 2022-01-31T22:01:23.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:23.549410+0000) 2022-01-31T22:01:24.054 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:23 smithi171 conmon[35325]: debug 2022-01-31T22:01:23.776+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:01:24.055 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:23 smithi171 conmon[51620]: debug 2022-01-31T22:01:23.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:23.904485+0000) 2022-01-31T22:01:24.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:23 smithi167 conmon[49112]: debug 2022-01-31T22:01:23.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:23.885257+0000) 2022-01-31T22:01:24.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:24 smithi171 conmon[41853]: debug 2022-01-31T22:01:24.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:24.261282+0000) 2022-01-31T22:01:24.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:24 smithi171 conmon[46715]: debug 2022-01-31T22:01:24.053+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:24.055516+0000) 2022-01-31T22:01:24.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:24 smithi167 conmon[54076]: debug 2022-01-31T22:01:24.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:24.549545+0000) 2022-01-31T22:01:24.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:24 smithi167 conmon[60316]: debug 2022-01-31T22:01:24.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:24.549548+0000) 2022-01-31T22:01:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:24 smithi167 conmon[49112]: debug 2022-01-31T22:01:24.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:24.885464+0000) 2022-01-31T22:01:25.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:25 smithi171 conmon[46715]: debug 2022-01-31T22:01:25.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:25.055706+0000) 2022-01-31T22:01:25.217 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:24 smithi171 conmon[51620]: debug 2022-01-31T22:01:24.902+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:24.904698+0000) 2022-01-31T22:01:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:25 smithi171 conmon[41853]: debug 2022-01-31T22:01:25.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:25.261460+0000) 2022-01-31T22:01:25.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:25 smithi167 conmon[54076]: debug 2022-01-31T22:01:25.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:25.549741+0000) 2022-01-31T22:01:25.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:25 smithi167 conmon[60316]: debug 2022-01-31T22:01:25.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:25.549741+0000) 2022-01-31T22:01:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:25 smithi167 conmon[49112]: debug 2022-01-31T22:01:25.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:25.885691+0000) 2022-01-31T22:01:26.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:26 smithi171 conmon[46715]: debug 2022-01-31T22:01:26.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:26.055886+0000) 2022-01-31T22:01:26.217 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:25 smithi171 conmon[51620]: debug 2022-01-31T22:01:25.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:25.904856+0000) 2022-01-31T22:01:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:26 smithi171 conmon[41853]: debug 2022-01-31T22:01:26.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:26.261666+0000) 2022-01-31T22:01:26.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:26 smithi167 conmon[54076]: debug 2022-01-31T22:01:26.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:26.549936+0000) 2022-01-31T22:01:26.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:26 smithi167 conmon[60316]: debug 2022-01-31T22:01:26.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:26.549942+0000) 2022-01-31T22:01:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:26 smithi167 conmon[49112]: debug 2022-01-31T22:01:26.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:26.885843+0000) 2022-01-31T22:01:27.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[46715]: debug 2022-01-31T22:01:27.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.056044+0000) 2022-01-31T22:01:27.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:26 smithi171 conmon[51620]: debug 2022-01-31T22:01:26.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:26.905010+0000) 2022-01-31T22:01:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[41853]: debug 2022-01-31T22:01:27.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.261872+0000) 2022-01-31T22:01:27.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:27 smithi167 conmon[54076]: debug 2022-01-31T22:01:27.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.550098+0000) 2022-01-31T22:01:27.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:27 smithi167 conmon[60316]: debug 2022-01-31T22:01:27.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.550090+0000) 2022-01-31T22:01:28.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[35325]: debug 2022-01-31T22:01:27.842+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176327 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:28.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[41853]: debug 2022-01-31T22:01:27.824+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.826671+0000) 2022-01-31T22:01:28.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[46715]: debug 2022-01-31T22:01:27.823+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.825166+0000) 2022-01-31T22:01:28.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:28 smithi171 conmon[46715]: debug 2022-01-31T22:01:28.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:28.056244+0000) 2022-01-31T22:01:28.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[51620]: debug 2022-01-31T22:01:27.823+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.825336+0000) 2022-01-31T22:01:28.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:27 smithi171 conmon[51620]: debug 2022-01-31T22:01:27.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.905194+0000) 2022-01-31T22:01:28.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:27 smithi167 conmon[54076]: debug 2022-01-31T22:01:27.824+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.825817+0000) 2022-01-31T22:01:28.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:27 smithi167 conmon[60316]: debug 2022-01-31T22:01:27.824+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.825529+0000) 2022-01-31T22:01:28.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:27 smithi167 conmon[49112]: debug 2022-01-31T22:01:27.824+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.825691+0000) 2022-01-31T22:01:28.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:27 smithi167 conmon[49112]: debug 2022-01-31T22:01:27.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:27.886027+0000) 2022-01-31T22:01:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:28 smithi171 conmon[41853]: debug 2022-01-31T22:01:28.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:28.262052+0000) 2022-01-31T22:01:28.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:28 smithi167 conmon[54076]: debug 2022-01-31T22:01:28.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:28.550277+0000) 2022-01-31T22:01:28.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:28 smithi167 conmon[60316]: debug 2022-01-31T22:01:28.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:28.550278+0000) 2022-01-31T22:01:29.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:28 smithi167 conmon[49112]: debug 2022-01-31T22:01:28.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:28.886236+0000) 2022-01-31T22:01:29.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:29 smithi171 conmon[46715]: debug 2022-01-31T22:01:29.054+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:29.056418+0000) 2022-01-31T22:01:29.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:28 smithi171 conmon[51620]: debug 2022-01-31T22:01:28.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:28.905338+0000) 2022-01-31T22:01:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:29 smithi171 conmon[41853]: debug 2022-01-31T22:01:29.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:29.262191+0000) 2022-01-31T22:01:29.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:29 smithi167 conmon[60316]: debug 2022-01-31T22:01:29.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:29.550455+0000) 2022-01-31T22:01:29.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:29 smithi167 conmon[54076]: debug 2022-01-31T22:01:29.548+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:29.550457+0000) 2022-01-31T22:01:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:29 smithi167 conmon[49112]: debug 2022-01-31T22:01:29.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:29.886434+0000) 2022-01-31T22:01:30.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:30 smithi171 conmon[46715]: debug 2022-01-31T22:01:30.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:30.056604+0000) 2022-01-31T22:01:30.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:29 smithi171 conmon[51620]: debug 2022-01-31T22:01:29.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:29.905488+0000) 2022-01-31T22:01:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:30 smithi171 conmon[41853]: debug 2022-01-31T22:01:30.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:30.262347+0000) 2022-01-31T22:01:30.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:30 smithi167 conmon[60316]: debug 2022-01-31T22:01:30.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:30.550690+0000) 2022-01-31T22:01:30.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:30 smithi167 conmon[54076]: debug 2022-01-31T22:01:30.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:30.550670+0000) 2022-01-31T22:01:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:30 smithi167 conmon[49112]: debug 2022-01-31T22:01:30.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:30.886670+0000) 2022-01-31T22:01:31.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:31 smithi171 conmon[46715]: debug 2022-01-31T22:01:31.055+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:31.056776+0000) 2022-01-31T22:01:31.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:30 smithi171 conmon[51620]: debug 2022-01-31T22:01:30.903+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:30.905626+0000) 2022-01-31T22:01:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:31 smithi171 conmon[41853]: debug 2022-01-31T22:01:31.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:31.262501+0000) 2022-01-31T22:01:31.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:31 smithi167 conmon[54076]: debug 2022-01-31T22:01:31.549+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:31.550828+0000) 2022-01-31T22:01:31.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:31 smithi167 conmon[60316]: debug 2022-01-31T22:01:31.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:31.550900+0000) 2022-01-31T22:01:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:31 smithi167 conmon[49112]: debug 2022-01-31T22:01:31.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:31.886886+0000) 2022-01-31T22:01:32.218 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[46715]: debug 2022-01-31T22:01:32.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.056963+0000) 2022-01-31T22:01:32.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:31 smithi171 conmon[51620]: debug 2022-01-31T22:01:31.904+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:31.905772+0000) 2022-01-31T22:01:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[41853]: debug 2022-01-31T22:01:32.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.262682+0000) 2022-01-31T22:01:32.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:32 smithi167 conmon[54076]: debug 2022-01-31T22:01:32.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.551019+0000) 2022-01-31T22:01:32.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:32 smithi167 conmon[60316]: debug 2022-01-31T22:01:32.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.551104+0000) 2022-01-31T22:01:33.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[35325]: debug 2022-01-31T22:01:32.878+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176453 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:33.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[41853]: debug 2022-01-31T22:01:32.846+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.847882+0000) 2022-01-31T22:01:33.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[46715]: debug 2022-01-31T22:01:32.845+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.847182+0000) 2022-01-31T22:01:33.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:33 smithi171 conmon[46715]: debug 2022-01-31T22:01:33.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:33.057170+0000) 2022-01-31T22:01:33.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[51620]: debug 2022-01-31T22:01:32.844+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.846564+0000) 2022-01-31T22:01:33.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:32 smithi171 conmon[51620]: debug 2022-01-31T22:01:32.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.905915+0000) 2022-01-31T22:01:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:32 smithi167 conmon[49112]: debug 2022-01-31T22:01:32.846+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.847636+0000) 2022-01-31T22:01:33.319 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:32 smithi167 conmon[49112]: debug 2022-01-31T22:01:32.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.887041+0000) 2022-01-31T22:01:33.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:32 smithi167 conmon[54076]: debug 2022-01-31T22:01:32.845+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.846904+0000) 2022-01-31T22:01:33.321 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:32 smithi167 conmon[60316]: debug 2022-01-31T22:01:32.846+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:32.847568+0000) 2022-01-31T22:01:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:33 smithi171 conmon[41853]: debug 2022-01-31T22:01:33.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:33.262914+0000) 2022-01-31T22:01:33.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:33 smithi167 conmon[54076]: debug 2022-01-31T22:01:33.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:33.551182+0000) 2022-01-31T22:01:33.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:33 smithi167 conmon[60316]: debug 2022-01-31T22:01:33.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:33.551293+0000) 2022-01-31T22:01:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:33 smithi167 conmon[49112]: debug 2022-01-31T22:01:33.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:33.887209+0000) 2022-01-31T22:01:34.218 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:34 smithi171 conmon[46715]: debug 2022-01-31T22:01:34.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:34.057369+0000) 2022-01-31T22:01:34.219 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:33 smithi171 conmon[51620]: debug 2022-01-31T22:01:33.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:33.906034+0000) 2022-01-31T22:01:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:34 smithi171 conmon[41853]: debug 2022-01-31T22:01:34.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:34.263070+0000) 2022-01-31T22:01:34.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:34 smithi167 conmon[54076]: debug 2022-01-31T22:01:34.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:34.551317+0000) 2022-01-31T22:01:34.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:34 smithi167 conmon[60316]: debug 2022-01-31T22:01:34.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:34.551435+0000) 2022-01-31T22:01:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:34 smithi167 conmon[49112]: debug 2022-01-31T22:01:34.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:34.887373+0000) 2022-01-31T22:01:35.218 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:35 smithi171 conmon[46715]: debug 2022-01-31T22:01:35.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:35.057527+0000) 2022-01-31T22:01:35.219 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:34 smithi171 conmon[51620]: debug 2022-01-31T22:01:34.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:34.906218+0000) 2022-01-31T22:01:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:35 smithi171 conmon[41853]: debug 2022-01-31T22:01:35.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:35.263253+0000) 2022-01-31T22:01:35.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:35 smithi167 conmon[60316]: debug 2022-01-31T22:01:35.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:35.551625+0000) 2022-01-31T22:01:35.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:35 smithi167 conmon[54076]: debug 2022-01-31T22:01:35.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:35.551513+0000) 2022-01-31T22:01:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:35 smithi167 conmon[49112]: debug 2022-01-31T22:01:35.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:35.887567+0000) 2022-01-31T22:01:36.218 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:36 smithi171 conmon[46715]: debug 2022-01-31T22:01:36.056+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:36.057708+0000) 2022-01-31T22:01:36.219 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:35 smithi171 conmon[51620]: debug 2022-01-31T22:01:35.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:35.906407+0000) 2022-01-31T22:01:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:36 smithi171 conmon[41853]: debug 2022-01-31T22:01:36.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:36.263435+0000) 2022-01-31T22:01:36.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:36 smithi167 conmon[54076]: debug 2022-01-31T22:01:36.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:36.551670+0000) 2022-01-31T22:01:36.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:36 smithi167 conmon[60316]: debug 2022-01-31T22:01:36.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:36.551750+0000) 2022-01-31T22:01:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:36 smithi167 conmon[49112]: debug 2022-01-31T22:01:36.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:36.887742+0000) 2022-01-31T22:01:37.218 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[46715]: debug 2022-01-31T22:01:37.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.057915+0000) 2022-01-31T22:01:37.219 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:36 smithi171 conmon[51620]: debug 2022-01-31T22:01:36.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:36.906586+0000) 2022-01-31T22:01:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[41853]: debug 2022-01-31T22:01:37.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.263618+0000) 2022-01-31T22:01:37.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:37 smithi167 conmon[54076]: debug 2022-01-31T22:01:37.550+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.551797+0000) 2022-01-31T22:01:37.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:37 smithi167 conmon[60316]: debug 2022-01-31T22:01:37.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.551855+0000) 2022-01-31T22:01:38.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:37 smithi167 conmon[54076]: debug 2022-01-31T22:01:37.881+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.883225+0000) 2022-01-31T22:01:38.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:37 smithi167 conmon[49112]: debug 2022-01-31T22:01:37.881+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.883117+0000) 2022-01-31T22:01:38.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:37 smithi167 conmon[49112]: debug 2022-01-31T22:01:37.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.887888+0000) 2022-01-31T22:01:38.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:37 smithi167 conmon[60316]: debug 2022-01-31T22:01:37.881+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.882886+0000) 2022-01-31T22:01:38.219 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[35325]: debug 2022-01-31T22:01:37.900+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176562 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:38.219 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[41853]: debug 2022-01-31T22:01:37.882+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.883718+0000) 2022-01-31T22:01:38.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[51620]: debug 2022-01-31T22:01:37.882+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.882805+0000) 2022-01-31T22:01:38.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[51620]: debug 2022-01-31T22:01:37.905+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.906717+0000) 2022-01-31T22:01:38.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:37 smithi171 conmon[46715]: debug 2022-01-31T22:01:37.881+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:37.882545+0000) 2022-01-31T22:01:38.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:38 smithi171 conmon[46715]: debug 2022-01-31T22:01:38.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:38.058105+0000) 2022-01-31T22:01:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:38 smithi171 conmon[41853]: debug 2022-01-31T22:01:38.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:38.263800+0000) 2022-01-31T22:01:38.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:38 smithi167 conmon[60316]: debug 2022-01-31T22:01:38.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:38.551985+0000) 2022-01-31T22:01:38.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:38 smithi167 conmon[54076]: debug 2022-01-31T22:01:38.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:38.551965+0000) 2022-01-31T22:01:39.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:38 smithi171 conmon[35325]: debug 2022-01-31T22:01:38.777+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:01:39.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:38 smithi171 conmon[51620]: debug 2022-01-31T22:01:38.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:38.906866+0000) 2022-01-31T22:01:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:38 smithi167 conmon[49112]: debug 2022-01-31T22:01:38.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:38.888053+0000) 2022-01-31T22:01:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:39 smithi171 conmon[41853]: debug 2022-01-31T22:01:39.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:39.263963+0000) 2022-01-31T22:01:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:39 smithi171 conmon[46715]: debug 2022-01-31T22:01:39.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:39.058267+0000) 2022-01-31T22:01:39.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:39 smithi167 conmon[60316]: debug 2022-01-31T22:01:39.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:39.552128+0000) 2022-01-31T22:01:39.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:39 smithi167 conmon[54076]: debug 2022-01-31T22:01:39.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:39.552127+0000) 2022-01-31T22:01:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:39 smithi167 conmon[49112]: debug 2022-01-31T22:01:39.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:39.888283+0000) 2022-01-31T22:01:40.219 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:40 smithi171 conmon[46715]: debug 2022-01-31T22:01:40.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:40.058455+0000) 2022-01-31T22:01:40.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:39 smithi171 conmon[51620]: debug 2022-01-31T22:01:39.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:39.907071+0000) 2022-01-31T22:01:40.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:40 smithi171 conmon[41853]: debug 2022-01-31T22:01:40.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:40.264147+0000) 2022-01-31T22:01:40.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:40 smithi167 conmon[54076]: debug 2022-01-31T22:01:40.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:40.552346+0000) 2022-01-31T22:01:40.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:40 smithi167 conmon[60316]: debug 2022-01-31T22:01:40.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:40.552327+0000) 2022-01-31T22:01:41.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:40 smithi167 conmon[49112]: debug 2022-01-31T22:01:40.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:40.888522+0000) 2022-01-31T22:01:41.219 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:41 smithi171 conmon[46715]: debug 2022-01-31T22:01:41.057+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:41.058631+0000) 2022-01-31T22:01:41.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:40 smithi171 conmon[51620]: debug 2022-01-31T22:01:40.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:40.907236+0000) 2022-01-31T22:01:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:41 smithi171 conmon[41853]: debug 2022-01-31T22:01:41.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:41.264333+0000) 2022-01-31T22:01:41.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:41 smithi167 conmon[54076]: debug 2022-01-31T22:01:41.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:41.552520+0000) 2022-01-31T22:01:41.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:41 smithi167 conmon[60316]: debug 2022-01-31T22:01:41.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:41.552521+0000) 2022-01-31T22:01:42.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:41 smithi167 conmon[49112]: debug 2022-01-31T22:01:41.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:41.888687+0000) 2022-01-31T22:01:42.219 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[46715]: debug 2022-01-31T22:01:42.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.058811+0000) 2022-01-31T22:01:42.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:41 smithi171 conmon[51620]: debug 2022-01-31T22:01:41.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:41.907390+0000) 2022-01-31T22:01:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[41853]: debug 2022-01-31T22:01:42.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.264514+0000) 2022-01-31T22:01:42.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:42 smithi167 conmon[54076]: debug 2022-01-31T22:01:42.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.552708+0000) 2022-01-31T22:01:42.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:42 smithi167 conmon[60316]: debug 2022-01-31T22:01:42.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.552674+0000) 2022-01-31T22:01:43.109 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:42 smithi167 conmon[60316]: debug 2022-01-31T22:01:42.902+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.903864+0000) 2022-01-31T22:01:43.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:42 smithi167 conmon[49112]: debug 2022-01-31T22:01:42.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.888882+0000) 2022-01-31T22:01:43.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:42 smithi167 conmon[49112]: debug 2022-01-31T22:01:42.904+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.905676+0000) 2022-01-31T22:01:43.111 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:42 smithi167 conmon[54076]: debug 2022-01-31T22:01:42.903+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.904976+0000) 2022-01-31T22:01:43.220 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[35325]: debug 2022-01-31T22:01:42.922+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176675 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:43.220 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[41853]: debug 2022-01-31T22:01:42.903+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.904449+0000) 2022-01-31T22:01:43.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[46715]: debug 2022-01-31T22:01:42.902+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.903501+0000) 2022-01-31T22:01:43.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:43 smithi171 conmon[46715]: debug 2022-01-31T22:01:43.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:43.059014+0000) 2022-01-31T22:01:43.222 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[51620]: debug 2022-01-31T22:01:42.904+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.904937+0000) 2022-01-31T22:01:43.222 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:42 smithi171 conmon[51620]: debug 2022-01-31T22:01:42.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:42.907534+0000) 2022-01-31T22:01:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:43 smithi171 conmon[41853]: debug 2022-01-31T22:01:43.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:43.264691+0000) 2022-01-31T22:01:43.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:43 smithi167 conmon[54076]: debug 2022-01-31T22:01:43.551+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:43.552865+0000) 2022-01-31T22:01:43.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:43 smithi167 conmon[60316]: debug 2022-01-31T22:01:43.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:43.552850+0000) 2022-01-31T22:01:44.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:43 smithi167 conmon[49112]: debug 2022-01-31T22:01:43.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:43.889079+0000) 2022-01-31T22:01:44.220 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:44 smithi171 conmon[46715]: debug 2022-01-31T22:01:44.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:44.059130+0000) 2022-01-31T22:01:44.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:43 smithi171 conmon[51620]: debug 2022-01-31T22:01:43.906+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:43.907666+0000) 2022-01-31T22:01:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:44 smithi171 conmon[41853]: debug 2022-01-31T22:01:44.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:44.264820+0000) 2022-01-31T22:01:44.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:44 smithi167 conmon[54076]: debug 2022-01-31T22:01:44.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:44.553023+0000) 2022-01-31T22:01:44.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:44 smithi167 conmon[60316]: debug 2022-01-31T22:01:44.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:44.553016+0000) 2022-01-31T22:01:45.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:44 smithi167 conmon[49112]: debug 2022-01-31T22:01:44.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:44.889280+0000) 2022-01-31T22:01:45.220 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:45 smithi171 conmon[46715]: debug 2022-01-31T22:01:45.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:45.059247+0000) 2022-01-31T22:01:45.220 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:44 smithi171 conmon[51620]: debug 2022-01-31T22:01:44.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:44.907871+0000) 2022-01-31T22:01:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:45 smithi171 conmon[41853]: debug 2022-01-31T22:01:45.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:45.264944+0000) 2022-01-31T22:01:45.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:45 smithi167 conmon[54076]: debug 2022-01-31T22:01:45.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:45.553227+0000) 2022-01-31T22:01:45.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:45 smithi167 conmon[60316]: debug 2022-01-31T22:01:45.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:45.553206+0000) 2022-01-31T22:01:46.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:45 smithi167 conmon[49112]: debug 2022-01-31T22:01:45.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:45.889519+0000) 2022-01-31T22:01:46.220 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:46 smithi171 conmon[46715]: debug 2022-01-31T22:01:46.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:46.059361+0000) 2022-01-31T22:01:46.221 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:45 smithi171 conmon[51620]: debug 2022-01-31T22:01:45.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:45.908048+0000) 2022-01-31T22:01:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:46 smithi171 conmon[41853]: debug 2022-01-31T22:01:46.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:46.265087+0000) 2022-01-31T22:01:46.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:46 smithi167 conmon[54076]: debug 2022-01-31T22:01:46.552+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:46.553415+0000) 2022-01-31T22:01:46.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:46 smithi167 conmon[60316]: debug 2022-01-31T22:01:46.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:46.553408+0000) 2022-01-31T22:01:47.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:46 smithi167 conmon[49112]: debug 2022-01-31T22:01:46.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:46.889733+0000) 2022-01-31T22:01:47.220 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[46715]: debug 2022-01-31T22:01:47.058+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.059506+0000) 2022-01-31T22:01:47.221 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:46 smithi171 conmon[51620]: debug 2022-01-31T22:01:46.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:46.908223+0000) 2022-01-31T22:01:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[41853]: debug 2022-01-31T22:01:47.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.265314+0000) 2022-01-31T22:01:47.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:47 smithi167 conmon[54076]: debug 2022-01-31T22:01:47.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.553618+0000) 2022-01-31T22:01:47.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:47 smithi167 conmon[60316]: debug 2022-01-31T22:01:47.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.553599+0000) 2022-01-31T22:01:48.121 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:47 smithi167 conmon[54076]: debug 2022-01-31T22:01:47.925+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.926147+0000) 2022-01-31T22:01:48.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:47 smithi167 conmon[60316]: debug 2022-01-31T22:01:47.925+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.925732+0000) 2022-01-31T22:01:48.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:47 smithi167 conmon[49112]: debug 2022-01-31T22:01:47.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.889942+0000) 2022-01-31T22:01:48.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:47 smithi167 conmon[49112]: debug 2022-01-31T22:01:47.926+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.926574+0000) 2022-01-31T22:01:48.221 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[35325]: debug 2022-01-31T22:01:47.943+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176784 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:48.222 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[41853]: debug 2022-01-31T22:01:47.925+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.926045+0000) 2022-01-31T22:01:48.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[46715]: debug 2022-01-31T22:01:47.924+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.925523+0000) 2022-01-31T22:01:48.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:48 smithi171 conmon[46715]: debug 2022-01-31T22:01:48.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:48.059688+0000) 2022-01-31T22:01:48.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[51620]: debug 2022-01-31T22:01:47.907+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.908394+0000) 2022-01-31T22:01:48.226 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:47 smithi171 conmon[51620]: debug 2022-01-31T22:01:47.926+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:47.926914+0000) 2022-01-31T22:01:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:48 smithi171 conmon[41853]: debug 2022-01-31T22:01:48.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:48.265490+0000) 2022-01-31T22:01:48.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:48 smithi167 conmon[54076]: debug 2022-01-31T22:01:48.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:48.553824+0000) 2022-01-31T22:01:48.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:48 smithi167 conmon[60316]: debug 2022-01-31T22:01:48.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:48.553813+0000) 2022-01-31T22:01:49.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:48 smithi167 conmon[49112]: debug 2022-01-31T22:01:48.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:48.890163+0000) 2022-01-31T22:01:49.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:49 smithi171 conmon[46715]: debug 2022-01-31T22:01:49.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:49.059858+0000) 2022-01-31T22:01:49.221 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:48 smithi171 conmon[51620]: debug 2022-01-31T22:01:48.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:48.908574+0000) 2022-01-31T22:01:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:49 smithi171 conmon[41853]: debug 2022-01-31T22:01:49.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:49.265648+0000) 2022-01-31T22:01:49.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:49 smithi167 conmon[54076]: debug 2022-01-31T22:01:49.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:49.553935+0000) 2022-01-31T22:01:49.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:49 smithi167 conmon[60316]: debug 2022-01-31T22:01:49.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:49.554001+0000) 2022-01-31T22:01:50.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:49 smithi167 conmon[49112]: debug 2022-01-31T22:01:49.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:49.890306+0000) 2022-01-31T22:01:50.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:50 smithi171 conmon[46715]: debug 2022-01-31T22:01:50.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:50.059991+0000) 2022-01-31T22:01:50.221 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:49 smithi171 conmon[51620]: debug 2022-01-31T22:01:49.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:49.908772+0000) 2022-01-31T22:01:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:50 smithi171 conmon[41853]: debug 2022-01-31T22:01:50.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:50.265855+0000) 2022-01-31T22:01:50.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:50 smithi167 conmon[54076]: debug 2022-01-31T22:01:50.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:50.554133+0000) 2022-01-31T22:01:50.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:50 smithi167 conmon[60316]: debug 2022-01-31T22:01:50.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:50.554160+0000) 2022-01-31T22:01:51.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:50 smithi167 conmon[49112]: debug 2022-01-31T22:01:50.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:50.890494+0000) 2022-01-31T22:01:51.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:51 smithi171 conmon[46715]: debug 2022-01-31T22:01:51.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:51.060154+0000) 2022-01-31T22:01:51.222 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:50 smithi171 conmon[51620]: debug 2022-01-31T22:01:50.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:50.908976+0000) 2022-01-31T22:01:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:51 smithi171 conmon[41853]: debug 2022-01-31T22:01:51.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:51.266036+0000) 2022-01-31T22:01:51.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:51 smithi167 conmon[60316]: debug 2022-01-31T22:01:51.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:51.554353+0000) 2022-01-31T22:01:51.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:51 smithi167 conmon[54076]: debug 2022-01-31T22:01:51.553+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:51.554298+0000) 2022-01-31T22:01:52.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:51 smithi167 conmon[49112]: debug 2022-01-31T22:01:51.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:51.890734+0000) 2022-01-31T22:01:52.221 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[46715]: debug 2022-01-31T22:01:52.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.060340+0000) 2022-01-31T22:01:52.222 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:51 smithi171 conmon[51620]: debug 2022-01-31T22:01:51.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:51.909131+0000) 2022-01-31T22:01:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[41853]: debug 2022-01-31T22:01:52.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.266215+0000) 2022-01-31T22:01:52.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:52 smithi167 conmon[54076]: debug 2022-01-31T22:01:52.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.554464+0000) 2022-01-31T22:01:52.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:52 smithi167 conmon[60316]: debug 2022-01-31T22:01:52.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.554479+0000) 2022-01-31T22:01:53.133 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:52 smithi167 conmon[54076]: debug 2022-01-31T22:01:52.946+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.947028+0000) 2022-01-31T22:01:53.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:52 smithi167 conmon[60316]: debug 2022-01-31T22:01:52.947+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.947557+0000) 2022-01-31T22:01:53.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:52 smithi167 conmon[49112]: debug 2022-01-31T22:01:52.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.890917+0000) 2022-01-31T22:01:53.135 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:52 smithi167 conmon[49112]: debug 2022-01-31T22:01:52.947+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.947618+0000) 2022-01-31T22:01:53.221 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[35325]: debug 2022-01-31T22:01:52.964+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 176897 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:53.222 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[41853]: debug 2022-01-31T22:01:52.946+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.946785+0000) 2022-01-31T22:01:53.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[46715]: debug 2022-01-31T22:01:52.945+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.946739+0000) 2022-01-31T22:01:53.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:53 smithi171 conmon[46715]: debug 2022-01-31T22:01:53.059+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:53.060482+0000) 2022-01-31T22:01:53.224 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[51620]: debug 2022-01-31T22:01:52.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.909260+0000) 2022-01-31T22:01:53.224 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:52 smithi171 conmon[51620]: debug 2022-01-31T22:01:52.946+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:52.947261+0000) 2022-01-31T22:01:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:53 smithi171 conmon[41853]: debug 2022-01-31T22:01:53.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:53.266440+0000) 2022-01-31T22:01:53.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:53 smithi167 conmon[54076]: debug 2022-01-31T22:01:53.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:53.554640+0000) 2022-01-31T22:01:53.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:53 smithi167 conmon[60316]: debug 2022-01-31T22:01:53.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:53.554664+0000) 2022-01-31T22:01:54.059 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:53 smithi171 conmon[35325]: debug 2022-01-31T22:01:53.778+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:01:54.060 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:53 smithi171 conmon[51620]: debug 2022-01-31T22:01:53.908+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:53.909438+0000) 2022-01-31T22:01:54.136 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:53 smithi167 conmon[49112]: debug 2022-01-31T22:01:53.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:53.891122+0000) 2022-01-31T22:01:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:54 smithi171 conmon[41853]: debug 2022-01-31T22:01:54.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:54.266580+0000) 2022-01-31T22:01:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:54 smithi171 conmon[46715]: debug 2022-01-31T22:01:54.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:54.060663+0000) 2022-01-31T22:01:54.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:54 smithi167 conmon[54076]: debug 2022-01-31T22:01:54.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:54.554812+0000) 2022-01-31T22:01:54.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:54 smithi167 conmon[60316]: debug 2022-01-31T22:01:54.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:54.554815+0000) 2022-01-31T22:01:55.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:54 smithi167 conmon[49112]: debug 2022-01-31T22:01:54.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:54.891302+0000) 2022-01-31T22:01:55.222 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:55 smithi171 conmon[46715]: debug 2022-01-31T22:01:55.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:55.060827+0000) 2022-01-31T22:01:55.223 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:54 smithi171 conmon[51620]: debug 2022-01-31T22:01:54.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:54.909694+0000) 2022-01-31T22:01:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:55 smithi171 conmon[41853]: debug 2022-01-31T22:01:55.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:55.266783+0000) 2022-01-31T22:01:55.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:55 smithi167 conmon[54076]: debug 2022-01-31T22:01:55.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:55.555016+0000) 2022-01-31T22:01:55.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:55 smithi167 conmon[60316]: debug 2022-01-31T22:01:55.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:55.554995+0000) 2022-01-31T22:01:56.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:55 smithi167 conmon[49112]: debug 2022-01-31T22:01:55.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:55.891497+0000) 2022-01-31T22:01:56.222 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:56 smithi171 conmon[46715]: debug 2022-01-31T22:01:56.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:56.061025+0000) 2022-01-31T22:01:56.223 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:55 smithi171 conmon[51620]: debug 2022-01-31T22:01:55.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:55.909932+0000) 2022-01-31T22:01:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:56 smithi171 conmon[41853]: debug 2022-01-31T22:01:56.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:56.266937+0000) 2022-01-31T22:01:56.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:56 smithi167 conmon[54076]: debug 2022-01-31T22:01:56.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:56.555227+0000) 2022-01-31T22:01:56.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:56 smithi167 conmon[60316]: debug 2022-01-31T22:01:56.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:56.555210+0000) 2022-01-31T22:01:57.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:56 smithi167 conmon[49112]: debug 2022-01-31T22:01:56.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:56.891676+0000) 2022-01-31T22:01:57.222 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[46715]: debug 2022-01-31T22:01:57.060+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.061161+0000) 2022-01-31T22:01:57.223 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:56 smithi171 conmon[51620]: debug 2022-01-31T22:01:56.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:56.910126+0000) 2022-01-31T22:01:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[41853]: debug 2022-01-31T22:01:57.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.267032+0000) 2022-01-31T22:01:57.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:57 smithi167 conmon[54076]: debug 2022-01-31T22:01:57.554+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.555429+0000) 2022-01-31T22:01:57.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:57 smithi167 conmon[60316]: debug 2022-01-31T22:01:57.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.555394+0000) 2022-01-31T22:01:58.145 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:57 smithi167 conmon[54076]: debug 2022-01-31T22:01:57.967+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.968080+0000) 2022-01-31T22:01:58.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:57 smithi167 conmon[60316]: debug 2022-01-31T22:01:57.968+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.968802+0000) 2022-01-31T22:01:58.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:57 smithi167 conmon[49112]: debug 2022-01-31T22:01:57.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.891861+0000) 2022-01-31T22:01:58.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:57 smithi167 conmon[49112]: debug 2022-01-31T22:01:57.968+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.968675+0000) 2022-01-31T22:01:58.222 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[35325]: debug 2022-01-31T22:01:57.985+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177007 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:01:58.223 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[41853]: debug 2022-01-31T22:01:57.966+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.967186+0000) 2022-01-31T22:01:58.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[46715]: debug 2022-01-31T22:01:57.967+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.968013+0000) 2022-01-31T22:01:58.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:58 smithi171 conmon[46715]: debug 2022-01-31T22:01:58.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:58.061352+0000) 2022-01-31T22:01:58.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[51620]: debug 2022-01-31T22:01:57.909+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.910284+0000) 2022-01-31T22:01:58.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:57 smithi171 conmon[51620]: debug 2022-01-31T22:01:57.966+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:57.967382+0000) 2022-01-31T22:01:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:58 smithi171 conmon[41853]: debug 2022-01-31T22:01:58.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:58.267184+0000) 2022-01-31T22:01:58.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:58 smithi167 conmon[54076]: debug 2022-01-31T22:01:58.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:58.555640+0000) 2022-01-31T22:01:58.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:58 smithi167 conmon[60316]: debug 2022-01-31T22:01:58.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:58.555606+0000) 2022-01-31T22:01:59.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:58 smithi167 conmon[49112]: debug 2022-01-31T22:01:58.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:58.892076+0000) 2022-01-31T22:01:59.222 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:01:59 smithi171 conmon[46715]: debug 2022-01-31T22:01:59.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:59.061504+0000) 2022-01-31T22:01:59.223 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:58 smithi171 conmon[51620]: debug 2022-01-31T22:01:58.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:58.910480+0000) 2022-01-31T22:01:59.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:01:59 smithi171 conmon[41853]: debug 2022-01-31T22:01:59.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:59.267336+0000) 2022-01-31T22:01:59.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:01:59 smithi167 conmon[54076]: debug 2022-01-31T22:01:59.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:59.555790+0000) 2022-01-31T22:01:59.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:01:59 smithi167 conmon[60316]: debug 2022-01-31T22:01:59.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:59.555752+0000) 2022-01-31T22:02:00.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:01:59 smithi167 conmon[49112]: debug 2022-01-31T22:01:59.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:59.892246+0000) 2022-01-31T22:02:00.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:00 smithi171 conmon[46715]: debug 2022-01-31T22:02:00.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:00.061680+0000) 2022-01-31T22:02:00.223 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:01:59 smithi171 conmon[51620]: debug 2022-01-31T22:01:59.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:01:59.910673+0000) 2022-01-31T22:02:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:00 smithi171 conmon[41853]: debug 2022-01-31T22:02:00.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:00.267520+0000) 2022-01-31T22:02:00.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:00 smithi167 conmon[54076]: debug 2022-01-31T22:02:00.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:00.555957+0000) 2022-01-31T22:02:00.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:00 smithi167 conmon[60316]: debug 2022-01-31T22:02:00.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:00.555936+0000) 2022-01-31T22:02:00.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:00 smithi171 conmon[51620]: debug 2022-01-31T22:02:00.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:00.910864+0000) 2022-01-31T22:02:01.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:00 smithi167 conmon[49112]: debug 2022-01-31T22:02:00.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:00.892454+0000) 2022-01-31T22:02:01.266 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:01 smithi171 conmon[46715]: debug 2022-01-31T22:02:01.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:01.061835+0000) 2022-01-31T22:02:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:01 smithi171 conmon[41853]: debug 2022-01-31T22:02:01.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:01.267703+0000) 2022-01-31T22:02:01.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:01 smithi167 conmon[60316]: debug 2022-01-31T22:02:01.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:01.556132+0000) 2022-01-31T22:02:01.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:01 smithi167 conmon[54076]: debug 2022-01-31T22:02:01.555+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:01.556151+0000) 2022-01-31T22:02:02.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:01 smithi167 conmon[49112]: debug 2022-01-31T22:02:01.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:01.892695+0000) 2022-01-31T22:02:02.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:02 smithi171 conmon[46715]: debug 2022-01-31T22:02:02.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.062026+0000) 2022-01-31T22:02:02.223 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:01 smithi171 conmon[51620]: debug 2022-01-31T22:02:01.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:01.911025+0000) 2022-01-31T22:02:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:02 smithi171 conmon[41853]: debug 2022-01-31T22:02:02.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.267900+0000) 2022-01-31T22:02:02.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:02 smithi167 conmon[60316]: debug 2022-01-31T22:02:02.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.556349+0000) 2022-01-31T22:02:02.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:02 smithi167 conmon[54076]: debug 2022-01-31T22:02:02.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.556351+0000) 2022-01-31T22:02:03.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:02 smithi167 conmon[49112]: debug 2022-01-31T22:02:02.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.892883+0000) 2022-01-31T22:02:03.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:02 smithi167 conmon[49112]: debug 2022-01-31T22:02:02.988+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.989355+0000) 2022-01-31T22:02:03.161 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:02 smithi167 conmon[54076]: debug 2022-01-31T22:02:02.991+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.991539+0000) 2022-01-31T22:02:03.161 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:02 smithi167 conmon[60316]: debug 2022-01-31T22:02:02.989+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.989624+0000) 2022-01-31T22:02:03.223 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:03 smithi171 conmon[35325]: debug 2022-01-31T22:02:03.007+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177119 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:03.224 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:02 smithi171 conmon[41853]: debug 2022-01-31T22:02:02.990+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.991470+0000) 2022-01-31T22:02:03.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:02 smithi171 conmon[46715]: debug 2022-01-31T22:02:02.990+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.991523+0000) 2022-01-31T22:02:03.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:03 smithi171 conmon[46715]: debug 2022-01-31T22:02:03.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:03.062185+0000) 2022-01-31T22:02:03.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:02 smithi171 conmon[51620]: debug 2022-01-31T22:02:02.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.911192+0000) 2022-01-31T22:02:03.226 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:02 smithi171 conmon[51620]: debug 2022-01-31T22:02:02.988+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:02.989667+0000) 2022-01-31T22:02:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:03 smithi171 conmon[41853]: debug 2022-01-31T22:02:03.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:03.268094+0000) 2022-01-31T22:02:03.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:03 smithi167 conmon[54076]: debug 2022-01-31T22:02:03.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:03.556552+0000) 2022-01-31T22:02:04.037 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:03 smithi167 conmon[60316]: debug 2022-01-31T22:02:03.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:03.556557+0000) 2022-01-31T22:02:04.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:03 smithi167 conmon[49112]: debug 2022-01-31T22:02:03.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:03.893079+0000) 2022-01-31T22:02:04.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:04 smithi171 conmon[46715]: debug 2022-01-31T22:02:04.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:04.062361+0000) 2022-01-31T22:02:04.224 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:03 smithi171 conmon[51620]: debug 2022-01-31T22:02:03.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:03.911291+0000) 2022-01-31T22:02:04.610 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:04 smithi171 conmon[41853]: debug 2022-01-31T22:02:04.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:04.268250+0000) 2022-01-31T22:02:04.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:04 smithi167 conmon[54076]: debug 2022-01-31T22:02:04.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:04.556714+0000) 2022-01-31T22:02:04.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:04 smithi167 conmon[60316]: debug 2022-01-31T22:02:04.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:04.556717+0000) 2022-01-31T22:02:05.167 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:04 smithi167 conmon[49112]: debug 2022-01-31T22:02:04.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:04.893222+0000) 2022-01-31T22:02:05.223 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:05 smithi171 conmon[46715]: debug 2022-01-31T22:02:05.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:05.062577+0000) 2022-01-31T22:02:05.224 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:04 smithi171 conmon[51620]: debug 2022-01-31T22:02:04.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:04.911440+0000) 2022-01-31T22:02:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:05 smithi171 conmon[41853]: debug 2022-01-31T22:02:05.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:05.268383+0000) 2022-01-31T22:02:05.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:05 smithi167 conmon[60316]: debug 2022-01-31T22:02:05.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:05.556918+0000) 2022-01-31T22:02:05.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:05 smithi167 conmon[54076]: debug 2022-01-31T22:02:05.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:05.556918+0000) 2022-01-31T22:02:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:05 smithi167 conmon[49112]: debug 2022-01-31T22:02:05.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:05.893369+0000) 2022-01-31T22:02:06.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:06 smithi171 conmon[46715]: debug 2022-01-31T22:02:06.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:06.062784+0000) 2022-01-31T22:02:06.224 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:05 smithi171 conmon[51620]: debug 2022-01-31T22:02:05.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:05.911646+0000) 2022-01-31T22:02:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:06 smithi171 conmon[41853]: debug 2022-01-31T22:02:06.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:06.268568+0000) 2022-01-31T22:02:06.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:06 smithi167 conmon[54076]: debug 2022-01-31T22:02:06.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:06.557114+0000) 2022-01-31T22:02:06.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:06 smithi167 conmon[60316]: debug 2022-01-31T22:02:06.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:06.557132+0000) 2022-01-31T22:02:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:06 smithi167 conmon[49112]: debug 2022-01-31T22:02:06.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:06.893559+0000) 2022-01-31T22:02:07.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:07 smithi171 conmon[46715]: debug 2022-01-31T22:02:07.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:07.062965+0000) 2022-01-31T22:02:07.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:06 smithi171 conmon[51620]: debug 2022-01-31T22:02:06.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:06.911838+0000) 2022-01-31T22:02:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:07 smithi171 conmon[41853]: debug 2022-01-31T22:02:07.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:07.268792+0000) 2022-01-31T22:02:07.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:07 smithi167 conmon[54076]: debug 2022-01-31T22:02:07.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:07.557336+0000) 2022-01-31T22:02:07.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:07 smithi167 conmon[60316]: debug 2022-01-31T22:02:07.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:07.557355+0000) 2022-01-31T22:02:08.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:08 smithi167 conmon[54076]: debug 2022-01-31T22:02:08.010+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.011280+0000) 2022-01-31T22:02:08.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:08 smithi167 conmon[60316]: debug 2022-01-31T22:02:08.010+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.011288+0000) 2022-01-31T22:02:08.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:07 smithi167 conmon[49112]: debug 2022-01-31T22:02:07.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:07.893719+0000) 2022-01-31T22:02:08.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:08 smithi167 conmon[49112]: debug 2022-01-31T22:02:08.010+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.010550+0000) 2022-01-31T22:02:08.224 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[35325]: debug 2022-01-31T22:02:08.028+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177229 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:08.225 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[41853]: debug 2022-01-31T22:02:08.010+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.012194+0000) 2022-01-31T22:02:08.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:07 smithi171 conmon[51620]: debug 2022-01-31T22:02:07.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:07.911984+0000) 2022-01-31T22:02:08.226 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[51620]: debug 2022-01-31T22:02:08.009+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.011582+0000) 2022-01-31T22:02:08.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[46715]: debug 2022-01-31T22:02:08.009+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.011666+0000) 2022-01-31T22:02:08.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[46715]: debug 2022-01-31T22:02:08.061+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.063146+0000) 2022-01-31T22:02:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[41853]: debug 2022-01-31T22:02:08.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.268979+0000) 2022-01-31T22:02:08.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:08 smithi167 conmon[54076]: debug 2022-01-31T22:02:08.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.557534+0000) 2022-01-31T22:02:08.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:08 smithi167 conmon[60316]: debug 2022-01-31T22:02:08.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.557543+0000) 2022-01-31T22:02:09.062 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[35325]: debug 2022-01-31T22:02:08.779+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:02:09.063 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:08 smithi171 conmon[51620]: debug 2022-01-31T22:02:08.910+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.912180+0000) 2022-01-31T22:02:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:08 smithi167 conmon[49112]: debug 2022-01-31T22:02:08.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:08.893950+0000) 2022-01-31T22:02:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:09 smithi171 conmon[46715]: debug 2022-01-31T22:02:09.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:09.063259+0000) 2022-01-31T22:02:09.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:09 smithi171 conmon[41853]: debug 2022-01-31T22:02:09.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:09.269139+0000) 2022-01-31T22:02:09.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:09 smithi167 conmon[54076]: debug 2022-01-31T22:02:09.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:09.557635+0000) 2022-01-31T22:02:09.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:09 smithi167 conmon[60316]: debug 2022-01-31T22:02:09.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:09.557657+0000) 2022-01-31T22:02:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:09 smithi167 conmon[49112]: debug 2022-01-31T22:02:09.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:09.894069+0000) 2022-01-31T22:02:10.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:10 smithi171 conmon[46715]: debug 2022-01-31T22:02:10.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:10.063383+0000) 2022-01-31T22:02:10.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:09 smithi171 conmon[51620]: debug 2022-01-31T22:02:09.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:09.912317+0000) 2022-01-31T22:02:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:10 smithi171 conmon[41853]: debug 2022-01-31T22:02:10.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:10.269327+0000) 2022-01-31T22:02:10.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:10 smithi167 conmon[60316]: debug 2022-01-31T22:02:10.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:10.557831+0000) 2022-01-31T22:02:10.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:10 smithi167 conmon[54076]: debug 2022-01-31T22:02:10.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:10.557835+0000) 2022-01-31T22:02:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:10 smithi167 conmon[49112]: debug 2022-01-31T22:02:10.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:10.894219+0000) 2022-01-31T22:02:11.224 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:11 smithi171 conmon[46715]: debug 2022-01-31T22:02:11.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:11.063605+0000) 2022-01-31T22:02:11.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:10 smithi171 conmon[51620]: debug 2022-01-31T22:02:10.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:10.912472+0000) 2022-01-31T22:02:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:11 smithi171 conmon[41853]: debug 2022-01-31T22:02:11.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:11.269468+0000) 2022-01-31T22:02:11.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:11 smithi167 conmon[60316]: debug 2022-01-31T22:02:11.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:11.558048+0000) 2022-01-31T22:02:11.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:11 smithi167 conmon[54076]: debug 2022-01-31T22:02:11.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:11.558047+0000) 2022-01-31T22:02:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:11 smithi167 conmon[49112]: debug 2022-01-31T22:02:11.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:11.894401+0000) 2022-01-31T22:02:12.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:12 smithi171 conmon[46715]: debug 2022-01-31T22:02:12.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:12.063827+0000) 2022-01-31T22:02:12.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:11 smithi171 conmon[51620]: debug 2022-01-31T22:02:11.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:11.912662+0000) 2022-01-31T22:02:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:12 smithi171 conmon[41853]: debug 2022-01-31T22:02:12.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:12.269676+0000) 2022-01-31T22:02:12.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:12 smithi167 conmon[60316]: debug 2022-01-31T22:02:12.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:12.558256+0000) 2022-01-31T22:02:12.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:12 smithi167 conmon[54076]: debug 2022-01-31T22:02:12.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:12.558255+0000) 2022-01-31T22:02:13.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:13 smithi167 conmon[54076]: debug 2022-01-31T22:02:13.031+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.032844+0000) 2022-01-31T22:02:13.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:13 smithi167 conmon[60316]: debug 2022-01-31T22:02:13.031+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.033469+0000) 2022-01-31T22:02:13.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:12 smithi167 conmon[49112]: debug 2022-01-31T22:02:12.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:12.894592+0000) 2022-01-31T22:02:13.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:13 smithi167 conmon[49112]: debug 2022-01-31T22:02:13.031+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.032543+0000) 2022-01-31T22:02:13.225 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[35325]: debug 2022-01-31T22:02:13.048+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177341 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:13.225 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[41853]: debug 2022-01-31T22:02:13.031+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.033680+0000) 2022-01-31T22:02:13.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[46715]: debug 2022-01-31T22:02:13.031+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.033429+0000) 2022-01-31T22:02:13.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[46715]: debug 2022-01-31T22:02:13.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.063888+0000) 2022-01-31T22:02:13.227 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:12 smithi171 conmon[51620]: debug 2022-01-31T22:02:12.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:12.912855+0000) 2022-01-31T22:02:13.227 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[51620]: debug 2022-01-31T22:02:13.031+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.033141+0000) 2022-01-31T22:02:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[41853]: debug 2022-01-31T22:02:13.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.269801+0000) 2022-01-31T22:02:13.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:13 smithi167 conmon[60316]: debug 2022-01-31T22:02:13.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.558460+0000) 2022-01-31T22:02:13.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:13 smithi167 conmon[54076]: debug 2022-01-31T22:02:13.556+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.558459+0000) 2022-01-31T22:02:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:13 smithi167 conmon[49112]: debug 2022-01-31T22:02:13.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.894755+0000) 2022-01-31T22:02:14.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:14 smithi171 conmon[46715]: debug 2022-01-31T22:02:14.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:14.064028+0000) 2022-01-31T22:02:14.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:13 smithi171 conmon[51620]: debug 2022-01-31T22:02:13.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:13.913067+0000) 2022-01-31T22:02:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:14 smithi171 conmon[41853]: debug 2022-01-31T22:02:14.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:14.269959+0000) 2022-01-31T22:02:14.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:14 smithi167 conmon[54076]: debug 2022-01-31T22:02:14.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:14.558627+0000) 2022-01-31T22:02:14.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:14 smithi167 conmon[60316]: debug 2022-01-31T22:02:14.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:14.558634+0000) 2022-01-31T22:02:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:14 smithi167 conmon[49112]: debug 2022-01-31T22:02:14.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:14.894913+0000) 2022-01-31T22:02:15.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:15 smithi171 conmon[46715]: debug 2022-01-31T22:02:15.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:15.064150+0000) 2022-01-31T22:02:15.226 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:14 smithi171 conmon[51620]: debug 2022-01-31T22:02:14.911+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:14.913215+0000) 2022-01-31T22:02:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:15 smithi171 conmon[41853]: debug 2022-01-31T22:02:15.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:15.270146+0000) 2022-01-31T22:02:15.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:15 smithi167 conmon[60316]: debug 2022-01-31T22:02:15.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:15.558830+0000) 2022-01-31T22:02:15.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:15 smithi167 conmon[54076]: debug 2022-01-31T22:02:15.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:15.558825+0000) 2022-01-31T22:02:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:15 smithi167 conmon[49112]: debug 2022-01-31T22:02:15.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:15.895077+0000) 2022-01-31T22:02:16.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:16 smithi171 conmon[46715]: debug 2022-01-31T22:02:16.062+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:16.064293+0000) 2022-01-31T22:02:16.226 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:15 smithi171 conmon[51620]: debug 2022-01-31T22:02:15.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:15.913398+0000) 2022-01-31T22:02:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:16 smithi171 conmon[41853]: debug 2022-01-31T22:02:16.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:16.270322+0000) 2022-01-31T22:02:16.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:16 smithi167 conmon[54076]: debug 2022-01-31T22:02:16.557+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:16.559002+0000) 2022-01-31T22:02:16.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:16 smithi167 conmon[60316]: debug 2022-01-31T22:02:16.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:16.558962+0000) 2022-01-31T22:02:17.063 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:16 smithi171 conmon[51620]: debug 2022-01-31T22:02:16.912+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:16.913589+0000) 2022-01-31T22:02:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:16 smithi167 conmon[49112]: debug 2022-01-31T22:02:16.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:16.895216+0000) 2022-01-31T22:02:17.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:17 smithi171 conmon[41853]: debug 2022-01-31T22:02:17.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:17.270504+0000) 2022-01-31T22:02:17.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:17 smithi171 conmon[46715]: debug 2022-01-31T22:02:17.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:17.064439+0000) 2022-01-31T22:02:17.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:17 smithi167 conmon[54076]: debug 2022-01-31T22:02:17.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:17.559155+0000) 2022-01-31T22:02:17.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:17 smithi167 conmon[60316]: debug 2022-01-31T22:02:17.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:17.559154+0000) 2022-01-31T22:02:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:17 smithi167 conmon[49112]: debug 2022-01-31T22:02:17.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:17.895378+0000) 2022-01-31T22:02:18.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:18 smithi167 conmon[49112]: debug 2022-01-31T22:02:18.052+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.053637+0000) 2022-01-31T22:02:18.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:18 smithi167 conmon[54076]: debug 2022-01-31T22:02:18.051+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.053058+0000) 2022-01-31T22:02:18.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:18 smithi167 conmon[60316]: debug 2022-01-31T22:02:18.051+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.053411+0000) 2022-01-31T22:02:18.226 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[35325]: debug 2022-01-31T22:02:18.071+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177452 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:18.227 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[41853]: debug 2022-01-31T22:02:18.052+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.053345+0000) 2022-01-31T22:02:18.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[46715]: debug 2022-01-31T22:02:18.052+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.053105+0000) 2022-01-31T22:02:18.228 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[46715]: debug 2022-01-31T22:02:18.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.064587+0000) 2022-01-31T22:02:18.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:17 smithi171 conmon[51620]: debug 2022-01-31T22:02:17.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:17.913799+0000) 2022-01-31T22:02:18.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[51620]: debug 2022-01-31T22:02:18.052+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.053441+0000) 2022-01-31T22:02:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[41853]: debug 2022-01-31T22:02:18.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.270681+0000) 2022-01-31T22:02:18.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:18 smithi167 conmon[54076]: debug 2022-01-31T22:02:18.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.559335+0000) 2022-01-31T22:02:18.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:18 smithi167 conmon[60316]: debug 2022-01-31T22:02:18.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.559335+0000) 2022-01-31T22:02:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:18 smithi167 conmon[49112]: debug 2022-01-31T22:02:18.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.895597+0000) 2022-01-31T22:02:19.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:19 smithi171 conmon[46715]: debug 2022-01-31T22:02:19.063+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:19.064679+0000) 2022-01-31T22:02:19.227 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:18 smithi171 conmon[51620]: debug 2022-01-31T22:02:18.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:18.913985+0000) 2022-01-31T22:02:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:19 smithi171 conmon[41853]: debug 2022-01-31T22:02:19.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:19.270841+0000) 2022-01-31T22:02:19.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:19 smithi167 conmon[54076]: debug 2022-01-31T22:02:19.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:19.559534+0000) 2022-01-31T22:02:19.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:19 smithi167 conmon[60316]: debug 2022-01-31T22:02:19.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:19.559526+0000) 2022-01-31T22:02:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:19 smithi167 conmon[49112]: debug 2022-01-31T22:02:19.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:19.895771+0000) 2022-01-31T22:02:20.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:20 smithi171 conmon[46715]: debug 2022-01-31T22:02:20.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:20.064794+0000) 2022-01-31T22:02:20.227 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:19 smithi171 conmon[51620]: debug 2022-01-31T22:02:19.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:19.914117+0000) 2022-01-31T22:02:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:20 smithi171 conmon[41853]: debug 2022-01-31T22:02:20.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:20.271000+0000) 2022-01-31T22:02:20.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:20 smithi167 conmon[54076]: debug 2022-01-31T22:02:20.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:20.559744+0000) 2022-01-31T22:02:20.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:20 smithi167 conmon[60316]: debug 2022-01-31T22:02:20.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:20.559745+0000) 2022-01-31T22:02:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:20 smithi167 conmon[49112]: debug 2022-01-31T22:02:20.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:20.895993+0000) 2022-01-31T22:02:21.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:21 smithi171 conmon[46715]: debug 2022-01-31T22:02:21.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:21.064928+0000) 2022-01-31T22:02:21.227 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:20 smithi171 conmon[51620]: debug 2022-01-31T22:02:20.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:20.914304+0000) 2022-01-31T22:02:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:21 smithi171 conmon[41853]: debug 2022-01-31T22:02:21.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:21.271176+0000) 2022-01-31T22:02:21.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:21 smithi167 conmon[54076]: debug 2022-01-31T22:02:21.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:21.559940+0000) 2022-01-31T22:02:21.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:21 smithi167 conmon[60316]: debug 2022-01-31T22:02:21.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:21.559952+0000) 2022-01-31T22:02:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:21 smithi167 conmon[49112]: debug 2022-01-31T22:02:21.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:21.896171+0000) 2022-01-31T22:02:22.226 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:22 smithi171 conmon[46715]: debug 2022-01-31T22:02:22.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:22.065040+0000) 2022-01-31T22:02:22.227 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:21 smithi171 conmon[51620]: debug 2022-01-31T22:02:21.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:21.914506+0000) 2022-01-31T22:02:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:22 smithi171 conmon[41853]: debug 2022-01-31T22:02:22.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:22.271359+0000) 2022-01-31T22:02:22.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:22 smithi167 conmon[54076]: debug 2022-01-31T22:02:22.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:22.560153+0000) 2022-01-31T22:02:22.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:22 smithi167 conmon[60316]: debug 2022-01-31T22:02:22.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:22.560156+0000) 2022-01-31T22:02:23.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:23 smithi167 conmon[54076]: debug 2022-01-31T22:02:23.072+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.074105+0000) 2022-01-31T22:02:23.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:23 smithi167 conmon[60316]: debug 2022-01-31T22:02:23.074+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.076127+0000) 2022-01-31T22:02:23.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:22 smithi167 conmon[49112]: debug 2022-01-31T22:02:22.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:22.896382+0000) 2022-01-31T22:02:23.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:23 smithi167 conmon[49112]: debug 2022-01-31T22:02:23.073+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.075252+0000) 2022-01-31T22:02:23.227 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[35325]: debug 2022-01-31T22:02:23.092+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177566 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:23.228 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[41853]: debug 2022-01-31T22:02:23.073+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.074675+0000) 2022-01-31T22:02:23.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:22 smithi171 conmon[51620]: debug 2022-01-31T22:02:22.913+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:22.914695+0000) 2022-01-31T22:02:23.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[51620]: debug 2022-01-31T22:02:23.074+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.075494+0000) 2022-01-31T22:02:23.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[46715]: debug 2022-01-31T22:02:23.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.065182+0000) 2022-01-31T22:02:23.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[46715]: debug 2022-01-31T22:02:23.074+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.075167+0000) 2022-01-31T22:02:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[41853]: debug 2022-01-31T22:02:23.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.271556+0000) 2022-01-31T22:02:23.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:23 smithi167 conmon[54076]: debug 2022-01-31T22:02:23.558+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.560367+0000) 2022-01-31T22:02:23.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:23 smithi167 conmon[60316]: debug 2022-01-31T22:02:23.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.560373+0000) 2022-01-31T22:02:24.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[51620]: debug 2022-01-31T22:02:23.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.914848+0000) 2022-01-31T22:02:24.065 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:23 smithi171 conmon[35325]: debug 2022-01-31T22:02:23.781+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:02:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:23 smithi167 conmon[49112]: debug 2022-01-31T22:02:23.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:23.896574+0000) 2022-01-31T22:02:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:24 smithi171 conmon[41853]: debug 2022-01-31T22:02:24.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:24.271726+0000) 2022-01-31T22:02:24.452 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:24 smithi171 conmon[46715]: debug 2022-01-31T22:02:24.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:24.065333+0000) 2022-01-31T22:02:24.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:24 smithi167 conmon[54076]: debug 2022-01-31T22:02:24.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:24.560541+0000) 2022-01-31T22:02:24.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:24 smithi167 conmon[60316]: debug 2022-01-31T22:02:24.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:24.560540+0000) 2022-01-31T22:02:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:24 smithi167 conmon[49112]: debug 2022-01-31T22:02:24.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:24.896732+0000) 2022-01-31T22:02:25.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:25 smithi171 conmon[46715]: debug 2022-01-31T22:02:25.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:25.065506+0000) 2022-01-31T22:02:25.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:24 smithi171 conmon[51620]: debug 2022-01-31T22:02:24.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:24.914974+0000) 2022-01-31T22:02:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:25 smithi171 conmon[41853]: debug 2022-01-31T22:02:25.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:25.271921+0000) 2022-01-31T22:02:25.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:25 smithi167 conmon[54076]: debug 2022-01-31T22:02:25.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:25.560722+0000) 2022-01-31T22:02:25.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:25 smithi167 conmon[60316]: debug 2022-01-31T22:02:25.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:25.560720+0000) 2022-01-31T22:02:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:25 smithi167 conmon[49112]: debug 2022-01-31T22:02:25.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:25.896908+0000) 2022-01-31T22:02:26.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:26 smithi171 conmon[46715]: debug 2022-01-31T22:02:26.064+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:26.065713+0000) 2022-01-31T22:02:26.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:25 smithi171 conmon[51620]: debug 2022-01-31T22:02:25.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:25.915154+0000) 2022-01-31T22:02:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:26 smithi171 conmon[41853]: debug 2022-01-31T22:02:26.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:26.272136+0000) 2022-01-31T22:02:26.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:26 smithi167 conmon[54076]: debug 2022-01-31T22:02:26.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:26.560909+0000) 2022-01-31T22:02:26.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:26 smithi167 conmon[60316]: debug 2022-01-31T22:02:26.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:26.560908+0000) 2022-01-31T22:02:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:26 smithi167 conmon[49112]: debug 2022-01-31T22:02:26.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:26.897061+0000) 2022-01-31T22:02:27.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:27 smithi171 conmon[46715]: debug 2022-01-31T22:02:27.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:27.065922+0000) 2022-01-31T22:02:27.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:26 smithi171 conmon[51620]: debug 2022-01-31T22:02:26.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:26.915317+0000) 2022-01-31T22:02:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:27 smithi171 conmon[41853]: debug 2022-01-31T22:02:27.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:27.272323+0000) 2022-01-31T22:02:27.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:27 smithi167 conmon[54076]: debug 2022-01-31T22:02:27.559+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:27.561112+0000) 2022-01-31T22:02:27.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:27 smithi167 conmon[60316]: debug 2022-01-31T22:02:27.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:27.561116+0000) 2022-01-31T22:02:28.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:28 smithi167 conmon[54076]: debug 2022-01-31T22:02:28.094+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.095978+0000) 2022-01-31T22:02:28.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:28 smithi167 conmon[60316]: debug 2022-01-31T22:02:28.095+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.096761+0000) 2022-01-31T22:02:28.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:27 smithi167 conmon[49112]: debug 2022-01-31T22:02:27.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:27.897237+0000) 2022-01-31T22:02:28.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:28 smithi167 conmon[49112]: debug 2022-01-31T22:02:28.093+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.095048+0000) 2022-01-31T22:02:28.227 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[35325]: debug 2022-01-31T22:02:28.113+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177677 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:28.228 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[41853]: debug 2022-01-31T22:02:28.095+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.095917+0000) 2022-01-31T22:02:28.228 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[46715]: debug 2022-01-31T22:02:28.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.066068+0000) 2022-01-31T22:02:28.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[46715]: debug 2022-01-31T22:02:28.096+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.096783+0000) 2022-01-31T22:02:28.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:27 smithi171 conmon[51620]: debug 2022-01-31T22:02:27.914+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:27.915491+0000) 2022-01-31T22:02:28.230 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[51620]: debug 2022-01-31T22:02:28.095+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.096148+0000) 2022-01-31T22:02:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[41853]: debug 2022-01-31T22:02:28.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.272500+0000) 2022-01-31T22:02:28.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:28 smithi167 conmon[54076]: debug 2022-01-31T22:02:28.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.561304+0000) 2022-01-31T22:02:28.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:28 smithi167 conmon[60316]: debug 2022-01-31T22:02:28.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.561304+0000) 2022-01-31T22:02:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:28 smithi167 conmon[49112]: debug 2022-01-31T22:02:28.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.897439+0000) 2022-01-31T22:02:29.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:29 smithi171 conmon[46715]: debug 2022-01-31T22:02:29.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:29.066281+0000) 2022-01-31T22:02:29.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:28 smithi171 conmon[51620]: debug 2022-01-31T22:02:28.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:28.915653+0000) 2022-01-31T22:02:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:29 smithi171 conmon[41853]: debug 2022-01-31T22:02:29.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:29.272675+0000) 2022-01-31T22:02:29.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:29 smithi167 conmon[54076]: debug 2022-01-31T22:02:29.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:29.561482+0000) 2022-01-31T22:02:29.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:29 smithi167 conmon[60316]: debug 2022-01-31T22:02:29.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:29.561482+0000) 2022-01-31T22:02:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:29 smithi167 conmon[49112]: debug 2022-01-31T22:02:29.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:29.897578+0000) 2022-01-31T22:02:30.228 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:30 smithi171 conmon[46715]: debug 2022-01-31T22:02:30.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:30.066435+0000) 2022-01-31T22:02:30.228 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:29 smithi171 conmon[51620]: debug 2022-01-31T22:02:29.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:29.915797+0000) 2022-01-31T22:02:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:30 smithi171 conmon[41853]: debug 2022-01-31T22:02:30.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:30.272875+0000) 2022-01-31T22:02:30.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:30 smithi167 conmon[54076]: debug 2022-01-31T22:02:30.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:30.561731+0000) 2022-01-31T22:02:30.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:30 smithi167 conmon[60316]: debug 2022-01-31T22:02:30.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:30.561732+0000) 2022-01-31T22:02:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:30 smithi167 conmon[49112]: debug 2022-01-31T22:02:30.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:30.897779+0000) 2022-01-31T22:02:31.228 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:31 smithi171 conmon[46715]: debug 2022-01-31T22:02:31.065+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:31.066558+0000) 2022-01-31T22:02:31.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:30 smithi171 conmon[51620]: debug 2022-01-31T22:02:30.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:30.915946+0000) 2022-01-31T22:02:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:31 smithi171 conmon[41853]: debug 2022-01-31T22:02:31.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:31.273013+0000) 2022-01-31T22:02:31.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:31 smithi167 conmon[54076]: debug 2022-01-31T22:02:31.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:31.561921+0000) 2022-01-31T22:02:31.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:31 smithi167 conmon[60316]: debug 2022-01-31T22:02:31.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:31.561895+0000) 2022-01-31T22:02:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:31 smithi167 conmon[49112]: debug 2022-01-31T22:02:31.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:31.897949+0000) 2022-01-31T22:02:32.228 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:32 smithi171 conmon[46715]: debug 2022-01-31T22:02:32.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:32.066768+0000) 2022-01-31T22:02:32.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:31 smithi171 conmon[51620]: debug 2022-01-31T22:02:31.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:31.916125+0000) 2022-01-31T22:02:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:32 smithi171 conmon[41853]: debug 2022-01-31T22:02:32.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:32.273249+0000) 2022-01-31T22:02:32.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:32 smithi167 conmon[54076]: debug 2022-01-31T22:02:32.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:32.562181+0000) 2022-01-31T22:02:32.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:32 smithi167 conmon[60316]: debug 2022-01-31T22:02:32.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:32.562134+0000) 2022-01-31T22:02:33.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:32 smithi167 conmon[49112]: debug 2022-01-31T22:02:32.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:32.898256+0000) 2022-01-31T22:02:33.228 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[35325]: debug 2022-01-31T22:02:33.140+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177788 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:33.229 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[41853]: debug 2022-01-31T22:02:33.115+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.117522+0000) 2022-01-31T22:02:33.230 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[46715]: debug 2022-01-31T22:02:33.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.066933+0000) 2022-01-31T22:02:33.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[46715]: debug 2022-01-31T22:02:33.115+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.118152+0000) 2022-01-31T22:02:33.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:32 smithi171 conmon[51620]: debug 2022-01-31T22:02:32.915+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:32.916410+0000) 2022-01-31T22:02:33.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[51620]: debug 2022-01-31T22:02:33.115+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.117998+0000) 2022-01-31T22:02:33.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:33 smithi167 conmon[49112]: debug 2022-01-31T22:02:33.114+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.116351+0000) 2022-01-31T22:02:33.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:33 smithi167 conmon[54076]: debug 2022-01-31T22:02:33.116+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.118129+0000) 2022-01-31T22:02:33.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:33 smithi167 conmon[60316]: debug 2022-01-31T22:02:33.116+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.117645+0000) 2022-01-31T22:02:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[41853]: debug 2022-01-31T22:02:33.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.273465+0000) 2022-01-31T22:02:33.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:33 smithi167 conmon[54076]: debug 2022-01-31T22:02:33.560+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.562336+0000) 2022-01-31T22:02:33.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:33 smithi167 conmon[60316]: debug 2022-01-31T22:02:33.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.562336+0000) 2022-01-31T22:02:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:33 smithi167 conmon[49112]: debug 2022-01-31T22:02:33.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.898407+0000) 2022-01-31T22:02:34.228 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:34 smithi171 conmon[46715]: debug 2022-01-31T22:02:34.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:34.067073+0000) 2022-01-31T22:02:34.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:33 smithi171 conmon[51620]: debug 2022-01-31T22:02:33.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:33.916592+0000) 2022-01-31T22:02:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:34 smithi171 conmon[41853]: debug 2022-01-31T22:02:34.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:34.273595+0000) 2022-01-31T22:02:34.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:34 smithi167 conmon[54076]: debug 2022-01-31T22:02:34.561+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:34.562527+0000) 2022-01-31T22:02:34.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:34 smithi167 conmon[60316]: debug 2022-01-31T22:02:34.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:34.562531+0000) 2022-01-31T22:02:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:34 smithi167 conmon[49112]: debug 2022-01-31T22:02:34.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:34.898551+0000) 2022-01-31T22:02:35.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:35 smithi171 conmon[46715]: debug 2022-01-31T22:02:35.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:35.067275+0000) 2022-01-31T22:02:35.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:34 smithi171 conmon[51620]: debug 2022-01-31T22:02:34.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:34.916740+0000) 2022-01-31T22:02:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:35 smithi171 conmon[41853]: debug 2022-01-31T22:02:35.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:35.273747+0000) 2022-01-31T22:02:35.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:35 smithi167 conmon[60316]: debug 2022-01-31T22:02:35.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:35.562723+0000) 2022-01-31T22:02:35.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:35 smithi167 conmon[54076]: debug 2022-01-31T22:02:35.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:35.562718+0000) 2022-01-31T22:02:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:35 smithi167 conmon[49112]: debug 2022-01-31T22:02:35.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:35.898759+0000) 2022-01-31T22:02:36.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:36 smithi171 conmon[46715]: debug 2022-01-31T22:02:36.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:36.067463+0000) 2022-01-31T22:02:36.230 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:35 smithi171 conmon[51620]: debug 2022-01-31T22:02:35.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:35.916969+0000) 2022-01-31T22:02:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:36 smithi171 conmon[41853]: debug 2022-01-31T22:02:36.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:36.273913+0000) 2022-01-31T22:02:36.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:36 smithi167 conmon[60316]: debug 2022-01-31T22:02:36.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:36.562918+0000) 2022-01-31T22:02:36.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:36 smithi167 conmon[54076]: debug 2022-01-31T22:02:36.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:36.562910+0000) 2022-01-31T22:02:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:36 smithi167 conmon[49112]: debug 2022-01-31T22:02:36.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:36.898951+0000) 2022-01-31T22:02:37.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:37 smithi171 conmon[46715]: debug 2022-01-31T22:02:37.066+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:37.067641+0000) 2022-01-31T22:02:37.230 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:36 smithi171 conmon[51620]: debug 2022-01-31T22:02:36.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:36.917151+0000) 2022-01-31T22:02:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:37 smithi171 conmon[41853]: debug 2022-01-31T22:02:37.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:37.274040+0000) 2022-01-31T22:02:37.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:37 smithi167 conmon[60316]: debug 2022-01-31T22:02:37.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:37.563132+0000) 2022-01-31T22:02:37.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:37 smithi167 conmon[54076]: debug 2022-01-31T22:02:37.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:37.563102+0000) 2022-01-31T22:02:38.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:37 smithi167 conmon[49112]: debug 2022-01-31T22:02:37.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:37.899138+0000) 2022-01-31T22:02:38.230 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[35325]: debug 2022-01-31T22:02:38.162+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 177900 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:38.230 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[41853]: debug 2022-01-31T22:02:38.142+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.143497+0000) 2022-01-31T22:02:38.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[46715]: debug 2022-01-31T22:02:38.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.067795+0000) 2022-01-31T22:02:38.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[46715]: debug 2022-01-31T22:02:38.143+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.144522+0000) 2022-01-31T22:02:38.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:37 smithi171 conmon[51620]: debug 2022-01-31T22:02:37.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:37.917315+0000) 2022-01-31T22:02:38.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[51620]: debug 2022-01-31T22:02:38.143+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.143770+0000) 2022-01-31T22:02:38.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:38 smithi167 conmon[49112]: debug 2022-01-31T22:02:38.143+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.144309+0000) 2022-01-31T22:02:38.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:38 smithi167 conmon[54076]: debug 2022-01-31T22:02:38.144+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.144869+0000) 2022-01-31T22:02:38.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:38 smithi167 conmon[60316]: debug 2022-01-31T22:02:38.145+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.145549+0000) 2022-01-31T22:02:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[41853]: debug 2022-01-31T22:02:38.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.274234+0000) 2022-01-31T22:02:38.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:38 smithi167 conmon[60316]: debug 2022-01-31T22:02:38.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.563313+0000) 2022-01-31T22:02:38.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:38 smithi167 conmon[54076]: debug 2022-01-31T22:02:38.562+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.563302+0000) 2022-01-31T22:02:39.067 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[51620]: debug 2022-01-31T22:02:38.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.917469+0000) 2022-01-31T22:02:39.067 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:38 smithi171 conmon[35325]: debug 2022-01-31T22:02:38.782+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:02:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:38 smithi167 conmon[49112]: debug 2022-01-31T22:02:38.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:38.899281+0000) 2022-01-31T22:02:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:39 smithi171 conmon[41853]: debug 2022-01-31T22:02:39.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:39.274383+0000) 2022-01-31T22:02:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:39 smithi171 conmon[46715]: debug 2022-01-31T22:02:39.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:39.067985+0000) 2022-01-31T22:02:39.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:39 smithi167 conmon[54076]: debug 2022-01-31T22:02:39.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:39.563474+0000) 2022-01-31T22:02:39.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:39 smithi167 conmon[60316]: debug 2022-01-31T22:02:39.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:39.563475+0000) 2022-01-31T22:02:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:39 smithi167 conmon[49112]: debug 2022-01-31T22:02:39.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:39.899444+0000) 2022-01-31T22:02:40.229 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:40 smithi171 conmon[46715]: debug 2022-01-31T22:02:40.067+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:40.068165+0000) 2022-01-31T22:02:40.230 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:39 smithi171 conmon[51620]: debug 2022-01-31T22:02:39.916+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:39.917639+0000) 2022-01-31T22:02:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:40 smithi171 conmon[41853]: debug 2022-01-31T22:02:40.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:40.274571+0000) 2022-01-31T22:02:40.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:40 smithi167 conmon[54076]: debug 2022-01-31T22:02:40.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:40.563711+0000) 2022-01-31T22:02:40.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:40 smithi167 conmon[60316]: debug 2022-01-31T22:02:40.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:40.563706+0000) 2022-01-31T22:02:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:40 smithi167 conmon[49112]: debug 2022-01-31T22:02:40.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:40.899661+0000) 2022-01-31T22:02:41.230 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:41 smithi171 conmon[46715]: debug 2022-01-31T22:02:41.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:41.068372+0000) 2022-01-31T22:02:41.230 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:40 smithi171 conmon[51620]: debug 2022-01-31T22:02:40.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:40.917824+0000) 2022-01-31T22:02:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:41 smithi171 conmon[41853]: debug 2022-01-31T22:02:41.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:41.274774+0000) 2022-01-31T22:02:41.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:41 smithi167 conmon[54076]: debug 2022-01-31T22:02:41.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:41.563935+0000) 2022-01-31T22:02:41.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:41 smithi167 conmon[60316]: debug 2022-01-31T22:02:41.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:41.563858+0000) 2022-01-31T22:02:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:41 smithi167 conmon[49112]: debug 2022-01-31T22:02:41.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:41.899864+0000) 2022-01-31T22:02:42.230 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:42 smithi171 conmon[46715]: debug 2022-01-31T22:02:42.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:42.068583+0000) 2022-01-31T22:02:42.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:41 smithi171 conmon[51620]: debug 2022-01-31T22:02:41.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:41.917998+0000) 2022-01-31T22:02:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:42 smithi171 conmon[41853]: debug 2022-01-31T22:02:42.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:42.274954+0000) 2022-01-31T22:02:42.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:42 smithi167 conmon[54076]: debug 2022-01-31T22:02:42.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:42.564139+0000) 2022-01-31T22:02:42.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:42 smithi167 conmon[60316]: debug 2022-01-31T22:02:42.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:42.564048+0000) 2022-01-31T22:02:43.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:42 smithi167 conmon[49112]: debug 2022-01-31T22:02:42.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:42.900070+0000) 2022-01-31T22:02:43.180 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[41853]: debug 2022-01-31T22:02:43.164+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.165023+0000) 2022-01-31T22:02:43.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[46715]: debug 2022-01-31T22:02:43.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.068791+0000) 2022-01-31T22:02:43.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[46715]: debug 2022-01-31T22:02:43.165+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.165965+0000) 2022-01-31T22:02:43.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:42 smithi171 conmon[51620]: debug 2022-01-31T22:02:42.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:42.918172+0000) 2022-01-31T22:02:43.182 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[51620]: debug 2022-01-31T22:02:43.164+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.165235+0000) 2022-01-31T22:02:43.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:43 smithi167 conmon[49112]: debug 2022-01-31T22:02:43.165+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.166106+0000) 2022-01-31T22:02:43.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:43 smithi167 conmon[54076]: debug 2022-01-31T22:02:43.165+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.165543+0000) 2022-01-31T22:02:43.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:43 smithi167 conmon[60316]: debug 2022-01-31T22:02:43.165+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.166019+0000) 2022-01-31T22:02:43.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[35325]: debug 2022-01-31T22:02:43.180+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178011 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:43.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[41853]: debug 2022-01-31T22:02:43.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.275121+0000) 2022-01-31T22:02:43.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:43 smithi167 conmon[54076]: debug 2022-01-31T22:02:43.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.564328+0000) 2022-01-31T22:02:43.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:43 smithi167 conmon[60316]: debug 2022-01-31T22:02:43.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.564243+0000) 2022-01-31T22:02:44.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:43 smithi167 conmon[49112]: debug 2022-01-31T22:02:43.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.900319+0000) 2022-01-31T22:02:44.230 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:44 smithi171 conmon[46715]: debug 2022-01-31T22:02:44.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:44.068961+0000) 2022-01-31T22:02:44.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:43 smithi171 conmon[51620]: debug 2022-01-31T22:02:43.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:43.918357+0000) 2022-01-31T22:02:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:44 smithi171 conmon[41853]: debug 2022-01-31T22:02:44.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:44.275274+0000) 2022-01-31T22:02:44.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:44 smithi167 conmon[60316]: debug 2022-01-31T22:02:44.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:44.564345+0000) 2022-01-31T22:02:44.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:44 smithi167 conmon[54076]: debug 2022-01-31T22:02:44.563+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:44.564414+0000) 2022-01-31T22:02:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:44 smithi167 conmon[49112]: debug 2022-01-31T22:02:44.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:44.900475+0000) 2022-01-31T22:02:45.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:45 smithi171 conmon[46715]: debug 2022-01-31T22:02:45.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:45.069121+0000) 2022-01-31T22:02:45.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:44 smithi171 conmon[51620]: debug 2022-01-31T22:02:44.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:44.918540+0000) 2022-01-31T22:02:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:45 smithi171 conmon[41853]: debug 2022-01-31T22:02:45.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:45.275477+0000) 2022-01-31T22:02:45.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:45 smithi167 conmon[60316]: debug 2022-01-31T22:02:45.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:45.564587+0000) 2022-01-31T22:02:45.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:45 smithi167 conmon[54076]: debug 2022-01-31T22:02:45.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:45.564589+0000) 2022-01-31T22:02:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:45 smithi167 conmon[49112]: debug 2022-01-31T22:02:45.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:45.900692+0000) 2022-01-31T22:02:46.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:46 smithi171 conmon[46715]: debug 2022-01-31T22:02:46.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:46.069322+0000) 2022-01-31T22:02:46.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:45 smithi171 conmon[51620]: debug 2022-01-31T22:02:45.917+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:45.918733+0000) 2022-01-31T22:02:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:46 smithi171 conmon[41853]: debug 2022-01-31T22:02:46.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:46.275658+0000) 2022-01-31T22:02:46.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:46 smithi167 conmon[60316]: debug 2022-01-31T22:02:46.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:46.564784+0000) 2022-01-31T22:02:46.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:46 smithi167 conmon[54076]: debug 2022-01-31T22:02:46.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:46.564764+0000) 2022-01-31T22:02:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:46 smithi167 conmon[49112]: debug 2022-01-31T22:02:46.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:46.900929+0000) 2022-01-31T22:02:47.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:47 smithi171 conmon[46715]: debug 2022-01-31T22:02:47.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:47.069461+0000) 2022-01-31T22:02:47.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:46 smithi171 conmon[51620]: debug 2022-01-31T22:02:46.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:46.918910+0000) 2022-01-31T22:02:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:47 smithi171 conmon[41853]: debug 2022-01-31T22:02:47.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:47.275822+0000) 2022-01-31T22:02:47.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:47 smithi167 conmon[54076]: debug 2022-01-31T22:02:47.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:47.564936+0000) 2022-01-31T22:02:47.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:47 smithi167 conmon[60316]: debug 2022-01-31T22:02:47.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:47.564939+0000) 2022-01-31T22:02:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:47 smithi167 conmon[49112]: debug 2022-01-31T22:02:47.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:47.901122+0000) 2022-01-31T22:02:48.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[46715]: debug 2022-01-31T22:02:48.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.069645+0000) 2022-01-31T22:02:48.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:47 smithi171 conmon[51620]: debug 2022-01-31T22:02:47.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:47.919035+0000) 2022-01-31T22:02:48.545 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:48 smithi167 conmon[49112]: debug 2022-01-31T22:02:48.184+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.184735+0000) 2022-01-31T22:02:48.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:48 smithi167 conmon[54076]: debug 2022-01-31T22:02:48.184+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.184712+0000) 2022-01-31T22:02:48.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:48 smithi167 conmon[60316]: debug 2022-01-31T22:02:48.183+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.184334+0000) 2022-01-31T22:02:48.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[35325]: debug 2022-01-31T22:02:48.202+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:48.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[41853]: debug 2022-01-31T22:02:48.184+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.185554+0000) 2022-01-31T22:02:48.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[41853]: debug 2022-01-31T22:02:48.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.275948+0000) 2022-01-31T22:02:48.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[46715]: debug 2022-01-31T22:02:48.184+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.185654+0000) 2022-01-31T22:02:48.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[51620]: debug 2022-01-31T22:02:48.184+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.185326+0000) 2022-01-31T22:02:48.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:48 smithi167 conmon[60316]: debug 2022-01-31T22:02:48.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.565145+0000) 2022-01-31T22:02:48.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:48 smithi167 conmon[54076]: debug 2022-01-31T22:02:48.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.565145+0000) 2022-01-31T22:02:49.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:48 smithi167 conmon[49112]: debug 2022-01-31T22:02:48.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.901290+0000) 2022-01-31T22:02:49.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:49 smithi171 conmon[46715]: debug 2022-01-31T22:02:49.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:49.069828+0000) 2022-01-31T22:02:49.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:48 smithi171 conmon[51620]: debug 2022-01-31T22:02:48.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:48.919204+0000) 2022-01-31T22:02:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:49 smithi171 conmon[41853]: debug 2022-01-31T22:02:49.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:49.276127+0000) 2022-01-31T22:02:49.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:49 smithi167 conmon[60316]: debug 2022-01-31T22:02:49.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:49.565303+0000) 2022-01-31T22:02:49.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:49 smithi167 conmon[54076]: debug 2022-01-31T22:02:49.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:49.565303+0000) 2022-01-31T22:02:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:49 smithi167 conmon[49112]: debug 2022-01-31T22:02:49.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:49.901441+0000) 2022-01-31T22:02:50.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:50 smithi171 conmon[46715]: debug 2022-01-31T22:02:50.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:50.069927+0000) 2022-01-31T22:02:50.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:49 smithi171 conmon[51620]: debug 2022-01-31T22:02:49.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:49.919301+0000) 2022-01-31T22:02:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:50 smithi171 conmon[41853]: debug 2022-01-31T22:02:50.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:50.276310+0000) 2022-01-31T22:02:50.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:50 smithi167 conmon[54076]: debug 2022-01-31T22:02:50.564+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:50.565473+0000) 2022-01-31T22:02:50.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:50 smithi167 conmon[60316]: debug 2022-01-31T22:02:50.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:50.565462+0000) 2022-01-31T22:02:51.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:50 smithi167 conmon[49112]: debug 2022-01-31T22:02:50.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:50.901667+0000) 2022-01-31T22:02:51.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:51 smithi171 conmon[46715]: debug 2022-01-31T22:02:51.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:51.070075+0000) 2022-01-31T22:02:51.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:50 smithi171 conmon[51620]: debug 2022-01-31T22:02:50.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:50.919466+0000) 2022-01-31T22:02:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:51 smithi171 conmon[41853]: debug 2022-01-31T22:02:51.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:51.276495+0000) 2022-01-31T22:02:51.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:51 smithi167 conmon[54076]: debug 2022-01-31T22:02:51.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:51.565659+0000) 2022-01-31T22:02:51.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:51 smithi167 conmon[60316]: debug 2022-01-31T22:02:51.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:51.565658+0000) 2022-01-31T22:02:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:51 smithi167 conmon[49112]: debug 2022-01-31T22:02:51.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:51.901866+0000) 2022-01-31T22:02:52.232 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:52 smithi171 conmon[46715]: debug 2022-01-31T22:02:52.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:52.070287+0000) 2022-01-31T22:02:52.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:51 smithi171 conmon[51620]: debug 2022-01-31T22:02:51.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:51.919736+0000) 2022-01-31T22:02:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:52 smithi171 conmon[41853]: debug 2022-01-31T22:02:52.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:52.276645+0000) 2022-01-31T22:02:52.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:52 smithi167 conmon[54076]: debug 2022-01-31T22:02:52.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:52.565784+0000) 2022-01-31T22:02:52.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:52 smithi167 conmon[60316]: debug 2022-01-31T22:02:52.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:52.565791+0000) 2022-01-31T22:02:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:52 smithi167 conmon[49112]: debug 2022-01-31T22:02:52.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:52.902083+0000) 2022-01-31T22:02:53.204 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:52 smithi171 conmon[51620]: debug 2022-01-31T22:02:52.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:52.919910+0000) 2022-01-31T22:02:53.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[46715]: debug 2022-01-31T22:02:53.068+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.070458+0000) 2022-01-31T22:02:53.547 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:53 smithi167 conmon[49112]: debug 2022-01-31T22:02:53.205+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.205548+0000) 2022-01-31T22:02:53.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:53 smithi167 conmon[54076]: debug 2022-01-31T22:02:53.206+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.206845+0000) 2022-01-31T22:02:53.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:53 smithi167 conmon[60316]: debug 2022-01-31T22:02:53.205+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.205689+0000) 2022-01-31T22:02:53.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[35325]: debug 2022-01-31T22:02:53.222+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178233 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:53.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[46715]: debug 2022-01-31T22:02:53.204+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.206212+0000) 2022-01-31T22:02:53.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[51620]: debug 2022-01-31T22:02:53.204+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.206538+0000) 2022-01-31T22:02:53.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[41853]: debug 2022-01-31T22:02:53.204+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.206350+0000) 2022-01-31T22:02:53.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[41853]: debug 2022-01-31T22:02:53.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.276801+0000) 2022-01-31T22:02:53.868 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:53 smithi167 conmon[54076]: debug 2022-01-31T22:02:53.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.565926+0000) 2022-01-31T22:02:53.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:53 smithi167 conmon[60316]: debug 2022-01-31T22:02:53.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.565962+0000) 2022-01-31T22:02:54.069 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[35325]: debug 2022-01-31T22:02:53.782+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:02:54.070 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:53 smithi171 conmon[51620]: debug 2022-01-31T22:02:53.918+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.920042+0000) 2022-01-31T22:02:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:53 smithi167 conmon[49112]: debug 2022-01-31T22:02:53.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:53.902304+0000) 2022-01-31T22:02:54.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:54 smithi171 conmon[41853]: debug 2022-01-31T22:02:54.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:54.276953+0000) 2022-01-31T22:02:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:54 smithi171 conmon[46715]: debug 2022-01-31T22:02:54.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:54.070643+0000) 2022-01-31T22:02:54.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:54 smithi167 conmon[54076]: debug 2022-01-31T22:02:54.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:54.566096+0000) 2022-01-31T22:02:54.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:54 smithi167 conmon[60316]: debug 2022-01-31T22:02:54.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:54.566104+0000) 2022-01-31T22:02:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:54 smithi167 conmon[49112]: debug 2022-01-31T22:02:54.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:54.902466+0000) 2022-01-31T22:02:55.232 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:55 smithi171 conmon[46715]: debug 2022-01-31T22:02:55.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:55.070788+0000) 2022-01-31T22:02:55.233 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:54 smithi171 conmon[51620]: debug 2022-01-31T22:02:54.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:54.920193+0000) 2022-01-31T22:02:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:55 smithi171 conmon[41853]: debug 2022-01-31T22:02:55.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:55.277110+0000) 2022-01-31T22:02:55.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:55 smithi167 conmon[54076]: debug 2022-01-31T22:02:55.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:55.566263+0000) 2022-01-31T22:02:55.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:55 smithi167 conmon[60316]: debug 2022-01-31T22:02:55.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:55.566298+0000) 2022-01-31T22:02:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:55 smithi167 conmon[49112]: debug 2022-01-31T22:02:55.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:55.902690+0000) 2022-01-31T22:02:56.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:55 smithi171 conmon[51620]: debug 2022-01-31T22:02:55.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:55.920366+0000) 2022-01-31T22:02:56.233 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:56 smithi171 conmon[46715]: debug 2022-01-31T22:02:56.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:56.070994+0000) 2022-01-31T22:02:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:56 smithi171 conmon[41853]: debug 2022-01-31T22:02:56.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:56.277266+0000) 2022-01-31T22:02:56.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:56 smithi167 conmon[60316]: debug 2022-01-31T22:02:56.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:56.566457+0000) 2022-01-31T22:02:56.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:56 smithi167 conmon[54076]: debug 2022-01-31T22:02:56.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:56.566451+0000) 2022-01-31T22:02:57.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:56 smithi167 conmon[49112]: debug 2022-01-31T22:02:56.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:56.902822+0000) 2022-01-31T22:02:57.233 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:57 smithi171 conmon[46715]: debug 2022-01-31T22:02:57.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:57.071172+0000) 2022-01-31T22:02:57.233 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:56 smithi171 conmon[51620]: debug 2022-01-31T22:02:56.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:56.920546+0000) 2022-01-31T22:02:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:57 smithi171 conmon[41853]: debug 2022-01-31T22:02:57.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:57.277417+0000) 2022-01-31T22:02:57.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:57 smithi167 conmon[54076]: debug 2022-01-31T22:02:57.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:57.566654+0000) 2022-01-31T22:02:57.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:57 smithi167 conmon[60316]: debug 2022-01-31T22:02:57.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:57.566658+0000) 2022-01-31T22:02:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:57 smithi167 conmon[49112]: debug 2022-01-31T22:02:57.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:57.903017+0000) 2022-01-31T22:02:58.225 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[46715]: debug 2022-01-31T22:02:58.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.071319+0000) 2022-01-31T22:02:58.225 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:57 smithi171 conmon[51620]: debug 2022-01-31T22:02:57.919+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:57.920790+0000) 2022-01-31T22:02:58.547 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:58 smithi167 conmon[49112]: debug 2022-01-31T22:02:58.226+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.227082+0000) 2022-01-31T22:02:58.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:58 smithi167 conmon[54076]: debug 2022-01-31T22:02:58.227+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.228113+0000) 2022-01-31T22:02:58.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:58 smithi167 conmon[60316]: debug 2022-01-31T22:02:58.226+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.227385+0000) 2022-01-31T22:02:58.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[35325]: debug 2022-01-31T22:02:58.243+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178344 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:02:58.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[46715]: debug 2022-01-31T22:02:58.225+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.226801+0000) 2022-01-31T22:02:58.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[51620]: debug 2022-01-31T22:02:58.225+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.227363+0000) 2022-01-31T22:02:58.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[41853]: debug 2022-01-31T22:02:58.225+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.227134+0000) 2022-01-31T22:02:58.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[41853]: debug 2022-01-31T22:02:58.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.277606+0000) 2022-01-31T22:02:58.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:58 smithi167 conmon[54076]: debug 2022-01-31T22:02:58.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.566819+0000) 2022-01-31T22:02:58.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:58 smithi167 conmon[60316]: debug 2022-01-31T22:02:58.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.566835+0000) 2022-01-31T22:02:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:58 smithi167 conmon[49112]: debug 2022-01-31T22:02:58.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.903230+0000) 2022-01-31T22:02:59.233 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:02:59 smithi171 conmon[46715]: debug 2022-01-31T22:02:59.070+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:59.071480+0000) 2022-01-31T22:02:59.234 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:58 smithi171 conmon[51620]: debug 2022-01-31T22:02:58.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:58.920985+0000) 2022-01-31T22:02:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:02:59 smithi171 conmon[41853]: debug 2022-01-31T22:02:59.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:59.277768+0000) 2022-01-31T22:02:59.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:02:59 smithi167 conmon[54076]: debug 2022-01-31T22:02:59.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:59.566978+0000) 2022-01-31T22:02:59.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:02:59 smithi167 conmon[60316]: debug 2022-01-31T22:02:59.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:59.566977+0000) 2022-01-31T22:03:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:02:59 smithi167 conmon[49112]: debug 2022-01-31T22:02:59.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:59.903410+0000) 2022-01-31T22:03:00.233 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:00 smithi171 conmon[46715]: debug 2022-01-31T22:03:00.069+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:00.071650+0000) 2022-01-31T22:03:00.234 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:02:59 smithi171 conmon[51620]: debug 2022-01-31T22:02:59.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:02:59.921177+0000) 2022-01-31T22:03:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:00 smithi171 conmon[41853]: debug 2022-01-31T22:03:00.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:00.277940+0000) 2022-01-31T22:03:00.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:00 smithi167 conmon[54076]: debug 2022-01-31T22:03:00.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:00.567171+0000) 2022-01-31T22:03:00.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:00 smithi167 conmon[60316]: debug 2022-01-31T22:03:00.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:00.567134+0000) 2022-01-31T22:03:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:00 smithi167 conmon[49112]: debug 2022-01-31T22:03:00.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:00.903557+0000) 2022-01-31T22:03:01.233 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:01 smithi171 conmon[46715]: debug 2022-01-31T22:03:01.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:01.071845+0000) 2022-01-31T22:03:01.234 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:00 smithi171 conmon[51620]: debug 2022-01-31T22:03:00.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:00.921382+0000) 2022-01-31T22:03:01.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:01 smithi171 conmon[41853]: debug 2022-01-31T22:03:01.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:01.278129+0000) 2022-01-31T22:03:01.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:01 smithi167 conmon[54076]: debug 2022-01-31T22:03:01.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:01.567340+0000) 2022-01-31T22:03:01.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:01 smithi167 conmon[60316]: debug 2022-01-31T22:03:01.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:01.567334+0000) 2022-01-31T22:03:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:01 smithi167 conmon[49112]: debug 2022-01-31T22:03:01.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:01.903692+0000) 2022-01-31T22:03:02.233 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:02 smithi171 conmon[46715]: debug 2022-01-31T22:03:02.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:02.072031+0000) 2022-01-31T22:03:02.234 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:01 smithi171 conmon[51620]: debug 2022-01-31T22:03:01.920+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:01.921587+0000) 2022-01-31T22:03:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:02 smithi171 conmon[41853]: debug 2022-01-31T22:03:02.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:02.278317+0000) 2022-01-31T22:03:02.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:02 smithi167 conmon[54076]: debug 2022-01-31T22:03:02.565+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:02.567472+0000) 2022-01-31T22:03:02.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:02 smithi167 conmon[60316]: debug 2022-01-31T22:03:02.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:02.567463+0000) 2022-01-31T22:03:03.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:02 smithi167 conmon[49112]: debug 2022-01-31T22:03:02.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:02.903884+0000) 2022-01-31T22:03:03.234 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[46715]: debug 2022-01-31T22:03:03.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.072218+0000) 2022-01-31T22:03:03.234 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:02 smithi171 conmon[51620]: debug 2022-01-31T22:03:02.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:02.921772+0000) 2022-01-31T22:03:03.548 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:03 smithi167 conmon[49112]: debug 2022-01-31T22:03:03.246+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.247803+0000) 2022-01-31T22:03:03.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:03 smithi167 conmon[54076]: debug 2022-01-31T22:03:03.247+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.248786+0000) 2022-01-31T22:03:03.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:03 smithi167 conmon[60316]: debug 2022-01-31T22:03:03.246+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.248247+0000) 2022-01-31T22:03:03.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[46715]: debug 2022-01-31T22:03:03.247+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.247844+0000) 2022-01-31T22:03:03.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[35325]: debug 2022-01-31T22:03:03.265+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178455 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:03.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[51620]: debug 2022-01-31T22:03:03.247+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.248639+0000) 2022-01-31T22:03:03.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[41853]: debug 2022-01-31T22:03:03.247+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.248487+0000) 2022-01-31T22:03:03.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[41853]: debug 2022-01-31T22:03:03.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.278537+0000) 2022-01-31T22:03:03.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:03 smithi167 conmon[54076]: debug 2022-01-31T22:03:03.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.567658+0000) 2022-01-31T22:03:03.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:03 smithi167 conmon[60316]: debug 2022-01-31T22:03:03.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.567653+0000) 2022-01-31T22:03:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:03 smithi167 conmon[49112]: debug 2022-01-31T22:03:03.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.904088+0000) 2022-01-31T22:03:04.234 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:04 smithi171 conmon[46715]: debug 2022-01-31T22:03:04.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:04.072436+0000) 2022-01-31T22:03:04.235 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:03 smithi171 conmon[51620]: debug 2022-01-31T22:03:03.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:03.921936+0000) 2022-01-31T22:03:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:04 smithi171 conmon[41853]: debug 2022-01-31T22:03:04.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:04.278689+0000) 2022-01-31T22:03:04.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:04 smithi167 conmon[54076]: debug 2022-01-31T22:03:04.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:04.567808+0000) 2022-01-31T22:03:04.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:04 smithi167 conmon[60316]: debug 2022-01-31T22:03:04.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:04.567808+0000) 2022-01-31T22:03:05.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:04 smithi167 conmon[49112]: debug 2022-01-31T22:03:04.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:04.904221+0000) 2022-01-31T22:03:05.234 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:05 smithi171 conmon[46715]: debug 2022-01-31T22:03:05.071+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:05.072588+0000) 2022-01-31T22:03:05.234 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:04 smithi171 conmon[51620]: debug 2022-01-31T22:03:04.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:04.922092+0000) 2022-01-31T22:03:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:05 smithi171 conmon[41853]: debug 2022-01-31T22:03:05.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:05.278902+0000) 2022-01-31T22:03:05.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:05 smithi167 conmon[54076]: debug 2022-01-31T22:03:05.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:05.567998+0000) 2022-01-31T22:03:05.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:05 smithi167 conmon[60316]: debug 2022-01-31T22:03:05.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:05.568003+0000) 2022-01-31T22:03:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:05 smithi167 conmon[49112]: debug 2022-01-31T22:03:05.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:05.904432+0000) 2022-01-31T22:03:06.234 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:06 smithi171 conmon[46715]: debug 2022-01-31T22:03:06.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:06.072789+0000) 2022-01-31T22:03:06.235 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:05 smithi171 conmon[51620]: debug 2022-01-31T22:03:05.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:05.922270+0000) 2022-01-31T22:03:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:06 smithi171 conmon[41853]: debug 2022-01-31T22:03:06.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:06.279099+0000) 2022-01-31T22:03:06.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:06 smithi167 conmon[54076]: debug 2022-01-31T22:03:06.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:06.568176+0000) 2022-01-31T22:03:06.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:06 smithi167 conmon[60316]: debug 2022-01-31T22:03:06.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:06.568192+0000) 2022-01-31T22:03:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:06 smithi167 conmon[49112]: debug 2022-01-31T22:03:06.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:06.904626+0000) 2022-01-31T22:03:07.234 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:07 smithi171 conmon[46715]: debug 2022-01-31T22:03:07.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:07.072969+0000) 2022-01-31T22:03:07.235 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:06 smithi171 conmon[51620]: debug 2022-01-31T22:03:06.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:06.922456+0000) 2022-01-31T22:03:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:07 smithi171 conmon[41853]: debug 2022-01-31T22:03:07.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:07.279212+0000) 2022-01-31T22:03:07.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:07 smithi167 conmon[54076]: debug 2022-01-31T22:03:07.566+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:07.568400+0000) 2022-01-31T22:03:07.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:07 smithi167 conmon[60316]: debug 2022-01-31T22:03:07.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:07.568403+0000) 2022-01-31T22:03:08.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:07 smithi167 conmon[49112]: debug 2022-01-31T22:03:07.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:07.904761+0000) 2022-01-31T22:03:08.234 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[46715]: debug 2022-01-31T22:03:08.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.073157+0000) 2022-01-31T22:03:08.235 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:07 smithi171 conmon[51620]: debug 2022-01-31T22:03:07.921+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:07.922632+0000) 2022-01-31T22:03:08.549 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:08 smithi167 conmon[49112]: debug 2022-01-31T22:03:08.267+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.269270+0000) 2022-01-31T22:03:08.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:08 smithi167 conmon[54076]: debug 2022-01-31T22:03:08.267+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.269186+0000) 2022-01-31T22:03:08.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:08 smithi167 conmon[60316]: debug 2022-01-31T22:03:08.266+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.268192+0000) 2022-01-31T22:03:08.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[35325]: debug 2022-01-31T22:03:08.287+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178567 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:08.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[46715]: debug 2022-01-31T22:03:08.267+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.267831+0000) 2022-01-31T22:03:08.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[51620]: debug 2022-01-31T22:03:08.268+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.268879+0000) 2022-01-31T22:03:08.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[41853]: debug 2022-01-31T22:03:08.267+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.268662+0000) 2022-01-31T22:03:08.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[41853]: debug 2022-01-31T22:03:08.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.279401+0000) 2022-01-31T22:03:08.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:08 smithi167 conmon[54076]: debug 2022-01-31T22:03:08.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.568600+0000) 2022-01-31T22:03:08.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:08 smithi167 conmon[60316]: debug 2022-01-31T22:03:08.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.568599+0000) 2022-01-31T22:03:09.072 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[35325]: debug 2022-01-31T22:03:08.783+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:03:09.073 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:08 smithi171 conmon[51620]: debug 2022-01-31T22:03:08.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.922737+0000) 2022-01-31T22:03:09.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:08 smithi167 conmon[49112]: debug 2022-01-31T22:03:08.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:08.904936+0000) 2022-01-31T22:03:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:09 smithi171 conmon[41853]: debug 2022-01-31T22:03:09.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:09.279555+0000) 2022-01-31T22:03:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:09 smithi171 conmon[46715]: debug 2022-01-31T22:03:09.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:09.073314+0000) 2022-01-31T22:03:09.620 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:09 smithi167 conmon[54076]: debug 2022-01-31T22:03:09.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:09.568758+0000) 2022-01-31T22:03:09.621 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:09 smithi167 conmon[60316]: debug 2022-01-31T22:03:09.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:09.568758+0000) 2022-01-31T22:03:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:09 smithi167 conmon[49112]: debug 2022-01-31T22:03:09.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:09.905202+0000) 2022-01-31T22:03:10.235 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:10 smithi171 conmon[46715]: debug 2022-01-31T22:03:10.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:10.073464+0000) 2022-01-31T22:03:10.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:09 smithi171 conmon[51620]: debug 2022-01-31T22:03:09.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:09.922969+0000) 2022-01-31T22:03:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:10 smithi171 conmon[41853]: debug 2022-01-31T22:03:10.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:10.279733+0000) 2022-01-31T22:03:10.713 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:10 smithi167 conmon[54076]: debug 2022-01-31T22:03:10.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:10.568957+0000) 2022-01-31T22:03:10.714 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:10 smithi167 conmon[60316]: debug 2022-01-31T22:03:10.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:10.568951+0000) 2022-01-31T22:03:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:10 smithi167 conmon[49112]: debug 2022-01-31T22:03:10.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:10.905359+0000) 2022-01-31T22:03:11.235 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:11 smithi171 conmon[46715]: debug 2022-01-31T22:03:11.072+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:11.073648+0000) 2022-01-31T22:03:11.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:10 smithi171 conmon[51620]: debug 2022-01-31T22:03:10.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:10.923166+0000) 2022-01-31T22:03:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:11 smithi171 conmon[41853]: debug 2022-01-31T22:03:11.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:11.279936+0000) 2022-01-31T22:03:11.621 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:11 smithi167 conmon[54076]: debug 2022-01-31T22:03:11.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:11.569154+0000) 2022-01-31T22:03:11.621 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:11 smithi167 conmon[60316]: debug 2022-01-31T22:03:11.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:11.569176+0000) 2022-01-31T22:03:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:11 smithi167 conmon[49112]: debug 2022-01-31T22:03:11.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:11.905607+0000) 2022-01-31T22:03:12.235 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:12 smithi171 conmon[46715]: debug 2022-01-31T22:03:12.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:12.073800+0000) 2022-01-31T22:03:12.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:11 smithi171 conmon[51620]: debug 2022-01-31T22:03:11.922+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:11.923376+0000) 2022-01-31T22:03:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:12 smithi171 conmon[41853]: debug 2022-01-31T22:03:12.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:12.280146+0000) 2022-01-31T22:03:12.621 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:12 smithi167 conmon[54076]: debug 2022-01-31T22:03:12.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:12.569275+0000) 2022-01-31T22:03:12.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:12 smithi167 conmon[60316]: debug 2022-01-31T22:03:12.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:12.569344+0000) 2022-01-31T22:03:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:12 smithi167 conmon[49112]: debug 2022-01-31T22:03:12.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:12.905752+0000) 2022-01-31T22:03:13.235 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[46715]: debug 2022-01-31T22:03:13.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.073987+0000) 2022-01-31T22:03:13.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:12 smithi171 conmon[51620]: debug 2022-01-31T22:03:12.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:12.923591+0000) 2022-01-31T22:03:13.550 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:13 smithi167 conmon[60316]: debug 2022-01-31T22:03:13.288+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.289848+0000) 2022-01-31T22:03:13.550 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:13 smithi167 conmon[49112]: debug 2022-01-31T22:03:13.289+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.291153+0000) 2022-01-31T22:03:13.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:13 smithi167 conmon[54076]: debug 2022-01-31T22:03:13.288+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.290441+0000) 2022-01-31T22:03:13.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[35325]: debug 2022-01-31T22:03:13.308+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178678 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:13.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[41853]: debug 2022-01-31T22:03:13.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.280310+0000) 2022-01-31T22:03:13.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[41853]: debug 2022-01-31T22:03:13.289+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.290666+0000) 2022-01-31T22:03:13.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[46715]: debug 2022-01-31T22:03:13.289+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.290461+0000) 2022-01-31T22:03:13.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[51620]: debug 2022-01-31T22:03:13.290+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.291014+0000) 2022-01-31T22:03:13.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:13 smithi167 conmon[54076]: debug 2022-01-31T22:03:13.567+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.569441+0000) 2022-01-31T22:03:13.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:13 smithi167 conmon[60316]: debug 2022-01-31T22:03:13.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.569536+0000) 2022-01-31T22:03:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:13 smithi167 conmon[49112]: debug 2022-01-31T22:03:13.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.905885+0000) 2022-01-31T22:03:14.236 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:14 smithi171 conmon[46715]: debug 2022-01-31T22:03:14.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:14.074144+0000) 2022-01-31T22:03:14.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:13 smithi171 conmon[51620]: debug 2022-01-31T22:03:13.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:13.923783+0000) 2022-01-31T22:03:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:14 smithi171 conmon[41853]: debug 2022-01-31T22:03:14.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:14.280513+0000) 2022-01-31T22:03:14.621 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:14 smithi167 conmon[54076]: debug 2022-01-31T22:03:14.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:14.569710+0000) 2022-01-31T22:03:14.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:14 smithi167 conmon[60316]: debug 2022-01-31T22:03:14.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:14.569639+0000) 2022-01-31T22:03:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:14 smithi167 conmon[49112]: debug 2022-01-31T22:03:14.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:14.906044+0000) 2022-01-31T22:03:15.236 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:15 smithi171 conmon[46715]: debug 2022-01-31T22:03:15.073+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:15.074325+0000) 2022-01-31T22:03:15.237 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:14 smithi171 conmon[51620]: debug 2022-01-31T22:03:14.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:14.923865+0000) 2022-01-31T22:03:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:15 smithi171 conmon[41853]: debug 2022-01-31T22:03:15.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:15.280672+0000) 2022-01-31T22:03:15.621 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:15 smithi167 conmon[54076]: debug 2022-01-31T22:03:15.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:15.569886+0000) 2022-01-31T22:03:15.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:15 smithi167 conmon[60316]: debug 2022-01-31T22:03:15.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:15.569875+0000) 2022-01-31T22:03:16.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:15 smithi167 conmon[49112]: debug 2022-01-31T22:03:15.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:15.906216+0000) 2022-01-31T22:03:16.236 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:16 smithi171 conmon[46715]: debug 2022-01-31T22:03:16.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:16.074483+0000) 2022-01-31T22:03:16.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:15 smithi171 conmon[51620]: debug 2022-01-31T22:03:15.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:15.924044+0000) 2022-01-31T22:03:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:16 smithi171 conmon[41853]: debug 2022-01-31T22:03:16.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:16.280834+0000) 2022-01-31T22:03:16.621 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:16 smithi167 conmon[54076]: debug 2022-01-31T22:03:16.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:16.570051+0000) 2022-01-31T22:03:16.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:16 smithi167 conmon[60316]: debug 2022-01-31T22:03:16.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:16.570050+0000) 2022-01-31T22:03:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:16 smithi167 conmon[49112]: debug 2022-01-31T22:03:16.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:16.906418+0000) 2022-01-31T22:03:17.236 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:17 smithi171 conmon[46715]: debug 2022-01-31T22:03:17.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:17.074614+0000) 2022-01-31T22:03:17.237 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:16 smithi171 conmon[51620]: debug 2022-01-31T22:03:16.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:16.924238+0000) 2022-01-31T22:03:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:17 smithi171 conmon[41853]: debug 2022-01-31T22:03:17.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:17.280940+0000) 2022-01-31T22:03:17.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:17 smithi167 conmon[60316]: debug 2022-01-31T22:03:17.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:17.570252+0000) 2022-01-31T22:03:17.622 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:17 smithi167 conmon[54076]: debug 2022-01-31T22:03:17.568+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:17.570252+0000) 2022-01-31T22:03:18.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:17 smithi171 conmon[51620]: debug 2022-01-31T22:03:17.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:17.924438+0000) 2022-01-31T22:03:18.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:17 smithi167 conmon[49112]: debug 2022-01-31T22:03:17.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:17.906623+0000) 2022-01-31T22:03:18.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[46715]: debug 2022-01-31T22:03:18.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.074783+0000) 2022-01-31T22:03:18.569 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:18 smithi167 conmon[49112]: debug 2022-01-31T22:03:18.310+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.312411+0000) 2022-01-31T22:03:18.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:18 smithi167 conmon[54076]: debug 2022-01-31T22:03:18.309+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.311457+0000) 2022-01-31T22:03:18.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:18 smithi167 conmon[60316]: debug 2022-01-31T22:03:18.311+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.312732+0000) 2022-01-31T22:03:18.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[35325]: debug 2022-01-31T22:03:18.329+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178790 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:18.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[46715]: debug 2022-01-31T22:03:18.311+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.312238+0000) 2022-01-31T22:03:18.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[51620]: debug 2022-01-31T22:03:18.311+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.312525+0000) 2022-01-31T22:03:18.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[41853]: debug 2022-01-31T22:03:18.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.281087+0000) 2022-01-31T22:03:18.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[41853]: debug 2022-01-31T22:03:18.310+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.311736+0000) 2022-01-31T22:03:18.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:18 smithi167 conmon[54076]: debug 2022-01-31T22:03:18.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.570411+0000) 2022-01-31T22:03:18.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:18 smithi167 conmon[60316]: debug 2022-01-31T22:03:18.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.570411+0000) 2022-01-31T22:03:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:18 smithi167 conmon[49112]: debug 2022-01-31T22:03:18.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.906830+0000) 2022-01-31T22:03:19.236 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:19 smithi171 conmon[46715]: debug 2022-01-31T22:03:19.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:19.074969+0000) 2022-01-31T22:03:19.237 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:18 smithi171 conmon[51620]: debug 2022-01-31T22:03:18.923+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:18.924659+0000) 2022-01-31T22:03:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:19 smithi171 conmon[41853]: debug 2022-01-31T22:03:19.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:19.281274+0000) 2022-01-31T22:03:19.622 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:19 smithi167 conmon[54076]: debug 2022-01-31T22:03:19.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:19.570604+0000) 2022-01-31T22:03:19.623 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:19 smithi167 conmon[60316]: debug 2022-01-31T22:03:19.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:19.570604+0000) 2022-01-31T22:03:20.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:19 smithi167 conmon[49112]: debug 2022-01-31T22:03:19.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:19.906987+0000) 2022-01-31T22:03:20.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:20 smithi171 conmon[46715]: debug 2022-01-31T22:03:20.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:20.075113+0000) 2022-01-31T22:03:20.237 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:19 smithi171 conmon[51620]: debug 2022-01-31T22:03:19.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:19.924863+0000) 2022-01-31T22:03:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:20 smithi171 conmon[41853]: debug 2022-01-31T22:03:20.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:20.281470+0000) 2022-01-31T22:03:20.622 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:20 smithi167 conmon[60316]: debug 2022-01-31T22:03:20.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:20.570775+0000) 2022-01-31T22:03:20.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:20 smithi167 conmon[54076]: debug 2022-01-31T22:03:20.569+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:20.570775+0000) 2022-01-31T22:03:21.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:20 smithi167 conmon[49112]: debug 2022-01-31T22:03:20.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:20.907173+0000) 2022-01-31T22:03:21.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:21 smithi171 conmon[46715]: debug 2022-01-31T22:03:21.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:21.075311+0000) 2022-01-31T22:03:21.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:20 smithi171 conmon[51620]: debug 2022-01-31T22:03:20.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:20.925097+0000) 2022-01-31T22:03:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:21 smithi171 conmon[41853]: debug 2022-01-31T22:03:21.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:21.281651+0000) 2022-01-31T22:03:21.622 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:21 smithi167 conmon[54076]: debug 2022-01-31T22:03:21.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:21.570938+0000) 2022-01-31T22:03:21.623 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:21 smithi167 conmon[60316]: debug 2022-01-31T22:03:21.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:21.570938+0000) 2022-01-31T22:03:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:21 smithi167 conmon[49112]: debug 2022-01-31T22:03:21.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:21.907353+0000) 2022-01-31T22:03:22.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:22 smithi171 conmon[46715]: debug 2022-01-31T22:03:22.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:22.075496+0000) 2022-01-31T22:03:22.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:21 smithi171 conmon[51620]: debug 2022-01-31T22:03:21.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:21.925333+0000) 2022-01-31T22:03:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:22 smithi171 conmon[41853]: debug 2022-01-31T22:03:22.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:22.281837+0000) 2022-01-31T22:03:22.623 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:22 smithi167 conmon[60316]: debug 2022-01-31T22:03:22.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:22.571122+0000) 2022-01-31T22:03:22.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:22 smithi167 conmon[54076]: debug 2022-01-31T22:03:22.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:22.571122+0000) 2022-01-31T22:03:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:22 smithi167 conmon[49112]: debug 2022-01-31T22:03:22.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:22.907563+0000) 2022-01-31T22:03:23.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[46715]: debug 2022-01-31T22:03:23.074+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.075669+0000) 2022-01-31T22:03:23.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:22 smithi171 conmon[51620]: debug 2022-01-31T22:03:22.924+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:22.925561+0000) 2022-01-31T22:03:23.354 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:23 smithi167 conmon[49112]: debug 2022-01-31T22:03:23.332+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.333624+0000) 2022-01-31T22:03:23.355 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:23 smithi167 conmon[54076]: debug 2022-01-31T22:03:23.331+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.333139+0000) 2022-01-31T22:03:23.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:23 smithi167 conmon[60316]: debug 2022-01-31T22:03:23.332+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.333779+0000) 2022-01-31T22:03:23.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[35325]: debug 2022-01-31T22:03:23.350+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 178900 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:23.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[41853]: debug 2022-01-31T22:03:23.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.282035+0000) 2022-01-31T22:03:23.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[41853]: debug 2022-01-31T22:03:23.333+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.333820+0000) 2022-01-31T22:03:23.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[46715]: debug 2022-01-31T22:03:23.333+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.334070+0000) 2022-01-31T22:03:23.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[51620]: debug 2022-01-31T22:03:23.331+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.332597+0000) 2022-01-31T22:03:23.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:23 smithi167 conmon[54076]: debug 2022-01-31T22:03:23.570+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.571293+0000) 2022-01-31T22:03:23.623 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:23 smithi167 conmon[60316]: debug 2022-01-31T22:03:23.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.571293+0000) 2022-01-31T22:03:24.075 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[35325]: debug 2022-01-31T22:03:23.784+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:03:24.075 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:23 smithi171 conmon[51620]: debug 2022-01-31T22:03:23.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.925788+0000) 2022-01-31T22:03:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:23 smithi167 conmon[49112]: debug 2022-01-31T22:03:23.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:23.907719+0000) 2022-01-31T22:03:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:24 smithi171 conmon[46715]: debug 2022-01-31T22:03:24.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:24.075880+0000) 2022-01-31T22:03:24.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:24 smithi171 conmon[41853]: debug 2022-01-31T22:03:24.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:24.282230+0000) 2022-01-31T22:03:24.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:24 smithi167 conmon[54076]: debug 2022-01-31T22:03:24.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:24.571495+0000) 2022-01-31T22:03:24.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:24 smithi167 conmon[60316]: debug 2022-01-31T22:03:24.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:24.571495+0000) 2022-01-31T22:03:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:24 smithi167 conmon[49112]: debug 2022-01-31T22:03:24.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:24.907941+0000) 2022-01-31T22:03:25.238 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:25 smithi171 conmon[46715]: debug 2022-01-31T22:03:25.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:25.076031+0000) 2022-01-31T22:03:25.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:24 smithi171 conmon[51620]: debug 2022-01-31T22:03:24.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:24.925940+0000) 2022-01-31T22:03:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:25 smithi171 conmon[41853]: debug 2022-01-31T22:03:25.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:25.282400+0000) 2022-01-31T22:03:25.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:25 smithi167 conmon[54076]: debug 2022-01-31T22:03:25.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:25.571673+0000) 2022-01-31T22:03:25.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:25 smithi167 conmon[60316]: debug 2022-01-31T22:03:25.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:25.571674+0000) 2022-01-31T22:03:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:25 smithi167 conmon[49112]: debug 2022-01-31T22:03:25.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:25.908135+0000) 2022-01-31T22:03:26.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:26 smithi171 conmon[46715]: debug 2022-01-31T22:03:26.075+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:26.076220+0000) 2022-01-31T22:03:26.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:25 smithi171 conmon[51620]: debug 2022-01-31T22:03:25.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:25.926157+0000) 2022-01-31T22:03:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:26 smithi171 conmon[41853]: debug 2022-01-31T22:03:26.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:26.282587+0000) 2022-01-31T22:03:26.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:26 smithi167 conmon[54076]: debug 2022-01-31T22:03:26.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:26.571854+0000) 2022-01-31T22:03:26.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:26 smithi167 conmon[60316]: debug 2022-01-31T22:03:26.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:26.571855+0000) 2022-01-31T22:03:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:26 smithi167 conmon[49112]: debug 2022-01-31T22:03:26.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:26.908364+0000) 2022-01-31T22:03:27.238 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:27 smithi171 conmon[46715]: debug 2022-01-31T22:03:27.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:27.076405+0000) 2022-01-31T22:03:27.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:26 smithi171 conmon[51620]: debug 2022-01-31T22:03:26.925+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:26.926368+0000) 2022-01-31T22:03:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:27 smithi171 conmon[41853]: debug 2022-01-31T22:03:27.282+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:27.282795+0000) 2022-01-31T22:03:27.623 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:27 smithi167 conmon[54076]: debug 2022-01-31T22:03:27.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:27.572072+0000) 2022-01-31T22:03:27.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:27 smithi167 conmon[60316]: debug 2022-01-31T22:03:27.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:27.572072+0000) 2022-01-31T22:03:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:27 smithi167 conmon[49112]: debug 2022-01-31T22:03:27.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:27.908525+0000) 2022-01-31T22:03:28.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:27 smithi171 conmon[51620]: debug 2022-01-31T22:03:27.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:27.926498+0000) 2022-01-31T22:03:28.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[46715]: debug 2022-01-31T22:03:28.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.076562+0000) 2022-01-31T22:03:28.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[35325]: debug 2022-01-31T22:03:28.371+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179011 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:28.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[46715]: debug 2022-01-31T22:03:28.352+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.353389+0000) 2022-01-31T22:03:28.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[51620]: debug 2022-01-31T22:03:28.352+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.353171+0000) 2022-01-31T22:03:28.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[41853]: debug 2022-01-31T22:03:28.282+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.283015+0000) 2022-01-31T22:03:28.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[41853]: debug 2022-01-31T22:03:28.353+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.354394+0000) 2022-01-31T22:03:28.624 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:28 smithi167 conmon[49112]: debug 2022-01-31T22:03:28.353+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.354113+0000) 2022-01-31T22:03:28.624 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:28 smithi167 conmon[54076]: debug 2022-01-31T22:03:28.354+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.354552+0000) 2022-01-31T22:03:28.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:28 smithi167 conmon[54076]: debug 2022-01-31T22:03:28.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.572259+0000) 2022-01-31T22:03:28.625 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:28 smithi167 conmon[60316]: debug 2022-01-31T22:03:28.353+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.353566+0000) 2022-01-31T22:03:28.625 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:28 smithi167 conmon[60316]: debug 2022-01-31T22:03:28.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.572259+0000) 2022-01-31T22:03:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:28 smithi167 conmon[49112]: debug 2022-01-31T22:03:28.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.908692+0000) 2022-01-31T22:03:29.238 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:29 smithi171 conmon[46715]: debug 2022-01-31T22:03:29.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:29.076720+0000) 2022-01-31T22:03:29.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:28 smithi171 conmon[51620]: debug 2022-01-31T22:03:28.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:28.926674+0000) 2022-01-31T22:03:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:29 smithi171 conmon[41853]: debug 2022-01-31T22:03:29.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:29.283219+0000) 2022-01-31T22:03:29.624 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:29 smithi167 conmon[54076]: debug 2022-01-31T22:03:29.571+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:29.572426+0000) 2022-01-31T22:03:29.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:29 smithi167 conmon[60316]: debug 2022-01-31T22:03:29.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:29.572426+0000) 2022-01-31T22:03:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:29 smithi167 conmon[49112]: debug 2022-01-31T22:03:29.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:29.908805+0000) 2022-01-31T22:03:30.238 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:30 smithi171 conmon[46715]: debug 2022-01-31T22:03:30.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:30.076873+0000) 2022-01-31T22:03:30.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:29 smithi171 conmon[51620]: debug 2022-01-31T22:03:29.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:29.926852+0000) 2022-01-31T22:03:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:30 smithi171 conmon[41853]: debug 2022-01-31T22:03:30.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:30.283377+0000) 2022-01-31T22:03:30.624 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:30 smithi167 conmon[54076]: debug 2022-01-31T22:03:30.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:30.572604+0000) 2022-01-31T22:03:30.624 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:30 smithi167 conmon[60316]: debug 2022-01-31T22:03:30.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:30.572604+0000) 2022-01-31T22:03:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:30 smithi167 conmon[49112]: debug 2022-01-31T22:03:30.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:30.909011+0000) 2022-01-31T22:03:31.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:31 smithi171 conmon[46715]: debug 2022-01-31T22:03:31.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:31.077025+0000) 2022-01-31T22:03:31.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:30 smithi171 conmon[51620]: debug 2022-01-31T22:03:30.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:30.927030+0000) 2022-01-31T22:03:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:31 smithi171 conmon[41853]: debug 2022-01-31T22:03:31.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:31.283557+0000) 2022-01-31T22:03:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:31 smithi167 conmon[54076]: debug 2022-01-31T22:03:31.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:31.572841+0000) 2022-01-31T22:03:31.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:31 smithi167 conmon[60316]: debug 2022-01-31T22:03:31.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:31.572841+0000) 2022-01-31T22:03:32.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:31 smithi167 conmon[49112]: debug 2022-01-31T22:03:31.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:31.909205+0000) 2022-01-31T22:03:32.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:32 smithi171 conmon[46715]: debug 2022-01-31T22:03:32.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:32.077160+0000) 2022-01-31T22:03:32.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:31 smithi171 conmon[51620]: debug 2022-01-31T22:03:31.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:31.927205+0000) 2022-01-31T22:03:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:32 smithi171 conmon[41853]: debug 2022-01-31T22:03:32.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:32.283717+0000) 2022-01-31T22:03:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:32 smithi167 conmon[54076]: debug 2022-01-31T22:03:32.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:32.573066+0000) 2022-01-31T22:03:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:32 smithi167 conmon[60316]: debug 2022-01-31T22:03:32.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:32.573066+0000) 2022-01-31T22:03:33.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:32 smithi167 conmon[49112]: debug 2022-01-31T22:03:32.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:32.909375+0000) 2022-01-31T22:03:33.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[46715]: debug 2022-01-31T22:03:33.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.077310+0000) 2022-01-31T22:03:33.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:32 smithi171 conmon[51620]: debug 2022-01-31T22:03:32.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:32.927393+0000) 2022-01-31T22:03:33.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[35325]: debug 2022-01-31T22:03:33.398+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:33.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[41853]: debug 2022-01-31T22:03:33.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.283931+0000) 2022-01-31T22:03:33.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[41853]: debug 2022-01-31T22:03:33.374+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.375004+0000) 2022-01-31T22:03:33.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[46715]: debug 2022-01-31T22:03:33.373+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.374486+0000) 2022-01-31T22:03:33.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[51620]: debug 2022-01-31T22:03:33.373+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.374372+0000) 2022-01-31T22:03:33.625 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:33 smithi167 conmon[49112]: debug 2022-01-31T22:03:33.375+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.375967+0000) 2022-01-31T22:03:33.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:33 smithi167 conmon[54076]: debug 2022-01-31T22:03:33.374+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.375369+0000) 2022-01-31T22:03:33.626 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:33 smithi167 conmon[54076]: debug 2022-01-31T22:03:33.572+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.573279+0000) 2022-01-31T22:03:33.626 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:33 smithi167 conmon[60316]: debug 2022-01-31T22:03:33.374+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.374975+0000) 2022-01-31T22:03:33.626 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:33 smithi167 conmon[60316]: debug 2022-01-31T22:03:33.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.573279+0000) 2022-01-31T22:03:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:33 smithi167 conmon[49112]: debug 2022-01-31T22:03:33.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.909588+0000) 2022-01-31T22:03:34.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:34 smithi171 conmon[46715]: debug 2022-01-31T22:03:34.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:34.077506+0000) 2022-01-31T22:03:34.239 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:33 smithi171 conmon[51620]: debug 2022-01-31T22:03:33.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:33.927582+0000) 2022-01-31T22:03:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:34 smithi171 conmon[41853]: debug 2022-01-31T22:03:34.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:34.284114+0000) 2022-01-31T22:03:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:34 smithi167 conmon[54076]: debug 2022-01-31T22:03:34.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:34.573446+0000) 2022-01-31T22:03:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:34 smithi167 conmon[60316]: debug 2022-01-31T22:03:34.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:34.573444+0000) 2022-01-31T22:03:35.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:34 smithi167 conmon[49112]: debug 2022-01-31T22:03:34.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:34.909724+0000) 2022-01-31T22:03:35.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:35 smithi171 conmon[46715]: debug 2022-01-31T22:03:35.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:35.077636+0000) 2022-01-31T22:03:35.240 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:34 smithi171 conmon[51620]: debug 2022-01-31T22:03:34.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:34.927739+0000) 2022-01-31T22:03:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:35 smithi171 conmon[41853]: debug 2022-01-31T22:03:35.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:35.284300+0000) 2022-01-31T22:03:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:35 smithi167 conmon[54076]: debug 2022-01-31T22:03:35.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:35.573681+0000) 2022-01-31T22:03:35.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:35 smithi167 conmon[60316]: debug 2022-01-31T22:03:35.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:35.573681+0000) 2022-01-31T22:03:36.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:35 smithi167 conmon[49112]: debug 2022-01-31T22:03:35.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:35.909912+0000) 2022-01-31T22:03:36.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:36 smithi171 conmon[46715]: debug 2022-01-31T22:03:36.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:36.077796+0000) 2022-01-31T22:03:36.240 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:35 smithi171 conmon[51620]: debug 2022-01-31T22:03:35.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:35.927973+0000) 2022-01-31T22:03:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:36 smithi171 conmon[41853]: debug 2022-01-31T22:03:36.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:36.284478+0000) 2022-01-31T22:03:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:36 smithi167 conmon[54076]: debug 2022-01-31T22:03:36.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:36.573829+0000) 2022-01-31T22:03:36.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:36 smithi167 conmon[60316]: debug 2022-01-31T22:03:36.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:36.573829+0000) 2022-01-31T22:03:37.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:36 smithi167 conmon[49112]: debug 2022-01-31T22:03:36.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:36.910117+0000) 2022-01-31T22:03:37.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:37 smithi171 conmon[46715]: debug 2022-01-31T22:03:37.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:37.077952+0000) 2022-01-31T22:03:37.240 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:36 smithi171 conmon[51620]: debug 2022-01-31T22:03:36.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:36.928176+0000) 2022-01-31T22:03:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:37 smithi171 conmon[41853]: debug 2022-01-31T22:03:37.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:37.284635+0000) 2022-01-31T22:03:37.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:37 smithi167 conmon[54076]: debug 2022-01-31T22:03:37.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:37.574024+0000) 2022-01-31T22:03:37.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:37 smithi167 conmon[60316]: debug 2022-01-31T22:03:37.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:37.574024+0000) 2022-01-31T22:03:38.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:37 smithi167 conmon[49112]: debug 2022-01-31T22:03:37.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:37.910334+0000) 2022-01-31T22:03:38.239 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[46715]: debug 2022-01-31T22:03:38.076+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.078122+0000) 2022-01-31T22:03:38.240 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:37 smithi171 conmon[51620]: debug 2022-01-31T22:03:37.926+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:37.928351+0000) 2022-01-31T22:03:38.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[35325]: debug 2022-01-31T22:03:38.418+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179234 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:38.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[46715]: debug 2022-01-31T22:03:38.399+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.400872+0000) 2022-01-31T22:03:38.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[51620]: debug 2022-01-31T22:03:38.398+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.400730+0000) 2022-01-31T22:03:38.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[41853]: debug 2022-01-31T22:03:38.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.284841+0000) 2022-01-31T22:03:38.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[41853]: debug 2022-01-31T22:03:38.400+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.402249+0000) 2022-01-31T22:03:38.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:38 smithi167 conmon[49112]: debug 2022-01-31T22:03:38.401+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.402241+0000) 2022-01-31T22:03:38.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:38 smithi167 conmon[54076]: debug 2022-01-31T22:03:38.400+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.401375+0000) 2022-01-31T22:03:38.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:38 smithi167 conmon[54076]: debug 2022-01-31T22:03:38.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.574242+0000) 2022-01-31T22:03:38.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:38 smithi167 conmon[60316]: debug 2022-01-31T22:03:38.400+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.401241+0000) 2022-01-31T22:03:38.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:38 smithi167 conmon[60316]: debug 2022-01-31T22:03:38.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.574242+0000) 2022-01-31T22:03:39.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:38 smithi167 conmon[49112]: debug 2022-01-31T22:03:38.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.910565+0000) 2022-01-31T22:03:39.077 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[35325]: debug 2022-01-31T22:03:38.784+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:03:39.077 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:38 smithi171 conmon[51620]: debug 2022-01-31T22:03:38.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:38.928540+0000) 2022-01-31T22:03:39.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:39 smithi171 conmon[41853]: debug 2022-01-31T22:03:39.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:39.285023+0000) 2022-01-31T22:03:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:39 smithi171 conmon[46715]: debug 2022-01-31T22:03:39.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:39.078283+0000) 2022-01-31T22:03:39.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:39 smithi167 conmon[60316]: debug 2022-01-31T22:03:39.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:39.574391+0000) 2022-01-31T22:03:39.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:39 smithi167 conmon[54076]: debug 2022-01-31T22:03:39.573+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:39.574392+0000) 2022-01-31T22:03:40.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:39 smithi167 conmon[49112]: debug 2022-01-31T22:03:39.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:39.910677+0000) 2022-01-31T22:03:40.240 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:40 smithi171 conmon[46715]: debug 2022-01-31T22:03:40.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:40.078493+0000) 2022-01-31T22:03:40.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:39 smithi171 conmon[51620]: debug 2022-01-31T22:03:39.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:39.928712+0000) 2022-01-31T22:03:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:40 smithi171 conmon[41853]: debug 2022-01-31T22:03:40.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:40.285138+0000) 2022-01-31T22:03:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:40 smithi167 conmon[60316]: debug 2022-01-31T22:03:40.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:40.574614+0000) 2022-01-31T22:03:40.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:40 smithi167 conmon[54076]: debug 2022-01-31T22:03:40.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:40.574614+0000) 2022-01-31T22:03:41.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:40 smithi167 conmon[49112]: debug 2022-01-31T22:03:40.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:40.910863+0000) 2022-01-31T22:03:41.240 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:41 smithi171 conmon[46715]: debug 2022-01-31T22:03:41.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:41.078683+0000) 2022-01-31T22:03:41.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:40 smithi171 conmon[51620]: debug 2022-01-31T22:03:40.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:40.928795+0000) 2022-01-31T22:03:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:41 smithi171 conmon[41853]: debug 2022-01-31T22:03:41.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:41.285298+0000) 2022-01-31T22:03:41.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:41 smithi167 conmon[60316]: debug 2022-01-31T22:03:41.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:41.574851+0000) 2022-01-31T22:03:41.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:41 smithi167 conmon[54076]: debug 2022-01-31T22:03:41.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:41.574851+0000) 2022-01-31T22:03:42.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:41 smithi167 conmon[49112]: debug 2022-01-31T22:03:41.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:41.911063+0000) 2022-01-31T22:03:42.240 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:42 smithi171 conmon[46715]: debug 2022-01-31T22:03:42.077+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:42.078847+0000) 2022-01-31T22:03:42.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:41 smithi171 conmon[51620]: debug 2022-01-31T22:03:41.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:41.928981+0000) 2022-01-31T22:03:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:42 smithi171 conmon[41853]: debug 2022-01-31T22:03:42.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:42.285477+0000) 2022-01-31T22:03:42.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:42 smithi167 conmon[60316]: debug 2022-01-31T22:03:42.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:42.575034+0000) 2022-01-31T22:03:42.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:42 smithi167 conmon[54076]: debug 2022-01-31T22:03:42.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:42.575033+0000) 2022-01-31T22:03:43.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:42 smithi167 conmon[49112]: debug 2022-01-31T22:03:42.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:42.911243+0000) 2022-01-31T22:03:43.240 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[46715]: debug 2022-01-31T22:03:43.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.079022+0000) 2022-01-31T22:03:43.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:42 smithi171 conmon[51620]: debug 2022-01-31T22:03:42.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:42.929174+0000) 2022-01-31T22:03:43.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[46715]: debug 2022-01-31T22:03:43.421+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.423164+0000) 2022-01-31T22:03:43.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[51620]: debug 2022-01-31T22:03:43.421+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.423252+0000) 2022-01-31T22:03:43.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[35325]: debug 2022-01-31T22:03:43.440+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179347 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:43.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[41853]: debug 2022-01-31T22:03:43.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.285710+0000) 2022-01-31T22:03:43.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[41853]: debug 2022-01-31T22:03:43.421+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.423679+0000) 2022-01-31T22:03:43.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:43 smithi167 conmon[49112]: debug 2022-01-31T22:03:43.423+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.423697+0000) 2022-01-31T22:03:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:43 smithi167 conmon[60316]: debug 2022-01-31T22:03:43.423+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.424424+0000) 2022-01-31T22:03:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:43 smithi167 conmon[60316]: debug 2022-01-31T22:03:43.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.575248+0000) 2022-01-31T22:03:43.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:43 smithi167 conmon[54076]: debug 2022-01-31T22:03:43.422+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.423288+0000) 2022-01-31T22:03:43.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:43 smithi167 conmon[54076]: debug 2022-01-31T22:03:43.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.575248+0000) 2022-01-31T22:03:44.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:43 smithi167 conmon[49112]: debug 2022-01-31T22:03:43.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.911372+0000) 2022-01-31T22:03:44.241 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:44 smithi171 conmon[46715]: debug 2022-01-31T22:03:44.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:44.079191+0000) 2022-01-31T22:03:44.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:43 smithi171 conmon[51620]: debug 2022-01-31T22:03:43.927+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:43.929383+0000) 2022-01-31T22:03:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:44 smithi171 conmon[41853]: debug 2022-01-31T22:03:44.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:44.285941+0000) 2022-01-31T22:03:44.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:44 smithi167 conmon[54076]: debug 2022-01-31T22:03:44.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:44.575435+0000) 2022-01-31T22:03:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:44 smithi167 conmon[60316]: debug 2022-01-31T22:03:44.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:44.575435+0000) 2022-01-31T22:03:45.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:44 smithi167 conmon[49112]: debug 2022-01-31T22:03:44.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:44.911540+0000) 2022-01-31T22:03:45.241 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:45 smithi171 conmon[46715]: debug 2022-01-31T22:03:45.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:45.079325+0000) 2022-01-31T22:03:45.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:44 smithi171 conmon[51620]: debug 2022-01-31T22:03:44.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:44.929556+0000) 2022-01-31T22:03:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:45 smithi171 conmon[41853]: debug 2022-01-31T22:03:45.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:45.286050+0000) 2022-01-31T22:03:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:45 smithi167 conmon[54076]: debug 2022-01-31T22:03:45.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:45.575634+0000) 2022-01-31T22:03:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:45 smithi167 conmon[60316]: debug 2022-01-31T22:03:45.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:45.575628+0000) 2022-01-31T22:03:46.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:45 smithi167 conmon[49112]: debug 2022-01-31T22:03:45.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:45.911695+0000) 2022-01-31T22:03:46.241 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:46 smithi171 conmon[46715]: debug 2022-01-31T22:03:46.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:46.079532+0000) 2022-01-31T22:03:46.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:45 smithi171 conmon[51620]: debug 2022-01-31T22:03:45.928+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:45.929748+0000) 2022-01-31T22:03:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:46 smithi171 conmon[41853]: debug 2022-01-31T22:03:46.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:46.286179+0000) 2022-01-31T22:03:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:46 smithi167 conmon[54076]: debug 2022-01-31T22:03:46.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:46.575847+0000) 2022-01-31T22:03:46.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:46 smithi167 conmon[60316]: debug 2022-01-31T22:03:46.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:46.575847+0000) 2022-01-31T22:03:47.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:46 smithi167 conmon[49112]: debug 2022-01-31T22:03:46.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:46.911888+0000) 2022-01-31T22:03:47.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:46 smithi171 conmon[51620]: debug 2022-01-31T22:03:46.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:46.929991+0000) 2022-01-31T22:03:47.242 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:47 smithi171 conmon[46715]: debug 2022-01-31T22:03:47.078+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:47.079715+0000) 2022-01-31T22:03:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:47 smithi171 conmon[41853]: debug 2022-01-31T22:03:47.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:47.286339+0000) 2022-01-31T22:03:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:47 smithi167 conmon[54076]: debug 2022-01-31T22:03:47.574+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:47.576054+0000) 2022-01-31T22:03:47.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:47 smithi167 conmon[60316]: debug 2022-01-31T22:03:47.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:47.576054+0000) 2022-01-31T22:03:48.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:47 smithi167 conmon[49112]: debug 2022-01-31T22:03:47.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:47.912040+0000) 2022-01-31T22:03:48.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:47 smithi171 conmon[51620]: debug 2022-01-31T22:03:47.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:47.930189+0000) 2022-01-31T22:03:48.242 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[46715]: debug 2022-01-31T22:03:48.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.079924+0000) 2022-01-31T22:03:48.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[46715]: debug 2022-01-31T22:03:48.445+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.445953+0000) 2022-01-31T22:03:48.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[35325]: debug 2022-01-31T22:03:48.463+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179460 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:48.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[41853]: debug 2022-01-31T22:03:48.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.286491+0000) 2022-01-31T22:03:48.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[41853]: debug 2022-01-31T22:03:48.443+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.444741+0000) 2022-01-31T22:03:48.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[51620]: debug 2022-01-31T22:03:48.443+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.444162+0000) 2022-01-31T22:03:48.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:48 smithi167 conmon[49112]: debug 2022-01-31T22:03:48.444+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.445719+0000) 2022-01-31T22:03:48.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:48 smithi167 conmon[54076]: debug 2022-01-31T22:03:48.443+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.444874+0000) 2022-01-31T22:03:48.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:48 smithi167 conmon[54076]: debug 2022-01-31T22:03:48.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.576249+0000) 2022-01-31T22:03:48.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:48 smithi167 conmon[60316]: debug 2022-01-31T22:03:48.443+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.444548+0000) 2022-01-31T22:03:48.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:48 smithi167 conmon[60316]: debug 2022-01-31T22:03:48.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.576249+0000) 2022-01-31T22:03:49.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:48 smithi167 conmon[49112]: debug 2022-01-31T22:03:48.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.912257+0000) 2022-01-31T22:03:49.241 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:48 smithi171 conmon[51620]: debug 2022-01-31T22:03:48.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:48.930354+0000) 2022-01-31T22:03:49.242 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:49 smithi171 conmon[46715]: debug 2022-01-31T22:03:49.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:49.080101+0000) 2022-01-31T22:03:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:49 smithi171 conmon[41853]: debug 2022-01-31T22:03:49.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:49.286657+0000) 2022-01-31T22:03:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:49 smithi167 conmon[54076]: debug 2022-01-31T22:03:49.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:49.576415+0000) 2022-01-31T22:03:49.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:49 smithi167 conmon[60316]: debug 2022-01-31T22:03:49.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:49.576415+0000) 2022-01-31T22:03:50.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:49 smithi167 conmon[49112]: debug 2022-01-31T22:03:49.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:49.912447+0000) 2022-01-31T22:03:50.242 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:49 smithi171 conmon[51620]: debug 2022-01-31T22:03:49.929+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:49.930553+0000) 2022-01-31T22:03:50.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:50 smithi171 conmon[46715]: debug 2022-01-31T22:03:50.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:50.080262+0000) 2022-01-31T22:03:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:50 smithi171 conmon[41853]: debug 2022-01-31T22:03:50.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:50.286818+0000) 2022-01-31T22:03:50.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:50 smithi167 conmon[54076]: debug 2022-01-31T22:03:50.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:50.576565+0000) 2022-01-31T22:03:50.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:50 smithi167 conmon[60316]: debug 2022-01-31T22:03:50.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:50.576566+0000) 2022-01-31T22:03:51.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:50 smithi167 conmon[49112]: debug 2022-01-31T22:03:50.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:50.912695+0000) 2022-01-31T22:03:51.242 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:51 smithi171 conmon[46715]: debug 2022-01-31T22:03:51.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:51.080446+0000) 2022-01-31T22:03:51.243 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:50 smithi171 conmon[51620]: debug 2022-01-31T22:03:50.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:50.930749+0000) 2022-01-31T22:03:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:51 smithi171 conmon[41853]: debug 2022-01-31T22:03:51.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:51.286994+0000) 2022-01-31T22:03:51.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:51 smithi167 conmon[54076]: debug 2022-01-31T22:03:51.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:51.576758+0000) 2022-01-31T22:03:51.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:51 smithi167 conmon[60316]: debug 2022-01-31T22:03:51.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:51.576757+0000) 2022-01-31T22:03:52.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:51 smithi167 conmon[49112]: debug 2022-01-31T22:03:51.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:51.912901+0000) 2022-01-31T22:03:52.242 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:51 smithi171 conmon[51620]: debug 2022-01-31T22:03:51.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:51.930937+0000) 2022-01-31T22:03:52.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:52 smithi171 conmon[46715]: debug 2022-01-31T22:03:52.079+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:52.080629+0000) 2022-01-31T22:03:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:52 smithi171 conmon[41853]: debug 2022-01-31T22:03:52.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:52.287168+0000) 2022-01-31T22:03:52.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:52 smithi167 conmon[54076]: debug 2022-01-31T22:03:52.575+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:52.576963+0000) 2022-01-31T22:03:52.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:52 smithi167 conmon[60316]: debug 2022-01-31T22:03:52.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:52.576964+0000) 2022-01-31T22:03:53.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:52 smithi167 conmon[49112]: debug 2022-01-31T22:03:52.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:52.913103+0000) 2022-01-31T22:03:53.242 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[46715]: debug 2022-01-31T22:03:53.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.080823+0000) 2022-01-31T22:03:53.243 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:52 smithi171 conmon[51620]: debug 2022-01-31T22:03:52.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:52.931129+0000) 2022-01-31T22:03:53.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[46715]: debug 2022-01-31T22:03:53.467+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.468253+0000) 2022-01-31T22:03:53.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[51620]: debug 2022-01-31T22:03:53.466+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.467644+0000) 2022-01-31T22:03:53.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[35325]: debug 2022-01-31T22:03:53.485+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179571 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:53.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[41853]: debug 2022-01-31T22:03:53.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.287358+0000) 2022-01-31T22:03:53.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[41853]: debug 2022-01-31T22:03:53.466+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.467575+0000) 2022-01-31T22:03:53.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:53 smithi167 conmon[49112]: debug 2022-01-31T22:03:53.466+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.468205+0000) 2022-01-31T22:03:53.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:53 smithi167 conmon[54076]: debug 2022-01-31T22:03:53.465+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.467431+0000) 2022-01-31T22:03:53.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:53 smithi167 conmon[54076]: debug 2022-01-31T22:03:53.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.577195+0000) 2022-01-31T22:03:53.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:53 smithi167 conmon[60316]: debug 2022-01-31T22:03:53.465+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.467324+0000) 2022-01-31T22:03:53.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:53 smithi167 conmon[60316]: debug 2022-01-31T22:03:53.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.577195+0000) 2022-01-31T22:03:54.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:53 smithi167 conmon[49112]: debug 2022-01-31T22:03:53.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.913260+0000) 2022-01-31T22:03:54.080 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[51620]: debug 2022-01-31T22:03:53.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:53.931288+0000) 2022-01-31T22:03:54.080 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:53 smithi171 conmon[35325]: debug 2022-01-31T22:03:53.785+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:03:54.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:54 smithi171 conmon[46715]: debug 2022-01-31T22:03:54.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:54.081017+0000) 2022-01-31T22:03:54.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:54 smithi171 conmon[41853]: debug 2022-01-31T22:03:54.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:54.287553+0000) 2022-01-31T22:03:54.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:54 smithi167 conmon[54076]: debug 2022-01-31T22:03:54.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:54.577359+0000) 2022-01-31T22:03:54.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:54 smithi167 conmon[60316]: debug 2022-01-31T22:03:54.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:54.577359+0000) 2022-01-31T22:03:55.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:54 smithi167 conmon[49112]: debug 2022-01-31T22:03:54.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:54.913406+0000) 2022-01-31T22:03:55.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:55 smithi171 conmon[46715]: debug 2022-01-31T22:03:55.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:55.081173+0000) 2022-01-31T22:03:55.243 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:54 smithi171 conmon[51620]: debug 2022-01-31T22:03:54.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:54.931439+0000) 2022-01-31T22:03:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:55 smithi171 conmon[41853]: debug 2022-01-31T22:03:55.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:55.287687+0000) 2022-01-31T22:03:55.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:55 smithi167 conmon[54076]: debug 2022-01-31T22:03:55.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:55.577552+0000) 2022-01-31T22:03:55.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:55 smithi167 conmon[60316]: debug 2022-01-31T22:03:55.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:55.577553+0000) 2022-01-31T22:03:56.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:55 smithi167 conmon[49112]: debug 2022-01-31T22:03:55.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:55.913619+0000) 2022-01-31T22:03:56.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:56 smithi171 conmon[46715]: debug 2022-01-31T22:03:56.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:56.081364+0000) 2022-01-31T22:03:56.243 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:55 smithi171 conmon[51620]: debug 2022-01-31T22:03:55.930+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:55.931624+0000) 2022-01-31T22:03:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:56 smithi171 conmon[41853]: debug 2022-01-31T22:03:56.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:56.287830+0000) 2022-01-31T22:03:56.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:56 smithi167 conmon[54076]: debug 2022-01-31T22:03:56.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:56.577763+0000) 2022-01-31T22:03:56.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:56 smithi167 conmon[60316]: debug 2022-01-31T22:03:56.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:56.577763+0000) 2022-01-31T22:03:57.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:56 smithi167 conmon[49112]: debug 2022-01-31T22:03:56.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:56.913777+0000) 2022-01-31T22:03:57.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:57 smithi171 conmon[46715]: debug 2022-01-31T22:03:57.080+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:57.081527+0000) 2022-01-31T22:03:57.244 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:56 smithi171 conmon[51620]: debug 2022-01-31T22:03:56.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:56.931811+0000) 2022-01-31T22:03:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:57 smithi171 conmon[41853]: debug 2022-01-31T22:03:57.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:57.287994+0000) 2022-01-31T22:03:57.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:57 smithi167 conmon[54076]: debug 2022-01-31T22:03:57.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:57.577947+0000) 2022-01-31T22:03:57.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:57 smithi167 conmon[60316]: debug 2022-01-31T22:03:57.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:57.577948+0000) 2022-01-31T22:03:58.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:57 smithi167 conmon[49112]: debug 2022-01-31T22:03:57.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:57.913952+0000) 2022-01-31T22:03:58.243 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:57 smithi171 conmon[51620]: debug 2022-01-31T22:03:57.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:57.931995+0000) 2022-01-31T22:03:58.244 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[46715]: debug 2022-01-31T22:03:58.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.081736+0000) 2022-01-31T22:03:58.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[46715]: debug 2022-01-31T22:03:58.486+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.487724+0000) 2022-01-31T22:03:58.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[51620]: debug 2022-01-31T22:03:58.488+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.489214+0000) 2022-01-31T22:03:58.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[41853]: debug 2022-01-31T22:03:58.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.288136+0000) 2022-01-31T22:03:58.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[41853]: debug 2022-01-31T22:03:58.487+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.487981+0000) 2022-01-31T22:03:58.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:58 smithi167 conmon[49112]: debug 2022-01-31T22:03:58.487+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.489444+0000) 2022-01-31T22:03:58.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:58 smithi167 conmon[60316]: debug 2022-01-31T22:03:58.487+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.489208+0000) 2022-01-31T22:03:58.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:58 smithi167 conmon[60316]: debug 2022-01-31T22:03:58.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.578152+0000) 2022-01-31T22:03:58.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:58 smithi167 conmon[54076]: debug 2022-01-31T22:03:58.487+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.488730+0000) 2022-01-31T22:03:58.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:58 smithi167 conmon[54076]: debug 2022-01-31T22:03:58.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.578152+0000) 2022-01-31T22:03:58.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[35325]: debug 2022-01-31T22:03:58.505+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179682 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:03:59.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:58 smithi167 conmon[49112]: debug 2022-01-31T22:03:58.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.914114+0000) 2022-01-31T22:03:59.243 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:03:59 smithi171 conmon[46715]: debug 2022-01-31T22:03:59.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:59.081938+0000) 2022-01-31T22:03:59.244 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:58 smithi171 conmon[51620]: debug 2022-01-31T22:03:58.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:58.932194+0000) 2022-01-31T22:03:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:03:59 smithi171 conmon[41853]: debug 2022-01-31T22:03:59.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:59.288252+0000) 2022-01-31T22:03:59.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:03:59 smithi167 conmon[54076]: debug 2022-01-31T22:03:59.576+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:59.578318+0000) 2022-01-31T22:03:59.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:03:59 smithi167 conmon[60316]: debug 2022-01-31T22:03:59.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:59.578318+0000) 2022-01-31T22:04:00.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:03:59 smithi167 conmon[49112]: debug 2022-01-31T22:03:59.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:59.914265+0000) 2022-01-31T22:04:00.244 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:00 smithi171 conmon[46715]: debug 2022-01-31T22:04:00.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:00.082064+0000) 2022-01-31T22:04:00.244 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:03:59 smithi171 conmon[51620]: debug 2022-01-31T22:03:59.931+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:03:59.932326+0000) 2022-01-31T22:04:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:00 smithi171 conmon[41853]: debug 2022-01-31T22:04:00.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:00.288358+0000) 2022-01-31T22:04:00.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:00 smithi167 conmon[54076]: debug 2022-01-31T22:04:00.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:00.578535+0000) 2022-01-31T22:04:00.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:00 smithi167 conmon[60316]: debug 2022-01-31T22:04:00.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:00.578535+0000) 2022-01-31T22:04:01.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:00 smithi167 conmon[49112]: debug 2022-01-31T22:04:00.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:00.914448+0000) 2022-01-31T22:04:01.244 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:01 smithi171 conmon[46715]: debug 2022-01-31T22:04:01.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:01.082218+0000) 2022-01-31T22:04:01.245 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:00 smithi171 conmon[51620]: debug 2022-01-31T22:04:00.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:00.932547+0000) 2022-01-31T22:04:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:01 smithi171 conmon[41853]: debug 2022-01-31T22:04:01.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:01.288509+0000) 2022-01-31T22:04:01.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:01 smithi167 conmon[54076]: debug 2022-01-31T22:04:01.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:01.578744+0000) 2022-01-31T22:04:01.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:01 smithi167 conmon[60316]: debug 2022-01-31T22:04:01.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:01.578744+0000) 2022-01-31T22:04:02.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:01 smithi171 conmon[51620]: debug 2022-01-31T22:04:01.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:01.932722+0000) 2022-01-31T22:04:02.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:01 smithi167 conmon[49112]: debug 2022-01-31T22:04:01.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:01.914639+0000) 2022-01-31T22:04:02.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:02 smithi171 conmon[46715]: debug 2022-01-31T22:04:02.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:02.082355+0000) 2022-01-31T22:04:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:02 smithi171 conmon[41853]: debug 2022-01-31T22:04:02.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:02.288638+0000) 2022-01-31T22:04:02.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:02 smithi167 conmon[54076]: debug 2022-01-31T22:04:02.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:02.578923+0000) 2022-01-31T22:04:02.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:02 smithi167 conmon[60316]: debug 2022-01-31T22:04:02.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:02.578923+0000) 2022-01-31T22:04:03.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:02 smithi167 conmon[49112]: debug 2022-01-31T22:04:02.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:02.914810+0000) 2022-01-31T22:04:03.244 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[46715]: debug 2022-01-31T22:04:03.081+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.082505+0000) 2022-01-31T22:04:03.245 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:02 smithi171 conmon[51620]: debug 2022-01-31T22:04:02.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:02.932897+0000) 2022-01-31T22:04:03.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[41853]: debug 2022-01-31T22:04:03.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.288823+0000) 2022-01-31T22:04:03.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[41853]: debug 2022-01-31T22:04:03.507+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.508690+0000) 2022-01-31T22:04:03.739 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:03 smithi167 conmon[49112]: debug 2022-01-31T22:04:03.507+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.509446+0000) 2022-01-31T22:04:03.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:03 smithi167 conmon[54076]: debug 2022-01-31T22:04:03.508+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.510010+0000) 2022-01-31T22:04:03.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:03 smithi167 conmon[54076]: debug 2022-01-31T22:04:03.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.579110+0000) 2022-01-31T22:04:03.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:03 smithi167 conmon[60316]: debug 2022-01-31T22:04:03.507+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.508823+0000) 2022-01-31T22:04:03.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:03 smithi167 conmon[60316]: debug 2022-01-31T22:04:03.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.579111+0000) 2022-01-31T22:04:03.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[35325]: debug 2022-01-31T22:04:03.527+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179793 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:03.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[46715]: debug 2022-01-31T22:04:03.508+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.508893+0000) 2022-01-31T22:04:03.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[51620]: debug 2022-01-31T22:04:03.509+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.509902+0000) 2022-01-31T22:04:04.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:03 smithi167 conmon[49112]: debug 2022-01-31T22:04:03.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.914994+0000) 2022-01-31T22:04:04.244 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:04 smithi171 conmon[46715]: debug 2022-01-31T22:04:04.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:04.082700+0000) 2022-01-31T22:04:04.245 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:03 smithi171 conmon[51620]: debug 2022-01-31T22:04:03.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:03.933062+0000) 2022-01-31T22:04:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:04 smithi171 conmon[41853]: debug 2022-01-31T22:04:04.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:04.288999+0000) 2022-01-31T22:04:04.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:04 smithi167 conmon[60316]: debug 2022-01-31T22:04:04.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:04.579274+0000) 2022-01-31T22:04:04.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:04 smithi167 conmon[54076]: debug 2022-01-31T22:04:04.577+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:04.579274+0000) 2022-01-31T22:04:05.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:04 smithi167 conmon[49112]: debug 2022-01-31T22:04:04.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:04.915172+0000) 2022-01-31T22:04:05.245 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:05 smithi171 conmon[46715]: debug 2022-01-31T22:04:05.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:05.082863+0000) 2022-01-31T22:04:05.245 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:04 smithi171 conmon[51620]: debug 2022-01-31T22:04:04.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:04.933261+0000) 2022-01-31T22:04:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:05 smithi171 conmon[41853]: debug 2022-01-31T22:04:05.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:05.289104+0000) 2022-01-31T22:04:05.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:05 smithi167 conmon[60316]: debug 2022-01-31T22:04:05.578+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:05.579531+0000) 2022-01-31T22:04:05.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:05 smithi167 conmon[54076]: debug 2022-01-31T22:04:05.578+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:05.579531+0000) 2022-01-31T22:04:06.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:05 smithi167 conmon[49112]: debug 2022-01-31T22:04:05.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:05.915373+0000) 2022-01-31T22:04:06.245 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:06 smithi171 conmon[46715]: debug 2022-01-31T22:04:06.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:06.083045+0000) 2022-01-31T22:04:06.246 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:05 smithi171 conmon[51620]: debug 2022-01-31T22:04:05.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:05.933455+0000) 2022-01-31T22:04:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:06 smithi171 conmon[41853]: debug 2022-01-31T22:04:06.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:06.289299+0000) 2022-01-31T22:04:06.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:06 smithi167 conmon[60316]: debug 2022-01-31T22:04:06.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:06.579704+0000) 2022-01-31T22:04:06.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:06 smithi167 conmon[54076]: debug 2022-01-31T22:04:06.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:06.579704+0000) 2022-01-31T22:04:07.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:06 smithi167 conmon[49112]: debug 2022-01-31T22:04:06.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:06.915572+0000) 2022-01-31T22:04:07.245 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:06 smithi171 conmon[51620]: debug 2022-01-31T22:04:06.932+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:06.933643+0000) 2022-01-31T22:04:07.246 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:07 smithi171 conmon[46715]: debug 2022-01-31T22:04:07.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:07.083204+0000) 2022-01-31T22:04:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:07 smithi171 conmon[41853]: debug 2022-01-31T22:04:07.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:07.289503+0000) 2022-01-31T22:04:07.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:07 smithi167 conmon[54076]: debug 2022-01-31T22:04:07.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:07.579924+0000) 2022-01-31T22:04:07.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:07 smithi167 conmon[60316]: debug 2022-01-31T22:04:07.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:07.579924+0000) 2022-01-31T22:04:08.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:07 smithi167 conmon[49112]: debug 2022-01-31T22:04:07.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:07.915726+0000) 2022-01-31T22:04:08.245 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[46715]: debug 2022-01-31T22:04:08.082+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.083377+0000) 2022-01-31T22:04:08.245 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:07 smithi171 conmon[51620]: debug 2022-01-31T22:04:07.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:07.933849+0000) 2022-01-31T22:04:08.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[41853]: debug 2022-01-31T22:04:08.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.289692+0000) 2022-01-31T22:04:08.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:08 smithi167 conmon[49112]: debug 2022-01-31T22:04:08.529+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.530593+0000) 2022-01-31T22:04:08.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:08 smithi167 conmon[60316]: debug 2022-01-31T22:04:08.529+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.530667+0000) 2022-01-31T22:04:08.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:08 smithi167 conmon[60316]: debug 2022-01-31T22:04:08.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.580106+0000) 2022-01-31T22:04:08.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:08 smithi167 conmon[54076]: debug 2022-01-31T22:04:08.529+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.531130+0000) 2022-01-31T22:04:08.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:08 smithi167 conmon[54076]: debug 2022-01-31T22:04:08.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.580105+0000) 2022-01-31T22:04:08.786 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[35325]: debug 2022-01-31T22:04:08.548+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 179905 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:08.786 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[41853]: debug 2022-01-31T22:04:08.528+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.529703+0000) 2022-01-31T22:04:08.787 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[46715]: debug 2022-01-31T22:04:08.530+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.530943+0000) 2022-01-31T22:04:08.787 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[51620]: debug 2022-01-31T22:04:08.531+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.531802+0000) 2022-01-31T22:04:09.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:08 smithi167 conmon[49112]: debug 2022-01-31T22:04:08.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.915888+0000) 2022-01-31T22:04:09.082 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[51620]: debug 2022-01-31T22:04:08.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:08.934038+0000) 2022-01-31T22:04:09.083 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:08 smithi171 conmon[35325]: debug 2022-01-31T22:04:08.786+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:04:09.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:09 smithi171 conmon[41853]: debug 2022-01-31T22:04:09.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:09.289871+0000) 2022-01-31T22:04:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:09 smithi171 conmon[46715]: debug 2022-01-31T22:04:09.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:09.083558+0000) 2022-01-31T22:04:09.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:09 smithi167 conmon[54076]: debug 2022-01-31T22:04:09.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:09.580233+0000) 2022-01-31T22:04:09.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:09 smithi167 conmon[60316]: debug 2022-01-31T22:04:09.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:09.580233+0000) 2022-01-31T22:04:10.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:09 smithi167 conmon[49112]: debug 2022-01-31T22:04:09.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:09.916065+0000) 2022-01-31T22:04:10.245 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:10 smithi171 conmon[46715]: debug 2022-01-31T22:04:10.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:10.083705+0000) 2022-01-31T22:04:10.246 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:09 smithi171 conmon[51620]: debug 2022-01-31T22:04:09.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:09.934240+0000) 2022-01-31T22:04:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:10 smithi171 conmon[41853]: debug 2022-01-31T22:04:10.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:10.290064+0000) 2022-01-31T22:04:10.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:10 smithi167 conmon[54076]: debug 2022-01-31T22:04:10.579+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:10.580363+0000) 2022-01-31T22:04:10.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:10 smithi167 conmon[60316]: debug 2022-01-31T22:04:10.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:10.580362+0000) 2022-01-31T22:04:11.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:10 smithi167 conmon[49112]: debug 2022-01-31T22:04:10.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:10.916259+0000) 2022-01-31T22:04:11.246 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:11 smithi171 conmon[46715]: debug 2022-01-31T22:04:11.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:11.083922+0000) 2022-01-31T22:04:11.246 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:10 smithi171 conmon[51620]: debug 2022-01-31T22:04:10.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:10.934382+0000) 2022-01-31T22:04:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:11 smithi171 conmon[41853]: debug 2022-01-31T22:04:11.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:11.290239+0000) 2022-01-31T22:04:11.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:11 smithi167 conmon[54076]: debug 2022-01-31T22:04:11.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:11.580550+0000) 2022-01-31T22:04:11.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:11 smithi167 conmon[60316]: debug 2022-01-31T22:04:11.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:11.580550+0000) 2022-01-31T22:04:12.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:11 smithi167 conmon[49112]: debug 2022-01-31T22:04:11.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:11.916445+0000) 2022-01-31T22:04:12.246 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:12 smithi171 conmon[46715]: debug 2022-01-31T22:04:12.083+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:12.084080+0000) 2022-01-31T22:04:12.404 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:11 smithi171 conmon[51620]: debug 2022-01-31T22:04:11.933+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:11.934547+0000) 2022-01-31T22:04:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:12 smithi171 conmon[41853]: debug 2022-01-31T22:04:12.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:12.290446+0000) 2022-01-31T22:04:12.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:12 smithi167 conmon[54076]: debug 2022-01-31T22:04:12.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:12.580681+0000) 2022-01-31T22:04:12.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:12 smithi167 conmon[60316]: debug 2022-01-31T22:04:12.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:12.580681+0000) 2022-01-31T22:04:13.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:12 smithi167 conmon[49112]: debug 2022-01-31T22:04:12.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:12.916615+0000) 2022-01-31T22:04:13.246 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[46715]: debug 2022-01-31T22:04:13.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.084256+0000) 2022-01-31T22:04:13.247 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:12 smithi171 conmon[51620]: debug 2022-01-31T22:04:12.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:12.934723+0000) 2022-01-31T22:04:13.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[51620]: debug 2022-01-31T22:04:13.550+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.551071+0000) 2022-01-31T22:04:13.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[41853]: debug 2022-01-31T22:04:13.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.290614+0000) 2022-01-31T22:04:13.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[41853]: debug 2022-01-31T22:04:13.550+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.551335+0000) 2022-01-31T22:04:13.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:13 smithi167 conmon[49112]: debug 2022-01-31T22:04:13.551+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.552161+0000) 2022-01-31T22:04:13.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:13 smithi167 conmon[54076]: debug 2022-01-31T22:04:13.551+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.551543+0000) 2022-01-31T22:04:13.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:13 smithi167 conmon[54076]: debug 2022-01-31T22:04:13.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.580831+0000) 2022-01-31T22:04:13.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:13 smithi167 conmon[60316]: debug 2022-01-31T22:04:13.551+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.551789+0000) 2022-01-31T22:04:13.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:13 smithi167 conmon[60316]: debug 2022-01-31T22:04:13.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.580830+0000) 2022-01-31T22:04:13.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[35325]: debug 2022-01-31T22:04:13.570+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180016 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:13.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[46715]: debug 2022-01-31T22:04:13.550+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.551011+0000) 2022-01-31T22:04:14.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:13 smithi167 conmon[49112]: debug 2022-01-31T22:04:13.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.916798+0000) 2022-01-31T22:04:14.246 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:13 smithi171 conmon[51620]: debug 2022-01-31T22:04:13.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:13.934943+0000) 2022-01-31T22:04:14.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:14 smithi171 conmon[46715]: debug 2022-01-31T22:04:14.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:14.084425+0000) 2022-01-31T22:04:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:14 smithi171 conmon[41853]: debug 2022-01-31T22:04:14.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:14.290792+0000) 2022-01-31T22:04:14.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:14 smithi167 conmon[54076]: debug 2022-01-31T22:04:14.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:14.580997+0000) 2022-01-31T22:04:14.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:14 smithi167 conmon[60316]: debug 2022-01-31T22:04:14.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:14.580993+0000) 2022-01-31T22:04:15.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:14 smithi167 conmon[49112]: debug 2022-01-31T22:04:14.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:14.916963+0000) 2022-01-31T22:04:15.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:15 smithi171 conmon[46715]: debug 2022-01-31T22:04:15.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:15.084553+0000) 2022-01-31T22:04:15.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:14 smithi171 conmon[51620]: debug 2022-01-31T22:04:14.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:14.935110+0000) 2022-01-31T22:04:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:15 smithi171 conmon[41853]: debug 2022-01-31T22:04:15.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:15.290943+0000) 2022-01-31T22:04:15.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:15 smithi167 conmon[54076]: debug 2022-01-31T22:04:15.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:15.581180+0000) 2022-01-31T22:04:15.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:15 smithi167 conmon[60316]: debug 2022-01-31T22:04:15.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:15.581201+0000) 2022-01-31T22:04:16.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:15 smithi167 conmon[49112]: debug 2022-01-31T22:04:15.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:15.917146+0000) 2022-01-31T22:04:16.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:16 smithi171 conmon[46715]: debug 2022-01-31T22:04:16.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:16.084731+0000) 2022-01-31T22:04:16.247 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:15 smithi171 conmon[51620]: debug 2022-01-31T22:04:15.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:15.935267+0000) 2022-01-31T22:04:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:16 smithi171 conmon[41853]: debug 2022-01-31T22:04:16.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:16.291132+0000) 2022-01-31T22:04:16.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:16 smithi167 conmon[54076]: debug 2022-01-31T22:04:16.580+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:16.581308+0000) 2022-01-31T22:04:16.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:16 smithi167 conmon[60316]: debug 2022-01-31T22:04:16.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:16.581351+0000) 2022-01-31T22:04:17.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:16 smithi167 conmon[49112]: debug 2022-01-31T22:04:16.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:16.917339+0000) 2022-01-31T22:04:17.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:17 smithi171 conmon[46715]: debug 2022-01-31T22:04:17.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:17.084932+0000) 2022-01-31T22:04:17.247 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:16 smithi171 conmon[51620]: debug 2022-01-31T22:04:16.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:16.935473+0000) 2022-01-31T22:04:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:17 smithi171 conmon[41853]: debug 2022-01-31T22:04:17.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:17.291309+0000) 2022-01-31T22:04:17.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:17 smithi167 conmon[54076]: debug 2022-01-31T22:04:17.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:17.581534+0000) 2022-01-31T22:04:17.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:17 smithi167 conmon[60316]: debug 2022-01-31T22:04:17.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:17.581535+0000) 2022-01-31T22:04:18.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:17 smithi167 conmon[49112]: debug 2022-01-31T22:04:17.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:17.917550+0000) 2022-01-31T22:04:18.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[46715]: debug 2022-01-31T22:04:18.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.085127+0000) 2022-01-31T22:04:18.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:17 smithi171 conmon[51620]: debug 2022-01-31T22:04:17.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:17.935658+0000) 2022-01-31T22:04:18.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[41853]: debug 2022-01-31T22:04:18.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.291497+0000) 2022-01-31T22:04:18.572 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[41853]: debug 2022-01-31T22:04:18.571+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.572657+0000) 2022-01-31T22:04:18.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:18 smithi167 conmon[49112]: debug 2022-01-31T22:04:18.574+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.575465+0000) 2022-01-31T22:04:18.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:18 smithi167 conmon[54076]: debug 2022-01-31T22:04:18.574+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.575102+0000) 2022-01-31T22:04:18.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:18 smithi167 conmon[54076]: debug 2022-01-31T22:04:18.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.581706+0000) 2022-01-31T22:04:18.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:18 smithi167 conmon[60316]: debug 2022-01-31T22:04:18.572+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.573159+0000) 2022-01-31T22:04:18.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:18 smithi167 conmon[60316]: debug 2022-01-31T22:04:18.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.581703+0000) 2022-01-31T22:04:18.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[35325]: debug 2022-01-31T22:04:18.590+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180127 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:18.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[46715]: debug 2022-01-31T22:04:18.574+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.575081+0000) 2022-01-31T22:04:18.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[51620]: debug 2022-01-31T22:04:18.571+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.572525+0000) 2022-01-31T22:04:19.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:18 smithi167 conmon[49112]: debug 2022-01-31T22:04:18.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.917764+0000) 2022-01-31T22:04:19.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:19 smithi171 conmon[46715]: debug 2022-01-31T22:04:19.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:19.085278+0000) 2022-01-31T22:04:19.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:18 smithi171 conmon[51620]: debug 2022-01-31T22:04:18.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:18.935801+0000) 2022-01-31T22:04:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:19 smithi171 conmon[41853]: debug 2022-01-31T22:04:19.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:19.291673+0000) 2022-01-31T22:04:19.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:19 smithi167 conmon[54076]: debug 2022-01-31T22:04:19.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:19.581873+0000) 2022-01-31T22:04:19.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:19 smithi167 conmon[60316]: debug 2022-01-31T22:04:19.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:19.581872+0000) 2022-01-31T22:04:20.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:19 smithi167 conmon[49112]: debug 2022-01-31T22:04:19.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:19.917886+0000) 2022-01-31T22:04:20.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:20 smithi171 conmon[46715]: debug 2022-01-31T22:04:20.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:20.085478+0000) 2022-01-31T22:04:20.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:19 smithi171 conmon[51620]: debug 2022-01-31T22:04:19.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:19.935919+0000) 2022-01-31T22:04:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:20 smithi171 conmon[41853]: debug 2022-01-31T22:04:20.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:20.291829+0000) 2022-01-31T22:04:20.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:20 smithi167 conmon[54076]: debug 2022-01-31T22:04:20.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:20.582068+0000) 2022-01-31T22:04:20.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:20 smithi167 conmon[60316]: debug 2022-01-31T22:04:20.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:20.582068+0000) 2022-01-31T22:04:21.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:20 smithi167 conmon[49112]: debug 2022-01-31T22:04:20.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:20.918042+0000) 2022-01-31T22:04:21.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:21 smithi171 conmon[46715]: debug 2022-01-31T22:04:21.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:21.085666+0000) 2022-01-31T22:04:21.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:20 smithi171 conmon[51620]: debug 2022-01-31T22:04:20.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:20.936111+0000) 2022-01-31T22:04:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:21 smithi171 conmon[41853]: debug 2022-01-31T22:04:21.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:21.292013+0000) 2022-01-31T22:04:21.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:21 smithi167 conmon[54076]: debug 2022-01-31T22:04:21.581+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:21.582253+0000) 2022-01-31T22:04:21.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:21 smithi167 conmon[60316]: debug 2022-01-31T22:04:21.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:21.582268+0000) 2022-01-31T22:04:22.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:21 smithi167 conmon[49112]: debug 2022-01-31T22:04:21.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:21.918220+0000) 2022-01-31T22:04:22.248 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:22 smithi171 conmon[46715]: debug 2022-01-31T22:04:22.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:22.085821+0000) 2022-01-31T22:04:22.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:21 smithi171 conmon[51620]: debug 2022-01-31T22:04:21.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:21.936280+0000) 2022-01-31T22:04:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:22 smithi171 conmon[41853]: debug 2022-01-31T22:04:22.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:22.292202+0000) 2022-01-31T22:04:22.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:22 smithi167 conmon[54076]: debug 2022-01-31T22:04:22.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:22.582457+0000) 2022-01-31T22:04:22.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:22 smithi167 conmon[60316]: debug 2022-01-31T22:04:22.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:22.582479+0000) 2022-01-31T22:04:23.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:22 smithi167 conmon[49112]: debug 2022-01-31T22:04:22.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:22.918416+0000) 2022-01-31T22:04:23.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:22 smithi171 conmon[51620]: debug 2022-01-31T22:04:22.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:22.936442+0000) 2022-01-31T22:04:23.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[46715]: debug 2022-01-31T22:04:23.084+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.085968+0000) 2022-01-31T22:04:23.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[41853]: debug 2022-01-31T22:04:23.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.292395+0000) 2022-01-31T22:04:23.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:23 smithi167 conmon[49112]: debug 2022-01-31T22:04:23.594+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.595002+0000) 2022-01-31T22:04:23.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:23 smithi167 conmon[54076]: debug 2022-01-31T22:04:23.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.582650+0000) 2022-01-31T22:04:23.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:23 smithi167 conmon[54076]: debug 2022-01-31T22:04:23.594+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.595320+0000) 2022-01-31T22:04:23.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:23 smithi167 conmon[60316]: debug 2022-01-31T22:04:23.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.582655+0000) 2022-01-31T22:04:23.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:23 smithi167 conmon[60316]: debug 2022-01-31T22:04:23.594+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.594655+0000) 2022-01-31T22:04:23.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[35325]: debug 2022-01-31T22:04:23.611+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180240 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:23.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[35325]: debug 2022-01-31T22:04:23.786+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:04:23.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[41853]: debug 2022-01-31T22:04:23.592+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.594194+0000) 2022-01-31T22:04:23.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[46715]: debug 2022-01-31T22:04:23.593+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.595068+0000) 2022-01-31T22:04:23.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[51620]: debug 2022-01-31T22:04:23.592+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.594458+0000) 2022-01-31T22:04:24.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:23 smithi167 conmon[49112]: debug 2022-01-31T22:04:23.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.918627+0000) 2022-01-31T22:04:24.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:23 smithi171 conmon[51620]: debug 2022-01-31T22:04:23.934+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:23.936645+0000) 2022-01-31T22:04:24.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:24 smithi171 conmon[46715]: debug 2022-01-31T22:04:24.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:24.086191+0000) 2022-01-31T22:04:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:24 smithi171 conmon[41853]: debug 2022-01-31T22:04:24.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:24.292621+0000) 2022-01-31T22:04:24.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:24 smithi167 conmon[54076]: debug 2022-01-31T22:04:24.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:24.582837+0000) 2022-01-31T22:04:24.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:24 smithi167 conmon[60316]: debug 2022-01-31T22:04:24.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:24.582861+0000) 2022-01-31T22:04:25.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:24 smithi167 conmon[49112]: debug 2022-01-31T22:04:24.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:24.918804+0000) 2022-01-31T22:04:25.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:24 smithi171 conmon[51620]: debug 2022-01-31T22:04:24.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:24.936803+0000) 2022-01-31T22:04:25.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:25 smithi171 conmon[46715]: debug 2022-01-31T22:04:25.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:25.086328+0000) 2022-01-31T22:04:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:25 smithi171 conmon[41853]: debug 2022-01-31T22:04:25.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:25.292768+0000) 2022-01-31T22:04:25.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:25 smithi167 conmon[54076]: debug 2022-01-31T22:04:25.582+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:25.582992+0000) 2022-01-31T22:04:25.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:25 smithi167 conmon[60316]: debug 2022-01-31T22:04:25.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:25.583017+0000) 2022-01-31T22:04:26.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:25 smithi167 conmon[49112]: debug 2022-01-31T22:04:25.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:25.919019+0000) 2022-01-31T22:04:26.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:25 smithi171 conmon[51620]: debug 2022-01-31T22:04:25.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:25.936990+0000) 2022-01-31T22:04:26.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:26 smithi171 conmon[46715]: debug 2022-01-31T22:04:26.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:26.086488+0000) 2022-01-31T22:04:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:26 smithi171 conmon[41853]: debug 2022-01-31T22:04:26.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:26.292867+0000) 2022-01-31T22:04:26.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:26 smithi167 conmon[54076]: debug 2022-01-31T22:04:26.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:26.583228+0000) 2022-01-31T22:04:26.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:26 smithi167 conmon[60316]: debug 2022-01-31T22:04:26.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:26.583237+0000) 2022-01-31T22:04:27.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:26 smithi167 conmon[49112]: debug 2022-01-31T22:04:26.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:26.919232+0000) 2022-01-31T22:04:27.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:26 smithi171 conmon[51620]: debug 2022-01-31T22:04:26.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:26.937150+0000) 2022-01-31T22:04:27.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:27 smithi171 conmon[46715]: debug 2022-01-31T22:04:27.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:27.086676+0000) 2022-01-31T22:04:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:27 smithi171 conmon[41853]: debug 2022-01-31T22:04:27.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:27.293033+0000) 2022-01-31T22:04:27.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:27 smithi167 conmon[54076]: debug 2022-01-31T22:04:27.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:27.583392+0000) 2022-01-31T22:04:27.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:27 smithi167 conmon[60316]: debug 2022-01-31T22:04:27.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:27.583412+0000) 2022-01-31T22:04:28.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:27 smithi167 conmon[49112]: debug 2022-01-31T22:04:27.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:27.919426+0000) 2022-01-31T22:04:28.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[46715]: debug 2022-01-31T22:04:28.085+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.086856+0000) 2022-01-31T22:04:28.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:27 smithi171 conmon[51620]: debug 2022-01-31T22:04:27.935+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:27.937303+0000) 2022-01-31T22:04:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[41853]: debug 2022-01-31T22:04:28.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.293215+0000) 2022-01-31T22:04:28.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:28 smithi167 conmon[49112]: debug 2022-01-31T22:04:28.614+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.615399+0000) 2022-01-31T22:04:28.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:28 smithi167 conmon[54076]: debug 2022-01-31T22:04:28.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.583575+0000) 2022-01-31T22:04:28.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:28 smithi167 conmon[54076]: debug 2022-01-31T22:04:28.615+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.616059+0000) 2022-01-31T22:04:28.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:28 smithi167 conmon[60316]: debug 2022-01-31T22:04:28.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.583592+0000) 2022-01-31T22:04:28.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:28 smithi167 conmon[60316]: debug 2022-01-31T22:04:28.615+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.615671+0000) 2022-01-31T22:04:28.937 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[35325]: debug 2022-01-31T22:04:28.633+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180351 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:28.937 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[41853]: debug 2022-01-31T22:04:28.613+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.615420+0000) 2022-01-31T22:04:28.938 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[46715]: debug 2022-01-31T22:04:28.615+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.616941+0000) 2022-01-31T22:04:28.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[51620]: debug 2022-01-31T22:04:28.614+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.615951+0000) 2022-01-31T22:04:29.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:28 smithi167 conmon[49112]: debug 2022-01-31T22:04:28.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.919579+0000) 2022-01-31T22:04:29.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:29 smithi171 conmon[46715]: debug 2022-01-31T22:04:29.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:29.087010+0000) 2022-01-31T22:04:29.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:28 smithi171 conmon[51620]: debug 2022-01-31T22:04:28.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:28.937460+0000) 2022-01-31T22:04:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:29 smithi171 conmon[41853]: debug 2022-01-31T22:04:29.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:29.293359+0000) 2022-01-31T22:04:29.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:29 smithi167 conmon[54076]: debug 2022-01-31T22:04:29.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:29.583773+0000) 2022-01-31T22:04:29.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:29 smithi167 conmon[60316]: debug 2022-01-31T22:04:29.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:29.583795+0000) 2022-01-31T22:04:30.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:29 smithi167 conmon[49112]: debug 2022-01-31T22:04:29.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:29.919717+0000) 2022-01-31T22:04:30.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:30 smithi171 conmon[46715]: debug 2022-01-31T22:04:30.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:30.087159+0000) 2022-01-31T22:04:30.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:29 smithi171 conmon[51620]: debug 2022-01-31T22:04:29.936+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:29.937696+0000) 2022-01-31T22:04:30.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:30 smithi171 conmon[41853]: debug 2022-01-31T22:04:30.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:30.293525+0000) 2022-01-31T22:04:30.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:30 smithi167 conmon[54076]: debug 2022-01-31T22:04:30.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:30.583891+0000) 2022-01-31T22:04:30.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:30 smithi167 conmon[60316]: debug 2022-01-31T22:04:30.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:30.583928+0000) 2022-01-31T22:04:31.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:30 smithi167 conmon[49112]: debug 2022-01-31T22:04:30.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:30.919925+0000) 2022-01-31T22:04:31.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:31 smithi171 conmon[46715]: debug 2022-01-31T22:04:31.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:31.087309+0000) 2022-01-31T22:04:31.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:30 smithi171 conmon[51620]: debug 2022-01-31T22:04:30.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:30.937902+0000) 2022-01-31T22:04:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:31 smithi171 conmon[41853]: debug 2022-01-31T22:04:31.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:31.293684+0000) 2022-01-31T22:04:31.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:31 smithi167 conmon[54076]: debug 2022-01-31T22:04:31.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:31.584066+0000) 2022-01-31T22:04:31.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:31 smithi167 conmon[60316]: debug 2022-01-31T22:04:31.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:31.584098+0000) 2022-01-31T22:04:32.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:31 smithi167 conmon[49112]: debug 2022-01-31T22:04:31.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:31.920136+0000) 2022-01-31T22:04:32.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:32 smithi171 conmon[46715]: debug 2022-01-31T22:04:32.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:32.087502+0000) 2022-01-31T22:04:32.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:31 smithi171 conmon[51620]: debug 2022-01-31T22:04:31.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:31.938092+0000) 2022-01-31T22:04:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:32 smithi171 conmon[41853]: debug 2022-01-31T22:04:32.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:32.293868+0000) 2022-01-31T22:04:32.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:32 smithi167 conmon[60316]: debug 2022-01-31T22:04:32.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:32.584296+0000) 2022-01-31T22:04:32.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:32 smithi167 conmon[54076]: debug 2022-01-31T22:04:32.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:32.584287+0000) 2022-01-31T22:04:33.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:32 smithi167 conmon[49112]: debug 2022-01-31T22:04:32.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:32.920344+0000) 2022-01-31T22:04:33.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[46715]: debug 2022-01-31T22:04:33.086+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.087652+0000) 2022-01-31T22:04:33.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:32 smithi171 conmon[51620]: debug 2022-01-31T22:04:32.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:32.938274+0000) 2022-01-31T22:04:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[41853]: debug 2022-01-31T22:04:33.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.294070+0000) 2022-01-31T22:04:33.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:33 smithi167 conmon[49112]: debug 2022-01-31T22:04:33.643+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.644359+0000) 2022-01-31T22:04:33.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:33 smithi167 conmon[54076]: debug 2022-01-31T22:04:33.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.584529+0000) 2022-01-31T22:04:33.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:33 smithi167 conmon[54076]: debug 2022-01-31T22:04:33.644+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.644509+0000) 2022-01-31T22:04:33.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:33 smithi167 conmon[60316]: debug 2022-01-31T22:04:33.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.584521+0000) 2022-01-31T22:04:33.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:33 smithi167 conmon[60316]: debug 2022-01-31T22:04:33.643+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.644101+0000) 2022-01-31T22:04:33.937 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[35325]: debug 2022-01-31T22:04:33.660+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180475 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:33.938 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[41853]: debug 2022-01-31T22:04:33.636+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.636827+0000) 2022-01-31T22:04:33.939 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[46715]: debug 2022-01-31T22:04:33.636+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.637175+0000) 2022-01-31T22:04:33.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[51620]: debug 2022-01-31T22:04:33.636+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.636747+0000) 2022-01-31T22:04:34.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:33 smithi167 conmon[49112]: debug 2022-01-31T22:04:33.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.920543+0000) 2022-01-31T22:04:34.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:33 smithi171 conmon[51620]: debug 2022-01-31T22:04:33.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:33.938454+0000) 2022-01-31T22:04:34.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:34 smithi171 conmon[46715]: debug 2022-01-31T22:04:34.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:34.087811+0000) 2022-01-31T22:04:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:34 smithi171 conmon[41853]: debug 2022-01-31T22:04:34.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:34.294260+0000) 2022-01-31T22:04:34.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:34 smithi167 conmon[54076]: debug 2022-01-31T22:04:34.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:34.584750+0000) 2022-01-31T22:04:34.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:34 smithi167 conmon[60316]: debug 2022-01-31T22:04:34.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:34.584749+0000) 2022-01-31T22:04:35.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:34 smithi167 conmon[49112]: debug 2022-01-31T22:04:34.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:34.920716+0000) 2022-01-31T22:04:35.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:35 smithi171 conmon[46715]: debug 2022-01-31T22:04:35.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:35.087922+0000) 2022-01-31T22:04:35.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:34 smithi171 conmon[51620]: debug 2022-01-31T22:04:34.937+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:34.938633+0000) 2022-01-31T22:04:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:35 smithi171 conmon[41853]: debug 2022-01-31T22:04:35.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:35.294414+0000) 2022-01-31T22:04:35.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:35 smithi167 conmon[54076]: debug 2022-01-31T22:04:35.583+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:35.584921+0000) 2022-01-31T22:04:35.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:35 smithi167 conmon[60316]: debug 2022-01-31T22:04:35.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:35.584916+0000) 2022-01-31T22:04:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:35 smithi167 conmon[49112]: debug 2022-01-31T22:04:35.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:35.920850+0000) 2022-01-31T22:04:36.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:36 smithi171 conmon[46715]: debug 2022-01-31T22:04:36.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:36.088123+0000) 2022-01-31T22:04:36.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:35 smithi171 conmon[51620]: debug 2022-01-31T22:04:35.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:35.938796+0000) 2022-01-31T22:04:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:36 smithi171 conmon[41853]: debug 2022-01-31T22:04:36.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:36.294597+0000) 2022-01-31T22:04:36.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:36 smithi167 conmon[54076]: debug 2022-01-31T22:04:36.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:36.585110+0000) 2022-01-31T22:04:36.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:36 smithi167 conmon[60316]: debug 2022-01-31T22:04:36.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:36.585105+0000) 2022-01-31T22:04:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:36 smithi167 conmon[49112]: debug 2022-01-31T22:04:36.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:36.921037+0000) 2022-01-31T22:04:37.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:37 smithi171 conmon[46715]: debug 2022-01-31T22:04:37.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:37.088305+0000) 2022-01-31T22:04:37.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:36 smithi171 conmon[51620]: debug 2022-01-31T22:04:36.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:36.938993+0000) 2022-01-31T22:04:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:37 smithi171 conmon[41853]: debug 2022-01-31T22:04:37.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:37.294756+0000) 2022-01-31T22:04:37.887 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:37 smithi167 conmon[60316]: debug 2022-01-31T22:04:37.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:37.585308+0000) 2022-01-31T22:04:37.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:37 smithi167 conmon[54076]: debug 2022-01-31T22:04:37.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:37.585308+0000) 2022-01-31T22:04:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:37 smithi167 conmon[49112]: debug 2022-01-31T22:04:37.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:37.921245+0000) 2022-01-31T22:04:38.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[46715]: debug 2022-01-31T22:04:38.087+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.088542+0000) 2022-01-31T22:04:38.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:37 smithi171 conmon[51620]: debug 2022-01-31T22:04:37.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:37.939169+0000) 2022-01-31T22:04:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[41853]: debug 2022-01-31T22:04:38.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.294930+0000) 2022-01-31T22:04:38.888 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:38 smithi167 conmon[49112]: debug 2022-01-31T22:04:38.663+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.664596+0000) 2022-01-31T22:04:38.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:38 smithi167 conmon[54076]: debug 2022-01-31T22:04:38.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.585513+0000) 2022-01-31T22:04:38.891 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:38 smithi167 conmon[54076]: debug 2022-01-31T22:04:38.662+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.663709+0000) 2022-01-31T22:04:38.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:38 smithi167 conmon[60316]: debug 2022-01-31T22:04:38.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.585481+0000) 2022-01-31T22:04:38.892 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:38 smithi167 conmon[60316]: debug 2022-01-31T22:04:38.662+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.663841+0000) 2022-01-31T22:04:38.938 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[41853]: debug 2022-01-31T22:04:38.662+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.662938+0000) 2022-01-31T22:04:38.939 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[46715]: debug 2022-01-31T22:04:38.663+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.664291+0000) 2022-01-31T22:04:38.939 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[51620]: debug 2022-01-31T22:04:38.663+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.664146+0000) 2022-01-31T22:04:38.940 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[35325]: debug 2022-01-31T22:04:38.681+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180586 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:38.940 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[35325]: debug 2022-01-31T22:04:38.787+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:04:39.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:38 smithi167 conmon[49112]: debug 2022-01-31T22:04:38.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.921431+0000) 2022-01-31T22:04:39.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:39 smithi171 conmon[46715]: debug 2022-01-31T22:04:39.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:39.088715+0000) 2022-01-31T22:04:39.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:38 smithi171 conmon[51620]: debug 2022-01-31T22:04:38.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:38.939338+0000) 2022-01-31T22:04:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:39 smithi171 conmon[41853]: debug 2022-01-31T22:04:39.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:39.295093+0000) 2022-01-31T22:04:39.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:39 smithi167 conmon[54076]: debug 2022-01-31T22:04:39.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:39.585685+0000) 2022-01-31T22:04:39.889 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:39 smithi167 conmon[60316]: debug 2022-01-31T22:04:39.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:39.585684+0000) 2022-01-31T22:04:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:39 smithi167 conmon[49112]: debug 2022-01-31T22:04:39.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:39.921569+0000) 2022-01-31T22:04:40.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:40 smithi171 conmon[46715]: debug 2022-01-31T22:04:40.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:40.088890+0000) 2022-01-31T22:04:40.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:39 smithi171 conmon[51620]: debug 2022-01-31T22:04:39.938+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:39.939470+0000) 2022-01-31T22:04:40.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:40 smithi171 conmon[41853]: debug 2022-01-31T22:04:40.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:40.295230+0000) 2022-01-31T22:04:40.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:40 smithi167 conmon[60316]: debug 2022-01-31T22:04:40.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:40.585857+0000) 2022-01-31T22:04:40.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:40 smithi167 conmon[54076]: debug 2022-01-31T22:04:40.584+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:40.585861+0000) 2022-01-31T22:04:41.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:40 smithi167 conmon[49112]: debug 2022-01-31T22:04:40.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:40.921726+0000) 2022-01-31T22:04:41.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:41 smithi171 conmon[46715]: debug 2022-01-31T22:04:41.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:41.089096+0000) 2022-01-31T22:04:41.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:40 smithi171 conmon[51620]: debug 2022-01-31T22:04:40.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:40.939660+0000) 2022-01-31T22:04:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:41 smithi171 conmon[41853]: debug 2022-01-31T22:04:41.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:41.295393+0000) 2022-01-31T22:04:41.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:41 smithi167 conmon[60316]: debug 2022-01-31T22:04:41.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:41.586097+0000) 2022-01-31T22:04:41.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:41 smithi167 conmon[54076]: debug 2022-01-31T22:04:41.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:41.586083+0000) 2022-01-31T22:04:42.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:41 smithi167 conmon[49112]: debug 2022-01-31T22:04:41.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:41.921906+0000) 2022-01-31T22:04:42.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:42 smithi171 conmon[46715]: debug 2022-01-31T22:04:42.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:42.089283+0000) 2022-01-31T22:04:42.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:41 smithi171 conmon[51620]: debug 2022-01-31T22:04:41.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:41.939823+0000) 2022-01-31T22:04:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:42 smithi171 conmon[41853]: debug 2022-01-31T22:04:42.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:42.295539+0000) 2022-01-31T22:04:42.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:42 smithi167 conmon[60316]: debug 2022-01-31T22:04:42.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:42.586298+0000) 2022-01-31T22:04:42.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:42 smithi167 conmon[54076]: debug 2022-01-31T22:04:42.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:42.586252+0000) 2022-01-31T22:04:43.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:42 smithi167 conmon[49112]: debug 2022-01-31T22:04:42.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:42.922115+0000) 2022-01-31T22:04:43.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[46715]: debug 2022-01-31T22:04:43.088+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.089486+0000) 2022-01-31T22:04:43.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:42 smithi171 conmon[51620]: debug 2022-01-31T22:04:42.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:42.939994+0000) 2022-01-31T22:04:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[41853]: debug 2022-01-31T22:04:43.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.295718+0000) 2022-01-31T22:04:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:43 smithi167 conmon[54076]: debug 2022-01-31T22:04:43.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.586436+0000) 2022-01-31T22:04:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:43 smithi167 conmon[60316]: debug 2022-01-31T22:04:43.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.586457+0000) 2022-01-31T22:04:43.939 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[35325]: debug 2022-01-31T22:04:43.703+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180697 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:43.940 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[46715]: debug 2022-01-31T22:04:43.685+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.686423+0000) 2022-01-31T22:04:43.941 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[51620]: debug 2022-01-31T22:04:43.684+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.685269+0000) 2022-01-31T22:04:43.941 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[41853]: debug 2022-01-31T22:04:43.684+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.684955+0000) 2022-01-31T22:04:44.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:43 smithi167 conmon[54076]: debug 2022-01-31T22:04:43.683+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.684569+0000) 2022-01-31T22:04:44.078 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:43 smithi167 conmon[60316]: debug 2022-01-31T22:04:43.683+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.684657+0000) 2022-01-31T22:04:44.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:43 smithi167 conmon[49112]: debug 2022-01-31T22:04:43.683+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.685008+0000) 2022-01-31T22:04:44.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:43 smithi167 conmon[49112]: debug 2022-01-31T22:04:43.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.922315+0000) 2022-01-31T22:04:44.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:43 smithi171 conmon[51620]: debug 2022-01-31T22:04:43.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:43.940210+0000) 2022-01-31T22:04:44.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:44 smithi171 conmon[46715]: debug 2022-01-31T22:04:44.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:44.089704+0000) 2022-01-31T22:04:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:44 smithi171 conmon[41853]: debug 2022-01-31T22:04:44.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:44.295926+0000) 2022-01-31T22:04:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:44 smithi167 conmon[54076]: debug 2022-01-31T22:04:44.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:44.586647+0000) 2022-01-31T22:04:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:44 smithi167 conmon[60316]: debug 2022-01-31T22:04:44.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:44.586651+0000) 2022-01-31T22:04:45.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:44 smithi167 conmon[49112]: debug 2022-01-31T22:04:44.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:44.922460+0000) 2022-01-31T22:04:45.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:45 smithi171 conmon[46715]: debug 2022-01-31T22:04:45.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:45.089851+0000) 2022-01-31T22:04:45.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:44 smithi171 conmon[51620]: debug 2022-01-31T22:04:44.939+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:44.940380+0000) 2022-01-31T22:04:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:45 smithi171 conmon[41853]: debug 2022-01-31T22:04:45.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:45.296040+0000) 2022-01-31T22:04:45.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:45 smithi167 conmon[54076]: debug 2022-01-31T22:04:45.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:45.586800+0000) 2022-01-31T22:04:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:45 smithi167 conmon[60316]: debug 2022-01-31T22:04:45.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:45.586783+0000) 2022-01-31T22:04:46.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:45 smithi167 conmon[49112]: debug 2022-01-31T22:04:45.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:45.922643+0000) 2022-01-31T22:04:46.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:46 smithi171 conmon[46715]: debug 2022-01-31T22:04:46.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:46.090033+0000) 2022-01-31T22:04:46.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:45 smithi171 conmon[51620]: debug 2022-01-31T22:04:45.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:45.940616+0000) 2022-01-31T22:04:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:46 smithi171 conmon[41853]: debug 2022-01-31T22:04:46.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:46.296227+0000) 2022-01-31T22:04:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:46 smithi167 conmon[54076]: debug 2022-01-31T22:04:46.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:46.587004+0000) 2022-01-31T22:04:46.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:46 smithi167 conmon[60316]: debug 2022-01-31T22:04:46.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:46.586982+0000) 2022-01-31T22:04:47.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:46 smithi167 conmon[49112]: debug 2022-01-31T22:04:46.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:46.922828+0000) 2022-01-31T22:04:47.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:47 smithi171 conmon[46715]: debug 2022-01-31T22:04:47.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:47.090180+0000) 2022-01-31T22:04:47.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:46 smithi171 conmon[51620]: debug 2022-01-31T22:04:46.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:46.940773+0000) 2022-01-31T22:04:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:47 smithi171 conmon[41853]: debug 2022-01-31T22:04:47.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:47.296403+0000) 2022-01-31T22:04:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:47 smithi167 conmon[54076]: debug 2022-01-31T22:04:47.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:47.587222+0000) 2022-01-31T22:04:47.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:47 smithi167 conmon[60316]: debug 2022-01-31T22:04:47.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:47.587223+0000) 2022-01-31T22:04:48.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:47 smithi167 conmon[49112]: debug 2022-01-31T22:04:47.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:47.923004+0000) 2022-01-31T22:04:48.252 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[46715]: debug 2022-01-31T22:04:48.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.090364+0000) 2022-01-31T22:04:48.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:47 smithi171 conmon[51620]: debug 2022-01-31T22:04:47.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:47.940962+0000) 2022-01-31T22:04:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[41853]: debug 2022-01-31T22:04:48.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.296564+0000) 2022-01-31T22:04:48.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:48 smithi167 conmon[54076]: debug 2022-01-31T22:04:48.585+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.587431+0000) 2022-01-31T22:04:48.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:48 smithi167 conmon[60316]: debug 2022-01-31T22:04:48.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.587435+0000) 2022-01-31T22:04:49.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[41853]: debug 2022-01-31T22:04:48.705+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.705813+0000) 2022-01-31T22:04:49.028 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[35325]: debug 2022-01-31T22:04:48.723+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180808 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:49.028 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[46715]: debug 2022-01-31T22:04:48.705+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.706496+0000) 2022-01-31T22:04:49.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[51620]: debug 2022-01-31T22:04:48.704+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.705682+0000) 2022-01-31T22:04:49.029 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:48 smithi171 conmon[51620]: debug 2022-01-31T22:04:48.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.941161+0000) 2022-01-31T22:04:49.078 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:48 smithi167 conmon[54076]: debug 2022-01-31T22:04:48.705+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.707127+0000) 2022-01-31T22:04:49.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:48 smithi167 conmon[49112]: debug 2022-01-31T22:04:48.705+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.707252+0000) 2022-01-31T22:04:49.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:48 smithi167 conmon[49112]: debug 2022-01-31T22:04:48.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.923224+0000) 2022-01-31T22:04:49.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:48 smithi167 conmon[60316]: debug 2022-01-31T22:04:48.705+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:48.706789+0000) 2022-01-31T22:04:49.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:49 smithi171 conmon[46715]: debug 2022-01-31T22:04:49.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:49.090541+0000) 2022-01-31T22:04:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:49 smithi171 conmon[41853]: debug 2022-01-31T22:04:49.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:49.296746+0000) 2022-01-31T22:04:49.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:49 smithi167 conmon[54076]: debug 2022-01-31T22:04:49.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:49.587652+0000) 2022-01-31T22:04:49.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:49 smithi167 conmon[60316]: debug 2022-01-31T22:04:49.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:49.587579+0000) 2022-01-31T22:04:50.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:49 smithi167 conmon[49112]: debug 2022-01-31T22:04:49.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:49.923391+0000) 2022-01-31T22:04:50.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:50 smithi171 conmon[46715]: debug 2022-01-31T22:04:50.089+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:50.090691+0000) 2022-01-31T22:04:50.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:49 smithi171 conmon[51620]: debug 2022-01-31T22:04:49.940+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:49.941362+0000) 2022-01-31T22:04:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:50 smithi171 conmon[41853]: debug 2022-01-31T22:04:50.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:50.296885+0000) 2022-01-31T22:04:50.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:50 smithi167 conmon[54076]: debug 2022-01-31T22:04:50.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:50.587779+0000) 2022-01-31T22:04:50.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:50 smithi167 conmon[60316]: debug 2022-01-31T22:04:50.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:50.587692+0000) 2022-01-31T22:04:51.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:50 smithi167 conmon[49112]: debug 2022-01-31T22:04:50.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:50.923574+0000) 2022-01-31T22:04:51.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:51 smithi171 conmon[46715]: debug 2022-01-31T22:04:51.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:51.090848+0000) 2022-01-31T22:04:51.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:50 smithi171 conmon[51620]: debug 2022-01-31T22:04:50.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:50.941541+0000) 2022-01-31T22:04:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:51 smithi171 conmon[41853]: debug 2022-01-31T22:04:51.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:51.297019+0000) 2022-01-31T22:04:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:51 smithi167 conmon[54076]: debug 2022-01-31T22:04:51.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:51.587893+0000) 2022-01-31T22:04:51.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:51 smithi167 conmon[60316]: debug 2022-01-31T22:04:51.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:51.587822+0000) 2022-01-31T22:04:52.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:51 smithi167 conmon[49112]: debug 2022-01-31T22:04:51.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:51.923734+0000) 2022-01-31T22:04:52.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:52 smithi171 conmon[46715]: debug 2022-01-31T22:04:52.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:52.090997+0000) 2022-01-31T22:04:52.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:51 smithi171 conmon[51620]: debug 2022-01-31T22:04:51.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:51.941731+0000) 2022-01-31T22:04:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:52 smithi171 conmon[41853]: debug 2022-01-31T22:04:52.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:52.297206+0000) 2022-01-31T22:04:52.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:52 smithi167 conmon[54076]: debug 2022-01-31T22:04:52.586+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:52.588054+0000) 2022-01-31T22:04:52.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:52 smithi167 conmon[60316]: debug 2022-01-31T22:04:52.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:52.587951+0000) 2022-01-31T22:04:53.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:52 smithi167 conmon[49112]: debug 2022-01-31T22:04:52.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:52.923887+0000) 2022-01-31T22:04:53.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[46715]: debug 2022-01-31T22:04:53.090+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.091189+0000) 2022-01-31T22:04:53.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:52 smithi171 conmon[51620]: debug 2022-01-31T22:04:52.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:52.941938+0000) 2022-01-31T22:04:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[41853]: debug 2022-01-31T22:04:53.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.297410+0000) 2022-01-31T22:04:53.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:53 smithi167 conmon[60316]: debug 2022-01-31T22:04:53.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.588131+0000) 2022-01-31T22:04:53.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:53 smithi167 conmon[54076]: debug 2022-01-31T22:04:53.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.588178+0000) 2022-01-31T22:04:54.027 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[41853]: debug 2022-01-31T22:04:53.726+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.727129+0000) 2022-01-31T22:04:54.028 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[46715]: debug 2022-01-31T22:04:53.726+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.726972+0000) 2022-01-31T22:04:54.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[35325]: debug 2022-01-31T22:04:53.745+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 180919 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:54.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[35325]: debug 2022-01-31T22:04:53.788+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:04:54.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[51620]: debug 2022-01-31T22:04:53.726+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.726853+0000) 2022-01-31T22:04:54.030 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:53 smithi171 conmon[51620]: debug 2022-01-31T22:04:53.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.942076+0000) 2022-01-31T22:04:54.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:53 smithi167 conmon[60316]: debug 2022-01-31T22:04:53.726+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.728058+0000) 2022-01-31T22:04:54.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:53 smithi167 conmon[49112]: debug 2022-01-31T22:04:53.726+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.727819+0000) 2022-01-31T22:04:54.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:53 smithi167 conmon[49112]: debug 2022-01-31T22:04:53.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.924051+0000) 2022-01-31T22:04:54.081 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:53 smithi167 conmon[54076]: debug 2022-01-31T22:04:53.725+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:53.727401+0000) 2022-01-31T22:04:54.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:54 smithi171 conmon[46715]: debug 2022-01-31T22:04:54.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:54.091371+0000) 2022-01-31T22:04:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:54 smithi171 conmon[41853]: debug 2022-01-31T22:04:54.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:54.297594+0000) 2022-01-31T22:04:54.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:54 smithi167 conmon[60316]: debug 2022-01-31T22:04:54.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:54.588282+0000) 2022-01-31T22:04:54.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:54 smithi167 conmon[54076]: debug 2022-01-31T22:04:54.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:54.588303+0000) 2022-01-31T22:04:55.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:54 smithi167 conmon[49112]: debug 2022-01-31T22:04:54.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:54.924163+0000) 2022-01-31T22:04:55.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:54 smithi171 conmon[51620]: debug 2022-01-31T22:04:54.941+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:54.942215+0000) 2022-01-31T22:04:55.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:55 smithi171 conmon[46715]: debug 2022-01-31T22:04:55.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:55.091510+0000) 2022-01-31T22:04:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:55 smithi171 conmon[41853]: debug 2022-01-31T22:04:55.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:55.297729+0000) 2022-01-31T22:04:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:55 smithi167 conmon[54076]: debug 2022-01-31T22:04:55.587+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:55.588445+0000) 2022-01-31T22:04:55.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:55 smithi167 conmon[60316]: debug 2022-01-31T22:04:55.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:55.588444+0000) 2022-01-31T22:04:56.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:55 smithi167 conmon[49112]: debug 2022-01-31T22:04:55.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:55.924350+0000) 2022-01-31T22:04:56.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:56 smithi171 conmon[46715]: debug 2022-01-31T22:04:56.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:56.091693+0000) 2022-01-31T22:04:56.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:55 smithi171 conmon[51620]: debug 2022-01-31T22:04:55.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:55.942403+0000) 2022-01-31T22:04:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:56 smithi171 conmon[41853]: debug 2022-01-31T22:04:56.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:56.297922+0000) 2022-01-31T22:04:56.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:56 smithi167 conmon[54076]: debug 2022-01-31T22:04:56.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:56.588636+0000) 2022-01-31T22:04:56.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:56 smithi167 conmon[60316]: debug 2022-01-31T22:04:56.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:56.588636+0000) 2022-01-31T22:04:57.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:56 smithi167 conmon[49112]: debug 2022-01-31T22:04:56.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:56.924534+0000) 2022-01-31T22:04:57.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:56 smithi171 conmon[51620]: debug 2022-01-31T22:04:56.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:56.942605+0000) 2022-01-31T22:04:57.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:57 smithi171 conmon[46715]: debug 2022-01-31T22:04:57.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:57.091920+0000) 2022-01-31T22:04:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:57 smithi171 conmon[41853]: debug 2022-01-31T22:04:57.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:57.298116+0000) 2022-01-31T22:04:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:57 smithi167 conmon[54076]: debug 2022-01-31T22:04:57.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:57.588881+0000) 2022-01-31T22:04:57.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:57 smithi167 conmon[60316]: debug 2022-01-31T22:04:57.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:57.588881+0000) 2022-01-31T22:04:58.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:57 smithi167 conmon[49112]: debug 2022-01-31T22:04:57.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:57.924724+0000) 2022-01-31T22:04:58.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[46715]: debug 2022-01-31T22:04:58.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.092106+0000) 2022-01-31T22:04:58.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:57 smithi171 conmon[51620]: debug 2022-01-31T22:04:57.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:57.942782+0000) 2022-01-31T22:04:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[41853]: debug 2022-01-31T22:04:58.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.298285+0000) 2022-01-31T22:04:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:58 smithi167 conmon[54076]: debug 2022-01-31T22:04:58.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.589118+0000) 2022-01-31T22:04:58.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:58 smithi167 conmon[60316]: debug 2022-01-31T22:04:58.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.589118+0000) 2022-01-31T22:04:59.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[35325]: debug 2022-01-31T22:04:58.766+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181030 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:04:59.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[41853]: debug 2022-01-31T22:04:58.748+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.748863+0000) 2022-01-31T22:04:59.031 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[46715]: debug 2022-01-31T22:04:58.747+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.748024+0000) 2022-01-31T22:04:59.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[51620]: debug 2022-01-31T22:04:58.747+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.748640+0000) 2022-01-31T22:04:59.033 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:58 smithi171 conmon[51620]: debug 2022-01-31T22:04:58.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.942936+0000) 2022-01-31T22:04:59.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:58 smithi167 conmon[49112]: debug 2022-01-31T22:04:58.747+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.749196+0000) 2022-01-31T22:04:59.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:58 smithi167 conmon[49112]: debug 2022-01-31T22:04:58.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.924949+0000) 2022-01-31T22:04:59.082 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:58 smithi167 conmon[60316]: debug 2022-01-31T22:04:58.746+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.748460+0000) 2022-01-31T22:04:59.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:58 smithi167 conmon[54076]: debug 2022-01-31T22:04:58.747+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:58.748623+0000) 2022-01-31T22:04:59.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:04:59 smithi171 conmon[46715]: debug 2022-01-31T22:04:59.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:59.092303+0000) 2022-01-31T22:04:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:04:59 smithi171 conmon[41853]: debug 2022-01-31T22:04:59.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:59.298437+0000) 2022-01-31T22:04:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:04:59 smithi167 conmon[54076]: debug 2022-01-31T22:04:59.588+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:59.589325+0000) 2022-01-31T22:04:59.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:04:59 smithi167 conmon[60316]: debug 2022-01-31T22:04:59.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:59.589325+0000) 2022-01-31T22:05:00.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:04:59 smithi167 conmon[49112]: debug 2022-01-31T22:04:59.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:59.925122+0000) 2022-01-31T22:05:00.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:00 smithi171 conmon[46715]: debug 2022-01-31T22:05:00.091+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:00.092468+0000) 2022-01-31T22:05:00.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:04:59 smithi171 conmon[51620]: debug 2022-01-31T22:04:59.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:04:59.943100+0000) 2022-01-31T22:05:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:00 smithi171 conmon[41853]: debug 2022-01-31T22:05:00.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:00.298600+0000) 2022-01-31T22:05:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:00 smithi167 conmon[54076]: debug 2022-01-31T22:05:00.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:00.589495+0000) 2022-01-31T22:05:00.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:00 smithi167 conmon[60316]: debug 2022-01-31T22:05:00.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:00.589495+0000) 2022-01-31T22:05:01.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:00 smithi167 conmon[49112]: debug 2022-01-31T22:05:00.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:00.925267+0000) 2022-01-31T22:05:01.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:01 smithi171 conmon[46715]: debug 2022-01-31T22:05:01.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:01.092657+0000) 2022-01-31T22:05:01.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:00 smithi171 conmon[51620]: debug 2022-01-31T22:05:00.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:00.943314+0000) 2022-01-31T22:05:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:01 smithi171 conmon[41853]: debug 2022-01-31T22:05:01.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:01.298815+0000) 2022-01-31T22:05:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:01 smithi167 conmon[54076]: debug 2022-01-31T22:05:01.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:01.589732+0000) 2022-01-31T22:05:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:01 smithi167 conmon[60316]: debug 2022-01-31T22:05:01.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:01.589732+0000) 2022-01-31T22:05:02.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:01 smithi167 conmon[49112]: debug 2022-01-31T22:05:01.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:01.925430+0000) 2022-01-31T22:05:02.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:02 smithi171 conmon[46715]: debug 2022-01-31T22:05:02.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:02.092815+0000) 2022-01-31T22:05:02.257 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:01 smithi171 conmon[51620]: debug 2022-01-31T22:05:01.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:01.943506+0000) 2022-01-31T22:05:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:02 smithi171 conmon[41853]: debug 2022-01-31T22:05:02.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:02.298911+0000) 2022-01-31T22:05:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:02 smithi167 conmon[54076]: debug 2022-01-31T22:05:02.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:02.589922+0000) 2022-01-31T22:05:02.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:02 smithi167 conmon[60316]: debug 2022-01-31T22:05:02.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:02.589921+0000) 2022-01-31T22:05:03.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:02 smithi167 conmon[49112]: debug 2022-01-31T22:05:02.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:02.925588+0000) 2022-01-31T22:05:03.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[46715]: debug 2022-01-31T22:05:03.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.092971+0000) 2022-01-31T22:05:03.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:02 smithi171 conmon[51620]: debug 2022-01-31T22:05:03.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:02 smithi171 conmon[51620]: 2022-01-31T22:05:02.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:02.943725+0000) 2022-01-31T22:05:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[41853]: debug 2022-01-31T22:05:03.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.299094+0000) 2022-01-31T22:05:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:03 smithi167 conmon[54076]: debug 2022-01-31T22:05:03.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.590084+0000) 2022-01-31T22:05:03.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:03 smithi167 conmon[60316]: debug 2022-01-31T22:05:03.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.590084+0000) 2022-01-31T22:05:04.029 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[35325]: debug 2022-01-31T22:05:03.787+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181142 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:04.030 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[41853]: debug 2022-01-31T22:05:03.768+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.769570+0000) 2022-01-31T22:05:04.031 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[46715]: debug 2022-01-31T22:05:03.770+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.770748+0000) 2022-01-31T22:05:04.031 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[51620]: debug 2022-01-31T22:05:03.769+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.770288+0000) 2022-01-31T22:05:04.032 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:03 smithi171 conmon[51620]: debug 2022-01-31T22:05:03.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.943904+0000) 2022-01-31T22:05:04.081 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:03 smithi167 conmon[54076]: debug 2022-01-31T22:05:03.770+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.770954+0000) 2022-01-31T22:05:04.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:03 smithi167 conmon[60316]: debug 2022-01-31T22:05:03.769+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.769542+0000) 2022-01-31T22:05:04.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:03 smithi167 conmon[49112]: debug 2022-01-31T22:05:03.769+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.770308+0000) 2022-01-31T22:05:04.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:03 smithi167 conmon[49112]: debug 2022-01-31T22:05:03.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:03.925766+0000) 2022-01-31T22:05:04.298 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:04 smithi171 conmon[46715]: debug 2022-01-31T22:05:04.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:04.093141+0000) 2022-01-31T22:05:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:04 smithi171 conmon[41853]: debug 2022-01-31T22:05:04.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:04.299290+0000) 2022-01-31T22:05:04.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:04 smithi167 conmon[54076]: debug 2022-01-31T22:05:04.589+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:04.590315+0000) 2022-01-31T22:05:04.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:04 smithi167 conmon[60316]: debug 2022-01-31T22:05:04.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:04.590315+0000) 2022-01-31T22:05:05.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:04 smithi167 conmon[49112]: debug 2022-01-31T22:05:04.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:04.925885+0000) 2022-01-31T22:05:05.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:05 smithi171 conmon[46715]: debug 2022-01-31T22:05:05.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:05.093307+0000) 2022-01-31T22:05:05.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:04 smithi171 conmon[51620]: debug 2022-01-31T22:05:04.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:04.944064+0000) 2022-01-31T22:05:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:05 smithi171 conmon[41853]: debug 2022-01-31T22:05:05.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:05.299429+0000) 2022-01-31T22:05:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:05 smithi167 conmon[54076]: debug 2022-01-31T22:05:05.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:05.590487+0000) 2022-01-31T22:05:05.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:05 smithi167 conmon[60316]: debug 2022-01-31T22:05:05.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:05.590485+0000) 2022-01-31T22:05:06.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:05 smithi167 conmon[49112]: debug 2022-01-31T22:05:05.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:05.926034+0000) 2022-01-31T22:05:06.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:06 smithi171 conmon[46715]: debug 2022-01-31T22:05:06.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:06.093493+0000) 2022-01-31T22:05:06.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:05 smithi171 conmon[51620]: debug 2022-01-31T22:05:05.942+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:05.944295+0000) 2022-01-31T22:05:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:06 smithi171 conmon[41853]: debug 2022-01-31T22:05:06.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:06.299581+0000) 2022-01-31T22:05:06.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:06 smithi167 conmon[60316]: debug 2022-01-31T22:05:06.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:06.590736+0000) 2022-01-31T22:05:06.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:06 smithi167 conmon[54076]: debug 2022-01-31T22:05:06.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:06.590736+0000) 2022-01-31T22:05:07.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:06 smithi167 conmon[49112]: debug 2022-01-31T22:05:06.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:06.926187+0000) 2022-01-31T22:05:07.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:06 smithi171 conmon[51620]: debug 2022-01-31T22:05:06.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:06.944459+0000) 2022-01-31T22:05:07.256 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:07 smithi171 conmon[46715]: debug 2022-01-31T22:05:07.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:07.093679+0000) 2022-01-31T22:05:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:07 smithi171 conmon[41853]: debug 2022-01-31T22:05:07.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:07.299771+0000) 2022-01-31T22:05:07.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:07 smithi167 conmon[54076]: debug 2022-01-31T22:05:07.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:07.590958+0000) 2022-01-31T22:05:07.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:07 smithi167 conmon[60316]: debug 2022-01-31T22:05:07.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:07.590956+0000) 2022-01-31T22:05:08.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:07 smithi167 conmon[49112]: debug 2022-01-31T22:05:07.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:07.926350+0000) 2022-01-31T22:05:08.256 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[46715]: debug 2022-01-31T22:05:08.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.093868+0000) 2022-01-31T22:05:08.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:07 smithi171 conmon[51620]: debug 2022-01-31T22:05:07.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:07.944663+0000) 2022-01-31T22:05:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[41853]: debug 2022-01-31T22:05:08.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.299986+0000) 2022-01-31T22:05:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:08 smithi167 conmon[60316]: debug 2022-01-31T22:05:08.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.591184+0000) 2022-01-31T22:05:08.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:08 smithi167 conmon[54076]: debug 2022-01-31T22:05:08.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.591184+0000) 2022-01-31T22:05:09.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:08 smithi167 conmon[49112]: debug 2022-01-31T22:05:08.789+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.790227+0000) 2022-01-31T22:05:09.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:08 smithi167 conmon[49112]: debug 2022-01-31T22:05:08.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.926537+0000) 2022-01-31T22:05:09.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:08 smithi167 conmon[54076]: debug 2022-01-31T22:05:08.790+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.791195+0000) 2022-01-31T22:05:09.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:08 smithi167 conmon[60316]: debug 2022-01-31T22:05:08.789+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.790023+0000) 2022-01-31T22:05:09.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[35325]: debug 2022-01-31T22:05:08.793+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:05:09.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[35325]: debug 2022-01-31T22:05:08.808+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181253 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:09.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[41853]: debug 2022-01-31T22:05:08.788+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.790001+0000) 2022-01-31T22:05:09.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[46715]: debug 2022-01-31T22:05:08.788+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.790633+0000) 2022-01-31T22:05:09.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[51620]: debug 2022-01-31T22:05:08.789+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.790895+0000) 2022-01-31T22:05:09.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:08 smithi171 conmon[51620]: debug 2022-01-31T22:05:08.943+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:08.944863+0000) 2022-01-31T22:05:09.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:09 smithi171 conmon[46715]: debug 2022-01-31T22:05:09.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:09.094023+0000) 2022-01-31T22:05:09.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:09 smithi171 conmon[41853]: debug 2022-01-31T22:05:09.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:09.300176+0000) 2022-01-31T22:05:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:09 smithi167 conmon[54076]: debug 2022-01-31T22:05:09.590+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:09.591338+0000) 2022-01-31T22:05:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:09 smithi167 conmon[60316]: debug 2022-01-31T22:05:09.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:09.591338+0000) 2022-01-31T22:05:10.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:09 smithi167 conmon[49112]: debug 2022-01-31T22:05:09.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:09.926627+0000) 2022-01-31T22:05:10.256 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:10 smithi171 conmon[46715]: debug 2022-01-31T22:05:10.092+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:10.094158+0000) 2022-01-31T22:05:10.257 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:09 smithi171 conmon[51620]: debug 2022-01-31T22:05:09.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:09.945072+0000) 2022-01-31T22:05:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:10 smithi171 conmon[41853]: debug 2022-01-31T22:05:10.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:10.300318+0000) 2022-01-31T22:05:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:10 smithi167 conmon[54076]: debug 2022-01-31T22:05:10.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:10.591487+0000) 2022-01-31T22:05:10.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:10 smithi167 conmon[60316]: debug 2022-01-31T22:05:10.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:10.591487+0000) 2022-01-31T22:05:11.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:10 smithi167 conmon[49112]: debug 2022-01-31T22:05:10.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:10.926778+0000) 2022-01-31T22:05:11.256 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:11 smithi171 conmon[46715]: debug 2022-01-31T22:05:11.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:11.094341+0000) 2022-01-31T22:05:11.257 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:10 smithi171 conmon[51620]: debug 2022-01-31T22:05:10.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:10.945269+0000) 2022-01-31T22:05:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:11 smithi171 conmon[41853]: debug 2022-01-31T22:05:11.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:11.300501+0000) 2022-01-31T22:05:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:11 smithi167 conmon[54076]: debug 2022-01-31T22:05:11.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:11.591726+0000) 2022-01-31T22:05:11.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:11 smithi167 conmon[60316]: debug 2022-01-31T22:05:11.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:11.591727+0000) 2022-01-31T22:05:12.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:11 smithi167 conmon[49112]: debug 2022-01-31T22:05:11.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:11.926952+0000) 2022-01-31T22:05:12.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:11 smithi171 conmon[51620]: debug 2022-01-31T22:05:11.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:11.945459+0000) 2022-01-31T22:05:12.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:12 smithi171 conmon[46715]: debug 2022-01-31T22:05:12.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:12.094524+0000) 2022-01-31T22:05:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:12 smithi171 conmon[41853]: debug 2022-01-31T22:05:12.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:12.300666+0000) 2022-01-31T22:05:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:12 smithi167 conmon[54076]: debug 2022-01-31T22:05:12.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:12.591898+0000) 2022-01-31T22:05:12.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:12 smithi167 conmon[60316]: debug 2022-01-31T22:05:12.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:12.591899+0000) 2022-01-31T22:05:13.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:12 smithi167 conmon[49112]: debug 2022-01-31T22:05:12.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:12.927082+0000) 2022-01-31T22:05:13.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[46715]: debug 2022-01-31T22:05:13.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.094658+0000) 2022-01-31T22:05:13.257 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:12 smithi171 conmon[51620]: debug 2022-01-31T22:05:12.944+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:12.945681+0000) 2022-01-31T22:05:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[41853]: debug 2022-01-31T22:05:13.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.300856+0000) 2022-01-31T22:05:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:13 smithi167 conmon[54076]: debug 2022-01-31T22:05:13.591+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.592081+0000) 2022-01-31T22:05:13.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:13 smithi167 conmon[60316]: debug 2022-01-31T22:05:13.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.592081+0000) 2022-01-31T22:05:14.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:13 smithi167 conmon[54076]: debug 2022-01-31T22:05:13.813+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.813974+0000) 2022-01-31T22:05:14.332 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:13 smithi167 conmon[60316]: debug 2022-01-31T22:05:13.811+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.811569+0000) 2022-01-31T22:05:14.333 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:13 smithi167 conmon[49112]: debug 2022-01-31T22:05:13.812+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.812916+0000) 2022-01-31T22:05:14.334 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:13 smithi167 conmon[49112]: debug 2022-01-31T22:05:13.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.927260+0000) 2022-01-31T22:05:14.334 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[35325]: debug 2022-01-31T22:05:13.829+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181365 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:14.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[41853]: debug 2022-01-31T22:05:13.810+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.812584+0000) 2022-01-31T22:05:14.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[46715]: debug 2022-01-31T22:05:13.809+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.811375+0000) 2022-01-31T22:05:14.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[51620]: debug 2022-01-31T22:05:13.810+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.812495+0000) 2022-01-31T22:05:14.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:13 smithi171 conmon[51620]: debug 2022-01-31T22:05:13.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:13.945885+0000) 2022-01-31T22:05:14.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:14 smithi171 conmon[41853]: debug 2022-01-31T22:05:14.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:14.301050+0000) 2022-01-31T22:05:14.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:14 smithi171 conmon[46715]: debug 2022-01-31T22:05:14.093+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:14.094848+0000) 2022-01-31T22:05:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:14 smithi167 conmon[54076]: debug 2022-01-31T22:05:14.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:14.592290+0000) 2022-01-31T22:05:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:14 smithi167 conmon[60316]: debug 2022-01-31T22:05:14.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:14.592290+0000) 2022-01-31T22:05:15.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:14 smithi167 conmon[49112]: debug 2022-01-31T22:05:14.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:14.927389+0000) 2022-01-31T22:05:15.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:15 smithi171 conmon[46715]: debug 2022-01-31T22:05:15.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:15.094974+0000) 2022-01-31T22:05:15.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:14 smithi171 conmon[51620]: debug 2022-01-31T22:05:14.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:14.946036+0000) 2022-01-31T22:05:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:15 smithi171 conmon[41853]: debug 2022-01-31T22:05:15.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:15.301197+0000) 2022-01-31T22:05:15.643 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:15 smithi167 conmon[54076]: debug 2022-01-31T22:05:15.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:15.592457+0000) 2022-01-31T22:05:15.643 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:15 smithi167 conmon[60316]: debug 2022-01-31T22:05:15.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:15.592457+0000) 2022-01-31T22:05:16.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:15 smithi167 conmon[49112]: debug 2022-01-31T22:05:15.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:15.927559+0000) 2022-01-31T22:05:16.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:16 smithi171 conmon[46715]: debug 2022-01-31T22:05:16.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:16.095150+0000) 2022-01-31T22:05:16.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:15 smithi171 conmon[51620]: debug 2022-01-31T22:05:15.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:15.946209+0000) 2022-01-31T22:05:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:16 smithi171 conmon[41853]: debug 2022-01-31T22:05:16.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:16.301397+0000) 2022-01-31T22:05:16.646 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:16 smithi167 conmon[54076]: debug 2022-01-31T22:05:16.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:16.592671+0000) 2022-01-31T22:05:16.647 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:16 smithi167 conmon[60316]: debug 2022-01-31T22:05:16.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:16.592671+0000) 2022-01-31T22:05:17.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:17 smithi171 conmon[46715]: debug 2022-01-31T22:05:17.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:17.095313+0000) 2022-01-31T22:05:17.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:16 smithi171 conmon[51620]: debug 2022-01-31T22:05:16.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:16.946400+0000) 2022-01-31T22:05:17.393 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:16 smithi167 conmon[49112]: debug 2022-01-31T22:05:16.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:16.927724+0000) 2022-01-31T22:05:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:17 smithi171 conmon[41853]: debug 2022-01-31T22:05:17.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:17.301561+0000) 2022-01-31T22:05:17.650 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:17 smithi167 conmon[54076]: debug 2022-01-31T22:05:17.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:17.592844+0000) 2022-01-31T22:05:17.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:17 smithi167 conmon[60316]: debug 2022-01-31T22:05:17.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:17.592844+0000) 2022-01-31T22:05:18.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[46715]: debug 2022-01-31T22:05:18.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.095495+0000) 2022-01-31T22:05:18.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:17 smithi171 conmon[51620]: debug 2022-01-31T22:05:17.945+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:17.946584+0000) 2022-01-31T22:05:18.393 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:17 smithi167 conmon[49112]: debug 2022-01-31T22:05:17.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:17.927834+0000) 2022-01-31T22:05:18.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[41853]: debug 2022-01-31T22:05:18.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.301715+0000) 2022-01-31T22:05:18.651 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:18 smithi167 conmon[60316]: debug 2022-01-31T22:05:18.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.593059+0000) 2022-01-31T22:05:18.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:18 smithi167 conmon[54076]: debug 2022-01-31T22:05:18.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.593059+0000) 2022-01-31T22:05:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:18 smithi167 conmon[49112]: debug 2022-01-31T22:05:18.833+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.833830+0000) 2022-01-31T22:05:18.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:18 smithi167 conmon[54076]: debug 2022-01-31T22:05:18.834+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.835082+0000) 2022-01-31T22:05:18.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:18 smithi167 conmon[60316]: debug 2022-01-31T22:05:18.833+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.834308+0000) 2022-01-31T22:05:19.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[35325]: debug 2022-01-31T22:05:18.851+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181476 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:19.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[41853]: debug 2022-01-31T22:05:18.835+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.835881+0000) 2022-01-31T22:05:19.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[46715]: debug 2022-01-31T22:05:18.832+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.833155+0000) 2022-01-31T22:05:19.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[51620]: debug 2022-01-31T22:05:18.833+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.834269+0000) 2022-01-31T22:05:19.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:18 smithi171 conmon[51620]: debug 2022-01-31T22:05:18.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.946781+0000) 2022-01-31T22:05:19.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:19 smithi171 conmon[46715]: debug 2022-01-31T22:05:19.094+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:19.095677+0000) 2022-01-31T22:05:19.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:19 smithi171 conmon[41853]: debug 2022-01-31T22:05:19.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:19.301885+0000) 2022-01-31T22:05:19.393 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:18 smithi167 conmon[49112]: debug 2022-01-31T22:05:18.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:18.927958+0000) 2022-01-31T22:05:19.652 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:19 smithi167 conmon[54076]: debug 2022-01-31T22:05:19.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:19.593278+0000) 2022-01-31T22:05:19.652 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:19 smithi167 conmon[60316]: debug 2022-01-31T22:05:19.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:19.593278+0000) 2022-01-31T22:05:20.258 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:20 smithi171 conmon[46715]: debug 2022-01-31T22:05:20.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:20.095835+0000) 2022-01-31T22:05:20.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:19 smithi171 conmon[51620]: debug 2022-01-31T22:05:19.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:19.946959+0000) 2022-01-31T22:05:20.393 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:19 smithi167 conmon[49112]: debug 2022-01-31T22:05:19.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:19.928146+0000) 2022-01-31T22:05:20.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:20 smithi171 conmon[41853]: debug 2022-01-31T22:05:20.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:20.302037+0000) 2022-01-31T22:05:20.656 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:20 smithi167 conmon[54076]: debug 2022-01-31T22:05:20.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:20.593456+0000) 2022-01-31T22:05:20.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:20 smithi167 conmon[60316]: debug 2022-01-31T22:05:20.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:20.593443+0000) 2022-01-31T22:05:21.258 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:21 smithi171 conmon[46715]: debug 2022-01-31T22:05:21.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:21.096033+0000) 2022-01-31T22:05:21.259 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:20 smithi171 conmon[51620]: debug 2022-01-31T22:05:20.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:20.947148+0000) 2022-01-31T22:05:21.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:20 smithi167 conmon[49112]: debug 2022-01-31T22:05:20.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:20.928363+0000) 2022-01-31T22:05:21.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:21 smithi171 conmon[41853]: debug 2022-01-31T22:05:21.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:21.302228+0000) 2022-01-31T22:05:21.659 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:21 smithi167 conmon[60316]: debug 2022-01-31T22:05:21.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:21.593632+0000) 2022-01-31T22:05:21.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:21 smithi167 conmon[54076]: debug 2022-01-31T22:05:21.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:21.593632+0000) 2022-01-31T22:05:22.258 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:22 smithi171 conmon[46715]: debug 2022-01-31T22:05:22.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:22.096217+0000) 2022-01-31T22:05:22.259 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:21 smithi171 conmon[51620]: debug 2022-01-31T22:05:21.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:21.947339+0000) 2022-01-31T22:05:22.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:21 smithi167 conmon[49112]: debug 2022-01-31T22:05:21.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:21.928569+0000) 2022-01-31T22:05:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:22 smithi171 conmon[41853]: debug 2022-01-31T22:05:22.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:22.302417+0000) 2022-01-31T22:05:22.662 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:22 smithi167 conmon[60316]: debug 2022-01-31T22:05:22.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:22.593816+0000) 2022-01-31T22:05:22.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:22 smithi167 conmon[54076]: debug 2022-01-31T22:05:22.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:22.593816+0000) 2022-01-31T22:05:23.258 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[46715]: debug 2022-01-31T22:05:23.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.096414+0000) 2022-01-31T22:05:23.259 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:22 smithi171 conmon[51620]: debug 2022-01-31T22:05:22.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:22.947550+0000) 2022-01-31T22:05:23.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:22 smithi167 conmon[49112]: debug 2022-01-31T22:05:22.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:22.928717+0000) 2022-01-31T22:05:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[41853]: debug 2022-01-31T22:05:23.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.302641+0000) 2022-01-31T22:05:23.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:23 smithi167 conmon[54076]: debug 2022-01-31T22:05:23.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.594033+0000) 2022-01-31T22:05:23.664 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:23 smithi167 conmon[60316]: debug 2022-01-31T22:05:23.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.594033+0000) 2022-01-31T22:05:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:23 smithi167 conmon[49112]: debug 2022-01-31T22:05:23.853+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.854745+0000) 2022-01-31T22:05:23.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:23 smithi167 conmon[54076]: debug 2022-01-31T22:05:23.852+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.854477+0000) 2022-01-31T22:05:23.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:23 smithi167 conmon[60316]: debug 2022-01-31T22:05:23.853+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.854663+0000) 2022-01-31T22:05:24.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[35325]: debug 2022-01-31T22:05:23.794+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:05:24.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[35325]: debug 2022-01-31T22:05:23.871+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181587 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:24.097 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[46715]: debug 2022-01-31T22:05:23.853+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.854593+0000) 2022-01-31T22:05:24.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[41853]: debug 2022-01-31T22:05:23.855+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.855738+0000) 2022-01-31T22:05:24.098 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[51620]: debug 2022-01-31T22:05:23.854+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.855304+0000) 2022-01-31T22:05:24.098 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:23 smithi171 conmon[51620]: debug 2022-01-31T22:05:23.946+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.947696+0000) 2022-01-31T22:05:24.356 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:24 smithi171 conmon[41853]: debug 2022-01-31T22:05:24.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:24.302769+0000) 2022-01-31T22:05:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:24 smithi171 conmon[46715]: debug 2022-01-31T22:05:24.095+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:24.096616+0000) 2022-01-31T22:05:24.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:23 smithi167 conmon[49112]: debug 2022-01-31T22:05:23.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:23.928861+0000) 2022-01-31T22:05:24.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:24 smithi167 conmon[54076]: debug 2022-01-31T22:05:24.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:24.594215+0000) 2022-01-31T22:05:24.665 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:24 smithi167 conmon[60316]: debug 2022-01-31T22:05:24.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:24.594215+0000) 2022-01-31T22:05:25.259 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:25 smithi171 conmon[46715]: debug 2022-01-31T22:05:25.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:25.096769+0000) 2022-01-31T22:05:25.260 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:24 smithi171 conmon[51620]: debug 2022-01-31T22:05:24.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:24.947875+0000) 2022-01-31T22:05:25.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:24 smithi167 conmon[49112]: debug 2022-01-31T22:05:24.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:24.928990+0000) 2022-01-31T22:05:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:25 smithi171 conmon[41853]: debug 2022-01-31T22:05:25.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:25.302912+0000) 2022-01-31T22:05:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:25 smithi167 conmon[54076]: debug 2022-01-31T22:05:25.592+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:25.594413+0000) 2022-01-31T22:05:25.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:25 smithi167 conmon[60316]: debug 2022-01-31T22:05:25.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:25.594414+0000) 2022-01-31T22:05:26.259 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:26 smithi171 conmon[46715]: debug 2022-01-31T22:05:26.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:26.096943+0000) 2022-01-31T22:05:26.260 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:25 smithi171 conmon[51620]: debug 2022-01-31T22:05:25.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:25.948070+0000) 2022-01-31T22:05:26.394 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:25 smithi167 conmon[49112]: debug 2022-01-31T22:05:25.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:25.929136+0000) 2022-01-31T22:05:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:26 smithi171 conmon[41853]: debug 2022-01-31T22:05:26.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:26.303095+0000) 2022-01-31T22:05:26.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:26 smithi167 conmon[54076]: debug 2022-01-31T22:05:26.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:26.594543+0000) 2022-01-31T22:05:26.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:26 smithi167 conmon[60316]: debug 2022-01-31T22:05:26.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:26.594543+0000) 2022-01-31T22:05:27.259 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:27 smithi171 conmon[46715]: debug 2022-01-31T22:05:27.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:27.097091+0000) 2022-01-31T22:05:27.260 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:26 smithi171 conmon[51620]: debug 2022-01-31T22:05:26.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:26.948239+0000) 2022-01-31T22:05:27.395 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:26 smithi167 conmon[49112]: debug 2022-01-31T22:05:26.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:26.929302+0000) 2022-01-31T22:05:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:27 smithi171 conmon[41853]: debug 2022-01-31T22:05:27.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:27.303307+0000) 2022-01-31T22:05:27.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:27 smithi167 conmon[60316]: debug 2022-01-31T22:05:27.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:27.594703+0000) 2022-01-31T22:05:27.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:27 smithi167 conmon[54076]: debug 2022-01-31T22:05:27.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:27.594703+0000) 2022-01-31T22:05:28.259 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[46715]: debug 2022-01-31T22:05:28.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.097247+0000) 2022-01-31T22:05:28.260 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:27 smithi171 conmon[51620]: debug 2022-01-31T22:05:27.947+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:27.948408+0000) 2022-01-31T22:05:28.395 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:27 smithi167 conmon[49112]: debug 2022-01-31T22:05:27.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:27.929496+0000) 2022-01-31T22:05:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[41853]: debug 2022-01-31T22:05:28.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.303496+0000) 2022-01-31T22:05:28.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:28 smithi167 conmon[54076]: debug 2022-01-31T22:05:28.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.594849+0000) 2022-01-31T22:05:28.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:28 smithi167 conmon[60316]: debug 2022-01-31T22:05:28.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.594849+0000) 2022-01-31T22:05:28.929 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:28 smithi167 conmon[60316]: debug 2022-01-31T22:05:28.873+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.875043+0000) 2022-01-31T22:05:28.930 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:28 smithi167 conmon[49112]: debug 2022-01-31T22:05:28.872+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.874204+0000) 2022-01-31T22:05:28.930 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:28 smithi167 conmon[54076]: debug 2022-01-31T22:05:28.873+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.874580+0000) 2022-01-31T22:05:29.259 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[35325]: debug 2022-01-31T22:05:28.893+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181698 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:29.260 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[41853]: debug 2022-01-31T22:05:28.874+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.875001+0000) 2022-01-31T22:05:29.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[46715]: debug 2022-01-31T22:05:28.873+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.873922+0000) 2022-01-31T22:05:29.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:29 smithi171 conmon[46715]: debug 2022-01-31T22:05:29.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:29.097447+0000) 2022-01-31T22:05:29.262 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[51620]: debug 2022-01-31T22:05:28.874+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.875267+0000) 2022-01-31T22:05:29.262 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:28 smithi171 conmon[51620]: debug 2022-01-31T22:05:28.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.948570+0000) 2022-01-31T22:05:29.395 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:28 smithi167 conmon[49112]: debug 2022-01-31T22:05:28.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:28.929674+0000) 2022-01-31T22:05:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:29 smithi171 conmon[41853]: debug 2022-01-31T22:05:29.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:29.303670+0000) 2022-01-31T22:05:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:29 smithi167 conmon[54076]: debug 2022-01-31T22:05:29.593+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:29.594983+0000) 2022-01-31T22:05:29.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:29 smithi167 conmon[60316]: debug 2022-01-31T22:05:29.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:29.594995+0000) 2022-01-31T22:05:30.260 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:30 smithi171 conmon[46715]: debug 2022-01-31T22:05:30.096+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:30.097616+0000) 2022-01-31T22:05:30.261 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:29 smithi171 conmon[51620]: debug 2022-01-31T22:05:29.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:29.948743+0000) 2022-01-31T22:05:30.395 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:29 smithi167 conmon[49112]: debug 2022-01-31T22:05:29.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:29.929845+0000) 2022-01-31T22:05:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:30 smithi171 conmon[41853]: debug 2022-01-31T22:05:30.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:30.303797+0000) 2022-01-31T22:05:30.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:30 smithi167 conmon[60316]: debug 2022-01-31T22:05:30.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:30.595154+0000) 2022-01-31T22:05:30.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:30 smithi167 conmon[54076]: debug 2022-01-31T22:05:30.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:30.595154+0000) 2022-01-31T22:05:30.937 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:30 smithi167 conmon[49112]: debug 2022-01-31T22:05:30.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:30.930060+0000) 2022-01-31T22:05:31.260 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:31 smithi171 conmon[46715]: debug 2022-01-31T22:05:31.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:31.097804+0000) 2022-01-31T22:05:31.261 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:30 smithi171 conmon[51620]: debug 2022-01-31T22:05:30.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:30.948951+0000) 2022-01-31T22:05:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:31 smithi171 conmon[41853]: debug 2022-01-31T22:05:31.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:31.303952+0000) 2022-01-31T22:05:31.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:31 smithi167 conmon[60316]: debug 2022-01-31T22:05:31.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:31.595339+0000) 2022-01-31T22:05:31.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:31 smithi167 conmon[54076]: debug 2022-01-31T22:05:31.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:31.595339+0000) 2022-01-31T22:05:31.937 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:31 smithi167 conmon[49112]: debug 2022-01-31T22:05:31.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:31.930275+0000) 2022-01-31T22:05:32.260 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:32 smithi171 conmon[46715]: debug 2022-01-31T22:05:32.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:32.097988+0000) 2022-01-31T22:05:32.261 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:31 smithi171 conmon[51620]: debug 2022-01-31T22:05:31.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:31.949137+0000) 2022-01-31T22:05:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:32 smithi171 conmon[41853]: debug 2022-01-31T22:05:32.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:32.304138+0000) 2022-01-31T22:05:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:32 smithi167 conmon[54076]: debug 2022-01-31T22:05:32.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:32.595544+0000) 2022-01-31T22:05:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:32 smithi167 conmon[60316]: debug 2022-01-31T22:05:32.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:32.595544+0000) 2022-01-31T22:05:33.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:32 smithi167 conmon[49112]: debug 2022-01-31T22:05:32.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:32.930492+0000) 2022-01-31T22:05:33.260 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[46715]: debug 2022-01-31T22:05:33.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.098177+0000) 2022-01-31T22:05:33.261 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:32 smithi171 conmon[51620]: debug 2022-01-31T22:05:32.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:32.949352+0000) 2022-01-31T22:05:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[41853]: debug 2022-01-31T22:05:33.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.304439+0000) 2022-01-31T22:05:33.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:33 smithi167 conmon[54076]: debug 2022-01-31T22:05:33.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.595800+0000) 2022-01-31T22:05:33.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:33 smithi167 conmon[60316]: debug 2022-01-31T22:05:33.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.595800+0000) 2022-01-31T22:05:34.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:33 smithi167 conmon[54076]: debug 2022-01-31T22:05:33.903+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.905032+0000) 2022-01-31T22:05:34.087 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:33 smithi167 conmon[60316]: debug 2022-01-31T22:05:33.903+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.904533+0000) 2022-01-31T22:05:34.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:33 smithi167 conmon[49112]: debug 2022-01-31T22:05:33.905+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.907241+0000) 2022-01-31T22:05:34.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:33 smithi167 conmon[49112]: debug 2022-01-31T22:05:33.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.930622+0000) 2022-01-31T22:05:34.260 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[35325]: debug 2022-01-31T22:05:33.920+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:34.261 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[41853]: debug 2022-01-31T22:05:33.895+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.896408+0000) 2022-01-31T22:05:34.262 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[46715]: debug 2022-01-31T22:05:33.895+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.896281+0000) 2022-01-31T22:05:34.262 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:34 smithi171 conmon[46715]: debug 2022-01-31T22:05:34.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:34.098364+0000) 2022-01-31T22:05:34.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[51620]: debug 2022-01-31T22:05:33.896+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.897388+0000) 2022-01-31T22:05:34.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:33 smithi171 conmon[51620]: debug 2022-01-31T22:05:33.948+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:33.949565+0000) 2022-01-31T22:05:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:34 smithi171 conmon[41853]: debug 2022-01-31T22:05:34.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:34.304575+0000) 2022-01-31T22:05:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:34 smithi167 conmon[54076]: debug 2022-01-31T22:05:34.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:34.595950+0000) 2022-01-31T22:05:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:34 smithi167 conmon[60316]: debug 2022-01-31T22:05:34.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:34.595950+0000) 2022-01-31T22:05:35.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:34 smithi167 conmon[49112]: debug 2022-01-31T22:05:34.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:34.930744+0000) 2022-01-31T22:05:35.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:35 smithi171 conmon[46715]: debug 2022-01-31T22:05:35.097+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:35.098526+0000) 2022-01-31T22:05:35.261 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:34 smithi171 conmon[51620]: debug 2022-01-31T22:05:34.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:34.949704+0000) 2022-01-31T22:05:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:35 smithi171 conmon[41853]: debug 2022-01-31T22:05:35.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:35.304683+0000) 2022-01-31T22:05:35.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:35 smithi167 conmon[60316]: debug 2022-01-31T22:05:35.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:35.596090+0000) 2022-01-31T22:05:35.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:35 smithi167 conmon[54076]: debug 2022-01-31T22:05:35.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:35.596090+0000) 2022-01-31T22:05:36.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:35 smithi167 conmon[49112]: debug 2022-01-31T22:05:35.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:35.930935+0000) 2022-01-31T22:05:36.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:36 smithi171 conmon[46715]: debug 2022-01-31T22:05:36.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:36.098705+0000) 2022-01-31T22:05:36.262 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:35 smithi171 conmon[51620]: debug 2022-01-31T22:05:35.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:35.949839+0000) 2022-01-31T22:05:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:36 smithi171 conmon[41853]: debug 2022-01-31T22:05:36.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:36.304867+0000) 2022-01-31T22:05:36.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:36 smithi167 conmon[60316]: debug 2022-01-31T22:05:36.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:36.596234+0000) 2022-01-31T22:05:36.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:36 smithi167 conmon[54076]: debug 2022-01-31T22:05:36.594+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:36.596234+0000) 2022-01-31T22:05:37.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:36 smithi167 conmon[49112]: debug 2022-01-31T22:05:36.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:36.931098+0000) 2022-01-31T22:05:37.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:37 smithi171 conmon[46715]: debug 2022-01-31T22:05:37.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:37.098859+0000) 2022-01-31T22:05:37.262 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:36 smithi171 conmon[51620]: debug 2022-01-31T22:05:36.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:36.950031+0000) 2022-01-31T22:05:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:37 smithi171 conmon[41853]: debug 2022-01-31T22:05:37.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:37.305019+0000) 2022-01-31T22:05:37.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:37 smithi167 conmon[60316]: debug 2022-01-31T22:05:37.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:37.596397+0000) 2022-01-31T22:05:37.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:37 smithi167 conmon[54076]: debug 2022-01-31T22:05:37.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:37.596397+0000) 2022-01-31T22:05:38.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:37 smithi167 conmon[49112]: debug 2022-01-31T22:05:37.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:37.931299+0000) 2022-01-31T22:05:38.261 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[46715]: debug 2022-01-31T22:05:38.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.099049+0000) 2022-01-31T22:05:38.262 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:37 smithi171 conmon[51620]: debug 2022-01-31T22:05:37.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:37.950250+0000) 2022-01-31T22:05:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[41853]: debug 2022-01-31T22:05:38.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.305187+0000) 2022-01-31T22:05:38.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:38 smithi167 conmon[60316]: debug 2022-01-31T22:05:38.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.596609+0000) 2022-01-31T22:05:38.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:38 smithi167 conmon[54076]: debug 2022-01-31T22:05:38.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.596609+0000) 2022-01-31T22:05:39.087 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:38 smithi167 conmon[54076]: debug 2022-01-31T22:05:38.922+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.924124+0000) 2022-01-31T22:05:39.088 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:38 smithi167 conmon[60316]: debug 2022-01-31T22:05:38.922+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.924033+0000) 2022-01-31T22:05:39.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:38 smithi167 conmon[49112]: debug 2022-01-31T22:05:38.922+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.924339+0000) 2022-01-31T22:05:39.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:38 smithi167 conmon[49112]: debug 2022-01-31T22:05:38.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.931563+0000) 2022-01-31T22:05:39.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[41853]: debug 2022-01-31T22:05:38.922+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.923679+0000) 2022-01-31T22:05:39.099 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[46715]: debug 2022-01-31T22:05:38.922+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.923173+0000) 2022-01-31T22:05:39.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[35325]: debug 2022-01-31T22:05:38.795+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:05:39.100 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[35325]: debug 2022-01-31T22:05:38.941+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 181921 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:39.100 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[51620]: debug 2022-01-31T22:05:38.923+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.923941+0000) 2022-01-31T22:05:39.101 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:38 smithi171 conmon[51620]: debug 2022-01-31T22:05:38.949+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:38.950377+0000) 2022-01-31T22:05:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:39 smithi171 conmon[46715]: debug 2022-01-31T22:05:39.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:39.099253+0000) 2022-01-31T22:05:39.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:39 smithi171 conmon[41853]: debug 2022-01-31T22:05:39.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:39.305385+0000) 2022-01-31T22:05:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:39 smithi167 conmon[54076]: debug 2022-01-31T22:05:39.595+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:39.596790+0000) 2022-01-31T22:05:39.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:39 smithi167 conmon[60316]: debug 2022-01-31T22:05:39.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:39.596790+0000) 2022-01-31T22:05:40.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:39 smithi167 conmon[49112]: debug 2022-01-31T22:05:39.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:39.931695+0000) 2022-01-31T22:05:40.262 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:40 smithi171 conmon[46715]: debug 2022-01-31T22:05:40.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:40.099402+0000) 2022-01-31T22:05:40.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:39 smithi171 conmon[51620]: debug 2022-01-31T22:05:39.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:39.950557+0000) 2022-01-31T22:05:40.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:40 smithi171 conmon[41853]: debug 2022-01-31T22:05:40.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:40.305557+0000) 2022-01-31T22:05:40.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:40 smithi167 conmon[54076]: debug 2022-01-31T22:05:40.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:40.596955+0000) 2022-01-31T22:05:40.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:40 smithi167 conmon[60316]: debug 2022-01-31T22:05:40.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:40.596955+0000) 2022-01-31T22:05:41.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:40 smithi167 conmon[49112]: debug 2022-01-31T22:05:40.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:40.931882+0000) 2022-01-31T22:05:41.262 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:41 smithi171 conmon[46715]: debug 2022-01-31T22:05:41.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:41.099562+0000) 2022-01-31T22:05:41.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:40 smithi171 conmon[51620]: debug 2022-01-31T22:05:40.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:40.950785+0000) 2022-01-31T22:05:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:41 smithi171 conmon[41853]: debug 2022-01-31T22:05:41.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:41.305726+0000) 2022-01-31T22:05:41.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:41 smithi167 conmon[54076]: debug 2022-01-31T22:05:41.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:41.597189+0000) 2022-01-31T22:05:41.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:41 smithi167 conmon[60316]: debug 2022-01-31T22:05:41.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:41.597189+0000) 2022-01-31T22:05:42.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:41 smithi167 conmon[49112]: debug 2022-01-31T22:05:41.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:41.932044+0000) 2022-01-31T22:05:42.262 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:42 smithi171 conmon[46715]: debug 2022-01-31T22:05:42.098+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:42.099746+0000) 2022-01-31T22:05:42.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:41 smithi171 conmon[51620]: debug 2022-01-31T22:05:41.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:41.950910+0000) 2022-01-31T22:05:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:42 smithi171 conmon[41853]: debug 2022-01-31T22:05:42.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:42.305914+0000) 2022-01-31T22:05:42.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:42 smithi167 conmon[54076]: debug 2022-01-31T22:05:42.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:42.597375+0000) 2022-01-31T22:05:42.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:42 smithi167 conmon[60316]: debug 2022-01-31T22:05:42.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:42.597375+0000) 2022-01-31T22:05:43.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:42 smithi167 conmon[49112]: debug 2022-01-31T22:05:42.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:42.932193+0000) 2022-01-31T22:05:43.262 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[46715]: debug 2022-01-31T22:05:43.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.099944+0000) 2022-01-31T22:05:43.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:42 smithi171 conmon[51620]: debug 2022-01-31T22:05:42.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:42.951100+0000) 2022-01-31T22:05:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[41853]: debug 2022-01-31T22:05:43.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.306105+0000) 2022-01-31T22:05:43.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:43 smithi167 conmon[60316]: debug 2022-01-31T22:05:43.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.597536+0000) 2022-01-31T22:05:43.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:43 smithi167 conmon[54076]: debug 2022-01-31T22:05:43.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.597536+0000) 2022-01-31T22:05:44.088 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:43 smithi167 conmon[54076]: debug 2022-01-31T22:05:43.944+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.945819+0000) 2022-01-31T22:05:44.089 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:43 smithi167 conmon[60316]: debug 2022-01-31T22:05:43.944+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.945953+0000) 2022-01-31T22:05:44.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:43 smithi167 conmon[49112]: debug 2022-01-31T22:05:43.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.932374+0000) 2022-01-31T22:05:44.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:43 smithi167 conmon[49112]: debug 2022-01-31T22:05:43.944+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.945623+0000) 2022-01-31T22:05:44.262 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[35325]: debug 2022-01-31T22:05:43.963+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182031 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:44.263 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[41853]: debug 2022-01-31T22:05:43.943+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.944372+0000) 2022-01-31T22:05:44.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[46715]: debug 2022-01-31T22:05:43.944+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.944932+0000) 2022-01-31T22:05:44.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:44 smithi171 conmon[46715]: debug 2022-01-31T22:05:44.099+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:44.100136+0000) 2022-01-31T22:05:44.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[51620]: debug 2022-01-31T22:05:43.944+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.945095+0000) 2022-01-31T22:05:44.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:43 smithi171 conmon[51620]: debug 2022-01-31T22:05:43.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:43.951275+0000) 2022-01-31T22:05:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:44 smithi171 conmon[41853]: debug 2022-01-31T22:05:44.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:44.306294+0000) 2022-01-31T22:05:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:44 smithi167 conmon[54076]: debug 2022-01-31T22:05:44.596+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:44.597724+0000) 2022-01-31T22:05:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:44 smithi167 conmon[60316]: debug 2022-01-31T22:05:44.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:44.597722+0000) 2022-01-31T22:05:45.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:44 smithi167 conmon[49112]: debug 2022-01-31T22:05:44.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:44.932547+0000) 2022-01-31T22:05:45.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:45 smithi171 conmon[46715]: debug 2022-01-31T22:05:45.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:45.100277+0000) 2022-01-31T22:05:45.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:44 smithi171 conmon[51620]: debug 2022-01-31T22:05:44.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:44.951436+0000) 2022-01-31T22:05:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:45 smithi171 conmon[41853]: debug 2022-01-31T22:05:45.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:45.306449+0000) 2022-01-31T22:05:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:45 smithi167 conmon[60316]: debug 2022-01-31T22:05:45.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:45.597895+0000) 2022-01-31T22:05:45.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:45 smithi167 conmon[54076]: debug 2022-01-31T22:05:45.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:45.597894+0000) 2022-01-31T22:05:46.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:45 smithi167 conmon[49112]: debug 2022-01-31T22:05:45.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:45.932664+0000) 2022-01-31T22:05:46.263 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:46 smithi171 conmon[46715]: debug 2022-01-31T22:05:46.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:46.100469+0000) 2022-01-31T22:05:46.263 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:45 smithi171 conmon[51620]: debug 2022-01-31T22:05:45.950+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:45.951638+0000) 2022-01-31T22:05:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:46 smithi171 conmon[41853]: debug 2022-01-31T22:05:46.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:46.306627+0000) 2022-01-31T22:05:46.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:46 smithi167 conmon[54076]: debug 2022-01-31T22:05:46.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:46.598064+0000) 2022-01-31T22:05:46.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:46 smithi167 conmon[60316]: debug 2022-01-31T22:05:46.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:46.598064+0000) 2022-01-31T22:05:47.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:46 smithi167 conmon[49112]: debug 2022-01-31T22:05:46.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:46.932856+0000) 2022-01-31T22:05:47.263 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:47 smithi171 conmon[46715]: debug 2022-01-31T22:05:47.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:47.100657+0000) 2022-01-31T22:05:47.264 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:46 smithi171 conmon[51620]: debug 2022-01-31T22:05:46.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:46.951824+0000) 2022-01-31T22:05:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:47 smithi171 conmon[41853]: debug 2022-01-31T22:05:47.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:47.306815+0000) 2022-01-31T22:05:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:47 smithi167 conmon[54076]: debug 2022-01-31T22:05:47.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:47.598245+0000) 2022-01-31T22:05:47.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:47 smithi167 conmon[60316]: debug 2022-01-31T22:05:47.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:47.598245+0000) 2022-01-31T22:05:48.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:47 smithi167 conmon[49112]: debug 2022-01-31T22:05:47.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:47.933034+0000) 2022-01-31T22:05:48.263 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[46715]: debug 2022-01-31T22:05:48.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.100824+0000) 2022-01-31T22:05:48.264 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:47 smithi171 conmon[51620]: debug 2022-01-31T22:05:47.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:47.951977+0000) 2022-01-31T22:05:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[41853]: debug 2022-01-31T22:05:48.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.306943+0000) 2022-01-31T22:05:48.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:48 smithi167 conmon[54076]: debug 2022-01-31T22:05:48.597+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.598440+0000) 2022-01-31T22:05:48.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:48 smithi167 conmon[60316]: debug 2022-01-31T22:05:48.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.598440+0000) 2022-01-31T22:05:49.089 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:48 smithi167 conmon[54076]: debug 2022-01-31T22:05:48.966+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.967165+0000) 2022-01-31T22:05:49.090 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:48 smithi167 conmon[60316]: debug 2022-01-31T22:05:48.966+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.967387+0000) 2022-01-31T22:05:49.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:48 smithi167 conmon[49112]: debug 2022-01-31T22:05:48.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.933238+0000) 2022-01-31T22:05:49.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:48 smithi167 conmon[49112]: debug 2022-01-31T22:05:48.966+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.967324+0000) 2022-01-31T22:05:49.263 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[35325]: debug 2022-01-31T22:05:48.983+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182143 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:49.264 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[41853]: debug 2022-01-31T22:05:48.965+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.965887+0000) 2022-01-31T22:05:49.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[46715]: debug 2022-01-31T22:05:48.965+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.966361+0000) 2022-01-31T22:05:49.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:49 smithi171 conmon[46715]: debug 2022-01-31T22:05:49.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:49.100949+0000) 2022-01-31T22:05:49.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[51620]: debug 2022-01-31T22:05:48.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.952156+0000) 2022-01-31T22:05:49.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:48 smithi171 conmon[51620]: debug 2022-01-31T22:05:48.965+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:48.966500+0000) 2022-01-31T22:05:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:49 smithi171 conmon[41853]: debug 2022-01-31T22:05:49.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:49.307078+0000) 2022-01-31T22:05:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:49 smithi167 conmon[60316]: debug 2022-01-31T22:05:49.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:49.598650+0000) 2022-01-31T22:05:49.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:49 smithi167 conmon[54076]: debug 2022-01-31T22:05:49.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:49.598650+0000) 2022-01-31T22:05:50.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:49 smithi167 conmon[49112]: debug 2022-01-31T22:05:49.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:49.933454+0000) 2022-01-31T22:05:50.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:50 smithi171 conmon[46715]: debug 2022-01-31T22:05:50.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:50.101122+0000) 2022-01-31T22:05:50.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:49 smithi171 conmon[51620]: debug 2022-01-31T22:05:49.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:49.952357+0000) 2022-01-31T22:05:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:50 smithi171 conmon[41853]: debug 2022-01-31T22:05:50.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:50.307226+0000) 2022-01-31T22:05:50.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:50 smithi167 conmon[54076]: debug 2022-01-31T22:05:50.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:50.598826+0000) 2022-01-31T22:05:50.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:50 smithi167 conmon[60316]: debug 2022-01-31T22:05:50.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:50.598829+0000) 2022-01-31T22:05:51.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:50 smithi167 conmon[49112]: debug 2022-01-31T22:05:50.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:50.933603+0000) 2022-01-31T22:05:51.264 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:51 smithi171 conmon[46715]: debug 2022-01-31T22:05:51.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:51.101284+0000) 2022-01-31T22:05:51.264 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:50 smithi171 conmon[51620]: debug 2022-01-31T22:05:50.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:50.952548+0000) 2022-01-31T22:05:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:51 smithi171 conmon[41853]: debug 2022-01-31T22:05:51.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:51.307427+0000) 2022-01-31T22:05:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:51 smithi167 conmon[54076]: debug 2022-01-31T22:05:51.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:51.599027+0000) 2022-01-31T22:05:51.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:51 smithi167 conmon[60316]: debug 2022-01-31T22:05:51.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:51.599030+0000) 2022-01-31T22:05:52.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:51 smithi167 conmon[49112]: debug 2022-01-31T22:05:51.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:51.933791+0000) 2022-01-31T22:05:52.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:52 smithi171 conmon[46715]: debug 2022-01-31T22:05:52.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:52.101486+0000) 2022-01-31T22:05:52.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:51 smithi171 conmon[51620]: debug 2022-01-31T22:05:51.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:51.952745+0000) 2022-01-31T22:05:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:52 smithi171 conmon[41853]: debug 2022-01-31T22:05:52.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:52.307609+0000) 2022-01-31T22:05:52.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:52 smithi167 conmon[60316]: debug 2022-01-31T22:05:52.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:52.599228+0000) 2022-01-31T22:05:52.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:52 smithi167 conmon[54076]: debug 2022-01-31T22:05:52.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:52.599228+0000) 2022-01-31T22:05:53.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:52 smithi167 conmon[49112]: debug 2022-01-31T22:05:52.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:52.933985+0000) 2022-01-31T22:05:53.266 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[46715]: debug 2022-01-31T22:05:53.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.101673+0000) 2022-01-31T22:05:53.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:52 smithi171 conmon[51620]: debug 2022-01-31T22:05:52.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:52.952885+0000) 2022-01-31T22:05:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[41853]: debug 2022-01-31T22:05:53.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.307832+0000) 2022-01-31T22:05:53.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:53 smithi167 conmon[54076]: debug 2022-01-31T22:05:53.598+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.599409+0000) 2022-01-31T22:05:53.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:53 smithi167 conmon[60316]: debug 2022-01-31T22:05:53.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.599409+0000) 2022-01-31T22:05:54.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:53 smithi167 conmon[54076]: debug 2022-01-31T22:05:53.986+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.986859+0000) 2022-01-31T22:05:54.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:53 smithi167 conmon[60316]: debug 2022-01-31T22:05:53.986+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.987315+0000) 2022-01-31T22:05:54.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:53 smithi167 conmon[49112]: debug 2022-01-31T22:05:53.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.934151+0000) 2022-01-31T22:05:54.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:53 smithi167 conmon[49112]: debug 2022-01-31T22:05:53.987+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.987751+0000) 2022-01-31T22:05:54.101 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[41853]: debug 2022-01-31T22:05:53.984+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.986367+0000) 2022-01-31T22:05:54.102 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[46715]: debug 2022-01-31T22:05:53.984+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.986619+0000) 2022-01-31T22:05:54.103 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[51620]: debug 2022-01-31T22:05:53.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.952990+0000) 2022-01-31T22:05:54.104 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[51620]: debug 2022-01-31T22:05:53.985+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:53.987671+0000) 2022-01-31T22:05:54.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:53 smithi171 conmon[35325]: debug 2022-01-31T22:05:53.795+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:05:54.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:54 smithi171 conmon[35325]: debug 2022-01-31T22:05:54.004+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182254 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:54 smithi171 conmon[46715]: debug 2022-01-31T22:05:54.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:54.101868+0000) 2022-01-31T22:05:54.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:54 smithi171 conmon[41853]: debug 2022-01-31T22:05:54.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:54.308028+0000) 2022-01-31T22:05:54.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:54 smithi167 conmon[54076]: debug 2022-01-31T22:05:54.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:54.599595+0000) 2022-01-31T22:05:54.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:54 smithi167 conmon[60316]: debug 2022-01-31T22:05:54.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:54.599595+0000) 2022-01-31T22:05:55.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:54 smithi167 conmon[49112]: debug 2022-01-31T22:05:54.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:54.934336+0000) 2022-01-31T22:05:55.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:55 smithi171 conmon[46715]: debug 2022-01-31T22:05:55.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:55.101976+0000) 2022-01-31T22:05:55.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:54 smithi171 conmon[51620]: debug 2022-01-31T22:05:54.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:54.953183+0000) 2022-01-31T22:05:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:55 smithi171 conmon[41853]: debug 2022-01-31T22:05:55.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:55.308176+0000) 2022-01-31T22:05:55.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:55 smithi167 conmon[54076]: debug 2022-01-31T22:05:55.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:55.599706+0000) 2022-01-31T22:05:55.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:55 smithi167 conmon[60316]: debug 2022-01-31T22:05:55.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:55.599706+0000) 2022-01-31T22:05:56.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:55 smithi167 conmon[49112]: debug 2022-01-31T22:05:55.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:55.934530+0000) 2022-01-31T22:05:56.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:56 smithi171 conmon[46715]: debug 2022-01-31T22:05:56.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:56.102173+0000) 2022-01-31T22:05:56.265 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:55 smithi171 conmon[51620]: debug 2022-01-31T22:05:55.951+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:55.953336+0000) 2022-01-31T22:05:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:56 smithi171 conmon[41853]: debug 2022-01-31T22:05:56.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:56.308307+0000) 2022-01-31T22:05:56.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:56 smithi167 conmon[60316]: debug 2022-01-31T22:05:56.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:56.599889+0000) 2022-01-31T22:05:56.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:56 smithi167 conmon[54076]: debug 2022-01-31T22:05:56.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:56.599891+0000) 2022-01-31T22:05:57.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:56 smithi167 conmon[49112]: debug 2022-01-31T22:05:56.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:56.934734+0000) 2022-01-31T22:05:57.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:57 smithi171 conmon[46715]: debug 2022-01-31T22:05:57.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:57.102382+0000) 2022-01-31T22:05:57.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:56 smithi171 conmon[51620]: debug 2022-01-31T22:05:56.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:56.953511+0000) 2022-01-31T22:05:57.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:57 smithi171 conmon[41853]: debug 2022-01-31T22:05:57.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:57.308467+0000) 2022-01-31T22:05:57.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:57 smithi167 conmon[60316]: debug 2022-01-31T22:05:57.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:57.600102+0000) 2022-01-31T22:05:57.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:57 smithi167 conmon[54076]: debug 2022-01-31T22:05:57.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:57.600100+0000) 2022-01-31T22:05:58.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:57 smithi167 conmon[49112]: debug 2022-01-31T22:05:57.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:57.934933+0000) 2022-01-31T22:05:58.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:58 smithi171 conmon[46715]: debug 2022-01-31T22:05:58.100+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:58.102581+0000) 2022-01-31T22:05:58.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:57 smithi171 conmon[51620]: debug 2022-01-31T22:05:57.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:57.953722+0000) 2022-01-31T22:05:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:58 smithi171 conmon[41853]: debug 2022-01-31T22:05:58.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:58.308655+0000) 2022-01-31T22:05:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:58 smithi167 conmon[54076]: debug 2022-01-31T22:05:58.599+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:58.600305+0000) 2022-01-31T22:05:58.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:58 smithi167 conmon[60316]: debug 2022-01-31T22:05:58.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:58.600304+0000) 2022-01-31T22:05:59.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:58 smithi167 conmon[49112]: debug 2022-01-31T22:05:58.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:58.935125+0000) 2022-01-31T22:05:59.265 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[35325]: debug 2022-01-31T22:05:59.026+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182365 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:05:59.266 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[41853]: debug 2022-01-31T22:05:59.007+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.009095+0000) 2022-01-31T22:05:59.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:58 smithi171 conmon[51620]: debug 2022-01-31T22:05:58.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:58.953895+0000) 2022-01-31T22:05:59.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[51620]: debug 2022-01-31T22:05:59.006+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.008272+0000) 2022-01-31T22:05:59.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[46715]: debug 2022-01-31T22:05:59.007+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.008925+0000) 2022-01-31T22:05:59.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[46715]: debug 2022-01-31T22:05:59.101+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.102769+0000) 2022-01-31T22:05:59.402 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:59 smithi167 conmon[49112]: debug 2022-01-31T22:05:59.009+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.010119+0000) 2022-01-31T22:05:59.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:59 smithi167 conmon[54076]: debug 2022-01-31T22:05:59.008+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.009475+0000) 2022-01-31T22:05:59.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:59 smithi167 conmon[60316]: debug 2022-01-31T22:05:59.008+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.009130+0000) 2022-01-31T22:05:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[41853]: debug 2022-01-31T22:05:59.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.308839+0000) 2022-01-31T22:05:59.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:05:59 smithi167 conmon[60316]: debug 2022-01-31T22:05:59.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.600537+0000) 2022-01-31T22:05:59.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:05:59 smithi167 conmon[54076]: debug 2022-01-31T22:05:59.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.600536+0000) 2022-01-31T22:06:00.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:05:59 smithi167 conmon[49112]: debug 2022-01-31T22:05:59.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.935299+0000) 2022-01-31T22:06:00.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:00 smithi171 conmon[46715]: debug 2022-01-31T22:06:00.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:00.102927+0000) 2022-01-31T22:06:00.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:05:59 smithi171 conmon[51620]: debug 2022-01-31T22:05:59.952+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:05:59.954091+0000) 2022-01-31T22:06:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:00 smithi171 conmon[41853]: debug 2022-01-31T22:06:00.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:00.308930+0000) 2022-01-31T22:06:00.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:00 smithi167 conmon[60316]: debug 2022-01-31T22:06:00.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:00.600709+0000) 2022-01-31T22:06:00.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:00 smithi167 conmon[54076]: debug 2022-01-31T22:06:00.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:00.600716+0000) 2022-01-31T22:06:01.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:00 smithi167 conmon[49112]: debug 2022-01-31T22:06:00.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:00.935483+0000) 2022-01-31T22:06:01.265 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:01 smithi171 conmon[46715]: debug 2022-01-31T22:06:01.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:01.103117+0000) 2022-01-31T22:06:01.266 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:00 smithi171 conmon[51620]: debug 2022-01-31T22:06:00.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:00.954254+0000) 2022-01-31T22:06:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:01 smithi171 conmon[41853]: debug 2022-01-31T22:06:01.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:01.309122+0000) 2022-01-31T22:06:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:01 smithi167 conmon[54076]: debug 2022-01-31T22:06:01.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:01.600873+0000) 2022-01-31T22:06:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:01 smithi167 conmon[60316]: debug 2022-01-31T22:06:01.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:01.600873+0000) 2022-01-31T22:06:02.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:01 smithi167 conmon[49112]: debug 2022-01-31T22:06:01.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:01.935714+0000) 2022-01-31T22:06:02.266 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:02 smithi171 conmon[46715]: debug 2022-01-31T22:06:02.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:02.103306+0000) 2022-01-31T22:06:02.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:01 smithi171 conmon[51620]: debug 2022-01-31T22:06:01.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:01.954458+0000) 2022-01-31T22:06:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:02 smithi171 conmon[41853]: debug 2022-01-31T22:06:02.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:02.309306+0000) 2022-01-31T22:06:02.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:02 smithi167 conmon[60316]: debug 2022-01-31T22:06:02.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:02.601059+0000) 2022-01-31T22:06:02.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:02 smithi167 conmon[54076]: debug 2022-01-31T22:06:02.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:02.601059+0000) 2022-01-31T22:06:03.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:02 smithi167 conmon[49112]: debug 2022-01-31T22:06:02.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:02.935892+0000) 2022-01-31T22:06:03.266 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:03 smithi171 conmon[46715]: debug 2022-01-31T22:06:03.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:03.103460+0000) 2022-01-31T22:06:03.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:02 smithi171 conmon[51620]: debug 2022-01-31T22:06:02.953+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:02.954642+0000) 2022-01-31T22:06:03.584 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:03 smithi171 conmon[41853]: debug 2022-01-31T22:06:03.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:03.309506+0000) 2022-01-31T22:06:03.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:03 smithi167 conmon[60316]: debug 2022-01-31T22:06:03.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:03.601240+0000) 2022-01-31T22:06:03.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:03 smithi167 conmon[54076]: debug 2022-01-31T22:06:03.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:03.601242+0000) 2022-01-31T22:06:04.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:03 smithi167 conmon[49112]: debug 2022-01-31T22:06:03.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:03.936062+0000) 2022-01-31T22:06:04.266 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[35325]: debug 2022-01-31T22:06:04.048+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182476 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:04.267 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[41853]: debug 2022-01-31T22:06:04.029+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.029880+0000) 2022-01-31T22:06:04.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:03 smithi171 conmon[51620]: debug 2022-01-31T22:06:03.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:03.954840+0000) 2022-01-31T22:06:04.268 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[51620]: debug 2022-01-31T22:06:04.029+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.030305+0000) 2022-01-31T22:06:04.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[46715]: debug 2022-01-31T22:06:04.030+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.031340+0000) 2022-01-31T22:06:04.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[46715]: debug 2022-01-31T22:06:04.102+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.103668+0000) 2022-01-31T22:06:04.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:04 smithi167 conmon[49112]: debug 2022-01-31T22:06:04.030+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.031285+0000) 2022-01-31T22:06:04.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:04 smithi167 conmon[54076]: debug 2022-01-31T22:06:04.029+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.030075+0000) 2022-01-31T22:06:04.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:04 smithi167 conmon[60316]: debug 2022-01-31T22:06:04.030+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.030598+0000) 2022-01-31T22:06:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[41853]: debug 2022-01-31T22:06:04.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.309717+0000) 2022-01-31T22:06:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:04 smithi167 conmon[54076]: debug 2022-01-31T22:06:04.600+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.601465+0000) 2022-01-31T22:06:04.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:04 smithi167 conmon[60316]: debug 2022-01-31T22:06:04.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.601465+0000) 2022-01-31T22:06:05.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:04 smithi167 conmon[49112]: debug 2022-01-31T22:06:04.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.936250+0000) 2022-01-31T22:06:05.266 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:05 smithi171 conmon[46715]: debug 2022-01-31T22:06:05.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:05.103831+0000) 2022-01-31T22:06:05.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:04 smithi171 conmon[51620]: debug 2022-01-31T22:06:04.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:04.955024+0000) 2022-01-31T22:06:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:05 smithi171 conmon[41853]: debug 2022-01-31T22:06:05.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:05.309896+0000) 2022-01-31T22:06:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:05 smithi167 conmon[54076]: debug 2022-01-31T22:06:05.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:05.601573+0000) 2022-01-31T22:06:05.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:05 smithi167 conmon[60316]: debug 2022-01-31T22:06:05.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:05.601763+0000) 2022-01-31T22:06:06.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:05 smithi167 conmon[49112]: debug 2022-01-31T22:06:05.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:05.936419+0000) 2022-01-31T22:06:06.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:06 smithi171 conmon[46715]: debug 2022-01-31T22:06:06.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:06.104016+0000) 2022-01-31T22:06:06.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:05 smithi171 conmon[51620]: debug 2022-01-31T22:06:05.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:05.955160+0000) 2022-01-31T22:06:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:06 smithi171 conmon[41853]: debug 2022-01-31T22:06:06.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:06.310055+0000) 2022-01-31T22:06:06.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:06 smithi167 conmon[54076]: debug 2022-01-31T22:06:06.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:06.601736+0000) 2022-01-31T22:06:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:06 smithi167 conmon[60316]: debug 2022-01-31T22:06:06.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:06.602542+0000) 2022-01-31T22:06:07.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:06 smithi167 conmon[49112]: debug 2022-01-31T22:06:06.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:06.936605+0000) 2022-01-31T22:06:07.266 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:07 smithi171 conmon[46715]: debug 2022-01-31T22:06:07.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:07.104178+0000) 2022-01-31T22:06:07.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:06 smithi171 conmon[51620]: debug 2022-01-31T22:06:06.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:06.955355+0000) 2022-01-31T22:06:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:07 smithi171 conmon[41853]: debug 2022-01-31T22:06:07.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:07.310206+0000) 2022-01-31T22:06:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:07 smithi167 conmon[54076]: debug 2022-01-31T22:06:07.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:07.601918+0000) 2022-01-31T22:06:07.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:07 smithi167 conmon[60316]: debug 2022-01-31T22:06:07.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:07.602648+0000) 2022-01-31T22:06:08.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:07 smithi167 conmon[49112]: debug 2022-01-31T22:06:07.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:07.936765+0000) 2022-01-31T22:06:08.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:08 smithi171 conmon[46715]: debug 2022-01-31T22:06:08.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:08.104329+0000) 2022-01-31T22:06:08.267 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:07 smithi171 conmon[51620]: debug 2022-01-31T22:06:07.954+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:07.955599+0000) 2022-01-31T22:06:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:08 smithi171 conmon[41853]: debug 2022-01-31T22:06:08.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:08.310400+0000) 2022-01-31T22:06:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:08 smithi167 conmon[54076]: debug 2022-01-31T22:06:08.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:08.602087+0000) 2022-01-31T22:06:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:08 smithi167 conmon[60316]: debug 2022-01-31T22:06:08.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:08.602780+0000) 2022-01-31T22:06:09.050 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:08 smithi167 conmon[49112]: debug 2022-01-31T22:06:08.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:08.936945+0000) 2022-01-31T22:06:09.051 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:08 smithi171 conmon[51620]: debug 2022-01-31T22:06:08.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:08.955828+0000) 2022-01-31T22:06:09.052 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:08 smithi171 conmon[35325]: debug 2022-01-31T22:06:08.796+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:06:09.307 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:09 smithi167 conmon[49112]: debug 2022-01-31T22:06:09.052+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.052658+0000) 2022-01-31T22:06:09.308 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:09 smithi167 conmon[54076]: debug 2022-01-31T22:06:09.050+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.050905+0000) 2022-01-31T22:06:09.309 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:09 smithi167 conmon[60316]: debug 2022-01-31T22:06:09.051+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.052355+0000) 2022-01-31T22:06:09.310 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[35325]: debug 2022-01-31T22:06:09.069+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182587 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:09.310 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[41853]: debug 2022-01-31T22:06:09.051+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.052016+0000) 2022-01-31T22:06:09.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[51620]: debug 2022-01-31T22:06:09.052+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.052834+0000) 2022-01-31T22:06:09.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[46715]: debug 2022-01-31T22:06:09.052+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.053008+0000) 2022-01-31T22:06:09.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[46715]: debug 2022-01-31T22:06:09.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.104517+0000) 2022-01-31T22:06:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[41853]: debug 2022-01-31T22:06:09.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.310584+0000) 2022-01-31T22:06:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:09 smithi167 conmon[54076]: debug 2022-01-31T22:06:09.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.602242+0000) 2022-01-31T22:06:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:09 smithi167 conmon[60316]: debug 2022-01-31T22:06:09.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.602879+0000) 2022-01-31T22:06:10.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:09 smithi167 conmon[49112]: debug 2022-01-31T22:06:09.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.937108+0000) 2022-01-31T22:06:10.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:10 smithi171 conmon[46715]: debug 2022-01-31T22:06:10.103+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:10.104655+0000) 2022-01-31T22:06:10.268 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:09 smithi171 conmon[51620]: debug 2022-01-31T22:06:09.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:09.956067+0000) 2022-01-31T22:06:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:10 smithi171 conmon[41853]: debug 2022-01-31T22:06:10.310+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:10.310733+0000) 2022-01-31T22:06:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:10 smithi167 conmon[54076]: debug 2022-01-31T22:06:10.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:10.602443+0000) 2022-01-31T22:06:10.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:10 smithi167 conmon[60316]: debug 2022-01-31T22:06:10.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:10.602957+0000) 2022-01-31T22:06:11.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:10 smithi167 conmon[49112]: debug 2022-01-31T22:06:10.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:10.937277+0000) 2022-01-31T22:06:11.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:11 smithi171 conmon[46715]: debug 2022-01-31T22:06:11.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:11.104833+0000) 2022-01-31T22:06:11.268 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:10 smithi171 conmon[51620]: debug 2022-01-31T22:06:10.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:10.956250+0000) 2022-01-31T22:06:11.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:11 smithi171 conmon[41853]: debug 2022-01-31T22:06:11.310+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:11.310898+0000) 2022-01-31T22:06:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:11 smithi167 conmon[54076]: debug 2022-01-31T22:06:11.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:11.602659+0000) 2022-01-31T22:06:11.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:11 smithi167 conmon[60316]: debug 2022-01-31T22:06:11.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:11.603130+0000) 2022-01-31T22:06:12.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:11 smithi167 conmon[49112]: debug 2022-01-31T22:06:11.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:11.937438+0000) 2022-01-31T22:06:12.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:12 smithi171 conmon[46715]: debug 2022-01-31T22:06:12.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:12.105024+0000) 2022-01-31T22:06:12.268 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:11 smithi171 conmon[51620]: debug 2022-01-31T22:06:11.955+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:11.956485+0000) 2022-01-31T22:06:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:12 smithi171 conmon[41853]: debug 2022-01-31T22:06:12.310+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:12.311088+0000) 2022-01-31T22:06:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:12 smithi167 conmon[54076]: debug 2022-01-31T22:06:12.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:12.602759+0000) 2022-01-31T22:06:12.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:12 smithi167 conmon[60316]: debug 2022-01-31T22:06:12.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:12.603288+0000) 2022-01-31T22:06:13.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:12 smithi167 conmon[49112]: debug 2022-01-31T22:06:12.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:12.937583+0000) 2022-01-31T22:06:13.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:13 smithi171 conmon[46715]: debug 2022-01-31T22:06:13.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:13.105214+0000) 2022-01-31T22:06:13.268 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:12 smithi171 conmon[51620]: debug 2022-01-31T22:06:12.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:12.956651+0000) 2022-01-31T22:06:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:13 smithi171 conmon[41853]: debug 2022-01-31T22:06:13.310+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:13.311295+0000) 2022-01-31T22:06:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:13 smithi167 conmon[54076]: debug 2022-01-31T22:06:13.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:13.602915+0000) 2022-01-31T22:06:13.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:13 smithi167 conmon[60316]: debug 2022-01-31T22:06:13.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:13.603420+0000) 2022-01-31T22:06:14.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:13 smithi167 conmon[49112]: debug 2022-01-31T22:06:13.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:13.937766+0000) 2022-01-31T22:06:14.268 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[35325]: debug 2022-01-31T22:06:14.091+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182698 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:14.269 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[41853]: debug 2022-01-31T22:06:14.071+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.072332+0000) 2022-01-31T22:06:14.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[46715]: debug 2022-01-31T22:06:14.071+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.072696+0000) 2022-01-31T22:06:14.270 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[46715]: debug 2022-01-31T22:06:14.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.105435+0000) 2022-01-31T22:06:14.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:13 smithi171 conmon[51620]: debug 2022-01-31T22:06:13.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:13.956818+0000) 2022-01-31T22:06:14.271 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[51620]: debug 2022-01-31T22:06:14.072+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.073228+0000) 2022-01-31T22:06:14.405 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:14 smithi167 conmon[49112]: debug 2022-01-31T22:06:14.071+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.073391+0000) 2022-01-31T22:06:14.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:14 smithi167 conmon[54076]: debug 2022-01-31T22:06:14.071+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.072942+0000) 2022-01-31T22:06:14.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:14 smithi167 conmon[60316]: debug 2022-01-31T22:06:14.072+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.073774+0000) 2022-01-31T22:06:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[41853]: debug 2022-01-31T22:06:14.310+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.311499+0000) 2022-01-31T22:06:14.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:14 smithi167 conmon[54076]: debug 2022-01-31T22:06:14.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.603073+0000) 2022-01-31T22:06:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:14 smithi167 conmon[60316]: debug 2022-01-31T22:06:14.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.603528+0000) 2022-01-31T22:06:15.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:14 smithi167 conmon[49112]: debug 2022-01-31T22:06:14.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.937927+0000) 2022-01-31T22:06:15.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:15 smithi171 conmon[46715]: debug 2022-01-31T22:06:15.104+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:15.105564+0000) 2022-01-31T22:06:15.269 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:14 smithi171 conmon[51620]: debug 2022-01-31T22:06:14.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:14.956990+0000) 2022-01-31T22:06:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:15 smithi171 conmon[41853]: debug 2022-01-31T22:06:15.311+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:15.311675+0000) 2022-01-31T22:06:15.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:15 smithi167 conmon[54076]: debug 2022-01-31T22:06:15.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:15.603226+0000) 2022-01-31T22:06:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:15 smithi167 conmon[60316]: debug 2022-01-31T22:06:15.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:15.603644+0000) 2022-01-31T22:06:16.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:15 smithi167 conmon[49112]: debug 2022-01-31T22:06:15.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:15.938071+0000) 2022-01-31T22:06:16.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:16 smithi171 conmon[46715]: debug 2022-01-31T22:06:16.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:16.105726+0000) 2022-01-31T22:06:16.269 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:15 smithi171 conmon[51620]: debug 2022-01-31T22:06:15.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:15.957181+0000) 2022-01-31T22:06:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:16 smithi171 conmon[41853]: debug 2022-01-31T22:06:16.311+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:16.311902+0000) 2022-01-31T22:06:16.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:16 smithi167 conmon[54076]: debug 2022-01-31T22:06:16.601+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:16.603410+0000) 2022-01-31T22:06:16.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:16 smithi167 conmon[60316]: debug 2022-01-31T22:06:16.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:16.603771+0000) 2022-01-31T22:06:17.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:16 smithi167 conmon[49112]: debug 2022-01-31T22:06:16.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:16.938261+0000) 2022-01-31T22:06:17.268 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:17 smithi171 conmon[46715]: debug 2022-01-31T22:06:17.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:17.105909+0000) 2022-01-31T22:06:17.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:16 smithi171 conmon[51620]: debug 2022-01-31T22:06:16.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:16.957370+0000) 2022-01-31T22:06:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:17 smithi171 conmon[41853]: debug 2022-01-31T22:06:17.311+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:17.312083+0000) 2022-01-31T22:06:17.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:17 smithi167 conmon[54076]: debug 2022-01-31T22:06:17.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:17.603546+0000) 2022-01-31T22:06:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:17 smithi167 conmon[60316]: debug 2022-01-31T22:06:17.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:17.603881+0000) 2022-01-31T22:06:18.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:17 smithi167 conmon[49112]: debug 2022-01-31T22:06:17.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:17.938457+0000) 2022-01-31T22:06:18.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:18 smithi171 conmon[46715]: debug 2022-01-31T22:06:18.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:18.106035+0000) 2022-01-31T22:06:18.269 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:17 smithi171 conmon[51620]: debug 2022-01-31T22:06:17.956+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:17.957598+0000) 2022-01-31T22:06:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:18 smithi171 conmon[41853]: debug 2022-01-31T22:06:18.311+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:18.312287+0000) 2022-01-31T22:06:18.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:18 smithi167 conmon[54076]: debug 2022-01-31T22:06:18.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:18.603682+0000) 2022-01-31T22:06:18.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:18 smithi167 conmon[60316]: debug 2022-01-31T22:06:18.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:18.603981+0000) 2022-01-31T22:06:19.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:18 smithi167 conmon[49112]: debug 2022-01-31T22:06:18.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:18.938649+0000) 2022-01-31T22:06:19.269 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[35325]: debug 2022-01-31T22:06:19.113+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182810 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:19.270 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[41853]: debug 2022-01-31T22:06:19.093+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.094237+0000) 2022-01-31T22:06:19.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:18 smithi171 conmon[51620]: debug 2022-01-31T22:06:18.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:18.957786+0000) 2022-01-31T22:06:19.271 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[51620]: debug 2022-01-31T22:06:19.094+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.094880+0000) 2022-01-31T22:06:19.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[46715]: debug 2022-01-31T22:06:19.093+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.094135+0000) 2022-01-31T22:06:19.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[46715]: debug 2022-01-31T22:06:19.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.106212+0000) 2022-01-31T22:06:19.406 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:19 smithi167 conmon[49112]: debug 2022-01-31T22:06:19.092+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.094441+0000) 2022-01-31T22:06:19.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:19 smithi167 conmon[54076]: debug 2022-01-31T22:06:19.093+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.095373+0000) 2022-01-31T22:06:19.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:19 smithi167 conmon[60316]: debug 2022-01-31T22:06:19.092+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.093751+0000) 2022-01-31T22:06:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[41853]: debug 2022-01-31T22:06:19.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.312448+0000) 2022-01-31T22:06:19.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:19 smithi167 conmon[54076]: debug 2022-01-31T22:06:19.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.603798+0000) 2022-01-31T22:06:19.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:19 smithi167 conmon[60316]: debug 2022-01-31T22:06:19.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.604083+0000) 2022-01-31T22:06:20.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:19 smithi167 conmon[49112]: debug 2022-01-31T22:06:19.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.938809+0000) 2022-01-31T22:06:20.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:20 smithi171 conmon[46715]: debug 2022-01-31T22:06:20.105+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:20.106390+0000) 2022-01-31T22:06:20.269 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:19 smithi171 conmon[51620]: debug 2022-01-31T22:06:19.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:19.958000+0000) 2022-01-31T22:06:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:20 smithi171 conmon[41853]: debug 2022-01-31T22:06:20.311+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:20.312524+0000) 2022-01-31T22:06:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:20 smithi167 conmon[54076]: debug 2022-01-31T22:06:20.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:20.603949+0000) 2022-01-31T22:06:20.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:20 smithi167 conmon[60316]: debug 2022-01-31T22:06:20.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:20.604178+0000) 2022-01-31T22:06:21.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:20 smithi167 conmon[49112]: debug 2022-01-31T22:06:20.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:20.938962+0000) 2022-01-31T22:06:21.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:21 smithi171 conmon[46715]: debug 2022-01-31T22:06:21.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:21.106581+0000) 2022-01-31T22:06:21.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:20 smithi171 conmon[51620]: debug 2022-01-31T22:06:20.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:20.958151+0000) 2022-01-31T22:06:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:21 smithi171 conmon[41853]: debug 2022-01-31T22:06:21.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:21.312697+0000) 2022-01-31T22:06:21.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:21 smithi167 conmon[54076]: debug 2022-01-31T22:06:21.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:21.604112+0000) 2022-01-31T22:06:21.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:21 smithi167 conmon[60316]: debug 2022-01-31T22:06:21.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:21.604329+0000) 2022-01-31T22:06:22.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:21 smithi167 conmon[49112]: debug 2022-01-31T22:06:21.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:21.939135+0000) 2022-01-31T22:06:22.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:22 smithi171 conmon[46715]: debug 2022-01-31T22:06:22.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:22.106792+0000) 2022-01-31T22:06:22.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:21 smithi171 conmon[51620]: debug 2022-01-31T22:06:21.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:21.958314+0000) 2022-01-31T22:06:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:22 smithi171 conmon[41853]: debug 2022-01-31T22:06:22.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:22.312876+0000) 2022-01-31T22:06:22.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:22 smithi167 conmon[54076]: debug 2022-01-31T22:06:22.602+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:22.604347+0000) 2022-01-31T22:06:22.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:22 smithi167 conmon[60316]: debug 2022-01-31T22:06:22.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:22.604512+0000) 2022-01-31T22:06:23.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:22 smithi167 conmon[49112]: debug 2022-01-31T22:06:22.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:22.939318+0000) 2022-01-31T22:06:23.270 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:23 smithi171 conmon[46715]: debug 2022-01-31T22:06:23.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:23.106956+0000) 2022-01-31T22:06:23.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:22 smithi171 conmon[51620]: debug 2022-01-31T22:06:22.957+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:22.958492+0000) 2022-01-31T22:06:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:23 smithi171 conmon[41853]: debug 2022-01-31T22:06:23.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:23.313057+0000) 2022-01-31T22:06:23.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:23 smithi167 conmon[60316]: debug 2022-01-31T22:06:23.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:23.604628+0000) 2022-01-31T22:06:23.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:23 smithi167 conmon[54076]: debug 2022-01-31T22:06:23.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:23.604521+0000) 2022-01-31T22:06:24.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:23 smithi167 conmon[49112]: debug 2022-01-31T22:06:23.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:23.939528+0000) 2022-01-31T22:06:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:23 smithi171 conmon[35325]: debug 2022-01-31T22:06:23.797+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:06:24.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:23 smithi171 conmon[51620]: debug 2022-01-31T22:06:23.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:23.958673+0000) 2022-01-31T22:06:24.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[51620]: debug 2022-01-31T22:06:24.117+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.118424+0000) 2022-01-31T22:06:24.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[35325]: debug 2022-01-31T22:06:24.135+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 182924 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:24.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[41853]: debug 2022-01-31T22:06:24.115+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.116575+0000) 2022-01-31T22:06:24.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[41853]: debug 2022-01-31T22:06:24.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.313244+0000) 2022-01-31T22:06:24.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[46715]: debug 2022-01-31T22:06:24.106+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.107121+0000) 2022-01-31T22:06:24.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[46715]: debug 2022-01-31T22:06:24.115+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.116295+0000) 2022-01-31T22:06:24.407 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:24 smithi167 conmon[49112]: debug 2022-01-31T22:06:24.115+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.117280+0000) 2022-01-31T22:06:24.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:24 smithi167 conmon[54076]: debug 2022-01-31T22:06:24.116+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.117533+0000) 2022-01-31T22:06:24.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:24 smithi167 conmon[60316]: debug 2022-01-31T22:06:24.115+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.116941+0000) 2022-01-31T22:06:24.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:24 smithi167 conmon[60316]: debug 2022-01-31T22:06:24.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.604797+0000) 2022-01-31T22:06:24.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:24 smithi167 conmon[54076]: debug 2022-01-31T22:06:24.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.604708+0000) 2022-01-31T22:06:25.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:24 smithi167 conmon[49112]: debug 2022-01-31T22:06:24.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.939728+0000) 2022-01-31T22:06:25.270 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:25 smithi171 conmon[46715]: debug 2022-01-31T22:06:25.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:25.107326+0000) 2022-01-31T22:06:25.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:24 smithi171 conmon[51620]: debug 2022-01-31T22:06:24.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:24.958850+0000) 2022-01-31T22:06:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:25 smithi171 conmon[41853]: debug 2022-01-31T22:06:25.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:25.313403+0000) 2022-01-31T22:06:25.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:25 smithi167 conmon[60316]: debug 2022-01-31T22:06:25.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:25.604948+0000) 2022-01-31T22:06:25.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:25 smithi167 conmon[54076]: debug 2022-01-31T22:06:25.604+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:25.604860+0000) 2022-01-31T22:06:26.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:25 smithi167 conmon[49112]: debug 2022-01-31T22:06:25.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:25.939848+0000) 2022-01-31T22:06:26.270 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:26 smithi171 conmon[46715]: debug 2022-01-31T22:06:26.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:26.107476+0000) 2022-01-31T22:06:26.271 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:25 smithi171 conmon[51620]: debug 2022-01-31T22:06:25.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:25.958967+0000) 2022-01-31T22:06:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:26 smithi171 conmon[41853]: debug 2022-01-31T22:06:26.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:26.313604+0000) 2022-01-31T22:06:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:26 smithi167 conmon[54076]: debug 2022-01-31T22:06:26.603+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:26.605132+0000) 2022-01-31T22:06:26.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:26 smithi167 conmon[60316]: debug 2022-01-31T22:06:26.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:26.605029+0000) 2022-01-31T22:06:27.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:26 smithi167 conmon[49112]: debug 2022-01-31T22:06:26.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:26.940056+0000) 2022-01-31T22:06:27.270 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:26 smithi171 conmon[51620]: debug 2022-01-31T22:06:26.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:26.959141+0000) 2022-01-31T22:06:27.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:27 smithi171 conmon[46715]: debug 2022-01-31T22:06:27.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:27.107663+0000) 2022-01-31T22:06:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:27 smithi171 conmon[41853]: debug 2022-01-31T22:06:27.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:27.313808+0000) 2022-01-31T22:06:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:27 smithi167 conmon[54076]: debug 2022-01-31T22:06:27.604+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:27.605318+0000) 2022-01-31T22:06:27.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:27 smithi167 conmon[60316]: debug 2022-01-31T22:06:27.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:27.605220+0000) 2022-01-31T22:06:28.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:27 smithi167 conmon[49112]: debug 2022-01-31T22:06:27.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:27.940269+0000) 2022-01-31T22:06:28.270 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:28 smithi171 conmon[46715]: debug 2022-01-31T22:06:28.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:28.107871+0000) 2022-01-31T22:06:28.271 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:27 smithi171 conmon[51620]: debug 2022-01-31T22:06:27.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:27.959340+0000) 2022-01-31T22:06:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:28 smithi171 conmon[41853]: debug 2022-01-31T22:06:28.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:28.313952+0000) 2022-01-31T22:06:28.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:28 smithi167 conmon[54076]: debug 2022-01-31T22:06:28.604+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:28.605543+0000) 2022-01-31T22:06:28.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:28 smithi167 conmon[60316]: debug 2022-01-31T22:06:28.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:28.605433+0000) 2022-01-31T22:06:29.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:28 smithi167 conmon[49112]: debug 2022-01-31T22:06:28.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:28.940461+0000) 2022-01-31T22:06:29.271 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[35325]: debug 2022-01-31T22:06:29.157+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183034 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:29.271 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[41853]: debug 2022-01-31T22:06:29.137+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.138150+0000) 2022-01-31T22:06:29.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:28 smithi171 conmon[51620]: debug 2022-01-31T22:06:28.958+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:28.959525+0000) 2022-01-31T22:06:29.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[51620]: debug 2022-01-31T22:06:29.138+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.139544+0000) 2022-01-31T22:06:29.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[46715]: debug 2022-01-31T22:06:29.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.108060+0000) 2022-01-31T22:06:29.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[46715]: debug 2022-01-31T22:06:29.137+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.137929+0000) 2022-01-31T22:06:29.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:29 smithi167 conmon[49112]: debug 2022-01-31T22:06:29.137+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.138971+0000) 2022-01-31T22:06:29.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:29 smithi167 conmon[54076]: debug 2022-01-31T22:06:29.137+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.139343+0000) 2022-01-31T22:06:29.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:29 smithi167 conmon[60316]: debug 2022-01-31T22:06:29.137+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.138722+0000) 2022-01-31T22:06:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[41853]: debug 2022-01-31T22:06:29.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.314084+0000) 2022-01-31T22:06:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:29 smithi167 conmon[54076]: debug 2022-01-31T22:06:29.604+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.605673+0000) 2022-01-31T22:06:29.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:29 smithi167 conmon[60316]: debug 2022-01-31T22:06:29.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.605590+0000) 2022-01-31T22:06:30.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:29 smithi167 conmon[49112]: debug 2022-01-31T22:06:29.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.940615+0000) 2022-01-31T22:06:30.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:30 smithi171 conmon[46715]: debug 2022-01-31T22:06:30.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:30.108247+0000) 2022-01-31T22:06:30.271 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:29 smithi171 conmon[51620]: debug 2022-01-31T22:06:29.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:29.959721+0000) 2022-01-31T22:06:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:30 smithi171 conmon[41853]: debug 2022-01-31T22:06:30.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:30.314229+0000) 2022-01-31T22:06:30.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:30 smithi167 conmon[54076]: debug 2022-01-31T22:06:30.605+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:30.605804+0000) 2022-01-31T22:06:30.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:30 smithi167 conmon[60316]: debug 2022-01-31T22:06:30.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:30.605758+0000) 2022-01-31T22:06:31.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:30 smithi167 conmon[49112]: debug 2022-01-31T22:06:30.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:30.940748+0000) 2022-01-31T22:06:31.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:31 smithi171 conmon[46715]: debug 2022-01-31T22:06:31.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:31.108371+0000) 2022-01-31T22:06:31.271 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:30 smithi171 conmon[51620]: debug 2022-01-31T22:06:30.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:30.959873+0000) 2022-01-31T22:06:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:31 smithi171 conmon[41853]: debug 2022-01-31T22:06:31.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:31.314417+0000) 2022-01-31T22:06:31.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:31 smithi167 conmon[54076]: debug 2022-01-31T22:06:31.605+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:31.605964+0000) 2022-01-31T22:06:31.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:31 smithi167 conmon[60316]: debug 2022-01-31T22:06:31.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:31.605954+0000) 2022-01-31T22:06:32.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:31 smithi167 conmon[49112]: debug 2022-01-31T22:06:31.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:31.940930+0000) 2022-01-31T22:06:32.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:32 smithi171 conmon[46715]: debug 2022-01-31T22:06:32.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:32.108548+0000) 2022-01-31T22:06:32.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:31 smithi171 conmon[51620]: debug 2022-01-31T22:06:31.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:31.960078+0000) 2022-01-31T22:06:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:32 smithi171 conmon[41853]: debug 2022-01-31T22:06:32.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:32.314605+0000) 2022-01-31T22:06:32.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:32 smithi167 conmon[54076]: debug 2022-01-31T22:06:32.605+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:32.606206+0000) 2022-01-31T22:06:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:32 smithi167 conmon[60316]: debug 2022-01-31T22:06:32.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:32.606169+0000) 2022-01-31T22:06:33.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:32 smithi167 conmon[49112]: debug 2022-01-31T22:06:32.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:32.941099+0000) 2022-01-31T22:06:33.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:33 smithi171 conmon[46715]: debug 2022-01-31T22:06:33.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:33.108735+0000) 2022-01-31T22:06:33.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:32 smithi171 conmon[51620]: debug 2022-01-31T22:06:32.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:32.960256+0000) 2022-01-31T22:06:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:33 smithi171 conmon[41853]: debug 2022-01-31T22:06:33.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:33.314807+0000) 2022-01-31T22:06:33.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:33 smithi167 conmon[54076]: debug 2022-01-31T22:06:33.605+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:33.606396+0000) 2022-01-31T22:06:33.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:33 smithi167 conmon[60316]: debug 2022-01-31T22:06:33.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:33.606316+0000) 2022-01-31T22:06:34.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:33 smithi167 conmon[49112]: debug 2022-01-31T22:06:33.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:33.941279+0000) 2022-01-31T22:06:34.271 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[35325]: debug 2022-01-31T22:06:34.179+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183146 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:34.272 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[41853]: debug 2022-01-31T22:06:34.159+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.159867+0000) 2022-01-31T22:06:34.273 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:33 smithi171 conmon[51620]: debug 2022-01-31T22:06:33.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:33.960428+0000) 2022-01-31T22:06:34.273 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[51620]: debug 2022-01-31T22:06:34.159+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.160540+0000) 2022-01-31T22:06:34.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[46715]: debug 2022-01-31T22:06:34.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.108921+0000) 2022-01-31T22:06:34.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[46715]: debug 2022-01-31T22:06:34.159+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.160423+0000) 2022-01-31T22:06:34.409 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[49112]: debug 2022-01-31T22:06:34.159+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.160550+0000) 2022-01-31T22:06:34.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[54076]: debug 2022-01-31T22:06:34.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[54076]: 2022-01-31T22:06:34.159+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.161194+0000) 2022-01-31T22:06:34.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[60316]: debug 2022-01-31T22:06:34.158+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.159979+0000) 2022-01-31T22:06:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[41853]: debug 2022-01-31T22:06:34.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.314978+0000) 2022-01-31T22:06:34.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[54076]: debug 2022-01-31T22:06:34.605+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.606484+0000) 2022-01-31T22:06:34.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[60316]: debug 2022-01-31T22:06:34.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.606558+0000) 2022-01-31T22:06:35.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:34 smithi167 conmon[49112]: debug 2022-01-31T22:06:34.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.941470+0000) 2022-01-31T22:06:35.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:35 smithi171 conmon[46715]: debug 2022-01-31T22:06:35.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:35.109114+0000) 2022-01-31T22:06:35.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:34 smithi171 conmon[51620]: debug 2022-01-31T22:06:34.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:34.960608+0000) 2022-01-31T22:06:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:35 smithi171 conmon[41853]: debug 2022-01-31T22:06:35.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:35.315136+0000) 2022-01-31T22:06:35.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:35 smithi167 conmon[54076]: debug 2022-01-31T22:06:35.606+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:35.606648+0000) 2022-01-31T22:06:35.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:35 smithi167 conmon[60316]: debug 2022-01-31T22:06:35.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:35.606678+0000) 2022-01-31T22:06:36.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:35 smithi167 conmon[49112]: debug 2022-01-31T22:06:35.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:35.941680+0000) 2022-01-31T22:06:36.271 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:36 smithi171 conmon[46715]: debug 2022-01-31T22:06:36.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:36.109244+0000) 2022-01-31T22:06:36.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:35 smithi171 conmon[51620]: debug 2022-01-31T22:06:35.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:35.960770+0000) 2022-01-31T22:06:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:36 smithi171 conmon[41853]: debug 2022-01-31T22:06:36.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:36.315319+0000) 2022-01-31T22:06:36.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:36 smithi167 conmon[54076]: debug 2022-01-31T22:06:36.606+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:36.606840+0000) 2022-01-31T22:06:36.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:36 smithi167 conmon[60316]: debug 2022-01-31T22:06:36.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:36.606838+0000) 2022-01-31T22:06:37.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:36 smithi167 conmon[49112]: debug 2022-01-31T22:06:36.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:36.941834+0000) 2022-01-31T22:06:37.272 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:37 smithi171 conmon[46715]: debug 2022-01-31T22:06:37.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:37.109416+0000) 2022-01-31T22:06:37.273 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:36 smithi171 conmon[51620]: debug 2022-01-31T22:06:36.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:36.960939+0000) 2022-01-31T22:06:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:37 smithi171 conmon[41853]: debug 2022-01-31T22:06:37.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:37.315508+0000) 2022-01-31T22:06:37.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:37 smithi167 conmon[54076]: debug 2022-01-31T22:06:37.606+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:37.607027+0000) 2022-01-31T22:06:37.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:37 smithi167 conmon[60316]: debug 2022-01-31T22:06:37.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:37.607027+0000) 2022-01-31T22:06:38.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:37 smithi167 conmon[49112]: debug 2022-01-31T22:06:37.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:37.942013+0000) 2022-01-31T22:06:38.272 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:38 smithi171 conmon[46715]: debug 2022-01-31T22:06:38.107+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:38.109597+0000) 2022-01-31T22:06:38.272 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:37 smithi171 conmon[51620]: debug 2022-01-31T22:06:37.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:37.961064+0000) 2022-01-31T22:06:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:38 smithi171 conmon[41853]: debug 2022-01-31T22:06:38.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:38.315689+0000) 2022-01-31T22:06:38.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:38 smithi167 conmon[54076]: debug 2022-01-31T22:06:38.606+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:38.607232+0000) 2022-01-31T22:06:38.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:38 smithi167 conmon[60316]: debug 2022-01-31T22:06:38.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:38.607232+0000) 2022-01-31T22:06:39.048 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:38 smithi171 conmon[35325]: debug 2022-01-31T22:06:38.797+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:06:39.049 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:38 smithi171 conmon[51620]: debug 2022-01-31T22:06:38.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:38.961223+0000) 2022-01-31T22:06:39.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:38 smithi167 conmon[49112]: debug 2022-01-31T22:06:38.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:38.942199+0000) 2022-01-31T22:06:39.315 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[35325]: debug 2022-01-31T22:06:39.206+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183252 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:39.315 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[41853]: debug 2022-01-31T22:06:39.180+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.182153+0000) 2022-01-31T22:06:39.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[51620]: debug 2022-01-31T22:06:39.181+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.183438+0000) 2022-01-31T22:06:39.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[46715]: debug 2022-01-31T22:06:39.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.109767+0000) 2022-01-31T22:06:39.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[46715]: debug 2022-01-31T22:06:39.181+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.182885+0000) 2022-01-31T22:06:39.409 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:39 smithi167 conmon[49112]: debug 2022-01-31T22:06:39.182+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.182828+0000) 2022-01-31T22:06:39.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:39 smithi167 conmon[54076]: debug 2022-01-31T22:06:39.182+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.183018+0000) 2022-01-31T22:06:39.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:39 smithi167 conmon[60316]: debug 2022-01-31T22:06:39.181+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.182408+0000) 2022-01-31T22:06:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[41853]: debug 2022-01-31T22:06:39.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.315867+0000) 2022-01-31T22:06:39.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:39 smithi167 conmon[54076]: debug 2022-01-31T22:06:39.606+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.607409+0000) 2022-01-31T22:06:39.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:39 smithi167 conmon[60316]: debug 2022-01-31T22:06:39.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.607409+0000) 2022-01-31T22:06:40.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:39 smithi167 conmon[49112]: debug 2022-01-31T22:06:39.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.942405+0000) 2022-01-31T22:06:40.272 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:40 smithi171 conmon[46715]: debug 2022-01-31T22:06:40.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:40.109975+0000) 2022-01-31T22:06:40.273 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:39 smithi171 conmon[51620]: debug 2022-01-31T22:06:39.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:39.961395+0000) 2022-01-31T22:06:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:40 smithi171 conmon[41853]: debug 2022-01-31T22:06:40.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:40.316004+0000) 2022-01-31T22:06:40.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:40 smithi167 conmon[54076]: debug 2022-01-31T22:06:40.607+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:40.607580+0000) 2022-01-31T22:06:40.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:40 smithi167 conmon[60316]: debug 2022-01-31T22:06:40.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:40.607581+0000) 2022-01-31T22:06:41.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:40 smithi167 conmon[49112]: debug 2022-01-31T22:06:40.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:40.942601+0000) 2022-01-31T22:06:41.272 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:41 smithi171 conmon[46715]: debug 2022-01-31T22:06:41.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:41.110121+0000) 2022-01-31T22:06:41.273 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:40 smithi171 conmon[51620]: debug 2022-01-31T22:06:40.959+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:40.961563+0000) 2022-01-31T22:06:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:41 smithi171 conmon[41853]: debug 2022-01-31T22:06:41.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:41.316200+0000) 2022-01-31T22:06:41.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:41 smithi167 conmon[54076]: debug 2022-01-31T22:06:41.607+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:41.607759+0000) 2022-01-31T22:06:41.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:41 smithi167 conmon[60316]: debug 2022-01-31T22:06:41.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:41.607759+0000) 2022-01-31T22:06:42.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:41 smithi167 conmon[49112]: debug 2022-01-31T22:06:41.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:41.942793+0000) 2022-01-31T22:06:42.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:42 smithi171 conmon[46715]: debug 2022-01-31T22:06:42.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:42.110285+0000) 2022-01-31T22:06:42.274 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:41 smithi171 conmon[51620]: debug 2022-01-31T22:06:41.960+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:41.961773+0000) 2022-01-31T22:06:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:42 smithi171 conmon[41853]: debug 2022-01-31T22:06:42.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:42.316355+0000) 2022-01-31T22:06:42.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:42 smithi167 conmon[54076]: debug 2022-01-31T22:06:42.607+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:42.607955+0000) 2022-01-31T22:06:42.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:42 smithi167 conmon[60316]: debug 2022-01-31T22:06:42.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:42.607956+0000) 2022-01-31T22:06:43.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:42 smithi167 conmon[49112]: debug 2022-01-31T22:06:42.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:42.942961+0000) 2022-01-31T22:06:43.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:43 smithi171 conmon[46715]: debug 2022-01-31T22:06:43.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:43.110465+0000) 2022-01-31T22:06:43.273 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:42 smithi171 conmon[51620]: debug 2022-01-31T22:06:42.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:42.961935+0000) 2022-01-31T22:06:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:43 smithi171 conmon[41853]: debug 2022-01-31T22:06:43.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:43.316578+0000) 2022-01-31T22:06:43.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:43 smithi167 conmon[54076]: debug 2022-01-31T22:06:43.607+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:43.608170+0000) 2022-01-31T22:06:43.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:43 smithi167 conmon[60316]: debug 2022-01-31T22:06:43.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:43.608170+0000) 2022-01-31T22:06:44.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:43 smithi167 conmon[49112]: debug 2022-01-31T22:06:43.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:43.943143+0000) 2022-01-31T22:06:44.229 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[41853]: debug 2022-01-31T22:06:44.208+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.210151+0000) 2022-01-31T22:06:44.230 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[46715]: debug 2022-01-31T22:06:44.108+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.110638+0000) 2022-01-31T22:06:44.230 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[46715]: debug 2022-01-31T22:06:44.209+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.210993+0000) 2022-01-31T22:06:44.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:43 smithi171 conmon[51620]: debug 2022-01-31T22:06:43.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:43.962092+0000) 2022-01-31T22:06:44.231 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[51620]: debug 2022-01-31T22:06:44.210+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.212104+0000) 2022-01-31T22:06:44.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:44 smithi167 conmon[49112]: debug 2022-01-31T22:06:44.211+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.211811+0000) 2022-01-31T22:06:44.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:44 smithi167 conmon[54076]: debug 2022-01-31T22:06:44.210+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.210632+0000) 2022-01-31T22:06:44.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:44 smithi167 conmon[60316]: debug 2022-01-31T22:06:44.209+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.210170+0000) 2022-01-31T22:06:44.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[35325]: debug 2022-01-31T22:06:44.228+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183364 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[41853]: debug 2022-01-31T22:06:44.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.316780+0000) 2022-01-31T22:06:44.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:44 smithi167 conmon[54076]: debug 2022-01-31T22:06:44.607+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.608371+0000) 2022-01-31T22:06:44.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:44 smithi167 conmon[60316]: debug 2022-01-31T22:06:44.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.608371+0000) 2022-01-31T22:06:45.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:44 smithi167 conmon[49112]: debug 2022-01-31T22:06:44.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.943244+0000) 2022-01-31T22:06:45.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:45 smithi171 conmon[46715]: debug 2022-01-31T22:06:45.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:45.110840+0000) 2022-01-31T22:06:45.274 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:44 smithi171 conmon[51620]: debug 2022-01-31T22:06:44.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:44.962297+0000) 2022-01-31T22:06:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:45 smithi171 conmon[41853]: debug 2022-01-31T22:06:45.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:45.316954+0000) 2022-01-31T22:06:45.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:45 smithi167 conmon[54076]: debug 2022-01-31T22:06:45.608+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:45.608541+0000) 2022-01-31T22:06:45.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:45 smithi167 conmon[60316]: debug 2022-01-31T22:06:45.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:45.608540+0000) 2022-01-31T22:06:46.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:45 smithi167 conmon[49112]: debug 2022-01-31T22:06:45.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:45.943406+0000) 2022-01-31T22:06:46.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:46 smithi171 conmon[46715]: debug 2022-01-31T22:06:46.109+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:46.110950+0000) 2022-01-31T22:06:46.274 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:45 smithi171 conmon[51620]: debug 2022-01-31T22:06:45.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:45.962452+0000) 2022-01-31T22:06:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:46 smithi171 conmon[41853]: debug 2022-01-31T22:06:46.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:46.317168+0000) 2022-01-31T22:06:46.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:46 smithi167 conmon[54076]: debug 2022-01-31T22:06:46.608+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:46.608759+0000) 2022-01-31T22:06:46.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:46 smithi167 conmon[60316]: debug 2022-01-31T22:06:46.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:46.608756+0000) 2022-01-31T22:06:47.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:46 smithi167 conmon[49112]: debug 2022-01-31T22:06:46.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:46.943559+0000) 2022-01-31T22:06:47.273 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:47 smithi171 conmon[46715]: debug 2022-01-31T22:06:47.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:47.111112+0000) 2022-01-31T22:06:47.274 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:46 smithi171 conmon[51620]: debug 2022-01-31T22:06:46.961+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:46.962622+0000) 2022-01-31T22:06:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:47 smithi171 conmon[41853]: debug 2022-01-31T22:06:47.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:47.317287+0000) 2022-01-31T22:06:47.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:47 smithi167 conmon[54076]: debug 2022-01-31T22:06:47.608+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:47.608896+0000) 2022-01-31T22:06:47.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:47 smithi167 conmon[60316]: debug 2022-01-31T22:06:47.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:47.608896+0000) 2022-01-31T22:06:48.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:47 smithi167 conmon[49112]: debug 2022-01-31T22:06:47.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:47.943753+0000) 2022-01-31T22:06:48.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:48 smithi171 conmon[46715]: debug 2022-01-31T22:06:48.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:48.111234+0000) 2022-01-31T22:06:48.274 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:47 smithi171 conmon[51620]: debug 2022-01-31T22:06:47.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:47.962802+0000) 2022-01-31T22:06:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:48 smithi171 conmon[41853]: debug 2022-01-31T22:06:48.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:48.317512+0000) 2022-01-31T22:06:48.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:48 smithi167 conmon[54076]: debug 2022-01-31T22:06:48.608+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:48.609050+0000) 2022-01-31T22:06:48.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:48 smithi167 conmon[60316]: debug 2022-01-31T22:06:48.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:48.609051+0000) 2022-01-31T22:06:49.099 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:48 smithi167 conmon[49112]: debug 2022-01-31T22:06:48.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:48.943947+0000) 2022-01-31T22:06:49.231 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[46715]: debug 2022-01-31T22:06:49.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.111374+0000) 2022-01-31T22:06:49.232 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:48 smithi171 conmon[51620]: debug 2022-01-31T22:06:48.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:48.962989+0000) 2022-01-31T22:06:49.411 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:49 smithi167 conmon[49112]: debug 2022-01-31T22:06:49.232+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.232805+0000) 2022-01-31T22:06:49.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:49 smithi167 conmon[54076]: debug 2022-01-31T22:06:49.232+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.233136+0000) 2022-01-31T22:06:49.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:49 smithi167 conmon[60316]: debug 2022-01-31T22:06:49.232+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.233318+0000) 2022-01-31T22:06:49.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[35325]: debug 2022-01-31T22:06:49.250+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183474 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:49.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[46715]: debug 2022-01-31T22:06:49.232+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.233154+0000) 2022-01-31T22:06:49.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[51620]: debug 2022-01-31T22:06:49.232+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.233304+0000) 2022-01-31T22:06:49.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[41853]: debug 2022-01-31T22:06:49.231+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.232505+0000) 2022-01-31T22:06:49.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[41853]: debug 2022-01-31T22:06:49.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.317701+0000) 2022-01-31T22:06:49.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:49 smithi167 conmon[60316]: debug 2022-01-31T22:06:49.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.609254+0000) 2022-01-31T22:06:49.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:49 smithi167 conmon[54076]: debug 2022-01-31T22:06:49.608+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.609239+0000) 2022-01-31T22:06:50.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:49 smithi167 conmon[49112]: debug 2022-01-31T22:06:49.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.944131+0000) 2022-01-31T22:06:50.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:50 smithi171 conmon[46715]: debug 2022-01-31T22:06:50.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:50.111550+0000) 2022-01-31T22:06:50.274 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:49 smithi171 conmon[51620]: debug 2022-01-31T22:06:49.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:49.963203+0000) 2022-01-31T22:06:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:50 smithi171 conmon[41853]: debug 2022-01-31T22:06:50.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:50.317860+0000) 2022-01-31T22:06:50.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:50 smithi167 conmon[60316]: debug 2022-01-31T22:06:50.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:50.609427+0000) 2022-01-31T22:06:50.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:50 smithi167 conmon[54076]: debug 2022-01-31T22:06:50.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:50.609395+0000) 2022-01-31T22:06:51.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:50 smithi167 conmon[49112]: debug 2022-01-31T22:06:50.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:50.944281+0000) 2022-01-31T22:06:51.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:51 smithi171 conmon[46715]: debug 2022-01-31T22:06:51.110+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:51.111716+0000) 2022-01-31T22:06:51.275 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:50 smithi171 conmon[51620]: debug 2022-01-31T22:06:50.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:50.963361+0000) 2022-01-31T22:06:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:51 smithi171 conmon[41853]: debug 2022-01-31T22:06:51.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:51.318041+0000) 2022-01-31T22:06:51.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:51 smithi167 conmon[54076]: debug 2022-01-31T22:06:51.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:51.609573+0000) 2022-01-31T22:06:51.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:51 smithi167 conmon[60316]: debug 2022-01-31T22:06:51.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:51.609626+0000) 2022-01-31T22:06:52.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:51 smithi167 conmon[49112]: debug 2022-01-31T22:06:51.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:51.944394+0000) 2022-01-31T22:06:52.275 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:52 smithi171 conmon[46715]: debug 2022-01-31T22:06:52.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:52.111825+0000) 2022-01-31T22:06:52.276 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:51 smithi171 conmon[51620]: debug 2022-01-31T22:06:51.962+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:51.963546+0000) 2022-01-31T22:06:52.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:52 smithi171 conmon[41853]: debug 2022-01-31T22:06:52.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:52.318198+0000) 2022-01-31T22:06:52.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:52 smithi167 conmon[54076]: debug 2022-01-31T22:06:52.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:52.609781+0000) 2022-01-31T22:06:52.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:52 smithi167 conmon[60316]: debug 2022-01-31T22:06:52.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:52.609795+0000) 2022-01-31T22:06:53.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:52 smithi167 conmon[49112]: debug 2022-01-31T22:06:52.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:52.944579+0000) 2022-01-31T22:06:53.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:53 smithi171 conmon[46715]: debug 2022-01-31T22:06:53.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:53.111942+0000) 2022-01-31T22:06:53.275 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:52 smithi171 conmon[51620]: debug 2022-01-31T22:06:52.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:52.963764+0000) 2022-01-31T22:06:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:53 smithi171 conmon[41853]: debug 2022-01-31T22:06:53.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:53.318393+0000) 2022-01-31T22:06:53.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:53 smithi167 conmon[54076]: debug 2022-01-31T22:06:53.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:53.609921+0000) 2022-01-31T22:06:53.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:53 smithi167 conmon[60316]: debug 2022-01-31T22:06:53.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:53.609921+0000) 2022-01-31T22:06:54.051 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:53 smithi171 conmon[35325]: debug 2022-01-31T22:06:53.798+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:06:54.052 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:53 smithi171 conmon[51620]: debug 2022-01-31T22:06:53.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:53.963950+0000) 2022-01-31T22:06:54.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:53 smithi167 conmon[49112]: debug 2022-01-31T22:06:53.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:53.944772+0000) 2022-01-31T22:06:54.317 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[41853]: debug 2022-01-31T22:06:54.253+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.253877+0000) 2022-01-31T22:06:54.318 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[51620]: debug 2022-01-31T22:06:54.252+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.253419+0000) 2022-01-31T22:06:54.319 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[35325]: debug 2022-01-31T22:06:54.272+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183586 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:54.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[46715]: debug 2022-01-31T22:06:54.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.112102+0000) 2022-01-31T22:06:54.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[46715]: debug 2022-01-31T22:06:54.252+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.253590+0000) 2022-01-31T22:06:54.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:54 smithi167 conmon[49112]: debug 2022-01-31T22:06:54.253+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.253843+0000) 2022-01-31T22:06:54.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:54 smithi167 conmon[54076]: debug 2022-01-31T22:06:54.253+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.254338+0000) 2022-01-31T22:06:54.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:54 smithi167 conmon[60316]: debug 2022-01-31T22:06:54.253+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.253952+0000) 2022-01-31T22:06:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[41853]: debug 2022-01-31T22:06:54.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.318560+0000) 2022-01-31T22:06:54.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:54 smithi167 conmon[54076]: debug 2022-01-31T22:06:54.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.610030+0000) 2022-01-31T22:06:54.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:54 smithi167 conmon[60316]: debug 2022-01-31T22:06:54.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.610029+0000) 2022-01-31T22:06:55.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:54 smithi167 conmon[49112]: debug 2022-01-31T22:06:54.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.944950+0000) 2022-01-31T22:06:55.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:55 smithi171 conmon[46715]: debug 2022-01-31T22:06:55.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:55.112291+0000) 2022-01-31T22:06:55.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:54 smithi171 conmon[51620]: debug 2022-01-31T22:06:54.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:54.964099+0000) 2022-01-31T22:06:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:55 smithi171 conmon[41853]: debug 2022-01-31T22:06:55.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:55.318731+0000) 2022-01-31T22:06:55.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:55 smithi167 conmon[54076]: debug 2022-01-31T22:06:55.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:55.610204+0000) 2022-01-31T22:06:55.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:55 smithi167 conmon[60316]: debug 2022-01-31T22:06:55.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:55.610204+0000) 2022-01-31T22:06:56.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:55 smithi167 conmon[49112]: debug 2022-01-31T22:06:55.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:55.945089+0000) 2022-01-31T22:06:56.276 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:56 smithi171 conmon[46715]: debug 2022-01-31T22:06:56.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:56.112460+0000) 2022-01-31T22:06:56.276 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:55 smithi171 conmon[51620]: debug 2022-01-31T22:06:55.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:55.964238+0000) 2022-01-31T22:06:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:56 smithi171 conmon[41853]: debug 2022-01-31T22:06:56.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:56.318891+0000) 2022-01-31T22:06:56.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:56 smithi167 conmon[54076]: debug 2022-01-31T22:06:56.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:56.610414+0000) 2022-01-31T22:06:56.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:56 smithi167 conmon[60316]: debug 2022-01-31T22:06:56.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:56.610414+0000) 2022-01-31T22:06:57.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:56 smithi167 conmon[49112]: debug 2022-01-31T22:06:56.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:56.945275+0000) 2022-01-31T22:06:57.276 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:57 smithi171 conmon[46715]: debug 2022-01-31T22:06:57.111+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:57.112647+0000) 2022-01-31T22:06:57.277 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:56 smithi171 conmon[51620]: debug 2022-01-31T22:06:56.963+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:56.964427+0000) 2022-01-31T22:06:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:57 smithi171 conmon[41853]: debug 2022-01-31T22:06:57.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:57.319009+0000) 2022-01-31T22:06:57.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:57 smithi167 conmon[54076]: debug 2022-01-31T22:06:57.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:57.610650+0000) 2022-01-31T22:06:57.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:57 smithi167 conmon[60316]: debug 2022-01-31T22:06:57.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:57.610650+0000) 2022-01-31T22:06:58.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:57 smithi167 conmon[49112]: debug 2022-01-31T22:06:57.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:57.945463+0000) 2022-01-31T22:06:58.276 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:58 smithi171 conmon[46715]: debug 2022-01-31T22:06:58.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:58.112805+0000) 2022-01-31T22:06:58.277 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:57 smithi171 conmon[51620]: debug 2022-01-31T22:06:57.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:57.964613+0000) 2022-01-31T22:06:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:58 smithi171 conmon[41853]: debug 2022-01-31T22:06:58.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:58.319234+0000) 2022-01-31T22:06:58.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:58 smithi167 conmon[54076]: debug 2022-01-31T22:06:58.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:58.610831+0000) 2022-01-31T22:06:58.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:58 smithi167 conmon[60316]: debug 2022-01-31T22:06:58.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:58.610831+0000) 2022-01-31T22:06:59.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:58 smithi167 conmon[49112]: debug 2022-01-31T22:06:58.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:58.945645+0000) 2022-01-31T22:06:59.274 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[46715]: debug 2022-01-31T22:06:59.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.112902+0000) 2022-01-31T22:06:59.275 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:58 smithi171 conmon[51620]: debug 2022-01-31T22:06:58.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:58.964811+0000) 2022-01-31T22:06:59.413 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:59 smithi167 conmon[49112]: debug 2022-01-31T22:06:59.274+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.275283+0000) 2022-01-31T22:06:59.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:59 smithi167 conmon[54076]: debug 2022-01-31T22:06:59.276+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.276775+0000) 2022-01-31T22:06:59.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:59 smithi167 conmon[60316]: debug 2022-01-31T22:06:59.274+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.275426+0000) 2022-01-31T22:06:59.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[35325]: debug 2022-01-31T22:06:59.294+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183696 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:06:59.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[46715]: debug 2022-01-31T22:06:59.275+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.276700+0000) 2022-01-31T22:06:59.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[51620]: debug 2022-01-31T22:06:59.275+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.275984+0000) 2022-01-31T22:06:59.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[41853]: debug 2022-01-31T22:06:59.276+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.277048+0000) 2022-01-31T22:06:59.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[41853]: debug 2022-01-31T22:06:59.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.319430+0000) 2022-01-31T22:06:59.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:06:59 smithi167 conmon[54076]: debug 2022-01-31T22:06:59.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.611007+0000) 2022-01-31T22:06:59.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:06:59 smithi167 conmon[60316]: debug 2022-01-31T22:06:59.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.611008+0000) 2022-01-31T22:07:00.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:06:59 smithi167 conmon[49112]: debug 2022-01-31T22:06:59.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.945782+0000) 2022-01-31T22:07:00.276 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:00 smithi171 conmon[46715]: debug 2022-01-31T22:07:00.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:00.113020+0000) 2022-01-31T22:07:00.277 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:06:59 smithi171 conmon[51620]: debug 2022-01-31T22:06:59.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:06:59.965015+0000) 2022-01-31T22:07:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:00 smithi171 conmon[41853]: debug 2022-01-31T22:07:00.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:00.319522+0000) 2022-01-31T22:07:00.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:00 smithi167 conmon[54076]: debug 2022-01-31T22:07:00.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:00.611170+0000) 2022-01-31T22:07:00.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:00 smithi167 conmon[60316]: debug 2022-01-31T22:07:00.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:00.611172+0000) 2022-01-31T22:07:01.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:00 smithi167 conmon[49112]: debug 2022-01-31T22:07:00.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:00.945947+0000) 2022-01-31T22:07:01.277 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:01 smithi171 conmon[46715]: debug 2022-01-31T22:07:01.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:01.113102+0000) 2022-01-31T22:07:01.277 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:00 smithi171 conmon[51620]: debug 2022-01-31T22:07:00.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:00.965134+0000) 2022-01-31T22:07:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:01 smithi171 conmon[41853]: debug 2022-01-31T22:07:01.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:01.319722+0000) 2022-01-31T22:07:01.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:01 smithi167 conmon[54076]: debug 2022-01-31T22:07:01.609+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:01.611385+0000) 2022-01-31T22:07:01.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:01 smithi167 conmon[60316]: debug 2022-01-31T22:07:01.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:01.611385+0000) 2022-01-31T22:07:02.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:01 smithi167 conmon[49112]: debug 2022-01-31T22:07:01.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:01.946152+0000) 2022-01-31T22:07:02.277 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:02 smithi171 conmon[46715]: debug 2022-01-31T22:07:02.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:02.113238+0000) 2022-01-31T22:07:02.277 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:01 smithi171 conmon[51620]: debug 2022-01-31T22:07:01.964+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:01.965322+0000) 2022-01-31T22:07:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:02 smithi171 conmon[41853]: debug 2022-01-31T22:07:02.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:02.319919+0000) 2022-01-31T22:07:02.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:02 smithi167 conmon[54076]: debug 2022-01-31T22:07:02.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:02.611581+0000) 2022-01-31T22:07:02.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:02 smithi167 conmon[60316]: debug 2022-01-31T22:07:02.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:02.611578+0000) 2022-01-31T22:07:03.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:02 smithi167 conmon[49112]: debug 2022-01-31T22:07:02.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:02.946304+0000) 2022-01-31T22:07:03.277 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:03 smithi171 conmon[46715]: debug 2022-01-31T22:07:03.112+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:03.113413+0000) 2022-01-31T22:07:03.278 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:02 smithi171 conmon[51620]: debug 2022-01-31T22:07:02.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:02.965483+0000) 2022-01-31T22:07:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:03 smithi171 conmon[41853]: debug 2022-01-31T22:07:03.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:03.320114+0000) 2022-01-31T22:07:03.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:03 smithi167 conmon[54076]: debug 2022-01-31T22:07:03.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:03.611799+0000) 2022-01-31T22:07:03.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:03 smithi167 conmon[60316]: debug 2022-01-31T22:07:03.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:03.611798+0000) 2022-01-31T22:07:04.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:03 smithi167 conmon[49112]: debug 2022-01-31T22:07:03.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:03.946459+0000) 2022-01-31T22:07:04.277 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[46715]: debug 2022-01-31T22:07:04.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.113615+0000) 2022-01-31T22:07:04.278 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:03 smithi171 conmon[51620]: debug 2022-01-31T22:07:03.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:03.965663+0000) 2022-01-31T22:07:04.414 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:04 smithi167 conmon[49112]: debug 2022-01-31T22:07:04.296+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.298147+0000) 2022-01-31T22:07:04.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:04 smithi167 conmon[54076]: debug 2022-01-31T22:07:04.296+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.298119+0000) 2022-01-31T22:07:04.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:04 smithi167 conmon[60316]: debug 2022-01-31T22:07:04.296+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.297942+0000) 2022-01-31T22:07:04.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[35325]: debug 2022-01-31T22:07:04.315+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:04.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[46715]: debug 2022-01-31T22:07:04.298+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.299049+0000) 2022-01-31T22:07:04.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[51620]: debug 2022-01-31T22:07:04.296+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.296986+0000) 2022-01-31T22:07:04.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[41853]: debug 2022-01-31T22:07:04.297+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.298316+0000) 2022-01-31T22:07:04.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[41853]: debug 2022-01-31T22:07:04.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.320267+0000) 2022-01-31T22:07:04.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:04 smithi167 conmon[54076]: debug 2022-01-31T22:07:04.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.612035+0000) 2022-01-31T22:07:04.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:04 smithi167 conmon[60316]: debug 2022-01-31T22:07:04.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.612035+0000) 2022-01-31T22:07:05.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:04 smithi171 conmon[51620]: debug 2022-01-31T22:07:04.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.965849+0000) 2022-01-31T22:07:05.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:04 smithi167 conmon[49112]: debug 2022-01-31T22:07:04.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:04.946657+0000) 2022-01-31T22:07:05.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:05 smithi171 conmon[46715]: debug 2022-01-31T22:07:05.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:05.113834+0000) 2022-01-31T22:07:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:05 smithi171 conmon[41853]: debug 2022-01-31T22:07:05.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:05.320387+0000) 2022-01-31T22:07:05.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:05 smithi167 conmon[54076]: debug 2022-01-31T22:07:05.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:05.612179+0000) 2022-01-31T22:07:05.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:05 smithi167 conmon[60316]: debug 2022-01-31T22:07:05.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:05.612182+0000) 2022-01-31T22:07:06.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:05 smithi167 conmon[49112]: debug 2022-01-31T22:07:05.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:05.946805+0000) 2022-01-31T22:07:06.278 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:06 smithi171 conmon[46715]: debug 2022-01-31T22:07:06.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:06.113992+0000) 2022-01-31T22:07:06.278 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:05 smithi171 conmon[51620]: debug 2022-01-31T22:07:05.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:05.965961+0000) 2022-01-31T22:07:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:06 smithi171 conmon[41853]: debug 2022-01-31T22:07:06.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:06.320561+0000) 2022-01-31T22:07:06.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:06 smithi167 conmon[54076]: debug 2022-01-31T22:07:06.610+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:06.612352+0000) 2022-01-31T22:07:06.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:06 smithi167 conmon[60316]: debug 2022-01-31T22:07:06.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:06.612353+0000) 2022-01-31T22:07:07.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:06 smithi167 conmon[49112]: debug 2022-01-31T22:07:06.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:06.947005+0000) 2022-01-31T22:07:07.278 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:07 smithi171 conmon[46715]: debug 2022-01-31T22:07:07.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:07.114203+0000) 2022-01-31T22:07:07.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:06 smithi171 conmon[51620]: debug 2022-01-31T22:07:06.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:06.966108+0000) 2022-01-31T22:07:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:07 smithi171 conmon[41853]: debug 2022-01-31T22:07:07.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:07.320727+0000) 2022-01-31T22:07:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:07 smithi167 conmon[54076]: debug 2022-01-31T22:07:07.611+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:07.612545+0000) 2022-01-31T22:07:07.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:07 smithi167 conmon[60316]: debug 2022-01-31T22:07:07.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:07.612545+0000) 2022-01-31T22:07:08.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:07 smithi167 conmon[49112]: debug 2022-01-31T22:07:07.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:07.947170+0000) 2022-01-31T22:07:08.278 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:08 smithi171 conmon[46715]: debug 2022-01-31T22:07:08.113+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:08.114394+0000) 2022-01-31T22:07:08.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:07 smithi171 conmon[51620]: debug 2022-01-31T22:07:07.965+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:07.966290+0000) 2022-01-31T22:07:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:08 smithi171 conmon[41853]: debug 2022-01-31T22:07:08.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:08.320935+0000) 2022-01-31T22:07:08.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:08 smithi167 conmon[54076]: debug 2022-01-31T22:07:08.611+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:08.612692+0000) 2022-01-31T22:07:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:08 smithi167 conmon[60316]: debug 2022-01-31T22:07:08.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:08.612692+0000) 2022-01-31T22:07:09.054 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:08 smithi171 conmon[51620]: debug 2022-01-31T22:07:08.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:08.966490+0000) 2022-01-31T22:07:09.055 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:08 smithi171 conmon[35325]: debug 2022-01-31T22:07:08.799+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:07:09.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:08 smithi167 conmon[49112]: debug 2022-01-31T22:07:08.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:08.947361+0000) 2022-01-31T22:07:09.318 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[51620]: debug 2022-01-31T22:07:09.317+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.318510+0000) 2022-01-31T22:07:09.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[46715]: debug 2022-01-31T22:07:09.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.114590+0000) 2022-01-31T22:07:09.415 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:09 smithi167 conmon[49112]: debug 2022-01-31T22:07:09.318+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.319514+0000) 2022-01-31T22:07:09.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:09 smithi167 conmon[54076]: debug 2022-01-31T22:07:09.318+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.319993+0000) 2022-01-31T22:07:09.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:09 smithi167 conmon[60316]: debug 2022-01-31T22:07:09.317+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.318964+0000) 2022-01-31T22:07:09.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[35325]: debug 2022-01-31T22:07:09.337+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 183918 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:09.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[41853]: debug 2022-01-31T22:07:09.319+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.319960+0000) 2022-01-31T22:07:09.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[41853]: debug 2022-01-31T22:07:09.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.321050+0000) 2022-01-31T22:07:09.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[46715]: debug 2022-01-31T22:07:09.319+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.320352+0000) 2022-01-31T22:07:09.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:09 smithi167 conmon[54076]: debug 2022-01-31T22:07:09.611+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.612879+0000) 2022-01-31T22:07:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:09 smithi167 conmon[60316]: debug 2022-01-31T22:07:09.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.612880+0000) 2022-01-31T22:07:10.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:09 smithi167 conmon[49112]: debug 2022-01-31T22:07:09.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.947574+0000) 2022-01-31T22:07:10.278 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:10 smithi171 conmon[46715]: debug 2022-01-31T22:07:10.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:10.114779+0000) 2022-01-31T22:07:10.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:09 smithi171 conmon[51620]: debug 2022-01-31T22:07:09.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:09.966718+0000) 2022-01-31T22:07:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:10 smithi171 conmon[41853]: debug 2022-01-31T22:07:10.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:10.321184+0000) 2022-01-31T22:07:10.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:10 smithi167 conmon[54076]: debug 2022-01-31T22:07:10.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:10.613047+0000) 2022-01-31T22:07:10.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:10 smithi167 conmon[60316]: debug 2022-01-31T22:07:10.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:10.613047+0000) 2022-01-31T22:07:11.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:10 smithi167 conmon[49112]: debug 2022-01-31T22:07:10.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:10.947726+0000) 2022-01-31T22:07:11.278 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:11 smithi171 conmon[46715]: debug 2022-01-31T22:07:11.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:11.114907+0000) 2022-01-31T22:07:11.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:10 smithi171 conmon[51620]: debug 2022-01-31T22:07:10.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:10.966899+0000) 2022-01-31T22:07:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:11 smithi171 conmon[41853]: debug 2022-01-31T22:07:11.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:11.321386+0000) 2022-01-31T22:07:11.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:11 smithi167 conmon[54076]: debug 2022-01-31T22:07:11.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:11.613291+0000) 2022-01-31T22:07:11.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:11 smithi167 conmon[60316]: debug 2022-01-31T22:07:11.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:11.613291+0000) 2022-01-31T22:07:12.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:11 smithi167 conmon[49112]: debug 2022-01-31T22:07:11.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:11.947882+0000) 2022-01-31T22:07:12.278 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:12 smithi171 conmon[46715]: debug 2022-01-31T22:07:12.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:12.115059+0000) 2022-01-31T22:07:12.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:11 smithi171 conmon[51620]: debug 2022-01-31T22:07:11.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:11.967067+0000) 2022-01-31T22:07:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:12 smithi171 conmon[41853]: debug 2022-01-31T22:07:12.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:12.321518+0000) 2022-01-31T22:07:12.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:12 smithi167 conmon[54076]: debug 2022-01-31T22:07:12.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:12.613549+0000) 2022-01-31T22:07:12.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:12 smithi167 conmon[60316]: debug 2022-01-31T22:07:12.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:12.613549+0000) 2022-01-31T22:07:13.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:12 smithi167 conmon[49112]: debug 2022-01-31T22:07:12.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:12.948007+0000) 2022-01-31T22:07:13.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:13 smithi171 conmon[46715]: debug 2022-01-31T22:07:13.114+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:13.115238+0000) 2022-01-31T22:07:13.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:12 smithi171 conmon[51620]: debug 2022-01-31T22:07:12.966+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:12.967221+0000) 2022-01-31T22:07:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:13 smithi171 conmon[41853]: debug 2022-01-31T22:07:13.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:13.321692+0000) 2022-01-31T22:07:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:13 smithi167 conmon[54076]: debug 2022-01-31T22:07:13.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:13.613755+0000) 2022-01-31T22:07:13.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:13 smithi167 conmon[60316]: debug 2022-01-31T22:07:13.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:13.613755+0000) 2022-01-31T22:07:14.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:13 smithi167 conmon[49112]: debug 2022-01-31T22:07:13.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:13.948178+0000) 2022-01-31T22:07:14.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[46715]: debug 2022-01-31T22:07:14.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.115407+0000) 2022-01-31T22:07:14.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:13 smithi171 conmon[51620]: debug 2022-01-31T22:07:13.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:13.967407+0000) 2022-01-31T22:07:14.416 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:14 smithi167 conmon[49112]: debug 2022-01-31T22:07:14.339+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.340582+0000) 2022-01-31T22:07:14.417 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:14 smithi167 conmon[54076]: debug 2022-01-31T22:07:14.339+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.341434+0000) 2022-01-31T22:07:14.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:14 smithi167 conmon[60316]: debug 2022-01-31T22:07:14.342+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.343752+0000) 2022-01-31T22:07:14.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[35325]: debug 2022-01-31T22:07:14.358+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184032 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:14.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[46715]: debug 2022-01-31T22:07:14.340+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.340941+0000) 2022-01-31T22:07:14.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[51620]: debug 2022-01-31T22:07:14.339+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.340017+0000) 2022-01-31T22:07:14.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[41853]: debug 2022-01-31T22:07:14.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.321942+0000) 2022-01-31T22:07:14.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[41853]: debug 2022-01-31T22:07:14.339+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.340399+0000) 2022-01-31T22:07:14.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:14 smithi167 conmon[54076]: debug 2022-01-31T22:07:14.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.613994+0000) 2022-01-31T22:07:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:14 smithi167 conmon[60316]: debug 2022-01-31T22:07:14.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.613993+0000) 2022-01-31T22:07:15.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:14 smithi167 conmon[49112]: debug 2022-01-31T22:07:14.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.948332+0000) 2022-01-31T22:07:15.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:15 smithi171 conmon[46715]: debug 2022-01-31T22:07:15.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:15.115594+0000) 2022-01-31T22:07:15.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:14 smithi171 conmon[51620]: debug 2022-01-31T22:07:14.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:14.967620+0000) 2022-01-31T22:07:15.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:15 smithi171 conmon[41853]: debug 2022-01-31T22:07:15.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:15.322058+0000) 2022-01-31T22:07:15.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:15 smithi167 conmon[60316]: debug 2022-01-31T22:07:15.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:15.614181+0000) 2022-01-31T22:07:15.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:15 smithi167 conmon[54076]: debug 2022-01-31T22:07:15.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:15.614181+0000) 2022-01-31T22:07:16.104 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:15 smithi167 conmon[49112]: debug 2022-01-31T22:07:15.947+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:15.948535+0000) 2022-01-31T22:07:16.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:16 smithi171 conmon[46715]: debug 2022-01-31T22:07:16.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:16.115769+0000) 2022-01-31T22:07:16.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:15 smithi171 conmon[51620]: debug 2022-01-31T22:07:15.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:15.967787+0000) 2022-01-31T22:07:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:16 smithi171 conmon[41853]: debug 2022-01-31T22:07:16.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:16.322185+0000) 2022-01-31T22:07:16.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:16 smithi167 conmon[54076]: debug 2022-01-31T22:07:16.612+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:16.614392+0000) 2022-01-31T22:07:16.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:16 smithi167 conmon[60316]: debug 2022-01-31T22:07:16.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:16.614392+0000) 2022-01-31T22:07:17.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:16 smithi167 conmon[49112]: debug 2022-01-31T22:07:16.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:16.948740+0000) 2022-01-31T22:07:17.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:17 smithi171 conmon[46715]: debug 2022-01-31T22:07:17.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:17.115930+0000) 2022-01-31T22:07:17.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:16 smithi171 conmon[51620]: debug 2022-01-31T22:07:16.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:16.967967+0000) 2022-01-31T22:07:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:17 smithi171 conmon[41853]: debug 2022-01-31T22:07:17.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:17.322376+0000) 2022-01-31T22:07:17.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:17 smithi167 conmon[54076]: debug 2022-01-31T22:07:17.613+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:17.614617+0000) 2022-01-31T22:07:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:17 smithi167 conmon[60316]: debug 2022-01-31T22:07:17.613+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:17.614617+0000) 2022-01-31T22:07:18.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:17 smithi167 conmon[49112]: debug 2022-01-31T22:07:17.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:17.948946+0000) 2022-01-31T22:07:18.279 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:18 smithi171 conmon[46715]: debug 2022-01-31T22:07:18.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:18.116118+0000) 2022-01-31T22:07:18.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:17 smithi171 conmon[51620]: debug 2022-01-31T22:07:17.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:17.968140+0000) 2022-01-31T22:07:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:18 smithi171 conmon[41853]: debug 2022-01-31T22:07:18.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:18.322530+0000) 2022-01-31T22:07:18.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:18 smithi167 conmon[60316]: debug 2022-01-31T22:07:18.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:18.614843+0000) 2022-01-31T22:07:18.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:18 smithi167 conmon[54076]: debug 2022-01-31T22:07:18.614+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:18.614874+0000) 2022-01-31T22:07:19.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:18 smithi167 conmon[49112]: debug 2022-01-31T22:07:18.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:18.949135+0000) 2022-01-31T22:07:19.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[46715]: debug 2022-01-31T22:07:19.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.116276+0000) 2022-01-31T22:07:19.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:18 smithi171 conmon[51620]: debug 2022-01-31T22:07:18.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:18.968312+0000) 2022-01-31T22:07:19.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[35325]: debug 2022-01-31T22:07:19.380+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184141 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:19.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[41853]: debug 2022-01-31T22:07:19.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.322773+0000) 2022-01-31T22:07:19.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[41853]: debug 2022-01-31T22:07:19.360+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.361251+0000) 2022-01-31T22:07:19.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[46715]: debug 2022-01-31T22:07:19.361+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.362194+0000) 2022-01-31T22:07:19.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[51620]: debug 2022-01-31T22:07:19.360+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.360869+0000) 2022-01-31T22:07:19.614 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:19 smithi167 conmon[49112]: debug 2022-01-31T22:07:19.359+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.361291+0000) 2022-01-31T22:07:19.615 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:19 smithi167 conmon[54076]: debug 2022-01-31T22:07:19.360+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.361783+0000) 2022-01-31T22:07:19.615 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:19 smithi167 conmon[60316]: debug 2022-01-31T22:07:19.359+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.361439+0000) 2022-01-31T22:07:19.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:19 smithi167 conmon[60316]: debug 2022-01-31T22:07:19.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.615078+0000) 2022-01-31T22:07:19.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:19 smithi167 conmon[54076]: debug 2022-01-31T22:07:19.614+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.615080+0000) 2022-01-31T22:07:20.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:20 smithi171 conmon[46715]: debug 2022-01-31T22:07:20.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:20.116474+0000) 2022-01-31T22:07:20.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:19 smithi171 conmon[51620]: debug 2022-01-31T22:07:19.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.968509+0000) 2022-01-31T22:07:20.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:19 smithi167 conmon[49112]: debug 2022-01-31T22:07:19.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:19.949297+0000) 2022-01-31T22:07:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:20 smithi171 conmon[41853]: debug 2022-01-31T22:07:20.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:20.322935+0000) 2022-01-31T22:07:20.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:20 smithi167 conmon[54076]: debug 2022-01-31T22:07:20.614+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:20.615252+0000) 2022-01-31T22:07:20.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:20 smithi167 conmon[60316]: debug 2022-01-31T22:07:20.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:20.615254+0000) 2022-01-31T22:07:21.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:20 smithi167 conmon[49112]: debug 2022-01-31T22:07:20.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:20.949456+0000) 2022-01-31T22:07:21.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:21 smithi171 conmon[46715]: debug 2022-01-31T22:07:21.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:21.116613+0000) 2022-01-31T22:07:21.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:20 smithi171 conmon[51620]: debug 2022-01-31T22:07:20.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:20.968630+0000) 2022-01-31T22:07:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:21 smithi171 conmon[41853]: debug 2022-01-31T22:07:21.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:21.323069+0000) 2022-01-31T22:07:21.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:21 smithi167 conmon[54076]: debug 2022-01-31T22:07:21.614+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:21.615484+0000) 2022-01-31T22:07:21.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:21 smithi167 conmon[60316]: debug 2022-01-31T22:07:21.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:21.615484+0000) 2022-01-31T22:07:22.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:21 smithi167 conmon[49112]: debug 2022-01-31T22:07:21.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:21.949668+0000) 2022-01-31T22:07:22.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:22 smithi171 conmon[46715]: debug 2022-01-31T22:07:22.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:22.116807+0000) 2022-01-31T22:07:22.281 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:21 smithi171 conmon[51620]: debug 2022-01-31T22:07:21.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:21.968844+0000) 2022-01-31T22:07:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:22 smithi171 conmon[41853]: debug 2022-01-31T22:07:22.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:22.323259+0000) 2022-01-31T22:07:22.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:22 smithi167 conmon[60316]: debug 2022-01-31T22:07:22.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:22.615731+0000) 2022-01-31T22:07:22.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:22 smithi167 conmon[54076]: debug 2022-01-31T22:07:22.615+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:22.615731+0000) 2022-01-31T22:07:23.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:22 smithi167 conmon[49112]: debug 2022-01-31T22:07:22.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:22.949865+0000) 2022-01-31T22:07:23.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:23 smithi171 conmon[46715]: debug 2022-01-31T22:07:23.115+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:23.116974+0000) 2022-01-31T22:07:23.281 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:22 smithi171 conmon[51620]: debug 2022-01-31T22:07:22.967+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:22.968991+0000) 2022-01-31T22:07:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:23 smithi171 conmon[41853]: debug 2022-01-31T22:07:23.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:23.323493+0000) 2022-01-31T22:07:23.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:23 smithi167 conmon[60316]: debug 2022-01-31T22:07:23.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:23.615933+0000) 2022-01-31T22:07:23.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:23 smithi167 conmon[54076]: debug 2022-01-31T22:07:23.615+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:23.615933+0000) 2022-01-31T22:07:24.058 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:23 smithi171 conmon[35325]: debug 2022-01-31T22:07:23.799+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:07:24.058 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:23 smithi171 conmon[51620]: debug 2022-01-31T22:07:23.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:23.969207+0000) 2022-01-31T22:07:24.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:23 smithi167 conmon[49112]: debug 2022-01-31T22:07:23.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:23.950036+0000) 2022-01-31T22:07:24.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[46715]: debug 2022-01-31T22:07:24.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.117165+0000) 2022-01-31T22:07:24.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[35325]: debug 2022-01-31T22:07:24.400+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184254 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:24.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[51620]: debug 2022-01-31T22:07:24.382+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.384216+0000) 2022-01-31T22:07:24.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[41853]: debug 2022-01-31T22:07:24.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.323685+0000) 2022-01-31T22:07:24.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[41853]: debug 2022-01-31T22:07:24.381+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.383376+0000) 2022-01-31T22:07:24.609 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[46715]: debug 2022-01-31T22:07:24.382+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.384445+0000) 2022-01-31T22:07:24.666 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:24 smithi167 conmon[49112]: debug 2022-01-31T22:07:24.382+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.382804+0000) 2022-01-31T22:07:24.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:24 smithi167 conmon[54076]: debug 2022-01-31T22:07:24.383+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.383588+0000) 2022-01-31T22:07:24.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:24 smithi167 conmon[54076]: debug 2022-01-31T22:07:24.615+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.616153+0000) 2022-01-31T22:07:24.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:24 smithi167 conmon[60316]: debug 2022-01-31T22:07:24.383+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.384038+0000) 2022-01-31T22:07:24.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:24 smithi167 conmon[60316]: debug 2022-01-31T22:07:24.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.616153+0000) 2022-01-31T22:07:25.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:25 smithi171 conmon[46715]: debug 2022-01-31T22:07:25.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:25.117367+0000) 2022-01-31T22:07:25.281 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:24 smithi171 conmon[51620]: debug 2022-01-31T22:07:24.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.969419+0000) 2022-01-31T22:07:25.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:24 smithi167 conmon[49112]: debug 2022-01-31T22:07:24.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:24.950230+0000) 2022-01-31T22:07:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:25 smithi171 conmon[41853]: debug 2022-01-31T22:07:25.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:25.323847+0000) 2022-01-31T22:07:25.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:25 smithi167 conmon[54076]: debug 2022-01-31T22:07:25.615+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:25.616272+0000) 2022-01-31T22:07:25.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:25 smithi167 conmon[60316]: debug 2022-01-31T22:07:25.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:25.616281+0000) 2022-01-31T22:07:26.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:25 smithi167 conmon[49112]: debug 2022-01-31T22:07:25.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:25.950326+0000) 2022-01-31T22:07:26.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:26 smithi171 conmon[46715]: debug 2022-01-31T22:07:26.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:26.117526+0000) 2022-01-31T22:07:26.281 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:25 smithi171 conmon[51620]: debug 2022-01-31T22:07:25.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:25.969578+0000) 2022-01-31T22:07:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:26 smithi171 conmon[41853]: debug 2022-01-31T22:07:26.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:26.323945+0000) 2022-01-31T22:07:26.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:26 smithi167 conmon[54076]: debug 2022-01-31T22:07:26.615+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:26.616378+0000) 2022-01-31T22:07:26.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:26 smithi167 conmon[60316]: debug 2022-01-31T22:07:26.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:26.616521+0000) 2022-01-31T22:07:27.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:26 smithi167 conmon[49112]: debug 2022-01-31T22:07:26.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:26.950496+0000) 2022-01-31T22:07:27.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:27 smithi171 conmon[46715]: debug 2022-01-31T22:07:27.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:27.117713+0000) 2022-01-31T22:07:27.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:26 smithi171 conmon[51620]: debug 2022-01-31T22:07:26.968+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:26.969781+0000) 2022-01-31T22:07:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:27 smithi171 conmon[41853]: debug 2022-01-31T22:07:27.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:27.324099+0000) 2022-01-31T22:07:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:27 smithi167 conmon[54076]: debug 2022-01-31T22:07:27.616+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:27.616558+0000) 2022-01-31T22:07:27.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:27 smithi167 conmon[60316]: debug 2022-01-31T22:07:27.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:27.616682+0000) 2022-01-31T22:07:28.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:27 smithi167 conmon[49112]: debug 2022-01-31T22:07:27.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:27.950717+0000) 2022-01-31T22:07:28.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:28 smithi171 conmon[46715]: debug 2022-01-31T22:07:28.116+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:28.117879+0000) 2022-01-31T22:07:28.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:27 smithi171 conmon[51620]: debug 2022-01-31T22:07:27.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:27.969967+0000) 2022-01-31T22:07:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:28 smithi171 conmon[41853]: debug 2022-01-31T22:07:28.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:28.324229+0000) 2022-01-31T22:07:28.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:28 smithi167 conmon[54076]: debug 2022-01-31T22:07:28.616+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:28.616759+0000) 2022-01-31T22:07:28.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:28 smithi167 conmon[60316]: debug 2022-01-31T22:07:28.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:28.616886+0000) 2022-01-31T22:07:29.106 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:28 smithi167 conmon[49112]: debug 2022-01-31T22:07:28.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:28.950907+0000) 2022-01-31T22:07:29.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[46715]: debug 2022-01-31T22:07:29.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.118080+0000) 2022-01-31T22:07:29.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:28 smithi171 conmon[51620]: debug 2022-01-31T22:07:28.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:28.970146+0000) 2022-01-31T22:07:29.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[35325]: debug 2022-01-31T22:07:29.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184363 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:29.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[51620]: debug 2022-01-31T22:07:29.403+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.405313+0000) 2022-01-31T22:07:29.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[41853]: debug 2022-01-31T22:07:29.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.324405+0000) 2022-01-31T22:07:29.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[41853]: debug 2022-01-31T22:07:29.402+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.404406+0000) 2022-01-31T22:07:29.609 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[46715]: debug 2022-01-31T22:07:29.403+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.405100+0000) 2022-01-31T22:07:29.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:29 smithi167 conmon[49112]: debug 2022-01-31T22:07:29.405+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.405909+0000) 2022-01-31T22:07:29.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:29 smithi167 conmon[54076]: debug 2022-01-31T22:07:29.405+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.405836+0000) 2022-01-31T22:07:29.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:29 smithi167 conmon[54076]: debug 2022-01-31T22:07:29.616+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.616950+0000) 2022-01-31T22:07:29.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:29 smithi167 conmon[60316]: debug 2022-01-31T22:07:29.404+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.405463+0000) 2022-01-31T22:07:29.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:29 smithi167 conmon[60316]: debug 2022-01-31T22:07:29.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.617044+0000) 2022-01-31T22:07:30.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:30 smithi171 conmon[46715]: debug 2022-01-31T22:07:30.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:30.118288+0000) 2022-01-31T22:07:30.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:29 smithi171 conmon[51620]: debug 2022-01-31T22:07:29.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.970340+0000) 2022-01-31T22:07:30.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:29 smithi167 conmon[49112]: debug 2022-01-31T22:07:29.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:29.951101+0000) 2022-01-31T22:07:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:30 smithi171 conmon[41853]: debug 2022-01-31T22:07:30.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:30.324599+0000) 2022-01-31T22:07:30.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:30 smithi167 conmon[54076]: debug 2022-01-31T22:07:30.616+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:30.617192+0000) 2022-01-31T22:07:30.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:30 smithi167 conmon[60316]: debug 2022-01-31T22:07:30.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:30.617199+0000) 2022-01-31T22:07:31.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:30 smithi167 conmon[49112]: debug 2022-01-31T22:07:30.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:30.951276+0000) 2022-01-31T22:07:31.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:31 smithi171 conmon[46715]: debug 2022-01-31T22:07:31.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:31.118462+0000) 2022-01-31T22:07:31.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:30 smithi171 conmon[51620]: debug 2022-01-31T22:07:30.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:30.970476+0000) 2022-01-31T22:07:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:31 smithi171 conmon[41853]: debug 2022-01-31T22:07:31.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:31.324765+0000) 2022-01-31T22:07:31.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:31 smithi167 conmon[54076]: debug 2022-01-31T22:07:31.616+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:31.617422+0000) 2022-01-31T22:07:31.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:31 smithi167 conmon[60316]: debug 2022-01-31T22:07:31.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:31.617423+0000) 2022-01-31T22:07:32.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:31 smithi167 conmon[49112]: debug 2022-01-31T22:07:31.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:31.951459+0000) 2022-01-31T22:07:32.282 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:32 smithi171 conmon[46715]: debug 2022-01-31T22:07:32.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:32.118638+0000) 2022-01-31T22:07:32.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:31 smithi171 conmon[51620]: debug 2022-01-31T22:07:31.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:31.970656+0000) 2022-01-31T22:07:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:32 smithi171 conmon[41853]: debug 2022-01-31T22:07:32.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:32.324950+0000) 2022-01-31T22:07:32.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:32 smithi167 conmon[60316]: debug 2022-01-31T22:07:32.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:32.617609+0000) 2022-01-31T22:07:32.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:32 smithi167 conmon[54076]: debug 2022-01-31T22:07:32.617+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:32.617608+0000) 2022-01-31T22:07:33.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:32 smithi167 conmon[49112]: debug 2022-01-31T22:07:32.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:32.951664+0000) 2022-01-31T22:07:33.282 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:33 smithi171 conmon[46715]: debug 2022-01-31T22:07:33.117+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:33.118800+0000) 2022-01-31T22:07:33.283 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:32 smithi171 conmon[51620]: debug 2022-01-31T22:07:32.969+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:32.970839+0000) 2022-01-31T22:07:33.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:33 smithi171 conmon[41853]: debug 2022-01-31T22:07:33.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:33.325182+0000) 2022-01-31T22:07:33.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:33 smithi167 conmon[60316]: debug 2022-01-31T22:07:33.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:33.617760+0000) 2022-01-31T22:07:33.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:33 smithi167 conmon[54076]: debug 2022-01-31T22:07:33.617+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:33.617755+0000) 2022-01-31T22:07:34.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:33 smithi167 conmon[49112]: debug 2022-01-31T22:07:33.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:33.951796+0000) 2022-01-31T22:07:34.282 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[46715]: debug 2022-01-31T22:07:34.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.119019+0000) 2022-01-31T22:07:34.283 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:33 smithi171 conmon[51620]: debug 2022-01-31T22:07:33.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:33.971004+0000) 2022-01-31T22:07:34.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[35325]: debug 2022-01-31T22:07:34.445+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184484 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:34.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[46715]: debug 2022-01-31T22:07:34.426+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.427052+0000) 2022-01-31T22:07:34.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[51620]: debug 2022-01-31T22:07:34.426+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.427223+0000) 2022-01-31T22:07:34.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[41853]: debug 2022-01-31T22:07:34.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.325347+0000) 2022-01-31T22:07:34.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[41853]: debug 2022-01-31T22:07:34.426+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.426864+0000) 2022-01-31T22:07:34.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:34 smithi167 conmon[49112]: debug 2022-01-31T22:07:34.428+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.428602+0000) 2022-01-31T22:07:34.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:34 smithi167 conmon[60316]: debug 2022-01-31T22:07:34.426+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.427326+0000) 2022-01-31T22:07:34.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:34 smithi167 conmon[60316]: debug 2022-01-31T22:07:34.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.617915+0000) 2022-01-31T22:07:34.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:34 smithi167 conmon[54076]: debug 2022-01-31T22:07:34.427+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.428188+0000) 2022-01-31T22:07:34.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:34 smithi167 conmon[54076]: debug 2022-01-31T22:07:34.617+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.617854+0000) 2022-01-31T22:07:35.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:34 smithi167 conmon[49112]: debug 2022-01-31T22:07:34.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.951976+0000) 2022-01-31T22:07:35.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:35 smithi171 conmon[46715]: debug 2022-01-31T22:07:35.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:35.119205+0000) 2022-01-31T22:07:35.283 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:34 smithi171 conmon[51620]: debug 2022-01-31T22:07:34.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:34.971187+0000) 2022-01-31T22:07:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:35 smithi171 conmon[41853]: debug 2022-01-31T22:07:35.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:35.325532+0000) 2022-01-31T22:07:35.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:35 smithi167 conmon[54076]: debug 2022-01-31T22:07:35.617+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:35.617988+0000) 2022-01-31T22:07:35.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:35 smithi167 conmon[60316]: debug 2022-01-31T22:07:35.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:35.618051+0000) 2022-01-31T22:07:36.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:35 smithi167 conmon[49112]: debug 2022-01-31T22:07:35.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:35.952131+0000) 2022-01-31T22:07:36.282 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:36 smithi171 conmon[46715]: debug 2022-01-31T22:07:36.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:36.119362+0000) 2022-01-31T22:07:36.283 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:35 smithi171 conmon[51620]: debug 2022-01-31T22:07:35.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:35.971340+0000) 2022-01-31T22:07:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:36 smithi171 conmon[41853]: debug 2022-01-31T22:07:36.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:36.325685+0000) 2022-01-31T22:07:36.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:36 smithi167 conmon[54076]: debug 2022-01-31T22:07:36.617+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:36.618193+0000) 2022-01-31T22:07:36.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:36 smithi167 conmon[60316]: debug 2022-01-31T22:07:36.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:36.618217+0000) 2022-01-31T22:07:37.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:36 smithi167 conmon[49112]: debug 2022-01-31T22:07:36.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:36.952318+0000) 2022-01-31T22:07:37.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:37 smithi171 conmon[46715]: debug 2022-01-31T22:07:37.118+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:37.119545+0000) 2022-01-31T22:07:37.283 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:36 smithi171 conmon[51620]: debug 2022-01-31T22:07:36.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:36.971499+0000) 2022-01-31T22:07:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:37 smithi171 conmon[41853]: debug 2022-01-31T22:07:37.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:37.325889+0000) 2022-01-31T22:07:37.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:37 smithi167 conmon[54076]: debug 2022-01-31T22:07:37.617+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:37.618390+0000) 2022-01-31T22:07:37.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:37 smithi167 conmon[60316]: debug 2022-01-31T22:07:37.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:37.618401+0000) 2022-01-31T22:07:38.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:37 smithi167 conmon[49112]: debug 2022-01-31T22:07:37.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:37.952516+0000) 2022-01-31T22:07:38.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:38 smithi171 conmon[46715]: debug 2022-01-31T22:07:38.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:38.119767+0000) 2022-01-31T22:07:38.284 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:37 smithi171 conmon[51620]: debug 2022-01-31T22:07:37.970+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:37.971673+0000) 2022-01-31T22:07:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:38 smithi171 conmon[41853]: debug 2022-01-31T22:07:38.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:38.326093+0000) 2022-01-31T22:07:38.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:38 smithi167 conmon[54076]: debug 2022-01-31T22:07:38.618+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:38.618596+0000) 2022-01-31T22:07:38.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:38 smithi167 conmon[60316]: debug 2022-01-31T22:07:38.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:38.618595+0000) 2022-01-31T22:07:39.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:38 smithi171 conmon[51620]: debug 2022-01-31T22:07:38.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:38.971858+0000) 2022-01-31T22:07:39.061 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:38 smithi171 conmon[35325]: debug 2022-01-31T22:07:38.800+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:07:39.108 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:38 smithi167 conmon[49112]: debug 2022-01-31T22:07:38.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:38.952643+0000) 2022-01-31T22:07:39.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[46715]: debug 2022-01-31T22:07:39.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.119911+0000) 2022-01-31T22:07:39.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[35325]: debug 2022-01-31T22:07:39.482+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184589 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:39.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[41853]: debug 2022-01-31T22:07:39.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.326279+0000) 2022-01-31T22:07:39.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[41853]: debug 2022-01-31T22:07:39.446+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.448578+0000) 2022-01-31T22:07:39.609 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[46715]: debug 2022-01-31T22:07:39.446+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.448343+0000) 2022-01-31T22:07:39.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[51620]: debug 2022-01-31T22:07:39.446+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.449360+0000) 2022-01-31T22:07:39.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:39 smithi167 conmon[49112]: debug 2022-01-31T22:07:39.449+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.449489+0000) 2022-01-31T22:07:39.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:39 smithi167 conmon[54076]: debug 2022-01-31T22:07:39.455+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.456135+0000) 2022-01-31T22:07:39.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:39 smithi167 conmon[54076]: debug 2022-01-31T22:07:39.618+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.618733+0000) 2022-01-31T22:07:39.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:39 smithi167 conmon[60316]: debug 2022-01-31T22:07:39.454+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.455339+0000) 2022-01-31T22:07:39.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:39 smithi167 conmon[60316]: debug 2022-01-31T22:07:39.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.618822+0000) 2022-01-31T22:07:40.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:39 smithi167 conmon[49112]: debug 2022-01-31T22:07:39.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.952806+0000) 2022-01-31T22:07:40.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:40 smithi171 conmon[46715]: debug 2022-01-31T22:07:40.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:40.120060+0000) 2022-01-31T22:07:40.284 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:39 smithi171 conmon[51620]: debug 2022-01-31T22:07:39.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:39.971983+0000) 2022-01-31T22:07:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:40 smithi171 conmon[41853]: debug 2022-01-31T22:07:40.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:40.326464+0000) 2022-01-31T22:07:40.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:40 smithi167 conmon[54076]: debug 2022-01-31T22:07:40.618+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:40.618872+0000) 2022-01-31T22:07:40.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:40 smithi167 conmon[60316]: debug 2022-01-31T22:07:40.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:40.618942+0000) 2022-01-31T22:07:41.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:40 smithi167 conmon[49112]: debug 2022-01-31T22:07:40.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:40.952980+0000) 2022-01-31T22:07:41.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:41 smithi171 conmon[46715]: debug 2022-01-31T22:07:41.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:41.120221+0000) 2022-01-31T22:07:41.284 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:40 smithi171 conmon[51620]: debug 2022-01-31T22:07:40.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:40.972154+0000) 2022-01-31T22:07:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:41 smithi171 conmon[41853]: debug 2022-01-31T22:07:41.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:41.326624+0000) 2022-01-31T22:07:41.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:41 smithi167 conmon[54076]: debug 2022-01-31T22:07:41.618+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:41.619023+0000) 2022-01-31T22:07:41.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:41 smithi167 conmon[60316]: debug 2022-01-31T22:07:41.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:41.619105+0000) 2022-01-31T22:07:42.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:41 smithi167 conmon[49112]: debug 2022-01-31T22:07:41.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:41.953179+0000) 2022-01-31T22:07:42.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:42 smithi171 conmon[46715]: debug 2022-01-31T22:07:42.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:42.120391+0000) 2022-01-31T22:07:42.284 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:41 smithi171 conmon[51620]: debug 2022-01-31T22:07:41.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:41.972348+0000) 2022-01-31T22:07:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:42 smithi171 conmon[41853]: debug 2022-01-31T22:07:42.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:42.326809+0000) 2022-01-31T22:07:42.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:42 smithi167 conmon[54076]: debug 2022-01-31T22:07:42.618+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:42.619174+0000) 2022-01-31T22:07:42.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:42 smithi167 conmon[60316]: debug 2022-01-31T22:07:42.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:42.619251+0000) 2022-01-31T22:07:43.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:42 smithi167 conmon[49112]: debug 2022-01-31T22:07:42.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:42.953364+0000) 2022-01-31T22:07:43.284 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:43 smithi171 conmon[46715]: debug 2022-01-31T22:07:43.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:43.120544+0000) 2022-01-31T22:07:43.284 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:42 smithi171 conmon[51620]: debug 2022-01-31T22:07:42.971+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:42.972529+0000) 2022-01-31T22:07:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:43 smithi171 conmon[41853]: debug 2022-01-31T22:07:43.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:43.327010+0000) 2022-01-31T22:07:43.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:43 smithi167 conmon[54076]: debug 2022-01-31T22:07:43.618+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:43.619382+0000) 2022-01-31T22:07:43.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:43 smithi167 conmon[60316]: debug 2022-01-31T22:07:43.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:43.619411+0000) 2022-01-31T22:07:44.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:43 smithi167 conmon[49112]: debug 2022-01-31T22:07:43.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:43.953590+0000) 2022-01-31T22:07:44.284 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[46715]: debug 2022-01-31T22:07:44.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.120746+0000) 2022-01-31T22:07:44.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:43 smithi171 conmon[51620]: debug 2022-01-31T22:07:43.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:43.972705+0000) 2022-01-31T22:07:44.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[35325]: debug 2022-01-31T22:07:44.503+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184704 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[41853]: debug 2022-01-31T22:07:44.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.327186+0000) 2022-01-31T22:07:44.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[41853]: debug 2022-01-31T22:07:44.484+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.485028+0000) 2022-01-31T22:07:44.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[46715]: debug 2022-01-31T22:07:44.484+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.485583+0000) 2022-01-31T22:07:44.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[51620]: debug 2022-01-31T22:07:44.486+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.487491+0000) 2022-01-31T22:07:44.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:44 smithi167 conmon[49112]: debug 2022-01-31T22:07:44.487+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.487589+0000) 2022-01-31T22:07:44.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:44 smithi167 conmon[54076]: debug 2022-01-31T22:07:44.487+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.488037+0000) 2022-01-31T22:07:44.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:44 smithi167 conmon[54076]: debug 2022-01-31T22:07:44.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.619603+0000) 2022-01-31T22:07:44.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:44 smithi167 conmon[60316]: debug 2022-01-31T22:07:44.485+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.485932+0000) 2022-01-31T22:07:44.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:44 smithi167 conmon[60316]: debug 2022-01-31T22:07:44.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.619603+0000) 2022-01-31T22:07:45.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:44 smithi167 conmon[49112]: debug 2022-01-31T22:07:44.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.953754+0000) 2022-01-31T22:07:45.284 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:45 smithi171 conmon[46715]: debug 2022-01-31T22:07:45.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:45.120949+0000) 2022-01-31T22:07:45.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:44 smithi171 conmon[51620]: debug 2022-01-31T22:07:44.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:44.972932+0000) 2022-01-31T22:07:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:45 smithi171 conmon[41853]: debug 2022-01-31T22:07:45.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:45.327365+0000) 2022-01-31T22:07:45.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:45 smithi167 conmon[54076]: debug 2022-01-31T22:07:45.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:45.619777+0000) 2022-01-31T22:07:45.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:45 smithi167 conmon[60316]: debug 2022-01-31T22:07:45.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:45.619760+0000) 2022-01-31T22:07:46.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:45 smithi167 conmon[49112]: debug 2022-01-31T22:07:45.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:45.953916+0000) 2022-01-31T22:07:46.284 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:46 smithi171 conmon[46715]: debug 2022-01-31T22:07:46.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:46.121086+0000) 2022-01-31T22:07:46.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:45 smithi171 conmon[51620]: debug 2022-01-31T22:07:45.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:45.973016+0000) 2022-01-31T22:07:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:46 smithi171 conmon[41853]: debug 2022-01-31T22:07:46.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:46.327518+0000) 2022-01-31T22:07:46.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:46 smithi167 conmon[54076]: debug 2022-01-31T22:07:46.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:46.619964+0000) 2022-01-31T22:07:46.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:46 smithi167 conmon[60316]: debug 2022-01-31T22:07:46.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:46.619960+0000) 2022-01-31T22:07:47.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:46 smithi167 conmon[49112]: debug 2022-01-31T22:07:46.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:46.954089+0000) 2022-01-31T22:07:47.284 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:47 smithi171 conmon[46715]: debug 2022-01-31T22:07:47.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:47.121244+0000) 2022-01-31T22:07:47.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:46 smithi171 conmon[51620]: debug 2022-01-31T22:07:46.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:46.973172+0000) 2022-01-31T22:07:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:47 smithi171 conmon[41853]: debug 2022-01-31T22:07:47.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:47.327746+0000) 2022-01-31T22:07:47.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:47 smithi167 conmon[54076]: debug 2022-01-31T22:07:47.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:47.620173+0000) 2022-01-31T22:07:47.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:47 smithi167 conmon[60316]: debug 2022-01-31T22:07:47.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:47.620172+0000) 2022-01-31T22:07:48.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:47 smithi167 conmon[49112]: debug 2022-01-31T22:07:47.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:47.954297+0000) 2022-01-31T22:07:48.284 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:48 smithi171 conmon[46715]: debug 2022-01-31T22:07:48.120+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:48.121371+0000) 2022-01-31T22:07:48.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:47 smithi171 conmon[51620]: debug 2022-01-31T22:07:47.972+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:47.973354+0000) 2022-01-31T22:07:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:48 smithi171 conmon[41853]: debug 2022-01-31T22:07:48.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:48.327959+0000) 2022-01-31T22:07:48.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:48 smithi167 conmon[54076]: debug 2022-01-31T22:07:48.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:48.620361+0000) 2022-01-31T22:07:48.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:48 smithi167 conmon[60316]: debug 2022-01-31T22:07:48.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:48.620363+0000) 2022-01-31T22:07:49.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:48 smithi167 conmon[49112]: debug 2022-01-31T22:07:48.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:48.954476+0000) 2022-01-31T22:07:49.285 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[46715]: debug 2022-01-31T22:07:49.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.121557+0000) 2022-01-31T22:07:49.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:48 smithi171 conmon[51620]: debug 2022-01-31T22:07:48.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:48.973539+0000) 2022-01-31T22:07:49.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[35325]: debug 2022-01-31T22:07:49.525+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184815 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:49.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[41853]: debug 2022-01-31T22:07:49.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.328107+0000) 2022-01-31T22:07:49.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[41853]: debug 2022-01-31T22:07:49.505+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.506546+0000) 2022-01-31T22:07:49.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[46715]: debug 2022-01-31T22:07:49.506+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.507598+0000) 2022-01-31T22:07:49.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[51620]: debug 2022-01-31T22:07:49.506+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.507111+0000) 2022-01-31T22:07:49.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:49 smithi167 conmon[49112]: debug 2022-01-31T22:07:49.505+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.507392+0000) 2022-01-31T22:07:49.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:49 smithi167 conmon[54076]: debug 2022-01-31T22:07:49.506+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.508218+0000) 2022-01-31T22:07:49.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:49 smithi167 conmon[54076]: debug 2022-01-31T22:07:49.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.620571+0000) 2022-01-31T22:07:49.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:49 smithi167 conmon[60316]: debug 2022-01-31T22:07:49.505+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.506827+0000) 2022-01-31T22:07:49.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:49 smithi167 conmon[60316]: debug 2022-01-31T22:07:49.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.620568+0000) 2022-01-31T22:07:50.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:49 smithi167 conmon[49112]: debug 2022-01-31T22:07:49.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.954664+0000) 2022-01-31T22:07:50.285 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:50 smithi171 conmon[46715]: debug 2022-01-31T22:07:50.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:50.121734+0000) 2022-01-31T22:07:50.285 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:49 smithi171 conmon[51620]: debug 2022-01-31T22:07:49.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:49.973745+0000) 2022-01-31T22:07:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:50 smithi171 conmon[41853]: debug 2022-01-31T22:07:50.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:50.328259+0000) 2022-01-31T22:07:50.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:50 smithi167 conmon[60316]: debug 2022-01-31T22:07:50.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:50.620679+0000) 2022-01-31T22:07:50.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:50 smithi167 conmon[54076]: debug 2022-01-31T22:07:50.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:50.620697+0000) 2022-01-31T22:07:51.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:50 smithi167 conmon[49112]: debug 2022-01-31T22:07:50.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:50.954814+0000) 2022-01-31T22:07:51.285 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:51 smithi171 conmon[46715]: debug 2022-01-31T22:07:51.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:51.121913+0000) 2022-01-31T22:07:51.286 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:50 smithi171 conmon[51620]: debug 2022-01-31T22:07:50.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:50.973965+0000) 2022-01-31T22:07:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:51 smithi171 conmon[41853]: debug 2022-01-31T22:07:51.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:51.328422+0000) 2022-01-31T22:07:51.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:51 smithi167 conmon[60316]: debug 2022-01-31T22:07:51.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:51.620900+0000) 2022-01-31T22:07:51.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:51 smithi167 conmon[54076]: debug 2022-01-31T22:07:51.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:51.620900+0000) 2022-01-31T22:07:52.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:51 smithi167 conmon[49112]: debug 2022-01-31T22:07:51.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:51.954970+0000) 2022-01-31T22:07:52.285 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:52 smithi171 conmon[46715]: debug 2022-01-31T22:07:52.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:52.122097+0000) 2022-01-31T22:07:52.286 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:51 smithi171 conmon[51620]: debug 2022-01-31T22:07:51.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:51.974166+0000) 2022-01-31T22:07:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:52 smithi171 conmon[41853]: debug 2022-01-31T22:07:52.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:52.328582+0000) 2022-01-31T22:07:52.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:52 smithi167 conmon[54076]: debug 2022-01-31T22:07:52.619+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:52.621095+0000) 2022-01-31T22:07:52.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:52 smithi167 conmon[60316]: debug 2022-01-31T22:07:52.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:52.621071+0000) 2022-01-31T22:07:53.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:52 smithi167 conmon[49112]: debug 2022-01-31T22:07:52.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:52.955150+0000) 2022-01-31T22:07:53.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:53 smithi171 conmon[46715]: debug 2022-01-31T22:07:53.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:53.122259+0000) 2022-01-31T22:07:53.286 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:52 smithi171 conmon[51620]: debug 2022-01-31T22:07:52.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:52.974373+0000) 2022-01-31T22:07:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:53 smithi171 conmon[41853]: debug 2022-01-31T22:07:53.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:53.328730+0000) 2022-01-31T22:07:53.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:53 smithi167 conmon[54076]: debug 2022-01-31T22:07:53.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:53.621274+0000) 2022-01-31T22:07:53.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:53 smithi167 conmon[60316]: debug 2022-01-31T22:07:53.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:53.621276+0000) 2022-01-31T22:07:54.065 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:53 smithi171 conmon[35325]: debug 2022-01-31T22:07:53.801+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:07:54.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:53 smithi171 conmon[51620]: debug 2022-01-31T22:07:53.973+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:53.974575+0000) 2022-01-31T22:07:54.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:53 smithi167 conmon[49112]: debug 2022-01-31T22:07:53.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:53.955277+0000) 2022-01-31T22:07:54.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[46715]: debug 2022-01-31T22:07:54.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.122437+0000) 2022-01-31T22:07:54.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[35325]: debug 2022-01-31T22:07:54.546+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 184928 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:54.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[46715]: debug 2022-01-31T22:07:54.527+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.528542+0000) 2022-01-31T22:07:54.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[51620]: debug 2022-01-31T22:07:54.526+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.527689+0000) 2022-01-31T22:07:54.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[41853]: debug 2022-01-31T22:07:54.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.328859+0000) 2022-01-31T22:07:54.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[41853]: debug 2022-01-31T22:07:54.527+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.527885+0000) 2022-01-31T22:07:54.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:54 smithi167 conmon[49112]: debug 2022-01-31T22:07:54.527+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.528644+0000) 2022-01-31T22:07:54.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:54 smithi167 conmon[60316]: debug 2022-01-31T22:07:54.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.529175+0000) 2022-01-31T22:07:54.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:54 smithi167 conmon[60316]: debug 2022-01-31T22:07:54.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.621468+0000) 2022-01-31T22:07:54.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:54 smithi167 conmon[54076]: debug 2022-01-31T22:07:54.527+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.529120+0000) 2022-01-31T22:07:54.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:54 smithi167 conmon[54076]: debug 2022-01-31T22:07:54.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.621467+0000) 2022-01-31T22:07:55.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:54 smithi167 conmon[49112]: debug 2022-01-31T22:07:54.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.955445+0000) 2022-01-31T22:07:55.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:55 smithi171 conmon[46715]: debug 2022-01-31T22:07:55.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:55.122646+0000) 2022-01-31T22:07:55.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:54 smithi171 conmon[51620]: debug 2022-01-31T22:07:54.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:54.974790+0000) 2022-01-31T22:07:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:55 smithi171 conmon[41853]: debug 2022-01-31T22:07:55.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:55.329066+0000) 2022-01-31T22:07:55.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:55 smithi167 conmon[54076]: debug 2022-01-31T22:07:55.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:55.621657+0000) 2022-01-31T22:07:55.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:55 smithi167 conmon[60316]: debug 2022-01-31T22:07:55.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:55.621619+0000) 2022-01-31T22:07:56.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:55 smithi167 conmon[49112]: debug 2022-01-31T22:07:55.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:55.955604+0000) 2022-01-31T22:07:56.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:56 smithi171 conmon[46715]: debug 2022-01-31T22:07:56.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:56.122796+0000) 2022-01-31T22:07:56.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:55 smithi171 conmon[51620]: debug 2022-01-31T22:07:55.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:55.974934+0000) 2022-01-31T22:07:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:56 smithi171 conmon[41853]: debug 2022-01-31T22:07:56.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:56.329224+0000) 2022-01-31T22:07:56.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:56 smithi167 conmon[54076]: debug 2022-01-31T22:07:56.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:56.621831+0000) 2022-01-31T22:07:56.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:56 smithi167 conmon[60316]: debug 2022-01-31T22:07:56.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:56.621820+0000) 2022-01-31T22:07:57.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:56 smithi167 conmon[49112]: debug 2022-01-31T22:07:56.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:56.955806+0000) 2022-01-31T22:07:57.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:57 smithi171 conmon[46715]: debug 2022-01-31T22:07:57.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:57.122945+0000) 2022-01-31T22:07:57.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:56 smithi171 conmon[51620]: debug 2022-01-31T22:07:56.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:56.975118+0000) 2022-01-31T22:07:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:57 smithi171 conmon[41853]: debug 2022-01-31T22:07:57.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:57.329380+0000) 2022-01-31T22:07:57.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:57 smithi167 conmon[60316]: debug 2022-01-31T22:07:57.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:57.622011+0000) 2022-01-31T22:07:57.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:57 smithi167 conmon[54076]: debug 2022-01-31T22:07:57.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:57.622010+0000) 2022-01-31T22:07:58.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:57 smithi167 conmon[49112]: debug 2022-01-31T22:07:57.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:57.955994+0000) 2022-01-31T22:07:58.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:58 smithi171 conmon[46715]: debug 2022-01-31T22:07:58.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:58.123104+0000) 2022-01-31T22:07:58.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:57 smithi171 conmon[51620]: debug 2022-01-31T22:07:57.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:57.975325+0000) 2022-01-31T22:07:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:58 smithi171 conmon[41853]: debug 2022-01-31T22:07:58.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:58.329545+0000) 2022-01-31T22:07:58.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:58 smithi167 conmon[54076]: debug 2022-01-31T22:07:58.621+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:58.622255+0000) 2022-01-31T22:07:58.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:58 smithi167 conmon[60316]: debug 2022-01-31T22:07:58.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:58.622206+0000) 2022-01-31T22:07:59.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:58 smithi167 conmon[49112]: debug 2022-01-31T22:07:58.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:58.956209+0000) 2022-01-31T22:07:59.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[46715]: debug 2022-01-31T22:07:59.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.123291+0000) 2022-01-31T22:07:59.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:58 smithi171 conmon[51620]: debug 2022-01-31T22:07:58.974+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:58.975553+0000) 2022-01-31T22:07:59.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[41853]: debug 2022-01-31T22:07:59.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.329719+0000) 2022-01-31T22:07:59.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:59 smithi167 conmon[49112]: debug 2022-01-31T22:07:59.549+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.551284+0000) 2022-01-31T22:07:59.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:59 smithi167 conmon[54076]: debug 2022-01-31T22:07:59.549+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.551144+0000) 2022-01-31T22:07:59.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:07:59 smithi167 conmon[54076]: debug 2022-01-31T22:07:59.620+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.622451+0000) 2022-01-31T22:07:59.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:59 smithi167 conmon[60316]: debug 2022-01-31T22:07:59.548+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.549904+0000) 2022-01-31T22:07:59.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:07:59 smithi167 conmon[60316]: debug 2022-01-31T22:07:59.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.622414+0000) 2022-01-31T22:07:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[41853]: debug 2022-01-31T22:07:59.549+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.549971+0000) 2022-01-31T22:07:59.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[46715]: debug 2022-01-31T22:07:59.548+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.549404+0000) 2022-01-31T22:07:59.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[51620]: debug 2022-01-31T22:07:59.549+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.549617+0000) 2022-01-31T22:07:59.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[35325]: debug 2022-01-31T22:07:59.569+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185037 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:07:59.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[35325]: 2022-01-31T22:08:00.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:07:59 smithi167 conmon[49112]: debug 2022-01-31T22:07:59.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.956345+0000) 2022-01-31T22:08:00.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:00 smithi171 conmon[46715]: debug 2022-01-31T22:08:00.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:00.123444+0000) 2022-01-31T22:08:00.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:07:59 smithi171 conmon[51620]: debug 2022-01-31T22:07:59.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:07:59.975749+0000) 2022-01-31T22:08:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:00 smithi171 conmon[41853]: debug 2022-01-31T22:08:00.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:00.329927+0000) 2022-01-31T22:08:00.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:00 smithi167 conmon[54076]: debug 2022-01-31T22:08:00.621+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:00.622670+0000) 2022-01-31T22:08:00.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:00 smithi167 conmon[60316]: debug 2022-01-31T22:08:00.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:00.622583+0000) 2022-01-31T22:08:01.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:00 smithi167 conmon[49112]: debug 2022-01-31T22:08:00.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:00.956485+0000) 2022-01-31T22:08:01.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:01 smithi171 conmon[46715]: debug 2022-01-31T22:08:01.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:01.123597+0000) 2022-01-31T22:08:01.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:00 smithi171 conmon[51620]: debug 2022-01-31T22:08:00.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:00.975933+0000) 2022-01-31T22:08:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:01 smithi171 conmon[41853]: debug 2022-01-31T22:08:01.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:01.330045+0000) 2022-01-31T22:08:01.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:01 smithi167 conmon[54076]: debug 2022-01-31T22:08:01.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:01.622916+0000) 2022-01-31T22:08:01.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:01 smithi167 conmon[60316]: debug 2022-01-31T22:08:01.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:01.622831+0000) 2022-01-31T22:08:02.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:01 smithi167 conmon[49112]: debug 2022-01-31T22:08:01.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:01.956716+0000) 2022-01-31T22:08:02.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:02 smithi171 conmon[46715]: debug 2022-01-31T22:08:02.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:02.123731+0000) 2022-01-31T22:08:02.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:01 smithi171 conmon[51620]: debug 2022-01-31T22:08:01.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:01.976124+0000) 2022-01-31T22:08:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:02 smithi171 conmon[41853]: debug 2022-01-31T22:08:02.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:02.330181+0000) 2022-01-31T22:08:02.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:02 smithi167 conmon[54076]: debug 2022-01-31T22:08:02.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:02.623098+0000) 2022-01-31T22:08:02.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:02 smithi167 conmon[60316]: debug 2022-01-31T22:08:02.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:02.623023+0000) 2022-01-31T22:08:03.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:02 smithi167 conmon[49112]: debug 2022-01-31T22:08:02.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:02.956913+0000) 2022-01-31T22:08:03.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:03 smithi171 conmon[46715]: debug 2022-01-31T22:08:03.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:03.123895+0000) 2022-01-31T22:08:03.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:02 smithi171 conmon[51620]: debug 2022-01-31T22:08:02.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:02.976310+0000) 2022-01-31T22:08:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:03 smithi171 conmon[41853]: debug 2022-01-31T22:08:03.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:03.330348+0000) 2022-01-31T22:08:03.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:03 smithi167 conmon[54076]: debug 2022-01-31T22:08:03.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:03.623267+0000) 2022-01-31T22:08:03.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:03 smithi167 conmon[60316]: debug 2022-01-31T22:08:03.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:03.623199+0000) 2022-01-31T22:08:04.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:03 smithi167 conmon[49112]: debug 2022-01-31T22:08:03.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:03.957105+0000) 2022-01-31T22:08:04.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[46715]: debug 2022-01-31T22:08:04.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.124100+0000) 2022-01-31T22:08:04.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:03 smithi171 conmon[51620]: debug 2022-01-31T22:08:03.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:03.976503+0000) 2022-01-31T22:08:04.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[41853]: debug 2022-01-31T22:08:04.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.330509+0000) 2022-01-31T22:08:04.825 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:04 smithi167 conmon[49112]: debug 2022-01-31T22:08:04.571+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.573425+0000) 2022-01-31T22:08:04.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:04 smithi167 conmon[60316]: debug 2022-01-31T22:08:04.570+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.572086+0000) 2022-01-31T22:08:04.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:04 smithi167 conmon[60316]: debug 2022-01-31T22:08:04.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.623328+0000) 2022-01-31T22:08:04.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:04 smithi167 conmon[54076]: debug 2022-01-31T22:08:04.571+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.573343+0000) 2022-01-31T22:08:04.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:04 smithi167 conmon[54076]: debug 2022-01-31T22:08:04.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.623404+0000) 2022-01-31T22:08:04.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[35325]: debug 2022-01-31T22:08:04.591+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185150 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:04.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[41853]: debug 2022-01-31T22:08:04.571+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.572518+0000) 2022-01-31T22:08:04.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[46715]: debug 2022-01-31T22:08:04.572+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.573009+0000) 2022-01-31T22:08:04.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[51620]: debug 2022-01-31T22:08:04.571+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.571844+0000) 2022-01-31T22:08:05.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:04 smithi167 conmon[49112]: debug 2022-01-31T22:08:04.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.957370+0000) 2022-01-31T22:08:05.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:05 smithi171 conmon[46715]: debug 2022-01-31T22:08:05.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:05.124284+0000) 2022-01-31T22:08:05.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:04 smithi171 conmon[51620]: debug 2022-01-31T22:08:04.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:04.976662+0000) 2022-01-31T22:08:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:05 smithi171 conmon[41853]: debug 2022-01-31T22:08:05.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:05.330663+0000) 2022-01-31T22:08:05.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:05 smithi167 conmon[54076]: debug 2022-01-31T22:08:05.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:05.623539+0000) 2022-01-31T22:08:05.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:05 smithi167 conmon[60316]: debug 2022-01-31T22:08:05.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:05.623473+0000) 2022-01-31T22:08:06.113 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:05 smithi167 conmon[49112]: debug 2022-01-31T22:08:05.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:05.957607+0000) 2022-01-31T22:08:06.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:06 smithi171 conmon[46715]: debug 2022-01-31T22:08:06.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:06.124455+0000) 2022-01-31T22:08:06.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:05 smithi171 conmon[51620]: debug 2022-01-31T22:08:05.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:05.976812+0000) 2022-01-31T22:08:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:06 smithi171 conmon[41853]: debug 2022-01-31T22:08:06.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:06.330795+0000) 2022-01-31T22:08:06.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:06 smithi167 conmon[54076]: debug 2022-01-31T22:08:06.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:06.623676+0000) 2022-01-31T22:08:06.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:06 smithi167 conmon[60316]: debug 2022-01-31T22:08:06.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:06.623672+0000) 2022-01-31T22:08:07.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:06 smithi167 conmon[49112]: debug 2022-01-31T22:08:06.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:06.957763+0000) 2022-01-31T22:08:07.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:07 smithi171 conmon[46715]: debug 2022-01-31T22:08:07.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:07.124608+0000) 2022-01-31T22:08:07.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:06 smithi171 conmon[51620]: debug 2022-01-31T22:08:06.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:06.977000+0000) 2022-01-31T22:08:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:07 smithi171 conmon[41853]: debug 2022-01-31T22:08:07.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:07.330969+0000) 2022-01-31T22:08:07.716 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:07 smithi167 conmon[54076]: debug 2022-01-31T22:08:07.623+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:07.623858+0000) 2022-01-31T22:08:07.716 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:07 smithi167 conmon[60316]: debug 2022-01-31T22:08:07.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:07.623863+0000) 2022-01-31T22:08:08.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:07 smithi167 conmon[49112]: debug 2022-01-31T22:08:07.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:07.957915+0000) 2022-01-31T22:08:08.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:08 smithi171 conmon[46715]: debug 2022-01-31T22:08:08.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:08.124785+0000) 2022-01-31T22:08:08.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:07 smithi171 conmon[51620]: debug 2022-01-31T22:08:07.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:07.977188+0000) 2022-01-31T22:08:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:08 smithi171 conmon[41853]: debug 2022-01-31T22:08:08.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:08.331156+0000) 2022-01-31T22:08:08.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:08 smithi167 conmon[54076]: debug 2022-01-31T22:08:08.623+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:08.624043+0000) 2022-01-31T22:08:08.826 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:08 smithi167 conmon[60316]: debug 2022-01-31T22:08:08.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:08.624000+0000) 2022-01-31T22:08:09.066 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:08 smithi171 conmon[35325]: debug 2022-01-31T22:08:08.801+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:08:09.068 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:08 smithi171 conmon[51620]: debug 2022-01-31T22:08:08.975+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:08.977392+0000) 2022-01-31T22:08:09.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:08 smithi167 conmon[49112]: debug 2022-01-31T22:08:08.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:08.958097+0000) 2022-01-31T22:08:09.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[46715]: debug 2022-01-31T22:08:09.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.124959+0000) 2022-01-31T22:08:09.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[41853]: debug 2022-01-31T22:08:09.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.331304+0000) 2022-01-31T22:08:09.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:09 smithi167 conmon[49112]: debug 2022-01-31T22:08:09.592+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.594373+0000) 2022-01-31T22:08:09.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:09 smithi167 conmon[54076]: debug 2022-01-31T22:08:09.593+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.594953+0000) 2022-01-31T22:08:09.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:09 smithi167 conmon[54076]: debug 2022-01-31T22:08:09.622+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.624148+0000) 2022-01-31T22:08:09.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:09 smithi167 conmon[60316]: debug 2022-01-31T22:08:09.592+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.593901+0000) 2022-01-31T22:08:09.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:09 smithi167 conmon[60316]: debug 2022-01-31T22:08:09.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.624153+0000) 2022-01-31T22:08:09.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[35325]: debug 2022-01-31T22:08:09.611+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185260 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:09.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[41853]: debug 2022-01-31T22:08:09.593+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.595426+0000) 2022-01-31T22:08:09.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[46715]: debug 2022-01-31T22:08:09.593+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.595395+0000) 2022-01-31T22:08:09.860 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[51620]: debug 2022-01-31T22:08:09.592+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.594736+0000) 2022-01-31T22:08:10.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:09 smithi167 conmon[49112]: debug 2022-01-31T22:08:09.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.958278+0000) 2022-01-31T22:08:10.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:10 smithi171 conmon[46715]: debug 2022-01-31T22:08:10.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:10.125124+0000) 2022-01-31T22:08:10.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:09 smithi171 conmon[51620]: debug 2022-01-31T22:08:09.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:09.977598+0000) 2022-01-31T22:08:10.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:10 smithi171 conmon[41853]: debug 2022-01-31T22:08:10.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:10.331477+0000) 2022-01-31T22:08:10.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:10 smithi167 conmon[54076]: debug 2022-01-31T22:08:10.623+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:10.624316+0000) 2022-01-31T22:08:10.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:10 smithi167 conmon[60316]: debug 2022-01-31T22:08:10.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:10.624317+0000) 2022-01-31T22:08:11.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:10 smithi167 conmon[49112]: debug 2022-01-31T22:08:10.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:10.958401+0000) 2022-01-31T22:08:11.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:11 smithi171 conmon[46715]: debug 2022-01-31T22:08:11.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:11.125260+0000) 2022-01-31T22:08:11.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:10 smithi171 conmon[51620]: debug 2022-01-31T22:08:10.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:10.977734+0000) 2022-01-31T22:08:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:11 smithi171 conmon[41853]: debug 2022-01-31T22:08:11.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:11.331628+0000) 2022-01-31T22:08:11.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:11 smithi167 conmon[54076]: debug 2022-01-31T22:08:11.623+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:11.624517+0000) 2022-01-31T22:08:11.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:11 smithi167 conmon[60316]: debug 2022-01-31T22:08:11.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:11.624475+0000) 2022-01-31T22:08:12.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:11 smithi167 conmon[49112]: debug 2022-01-31T22:08:11.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:11.958562+0000) 2022-01-31T22:08:12.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:12 smithi171 conmon[46715]: debug 2022-01-31T22:08:12.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:12.125358+0000) 2022-01-31T22:08:12.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:11 smithi171 conmon[51620]: debug 2022-01-31T22:08:11.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:11.977893+0000) 2022-01-31T22:08:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:12 smithi171 conmon[41853]: debug 2022-01-31T22:08:12.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:12.331767+0000) 2022-01-31T22:08:12.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:12 smithi167 conmon[54076]: debug 2022-01-31T22:08:12.624+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:12.624659+0000) 2022-01-31T22:08:12.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:12 smithi167 conmon[60316]: debug 2022-01-31T22:08:12.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:12.624663+0000) 2022-01-31T22:08:13.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:12 smithi167 conmon[49112]: debug 2022-01-31T22:08:12.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:12.958766+0000) 2022-01-31T22:08:13.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:13 smithi171 conmon[46715]: debug 2022-01-31T22:08:13.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:13.125537+0000) 2022-01-31T22:08:13.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:12 smithi171 conmon[51620]: debug 2022-01-31T22:08:12.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:12.978086+0000) 2022-01-31T22:08:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:13 smithi171 conmon[41853]: debug 2022-01-31T22:08:13.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:13.331957+0000) 2022-01-31T22:08:13.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:13 smithi167 conmon[54076]: debug 2022-01-31T22:08:13.624+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:13.624843+0000) 2022-01-31T22:08:13.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:13 smithi167 conmon[60316]: debug 2022-01-31T22:08:13.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:13.624840+0000) 2022-01-31T22:08:14.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:13 smithi167 conmon[49112]: debug 2022-01-31T22:08:13.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:13.958896+0000) 2022-01-31T22:08:14.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[46715]: debug 2022-01-31T22:08:14.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.125741+0000) 2022-01-31T22:08:14.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:13 smithi171 conmon[51620]: debug 2022-01-31T22:08:13.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:13.978272+0000) 2022-01-31T22:08:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[41853]: debug 2022-01-31T22:08:14.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.332167+0000) 2022-01-31T22:08:14.827 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:14 smithi167 conmon[49112]: debug 2022-01-31T22:08:14.616+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.617030+0000) 2022-01-31T22:08:14.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:14 smithi167 conmon[60316]: debug 2022-01-31T22:08:14.615+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.616340+0000) 2022-01-31T22:08:14.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:14 smithi167 conmon[60316]: debug 2022-01-31T22:08:14.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.625028+0000) 2022-01-31T22:08:14.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:14 smithi167 conmon[54076]: debug 2022-01-31T22:08:14.614+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.615458+0000) 2022-01-31T22:08:14.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:14 smithi167 conmon[54076]: debug 2022-01-31T22:08:14.624+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.625003+0000) 2022-01-31T22:08:14.978 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[35325]: debug 2022-01-31T22:08:14.633+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185373 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:14.979 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[41853]: debug 2022-01-31T22:08:14.615+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.617245+0000) 2022-01-31T22:08:14.980 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[46715]: debug 2022-01-31T22:08:14.615+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.617018+0000) 2022-01-31T22:08:14.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[51620]: debug 2022-01-31T22:08:14.614+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.616600+0000) 2022-01-31T22:08:15.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:14 smithi167 conmon[49112]: debug 2022-01-31T22:08:14.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.959094+0000) 2022-01-31T22:08:15.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:15 smithi171 conmon[46715]: debug 2022-01-31T22:08:15.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:15.125939+0000) 2022-01-31T22:08:15.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:14 smithi171 conmon[51620]: debug 2022-01-31T22:08:14.976+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:14.978471+0000) 2022-01-31T22:08:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:15 smithi171 conmon[41853]: debug 2022-01-31T22:08:15.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:15.332349+0000) 2022-01-31T22:08:15.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:15 smithi167 conmon[54076]: debug 2022-01-31T22:08:15.624+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:15.625157+0000) 2022-01-31T22:08:15.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:15 smithi167 conmon[60316]: debug 2022-01-31T22:08:15.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:15.625176+0000) 2022-01-31T22:08:16.102 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:15 smithi167 conmon[49112]: debug 2022-01-31T22:08:15.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:15.959266+0000) 2022-01-31T22:08:16.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:16 smithi171 conmon[46715]: debug 2022-01-31T22:08:16.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:16.126087+0000) 2022-01-31T22:08:16.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:15 smithi171 conmon[51620]: debug 2022-01-31T22:08:15.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:15.978626+0000) 2022-01-31T22:08:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:16 smithi171 conmon[41853]: debug 2022-01-31T22:08:16.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:16.332500+0000) 2022-01-31T22:08:16.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:16 smithi167 conmon[54076]: debug 2022-01-31T22:08:16.624+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:16.625398+0000) 2022-01-31T22:08:16.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:16 smithi167 conmon[60316]: debug 2022-01-31T22:08:16.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:16.625397+0000) 2022-01-31T22:08:17.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:16 smithi167 conmon[49112]: debug 2022-01-31T22:08:16.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:16.959480+0000) 2022-01-31T22:08:17.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:17 smithi171 conmon[46715]: debug 2022-01-31T22:08:17.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:17.126280+0000) 2022-01-31T22:08:17.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:16 smithi171 conmon[51620]: debug 2022-01-31T22:08:16.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:16.978811+0000) 2022-01-31T22:08:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:17 smithi171 conmon[41853]: debug 2022-01-31T22:08:17.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:17.332655+0000) 2022-01-31T22:08:17.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:17 smithi167 conmon[54076]: debug 2022-01-31T22:08:17.625+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:17.625630+0000) 2022-01-31T22:08:17.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:17 smithi167 conmon[60316]: debug 2022-01-31T22:08:17.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:17.625636+0000) 2022-01-31T22:08:18.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:17 smithi167 conmon[49112]: debug 2022-01-31T22:08:17.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:17.959689+0000) 2022-01-31T22:08:18.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:18 smithi171 conmon[46715]: debug 2022-01-31T22:08:18.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:18.126440+0000) 2022-01-31T22:08:18.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:17 smithi171 conmon[51620]: debug 2022-01-31T22:08:17.977+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:17.978931+0000) 2022-01-31T22:08:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:18 smithi171 conmon[41853]: debug 2022-01-31T22:08:18.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:18.332821+0000) 2022-01-31T22:08:18.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:18 smithi167 conmon[54076]: debug 2022-01-31T22:08:18.625+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:18.625846+0000) 2022-01-31T22:08:18.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:18 smithi167 conmon[60316]: debug 2022-01-31T22:08:18.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:18.625847+0000) 2022-01-31T22:08:19.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:18 smithi167 conmon[49112]: debug 2022-01-31T22:08:18.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:18.959855+0000) 2022-01-31T22:08:19.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[46715]: debug 2022-01-31T22:08:19.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.126630+0000) 2022-01-31T22:08:19.291 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:18 smithi171 conmon[51620]: debug 2022-01-31T22:08:18.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:18.979098+0000) 2022-01-31T22:08:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[41853]: debug 2022-01-31T22:08:19.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.333007+0000) 2022-01-31T22:08:19.828 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:19 smithi167 conmon[49112]: debug 2022-01-31T22:08:19.638+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.638699+0000) 2022-01-31T22:08:19.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:19 smithi167 conmon[54076]: debug 2022-01-31T22:08:19.625+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.626058+0000) 2022-01-31T22:08:19.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:19 smithi167 conmon[54076]: debug 2022-01-31T22:08:19.636+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.637126+0000) 2022-01-31T22:08:19.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:19 smithi167 conmon[60316]: debug 2022-01-31T22:08:19.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.626033+0000) 2022-01-31T22:08:19.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:19 smithi167 conmon[60316]: debug 2022-01-31T22:08:19.637+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.638150+0000) 2022-01-31T22:08:19.978 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[35325]: debug 2022-01-31T22:08:19.655+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185482 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:20.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[41853]: debug 2022-01-31T22:08:19.637+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.639054+0000) 2022-01-31T22:08:20.098 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[46715]: debug 2022-01-31T22:08:19.637+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.638917+0000) 2022-01-31T22:08:20.099 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[51620]: debug 2022-01-31T22:08:19.636+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.638519+0000) 2022-01-31T22:08:20.111 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:19 smithi167 conmon[49112]: debug 2022-01-31T22:08:19.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.960045+0000) 2022-01-31T22:08:20.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:20 smithi171 conmon[46715]: debug 2022-01-31T22:08:20.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:20.126823+0000) 2022-01-31T22:08:20.291 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:19 smithi171 conmon[51620]: debug 2022-01-31T22:08:19.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:19.979283+0000) 2022-01-31T22:08:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:20 smithi171 conmon[41853]: debug 2022-01-31T22:08:20.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:20.333135+0000) 2022-01-31T22:08:20.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:20 smithi167 conmon[54076]: debug 2022-01-31T22:08:20.625+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:20.626246+0000) 2022-01-31T22:08:20.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:20 smithi167 conmon[60316]: debug 2022-01-31T22:08:20.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:20.626254+0000) 2022-01-31T22:08:21.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:20 smithi167 conmon[49112]: debug 2022-01-31T22:08:20.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:20.960159+0000) 2022-01-31T22:08:21.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:21 smithi171 conmon[46715]: debug 2022-01-31T22:08:21.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:21.126982+0000) 2022-01-31T22:08:21.291 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:20 smithi171 conmon[51620]: debug 2022-01-31T22:08:20.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:20.979436+0000) 2022-01-31T22:08:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:21 smithi171 conmon[41853]: debug 2022-01-31T22:08:21.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:21.333297+0000) 2022-01-31T22:08:21.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:21 smithi167 conmon[54076]: debug 2022-01-31T22:08:21.626+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:21.626427+0000) 2022-01-31T22:08:21.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:21 smithi167 conmon[60316]: debug 2022-01-31T22:08:21.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:21.626431+0000) 2022-01-31T22:08:22.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:21 smithi167 conmon[49112]: debug 2022-01-31T22:08:21.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:21.960362+0000) 2022-01-31T22:08:22.291 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:22 smithi171 conmon[46715]: debug 2022-01-31T22:08:22.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:22.127109+0000) 2022-01-31T22:08:22.291 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:21 smithi171 conmon[51620]: debug 2022-01-31T22:08:21.978+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:21.979625+0000) 2022-01-31T22:08:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:22 smithi171 conmon[41853]: debug 2022-01-31T22:08:22.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:22.333489+0000) 2022-01-31T22:08:22.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:22 smithi167 conmon[54076]: debug 2022-01-31T22:08:22.626+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:22.626643+0000) 2022-01-31T22:08:22.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:22 smithi167 conmon[60316]: debug 2022-01-31T22:08:22.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:22.626575+0000) 2022-01-31T22:08:23.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:22 smithi167 conmon[49112]: debug 2022-01-31T22:08:22.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:22.960581+0000) 2022-01-31T22:08:23.291 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:23 smithi171 conmon[46715]: debug 2022-01-31T22:08:23.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:23.127241+0000) 2022-01-31T22:08:23.292 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:22 smithi171 conmon[51620]: debug 2022-01-31T22:08:22.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:22.979872+0000) 2022-01-31T22:08:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:23 smithi171 conmon[41853]: debug 2022-01-31T22:08:23.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:23.333690+0000) 2022-01-31T22:08:23.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:23 smithi167 conmon[54076]: debug 2022-01-31T22:08:23.626+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:23.626860+0000) 2022-01-31T22:08:23.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:23 smithi167 conmon[60316]: debug 2022-01-31T22:08:23.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:23.626766+0000) 2022-01-31T22:08:24.069 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:23 smithi171 conmon[51620]: debug 2022-01-31T22:08:23.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:23.980041+0000) 2022-01-31T22:08:24.070 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:23 smithi171 conmon[35325]: debug 2022-01-31T22:08:23.802+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:08:24.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:23 smithi167 conmon[49112]: debug 2022-01-31T22:08:23.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:23.960741+0000) 2022-01-31T22:08:24.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[46715]: debug 2022-01-31T22:08:24.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.127441+0000) 2022-01-31T22:08:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[41853]: debug 2022-01-31T22:08:24.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.333918+0000) 2022-01-31T22:08:24.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:24 smithi167 conmon[49112]: debug 2022-01-31T22:08:24.660+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.660844+0000) 2022-01-31T22:08:24.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:24 smithi167 conmon[54076]: debug 2022-01-31T22:08:24.626+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.627007+0000) 2022-01-31T22:08:24.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:24 smithi167 conmon[54076]: debug 2022-01-31T22:08:24.660+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.660715+0000) 2022-01-31T22:08:24.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:24 smithi167 conmon[60316]: debug 2022-01-31T22:08:24.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.626972+0000) 2022-01-31T22:08:24.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:24 smithi167 conmon[60316]: debug 2022-01-31T22:08:24.659+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.660093+0000) 2022-01-31T22:08:24.979 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[35325]: debug 2022-01-31T22:08:24.678+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185597 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:24.980 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[41853]: debug 2022-01-31T22:08:24.659+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.659964+0000) 2022-01-31T22:08:24.981 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[46715]: debug 2022-01-31T22:08:24.660+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.661088+0000) 2022-01-31T22:08:24.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[51620]: debug 2022-01-31T22:08:24.659+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.660166+0000) 2022-01-31T22:08:25.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:24 smithi167 conmon[49112]: debug 2022-01-31T22:08:24.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.960934+0000) 2022-01-31T22:08:25.291 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:25 smithi171 conmon[46715]: debug 2022-01-31T22:08:25.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:25.127600+0000) 2022-01-31T22:08:25.292 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:24 smithi171 conmon[51620]: debug 2022-01-31T22:08:24.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:24.980197+0000) 2022-01-31T22:08:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:25 smithi171 conmon[41853]: debug 2022-01-31T22:08:25.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:25.334118+0000) 2022-01-31T22:08:25.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:25 smithi167 conmon[54076]: debug 2022-01-31T22:08:25.626+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:25.627129+0000) 2022-01-31T22:08:25.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:25 smithi167 conmon[60316]: debug 2022-01-31T22:08:25.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:25.627040+0000) 2022-01-31T22:08:26.116 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:25 smithi167 conmon[49112]: debug 2022-01-31T22:08:25.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:25.961110+0000) 2022-01-31T22:08:26.291 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:26 smithi171 conmon[46715]: debug 2022-01-31T22:08:26.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:26.127743+0000) 2022-01-31T22:08:26.292 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:25 smithi171 conmon[51620]: debug 2022-01-31T22:08:25.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:25.980344+0000) 2022-01-31T22:08:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:26 smithi171 conmon[41853]: debug 2022-01-31T22:08:26.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:26.334248+0000) 2022-01-31T22:08:26.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:26 smithi167 conmon[54076]: debug 2022-01-31T22:08:26.626+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:26.627268+0000) 2022-01-31T22:08:26.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:26 smithi167 conmon[60316]: debug 2022-01-31T22:08:26.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:26.627198+0000) 2022-01-31T22:08:27.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:26 smithi167 conmon[49112]: debug 2022-01-31T22:08:26.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:26.961235+0000) 2022-01-31T22:08:27.292 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:27 smithi171 conmon[46715]: debug 2022-01-31T22:08:27.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:27.127979+0000) 2022-01-31T22:08:27.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:26 smithi171 conmon[51620]: debug 2022-01-31T22:08:26.979+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:26.980505+0000) 2022-01-31T22:08:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:27 smithi171 conmon[41853]: debug 2022-01-31T22:08:27.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:27.334402+0000) 2022-01-31T22:08:27.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:27 smithi167 conmon[54076]: debug 2022-01-31T22:08:27.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:27.627524+0000) 2022-01-31T22:08:27.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:27 smithi167 conmon[60316]: debug 2022-01-31T22:08:27.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:27.627407+0000) 2022-01-31T22:08:28.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:27 smithi167 conmon[49112]: debug 2022-01-31T22:08:27.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:27.961426+0000) 2022-01-31T22:08:28.292 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:28 smithi171 conmon[46715]: debug 2022-01-31T22:08:28.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:28.128172+0000) 2022-01-31T22:08:28.292 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:27 smithi171 conmon[51620]: debug 2022-01-31T22:08:27.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:27.980683+0000) 2022-01-31T22:08:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:28 smithi171 conmon[41853]: debug 2022-01-31T22:08:28.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:28.334590+0000) 2022-01-31T22:08:28.829 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:28 smithi167 conmon[54076]: debug 2022-01-31T22:08:28.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:28.627600+0000) 2022-01-31T22:08:28.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:28 smithi167 conmon[60316]: debug 2022-01-31T22:08:28.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:28.627705+0000) 2022-01-31T22:08:29.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:28 smithi167 conmon[49112]: debug 2022-01-31T22:08:28.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:28.961612+0000) 2022-01-31T22:08:29.292 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[46715]: debug 2022-01-31T22:08:29.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.128373+0000) 2022-01-31T22:08:29.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:28 smithi171 conmon[51620]: debug 2022-01-31T22:08:28.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:28.980858+0000) 2022-01-31T22:08:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[41853]: debug 2022-01-31T22:08:29.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.334803+0000) 2022-01-31T22:08:29.680 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:29 smithi167 conmon[54076]: debug 2022-01-31T22:08:29.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.627791+0000) 2022-01-31T22:08:29.681 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:29 smithi167 conmon[60316]: debug 2022-01-31T22:08:29.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.627851+0000) 2022-01-31T22:08:29.961 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:29 smithi167 conmon[49112]: debug 2022-01-31T22:08:29.682+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.682577+0000) 2022-01-31T22:08:29.962 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:29 smithi167 conmon[54076]: debug 2022-01-31T22:08:29.682+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.682568+0000) 2022-01-31T22:08:29.962 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:29 smithi167 conmon[60316]: debug 2022-01-31T22:08:29.681+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.681992+0000) 2022-01-31T22:08:29.980 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[35325]: debug 2022-01-31T22:08:29.700+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185706 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:29.980 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[46715]: debug 2022-01-31T22:08:29.682+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.682954+0000) 2022-01-31T22:08:29.981 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[41853]: debug 2022-01-31T22:08:29.680+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.681243+0000) 2022-01-31T22:08:29.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[51620]: debug 2022-01-31T22:08:29.680+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.681159+0000) 2022-01-31T22:08:30.292 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:30 smithi171 conmon[46715]: debug 2022-01-31T22:08:30.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:30.128555+0000) 2022-01-31T22:08:30.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:29 smithi171 conmon[51620]: debug 2022-01-31T22:08:29.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.981056+0000) 2022-01-31T22:08:30.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:29 smithi167 conmon[49112]: debug 2022-01-31T22:08:29.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:29.961804+0000) 2022-01-31T22:08:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:30 smithi171 conmon[41853]: debug 2022-01-31T22:08:30.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:30.334965+0000) 2022-01-31T22:08:30.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:30 smithi167 conmon[54076]: debug 2022-01-31T22:08:30.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:30.627995+0000) 2022-01-31T22:08:30.830 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:30 smithi167 conmon[60316]: debug 2022-01-31T22:08:30.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:30.628027+0000) 2022-01-31T22:08:31.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:30 smithi167 conmon[49112]: debug 2022-01-31T22:08:30.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:30.961969+0000) 2022-01-31T22:08:31.292 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:31 smithi171 conmon[46715]: debug 2022-01-31T22:08:31.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:31.128671+0000) 2022-01-31T22:08:31.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:30 smithi171 conmon[51620]: debug 2022-01-31T22:08:30.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:30.981220+0000) 2022-01-31T22:08:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:31 smithi171 conmon[41853]: debug 2022-01-31T22:08:31.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:31.335099+0000) 2022-01-31T22:08:31.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:31 smithi167 conmon[54076]: debug 2022-01-31T22:08:31.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:31.628135+0000) 2022-01-31T22:08:31.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:31 smithi167 conmon[60316]: debug 2022-01-31T22:08:31.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:31.628213+0000) 2022-01-31T22:08:32.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:31 smithi167 conmon[49112]: debug 2022-01-31T22:08:31.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:31.962168+0000) 2022-01-31T22:08:32.292 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:32 smithi171 conmon[46715]: debug 2022-01-31T22:08:32.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:32.128845+0000) 2022-01-31T22:08:32.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:31 smithi171 conmon[51620]: debug 2022-01-31T22:08:31.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:31.981409+0000) 2022-01-31T22:08:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:32 smithi171 conmon[41853]: debug 2022-01-31T22:08:32.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:32.335280+0000) 2022-01-31T22:08:32.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:32 smithi167 conmon[54076]: debug 2022-01-31T22:08:32.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:32.628327+0000) 2022-01-31T22:08:32.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:32 smithi167 conmon[60316]: debug 2022-01-31T22:08:32.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:32.628439+0000) 2022-01-31T22:08:33.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:32 smithi167 conmon[49112]: debug 2022-01-31T22:08:32.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:32.962376+0000) 2022-01-31T22:08:33.293 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:33 smithi171 conmon[46715]: debug 2022-01-31T22:08:33.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:33.129020+0000) 2022-01-31T22:08:33.294 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:32 smithi171 conmon[51620]: debug 2022-01-31T22:08:32.980+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:32.981607+0000) 2022-01-31T22:08:33.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:33 smithi171 conmon[41853]: debug 2022-01-31T22:08:33.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:33.335473+0000) 2022-01-31T22:08:33.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:33 smithi167 conmon[54076]: debug 2022-01-31T22:08:33.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:33.628543+0000) 2022-01-31T22:08:33.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:33 smithi167 conmon[60316]: debug 2022-01-31T22:08:33.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:33.628547+0000) 2022-01-31T22:08:34.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:33 smithi167 conmon[49112]: debug 2022-01-31T22:08:33.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:33.962600+0000) 2022-01-31T22:08:34.293 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[46715]: debug 2022-01-31T22:08:34.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.129268+0000) 2022-01-31T22:08:34.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:33 smithi171 conmon[51620]: debug 2022-01-31T22:08:33.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:33.981837+0000) 2022-01-31T22:08:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[41853]: debug 2022-01-31T22:08:34.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.335700+0000) 2022-01-31T22:08:34.702 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:34 smithi167 conmon[54076]: debug 2022-01-31T22:08:34.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.628750+0000) 2022-01-31T22:08:34.703 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:34 smithi167 conmon[60316]: debug 2022-01-31T22:08:34.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.628750+0000) 2022-01-31T22:08:34.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:34 smithi167 conmon[49112]: debug 2022-01-31T22:08:34.703+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.703616+0000) 2022-01-31T22:08:34.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:34 smithi167 conmon[54076]: debug 2022-01-31T22:08:34.703+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.704201+0000) 2022-01-31T22:08:34.963 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:34 smithi167 conmon[60316]: debug 2022-01-31T22:08:34.703+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.703582+0000) 2022-01-31T22:08:34.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[35325]: debug 2022-01-31T22:08:34.722+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185819 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:34.982 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[41853]: debug 2022-01-31T22:08:34.703+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.703814+0000) 2022-01-31T22:08:34.982 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[46715]: debug 2022-01-31T22:08:34.703+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.704656+0000) 2022-01-31T22:08:34.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[51620]: debug 2022-01-31T22:08:34.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[51620]: 2022-01-31T22:08:34.702+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.702848+0000) 2022-01-31T22:08:35.293 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:35 smithi171 conmon[46715]: debug 2022-01-31T22:08:35.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:35.129451+0000) 2022-01-31T22:08:35.293 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:34 smithi171 conmon[51620]: debug 2022-01-31T22:08:34.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.981932+0000) 2022-01-31T22:08:35.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:34 smithi167 conmon[49112]: debug 2022-01-31T22:08:34.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:34.962728+0000) 2022-01-31T22:08:35.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:35 smithi171 conmon[41853]: debug 2022-01-31T22:08:35.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:35.335876+0000) 2022-01-31T22:08:35.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:35 smithi167 conmon[54076]: debug 2022-01-31T22:08:35.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:35.628847+0000) 2022-01-31T22:08:35.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:35 smithi167 conmon[60316]: debug 2022-01-31T22:08:35.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:35.628847+0000) 2022-01-31T22:08:36.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:35 smithi167 conmon[49112]: debug 2022-01-31T22:08:35.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:35.962896+0000) 2022-01-31T22:08:36.293 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:36 smithi171 conmon[46715]: debug 2022-01-31T22:08:36.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:36.129594+0000) 2022-01-31T22:08:36.294 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:35 smithi171 conmon[51620]: debug 2022-01-31T22:08:35.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:35.982076+0000) 2022-01-31T22:08:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:36 smithi171 conmon[41853]: debug 2022-01-31T22:08:36.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:36.336012+0000) 2022-01-31T22:08:36.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:36 smithi167 conmon[54076]: debug 2022-01-31T22:08:36.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:36.629034+0000) 2022-01-31T22:08:36.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:36 smithi167 conmon[60316]: debug 2022-01-31T22:08:36.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:36.629034+0000) 2022-01-31T22:08:37.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:36 smithi167 conmon[49112]: debug 2022-01-31T22:08:36.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:36.963097+0000) 2022-01-31T22:08:37.293 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:37 smithi171 conmon[46715]: debug 2022-01-31T22:08:37.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:37.129783+0000) 2022-01-31T22:08:37.294 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:36 smithi171 conmon[51620]: debug 2022-01-31T22:08:36.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:36.982273+0000) 2022-01-31T22:08:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:37 smithi171 conmon[41853]: debug 2022-01-31T22:08:37.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:37.336204+0000) 2022-01-31T22:08:37.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:37 smithi167 conmon[54076]: debug 2022-01-31T22:08:37.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:37.629272+0000) 2022-01-31T22:08:37.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:37 smithi167 conmon[60316]: debug 2022-01-31T22:08:37.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:37.629272+0000) 2022-01-31T22:08:38.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:37 smithi167 conmon[49112]: debug 2022-01-31T22:08:37.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:37.963241+0000) 2022-01-31T22:08:38.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:38 smithi171 conmon[46715]: debug 2022-01-31T22:08:38.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:38.129935+0000) 2022-01-31T22:08:38.294 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:37 smithi171 conmon[51620]: debug 2022-01-31T22:08:37.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:37.982456+0000) 2022-01-31T22:08:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:38 smithi171 conmon[41853]: debug 2022-01-31T22:08:38.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:38.336361+0000) 2022-01-31T22:08:38.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:38 smithi167 conmon[54076]: debug 2022-01-31T22:08:38.627+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:38.629431+0000) 2022-01-31T22:08:38.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:38 smithi167 conmon[60316]: debug 2022-01-31T22:08:38.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:38.629431+0000) 2022-01-31T22:08:39.073 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:38 smithi171 conmon[35325]: debug 2022-01-31T22:08:38.803+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:08:39.074 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:38 smithi171 conmon[51620]: debug 2022-01-31T22:08:38.981+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:38.982648+0000) 2022-01-31T22:08:39.118 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:38 smithi167 conmon[49112]: debug 2022-01-31T22:08:38.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:38.963474+0000) 2022-01-31T22:08:39.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[46715]: debug 2022-01-31T22:08:39.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.130097+0000) 2022-01-31T22:08:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[41853]: debug 2022-01-31T22:08:39.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.336475+0000) 2022-01-31T22:08:39.725 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[60316]: debug 2022-01-31T22:08:39.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.629613+0000) 2022-01-31T22:08:39.726 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[54076]: debug 2022-01-31T22:08:39.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.629614+0000) 2022-01-31T22:08:39.982 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[35325]: debug 2022-01-31T22:08:39.750+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 185928 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:39.983 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[41853]: debug 2022-01-31T22:08:39.724+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.725626+0000) 2022-01-31T22:08:39.983 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[46715]: debug 2022-01-31T22:08:39.725+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.725790+0000) 2022-01-31T22:08:39.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[51620]: debug 2022-01-31T22:08:39.724+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.725472+0000) 2022-01-31T22:08:40.119 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[54076]: debug 2022-01-31T22:08:39.725+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.726942+0000) 2022-01-31T22:08:40.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[60316]: debug 2022-01-31T22:08:39.725+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.726571+0000) 2022-01-31T22:08:40.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[49112]: debug 2022-01-31T22:08:40.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[49112]: 2022-01-31T22:08:39.732+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.733775+0000) 2022-01-31T22:08:40.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:39 smithi167 conmon[49112]: debug 2022-01-31T22:08:39.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.963671+0000) 2022-01-31T22:08:40.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:40 smithi171 conmon[46715]: debug 2022-01-31T22:08:40.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:40.130276+0000) 2022-01-31T22:08:40.294 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:39 smithi171 conmon[51620]: debug 2022-01-31T22:08:39.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:39.982838+0000) 2022-01-31T22:08:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:40 smithi171 conmon[41853]: debug 2022-01-31T22:08:40.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:40.336660+0000) 2022-01-31T22:08:40.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:40 smithi167 conmon[54076]: debug 2022-01-31T22:08:40.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:40.629790+0000) 2022-01-31T22:08:40.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:40 smithi167 conmon[60316]: debug 2022-01-31T22:08:40.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:40.629790+0000) 2022-01-31T22:08:41.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:40 smithi167 conmon[49112]: debug 2022-01-31T22:08:40.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:40.963820+0000) 2022-01-31T22:08:41.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:41 smithi171 conmon[46715]: debug 2022-01-31T22:08:41.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:41.130422+0000) 2022-01-31T22:08:41.295 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:40 smithi171 conmon[51620]: debug 2022-01-31T22:08:40.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:40.982968+0000) 2022-01-31T22:08:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:41 smithi171 conmon[41853]: debug 2022-01-31T22:08:41.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:41.336927+0000) 2022-01-31T22:08:41.831 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:41 smithi167 conmon[54076]: debug 2022-01-31T22:08:41.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:41.629936+0000) 2022-01-31T22:08:41.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:41 smithi167 conmon[60316]: debug 2022-01-31T22:08:41.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:41.629936+0000) 2022-01-31T22:08:42.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:41 smithi167 conmon[49112]: debug 2022-01-31T22:08:41.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:41.963985+0000) 2022-01-31T22:08:42.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:42 smithi171 conmon[46715]: debug 2022-01-31T22:08:42.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:42.130627+0000) 2022-01-31T22:08:42.295 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:41 smithi171 conmon[51620]: debug 2022-01-31T22:08:41.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:41.983111+0000) 2022-01-31T22:08:42.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:42 smithi171 conmon[41853]: debug 2022-01-31T22:08:42.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:42.337090+0000) 2022-01-31T22:08:42.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:42 smithi167 conmon[60316]: debug 2022-01-31T22:08:42.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:42.630147+0000) 2022-01-31T22:08:42.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:42 smithi167 conmon[54076]: debug 2022-01-31T22:08:42.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:42.630146+0000) 2022-01-31T22:08:43.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:42 smithi167 conmon[49112]: debug 2022-01-31T22:08:42.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:42.964155+0000) 2022-01-31T22:08:43.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:43 smithi171 conmon[46715]: debug 2022-01-31T22:08:43.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:43.130824+0000) 2022-01-31T22:08:43.295 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:42 smithi171 conmon[51620]: debug 2022-01-31T22:08:42.983+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:42.983300+0000) 2022-01-31T22:08:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:43 smithi171 conmon[41853]: debug 2022-01-31T22:08:43.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:43.337292+0000) 2022-01-31T22:08:43.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:43 smithi167 conmon[54076]: debug 2022-01-31T22:08:43.628+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:43.630351+0000) 2022-01-31T22:08:43.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:43 smithi167 conmon[60316]: debug 2022-01-31T22:08:43.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:43.630351+0000) 2022-01-31T22:08:44.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:43 smithi167 conmon[49112]: debug 2022-01-31T22:08:43.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:43.964332+0000) 2022-01-31T22:08:44.294 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[46715]: debug 2022-01-31T22:08:44.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.131001+0000) 2022-01-31T22:08:44.295 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:43 smithi171 conmon[51620]: debug 2022-01-31T22:08:43.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:43.983530+0000) 2022-01-31T22:08:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[41853]: debug 2022-01-31T22:08:44.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.337490+0000) 2022-01-31T22:08:44.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:44 smithi167 conmon[54076]: debug 2022-01-31T22:08:44.629+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.630531+0000) 2022-01-31T22:08:44.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:44 smithi167 conmon[60316]: debug 2022-01-31T22:08:44.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.630532+0000) 2022-01-31T22:08:45.073 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[35325]: debug 2022-01-31T22:08:44.772+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186041 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:45.074 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[41853]: debug 2022-01-31T22:08:44.752+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.753118+0000) 2022-01-31T22:08:45.075 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[46715]: debug 2022-01-31T22:08:44.753+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.754219+0000) 2022-01-31T22:08:45.076 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[51620]: debug 2022-01-31T22:08:44.752+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.753384+0000) 2022-01-31T22:08:45.076 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:44 smithi171 conmon[51620]: debug 2022-01-31T22:08:44.982+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.983729+0000) 2022-01-31T22:08:45.120 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:44 smithi167 conmon[54076]: debug 2022-01-31T22:08:44.753+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.754753+0000) 2022-01-31T22:08:45.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:44 smithi167 conmon[60316]: debug 2022-01-31T22:08:44.753+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.754627+0000) 2022-01-31T22:08:45.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:44 smithi167 conmon[49112]: debug 2022-01-31T22:08:44.753+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.754490+0000) 2022-01-31T22:08:45.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:44 smithi167 conmon[49112]: debug 2022-01-31T22:08:44.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:44.964523+0000) 2022-01-31T22:08:45.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:45 smithi171 conmon[46715]: debug 2022-01-31T22:08:45.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:45.131166+0000) 2022-01-31T22:08:45.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:45 smithi171 conmon[41853]: debug 2022-01-31T22:08:45.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:45.337641+0000) 2022-01-31T22:08:45.832 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:45 smithi167 conmon[60316]: debug 2022-01-31T22:08:45.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:45.630669+0000) 2022-01-31T22:08:45.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:45 smithi167 conmon[54076]: debug 2022-01-31T22:08:45.629+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:45.630669+0000) 2022-01-31T22:08:46.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:45 smithi167 conmon[49112]: debug 2022-01-31T22:08:45.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:45.964666+0000) 2022-01-31T22:08:46.295 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:46 smithi171 conmon[46715]: debug 2022-01-31T22:08:46.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:46.131339+0000) 2022-01-31T22:08:46.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:45 smithi171 conmon[51620]: debug 2022-01-31T22:08:45.983+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:45.983906+0000) 2022-01-31T22:08:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:46 smithi171 conmon[41853]: debug 2022-01-31T22:08:46.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:46.337829+0000) 2022-01-31T22:08:46.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:46 smithi167 conmon[54076]: debug 2022-01-31T22:08:46.629+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:46.630777+0000) 2022-01-31T22:08:46.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:46 smithi167 conmon[60316]: debug 2022-01-31T22:08:46.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:46.630777+0000) 2022-01-31T22:08:47.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:46 smithi167 conmon[49112]: debug 2022-01-31T22:08:46.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:46.964888+0000) 2022-01-31T22:08:47.295 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:47 smithi171 conmon[46715]: debug 2022-01-31T22:08:47.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:47.131525+0000) 2022-01-31T22:08:47.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:46 smithi171 conmon[51620]: debug 2022-01-31T22:08:46.983+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:46.984090+0000) 2022-01-31T22:08:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:47 smithi171 conmon[41853]: debug 2022-01-31T22:08:47.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:47.337997+0000) 2022-01-31T22:08:47.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:47 smithi167 conmon[54076]: debug 2022-01-31T22:08:47.629+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:47.630988+0000) 2022-01-31T22:08:47.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:47 smithi167 conmon[60316]: debug 2022-01-31T22:08:47.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:47.630988+0000) 2022-01-31T22:08:48.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:47 smithi167 conmon[49112]: debug 2022-01-31T22:08:47.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:47.965067+0000) 2022-01-31T22:08:48.295 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:48 smithi171 conmon[46715]: debug 2022-01-31T22:08:48.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:48.131712+0000) 2022-01-31T22:08:48.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:47 smithi171 conmon[51620]: debug 2022-01-31T22:08:47.983+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:47.984285+0000) 2022-01-31T22:08:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:48 smithi171 conmon[41853]: debug 2022-01-31T22:08:48.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:48.338184+0000) 2022-01-31T22:08:48.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:48 smithi167 conmon[54076]: debug 2022-01-31T22:08:48.629+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:48.631211+0000) 2022-01-31T22:08:48.834 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:48 smithi167 conmon[60316]: debug 2022-01-31T22:08:48.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:48.631211+0000) 2022-01-31T22:08:49.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:48 smithi167 conmon[49112]: debug 2022-01-31T22:08:48.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:48.965246+0000) 2022-01-31T22:08:49.295 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[46715]: debug 2022-01-31T22:08:49.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.131946+0000) 2022-01-31T22:08:49.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:48 smithi171 conmon[51620]: debug 2022-01-31T22:08:48.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:48.984473+0000) 2022-01-31T22:08:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[41853]: debug 2022-01-31T22:08:49.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.338373+0000) 2022-01-31T22:08:49.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:49 smithi167 conmon[54076]: debug 2022-01-31T22:08:49.629+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.631344+0000) 2022-01-31T22:08:49.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:49 smithi167 conmon[60316]: debug 2022-01-31T22:08:49.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.631373+0000) 2022-01-31T22:08:50.075 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[35325]: debug 2022-01-31T22:08:49.794+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186150 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:50.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[41853]: debug 2022-01-31T22:08:49.774+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.775413+0000) 2022-01-31T22:08:50.077 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[46715]: debug 2022-01-31T22:08:49.776+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.776976+0000) 2022-01-31T22:08:50.077 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[51620]: debug 2022-01-31T22:08:49.774+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.775736+0000) 2022-01-31T22:08:50.078 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:49 smithi171 conmon[51620]: debug 2022-01-31T22:08:49.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.984686+0000) 2022-01-31T22:08:50.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:49 smithi167 conmon[49112]: debug 2022-01-31T22:08:49.774+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.776191+0000) 2022-01-31T22:08:50.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:49 smithi167 conmon[49112]: debug 2022-01-31T22:08:49.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.965447+0000) 2022-01-31T22:08:50.121 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:49 smithi167 conmon[54076]: debug 2022-01-31T22:08:49.775+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.777136+0000) 2022-01-31T22:08:50.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:49 smithi167 conmon[60316]: debug 2022-01-31T22:08:49.775+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:49.777101+0000) 2022-01-31T22:08:50.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:50 smithi171 conmon[46715]: debug 2022-01-31T22:08:50.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:50.132105+0000) 2022-01-31T22:08:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:50 smithi171 conmon[41853]: debug 2022-01-31T22:08:50.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:50.338535+0000) 2022-01-31T22:08:50.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:50 smithi167 conmon[60316]: debug 2022-01-31T22:08:50.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:50.631547+0000) 2022-01-31T22:08:50.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:50 smithi167 conmon[54076]: debug 2022-01-31T22:08:50.630+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:50.631543+0000) 2022-01-31T22:08:51.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:50 smithi167 conmon[49112]: debug 2022-01-31T22:08:50.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:50.965624+0000) 2022-01-31T22:08:51.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:51 smithi171 conmon[46715]: debug 2022-01-31T22:08:51.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:51.132242+0000) 2022-01-31T22:08:51.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:50 smithi171 conmon[51620]: debug 2022-01-31T22:08:50.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:50.984848+0000) 2022-01-31T22:08:51.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:51 smithi171 conmon[41853]: debug 2022-01-31T22:08:51.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:51.338686+0000) 2022-01-31T22:08:51.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:51 smithi167 conmon[60316]: debug 2022-01-31T22:08:51.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:51.631693+0000) 2022-01-31T22:08:51.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:51 smithi167 conmon[54076]: debug 2022-01-31T22:08:51.630+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:51.631674+0000) 2022-01-31T22:08:52.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:51 smithi167 conmon[49112]: debug 2022-01-31T22:08:51.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:51.965812+0000) 2022-01-31T22:08:52.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:52 smithi171 conmon[46715]: debug 2022-01-31T22:08:52.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:52.132375+0000) 2022-01-31T22:08:52.296 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:51 smithi171 conmon[51620]: debug 2022-01-31T22:08:51.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:51.984996+0000) 2022-01-31T22:08:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:52 smithi171 conmon[41853]: debug 2022-01-31T22:08:52.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:52.338872+0000) 2022-01-31T22:08:52.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:52 smithi167 conmon[54076]: debug 2022-01-31T22:08:52.631+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:52.631882+0000) 2022-01-31T22:08:52.834 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:52 smithi167 conmon[60316]: debug 2022-01-31T22:08:52.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:52.631880+0000) 2022-01-31T22:08:53.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:52 smithi167 conmon[49112]: debug 2022-01-31T22:08:52.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:52.966029+0000) 2022-01-31T22:08:53.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:53 smithi171 conmon[46715]: debug 2022-01-31T22:08:53.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:53.132577+0000) 2022-01-31T22:08:53.297 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:52 smithi171 conmon[51620]: debug 2022-01-31T22:08:52.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:52.985192+0000) 2022-01-31T22:08:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:53 smithi171 conmon[41853]: debug 2022-01-31T22:08:53.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:53.339105+0000) 2022-01-31T22:08:53.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:53 smithi167 conmon[54076]: debug 2022-01-31T22:08:53.631+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:53.632115+0000) 2022-01-31T22:08:53.834 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:53 smithi167 conmon[60316]: debug 2022-01-31T22:08:53.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:53.632114+0000) 2022-01-31T22:08:54.075 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:53 smithi171 conmon[35325]: debug 2022-01-31T22:08:53.804+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:08:54.076 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:53 smithi171 conmon[51620]: debug 2022-01-31T22:08:53.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:53.985410+0000) 2022-01-31T22:08:54.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:53 smithi167 conmon[49112]: debug 2022-01-31T22:08:53.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:53.966224+0000) 2022-01-31T22:08:54.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[46715]: debug 2022-01-31T22:08:54.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.132799+0000) 2022-01-31T22:08:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[41853]: debug 2022-01-31T22:08:54.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.339315+0000) 2022-01-31T22:08:54.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:54 smithi167 conmon[60316]: debug 2022-01-31T22:08:54.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.632307+0000) 2022-01-31T22:08:54.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:54 smithi167 conmon[54076]: debug 2022-01-31T22:08:54.630+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.632307+0000) 2022-01-31T22:08:55.076 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[35325]: debug 2022-01-31T22:08:54.816+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186263 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:08:55.076 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[41853]: debug 2022-01-31T22:08:54.796+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.797907+0000) 2022-01-31T22:08:55.077 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[46715]: debug 2022-01-31T22:08:54.796+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.798199+0000) 2022-01-31T22:08:55.078 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[51620]: debug 2022-01-31T22:08:54.797+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.799230+0000) 2022-01-31T22:08:55.078 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:54 smithi171 conmon[51620]: debug 2022-01-31T22:08:54.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.985625+0000) 2022-01-31T22:08:55.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:54 smithi167 conmon[54076]: debug 2022-01-31T22:08:54.797+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.799070+0000) 2022-01-31T22:08:55.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:54 smithi167 conmon[49112]: debug 2022-01-31T22:08:54.797+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.798906+0000) 2022-01-31T22:08:55.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:54 smithi167 conmon[49112]: debug 2022-01-31T22:08:54.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.966411+0000) 2022-01-31T22:08:55.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:54 smithi167 conmon[60316]: debug 2022-01-31T22:08:54.796+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:54.798204+0000) 2022-01-31T22:08:55.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:55 smithi171 conmon[46715]: debug 2022-01-31T22:08:55.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:55.133010+0000) 2022-01-31T22:08:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:55 smithi171 conmon[41853]: debug 2022-01-31T22:08:55.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:55.339486+0000) 2022-01-31T22:08:55.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:55 smithi167 conmon[54076]: debug 2022-01-31T22:08:55.631+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:55.632520+0000) 2022-01-31T22:08:55.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:55 smithi167 conmon[60316]: debug 2022-01-31T22:08:55.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:55.632520+0000) 2022-01-31T22:08:56.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:55 smithi167 conmon[49112]: debug 2022-01-31T22:08:55.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:55.966590+0000) 2022-01-31T22:08:56.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:56 smithi171 conmon[46715]: debug 2022-01-31T22:08:56.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:56.133136+0000) 2022-01-31T22:08:56.298 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:55 smithi171 conmon[51620]: debug 2022-01-31T22:08:55.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:55.985807+0000) 2022-01-31T22:08:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:56 smithi171 conmon[41853]: debug 2022-01-31T22:08:56.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:56.339624+0000) 2022-01-31T22:08:56.834 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:56 smithi167 conmon[60316]: debug 2022-01-31T22:08:56.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:56.632711+0000) 2022-01-31T22:08:56.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:56 smithi167 conmon[54076]: debug 2022-01-31T22:08:56.632+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:56.632711+0000) 2022-01-31T22:08:57.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:56 smithi167 conmon[49112]: debug 2022-01-31T22:08:56.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:56.966786+0000) 2022-01-31T22:08:57.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:57 smithi171 conmon[46715]: debug 2022-01-31T22:08:57.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:57.133304+0000) 2022-01-31T22:08:57.298 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:56 smithi171 conmon[51620]: debug 2022-01-31T22:08:56.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:56.985983+0000) 2022-01-31T22:08:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:57 smithi171 conmon[41853]: debug 2022-01-31T22:08:57.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:57.339791+0000) 2022-01-31T22:08:57.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:57 smithi167 conmon[54076]: debug 2022-01-31T22:08:57.632+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:57.632958+0000) 2022-01-31T22:08:57.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:57 smithi167 conmon[60316]: debug 2022-01-31T22:08:57.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:57.632958+0000) 2022-01-31T22:08:58.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:57 smithi167 conmon[49112]: debug 2022-01-31T22:08:57.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:57.966973+0000) 2022-01-31T22:08:58.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:58 smithi171 conmon[46715]: debug 2022-01-31T22:08:58.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:58.133480+0000) 2022-01-31T22:08:58.297 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:57 smithi171 conmon[51620]: debug 2022-01-31T22:08:57.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:57.986145+0000) 2022-01-31T22:08:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:58 smithi171 conmon[41853]: debug 2022-01-31T22:08:58.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:58.339985+0000) 2022-01-31T22:08:58.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:58 smithi167 conmon[54076]: debug 2022-01-31T22:08:58.632+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:58.633215+0000) 2022-01-31T22:08:58.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:58 smithi167 conmon[60316]: debug 2022-01-31T22:08:58.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:58.633215+0000) 2022-01-31T22:08:59.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:58 smithi167 conmon[49112]: debug 2022-01-31T22:08:58.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:58.967167+0000) 2022-01-31T22:08:59.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[46715]: debug 2022-01-31T22:08:59.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.133640+0000) 2022-01-31T22:08:59.298 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:58 smithi171 conmon[51620]: debug 2022-01-31T22:08:58.984+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:58.986335+0000) 2022-01-31T22:08:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[41853]: debug 2022-01-31T22:08:59.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.340216+0000) 2022-01-31T22:08:59.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:59 smithi167 conmon[54076]: debug 2022-01-31T22:08:59.632+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.633361+0000) 2022-01-31T22:08:59.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:59 smithi167 conmon[60316]: debug 2022-01-31T22:08:59.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.633361+0000) 2022-01-31T22:08:59.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:08:59 smithi167 conmon[60316]: debug 2022-01-31T22:08:59.818+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.819862+0000) 2022-01-31T22:09:00.077 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[35325]: debug 2022-01-31T22:08:59.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186372 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:00.078 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[46715]: debug 2022-01-31T22:08:59.818+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.820561+0000) 2022-01-31T22:09:00.078 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[41853]: debug 2022-01-31T22:08:59.819+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.821156+0000) 2022-01-31T22:09:00.079 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[51620]: debug 2022-01-31T22:08:59.820+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.821774+0000) 2022-01-31T22:09:00.079 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:08:59 smithi171 conmon[51620]: debug 2022-01-31T22:08:59.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.986540+0000) 2022-01-31T22:09:00.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:08:59 smithi167 conmon[54076]: debug 2022-01-31T22:08:59.819+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.821251+0000) 2022-01-31T22:09:00.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:59 smithi167 conmon[49112]: debug 2022-01-31T22:08:59.819+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.821282+0000) 2022-01-31T22:09:00.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:08:59 smithi167 conmon[49112]: debug 2022-01-31T22:08:59.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:08:59.967324+0000) 2022-01-31T22:09:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:00 smithi171 conmon[46715]: debug 2022-01-31T22:09:00.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:00.133791+0000) 2022-01-31T22:09:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:00 smithi171 conmon[41853]: debug 2022-01-31T22:09:00.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:00.340447+0000) 2022-01-31T22:09:00.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:00 smithi167 conmon[54076]: debug 2022-01-31T22:09:00.633+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:00.633545+0000) 2022-01-31T22:09:00.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:00 smithi167 conmon[60316]: debug 2022-01-31T22:09:00.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:00.633544+0000) 2022-01-31T22:09:01.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:00 smithi167 conmon[49112]: debug 2022-01-31T22:09:00.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:00.967486+0000) 2022-01-31T22:09:01.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:01 smithi171 conmon[46715]: debug 2022-01-31T22:09:01.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:01.133923+0000) 2022-01-31T22:09:01.298 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:00 smithi171 conmon[51620]: debug 2022-01-31T22:09:00.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:00.986732+0000) 2022-01-31T22:09:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:01 smithi171 conmon[41853]: debug 2022-01-31T22:09:01.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:01.340618+0000) 2022-01-31T22:09:01.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:01 smithi167 conmon[54076]: debug 2022-01-31T22:09:01.633+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:01.633679+0000) 2022-01-31T22:09:01.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:01 smithi167 conmon[60316]: debug 2022-01-31T22:09:01.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:01.633694+0000) 2022-01-31T22:09:02.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:01 smithi167 conmon[49112]: debug 2022-01-31T22:09:01.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:01.967718+0000) 2022-01-31T22:09:02.297 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:02 smithi171 conmon[46715]: debug 2022-01-31T22:09:02.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:02.134090+0000) 2022-01-31T22:09:02.298 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:01 smithi171 conmon[51620]: debug 2022-01-31T22:09:01.985+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:01.986893+0000) 2022-01-31T22:09:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:02 smithi171 conmon[41853]: debug 2022-01-31T22:09:02.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:02.340808+0000) 2022-01-31T22:09:02.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:02 smithi167 conmon[54076]: debug 2022-01-31T22:09:02.633+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:02.633901+0000) 2022-01-31T22:09:02.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:02 smithi167 conmon[60316]: debug 2022-01-31T22:09:02.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:02.633902+0000) 2022-01-31T22:09:03.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:02 smithi167 conmon[49112]: debug 2022-01-31T22:09:02.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:02.967904+0000) 2022-01-31T22:09:03.298 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:03 smithi171 conmon[46715]: debug 2022-01-31T22:09:03.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:03.134294+0000) 2022-01-31T22:09:03.299 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:02 smithi171 conmon[51620]: debug 2022-01-31T22:09:02.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:02.987078+0000) 2022-01-31T22:09:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:03 smithi171 conmon[41853]: debug 2022-01-31T22:09:03.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:03.341006+0000) 2022-01-31T22:09:03.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:03 smithi167 conmon[54076]: debug 2022-01-31T22:09:03.633+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:03.634113+0000) 2022-01-31T22:09:03.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:03 smithi167 conmon[60316]: debug 2022-01-31T22:09:03.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:03.634113+0000) 2022-01-31T22:09:04.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:03 smithi167 conmon[49112]: debug 2022-01-31T22:09:03.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:03.968100+0000) 2022-01-31T22:09:04.298 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[46715]: debug 2022-01-31T22:09:04.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.134471+0000) 2022-01-31T22:09:04.299 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:03 smithi171 conmon[51620]: debug 2022-01-31T22:09:03.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:03.987260+0000) 2022-01-31T22:09:04.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[41853]: debug 2022-01-31T22:09:04.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.341201+0000) 2022-01-31T22:09:04.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:04 smithi167 conmon[54076]: debug 2022-01-31T22:09:04.633+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.634327+0000) 2022-01-31T22:09:04.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:04 smithi167 conmon[60316]: debug 2022-01-31T22:09:04.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.634327+0000) 2022-01-31T22:09:05.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[35325]: debug 2022-01-31T22:09:04.860+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186486 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:05.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[41853]: debug 2022-01-31T22:09:04.841+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.843388+0000) 2022-01-31T22:09:05.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[46715]: debug 2022-01-31T22:09:04.840+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.841905+0000) 2022-01-31T22:09:05.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[51620]: debug 2022-01-31T22:09:04.841+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.843643+0000) 2022-01-31T22:09:05.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:04 smithi171 conmon[51620]: debug 2022-01-31T22:09:04.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.987451+0000) 2022-01-31T22:09:05.123 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:04 smithi167 conmon[54076]: debug 2022-01-31T22:09:04.842+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.843310+0000) 2022-01-31T22:09:05.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:04 smithi167 conmon[60316]: debug 2022-01-31T22:09:04.842+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.843108+0000) 2022-01-31T22:09:05.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:04 smithi167 conmon[49112]: debug 2022-01-31T22:09:04.842+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.842805+0000) 2022-01-31T22:09:05.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:04 smithi167 conmon[49112]: debug 2022-01-31T22:09:04.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:04.968295+0000) 2022-01-31T22:09:05.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:05 smithi171 conmon[46715]: debug 2022-01-31T22:09:05.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:05.134670+0000) 2022-01-31T22:09:05.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:05 smithi171 conmon[41853]: debug 2022-01-31T22:09:05.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:05.341357+0000) 2022-01-31T22:09:05.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:05 smithi167 conmon[60316]: debug 2022-01-31T22:09:05.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:05.634464+0000) 2022-01-31T22:09:05.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:05 smithi167 conmon[54076]: debug 2022-01-31T22:09:05.634+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:05.634475+0000) 2022-01-31T22:09:06.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:05 smithi167 conmon[49112]: debug 2022-01-31T22:09:05.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:05.968456+0000) 2022-01-31T22:09:06.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:06 smithi171 conmon[46715]: debug 2022-01-31T22:09:06.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:06.134794+0000) 2022-01-31T22:09:06.300 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:05 smithi171 conmon[51620]: debug 2022-01-31T22:09:05.986+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:05.987562+0000) 2022-01-31T22:09:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:06 smithi171 conmon[41853]: debug 2022-01-31T22:09:06.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:06.341495+0000) 2022-01-31T22:09:06.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:06 smithi167 conmon[60316]: debug 2022-01-31T22:09:06.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:06.634638+0000) 2022-01-31T22:09:06.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:06 smithi167 conmon[54076]: debug 2022-01-31T22:09:06.634+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:06.634637+0000) 2022-01-31T22:09:07.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:06 smithi167 conmon[49112]: debug 2022-01-31T22:09:06.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:06.968653+0000) 2022-01-31T22:09:07.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:07 smithi171 conmon[46715]: debug 2022-01-31T22:09:07.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:07.134975+0000) 2022-01-31T22:09:07.299 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:06 smithi171 conmon[51620]: debug 2022-01-31T22:09:06.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:06.987738+0000) 2022-01-31T22:09:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:07 smithi171 conmon[41853]: debug 2022-01-31T22:09:07.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:07.341656+0000) 2022-01-31T22:09:07.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:07 smithi167 conmon[60316]: debug 2022-01-31T22:09:07.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:07.634854+0000) 2022-01-31T22:09:07.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:07 smithi167 conmon[54076]: debug 2022-01-31T22:09:07.634+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:07.634854+0000) 2022-01-31T22:09:08.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:07 smithi167 conmon[49112]: debug 2022-01-31T22:09:07.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:07.968792+0000) 2022-01-31T22:09:08.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:08 smithi171 conmon[46715]: debug 2022-01-31T22:09:08.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:08.135173+0000) 2022-01-31T22:09:08.299 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:07 smithi171 conmon[51620]: debug 2022-01-31T22:09:07.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:07.987933+0000) 2022-01-31T22:09:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:08 smithi171 conmon[41853]: debug 2022-01-31T22:09:08.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:08.341881+0000) 2022-01-31T22:09:08.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:08 smithi167 conmon[60316]: debug 2022-01-31T22:09:08.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:08.635069+0000) 2022-01-31T22:09:08.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:08 smithi167 conmon[54076]: debug 2022-01-31T22:09:08.634+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:08.635069+0000) 2022-01-31T22:09:09.079 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:08 smithi171 conmon[35325]: debug 2022-01-31T22:09:08.804+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:09:09.080 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:08 smithi171 conmon[51620]: debug 2022-01-31T22:09:08.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:08.988135+0000) 2022-01-31T22:09:09.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:08 smithi167 conmon[49112]: debug 2022-01-31T22:09:08.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:08.968947+0000) 2022-01-31T22:09:09.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[46715]: debug 2022-01-31T22:09:09.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.135359+0000) 2022-01-31T22:09:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[41853]: debug 2022-01-31T22:09:09.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.342031+0000) 2022-01-31T22:09:09.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:09 smithi167 conmon[60316]: debug 2022-01-31T22:09:09.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.635290+0000) 2022-01-31T22:09:09.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:09 smithi167 conmon[54076]: debug 2022-01-31T22:09:09.634+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.635290+0000) 2022-01-31T22:09:10.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:09 smithi167 conmon[49112]: debug 2022-01-31T22:09:09.864+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.864753+0000) 2022-01-31T22:09:10.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:09 smithi167 conmon[49112]: debug 2022-01-31T22:09:09.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.969129+0000) 2022-01-31T22:09:10.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:09 smithi167 conmon[54076]: debug 2022-01-31T22:09:09.865+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.865601+0000) 2022-01-31T22:09:10.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:09 smithi167 conmon[60316]: debug 2022-01-31T22:09:09.864+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.865294+0000) 2022-01-31T22:09:10.134 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[46715]: debug 2022-01-31T22:09:09.864+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.865253+0000) 2022-01-31T22:09:10.135 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[51620]: debug 2022-01-31T22:09:09.864+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.865196+0000) 2022-01-31T22:09:10.135 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[51620]: debug 2022-01-31T22:09:09.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.988374+0000) 2022-01-31T22:09:10.136 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[35325]: debug 2022-01-31T22:09:09.883+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186595 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:10.136 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:09 smithi171 conmon[41853]: debug 2022-01-31T22:09:09.864+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:09.865074+0000) 2022-01-31T22:09:10.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:10 smithi171 conmon[46715]: debug 2022-01-31T22:09:10.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:10.135549+0000) 2022-01-31T22:09:10.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:10 smithi171 conmon[41853]: debug 2022-01-31T22:09:10.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:10.342166+0000) 2022-01-31T22:09:10.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:10 smithi167 conmon[54076]: debug 2022-01-31T22:09:10.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:10.635526+0000) 2022-01-31T22:09:10.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:10 smithi167 conmon[60316]: debug 2022-01-31T22:09:10.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:10.635526+0000) 2022-01-31T22:09:11.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:10 smithi167 conmon[49112]: debug 2022-01-31T22:09:10.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:10.969285+0000) 2022-01-31T22:09:11.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:11 smithi171 conmon[46715]: debug 2022-01-31T22:09:11.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:11.135688+0000) 2022-01-31T22:09:11.300 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:10 smithi171 conmon[51620]: debug 2022-01-31T22:09:10.987+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:10.988517+0000) 2022-01-31T22:09:11.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:11 smithi171 conmon[41853]: debug 2022-01-31T22:09:11.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:11.342350+0000) 2022-01-31T22:09:11.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:11 smithi167 conmon[60316]: debug 2022-01-31T22:09:11.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:11.635682+0000) 2022-01-31T22:09:11.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:11 smithi167 conmon[54076]: debug 2022-01-31T22:09:11.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:11.635665+0000) 2022-01-31T22:09:12.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:11 smithi167 conmon[49112]: debug 2022-01-31T22:09:11.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:11.969484+0000) 2022-01-31T22:09:12.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:12 smithi171 conmon[46715]: debug 2022-01-31T22:09:12.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:12.135873+0000) 2022-01-31T22:09:12.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:11 smithi171 conmon[51620]: debug 2022-01-31T22:09:11.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:11.988710+0000) 2022-01-31T22:09:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:12 smithi171 conmon[41853]: debug 2022-01-31T22:09:12.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:12.342508+0000) 2022-01-31T22:09:12.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:12 smithi167 conmon[54076]: debug 2022-01-31T22:09:12.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:12.635835+0000) 2022-01-31T22:09:12.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:12 smithi167 conmon[60316]: debug 2022-01-31T22:09:12.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:12.635835+0000) 2022-01-31T22:09:13.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:12 smithi167 conmon[49112]: debug 2022-01-31T22:09:12.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:12.969690+0000) 2022-01-31T22:09:13.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:13 smithi171 conmon[46715]: debug 2022-01-31T22:09:13.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:13.135999+0000) 2022-01-31T22:09:13.300 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:12 smithi171 conmon[51620]: debug 2022-01-31T22:09:12.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:12.988905+0000) 2022-01-31T22:09:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:13 smithi171 conmon[41853]: debug 2022-01-31T22:09:13.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:13.342718+0000) 2022-01-31T22:09:13.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:13 smithi167 conmon[54076]: debug 2022-01-31T22:09:13.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:13.636013+0000) 2022-01-31T22:09:13.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:13 smithi167 conmon[60316]: debug 2022-01-31T22:09:13.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:13.636013+0000) 2022-01-31T22:09:14.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:13 smithi167 conmon[49112]: debug 2022-01-31T22:09:13.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:13.969889+0000) 2022-01-31T22:09:14.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[46715]: debug 2022-01-31T22:09:14.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.136181+0000) 2022-01-31T22:09:14.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:13 smithi171 conmon[51620]: debug 2022-01-31T22:09:13.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:13.989027+0000) 2022-01-31T22:09:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[41853]: debug 2022-01-31T22:09:14.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.342948+0000) 2022-01-31T22:09:14.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:14 smithi167 conmon[54076]: debug 2022-01-31T22:09:14.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.636201+0000) 2022-01-31T22:09:14.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:14 smithi167 conmon[60316]: debug 2022-01-31T22:09:14.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.636201+0000) 2022-01-31T22:09:15.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:14 smithi167 conmon[54076]: debug 2022-01-31T22:09:14.887+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.888170+0000) 2022-01-31T22:09:15.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:14 smithi167 conmon[49112]: debug 2022-01-31T22:09:14.887+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.888027+0000) 2022-01-31T22:09:15.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:14 smithi167 conmon[49112]: debug 2022-01-31T22:09:14.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.970075+0000) 2022-01-31T22:09:15.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:14 smithi167 conmon[60316]: debug 2022-01-31T22:09:14.886+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.886604+0000) 2022-01-31T22:09:15.300 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[41853]: debug 2022-01-31T22:09:14.886+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.887552+0000) 2022-01-31T22:09:15.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[51620]: debug 2022-01-31T22:09:14.887+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.887908+0000) 2022-01-31T22:09:15.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[51620]: debug 2022-01-31T22:09:14.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.989216+0000) 2022-01-31T22:09:15.302 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[35325]: debug 2022-01-31T22:09:14.905+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186709 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:15.302 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:14 smithi171 conmon[46715]: debug 2022-01-31T22:09:14.886+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:14.887740+0000) 2022-01-31T22:09:15.302 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:15 smithi171 conmon[46715]: debug 2022-01-31T22:09:15.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:15.136308+0000) 2022-01-31T22:09:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:15 smithi171 conmon[41853]: debug 2022-01-31T22:09:15.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:15.343108+0000) 2022-01-31T22:09:15.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:15 smithi167 conmon[54076]: debug 2022-01-31T22:09:15.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:15.636367+0000) 2022-01-31T22:09:15.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:15 smithi167 conmon[60316]: debug 2022-01-31T22:09:15.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:15.636367+0000) 2022-01-31T22:09:16.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:15 smithi167 conmon[49112]: debug 2022-01-31T22:09:15.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:15.970289+0000) 2022-01-31T22:09:16.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:16 smithi171 conmon[46715]: debug 2022-01-31T22:09:16.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:16.136448+0000) 2022-01-31T22:09:16.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:15 smithi171 conmon[51620]: debug 2022-01-31T22:09:15.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:15.989367+0000) 2022-01-31T22:09:16.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:16 smithi171 conmon[41853]: debug 2022-01-31T22:09:16.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:16.343259+0000) 2022-01-31T22:09:16.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:16 smithi167 conmon[54076]: debug 2022-01-31T22:09:16.635+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:16.636473+0000) 2022-01-31T22:09:16.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:16 smithi167 conmon[60316]: debug 2022-01-31T22:09:16.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:16.636473+0000) 2022-01-31T22:09:17.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:16 smithi167 conmon[49112]: debug 2022-01-31T22:09:16.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:16.970490+0000) 2022-01-31T22:09:17.301 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:17 smithi171 conmon[46715]: debug 2022-01-31T22:09:17.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:17.136655+0000) 2022-01-31T22:09:17.302 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:16 smithi171 conmon[51620]: debug 2022-01-31T22:09:16.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:16.989546+0000) 2022-01-31T22:09:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:17 smithi171 conmon[41853]: debug 2022-01-31T22:09:17.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:17.343417+0000) 2022-01-31T22:09:17.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:17 smithi167 conmon[54076]: debug 2022-01-31T22:09:17.636+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:17.636617+0000) 2022-01-31T22:09:17.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:17 smithi167 conmon[60316]: debug 2022-01-31T22:09:17.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:17.636617+0000) 2022-01-31T22:09:18.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:17 smithi167 conmon[49112]: debug 2022-01-31T22:09:17.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:17.970704+0000) 2022-01-31T22:09:18.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:18 smithi171 conmon[46715]: debug 2022-01-31T22:09:18.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:18.136845+0000) 2022-01-31T22:09:18.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:17 smithi171 conmon[51620]: debug 2022-01-31T22:09:17.988+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:17.989733+0000) 2022-01-31T22:09:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:18 smithi171 conmon[41853]: debug 2022-01-31T22:09:18.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:18.343601+0000) 2022-01-31T22:09:18.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:18 smithi167 conmon[54076]: debug 2022-01-31T22:09:18.636+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:18.636774+0000) 2022-01-31T22:09:18.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:18 smithi167 conmon[60316]: debug 2022-01-31T22:09:18.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:18.636773+0000) 2022-01-31T22:09:19.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:18 smithi167 conmon[49112]: debug 2022-01-31T22:09:18.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:18.970870+0000) 2022-01-31T22:09:19.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[46715]: debug 2022-01-31T22:09:19.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.137039+0000) 2022-01-31T22:09:19.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:18 smithi171 conmon[51620]: debug 2022-01-31T22:09:18.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:18.989931+0000) 2022-01-31T22:09:19.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[41853]: debug 2022-01-31T22:09:19.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.343766+0000) 2022-01-31T22:09:19.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:19 smithi167 conmon[54076]: debug 2022-01-31T22:09:19.636+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.636954+0000) 2022-01-31T22:09:19.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:19 smithi167 conmon[60316]: debug 2022-01-31T22:09:19.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.636956+0000) 2022-01-31T22:09:20.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:19 smithi167 conmon[49112]: debug 2022-01-31T22:09:19.908+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.909072+0000) 2022-01-31T22:09:20.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:19 smithi167 conmon[49112]: debug 2022-01-31T22:09:19.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.971029+0000) 2022-01-31T22:09:20.127 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:19 smithi167 conmon[54076]: debug 2022-01-31T22:09:19.909+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.909823+0000) 2022-01-31T22:09:20.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:19 smithi167 conmon[60316]: debug 2022-01-31T22:09:19.908+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.908851+0000) 2022-01-31T22:09:20.301 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[35325]: debug 2022-01-31T22:09:19.927+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186818 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:20.302 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[41853]: debug 2022-01-31T22:09:19.908+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.909600+0000) 2022-01-31T22:09:20.302 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[46715]: debug 2022-01-31T22:09:19.909+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.910093+0000) 2022-01-31T22:09:20.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:20 smithi171 conmon[46715]: debug 2022-01-31T22:09:20.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:20.137225+0000) 2022-01-31T22:09:20.303 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[51620]: debug 2022-01-31T22:09:19.908+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.909450+0000) 2022-01-31T22:09:20.303 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:19 smithi171 conmon[51620]: debug 2022-01-31T22:09:19.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:19.990125+0000) 2022-01-31T22:09:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:20 smithi171 conmon[41853]: debug 2022-01-31T22:09:20.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:20.343919+0000) 2022-01-31T22:09:20.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:20 smithi167 conmon[54076]: debug 2022-01-31T22:09:20.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:20.637173+0000) 2022-01-31T22:09:20.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:20 smithi167 conmon[60316]: debug 2022-01-31T22:09:20.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:20.637173+0000) 2022-01-31T22:09:21.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:20 smithi167 conmon[49112]: debug 2022-01-31T22:09:20.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:20.971192+0000) 2022-01-31T22:09:21.301 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:21 smithi171 conmon[46715]: debug 2022-01-31T22:09:21.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:21.137360+0000) 2022-01-31T22:09:21.302 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:20 smithi171 conmon[51620]: debug 2022-01-31T22:09:20.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:20.990252+0000) 2022-01-31T22:09:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:21 smithi171 conmon[41853]: debug 2022-01-31T22:09:21.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:21.344069+0000) 2022-01-31T22:09:21.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:21 smithi167 conmon[54076]: debug 2022-01-31T22:09:21.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:21.637362+0000) 2022-01-31T22:09:21.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:21 smithi167 conmon[60316]: debug 2022-01-31T22:09:21.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:21.637362+0000) 2022-01-31T22:09:22.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:21 smithi167 conmon[49112]: debug 2022-01-31T22:09:21.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:21.971376+0000) 2022-01-31T22:09:22.301 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:22 smithi171 conmon[46715]: debug 2022-01-31T22:09:22.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:22.137543+0000) 2022-01-31T22:09:22.302 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:21 smithi171 conmon[51620]: debug 2022-01-31T22:09:21.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:21.990411+0000) 2022-01-31T22:09:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:22 smithi171 conmon[41853]: debug 2022-01-31T22:09:22.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:22.344230+0000) 2022-01-31T22:09:22.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:22 smithi167 conmon[54076]: debug 2022-01-31T22:09:22.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:22.637586+0000) 2022-01-31T22:09:22.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:22 smithi167 conmon[60316]: debug 2022-01-31T22:09:22.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:22.637588+0000) 2022-01-31T22:09:23.081 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:22 smithi171 conmon[51620]: debug 2022-01-31T22:09:22.989+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:22.990593+0000) 2022-01-31T22:09:23.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:22 smithi167 conmon[49112]: debug 2022-01-31T22:09:22.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:22.971578+0000) 2022-01-31T22:09:23.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:23 smithi171 conmon[46715]: debug 2022-01-31T22:09:23.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:23.137700+0000) 2022-01-31T22:09:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:23 smithi171 conmon[41853]: debug 2022-01-31T22:09:23.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:23.344431+0000) 2022-01-31T22:09:23.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:23 smithi167 conmon[54076]: debug 2022-01-31T22:09:23.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:23.637766+0000) 2022-01-31T22:09:23.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:23 smithi167 conmon[60316]: debug 2022-01-31T22:09:23.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:23.637766+0000) 2022-01-31T22:09:24.082 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:23 smithi171 conmon[35325]: debug 2022-01-31T22:09:23.805+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:09:24.082 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:23 smithi171 conmon[51620]: debug 2022-01-31T22:09:23.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:23.990794+0000) 2022-01-31T22:09:24.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:23 smithi167 conmon[49112]: debug 2022-01-31T22:09:23.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:23.971727+0000) 2022-01-31T22:09:24.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[46715]: debug 2022-01-31T22:09:24.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.137927+0000) 2022-01-31T22:09:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[41853]: debug 2022-01-31T22:09:24.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.344634+0000) 2022-01-31T22:09:24.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:24 smithi167 conmon[54076]: debug 2022-01-31T22:09:24.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.637980+0000) 2022-01-31T22:09:24.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:24 smithi167 conmon[60316]: debug 2022-01-31T22:09:24.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.637980+0000) 2022-01-31T22:09:25.127 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:24 smithi167 conmon[54076]: debug 2022-01-31T22:09:24.930+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.930779+0000) 2022-01-31T22:09:25.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:24 smithi167 conmon[60316]: debug 2022-01-31T22:09:24.929+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.930150+0000) 2022-01-31T22:09:25.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:24 smithi167 conmon[49112]: debug 2022-01-31T22:09:24.930+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.930616+0000) 2022-01-31T22:09:25.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:24 smithi167 conmon[49112]: debug 2022-01-31T22:09:24.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.971951+0000) 2022-01-31T22:09:25.302 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[35325]: debug 2022-01-31T22:09:24.949+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 186931 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:25.303 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[41853]: debug 2022-01-31T22:09:24.930+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.931475+0000) 2022-01-31T22:09:25.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[46715]: debug 2022-01-31T22:09:24.930+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.931610+0000) 2022-01-31T22:09:25.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:25 smithi171 conmon[46715]: debug 2022-01-31T22:09:25.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:25.138083+0000) 2022-01-31T22:09:25.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[51620]: debug 2022-01-31T22:09:24.931+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.931815+0000) 2022-01-31T22:09:25.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:24 smithi171 conmon[51620]: debug 2022-01-31T22:09:24.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:24.991009+0000) 2022-01-31T22:09:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:25 smithi171 conmon[41853]: debug 2022-01-31T22:09:25.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:25.344821+0000) 2022-01-31T22:09:25.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:25 smithi167 conmon[54076]: debug 2022-01-31T22:09:25.636+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:25.638194+0000) 2022-01-31T22:09:25.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:25 smithi167 conmon[60316]: debug 2022-01-31T22:09:25.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:25.638194+0000) 2022-01-31T22:09:26.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:25 smithi167 conmon[49112]: debug 2022-01-31T22:09:25.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:25.972066+0000) 2022-01-31T22:09:26.302 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:26 smithi171 conmon[46715]: debug 2022-01-31T22:09:26.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:26.138246+0000) 2022-01-31T22:09:26.303 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:25 smithi171 conmon[51620]: debug 2022-01-31T22:09:25.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:25.991194+0000) 2022-01-31T22:09:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:26 smithi171 conmon[41853]: debug 2022-01-31T22:09:26.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:26.344935+0000) 2022-01-31T22:09:26.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:26 smithi167 conmon[54076]: debug 2022-01-31T22:09:26.636+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:26.638357+0000) 2022-01-31T22:09:26.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:26 smithi167 conmon[60316]: debug 2022-01-31T22:09:26.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:26.638357+0000) 2022-01-31T22:09:27.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:26 smithi167 conmon[49112]: debug 2022-01-31T22:09:26.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:26.972249+0000) 2022-01-31T22:09:27.302 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:27 smithi171 conmon[46715]: debug 2022-01-31T22:09:27.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:27.138449+0000) 2022-01-31T22:09:27.303 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:26 smithi171 conmon[51620]: debug 2022-01-31T22:09:26.990+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:26.991388+0000) 2022-01-31T22:09:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:27 smithi171 conmon[41853]: debug 2022-01-31T22:09:27.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:27.345117+0000) 2022-01-31T22:09:27.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:27 smithi167 conmon[54076]: debug 2022-01-31T22:09:27.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:27.638556+0000) 2022-01-31T22:09:27.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:27 smithi167 conmon[60316]: debug 2022-01-31T22:09:27.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:27.638573+0000) 2022-01-31T22:09:28.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:27 smithi167 conmon[49112]: debug 2022-01-31T22:09:27.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:27.972438+0000) 2022-01-31T22:09:28.302 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:28 smithi171 conmon[46715]: debug 2022-01-31T22:09:28.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:28.138622+0000) 2022-01-31T22:09:28.303 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:27 smithi171 conmon[51620]: debug 2022-01-31T22:09:27.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:27.991584+0000) 2022-01-31T22:09:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:28 smithi171 conmon[41853]: debug 2022-01-31T22:09:28.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:28.345298+0000) 2022-01-31T22:09:28.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:28 smithi167 conmon[54076]: debug 2022-01-31T22:09:28.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:28.638747+0000) 2022-01-31T22:09:28.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:28 smithi167 conmon[60316]: debug 2022-01-31T22:09:28.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:28.638769+0000) 2022-01-31T22:09:29.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:28 smithi167 conmon[49112]: debug 2022-01-31T22:09:28.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:28.972531+0000) 2022-01-31T22:09:29.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[46715]: debug 2022-01-31T22:09:29.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.138804+0000) 2022-01-31T22:09:29.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:28 smithi171 conmon[51620]: debug 2022-01-31T22:09:28.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:28.991775+0000) 2022-01-31T22:09:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[41853]: debug 2022-01-31T22:09:29.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.345475+0000) 2022-01-31T22:09:29.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:29 smithi167 conmon[54076]: debug 2022-01-31T22:09:29.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.638922+0000) 2022-01-31T22:09:29.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:29 smithi167 conmon[60316]: debug 2022-01-31T22:09:29.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.638935+0000) 2022-01-31T22:09:30.127 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:29 smithi167 conmon[54076]: debug 2022-01-31T22:09:29.950+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.952188+0000) 2022-01-31T22:09:30.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:29 smithi167 conmon[60316]: debug 2022-01-31T22:09:29.951+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.952866+0000) 2022-01-31T22:09:30.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:29 smithi167 conmon[49112]: debug 2022-01-31T22:09:29.951+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.952651+0000) 2022-01-31T22:09:30.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:29 smithi167 conmon[49112]: debug 2022-01-31T22:09:29.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.972632+0000) 2022-01-31T22:09:30.303 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[35325]: debug 2022-01-31T22:09:29.971+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187040 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:30.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[41853]: debug 2022-01-31T22:09:29.952+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.953235+0000) 2022-01-31T22:09:30.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[46715]: debug 2022-01-31T22:09:29.951+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.952562+0000) 2022-01-31T22:09:30.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:30 smithi171 conmon[46715]: debug 2022-01-31T22:09:30.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:30.138967+0000) 2022-01-31T22:09:30.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[51620]: debug 2022-01-31T22:09:29.952+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.953373+0000) 2022-01-31T22:09:30.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:29 smithi171 conmon[51620]: debug 2022-01-31T22:09:29.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:29.991945+0000) 2022-01-31T22:09:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:30 smithi171 conmon[41853]: debug 2022-01-31T22:09:30.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:30.345696+0000) 2022-01-31T22:09:30.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:30 smithi167 conmon[54076]: debug 2022-01-31T22:09:30.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:30.639112+0000) 2022-01-31T22:09:30.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:30 smithi167 conmon[60316]: debug 2022-01-31T22:09:30.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:30.639127+0000) 2022-01-31T22:09:31.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:30 smithi167 conmon[49112]: debug 2022-01-31T22:09:30.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:30.972773+0000) 2022-01-31T22:09:31.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:31 smithi171 conmon[46715]: debug 2022-01-31T22:09:31.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:31.139123+0000) 2022-01-31T22:09:31.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:30 smithi171 conmon[51620]: debug 2022-01-31T22:09:30.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:30.992079+0000) 2022-01-31T22:09:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:31 smithi171 conmon[41853]: debug 2022-01-31T22:09:31.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:31.345850+0000) 2022-01-31T22:09:31.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:31 smithi167 conmon[54076]: debug 2022-01-31T22:09:31.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:31.639270+0000) 2022-01-31T22:09:31.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:31 smithi167 conmon[60316]: debug 2022-01-31T22:09:31.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:31.639312+0000) 2022-01-31T22:09:32.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:31 smithi167 conmon[49112]: debug 2022-01-31T22:09:31.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:31.973011+0000) 2022-01-31T22:09:32.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:32 smithi171 conmon[46715]: debug 2022-01-31T22:09:32.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:32.139282+0000) 2022-01-31T22:09:32.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:31 smithi171 conmon[51620]: debug 2022-01-31T22:09:31.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:31.992280+0000) 2022-01-31T22:09:32.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:32 smithi171 conmon[41853]: debug 2022-01-31T22:09:32.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:32.345968+0000) 2022-01-31T22:09:32.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:32 smithi167 conmon[54076]: debug 2022-01-31T22:09:32.637+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:32.639469+0000) 2022-01-31T22:09:32.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:32 smithi167 conmon[60316]: debug 2022-01-31T22:09:32.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:32.639469+0000) 2022-01-31T22:09:33.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:32 smithi167 conmon[49112]: debug 2022-01-31T22:09:32.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:32.973200+0000) 2022-01-31T22:09:33.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:33 smithi171 conmon[46715]: debug 2022-01-31T22:09:33.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:33.139448+0000) 2022-01-31T22:09:33.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:32 smithi171 conmon[51620]: debug 2022-01-31T22:09:32.991+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:32.992500+0000) 2022-01-31T22:09:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:33 smithi171 conmon[41853]: debug 2022-01-31T22:09:33.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:33.346142+0000) 2022-01-31T22:09:33.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:33 smithi167 conmon[54076]: debug 2022-01-31T22:09:33.638+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:33.639670+0000) 2022-01-31T22:09:33.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:33 smithi167 conmon[60316]: debug 2022-01-31T22:09:33.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:33.639633+0000) 2022-01-31T22:09:34.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:33 smithi167 conmon[49112]: debug 2022-01-31T22:09:33.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:33.973866+0000) 2022-01-31T22:09:34.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[46715]: debug 2022-01-31T22:09:34.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.139647+0000) 2022-01-31T22:09:34.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:33 smithi171 conmon[51620]: debug 2022-01-31T22:09:33.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:33.992666+0000) 2022-01-31T22:09:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[41853]: debug 2022-01-31T22:09:34.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.346297+0000) 2022-01-31T22:09:34.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:34 smithi167 conmon[54076]: debug 2022-01-31T22:09:34.638+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.639774+0000) 2022-01-31T22:09:34.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:34 smithi167 conmon[60316]: debug 2022-01-31T22:09:34.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.639799+0000) 2022-01-31T22:09:35.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:34 smithi167 conmon[49112]: debug 2022-01-31T22:09:34.974+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.975951+0000) 2022-01-31T22:09:35.129 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:34 smithi167 conmon[54076]: debug 2022-01-31T22:09:34.973+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.975147+0000) 2022-01-31T22:09:35.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:34 smithi167 conmon[60316]: debug 2022-01-31T22:09:34.973+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.974827+0000) 2022-01-31T22:09:35.303 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[35325]: debug 2022-01-31T22:09:34.993+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187154 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:35.304 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[41853]: debug 2022-01-31T22:09:34.973+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.974606+0000) 2022-01-31T22:09:35.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[51620]: debug 2022-01-31T22:09:34.974+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.975160+0000) 2022-01-31T22:09:35.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[51620]: debug 2022-01-31T22:09:34.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.992846+0000) 2022-01-31T22:09:35.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:34 smithi171 conmon[46715]: debug 2022-01-31T22:09:34.973+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:34.974181+0000) 2022-01-31T22:09:35.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:35 smithi171 conmon[46715]: debug 2022-01-31T22:09:35.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:35.139823+0000) 2022-01-31T22:09:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:35 smithi171 conmon[41853]: debug 2022-01-31T22:09:35.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:35.346484+0000) 2022-01-31T22:09:35.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:35 smithi167 conmon[54076]: debug 2022-01-31T22:09:35.638+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:35.639966+0000) 2022-01-31T22:09:35.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:35 smithi167 conmon[60316]: debug 2022-01-31T22:09:35.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:35.639950+0000) 2022-01-31T22:09:36.303 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:36 smithi171 conmon[46715]: debug 2022-01-31T22:09:36.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:36.139923+0000) 2022-01-31T22:09:36.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:35 smithi171 conmon[51620]: debug 2022-01-31T22:09:35.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:35.993019+0000) 2022-01-31T22:09:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:36 smithi171 conmon[41853]: debug 2022-01-31T22:09:36.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:36.346636+0000) 2022-01-31T22:09:36.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:36 smithi167 conmon[54076]: debug 2022-01-31T22:09:36.638+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:36.640105+0000) 2022-01-31T22:09:36.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:36 smithi167 conmon[60316]: debug 2022-01-31T22:09:36.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:36.640097+0000) 2022-01-31T22:09:37.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:37 smithi171 conmon[46715]: debug 2022-01-31T22:09:37.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:37.140035+0000) 2022-01-31T22:09:37.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:36 smithi171 conmon[51620]: debug 2022-01-31T22:09:36.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:36.993217+0000) 2022-01-31T22:09:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:37 smithi171 conmon[41853]: debug 2022-01-31T22:09:37.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:37.346796+0000) 2022-01-31T22:09:37.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:37 smithi167 conmon[54076]: debug 2022-01-31T22:09:37.638+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:37.640292+0000) 2022-01-31T22:09:37.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:37 smithi167 conmon[60316]: debug 2022-01-31T22:09:37.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:37.640287+0000) 2022-01-31T22:09:38.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:38 smithi171 conmon[46715]: debug 2022-01-31T22:09:38.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:38.140198+0000) 2022-01-31T22:09:38.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:37 smithi171 conmon[51620]: debug 2022-01-31T22:09:37.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:37.993377+0000) 2022-01-31T22:09:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:38 smithi171 conmon[41853]: debug 2022-01-31T22:09:38.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:38.346929+0000) 2022-01-31T22:09:38.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:38 smithi167 conmon[54076]: debug 2022-01-31T22:09:38.639+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:38.640531+0000) 2022-01-31T22:09:38.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:38 smithi167 conmon[60316]: debug 2022-01-31T22:09:38.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:38.640531+0000) 2022-01-31T22:09:39.084 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:38 smithi171 conmon[51620]: debug 2022-01-31T22:09:38.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:38.993592+0000) 2022-01-31T22:09:39.085 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:38 smithi171 conmon[35325]: debug 2022-01-31T22:09:38.806+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:09:39.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:39 smithi171 conmon[46715]: debug 2022-01-31T22:09:39.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.140401+0000) 2022-01-31T22:09:39.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:39 smithi171 conmon[41853]: debug 2022-01-31T22:09:39.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.347110+0000) 2022-01-31T22:09:39.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:39 smithi167 conmon[54076]: debug 2022-01-31T22:09:39.639+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.640700+0000) 2022-01-31T22:09:39.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:39 smithi167 conmon[60316]: debug 2022-01-31T22:09:39.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.640650+0000) 2022-01-31T22:09:40.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:40 smithi167 conmon[49112]: debug 2022-01-31T22:09:40.002+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:40.004248+0000) 2022-01-31T22:09:40.130 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:39 smithi167 conmon[54076]: debug 2022-01-31T22:09:39.996+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.997788+0000) 2022-01-31T22:09:40.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:39 smithi167 conmon[60316]: debug 2022-01-31T22:09:39.995+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.996905+0000) 2022-01-31T22:09:40.304 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:40 smithi171 conmon[35325]: debug 2022-01-31T22:09:40.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187263 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:40.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:39 smithi171 conmon[41853]: debug 2022-01-31T22:09:39.995+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.996667+0000) 2022-01-31T22:09:40.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:39 smithi171 conmon[46715]: debug 2022-01-31T22:09:39.995+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.996582+0000) 2022-01-31T22:09:40.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:40 smithi171 conmon[46715]: debug 2022-01-31T22:09:40.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:40.140560+0000) 2022-01-31T22:09:40.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:39 smithi171 conmon[51620]: debug 2022-01-31T22:09:39.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.993798+0000) 2022-01-31T22:09:40.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:39 smithi171 conmon[51620]: debug 2022-01-31T22:09:39.996+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:39.996912+0000) 2022-01-31T22:09:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:40 smithi171 conmon[41853]: debug 2022-01-31T22:09:40.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:40.347304+0000) 2022-01-31T22:09:40.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:40 smithi167 conmon[54076]: debug 2022-01-31T22:09:40.640+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:40.640888+0000) 2022-01-31T22:09:40.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:40 smithi167 conmon[60316]: debug 2022-01-31T22:09:40.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:40.640832+0000) 2022-01-31T22:09:41.304 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:41 smithi171 conmon[46715]: debug 2022-01-31T22:09:41.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:41.140702+0000) 2022-01-31T22:09:41.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:40 smithi171 conmon[51620]: debug 2022-01-31T22:09:40.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:40.993988+0000) 2022-01-31T22:09:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:41 smithi171 conmon[41853]: debug 2022-01-31T22:09:41.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:41.347445+0000) 2022-01-31T22:09:41.723 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:41 smithi167 conmon[54076]: debug 2022-01-31T22:09:41.640+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:41.641042+0000) 2022-01-31T22:09:41.724 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:41 smithi167 conmon[60316]: debug 2022-01-31T22:09:41.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:41.641026+0000) 2022-01-31T22:09:41.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:41 smithi167 conmon[49112]: debug 2022-01-31T22:09:41.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:41.724271+0000) 2022-01-31T22:09:42.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:42 smithi171 conmon[46715]: debug 2022-01-31T22:09:42.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:42.140908+0000) 2022-01-31T22:09:42.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:41 smithi171 conmon[51620]: debug 2022-01-31T22:09:41.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:41.994131+0000) 2022-01-31T22:09:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:42 smithi171 conmon[41853]: debug 2022-01-31T22:09:42.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:42.347660+0000) 2022-01-31T22:09:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:42 smithi167 conmon[49112]: debug 2022-01-31T22:09:42.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:42.724469+0000) 2022-01-31T22:09:42.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:42 smithi167 conmon[54076]: debug 2022-01-31T22:09:42.640+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:42.641233+0000) 2022-01-31T22:09:42.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:42 smithi167 conmon[60316]: debug 2022-01-31T22:09:42.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:42.641227+0000) 2022-01-31T22:09:43.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:43 smithi171 conmon[46715]: debug 2022-01-31T22:09:43.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:43.141044+0000) 2022-01-31T22:09:43.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:42 smithi171 conmon[51620]: debug 2022-01-31T22:09:42.992+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:42.994331+0000) 2022-01-31T22:09:43.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:43 smithi171 conmon[41853]: debug 2022-01-31T22:09:43.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:43.347843+0000) 2022-01-31T22:09:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:43 smithi167 conmon[49112]: debug 2022-01-31T22:09:43.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:43.724638+0000) 2022-01-31T22:09:43.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:43 smithi167 conmon[54076]: debug 2022-01-31T22:09:43.640+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:43.641449+0000) 2022-01-31T22:09:43.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:43 smithi167 conmon[60316]: debug 2022-01-31T22:09:43.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:43.641425+0000) 2022-01-31T22:09:44.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:44 smithi171 conmon[46715]: debug 2022-01-31T22:09:44.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:44.141216+0000) 2022-01-31T22:09:44.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:43 smithi171 conmon[51620]: debug 2022-01-31T22:09:43.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:43.994544+0000) 2022-01-31T22:09:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:44 smithi171 conmon[41853]: debug 2022-01-31T22:09:44.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:44.348033+0000) 2022-01-31T22:09:44.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:44 smithi167 conmon[49112]: debug 2022-01-31T22:09:44.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:44.724812+0000) 2022-01-31T22:09:44.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:44 smithi167 conmon[54076]: debug 2022-01-31T22:09:44.640+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:44.641610+0000) 2022-01-31T22:09:44.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:44 smithi167 conmon[60316]: debug 2022-01-31T22:09:44.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:44.641610+0000) 2022-01-31T22:09:45.305 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[35325]: debug 2022-01-31T22:09:45.052+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187377 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:45.306 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[41853]: debug 2022-01-31T22:09:45.032+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.034680+0000) 2022-01-31T22:09:45.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[46715]: debug 2022-01-31T22:09:45.032+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.034408+0000) 2022-01-31T22:09:45.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[46715]: debug 2022-01-31T22:09:45.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.141411+0000) 2022-01-31T22:09:45.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:44 smithi171 conmon[51620]: debug 2022-01-31T22:09:44.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:44.994737+0000) 2022-01-31T22:09:45.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[51620]: debug 2022-01-31T22:09:45.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[51620]: 2022-01-31T22:09:45.032+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.033762+0000) 2022-01-31T22:09:45.334 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:45 smithi167 conmon[49112]: debug 2022-01-31T22:09:45.033+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.035462+0000) 2022-01-31T22:09:45.335 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:45 smithi167 conmon[54076]: debug 2022-01-31T22:09:45.033+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.034565+0000) 2022-01-31T22:09:45.336 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:45 smithi167 conmon[60316]: debug 2022-01-31T22:09:45.032+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.033914+0000) 2022-01-31T22:09:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[41853]: debug 2022-01-31T22:09:45.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.348182+0000) 2022-01-31T22:09:45.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:45 smithi167 conmon[49112]: debug 2022-01-31T22:09:45.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.724965+0000) 2022-01-31T22:09:45.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:45 smithi167 conmon[54076]: debug 2022-01-31T22:09:45.641+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.641807+0000) 2022-01-31T22:09:45.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:45 smithi167 conmon[60316]: debug 2022-01-31T22:09:45.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.641812+0000) 2022-01-31T22:09:46.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:46 smithi171 conmon[46715]: debug 2022-01-31T22:09:46.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:46.141563+0000) 2022-01-31T22:09:46.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:45 smithi171 conmon[51620]: debug 2022-01-31T22:09:45.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:45.994921+0000) 2022-01-31T22:09:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:46 smithi171 conmon[41853]: debug 2022-01-31T22:09:46.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:46.348336+0000) 2022-01-31T22:09:46.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:46 smithi167 conmon[54076]: debug 2022-01-31T22:09:46.641+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:46.641921+0000) 2022-01-31T22:09:46.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:46 smithi167 conmon[60316]: debug 2022-01-31T22:09:46.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:46.641944+0000) 2022-01-31T22:09:46.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:46 smithi167 conmon[49112]: debug 2022-01-31T22:09:46.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:46.725108+0000) 2022-01-31T22:09:47.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:47 smithi171 conmon[46715]: debug 2022-01-31T22:09:47.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:47.141726+0000) 2022-01-31T22:09:47.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:46 smithi171 conmon[51620]: debug 2022-01-31T22:09:46.993+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:46.995059+0000) 2022-01-31T22:09:47.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:47 smithi171 conmon[41853]: debug 2022-01-31T22:09:47.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:47.348502+0000) 2022-01-31T22:09:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:47 smithi167 conmon[49112]: debug 2022-01-31T22:09:47.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:47.725204+0000) 2022-01-31T22:09:47.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:47 smithi167 conmon[54076]: debug 2022-01-31T22:09:47.641+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:47.642100+0000) 2022-01-31T22:09:47.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:47 smithi167 conmon[60316]: debug 2022-01-31T22:09:47.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:47.642115+0000) 2022-01-31T22:09:48.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:48 smithi171 conmon[46715]: debug 2022-01-31T22:09:48.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:48.141897+0000) 2022-01-31T22:09:48.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:47 smithi171 conmon[51620]: debug 2022-01-31T22:09:47.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:47.995256+0000) 2022-01-31T22:09:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:48 smithi171 conmon[41853]: debug 2022-01-31T22:09:48.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:48.348646+0000) 2022-01-31T22:09:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:48 smithi167 conmon[49112]: debug 2022-01-31T22:09:48.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:48.725369+0000) 2022-01-31T22:09:48.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:48 smithi167 conmon[60316]: debug 2022-01-31T22:09:48.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:48.642313+0000) 2022-01-31T22:09:48.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:48 smithi167 conmon[54076]: debug 2022-01-31T22:09:48.640+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:48.642312+0000) 2022-01-31T22:09:49.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:49 smithi171 conmon[46715]: debug 2022-01-31T22:09:49.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:49.142091+0000) 2022-01-31T22:09:49.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:48 smithi171 conmon[51620]: debug 2022-01-31T22:09:48.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:48.995468+0000) 2022-01-31T22:09:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:49 smithi171 conmon[41853]: debug 2022-01-31T22:09:49.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:49.348867+0000) 2022-01-31T22:09:49.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:49 smithi167 conmon[49112]: debug 2022-01-31T22:09:49.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:49.725585+0000) 2022-01-31T22:09:49.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:49 smithi167 conmon[60316]: debug 2022-01-31T22:09:49.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:49.642529+0000) 2022-01-31T22:09:49.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:49 smithi167 conmon[54076]: debug 2022-01-31T22:09:49.642+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:49.642529+0000) 2022-01-31T22:09:50.306 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[35325]: debug 2022-01-31T22:09:50.074+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187487 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:50.307 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[41853]: debug 2022-01-31T22:09:50.055+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.057269+0000) 2022-01-31T22:09:50.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[46715]: debug 2022-01-31T22:09:50.054+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.056266+0000) 2022-01-31T22:09:50.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[46715]: debug 2022-01-31T22:09:50.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.142292+0000) 2022-01-31T22:09:50.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:49 smithi171 conmon[51620]: debug 2022-01-31T22:09:49.994+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:49.995674+0000) 2022-01-31T22:09:50.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[51620]: debug 2022-01-31T22:09:50.054+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.056561+0000) 2022-01-31T22:09:50.347 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:50 smithi167 conmon[49112]: debug 2022-01-31T22:09:50.056+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.057040+0000) 2022-01-31T22:09:50.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:50 smithi167 conmon[60316]: debug 2022-01-31T22:09:50.056+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.056599+0000) 2022-01-31T22:09:50.348 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:50 smithi167 conmon[54076]: debug 2022-01-31T22:09:50.055+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.056365+0000) 2022-01-31T22:09:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[41853]: debug 2022-01-31T22:09:50.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.349028+0000) 2022-01-31T22:09:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:50 smithi167 conmon[49112]: debug 2022-01-31T22:09:50.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.725754+0000) 2022-01-31T22:09:50.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:50 smithi167 conmon[54076]: debug 2022-01-31T22:09:50.642+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.642698+0000) 2022-01-31T22:09:50.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:50 smithi167 conmon[60316]: debug 2022-01-31T22:09:50.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.642695+0000) 2022-01-31T22:09:51.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:51 smithi171 conmon[46715]: debug 2022-01-31T22:09:51.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:51.142441+0000) 2022-01-31T22:09:51.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:50 smithi171 conmon[51620]: debug 2022-01-31T22:09:50.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:50.995875+0000) 2022-01-31T22:09:51.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:51 smithi171 conmon[41853]: debug 2022-01-31T22:09:51.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:51.349131+0000) 2022-01-31T22:09:51.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:51 smithi167 conmon[49112]: debug 2022-01-31T22:09:51.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:51.725866+0000) 2022-01-31T22:09:51.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:51 smithi167 conmon[60316]: debug 2022-01-31T22:09:51.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:51.642868+0000) 2022-01-31T22:09:51.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:51 smithi167 conmon[54076]: debug 2022-01-31T22:09:51.642+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:51.642839+0000) 2022-01-31T22:09:52.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:52 smithi171 conmon[46715]: debug 2022-01-31T22:09:52.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:52.142639+0000) 2022-01-31T22:09:52.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:51 smithi171 conmon[51620]: debug 2022-01-31T22:09:51.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:51.996050+0000) 2022-01-31T22:09:52.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:52 smithi171 conmon[41853]: debug 2022-01-31T22:09:52.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:52.349360+0000) 2022-01-31T22:09:52.922 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:52 smithi167 conmon[49112]: debug 2022-01-31T22:09:52.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:52.726082+0000) 2022-01-31T22:09:53.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:52 smithi167 conmon[54076]: debug 2022-01-31T22:09:52.642+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:52.643048+0000) 2022-01-31T22:09:53.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:52 smithi167 conmon[60316]: debug 2022-01-31T22:09:52.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:52.643050+0000) 2022-01-31T22:09:53.364 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:53 smithi171 conmon[46715]: debug 2022-01-31T22:09:53.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:53.142846+0000) 2022-01-31T22:09:53.365 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:52 smithi171 conmon[51620]: debug 2022-01-31T22:09:52.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:52.996235+0000) 2022-01-31T22:09:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:53 smithi171 conmon[41853]: debug 2022-01-31T22:09:53.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:53.349577+0000) 2022-01-31T22:09:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:53 smithi167 conmon[49112]: debug 2022-01-31T22:09:53.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:53.726295+0000) 2022-01-31T22:09:53.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:53 smithi167 conmon[54076]: debug 2022-01-31T22:09:53.642+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:53.643244+0000) 2022-01-31T22:09:53.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:53 smithi167 conmon[60316]: debug 2022-01-31T22:09:53.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:53.643245+0000) 2022-01-31T22:09:54.088 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:53 smithi171 conmon[35325]: debug 2022-01-31T22:09:53.806+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:09:54.089 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:53 smithi171 conmon[51620]: debug 2022-01-31T22:09:53.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:53.996401+0000) 2022-01-31T22:09:54.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:54 smithi171 conmon[46715]: debug 2022-01-31T22:09:54.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:54.143006+0000) 2022-01-31T22:09:54.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:54 smithi171 conmon[41853]: debug 2022-01-31T22:09:54.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:54.349783+0000) 2022-01-31T22:09:54.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:54 smithi167 conmon[54076]: debug 2022-01-31T22:09:54.642+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:54.643462+0000) 2022-01-31T22:09:54.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:54 smithi167 conmon[49112]: debug 2022-01-31T22:09:54.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:54.726472+0000) 2022-01-31T22:09:54.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:54 smithi167 conmon[60316]: debug 2022-01-31T22:09:54.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:54.643462+0000) 2022-01-31T22:09:55.307 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[35325]: debug 2022-01-31T22:09:55.097+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:09:55.307 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[41853]: debug 2022-01-31T22:09:55.077+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.078646+0000) 2022-01-31T22:09:55.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[46715]: debug 2022-01-31T22:09:55.079+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.080055+0000) 2022-01-31T22:09:55.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[46715]: debug 2022-01-31T22:09:55.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.143124+0000) 2022-01-31T22:09:55.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:54 smithi171 conmon[51620]: debug 2022-01-31T22:09:54.995+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:54.996557+0000) 2022-01-31T22:09:55.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[51620]: debug 2022-01-31T22:09:55.078+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.078878+0000) 2022-01-31T22:09:55.359 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:55 smithi167 conmon[49112]: debug 2022-01-31T22:09:55.077+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.078307+0000) 2022-01-31T22:09:55.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:55 smithi167 conmon[54076]: debug 2022-01-31T22:09:55.078+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.079294+0000) 2022-01-31T22:09:55.361 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:55 smithi167 conmon[60316]: debug 2022-01-31T22:09:55.079+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.079676+0000) 2022-01-31T22:09:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[41853]: debug 2022-01-31T22:09:55.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.349910+0000) 2022-01-31T22:09:55.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:55 smithi167 conmon[49112]: debug 2022-01-31T22:09:55.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.726591+0000) 2022-01-31T22:09:55.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:55 smithi167 conmon[60316]: debug 2022-01-31T22:09:55.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.643691+0000) 2022-01-31T22:09:55.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:55 smithi167 conmon[54076]: debug 2022-01-31T22:09:55.643+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.643686+0000) 2022-01-31T22:09:56.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:56 smithi171 conmon[46715]: debug 2022-01-31T22:09:56.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:56.143272+0000) 2022-01-31T22:09:56.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:55 smithi171 conmon[51620]: debug 2022-01-31T22:09:55.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:55.996749+0000) 2022-01-31T22:09:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:56 smithi171 conmon[41853]: debug 2022-01-31T22:09:56.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:56.350064+0000) 2022-01-31T22:09:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:56 smithi167 conmon[49112]: debug 2022-01-31T22:09:56.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:56.726719+0000) 2022-01-31T22:09:56.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:56 smithi167 conmon[54076]: debug 2022-01-31T22:09:56.643+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:56.643857+0000) 2022-01-31T22:09:56.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:56 smithi167 conmon[60316]: debug 2022-01-31T22:09:56.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:56.643860+0000) 2022-01-31T22:09:57.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:57 smithi171 conmon[46715]: debug 2022-01-31T22:09:57.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:57.143380+0000) 2022-01-31T22:09:57.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:56 smithi171 conmon[51620]: debug 2022-01-31T22:09:56.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:56.996989+0000) 2022-01-31T22:09:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:57 smithi171 conmon[41853]: debug 2022-01-31T22:09:57.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:57.350253+0000) 2022-01-31T22:09:57.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:57 smithi167 conmon[49112]: debug 2022-01-31T22:09:57.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:57.726875+0000) 2022-01-31T22:09:57.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:57 smithi167 conmon[54076]: debug 2022-01-31T22:09:57.643+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:57.644058+0000) 2022-01-31T22:09:57.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:57 smithi167 conmon[60316]: debug 2022-01-31T22:09:57.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:57.644074+0000) 2022-01-31T22:09:58.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:58 smithi171 conmon[46715]: debug 2022-01-31T22:09:58.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:58.143536+0000) 2022-01-31T22:09:58.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:57 smithi171 conmon[51620]: debug 2022-01-31T22:09:57.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:57.997121+0000) 2022-01-31T22:09:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:58 smithi171 conmon[41853]: debug 2022-01-31T22:09:58.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:58.350457+0000) 2022-01-31T22:09:58.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:58 smithi167 conmon[49112]: debug 2022-01-31T22:09:58.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:58.727023+0000) 2022-01-31T22:09:58.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:58 smithi167 conmon[60316]: debug 2022-01-31T22:09:58.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:58.644305+0000) 2022-01-31T22:09:58.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:58 smithi167 conmon[54076]: debug 2022-01-31T22:09:58.643+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:58.644282+0000) 2022-01-31T22:09:59.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:09:59 smithi171 conmon[46715]: debug 2022-01-31T22:09:59.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:59.143716+0000) 2022-01-31T22:09:59.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:58 smithi171 conmon[51620]: debug 2022-01-31T22:09:58.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:58.997265+0000) 2022-01-31T22:09:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:09:59 smithi171 conmon[41853]: debug 2022-01-31T22:09:59.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:59.350640+0000) 2022-01-31T22:09:59.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:09:59 smithi167 conmon[49112]: debug 2022-01-31T22:09:59.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:59.727198+0000) 2022-01-31T22:09:59.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:09:59 smithi167 conmon[54076]: debug 2022-01-31T22:09:59.644+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:59.644449+0000) 2022-01-31T22:09:59.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:09:59 smithi167 conmon[60316]: debug 2022-01-31T22:09:59.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:59.644449+0000) 2022-01-31T22:10:00.308 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[35325]: debug 2022-01-31T22:10:00.120+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187709 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:00.308 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[41853]: debug 2022-01-31T22:10:00.100+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.101174+0000) 2022-01-31T22:10:00.309 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[46715]: debug 2022-01-31T22:10:00.100+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.101629+0000) 2022-01-31T22:10:00.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[46715]: debug 2022-01-31T22:10:00.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.143938+0000) 2022-01-31T22:10:00.310 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:09:59 smithi171 conmon[51620]: debug 2022-01-31T22:09:59.996+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:09:59.997460+0000) 2022-01-31T22:10:00.310 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[51620]: debug 2022-01-31T22:10:00.101+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.101914+0000) 2022-01-31T22:10:00.372 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:00 smithi167 conmon[49112]: debug 2022-01-31T22:10:00.100+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.100881+0000) 2022-01-31T22:10:00.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:00 smithi167 conmon[54076]: debug 2022-01-31T22:10:00.099+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.100338+0000) 2022-01-31T22:10:00.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:00 smithi167 conmon[60316]: debug 2022-01-31T22:10:00.101+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.101599+0000) 2022-01-31T22:10:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[41853]: debug 2022-01-31T22:10:00.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.350820+0000) 2022-01-31T22:10:00.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:00 smithi167 conmon[60316]: debug 2022-01-31T22:10:00.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.644637+0000) 2022-01-31T22:10:00.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:00 smithi167 conmon[49112]: debug 2022-01-31T22:10:00.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.727385+0000) 2022-01-31T22:10:00.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:00 smithi167 conmon[54076]: debug 2022-01-31T22:10:00.644+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.644704+0000) 2022-01-31T22:10:01.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:01 smithi171 conmon[46715]: debug 2022-01-31T22:10:01.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:01.144086+0000) 2022-01-31T22:10:01.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:00 smithi171 conmon[51620]: debug 2022-01-31T22:10:00.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:00.997663+0000) 2022-01-31T22:10:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:01 smithi171 conmon[41853]: debug 2022-01-31T22:10:01.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:01.350978+0000) 2022-01-31T22:10:01.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:01 smithi167 conmon[49112]: debug 2022-01-31T22:10:01.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:01.727537+0000) 2022-01-31T22:10:01.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:01 smithi167 conmon[54076]: debug 2022-01-31T22:10:01.644+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:01.644863+0000) 2022-01-31T22:10:01.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:01 smithi167 conmon[60316]: debug 2022-01-31T22:10:01.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:01.644824+0000) 2022-01-31T22:10:02.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:02 smithi171 conmon[46715]: debug 2022-01-31T22:10:02.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:02.144272+0000) 2022-01-31T22:10:02.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:01 smithi171 conmon[51620]: debug 2022-01-31T22:10:01.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:01.997837+0000) 2022-01-31T22:10:02.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:02 smithi171 conmon[41853]: debug 2022-01-31T22:10:02.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:02.351162+0000) 2022-01-31T22:10:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:02 smithi167 conmon[49112]: debug 2022-01-31T22:10:02.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:02.727704+0000) 2022-01-31T22:10:02.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:02 smithi167 conmon[54076]: debug 2022-01-31T22:10:02.644+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:02.645090+0000) 2022-01-31T22:10:02.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:02 smithi167 conmon[60316]: debug 2022-01-31T22:10:02.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:02.645013+0000) 2022-01-31T22:10:03.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:03 smithi171 conmon[46715]: debug 2022-01-31T22:10:03.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:03.144376+0000) 2022-01-31T22:10:03.310 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:02 smithi171 conmon[51620]: debug 2022-01-31T22:10:02.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:02.998035+0000) 2022-01-31T22:10:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:03 smithi171 conmon[41853]: debug 2022-01-31T22:10:03.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:03.351358+0000) 2022-01-31T22:10:03.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:03 smithi167 conmon[49112]: debug 2022-01-31T22:10:03.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:03.727857+0000) 2022-01-31T22:10:03.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:03 smithi167 conmon[54076]: debug 2022-01-31T22:10:03.644+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:03.645318+0000) 2022-01-31T22:10:03.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:03 smithi167 conmon[60316]: debug 2022-01-31T22:10:03.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:03.645201+0000) 2022-01-31T22:10:04.308 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:04 smithi171 conmon[46715]: debug 2022-01-31T22:10:04.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:04.144553+0000) 2022-01-31T22:10:04.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:03 smithi171 conmon[51620]: debug 2022-01-31T22:10:03.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:03.998228+0000) 2022-01-31T22:10:04.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:04 smithi171 conmon[41853]: debug 2022-01-31T22:10:04.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:04.351543+0000) 2022-01-31T22:10:04.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:04 smithi167 conmon[54076]: debug 2022-01-31T22:10:04.645+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:04.645496+0000) 2022-01-31T22:10:04.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:04 smithi167 conmon[60316]: debug 2022-01-31T22:10:04.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:04.645390+0000) 2022-01-31T22:10:04.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:04 smithi167 conmon[49112]: debug 2022-01-31T22:10:04.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:04.728050+0000) 2022-01-31T22:10:05.308 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[35325]: debug 2022-01-31T22:10:05.142+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187821 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:05.309 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[41853]: debug 2022-01-31T22:10:05.123+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.124226+0000) 2022-01-31T22:10:05.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[46715]: debug 2022-01-31T22:10:05.122+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.123000+0000) 2022-01-31T22:10:05.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[46715]: debug 2022-01-31T22:10:05.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.144686+0000) 2022-01-31T22:10:05.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:04 smithi171 conmon[51620]: debug 2022-01-31T22:10:04.997+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:04.998433+0000) 2022-01-31T22:10:05.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[51620]: debug 2022-01-31T22:10:05.124+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.124957+0000) 2022-01-31T22:10:05.385 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:05 smithi167 conmon[49112]: debug 2022-01-31T22:10:05.123+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.124210+0000) 2022-01-31T22:10:05.386 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:05 smithi167 conmon[54076]: debug 2022-01-31T22:10:05.122+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.123325+0000) 2022-01-31T22:10:05.387 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:05 smithi167 conmon[60316]: debug 2022-01-31T22:10:05.124+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.124570+0000) 2022-01-31T22:10:05.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[41853]: debug 2022-01-31T22:10:05.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.351731+0000) 2022-01-31T22:10:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:05 smithi167 conmon[49112]: debug 2022-01-31T22:10:05.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.728206+0000) 2022-01-31T22:10:05.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:05 smithi167 conmon[54076]: debug 2022-01-31T22:10:05.645+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.645677+0000) 2022-01-31T22:10:05.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:05 smithi167 conmon[60316]: debug 2022-01-31T22:10:05.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.645607+0000) 2022-01-31T22:10:06.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:06 smithi171 conmon[46715]: debug 2022-01-31T22:10:06.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:06.144839+0000) 2022-01-31T22:10:06.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:05 smithi171 conmon[51620]: debug 2022-01-31T22:10:05.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:05.998611+0000) 2022-01-31T22:10:06.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:06 smithi171 conmon[41853]: debug 2022-01-31T22:10:06.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:06.351846+0000) 2022-01-31T22:10:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:06 smithi167 conmon[49112]: debug 2022-01-31T22:10:06.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:06.728344+0000) 2022-01-31T22:10:06.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:06 smithi167 conmon[54076]: debug 2022-01-31T22:10:06.645+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:06.645799+0000) 2022-01-31T22:10:06.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:06 smithi167 conmon[60316]: debug 2022-01-31T22:10:06.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:06.645756+0000) 2022-01-31T22:10:07.309 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:07 smithi171 conmon[46715]: debug 2022-01-31T22:10:07.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:07.145025+0000) 2022-01-31T22:10:07.310 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:06 smithi171 conmon[51620]: debug 2022-01-31T22:10:06.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:06.998789+0000) 2022-01-31T22:10:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:07 smithi171 conmon[41853]: debug 2022-01-31T22:10:07.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:07.352009+0000) 2022-01-31T22:10:07.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:07 smithi167 conmon[54076]: debug 2022-01-31T22:10:07.645+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:07.645992+0000) 2022-01-31T22:10:07.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:07 smithi167 conmon[49112]: debug 2022-01-31T22:10:07.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:07.728601+0000) 2022-01-31T22:10:07.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:07 smithi167 conmon[60316]: debug 2022-01-31T22:10:07.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:07.645950+0000) 2022-01-31T22:10:08.089 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:07 smithi171 conmon[51620]: debug 2022-01-31T22:10:07.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:07.998939+0000) 2022-01-31T22:10:08.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:08 smithi171 conmon[46715]: debug 2022-01-31T22:10:08.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:08.145224+0000) 2022-01-31T22:10:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:08 smithi171 conmon[41853]: debug 2022-01-31T22:10:08.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:08.352219+0000) 2022-01-31T22:10:08.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:08 smithi167 conmon[60316]: debug 2022-01-31T22:10:08.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:08.646163+0000) 2022-01-31T22:10:08.670 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:08 smithi167 conmon[54076]: debug 2022-01-31T22:10:08.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:08.646192+0000) 2022-01-31T22:10:08.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:08 smithi167 conmon[49112]: debug 2022-01-31T22:10:08.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:08.728785+0000) 2022-01-31T22:10:08.998 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:08 smithi171 conmon[35325]: debug 2022-01-31T22:10:08.807+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:10:09.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:08 smithi171 conmon[51620]: debug 2022-01-31T22:10:08.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:08.999116+0000) 2022-01-31T22:10:09.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:09 smithi171 conmon[46715]: debug 2022-01-31T22:10:09.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:09.145433+0000) 2022-01-31T22:10:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:09 smithi171 conmon[41853]: debug 2022-01-31T22:10:09.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:09.352415+0000) 2022-01-31T22:10:09.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:09 smithi167 conmon[54076]: debug 2022-01-31T22:10:09.645+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:09.646406+0000) 2022-01-31T22:10:09.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:09 smithi167 conmon[60316]: debug 2022-01-31T22:10:09.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:09.646328+0000) 2022-01-31T22:10:09.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:09 smithi167 conmon[49112]: debug 2022-01-31T22:10:09.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:09.728971+0000) 2022-01-31T22:10:10.305 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[35325]: debug 2022-01-31T22:10:10.164+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 187931 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:10.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[41853]: debug 2022-01-31T22:10:10.144+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.145612+0000) 2022-01-31T22:10:10.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:09 smithi171 conmon[51620]: debug 2022-01-31T22:10:09.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:09.999321+0000) 2022-01-31T22:10:10.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[51620]: debug 2022-01-31T22:10:10.145+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.146224+0000) 2022-01-31T22:10:10.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[46715]: debug 2022-01-31T22:10:10.144+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.144926+0000) 2022-01-31T22:10:10.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[46715]: debug 2022-01-31T22:10:10.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.145744+0000) 2022-01-31T22:10:10.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:10 smithi167 conmon[54076]: debug 2022-01-31T22:10:10.144+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.144970+0000) 2022-01-31T22:10:10.399 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:10 smithi167 conmon[60316]: debug 2022-01-31T22:10:10.145+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.146293+0000) 2022-01-31T22:10:10.400 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:10 smithi167 conmon[49112]: debug 2022-01-31T22:10:10.145+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.145806+0000) 2022-01-31T22:10:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[41853]: debug 2022-01-31T22:10:10.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.352615+0000) 2022-01-31T22:10:10.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:10 smithi167 conmon[60316]: debug 2022-01-31T22:10:10.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.646450+0000) 2022-01-31T22:10:10.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:10 smithi167 conmon[54076]: debug 2022-01-31T22:10:10.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.646587+0000) 2022-01-31T22:10:10.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:10 smithi167 conmon[49112]: debug 2022-01-31T22:10:10.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.729182+0000) 2022-01-31T22:10:11.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:11 smithi171 conmon[46715]: debug 2022-01-31T22:10:11.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:11.145883+0000) 2022-01-31T22:10:11.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:10 smithi171 conmon[51620]: debug 2022-01-31T22:10:10.998+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:10.999515+0000) 2022-01-31T22:10:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:11 smithi171 conmon[41853]: debug 2022-01-31T22:10:11.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:11.352781+0000) 2022-01-31T22:10:11.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:11 smithi167 conmon[49112]: debug 2022-01-31T22:10:11.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:11.729321+0000) 2022-01-31T22:10:11.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:11 smithi167 conmon[54076]: debug 2022-01-31T22:10:11.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:11.646725+0000) 2022-01-31T22:10:11.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:11 smithi167 conmon[60316]: debug 2022-01-31T22:10:11.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:11.646694+0000) 2022-01-31T22:10:12.309 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:11 smithi171 conmon[51620]: debug 2022-01-31T22:10:11.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:11.999694+0000) 2022-01-31T22:10:12.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:12 smithi171 conmon[46715]: debug 2022-01-31T22:10:12.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:12.146040+0000) 2022-01-31T22:10:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:12 smithi171 conmon[41853]: debug 2022-01-31T22:10:12.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:12 smithi171 conmon[41853]: 2022-01-31T22:10:12.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:12.352941+0000) 2022-01-31T22:10:12.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:12 smithi167 conmon[54076]: debug 2022-01-31T22:10:12.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:12.646927+0000) 2022-01-31T22:10:12.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:12 smithi167 conmon[60316]: debug 2022-01-31T22:10:12.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:12.646913+0000) 2022-01-31T22:10:12.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:12 smithi167 conmon[49112]: debug 2022-01-31T22:10:12.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:12.729541+0000) 2022-01-31T22:10:13.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:13 smithi171 conmon[46715]: debug 2022-01-31T22:10:13.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:13.146171+0000) 2022-01-31T22:10:13.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:13 smithi171 conmon[51620]: debug 2022-01-31T22:10:12.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:12.999893+0000) 2022-01-31T22:10:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:13 smithi171 conmon[41853]: debug 2022-01-31T22:10:13.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:13.353080+0000) 2022-01-31T22:10:13.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:13 smithi167 conmon[54076]: debug 2022-01-31T22:10:13.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:13.647131+0000) 2022-01-31T22:10:13.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:13 smithi167 conmon[60316]: debug 2022-01-31T22:10:13.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:13.647132+0000) 2022-01-31T22:10:13.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:13 smithi167 conmon[49112]: debug 2022-01-31T22:10:13.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:13.729700+0000) 2022-01-31T22:10:14.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:14 smithi171 conmon[46715]: debug 2022-01-31T22:10:14.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:14.146332+0000) 2022-01-31T22:10:14.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:14 smithi171 conmon[51620]: debug 2022-01-31T22:10:13.999+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:14.000101+0000) 2022-01-31T22:10:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:14 smithi171 conmon[41853]: debug 2022-01-31T22:10:14.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:14.353270+0000) 2022-01-31T22:10:14.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:14 smithi167 conmon[54076]: debug 2022-01-31T22:10:14.645+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:14.647326+0000) 2022-01-31T22:10:14.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:14 smithi167 conmon[60316]: debug 2022-01-31T22:10:14.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:14.647327+0000) 2022-01-31T22:10:14.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:14 smithi167 conmon[49112]: debug 2022-01-31T22:10:14.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:14.729831+0000) 2022-01-31T22:10:15.310 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[35325]: debug 2022-01-31T22:10:15.186+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188044 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:15.311 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[41853]: debug 2022-01-31T22:10:15.165+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.166692+0000) 2022-01-31T22:10:15.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[46715]: debug 2022-01-31T22:10:15.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.146487+0000) 2022-01-31T22:10:15.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[46715]: debug 2022-01-31T22:10:15.166+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.167632+0000) 2022-01-31T22:10:15.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[51620]: debug 2022-01-31T22:10:15.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.000290+0000) 2022-01-31T22:10:15.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[51620]: debug 2022-01-31T22:10:15.168+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.168852+0000) 2022-01-31T22:10:15.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:15 smithi167 conmon[49112]: debug 2022-01-31T22:10:15.166+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.168178+0000) 2022-01-31T22:10:15.412 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:15 smithi167 conmon[54076]: debug 2022-01-31T22:10:15.165+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.167296+0000) 2022-01-31T22:10:15.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:15 smithi167 conmon[60316]: debug 2022-01-31T22:10:15.167+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.168935+0000) 2022-01-31T22:10:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:15 smithi171 conmon[41853]: debug 2022-01-31T22:10:15.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.353395+0000) 2022-01-31T22:10:15.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:15 smithi167 conmon[54076]: debug 2022-01-31T22:10:15.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.647540+0000) 2022-01-31T22:10:15.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:15 smithi167 conmon[60316]: debug 2022-01-31T22:10:15.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.647539+0000) 2022-01-31T22:10:15.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:15 smithi167 conmon[49112]: debug 2022-01-31T22:10:15.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:15.729988+0000) 2022-01-31T22:10:16.310 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:16 smithi171 conmon[46715]: debug 2022-01-31T22:10:16.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:16.146684+0000) 2022-01-31T22:10:16.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:16 smithi171 conmon[51620]: debug 2022-01-31T22:10:16.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:16.000459+0000) 2022-01-31T22:10:16.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:16 smithi171 conmon[41853]: debug 2022-01-31T22:10:16.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:16.353525+0000) 2022-01-31T22:10:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:16 smithi167 conmon[49112]: debug 2022-01-31T22:10:16.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:16.730143+0000) 2022-01-31T22:10:16.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:16 smithi167 conmon[54076]: debug 2022-01-31T22:10:16.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:16.647690+0000) 2022-01-31T22:10:16.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:16 smithi167 conmon[60316]: debug 2022-01-31T22:10:16.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:16.647683+0000) 2022-01-31T22:10:17.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:17 smithi171 conmon[51620]: debug 2022-01-31T22:10:17.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:17.000618+0000) 2022-01-31T22:10:17.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:17 smithi171 conmon[46715]: debug 2022-01-31T22:10:17.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:17.146835+0000) 2022-01-31T22:10:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:17 smithi171 conmon[41853]: debug 2022-01-31T22:10:17.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:17.353702+0000) 2022-01-31T22:10:17.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:17 smithi167 conmon[54076]: debug 2022-01-31T22:10:17.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:17.647885+0000) 2022-01-31T22:10:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:17 smithi167 conmon[60316]: debug 2022-01-31T22:10:17.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:17.647878+0000) 2022-01-31T22:10:17.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:17 smithi167 conmon[49112]: debug 2022-01-31T22:10:17.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:17.730243+0000) 2022-01-31T22:10:18.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:18 smithi171 conmon[46715]: debug 2022-01-31T22:10:18.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:18.147017+0000) 2022-01-31T22:10:18.311 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:18 smithi171 conmon[51620]: debug 2022-01-31T22:10:18.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:18.000793+0000) 2022-01-31T22:10:18.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:18 smithi171 conmon[41853]: debug 2022-01-31T22:10:18.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:18.353858+0000) 2022-01-31T22:10:18.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:18 smithi167 conmon[54076]: debug 2022-01-31T22:10:18.647+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:18.648072+0000) 2022-01-31T22:10:18.698 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:18 smithi167 conmon[60316]: debug 2022-01-31T22:10:18.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:18.648071+0000) 2022-01-31T22:10:18.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:18 smithi167 conmon[49112]: debug 2022-01-31T22:10:18.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:18.730429+0000) 2022-01-31T22:10:19.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:19 smithi171 conmon[46715]: debug 2022-01-31T22:10:19.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:19.147217+0000) 2022-01-31T22:10:19.312 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:19 smithi171 conmon[51620]: debug 2022-01-31T22:10:19.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:19.000980+0000) 2022-01-31T22:10:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:19 smithi171 conmon[41853]: debug 2022-01-31T22:10:19.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:19.354009+0000) 2022-01-31T22:10:19.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:19 smithi167 conmon[54076]: debug 2022-01-31T22:10:19.646+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:19.648341+0000) 2022-01-31T22:10:19.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:19 smithi167 conmon[60316]: debug 2022-01-31T22:10:19.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:19.648266+0000) 2022-01-31T22:10:19.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:19 smithi167 conmon[49112]: debug 2022-01-31T22:10:19.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:19.730525+0000) 2022-01-31T22:10:20.311 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[35325]: debug 2022-01-31T22:10:20.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188157 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:20.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[41853]: debug 2022-01-31T22:10:20.189+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.190143+0000) 2022-01-31T22:10:20.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[46715]: debug 2022-01-31T22:10:20.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.147397+0000) 2022-01-31T22:10:20.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[46715]: debug 2022-01-31T22:10:20.188+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.189382+0000) 2022-01-31T22:10:20.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[51620]: debug 2022-01-31T22:10:20.000+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.001163+0000) 2022-01-31T22:10:20.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[51620]: debug 2022-01-31T22:10:20.188+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.189009+0000) 2022-01-31T22:10:20.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:20 smithi167 conmon[49112]: debug 2022-01-31T22:10:20.188+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.190445+0000) 2022-01-31T22:10:20.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:20 smithi167 conmon[54076]: debug 2022-01-31T22:10:20.188+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.189633+0000) 2022-01-31T22:10:20.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:20 smithi167 conmon[60316]: debug 2022-01-31T22:10:20.188+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.190126+0000) 2022-01-31T22:10:20.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:20 smithi171 conmon[41853]: debug 2022-01-31T22:10:20.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.354192+0000) 2022-01-31T22:10:20.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:20 smithi167 conmon[54076]: debug 2022-01-31T22:10:20.647+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.648494+0000) 2022-01-31T22:10:20.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:20 smithi167 conmon[60316]: debug 2022-01-31T22:10:20.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.648425+0000) 2022-01-31T22:10:20.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:20 smithi167 conmon[49112]: debug 2022-01-31T22:10:20.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:20.730693+0000) 2022-01-31T22:10:21.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:21 smithi171 conmon[46715]: debug 2022-01-31T22:10:21.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:21.147559+0000) 2022-01-31T22:10:21.312 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:21 smithi171 conmon[51620]: debug 2022-01-31T22:10:21.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:21.001379+0000) 2022-01-31T22:10:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:21 smithi171 conmon[41853]: debug 2022-01-31T22:10:21.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:21.354343+0000) 2022-01-31T22:10:21.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:21 smithi167 conmon[54076]: debug 2022-01-31T22:10:21.647+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:21.648610+0000) 2022-01-31T22:10:21.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:21 smithi167 conmon[60316]: debug 2022-01-31T22:10:21.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:21.648707+0000) 2022-01-31T22:10:21.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:21 smithi167 conmon[49112]: debug 2022-01-31T22:10:21.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:21.730830+0000) 2022-01-31T22:10:22.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:22 smithi171 conmon[46715]: debug 2022-01-31T22:10:22.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:22.147703+0000) 2022-01-31T22:10:22.312 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:22 smithi171 conmon[51620]: debug 2022-01-31T22:10:22.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:22.001566+0000) 2022-01-31T22:10:22.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:22 smithi171 conmon[41853]: debug 2022-01-31T22:10:22.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:22.354549+0000) 2022-01-31T22:10:22.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:22 smithi167 conmon[54076]: debug 2022-01-31T22:10:22.647+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:22.648809+0000) 2022-01-31T22:10:22.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:22 smithi167 conmon[60316]: debug 2022-01-31T22:10:22.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:22.648906+0000) 2022-01-31T22:10:22.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:22 smithi167 conmon[49112]: debug 2022-01-31T22:10:22.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:22.730970+0000) 2022-01-31T22:10:23.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:23 smithi171 conmon[46715]: debug 2022-01-31T22:10:23.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:23.147855+0000) 2022-01-31T22:10:23.312 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:23 smithi171 conmon[51620]: debug 2022-01-31T22:10:23.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:23.001776+0000) 2022-01-31T22:10:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:23 smithi171 conmon[41853]: debug 2022-01-31T22:10:23.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:23.354744+0000) 2022-01-31T22:10:23.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:23 smithi167 conmon[54076]: debug 2022-01-31T22:10:23.647+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:23.649029+0000) 2022-01-31T22:10:23.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:23 smithi167 conmon[60316]: debug 2022-01-31T22:10:23.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:23.649095+0000) 2022-01-31T22:10:23.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:23 smithi167 conmon[49112]: debug 2022-01-31T22:10:23.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:23.731149+0000) 2022-01-31T22:10:24.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:24 smithi171 conmon[51620]: debug 2022-01-31T22:10:24.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:24.001987+0000) 2022-01-31T22:10:24.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:23 smithi171 conmon[35325]: debug 2022-01-31T22:10:23.808+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:10:24.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:24 smithi171 conmon[46715]: debug 2022-01-31T22:10:24.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:24.148047+0000) 2022-01-31T22:10:24.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:24 smithi171 conmon[41853]: debug 2022-01-31T22:10:24.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:24.354939+0000) 2022-01-31T22:10:24.699 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:24 smithi167 conmon[54076]: debug 2022-01-31T22:10:24.648+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:24.649245+0000) 2022-01-31T22:10:24.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:24 smithi167 conmon[60316]: debug 2022-01-31T22:10:24.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:24.649290+0000) 2022-01-31T22:10:24.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:24 smithi167 conmon[49112]: debug 2022-01-31T22:10:24.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:24.731335+0000) 2022-01-31T22:10:25.312 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[35325]: debug 2022-01-31T22:10:25.231+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188268 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:25.313 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[41853]: debug 2022-01-31T22:10:25.211+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.211795+0000) 2022-01-31T22:10:25.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[51620]: debug 2022-01-31T22:10:25.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.002192+0000) 2022-01-31T22:10:25.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[51620]: debug 2022-01-31T22:10:25.210+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.211315+0000) 2022-01-31T22:10:25.314 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[46715]: debug 2022-01-31T22:10:25.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.148230+0000) 2022-01-31T22:10:25.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[46715]: debug 2022-01-31T22:10:25.210+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.211471+0000) 2022-01-31T22:10:25.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:25 smithi167 conmon[49112]: debug 2022-01-31T22:10:25.210+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.212356+0000) 2022-01-31T22:10:25.424 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:25 smithi167 conmon[54076]: debug 2022-01-31T22:10:25.211+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.212809+0000) 2022-01-31T22:10:25.425 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:25 smithi167 conmon[60316]: debug 2022-01-31T22:10:25.210+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.212207+0000) 2022-01-31T22:10:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:25 smithi171 conmon[41853]: debug 2022-01-31T22:10:25.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.355111+0000) 2022-01-31T22:10:25.692 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:25 smithi167 conmon[54076]: debug 2022-01-31T22:10:25.648+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.649449+0000) 2022-01-31T22:10:25.692 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:25 smithi167 conmon[60316]: debug 2022-01-31T22:10:25.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.649493+0000) 2022-01-31T22:10:25.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:25 smithi167 conmon[49112]: debug 2022-01-31T22:10:25.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:25.731555+0000) 2022-01-31T22:10:26.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:26 smithi171 conmon[46715]: debug 2022-01-31T22:10:26.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:26.148367+0000) 2022-01-31T22:10:26.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:26 smithi171 conmon[51620]: debug 2022-01-31T22:10:26.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:26.002403+0000) 2022-01-31T22:10:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:26 smithi171 conmon[41853]: debug 2022-01-31T22:10:26.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:26.355278+0000) 2022-01-31T22:10:26.699 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:26 smithi167 conmon[54076]: debug 2022-01-31T22:10:26.648+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:26.649599+0000) 2022-01-31T22:10:26.700 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:26 smithi167 conmon[60316]: debug 2022-01-31T22:10:26.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:26.649654+0000) 2022-01-31T22:10:26.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:26 smithi167 conmon[49112]: debug 2022-01-31T22:10:26.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:26.731667+0000) 2022-01-31T22:10:27.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:27 smithi171 conmon[46715]: debug 2022-01-31T22:10:27.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:27.148525+0000) 2022-01-31T22:10:27.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:27 smithi171 conmon[51620]: debug 2022-01-31T22:10:27.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:27.002573+0000) 2022-01-31T22:10:27.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:27 smithi171 conmon[41853]: debug 2022-01-31T22:10:27.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:27.355461+0000) 2022-01-31T22:10:27.700 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:27 smithi167 conmon[60316]: debug 2022-01-31T22:10:27.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:27.649809+0000) 2022-01-31T22:10:27.700 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:27 smithi167 conmon[54076]: debug 2022-01-31T22:10:27.649+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:27.649791+0000) 2022-01-31T22:10:27.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:27 smithi167 conmon[49112]: debug 2022-01-31T22:10:27.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:27.731848+0000) 2022-01-31T22:10:28.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:28 smithi171 conmon[46715]: debug 2022-01-31T22:10:28.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:28.148710+0000) 2022-01-31T22:10:28.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:28 smithi171 conmon[51620]: debug 2022-01-31T22:10:28.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:28.002777+0000) 2022-01-31T22:10:28.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:28 smithi171 conmon[41853]: debug 2022-01-31T22:10:28.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:28.355646+0000) 2022-01-31T22:10:28.699 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:28 smithi167 conmon[54076]: debug 2022-01-31T22:10:28.649+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:28.649977+0000) 2022-01-31T22:10:28.700 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:28 smithi167 conmon[60316]: debug 2022-01-31T22:10:28.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:28.649970+0000) 2022-01-31T22:10:28.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:28 smithi167 conmon[49112]: debug 2022-01-31T22:10:28.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:28.732080+0000) 2022-01-31T22:10:29.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:29 smithi171 conmon[46715]: debug 2022-01-31T22:10:29.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:29.148923+0000) 2022-01-31T22:10:29.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:29 smithi171 conmon[51620]: debug 2022-01-31T22:10:29.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:29.002937+0000) 2022-01-31T22:10:29.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:29 smithi171 conmon[41853]: debug 2022-01-31T22:10:29.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:29.355844+0000) 2022-01-31T22:10:29.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:29 smithi167 conmon[54076]: debug 2022-01-31T22:10:29.649+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:29.650202+0000) 2022-01-31T22:10:29.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:29 smithi167 conmon[60316]: debug 2022-01-31T22:10:29.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:29.650169+0000) 2022-01-31T22:10:29.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:29 smithi167 conmon[49112]: debug 2022-01-31T22:10:29.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:29.732276+0000) 2022-01-31T22:10:30.253 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[41853]: debug 2022-01-31T22:10:30.232+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.233847+0000) 2022-01-31T22:10:30.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[46715]: debug 2022-01-31T22:10:30.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.149135+0000) 2022-01-31T22:10:30.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[46715]: debug 2022-01-31T22:10:30.232+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.234140+0000) 2022-01-31T22:10:30.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[51620]: debug 2022-01-31T22:10:30.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.003130+0000) 2022-01-31T22:10:30.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[51620]: debug 2022-01-31T22:10:30.232+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.234500+0000) 2022-01-31T22:10:30.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:30 smithi167 conmon[49112]: debug 2022-01-31T22:10:30.233+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.235151+0000) 2022-01-31T22:10:30.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:30 smithi167 conmon[54076]: debug 2022-01-31T22:10:30.232+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.234424+0000) 2022-01-31T22:10:30.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:30 smithi167 conmon[60316]: debug 2022-01-31T22:10:30.234+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.235579+0000) 2022-01-31T22:10:30.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[35325]: debug 2022-01-31T22:10:30.252+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188380 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:30.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:30 smithi171 conmon[41853]: debug 2022-01-31T22:10:30.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.355943+0000) 2022-01-31T22:10:30.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:30 smithi167 conmon[60316]: debug 2022-01-31T22:10:30.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.650403+0000) 2022-01-31T22:10:30.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:30 smithi167 conmon[54076]: debug 2022-01-31T22:10:30.648+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.650411+0000) 2022-01-31T22:10:30.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:30 smithi167 conmon[49112]: debug 2022-01-31T22:10:30.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:30.732418+0000) 2022-01-31T22:10:31.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:31 smithi171 conmon[46715]: debug 2022-01-31T22:10:31.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:31.149287+0000) 2022-01-31T22:10:31.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:31 smithi171 conmon[51620]: debug 2022-01-31T22:10:31.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:31.003353+0000) 2022-01-31T22:10:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:31 smithi171 conmon[41853]: debug 2022-01-31T22:10:31.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:31.356080+0000) 2022-01-31T22:10:31.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:31 smithi167 conmon[60316]: debug 2022-01-31T22:10:31.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:31.650611+0000) 2022-01-31T22:10:31.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:31 smithi167 conmon[54076]: debug 2022-01-31T22:10:31.649+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:31.650566+0000) 2022-01-31T22:10:31.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:31 smithi167 conmon[49112]: debug 2022-01-31T22:10:31.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:31.732617+0000) 2022-01-31T22:10:32.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:32 smithi171 conmon[46715]: debug 2022-01-31T22:10:32.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:32.149453+0000) 2022-01-31T22:10:32.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:32 smithi171 conmon[51620]: debug 2022-01-31T22:10:32.001+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:32.003511+0000) 2022-01-31T22:10:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:32 smithi171 conmon[41853]: debug 2022-01-31T22:10:32.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:32.356231+0000) 2022-01-31T22:10:32.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:32 smithi167 conmon[60316]: debug 2022-01-31T22:10:32.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:32.650788+0000) 2022-01-31T22:10:32.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:32 smithi167 conmon[54076]: debug 2022-01-31T22:10:32.650+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:32.650754+0000) 2022-01-31T22:10:32.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:32 smithi167 conmon[49112]: debug 2022-01-31T22:10:32.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:32.732784+0000) 2022-01-31T22:10:33.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:33 smithi171 conmon[46715]: debug 2022-01-31T22:10:33.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:33.149604+0000) 2022-01-31T22:10:33.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:33 smithi171 conmon[51620]: debug 2022-01-31T22:10:33.002+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:33.003703+0000) 2022-01-31T22:10:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:33 smithi171 conmon[41853]: debug 2022-01-31T22:10:33.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:33.356399+0000) 2022-01-31T22:10:33.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:33 smithi167 conmon[60316]: debug 2022-01-31T22:10:33.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:33.651003+0000) 2022-01-31T22:10:33.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:33 smithi167 conmon[54076]: debug 2022-01-31T22:10:33.650+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:33.650950+0000) 2022-01-31T22:10:33.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:33 smithi167 conmon[49112]: debug 2022-01-31T22:10:33.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:33.732966+0000) 2022-01-31T22:10:34.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:34 smithi171 conmon[46715]: debug 2022-01-31T22:10:34.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:34.149800+0000) 2022-01-31T22:10:34.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:34 smithi171 conmon[51620]: debug 2022-01-31T22:10:34.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:34.003925+0000) 2022-01-31T22:10:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:34 smithi171 conmon[41853]: debug 2022-01-31T22:10:34.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:34.356550+0000) 2022-01-31T22:10:34.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:34 smithi167 conmon[54076]: debug 2022-01-31T22:10:34.649+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:34.651184+0000) 2022-01-31T22:10:34.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:34 smithi167 conmon[60316]: debug 2022-01-31T22:10:34.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:34.651185+0000) 2022-01-31T22:10:34.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:34 smithi167 conmon[49112]: debug 2022-01-31T22:10:34.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:34.733152+0000) 2022-01-31T22:10:35.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[46715]: debug 2022-01-31T22:10:35.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.149953+0000) 2022-01-31T22:10:35.256 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[51620]: debug 2022-01-31T22:10:35.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.004130+0000) 2022-01-31T22:10:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[35325]: debug 2022-01-31T22:10:35.274+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188506 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[46715]: debug 2022-01-31T22:10:35.254+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.255857+0000) 2022-01-31T22:10:35.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[51620]: debug 2022-01-31T22:10:35.254+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.256457+0000) 2022-01-31T22:10:35.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[41853]: debug 2022-01-31T22:10:35.254+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.256125+0000) 2022-01-31T22:10:35.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:35 smithi171 conmon[41853]: debug 2022-01-31T22:10:35.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.356720+0000) 2022-01-31T22:10:35.628 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:35 smithi167 conmon[49112]: debug 2022-01-31T22:10:35.255+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.257309+0000) 2022-01-31T22:10:35.629 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:35 smithi167 conmon[54076]: debug 2022-01-31T22:10:35.255+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.256541+0000) 2022-01-31T22:10:35.629 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:35 smithi167 conmon[60316]: debug 2022-01-31T22:10:35.256+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.257660+0000) 2022-01-31T22:10:35.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:35 smithi167 conmon[60316]: debug 2022-01-31T22:10:35.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.651347+0000) 2022-01-31T22:10:35.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:35 smithi167 conmon[49112]: debug 2022-01-31T22:10:35.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.733333+0000) 2022-01-31T22:10:35.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:35 smithi167 conmon[54076]: debug 2022-01-31T22:10:35.650+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:35.651377+0000) 2022-01-31T22:10:36.314 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:36 smithi171 conmon[46715]: debug 2022-01-31T22:10:36.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:36.150102+0000) 2022-01-31T22:10:36.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:36 smithi171 conmon[51620]: debug 2022-01-31T22:10:36.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:36.004294+0000) 2022-01-31T22:10:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:36 smithi171 conmon[41853]: debug 2022-01-31T22:10:36.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:36.356872+0000) 2022-01-31T22:10:36.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:36 smithi167 conmon[54076]: debug 2022-01-31T22:10:36.650+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:36.651536+0000) 2022-01-31T22:10:36.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:36 smithi167 conmon[60316]: debug 2022-01-31T22:10:36.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:36.651507+0000) 2022-01-31T22:10:36.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:36 smithi167 conmon[49112]: debug 2022-01-31T22:10:36.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:36.733464+0000) 2022-01-31T22:10:37.314 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:37 smithi171 conmon[46715]: debug 2022-01-31T22:10:37.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:37.150233+0000) 2022-01-31T22:10:37.315 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:37 smithi171 conmon[51620]: debug 2022-01-31T22:10:37.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:37.004468+0000) 2022-01-31T22:10:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:37 smithi171 conmon[41853]: debug 2022-01-31T22:10:37.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:37.357056+0000) 2022-01-31T22:10:37.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:37 smithi167 conmon[54076]: debug 2022-01-31T22:10:37.651+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:37.651730+0000) 2022-01-31T22:10:37.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:37 smithi167 conmon[60316]: debug 2022-01-31T22:10:37.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:37.651697+0000) 2022-01-31T22:10:37.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:37 smithi167 conmon[49112]: debug 2022-01-31T22:10:37.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:37.733670+0000) 2022-01-31T22:10:38.314 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:38 smithi171 conmon[46715]: debug 2022-01-31T22:10:38.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:38.150420+0000) 2022-01-31T22:10:38.315 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:38 smithi171 conmon[51620]: debug 2022-01-31T22:10:38.003+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:38.004660+0000) 2022-01-31T22:10:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:38 smithi171 conmon[41853]: debug 2022-01-31T22:10:38.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:38.357253+0000) 2022-01-31T22:10:38.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:38 smithi167 conmon[54076]: debug 2022-01-31T22:10:38.651+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:38.651887+0000) 2022-01-31T22:10:38.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:38 smithi167 conmon[60316]: debug 2022-01-31T22:10:38.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:38.651886+0000) 2022-01-31T22:10:38.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:38 smithi167 conmon[49112]: debug 2022-01-31T22:10:38.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:38.733799+0000) 2022-01-31T22:10:39.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:38 smithi171 conmon[35325]: debug 2022-01-31T22:10:38.808+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:10:39.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:39 smithi171 conmon[51620]: debug 2022-01-31T22:10:39.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:39.004862+0000) 2022-01-31T22:10:39.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:39 smithi171 conmon[46715]: debug 2022-01-31T22:10:39.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:39.150595+0000) 2022-01-31T22:10:39.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:39 smithi167 conmon[54076]: debug 2022-01-31T22:10:39.651+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:39.652076+0000) 2022-01-31T22:10:39.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:39 smithi167 conmon[60316]: debug 2022-01-31T22:10:39.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:39.652089+0000) 2022-01-31T22:10:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:39 smithi171 conmon[41853]: debug 2022-01-31T22:10:39.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:39.357408+0000) 2022-01-31T22:10:39.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:39 smithi167 conmon[49112]: debug 2022-01-31T22:10:39.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:39.733927+0000) 2022-01-31T22:10:40.277 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[46715]: debug 2022-01-31T22:10:40.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.150789+0000) 2022-01-31T22:10:40.278 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[51620]: debug 2022-01-31T22:10:40.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.005057+0000) 2022-01-31T22:10:40.278 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[51620]: debug 2022-01-31T22:10:40.277+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.278274+0000) 2022-01-31T22:10:40.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[35325]: debug 2022-01-31T22:10:40.303+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188617 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:40.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[46715]: debug 2022-01-31T22:10:40.277+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.278475+0000) 2022-01-31T22:10:40.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[41853]: debug 2022-01-31T22:10:40.278+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.279328+0000) 2022-01-31T22:10:40.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:40 smithi171 conmon[41853]: debug 2022-01-31T22:10:40.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.357627+0000) 2022-01-31T22:10:40.628 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:40 smithi167 conmon[49112]: debug 2022-01-31T22:10:40.278+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.278897+0000) 2022-01-31T22:10:40.629 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:40 smithi167 conmon[54076]: debug 2022-01-31T22:10:40.278+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.278524+0000) 2022-01-31T22:10:40.630 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:40 smithi167 conmon[60316]: debug 2022-01-31T22:10:40.285+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.285676+0000) 2022-01-31T22:10:40.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:40 smithi167 conmon[49112]: debug 2022-01-31T22:10:40.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.734104+0000) 2022-01-31T22:10:40.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:40 smithi167 conmon[54076]: debug 2022-01-31T22:10:40.651+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.652284+0000) 2022-01-31T22:10:40.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:40 smithi167 conmon[60316]: debug 2022-01-31T22:10:40.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:40.652304+0000) 2022-01-31T22:10:41.314 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:41 smithi171 conmon[46715]: debug 2022-01-31T22:10:41.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:41.150972+0000) 2022-01-31T22:10:41.315 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:41 smithi171 conmon[51620]: debug 2022-01-31T22:10:41.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:41.005237+0000) 2022-01-31T22:10:41.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:41 smithi171 conmon[41853]: debug 2022-01-31T22:10:41.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:41.357793+0000) 2022-01-31T22:10:41.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:41 smithi167 conmon[54076]: debug 2022-01-31T22:10:41.652+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:41.652466+0000) 2022-01-31T22:10:41.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:41 smithi167 conmon[60316]: debug 2022-01-31T22:10:41.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:41.652462+0000) 2022-01-31T22:10:41.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:41 smithi167 conmon[49112]: debug 2022-01-31T22:10:41.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:41.734283+0000) 2022-01-31T22:10:42.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:42 smithi171 conmon[46715]: debug 2022-01-31T22:10:42.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:42.151132+0000) 2022-01-31T22:10:42.315 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:42 smithi171 conmon[51620]: debug 2022-01-31T22:10:42.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:42.005434+0000) 2022-01-31T22:10:42.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:42 smithi171 conmon[41853]: debug 2022-01-31T22:10:42.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:42.357976+0000) 2022-01-31T22:10:42.733 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:42 smithi167 conmon[54076]: debug 2022-01-31T22:10:42.652+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:42.652672+0000) 2022-01-31T22:10:42.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:42 smithi167 conmon[60316]: debug 2022-01-31T22:10:42.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:42.652672+0000) 2022-01-31T22:10:42.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:42 smithi167 conmon[49112]: debug 2022-01-31T22:10:42.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:42.734539+0000) 2022-01-31T22:10:43.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:43 smithi171 conmon[46715]: debug 2022-01-31T22:10:43.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:43.151281+0000) 2022-01-31T22:10:43.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:43 smithi171 conmon[51620]: debug 2022-01-31T22:10:43.004+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:43.005613+0000) 2022-01-31T22:10:43.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:43 smithi171 conmon[41853]: debug 2022-01-31T22:10:43.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:43.358148+0000) 2022-01-31T22:10:43.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:43 smithi167 conmon[54076]: debug 2022-01-31T22:10:43.652+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:43.652869+0000) 2022-01-31T22:10:43.734 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:43 smithi167 conmon[60316]: debug 2022-01-31T22:10:43.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:43.652866+0000) 2022-01-31T22:10:43.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:43 smithi167 conmon[49112]: debug 2022-01-31T22:10:43.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:43.734679+0000) 2022-01-31T22:10:44.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:44 smithi171 conmon[46715]: debug 2022-01-31T22:10:44.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:44.151495+0000) 2022-01-31T22:10:44.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:44 smithi171 conmon[51620]: debug 2022-01-31T22:10:44.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:44.005834+0000) 2022-01-31T22:10:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:44 smithi171 conmon[41853]: debug 2022-01-31T22:10:44.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:44.358326+0000) 2022-01-31T22:10:44.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:44 smithi167 conmon[54076]: debug 2022-01-31T22:10:44.652+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:44.653077+0000) 2022-01-31T22:10:44.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:44 smithi167 conmon[60316]: debug 2022-01-31T22:10:44.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:44.653057+0000) 2022-01-31T22:10:44.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:44 smithi167 conmon[49112]: debug 2022-01-31T22:10:44.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:44.734838+0000) 2022-01-31T22:10:45.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[51620]: debug 2022-01-31T22:10:45.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.006014+0000) 2022-01-31T22:10:45.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[46715]: debug 2022-01-31T22:10:45.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.151675+0000) 2022-01-31T22:10:45.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[46715]: debug 2022-01-31T22:10:45.305+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.306035+0000) 2022-01-31T22:10:45.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[35325]: debug 2022-01-31T22:10:45.325+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188729 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:45.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[51620]: debug 2022-01-31T22:10:45.306+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.306993+0000) 2022-01-31T22:10:45.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[41853]: debug 2022-01-31T22:10:45.306+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.307255+0000) 2022-01-31T22:10:45.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:45 smithi171 conmon[41853]: debug 2022-01-31T22:10:45.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.358490+0000) 2022-01-31T22:10:45.629 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:45 smithi167 conmon[49112]: debug 2022-01-31T22:10:45.307+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.307866+0000) 2022-01-31T22:10:45.630 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:45 smithi167 conmon[54076]: debug 2022-01-31T22:10:45.306+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.307457+0000) 2022-01-31T22:10:45.631 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:45 smithi167 conmon[60316]: debug 2022-01-31T22:10:45.308+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.309184+0000) 2022-01-31T22:10:45.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:45 smithi167 conmon[49112]: debug 2022-01-31T22:10:45.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.734958+0000) 2022-01-31T22:10:45.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:45 smithi167 conmon[54076]: debug 2022-01-31T22:10:45.652+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.653269+0000) 2022-01-31T22:10:45.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:45 smithi167 conmon[60316]: debug 2022-01-31T22:10:45.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:45.653246+0000) 2022-01-31T22:10:46.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:46 smithi171 conmon[46715]: debug 2022-01-31T22:10:46.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:46.151820+0000) 2022-01-31T22:10:46.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:46 smithi171 conmon[51620]: debug 2022-01-31T22:10:46.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:46.006211+0000) 2022-01-31T22:10:46.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:46 smithi171 conmon[41853]: debug 2022-01-31T22:10:46.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:46.358602+0000) 2022-01-31T22:10:46.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:46 smithi167 conmon[60316]: debug 2022-01-31T22:10:46.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:46.653443+0000) 2022-01-31T22:10:46.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:46 smithi167 conmon[49112]: debug 2022-01-31T22:10:46.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:46.735126+0000) 2022-01-31T22:10:46.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:46 smithi167 conmon[54076]: debug 2022-01-31T22:10:46.653+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:46.653445+0000) 2022-01-31T22:10:47.316 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:47 smithi171 conmon[46715]: debug 2022-01-31T22:10:47.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:47.151992+0000) 2022-01-31T22:10:47.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:47 smithi171 conmon[51620]: debug 2022-01-31T22:10:47.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:47.006377+0000) 2022-01-31T22:10:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:47 smithi171 conmon[41853]: debug 2022-01-31T22:10:47.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:47.358768+0000) 2022-01-31T22:10:47.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:47 smithi167 conmon[54076]: debug 2022-01-31T22:10:47.653+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:47.653671+0000) 2022-01-31T22:10:47.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:47 smithi167 conmon[60316]: debug 2022-01-31T22:10:47.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:47.653663+0000) 2022-01-31T22:10:47.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:47 smithi167 conmon[49112]: debug 2022-01-31T22:10:47.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:47.735341+0000) 2022-01-31T22:10:48.316 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:48 smithi171 conmon[46715]: debug 2022-01-31T22:10:48.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:48.152182+0000) 2022-01-31T22:10:48.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:48 smithi171 conmon[51620]: debug 2022-01-31T22:10:48.005+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:48.006611+0000) 2022-01-31T22:10:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:48 smithi171 conmon[41853]: debug 2022-01-31T22:10:48.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:48.358944+0000) 2022-01-31T22:10:48.734 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:48 smithi167 conmon[54076]: debug 2022-01-31T22:10:48.653+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:48.653866+0000) 2022-01-31T22:10:48.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:48 smithi167 conmon[60316]: debug 2022-01-31T22:10:48.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:48.653857+0000) 2022-01-31T22:10:48.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:48 smithi167 conmon[49112]: debug 2022-01-31T22:10:48.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:48.735562+0000) 2022-01-31T22:10:49.316 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:49 smithi171 conmon[46715]: debug 2022-01-31T22:10:49.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:49.152326+0000) 2022-01-31T22:10:49.317 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:49 smithi171 conmon[51620]: debug 2022-01-31T22:10:49.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:49.006793+0000) 2022-01-31T22:10:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:49 smithi171 conmon[41853]: debug 2022-01-31T22:10:49.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:49.359113+0000) 2022-01-31T22:10:49.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:49 smithi167 conmon[54076]: debug 2022-01-31T22:10:49.653+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:49.654029+0000) 2022-01-31T22:10:49.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:49 smithi167 conmon[60316]: debug 2022-01-31T22:10:49.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:49.654031+0000) 2022-01-31T22:10:49.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:49 smithi167 conmon[49112]: debug 2022-01-31T22:10:49.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:49.735716+0000) 2022-01-31T22:10:50.316 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[46715]: debug 2022-01-31T22:10:50.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.152493+0000) 2022-01-31T22:10:50.317 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[51620]: debug 2022-01-31T22:10:50.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.007002+0000) 2022-01-31T22:10:50.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[46715]: debug 2022-01-31T22:10:50.327+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.328287+0000) 2022-01-31T22:10:50.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[51620]: debug 2022-01-31T22:10:50.327+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.328475+0000) 2022-01-31T22:10:50.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[35325]: debug 2022-01-31T22:10:50.347+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188840 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:50.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[41853]: debug 2022-01-31T22:10:50.327+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.328577+0000) 2022-01-31T22:10:50.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:50 smithi171 conmon[41853]: debug 2022-01-31T22:10:50.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.359241+0000) 2022-01-31T22:10:50.630 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:50 smithi167 conmon[49112]: debug 2022-01-31T22:10:50.328+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.329205+0000) 2022-01-31T22:10:50.631 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:50 smithi167 conmon[60316]: debug 2022-01-31T22:10:50.329+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.329664+0000) 2022-01-31T22:10:50.632 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:50 smithi167 conmon[54076]: debug 2022-01-31T22:10:50.329+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.329487+0000) 2022-01-31T22:10:50.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:50 smithi167 conmon[60316]: debug 2022-01-31T22:10:50.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.654218+0000) 2022-01-31T22:10:50.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:50 smithi167 conmon[49112]: debug 2022-01-31T22:10:50.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.735886+0000) 2022-01-31T22:10:50.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:50 smithi167 conmon[54076]: debug 2022-01-31T22:10:50.653+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:50.654217+0000) 2022-01-31T22:10:51.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:51 smithi171 conmon[51620]: debug 2022-01-31T22:10:51.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:51.007190+0000) 2022-01-31T22:10:51.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:51 smithi171 conmon[46715]: debug 2022-01-31T22:10:51.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:51.152703+0000) 2022-01-31T22:10:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:51 smithi171 conmon[41853]: debug 2022-01-31T22:10:51.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:51.359403+0000) 2022-01-31T22:10:51.735 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:51 smithi167 conmon[60316]: debug 2022-01-31T22:10:51.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:51.654408+0000) 2022-01-31T22:10:51.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:51 smithi167 conmon[54076]: debug 2022-01-31T22:10:51.654+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:51.654373+0000) 2022-01-31T22:10:51.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:51 smithi167 conmon[49112]: debug 2022-01-31T22:10:51.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:51.736019+0000) 2022-01-31T22:10:52.316 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:52 smithi171 conmon[46715]: debug 2022-01-31T22:10:52.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:52.152848+0000) 2022-01-31T22:10:52.317 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:52 smithi171 conmon[51620]: debug 2022-01-31T22:10:52.006+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:52.007359+0000) 2022-01-31T22:10:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:52 smithi171 conmon[41853]: debug 2022-01-31T22:10:52.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:52.359563+0000) 2022-01-31T22:10:52.735 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:52 smithi167 conmon[54076]: debug 2022-01-31T22:10:52.654+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:52.654597+0000) 2022-01-31T22:10:52.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:52 smithi167 conmon[60316]: debug 2022-01-31T22:10:52.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:52.654610+0000) 2022-01-31T22:10:52.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:52 smithi167 conmon[49112]: debug 2022-01-31T22:10:52.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:52.736210+0000) 2022-01-31T22:10:53.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:53 smithi171 conmon[46715]: debug 2022-01-31T22:10:53.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:53.152980+0000) 2022-01-31T22:10:53.317 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:53 smithi171 conmon[51620]: debug 2022-01-31T22:10:53.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:53.007557+0000) 2022-01-31T22:10:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:53 smithi171 conmon[41853]: debug 2022-01-31T22:10:53.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:53.359774+0000) 2022-01-31T22:10:53.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:53 smithi167 conmon[54076]: debug 2022-01-31T22:10:53.654+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:53.654810+0000) 2022-01-31T22:10:53.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:53 smithi167 conmon[60316]: debug 2022-01-31T22:10:53.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:53.654757+0000) 2022-01-31T22:10:53.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:53 smithi167 conmon[49112]: debug 2022-01-31T22:10:53.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:53.736384+0000) 2022-01-31T22:10:54.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:53 smithi171 conmon[35325]: debug 2022-01-31T22:10:53.809+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:10:54.099 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:54 smithi171 conmon[51620]: debug 2022-01-31T22:10:54.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:54.007739+0000) 2022-01-31T22:10:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:54 smithi171 conmon[46715]: debug 2022-01-31T22:10:54.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:54.153154+0000) 2022-01-31T22:10:54.827 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:54 smithi167 conmon[54076]: debug 2022-01-31T22:10:54.654+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:54.654973+0000) 2022-01-31T22:10:54.828 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:54 smithi167 conmon[60316]: debug 2022-01-31T22:10:54.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:54.654894+0000) 2022-01-31T22:10:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:54 smithi171 conmon[41853]: debug 2022-01-31T22:10:54.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:54.359927+0000) 2022-01-31T22:10:54.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:54 smithi167 conmon[49112]: debug 2022-01-31T22:10:54.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:54.736584+0000) 2022-01-31T22:10:55.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[46715]: debug 2022-01-31T22:10:55.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.153334+0000) 2022-01-31T22:10:55.318 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[51620]: debug 2022-01-31T22:10:55.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.007977+0000) 2022-01-31T22:10:55.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[35325]: debug 2022-01-31T22:10:55.370+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 188951 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:10:55.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[46715]: debug 2022-01-31T22:10:55.349+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.350716+0000) 2022-01-31T22:10:55.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[51620]: debug 2022-01-31T22:10:55.350+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.351028+0000) 2022-01-31T22:10:55.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[41853]: debug 2022-01-31T22:10:55.350+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.351617+0000) 2022-01-31T22:10:55.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:55 smithi171 conmon[41853]: debug 2022-01-31T22:10:55.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.360074+0000) 2022-01-31T22:10:55.631 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:55 smithi167 conmon[49112]: debug 2022-01-31T22:10:55.351+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.351985+0000) 2022-01-31T22:10:55.632 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:55 smithi167 conmon[54076]: debug 2022-01-31T22:10:55.351+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.352188+0000) 2022-01-31T22:10:55.632 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:55 smithi167 conmon[60316]: debug 2022-01-31T22:10:55.351+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.351855+0000) 2022-01-31T22:10:55.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:55 smithi167 conmon[60316]: debug 2022-01-31T22:10:55.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.655099+0000) 2022-01-31T22:10:55.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:55 smithi167 conmon[49112]: debug 2022-01-31T22:10:55.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.736737+0000) 2022-01-31T22:10:55.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:55 smithi167 conmon[54076]: debug 2022-01-31T22:10:55.654+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:55.655156+0000) 2022-01-31T22:10:56.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:56 smithi171 conmon[46715]: debug 2022-01-31T22:10:56.152+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:56.153486+0000) 2022-01-31T22:10:56.318 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:56 smithi171 conmon[51620]: debug 2022-01-31T22:10:56.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:56.008147+0000) 2022-01-31T22:10:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:56 smithi171 conmon[41853]: debug 2022-01-31T22:10:56.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:56.360241+0000) 2022-01-31T22:10:56.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:56 smithi167 conmon[60316]: debug 2022-01-31T22:10:56.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:56.655281+0000) 2022-01-31T22:10:56.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:56 smithi167 conmon[54076]: debug 2022-01-31T22:10:56.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:56.655303+0000) 2022-01-31T22:10:56.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:56 smithi167 conmon[49112]: debug 2022-01-31T22:10:56.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:56.736888+0000) 2022-01-31T22:10:57.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:57 smithi171 conmon[46715]: debug 2022-01-31T22:10:57.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:57.153616+0000) 2022-01-31T22:10:57.318 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:57 smithi171 conmon[51620]: debug 2022-01-31T22:10:57.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:57.008329+0000) 2022-01-31T22:10:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:57 smithi171 conmon[41853]: debug 2022-01-31T22:10:57.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:57.360445+0000) 2022-01-31T22:10:57.736 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:57 smithi167 conmon[60316]: debug 2022-01-31T22:10:57.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:57.655490+0000) 2022-01-31T22:10:57.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:57 smithi167 conmon[54076]: debug 2022-01-31T22:10:57.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:57.655489+0000) 2022-01-31T22:10:57.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:57 smithi167 conmon[49112]: debug 2022-01-31T22:10:57.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:57.737068+0000) 2022-01-31T22:10:58.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:58 smithi171 conmon[46715]: debug 2022-01-31T22:10:58.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:58.153798+0000) 2022-01-31T22:10:58.318 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:58 smithi171 conmon[51620]: debug 2022-01-31T22:10:58.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:58.008541+0000) 2022-01-31T22:10:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:58 smithi171 conmon[41853]: debug 2022-01-31T22:10:58.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:58.360655+0000) 2022-01-31T22:10:58.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:58 smithi167 conmon[54076]: debug 2022-01-31T22:10:58.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:58.655708+0000) 2022-01-31T22:10:58.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:58 smithi167 conmon[60316]: debug 2022-01-31T22:10:58.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:58.655733+0000) 2022-01-31T22:10:58.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:58 smithi167 conmon[49112]: debug 2022-01-31T22:10:58.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:58.737225+0000) 2022-01-31T22:10:59.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:10:59 smithi171 conmon[46715]: debug 2022-01-31T22:10:59.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:59.154029+0000) 2022-01-31T22:10:59.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:10:59 smithi171 conmon[51620]: debug 2022-01-31T22:10:59.007+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:59.008700+0000) 2022-01-31T22:10:59.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:10:59 smithi171 conmon[41853]: debug 2022-01-31T22:10:59.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:59.360845+0000) 2022-01-31T22:10:59.736 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:10:59 smithi167 conmon[54076]: debug 2022-01-31T22:10:59.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:59.655900+0000) 2022-01-31T22:10:59.737 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:10:59 smithi167 conmon[60316]: debug 2022-01-31T22:10:59.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:59.655903+0000) 2022-01-31T22:10:59.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:10:59 smithi167 conmon[49112]: debug 2022-01-31T22:10:59.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:10:59.737419+0000) 2022-01-31T22:11:00.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[46715]: debug 2022-01-31T22:11:00.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.154182+0000) 2022-01-31T22:11:00.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[51620]: debug 2022-01-31T22:11:00.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.008864+0000) 2022-01-31T22:11:00.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[35325]: debug 2022-01-31T22:11:00.393+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189062 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:00.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[46715]: debug 2022-01-31T22:11:00.372+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.373434+0000) 2022-01-31T22:11:00.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[51620]: debug 2022-01-31T22:11:00.373+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.373807+0000) 2022-01-31T22:11:00.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[41853]: debug 2022-01-31T22:11:00.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.361028+0000) 2022-01-31T22:11:00.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:00 smithi171 conmon[41853]: debug 2022-01-31T22:11:00.372+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.373609+0000) 2022-01-31T22:11:00.632 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:00 smithi167 conmon[49112]: debug 2022-01-31T22:11:00.373+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.374057+0000) 2022-01-31T22:11:00.633 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:00 smithi167 conmon[54076]: debug 2022-01-31T22:11:00.374+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.375017+0000) 2022-01-31T22:11:00.634 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:00 smithi167 conmon[60316]: debug 2022-01-31T22:11:00.374+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.374889+0000) 2022-01-31T22:11:00.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:00 smithi167 conmon[54076]: debug 2022-01-31T22:11:00.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.656123+0000) 2022-01-31T22:11:00.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:00 smithi167 conmon[60316]: debug 2022-01-31T22:11:00.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.656123+0000) 2022-01-31T22:11:00.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:00 smithi167 conmon[49112]: debug 2022-01-31T22:11:00.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:00.737603+0000) 2022-01-31T22:11:01.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:01 smithi171 conmon[46715]: debug 2022-01-31T22:11:01.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:01.154348+0000) 2022-01-31T22:11:01.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:01 smithi171 conmon[51620]: debug 2022-01-31T22:11:01.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:01.009028+0000) 2022-01-31T22:11:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:01 smithi171 conmon[41853]: debug 2022-01-31T22:11:01.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:01.361238+0000) 2022-01-31T22:11:01.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:01 smithi167 conmon[54076]: debug 2022-01-31T22:11:01.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:01.656279+0000) 2022-01-31T22:11:01.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:01 smithi167 conmon[60316]: debug 2022-01-31T22:11:01.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:01.656301+0000) 2022-01-31T22:11:01.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:01 smithi167 conmon[49112]: debug 2022-01-31T22:11:01.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:01.737756+0000) 2022-01-31T22:11:02.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:02 smithi171 conmon[46715]: debug 2022-01-31T22:11:02.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:02.154528+0000) 2022-01-31T22:11:02.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:02 smithi171 conmon[51620]: debug 2022-01-31T22:11:02.008+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:02.009213+0000) 2022-01-31T22:11:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:02 smithi171 conmon[41853]: debug 2022-01-31T22:11:02.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:02.361376+0000) 2022-01-31T22:11:02.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:02 smithi167 conmon[54076]: debug 2022-01-31T22:11:02.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:02.656474+0000) 2022-01-31T22:11:02.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:02 smithi167 conmon[60316]: debug 2022-01-31T22:11:02.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:02.656520+0000) 2022-01-31T22:11:02.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:02 smithi167 conmon[49112]: debug 2022-01-31T22:11:02.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:02.737915+0000) 2022-01-31T22:11:03.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:03 smithi171 conmon[46715]: debug 2022-01-31T22:11:03.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:03.154737+0000) 2022-01-31T22:11:03.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:03 smithi171 conmon[51620]: debug 2022-01-31T22:11:03.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:03.009405+0000) 2022-01-31T22:11:03.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:03 smithi171 conmon[41853]: debug 2022-01-31T22:11:03.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:03.361521+0000) 2022-01-31T22:11:03.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:03 smithi167 conmon[54076]: debug 2022-01-31T22:11:03.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:03.656675+0000) 2022-01-31T22:11:03.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:03 smithi167 conmon[60316]: debug 2022-01-31T22:11:03.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:03.656682+0000) 2022-01-31T22:11:03.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:03 smithi167 conmon[49112]: debug 2022-01-31T22:11:03.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:03.738061+0000) 2022-01-31T22:11:04.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:04 smithi171 conmon[46715]: debug 2022-01-31T22:11:04.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:04.154953+0000) 2022-01-31T22:11:04.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:04 smithi171 conmon[51620]: debug 2022-01-31T22:11:04.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:04.009612+0000) 2022-01-31T22:11:04.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:04 smithi171 conmon[41853]: debug 2022-01-31T22:11:04.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:04.361638+0000) 2022-01-31T22:11:04.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:04 smithi167 conmon[54076]: debug 2022-01-31T22:11:04.655+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:04.656829+0000) 2022-01-31T22:11:04.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:04 smithi167 conmon[60316]: debug 2022-01-31T22:11:04.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:04.656904+0000) 2022-01-31T22:11:04.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:04 smithi167 conmon[49112]: debug 2022-01-31T22:11:04.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:04.738233+0000) 2022-01-31T22:11:05.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[51620]: debug 2022-01-31T22:11:05.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.009816+0000) 2022-01-31T22:11:05.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[46715]: debug 2022-01-31T22:11:05.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.155135+0000) 2022-01-31T22:11:05.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[35325]: debug 2022-01-31T22:11:05.415+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189174 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:05.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[41853]: debug 2022-01-31T22:11:05.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.361815+0000) 2022-01-31T22:11:05.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[41853]: debug 2022-01-31T22:11:05.395+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.396634+0000) 2022-01-31T22:11:05.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[46715]: debug 2022-01-31T22:11:05.396+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.396958+0000) 2022-01-31T22:11:05.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:05 smithi171 conmon[51620]: debug 2022-01-31T22:11:05.396+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.397169+0000) 2022-01-31T22:11:05.656 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:05 smithi167 conmon[49112]: debug 2022-01-31T22:11:05.395+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.397282+0000) 2022-01-31T22:11:05.657 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:05 smithi167 conmon[60316]: debug 2022-01-31T22:11:05.396+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.397669+0000) 2022-01-31T22:11:05.657 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:05 smithi167 conmon[54076]: debug 2022-01-31T22:11:05.395+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.396999+0000) 2022-01-31T22:11:05.658 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:05 smithi167 conmon[54076]: debug 2022-01-31T22:11:05.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.657019+0000) 2022-01-31T22:11:05.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:05 smithi167 conmon[60316]: debug 2022-01-31T22:11:05.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.657064+0000) 2022-01-31T22:11:05.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:05 smithi167 conmon[49112]: debug 2022-01-31T22:11:05.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:05.738437+0000) 2022-01-31T22:11:06.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:06 smithi171 conmon[46715]: debug 2022-01-31T22:11:06.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:06.155292+0000) 2022-01-31T22:11:06.320 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:06 smithi171 conmon[51620]: debug 2022-01-31T22:11:06.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:06.010021+0000) 2022-01-31T22:11:06.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:06 smithi171 conmon[41853]: debug 2022-01-31T22:11:06.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:06.361973+0000) 2022-01-31T22:11:06.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:06 smithi167 conmon[54076]: debug 2022-01-31T22:11:06.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:06.657178+0000) 2022-01-31T22:11:06.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:06 smithi167 conmon[60316]: debug 2022-01-31T22:11:06.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:06.657245+0000) 2022-01-31T22:11:06.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:06 smithi167 conmon[49112]: debug 2022-01-31T22:11:06.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:06.738575+0000) 2022-01-31T22:11:07.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:07 smithi171 conmon[51620]: debug 2022-01-31T22:11:07.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:07.010177+0000) 2022-01-31T22:11:07.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:07 smithi171 conmon[46715]: debug 2022-01-31T22:11:07.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:07.155443+0000) 2022-01-31T22:11:07.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:07 smithi171 conmon[41853]: debug 2022-01-31T22:11:07.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:07.362114+0000) 2022-01-31T22:11:07.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:07 smithi167 conmon[54076]: debug 2022-01-31T22:11:07.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:07.657393+0000) 2022-01-31T22:11:07.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:07 smithi167 conmon[60316]: debug 2022-01-31T22:11:07.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:07.657525+0000) 2022-01-31T22:11:07.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:07 smithi167 conmon[49112]: debug 2022-01-31T22:11:07.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:07.738757+0000) 2022-01-31T22:11:08.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:08 smithi171 conmon[46715]: debug 2022-01-31T22:11:08.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:08.155637+0000) 2022-01-31T22:11:08.320 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:08 smithi171 conmon[51620]: debug 2022-01-31T22:11:08.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:08.010362+0000) 2022-01-31T22:11:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:08 smithi171 conmon[41853]: debug 2022-01-31T22:11:08.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:08.362316+0000) 2022-01-31T22:11:08.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:08 smithi167 conmon[54076]: debug 2022-01-31T22:11:08.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:08.657556+0000) 2022-01-31T22:11:08.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:08 smithi167 conmon[60316]: debug 2022-01-31T22:11:08.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:08.657648+0000) 2022-01-31T22:11:08.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:08 smithi167 conmon[49112]: debug 2022-01-31T22:11:08.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:08.738951+0000) 2022-01-31T22:11:09.009 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:08 smithi171 conmon[35325]: debug 2022-01-31T22:11:08.810+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:11:09.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:09 smithi171 conmon[46715]: debug 2022-01-31T22:11:09.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:09.155768+0000) 2022-01-31T22:11:09.320 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:09 smithi171 conmon[51620]: debug 2022-01-31T22:11:09.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:09.010546+0000) 2022-01-31T22:11:09.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:09 smithi171 conmon[41853]: debug 2022-01-31T22:11:09.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:09.362521+0000) 2022-01-31T22:11:09.724 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:09 smithi167 conmon[54076]: debug 2022-01-31T22:11:09.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:09.657681+0000) 2022-01-31T22:11:09.725 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:09 smithi167 conmon[60316]: debug 2022-01-31T22:11:09.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:09.657784+0000) 2022-01-31T22:11:09.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:09 smithi167 conmon[49112]: debug 2022-01-31T22:11:09.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:09.739136+0000) 2022-01-31T22:11:10.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[46715]: debug 2022-01-31T22:11:10.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.155950+0000) 2022-01-31T22:11:10.320 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[51620]: debug 2022-01-31T22:11:10.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.010672+0000) 2022-01-31T22:11:10.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[35325]: debug 2022-01-31T22:11:10.437+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189285 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:10.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[46715]: debug 2022-01-31T22:11:10.418+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.418941+0000) 2022-01-31T22:11:10.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[51620]: debug 2022-01-31T22:11:10.418+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.418861+0000) 2022-01-31T22:11:10.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[41853]: debug 2022-01-31T22:11:10.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.362709+0000) 2022-01-31T22:11:10.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:10 smithi171 conmon[41853]: debug 2022-01-31T22:11:10.418+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.419302+0000) 2022-01-31T22:11:10.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:10 smithi167 conmon[54076]: debug 2022-01-31T22:11:10.418+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.420385+0000) 2022-01-31T22:11:10.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:10 smithi167 conmon[54076]: debug 2022-01-31T22:11:10.657+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.657915+0000) 2022-01-31T22:11:10.670 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:10 smithi167 conmon[49112]: debug 2022-01-31T22:11:10.418+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.419522+0000) 2022-01-31T22:11:10.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:10 smithi167 conmon[60316]: debug 2022-01-31T22:11:10.417+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.418959+0000) 2022-01-31T22:11:10.671 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:10 smithi167 conmon[60316]: debug 2022-01-31T22:11:10.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.657974+0000) 2022-01-31T22:11:10.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:10 smithi167 conmon[49112]: debug 2022-01-31T22:11:10.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:10.739304+0000) 2022-01-31T22:11:11.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:11 smithi171 conmon[46715]: debug 2022-01-31T22:11:11.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:11.156153+0000) 2022-01-31T22:11:11.321 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:11 smithi171 conmon[51620]: debug 2022-01-31T22:11:11.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:11.010866+0000) 2022-01-31T22:11:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:11 smithi171 conmon[41853]: debug 2022-01-31T22:11:11.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:11.363050+0000) 2022-01-31T22:11:11.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:11 smithi167 conmon[60316]: debug 2022-01-31T22:11:11.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:11.658155+0000) 2022-01-31T22:11:11.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:11 smithi167 conmon[54076]: debug 2022-01-31T22:11:11.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:11.658068+0000) 2022-01-31T22:11:11.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:11 smithi167 conmon[49112]: debug 2022-01-31T22:11:11.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:11.739465+0000) 2022-01-31T22:11:12.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:12 smithi171 conmon[46715]: debug 2022-01-31T22:11:12.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:12.156310+0000) 2022-01-31T22:11:12.321 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:12 smithi171 conmon[51620]: debug 2022-01-31T22:11:12.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:12.011032+0000) 2022-01-31T22:11:12.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:12 smithi171 conmon[41853]: debug 2022-01-31T22:11:12.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:12.363427+0000) 2022-01-31T22:11:12.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:12 smithi167 conmon[60316]: debug 2022-01-31T22:11:12.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:12.658304+0000) 2022-01-31T22:11:12.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:12 smithi167 conmon[54076]: debug 2022-01-31T22:11:12.657+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:12.658188+0000) 2022-01-31T22:11:12.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:12 smithi167 conmon[49112]: debug 2022-01-31T22:11:12.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:12.739656+0000) 2022-01-31T22:11:13.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:13 smithi171 conmon[46715]: debug 2022-01-31T22:11:13.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:13.156491+0000) 2022-01-31T22:11:13.321 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:13 smithi171 conmon[51620]: debug 2022-01-31T22:11:13.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:13.011223+0000) 2022-01-31T22:11:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:13 smithi171 conmon[41853]: debug 2022-01-31T22:11:13.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:13.363636+0000) 2022-01-31T22:11:13.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:13 smithi167 conmon[54076]: debug 2022-01-31T22:11:13.656+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:13.658385+0000) 2022-01-31T22:11:13.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:13 smithi167 conmon[60316]: debug 2022-01-31T22:11:13.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:13.658544+0000) 2022-01-31T22:11:13.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:13 smithi167 conmon[49112]: debug 2022-01-31T22:11:13.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:13.739787+0000) 2022-01-31T22:11:14.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:14 smithi171 conmon[46715]: debug 2022-01-31T22:11:14.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:14.156686+0000) 2022-01-31T22:11:14.321 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:14 smithi171 conmon[51620]: debug 2022-01-31T22:11:14.009+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:14.011433+0000) 2022-01-31T22:11:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:14 smithi171 conmon[41853]: debug 2022-01-31T22:11:14.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:14.363823+0000) 2022-01-31T22:11:14.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:14 smithi167 conmon[54076]: debug 2022-01-31T22:11:14.657+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:14.658609+0000) 2022-01-31T22:11:14.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:14 smithi167 conmon[60316]: debug 2022-01-31T22:11:14.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:14.658672+0000) 2022-01-31T22:11:14.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:14 smithi167 conmon[49112]: debug 2022-01-31T22:11:14.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:14.739947+0000) 2022-01-31T22:11:15.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[46715]: debug 2022-01-31T22:11:15.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.156912+0000) 2022-01-31T22:11:15.321 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[51620]: debug 2022-01-31T22:11:15.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.011611+0000) 2022-01-31T22:11:15.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[35325]: debug 2022-01-31T22:11:15.459+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189396 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:15.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[46715]: debug 2022-01-31T22:11:15.438+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.440524+0000) 2022-01-31T22:11:15.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[51620]: debug 2022-01-31T22:11:15.440+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.441835+0000) 2022-01-31T22:11:15.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[41853]: debug 2022-01-31T22:11:15.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.364008+0000) 2022-01-31T22:11:15.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:15 smithi171 conmon[41853]: debug 2022-01-31T22:11:15.440+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.441975+0000) 2022-01-31T22:11:15.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:15 smithi167 conmon[49112]: debug 2022-01-31T22:11:15.439+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.441137+0000) 2022-01-31T22:11:15.709 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:15 smithi167 conmon[54076]: debug 2022-01-31T22:11:15.439+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.440788+0000) 2022-01-31T22:11:15.709 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:15 smithi167 conmon[54076]: debug 2022-01-31T22:11:15.657+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.658799+0000) 2022-01-31T22:11:15.709 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:15 smithi167 conmon[60316]: debug 2022-01-31T22:11:15.439+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.441408+0000) 2022-01-31T22:11:15.710 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:15 smithi167 conmon[60316]: debug 2022-01-31T22:11:15.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.658881+0000) 2022-01-31T22:11:15.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:15 smithi167 conmon[49112]: debug 2022-01-31T22:11:15.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:15.740145+0000) 2022-01-31T22:11:16.321 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:16 smithi171 conmon[51620]: debug 2022-01-31T22:11:16.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:16.011735+0000) 2022-01-31T22:11:16.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:16 smithi171 conmon[46715]: debug 2022-01-31T22:11:16.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:16.157085+0000) 2022-01-31T22:11:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:16 smithi171 conmon[41853]: debug 2022-01-31T22:11:16.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:16.364170+0000) 2022-01-31T22:11:16.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:16 smithi167 conmon[49112]: debug 2022-01-31T22:11:16.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:16.740257+0000) 2022-01-31T22:11:16.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:16 smithi167 conmon[54076]: debug 2022-01-31T22:11:16.658+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:16.659042+0000) 2022-01-31T22:11:16.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:16 smithi167 conmon[60316]: debug 2022-01-31T22:11:16.658+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:16.658950+0000) 2022-01-31T22:11:17.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:17 smithi171 conmon[46715]: debug 2022-01-31T22:11:17.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:17.157198+0000) 2022-01-31T22:11:17.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:17 smithi171 conmon[51620]: debug 2022-01-31T22:11:17.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:17.011906+0000) 2022-01-31T22:11:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:17 smithi171 conmon[41853]: debug 2022-01-31T22:11:17.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:17.364336+0000) 2022-01-31T22:11:17.739 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:17 smithi167 conmon[54076]: debug 2022-01-31T22:11:17.657+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:17.659231+0000) 2022-01-31T22:11:17.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:17 smithi167 conmon[60316]: debug 2022-01-31T22:11:17.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:17.659138+0000) 2022-01-31T22:11:17.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:17 smithi167 conmon[49112]: debug 2022-01-31T22:11:17.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:17.740424+0000) 2022-01-31T22:11:18.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:18 smithi171 conmon[46715]: debug 2022-01-31T22:11:18.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:18.157356+0000) 2022-01-31T22:11:18.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:18 smithi171 conmon[51620]: debug 2022-01-31T22:11:18.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:18.012074+0000) 2022-01-31T22:11:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:18 smithi171 conmon[41853]: debug 2022-01-31T22:11:18.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:18.364521+0000) 2022-01-31T22:11:18.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:18 smithi167 conmon[54076]: debug 2022-01-31T22:11:18.657+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:18.659393+0000) 2022-01-31T22:11:18.740 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:18 smithi167 conmon[60316]: debug 2022-01-31T22:11:18.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:18.659337+0000) 2022-01-31T22:11:18.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:18 smithi167 conmon[49112]: debug 2022-01-31T22:11:18.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:18.740579+0000) 2022-01-31T22:11:19.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:19 smithi171 conmon[46715]: debug 2022-01-31T22:11:19.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:19.157541+0000) 2022-01-31T22:11:19.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:19 smithi171 conmon[51620]: debug 2022-01-31T22:11:19.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:19.012315+0000) 2022-01-31T22:11:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:19 smithi171 conmon[41853]: debug 2022-01-31T22:11:19.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:19.364704+0000) 2022-01-31T22:11:19.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:19 smithi167 conmon[54076]: debug 2022-01-31T22:11:19.658+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:19.659573+0000) 2022-01-31T22:11:19.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:19 smithi167 conmon[60316]: debug 2022-01-31T22:11:19.658+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:19.659558+0000) 2022-01-31T22:11:19.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:19 smithi167 conmon[49112]: debug 2022-01-31T22:11:19.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:19.740768+0000) 2022-01-31T22:11:20.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[46715]: debug 2022-01-31T22:11:20.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.157744+0000) 2022-01-31T22:11:20.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[51620]: debug 2022-01-31T22:11:20.010+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.012548+0000) 2022-01-31T22:11:20.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[35325]: debug 2022-01-31T22:11:20.481+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189507 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:20.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[46715]: debug 2022-01-31T22:11:20.461+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.463494+0000) 2022-01-31T22:11:20.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[51620]: debug 2022-01-31T22:11:20.463+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.465013+0000) 2022-01-31T22:11:20.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[41853]: debug 2022-01-31T22:11:20.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.364908+0000) 2022-01-31T22:11:20.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:20 smithi171 conmon[41853]: debug 2022-01-31T22:11:20.462+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.464622+0000) 2022-01-31T22:11:20.740 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:20 smithi167 conmon[49112]: debug 2022-01-31T22:11:20.462+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.463516+0000) 2022-01-31T22:11:20.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:20 smithi167 conmon[54076]: debug 2022-01-31T22:11:20.461+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.463153+0000) 2022-01-31T22:11:20.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:20 smithi167 conmon[54076]: debug 2022-01-31T22:11:20.659+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.659791+0000) 2022-01-31T22:11:20.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:20 smithi167 conmon[60316]: debug 2022-01-31T22:11:20.462+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.464107+0000) 2022-01-31T22:11:20.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:20 smithi167 conmon[60316]: debug 2022-01-31T22:11:20.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.659790+0000) 2022-01-31T22:11:20.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:20 smithi167 conmon[49112]: debug 2022-01-31T22:11:20.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:20.740931+0000) 2022-01-31T22:11:21.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:21 smithi171 conmon[46715]: debug 2022-01-31T22:11:21.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:21.157968+0000) 2022-01-31T22:11:21.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:21 smithi171 conmon[51620]: debug 2022-01-31T22:11:21.011+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:21.012745+0000) 2022-01-31T22:11:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:21 smithi171 conmon[41853]: debug 2022-01-31T22:11:21.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:21.365112+0000) 2022-01-31T22:11:21.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:21 smithi167 conmon[60316]: debug 2022-01-31T22:11:21.658+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:21.659954+0000) 2022-01-31T22:11:21.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:21 smithi167 conmon[54076]: debug 2022-01-31T22:11:21.658+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:21.659954+0000) 2022-01-31T22:11:21.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:21 smithi167 conmon[49112]: debug 2022-01-31T22:11:21.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:21.741074+0000) 2022-01-31T22:11:22.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:22 smithi171 conmon[46715]: debug 2022-01-31T22:11:22.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:22.158119+0000) 2022-01-31T22:11:22.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:22 smithi171 conmon[51620]: debug 2022-01-31T22:11:22.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:22.012954+0000) 2022-01-31T22:11:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:22 smithi171 conmon[41853]: debug 2022-01-31T22:11:22.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:22.365260+0000) 2022-01-31T22:11:22.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:22 smithi167 conmon[54076]: debug 2022-01-31T22:11:22.658+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:22.660148+0000) 2022-01-31T22:11:22.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:22 smithi167 conmon[60316]: debug 2022-01-31T22:11:22.658+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:22.660142+0000) 2022-01-31T22:11:23.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:22 smithi167 conmon[49112]: debug 2022-01-31T22:11:22.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:22.741210+0000) 2022-01-31T22:11:23.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:23 smithi171 conmon[46715]: debug 2022-01-31T22:11:23.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:23.158272+0000) 2022-01-31T22:11:23.323 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:23 smithi171 conmon[51620]: debug 2022-01-31T22:11:23.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:23.013132+0000) 2022-01-31T22:11:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:23 smithi171 conmon[41853]: debug 2022-01-31T22:11:23.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:23.365440+0000) 2022-01-31T22:11:23.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:23 smithi167 conmon[54076]: debug 2022-01-31T22:11:23.659+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:23.660352+0000) 2022-01-31T22:11:23.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:23 smithi167 conmon[60316]: debug 2022-01-31T22:11:23.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:23.660353+0000) 2022-01-31T22:11:24.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:23 smithi167 conmon[49112]: debug 2022-01-31T22:11:23.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:23.741385+0000) 2022-01-31T22:11:24.104 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:23 smithi171 conmon[35325]: debug 2022-01-31T22:11:23.810+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:11:24.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:24 smithi171 conmon[51620]: debug 2022-01-31T22:11:24.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:24.013339+0000) 2022-01-31T22:11:24.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:24 smithi171 conmon[46715]: debug 2022-01-31T22:11:24.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:24.158449+0000) 2022-01-31T22:11:24.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:24 smithi167 conmon[60316]: debug 2022-01-31T22:11:24.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:24.660541+0000) 2022-01-31T22:11:24.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:24 smithi167 conmon[54076]: debug 2022-01-31T22:11:24.659+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:24.660561+0000) 2022-01-31T22:11:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:24 smithi171 conmon[41853]: debug 2022-01-31T22:11:24.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:24.365665+0000) 2022-01-31T22:11:25.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:24 smithi167 conmon[49112]: debug 2022-01-31T22:11:24.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:24.741578+0000) 2022-01-31T22:11:25.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[46715]: debug 2022-01-31T22:11:25.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.158627+0000) 2022-01-31T22:11:25.323 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[51620]: debug 2022-01-31T22:11:25.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.013467+0000) 2022-01-31T22:11:25.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[35325]: debug 2022-01-31T22:11:25.505+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189618 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:25.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[46715]: debug 2022-01-31T22:11:25.485+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.486067+0000) 2022-01-31T22:11:25.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[51620]: debug 2022-01-31T22:11:25.485+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.486721+0000) 2022-01-31T22:11:25.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[41853]: debug 2022-01-31T22:11:25.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.365849+0000) 2022-01-31T22:11:25.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:25 smithi171 conmon[41853]: debug 2022-01-31T22:11:25.485+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.486363+0000) 2022-01-31T22:11:25.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:25 smithi167 conmon[49112]: debug 2022-01-31T22:11:25.484+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.486035+0000) 2022-01-31T22:11:25.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:25 smithi167 conmon[54076]: debug 2022-01-31T22:11:25.484+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.485797+0000) 2022-01-31T22:11:25.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:25 smithi167 conmon[54076]: debug 2022-01-31T22:11:25.660+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.660689+0000) 2022-01-31T22:11:25.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:25 smithi167 conmon[60316]: debug 2022-01-31T22:11:25.485+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.486782+0000) 2022-01-31T22:11:25.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:25 smithi167 conmon[60316]: debug 2022-01-31T22:11:25.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.660712+0000) 2022-01-31T22:11:26.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:25 smithi167 conmon[49112]: debug 2022-01-31T22:11:25.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:25.741759+0000) 2022-01-31T22:11:26.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:26 smithi171 conmon[46715]: debug 2022-01-31T22:11:26.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:26.158820+0000) 2022-01-31T22:11:26.324 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:26 smithi171 conmon[51620]: debug 2022-01-31T22:11:26.012+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:26.013677+0000) 2022-01-31T22:11:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:26 smithi171 conmon[41853]: debug 2022-01-31T22:11:26.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:26.366037+0000) 2022-01-31T22:11:26.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:26 smithi167 conmon[54076]: debug 2022-01-31T22:11:26.660+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:26.660842+0000) 2022-01-31T22:11:26.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:26 smithi167 conmon[60316]: debug 2022-01-31T22:11:26.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:26.660873+0000) 2022-01-31T22:11:27.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:26 smithi167 conmon[49112]: debug 2022-01-31T22:11:26.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:26.741910+0000) 2022-01-31T22:11:27.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:27 smithi171 conmon[46715]: debug 2022-01-31T22:11:27.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:27.158986+0000) 2022-01-31T22:11:27.324 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:27 smithi171 conmon[51620]: debug 2022-01-31T22:11:27.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:27.013845+0000) 2022-01-31T22:11:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:27 smithi171 conmon[41853]: debug 2022-01-31T22:11:27.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:27.366203+0000) 2022-01-31T22:11:27.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:27 smithi167 conmon[54076]: debug 2022-01-31T22:11:27.660+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:27.661040+0000) 2022-01-31T22:11:27.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:27 smithi167 conmon[60316]: debug 2022-01-31T22:11:27.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:27.661059+0000) 2022-01-31T22:11:28.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:27 smithi167 conmon[49112]: debug 2022-01-31T22:11:27.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:27.742104+0000) 2022-01-31T22:11:28.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:28 smithi171 conmon[46715]: debug 2022-01-31T22:11:28.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:28.159104+0000) 2022-01-31T22:11:28.323 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:28 smithi171 conmon[51620]: debug 2022-01-31T22:11:28.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:28.014034+0000) 2022-01-31T22:11:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:28 smithi171 conmon[41853]: debug 2022-01-31T22:11:28.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:28.366409+0000) 2022-01-31T22:11:28.741 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:28 smithi167 conmon[54076]: debug 2022-01-31T22:11:28.660+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:28.661245+0000) 2022-01-31T22:11:28.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:28 smithi167 conmon[60316]: debug 2022-01-31T22:11:28.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:28.661249+0000) 2022-01-31T22:11:29.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:28 smithi167 conmon[49112]: debug 2022-01-31T22:11:28.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:28.742311+0000) 2022-01-31T22:11:29.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:29 smithi171 conmon[46715]: debug 2022-01-31T22:11:29.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:29.159287+0000) 2022-01-31T22:11:29.324 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:29 smithi171 conmon[51620]: debug 2022-01-31T22:11:29.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:29.014225+0000) 2022-01-31T22:11:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:29 smithi171 conmon[41853]: debug 2022-01-31T22:11:29.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:29.366601+0000) 2022-01-31T22:11:29.742 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:29 smithi167 conmon[60316]: debug 2022-01-31T22:11:29.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:29.661461+0000) 2022-01-31T22:11:29.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:29 smithi167 conmon[54076]: debug 2022-01-31T22:11:29.660+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:29.661460+0000) 2022-01-31T22:11:30.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:29 smithi167 conmon[49112]: debug 2022-01-31T22:11:29.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:29.742479+0000) 2022-01-31T22:11:30.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[46715]: debug 2022-01-31T22:11:30.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.159462+0000) 2022-01-31T22:11:30.324 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[51620]: debug 2022-01-31T22:11:30.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.014405+0000) 2022-01-31T22:11:30.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[35325]: debug 2022-01-31T22:11:30.528+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189729 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:30.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[46715]: debug 2022-01-31T22:11:30.507+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.508250+0000) 2022-01-31T22:11:30.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[51620]: debug 2022-01-31T22:11:30.507+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.508397+0000) 2022-01-31T22:11:30.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[41853]: debug 2022-01-31T22:11:30.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.366790+0000) 2022-01-31T22:11:30.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:30 smithi171 conmon[41853]: debug 2022-01-31T22:11:30.507+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.508469+0000) 2022-01-31T22:11:30.742 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:30 smithi167 conmon[49112]: debug 2022-01-31T22:11:30.507+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.508027+0000) 2022-01-31T22:11:30.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:30 smithi167 conmon[54076]: debug 2022-01-31T22:11:30.508+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.508804+0000) 2022-01-31T22:11:30.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:30 smithi167 conmon[54076]: debug 2022-01-31T22:11:30.661+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.661687+0000) 2022-01-31T22:11:30.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:30 smithi167 conmon[60316]: debug 2022-01-31T22:11:30.508+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.508600+0000) 2022-01-31T22:11:30.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:30 smithi167 conmon[60316]: debug 2022-01-31T22:11:30.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.661683+0000) 2022-01-31T22:11:31.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:30 smithi167 conmon[49112]: debug 2022-01-31T22:11:30.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:30.742643+0000) 2022-01-31T22:11:31.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:31 smithi171 conmon[46715]: debug 2022-01-31T22:11:31.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:31.159638+0000) 2022-01-31T22:11:31.324 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:31 smithi171 conmon[51620]: debug 2022-01-31T22:11:31.013+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:31.014586+0000) 2022-01-31T22:11:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:31 smithi171 conmon[41853]: debug 2022-01-31T22:11:31.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:31.366948+0000) 2022-01-31T22:11:31.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:31 smithi167 conmon[54076]: debug 2022-01-31T22:11:31.661+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:31.661843+0000) 2022-01-31T22:11:31.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:31 smithi167 conmon[60316]: debug 2022-01-31T22:11:31.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:31.661861+0000) 2022-01-31T22:11:32.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:31 smithi167 conmon[49112]: debug 2022-01-31T22:11:31.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:31.742792+0000) 2022-01-31T22:11:32.323 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:32 smithi171 conmon[46715]: debug 2022-01-31T22:11:32.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:32.159787+0000) 2022-01-31T22:11:32.324 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:32 smithi171 conmon[51620]: debug 2022-01-31T22:11:32.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:32.014784+0000) 2022-01-31T22:11:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:32 smithi171 conmon[41853]: debug 2022-01-31T22:11:32.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:32.367074+0000) 2022-01-31T22:11:32.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:32 smithi167 conmon[54076]: debug 2022-01-31T22:11:32.661+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:32.662049+0000) 2022-01-31T22:11:32.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:32 smithi167 conmon[60316]: debug 2022-01-31T22:11:32.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:32.662047+0000) 2022-01-31T22:11:33.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:32 smithi167 conmon[49112]: debug 2022-01-31T22:11:32.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:32.742996+0000) 2022-01-31T22:11:33.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:33 smithi171 conmon[46715]: debug 2022-01-31T22:11:33.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:33.159929+0000) 2022-01-31T22:11:33.325 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:33 smithi171 conmon[51620]: debug 2022-01-31T22:11:33.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:33.014983+0000) 2022-01-31T22:11:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:33 smithi171 conmon[41853]: debug 2022-01-31T22:11:33.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:33.367226+0000) 2022-01-31T22:11:33.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:33 smithi167 conmon[54076]: debug 2022-01-31T22:11:33.661+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:33.662216+0000) 2022-01-31T22:11:33.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:33 smithi167 conmon[60316]: debug 2022-01-31T22:11:33.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:33.662231+0000) 2022-01-31T22:11:34.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:33 smithi167 conmon[49112]: debug 2022-01-31T22:11:33.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:33.743151+0000) 2022-01-31T22:11:34.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:34 smithi171 conmon[46715]: debug 2022-01-31T22:11:34.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:34.160110+0000) 2022-01-31T22:11:34.325 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:34 smithi171 conmon[51620]: debug 2022-01-31T22:11:34.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:34.015184+0000) 2022-01-31T22:11:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:34 smithi171 conmon[41853]: debug 2022-01-31T22:11:34.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:34.367367+0000) 2022-01-31T22:11:34.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:34 smithi167 conmon[54076]: debug 2022-01-31T22:11:34.661+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:34.662434+0000) 2022-01-31T22:11:34.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:34 smithi167 conmon[60316]: debug 2022-01-31T22:11:34.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:34.662431+0000) 2022-01-31T22:11:35.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:34 smithi167 conmon[49112]: debug 2022-01-31T22:11:34.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:34.743404+0000) 2022-01-31T22:11:35.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[46715]: debug 2022-01-31T22:11:35.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.160365+0000) 2022-01-31T22:11:35.325 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[51620]: debug 2022-01-31T22:11:35.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.015472+0000) 2022-01-31T22:11:35.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[35325]: debug 2022-01-31T22:11:35.551+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189840 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:35.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[46715]: debug 2022-01-31T22:11:35.533+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.533814+0000) 2022-01-31T22:11:35.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[51620]: debug 2022-01-31T22:11:35.532+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.533123+0000) 2022-01-31T22:11:35.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[41853]: debug 2022-01-31T22:11:35.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.367557+0000) 2022-01-31T22:11:35.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:35 smithi171 conmon[41853]: debug 2022-01-31T22:11:35.531+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.532716+0000) 2022-01-31T22:11:35.743 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:35 smithi167 conmon[49112]: debug 2022-01-31T22:11:35.532+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.532762+0000) 2022-01-31T22:11:35.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:35 smithi167 conmon[54076]: debug 2022-01-31T22:11:35.532+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.533083+0000) 2022-01-31T22:11:35.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:35 smithi167 conmon[54076]: debug 2022-01-31T22:11:35.662+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.662648+0000) 2022-01-31T22:11:35.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:35 smithi167 conmon[60316]: debug 2022-01-31T22:11:35.532+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.532582+0000) 2022-01-31T22:11:35.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:35 smithi167 conmon[60316]: debug 2022-01-31T22:11:35.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.662593+0000) 2022-01-31T22:11:36.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:35 smithi167 conmon[49112]: debug 2022-01-31T22:11:35.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:35.743588+0000) 2022-01-31T22:11:36.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:36 smithi171 conmon[46715]: debug 2022-01-31T22:11:36.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:36.160525+0000) 2022-01-31T22:11:36.325 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:36 smithi171 conmon[51620]: debug 2022-01-31T22:11:36.014+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:36.015652+0000) 2022-01-31T22:11:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:36 smithi171 conmon[41853]: debug 2022-01-31T22:11:36.367+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:36.367705+0000) 2022-01-31T22:11:36.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:36 smithi167 conmon[54076]: debug 2022-01-31T22:11:36.662+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:36.662791+0000) 2022-01-31T22:11:36.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:36 smithi167 conmon[60316]: debug 2022-01-31T22:11:36.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:36.662692+0000) 2022-01-31T22:11:37.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:36 smithi167 conmon[49112]: debug 2022-01-31T22:11:36.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:36.743793+0000) 2022-01-31T22:11:37.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:37 smithi171 conmon[46715]: debug 2022-01-31T22:11:37.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:37.160679+0000) 2022-01-31T22:11:37.325 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:37 smithi171 conmon[51620]: debug 2022-01-31T22:11:37.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:37.015815+0000) 2022-01-31T22:11:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:37 smithi171 conmon[41853]: debug 2022-01-31T22:11:37.367+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:37.367853+0000) 2022-01-31T22:11:37.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:37 smithi167 conmon[54076]: debug 2022-01-31T22:11:37.662+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:37.662990+0000) 2022-01-31T22:11:37.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:37 smithi167 conmon[60316]: debug 2022-01-31T22:11:37.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:37.662883+0000) 2022-01-31T22:11:38.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:37 smithi167 conmon[49112]: debug 2022-01-31T22:11:37.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:37.744001+0000) 2022-01-31T22:11:38.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:38 smithi171 conmon[46715]: debug 2022-01-31T22:11:38.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:38.160871+0000) 2022-01-31T22:11:38.325 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:38 smithi171 conmon[51620]: debug 2022-01-31T22:11:38.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:38.016014+0000) 2022-01-31T22:11:38.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:38 smithi171 conmon[41853]: debug 2022-01-31T22:11:38.367+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:38.368034+0000) 2022-01-31T22:11:38.743 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:38 smithi167 conmon[54076]: debug 2022-01-31T22:11:38.662+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:38.663179+0000) 2022-01-31T22:11:38.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:38 smithi167 conmon[60316]: debug 2022-01-31T22:11:38.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:38.663073+0000) 2022-01-31T22:11:39.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:38 smithi167 conmon[49112]: debug 2022-01-31T22:11:38.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:38.744210+0000) 2022-01-31T22:11:39.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:39 smithi171 conmon[51620]: debug 2022-01-31T22:11:39.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:39.016153+0000) 2022-01-31T22:11:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:38 smithi171 conmon[35325]: debug 2022-01-31T22:11:38.811+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:11:39.367 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:39 smithi171 conmon[46715]: debug 2022-01-31T22:11:39.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:39.161075+0000) 2022-01-31T22:11:39.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:39 smithi167 conmon[54076]: debug 2022-01-31T22:11:39.662+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:39.663377+0000) 2022-01-31T22:11:39.744 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:39 smithi167 conmon[60316]: debug 2022-01-31T22:11:39.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:39.663259+0000) 2022-01-31T22:11:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:39 smithi171 conmon[41853]: debug 2022-01-31T22:11:39.367+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:39.368186+0000) 2022-01-31T22:11:40.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:39 smithi167 conmon[49112]: debug 2022-01-31T22:11:39.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:39.744388+0000) 2022-01-31T22:11:40.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[46715]: debug 2022-01-31T22:11:40.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.161269+0000) 2022-01-31T22:11:40.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[51620]: debug 2022-01-31T22:11:40.015+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.016344+0000) 2022-01-31T22:11:40.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[46715]: debug 2022-01-31T22:11:40.554+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.555037+0000) 2022-01-31T22:11:40.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[51620]: debug 2022-01-31T22:11:40.554+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.554945+0000) 2022-01-31T22:11:40.580 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[41853]: debug 2022-01-31T22:11:40.367+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.368367+0000) 2022-01-31T22:11:40.580 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[41853]: debug 2022-01-31T22:11:40.553+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.554653+0000) 2022-01-31T22:11:40.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[49112]: debug 2022-01-31T22:11:40.560+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.561091+0000) 2022-01-31T22:11:40.744 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[49112]: 2022-01-31T22:11:40.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[54076]: debug 2022-01-31T22:11:40.553+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.554328+0000) 2022-01-31T22:11:40.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[54076]: debug 2022-01-31T22:11:40.663+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.663574+0000) 2022-01-31T22:11:40.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[60316]: debug 2022-01-31T22:11:40.561+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.561698+0000) 2022-01-31T22:11:40.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[60316]: debug 2022-01-31T22:11:40.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.663465+0000) 2022-01-31T22:11:40.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:40 smithi171 conmon[35325]: debug 2022-01-31T22:11:40.578+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 189950 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:41.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:40 smithi167 conmon[49112]: debug 2022-01-31T22:11:40.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:40.744610+0000) 2022-01-31T22:11:41.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:41 smithi171 conmon[46715]: debug 2022-01-31T22:11:41.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:41.161441+0000) 2022-01-31T22:11:41.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:41 smithi171 conmon[51620]: debug 2022-01-31T22:11:41.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:41.016547+0000) 2022-01-31T22:11:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:41 smithi171 conmon[41853]: debug 2022-01-31T22:11:41.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:41.368551+0000) 2022-01-31T22:11:41.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:41 smithi167 conmon[54076]: debug 2022-01-31T22:11:41.663+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:41.663731+0000) 2022-01-31T22:11:41.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:41 smithi167 conmon[60316]: debug 2022-01-31T22:11:41.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:41.663687+0000) 2022-01-31T22:11:42.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:41 smithi167 conmon[49112]: debug 2022-01-31T22:11:41.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:41.744763+0000) 2022-01-31T22:11:42.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:42 smithi171 conmon[46715]: debug 2022-01-31T22:11:42.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:42.161613+0000) 2022-01-31T22:11:42.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:42 smithi171 conmon[51620]: debug 2022-01-31T22:11:42.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:42.016685+0000) 2022-01-31T22:11:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:42 smithi171 conmon[41853]: debug 2022-01-31T22:11:42.367+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:42.368705+0000) 2022-01-31T22:11:42.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:42 smithi167 conmon[54076]: debug 2022-01-31T22:11:42.663+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:42.663928+0000) 2022-01-31T22:11:42.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:42 smithi167 conmon[60316]: debug 2022-01-31T22:11:42.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:42.663837+0000) 2022-01-31T22:11:43.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:42 smithi167 conmon[49112]: debug 2022-01-31T22:11:42.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:42.744956+0000) 2022-01-31T22:11:43.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:43 smithi171 conmon[46715]: debug 2022-01-31T22:11:43.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:43.161793+0000) 2022-01-31T22:11:43.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:43 smithi171 conmon[51620]: debug 2022-01-31T22:11:43.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:43.016897+0000) 2022-01-31T22:11:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:43 smithi171 conmon[41853]: debug 2022-01-31T22:11:43.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:43.368933+0000) 2022-01-31T22:11:43.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:43 smithi167 conmon[54076]: debug 2022-01-31T22:11:43.663+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:43.664087+0000) 2022-01-31T22:11:43.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:43 smithi167 conmon[60316]: debug 2022-01-31T22:11:43.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:43.664015+0000) 2022-01-31T22:11:44.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:43 smithi167 conmon[49112]: debug 2022-01-31T22:11:43.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:43.745147+0000) 2022-01-31T22:11:44.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:44 smithi171 conmon[46715]: debug 2022-01-31T22:11:44.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:44.162019+0000) 2022-01-31T22:11:44.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:44 smithi171 conmon[51620]: debug 2022-01-31T22:11:44.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:44.017139+0000) 2022-01-31T22:11:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:44 smithi171 conmon[41853]: debug 2022-01-31T22:11:44.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:44.369132+0000) 2022-01-31T22:11:44.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:44 smithi167 conmon[54076]: debug 2022-01-31T22:11:44.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:44.664280+0000) 2022-01-31T22:11:44.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:44 smithi167 conmon[60316]: debug 2022-01-31T22:11:44.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:44.664212+0000) 2022-01-31T22:11:45.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:44 smithi167 conmon[49112]: debug 2022-01-31T22:11:44.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:44.745306+0000) 2022-01-31T22:11:45.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[46715]: debug 2022-01-31T22:11:45.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.162204+0000) 2022-01-31T22:11:45.327 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[51620]: debug 2022-01-31T22:11:45.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.017310+0000) 2022-01-31T22:11:45.581 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[41853]: debug 2022-01-31T22:11:45.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.369304+0000) 2022-01-31T22:11:45.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:45 smithi167 conmon[49112]: debug 2022-01-31T22:11:45.581+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.582022+0000) 2022-01-31T22:11:45.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:45 smithi167 conmon[54076]: debug 2022-01-31T22:11:45.581+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.581859+0000) 2022-01-31T22:11:45.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:45 smithi167 conmon[54076]: debug 2022-01-31T22:11:45.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.664486+0000) 2022-01-31T22:11:45.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:45 smithi167 conmon[60316]: debug 2022-01-31T22:11:45.582+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.582787+0000) 2022-01-31T22:11:45.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:45 smithi167 conmon[60316]: debug 2022-01-31T22:11:45.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.664439+0000) 2022-01-31T22:11:45.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[35325]: debug 2022-01-31T22:11:45.601+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190061 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[41853]: debug 2022-01-31T22:11:45.580+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.581580+0000) 2022-01-31T22:11:45.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[46715]: debug 2022-01-31T22:11:45.582+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.582748+0000) 2022-01-31T22:11:45.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:45 smithi171 conmon[51620]: debug 2022-01-31T22:11:45.582+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.582982+0000) 2022-01-31T22:11:46.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:45 smithi167 conmon[49112]: debug 2022-01-31T22:11:45.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:45.745520+0000) 2022-01-31T22:11:46.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:46 smithi171 conmon[46715]: debug 2022-01-31T22:11:46.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:46.162382+0000) 2022-01-31T22:11:46.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:46 smithi171 conmon[51620]: debug 2022-01-31T22:11:46.016+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:46.017495+0000) 2022-01-31T22:11:46.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:46 smithi171 conmon[41853]: debug 2022-01-31T22:11:46.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:46.369469+0000) 2022-01-31T22:11:46.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:46 smithi167 conmon[60316]: debug 2022-01-31T22:11:46.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:46.664595+0000) 2022-01-31T22:11:46.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:46 smithi167 conmon[49112]: debug 2022-01-31T22:11:46.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:46.745691+0000) 2022-01-31T22:11:46.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:46 smithi167 conmon[54076]: debug 2022-01-31T22:11:46.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:46.664698+0000) 2022-01-31T22:11:47.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:47 smithi171 conmon[46715]: debug 2022-01-31T22:11:47.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:47.162552+0000) 2022-01-31T22:11:47.327 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:47 smithi171 conmon[51620]: debug 2022-01-31T22:11:47.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:47.017657+0000) 2022-01-31T22:11:47.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:47 smithi171 conmon[41853]: debug 2022-01-31T22:11:47.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:47.369625+0000) 2022-01-31T22:11:47.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:47 smithi167 conmon[54076]: debug 2022-01-31T22:11:47.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:47.664797+0000) 2022-01-31T22:11:47.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:47 smithi167 conmon[60316]: debug 2022-01-31T22:11:47.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:47.664722+0000) 2022-01-31T22:11:48.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:47 smithi167 conmon[49112]: debug 2022-01-31T22:11:47.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:47.745830+0000) 2022-01-31T22:11:48.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:48 smithi171 conmon[46715]: debug 2022-01-31T22:11:48.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:48.162771+0000) 2022-01-31T22:11:48.327 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:48 smithi171 conmon[51620]: debug 2022-01-31T22:11:48.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:48.017851+0000) 2022-01-31T22:11:48.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:48 smithi171 conmon[41853]: debug 2022-01-31T22:11:48.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:48.369830+0000) 2022-01-31T22:11:48.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:48 smithi167 conmon[54076]: debug 2022-01-31T22:11:48.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:48.664927+0000) 2022-01-31T22:11:48.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:48 smithi167 conmon[60316]: debug 2022-01-31T22:11:48.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:48.664845+0000) 2022-01-31T22:11:49.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:48 smithi167 conmon[49112]: debug 2022-01-31T22:11:48.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:48.746011+0000) 2022-01-31T22:11:49.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:49 smithi171 conmon[46715]: debug 2022-01-31T22:11:49.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:49.162962+0000) 2022-01-31T22:11:49.327 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:49 smithi171 conmon[51620]: debug 2022-01-31T22:11:49.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:49.018052+0000) 2022-01-31T22:11:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:49 smithi171 conmon[41853]: debug 2022-01-31T22:11:49.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:49.369984+0000) 2022-01-31T22:11:49.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:49 smithi167 conmon[54076]: debug 2022-01-31T22:11:49.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:49.665053+0000) 2022-01-31T22:11:49.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:49 smithi167 conmon[60316]: debug 2022-01-31T22:11:49.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:49.664974+0000) 2022-01-31T22:11:50.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:49 smithi167 conmon[49112]: debug 2022-01-31T22:11:49.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:49.746224+0000) 2022-01-31T22:11:50.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[46715]: debug 2022-01-31T22:11:50.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.163146+0000) 2022-01-31T22:11:50.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[51620]: debug 2022-01-31T22:11:50.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.018285+0000) 2022-01-31T22:11:50.604 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[41853]: debug 2022-01-31T22:11:50.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.370135+0000) 2022-01-31T22:11:50.604 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[41853]: debug 2022-01-31T22:11:50.603+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.604203+0000) 2022-01-31T22:11:50.745 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:50 smithi167 conmon[49112]: debug 2022-01-31T22:11:50.605+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.605544+0000) 2022-01-31T22:11:50.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:50 smithi167 conmon[54076]: debug 2022-01-31T22:11:50.604+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.604698+0000) 2022-01-31T22:11:50.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:50 smithi167 conmon[54076]: debug 2022-01-31T22:11:50.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.665196+0000) 2022-01-31T22:11:50.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:50 smithi167 conmon[60316]: debug 2022-01-31T22:11:50.604+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.604979+0000) 2022-01-31T22:11:50.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:50 smithi167 conmon[60316]: debug 2022-01-31T22:11:50.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.665114+0000) 2022-01-31T22:11:50.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[46715]: debug 2022-01-31T22:11:50.604+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.605200+0000) 2022-01-31T22:11:50.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[35325]: debug 2022-01-31T22:11:50.624+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190172 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:50.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:50 smithi171 conmon[51620]: debug 2022-01-31T22:11:50.604+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.605451+0000) 2022-01-31T22:11:51.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:50 smithi167 conmon[49112]: debug 2022-01-31T22:11:50.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:50.746388+0000) 2022-01-31T22:11:51.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:51 smithi171 conmon[46715]: debug 2022-01-31T22:11:51.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:51.163333+0000) 2022-01-31T22:11:51.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:51 smithi171 conmon[51620]: debug 2022-01-31T22:11:51.017+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:51.018495+0000) 2022-01-31T22:11:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:51 smithi171 conmon[41853]: debug 2022-01-31T22:11:51.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:51.370299+0000) 2022-01-31T22:11:51.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:51 smithi167 conmon[54076]: debug 2022-01-31T22:11:51.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:51.665341+0000) 2022-01-31T22:11:51.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:51 smithi167 conmon[60316]: debug 2022-01-31T22:11:51.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:51.665309+0000) 2022-01-31T22:11:52.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:51 smithi167 conmon[49112]: debug 2022-01-31T22:11:51.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:51.746519+0000) 2022-01-31T22:11:52.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:52 smithi171 conmon[46715]: debug 2022-01-31T22:11:52.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:52.163485+0000) 2022-01-31T22:11:52.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:52 smithi171 conmon[51620]: debug 2022-01-31T22:11:52.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:52.018673+0000) 2022-01-31T22:11:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:52 smithi171 conmon[41853]: debug 2022-01-31T22:11:52.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:52.370449+0000) 2022-01-31T22:11:52.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:52 smithi167 conmon[54076]: debug 2022-01-31T22:11:52.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:52.665556+0000) 2022-01-31T22:11:52.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:52 smithi167 conmon[60316]: debug 2022-01-31T22:11:52.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:52.665556+0000) 2022-01-31T22:11:53.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:52 smithi167 conmon[49112]: debug 2022-01-31T22:11:52.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:52.746687+0000) 2022-01-31T22:11:53.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:53 smithi171 conmon[46715]: debug 2022-01-31T22:11:53.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:53.163668+0000) 2022-01-31T22:11:53.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:53 smithi171 conmon[51620]: debug 2022-01-31T22:11:53.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:53.018863+0000) 2022-01-31T22:11:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:53 smithi171 conmon[41853]: debug 2022-01-31T22:11:53.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:53.370665+0000) 2022-01-31T22:11:53.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:53 smithi167 conmon[60316]: debug 2022-01-31T22:11:53.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:53.665715+0000) 2022-01-31T22:11:53.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:53 smithi167 conmon[54076]: debug 2022-01-31T22:11:53.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:53.665715+0000) 2022-01-31T22:11:54.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:53 smithi167 conmon[49112]: debug 2022-01-31T22:11:53.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:53.746841+0000) 2022-01-31T22:11:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:53 smithi171 conmon[35325]: debug 2022-01-31T22:11:53.812+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:11:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:54 smithi171 conmon[51620]: debug 2022-01-31T22:11:54.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:54.019025+0000) 2022-01-31T22:11:54.370 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:54 smithi171 conmon[46715]: debug 2022-01-31T22:11:54.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:54.163883+0000) 2022-01-31T22:11:54.746 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:54 smithi167 conmon[54076]: debug 2022-01-31T22:11:54.664+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:54.665892+0000) 2022-01-31T22:11:54.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:54 smithi167 conmon[60316]: debug 2022-01-31T22:11:54.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:54.665892+0000) 2022-01-31T22:11:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:54 smithi171 conmon[41853]: debug 2022-01-31T22:11:54.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:54.370829+0000) 2022-01-31T22:11:55.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:54 smithi167 conmon[49112]: debug 2022-01-31T22:11:54.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:54.747040+0000) 2022-01-31T22:11:55.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[46715]: debug 2022-01-31T22:11:55.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.164028+0000) 2022-01-31T22:11:55.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[51620]: debug 2022-01-31T22:11:55.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.019181+0000) 2022-01-31T22:11:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[41853]: debug 2022-01-31T22:11:55.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.370980+0000) 2022-01-31T22:11:55.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:55 smithi167 conmon[49112]: debug 2022-01-31T22:11:55.625+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.626842+0000) 2022-01-31T22:11:55.935 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:55 smithi167 conmon[54076]: debug 2022-01-31T22:11:55.625+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.627146+0000) 2022-01-31T22:11:55.936 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:55 smithi167 conmon[54076]: debug 2022-01-31T22:11:55.665+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.666142+0000) 2022-01-31T22:11:55.936 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:55 smithi167 conmon[60316]: debug 2022-01-31T22:11:55.625+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.627251+0000) 2022-01-31T22:11:55.937 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:55 smithi167 conmon[60316]: debug 2022-01-31T22:11:55.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.666142+0000) 2022-01-31T22:11:56.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:55 smithi167 conmon[49112]: debug 2022-01-31T22:11:55.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.747229+0000) 2022-01-31T22:11:56.018 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[35325]: debug 2022-01-31T22:11:55.645+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190283 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:11:56.019 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[41853]: debug 2022-01-31T22:11:55.626+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.627167+0000) 2022-01-31T22:11:56.020 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[46715]: debug 2022-01-31T22:11:55.627+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.627836+0000) 2022-01-31T22:11:56.020 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:55 smithi171 conmon[51620]: debug 2022-01-31T22:11:55.627+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:55.628610+0000) 2022-01-31T22:11:56.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:56 smithi171 conmon[46715]: debug 2022-01-31T22:11:56.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:56.164156+0000) 2022-01-31T22:11:56.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:56 smithi171 conmon[51620]: debug 2022-01-31T22:11:56.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:56.019296+0000) 2022-01-31T22:11:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:56 smithi171 conmon[41853]: debug 2022-01-31T22:11:56.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:56.371079+0000) 2022-01-31T22:11:56.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:56 smithi167 conmon[60316]: debug 2022-01-31T22:11:56.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:56.666315+0000) 2022-01-31T22:11:56.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:56 smithi167 conmon[54076]: debug 2022-01-31T22:11:56.665+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:56.666315+0000) 2022-01-31T22:11:57.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:56 smithi167 conmon[49112]: debug 2022-01-31T22:11:56.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:56.747402+0000) 2022-01-31T22:11:57.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:57 smithi171 conmon[46715]: debug 2022-01-31T22:11:57.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:57.164279+0000) 2022-01-31T22:11:57.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:57 smithi171 conmon[51620]: debug 2022-01-31T22:11:57.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:57.019455+0000) 2022-01-31T22:11:57.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:57 smithi171 conmon[41853]: debug 2022-01-31T22:11:57.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:57.371218+0000) 2022-01-31T22:11:57.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:57 smithi167 conmon[54076]: debug 2022-01-31T22:11:57.665+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:57.666552+0000) 2022-01-31T22:11:57.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:57 smithi167 conmon[60316]: debug 2022-01-31T22:11:57.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:57.666552+0000) 2022-01-31T22:11:58.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:57 smithi167 conmon[49112]: debug 2022-01-31T22:11:57.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:57.747594+0000) 2022-01-31T22:11:58.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:58 smithi171 conmon[46715]: debug 2022-01-31T22:11:58.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:58.164458+0000) 2022-01-31T22:11:58.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:58 smithi171 conmon[51620]: debug 2022-01-31T22:11:58.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:58.019643+0000) 2022-01-31T22:11:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:58 smithi171 conmon[41853]: debug 2022-01-31T22:11:58.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:58.371378+0000) 2022-01-31T22:11:58.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:58 smithi167 conmon[54076]: debug 2022-01-31T22:11:58.665+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:58.666703+0000) 2022-01-31T22:11:58.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:58 smithi167 conmon[60316]: debug 2022-01-31T22:11:58.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:58.666703+0000) 2022-01-31T22:11:59.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:58 smithi167 conmon[49112]: debug 2022-01-31T22:11:58.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:58.747790+0000) 2022-01-31T22:11:59.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:11:59 smithi171 conmon[46715]: debug 2022-01-31T22:11:59.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:59.164664+0000) 2022-01-31T22:11:59.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:11:59 smithi171 conmon[51620]: debug 2022-01-31T22:11:59.018+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:59.019823+0000) 2022-01-31T22:11:59.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:11:59 smithi171 conmon[41853]: debug 2022-01-31T22:11:59.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:59.371557+0000) 2022-01-31T22:11:59.747 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:11:59 smithi167 conmon[60316]: debug 2022-01-31T22:11:59.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:59.666872+0000) 2022-01-31T22:11:59.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:11:59 smithi167 conmon[54076]: debug 2022-01-31T22:11:59.665+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:59.666872+0000) 2022-01-31T22:12:00.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:11:59 smithi167 conmon[49112]: debug 2022-01-31T22:11:59.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:11:59.747985+0000) 2022-01-31T22:12:00.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[46715]: debug 2022-01-31T22:12:00.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.164864+0000) 2022-01-31T22:12:00.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[51620]: debug 2022-01-31T22:12:00.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.019990+0000) 2022-01-31T22:12:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[41853]: debug 2022-01-31T22:12:00.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.371711+0000) 2022-01-31T22:12:00.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:00 smithi167 conmon[49112]: debug 2022-01-31T22:12:00.647+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.648636+0000) 2022-01-31T22:12:00.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:00 smithi167 conmon[54076]: debug 2022-01-31T22:12:00.648+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.650090+0000) 2022-01-31T22:12:00.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:00 smithi167 conmon[54076]: debug 2022-01-31T22:12:00.665+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.667105+0000) 2022-01-31T22:12:00.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:00 smithi167 conmon[60316]: debug 2022-01-31T22:12:00.648+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.649900+0000) 2022-01-31T22:12:00.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:00 smithi167 conmon[60316]: debug 2022-01-31T22:12:00.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.667108+0000) 2022-01-31T22:12:01.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:00 smithi167 conmon[49112]: debug 2022-01-31T22:12:00.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.748134+0000) 2022-01-31T22:12:01.019 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[35325]: debug 2022-01-31T22:12:00.667+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190395 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:01.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[41853]: debug 2022-01-31T22:12:00.646+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.648408+0000) 2022-01-31T22:12:01.020 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[46715]: debug 2022-01-31T22:12:00.646+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.648600+0000) 2022-01-31T22:12:01.021 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:00 smithi171 conmon[51620]: debug 2022-01-31T22:12:00.647+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:00.649196+0000) 2022-01-31T22:12:01.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:01 smithi171 conmon[46715]: debug 2022-01-31T22:12:01.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:01.165048+0000) 2022-01-31T22:12:01.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:01 smithi171 conmon[51620]: debug 2022-01-31T22:12:01.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:01.020228+0000) 2022-01-31T22:12:01.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:01 smithi171 conmon[41853]: debug 2022-01-31T22:12:01.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:01.371927+0000) 2022-01-31T22:12:01.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:01 smithi167 conmon[54076]: debug 2022-01-31T22:12:01.666+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:01.667275+0000) 2022-01-31T22:12:01.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:01 smithi167 conmon[60316]: debug 2022-01-31T22:12:01.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:01.667275+0000) 2022-01-31T22:12:02.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:01 smithi167 conmon[49112]: debug 2022-01-31T22:12:01.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:01.748288+0000) 2022-01-31T22:12:02.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:02 smithi171 conmon[46715]: debug 2022-01-31T22:12:02.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:02.165201+0000) 2022-01-31T22:12:02.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:02 smithi171 conmon[51620]: debug 2022-01-31T22:12:02.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:02.020417+0000) 2022-01-31T22:12:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:02 smithi171 conmon[41853]: debug 2022-01-31T22:12:02.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:02.372078+0000) 2022-01-31T22:12:02.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:02 smithi167 conmon[54076]: debug 2022-01-31T22:12:02.666+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:02.667528+0000) 2022-01-31T22:12:02.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:02 smithi167 conmon[60316]: debug 2022-01-31T22:12:02.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:02.667528+0000) 2022-01-31T22:12:03.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:02 smithi167 conmon[49112]: debug 2022-01-31T22:12:02.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:02.748520+0000) 2022-01-31T22:12:03.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:03 smithi171 conmon[46715]: debug 2022-01-31T22:12:03.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:03.165416+0000) 2022-01-31T22:12:03.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:03 smithi171 conmon[51620]: debug 2022-01-31T22:12:03.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:03.020622+0000) 2022-01-31T22:12:03.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:03 smithi171 conmon[41853]: debug 2022-01-31T22:12:03.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:03.372276+0000) 2022-01-31T22:12:03.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:03 smithi167 conmon[54076]: debug 2022-01-31T22:12:03.666+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:03.667738+0000) 2022-01-31T22:12:03.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:03 smithi167 conmon[60316]: debug 2022-01-31T22:12:03.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:03.667739+0000) 2022-01-31T22:12:04.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:03 smithi167 conmon[49112]: debug 2022-01-31T22:12:03.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:03.748676+0000) 2022-01-31T22:12:04.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:04 smithi171 conmon[46715]: debug 2022-01-31T22:12:04.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:04.165676+0000) 2022-01-31T22:12:04.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:04 smithi171 conmon[51620]: debug 2022-01-31T22:12:04.019+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:04.020774+0000) 2022-01-31T22:12:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:04 smithi171 conmon[41853]: debug 2022-01-31T22:12:04.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:04.372408+0000) 2022-01-31T22:12:04.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:04 smithi167 conmon[54076]: debug 2022-01-31T22:12:04.667+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:04.667950+0000) 2022-01-31T22:12:04.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:04 smithi167 conmon[60316]: debug 2022-01-31T22:12:04.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:04.667950+0000) 2022-01-31T22:12:05.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:04 smithi167 conmon[49112]: debug 2022-01-31T22:12:04.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:04.748861+0000) 2022-01-31T22:12:05.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[46715]: debug 2022-01-31T22:12:05.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.165897+0000) 2022-01-31T22:12:05.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[51620]: debug 2022-01-31T22:12:05.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.020950+0000) 2022-01-31T22:12:05.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[41853]: debug 2022-01-31T22:12:05.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.372561+0000) 2022-01-31T22:12:05.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:05 smithi167 conmon[49112]: debug 2022-01-31T22:12:05.669+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.670812+0000) 2022-01-31T22:12:05.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:05 smithi167 conmon[60316]: debug 2022-01-31T22:12:05.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.668083+0000) 2022-01-31T22:12:05.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:05 smithi167 conmon[60316]: debug 2022-01-31T22:12:05.670+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.672020+0000) 2022-01-31T22:12:05.750 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:05 smithi167 conmon[54076]: debug 2022-01-31T22:12:05.666+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.668083+0000) 2022-01-31T22:12:05.750 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:05 smithi167 conmon[54076]: debug 2022-01-31T22:12:05.670+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.672468+0000) 2022-01-31T22:12:06.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:05 smithi167 conmon[49112]: debug 2022-01-31T22:12:05.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.749018+0000) 2022-01-31T22:12:06.020 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[35325]: debug 2022-01-31T22:12:05.690+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190506 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:06.020 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[41853]: debug 2022-01-31T22:12:05.669+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.671279+0000) 2022-01-31T22:12:06.021 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[46715]: debug 2022-01-31T22:12:05.669+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.671391+0000) 2022-01-31T22:12:06.022 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:05 smithi171 conmon[51620]: debug 2022-01-31T22:12:05.669+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:05.671611+0000) 2022-01-31T22:12:06.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:06 smithi171 conmon[46715]: debug 2022-01-31T22:12:06.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:06.165998+0000) 2022-01-31T22:12:06.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:06 smithi171 conmon[51620]: debug 2022-01-31T22:12:06.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:06.021134+0000) 2022-01-31T22:12:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:06 smithi171 conmon[41853]: debug 2022-01-31T22:12:06.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:06.372768+0000) 2022-01-31T22:12:06.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:06 smithi167 conmon[54076]: debug 2022-01-31T22:12:06.666+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:06.668316+0000) 2022-01-31T22:12:06.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:06 smithi167 conmon[60316]: debug 2022-01-31T22:12:06.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:06.668316+0000) 2022-01-31T22:12:07.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:06 smithi167 conmon[49112]: debug 2022-01-31T22:12:06.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:06.749170+0000) 2022-01-31T22:12:07.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:07 smithi171 conmon[46715]: debug 2022-01-31T22:12:07.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:07.166133+0000) 2022-01-31T22:12:07.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:07 smithi171 conmon[51620]: debug 2022-01-31T22:12:07.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:07.021297+0000) 2022-01-31T22:12:07.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:07 smithi171 conmon[41853]: debug 2022-01-31T22:12:07.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:07.372918+0000) 2022-01-31T22:12:07.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:07 smithi167 conmon[54076]: debug 2022-01-31T22:12:07.667+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:07.668485+0000) 2022-01-31T22:12:07.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:07 smithi167 conmon[60316]: debug 2022-01-31T22:12:07.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:07.668485+0000) 2022-01-31T22:12:08.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:07 smithi167 conmon[49112]: debug 2022-01-31T22:12:07.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:07.749328+0000) 2022-01-31T22:12:08.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:08 smithi171 conmon[46715]: debug 2022-01-31T22:12:08.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:08.166315+0000) 2022-01-31T22:12:08.331 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:08 smithi171 conmon[51620]: debug 2022-01-31T22:12:08.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:08.021490+0000) 2022-01-31T22:12:08.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:08 smithi171 conmon[41853]: debug 2022-01-31T22:12:08.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:08.373105+0000) 2022-01-31T22:12:08.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:08 smithi167 conmon[54076]: debug 2022-01-31T22:12:08.668+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:08.668732+0000) 2022-01-31T22:12:08.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:08 smithi167 conmon[60316]: debug 2022-01-31T22:12:08.668+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:08.668732+0000) 2022-01-31T22:12:09.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:08 smithi167 conmon[49112]: debug 2022-01-31T22:12:08.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:08.749538+0000) 2022-01-31T22:12:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:08 smithi171 conmon[35325]: debug 2022-01-31T22:12:08.812+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:12:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:09 smithi171 conmon[51620]: debug 2022-01-31T22:12:09.020+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:09.021678+0000) 2022-01-31T22:12:09.372 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:09 smithi171 conmon[46715]: debug 2022-01-31T22:12:09.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:09.166524+0000) 2022-01-31T22:12:09.666 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:09 smithi171 conmon[41853]: debug 2022-01-31T22:12:09.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:09.373265+0000) 2022-01-31T22:12:09.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:09 smithi167 conmon[54076]: debug 2022-01-31T22:12:09.667+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:09.668933+0000) 2022-01-31T22:12:09.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:09 smithi167 conmon[60316]: debug 2022-01-31T22:12:09.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:09.668933+0000) 2022-01-31T22:12:10.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:09 smithi167 conmon[49112]: debug 2022-01-31T22:12:09.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:09.749735+0000) 2022-01-31T22:12:10.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[46715]: debug 2022-01-31T22:12:10.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.166709+0000) 2022-01-31T22:12:10.331 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[51620]: debug 2022-01-31T22:12:10.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.021855+0000) 2022-01-31T22:12:10.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[41853]: debug 2022-01-31T22:12:10.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.373440+0000) 2022-01-31T22:12:10.749 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:10 smithi167 conmon[49112]: debug 2022-01-31T22:12:10.693+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.694724+0000) 2022-01-31T22:12:10.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:10 smithi167 conmon[60316]: debug 2022-01-31T22:12:10.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.669104+0000) 2022-01-31T22:12:10.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:10 smithi167 conmon[60316]: debug 2022-01-31T22:12:10.693+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.695007+0000) 2022-01-31T22:12:10.751 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:10 smithi167 conmon[54076]: debug 2022-01-31T22:12:10.667+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.669104+0000) 2022-01-31T22:12:10.751 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:10 smithi167 conmon[54076]: debug 2022-01-31T22:12:10.693+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.695108+0000) 2022-01-31T22:12:11.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:10 smithi167 conmon[49112]: debug 2022-01-31T22:12:10.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.749951+0000) 2022-01-31T22:12:11.021 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[35325]: debug 2022-01-31T22:12:10.713+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190618 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:11.022 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[41853]: debug 2022-01-31T22:12:10.693+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.693787+0000) 2022-01-31T22:12:11.022 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[46715]: debug 2022-01-31T22:12:10.693+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.694123+0000) 2022-01-31T22:12:11.023 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:10 smithi171 conmon[51620]: debug 2022-01-31T22:12:10.694+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:10.695253+0000) 2022-01-31T22:12:11.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:11 smithi171 conmon[46715]: debug 2022-01-31T22:12:11.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:11.166919+0000) 2022-01-31T22:12:11.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:11 smithi171 conmon[51620]: debug 2022-01-31T22:12:11.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:11.022004+0000) 2022-01-31T22:12:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:11 smithi171 conmon[41853]: debug 2022-01-31T22:12:11.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:11.373631+0000) 2022-01-31T22:12:11.718 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:11 smithi167 conmon[54076]: debug 2022-01-31T22:12:11.667+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:11.669354+0000) 2022-01-31T22:12:11.718 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:11 smithi167 conmon[60316]: debug 2022-01-31T22:12:11.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:11.669354+0000) 2022-01-31T22:12:12.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:11 smithi167 conmon[49112]: debug 2022-01-31T22:12:11.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:11.750098+0000) 2022-01-31T22:12:12.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:12 smithi171 conmon[46715]: debug 2022-01-31T22:12:12.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:12.167078+0000) 2022-01-31T22:12:12.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:12 smithi171 conmon[51620]: debug 2022-01-31T22:12:12.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:12.022115+0000) 2022-01-31T22:12:12.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:12 smithi171 conmon[41853]: debug 2022-01-31T22:12:12.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:12.373821+0000) 2022-01-31T22:12:12.725 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:12 smithi167 conmon[54076]: debug 2022-01-31T22:12:12.668+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:12.669542+0000) 2022-01-31T22:12:12.726 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:12 smithi167 conmon[60316]: debug 2022-01-31T22:12:12.668+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:12.669541+0000) 2022-01-31T22:12:13.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:12 smithi167 conmon[49112]: debug 2022-01-31T22:12:12.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:12.750259+0000) 2022-01-31T22:12:13.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:13 smithi171 conmon[46715]: debug 2022-01-31T22:12:13.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:13.167266+0000) 2022-01-31T22:12:13.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:13 smithi171 conmon[51620]: debug 2022-01-31T22:12:13.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:13.022301+0000) 2022-01-31T22:12:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:13 smithi171 conmon[41853]: debug 2022-01-31T22:12:13.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:13.373987+0000) 2022-01-31T22:12:13.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:13 smithi167 conmon[54076]: debug 2022-01-31T22:12:13.669+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:13.669722+0000) 2022-01-31T22:12:13.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:13 smithi167 conmon[60316]: debug 2022-01-31T22:12:13.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:13.669722+0000) 2022-01-31T22:12:14.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:13 smithi167 conmon[49112]: debug 2022-01-31T22:12:13.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:13.750451+0000) 2022-01-31T22:12:14.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:14 smithi171 conmon[46715]: debug 2022-01-31T22:12:14.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:14.167433+0000) 2022-01-31T22:12:14.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:14 smithi171 conmon[51620]: debug 2022-01-31T22:12:14.021+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:14.022502+0000) 2022-01-31T22:12:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:14 smithi171 conmon[41853]: debug 2022-01-31T22:12:14.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:14.374174+0000) 2022-01-31T22:12:14.731 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:14 smithi167 conmon[54076]: debug 2022-01-31T22:12:14.669+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:14.669932+0000) 2022-01-31T22:12:14.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:14 smithi167 conmon[60316]: debug 2022-01-31T22:12:14.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:14.669932+0000) 2022-01-31T22:12:15.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:14 smithi167 conmon[49112]: debug 2022-01-31T22:12:14.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:14.750583+0000) 2022-01-31T22:12:15.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[46715]: debug 2022-01-31T22:12:15.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.167591+0000) 2022-01-31T22:12:15.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[51620]: debug 2022-01-31T22:12:15.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.022685+0000) 2022-01-31T22:12:15.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[51620]: 2022-01-31T22:12:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[41853]: debug 2022-01-31T22:12:15.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.374331+0000) 2022-01-31T22:12:15.721 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:15 smithi167 conmon[49112]: debug 2022-01-31T22:12:15.715+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.717215+0000) 2022-01-31T22:12:15.722 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:15 smithi167 conmon[54076]: debug 2022-01-31T22:12:15.669+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.670147+0000) 2022-01-31T22:12:15.722 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:15 smithi167 conmon[54076]: debug 2022-01-31T22:12:15.714+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.716326+0000) 2022-01-31T22:12:15.723 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:15 smithi167 conmon[60316]: debug 2022-01-31T22:12:15.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.670147+0000) 2022-01-31T22:12:15.723 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:15 smithi167 conmon[60316]: debug 2022-01-31T22:12:15.715+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.716547+0000) 2022-01-31T22:12:16.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:15 smithi167 conmon[49112]: debug 2022-01-31T22:12:15.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.750723+0000) 2022-01-31T22:12:16.022 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[35325]: debug 2022-01-31T22:12:15.736+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190730 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:16.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[41853]: debug 2022-01-31T22:12:15.715+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.715976+0000) 2022-01-31T22:12:16.023 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[46715]: debug 2022-01-31T22:12:15.716+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.717034+0000) 2022-01-31T22:12:16.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:15 smithi171 conmon[51620]: debug 2022-01-31T22:12:15.717+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:15.717932+0000) 2022-01-31T22:12:16.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:16 smithi171 conmon[46715]: debug 2022-01-31T22:12:16.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:16.167793+0000) 2022-01-31T22:12:16.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:16 smithi171 conmon[51620]: debug 2022-01-31T22:12:16.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:16.022842+0000) 2022-01-31T22:12:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:16 smithi171 conmon[41853]: debug 2022-01-31T22:12:16.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:16.374502+0000) 2022-01-31T22:12:16.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:16 smithi167 conmon[49112]: debug 2022-01-31T22:12:16.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:16.750894+0000) 2022-01-31T22:12:16.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:16 smithi167 conmon[54076]: debug 2022-01-31T22:12:16.669+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:16.670339+0000) 2022-01-31T22:12:16.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:16 smithi167 conmon[60316]: debug 2022-01-31T22:12:16.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:16.670339+0000) 2022-01-31T22:12:17.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:17 smithi171 conmon[46715]: debug 2022-01-31T22:12:17.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:17.167948+0000) 2022-01-31T22:12:17.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:17 smithi171 conmon[51620]: debug 2022-01-31T22:12:17.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:17.022979+0000) 2022-01-31T22:12:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:17 smithi171 conmon[41853]: debug 2022-01-31T22:12:17.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:17.374690+0000) 2022-01-31T22:12:17.738 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:17 smithi167 conmon[54076]: debug 2022-01-31T22:12:17.670+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:17.670530+0000) 2022-01-31T22:12:17.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:17 smithi167 conmon[60316]: debug 2022-01-31T22:12:17.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:17.670527+0000) 2022-01-31T22:12:18.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:17 smithi167 conmon[49112]: debug 2022-01-31T22:12:17.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:17.751055+0000) 2022-01-31T22:12:18.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:18 smithi171 conmon[46715]: debug 2022-01-31T22:12:18.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:18.168145+0000) 2022-01-31T22:12:18.333 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:18 smithi171 conmon[51620]: debug 2022-01-31T22:12:18.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:18.023179+0000) 2022-01-31T22:12:18.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:18 smithi171 conmon[41853]: debug 2022-01-31T22:12:18.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:18.374916+0000) 2022-01-31T22:12:18.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:18 smithi167 conmon[54076]: debug 2022-01-31T22:12:18.670+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:18.670708+0000) 2022-01-31T22:12:18.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:18 smithi167 conmon[60316]: debug 2022-01-31T22:12:18.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:18.670709+0000) 2022-01-31T22:12:19.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:18 smithi167 conmon[49112]: debug 2022-01-31T22:12:18.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:18.751182+0000) 2022-01-31T22:12:19.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:19 smithi171 conmon[46715]: debug 2022-01-31T22:12:19.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:19.168348+0000) 2022-01-31T22:12:19.333 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:19 smithi171 conmon[51620]: debug 2022-01-31T22:12:19.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:19.023363+0000) 2022-01-31T22:12:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:19 smithi171 conmon[41853]: debug 2022-01-31T22:12:19.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:19.375082+0000) 2022-01-31T22:12:19.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:19 smithi167 conmon[54076]: debug 2022-01-31T22:12:19.670+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:19.670861+0000) 2022-01-31T22:12:19.745 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:19 smithi167 conmon[60316]: debug 2022-01-31T22:12:19.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:19.670861+0000) 2022-01-31T22:12:20.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:19 smithi167 conmon[49112]: debug 2022-01-31T22:12:19.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:19.751379+0000) 2022-01-31T22:12:20.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[46715]: debug 2022-01-31T22:12:20.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.168555+0000) 2022-01-31T22:12:20.335 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[51620]: debug 2022-01-31T22:12:20.022+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.023551+0000) 2022-01-31T22:12:20.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[41853]: debug 2022-01-31T22:12:20.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.375275+0000) 2022-01-31T22:12:20.740 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:20 smithi167 conmon[54076]: debug 2022-01-31T22:12:20.670+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.671053+0000) 2022-01-31T22:12:20.741 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:20 smithi167 conmon[60316]: debug 2022-01-31T22:12:20.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.671053+0000) 2022-01-31T22:12:21.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:20 smithi167 conmon[49112]: debug 2022-01-31T22:12:20.740+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.741216+0000) 2022-01-31T22:12:21.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:20 smithi167 conmon[49112]: debug 2022-01-31T22:12:20.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.751571+0000) 2022-01-31T22:12:21.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:20 smithi167 conmon[54076]: debug 2022-01-31T22:12:20.739+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.740195+0000) 2022-01-31T22:12:21.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:20 smithi167 conmon[60316]: debug 2022-01-31T22:12:20.739+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.739921+0000) 2022-01-31T22:12:21.023 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[35325]: debug 2022-01-31T22:12:20.759+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190841 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:21.023 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[41853]: debug 2022-01-31T22:12:20.738+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.739397+0000) 2022-01-31T22:12:21.024 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[46715]: debug 2022-01-31T22:12:20.739+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.739746+0000) 2022-01-31T22:12:21.024 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:20 smithi171 conmon[51620]: debug 2022-01-31T22:12:20.738+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:20.739505+0000) 2022-01-31T22:12:21.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:21 smithi171 conmon[46715]: debug 2022-01-31T22:12:21.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:21.168739+0000) 2022-01-31T22:12:21.333 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:21 smithi171 conmon[51620]: debug 2022-01-31T22:12:21.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:21.023742+0000) 2022-01-31T22:12:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:21 smithi171 conmon[41853]: debug 2022-01-31T22:12:21.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:21.375436+0000) 2022-01-31T22:12:21.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:21 smithi167 conmon[54076]: debug 2022-01-31T22:12:21.670+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:21.671283+0000) 2022-01-31T22:12:21.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:21 smithi167 conmon[60316]: debug 2022-01-31T22:12:21.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:21.671281+0000) 2022-01-31T22:12:22.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:21 smithi167 conmon[49112]: debug 2022-01-31T22:12:21.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:21.751677+0000) 2022-01-31T22:12:22.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:22 smithi171 conmon[46715]: debug 2022-01-31T22:12:22.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:22.168929+0000) 2022-01-31T22:12:22.333 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:22 smithi171 conmon[51620]: debug 2022-01-31T22:12:22.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:22.023912+0000) 2022-01-31T22:12:22.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:22 smithi171 conmon[41853]: debug 2022-01-31T22:12:22.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:22.375586+0000) 2022-01-31T22:12:22.751 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:22 smithi167 conmon[54076]: debug 2022-01-31T22:12:22.671+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:22.671454+0000) 2022-01-31T22:12:22.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:22 smithi167 conmon[60316]: debug 2022-01-31T22:12:22.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:22.671454+0000) 2022-01-31T22:12:23.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:22 smithi167 conmon[49112]: debug 2022-01-31T22:12:22.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:22.751885+0000) 2022-01-31T22:12:23.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:23 smithi171 conmon[46715]: debug 2022-01-31T22:12:23.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:23.169080+0000) 2022-01-31T22:12:23.334 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:23 smithi171 conmon[51620]: debug 2022-01-31T22:12:23.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:23.024097+0000) 2022-01-31T22:12:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:23 smithi171 conmon[41853]: debug 2022-01-31T22:12:23.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:23.375785+0000) 2022-01-31T22:12:23.751 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:23 smithi167 conmon[54076]: debug 2022-01-31T22:12:23.671+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:23.671613+0000) 2022-01-31T22:12:23.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:23 smithi167 conmon[60316]: debug 2022-01-31T22:12:23.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:23.671614+0000) 2022-01-31T22:12:24.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:23 smithi167 conmon[49112]: debug 2022-01-31T22:12:23.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:23.752076+0000) 2022-01-31T22:12:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:23 smithi171 conmon[35325]: debug 2022-01-31T22:12:23.813+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:12:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:24 smithi171 conmon[51620]: debug 2022-01-31T22:12:24.023+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:24.024297+0000) 2022-01-31T22:12:24.375 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:24 smithi171 conmon[46715]: debug 2022-01-31T22:12:24.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:24.169315+0000) 2022-01-31T22:12:24.751 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:24 smithi167 conmon[60316]: debug 2022-01-31T22:12:24.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:24.671794+0000) 2022-01-31T22:12:24.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:24 smithi167 conmon[54076]: debug 2022-01-31T22:12:24.671+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:24.671794+0000) 2022-01-31T22:12:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:24 smithi171 conmon[41853]: debug 2022-01-31T22:12:24.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:24.375972+0000) 2022-01-31T22:12:25.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:24 smithi167 conmon[49112]: debug 2022-01-31T22:12:24.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:24.752226+0000) 2022-01-31T22:12:25.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[46715]: debug 2022-01-31T22:12:25.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.169513+0000) 2022-01-31T22:12:25.334 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[51620]: debug 2022-01-31T22:12:25.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.024506+0000) 2022-01-31T22:12:25.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[41853]: debug 2022-01-31T22:12:25.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.376160+0000) 2022-01-31T22:12:25.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:25 smithi167 conmon[54076]: debug 2022-01-31T22:12:25.671+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.672015+0000) 2022-01-31T22:12:25.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:25 smithi167 conmon[60316]: debug 2022-01-31T22:12:25.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.672015+0000) 2022-01-31T22:12:26.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:25 smithi167 conmon[54076]: debug 2022-01-31T22:12:25.762+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.763357+0000) 2022-01-31T22:12:26.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:25 smithi167 conmon[60316]: debug 2022-01-31T22:12:25.762+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.763045+0000) 2022-01-31T22:12:26.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:25 smithi167 conmon[49112]: debug 2022-01-31T22:12:25.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.752417+0000) 2022-01-31T22:12:26.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:25 smithi167 conmon[49112]: debug 2022-01-31T22:12:25.763+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.764171+0000) 2022-01-31T22:12:26.023 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[35325]: debug 2022-01-31T22:12:25.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 190952 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:26.024 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[41853]: debug 2022-01-31T22:12:25.762+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.762782+0000) 2022-01-31T22:12:26.025 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[46715]: debug 2022-01-31T22:12:25.762+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.762990+0000) 2022-01-31T22:12:26.025 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:25 smithi171 conmon[51620]: debug 2022-01-31T22:12:25.762+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:25.763682+0000) 2022-01-31T22:12:26.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:26 smithi171 conmon[46715]: debug 2022-01-31T22:12:26.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:26.169650+0000) 2022-01-31T22:12:26.334 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:26 smithi171 conmon[51620]: debug 2022-01-31T22:12:26.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:26.024702+0000) 2022-01-31T22:12:26.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:26 smithi171 conmon[41853]: debug 2022-01-31T22:12:26.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:26.376365+0000) 2022-01-31T22:12:26.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:26 smithi167 conmon[54076]: debug 2022-01-31T22:12:26.671+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:26.672244+0000) 2022-01-31T22:12:26.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:26 smithi167 conmon[60316]: debug 2022-01-31T22:12:26.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:26.672244+0000) 2022-01-31T22:12:27.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:26 smithi167 conmon[49112]: debug 2022-01-31T22:12:26.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:26.752612+0000) 2022-01-31T22:12:27.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:27 smithi171 conmon[46715]: debug 2022-01-31T22:12:27.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:27.169852+0000) 2022-01-31T22:12:27.334 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:27 smithi171 conmon[51620]: debug 2022-01-31T22:12:27.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:27.024867+0000) 2022-01-31T22:12:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:27 smithi171 conmon[41853]: debug 2022-01-31T22:12:27.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:27.376519+0000) 2022-01-31T22:12:27.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:27 smithi167 conmon[54076]: debug 2022-01-31T22:12:27.671+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:27.672379+0000) 2022-01-31T22:12:27.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:27 smithi167 conmon[60316]: debug 2022-01-31T22:12:27.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:27.672379+0000) 2022-01-31T22:12:28.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:27 smithi167 conmon[49112]: debug 2022-01-31T22:12:27.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:27.752762+0000) 2022-01-31T22:12:28.333 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:28 smithi171 conmon[46715]: debug 2022-01-31T22:12:28.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:28.170009+0000) 2022-01-31T22:12:28.334 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:28 smithi171 conmon[51620]: debug 2022-01-31T22:12:28.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:28.024994+0000) 2022-01-31T22:12:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:28 smithi171 conmon[41853]: debug 2022-01-31T22:12:28.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:28.376706+0000) 2022-01-31T22:12:28.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:28 smithi167 conmon[60316]: debug 2022-01-31T22:12:28.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:28.672545+0000) 2022-01-31T22:12:28.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:28 smithi167 conmon[54076]: debug 2022-01-31T22:12:28.672+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:28.672546+0000) 2022-01-31T22:12:29.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:28 smithi167 conmon[49112]: debug 2022-01-31T22:12:28.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:28.752969+0000) 2022-01-31T22:12:29.334 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:29 smithi171 conmon[46715]: debug 2022-01-31T22:12:29.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:29.170188+0000) 2022-01-31T22:12:29.334 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:29 smithi171 conmon[51620]: debug 2022-01-31T22:12:29.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:29.025187+0000) 2022-01-31T22:12:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:29 smithi171 conmon[41853]: debug 2022-01-31T22:12:29.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:29.376845+0000) 2022-01-31T22:12:29.752 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:29 smithi167 conmon[60316]: debug 2022-01-31T22:12:29.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:29.672658+0000) 2022-01-31T22:12:29.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:29 smithi167 conmon[54076]: debug 2022-01-31T22:12:29.672+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:29.672658+0000) 2022-01-31T22:12:30.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:29 smithi167 conmon[49112]: debug 2022-01-31T22:12:29.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:29.753124+0000) 2022-01-31T22:12:30.334 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[46715]: debug 2022-01-31T22:12:30.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.170310+0000) 2022-01-31T22:12:30.335 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[51620]: debug 2022-01-31T22:12:30.024+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.025415+0000) 2022-01-31T22:12:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[41853]: debug 2022-01-31T22:12:30.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.377024+0000) 2022-01-31T22:12:30.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:30 smithi167 conmon[54076]: debug 2022-01-31T22:12:30.672+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.672839+0000) 2022-01-31T22:12:30.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:30 smithi167 conmon[60316]: debug 2022-01-31T22:12:30.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.672839+0000) 2022-01-31T22:12:31.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:30 smithi167 conmon[54076]: debug 2022-01-31T22:12:30.784+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.785126+0000) 2022-01-31T22:12:31.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:30 smithi167 conmon[60316]: debug 2022-01-31T22:12:30.785+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.785868+0000) 2022-01-31T22:12:31.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:30 smithi167 conmon[49112]: debug 2022-01-31T22:12:30.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.753279+0000) 2022-01-31T22:12:31.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:30 smithi167 conmon[49112]: debug 2022-01-31T22:12:30.785+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.785676+0000) 2022-01-31T22:12:31.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[35325]: debug 2022-01-31T22:12:30.805+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191063 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:31.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[41853]: debug 2022-01-31T22:12:30.784+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.785544+0000) 2022-01-31T22:12:31.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[46715]: debug 2022-01-31T22:12:30.783+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.784679+0000) 2022-01-31T22:12:31.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:30 smithi171 conmon[51620]: debug 2022-01-31T22:12:30.784+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:30.785484+0000) 2022-01-31T22:12:31.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:31 smithi171 conmon[51620]: debug 2022-01-31T22:12:31.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:31.025636+0000) 2022-01-31T22:12:31.376 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:31 smithi171 conmon[46715]: debug 2022-01-31T22:12:31.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:31.170414+0000) 2022-01-31T22:12:31.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:31 smithi167 conmon[60316]: debug 2022-01-31T22:12:31.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:31.673047+0000) 2022-01-31T22:12:31.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:31 smithi167 conmon[54076]: debug 2022-01-31T22:12:31.672+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:31.673047+0000) 2022-01-31T22:12:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:31 smithi171 conmon[41853]: debug 2022-01-31T22:12:31.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:31.377198+0000) 2022-01-31T22:12:32.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:31 smithi167 conmon[49112]: debug 2022-01-31T22:12:31.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:31.753456+0000) 2022-01-31T22:12:32.334 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:32 smithi171 conmon[46715]: debug 2022-01-31T22:12:32.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:32.170543+0000) 2022-01-31T22:12:32.335 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:32 smithi171 conmon[51620]: debug 2022-01-31T22:12:32.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:32.025812+0000) 2022-01-31T22:12:32.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:32 smithi171 conmon[41853]: debug 2022-01-31T22:12:32.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:32.377342+0000) 2022-01-31T22:12:32.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:32 smithi167 conmon[60316]: debug 2022-01-31T22:12:32.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:32.673220+0000) 2022-01-31T22:12:32.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:32 smithi167 conmon[54076]: debug 2022-01-31T22:12:32.673+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:32.673220+0000) 2022-01-31T22:12:33.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:32 smithi167 conmon[49112]: debug 2022-01-31T22:12:32.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:32.753654+0000) 2022-01-31T22:12:33.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:33 smithi171 conmon[46715]: debug 2022-01-31T22:12:33.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:33.170674+0000) 2022-01-31T22:12:33.335 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:33 smithi171 conmon[51620]: debug 2022-01-31T22:12:33.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:33.025977+0000) 2022-01-31T22:12:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:33 smithi171 conmon[41853]: debug 2022-01-31T22:12:33.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:33.377541+0000) 2022-01-31T22:12:33.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:33 smithi167 conmon[60316]: debug 2022-01-31T22:12:33.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:33.673440+0000) 2022-01-31T22:12:33.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:33 smithi167 conmon[54076]: debug 2022-01-31T22:12:33.672+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:33.673440+0000) 2022-01-31T22:12:34.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:33 smithi167 conmon[49112]: debug 2022-01-31T22:12:33.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:33.753846+0000) 2022-01-31T22:12:34.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:34 smithi171 conmon[46715]: debug 2022-01-31T22:12:34.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:34.170824+0000) 2022-01-31T22:12:34.335 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:34 smithi171 conmon[51620]: debug 2022-01-31T22:12:34.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:34.026144+0000) 2022-01-31T22:12:34.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:34 smithi171 conmon[41853]: debug 2022-01-31T22:12:34.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:34.377746+0000) 2022-01-31T22:12:34.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:34 smithi167 conmon[60316]: debug 2022-01-31T22:12:34.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:34.673649+0000) 2022-01-31T22:12:34.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:34 smithi167 conmon[54076]: debug 2022-01-31T22:12:34.673+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:34.673649+0000) 2022-01-31T22:12:35.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:34 smithi167 conmon[49112]: debug 2022-01-31T22:12:34.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:34.754040+0000) 2022-01-31T22:12:35.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[46715]: debug 2022-01-31T22:12:35.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.170926+0000) 2022-01-31T22:12:35.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[51620]: debug 2022-01-31T22:12:35.025+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.026376+0000) 2022-01-31T22:12:35.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[41853]: debug 2022-01-31T22:12:35.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.377959+0000) 2022-01-31T22:12:35.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:35 smithi167 conmon[54076]: debug 2022-01-31T22:12:35.673+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.673860+0000) 2022-01-31T22:12:35.754 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:35 smithi167 conmon[60316]: debug 2022-01-31T22:12:35.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.673862+0000) 2022-01-31T22:12:36.013 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:35 smithi167 conmon[54076]: debug 2022-01-31T22:12:35.808+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.808625+0000) 2022-01-31T22:12:36.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:35 smithi167 conmon[49112]: debug 2022-01-31T22:12:35.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.754223+0000) 2022-01-31T22:12:36.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:35 smithi167 conmon[49112]: debug 2022-01-31T22:12:35.808+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.809249+0000) 2022-01-31T22:12:36.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:35 smithi167 conmon[60316]: debug 2022-01-31T22:12:35.808+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.809047+0000) 2022-01-31T22:12:36.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[41853]: debug 2022-01-31T22:12:35.808+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.809052+0000) 2022-01-31T22:12:36.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[46715]: debug 2022-01-31T22:12:35.807+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.807834+0000) 2022-01-31T22:12:36.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[35325]: debug 2022-01-31T22:12:35.827+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191173 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:36.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:35 smithi171 conmon[51620]: debug 2022-01-31T22:12:35.807+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:35.808333+0000) 2022-01-31T22:12:36.110 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:36 smithi171 conmon[51620]: debug 2022-01-31T22:12:36.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:36.026553+0000) 2022-01-31T22:12:36.377 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:36 smithi171 conmon[46715]: debug 2022-01-31T22:12:36.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:36.171045+0000) 2022-01-31T22:12:36.753 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:36 smithi167 conmon[54076]: debug 2022-01-31T22:12:36.673+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:36.674089+0000) 2022-01-31T22:12:36.754 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:36 smithi167 conmon[60316]: debug 2022-01-31T22:12:36.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:36.674089+0000) 2022-01-31T22:12:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:36 smithi171 conmon[41853]: debug 2022-01-31T22:12:36.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:36.378077+0000) 2022-01-31T22:12:37.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:36 smithi167 conmon[49112]: debug 2022-01-31T22:12:36.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:36.754405+0000) 2022-01-31T22:12:37.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:37 smithi171 conmon[46715]: debug 2022-01-31T22:12:37.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:37.171199+0000) 2022-01-31T22:12:37.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:37 smithi171 conmon[51620]: debug 2022-01-31T22:12:37.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:37.026708+0000) 2022-01-31T22:12:37.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:37 smithi171 conmon[41853]: debug 2022-01-31T22:12:37.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:37.378216+0000) 2022-01-31T22:12:37.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:37 smithi167 conmon[54076]: debug 2022-01-31T22:12:37.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:37.674241+0000) 2022-01-31T22:12:37.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:37 smithi167 conmon[60316]: debug 2022-01-31T22:12:37.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:37.674240+0000) 2022-01-31T22:12:38.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:37 smithi167 conmon[49112]: debug 2022-01-31T22:12:37.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:37.754575+0000) 2022-01-31T22:12:38.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:38 smithi171 conmon[46715]: debug 2022-01-31T22:12:38.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:38.171363+0000) 2022-01-31T22:12:38.336 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:38 smithi171 conmon[51620]: debug 2022-01-31T22:12:38.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:38.026943+0000) 2022-01-31T22:12:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:38 smithi171 conmon[41853]: debug 2022-01-31T22:12:38.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:38.378407+0000) 2022-01-31T22:12:38.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:38 smithi167 conmon[54076]: debug 2022-01-31T22:12:38.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:38.674408+0000) 2022-01-31T22:12:38.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:38 smithi167 conmon[60316]: debug 2022-01-31T22:12:38.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:38.674408+0000) 2022-01-31T22:12:39.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:38 smithi167 conmon[49112]: debug 2022-01-31T22:12:38.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:38.754765+0000) 2022-01-31T22:12:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:38 smithi171 conmon[35325]: debug 2022-01-31T22:12:38.814+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:12:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:39 smithi171 conmon[51620]: debug 2022-01-31T22:12:39.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:39.027129+0000) 2022-01-31T22:12:39.378 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:39 smithi171 conmon[46715]: debug 2022-01-31T22:12:39.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:39.171510+0000) 2022-01-31T22:12:39.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:39 smithi167 conmon[54076]: debug 2022-01-31T22:12:39.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:39.674599+0000) 2022-01-31T22:12:39.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:39 smithi167 conmon[60316]: debug 2022-01-31T22:12:39.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:39.674598+0000) 2022-01-31T22:12:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:39 smithi171 conmon[41853]: debug 2022-01-31T22:12:39.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:39.378596+0000) 2022-01-31T22:12:40.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:39 smithi167 conmon[49112]: debug 2022-01-31T22:12:39.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:39.754951+0000) 2022-01-31T22:12:40.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[46715]: debug 2022-01-31T22:12:40.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.171685+0000) 2022-01-31T22:12:40.337 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[51620]: debug 2022-01-31T22:12:40.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.027359+0000) 2022-01-31T22:12:40.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[41853]: debug 2022-01-31T22:12:40.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.378806+0000) 2022-01-31T22:12:40.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:40 smithi167 conmon[54076]: debug 2022-01-31T22:12:40.673+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.674808+0000) 2022-01-31T22:12:40.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:40 smithi167 conmon[60316]: debug 2022-01-31T22:12:40.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.674808+0000) 2022-01-31T22:12:41.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:40 smithi167 conmon[49112]: debug 2022-01-31T22:12:40.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.755108+0000) 2022-01-31T22:12:41.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:40 smithi167 conmon[49112]: debug 2022-01-31T22:12:40.830+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.831836+0000) 2022-01-31T22:12:41.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:40 smithi167 conmon[54076]: debug 2022-01-31T22:12:40.830+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.832227+0000) 2022-01-31T22:12:41.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:40 smithi167 conmon[60316]: debug 2022-01-31T22:12:40.836+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.838003+0000) 2022-01-31T22:12:41.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[35325]: debug 2022-01-31T22:12:40.865+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191285 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:41.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[41853]: debug 2022-01-31T22:12:40.831+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.832003+0000) 2022-01-31T22:12:41.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[46715]: debug 2022-01-31T22:12:40.830+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.830880+0000) 2022-01-31T22:12:41.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:40 smithi171 conmon[51620]: debug 2022-01-31T22:12:40.830+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:40.831422+0000) 2022-01-31T22:12:41.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:41 smithi171 conmon[51620]: debug 2022-01-31T22:12:41.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:41.027558+0000) 2022-01-31T22:12:41.378 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:41 smithi171 conmon[46715]: debug 2022-01-31T22:12:41.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:41.171864+0000) 2022-01-31T22:12:41.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:41 smithi167 conmon[54076]: debug 2022-01-31T22:12:41.673+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:41.674973+0000) 2022-01-31T22:12:41.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:41 smithi167 conmon[60316]: debug 2022-01-31T22:12:41.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:41.674973+0000) 2022-01-31T22:12:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:41 smithi171 conmon[41853]: debug 2022-01-31T22:12:41.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:41.378958+0000) 2022-01-31T22:12:42.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:41 smithi167 conmon[49112]: debug 2022-01-31T22:12:41.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:41.755297+0000) 2022-01-31T22:12:42.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:42 smithi171 conmon[46715]: debug 2022-01-31T22:12:42.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:42.172038+0000) 2022-01-31T22:12:42.337 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:42 smithi171 conmon[51620]: debug 2022-01-31T22:12:42.026+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:42.027732+0000) 2022-01-31T22:12:42.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:42 smithi171 conmon[41853]: debug 2022-01-31T22:12:42.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:42.379097+0000) 2022-01-31T22:12:42.754 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:42 smithi167 conmon[54076]: debug 2022-01-31T22:12:42.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:42.675122+0000) 2022-01-31T22:12:42.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:42 smithi167 conmon[60316]: debug 2022-01-31T22:12:42.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:42.675122+0000) 2022-01-31T22:12:43.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:42 smithi167 conmon[49112]: debug 2022-01-31T22:12:42.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:42.755447+0000) 2022-01-31T22:12:43.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:43 smithi171 conmon[46715]: debug 2022-01-31T22:12:43.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:43.172231+0000) 2022-01-31T22:12:43.337 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:43 smithi171 conmon[51620]: debug 2022-01-31T22:12:43.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:43.027962+0000) 2022-01-31T22:12:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:43 smithi171 conmon[41853]: debug 2022-01-31T22:12:43.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:43.379313+0000) 2022-01-31T22:12:43.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:43 smithi167 conmon[54076]: debug 2022-01-31T22:12:43.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:43.675327+0000) 2022-01-31T22:12:43.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:43 smithi167 conmon[60316]: debug 2022-01-31T22:12:43.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:43.675327+0000) 2022-01-31T22:12:44.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:43 smithi167 conmon[49112]: debug 2022-01-31T22:12:43.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:43.755688+0000) 2022-01-31T22:12:44.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:44 smithi171 conmon[46715]: debug 2022-01-31T22:12:44.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:44.172445+0000) 2022-01-31T22:12:44.337 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:44 smithi171 conmon[51620]: debug 2022-01-31T22:12:44.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:44.028183+0000) 2022-01-31T22:12:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:44 smithi171 conmon[41853]: debug 2022-01-31T22:12:44.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:44.379522+0000) 2022-01-31T22:12:44.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:44 smithi167 conmon[60316]: debug 2022-01-31T22:12:44.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:44.675565+0000) 2022-01-31T22:12:44.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:44 smithi167 conmon[54076]: debug 2022-01-31T22:12:44.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:44.675565+0000) 2022-01-31T22:12:45.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:44 smithi167 conmon[49112]: debug 2022-01-31T22:12:44.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:44.755817+0000) 2022-01-31T22:12:45.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[46715]: debug 2022-01-31T22:12:45.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.172628+0000) 2022-01-31T22:12:45.337 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[51620]: debug 2022-01-31T22:12:45.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.028384+0000) 2022-01-31T22:12:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[41853]: debug 2022-01-31T22:12:45.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.379706+0000) 2022-01-31T22:12:45.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:45 smithi167 conmon[54076]: debug 2022-01-31T22:12:45.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.675781+0000) 2022-01-31T22:12:45.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:45 smithi167 conmon[60316]: debug 2022-01-31T22:12:45.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.675782+0000) 2022-01-31T22:12:46.015 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:45 smithi167 conmon[60316]: debug 2022-01-31T22:12:45.873+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.875284+0000) 2022-01-31T22:12:46.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:45 smithi167 conmon[49112]: debug 2022-01-31T22:12:45.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.755999+0000) 2022-01-31T22:12:46.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:45 smithi167 conmon[49112]: debug 2022-01-31T22:12:45.873+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.874945+0000) 2022-01-31T22:12:46.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:45 smithi167 conmon[54076]: debug 2022-01-31T22:12:45.866+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.868477+0000) 2022-01-31T22:12:46.120 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[35325]: debug 2022-01-31T22:12:45.886+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191395 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:46.121 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[41853]: debug 2022-01-31T22:12:45.866+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.868272+0000) 2022-01-31T22:12:46.121 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[46715]: debug 2022-01-31T22:12:45.866+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.868091+0000) 2022-01-31T22:12:46.122 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:45 smithi171 conmon[51620]: debug 2022-01-31T22:12:45.866+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:45.867966+0000) 2022-01-31T22:12:46.122 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:46 smithi171 conmon[51620]: debug 2022-01-31T22:12:46.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:46.028532+0000) 2022-01-31T22:12:46.379 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:46 smithi171 conmon[46715]: debug 2022-01-31T22:12:46.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:46.172797+0000) 2022-01-31T22:12:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:46 smithi171 conmon[41853]: debug 2022-01-31T22:12:46.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:46.379907+0000) 2022-01-31T22:12:46.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:46 smithi167 conmon[49112]: debug 2022-01-31T22:12:46.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:46.756132+0000) 2022-01-31T22:12:46.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:46 smithi167 conmon[54076]: debug 2022-01-31T22:12:46.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:46.675912+0000) 2022-01-31T22:12:46.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:46 smithi167 conmon[60316]: debug 2022-01-31T22:12:46.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:46.675907+0000) 2022-01-31T22:12:47.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:47 smithi171 conmon[46715]: debug 2022-01-31T22:12:47.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:47.172950+0000) 2022-01-31T22:12:47.338 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:47 smithi171 conmon[51620]: debug 2022-01-31T22:12:47.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:47.028675+0000) 2022-01-31T22:12:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:47 smithi171 conmon[41853]: debug 2022-01-31T22:12:47.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:47.380065+0000) 2022-01-31T22:12:47.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:47 smithi167 conmon[54076]: debug 2022-01-31T22:12:47.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:47.676007+0000) 2022-01-31T22:12:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:47 smithi167 conmon[49112]: debug 2022-01-31T22:12:47.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:47.756240+0000) 2022-01-31T22:12:47.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:47 smithi167 conmon[60316]: debug 2022-01-31T22:12:47.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:47.676001+0000) 2022-01-31T22:12:48.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:48 smithi171 conmon[46715]: debug 2022-01-31T22:12:48.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:48.173126+0000) 2022-01-31T22:12:48.338 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:48 smithi171 conmon[51620]: debug 2022-01-31T22:12:48.027+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:48.028831+0000) 2022-01-31T22:12:48.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:48 smithi171 conmon[41853]: debug 2022-01-31T22:12:48.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:48.380226+0000) 2022-01-31T22:12:48.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:48 smithi167 conmon[49112]: debug 2022-01-31T22:12:48.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:48.756369+0000) 2022-01-31T22:12:48.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:48 smithi167 conmon[54076]: debug 2022-01-31T22:12:48.674+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:48.676163+0000) 2022-01-31T22:12:48.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:48 smithi167 conmon[60316]: debug 2022-01-31T22:12:48.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:48.676172+0000) 2022-01-31T22:12:49.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:49 smithi171 conmon[46715]: debug 2022-01-31T22:12:49.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:49.173336+0000) 2022-01-31T22:12:49.338 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:49 smithi171 conmon[51620]: debug 2022-01-31T22:12:49.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:49.028998+0000) 2022-01-31T22:12:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:49 smithi171 conmon[41853]: debug 2022-01-31T22:12:49.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:49.380424+0000) 2022-01-31T22:12:49.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:49 smithi167 conmon[54076]: debug 2022-01-31T22:12:49.675+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:49.676321+0000) 2022-01-31T22:12:49.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:49 smithi167 conmon[60316]: debug 2022-01-31T22:12:49.675+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:49.676340+0000) 2022-01-31T22:12:50.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:49 smithi167 conmon[49112]: debug 2022-01-31T22:12:49.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:49.756574+0000) 2022-01-31T22:12:50.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[46715]: debug 2022-01-31T22:12:50.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.173520+0000) 2022-01-31T22:12:50.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[51620]: debug 2022-01-31T22:12:50.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.029201+0000) 2022-01-31T22:12:50.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[41853]: debug 2022-01-31T22:12:50.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.380607+0000) 2022-01-31T22:12:50.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[49112]: debug 2022-01-31T22:12:50.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.756763+0000) 2022-01-31T22:12:50.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[49112]: debug 2022-01-31T22:12:50.890+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.891574+0000) 2022-01-31T22:12:50.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[54076]: debug 2022-01-31T22:12:50.675+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.676535+0000) 2022-01-31T22:12:50.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[54076]: debug 2022-01-31T22:12:50.889+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.890875+0000) 2022-01-31T22:12:50.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[60316]: debug 2022-01-31T22:12:50.675+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.676550+0000) 2022-01-31T22:12:50.922 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[60316]: debug 2022-01-31T22:12:50.922 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:50 smithi167 conmon[60316]: 2022-01-31T22:12:50.890+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.891713+0000) 2022-01-31T22:12:51.173 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[35325]: debug 2022-01-31T22:12:50.908+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191508 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:51.174 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[41853]: debug 2022-01-31T22:12:50.890+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.891995+0000) 2022-01-31T22:12:51.174 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[46715]: debug 2022-01-31T22:12:50.888+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.890244+0000) 2022-01-31T22:12:51.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:50 smithi171 conmon[51620]: debug 2022-01-31T22:12:50.888+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:50.890729+0000) 2022-01-31T22:12:51.175 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:51 smithi171 conmon[51620]: debug 2022-01-31T22:12:51.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:51.029359+0000) 2022-01-31T22:12:51.606 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:51 smithi171 conmon[46715]: debug 2022-01-31T22:12:51.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:51.173690+0000) 2022-01-31T22:12:51.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:51 smithi171 conmon[41853]: debug 2022-01-31T22:12:51.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:51.380818+0000) 2022-01-31T22:12:51.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:51 smithi167 conmon[54076]: debug 2022-01-31T22:12:51.675+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:51.676744+0000) 2022-01-31T22:12:51.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:51 smithi167 conmon[60316]: debug 2022-01-31T22:12:51.675+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:51.676726+0000) 2022-01-31T22:12:52.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:51 smithi167 conmon[49112]: debug 2022-01-31T22:12:51.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:51.756950+0000) 2022-01-31T22:12:52.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:52 smithi171 conmon[46715]: debug 2022-01-31T22:12:52.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:52.173847+0000) 2022-01-31T22:12:52.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:52 smithi171 conmon[51620]: debug 2022-01-31T22:12:52.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:52.029527+0000) 2022-01-31T22:12:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:52 smithi171 conmon[41853]: debug 2022-01-31T22:12:52.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:52.380981+0000) 2022-01-31T22:12:52.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:52 smithi167 conmon[54076]: debug 2022-01-31T22:12:52.676+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:52.676926+0000) 2022-01-31T22:12:52.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:52 smithi167 conmon[60316]: debug 2022-01-31T22:12:52.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:52.676881+0000) 2022-01-31T22:12:53.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:52 smithi167 conmon[49112]: debug 2022-01-31T22:12:52.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:52.757100+0000) 2022-01-31T22:12:53.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:53 smithi171 conmon[46715]: debug 2022-01-31T22:12:53.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:53.173975+0000) 2022-01-31T22:12:53.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:53 smithi171 conmon[51620]: debug 2022-01-31T22:12:53.028+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:53.029719+0000) 2022-01-31T22:12:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:53 smithi171 conmon[41853]: debug 2022-01-31T22:12:53.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:53.381110+0000) 2022-01-31T22:12:53.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:53 smithi167 conmon[54076]: debug 2022-01-31T22:12:53.675+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:53.677142+0000) 2022-01-31T22:12:53.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:53 smithi167 conmon[60316]: debug 2022-01-31T22:12:53.675+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:53.677094+0000) 2022-01-31T22:12:54.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:53 smithi167 conmon[49112]: debug 2022-01-31T22:12:53.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:53.757256+0000) 2022-01-31T22:12:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:53 smithi171 conmon[35325]: debug 2022-01-31T22:12:53.814+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:12:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:54 smithi171 conmon[51620]: debug 2022-01-31T22:12:54.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:54.029943+0000) 2022-01-31T22:12:54.381 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:54 smithi171 conmon[46715]: debug 2022-01-31T22:12:54.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:54.174193+0000) 2022-01-31T22:12:54.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:54 smithi167 conmon[54076]: debug 2022-01-31T22:12:54.675+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:54.677345+0000) 2022-01-31T22:12:54.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:54 smithi167 conmon[60316]: debug 2022-01-31T22:12:54.675+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:54.677305+0000) 2022-01-31T22:12:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:54 smithi171 conmon[41853]: debug 2022-01-31T22:12:54.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:54.381297+0000) 2022-01-31T22:12:55.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:54 smithi167 conmon[49112]: debug 2022-01-31T22:12:54.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:54.757529+0000) 2022-01-31T22:12:55.338 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[46715]: debug 2022-01-31T22:12:55.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.174391+0000) 2022-01-31T22:12:55.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[51620]: debug 2022-01-31T22:12:55.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.030123+0000) 2022-01-31T22:12:55.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[41853]: debug 2022-01-31T22:12:55.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.381460+0000) 2022-01-31T22:12:55.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:55 smithi167 conmon[54076]: debug 2022-01-31T22:12:55.676+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.677521+0000) 2022-01-31T22:12:55.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:55 smithi167 conmon[60316]: debug 2022-01-31T22:12:55.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.677473+0000) 2022-01-31T22:12:56.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:55 smithi167 conmon[54076]: debug 2022-01-31T22:12:55.911+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.913068+0000) 2022-01-31T22:12:56.017 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:55 smithi167 conmon[60316]: debug 2022-01-31T22:12:55.911+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.913267+0000) 2022-01-31T22:12:56.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:55 smithi167 conmon[49112]: debug 2022-01-31T22:12:55.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.757706+0000) 2022-01-31T22:12:56.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:55 smithi167 conmon[49112]: debug 2022-01-31T22:12:55.911+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.912980+0000) 2022-01-31T22:12:56.174 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[35325]: debug 2022-01-31T22:12:55.931+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191618 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:12:56.174 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[41853]: debug 2022-01-31T22:12:55.912+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.913510+0000) 2022-01-31T22:12:56.175 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[46715]: debug 2022-01-31T22:12:55.912+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.912966+0000) 2022-01-31T22:12:56.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:55 smithi171 conmon[51620]: debug 2022-01-31T22:12:55.911+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:55.912110+0000) 2022-01-31T22:12:56.176 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:56 smithi171 conmon[51620]: debug 2022-01-31T22:12:56.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:56.030317+0000) 2022-01-31T22:12:56.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:56 smithi171 conmon[41853]: debug 2022-01-31T22:12:56.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:56.381656+0000) 2022-01-31T22:12:56.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:56 smithi171 conmon[46715]: debug 2022-01-31T22:12:56.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:56.174605+0000) 2022-01-31T22:12:56.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:56 smithi167 conmon[54076]: debug 2022-01-31T22:12:56.676+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:56.677689+0000) 2022-01-31T22:12:56.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:56 smithi167 conmon[60316]: debug 2022-01-31T22:12:56.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:56.677686+0000) 2022-01-31T22:12:57.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:56 smithi167 conmon[49112]: debug 2022-01-31T22:12:56.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:56.757807+0000) 2022-01-31T22:12:57.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:57 smithi171 conmon[46715]: debug 2022-01-31T22:12:57.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:57.174773+0000) 2022-01-31T22:12:57.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:57 smithi171 conmon[51620]: debug 2022-01-31T22:12:57.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:57.030483+0000) 2022-01-31T22:12:57.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:57 smithi171 conmon[41853]: debug 2022-01-31T22:12:57.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:57.381793+0000) 2022-01-31T22:12:57.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:57 smithi167 conmon[54076]: debug 2022-01-31T22:12:57.677+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:57.677872+0000) 2022-01-31T22:12:57.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:57 smithi167 conmon[60316]: debug 2022-01-31T22:12:57.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:57.677868+0000) 2022-01-31T22:12:58.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:57 smithi167 conmon[49112]: debug 2022-01-31T22:12:57.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:57.757979+0000) 2022-01-31T22:12:58.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:58 smithi171 conmon[46715]: debug 2022-01-31T22:12:58.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:58.174951+0000) 2022-01-31T22:12:58.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:58 smithi171 conmon[51620]: debug 2022-01-31T22:12:58.029+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:58.030699+0000) 2022-01-31T22:12:58.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:58 smithi171 conmon[41853]: debug 2022-01-31T22:12:58.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:58.381980+0000) 2022-01-31T22:12:58.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:58 smithi167 conmon[54076]: debug 2022-01-31T22:12:58.677+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:58.678091+0000) 2022-01-31T22:12:58.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:58 smithi167 conmon[60316]: debug 2022-01-31T22:12:58.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:58.678090+0000) 2022-01-31T22:12:59.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:58 smithi167 conmon[49112]: debug 2022-01-31T22:12:58.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:58.758186+0000) 2022-01-31T22:12:59.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:12:59 smithi171 conmon[46715]: debug 2022-01-31T22:12:59.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:59.175136+0000) 2022-01-31T22:12:59.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:12:59 smithi171 conmon[51620]: debug 2022-01-31T22:12:59.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:59.030853+0000) 2022-01-31T22:12:59.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:12:59 smithi171 conmon[41853]: debug 2022-01-31T22:12:59.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:59.382123+0000) 2022-01-31T22:12:59.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:12:59 smithi167 conmon[49112]: debug 2022-01-31T22:12:59.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:59.758373+0000) 2022-01-31T22:12:59.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:12:59 smithi167 conmon[54076]: debug 2022-01-31T22:12:59.676+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:59.678293+0000) 2022-01-31T22:12:59.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:12:59 smithi167 conmon[60316]: debug 2022-01-31T22:12:59.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:12:59.678289+0000) 2022-01-31T22:13:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[46715]: debug 2022-01-31T22:13:00.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.175350+0000) 2022-01-31T22:13:00.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[51620]: debug 2022-01-31T22:13:00.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.031062+0000) 2022-01-31T22:13:00.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[41853]: debug 2022-01-31T22:13:00.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.382321+0000) 2022-01-31T22:13:00.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:00 smithi167 conmon[54076]: debug 2022-01-31T22:13:00.677+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.678520+0000) 2022-01-31T22:13:00.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:00 smithi167 conmon[60316]: debug 2022-01-31T22:13:00.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.678517+0000) 2022-01-31T22:13:00.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:00 smithi167 conmon[60316]: debug 2022-01-31T22:13:00.933+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.935205+0000) 2022-01-31T22:13:00.941 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:00 smithi167 conmon[49112]: debug 2022-01-31T22:13:00.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.758535+0000) 2022-01-31T22:13:00.941 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:00 smithi167 conmon[49112]: debug 2022-01-31T22:13:00.937+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.939270+0000) 2022-01-31T22:13:01.017 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:00 smithi167 conmon[54076]: debug 2022-01-31T22:13:00.939+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.941390+0000) 2022-01-31T22:13:01.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[35325]: debug 2022-01-31T22:13:00.953+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191731 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:01.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[41853]: debug 2022-01-31T22:13:00.933+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.934647+0000) 2022-01-31T22:13:01.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[51620]: debug 2022-01-31T22:13:00.933+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.934463+0000) 2022-01-31T22:13:01.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:01 smithi171 conmon[51620]: debug 2022-01-31T22:13:01.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:01.031249+0000) 2022-01-31T22:13:01.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:00 smithi171 conmon[46715]: debug 2022-01-31T22:13:00.934+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:00.935508+0000) 2022-01-31T22:13:01.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:01 smithi171 conmon[46715]: debug 2022-01-31T22:13:01.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:01.175565+0000) 2022-01-31T22:13:01.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:01 smithi171 conmon[41853]: debug 2022-01-31T22:13:01.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:01.382490+0000) 2022-01-31T22:13:01.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:01 smithi167 conmon[54076]: debug 2022-01-31T22:13:01.678+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:01.678728+0000) 2022-01-31T22:13:01.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:01 smithi167 conmon[60316]: debug 2022-01-31T22:13:01.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:01.678732+0000) 2022-01-31T22:13:02.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:01 smithi167 conmon[49112]: debug 2022-01-31T22:13:01.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:01.758722+0000) 2022-01-31T22:13:02.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:02 smithi171 conmon[46715]: debug 2022-01-31T22:13:02.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:02.175719+0000) 2022-01-31T22:13:02.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:02 smithi171 conmon[51620]: debug 2022-01-31T22:13:02.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:02.031428+0000) 2022-01-31T22:13:02.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:02 smithi171 conmon[41853]: debug 2022-01-31T22:13:02.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:02.382653+0000) 2022-01-31T22:13:02.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:02 smithi167 conmon[54076]: debug 2022-01-31T22:13:02.678+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:02.678905+0000) 2022-01-31T22:13:02.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:02 smithi167 conmon[60316]: debug 2022-01-31T22:13:02.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:02.678891+0000) 2022-01-31T22:13:03.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:02 smithi167 conmon[49112]: debug 2022-01-31T22:13:02.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:02.758913+0000) 2022-01-31T22:13:03.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:03 smithi171 conmon[46715]: debug 2022-01-31T22:13:03.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:03.175900+0000) 2022-01-31T22:13:03.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:03 smithi171 conmon[51620]: debug 2022-01-31T22:13:03.030+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:03.031570+0000) 2022-01-31T22:13:03.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:03 smithi171 conmon[41853]: debug 2022-01-31T22:13:03.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:03.382857+0000) 2022-01-31T22:13:03.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:03 smithi167 conmon[54076]: debug 2022-01-31T22:13:03.678+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:03.679078+0000) 2022-01-31T22:13:03.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:03 smithi167 conmon[60316]: debug 2022-01-31T22:13:03.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:03.679070+0000) 2022-01-31T22:13:04.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:03 smithi167 conmon[49112]: debug 2022-01-31T22:13:03.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:03.759125+0000) 2022-01-31T22:13:04.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:04 smithi171 conmon[46715]: debug 2022-01-31T22:13:04.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:04.176110+0000) 2022-01-31T22:13:04.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:04 smithi171 conmon[51620]: debug 2022-01-31T22:13:04.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:04.031796+0000) 2022-01-31T22:13:04.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:04 smithi171 conmon[41853]: debug 2022-01-31T22:13:04.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:04.383038+0000) 2022-01-31T22:13:04.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:04 smithi167 conmon[54076]: debug 2022-01-31T22:13:04.678+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:04.679287+0000) 2022-01-31T22:13:04.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:04 smithi167 conmon[60316]: debug 2022-01-31T22:13:04.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:04.679253+0000) 2022-01-31T22:13:05.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:04 smithi167 conmon[49112]: debug 2022-01-31T22:13:04.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:04.759332+0000) 2022-01-31T22:13:05.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[51620]: debug 2022-01-31T22:13:05.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.031940+0000) 2022-01-31T22:13:05.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[46715]: debug 2022-01-31T22:13:05.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.176268+0000) 2022-01-31T22:13:05.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[41853]: debug 2022-01-31T22:13:05.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.383182+0000) 2022-01-31T22:13:05.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:05 smithi167 conmon[54076]: debug 2022-01-31T22:13:05.678+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.679454+0000) 2022-01-31T22:13:05.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:05 smithi167 conmon[60316]: debug 2022-01-31T22:13:05.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.679450+0000) 2022-01-31T22:13:06.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:05 smithi167 conmon[54076]: debug 2022-01-31T22:13:05.957+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.957969+0000) 2022-01-31T22:13:06.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:05 smithi167 conmon[60316]: debug 2022-01-31T22:13:05.956+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.957137+0000) 2022-01-31T22:13:06.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:05 smithi167 conmon[49112]: debug 2022-01-31T22:13:05.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.759542+0000) 2022-01-31T22:13:06.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:05 smithi167 conmon[49112]: debug 2022-01-31T22:13:05.957+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.958088+0000) 2022-01-31T22:13:06.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[35325]: debug 2022-01-31T22:13:05.976+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191841 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:06.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[41853]: debug 2022-01-31T22:13:05.956+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.957497+0000) 2022-01-31T22:13:06.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[46715]: debug 2022-01-31T22:13:05.956+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.957017+0000) 2022-01-31T22:13:06.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:06 smithi171 conmon[46715]: debug 2022-01-31T22:13:06.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:06.176452+0000) 2022-01-31T22:13:06.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:05 smithi171 conmon[51620]: debug 2022-01-31T22:13:05.955+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:05.956608+0000) 2022-01-31T22:13:06.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:06 smithi171 conmon[51620]: debug 2022-01-31T22:13:06.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:06.032123+0000) 2022-01-31T22:13:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:06 smithi171 conmon[41853]: debug 2022-01-31T22:13:06.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:06.383356+0000) 2022-01-31T22:13:06.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:06 smithi167 conmon[54076]: debug 2022-01-31T22:13:06.679+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:06.679663+0000) 2022-01-31T22:13:06.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:06 smithi167 conmon[60316]: debug 2022-01-31T22:13:06.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:06.679592+0000) 2022-01-31T22:13:07.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:06 smithi167 conmon[49112]: debug 2022-01-31T22:13:06.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:06.759724+0000) 2022-01-31T22:13:07.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:07 smithi171 conmon[51620]: debug 2022-01-31T22:13:07.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:07.032279+0000) 2022-01-31T22:13:07.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:07 smithi171 conmon[46715]: debug 2022-01-31T22:13:07.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:07.176624+0000) 2022-01-31T22:13:07.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:07 smithi171 conmon[41853]: debug 2022-01-31T22:13:07.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:07.383503+0000) 2022-01-31T22:13:07.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:07 smithi167 conmon[54076]: debug 2022-01-31T22:13:07.679+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:07.679775+0000) 2022-01-31T22:13:07.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:07 smithi167 conmon[60316]: debug 2022-01-31T22:13:07.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:07.679703+0000) 2022-01-31T22:13:08.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:07 smithi167 conmon[49112]: debug 2022-01-31T22:13:07.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:07.759872+0000) 2022-01-31T22:13:08.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:08 smithi171 conmon[46715]: debug 2022-01-31T22:13:08.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:08.176809+0000) 2022-01-31T22:13:08.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:08 smithi171 conmon[51620]: debug 2022-01-31T22:13:08.031+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:08.032488+0000) 2022-01-31T22:13:08.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:08 smithi171 conmon[41853]: debug 2022-01-31T22:13:08.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:08.383656+0000) 2022-01-31T22:13:08.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:08 smithi167 conmon[54076]: debug 2022-01-31T22:13:08.679+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:08.679911+0000) 2022-01-31T22:13:08.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:08 smithi167 conmon[60316]: debug 2022-01-31T22:13:08.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:08.679911+0000) 2022-01-31T22:13:09.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:08 smithi167 conmon[49112]: debug 2022-01-31T22:13:08.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:08.760065+0000) 2022-01-31T22:13:09.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:08 smithi171 conmon[35325]: debug 2022-01-31T22:13:08.815+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:13:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:09 smithi171 conmon[51620]: debug 2022-01-31T22:13:09.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:09.032655+0000) 2022-01-31T22:13:09.383 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:09 smithi171 conmon[46715]: debug 2022-01-31T22:13:09.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:09.176994+0000) 2022-01-31T22:13:09.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:09 smithi167 conmon[54076]: debug 2022-01-31T22:13:09.679+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:09.680143+0000) 2022-01-31T22:13:09.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:09 smithi167 conmon[60316]: debug 2022-01-31T22:13:09.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:09.680143+0000) 2022-01-31T22:13:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:09 smithi171 conmon[41853]: debug 2022-01-31T22:13:09.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:09.383840+0000) 2022-01-31T22:13:10.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:09 smithi167 conmon[49112]: debug 2022-01-31T22:13:09.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:09.760290+0000) 2022-01-31T22:13:10.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[46715]: debug 2022-01-31T22:13:10.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.177174+0000) 2022-01-31T22:13:10.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[51620]: debug 2022-01-31T22:13:10.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.032874+0000) 2022-01-31T22:13:10.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[41853]: debug 2022-01-31T22:13:10.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.383992+0000) 2022-01-31T22:13:10.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:10 smithi167 conmon[54076]: debug 2022-01-31T22:13:10.679+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.680329+0000) 2022-01-31T22:13:10.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:10 smithi167 conmon[60316]: debug 2022-01-31T22:13:10.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.680329+0000) 2022-01-31T22:13:11.019 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:10 smithi167 conmon[54076]: debug 2022-01-31T22:13:10.979+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.980360+0000) 2022-01-31T22:13:11.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:10 smithi167 conmon[49112]: debug 2022-01-31T22:13:10.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.760492+0000) 2022-01-31T22:13:11.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:10 smithi167 conmon[49112]: debug 2022-01-31T22:13:10.980+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.980721+0000) 2022-01-31T22:13:11.021 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:10 smithi167 conmon[60316]: debug 2022-01-31T22:13:10.978+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.979226+0000) 2022-01-31T22:13:11.125 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[41853]: debug 2022-01-31T22:13:10.978+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.979198+0000) 2022-01-31T22:13:11.126 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[46715]: debug 2022-01-31T22:13:10.979+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.979746+0000) 2022-01-31T22:13:11.127 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[35325]: debug 2022-01-31T22:13:10.998+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 191953 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:11.127 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:10 smithi171 conmon[51620]: debug 2022-01-31T22:13:10.978+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:10.978798+0000) 2022-01-31T22:13:11.128 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:11 smithi171 conmon[51620]: debug 2022-01-31T22:13:11.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:11.033021+0000) 2022-01-31T22:13:11.383 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:11 smithi171 conmon[46715]: debug 2022-01-31T22:13:11.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:11.177346+0000) 2022-01-31T22:13:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:11 smithi171 conmon[41853]: debug 2022-01-31T22:13:11.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:11.384199+0000) 2022-01-31T22:13:11.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:11 smithi167 conmon[49112]: debug 2022-01-31T22:13:11.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:11.760668+0000) 2022-01-31T22:13:11.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:11 smithi167 conmon[54076]: debug 2022-01-31T22:13:11.680+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:11.680546+0000) 2022-01-31T22:13:11.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:11 smithi167 conmon[60316]: debug 2022-01-31T22:13:11.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:11.680546+0000) 2022-01-31T22:13:12.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:12 smithi171 conmon[46715]: debug 2022-01-31T22:13:12.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:12.177477+0000) 2022-01-31T22:13:12.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:12 smithi171 conmon[51620]: debug 2022-01-31T22:13:12.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:12.033187+0000) 2022-01-31T22:13:12.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:12 smithi171 conmon[41853]: debug 2022-01-31T22:13:12.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:12.384369+0000) 2022-01-31T22:13:12.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:12 smithi167 conmon[54076]: debug 2022-01-31T22:13:12.680+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:12.680657+0000) 2022-01-31T22:13:12.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:12 smithi167 conmon[60316]: debug 2022-01-31T22:13:12.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:12.680657+0000) 2022-01-31T22:13:13.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:12 smithi167 conmon[49112]: debug 2022-01-31T22:13:12.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:12.760850+0000) 2022-01-31T22:13:13.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:13 smithi171 conmon[46715]: debug 2022-01-31T22:13:13.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:13.177663+0000) 2022-01-31T22:13:13.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:13 smithi171 conmon[51620]: debug 2022-01-31T22:13:13.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:13.033355+0000) 2022-01-31T22:13:13.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:13 smithi171 conmon[41853]: debug 2022-01-31T22:13:13.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:13.384571+0000) 2022-01-31T22:13:13.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:13 smithi167 conmon[54076]: debug 2022-01-31T22:13:13.680+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:13.680843+0000) 2022-01-31T22:13:13.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:13 smithi167 conmon[60316]: debug 2022-01-31T22:13:13.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:13.680843+0000) 2022-01-31T22:13:14.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:13 smithi167 conmon[49112]: debug 2022-01-31T22:13:13.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:13.761040+0000) 2022-01-31T22:13:14.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:14 smithi171 conmon[46715]: debug 2022-01-31T22:13:14.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:14.177872+0000) 2022-01-31T22:13:14.306 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:14 smithi171 conmon[51620]: debug 2022-01-31T22:13:14.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:14.033549+0000) 2022-01-31T22:13:14.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:14 smithi171 conmon[41853]: debug 2022-01-31T22:13:14.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:14.384800+0000) 2022-01-31T22:13:14.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:14 smithi167 conmon[54076]: debug 2022-01-31T22:13:14.680+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:14.681062+0000) 2022-01-31T22:13:14.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:14 smithi167 conmon[60316]: debug 2022-01-31T22:13:14.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:14.681062+0000) 2022-01-31T22:13:15.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:14 smithi167 conmon[49112]: debug 2022-01-31T22:13:14.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:14.761244+0000) 2022-01-31T22:13:15.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:15 smithi171 conmon[46715]: debug 2022-01-31T22:13:15.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:15.178053+0000) 2022-01-31T22:13:15.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:15 smithi171 conmon[51620]: debug 2022-01-31T22:13:15.032+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:15.033721+0000) 2022-01-31T22:13:15.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:15 smithi171 conmon[41853]: debug 2022-01-31T22:13:15.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:15.384921+0000) 2022-01-31T22:13:15.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:15 smithi167 conmon[54076]: debug 2022-01-31T22:13:15.680+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:15.681267+0000) 2022-01-31T22:13:15.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:15 smithi167 conmon[60316]: debug 2022-01-31T22:13:15.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:15.681267+0000) 2022-01-31T22:13:16.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:16 smithi167 conmon[54076]: debug 2022-01-31T22:13:16.001+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.001974+0000) 2022-01-31T22:13:16.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:15 smithi167 conmon[49112]: debug 2022-01-31T22:13:15.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:15.761430+0000) 2022-01-31T22:13:16.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:16 smithi167 conmon[49112]: debug 2022-01-31T22:13:16.001+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.001691+0000) 2022-01-31T22:13:16.022 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:16 smithi167 conmon[60316]: debug 2022-01-31T22:13:16.000+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.000813+0000) 2022-01-31T22:13:16.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[35325]: debug 2022-01-31T22:13:16.020+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192063 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:16.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[41853]: debug 2022-01-31T22:13:15.999+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.000695+0000) 2022-01-31T22:13:16.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[46715]: debug 2022-01-31T22:13:16.001+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.001924+0000) 2022-01-31T22:13:16.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[46715]: debug 2022-01-31T22:13:16.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.178284+0000) 2022-01-31T22:13:16.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[51620]: debug 2022-01-31T22:13:15.999+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.000358+0000) 2022-01-31T22:13:16.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[51620]: debug 2022-01-31T22:13:16.033+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.033918+0000) 2022-01-31T22:13:16.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:16 smithi171 conmon[41853]: debug 2022-01-31T22:13:16.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.385075+0000) 2022-01-31T22:13:16.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:16 smithi167 conmon[54076]: debug 2022-01-31T22:13:16.680+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.681459+0000) 2022-01-31T22:13:16.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:16 smithi167 conmon[49112]: debug 2022-01-31T22:13:16.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.761594+0000) 2022-01-31T22:13:16.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:16 smithi167 conmon[60316]: debug 2022-01-31T22:13:16.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:16.681459+0000) 2022-01-31T22:13:17.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:17 smithi171 conmon[46715]: debug 2022-01-31T22:13:17.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:17.178443+0000) 2022-01-31T22:13:17.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:17 smithi171 conmon[51620]: debug 2022-01-31T22:13:17.033+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:17.034076+0000) 2022-01-31T22:13:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:17 smithi171 conmon[41853]: debug 2022-01-31T22:13:17.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:17.385245+0000) 2022-01-31T22:13:17.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:17 smithi167 conmon[54076]: debug 2022-01-31T22:13:17.681+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:17.681625+0000) 2022-01-31T22:13:17.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:17 smithi167 conmon[60316]: debug 2022-01-31T22:13:17.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:17.681625+0000) 2022-01-31T22:13:18.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:17 smithi167 conmon[49112]: debug 2022-01-31T22:13:17.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:17.761689+0000) 2022-01-31T22:13:18.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:18 smithi171 conmon[46715]: debug 2022-01-31T22:13:18.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:18.178646+0000) 2022-01-31T22:13:18.338 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:18 smithi171 conmon[51620]: debug 2022-01-31T22:13:18.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:18.034286+0000) 2022-01-31T22:13:18.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:18 smithi171 conmon[41853]: debug 2022-01-31T22:13:18.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:18.385447+0000) 2022-01-31T22:13:18.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:18 smithi167 conmon[54076]: debug 2022-01-31T22:13:18.681+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:18.681773+0000) 2022-01-31T22:13:18.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:18 smithi167 conmon[60316]: debug 2022-01-31T22:13:18.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:18.681773+0000) 2022-01-31T22:13:19.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:18 smithi167 conmon[49112]: debug 2022-01-31T22:13:18.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:18.761853+0000) 2022-01-31T22:13:19.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:19 smithi171 conmon[46715]: debug 2022-01-31T22:13:19.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:19.178802+0000) 2022-01-31T22:13:19.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:19 smithi171 conmon[51620]: debug 2022-01-31T22:13:19.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:19.034502+0000) 2022-01-31T22:13:19.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:19 smithi171 conmon[41853]: debug 2022-01-31T22:13:19.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:19.385648+0000) 2022-01-31T22:13:19.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:19 smithi167 conmon[54076]: debug 2022-01-31T22:13:19.681+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:19.681973+0000) 2022-01-31T22:13:19.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:19 smithi167 conmon[60316]: debug 2022-01-31T22:13:19.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:19.681973+0000) 2022-01-31T22:13:20.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:19 smithi167 conmon[49112]: debug 2022-01-31T22:13:19.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:19.762040+0000) 2022-01-31T22:13:20.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:20 smithi171 conmon[46715]: debug 2022-01-31T22:13:20.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:20.179025+0000) 2022-01-31T22:13:20.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:20 smithi171 conmon[51620]: debug 2022-01-31T22:13:20.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:20.034717+0000) 2022-01-31T22:13:20.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:20 smithi171 conmon[41853]: debug 2022-01-31T22:13:20.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:20.385820+0000) 2022-01-31T22:13:20.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:20 smithi167 conmon[54076]: debug 2022-01-31T22:13:20.681+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:20.682159+0000) 2022-01-31T22:13:20.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:20 smithi167 conmon[60316]: debug 2022-01-31T22:13:20.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:20.682159+0000) 2022-01-31T22:13:21.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:20 smithi167 conmon[49112]: debug 2022-01-31T22:13:20.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:20.762224+0000) 2022-01-31T22:13:21.343 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[35325]: debug 2022-01-31T22:13:21.042+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192175 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:21.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[41853]: debug 2022-01-31T22:13:21.021+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.022707+0000) 2022-01-31T22:13:21.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[46715]: debug 2022-01-31T22:13:21.023+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.024176+0000) 2022-01-31T22:13:21.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[46715]: debug 2022-01-31T22:13:21.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.179217+0000) 2022-01-31T22:13:21.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[51620]: debug 2022-01-31T22:13:21.022+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.023480+0000) 2022-01-31T22:13:21.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[51620]: debug 2022-01-31T22:13:21.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.034882+0000) 2022-01-31T22:13:21.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:21 smithi167 conmon[49112]: debug 2022-01-31T22:13:21.023+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.023762+0000) 2022-01-31T22:13:21.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:21 smithi167 conmon[60316]: debug 2022-01-31T22:13:21.023+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.024004+0000) 2022-01-31T22:13:21.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:21 smithi167 conmon[54076]: debug 2022-01-31T22:13:21.022+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.023322+0000) 2022-01-31T22:13:21.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:21 smithi171 conmon[41853]: debug 2022-01-31T22:13:21.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.386015+0000) 2022-01-31T22:13:21.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:21 smithi167 conmon[54076]: debug 2022-01-31T22:13:21.681+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.682316+0000) 2022-01-31T22:13:21.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:21 smithi167 conmon[60316]: debug 2022-01-31T22:13:21.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.682338+0000) 2022-01-31T22:13:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:21 smithi167 conmon[49112]: debug 2022-01-31T22:13:21.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:21.762406+0000) 2022-01-31T22:13:22.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:22 smithi171 conmon[46715]: debug 2022-01-31T22:13:22.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:22.179329+0000) 2022-01-31T22:13:22.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:22 smithi171 conmon[51620]: debug 2022-01-31T22:13:22.034+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:22.035049+0000) 2022-01-31T22:13:22.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:22 smithi171 conmon[41853]: debug 2022-01-31T22:13:22.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:22.386168+0000) 2022-01-31T22:13:22.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:22 smithi167 conmon[54076]: debug 2022-01-31T22:13:22.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:22.682545+0000) 2022-01-31T22:13:22.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:22 smithi167 conmon[60316]: debug 2022-01-31T22:13:22.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:22.682545+0000) 2022-01-31T22:13:23.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:22 smithi167 conmon[49112]: debug 2022-01-31T22:13:22.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:22.762587+0000) 2022-01-31T22:13:23.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:23 smithi171 conmon[46715]: debug 2022-01-31T22:13:23.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:23.179490+0000) 2022-01-31T22:13:23.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:23 smithi171 conmon[51620]: debug 2022-01-31T22:13:23.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:23.035227+0000) 2022-01-31T22:13:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:23 smithi171 conmon[41853]: debug 2022-01-31T22:13:23.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:23.386372+0000) 2022-01-31T22:13:23.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:23 smithi167 conmon[60316]: debug 2022-01-31T22:13:23.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:23.682740+0000) 2022-01-31T22:13:23.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:23 smithi167 conmon[54076]: debug 2022-01-31T22:13:23.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:23.682740+0000) 2022-01-31T22:13:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:23 smithi171 conmon[35325]: debug 2022-01-31T22:13:23.816+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:13:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:24 smithi171 conmon[51620]: debug 2022-01-31T22:13:24.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:24.035398+0000) 2022-01-31T22:13:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:23 smithi167 conmon[49112]: debug 2022-01-31T22:13:23.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:23.762687+0000) 2022-01-31T22:13:24.385 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:24 smithi171 conmon[46715]: debug 2022-01-31T22:13:24.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:24.179740+0000) 2022-01-31T22:13:24.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:24 smithi167 conmon[60316]: debug 2022-01-31T22:13:24.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:24.682904+0000) 2022-01-31T22:13:24.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:24 smithi167 conmon[54076]: debug 2022-01-31T22:13:24.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:24.682904+0000) 2022-01-31T22:13:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:24 smithi171 conmon[41853]: debug 2022-01-31T22:13:24.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:24.386563+0000) 2022-01-31T22:13:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:24 smithi167 conmon[49112]: debug 2022-01-31T22:13:24.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:24.762861+0000) 2022-01-31T22:13:25.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:25 smithi171 conmon[46715]: debug 2022-01-31T22:13:25.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:25.179935+0000) 2022-01-31T22:13:25.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:25 smithi171 conmon[51620]: debug 2022-01-31T22:13:25.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:25.035556+0000) 2022-01-31T22:13:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:25 smithi171 conmon[41853]: debug 2022-01-31T22:13:25.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:25.386734+0000) 2022-01-31T22:13:25.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:25 smithi167 conmon[54076]: debug 2022-01-31T22:13:25.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:25.683103+0000) 2022-01-31T22:13:25.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:25 smithi167 conmon[60316]: debug 2022-01-31T22:13:25.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:25.683103+0000) 2022-01-31T22:13:26.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:25 smithi167 conmon[49112]: debug 2022-01-31T22:13:25.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:25.763047+0000) 2022-01-31T22:13:26.344 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[35325]: debug 2022-01-31T22:13:26.064+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192286 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:26.345 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[41853]: debug 2022-01-31T22:13:26.044+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.045565+0000) 2022-01-31T22:13:26.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[46715]: debug 2022-01-31T22:13:26.044+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.045170+0000) 2022-01-31T22:13:26.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[46715]: debug 2022-01-31T22:13:26.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.180114+0000) 2022-01-31T22:13:26.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[51620]: debug 2022-01-31T22:13:26.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.035738+0000) 2022-01-31T22:13:26.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[51620]: debug 2022-01-31T22:13:26.044+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.045653+0000) 2022-01-31T22:13:26.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:26 smithi167 conmon[49112]: debug 2022-01-31T22:13:26.045+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.045742+0000) 2022-01-31T22:13:26.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:26 smithi167 conmon[54076]: debug 2022-01-31T22:13:26.044+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.044925+0000) 2022-01-31T22:13:26.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:26 smithi167 conmon[60316]: debug 2022-01-31T22:13:26.044+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.045331+0000) 2022-01-31T22:13:26.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:26 smithi171 conmon[41853]: debug 2022-01-31T22:13:26.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.386929+0000) 2022-01-31T22:13:26.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:26 smithi167 conmon[54076]: debug 2022-01-31T22:13:26.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.683301+0000) 2022-01-31T22:13:26.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:26 smithi167 conmon[60316]: debug 2022-01-31T22:13:26.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.683301+0000) 2022-01-31T22:13:27.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:26 smithi167 conmon[49112]: debug 2022-01-31T22:13:26.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:26.763254+0000) 2022-01-31T22:13:27.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:27 smithi171 conmon[46715]: debug 2022-01-31T22:13:27.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:27.180261+0000) 2022-01-31T22:13:27.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:27 smithi171 conmon[51620]: debug 2022-01-31T22:13:27.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:27.035955+0000) 2022-01-31T22:13:27.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:27 smithi171 conmon[41853]: debug 2022-01-31T22:13:27.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:27.387057+0000) 2022-01-31T22:13:27.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:27 smithi167 conmon[54076]: debug 2022-01-31T22:13:27.683+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:27.683519+0000) 2022-01-31T22:13:27.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:27 smithi167 conmon[60316]: debug 2022-01-31T22:13:27.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:27.683519+0000) 2022-01-31T22:13:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:27 smithi167 conmon[49112]: debug 2022-01-31T22:13:27.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:27.763390+0000) 2022-01-31T22:13:28.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:28 smithi171 conmon[46715]: debug 2022-01-31T22:13:28.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:28.180447+0000) 2022-01-31T22:13:28.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:28 smithi171 conmon[51620]: debug 2022-01-31T22:13:28.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:28.036112+0000) 2022-01-31T22:13:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:28 smithi171 conmon[41853]: debug 2022-01-31T22:13:28.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:28.387219+0000) 2022-01-31T22:13:28.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:28 smithi167 conmon[54076]: debug 2022-01-31T22:13:28.683+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:28.683702+0000) 2022-01-31T22:13:28.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:28 smithi167 conmon[60316]: debug 2022-01-31T22:13:28.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:28.683702+0000) 2022-01-31T22:13:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:28 smithi167 conmon[49112]: debug 2022-01-31T22:13:28.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:28.763573+0000) 2022-01-31T22:13:29.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:29 smithi171 conmon[46715]: debug 2022-01-31T22:13:29.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:29.180654+0000) 2022-01-31T22:13:29.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:29 smithi171 conmon[51620]: debug 2022-01-31T22:13:29.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:29.036312+0000) 2022-01-31T22:13:29.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:29 smithi171 conmon[41853]: debug 2022-01-31T22:13:29.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:29.387374+0000) 2022-01-31T22:13:29.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:29 smithi167 conmon[54076]: debug 2022-01-31T22:13:29.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:29.683896+0000) 2022-01-31T22:13:29.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:29 smithi167 conmon[60316]: debug 2022-01-31T22:13:29.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:29.683896+0000) 2022-01-31T22:13:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:29 smithi167 conmon[49112]: debug 2022-01-31T22:13:29.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:29.763718+0000) 2022-01-31T22:13:30.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:30 smithi171 conmon[46715]: debug 2022-01-31T22:13:30.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:30.180840+0000) 2022-01-31T22:13:30.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:30 smithi171 conmon[51620]: debug 2022-01-31T22:13:30.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:30.036516+0000) 2022-01-31T22:13:30.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:30 smithi171 conmon[41853]: debug 2022-01-31T22:13:30.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:30.387560+0000) 2022-01-31T22:13:30.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:30 smithi167 conmon[54076]: debug 2022-01-31T22:13:30.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:30.684036+0000) 2022-01-31T22:13:30.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:30 smithi167 conmon[60316]: debug 2022-01-31T22:13:30.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:30.684036+0000) 2022-01-31T22:13:31.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:30 smithi167 conmon[49112]: debug 2022-01-31T22:13:30.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:30.763868+0000) 2022-01-31T22:13:31.345 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[41853]: debug 2022-01-31T22:13:31.066+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.068527+0000) 2022-01-31T22:13:31.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[35325]: debug 2022-01-31T22:13:31.086+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192397 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:31.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[46715]: debug 2022-01-31T22:13:31.066+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.068172+0000) 2022-01-31T22:13:31.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[46715]: debug 2022-01-31T22:13:31.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.181043+0000) 2022-01-31T22:13:31.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[51620]: debug 2022-01-31T22:13:31.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.036699+0000) 2022-01-31T22:13:31.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[51620]: debug 2022-01-31T22:13:31.066+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.067927+0000) 2022-01-31T22:13:31.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:31 smithi167 conmon[49112]: debug 2022-01-31T22:13:31.066+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.068310+0000) 2022-01-31T22:13:31.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:31 smithi167 conmon[54076]: debug 2022-01-31T22:13:31.066+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.068189+0000) 2022-01-31T22:13:31.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:31 smithi167 conmon[60316]: debug 2022-01-31T22:13:31.066+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.067977+0000) 2022-01-31T22:13:31.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:31 smithi171 conmon[41853]: debug 2022-01-31T22:13:31.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.387737+0000) 2022-01-31T22:13:31.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:31 smithi167 conmon[54076]: debug 2022-01-31T22:13:31.682+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.684230+0000) 2022-01-31T22:13:31.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:31 smithi167 conmon[60316]: debug 2022-01-31T22:13:31.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.684230+0000) 2022-01-31T22:13:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:31 smithi167 conmon[49112]: debug 2022-01-31T22:13:31.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:31.764020+0000) 2022-01-31T22:13:32.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:32 smithi171 conmon[46715]: debug 2022-01-31T22:13:32.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:32.181181+0000) 2022-01-31T22:13:32.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:32 smithi171 conmon[51620]: debug 2022-01-31T22:13:32.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:32.036885+0000) 2022-01-31T22:13:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:32 smithi171 conmon[41853]: debug 2022-01-31T22:13:32.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:32.387909+0000) 2022-01-31T22:13:32.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:32 smithi167 conmon[54076]: debug 2022-01-31T22:13:32.683+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:32.684366+0000) 2022-01-31T22:13:32.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:32 smithi167 conmon[60316]: debug 2022-01-31T22:13:32.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:32.684366+0000) 2022-01-31T22:13:33.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:32 smithi167 conmon[49112]: debug 2022-01-31T22:13:32.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:32.764262+0000) 2022-01-31T22:13:33.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:33 smithi171 conmon[46715]: debug 2022-01-31T22:13:33.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:33.181338+0000) 2022-01-31T22:13:33.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:33 smithi171 conmon[51620]: debug 2022-01-31T22:13:33.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:33.037027+0000) 2022-01-31T22:13:33.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:33 smithi171 conmon[41853]: debug 2022-01-31T22:13:33.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:33.388115+0000) 2022-01-31T22:13:33.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:33 smithi167 conmon[54076]: debug 2022-01-31T22:13:33.683+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:33.684539+0000) 2022-01-31T22:13:33.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:33 smithi167 conmon[60316]: debug 2022-01-31T22:13:33.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:33.684539+0000) 2022-01-31T22:13:34.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:33 smithi167 conmon[49112]: debug 2022-01-31T22:13:33.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:33.764448+0000) 2022-01-31T22:13:34.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:34 smithi171 conmon[46715]: debug 2022-01-31T22:13:34.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:34.181572+0000) 2022-01-31T22:13:34.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:34 smithi171 conmon[51620]: debug 2022-01-31T22:13:34.035+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:34.037217+0000) 2022-01-31T22:13:34.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:34 smithi171 conmon[41853]: debug 2022-01-31T22:13:34.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:34.388259+0000) 2022-01-31T22:13:34.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:34 smithi167 conmon[54076]: debug 2022-01-31T22:13:34.683+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:34.684716+0000) 2022-01-31T22:13:34.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:34 smithi167 conmon[60316]: debug 2022-01-31T22:13:34.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:34.684716+0000) 2022-01-31T22:13:35.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:34 smithi167 conmon[49112]: debug 2022-01-31T22:13:34.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:34.764612+0000) 2022-01-31T22:13:35.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:35 smithi171 conmon[46715]: debug 2022-01-31T22:13:35.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:35.181762+0000) 2022-01-31T22:13:35.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:35 smithi171 conmon[51620]: debug 2022-01-31T22:13:35.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:35.037409+0000) 2022-01-31T22:13:35.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:35 smithi171 conmon[41853]: debug 2022-01-31T22:13:35.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:35.388365+0000) 2022-01-31T22:13:35.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:35 smithi167 conmon[54076]: debug 2022-01-31T22:13:35.684+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:35.684934+0000) 2022-01-31T22:13:35.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:35 smithi167 conmon[60316]: debug 2022-01-31T22:13:35.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:35.684934+0000) 2022-01-31T22:13:36.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:35 smithi167 conmon[49112]: debug 2022-01-31T22:13:35.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:35.764743+0000) 2022-01-31T22:13:36.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[35325]: debug 2022-01-31T22:13:36.108+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192520 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:36.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[41853]: debug 2022-01-31T22:13:36.089+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.091012+0000) 2022-01-31T22:13:36.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[46715]: debug 2022-01-31T22:13:36.089+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.090860+0000) 2022-01-31T22:13:36.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[46715]: debug 2022-01-31T22:13:36.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.181937+0000) 2022-01-31T22:13:36.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[51620]: debug 2022-01-31T22:13:36.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.037561+0000) 2022-01-31T22:13:36.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[51620]: debug 2022-01-31T22:13:36.088+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.090031+0000) 2022-01-31T22:13:36.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:36 smithi167 conmon[49112]: debug 2022-01-31T22:13:36.089+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.090529+0000) 2022-01-31T22:13:36.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:36 smithi167 conmon[54076]: debug 2022-01-31T22:13:36.088+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.090267+0000) 2022-01-31T22:13:36.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:36 smithi167 conmon[60316]: debug 2022-01-31T22:13:36.089+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.091303+0000) 2022-01-31T22:13:36.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:36 smithi171 conmon[41853]: debug 2022-01-31T22:13:36.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.388554+0000) 2022-01-31T22:13:36.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:36 smithi167 conmon[54076]: debug 2022-01-31T22:13:36.684+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.685143+0000) 2022-01-31T22:13:36.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:36 smithi167 conmon[60316]: debug 2022-01-31T22:13:36.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.685143+0000) 2022-01-31T22:13:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:36 smithi167 conmon[49112]: debug 2022-01-31T22:13:36.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:36.764914+0000) 2022-01-31T22:13:37.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:37 smithi171 conmon[46715]: debug 2022-01-31T22:13:37.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:37.182082+0000) 2022-01-31T22:13:37.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:37 smithi171 conmon[51620]: debug 2022-01-31T22:13:37.036+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:37.037780+0000) 2022-01-31T22:13:37.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:37 smithi171 conmon[41853]: debug 2022-01-31T22:13:37.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:37.388721+0000) 2022-01-31T22:13:37.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:37 smithi167 conmon[54076]: debug 2022-01-31T22:13:37.683+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:37.685278+0000) 2022-01-31T22:13:37.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:37 smithi167 conmon[60316]: debug 2022-01-31T22:13:37.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:37.685278+0000) 2022-01-31T22:13:38.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:37 smithi167 conmon[49112]: debug 2022-01-31T22:13:37.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:37.765062+0000) 2022-01-31T22:13:38.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:38 smithi171 conmon[46715]: debug 2022-01-31T22:13:38.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:38.182240+0000) 2022-01-31T22:13:38.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:38 smithi171 conmon[51620]: debug 2022-01-31T22:13:38.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:38.037944+0000) 2022-01-31T22:13:38.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:38 smithi171 conmon[41853]: debug 2022-01-31T22:13:38.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:38.388922+0000) 2022-01-31T22:13:38.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:38 smithi167 conmon[54076]: debug 2022-01-31T22:13:38.684+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:38.685518+0000) 2022-01-31T22:13:38.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:38 smithi167 conmon[60316]: debug 2022-01-31T22:13:38.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:38.685518+0000) 2022-01-31T22:13:39.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:38 smithi171 conmon[35325]: debug 2022-01-31T22:13:38.816+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:13:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:39 smithi171 conmon[51620]: debug 2022-01-31T22:13:39.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:39.038137+0000) 2022-01-31T22:13:39.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:38 smithi167 conmon[49112]: debug 2022-01-31T22:13:38.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:38.765251+0000) 2022-01-31T22:13:39.388 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:39 smithi171 conmon[46715]: debug 2022-01-31T22:13:39.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:39.182422+0000) 2022-01-31T22:13:39.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:39 smithi167 conmon[54076]: debug 2022-01-31T22:13:39.684+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:39.685753+0000) 2022-01-31T22:13:39.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:39 smithi167 conmon[60316]: debug 2022-01-31T22:13:39.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:39.685753+0000) 2022-01-31T22:13:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:39 smithi171 conmon[41853]: debug 2022-01-31T22:13:39.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:39.389100+0000) 2022-01-31T22:13:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:39 smithi167 conmon[49112]: debug 2022-01-31T22:13:39.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:39.765470+0000) 2022-01-31T22:13:40.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:40 smithi171 conmon[46715]: debug 2022-01-31T22:13:40.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:40.182602+0000) 2022-01-31T22:13:40.347 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:40 smithi171 conmon[51620]: debug 2022-01-31T22:13:40.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:40.038341+0000) 2022-01-31T22:13:40.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:40 smithi171 conmon[41853]: debug 2022-01-31T22:13:40.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:40.389316+0000) 2022-01-31T22:13:40.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:40 smithi167 conmon[54076]: debug 2022-01-31T22:13:40.684+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:40.685938+0000) 2022-01-31T22:13:40.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:40 smithi167 conmon[60316]: debug 2022-01-31T22:13:40.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:40.685938+0000) 2022-01-31T22:13:41.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:40 smithi167 conmon[49112]: debug 2022-01-31T22:13:40.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:40.765556+0000) 2022-01-31T22:13:41.347 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[35325]: debug 2022-01-31T22:13:41.132+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192631 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:41.348 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[41853]: debug 2022-01-31T22:13:41.112+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.112895+0000) 2022-01-31T22:13:41.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[46715]: debug 2022-01-31T22:13:41.112+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.113357+0000) 2022-01-31T22:13:41.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[46715]: debug 2022-01-31T22:13:41.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.182764+0000) 2022-01-31T22:13:41.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[51620]: debug 2022-01-31T22:13:41.037+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.038553+0000) 2022-01-31T22:13:41.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[51620]: debug 2022-01-31T22:13:41.111+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.112583+0000) 2022-01-31T22:13:41.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:41 smithi167 conmon[49112]: debug 2022-01-31T22:13:41.111+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.113304+0000) 2022-01-31T22:13:41.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:41 smithi167 conmon[54076]: debug 2022-01-31T22:13:41.111+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.113385+0000) 2022-01-31T22:13:41.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:41 smithi167 conmon[60316]: debug 2022-01-31T22:13:41.112+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.114261+0000) 2022-01-31T22:13:41.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:41 smithi171 conmon[41853]: debug 2022-01-31T22:13:41.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.389527+0000) 2022-01-31T22:13:41.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:41 smithi167 conmon[60316]: debug 2022-01-31T22:13:41.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.686134+0000) 2022-01-31T22:13:41.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:41 smithi167 conmon[54076]: debug 2022-01-31T22:13:41.685+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.686134+0000) 2022-01-31T22:13:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:41 smithi167 conmon[49112]: debug 2022-01-31T22:13:41.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:41.765674+0000) 2022-01-31T22:13:42.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:42 smithi171 conmon[46715]: debug 2022-01-31T22:13:42.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:42.182925+0000) 2022-01-31T22:13:42.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:42 smithi171 conmon[51620]: debug 2022-01-31T22:13:42.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:42.038760+0000) 2022-01-31T22:13:42.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:42 smithi171 conmon[41853]: debug 2022-01-31T22:13:42.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:42.389665+0000) 2022-01-31T22:13:42.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:42 smithi167 conmon[54076]: debug 2022-01-31T22:13:42.685+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:42.686309+0000) 2022-01-31T22:13:42.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:42 smithi167 conmon[60316]: debug 2022-01-31T22:13:42.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:42.686309+0000) 2022-01-31T22:13:43.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:42 smithi167 conmon[49112]: debug 2022-01-31T22:13:42.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:42.765842+0000) 2022-01-31T22:13:43.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:43 smithi171 conmon[46715]: debug 2022-01-31T22:13:43.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:43.183070+0000) 2022-01-31T22:13:43.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:43 smithi171 conmon[51620]: debug 2022-01-31T22:13:43.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:43.038937+0000) 2022-01-31T22:13:43.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:43 smithi171 conmon[41853]: debug 2022-01-31T22:13:43.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:43.389858+0000) 2022-01-31T22:13:43.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:43 smithi167 conmon[54076]: debug 2022-01-31T22:13:43.685+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:43.686511+0000) 2022-01-31T22:13:43.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:43 smithi167 conmon[60316]: debug 2022-01-31T22:13:43.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:43.686511+0000) 2022-01-31T22:13:44.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:43 smithi167 conmon[49112]: debug 2022-01-31T22:13:43.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:43.765994+0000) 2022-01-31T22:13:44.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:44 smithi171 conmon[46715]: debug 2022-01-31T22:13:44.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:44.183265+0000) 2022-01-31T22:13:44.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:44 smithi171 conmon[51620]: debug 2022-01-31T22:13:44.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:44.039149+0000) 2022-01-31T22:13:44.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:44 smithi171 conmon[41853]: debug 2022-01-31T22:13:44.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:44.390011+0000) 2022-01-31T22:13:44.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:44 smithi167 conmon[54076]: debug 2022-01-31T22:13:44.686+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:44.686745+0000) 2022-01-31T22:13:44.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:44 smithi167 conmon[60316]: debug 2022-01-31T22:13:44.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:44.686745+0000) 2022-01-31T22:13:45.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:44 smithi167 conmon[49112]: debug 2022-01-31T22:13:44.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:44.766182+0000) 2022-01-31T22:13:45.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:45 smithi171 conmon[46715]: debug 2022-01-31T22:13:45.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:45.183421+0000) 2022-01-31T22:13:45.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:45 smithi171 conmon[51620]: debug 2022-01-31T22:13:45.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:45.039334+0000) 2022-01-31T22:13:45.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:45 smithi171 conmon[41853]: debug 2022-01-31T22:13:45.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:45.390171+0000) 2022-01-31T22:13:45.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:45 smithi167 conmon[54076]: debug 2022-01-31T22:13:45.685+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:45.686900+0000) 2022-01-31T22:13:45.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:45 smithi167 conmon[60316]: debug 2022-01-31T22:13:45.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:45.686901+0000) 2022-01-31T22:13:46.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:45 smithi167 conmon[49112]: debug 2022-01-31T22:13:45.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:45.766365+0000) 2022-01-31T22:13:46.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[41853]: debug 2022-01-31T22:13:46.135+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.136453+0000) 2022-01-31T22:13:46.348 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[35325]: debug 2022-01-31T22:13:46.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192742 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:46.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[46715]: debug 2022-01-31T22:13:46.135+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.136227+0000) 2022-01-31T22:13:46.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[46715]: debug 2022-01-31T22:13:46.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.183604+0000) 2022-01-31T22:13:46.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[51620]: debug 2022-01-31T22:13:46.038+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.039529+0000) 2022-01-31T22:13:46.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[51620]: debug 2022-01-31T22:13:46.135+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.135976+0000) 2022-01-31T22:13:46.418 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:46 smithi167 conmon[49112]: debug 2022-01-31T22:13:46.133+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.135414+0000) 2022-01-31T22:13:46.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:46 smithi167 conmon[54076]: debug 2022-01-31T22:13:46.134+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.135750+0000) 2022-01-31T22:13:46.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:46 smithi167 conmon[60316]: debug 2022-01-31T22:13:46.135+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.136739+0000) 2022-01-31T22:13:46.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:46 smithi171 conmon[41853]: debug 2022-01-31T22:13:46.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.390362+0000) 2022-01-31T22:13:46.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:46 smithi167 conmon[54076]: debug 2022-01-31T22:13:46.686+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.687135+0000) 2022-01-31T22:13:46.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:46 smithi167 conmon[60316]: debug 2022-01-31T22:13:46.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.687135+0000) 2022-01-31T22:13:47.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:46 smithi167 conmon[49112]: debug 2022-01-31T22:13:46.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:46.766553+0000) 2022-01-31T22:13:47.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:47 smithi171 conmon[46715]: debug 2022-01-31T22:13:47.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:47.183770+0000) 2022-01-31T22:13:47.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:47 smithi171 conmon[51620]: debug 2022-01-31T22:13:47.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:47.039717+0000) 2022-01-31T22:13:47.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:47 smithi171 conmon[41853]: debug 2022-01-31T22:13:47.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:47.390530+0000) 2022-01-31T22:13:47.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:47 smithi167 conmon[54076]: debug 2022-01-31T22:13:47.686+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:47.687285+0000) 2022-01-31T22:13:47.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:47 smithi167 conmon[60316]: debug 2022-01-31T22:13:47.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:47.687285+0000) 2022-01-31T22:13:48.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:47 smithi167 conmon[49112]: debug 2022-01-31T22:13:47.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:47.766653+0000) 2022-01-31T22:13:48.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:48 smithi171 conmon[46715]: debug 2022-01-31T22:13:48.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:48.183960+0000) 2022-01-31T22:13:48.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:48 smithi171 conmon[51620]: debug 2022-01-31T22:13:48.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:48.039854+0000) 2022-01-31T22:13:48.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:48 smithi171 conmon[41853]: debug 2022-01-31T22:13:48.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:48.390737+0000) 2022-01-31T22:13:48.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:48 smithi167 conmon[54076]: debug 2022-01-31T22:13:48.686+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:48.687535+0000) 2022-01-31T22:13:48.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:48 smithi167 conmon[60316]: debug 2022-01-31T22:13:48.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:48.687537+0000) 2022-01-31T22:13:49.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:48 smithi167 conmon[49112]: debug 2022-01-31T22:13:48.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:48.766838+0000) 2022-01-31T22:13:49.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:49 smithi171 conmon[46715]: debug 2022-01-31T22:13:49.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:49.184145+0000) 2022-01-31T22:13:49.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:49 smithi171 conmon[51620]: debug 2022-01-31T22:13:49.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:49.039979+0000) 2022-01-31T22:13:49.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:49 smithi171 conmon[41853]: debug 2022-01-31T22:13:49.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:49.390906+0000) 2022-01-31T22:13:49.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:49 smithi167 conmon[54076]: debug 2022-01-31T22:13:49.687+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:49.687765+0000) 2022-01-31T22:13:49.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:49 smithi167 conmon[60316]: debug 2022-01-31T22:13:49.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:49.687765+0000) 2022-01-31T22:13:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:49 smithi167 conmon[49112]: debug 2022-01-31T22:13:49.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:49.766964+0000) 2022-01-31T22:13:50.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:50 smithi171 conmon[46715]: debug 2022-01-31T22:13:50.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:50.184329+0000) 2022-01-31T22:13:50.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:50 smithi171 conmon[51620]: debug 2022-01-31T22:13:50.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:50.040163+0000) 2022-01-31T22:13:50.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:50 smithi171 conmon[41853]: debug 2022-01-31T22:13:50.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:50.391113+0000) 2022-01-31T22:13:50.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:50 smithi167 conmon[54076]: debug 2022-01-31T22:13:50.686+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:50.687924+0000) 2022-01-31T22:13:50.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:50 smithi167 conmon[60316]: debug 2022-01-31T22:13:50.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:50.687917+0000) 2022-01-31T22:13:51.163 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:50 smithi167 conmon[49112]: debug 2022-01-31T22:13:50.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:50.767127+0000) 2022-01-31T22:13:51.348 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[35325]: debug 2022-01-31T22:13:51.184+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192853 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:51.349 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[41853]: debug 2022-01-31T22:13:51.163+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.164734+0000) 2022-01-31T22:13:51.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[51620]: debug 2022-01-31T22:13:51.039+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.040369+0000) 2022-01-31T22:13:51.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[51620]: debug 2022-01-31T22:13:51.163+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.164151+0000) 2022-01-31T22:13:51.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[46715]: debug 2022-01-31T22:13:51.164+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.165633+0000) 2022-01-31T22:13:51.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[46715]: debug 2022-01-31T22:13:51.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.184498+0000) 2022-01-31T22:13:51.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:51 smithi167 conmon[49112]: debug 2022-01-31T22:13:51.162+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.164371+0000) 2022-01-31T22:13:51.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:51 smithi167 conmon[54076]: debug 2022-01-31T22:13:51.163+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.164701+0000) 2022-01-31T22:13:51.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:51 smithi167 conmon[60316]: debug 2022-01-31T22:13:51.163+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.164883+0000) 2022-01-31T22:13:51.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:51 smithi171 conmon[41853]: debug 2022-01-31T22:13:51.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.391295+0000) 2022-01-31T22:13:51.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:51 smithi167 conmon[54076]: debug 2022-01-31T22:13:51.687+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.688115+0000) 2022-01-31T22:13:51.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:51 smithi167 conmon[60316]: debug 2022-01-31T22:13:51.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.688114+0000) 2022-01-31T22:13:52.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:51 smithi167 conmon[49112]: debug 2022-01-31T22:13:51.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:51.767312+0000) 2022-01-31T22:13:52.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:52 smithi171 conmon[46715]: debug 2022-01-31T22:13:52.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:52.184669+0000) 2022-01-31T22:13:52.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:52 smithi171 conmon[51620]: debug 2022-01-31T22:13:52.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:52.040530+0000) 2022-01-31T22:13:52.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:52 smithi171 conmon[41853]: debug 2022-01-31T22:13:52.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:52.391449+0000) 2022-01-31T22:13:52.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:52 smithi167 conmon[54076]: debug 2022-01-31T22:13:52.687+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:52.688276+0000) 2022-01-31T22:13:52.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:52 smithi167 conmon[60316]: debug 2022-01-31T22:13:52.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:52.688276+0000) 2022-01-31T22:13:53.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:52 smithi167 conmon[49112]: debug 2022-01-31T22:13:52.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:52.767460+0000) 2022-01-31T22:13:53.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:53 smithi171 conmon[46715]: debug 2022-01-31T22:13:53.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:53.184878+0000) 2022-01-31T22:13:53.349 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:53 smithi171 conmon[51620]: debug 2022-01-31T22:13:53.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:53.040670+0000) 2022-01-31T22:13:53.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:53 smithi171 conmon[41853]: debug 2022-01-31T22:13:53.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:53.391653+0000) 2022-01-31T22:13:53.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:53 smithi167 conmon[54076]: debug 2022-01-31T22:13:53.687+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:53.688438+0000) 2022-01-31T22:13:53.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:53 smithi167 conmon[60316]: debug 2022-01-31T22:13:53.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:53.688438+0000) 2022-01-31T22:13:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:53 smithi171 conmon[35325]: debug 2022-01-31T22:13:53.817+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:13:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:54 smithi171 conmon[51620]: debug 2022-01-31T22:13:54.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:54.040889+0000) 2022-01-31T22:13:54.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:53 smithi167 conmon[49112]: debug 2022-01-31T22:13:53.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:53.767599+0000) 2022-01-31T22:13:54.391 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:54 smithi171 conmon[46715]: debug 2022-01-31T22:13:54.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:54.185085+0000) 2022-01-31T22:13:54.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:54 smithi167 conmon[54076]: debug 2022-01-31T22:13:54.688+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:54.688608+0000) 2022-01-31T22:13:54.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:54 smithi167 conmon[60316]: debug 2022-01-31T22:13:54.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:54.688608+0000) 2022-01-31T22:13:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:54 smithi171 conmon[41853]: debug 2022-01-31T22:13:54.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:54.391825+0000) 2022-01-31T22:13:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:54 smithi167 conmon[49112]: debug 2022-01-31T22:13:54.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:54.767789+0000) 2022-01-31T22:13:55.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:55 smithi171 conmon[46715]: debug 2022-01-31T22:13:55.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:55.185238+0000) 2022-01-31T22:13:55.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:55 smithi171 conmon[51620]: debug 2022-01-31T22:13:55.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:55.041076+0000) 2022-01-31T22:13:55.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:55 smithi171 conmon[41853]: debug 2022-01-31T22:13:55.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:55.391969+0000) 2022-01-31T22:13:55.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:55 smithi167 conmon[54076]: debug 2022-01-31T22:13:55.688+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:55.688831+0000) 2022-01-31T22:13:55.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:55 smithi167 conmon[60316]: debug 2022-01-31T22:13:55.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:55.688832+0000) 2022-01-31T22:13:56.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:55 smithi167 conmon[49112]: debug 2022-01-31T22:13:55.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:55.767973+0000) 2022-01-31T22:13:56.349 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[35325]: debug 2022-01-31T22:13:56.206+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 192964 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:13:56.350 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[41853]: debug 2022-01-31T22:13:56.186+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.187229+0000) 2022-01-31T22:13:56.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[46715]: debug 2022-01-31T22:13:56.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.185456+0000) 2022-01-31T22:13:56.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[46715]: debug 2022-01-31T22:13:56.186+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.187687+0000) 2022-01-31T22:13:56.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[51620]: debug 2022-01-31T22:13:56.040+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.041261+0000) 2022-01-31T22:13:56.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[51620]: debug 2022-01-31T22:13:56.186+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.186858+0000) 2022-01-31T22:13:56.490 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:56 smithi167 conmon[49112]: debug 2022-01-31T22:13:56.187+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.188268+0000) 2022-01-31T22:13:56.490 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:56 smithi167 conmon[54076]: debug 2022-01-31T22:13:56.186+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.187325+0000) 2022-01-31T22:13:56.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:56 smithi167 conmon[60316]: debug 2022-01-31T22:13:56.187+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.187860+0000) 2022-01-31T22:13:56.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:56 smithi171 conmon[41853]: debug 2022-01-31T22:13:56.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.392194+0000) 2022-01-31T22:13:56.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:56 smithi167 conmon[54076]: debug 2022-01-31T22:13:56.688+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.689030+0000) 2022-01-31T22:13:56.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:56 smithi167 conmon[60316]: debug 2022-01-31T22:13:56.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.689029+0000) 2022-01-31T22:13:57.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:56 smithi167 conmon[49112]: debug 2022-01-31T22:13:56.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:56.768165+0000) 2022-01-31T22:13:57.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:57 smithi171 conmon[46715]: debug 2022-01-31T22:13:57.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:57.185625+0000) 2022-01-31T22:13:57.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:57 smithi171 conmon[51620]: debug 2022-01-31T22:13:57.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:57.041467+0000) 2022-01-31T22:13:57.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:57 smithi171 conmon[41853]: debug 2022-01-31T22:13:57.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:57.392376+0000) 2022-01-31T22:13:57.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:57 smithi167 conmon[54076]: debug 2022-01-31T22:13:57.688+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:57.689217+0000) 2022-01-31T22:13:57.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:57 smithi167 conmon[60316]: debug 2022-01-31T22:13:57.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:57.689217+0000) 2022-01-31T22:13:58.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:57 smithi167 conmon[49112]: debug 2022-01-31T22:13:57.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:57.768327+0000) 2022-01-31T22:13:58.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:58 smithi171 conmon[46715]: debug 2022-01-31T22:13:58.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:58.185772+0000) 2022-01-31T22:13:58.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:58 smithi171 conmon[51620]: debug 2022-01-31T22:13:58.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:58.041645+0000) 2022-01-31T22:13:58.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:58 smithi171 conmon[41853]: debug 2022-01-31T22:13:58.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:58.392567+0000) 2022-01-31T22:13:58.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:58 smithi167 conmon[54076]: debug 2022-01-31T22:13:58.688+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:58.689425+0000) 2022-01-31T22:13:58.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:58 smithi167 conmon[60316]: debug 2022-01-31T22:13:58.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:58.689425+0000) 2022-01-31T22:13:59.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:58 smithi167 conmon[49112]: debug 2022-01-31T22:13:58.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:58.768529+0000) 2022-01-31T22:13:59.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:13:59 smithi171 conmon[46715]: debug 2022-01-31T22:13:59.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:59.185969+0000) 2022-01-31T22:13:59.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:13:59 smithi171 conmon[51620]: debug 2022-01-31T22:13:59.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:59.041837+0000) 2022-01-31T22:13:59.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:13:59 smithi171 conmon[41853]: debug 2022-01-31T22:13:59.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:59.392719+0000) 2022-01-31T22:13:59.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:13:59 smithi167 conmon[54076]: debug 2022-01-31T22:13:59.689+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:59.689623+0000) 2022-01-31T22:13:59.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:13:59 smithi167 conmon[60316]: debug 2022-01-31T22:13:59.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:59.689577+0000) 2022-01-31T22:14:00.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:13:59 smithi167 conmon[49112]: debug 2022-01-31T22:13:59.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:13:59.768715+0000) 2022-01-31T22:14:00.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:00 smithi171 conmon[46715]: debug 2022-01-31T22:14:00.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:00.186146+0000) 2022-01-31T22:14:00.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:00 smithi171 conmon[51620]: debug 2022-01-31T22:14:00.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:00.042039+0000) 2022-01-31T22:14:00.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:00 smithi171 conmon[41853]: debug 2022-01-31T22:14:00.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:00.392917+0000) 2022-01-31T22:14:00.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:00 smithi167 conmon[54076]: debug 2022-01-31T22:14:00.689+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:00.689800+0000) 2022-01-31T22:14:00.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:00 smithi167 conmon[60316]: debug 2022-01-31T22:14:00.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:00.689749+0000) 2022-01-31T22:14:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:00 smithi167 conmon[49112]: debug 2022-01-31T22:14:00.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:00.768893+0000) 2022-01-31T22:14:01.350 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[35325]: debug 2022-01-31T22:14:01.229+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193076 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:01.351 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[41853]: debug 2022-01-31T22:14:01.209+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.210127+0000) 2022-01-31T22:14:01.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[51620]: debug 2022-01-31T22:14:01.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.042271+0000) 2022-01-31T22:14:01.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[51620]: debug 2022-01-31T22:14:01.208+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.209060+0000) 2022-01-31T22:14:01.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[46715]: debug 2022-01-31T22:14:01.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.186345+0000) 2022-01-31T22:14:01.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[46715]: debug 2022-01-31T22:14:01.209+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.209827+0000) 2022-01-31T22:14:01.491 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:01 smithi167 conmon[49112]: debug 2022-01-31T22:14:01.209+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.210200+0000) 2022-01-31T22:14:01.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:01 smithi167 conmon[54076]: debug 2022-01-31T22:14:01.210+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.210931+0000) 2022-01-31T22:14:01.492 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:01 smithi167 conmon[60316]: debug 2022-01-31T22:14:01.209+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.210453+0000) 2022-01-31T22:14:01.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:01 smithi171 conmon[41853]: debug 2022-01-31T22:14:01.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.393063+0000) 2022-01-31T22:14:01.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:01 smithi167 conmon[54076]: debug 2022-01-31T22:14:01.689+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.689983+0000) 2022-01-31T22:14:01.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:01 smithi167 conmon[60316]: debug 2022-01-31T22:14:01.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.689983+0000) 2022-01-31T22:14:02.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:01 smithi167 conmon[49112]: debug 2022-01-31T22:14:01.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:01.769050+0000) 2022-01-31T22:14:02.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:02 smithi171 conmon[46715]: debug 2022-01-31T22:14:02.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:02.186502+0000) 2022-01-31T22:14:02.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:02 smithi171 conmon[51620]: debug 2022-01-31T22:14:02.041+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:02.042460+0000) 2022-01-31T22:14:02.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:02 smithi171 conmon[41853]: debug 2022-01-31T22:14:02.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:02.393211+0000) 2022-01-31T22:14:02.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:02 smithi167 conmon[54076]: debug 2022-01-31T22:14:02.689+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:02.690120+0000) 2022-01-31T22:14:02.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:02 smithi167 conmon[60316]: debug 2022-01-31T22:14:02.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:02.690120+0000) 2022-01-31T22:14:03.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:02 smithi167 conmon[49112]: debug 2022-01-31T22:14:02.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:02.769243+0000) 2022-01-31T22:14:03.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:03 smithi171 conmon[46715]: debug 2022-01-31T22:14:03.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:03.186674+0000) 2022-01-31T22:14:03.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:03 smithi171 conmon[51620]: debug 2022-01-31T22:14:03.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:03.042626+0000) 2022-01-31T22:14:03.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:03 smithi171 conmon[41853]: debug 2022-01-31T22:14:03.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:03.393432+0000) 2022-01-31T22:14:03.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:03 smithi167 conmon[54076]: debug 2022-01-31T22:14:03.689+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:03.690268+0000) 2022-01-31T22:14:03.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:03 smithi167 conmon[60316]: debug 2022-01-31T22:14:03.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:03.690227+0000) 2022-01-31T22:14:04.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:03 smithi167 conmon[49112]: debug 2022-01-31T22:14:03.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:03.769425+0000) 2022-01-31T22:14:04.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:04 smithi171 conmon[46715]: debug 2022-01-31T22:14:04.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:04.186912+0000) 2022-01-31T22:14:04.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:04 smithi171 conmon[51620]: debug 2022-01-31T22:14:04.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:04.042812+0000) 2022-01-31T22:14:04.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:04 smithi171 conmon[41853]: debug 2022-01-31T22:14:04.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:04.393611+0000) 2022-01-31T22:14:04.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:04 smithi167 conmon[54076]: debug 2022-01-31T22:14:04.689+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:04.690427+0000) 2022-01-31T22:14:04.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:04 smithi167 conmon[60316]: debug 2022-01-31T22:14:04.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:04.690427+0000) 2022-01-31T22:14:05.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:04 smithi167 conmon[49112]: debug 2022-01-31T22:14:04.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:04.769608+0000) 2022-01-31T22:14:05.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:05 smithi171 conmon[46715]: debug 2022-01-31T22:14:05.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:05.187092+0000) 2022-01-31T22:14:05.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:05 smithi171 conmon[51620]: debug 2022-01-31T22:14:05.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:05.042962+0000) 2022-01-31T22:14:05.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:05 smithi171 conmon[41853]: debug 2022-01-31T22:14:05.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:05.393763+0000) 2022-01-31T22:14:05.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:05 smithi167 conmon[54076]: debug 2022-01-31T22:14:05.690+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:05.690565+0000) 2022-01-31T22:14:05.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:05 smithi167 conmon[60316]: debug 2022-01-31T22:14:05.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:05.690565+0000) 2022-01-31T22:14:06.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:05 smithi167 conmon[49112]: debug 2022-01-31T22:14:05.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:05.769793+0000) 2022-01-31T22:14:06.351 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[35325]: debug 2022-01-31T22:14:06.251+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193190 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:06.352 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[41853]: debug 2022-01-31T22:14:06.231+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.232331+0000) 2022-01-31T22:14:06.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[51620]: debug 2022-01-31T22:14:06.042+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.043162+0000) 2022-01-31T22:14:06.353 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[51620]: debug 2022-01-31T22:14:06.231+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.232477+0000) 2022-01-31T22:14:06.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[46715]: debug 2022-01-31T22:14:06.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.187265+0000) 2022-01-31T22:14:06.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[46715]: debug 2022-01-31T22:14:06.232+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.233444+0000) 2022-01-31T22:14:06.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:06 smithi167 conmon[49112]: debug 2022-01-31T22:14:06.233+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.234190+0000) 2022-01-31T22:14:06.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:06 smithi167 conmon[54076]: debug 2022-01-31T22:14:06.233+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.233523+0000) 2022-01-31T22:14:06.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:06 smithi167 conmon[60316]: debug 2022-01-31T22:14:06.233+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.233650+0000) 2022-01-31T22:14:06.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:06 smithi171 conmon[41853]: debug 2022-01-31T22:14:06.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.393918+0000) 2022-01-31T22:14:06.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:06 smithi167 conmon[60316]: debug 2022-01-31T22:14:06.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.690696+0000) 2022-01-31T22:14:06.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:06 smithi167 conmon[54076]: debug 2022-01-31T22:14:06.690+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.690696+0000) 2022-01-31T22:14:07.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:06 smithi167 conmon[49112]: debug 2022-01-31T22:14:06.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:06.769951+0000) 2022-01-31T22:14:07.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:07 smithi171 conmon[46715]: debug 2022-01-31T22:14:07.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:07.187434+0000) 2022-01-31T22:14:07.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:07 smithi171 conmon[51620]: debug 2022-01-31T22:14:07.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:07.043341+0000) 2022-01-31T22:14:07.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:07 smithi171 conmon[41853]: debug 2022-01-31T22:14:07.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:07.394027+0000) 2022-01-31T22:14:07.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:07 smithi167 conmon[54076]: debug 2022-01-31T22:14:07.690+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:07.690830+0000) 2022-01-31T22:14:07.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:07 smithi167 conmon[60316]: debug 2022-01-31T22:14:07.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:07.690833+0000) 2022-01-31T22:14:08.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:07 smithi167 conmon[49112]: debug 2022-01-31T22:14:07.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:07.770100+0000) 2022-01-31T22:14:08.351 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:08 smithi171 conmon[46715]: debug 2022-01-31T22:14:08.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:08.187584+0000) 2022-01-31T22:14:08.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:08 smithi171 conmon[51620]: debug 2022-01-31T22:14:08.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:08.043493+0000) 2022-01-31T22:14:08.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:08 smithi171 conmon[41853]: debug 2022-01-31T22:14:08.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:08.394214+0000) 2022-01-31T22:14:08.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:08 smithi167 conmon[60316]: debug 2022-01-31T22:14:08.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:08.690986+0000) 2022-01-31T22:14:08.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:08 smithi167 conmon[54076]: debug 2022-01-31T22:14:08.690+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:08.690986+0000) 2022-01-31T22:14:09.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:08 smithi167 conmon[49112]: debug 2022-01-31T22:14:08.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:08.770313+0000) 2022-01-31T22:14:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:08 smithi171 conmon[35325]: debug 2022-01-31T22:14:08.818+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:14:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:09 smithi171 conmon[51620]: debug 2022-01-31T22:14:09.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:09.043693+0000) 2022-01-31T22:14:09.393 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:09 smithi171 conmon[46715]: debug 2022-01-31T22:14:09.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:09.187782+0000) 2022-01-31T22:14:09.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:09 smithi167 conmon[54076]: debug 2022-01-31T22:14:09.690+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:09.691180+0000) 2022-01-31T22:14:09.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:09 smithi167 conmon[60316]: debug 2022-01-31T22:14:09.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:09.691181+0000) 2022-01-31T22:14:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:09 smithi171 conmon[41853]: debug 2022-01-31T22:14:09.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:09.394418+0000) 2022-01-31T22:14:10.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:09 smithi167 conmon[49112]: debug 2022-01-31T22:14:09.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:09.770488+0000) 2022-01-31T22:14:10.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:10 smithi171 conmon[46715]: debug 2022-01-31T22:14:10.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:10.187979+0000) 2022-01-31T22:14:10.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:10 smithi171 conmon[51620]: debug 2022-01-31T22:14:10.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:10.043902+0000) 2022-01-31T22:14:10.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:10 smithi171 conmon[41853]: debug 2022-01-31T22:14:10.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:10.394670+0000) 2022-01-31T22:14:10.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:10 smithi167 conmon[60316]: debug 2022-01-31T22:14:10.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:10.691412+0000) 2022-01-31T22:14:10.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:10 smithi167 conmon[54076]: debug 2022-01-31T22:14:10.690+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:10.691412+0000) 2022-01-31T22:14:11.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:10 smithi167 conmon[49112]: debug 2022-01-31T22:14:10.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:10.770714+0000) 2022-01-31T22:14:11.352 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[35325]: debug 2022-01-31T22:14:11.274+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193300 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:11.353 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[41853]: debug 2022-01-31T22:14:11.254+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.254939+0000) 2022-01-31T22:14:11.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[46715]: debug 2022-01-31T22:14:11.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.188126+0000) 2022-01-31T22:14:11.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[46715]: debug 2022-01-31T22:14:11.255+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.255928+0000) 2022-01-31T22:14:11.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[51620]: debug 2022-01-31T22:14:11.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.044058+0000) 2022-01-31T22:14:11.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[51620]: debug 2022-01-31T22:14:11.253+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.254263+0000) 2022-01-31T22:14:11.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:11 smithi167 conmon[49112]: debug 2022-01-31T22:14:11.255+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.255775+0000) 2022-01-31T22:14:11.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:11 smithi167 conmon[54076]: debug 2022-01-31T22:14:11.255+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.255925+0000) 2022-01-31T22:14:11.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:11 smithi167 conmon[60316]: debug 2022-01-31T22:14:11.255+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.255648+0000) 2022-01-31T22:14:11.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:11 smithi171 conmon[41853]: debug 2022-01-31T22:14:11.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.394829+0000) 2022-01-31T22:14:11.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:11 smithi167 conmon[54076]: debug 2022-01-31T22:14:11.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.691614+0000) 2022-01-31T22:14:11.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:11 smithi167 conmon[60316]: debug 2022-01-31T22:14:11.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.691614+0000) 2022-01-31T22:14:12.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:11 smithi167 conmon[49112]: debug 2022-01-31T22:14:11.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:11.770889+0000) 2022-01-31T22:14:12.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:12 smithi171 conmon[51620]: debug 2022-01-31T22:14:12.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:12.044255+0000) 2022-01-31T22:14:12.394 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:12 smithi171 conmon[46715]: debug 2022-01-31T22:14:12.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:12.188310+0000) 2022-01-31T22:14:12.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:12 smithi167 conmon[54076]: debug 2022-01-31T22:14:12.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:12.691757+0000) 2022-01-31T22:14:12.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:12 smithi167 conmon[60316]: debug 2022-01-31T22:14:12.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:12.691757+0000) 2022-01-31T22:14:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:12 smithi171 conmon[41853]: debug 2022-01-31T22:14:12.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:12.394981+0000) 2022-01-31T22:14:13.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:12 smithi167 conmon[49112]: debug 2022-01-31T22:14:12.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:12.771044+0000) 2022-01-31T22:14:13.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:13 smithi171 conmon[46715]: debug 2022-01-31T22:14:13.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:13.188478+0000) 2022-01-31T22:14:13.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:13 smithi171 conmon[51620]: debug 2022-01-31T22:14:13.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:13.044427+0000) 2022-01-31T22:14:13.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:13 smithi171 conmon[41853]: debug 2022-01-31T22:14:13.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:13.395214+0000) 2022-01-31T22:14:13.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:13 smithi167 conmon[54076]: debug 2022-01-31T22:14:13.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:13.691930+0000) 2022-01-31T22:14:13.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:13 smithi167 conmon[60316]: debug 2022-01-31T22:14:13.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:13.691930+0000) 2022-01-31T22:14:14.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:13 smithi167 conmon[49112]: debug 2022-01-31T22:14:13.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:13.771199+0000) 2022-01-31T22:14:14.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:14 smithi171 conmon[46715]: debug 2022-01-31T22:14:14.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:14.188713+0000) 2022-01-31T22:14:14.353 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:14 smithi171 conmon[51620]: debug 2022-01-31T22:14:14.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:14.044635+0000) 2022-01-31T22:14:14.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:14 smithi171 conmon[41853]: debug 2022-01-31T22:14:14.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:14.395377+0000) 2022-01-31T22:14:14.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:14 smithi167 conmon[60316]: debug 2022-01-31T22:14:14.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:14.692100+0000) 2022-01-31T22:14:14.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:14 smithi167 conmon[54076]: debug 2022-01-31T22:14:14.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:14.692100+0000) 2022-01-31T22:14:15.053 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:14 smithi167 conmon[49112]: debug 2022-01-31T22:14:14.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:14.771372+0000) 2022-01-31T22:14:15.297 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:15 smithi171 conmon[51620]: debug 2022-01-31T22:14:15.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:15.044840+0000) 2022-01-31T22:14:15.298 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:15 smithi171 conmon[46715]: debug 2022-01-31T22:14:15.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:15.188925+0000) 2022-01-31T22:14:15.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:15 smithi171 conmon[41853]: debug 2022-01-31T22:14:15.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:15.395476+0000) 2022-01-31T22:14:15.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:15 smithi167 conmon[54076]: debug 2022-01-31T22:14:15.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:15.692309+0000) 2022-01-31T22:14:15.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:15 smithi167 conmon[60316]: debug 2022-01-31T22:14:15.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:15.692309+0000) 2022-01-31T22:14:16.056 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:15 smithi167 conmon[49112]: debug 2022-01-31T22:14:15.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:15.771551+0000) 2022-01-31T22:14:16.298 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[41853]: debug 2022-01-31T22:14:16.277+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.278917+0000) 2022-01-31T22:14:16.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[46715]: debug 2022-01-31T22:14:16.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.189086+0000) 2022-01-31T22:14:16.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[46715]: debug 2022-01-31T22:14:16.275+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.277518+0000) 2022-01-31T22:14:16.300 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[51620]: debug 2022-01-31T22:14:16.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.045026+0000) 2022-01-31T22:14:16.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[51620]: debug 2022-01-31T22:14:16.275+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.277451+0000) 2022-01-31T22:14:16.322 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:16 smithi167 conmon[49112]: debug 2022-01-31T22:14:16.278+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.279094+0000) 2022-01-31T22:14:16.323 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:16 smithi167 conmon[54076]: debug 2022-01-31T22:14:16.277+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.278005+0000) 2022-01-31T22:14:16.324 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:16 smithi167 conmon[60316]: debug 2022-01-31T22:14:16.277+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.277785+0000) 2022-01-31T22:14:16.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[35325]: debug 2022-01-31T22:14:16.296+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193412 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:16.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:16 smithi171 conmon[41853]: debug 2022-01-31T22:14:16.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.395630+0000) 2022-01-31T22:14:16.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:16 smithi167 conmon[54076]: debug 2022-01-31T22:14:16.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.692537+0000) 2022-01-31T22:14:16.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:16 smithi167 conmon[60316]: debug 2022-01-31T22:14:16.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.692537+0000) 2022-01-31T22:14:16.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:16 smithi167 conmon[49112]: debug 2022-01-31T22:14:16.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:16.771724+0000) 2022-01-31T22:14:17.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:17 smithi171 conmon[46715]: debug 2022-01-31T22:14:17.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:17.189244+0000) 2022-01-31T22:14:17.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:17 smithi171 conmon[51620]: debug 2022-01-31T22:14:17.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:17.045192+0000) 2022-01-31T22:14:17.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:17 smithi171 conmon[41853]: debug 2022-01-31T22:14:17.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:17.395806+0000) 2022-01-31T22:14:17.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:17 smithi167 conmon[54076]: debug 2022-01-31T22:14:17.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:17.692650+0000) 2022-01-31T22:14:17.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:17 smithi167 conmon[60316]: debug 2022-01-31T22:14:17.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:17.692650+0000) 2022-01-31T22:14:18.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:17 smithi167 conmon[49112]: debug 2022-01-31T22:14:17.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:17.771839+0000) 2022-01-31T22:14:18.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:18 smithi171 conmon[46715]: debug 2022-01-31T22:14:18.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:18.189385+0000) 2022-01-31T22:14:18.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:18 smithi171 conmon[51620]: debug 2022-01-31T22:14:18.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:18.045360+0000) 2022-01-31T22:14:18.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:18 smithi171 conmon[41853]: debug 2022-01-31T22:14:18.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:18.395986+0000) 2022-01-31T22:14:18.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:18 smithi167 conmon[60316]: debug 2022-01-31T22:14:18.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:18.692807+0000) 2022-01-31T22:14:18.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:18 smithi167 conmon[54076]: debug 2022-01-31T22:14:18.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:18.692807+0000) 2022-01-31T22:14:19.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:18 smithi167 conmon[49112]: debug 2022-01-31T22:14:18.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:18.772020+0000) 2022-01-31T22:14:19.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:19 smithi171 conmon[46715]: debug 2022-01-31T22:14:19.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:19.189576+0000) 2022-01-31T22:14:19.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:19 smithi171 conmon[51620]: debug 2022-01-31T22:14:19.043+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:19.045524+0000) 2022-01-31T22:14:19.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:19 smithi171 conmon[41853]: debug 2022-01-31T22:14:19.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:19.396124+0000) 2022-01-31T22:14:19.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:19 smithi167 conmon[54076]: debug 2022-01-31T22:14:19.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:19.693003+0000) 2022-01-31T22:14:19.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:19 smithi167 conmon[60316]: debug 2022-01-31T22:14:19.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:19.693003+0000) 2022-01-31T22:14:20.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:19 smithi167 conmon[49112]: debug 2022-01-31T22:14:19.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:19.772176+0000) 2022-01-31T22:14:20.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:20 smithi171 conmon[46715]: debug 2022-01-31T22:14:20.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:20.189777+0000) 2022-01-31T22:14:20.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:20 smithi171 conmon[51620]: debug 2022-01-31T22:14:20.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:20.045750+0000) 2022-01-31T22:14:20.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:20 smithi171 conmon[41853]: debug 2022-01-31T22:14:20.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:20.396279+0000) 2022-01-31T22:14:20.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:20 smithi167 conmon[54076]: debug 2022-01-31T22:14:20.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:20.693214+0000) 2022-01-31T22:14:20.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:20 smithi167 conmon[60316]: debug 2022-01-31T22:14:20.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:20.693214+0000) 2022-01-31T22:14:21.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:20 smithi167 conmon[49112]: debug 2022-01-31T22:14:20.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:20.772388+0000) 2022-01-31T22:14:21.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[46715]: debug 2022-01-31T22:14:21.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.189937+0000) 2022-01-31T22:14:21.300 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[51620]: debug 2022-01-31T22:14:21.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.045974+0000) 2022-01-31T22:14:21.339 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:21 smithi167 conmon[49112]: debug 2022-01-31T22:14:21.300+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.301852+0000) 2022-01-31T22:14:21.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:21 smithi167 conmon[54076]: debug 2022-01-31T22:14:21.300+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.301972+0000) 2022-01-31T22:14:21.341 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:21 smithi167 conmon[60316]: debug 2022-01-31T22:14:21.298+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.300178+0000) 2022-01-31T22:14:21.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[35325]: debug 2022-01-31T22:14:21.319+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193523 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:21.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[41853]: debug 2022-01-31T22:14:21.299+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.301278+0000) 2022-01-31T22:14:21.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[41853]: debug 2022-01-31T22:14:21.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.396451+0000) 2022-01-31T22:14:21.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[46715]: debug 2022-01-31T22:14:21.298+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.300540+0000) 2022-01-31T22:14:21.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:21 smithi171 conmon[51620]: debug 2022-01-31T22:14:21.299+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.300787+0000) 2022-01-31T22:14:21.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:21 smithi167 conmon[60316]: debug 2022-01-31T22:14:21.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.693447+0000) 2022-01-31T22:14:21.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:21 smithi167 conmon[54076]: debug 2022-01-31T22:14:21.691+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.693445+0000) 2022-01-31T22:14:22.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:21 smithi167 conmon[49112]: debug 2022-01-31T22:14:21.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:21.772603+0000) 2022-01-31T22:14:22.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:22 smithi171 conmon[46715]: debug 2022-01-31T22:14:22.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:22.190117+0000) 2022-01-31T22:14:22.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:22 smithi171 conmon[51620]: debug 2022-01-31T22:14:22.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:22.046176+0000) 2022-01-31T22:14:22.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:22 smithi171 conmon[41853]: debug 2022-01-31T22:14:22.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:22.396621+0000) 2022-01-31T22:14:22.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:22 smithi167 conmon[54076]: debug 2022-01-31T22:14:22.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:22.693552+0000) 2022-01-31T22:14:22.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:22 smithi167 conmon[60316]: debug 2022-01-31T22:14:22.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:22.693552+0000) 2022-01-31T22:14:23.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:22 smithi167 conmon[49112]: debug 2022-01-31T22:14:22.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:22.772766+0000) 2022-01-31T22:14:23.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:23 smithi171 conmon[46715]: debug 2022-01-31T22:14:23.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:23.190243+0000) 2022-01-31T22:14:23.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:23 smithi171 conmon[51620]: debug 2022-01-31T22:14:23.044+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:23.046365+0000) 2022-01-31T22:14:23.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:23 smithi171 conmon[41853]: debug 2022-01-31T22:14:23.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:23.396796+0000) 2022-01-31T22:14:23.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:23 smithi167 conmon[54076]: debug 2022-01-31T22:14:23.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:23.693724+0000) 2022-01-31T22:14:23.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:23 smithi167 conmon[60316]: debug 2022-01-31T22:14:23.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:23.693724+0000) 2022-01-31T22:14:24.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:23 smithi167 conmon[49112]: debug 2022-01-31T22:14:23.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:23.772918+0000) 2022-01-31T22:14:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:23 smithi171 conmon[35325]: debug 2022-01-31T22:14:23.818+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:14:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:24 smithi171 conmon[51620]: debug 2022-01-31T22:14:24.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:24.046555+0000) 2022-01-31T22:14:24.396 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:24 smithi171 conmon[46715]: debug 2022-01-31T22:14:24.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:24.190374+0000) 2022-01-31T22:14:24.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:24 smithi167 conmon[54076]: debug 2022-01-31T22:14:24.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:24.693944+0000) 2022-01-31T22:14:24.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:24 smithi167 conmon[60316]: debug 2022-01-31T22:14:24.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:24.693944+0000) 2022-01-31T22:14:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:24 smithi171 conmon[41853]: debug 2022-01-31T22:14:24.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:24.396961+0000) 2022-01-31T22:14:25.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:24 smithi167 conmon[49112]: debug 2022-01-31T22:14:24.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:24.773080+0000) 2022-01-31T22:14:25.354 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:25 smithi171 conmon[46715]: debug 2022-01-31T22:14:25.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:25.190502+0000) 2022-01-31T22:14:25.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:25 smithi171 conmon[51620]: debug 2022-01-31T22:14:25.045+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:25.046749+0000) 2022-01-31T22:14:25.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:25 smithi171 conmon[41853]: debug 2022-01-31T22:14:25.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:25.397116+0000) 2022-01-31T22:14:25.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:25 smithi167 conmon[54076]: debug 2022-01-31T22:14:25.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:25.694126+0000) 2022-01-31T22:14:25.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:25 smithi167 conmon[60316]: debug 2022-01-31T22:14:25.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:25.694126+0000) 2022-01-31T22:14:26.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:25 smithi167 conmon[49112]: debug 2022-01-31T22:14:25.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:25.773264+0000) 2022-01-31T22:14:26.323 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[51620]: debug 2022-01-31T22:14:26.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.046932+0000) 2022-01-31T22:14:26.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[46715]: debug 2022-01-31T22:14:26.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.190668+0000) 2022-01-31T22:14:26.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:26 smithi167 conmon[49112]: debug 2022-01-31T22:14:26.322+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.324477+0000) 2022-01-31T22:14:26.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:26 smithi167 conmon[54076]: debug 2022-01-31T22:14:26.322+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.324197+0000) 2022-01-31T22:14:26.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:26 smithi167 conmon[60316]: debug 2022-01-31T22:14:26.322+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.324292+0000) 2022-01-31T22:14:26.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[35325]: debug 2022-01-31T22:14:26.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193637 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:26.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[46715]: debug 2022-01-31T22:14:26.323+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.324465+0000) 2022-01-31T22:14:26.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[51620]: debug 2022-01-31T22:14:26.322+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.323679+0000) 2022-01-31T22:14:26.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[41853]: debug 2022-01-31T22:14:26.323+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.323777+0000) 2022-01-31T22:14:26.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:26 smithi171 conmon[41853]: debug 2022-01-31T22:14:26.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.397298+0000) 2022-01-31T22:14:26.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:26 smithi167 conmon[54076]: debug 2022-01-31T22:14:26.692+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.694285+0000) 2022-01-31T22:14:26.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:26 smithi167 conmon[60316]: debug 2022-01-31T22:14:26.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.694330+0000) 2022-01-31T22:14:27.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:26 smithi167 conmon[49112]: debug 2022-01-31T22:14:26.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:26.773468+0000) 2022-01-31T22:14:27.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:27 smithi171 conmon[46715]: debug 2022-01-31T22:14:27.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:27.190854+0000) 2022-01-31T22:14:27.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:27 smithi171 conmon[51620]: debug 2022-01-31T22:14:27.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:27.047108+0000) 2022-01-31T22:14:27.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:27 smithi171 conmon[41853]: debug 2022-01-31T22:14:27.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:27.397436+0000) 2022-01-31T22:14:27.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:27 smithi167 conmon[54076]: debug 2022-01-31T22:14:27.693+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:27.694481+0000) 2022-01-31T22:14:27.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:27 smithi167 conmon[60316]: debug 2022-01-31T22:14:27.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:27.694481+0000) 2022-01-31T22:14:28.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:27 smithi167 conmon[49112]: debug 2022-01-31T22:14:27.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:27.773617+0000) 2022-01-31T22:14:28.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:28 smithi171 conmon[46715]: debug 2022-01-31T22:14:28.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:28.191000+0000) 2022-01-31T22:14:28.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:28 smithi171 conmon[51620]: debug 2022-01-31T22:14:28.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:28.047269+0000) 2022-01-31T22:14:28.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:28 smithi171 conmon[41853]: debug 2022-01-31T22:14:28.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:28.397641+0000) 2022-01-31T22:14:28.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:28 smithi167 conmon[60316]: debug 2022-01-31T22:14:28.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:28.694632+0000) 2022-01-31T22:14:28.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:28 smithi167 conmon[54076]: debug 2022-01-31T22:14:28.693+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:28.694606+0000) 2022-01-31T22:14:29.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:28 smithi167 conmon[49112]: debug 2022-01-31T22:14:28.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:28.773774+0000) 2022-01-31T22:14:29.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:29 smithi171 conmon[46715]: debug 2022-01-31T22:14:29.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:29.191188+0000) 2022-01-31T22:14:29.401 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:29 smithi171 conmon[51620]: debug 2022-01-31T22:14:29.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:29.047478+0000) 2022-01-31T22:14:29.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:29 smithi171 conmon[41853]: debug 2022-01-31T22:14:29.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:29.397801+0000) 2022-01-31T22:14:29.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:29 smithi167 conmon[54076]: debug 2022-01-31T22:14:29.693+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:29.694765+0000) 2022-01-31T22:14:29.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:29 smithi167 conmon[60316]: debug 2022-01-31T22:14:29.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:29.694767+0000) 2022-01-31T22:14:30.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:29 smithi167 conmon[49112]: debug 2022-01-31T22:14:29.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:29.773937+0000) 2022-01-31T22:14:30.355 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:30 smithi171 conmon[46715]: debug 2022-01-31T22:14:30.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:30.191345+0000) 2022-01-31T22:14:30.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:30 smithi171 conmon[51620]: debug 2022-01-31T22:14:30.046+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:30.047682+0000) 2022-01-31T22:14:30.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:30 smithi171 conmon[41853]: debug 2022-01-31T22:14:30.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:30.397977+0000) 2022-01-31T22:14:30.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:30 smithi167 conmon[54076]: debug 2022-01-31T22:14:30.693+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:30.694931+0000) 2022-01-31T22:14:30.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:30 smithi167 conmon[60316]: debug 2022-01-31T22:14:30.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:30.694938+0000) 2022-01-31T22:14:31.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:30 smithi167 conmon[49112]: debug 2022-01-31T22:14:30.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:30.774070+0000) 2022-01-31T22:14:31.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[46715]: debug 2022-01-31T22:14:31.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.191545+0000) 2022-01-31T22:14:31.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[51620]: debug 2022-01-31T22:14:31.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.047886+0000) 2022-01-31T22:14:31.347 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:31 smithi167 conmon[54076]: debug 2022-01-31T22:14:31.345+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.346730+0000) 2022-01-31T22:14:31.347 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:31 smithi167 conmon[60316]: debug 2022-01-31T22:14:31.344+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.346122+0000) 2022-01-31T22:14:31.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[46715]: debug 2022-01-31T22:14:31.347+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.348447+0000) 2022-01-31T22:14:31.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[51620]: debug 2022-01-31T22:14:31.348+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.348815+0000) 2022-01-31T22:14:31.608 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[35325]: debug 2022-01-31T22:14:31.366+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193747 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:31.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[41853]: debug 2022-01-31T22:14:31.345+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.346048+0000) 2022-01-31T22:14:31.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:31 smithi171 conmon[41853]: debug 2022-01-31T22:14:31.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.398178+0000) 2022-01-31T22:14:31.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:31 smithi167 conmon[49112]: debug 2022-01-31T22:14:31.347+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.349070+0000) 2022-01-31T22:14:32.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:31 smithi167 conmon[49112]: debug 2022-01-31T22:14:31.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.774209+0000) 2022-01-31T22:14:32.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:31 smithi167 conmon[54076]: debug 2022-01-31T22:14:31.694+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.695148+0000) 2022-01-31T22:14:32.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:31 smithi167 conmon[60316]: debug 2022-01-31T22:14:31.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:31.695148+0000) 2022-01-31T22:14:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:32 smithi171 conmon[46715]: debug 2022-01-31T22:14:32.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:32.191644+0000) 2022-01-31T22:14:32.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:32 smithi171 conmon[51620]: debug 2022-01-31T22:14:32.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:32.048050+0000) 2022-01-31T22:14:32.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:32 smithi171 conmon[41853]: debug 2022-01-31T22:14:32.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:32.398291+0000) 2022-01-31T22:14:32.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:32 smithi167 conmon[54076]: debug 2022-01-31T22:14:32.693+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:32.695315+0000) 2022-01-31T22:14:32.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:32 smithi167 conmon[60316]: debug 2022-01-31T22:14:32.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:32.695319+0000) 2022-01-31T22:14:33.101 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:32 smithi167 conmon[49112]: debug 2022-01-31T22:14:32.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:32.774414+0000) 2022-01-31T22:14:33.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:33 smithi171 conmon[46715]: debug 2022-01-31T22:14:33.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:33.191811+0000) 2022-01-31T22:14:33.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:33 smithi171 conmon[51620]: debug 2022-01-31T22:14:33.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:33.048209+0000) 2022-01-31T22:14:33.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:33 smithi171 conmon[41853]: debug 2022-01-31T22:14:33.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:33.398473+0000) 2022-01-31T22:14:33.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:33 smithi167 conmon[54076]: debug 2022-01-31T22:14:33.694+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:33.695550+0000) 2022-01-31T22:14:33.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:33 smithi167 conmon[60316]: debug 2022-01-31T22:14:33.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:33.695550+0000) 2022-01-31T22:14:34.105 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:33 smithi167 conmon[49112]: debug 2022-01-31T22:14:33.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:33.774599+0000) 2022-01-31T22:14:34.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:34 smithi171 conmon[46715]: debug 2022-01-31T22:14:34.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:34.192023+0000) 2022-01-31T22:14:34.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:34 smithi171 conmon[51620]: debug 2022-01-31T22:14:34.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:34.048421+0000) 2022-01-31T22:14:34.606 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:34 smithi171 conmon[41853]: debug 2022-01-31T22:14:34.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:34.398672+0000) 2022-01-31T22:14:34.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:34 smithi167 conmon[54076]: debug 2022-01-31T22:14:34.695+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:34.695762+0000) 2022-01-31T22:14:34.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:34 smithi167 conmon[60316]: debug 2022-01-31T22:14:34.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:34.695762+0000) 2022-01-31T22:14:35.109 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:34 smithi167 conmon[49112]: debug 2022-01-31T22:14:34.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:34.774808+0000) 2022-01-31T22:14:35.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:35 smithi171 conmon[46715]: debug 2022-01-31T22:14:35.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:35.192228+0000) 2022-01-31T22:14:35.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:35 smithi171 conmon[51620]: debug 2022-01-31T22:14:35.047+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:35.048621+0000) 2022-01-31T22:14:35.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:35 smithi171 conmon[41853]: debug 2022-01-31T22:14:35.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:35.398846+0000) 2022-01-31T22:14:35.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:35 smithi167 conmon[54076]: debug 2022-01-31T22:14:35.694+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:35.695966+0000) 2022-01-31T22:14:35.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:35 smithi167 conmon[60316]: debug 2022-01-31T22:14:35.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:35.695966+0000) 2022-01-31T22:14:36.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:35 smithi167 conmon[49112]: debug 2022-01-31T22:14:35.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:35.775049+0000) 2022-01-31T22:14:36.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[46715]: debug 2022-01-31T22:14:36.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.192437+0000) 2022-01-31T22:14:36.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[51620]: debug 2022-01-31T22:14:36.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.048862+0000) 2022-01-31T22:14:36.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[46715]: debug 2022-01-31T22:14:36.368+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.369302+0000) 2022-01-31T22:14:36.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[35325]: debug 2022-01-31T22:14:36.388+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193859 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:36.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[51620]: debug 2022-01-31T22:14:36.370+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.371275+0000) 2022-01-31T22:14:36.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[41853]: debug 2022-01-31T22:14:36.368+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.369608+0000) 2022-01-31T22:14:36.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:36 smithi171 conmon[41853]: debug 2022-01-31T22:14:36.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.398996+0000) 2022-01-31T22:14:36.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:36 smithi167 conmon[49112]: debug 2022-01-31T22:14:36.369+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.370603+0000) 2022-01-31T22:14:36.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:36 smithi167 conmon[54076]: debug 2022-01-31T22:14:36.368+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.369932+0000) 2022-01-31T22:14:36.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:36 smithi167 conmon[60316]: debug 2022-01-31T22:14:36.368+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.369683+0000) 2022-01-31T22:14:37.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:36 smithi167 conmon[49112]: debug 2022-01-31T22:14:36.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.775220+0000) 2022-01-31T22:14:37.113 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:36 smithi167 conmon[54076]: debug 2022-01-31T22:14:36.695+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.696128+0000) 2022-01-31T22:14:37.113 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:36 smithi167 conmon[60316]: debug 2022-01-31T22:14:36.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:36.696128+0000) 2022-01-31T22:14:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:37 smithi171 conmon[46715]: debug 2022-01-31T22:14:37.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:37.192743+0000) 2022-01-31T22:14:37.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:37 smithi171 conmon[51620]: debug 2022-01-31T22:14:37.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:37.049044+0000) 2022-01-31T22:14:37.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:37 smithi171 conmon[41853]: debug 2022-01-31T22:14:37.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:37.399154+0000) 2022-01-31T22:14:37.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:37 smithi167 conmon[60316]: debug 2022-01-31T22:14:37.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:37.696308+0000) 2022-01-31T22:14:37.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:37 smithi167 conmon[54076]: debug 2022-01-31T22:14:37.695+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:37.696307+0000) 2022-01-31T22:14:38.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:37 smithi167 conmon[49112]: debug 2022-01-31T22:14:37.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:37.775397+0000) 2022-01-31T22:14:38.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:38 smithi171 conmon[46715]: debug 2022-01-31T22:14:38.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:38.192894+0000) 2022-01-31T22:14:38.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:38 smithi171 conmon[51620]: debug 2022-01-31T22:14:38.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:38.049236+0000) 2022-01-31T22:14:38.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:38 smithi167 conmon[54076]: debug 2022-01-31T22:14:38.695+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:38.696491+0000) 2022-01-31T22:14:38.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:38 smithi167 conmon[60316]: debug 2022-01-31T22:14:38.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:38.696491+0000) 2022-01-31T22:14:38.819 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:38 smithi171 conmon[41853]: debug 2022-01-31T22:14:38.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:38.399343+0000) 2022-01-31T22:14:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:38 smithi171 conmon[35325]: debug 2022-01-31T22:14:38.819+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:14:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:39 smithi171 conmon[51620]: debug 2022-01-31T22:14:39.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:39.049392+0000) 2022-01-31T22:14:39.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:38 smithi167 conmon[49112]: debug 2022-01-31T22:14:38.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:38.775611+0000) 2022-01-31T22:14:39.399 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:39 smithi171 conmon[46715]: debug 2022-01-31T22:14:39.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:39.193052+0000) 2022-01-31T22:14:39.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:39 smithi167 conmon[54076]: debug 2022-01-31T22:14:39.696+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:39.696696+0000) 2022-01-31T22:14:39.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:39 smithi167 conmon[60316]: debug 2022-01-31T22:14:39.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:39.696694+0000) 2022-01-31T22:14:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:39 smithi171 conmon[41853]: debug 2022-01-31T22:14:39.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:39.399548+0000) 2022-01-31T22:14:40.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:39 smithi167 conmon[49112]: debug 2022-01-31T22:14:39.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:39.775804+0000) 2022-01-31T22:14:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:40 smithi171 conmon[46715]: debug 2022-01-31T22:14:40.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:40.193242+0000) 2022-01-31T22:14:40.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:40 smithi171 conmon[51620]: debug 2022-01-31T22:14:40.048+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:40.049567+0000) 2022-01-31T22:14:40.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:40 smithi167 conmon[54076]: debug 2022-01-31T22:14:40.696+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:40.696886+0000) 2022-01-31T22:14:40.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:40 smithi167 conmon[60316]: debug 2022-01-31T22:14:40.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:40.696886+0000) 2022-01-31T22:14:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:40 smithi171 conmon[41853]: debug 2022-01-31T22:14:40.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:40.399770+0000) 2022-01-31T22:14:41.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:40 smithi167 conmon[49112]: debug 2022-01-31T22:14:40.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:40.775991+0000) 2022-01-31T22:14:41.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[51620]: debug 2022-01-31T22:14:41.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.049744+0000) 2022-01-31T22:14:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[46715]: debug 2022-01-31T22:14:41.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.193444+0000) 2022-01-31T22:14:41.669 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:41 smithi167 conmon[49112]: debug 2022-01-31T22:14:41.391+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.392726+0000) 2022-01-31T22:14:41.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:41 smithi167 conmon[54076]: debug 2022-01-31T22:14:41.390+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.392440+0000) 2022-01-31T22:14:41.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:41 smithi167 conmon[60316]: debug 2022-01-31T22:14:41.391+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.393124+0000) 2022-01-31T22:14:41.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[35325]: debug 2022-01-31T22:14:41.412+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 193969 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:41.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[46715]: debug 2022-01-31T22:14:41.391+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.392599+0000) 2022-01-31T22:14:41.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[51620]: debug 2022-01-31T22:14:41.392+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.393421+0000) 2022-01-31T22:14:41.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[41853]: debug 2022-01-31T22:14:41.390+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.391574+0000) 2022-01-31T22:14:41.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:41 smithi171 conmon[41853]: debug 2022-01-31T22:14:41.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.399882+0000) 2022-01-31T22:14:42.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:41 smithi167 conmon[54076]: debug 2022-01-31T22:14:41.695+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.697070+0000) 2022-01-31T22:14:42.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:41 smithi167 conmon[49112]: debug 2022-01-31T22:14:41.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.776176+0000) 2022-01-31T22:14:42.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:41 smithi167 conmon[60316]: debug 2022-01-31T22:14:41.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:41.697070+0000) 2022-01-31T22:14:42.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:42 smithi171 conmon[51620]: debug 2022-01-31T22:14:42.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:42.049883+0000) 2022-01-31T22:14:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:42 smithi171 conmon[46715]: debug 2022-01-31T22:14:42.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:42.193644+0000) 2022-01-31T22:14:42.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:42 smithi167 conmon[60316]: debug 2022-01-31T22:14:42.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:42.697240+0000) 2022-01-31T22:14:42.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:42 smithi167 conmon[54076]: debug 2022-01-31T22:14:42.696+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:42.697240+0000) 2022-01-31T22:14:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:42 smithi171 conmon[41853]: debug 2022-01-31T22:14:42.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:42.400073+0000) 2022-01-31T22:14:43.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:42 smithi167 conmon[49112]: debug 2022-01-31T22:14:42.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:42.776369+0000) 2022-01-31T22:14:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:43 smithi171 conmon[46715]: debug 2022-01-31T22:14:43.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:43.193794+0000) 2022-01-31T22:14:43.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:43 smithi171 conmon[51620]: debug 2022-01-31T22:14:43.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:43.050047+0000) 2022-01-31T22:14:43.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:43 smithi167 conmon[54076]: debug 2022-01-31T22:14:43.696+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:43.697464+0000) 2022-01-31T22:14:43.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:43 smithi167 conmon[60316]: debug 2022-01-31T22:14:43.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:43.697464+0000) 2022-01-31T22:14:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:43 smithi171 conmon[41853]: debug 2022-01-31T22:14:43.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:43.400289+0000) 2022-01-31T22:14:44.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:43 smithi167 conmon[49112]: debug 2022-01-31T22:14:43.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:43.776561+0000) 2022-01-31T22:14:44.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:44 smithi171 conmon[46715]: debug 2022-01-31T22:14:44.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:44.194003+0000) 2022-01-31T22:14:44.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:44 smithi171 conmon[51620]: debug 2022-01-31T22:14:44.049+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:44.050270+0000) 2022-01-31T22:14:44.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:44 smithi167 conmon[54076]: debug 2022-01-31T22:14:44.697+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:44.697727+0000) 2022-01-31T22:14:44.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:44 smithi167 conmon[60316]: debug 2022-01-31T22:14:44.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:44.697727+0000) 2022-01-31T22:14:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:44 smithi171 conmon[41853]: debug 2022-01-31T22:14:44.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:44.400526+0000) 2022-01-31T22:14:45.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:44 smithi167 conmon[49112]: debug 2022-01-31T22:14:44.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:44.776723+0000) 2022-01-31T22:14:45.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:45 smithi171 conmon[46715]: debug 2022-01-31T22:14:45.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:45.194153+0000) 2022-01-31T22:14:45.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:45 smithi171 conmon[51620]: debug 2022-01-31T22:14:45.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:45.050450+0000) 2022-01-31T22:14:45.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:45 smithi167 conmon[54076]: debug 2022-01-31T22:14:45.697+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:45.697941+0000) 2022-01-31T22:14:45.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:45 smithi167 conmon[60316]: debug 2022-01-31T22:14:45.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:45.697941+0000) 2022-01-31T22:14:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:45 smithi171 conmon[41853]: debug 2022-01-31T22:14:45.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:45.400658+0000) 2022-01-31T22:14:46.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:45 smithi167 conmon[49112]: debug 2022-01-31T22:14:45.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:45.776898+0000) 2022-01-31T22:14:46.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[46715]: debug 2022-01-31T22:14:46.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.194266+0000) 2022-01-31T22:14:46.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[51620]: debug 2022-01-31T22:14:46.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.050641+0000) 2022-01-31T22:14:46.669 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:46 smithi167 conmon[49112]: debug 2022-01-31T22:14:46.415+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.415520+0000) 2022-01-31T22:14:46.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:46 smithi167 conmon[54076]: debug 2022-01-31T22:14:46.414+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.415284+0000) 2022-01-31T22:14:46.670 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:46 smithi167 conmon[60316]: debug 2022-01-31T22:14:46.415+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.415843+0000) 2022-01-31T22:14:46.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[35325]: debug 2022-01-31T22:14:46.450+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194081 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:46.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[46715]: debug 2022-01-31T22:14:46.415+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.416230+0000) 2022-01-31T22:14:46.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[51620]: debug 2022-01-31T22:14:46.415+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.416142+0000) 2022-01-31T22:14:46.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[41853]: debug 2022-01-31T22:14:46.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.400790+0000) 2022-01-31T22:14:46.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:46 smithi171 conmon[41853]: debug 2022-01-31T22:14:46.413+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.414473+0000) 2022-01-31T22:14:47.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:46 smithi167 conmon[49112]: debug 2022-01-31T22:14:46.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.777080+0000) 2022-01-31T22:14:47.140 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:46 smithi167 conmon[54076]: debug 2022-01-31T22:14:46.697+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.698153+0000) 2022-01-31T22:14:47.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:46 smithi167 conmon[60316]: debug 2022-01-31T22:14:46.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:46.698153+0000) 2022-01-31T22:14:47.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:47 smithi171 conmon[46715]: debug 2022-01-31T22:14:47.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:47.194448+0000) 2022-01-31T22:14:47.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:47 smithi171 conmon[51620]: debug 2022-01-31T22:14:47.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:47.050841+0000) 2022-01-31T22:14:47.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:47 smithi167 conmon[54076]: debug 2022-01-31T22:14:47.697+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:47.698345+0000) 2022-01-31T22:14:47.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:47 smithi167 conmon[60316]: debug 2022-01-31T22:14:47.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:47.698345+0000) 2022-01-31T22:14:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:47 smithi171 conmon[41853]: debug 2022-01-31T22:14:47.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:47.400953+0000) 2022-01-31T22:14:48.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:47 smithi167 conmon[49112]: debug 2022-01-31T22:14:47.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:47.777247+0000) 2022-01-31T22:14:48.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:48 smithi171 conmon[46715]: debug 2022-01-31T22:14:48.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:48.194559+0000) 2022-01-31T22:14:48.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:48 smithi171 conmon[51620]: debug 2022-01-31T22:14:48.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:48.050991+0000) 2022-01-31T22:14:48.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:48 smithi167 conmon[54076]: debug 2022-01-31T22:14:48.698+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:48.698532+0000) 2022-01-31T22:14:48.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:48 smithi167 conmon[60316]: debug 2022-01-31T22:14:48.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:48.698538+0000) 2022-01-31T22:14:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:48 smithi171 conmon[41853]: debug 2022-01-31T22:14:48.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:48.401104+0000) 2022-01-31T22:14:49.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:48 smithi167 conmon[49112]: debug 2022-01-31T22:14:48.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:48.777434+0000) 2022-01-31T22:14:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:49 smithi171 conmon[46715]: debug 2022-01-31T22:14:49.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:49.194742+0000) 2022-01-31T22:14:49.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:49 smithi171 conmon[51620]: debug 2022-01-31T22:14:49.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:49.051210+0000) 2022-01-31T22:14:49.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:49 smithi167 conmon[54076]: debug 2022-01-31T22:14:49.698+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:49.698748+0000) 2022-01-31T22:14:49.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:49 smithi167 conmon[60316]: debug 2022-01-31T22:14:49.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:49.698748+0000) 2022-01-31T22:14:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:49 smithi171 conmon[41853]: debug 2022-01-31T22:14:49.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:49.401278+0000) 2022-01-31T22:14:50.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:49 smithi167 conmon[49112]: debug 2022-01-31T22:14:49.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:49.777631+0000) 2022-01-31T22:14:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:50 smithi171 conmon[46715]: debug 2022-01-31T22:14:50.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:50.194880+0000) 2022-01-31T22:14:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:50 smithi171 conmon[51620]: debug 2022-01-31T22:14:50.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:50.051336+0000) 2022-01-31T22:14:50.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:50 smithi167 conmon[54076]: debug 2022-01-31T22:14:50.698+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:50.698974+0000) 2022-01-31T22:14:50.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:50 smithi167 conmon[60316]: debug 2022-01-31T22:14:50.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:50.698973+0000) 2022-01-31T22:14:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:50 smithi171 conmon[41853]: debug 2022-01-31T22:14:50.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:50.401493+0000) 2022-01-31T22:14:51.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:50 smithi167 conmon[49112]: debug 2022-01-31T22:14:50.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:50.777805+0000) 2022-01-31T22:14:51.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[46715]: debug 2022-01-31T22:14:51.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.195015+0000) 2022-01-31T22:14:51.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[51620]: debug 2022-01-31T22:14:51.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.051490+0000) 2022-01-31T22:14:51.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:51 smithi167 conmon[49112]: debug 2022-01-31T22:14:51.453+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.453773+0000) 2022-01-31T22:14:51.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:51 smithi167 conmon[54076]: debug 2022-01-31T22:14:51.452+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.452921+0000) 2022-01-31T22:14:51.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:51 smithi167 conmon[54076]: debug 2022-01-31T22:14:51.698+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.699164+0000) 2022-01-31T22:14:51.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:51 smithi167 conmon[60316]: debug 2022-01-31T22:14:51.453+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.454070+0000) 2022-01-31T22:14:51.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:51 smithi167 conmon[60316]: debug 2022-01-31T22:14:51.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.699164+0000) 2022-01-31T22:14:51.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[35325]: debug 2022-01-31T22:14:51.473+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:51.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[46715]: debug 2022-01-31T22:14:51.453+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.454675+0000) 2022-01-31T22:14:51.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[51620]: debug 2022-01-31T22:14:51.454+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.454929+0000) 2022-01-31T22:14:51.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[41853]: debug 2022-01-31T22:14:51.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.401666+0000) 2022-01-31T22:14:51.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:51 smithi171 conmon[41853]: debug 2022-01-31T22:14:51.452+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.453319+0000) 2022-01-31T22:14:52.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:51 smithi167 conmon[49112]: debug 2022-01-31T22:14:51.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:51.777939+0000) 2022-01-31T22:14:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:52 smithi171 conmon[46715]: debug 2022-01-31T22:14:52.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:52.195219+0000) 2022-01-31T22:14:52.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:52 smithi171 conmon[51620]: debug 2022-01-31T22:14:52.050+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:52.051683+0000) 2022-01-31T22:14:52.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:52 smithi167 conmon[54076]: debug 2022-01-31T22:14:52.699+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:52.699311+0000) 2022-01-31T22:14:52.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:52 smithi167 conmon[60316]: debug 2022-01-31T22:14:52.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:52.699311+0000) 2022-01-31T22:14:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:52 smithi171 conmon[41853]: debug 2022-01-31T22:14:52.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:52.401850+0000) 2022-01-31T22:14:53.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:52 smithi167 conmon[49112]: debug 2022-01-31T22:14:52.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:52.778144+0000) 2022-01-31T22:14:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:53 smithi171 conmon[46715]: debug 2022-01-31T22:14:53.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:53.195298+0000) 2022-01-31T22:14:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:53 smithi171 conmon[51620]: debug 2022-01-31T22:14:53.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:53.051869+0000) 2022-01-31T22:14:53.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:53 smithi167 conmon[54076]: debug 2022-01-31T22:14:53.699+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:53.699527+0000) 2022-01-31T22:14:53.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:53 smithi167 conmon[60316]: debug 2022-01-31T22:14:53.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:53.699527+0000) 2022-01-31T22:14:53.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:53 smithi171 conmon[41853]: debug 2022-01-31T22:14:53.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:53.402025+0000) 2022-01-31T22:14:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:53 smithi171 conmon[35325]: debug 2022-01-31T22:14:53.820+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:14:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:54 smithi171 conmon[51620]: debug 2022-01-31T22:14:54.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:54.052086+0000) 2022-01-31T22:14:54.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:53 smithi167 conmon[49112]: debug 2022-01-31T22:14:53.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:53.778288+0000) 2022-01-31T22:14:54.401 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:54 smithi171 conmon[46715]: debug 2022-01-31T22:14:54.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:54.195415+0000) 2022-01-31T22:14:54.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:54 smithi167 conmon[54076]: debug 2022-01-31T22:14:54.699+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:54.699749+0000) 2022-01-31T22:14:54.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:54 smithi167 conmon[60316]: debug 2022-01-31T22:14:54.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:54.699751+0000) 2022-01-31T22:14:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:54 smithi171 conmon[41853]: debug 2022-01-31T22:14:54.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:54.402185+0000) 2022-01-31T22:14:55.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:54 smithi167 conmon[49112]: debug 2022-01-31T22:14:54.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:54.778460+0000) 2022-01-31T22:14:55.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:55 smithi171 conmon[46715]: debug 2022-01-31T22:14:55.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:55.195610+0000) 2022-01-31T22:14:55.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:55 smithi171 conmon[51620]: debug 2022-01-31T22:14:55.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:55.052308+0000) 2022-01-31T22:14:55.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:55 smithi167 conmon[54076]: debug 2022-01-31T22:14:55.699+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:55.699994+0000) 2022-01-31T22:14:55.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:55 smithi167 conmon[60316]: debug 2022-01-31T22:14:55.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:55.699994+0000) 2022-01-31T22:14:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:55 smithi171 conmon[41853]: debug 2022-01-31T22:14:55.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:55.402387+0000) 2022-01-31T22:14:56.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:55 smithi167 conmon[49112]: debug 2022-01-31T22:14:55.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:55.778541+0000) 2022-01-31T22:14:56.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[51620]: debug 2022-01-31T22:14:56.051+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.052473+0000) 2022-01-31T22:14:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[46715]: debug 2022-01-31T22:14:56.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.195781+0000) 2022-01-31T22:14:56.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[49112]: debug 2022-01-31T22:14:56.475+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.476158+0000) 2022-01-31T22:14:56.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[49112]: 2022-01-31T22:14:56.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[54076]: debug 2022-01-31T22:14:56.476+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.476577+0000) 2022-01-31T22:14:56.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[54076]: debug 2022-01-31T22:14:56.699+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.700194+0000) 2022-01-31T22:14:56.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[60316]: debug 2022-01-31T22:14:56.476+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.476682+0000) 2022-01-31T22:14:56.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[60316]: debug 2022-01-31T22:14:56.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.700194+0000) 2022-01-31T22:14:56.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[35325]: debug 2022-01-31T22:14:56.497+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194304 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:14:56.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[46715]: debug 2022-01-31T22:14:56.475+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.475975+0000) 2022-01-31T22:14:56.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[51620]: debug 2022-01-31T22:14:56.477+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.477794+0000) 2022-01-31T22:14:56.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[41853]: debug 2022-01-31T22:14:56.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.402601+0000) 2022-01-31T22:14:56.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:56 smithi171 conmon[41853]: debug 2022-01-31T22:14:56.476+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.477245+0000) 2022-01-31T22:14:57.166 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:56 smithi167 conmon[49112]: debug 2022-01-31T22:14:56.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:56.778691+0000) 2022-01-31T22:14:57.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:57 smithi171 conmon[51620]: debug 2022-01-31T22:14:57.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:57.052660+0000) 2022-01-31T22:14:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:57 smithi171 conmon[46715]: debug 2022-01-31T22:14:57.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:57.195976+0000) 2022-01-31T22:14:57.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:57 smithi167 conmon[54076]: debug 2022-01-31T22:14:57.700+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:57.700372+0000) 2022-01-31T22:14:57.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:57 smithi167 conmon[60316]: debug 2022-01-31T22:14:57.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:57.700372+0000) 2022-01-31T22:14:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:57 smithi171 conmon[41853]: debug 2022-01-31T22:14:57.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:57.402819+0000) 2022-01-31T22:14:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:57 smithi167 conmon[49112]: debug 2022-01-31T22:14:57.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:57.778846+0000) 2022-01-31T22:14:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:58 smithi171 conmon[46715]: debug 2022-01-31T22:14:58.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:58.196139+0000) 2022-01-31T22:14:58.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:58 smithi171 conmon[51620]: debug 2022-01-31T22:14:58.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:58.052824+0000) 2022-01-31T22:14:58.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:58 smithi167 conmon[54076]: debug 2022-01-31T22:14:58.700+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:58.700559+0000) 2022-01-31T22:14:58.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:58 smithi167 conmon[60316]: debug 2022-01-31T22:14:58.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:58.700562+0000) 2022-01-31T22:14:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:58 smithi171 conmon[41853]: debug 2022-01-31T22:14:58.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:58.402955+0000) 2022-01-31T22:14:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:58 smithi167 conmon[49112]: debug 2022-01-31T22:14:58.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:58.778978+0000) 2022-01-31T22:14:59.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:14:59 smithi171 conmon[46715]: debug 2022-01-31T22:14:59.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:59.196293+0000) 2022-01-31T22:14:59.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:14:59 smithi171 conmon[51620]: debug 2022-01-31T22:14:59.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:59.053031+0000) 2022-01-31T22:14:59.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:14:59 smithi167 conmon[54076]: debug 2022-01-31T22:14:59.700+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:59.700760+0000) 2022-01-31T22:14:59.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:14:59 smithi167 conmon[60316]: debug 2022-01-31T22:14:59.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:59.700760+0000) 2022-01-31T22:14:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:14:59 smithi171 conmon[41853]: debug 2022-01-31T22:14:59.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:59.403136+0000) 2022-01-31T22:15:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:14:59 smithi167 conmon[49112]: debug 2022-01-31T22:14:59.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:14:59.779136+0000) 2022-01-31T22:15:00.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:00 smithi171 conmon[46715]: debug 2022-01-31T22:15:00.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:00.196444+0000) 2022-01-31T22:15:00.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:00 smithi171 conmon[51620]: debug 2022-01-31T22:15:00.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:00.053255+0000) 2022-01-31T22:15:00.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:00 smithi167 conmon[54076]: debug 2022-01-31T22:15:00.700+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:00.700946+0000) 2022-01-31T22:15:00.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:00 smithi167 conmon[60316]: debug 2022-01-31T22:15:00.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:00.700946+0000) 2022-01-31T22:15:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:00 smithi171 conmon[41853]: debug 2022-01-31T22:15:00.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:00.403325+0000) 2022-01-31T22:15:01.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:00 smithi167 conmon[49112]: debug 2022-01-31T22:15:00.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:00.779322+0000) 2022-01-31T22:15:01.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[46715]: debug 2022-01-31T22:15:01.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.196600+0000) 2022-01-31T22:15:01.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[51620]: debug 2022-01-31T22:15:01.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.053438+0000) 2022-01-31T22:15:01.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:01 smithi167 conmon[49112]: debug 2022-01-31T22:15:01.500+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.500965+0000) 2022-01-31T22:15:01.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:01 smithi167 conmon[54076]: debug 2022-01-31T22:15:01.501+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.501878+0000) 2022-01-31T22:15:01.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:01 smithi167 conmon[54076]: debug 2022-01-31T22:15:01.700+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.701162+0000) 2022-01-31T22:15:01.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:01 smithi167 conmon[60316]: debug 2022-01-31T22:15:01.500+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.501396+0000) 2022-01-31T22:15:01.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:01 smithi167 conmon[60316]: debug 2022-01-31T22:15:01.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.701162+0000) 2022-01-31T22:15:01.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[35325]: debug 2022-01-31T22:15:01.519+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194414 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:01.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[46715]: debug 2022-01-31T22:15:01.498+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.500726+0000) 2022-01-31T22:15:01.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[51620]: debug 2022-01-31T22:15:01.500+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.502533+0000) 2022-01-31T22:15:01.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[41853]: debug 2022-01-31T22:15:01.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.403507+0000) 2022-01-31T22:15:01.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:01 smithi171 conmon[41853]: debug 2022-01-31T22:15:01.500+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.502243+0000) 2022-01-31T22:15:02.171 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:01 smithi167 conmon[49112]: debug 2022-01-31T22:15:01.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:01.779542+0000) 2022-01-31T22:15:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:02 smithi171 conmon[46715]: debug 2022-01-31T22:15:02.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:02.196773+0000) 2022-01-31T22:15:02.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:02 smithi171 conmon[51620]: debug 2022-01-31T22:15:02.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:02.053637+0000) 2022-01-31T22:15:02.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:02 smithi167 conmon[54076]: debug 2022-01-31T22:15:02.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:02.701333+0000) 2022-01-31T22:15:02.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:02 smithi167 conmon[60316]: debug 2022-01-31T22:15:02.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:02.701332+0000) 2022-01-31T22:15:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:02 smithi171 conmon[41853]: debug 2022-01-31T22:15:02.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:02.403697+0000) 2022-01-31T22:15:03.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:02 smithi167 conmon[49112]: debug 2022-01-31T22:15:02.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:02.779683+0000) 2022-01-31T22:15:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:03 smithi171 conmon[46715]: debug 2022-01-31T22:15:03.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:03.196925+0000) 2022-01-31T22:15:03.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:03 smithi171 conmon[51620]: debug 2022-01-31T22:15:03.052+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:03.053809+0000) 2022-01-31T22:15:03.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:03 smithi167 conmon[60316]: debug 2022-01-31T22:15:03.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:03.701532+0000) 2022-01-31T22:15:03.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:03 smithi167 conmon[54076]: debug 2022-01-31T22:15:03.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:03.701532+0000) 2022-01-31T22:15:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:03 smithi171 conmon[41853]: debug 2022-01-31T22:15:03.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:03.403873+0000) 2022-01-31T22:15:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:03 smithi167 conmon[49112]: debug 2022-01-31T22:15:03.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:03.779884+0000) 2022-01-31T22:15:04.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:04 smithi171 conmon[46715]: debug 2022-01-31T22:15:04.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:04.197110+0000) 2022-01-31T22:15:04.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:04 smithi171 conmon[51620]: debug 2022-01-31T22:15:04.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:04.054012+0000) 2022-01-31T22:15:04.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:04 smithi167 conmon[54076]: debug 2022-01-31T22:15:04.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:04.701732+0000) 2022-01-31T22:15:04.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:04 smithi167 conmon[60316]: debug 2022-01-31T22:15:04.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:04.701732+0000) 2022-01-31T22:15:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:04 smithi171 conmon[41853]: debug 2022-01-31T22:15:04.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:04.404060+0000) 2022-01-31T22:15:05.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:04 smithi167 conmon[49112]: debug 2022-01-31T22:15:04.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:04.780069+0000) 2022-01-31T22:15:05.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:05 smithi171 conmon[46715]: debug 2022-01-31T22:15:05.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:05.197253+0000) 2022-01-31T22:15:05.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:05 smithi171 conmon[51620]: debug 2022-01-31T22:15:05.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:05.054195+0000) 2022-01-31T22:15:05.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:05 smithi167 conmon[54076]: debug 2022-01-31T22:15:05.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:05.701948+0000) 2022-01-31T22:15:05.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:05 smithi167 conmon[60316]: debug 2022-01-31T22:15:05.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:05.701947+0000) 2022-01-31T22:15:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:05 smithi171 conmon[41853]: debug 2022-01-31T22:15:05.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:05.404234+0000) 2022-01-31T22:15:06.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:05 smithi167 conmon[49112]: debug 2022-01-31T22:15:05.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:05.780180+0000) 2022-01-31T22:15:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[46715]: debug 2022-01-31T22:15:06.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.197413+0000) 2022-01-31T22:15:06.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[51620]: debug 2022-01-31T22:15:06.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.054377+0000) 2022-01-31T22:15:06.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:06 smithi167 conmon[49112]: debug 2022-01-31T22:15:06.522+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.524154+0000) 2022-01-31T22:15:06.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:06 smithi167 conmon[54076]: debug 2022-01-31T22:15:06.522+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.523872+0000) 2022-01-31T22:15:06.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:06 smithi167 conmon[54076]: debug 2022-01-31T22:15:06.700+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.702165+0000) 2022-01-31T22:15:06.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:06 smithi167 conmon[60316]: debug 2022-01-31T22:15:06.522+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.523778+0000) 2022-01-31T22:15:06.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:06 smithi167 conmon[60316]: debug 2022-01-31T22:15:06.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.702165+0000) 2022-01-31T22:15:06.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[35325]: debug 2022-01-31T22:15:06.542+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194526 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[41853]: debug 2022-01-31T22:15:06.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.404391+0000) 2022-01-31T22:15:06.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[41853]: debug 2022-01-31T22:15:06.522+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.524406+0000) 2022-01-31T22:15:06.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[46715]: debug 2022-01-31T22:15:06.521+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.522822+0000) 2022-01-31T22:15:06.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:06 smithi171 conmon[51620]: debug 2022-01-31T22:15:06.523+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.524877+0000) 2022-01-31T22:15:07.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:06 smithi167 conmon[49112]: debug 2022-01-31T22:15:06.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:06.780368+0000) 2022-01-31T22:15:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:07 smithi171 conmon[46715]: debug 2022-01-31T22:15:07.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:07.197573+0000) 2022-01-31T22:15:07.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:07 smithi171 conmon[51620]: debug 2022-01-31T22:15:07.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:07.054564+0000) 2022-01-31T22:15:07.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:07 smithi167 conmon[54076]: debug 2022-01-31T22:15:07.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:07.702336+0000) 2022-01-31T22:15:07.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:07 smithi167 conmon[60316]: debug 2022-01-31T22:15:07.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:07.702336+0000) 2022-01-31T22:15:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:07 smithi171 conmon[41853]: debug 2022-01-31T22:15:07.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:07.404547+0000) 2022-01-31T22:15:08.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:07 smithi167 conmon[49112]: debug 2022-01-31T22:15:07.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:07.780538+0000) 2022-01-31T22:15:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:08 smithi171 conmon[46715]: debug 2022-01-31T22:15:08.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:08.197714+0000) 2022-01-31T22:15:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:08 smithi171 conmon[51620]: debug 2022-01-31T22:15:08.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:08.054748+0000) 2022-01-31T22:15:08.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:08 smithi167 conmon[54076]: debug 2022-01-31T22:15:08.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:08.702548+0000) 2022-01-31T22:15:08.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:08 smithi167 conmon[60316]: debug 2022-01-31T22:15:08.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:08.702548+0000) 2022-01-31T22:15:08.821 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:08 smithi171 conmon[41853]: debug 2022-01-31T22:15:08.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:08.404690+0000) 2022-01-31T22:15:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:08 smithi171 conmon[35325]: debug 2022-01-31T22:15:08.820+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:15:09.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:09 smithi171 conmon[51620]: debug 2022-01-31T22:15:09.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:09.054895+0000) 2022-01-31T22:15:09.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:08 smithi167 conmon[49112]: debug 2022-01-31T22:15:08.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:08.780697+0000) 2022-01-31T22:15:09.404 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:09 smithi171 conmon[46715]: debug 2022-01-31T22:15:09.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:09.197769+0000) 2022-01-31T22:15:09.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:09 smithi167 conmon[54076]: debug 2022-01-31T22:15:09.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:09.702747+0000) 2022-01-31T22:15:09.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:09 smithi167 conmon[60316]: debug 2022-01-31T22:15:09.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:09.702748+0000) 2022-01-31T22:15:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:09 smithi171 conmon[41853]: debug 2022-01-31T22:15:09.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:09.404928+0000) 2022-01-31T22:15:10.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:09 smithi167 conmon[49112]: debug 2022-01-31T22:15:09.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:09.780860+0000) 2022-01-31T22:15:10.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:10 smithi171 conmon[46715]: debug 2022-01-31T22:15:10.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:10.197923+0000) 2022-01-31T22:15:10.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:10 smithi171 conmon[51620]: debug 2022-01-31T22:15:10.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:10.055057+0000) 2022-01-31T22:15:10.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:10 smithi167 conmon[60316]: debug 2022-01-31T22:15:10.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:10.702927+0000) 2022-01-31T22:15:10.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:10 smithi167 conmon[54076]: debug 2022-01-31T22:15:10.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:10.702926+0000) 2022-01-31T22:15:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:10 smithi171 conmon[41853]: debug 2022-01-31T22:15:10.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:10.405044+0000) 2022-01-31T22:15:11.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:10 smithi167 conmon[49112]: debug 2022-01-31T22:15:10.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:10.781041+0000) 2022-01-31T22:15:11.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[46715]: debug 2022-01-31T22:15:11.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.198106+0000) 2022-01-31T22:15:11.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[51620]: debug 2022-01-31T22:15:11.053+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.055256+0000) 2022-01-31T22:15:11.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:11 smithi167 conmon[49112]: debug 2022-01-31T22:15:11.545+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.547110+0000) 2022-01-31T22:15:11.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:11 smithi167 conmon[54076]: debug 2022-01-31T22:15:11.544+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.546281+0000) 2022-01-31T22:15:11.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:11 smithi167 conmon[54076]: debug 2022-01-31T22:15:11.702+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.703142+0000) 2022-01-31T22:15:11.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:11 smithi167 conmon[60316]: debug 2022-01-31T22:15:11.545+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.547461+0000) 2022-01-31T22:15:11.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:11 smithi167 conmon[60316]: debug 2022-01-31T22:15:11.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.703142+0000) 2022-01-31T22:15:11.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[35325]: debug 2022-01-31T22:15:11.565+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194636 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:11.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[46715]: debug 2022-01-31T22:15:11.545+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.546471+0000) 2022-01-31T22:15:11.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[51620]: debug 2022-01-31T22:15:11.546+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.547700+0000) 2022-01-31T22:15:11.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[41853]: debug 2022-01-31T22:15:11.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.405204+0000) 2022-01-31T22:15:11.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:11 smithi171 conmon[41853]: debug 2022-01-31T22:15:11.546+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.547008+0000) 2022-01-31T22:15:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:11 smithi167 conmon[49112]: debug 2022-01-31T22:15:11.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:11.781252+0000) 2022-01-31T22:15:12.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:12 smithi171 conmon[46715]: debug 2022-01-31T22:15:12.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:12.198272+0000) 2022-01-31T22:15:12.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:12 smithi171 conmon[51620]: debug 2022-01-31T22:15:12.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:12.055454+0000) 2022-01-31T22:15:12.729 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:12 smithi171 conmon[41853]: debug 2022-01-31T22:15:12.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:12.405331+0000) 2022-01-31T22:15:12.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:12 smithi167 conmon[60316]: debug 2022-01-31T22:15:12.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:12.703335+0000) 2022-01-31T22:15:12.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:12 smithi167 conmon[54076]: debug 2022-01-31T22:15:12.701+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:12.703335+0000) 2022-01-31T22:15:13.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:12 smithi167 conmon[49112]: debug 2022-01-31T22:15:12.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:12.781435+0000) 2022-01-31T22:15:13.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:13 smithi171 conmon[46715]: debug 2022-01-31T22:15:13.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:13.198407+0000) 2022-01-31T22:15:13.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:13 smithi171 conmon[51620]: debug 2022-01-31T22:15:13.054+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:13.055586+0000) 2022-01-31T22:15:13.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:13 smithi167 conmon[54076]: debug 2022-01-31T22:15:13.702+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:13.703549+0000) 2022-01-31T22:15:13.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:13 smithi167 conmon[60316]: debug 2022-01-31T22:15:13.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:13.703547+0000) 2022-01-31T22:15:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:13 smithi171 conmon[41853]: debug 2022-01-31T22:15:13.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:13.405501+0000) 2022-01-31T22:15:14.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:13 smithi167 conmon[49112]: debug 2022-01-31T22:15:13.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:13.781629+0000) 2022-01-31T22:15:14.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:14 smithi171 conmon[46715]: debug 2022-01-31T22:15:14.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:14.198596+0000) 2022-01-31T22:15:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:14 smithi171 conmon[51620]: debug 2022-01-31T22:15:14.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:14.055780+0000) 2022-01-31T22:15:14.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:14 smithi167 conmon[60316]: debug 2022-01-31T22:15:14.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:14.703758+0000) 2022-01-31T22:15:14.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:14 smithi167 conmon[54076]: debug 2022-01-31T22:15:14.702+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:14.703758+0000) 2022-01-31T22:15:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:14 smithi171 conmon[41853]: debug 2022-01-31T22:15:14.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:14.405694+0000) 2022-01-31T22:15:15.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:14 smithi167 conmon[49112]: debug 2022-01-31T22:15:14.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:14.781834+0000) 2022-01-31T22:15:15.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:15 smithi171 conmon[46715]: debug 2022-01-31T22:15:15.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:15.198793+0000) 2022-01-31T22:15:15.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:15 smithi171 conmon[51620]: debug 2022-01-31T22:15:15.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:15.055980+0000) 2022-01-31T22:15:15.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:15 smithi167 conmon[54076]: debug 2022-01-31T22:15:15.703+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:15.703979+0000) 2022-01-31T22:15:15.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:15 smithi167 conmon[60316]: debug 2022-01-31T22:15:15.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:15.703980+0000) 2022-01-31T22:15:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:15 smithi171 conmon[41853]: debug 2022-01-31T22:15:15.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:15.405898+0000) 2022-01-31T22:15:16.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:15 smithi167 conmon[49112]: debug 2022-01-31T22:15:15.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:15.782049+0000) 2022-01-31T22:15:16.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[46715]: debug 2022-01-31T22:15:16.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.198965+0000) 2022-01-31T22:15:16.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[51620]: debug 2022-01-31T22:15:16.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.056170+0000) 2022-01-31T22:15:16.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:16 smithi167 conmon[49112]: debug 2022-01-31T22:15:16.567+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.569280+0000) 2022-01-31T22:15:16.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:16 smithi167 conmon[54076]: debug 2022-01-31T22:15:16.568+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.570066+0000) 2022-01-31T22:15:16.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:16 smithi167 conmon[54076]: debug 2022-01-31T22:15:16.702+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.704157+0000) 2022-01-31T22:15:16.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:16 smithi167 conmon[60316]: debug 2022-01-31T22:15:16.567+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.568994+0000) 2022-01-31T22:15:16.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:16 smithi167 conmon[60316]: debug 2022-01-31T22:15:16.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.704157+0000) 2022-01-31T22:15:16.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[35325]: debug 2022-01-31T22:15:16.588+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194748 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:16.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[46715]: debug 2022-01-31T22:15:16.569+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.569925+0000) 2022-01-31T22:15:16.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[51620]: debug 2022-01-31T22:15:16.569+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.570331+0000) 2022-01-31T22:15:16.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[41853]: debug 2022-01-31T22:15:16.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.406105+0000) 2022-01-31T22:15:16.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:16 smithi171 conmon[41853]: debug 2022-01-31T22:15:16.568+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.569151+0000) 2022-01-31T22:15:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:16 smithi167 conmon[49112]: debug 2022-01-31T22:15:16.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:16.782244+0000) 2022-01-31T22:15:17.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:17 smithi171 conmon[46715]: debug 2022-01-31T22:15:17.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:17.199066+0000) 2022-01-31T22:15:17.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:17 smithi171 conmon[51620]: debug 2022-01-31T22:15:17.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:17.056362+0000) 2022-01-31T22:15:17.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:17 smithi167 conmon[54076]: debug 2022-01-31T22:15:17.702+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:17.704345+0000) 2022-01-31T22:15:17.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:17 smithi167 conmon[60316]: debug 2022-01-31T22:15:17.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:17.704346+0000) 2022-01-31T22:15:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:17 smithi171 conmon[41853]: debug 2022-01-31T22:15:17.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:17.406289+0000) 2022-01-31T22:15:18.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:17 smithi167 conmon[49112]: debug 2022-01-31T22:15:17.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:17.782398+0000) 2022-01-31T22:15:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:18 smithi171 conmon[46715]: debug 2022-01-31T22:15:18.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:18.199204+0000) 2022-01-31T22:15:18.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:18 smithi171 conmon[51620]: debug 2022-01-31T22:15:18.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:18.056513+0000) 2022-01-31T22:15:18.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:18 smithi167 conmon[54076]: debug 2022-01-31T22:15:18.703+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:18.704552+0000) 2022-01-31T22:15:18.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:18 smithi167 conmon[60316]: debug 2022-01-31T22:15:18.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:18.704552+0000) 2022-01-31T22:15:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:18 smithi171 conmon[41853]: debug 2022-01-31T22:15:18.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:18.406446+0000) 2022-01-31T22:15:19.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:18 smithi167 conmon[49112]: debug 2022-01-31T22:15:18.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:18.782555+0000) 2022-01-31T22:15:19.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:19 smithi171 conmon[51620]: debug 2022-01-31T22:15:19.055+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:19.056705+0000) 2022-01-31T22:15:19.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:19 smithi171 conmon[46715]: debug 2022-01-31T22:15:19.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:19.199394+0000) 2022-01-31T22:15:19.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:19 smithi167 conmon[60316]: debug 2022-01-31T22:15:19.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:19.704741+0000) 2022-01-31T22:15:19.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:19 smithi167 conmon[54076]: debug 2022-01-31T22:15:19.703+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:19.704741+0000) 2022-01-31T22:15:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:19 smithi171 conmon[41853]: debug 2022-01-31T22:15:19.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:19.406628+0000) 2022-01-31T22:15:20.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:19 smithi167 conmon[49112]: debug 2022-01-31T22:15:19.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:19.782757+0000) 2022-01-31T22:15:20.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:20 smithi171 conmon[46715]: debug 2022-01-31T22:15:20.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:20.199598+0000) 2022-01-31T22:15:20.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:20 smithi171 conmon[51620]: debug 2022-01-31T22:15:20.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:20.056892+0000) 2022-01-31T22:15:20.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:20 smithi167 conmon[54076]: debug 2022-01-31T22:15:20.704+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:20.704957+0000) 2022-01-31T22:15:20.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:20 smithi167 conmon[60316]: debug 2022-01-31T22:15:20.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:20.704957+0000) 2022-01-31T22:15:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:20 smithi171 conmon[41853]: debug 2022-01-31T22:15:20.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:20.406818+0000) 2022-01-31T22:15:21.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:20 smithi167 conmon[49112]: debug 2022-01-31T22:15:20.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:20.782964+0000) 2022-01-31T22:15:21.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[51620]: debug 2022-01-31T22:15:21.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.057044+0000) 2022-01-31T22:15:21.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[46715]: debug 2022-01-31T22:15:21.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.199816+0000) 2022-01-31T22:15:21.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:21 smithi167 conmon[49112]: debug 2022-01-31T22:15:21.591+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.592615+0000) 2022-01-31T22:15:21.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:21 smithi167 conmon[54076]: debug 2022-01-31T22:15:21.590+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.591777+0000) 2022-01-31T22:15:21.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:21 smithi167 conmon[54076]: debug 2022-01-31T22:15:21.704+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.705172+0000) 2022-01-31T22:15:21.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:21 smithi167 conmon[60316]: debug 2022-01-31T22:15:21.590+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.592215+0000) 2022-01-31T22:15:21.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:21 smithi167 conmon[60316]: debug 2022-01-31T22:15:21.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.705172+0000) 2022-01-31T22:15:21.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[35325]: debug 2022-01-31T22:15:21.611+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194858 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:21.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[41853]: debug 2022-01-31T22:15:21.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.406977+0000) 2022-01-31T22:15:21.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[41853]: debug 2022-01-31T22:15:21.591+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.592610+0000) 2022-01-31T22:15:21.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[46715]: debug 2022-01-31T22:15:21.591+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.592442+0000) 2022-01-31T22:15:21.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:21 smithi171 conmon[51620]: debug 2022-01-31T22:15:21.591+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.592028+0000) 2022-01-31T22:15:22.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:21 smithi167 conmon[49112]: debug 2022-01-31T22:15:21.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:21.783129+0000) 2022-01-31T22:15:22.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:22 smithi171 conmon[46715]: debug 2022-01-31T22:15:22.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:22.199983+0000) 2022-01-31T22:15:22.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:22 smithi171 conmon[51620]: debug 2022-01-31T22:15:22.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:22.057265+0000) 2022-01-31T22:15:22.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:22 smithi167 conmon[54076]: debug 2022-01-31T22:15:22.704+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:22.705346+0000) 2022-01-31T22:15:22.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:22 smithi167 conmon[60316]: debug 2022-01-31T22:15:22.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:22.705346+0000) 2022-01-31T22:15:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:22 smithi171 conmon[41853]: debug 2022-01-31T22:15:22.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:22.407124+0000) 2022-01-31T22:15:23.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:22 smithi167 conmon[49112]: debug 2022-01-31T22:15:22.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:22.783309+0000) 2022-01-31T22:15:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:23 smithi171 conmon[46715]: debug 2022-01-31T22:15:23.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:23.200142+0000) 2022-01-31T22:15:23.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:23 smithi171 conmon[51620]: debug 2022-01-31T22:15:23.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:23.057424+0000) 2022-01-31T22:15:23.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:23 smithi167 conmon[54076]: debug 2022-01-31T22:15:23.704+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:23.705563+0000) 2022-01-31T22:15:23.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:23 smithi167 conmon[60316]: debug 2022-01-31T22:15:23.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:23.705584+0000) 2022-01-31T22:15:23.821 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:23 smithi171 conmon[41853]: debug 2022-01-31T22:15:23.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:23.407306+0000) 2022-01-31T22:15:24.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:24 smithi171 conmon[51620]: debug 2022-01-31T22:15:24.056+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:24.057637+0000) 2022-01-31T22:15:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:23 smithi171 conmon[35325]: debug 2022-01-31T22:15:23.821+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:15:24.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:23 smithi167 conmon[49112]: debug 2022-01-31T22:15:23.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:23.783539+0000) 2022-01-31T22:15:24.407 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:24 smithi171 conmon[46715]: debug 2022-01-31T22:15:24.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:24.200352+0000) 2022-01-31T22:15:24.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:24 smithi167 conmon[54076]: debug 2022-01-31T22:15:24.705+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:24.705754+0000) 2022-01-31T22:15:24.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:24 smithi167 conmon[60316]: debug 2022-01-31T22:15:24.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:24.705754+0000) 2022-01-31T22:15:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:24 smithi171 conmon[41853]: debug 2022-01-31T22:15:24.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:24.407492+0000) 2022-01-31T22:15:25.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:24 smithi167 conmon[49112]: debug 2022-01-31T22:15:24.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:24.783699+0000) 2022-01-31T22:15:25.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:25 smithi171 conmon[46715]: debug 2022-01-31T22:15:25.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:25.200562+0000) 2022-01-31T22:15:25.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:25 smithi171 conmon[51620]: debug 2022-01-31T22:15:25.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:25.057811+0000) 2022-01-31T22:15:25.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:25 smithi167 conmon[60316]: debug 2022-01-31T22:15:25.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:25.705869+0000) 2022-01-31T22:15:25.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:25 smithi167 conmon[54076]: debug 2022-01-31T22:15:25.704+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:25.705869+0000) 2022-01-31T22:15:25.866 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:25 smithi171 conmon[41853]: debug 2022-01-31T22:15:25.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:25.407679+0000) 2022-01-31T22:15:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:25 smithi167 conmon[49112]: debug 2022-01-31T22:15:25.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:25.783878+0000) 2022-01-31T22:15:26.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[46715]: debug 2022-01-31T22:15:26.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.200740+0000) 2022-01-31T22:15:26.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[51620]: debug 2022-01-31T22:15:26.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.057974+0000) 2022-01-31T22:15:26.614 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[46715]: debug 2022-01-31T22:15:26.614+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.615017+0000) 2022-01-31T22:15:26.615 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[51620]: debug 2022-01-31T22:15:26.614+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.615286+0000) 2022-01-31T22:15:26.616 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[41853]: debug 2022-01-31T22:15:26.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.407874+0000) 2022-01-31T22:15:26.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:26 smithi167 conmon[49112]: debug 2022-01-31T22:15:26.614+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.615652+0000) 2022-01-31T22:15:26.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:26 smithi167 conmon[54076]: debug 2022-01-31T22:15:26.612+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.613995+0000) 2022-01-31T22:15:26.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:26 smithi167 conmon[54076]: debug 2022-01-31T22:15:26.704+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.706060+0000) 2022-01-31T22:15:26.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:26 smithi167 conmon[60316]: debug 2022-01-31T22:15:26.613+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.614573+0000) 2022-01-31T22:15:26.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:26 smithi167 conmon[60316]: debug 2022-01-31T22:15:26.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.706060+0000) 2022-01-31T22:15:27.057 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[35325]: debug 2022-01-31T22:15:26.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 194968 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:27.058 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:26 smithi171 conmon[41853]: debug 2022-01-31T22:15:26.614+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.615230+0000) 2022-01-31T22:15:27.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:26 smithi167 conmon[49112]: debug 2022-01-31T22:15:26.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:26.784064+0000) 2022-01-31T22:15:27.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:27 smithi171 conmon[46715]: debug 2022-01-31T22:15:27.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:27.200917+0000) 2022-01-31T22:15:27.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:27 smithi171 conmon[51620]: debug 2022-01-31T22:15:27.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:27.058167+0000) 2022-01-31T22:15:27.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:27 smithi167 conmon[60316]: debug 2022-01-31T22:15:27.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:27.706232+0000) 2022-01-31T22:15:27.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:27 smithi167 conmon[54076]: debug 2022-01-31T22:15:27.705+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:27.706232+0000) 2022-01-31T22:15:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:27 smithi171 conmon[41853]: debug 2022-01-31T22:15:27.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:27.408021+0000) 2022-01-31T22:15:28.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:27 smithi167 conmon[49112]: debug 2022-01-31T22:15:27.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:27.784234+0000) 2022-01-31T22:15:28.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:28 smithi171 conmon[46715]: debug 2022-01-31T22:15:28.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:28.201071+0000) 2022-01-31T22:15:28.322 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:28 smithi171 conmon[51620]: debug 2022-01-31T22:15:28.057+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:28.058283+0000) 2022-01-31T22:15:28.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:28 smithi167 conmon[54076]: debug 2022-01-31T22:15:28.705+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:28.706404+0000) 2022-01-31T22:15:28.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:28 smithi167 conmon[60316]: debug 2022-01-31T22:15:28.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:28.706404+0000) 2022-01-31T22:15:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:28 smithi171 conmon[41853]: debug 2022-01-31T22:15:28.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:28.408149+0000) 2022-01-31T22:15:29.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:28 smithi167 conmon[49112]: debug 2022-01-31T22:15:28.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:28.784455+0000) 2022-01-31T22:15:29.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:29 smithi171 conmon[46715]: debug 2022-01-31T22:15:29.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:29.201237+0000) 2022-01-31T22:15:29.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:29 smithi171 conmon[51620]: debug 2022-01-31T22:15:29.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:29.058514+0000) 2022-01-31T22:15:29.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:29 smithi167 conmon[54076]: debug 2022-01-31T22:15:29.706+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:29.706582+0000) 2022-01-31T22:15:29.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:29 smithi167 conmon[60316]: debug 2022-01-31T22:15:29.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:29.706585+0000) 2022-01-31T22:15:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:29 smithi171 conmon[41853]: debug 2022-01-31T22:15:29.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:29.408328+0000) 2022-01-31T22:15:30.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:29 smithi167 conmon[49112]: debug 2022-01-31T22:15:29.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:29.784661+0000) 2022-01-31T22:15:30.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:30 smithi171 conmon[51620]: debug 2022-01-31T22:15:30.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:30.058725+0000) 2022-01-31T22:15:30.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:30 smithi171 conmon[46715]: debug 2022-01-31T22:15:30.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:30.201403+0000) 2022-01-31T22:15:30.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:30 smithi167 conmon[54076]: debug 2022-01-31T22:15:30.706+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:30.706762+0000) 2022-01-31T22:15:30.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:30 smithi167 conmon[60316]: debug 2022-01-31T22:15:30.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:30.706762+0000) 2022-01-31T22:15:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:30 smithi171 conmon[41853]: debug 2022-01-31T22:15:30.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:30.408481+0000) 2022-01-31T22:15:31.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:30 smithi167 conmon[49112]: debug 2022-01-31T22:15:30.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:30.784897+0000) 2022-01-31T22:15:31.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[46715]: debug 2022-01-31T22:15:31.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.201548+0000) 2022-01-31T22:15:31.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[51620]: debug 2022-01-31T22:15:31.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.058913+0000) 2022-01-31T22:15:31.637 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[41853]: debug 2022-01-31T22:15:31.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.408637+0000) 2022-01-31T22:15:31.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:31 smithi167 conmon[49112]: debug 2022-01-31T22:15:31.639+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.639672+0000) 2022-01-31T22:15:31.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:31 smithi167 conmon[60316]: debug 2022-01-31T22:15:31.638+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.639339+0000) 2022-01-31T22:15:31.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:31 smithi167 conmon[60316]: debug 2022-01-31T22:15:31.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.706968+0000) 2022-01-31T22:15:31.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:31 smithi167 conmon[54076]: debug 2022-01-31T22:15:31.637+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.637713+0000) 2022-01-31T22:15:31.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:31 smithi167 conmon[54076]: debug 2022-01-31T22:15:31.706+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.706967+0000) 2022-01-31T22:15:32.058 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[46715]: debug 2022-01-31T22:15:31.637+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.638279+0000) 2022-01-31T22:15:32.059 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[51620]: debug 2022-01-31T22:15:31.638+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.639253+0000) 2022-01-31T22:15:32.060 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[35325]: debug 2022-01-31T22:15:31.657+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195078 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:32.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[41853]: debug 2022-01-31T22:15:31.637+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.638598+0000) 2022-01-31T22:15:32.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:31 smithi171 conmon[41853]: 2022-01-31T22:15:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:31 smithi167 conmon[49112]: debug 2022-01-31T22:15:31.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:31.785074+0000) 2022-01-31T22:15:32.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:32 smithi171 conmon[46715]: debug 2022-01-31T22:15:32.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:32.201738+0000) 2022-01-31T22:15:32.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:32 smithi171 conmon[51620]: debug 2022-01-31T22:15:32.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:32.059101+0000) 2022-01-31T22:15:32.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:32 smithi167 conmon[54076]: debug 2022-01-31T22:15:32.706+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:32.707124+0000) 2022-01-31T22:15:32.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:32 smithi167 conmon[60316]: debug 2022-01-31T22:15:32.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:32.707124+0000) 2022-01-31T22:15:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:32 smithi171 conmon[41853]: debug 2022-01-31T22:15:32.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:32.408870+0000) 2022-01-31T22:15:33.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:32 smithi167 conmon[49112]: debug 2022-01-31T22:15:32.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:32.785256+0000) 2022-01-31T22:15:33.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:33 smithi171 conmon[51620]: debug 2022-01-31T22:15:33.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:33.059263+0000) 2022-01-31T22:15:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:33 smithi171 conmon[46715]: debug 2022-01-31T22:15:33.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:33.201926+0000) 2022-01-31T22:15:33.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:33 smithi167 conmon[54076]: debug 2022-01-31T22:15:33.706+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:33.707331+0000) 2022-01-31T22:15:33.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:33 smithi167 conmon[60316]: debug 2022-01-31T22:15:33.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:33.707329+0000) 2022-01-31T22:15:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:33 smithi171 conmon[41853]: debug 2022-01-31T22:15:33.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:33.409016+0000) 2022-01-31T22:15:34.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:33 smithi167 conmon[49112]: debug 2022-01-31T22:15:33.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:33.785435+0000) 2022-01-31T22:15:34.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:34 smithi171 conmon[51620]: debug 2022-01-31T22:15:34.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:34.059395+0000) 2022-01-31T22:15:34.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:34 smithi171 conmon[46715]: debug 2022-01-31T22:15:34.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:34.202055+0000) 2022-01-31T22:15:34.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:34 smithi167 conmon[54076]: debug 2022-01-31T22:15:34.707+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:34.707567+0000) 2022-01-31T22:15:34.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:34 smithi167 conmon[60316]: debug 2022-01-31T22:15:34.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:34.707567+0000) 2022-01-31T22:15:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:34 smithi171 conmon[41853]: debug 2022-01-31T22:15:34.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:34.409220+0000) 2022-01-31T22:15:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:34 smithi167 conmon[49112]: debug 2022-01-31T22:15:34.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:34.785592+0000) 2022-01-31T22:15:35.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:35 smithi171 conmon[46715]: debug 2022-01-31T22:15:35.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:35.202203+0000) 2022-01-31T22:15:35.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:35 smithi171 conmon[51620]: debug 2022-01-31T22:15:35.058+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:35.059610+0000) 2022-01-31T22:15:35.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:35 smithi167 conmon[54076]: debug 2022-01-31T22:15:35.707+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:35.707727+0000) 2022-01-31T22:15:35.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:35 smithi167 conmon[60316]: debug 2022-01-31T22:15:35.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:35.707727+0000) 2022-01-31T22:15:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:35 smithi171 conmon[41853]: debug 2022-01-31T22:15:35.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:35.409442+0000) 2022-01-31T22:15:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:35 smithi167 conmon[49112]: debug 2022-01-31T22:15:35.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:35.785753+0000) 2022-01-31T22:15:36.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[46715]: debug 2022-01-31T22:15:36.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.202363+0000) 2022-01-31T22:15:36.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[51620]: debug 2022-01-31T22:15:36.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.059785+0000) 2022-01-31T22:15:36.659 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[41853]: debug 2022-01-31T22:15:36.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.409658+0000) 2022-01-31T22:15:36.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:36 smithi167 conmon[49112]: debug 2022-01-31T22:15:36.659+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.660429+0000) 2022-01-31T22:15:36.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:36 smithi167 conmon[54076]: debug 2022-01-31T22:15:36.660+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.660726+0000) 2022-01-31T22:15:36.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:36 smithi167 conmon[54076]: debug 2022-01-31T22:15:36.707+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.707940+0000) 2022-01-31T22:15:36.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:36 smithi167 conmon[60316]: debug 2022-01-31T22:15:36.661+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.662027+0000) 2022-01-31T22:15:36.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:36 smithi167 conmon[60316]: debug 2022-01-31T22:15:36.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.707940+0000) 2022-01-31T22:15:37.059 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[35325]: debug 2022-01-31T22:15:36.680+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195190 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:37.060 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[46715]: debug 2022-01-31T22:15:36.660+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.661271+0000) 2022-01-31T22:15:37.061 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[51620]: debug 2022-01-31T22:15:36.659+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.660462+0000) 2022-01-31T22:15:37.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:36 smithi171 conmon[41853]: debug 2022-01-31T22:15:36.660+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.661697+0000) 2022-01-31T22:15:37.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:36 smithi167 conmon[49112]: debug 2022-01-31T22:15:36.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:36.785955+0000) 2022-01-31T22:15:37.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:37 smithi171 conmon[46715]: debug 2022-01-31T22:15:37.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:37.202518+0000) 2022-01-31T22:15:37.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:37 smithi171 conmon[51620]: debug 2022-01-31T22:15:37.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:37.059896+0000) 2022-01-31T22:15:37.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:37 smithi167 conmon[54076]: debug 2022-01-31T22:15:37.707+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:37.708112+0000) 2022-01-31T22:15:37.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:37 smithi167 conmon[60316]: debug 2022-01-31T22:15:37.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:37.708112+0000) 2022-01-31T22:15:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:37 smithi171 conmon[41853]: debug 2022-01-31T22:15:37.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:37.409869+0000) 2022-01-31T22:15:38.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:37 smithi167 conmon[49112]: debug 2022-01-31T22:15:37.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:37.786107+0000) 2022-01-31T22:15:38.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:38 smithi171 conmon[51620]: debug 2022-01-31T22:15:38.059+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:38.060048+0000) 2022-01-31T22:15:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:38 smithi171 conmon[46715]: debug 2022-01-31T22:15:38.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:38.202674+0000) 2022-01-31T22:15:38.785 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:38 smithi167 conmon[60316]: debug 2022-01-31T22:15:38.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:38.708329+0000) 2022-01-31T22:15:38.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:38 smithi167 conmon[54076]: debug 2022-01-31T22:15:38.707+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:38.708329+0000) 2022-01-31T22:15:38.822 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:38 smithi171 conmon[41853]: debug 2022-01-31T22:15:38.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:38.410014+0000) 2022-01-31T22:15:39.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:39 smithi171 conmon[51620]: debug 2022-01-31T22:15:39.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:39.060277+0000) 2022-01-31T22:15:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:38 smithi171 conmon[35325]: debug 2022-01-31T22:15:38.822+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:15:39.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:38 smithi167 conmon[49112]: debug 2022-01-31T22:15:38.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:38.786299+0000) 2022-01-31T22:15:39.409 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:39 smithi171 conmon[46715]: debug 2022-01-31T22:15:39.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:39.202859+0000) 2022-01-31T22:15:39.785 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:39 smithi167 conmon[54076]: debug 2022-01-31T22:15:39.708+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:39.708521+0000) 2022-01-31T22:15:39.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:39 smithi167 conmon[60316]: debug 2022-01-31T22:15:39.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:39.708521+0000) 2022-01-31T22:15:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:39 smithi171 conmon[41853]: debug 2022-01-31T22:15:39.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:39.410197+0000) 2022-01-31T22:15:40.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:39 smithi167 conmon[49112]: debug 2022-01-31T22:15:39.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:39.786459+0000) 2022-01-31T22:15:40.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:40 smithi171 conmon[51620]: debug 2022-01-31T22:15:40.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:40.060468+0000) 2022-01-31T22:15:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:40 smithi171 conmon[46715]: debug 2022-01-31T22:15:40.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:40.202998+0000) 2022-01-31T22:15:40.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:40 smithi167 conmon[60316]: debug 2022-01-31T22:15:40.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:40.708676+0000) 2022-01-31T22:15:40.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:40 smithi167 conmon[54076]: debug 2022-01-31T22:15:40.708+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:40.708677+0000) 2022-01-31T22:15:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:40 smithi171 conmon[41853]: debug 2022-01-31T22:15:40.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:40.410387+0000) 2022-01-31T22:15:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:40 smithi167 conmon[49112]: debug 2022-01-31T22:15:40.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:40.786666+0000) 2022-01-31T22:15:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[46715]: debug 2022-01-31T22:15:41.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.203180+0000) 2022-01-31T22:15:41.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[51620]: debug 2022-01-31T22:15:41.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.060662+0000) 2022-01-31T22:15:41.683 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[41853]: debug 2022-01-31T22:15:41.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.410573+0000) 2022-01-31T22:15:41.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:41 smithi167 conmon[49112]: debug 2022-01-31T22:15:41.682+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.683114+0000) 2022-01-31T22:15:41.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:41 smithi167 conmon[60316]: debug 2022-01-31T22:15:41.684+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.685397+0000) 2022-01-31T22:15:41.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:41 smithi167 conmon[60316]: debug 2022-01-31T22:15:41.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.708855+0000) 2022-01-31T22:15:41.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:41 smithi167 conmon[54076]: debug 2022-01-31T22:15:41.683+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.684461+0000) 2022-01-31T22:15:41.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:41 smithi167 conmon[54076]: debug 2022-01-31T22:15:41.708+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.708834+0000) 2022-01-31T22:15:42.060 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[35325]: debug 2022-01-31T22:15:41.703+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195300 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:42.061 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[41853]: debug 2022-01-31T22:15:41.683+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.684662+0000) 2022-01-31T22:15:42.061 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[46715]: debug 2022-01-31T22:15:41.683+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.683831+0000) 2022-01-31T22:15:42.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:41 smithi171 conmon[51620]: debug 2022-01-31T22:15:41.682+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.683585+0000) 2022-01-31T22:15:42.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:41 smithi167 conmon[49112]: debug 2022-01-31T22:15:41.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:41.786874+0000) 2022-01-31T22:15:42.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:42 smithi171 conmon[46715]: debug 2022-01-31T22:15:42.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:42.203310+0000) 2022-01-31T22:15:42.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:42 smithi171 conmon[51620]: debug 2022-01-31T22:15:42.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:42.060834+0000) 2022-01-31T22:15:42.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:42 smithi167 conmon[54076]: debug 2022-01-31T22:15:42.708+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:42.709013+0000) 2022-01-31T22:15:42.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:42 smithi167 conmon[60316]: debug 2022-01-31T22:15:42.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:42.709008+0000) 2022-01-31T22:15:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:42 smithi171 conmon[41853]: debug 2022-01-31T22:15:42.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:42.410728+0000) 2022-01-31T22:15:43.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:42 smithi167 conmon[49112]: debug 2022-01-31T22:15:42.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:42.787055+0000) 2022-01-31T22:15:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:43 smithi171 conmon[46715]: debug 2022-01-31T22:15:43.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:43.203456+0000) 2022-01-31T22:15:43.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:43 smithi171 conmon[51620]: debug 2022-01-31T22:15:43.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:43.060963+0000) 2022-01-31T22:15:43.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:43 smithi167 conmon[54076]: debug 2022-01-31T22:15:43.708+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:43.709210+0000) 2022-01-31T22:15:43.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:43 smithi167 conmon[60316]: debug 2022-01-31T22:15:43.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:43.709232+0000) 2022-01-31T22:15:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:43 smithi171 conmon[41853]: debug 2022-01-31T22:15:43.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:43.410926+0000) 2022-01-31T22:15:44.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:43 smithi167 conmon[49112]: debug 2022-01-31T22:15:43.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:43.787217+0000) 2022-01-31T22:15:44.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:44 smithi171 conmon[46715]: debug 2022-01-31T22:15:44.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:44.203666+0000) 2022-01-31T22:15:44.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:44 smithi171 conmon[51620]: debug 2022-01-31T22:15:44.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:44.061170+0000) 2022-01-31T22:15:44.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:44 smithi167 conmon[54076]: debug 2022-01-31T22:15:44.708+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:44.709470+0000) 2022-01-31T22:15:44.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:44 smithi167 conmon[60316]: debug 2022-01-31T22:15:44.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:44.709466+0000) 2022-01-31T22:15:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:44 smithi171 conmon[41853]: debug 2022-01-31T22:15:44.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:44.411108+0000) 2022-01-31T22:15:45.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:44 smithi167 conmon[49112]: debug 2022-01-31T22:15:44.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:44.787405+0000) 2022-01-31T22:15:45.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:45 smithi171 conmon[46715]: debug 2022-01-31T22:15:45.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:45.203854+0000) 2022-01-31T22:15:45.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:45 smithi171 conmon[51620]: debug 2022-01-31T22:15:45.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:45.061339+0000) 2022-01-31T22:15:45.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:45 smithi167 conmon[54076]: debug 2022-01-31T22:15:45.709+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:45.709630+0000) 2022-01-31T22:15:45.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:45 smithi167 conmon[60316]: debug 2022-01-31T22:15:45.709+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:45.709629+0000) 2022-01-31T22:15:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:45 smithi171 conmon[41853]: debug 2022-01-31T22:15:45.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:45.411322+0000) 2022-01-31T22:15:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:45 smithi167 conmon[49112]: debug 2022-01-31T22:15:45.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:45.787581+0000) 2022-01-31T22:15:46.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[46715]: debug 2022-01-31T22:15:46.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.204053+0000) 2022-01-31T22:15:46.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[51620]: debug 2022-01-31T22:15:46.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.061538+0000) 2022-01-31T22:15:46.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[41853]: debug 2022-01-31T22:15:46.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.411477+0000) 2022-01-31T22:15:46.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[41853]: debug 2022-01-31T22:15:46.705+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.707549+0000) 2022-01-31T22:15:46.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:46 smithi167 conmon[49112]: debug 2022-01-31T22:15:46.705+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.706047+0000) 2022-01-31T22:15:46.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:46 smithi167 conmon[54076]: debug 2022-01-31T22:15:46.706+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.707233+0000) 2022-01-31T22:15:46.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:46 smithi167 conmon[54076]: debug 2022-01-31T22:15:46.709+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.709735+0000) 2022-01-31T22:15:46.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:46 smithi167 conmon[60316]: debug 2022-01-31T22:15:46.713+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.714427+0000) 2022-01-31T22:15:47.061 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[35325]: debug 2022-01-31T22:15:46.740+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195413 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:47.062 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[46715]: debug 2022-01-31T22:15:46.704+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.706680+0000) 2022-01-31T22:15:47.062 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:46 smithi171 conmon[51620]: debug 2022-01-31T22:15:46.705+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.707395+0000) 2022-01-31T22:15:47.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:46 smithi167 conmon[49112]: debug 2022-01-31T22:15:46.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:46.787722+0000) 2022-01-31T22:15:47.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:47 smithi171 conmon[46715]: debug 2022-01-31T22:15:47.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:47.204227+0000) 2022-01-31T22:15:47.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:47 smithi171 conmon[51620]: debug 2022-01-31T22:15:47.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:47.061704+0000) 2022-01-31T22:15:47.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:47 smithi167 conmon[54076]: debug 2022-01-31T22:15:47.709+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:47.709845+0000) 2022-01-31T22:15:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:47 smithi171 conmon[41853]: debug 2022-01-31T22:15:47.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:47.411675+0000) 2022-01-31T22:15:48.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:47 smithi167 conmon[49112]: debug 2022-01-31T22:15:47.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:47.787823+0000) 2022-01-31T22:15:48.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:48 smithi171 conmon[51620]: debug 2022-01-31T22:15:48.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:48.061857+0000) 2022-01-31T22:15:48.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:48 smithi171 conmon[46715]: debug 2022-01-31T22:15:48.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:48.204398+0000) 2022-01-31T22:15:48.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:48 smithi167 conmon[54076]: debug 2022-01-31T22:15:48.709+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:48.710013+0000) 2022-01-31T22:15:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:48 smithi171 conmon[41853]: debug 2022-01-31T22:15:48.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:48.411839+0000) 2022-01-31T22:15:49.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:48 smithi167 conmon[49112]: debug 2022-01-31T22:15:48.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:48.788000+0000) 2022-01-31T22:15:49.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:49 smithi171 conmon[46715]: debug 2022-01-31T22:15:49.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:49.204601+0000) 2022-01-31T22:15:49.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:49 smithi171 conmon[51620]: debug 2022-01-31T22:15:49.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:49.062017+0000) 2022-01-31T22:15:49.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:49 smithi167 conmon[54076]: debug 2022-01-31T22:15:49.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:49.710189+0000) 2022-01-31T22:15:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:49 smithi171 conmon[41853]: debug 2022-01-31T22:15:49.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:49.411986+0000) 2022-01-31T22:15:50.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:49 smithi167 conmon[49112]: debug 2022-01-31T22:15:49.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:49.788203+0000) 2022-01-31T22:15:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:50 smithi171 conmon[46715]: debug 2022-01-31T22:15:50.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:50.204700+0000) 2022-01-31T22:15:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:50 smithi171 conmon[51620]: debug 2022-01-31T22:15:50.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:50.062192+0000) 2022-01-31T22:15:50.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:50 smithi167 conmon[54076]: debug 2022-01-31T22:15:50.709+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:50.710401+0000) 2022-01-31T22:15:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:50 smithi171 conmon[41853]: debug 2022-01-31T22:15:50.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:50.412175+0000) 2022-01-31T22:15:51.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:50 smithi167 conmon[49112]: debug 2022-01-31T22:15:50.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:50.788414+0000) 2022-01-31T22:15:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[46715]: debug 2022-01-31T22:15:51.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.204863+0000) 2022-01-31T22:15:51.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[51620]: debug 2022-01-31T22:15:51.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.062392+0000) 2022-01-31T22:15:51.743 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[41853]: debug 2022-01-31T22:15:51.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.412352+0000) 2022-01-31T22:15:51.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:51 smithi167 conmon[49112]: debug 2022-01-31T22:15:51.743+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.744300+0000) 2022-01-31T22:15:51.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:51 smithi167 conmon[60316]: debug 2022-01-31T22:15:51.745+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.745960+0000) 2022-01-31T22:15:51.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:51 smithi167 conmon[54076]: debug 2022-01-31T22:15:51.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.710569+0000) 2022-01-31T22:15:51.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:51 smithi167 conmon[54076]: debug 2022-01-31T22:15:51.744+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.744551+0000) 2022-01-31T22:15:52.062 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[35325]: debug 2022-01-31T22:15:51.763+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195523 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:52.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[41853]: debug 2022-01-31T22:15:51.743+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.745420+0000) 2022-01-31T22:15:52.063 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[46715]: debug 2022-01-31T22:15:51.743+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.744810+0000) 2022-01-31T22:15:52.064 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:51 smithi171 conmon[51620]: debug 2022-01-31T22:15:51.744+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.745958+0000) 2022-01-31T22:15:52.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:51 smithi167 conmon[49112]: debug 2022-01-31T22:15:51.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:51.788608+0000) 2022-01-31T22:15:52.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:52 smithi171 conmon[46715]: debug 2022-01-31T22:15:52.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:52.205067+0000) 2022-01-31T22:15:52.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:52 smithi171 conmon[51620]: debug 2022-01-31T22:15:52.061+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:52.062603+0000) 2022-01-31T22:15:52.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:52 smithi167 conmon[54076]: debug 2022-01-31T22:15:52.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:52.710720+0000) 2022-01-31T22:15:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:52 smithi171 conmon[41853]: debug 2022-01-31T22:15:52.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:52.412556+0000) 2022-01-31T22:15:53.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:52 smithi167 conmon[49112]: debug 2022-01-31T22:15:52.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:52.788812+0000) 2022-01-31T22:15:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:53 smithi171 conmon[46715]: debug 2022-01-31T22:15:53.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:53.205223+0000) 2022-01-31T22:15:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:53 smithi171 conmon[51620]: debug 2022-01-31T22:15:53.060+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:53.062735+0000) 2022-01-31T22:15:53.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:53 smithi167 conmon[60316]: debug 2022-01-31T22:15:53.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:53.463081+0000) 2022-01-31T22:15:53.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:53 smithi167 conmon[54076]: debug 2022-01-31T22:15:53.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:53.710836+0000) 2022-01-31T22:15:53.823 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:53 smithi171 conmon[41853]: debug 2022-01-31T22:15:53.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:53.412705+0000) 2022-01-31T22:15:54.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:53 smithi171 conmon[35325]: debug 2022-01-31T22:15:53.822+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:15:54.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:54 smithi171 conmon[51620]: debug 2022-01-31T22:15:54.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:54.062909+0000) 2022-01-31T22:15:54.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:53 smithi167 conmon[49112]: debug 2022-01-31T22:15:53.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:53.789009+0000) 2022-01-31T22:15:54.412 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:54 smithi171 conmon[46715]: debug 2022-01-31T22:15:54.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:54.205432+0000) 2022-01-31T22:15:54.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:54 smithi167 conmon[54076]: debug 2022-01-31T22:15:54.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:54.710998+0000) 2022-01-31T22:15:54.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:54 smithi167 conmon[60316]: debug 2022-01-31T22:15:54.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:54.463256+0000) 2022-01-31T22:15:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:54 smithi171 conmon[41853]: debug 2022-01-31T22:15:54.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:54.412869+0000) 2022-01-31T22:15:55.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:54 smithi167 conmon[49112]: debug 2022-01-31T22:15:54.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:54.789168+0000) 2022-01-31T22:15:55.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:55 smithi171 conmon[46715]: debug 2022-01-31T22:15:55.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:55.205618+0000) 2022-01-31T22:15:55.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:55 smithi171 conmon[51620]: debug 2022-01-31T22:15:55.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:55.063096+0000) 2022-01-31T22:15:55.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:55 smithi167 conmon[54076]: debug 2022-01-31T22:15:55.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:55.711172+0000) 2022-01-31T22:15:55.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:55 smithi167 conmon[60316]: debug 2022-01-31T22:15:55.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:55.463469+0000) 2022-01-31T22:15:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:55 smithi171 conmon[41853]: debug 2022-01-31T22:15:55.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:55.413061+0000) 2022-01-31T22:15:56.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:55 smithi167 conmon[49112]: debug 2022-01-31T22:15:55.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:55.789323+0000) 2022-01-31T22:15:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[46715]: debug 2022-01-31T22:15:56.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.205791+0000) 2022-01-31T22:15:56.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[51620]: debug 2022-01-31T22:15:56.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.063288+0000) 2022-01-31T22:15:56.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:56 smithi167 conmon[54076]: debug 2022-01-31T22:15:56.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.711332+0000) 2022-01-31T22:15:56.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:56 smithi167 conmon[60316]: debug 2022-01-31T22:15:56.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.463679+0000) 2022-01-31T22:15:56.766 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[41853]: debug 2022-01-31T22:15:56.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.413195+0000) 2022-01-31T22:15:57.063 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[35325]: debug 2022-01-31T22:15:56.786+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195636 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:15:57.063 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[41853]: debug 2022-01-31T22:15:56.766+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.767830+0000) 2022-01-31T22:15:57.064 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[46715]: debug 2022-01-31T22:15:56.765+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.767493+0000) 2022-01-31T22:15:57.065 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:56 smithi171 conmon[51620]: debug 2022-01-31T22:15:56.766+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.768344+0000) 2022-01-31T22:15:57.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:56 smithi167 conmon[54076]: debug 2022-01-31T22:15:56.765+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.767170+0000) 2022-01-31T22:15:57.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:56 smithi167 conmon[60316]: debug 2022-01-31T22:15:56.766+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.768154+0000) 2022-01-31T22:15:57.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:56 smithi167 conmon[49112]: debug 2022-01-31T22:15:56.766+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.767915+0000) 2022-01-31T22:15:57.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:56 smithi167 conmon[49112]: debug 2022-01-31T22:15:56.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:56.789474+0000) 2022-01-31T22:15:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:57 smithi171 conmon[46715]: debug 2022-01-31T22:15:57.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:57.205985+0000) 2022-01-31T22:15:57.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:57 smithi171 conmon[51620]: debug 2022-01-31T22:15:57.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:57.063498+0000) 2022-01-31T22:15:57.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:57 smithi167 conmon[54076]: debug 2022-01-31T22:15:57.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:57.711535+0000) 2022-01-31T22:15:57.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:57 smithi167 conmon[60316]: debug 2022-01-31T22:15:57.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:57.463887+0000) 2022-01-31T22:15:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:57 smithi171 conmon[41853]: debug 2022-01-31T22:15:57.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:57.413299+0000) 2022-01-31T22:15:58.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:57 smithi167 conmon[49112]: debug 2022-01-31T22:15:57.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:57.789627+0000) 2022-01-31T22:15:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:58 smithi171 conmon[46715]: debug 2022-01-31T22:15:58.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:58.206101+0000) 2022-01-31T22:15:58.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:58 smithi171 conmon[51620]: debug 2022-01-31T22:15:58.062+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:58.063683+0000) 2022-01-31T22:15:58.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:58 smithi167 conmon[54076]: debug 2022-01-31T22:15:58.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:58.711676+0000) 2022-01-31T22:15:58.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:58 smithi167 conmon[60316]: debug 2022-01-31T22:15:58.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:58.464035+0000) 2022-01-31T22:15:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:58 smithi171 conmon[41853]: debug 2022-01-31T22:15:58.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:58.413465+0000) 2022-01-31T22:15:59.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:58 smithi167 conmon[49112]: debug 2022-01-31T22:15:58.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:58.789726+0000) 2022-01-31T22:15:59.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:15:59 smithi171 conmon[46715]: debug 2022-01-31T22:15:59.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:59.206252+0000) 2022-01-31T22:15:59.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:15:59 smithi171 conmon[51620]: debug 2022-01-31T22:15:59.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:59.063915+0000) 2022-01-31T22:15:59.757 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:15:59 smithi167 conmon[54076]: debug 2022-01-31T22:15:59.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:59.711866+0000) 2022-01-31T22:15:59.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:15:59 smithi167 conmon[60316]: debug 2022-01-31T22:15:59.462+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:59.464219+0000) 2022-01-31T22:15:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:15:59 smithi171 conmon[41853]: debug 2022-01-31T22:15:59.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:59.413631+0000) 2022-01-31T22:16:00.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:15:59 smithi167 conmon[49112]: debug 2022-01-31T22:15:59.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:15:59.789881+0000) 2022-01-31T22:16:00.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:00 smithi171 conmon[46715]: debug 2022-01-31T22:16:00.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:00.206415+0000) 2022-01-31T22:16:00.363 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:00 smithi171 conmon[51620]: debug 2022-01-31T22:16:00.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:00.064123+0000) 2022-01-31T22:16:00.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:00 smithi167 conmon[54076]: debug 2022-01-31T22:16:00.711+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:00.712077+0000) 2022-01-31T22:16:00.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:00 smithi167 conmon[60316]: debug 2022-01-31T22:16:00.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:00.464404+0000) 2022-01-31T22:16:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:00 smithi171 conmon[41853]: debug 2022-01-31T22:16:00.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:00.413821+0000) 2022-01-31T22:16:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:00 smithi167 conmon[49112]: debug 2022-01-31T22:16:00.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:00.790036+0000) 2022-01-31T22:16:01.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[51620]: debug 2022-01-31T22:16:01.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.064325+0000) 2022-01-31T22:16:01.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[46715]: debug 2022-01-31T22:16:01.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.206623+0000) 2022-01-31T22:16:01.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:01 smithi167 conmon[54076]: debug 2022-01-31T22:16:01.710+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.712279+0000) 2022-01-31T22:16:01.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:01 smithi167 conmon[60316]: debug 2022-01-31T22:16:01.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.464579+0000) 2022-01-31T22:16:01.789 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[41853]: debug 2022-01-31T22:16:01.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.414024+0000) 2022-01-31T22:16:02.064 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[35325]: debug 2022-01-31T22:16:01.810+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195746 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:02.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[41853]: debug 2022-01-31T22:16:01.791+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.792348+0000) 2022-01-31T22:16:02.065 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[46715]: debug 2022-01-31T22:16:01.789+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.790180+0000) 2022-01-31T22:16:02.066 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:01 smithi171 conmon[51620]: debug 2022-01-31T22:16:01.790+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.791402+0000) 2022-01-31T22:16:02.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:01 smithi167 conmon[49112]: debug 2022-01-31T22:16:01.790+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.792113+0000) 2022-01-31T22:16:02.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:01 smithi167 conmon[54076]: debug 2022-01-31T22:16:01.790+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.791793+0000) 2022-01-31T22:16:02.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:01 smithi167 conmon[60316]: debug 2022-01-31T22:16:01.790+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:01.792038+0000) 2022-01-31T22:16:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:02 smithi171 conmon[46715]: debug 2022-01-31T22:16:02.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:02.206829+0000) 2022-01-31T22:16:02.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:02 smithi171 conmon[51620]: debug 2022-01-31T22:16:02.063+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:02.064538+0000) 2022-01-31T22:16:02.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:02 smithi167 conmon[54076]: debug 2022-01-31T22:16:02.711+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:02.712488+0000) 2022-01-31T22:16:02.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:02 smithi167 conmon[60316]: debug 2022-01-31T22:16:02.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:02.464758+0000) 2022-01-31T22:16:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:02 smithi171 conmon[41853]: debug 2022-01-31T22:16:02.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:02.414226+0000) 2022-01-31T22:16:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:03 smithi171 conmon[46715]: debug 2022-01-31T22:16:03.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:03.206991+0000) 2022-01-31T22:16:03.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:03 smithi171 conmon[51620]: debug 2022-01-31T22:16:03.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:03.064706+0000) 2022-01-31T22:16:03.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:03 smithi167 conmon[54076]: debug 2022-01-31T22:16:03.711+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:03.712605+0000) 2022-01-31T22:16:03.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:03 smithi167 conmon[60316]: debug 2022-01-31T22:16:03.463+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:03.464924+0000) 2022-01-31T22:16:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:03 smithi171 conmon[41853]: debug 2022-01-31T22:16:03.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:03.414402+0000) 2022-01-31T22:16:04.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:04 smithi171 conmon[51620]: debug 2022-01-31T22:16:04.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:04.064883+0000) 2022-01-31T22:16:04.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:04 smithi171 conmon[46715]: debug 2022-01-31T22:16:04.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:04.207207+0000) 2022-01-31T22:16:04.758 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:04 smithi167 conmon[54076]: debug 2022-01-31T22:16:04.711+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:04.712796+0000) 2022-01-31T22:16:04.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:04 smithi167 conmon[60316]: debug 2022-01-31T22:16:04.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:04.465126+0000) 2022-01-31T22:16:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:04 smithi171 conmon[41853]: debug 2022-01-31T22:16:04.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:04.414612+0000) 2022-01-31T22:16:05.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:05 smithi171 conmon[46715]: debug 2022-01-31T22:16:05.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:05.207394+0000) 2022-01-31T22:16:05.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:05 smithi171 conmon[51620]: debug 2022-01-31T22:16:05.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:05.065062+0000) 2022-01-31T22:16:05.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:05 smithi167 conmon[54076]: debug 2022-01-31T22:16:05.712+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:05.712961+0000) 2022-01-31T22:16:05.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:05 smithi167 conmon[60316]: debug 2022-01-31T22:16:05.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:05.465325+0000) 2022-01-31T22:16:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:05 smithi171 conmon[41853]: debug 2022-01-31T22:16:05.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:05.414794+0000) 2022-01-31T22:16:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[46715]: debug 2022-01-31T22:16:06.206+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.207617+0000) 2022-01-31T22:16:06.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[51620]: debug 2022-01-31T22:16:06.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.065301+0000) 2022-01-31T22:16:06.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:06 smithi167 conmon[54076]: debug 2022-01-31T22:16:06.712+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.713113+0000) 2022-01-31T22:16:06.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:06 smithi167 conmon[60316]: debug 2022-01-31T22:16:06.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.465542+0000) 2022-01-31T22:16:06.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[41853]: debug 2022-01-31T22:16:06.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.414947+0000) 2022-01-31T22:16:07.064 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[35325]: debug 2022-01-31T22:16:06.833+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195858 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:07.065 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[41853]: debug 2022-01-31T22:16:06.814+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.815117+0000) 2022-01-31T22:16:07.066 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[46715]: debug 2022-01-31T22:16:06.812+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.813458+0000) 2022-01-31T22:16:07.066 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:06 smithi171 conmon[51620]: debug 2022-01-31T22:16:06.813+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.814433+0000) 2022-01-31T22:16:07.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:06 smithi167 conmon[49112]: debug 2022-01-31T22:16:06.813+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.815292+0000) 2022-01-31T22:16:07.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:06 smithi167 conmon[54076]: debug 2022-01-31T22:16:06.813+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.814995+0000) 2022-01-31T22:16:07.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:06 smithi167 conmon[60316]: debug 2022-01-31T22:16:06.814+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:06.815516+0000) 2022-01-31T22:16:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:07 smithi171 conmon[46715]: debug 2022-01-31T22:16:07.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:07.207783+0000) 2022-01-31T22:16:07.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:07 smithi171 conmon[51620]: debug 2022-01-31T22:16:07.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:07.065519+0000) 2022-01-31T22:16:07.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:07 smithi167 conmon[54076]: debug 2022-01-31T22:16:07.712+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:07.713246+0000) 2022-01-31T22:16:07.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:07 smithi167 conmon[60316]: debug 2022-01-31T22:16:07.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:07.465731+0000) 2022-01-31T22:16:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:07 smithi171 conmon[41853]: debug 2022-01-31T22:16:07.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:07.415132+0000) 2022-01-31T22:16:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:08 smithi171 conmon[46715]: debug 2022-01-31T22:16:08.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:08.207943+0000) 2022-01-31T22:16:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:08 smithi171 conmon[51620]: debug 2022-01-31T22:16:08.064+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:08.065625+0000) 2022-01-31T22:16:08.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:08 smithi167 conmon[49112]: debug 2022-01-31T22:16:08.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:08.540444+0000) 2022-01-31T22:16:08.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:08 smithi167 conmon[54076]: debug 2022-01-31T22:16:08.711+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:08.713337+0000) 2022-01-31T22:16:08.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:08 smithi167 conmon[60316]: debug 2022-01-31T22:16:08.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:08.465888+0000) 2022-01-31T22:16:08.823 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:08 smithi171 conmon[41853]: debug 2022-01-31T22:16:08.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:08.415273+0000) 2022-01-31T22:16:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:08 smithi171 conmon[35325]: debug 2022-01-31T22:16:08.823+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:16:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:09 smithi171 conmon[51620]: debug 2022-01-31T22:16:09.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:09.065840+0000) 2022-01-31T22:16:09.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:09 smithi171 conmon[46715]: debug 2022-01-31T22:16:09.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:09.208127+0000) 2022-01-31T22:16:09.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:09 smithi167 conmon[49112]: debug 2022-01-31T22:16:09.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:09.540668+0000) 2022-01-31T22:16:09.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:09 smithi167 conmon[54076]: debug 2022-01-31T22:16:09.712+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:09.713546+0000) 2022-01-31T22:16:09.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:09 smithi167 conmon[60316]: debug 2022-01-31T22:16:09.464+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:09.466077+0000) 2022-01-31T22:16:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:09 smithi171 conmon[41853]: debug 2022-01-31T22:16:09.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:09.415488+0000) 2022-01-31T22:16:10.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:10 smithi171 conmon[46715]: debug 2022-01-31T22:16:10.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:10.208319+0000) 2022-01-31T22:16:10.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:10 smithi171 conmon[51620]: debug 2022-01-31T22:16:10.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:10.066036+0000) 2022-01-31T22:16:10.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:10 smithi167 conmon[49112]: debug 2022-01-31T22:16:10.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:10.540849+0000) 2022-01-31T22:16:10.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:10 smithi167 conmon[54076]: debug 2022-01-31T22:16:10.713+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:10.713752+0000) 2022-01-31T22:16:10.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:10 smithi167 conmon[60316]: debug 2022-01-31T22:16:10.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:10.466298+0000) 2022-01-31T22:16:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:10 smithi171 conmon[41853]: debug 2022-01-31T22:16:10.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:10.415698+0000) 2022-01-31T22:16:11.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[46715]: debug 2022-01-31T22:16:11.207+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.208521+0000) 2022-01-31T22:16:11.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[51620]: debug 2022-01-31T22:16:11.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.066198+0000) 2022-01-31T22:16:11.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:11 smithi167 conmon[49112]: debug 2022-01-31T22:16:11.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.541045+0000) 2022-01-31T22:16:11.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:11 smithi167 conmon[54076]: debug 2022-01-31T22:16:11.713+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.713944+0000) 2022-01-31T22:16:11.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:11 smithi167 conmon[60316]: debug 2022-01-31T22:16:11.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.466529+0000) 2022-01-31T22:16:11.836 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[41853]: debug 2022-01-31T22:16:11.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.415914+0000) 2022-01-31T22:16:11.836 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[41853]: debug 2022-01-31T22:16:11.836+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.836838+0000) 2022-01-31T22:16:12.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[35325]: debug 2022-01-31T22:16:11.856+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 195968 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:12.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[46715]: debug 2022-01-31T22:16:11.837+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.837876+0000) 2022-01-31T22:16:12.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:11 smithi171 conmon[51620]: debug 2022-01-31T22:16:11.835+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.836615+0000) 2022-01-31T22:16:12.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:12 smithi171 conmon[51620]: debug 2022-01-31T22:16:12.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:12.066425+0000) 2022-01-31T22:16:12.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:11 smithi167 conmon[49112]: debug 2022-01-31T22:16:11.836+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.838218+0000) 2022-01-31T22:16:12.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:11 smithi167 conmon[54076]: debug 2022-01-31T22:16:11.836+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.837596+0000) 2022-01-31T22:16:12.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:11 smithi167 conmon[60316]: debug 2022-01-31T22:16:11.836+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:11.838027+0000) 2022-01-31T22:16:12.415 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:12 smithi171 conmon[46715]: debug 2022-01-31T22:16:12.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:12.208707+0000) 2022-01-31T22:16:12.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:12 smithi167 conmon[60316]: debug 2022-01-31T22:16:12.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:12.466676+0000) 2022-01-31T22:16:12.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:12 smithi167 conmon[49112]: debug 2022-01-31T22:16:12.539+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:12.541242+0000) 2022-01-31T22:16:12.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:12 smithi167 conmon[54076]: debug 2022-01-31T22:16:12.713+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:12.714149+0000) 2022-01-31T22:16:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:12 smithi171 conmon[41853]: debug 2022-01-31T22:16:12.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:12.416070+0000) 2022-01-31T22:16:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:13 smithi171 conmon[46715]: debug 2022-01-31T22:16:13.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:13.208863+0000) 2022-01-31T22:16:13.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:13 smithi171 conmon[51620]: debug 2022-01-31T22:16:13.065+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:13.066552+0000) 2022-01-31T22:16:13.657 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:13 smithi171 conmon[41853]: debug 2022-01-31T22:16:13.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:13.416220+0000) 2022-01-31T22:16:13.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:13 smithi167 conmon[49112]: debug 2022-01-31T22:16:13.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:13.541379+0000) 2022-01-31T22:16:13.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:13 smithi167 conmon[54076]: debug 2022-01-31T22:16:13.713+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:13.714324+0000) 2022-01-31T22:16:13.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:13 smithi167 conmon[60316]: debug 2022-01-31T22:16:13.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:13.466826+0000) 2022-01-31T22:16:14.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:14 smithi171 conmon[46715]: debug 2022-01-31T22:16:14.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:14.209079+0000) 2022-01-31T22:16:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:14 smithi171 conmon[51620]: debug 2022-01-31T22:16:14.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:14.066781+0000) 2022-01-31T22:16:14.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:14 smithi167 conmon[49112]: debug 2022-01-31T22:16:14.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:14.541536+0000) 2022-01-31T22:16:14.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:14 smithi167 conmon[54076]: debug 2022-01-31T22:16:14.713+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:14.714526+0000) 2022-01-31T22:16:14.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:14 smithi167 conmon[60316]: debug 2022-01-31T22:16:14.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:14.467032+0000) 2022-01-31T22:16:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:14 smithi171 conmon[41853]: debug 2022-01-31T22:16:14.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:14.416381+0000) 2022-01-31T22:16:15.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:15 smithi171 conmon[46715]: debug 2022-01-31T22:16:15.208+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:15.209275+0000) 2022-01-31T22:16:15.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:15 smithi171 conmon[51620]: debug 2022-01-31T22:16:15.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:15.066976+0000) 2022-01-31T22:16:15.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:15 smithi167 conmon[49112]: debug 2022-01-31T22:16:15.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:15.541733+0000) 2022-01-31T22:16:15.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:15 smithi167 conmon[54076]: debug 2022-01-31T22:16:15.714+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:15.714702+0000) 2022-01-31T22:16:15.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:15 smithi167 conmon[60316]: debug 2022-01-31T22:16:15.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:15.467244+0000) 2022-01-31T22:16:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:15 smithi171 conmon[41853]: debug 2022-01-31T22:16:15.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:15.416564+0000) 2022-01-31T22:16:16.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[46715]: debug 2022-01-31T22:16:16.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.209462+0000) 2022-01-31T22:16:16.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[51620]: debug 2022-01-31T22:16:16.066+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.067201+0000) 2022-01-31T22:16:16.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:16 smithi167 conmon[49112]: debug 2022-01-31T22:16:16.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.541887+0000) 2022-01-31T22:16:16.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:16 smithi167 conmon[54076]: debug 2022-01-31T22:16:16.714+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.714897+0000) 2022-01-31T22:16:16.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:16 smithi167 conmon[60316]: debug 2022-01-31T22:16:16.465+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.467461+0000) 2022-01-31T22:16:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[41853]: debug 2022-01-31T22:16:16.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.416746+0000) 2022-01-31T22:16:17.162 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[46715]: debug 2022-01-31T22:16:16.859+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.860207+0000) 2022-01-31T22:16:17.163 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[35325]: debug 2022-01-31T22:16:16.879+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196080 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:17.163 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[41853]: debug 2022-01-31T22:16:16.859+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.860369+0000) 2022-01-31T22:16:17.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:16 smithi171 conmon[51620]: debug 2022-01-31T22:16:16.858+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.858794+0000) 2022-01-31T22:16:17.164 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:17 smithi171 conmon[51620]: debug 2022-01-31T22:16:17.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:17.067434+0000) 2022-01-31T22:16:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:16 smithi167 conmon[49112]: debug 2022-01-31T22:16:16.858+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.860338+0000) 2022-01-31T22:16:17.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:16 smithi167 conmon[54076]: debug 2022-01-31T22:16:16.858+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.859831+0000) 2022-01-31T22:16:17.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:16 smithi167 conmon[60316]: debug 2022-01-31T22:16:16.857+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:16.859156+0000) 2022-01-31T22:16:17.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:17 smithi171 conmon[46715]: debug 2022-01-31T22:16:17.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:17.209648+0000) 2022-01-31T22:16:17.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:17 smithi167 conmon[60316]: debug 2022-01-31T22:16:17.466+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:17.467591+0000) 2022-01-31T22:16:17.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:17 smithi167 conmon[49112]: debug 2022-01-31T22:16:17.540+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:17.542092+0000) 2022-01-31T22:16:17.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:17 smithi167 conmon[54076]: debug 2022-01-31T22:16:17.714+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:17.715087+0000) 2022-01-31T22:16:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:17 smithi171 conmon[41853]: debug 2022-01-31T22:16:17.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:17.416861+0000) 2022-01-31T22:16:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:18 smithi171 conmon[46715]: debug 2022-01-31T22:16:18.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:18.209847+0000) 2022-01-31T22:16:18.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:18 smithi171 conmon[51620]: debug 2022-01-31T22:16:18.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:18.067620+0000) 2022-01-31T22:16:18.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:18 smithi167 conmon[49112]: debug 2022-01-31T22:16:18.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:18.542243+0000) 2022-01-31T22:16:18.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:18 smithi167 conmon[54076]: debug 2022-01-31T22:16:18.713+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:18.715181+0000) 2022-01-31T22:16:18.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:18 smithi167 conmon[60316]: debug 2022-01-31T22:16:18.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:18.467745+0000) 2022-01-31T22:16:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:18 smithi171 conmon[41853]: debug 2022-01-31T22:16:18.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:18.416985+0000) 2022-01-31T22:16:19.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:19 smithi171 conmon[46715]: debug 2022-01-31T22:16:19.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:19.210041+0000) 2022-01-31T22:16:19.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:19 smithi171 conmon[51620]: debug 2022-01-31T22:16:19.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:19.067841+0000) 2022-01-31T22:16:19.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:19 smithi167 conmon[49112]: debug 2022-01-31T22:16:19.541+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:19.542425+0000) 2022-01-31T22:16:19.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:19 smithi167 conmon[54076]: debug 2022-01-31T22:16:19.714+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:19.715292+0000) 2022-01-31T22:16:19.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:19 smithi167 conmon[60316]: debug 2022-01-31T22:16:19.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:19.467949+0000) 2022-01-31T22:16:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:19 smithi171 conmon[41853]: debug 2022-01-31T22:16:19.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:19.417187+0000) 2022-01-31T22:16:20.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:20 smithi171 conmon[46715]: debug 2022-01-31T22:16:20.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:20.210221+0000) 2022-01-31T22:16:20.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:20 smithi171 conmon[51620]: debug 2022-01-31T22:16:20.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:20.068054+0000) 2022-01-31T22:16:20.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:20 smithi167 conmon[49112]: debug 2022-01-31T22:16:20.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:20.542548+0000) 2022-01-31T22:16:20.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:20 smithi167 conmon[54076]: debug 2022-01-31T22:16:20.715+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:20.715489+0000) 2022-01-31T22:16:20.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:20 smithi167 conmon[60316]: debug 2022-01-31T22:16:20.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:20.468145+0000) 2022-01-31T22:16:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:20 smithi171 conmon[41853]: debug 2022-01-31T22:16:20.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:20.417386+0000) 2022-01-31T22:16:21.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[46715]: debug 2022-01-31T22:16:21.209+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.210437+0000) 2022-01-31T22:16:21.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[51620]: debug 2022-01-31T22:16:21.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.068256+0000) 2022-01-31T22:16:21.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:21 smithi167 conmon[49112]: debug 2022-01-31T22:16:21.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.542740+0000) 2022-01-31T22:16:21.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:21 smithi167 conmon[54076]: debug 2022-01-31T22:16:21.715+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.715693+0000) 2022-01-31T22:16:21.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:21 smithi167 conmon[60316]: debug 2022-01-31T22:16:21.467+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.468348+0000) 2022-01-31T22:16:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[41853]: debug 2022-01-31T22:16:21.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.417586+0000) 2022-01-31T22:16:22.163 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[41853]: debug 2022-01-31T22:16:21.882+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.883529+0000) 2022-01-31T22:16:22.164 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[46715]: debug 2022-01-31T22:16:21.882+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.883343+0000) 2022-01-31T22:16:22.165 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[35325]: debug 2022-01-31T22:16:21.902+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196190 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:22.165 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:21 smithi171 conmon[51620]: debug 2022-01-31T22:16:21.881+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.881886+0000) 2022-01-31T22:16:22.166 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:22 smithi171 conmon[51620]: debug 2022-01-31T22:16:22.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:22.068424+0000) 2022-01-31T22:16:22.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:21 smithi167 conmon[60316]: debug 2022-01-31T22:16:21.882+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.882758+0000) 2022-01-31T22:16:22.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:21 smithi167 conmon[49112]: debug 2022-01-31T22:16:21.882+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.882865+0000) 2022-01-31T22:16:22.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:21 smithi167 conmon[54076]: debug 2022-01-31T22:16:21.881+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:21.882028+0000) 2022-01-31T22:16:22.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:22 smithi171 conmon[46715]: debug 2022-01-31T22:16:22.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:22.210625+0000) 2022-01-31T22:16:22.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:22 smithi167 conmon[49112]: debug 2022-01-31T22:16:22.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:22.542925+0000) 2022-01-31T22:16:22.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:22 smithi167 conmon[54076]: debug 2022-01-31T22:16:22.715+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:22.715887+0000) 2022-01-31T22:16:22.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:22 smithi167 conmon[60316]: debug 2022-01-31T22:16:22.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:22.468538+0000) 2022-01-31T22:16:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:22 smithi171 conmon[41853]: debug 2022-01-31T22:16:22.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:22.417787+0000) 2022-01-31T22:16:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:23 smithi171 conmon[46715]: debug 2022-01-31T22:16:23.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:23.210797+0000) 2022-01-31T22:16:23.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:23 smithi171 conmon[51620]: debug 2022-01-31T22:16:23.067+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:23.068595+0000) 2022-01-31T22:16:23.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:23 smithi167 conmon[49112]: debug 2022-01-31T22:16:23.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:23.543116+0000) 2022-01-31T22:16:23.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:23 smithi167 conmon[54076]: debug 2022-01-31T22:16:23.715+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:23.716033+0000) 2022-01-31T22:16:23.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:23 smithi167 conmon[60316]: debug 2022-01-31T22:16:23.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:23.468690+0000) 2022-01-31T22:16:23.824 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:23 smithi171 conmon[41853]: debug 2022-01-31T22:16:23.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:23.417958+0000) 2022-01-31T22:16:24.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:24 smithi171 conmon[51620]: debug 2022-01-31T22:16:24.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:24.068811+0000) 2022-01-31T22:16:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:23 smithi171 conmon[35325]: debug 2022-01-31T22:16:23.824+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:16:24.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:24 smithi171 conmon[46715]: debug 2022-01-31T22:16:24.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:24.210965+0000) 2022-01-31T22:16:24.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:24 smithi167 conmon[49112]: debug 2022-01-31T22:16:24.542+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:24.543347+0000) 2022-01-31T22:16:24.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:24 smithi167 conmon[54076]: debug 2022-01-31T22:16:24.715+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:24.716186+0000) 2022-01-31T22:16:24.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:24 smithi167 conmon[60316]: debug 2022-01-31T22:16:24.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:24.468912+0000) 2022-01-31T22:16:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:24 smithi171 conmon[41853]: debug 2022-01-31T22:16:24.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:24.418161+0000) 2022-01-31T22:16:25.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:25 smithi171 conmon[46715]: debug 2022-01-31T22:16:25.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:25.211126+0000) 2022-01-31T22:16:25.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:25 smithi171 conmon[51620]: debug 2022-01-31T22:16:25.068+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:25.069019+0000) 2022-01-31T22:16:25.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:25 smithi167 conmon[49112]: debug 2022-01-31T22:16:25.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:25.543562+0000) 2022-01-31T22:16:25.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:25 smithi167 conmon[54076]: debug 2022-01-31T22:16:25.715+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:25.716350+0000) 2022-01-31T22:16:25.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:25 smithi167 conmon[60316]: debug 2022-01-31T22:16:25.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:25.469109+0000) 2022-01-31T22:16:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:25 smithi171 conmon[41853]: debug 2022-01-31T22:16:25.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:25.418350+0000) 2022-01-31T22:16:26.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[51620]: debug 2022-01-31T22:16:26.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.069234+0000) 2022-01-31T22:16:26.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[46715]: debug 2022-01-31T22:16:26.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.211276+0000) 2022-01-31T22:16:26.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:26 smithi167 conmon[49112]: debug 2022-01-31T22:16:26.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.543776+0000) 2022-01-31T22:16:26.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:26 smithi167 conmon[54076]: debug 2022-01-31T22:16:26.716+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.716538+0000) 2022-01-31T22:16:26.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:26 smithi167 conmon[60316]: debug 2022-01-31T22:16:26.468+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.469309+0000) 2022-01-31T22:16:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[41853]: debug 2022-01-31T22:16:26.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.418578+0000) 2022-01-31T22:16:27.164 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[35325]: debug 2022-01-31T22:16:26.926+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196302 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:27.165 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[41853]: debug 2022-01-31T22:16:26.905+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.906612+0000) 2022-01-31T22:16:27.166 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[46715]: debug 2022-01-31T22:16:26.906+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.906904+0000) 2022-01-31T22:16:27.166 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:26 smithi171 conmon[51620]: debug 2022-01-31T22:16:26.904+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.905038+0000) 2022-01-31T22:16:27.167 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:27 smithi171 conmon[51620]: debug 2022-01-31T22:16:27.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:27.069430+0000) 2022-01-31T22:16:27.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:26 smithi167 conmon[49112]: debug 2022-01-31T22:16:26.905+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.905743+0000) 2022-01-31T22:16:27.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:26 smithi167 conmon[54076]: debug 2022-01-31T22:16:26.905+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.906014+0000) 2022-01-31T22:16:27.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:26 smithi167 conmon[60316]: debug 2022-01-31T22:16:26.905+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:26.906461+0000) 2022-01-31T22:16:27.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:27 smithi171 conmon[46715]: debug 2022-01-31T22:16:27.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:27.211437+0000) 2022-01-31T22:16:27.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:27 smithi167 conmon[49112]: debug 2022-01-31T22:16:27.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:27.543954+0000) 2022-01-31T22:16:27.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:27 smithi167 conmon[54076]: debug 2022-01-31T22:16:27.716+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:27.716635+0000) 2022-01-31T22:16:27.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:27 smithi167 conmon[60316]: debug 2022-01-31T22:16:27.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:27.469514+0000) 2022-01-31T22:16:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:27 smithi171 conmon[41853]: debug 2022-01-31T22:16:27.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:27.418785+0000) 2022-01-31T22:16:28.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:28 smithi171 conmon[46715]: debug 2022-01-31T22:16:28.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:28.211566+0000) 2022-01-31T22:16:28.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:28 smithi171 conmon[51620]: debug 2022-01-31T22:16:28.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:28.069600+0000) 2022-01-31T22:16:28.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:28 smithi167 conmon[49112]: debug 2022-01-31T22:16:28.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:28.544085+0000) 2022-01-31T22:16:28.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:28 smithi167 conmon[54076]: debug 2022-01-31T22:16:28.716+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:28.716747+0000) 2022-01-31T22:16:28.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:28 smithi167 conmon[60316]: debug 2022-01-31T22:16:28.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:28.469582+0000) 2022-01-31T22:16:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:28 smithi171 conmon[41853]: debug 2022-01-31T22:16:28.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:28.418914+0000) 2022-01-31T22:16:29.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:29 smithi171 conmon[46715]: debug 2022-01-31T22:16:29.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:29.211732+0000) 2022-01-31T22:16:29.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:29 smithi171 conmon[51620]: debug 2022-01-31T22:16:29.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:29.069818+0000) 2022-01-31T22:16:29.761 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:29 smithi171 conmon[41853]: debug 2022-01-31T22:16:29.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:29.419074+0000) 2022-01-31T22:16:29.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:29 smithi167 conmon[49112]: debug 2022-01-31T22:16:29.543+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:29.544315+0000) 2022-01-31T22:16:29.763 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:29 smithi167 conmon[54076]: debug 2022-01-31T22:16:29.716+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:29.716955+0000) 2022-01-31T22:16:29.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:29 smithi167 conmon[60316]: debug 2022-01-31T22:16:29.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:29.469774+0000) 2022-01-31T22:16:30.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:30 smithi171 conmon[46715]: debug 2022-01-31T22:16:30.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:30.211906+0000) 2022-01-31T22:16:30.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:30 smithi171 conmon[51620]: debug 2022-01-31T22:16:30.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:30.070037+0000) 2022-01-31T22:16:30.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:30 smithi167 conmon[49112]: debug 2022-01-31T22:16:30.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:30.544483+0000) 2022-01-31T22:16:30.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:30 smithi167 conmon[54076]: debug 2022-01-31T22:16:30.716+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:30.717110+0000) 2022-01-31T22:16:30.764 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:30 smithi167 conmon[60316]: debug 2022-01-31T22:16:30.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:30.469938+0000) 2022-01-31T22:16:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:30 smithi171 conmon[41853]: debug 2022-01-31T22:16:30.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:30.419256+0000) 2022-01-31T22:16:31.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[46715]: debug 2022-01-31T22:16:31.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.212091+0000) 2022-01-31T22:16:31.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[51620]: debug 2022-01-31T22:16:31.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.070245+0000) 2022-01-31T22:16:31.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:31 smithi167 conmon[49112]: debug 2022-01-31T22:16:31.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.544663+0000) 2022-01-31T22:16:31.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:31 smithi167 conmon[54076]: debug 2022-01-31T22:16:31.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.717292+0000) 2022-01-31T22:16:31.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:31 smithi167 conmon[60316]: debug 2022-01-31T22:16:31.469+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.470140+0000) 2022-01-31T22:16:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[41853]: debug 2022-01-31T22:16:31.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.419445+0000) 2022-01-31T22:16:32.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:31 smithi167 conmon[49112]: debug 2022-01-31T22:16:31.929+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.929824+0000) 2022-01-31T22:16:32.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:31 smithi167 conmon[54076]: debug 2022-01-31T22:16:31.928+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.929424+0000) 2022-01-31T22:16:32.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:31 smithi167 conmon[60316]: debug 2022-01-31T22:16:31.929+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.930070+0000) 2022-01-31T22:16:32.211 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[35325]: debug 2022-01-31T22:16:31.948+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196412 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:32.212 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[41853]: debug 2022-01-31T22:16:31.928+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.930308+0000) 2022-01-31T22:16:32.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[46715]: debug 2022-01-31T22:16:31.928+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.930485+0000) 2022-01-31T22:16:32.213 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:31 smithi171 conmon[51620]: debug 2022-01-31T22:16:31.926+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:31.928266+0000) 2022-01-31T22:16:32.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:32 smithi171 conmon[51620]: debug 2022-01-31T22:16:32.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:32.070458+0000) 2022-01-31T22:16:32.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:32 smithi171 conmon[41853]: debug 2022-01-31T22:16:32.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:32.419593+0000) 2022-01-31T22:16:32.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:32 smithi171 conmon[46715]: debug 2022-01-31T22:16:32.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:32.212265+0000) 2022-01-31T22:16:32.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:32 smithi167 conmon[49112]: debug 2022-01-31T22:16:32.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:32.544871+0000) 2022-01-31T22:16:32.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:32 smithi167 conmon[54076]: debug 2022-01-31T22:16:32.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:32.717479+0000) 2022-01-31T22:16:32.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:32 smithi167 conmon[60316]: debug 2022-01-31T22:16:32.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:32.470348+0000) 2022-01-31T22:16:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:33 smithi171 conmon[46715]: debug 2022-01-31T22:16:33.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:33.212426+0000) 2022-01-31T22:16:33.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:33 smithi171 conmon[51620]: debug 2022-01-31T22:16:33.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:33.070578+0000) 2022-01-31T22:16:33.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:33 smithi167 conmon[49112]: debug 2022-01-31T22:16:33.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:33.545011+0000) 2022-01-31T22:16:33.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:33 smithi167 conmon[54076]: debug 2022-01-31T22:16:33.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:33.717616+0000) 2022-01-31T22:16:33.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:33 smithi167 conmon[60316]: debug 2022-01-31T22:16:33.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:33.470484+0000) 2022-01-31T22:16:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:33 smithi171 conmon[41853]: debug 2022-01-31T22:16:33.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:33.419803+0000) 2022-01-31T22:16:34.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:34 smithi171 conmon[51620]: debug 2022-01-31T22:16:34.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:34.070776+0000) 2022-01-31T22:16:34.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:34 smithi171 conmon[46715]: debug 2022-01-31T22:16:34.210+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:34.212612+0000) 2022-01-31T22:16:34.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:34 smithi167 conmon[49112]: debug 2022-01-31T22:16:34.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:34.545195+0000) 2022-01-31T22:16:34.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:34 smithi167 conmon[54076]: debug 2022-01-31T22:16:34.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:34.717753+0000) 2022-01-31T22:16:34.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:34 smithi167 conmon[60316]: debug 2022-01-31T22:16:34.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:34.470715+0000) 2022-01-31T22:16:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:34 smithi171 conmon[41853]: debug 2022-01-31T22:16:34.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:34.419955+0000) 2022-01-31T22:16:35.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:35 smithi171 conmon[51620]: debug 2022-01-31T22:16:35.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:35.070938+0000) 2022-01-31T22:16:35.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:35 smithi171 conmon[46715]: debug 2022-01-31T22:16:35.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:35.212739+0000) 2022-01-31T22:16:35.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:35 smithi167 conmon[49112]: debug 2022-01-31T22:16:35.544+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:35.545384+0000) 2022-01-31T22:16:35.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:35 smithi167 conmon[54076]: debug 2022-01-31T22:16:35.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:35.717891+0000) 2022-01-31T22:16:35.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:35 smithi167 conmon[60316]: debug 2022-01-31T22:16:35.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:35.470902+0000) 2022-01-31T22:16:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:35 smithi171 conmon[41853]: debug 2022-01-31T22:16:35.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:35.420121+0000) 2022-01-31T22:16:36.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[46715]: debug 2022-01-31T22:16:36.211+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.212858+0000) 2022-01-31T22:16:36.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[51620]: debug 2022-01-31T22:16:36.069+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.071141+0000) 2022-01-31T22:16:36.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:36 smithi167 conmon[49112]: debug 2022-01-31T22:16:36.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.545577+0000) 2022-01-31T22:16:36.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:36 smithi167 conmon[54076]: debug 2022-01-31T22:16:36.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.718089+0000) 2022-01-31T22:16:36.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:36 smithi167 conmon[60316]: debug 2022-01-31T22:16:36.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.471109+0000) 2022-01-31T22:16:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[41853]: debug 2022-01-31T22:16:36.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.420274+0000) 2022-01-31T22:16:37.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:36 smithi167 conmon[49112]: debug 2022-01-31T22:16:36.951+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.952436+0000) 2022-01-31T22:16:37.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:36 smithi167 conmon[54076]: debug 2022-01-31T22:16:36.952+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.953434+0000) 2022-01-31T22:16:37.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:36 smithi167 conmon[60316]: debug 2022-01-31T22:16:36.951+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.952202+0000) 2022-01-31T22:16:37.212 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[35325]: debug 2022-01-31T22:16:36.970+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196539 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:37.213 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[41853]: debug 2022-01-31T22:16:36.951+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.953506+0000) 2022-01-31T22:16:37.214 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[46715]: debug 2022-01-31T22:16:36.950+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.952352+0000) 2022-01-31T22:16:37.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:36 smithi171 conmon[51620]: debug 2022-01-31T22:16:36.949+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:36.951405+0000) 2022-01-31T22:16:37.215 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:37 smithi171 conmon[51620]: debug 2022-01-31T22:16:37.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:37.071323+0000) 2022-01-31T22:16:37.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:37 smithi171 conmon[46715]: debug 2022-01-31T22:16:37.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:37.213008+0000) 2022-01-31T22:16:37.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:37 smithi171 conmon[41853]: debug 2022-01-31T22:16:37.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:37.420467+0000) 2022-01-31T22:16:37.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:37 smithi167 conmon[49112]: debug 2022-01-31T22:16:37.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:37.545759+0000) 2022-01-31T22:16:37.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:37 smithi167 conmon[54076]: debug 2022-01-31T22:16:37.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:37.718262+0000) 2022-01-31T22:16:37.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:37 smithi167 conmon[60316]: debug 2022-01-31T22:16:37.470+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:37.471329+0000) 2022-01-31T22:16:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:38 smithi171 conmon[46715]: debug 2022-01-31T22:16:38.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:38.213147+0000) 2022-01-31T22:16:38.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:38 smithi171 conmon[51620]: debug 2022-01-31T22:16:38.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:38.071408+0000) 2022-01-31T22:16:38.764 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:38 smithi167 conmon[49112]: debug 2022-01-31T22:16:38.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:38.545919+0000) 2022-01-31T22:16:38.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:38 smithi167 conmon[54076]: debug 2022-01-31T22:16:38.717+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:38.718409+0000) 2022-01-31T22:16:38.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:38 smithi167 conmon[60316]: debug 2022-01-31T22:16:38.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:38.471463+0000) 2022-01-31T22:16:38.825 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:38 smithi171 conmon[41853]: debug 2022-01-31T22:16:38.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:38.420621+0000) 2022-01-31T22:16:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:38 smithi171 conmon[35325]: debug 2022-01-31T22:16:38.824+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:16:39.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:39 smithi171 conmon[51620]: debug 2022-01-31T22:16:39.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:39.071574+0000) 2022-01-31T22:16:39.420 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:39 smithi171 conmon[46715]: debug 2022-01-31T22:16:39.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:39.213303+0000) 2022-01-31T22:16:39.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:39 smithi167 conmon[49112]: debug 2022-01-31T22:16:39.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:39.546141+0000) 2022-01-31T22:16:39.765 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:39 smithi167 conmon[54076]: debug 2022-01-31T22:16:39.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:39.718606+0000) 2022-01-31T22:16:39.766 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:39 smithi167 conmon[60316]: debug 2022-01-31T22:16:39.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:39.471577+0000) 2022-01-31T22:16:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:39 smithi171 conmon[41853]: debug 2022-01-31T22:16:39.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:39.420802+0000) 2022-01-31T22:16:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:40 smithi171 conmon[46715]: debug 2022-01-31T22:16:40.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:40.213476+0000) 2022-01-31T22:16:40.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:40 smithi171 conmon[51620]: debug 2022-01-31T22:16:40.070+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:40.071766+0000) 2022-01-31T22:16:40.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:40 smithi167 conmon[49112]: debug 2022-01-31T22:16:40.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:40.546336+0000) 2022-01-31T22:16:40.719 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:40 smithi167 conmon[60316]: debug 2022-01-31T22:16:40.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:40.471742+0000) 2022-01-31T22:16:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:40 smithi171 conmon[41853]: debug 2022-01-31T22:16:40.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:40.420965+0000) 2022-01-31T22:16:41.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:40 smithi167 conmon[54076]: debug 2022-01-31T22:16:40.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:40.718799+0000) 2022-01-31T22:16:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[46715]: debug 2022-01-31T22:16:41.212+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.213682+0000) 2022-01-31T22:16:41.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[51620]: debug 2022-01-31T22:16:41.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.071950+0000) 2022-01-31T22:16:41.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:41 smithi167 conmon[49112]: debug 2022-01-31T22:16:41.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.546532+0000) 2022-01-31T22:16:41.719 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:41 smithi167 conmon[60316]: debug 2022-01-31T22:16:41.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.471965+0000) 2022-01-31T22:16:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[41853]: debug 2022-01-31T22:16:41.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.421107+0000) 2022-01-31T22:16:41.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:41 smithi167 conmon[54076]: debug 2022-01-31T22:16:41.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.719004+0000) 2022-01-31T22:16:42.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[35325]: debug 2022-01-31T22:16:41.994+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196649 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:42.359 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[41853]: debug 2022-01-31T22:16:41.972+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.974396+0000) 2022-01-31T22:16:42.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[46715]: debug 2022-01-31T22:16:41.972+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.974679+0000) 2022-01-31T22:16:42.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:42 smithi171 conmon[46715]: debug 2022-01-31T22:16:42.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:42.213843+0000) 2022-01-31T22:16:42.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:41 smithi171 conmon[51620]: debug 2022-01-31T22:16:41.973+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.975680+0000) 2022-01-31T22:16:42.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:42 smithi171 conmon[51620]: debug 2022-01-31T22:16:42.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:42.072162+0000) 2022-01-31T22:16:42.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:41 smithi167 conmon[49112]: debug 2022-01-31T22:16:41.975+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.975774+0000) 2022-01-31T22:16:42.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:41 smithi167 conmon[54076]: debug 2022-01-31T22:16:41.974+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.974918+0000) 2022-01-31T22:16:42.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:41 smithi167 conmon[60316]: debug 2022-01-31T22:16:41.974+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:41.975394+0000) 2022-01-31T22:16:42.718 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:42 smithi167 conmon[60316]: debug 2022-01-31T22:16:42.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:42.472120+0000) 2022-01-31T22:16:42.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:42 smithi167 conmon[49112]: debug 2022-01-31T22:16:42.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:42.546690+0000) 2022-01-31T22:16:42.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:42 smithi171 conmon[41853]: debug 2022-01-31T22:16:42.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:42.421267+0000) 2022-01-31T22:16:43.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:42 smithi167 conmon[54076]: debug 2022-01-31T22:16:42.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:42.719208+0000) 2022-01-31T22:16:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:43 smithi171 conmon[46715]: debug 2022-01-31T22:16:43.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:43.213955+0000) 2022-01-31T22:16:43.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:43 smithi171 conmon[51620]: debug 2022-01-31T22:16:43.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:43.072309+0000) 2022-01-31T22:16:43.718 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:43 smithi167 conmon[49112]: debug 2022-01-31T22:16:43.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:43.546821+0000) 2022-01-31T22:16:43.719 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:43 smithi167 conmon[60316]: debug 2022-01-31T22:16:43.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:43.472300+0000) 2022-01-31T22:16:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:43 smithi171 conmon[41853]: debug 2022-01-31T22:16:43.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:43.421432+0000) 2022-01-31T22:16:44.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:43 smithi167 conmon[54076]: debug 2022-01-31T22:16:43.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:43.719321+0000) 2022-01-31T22:16:44.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:44 smithi171 conmon[46715]: debug 2022-01-31T22:16:44.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:44.214126+0000) 2022-01-31T22:16:44.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:44 smithi171 conmon[51620]: debug 2022-01-31T22:16:44.071+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:44.072536+0000) 2022-01-31T22:16:44.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:44 smithi167 conmon[49112]: debug 2022-01-31T22:16:44.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:44.547012+0000) 2022-01-31T22:16:44.719 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:44 smithi167 conmon[60316]: debug 2022-01-31T22:16:44.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:44.472473+0000) 2022-01-31T22:16:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:44 smithi171 conmon[41853]: debug 2022-01-31T22:16:44.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:44.421630+0000) 2022-01-31T22:16:45.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:44 smithi167 conmon[54076]: debug 2022-01-31T22:16:44.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:44.719514+0000) 2022-01-31T22:16:45.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:45 smithi171 conmon[46715]: debug 2022-01-31T22:16:45.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:45.214332+0000) 2022-01-31T22:16:45.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:45 smithi171 conmon[51620]: debug 2022-01-31T22:16:45.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:45.072773+0000) 2022-01-31T22:16:45.719 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:45 smithi167 conmon[49112]: debug 2022-01-31T22:16:45.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:45.547180+0000) 2022-01-31T22:16:45.719 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:45 smithi167 conmon[60316]: debug 2022-01-31T22:16:45.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:45.472615+0000) 2022-01-31T22:16:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:45 smithi171 conmon[41853]: debug 2022-01-31T22:16:45.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:45.421817+0000) 2022-01-31T22:16:46.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:45 smithi167 conmon[54076]: debug 2022-01-31T22:16:45.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:45.719701+0000) 2022-01-31T22:16:46.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:46 smithi171 conmon[46715]: debug 2022-01-31T22:16:46.213+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.214539+0000) 2022-01-31T22:16:46.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:46 smithi171 conmon[51620]: debug 2022-01-31T22:16:46.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.072937+0000) 2022-01-31T22:16:46.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:46 smithi167 conmon[49112]: debug 2022-01-31T22:16:46.545+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.547334+0000) 2022-01-31T22:16:46.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:46 smithi167 conmon[54076]: debug 2022-01-31T22:16:46.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.719913+0000) 2022-01-31T22:16:46.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:46 smithi167 conmon[60316]: debug 2022-01-31T22:16:46.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.472725+0000) 2022-01-31T22:16:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:46 smithi171 conmon[41853]: debug 2022-01-31T22:16:46.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.421994+0000) 2022-01-31T22:16:47.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:46 smithi167 conmon[49112]: debug 2022-01-31T22:16:46.996+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.998381+0000) 2022-01-31T22:16:47.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:46 smithi167 conmon[54076]: debug 2022-01-31T22:16:46.997+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.998522+0000) 2022-01-31T22:16:47.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:47 smithi167 conmon[60316]: debug 2022-01-31T22:16:47.004+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.005705+0000) 2022-01-31T22:16:47.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:47 smithi171 conmon[35325]: debug 2022-01-31T22:16:47.025+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196762 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:47.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:46 smithi171 conmon[41853]: debug 2022-01-31T22:16:46.997+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.998524+0000) 2022-01-31T22:16:47.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:46 smithi171 conmon[46715]: debug 2022-01-31T22:16:46.997+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.997810+0000) 2022-01-31T22:16:47.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:47 smithi171 conmon[46715]: debug 2022-01-31T22:16:47.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.214719+0000) 2022-01-31T22:16:47.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:46 smithi171 conmon[51620]: debug 2022-01-31T22:16:46.998+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:46.998946+0000) 2022-01-31T22:16:47.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:47 smithi171 conmon[51620]: debug 2022-01-31T22:16:47.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.073076+0000) 2022-01-31T22:16:47.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:47 smithi167 conmon[49112]: debug 2022-01-31T22:16:47.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.547519+0000) 2022-01-31T22:16:47.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:47 smithi167 conmon[60316]: debug 2022-01-31T22:16:47.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.472845+0000) 2022-01-31T22:16:47.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:47 smithi167 conmon[54076]: debug 2022-01-31T22:16:47.719+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.720070+0000) 2022-01-31T22:16:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:47 smithi171 conmon[41853]: debug 2022-01-31T22:16:47.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:47.422148+0000) 2022-01-31T22:16:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:48 smithi171 conmon[46715]: debug 2022-01-31T22:16:48.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:48.214874+0000) 2022-01-31T22:16:48.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:48 smithi171 conmon[51620]: debug 2022-01-31T22:16:48.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:48.073220+0000) 2022-01-31T22:16:48.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:48 smithi167 conmon[49112]: debug 2022-01-31T22:16:48.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:48.547657+0000) 2022-01-31T22:16:48.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:48 smithi167 conmon[60316]: debug 2022-01-31T22:16:48.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:48.473003+0000) 2022-01-31T22:16:48.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:48 smithi167 conmon[54076]: debug 2022-01-31T22:16:48.718+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:48.720120+0000) 2022-01-31T22:16:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:48 smithi171 conmon[41853]: debug 2022-01-31T22:16:48.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:48.422338+0000) 2022-01-31T22:16:49.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:49 smithi171 conmon[46715]: debug 2022-01-31T22:16:49.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:49.215083+0000) 2022-01-31T22:16:49.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:49 smithi171 conmon[51620]: debug 2022-01-31T22:16:49.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:49.073427+0000) 2022-01-31T22:16:49.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:49 smithi167 conmon[49112]: debug 2022-01-31T22:16:49.546+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:49.547859+0000) 2022-01-31T22:16:49.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:49 smithi167 conmon[60316]: debug 2022-01-31T22:16:49.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:49.473192+0000) 2022-01-31T22:16:49.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:49 smithi167 conmon[54076]: debug 2022-01-31T22:16:49.719+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:49.720308+0000) 2022-01-31T22:16:49.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:49 smithi171 conmon[41853]: debug 2022-01-31T22:16:49.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:49.422511+0000) 2022-01-31T22:16:50.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:50 smithi171 conmon[46715]: debug 2022-01-31T22:16:50.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:50.215249+0000) 2022-01-31T22:16:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:50 smithi171 conmon[51620]: debug 2022-01-31T22:16:50.072+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:50.073594+0000) 2022-01-31T22:16:50.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:50 smithi167 conmon[49112]: debug 2022-01-31T22:16:50.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:50.548072+0000) 2022-01-31T22:16:50.767 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:50 smithi167 conmon[60316]: debug 2022-01-31T22:16:50.471+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:50.473405+0000) 2022-01-31T22:16:50.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:50 smithi167 conmon[54076]: debug 2022-01-31T22:16:50.719+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:50.720520+0000) 2022-01-31T22:16:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:50 smithi171 conmon[41853]: debug 2022-01-31T22:16:50.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:50.422689+0000) 2022-01-31T22:16:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:51 smithi171 conmon[46715]: debug 2022-01-31T22:16:51.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:51.215432+0000) 2022-01-31T22:16:51.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:51 smithi171 conmon[51620]: debug 2022-01-31T22:16:51.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:51.073794+0000) 2022-01-31T22:16:51.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:51 smithi167 conmon[49112]: debug 2022-01-31T22:16:51.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:51.548282+0000) 2022-01-31T22:16:51.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:51 smithi167 conmon[60316]: debug 2022-01-31T22:16:51.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:51.473633+0000) 2022-01-31T22:16:51.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:51 smithi167 conmon[54076]: debug 2022-01-31T22:16:51.719+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:51.720760+0000) 2022-01-31T22:16:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:51 smithi171 conmon[41853]: debug 2022-01-31T22:16:51.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:51.422867+0000) 2022-01-31T22:16:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[46715]: debug 2022-01-31T22:16:52.027+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.027787+0000) 2022-01-31T22:16:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[46715]: debug 2022-01-31T22:16:52.214+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.215624+0000) 2022-01-31T22:16:52.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[51620]: debug 2022-01-31T22:16:52.028+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.029053+0000) 2022-01-31T22:16:52.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[51620]: debug 2022-01-31T22:16:52.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.074012+0000) 2022-01-31T22:16:52.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[35325]: debug 2022-01-31T22:16:52.048+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196872 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:52.359 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[41853]: debug 2022-01-31T22:16:52.028+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.029184+0000) 2022-01-31T22:16:52.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[49112]: debug 2022-01-31T22:16:52.027+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.028875+0000) 2022-01-31T22:16:52.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[54076]: debug 2022-01-31T22:16:52.027+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.028535+0000) 2022-01-31T22:16:52.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[60316]: debug 2022-01-31T22:16:52.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[60316]: 2022-01-31T22:16:52.028+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.029587+0000) 2022-01-31T22:16:52.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[49112]: debug 2022-01-31T22:16:52.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.548443+0000) 2022-01-31T22:16:52.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[54076]: debug 2022-01-31T22:16:52.719+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.720893+0000) 2022-01-31T22:16:52.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:52 smithi167 conmon[60316]: debug 2022-01-31T22:16:52.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.473819+0000) 2022-01-31T22:16:52.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:52 smithi171 conmon[41853]: debug 2022-01-31T22:16:52.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:52.423081+0000) 2022-01-31T22:16:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:53 smithi171 conmon[46715]: debug 2022-01-31T22:16:53.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:53.215789+0000) 2022-01-31T22:16:53.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:53 smithi171 conmon[51620]: debug 2022-01-31T22:16:53.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:53.074117+0000) 2022-01-31T22:16:53.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:53 smithi167 conmon[49112]: debug 2022-01-31T22:16:53.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:53.548594+0000) 2022-01-31T22:16:53.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:53 smithi167 conmon[54076]: debug 2022-01-31T22:16:53.719+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:53.721000+0000) 2022-01-31T22:16:53.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:53 smithi167 conmon[60316]: debug 2022-01-31T22:16:53.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:53.474030+0000) 2022-01-31T22:16:53.825 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:53 smithi171 conmon[41853]: debug 2022-01-31T22:16:53.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:53.423261+0000) 2022-01-31T22:16:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:53 smithi171 conmon[35325]: debug 2022-01-31T22:16:53.825+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:16:54.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:54 smithi171 conmon[51620]: debug 2022-01-31T22:16:54.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:54.074307+0000) 2022-01-31T22:16:54.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:54 smithi171 conmon[46715]: debug 2022-01-31T22:16:54.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:54.215968+0000) 2022-01-31T22:16:54.767 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:54 smithi167 conmon[49112]: debug 2022-01-31T22:16:54.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:54.548775+0000) 2022-01-31T22:16:54.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:54 smithi167 conmon[54076]: debug 2022-01-31T22:16:54.720+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:54.721185+0000) 2022-01-31T22:16:54.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:54 smithi167 conmon[60316]: debug 2022-01-31T22:16:54.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:54.474230+0000) 2022-01-31T22:16:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:54 smithi171 conmon[41853]: debug 2022-01-31T22:16:54.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:54.423468+0000) 2022-01-31T22:16:55.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:55 smithi171 conmon[46715]: debug 2022-01-31T22:16:55.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:55.216121+0000) 2022-01-31T22:16:55.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:55 smithi171 conmon[51620]: debug 2022-01-31T22:16:55.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:55.074519+0000) 2022-01-31T22:16:55.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:55 smithi167 conmon[54076]: debug 2022-01-31T22:16:55.720+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:55.721369+0000) 2022-01-31T22:16:55.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:55 smithi167 conmon[60316]: debug 2022-01-31T22:16:55.472+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:55.474438+0000) 2022-01-31T22:16:55.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:55 smithi167 conmon[49112]: debug 2022-01-31T22:16:55.547+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:55.548958+0000) 2022-01-31T22:16:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:55 smithi171 conmon[41853]: debug 2022-01-31T22:16:55.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:55.423675+0000) 2022-01-31T22:16:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:56 smithi171 conmon[46715]: debug 2022-01-31T22:16:56.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:56.216242+0000) 2022-01-31T22:16:56.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:56 smithi171 conmon[51620]: debug 2022-01-31T22:16:56.073+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:56.074724+0000) 2022-01-31T22:16:56.767 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:56 smithi167 conmon[54076]: debug 2022-01-31T22:16:56.720+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:56.721547+0000) 2022-01-31T22:16:56.768 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:56 smithi167 conmon[60316]: debug 2022-01-31T22:16:56.473+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:56.474595+0000) 2022-01-31T22:16:56.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:56 smithi167 conmon[49112]: debug 2022-01-31T22:16:56.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:56.549169+0000) 2022-01-31T22:16:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:56 smithi171 conmon[41853]: debug 2022-01-31T22:16:56.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:56.423857+0000) 2022-01-31T22:16:57.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[35325]: debug 2022-01-31T22:16:57.071+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 196984 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:16:57.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[41853]: debug 2022-01-31T22:16:57.051+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.052077+0000) 2022-01-31T22:16:57.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[46715]: debug 2022-01-31T22:16:57.051+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.051945+0000) 2022-01-31T22:16:57.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[46715]: debug 2022-01-31T22:16:57.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.216434+0000) 2022-01-31T22:16:57.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[51620]: debug 2022-01-31T22:16:57.050+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.050806+0000) 2022-01-31T22:16:57.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[51620]: debug 2022-01-31T22:16:57.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.074884+0000) 2022-01-31T22:16:57.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:57 smithi167 conmon[49112]: debug 2022-01-31T22:16:57.051+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.052531+0000) 2022-01-31T22:16:57.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:57 smithi167 conmon[54076]: debug 2022-01-31T22:16:57.049+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.051191+0000) 2022-01-31T22:16:57.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:57 smithi167 conmon[60316]: debug 2022-01-31T22:16:57.050+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.051797+0000) 2022-01-31T22:16:57.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:57 smithi167 conmon[49112]: debug 2022-01-31T22:16:57.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.549332+0000) 2022-01-31T22:16:57.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:57 smithi167 conmon[60316]: debug 2022-01-31T22:16:57.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.474787+0000) 2022-01-31T22:16:57.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:57 smithi167 conmon[54076]: debug 2022-01-31T22:16:57.720+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.721729+0000) 2022-01-31T22:16:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:57 smithi171 conmon[41853]: debug 2022-01-31T22:16:57.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:57.424042+0000) 2022-01-31T22:16:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:58 smithi171 conmon[46715]: debug 2022-01-31T22:16:58.215+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:58.216633+0000) 2022-01-31T22:16:58.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:58 smithi171 conmon[51620]: debug 2022-01-31T22:16:58.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:58.075053+0000) 2022-01-31T22:16:58.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:58 smithi167 conmon[49112]: debug 2022-01-31T22:16:58.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:58.549454+0000) 2022-01-31T22:16:58.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:58 smithi167 conmon[54076]: debug 2022-01-31T22:16:58.720+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:58.721836+0000) 2022-01-31T22:16:58.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:58 smithi167 conmon[60316]: debug 2022-01-31T22:16:58.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:58.474950+0000) 2022-01-31T22:16:58.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:58 smithi171 conmon[41853]: debug 2022-01-31T22:16:58.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:58.424174+0000) 2022-01-31T22:16:59.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:16:59 smithi171 conmon[46715]: debug 2022-01-31T22:16:59.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:59.216840+0000) 2022-01-31T22:16:59.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:16:59 smithi171 conmon[51620]: debug 2022-01-31T22:16:59.074+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:59.075274+0000) 2022-01-31T22:16:59.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:16:59 smithi167 conmon[49112]: debug 2022-01-31T22:16:59.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:59.549627+0000) 2022-01-31T22:16:59.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:16:59 smithi167 conmon[54076]: debug 2022-01-31T22:16:59.721+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:59.721953+0000) 2022-01-31T22:16:59.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:16:59 smithi167 conmon[60316]: debug 2022-01-31T22:16:59.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:59.475137+0000) 2022-01-31T22:16:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:16:59 smithi171 conmon[41853]: debug 2022-01-31T22:16:59.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:16:59.424337+0000) 2022-01-31T22:17:00.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:00 smithi171 conmon[46715]: debug 2022-01-31T22:17:00.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:00.217022+0000) 2022-01-31T22:17:00.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:00 smithi171 conmon[51620]: debug 2022-01-31T22:17:00.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:00.075441+0000) 2022-01-31T22:17:00.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:00 smithi167 conmon[49112]: debug 2022-01-31T22:17:00.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:00.549798+0000) 2022-01-31T22:17:00.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:00 smithi167 conmon[60316]: debug 2022-01-31T22:17:00.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:00.475301+0000) 2022-01-31T22:17:00.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:00 smithi167 conmon[54076]: debug 2022-01-31T22:17:00.720+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:00.722169+0000) 2022-01-31T22:17:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:00 smithi171 conmon[41853]: debug 2022-01-31T22:17:00.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:00.424528+0000) 2022-01-31T22:17:01.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:01 smithi171 conmon[46715]: debug 2022-01-31T22:17:01.216+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:01.217245+0000) 2022-01-31T22:17:01.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:01 smithi171 conmon[51620]: debug 2022-01-31T22:17:01.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:01.075629+0000) 2022-01-31T22:17:01.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:01 smithi167 conmon[49112]: debug 2022-01-31T22:17:01.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:01.549994+0000) 2022-01-31T22:17:01.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:01 smithi167 conmon[54076]: debug 2022-01-31T22:17:01.721+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:01.722354+0000) 2022-01-31T22:17:01.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:01 smithi167 conmon[60316]: debug 2022-01-31T22:17:01.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:01.475465+0000) 2022-01-31T22:17:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:01 smithi171 conmon[41853]: debug 2022-01-31T22:17:01.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:01.424706+0000) 2022-01-31T22:17:02.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[41853]: debug 2022-01-31T22:17:02.073+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.074119+0000) 2022-01-31T22:17:02.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[35325]: debug 2022-01-31T22:17:02.094+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197095 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:02.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[46715]: debug 2022-01-31T22:17:02.074+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.075717+0000) 2022-01-31T22:17:02.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[46715]: debug 2022-01-31T22:17:02.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.217400+0000) 2022-01-31T22:17:02.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[51620]: debug 2022-01-31T22:17:02.074+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.075455+0000) 2022-01-31T22:17:02.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[51620]: debug 2022-01-31T22:17:02.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.075766+0000) 2022-01-31T22:17:02.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:02 smithi167 conmon[49112]: debug 2022-01-31T22:17:02.074+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.075789+0000) 2022-01-31T22:17:02.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:02 smithi167 conmon[54076]: debug 2022-01-31T22:17:02.073+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.074557+0000) 2022-01-31T22:17:02.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:02 smithi167 conmon[60316]: debug 2022-01-31T22:17:02.073+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.075290+0000) 2022-01-31T22:17:02.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:02 smithi167 conmon[49112]: debug 2022-01-31T22:17:02.548+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.550195+0000) 2022-01-31T22:17:02.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:02 smithi167 conmon[60316]: debug 2022-01-31T22:17:02.474+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.475629+0000) 2022-01-31T22:17:02.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:02 smithi167 conmon[54076]: debug 2022-01-31T22:17:02.721+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.722543+0000) 2022-01-31T22:17:02.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:02 smithi171 conmon[41853]: debug 2022-01-31T22:17:02.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:02.424910+0000) 2022-01-31T22:17:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:03 smithi171 conmon[46715]: debug 2022-01-31T22:17:03.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:03.217539+0000) 2022-01-31T22:17:03.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:03 smithi171 conmon[51620]: debug 2022-01-31T22:17:03.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:03.075882+0000) 2022-01-31T22:17:03.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:03 smithi167 conmon[49112]: debug 2022-01-31T22:17:03.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:03.550348+0000) 2022-01-31T22:17:03.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:03 smithi167 conmon[54076]: debug 2022-01-31T22:17:03.722+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:03.722709+0000) 2022-01-31T22:17:03.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:03 smithi167 conmon[60316]: debug 2022-01-31T22:17:03.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:03.475777+0000) 2022-01-31T22:17:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:03 smithi171 conmon[41853]: debug 2022-01-31T22:17:03.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:03.425074+0000) 2022-01-31T22:17:04.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:04 smithi171 conmon[46715]: debug 2022-01-31T22:17:04.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:04.217788+0000) 2022-01-31T22:17:04.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:04 smithi171 conmon[51620]: debug 2022-01-31T22:17:04.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:04.076034+0000) 2022-01-31T22:17:04.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:04 smithi167 conmon[49112]: debug 2022-01-31T22:17:04.549+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:04.550588+0000) 2022-01-31T22:17:04.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:04 smithi167 conmon[54076]: debug 2022-01-31T22:17:04.722+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:04.722900+0000) 2022-01-31T22:17:04.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:04 smithi167 conmon[60316]: debug 2022-01-31T22:17:04.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:04.475981+0000) 2022-01-31T22:17:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:04 smithi171 conmon[41853]: debug 2022-01-31T22:17:04.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:04.425235+0000) 2022-01-31T22:17:05.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:05 smithi171 conmon[46715]: debug 2022-01-31T22:17:05.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:05.217975+0000) 2022-01-31T22:17:05.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:05 smithi171 conmon[51620]: debug 2022-01-31T22:17:05.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:05.076234+0000) 2022-01-31T22:17:05.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:05 smithi167 conmon[49112]: debug 2022-01-31T22:17:05.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:05.550732+0000) 2022-01-31T22:17:05.770 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:05 smithi167 conmon[60316]: debug 2022-01-31T22:17:05.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:05.476178+0000) 2022-01-31T22:17:05.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:05 smithi167 conmon[54076]: debug 2022-01-31T22:17:05.721+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:05.723109+0000) 2022-01-31T22:17:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:05 smithi171 conmon[41853]: debug 2022-01-31T22:17:05.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:05.425386+0000) 2022-01-31T22:17:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:06 smithi171 conmon[46715]: debug 2022-01-31T22:17:06.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:06.218122+0000) 2022-01-31T22:17:06.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:06 smithi171 conmon[51620]: debug 2022-01-31T22:17:06.075+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:06.076376+0000) 2022-01-31T22:17:06.769 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:06 smithi167 conmon[49112]: debug 2022-01-31T22:17:06.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:06.550871+0000) 2022-01-31T22:17:06.770 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:06 smithi167 conmon[54076]: debug 2022-01-31T22:17:06.722+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:06.723320+0000) 2022-01-31T22:17:06.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:06 smithi167 conmon[60316]: debug 2022-01-31T22:17:06.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:06.476308+0000) 2022-01-31T22:17:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:06 smithi171 conmon[41853]: debug 2022-01-31T22:17:06.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:06.425540+0000) 2022-01-31T22:17:07.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[35325]: debug 2022-01-31T22:17:07.116+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197207 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:07.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[41853]: debug 2022-01-31T22:17:07.096+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.097237+0000) 2022-01-31T22:17:07.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[46715]: debug 2022-01-31T22:17:07.097+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.098228+0000) 2022-01-31T22:17:07.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[46715]: debug 2022-01-31T22:17:07.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.218286+0000) 2022-01-31T22:17:07.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[51620]: debug 2022-01-31T22:17:07.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.076558+0000) 2022-01-31T22:17:07.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[51620]: debug 2022-01-31T22:17:07.096+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.096942+0000) 2022-01-31T22:17:07.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:07 smithi167 conmon[49112]: debug 2022-01-31T22:17:07.096+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.098016+0000) 2022-01-31T22:17:07.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:07 smithi167 conmon[54076]: debug 2022-01-31T22:17:07.097+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.098586+0000) 2022-01-31T22:17:07.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:07 smithi167 conmon[60316]: debug 2022-01-31T22:17:07.096+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.097723+0000) 2022-01-31T22:17:07.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:07 smithi167 conmon[49112]: debug 2022-01-31T22:17:07.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.551050+0000) 2022-01-31T22:17:07.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:07 smithi167 conmon[60316]: debug 2022-01-31T22:17:07.475+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.476476+0000) 2022-01-31T22:17:07.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:07 smithi167 conmon[54076]: debug 2022-01-31T22:17:07.723+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.723526+0000) 2022-01-31T22:17:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:07 smithi171 conmon[41853]: debug 2022-01-31T22:17:07.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:07.425697+0000) 2022-01-31T22:17:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:08 smithi171 conmon[46715]: debug 2022-01-31T22:17:08.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:08.218422+0000) 2022-01-31T22:17:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:08 smithi171 conmon[51620]: debug 2022-01-31T22:17:08.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:08.076741+0000) 2022-01-31T22:17:08.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:08 smithi167 conmon[49112]: debug 2022-01-31T22:17:08.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:08.551227+0000) 2022-01-31T22:17:08.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:08 smithi167 conmon[54076]: debug 2022-01-31T22:17:08.723+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:08.723669+0000) 2022-01-31T22:17:08.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:08 smithi167 conmon[60316]: debug 2022-01-31T22:17:08.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:08.476643+0000) 2022-01-31T22:17:08.826 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:08 smithi171 conmon[41853]: debug 2022-01-31T22:17:08.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:08.425841+0000) 2022-01-31T22:17:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:08 smithi171 conmon[35325]: debug 2022-01-31T22:17:08.826+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:17:09.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:09 smithi171 conmon[51620]: debug 2022-01-31T22:17:09.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:09.076883+0000) 2022-01-31T22:17:09.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:09 smithi171 conmon[46715]: debug 2022-01-31T22:17:09.217+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:09.218613+0000) 2022-01-31T22:17:09.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:09 smithi167 conmon[49112]: debug 2022-01-31T22:17:09.550+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:09.551421+0000) 2022-01-31T22:17:09.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:09 smithi167 conmon[54076]: debug 2022-01-31T22:17:09.723+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:09.723789+0000) 2022-01-31T22:17:09.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:09 smithi167 conmon[60316]: debug 2022-01-31T22:17:09.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:09.476822+0000) 2022-01-31T22:17:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:09 smithi171 conmon[41853]: debug 2022-01-31T22:17:09.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:09.426033+0000) 2022-01-31T22:17:10.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:10 smithi171 conmon[46715]: debug 2022-01-31T22:17:10.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:10.218804+0000) 2022-01-31T22:17:10.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:10 smithi171 conmon[51620]: debug 2022-01-31T22:17:10.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:10.077037+0000) 2022-01-31T22:17:10.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:10 smithi167 conmon[49112]: debug 2022-01-31T22:17:10.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:10.551618+0000) 2022-01-31T22:17:10.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:10 smithi167 conmon[60316]: debug 2022-01-31T22:17:10.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:10.476954+0000) 2022-01-31T22:17:10.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:10 smithi167 conmon[54076]: debug 2022-01-31T22:17:10.723+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:10.723974+0000) 2022-01-31T22:17:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:10 smithi171 conmon[41853]: debug 2022-01-31T22:17:10.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:10.426186+0000) 2022-01-31T22:17:11.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:11 smithi171 conmon[46715]: debug 2022-01-31T22:17:11.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:11.218959+0000) 2022-01-31T22:17:11.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:11 smithi171 conmon[51620]: debug 2022-01-31T22:17:11.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:11.077227+0000) 2022-01-31T22:17:11.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:11 smithi167 conmon[49112]: debug 2022-01-31T22:17:11.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:11.551776+0000) 2022-01-31T22:17:11.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:11 smithi167 conmon[54076]: debug 2022-01-31T22:17:11.723+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:11.724175+0000) 2022-01-31T22:17:11.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:11 smithi167 conmon[60316]: debug 2022-01-31T22:17:11.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:11.477137+0000) 2022-01-31T22:17:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:11 smithi171 conmon[41853]: debug 2022-01-31T22:17:11.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:11.426359+0000) 2022-01-31T22:17:12.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[35325]: debug 2022-01-31T22:17:12.139+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197318 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:12.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[41853]: debug 2022-01-31T22:17:12.119+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.119932+0000) 2022-01-31T22:17:12.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[46715]: debug 2022-01-31T22:17:12.120+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.121244+0000) 2022-01-31T22:17:12.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[46715]: debug 2022-01-31T22:17:12.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.219140+0000) 2022-01-31T22:17:12.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[51620]: debug 2022-01-31T22:17:12.076+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.077380+0000) 2022-01-31T22:17:12.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[51620]: debug 2022-01-31T22:17:12.119+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.120505+0000) 2022-01-31T22:17:12.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:12 smithi167 conmon[49112]: debug 2022-01-31T22:17:12.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.120030+0000) 2022-01-31T22:17:12.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:12 smithi167 conmon[54076]: debug 2022-01-31T22:17:12.120+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.120798+0000) 2022-01-31T22:17:12.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:12 smithi167 conmon[60316]: debug 2022-01-31T22:17:12.119+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.120399+0000) 2022-01-31T22:17:12.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:12 smithi167 conmon[54076]: debug 2022-01-31T22:17:12.723+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.724331+0000) 2022-01-31T22:17:12.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:12 smithi167 conmon[60316]: debug 2022-01-31T22:17:12.476+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.477297+0000) 2022-01-31T22:17:12.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:12 smithi167 conmon[49112]: debug 2022-01-31T22:17:12.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.551939+0000) 2022-01-31T22:17:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:12 smithi171 conmon[41853]: debug 2022-01-31T22:17:12.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:12.426542+0000) 2022-01-31T22:17:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:13 smithi171 conmon[46715]: debug 2022-01-31T22:17:13.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:13.219284+0000) 2022-01-31T22:17:13.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:13 smithi171 conmon[51620]: debug 2022-01-31T22:17:13.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:13.077549+0000) 2022-01-31T22:17:13.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:13 smithi167 conmon[54076]: debug 2022-01-31T22:17:13.724+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:13.724475+0000) 2022-01-31T22:17:13.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:13 smithi167 conmon[60316]: debug 2022-01-31T22:17:13.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:13.477458+0000) 2022-01-31T22:17:13.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:13 smithi167 conmon[49112]: debug 2022-01-31T22:17:13.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:13.552100+0000) 2022-01-31T22:17:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:13 smithi171 conmon[41853]: debug 2022-01-31T22:17:13.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:13.426737+0000) 2022-01-31T22:17:14.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:14 smithi171 conmon[46715]: debug 2022-01-31T22:17:14.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:14.219486+0000) 2022-01-31T22:17:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:14 smithi171 conmon[51620]: debug 2022-01-31T22:17:14.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:14.077704+0000) 2022-01-31T22:17:14.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:14 smithi167 conmon[49112]: debug 2022-01-31T22:17:14.551+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:14.552245+0000) 2022-01-31T22:17:14.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:14 smithi167 conmon[60316]: debug 2022-01-31T22:17:14.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:14.477660+0000) 2022-01-31T22:17:14.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:14 smithi167 conmon[54076]: debug 2022-01-31T22:17:14.724+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:14.724666+0000) 2022-01-31T22:17:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:14 smithi171 conmon[41853]: debug 2022-01-31T22:17:14.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:14.426934+0000) 2022-01-31T22:17:15.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:15 smithi171 conmon[46715]: debug 2022-01-31T22:17:15.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:15.219653+0000) 2022-01-31T22:17:15.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:15 smithi171 conmon[51620]: debug 2022-01-31T22:17:15.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:15.077893+0000) 2022-01-31T22:17:15.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:15 smithi167 conmon[49112]: debug 2022-01-31T22:17:15.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:15.552424+0000) 2022-01-31T22:17:15.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:15 smithi167 conmon[54076]: debug 2022-01-31T22:17:15.724+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:15.724827+0000) 2022-01-31T22:17:15.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:15 smithi167 conmon[60316]: debug 2022-01-31T22:17:15.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:15.477837+0000) 2022-01-31T22:17:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:15 smithi171 conmon[41853]: debug 2022-01-31T22:17:15.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:15.427093+0000) 2022-01-31T22:17:16.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:16 smithi171 conmon[46715]: debug 2022-01-31T22:17:16.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:16.219785+0000) 2022-01-31T22:17:16.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:16 smithi171 conmon[51620]: debug 2022-01-31T22:17:16.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:16.078077+0000) 2022-01-31T22:17:16.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:16 smithi167 conmon[49112]: debug 2022-01-31T22:17:16.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:16.552583+0000) 2022-01-31T22:17:16.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:16 smithi167 conmon[60316]: debug 2022-01-31T22:17:16.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:16.478002+0000) 2022-01-31T22:17:16.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:16 smithi167 conmon[54076]: debug 2022-01-31T22:17:16.724+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:16.724984+0000) 2022-01-31T22:17:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:16 smithi171 conmon[41853]: debug 2022-01-31T22:17:16.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:16.427284+0000) 2022-01-31T22:17:17.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[35325]: debug 2022-01-31T22:17:17.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197429 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:17.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[41853]: debug 2022-01-31T22:17:17.139+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.141565+0000) 2022-01-31T22:17:17.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[46715]: debug 2022-01-31T22:17:17.140+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.141939+0000) 2022-01-31T22:17:17.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[46715]: debug 2022-01-31T22:17:17.218+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.219941+0000) 2022-01-31T22:17:17.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[51620]: debug 2022-01-31T22:17:17.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.078263+0000) 2022-01-31T22:17:17.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[51620]: debug 2022-01-31T22:17:17.141+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.143371+0000) 2022-01-31T22:17:17.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:17 smithi167 conmon[49112]: debug 2022-01-31T22:17:17.141+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.142294+0000) 2022-01-31T22:17:17.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:17 smithi167 conmon[54076]: debug 2022-01-31T22:17:17.143+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.143524+0000) 2022-01-31T22:17:17.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:17 smithi167 conmon[60316]: debug 2022-01-31T22:17:17.142+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.142924+0000) 2022-01-31T22:17:17.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:17 smithi167 conmon[49112]: debug 2022-01-31T22:17:17.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.552787+0000) 2022-01-31T22:17:17.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:17 smithi167 conmon[60316]: debug 2022-01-31T22:17:17.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.478202+0000) 2022-01-31T22:17:17.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:17 smithi167 conmon[54076]: debug 2022-01-31T22:17:17.724+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.725171+0000) 2022-01-31T22:17:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:17 smithi171 conmon[41853]: debug 2022-01-31T22:17:17.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:17.427458+0000) 2022-01-31T22:17:18.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:18 smithi171 conmon[46715]: debug 2022-01-31T22:17:18.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:18.220096+0000) 2022-01-31T22:17:18.338 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:18 smithi171 conmon[51620]: debug 2022-01-31T22:17:18.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:18.078454+0000) 2022-01-31T22:17:18.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:18 smithi167 conmon[49112]: debug 2022-01-31T22:17:18.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:18.552935+0000) 2022-01-31T22:17:18.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:18 smithi167 conmon[54076]: debug 2022-01-31T22:17:18.724+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:18.725304+0000) 2022-01-31T22:17:18.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:18 smithi167 conmon[60316]: debug 2022-01-31T22:17:18.477+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:18.478355+0000) 2022-01-31T22:17:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:18 smithi171 conmon[41853]: debug 2022-01-31T22:17:18.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:18.427598+0000) 2022-01-31T22:17:19.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:19 smithi171 conmon[46715]: debug 2022-01-31T22:17:19.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:19.220253+0000) 2022-01-31T22:17:19.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:19 smithi171 conmon[51620]: debug 2022-01-31T22:17:19.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:19.078605+0000) 2022-01-31T22:17:19.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:19 smithi167 conmon[49112]: debug 2022-01-31T22:17:19.552+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:19.553160+0000) 2022-01-31T22:17:19.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:19 smithi167 conmon[54076]: debug 2022-01-31T22:17:19.725+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:19.725491+0000) 2022-01-31T22:17:19.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:19 smithi167 conmon[60316]: debug 2022-01-31T22:17:19.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:19.478516+0000) 2022-01-31T22:17:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:19 smithi171 conmon[41853]: debug 2022-01-31T22:17:19.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:19.427781+0000) 2022-01-31T22:17:20.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:20 smithi171 conmon[46715]: debug 2022-01-31T22:17:20.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:20.220439+0000) 2022-01-31T22:17:20.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:20 smithi171 conmon[51620]: debug 2022-01-31T22:17:20.077+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:20.078796+0000) 2022-01-31T22:17:20.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:20 smithi167 conmon[49112]: debug 2022-01-31T22:17:20.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:20.553316+0000) 2022-01-31T22:17:20.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:20 smithi167 conmon[54076]: debug 2022-01-31T22:17:20.725+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:20.725672+0000) 2022-01-31T22:17:20.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:20 smithi167 conmon[60316]: debug 2022-01-31T22:17:20.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:20.478696+0000) 2022-01-31T22:17:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:20 smithi171 conmon[41853]: debug 2022-01-31T22:17:20.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:20.427947+0000) 2022-01-31T22:17:21.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:21 smithi171 conmon[46715]: debug 2022-01-31T22:17:21.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:21.220621+0000) 2022-01-31T22:17:21.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:21 smithi171 conmon[51620]: debug 2022-01-31T22:17:21.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:21.078977+0000) 2022-01-31T22:17:21.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:21 smithi167 conmon[60316]: debug 2022-01-31T22:17:21.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:21.478849+0000) 2022-01-31T22:17:21.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:21 smithi167 conmon[49112]: debug 2022-01-31T22:17:21.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:21.553517+0000) 2022-01-31T22:17:21.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:21 smithi167 conmon[54076]: debug 2022-01-31T22:17:21.725+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:21.725824+0000) 2022-01-31T22:17:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:21 smithi171 conmon[41853]: debug 2022-01-31T22:17:21.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:21.428152+0000) 2022-01-31T22:17:22.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[35325]: debug 2022-01-31T22:17:22.184+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197540 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:22.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[41853]: debug 2022-01-31T22:17:22.163+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.165125+0000) 2022-01-31T22:17:22.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[46715]: debug 2022-01-31T22:17:22.163+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.165095+0000) 2022-01-31T22:17:22.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[46715]: debug 2022-01-31T22:17:22.219+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.220804+0000) 2022-01-31T22:17:22.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[51620]: debug 2022-01-31T22:17:22.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.079168+0000) 2022-01-31T22:17:22.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[51620]: debug 2022-01-31T22:17:22.163+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.165416+0000) 2022-01-31T22:17:22.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:22 smithi167 conmon[49112]: debug 2022-01-31T22:17:22.165+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.165834+0000) 2022-01-31T22:17:22.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:22 smithi167 conmon[54076]: debug 2022-01-31T22:17:22.165+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.166212+0000) 2022-01-31T22:17:22.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:22 smithi167 conmon[60316]: debug 2022-01-31T22:17:22.165+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.166202+0000) 2022-01-31T22:17:22.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:22 smithi167 conmon[49112]: debug 2022-01-31T22:17:22.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.553697+0000) 2022-01-31T22:17:22.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:22 smithi167 conmon[54076]: debug 2022-01-31T22:17:22.725+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.725953+0000) 2022-01-31T22:17:22.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:22 smithi167 conmon[60316]: debug 2022-01-31T22:17:22.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.479035+0000) 2022-01-31T22:17:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:22 smithi171 conmon[41853]: debug 2022-01-31T22:17:22.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:22.428304+0000) 2022-01-31T22:17:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:23 smithi171 conmon[46715]: debug 2022-01-31T22:17:23.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:23.220956+0000) 2022-01-31T22:17:23.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:23 smithi171 conmon[51620]: debug 2022-01-31T22:17:23.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:23.079360+0000) 2022-01-31T22:17:23.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:23 smithi167 conmon[49112]: debug 2022-01-31T22:17:23.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:23.553837+0000) 2022-01-31T22:17:23.773 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:23 smithi167 conmon[54076]: debug 2022-01-31T22:17:23.725+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:23.726065+0000) 2022-01-31T22:17:23.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:23 smithi167 conmon[60316]: debug 2022-01-31T22:17:23.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:23.479182+0000) 2022-01-31T22:17:23.827 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:23 smithi171 conmon[41853]: debug 2022-01-31T22:17:23.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:23.428441+0000) 2022-01-31T22:17:24.079 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:23 smithi171 conmon[35325]: debug 2022-01-31T22:17:23.826+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:17:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:24 smithi171 conmon[46715]: debug 2022-01-31T22:17:24.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:24.221179+0000) 2022-01-31T22:17:24.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:24 smithi171 conmon[51620]: debug 2022-01-31T22:17:24.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:24.079546+0000) 2022-01-31T22:17:24.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:24 smithi167 conmon[49112]: debug 2022-01-31T22:17:24.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:24.553996+0000) 2022-01-31T22:17:24.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:24 smithi167 conmon[54076]: debug 2022-01-31T22:17:24.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:24.726257+0000) 2022-01-31T22:17:24.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:24 smithi167 conmon[60316]: debug 2022-01-31T22:17:24.478+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:24.479303+0000) 2022-01-31T22:17:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:24 smithi171 conmon[41853]: debug 2022-01-31T22:17:24.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:24.428616+0000) 2022-01-31T22:17:25.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:25 smithi171 conmon[51620]: debug 2022-01-31T22:17:25.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:25.079727+0000) 2022-01-31T22:17:25.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:25 smithi171 conmon[46715]: debug 2022-01-31T22:17:25.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:25.221372+0000) 2022-01-31T22:17:25.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:25 smithi167 conmon[49112]: debug 2022-01-31T22:17:25.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:25.554194+0000) 2022-01-31T22:17:25.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:25 smithi167 conmon[54076]: debug 2022-01-31T22:17:25.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:25.726421+0000) 2022-01-31T22:17:25.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:25 smithi167 conmon[60316]: debug 2022-01-31T22:17:25.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:25.479464+0000) 2022-01-31T22:17:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:25 smithi171 conmon[41853]: debug 2022-01-31T22:17:25.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:25.428823+0000) 2022-01-31T22:17:26.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:26 smithi171 conmon[46715]: debug 2022-01-31T22:17:26.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:26.221560+0000) 2022-01-31T22:17:26.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:26 smithi171 conmon[51620]: debug 2022-01-31T22:17:26.078+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:26.079920+0000) 2022-01-31T22:17:26.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:26 smithi167 conmon[49112]: debug 2022-01-31T22:17:26.553+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:26.554374+0000) 2022-01-31T22:17:26.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:26 smithi167 conmon[54076]: debug 2022-01-31T22:17:26.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:26.726602+0000) 2022-01-31T22:17:26.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:26 smithi167 conmon[60316]: debug 2022-01-31T22:17:26.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:26.479676+0000) 2022-01-31T22:17:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:26 smithi171 conmon[41853]: debug 2022-01-31T22:17:26.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:26.429022+0000) 2022-01-31T22:17:27.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[35325]: debug 2022-01-31T22:17:27.208+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197651 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:27.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[41853]: debug 2022-01-31T22:17:27.188+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.190156+0000) 2022-01-31T22:17:27.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[46715]: debug 2022-01-31T22:17:27.187+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.189346+0000) 2022-01-31T22:17:27.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[46715]: debug 2022-01-31T22:17:27.220+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.221707+0000) 2022-01-31T22:17:27.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[51620]: debug 2022-01-31T22:17:27.079+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.080122+0000) 2022-01-31T22:17:27.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[51620]: debug 2022-01-31T22:17:27.186+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.188453+0000) 2022-01-31T22:17:27.479 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:27 smithi167 conmon[49112]: debug 2022-01-31T22:17:27.189+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.189578+0000) 2022-01-31T22:17:27.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:27 smithi167 conmon[54076]: debug 2022-01-31T22:17:27.189+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.190048+0000) 2022-01-31T22:17:27.480 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:27 smithi167 conmon[60316]: debug 2022-01-31T22:17:27.189+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.189493+0000) 2022-01-31T22:17:27.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:27 smithi167 conmon[49112]: debug 2022-01-31T22:17:27.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.554560+0000) 2022-01-31T22:17:27.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:27 smithi167 conmon[54076]: debug 2022-01-31T22:17:27.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.726783+0000) 2022-01-31T22:17:27.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:27 smithi167 conmon[60316]: debug 2022-01-31T22:17:27.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.479830+0000) 2022-01-31T22:17:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:27 smithi171 conmon[41853]: debug 2022-01-31T22:17:27.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:27.429225+0000) 2022-01-31T22:17:28.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:28 smithi171 conmon[46715]: debug 2022-01-31T22:17:28.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:28.221827+0000) 2022-01-31T22:17:28.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:28 smithi171 conmon[51620]: debug 2022-01-31T22:17:28.079+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:28.080308+0000) 2022-01-31T22:17:28.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:28 smithi167 conmon[49112]: debug 2022-01-31T22:17:28.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:28.554716+0000) 2022-01-31T22:17:28.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:28 smithi167 conmon[54076]: debug 2022-01-31T22:17:28.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:28.726927+0000) 2022-01-31T22:17:28.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:28 smithi167 conmon[60316]: debug 2022-01-31T22:17:28.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:28.479983+0000) 2022-01-31T22:17:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:28 smithi171 conmon[41853]: debug 2022-01-31T22:17:28.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:28.429415+0000) 2022-01-31T22:17:29.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:29 smithi171 conmon[46715]: debug 2022-01-31T22:17:29.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:29.222012+0000) 2022-01-31T22:17:29.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:29 smithi171 conmon[51620]: debug 2022-01-31T22:17:29.079+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:29.080508+0000) 2022-01-31T22:17:29.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:29 smithi167 conmon[49112]: debug 2022-01-31T22:17:29.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:29.554884+0000) 2022-01-31T22:17:29.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:29 smithi167 conmon[54076]: debug 2022-01-31T22:17:29.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:29.727137+0000) 2022-01-31T22:17:29.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:29 smithi167 conmon[60316]: debug 2022-01-31T22:17:29.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:29.480166+0000) 2022-01-31T22:17:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:29 smithi171 conmon[41853]: debug 2022-01-31T22:17:29.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:29.429617+0000) 2022-01-31T22:17:30.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:30 smithi171 conmon[46715]: debug 2022-01-31T22:17:30.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:30.222136+0000) 2022-01-31T22:17:30.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:30 smithi171 conmon[51620]: debug 2022-01-31T22:17:30.079+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:30.080709+0000) 2022-01-31T22:17:30.689 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:30 smithi171 conmon[41853]: debug 2022-01-31T22:17:30.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:30.429796+0000) 2022-01-31T22:17:30.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:30 smithi167 conmon[49112]: debug 2022-01-31T22:17:30.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:30.555046+0000) 2022-01-31T22:17:30.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:30 smithi167 conmon[54076]: debug 2022-01-31T22:17:30.727+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:30.727328+0000) 2022-01-31T22:17:30.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:30 smithi167 conmon[60316]: debug 2022-01-31T22:17:30.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:30.480349+0000) 2022-01-31T22:17:31.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:31 smithi171 conmon[46715]: debug 2022-01-31T22:17:31.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:31.222298+0000) 2022-01-31T22:17:31.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:31 smithi171 conmon[51620]: debug 2022-01-31T22:17:31.080+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:31.080933+0000) 2022-01-31T22:17:31.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:31 smithi167 conmon[49112]: debug 2022-01-31T22:17:31.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:31.555235+0000) 2022-01-31T22:17:31.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:31 smithi167 conmon[54076]: debug 2022-01-31T22:17:31.727+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:31.727485+0000) 2022-01-31T22:17:31.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:31 smithi167 conmon[60316]: debug 2022-01-31T22:17:31.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:31.480561+0000) 2022-01-31T22:17:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:31 smithi171 conmon[41853]: debug 2022-01-31T22:17:31.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:31.429953+0000) 2022-01-31T22:17:32.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[35325]: debug 2022-01-31T22:17:32.231+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197762 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:32.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[41853]: debug 2022-01-31T22:17:32.211+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.212672+0000) 2022-01-31T22:17:32.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[46715]: debug 2022-01-31T22:17:32.210+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.210904+0000) 2022-01-31T22:17:32.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[46715]: debug 2022-01-31T22:17:32.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.222463+0000) 2022-01-31T22:17:32.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[51620]: debug 2022-01-31T22:17:32.080+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.081049+0000) 2022-01-31T22:17:32.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[51620]: debug 2022-01-31T22:17:32.209+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.210679+0000) 2022-01-31T22:17:32.480 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:32 smithi167 conmon[49112]: debug 2022-01-31T22:17:32.211+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.211612+0000) 2022-01-31T22:17:32.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:32 smithi167 conmon[54076]: debug 2022-01-31T22:17:32.211+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.211600+0000) 2022-01-31T22:17:32.482 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:32 smithi167 conmon[60316]: debug 2022-01-31T22:17:32.212+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.212597+0000) 2022-01-31T22:17:32.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:32 smithi167 conmon[49112]: debug 2022-01-31T22:17:32.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.555456+0000) 2022-01-31T22:17:32.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:32 smithi167 conmon[54076]: debug 2022-01-31T22:17:32.727+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.727622+0000) 2022-01-31T22:17:32.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:32 smithi167 conmon[60316]: debug 2022-01-31T22:17:32.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.480785+0000) 2022-01-31T22:17:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:32 smithi171 conmon[41853]: debug 2022-01-31T22:17:32.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:32.430104+0000) 2022-01-31T22:17:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:33 smithi171 conmon[46715]: debug 2022-01-31T22:17:33.221+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:33.222642+0000) 2022-01-31T22:17:33.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:33 smithi171 conmon[51620]: debug 2022-01-31T22:17:33.080+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:33.081240+0000) 2022-01-31T22:17:33.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:33 smithi167 conmon[49112]: debug 2022-01-31T22:17:33.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:33.555600+0000) 2022-01-31T22:17:33.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:33 smithi167 conmon[54076]: debug 2022-01-31T22:17:33.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:33.727767+0000) 2022-01-31T22:17:33.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:33 smithi167 conmon[60316]: debug 2022-01-31T22:17:33.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:33.480964+0000) 2022-01-31T22:17:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:33 smithi171 conmon[41853]: debug 2022-01-31T22:17:33.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:33.430288+0000) 2022-01-31T22:17:34.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:34 smithi171 conmon[46715]: debug 2022-01-31T22:17:34.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:34.222893+0000) 2022-01-31T22:17:34.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:34 smithi171 conmon[51620]: debug 2022-01-31T22:17:34.080+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:34.081390+0000) 2022-01-31T22:17:34.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:34 smithi167 conmon[49112]: debug 2022-01-31T22:17:34.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:34.555786+0000) 2022-01-31T22:17:34.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:34 smithi167 conmon[54076]: debug 2022-01-31T22:17:34.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:34.727970+0000) 2022-01-31T22:17:34.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:34 smithi167 conmon[60316]: debug 2022-01-31T22:17:34.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:34.481160+0000) 2022-01-31T22:17:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:34 smithi171 conmon[41853]: debug 2022-01-31T22:17:34.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:34.430449+0000) 2022-01-31T22:17:35.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:35 smithi171 conmon[46715]: debug 2022-01-31T22:17:35.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:35.223051+0000) 2022-01-31T22:17:35.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:35 smithi171 conmon[51620]: debug 2022-01-31T22:17:35.080+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:35.081580+0000) 2022-01-31T22:17:35.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:35 smithi167 conmon[49112]: debug 2022-01-31T22:17:35.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:35.555963+0000) 2022-01-31T22:17:35.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:35 smithi167 conmon[54076]: debug 2022-01-31T22:17:35.726+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:35.728157+0000) 2022-01-31T22:17:35.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:35 smithi167 conmon[60316]: debug 2022-01-31T22:17:35.479+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:35.481348+0000) 2022-01-31T22:17:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:35 smithi171 conmon[41853]: debug 2022-01-31T22:17:35.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:35.430610+0000) 2022-01-31T22:17:36.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:36 smithi171 conmon[46715]: debug 2022-01-31T22:17:36.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:36.223202+0000) 2022-01-31T22:17:36.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:36 smithi171 conmon[51620]: debug 2022-01-31T22:17:36.081+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:36.081801+0000) 2022-01-31T22:17:36.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:36 smithi167 conmon[49112]: debug 2022-01-31T22:17:36.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:36.556192+0000) 2022-01-31T22:17:36.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:36 smithi167 conmon[60316]: debug 2022-01-31T22:17:36.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:36.481578+0000) 2022-01-31T22:17:36.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:36 smithi167 conmon[54076]: debug 2022-01-31T22:17:36.727+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:36.728388+0000) 2022-01-31T22:17:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:36 smithi171 conmon[41853]: debug 2022-01-31T22:17:36.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:36.430888+0000) 2022-01-31T22:17:37.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[35325]: debug 2022-01-31T22:17:37.254+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197874 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:37.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[41853]: debug 2022-01-31T22:17:37.235+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.236529+0000) 2022-01-31T22:17:37.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[51620]: debug 2022-01-31T22:17:37.081+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.082032+0000) 2022-01-31T22:17:37.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[51620]: debug 2022-01-31T22:17:37.234+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.234877+0000) 2022-01-31T22:17:37.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[46715]: debug 2022-01-31T22:17:37.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.223388+0000) 2022-01-31T22:17:37.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[46715]: debug 2022-01-31T22:17:37.236+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.236896+0000) 2022-01-31T22:17:37.481 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:37 smithi167 conmon[49112]: debug 2022-01-31T22:17:37.234+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.235901+0000) 2022-01-31T22:17:37.482 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:37 smithi167 conmon[54076]: debug 2022-01-31T22:17:37.233+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.235399+0000) 2022-01-31T22:17:37.482 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:37 smithi167 conmon[60316]: debug 2022-01-31T22:17:37.235+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.237335+0000) 2022-01-31T22:17:37.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:37 smithi167 conmon[49112]: debug 2022-01-31T22:17:37.554+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.556396+0000) 2022-01-31T22:17:37.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:37 smithi167 conmon[60316]: debug 2022-01-31T22:17:37.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.481680+0000) 2022-01-31T22:17:37.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:37 smithi167 conmon[54076]: debug 2022-01-31T22:17:37.727+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.728548+0000) 2022-01-31T22:17:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:37 smithi171 conmon[41853]: debug 2022-01-31T22:17:37.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:37.431009+0000) 2022-01-31T22:17:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:38 smithi171 conmon[46715]: debug 2022-01-31T22:17:38.222+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:38.223562+0000) 2022-01-31T22:17:38.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:38 smithi171 conmon[51620]: debug 2022-01-31T22:17:38.081+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:38.082208+0000) 2022-01-31T22:17:38.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:38 smithi167 conmon[49112]: debug 2022-01-31T22:17:38.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:38.556550+0000) 2022-01-31T22:17:38.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:38 smithi167 conmon[54076]: debug 2022-01-31T22:17:38.727+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:38.728726+0000) 2022-01-31T22:17:38.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:38 smithi167 conmon[60316]: debug 2022-01-31T22:17:38.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:38.481815+0000) 2022-01-31T22:17:38.827 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:38 smithi171 conmon[41853]: debug 2022-01-31T22:17:38.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:38.431192+0000) 2022-01-31T22:17:39.081 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:38 smithi171 conmon[35325]: debug 2022-01-31T22:17:38.827+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:17:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:39 smithi171 conmon[46715]: debug 2022-01-31T22:17:39.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:39.223711+0000) 2022-01-31T22:17:39.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:39 smithi171 conmon[51620]: debug 2022-01-31T22:17:39.081+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:39.082358+0000) 2022-01-31T22:17:39.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:39 smithi167 conmon[49112]: debug 2022-01-31T22:17:39.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:39.556702+0000) 2022-01-31T22:17:39.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:39 smithi167 conmon[54076]: debug 2022-01-31T22:17:39.728+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:39.728913+0000) 2022-01-31T22:17:39.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:39 smithi167 conmon[60316]: debug 2022-01-31T22:17:39.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:39.481987+0000) 2022-01-31T22:17:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:39 smithi171 conmon[41853]: debug 2022-01-31T22:17:39.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:39.431417+0000) 2022-01-31T22:17:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:40 smithi171 conmon[46715]: debug 2022-01-31T22:17:40.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:40.223860+0000) 2022-01-31T22:17:40.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:40 smithi171 conmon[51620]: debug 2022-01-31T22:17:40.082+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:40.082555+0000) 2022-01-31T22:17:40.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:40 smithi167 conmon[54076]: debug 2022-01-31T22:17:40.728+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:40.729114+0000) 2022-01-31T22:17:40.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:40 smithi167 conmon[60316]: debug 2022-01-31T22:17:40.480+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:40.482167+0000) 2022-01-31T22:17:40.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:40 smithi167 conmon[49112]: debug 2022-01-31T22:17:40.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:40.556843+0000) 2022-01-31T22:17:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:40 smithi171 conmon[41853]: debug 2022-01-31T22:17:40.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:40.431641+0000) 2022-01-31T22:17:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:41 smithi171 conmon[46715]: debug 2022-01-31T22:17:41.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:41.224003+0000) 2022-01-31T22:17:41.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:41 smithi171 conmon[51620]: debug 2022-01-31T22:17:41.082+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:41.082735+0000) 2022-01-31T22:17:41.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:41 smithi167 conmon[54076]: debug 2022-01-31T22:17:41.728+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:41.729309+0000) 2022-01-31T22:17:41.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:41 smithi167 conmon[60316]: debug 2022-01-31T22:17:41.481+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:41.482365+0000) 2022-01-31T22:17:41.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:41 smithi167 conmon[49112]: debug 2022-01-31T22:17:41.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:41.557048+0000) 2022-01-31T22:17:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:41 smithi171 conmon[41853]: debug 2022-01-31T22:17:41.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:41.431829+0000) 2022-01-31T22:17:42.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[41853]: debug 2022-01-31T22:17:42.257+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.257809+0000) 2022-01-31T22:17:42.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[35325]: debug 2022-01-31T22:17:42.278+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 197984 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:42.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[46715]: debug 2022-01-31T22:17:42.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.224118+0000) 2022-01-31T22:17:42.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[46715]: debug 2022-01-31T22:17:42.257+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.258099+0000) 2022-01-31T22:17:42.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[51620]: debug 2022-01-31T22:17:42.082+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.082948+0000) 2022-01-31T22:17:42.362 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[51620]: debug 2022-01-31T22:17:42.257+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.258345+0000) 2022-01-31T22:17:42.482 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:42 smithi167 conmon[49112]: debug 2022-01-31T22:17:42.257+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.258850+0000) 2022-01-31T22:17:42.483 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:42 smithi167 conmon[54076]: debug 2022-01-31T22:17:42.258+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.259624+0000) 2022-01-31T22:17:42.483 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:42 smithi167 conmon[60316]: debug 2022-01-31T22:17:42.257+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.259245+0000) 2022-01-31T22:17:42.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:42 smithi167 conmon[49112]: debug 2022-01-31T22:17:42.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.557233+0000) 2022-01-31T22:17:42.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:42 smithi167 conmon[54076]: debug 2022-01-31T22:17:42.728+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.729462+0000) 2022-01-31T22:17:42.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:42 smithi167 conmon[60316]: debug 2022-01-31T22:17:42.481+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.482549+0000) 2022-01-31T22:17:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:42 smithi171 conmon[41853]: debug 2022-01-31T22:17:42.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:42.431978+0000) 2022-01-31T22:17:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:43 smithi171 conmon[46715]: debug 2022-01-31T22:17:43.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:43.224269+0000) 2022-01-31T22:17:43.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:43 smithi171 conmon[51620]: debug 2022-01-31T22:17:43.082+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:43.083128+0000) 2022-01-31T22:17:43.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:43 smithi167 conmon[49112]: debug 2022-01-31T22:17:43.555+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:43.557393+0000) 2022-01-31T22:17:43.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:43 smithi167 conmon[54076]: debug 2022-01-31T22:17:43.728+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:43.729614+0000) 2022-01-31T22:17:43.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:43 smithi167 conmon[60316]: debug 2022-01-31T22:17:43.481+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:43.482675+0000) 2022-01-31T22:17:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:43 smithi171 conmon[41853]: debug 2022-01-31T22:17:43.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:43.432119+0000) 2022-01-31T22:17:44.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:44 smithi171 conmon[46715]: debug 2022-01-31T22:17:44.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:44.224475+0000) 2022-01-31T22:17:44.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:44 smithi171 conmon[51620]: debug 2022-01-31T22:17:44.082+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:44.083310+0000) 2022-01-31T22:17:44.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:44 smithi167 conmon[49112]: debug 2022-01-31T22:17:44.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:44.557575+0000) 2022-01-31T22:17:44.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:44 smithi167 conmon[54076]: debug 2022-01-31T22:17:44.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:44.729788+0000) 2022-01-31T22:17:44.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:44 smithi167 conmon[60316]: debug 2022-01-31T22:17:44.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:44.482867+0000) 2022-01-31T22:17:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:44 smithi171 conmon[41853]: debug 2022-01-31T22:17:44.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:44.432313+0000) 2022-01-31T22:17:45.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:45 smithi171 conmon[46715]: debug 2022-01-31T22:17:45.223+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:45.224681+0000) 2022-01-31T22:17:45.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:45 smithi171 conmon[51620]: debug 2022-01-31T22:17:45.083+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:45.083497+0000) 2022-01-31T22:17:45.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:45 smithi167 conmon[49112]: debug 2022-01-31T22:17:45.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:45.557705+0000) 2022-01-31T22:17:45.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:45 smithi167 conmon[60316]: debug 2022-01-31T22:17:45.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:45.483020+0000) 2022-01-31T22:17:45.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:45 smithi167 conmon[54076]: debug 2022-01-31T22:17:45.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:45.729958+0000) 2022-01-31T22:17:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:45 smithi171 conmon[41853]: debug 2022-01-31T22:17:45.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:45.432425+0000) 2022-01-31T22:17:46.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:46 smithi171 conmon[46715]: debug 2022-01-31T22:17:46.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:46.224843+0000) 2022-01-31T22:17:46.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:46 smithi171 conmon[51620]: debug 2022-01-31T22:17:46.083+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:46.083675+0000) 2022-01-31T22:17:46.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:46 smithi167 conmon[49112]: debug 2022-01-31T22:17:46.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:46.557861+0000) 2022-01-31T22:17:46.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:46 smithi167 conmon[54076]: debug 2022-01-31T22:17:46.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:46.730125+0000) 2022-01-31T22:17:46.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:46 smithi167 conmon[60316]: debug 2022-01-31T22:17:46.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:46.483198+0000) 2022-01-31T22:17:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:46 smithi171 conmon[41853]: debug 2022-01-31T22:17:46.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:46.432603+0000) 2022-01-31T22:17:47.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[35325]: debug 2022-01-31T22:17:47.316+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198096 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:47.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[41853]: debug 2022-01-31T22:17:47.281+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.282062+0000) 2022-01-31T22:17:47.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[46715]: debug 2022-01-31T22:17:47.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.224962+0000) 2022-01-31T22:17:47.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[46715]: debug 2022-01-31T22:17:47.280+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.281399+0000) 2022-01-31T22:17:47.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[51620]: debug 2022-01-31T22:17:47.083+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.083838+0000) 2022-01-31T22:17:47.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[51620]: debug 2022-01-31T22:17:47.281+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.281870+0000) 2022-01-31T22:17:47.483 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:47 smithi167 conmon[49112]: debug 2022-01-31T22:17:47.279+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.281481+0000) 2022-01-31T22:17:47.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:47 smithi167 conmon[54076]: debug 2022-01-31T22:17:47.280+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.281910+0000) 2022-01-31T22:17:47.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:47 smithi167 conmon[60316]: debug 2022-01-31T22:17:47.281+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.282720+0000) 2022-01-31T22:17:47.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:47 smithi167 conmon[49112]: debug 2022-01-31T22:17:47.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.558045+0000) 2022-01-31T22:17:47.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:47 smithi167 conmon[60316]: debug 2022-01-31T22:17:47.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.483381+0000) 2022-01-31T22:17:47.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:47 smithi167 conmon[54076]: debug 2022-01-31T22:17:47.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.730336+0000) 2022-01-31T22:17:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:47 smithi171 conmon[41853]: debug 2022-01-31T22:17:47.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:47.432797+0000) 2022-01-31T22:17:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:48 smithi171 conmon[46715]: debug 2022-01-31T22:17:48.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:48.225108+0000) 2022-01-31T22:17:48.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:48 smithi171 conmon[51620]: debug 2022-01-31T22:17:48.083+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:48.084030+0000) 2022-01-31T22:17:48.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:48 smithi167 conmon[49112]: debug 2022-01-31T22:17:48.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:48.558190+0000) 2022-01-31T22:17:48.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:48 smithi167 conmon[54076]: debug 2022-01-31T22:17:48.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:48.730494+0000) 2022-01-31T22:17:48.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:48 smithi167 conmon[60316]: debug 2022-01-31T22:17:48.482+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:48.483531+0000) 2022-01-31T22:17:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:48 smithi171 conmon[41853]: debug 2022-01-31T22:17:48.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:48.432901+0000) 2022-01-31T22:17:49.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:49 smithi171 conmon[46715]: debug 2022-01-31T22:17:49.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:49.225208+0000) 2022-01-31T22:17:49.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:49 smithi171 conmon[51620]: debug 2022-01-31T22:17:49.083+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:49.084184+0000) 2022-01-31T22:17:49.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:49 smithi167 conmon[49112]: debug 2022-01-31T22:17:49.556+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:49.558410+0000) 2022-01-31T22:17:49.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:49 smithi167 conmon[54076]: debug 2022-01-31T22:17:49.730+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:49.730706+0000) 2022-01-31T22:17:49.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:49 smithi167 conmon[60316]: debug 2022-01-31T22:17:49.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:49.483739+0000) 2022-01-31T22:17:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:49 smithi171 conmon[41853]: debug 2022-01-31T22:17:49.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:49.433058+0000) 2022-01-31T22:17:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:50 smithi171 conmon[46715]: debug 2022-01-31T22:17:50.224+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:50.225365+0000) 2022-01-31T22:17:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:50 smithi171 conmon[51620]: debug 2022-01-31T22:17:50.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:50.084373+0000) 2022-01-31T22:17:50.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:50 smithi167 conmon[49112]: debug 2022-01-31T22:17:50.557+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:50.558598+0000) 2022-01-31T22:17:50.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:50 smithi167 conmon[54076]: debug 2022-01-31T22:17:50.730+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:50.730904+0000) 2022-01-31T22:17:50.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:50 smithi167 conmon[60316]: debug 2022-01-31T22:17:50.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:50.483956+0000) 2022-01-31T22:17:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:50 smithi171 conmon[41853]: debug 2022-01-31T22:17:50.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:50.433256+0000) 2022-01-31T22:17:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:51 smithi171 conmon[46715]: debug 2022-01-31T22:17:51.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:51.225576+0000) 2022-01-31T22:17:51.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:51 smithi171 conmon[51620]: debug 2022-01-31T22:17:51.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:51.084584+0000) 2022-01-31T22:17:51.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:51 smithi167 conmon[49112]: debug 2022-01-31T22:17:51.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:51.558782+0000) 2022-01-31T22:17:51.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:51 smithi167 conmon[54076]: debug 2022-01-31T22:17:51.730+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:51.731105+0000) 2022-01-31T22:17:51.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:51 smithi167 conmon[60316]: debug 2022-01-31T22:17:51.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:51.484133+0000) 2022-01-31T22:17:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:51 smithi171 conmon[41853]: debug 2022-01-31T22:17:51.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:51.433461+0000) 2022-01-31T22:17:52.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[41853]: debug 2022-01-31T22:17:52.319+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.320080+0000) 2022-01-31T22:17:52.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[46715]: debug 2022-01-31T22:17:52.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.225759+0000) 2022-01-31T22:17:52.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[46715]: debug 2022-01-31T22:17:52.318+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.319731+0000) 2022-01-31T22:17:52.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[51620]: debug 2022-01-31T22:17:52.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.084797+0000) 2022-01-31T22:17:52.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[51620]: debug 2022-01-31T22:17:52.319+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.319945+0000) 2022-01-31T22:17:52.483 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:52 smithi167 conmon[49112]: debug 2022-01-31T22:17:52.319+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.320679+0000) 2022-01-31T22:17:52.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:52 smithi167 conmon[54076]: debug 2022-01-31T22:17:52.318+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.319556+0000) 2022-01-31T22:17:52.485 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:52 smithi167 conmon[60316]: debug 2022-01-31T22:17:52.319+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.321292+0000) 2022-01-31T22:17:52.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[35325]: debug 2022-01-31T22:17:52.340+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198205 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:52.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:52 smithi171 conmon[41853]: debug 2022-01-31T22:17:52.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.433677+0000) 2022-01-31T22:17:52.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:52 smithi167 conmon[49112]: debug 2022-01-31T22:17:52.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.558986+0000) 2022-01-31T22:17:52.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:52 smithi167 conmon[60316]: debug 2022-01-31T22:17:52.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.484288+0000) 2022-01-31T22:17:52.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:52 smithi167 conmon[54076]: debug 2022-01-31T22:17:52.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:52.731254+0000) 2022-01-31T22:17:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:53 smithi171 conmon[46715]: debug 2022-01-31T22:17:53.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:53.225919+0000) 2022-01-31T22:17:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:53 smithi171 conmon[51620]: debug 2022-01-31T22:17:53.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:53.084983+0000) 2022-01-31T22:17:53.534 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:53 smithi167 conmon[60316]: debug 2022-01-31T22:17:53.483+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:53.484432+0000) 2022-01-31T22:17:53.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:53 smithi167 conmon[49112]: debug 2022-01-31T22:17:53.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:53.559139+0000) 2022-01-31T22:17:53.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:53 smithi167 conmon[54076]: debug 2022-01-31T22:17:53.729+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:53.731411+0000) 2022-01-31T22:17:53.828 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:53 smithi171 conmon[41853]: debug 2022-01-31T22:17:53.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:53.433832+0000) 2022-01-31T22:17:54.084 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:53 smithi171 conmon[35325]: debug 2022-01-31T22:17:53.828+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:17:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:54 smithi171 conmon[46715]: debug 2022-01-31T22:17:54.225+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:54.226102+0000) 2022-01-31T22:17:54.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:54 smithi171 conmon[51620]: debug 2022-01-31T22:17:54.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:54.085194+0000) 2022-01-31T22:17:54.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:54 smithi167 conmon[49112]: debug 2022-01-31T22:17:54.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:54.559325+0000) 2022-01-31T22:17:54.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:54 smithi167 conmon[54076]: debug 2022-01-31T22:17:54.730+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:54.731519+0000) 2022-01-31T22:17:54.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:54 smithi167 conmon[60316]: debug 2022-01-31T22:17:54.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:54.484651+0000) 2022-01-31T22:17:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:54 smithi171 conmon[41853]: debug 2022-01-31T22:17:54.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:54.433978+0000) 2022-01-31T22:17:55.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:55 smithi171 conmon[46715]: debug 2022-01-31T22:17:55.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:55.226258+0000) 2022-01-31T22:17:55.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:55 smithi171 conmon[51620]: debug 2022-01-31T22:17:55.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:55.085363+0000) 2022-01-31T22:17:55.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:55 smithi167 conmon[49112]: debug 2022-01-31T22:17:55.558+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:55.559568+0000) 2022-01-31T22:17:55.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:55 smithi167 conmon[54076]: debug 2022-01-31T22:17:55.730+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:55.731623+0000) 2022-01-31T22:17:55.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:55 smithi167 conmon[60316]: debug 2022-01-31T22:17:55.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:55.484861+0000) 2022-01-31T22:17:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:55 smithi171 conmon[41853]: debug 2022-01-31T22:17:55.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:55.434131+0000) 2022-01-31T22:17:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:56 smithi171 conmon[46715]: debug 2022-01-31T22:17:56.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:56.226441+0000) 2022-01-31T22:17:56.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:56 smithi171 conmon[51620]: debug 2022-01-31T22:17:56.084+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:56.085571+0000) 2022-01-31T22:17:56.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:56 smithi167 conmon[49112]: debug 2022-01-31T22:17:56.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:56.559757+0000) 2022-01-31T22:17:56.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:56 smithi167 conmon[54076]: debug 2022-01-31T22:17:56.730+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:56.731743+0000) 2022-01-31T22:17:56.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:56 smithi167 conmon[60316]: debug 2022-01-31T22:17:56.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:56.485087+0000) 2022-01-31T22:17:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:56 smithi171 conmon[41853]: debug 2022-01-31T22:17:56.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:56.434338+0000) 2022-01-31T22:17:57.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[46715]: debug 2022-01-31T22:17:57.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.226629+0000) 2022-01-31T22:17:57.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[51620]: debug 2022-01-31T22:17:57.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.085748+0000) 2022-01-31T22:17:57.485 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:57 smithi167 conmon[49112]: debug 2022-01-31T22:17:57.344+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.344590+0000) 2022-01-31T22:17:57.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:57 smithi167 conmon[54076]: debug 2022-01-31T22:17:57.343+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.343725+0000) 2022-01-31T22:17:57.486 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:57 smithi167 conmon[60316]: debug 2022-01-31T22:17:57.344+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.345172+0000) 2022-01-31T22:17:57.607 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[35325]: debug 2022-01-31T22:17:57.363+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198317 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:17:57.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[46715]: debug 2022-01-31T22:17:57.342+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.343585+0000) 2022-01-31T22:17:57.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[51620]: debug 2022-01-31T22:17:57.342+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.343049+0000) 2022-01-31T22:17:57.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[41853]: debug 2022-01-31T22:17:57.342+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.342891+0000) 2022-01-31T22:17:57.609 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:57 smithi171 conmon[41853]: debug 2022-01-31T22:17:57.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.434511+0000) 2022-01-31T22:17:57.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:57 smithi167 conmon[49112]: debug 2022-01-31T22:17:57.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.559942+0000) 2022-01-31T22:17:57.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:57 smithi167 conmon[54076]: debug 2022-01-31T22:17:57.731+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.731879+0000) 2022-01-31T22:17:57.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:57 smithi167 conmon[60316]: debug 2022-01-31T22:17:57.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:57.485244+0000) 2022-01-31T22:17:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:58 smithi171 conmon[46715]: debug 2022-01-31T22:17:58.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:58.226827+0000) 2022-01-31T22:17:58.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:58 smithi171 conmon[51620]: debug 2022-01-31T22:17:58.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:58.085935+0000) 2022-01-31T22:17:58.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:58 smithi167 conmon[49112]: debug 2022-01-31T22:17:58.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:58.560126+0000) 2022-01-31T22:17:58.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:58 smithi167 conmon[54076]: debug 2022-01-31T22:17:58.731+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:58.732024+0000) 2022-01-31T22:17:58.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:58 smithi167 conmon[60316]: debug 2022-01-31T22:17:58.484+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:58.485399+0000) 2022-01-31T22:17:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:58 smithi171 conmon[41853]: debug 2022-01-31T22:17:58.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:58.434669+0000) 2022-01-31T22:17:59.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:17:59 smithi171 conmon[46715]: debug 2022-01-31T22:17:59.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:59.226968+0000) 2022-01-31T22:17:59.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:17:59 smithi171 conmon[51620]: debug 2022-01-31T22:17:59.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:59.086035+0000) 2022-01-31T22:17:59.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:17:59 smithi167 conmon[49112]: debug 2022-01-31T22:17:59.559+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:59.560300+0000) 2022-01-31T22:17:59.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:17:59 smithi167 conmon[54076]: debug 2022-01-31T22:17:59.731+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:59.732181+0000) 2022-01-31T22:17:59.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:17:59 smithi167 conmon[60316]: debug 2022-01-31T22:17:59.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:59.485613+0000) 2022-01-31T22:17:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:17:59 smithi171 conmon[41853]: debug 2022-01-31T22:17:59.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:17:59.434848+0000) 2022-01-31T22:18:00.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:00 smithi171 conmon[46715]: debug 2022-01-31T22:18:00.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:00.227136+0000) 2022-01-31T22:18:00.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:00 smithi171 conmon[51620]: debug 2022-01-31T22:18:00.086+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:00.086257+0000) 2022-01-31T22:18:00.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:00 smithi167 conmon[49112]: debug 2022-01-31T22:18:00.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:00.560473+0000) 2022-01-31T22:18:00.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:00 smithi167 conmon[60316]: debug 2022-01-31T22:18:00.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:00.485804+0000) 2022-01-31T22:18:00.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:00 smithi167 conmon[54076]: debug 2022-01-31T22:18:00.731+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:00.732383+0000) 2022-01-31T22:18:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:00 smithi171 conmon[41853]: debug 2022-01-31T22:18:00.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:00.434979+0000) 2022-01-31T22:18:01.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:01 smithi171 conmon[51620]: debug 2022-01-31T22:18:01.086+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:01.086456+0000) 2022-01-31T22:18:01.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:01 smithi171 conmon[46715]: debug 2022-01-31T22:18:01.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:01.227296+0000) 2022-01-31T22:18:01.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:01 smithi167 conmon[49112]: debug 2022-01-31T22:18:01.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:01.560656+0000) 2022-01-31T22:18:01.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:01 smithi167 conmon[54076]: debug 2022-01-31T22:18:01.732+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:01.732573+0000) 2022-01-31T22:18:01.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:01 smithi167 conmon[60316]: debug 2022-01-31T22:18:01.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:01.485961+0000) 2022-01-31T22:18:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:01 smithi171 conmon[41853]: debug 2022-01-31T22:18:01.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:01.435082+0000) 2022-01-31T22:18:02.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[51620]: debug 2022-01-31T22:18:02.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.086681+0000) 2022-01-31T22:18:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[46715]: debug 2022-01-31T22:18:02.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.227447+0000) 2022-01-31T22:18:02.485 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:02 smithi167 conmon[49112]: debug 2022-01-31T22:18:02.367+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.367604+0000) 2022-01-31T22:18:02.486 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:02 smithi167 conmon[54076]: debug 2022-01-31T22:18:02.366+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.366953+0000) 2022-01-31T22:18:02.487 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:02 smithi167 conmon[60316]: debug 2022-01-31T22:18:02.367+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.367817+0000) 2022-01-31T22:18:02.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:02 smithi167 conmon[49112]: debug 2022-01-31T22:18:02.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.560848+0000) 2022-01-31T22:18:02.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:02 smithi167 conmon[54076]: debug 2022-01-31T22:18:02.732+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.732754+0000) 2022-01-31T22:18:02.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:02 smithi167 conmon[60316]: debug 2022-01-31T22:18:02.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.486138+0000) 2022-01-31T22:18:02.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[35325]: debug 2022-01-31T22:18:02.385+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198429 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:02.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[41853]: debug 2022-01-31T22:18:02.364+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.366573+0000) 2022-01-31T22:18:02.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[41853]: debug 2022-01-31T22:18:02.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.435180+0000) 2022-01-31T22:18:02.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[46715]: debug 2022-01-31T22:18:02.364+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.365891+0000) 2022-01-31T22:18:02.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:02 smithi171 conmon[51620]: debug 2022-01-31T22:18:02.364+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:02.366365+0000) 2022-01-31T22:18:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:03 smithi171 conmon[46715]: debug 2022-01-31T22:18:03.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:03.227610+0000) 2022-01-31T22:18:03.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:03 smithi171 conmon[51620]: debug 2022-01-31T22:18:03.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:03.086836+0000) 2022-01-31T22:18:03.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:03 smithi167 conmon[54076]: debug 2022-01-31T22:18:03.732+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:03.732888+0000) 2022-01-31T22:18:03.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:03 smithi167 conmon[60316]: debug 2022-01-31T22:18:03.485+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:03.486282+0000) 2022-01-31T22:18:03.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:03 smithi167 conmon[49112]: debug 2022-01-31T22:18:03.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:03.560982+0000) 2022-01-31T22:18:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:03 smithi171 conmon[41853]: debug 2022-01-31T22:18:03.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:03.435364+0000) 2022-01-31T22:18:04.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:04 smithi171 conmon[46715]: debug 2022-01-31T22:18:04.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:04.227795+0000) 2022-01-31T22:18:04.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:04 smithi171 conmon[51620]: debug 2022-01-31T22:18:04.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:04.087006+0000) 2022-01-31T22:18:04.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:04 smithi167 conmon[49112]: debug 2022-01-31T22:18:04.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:04.561129+0000) 2022-01-31T22:18:04.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:04 smithi167 conmon[54076]: debug 2022-01-31T22:18:04.732+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:04.733093+0000) 2022-01-31T22:18:04.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:04 smithi167 conmon[60316]: debug 2022-01-31T22:18:04.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:04.486470+0000) 2022-01-31T22:18:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:04 smithi171 conmon[41853]: debug 2022-01-31T22:18:04.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:04.435547+0000) 2022-01-31T22:18:05.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:05 smithi171 conmon[46715]: debug 2022-01-31T22:18:05.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:05.227945+0000) 2022-01-31T22:18:05.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:05 smithi171 conmon[51620]: debug 2022-01-31T22:18:05.085+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:05.087157+0000) 2022-01-31T22:18:05.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:05 smithi167 conmon[49112]: debug 2022-01-31T22:18:05.560+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:05.561312+0000) 2022-01-31T22:18:05.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:05 smithi167 conmon[54076]: debug 2022-01-31T22:18:05.732+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:05.733278+0000) 2022-01-31T22:18:05.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:05 smithi167 conmon[60316]: debug 2022-01-31T22:18:05.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:05.486683+0000) 2022-01-31T22:18:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:05 smithi171 conmon[41853]: debug 2022-01-31T22:18:05.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:05.435791+0000) 2022-01-31T22:18:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:06 smithi171 conmon[46715]: debug 2022-01-31T22:18:06.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:06.228138+0000) 2022-01-31T22:18:06.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:06 smithi171 conmon[51620]: debug 2022-01-31T22:18:06.086+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:06.087356+0000) 2022-01-31T22:18:06.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:06 smithi167 conmon[54076]: debug 2022-01-31T22:18:06.733+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:06.733464+0000) 2022-01-31T22:18:06.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:06 smithi167 conmon[49112]: debug 2022-01-31T22:18:06.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:06.561496+0000) 2022-01-31T22:18:06.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:06 smithi167 conmon[60316]: debug 2022-01-31T22:18:06.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:06.486868+0000) 2022-01-31T22:18:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:06 smithi171 conmon[41853]: debug 2022-01-31T22:18:06.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:06.435945+0000) 2022-01-31T22:18:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[46715]: debug 2022-01-31T22:18:07.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.228327+0000) 2022-01-31T22:18:07.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[51620]: debug 2022-01-31T22:18:07.086+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.087537+0000) 2022-01-31T22:18:07.486 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:07 smithi167 conmon[49112]: debug 2022-01-31T22:18:07.389+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.390211+0000) 2022-01-31T22:18:07.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:07 smithi167 conmon[54076]: debug 2022-01-31T22:18:07.389+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.390125+0000) 2022-01-31T22:18:07.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:07 smithi167 conmon[60316]: debug 2022-01-31T22:18:07.390+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.390559+0000) 2022-01-31T22:18:07.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:07 smithi167 conmon[49112]: debug 2022-01-31T22:18:07.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.561646+0000) 2022-01-31T22:18:07.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:07 smithi167 conmon[54076]: debug 2022-01-31T22:18:07.733+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.733660+0000) 2022-01-31T22:18:07.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:07 smithi167 conmon[60316]: debug 2022-01-31T22:18:07.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.487051+0000) 2022-01-31T22:18:07.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[35325]: debug 2022-01-31T22:18:07.408+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198536 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:07.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[46715]: debug 2022-01-31T22:18:07.387+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.389626+0000) 2022-01-31T22:18:07.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[51620]: debug 2022-01-31T22:18:07.388+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.390293+0000) 2022-01-31T22:18:07.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[41853]: debug 2022-01-31T22:18:07.387+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.389247+0000) 2022-01-31T22:18:07.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:07 smithi171 conmon[41853]: debug 2022-01-31T22:18:07.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:07.436059+0000) 2022-01-31T22:18:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:08 smithi171 conmon[46715]: debug 2022-01-31T22:18:08.226+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:08.228551+0000) 2022-01-31T22:18:08.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:08 smithi171 conmon[51620]: debug 2022-01-31T22:18:08.086+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:08.087733+0000) 2022-01-31T22:18:08.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:08 smithi167 conmon[49112]: debug 2022-01-31T22:18:08.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:08.561805+0000) 2022-01-31T22:18:08.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:08 smithi167 conmon[54076]: debug 2022-01-31T22:18:08.733+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:08.733785+0000) 2022-01-31T22:18:08.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:08 smithi167 conmon[60316]: debug 2022-01-31T22:18:08.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:08.487191+0000) 2022-01-31T22:18:08.829 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:08 smithi171 conmon[41853]: debug 2022-01-31T22:18:08.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:08.436152+0000) 2022-01-31T22:18:09.087 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:08 smithi171 conmon[35325]: debug 2022-01-31T22:18:08.828+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:18:09.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:09 smithi171 conmon[51620]: debug 2022-01-31T22:18:09.086+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:09.087867+0000) 2022-01-31T22:18:09.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:09 smithi171 conmon[46715]: debug 2022-01-31T22:18:09.227+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:09.228731+0000) 2022-01-31T22:18:09.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:09 smithi167 conmon[49112]: debug 2022-01-31T22:18:09.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:09.561971+0000) 2022-01-31T22:18:09.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:09 smithi167 conmon[54076]: debug 2022-01-31T22:18:09.733+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:09.733995+0000) 2022-01-31T22:18:09.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:09 smithi167 conmon[60316]: debug 2022-01-31T22:18:09.486+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:09.487346+0000) 2022-01-31T22:18:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:09 smithi171 conmon[41853]: debug 2022-01-31T22:18:09.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:09.436287+0000) 2022-01-31T22:18:10.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:10 smithi171 conmon[46715]: debug 2022-01-31T22:18:10.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:10.228960+0000) 2022-01-31T22:18:10.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:10 smithi171 conmon[51620]: debug 2022-01-31T22:18:10.087+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:10.088051+0000) 2022-01-31T22:18:10.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:10 smithi167 conmon[49112]: debug 2022-01-31T22:18:10.561+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:10.562172+0000) 2022-01-31T22:18:10.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:10 smithi167 conmon[54076]: debug 2022-01-31T22:18:10.733+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:10.734196+0000) 2022-01-31T22:18:10.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:10 smithi167 conmon[60316]: debug 2022-01-31T22:18:10.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:10.487496+0000) 2022-01-31T22:18:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:10 smithi171 conmon[41853]: debug 2022-01-31T22:18:10.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:10.436416+0000) 2022-01-31T22:18:11.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:11 smithi171 conmon[46715]: debug 2022-01-31T22:18:11.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:11.229112+0000) 2022-01-31T22:18:11.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:11 smithi171 conmon[51620]: debug 2022-01-31T22:18:11.087+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:11.088248+0000) 2022-01-31T22:18:11.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:11 smithi167 conmon[49112]: debug 2022-01-31T22:18:11.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:11.562359+0000) 2022-01-31T22:18:11.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:11 smithi167 conmon[54076]: debug 2022-01-31T22:18:11.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:11.734382+0000) 2022-01-31T22:18:11.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:11 smithi167 conmon[60316]: debug 2022-01-31T22:18:11.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:11.487698+0000) 2022-01-31T22:18:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:11 smithi171 conmon[41853]: debug 2022-01-31T22:18:11.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:11.436519+0000) 2022-01-31T22:18:12.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[46715]: debug 2022-01-31T22:18:12.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.229279+0000) 2022-01-31T22:18:12.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[51620]: debug 2022-01-31T22:18:12.087+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.088437+0000) 2022-01-31T22:18:12.487 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:12 smithi167 conmon[49112]: debug 2022-01-31T22:18:12.412+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.413090+0000) 2022-01-31T22:18:12.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:12 smithi167 conmon[54076]: debug 2022-01-31T22:18:12.412+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.412566+0000) 2022-01-31T22:18:12.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:12 smithi167 conmon[60316]: debug 2022-01-31T22:18:12.412+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.413270+0000) 2022-01-31T22:18:12.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:12 smithi167 conmon[49112]: debug 2022-01-31T22:18:12.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.562554+0000) 2022-01-31T22:18:12.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:12 smithi167 conmon[60316]: debug 2022-01-31T22:18:12.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.487899+0000) 2022-01-31T22:18:12.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:12 smithi167 conmon[54076]: debug 2022-01-31T22:18:12.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.734562+0000) 2022-01-31T22:18:12.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[35325]: debug 2022-01-31T22:18:12.431+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198648 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:12.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[46715]: debug 2022-01-31T22:18:12.409+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.411728+0000) 2022-01-31T22:18:12.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[51620]: debug 2022-01-31T22:18:12.410+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.412448+0000) 2022-01-31T22:18:12.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[41853]: debug 2022-01-31T22:18:12.410+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.411996+0000) 2022-01-31T22:18:12.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:12 smithi171 conmon[41853]: debug 2022-01-31T22:18:12.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:12.436628+0000) 2022-01-31T22:18:13.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:13 smithi171 conmon[51620]: debug 2022-01-31T22:18:13.087+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:13.088634+0000) 2022-01-31T22:18:13.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:13 smithi171 conmon[46715]: debug 2022-01-31T22:18:13.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:13.229486+0000) 2022-01-31T22:18:13.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:13 smithi167 conmon[49112]: debug 2022-01-31T22:18:13.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:13.562744+0000) 2022-01-31T22:18:13.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:13 smithi167 conmon[54076]: debug 2022-01-31T22:18:13.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:13.734663+0000) 2022-01-31T22:18:13.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:13 smithi167 conmon[60316]: debug 2022-01-31T22:18:13.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:13.488080+0000) 2022-01-31T22:18:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:13 smithi171 conmon[41853]: debug 2022-01-31T22:18:13.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:13.436819+0000) 2022-01-31T22:18:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:14 smithi171 conmon[51620]: debug 2022-01-31T22:18:14.088+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:14.088798+0000) 2022-01-31T22:18:14.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:14 smithi171 conmon[46715]: debug 2022-01-31T22:18:14.228+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:14.229669+0000) 2022-01-31T22:18:14.722 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:14 smithi171 conmon[41853]: debug 2022-01-31T22:18:14.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:14.436971+0000) 2022-01-31T22:18:14.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:14 smithi167 conmon[49112]: debug 2022-01-31T22:18:14.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:14.562932+0000) 2022-01-31T22:18:14.782 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:14 smithi167 conmon[54076]: debug 2022-01-31T22:18:14.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:14.734793+0000) 2022-01-31T22:18:14.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:14 smithi167 conmon[60316]: debug 2022-01-31T22:18:14.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:14.488284+0000) 2022-01-31T22:18:15.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:15 smithi171 conmon[46715]: debug 2022-01-31T22:18:15.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:15.229855+0000) 2022-01-31T22:18:15.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:15 smithi171 conmon[51620]: debug 2022-01-31T22:18:15.088+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:15.088961+0000) 2022-01-31T22:18:15.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:15 smithi167 conmon[60316]: debug 2022-01-31T22:18:15.487+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:15.488461+0000) 2022-01-31T22:18:15.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:15 smithi167 conmon[49112]: debug 2022-01-31T22:18:15.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:15.563143+0000) 2022-01-31T22:18:15.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:15 smithi167 conmon[54076]: debug 2022-01-31T22:18:15.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:15.734946+0000) 2022-01-31T22:18:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:15 smithi171 conmon[41853]: debug 2022-01-31T22:18:15.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:15.437123+0000) 2022-01-31T22:18:16.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:16 smithi171 conmon[46715]: debug 2022-01-31T22:18:16.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:16.230030+0000) 2022-01-31T22:18:16.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:16 smithi171 conmon[51620]: debug 2022-01-31T22:18:16.088+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:16.089148+0000) 2022-01-31T22:18:16.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:16 smithi167 conmon[49112]: debug 2022-01-31T22:18:16.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:16.563328+0000) 2022-01-31T22:18:16.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:16 smithi167 conmon[54076]: debug 2022-01-31T22:18:16.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:16.735092+0000) 2022-01-31T22:18:16.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:16 smithi167 conmon[60316]: debug 2022-01-31T22:18:16.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:16.488626+0000) 2022-01-31T22:18:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:16 smithi171 conmon[41853]: debug 2022-01-31T22:18:16.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:16.437306+0000) 2022-01-31T22:18:17.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[46715]: debug 2022-01-31T22:18:17.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.230183+0000) 2022-01-31T22:18:17.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[51620]: debug 2022-01-31T22:18:17.088+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.089311+0000) 2022-01-31T22:18:17.488 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[49112]: debug 2022-01-31T22:18:17.435+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.436321+0000) 2022-01-31T22:18:17.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[54076]: debug 2022-01-31T22:18:17.435+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.436089+0000) 2022-01-31T22:18:17.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[60316]: debug 2022-01-31T22:18:17.436+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.436532+0000) 2022-01-31T22:18:17.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[60316]: 2022-01-31T22:18:17.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[49112]: debug 2022-01-31T22:18:17.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.563550+0000) 2022-01-31T22:18:17.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[54076]: debug 2022-01-31T22:18:17.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.735305+0000) 2022-01-31T22:18:17.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:17 smithi167 conmon[60316]: debug 2022-01-31T22:18:17.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.488780+0000) 2022-01-31T22:18:17.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[35325]: debug 2022-01-31T22:18:17.455+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198758 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:17.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[46715]: debug 2022-01-31T22:18:17.434+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.434981+0000) 2022-01-31T22:18:17.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[51620]: debug 2022-01-31T22:18:17.435+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.436624+0000) 2022-01-31T22:18:17.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[41853]: debug 2022-01-31T22:18:17.434+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.435456+0000) 2022-01-31T22:18:17.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:17 smithi171 conmon[41853]: debug 2022-01-31T22:18:17.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:17.437433+0000) 2022-01-31T22:18:18.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:18 smithi171 conmon[51620]: debug 2022-01-31T22:18:18.088+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:18.089514+0000) 2022-01-31T22:18:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:18 smithi171 conmon[46715]: debug 2022-01-31T22:18:18.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:18.230382+0000) 2022-01-31T22:18:18.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:18 smithi167 conmon[49112]: debug 2022-01-31T22:18:18.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:18.563647+0000) 2022-01-31T22:18:18.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:18 smithi167 conmon[54076]: debug 2022-01-31T22:18:18.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:18.735442+0000) 2022-01-31T22:18:18.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:18 smithi167 conmon[60316]: debug 2022-01-31T22:18:18.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:18.488953+0000) 2022-01-31T22:18:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:18 smithi171 conmon[41853]: debug 2022-01-31T22:18:18.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:18.437555+0000) 2022-01-31T22:18:19.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:19 smithi171 conmon[46715]: debug 2022-01-31T22:18:19.229+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:19.230554+0000) 2022-01-31T22:18:19.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:19 smithi171 conmon[51620]: debug 2022-01-31T22:18:19.088+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:19.089657+0000) 2022-01-31T22:18:19.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:19 smithi167 conmon[49112]: debug 2022-01-31T22:18:19.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:19.563848+0000) 2022-01-31T22:18:19.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:19 smithi167 conmon[54076]: debug 2022-01-31T22:18:19.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:19.735649+0000) 2022-01-31T22:18:19.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:19 smithi167 conmon[60316]: debug 2022-01-31T22:18:19.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:19.489127+0000) 2022-01-31T22:18:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:19 smithi171 conmon[41853]: debug 2022-01-31T22:18:19.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:19.437717+0000) 2022-01-31T22:18:20.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:20 smithi171 conmon[46715]: debug 2022-01-31T22:18:20.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:20.230780+0000) 2022-01-31T22:18:20.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:20 smithi171 conmon[51620]: debug 2022-01-31T22:18:20.089+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:20.089864+0000) 2022-01-31T22:18:20.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:20 smithi167 conmon[54076]: debug 2022-01-31T22:18:20.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:20.735828+0000) 2022-01-31T22:18:20.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:20 smithi167 conmon[60316]: debug 2022-01-31T22:18:20.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:20.489311+0000) 2022-01-31T22:18:20.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:20 smithi167 conmon[49112]: debug 2022-01-31T22:18:20.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:20.564060+0000) 2022-01-31T22:18:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:20 smithi171 conmon[41853]: debug 2022-01-31T22:18:20.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:20.437875+0000) 2022-01-31T22:18:21.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:21 smithi171 conmon[46715]: debug 2022-01-31T22:18:21.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:21.230956+0000) 2022-01-31T22:18:21.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:21 smithi171 conmon[51620]: debug 2022-01-31T22:18:21.089+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:21.090029+0000) 2022-01-31T22:18:21.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:21 smithi167 conmon[49112]: debug 2022-01-31T22:18:21.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:21.564216+0000) 2022-01-31T22:18:21.783 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:21 smithi167 conmon[60316]: debug 2022-01-31T22:18:21.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:21.489465+0000) 2022-01-31T22:18:21.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:21 smithi167 conmon[54076]: debug 2022-01-31T22:18:21.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:21.736034+0000) 2022-01-31T22:18:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:21 smithi171 conmon[41853]: debug 2022-01-31T22:18:21.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:21.438071+0000) 2022-01-31T22:18:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[46715]: debug 2022-01-31T22:18:22.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.231127+0000) 2022-01-31T22:18:22.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[51620]: debug 2022-01-31T22:18:22.089+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.090195+0000) 2022-01-31T22:18:22.489 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:22 smithi167 conmon[49112]: debug 2022-01-31T22:18:22.458+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.459587+0000) 2022-01-31T22:18:22.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:22 smithi167 conmon[54076]: debug 2022-01-31T22:18:22.458+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.459701+0000) 2022-01-31T22:18:22.490 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:22 smithi167 conmon[60316]: debug 2022-01-31T22:18:22.456+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.458394+0000) 2022-01-31T22:18:22.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:22 smithi167 conmon[49112]: debug 2022-01-31T22:18:22.562+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.564416+0000) 2022-01-31T22:18:22.784 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:22 smithi167 conmon[54076]: debug 2022-01-31T22:18:22.734+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.736166+0000) 2022-01-31T22:18:22.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:22 smithi167 conmon[60316]: debug 2022-01-31T22:18:22.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.489581+0000) 2022-01-31T22:18:22.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[46715]: debug 2022-01-31T22:18:22.457+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.458387+0000) 2022-01-31T22:18:22.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[51620]: debug 2022-01-31T22:18:22.458+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.458984+0000) 2022-01-31T22:18:22.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[35325]: debug 2022-01-31T22:18:22.479+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198870 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:22.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[41853]: debug 2022-01-31T22:18:22.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.438228+0000) 2022-01-31T22:18:22.860 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:22 smithi171 conmon[41853]: debug 2022-01-31T22:18:22.457+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:22.458228+0000) 2022-01-31T22:18:23.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:23 smithi171 conmon[46715]: debug 2022-01-31T22:18:23.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:23.231262+0000) 2022-01-31T22:18:23.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:23 smithi171 conmon[51620]: debug 2022-01-31T22:18:23.089+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:23.090362+0000) 2022-01-31T22:18:23.783 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:23 smithi167 conmon[54076]: debug 2022-01-31T22:18:23.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:23.736329+0000) 2022-01-31T22:18:23.784 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:23 smithi167 conmon[60316]: debug 2022-01-31T22:18:23.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:23.489693+0000) 2022-01-31T22:18:23.784 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:23 smithi167 conmon[49112]: debug 2022-01-31T22:18:23.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:23.564576+0000) 2022-01-31T22:18:23.829 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:23 smithi171 conmon[41853]: debug 2022-01-31T22:18:23.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:23.438432+0000) 2022-01-31T22:18:24.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:23 smithi171 conmon[35325]: debug 2022-01-31T22:18:23.829+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:18:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:24 smithi171 conmon[46715]: debug 2022-01-31T22:18:24.230+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:24.231387+0000) 2022-01-31T22:18:24.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:24 smithi171 conmon[51620]: debug 2022-01-31T22:18:24.089+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:24.090498+0000) 2022-01-31T22:18:24.741 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:24 smithi167 conmon[49112]: debug 2022-01-31T22:18:24.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:24.564729+0000) 2022-01-31T22:18:24.742 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:24 smithi167 conmon[54076]: debug 2022-01-31T22:18:24.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:24.736530+0000) 2022-01-31T22:18:24.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:24 smithi167 conmon[60316]: debug 2022-01-31T22:18:24.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:24.489885+0000) 2022-01-31T22:18:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:24 smithi171 conmon[41853]: debug 2022-01-31T22:18:24.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:24.438511+0000) 2022-01-31T22:18:25.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:25 smithi171 conmon[46715]: debug 2022-01-31T22:18:25.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:25.231536+0000) 2022-01-31T22:18:25.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:25 smithi171 conmon[51620]: debug 2022-01-31T22:18:25.089+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:25.090651+0000) 2022-01-31T22:18:25.745 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:25 smithi167 conmon[54076]: debug 2022-01-31T22:18:25.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:25.736726+0000) 2022-01-31T22:18:25.746 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:25 smithi167 conmon[60316]: debug 2022-01-31T22:18:25.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:25.490081+0000) 2022-01-31T22:18:25.746 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:25 smithi167 conmon[49112]: debug 2022-01-31T22:18:25.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:25.564913+0000) 2022-01-31T22:18:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:25 smithi171 conmon[41853]: debug 2022-01-31T22:18:25.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:25.438679+0000) 2022-01-31T22:18:26.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:26 smithi171 conmon[46715]: debug 2022-01-31T22:18:26.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:26.231728+0000) 2022-01-31T22:18:26.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:26 smithi171 conmon[51620]: debug 2022-01-31T22:18:26.090+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:26.090842+0000) 2022-01-31T22:18:26.747 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:26 smithi167 conmon[54076]: debug 2022-01-31T22:18:26.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:26.736906+0000) 2022-01-31T22:18:26.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:26 smithi167 conmon[60316]: debug 2022-01-31T22:18:26.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:26.490271+0000) 2022-01-31T22:18:26.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:26 smithi167 conmon[49112]: debug 2022-01-31T22:18:26.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:26.565056+0000) 2022-01-31T22:18:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:26 smithi171 conmon[41853]: debug 2022-01-31T22:18:26.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:26.438850+0000) 2022-01-31T22:18:27.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[46715]: debug 2022-01-31T22:18:27.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.231896+0000) 2022-01-31T22:18:27.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[51620]: debug 2022-01-31T22:18:27.090+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.091037+0000) 2022-01-31T22:18:27.736 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:27 smithi167 conmon[49112]: debug 2022-01-31T22:18:27.481+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.482574+0000) 2022-01-31T22:18:27.737 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:27 smithi167 conmon[49112]: debug 2022-01-31T22:18:27.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.565191+0000) 2022-01-31T22:18:27.737 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:27 smithi167 conmon[54076]: debug 2022-01-31T22:18:27.481+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.483084+0000) 2022-01-31T22:18:27.738 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:27 smithi167 conmon[60316]: debug 2022-01-31T22:18:27.480+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.482333+0000) 2022-01-31T22:18:27.739 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:27 smithi167 conmon[60316]: debug 2022-01-31T22:18:27.488+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.490398+0000) 2022-01-31T22:18:27.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[35325]: debug 2022-01-31T22:18:27.502+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 198980 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:27.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[41853]: debug 2022-01-31T22:18:27.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.439000+0000) 2022-01-31T22:18:27.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[41853]: debug 2022-01-31T22:18:27.481+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.481982+0000) 2022-01-31T22:18:27.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[46715]: debug 2022-01-31T22:18:27.481+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.482505+0000) 2022-01-31T22:18:27.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:27 smithi171 conmon[51620]: debug 2022-01-31T22:18:27.481+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.482247+0000) 2022-01-31T22:18:28.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:27 smithi167 conmon[54076]: debug 2022-01-31T22:18:27.736+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:27.737092+0000) 2022-01-31T22:18:28.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:28 smithi171 conmon[46715]: debug 2022-01-31T22:18:28.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:28.231994+0000) 2022-01-31T22:18:28.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:28 smithi171 conmon[51620]: debug 2022-01-31T22:18:28.090+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:28.091243+0000) 2022-01-31T22:18:28.751 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:28 smithi167 conmon[49112]: debug 2022-01-31T22:18:28.563+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:28.565361+0000) 2022-01-31T22:18:28.752 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:28 smithi167 conmon[54076]: debug 2022-01-31T22:18:28.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:28.737240+0000) 2022-01-31T22:18:28.753 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:28 smithi167 conmon[60316]: debug 2022-01-31T22:18:28.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:28.490541+0000) 2022-01-31T22:18:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:28 smithi171 conmon[41853]: debug 2022-01-31T22:18:28.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:28.439184+0000) 2022-01-31T22:18:29.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:29 smithi171 conmon[46715]: debug 2022-01-31T22:18:29.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:29.232137+0000) 2022-01-31T22:18:29.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:29 smithi171 conmon[51620]: debug 2022-01-31T22:18:29.090+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:29.091377+0000) 2022-01-31T22:18:29.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:29 smithi167 conmon[49112]: debug 2022-01-31T22:18:29.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:29.565577+0000) 2022-01-31T22:18:29.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:29 smithi167 conmon[54076]: debug 2022-01-31T22:18:29.735+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:29.737403+0000) 2022-01-31T22:18:29.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:29 smithi167 conmon[60316]: debug 2022-01-31T22:18:29.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:29.490729+0000) 2022-01-31T22:18:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:29 smithi171 conmon[41853]: debug 2022-01-31T22:18:29.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:29.439331+0000) 2022-01-31T22:18:30.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:30 smithi171 conmon[46715]: debug 2022-01-31T22:18:30.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:30.232325+0000) 2022-01-31T22:18:30.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:30 smithi171 conmon[51620]: debug 2022-01-31T22:18:30.091+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:30.091569+0000) 2022-01-31T22:18:30.758 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:30 smithi167 conmon[49112]: debug 2022-01-31T22:18:30.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:30.565722+0000) 2022-01-31T22:18:30.759 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:30 smithi167 conmon[54076]: debug 2022-01-31T22:18:30.736+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:30.737561+0000) 2022-01-31T22:18:30.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:30 smithi167 conmon[60316]: debug 2022-01-31T22:18:30.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:30.490920+0000) 2022-01-31T22:18:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:30 smithi171 conmon[41853]: debug 2022-01-31T22:18:30.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:30.439522+0000) 2022-01-31T22:18:31.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:31 smithi171 conmon[46715]: debug 2022-01-31T22:18:31.231+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:31.232474+0000) 2022-01-31T22:18:31.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:31 smithi171 conmon[51620]: debug 2022-01-31T22:18:31.091+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:31.091726+0000) 2022-01-31T22:18:31.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:31 smithi167 conmon[49112]: debug 2022-01-31T22:18:31.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:31.565917+0000) 2022-01-31T22:18:31.761 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:31 smithi167 conmon[54076]: debug 2022-01-31T22:18:31.736+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:31.737693+0000) 2022-01-31T22:18:31.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:31 smithi167 conmon[60316]: debug 2022-01-31T22:18:31.489+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:31.491141+0000) 2022-01-31T22:18:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:31 smithi171 conmon[41853]: debug 2022-01-31T22:18:31.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:31.439698+0000) 2022-01-31T22:18:32.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[46715]: debug 2022-01-31T22:18:32.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.232675+0000) 2022-01-31T22:18:32.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[51620]: debug 2022-01-31T22:18:32.091+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.091941+0000) 2022-01-31T22:18:32.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:32 smithi167 conmon[49112]: debug 2022-01-31T22:18:32.505+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.506934+0000) 2022-01-31T22:18:32.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:32 smithi167 conmon[49112]: debug 2022-01-31T22:18:32.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.566080+0000) 2022-01-31T22:18:32.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:32 smithi167 conmon[54076]: debug 2022-01-31T22:18:32.505+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.507094+0000) 2022-01-31T22:18:32.762 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:32 smithi167 conmon[54076]: debug 2022-01-31T22:18:32.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.737872+0000) 2022-01-31T22:18:32.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:32 smithi167 conmon[60316]: debug 2022-01-31T22:18:32.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.491313+0000) 2022-01-31T22:18:32.763 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:32 smithi167 conmon[60316]: debug 2022-01-31T22:18:32.504+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.506216+0000) 2022-01-31T22:18:32.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[35325]: debug 2022-01-31T22:18:32.526+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199093 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:32.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[46715]: debug 2022-01-31T22:18:32.505+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.506310+0000) 2022-01-31T22:18:32.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[51620]: debug 2022-01-31T22:18:32.505+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.505828+0000) 2022-01-31T22:18:32.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[41853]: debug 2022-01-31T22:18:32.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.439806+0000) 2022-01-31T22:18:32.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:32 smithi171 conmon[41853]: debug 2022-01-31T22:18:32.505+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:32.506609+0000) 2022-01-31T22:18:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:33 smithi171 conmon[46715]: debug 2022-01-31T22:18:33.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:33.232879+0000) 2022-01-31T22:18:33.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:33 smithi171 conmon[51620]: debug 2022-01-31T22:18:33.091+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:33.092129+0000) 2022-01-31T22:18:33.764 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:33 smithi167 conmon[54076]: debug 2022-01-31T22:18:33.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:33.738031+0000) 2022-01-31T22:18:33.765 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:33 smithi167 conmon[60316]: debug 2022-01-31T22:18:33.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:33.491491+0000) 2022-01-31T22:18:33.765 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:33 smithi167 conmon[49112]: debug 2022-01-31T22:18:33.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:33.566236+0000) 2022-01-31T22:18:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:33 smithi171 conmon[41853]: debug 2022-01-31T22:18:33.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:33.439989+0000) 2022-01-31T22:18:34.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:34 smithi171 conmon[46715]: debug 2022-01-31T22:18:34.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:34.233057+0000) 2022-01-31T22:18:34.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:34 smithi171 conmon[51620]: debug 2022-01-31T22:18:34.091+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:34.092316+0000) 2022-01-31T22:18:34.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:34 smithi167 conmon[49112]: debug 2022-01-31T22:18:34.564+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:34.566419+0000) 2022-01-31T22:18:34.769 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:34 smithi167 conmon[54076]: debug 2022-01-31T22:18:34.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:34.738243+0000) 2022-01-31T22:18:34.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:34 smithi167 conmon[60316]: debug 2022-01-31T22:18:34.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:34.491699+0000) 2022-01-31T22:18:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:34 smithi171 conmon[41853]: debug 2022-01-31T22:18:34.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:34.440121+0000) 2022-01-31T22:18:35.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:35 smithi171 conmon[51620]: debug 2022-01-31T22:18:35.091+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:35.092518+0000) 2022-01-31T22:18:35.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:35 smithi171 conmon[46715]: debug 2022-01-31T22:18:35.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:35.233245+0000) 2022-01-31T22:18:35.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:35 smithi167 conmon[49112]: debug 2022-01-31T22:18:35.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:35.566581+0000) 2022-01-31T22:18:35.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:35 smithi167 conmon[60316]: debug 2022-01-31T22:18:35.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:35.491875+0000) 2022-01-31T22:18:35.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:35 smithi167 conmon[54076]: debug 2022-01-31T22:18:35.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:35.738425+0000) 2022-01-31T22:18:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:35 smithi171 conmon[41853]: debug 2022-01-31T22:18:35.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:35.440278+0000) 2022-01-31T22:18:36.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:36 smithi171 conmon[51620]: debug 2022-01-31T22:18:36.092+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:36.092698+0000) 2022-01-31T22:18:36.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:36 smithi171 conmon[46715]: debug 2022-01-31T22:18:36.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:36.233411+0000) 2022-01-31T22:18:36.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:36 smithi167 conmon[49112]: debug 2022-01-31T22:18:36.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:36.566738+0000) 2022-01-31T22:18:36.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:36 smithi167 conmon[54076]: debug 2022-01-31T22:18:36.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:36.738594+0000) 2022-01-31T22:18:36.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:36 smithi167 conmon[60316]: debug 2022-01-31T22:18:36.490+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:36.492066+0000) 2022-01-31T22:18:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:36 smithi171 conmon[41853]: debug 2022-01-31T22:18:36.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:36.440475+0000) 2022-01-31T22:18:37.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[51620]: debug 2022-01-31T22:18:37.092+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.092853+0000) 2022-01-31T22:18:37.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[46715]: debug 2022-01-31T22:18:37.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.233569+0000) 2022-01-31T22:18:37.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[49112]: debug 2022-01-31T22:18:37.530+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.531525+0000) 2022-01-31T22:18:37.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[49112]: debug 2022-01-31T22:18:37.565+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.566897+0000) 2022-01-31T22:18:37.777 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[54076]: debug 2022-01-31T22:18:37.528+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.530176+0000) 2022-01-31T22:18:37.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[54076]: debug 2022-01-31T22:18:37.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.738764+0000) 2022-01-31T22:18:37.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[60316]: debug 2022-01-31T22:18:37.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.492263+0000) 2022-01-31T22:18:37.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[60316]: debug 2022-01-31T22:18:37.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:37 smithi167 conmon[60316]: 2022-01-31T22:18:37.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.529299+0000) 2022-01-31T22:18:37.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[35325]: debug 2022-01-31T22:18:37.549+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199203 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:37.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[46715]: debug 2022-01-31T22:18:37.528+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.529567+0000) 2022-01-31T22:18:37.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[51620]: debug 2022-01-31T22:18:37.528+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.529665+0000) 2022-01-31T22:18:37.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[41853]: debug 2022-01-31T22:18:37.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.440677+0000) 2022-01-31T22:18:37.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:37 smithi171 conmon[41853]: debug 2022-01-31T22:18:37.529+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:37.530210+0000) 2022-01-31T22:18:38.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:38 smithi171 conmon[51620]: debug 2022-01-31T22:18:38.092+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:38.093030+0000) 2022-01-31T22:18:38.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:38 smithi171 conmon[46715]: debug 2022-01-31T22:18:38.232+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:38.233727+0000) 2022-01-31T22:18:38.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:38 smithi167 conmon[49112]: debug 2022-01-31T22:18:38.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:38.567048+0000) 2022-01-31T22:18:38.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:38 smithi167 conmon[54076]: debug 2022-01-31T22:18:38.737+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:38.738901+0000) 2022-01-31T22:18:38.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:38 smithi167 conmon[60316]: debug 2022-01-31T22:18:38.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:38.492461+0000) 2022-01-31T22:18:38.831 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:38 smithi171 conmon[41853]: debug 2022-01-31T22:18:38.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:38.440874+0000) 2022-01-31T22:18:39.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:38 smithi171 conmon[35325]: debug 2022-01-31T22:18:38.830+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:18:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:39 smithi171 conmon[46715]: debug 2022-01-31T22:18:39.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:39.233866+0000) 2022-01-31T22:18:39.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:39 smithi171 conmon[51620]: debug 2022-01-31T22:18:39.092+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:39.093197+0000) 2022-01-31T22:18:39.785 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:39 smithi167 conmon[49112]: debug 2022-01-31T22:18:39.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:39.567262+0000) 2022-01-31T22:18:39.786 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:39 smithi167 conmon[54076]: debug 2022-01-31T22:18:39.738+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:39.739089+0000) 2022-01-31T22:18:39.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:39 smithi167 conmon[60316]: debug 2022-01-31T22:18:39.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:39.492605+0000) 2022-01-31T22:18:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:39 smithi171 conmon[41853]: debug 2022-01-31T22:18:39.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:39.440998+0000) 2022-01-31T22:18:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:40 smithi171 conmon[46715]: debug 2022-01-31T22:18:40.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:40.234073+0000) 2022-01-31T22:18:40.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:40 smithi171 conmon[51620]: debug 2022-01-31T22:18:40.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:40.093393+0000) 2022-01-31T22:18:40.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:40 smithi167 conmon[49112]: debug 2022-01-31T22:18:40.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:40.567477+0000) 2022-01-31T22:18:40.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:40 smithi167 conmon[54076]: debug 2022-01-31T22:18:40.738+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:40.739307+0000) 2022-01-31T22:18:40.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:40 smithi167 conmon[60316]: debug 2022-01-31T22:18:40.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:40.492762+0000) 2022-01-31T22:18:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:40 smithi171 conmon[41853]: debug 2022-01-31T22:18:40.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:40.441189+0000) 2022-01-31T22:18:41.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:41 smithi171 conmon[46715]: debug 2022-01-31T22:18:41.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:41.234259+0000) 2022-01-31T22:18:41.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:41 smithi171 conmon[51620]: debug 2022-01-31T22:18:41.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:41.093551+0000) 2022-01-31T22:18:41.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:41 smithi167 conmon[49112]: debug 2022-01-31T22:18:41.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:41.567685+0000) 2022-01-31T22:18:41.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:41 smithi167 conmon[54076]: debug 2022-01-31T22:18:41.738+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:41.739464+0000) 2022-01-31T22:18:41.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:41 smithi167 conmon[60316]: debug 2022-01-31T22:18:41.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:41.492957+0000) 2022-01-31T22:18:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:41 smithi171 conmon[41853]: debug 2022-01-31T22:18:41.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:41.441373+0000) 2022-01-31T22:18:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[46715]: debug 2022-01-31T22:18:42.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.234447+0000) 2022-01-31T22:18:42.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[51620]: debug 2022-01-31T22:18:42.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.093783+0000) 2022-01-31T22:18:42.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:42 smithi167 conmon[49112]: debug 2022-01-31T22:18:42.551+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.553310+0000) 2022-01-31T22:18:42.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:42 smithi167 conmon[49112]: debug 2022-01-31T22:18:42.566+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.567861+0000) 2022-01-31T22:18:42.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:42 smithi167 conmon[54076]: debug 2022-01-31T22:18:42.551+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.553215+0000) 2022-01-31T22:18:42.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:42 smithi167 conmon[54076]: debug 2022-01-31T22:18:42.739+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.739663+0000) 2022-01-31T22:18:42.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:42 smithi167 conmon[60316]: debug 2022-01-31T22:18:42.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.493174+0000) 2022-01-31T22:18:42.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:42 smithi167 conmon[60316]: debug 2022-01-31T22:18:42.550+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.552319+0000) 2022-01-31T22:18:42.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[35325]: debug 2022-01-31T22:18:42.572+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199315 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:42.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[41853]: debug 2022-01-31T22:18:42.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.441574+0000) 2022-01-31T22:18:42.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[41853]: debug 2022-01-31T22:18:42.552+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.553097+0000) 2022-01-31T22:18:42.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[46715]: debug 2022-01-31T22:18:42.551+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.552208+0000) 2022-01-31T22:18:42.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:42 smithi171 conmon[51620]: debug 2022-01-31T22:18:42.553+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:42.554403+0000) 2022-01-31T22:18:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:43 smithi171 conmon[46715]: debug 2022-01-31T22:18:43.233+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:43.234650+0000) 2022-01-31T22:18:43.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:43 smithi171 conmon[51620]: debug 2022-01-31T22:18:43.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:43.093948+0000) 2022-01-31T22:18:43.786 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:43 smithi167 conmon[60316]: debug 2022-01-31T22:18:43.491+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:43.493366+0000) 2022-01-31T22:18:43.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:43 smithi167 conmon[49112]: debug 2022-01-31T22:18:43.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:43.568012+0000) 2022-01-31T22:18:43.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:43 smithi167 conmon[54076]: debug 2022-01-31T22:18:43.738+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:43.739749+0000) 2022-01-31T22:18:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:43 smithi171 conmon[41853]: debug 2022-01-31T22:18:43.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:43.441791+0000) 2022-01-31T22:18:44.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:44 smithi171 conmon[51620]: debug 2022-01-31T22:18:44.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:44.094129+0000) 2022-01-31T22:18:44.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:44 smithi171 conmon[46715]: debug 2022-01-31T22:18:44.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:44.234834+0000) 2022-01-31T22:18:44.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:44 smithi167 conmon[60316]: debug 2022-01-31T22:18:44.492+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:44.493535+0000) 2022-01-31T22:18:44.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:44 smithi167 conmon[49112]: debug 2022-01-31T22:18:44.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:44.568149+0000) 2022-01-31T22:18:44.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:44 smithi167 conmon[54076]: debug 2022-01-31T22:18:44.739+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:44.739910+0000) 2022-01-31T22:18:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:44 smithi171 conmon[41853]: debug 2022-01-31T22:18:44.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:44.441943+0000) 2022-01-31T22:18:45.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:45 smithi171 conmon[46715]: debug 2022-01-31T22:18:45.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:45.235024+0000) 2022-01-31T22:18:45.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:45 smithi171 conmon[51620]: debug 2022-01-31T22:18:45.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:45.094287+0000) 2022-01-31T22:18:45.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:45 smithi167 conmon[60316]: debug 2022-01-31T22:18:45.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:45.493723+0000) 2022-01-31T22:18:45.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:45 smithi167 conmon[49112]: debug 2022-01-31T22:18:45.567+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:45.568304+0000) 2022-01-31T22:18:45.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:45 smithi167 conmon[54076]: debug 2022-01-31T22:18:45.739+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:45.740054+0000) 2022-01-31T22:18:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:45 smithi171 conmon[41853]: debug 2022-01-31T22:18:45.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:45.442092+0000) 2022-01-31T22:18:46.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:46 smithi171 conmon[46715]: debug 2022-01-31T22:18:46.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:46.235217+0000) 2022-01-31T22:18:46.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:46 smithi171 conmon[51620]: debug 2022-01-31T22:18:46.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:46.094469+0000) 2022-01-31T22:18:46.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:46 smithi167 conmon[60316]: debug 2022-01-31T22:18:46.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:46.493915+0000) 2022-01-31T22:18:46.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:46 smithi167 conmon[49112]: debug 2022-01-31T22:18:46.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:46.568497+0000) 2022-01-31T22:18:46.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:46 smithi167 conmon[54076]: debug 2022-01-31T22:18:46.739+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:46.740218+0000) 2022-01-31T22:18:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:46 smithi171 conmon[41853]: debug 2022-01-31T22:18:46.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:46.442271+0000) 2022-01-31T22:18:47.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[46715]: debug 2022-01-31T22:18:47.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.235397+0000) 2022-01-31T22:18:47.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[51620]: debug 2022-01-31T22:18:47.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.094668+0000) 2022-01-31T22:18:47.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:47 smithi167 conmon[60316]: debug 2022-01-31T22:18:47.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.494112+0000) 2022-01-31T22:18:47.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:47 smithi167 conmon[60316]: debug 2022-01-31T22:18:47.575+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.575494+0000) 2022-01-31T22:18:47.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:47 smithi167 conmon[49112]: debug 2022-01-31T22:18:47.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.568712+0000) 2022-01-31T22:18:47.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:47 smithi167 conmon[49112]: debug 2022-01-31T22:18:47.574+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.575372+0000) 2022-01-31T22:18:47.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:47 smithi167 conmon[54076]: debug 2022-01-31T22:18:47.581+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.582311+0000) 2022-01-31T22:18:47.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:47 smithi167 conmon[54076]: debug 2022-01-31T22:18:47.739+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.740373+0000) 2022-01-31T22:18:47.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[35325]: debug 2022-01-31T22:18:47.600+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199425 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:47.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[41853]: debug 2022-01-31T22:18:47.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.442429+0000) 2022-01-31T22:18:47.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[41853]: debug 2022-01-31T22:18:47.572+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.576208+0000) 2022-01-31T22:18:47.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[46715]: debug 2022-01-31T22:18:47.572+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.575112+0000) 2022-01-31T22:18:47.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:47 smithi171 conmon[51620]: debug 2022-01-31T22:18:47.572+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:47.576648+0000) 2022-01-31T22:18:48.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:48 smithi171 conmon[46715]: debug 2022-01-31T22:18:48.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:48.235520+0000) 2022-01-31T22:18:48.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:48 smithi171 conmon[51620]: debug 2022-01-31T22:18:48.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:48.094866+0000) 2022-01-31T22:18:48.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:48 smithi167 conmon[60316]: debug 2022-01-31T22:18:48.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:48.494268+0000) 2022-01-31T22:18:48.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:48 smithi167 conmon[49112]: debug 2022-01-31T22:18:48.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:48.568868+0000) 2022-01-31T22:18:48.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:48 smithi167 conmon[54076]: debug 2022-01-31T22:18:48.740+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:48.740547+0000) 2022-01-31T22:18:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:48 smithi171 conmon[41853]: debug 2022-01-31T22:18:48.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:48.442573+0000) 2022-01-31T22:18:49.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:49 smithi171 conmon[46715]: debug 2022-01-31T22:18:49.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:49.235664+0000) 2022-01-31T22:18:49.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:49 smithi171 conmon[51620]: debug 2022-01-31T22:18:49.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:49.095038+0000) 2022-01-31T22:18:49.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:49 smithi167 conmon[60316]: debug 2022-01-31T22:18:49.493+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:49.494408+0000) 2022-01-31T22:18:49.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:49 smithi167 conmon[49112]: debug 2022-01-31T22:18:49.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:49.569040+0000) 2022-01-31T22:18:49.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:49 smithi167 conmon[54076]: debug 2022-01-31T22:18:49.740+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:49.740711+0000) 2022-01-31T22:18:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:49 smithi171 conmon[41853]: debug 2022-01-31T22:18:49.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:49.442709+0000) 2022-01-31T22:18:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:50 smithi171 conmon[46715]: debug 2022-01-31T22:18:50.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:50.235816+0000) 2022-01-31T22:18:50.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:50 smithi171 conmon[51620]: debug 2022-01-31T22:18:50.093+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:50.095212+0000) 2022-01-31T22:18:50.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:50 smithi167 conmon[60316]: debug 2022-01-31T22:18:50.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:50.494597+0000) 2022-01-31T22:18:50.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:50 smithi167 conmon[49112]: debug 2022-01-31T22:18:50.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:50.569200+0000) 2022-01-31T22:18:50.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:50 smithi167 conmon[54076]: debug 2022-01-31T22:18:50.740+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:50.740867+0000) 2022-01-31T22:18:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:50 smithi171 conmon[41853]: debug 2022-01-31T22:18:50.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:50.442923+0000) 2022-01-31T22:18:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:51 smithi171 conmon[46715]: debug 2022-01-31T22:18:51.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:51.235966+0000) 2022-01-31T22:18:51.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:51 smithi171 conmon[51620]: debug 2022-01-31T22:18:51.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:51.095373+0000) 2022-01-31T22:18:51.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:51 smithi167 conmon[49112]: debug 2022-01-31T22:18:51.568+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:51.569360+0000) 2022-01-31T22:18:51.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:51 smithi167 conmon[54076]: debug 2022-01-31T22:18:51.740+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:51.741060+0000) 2022-01-31T22:18:51.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:51 smithi167 conmon[60316]: debug 2022-01-31T22:18:51.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:51.494779+0000) 2022-01-31T22:18:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:51 smithi171 conmon[41853]: debug 2022-01-31T22:18:51.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:51.443104+0000) 2022-01-31T22:18:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[46715]: debug 2022-01-31T22:18:52.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.236129+0000) 2022-01-31T22:18:52.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[51620]: debug 2022-01-31T22:18:52.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.095555+0000) 2022-01-31T22:18:52.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:52 smithi167 conmon[49112]: debug 2022-01-31T22:18:52.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.569568+0000) 2022-01-31T22:18:52.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:52 smithi167 conmon[49112]: debug 2022-01-31T22:18:52.604+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.604524+0000) 2022-01-31T22:18:52.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:52 smithi167 conmon[54076]: debug 2022-01-31T22:18:52.604+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.604986+0000) 2022-01-31T22:18:52.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:52 smithi167 conmon[54076]: debug 2022-01-31T22:18:52.740+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.741275+0000) 2022-01-31T22:18:52.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:52 smithi167 conmon[60316]: debug 2022-01-31T22:18:52.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.494980+0000) 2022-01-31T22:18:52.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:52 smithi167 conmon[60316]: debug 2022-01-31T22:18:52.606+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.606583+0000) 2022-01-31T22:18:52.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[35325]: debug 2022-01-31T22:18:52.624+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199537 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:52.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[41853]: debug 2022-01-31T22:18:52.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.443280+0000) 2022-01-31T22:18:52.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[41853]: debug 2022-01-31T22:18:52.602+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.604603+0000) 2022-01-31T22:18:52.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[46715]: debug 2022-01-31T22:18:52.603+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.604999+0000) 2022-01-31T22:18:52.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:52 smithi171 conmon[51620]: debug 2022-01-31T22:18:52.603+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:52.605552+0000) 2022-01-31T22:18:53.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:53 smithi171 conmon[46715]: debug 2022-01-31T22:18:53.234+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:53.236233+0000) 2022-01-31T22:18:53.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:53 smithi171 conmon[51620]: debug 2022-01-31T22:18:53.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:53.095680+0000) 2022-01-31T22:18:53.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:53 smithi167 conmon[49112]: debug 2022-01-31T22:18:53.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:53.569785+0000) 2022-01-31T22:18:53.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:53 smithi167 conmon[54076]: debug 2022-01-31T22:18:53.741+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:53.741410+0000) 2022-01-31T22:18:53.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:53 smithi167 conmon[60316]: debug 2022-01-31T22:18:53.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:53.495207+0000) 2022-01-31T22:18:53.831 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:53 smithi171 conmon[41853]: debug 2022-01-31T22:18:53.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:53.443479+0000) 2022-01-31T22:18:54.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:53 smithi171 conmon[35325]: debug 2022-01-31T22:18:53.830+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:18:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:54 smithi171 conmon[46715]: debug 2022-01-31T22:18:54.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:54.236403+0000) 2022-01-31T22:18:54.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:54 smithi171 conmon[51620]: debug 2022-01-31T22:18:54.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:54.095855+0000) 2022-01-31T22:18:54.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:54 smithi167 conmon[49112]: debug 2022-01-31T22:18:54.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:54.569913+0000) 2022-01-31T22:18:54.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:54 smithi167 conmon[60316]: debug 2022-01-31T22:18:54.494+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:54.495409+0000) 2022-01-31T22:18:54.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:54 smithi167 conmon[54076]: debug 2022-01-31T22:18:54.741+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:54.741608+0000) 2022-01-31T22:18:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:54 smithi171 conmon[41853]: debug 2022-01-31T22:18:54.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:54.443605+0000) 2022-01-31T22:18:55.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:55 smithi171 conmon[46715]: debug 2022-01-31T22:18:55.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:55.236585+0000) 2022-01-31T22:18:55.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:55 smithi171 conmon[51620]: debug 2022-01-31T22:18:55.095+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:55.096037+0000) 2022-01-31T22:18:55.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:55 smithi167 conmon[49112]: debug 2022-01-31T22:18:55.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:55.570102+0000) 2022-01-31T22:18:55.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:55 smithi167 conmon[54076]: debug 2022-01-31T22:18:55.741+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:55.741739+0000) 2022-01-31T22:18:55.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:55 smithi167 conmon[60316]: debug 2022-01-31T22:18:55.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:55.495629+0000) 2022-01-31T22:18:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:55 smithi171 conmon[41853]: debug 2022-01-31T22:18:55.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:55.443774+0000) 2022-01-31T22:18:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:56 smithi171 conmon[46715]: debug 2022-01-31T22:18:56.235+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:56.236742+0000) 2022-01-31T22:18:56.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:56 smithi171 conmon[51620]: debug 2022-01-31T22:18:56.094+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:56.096214+0000) 2022-01-31T22:18:56.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:56 smithi167 conmon[49112]: debug 2022-01-31T22:18:56.569+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:56.570275+0000) 2022-01-31T22:18:56.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:56 smithi167 conmon[60316]: debug 2022-01-31T22:18:56.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:56.495768+0000) 2022-01-31T22:18:56.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:56 smithi167 conmon[54076]: debug 2022-01-31T22:18:56.741+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:56.741897+0000) 2022-01-31T22:18:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:56 smithi171 conmon[41853]: debug 2022-01-31T22:18:56.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:56.443963+0000) 2022-01-31T22:18:57.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[46715]: debug 2022-01-31T22:18:57.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.236947+0000) 2022-01-31T22:18:57.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[51620]: debug 2022-01-31T22:18:57.095+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.096437+0000) 2022-01-31T22:18:57.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:57 smithi167 conmon[49112]: debug 2022-01-31T22:18:57.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.570423+0000) 2022-01-31T22:18:57.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:57 smithi167 conmon[49112]: debug 2022-01-31T22:18:57.629+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.629719+0000) 2022-01-31T22:18:57.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:57 smithi167 conmon[54076]: debug 2022-01-31T22:18:57.627+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.628110+0000) 2022-01-31T22:18:57.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:57 smithi167 conmon[54076]: debug 2022-01-31T22:18:57.741+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.742050+0000) 2022-01-31T22:18:57.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:57 smithi167 conmon[60316]: debug 2022-01-31T22:18:57.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.495926+0000) 2022-01-31T22:18:57.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:57 smithi167 conmon[60316]: debug 2022-01-31T22:18:57.629+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.629571+0000) 2022-01-31T22:18:57.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[35325]: debug 2022-01-31T22:18:57.647+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199647 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:18:57.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[46715]: debug 2022-01-31T22:18:57.627+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.629186+0000) 2022-01-31T22:18:57.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[51620]: debug 2022-01-31T22:18:57.627+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.629672+0000) 2022-01-31T22:18:57.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[41853]: debug 2022-01-31T22:18:57.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.444139+0000) 2022-01-31T22:18:57.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:57 smithi171 conmon[41853]: debug 2022-01-31T22:18:57.626+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:57.628543+0000) 2022-01-31T22:18:58.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:58 smithi171 conmon[51620]: debug 2022-01-31T22:18:58.095+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:58.096655+0000) 2022-01-31T22:18:58.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:58 smithi171 conmon[46715]: debug 2022-01-31T22:18:58.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:58.237125+0000) 2022-01-31T22:18:58.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:58 smithi167 conmon[49112]: debug 2022-01-31T22:18:58.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:58.570578+0000) 2022-01-31T22:18:58.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:58 smithi167 conmon[54076]: debug 2022-01-31T22:18:58.741+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:58.742234+0000) 2022-01-31T22:18:58.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:58 smithi167 conmon[60316]: debug 2022-01-31T22:18:58.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:58.496076+0000) 2022-01-31T22:18:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:58 smithi171 conmon[41853]: debug 2022-01-31T22:18:58.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:58.444302+0000) 2022-01-31T22:18:59.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:18:59 smithi171 conmon[46715]: debug 2022-01-31T22:18:59.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:59.237284+0000) 2022-01-31T22:18:59.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:18:59 smithi171 conmon[51620]: debug 2022-01-31T22:18:59.096+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:59.096816+0000) 2022-01-31T22:18:59.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:18:59 smithi167 conmon[49112]: debug 2022-01-31T22:18:59.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:59.570748+0000) 2022-01-31T22:18:59.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:18:59 smithi167 conmon[54076]: debug 2022-01-31T22:18:59.742+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:59.742426+0000) 2022-01-31T22:18:59.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:18:59 smithi167 conmon[60316]: debug 2022-01-31T22:18:59.495+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:59.496218+0000) 2022-01-31T22:18:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:18:59 smithi171 conmon[41853]: debug 2022-01-31T22:18:59.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:18:59.444463+0000) 2022-01-31T22:19:00.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:00 smithi171 conmon[46715]: debug 2022-01-31T22:19:00.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:00.237470+0000) 2022-01-31T22:19:00.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:00 smithi171 conmon[51620]: debug 2022-01-31T22:19:00.096+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:00.097007+0000) 2022-01-31T22:19:00.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:00 smithi167 conmon[49112]: debug 2022-01-31T22:19:00.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:00.570933+0000) 2022-01-31T22:19:00.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:00 smithi167 conmon[54076]: debug 2022-01-31T22:19:00.742+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:00.742576+0000) 2022-01-31T22:19:00.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:00 smithi167 conmon[60316]: debug 2022-01-31T22:19:00.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:00.496375+0000) 2022-01-31T22:19:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:00 smithi171 conmon[41853]: debug 2022-01-31T22:19:00.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:00.444643+0000) 2022-01-31T22:19:01.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:01 smithi171 conmon[51620]: debug 2022-01-31T22:19:01.096+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:01.097195+0000) 2022-01-31T22:19:01.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:01 smithi171 conmon[46715]: debug 2022-01-31T22:19:01.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:01.237565+0000) 2022-01-31T22:19:01.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:01 smithi167 conmon[49112]: debug 2022-01-31T22:19:01.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:01.571119+0000) 2022-01-31T22:19:01.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:01 smithi167 conmon[54076]: debug 2022-01-31T22:19:01.742+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:01.742768+0000) 2022-01-31T22:19:01.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:01 smithi167 conmon[60316]: debug 2022-01-31T22:19:01.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:01.496529+0000) 2022-01-31T22:19:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:01 smithi171 conmon[41853]: debug 2022-01-31T22:19:01.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:01.444825+0000) 2022-01-31T22:19:02.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[46715]: debug 2022-01-31T22:19:02.236+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.237668+0000) 2022-01-31T22:19:02.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[51620]: debug 2022-01-31T22:19:02.096+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.097378+0000) 2022-01-31T22:19:02.672 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[46715]: debug 2022-01-31T22:19:02.652+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.652881+0000) 2022-01-31T22:19:02.673 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[51620]: debug 2022-01-31T22:19:02.651+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.652457+0000) 2022-01-31T22:19:02.673 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[41853]: debug 2022-01-31T22:19:02.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.444992+0000) 2022-01-31T22:19:02.673 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[41853]: debug 2022-01-31T22:19:02.651+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.651804+0000) 2022-01-31T22:19:02.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:02 smithi167 conmon[49112]: debug 2022-01-31T22:19:02.570+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.571329+0000) 2022-01-31T22:19:02.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:02 smithi167 conmon[49112]: debug 2022-01-31T22:19:02.651+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.652480+0000) 2022-01-31T22:19:02.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:02 smithi167 conmon[54076]: debug 2022-01-31T22:19:02.652+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.653066+0000) 2022-01-31T22:19:02.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:02 smithi167 conmon[54076]: debug 2022-01-31T22:19:02.742+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.742976+0000) 2022-01-31T22:19:02.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:02 smithi167 conmon[60316]: debug 2022-01-31T22:19:02.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.496690+0000) 2022-01-31T22:19:02.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:02 smithi167 conmon[60316]: debug 2022-01-31T22:19:02.652+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:02.653151+0000) 2022-01-31T22:19:03.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:02 smithi171 conmon[35325]: debug 2022-01-31T22:19:02.671+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199759 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:03.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:03 smithi171 conmon[46715]: debug 2022-01-31T22:19:03.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:03.237838+0000) 2022-01-31T22:19:03.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:03 smithi171 conmon[51620]: debug 2022-01-31T22:19:03.096+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:03.097578+0000) 2022-01-31T22:19:03.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:03 smithi167 conmon[49112]: debug 2022-01-31T22:19:03.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:03.571520+0000) 2022-01-31T22:19:03.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:03 smithi167 conmon[54076]: debug 2022-01-31T22:19:03.742+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:03.743128+0000) 2022-01-31T22:19:03.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:03 smithi167 conmon[60316]: debug 2022-01-31T22:19:03.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:03.496806+0000) 2022-01-31T22:19:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:03 smithi171 conmon[41853]: debug 2022-01-31T22:19:03.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:03.445168+0000) 2022-01-31T22:19:04.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:04 smithi171 conmon[46715]: debug 2022-01-31T22:19:04.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:04.237990+0000) 2022-01-31T22:19:04.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:04 smithi171 conmon[51620]: debug 2022-01-31T22:19:04.097+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:04.097783+0000) 2022-01-31T22:19:04.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:04 smithi167 conmon[49112]: debug 2022-01-31T22:19:04.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:04.571613+0000) 2022-01-31T22:19:04.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:04 smithi167 conmon[54076]: debug 2022-01-31T22:19:04.742+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:04.743308+0000) 2022-01-31T22:19:04.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:04 smithi167 conmon[60316]: debug 2022-01-31T22:19:04.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:04.496988+0000) 2022-01-31T22:19:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:04 smithi171 conmon[41853]: debug 2022-01-31T22:19:04.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:04.445323+0000) 2022-01-31T22:19:05.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:05 smithi171 conmon[51620]: debug 2022-01-31T22:19:05.097+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:05.097952+0000) 2022-01-31T22:19:05.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:05 smithi171 conmon[46715]: debug 2022-01-31T22:19:05.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:05.238203+0000) 2022-01-31T22:19:05.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:05 smithi167 conmon[49112]: debug 2022-01-31T22:19:05.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:05.571803+0000) 2022-01-31T22:19:05.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:05 smithi167 conmon[54076]: debug 2022-01-31T22:19:05.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:05.743486+0000) 2022-01-31T22:19:05.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:05 smithi167 conmon[60316]: debug 2022-01-31T22:19:05.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:05.497207+0000) 2022-01-31T22:19:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:05 smithi171 conmon[41853]: debug 2022-01-31T22:19:05.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:05.445505+0000) 2022-01-31T22:19:06.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:06 smithi171 conmon[46715]: debug 2022-01-31T22:19:06.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:06.238356+0000) 2022-01-31T22:19:06.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:06 smithi171 conmon[51620]: debug 2022-01-31T22:19:06.097+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:06.098136+0000) 2022-01-31T22:19:06.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:06 smithi167 conmon[49112]: debug 2022-01-31T22:19:06.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:06.571987+0000) 2022-01-31T22:19:06.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:06 smithi167 conmon[54076]: debug 2022-01-31T22:19:06.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:06.743677+0000) 2022-01-31T22:19:06.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:06 smithi167 conmon[60316]: debug 2022-01-31T22:19:06.496+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:06.497357+0000) 2022-01-31T22:19:06.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:06 smithi171 conmon[41853]: debug 2022-01-31T22:19:06.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:06.445674+0000) 2022-01-31T22:19:07.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[46715]: debug 2022-01-31T22:19:07.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.238509+0000) 2022-01-31T22:19:07.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[51620]: debug 2022-01-31T22:19:07.097+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.098320+0000) 2022-01-31T22:19:07.674 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[41853]: debug 2022-01-31T22:19:07.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.445836+0000) 2022-01-31T22:19:07.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:07 smithi167 conmon[49112]: debug 2022-01-31T22:19:07.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.572195+0000) 2022-01-31T22:19:07.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:07 smithi167 conmon[49112]: debug 2022-01-31T22:19:07.674+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.675299+0000) 2022-01-31T22:19:07.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:07 smithi167 conmon[54076]: debug 2022-01-31T22:19:07.674+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.675482+0000) 2022-01-31T22:19:07.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:07 smithi167 conmon[54076]: debug 2022-01-31T22:19:07.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.743806+0000) 2022-01-31T22:19:07.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:07 smithi167 conmon[60316]: debug 2022-01-31T22:19:07.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.497541+0000) 2022-01-31T22:19:07.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:07 smithi167 conmon[60316]: debug 2022-01-31T22:19:07.674+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.674798+0000) 2022-01-31T22:19:08.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[35325]: debug 2022-01-31T22:19:07.695+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199871 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:08.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[41853]: debug 2022-01-31T22:19:07.675+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.676015+0000) 2022-01-31T22:19:08.099 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[46715]: debug 2022-01-31T22:19:07.673+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.674704+0000) 2022-01-31T22:19:08.100 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:07 smithi171 conmon[51620]: debug 2022-01-31T22:19:07.675+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:07.676179+0000) 2022-01-31T22:19:08.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:08 smithi171 conmon[51620]: debug 2022-01-31T22:19:08.097+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:08.098519+0000) 2022-01-31T22:19:08.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:08 smithi171 conmon[46715]: debug 2022-01-31T22:19:08.237+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:08.238680+0000) 2022-01-31T22:19:08.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:08 smithi167 conmon[49112]: debug 2022-01-31T22:19:08.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:08.572359+0000) 2022-01-31T22:19:08.791 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:08 smithi167 conmon[54076]: debug 2022-01-31T22:19:08.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:08.743954+0000) 2022-01-31T22:19:08.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:08 smithi167 conmon[60316]: debug 2022-01-31T22:19:08.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:08.497735+0000) 2022-01-31T22:19:08.831 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:08 smithi171 conmon[41853]: debug 2022-01-31T22:19:08.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:08.445987+0000) 2022-01-31T22:19:09.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:08 smithi171 conmon[35325]: debug 2022-01-31T22:19:08.831+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:19:09.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:09 smithi171 conmon[46715]: debug 2022-01-31T22:19:09.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:09.238771+0000) 2022-01-31T22:19:09.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:09 smithi171 conmon[51620]: debug 2022-01-31T22:19:09.098+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:09.098691+0000) 2022-01-31T22:19:09.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:09 smithi167 conmon[49112]: debug 2022-01-31T22:19:09.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:09.572566+0000) 2022-01-31T22:19:09.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:09 smithi167 conmon[54076]: debug 2022-01-31T22:19:09.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:09.744142+0000) 2022-01-31T22:19:09.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:09 smithi167 conmon[60316]: debug 2022-01-31T22:19:09.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:09.497910+0000) 2022-01-31T22:19:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:09 smithi171 conmon[41853]: debug 2022-01-31T22:19:09.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:09.446087+0000) 2022-01-31T22:19:10.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:10 smithi171 conmon[46715]: debug 2022-01-31T22:19:10.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:10.238894+0000) 2022-01-31T22:19:10.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:10 smithi171 conmon[51620]: debug 2022-01-31T22:19:10.098+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:10.098906+0000) 2022-01-31T22:19:10.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:10 smithi167 conmon[49112]: debug 2022-01-31T22:19:10.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:10.572704+0000) 2022-01-31T22:19:10.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:10 smithi167 conmon[54076]: debug 2022-01-31T22:19:10.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:10.744320+0000) 2022-01-31T22:19:10.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:10 smithi167 conmon[60316]: debug 2022-01-31T22:19:10.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:10.498099+0000) 2022-01-31T22:19:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:10 smithi171 conmon[41853]: debug 2022-01-31T22:19:10.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:10.446260+0000) 2022-01-31T22:19:11.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:11 smithi171 conmon[51620]: debug 2022-01-31T22:19:11.098+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:11.099092+0000) 2022-01-31T22:19:11.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:11 smithi171 conmon[46715]: debug 2022-01-31T22:19:11.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:11.239094+0000) 2022-01-31T22:19:11.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:11 smithi167 conmon[49112]: debug 2022-01-31T22:19:11.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:11.572889+0000) 2022-01-31T22:19:11.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:11 smithi167 conmon[54076]: debug 2022-01-31T22:19:11.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:11.744490+0000) 2022-01-31T22:19:11.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:11 smithi167 conmon[60316]: debug 2022-01-31T22:19:11.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:11.498267+0000) 2022-01-31T22:19:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:11 smithi171 conmon[41853]: debug 2022-01-31T22:19:11.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:11.446411+0000) 2022-01-31T22:19:12.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[46715]: debug 2022-01-31T22:19:12.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.239221+0000) 2022-01-31T22:19:12.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[51620]: debug 2022-01-31T22:19:12.098+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.099273+0000) 2022-01-31T22:19:12.698 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[41853]: debug 2022-01-31T22:19:12.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.446569+0000) 2022-01-31T22:19:12.699 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[41853]: debug 2022-01-31T22:19:12.697+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.698467+0000) 2022-01-31T22:19:12.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:12 smithi167 conmon[60316]: debug 2022-01-31T22:19:12.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.498397+0000) 2022-01-31T22:19:12.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:12 smithi167 conmon[60316]: debug 2022-01-31T22:19:12.696+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.698299+0000) 2022-01-31T22:19:12.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:12 smithi167 conmon[49112]: debug 2022-01-31T22:19:12.571+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.573073+0000) 2022-01-31T22:19:12.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:12 smithi167 conmon[49112]: debug 2022-01-31T22:19:12.696+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.698108+0000) 2022-01-31T22:19:12.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:12 smithi167 conmon[54076]: debug 2022-01-31T22:19:12.697+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.699415+0000) 2022-01-31T22:19:12.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:12 smithi167 conmon[54076]: debug 2022-01-31T22:19:12.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.744642+0000) 2022-01-31T22:19:13.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[35325]: debug 2022-01-31T22:19:12.718+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 199983 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:13.099 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[46715]: debug 2022-01-31T22:19:12.697+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.698522+0000) 2022-01-31T22:19:13.100 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:12 smithi171 conmon[51620]: debug 2022-01-31T22:19:12.698+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:12.699018+0000) 2022-01-31T22:19:13.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:13 smithi171 conmon[46715]: debug 2022-01-31T22:19:13.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:13.239375+0000) 2022-01-31T22:19:13.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:13 smithi171 conmon[51620]: debug 2022-01-31T22:19:13.098+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:13.099461+0000) 2022-01-31T22:19:13.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:13 smithi167 conmon[49112]: debug 2022-01-31T22:19:13.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:13.573259+0000) 2022-01-31T22:19:13.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:13 smithi167 conmon[54076]: debug 2022-01-31T22:19:13.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:13.744751+0000) 2022-01-31T22:19:13.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:13 smithi167 conmon[60316]: debug 2022-01-31T22:19:13.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:13.498559+0000) 2022-01-31T22:19:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:13 smithi171 conmon[41853]: debug 2022-01-31T22:19:13.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:13.446771+0000) 2022-01-31T22:19:14.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:14 smithi171 conmon[46715]: debug 2022-01-31T22:19:14.238+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:14.239552+0000) 2022-01-31T22:19:14.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:14 smithi171 conmon[51620]: debug 2022-01-31T22:19:14.099+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:14.099632+0000) 2022-01-31T22:19:14.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:14 smithi167 conmon[49112]: debug 2022-01-31T22:19:14.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:14.573445+0000) 2022-01-31T22:19:14.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:14 smithi167 conmon[54076]: debug 2022-01-31T22:19:14.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:14.744890+0000) 2022-01-31T22:19:14.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:14 smithi167 conmon[60316]: debug 2022-01-31T22:19:14.497+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:14.498815+0000) 2022-01-31T22:19:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:14 smithi171 conmon[41853]: debug 2022-01-31T22:19:14.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:14.446915+0000) 2022-01-31T22:19:15.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:15 smithi171 conmon[46715]: debug 2022-01-31T22:19:15.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:15.239767+0000) 2022-01-31T22:19:15.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:15 smithi171 conmon[51620]: debug 2022-01-31T22:19:15.099+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:15.099776+0000) 2022-01-31T22:19:15.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:15 smithi167 conmon[49112]: debug 2022-01-31T22:19:15.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:15.573599+0000) 2022-01-31T22:19:15.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:15 smithi167 conmon[54076]: debug 2022-01-31T22:19:15.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:15.745093+0000) 2022-01-31T22:19:15.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:15 smithi167 conmon[60316]: debug 2022-01-31T22:19:15.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:15.499028+0000) 2022-01-31T22:19:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:15 smithi171 conmon[41853]: debug 2022-01-31T22:19:15.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:15.447088+0000) 2022-01-31T22:19:16.198 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:16 smithi171 conmon[51620]: debug 2022-01-31T22:19:16.099+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:16.099988+0000) 2022-01-31T22:19:16.457 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:16 smithi171 conmon[41853]: debug 2022-01-31T22:19:16.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:16.447245+0000) 2022-01-31T22:19:16.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:16 smithi171 conmon[46715]: debug 2022-01-31T22:19:16.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:16.239970+0000) 2022-01-31T22:19:16.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:16 smithi167 conmon[49112]: debug 2022-01-31T22:19:16.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:16.573809+0000) 2022-01-31T22:19:16.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:16 smithi167 conmon[54076]: debug 2022-01-31T22:19:16.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:16.745300+0000) 2022-01-31T22:19:16.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:16 smithi167 conmon[60316]: debug 2022-01-31T22:19:16.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:16.499158+0000) 2022-01-31T22:19:17.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[46715]: debug 2022-01-31T22:19:17.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.240161+0000) 2022-01-31T22:19:17.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[51620]: debug 2022-01-31T22:19:17.099+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.100121+0000) 2022-01-31T22:19:17.720 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[41853]: debug 2022-01-31T22:19:17.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.447430+0000) 2022-01-31T22:19:17.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:17 smithi167 conmon[49112]: debug 2022-01-31T22:19:17.572+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.573976+0000) 2022-01-31T22:19:17.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:17 smithi167 conmon[49112]: debug 2022-01-31T22:19:17.720+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.721829+0000) 2022-01-31T22:19:17.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:17 smithi167 conmon[54076]: debug 2022-01-31T22:19:17.721+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.722659+0000) 2022-01-31T22:19:17.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:17 smithi167 conmon[54076]: debug 2022-01-31T22:19:17.743+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.745454+0000) 2022-01-31T22:19:17.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:17 smithi167 conmon[60316]: debug 2022-01-31T22:19:17.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.499321+0000) 2022-01-31T22:19:17.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:17 smithi167 conmon[60316]: debug 2022-01-31T22:19:17.719+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.721074+0000) 2022-01-31T22:19:18.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[35325]: debug 2022-01-31T22:19:17.741+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200093 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:18.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[41853]: debug 2022-01-31T22:19:17.720+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.720891+0000) 2022-01-31T22:19:18.101 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[46715]: debug 2022-01-31T22:19:17.720+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.721549+0000) 2022-01-31T22:19:18.101 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:17 smithi171 conmon[51620]: debug 2022-01-31T22:19:17.720+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:17.721380+0000) 2022-01-31T22:19:18.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:18 smithi171 conmon[46715]: debug 2022-01-31T22:19:18.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:18.240352+0000) 2022-01-31T22:19:18.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:18 smithi171 conmon[51620]: debug 2022-01-31T22:19:18.099+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:18.100274+0000) 2022-01-31T22:19:18.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:18 smithi167 conmon[49112]: debug 2022-01-31T22:19:18.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:18.574133+0000) 2022-01-31T22:19:18.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:18 smithi167 conmon[54076]: debug 2022-01-31T22:19:18.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:18.745573+0000) 2022-01-31T22:19:18.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:18 smithi167 conmon[60316]: debug 2022-01-31T22:19:18.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:18.499543+0000) 2022-01-31T22:19:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:18 smithi171 conmon[41853]: debug 2022-01-31T22:19:18.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:18.447610+0000) 2022-01-31T22:19:19.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:19 smithi171 conmon[46715]: debug 2022-01-31T22:19:19.239+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:19.240525+0000) 2022-01-31T22:19:19.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:19 smithi171 conmon[51620]: debug 2022-01-31T22:19:19.100+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:19.100413+0000) 2022-01-31T22:19:19.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:19 smithi167 conmon[49112]: debug 2022-01-31T22:19:19.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:19.574319+0000) 2022-01-31T22:19:19.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:19 smithi167 conmon[54076]: debug 2022-01-31T22:19:19.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:19.745754+0000) 2022-01-31T22:19:19.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:19 smithi167 conmon[60316]: debug 2022-01-31T22:19:19.498+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:19.499700+0000) 2022-01-31T22:19:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:19 smithi171 conmon[41853]: debug 2022-01-31T22:19:19.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:19.447791+0000) 2022-01-31T22:19:20.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:20 smithi171 conmon[46715]: debug 2022-01-31T22:19:20.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:20.240710+0000) 2022-01-31T22:19:20.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:20 smithi171 conmon[51620]: debug 2022-01-31T22:19:20.100+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:20.100618+0000) 2022-01-31T22:19:20.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:20 smithi167 conmon[49112]: debug 2022-01-31T22:19:20.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:20.574536+0000) 2022-01-31T22:19:20.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:20 smithi167 conmon[54076]: debug 2022-01-31T22:19:20.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:20.745978+0000) 2022-01-31T22:19:20.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:20 smithi167 conmon[60316]: debug 2022-01-31T22:19:20.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:20.499907+0000) 2022-01-31T22:19:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:20 smithi171 conmon[41853]: debug 2022-01-31T22:19:20.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:20.447953+0000) 2022-01-31T22:19:21.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:21 smithi171 conmon[46715]: debug 2022-01-31T22:19:21.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:21.240930+0000) 2022-01-31T22:19:21.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:21 smithi171 conmon[51620]: debug 2022-01-31T22:19:21.100+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:21.100829+0000) 2022-01-31T22:19:21.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:21 smithi167 conmon[49112]: debug 2022-01-31T22:19:21.573+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:21.574744+0000) 2022-01-31T22:19:21.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:21 smithi167 conmon[54076]: debug 2022-01-31T22:19:21.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:21.746158+0000) 2022-01-31T22:19:21.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:21 smithi167 conmon[60316]: debug 2022-01-31T22:19:21.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:21.500093+0000) 2022-01-31T22:19:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:21 smithi171 conmon[41853]: debug 2022-01-31T22:19:21.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:21.448141+0000) 2022-01-31T22:19:22.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[51620]: debug 2022-01-31T22:19:22.100+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.101010+0000) 2022-01-31T22:19:22.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[46715]: debug 2022-01-31T22:19:22.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.241078+0000) 2022-01-31T22:19:22.744 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[41853]: debug 2022-01-31T22:19:22.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.448291+0000) 2022-01-31T22:19:22.744 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[41853]: debug 2022-01-31T22:19:22.743+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.744268+0000) 2022-01-31T22:19:22.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:22 smithi167 conmon[49112]: debug 2022-01-31T22:19:22.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.574924+0000) 2022-01-31T22:19:22.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:22 smithi167 conmon[49112]: debug 2022-01-31T22:19:22.744+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.746195+0000) 2022-01-31T22:19:22.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:22 smithi167 conmon[54076]: debug 2022-01-31T22:19:22.744+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.745856+0000) 2022-01-31T22:19:22.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:22 smithi167 conmon[54076]: debug 2022-01-31T22:19:22.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.746314+0000) 2022-01-31T22:19:22.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:22 smithi167 conmon[60316]: debug 2022-01-31T22:19:22.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.500291+0000) 2022-01-31T22:19:22.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:22 smithi167 conmon[60316]: debug 2022-01-31T22:19:22.743+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.745097+0000) 2022-01-31T22:19:23.100 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[35325]: debug 2022-01-31T22:19:22.764+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200205 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:23.101 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[46715]: debug 2022-01-31T22:19:22.743+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.744539+0000) 2022-01-31T22:19:23.102 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:22 smithi171 conmon[51620]: debug 2022-01-31T22:19:22.743+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:22.744645+0000) 2022-01-31T22:19:23.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:23 smithi171 conmon[51620]: debug 2022-01-31T22:19:23.100+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:23.101248+0000) 2022-01-31T22:19:23.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:23 smithi171 conmon[46715]: debug 2022-01-31T22:19:23.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:23.241233+0000) 2022-01-31T22:19:23.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:23 smithi167 conmon[49112]: debug 2022-01-31T22:19:23.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:23.575112+0000) 2022-01-31T22:19:23.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:23 smithi167 conmon[54076]: debug 2022-01-31T22:19:23.744+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:23.746475+0000) 2022-01-31T22:19:23.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:23 smithi167 conmon[60316]: debug 2022-01-31T22:19:23.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:23.500445+0000) 2022-01-31T22:19:23.832 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:23 smithi171 conmon[41853]: debug 2022-01-31T22:19:23.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:23.448491+0000) 2022-01-31T22:19:24.100 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:23 smithi171 conmon[35325]: debug 2022-01-31T22:19:23.832+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:19:24.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:24 smithi171 conmon[46715]: debug 2022-01-31T22:19:24.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:24.241469+0000) 2022-01-31T22:19:24.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:24 smithi171 conmon[51620]: debug 2022-01-31T22:19:24.101+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:24.101439+0000) 2022-01-31T22:19:24.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:24 smithi167 conmon[49112]: debug 2022-01-31T22:19:24.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:24.575235+0000) 2022-01-31T22:19:24.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:24 smithi167 conmon[54076]: debug 2022-01-31T22:19:24.745+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:24.746677+0000) 2022-01-31T22:19:24.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:24 smithi167 conmon[60316]: debug 2022-01-31T22:19:24.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:24.500671+0000) 2022-01-31T22:19:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:24 smithi171 conmon[41853]: debug 2022-01-31T22:19:24.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:24.448645+0000) 2022-01-31T22:19:25.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:25 smithi171 conmon[46715]: debug 2022-01-31T22:19:25.240+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:25.241678+0000) 2022-01-31T22:19:25.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:25 smithi171 conmon[51620]: debug 2022-01-31T22:19:25.100+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:25.101650+0000) 2022-01-31T22:19:25.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:25 smithi167 conmon[49112]: debug 2022-01-31T22:19:25.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:25.575387+0000) 2022-01-31T22:19:25.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:25 smithi167 conmon[54076]: debug 2022-01-31T22:19:25.746+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:25.746871+0000) 2022-01-31T22:19:25.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:25 smithi167 conmon[60316]: debug 2022-01-31T22:19:25.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:25.500827+0000) 2022-01-31T22:19:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:25 smithi171 conmon[41853]: debug 2022-01-31T22:19:25.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:25.448837+0000) 2022-01-31T22:19:26.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:26 smithi171 conmon[46715]: debug 2022-01-31T22:19:26.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:26.241794+0000) 2022-01-31T22:19:26.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:26 smithi171 conmon[51620]: debug 2022-01-31T22:19:26.101+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:26.101840+0000) 2022-01-31T22:19:26.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:26 smithi167 conmon[49112]: debug 2022-01-31T22:19:26.574+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:26.575572+0000) 2022-01-31T22:19:26.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:26 smithi167 conmon[54076]: debug 2022-01-31T22:19:26.746+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:26.746998+0000) 2022-01-31T22:19:26.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:26 smithi167 conmon[60316]: debug 2022-01-31T22:19:26.499+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:26.500992+0000) 2022-01-31T22:19:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:26 smithi171 conmon[41853]: debug 2022-01-31T22:19:26.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:26.448993+0000) 2022-01-31T22:19:27.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[51620]: debug 2022-01-31T22:19:27.101+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.102049+0000) 2022-01-31T22:19:27.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[46715]: debug 2022-01-31T22:19:27.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.241982+0000) 2022-01-31T22:19:27.767 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[41853]: debug 2022-01-31T22:19:27.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.449134+0000) 2022-01-31T22:19:27.768 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:27 smithi167 conmon[49112]: debug 2022-01-31T22:19:27.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.575753+0000) 2022-01-31T22:19:27.768 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:27 smithi167 conmon[54076]: debug 2022-01-31T22:19:27.745+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.747173+0000) 2022-01-31T22:19:27.769 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:27 smithi167 conmon[60316]: debug 2022-01-31T22:19:27.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.501175+0000) 2022-01-31T22:19:28.101 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[35325]: debug 2022-01-31T22:19:27.788+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200316 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:28.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[41853]: debug 2022-01-31T22:19:27.766+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.767392+0000) 2022-01-31T22:19:28.103 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[46715]: debug 2022-01-31T22:19:27.766+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.767650+0000) 2022-01-31T22:19:28.103 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:27 smithi171 conmon[51620]: debug 2022-01-31T22:19:27.768+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.768922+0000) 2022-01-31T22:19:28.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:27 smithi167 conmon[49112]: debug 2022-01-31T22:19:27.768+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.769529+0000) 2022-01-31T22:19:28.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:27 smithi167 conmon[54076]: debug 2022-01-31T22:19:27.769+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.770665+0000) 2022-01-31T22:19:28.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:27 smithi167 conmon[60316]: debug 2022-01-31T22:19:27.767+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:27.769472+0000) 2022-01-31T22:19:28.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:28 smithi171 conmon[46715]: debug 2022-01-31T22:19:28.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:28.242131+0000) 2022-01-31T22:19:28.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:28 smithi171 conmon[51620]: debug 2022-01-31T22:19:28.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:28.102250+0000) 2022-01-31T22:19:28.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:28 smithi167 conmon[49112]: debug 2022-01-31T22:19:28.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:28.575956+0000) 2022-01-31T22:19:28.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:28 smithi167 conmon[54076]: debug 2022-01-31T22:19:28.745+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:28.747389+0000) 2022-01-31T22:19:28.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:28 smithi167 conmon[60316]: debug 2022-01-31T22:19:28.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:28.501362+0000) 2022-01-31T22:19:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:28 smithi171 conmon[41853]: debug 2022-01-31T22:19:28.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:28.449326+0000) 2022-01-31T22:19:29.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:29 smithi171 conmon[46715]: debug 2022-01-31T22:19:29.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:29.242301+0000) 2022-01-31T22:19:29.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:29 smithi171 conmon[51620]: debug 2022-01-31T22:19:29.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:29.102423+0000) 2022-01-31T22:19:29.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:29 smithi167 conmon[49112]: debug 2022-01-31T22:19:29.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:29.576130+0000) 2022-01-31T22:19:29.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:29 smithi167 conmon[54076]: debug 2022-01-31T22:19:29.746+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:29.747535+0000) 2022-01-31T22:19:29.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:29 smithi167 conmon[60316]: debug 2022-01-31T22:19:29.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:29.501511+0000) 2022-01-31T22:19:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:29 smithi171 conmon[41853]: debug 2022-01-31T22:19:29.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:29.449472+0000) 2022-01-31T22:19:30.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:30 smithi171 conmon[46715]: debug 2022-01-31T22:19:30.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:30.242515+0000) 2022-01-31T22:19:30.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:30 smithi171 conmon[51620]: debug 2022-01-31T22:19:30.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:30.102614+0000) 2022-01-31T22:19:30.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:30 smithi167 conmon[49112]: debug 2022-01-31T22:19:30.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:30.576292+0000) 2022-01-31T22:19:30.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:30 smithi167 conmon[54076]: debug 2022-01-31T22:19:30.746+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:30.747706+0000) 2022-01-31T22:19:30.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:30 smithi167 conmon[60316]: debug 2022-01-31T22:19:30.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:30.501731+0000) 2022-01-31T22:19:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:30 smithi171 conmon[41853]: debug 2022-01-31T22:19:30.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:30.449658+0000) 2022-01-31T22:19:31.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:31 smithi171 conmon[46715]: debug 2022-01-31T22:19:31.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:31.242708+0000) 2022-01-31T22:19:31.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:31 smithi171 conmon[51620]: debug 2022-01-31T22:19:31.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:31.102791+0000) 2022-01-31T22:19:31.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:31 smithi167 conmon[49112]: debug 2022-01-31T22:19:31.575+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:31.576487+0000) 2022-01-31T22:19:31.795 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:31 smithi167 conmon[54076]: debug 2022-01-31T22:19:31.746+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:31.747906+0000) 2022-01-31T22:19:31.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:31 smithi167 conmon[60316]: debug 2022-01-31T22:19:31.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:31.501899+0000) 2022-01-31T22:19:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:31 smithi171 conmon[41853]: debug 2022-01-31T22:19:31.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:31.449841+0000) 2022-01-31T22:19:32.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[46715]: debug 2022-01-31T22:19:32.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.242864+0000) 2022-01-31T22:19:32.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[51620]: debug 2022-01-31T22:19:32.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.102971+0000) 2022-01-31T22:19:32.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:32 smithi167 conmon[49112]: debug 2022-01-31T22:19:32.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.576709+0000) 2022-01-31T22:19:32.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:32 smithi167 conmon[54076]: debug 2022-01-31T22:19:32.747+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.748096+0000) 2022-01-31T22:19:32.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:32 smithi167 conmon[60316]: debug 2022-01-31T22:19:32.500+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.502037+0000) 2022-01-31T22:19:32.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[41853]: debug 2022-01-31T22:19:32.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.450020+0000) 2022-01-31T22:19:32.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[41853]: debug 2022-01-31T22:19:32.790+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.791863+0000) 2022-01-31T22:19:33.102 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[35325]: debug 2022-01-31T22:19:32.811+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200428 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:33.103 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[46715]: debug 2022-01-31T22:19:32.790+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.792642+0000) 2022-01-31T22:19:33.104 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:32 smithi171 conmon[51620]: debug 2022-01-31T22:19:32.791+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.793049+0000) 2022-01-31T22:19:33.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:32 smithi167 conmon[49112]: debug 2022-01-31T22:19:32.790+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.792298+0000) 2022-01-31T22:19:33.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:32 smithi167 conmon[54076]: debug 2022-01-31T22:19:32.790+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.791880+0000) 2022-01-31T22:19:33.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:32 smithi167 conmon[60316]: debug 2022-01-31T22:19:32.790+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:32.791751+0000) 2022-01-31T22:19:33.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:33 smithi171 conmon[46715]: debug 2022-01-31T22:19:33.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:33.243010+0000) 2022-01-31T22:19:33.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:33 smithi171 conmon[51620]: debug 2022-01-31T22:19:33.101+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:33.103150+0000) 2022-01-31T22:19:33.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:33 smithi167 conmon[49112]: debug 2022-01-31T22:19:33.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:33.576923+0000) 2022-01-31T22:19:33.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:33 smithi167 conmon[54076]: debug 2022-01-31T22:19:33.747+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:33.748298+0000) 2022-01-31T22:19:33.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:33 smithi167 conmon[60316]: debug 2022-01-31T22:19:33.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:33.502209+0000) 2022-01-31T22:19:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:33 smithi171 conmon[41853]: debug 2022-01-31T22:19:33.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:33.450217+0000) 2022-01-31T22:19:34.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:34 smithi171 conmon[46715]: debug 2022-01-31T22:19:34.241+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:34.243172+0000) 2022-01-31T22:19:34.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:34 smithi171 conmon[51620]: debug 2022-01-31T22:19:34.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:34.103321+0000) 2022-01-31T22:19:34.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:34 smithi167 conmon[49112]: debug 2022-01-31T22:19:34.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:34.577066+0000) 2022-01-31T22:19:34.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:34 smithi167 conmon[54076]: debug 2022-01-31T22:19:34.747+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:34.748468+0000) 2022-01-31T22:19:34.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:34 smithi167 conmon[60316]: debug 2022-01-31T22:19:34.501+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:34.502382+0000) 2022-01-31T22:19:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:34 smithi171 conmon[41853]: debug 2022-01-31T22:19:34.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:34.450375+0000) 2022-01-31T22:19:35.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:35 smithi171 conmon[46715]: debug 2022-01-31T22:19:35.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:35.243305+0000) 2022-01-31T22:19:35.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:35 smithi171 conmon[51620]: debug 2022-01-31T22:19:35.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:35.103535+0000) 2022-01-31T22:19:35.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:35 smithi167 conmon[49112]: debug 2022-01-31T22:19:35.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:35.577275+0000) 2022-01-31T22:19:35.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:35 smithi167 conmon[54076]: debug 2022-01-31T22:19:35.748+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:35.748668+0000) 2022-01-31T22:19:35.797 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:35 smithi167 conmon[60316]: debug 2022-01-31T22:19:35.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:35.502550+0000) 2022-01-31T22:19:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:35 smithi171 conmon[41853]: debug 2022-01-31T22:19:35.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:35.450569+0000) 2022-01-31T22:19:36.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:36 smithi171 conmon[46715]: debug 2022-01-31T22:19:36.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:36.243443+0000) 2022-01-31T22:19:36.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:36 smithi171 conmon[51620]: debug 2022-01-31T22:19:36.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:36.103742+0000) 2022-01-31T22:19:36.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:36 smithi167 conmon[49112]: debug 2022-01-31T22:19:36.576+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:36.577438+0000) 2022-01-31T22:19:36.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:36 smithi167 conmon[54076]: debug 2022-01-31T22:19:36.748+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:36.748853+0000) 2022-01-31T22:19:36.797 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:36 smithi167 conmon[60316]: debug 2022-01-31T22:19:36.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:36.502703+0000) 2022-01-31T22:19:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:36 smithi171 conmon[41853]: debug 2022-01-31T22:19:36.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:36.450730+0000) 2022-01-31T22:19:37.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[46715]: debug 2022-01-31T22:19:37.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.243580+0000) 2022-01-31T22:19:37.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[51620]: debug 2022-01-31T22:19:37.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.103914+0000) 2022-01-31T22:19:37.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:37 smithi167 conmon[49112]: debug 2022-01-31T22:19:37.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.577572+0000) 2022-01-31T22:19:37.796 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:37 smithi167 conmon[54076]: debug 2022-01-31T22:19:37.748+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.749015+0000) 2022-01-31T22:19:37.797 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:37 smithi167 conmon[60316]: debug 2022-01-31T22:19:37.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.502868+0000) 2022-01-31T22:19:37.814 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[41853]: debug 2022-01-31T22:19:37.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.450964+0000) 2022-01-31T22:19:38.103 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[35325]: debug 2022-01-31T22:19:37.835+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200556 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:38.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[41853]: debug 2022-01-31T22:19:37.812+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.814652+0000) 2022-01-31T22:19:38.104 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[46715]: debug 2022-01-31T22:19:37.815+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.816894+0000) 2022-01-31T22:19:38.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:37 smithi171 conmon[51620]: debug 2022-01-31T22:19:37.814+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.816018+0000) 2022-01-31T22:19:38.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:37 smithi167 conmon[60316]: debug 2022-01-31T22:19:37.815+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.815739+0000) 2022-01-31T22:19:38.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:37 smithi167 conmon[49112]: debug 2022-01-31T22:19:37.815+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.815840+0000) 2022-01-31T22:19:38.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:37 smithi167 conmon[54076]: debug 2022-01-31T22:19:37.815+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:37.815939+0000) 2022-01-31T22:19:38.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:38 smithi171 conmon[46715]: debug 2022-01-31T22:19:38.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:38.243737+0000) 2022-01-31T22:19:38.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:38 smithi171 conmon[51620]: debug 2022-01-31T22:19:38.103+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:38.104086+0000) 2022-01-31T22:19:38.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:38 smithi167 conmon[60316]: debug 2022-01-31T22:19:38.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:38.503033+0000) 2022-01-31T22:19:38.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:38 smithi167 conmon[49112]: debug 2022-01-31T22:19:38.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:38.577769+0000) 2022-01-31T22:19:38.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:38 smithi167 conmon[54076]: debug 2022-01-31T22:19:38.748+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:38.749218+0000) 2022-01-31T22:19:38.833 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:38 smithi171 conmon[41853]: debug 2022-01-31T22:19:38.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:38.451138+0000) 2022-01-31T22:19:39.103 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:38 smithi171 conmon[35325]: debug 2022-01-31T22:19:38.832+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:19:39.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:39 smithi171 conmon[46715]: debug 2022-01-31T22:19:39.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:39.243891+0000) 2022-01-31T22:19:39.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:39 smithi171 conmon[51620]: debug 2022-01-31T22:19:39.103+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:39.104234+0000) 2022-01-31T22:19:39.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:39 smithi167 conmon[49112]: debug 2022-01-31T22:19:39.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:39.577942+0000) 2022-01-31T22:19:39.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:39 smithi167 conmon[54076]: debug 2022-01-31T22:19:39.748+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:39.749394+0000) 2022-01-31T22:19:39.797 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:39 smithi167 conmon[60316]: debug 2022-01-31T22:19:39.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:39.503170+0000) 2022-01-31T22:19:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:39 smithi171 conmon[41853]: debug 2022-01-31T22:19:39.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:39.451260+0000) 2022-01-31T22:19:40.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:40 smithi171 conmon[51620]: debug 2022-01-31T22:19:40.102+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:40.104423+0000) 2022-01-31T22:19:40.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:40 smithi171 conmon[46715]: debug 2022-01-31T22:19:40.242+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:40.244029+0000) 2022-01-31T22:19:40.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:40 smithi167 conmon[49112]: debug 2022-01-31T22:19:40.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:40.578101+0000) 2022-01-31T22:19:40.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:40 smithi167 conmon[54076]: debug 2022-01-31T22:19:40.749+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:40.749584+0000) 2022-01-31T22:19:40.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:40 smithi167 conmon[60316]: debug 2022-01-31T22:19:40.502+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:40.503305+0000) 2022-01-31T22:19:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:40 smithi171 conmon[41853]: debug 2022-01-31T22:19:40.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:40.451393+0000) 2022-01-31T22:19:41.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:41 smithi171 conmon[46715]: debug 2022-01-31T22:19:41.243+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:41.244231+0000) 2022-01-31T22:19:41.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:41 smithi171 conmon[51620]: debug 2022-01-31T22:19:41.103+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:41.104597+0000) 2022-01-31T22:19:41.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:41 smithi167 conmon[49112]: debug 2022-01-31T22:19:41.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:41.578260+0000) 2022-01-31T22:19:41.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:41 smithi167 conmon[54076]: debug 2022-01-31T22:19:41.749+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:41.749732+0000) 2022-01-31T22:19:41.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:41 smithi167 conmon[60316]: debug 2022-01-31T22:19:41.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:41.503479+0000) 2022-01-31T22:19:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:41 smithi171 conmon[41853]: debug 2022-01-31T22:19:41.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:41.451577+0000) 2022-01-31T22:19:42.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[46715]: debug 2022-01-31T22:19:42.243+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.244330+0000) 2022-01-31T22:19:42.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[51620]: debug 2022-01-31T22:19:42.103+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.104747+0000) 2022-01-31T22:19:42.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:42 smithi167 conmon[49112]: debug 2022-01-31T22:19:42.577+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.578429+0000) 2022-01-31T22:19:42.797 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:42 smithi167 conmon[54076]: debug 2022-01-31T22:19:42.749+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.749862+0000) 2022-01-31T22:19:42.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:42 smithi167 conmon[60316]: debug 2022-01-31T22:19:42.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.503630+0000) 2022-01-31T22:19:42.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[41853]: debug 2022-01-31T22:19:42.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.451749+0000) 2022-01-31T22:19:43.104 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[35325]: debug 2022-01-31T22:19:42.858+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200668 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:43.105 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[41853]: debug 2022-01-31T22:19:42.837+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.839228+0000) 2022-01-31T22:19:43.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[46715]: debug 2022-01-31T22:19:42.838+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.840109+0000) 2022-01-31T22:19:43.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:42 smithi171 conmon[51620]: debug 2022-01-31T22:19:42.838+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.840477+0000) 2022-01-31T22:19:43.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:42 smithi167 conmon[49112]: debug 2022-01-31T22:19:42.838+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.839381+0000) 2022-01-31T22:19:43.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:42 smithi167 conmon[54076]: debug 2022-01-31T22:19:42.839+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.839646+0000) 2022-01-31T22:19:43.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:42 smithi167 conmon[60316]: debug 2022-01-31T22:19:42.839+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:42.840266+0000) 2022-01-31T22:19:43.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:43 smithi171 conmon[46715]: debug 2022-01-31T22:19:43.243+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:43.244535+0000) 2022-01-31T22:19:43.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:43 smithi171 conmon[51620]: debug 2022-01-31T22:19:43.104+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:43.104977+0000) 2022-01-31T22:19:43.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:43 smithi167 conmon[49112]: debug 2022-01-31T22:19:43.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:43.578609+0000) 2022-01-31T22:19:43.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:43 smithi167 conmon[54076]: debug 2022-01-31T22:19:43.749+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:43.750005+0000) 2022-01-31T22:19:43.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:43 smithi167 conmon[60316]: debug 2022-01-31T22:19:43.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:43.503789+0000) 2022-01-31T22:19:43.860 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:43 smithi171 conmon[41853]: debug 2022-01-31T22:19:43.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:43.451923+0000) 2022-01-31T22:19:44.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:44 smithi171 conmon[46715]: debug 2022-01-31T22:19:44.243+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:44.244692+0000) 2022-01-31T22:19:44.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:44 smithi171 conmon[51620]: debug 2022-01-31T22:19:44.104+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:44.105105+0000) 2022-01-31T22:19:44.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:44 smithi167 conmon[49112]: debug 2022-01-31T22:19:44.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:44.578764+0000) 2022-01-31T22:19:44.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:44 smithi167 conmon[54076]: debug 2022-01-31T22:19:44.749+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:44.750158+0000) 2022-01-31T22:19:44.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:44 smithi167 conmon[60316]: debug 2022-01-31T22:19:44.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:44.503956+0000) 2022-01-31T22:19:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:44 smithi171 conmon[41853]: debug 2022-01-31T22:19:44.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:44.452058+0000) 2022-01-31T22:19:45.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:45 smithi171 conmon[46715]: debug 2022-01-31T22:19:45.244+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:45.244845+0000) 2022-01-31T22:19:45.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:45 smithi171 conmon[51620]: debug 2022-01-31T22:19:45.104+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:45.105279+0000) 2022-01-31T22:19:45.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:45 smithi167 conmon[49112]: debug 2022-01-31T22:19:45.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:45.578896+0000) 2022-01-31T22:19:45.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:45 smithi167 conmon[54076]: debug 2022-01-31T22:19:45.749+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:45.750334+0000) 2022-01-31T22:19:45.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:45 smithi167 conmon[60316]: debug 2022-01-31T22:19:45.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:45.504120+0000) 2022-01-31T22:19:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:45 smithi171 conmon[41853]: debug 2022-01-31T22:19:45.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:45.452213+0000) 2022-01-31T22:19:46.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:46 smithi171 conmon[46715]: debug 2022-01-31T22:19:46.244+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:46.245054+0000) 2022-01-31T22:19:46.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:46 smithi171 conmon[51620]: debug 2022-01-31T22:19:46.104+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:46.105451+0000) 2022-01-31T22:19:46.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:46 smithi167 conmon[49112]: debug 2022-01-31T22:19:46.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:46.579050+0000) 2022-01-31T22:19:46.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:46 smithi167 conmon[54076]: debug 2022-01-31T22:19:46.750+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:46.750540+0000) 2022-01-31T22:19:46.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:46 smithi167 conmon[60316]: debug 2022-01-31T22:19:46.503+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:46.504279+0000) 2022-01-31T22:19:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:46 smithi171 conmon[41853]: debug 2022-01-31T22:19:46.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:46.452399+0000) 2022-01-31T22:19:47.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[46715]: debug 2022-01-31T22:19:47.244+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.245222+0000) 2022-01-31T22:19:47.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[51620]: debug 2022-01-31T22:19:47.104+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.105635+0000) 2022-01-31T22:19:47.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:47 smithi167 conmon[49112]: debug 2022-01-31T22:19:47.578+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.579225+0000) 2022-01-31T22:19:47.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:47 smithi167 conmon[54076]: debug 2022-01-31T22:19:47.750+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.750657+0000) 2022-01-31T22:19:47.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:47 smithi167 conmon[60316]: debug 2022-01-31T22:19:47.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.504465+0000) 2022-01-31T22:19:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[41853]: debug 2022-01-31T22:19:47.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.452590+0000) 2022-01-31T22:19:48.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:47 smithi167 conmon[49112]: debug 2022-01-31T22:19:47.862+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.862693+0000) 2022-01-31T22:19:48.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:47 smithi167 conmon[54076]: debug 2022-01-31T22:19:47.861+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.862430+0000) 2022-01-31T22:19:48.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:47 smithi167 conmon[60316]: debug 2022-01-31T22:19:47.861+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.862047+0000) 2022-01-31T22:19:48.204 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[35325]: debug 2022-01-31T22:19:47.889+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200778 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:48.205 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[41853]: debug 2022-01-31T22:19:47.861+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.862191+0000) 2022-01-31T22:19:48.206 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[46715]: debug 2022-01-31T22:19:47.862+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.863037+0000) 2022-01-31T22:19:48.206 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:47 smithi171 conmon[51620]: debug 2022-01-31T22:19:47.862+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:47.863259+0000) 2022-01-31T22:19:48.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:48 smithi171 conmon[51620]: debug 2022-01-31T22:19:48.105+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:48.105806+0000) 2022-01-31T22:19:48.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:48 smithi171 conmon[46715]: debug 2022-01-31T22:19:48.244+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:48.245379+0000) 2022-01-31T22:19:48.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:48 smithi171 conmon[41853]: debug 2022-01-31T22:19:48.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:48.452809+0000) 2022-01-31T22:19:48.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:48 smithi167 conmon[54076]: debug 2022-01-31T22:19:48.750+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:48.750774+0000) 2022-01-31T22:19:48.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:48 smithi167 conmon[60316]: debug 2022-01-31T22:19:48.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:48.504691+0000) 2022-01-31T22:19:48.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:48 smithi167 conmon[49112]: debug 2022-01-31T22:19:48.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:48.579443+0000) 2022-01-31T22:19:49.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:49 smithi171 conmon[46715]: debug 2022-01-31T22:19:49.244+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:49.245571+0000) 2022-01-31T22:19:49.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:49 smithi171 conmon[51620]: debug 2022-01-31T22:19:49.105+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:49.105918+0000) 2022-01-31T22:19:49.798 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:49 smithi167 conmon[54076]: debug 2022-01-31T22:19:49.750+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:49.750946+0000) 2022-01-31T22:19:49.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:49 smithi167 conmon[60316]: debug 2022-01-31T22:19:49.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:49.504833+0000) 2022-01-31T22:19:49.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:49 smithi167 conmon[49112]: debug 2022-01-31T22:19:49.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:49.579625+0000) 2022-01-31T22:19:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:49 smithi171 conmon[41853]: debug 2022-01-31T22:19:49.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:49.452948+0000) 2022-01-31T22:19:50.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:50 smithi171 conmon[46715]: debug 2022-01-31T22:19:50.245+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:50.245769+0000) 2022-01-31T22:19:50.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:50 smithi171 conmon[51620]: debug 2022-01-31T22:19:50.105+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:50.106070+0000) 2022-01-31T22:19:50.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:50 smithi167 conmon[49112]: debug 2022-01-31T22:19:50.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:50.579808+0000) 2022-01-31T22:19:50.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:50 smithi167 conmon[54076]: debug 2022-01-31T22:19:50.750+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:50.751180+0000) 2022-01-31T22:19:50.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:50 smithi167 conmon[60316]: debug 2022-01-31T22:19:50.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:50.505024+0000) 2022-01-31T22:19:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:50 smithi171 conmon[41853]: debug 2022-01-31T22:19:50.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:50.453156+0000) 2022-01-31T22:19:51.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:51 smithi171 conmon[46715]: debug 2022-01-31T22:19:51.245+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:51.245949+0000) 2022-01-31T22:19:51.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:51 smithi171 conmon[51620]: debug 2022-01-31T22:19:51.105+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:51.106265+0000) 2022-01-31T22:19:51.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:51 smithi167 conmon[49112]: debug 2022-01-31T22:19:51.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:51.579989+0000) 2022-01-31T22:19:51.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:51 smithi167 conmon[54076]: debug 2022-01-31T22:19:51.750+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:51.751361+0000) 2022-01-31T22:19:51.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:51 smithi167 conmon[60316]: debug 2022-01-31T22:19:51.504+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:51.505224+0000) 2022-01-31T22:19:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:51 smithi171 conmon[41853]: debug 2022-01-31T22:19:51.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:51.453353+0000) 2022-01-31T22:19:52.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[46715]: debug 2022-01-31T22:19:52.245+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.246103+0000) 2022-01-31T22:19:52.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[51620]: debug 2022-01-31T22:19:52.105+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.106441+0000) 2022-01-31T22:19:52.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[49112]: debug 2022-01-31T22:19:52.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.580146+0000) 2022-01-31T22:19:52.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[54076]: debug 2022-01-31T22:19:52.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.751571+0000) 2022-01-31T22:19:52.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[60316]: debug 2022-01-31T22:19:52.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.505390+0000) 2022-01-31T22:19:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[41853]: debug 2022-01-31T22:19:52.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.453508+0000) 2022-01-31T22:19:53.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[54076]: debug 2022-01-31T22:19:52.893+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.893893+0000) 2022-01-31T22:19:53.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[60316]: debug 2022-01-31T22:19:52.892+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.893270+0000) 2022-01-31T22:19:53.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[49112]: debug 2022-01-31T22:19:53.171 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:52 smithi167 conmon[49112]: 2022-01-31T22:19:52.899+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.899856+0000) 2022-01-31T22:19:53.205 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[35325]: debug 2022-01-31T22:19:52.913+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 200890 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:53.206 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[41853]: debug 2022-01-31T22:19:52.891+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.892404+0000) 2022-01-31T22:19:53.207 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[46715]: debug 2022-01-31T22:19:52.892+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.893686+0000) 2022-01-31T22:19:53.207 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:52 smithi171 conmon[51620]: debug 2022-01-31T22:19:52.893+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:52.894617+0000) 2022-01-31T22:19:53.208 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:53 smithi171 conmon[51620]: debug 2022-01-31T22:19:53.105+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:53.106597+0000) 2022-01-31T22:19:53.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:53 smithi171 conmon[41853]: debug 2022-01-31T22:19:53.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:53.453721+0000) 2022-01-31T22:19:53.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:53 smithi171 conmon[46715]: debug 2022-01-31T22:19:53.245+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:53.246333+0000) 2022-01-31T22:19:53.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:53 smithi167 conmon[54076]: debug 2022-01-31T22:19:53.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:53.751719+0000) 2022-01-31T22:19:53.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:53 smithi167 conmon[60316]: debug 2022-01-31T22:19:53.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:53.505552+0000) 2022-01-31T22:19:53.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:53 smithi167 conmon[49112]: debug 2022-01-31T22:19:53.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:53.580309+0000) 2022-01-31T22:19:54.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:53 smithi171 conmon[35325]: debug 2022-01-31T22:19:53.833+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:19:54.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:54 smithi171 conmon[51620]: debug 2022-01-31T22:19:54.106+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:54.106741+0000) 2022-01-31T22:19:54.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:54 smithi171 conmon[46715]: debug 2022-01-31T22:19:54.245+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:54.246506+0000) 2022-01-31T22:19:54.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:54 smithi167 conmon[49112]: debug 2022-01-31T22:19:54.579+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:54.580437+0000) 2022-01-31T22:19:54.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:54 smithi167 conmon[54076]: debug 2022-01-31T22:19:54.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:54.751837+0000) 2022-01-31T22:19:54.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:54 smithi167 conmon[60316]: debug 2022-01-31T22:19:54.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:54.505754+0000) 2022-01-31T22:19:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:54 smithi171 conmon[41853]: debug 2022-01-31T22:19:54.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:54.453873+0000) 2022-01-31T22:19:55.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:55 smithi171 conmon[46715]: debug 2022-01-31T22:19:55.246+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:55.246712+0000) 2022-01-31T22:19:55.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:55 smithi171 conmon[51620]: debug 2022-01-31T22:19:55.106+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:55.106916+0000) 2022-01-31T22:19:55.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:55 smithi167 conmon[49112]: debug 2022-01-31T22:19:55.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:55.580627+0000) 2022-01-31T22:19:55.800 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:55 smithi167 conmon[54076]: debug 2022-01-31T22:19:55.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:55.751991+0000) 2022-01-31T22:19:55.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:55 smithi167 conmon[60316]: debug 2022-01-31T22:19:55.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:55.505926+0000) 2022-01-31T22:19:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:55 smithi171 conmon[41853]: debug 2022-01-31T22:19:55.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:55.454056+0000) 2022-01-31T22:19:56.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:56 smithi171 conmon[51620]: debug 2022-01-31T22:19:56.106+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:56.107086+0000) 2022-01-31T22:19:56.357 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:56 smithi171 conmon[46715]: debug 2022-01-31T22:19:56.246+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:56.246935+0000) 2022-01-31T22:19:56.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:56 smithi167 conmon[54076]: debug 2022-01-31T22:19:56.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:56.752149+0000) 2022-01-31T22:19:56.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:56 smithi167 conmon[60316]: debug 2022-01-31T22:19:56.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:56.506129+0000) 2022-01-31T22:19:56.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:56 smithi167 conmon[49112]: debug 2022-01-31T22:19:56.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:56.580758+0000) 2022-01-31T22:19:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:56 smithi171 conmon[41853]: debug 2022-01-31T22:19:56.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:56.454184+0000) 2022-01-31T22:19:57.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[46715]: debug 2022-01-31T22:19:57.246+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.247147+0000) 2022-01-31T22:19:57.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[51620]: debug 2022-01-31T22:19:57.106+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.107278+0000) 2022-01-31T22:19:57.799 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:57 smithi167 conmon[54076]: debug 2022-01-31T22:19:57.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.752288+0000) 2022-01-31T22:19:57.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:57 smithi167 conmon[60316]: debug 2022-01-31T22:19:57.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.506289+0000) 2022-01-31T22:19:57.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:57 smithi167 conmon[49112]: debug 2022-01-31T22:19:57.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.580922+0000) 2022-01-31T22:19:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[41853]: debug 2022-01-31T22:19:57.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.454355+0000) 2022-01-31T22:19:58.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:57 smithi167 conmon[49112]: debug 2022-01-31T22:19:57.917+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.917803+0000) 2022-01-31T22:19:58.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:57 smithi167 conmon[54076]: debug 2022-01-31T22:19:57.917+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.917517+0000) 2022-01-31T22:19:58.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:57 smithi167 conmon[60316]: debug 2022-01-31T22:19:57.916+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.916951+0000) 2022-01-31T22:19:58.207 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[35325]: debug 2022-01-31T22:19:57.937+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201000 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:19:58.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[41853]: debug 2022-01-31T22:19:57.915+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.916567+0000) 2022-01-31T22:19:58.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[46715]: debug 2022-01-31T22:19:57.916+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.917099+0000) 2022-01-31T22:19:58.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:57 smithi171 conmon[51620]: debug 2022-01-31T22:19:57.917+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:57.917874+0000) 2022-01-31T22:19:58.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:58 smithi171 conmon[51620]: debug 2022-01-31T22:19:58.106+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:58.107456+0000) 2022-01-31T22:19:58.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:58 smithi171 conmon[46715]: debug 2022-01-31T22:19:58.246+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:58.247351+0000) 2022-01-31T22:19:58.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:58 smithi171 conmon[41853]: debug 2022-01-31T22:19:58.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:58.454568+0000) 2022-01-31T22:19:58.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:58 smithi167 conmon[49112]: debug 2022-01-31T22:19:58.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:58.581150+0000) 2022-01-31T22:19:58.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:58 smithi167 conmon[54076]: debug 2022-01-31T22:19:58.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:58.752408+0000) 2022-01-31T22:19:58.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:58 smithi167 conmon[60316]: debug 2022-01-31T22:19:58.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:58.506448+0000) 2022-01-31T22:19:59.412 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:19:59 smithi171 conmon[46715]: debug 2022-01-31T22:19:59.247+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:59.247505+0000) 2022-01-31T22:19:59.413 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:19:59 smithi171 conmon[51620]: debug 2022-01-31T22:19:59.106+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:59.107592+0000) 2022-01-31T22:19:59.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:19:59 smithi167 conmon[49112]: debug 2022-01-31T22:19:59.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:59.581330+0000) 2022-01-31T22:19:59.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:19:59 smithi167 conmon[54076]: debug 2022-01-31T22:19:59.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:59.752525+0000) 2022-01-31T22:19:59.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:19:59 smithi167 conmon[60316]: debug 2022-01-31T22:19:59.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:59.506604+0000) 2022-01-31T22:19:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:19:59 smithi171 conmon[41853]: debug 2022-01-31T22:19:59.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:19:59.454715+0000) 2022-01-31T22:20:00.412 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:00 smithi171 conmon[46715]: debug 2022-01-31T22:20:00.247+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:00.247698+0000) 2022-01-31T22:20:00.413 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:00 smithi171 conmon[51620]: debug 2022-01-31T22:20:00.107+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:00.107822+0000) 2022-01-31T22:20:00.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:00 smithi167 conmon[49112]: debug 2022-01-31T22:20:00.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:00.581490+0000) 2022-01-31T22:20:00.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:00 smithi167 conmon[54076]: debug 2022-01-31T22:20:00.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:00.752713+0000) 2022-01-31T22:20:00.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:00 smithi167 conmon[60316]: debug 2022-01-31T22:20:00.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:00.506769+0000) 2022-01-31T22:20:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:00 smithi171 conmon[41853]: debug 2022-01-31T22:20:00.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:00.454893+0000) 2022-01-31T22:20:01.412 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:01 smithi171 conmon[46715]: debug 2022-01-31T22:20:01.247+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:01.247943+0000) 2022-01-31T22:20:01.413 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:01 smithi171 conmon[51620]: debug 2022-01-31T22:20:01.107+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:01.107968+0000) 2022-01-31T22:20:01.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:01 smithi167 conmon[49112]: debug 2022-01-31T22:20:01.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:01.581702+0000) 2022-01-31T22:20:01.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:01 smithi167 conmon[54076]: debug 2022-01-31T22:20:01.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:01.752903+0000) 2022-01-31T22:20:01.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:01 smithi167 conmon[60316]: debug 2022-01-31T22:20:01.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:01.507000+0000) 2022-01-31T22:20:01.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:01 smithi171 conmon[41853]: debug 2022-01-31T22:20:01.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:01.455098+0000) 2022-01-31T22:20:02.413 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[51620]: debug 2022-01-31T22:20:02.107+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.108091+0000) 2022-01-31T22:20:02.413 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[46715]: debug 2022-01-31T22:20:02.247+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.248101+0000) 2022-01-31T22:20:02.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:02 smithi167 conmon[49112]: debug 2022-01-31T22:20:02.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.581856+0000) 2022-01-31T22:20:02.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:02 smithi167 conmon[54076]: debug 2022-01-31T22:20:02.752+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.753083+0000) 2022-01-31T22:20:02.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:02 smithi167 conmon[60316]: debug 2022-01-31T22:20:02.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.507204+0000) 2022-01-31T22:20:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[41853]: debug 2022-01-31T22:20:02.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.455265+0000) 2022-01-31T22:20:03.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:02 smithi167 conmon[49112]: debug 2022-01-31T22:20:02.940+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.941637+0000) 2022-01-31T22:20:03.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:02 smithi167 conmon[54076]: debug 2022-01-31T22:20:02.941+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.942704+0000) 2022-01-31T22:20:03.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:02 smithi167 conmon[60316]: debug 2022-01-31T22:20:02.940+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.942195+0000) 2022-01-31T22:20:03.207 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[35325]: debug 2022-01-31T22:20:02.961+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:03.208 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[41853]: debug 2022-01-31T22:20:02.940+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.941209+0000) 2022-01-31T22:20:03.209 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[46715]: debug 2022-01-31T22:20:02.940+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.941033+0000) 2022-01-31T22:20:03.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:02 smithi171 conmon[51620]: debug 2022-01-31T22:20:02.940+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:02.941415+0000) 2022-01-31T22:20:03.209 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:03 smithi171 conmon[51620]: debug 2022-01-31T22:20:03.107+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:03.108263+0000) 2022-01-31T22:20:03.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:03 smithi171 conmon[41853]: debug 2022-01-31T22:20:03.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:03.455411+0000) 2022-01-31T22:20:03.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:03 smithi171 conmon[46715]: debug 2022-01-31T22:20:03.247+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:03.248300+0000) 2022-01-31T22:20:03.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:03 smithi167 conmon[49112]: debug 2022-01-31T22:20:03.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:03.582039+0000) 2022-01-31T22:20:03.801 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:03 smithi167 conmon[54076]: debug 2022-01-31T22:20:03.752+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:03.753263+0000) 2022-01-31T22:20:03.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:03 smithi167 conmon[60316]: debug 2022-01-31T22:20:03.505+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:03.507386+0000) 2022-01-31T22:20:04.413 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:04 smithi171 conmon[51620]: debug 2022-01-31T22:20:04.107+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:04.108429+0000) 2022-01-31T22:20:04.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:04 smithi171 conmon[46715]: debug 2022-01-31T22:20:04.248+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:04.248487+0000) 2022-01-31T22:20:04.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:04 smithi167 conmon[49112]: debug 2022-01-31T22:20:04.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:04.582195+0000) 2022-01-31T22:20:04.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:04 smithi167 conmon[54076]: debug 2022-01-31T22:20:04.751+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:04.753407+0000) 2022-01-31T22:20:04.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:04 smithi167 conmon[60316]: debug 2022-01-31T22:20:04.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:04.507568+0000) 2022-01-31T22:20:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:04 smithi171 conmon[41853]: debug 2022-01-31T22:20:04.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:04.455585+0000) 2022-01-31T22:20:05.413 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:05 smithi171 conmon[46715]: debug 2022-01-31T22:20:05.247+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:05.248618+0000) 2022-01-31T22:20:05.414 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:05 smithi171 conmon[51620]: debug 2022-01-31T22:20:05.108+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:05.108599+0000) 2022-01-31T22:20:05.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:05 smithi167 conmon[49112]: debug 2022-01-31T22:20:05.580+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:05.582375+0000) 2022-01-31T22:20:05.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:05 smithi167 conmon[54076]: debug 2022-01-31T22:20:05.752+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:05.753518+0000) 2022-01-31T22:20:05.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:05 smithi167 conmon[60316]: debug 2022-01-31T22:20:05.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:05.507766+0000) 2022-01-31T22:20:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:05 smithi171 conmon[41853]: debug 2022-01-31T22:20:05.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:05.455786+0000) 2022-01-31T22:20:06.413 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:06 smithi171 conmon[46715]: debug 2022-01-31T22:20:06.248+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:06.248775+0000) 2022-01-31T22:20:06.414 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:06 smithi171 conmon[51620]: debug 2022-01-31T22:20:06.108+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:06.108789+0000) 2022-01-31T22:20:06.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:06 smithi167 conmon[49112]: debug 2022-01-31T22:20:06.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:06.582506+0000) 2022-01-31T22:20:06.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:06 smithi167 conmon[54076]: debug 2022-01-31T22:20:06.752+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:06.753618+0000) 2022-01-31T22:20:06.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:06 smithi167 conmon[60316]: debug 2022-01-31T22:20:06.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:06.507978+0000) 2022-01-31T22:20:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:06 smithi171 conmon[41853]: debug 2022-01-31T22:20:06.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:06.455989+0000) 2022-01-31T22:20:07.413 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[46715]: debug 2022-01-31T22:20:07.248+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.248925+0000) 2022-01-31T22:20:07.414 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[51620]: debug 2022-01-31T22:20:07.108+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.108955+0000) 2022-01-31T22:20:07.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:07 smithi167 conmon[49112]: debug 2022-01-31T22:20:07.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.582668+0000) 2022-01-31T22:20:07.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:07 smithi167 conmon[54076]: debug 2022-01-31T22:20:07.752+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.753764+0000) 2022-01-31T22:20:07.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:07 smithi167 conmon[60316]: debug 2022-01-31T22:20:07.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.508144+0000) 2022-01-31T22:20:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[41853]: debug 2022-01-31T22:20:07.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.456175+0000) 2022-01-31T22:20:08.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:07 smithi167 conmon[49112]: debug 2022-01-31T22:20:07.963+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.964908+0000) 2022-01-31T22:20:08.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:07 smithi167 conmon[54076]: debug 2022-01-31T22:20:07.964+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.965795+0000) 2022-01-31T22:20:08.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:07 smithi167 conmon[60316]: debug 2022-01-31T22:20:07.964+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.965642+0000) 2022-01-31T22:20:08.248 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[35325]: debug 2022-01-31T22:20:07.984+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201222 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:08.249 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[41853]: debug 2022-01-31T22:20:07.963+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.964421+0000) 2022-01-31T22:20:08.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[46715]: debug 2022-01-31T22:20:07.963+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.963746+0000) 2022-01-31T22:20:08.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:07 smithi171 conmon[51620]: debug 2022-01-31T22:20:07.963+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:07.964287+0000) 2022-01-31T22:20:08.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:08 smithi171 conmon[51620]: debug 2022-01-31T22:20:08.108+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:08.109127+0000) 2022-01-31T22:20:08.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:08 smithi171 conmon[46715]: debug 2022-01-31T22:20:08.248+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:08.249060+0000) 2022-01-31T22:20:08.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:08 smithi171 conmon[41853]: debug 2022-01-31T22:20:08.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:08.456323+0000) 2022-01-31T22:20:08.801 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:08 smithi167 conmon[49112]: debug 2022-01-31T22:20:08.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:08.582888+0000) 2022-01-31T22:20:08.802 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:08 smithi167 conmon[60316]: debug 2022-01-31T22:20:08.506+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:08.508340+0000) 2022-01-31T22:20:08.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:08 smithi167 conmon[54076]: debug 2022-01-31T22:20:08.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:08.753931+0000) 2022-01-31T22:20:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:08 smithi171 conmon[35325]: debug 2022-01-31T22:20:08.834+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:20:09.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:09 smithi171 conmon[46715]: debug 2022-01-31T22:20:09.248+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:09.249167+0000) 2022-01-31T22:20:09.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:09 smithi171 conmon[51620]: debug 2022-01-31T22:20:09.108+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:09.109242+0000) 2022-01-31T22:20:09.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:09 smithi167 conmon[49112]: debug 2022-01-31T22:20:09.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:09.583081+0000) 2022-01-31T22:20:09.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:09 smithi167 conmon[60316]: debug 2022-01-31T22:20:09.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:09.508455+0000) 2022-01-31T22:20:09.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:09 smithi167 conmon[54076]: debug 2022-01-31T22:20:09.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:09.754089+0000) 2022-01-31T22:20:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:09 smithi171 conmon[41853]: debug 2022-01-31T22:20:09.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:09.456458+0000) 2022-01-31T22:20:10.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:10 smithi171 conmon[46715]: debug 2022-01-31T22:20:10.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:10.249342+0000) 2022-01-31T22:20:10.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:10 smithi171 conmon[51620]: debug 2022-01-31T22:20:10.108+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:10.109415+0000) 2022-01-31T22:20:10.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:10 smithi167 conmon[49112]: debug 2022-01-31T22:20:10.581+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:10.583239+0000) 2022-01-31T22:20:10.802 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:10 smithi167 conmon[54076]: debug 2022-01-31T22:20:10.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:10.754310+0000) 2022-01-31T22:20:10.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:10 smithi167 conmon[60316]: debug 2022-01-31T22:20:10.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:10.508610+0000) 2022-01-31T22:20:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:10 smithi171 conmon[41853]: debug 2022-01-31T22:20:10.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:10.456665+0000) 2022-01-31T22:20:11.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:11 smithi171 conmon[46715]: debug 2022-01-31T22:20:11.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:11.249514+0000) 2022-01-31T22:20:11.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:11 smithi171 conmon[51620]: debug 2022-01-31T22:20:11.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:11.109587+0000) 2022-01-31T22:20:11.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:11 smithi167 conmon[49112]: debug 2022-01-31T22:20:11.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:11.583393+0000) 2022-01-31T22:20:11.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:11 smithi167 conmon[54076]: debug 2022-01-31T22:20:11.752+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:11.754436+0000) 2022-01-31T22:20:11.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:11 smithi167 conmon[60316]: debug 2022-01-31T22:20:11.507+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:11.508792+0000) 2022-01-31T22:20:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:11 smithi171 conmon[41853]: debug 2022-01-31T22:20:11.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:11.456850+0000) 2022-01-31T22:20:12.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:12 smithi171 conmon[46715]: debug 2022-01-31T22:20:12.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.249672+0000) 2022-01-31T22:20:12.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:12 smithi171 conmon[51620]: debug 2022-01-31T22:20:12.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.109729+0000) 2022-01-31T22:20:12.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:12 smithi167 conmon[49112]: debug 2022-01-31T22:20:12.582+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.583584+0000) 2022-01-31T22:20:12.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:12 smithi167 conmon[54076]: debug 2022-01-31T22:20:12.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.754581+0000) 2022-01-31T22:20:12.803 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:12 smithi167 conmon[60316]: debug 2022-01-31T22:20:12.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.508996+0000) 2022-01-31T22:20:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:12 smithi171 conmon[41853]: debug 2022-01-31T22:20:12.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.457043+0000) 2022-01-31T22:20:13.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:12 smithi167 conmon[49112]: debug 2022-01-31T22:20:12.986+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.987727+0000) 2022-01-31T22:20:13.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:12 smithi167 conmon[54076]: debug 2022-01-31T22:20:12.986+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.987604+0000) 2022-01-31T22:20:13.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:12 smithi167 conmon[60316]: debug 2022-01-31T22:20:12.986+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.988409+0000) 2022-01-31T22:20:13.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:13 smithi171 conmon[35325]: debug 2022-01-31T22:20:13.007+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201335 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:13.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:12 smithi171 conmon[41853]: debug 2022-01-31T22:20:12.986+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.986928+0000) 2022-01-31T22:20:13.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:12 smithi171 conmon[46715]: debug 2022-01-31T22:20:12.987+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.987937+0000) 2022-01-31T22:20:13.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:12 smithi171 conmon[51620]: debug 2022-01-31T22:20:12.986+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:12.987621+0000) 2022-01-31T22:20:13.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:13 smithi171 conmon[51620]: debug 2022-01-31T22:20:13.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:13.109869+0000) 2022-01-31T22:20:13.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:13 smithi171 conmon[46715]: debug 2022-01-31T22:20:13.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:13.249798+0000) 2022-01-31T22:20:13.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:13 smithi171 conmon[41853]: debug 2022-01-31T22:20:13.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:13.457209+0000) 2022-01-31T22:20:13.802 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:13 smithi167 conmon[49112]: debug 2022-01-31T22:20:13.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:13.583773+0000) 2022-01-31T22:20:13.803 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:13 smithi167 conmon[54076]: debug 2022-01-31T22:20:13.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:13.754732+0000) 2022-01-31T22:20:13.804 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:13 smithi167 conmon[60316]: debug 2022-01-31T22:20:13.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:13.509184+0000) 2022-01-31T22:20:14.415 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:14 smithi171 conmon[46715]: debug 2022-01-31T22:20:14.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:14.249954+0000) 2022-01-31T22:20:14.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:14 smithi171 conmon[51620]: debug 2022-01-31T22:20:14.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:14.109966+0000) 2022-01-31T22:20:14.754 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:14 smithi167 conmon[49112]: debug 2022-01-31T22:20:14.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:14.583921+0000) 2022-01-31T22:20:14.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:14 smithi167 conmon[60316]: debug 2022-01-31T22:20:14.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:14.509365+0000) 2022-01-31T22:20:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:14 smithi171 conmon[41853]: debug 2022-01-31T22:20:14.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:14.457367+0000) 2022-01-31T22:20:15.057 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:14 smithi167 conmon[54076]: debug 2022-01-31T22:20:14.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:14.754829+0000) 2022-01-31T22:20:15.415 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:15 smithi171 conmon[46715]: debug 2022-01-31T22:20:15.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:15.250113+0000) 2022-01-31T22:20:15.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:15 smithi171 conmon[51620]: debug 2022-01-31T22:20:15.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:15.110170+0000) 2022-01-31T22:20:15.754 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:15 smithi167 conmon[49112]: debug 2022-01-31T22:20:15.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:15.584104+0000) 2022-01-31T22:20:15.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:15 smithi167 conmon[60316]: debug 2022-01-31T22:20:15.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:15.509618+0000) 2022-01-31T22:20:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:15 smithi171 conmon[41853]: debug 2022-01-31T22:20:15.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:15.457558+0000) 2022-01-31T22:20:16.060 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:15 smithi167 conmon[54076]: debug 2022-01-31T22:20:15.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:15.755026+0000) 2022-01-31T22:20:16.415 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:16 smithi171 conmon[46715]: debug 2022-01-31T22:20:16.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:16.250296+0000) 2022-01-31T22:20:16.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:16 smithi171 conmon[51620]: debug 2022-01-31T22:20:16.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:16.110417+0000) 2022-01-31T22:20:16.754 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:16 smithi167 conmon[49112]: debug 2022-01-31T22:20:16.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:16.584301+0000) 2022-01-31T22:20:16.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:16 smithi167 conmon[60316]: debug 2022-01-31T22:20:16.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:16.509776+0000) 2022-01-31T22:20:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:16 smithi171 conmon[41853]: debug 2022-01-31T22:20:16.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:16.457714+0000) 2022-01-31T22:20:17.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:16 smithi167 conmon[54076]: debug 2022-01-31T22:20:16.754+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:16.755202+0000) 2022-01-31T22:20:17.385 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:17 smithi171 conmon[51620]: debug 2022-01-31T22:20:17.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:17.110580+0000) 2022-01-31T22:20:17.386 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:17 smithi171 conmon[46715]: debug 2022-01-31T22:20:17.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:17.250518+0000) 2022-01-31T22:20:17.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:17 smithi167 conmon[49112]: debug 2022-01-31T22:20:17.583+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:17.584479+0000) 2022-01-31T22:20:17.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:17 smithi167 conmon[60316]: debug 2022-01-31T22:20:17.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:17.509935+0000) 2022-01-31T22:20:17.914 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:17 smithi171 conmon[41853]: debug 2022-01-31T22:20:17.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:17.457894+0000) 2022-01-31T22:20:18.010 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:17 smithi167 conmon[54076]: debug 2022-01-31T22:20:17.753+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:17.755408+0000) 2022-01-31T22:20:18.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[46715]: debug 2022-01-31T22:20:18.010+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.011187+0000) 2022-01-31T22:20:18.307 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[46715]: debug 2022-01-31T22:20:18.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.250667+0000) 2022-01-31T22:20:18.307 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[51620]: debug 2022-01-31T22:20:18.010+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.011664+0000) 2022-01-31T22:20:18.308 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[51620]: debug 2022-01-31T22:20:18.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.110714+0000) 2022-01-31T22:20:18.308 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[35325]: debug 2022-01-31T22:20:18.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201445 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:18.309 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[41853]: debug 2022-01-31T22:20:18.010+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.010852+0000) 2022-01-31T22:20:18.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:18 smithi167 conmon[49112]: debug 2022-01-31T22:20:18.009+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.011364+0000) 2022-01-31T22:20:18.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:18 smithi167 conmon[54076]: debug 2022-01-31T22:20:18.009+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.010708+0000) 2022-01-31T22:20:18.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:18 smithi167 conmon[60316]: debug 2022-01-31T22:20:18.011+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.012540+0000) 2022-01-31T22:20:18.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:18 smithi167 conmon[49112]: debug 2022-01-31T22:20:18.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.584657+0000) 2022-01-31T22:20:18.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:18 smithi167 conmon[60316]: debug 2022-01-31T22:20:18.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.510136+0000) 2022-01-31T22:20:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:18 smithi171 conmon[41853]: debug 2022-01-31T22:20:18.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.458072+0000) 2022-01-31T22:20:19.067 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:18 smithi167 conmon[54076]: debug 2022-01-31T22:20:18.754+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:18.755564+0000) 2022-01-31T22:20:19.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:19 smithi171 conmon[46715]: debug 2022-01-31T22:20:19.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:19.250806+0000) 2022-01-31T22:20:19.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:19 smithi171 conmon[51620]: debug 2022-01-31T22:20:19.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:19.110862+0000) 2022-01-31T22:20:19.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:19 smithi167 conmon[49112]: debug 2022-01-31T22:20:19.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:19.584828+0000) 2022-01-31T22:20:19.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:19 smithi167 conmon[60316]: debug 2022-01-31T22:20:19.508+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:19.510311+0000) 2022-01-31T22:20:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:19 smithi171 conmon[41853]: debug 2022-01-31T22:20:19.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:19.458206+0000) 2022-01-31T22:20:20.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:19 smithi167 conmon[54076]: debug 2022-01-31T22:20:19.755+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:19.755709+0000) 2022-01-31T22:20:20.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:20 smithi171 conmon[46715]: debug 2022-01-31T22:20:20.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:20.250995+0000) 2022-01-31T22:20:20.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:20 smithi171 conmon[51620]: debug 2022-01-31T22:20:20.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:20.111061+0000) 2022-01-31T22:20:20.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:20 smithi167 conmon[49112]: debug 2022-01-31T22:20:20.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:20.585054+0000) 2022-01-31T22:20:20.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:20 smithi167 conmon[60316]: debug 2022-01-31T22:20:20.509+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:20.510496+0000) 2022-01-31T22:20:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:20 smithi171 conmon[41853]: debug 2022-01-31T22:20:20.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:20.458358+0000) 2022-01-31T22:20:21.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:20 smithi167 conmon[54076]: debug 2022-01-31T22:20:20.755+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:20.755912+0000) 2022-01-31T22:20:21.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:21 smithi171 conmon[46715]: debug 2022-01-31T22:20:21.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:21.251200+0000) 2022-01-31T22:20:21.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:21 smithi171 conmon[51620]: debug 2022-01-31T22:20:21.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:21.111216+0000) 2022-01-31T22:20:21.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:21 smithi167 conmon[49112]: debug 2022-01-31T22:20:21.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:21.585237+0000) 2022-01-31T22:20:21.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:21 smithi167 conmon[60316]: debug 2022-01-31T22:20:21.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:21.510694+0000) 2022-01-31T22:20:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:21 smithi171 conmon[41853]: debug 2022-01-31T22:20:21.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:21.458541+0000) 2022-01-31T22:20:22.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:21 smithi167 conmon[54076]: debug 2022-01-31T22:20:21.755+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:21.756102+0000) 2022-01-31T22:20:22.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:22 smithi171 conmon[46715]: debug 2022-01-31T22:20:22.249+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:22.251347+0000) 2022-01-31T22:20:22.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:22 smithi171 conmon[51620]: debug 2022-01-31T22:20:22.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:22.111405+0000) 2022-01-31T22:20:22.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:22 smithi167 conmon[49112]: debug 2022-01-31T22:20:22.584+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:22.585420+0000) 2022-01-31T22:20:22.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:22 smithi167 conmon[60316]: debug 2022-01-31T22:20:22.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:22.510855+0000) 2022-01-31T22:20:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:22 smithi171 conmon[41853]: debug 2022-01-31T22:20:22.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:22.458730+0000) 2022-01-31T22:20:23.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:22 smithi167 conmon[54076]: debug 2022-01-31T22:20:22.755+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:22.756277+0000) 2022-01-31T22:20:23.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[41853]: debug 2022-01-31T22:20:23.033+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.035241+0000) 2022-01-31T22:20:23.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[35325]: debug 2022-01-31T22:20:23.053+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201558 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:23.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[46715]: debug 2022-01-31T22:20:23.032+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.034574+0000) 2022-01-31T22:20:23.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[46715]: debug 2022-01-31T22:20:23.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.251500+0000) 2022-01-31T22:20:23.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[51620]: debug 2022-01-31T22:20:23.033+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.034819+0000) 2022-01-31T22:20:23.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[51620]: debug 2022-01-31T22:20:23.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.111594+0000) 2022-01-31T22:20:23.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:23 smithi167 conmon[49112]: debug 2022-01-31T22:20:23.033+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.034728+0000) 2022-01-31T22:20:23.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:23 smithi167 conmon[54076]: debug 2022-01-31T22:20:23.034+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.036130+0000) 2022-01-31T22:20:23.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:23 smithi167 conmon[60316]: debug 2022-01-31T22:20:23.034+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.036264+0000) 2022-01-31T22:20:23.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:23 smithi167 conmon[49112]: debug 2022-01-31T22:20:23.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.585667+0000) 2022-01-31T22:20:23.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:23 smithi167 conmon[60316]: debug 2022-01-31T22:20:23.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.511038+0000) 2022-01-31T22:20:23.837 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[41853]: debug 2022-01-31T22:20:23.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.458944+0000) 2022-01-31T22:20:24.081 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:23 smithi167 conmon[54076]: debug 2022-01-31T22:20:23.755+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:23.756507+0000) 2022-01-31T22:20:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:23 smithi171 conmon[35325]: debug 2022-01-31T22:20:23.834+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:20:24.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:24 smithi171 conmon[46715]: debug 2022-01-31T22:20:24.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:24.251662+0000) 2022-01-31T22:20:24.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:24 smithi171 conmon[51620]: debug 2022-01-31T22:20:24.109+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:24.111731+0000) 2022-01-31T22:20:24.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:24 smithi167 conmon[49112]: debug 2022-01-31T22:20:24.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:24.585820+0000) 2022-01-31T22:20:24.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:24 smithi167 conmon[60316]: debug 2022-01-31T22:20:24.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:24.511218+0000) 2022-01-31T22:20:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:24 smithi171 conmon[41853]: debug 2022-01-31T22:20:24.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:24.459084+0000) 2022-01-31T22:20:25.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:24 smithi167 conmon[54076]: debug 2022-01-31T22:20:24.756+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:24.756631+0000) 2022-01-31T22:20:25.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:25 smithi171 conmon[46715]: debug 2022-01-31T22:20:25.250+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:25.251828+0000) 2022-01-31T22:20:25.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:25 smithi171 conmon[51620]: debug 2022-01-31T22:20:25.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:25.111911+0000) 2022-01-31T22:20:25.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:25 smithi167 conmon[49112]: debug 2022-01-31T22:20:25.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:25.585949+0000) 2022-01-31T22:20:25.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:25 smithi167 conmon[60316]: debug 2022-01-31T22:20:25.510+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:25.511427+0000) 2022-01-31T22:20:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:25 smithi171 conmon[41853]: debug 2022-01-31T22:20:25.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:25.459212+0000) 2022-01-31T22:20:26.088 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:25 smithi167 conmon[54076]: debug 2022-01-31T22:20:25.756+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:25.756790+0000) 2022-01-31T22:20:26.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:26 smithi171 conmon[46715]: debug 2022-01-31T22:20:26.251+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:26.251987+0000) 2022-01-31T22:20:26.418 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:26 smithi171 conmon[51620]: debug 2022-01-31T22:20:26.110+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:26.112040+0000) 2022-01-31T22:20:26.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:26 smithi167 conmon[60316]: debug 2022-01-31T22:20:26.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:26.511593+0000) 2022-01-31T22:20:26.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:26 smithi167 conmon[49112]: debug 2022-01-31T22:20:26.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:26.586047+0000) 2022-01-31T22:20:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:26 smithi171 conmon[41853]: debug 2022-01-31T22:20:26.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:26.459383+0000) 2022-01-31T22:20:27.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:26 smithi167 conmon[54076]: debug 2022-01-31T22:20:26.756+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:26.756986+0000) 2022-01-31T22:20:27.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:27 smithi171 conmon[46715]: debug 2022-01-31T22:20:27.251+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:27.252173+0000) 2022-01-31T22:20:27.418 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:27 smithi171 conmon[51620]: debug 2022-01-31T22:20:27.111+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:27.112234+0000) 2022-01-31T22:20:27.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:27 smithi167 conmon[49112]: debug 2022-01-31T22:20:27.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:27.586205+0000) 2022-01-31T22:20:27.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:27 smithi167 conmon[60316]: debug 2022-01-31T22:20:27.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:27.511785+0000) 2022-01-31T22:20:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:27 smithi171 conmon[41853]: debug 2022-01-31T22:20:27.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:27.459537+0000) 2022-01-31T22:20:28.057 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:27 smithi167 conmon[54076]: debug 2022-01-31T22:20:27.756+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:27.757192+0000) 2022-01-31T22:20:28.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[41853]: debug 2022-01-31T22:20:28.055+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.057386+0000) 2022-01-31T22:20:28.358 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[35325]: debug 2022-01-31T22:20:28.077+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201668 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:28.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[46715]: debug 2022-01-31T22:20:28.055+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.057173+0000) 2022-01-31T22:20:28.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[46715]: debug 2022-01-31T22:20:28.251+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.252330+0000) 2022-01-31T22:20:28.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[51620]: debug 2022-01-31T22:20:28.056+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.058111+0000) 2022-01-31T22:20:28.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[51620]: debug 2022-01-31T22:20:28.111+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.112392+0000) 2022-01-31T22:20:28.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:28 smithi167 conmon[49112]: debug 2022-01-31T22:20:28.057+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.058257+0000) 2022-01-31T22:20:28.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:28 smithi167 conmon[54076]: debug 2022-01-31T22:20:28.057+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.058142+0000) 2022-01-31T22:20:28.421 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:28 smithi167 conmon[60316]: debug 2022-01-31T22:20:28.057+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.058451+0000) 2022-01-31T22:20:28.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:28 smithi167 conmon[49112]: debug 2022-01-31T22:20:28.585+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.586360+0000) 2022-01-31T22:20:28.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:28 smithi167 conmon[60316]: debug 2022-01-31T22:20:28.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.511980+0000) 2022-01-31T22:20:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:28 smithi171 conmon[41853]: debug 2022-01-31T22:20:28.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.459668+0000) 2022-01-31T22:20:29.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:28 smithi167 conmon[54076]: debug 2022-01-31T22:20:28.756+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:28.757346+0000) 2022-01-31T22:20:29.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:29 smithi171 conmon[46715]: debug 2022-01-31T22:20:29.251+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:29.252469+0000) 2022-01-31T22:20:29.418 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:29 smithi171 conmon[51620]: debug 2022-01-31T22:20:29.111+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:29.112555+0000) 2022-01-31T22:20:29.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:29 smithi167 conmon[49112]: debug 2022-01-31T22:20:29.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:29.586553+0000) 2022-01-31T22:20:29.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:29 smithi167 conmon[60316]: debug 2022-01-31T22:20:29.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:29.512149+0000) 2022-01-31T22:20:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:29 smithi171 conmon[41853]: debug 2022-01-31T22:20:29.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:29.459815+0000) 2022-01-31T22:20:30.098 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:29 smithi167 conmon[54076]: debug 2022-01-31T22:20:29.757+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:29.757531+0000) 2022-01-31T22:20:30.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:30 smithi171 conmon[46715]: debug 2022-01-31T22:20:30.251+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:30.252671+0000) 2022-01-31T22:20:30.418 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:30 smithi171 conmon[51620]: debug 2022-01-31T22:20:30.111+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:30.112708+0000) 2022-01-31T22:20:30.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:30 smithi167 conmon[49112]: debug 2022-01-31T22:20:30.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:30.586727+0000) 2022-01-31T22:20:30.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:30 smithi167 conmon[60316]: debug 2022-01-31T22:20:30.511+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:30.512308+0000) 2022-01-31T22:20:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:30 smithi171 conmon[41853]: debug 2022-01-31T22:20:30.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:30.459947+0000) 2022-01-31T22:20:31.101 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:30 smithi167 conmon[54076]: debug 2022-01-31T22:20:30.757+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:30.757722+0000) 2022-01-31T22:20:31.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:31 smithi171 conmon[46715]: debug 2022-01-31T22:20:31.252+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:31.252825+0000) 2022-01-31T22:20:31.418 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:31 smithi171 conmon[51620]: debug 2022-01-31T22:20:31.112+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:31.112855+0000) 2022-01-31T22:20:31.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:31 smithi167 conmon[49112]: debug 2022-01-31T22:20:31.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:31.586945+0000) 2022-01-31T22:20:31.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:31 smithi167 conmon[60316]: debug 2022-01-31T22:20:31.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:31.512522+0000) 2022-01-31T22:20:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:31 smithi171 conmon[41853]: debug 2022-01-31T22:20:31.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:31.460079+0000) 2022-01-31T22:20:32.104 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:31 smithi167 conmon[54076]: debug 2022-01-31T22:20:31.757+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:31.757901+0000) 2022-01-31T22:20:32.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:32 smithi171 conmon[46715]: debug 2022-01-31T22:20:32.252+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:32.253010+0000) 2022-01-31T22:20:32.419 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:32 smithi171 conmon[51620]: debug 2022-01-31T22:20:32.112+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:32.113019+0000) 2022-01-31T22:20:32.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:32 smithi167 conmon[49112]: debug 2022-01-31T22:20:32.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:32.587159+0000) 2022-01-31T22:20:32.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:32 smithi167 conmon[60316]: debug 2022-01-31T22:20:32.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:32.512629+0000) 2022-01-31T22:20:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:32 smithi171 conmon[41853]: debug 2022-01-31T22:20:32.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:32.460208+0000) 2022-01-31T22:20:33.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:32 smithi167 conmon[54076]: debug 2022-01-31T22:20:32.757+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:32.758099+0000) 2022-01-31T22:20:33.082 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[41853]: debug 2022-01-31T22:20:33.080+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.081484+0000) 2022-01-31T22:20:33.083 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[46715]: debug 2022-01-31T22:20:33.080+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.080938+0000) 2022-01-31T22:20:33.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[35325]: debug 2022-01-31T22:20:33.102+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201781 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:33.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[46715]: debug 2022-01-31T22:20:33.252+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.253209+0000) 2022-01-31T22:20:33.358 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[51620]: debug 2022-01-31T22:20:33.081+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.082280+0000) 2022-01-31T22:20:33.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[51620]: debug 2022-01-31T22:20:33.112+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.113145+0000) 2022-01-31T22:20:33.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:33 smithi167 conmon[49112]: debug 2022-01-31T22:20:33.081+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.081727+0000) 2022-01-31T22:20:33.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:33 smithi167 conmon[54076]: debug 2022-01-31T22:20:33.081+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.081637+0000) 2022-01-31T22:20:33.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:33 smithi167 conmon[60316]: debug 2022-01-31T22:20:33.082+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.082828+0000) 2022-01-31T22:20:33.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:33 smithi167 conmon[49112]: debug 2022-01-31T22:20:33.586+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.587307+0000) 2022-01-31T22:20:33.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:33 smithi167 conmon[60316]: debug 2022-01-31T22:20:33.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.512826+0000) 2022-01-31T22:20:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:33 smithi171 conmon[41853]: debug 2022-01-31T22:20:33.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.460399+0000) 2022-01-31T22:20:34.109 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:33 smithi167 conmon[54076]: debug 2022-01-31T22:20:33.757+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:33.758258+0000) 2022-01-31T22:20:34.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:34 smithi171 conmon[46715]: debug 2022-01-31T22:20:34.252+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:34.253385+0000) 2022-01-31T22:20:34.419 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:34 smithi171 conmon[51620]: debug 2022-01-31T22:20:34.112+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:34.113281+0000) 2022-01-31T22:20:34.757 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:34 smithi167 conmon[49112]: debug 2022-01-31T22:20:34.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:34.587476+0000) 2022-01-31T22:20:34.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:34 smithi167 conmon[60316]: debug 2022-01-31T22:20:34.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:34.513032+0000) 2022-01-31T22:20:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:34 smithi171 conmon[41853]: debug 2022-01-31T22:20:34.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:34.460559+0000) 2022-01-31T22:20:35.112 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:34 smithi167 conmon[54076]: debug 2022-01-31T22:20:34.757+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:34.758385+0000) 2022-01-31T22:20:35.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:35 smithi171 conmon[46715]: debug 2022-01-31T22:20:35.252+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:35.253569+0000) 2022-01-31T22:20:35.419 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:35 smithi171 conmon[51620]: debug 2022-01-31T22:20:35.112+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:35.113443+0000) 2022-01-31T22:20:35.758 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:35 smithi167 conmon[49112]: debug 2022-01-31T22:20:35.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:35.587709+0000) 2022-01-31T22:20:35.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:35 smithi167 conmon[60316]: debug 2022-01-31T22:20:35.512+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:35.513232+0000) 2022-01-31T22:20:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:35 smithi171 conmon[41853]: debug 2022-01-31T22:20:35.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:35.460716+0000) 2022-01-31T22:20:36.116 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:35 smithi167 conmon[54076]: debug 2022-01-31T22:20:35.758+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:35.758518+0000) 2022-01-31T22:20:36.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:36 smithi171 conmon[46715]: debug 2022-01-31T22:20:36.253+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:36.253745+0000) 2022-01-31T22:20:36.419 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:36 smithi171 conmon[51620]: debug 2022-01-31T22:20:36.112+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:36.113626+0000) 2022-01-31T22:20:36.758 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:36 smithi167 conmon[49112]: debug 2022-01-31T22:20:36.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:36.587845+0000) 2022-01-31T22:20:36.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:36 smithi167 conmon[60316]: debug 2022-01-31T22:20:36.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:36.513405+0000) 2022-01-31T22:20:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:36 smithi171 conmon[41853]: debug 2022-01-31T22:20:36.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:36.460930+0000) 2022-01-31T22:20:37.119 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:36 smithi167 conmon[54076]: debug 2022-01-31T22:20:36.758+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:36.758670+0000) 2022-01-31T22:20:37.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:37 smithi171 conmon[46715]: debug 2022-01-31T22:20:37.253+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:37.253983+0000) 2022-01-31T22:20:37.420 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:37 smithi171 conmon[51620]: debug 2022-01-31T22:20:37.113+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:37.113796+0000) 2022-01-31T22:20:37.758 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:37 smithi167 conmon[49112]: debug 2022-01-31T22:20:37.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:37.588077+0000) 2022-01-31T22:20:37.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:37 smithi167 conmon[60316]: debug 2022-01-31T22:20:37.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:37.513678+0000) 2022-01-31T22:20:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:37 smithi171 conmon[41853]: debug 2022-01-31T22:20:37.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:37.461173+0000) 2022-01-31T22:20:38.105 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:37 smithi167 conmon[54076]: debug 2022-01-31T22:20:37.758+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:37.758871+0000) 2022-01-31T22:20:38.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[35325]: debug 2022-01-31T22:20:38.125+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 201892 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:38.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[41853]: debug 2022-01-31T22:20:38.104+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.104972+0000) 2022-01-31T22:20:38.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[46715]: debug 2022-01-31T22:20:38.105+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.106262+0000) 2022-01-31T22:20:38.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[46715]: debug 2022-01-31T22:20:38.253+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.254121+0000) 2022-01-31T22:20:38.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[51620]: debug 2022-01-31T22:20:38.106+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.106956+0000) 2022-01-31T22:20:38.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[51620]: debug 2022-01-31T22:20:38.113+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.113939+0000) 2022-01-31T22:20:38.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:38 smithi167 conmon[60316]: debug 2022-01-31T22:20:38.106+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.107074+0000) 2022-01-31T22:20:38.420 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:38 smithi167 conmon[49112]: debug 2022-01-31T22:20:38.106+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.106742+0000) 2022-01-31T22:20:38.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:38 smithi167 conmon[54076]: debug 2022-01-31T22:20:38.106+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.106847+0000) 2022-01-31T22:20:38.758 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:38 smithi167 conmon[60316]: debug 2022-01-31T22:20:38.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.513852+0000) 2022-01-31T22:20:38.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:38 smithi167 conmon[49112]: debug 2022-01-31T22:20:38.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.588235+0000) 2022-01-31T22:20:38.835 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[41853]: debug 2022-01-31T22:20:38.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.461333+0000) 2022-01-31T22:20:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:38 smithi171 conmon[35325]: debug 2022-01-31T22:20:38.835+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:20:39.124 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:38 smithi167 conmon[54076]: debug 2022-01-31T22:20:38.758+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:38.759018+0000) 2022-01-31T22:20:39.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:39 smithi171 conmon[46715]: debug 2022-01-31T22:20:39.253+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:39.254294+0000) 2022-01-31T22:20:39.420 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:39 smithi171 conmon[51620]: debug 2022-01-31T22:20:39.113+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:39.114050+0000) 2022-01-31T22:20:39.758 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:39 smithi167 conmon[49112]: debug 2022-01-31T22:20:39.587+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:39.588405+0000) 2022-01-31T22:20:39.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:39 smithi167 conmon[60316]: debug 2022-01-31T22:20:39.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:39.514002+0000) 2022-01-31T22:20:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:39 smithi171 conmon[41853]: debug 2022-01-31T22:20:39.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:39.461505+0000) 2022-01-31T22:20:40.128 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:39 smithi167 conmon[54076]: debug 2022-01-31T22:20:39.758+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:39.759127+0000) 2022-01-31T22:20:40.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:40 smithi171 conmon[46715]: debug 2022-01-31T22:20:40.253+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:40.254500+0000) 2022-01-31T22:20:40.420 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:40 smithi171 conmon[51620]: debug 2022-01-31T22:20:40.113+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:40.114232+0000) 2022-01-31T22:20:40.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:40 smithi167 conmon[49112]: debug 2022-01-31T22:20:40.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:40.588594+0000) 2022-01-31T22:20:40.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:40 smithi167 conmon[60316]: debug 2022-01-31T22:20:40.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:40.514217+0000) 2022-01-31T22:20:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:40 smithi171 conmon[41853]: debug 2022-01-31T22:20:40.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:40.461707+0000) 2022-01-31T22:20:41.132 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:40 smithi167 conmon[54076]: debug 2022-01-31T22:20:40.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:40.759315+0000) 2022-01-31T22:20:41.419 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:41 smithi171 conmon[46715]: debug 2022-01-31T22:20:41.253+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:41.254688+0000) 2022-01-31T22:20:41.420 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:41 smithi171 conmon[51620]: debug 2022-01-31T22:20:41.113+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:41.114416+0000) 2022-01-31T22:20:41.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:41 smithi167 conmon[49112]: debug 2022-01-31T22:20:41.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:41.588784+0000) 2022-01-31T22:20:41.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:41 smithi167 conmon[60316]: debug 2022-01-31T22:20:41.513+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:41.514432+0000) 2022-01-31T22:20:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:41 smithi171 conmon[41853]: debug 2022-01-31T22:20:41.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:41.461913+0000) 2022-01-31T22:20:42.134 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:41 smithi167 conmon[54076]: debug 2022-01-31T22:20:41.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:41.759514+0000) 2022-01-31T22:20:42.420 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:42 smithi171 conmon[46715]: debug 2022-01-31T22:20:42.254+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:42.254844+0000) 2022-01-31T22:20:42.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:42 smithi171 conmon[51620]: debug 2022-01-31T22:20:42.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:42.114589+0000) 2022-01-31T22:20:42.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:42 smithi167 conmon[49112]: debug 2022-01-31T22:20:42.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:42.588944+0000) 2022-01-31T22:20:42.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:42 smithi167 conmon[60316]: debug 2022-01-31T22:20:42.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:42.514594+0000) 2022-01-31T22:20:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:42 smithi171 conmon[41853]: debug 2022-01-31T22:20:42.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:42.462123+0000) 2022-01-31T22:20:43.128 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:42 smithi167 conmon[54076]: debug 2022-01-31T22:20:42.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:42.759673+0000) 2022-01-31T22:20:43.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:43 smithi167 conmon[49112]: debug 2022-01-31T22:20:43.129+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.129519+0000) 2022-01-31T22:20:43.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:43 smithi167 conmon[54076]: debug 2022-01-31T22:20:43.129+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.130021+0000) 2022-01-31T22:20:43.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:43 smithi167 conmon[60316]: debug 2022-01-31T22:20:43.129+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.129838+0000) 2022-01-31T22:20:43.421 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[35325]: debug 2022-01-31T22:20:43.150+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202004 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:43.421 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[41853]: debug 2022-01-31T22:20:43.127+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.128361+0000) 2022-01-31T22:20:43.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[46715]: debug 2022-01-31T22:20:43.129+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.130265+0000) 2022-01-31T22:20:43.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[46715]: debug 2022-01-31T22:20:43.254+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.255036+0000) 2022-01-31T22:20:43.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[51620]: debug 2022-01-31T22:20:43.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.114733+0000) 2022-01-31T22:20:43.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[51620]: debug 2022-01-31T22:20:43.129+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.129809+0000) 2022-01-31T22:20:43.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:43 smithi167 conmon[60316]: debug 2022-01-31T22:20:43.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.514750+0000) 2022-01-31T22:20:43.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:43 smithi167 conmon[49112]: debug 2022-01-31T22:20:43.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.589125+0000) 2022-01-31T22:20:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:43 smithi171 conmon[41853]: debug 2022-01-31T22:20:43.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.462334+0000) 2022-01-31T22:20:44.139 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:43 smithi167 conmon[54076]: debug 2022-01-31T22:20:43.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:43.759861+0000) 2022-01-31T22:20:44.420 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:44 smithi171 conmon[46715]: debug 2022-01-31T22:20:44.254+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:44.255214+0000) 2022-01-31T22:20:44.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:44 smithi171 conmon[51620]: debug 2022-01-31T22:20:44.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:44.114903+0000) 2022-01-31T22:20:44.759 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:44 smithi167 conmon[49112]: debug 2022-01-31T22:20:44.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:44.589296+0000) 2022-01-31T22:20:44.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:44 smithi167 conmon[60316]: debug 2022-01-31T22:20:44.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:44.514982+0000) 2022-01-31T22:20:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:44 smithi171 conmon[41853]: debug 2022-01-31T22:20:44.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:44.462480+0000) 2022-01-31T22:20:45.143 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:44 smithi167 conmon[54076]: debug 2022-01-31T22:20:44.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:44.759999+0000) 2022-01-31T22:20:45.420 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:45 smithi171 conmon[46715]: debug 2022-01-31T22:20:45.254+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:45.255381+0000) 2022-01-31T22:20:45.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:45 smithi171 conmon[51620]: debug 2022-01-31T22:20:45.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:45.115029+0000) 2022-01-31T22:20:45.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:45 smithi167 conmon[60316]: debug 2022-01-31T22:20:45.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:45.515149+0000) 2022-01-31T22:20:45.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:45 smithi167 conmon[49112]: debug 2022-01-31T22:20:45.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:45.589492+0000) 2022-01-31T22:20:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:45 smithi171 conmon[41853]: debug 2022-01-31T22:20:45.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:45.462668+0000) 2022-01-31T22:20:46.148 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:45 smithi167 conmon[54076]: debug 2022-01-31T22:20:45.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:45.760161+0000) 2022-01-31T22:20:46.420 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:46 smithi171 conmon[46715]: debug 2022-01-31T22:20:46.254+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:46.255477+0000) 2022-01-31T22:20:46.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:46 smithi171 conmon[51620]: debug 2022-01-31T22:20:46.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:46.115130+0000) 2022-01-31T22:20:46.759 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:46 smithi167 conmon[60316]: debug 2022-01-31T22:20:46.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:46.515275+0000) 2022-01-31T22:20:46.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:46 smithi167 conmon[49112]: debug 2022-01-31T22:20:46.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:46.589629+0000) 2022-01-31T22:20:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:46 smithi171 conmon[41853]: debug 2022-01-31T22:20:46.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:46.462889+0000) 2022-01-31T22:20:47.151 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:46 smithi167 conmon[54076]: debug 2022-01-31T22:20:46.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:46.760354+0000) 2022-01-31T22:20:47.421 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:47 smithi171 conmon[46715]: debug 2022-01-31T22:20:47.255+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:47.255636+0000) 2022-01-31T22:20:47.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:47 smithi171 conmon[51620]: debug 2022-01-31T22:20:47.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:47.115309+0000) 2022-01-31T22:20:47.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:47 smithi167 conmon[60316]: debug 2022-01-31T22:20:47.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:47.515457+0000) 2022-01-31T22:20:47.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:47 smithi167 conmon[49112]: debug 2022-01-31T22:20:47.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:47.589779+0000) 2022-01-31T22:20:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:47 smithi171 conmon[41853]: debug 2022-01-31T22:20:47.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:47.463071+0000) 2022-01-31T22:20:48.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:47 smithi167 conmon[54076]: debug 2022-01-31T22:20:47.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:47.760534+0000) 2022-01-31T22:20:48.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:48 smithi167 conmon[60316]: debug 2022-01-31T22:20:48.152+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.153337+0000) 2022-01-31T22:20:48.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:48 smithi167 conmon[49112]: debug 2022-01-31T22:20:48.153+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.153522+0000) 2022-01-31T22:20:48.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:48 smithi167 conmon[54076]: debug 2022-01-31T22:20:48.153+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.154123+0000) 2022-01-31T22:20:48.421 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[35325]: debug 2022-01-31T22:20:48.180+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202116 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:48.421 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[41853]: debug 2022-01-31T22:20:48.152+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.153015+0000) 2022-01-31T22:20:48.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[46715]: debug 2022-01-31T22:20:48.153+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.153852+0000) 2022-01-31T22:20:48.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[46715]: debug 2022-01-31T22:20:48.255+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.255786+0000) 2022-01-31T22:20:48.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[51620]: debug 2022-01-31T22:20:48.115+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.115480+0000) 2022-01-31T22:20:48.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[51620]: debug 2022-01-31T22:20:48.154+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.155049+0000) 2022-01-31T22:20:48.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:48 smithi167 conmon[49112]: debug 2022-01-31T22:20:48.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.589966+0000) 2022-01-31T22:20:48.760 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:48 smithi167 conmon[60316]: debug 2022-01-31T22:20:48.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.515694+0000) 2022-01-31T22:20:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:48 smithi171 conmon[41853]: debug 2022-01-31T22:20:48.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.463293+0000) 2022-01-31T22:20:49.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:48 smithi167 conmon[54076]: debug 2022-01-31T22:20:48.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:48.760700+0000) 2022-01-31T22:20:49.421 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:49 smithi171 conmon[46715]: debug 2022-01-31T22:20:49.255+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:49.255925+0000) 2022-01-31T22:20:49.422 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:49 smithi171 conmon[51620]: debug 2022-01-31T22:20:49.114+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:49.115674+0000) 2022-01-31T22:20:49.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:49 smithi167 conmon[49112]: debug 2022-01-31T22:20:49.588+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:49.590145+0000) 2022-01-31T22:20:49.809 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:49 smithi167 conmon[54076]: debug 2022-01-31T22:20:49.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:49.760844+0000) 2022-01-31T22:20:49.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:49 smithi167 conmon[60316]: debug 2022-01-31T22:20:49.514+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:49.515857+0000) 2022-01-31T22:20:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:49 smithi171 conmon[41853]: debug 2022-01-31T22:20:49.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:49.463434+0000) 2022-01-31T22:20:50.421 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:50 smithi171 conmon[46715]: debug 2022-01-31T22:20:50.255+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:50.256123+0000) 2022-01-31T22:20:50.422 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:50 smithi171 conmon[51620]: debug 2022-01-31T22:20:50.115+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:50.115835+0000) 2022-01-31T22:20:50.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:50 smithi167 conmon[49112]: debug 2022-01-31T22:20:50.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:50.590269+0000) 2022-01-31T22:20:50.810 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:50 smithi167 conmon[60316]: debug 2022-01-31T22:20:50.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:50.515991+0000) 2022-01-31T22:20:50.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:50 smithi167 conmon[54076]: debug 2022-01-31T22:20:50.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:50.761058+0000) 2022-01-31T22:20:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:50 smithi171 conmon[41853]: debug 2022-01-31T22:20:50.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:50.463556+0000) 2022-01-31T22:20:51.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:51 smithi171 conmon[51620]: debug 2022-01-31T22:20:51.115+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:51.115999+0000) 2022-01-31T22:20:51.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:51 smithi171 conmon[46715]: debug 2022-01-31T22:20:51.255+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:51.256305+0000) 2022-01-31T22:20:51.809 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:51 smithi167 conmon[60316]: debug 2022-01-31T22:20:51.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:51.516145+0000) 2022-01-31T22:20:51.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:51 smithi167 conmon[49112]: debug 2022-01-31T22:20:51.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:51.590446+0000) 2022-01-31T22:20:51.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:51 smithi167 conmon[54076]: debug 2022-01-31T22:20:51.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:51.761249+0000) 2022-01-31T22:20:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:51 smithi171 conmon[41853]: debug 2022-01-31T22:20:51.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:51.463747+0000) 2022-01-31T22:20:52.421 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:52 smithi171 conmon[51620]: debug 2022-01-31T22:20:52.115+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:52.116148+0000) 2022-01-31T22:20:52.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:52 smithi171 conmon[46715]: debug 2022-01-31T22:20:52.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:52.256490+0000) 2022-01-31T22:20:52.809 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:52 smithi167 conmon[60316]: debug 2022-01-31T22:20:52.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:52.516327+0000) 2022-01-31T22:20:52.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:52 smithi167 conmon[49112]: debug 2022-01-31T22:20:52.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:52.590652+0000) 2022-01-31T22:20:52.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:52 smithi167 conmon[54076]: debug 2022-01-31T22:20:52.759+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:52.761449+0000) 2022-01-31T22:20:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:52 smithi171 conmon[41853]: debug 2022-01-31T22:20:52.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:52.463967+0000) 2022-01-31T22:20:53.422 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[35325]: debug 2022-01-31T22:20:53.202+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202227 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:53.422 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[41853]: debug 2022-01-31T22:20:53.181+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.182687+0000) 2022-01-31T22:20:53.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[46715]: debug 2022-01-31T22:20:53.182+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.183290+0000) 2022-01-31T22:20:53.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[46715]: debug 2022-01-31T22:20:53.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.256645+0000) 2022-01-31T22:20:53.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[51620]: debug 2022-01-31T22:20:53.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.116332+0000) 2022-01-31T22:20:53.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[51620]: debug 2022-01-31T22:20:53.183+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.184246+0000) 2022-01-31T22:20:53.486 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:53 smithi167 conmon[54076]: debug 2022-01-31T22:20:53.180+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.183708+0000) 2022-01-31T22:20:53.487 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:53 smithi167 conmon[60316]: debug 2022-01-31T22:20:53.188+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.190417+0000) 2022-01-31T22:20:53.488 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:53 smithi167 conmon[49112]: debug 2022-01-31T22:20:53.188+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.190113+0000) 2022-01-31T22:20:53.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:53 smithi167 conmon[49112]: debug 2022-01-31T22:20:53.589+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.590815+0000) 2022-01-31T22:20:53.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:53 smithi167 conmon[60316]: debug 2022-01-31T22:20:53.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.516543+0000) 2022-01-31T22:20:53.836 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[41853]: debug 2022-01-31T22:20:53.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.464156+0000) 2022-01-31T22:20:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:53 smithi171 conmon[35325]: debug 2022-01-31T22:20:53.836+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:20:54.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:53 smithi167 conmon[54076]: debug 2022-01-31T22:20:53.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:53.761580+0000) 2022-01-31T22:20:54.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:54 smithi171 conmon[46715]: debug 2022-01-31T22:20:54.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:54.256804+0000) 2022-01-31T22:20:54.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:54 smithi171 conmon[51620]: debug 2022-01-31T22:20:54.115+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:54.116478+0000) 2022-01-31T22:20:54.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:54 smithi167 conmon[49112]: debug 2022-01-31T22:20:54.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:54.590969+0000) 2022-01-31T22:20:54.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:54 smithi167 conmon[54076]: debug 2022-01-31T22:20:54.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:54.761690+0000) 2022-01-31T22:20:54.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:54 smithi167 conmon[60316]: debug 2022-01-31T22:20:54.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:54.516756+0000) 2022-01-31T22:20:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:54 smithi171 conmon[41853]: debug 2022-01-31T22:20:54.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:54.464319+0000) 2022-01-31T22:20:55.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:55 smithi171 conmon[46715]: debug 2022-01-31T22:20:55.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:55.256990+0000) 2022-01-31T22:20:55.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:55 smithi171 conmon[51620]: debug 2022-01-31T22:20:55.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:55.116659+0000) 2022-01-31T22:20:55.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:55 smithi167 conmon[49112]: debug 2022-01-31T22:20:55.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:55.591126+0000) 2022-01-31T22:20:55.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:55 smithi167 conmon[60316]: debug 2022-01-31T22:20:55.515+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:55.516891+0000) 2022-01-31T22:20:55.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:55 smithi167 conmon[54076]: debug 2022-01-31T22:20:55.760+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:55.761842+0000) 2022-01-31T22:20:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:55 smithi171 conmon[41853]: debug 2022-01-31T22:20:55.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:55.464502+0000) 2022-01-31T22:20:56.422 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:56 smithi171 conmon[46715]: debug 2022-01-31T22:20:56.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:56.257187+0000) 2022-01-31T22:20:56.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:56 smithi171 conmon[51620]: debug 2022-01-31T22:20:56.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:56.116880+0000) 2022-01-31T22:20:56.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:56 smithi167 conmon[49112]: debug 2022-01-31T22:20:56.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:56.591324+0000) 2022-01-31T22:20:56.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:56 smithi167 conmon[60316]: debug 2022-01-31T22:20:56.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:56.517091+0000) 2022-01-31T22:20:56.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:56 smithi167 conmon[54076]: debug 2022-01-31T22:20:56.761+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:56.762002+0000) 2022-01-31T22:20:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:56 smithi171 conmon[41853]: debug 2022-01-31T22:20:56.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:56.464685+0000) 2022-01-31T22:20:57.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:57 smithi171 conmon[46715]: debug 2022-01-31T22:20:57.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:57.257371+0000) 2022-01-31T22:20:57.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:57 smithi171 conmon[51620]: debug 2022-01-31T22:20:57.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:57.117046+0000) 2022-01-31T22:20:57.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:57 smithi167 conmon[49112]: debug 2022-01-31T22:20:57.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:57.591478+0000) 2022-01-31T22:20:57.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:57 smithi167 conmon[60316]: debug 2022-01-31T22:20:57.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:57.517280+0000) 2022-01-31T22:20:57.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:57 smithi167 conmon[54076]: debug 2022-01-31T22:20:57.761+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:57.762187+0000) 2022-01-31T22:20:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:57 smithi171 conmon[41853]: debug 2022-01-31T22:20:57.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:57.464838+0000) 2022-01-31T22:20:58.423 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[35325]: debug 2022-01-31T22:20:58.225+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202338 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:20:58.424 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[41853]: debug 2022-01-31T22:20:58.204+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.205627+0000) 2022-01-31T22:20:58.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[46715]: debug 2022-01-31T22:20:58.205+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.206431+0000) 2022-01-31T22:20:58.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[46715]: debug 2022-01-31T22:20:58.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.257552+0000) 2022-01-31T22:20:58.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[51620]: debug 2022-01-31T22:20:58.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.117221+0000) 2022-01-31T22:20:58.428 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[51620]: debug 2022-01-31T22:20:58.205+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.206048+0000) 2022-01-31T22:20:58.487 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:58 smithi167 conmon[49112]: debug 2022-01-31T22:20:58.204+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.206335+0000) 2022-01-31T22:20:58.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:58 smithi167 conmon[54076]: debug 2022-01-31T22:20:58.204+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.205545+0000) 2022-01-31T22:20:58.488 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:58 smithi167 conmon[60316]: debug 2022-01-31T22:20:58.204+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.206167+0000) 2022-01-31T22:20:58.762 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:58 smithi167 conmon[49112]: debug 2022-01-31T22:20:58.590+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.591689+0000) 2022-01-31T22:20:58.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:58 smithi167 conmon[60316]: debug 2022-01-31T22:20:58.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.517470+0000) 2022-01-31T22:20:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:58 smithi171 conmon[41853]: debug 2022-01-31T22:20:58.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.465021+0000) 2022-01-31T22:20:59.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:58 smithi167 conmon[54076]: debug 2022-01-31T22:20:58.761+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:58.762376+0000) 2022-01-31T22:20:59.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:20:59 smithi171 conmon[46715]: debug 2022-01-31T22:20:59.256+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:59.257733+0000) 2022-01-31T22:20:59.423 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:20:59 smithi171 conmon[51620]: debug 2022-01-31T22:20:59.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:59.117359+0000) 2022-01-31T22:20:59.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:20:59 smithi167 conmon[49112]: debug 2022-01-31T22:20:59.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:59.591873+0000) 2022-01-31T22:20:59.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:20:59 smithi167 conmon[54076]: debug 2022-01-31T22:20:59.761+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:59.762515+0000) 2022-01-31T22:20:59.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:20:59 smithi167 conmon[60316]: debug 2022-01-31T22:20:59.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:59.517648+0000) 2022-01-31T22:20:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:20:59 smithi171 conmon[41853]: debug 2022-01-31T22:20:59.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:20:59.465144+0000) 2022-01-31T22:21:00.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:00 smithi171 conmon[46715]: debug 2022-01-31T22:21:00.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:00.257900+0000) 2022-01-31T22:21:00.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:00 smithi171 conmon[51620]: debug 2022-01-31T22:21:00.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:00.117493+0000) 2022-01-31T22:21:00.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:00 smithi167 conmon[49112]: debug 2022-01-31T22:21:00.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:00.592063+0000) 2022-01-31T22:21:00.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:00 smithi167 conmon[54076]: debug 2022-01-31T22:21:00.761+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:00.762655+0000) 2022-01-31T22:21:00.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:00 smithi167 conmon[60316]: debug 2022-01-31T22:21:00.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:00.517779+0000) 2022-01-31T22:21:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:00 smithi171 conmon[41853]: debug 2022-01-31T22:21:00.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:00.465342+0000) 2022-01-31T22:21:01.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:01 smithi171 conmon[46715]: debug 2022-01-31T22:21:01.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:01.258116+0000) 2022-01-31T22:21:01.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:01 smithi171 conmon[51620]: debug 2022-01-31T22:21:01.116+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:01.117666+0000) 2022-01-31T22:21:01.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:01 smithi167 conmon[49112]: debug 2022-01-31T22:21:01.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:01.592245+0000) 2022-01-31T22:21:01.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:01 smithi167 conmon[54076]: debug 2022-01-31T22:21:01.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:01.762843+0000) 2022-01-31T22:21:01.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:01 smithi167 conmon[60316]: debug 2022-01-31T22:21:01.517+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:01.517957+0000) 2022-01-31T22:21:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:01 smithi171 conmon[41853]: debug 2022-01-31T22:21:01.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:01.465518+0000) 2022-01-31T22:21:02.423 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:02 smithi171 conmon[46715]: debug 2022-01-31T22:21:02.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:02.258301+0000) 2022-01-31T22:21:02.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:02 smithi171 conmon[51620]: debug 2022-01-31T22:21:02.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:02.117827+0000) 2022-01-31T22:21:02.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:02 smithi167 conmon[49112]: debug 2022-01-31T22:21:02.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:02.592446+0000) 2022-01-31T22:21:02.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:02 smithi167 conmon[54076]: debug 2022-01-31T22:21:02.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:02.763010+0000) 2022-01-31T22:21:02.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:02 smithi167 conmon[60316]: debug 2022-01-31T22:21:02.517+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:02.518150+0000) 2022-01-31T22:21:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:02 smithi171 conmon[41853]: debug 2022-01-31T22:21:02.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:02.465742+0000) 2022-01-31T22:21:03.423 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[35325]: debug 2022-01-31T22:21:03.249+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202449 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:03.424 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[41853]: debug 2022-01-31T22:21:03.227+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.228185+0000) 2022-01-31T22:21:03.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[46715]: debug 2022-01-31T22:21:03.229+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.229819+0000) 2022-01-31T22:21:03.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[46715]: debug 2022-01-31T22:21:03.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.258434+0000) 2022-01-31T22:21:03.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[51620]: debug 2022-01-31T22:21:03.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.117979+0000) 2022-01-31T22:21:03.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[51620]: debug 2022-01-31T22:21:03.228+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.228900+0000) 2022-01-31T22:21:03.488 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:03 smithi167 conmon[49112]: debug 2022-01-31T22:21:03.228+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.229594+0000) 2022-01-31T22:21:03.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:03 smithi167 conmon[54076]: debug 2022-01-31T22:21:03.227+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.229110+0000) 2022-01-31T22:21:03.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:03 smithi167 conmon[60316]: debug 2022-01-31T22:21:03.228+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.230023+0000) 2022-01-31T22:21:03.762 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:03 smithi167 conmon[60316]: debug 2022-01-31T22:21:03.516+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.518339+0000) 2022-01-31T22:21:03.763 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:03 smithi167 conmon[49112]: debug 2022-01-31T22:21:03.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.592600+0000) 2022-01-31T22:21:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:03 smithi171 conmon[41853]: debug 2022-01-31T22:21:03.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.466011+0000) 2022-01-31T22:21:04.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:03 smithi167 conmon[54076]: debug 2022-01-31T22:21:03.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:03.763135+0000) 2022-01-31T22:21:04.424 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:04 smithi171 conmon[46715]: debug 2022-01-31T22:21:04.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:04.258579+0000) 2022-01-31T22:21:04.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:04 smithi171 conmon[51620]: debug 2022-01-31T22:21:04.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:04.118132+0000) 2022-01-31T22:21:04.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:04 smithi167 conmon[54076]: debug 2022-01-31T22:21:04.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:04.763252+0000) 2022-01-31T22:21:04.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:04 smithi167 conmon[60316]: debug 2022-01-31T22:21:04.517+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:04.518543+0000) 2022-01-31T22:21:04.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:04 smithi167 conmon[49112]: debug 2022-01-31T22:21:04.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:04.592774+0000) 2022-01-31T22:21:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:04 smithi171 conmon[41853]: debug 2022-01-31T22:21:04.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:04.466139+0000) 2022-01-31T22:21:05.424 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:05 smithi171 conmon[46715]: debug 2022-01-31T22:21:05.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:05.258797+0000) 2022-01-31T22:21:05.425 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:05 smithi171 conmon[51620]: debug 2022-01-31T22:21:05.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:05.118268+0000) 2022-01-31T22:21:05.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:05 smithi167 conmon[54076]: debug 2022-01-31T22:21:05.761+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:05.763434+0000) 2022-01-31T22:21:05.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:05 smithi167 conmon[49112]: debug 2022-01-31T22:21:05.591+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:05.592939+0000) 2022-01-31T22:21:05.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:05 smithi167 conmon[60316]: debug 2022-01-31T22:21:05.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:05.518715+0000) 2022-01-31T22:21:05.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:05 smithi171 conmon[41853]: debug 2022-01-31T22:21:05.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:05.466295+0000) 2022-01-31T22:21:06.424 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:06 smithi171 conmon[51620]: debug 2022-01-31T22:21:06.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:06.118407+0000) 2022-01-31T22:21:06.424 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:06 smithi171 conmon[46715]: debug 2022-01-31T22:21:06.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:06.258980+0000) 2022-01-31T22:21:06.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:06 smithi167 conmon[49112]: debug 2022-01-31T22:21:06.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:06.593140+0000) 2022-01-31T22:21:06.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:06 smithi167 conmon[54076]: debug 2022-01-31T22:21:06.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:06.763629+0000) 2022-01-31T22:21:06.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:06 smithi167 conmon[60316]: debug 2022-01-31T22:21:06.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:06.518879+0000) 2022-01-31T22:21:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:06 smithi171 conmon[41853]: debug 2022-01-31T22:21:06.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:06.466490+0000) 2022-01-31T22:21:07.424 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:07 smithi171 conmon[46715]: debug 2022-01-31T22:21:07.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:07.259149+0000) 2022-01-31T22:21:07.425 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:07 smithi171 conmon[51620]: debug 2022-01-31T22:21:07.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:07.118535+0000) 2022-01-31T22:21:07.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:07 smithi167 conmon[49112]: debug 2022-01-31T22:21:07.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:07.593324+0000) 2022-01-31T22:21:07.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:07 smithi167 conmon[54076]: debug 2022-01-31T22:21:07.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:07.763781+0000) 2022-01-31T22:21:07.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:07 smithi167 conmon[60316]: debug 2022-01-31T22:21:07.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:07.519089+0000) 2022-01-31T22:21:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:07 smithi171 conmon[41853]: debug 2022-01-31T22:21:07.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:07.466711+0000) 2022-01-31T22:21:08.424 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[35325]: debug 2022-01-31T22:21:08.272+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202559 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:08.425 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[41853]: debug 2022-01-31T22:21:08.250+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.252482+0000) 2022-01-31T22:21:08.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[46715]: debug 2022-01-31T22:21:08.252+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.253852+0000) 2022-01-31T22:21:08.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[46715]: debug 2022-01-31T22:21:08.257+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.259286+0000) 2022-01-31T22:21:08.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[51620]: debug 2022-01-31T22:21:08.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.118667+0000) 2022-01-31T22:21:08.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[51620]: debug 2022-01-31T22:21:08.250+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.252000+0000) 2022-01-31T22:21:08.518 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:08 smithi167 conmon[49112]: debug 2022-01-31T22:21:08.250+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.252319+0000) 2022-01-31T22:21:08.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:08 smithi167 conmon[54076]: debug 2022-01-31T22:21:08.251+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.252576+0000) 2022-01-31T22:21:08.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:08 smithi167 conmon[60316]: debug 2022-01-31T22:21:08.252+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.253494+0000) 2022-01-31T22:21:08.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:08 smithi167 conmon[49112]: debug 2022-01-31T22:21:08.592+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.593538+0000) 2022-01-31T22:21:08.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:08 smithi167 conmon[54076]: debug 2022-01-31T22:21:08.763+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.763945+0000) 2022-01-31T22:21:08.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:08 smithi167 conmon[60316]: debug 2022-01-31T22:21:08.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.519261+0000) 2022-01-31T22:21:08.836 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[41853]: debug 2022-01-31T22:21:08.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:08.466892+0000) 2022-01-31T22:21:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:08 smithi171 conmon[35325]: debug 2022-01-31T22:21:08.836+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:21:09.424 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:09 smithi171 conmon[46715]: debug 2022-01-31T22:21:09.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:09.259437+0000) 2022-01-31T22:21:09.425 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:09 smithi171 conmon[51620]: debug 2022-01-31T22:21:09.117+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:09.118852+0000) 2022-01-31T22:21:09.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:09 smithi167 conmon[49112]: debug 2022-01-31T22:21:09.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:09.593724+0000) 2022-01-31T22:21:09.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:09 smithi167 conmon[54076]: debug 2022-01-31T22:21:09.762+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:09.764058+0000) 2022-01-31T22:21:09.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:09 smithi167 conmon[60316]: debug 2022-01-31T22:21:09.518+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:09.519415+0000) 2022-01-31T22:21:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:09 smithi171 conmon[41853]: debug 2022-01-31T22:21:09.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:09.467015+0000) 2022-01-31T22:21:10.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:10 smithi171 conmon[46715]: debug 2022-01-31T22:21:10.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:10.259625+0000) 2022-01-31T22:21:10.425 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:10 smithi171 conmon[51620]: debug 2022-01-31T22:21:10.118+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:10.119019+0000) 2022-01-31T22:21:10.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:10 smithi167 conmon[49112]: debug 2022-01-31T22:21:10.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:10.593909+0000) 2022-01-31T22:21:10.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:10 smithi167 conmon[54076]: debug 2022-01-31T22:21:10.763+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:10.764275+0000) 2022-01-31T22:21:10.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:10 smithi167 conmon[60316]: debug 2022-01-31T22:21:10.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:10.519599+0000) 2022-01-31T22:21:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:10 smithi171 conmon[41853]: debug 2022-01-31T22:21:10.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:10.467205+0000) 2022-01-31T22:21:11.425 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:11 smithi171 conmon[51620]: debug 2022-01-31T22:21:11.118+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:11.119192+0000) 2022-01-31T22:21:11.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:11 smithi171 conmon[46715]: debug 2022-01-31T22:21:11.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:11.259814+0000) 2022-01-31T22:21:11.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:11 smithi167 conmon[49112]: debug 2022-01-31T22:21:11.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:11.594068+0000) 2022-01-31T22:21:11.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:11 smithi167 conmon[54076]: debug 2022-01-31T22:21:11.763+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:11.764493+0000) 2022-01-31T22:21:11.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:11 smithi167 conmon[60316]: debug 2022-01-31T22:21:11.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:11.519759+0000) 2022-01-31T22:21:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:11 smithi171 conmon[41853]: debug 2022-01-31T22:21:11.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:11.467354+0000) 2022-01-31T22:21:12.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:12 smithi171 conmon[46715]: debug 2022-01-31T22:21:12.259+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:12.260021+0000) 2022-01-31T22:21:12.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:12 smithi171 conmon[51620]: debug 2022-01-31T22:21:12.118+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:12.119370+0000) 2022-01-31T22:21:12.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:12 smithi167 conmon[49112]: debug 2022-01-31T22:21:12.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:12.594255+0000) 2022-01-31T22:21:12.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:12 smithi167 conmon[54076]: debug 2022-01-31T22:21:12.764+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:12.764612+0000) 2022-01-31T22:21:12.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:12 smithi167 conmon[60316]: debug 2022-01-31T22:21:12.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:12.519971+0000) 2022-01-31T22:21:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:12 smithi171 conmon[41853]: debug 2022-01-31T22:21:12.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:12.467537+0000) 2022-01-31T22:21:13.425 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[41853]: debug 2022-01-31T22:21:13.274+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.276737+0000) 2022-01-31T22:21:13.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[46715]: debug 2022-01-31T22:21:13.258+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.260150+0000) 2022-01-31T22:21:13.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[46715]: debug 2022-01-31T22:21:13.274+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.276004+0000) 2022-01-31T22:21:13.427 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[35325]: debug 2022-01-31T22:21:13.295+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202671 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:13.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[51620]: debug 2022-01-31T22:21:13.118+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.119561+0000) 2022-01-31T22:21:13.428 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[51620]: debug 2022-01-31T22:21:13.273+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.275520+0000) 2022-01-31T22:21:13.519 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:13 smithi167 conmon[49112]: debug 2022-01-31T22:21:13.275+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.275868+0000) 2022-01-31T22:21:13.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:13 smithi167 conmon[54076]: debug 2022-01-31T22:21:13.276+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.276901+0000) 2022-01-31T22:21:13.520 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:13 smithi167 conmon[60316]: debug 2022-01-31T22:21:13.275+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.276399+0000) 2022-01-31T22:21:13.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:13 smithi167 conmon[49112]: debug 2022-01-31T22:21:13.593+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.594442+0000) 2022-01-31T22:21:13.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:13 smithi167 conmon[54076]: debug 2022-01-31T22:21:13.764+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.764792+0000) 2022-01-31T22:21:13.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:13 smithi167 conmon[60316]: debug 2022-01-31T22:21:13.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.520155+0000) 2022-01-31T22:21:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:13 smithi171 conmon[41853]: debug 2022-01-31T22:21:13.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:13.467703+0000) 2022-01-31T22:21:14.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:14 smithi171 conmon[46715]: debug 2022-01-31T22:21:14.259+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:14.260320+0000) 2022-01-31T22:21:14.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:14 smithi171 conmon[51620]: debug 2022-01-31T22:21:14.118+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:14.119780+0000) 2022-01-31T22:21:14.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:14 smithi167 conmon[49112]: debug 2022-01-31T22:21:14.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:14.594597+0000) 2022-01-31T22:21:14.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:14 smithi167 conmon[54076]: debug 2022-01-31T22:21:14.764+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:14.764943+0000) 2022-01-31T22:21:14.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:14 smithi167 conmon[60316]: debug 2022-01-31T22:21:14.519+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:14.520336+0000) 2022-01-31T22:21:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:14 smithi171 conmon[41853]: debug 2022-01-31T22:21:14.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:14.467863+0000) 2022-01-31T22:21:15.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:15 smithi171 conmon[46715]: debug 2022-01-31T22:21:15.259+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:15.260470+0000) 2022-01-31T22:21:15.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:15 smithi171 conmon[51620]: debug 2022-01-31T22:21:15.119+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:15.119859+0000) 2022-01-31T22:21:15.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:15 smithi167 conmon[49112]: debug 2022-01-31T22:21:15.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:15.594760+0000) 2022-01-31T22:21:15.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:15 smithi167 conmon[54076]: debug 2022-01-31T22:21:15.764+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:15.765110+0000) 2022-01-31T22:21:15.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:15 smithi167 conmon[60316]: debug 2022-01-31T22:21:15.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:15.520534+0000) 2022-01-31T22:21:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:15 smithi171 conmon[41853]: debug 2022-01-31T22:21:15.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:15.468030+0000) 2022-01-31T22:21:16.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:16 smithi171 conmon[46715]: debug 2022-01-31T22:21:16.259+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:16.260675+0000) 2022-01-31T22:21:16.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:16 smithi171 conmon[51620]: debug 2022-01-31T22:21:16.119+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:16.120045+0000) 2022-01-31T22:21:16.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:16 smithi167 conmon[49112]: debug 2022-01-31T22:21:16.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:16.594942+0000) 2022-01-31T22:21:16.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:16 smithi167 conmon[54076]: debug 2022-01-31T22:21:16.764+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:16.765267+0000) 2022-01-31T22:21:16.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:16 smithi167 conmon[60316]: debug 2022-01-31T22:21:16.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:16.520723+0000) 2022-01-31T22:21:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:16 smithi171 conmon[41853]: debug 2022-01-31T22:21:16.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:16.468219+0000) 2022-01-31T22:21:17.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:17 smithi171 conmon[46715]: debug 2022-01-31T22:21:17.260+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:17.260855+0000) 2022-01-31T22:21:17.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:17 smithi171 conmon[51620]: debug 2022-01-31T22:21:17.119+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:17.120263+0000) 2022-01-31T22:21:17.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:17 smithi167 conmon[49112]: debug 2022-01-31T22:21:17.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:17.595126+0000) 2022-01-31T22:21:17.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:17 smithi167 conmon[54076]: debug 2022-01-31T22:21:17.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:17.765461+0000) 2022-01-31T22:21:17.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:17 smithi167 conmon[60316]: debug 2022-01-31T22:21:17.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:17.520917+0000) 2022-01-31T22:21:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:17 smithi171 conmon[41853]: debug 2022-01-31T22:21:17.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:17.468374+0000) 2022-01-31T22:21:18.426 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[35325]: debug 2022-01-31T22:21:18.320+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202781 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:18.427 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[41853]: debug 2022-01-31T22:21:18.299+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.300198+0000) 2022-01-31T22:21:18.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[46715]: debug 2022-01-31T22:21:18.260+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.261009+0000) 2022-01-31T22:21:18.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[46715]: debug 2022-01-31T22:21:18.299+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.300154+0000) 2022-01-31T22:21:18.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[51620]: debug 2022-01-31T22:21:18.119+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.120486+0000) 2022-01-31T22:21:18.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[51620]: debug 2022-01-31T22:21:18.298+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.299381+0000) 2022-01-31T22:21:18.520 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:18 smithi167 conmon[49112]: debug 2022-01-31T22:21:18.300+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.301090+0000) 2022-01-31T22:21:18.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:18 smithi167 conmon[54076]: debug 2022-01-31T22:21:18.300+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.300799+0000) 2022-01-31T22:21:18.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:18 smithi167 conmon[60316]: debug 2022-01-31T22:21:18.299+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.299978+0000) 2022-01-31T22:21:18.810 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:18 smithi171 conmon[41853]: debug 2022-01-31T22:21:18.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.468557+0000) 2022-01-31T22:21:18.814 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:18 smithi167 conmon[54076]: debug 2022-01-31T22:21:18.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.765677+0000) 2022-01-31T22:21:18.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:18 smithi167 conmon[49112]: debug 2022-01-31T22:21:18.594+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.595311+0000) 2022-01-31T22:21:18.815 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:18 smithi167 conmon[60316]: debug 2022-01-31T22:21:18.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:18.521078+0000) 2022-01-31T22:21:19.426 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:19 smithi171 conmon[46715]: debug 2022-01-31T22:21:19.260+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:19.261245+0000) 2022-01-31T22:21:19.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:19 smithi171 conmon[51620]: debug 2022-01-31T22:21:19.119+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:19.120676+0000) 2022-01-31T22:21:19.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:19 smithi167 conmon[49112]: debug 2022-01-31T22:21:19.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:19.595523+0000) 2022-01-31T22:21:19.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:19 smithi167 conmon[60316]: debug 2022-01-31T22:21:19.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:19.521224+0000) 2022-01-31T22:21:19.781 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:19 smithi167 conmon[54076]: debug 2022-01-31T22:21:19.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:19.765768+0000) 2022-01-31T22:21:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:19 smithi171 conmon[41853]: debug 2022-01-31T22:21:19.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:19.468727+0000) 2022-01-31T22:21:20.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:20 smithi171 conmon[46715]: debug 2022-01-31T22:21:20.260+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:20.261408+0000) 2022-01-31T22:21:20.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:20 smithi171 conmon[51620]: debug 2022-01-31T22:21:20.120+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:20.120800+0000) 2022-01-31T22:21:20.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:20 smithi167 conmon[49112]: debug 2022-01-31T22:21:20.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:20.595705+0000) 2022-01-31T22:21:20.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:20 smithi167 conmon[54076]: debug 2022-01-31T22:21:20.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:20.765878+0000) 2022-01-31T22:21:20.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:20 smithi167 conmon[60316]: debug 2022-01-31T22:21:20.520+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:20.521380+0000) 2022-01-31T22:21:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:20 smithi171 conmon[41853]: debug 2022-01-31T22:21:20.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:20.468941+0000) 2022-01-31T22:21:21.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:21 smithi171 conmon[46715]: debug 2022-01-31T22:21:21.260+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:21.261564+0000) 2022-01-31T22:21:21.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:21 smithi171 conmon[51620]: debug 2022-01-31T22:21:21.120+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:21.121014+0000) 2022-01-31T22:21:21.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:21 smithi167 conmon[49112]: debug 2022-01-31T22:21:21.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:21.595891+0000) 2022-01-31T22:21:21.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:21 smithi167 conmon[54076]: debug 2022-01-31T22:21:21.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:21.766049+0000) 2022-01-31T22:21:21.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:21 smithi167 conmon[60316]: debug 2022-01-31T22:21:21.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:21.521587+0000) 2022-01-31T22:21:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:21 smithi171 conmon[41853]: debug 2022-01-31T22:21:21.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:21.469144+0000) 2022-01-31T22:21:22.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:22 smithi171 conmon[46715]: debug 2022-01-31T22:21:22.261+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:22.261768+0000) 2022-01-31T22:21:22.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:22 smithi171 conmon[51620]: debug 2022-01-31T22:21:22.120+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:22.121230+0000) 2022-01-31T22:21:22.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:22 smithi167 conmon[49112]: debug 2022-01-31T22:21:22.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:22.596130+0000) 2022-01-31T22:21:22.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:22 smithi167 conmon[54076]: debug 2022-01-31T22:21:22.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:22.766223+0000) 2022-01-31T22:21:22.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:22 smithi167 conmon[60316]: debug 2022-01-31T22:21:22.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:22.521790+0000) 2022-01-31T22:21:22.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:22 smithi171 conmon[41853]: debug 2022-01-31T22:21:22.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:22.469325+0000) 2022-01-31T22:21:23.427 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[35325]: debug 2022-01-31T22:21:23.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 202889 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:23.428 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[41853]: debug 2022-01-31T22:21:23.322+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.323324+0000) 2022-01-31T22:21:23.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[46715]: debug 2022-01-31T22:21:23.261+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.261951+0000) 2022-01-31T22:21:23.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[46715]: debug 2022-01-31T22:21:23.323+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.323983+0000) 2022-01-31T22:21:23.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[51620]: debug 2022-01-31T22:21:23.120+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.121397+0000) 2022-01-31T22:21:23.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[51620]: debug 2022-01-31T22:21:23.322+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.322790+0000) 2022-01-31T22:21:23.521 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:23 smithi167 conmon[49112]: debug 2022-01-31T22:21:23.323+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.324269+0000) 2022-01-31T22:21:23.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:23 smithi167 conmon[54076]: debug 2022-01-31T22:21:23.324+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.324812+0000) 2022-01-31T22:21:23.522 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:23 smithi167 conmon[60316]: debug 2022-01-31T22:21:23.324+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.324595+0000) 2022-01-31T22:21:23.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:23 smithi167 conmon[49112]: debug 2022-01-31T22:21:23.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.596285+0000) 2022-01-31T22:21:23.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:23 smithi167 conmon[54076]: debug 2022-01-31T22:21:23.765+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.766416+0000) 2022-01-31T22:21:23.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:23 smithi167 conmon[60316]: debug 2022-01-31T22:21:23.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.521988+0000) 2022-01-31T22:21:23.837 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[41853]: debug 2022-01-31T22:21:23.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:23.469480+0000) 2022-01-31T22:21:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:23 smithi171 conmon[35325]: debug 2022-01-31T22:21:23.837+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:21:24.427 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:24 smithi171 conmon[51620]: debug 2022-01-31T22:21:24.120+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:24.121505+0000) 2022-01-31T22:21:24.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:24 smithi171 conmon[46715]: debug 2022-01-31T22:21:24.261+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:24.262190+0000) 2022-01-31T22:21:24.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:24 smithi167 conmon[49112]: debug 2022-01-31T22:21:24.595+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:24.596446+0000) 2022-01-31T22:21:24.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:24 smithi167 conmon[54076]: debug 2022-01-31T22:21:24.766+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:24.766575+0000) 2022-01-31T22:21:24.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:24 smithi167 conmon[60316]: debug 2022-01-31T22:21:24.521+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:24.522153+0000) 2022-01-31T22:21:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:24 smithi171 conmon[41853]: debug 2022-01-31T22:21:24.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:24.469662+0000) 2022-01-31T22:21:25.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:25 smithi171 conmon[46715]: debug 2022-01-31T22:21:25.261+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:25.262338+0000) 2022-01-31T22:21:25.428 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:25 smithi171 conmon[51620]: debug 2022-01-31T22:21:25.120+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:25.121593+0000) 2022-01-31T22:21:25.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:25 smithi167 conmon[49112]: debug 2022-01-31T22:21:25.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:25.596672+0000) 2022-01-31T22:21:25.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:25 smithi167 conmon[54076]: debug 2022-01-31T22:21:25.766+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:25.766758+0000) 2022-01-31T22:21:25.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:25 smithi167 conmon[60316]: debug 2022-01-31T22:21:25.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:25.522313+0000) 2022-01-31T22:21:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:25 smithi171 conmon[41853]: debug 2022-01-31T22:21:25.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:25.469826+0000) 2022-01-31T22:21:26.427 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:26 smithi171 conmon[46715]: debug 2022-01-31T22:21:26.261+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:26.262493+0000) 2022-01-31T22:21:26.428 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:26 smithi171 conmon[51620]: debug 2022-01-31T22:21:26.121+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:26.121775+0000) 2022-01-31T22:21:26.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:26 smithi167 conmon[49112]: debug 2022-01-31T22:21:26.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:26.596859+0000) 2022-01-31T22:21:26.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:26 smithi167 conmon[54076]: debug 2022-01-31T22:21:26.766+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:26.766918+0000) 2022-01-31T22:21:26.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:26 smithi167 conmon[60316]: debug 2022-01-31T22:21:26.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:26.522486+0000) 2022-01-31T22:21:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:26 smithi171 conmon[41853]: debug 2022-01-31T22:21:26.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:26.470011+0000) 2022-01-31T22:21:27.428 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:27 smithi171 conmon[51620]: debug 2022-01-31T22:21:27.121+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:27.121971+0000) 2022-01-31T22:21:27.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:27 smithi171 conmon[46715]: debug 2022-01-31T22:21:27.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:27.262674+0000) 2022-01-31T22:21:27.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:27 smithi167 conmon[49112]: debug 2022-01-31T22:21:27.596+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:27.597070+0000) 2022-01-31T22:21:27.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:27 smithi167 conmon[54076]: debug 2022-01-31T22:21:27.766+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:27.767130+0000) 2022-01-31T22:21:27.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:27 smithi167 conmon[60316]: debug 2022-01-31T22:21:27.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:27.522702+0000) 2022-01-31T22:21:27.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:27 smithi171 conmon[41853]: debug 2022-01-31T22:21:27.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:27.470197+0000) 2022-01-31T22:21:28.428 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[35325]: debug 2022-01-31T22:21:28.367+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203002 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:28.429 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[41853]: debug 2022-01-31T22:21:28.346+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.347001+0000) 2022-01-31T22:21:28.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[46715]: debug 2022-01-31T22:21:28.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.262848+0000) 2022-01-31T22:21:28.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[46715]: debug 2022-01-31T22:21:28.345+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.346204+0000) 2022-01-31T22:21:28.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[51620]: debug 2022-01-31T22:21:28.121+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.122163+0000) 2022-01-31T22:21:28.431 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[51620]: debug 2022-01-31T22:21:28.345+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.345994+0000) 2022-01-31T22:21:28.522 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:28 smithi167 conmon[49112]: debug 2022-01-31T22:21:28.347+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.347832+0000) 2022-01-31T22:21:28.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:28 smithi167 conmon[54076]: debug 2022-01-31T22:21:28.346+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.347130+0000) 2022-01-31T22:21:28.523 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:28 smithi167 conmon[60316]: debug 2022-01-31T22:21:28.347+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.347691+0000) 2022-01-31T22:21:28.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:28 smithi167 conmon[60316]: debug 2022-01-31T22:21:28.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.522879+0000) 2022-01-31T22:21:28.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:28 smithi167 conmon[49112]: debug 2022-01-31T22:21:28.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.597260+0000) 2022-01-31T22:21:28.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:28 smithi167 conmon[54076]: debug 2022-01-31T22:21:28.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.767319+0000) 2022-01-31T22:21:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:28 smithi171 conmon[41853]: debug 2022-01-31T22:21:28.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:28.470344+0000) 2022-01-31T22:21:29.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:29 smithi171 conmon[46715]: debug 2022-01-31T22:21:29.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:29.263047+0000) 2022-01-31T22:21:29.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:29 smithi171 conmon[51620]: debug 2022-01-31T22:21:29.121+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:29.122333+0000) 2022-01-31T22:21:29.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:29 smithi167 conmon[49112]: debug 2022-01-31T22:21:29.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:29.597441+0000) 2022-01-31T22:21:29.816 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:29 smithi167 conmon[54076]: debug 2022-01-31T22:21:29.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:29.767478+0000) 2022-01-31T22:21:29.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:29 smithi167 conmon[60316]: debug 2022-01-31T22:21:29.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:29.523036+0000) 2022-01-31T22:21:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:29 smithi171 conmon[41853]: debug 2022-01-31T22:21:29.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:29.470520+0000) 2022-01-31T22:21:30.428 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:30 smithi171 conmon[46715]: debug 2022-01-31T22:21:30.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:30.263203+0000) 2022-01-31T22:21:30.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:30 smithi171 conmon[51620]: debug 2022-01-31T22:21:30.121+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:30.122491+0000) 2022-01-31T22:21:30.816 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:30 smithi167 conmon[49112]: debug 2022-01-31T22:21:30.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:30.597577+0000) 2022-01-31T22:21:30.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:30 smithi167 conmon[54076]: debug 2022-01-31T22:21:30.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:30.767661+0000) 2022-01-31T22:21:30.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:30 smithi167 conmon[60316]: debug 2022-01-31T22:21:30.522+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:30.523218+0000) 2022-01-31T22:21:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:30 smithi171 conmon[41853]: debug 2022-01-31T22:21:30.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:30.470716+0000) 2022-01-31T22:21:31.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:31 smithi171 conmon[46715]: debug 2022-01-31T22:21:31.262+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:31.263386+0000) 2022-01-31T22:21:31.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:31 smithi171 conmon[51620]: debug 2022-01-31T22:21:31.121+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:31.122614+0000) 2022-01-31T22:21:31.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:31 smithi167 conmon[60316]: debug 2022-01-31T22:21:31.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:31.523416+0000) 2022-01-31T22:21:31.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:31 smithi167 conmon[49112]: debug 2022-01-31T22:21:31.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:31.597707+0000) 2022-01-31T22:21:31.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:31 smithi167 conmon[54076]: debug 2022-01-31T22:21:31.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:31.767818+0000) 2022-01-31T22:21:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:31 smithi171 conmon[41853]: debug 2022-01-31T22:21:31.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:31.470934+0000) 2022-01-31T22:21:32.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:32 smithi171 conmon[46715]: debug 2022-01-31T22:21:32.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:32.263541+0000) 2022-01-31T22:21:32.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:32 smithi171 conmon[51620]: debug 2022-01-31T22:21:32.122+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:32.122766+0000) 2022-01-31T22:21:32.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:32 smithi167 conmon[60316]: debug 2022-01-31T22:21:32.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:32.523612+0000) 2022-01-31T22:21:32.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:32 smithi167 conmon[49112]: debug 2022-01-31T22:21:32.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:32.597842+0000) 2022-01-31T22:21:32.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:32 smithi167 conmon[54076]: debug 2022-01-31T22:21:32.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:32.767971+0000) 2022-01-31T22:21:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:32 smithi171 conmon[41853]: debug 2022-01-31T22:21:32.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:32.471138+0000) 2022-01-31T22:21:33.391 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[46715]: debug 2022-01-31T22:21:33.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.263722+0000) 2022-01-31T22:21:33.392 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[46715]: debug 2022-01-31T22:21:33.369+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.369790+0000) 2022-01-31T22:21:33.393 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[51620]: debug 2022-01-31T22:21:33.122+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.122909+0000) 2022-01-31T22:21:33.393 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[51620]: debug 2022-01-31T22:21:33.368+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.369437+0000) 2022-01-31T22:21:33.394 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[41853]: debug 2022-01-31T22:21:33.369+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.370616+0000) 2022-01-31T22:21:33.523 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:33 smithi167 conmon[49112]: debug 2022-01-31T22:21:33.370+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.371171+0000) 2022-01-31T22:21:33.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:33 smithi167 conmon[54076]: debug 2022-01-31T22:21:33.369+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.370310+0000) 2022-01-31T22:21:33.526 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:33 smithi167 conmon[60316]: debug 2022-01-31T22:21:33.370+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.371088+0000) 2022-01-31T22:21:33.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:33 smithi167 conmon[54076]: debug 2022-01-31T22:21:33.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.768181+0000) 2022-01-31T22:21:33.817 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:33 smithi167 conmon[60316]: debug 2022-01-31T22:21:33.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.523756+0000) 2022-01-31T22:21:33.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:33 smithi167 conmon[49112]: debug 2022-01-31T22:21:33.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.598020+0000) 2022-01-31T22:21:33.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[35325]: debug 2022-01-31T22:21:33.390+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:33.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:33 smithi171 conmon[41853]: debug 2022-01-31T22:21:33.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:33.471392+0000) 2022-01-31T22:21:34.401 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:34 smithi171 conmon[46715]: debug 2022-01-31T22:21:34.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:34.263930+0000) 2022-01-31T22:21:34.402 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:34 smithi171 conmon[51620]: debug 2022-01-31T22:21:34.122+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:34.123089+0000) 2022-01-31T22:21:34.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:34 smithi167 conmon[49112]: debug 2022-01-31T22:21:34.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:34.598173+0000) 2022-01-31T22:21:34.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:34 smithi167 conmon[54076]: debug 2022-01-31T22:21:34.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:34.768359+0000) 2022-01-31T22:21:34.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:34 smithi167 conmon[60316]: debug 2022-01-31T22:21:34.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:34.523971+0000) 2022-01-31T22:21:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:34 smithi171 conmon[41853]: debug 2022-01-31T22:21:34.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:34.471552+0000) 2022-01-31T22:21:35.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:35 smithi171 conmon[46715]: debug 2022-01-31T22:21:35.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:35.264080+0000) 2022-01-31T22:21:35.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:35 smithi171 conmon[51620]: debug 2022-01-31T22:21:35.122+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:35.123222+0000) 2022-01-31T22:21:35.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:35 smithi167 conmon[49112]: debug 2022-01-31T22:21:35.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:35.598313+0000) 2022-01-31T22:21:35.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:35 smithi167 conmon[54076]: debug 2022-01-31T22:21:35.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:35.768532+0000) 2022-01-31T22:21:35.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:35 smithi167 conmon[60316]: debug 2022-01-31T22:21:35.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:35.524193+0000) 2022-01-31T22:21:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:35 smithi171 conmon[41853]: debug 2022-01-31T22:21:35.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:35.471729+0000) 2022-01-31T22:21:36.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:36 smithi171 conmon[46715]: debug 2022-01-31T22:21:36.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:36.264258+0000) 2022-01-31T22:21:36.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:36 smithi171 conmon[51620]: debug 2022-01-31T22:21:36.123+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:36.123375+0000) 2022-01-31T22:21:36.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:36 smithi167 conmon[49112]: debug 2022-01-31T22:21:36.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:36.598519+0000) 2022-01-31T22:21:36.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:36 smithi167 conmon[54076]: debug 2022-01-31T22:21:36.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:36.768720+0000) 2022-01-31T22:21:36.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:36 smithi167 conmon[60316]: debug 2022-01-31T22:21:36.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:36.524378+0000) 2022-01-31T22:21:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:36 smithi171 conmon[41853]: debug 2022-01-31T22:21:36.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:36.471917+0000) 2022-01-31T22:21:37.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:37 smithi171 conmon[46715]: debug 2022-01-31T22:21:37.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:37.264437+0000) 2022-01-31T22:21:37.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:37 smithi171 conmon[51620]: debug 2022-01-31T22:21:37.122+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:37.123567+0000) 2022-01-31T22:21:37.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:37 smithi167 conmon[49112]: debug 2022-01-31T22:21:37.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:37.598712+0000) 2022-01-31T22:21:37.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:37 smithi167 conmon[54076]: debug 2022-01-31T22:21:37.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:37.768862+0000) 2022-01-31T22:21:37.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:37 smithi167 conmon[60316]: debug 2022-01-31T22:21:37.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:37.524575+0000) 2022-01-31T22:21:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:37 smithi171 conmon[41853]: debug 2022-01-31T22:21:37.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:37.472100+0000) 2022-01-31T22:21:38.393 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[46715]: debug 2022-01-31T22:21:38.263+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.264615+0000) 2022-01-31T22:21:38.393 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[51620]: debug 2022-01-31T22:21:38.123+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.123747+0000) 2022-01-31T22:21:38.524 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:38 smithi167 conmon[49112]: debug 2022-01-31T22:21:38.393+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.394926+0000) 2022-01-31T22:21:38.525 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:38 smithi167 conmon[54076]: debug 2022-01-31T22:21:38.393+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.395161+0000) 2022-01-31T22:21:38.525 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:38 smithi167 conmon[60316]: debug 2022-01-31T22:21:38.392+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.393670+0000) 2022-01-31T22:21:38.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:38 smithi167 conmon[49112]: debug 2022-01-31T22:21:38.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.598904+0000) 2022-01-31T22:21:38.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:38 smithi167 conmon[54076]: debug 2022-01-31T22:21:38.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.768975+0000) 2022-01-31T22:21:38.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:38 smithi167 conmon[60316]: debug 2022-01-31T22:21:38.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.524755+0000) 2022-01-31T22:21:38.838 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[35325]: debug 2022-01-31T22:21:38.414+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203225 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:38.839 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[46715]: debug 2022-01-31T22:21:38.392+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.393539+0000) 2022-01-31T22:21:38.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[51620]: debug 2022-01-31T22:21:38.392+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.393275+0000) 2022-01-31T22:21:38.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[41853]: debug 2022-01-31T22:21:38.393+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.394491+0000) 2022-01-31T22:21:38.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[41853]: debug 2022-01-31T22:21:38.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:38.472284+0000) 2022-01-31T22:21:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:38 smithi171 conmon[35325]: debug 2022-01-31T22:21:38.838+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:21:39.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:39 smithi171 conmon[46715]: debug 2022-01-31T22:21:39.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:39.264779+0000) 2022-01-31T22:21:39.431 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:39 smithi171 conmon[51620]: debug 2022-01-31T22:21:39.123+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:39.123953+0000) 2022-01-31T22:21:39.817 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:39 smithi167 conmon[54076]: debug 2022-01-31T22:21:39.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:39.769093+0000) 2022-01-31T22:21:39.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:39 smithi167 conmon[60316]: debug 2022-01-31T22:21:39.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:39.524892+0000) 2022-01-31T22:21:39.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:39 smithi167 conmon[49112]: debug 2022-01-31T22:21:39.597+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:39.599103+0000) 2022-01-31T22:21:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:39 smithi171 conmon[41853]: debug 2022-01-31T22:21:39.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:39.472486+0000) 2022-01-31T22:21:40.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:40 smithi171 conmon[46715]: debug 2022-01-31T22:21:40.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:40.264933+0000) 2022-01-31T22:21:40.431 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:40 smithi171 conmon[51620]: debug 2022-01-31T22:21:40.123+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:40.124149+0000) 2022-01-31T22:21:40.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:40 smithi167 conmon[49112]: debug 2022-01-31T22:21:40.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:40.599318+0000) 2022-01-31T22:21:40.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:40 smithi167 conmon[54076]: debug 2022-01-31T22:21:40.767+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:40.769295+0000) 2022-01-31T22:21:40.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:40 smithi167 conmon[60316]: debug 2022-01-31T22:21:40.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:40.525083+0000) 2022-01-31T22:21:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:40 smithi171 conmon[41853]: debug 2022-01-31T22:21:40.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:40.472701+0000) 2022-01-31T22:21:41.430 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:41 smithi171 conmon[46715]: debug 2022-01-31T22:21:41.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:41.265118+0000) 2022-01-31T22:21:41.431 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:41 smithi171 conmon[51620]: debug 2022-01-31T22:21:41.124+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:41.124343+0000) 2022-01-31T22:21:41.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:41 smithi167 conmon[49112]: debug 2022-01-31T22:21:41.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:41.599513+0000) 2022-01-31T22:21:41.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:41 smithi167 conmon[54076]: debug 2022-01-31T22:21:41.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:41.769425+0000) 2022-01-31T22:21:41.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:41 smithi167 conmon[60316]: debug 2022-01-31T22:21:41.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:41.525273+0000) 2022-01-31T22:21:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:41 smithi171 conmon[41853]: debug 2022-01-31T22:21:41.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:41.472850+0000) 2022-01-31T22:21:42.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:42 smithi171 conmon[46715]: debug 2022-01-31T22:21:42.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:42.265316+0000) 2022-01-31T22:21:42.431 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:42 smithi171 conmon[51620]: debug 2022-01-31T22:21:42.124+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:42.124544+0000) 2022-01-31T22:21:42.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:42 smithi167 conmon[49112]: debug 2022-01-31T22:21:42.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:42.599667+0000) 2022-01-31T22:21:42.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:42 smithi167 conmon[54076]: debug 2022-01-31T22:21:42.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:42.769587+0000) 2022-01-31T22:21:42.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:42 smithi167 conmon[60316]: debug 2022-01-31T22:21:42.523+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:42.525482+0000) 2022-01-31T22:21:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:42 smithi171 conmon[41853]: debug 2022-01-31T22:21:42.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:42.473037+0000) 2022-01-31T22:21:43.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[46715]: debug 2022-01-31T22:21:43.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.265466+0000) 2022-01-31T22:21:43.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[51620]: debug 2022-01-31T22:21:43.124+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.124800+0000) 2022-01-31T22:21:43.417 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[51620]: debug 2022-01-31T22:21:43.416+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.416981+0000) 2022-01-31T22:21:43.525 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:43 smithi167 conmon[49112]: debug 2022-01-31T22:21:43.416+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.418382+0000) 2022-01-31T22:21:43.526 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:43 smithi167 conmon[54076]: debug 2022-01-31T22:21:43.417+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.418564+0000) 2022-01-31T22:21:43.526 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:43 smithi167 conmon[60316]: debug 2022-01-31T22:21:43.416+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.418161+0000) 2022-01-31T22:21:43.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:43 smithi167 conmon[49112]: debug 2022-01-31T22:21:43.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.599808+0000) 2022-01-31T22:21:43.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:43 smithi167 conmon[54076]: debug 2022-01-31T22:21:43.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.769773+0000) 2022-01-31T22:21:43.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:43 smithi167 conmon[60316]: debug 2022-01-31T22:21:43.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.525651+0000) 2022-01-31T22:21:43.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[35325]: debug 2022-01-31T22:21:43.438+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203335 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:43.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[46715]: debug 2022-01-31T22:21:43.416+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.417506+0000) 2022-01-31T22:21:43.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[41853]: debug 2022-01-31T22:21:43.416+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.417620+0000) 2022-01-31T22:21:43.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:43 smithi171 conmon[41853]: debug 2022-01-31T22:21:43.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:43.473236+0000) 2022-01-31T22:21:44.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:44 smithi171 conmon[46715]: debug 2022-01-31T22:21:44.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:44.265669+0000) 2022-01-31T22:21:44.432 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:44 smithi171 conmon[51620]: debug 2022-01-31T22:21:44.124+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:44.124971+0000) 2022-01-31T22:21:44.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:44 smithi167 conmon[49112]: debug 2022-01-31T22:21:44.598+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:44.599933+0000) 2022-01-31T22:21:44.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:44 smithi167 conmon[54076]: debug 2022-01-31T22:21:44.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:44.769899+0000) 2022-01-31T22:21:44.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:44 smithi167 conmon[60316]: debug 2022-01-31T22:21:44.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:44.525843+0000) 2022-01-31T22:21:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:44 smithi171 conmon[41853]: debug 2022-01-31T22:21:44.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:44.473408+0000) 2022-01-31T22:21:45.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:45 smithi171 conmon[46715]: debug 2022-01-31T22:21:45.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:45.265800+0000) 2022-01-31T22:21:45.432 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:45 smithi171 conmon[51620]: debug 2022-01-31T22:21:45.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:45.125161+0000) 2022-01-31T22:21:45.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:45 smithi167 conmon[49112]: debug 2022-01-31T22:21:45.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:45.600140+0000) 2022-01-31T22:21:45.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:45 smithi167 conmon[54076]: debug 2022-01-31T22:21:45.769+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:45.770073+0000) 2022-01-31T22:21:45.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:45 smithi167 conmon[60316]: debug 2022-01-31T22:21:45.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:45.526029+0000) 2022-01-31T22:21:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:45 smithi171 conmon[41853]: debug 2022-01-31T22:21:45.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:45.473612+0000) 2022-01-31T22:21:46.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:46 smithi171 conmon[46715]: debug 2022-01-31T22:21:46.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:46.265954+0000) 2022-01-31T22:21:46.432 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:46 smithi171 conmon[51620]: debug 2022-01-31T22:21:46.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:46.125387+0000) 2022-01-31T22:21:46.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:46 smithi167 conmon[49112]: debug 2022-01-31T22:21:46.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:46.600334+0000) 2022-01-31T22:21:46.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:46 smithi167 conmon[54076]: debug 2022-01-31T22:21:46.769+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:46.770254+0000) 2022-01-31T22:21:46.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:46 smithi167 conmon[60316]: debug 2022-01-31T22:21:46.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:46.526194+0000) 2022-01-31T22:21:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:46 smithi171 conmon[41853]: debug 2022-01-31T22:21:46.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:46.473757+0000) 2022-01-31T22:21:47.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:47 smithi171 conmon[46715]: debug 2022-01-31T22:21:47.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:47.266135+0000) 2022-01-31T22:21:47.432 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:47 smithi171 conmon[51620]: debug 2022-01-31T22:21:47.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:47.125545+0000) 2022-01-31T22:21:47.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:47 smithi167 conmon[49112]: debug 2022-01-31T22:21:47.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:47.600539+0000) 2022-01-31T22:21:47.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:47 smithi167 conmon[54076]: debug 2022-01-31T22:21:47.768+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:47.770386+0000) 2022-01-31T22:21:47.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:47 smithi167 conmon[60316]: debug 2022-01-31T22:21:47.524+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:47.526426+0000) 2022-01-31T22:21:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:47 smithi171 conmon[41853]: debug 2022-01-31T22:21:47.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:47.473910+0000) 2022-01-31T22:21:48.431 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[46715]: debug 2022-01-31T22:21:48.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.266342+0000) 2022-01-31T22:21:48.432 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[51620]: debug 2022-01-31T22:21:48.124+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.125718+0000) 2022-01-31T22:21:48.526 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:48 smithi167 conmon[49112]: debug 2022-01-31T22:21:48.440+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.441684+0000) 2022-01-31T22:21:48.526 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:48 smithi167 conmon[54076]: debug 2022-01-31T22:21:48.441+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.442806+0000) 2022-01-31T22:21:48.527 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:48 smithi167 conmon[60316]: debug 2022-01-31T22:21:48.440+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.442460+0000) 2022-01-31T22:21:48.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:48 smithi167 conmon[60316]: debug 2022-01-31T22:21:48.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.526584+0000) 2022-01-31T22:21:48.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:48 smithi167 conmon[49112]: debug 2022-01-31T22:21:48.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.600719+0000) 2022-01-31T22:21:48.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:48 smithi167 conmon[54076]: debug 2022-01-31T22:21:48.769+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.770542+0000) 2022-01-31T22:21:48.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[46715]: debug 2022-01-31T22:21:48.440+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.440803+0000) 2022-01-31T22:21:48.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[35325]: debug 2022-01-31T22:21:48.461+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203443 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:48.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[41853]: debug 2022-01-31T22:21:48.440+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.441531+0000) 2022-01-31T22:21:48.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[41853]: debug 2022-01-31T22:21:48.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.474035+0000) 2022-01-31T22:21:48.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:48 smithi171 conmon[51620]: debug 2022-01-31T22:21:48.440+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:48.440928+0000) 2022-01-31T22:21:49.432 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:49 smithi171 conmon[46715]: debug 2022-01-31T22:21:49.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:49.266469+0000) 2022-01-31T22:21:49.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:49 smithi171 conmon[51620]: debug 2022-01-31T22:21:49.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:49.125901+0000) 2022-01-31T22:21:49.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:49 smithi167 conmon[49112]: debug 2022-01-31T22:21:49.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:49.600886+0000) 2022-01-31T22:21:49.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:49 smithi167 conmon[54076]: debug 2022-01-31T22:21:49.769+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:49.770698+0000) 2022-01-31T22:21:49.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:49 smithi167 conmon[60316]: debug 2022-01-31T22:21:49.525+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:49.526703+0000) 2022-01-31T22:21:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:49 smithi171 conmon[41853]: debug 2022-01-31T22:21:49.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:49.474174+0000) 2022-01-31T22:21:50.432 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:50 smithi171 conmon[46715]: debug 2022-01-31T22:21:50.264+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:50.266607+0000) 2022-01-31T22:21:50.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:50 smithi171 conmon[51620]: debug 2022-01-31T22:21:50.124+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:50.126005+0000) 2022-01-31T22:21:50.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:50 smithi167 conmon[54076]: debug 2022-01-31T22:21:50.770+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:50.770891+0000) 2022-01-31T22:21:50.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:50 smithi167 conmon[60316]: debug 2022-01-31T22:21:50.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:50.526893+0000) 2022-01-31T22:21:50.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:50 smithi167 conmon[49112]: debug 2022-01-31T22:21:50.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:50.601062+0000) 2022-01-31T22:21:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:50 smithi171 conmon[41853]: debug 2022-01-31T22:21:50.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:50.474306+0000) 2022-01-31T22:21:51.432 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:51 smithi171 conmon[46715]: debug 2022-01-31T22:21:51.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:51.266794+0000) 2022-01-31T22:21:51.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:51 smithi171 conmon[51620]: debug 2022-01-31T22:21:51.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:51.126175+0000) 2022-01-31T22:21:51.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:51 smithi167 conmon[49112]: debug 2022-01-31T22:21:51.599+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:51.601281+0000) 2022-01-31T22:21:51.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:51 smithi167 conmon[54076]: debug 2022-01-31T22:21:51.770+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:51.771071+0000) 2022-01-31T22:21:51.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:51 smithi167 conmon[60316]: debug 2022-01-31T22:21:51.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:51.527071+0000) 2022-01-31T22:21:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:51 smithi171 conmon[41853]: debug 2022-01-31T22:21:51.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:51.474439+0000) 2022-01-31T22:21:52.432 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:52 smithi171 conmon[46715]: debug 2022-01-31T22:21:52.265+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:52.266975+0000) 2022-01-31T22:21:52.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:52 smithi171 conmon[51620]: debug 2022-01-31T22:21:52.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:52.126363+0000) 2022-01-31T22:21:52.770 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:52 smithi167 conmon[49112]: debug 2022-01-31T22:21:52.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:52.601466+0000) 2022-01-31T22:21:52.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:52 smithi167 conmon[60316]: debug 2022-01-31T22:21:52.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:52.527227+0000) 2022-01-31T22:21:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:52 smithi171 conmon[41853]: debug 2022-01-31T22:21:52.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:52.474614+0000) 2022-01-31T22:21:53.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:52 smithi167 conmon[54076]: debug 2022-01-31T22:21:52.769+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:52.771254+0000) 2022-01-31T22:21:53.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[46715]: debug 2022-01-31T22:21:53.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.267220+0000) 2022-01-31T22:21:53.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[51620]: debug 2022-01-31T22:21:53.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.126547+0000) 2022-01-31T22:21:53.771 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:53 smithi167 conmon[54076]: debug 2022-01-31T22:21:53.463+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.465177+0000) 2022-01-31T22:21:53.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:53 smithi167 conmon[49112]: debug 2022-01-31T22:21:53.471+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.473104+0000) 2022-01-31T22:21:53.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:53 smithi167 conmon[49112]: debug 2022-01-31T22:21:53.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.601706+0000) 2022-01-31T22:21:53.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:53 smithi167 conmon[60316]: debug 2022-01-31T22:21:53.472+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.474102+0000) 2022-01-31T22:21:53.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:53 smithi167 conmon[60316]: debug 2022-01-31T22:21:53.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.527384+0000) 2022-01-31T22:21:53.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[41853]: debug 2022-01-31T22:21:53.463+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.465560+0000) 2022-01-31T22:21:53.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[41853]: debug 2022-01-31T22:21:53.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.475083+0000) 2022-01-31T22:21:53.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[46715]: debug 2022-01-31T22:21:53.462+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.464665+0000) 2022-01-31T22:21:53.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[51620]: debug 2022-01-31T22:21:53.462+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.464412+0000) 2022-01-31T22:21:53.842 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[35325]: debug 2022-01-31T22:21:53.490+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203553 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:54.076 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:53 smithi167 conmon[54076]: debug 2022-01-31T22:21:53.769+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:53.771457+0000) 2022-01-31T22:21:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:53 smithi171 conmon[35325]: debug 2022-01-31T22:21:53.838+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:21:54.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:54 smithi171 conmon[46715]: debug 2022-01-31T22:21:54.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:54.267460+0000) 2022-01-31T22:21:54.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:54 smithi171 conmon[51620]: debug 2022-01-31T22:21:54.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:54.126647+0000) 2022-01-31T22:21:54.771 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:54 smithi167 conmon[60316]: debug 2022-01-31T22:21:54.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:54.527591+0000) 2022-01-31T22:21:54.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:54 smithi167 conmon[49112]: debug 2022-01-31T22:21:54.600+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:54.601829+0000) 2022-01-31T22:21:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:54 smithi171 conmon[41853]: debug 2022-01-31T22:21:54.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:54.475591+0000) 2022-01-31T22:21:55.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:54 smithi167 conmon[54076]: debug 2022-01-31T22:21:54.770+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:54.771567+0000) 2022-01-31T22:21:55.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:55 smithi171 conmon[46715]: debug 2022-01-31T22:21:55.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:55.267609+0000) 2022-01-31T22:21:55.433 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:55 smithi171 conmon[51620]: debug 2022-01-31T22:21:55.125+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:55.126796+0000) 2022-01-31T22:21:55.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:55 smithi167 conmon[49112]: debug 2022-01-31T22:21:55.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:55.602011+0000) 2022-01-31T22:21:55.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:55 smithi167 conmon[60316]: debug 2022-01-31T22:21:55.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:55.527767+0000) 2022-01-31T22:21:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:55 smithi171 conmon[41853]: debug 2022-01-31T22:21:55.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:55.475748+0000) 2022-01-31T22:21:56.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:55 smithi167 conmon[54076]: debug 2022-01-31T22:21:55.771+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:55.771784+0000) 2022-01-31T22:21:56.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:56 smithi171 conmon[46715]: debug 2022-01-31T22:21:56.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:56.267781+0000) 2022-01-31T22:21:56.434 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:56 smithi171 conmon[51620]: debug 2022-01-31T22:21:56.126+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:56.126969+0000) 2022-01-31T22:21:56.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:56 smithi167 conmon[49112]: debug 2022-01-31T22:21:56.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:56.602199+0000) 2022-01-31T22:21:56.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:56 smithi167 conmon[60316]: debug 2022-01-31T22:21:56.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:56.527942+0000) 2022-01-31T22:21:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:56 smithi171 conmon[41853]: debug 2022-01-31T22:21:56.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:56.475907+0000) 2022-01-31T22:21:57.087 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:56 smithi167 conmon[54076]: debug 2022-01-31T22:21:56.771+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:56.771917+0000) 2022-01-31T22:21:57.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:57 smithi171 conmon[46715]: debug 2022-01-31T22:21:57.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:57.267975+0000) 2022-01-31T22:21:57.434 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:57 smithi171 conmon[51620]: debug 2022-01-31T22:21:57.126+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:57.127172+0000) 2022-01-31T22:21:57.771 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:57 smithi167 conmon[49112]: debug 2022-01-31T22:21:57.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:57.602357+0000) 2022-01-31T22:21:57.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:57 smithi167 conmon[60316]: debug 2022-01-31T22:21:57.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:57.528068+0000) 2022-01-31T22:21:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:57 smithi171 conmon[41853]: debug 2022-01-31T22:21:57.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:57.476091+0000) 2022-01-31T22:21:58.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:57 smithi167 conmon[54076]: debug 2022-01-31T22:21:57.771+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:57.772066+0000) 2022-01-31T22:21:58.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[46715]: debug 2022-01-31T22:21:58.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.268124+0000) 2022-01-31T22:21:58.434 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[51620]: debug 2022-01-31T22:21:58.126+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.127385+0000) 2022-01-31T22:21:58.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:58 smithi167 conmon[54076]: debug 2022-01-31T22:21:58.493+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.494964+0000) 2022-01-31T22:21:58.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:58 smithi167 conmon[60316]: debug 2022-01-31T22:21:58.494+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.496239+0000) 2022-01-31T22:21:58.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:58 smithi167 conmon[60316]: debug 2022-01-31T22:21:58.526+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.528202+0000) 2022-01-31T22:21:58.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:58 smithi167 conmon[49112]: debug 2022-01-31T22:21:58.494+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.495837+0000) 2022-01-31T22:21:58.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:58 smithi167 conmon[49112]: debug 2022-01-31T22:21:58.601+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.602542+0000) 2022-01-31T22:21:58.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[35325]: debug 2022-01-31T22:21:58.514+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203665 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:21:58.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[41853]: debug 2022-01-31T22:21:58.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.476281+0000) 2022-01-31T22:21:58.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[41853]: debug 2022-01-31T22:21:58.493+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.494811+0000) 2022-01-31T22:21:58.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[46715]: debug 2022-01-31T22:21:58.493+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.495462+0000) 2022-01-31T22:21:58.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:58 smithi171 conmon[51620]: debug 2022-01-31T22:21:58.492+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.494492+0000) 2022-01-31T22:21:59.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:58 smithi167 conmon[54076]: debug 2022-01-31T22:21:58.770+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:58.772242+0000) 2022-01-31T22:21:59.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:21:59 smithi171 conmon[46715]: debug 2022-01-31T22:21:59.266+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:59.268265+0000) 2022-01-31T22:21:59.434 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:21:59 smithi171 conmon[51620]: debug 2022-01-31T22:21:59.126+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:59.127550+0000) 2022-01-31T22:21:59.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:21:59 smithi167 conmon[49112]: debug 2022-01-31T22:21:59.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:59.602721+0000) 2022-01-31T22:21:59.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:21:59 smithi167 conmon[60316]: debug 2022-01-31T22:21:59.527+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:59.528354+0000) 2022-01-31T22:21:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:21:59 smithi171 conmon[41853]: debug 2022-01-31T22:21:59.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:59.476482+0000) 2022-01-31T22:22:00.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:21:59 smithi167 conmon[54076]: debug 2022-01-31T22:21:59.771+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:21:59.772422+0000) 2022-01-31T22:22:00.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:00 smithi171 conmon[46715]: debug 2022-01-31T22:22:00.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:00.268427+0000) 2022-01-31T22:22:00.434 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:00 smithi171 conmon[51620]: debug 2022-01-31T22:22:00.126+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:00.127732+0000) 2022-01-31T22:22:00.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:00 smithi167 conmon[49112]: debug 2022-01-31T22:22:00.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:00.602902+0000) 2022-01-31T22:22:00.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:00 smithi167 conmon[60316]: debug 2022-01-31T22:22:00.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:00.528583+0000) 2022-01-31T22:22:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:00 smithi171 conmon[41853]: debug 2022-01-31T22:22:00.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:00.476615+0000) 2022-01-31T22:22:01.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:00 smithi167 conmon[54076]: debug 2022-01-31T22:22:00.772+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:00.772665+0000) 2022-01-31T22:22:01.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:01 smithi171 conmon[46715]: debug 2022-01-31T22:22:01.267+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:01.268616+0000) 2022-01-31T22:22:01.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:01 smithi171 conmon[51620]: debug 2022-01-31T22:22:01.127+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:01.127915+0000) 2022-01-31T22:22:01.772 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:01 smithi167 conmon[49112]: debug 2022-01-31T22:22:01.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:01.603087+0000) 2022-01-31T22:22:01.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:01 smithi167 conmon[60316]: debug 2022-01-31T22:22:01.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:01.528775+0000) 2022-01-31T22:22:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:01 smithi171 conmon[41853]: debug 2022-01-31T22:22:01.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:01.476821+0000) 2022-01-31T22:22:02.101 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:01 smithi167 conmon[54076]: debug 2022-01-31T22:22:01.772+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:01.772873+0000) 2022-01-31T22:22:02.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:02 smithi171 conmon[46715]: debug 2022-01-31T22:22:02.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:02.268796+0000) 2022-01-31T22:22:02.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:02 smithi171 conmon[51620]: debug 2022-01-31T22:22:02.127+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:02.128049+0000) 2022-01-31T22:22:02.772 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:02 smithi167 conmon[60316]: debug 2022-01-31T22:22:02.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:02.528942+0000) 2022-01-31T22:22:02.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:02 smithi167 conmon[49112]: debug 2022-01-31T22:22:02.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:02.603292+0000) 2022-01-31T22:22:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:02 smithi171 conmon[41853]: debug 2022-01-31T22:22:02.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:02.476995+0000) 2022-01-31T22:22:03.104 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:02 smithi167 conmon[54076]: debug 2022-01-31T22:22:02.772+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:02.773060+0000) 2022-01-31T22:22:03.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[46715]: debug 2022-01-31T22:22:03.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.268951+0000) 2022-01-31T22:22:03.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[51620]: debug 2022-01-31T22:22:03.127+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.128200+0000) 2022-01-31T22:22:03.772 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:03 smithi167 conmon[54076]: debug 2022-01-31T22:22:03.519+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.519859+0000) 2022-01-31T22:22:03.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:03 smithi167 conmon[60316]: debug 2022-01-31T22:22:03.519+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.519954+0000) 2022-01-31T22:22:03.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:03 smithi167 conmon[60316]: debug 2022-01-31T22:22:03.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.529130+0000) 2022-01-31T22:22:03.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:03 smithi167 conmon[49112]: debug 2022-01-31T22:22:03.519+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.519897+0000) 2022-01-31T22:22:03.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:03 smithi167 conmon[49112]: debug 2022-01-31T22:22:03.602+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.603468+0000) 2022-01-31T22:22:03.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[35325]: debug 2022-01-31T22:22:03.540+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203775 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:03.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[46715]: debug 2022-01-31T22:22:03.518+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.518892+0000) 2022-01-31T22:22:03.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[51620]: debug 2022-01-31T22:22:03.517+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.518091+0000) 2022-01-31T22:22:03.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[41853]: debug 2022-01-31T22:22:03.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.477162+0000) 2022-01-31T22:22:03.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:03 smithi171 conmon[41853]: debug 2022-01-31T22:22:03.517+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.518208+0000) 2022-01-31T22:22:04.104 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:03 smithi167 conmon[54076]: debug 2022-01-31T22:22:03.772+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:03.773271+0000) 2022-01-31T22:22:04.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:04 smithi171 conmon[46715]: debug 2022-01-31T22:22:04.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:04.269174+0000) 2022-01-31T22:22:04.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:04 smithi171 conmon[51620]: debug 2022-01-31T22:22:04.127+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:04.128393+0000) 2022-01-31T22:22:04.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:04 smithi167 conmon[49112]: debug 2022-01-31T22:22:04.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:04.603612+0000) 2022-01-31T22:22:04.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:04 smithi167 conmon[60316]: debug 2022-01-31T22:22:04.528+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:04.529309+0000) 2022-01-31T22:22:04.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:04 smithi171 conmon[41853]: debug 2022-01-31T22:22:04.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:04.477322+0000) 2022-01-31T22:22:05.108 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:04 smithi167 conmon[54076]: debug 2022-01-31T22:22:04.772+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:04.773448+0000) 2022-01-31T22:22:05.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:05 smithi171 conmon[46715]: debug 2022-01-31T22:22:05.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:05.269342+0000) 2022-01-31T22:22:05.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:05 smithi171 conmon[51620]: debug 2022-01-31T22:22:05.127+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:05.128564+0000) 2022-01-31T22:22:05.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:05 smithi167 conmon[49112]: debug 2022-01-31T22:22:05.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:05.603822+0000) 2022-01-31T22:22:05.773 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:05 smithi167 conmon[60316]: debug 2022-01-31T22:22:05.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:05.529531+0000) 2022-01-31T22:22:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:05 smithi171 conmon[41853]: debug 2022-01-31T22:22:05.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:05.477470+0000) 2022-01-31T22:22:06.112 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:05 smithi167 conmon[54076]: debug 2022-01-31T22:22:05.773+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:05.773646+0000) 2022-01-31T22:22:06.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:06 smithi171 conmon[46715]: debug 2022-01-31T22:22:06.268+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:06.269540+0000) 2022-01-31T22:22:06.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:06 smithi171 conmon[51620]: debug 2022-01-31T22:22:06.128+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:06.128742+0000) 2022-01-31T22:22:06.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:06 smithi167 conmon[49112]: debug 2022-01-31T22:22:06.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:06.604008+0000) 2022-01-31T22:22:06.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:06 smithi167 conmon[60316]: debug 2022-01-31T22:22:06.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:06.529714+0000) 2022-01-31T22:22:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:06 smithi171 conmon[41853]: debug 2022-01-31T22:22:06.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:06.477598+0000) 2022-01-31T22:22:07.115 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:06 smithi167 conmon[54076]: debug 2022-01-31T22:22:06.773+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:06.773820+0000) 2022-01-31T22:22:07.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:07 smithi171 conmon[46715]: debug 2022-01-31T22:22:07.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:07.269746+0000) 2022-01-31T22:22:07.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:07 smithi171 conmon[51620]: debug 2022-01-31T22:22:07.128+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:07.128925+0000) 2022-01-31T22:22:07.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:07 smithi167 conmon[49112]: debug 2022-01-31T22:22:07.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:07.604191+0000) 2022-01-31T22:22:07.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:07 smithi167 conmon[60316]: debug 2022-01-31T22:22:07.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:07.529875+0000) 2022-01-31T22:22:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:07 smithi171 conmon[41853]: debug 2022-01-31T22:22:07.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:07.477776+0000) 2022-01-31T22:22:08.118 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:07 smithi167 conmon[54076]: debug 2022-01-31T22:22:07.773+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:07.773988+0000) 2022-01-31T22:22:08.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[46715]: debug 2022-01-31T22:22:08.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.269947+0000) 2022-01-31T22:22:08.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[51620]: debug 2022-01-31T22:22:08.128+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.129089+0000) 2022-01-31T22:22:08.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:08 smithi167 conmon[49112]: debug 2022-01-31T22:22:08.544+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.544689+0000) 2022-01-31T22:22:08.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:08 smithi167 conmon[49112]: debug 2022-01-31T22:22:08.603+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.604396+0000) 2022-01-31T22:22:08.775 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:08 smithi167 conmon[54076]: debug 2022-01-31T22:22:08.544+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.544799+0000) 2022-01-31T22:22:08.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:08 smithi167 conmon[60316]: debug 2022-01-31T22:22:08.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.530065+0000) 2022-01-31T22:22:08.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:08 smithi167 conmon[60316]: debug 2022-01-31T22:22:08.544+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.544915+0000) 2022-01-31T22:22:08.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[35325]: debug 2022-01-31T22:22:08.563+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:08.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[46715]: debug 2022-01-31T22:22:08.542+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.543591+0000) 2022-01-31T22:22:08.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[51620]: debug 2022-01-31T22:22:08.544+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.544803+0000) 2022-01-31T22:22:08.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[41853]: debug 2022-01-31T22:22:08.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.477907+0000) 2022-01-31T22:22:08.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[41853]: debug 2022-01-31T22:22:08.542+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.542880+0000) 2022-01-31T22:22:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:08 smithi171 conmon[35325]: debug 2022-01-31T22:22:08.839+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:22:09.118 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:08 smithi167 conmon[54076]: debug 2022-01-31T22:22:08.773+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:08.774163+0000) 2022-01-31T22:22:09.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:09 smithi171 conmon[46715]: debug 2022-01-31T22:22:09.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:09.270104+0000) 2022-01-31T22:22:09.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:09 smithi171 conmon[51620]: debug 2022-01-31T22:22:09.128+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:09.129211+0000) 2022-01-31T22:22:09.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:09 smithi167 conmon[49112]: debug 2022-01-31T22:22:09.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:09.604594+0000) 2022-01-31T22:22:09.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:09 smithi167 conmon[60316]: debug 2022-01-31T22:22:09.529+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:09.530270+0000) 2022-01-31T22:22:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:09 smithi171 conmon[41853]: debug 2022-01-31T22:22:09.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:09.478042+0000) 2022-01-31T22:22:10.122 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:09 smithi167 conmon[54076]: debug 2022-01-31T22:22:09.773+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:09.774305+0000) 2022-01-31T22:22:10.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:10 smithi171 conmon[46715]: debug 2022-01-31T22:22:10.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:10.270253+0000) 2022-01-31T22:22:10.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:10 smithi171 conmon[51620]: debug 2022-01-31T22:22:10.128+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:10.129313+0000) 2022-01-31T22:22:10.773 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:10 smithi167 conmon[49112]: debug 2022-01-31T22:22:10.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:10.604777+0000) 2022-01-31T22:22:10.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:10 smithi167 conmon[60316]: debug 2022-01-31T22:22:10.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:10.530412+0000) 2022-01-31T22:22:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:10 smithi171 conmon[41853]: debug 2022-01-31T22:22:10.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:10.478163+0000) 2022-01-31T22:22:11.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:10 smithi167 conmon[54076]: debug 2022-01-31T22:22:10.773+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:10.774493+0000) 2022-01-31T22:22:11.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:11 smithi171 conmon[46715]: debug 2022-01-31T22:22:11.269+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:11.270409+0000) 2022-01-31T22:22:11.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:11 smithi171 conmon[51620]: debug 2022-01-31T22:22:11.128+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:11.129503+0000) 2022-01-31T22:22:11.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:11 smithi167 conmon[49112]: debug 2022-01-31T22:22:11.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:11.604972+0000) 2022-01-31T22:22:11.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:11 smithi167 conmon[60316]: debug 2022-01-31T22:22:11.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:11.530621+0000) 2022-01-31T22:22:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:11 smithi171 conmon[41853]: debug 2022-01-31T22:22:11.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:11.478329+0000) 2022-01-31T22:22:12.129 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:11 smithi167 conmon[54076]: debug 2022-01-31T22:22:11.774+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:11.774660+0000) 2022-01-31T22:22:12.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:12 smithi171 conmon[46715]: debug 2022-01-31T22:22:12.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:12.270590+0000) 2022-01-31T22:22:12.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:12 smithi171 conmon[51620]: debug 2022-01-31T22:22:12.129+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:12.129729+0000) 2022-01-31T22:22:12.774 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:12 smithi167 conmon[60316]: debug 2022-01-31T22:22:12.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:12.530816+0000) 2022-01-31T22:22:12.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:12 smithi167 conmon[49112]: debug 2022-01-31T22:22:12.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:12.605128+0000) 2022-01-31T22:22:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:12 smithi171 conmon[41853]: debug 2022-01-31T22:22:12.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:12.478460+0000) 2022-01-31T22:22:13.132 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:12 smithi167 conmon[54076]: debug 2022-01-31T22:22:12.774+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:12.774819+0000) 2022-01-31T22:22:13.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[46715]: debug 2022-01-31T22:22:13.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.270794+0000) 2022-01-31T22:22:13.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[51620]: debug 2022-01-31T22:22:13.129+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.129922+0000) 2022-01-31T22:22:13.774 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:13 smithi167 conmon[54076]: debug 2022-01-31T22:22:13.566+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.567401+0000) 2022-01-31T22:22:13.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:13 smithi167 conmon[49112]: debug 2022-01-31T22:22:13.566+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.566842+0000) 2022-01-31T22:22:13.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:13 smithi167 conmon[49112]: debug 2022-01-31T22:22:13.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.605280+0000) 2022-01-31T22:22:13.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:13 smithi167 conmon[60316]: debug 2022-01-31T22:22:13.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.531027+0000) 2022-01-31T22:22:13.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:13 smithi167 conmon[60316]: debug 2022-01-31T22:22:13.566+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.566988+0000) 2022-01-31T22:22:13.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[35325]: debug 2022-01-31T22:22:13.586+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 203998 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:13.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[46715]: debug 2022-01-31T22:22:13.566+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.567004+0000) 2022-01-31T22:22:13.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[51620]: debug 2022-01-31T22:22:13.564+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.565719+0000) 2022-01-31T22:22:13.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[41853]: debug 2022-01-31T22:22:13.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.478648+0000) 2022-01-31T22:22:13.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:13 smithi171 conmon[41853]: debug 2022-01-31T22:22:13.565+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.566205+0000) 2022-01-31T22:22:14.132 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:13 smithi167 conmon[54076]: debug 2022-01-31T22:22:13.774+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:13.775000+0000) 2022-01-31T22:22:14.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:14 smithi171 conmon[46715]: debug 2022-01-31T22:22:14.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:14.270982+0000) 2022-01-31T22:22:14.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:14 smithi171 conmon[51620]: debug 2022-01-31T22:22:14.129+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:14.130079+0000) 2022-01-31T22:22:14.774 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:14 smithi167 conmon[49112]: debug 2022-01-31T22:22:14.604+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:14.605435+0000) 2022-01-31T22:22:14.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:14 smithi167 conmon[60316]: debug 2022-01-31T22:22:14.530+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:14.531227+0000) 2022-01-31T22:22:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:14 smithi171 conmon[41853]: debug 2022-01-31T22:22:14.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:14.478833+0000) 2022-01-31T22:22:15.136 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:14 smithi167 conmon[54076]: debug 2022-01-31T22:22:14.774+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:14.775112+0000) 2022-01-31T22:22:15.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:15 smithi171 conmon[46715]: debug 2022-01-31T22:22:15.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:15.271120+0000) 2022-01-31T22:22:15.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:15 smithi171 conmon[51620]: debug 2022-01-31T22:22:15.129+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:15.130241+0000) 2022-01-31T22:22:15.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:15 smithi167 conmon[49112]: debug 2022-01-31T22:22:15.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:15.605562+0000) 2022-01-31T22:22:15.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:15 smithi167 conmon[60316]: debug 2022-01-31T22:22:15.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:15.531388+0000) 2022-01-31T22:22:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:15 smithi171 conmon[41853]: debug 2022-01-31T22:22:15.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:15.478944+0000) 2022-01-31T22:22:16.139 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:15 smithi167 conmon[54076]: debug 2022-01-31T22:22:15.774+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:15.775267+0000) 2022-01-31T22:22:16.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:16 smithi171 conmon[46715]: debug 2022-01-31T22:22:16.270+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:16.271307+0000) 2022-01-31T22:22:16.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:16 smithi171 conmon[51620]: debug 2022-01-31T22:22:16.129+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:16.130481+0000) 2022-01-31T22:22:16.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:16 smithi167 conmon[49112]: debug 2022-01-31T22:22:16.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:16.605748+0000) 2022-01-31T22:22:16.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:16 smithi167 conmon[60316]: debug 2022-01-31T22:22:16.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:16.531596+0000) 2022-01-31T22:22:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:16 smithi171 conmon[41853]: debug 2022-01-31T22:22:16.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:16.479133+0000) 2022-01-31T22:22:17.143 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:16 smithi167 conmon[54076]: debug 2022-01-31T22:22:16.775+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:16.775485+0000) 2022-01-31T22:22:17.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:17 smithi171 conmon[46715]: debug 2022-01-31T22:22:17.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:17.271489+0000) 2022-01-31T22:22:17.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:17 smithi171 conmon[51620]: debug 2022-01-31T22:22:17.129+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:17.130690+0000) 2022-01-31T22:22:17.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:17 smithi167 conmon[49112]: debug 2022-01-31T22:22:17.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:17.605926+0000) 2022-01-31T22:22:17.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:17 smithi167 conmon[60316]: debug 2022-01-31T22:22:17.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:17.531791+0000) 2022-01-31T22:22:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:17 smithi171 conmon[41853]: debug 2022-01-31T22:22:17.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:17.479288+0000) 2022-01-31T22:22:18.145 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:17 smithi167 conmon[54076]: debug 2022-01-31T22:22:17.775+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:17.775688+0000) 2022-01-31T22:22:18.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[46715]: debug 2022-01-31T22:22:18.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.271641+0000) 2022-01-31T22:22:18.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[51620]: debug 2022-01-31T22:22:18.130+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.130927+0000) 2022-01-31T22:22:18.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:18 smithi167 conmon[49112]: debug 2022-01-31T22:22:18.589+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.590161+0000) 2022-01-31T22:22:18.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:18 smithi167 conmon[49112]: debug 2022-01-31T22:22:18.605+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.606086+0000) 2022-01-31T22:22:18.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:18 smithi167 conmon[54076]: debug 2022-01-31T22:22:18.590+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.591275+0000) 2022-01-31T22:22:18.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:18 smithi167 conmon[60316]: debug 2022-01-31T22:22:18.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.531986+0000) 2022-01-31T22:22:18.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:18 smithi167 conmon[60316]: debug 2022-01-31T22:22:18.590+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.591410+0000) 2022-01-31T22:22:18.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[35325]: debug 2022-01-31T22:22:18.610+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204110 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[41853]: debug 2022-01-31T22:22:18.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.479487+0000) 2022-01-31T22:22:18.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[41853]: debug 2022-01-31T22:22:18.588+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.589416+0000) 2022-01-31T22:22:18.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[46715]: debug 2022-01-31T22:22:18.590+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.591455+0000) 2022-01-31T22:22:18.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:18 smithi171 conmon[51620]: debug 2022-01-31T22:22:18.588+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.589511+0000) 2022-01-31T22:22:19.146 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:18 smithi167 conmon[54076]: debug 2022-01-31T22:22:18.775+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:18.775842+0000) 2022-01-31T22:22:19.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:19 smithi171 conmon[46715]: debug 2022-01-31T22:22:19.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:19.271846+0000) 2022-01-31T22:22:19.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:19 smithi171 conmon[51620]: debug 2022-01-31T22:22:19.130+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:19.131131+0000) 2022-01-31T22:22:19.737 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:19 smithi171 conmon[41853]: debug 2022-01-31T22:22:19.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:19.479663+0000) 2022-01-31T22:22:19.775 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:19 smithi167 conmon[49112]: debug 2022-01-31T22:22:19.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:19.606260+0000) 2022-01-31T22:22:19.775 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:19 smithi167 conmon[60316]: debug 2022-01-31T22:22:19.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:19.532146+0000) 2022-01-31T22:22:20.149 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:19 smithi167 conmon[54076]: debug 2022-01-31T22:22:19.775+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:19.775976+0000) 2022-01-31T22:22:20.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:20 smithi171 conmon[46715]: debug 2022-01-31T22:22:20.271+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:20.272155+0000) 2022-01-31T22:22:20.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:20 smithi171 conmon[51620]: debug 2022-01-31T22:22:20.130+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:20.131227+0000) 2022-01-31T22:22:20.708 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:20 smithi167 conmon[49112]: debug 2022-01-31T22:22:20.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:20.606447+0000) 2022-01-31T22:22:20.708 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:20 smithi167 conmon[60316]: debug 2022-01-31T22:22:20.531+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:20.532309+0000) 2022-01-31T22:22:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:20 smithi171 conmon[41853]: debug 2022-01-31T22:22:20.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:20.479828+0000) 2022-01-31T22:22:21.153 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:20 smithi167 conmon[54076]: debug 2022-01-31T22:22:20.775+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:20.776145+0000) 2022-01-31T22:22:21.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:21 smithi171 conmon[46715]: debug 2022-01-31T22:22:21.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:21.272831+0000) 2022-01-31T22:22:21.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:21 smithi171 conmon[51620]: debug 2022-01-31T22:22:21.131+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:21.131419+0000) 2022-01-31T22:22:21.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:21 smithi167 conmon[49112]: debug 2022-01-31T22:22:21.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:21.606632+0000) 2022-01-31T22:22:21.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:21 smithi167 conmon[60316]: debug 2022-01-31T22:22:21.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:21.532471+0000) 2022-01-31T22:22:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:21 smithi171 conmon[41853]: debug 2022-01-31T22:22:21.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:21.480003+0000) 2022-01-31T22:22:22.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:21 smithi167 conmon[54076]: debug 2022-01-31T22:22:21.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:21.776325+0000) 2022-01-31T22:22:22.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:22 smithi171 conmon[46715]: debug 2022-01-31T22:22:22.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:22.272991+0000) 2022-01-31T22:22:22.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:22 smithi171 conmon[51620]: debug 2022-01-31T22:22:22.130+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:22.131608+0000) 2022-01-31T22:22:22.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:22 smithi167 conmon[49112]: debug 2022-01-31T22:22:22.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:22.606797+0000) 2022-01-31T22:22:22.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:22 smithi167 conmon[60316]: debug 2022-01-31T22:22:22.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:22.532672+0000) 2022-01-31T22:22:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:22 smithi171 conmon[41853]: debug 2022-01-31T22:22:22.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:22.480207+0000) 2022-01-31T22:22:23.159 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:22 smithi167 conmon[54076]: debug 2022-01-31T22:22:22.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:22.776541+0000) 2022-01-31T22:22:23.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[46715]: debug 2022-01-31T22:22:23.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.273119+0000) 2022-01-31T22:22:23.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[51620]: debug 2022-01-31T22:22:23.130+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.131739+0000) 2022-01-31T22:22:23.776 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:23 smithi167 conmon[54076]: debug 2022-01-31T22:22:23.614+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.614869+0000) 2022-01-31T22:22:23.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:23 smithi167 conmon[60316]: debug 2022-01-31T22:22:23.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.532809+0000) 2022-01-31T22:22:23.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:23 smithi167 conmon[60316]: debug 2022-01-31T22:22:23.613+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.614181+0000) 2022-01-31T22:22:23.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:23 smithi167 conmon[49112]: debug 2022-01-31T22:22:23.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.606963+0000) 2022-01-31T22:22:23.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:23 smithi167 conmon[49112]: debug 2022-01-31T22:22:23.613+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.613692+0000) 2022-01-31T22:22:23.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[35325]: debug 2022-01-31T22:22:23.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204220 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:23.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[46715]: debug 2022-01-31T22:22:23.614+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.615409+0000) 2022-01-31T22:22:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[51620]: debug 2022-01-31T22:22:23.613+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.613988+0000) 2022-01-31T22:22:23.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[41853]: debug 2022-01-31T22:22:23.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.480407+0000) 2022-01-31T22:22:23.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[41853]: debug 2022-01-31T22:22:23.613+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.613927+0000) 2022-01-31T22:22:24.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:23 smithi171 conmon[35325]: debug 2022-01-31T22:22:23.840+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:22:24.159 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:23 smithi167 conmon[54076]: debug 2022-01-31T22:22:23.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:23.776660+0000) 2022-01-31T22:22:24.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:24 smithi171 conmon[46715]: debug 2022-01-31T22:22:24.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:24.273326+0000) 2022-01-31T22:22:24.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:24 smithi171 conmon[51620]: debug 2022-01-31T22:22:24.131+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:24.131860+0000) 2022-01-31T22:22:24.776 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:24 smithi167 conmon[49112]: debug 2022-01-31T22:22:24.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:24.607118+0000) 2022-01-31T22:22:24.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:24 smithi167 conmon[60316]: debug 2022-01-31T22:22:24.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:24.532964+0000) 2022-01-31T22:22:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:24 smithi171 conmon[41853]: debug 2022-01-31T22:22:24.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:24.480601+0000) 2022-01-31T22:22:25.163 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:24 smithi167 conmon[54076]: debug 2022-01-31T22:22:24.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:24.776774+0000) 2022-01-31T22:22:25.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:25 smithi171 conmon[46715]: debug 2022-01-31T22:22:25.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:25.273477+0000) 2022-01-31T22:22:25.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:25 smithi171 conmon[51620]: debug 2022-01-31T22:22:25.131+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:25.132008+0000) 2022-01-31T22:22:25.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:25 smithi167 conmon[60316]: debug 2022-01-31T22:22:25.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:25.533113+0000) 2022-01-31T22:22:25.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:25 smithi167 conmon[49112]: debug 2022-01-31T22:22:25.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:25.607255+0000) 2022-01-31T22:22:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:25 smithi171 conmon[41853]: debug 2022-01-31T22:22:25.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:25.480796+0000) 2022-01-31T22:22:26.167 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:25 smithi167 conmon[54076]: debug 2022-01-31T22:22:25.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:25.776926+0000) 2022-01-31T22:22:26.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:26 smithi171 conmon[46715]: debug 2022-01-31T22:22:26.272+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:26.273676+0000) 2022-01-31T22:22:26.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:26 smithi171 conmon[51620]: debug 2022-01-31T22:22:26.131+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:26.132108+0000) 2022-01-31T22:22:26.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:26 smithi167 conmon[60316]: debug 2022-01-31T22:22:26.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:26.533283+0000) 2022-01-31T22:22:26.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:26 smithi167 conmon[49112]: debug 2022-01-31T22:22:26.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:26.607457+0000) 2022-01-31T22:22:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:26 smithi171 conmon[41853]: debug 2022-01-31T22:22:26.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:26.480945+0000) 2022-01-31T22:22:27.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:26 smithi167 conmon[54076]: debug 2022-01-31T22:22:26.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:26.777119+0000) 2022-01-31T22:22:27.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:27 smithi171 conmon[46715]: debug 2022-01-31T22:22:27.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:27.273865+0000) 2022-01-31T22:22:27.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:27 smithi171 conmon[51620]: debug 2022-01-31T22:22:27.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:27.132258+0000) 2022-01-31T22:22:27.776 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:27 smithi167 conmon[60316]: debug 2022-01-31T22:22:27.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:27.533467+0000) 2022-01-31T22:22:27.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:27 smithi167 conmon[49112]: debug 2022-01-31T22:22:27.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:27.607589+0000) 2022-01-31T22:22:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:27 smithi171 conmon[41853]: debug 2022-01-31T22:22:27.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:27.481081+0000) 2022-01-31T22:22:28.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:27 smithi167 conmon[54076]: debug 2022-01-31T22:22:27.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:27.777304+0000) 2022-01-31T22:22:28.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[46715]: debug 2022-01-31T22:22:28.273+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.274050+0000) 2022-01-31T22:22:28.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[51620]: debug 2022-01-31T22:22:28.131+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.132413+0000) 2022-01-31T22:22:28.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:28 smithi167 conmon[49112]: debug 2022-01-31T22:22:28.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.607682+0000) 2022-01-31T22:22:28.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:28 smithi167 conmon[49112]: debug 2022-01-31T22:22:28.636+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.638341+0000) 2022-01-31T22:22:28.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:28 smithi167 conmon[54076]: debug 2022-01-31T22:22:28.636+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.638001+0000) 2022-01-31T22:22:28.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:28 smithi167 conmon[60316]: debug 2022-01-31T22:22:28.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.533630+0000) 2022-01-31T22:22:28.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:28 smithi167 conmon[60316]: debug 2022-01-31T22:22:28.635+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.637463+0000) 2022-01-31T22:22:28.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[35325]: debug 2022-01-31T22:22:28.657+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204332 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:28.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[46715]: debug 2022-01-31T22:22:28.636+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.637313+0000) 2022-01-31T22:22:28.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[51620]: debug 2022-01-31T22:22:28.637+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.637822+0000) 2022-01-31T22:22:28.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[41853]: debug 2022-01-31T22:22:28.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.481195+0000) 2022-01-31T22:22:28.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:28 smithi171 conmon[41853]: debug 2022-01-31T22:22:28.637+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.638371+0000) 2022-01-31T22:22:29.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:28 smithi167 conmon[54076]: debug 2022-01-31T22:22:28.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:28.777467+0000) 2022-01-31T22:22:29.439 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:29 smithi171 conmon[46715]: debug 2022-01-31T22:22:29.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:29.274237+0000) 2022-01-31T22:22:29.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:29 smithi171 conmon[51620]: debug 2022-01-31T22:22:29.131+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:29.132651+0000) 2022-01-31T22:22:29.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:29 smithi167 conmon[49112]: debug 2022-01-31T22:22:29.606+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:29.607881+0000) 2022-01-31T22:22:29.777 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:29 smithi167 conmon[60316]: debug 2022-01-31T22:22:29.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:29.533834+0000) 2022-01-31T22:22:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:29 smithi171 conmon[41853]: debug 2022-01-31T22:22:29.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:29.481379+0000) 2022-01-31T22:22:30.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:29 smithi167 conmon[54076]: debug 2022-01-31T22:22:29.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:29.777675+0000) 2022-01-31T22:22:30.439 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:30 smithi171 conmon[46715]: debug 2022-01-31T22:22:30.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:30.274378+0000) 2022-01-31T22:22:30.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:30 smithi171 conmon[51620]: debug 2022-01-31T22:22:30.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:30.132821+0000) 2022-01-31T22:22:30.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:30 smithi167 conmon[49112]: debug 2022-01-31T22:22:30.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:30.608006+0000) 2022-01-31T22:22:30.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:30 smithi167 conmon[60316]: debug 2022-01-31T22:22:30.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:30.533929+0000) 2022-01-31T22:22:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:30 smithi171 conmon[41853]: debug 2022-01-31T22:22:30.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:30.481491+0000) 2022-01-31T22:22:31.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:30 smithi167 conmon[54076]: debug 2022-01-31T22:22:30.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:30.777838+0000) 2022-01-31T22:22:31.439 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:31 smithi171 conmon[46715]: debug 2022-01-31T22:22:31.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:31.274538+0000) 2022-01-31T22:22:31.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:31 smithi171 conmon[51620]: debug 2022-01-31T22:22:31.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:31.132958+0000) 2022-01-31T22:22:31.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:31 smithi167 conmon[49112]: debug 2022-01-31T22:22:31.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:31.608178+0000) 2022-01-31T22:22:31.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:31 smithi167 conmon[60316]: debug 2022-01-31T22:22:31.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:31.534104+0000) 2022-01-31T22:22:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:31 smithi171 conmon[41853]: debug 2022-01-31T22:22:31.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:31.481669+0000) 2022-01-31T22:22:32.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:31 smithi167 conmon[54076]: debug 2022-01-31T22:22:31.776+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:31.778009+0000) 2022-01-31T22:22:32.439 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:32 smithi171 conmon[46715]: debug 2022-01-31T22:22:32.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:32.274724+0000) 2022-01-31T22:22:32.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:32 smithi171 conmon[51620]: debug 2022-01-31T22:22:32.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:32.133116+0000) 2022-01-31T22:22:32.777 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:32 smithi167 conmon[49112]: debug 2022-01-31T22:22:32.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:32.608332+0000) 2022-01-31T22:22:32.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:32 smithi167 conmon[60316]: debug 2022-01-31T22:22:32.532+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:32.534285+0000) 2022-01-31T22:22:32.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:32 smithi171 conmon[41853]: debug 2022-01-31T22:22:32.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:32.481801+0000) 2022-01-31T22:22:33.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:32 smithi167 conmon[54076]: debug 2022-01-31T22:22:32.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:32.778205+0000) 2022-01-31T22:22:33.439 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[46715]: debug 2022-01-31T22:22:33.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.274946+0000) 2022-01-31T22:22:33.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[51620]: debug 2022-01-31T22:22:33.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.133274+0000) 2022-01-31T22:22:33.778 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:33 smithi167 conmon[54076]: debug 2022-01-31T22:22:33.659+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.661054+0000) 2022-01-31T22:22:33.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:33 smithi167 conmon[60316]: debug 2022-01-31T22:22:33.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.534445+0000) 2022-01-31T22:22:33.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:33 smithi167 conmon[60316]: debug 2022-01-31T22:22:33.659+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.660910+0000) 2022-01-31T22:22:33.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:33 smithi167 conmon[49112]: debug 2022-01-31T22:22:33.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.608536+0000) 2022-01-31T22:22:33.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:33 smithi167 conmon[49112]: debug 2022-01-31T22:22:33.659+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.661324+0000) 2022-01-31T22:22:33.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[35325]: debug 2022-01-31T22:22:33.681+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204442 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:33.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[46715]: debug 2022-01-31T22:22:33.660+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.661531+0000) 2022-01-31T22:22:33.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[51620]: debug 2022-01-31T22:22:33.660+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.661163+0000) 2022-01-31T22:22:33.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[41853]: debug 2022-01-31T22:22:33.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.482002+0000) 2022-01-31T22:22:33.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:33 smithi171 conmon[41853]: debug 2022-01-31T22:22:33.660+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.661067+0000) 2022-01-31T22:22:34.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:33 smithi167 conmon[54076]: debug 2022-01-31T22:22:33.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:33.778367+0000) 2022-01-31T22:22:34.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:34 smithi171 conmon[46715]: debug 2022-01-31T22:22:34.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:34.275156+0000) 2022-01-31T22:22:34.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:34 smithi171 conmon[51620]: debug 2022-01-31T22:22:34.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:34.133376+0000) 2022-01-31T22:22:34.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:34 smithi167 conmon[49112]: debug 2022-01-31T22:22:34.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:34.608685+0000) 2022-01-31T22:22:34.778 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:34 smithi167 conmon[60316]: debug 2022-01-31T22:22:34.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:34.534580+0000) 2022-01-31T22:22:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:34 smithi171 conmon[41853]: debug 2022-01-31T22:22:34.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:34.482188+0000) 2022-01-31T22:22:35.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:34 smithi167 conmon[54076]: debug 2022-01-31T22:22:34.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:34.778546+0000) 2022-01-31T22:22:35.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:35 smithi171 conmon[46715]: debug 2022-01-31T22:22:35.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:35.275309+0000) 2022-01-31T22:22:35.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:35 smithi171 conmon[51620]: debug 2022-01-31T22:22:35.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:35.133518+0000) 2022-01-31T22:22:35.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:35 smithi167 conmon[49112]: debug 2022-01-31T22:22:35.607+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:35.608811+0000) 2022-01-31T22:22:35.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:35 smithi167 conmon[60316]: debug 2022-01-31T22:22:35.533+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:35.534750+0000) 2022-01-31T22:22:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:35 smithi171 conmon[41853]: debug 2022-01-31T22:22:35.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:35.482345+0000) 2022-01-31T22:22:36.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:35 smithi167 conmon[54076]: debug 2022-01-31T22:22:35.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:35.778716+0000) 2022-01-31T22:22:36.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:36 smithi171 conmon[46715]: debug 2022-01-31T22:22:36.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:36.275479+0000) 2022-01-31T22:22:36.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:36 smithi171 conmon[51620]: debug 2022-01-31T22:22:36.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:36.133667+0000) 2022-01-31T22:22:36.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:36 smithi167 conmon[49112]: debug 2022-01-31T22:22:36.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:36.608969+0000) 2022-01-31T22:22:36.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:36 smithi167 conmon[60316]: debug 2022-01-31T22:22:36.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:36.534992+0000) 2022-01-31T22:22:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:36 smithi171 conmon[41853]: debug 2022-01-31T22:22:36.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:36.482511+0000) 2022-01-31T22:22:37.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:36 smithi167 conmon[54076]: debug 2022-01-31T22:22:36.778+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:36.778946+0000) 2022-01-31T22:22:37.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:37 smithi171 conmon[46715]: debug 2022-01-31T22:22:37.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:37.275628+0000) 2022-01-31T22:22:37.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:37 smithi171 conmon[51620]: debug 2022-01-31T22:22:37.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:37.133830+0000) 2022-01-31T22:22:37.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:37 smithi167 conmon[49112]: debug 2022-01-31T22:22:37.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:37.609161+0000) 2022-01-31T22:22:37.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:37 smithi167 conmon[60316]: debug 2022-01-31T22:22:37.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:37.535122+0000) 2022-01-31T22:22:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:37 smithi171 conmon[41853]: debug 2022-01-31T22:22:37.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:37.482695+0000) 2022-01-31T22:22:38.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:37 smithi167 conmon[54076]: debug 2022-01-31T22:22:37.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:37.779130+0000) 2022-01-31T22:22:38.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[46715]: debug 2022-01-31T22:22:38.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.275839+0000) 2022-01-31T22:22:38.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[51620]: debug 2022-01-31T22:22:38.132+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.133981+0000) 2022-01-31T22:22:38.705 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[46715]: debug 2022-01-31T22:22:38.683+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.684996+0000) 2022-01-31T22:22:38.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[51620]: debug 2022-01-31T22:22:38.683+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.685304+0000) 2022-01-31T22:22:38.706 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[41853]: debug 2022-01-31T22:22:38.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.482896+0000) 2022-01-31T22:22:38.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[41853]: debug 2022-01-31T22:22:38.682+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.684386+0000) 2022-01-31T22:22:38.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:38 smithi167 conmon[49112]: debug 2022-01-31T22:22:38.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.609322+0000) 2022-01-31T22:22:38.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:38 smithi167 conmon[49112]: debug 2022-01-31T22:22:38.683+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.684978+0000) 2022-01-31T22:22:38.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:38 smithi167 conmon[54076]: debug 2022-01-31T22:22:38.683+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.685175+0000) 2022-01-31T22:22:38.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:38 smithi167 conmon[60316]: debug 2022-01-31T22:22:38.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.535308+0000) 2022-01-31T22:22:38.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:38 smithi167 conmon[60316]: debug 2022-01-31T22:22:38.683+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.685279+0000) 2022-01-31T22:22:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[35325]: debug 2022-01-31T22:22:38.704+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204566 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:38 smithi171 conmon[35325]: debug 2022-01-31T22:22:38.840+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:22:39.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:38 smithi167 conmon[54076]: debug 2022-01-31T22:22:38.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:38.779304+0000) 2022-01-31T22:22:39.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:39 smithi171 conmon[46715]: debug 2022-01-31T22:22:39.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:39.276047+0000) 2022-01-31T22:22:39.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:39 smithi171 conmon[51620]: debug 2022-01-31T22:22:39.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:39.134188+0000) 2022-01-31T22:22:39.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:39 smithi167 conmon[49112]: debug 2022-01-31T22:22:39.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:39.609479+0000) 2022-01-31T22:22:39.779 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:39 smithi167 conmon[60316]: debug 2022-01-31T22:22:39.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:39.535472+0000) 2022-01-31T22:22:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:39 smithi171 conmon[41853]: debug 2022-01-31T22:22:39.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:39.483054+0000) 2022-01-31T22:22:40.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:39 smithi167 conmon[54076]: debug 2022-01-31T22:22:39.777+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:39.779478+0000) 2022-01-31T22:22:40.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:40 smithi171 conmon[46715]: debug 2022-01-31T22:22:40.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:40.276173+0000) 2022-01-31T22:22:40.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:40 smithi171 conmon[51620]: debug 2022-01-31T22:22:40.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:40.134366+0000) 2022-01-31T22:22:40.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:40 smithi167 conmon[49112]: debug 2022-01-31T22:22:40.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:40.609653+0000) 2022-01-31T22:22:40.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:40 smithi167 conmon[60316]: debug 2022-01-31T22:22:40.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:40.535623+0000) 2022-01-31T22:22:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:40 smithi171 conmon[41853]: debug 2022-01-31T22:22:40.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:40.483195+0000) 2022-01-31T22:22:41.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:40 smithi167 conmon[54076]: debug 2022-01-31T22:22:40.778+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:40.779598+0000) 2022-01-31T22:22:41.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:41 smithi171 conmon[46715]: debug 2022-01-31T22:22:41.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:41.276360+0000) 2022-01-31T22:22:41.441 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:41 smithi171 conmon[51620]: debug 2022-01-31T22:22:41.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:41.134562+0000) 2022-01-31T22:22:41.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:41 smithi167 conmon[49112]: debug 2022-01-31T22:22:41.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:41.609798+0000) 2022-01-31T22:22:41.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:41 smithi167 conmon[60316]: debug 2022-01-31T22:22:41.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:41.535813+0000) 2022-01-31T22:22:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:41 smithi171 conmon[41853]: debug 2022-01-31T22:22:41.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:41.483353+0000) 2022-01-31T22:22:42.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:41 smithi167 conmon[54076]: debug 2022-01-31T22:22:41.779+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:41.779783+0000) 2022-01-31T22:22:42.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:42 smithi171 conmon[46715]: debug 2022-01-31T22:22:42.275+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:42.276550+0000) 2022-01-31T22:22:42.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:42 smithi171 conmon[51620]: debug 2022-01-31T22:22:42.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:42.134744+0000) 2022-01-31T22:22:42.779 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:42 smithi167 conmon[49112]: debug 2022-01-31T22:22:42.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:42.609952+0000) 2022-01-31T22:22:42.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:42 smithi167 conmon[60316]: debug 2022-01-31T22:22:42.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:42.536017+0000) 2022-01-31T22:22:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:42 smithi171 conmon[41853]: debug 2022-01-31T22:22:42.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:42.483531+0000) 2022-01-31T22:22:43.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:42 smithi167 conmon[54076]: debug 2022-01-31T22:22:42.779+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:42.779938+0000) 2022-01-31T22:22:43.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[46715]: debug 2022-01-31T22:22:43.274+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.276749+0000) 2022-01-31T22:22:43.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[51620]: debug 2022-01-31T22:22:43.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.134895+0000) 2022-01-31T22:22:43.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[41853]: debug 2022-01-31T22:22:43.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.483731+0000) 2022-01-31T22:22:43.708 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[41853]: debug 2022-01-31T22:22:43.706+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.707984+0000) 2022-01-31T22:22:43.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:43 smithi167 conmon[54076]: debug 2022-01-31T22:22:43.708+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.710178+0000) 2022-01-31T22:22:43.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:43 smithi167 conmon[49112]: debug 2022-01-31T22:22:43.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.610121+0000) 2022-01-31T22:22:43.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:43 smithi167 conmon[49112]: debug 2022-01-31T22:22:43.707+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.709485+0000) 2022-01-31T22:22:43.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:43 smithi167 conmon[60316]: debug 2022-01-31T22:22:43.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.536178+0000) 2022-01-31T22:22:43.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:43 smithi167 conmon[60316]: debug 2022-01-31T22:22:43.707+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.709365+0000) 2022-01-31T22:22:44.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[35325]: debug 2022-01-31T22:22:43.728+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204678 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:44.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[46715]: debug 2022-01-31T22:22:43.707+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.708910+0000) 2022-01-31T22:22:44.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:43 smithi171 conmon[51620]: debug 2022-01-31T22:22:43.708+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.709928+0000) 2022-01-31T22:22:44.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:43 smithi167 conmon[54076]: debug 2022-01-31T22:22:43.779+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:43.780110+0000) 2022-01-31T22:22:44.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:44 smithi171 conmon[46715]: debug 2022-01-31T22:22:44.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:44.276964+0000) 2022-01-31T22:22:44.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:44 smithi171 conmon[51620]: debug 2022-01-31T22:22:44.133+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:44.135062+0000) 2022-01-31T22:22:44.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:44 smithi167 conmon[60316]: debug 2022-01-31T22:22:44.534+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:44.536385+0000) 2022-01-31T22:22:44.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:44 smithi167 conmon[49112]: debug 2022-01-31T22:22:44.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:44.610298+0000) 2022-01-31T22:22:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:44 smithi171 conmon[41853]: debug 2022-01-31T22:22:44.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:44.483963+0000) 2022-01-31T22:22:45.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:44 smithi167 conmon[54076]: debug 2022-01-31T22:22:44.779+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:44.780271+0000) 2022-01-31T22:22:45.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:45 smithi171 conmon[46715]: debug 2022-01-31T22:22:45.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:45.277111+0000) 2022-01-31T22:22:45.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:45 smithi171 conmon[51620]: debug 2022-01-31T22:22:45.134+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:45.135213+0000) 2022-01-31T22:22:45.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:45 smithi167 conmon[49112]: debug 2022-01-31T22:22:45.608+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:45.610427+0000) 2022-01-31T22:22:45.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:45 smithi167 conmon[60316]: debug 2022-01-31T22:22:45.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:45.536541+0000) 2022-01-31T22:22:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:45 smithi171 conmon[41853]: debug 2022-01-31T22:22:45.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:45.484108+0000) 2022-01-31T22:22:46.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:45 smithi167 conmon[54076]: debug 2022-01-31T22:22:45.779+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:45.780487+0000) 2022-01-31T22:22:46.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:46 smithi171 conmon[46715]: debug 2022-01-31T22:22:46.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:46.277288+0000) 2022-01-31T22:22:46.443 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:46 smithi171 conmon[51620]: debug 2022-01-31T22:22:46.134+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:46.135425+0000) 2022-01-31T22:22:46.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:46 smithi167 conmon[49112]: debug 2022-01-31T22:22:46.609+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:46.610628+0000) 2022-01-31T22:22:46.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:46 smithi167 conmon[60316]: debug 2022-01-31T22:22:46.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:46.536711+0000) 2022-01-31T22:22:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:46 smithi171 conmon[41853]: debug 2022-01-31T22:22:46.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:46.484300+0000) 2022-01-31T22:22:47.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:46 smithi167 conmon[54076]: debug 2022-01-31T22:22:46.779+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:46.780657+0000) 2022-01-31T22:22:47.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:47 smithi171 conmon[46715]: debug 2022-01-31T22:22:47.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:47.277452+0000) 2022-01-31T22:22:47.443 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:47 smithi171 conmon[51620]: debug 2022-01-31T22:22:47.134+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:47.135642+0000) 2022-01-31T22:22:47.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:47 smithi167 conmon[49112]: debug 2022-01-31T22:22:47.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:47.610827+0000) 2022-01-31T22:22:47.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:47 smithi167 conmon[60316]: debug 2022-01-31T22:22:47.535+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:47.536838+0000) 2022-01-31T22:22:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:47 smithi171 conmon[41853]: debug 2022-01-31T22:22:47.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:47.484404+0000) 2022-01-31T22:22:48.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:47 smithi167 conmon[54076]: debug 2022-01-31T22:22:47.780+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:47.780848+0000) 2022-01-31T22:22:48.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[46715]: debug 2022-01-31T22:22:48.276+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.277645+0000) 2022-01-31T22:22:48.443 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[51620]: debug 2022-01-31T22:22:48.135+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.135846+0000) 2022-01-31T22:22:48.732 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[41853]: debug 2022-01-31T22:22:48.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.484586+0000) 2022-01-31T22:22:48.780 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:48 smithi167 conmon[54076]: debug 2022-01-31T22:22:48.732+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.734164+0000) 2022-01-31T22:22:48.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:48 smithi167 conmon[49112]: debug 2022-01-31T22:22:48.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.611015+0000) 2022-01-31T22:22:48.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:48 smithi167 conmon[49112]: debug 2022-01-31T22:22:48.731+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.733379+0000) 2022-01-31T22:22:48.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:48 smithi167 conmon[60316]: debug 2022-01-31T22:22:48.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.537024+0000) 2022-01-31T22:22:48.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:48 smithi167 conmon[60316]: debug 2022-01-31T22:22:48.732+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.733724+0000) 2022-01-31T22:22:49.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[35325]: debug 2022-01-31T22:22:48.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204789 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:49.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[41853]: debug 2022-01-31T22:22:48.731+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.732586+0000) 2022-01-31T22:22:49.109 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[46715]: debug 2022-01-31T22:22:48.733+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.733906+0000) 2022-01-31T22:22:49.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:48 smithi171 conmon[51620]: debug 2022-01-31T22:22:48.732+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.732802+0000) 2022-01-31T22:22:49.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:48 smithi167 conmon[54076]: debug 2022-01-31T22:22:48.780+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:48.781053+0000) 2022-01-31T22:22:49.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:49 smithi171 conmon[51620]: debug 2022-01-31T22:22:49.135+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:49.136045+0000) 2022-01-31T22:22:49.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:49 smithi171 conmon[46715]: debug 2022-01-31T22:22:49.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:49.277850+0000) 2022-01-31T22:22:49.780 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:49 smithi167 conmon[49112]: debug 2022-01-31T22:22:49.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:49.611193+0000) 2022-01-31T22:22:49.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:49 smithi167 conmon[60316]: debug 2022-01-31T22:22:49.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:49.537211+0000) 2022-01-31T22:22:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:49 smithi171 conmon[41853]: debug 2022-01-31T22:22:49.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:49.484788+0000) 2022-01-31T22:22:50.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:49 smithi167 conmon[54076]: debug 2022-01-31T22:22:49.780+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:49.781231+0000) 2022-01-31T22:22:50.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:50 smithi171 conmon[51620]: debug 2022-01-31T22:22:50.135+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:50.136213+0000) 2022-01-31T22:22:50.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:50 smithi171 conmon[46715]: debug 2022-01-31T22:22:50.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:50.278005+0000) 2022-01-31T22:22:50.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:50 smithi167 conmon[49112]: debug 2022-01-31T22:22:50.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:50.611343+0000) 2022-01-31T22:22:50.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:50 smithi167 conmon[60316]: debug 2022-01-31T22:22:50.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:50.537399+0000) 2022-01-31T22:22:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:50 smithi171 conmon[41853]: debug 2022-01-31T22:22:50.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:50.484941+0000) 2022-01-31T22:22:51.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:50 smithi167 conmon[54076]: debug 2022-01-31T22:22:50.780+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:50.781425+0000) 2022-01-31T22:22:51.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:51 smithi171 conmon[46715]: debug 2022-01-31T22:22:51.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:51.278208+0000) 2022-01-31T22:22:51.443 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:51 smithi171 conmon[51620]: debug 2022-01-31T22:22:51.135+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:51.136420+0000) 2022-01-31T22:22:51.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:51 smithi167 conmon[49112]: debug 2022-01-31T22:22:51.610+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:51.611528+0000) 2022-01-31T22:22:51.781 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:51 smithi167 conmon[60316]: debug 2022-01-31T22:22:51.536+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:51.537548+0000) 2022-01-31T22:22:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:51 smithi171 conmon[41853]: debug 2022-01-31T22:22:51.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:51.485130+0000) 2022-01-31T22:22:52.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:51 smithi167 conmon[54076]: debug 2022-01-31T22:22:51.781+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:51.781574+0000) 2022-01-31T22:22:52.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:52 smithi171 conmon[46715]: debug 2022-01-31T22:22:52.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:52.278371+0000) 2022-01-31T22:22:52.444 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:52 smithi171 conmon[51620]: debug 2022-01-31T22:22:52.135+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:52.136611+0000) 2022-01-31T22:22:52.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:52 smithi167 conmon[49112]: debug 2022-01-31T22:22:52.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:52.611713+0000) 2022-01-31T22:22:52.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:52 smithi167 conmon[60316]: debug 2022-01-31T22:22:52.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:52.537724+0000) 2022-01-31T22:22:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:52 smithi171 conmon[41853]: debug 2022-01-31T22:22:52.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:52.485369+0000) 2022-01-31T22:22:53.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:52 smithi167 conmon[54076]: debug 2022-01-31T22:22:52.781+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:52.781729+0000) 2022-01-31T22:22:53.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[46715]: debug 2022-01-31T22:22:53.277+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.278530+0000) 2022-01-31T22:22:53.444 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[51620]: debug 2022-01-31T22:22:53.136+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.136804+0000) 2022-01-31T22:22:53.755 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[41853]: debug 2022-01-31T22:22:53.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.485612+0000) 2022-01-31T22:22:53.760 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:53 smithi167 conmon[54076]: debug 2022-01-31T22:22:53.756+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.756906+0000) 2022-01-31T22:22:53.760 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:53 smithi167 conmon[49112]: debug 2022-01-31T22:22:53.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.611894+0000) 2022-01-31T22:22:53.761 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:53 smithi167 conmon[49112]: debug 2022-01-31T22:22:53.755+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.755961+0000) 2022-01-31T22:22:53.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:53 smithi167 conmon[60316]: debug 2022-01-31T22:22:53.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.537916+0000) 2022-01-31T22:22:53.761 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:53 smithi167 conmon[60316]: debug 2022-01-31T22:22:53.755+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.756275+0000) 2022-01-31T22:22:54.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[41853]: debug 2022-01-31T22:22:53.756+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.757217+0000) 2022-01-31T22:22:54.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[46715]: debug 2022-01-31T22:22:53.755+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.756433+0000) 2022-01-31T22:22:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[35325]: debug 2022-01-31T22:22:53.792+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 204891 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[35325]: debug 2022-01-31T22:22:53.841+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:22:54.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:53 smithi171 conmon[51620]: debug 2022-01-31T22:22:53.755+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.756469+0000) 2022-01-31T22:22:54.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:53 smithi167 conmon[54076]: debug 2022-01-31T22:22:53.781+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:53.781848+0000) 2022-01-31T22:22:54.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:54 smithi171 conmon[46715]: debug 2022-01-31T22:22:54.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:54.278742+0000) 2022-01-31T22:22:54.444 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:54 smithi171 conmon[51620]: debug 2022-01-31T22:22:54.136+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:54.137023+0000) 2022-01-31T22:22:54.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:54 smithi167 conmon[49112]: debug 2022-01-31T22:22:54.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:54.612100+0000) 2022-01-31T22:22:54.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:54 smithi167 conmon[60316]: debug 2022-01-31T22:22:54.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:54.538074+0000) 2022-01-31T22:22:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:54 smithi171 conmon[41853]: debug 2022-01-31T22:22:54.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:54.485788+0000) 2022-01-31T22:22:55.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:54 smithi167 conmon[54076]: debug 2022-01-31T22:22:54.781+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:54.781982+0000) 2022-01-31T22:22:55.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:55 smithi171 conmon[46715]: debug 2022-01-31T22:22:55.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:55.278898+0000) 2022-01-31T22:22:55.444 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:55 smithi171 conmon[51620]: debug 2022-01-31T22:22:55.136+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:55.137184+0000) 2022-01-31T22:22:55.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:55 smithi167 conmon[49112]: debug 2022-01-31T22:22:55.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:55.612246+0000) 2022-01-31T22:22:55.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:55 smithi167 conmon[60316]: debug 2022-01-31T22:22:55.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:55.538228+0000) 2022-01-31T22:22:55.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:55 smithi171 conmon[41853]: debug 2022-01-31T22:22:55.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:55.485967+0000) 2022-01-31T22:22:56.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:55 smithi167 conmon[54076]: debug 2022-01-31T22:22:55.781+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:55.782186+0000) 2022-01-31T22:22:56.443 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:56 smithi171 conmon[46715]: debug 2022-01-31T22:22:56.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:56.279080+0000) 2022-01-31T22:22:56.444 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:56 smithi171 conmon[51620]: debug 2022-01-31T22:22:56.136+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:56.137367+0000) 2022-01-31T22:22:56.781 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:56 smithi167 conmon[49112]: debug 2022-01-31T22:22:56.611+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:56.612456+0000) 2022-01-31T22:22:56.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:56 smithi167 conmon[60316]: debug 2022-01-31T22:22:56.537+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:56.538418+0000) 2022-01-31T22:22:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:56 smithi171 conmon[41853]: debug 2022-01-31T22:22:56.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:56.486102+0000) 2022-01-31T22:22:57.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:56 smithi167 conmon[54076]: debug 2022-01-31T22:22:56.781+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:56.782364+0000) 2022-01-31T22:22:57.444 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:57 smithi171 conmon[46715]: debug 2022-01-31T22:22:57.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:57.279292+0000) 2022-01-31T22:22:57.444 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:57 smithi171 conmon[51620]: debug 2022-01-31T22:22:57.136+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:57.137550+0000) 2022-01-31T22:22:57.782 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:57 smithi167 conmon[49112]: debug 2022-01-31T22:22:57.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:57.612614+0000) 2022-01-31T22:22:57.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:57 smithi167 conmon[60316]: debug 2022-01-31T22:22:57.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:57.538644+0000) 2022-01-31T22:22:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:57 smithi171 conmon[41853]: debug 2022-01-31T22:22:57.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:57.486270+0000) 2022-01-31T22:22:58.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:57 smithi167 conmon[54076]: debug 2022-01-31T22:22:57.782+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:57.782574+0000) 2022-01-31T22:22:58.444 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[46715]: debug 2022-01-31T22:22:58.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.279480+0000) 2022-01-31T22:22:58.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[51620]: debug 2022-01-31T22:22:58.137+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.137704+0000) 2022-01-31T22:22:58.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:58 smithi167 conmon[60316]: debug 2022-01-31T22:22:58.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.538834+0000) 2022-01-31T22:22:58.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:58 smithi167 conmon[49112]: debug 2022-01-31T22:22:58.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.612802+0000) 2022-01-31T22:22:58.794 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[41853]: debug 2022-01-31T22:22:58.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.486473+0000) 2022-01-31T22:22:59.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[35325]: debug 2022-01-31T22:22:58.816+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205002 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:22:59.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[41853]: debug 2022-01-31T22:22:58.795+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.796627+0000) 2022-01-31T22:22:59.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[46715]: debug 2022-01-31T22:22:58.795+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.795847+0000) 2022-01-31T22:22:59.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:58 smithi171 conmon[51620]: debug 2022-01-31T22:22:58.794+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.795003+0000) 2022-01-31T22:22:59.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:58 smithi167 conmon[49112]: debug 2022-01-31T22:22:58.795+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.796292+0000) 2022-01-31T22:22:59.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:58 smithi167 conmon[60316]: debug 2022-01-31T22:22:58.796+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.797131+0000) 2022-01-31T22:22:59.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:58 smithi167 conmon[54076]: debug 2022-01-31T22:22:58.782+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.782758+0000) 2022-01-31T22:22:59.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:58 smithi167 conmon[54076]: debug 2022-01-31T22:22:58.796+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:58.796612+0000) 2022-01-31T22:22:59.444 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:22:59 smithi171 conmon[46715]: debug 2022-01-31T22:22:59.278+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:59.279666+0000) 2022-01-31T22:22:59.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:22:59 smithi171 conmon[51620]: debug 2022-01-31T22:22:59.137+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:59.137882+0000) 2022-01-31T22:22:59.782 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:22:59 smithi167 conmon[60316]: debug 2022-01-31T22:22:59.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:59.539023+0000) 2022-01-31T22:22:59.783 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:22:59 smithi167 conmon[49112]: debug 2022-01-31T22:22:59.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:59.612972+0000) 2022-01-31T22:22:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:22:59 smithi171 conmon[41853]: debug 2022-01-31T22:22:59.485+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:59.486682+0000) 2022-01-31T22:23:00.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:22:59 smithi167 conmon[54076]: debug 2022-01-31T22:22:59.782+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:22:59.782886+0000) 2022-01-31T22:23:00.444 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:00 smithi171 conmon[46715]: debug 2022-01-31T22:23:00.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:00.279851+0000) 2022-01-31T22:23:00.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:00 smithi171 conmon[51620]: debug 2022-01-31T22:23:00.137+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:00.138021+0000) 2022-01-31T22:23:00.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:00 smithi167 conmon[54076]: debug 2022-01-31T22:23:00.782+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:00.783084+0000) 2022-01-31T22:23:00.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:00 smithi167 conmon[60316]: debug 2022-01-31T22:23:00.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:00.539184+0000) 2022-01-31T22:23:00.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:00 smithi167 conmon[49112]: debug 2022-01-31T22:23:00.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:00.613130+0000) 2022-01-31T22:23:00.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:00 smithi171 conmon[41853]: debug 2022-01-31T22:23:00.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:00.486837+0000) 2022-01-31T22:23:01.444 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:01 smithi171 conmon[46715]: debug 2022-01-31T22:23:01.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:01.280026+0000) 2022-01-31T22:23:01.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:01 smithi171 conmon[51620]: debug 2022-01-31T22:23:01.137+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:01.138207+0000) 2022-01-31T22:23:01.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:01 smithi167 conmon[54076]: debug 2022-01-31T22:23:01.782+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:01.783286+0000) 2022-01-31T22:23:02.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:01 smithi167 conmon[60316]: debug 2022-01-31T22:23:01.538+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:01.539367+0000) 2022-01-31T22:23:02.035 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:01 smithi167 conmon[49112]: debug 2022-01-31T22:23:01.612+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:01.613333+0000) 2022-01-31T22:23:02.036 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:01 smithi171 conmon[41853]: debug 2022-01-31T22:23:01.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:01.487031+0000) 2022-01-31T22:23:02.444 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:02 smithi171 conmon[46715]: debug 2022-01-31T22:23:02.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:02.280181+0000) 2022-01-31T22:23:02.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:02 smithi171 conmon[51620]: debug 2022-01-31T22:23:02.137+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:02.138387+0000) 2022-01-31T22:23:02.832 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:02 smithi167 conmon[54076]: debug 2022-01-31T22:23:02.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:02.783496+0000) 2022-01-31T22:23:02.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:02 smithi167 conmon[60316]: debug 2022-01-31T22:23:02.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:02.539542+0000) 2022-01-31T22:23:02.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:02 smithi167 conmon[49112]: debug 2022-01-31T22:23:02.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:02.613493+0000) 2022-01-31T22:23:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:02 smithi171 conmon[41853]: debug 2022-01-31T22:23:02.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:02.487220+0000) 2022-01-31T22:23:03.445 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[46715]: debug 2022-01-31T22:23:03.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.280358+0000) 2022-01-31T22:23:03.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[51620]: debug 2022-01-31T22:23:03.137+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.138662+0000) 2022-01-31T22:23:03.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:03 smithi167 conmon[49112]: debug 2022-01-31T22:23:03.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.613673+0000) 2022-01-31T22:23:03.820 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[41853]: debug 2022-01-31T22:23:03.486+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.487410+0000) 2022-01-31T22:23:03.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:03 smithi167 conmon[54076]: debug 2022-01-31T22:23:03.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.783695+0000) 2022-01-31T22:23:03.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:03 smithi167 conmon[60316]: debug 2022-01-31T22:23:03.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.539725+0000) 2022-01-31T22:23:04.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[35325]: debug 2022-01-31T22:23:03.840+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205114 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:04.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[41853]: debug 2022-01-31T22:23:03.819+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.820623+0000) 2022-01-31T22:23:04.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[46715]: debug 2022-01-31T22:23:03.819+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.819772+0000) 2022-01-31T22:23:04.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:03 smithi171 conmon[51620]: debug 2022-01-31T22:23:03.818+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.819385+0000) 2022-01-31T22:23:04.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:03 smithi167 conmon[49112]: debug 2022-01-31T22:23:03.818+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.819174+0000) 2022-01-31T22:23:04.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:03 smithi167 conmon[54076]: debug 2022-01-31T22:23:03.819+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.819776+0000) 2022-01-31T22:23:04.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:03 smithi167 conmon[60316]: debug 2022-01-31T22:23:03.819+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:03.820417+0000) 2022-01-31T22:23:04.445 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:04 smithi171 conmon[46715]: debug 2022-01-31T22:23:04.279+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:04.280526+0000) 2022-01-31T22:23:04.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:04 smithi171 conmon[51620]: debug 2022-01-31T22:23:04.138+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:04.138845+0000) 2022-01-31T22:23:04.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:04 smithi167 conmon[49112]: debug 2022-01-31T22:23:04.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:04.613773+0000) 2022-01-31T22:23:04.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:04 smithi167 conmon[54076]: debug 2022-01-31T22:23:04.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:04.783792+0000) 2022-01-31T22:23:04.834 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:04 smithi167 conmon[60316]: debug 2022-01-31T22:23:04.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:04.539903+0000) 2022-01-31T22:23:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:04 smithi171 conmon[41853]: debug 2022-01-31T22:23:04.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:04.487602+0000) 2022-01-31T22:23:05.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:05 smithi171 conmon[51620]: debug 2022-01-31T22:23:05.138+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:05.138973+0000) 2022-01-31T22:23:05.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:05 smithi171 conmon[46715]: debug 2022-01-31T22:23:05.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:05 smithi171 conmon[46715]: 2022-01-31T22:23:05.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:05.280679+0000) 2022-01-31T22:23:05.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:05 smithi167 conmon[49112]: debug 2022-01-31T22:23:05.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:05.613918+0000) 2022-01-31T22:23:05.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:05 smithi167 conmon[54076]: debug 2022-01-31T22:23:05.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:05.783890+0000) 2022-01-31T22:23:05.834 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:05 smithi167 conmon[60316]: debug 2022-01-31T22:23:05.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:05.540051+0000) 2022-01-31T22:23:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:05 smithi171 conmon[41853]: debug 2022-01-31T22:23:05.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:05.487782+0000) 2022-01-31T22:23:06.445 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:06 smithi171 conmon[46715]: debug 2022-01-31T22:23:06.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:06.280774+0000) 2022-01-31T22:23:06.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:06 smithi171 conmon[51620]: debug 2022-01-31T22:23:06.138+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:06.139206+0000) 2022-01-31T22:23:06.833 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:06 smithi167 conmon[49112]: debug 2022-01-31T22:23:06.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:06.614095+0000) 2022-01-31T22:23:06.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:06 smithi167 conmon[54076]: debug 2022-01-31T22:23:06.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:06.784065+0000) 2022-01-31T22:23:06.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:06 smithi167 conmon[60316]: debug 2022-01-31T22:23:06.539+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:06.540239+0000) 2022-01-31T22:23:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:06 smithi171 conmon[41853]: debug 2022-01-31T22:23:06.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:06.487949+0000) 2022-01-31T22:23:07.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:07 smithi171 conmon[46715]: debug 2022-01-31T22:23:07.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:07.280956+0000) 2022-01-31T22:23:07.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:07 smithi171 conmon[51620]: debug 2022-01-31T22:23:07.138+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:07.139400+0000) 2022-01-31T22:23:07.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:07 smithi167 conmon[49112]: debug 2022-01-31T22:23:07.613+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:07.614303+0000) 2022-01-31T22:23:07.834 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:07 smithi167 conmon[54076]: debug 2022-01-31T22:23:07.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:07.784268+0000) 2022-01-31T22:23:07.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:07 smithi167 conmon[60316]: debug 2022-01-31T22:23:07.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:07.540480+0000) 2022-01-31T22:23:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:07 smithi171 conmon[41853]: debug 2022-01-31T22:23:07.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:07.488139+0000) 2022-01-31T22:23:08.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[46715]: debug 2022-01-31T22:23:08.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.281133+0000) 2022-01-31T22:23:08.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[51620]: debug 2022-01-31T22:23:08.139+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.139602+0000) 2022-01-31T22:23:08.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:08 smithi167 conmon[49112]: debug 2022-01-31T22:23:08.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.614511+0000) 2022-01-31T22:23:08.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:08 smithi167 conmon[54076]: debug 2022-01-31T22:23:08.783+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.784437+0000) 2022-01-31T22:23:08.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:08 smithi167 conmon[60316]: debug 2022-01-31T22:23:08.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.540652+0000) 2022-01-31T22:23:08.843 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[41853]: debug 2022-01-31T22:23:08.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.488298+0000) 2022-01-31T22:23:09.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[41853]: debug 2022-01-31T22:23:08.844+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.845484+0000) 2022-01-31T22:23:09.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[46715]: debug 2022-01-31T22:23:08.844+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.844807+0000) 2022-01-31T22:23:09.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[51620]: debug 2022-01-31T22:23:08.843+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.844150+0000) 2022-01-31T22:23:09.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[35325]: debug 2022-01-31T22:23:08.844+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:23:09.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:08 smithi171 conmon[35325]: debug 2022-01-31T22:23:08.864+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205225 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:09.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:08 smithi167 conmon[49112]: debug 2022-01-31T22:23:08.843+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.844282+0000) 2022-01-31T22:23:09.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:08 smithi167 conmon[54076]: debug 2022-01-31T22:23:08.844+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.845367+0000) 2022-01-31T22:23:09.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:08 smithi167 conmon[60316]: debug 2022-01-31T22:23:08.843+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:08.844025+0000) 2022-01-31T22:23:09.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:09 smithi171 conmon[46715]: debug 2022-01-31T22:23:09.280+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:09.281306+0000) 2022-01-31T22:23:09.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:09 smithi171 conmon[51620]: debug 2022-01-31T22:23:09.139+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:09.139782+0000) 2022-01-31T22:23:09.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:09 smithi167 conmon[49112]: debug 2022-01-31T22:23:09.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:09.614635+0000) 2022-01-31T22:23:09.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:09 smithi167 conmon[54076]: debug 2022-01-31T22:23:09.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:09.784574+0000) 2022-01-31T22:23:09.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:09 smithi167 conmon[60316]: debug 2022-01-31T22:23:09.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:09.540819+0000) 2022-01-31T22:23:09.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:09 smithi171 conmon[41853]: debug 2022-01-31T22:23:09.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:09.488452+0000) 2022-01-31T22:23:10.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:10 smithi171 conmon[46715]: debug 2022-01-31T22:23:10.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:10.281459+0000) 2022-01-31T22:23:10.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:10 smithi171 conmon[51620]: debug 2022-01-31T22:23:10.139+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:10.139968+0000) 2022-01-31T22:23:10.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:10 smithi167 conmon[49112]: debug 2022-01-31T22:23:10.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:10.614716+0000) 2022-01-31T22:23:10.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:10 smithi167 conmon[54076]: debug 2022-01-31T22:23:10.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:10.784736+0000) 2022-01-31T22:23:10.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:10 smithi167 conmon[60316]: debug 2022-01-31T22:23:10.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:10.540974+0000) 2022-01-31T22:23:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:10 smithi171 conmon[41853]: debug 2022-01-31T22:23:10.487+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:10.488625+0000) 2022-01-31T22:23:11.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:11 smithi171 conmon[46715]: debug 2022-01-31T22:23:11.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:11.281646+0000) 2022-01-31T22:23:11.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:11 smithi171 conmon[51620]: debug 2022-01-31T22:23:11.139+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:11.140152+0000) 2022-01-31T22:23:11.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:11 smithi167 conmon[49112]: debug 2022-01-31T22:23:11.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:11.614834+0000) 2022-01-31T22:23:11.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:11 smithi167 conmon[54076]: debug 2022-01-31T22:23:11.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:11.784899+0000) 2022-01-31T22:23:11.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:11 smithi167 conmon[60316]: debug 2022-01-31T22:23:11.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:11.541169+0000) 2022-01-31T22:23:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:11 smithi171 conmon[41853]: debug 2022-01-31T22:23:11.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:11.488828+0000) 2022-01-31T22:23:12.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:12 smithi171 conmon[46715]: debug 2022-01-31T22:23:12.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:12.281831+0000) 2022-01-31T22:23:12.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:12 smithi171 conmon[51620]: debug 2022-01-31T22:23:12.139+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:12.140358+0000) 2022-01-31T22:23:12.834 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:12 smithi167 conmon[49112]: debug 2022-01-31T22:23:12.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:12.615047+0000) 2022-01-31T22:23:12.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:12 smithi167 conmon[54076]: debug 2022-01-31T22:23:12.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:12.785105+0000) 2022-01-31T22:23:12.835 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:12 smithi167 conmon[60316]: debug 2022-01-31T22:23:12.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:12.541297+0000) 2022-01-31T22:23:12.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:12 smithi171 conmon[41853]: debug 2022-01-31T22:23:12.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:12.488952+0000) 2022-01-31T22:23:13.446 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[46715]: debug 2022-01-31T22:23:13.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.281997+0000) 2022-01-31T22:23:13.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[51620]: debug 2022-01-31T22:23:13.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.140529+0000) 2022-01-31T22:23:13.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:13 smithi167 conmon[49112]: debug 2022-01-31T22:23:13.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.615226+0000) 2022-01-31T22:23:13.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:13 smithi167 conmon[54076]: debug 2022-01-31T22:23:13.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.785316+0000) 2022-01-31T22:23:13.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:13 smithi167 conmon[60316]: debug 2022-01-31T22:23:13.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.541496+0000) 2022-01-31T22:23:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[41853]: debug 2022-01-31T22:23:13.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.489081+0000) 2022-01-31T22:23:14.140 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[35325]: debug 2022-01-31T22:23:13.888+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205337 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:14.141 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[41853]: debug 2022-01-31T22:23:13.867+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.868432+0000) 2022-01-31T22:23:14.141 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[46715]: debug 2022-01-31T22:23:13.868+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.869222+0000) 2022-01-31T22:23:14.142 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:13 smithi171 conmon[51620]: debug 2022-01-31T22:23:13.866+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.867243+0000) 2022-01-31T22:23:14.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:13 smithi167 conmon[49112]: debug 2022-01-31T22:23:13.867+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.868026+0000) 2022-01-31T22:23:14.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:13 smithi167 conmon[54076]: debug 2022-01-31T22:23:13.868+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.869309+0000) 2022-01-31T22:23:14.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:13 smithi167 conmon[60316]: debug 2022-01-31T22:23:13.868+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:13.869251+0000) 2022-01-31T22:23:14.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:14 smithi171 conmon[46715]: debug 2022-01-31T22:23:14.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:14.282198+0000) 2022-01-31T22:23:14.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:14 smithi171 conmon[51620]: debug 2022-01-31T22:23:14.139+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:14.140675+0000) 2022-01-31T22:23:14.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:14 smithi167 conmon[49112]: debug 2022-01-31T22:23:14.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:14.615381+0000) 2022-01-31T22:23:14.835 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:14 smithi167 conmon[54076]: debug 2022-01-31T22:23:14.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:14.785468+0000) 2022-01-31T22:23:14.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:14 smithi167 conmon[60316]: debug 2022-01-31T22:23:14.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:14.541720+0000) 2022-01-31T22:23:14.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:14 smithi171 conmon[41853]: debug 2022-01-31T22:23:14.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:14.489286+0000) 2022-01-31T22:23:15.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:15 smithi171 conmon[46715]: debug 2022-01-31T22:23:15.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:15.282355+0000) 2022-01-31T22:23:15.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:15 smithi171 conmon[51620]: debug 2022-01-31T22:23:15.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:15.140811+0000) 2022-01-31T22:23:15.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:15 smithi167 conmon[49112]: debug 2022-01-31T22:23:15.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:15.615530+0000) 2022-01-31T22:23:15.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:15 smithi167 conmon[54076]: debug 2022-01-31T22:23:15.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:15.785620+0000) 2022-01-31T22:23:15.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:15 smithi167 conmon[60316]: debug 2022-01-31T22:23:15.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:15.541818+0000) 2022-01-31T22:23:15.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:15 smithi171 conmon[41853]: debug 2022-01-31T22:23:15.488+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:15.489414+0000) 2022-01-31T22:23:16.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:16 smithi171 conmon[46715]: debug 2022-01-31T22:23:16.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:16.282541+0000) 2022-01-31T22:23:16.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:16 smithi171 conmon[51620]: debug 2022-01-31T22:23:16.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:16.140990+0000) 2022-01-31T22:23:16.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:16 smithi167 conmon[49112]: debug 2022-01-31T22:23:16.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:16.615653+0000) 2022-01-31T22:23:16.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:16 smithi167 conmon[54076]: debug 2022-01-31T22:23:16.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:16.785776+0000) 2022-01-31T22:23:16.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:16 smithi167 conmon[60316]: debug 2022-01-31T22:23:16.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:16.542012+0000) 2022-01-31T22:23:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:16 smithi171 conmon[41853]: debug 2022-01-31T22:23:16.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:16.489578+0000) 2022-01-31T22:23:17.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:17 smithi171 conmon[46715]: debug 2022-01-31T22:23:17.281+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:17.282730+0000) 2022-01-31T22:23:17.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:17 smithi171 conmon[51620]: debug 2022-01-31T22:23:17.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:17.141199+0000) 2022-01-31T22:23:17.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:17 smithi167 conmon[49112]: debug 2022-01-31T22:23:17.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:17.615814+0000) 2022-01-31T22:23:17.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:17 smithi167 conmon[54076]: debug 2022-01-31T22:23:17.784+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:17.785955+0000) 2022-01-31T22:23:17.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:17 smithi167 conmon[60316]: debug 2022-01-31T22:23:17.540+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:17.542217+0000) 2022-01-31T22:23:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:17 smithi171 conmon[41853]: debug 2022-01-31T22:23:17.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:17.489786+0000) 2022-01-31T22:23:18.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[46715]: debug 2022-01-31T22:23:18.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.282939+0000) 2022-01-31T22:23:18.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[51620]: debug 2022-01-31T22:23:18.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.141368+0000) 2022-01-31T22:23:18.835 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:18 smithi167 conmon[49112]: debug 2022-01-31T22:23:18.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.615984+0000) 2022-01-31T22:23:18.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:18 smithi167 conmon[54076]: debug 2022-01-31T22:23:18.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.786180+0000) 2022-01-31T22:23:18.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:18 smithi167 conmon[60316]: debug 2022-01-31T22:23:18.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.542382+0000) 2022-01-31T22:23:18.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[41853]: debug 2022-01-31T22:23:18.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.489987+0000) 2022-01-31T22:23:19.141 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[35325]: debug 2022-01-31T22:23:18.912+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205448 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:19.142 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[41853]: debug 2022-01-31T22:23:18.889+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.890549+0000) 2022-01-31T22:23:19.142 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[46715]: debug 2022-01-31T22:23:18.891+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.892271+0000) 2022-01-31T22:23:19.143 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:18 smithi171 conmon[51620]: debug 2022-01-31T22:23:18.891+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.892098+0000) 2022-01-31T22:23:19.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:18 smithi167 conmon[49112]: debug 2022-01-31T22:23:18.890+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.892100+0000) 2022-01-31T22:23:19.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:18 smithi167 conmon[54076]: debug 2022-01-31T22:23:18.889+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.891423+0000) 2022-01-31T22:23:19.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:18 smithi167 conmon[60316]: debug 2022-01-31T22:23:18.890+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:18.891564+0000) 2022-01-31T22:23:19.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:19 smithi171 conmon[46715]: debug 2022-01-31T22:23:19.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:19.283088+0000) 2022-01-31T22:23:19.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:19 smithi171 conmon[51620]: debug 2022-01-31T22:23:19.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:19.141542+0000) 2022-01-31T22:23:19.836 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:19 smithi167 conmon[60316]: debug 2022-01-31T22:23:19.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:19.542585+0000) 2022-01-31T22:23:19.836 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:19 smithi167 conmon[49112]: debug 2022-01-31T22:23:19.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:19.616154+0000) 2022-01-31T22:23:19.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:19 smithi167 conmon[54076]: debug 2022-01-31T22:23:19.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:19.786394+0000) 2022-01-31T22:23:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:19 smithi171 conmon[41853]: debug 2022-01-31T22:23:19.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:19.490170+0000) 2022-01-31T22:23:20.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:20 smithi171 conmon[46715]: debug 2022-01-31T22:23:20.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:20.283243+0000) 2022-01-31T22:23:20.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:20 smithi171 conmon[51620]: debug 2022-01-31T22:23:20.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:20.141713+0000) 2022-01-31T22:23:20.836 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:20 smithi167 conmon[49112]: debug 2022-01-31T22:23:20.614+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:20.616337+0000) 2022-01-31T22:23:20.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:20 smithi167 conmon[54076]: debug 2022-01-31T22:23:20.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:20.786553+0000) 2022-01-31T22:23:20.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:20 smithi167 conmon[60316]: debug 2022-01-31T22:23:20.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:20.542687+0000) 2022-01-31T22:23:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:20 smithi171 conmon[41853]: debug 2022-01-31T22:23:20.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:20.490314+0000) 2022-01-31T22:23:21.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:21 smithi171 conmon[46715]: debug 2022-01-31T22:23:21.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:21.283431+0000) 2022-01-31T22:23:21.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:21 smithi171 conmon[51620]: debug 2022-01-31T22:23:21.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:21.141932+0000) 2022-01-31T22:23:21.836 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:21 smithi167 conmon[49112]: debug 2022-01-31T22:23:21.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:21.616558+0000) 2022-01-31T22:23:21.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:21 smithi167 conmon[54076]: debug 2022-01-31T22:23:21.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:21.786724+0000) 2022-01-31T22:23:21.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:21 smithi167 conmon[60316]: debug 2022-01-31T22:23:21.541+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:21.542898+0000) 2022-01-31T22:23:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:21 smithi171 conmon[41853]: debug 2022-01-31T22:23:21.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:21.490501+0000) 2022-01-31T22:23:22.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:22 smithi171 conmon[46715]: debug 2022-01-31T22:23:22.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:22.283596+0000) 2022-01-31T22:23:22.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:22 smithi171 conmon[51620]: debug 2022-01-31T22:23:22.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:22.142090+0000) 2022-01-31T22:23:22.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:22 smithi167 conmon[54076]: debug 2022-01-31T22:23:22.785+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:22.786905+0000) 2022-01-31T22:23:22.837 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:22 smithi167 conmon[49112]: debug 2022-01-31T22:23:22.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:22.616721+0000) 2022-01-31T22:23:22.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:22 smithi167 conmon[60316]: debug 2022-01-31T22:23:22.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:22.543093+0000) 2022-01-31T22:23:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:22 smithi171 conmon[41853]: debug 2022-01-31T22:23:22.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:22.490714+0000) 2022-01-31T22:23:23.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[46715]: debug 2022-01-31T22:23:23.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.283784+0000) 2022-01-31T22:23:23.449 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[51620]: debug 2022-01-31T22:23:23.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.142285+0000) 2022-01-31T22:23:23.836 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:23 smithi167 conmon[49112]: debug 2022-01-31T22:23:23.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.616873+0000) 2022-01-31T22:23:23.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:23 smithi167 conmon[54076]: debug 2022-01-31T22:23:23.786+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.787068+0000) 2022-01-31T22:23:23.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:23 smithi167 conmon[60316]: debug 2022-01-31T22:23:23.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.543297+0000) 2022-01-31T22:23:23.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[41853]: debug 2022-01-31T22:23:23.489+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.490883+0000) 2022-01-31T22:23:24.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[46715]: debug 2022-01-31T22:23:23.913+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.915534+0000) 2022-01-31T22:23:24.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[51620]: debug 2022-01-31T22:23:23.913+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.915082+0000) 2022-01-31T22:23:24.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[35325]: debug 2022-01-31T22:23:23.844+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:23:24.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[35325]: debug 2022-01-31T22:23:23.935+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205559 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:24.109 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:23 smithi171 conmon[41853]: debug 2022-01-31T22:23:23.914+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.916262+0000) 2022-01-31T22:23:24.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:23 smithi167 conmon[54076]: debug 2022-01-31T22:23:23.914+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.915567+0000) 2022-01-31T22:23:24.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:23 smithi167 conmon[60316]: debug 2022-01-31T22:23:23.914+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.915900+0000) 2022-01-31T22:23:24.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:23 smithi167 conmon[49112]: debug 2022-01-31T22:23:23.914+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:23.915979+0000) 2022-01-31T22:23:24.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:24 smithi171 conmon[46715]: debug 2022-01-31T22:23:24.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:24.283925+0000) 2022-01-31T22:23:24.449 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:24 smithi171 conmon[51620]: debug 2022-01-31T22:23:24.140+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:24.142458+0000) 2022-01-31T22:23:24.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:24 smithi167 conmon[54076]: debug 2022-01-31T22:23:24.786+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:24.787249+0000) 2022-01-31T22:23:24.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:24 smithi167 conmon[60316]: debug 2022-01-31T22:23:24.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:24.543518+0000) 2022-01-31T22:23:24.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:24 smithi167 conmon[49112]: debug 2022-01-31T22:23:24.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:24.617025+0000) 2022-01-31T22:23:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:24 smithi171 conmon[41853]: debug 2022-01-31T22:23:24.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:24.491095+0000) 2022-01-31T22:23:25.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:25 smithi171 conmon[46715]: debug 2022-01-31T22:23:25.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:25.284086+0000) 2022-01-31T22:23:25.449 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:25 smithi171 conmon[51620]: debug 2022-01-31T22:23:25.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:25.142640+0000) 2022-01-31T22:23:25.837 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:25 smithi167 conmon[49112]: debug 2022-01-31T22:23:25.615+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:25.617166+0000) 2022-01-31T22:23:25.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:25 smithi167 conmon[54076]: debug 2022-01-31T22:23:25.786+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:25.787385+0000) 2022-01-31T22:23:25.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:25 smithi167 conmon[60316]: debug 2022-01-31T22:23:25.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:25.543611+0000) 2022-01-31T22:23:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:25 smithi171 conmon[41853]: debug 2022-01-31T22:23:25.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:25.491255+0000) 2022-01-31T22:23:26.449 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:26 smithi171 conmon[46715]: debug 2022-01-31T22:23:26.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:26.284274+0000) 2022-01-31T22:23:26.449 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:26 smithi171 conmon[51620]: debug 2022-01-31T22:23:26.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:26.142849+0000) 2022-01-31T22:23:26.837 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:26 smithi167 conmon[49112]: debug 2022-01-31T22:23:26.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:26.617318+0000) 2022-01-31T22:23:26.837 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:26 smithi167 conmon[54076]: debug 2022-01-31T22:23:26.786+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:26.787570+0000) 2022-01-31T22:23:26.838 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:26 smithi167 conmon[60316]: debug 2022-01-31T22:23:26.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:26.543747+0000) 2022-01-31T22:23:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:26 smithi171 conmon[41853]: debug 2022-01-31T22:23:26.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:26.491451+0000) 2022-01-31T22:23:27.449 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:27 smithi171 conmon[46715]: debug 2022-01-31T22:23:27.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:27.284503+0000) 2022-01-31T22:23:27.449 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:27 smithi171 conmon[51620]: debug 2022-01-31T22:23:27.142+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:27.143055+0000) 2022-01-31T22:23:27.837 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:27 smithi167 conmon[49112]: debug 2022-01-31T22:23:27.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:27.617520+0000) 2022-01-31T22:23:27.838 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:27 smithi167 conmon[54076]: debug 2022-01-31T22:23:27.786+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:27.787727+0000) 2022-01-31T22:23:27.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:27 smithi167 conmon[60316]: debug 2022-01-31T22:23:27.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:27.543941+0000) 2022-01-31T22:23:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:27 smithi171 conmon[41853]: debug 2022-01-31T22:23:27.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:27.491603+0000) 2022-01-31T22:23:28.449 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[46715]: debug 2022-01-31T22:23:28.282+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.284728+0000) 2022-01-31T22:23:28.450 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[51620]: debug 2022-01-31T22:23:28.142+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.143229+0000) 2022-01-31T22:23:28.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:28 smithi167 conmon[49112]: debug 2022-01-31T22:23:28.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.617728+0000) 2022-01-31T22:23:28.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:28 smithi167 conmon[54076]: debug 2022-01-31T22:23:28.786+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.787871+0000) 2022-01-31T22:23:28.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:28 smithi167 conmon[60316]: debug 2022-01-31T22:23:28.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.544079+0000) 2022-01-31T22:23:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[41853]: debug 2022-01-31T22:23:28.490+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.491788+0000) 2022-01-31T22:23:29.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:28 smithi167 conmon[49112]: debug 2022-01-31T22:23:28.938+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.940278+0000) 2022-01-31T22:23:29.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:28 smithi167 conmon[54076]: debug 2022-01-31T22:23:28.938+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.940129+0000) 2022-01-31T22:23:29.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:28 smithi167 conmon[60316]: debug 2022-01-31T22:23:28.939+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.940678+0000) 2022-01-31T22:23:29.246 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[35325]: debug 2022-01-31T22:23:28.959+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205670 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:29.247 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[41853]: debug 2022-01-31T22:23:28.938+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.940403+0000) 2022-01-31T22:23:29.248 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[46715]: debug 2022-01-31T22:23:28.939+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.940929+0000) 2022-01-31T22:23:29.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:28 smithi171 conmon[51620]: debug 2022-01-31T22:23:28.937+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:28.939628+0000) 2022-01-31T22:23:29.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:29 smithi171 conmon[51620]: debug 2022-01-31T22:23:29.141+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:29.143439+0000) 2022-01-31T22:23:29.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:29 smithi171 conmon[46715]: debug 2022-01-31T22:23:29.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:29.284820+0000) 2022-01-31T22:23:29.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:29 smithi171 conmon[41853]: debug 2022-01-31T22:23:29.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:29.491995+0000) 2022-01-31T22:23:29.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:29 smithi167 conmon[49112]: debug 2022-01-31T22:23:29.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:29.617953+0000) 2022-01-31T22:23:29.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:29 smithi167 conmon[54076]: debug 2022-01-31T22:23:29.787+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:29.788057+0000) 2022-01-31T22:23:29.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:29 smithi167 conmon[60316]: debug 2022-01-31T22:23:29.542+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:29.544291+0000) 2022-01-31T22:23:30.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:30 smithi171 conmon[46715]: debug 2022-01-31T22:23:30.283+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:30.284918+0000) 2022-01-31T22:23:30.451 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:30 smithi171 conmon[51620]: debug 2022-01-31T22:23:30.142+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:30.143619+0000) 2022-01-31T22:23:30.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:30 smithi167 conmon[49112]: debug 2022-01-31T22:23:30.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:30.618077+0000) 2022-01-31T22:23:30.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:30 smithi167 conmon[54076]: debug 2022-01-31T22:23:30.787+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:30.788193+0000) 2022-01-31T22:23:30.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:30 smithi167 conmon[60316]: debug 2022-01-31T22:23:30.543+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:30.544427+0000) 2022-01-31T22:23:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:30 smithi171 conmon[41853]: debug 2022-01-31T22:23:30.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:30.492141+0000) 2022-01-31T22:23:31.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:31 smithi171 conmon[46715]: debug 2022-01-31T22:23:31.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:31.285037+0000) 2022-01-31T22:23:31.450 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:31 smithi171 conmon[51620]: debug 2022-01-31T22:23:31.143+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:31.143854+0000) 2022-01-31T22:23:31.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:31 smithi167 conmon[49112]: debug 2022-01-31T22:23:31.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:31.618224+0000) 2022-01-31T22:23:31.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:31 smithi167 conmon[54076]: debug 2022-01-31T22:23:31.787+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:31.788350+0000) 2022-01-31T22:23:31.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:31 smithi167 conmon[60316]: debug 2022-01-31T22:23:31.543+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:31.544622+0000) 2022-01-31T22:23:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:31 smithi171 conmon[41853]: debug 2022-01-31T22:23:31.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:31.492324+0000) 2022-01-31T22:23:32.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:32 smithi171 conmon[46715]: debug 2022-01-31T22:23:32.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:32.285212+0000) 2022-01-31T22:23:32.645 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:32 smithi171 conmon[51620]: debug 2022-01-31T22:23:32.143+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:32.144077+0000) 2022-01-31T22:23:32.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:32 smithi167 conmon[49112]: debug 2022-01-31T22:23:32.616+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:32.618406+0000) 2022-01-31T22:23:32.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:32 smithi167 conmon[54076]: debug 2022-01-31T22:23:32.787+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:32.788555+0000) 2022-01-31T22:23:32.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:32 smithi167 conmon[60316]: debug 2022-01-31T22:23:32.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:32.544814+0000) 2022-01-31T22:23:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:32 smithi171 conmon[41853]: debug 2022-01-31T22:23:32.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:32.492504+0000) 2022-01-31T22:23:33.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[46715]: debug 2022-01-31T22:23:33.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.285330+0000) 2022-01-31T22:23:33.451 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[51620]: debug 2022-01-31T22:23:33.143+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.144209+0000) 2022-01-31T22:23:33.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:33 smithi167 conmon[49112]: debug 2022-01-31T22:23:33.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.618535+0000) 2022-01-31T22:23:33.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:33 smithi167 conmon[54076]: debug 2022-01-31T22:23:33.788+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.788707+0000) 2022-01-31T22:23:33.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:33 smithi167 conmon[60316]: debug 2022-01-31T22:23:33.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.544974+0000) 2022-01-31T22:23:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[41853]: debug 2022-01-31T22:23:33.491+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.492658+0000) 2022-01-31T22:23:34.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:33 smithi167 conmon[49112]: debug 2022-01-31T22:23:33.963+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.964549+0000) 2022-01-31T22:23:34.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:33 smithi167 conmon[54076]: debug 2022-01-31T22:23:33.962+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.964354+0000) 2022-01-31T22:23:34.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:33 smithi167 conmon[60316]: debug 2022-01-31T22:23:33.962+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.964405+0000) 2022-01-31T22:23:34.247 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[35325]: debug 2022-01-31T22:23:33.983+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205781 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:34.248 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[41853]: debug 2022-01-31T22:23:33.964+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.964965+0000) 2022-01-31T22:23:34.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[46715]: debug 2022-01-31T22:23:33.963+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.964165+0000) 2022-01-31T22:23:34.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:33 smithi171 conmon[51620]: debug 2022-01-31T22:23:33.962+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:33.963687+0000) 2022-01-31T22:23:34.250 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:34 smithi171 conmon[51620]: debug 2022-01-31T22:23:34.143+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:34.144366+0000) 2022-01-31T22:23:34.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:34 smithi171 conmon[46715]: debug 2022-01-31T22:23:34.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:34.285477+0000) 2022-01-31T22:23:34.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:34 smithi171 conmon[41853]: debug 2022-01-31T22:23:34.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:34.492842+0000) 2022-01-31T22:23:34.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:34 smithi167 conmon[49112]: debug 2022-01-31T22:23:34.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:34.618668+0000) 2022-01-31T22:23:34.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:34 smithi167 conmon[54076]: debug 2022-01-31T22:23:34.788+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:34.788912+0000) 2022-01-31T22:23:34.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:34 smithi167 conmon[60316]: debug 2022-01-31T22:23:34.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:34.545183+0000) 2022-01-31T22:23:35.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:35 smithi171 conmon[46715]: debug 2022-01-31T22:23:35.284+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:35.285634+0000) 2022-01-31T22:23:35.451 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:35 smithi171 conmon[51620]: debug 2022-01-31T22:23:35.143+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:35.144527+0000) 2022-01-31T22:23:35.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:35 smithi167 conmon[49112]: debug 2022-01-31T22:23:35.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:35.618808+0000) 2022-01-31T22:23:35.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:35 smithi167 conmon[54076]: debug 2022-01-31T22:23:35.788+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:35.789066+0000) 2022-01-31T22:23:35.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:35 smithi167 conmon[60316]: debug 2022-01-31T22:23:35.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:35.545334+0000) 2022-01-31T22:23:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:35 smithi171 conmon[41853]: debug 2022-01-31T22:23:35.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:35.493009+0000) 2022-01-31T22:23:36.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:36 smithi171 conmon[46715]: debug 2022-01-31T22:23:36.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:36.285820+0000) 2022-01-31T22:23:36.451 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:36 smithi171 conmon[51620]: debug 2022-01-31T22:23:36.143+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:36.144726+0000) 2022-01-31T22:23:36.770 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:36 smithi171 conmon[41853]: debug 2022-01-31T22:23:36.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:36.493189+0000) 2022-01-31T22:23:36.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:36 smithi167 conmon[49112]: debug 2022-01-31T22:23:36.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:36.619013+0000) 2022-01-31T22:23:36.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:36 smithi167 conmon[54076]: debug 2022-01-31T22:23:36.787+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:36.789264+0000) 2022-01-31T22:23:36.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:36 smithi167 conmon[60316]: debug 2022-01-31T22:23:36.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:36.545521+0000) 2022-01-31T22:23:37.452 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:37 smithi171 conmon[46715]: debug 2022-01-31T22:23:37.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:37.286004+0000) 2022-01-31T22:23:37.454 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:37 smithi171 conmon[51620]: debug 2022-01-31T22:23:37.144+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:37.144948+0000) 2022-01-31T22:23:37.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:37 smithi167 conmon[49112]: debug 2022-01-31T22:23:37.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:37.619197+0000) 2022-01-31T22:23:37.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:37 smithi167 conmon[54076]: debug 2022-01-31T22:23:37.787+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:37.789468+0000) 2022-01-31T22:23:37.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:37 smithi167 conmon[60316]: debug 2022-01-31T22:23:37.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:37.545732+0000) 2022-01-31T22:23:37.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:37 smithi171 conmon[41853]: debug 2022-01-31T22:23:37.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:37.493359+0000) 2022-01-31T22:23:38.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[46715]: debug 2022-01-31T22:23:38.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.286199+0000) 2022-01-31T22:23:38.452 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[51620]: debug 2022-01-31T22:23:38.144+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.145218+0000) 2022-01-31T22:23:38.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:38 smithi167 conmon[49112]: debug 2022-01-31T22:23:38.617+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.619416+0000) 2022-01-31T22:23:38.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:38 smithi167 conmon[54076]: debug 2022-01-31T22:23:38.788+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.789752+0000) 2022-01-31T22:23:38.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:38 smithi167 conmon[60316]: debug 2022-01-31T22:23:38.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.546025+0000) 2022-01-31T22:23:38.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[41853]: debug 2022-01-31T22:23:38.492+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.493641+0000) 2022-01-31T22:23:39.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[41853]: debug 2022-01-31T22:23:38.986+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.986957+0000) 2022-01-31T22:23:39.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[46715]: debug 2022-01-31T22:23:38.985+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.986545+0000) 2022-01-31T22:23:39.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[51620]: debug 2022-01-31T22:23:38.985+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.986675+0000) 2022-01-31T22:23:39.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:38 smithi171 conmon[35325]: debug 2022-01-31T22:23:38.845+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:23:39.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:39 smithi171 conmon[35325]: debug 2022-01-31T22:23:39.007+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 205892 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:39.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:38 smithi167 conmon[49112]: debug 2022-01-31T22:23:38.984+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.986389+0000) 2022-01-31T22:23:39.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:38 smithi167 conmon[54076]: debug 2022-01-31T22:23:38.986+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.987743+0000) 2022-01-31T22:23:39.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:38 smithi167 conmon[60316]: debug 2022-01-31T22:23:38.985+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:38.987440+0000) 2022-01-31T22:23:39.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:39 smithi171 conmon[46715]: debug 2022-01-31T22:23:39.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:39.286355+0000) 2022-01-31T22:23:39.452 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:39 smithi171 conmon[51620]: debug 2022-01-31T22:23:39.144+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:39.145370+0000) 2022-01-31T22:23:39.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:39 smithi167 conmon[49112]: debug 2022-01-31T22:23:39.618+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:39.619562+0000) 2022-01-31T22:23:39.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:39 smithi167 conmon[54076]: debug 2022-01-31T22:23:39.789+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:39.789942+0000) 2022-01-31T22:23:39.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:39 smithi167 conmon[60316]: debug 2022-01-31T22:23:39.545+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:39.546200+0000) 2022-01-31T22:23:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:39 smithi171 conmon[41853]: debug 2022-01-31T22:23:39.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:39.493830+0000) 2022-01-31T22:23:40.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:40 smithi171 conmon[46715]: debug 2022-01-31T22:23:40.285+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:40.286494+0000) 2022-01-31T22:23:40.452 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:40 smithi171 conmon[51620]: debug 2022-01-31T22:23:40.144+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:40.145551+0000) 2022-01-31T22:23:40.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:40 smithi167 conmon[49112]: debug 2022-01-31T22:23:40.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:40.619704+0000) 2022-01-31T22:23:40.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:40 smithi167 conmon[54076]: debug 2022-01-31T22:23:40.789+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:40.790146+0000) 2022-01-31T22:23:40.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:40 smithi167 conmon[60316]: debug 2022-01-31T22:23:40.544+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:40.546377+0000) 2022-01-31T22:23:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:40 smithi171 conmon[41853]: debug 2022-01-31T22:23:40.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:40.493967+0000) 2022-01-31T22:23:41.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:41 smithi171 conmon[46715]: debug 2022-01-31T22:23:41.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:41.286685+0000) 2022-01-31T22:23:41.452 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:41 smithi171 conmon[51620]: debug 2022-01-31T22:23:41.145+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:41.145764+0000) 2022-01-31T22:23:41.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:41 smithi167 conmon[49112]: debug 2022-01-31T22:23:41.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:41.619889+0000) 2022-01-31T22:23:41.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:41 smithi167 conmon[54076]: debug 2022-01-31T22:23:41.789+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:41.790346+0000) 2022-01-31T22:23:41.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:41 smithi167 conmon[60316]: debug 2022-01-31T22:23:41.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:41.546596+0000) 2022-01-31T22:23:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:41 smithi171 conmon[41853]: debug 2022-01-31T22:23:41.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:41.494168+0000) 2022-01-31T22:23:42.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:42 smithi171 conmon[46715]: debug 2022-01-31T22:23:42.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:42.286842+0000) 2022-01-31T22:23:42.452 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:42 smithi171 conmon[51620]: debug 2022-01-31T22:23:42.145+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:42.145949+0000) 2022-01-31T22:23:42.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:42 smithi167 conmon[49112]: debug 2022-01-31T22:23:42.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:42.620079+0000) 2022-01-31T22:23:42.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:42 smithi167 conmon[54076]: debug 2022-01-31T22:23:42.790+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:42.790525+0000) 2022-01-31T22:23:42.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:42 smithi167 conmon[60316]: debug 2022-01-31T22:23:42.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:42.546806+0000) 2022-01-31T22:23:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:42 smithi171 conmon[41853]: debug 2022-01-31T22:23:42.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:42.494365+0000) 2022-01-31T22:23:43.452 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:43 smithi171 conmon[46715]: debug 2022-01-31T22:23:43.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:43.286988+0000) 2022-01-31T22:23:43.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:43 smithi171 conmon[51620]: debug 2022-01-31T22:23:43.145+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:43.146099+0000) 2022-01-31T22:23:43.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:43 smithi167 conmon[49112]: debug 2022-01-31T22:23:43.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:43.620264+0000) 2022-01-31T22:23:43.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:43 smithi167 conmon[54076]: debug 2022-01-31T22:23:43.790+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:43.790661+0000) 2022-01-31T22:23:43.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:43 smithi167 conmon[60316]: debug 2022-01-31T22:23:43.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:43.546982+0000) 2022-01-31T22:23:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:43 smithi171 conmon[41853]: debug 2022-01-31T22:23:43.493+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:43.494523+0000) 2022-01-31T22:23:44.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:44 smithi167 conmon[49112]: debug 2022-01-31T22:23:44.010+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.011228+0000) 2022-01-31T22:23:44.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:44 smithi167 conmon[54076]: debug 2022-01-31T22:23:44.009+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.010231+0000) 2022-01-31T22:23:44.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:44 smithi167 conmon[60316]: debug 2022-01-31T22:23:44.011+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.011811+0000) 2022-01-31T22:23:44.286 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[35325]: debug 2022-01-31T22:23:44.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206003 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:44.287 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[41853]: debug 2022-01-31T22:23:44.010+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.011005+0000) 2022-01-31T22:23:44.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[46715]: debug 2022-01-31T22:23:44.009+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.009890+0000) 2022-01-31T22:23:44.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[51620]: debug 2022-01-31T22:23:44.010+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.010798+0000) 2022-01-31T22:23:44.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[51620]: debug 2022-01-31T22:23:44.145+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.146300+0000) 2022-01-31T22:23:44.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[41853]: debug 2022-01-31T22:23:44.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.494711+0000) 2022-01-31T22:23:44.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:44 smithi171 conmon[46715]: debug 2022-01-31T22:23:44.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.287218+0000) 2022-01-31T22:23:44.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:44 smithi167 conmon[49112]: debug 2022-01-31T22:23:44.619+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.620454+0000) 2022-01-31T22:23:44.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:44 smithi167 conmon[54076]: debug 2022-01-31T22:23:44.790+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.790856+0000) 2022-01-31T22:23:44.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:44 smithi167 conmon[60316]: debug 2022-01-31T22:23:44.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:44.547170+0000) 2022-01-31T22:23:45.452 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:45 smithi171 conmon[46715]: debug 2022-01-31T22:23:45.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:45.287293+0000) 2022-01-31T22:23:45.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:45 smithi171 conmon[51620]: debug 2022-01-31T22:23:45.145+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:45.146488+0000) 2022-01-31T22:23:45.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:45 smithi167 conmon[49112]: debug 2022-01-31T22:23:45.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:45.620608+0000) 2022-01-31T22:23:45.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:45 smithi167 conmon[54076]: debug 2022-01-31T22:23:45.790+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:45.790973+0000) 2022-01-31T22:23:45.841 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:45 smithi167 conmon[60316]: debug 2022-01-31T22:23:45.546+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:45.547340+0000) 2022-01-31T22:23:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:45 smithi171 conmon[41853]: debug 2022-01-31T22:23:45.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:45.494870+0000) 2022-01-31T22:23:46.452 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:46 smithi171 conmon[46715]: debug 2022-01-31T22:23:46.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:46.287472+0000) 2022-01-31T22:23:46.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:46 smithi171 conmon[51620]: debug 2022-01-31T22:23:46.146+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:46.146696+0000) 2022-01-31T22:23:46.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:46 smithi167 conmon[49112]: debug 2022-01-31T22:23:46.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:46.620813+0000) 2022-01-31T22:23:46.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:46 smithi167 conmon[54076]: debug 2022-01-31T22:23:46.790+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:46.791139+0000) 2022-01-31T22:23:46.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:46 smithi167 conmon[60316]: debug 2022-01-31T22:23:46.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:46.547546+0000) 2022-01-31T22:23:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:46 smithi171 conmon[41853]: debug 2022-01-31T22:23:46.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:46.494986+0000) 2022-01-31T22:23:47.452 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:47 smithi171 conmon[46715]: debug 2022-01-31T22:23:47.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:47.287644+0000) 2022-01-31T22:23:47.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:47 smithi171 conmon[51620]: debug 2022-01-31T22:23:47.146+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:47.146899+0000) 2022-01-31T22:23:47.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:47 smithi167 conmon[49112]: debug 2022-01-31T22:23:47.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:47.620968+0000) 2022-01-31T22:23:47.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:47 smithi167 conmon[54076]: debug 2022-01-31T22:23:47.790+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:47.791348+0000) 2022-01-31T22:23:47.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:47 smithi167 conmon[60316]: debug 2022-01-31T22:23:47.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:47.547780+0000) 2022-01-31T22:23:47.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:47 smithi171 conmon[41853]: debug 2022-01-31T22:23:47.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:47.495172+0000) 2022-01-31T22:23:48.453 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:48 smithi171 conmon[46715]: debug 2022-01-31T22:23:48.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:48.287848+0000) 2022-01-31T22:23:48.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:48 smithi171 conmon[51620]: debug 2022-01-31T22:23:48.146+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:48.147063+0000) 2022-01-31T22:23:48.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:48 smithi167 conmon[49112]: debug 2022-01-31T22:23:48.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:48.621108+0000) 2022-01-31T22:23:48.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:48 smithi167 conmon[54076]: debug 2022-01-31T22:23:48.791+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:48.791533+0000) 2022-01-31T22:23:48.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:48 smithi167 conmon[60316]: debug 2022-01-31T22:23:48.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:48.547964+0000) 2022-01-31T22:23:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:48 smithi171 conmon[41853]: debug 2022-01-31T22:23:48.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:48.495363+0000) 2022-01-31T22:23:49.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:49 smithi167 conmon[54076]: debug 2022-01-31T22:23:49.034+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.034851+0000) 2022-01-31T22:23:49.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:49 smithi167 conmon[60316]: debug 2022-01-31T22:23:49.035+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.036450+0000) 2022-01-31T22:23:49.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:49 smithi167 conmon[49112]: debug 2022-01-31T22:23:49.035+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.036006+0000) 2022-01-31T22:23:49.287 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[35325]: debug 2022-01-31T22:23:49.055+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206115 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:49.288 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[41853]: debug 2022-01-31T22:23:49.035+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.036419+0000) 2022-01-31T22:23:49.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[46715]: debug 2022-01-31T22:23:49.033+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.034579+0000) 2022-01-31T22:23:49.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[51620]: debug 2022-01-31T22:23:49.033+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.034476+0000) 2022-01-31T22:23:49.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[51620]: debug 2022-01-31T22:23:49.146+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.147245+0000) 2022-01-31T22:23:49.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[41853]: debug 2022-01-31T22:23:49.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.495574+0000) 2022-01-31T22:23:49.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:49 smithi171 conmon[46715]: debug 2022-01-31T22:23:49.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.288003+0000) 2022-01-31T22:23:49.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:49 smithi167 conmon[49112]: debug 2022-01-31T22:23:49.620+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.621228+0000) 2022-01-31T22:23:49.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:49 smithi167 conmon[54076]: debug 2022-01-31T22:23:49.791+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.791695+0000) 2022-01-31T22:23:49.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:49 smithi167 conmon[60316]: debug 2022-01-31T22:23:49.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:49.548124+0000) 2022-01-31T22:23:50.453 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:50 smithi171 conmon[46715]: debug 2022-01-31T22:23:50.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:50.288142+0000) 2022-01-31T22:23:50.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:50 smithi171 conmon[51620]: debug 2022-01-31T22:23:50.147+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:50.147436+0000) 2022-01-31T22:23:50.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:50 smithi167 conmon[49112]: debug 2022-01-31T22:23:50.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:50.621368+0000) 2022-01-31T22:23:50.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:50 smithi167 conmon[54076]: debug 2022-01-31T22:23:50.791+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:50.791851+0000) 2022-01-31T22:23:50.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:50 smithi167 conmon[60316]: debug 2022-01-31T22:23:50.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:50.548263+0000) 2022-01-31T22:23:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:50 smithi171 conmon[41853]: debug 2022-01-31T22:23:50.494+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:50.495729+0000) 2022-01-31T22:23:51.453 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:51 smithi171 conmon[46715]: debug 2022-01-31T22:23:51.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:51.288320+0000) 2022-01-31T22:23:51.454 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:51 smithi171 conmon[51620]: debug 2022-01-31T22:23:51.147+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:51.147609+0000) 2022-01-31T22:23:51.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:51 smithi167 conmon[49112]: debug 2022-01-31T22:23:51.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:51.621546+0000) 2022-01-31T22:23:51.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:51 smithi167 conmon[54076]: debug 2022-01-31T22:23:51.791+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:51.792007+0000) 2022-01-31T22:23:51.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:51 smithi167 conmon[60316]: debug 2022-01-31T22:23:51.547+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:51.548390+0000) 2022-01-31T22:23:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:51 smithi171 conmon[41853]: debug 2022-01-31T22:23:51.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:51.495898+0000) 2022-01-31T22:23:52.453 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:52 smithi171 conmon[51620]: debug 2022-01-31T22:23:52.147+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:52.147816+0000) 2022-01-31T22:23:52.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:52 smithi171 conmon[46715]: debug 2022-01-31T22:23:52.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:52.288505+0000) 2022-01-31T22:23:52.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:52 smithi167 conmon[49112]: debug 2022-01-31T22:23:52.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:52.621694+0000) 2022-01-31T22:23:52.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:52 smithi167 conmon[60316]: debug 2022-01-31T22:23:52.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:52.548550+0000) 2022-01-31T22:23:52.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:52 smithi167 conmon[54076]: debug 2022-01-31T22:23:52.791+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:52.792169+0000) 2022-01-31T22:23:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:52 smithi171 conmon[41853]: debug 2022-01-31T22:23:52.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:52.496090+0000) 2022-01-31T22:23:53.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:53 smithi171 conmon[46715]: debug 2022-01-31T22:23:53.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:53.288669+0000) 2022-01-31T22:23:53.455 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:53 smithi171 conmon[51620]: debug 2022-01-31T22:23:53.147+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:53.147974+0000) 2022-01-31T22:23:53.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:53 smithi167 conmon[49112]: debug 2022-01-31T22:23:53.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:53.621840+0000) 2022-01-31T22:23:53.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:53 smithi167 conmon[54076]: debug 2022-01-31T22:23:53.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:53.792323+0000) 2022-01-31T22:23:53.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:53 smithi167 conmon[60316]: debug 2022-01-31T22:23:53.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:53.548740+0000) 2022-01-31T22:23:53.846 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:53 smithi171 conmon[41853]: debug 2022-01-31T22:23:53.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:53.496295+0000) 2022-01-31T22:23:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:53 smithi171 conmon[35325]: debug 2022-01-31T22:23:53.846+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:23:54.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[35325]: debug 2022-01-31T22:23:54.095+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206227 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:54.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[41853]: debug 2022-01-31T22:23:54.058+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.059723+0000) 2022-01-31T22:23:54.109 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[46715]: debug 2022-01-31T22:23:54.058+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.058771+0000) 2022-01-31T22:23:54.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[51620]: debug 2022-01-31T22:23:54.057+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.058404+0000) 2022-01-31T22:23:54.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:54 smithi167 conmon[49112]: debug 2022-01-31T22:23:54.057+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.058397+0000) 2022-01-31T22:23:54.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:54 smithi167 conmon[54076]: debug 2022-01-31T22:23:54.058+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.059091+0000) 2022-01-31T22:23:54.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:54 smithi167 conmon[60316]: debug 2022-01-31T22:23:54.058+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.058640+0000) 2022-01-31T22:23:54.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[46715]: debug 2022-01-31T22:23:54.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.288821+0000) 2022-01-31T22:23:54.454 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[51620]: debug 2022-01-31T22:23:54.147+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.148154+0000) 2022-01-31T22:23:54.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:54 smithi167 conmon[49112]: debug 2022-01-31T22:23:54.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.622018+0000) 2022-01-31T22:23:54.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:54 smithi167 conmon[54076]: debug 2022-01-31T22:23:54.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.792464+0000) 2022-01-31T22:23:54.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:54 smithi167 conmon[60316]: debug 2022-01-31T22:23:54.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.548884+0000) 2022-01-31T22:23:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:54 smithi171 conmon[41853]: debug 2022-01-31T22:23:54.495+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:54.496501+0000) 2022-01-31T22:23:55.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:55 smithi171 conmon[46715]: debug 2022-01-31T22:23:55.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:55.288978+0000) 2022-01-31T22:23:55.455 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:55 smithi171 conmon[51620]: debug 2022-01-31T22:23:55.147+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:55.148297+0000) 2022-01-31T22:23:55.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:55 smithi167 conmon[49112]: debug 2022-01-31T22:23:55.621+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:55.622157+0000) 2022-01-31T22:23:55.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:55 smithi167 conmon[54076]: debug 2022-01-31T22:23:55.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:55.792603+0000) 2022-01-31T22:23:55.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:55 smithi167 conmon[60316]: debug 2022-01-31T22:23:55.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:55.549032+0000) 2022-01-31T22:23:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:55 smithi171 conmon[41853]: debug 2022-01-31T22:23:55.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:55.496658+0000) 2022-01-31T22:23:56.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:56 smithi171 conmon[46715]: debug 2022-01-31T22:23:56.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:56.289202+0000) 2022-01-31T22:23:56.455 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:56 smithi171 conmon[51620]: debug 2022-01-31T22:23:56.148+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:56.148505+0000) 2022-01-31T22:23:56.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:56 smithi167 conmon[49112]: debug 2022-01-31T22:23:56.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:56.622348+0000) 2022-01-31T22:23:56.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:56 smithi167 conmon[54076]: debug 2022-01-31T22:23:56.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:56.792749+0000) 2022-01-31T22:23:56.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:56 smithi167 conmon[60316]: debug 2022-01-31T22:23:56.548+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:56.549225+0000) 2022-01-31T22:23:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:56 smithi171 conmon[41853]: debug 2022-01-31T22:23:56.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:56.496837+0000) 2022-01-31T22:23:57.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:57 smithi171 conmon[46715]: debug 2022-01-31T22:23:57.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:57.289433+0000) 2022-01-31T22:23:57.455 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:57 smithi171 conmon[51620]: debug 2022-01-31T22:23:57.148+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:57.148691+0000) 2022-01-31T22:23:57.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:57 smithi167 conmon[49112]: debug 2022-01-31T22:23:57.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:57.622534+0000) 2022-01-31T22:23:57.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:57 smithi167 conmon[54076]: debug 2022-01-31T22:23:57.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:57.792943+0000) 2022-01-31T22:23:57.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:57 smithi167 conmon[60316]: debug 2022-01-31T22:23:57.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:57.549434+0000) 2022-01-31T22:23:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:57 smithi171 conmon[41853]: debug 2022-01-31T22:23:57.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:57.496987+0000) 2022-01-31T22:23:58.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:58 smithi171 conmon[46715]: debug 2022-01-31T22:23:58.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:58.289648+0000) 2022-01-31T22:23:58.455 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:58 smithi171 conmon[51620]: debug 2022-01-31T22:23:58.148+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:58.148891+0000) 2022-01-31T22:23:58.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:58 smithi167 conmon[49112]: debug 2022-01-31T22:23:58.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:58.622723+0000) 2022-01-31T22:23:58.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:58 smithi167 conmon[54076]: debug 2022-01-31T22:23:58.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:58.793149+0000) 2022-01-31T22:23:58.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:58 smithi167 conmon[60316]: debug 2022-01-31T22:23:58.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:58.549610+0000) 2022-01-31T22:23:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:58 smithi171 conmon[41853]: debug 2022-01-31T22:23:58.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:58.497178+0000) 2022-01-31T22:23:59.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[35325]: debug 2022-01-31T22:23:59.119+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206339 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:23:59.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[46715]: debug 2022-01-31T22:23:59.098+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.099242+0000) 2022-01-31T22:23:59.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[46715]: debug 2022-01-31T22:23:59.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.289814+0000) 2022-01-31T22:23:59.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[51620]: debug 2022-01-31T22:23:59.097+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.098648+0000) 2022-01-31T22:23:59.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[51620]: debug 2022-01-31T22:23:59.148+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.149049+0000) 2022-01-31T22:23:59.359 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[41853]: debug 2022-01-31T22:23:59.098+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.099639+0000) 2022-01-31T22:23:59.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:59 smithi167 conmon[49112]: debug 2022-01-31T22:23:59.098+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.098936+0000) 2022-01-31T22:23:59.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:59 smithi167 conmon[54076]: debug 2022-01-31T22:23:59.097+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.098013+0000) 2022-01-31T22:23:59.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:59 smithi167 conmon[60316]: debug 2022-01-31T22:23:59.097+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.098146+0000) 2022-01-31T22:23:59.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:23:59 smithi167 conmon[49112]: debug 2022-01-31T22:23:59.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.622908+0000) 2022-01-31T22:23:59.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:23:59 smithi167 conmon[60316]: debug 2022-01-31T22:23:59.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.549796+0000) 2022-01-31T22:23:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:23:59 smithi171 conmon[41853]: debug 2022-01-31T22:23:59.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.497363+0000) 2022-01-31T22:24:00.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:23:59 smithi167 conmon[54076]: debug 2022-01-31T22:23:59.792+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:23:59.793327+0000) 2022-01-31T22:24:00.455 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:00 smithi171 conmon[51620]: debug 2022-01-31T22:24:00.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:00.149224+0000) 2022-01-31T22:24:00.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:00 smithi171 conmon[46715]: debug 2022-01-31T22:24:00.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:00.290007+0000) 2022-01-31T22:24:00.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:00 smithi167 conmon[54076]: debug 2022-01-31T22:24:00.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:00.793510+0000) 2022-01-31T22:24:00.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:00 smithi167 conmon[60316]: debug 2022-01-31T22:24:00.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:00.549927+0000) 2022-01-31T22:24:00.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:00 smithi167 conmon[49112]: debug 2022-01-31T22:24:00.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:00.623045+0000) 2022-01-31T22:24:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:00 smithi171 conmon[41853]: debug 2022-01-31T22:24:00.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:00.497510+0000) 2022-01-31T22:24:01.455 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:01 smithi171 conmon[46715]: debug 2022-01-31T22:24:01.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:01.290192+0000) 2022-01-31T22:24:01.456 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:01 smithi171 conmon[51620]: debug 2022-01-31T22:24:01.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:01.149384+0000) 2022-01-31T22:24:01.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:01 smithi167 conmon[49112]: debug 2022-01-31T22:24:01.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:01.623252+0000) 2022-01-31T22:24:01.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:01 smithi167 conmon[60316]: debug 2022-01-31T22:24:01.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:01.550106+0000) 2022-01-31T22:24:01.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:01 smithi167 conmon[54076]: debug 2022-01-31T22:24:01.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:01.793664+0000) 2022-01-31T22:24:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:01 smithi171 conmon[41853]: debug 2022-01-31T22:24:01.496+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:01.497697+0000) 2022-01-31T22:24:02.455 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:02 smithi171 conmon[46715]: debug 2022-01-31T22:24:02.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:02.290343+0000) 2022-01-31T22:24:02.456 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:02 smithi171 conmon[51620]: debug 2022-01-31T22:24:02.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:02.149545+0000) 2022-01-31T22:24:02.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:02 smithi167 conmon[49112]: debug 2022-01-31T22:24:02.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:02.623406+0000) 2022-01-31T22:24:02.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:02 smithi167 conmon[60316]: debug 2022-01-31T22:24:02.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:02.550291+0000) 2022-01-31T22:24:02.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:02 smithi167 conmon[54076]: debug 2022-01-31T22:24:02.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:02.793810+0000) 2022-01-31T22:24:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:02 smithi171 conmon[41853]: debug 2022-01-31T22:24:02.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:02.497924+0000) 2022-01-31T22:24:03.455 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:03 smithi171 conmon[46715]: debug 2022-01-31T22:24:03.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:03.290520+0000) 2022-01-31T22:24:03.456 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:03 smithi171 conmon[51620]: debug 2022-01-31T22:24:03.148+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:03.149649+0000) 2022-01-31T22:24:03.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:03 smithi167 conmon[49112]: debug 2022-01-31T22:24:03.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:03.623580+0000) 2022-01-31T22:24:03.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:03 smithi167 conmon[54076]: debug 2022-01-31T22:24:03.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:03.793986+0000) 2022-01-31T22:24:03.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:03 smithi167 conmon[60316]: debug 2022-01-31T22:24:03.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:03.550479+0000) 2022-01-31T22:24:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:03 smithi171 conmon[41853]: debug 2022-01-31T22:24:03.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:03.498154+0000) 2022-01-31T22:24:04.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:04 smithi167 conmon[49112]: debug 2022-01-31T22:24:04.122+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.123436+0000) 2022-01-31T22:24:04.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:04 smithi167 conmon[54076]: debug 2022-01-31T22:24:04.122+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.122782+0000) 2022-01-31T22:24:04.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:04 smithi167 conmon[60316]: debug 2022-01-31T22:24:04.122+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.122631+0000) 2022-01-31T22:24:04.455 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[35325]: debug 2022-01-31T22:24:04.142+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206452 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:04.456 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[41853]: debug 2022-01-31T22:24:04.121+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.122437+0000) 2022-01-31T22:24:04.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[46715]: debug 2022-01-31T22:24:04.121+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.122122+0000) 2022-01-31T22:24:04.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[46715]: debug 2022-01-31T22:24:04.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.290726+0000) 2022-01-31T22:24:04.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[51620]: debug 2022-01-31T22:24:04.121+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.121763+0000) 2022-01-31T22:24:04.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[51620]: debug 2022-01-31T22:24:04.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.149769+0000) 2022-01-31T22:24:04.793 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:04 smithi167 conmon[60316]: debug 2022-01-31T22:24:04.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.550701+0000) 2022-01-31T22:24:04.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:04 smithi167 conmon[49112]: debug 2022-01-31T22:24:04.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.623740+0000) 2022-01-31T22:24:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:04 smithi171 conmon[41853]: debug 2022-01-31T22:24:04.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.498358+0000) 2022-01-31T22:24:05.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:04 smithi167 conmon[54076]: debug 2022-01-31T22:24:04.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:04.794145+0000) 2022-01-31T22:24:05.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:05 smithi171 conmon[46715]: debug 2022-01-31T22:24:05.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:05.290904+0000) 2022-01-31T22:24:05.456 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:05 smithi171 conmon[51620]: debug 2022-01-31T22:24:05.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:05.149917+0000) 2022-01-31T22:24:05.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:05 smithi167 conmon[49112]: debug 2022-01-31T22:24:05.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:05.623916+0000) 2022-01-31T22:24:05.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:05 smithi167 conmon[54076]: debug 2022-01-31T22:24:05.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:05.794266+0000) 2022-01-31T22:24:05.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:05 smithi167 conmon[60316]: debug 2022-01-31T22:24:05.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:05.550795+0000) 2022-01-31T22:24:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:05 smithi171 conmon[41853]: debug 2022-01-31T22:24:05.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:05.498496+0000) 2022-01-31T22:24:06.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:06 smithi171 conmon[46715]: debug 2022-01-31T22:24:06.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:06.291059+0000) 2022-01-31T22:24:06.457 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:06 smithi171 conmon[51620]: debug 2022-01-31T22:24:06.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:06.150066+0000) 2022-01-31T22:24:06.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:06 smithi167 conmon[49112]: debug 2022-01-31T22:24:06.622+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:06.624134+0000) 2022-01-31T22:24:06.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:06 smithi167 conmon[60316]: debug 2022-01-31T22:24:06.549+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:06.550987+0000) 2022-01-31T22:24:06.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:06 smithi167 conmon[54076]: debug 2022-01-31T22:24:06.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:06.794473+0000) 2022-01-31T22:24:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:06 smithi171 conmon[41853]: debug 2022-01-31T22:24:06.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:06.498636+0000) 2022-01-31T22:24:07.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:07 smithi171 conmon[46715]: debug 2022-01-31T22:24:07.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:07.291211+0000) 2022-01-31T22:24:07.457 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:07 smithi171 conmon[51620]: debug 2022-01-31T22:24:07.150+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:07.150224+0000) 2022-01-31T22:24:07.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:07 smithi167 conmon[49112]: debug 2022-01-31T22:24:07.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:07.624314+0000) 2022-01-31T22:24:07.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:07 smithi167 conmon[60316]: debug 2022-01-31T22:24:07.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:07.551153+0000) 2022-01-31T22:24:07.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:07 smithi167 conmon[54076]: debug 2022-01-31T22:24:07.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:07.794680+0000) 2022-01-31T22:24:07.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:07 smithi171 conmon[41853]: debug 2022-01-31T22:24:07.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:07.498814+0000) 2022-01-31T22:24:08.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:08 smithi171 conmon[46715]: debug 2022-01-31T22:24:08.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:08.291388+0000) 2022-01-31T22:24:08.457 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:08 smithi171 conmon[51620]: debug 2022-01-31T22:24:08.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:08.150361+0000) 2022-01-31T22:24:08.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:08 smithi167 conmon[49112]: debug 2022-01-31T22:24:08.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:08.624520+0000) 2022-01-31T22:24:08.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:08 smithi167 conmon[54076]: debug 2022-01-31T22:24:08.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:08.794868+0000) 2022-01-31T22:24:08.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:08 smithi167 conmon[60316]: debug 2022-01-31T22:24:08.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:08.551349+0000) 2022-01-31T22:24:08.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:08 smithi171 conmon[41853]: debug 2022-01-31T22:24:08.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:08.499007+0000) 2022-01-31T22:24:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:08 smithi171 conmon[35325]: debug 2022-01-31T22:24:08.846+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:24:09.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:09 smithi167 conmon[49112]: debug 2022-01-31T22:24:09.145+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.146893+0000) 2022-01-31T22:24:09.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:09 smithi167 conmon[54076]: debug 2022-01-31T22:24:09.145+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.147350+0000) 2022-01-31T22:24:09.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:09 smithi167 conmon[60316]: debug 2022-01-31T22:24:09.145+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.147217+0000) 2022-01-31T22:24:09.456 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[35325]: debug 2022-01-31T22:24:09.164+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206563 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:09.457 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[41853]: debug 2022-01-31T22:24:09.143+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.145394+0000) 2022-01-31T22:24:09.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[46715]: debug 2022-01-31T22:24:09.143+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.145733+0000) 2022-01-31T22:24:09.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[46715]: debug 2022-01-31T22:24:09.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.291568+0000) 2022-01-31T22:24:09.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[51620]: debug 2022-01-31T22:24:09.143+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.145553+0000) 2022-01-31T22:24:09.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[51620]: debug 2022-01-31T22:24:09.148+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.150470+0000) 2022-01-31T22:24:09.794 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:09 smithi167 conmon[49112]: debug 2022-01-31T22:24:09.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.624719+0000) 2022-01-31T22:24:09.795 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:09 smithi167 conmon[60316]: debug 2022-01-31T22:24:09.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.551555+0000) 2022-01-31T22:24:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:09 smithi171 conmon[41853]: debug 2022-01-31T22:24:09.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.499184+0000) 2022-01-31T22:24:10.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:09 smithi167 conmon[54076]: debug 2022-01-31T22:24:09.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:09.795028+0000) 2022-01-31T22:24:10.456 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:10 smithi171 conmon[46715]: debug 2022-01-31T22:24:10.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:10.291715+0000) 2022-01-31T22:24:10.457 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:10 smithi171 conmon[51620]: debug 2022-01-31T22:24:10.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:10.150653+0000) 2022-01-31T22:24:10.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:10 smithi167 conmon[49112]: debug 2022-01-31T22:24:10.623+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:10.624913+0000) 2022-01-31T22:24:10.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:10 smithi167 conmon[54076]: debug 2022-01-31T22:24:10.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:10.795200+0000) 2022-01-31T22:24:10.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:10 smithi167 conmon[60316]: debug 2022-01-31T22:24:10.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:10.551695+0000) 2022-01-31T22:24:10.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:10 smithi171 conmon[41853]: debug 2022-01-31T22:24:10.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:10.499337+0000) 2022-01-31T22:24:11.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:11 smithi171 conmon[46715]: debug 2022-01-31T22:24:11.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:11.291852+0000) 2022-01-31T22:24:11.457 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:11 smithi171 conmon[51620]: debug 2022-01-31T22:24:11.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:11.150826+0000) 2022-01-31T22:24:11.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:11 smithi167 conmon[49112]: debug 2022-01-31T22:24:11.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:11.625084+0000) 2022-01-31T22:24:11.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:11 smithi167 conmon[54076]: debug 2022-01-31T22:24:11.793+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:11.795391+0000) 2022-01-31T22:24:11.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:11 smithi167 conmon[60316]: debug 2022-01-31T22:24:11.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:11.551853+0000) 2022-01-31T22:24:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:11 smithi171 conmon[41853]: debug 2022-01-31T22:24:11.497+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:11.499518+0000) 2022-01-31T22:24:12.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:12 smithi171 conmon[46715]: debug 2022-01-31T22:24:12.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:12.292007+0000) 2022-01-31T22:24:12.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:12 smithi171 conmon[51620]: debug 2022-01-31T22:24:12.150+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:12.151018+0000) 2022-01-31T22:24:12.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:12 smithi167 conmon[54076]: debug 2022-01-31T22:24:12.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:12.795581+0000) 2022-01-31T22:24:12.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:12 smithi167 conmon[49112]: debug 2022-01-31T22:24:12.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:12.625242+0000) 2022-01-31T22:24:12.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:12 smithi167 conmon[60316]: debug 2022-01-31T22:24:12.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:12.552043+0000) 2022-01-31T22:24:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:12 smithi171 conmon[41853]: debug 2022-01-31T22:24:12.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:12.499737+0000) 2022-01-31T22:24:13.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:13 smithi171 conmon[46715]: debug 2022-01-31T22:24:13.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:13.292212+0000) 2022-01-31T22:24:13.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:13 smithi171 conmon[51620]: debug 2022-01-31T22:24:13.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:13.151118+0000) 2022-01-31T22:24:13.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:13 smithi167 conmon[49112]: debug 2022-01-31T22:24:13.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:13.625430+0000) 2022-01-31T22:24:13.846 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:13 smithi167 conmon[54076]: debug 2022-01-31T22:24:13.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:13.795791+0000) 2022-01-31T22:24:13.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:13 smithi167 conmon[60316]: debug 2022-01-31T22:24:13.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:13.552169+0000) 2022-01-31T22:24:13.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:13 smithi171 conmon[41853]: debug 2022-01-31T22:24:13.498+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:13.499970+0000) 2022-01-31T22:24:14.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:14 smithi167 conmon[49112]: debug 2022-01-31T22:24:14.170+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.171548+0000) 2022-01-31T22:24:14.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:14 smithi167 conmon[54076]: debug 2022-01-31T22:24:14.168+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.169832+0000) 2022-01-31T22:24:14.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:14 smithi167 conmon[60316]: debug 2022-01-31T22:24:14.168+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.170212+0000) 2022-01-31T22:24:14.457 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[35325]: debug 2022-01-31T22:24:14.188+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206675 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:14.458 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[41853]: debug 2022-01-31T22:24:14.166+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.168159+0000) 2022-01-31T22:24:14.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[46715]: debug 2022-01-31T22:24:14.167+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.169258+0000) 2022-01-31T22:24:14.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[46715]: debug 2022-01-31T22:24:14.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.292333+0000) 2022-01-31T22:24:14.460 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[51620]: debug 2022-01-31T22:24:14.149+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.151279+0000) 2022-01-31T22:24:14.460 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[51620]: debug 2022-01-31T22:24:14.166+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.168058+0000) 2022-01-31T22:24:14.795 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:14 smithi167 conmon[49112]: debug 2022-01-31T22:24:14.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.625585+0000) 2022-01-31T22:24:14.796 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:14 smithi167 conmon[60316]: debug 2022-01-31T22:24:14.550+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.552349+0000) 2022-01-31T22:24:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:14 smithi171 conmon[41853]: debug 2022-01-31T22:24:14.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.500144+0000) 2022-01-31T22:24:15.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:14 smithi167 conmon[54076]: debug 2022-01-31T22:24:14.795+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:14.795911+0000) 2022-01-31T22:24:15.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:15 smithi171 conmon[46715]: debug 2022-01-31T22:24:15.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:15.292453+0000) 2022-01-31T22:24:15.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:15 smithi171 conmon[51620]: debug 2022-01-31T22:24:15.150+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:15.151448+0000) 2022-01-31T22:24:15.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:15 smithi167 conmon[60316]: debug 2022-01-31T22:24:15.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:15.552481+0000) 2022-01-31T22:24:15.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:15 smithi167 conmon[49112]: debug 2022-01-31T22:24:15.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:15.625743+0000) 2022-01-31T22:24:15.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:15 smithi167 conmon[54076]: debug 2022-01-31T22:24:15.794+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:15.796068+0000) 2022-01-31T22:24:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:15 smithi171 conmon[41853]: debug 2022-01-31T22:24:15.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:15.500305+0000) 2022-01-31T22:24:16.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:16 smithi171 conmon[46715]: debug 2022-01-31T22:24:16.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:16.292589+0000) 2022-01-31T22:24:16.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:16 smithi171 conmon[51620]: debug 2022-01-31T22:24:16.150+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:16.151617+0000) 2022-01-31T22:24:16.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:16 smithi167 conmon[49112]: debug 2022-01-31T22:24:16.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:16.625924+0000) 2022-01-31T22:24:16.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:16 smithi167 conmon[54076]: debug 2022-01-31T22:24:16.795+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:16.796257+0000) 2022-01-31T22:24:16.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:16 smithi167 conmon[60316]: debug 2022-01-31T22:24:16.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:16.552654+0000) 2022-01-31T22:24:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:16 smithi171 conmon[41853]: debug 2022-01-31T22:24:16.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:16.500489+0000) 2022-01-31T22:24:17.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:17 smithi171 conmon[46715]: debug 2022-01-31T22:24:17.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:17.292734+0000) 2022-01-31T22:24:17.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:17 smithi171 conmon[51620]: debug 2022-01-31T22:24:17.150+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:17.151791+0000) 2022-01-31T22:24:17.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:17 smithi167 conmon[49112]: debug 2022-01-31T22:24:17.624+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:17.626028+0000) 2022-01-31T22:24:17.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:17 smithi167 conmon[60316]: debug 2022-01-31T22:24:17.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:17.552898+0000) 2022-01-31T22:24:17.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:17 smithi167 conmon[54076]: debug 2022-01-31T22:24:17.795+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:17.796445+0000) 2022-01-31T22:24:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:17 smithi171 conmon[41853]: debug 2022-01-31T22:24:17.499+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:17.500643+0000) 2022-01-31T22:24:18.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:18 smithi171 conmon[46715]: debug 2022-01-31T22:24:18.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:18.292900+0000) 2022-01-31T22:24:18.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:18 smithi171 conmon[51620]: debug 2022-01-31T22:24:18.151+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:18.151988+0000) 2022-01-31T22:24:18.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:18 smithi167 conmon[60316]: debug 2022-01-31T22:24:18.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:18.553104+0000) 2022-01-31T22:24:18.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:18 smithi167 conmon[49112]: debug 2022-01-31T22:24:18.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:18.626180+0000) 2022-01-31T22:24:18.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:18 smithi167 conmon[54076]: debug 2022-01-31T22:24:18.795+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:18.796646+0000) 2022-01-31T22:24:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:18 smithi171 conmon[41853]: debug 2022-01-31T22:24:18.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:18.500858+0000) 2022-01-31T22:24:19.458 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[35325]: debug 2022-01-31T22:24:19.214+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206786 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:19.459 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[41853]: debug 2022-01-31T22:24:19.191+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.191969+0000) 2022-01-31T22:24:19.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[46715]: debug 2022-01-31T22:24:19.191+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.192701+0000) 2022-01-31T22:24:19.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[46715]: debug 2022-01-31T22:24:19.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.293060+0000) 2022-01-31T22:24:19.460 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[51620]: debug 2022-01-31T22:24:19.151+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.152188+0000) 2022-01-31T22:24:19.461 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[51620]: debug 2022-01-31T22:24:19.190+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.191625+0000) 2022-01-31T22:24:19.488 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:19 smithi167 conmon[49112]: debug 2022-01-31T22:24:19.192+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.193849+0000) 2022-01-31T22:24:19.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:19 smithi167 conmon[54076]: debug 2022-01-31T22:24:19.191+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.192640+0000) 2022-01-31T22:24:19.489 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:19 smithi167 conmon[60316]: debug 2022-01-31T22:24:19.191+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.193059+0000) 2022-01-31T22:24:19.796 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:19 smithi167 conmon[49112]: debug 2022-01-31T22:24:19.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.626333+0000) 2022-01-31T22:24:19.797 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:19 smithi167 conmon[60316]: debug 2022-01-31T22:24:19.551+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.553281+0000) 2022-01-31T22:24:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:19 smithi171 conmon[41853]: debug 2022-01-31T22:24:19.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.501021+0000) 2022-01-31T22:24:20.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:19 smithi167 conmon[54076]: debug 2022-01-31T22:24:19.795+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:19.796822+0000) 2022-01-31T22:24:20.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:20 smithi171 conmon[46715]: debug 2022-01-31T22:24:20.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:20.293211+0000) 2022-01-31T22:24:20.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:20 smithi171 conmon[51620]: debug 2022-01-31T22:24:20.151+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:20.152372+0000) 2022-01-31T22:24:20.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:20 smithi167 conmon[49112]: debug 2022-01-31T22:24:20.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:20.626485+0000) 2022-01-31T22:24:20.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:20 smithi167 conmon[54076]: debug 2022-01-31T22:24:20.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:20.796999+0000) 2022-01-31T22:24:20.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:20 smithi167 conmon[60316]: debug 2022-01-31T22:24:20.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:20.553440+0000) 2022-01-31T22:24:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:20 smithi171 conmon[41853]: debug 2022-01-31T22:24:20.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:20.501101+0000) 2022-01-31T22:24:21.458 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:21 smithi171 conmon[46715]: debug 2022-01-31T22:24:21.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:21.293374+0000) 2022-01-31T22:24:21.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:21 smithi171 conmon[51620]: debug 2022-01-31T22:24:21.151+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:21.152544+0000) 2022-01-31T22:24:21.847 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:21 smithi167 conmon[49112]: debug 2022-01-31T22:24:21.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:21.626694+0000) 2022-01-31T22:24:21.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:21 smithi167 conmon[60316]: debug 2022-01-31T22:24:21.552+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:21.553627+0000) 2022-01-31T22:24:21.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:21 smithi167 conmon[54076]: debug 2022-01-31T22:24:21.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:21.797160+0000) 2022-01-31T22:24:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:21 smithi171 conmon[41853]: debug 2022-01-31T22:24:21.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:21.501229+0000) 2022-01-31T22:24:22.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:22 smithi171 conmon[46715]: debug 2022-01-31T22:24:22.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:22.293556+0000) 2022-01-31T22:24:22.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:22 smithi171 conmon[51620]: debug 2022-01-31T22:24:22.151+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:22.152735+0000) 2022-01-31T22:24:22.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:22 smithi167 conmon[49112]: debug 2022-01-31T22:24:22.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:22.626899+0000) 2022-01-31T22:24:22.812 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:22 smithi167 conmon[60316]: debug 2022-01-31T22:24:22.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:22.553792+0000) 2022-01-31T22:24:22.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:22 smithi167 conmon[54076]: debug 2022-01-31T22:24:22.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:22.797312+0000) 2022-01-31T22:24:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:22 smithi171 conmon[41853]: debug 2022-01-31T22:24:22.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:22.501391+0000) 2022-01-31T22:24:23.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:23 smithi171 conmon[46715]: debug 2022-01-31T22:24:23.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:23.293742+0000) 2022-01-31T22:24:23.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:23 smithi171 conmon[51620]: debug 2022-01-31T22:24:23.152+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:23.152911+0000) 2022-01-31T22:24:23.847 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:23 smithi167 conmon[60316]: debug 2022-01-31T22:24:23.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:23.554036+0000) 2022-01-31T22:24:23.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:23 smithi167 conmon[49112]: debug 2022-01-31T22:24:23.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:23.627082+0000) 2022-01-31T22:24:23.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:23 smithi167 conmon[54076]: debug 2022-01-31T22:24:23.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:23.797442+0000) 2022-01-31T22:24:23.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:23 smithi171 conmon[41853]: debug 2022-01-31T22:24:23.500+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:23.501593+0000) 2022-01-31T22:24:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:23 smithi171 conmon[35325]: debug 2022-01-31T22:24:23.847+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:24:24.459 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[35325]: debug 2022-01-31T22:24:24.238+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 206899 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:24.460 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[41853]: debug 2022-01-31T22:24:24.216+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.217450+0000) 2022-01-31T22:24:24.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[46715]: debug 2022-01-31T22:24:24.217+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.217817+0000) 2022-01-31T22:24:24.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[46715]: debug 2022-01-31T22:24:24.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.293955+0000) 2022-01-31T22:24:24.461 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[51620]: debug 2022-01-31T22:24:24.152+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.153083+0000) 2022-01-31T22:24:24.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[51620]: debug 2022-01-31T22:24:24.216+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.217352+0000) 2022-01-31T22:24:24.503 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:24 smithi167 conmon[49112]: debug 2022-01-31T22:24:24.217+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.219347+0000) 2022-01-31T22:24:24.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:24 smithi167 conmon[54076]: debug 2022-01-31T22:24:24.216+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.217809+0000) 2022-01-31T22:24:24.504 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:24 smithi167 conmon[60316]: debug 2022-01-31T22:24:24.216+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.218122+0000) 2022-01-31T22:24:24.797 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:24 smithi167 conmon[49112]: debug 2022-01-31T22:24:24.625+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.627258+0000) 2022-01-31T22:24:24.797 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:24 smithi167 conmon[60316]: debug 2022-01-31T22:24:24.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.554229+0000) 2022-01-31T22:24:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:24 smithi171 conmon[41853]: debug 2022-01-31T22:24:24.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.501762+0000) 2022-01-31T22:24:25.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:24 smithi167 conmon[54076]: debug 2022-01-31T22:24:24.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:24.797584+0000) 2022-01-31T22:24:25.459 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:25 smithi171 conmon[46715]: debug 2022-01-31T22:24:25.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:25.294097+0000) 2022-01-31T22:24:25.460 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:25 smithi171 conmon[51620]: debug 2022-01-31T22:24:25.152+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:25.153269+0000) 2022-01-31T22:24:25.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:25 smithi167 conmon[49112]: debug 2022-01-31T22:24:25.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:25.627416+0000) 2022-01-31T22:24:25.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:25 smithi167 conmon[60316]: debug 2022-01-31T22:24:25.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:25.554389+0000) 2022-01-31T22:24:25.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:25 smithi167 conmon[54076]: debug 2022-01-31T22:24:25.797+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:25.797746+0000) 2022-01-31T22:24:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:25 smithi171 conmon[41853]: debug 2022-01-31T22:24:25.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:25.501877+0000) 2022-01-31T22:24:26.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:26 smithi171 conmon[51620]: debug 2022-01-31T22:24:26.152+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:26.153404+0000) 2022-01-31T22:24:26.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:26 smithi171 conmon[46715]: debug 2022-01-31T22:24:26.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:26.294265+0000) 2022-01-31T22:24:26.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:26 smithi167 conmon[49112]: debug 2022-01-31T22:24:26.626+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:26.627602+0000) 2022-01-31T22:24:26.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:26 smithi167 conmon[54076]: debug 2022-01-31T22:24:26.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:26.797874+0000) 2022-01-31T22:24:26.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:26 smithi167 conmon[60316]: debug 2022-01-31T22:24:26.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:26.554599+0000) 2022-01-31T22:24:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:26 smithi171 conmon[41853]: debug 2022-01-31T22:24:26.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:26.502030+0000) 2022-01-31T22:24:27.459 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:27 smithi171 conmon[51620]: debug 2022-01-31T22:24:27.152+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:27.153567+0000) 2022-01-31T22:24:27.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:27 smithi171 conmon[46715]: debug 2022-01-31T22:24:27.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:27.294428+0000) 2022-01-31T22:24:27.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:27 smithi167 conmon[49112]: debug 2022-01-31T22:24:27.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:27.627786+0000) 2022-01-31T22:24:27.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:27 smithi167 conmon[54076]: debug 2022-01-31T22:24:27.796+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:27.798061+0000) 2022-01-31T22:24:27.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:27 smithi167 conmon[60316]: debug 2022-01-31T22:24:27.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:27.554807+0000) 2022-01-31T22:24:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:27 smithi171 conmon[41853]: debug 2022-01-31T22:24:27.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:27.502191+0000) 2022-01-31T22:24:28.460 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:28 smithi171 conmon[51620]: debug 2022-01-31T22:24:28.152+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:28.153720+0000) 2022-01-31T22:24:28.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:28 smithi171 conmon[46715]: debug 2022-01-31T22:24:28.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:28.294611+0000) 2022-01-31T22:24:28.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:28 smithi167 conmon[49112]: debug 2022-01-31T22:24:28.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:28.627966+0000) 2022-01-31T22:24:28.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:28 smithi167 conmon[54076]: debug 2022-01-31T22:24:28.797+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:28.798264+0000) 2022-01-31T22:24:28.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:28 smithi167 conmon[60316]: debug 2022-01-31T22:24:28.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:28.554994+0000) 2022-01-31T22:24:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:28 smithi171 conmon[41853]: debug 2022-01-31T22:24:28.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:28.502318+0000) 2022-01-31T22:24:29.460 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[35325]: debug 2022-01-31T22:24:29.262+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207010 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:29.461 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[41853]: debug 2022-01-31T22:24:29.240+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.241151+0000) 2022-01-31T22:24:29.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[46715]: debug 2022-01-31T22:24:29.240+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.241568+0000) 2022-01-31T22:24:29.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[46715]: debug 2022-01-31T22:24:29.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.294828+0000) 2022-01-31T22:24:29.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[51620]: debug 2022-01-31T22:24:29.153+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.153911+0000) 2022-01-31T22:24:29.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[51620]: debug 2022-01-31T22:24:29.240+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.240822+0000) 2022-01-31T22:24:29.518 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:29 smithi167 conmon[49112]: debug 2022-01-31T22:24:29.240+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.242251+0000) 2022-01-31T22:24:29.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:29 smithi167 conmon[54076]: debug 2022-01-31T22:24:29.240+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.241934+0000) 2022-01-31T22:24:29.519 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:29 smithi167 conmon[60316]: debug 2022-01-31T22:24:29.241+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.242602+0000) 2022-01-31T22:24:29.798 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:29 smithi167 conmon[49112]: debug 2022-01-31T22:24:29.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.628171+0000) 2022-01-31T22:24:29.798 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:29 smithi167 conmon[60316]: debug 2022-01-31T22:24:29.553+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.555160+0000) 2022-01-31T22:24:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:29 smithi171 conmon[41853]: debug 2022-01-31T22:24:29.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.502489+0000) 2022-01-31T22:24:30.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:29 smithi167 conmon[54076]: debug 2022-01-31T22:24:29.797+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:29.798474+0000) 2022-01-31T22:24:30.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:30 smithi171 conmon[46715]: debug 2022-01-31T22:24:30.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:30.294967+0000) 2022-01-31T22:24:30.461 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:30 smithi171 conmon[51620]: debug 2022-01-31T22:24:30.153+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:30.154103+0000) 2022-01-31T22:24:30.848 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:30 smithi167 conmon[54076]: debug 2022-01-31T22:24:30.798+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:30.798675+0000) 2022-01-31T22:24:30.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:30 smithi167 conmon[60316]: debug 2022-01-31T22:24:30.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:30.555274+0000) 2022-01-31T22:24:30.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:30 smithi167 conmon[49112]: debug 2022-01-31T22:24:30.627+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:30.628337+0000) 2022-01-31T22:24:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:30 smithi171 conmon[41853]: debug 2022-01-31T22:24:30.501+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:30.502662+0000) 2022-01-31T22:24:31.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:31 smithi171 conmon[46715]: debug 2022-01-31T22:24:31.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:31.295122+0000) 2022-01-31T22:24:31.461 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:31 smithi171 conmon[51620]: debug 2022-01-31T22:24:31.153+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:31.154267+0000) 2022-01-31T22:24:31.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:31 smithi167 conmon[60316]: debug 2022-01-31T22:24:31.554+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:31.555450+0000) 2022-01-31T22:24:31.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:31 smithi167 conmon[49112]: debug 2022-01-31T22:24:31.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:31.628490+0000) 2022-01-31T22:24:31.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:31 smithi167 conmon[54076]: debug 2022-01-31T22:24:31.798+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:31.798878+0000) 2022-01-31T22:24:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:31 smithi171 conmon[41853]: debug 2022-01-31T22:24:31.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:31.502849+0000) 2022-01-31T22:24:32.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:32 smithi171 conmon[46715]: debug 2022-01-31T22:24:32.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:32.295273+0000) 2022-01-31T22:24:32.461 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:32 smithi171 conmon[51620]: debug 2022-01-31T22:24:32.153+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:32.154450+0000) 2022-01-31T22:24:32.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:32 smithi167 conmon[60316]: debug 2022-01-31T22:24:32.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:32.555659+0000) 2022-01-31T22:24:32.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:32 smithi167 conmon[49112]: debug 2022-01-31T22:24:32.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:32.628653+0000) 2022-01-31T22:24:32.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:32 smithi167 conmon[54076]: debug 2022-01-31T22:24:32.798+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:32.799063+0000) 2022-01-31T22:24:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:32 smithi171 conmon[41853]: debug 2022-01-31T22:24:32.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:32.503050+0000) 2022-01-31T22:24:33.460 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:33 smithi171 conmon[46715]: debug 2022-01-31T22:24:33.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:33.295438+0000) 2022-01-31T22:24:33.461 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:33 smithi171 conmon[51620]: debug 2022-01-31T22:24:33.153+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:33.154659+0000) 2022-01-31T22:24:33.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:33 smithi167 conmon[49112]: debug 2022-01-31T22:24:33.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:33.628819+0000) 2022-01-31T22:24:33.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:33 smithi167 conmon[54076]: debug 2022-01-31T22:24:33.798+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:33.799268+0000) 2022-01-31T22:24:33.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:33 smithi167 conmon[60316]: debug 2022-01-31T22:24:33.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:33.555825+0000) 2022-01-31T22:24:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:33 smithi171 conmon[41853]: debug 2022-01-31T22:24:33.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:33.503284+0000) 2022-01-31T22:24:34.461 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[35325]: debug 2022-01-31T22:24:34.286+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:34.461 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[41853]: debug 2022-01-31T22:24:34.264+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.265638+0000) 2022-01-31T22:24:34.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[46715]: debug 2022-01-31T22:24:34.265+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.265952+0000) 2022-01-31T22:24:34.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[46715]: debug 2022-01-31T22:24:34.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.295557+0000) 2022-01-31T22:24:34.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[51620]: debug 2022-01-31T22:24:34.154+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.154848+0000) 2022-01-31T22:24:34.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[51620]: debug 2022-01-31T22:24:34.264+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.265145+0000) 2022-01-31T22:24:34.525 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:34 smithi167 conmon[49112]: debug 2022-01-31T22:24:34.266+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.267079+0000) 2022-01-31T22:24:34.525 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:34 smithi167 conmon[54076]: debug 2022-01-31T22:24:34.266+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.266753+0000) 2022-01-31T22:24:34.526 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:34 smithi167 conmon[60316]: debug 2022-01-31T22:24:34.266+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.267187+0000) 2022-01-31T22:24:34.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:34 smithi167 conmon[60316]: debug 2022-01-31T22:24:34.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.556007+0000) 2022-01-31T22:24:34.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:34 smithi167 conmon[49112]: debug 2022-01-31T22:24:34.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.629025+0000) 2022-01-31T22:24:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:34 smithi171 conmon[41853]: debug 2022-01-31T22:24:34.502+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.503436+0000) 2022-01-31T22:24:35.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:34 smithi167 conmon[54076]: debug 2022-01-31T22:24:34.798+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:34.799424+0000) 2022-01-31T22:24:35.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:35 smithi171 conmon[46715]: debug 2022-01-31T22:24:35.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:35.295709+0000) 2022-01-31T22:24:35.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:35 smithi171 conmon[51620]: debug 2022-01-31T22:24:35.154+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:35.155033+0000) 2022-01-31T22:24:35.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:35 smithi167 conmon[49112]: debug 2022-01-31T22:24:35.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:35.629183+0000) 2022-01-31T22:24:35.799 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:35 smithi167 conmon[60316]: debug 2022-01-31T22:24:35.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:35.556104+0000) 2022-01-31T22:24:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:35 smithi171 conmon[41853]: debug 2022-01-31T22:24:35.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:35.503581+0000) 2022-01-31T22:24:36.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:35 smithi167 conmon[54076]: debug 2022-01-31T22:24:35.799+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:35.799546+0000) 2022-01-31T22:24:36.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:36 smithi171 conmon[46715]: debug 2022-01-31T22:24:36.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:36.295940+0000) 2022-01-31T22:24:36.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:36 smithi171 conmon[51620]: debug 2022-01-31T22:24:36.154+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:36.155227+0000) 2022-01-31T22:24:36.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:36 smithi167 conmon[49112]: debug 2022-01-31T22:24:36.628+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:36.629367+0000) 2022-01-31T22:24:36.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:36 smithi167 conmon[60316]: debug 2022-01-31T22:24:36.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:36.556293+0000) 2022-01-31T22:24:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:36 smithi171 conmon[41853]: debug 2022-01-31T22:24:36.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:36.503794+0000) 2022-01-31T22:24:37.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:36 smithi167 conmon[54076]: debug 2022-01-31T22:24:36.799+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:36.799691+0000) 2022-01-31T22:24:37.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:37 smithi171 conmon[46715]: debug 2022-01-31T22:24:37.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:37.296132+0000) 2022-01-31T22:24:37.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:37 smithi171 conmon[51620]: debug 2022-01-31T22:24:37.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:37.155410+0000) 2022-01-31T22:24:37.800 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:37 smithi167 conmon[49112]: debug 2022-01-31T22:24:37.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:37.629571+0000) 2022-01-31T22:24:37.801 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:37 smithi167 conmon[60316]: debug 2022-01-31T22:24:37.555+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:37.556475+0000) 2022-01-31T22:24:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:37 smithi171 conmon[41853]: debug 2022-01-31T22:24:37.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:37.503920+0000) 2022-01-31T22:24:38.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:37 smithi167 conmon[54076]: debug 2022-01-31T22:24:37.799+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:37.799851+0000) 2022-01-31T22:24:38.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:38 smithi171 conmon[46715]: debug 2022-01-31T22:24:38.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:38.296329+0000) 2022-01-31T22:24:38.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:38 smithi171 conmon[51620]: debug 2022-01-31T22:24:38.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:38.155622+0000) 2022-01-31T22:24:38.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:38 smithi167 conmon[49112]: debug 2022-01-31T22:24:38.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:38.629729+0000) 2022-01-31T22:24:38.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:38 smithi167 conmon[60316]: debug 2022-01-31T22:24:38.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:38.556695+0000) 2022-01-31T22:24:38.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:38 smithi171 conmon[41853]: debug 2022-01-31T22:24:38.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:38.504099+0000) 2022-01-31T22:24:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:38 smithi171 conmon[35325]: debug 2022-01-31T22:24:38.848+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:24:39.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:38 smithi167 conmon[54076]: debug 2022-01-31T22:24:38.799+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:38.800058+0000) 2022-01-31T22:24:39.462 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[35325]: debug 2022-01-31T22:24:39.310+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207232 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:39.462 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[41853]: debug 2022-01-31T22:24:39.288+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.289184+0000) 2022-01-31T22:24:39.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[46715]: debug 2022-01-31T22:24:39.289+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.289877+0000) 2022-01-31T22:24:39.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[46715]: debug 2022-01-31T22:24:39.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.296467+0000) 2022-01-31T22:24:39.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[51620]: debug 2022-01-31T22:24:39.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.155802+0000) 2022-01-31T22:24:39.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[51620]: debug 2022-01-31T22:24:39.288+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.289654+0000) 2022-01-31T22:24:39.548 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:39 smithi167 conmon[49112]: debug 2022-01-31T22:24:39.289+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.290195+0000) 2022-01-31T22:24:39.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:39 smithi167 conmon[54076]: debug 2022-01-31T22:24:39.290+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.290689+0000) 2022-01-31T22:24:39.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:39 smithi167 conmon[60316]: debug 2022-01-31T22:24:39.290+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.290994+0000) 2022-01-31T22:24:39.799 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:39 smithi167 conmon[49112]: debug 2022-01-31T22:24:39.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.629901+0000) 2022-01-31T22:24:39.800 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:39 smithi167 conmon[60316]: debug 2022-01-31T22:24:39.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.556844+0000) 2022-01-31T22:24:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:39 smithi171 conmon[41853]: debug 2022-01-31T22:24:39.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.504286+0000) 2022-01-31T22:24:40.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:39 smithi167 conmon[54076]: debug 2022-01-31T22:24:39.799+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:39.800219+0000) 2022-01-31T22:24:40.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:40 smithi171 conmon[46715]: debug 2022-01-31T22:24:40.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:40.296632+0000) 2022-01-31T22:24:40.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:40 smithi171 conmon[51620]: debug 2022-01-31T22:24:40.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:40.155979+0000) 2022-01-31T22:24:40.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:40 smithi167 conmon[49112]: debug 2022-01-31T22:24:40.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:40.630051+0000) 2022-01-31T22:24:40.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:40 smithi167 conmon[60316]: debug 2022-01-31T22:24:40.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:40.556966+0000) 2022-01-31T22:24:40.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:40 smithi167 conmon[54076]: debug 2022-01-31T22:24:40.799+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:40.800395+0000) 2022-01-31T22:24:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:40 smithi171 conmon[41853]: debug 2022-01-31T22:24:40.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:40.504458+0000) 2022-01-31T22:24:41.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:41 smithi171 conmon[46715]: debug 2022-01-31T22:24:41.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:41.296830+0000) 2022-01-31T22:24:41.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:41 smithi171 conmon[51620]: debug 2022-01-31T22:24:41.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:41.156122+0000) 2022-01-31T22:24:41.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:41 smithi167 conmon[49112]: debug 2022-01-31T22:24:41.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:41.630207+0000) 2022-01-31T22:24:41.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:41 smithi167 conmon[54076]: debug 2022-01-31T22:24:41.800+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:41.800598+0000) 2022-01-31T22:24:41.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:41 smithi167 conmon[60316]: debug 2022-01-31T22:24:41.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:41.557100+0000) 2022-01-31T22:24:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:41 smithi171 conmon[41853]: debug 2022-01-31T22:24:41.503+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:41.504660+0000) 2022-01-31T22:24:42.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:42 smithi171 conmon[46715]: debug 2022-01-31T22:24:42.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:42.297019+0000) 2022-01-31T22:24:42.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:42 smithi171 conmon[51620]: debug 2022-01-31T22:24:42.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:42.156293+0000) 2022-01-31T22:24:42.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:42 smithi167 conmon[49112]: debug 2022-01-31T22:24:42.629+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:42.630392+0000) 2022-01-31T22:24:42.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:42 smithi167 conmon[54076]: debug 2022-01-31T22:24:42.800+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:42.800805+0000) 2022-01-31T22:24:42.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:42 smithi167 conmon[60316]: debug 2022-01-31T22:24:42.556+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:42.557256+0000) 2022-01-31T22:24:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:42 smithi171 conmon[41853]: debug 2022-01-31T22:24:42.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:42.504849+0000) 2022-01-31T22:24:43.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:43 smithi171 conmon[46715]: debug 2022-01-31T22:24:43.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:43.297166+0000) 2022-01-31T22:24:43.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:43 smithi171 conmon[51620]: debug 2022-01-31T22:24:43.155+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:43.156497+0000) 2022-01-31T22:24:43.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:43 smithi167 conmon[49112]: debug 2022-01-31T22:24:43.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:43.630579+0000) 2022-01-31T22:24:43.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:43 smithi167 conmon[54076]: debug 2022-01-31T22:24:43.800+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:43.800997+0000) 2022-01-31T22:24:43.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:43 smithi167 conmon[60316]: debug 2022-01-31T22:24:43.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:43.557444+0000) 2022-01-31T22:24:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:43 smithi171 conmon[41853]: debug 2022-01-31T22:24:43.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:43.505018+0000) 2022-01-31T22:24:44.462 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[35325]: debug 2022-01-31T22:24:44.334+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207344 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:44.463 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[41853]: debug 2022-01-31T22:24:44.313+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.314336+0000) 2022-01-31T22:24:44.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[46715]: debug 2022-01-31T22:24:44.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.297350+0000) 2022-01-31T22:24:44.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[46715]: debug 2022-01-31T22:24:44.314+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.315034+0000) 2022-01-31T22:24:44.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[51620]: debug 2022-01-31T22:24:44.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.156654+0000) 2022-01-31T22:24:44.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[51620]: debug 2022-01-31T22:24:44.313+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.313976+0000) 2022-01-31T22:24:44.557 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:44 smithi167 conmon[49112]: debug 2022-01-31T22:24:44.313+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.313734+0000) 2022-01-31T22:24:44.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:44 smithi167 conmon[54076]: debug 2022-01-31T22:24:44.313+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.314254+0000) 2022-01-31T22:24:44.558 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:44 smithi167 conmon[60316]: debug 2022-01-31T22:24:44.314+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.315319+0000) 2022-01-31T22:24:44.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:44 smithi167 conmon[49112]: debug 2022-01-31T22:24:44.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.630714+0000) 2022-01-31T22:24:44.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:44 smithi167 conmon[54076]: debug 2022-01-31T22:24:44.800+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.801188+0000) 2022-01-31T22:24:44.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:44 smithi167 conmon[60316]: debug 2022-01-31T22:24:44.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.557621+0000) 2022-01-31T22:24:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:44 smithi171 conmon[41853]: debug 2022-01-31T22:24:44.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:44.505222+0000) 2022-01-31T22:24:45.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:45 smithi171 conmon[46715]: debug 2022-01-31T22:24:45.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:45.297530+0000) 2022-01-31T22:24:45.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:45 smithi171 conmon[51620]: debug 2022-01-31T22:24:45.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:45.156836+0000) 2022-01-31T22:24:45.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:45 smithi167 conmon[49112]: debug 2022-01-31T22:24:45.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:45.630861+0000) 2022-01-31T22:24:45.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:45 smithi167 conmon[54076]: debug 2022-01-31T22:24:45.801+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:45.801378+0000) 2022-01-31T22:24:45.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:45 smithi167 conmon[60316]: debug 2022-01-31T22:24:45.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:45.557725+0000) 2022-01-31T22:24:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:45 smithi171 conmon[41853]: debug 2022-01-31T22:24:45.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:45.505360+0000) 2022-01-31T22:24:46.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:46 smithi171 conmon[46715]: debug 2022-01-31T22:24:46.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:46.297688+0000) 2022-01-31T22:24:46.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:46 smithi171 conmon[51620]: debug 2022-01-31T22:24:46.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:46.156929+0000) 2022-01-31T22:24:46.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:46 smithi167 conmon[49112]: debug 2022-01-31T22:24:46.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:46.631052+0000) 2022-01-31T22:24:46.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:46 smithi167 conmon[54076]: debug 2022-01-31T22:24:46.801+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:46.801566+0000) 2022-01-31T22:24:46.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:46 smithi167 conmon[60316]: debug 2022-01-31T22:24:46.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:46.557915+0000) 2022-01-31T22:24:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:46 smithi171 conmon[41853]: debug 2022-01-31T22:24:46.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:46.505532+0000) 2022-01-31T22:24:47.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:47 smithi171 conmon[46715]: debug 2022-01-31T22:24:47.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:47.297839+0000) 2022-01-31T22:24:47.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:47 smithi171 conmon[51620]: debug 2022-01-31T22:24:47.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:47.157136+0000) 2022-01-31T22:24:47.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:47 smithi167 conmon[49112]: debug 2022-01-31T22:24:47.630+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:47.631249+0000) 2022-01-31T22:24:47.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:47 smithi167 conmon[54076]: debug 2022-01-31T22:24:47.801+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:47.801753+0000) 2022-01-31T22:24:47.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:47 smithi167 conmon[60316]: debug 2022-01-31T22:24:47.557+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:47.558127+0000) 2022-01-31T22:24:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:47 smithi171 conmon[41853]: debug 2022-01-31T22:24:47.504+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:47.505716+0000) 2022-01-31T22:24:48.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:48 smithi171 conmon[51620]: debug 2022-01-31T22:24:48.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:48.157342+0000) 2022-01-31T22:24:48.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:48 smithi171 conmon[46715]: debug 2022-01-31T22:24:48.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:48.298026+0000) 2022-01-31T22:24:48.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:48 smithi167 conmon[49112]: debug 2022-01-31T22:24:48.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:48.631403+0000) 2022-01-31T22:24:48.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:48 smithi167 conmon[60316]: debug 2022-01-31T22:24:48.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:48.558311+0000) 2022-01-31T22:24:48.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:48 smithi167 conmon[54076]: debug 2022-01-31T22:24:48.801+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:48.801934+0000) 2022-01-31T22:24:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:48 smithi171 conmon[41853]: debug 2022-01-31T22:24:48.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:48.505930+0000) 2022-01-31T22:24:49.465 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[35325]: debug 2022-01-31T22:24:49.358+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207454 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:49.466 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[41853]: debug 2022-01-31T22:24:49.337+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.337978+0000) 2022-01-31T22:24:49.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[46715]: debug 2022-01-31T22:24:49.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.298208+0000) 2022-01-31T22:24:49.467 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[46715]: debug 2022-01-31T22:24:49.337+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.338524+0000) 2022-01-31T22:24:49.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[51620]: debug 2022-01-31T22:24:49.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.157489+0000) 2022-01-31T22:24:49.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[51620]: debug 2022-01-31T22:24:49.336+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.337088+0000) 2022-01-31T22:24:49.558 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:49 smithi167 conmon[49112]: debug 2022-01-31T22:24:49.337+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.337913+0000) 2022-01-31T22:24:49.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:49 smithi167 conmon[54076]: debug 2022-01-31T22:24:49.337+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.338189+0000) 2022-01-31T22:24:49.559 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:49 smithi167 conmon[60316]: debug 2022-01-31T22:24:49.337+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.338457+0000) 2022-01-31T22:24:49.851 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:49 smithi167 conmon[49112]: debug 2022-01-31T22:24:49.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.631602+0000) 2022-01-31T22:24:49.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:49 smithi167 conmon[54076]: debug 2022-01-31T22:24:49.801+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.802092+0000) 2022-01-31T22:24:49.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:49 smithi167 conmon[60316]: debug 2022-01-31T22:24:49.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.558528+0000) 2022-01-31T22:24:49.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:49 smithi171 conmon[41853]: debug 2022-01-31T22:24:49.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:49.506056+0000) 2022-01-31T22:24:50.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:50 smithi171 conmon[46715]: debug 2022-01-31T22:24:50.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:50.298399+0000) 2022-01-31T22:24:50.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:50 smithi171 conmon[51620]: debug 2022-01-31T22:24:50.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:50.157682+0000) 2022-01-31T22:24:50.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:50 smithi167 conmon[49112]: debug 2022-01-31T22:24:50.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:50.631757+0000) 2022-01-31T22:24:50.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:50 smithi167 conmon[54076]: debug 2022-01-31T22:24:50.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:50.802266+0000) 2022-01-31T22:24:50.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:50 smithi167 conmon[60316]: debug 2022-01-31T22:24:50.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:50.558701+0000) 2022-01-31T22:24:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:50 smithi171 conmon[41853]: debug 2022-01-31T22:24:50.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:50.506174+0000) 2022-01-31T22:24:51.463 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:51 smithi171 conmon[46715]: debug 2022-01-31T22:24:51.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:51.298527+0000) 2022-01-31T22:24:51.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:51 smithi171 conmon[51620]: debug 2022-01-31T22:24:51.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:51.157851+0000) 2022-01-31T22:24:51.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:51 smithi167 conmon[49112]: debug 2022-01-31T22:24:51.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:51.631944+0000) 2022-01-31T22:24:51.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:51 smithi167 conmon[60316]: debug 2022-01-31T22:24:51.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:51.558889+0000) 2022-01-31T22:24:51.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:51 smithi167 conmon[54076]: debug 2022-01-31T22:24:51.801+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:51.802453+0000) 2022-01-31T22:24:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:51 smithi171 conmon[41853]: debug 2022-01-31T22:24:51.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:51.506358+0000) 2022-01-31T22:24:52.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:52 smithi171 conmon[46715]: debug 2022-01-31T22:24:52.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:52.298709+0000) 2022-01-31T22:24:52.464 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:52 smithi171 conmon[51620]: debug 2022-01-31T22:24:52.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:52.158030+0000) 2022-01-31T22:24:52.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:52 smithi167 conmon[49112]: debug 2022-01-31T22:24:52.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:52.632150+0000) 2022-01-31T22:24:52.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:52 smithi167 conmon[54076]: debug 2022-01-31T22:24:52.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:52.802648+0000) 2022-01-31T22:24:52.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:52 smithi167 conmon[60316]: debug 2022-01-31T22:24:52.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:52.559046+0000) 2022-01-31T22:24:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:52 smithi171 conmon[41853]: debug 2022-01-31T22:24:52.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:52.506512+0000) 2022-01-31T22:24:53.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:53 smithi171 conmon[46715]: debug 2022-01-31T22:24:53.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:53.298928+0000) 2022-01-31T22:24:53.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:53 smithi171 conmon[51620]: debug 2022-01-31T22:24:53.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:53.158134+0000) 2022-01-31T22:24:53.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:53 smithi171 conmon[41853]: debug 2022-01-31T22:24:53.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:53.506671+0000) 2022-01-31T22:24:53.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:53 smithi167 conmon[49112]: debug 2022-01-31T22:24:53.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:53.632371+0000) 2022-01-31T22:24:53.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:53 smithi167 conmon[54076]: debug 2022-01-31T22:24:53.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:53.802777+0000) 2022-01-31T22:24:53.853 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:53 smithi167 conmon[60316]: debug 2022-01-31T22:24:53.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:53.559194+0000) 2022-01-31T22:24:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:53 smithi171 conmon[35325]: debug 2022-01-31T22:24:53.848+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:24:54.464 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[35325]: debug 2022-01-31T22:24:54.387+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207567 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:54.465 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[41853]: debug 2022-01-31T22:24:54.358+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.362858+0000) 2022-01-31T22:24:54.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[46715]: debug 2022-01-31T22:24:54.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.299127+0000) 2022-01-31T22:24:54.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[46715]: debug 2022-01-31T22:24:54.358+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.361432+0000) 2022-01-31T22:24:54.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[51620]: debug 2022-01-31T22:24:54.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.158318+0000) 2022-01-31T22:24:54.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[51620]: debug 2022-01-31T22:24:54.358+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.360938+0000) 2022-01-31T22:24:54.558 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:54 smithi167 conmon[49112]: debug 2022-01-31T22:24:54.359+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.362237+0000) 2022-01-31T22:24:54.559 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:54 smithi167 conmon[54076]: debug 2022-01-31T22:24:54.359+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.360852+0000) 2022-01-31T22:24:54.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:54 smithi167 conmon[60316]: debug 2022-01-31T22:24:54.359+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.362451+0000) 2022-01-31T22:24:54.560 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:54 smithi167 conmon[60316]: debug 2022-01-31T22:24:54.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.559371+0000) 2022-01-31T22:24:54.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:54 smithi167 conmon[49112]: debug 2022-01-31T22:24:54.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.632563+0000) 2022-01-31T22:24:54.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:54 smithi167 conmon[54076]: debug 2022-01-31T22:24:54.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.802930+0000) 2022-01-31T22:24:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:54 smithi171 conmon[41853]: debug 2022-01-31T22:24:54.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:54.506875+0000) 2022-01-31T22:24:55.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:55 smithi171 conmon[46715]: debug 2022-01-31T22:24:55.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:55.299308+0000) 2022-01-31T22:24:55.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:55 smithi171 conmon[51620]: debug 2022-01-31T22:24:55.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:55.158472+0000) 2022-01-31T22:24:55.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:55 smithi167 conmon[49112]: debug 2022-01-31T22:24:55.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:55.632736+0000) 2022-01-31T22:24:55.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:55 smithi167 conmon[54076]: debug 2022-01-31T22:24:55.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:55.803077+0000) 2022-01-31T22:24:55.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:55 smithi167 conmon[60316]: debug 2022-01-31T22:24:55.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:55.559538+0000) 2022-01-31T22:24:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:55 smithi171 conmon[41853]: debug 2022-01-31T22:24:55.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:55.507017+0000) 2022-01-31T22:24:56.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:56 smithi171 conmon[46715]: debug 2022-01-31T22:24:56.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:56.299463+0000) 2022-01-31T22:24:56.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:56 smithi171 conmon[51620]: debug 2022-01-31T22:24:56.156+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:56.158656+0000) 2022-01-31T22:24:56.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:56 smithi167 conmon[49112]: debug 2022-01-31T22:24:56.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:56.632902+0000) 2022-01-31T22:24:56.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:56 smithi167 conmon[54076]: debug 2022-01-31T22:24:56.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:56.803236+0000) 2022-01-31T22:24:56.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:56 smithi167 conmon[60316]: debug 2022-01-31T22:24:56.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:56.559726+0000) 2022-01-31T22:24:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:56 smithi171 conmon[41853]: debug 2022-01-31T22:24:56.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:56.507183+0000) 2022-01-31T22:24:57.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:57 smithi171 conmon[46715]: debug 2022-01-31T22:24:57.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:57.299626+0000) 2022-01-31T22:24:57.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:57 smithi171 conmon[51620]: debug 2022-01-31T22:24:57.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:57.158831+0000) 2022-01-31T22:24:57.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:57 smithi167 conmon[49112]: debug 2022-01-31T22:24:57.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:57.633110+0000) 2022-01-31T22:24:57.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:57 smithi167 conmon[54076]: debug 2022-01-31T22:24:57.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:57.803393+0000) 2022-01-31T22:24:57.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:57 smithi167 conmon[60316]: debug 2022-01-31T22:24:57.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:57.559884+0000) 2022-01-31T22:24:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:57 smithi171 conmon[41853]: debug 2022-01-31T22:24:57.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:57.507294+0000) 2022-01-31T22:24:58.465 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:58 smithi171 conmon[46715]: debug 2022-01-31T22:24:58.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:58.299792+0000) 2022-01-31T22:24:58.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:58 smithi171 conmon[51620]: debug 2022-01-31T22:24:58.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:58.159016+0000) 2022-01-31T22:24:58.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:58 smithi167 conmon[49112]: debug 2022-01-31T22:24:58.631+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:58.633272+0000) 2022-01-31T22:24:58.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:58 smithi167 conmon[54076]: debug 2022-01-31T22:24:58.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:58.803549+0000) 2022-01-31T22:24:58.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:58 smithi167 conmon[60316]: debug 2022-01-31T22:24:58.558+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:58.560029+0000) 2022-01-31T22:24:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:58 smithi171 conmon[41853]: debug 2022-01-31T22:24:58.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:58.507481+0000) 2022-01-31T22:24:59.413 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[41853]: debug 2022-01-31T22:24:59.390+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.391922+0000) 2022-01-31T22:24:59.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[46715]: debug 2022-01-31T22:24:59.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.299970+0000) 2022-01-31T22:24:59.414 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[46715]: debug 2022-01-31T22:24:59.389+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.391383+0000) 2022-01-31T22:24:59.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[51620]: debug 2022-01-31T22:24:59.158+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.159203+0000) 2022-01-31T22:24:59.415 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[51620]: debug 2022-01-31T22:24:59.388+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.390559+0000) 2022-01-31T22:24:59.560 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:59 smithi167 conmon[49112]: debug 2022-01-31T22:24:59.389+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.391167+0000) 2022-01-31T22:24:59.560 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:59 smithi167 conmon[54076]: debug 2022-01-31T22:24:59.390+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.392433+0000) 2022-01-31T22:24:59.561 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:59 smithi167 conmon[60316]: debug 2022-01-31T22:24:59.389+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.391321+0000) 2022-01-31T22:24:59.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:24:59 smithi167 conmon[49112]: debug 2022-01-31T22:24:59.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.633435+0000) 2022-01-31T22:24:59.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:24:59 smithi167 conmon[54076]: debug 2022-01-31T22:24:59.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.803729+0000) 2022-01-31T22:24:59.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:24:59 smithi167 conmon[60316]: debug 2022-01-31T22:24:59.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.560251+0000) 2022-01-31T22:24:59.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[35325]: debug 2022-01-31T22:24:59.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207677 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:24:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:24:59 smithi171 conmon[41853]: debug 2022-01-31T22:24:59.505+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:24:59.507635+0000) 2022-01-31T22:25:00.465 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:00 smithi171 conmon[46715]: debug 2022-01-31T22:25:00.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:00.300175+0000) 2022-01-31T22:25:00.466 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:00 smithi171 conmon[51620]: debug 2022-01-31T22:25:00.158+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:00.159416+0000) 2022-01-31T22:25:00.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:00 smithi167 conmon[49112]: debug 2022-01-31T22:25:00.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:00.633597+0000) 2022-01-31T22:25:00.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:00 smithi167 conmon[54076]: debug 2022-01-31T22:25:00.802+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:00.803920+0000) 2022-01-31T22:25:00.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:00 smithi167 conmon[60316]: debug 2022-01-31T22:25:00.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:00.560437+0000) 2022-01-31T22:25:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:00 smithi171 conmon[41853]: debug 2022-01-31T22:25:00.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:00.507807+0000) 2022-01-31T22:25:01.465 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:01 smithi171 conmon[46715]: debug 2022-01-31T22:25:01.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:01.300325+0000) 2022-01-31T22:25:01.466 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:01 smithi171 conmon[51620]: debug 2022-01-31T22:25:01.157+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:01.159578+0000) 2022-01-31T22:25:01.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:01 smithi167 conmon[49112]: debug 2022-01-31T22:25:01.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:01.633761+0000) 2022-01-31T22:25:01.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:01 smithi167 conmon[54076]: debug 2022-01-31T22:25:01.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:01.804103+0000) 2022-01-31T22:25:01.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:01 smithi167 conmon[60316]: debug 2022-01-31T22:25:01.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:01.560608+0000) 2022-01-31T22:25:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:01 smithi171 conmon[41853]: debug 2022-01-31T22:25:01.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:01.507932+0000) 2022-01-31T22:25:02.465 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:02 smithi171 conmon[46715]: debug 2022-01-31T22:25:02.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:02.300507+0000) 2022-01-31T22:25:02.466 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:02 smithi171 conmon[51620]: debug 2022-01-31T22:25:02.158+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:02.159795+0000) 2022-01-31T22:25:02.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:02 smithi167 conmon[49112]: debug 2022-01-31T22:25:02.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:02.633866+0000) 2022-01-31T22:25:02.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:02 smithi167 conmon[60316]: debug 2022-01-31T22:25:02.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:02.560787+0000) 2022-01-31T22:25:02.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:02 smithi167 conmon[54076]: debug 2022-01-31T22:25:02.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:02.804312+0000) 2022-01-31T22:25:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:02 smithi171 conmon[41853]: debug 2022-01-31T22:25:02.506+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:02.508032+0000) 2022-01-31T22:25:03.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:03 smithi171 conmon[46715]: debug 2022-01-31T22:25:03.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:03.300663+0000) 2022-01-31T22:25:03.466 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:03 smithi171 conmon[51620]: debug 2022-01-31T22:25:03.158+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:03.159953+0000) 2022-01-31T22:25:03.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:03 smithi167 conmon[49112]: debug 2022-01-31T22:25:03.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:03.634030+0000) 2022-01-31T22:25:03.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:03 smithi167 conmon[60316]: debug 2022-01-31T22:25:03.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:03.560949+0000) 2022-01-31T22:25:03.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:03 smithi167 conmon[54076]: debug 2022-01-31T22:25:03.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:03.804492+0000) 2022-01-31T22:25:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:03 smithi171 conmon[41853]: debug 2022-01-31T22:25:03.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:03.508214+0000) 2022-01-31T22:25:04.415 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[46715]: debug 2022-01-31T22:25:04.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.300831+0000) 2022-01-31T22:25:04.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[51620]: debug 2022-01-31T22:25:04.159+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.160111+0000) 2022-01-31T22:25:04.560 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:04 smithi167 conmon[49112]: debug 2022-01-31T22:25:04.414+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.416444+0000) 2022-01-31T22:25:04.561 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:04 smithi167 conmon[54076]: debug 2022-01-31T22:25:04.414+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.415802+0000) 2022-01-31T22:25:04.562 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:04 smithi167 conmon[60316]: debug 2022-01-31T22:25:04.414+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.415727+0000) 2022-01-31T22:25:04.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:04 smithi167 conmon[49112]: debug 2022-01-31T22:25:04.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.634250+0000) 2022-01-31T22:25:04.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:04 smithi167 conmon[54076]: debug 2022-01-31T22:25:04.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.804653+0000) 2022-01-31T22:25:04.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:04 smithi167 conmon[60316]: debug 2022-01-31T22:25:04.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.561098+0000) 2022-01-31T22:25:04.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[35325]: debug 2022-01-31T22:25:04.435+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207791 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[41853]: debug 2022-01-31T22:25:04.414+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.416192+0000) 2022-01-31T22:25:04.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[41853]: debug 2022-01-31T22:25:04.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.508397+0000) 2022-01-31T22:25:04.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[46715]: debug 2022-01-31T22:25:04.414+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.416662+0000) 2022-01-31T22:25:04.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:04 smithi171 conmon[51620]: debug 2022-01-31T22:25:04.413+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:04.415385+0000) 2022-01-31T22:25:05.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:05 smithi171 conmon[46715]: debug 2022-01-31T22:25:05.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:05.300983+0000) 2022-01-31T22:25:05.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:05 smithi171 conmon[51620]: debug 2022-01-31T22:25:05.159+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:05.160299+0000) 2022-01-31T22:25:05.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:05 smithi167 conmon[49112]: debug 2022-01-31T22:25:05.632+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:05.634361+0000) 2022-01-31T22:25:05.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:05 smithi167 conmon[54076]: debug 2022-01-31T22:25:05.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:05.804806+0000) 2022-01-31T22:25:05.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:05 smithi167 conmon[60316]: debug 2022-01-31T22:25:05.559+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:05.561258+0000) 2022-01-31T22:25:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:05 smithi171 conmon[41853]: debug 2022-01-31T22:25:05.507+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:05.508549+0000) 2022-01-31T22:25:06.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:06 smithi171 conmon[46715]: debug 2022-01-31T22:25:06.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:06.301129+0000) 2022-01-31T22:25:06.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:06 smithi171 conmon[51620]: debug 2022-01-31T22:25:06.159+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:06.160464+0000) 2022-01-31T22:25:06.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:06 smithi167 conmon[49112]: debug 2022-01-31T22:25:06.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:06.634505+0000) 2022-01-31T22:25:06.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:06 smithi167 conmon[54076]: debug 2022-01-31T22:25:06.804+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:06.804992+0000) 2022-01-31T22:25:06.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:06 smithi167 conmon[60316]: debug 2022-01-31T22:25:06.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:06.561442+0000) 2022-01-31T22:25:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:06 smithi171 conmon[41853]: debug 2022-01-31T22:25:06.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:06.508736+0000) 2022-01-31T22:25:07.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:07 smithi171 conmon[46715]: debug 2022-01-31T22:25:07.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:07.301281+0000) 2022-01-31T22:25:07.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:07 smithi171 conmon[51620]: debug 2022-01-31T22:25:07.159+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:07.160692+0000) 2022-01-31T22:25:07.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:07 smithi167 conmon[49112]: debug 2022-01-31T22:25:07.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:07.634670+0000) 2022-01-31T22:25:07.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:07 smithi167 conmon[60316]: debug 2022-01-31T22:25:07.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:07.561587+0000) 2022-01-31T22:25:07.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:07 smithi167 conmon[54076]: debug 2022-01-31T22:25:07.804+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:07.805165+0000) 2022-01-31T22:25:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:07 smithi171 conmon[41853]: debug 2022-01-31T22:25:07.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:07.508939+0000) 2022-01-31T22:25:08.467 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:08 smithi171 conmon[46715]: debug 2022-01-31T22:25:08.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:08.301465+0000) 2022-01-31T22:25:08.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:08 smithi171 conmon[51620]: debug 2022-01-31T22:25:08.160+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:08.160859+0000) 2022-01-31T22:25:08.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:08 smithi171 conmon[41853]: debug 2022-01-31T22:25:08.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:08.509123+0000) 2022-01-31T22:25:08.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:08 smithi167 conmon[49112]: debug 2022-01-31T22:25:08.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:08.634834+0000) 2022-01-31T22:25:08.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:08 smithi167 conmon[54076]: debug 2022-01-31T22:25:08.803+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:08.805384+0000) 2022-01-31T22:25:08.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:08 smithi167 conmon[60316]: debug 2022-01-31T22:25:08.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:08.561809+0000) 2022-01-31T22:25:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:08 smithi171 conmon[35325]: debug 2022-01-31T22:25:08.849+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:25:09.439 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[46715]: debug 2022-01-31T22:25:09.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.301655+0000) 2022-01-31T22:25:09.440 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[51620]: debug 2022-01-31T22:25:09.160+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.161029+0000) 2022-01-31T22:25:09.440 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[41853]: debug 2022-01-31T22:25:09.438+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.439400+0000) 2022-01-31T22:25:09.561 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:09 smithi167 conmon[49112]: debug 2022-01-31T22:25:09.439+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.441240+0000) 2022-01-31T22:25:09.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:09 smithi167 conmon[54076]: debug 2022-01-31T22:25:09.438+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.440035+0000) 2022-01-31T22:25:09.563 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:09 smithi167 conmon[60316]: debug 2022-01-31T22:25:09.438+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.440113+0000) 2022-01-31T22:25:09.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:09 smithi167 conmon[49112]: debug 2022-01-31T22:25:09.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.635067+0000) 2022-01-31T22:25:09.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:09 smithi167 conmon[54076]: debug 2022-01-31T22:25:09.804+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.805569+0000) 2022-01-31T22:25:09.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:09 smithi167 conmon[60316]: debug 2022-01-31T22:25:09.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.561942+0000) 2022-01-31T22:25:09.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[35325]: debug 2022-01-31T22:25:09.462+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 207900 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[41853]: debug 2022-01-31T22:25:09.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.509272+0000) 2022-01-31T22:25:09.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[46715]: debug 2022-01-31T22:25:09.440+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.441307+0000) 2022-01-31T22:25:09.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:09 smithi171 conmon[51620]: debug 2022-01-31T22:25:09.438+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:09.439513+0000) 2022-01-31T22:25:10.467 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:10 smithi171 conmon[46715]: debug 2022-01-31T22:25:10.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:10.301844+0000) 2022-01-31T22:25:10.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:10 smithi171 conmon[51620]: debug 2022-01-31T22:25:10.160+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:10.161236+0000) 2022-01-31T22:25:10.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:10 smithi167 conmon[49112]: debug 2022-01-31T22:25:10.633+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:10.635234+0000) 2022-01-31T22:25:10.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:10 smithi167 conmon[54076]: debug 2022-01-31T22:25:10.805+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:10.805758+0000) 2022-01-31T22:25:10.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:10 smithi167 conmon[60316]: debug 2022-01-31T22:25:10.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:10.562048+0000) 2022-01-31T22:25:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:10 smithi171 conmon[41853]: debug 2022-01-31T22:25:10.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:10.509417+0000) 2022-01-31T22:25:11.467 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:11 smithi171 conmon[46715]: debug 2022-01-31T22:25:11.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:11.301998+0000) 2022-01-31T22:25:11.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:11 smithi171 conmon[51620]: debug 2022-01-31T22:25:11.160+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:11.161429+0000) 2022-01-31T22:25:11.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:11 smithi167 conmon[49112]: debug 2022-01-31T22:25:11.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:11.635415+0000) 2022-01-31T22:25:11.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:11 smithi167 conmon[54076]: debug 2022-01-31T22:25:11.804+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:11.805964+0000) 2022-01-31T22:25:11.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:11 smithi167 conmon[60316]: debug 2022-01-31T22:25:11.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:11.562209+0000) 2022-01-31T22:25:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:11 smithi171 conmon[41853]: debug 2022-01-31T22:25:11.508+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:11.509596+0000) 2022-01-31T22:25:12.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:12 smithi171 conmon[46715]: debug 2022-01-31T22:25:12.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:12.302193+0000) 2022-01-31T22:25:12.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:12 smithi171 conmon[51620]: debug 2022-01-31T22:25:12.160+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:12.161639+0000) 2022-01-31T22:25:12.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:12 smithi167 conmon[49112]: debug 2022-01-31T22:25:12.634+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:12.635597+0000) 2022-01-31T22:25:12.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:12 smithi167 conmon[54076]: debug 2022-01-31T22:25:12.804+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:12.806122+0000) 2022-01-31T22:25:12.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:12 smithi167 conmon[60316]: debug 2022-01-31T22:25:12.560+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:12.562387+0000) 2022-01-31T22:25:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:12 smithi171 conmon[41853]: debug 2022-01-31T22:25:12.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:12.509772+0000) 2022-01-31T22:25:13.467 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:13 smithi171 conmon[46715]: debug 2022-01-31T22:25:13.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:13.302377+0000) 2022-01-31T22:25:13.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:13 smithi171 conmon[51620]: debug 2022-01-31T22:25:13.161+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:13.161843+0000) 2022-01-31T22:25:13.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:13 smithi167 conmon[49112]: debug 2022-01-31T22:25:13.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:13.635779+0000) 2022-01-31T22:25:13.856 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:13 smithi167 conmon[54076]: debug 2022-01-31T22:25:13.804+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:13.806302+0000) 2022-01-31T22:25:13.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:13 smithi167 conmon[60316]: debug 2022-01-31T22:25:13.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:13.562541+0000) 2022-01-31T22:25:13.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:13 smithi171 conmon[41853]: debug 2022-01-31T22:25:13.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:13.510014+0000) 2022-01-31T22:25:14.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[46715]: debug 2022-01-31T22:25:14.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.302569+0000) 2022-01-31T22:25:14.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[51620]: debug 2022-01-31T22:25:14.161+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.162037+0000) 2022-01-31T22:25:14.466 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[41853]: debug 2022-01-31T22:25:14.464+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.465023+0000) 2022-01-31T22:25:14.562 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:14 smithi167 conmon[49112]: debug 2022-01-31T22:25:14.465+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.466977+0000) 2022-01-31T22:25:14.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:14 smithi167 conmon[54076]: debug 2022-01-31T22:25:14.463+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.465475+0000) 2022-01-31T22:25:14.563 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:14 smithi167 conmon[60316]: debug 2022-01-31T22:25:14.464+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.465772+0000) 2022-01-31T22:25:14.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:14 smithi167 conmon[49112]: debug 2022-01-31T22:25:14.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.635969+0000) 2022-01-31T22:25:14.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:14 smithi167 conmon[60316]: debug 2022-01-31T22:25:14.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.562722+0000) 2022-01-31T22:25:14.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:14 smithi167 conmon[54076]: debug 2022-01-31T22:25:14.805+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.806508+0000) 2022-01-31T22:25:14.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[41853]: debug 2022-01-31T22:25:14.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.510209+0000) 2022-01-31T22:25:14.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[46715]: debug 2022-01-31T22:25:14.465+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.466313+0000) 2022-01-31T22:25:14.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[51620]: debug 2022-01-31T22:25:14.465+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:14.465871+0000) 2022-01-31T22:25:14.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:14 smithi171 conmon[35325]: debug 2022-01-31T22:25:14.486+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208014 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:15.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:15 smithi171 conmon[46715]: debug 2022-01-31T22:25:15.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:15.302773+0000) 2022-01-31T22:25:15.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:15 smithi171 conmon[51620]: debug 2022-01-31T22:25:15.161+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:15.162196+0000) 2022-01-31T22:25:15.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:15 smithi167 conmon[49112]: debug 2022-01-31T22:25:15.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:15.636113+0000) 2022-01-31T22:25:15.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:15 smithi167 conmon[60316]: debug 2022-01-31T22:25:15.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:15.562881+0000) 2022-01-31T22:25:15.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:15 smithi167 conmon[54076]: debug 2022-01-31T22:25:15.805+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:15.806601+0000) 2022-01-31T22:25:15.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:15 smithi171 conmon[41853]: debug 2022-01-31T22:25:15.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:15.510384+0000) 2022-01-31T22:25:16.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:16 smithi171 conmon[46715]: debug 2022-01-31T22:25:16.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:16.302931+0000) 2022-01-31T22:25:16.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:16 smithi171 conmon[51620]: debug 2022-01-31T22:25:16.161+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:16.162341+0000) 2022-01-31T22:25:16.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:16 smithi167 conmon[49112]: debug 2022-01-31T22:25:16.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:16.636305+0000) 2022-01-31T22:25:16.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:16 smithi167 conmon[60316]: debug 2022-01-31T22:25:16.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:16.563025+0000) 2022-01-31T22:25:16.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:16 smithi167 conmon[54076]: debug 2022-01-31T22:25:16.806+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:16.806777+0000) 2022-01-31T22:25:16.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:16 smithi171 conmon[41853]: debug 2022-01-31T22:25:16.509+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:16.510560+0000) 2022-01-31T22:25:17.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:17 smithi171 conmon[46715]: debug 2022-01-31T22:25:17.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:17.303080+0000) 2022-01-31T22:25:17.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:17 smithi171 conmon[51620]: debug 2022-01-31T22:25:17.161+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:17.162563+0000) 2022-01-31T22:25:17.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:17 smithi167 conmon[49112]: debug 2022-01-31T22:25:17.635+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:17.636538+0000) 2022-01-31T22:25:17.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:17 smithi167 conmon[54076]: debug 2022-01-31T22:25:17.806+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:17.806988+0000) 2022-01-31T22:25:17.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:17 smithi167 conmon[60316]: debug 2022-01-31T22:25:17.561+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:17.563224+0000) 2022-01-31T22:25:17.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:17 smithi171 conmon[41853]: debug 2022-01-31T22:25:17.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:17.510764+0000) 2022-01-31T22:25:18.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:18 smithi171 conmon[46715]: debug 2022-01-31T22:25:18.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:18.303211+0000) 2022-01-31T22:25:18.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:18 smithi171 conmon[51620]: debug 2022-01-31T22:25:18.162+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:18.162790+0000) 2022-01-31T22:25:18.856 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:18 smithi167 conmon[49112]: debug 2022-01-31T22:25:18.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:18.636697+0000) 2022-01-31T22:25:18.857 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:18 smithi167 conmon[60316]: debug 2022-01-31T22:25:18.562+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:18.563383+0000) 2022-01-31T22:25:18.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:18 smithi167 conmon[54076]: debug 2022-01-31T22:25:18.806+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:18.807164+0000) 2022-01-31T22:25:18.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:18 smithi171 conmon[41853]: debug 2022-01-31T22:25:18.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:18.510925+0000) 2022-01-31T22:25:19.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[46715]: debug 2022-01-31T22:25:19.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.303387+0000) 2022-01-31T22:25:19.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[51620]: debug 2022-01-31T22:25:19.162+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.163000+0000) 2022-01-31T22:25:19.563 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:19 smithi167 conmon[49112]: debug 2022-01-31T22:25:19.488+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.490336+0000) 2022-01-31T22:25:19.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:19 smithi167 conmon[54076]: debug 2022-01-31T22:25:19.488+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.490234+0000) 2022-01-31T22:25:19.564 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:19 smithi167 conmon[60316]: debug 2022-01-31T22:25:19.488+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.489609+0000) 2022-01-31T22:25:19.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:19 smithi167 conmon[49112]: debug 2022-01-31T22:25:19.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.636895+0000) 2022-01-31T22:25:19.857 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:19 smithi167 conmon[54076]: debug 2022-01-31T22:25:19.806+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.807338+0000) 2022-01-31T22:25:19.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:19 smithi167 conmon[60316]: debug 2022-01-31T22:25:19.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.563569+0000) 2022-01-31T22:25:19.859 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[35325]: debug 2022-01-31T22:25:19.508+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:19.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[46715]: debug 2022-01-31T22:25:19.489+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.490310+0000) 2022-01-31T22:25:19.860 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[51620]: debug 2022-01-31T22:25:19.489+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.490020+0000) 2022-01-31T22:25:19.860 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[41853]: debug 2022-01-31T22:25:19.488+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.489338+0000) 2022-01-31T22:25:19.861 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:19 smithi171 conmon[41853]: debug 2022-01-31T22:25:19.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:19.511053+0000) 2022-01-31T22:25:20.469 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:20 smithi171 conmon[46715]: debug 2022-01-31T22:25:20.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:20.303570+0000) 2022-01-31T22:25:20.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:20 smithi171 conmon[51620]: debug 2022-01-31T22:25:20.162+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:20.163215+0000) 2022-01-31T22:25:20.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:20 smithi167 conmon[49112]: debug 2022-01-31T22:25:20.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:20.637048+0000) 2022-01-31T22:25:20.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:20 smithi167 conmon[60316]: debug 2022-01-31T22:25:20.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:20.563731+0000) 2022-01-31T22:25:20.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:20 smithi167 conmon[54076]: debug 2022-01-31T22:25:20.807+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:20.807496+0000) 2022-01-31T22:25:20.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:20 smithi171 conmon[41853]: debug 2022-01-31T22:25:20.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:20.511193+0000) 2022-01-31T22:25:21.469 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:21 smithi171 conmon[46715]: debug 2022-01-31T22:25:21.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:21.303744+0000) 2022-01-31T22:25:21.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:21 smithi171 conmon[51620]: debug 2022-01-31T22:25:21.162+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:21.163378+0000) 2022-01-31T22:25:21.730 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:21 smithi171 conmon[41853]: debug 2022-01-31T22:25:21.510+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:21.511374+0000) 2022-01-31T22:25:21.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:21 smithi167 conmon[49112]: debug 2022-01-31T22:25:21.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:21.637193+0000) 2022-01-31T22:25:21.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:21 smithi167 conmon[54076]: debug 2022-01-31T22:25:21.807+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:21.807690+0000) 2022-01-31T22:25:21.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:21 smithi167 conmon[60316]: debug 2022-01-31T22:25:21.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:21.563867+0000) 2022-01-31T22:25:22.268 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:22 smithi171 conmon[51620]: debug 2022-01-31T22:25:22.163+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:22.163589+0000) 2022-01-31T22:25:22.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:22 smithi171 conmon[41853]: debug 2022-01-31T22:25:22.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:22.511527+0000) 2022-01-31T22:25:22.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:22 smithi171 conmon[46715]: debug 2022-01-31T22:25:22.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:22.303972+0000) 2022-01-31T22:25:22.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:22 smithi167 conmon[49112]: debug 2022-01-31T22:25:22.636+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:22.637367+0000) 2022-01-31T22:25:22.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:22 smithi167 conmon[54076]: debug 2022-01-31T22:25:22.807+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:22.807902+0000) 2022-01-31T22:25:22.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:22 smithi167 conmon[60316]: debug 2022-01-31T22:25:22.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:22.564080+0000) 2022-01-31T22:25:23.469 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:23 smithi171 conmon[46715]: debug 2022-01-31T22:25:23.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:23.304165+0000) 2022-01-31T22:25:23.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:23 smithi171 conmon[51620]: debug 2022-01-31T22:25:23.163+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:23.163795+0000) 2022-01-31T22:25:23.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:23 smithi171 conmon[41853]: debug 2022-01-31T22:25:23.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:23.511742+0000) 2022-01-31T22:25:23.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:23 smithi167 conmon[49112]: debug 2022-01-31T22:25:23.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:23.637573+0000) 2022-01-31T22:25:23.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:23 smithi167 conmon[54076]: debug 2022-01-31T22:25:23.807+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:23.808064+0000) 2022-01-31T22:25:23.858 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:23 smithi167 conmon[60316]: debug 2022-01-31T22:25:23.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:23.564207+0000) 2022-01-31T22:25:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:23 smithi171 conmon[35325]: debug 2022-01-31T22:25:23.850+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:25:24.469 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[46715]: debug 2022-01-31T22:25:24.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.304353+0000) 2022-01-31T22:25:24.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[51620]: debug 2022-01-31T22:25:24.163+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.163979+0000) 2022-01-31T22:25:24.563 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:24 smithi167 conmon[49112]: debug 2022-01-31T22:25:24.509+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.513203+0000) 2022-01-31T22:25:24.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:24 smithi167 conmon[54076]: debug 2022-01-31T22:25:24.518+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.519317+0000) 2022-01-31T22:25:24.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:24 smithi167 conmon[60316]: debug 2022-01-31T22:25:24.509+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.511493+0000) 2022-01-31T22:25:24.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[35325]: debug 2022-01-31T22:25:24.532+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208236 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:24.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[41853]: debug 2022-01-31T22:25:24.510+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.511707+0000) 2022-01-31T22:25:24.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[41853]: debug 2022-01-31T22:25:24.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.511891+0000) 2022-01-31T22:25:24.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[46715]: debug 2022-01-31T22:25:24.511+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.512328+0000) 2022-01-31T22:25:24.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:24 smithi171 conmon[51620]: debug 2022-01-31T22:25:24.510+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.511464+0000) 2022-01-31T22:25:24.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:24 smithi167 conmon[49112]: debug 2022-01-31T22:25:24.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.637735+0000) 2022-01-31T22:25:24.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:24 smithi167 conmon[54076]: debug 2022-01-31T22:25:24.807+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.808256+0000) 2022-01-31T22:25:24.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:24 smithi167 conmon[60316]: debug 2022-01-31T22:25:24.563+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:24.564354+0000) 2022-01-31T22:25:25.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:25 smithi171 conmon[46715]: debug 2022-01-31T22:25:25.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:25.304558+0000) 2022-01-31T22:25:25.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:25 smithi171 conmon[51620]: debug 2022-01-31T22:25:25.163+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:25.164180+0000) 2022-01-31T22:25:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:25 smithi171 conmon[41853]: debug 2022-01-31T22:25:25.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:25.512038+0000) 2022-01-31T22:25:25.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:25 smithi167 conmon[49112]: debug 2022-01-31T22:25:25.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:25.637874+0000) 2022-01-31T22:25:25.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:25 smithi167 conmon[54076]: debug 2022-01-31T22:25:25.807+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:25.808382+0000) 2022-01-31T22:25:25.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:25 smithi167 conmon[60316]: debug 2022-01-31T22:25:25.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:25.564495+0000) 2022-01-31T22:25:26.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:26 smithi171 conmon[46715]: debug 2022-01-31T22:25:26.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:26.304691+0000) 2022-01-31T22:25:26.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:26 smithi171 conmon[51620]: debug 2022-01-31T22:25:26.164+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:26.164348+0000) 2022-01-31T22:25:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:26 smithi171 conmon[41853]: debug 2022-01-31T22:25:26.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:26.512187+0000) 2022-01-31T22:25:26.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:26 smithi167 conmon[49112]: debug 2022-01-31T22:25:26.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:26.638045+0000) 2022-01-31T22:25:26.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:26 smithi167 conmon[54076]: debug 2022-01-31T22:25:26.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:26.808530+0000) 2022-01-31T22:25:26.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:26 smithi167 conmon[60316]: debug 2022-01-31T22:25:26.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:26.564705+0000) 2022-01-31T22:25:27.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:27 smithi171 conmon[46715]: debug 2022-01-31T22:25:27.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:27.304864+0000) 2022-01-31T22:25:27.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:27 smithi171 conmon[51620]: debug 2022-01-31T22:25:27.164+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:27.164565+0000) 2022-01-31T22:25:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:27 smithi171 conmon[41853]: debug 2022-01-31T22:25:27.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:27.512403+0000) 2022-01-31T22:25:27.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:27 smithi167 conmon[54076]: debug 2022-01-31T22:25:27.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:27.808660+0000) 2022-01-31T22:25:27.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:27 smithi167 conmon[49112]: debug 2022-01-31T22:25:27.637+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:27.638240+0000) 2022-01-31T22:25:27.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:27 smithi167 conmon[60316]: debug 2022-01-31T22:25:27.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:27.564892+0000) 2022-01-31T22:25:28.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:28 smithi171 conmon[46715]: debug 2022-01-31T22:25:28.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:28.305015+0000) 2022-01-31T22:25:28.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:28 smithi171 conmon[51620]: debug 2022-01-31T22:25:28.164+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:28.164785+0000) 2022-01-31T22:25:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:28 smithi171 conmon[41853]: debug 2022-01-31T22:25:28.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:28.512577+0000) 2022-01-31T22:25:28.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:28 smithi167 conmon[49112]: debug 2022-01-31T22:25:28.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:28.638431+0000) 2022-01-31T22:25:28.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:28 smithi167 conmon[54076]: debug 2022-01-31T22:25:28.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:28.808830+0000) 2022-01-31T22:25:28.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:28 smithi167 conmon[60316]: debug 2022-01-31T22:25:28.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:28.565057+0000) 2022-01-31T22:25:29.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[46715]: debug 2022-01-31T22:25:29.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.305205+0000) 2022-01-31T22:25:29.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[51620]: debug 2022-01-31T22:25:29.164+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.164989+0000) 2022-01-31T22:25:29.564 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:29 smithi167 conmon[49112]: debug 2022-01-31T22:25:29.536+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.536561+0000) 2022-01-31T22:25:29.565 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:29 smithi167 conmon[54076]: debug 2022-01-31T22:25:29.535+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.536253+0000) 2022-01-31T22:25:29.565 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:29 smithi167 conmon[60316]: debug 2022-01-31T22:25:29.536+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.536892+0000) 2022-01-31T22:25:29.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[35325]: debug 2022-01-31T22:25:29.555+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208344 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:29.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[41853]: debug 2022-01-31T22:25:29.511+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.512715+0000) 2022-01-31T22:25:29.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[41853]: debug 2022-01-31T22:25:29.535+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.536188+0000) 2022-01-31T22:25:29.860 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[46715]: debug 2022-01-31T22:25:29.534+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.535554+0000) 2022-01-31T22:25:29.861 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:29 smithi171 conmon[51620]: debug 2022-01-31T22:25:29.535+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.536430+0000) 2022-01-31T22:25:29.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:29 smithi167 conmon[49112]: debug 2022-01-31T22:25:29.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.638634+0000) 2022-01-31T22:25:29.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:29 smithi167 conmon[54076]: debug 2022-01-31T22:25:29.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.808970+0000) 2022-01-31T22:25:29.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:29 smithi167 conmon[60316]: debug 2022-01-31T22:25:29.564+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:29.565213+0000) 2022-01-31T22:25:30.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:30 smithi171 conmon[46715]: debug 2022-01-31T22:25:30.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:30.305366+0000) 2022-01-31T22:25:30.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:30 smithi171 conmon[51620]: debug 2022-01-31T22:25:30.164+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:30.165220+0000) 2022-01-31T22:25:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:30 smithi171 conmon[41853]: debug 2022-01-31T22:25:30.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:30.512922+0000) 2022-01-31T22:25:30.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:30 smithi167 conmon[54076]: debug 2022-01-31T22:25:30.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:30.809133+0000) 2022-01-31T22:25:30.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:30 smithi167 conmon[60316]: debug 2022-01-31T22:25:30.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:30.565369+0000) 2022-01-31T22:25:30.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:30 smithi167 conmon[49112]: debug 2022-01-31T22:25:30.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:30.638770+0000) 2022-01-31T22:25:31.470 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:31 smithi171 conmon[46715]: debug 2022-01-31T22:25:31.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:31.305501+0000) 2022-01-31T22:25:31.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:31 smithi171 conmon[51620]: debug 2022-01-31T22:25:31.164+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:31.165386+0000) 2022-01-31T22:25:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:31 smithi171 conmon[41853]: debug 2022-01-31T22:25:31.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:31.513069+0000) 2022-01-31T22:25:31.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:31 smithi167 conmon[49112]: debug 2022-01-31T22:25:31.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:31.638885+0000) 2022-01-31T22:25:31.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:31 smithi167 conmon[54076]: debug 2022-01-31T22:25:31.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:31.809271+0000) 2022-01-31T22:25:31.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:31 smithi167 conmon[60316]: debug 2022-01-31T22:25:31.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:31.565598+0000) 2022-01-31T22:25:32.471 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:32 smithi171 conmon[46715]: debug 2022-01-31T22:25:32.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:32.305687+0000) 2022-01-31T22:25:32.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:32 smithi171 conmon[51620]: debug 2022-01-31T22:25:32.165+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:32.165591+0000) 2022-01-31T22:25:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:32 smithi171 conmon[41853]: debug 2022-01-31T22:25:32.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:32.513235+0000) 2022-01-31T22:25:32.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:32 smithi167 conmon[49112]: debug 2022-01-31T22:25:32.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:32.639068+0000) 2022-01-31T22:25:32.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:32 smithi167 conmon[60316]: debug 2022-01-31T22:25:32.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:32.565782+0000) 2022-01-31T22:25:32.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:32 smithi167 conmon[54076]: debug 2022-01-31T22:25:32.808+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:32.809396+0000) 2022-01-31T22:25:33.471 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:33 smithi171 conmon[46715]: debug 2022-01-31T22:25:33.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:33.305809+0000) 2022-01-31T22:25:33.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:33 smithi171 conmon[51620]: debug 2022-01-31T22:25:33.165+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:33.165839+0000) 2022-01-31T22:25:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:33 smithi171 conmon[41853]: debug 2022-01-31T22:25:33.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:33.513476+0000) 2022-01-31T22:25:33.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:33 smithi167 conmon[49112]: debug 2022-01-31T22:25:33.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:33.639271+0000) 2022-01-31T22:25:33.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:33 smithi167 conmon[60316]: debug 2022-01-31T22:25:33.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:33.565970+0000) 2022-01-31T22:25:33.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:33 smithi167 conmon[54076]: debug 2022-01-31T22:25:33.809+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:33.809566+0000) 2022-01-31T22:25:34.471 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[46715]: debug 2022-01-31T22:25:34.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.305987+0000) 2022-01-31T22:25:34.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[51620]: debug 2022-01-31T22:25:34.165+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.165994+0000) 2022-01-31T22:25:34.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:34 smithi167 conmon[49112]: debug 2022-01-31T22:25:34.559+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.560437+0000) 2022-01-31T22:25:34.809 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:34 smithi167 conmon[49112]: debug 2022-01-31T22:25:34.638+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.639421+0000) 2022-01-31T22:25:34.810 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:34 smithi167 conmon[54076]: debug 2022-01-31T22:25:34.559+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.559684+0000) 2022-01-31T22:25:34.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:34 smithi167 conmon[60316]: debug 2022-01-31T22:25:34.560+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.560648+0000) 2022-01-31T22:25:34.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:34 smithi167 conmon[60316]: debug 2022-01-31T22:25:34.565+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.566146+0000) 2022-01-31T22:25:34.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[35325]: debug 2022-01-31T22:25:34.579+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208458 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:34.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[46715]: debug 2022-01-31T22:25:34.558+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.559621+0000) 2022-01-31T22:25:34.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[51620]: debug 2022-01-31T22:25:34.559+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.560203+0000) 2022-01-31T22:25:34.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[41853]: debug 2022-01-31T22:25:34.512+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.513718+0000) 2022-01-31T22:25:34.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:34 smithi171 conmon[41853]: debug 2022-01-31T22:25:34.559+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.560000+0000) 2022-01-31T22:25:35.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:34 smithi167 conmon[54076]: debug 2022-01-31T22:25:34.809+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:34.809753+0000) 2022-01-31T22:25:35.471 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:35 smithi171 conmon[46715]: debug 2022-01-31T22:25:35.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:35.306191+0000) 2022-01-31T22:25:35.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:35 smithi171 conmon[51620]: debug 2022-01-31T22:25:35.166+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:35.166201+0000) 2022-01-31T22:25:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:35 smithi171 conmon[41853]: debug 2022-01-31T22:25:35.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:35.513971+0000) 2022-01-31T22:25:35.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:35 smithi167 conmon[49112]: debug 2022-01-31T22:25:35.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:35.639553+0000) 2022-01-31T22:25:35.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:35 smithi167 conmon[54076]: debug 2022-01-31T22:25:35.809+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:35.809868+0000) 2022-01-31T22:25:35.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:35 smithi167 conmon[60316]: debug 2022-01-31T22:25:35.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:35.566277+0000) 2022-01-31T22:25:36.471 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:36 smithi171 conmon[46715]: debug 2022-01-31T22:25:36.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:36.306400+0000) 2022-01-31T22:25:36.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:36 smithi171 conmon[51620]: debug 2022-01-31T22:25:36.166+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:36.166335+0000) 2022-01-31T22:25:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:36 smithi171 conmon[41853]: debug 2022-01-31T22:25:36.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:36.514142+0000) 2022-01-31T22:25:36.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:36 smithi167 conmon[49112]: debug 2022-01-31T22:25:36.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:36.639707+0000) 2022-01-31T22:25:36.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:36 smithi167 conmon[54076]: debug 2022-01-31T22:25:36.809+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:36.810039+0000) 2022-01-31T22:25:36.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:36 smithi167 conmon[60316]: debug 2022-01-31T22:25:36.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:36.566406+0000) 2022-01-31T22:25:37.472 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:37 smithi171 conmon[46715]: debug 2022-01-31T22:25:37.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:37.306546+0000) 2022-01-31T22:25:37.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:37 smithi171 conmon[51620]: debug 2022-01-31T22:25:37.165+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:37.166539+0000) 2022-01-31T22:25:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:37 smithi171 conmon[41853]: debug 2022-01-31T22:25:37.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:37.514294+0000) 2022-01-31T22:25:37.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:37 smithi167 conmon[49112]: debug 2022-01-31T22:25:37.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:37.639830+0000) 2022-01-31T22:25:37.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:37 smithi167 conmon[54076]: debug 2022-01-31T22:25:37.809+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:37.810220+0000) 2022-01-31T22:25:37.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:37 smithi167 conmon[60316]: debug 2022-01-31T22:25:37.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:37.566535+0000) 2022-01-31T22:25:38.472 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:38 smithi171 conmon[46715]: debug 2022-01-31T22:25:38.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:38.306723+0000) 2022-01-31T22:25:38.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:38 smithi171 conmon[51620]: debug 2022-01-31T22:25:38.166+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:38.166748+0000) 2022-01-31T22:25:38.851 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:38 smithi171 conmon[41853]: debug 2022-01-31T22:25:38.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:38.514459+0000) 2022-01-31T22:25:38.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:38 smithi167 conmon[49112]: debug 2022-01-31T22:25:38.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:38.640017+0000) 2022-01-31T22:25:38.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:38 smithi167 conmon[60316]: debug 2022-01-31T22:25:38.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:38.566719+0000) 2022-01-31T22:25:38.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:38 smithi167 conmon[54076]: debug 2022-01-31T22:25:38.809+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:38.810401+0000) 2022-01-31T22:25:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:38 smithi171 conmon[35325]: debug 2022-01-31T22:25:38.851+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:25:39.472 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[46715]: debug 2022-01-31T22:25:39.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.306871+0000) 2022-01-31T22:25:39.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[51620]: debug 2022-01-31T22:25:39.166+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.166984+0000) 2022-01-31T22:25:39.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[46715]: debug 2022-01-31T22:25:39.583+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.584654+0000) 2022-01-31T22:25:39.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[35325]: debug 2022-01-31T22:25:39.604+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208581 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:39.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[41853]: debug 2022-01-31T22:25:39.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.514621+0000) 2022-01-31T22:25:39.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[41853]: debug 2022-01-31T22:25:39.583+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.584332+0000) 2022-01-31T22:25:39.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:39 smithi171 conmon[51620]: debug 2022-01-31T22:25:39.583+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.584707+0000) 2022-01-31T22:25:39.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:39 smithi167 conmon[49112]: debug 2022-01-31T22:25:39.583+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.584323+0000) 2022-01-31T22:25:39.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:39 smithi167 conmon[49112]: debug 2022-01-31T22:25:39.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.640215+0000) 2022-01-31T22:25:39.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:39 smithi167 conmon[54076]: debug 2022-01-31T22:25:39.583+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.583698+0000) 2022-01-31T22:25:39.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:39 smithi167 conmon[54076]: debug 2022-01-31T22:25:39.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.810606+0000) 2022-01-31T22:25:39.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:39 smithi167 conmon[60316]: debug 2022-01-31T22:25:39.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.566894+0000) 2022-01-31T22:25:39.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:39 smithi167 conmon[60316]: debug 2022-01-31T22:25:39.583+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:39.583839+0000) 2022-01-31T22:25:40.472 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:40 smithi171 conmon[46715]: debug 2022-01-31T22:25:40.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:40.307027+0000) 2022-01-31T22:25:40.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:40 smithi171 conmon[51620]: debug 2022-01-31T22:25:40.165+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:40.167164+0000) 2022-01-31T22:25:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:40 smithi171 conmon[41853]: debug 2022-01-31T22:25:40.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:40.514747+0000) 2022-01-31T22:25:40.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:40 smithi167 conmon[49112]: debug 2022-01-31T22:25:40.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:40.640392+0000) 2022-01-31T22:25:40.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:40 smithi167 conmon[54076]: debug 2022-01-31T22:25:40.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:40.810817+0000) 2022-01-31T22:25:40.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:40 smithi167 conmon[60316]: debug 2022-01-31T22:25:40.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:40.567078+0000) 2022-01-31T22:25:41.472 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:41 smithi171 conmon[46715]: debug 2022-01-31T22:25:41.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:41.307179+0000) 2022-01-31T22:25:41.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:41 smithi171 conmon[51620]: debug 2022-01-31T22:25:41.166+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:41.167308+0000) 2022-01-31T22:25:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:41 smithi171 conmon[41853]: debug 2022-01-31T22:25:41.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:41.514890+0000) 2022-01-31T22:25:41.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:41 smithi167 conmon[49112]: debug 2022-01-31T22:25:41.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:41.640577+0000) 2022-01-31T22:25:41.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:41 smithi167 conmon[54076]: debug 2022-01-31T22:25:41.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:41.811024+0000) 2022-01-31T22:25:41.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:41 smithi167 conmon[60316]: debug 2022-01-31T22:25:41.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:41.567266+0000) 2022-01-31T22:25:42.472 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:42 smithi171 conmon[46715]: debug 2022-01-31T22:25:42.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:42.307364+0000) 2022-01-31T22:25:42.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:42 smithi171 conmon[51620]: debug 2022-01-31T22:25:42.166+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:42.167528+0000) 2022-01-31T22:25:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:42 smithi171 conmon[41853]: debug 2022-01-31T22:25:42.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:42.515053+0000) 2022-01-31T22:25:42.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:42 smithi167 conmon[49112]: debug 2022-01-31T22:25:42.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:42.640724+0000) 2022-01-31T22:25:42.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:42 smithi167 conmon[54076]: debug 2022-01-31T22:25:42.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:42.811216+0000) 2022-01-31T22:25:42.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:42 smithi167 conmon[60316]: debug 2022-01-31T22:25:42.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:42.567474+0000) 2022-01-31T22:25:43.473 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:43 smithi171 conmon[46715]: debug 2022-01-31T22:25:43.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:43.307546+0000) 2022-01-31T22:25:43.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:43 smithi171 conmon[51620]: debug 2022-01-31T22:25:43.165+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:43.167698+0000) 2022-01-31T22:25:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:43 smithi171 conmon[41853]: debug 2022-01-31T22:25:43.513+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:43.515241+0000) 2022-01-31T22:25:43.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:43 smithi167 conmon[49112]: debug 2022-01-31T22:25:43.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:43.640916+0000) 2022-01-31T22:25:43.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:43 smithi167 conmon[54076]: debug 2022-01-31T22:25:43.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:43.811382+0000) 2022-01-31T22:25:43.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:43 smithi167 conmon[60316]: debug 2022-01-31T22:25:43.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:43.567638+0000) 2022-01-31T22:25:44.473 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[46715]: debug 2022-01-31T22:25:44.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.307775+0000) 2022-01-31T22:25:44.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[51620]: debug 2022-01-31T22:25:44.167+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.167956+0000) 2022-01-31T22:25:44.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[35325]: debug 2022-01-31T22:25:44.627+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208696 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:44.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[46715]: debug 2022-01-31T22:25:44.607+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.609231+0000) 2022-01-31T22:25:44.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[51620]: debug 2022-01-31T22:25:44.606+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.608112+0000) 2022-01-31T22:25:44.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[41853]: debug 2022-01-31T22:25:44.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.515435+0000) 2022-01-31T22:25:44.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:44 smithi171 conmon[41853]: debug 2022-01-31T22:25:44.606+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.608398+0000) 2022-01-31T22:25:44.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:44 smithi167 conmon[49112]: debug 2022-01-31T22:25:44.608+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.608637+0000) 2022-01-31T22:25:44.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:44 smithi167 conmon[49112]: debug 2022-01-31T22:25:44.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.641081+0000) 2022-01-31T22:25:44.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:44 smithi167 conmon[54076]: debug 2022-01-31T22:25:44.607+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.607810+0000) 2022-01-31T22:25:44.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:44 smithi167 conmon[54076]: debug 2022-01-31T22:25:44.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.811559+0000) 2022-01-31T22:25:44.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:44 smithi167 conmon[60316]: debug 2022-01-31T22:25:44.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.567861+0000) 2022-01-31T22:25:44.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:44 smithi167 conmon[60316]: debug 2022-01-31T22:25:44.607+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:44.607657+0000) 2022-01-31T22:25:45.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:45 smithi171 conmon[51620]: debug 2022-01-31T22:25:45.167+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:45.168181+0000) 2022-01-31T22:25:45.473 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:45 smithi171 conmon[46715]: debug 2022-01-31T22:25:45.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:45.307912+0000) 2022-01-31T22:25:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:45 smithi171 conmon[41853]: debug 2022-01-31T22:25:45.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:45.515621+0000) 2022-01-31T22:25:45.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:45 smithi167 conmon[49112]: debug 2022-01-31T22:25:45.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:45.641224+0000) 2022-01-31T22:25:45.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:45 smithi167 conmon[54076]: debug 2022-01-31T22:25:45.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:45.811736+0000) 2022-01-31T22:25:45.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:45 smithi167 conmon[60316]: debug 2022-01-31T22:25:45.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:45.568057+0000) 2022-01-31T22:25:46.473 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:46 smithi171 conmon[46715]: debug 2022-01-31T22:25:46.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:46.308054+0000) 2022-01-31T22:25:46.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:46 smithi171 conmon[51620]: debug 2022-01-31T22:25:46.167+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:46.168328+0000) 2022-01-31T22:25:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:46 smithi171 conmon[41853]: debug 2022-01-31T22:25:46.514+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:46.515788+0000) 2022-01-31T22:25:46.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:46 smithi167 conmon[49112]: debug 2022-01-31T22:25:46.639+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:46.641382+0000) 2022-01-31T22:25:46.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:46 smithi167 conmon[54076]: debug 2022-01-31T22:25:46.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:46.811881+0000) 2022-01-31T22:25:46.862 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:46 smithi167 conmon[60316]: debug 2022-01-31T22:25:46.566+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:46.568183+0000) 2022-01-31T22:25:47.473 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:47 smithi171 conmon[46715]: debug 2022-01-31T22:25:47.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:47.308240+0000) 2022-01-31T22:25:47.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:47 smithi171 conmon[51620]: debug 2022-01-31T22:25:47.167+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:47.168511+0000) 2022-01-31T22:25:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:47 smithi171 conmon[41853]: debug 2022-01-31T22:25:47.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:47.515999+0000) 2022-01-31T22:25:47.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:47 smithi167 conmon[49112]: debug 2022-01-31T22:25:47.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:47.641579+0000) 2022-01-31T22:25:47.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:47 smithi167 conmon[54076]: debug 2022-01-31T22:25:47.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:47.812028+0000) 2022-01-31T22:25:47.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:47 smithi167 conmon[60316]: debug 2022-01-31T22:25:47.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:47.568347+0000) 2022-01-31T22:25:48.473 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:48 smithi171 conmon[46715]: debug 2022-01-31T22:25:48.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:48.308419+0000) 2022-01-31T22:25:48.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:48 smithi171 conmon[51620]: debug 2022-01-31T22:25:48.167+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:48.168708+0000) 2022-01-31T22:25:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:48 smithi171 conmon[41853]: debug 2022-01-31T22:25:48.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:48.516157+0000) 2022-01-31T22:25:48.861 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:48 smithi167 conmon[49112]: debug 2022-01-31T22:25:48.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:48.641752+0000) 2022-01-31T22:25:48.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:48 smithi167 conmon[54076]: debug 2022-01-31T22:25:48.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:48.812233+0000) 2022-01-31T22:25:48.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:48 smithi167 conmon[60316]: debug 2022-01-31T22:25:48.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:48.568582+0000) 2022-01-31T22:25:49.474 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[46715]: debug 2022-01-31T22:25:49.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.308571+0000) 2022-01-31T22:25:49.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[51620]: debug 2022-01-31T22:25:49.167+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.168912+0000) 2022-01-31T22:25:49.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[35325]: debug 2022-01-31T22:25:49.652+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208805 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:49.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[46715]: debug 2022-01-31T22:25:49.630+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.632522+0000) 2022-01-31T22:25:49.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[51620]: debug 2022-01-31T22:25:49.629+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.631403+0000) 2022-01-31T22:25:49.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[41853]: debug 2022-01-31T22:25:49.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.516317+0000) 2022-01-31T22:25:49.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:49 smithi171 conmon[41853]: debug 2022-01-31T22:25:49.629+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.630897+0000) 2022-01-31T22:25:49.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:49 smithi167 conmon[49112]: debug 2022-01-31T22:25:49.631+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.633250+0000) 2022-01-31T22:25:49.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:49 smithi167 conmon[49112]: debug 2022-01-31T22:25:49.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.641910+0000) 2022-01-31T22:25:49.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:49 smithi167 conmon[54076]: debug 2022-01-31T22:25:49.629+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.631227+0000) 2022-01-31T22:25:49.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:49 smithi167 conmon[54076]: debug 2022-01-31T22:25:49.810+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.812388+0000) 2022-01-31T22:25:49.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:49 smithi167 conmon[60316]: debug 2022-01-31T22:25:49.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.568752+0000) 2022-01-31T22:25:49.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:49 smithi167 conmon[60316]: debug 2022-01-31T22:25:49.629+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:49.631480+0000) 2022-01-31T22:25:50.474 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:50 smithi171 conmon[46715]: debug 2022-01-31T22:25:50.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:50.308772+0000) 2022-01-31T22:25:50.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:50 smithi171 conmon[51620]: debug 2022-01-31T22:25:50.168+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:50.169058+0000) 2022-01-31T22:25:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:50 smithi171 conmon[41853]: debug 2022-01-31T22:25:50.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:50.516519+0000) 2022-01-31T22:25:50.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:50 smithi167 conmon[49112]: debug 2022-01-31T22:25:50.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:50.642056+0000) 2022-01-31T22:25:50.862 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:50 smithi167 conmon[54076]: debug 2022-01-31T22:25:50.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:50.812570+0000) 2022-01-31T22:25:50.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:50 smithi167 conmon[60316]: debug 2022-01-31T22:25:50.567+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:50.568940+0000) 2022-01-31T22:25:51.474 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:51 smithi171 conmon[46715]: debug 2022-01-31T22:25:51.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:51.308923+0000) 2022-01-31T22:25:51.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:51 smithi171 conmon[51620]: debug 2022-01-31T22:25:51.168+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:51.169213+0000) 2022-01-31T22:25:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:51 smithi171 conmon[41853]: debug 2022-01-31T22:25:51.515+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:51.516669+0000) 2022-01-31T22:25:51.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:51 smithi167 conmon[49112]: debug 2022-01-31T22:25:51.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:51.642241+0000) 2022-01-31T22:25:51.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:51 smithi167 conmon[54076]: debug 2022-01-31T22:25:51.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:51.812753+0000) 2022-01-31T22:25:51.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:51 smithi167 conmon[60316]: debug 2022-01-31T22:25:51.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:51.569092+0000) 2022-01-31T22:25:52.474 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:52 smithi171 conmon[46715]: debug 2022-01-31T22:25:52.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:52.309102+0000) 2022-01-31T22:25:52.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:52 smithi171 conmon[51620]: debug 2022-01-31T22:25:52.168+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:52.169422+0000) 2022-01-31T22:25:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:52 smithi171 conmon[41853]: debug 2022-01-31T22:25:52.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:52.516822+0000) 2022-01-31T22:25:52.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:52 smithi167 conmon[49112]: debug 2022-01-31T22:25:52.640+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:52.642385+0000) 2022-01-31T22:25:52.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:52 smithi167 conmon[54076]: debug 2022-01-31T22:25:52.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:52.812934+0000) 2022-01-31T22:25:52.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:52 smithi167 conmon[60316]: debug 2022-01-31T22:25:52.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:52.569280+0000) 2022-01-31T22:25:53.474 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:53 smithi171 conmon[46715]: debug 2022-01-31T22:25:53.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:53.309305+0000) 2022-01-31T22:25:53.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:53 smithi171 conmon[51620]: debug 2022-01-31T22:25:53.168+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:53.169652+0000) 2022-01-31T22:25:53.851 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:53 smithi171 conmon[41853]: debug 2022-01-31T22:25:53.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:53.517035+0000) 2022-01-31T22:25:53.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:53 smithi167 conmon[49112]: debug 2022-01-31T22:25:53.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:53.642592+0000) 2022-01-31T22:25:53.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:53 smithi167 conmon[54076]: debug 2022-01-31T22:25:53.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:53.813141+0000) 2022-01-31T22:25:53.863 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:53 smithi167 conmon[60316]: debug 2022-01-31T22:25:53.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:53.569472+0000) 2022-01-31T22:25:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:53 smithi171 conmon[35325]: debug 2022-01-31T22:25:53.851+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:25:54.475 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[46715]: debug 2022-01-31T22:25:54.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.309505+0000) 2022-01-31T22:25:54.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[51620]: debug 2022-01-31T22:25:54.169+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.169866+0000) 2022-01-31T22:25:54.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[35325]: debug 2022-01-31T22:25:54.691+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 208918 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:54.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[46715]: debug 2022-01-31T22:25:54.655+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.656322+0000) 2022-01-31T22:25:54.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[51620]: debug 2022-01-31T22:25:54.655+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.656594+0000) 2022-01-31T22:25:54.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[41853]: debug 2022-01-31T22:25:54.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.517232+0000) 2022-01-31T22:25:54.860 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:54 smithi171 conmon[41853]: debug 2022-01-31T22:25:54.655+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.655970+0000) 2022-01-31T22:25:54.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:54 smithi167 conmon[54076]: debug 2022-01-31T22:25:54.654+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.656312+0000) 2022-01-31T22:25:54.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:54 smithi167 conmon[54076]: debug 2022-01-31T22:25:54.812+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.813301+0000) 2022-01-31T22:25:54.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:54 smithi167 conmon[60316]: debug 2022-01-31T22:25:54.568+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.569681+0000) 2022-01-31T22:25:54.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:54 smithi167 conmon[60316]: debug 2022-01-31T22:25:54.663+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.664612+0000) 2022-01-31T22:25:54.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:54 smithi167 conmon[49112]: debug 2022-01-31T22:25:54.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.642737+0000) 2022-01-31T22:25:54.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:54 smithi167 conmon[49112]: debug 2022-01-31T22:25:54.662+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:54.663821+0000) 2022-01-31T22:25:55.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:55 smithi171 conmon[51620]: debug 2022-01-31T22:25:55.169+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:55.170106+0000) 2022-01-31T22:25:55.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:55 smithi171 conmon[46715]: debug 2022-01-31T22:25:55.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:55.309689+0000) 2022-01-31T22:25:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:55 smithi171 conmon[41853]: debug 2022-01-31T22:25:55.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:55.517388+0000) 2022-01-31T22:25:55.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:55 smithi167 conmon[49112]: debug 2022-01-31T22:25:55.641+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:55.642889+0000) 2022-01-31T22:25:55.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:55 smithi167 conmon[54076]: debug 2022-01-31T22:25:55.811+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:55.813438+0000) 2022-01-31T22:25:55.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:55 smithi167 conmon[60316]: debug 2022-01-31T22:25:55.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:55.569828+0000) 2022-01-31T22:25:56.475 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:56 smithi171 conmon[46715]: debug 2022-01-31T22:25:56.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:56.309840+0000) 2022-01-31T22:25:56.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:56 smithi171 conmon[51620]: debug 2022-01-31T22:25:56.169+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:56.170291+0000) 2022-01-31T22:25:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:56 smithi171 conmon[41853]: debug 2022-01-31T22:25:56.516+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:56.517521+0000) 2022-01-31T22:25:56.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:56 smithi167 conmon[49112]: debug 2022-01-31T22:25:56.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:56.643046+0000) 2022-01-31T22:25:56.863 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:56 smithi167 conmon[54076]: debug 2022-01-31T22:25:56.812+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:56.813571+0000) 2022-01-31T22:25:56.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:56 smithi167 conmon[60316]: debug 2022-01-31T22:25:56.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:56.569966+0000) 2022-01-31T22:25:57.475 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:57 smithi171 conmon[46715]: debug 2022-01-31T22:25:57.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:57.310051+0000) 2022-01-31T22:25:57.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:57 smithi171 conmon[51620]: debug 2022-01-31T22:25:57.169+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:57.170508+0000) 2022-01-31T22:25:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:57 smithi171 conmon[41853]: debug 2022-01-31T22:25:57.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:57.517673+0000) 2022-01-31T22:25:57.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:57 smithi167 conmon[49112]: debug 2022-01-31T22:25:57.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:57.643231+0000) 2022-01-31T22:25:57.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:57 smithi167 conmon[54076]: debug 2022-01-31T22:25:57.812+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:57.813755+0000) 2022-01-31T22:25:57.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:57 smithi167 conmon[60316]: debug 2022-01-31T22:25:57.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:57.570124+0000) 2022-01-31T22:25:58.475 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:58 smithi171 conmon[46715]: debug 2022-01-31T22:25:58.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:58.310229+0000) 2022-01-31T22:25:58.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:58 smithi171 conmon[51620]: debug 2022-01-31T22:25:58.170+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:58.170699+0000) 2022-01-31T22:25:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:58 smithi171 conmon[41853]: debug 2022-01-31T22:25:58.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:58.517856+0000) 2022-01-31T22:25:58.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:58 smithi167 conmon[49112]: debug 2022-01-31T22:25:58.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:58.643438+0000) 2022-01-31T22:25:58.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:58 smithi167 conmon[54076]: debug 2022-01-31T22:25:58.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:58.813909+0000) 2022-01-31T22:25:58.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:58 smithi167 conmon[60316]: debug 2022-01-31T22:25:58.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:58.570309+0000) 2022-01-31T22:25:59.475 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[46715]: debug 2022-01-31T22:25:59.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.310419+0000) 2022-01-31T22:25:59.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[51620]: debug 2022-01-31T22:25:59.170+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.170935+0000) 2022-01-31T22:25:59.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[35325]: debug 2022-01-31T22:25:59.715+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209027 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:25:59.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[46715]: debug 2022-01-31T22:25:59.692+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.693567+0000) 2022-01-31T22:25:59.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[51620]: debug 2022-01-31T22:25:59.693+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.694352+0000) 2022-01-31T22:25:59.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[41853]: debug 2022-01-31T22:25:59.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.517990+0000) 2022-01-31T22:25:59.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:25:59 smithi171 conmon[41853]: debug 2022-01-31T22:25:59.693+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.694548+0000) 2022-01-31T22:25:59.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:59 smithi167 conmon[49112]: debug 2022-01-31T22:25:59.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.643681+0000) 2022-01-31T22:25:59.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:25:59 smithi167 conmon[49112]: debug 2022-01-31T22:25:59.693+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.695251+0000) 2022-01-31T22:25:59.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:59 smithi167 conmon[54076]: debug 2022-01-31T22:25:59.693+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.694939+0000) 2022-01-31T22:25:59.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:25:59 smithi167 conmon[54076]: debug 2022-01-31T22:25:59.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.814089+0000) 2022-01-31T22:25:59.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:59 smithi167 conmon[60316]: debug 2022-01-31T22:25:59.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.570550+0000) 2022-01-31T22:25:59.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:25:59 smithi167 conmon[60316]: debug 2022-01-31T22:25:59.693+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:25:59.695378+0000) 2022-01-31T22:26:00.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:00 smithi171 conmon[46715]: debug 2022-01-31T22:26:00.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:00.310584+0000) 2022-01-31T22:26:00.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:00 smithi171 conmon[51620]: debug 2022-01-31T22:26:00.170+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:00.171150+0000) 2022-01-31T22:26:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:00 smithi171 conmon[41853]: debug 2022-01-31T22:26:00.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:00.518149+0000) 2022-01-31T22:26:00.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:00 smithi167 conmon[54076]: debug 2022-01-31T22:26:00.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:00.814299+0000) 2022-01-31T22:26:00.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:00 smithi167 conmon[49112]: debug 2022-01-31T22:26:00.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:00.643863+0000) 2022-01-31T22:26:00.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:00 smithi167 conmon[60316]: debug 2022-01-31T22:26:00.569+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:00.570733+0000) 2022-01-31T22:26:01.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:01 smithi171 conmon[46715]: debug 2022-01-31T22:26:01.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:01.310742+0000) 2022-01-31T22:26:01.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:01 smithi171 conmon[51620]: debug 2022-01-31T22:26:01.170+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:01.171320+0000) 2022-01-31T22:26:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:01 smithi171 conmon[41853]: debug 2022-01-31T22:26:01.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:01.518297+0000) 2022-01-31T22:26:01.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:01 smithi167 conmon[49112]: debug 2022-01-31T22:26:01.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:01.644012+0000) 2022-01-31T22:26:01.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:01 smithi167 conmon[54076]: debug 2022-01-31T22:26:01.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:01.814493+0000) 2022-01-31T22:26:01.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:01 smithi167 conmon[60316]: debug 2022-01-31T22:26:01.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:01.570887+0000) 2022-01-31T22:26:02.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:02 smithi171 conmon[46715]: debug 2022-01-31T22:26:02.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:02.310968+0000) 2022-01-31T22:26:02.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:02 smithi171 conmon[51620]: debug 2022-01-31T22:26:02.170+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:02.171557+0000) 2022-01-31T22:26:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:02 smithi171 conmon[41853]: debug 2022-01-31T22:26:02.517+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:02.518451+0000) 2022-01-31T22:26:02.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:02 smithi167 conmon[54076]: debug 2022-01-31T22:26:02.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:02.814667+0000) 2022-01-31T22:26:02.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:02 smithi167 conmon[60316]: debug 2022-01-31T22:26:02.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:02.571103+0000) 2022-01-31T22:26:02.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:02 smithi167 conmon[49112]: debug 2022-01-31T22:26:02.642+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:02.644218+0000) 2022-01-31T22:26:03.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:03 smithi171 conmon[46715]: debug 2022-01-31T22:26:03.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:03.311116+0000) 2022-01-31T22:26:03.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:03 smithi171 conmon[51620]: debug 2022-01-31T22:26:03.171+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:03.171799+0000) 2022-01-31T22:26:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:03 smithi171 conmon[41853]: debug 2022-01-31T22:26:03.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:03.518633+0000) 2022-01-31T22:26:03.864 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:03 smithi167 conmon[54076]: debug 2022-01-31T22:26:03.814+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:03.814879+0000) 2022-01-31T22:26:03.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:03 smithi167 conmon[60316]: debug 2022-01-31T22:26:03.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:03.571251+0000) 2022-01-31T22:26:03.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:03 smithi167 conmon[49112]: debug 2022-01-31T22:26:03.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:03.644418+0000) 2022-01-31T22:26:04.476 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[46715]: debug 2022-01-31T22:26:04.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.311265+0000) 2022-01-31T22:26:04.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[51620]: debug 2022-01-31T22:26:04.171+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.172030+0000) 2022-01-31T22:26:04.739 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[46715]: debug 2022-01-31T22:26:04.717+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.718170+0000) 2022-01-31T22:26:04.740 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[41853]: debug 2022-01-31T22:26:04.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.518820+0000) 2022-01-31T22:26:04.741 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[41853]: debug 2022-01-31T22:26:04.719+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.720005+0000) 2022-01-31T22:26:04.741 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[51620]: debug 2022-01-31T22:26:04.718+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.719347+0000) 2022-01-31T22:26:04.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:04 smithi167 conmon[49112]: debug 2022-01-31T22:26:04.643+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.644649+0000) 2022-01-31T22:26:04.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:04 smithi167 conmon[49112]: debug 2022-01-31T22:26:04.718+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.720457+0000) 2022-01-31T22:26:04.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:04 smithi167 conmon[54076]: debug 2022-01-31T22:26:04.718+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.720093+0000) 2022-01-31T22:26:04.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:04 smithi167 conmon[54076]: debug 2022-01-31T22:26:04.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.815065+0000) 2022-01-31T22:26:04.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:04 smithi167 conmon[60316]: debug 2022-01-31T22:26:04.570+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.571479+0000) 2022-01-31T22:26:04.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:04 smithi167 conmon[60316]: debug 2022-01-31T22:26:04.718+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:04.719951+0000) 2022-01-31T22:26:05.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:04 smithi171 conmon[35325]: debug 2022-01-31T22:26:04.739+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209140 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:05.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:05 smithi171 conmon[46715]: debug 2022-01-31T22:26:05.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:05.311440+0000) 2022-01-31T22:26:05.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:05 smithi171 conmon[51620]: debug 2022-01-31T22:26:05.171+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:05.172238+0000) 2022-01-31T22:26:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:05 smithi171 conmon[41853]: debug 2022-01-31T22:26:05.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:05.518999+0000) 2022-01-31T22:26:05.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:05 smithi167 conmon[49112]: debug 2022-01-31T22:26:05.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:05.644838+0000) 2022-01-31T22:26:05.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:05 smithi167 conmon[54076]: debug 2022-01-31T22:26:05.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:05.815162+0000) 2022-01-31T22:26:05.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:05 smithi167 conmon[60316]: debug 2022-01-31T22:26:05.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:05.571654+0000) 2022-01-31T22:26:06.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:06 smithi171 conmon[46715]: debug 2022-01-31T22:26:06.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:06.311598+0000) 2022-01-31T22:26:06.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:06 smithi171 conmon[51620]: debug 2022-01-31T22:26:06.171+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:06.172411+0000) 2022-01-31T22:26:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:06 smithi171 conmon[41853]: debug 2022-01-31T22:26:06.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:06.519131+0000) 2022-01-31T22:26:06.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:06 smithi167 conmon[49112]: debug 2022-01-31T22:26:06.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:06.645011+0000) 2022-01-31T22:26:06.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:06 smithi167 conmon[54076]: debug 2022-01-31T22:26:06.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:06.815265+0000) 2022-01-31T22:26:06.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:06 smithi167 conmon[60316]: debug 2022-01-31T22:26:06.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:06.571835+0000) 2022-01-31T22:26:07.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:07 smithi171 conmon[46715]: debug 2022-01-31T22:26:07.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:07.311802+0000) 2022-01-31T22:26:07.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:07 smithi171 conmon[51620]: debug 2022-01-31T22:26:07.172+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:07.172625+0000) 2022-01-31T22:26:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:07 smithi171 conmon[41853]: debug 2022-01-31T22:26:07.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:07.519308+0000) 2022-01-31T22:26:07.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:07 smithi167 conmon[49112]: debug 2022-01-31T22:26:07.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:07.645166+0000) 2022-01-31T22:26:07.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:07 smithi167 conmon[54076]: debug 2022-01-31T22:26:07.813+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:07.815365+0000) 2022-01-31T22:26:07.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:07 smithi167 conmon[60316]: debug 2022-01-31T22:26:07.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:07.572004+0000) 2022-01-31T22:26:08.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:08 smithi171 conmon[46715]: debug 2022-01-31T22:26:08.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:08.311988+0000) 2022-01-31T22:26:08.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:08 smithi171 conmon[51620]: debug 2022-01-31T22:26:08.172+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:08.172846+0000) 2022-01-31T22:26:08.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:08 smithi171 conmon[41853]: debug 2022-01-31T22:26:08.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:08.519458+0000) 2022-01-31T22:26:08.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:08 smithi167 conmon[49112]: debug 2022-01-31T22:26:08.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:08.645370+0000) 2022-01-31T22:26:08.866 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:08 smithi167 conmon[54076]: debug 2022-01-31T22:26:08.814+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:08.815535+0000) 2022-01-31T22:26:08.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:08 smithi167 conmon[60316]: debug 2022-01-31T22:26:08.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:08.572235+0000) 2022-01-31T22:26:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:08 smithi171 conmon[35325]: debug 2022-01-31T22:26:08.852+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:26:09.477 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[46715]: debug 2022-01-31T22:26:09.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.312175+0000) 2022-01-31T22:26:09.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[51620]: debug 2022-01-31T22:26:09.172+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.173058+0000) 2022-01-31T22:26:09.742 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[41853]: debug 2022-01-31T22:26:09.518+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.519659+0000) 2022-01-31T22:26:09.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:09 smithi167 conmon[49112]: debug 2022-01-31T22:26:09.644+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.645574+0000) 2022-01-31T22:26:09.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:09 smithi167 conmon[49112]: debug 2022-01-31T22:26:09.742+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.744477+0000) 2022-01-31T22:26:09.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:09 smithi167 conmon[54076]: debug 2022-01-31T22:26:09.741+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.743059+0000) 2022-01-31T22:26:09.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:09 smithi167 conmon[54076]: debug 2022-01-31T22:26:09.814+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.815673+0000) 2022-01-31T22:26:09.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:09 smithi167 conmon[60316]: debug 2022-01-31T22:26:09.571+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.572439+0000) 2022-01-31T22:26:09.868 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:09 smithi167 conmon[60316]: debug 2022-01-31T22:26:09.741+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.742308+0000) 2022-01-31T22:26:10.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[35325]: debug 2022-01-31T22:26:09.764+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209249 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:10.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[41853]: debug 2022-01-31T22:26:09.743+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.744409+0000) 2022-01-31T22:26:10.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[46715]: debug 2022-01-31T22:26:09.741+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.742279+0000) 2022-01-31T22:26:10.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:09 smithi171 conmon[51620]: debug 2022-01-31T22:26:09.742+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:09.743015+0000) 2022-01-31T22:26:10.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:10 smithi171 conmon[46715]: debug 2022-01-31T22:26:10.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:10.312359+0000) 2022-01-31T22:26:10.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:10 smithi171 conmon[51620]: debug 2022-01-31T22:26:10.172+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:10.173268+0000) 2022-01-31T22:26:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:10 smithi171 conmon[41853]: debug 2022-01-31T22:26:10.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:10.519841+0000) 2022-01-31T22:26:10.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:10 smithi167 conmon[49112]: debug 2022-01-31T22:26:10.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:10.645721+0000) 2022-01-31T22:26:10.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:10 smithi167 conmon[60316]: debug 2022-01-31T22:26:10.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:10.572667+0000) 2022-01-31T22:26:10.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:10 smithi167 conmon[54076]: debug 2022-01-31T22:26:10.815+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:10.815792+0000) 2022-01-31T22:26:11.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:11 smithi171 conmon[46715]: debug 2022-01-31T22:26:11.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:11.312458+0000) 2022-01-31T22:26:11.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:11 smithi171 conmon[51620]: debug 2022-01-31T22:26:11.173+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:11.173438+0000) 2022-01-31T22:26:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:11 smithi171 conmon[41853]: debug 2022-01-31T22:26:11.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:11.519972+0000) 2022-01-31T22:26:11.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:11 smithi167 conmon[49112]: debug 2022-01-31T22:26:11.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:11.645876+0000) 2022-01-31T22:26:11.867 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:11 smithi167 conmon[60316]: debug 2022-01-31T22:26:11.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:11.572780+0000) 2022-01-31T22:26:11.867 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:11 smithi167 conmon[54076]: debug 2022-01-31T22:26:11.815+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:11.816012+0000) 2022-01-31T22:26:12.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:12 smithi171 conmon[46715]: debug 2022-01-31T22:26:12.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:12.312651+0000) 2022-01-31T22:26:12.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:12 smithi171 conmon[51620]: debug 2022-01-31T22:26:12.173+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:12.173645+0000) 2022-01-31T22:26:12.826 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:12 smithi167 conmon[49112]: debug 2022-01-31T22:26:12.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:12.646042+0000) 2022-01-31T22:26:12.826 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:12 smithi167 conmon[54076]: debug 2022-01-31T22:26:12.815+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:12.816216+0000) 2022-01-31T22:26:12.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:12 smithi167 conmon[60316]: debug 2022-01-31T22:26:12.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:12.572915+0000) 2022-01-31T22:26:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:12 smithi171 conmon[41853]: debug 2022-01-31T22:26:12.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:12.520125+0000) 2022-01-31T22:26:13.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:13 smithi171 conmon[46715]: debug 2022-01-31T22:26:13.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:13.312810+0000) 2022-01-31T22:26:13.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:13 smithi171 conmon[51620]: debug 2022-01-31T22:26:13.173+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:13.173872+0000) 2022-01-31T22:26:13.828 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:13 smithi167 conmon[54076]: debug 2022-01-31T22:26:13.815+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:13.816403+0000) 2022-01-31T22:26:13.829 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:13 smithi167 conmon[60316]: debug 2022-01-31T22:26:13.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:13.573058+0000) 2022-01-31T22:26:13.830 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:13 smithi167 conmon[49112]: debug 2022-01-31T22:26:13.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:13.646202+0000) 2022-01-31T22:26:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:13 smithi171 conmon[41853]: debug 2022-01-31T22:26:13.519+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:13.520343+0000) 2022-01-31T22:26:14.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[46715]: debug 2022-01-31T22:26:14.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.312998+0000) 2022-01-31T22:26:14.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[51620]: debug 2022-01-31T22:26:14.173+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.174085+0000) 2022-01-31T22:26:14.767 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[41853]: debug 2022-01-31T22:26:14.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.520538+0000) 2022-01-31T22:26:14.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:14 smithi167 conmon[49112]: debug 2022-01-31T22:26:14.645+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.646384+0000) 2022-01-31T22:26:14.829 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:14 smithi167 conmon[49112]: debug 2022-01-31T22:26:14.766+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.767437+0000) 2022-01-31T22:26:14.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:14 smithi167 conmon[54076]: debug 2022-01-31T22:26:14.768+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.769124+0000) 2022-01-31T22:26:14.830 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:14 smithi167 conmon[54076]: debug 2022-01-31T22:26:14.816+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.816620+0000) 2022-01-31T22:26:14.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:14 smithi167 conmon[60316]: debug 2022-01-31T22:26:14.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.573182+0000) 2022-01-31T22:26:14.831 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:14 smithi167 conmon[60316]: debug 2022-01-31T22:26:14.767+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.767723+0000) 2022-01-31T22:26:15.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[41853]: debug 2022-01-31T22:26:14.767+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.768461+0000) 2022-01-31T22:26:15.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[46715]: debug 2022-01-31T22:26:14.766+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.767686+0000) 2022-01-31T22:26:15.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[51620]: debug 2022-01-31T22:26:14.766+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:14.767465+0000) 2022-01-31T22:26:15.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:14 smithi171 conmon[35325]: debug 2022-01-31T22:26:14.788+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209364 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:15.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:15 smithi171 conmon[46715]: debug 2022-01-31T22:26:15.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:15.313127+0000) 2022-01-31T22:26:15.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:15 smithi171 conmon[51620]: debug 2022-01-31T22:26:15.173+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:15.174244+0000) 2022-01-31T22:26:15.832 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:15 smithi167 conmon[49112]: debug 2022-01-31T22:26:15.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:15.646567+0000) 2022-01-31T22:26:15.833 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:15 smithi167 conmon[54076]: debug 2022-01-31T22:26:15.816+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:15.816741+0000) 2022-01-31T22:26:15.833 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:15 smithi167 conmon[60316]: debug 2022-01-31T22:26:15.572+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:15.573332+0000) 2022-01-31T22:26:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:15 smithi171 conmon[41853]: debug 2022-01-31T22:26:15.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:15.520718+0000) 2022-01-31T22:26:16.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:16 smithi171 conmon[46715]: debug 2022-01-31T22:26:16.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:16.313210+0000) 2022-01-31T22:26:16.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:16 smithi171 conmon[51620]: debug 2022-01-31T22:26:16.173+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:16.174411+0000) 2022-01-31T22:26:16.836 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:16 smithi167 conmon[49112]: debug 2022-01-31T22:26:16.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:16.646723+0000) 2022-01-31T22:26:16.836 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:16 smithi167 conmon[54076]: debug 2022-01-31T22:26:16.816+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:16.816903+0000) 2022-01-31T22:26:16.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:16 smithi167 conmon[60316]: debug 2022-01-31T22:26:16.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:16.573497+0000) 2022-01-31T22:26:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:16 smithi171 conmon[41853]: debug 2022-01-31T22:26:16.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:16.520873+0000) 2022-01-31T22:26:17.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:17 smithi171 conmon[46715]: debug 2022-01-31T22:26:17.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:17.313392+0000) 2022-01-31T22:26:17.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:17 smithi171 conmon[51620]: debug 2022-01-31T22:26:17.174+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:17.174595+0000) 2022-01-31T22:26:17.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:17 smithi167 conmon[49112]: debug 2022-01-31T22:26:17.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:17.646866+0000) 2022-01-31T22:26:17.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:17 smithi167 conmon[54076]: debug 2022-01-31T22:26:17.816+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:17.817086+0000) 2022-01-31T22:26:17.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:17 smithi167 conmon[60316]: debug 2022-01-31T22:26:17.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:17.573701+0000) 2022-01-31T22:26:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:17 smithi171 conmon[41853]: debug 2022-01-31T22:26:17.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:17.521038+0000) 2022-01-31T22:26:18.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:18 smithi171 conmon[46715]: debug 2022-01-31T22:26:18.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:18.313576+0000) 2022-01-31T22:26:18.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:18 smithi171 conmon[51620]: debug 2022-01-31T22:26:18.174+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:18.174765+0000) 2022-01-31T22:26:18.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:18 smithi167 conmon[49112]: debug 2022-01-31T22:26:18.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:18.647047+0000) 2022-01-31T22:26:18.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:18 smithi167 conmon[60316]: debug 2022-01-31T22:26:18.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:18.573872+0000) 2022-01-31T22:26:18.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:18 smithi167 conmon[54076]: debug 2022-01-31T22:26:18.816+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:18.817267+0000) 2022-01-31T22:26:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:18 smithi171 conmon[41853]: debug 2022-01-31T22:26:18.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:18.521254+0000) 2022-01-31T22:26:19.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[46715]: debug 2022-01-31T22:26:19.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.313789+0000) 2022-01-31T22:26:19.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[51620]: debug 2022-01-31T22:26:19.174+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.174958+0000) 2022-01-31T22:26:19.791 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[41853]: debug 2022-01-31T22:26:19.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.521464+0000) 2022-01-31T22:26:19.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:19 smithi167 conmon[49112]: debug 2022-01-31T22:26:19.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.647248+0000) 2022-01-31T22:26:19.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:19 smithi167 conmon[49112]: debug 2022-01-31T22:26:19.792+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.792746+0000) 2022-01-31T22:26:19.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:19 smithi167 conmon[54076]: debug 2022-01-31T22:26:19.792+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.793256+0000) 2022-01-31T22:26:19.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:19 smithi167 conmon[54076]: debug 2022-01-31T22:26:19.816+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.817417+0000) 2022-01-31T22:26:19.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:19 smithi167 conmon[60316]: debug 2022-01-31T22:26:19.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.574032+0000) 2022-01-31T22:26:19.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:19 smithi167 conmon[60316]: debug 2022-01-31T22:26:19.793+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.793624+0000) 2022-01-31T22:26:20.107 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[41853]: debug 2022-01-31T22:26:19.791+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.792502+0000) 2022-01-31T22:26:20.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[46715]: debug 2022-01-31T22:26:19.791+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.792288+0000) 2022-01-31T22:26:20.108 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[51620]: debug 2022-01-31T22:26:19.790+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:19.791211+0000) 2022-01-31T22:26:20.109 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:19 smithi171 conmon[35325]: debug 2022-01-31T22:26:19.812+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209473 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:20.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:20 smithi171 conmon[46715]: debug 2022-01-31T22:26:20.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:20.313968+0000) 2022-01-31T22:26:20.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:20 smithi171 conmon[51620]: debug 2022-01-31T22:26:20.174+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:20.175172+0000) 2022-01-31T22:26:20.846 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:20 smithi167 conmon[49112]: debug 2022-01-31T22:26:20.646+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:20.647378+0000) 2022-01-31T22:26:20.847 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:20 smithi167 conmon[54076]: debug 2022-01-31T22:26:20.817+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:20.817569+0000) 2022-01-31T22:26:20.848 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:20 smithi167 conmon[60316]: debug 2022-01-31T22:26:20.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:20.574222+0000) 2022-01-31T22:26:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:20 smithi171 conmon[41853]: debug 2022-01-31T22:26:20.520+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:20.521640+0000) 2022-01-31T22:26:21.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:21 smithi171 conmon[46715]: debug 2022-01-31T22:26:21.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:21.314120+0000) 2022-01-31T22:26:21.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:21 smithi171 conmon[51620]: debug 2022-01-31T22:26:21.174+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:21.175321+0000) 2022-01-31T22:26:21.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:21 smithi167 conmon[49112]: debug 2022-01-31T22:26:21.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:21.647533+0000) 2022-01-31T22:26:21.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:21 smithi167 conmon[54076]: debug 2022-01-31T22:26:21.817+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:21.817759+0000) 2022-01-31T22:26:21.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:21 smithi167 conmon[60316]: debug 2022-01-31T22:26:21.573+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:21.574377+0000) 2022-01-31T22:26:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:21 smithi171 conmon[41853]: debug 2022-01-31T22:26:21.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:21.521816+0000) 2022-01-31T22:26:22.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:22 smithi171 conmon[51620]: debug 2022-01-31T22:26:22.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:22.175545+0000) 2022-01-31T22:26:22.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:22 smithi171 conmon[46715]: debug 2022-01-31T22:26:22.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:22.314283+0000) 2022-01-31T22:26:22.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:22 smithi167 conmon[49112]: debug 2022-01-31T22:26:22.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:22.647713+0000) 2022-01-31T22:26:22.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:22 smithi167 conmon[54076]: debug 2022-01-31T22:26:22.817+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:22.817892+0000) 2022-01-31T22:26:22.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:22 smithi167 conmon[60316]: debug 2022-01-31T22:26:22.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:22.574599+0000) 2022-01-31T22:26:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:22 smithi171 conmon[41853]: debug 2022-01-31T22:26:22.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:22.522022+0000) 2022-01-31T22:26:23.280 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:23 smithi171 conmon[51620]: debug 2022-01-31T22:26:23.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:23.175769+0000) 2022-01-31T22:26:23.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:23 smithi171 conmon[46715]: debug 2022-01-31T22:26:23.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:23.314436+0000) 2022-01-31T22:26:23.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:23 smithi171 conmon[41853]: debug 2022-01-31T22:26:23.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:23.522240+0000) 2022-01-31T22:26:23.857 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:23 smithi167 conmon[49112]: debug 2022-01-31T22:26:23.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:23.647893+0000) 2022-01-31T22:26:23.858 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:23 smithi167 conmon[54076]: debug 2022-01-31T22:26:23.817+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:23.818042+0000) 2022-01-31T22:26:23.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:23 smithi167 conmon[60316]: debug 2022-01-31T22:26:23.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:23.574811+0000) 2022-01-31T22:26:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:23 smithi171 conmon[35325]: debug 2022-01-31T22:26:23.853+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:26:24.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[46715]: debug 2022-01-31T22:26:24.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.314610+0000) 2022-01-31T22:26:24.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[51620]: debug 2022-01-31T22:26:24.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.176014+0000) 2022-01-31T22:26:24.815 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[41853]: debug 2022-01-31T22:26:24.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.522447+0000) 2022-01-31T22:26:24.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:24 smithi167 conmon[49112]: debug 2022-01-31T22:26:24.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.648085+0000) 2022-01-31T22:26:24.858 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:24 smithi167 conmon[49112]: debug 2022-01-31T22:26:24.816+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.817091+0000) 2022-01-31T22:26:24.859 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:24 smithi167 conmon[54076]: debug 2022-01-31T22:26:24.816+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.817034+0000) 2022-01-31T22:26:24.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:24 smithi167 conmon[54076]: debug 2022-01-31T22:26:24.817+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.818139+0000) 2022-01-31T22:26:24.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:24 smithi167 conmon[60316]: debug 2022-01-31T22:26:24.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.574990+0000) 2022-01-31T22:26:24.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:24 smithi167 conmon[60316]: debug 2022-01-31T22:26:24.815+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.816482+0000) 2022-01-31T22:26:25.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[35325]: debug 2022-01-31T22:26:24.837+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209586 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:25.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[41853]: debug 2022-01-31T22:26:24.816+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.816788+0000) 2022-01-31T22:26:25.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[46715]: debug 2022-01-31T22:26:24.815+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.816527+0000) 2022-01-31T22:26:25.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:24 smithi171 conmon[51620]: debug 2022-01-31T22:26:24.814+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:24.815528+0000) 2022-01-31T22:26:25.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:25 smithi171 conmon[46715]: debug 2022-01-31T22:26:25.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:25.314783+0000) 2022-01-31T22:26:25.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:25 smithi171 conmon[51620]: debug 2022-01-31T22:26:25.176+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:25.176205+0000) 2022-01-31T22:26:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:25 smithi171 conmon[41853]: debug 2022-01-31T22:26:25.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:25.522626+0000) 2022-01-31T22:26:25.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:25 smithi167 conmon[49112]: debug 2022-01-31T22:26:25.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:25.648238+0000) 2022-01-31T22:26:25.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:25 smithi167 conmon[60316]: debug 2022-01-31T22:26:25.574+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:25.575135+0000) 2022-01-31T22:26:25.861 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:25 smithi167 conmon[54076]: debug 2022-01-31T22:26:25.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:25.818270+0000) 2022-01-31T22:26:26.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:26 smithi171 conmon[46715]: debug 2022-01-31T22:26:26.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:26.316034+0000) 2022-01-31T22:26:26.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:26 smithi171 conmon[51620]: debug 2022-01-31T22:26:26.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:26.176382+0000) 2022-01-31T22:26:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:26 smithi171 conmon[41853]: debug 2022-01-31T22:26:26.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:26.522772+0000) 2022-01-31T22:26:26.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:26 smithi167 conmon[49112]: debug 2022-01-31T22:26:26.647+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:26.648408+0000) 2022-01-31T22:26:26.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:26 smithi167 conmon[54076]: debug 2022-01-31T22:26:26.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:26.818450+0000) 2022-01-31T22:26:26.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:26 smithi167 conmon[60316]: debug 2022-01-31T22:26:26.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:26.575274+0000) 2022-01-31T22:26:27.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:27 smithi171 conmon[46715]: debug 2022-01-31T22:26:27.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:27.316713+0000) 2022-01-31T22:26:27.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:27 smithi171 conmon[51620]: debug 2022-01-31T22:26:27.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:27.176593+0000) 2022-01-31T22:26:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:27 smithi171 conmon[41853]: debug 2022-01-31T22:26:27.521+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:27.522949+0000) 2022-01-31T22:26:27.868 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:27 smithi167 conmon[49112]: debug 2022-01-31T22:26:27.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:27.648587+0000) 2022-01-31T22:26:27.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:27 smithi167 conmon[54076]: debug 2022-01-31T22:26:27.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:27.818673+0000) 2022-01-31T22:26:27.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:27 smithi167 conmon[60316]: debug 2022-01-31T22:26:27.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:27.575444+0000) 2022-01-31T22:26:28.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:28 smithi171 conmon[46715]: debug 2022-01-31T22:26:28.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:28.316924+0000) 2022-01-31T22:26:28.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:28 smithi171 conmon[51620]: debug 2022-01-31T22:26:28.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:28.176803+0000) 2022-01-31T22:26:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:28 smithi171 conmon[41853]: debug 2022-01-31T22:26:28.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:28.523137+0000) 2022-01-31T22:26:28.869 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:28 smithi167 conmon[54076]: debug 2022-01-31T22:26:28.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:28.818819+0000) 2022-01-31T22:26:28.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:28 smithi167 conmon[60316]: debug 2022-01-31T22:26:28.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:28.575659+0000) 2022-01-31T22:26:28.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:28 smithi167 conmon[49112]: debug 2022-01-31T22:26:28.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:28.648709+0000) 2022-01-31T22:26:29.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[46715]: debug 2022-01-31T22:26:29.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.317074+0000) 2022-01-31T22:26:29.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[51620]: debug 2022-01-31T22:26:29.175+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.176963+0000) 2022-01-31T22:26:29.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:29 smithi167 conmon[49112]: debug 2022-01-31T22:26:29.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.648910+0000) 2022-01-31T22:26:29.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:29 smithi167 conmon[60316]: debug 2022-01-31T22:26:29.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.575848+0000) 2022-01-31T22:26:29.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:29 smithi167 conmon[54076]: debug 2022-01-31T22:26:29.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.819027+0000) 2022-01-31T22:26:29.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[41853]: debug 2022-01-31T22:26:29.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.523319+0000) 2022-01-31T22:26:30.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[35325]: debug 2022-01-31T22:26:29.860+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209695 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:30.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[41853]: debug 2022-01-31T22:26:29.839+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.840853+0000) 2022-01-31T22:26:30.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[46715]: debug 2022-01-31T22:26:29.840+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.842131+0000) 2022-01-31T22:26:30.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:29 smithi171 conmon[51620]: debug 2022-01-31T22:26:29.838+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.840601+0000) 2022-01-31T22:26:30.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:29 smithi167 conmon[49112]: debug 2022-01-31T22:26:29.840+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.841282+0000) 2022-01-31T22:26:30.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:29 smithi167 conmon[60316]: debug 2022-01-31T22:26:29.839+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.840304+0000) 2022-01-31T22:26:30.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:29 smithi167 conmon[54076]: debug 2022-01-31T22:26:29.841+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:29.841902+0000) 2022-01-31T22:26:30.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:30 smithi171 conmon[46715]: debug 2022-01-31T22:26:30.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:30.317260+0000) 2022-01-31T22:26:30.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:30 smithi171 conmon[51620]: debug 2022-01-31T22:26:30.176+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:30.177193+0000) 2022-01-31T22:26:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:30 smithi171 conmon[41853]: debug 2022-01-31T22:26:30.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:30.523511+0000) 2022-01-31T22:26:30.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:30 smithi167 conmon[49112]: debug 2022-01-31T22:26:30.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:30.649070+0000) 2022-01-31T22:26:30.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:30 smithi167 conmon[60316]: debug 2022-01-31T22:26:30.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:30.576036+0000) 2022-01-31T22:26:30.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:30 smithi167 conmon[54076]: debug 2022-01-31T22:26:30.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:30.819251+0000) 2022-01-31T22:26:31.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:31 smithi171 conmon[46715]: debug 2022-01-31T22:26:31.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:31.317406+0000) 2022-01-31T22:26:31.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:31 smithi171 conmon[51620]: debug 2022-01-31T22:26:31.176+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:31.177369+0000) 2022-01-31T22:26:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:31 smithi171 conmon[41853]: debug 2022-01-31T22:26:31.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:31.523652+0000) 2022-01-31T22:26:31.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:31 smithi167 conmon[49112]: debug 2022-01-31T22:26:31.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:31.649211+0000) 2022-01-31T22:26:31.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:31 smithi167 conmon[54076]: debug 2022-01-31T22:26:31.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:31.819402+0000) 2022-01-31T22:26:31.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:31 smithi167 conmon[60316]: debug 2022-01-31T22:26:31.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:31.576175+0000) 2022-01-31T22:26:32.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:32 smithi171 conmon[46715]: debug 2022-01-31T22:26:32.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:32.317604+0000) 2022-01-31T22:26:32.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:32 smithi171 conmon[51620]: debug 2022-01-31T22:26:32.176+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:32.177580+0000) 2022-01-31T22:26:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:32 smithi171 conmon[41853]: debug 2022-01-31T22:26:32.522+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:32.523815+0000) 2022-01-31T22:26:32.869 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:32 smithi167 conmon[60316]: debug 2022-01-31T22:26:32.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:32.576360+0000) 2022-01-31T22:26:32.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:32 smithi167 conmon[49112]: debug 2022-01-31T22:26:32.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:32.649424+0000) 2022-01-31T22:26:32.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:32 smithi167 conmon[54076]: debug 2022-01-31T22:26:32.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:32.819591+0000) 2022-01-31T22:26:33.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:33 smithi171 conmon[46715]: debug 2022-01-31T22:26:33.316+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:33.317772+0000) 2022-01-31T22:26:33.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:33 smithi171 conmon[51620]: debug 2022-01-31T22:26:33.176+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:33.177799+0000) 2022-01-31T22:26:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:33 smithi171 conmon[41853]: debug 2022-01-31T22:26:33.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:33.524017+0000) 2022-01-31T22:26:33.869 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:33 smithi167 conmon[49112]: debug 2022-01-31T22:26:33.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:33.649620+0000) 2022-01-31T22:26:33.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:33 smithi167 conmon[54076]: debug 2022-01-31T22:26:33.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:33.819782+0000) 2022-01-31T22:26:33.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:33 smithi167 conmon[60316]: debug 2022-01-31T22:26:33.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:33.576575+0000) 2022-01-31T22:26:34.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[46715]: debug 2022-01-31T22:26:34.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.317977+0000) 2022-01-31T22:26:34.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[51620]: debug 2022-01-31T22:26:34.177+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.178017+0000) 2022-01-31T22:26:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[41853]: debug 2022-01-31T22:26:34.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.524218+0000) 2022-01-31T22:26:34.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:34 smithi167 conmon[49112]: debug 2022-01-31T22:26:34.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.649756+0000) 2022-01-31T22:26:34.865 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:34 smithi167 conmon[60316]: debug 2022-01-31T22:26:34.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.576777+0000) 2022-01-31T22:26:34.865 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:34 smithi167 conmon[54076]: debug 2022-01-31T22:26:34.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.819946+0000) 2022-01-31T22:26:35.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:34 smithi167 conmon[49112]: debug 2022-01-31T22:26:34.865+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.866244+0000) 2022-01-31T22:26:35.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:34 smithi167 conmon[54076]: debug 2022-01-31T22:26:34.864+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.865043+0000) 2022-01-31T22:26:35.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:34 smithi167 conmon[60316]: debug 2022-01-31T22:26:34.864+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.864870+0000) 2022-01-31T22:26:35.177 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[35325]: debug 2022-01-31T22:26:34.885+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209809 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:35.178 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[41853]: debug 2022-01-31T22:26:34.864+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.865921+0000) 2022-01-31T22:26:35.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[46715]: debug 2022-01-31T22:26:34.863+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.865153+0000) 2022-01-31T22:26:35.179 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:34 smithi171 conmon[51620]: debug 2022-01-31T22:26:34.863+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:34.865094+0000) 2022-01-31T22:26:35.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:35 smithi171 conmon[46715]: debug 2022-01-31T22:26:35.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:35.318125+0000) 2022-01-31T22:26:35.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:35 smithi171 conmon[51620]: debug 2022-01-31T22:26:35.177+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:35.178190+0000) 2022-01-31T22:26:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:35 smithi171 conmon[41853]: debug 2022-01-31T22:26:35.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:35.524401+0000) 2022-01-31T22:26:35.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:35 smithi167 conmon[54076]: debug 2022-01-31T22:26:35.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:35.820135+0000) 2022-01-31T22:26:35.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:35 smithi167 conmon[60316]: debug 2022-01-31T22:26:35.575+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:35.576959+0000) 2022-01-31T22:26:35.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:35 smithi167 conmon[49112]: debug 2022-01-31T22:26:35.648+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:35.649914+0000) 2022-01-31T22:26:36.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:36 smithi171 conmon[46715]: debug 2022-01-31T22:26:36.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:36.318277+0000) 2022-01-31T22:26:36.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:36 smithi171 conmon[51620]: debug 2022-01-31T22:26:36.177+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:36.178359+0000) 2022-01-31T22:26:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:36 smithi171 conmon[41853]: debug 2022-01-31T22:26:36.523+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:36.524570+0000) 2022-01-31T22:26:36.870 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:36 smithi167 conmon[54076]: debug 2022-01-31T22:26:36.818+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:36.820293+0000) 2022-01-31T22:26:36.870 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:36 smithi167 conmon[60316]: debug 2022-01-31T22:26:36.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:36.577058+0000) 2022-01-31T22:26:36.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:36 smithi167 conmon[49112]: debug 2022-01-31T22:26:36.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:36.650092+0000) 2022-01-31T22:26:37.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:37 smithi171 conmon[46715]: debug 2022-01-31T22:26:37.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:37.318471+0000) 2022-01-31T22:26:37.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:37 smithi171 conmon[51620]: debug 2022-01-31T22:26:37.177+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:37.178558+0000) 2022-01-31T22:26:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:37 smithi171 conmon[41853]: debug 2022-01-31T22:26:37.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:37.524770+0000) 2022-01-31T22:26:37.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:37 smithi167 conmon[49112]: debug 2022-01-31T22:26:37.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:37.650303+0000) 2022-01-31T22:26:37.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:37 smithi167 conmon[54076]: debug 2022-01-31T22:26:37.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:37.820478+0000) 2022-01-31T22:26:37.871 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:37 smithi167 conmon[60316]: debug 2022-01-31T22:26:37.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:37.577223+0000) 2022-01-31T22:26:38.482 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:38 smithi171 conmon[46715]: debug 2022-01-31T22:26:38.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:38.318658+0000) 2022-01-31T22:26:38.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:38 smithi171 conmon[51620]: debug 2022-01-31T22:26:38.178+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:38.178802+0000) 2022-01-31T22:26:38.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:38 smithi171 conmon[41853]: debug 2022-01-31T22:26:38.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:38.524872+0000) 2022-01-31T22:26:38.870 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:38 smithi167 conmon[49112]: debug 2022-01-31T22:26:38.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:38.650532+0000) 2022-01-31T22:26:38.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:38 smithi167 conmon[54076]: debug 2022-01-31T22:26:38.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:38.820680+0000) 2022-01-31T22:26:38.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:38 smithi167 conmon[60316]: debug 2022-01-31T22:26:38.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:38.577383+0000) 2022-01-31T22:26:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:38 smithi171 conmon[35325]: debug 2022-01-31T22:26:38.853+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:26:39.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[46715]: debug 2022-01-31T22:26:39.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.318889+0000) 2022-01-31T22:26:39.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[51620]: debug 2022-01-31T22:26:39.178+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.179049+0000) 2022-01-31T22:26:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[41853]: debug 2022-01-31T22:26:39.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.525086+0000) 2022-01-31T22:26:39.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:39 smithi167 conmon[49112]: debug 2022-01-31T22:26:39.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.650740+0000) 2022-01-31T22:26:39.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:39 smithi167 conmon[54076]: debug 2022-01-31T22:26:39.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.820883+0000) 2022-01-31T22:26:39.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:39 smithi167 conmon[60316]: debug 2022-01-31T22:26:39.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.577621+0000) 2022-01-31T22:26:40.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:39 smithi167 conmon[49112]: debug 2022-01-31T22:26:39.888+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.889623+0000) 2022-01-31T22:26:40.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:39 smithi167 conmon[54076]: debug 2022-01-31T22:26:39.889+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.890780+0000) 2022-01-31T22:26:40.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:39 smithi167 conmon[60316]: debug 2022-01-31T22:26:39.888+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.889861+0000) 2022-01-31T22:26:40.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[35325]: debug 2022-01-31T22:26:39.911+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 209919 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:40.179 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[41853]: debug 2022-01-31T22:26:39.890+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.891282+0000) 2022-01-31T22:26:40.180 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[46715]: debug 2022-01-31T22:26:39.889+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.890063+0000) 2022-01-31T22:26:40.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:39 smithi171 conmon[51620]: debug 2022-01-31T22:26:39.888+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:39.889192+0000) 2022-01-31T22:26:40.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:40 smithi171 conmon[46715]: debug 2022-01-31T22:26:40.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:40.319082+0000) 2022-01-31T22:26:40.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:40 smithi171 conmon[51620]: debug 2022-01-31T22:26:40.178+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:40.179265+0000) 2022-01-31T22:26:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:40 smithi171 conmon[41853]: debug 2022-01-31T22:26:40.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:40.525282+0000) 2022-01-31T22:26:40.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:40 smithi167 conmon[49112]: debug 2022-01-31T22:26:40.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:40.650897+0000) 2022-01-31T22:26:40.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:40 smithi167 conmon[54076]: debug 2022-01-31T22:26:40.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:40.821072+0000) 2022-01-31T22:26:40.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:40 smithi167 conmon[60316]: debug 2022-01-31T22:26:40.576+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:40.577786+0000) 2022-01-31T22:26:41.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:41 smithi171 conmon[46715]: debug 2022-01-31T22:26:41.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:41.319248+0000) 2022-01-31T22:26:41.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:41 smithi171 conmon[51620]: debug 2022-01-31T22:26:41.178+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:41.179413+0000) 2022-01-31T22:26:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:41 smithi171 conmon[41853]: debug 2022-01-31T22:26:41.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:41.525439+0000) 2022-01-31T22:26:41.871 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:41 smithi167 conmon[49112]: debug 2022-01-31T22:26:41.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:41.650997+0000) 2022-01-31T22:26:41.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:41 smithi167 conmon[54076]: debug 2022-01-31T22:26:41.819+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:41.821228+0000) 2022-01-31T22:26:41.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:41 smithi167 conmon[60316]: debug 2022-01-31T22:26:41.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:41.577957+0000) 2022-01-31T22:26:42.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:42 smithi171 conmon[46715]: debug 2022-01-31T22:26:42.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:42.319424+0000) 2022-01-31T22:26:42.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:42 smithi171 conmon[51620]: debug 2022-01-31T22:26:42.178+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:42.179578+0000) 2022-01-31T22:26:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:42 smithi171 conmon[41853]: debug 2022-01-31T22:26:42.524+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:42.525668+0000) 2022-01-31T22:26:42.871 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:42 smithi167 conmon[54076]: debug 2022-01-31T22:26:42.820+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:42.821438+0000) 2022-01-31T22:26:42.872 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:42 smithi167 conmon[60316]: debug 2022-01-31T22:26:42.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:42.578146+0000) 2022-01-31T22:26:42.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:42 smithi167 conmon[49112]: debug 2022-01-31T22:26:42.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:42.651191+0000) 2022-01-31T22:26:43.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:43 smithi171 conmon[46715]: debug 2022-01-31T22:26:43.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:43.319601+0000) 2022-01-31T22:26:43.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:43 smithi171 conmon[51620]: debug 2022-01-31T22:26:43.179+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:43.179820+0000) 2022-01-31T22:26:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:43 smithi171 conmon[41853]: debug 2022-01-31T22:26:43.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:43.525909+0000) 2022-01-31T22:26:43.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:43 smithi167 conmon[54076]: debug 2022-01-31T22:26:43.820+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:43.821598+0000) 2022-01-31T22:26:43.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:43 smithi167 conmon[60316]: debug 2022-01-31T22:26:43.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:43.578331+0000) 2022-01-31T22:26:43.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:43 smithi167 conmon[49112]: debug 2022-01-31T22:26:43.649+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:43.651371+0000) 2022-01-31T22:26:44.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[46715]: debug 2022-01-31T22:26:44.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.319802+0000) 2022-01-31T22:26:44.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[51620]: debug 2022-01-31T22:26:44.179+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.179999+0000) 2022-01-31T22:26:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[41853]: debug 2022-01-31T22:26:44.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.526088+0000) 2022-01-31T22:26:44.872 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:44 smithi167 conmon[54076]: debug 2022-01-31T22:26:44.820+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.821752+0000) 2022-01-31T22:26:44.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:44 smithi167 conmon[60316]: debug 2022-01-31T22:26:44.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.578528+0000) 2022-01-31T22:26:44.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:44 smithi167 conmon[49112]: debug 2022-01-31T22:26:44.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.651560+0000) 2022-01-31T22:26:45.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:44 smithi167 conmon[49112]: debug 2022-01-31T22:26:44.912+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.914179+0000) 2022-01-31T22:26:45.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:44 smithi167 conmon[54076]: debug 2022-01-31T22:26:44.913+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.914675+0000) 2022-01-31T22:26:45.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:44 smithi167 conmon[60316]: debug 2022-01-31T22:26:44.913+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.914492+0000) 2022-01-31T22:26:45.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[35325]: debug 2022-01-31T22:26:44.936+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210033 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:45.180 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[41853]: debug 2022-01-31T22:26:44.914+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.915486+0000) 2022-01-31T22:26:45.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[46715]: debug 2022-01-31T22:26:44.913+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.914229+0000) 2022-01-31T22:26:45.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:44 smithi171 conmon[51620]: debug 2022-01-31T22:26:44.914+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:44.915548+0000) 2022-01-31T22:26:45.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:45 smithi171 conmon[51620]: debug 2022-01-31T22:26:45.179+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:45.180183+0000) 2022-01-31T22:26:45.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:45 smithi171 conmon[46715]: debug 2022-01-31T22:26:45.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:45.319988+0000) 2022-01-31T22:26:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:45 smithi171 conmon[41853]: debug 2022-01-31T22:26:45.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:45.526282+0000) 2022-01-31T22:26:45.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:45 smithi167 conmon[49112]: debug 2022-01-31T22:26:45.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:45.651690+0000) 2022-01-31T22:26:45.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:45 smithi167 conmon[54076]: debug 2022-01-31T22:26:45.820+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:45.821936+0000) 2022-01-31T22:26:45.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:45 smithi167 conmon[60316]: debug 2022-01-31T22:26:45.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:45.578654+0000) 2022-01-31T22:26:46.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:46 smithi171 conmon[46715]: debug 2022-01-31T22:26:46.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:46.320146+0000) 2022-01-31T22:26:46.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:46 smithi171 conmon[51620]: debug 2022-01-31T22:26:46.179+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:46.180332+0000) 2022-01-31T22:26:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:46 smithi171 conmon[41853]: debug 2022-01-31T22:26:46.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:46.526434+0000) 2022-01-31T22:26:46.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:46 smithi167 conmon[49112]: debug 2022-01-31T22:26:46.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:46.651813+0000) 2022-01-31T22:26:46.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:46 smithi167 conmon[54076]: debug 2022-01-31T22:26:46.821+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:46.822105+0000) 2022-01-31T22:26:46.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:46 smithi167 conmon[60316]: debug 2022-01-31T22:26:46.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:46.578798+0000) 2022-01-31T22:26:47.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:47 smithi171 conmon[46715]: debug 2022-01-31T22:26:47.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:47.320328+0000) 2022-01-31T22:26:47.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:47 smithi171 conmon[51620]: debug 2022-01-31T22:26:47.179+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:47.180532+0000) 2022-01-31T22:26:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:47 smithi171 conmon[41853]: debug 2022-01-31T22:26:47.525+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:47.526537+0000) 2022-01-31T22:26:47.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:47 smithi167 conmon[49112]: debug 2022-01-31T22:26:47.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:47.651988+0000) 2022-01-31T22:26:47.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:47 smithi167 conmon[54076]: debug 2022-01-31T22:26:47.820+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:47.822258+0000) 2022-01-31T22:26:47.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:47 smithi167 conmon[60316]: debug 2022-01-31T22:26:47.578+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:47.579007+0000) 2022-01-31T22:26:48.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:48 smithi171 conmon[46715]: debug 2022-01-31T22:26:48.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:48.320515+0000) 2022-01-31T22:26:48.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:48 smithi171 conmon[51620]: debug 2022-01-31T22:26:48.180+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:48.180739+0000) 2022-01-31T22:26:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:48 smithi171 conmon[41853]: debug 2022-01-31T22:26:48.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:48.526709+0000) 2022-01-31T22:26:48.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:48 smithi167 conmon[49112]: debug 2022-01-31T22:26:48.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:48.652172+0000) 2022-01-31T22:26:48.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:48 smithi167 conmon[60316]: debug 2022-01-31T22:26:48.578+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:48.579193+0000) 2022-01-31T22:26:48.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:48 smithi167 conmon[54076]: debug 2022-01-31T22:26:48.820+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:48.822439+0000) 2022-01-31T22:26:49.484 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[46715]: debug 2022-01-31T22:26:49.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.320712+0000) 2022-01-31T22:26:49.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[51620]: debug 2022-01-31T22:26:49.180+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.180937+0000) 2022-01-31T22:26:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[41853]: debug 2022-01-31T22:26:49.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.526947+0000) 2022-01-31T22:26:49.872 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:49 smithi167 conmon[49112]: debug 2022-01-31T22:26:49.650+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.652334+0000) 2022-01-31T22:26:49.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:49 smithi167 conmon[54076]: debug 2022-01-31T22:26:49.821+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.822587+0000) 2022-01-31T22:26:49.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:49 smithi167 conmon[60316]: debug 2022-01-31T22:26:49.577+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.579401+0000) 2022-01-31T22:26:50.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:49 smithi167 conmon[49112]: debug 2022-01-31T22:26:49.937+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.939401+0000) 2022-01-31T22:26:50.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:49 smithi167 conmon[54076]: debug 2022-01-31T22:26:49.937+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.939269+0000) 2022-01-31T22:26:50.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:49 smithi167 conmon[60316]: debug 2022-01-31T22:26:49.938+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.939616+0000) 2022-01-31T22:26:50.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[41853]: debug 2022-01-31T22:26:49.938+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.939117+0000) 2022-01-31T22:26:50.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[46715]: debug 2022-01-31T22:26:49.939+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.940092+0000) 2022-01-31T22:26:50.287 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[35325]: debug 2022-01-31T22:26:49.960+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210142 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:50.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:49 smithi171 conmon[51620]: debug 2022-01-31T22:26:49.938+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:49.938871+0000) 2022-01-31T22:26:50.288 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:50 smithi171 conmon[51620]: debug 2022-01-31T22:26:50.180+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:50.181096+0000) 2022-01-31T22:26:50.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:50 smithi171 conmon[41853]: debug 2022-01-31T22:26:50.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:50.527133+0000) 2022-01-31T22:26:50.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:50 smithi171 conmon[46715]: debug 2022-01-31T22:26:50.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:50.320906+0000) 2022-01-31T22:26:50.873 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:50 smithi167 conmon[54076]: debug 2022-01-31T22:26:50.821+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:50.822798+0000) 2022-01-31T22:26:50.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:50 smithi167 conmon[60316]: debug 2022-01-31T22:26:50.578+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:50.579560+0000) 2022-01-31T22:26:50.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:50 smithi167 conmon[49112]: debug 2022-01-31T22:26:50.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:50.652491+0000) 2022-01-31T22:26:51.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:51 smithi171 conmon[46715]: debug 2022-01-31T22:26:51.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:51.321069+0000) 2022-01-31T22:26:51.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:51 smithi171 conmon[51620]: debug 2022-01-31T22:26:51.180+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:51.181264+0000) 2022-01-31T22:26:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:51 smithi171 conmon[41853]: debug 2022-01-31T22:26:51.526+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:51.527302+0000) 2022-01-31T22:26:51.873 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:51 smithi167 conmon[60316]: debug 2022-01-31T22:26:51.578+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:51.579704+0000) 2022-01-31T22:26:51.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:51 smithi167 conmon[49112]: debug 2022-01-31T22:26:51.651+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:51.652662+0000) 2022-01-31T22:26:51.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:51 smithi167 conmon[54076]: debug 2022-01-31T22:26:51.822+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:51.822949+0000) 2022-01-31T22:26:52.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:52 smithi171 conmon[46715]: debug 2022-01-31T22:26:52.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:52.321271+0000) 2022-01-31T22:26:52.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:52 smithi171 conmon[51620]: debug 2022-01-31T22:26:52.180+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:52.181463+0000) 2022-01-31T22:26:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:52 smithi171 conmon[41853]: debug 2022-01-31T22:26:52.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:52.527503+0000) 2022-01-31T22:26:52.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:52 smithi167 conmon[49112]: debug 2022-01-31T22:26:52.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:52.652879+0000) 2022-01-31T22:26:52.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:52 smithi167 conmon[54076]: debug 2022-01-31T22:26:52.822+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:52.823075+0000) 2022-01-31T22:26:52.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:52 smithi167 conmon[60316]: debug 2022-01-31T22:26:52.578+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:52.579865+0000) 2022-01-31T22:26:53.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:53 smithi171 conmon[46715]: debug 2022-01-31T22:26:53.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:53.321453+0000) 2022-01-31T22:26:53.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:53 smithi171 conmon[51620]: debug 2022-01-31T22:26:53.180+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:53.181672+0000) 2022-01-31T22:26:53.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:53 smithi171 conmon[41853]: debug 2022-01-31T22:26:53.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:53.527699+0000) 2022-01-31T22:26:53.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:53 smithi167 conmon[49112]: debug 2022-01-31T22:26:53.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:53.653109+0000) 2022-01-31T22:26:53.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:53 smithi167 conmon[54076]: debug 2022-01-31T22:26:53.821+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:53.823259+0000) 2022-01-31T22:26:53.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:53 smithi167 conmon[60316]: debug 2022-01-31T22:26:53.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:53.580054+0000) 2022-01-31T22:26:54.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:53 smithi171 conmon[35325]: debug 2022-01-31T22:26:53.854+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:26:54.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[46715]: debug 2022-01-31T22:26:54.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.321611+0000) 2022-01-31T22:26:54.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[51620]: debug 2022-01-31T22:26:54.181+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.181877+0000) 2022-01-31T22:26:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[41853]: debug 2022-01-31T22:26:54.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.527923+0000) 2022-01-31T22:26:54.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:54 smithi167 conmon[49112]: debug 2022-01-31T22:26:54.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.653263+0000) 2022-01-31T22:26:54.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:54 smithi167 conmon[54076]: debug 2022-01-31T22:26:54.821+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.823440+0000) 2022-01-31T22:26:54.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:54 smithi167 conmon[60316]: debug 2022-01-31T22:26:54.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.580254+0000) 2022-01-31T22:26:55.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:54 smithi167 conmon[49112]: debug 2022-01-31T22:26:54.962+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.963867+0000) 2022-01-31T22:26:55.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:54 smithi167 conmon[54076]: debug 2022-01-31T22:26:54.962+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.964355+0000) 2022-01-31T22:26:55.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:54 smithi167 conmon[60316]: debug 2022-01-31T22:26:54.962+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.964062+0000) 2022-01-31T22:26:55.287 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[35325]: debug 2022-01-31T22:26:54.991+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210255 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:26:55.287 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[41853]: debug 2022-01-31T22:26:54.962+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.963275+0000) 2022-01-31T22:26:55.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[46715]: debug 2022-01-31T22:26:54.964+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.965454+0000) 2022-01-31T22:26:55.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:54 smithi171 conmon[51620]: debug 2022-01-31T22:26:54.963+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:54.964552+0000) 2022-01-31T22:26:55.289 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:55 smithi171 conmon[51620]: debug 2022-01-31T22:26:55.181+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:55.182073+0000) 2022-01-31T22:26:55.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:55 smithi171 conmon[41853]: debug 2022-01-31T22:26:55.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:55.528110+0000) 2022-01-31T22:26:55.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:55 smithi171 conmon[46715]: debug 2022-01-31T22:26:55.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:55.321789+0000) 2022-01-31T22:26:55.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:55 smithi167 conmon[49112]: debug 2022-01-31T22:26:55.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:55.653405+0000) 2022-01-31T22:26:55.874 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:55 smithi167 conmon[54076]: debug 2022-01-31T22:26:55.823+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:55.823642+0000) 2022-01-31T22:26:55.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:55 smithi167 conmon[60316]: debug 2022-01-31T22:26:55.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:55.580445+0000) 2022-01-31T22:26:56.485 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:56 smithi171 conmon[46715]: debug 2022-01-31T22:26:56.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:56.321932+0000) 2022-01-31T22:26:56.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:56 smithi171 conmon[51620]: debug 2022-01-31T22:26:56.181+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:56.182250+0000) 2022-01-31T22:26:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:56 smithi171 conmon[41853]: debug 2022-01-31T22:26:56.527+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:56.528310+0000) 2022-01-31T22:26:56.873 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:56 smithi167 conmon[49112]: debug 2022-01-31T22:26:56.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:56.653557+0000) 2022-01-31T22:26:56.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:56 smithi167 conmon[60316]: debug 2022-01-31T22:26:56.579+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:56.580577+0000) 2022-01-31T22:26:56.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:56 smithi167 conmon[54076]: debug 2022-01-31T22:26:56.823+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:56.823811+0000) 2022-01-31T22:26:57.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:57 smithi171 conmon[46715]: debug 2022-01-31T22:26:57.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:57.322121+0000) 2022-01-31T22:26:57.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:57 smithi171 conmon[51620]: debug 2022-01-31T22:26:57.181+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:57.182488+0000) 2022-01-31T22:26:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:57 smithi171 conmon[41853]: debug 2022-01-31T22:26:57.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:57.528505+0000) 2022-01-31T22:26:57.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:57 smithi167 conmon[49112]: debug 2022-01-31T22:26:57.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:57.653733+0000) 2022-01-31T22:26:57.874 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:57 smithi167 conmon[60316]: debug 2022-01-31T22:26:57.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:57.580710+0000) 2022-01-31T22:26:57.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:57 smithi167 conmon[54076]: debug 2022-01-31T22:26:57.823+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:57.823941+0000) 2022-01-31T22:26:58.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:58 smithi171 conmon[51620]: debug 2022-01-31T22:26:58.182+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:58.182741+0000) 2022-01-31T22:26:58.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:58 smithi171 conmon[46715]: debug 2022-01-31T22:26:58.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:58.322329+0000) 2022-01-31T22:26:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:58 smithi171 conmon[41853]: debug 2022-01-31T22:26:58.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:58.528618+0000) 2022-01-31T22:26:58.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:58 smithi167 conmon[49112]: debug 2022-01-31T22:26:58.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:58.653872+0000) 2022-01-31T22:26:58.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:58 smithi167 conmon[60316]: debug 2022-01-31T22:26:58.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:58.580868+0000) 2022-01-31T22:26:58.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:58 smithi167 conmon[54076]: debug 2022-01-31T22:26:58.823+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:58.824105+0000) 2022-01-31T22:26:59.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:59 smithi171 conmon[51620]: debug 2022-01-31T22:26:59.182+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.182969+0000) 2022-01-31T22:26:59.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:59 smithi171 conmon[46715]: debug 2022-01-31T22:26:59.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.322512+0000) 2022-01-31T22:26:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:59 smithi171 conmon[41853]: debug 2022-01-31T22:26:59.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.528864+0000) 2022-01-31T22:26:59.874 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:59 smithi167 conmon[49112]: debug 2022-01-31T22:26:59.652+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.654069+0000) 2022-01-31T22:26:59.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:59 smithi167 conmon[60316]: debug 2022-01-31T22:26:59.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.581062+0000) 2022-01-31T22:26:59.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:59 smithi167 conmon[54076]: debug 2022-01-31T22:26:59.823+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.824299+0000) 2022-01-31T22:27:00.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:26:59 smithi167 conmon[49112]: debug 2022-01-31T22:26:59.994+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.995873+0000) 2022-01-31T22:27:00.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:26:59 smithi167 conmon[54076]: debug 2022-01-31T22:26:59.993+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.995308+0000) 2022-01-31T22:27:00.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:26:59 smithi167 conmon[60316]: debug 2022-01-31T22:26:59.993+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.995076+0000) 2022-01-31T22:27:00.288 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:00 smithi171 conmon[35325]: debug 2022-01-31T22:27:00.015+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210364 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:00.288 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:26:59 smithi171 conmon[41853]: debug 2022-01-31T22:26:59.993+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.993975+0000) 2022-01-31T22:27:00.289 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:26:59 smithi171 conmon[46715]: debug 2022-01-31T22:26:59.994+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.994823+0000) 2022-01-31T22:27:00.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:26:59 smithi171 conmon[51620]: debug 2022-01-31T22:26:59.993+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:26:59.994421+0000) 2022-01-31T22:27:00.290 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:00 smithi171 conmon[51620]: debug 2022-01-31T22:27:00.182+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:00.183169+0000) 2022-01-31T22:27:00.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:00 smithi171 conmon[46715]: debug 2022-01-31T22:27:00.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:00.322726+0000) 2022-01-31T22:27:00.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:00 smithi171 conmon[41853]: debug 2022-01-31T22:27:00.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:00.529053+0000) 2022-01-31T22:27:00.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:00 smithi167 conmon[49112]: debug 2022-01-31T22:27:00.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:00.654196+0000) 2022-01-31T22:27:00.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:00 smithi167 conmon[60316]: debug 2022-01-31T22:27:00.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:00.581249+0000) 2022-01-31T22:27:00.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:00 smithi167 conmon[54076]: debug 2022-01-31T22:27:00.823+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:00.824475+0000) 2022-01-31T22:27:01.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:01 smithi171 conmon[46715]: debug 2022-01-31T22:27:01.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:01.322866+0000) 2022-01-31T22:27:01.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:01 smithi171 conmon[51620]: debug 2022-01-31T22:27:01.183+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:01.183316+0000) 2022-01-31T22:27:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:01 smithi171 conmon[41853]: debug 2022-01-31T22:27:01.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:01.529177+0000) 2022-01-31T22:27:01.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:01 smithi167 conmon[49112]: debug 2022-01-31T22:27:01.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:01.654297+0000) 2022-01-31T22:27:01.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:01 smithi167 conmon[54076]: debug 2022-01-31T22:27:01.824+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:01.824649+0000) 2022-01-31T22:27:01.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:01 smithi167 conmon[60316]: debug 2022-01-31T22:27:01.580+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:01.581395+0000) 2022-01-31T22:27:02.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:02 smithi171 conmon[46715]: debug 2022-01-31T22:27:02.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:02.322991+0000) 2022-01-31T22:27:02.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:02 smithi171 conmon[51620]: debug 2022-01-31T22:27:02.182+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:02.183464+0000) 2022-01-31T22:27:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:02 smithi171 conmon[41853]: debug 2022-01-31T22:27:02.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:02.529348+0000) 2022-01-31T22:27:02.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:02 smithi167 conmon[54076]: debug 2022-01-31T22:27:02.824+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:02.824794+0000) 2022-01-31T22:27:02.875 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:02 smithi167 conmon[60316]: debug 2022-01-31T22:27:02.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:02.581605+0000) 2022-01-31T22:27:02.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:02 smithi167 conmon[49112]: debug 2022-01-31T22:27:02.653+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:02.654475+0000) 2022-01-31T22:27:03.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:03 smithi171 conmon[46715]: debug 2022-01-31T22:27:03.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:03.323206+0000) 2022-01-31T22:27:03.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:03 smithi171 conmon[51620]: debug 2022-01-31T22:27:03.182+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:03.183669+0000) 2022-01-31T22:27:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:03 smithi171 conmon[41853]: debug 2022-01-31T22:27:03.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:03.529528+0000) 2022-01-31T22:27:03.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:03 smithi167 conmon[49112]: debug 2022-01-31T22:27:03.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:03.654689+0000) 2022-01-31T22:27:03.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:03 smithi167 conmon[54076]: debug 2022-01-31T22:27:03.824+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:03.824963+0000) 2022-01-31T22:27:03.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:03 smithi167 conmon[60316]: debug 2022-01-31T22:27:03.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:03.581766+0000) 2022-01-31T22:27:04.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:04 smithi171 conmon[46715]: debug 2022-01-31T22:27:04.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:04.323404+0000) 2022-01-31T22:27:04.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:04 smithi171 conmon[51620]: debug 2022-01-31T22:27:04.183+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:04.183879+0000) 2022-01-31T22:27:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:04 smithi171 conmon[41853]: debug 2022-01-31T22:27:04.528+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:04.529678+0000) 2022-01-31T22:27:04.875 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:04 smithi167 conmon[54076]: debug 2022-01-31T22:27:04.824+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:04.825158+0000) 2022-01-31T22:27:04.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:04 smithi167 conmon[60316]: debug 2022-01-31T22:27:04.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:04.582014+0000) 2022-01-31T22:27:04.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:04 smithi167 conmon[49112]: debug 2022-01-31T22:27:04.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:04.654887+0000) 2022-01-31T22:27:05.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:05 smithi167 conmon[49112]: debug 2022-01-31T22:27:05.019+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.019491+0000) 2022-01-31T22:27:05.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:05 smithi167 conmon[54076]: debug 2022-01-31T22:27:05.019+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.019849+0000) 2022-01-31T22:27:05.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:05 smithi167 conmon[60316]: debug 2022-01-31T22:27:05.019+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.020212+0000) 2022-01-31T22:27:05.289 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[35325]: debug 2022-01-31T22:27:05.039+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210477 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:05.289 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[41853]: debug 2022-01-31T22:27:05.017+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.018594+0000) 2022-01-31T22:27:05.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[46715]: debug 2022-01-31T22:27:05.018+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.019460+0000) 2022-01-31T22:27:05.291 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[51620]: debug 2022-01-31T22:27:05.018+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.018916+0000) 2022-01-31T22:27:05.291 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[51620]: debug 2022-01-31T22:27:05.183+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.184029+0000) 2022-01-31T22:27:05.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[46715]: debug 2022-01-31T22:27:05.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.323626+0000) 2022-01-31T22:27:05.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:05 smithi171 conmon[41853]: debug 2022-01-31T22:27:05.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.529792+0000) 2022-01-31T22:27:05.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:05 smithi167 conmon[49112]: debug 2022-01-31T22:27:05.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.655076+0000) 2022-01-31T22:27:05.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:05 smithi167 conmon[60316]: debug 2022-01-31T22:27:05.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.582200+0000) 2022-01-31T22:27:05.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:05 smithi167 conmon[54076]: debug 2022-01-31T22:27:05.824+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:05.825314+0000) 2022-01-31T22:27:06.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:06 smithi171 conmon[46715]: debug 2022-01-31T22:27:06.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:06.323792+0000) 2022-01-31T22:27:06.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:06 smithi171 conmon[51620]: debug 2022-01-31T22:27:06.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:06.184207+0000) 2022-01-31T22:27:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:06 smithi171 conmon[41853]: debug 2022-01-31T22:27:06.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:06.529933+0000) 2022-01-31T22:27:06.875 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:06 smithi167 conmon[49112]: debug 2022-01-31T22:27:06.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:06.655233+0000) 2022-01-31T22:27:06.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:06 smithi167 conmon[54076]: debug 2022-01-31T22:27:06.825+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:06.825488+0000) 2022-01-31T22:27:06.876 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:06 smithi167 conmon[60316]: debug 2022-01-31T22:27:06.581+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:06.582365+0000) 2022-01-31T22:27:07.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:07 smithi171 conmon[46715]: debug 2022-01-31T22:27:07.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:07.323987+0000) 2022-01-31T22:27:07.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:07 smithi171 conmon[51620]: debug 2022-01-31T22:27:07.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:07.184373+0000) 2022-01-31T22:27:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:07 smithi171 conmon[41853]: debug 2022-01-31T22:27:07.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:07.530108+0000) 2022-01-31T22:27:07.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:07 smithi167 conmon[54076]: debug 2022-01-31T22:27:07.825+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:07.825625+0000) 2022-01-31T22:27:07.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:07 smithi167 conmon[49112]: debug 2022-01-31T22:27:07.654+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:07.655435+0000) 2022-01-31T22:27:07.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:07 smithi167 conmon[49112]: 2022-01-31T22:27:07.877 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:07 smithi167 conmon[60316]: debug 2022-01-31T22:27:07.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:07.582560+0000) 2022-01-31T22:27:08.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:08 smithi171 conmon[46715]: debug 2022-01-31T22:27:08.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:08.324177+0000) 2022-01-31T22:27:08.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:08 smithi171 conmon[51620]: debug 2022-01-31T22:27:08.183+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:08.184559+0000) 2022-01-31T22:27:08.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:08 smithi171 conmon[41853]: debug 2022-01-31T22:27:08.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:08.530263+0000) 2022-01-31T22:27:08.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:08 smithi167 conmon[49112]: debug 2022-01-31T22:27:08.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:08.655623+0000) 2022-01-31T22:27:08.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:08 smithi167 conmon[54076]: debug 2022-01-31T22:27:08.825+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:08.825783+0000) 2022-01-31T22:27:08.877 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:08 smithi167 conmon[60316]: debug 2022-01-31T22:27:08.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:08.582703+0000) 2022-01-31T22:27:09.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:08 smithi171 conmon[35325]: debug 2022-01-31T22:27:08.855+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:27:09.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:09 smithi171 conmon[46715]: debug 2022-01-31T22:27:09.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:09.324397+0000) 2022-01-31T22:27:09.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:09 smithi171 conmon[51620]: debug 2022-01-31T22:27:09.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:09.184723+0000) 2022-01-31T22:27:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:09 smithi171 conmon[41853]: debug 2022-01-31T22:27:09.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:09.530476+0000) 2022-01-31T22:27:09.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:09 smithi167 conmon[49112]: debug 2022-01-31T22:27:09.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:09.655818+0000) 2022-01-31T22:27:09.877 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:09 smithi167 conmon[60316]: debug 2022-01-31T22:27:09.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:09.582896+0000) 2022-01-31T22:27:09.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:09 smithi167 conmon[54076]: debug 2022-01-31T22:27:09.825+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:09.825957+0000) 2022-01-31T22:27:10.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:10 smithi167 conmon[54076]: debug 2022-01-31T22:27:10.042+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.042605+0000) 2022-01-31T22:27:10.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:10 smithi167 conmon[60316]: debug 2022-01-31T22:27:10.042+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.043176+0000) 2022-01-31T22:27:10.170 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:10 smithi167 conmon[49112]: debug 2022-01-31T22:27:10.042+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.042802+0000) 2022-01-31T22:27:10.324 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[35325]: debug 2022-01-31T22:27:10.063+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210587 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:10.325 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[41853]: debug 2022-01-31T22:27:10.041+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.041877+0000) 2022-01-31T22:27:10.325 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[46715]: debug 2022-01-31T22:27:10.042+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.042914+0000) 2022-01-31T22:27:10.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[51620]: debug 2022-01-31T22:27:10.041+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.042740+0000) 2022-01-31T22:27:10.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[51620]: debug 2022-01-31T22:27:10.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.184877+0000) 2022-01-31T22:27:10.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[46715]: debug 2022-01-31T22:27:10.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.324582+0000) 2022-01-31T22:27:10.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:10 smithi171 conmon[41853]: debug 2022-01-31T22:27:10.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.530675+0000) 2022-01-31T22:27:10.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:10 smithi167 conmon[49112]: debug 2022-01-31T22:27:10.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.655973+0000) 2022-01-31T22:27:10.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:10 smithi167 conmon[54076]: debug 2022-01-31T22:27:10.825+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.826141+0000) 2022-01-31T22:27:10.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:10 smithi167 conmon[60316]: debug 2022-01-31T22:27:10.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:10.583110+0000) 2022-01-31T22:27:11.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:11 smithi171 conmon[46715]: debug 2022-01-31T22:27:11.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:11.324735+0000) 2022-01-31T22:27:11.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:11 smithi171 conmon[51620]: debug 2022-01-31T22:27:11.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:11.184993+0000) 2022-01-31T22:27:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:11 smithi171 conmon[41853]: debug 2022-01-31T22:27:11.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:11.530825+0000) 2022-01-31T22:27:11.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:11 smithi167 conmon[49112]: debug 2022-01-31T22:27:11.655+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:11.656137+0000) 2022-01-31T22:27:11.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:11 smithi167 conmon[54076]: debug 2022-01-31T22:27:11.825+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:11.826286+0000) 2022-01-31T22:27:11.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:11 smithi167 conmon[60316]: debug 2022-01-31T22:27:11.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:11.583266+0000) 2022-01-31T22:27:12.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:12 smithi171 conmon[46715]: debug 2022-01-31T22:27:12.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:12.324963+0000) 2022-01-31T22:27:12.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:12 smithi171 conmon[51620]: debug 2022-01-31T22:27:12.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:12.185193+0000) 2022-01-31T22:27:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:12 smithi171 conmon[41853]: debug 2022-01-31T22:27:12.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:12.530985+0000) 2022-01-31T22:27:12.876 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:12 smithi167 conmon[54076]: debug 2022-01-31T22:27:12.826+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:12.826481+0000) 2022-01-31T22:27:12.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:12 smithi167 conmon[49112]: debug 2022-01-31T22:27:12.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:12.656323+0000) 2022-01-31T22:27:12.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:12 smithi167 conmon[60316]: debug 2022-01-31T22:27:12.582+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:12.583381+0000) 2022-01-31T22:27:13.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:13 smithi171 conmon[46715]: debug 2022-01-31T22:27:13.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:13.325176+0000) 2022-01-31T22:27:13.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:13 smithi171 conmon[51620]: debug 2022-01-31T22:27:13.183+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:13.185426+0000) 2022-01-31T22:27:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:13 smithi171 conmon[41853]: debug 2022-01-31T22:27:13.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:13.531175+0000) 2022-01-31T22:27:13.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:13 smithi167 conmon[49112]: debug 2022-01-31T22:27:13.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:13.656547+0000) 2022-01-31T22:27:13.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:13 smithi167 conmon[54076]: debug 2022-01-31T22:27:13.826+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:13.826666+0000) 2022-01-31T22:27:13.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:13 smithi167 conmon[60316]: debug 2022-01-31T22:27:13.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:13.583595+0000) 2022-01-31T22:27:14.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:14 smithi171 conmon[46715]: debug 2022-01-31T22:27:14.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:14.325384+0000) 2022-01-31T22:27:14.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:14 smithi171 conmon[51620]: debug 2022-01-31T22:27:14.183+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:14.185670+0000) 2022-01-31T22:27:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:14 smithi171 conmon[41853]: debug 2022-01-31T22:27:14.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:14.531399+0000) 2022-01-31T22:27:14.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:14 smithi167 conmon[49112]: debug 2022-01-31T22:27:14.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:14.656747+0000) 2022-01-31T22:27:14.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:14 smithi167 conmon[54076]: debug 2022-01-31T22:27:14.826+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:14.826836+0000) 2022-01-31T22:27:14.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:14 smithi167 conmon[60316]: debug 2022-01-31T22:27:14.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:14.583848+0000) 2022-01-31T22:27:15.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:15 smithi167 conmon[49112]: debug 2022-01-31T22:27:15.067+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.068042+0000) 2022-01-31T22:27:15.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:15 smithi167 conmon[54076]: debug 2022-01-31T22:27:15.066+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.067041+0000) 2022-01-31T22:27:15.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:15 smithi167 conmon[60316]: debug 2022-01-31T22:27:15.066+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.067317+0000) 2022-01-31T22:27:15.325 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[35325]: debug 2022-01-31T22:27:15.087+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210699 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:15.326 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[41853]: debug 2022-01-31T22:27:15.065+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.067442+0000) 2022-01-31T22:27:15.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[46715]: debug 2022-01-31T22:27:15.066+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.067938+0000) 2022-01-31T22:27:15.327 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[51620]: debug 2022-01-31T22:27:15.066+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.068308+0000) 2022-01-31T22:27:15.327 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[51620]: debug 2022-01-31T22:27:15.184+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.185907+0000) 2022-01-31T22:27:15.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[41853]: debug 2022-01-31T22:27:15.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.531570+0000) 2022-01-31T22:27:15.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:15 smithi171 conmon[46715]: debug 2022-01-31T22:27:15.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.325535+0000) 2022-01-31T22:27:15.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:15 smithi167 conmon[49112]: debug 2022-01-31T22:27:15.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.656933+0000) 2022-01-31T22:27:15.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:15 smithi167 conmon[60316]: debug 2022-01-31T22:27:15.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.584036+0000) 2022-01-31T22:27:15.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:15 smithi167 conmon[54076]: debug 2022-01-31T22:27:15.826+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:15.826990+0000) 2022-01-31T22:27:16.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:16 smithi171 conmon[46715]: debug 2022-01-31T22:27:16.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:16.325681+0000) 2022-01-31T22:27:16.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:16 smithi171 conmon[51620]: debug 2022-01-31T22:27:16.185+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:16.186045+0000) 2022-01-31T22:27:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:16 smithi171 conmon[41853]: debug 2022-01-31T22:27:16.529+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:16.531728+0000) 2022-01-31T22:27:16.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:16 smithi167 conmon[49112]: debug 2022-01-31T22:27:16.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:16.657059+0000) 2022-01-31T22:27:16.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:16 smithi167 conmon[60316]: debug 2022-01-31T22:27:16.583+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:16.584176+0000) 2022-01-31T22:27:16.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:16 smithi167 conmon[54076]: debug 2022-01-31T22:27:16.826+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:16.827143+0000) 2022-01-31T22:27:17.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:17 smithi171 conmon[46715]: debug 2022-01-31T22:27:17.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:17.325839+0000) 2022-01-31T22:27:17.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:17 smithi171 conmon[51620]: debug 2022-01-31T22:27:17.185+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:17.186223+0000) 2022-01-31T22:27:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:17 smithi171 conmon[41853]: debug 2022-01-31T22:27:17.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:17.531862+0000) 2022-01-31T22:27:17.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:17 smithi167 conmon[49112]: debug 2022-01-31T22:27:17.656+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:17.657236+0000) 2022-01-31T22:27:17.878 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:17 smithi167 conmon[60316]: debug 2022-01-31T22:27:17.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:17.584372+0000) 2022-01-31T22:27:17.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:17 smithi167 conmon[54076]: debug 2022-01-31T22:27:17.826+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:17.827317+0000) 2022-01-31T22:27:18.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:18 smithi171 conmon[46715]: debug 2022-01-31T22:27:18.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:18.326058+0000) 2022-01-31T22:27:18.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:18 smithi171 conmon[51620]: debug 2022-01-31T22:27:18.185+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:18.186430+0000) 2022-01-31T22:27:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:18 smithi171 conmon[41853]: debug 2022-01-31T22:27:18.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:18.532023+0000) 2022-01-31T22:27:18.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:18 smithi167 conmon[49112]: debug 2022-01-31T22:27:18.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:18.657390+0000) 2022-01-31T22:27:18.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:18 smithi167 conmon[54076]: debug 2022-01-31T22:27:18.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:18.827476+0000) 2022-01-31T22:27:18.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:18 smithi167 conmon[60316]: debug 2022-01-31T22:27:18.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:18.584590+0000) 2022-01-31T22:27:19.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:19 smithi171 conmon[46715]: debug 2022-01-31T22:27:19.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:19.326274+0000) 2022-01-31T22:27:19.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:19 smithi171 conmon[51620]: debug 2022-01-31T22:27:19.185+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:19.186642+0000) 2022-01-31T22:27:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:19 smithi171 conmon[41853]: debug 2022-01-31T22:27:19.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:19.532208+0000) 2022-01-31T22:27:19.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:19 smithi167 conmon[49112]: debug 2022-01-31T22:27:19.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:19.657551+0000) 2022-01-31T22:27:19.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:19 smithi167 conmon[54076]: debug 2022-01-31T22:27:19.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:19.827687+0000) 2022-01-31T22:27:19.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:19 smithi167 conmon[60316]: debug 2022-01-31T22:27:19.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:19.584796+0000) 2022-01-31T22:27:20.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:20 smithi167 conmon[49112]: debug 2022-01-31T22:27:20.091+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.091836+0000) 2022-01-31T22:27:20.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:20 smithi167 conmon[54076]: debug 2022-01-31T22:27:20.091+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.091646+0000) 2022-01-31T22:27:20.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:20 smithi167 conmon[60316]: debug 2022-01-31T22:27:20.091+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.091519+0000) 2022-01-31T22:27:20.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[35325]: debug 2022-01-31T22:27:20.111+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210810 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:20.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[41853]: debug 2022-01-31T22:27:20.089+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.090880+0000) 2022-01-31T22:27:20.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[46715]: debug 2022-01-31T22:27:20.089+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.091541+0000) 2022-01-31T22:27:20.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[46715]: debug 2022-01-31T22:27:20.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.326481+0000) 2022-01-31T22:27:20.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[51620]: debug 2022-01-31T22:27:20.090+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.092326+0000) 2022-01-31T22:27:20.362 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[51620]: debug 2022-01-31T22:27:20.185+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.186835+0000) 2022-01-31T22:27:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:20 smithi171 conmon[41853]: debug 2022-01-31T22:27:20.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.532342+0000) 2022-01-31T22:27:20.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:20 smithi167 conmon[49112]: debug 2022-01-31T22:27:20.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.657746+0000) 2022-01-31T22:27:20.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:20 smithi167 conmon[54076]: debug 2022-01-31T22:27:20.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.827885+0000) 2022-01-31T22:27:20.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:20 smithi167 conmon[60316]: debug 2022-01-31T22:27:20.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:20.584979+0000) 2022-01-31T22:27:21.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:21 smithi171 conmon[46715]: debug 2022-01-31T22:27:21.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:21.326592+0000) 2022-01-31T22:27:21.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:21 smithi171 conmon[51620]: debug 2022-01-31T22:27:21.186+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:21.186995+0000) 2022-01-31T22:27:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:21 smithi171 conmon[41853]: debug 2022-01-31T22:27:21.530+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:21.532437+0000) 2022-01-31T22:27:21.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:21 smithi167 conmon[49112]: debug 2022-01-31T22:27:21.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:21.657897+0000) 2022-01-31T22:27:21.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:21 smithi167 conmon[60316]: debug 2022-01-31T22:27:21.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:21.585120+0000) 2022-01-31T22:27:21.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:21 smithi167 conmon[54076]: debug 2022-01-31T22:27:21.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:21.828047+0000) 2022-01-31T22:27:22.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:22 smithi171 conmon[46715]: debug 2022-01-31T22:27:22.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:22.326782+0000) 2022-01-31T22:27:22.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:22 smithi171 conmon[51620]: debug 2022-01-31T22:27:22.186+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:22.187138+0000) 2022-01-31T22:27:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:22 smithi171 conmon[41853]: debug 2022-01-31T22:27:22.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:22.532593+0000) 2022-01-31T22:27:22.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:22 smithi167 conmon[49112]: debug 2022-01-31T22:27:22.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:22.658083+0000) 2022-01-31T22:27:22.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:22 smithi167 conmon[54076]: debug 2022-01-31T22:27:22.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:22.828200+0000) 2022-01-31T22:27:22.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:22 smithi167 conmon[60316]: debug 2022-01-31T22:27:22.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:22.585317+0000) 2022-01-31T22:27:23.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:23 smithi171 conmon[46715]: debug 2022-01-31T22:27:23.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:23.326969+0000) 2022-01-31T22:27:23.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:23 smithi171 conmon[51620]: debug 2022-01-31T22:27:23.185+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:23.187261+0000) 2022-01-31T22:27:23.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:23 smithi171 conmon[41853]: debug 2022-01-31T22:27:23.531+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:23.532797+0000) 2022-01-31T22:27:23.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:23 smithi167 conmon[49112]: debug 2022-01-31T22:27:23.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:23.658268+0000) 2022-01-31T22:27:23.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:23 smithi167 conmon[54076]: debug 2022-01-31T22:27:23.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:23.828384+0000) 2022-01-31T22:27:23.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:23 smithi167 conmon[60316]: debug 2022-01-31T22:27:23.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:23.585524+0000) 2022-01-31T22:27:24.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:23 smithi171 conmon[35325]: debug 2022-01-31T22:27:23.855+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:27:24.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:24 smithi171 conmon[46715]: debug 2022-01-31T22:27:24.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:24.327094+0000) 2022-01-31T22:27:24.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:24 smithi171 conmon[51620]: debug 2022-01-31T22:27:24.186+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:24.187432+0000) 2022-01-31T22:27:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:24 smithi171 conmon[41853]: debug 2022-01-31T22:27:24.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:24.533015+0000) 2022-01-31T22:27:24.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:24 smithi167 conmon[49112]: debug 2022-01-31T22:27:24.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:24.658454+0000) 2022-01-31T22:27:24.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:24 smithi167 conmon[60316]: debug 2022-01-31T22:27:24.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:24.585749+0000) 2022-01-31T22:27:24.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:24 smithi167 conmon[54076]: debug 2022-01-31T22:27:24.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:24.828539+0000) 2022-01-31T22:27:25.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:25 smithi167 conmon[49112]: debug 2022-01-31T22:27:25.115+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.117148+0000) 2022-01-31T22:27:25.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:25 smithi167 conmon[54076]: debug 2022-01-31T22:27:25.115+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.116800+0000) 2022-01-31T22:27:25.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:25 smithi167 conmon[60316]: debug 2022-01-31T22:27:25.115+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.116571+0000) 2022-01-31T22:27:25.491 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[35325]: debug 2022-01-31T22:27:25.136+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 210922 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:25.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[41853]: debug 2022-01-31T22:27:25.115+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.115956+0000) 2022-01-31T22:27:25.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[46715]: debug 2022-01-31T22:27:25.115+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.116279+0000) 2022-01-31T22:27:25.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[46715]: debug 2022-01-31T22:27:25.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.327241+0000) 2022-01-31T22:27:25.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[51620]: debug 2022-01-31T22:27:25.115+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.115820+0000) 2022-01-31T22:27:25.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[51620]: debug 2022-01-31T22:27:25.186+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.187646+0000) 2022-01-31T22:27:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:25 smithi171 conmon[41853]: debug 2022-01-31T22:27:25.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.533196+0000) 2022-01-31T22:27:25.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:25 smithi167 conmon[49112]: debug 2022-01-31T22:27:25.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.658600+0000) 2022-01-31T22:27:25.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:25 smithi167 conmon[54076]: debug 2022-01-31T22:27:25.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.828696+0000) 2022-01-31T22:27:25.861 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:25 smithi167 conmon[60316]: debug 2022-01-31T22:27:25.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:25.585947+0000) 2022-01-31T22:27:26.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:26 smithi171 conmon[46715]: debug 2022-01-31T22:27:26.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:26.327387+0000) 2022-01-31T22:27:26.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:26 smithi171 conmon[51620]: debug 2022-01-31T22:27:26.187+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:26.187804+0000) 2022-01-31T22:27:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:26 smithi171 conmon[41853]: debug 2022-01-31T22:27:26.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:26.533362+0000) 2022-01-31T22:27:26.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:26 smithi167 conmon[49112]: debug 2022-01-31T22:27:26.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:26.658744+0000) 2022-01-31T22:27:26.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:26 smithi167 conmon[54076]: debug 2022-01-31T22:27:26.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:26.828845+0000) 2022-01-31T22:27:26.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:26 smithi167 conmon[60316]: debug 2022-01-31T22:27:26.584+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:26.586125+0000) 2022-01-31T22:27:27.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:27 smithi171 conmon[46715]: debug 2022-01-31T22:27:27.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:27.327567+0000) 2022-01-31T22:27:27.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:27 smithi171 conmon[51620]: debug 2022-01-31T22:27:27.187+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:27.188019+0000) 2022-01-31T22:27:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:27 smithi171 conmon[41853]: debug 2022-01-31T22:27:27.532+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:27.533545+0000) 2022-01-31T22:27:27.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:27 smithi167 conmon[49112]: debug 2022-01-31T22:27:27.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:27.658961+0000) 2022-01-31T22:27:27.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:27 smithi167 conmon[54076]: debug 2022-01-31T22:27:27.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:27.829006+0000) 2022-01-31T22:27:27.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:27 smithi167 conmon[60316]: debug 2022-01-31T22:27:27.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:27.586318+0000) 2022-01-31T22:27:28.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:28 smithi171 conmon[46715]: debug 2022-01-31T22:27:28.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:28.327778+0000) 2022-01-31T22:27:28.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:28 smithi171 conmon[51620]: debug 2022-01-31T22:27:28.187+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:28.188251+0000) 2022-01-31T22:27:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:28 smithi171 conmon[41853]: debug 2022-01-31T22:27:28.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:28.533746+0000) 2022-01-31T22:27:28.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:28 smithi167 conmon[49112]: debug 2022-01-31T22:27:28.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:28.659114+0000) 2022-01-31T22:27:28.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:28 smithi167 conmon[54076]: debug 2022-01-31T22:27:28.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:28.829188+0000) 2022-01-31T22:27:28.882 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:28 smithi167 conmon[60316]: debug 2022-01-31T22:27:28.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:28.586523+0000) 2022-01-31T22:27:29.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:29 smithi171 conmon[46715]: debug 2022-01-31T22:27:29.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:29.327959+0000) 2022-01-31T22:27:29.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:29 smithi171 conmon[51620]: debug 2022-01-31T22:27:29.187+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:29.188475+0000) 2022-01-31T22:27:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:29 smithi171 conmon[41853]: debug 2022-01-31T22:27:29.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:29.533959+0000) 2022-01-31T22:27:29.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:29 smithi167 conmon[49112]: debug 2022-01-31T22:27:29.657+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:29.659309+0000) 2022-01-31T22:27:29.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:29 smithi167 conmon[54076]: debug 2022-01-31T22:27:29.827+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:29.829391+0000) 2022-01-31T22:27:29.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:29 smithi167 conmon[60316]: debug 2022-01-31T22:27:29.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:29.586712+0000) 2022-01-31T22:27:30.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:30 smithi167 conmon[49112]: debug 2022-01-31T22:27:30.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.141183+0000) 2022-01-31T22:27:30.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:30 smithi167 conmon[54076]: debug 2022-01-31T22:27:30.138+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.139934+0000) 2022-01-31T22:27:30.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:30 smithi167 conmon[60316]: debug 2022-01-31T22:27:30.138+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.139771+0000) 2022-01-31T22:27:30.492 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[35325]: debug 2022-01-31T22:27:30.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211033 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:30.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[41853]: debug 2022-01-31T22:27:30.140+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.141069+0000) 2022-01-31T22:27:30.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[46715]: debug 2022-01-31T22:27:30.140+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.141150+0000) 2022-01-31T22:27:30.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[46715]: debug 2022-01-31T22:27:30.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.328133+0000) 2022-01-31T22:27:30.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[51620]: debug 2022-01-31T22:27:30.139+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.140431+0000) 2022-01-31T22:27:30.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[51620]: debug 2022-01-31T22:27:30.187+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.188668+0000) 2022-01-31T22:27:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:30 smithi171 conmon[41853]: debug 2022-01-31T22:27:30.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.534162+0000) 2022-01-31T22:27:30.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:30 smithi167 conmon[49112]: debug 2022-01-31T22:27:30.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.659471+0000) 2022-01-31T22:27:30.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:30 smithi167 conmon[54076]: debug 2022-01-31T22:27:30.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.829589+0000) 2022-01-31T22:27:30.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:30 smithi167 conmon[60316]: debug 2022-01-31T22:27:30.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:30.586901+0000) 2022-01-31T22:27:31.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:31 smithi171 conmon[46715]: debug 2022-01-31T22:27:31.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:31.328241+0000) 2022-01-31T22:27:31.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:31 smithi171 conmon[51620]: debug 2022-01-31T22:27:31.188+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:31.188832+0000) 2022-01-31T22:27:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:31 smithi171 conmon[41853]: debug 2022-01-31T22:27:31.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:31.534322+0000) 2022-01-31T22:27:31.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:31 smithi167 conmon[49112]: debug 2022-01-31T22:27:31.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:31.659609+0000) 2022-01-31T22:27:31.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:31 smithi167 conmon[54076]: debug 2022-01-31T22:27:31.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:31.829705+0000) 2022-01-31T22:27:31.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:31 smithi167 conmon[60316]: debug 2022-01-31T22:27:31.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:31.587033+0000) 2022-01-31T22:27:32.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:32 smithi171 conmon[46715]: debug 2022-01-31T22:27:32.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:32.328428+0000) 2022-01-31T22:27:32.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:32 smithi171 conmon[51620]: debug 2022-01-31T22:27:32.188+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:32.188991+0000) 2022-01-31T22:27:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:32 smithi171 conmon[41853]: debug 2022-01-31T22:27:32.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:32.534485+0000) 2022-01-31T22:27:32.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:32 smithi167 conmon[49112]: debug 2022-01-31T22:27:32.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:32.659765+0000) 2022-01-31T22:27:32.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:32 smithi167 conmon[54076]: debug 2022-01-31T22:27:32.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:32.829877+0000) 2022-01-31T22:27:32.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:32 smithi167 conmon[60316]: debug 2022-01-31T22:27:32.585+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:32.587151+0000) 2022-01-31T22:27:33.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:33 smithi171 conmon[46715]: debug 2022-01-31T22:27:33.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:33.328555+0000) 2022-01-31T22:27:33.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:33 smithi171 conmon[51620]: debug 2022-01-31T22:27:33.188+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:33.189165+0000) 2022-01-31T22:27:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:33 smithi171 conmon[41853]: debug 2022-01-31T22:27:33.533+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:33.534659+0000) 2022-01-31T22:27:33.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:33 smithi167 conmon[54076]: debug 2022-01-31T22:27:33.828+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:33.830063+0000) 2022-01-31T22:27:33.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:33 smithi167 conmon[49112]: debug 2022-01-31T22:27:33.658+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:33.659945+0000) 2022-01-31T22:27:33.882 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:33 smithi167 conmon[60316]: debug 2022-01-31T22:27:33.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:33.587342+0000) 2022-01-31T22:27:34.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:34 smithi171 conmon[46715]: debug 2022-01-31T22:27:34.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:34.328709+0000) 2022-01-31T22:27:34.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:34 smithi171 conmon[51620]: debug 2022-01-31T22:27:34.188+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:34.189389+0000) 2022-01-31T22:27:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:34 smithi171 conmon[41853]: debug 2022-01-31T22:27:34.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:34.534838+0000) 2022-01-31T22:27:34.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:34 smithi167 conmon[49112]: debug 2022-01-31T22:27:34.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:34.660172+0000) 2022-01-31T22:27:34.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:34 smithi167 conmon[54076]: debug 2022-01-31T22:27:34.829+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:34.830235+0000) 2022-01-31T22:27:34.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:34 smithi167 conmon[60316]: debug 2022-01-31T22:27:34.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:34.587523+0000) 2022-01-31T22:27:35.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[54076]: debug 2022-01-31T22:27:35.164+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.166208+0000) 2022-01-31T22:27:35.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[60316]: debug 2022-01-31T22:27:35.163+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.165175+0000) 2022-01-31T22:27:35.420 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[49112]: debug 2022-01-31T22:27:35.421 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[49112]: 2022-01-31T22:27:35.163+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.165293+0000) 2022-01-31T22:27:35.493 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[35325]: debug 2022-01-31T22:27:35.185+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211141 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:35.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[41853]: debug 2022-01-31T22:27:35.164+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.164854+0000) 2022-01-31T22:27:35.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[46715]: debug 2022-01-31T22:27:35.164+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.165390+0000) 2022-01-31T22:27:35.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[46715]: debug 2022-01-31T22:27:35.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.328853+0000) 2022-01-31T22:27:35.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[51620]: debug 2022-01-31T22:27:35.163+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.164625+0000) 2022-01-31T22:27:35.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[51620]: debug 2022-01-31T22:27:35.188+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.189547+0000) 2022-01-31T22:27:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:35 smithi171 conmon[41853]: debug 2022-01-31T22:27:35.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.534989+0000) 2022-01-31T22:27:35.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[54076]: debug 2022-01-31T22:27:35.829+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.830445+0000) 2022-01-31T22:27:35.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[60316]: debug 2022-01-31T22:27:35.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.587659+0000) 2022-01-31T22:27:35.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:35 smithi167 conmon[49112]: debug 2022-01-31T22:27:35.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:35.660362+0000) 2022-01-31T22:27:36.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:36 smithi171 conmon[46715]: debug 2022-01-31T22:27:36.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:36.329000+0000) 2022-01-31T22:27:36.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:36 smithi171 conmon[51620]: debug 2022-01-31T22:27:36.189+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:36.189713+0000) 2022-01-31T22:27:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:36 smithi171 conmon[41853]: debug 2022-01-31T22:27:36.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:36.535138+0000) 2022-01-31T22:27:36.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:36 smithi167 conmon[54076]: debug 2022-01-31T22:27:36.829+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:36.830593+0000) 2022-01-31T22:27:36.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:36 smithi167 conmon[60316]: debug 2022-01-31T22:27:36.586+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:36.587795+0000) 2022-01-31T22:27:36.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:36 smithi167 conmon[49112]: debug 2022-01-31T22:27:36.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:36.660532+0000) 2022-01-31T22:27:37.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:37 smithi171 conmon[46715]: debug 2022-01-31T22:27:37.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:37.329194+0000) 2022-01-31T22:27:37.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:37 smithi171 conmon[51620]: debug 2022-01-31T22:27:37.189+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:37.189919+0000) 2022-01-31T22:27:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:37 smithi171 conmon[41853]: debug 2022-01-31T22:27:37.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:37.535285+0000) 2022-01-31T22:27:37.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:37 smithi167 conmon[54076]: debug 2022-01-31T22:27:37.829+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:37.830760+0000) 2022-01-31T22:27:37.882 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:37 smithi167 conmon[60316]: debug 2022-01-31T22:27:37.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:37.587972+0000) 2022-01-31T22:27:37.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:37 smithi167 conmon[49112]: debug 2022-01-31T22:27:37.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:37.660682+0000) 2022-01-31T22:27:38.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:38 smithi171 conmon[46715]: debug 2022-01-31T22:27:38.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:38.329375+0000) 2022-01-31T22:27:38.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:38 smithi171 conmon[51620]: debug 2022-01-31T22:27:38.189+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:38.190053+0000) 2022-01-31T22:27:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:38 smithi171 conmon[41853]: debug 2022-01-31T22:27:38.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:38.535499+0000) 2022-01-31T22:27:38.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:38 smithi167 conmon[54076]: debug 2022-01-31T22:27:38.830+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:38.830997+0000) 2022-01-31T22:27:38.882 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:38 smithi167 conmon[60316]: debug 2022-01-31T22:27:38.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:38.588188+0000) 2022-01-31T22:27:38.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:38 smithi167 conmon[49112]: debug 2022-01-31T22:27:38.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:38.660869+0000) 2022-01-31T22:27:39.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:38 smithi171 conmon[35325]: debug 2022-01-31T22:27:38.856+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:27:39.457 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:39 smithi171 conmon[46715]: debug 2022-01-31T22:27:39.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:39.329554+0000) 2022-01-31T22:27:39.458 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:39 smithi171 conmon[51620]: debug 2022-01-31T22:27:39.189+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:39.190239+0000) 2022-01-31T22:27:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:39 smithi171 conmon[41853]: debug 2022-01-31T22:27:39.534+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:39.535675+0000) 2022-01-31T22:27:39.881 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:39 smithi167 conmon[49112]: debug 2022-01-31T22:27:39.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:39.661044+0000) 2022-01-31T22:27:39.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:39 smithi167 conmon[54076]: debug 2022-01-31T22:27:39.830+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:39.831159+0000) 2022-01-31T22:27:39.882 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:39 smithi167 conmon[60316]: debug 2022-01-31T22:27:39.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:39.588360+0000) 2022-01-31T22:27:40.493 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[35325]: debug 2022-01-31T22:27:40.209+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211252 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:40.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[41853]: debug 2022-01-31T22:27:40.187+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.188714+0000) 2022-01-31T22:27:40.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[46715]: debug 2022-01-31T22:27:40.188+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.188776+0000) 2022-01-31T22:27:40.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[46715]: debug 2022-01-31T22:27:40.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.329727+0000) 2022-01-31T22:27:40.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[51620]: debug 2022-01-31T22:27:40.187+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.187953+0000) 2022-01-31T22:27:40.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[51620]: debug 2022-01-31T22:27:40.189+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.190381+0000) 2022-01-31T22:27:40.588 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:40 smithi167 conmon[49112]: debug 2022-01-31T22:27:40.186+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.188442+0000) 2022-01-31T22:27:40.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:40 smithi167 conmon[54076]: debug 2022-01-31T22:27:40.188+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.189898+0000) 2022-01-31T22:27:40.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:40 smithi167 conmon[60316]: debug 2022-01-31T22:27:40.188+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.189662+0000) 2022-01-31T22:27:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:40 smithi171 conmon[41853]: debug 2022-01-31T22:27:40.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.535852+0000) 2022-01-31T22:27:40.881 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:40 smithi167 conmon[54076]: debug 2022-01-31T22:27:40.830+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.831353+0000) 2022-01-31T22:27:40.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:40 smithi167 conmon[49112]: debug 2022-01-31T22:27:40.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.661226+0000) 2022-01-31T22:27:40.882 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:40 smithi167 conmon[60316]: debug 2022-01-31T22:27:40.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:40.588549+0000) 2022-01-31T22:27:41.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:41 smithi171 conmon[46715]: debug 2022-01-31T22:27:41.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:41.329798+0000) 2022-01-31T22:27:41.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:41 smithi171 conmon[51620]: debug 2022-01-31T22:27:41.190+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:41.190542+0000) 2022-01-31T22:27:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:41 smithi171 conmon[41853]: debug 2022-01-31T22:27:41.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:41.536000+0000) 2022-01-31T22:27:41.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:41 smithi167 conmon[49112]: debug 2022-01-31T22:27:41.659+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:41.661404+0000) 2022-01-31T22:27:41.883 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:41 smithi167 conmon[54076]: debug 2022-01-31T22:27:41.830+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:41.831509+0000) 2022-01-31T22:27:41.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:41 smithi167 conmon[60316]: debug 2022-01-31T22:27:41.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:41.588663+0000) 2022-01-31T22:27:42.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:42 smithi171 conmon[46715]: debug 2022-01-31T22:27:42.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:42.330002+0000) 2022-01-31T22:27:42.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:42 smithi171 conmon[51620]: debug 2022-01-31T22:27:42.189+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:42.190692+0000) 2022-01-31T22:27:42.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:42 smithi171 conmon[41853]: debug 2022-01-31T22:27:42.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:42.536185+0000) 2022-01-31T22:27:42.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:42 smithi167 conmon[49112]: debug 2022-01-31T22:27:42.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:42.661587+0000) 2022-01-31T22:27:42.883 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:42 smithi167 conmon[54076]: debug 2022-01-31T22:27:42.830+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:42.831667+0000) 2022-01-31T22:27:42.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:42 smithi167 conmon[60316]: debug 2022-01-31T22:27:42.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:42.588814+0000) 2022-01-31T22:27:43.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:43 smithi171 conmon[46715]: debug 2022-01-31T22:27:43.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:43.330160+0000) 2022-01-31T22:27:43.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:43 smithi171 conmon[51620]: debug 2022-01-31T22:27:43.190+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:43.190895+0000) 2022-01-31T22:27:43.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:43 smithi171 conmon[41853]: debug 2022-01-31T22:27:43.535+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:43.536360+0000) 2022-01-31T22:27:43.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:43 smithi167 conmon[49112]: debug 2022-01-31T22:27:43.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:43.661752+0000) 2022-01-31T22:27:43.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:43 smithi167 conmon[54076]: debug 2022-01-31T22:27:43.831+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:43.831877+0000) 2022-01-31T22:27:43.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:43 smithi167 conmon[60316]: debug 2022-01-31T22:27:43.587+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:43.589013+0000) 2022-01-31T22:27:44.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:44 smithi171 conmon[46715]: debug 2022-01-31T22:27:44.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:44.330357+0000) 2022-01-31T22:27:44.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:44 smithi171 conmon[51620]: debug 2022-01-31T22:27:44.190+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:44.191096+0000) 2022-01-31T22:27:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:44 smithi171 conmon[41853]: debug 2022-01-31T22:27:44.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:44.536540+0000) 2022-01-31T22:27:44.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:44 smithi167 conmon[49112]: debug 2022-01-31T22:27:44.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:44.661972+0000) 2022-01-31T22:27:44.883 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:44 smithi167 conmon[54076]: debug 2022-01-31T22:27:44.830+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:44.832078+0000) 2022-01-31T22:27:44.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:44 smithi167 conmon[60316]: debug 2022-01-31T22:27:44.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:44.589143+0000) 2022-01-31T22:27:45.495 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[35325]: debug 2022-01-31T22:27:45.234+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211364 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:45.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[41853]: debug 2022-01-31T22:27:45.212+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.213038+0000) 2022-01-31T22:27:45.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[46715]: debug 2022-01-31T22:27:45.211+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.212682+0000) 2022-01-31T22:27:45.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[46715]: debug 2022-01-31T22:27:45.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.330548+0000) 2022-01-31T22:27:45.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[51620]: debug 2022-01-31T22:27:45.190+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.191277+0000) 2022-01-31T22:27:45.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[51620]: debug 2022-01-31T22:27:45.212+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.213673+0000) 2022-01-31T22:27:45.589 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:45 smithi167 conmon[49112]: debug 2022-01-31T22:27:45.212+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.214130+0000) 2022-01-31T22:27:45.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:45 smithi167 conmon[54076]: debug 2022-01-31T22:27:45.213+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.214528+0000) 2022-01-31T22:27:45.590 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:45 smithi167 conmon[60316]: debug 2022-01-31T22:27:45.212+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.214185+0000) 2022-01-31T22:27:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:45 smithi171 conmon[41853]: debug 2022-01-31T22:27:45.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.536694+0000) 2022-01-31T22:27:45.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:45 smithi167 conmon[49112]: debug 2022-01-31T22:27:45.660+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.662149+0000) 2022-01-31T22:27:45.883 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:45 smithi167 conmon[54076]: debug 2022-01-31T22:27:45.831+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.832240+0000) 2022-01-31T22:27:45.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:45 smithi167 conmon[60316]: debug 2022-01-31T22:27:45.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:45.589310+0000) 2022-01-31T22:27:46.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:46 smithi171 conmon[51620]: debug 2022-01-31T22:27:46.190+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:46.191434+0000) 2022-01-31T22:27:46.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:46 smithi171 conmon[46715]: debug 2022-01-31T22:27:46.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:46.330680+0000) 2022-01-31T22:27:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:46 smithi171 conmon[41853]: debug 2022-01-31T22:27:46.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:46.536822+0000) 2022-01-31T22:27:46.882 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:46 smithi167 conmon[49112]: debug 2022-01-31T22:27:46.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:46.662330+0000) 2022-01-31T22:27:46.884 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:46 smithi167 conmon[60316]: debug 2022-01-31T22:27:46.588+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:46.589472+0000) 2022-01-31T22:27:46.884 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:46 smithi167 conmon[54076]: debug 2022-01-31T22:27:46.831+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:46.832426+0000) 2022-01-31T22:27:47.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:47 smithi171 conmon[46715]: debug 2022-01-31T22:27:47.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:47.330840+0000) 2022-01-31T22:27:47.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:47 smithi171 conmon[51620]: debug 2022-01-31T22:27:47.191+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:47.191561+0000) 2022-01-31T22:27:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:47 smithi171 conmon[41853]: debug 2022-01-31T22:27:47.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:47.537028+0000) 2022-01-31T22:27:47.882 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:47 smithi167 conmon[54076]: debug 2022-01-31T22:27:47.831+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:47.832607+0000) 2022-01-31T22:27:47.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:47 smithi167 conmon[60316]: debug 2022-01-31T22:27:47.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:47.589682+0000) 2022-01-31T22:27:47.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:47 smithi167 conmon[49112]: debug 2022-01-31T22:27:47.661+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:47.662558+0000) 2022-01-31T22:27:48.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:48 smithi171 conmon[46715]: debug 2022-01-31T22:27:48.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:48.331022+0000) 2022-01-31T22:27:48.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:48 smithi171 conmon[51620]: debug 2022-01-31T22:27:48.191+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:48.191709+0000) 2022-01-31T22:27:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:48 smithi171 conmon[41853]: debug 2022-01-31T22:27:48.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:48.537174+0000) 2022-01-31T22:27:48.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:48 smithi167 conmon[49112]: debug 2022-01-31T22:27:48.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:48.662721+0000) 2022-01-31T22:27:48.884 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:48 smithi167 conmon[54076]: debug 2022-01-31T22:27:48.831+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:48.832755+0000) 2022-01-31T22:27:48.885 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:48 smithi167 conmon[60316]: debug 2022-01-31T22:27:48.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:48.589904+0000) 2022-01-31T22:27:49.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:49 smithi171 conmon[46715]: debug 2022-01-31T22:27:49.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:49.331223+0000) 2022-01-31T22:27:49.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:49 smithi171 conmon[51620]: debug 2022-01-31T22:27:49.191+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:49.191905+0000) 2022-01-31T22:27:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:49 smithi171 conmon[41853]: debug 2022-01-31T22:27:49.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:49.537340+0000) 2022-01-31T22:27:49.883 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:49 smithi167 conmon[60316]: debug 2022-01-31T22:27:49.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:49.590071+0000) 2022-01-31T22:27:49.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:49 smithi167 conmon[49112]: debug 2022-01-31T22:27:49.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:49.662910+0000) 2022-01-31T22:27:49.884 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:49 smithi167 conmon[54076]: debug 2022-01-31T22:27:49.832+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:49.832917+0000) 2022-01-31T22:27:50.495 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[35325]: debug 2022-01-31T22:27:50.258+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211475 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:50.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[46715]: debug 2022-01-31T22:27:50.236+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.237240+0000) 2022-01-31T22:27:50.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[46715]: debug 2022-01-31T22:27:50.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.331405+0000) 2022-01-31T22:27:50.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[41853]: debug 2022-01-31T22:27:50.237+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.238158+0000) 2022-01-31T22:27:50.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[51620]: debug 2022-01-31T22:27:50.191+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.192094+0000) 2022-01-31T22:27:50.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[51620]: debug 2022-01-31T22:27:50.238+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.239075+0000) 2022-01-31T22:27:50.589 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:50 smithi167 conmon[49112]: debug 2022-01-31T22:27:50.237+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.238485+0000) 2022-01-31T22:27:50.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:50 smithi167 conmon[54076]: debug 2022-01-31T22:27:50.237+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.238252+0000) 2022-01-31T22:27:50.591 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:50 smithi167 conmon[60316]: debug 2022-01-31T22:27:50.238+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.238834+0000) 2022-01-31T22:27:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:50 smithi171 conmon[41853]: debug 2022-01-31T22:27:50.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.537505+0000) 2022-01-31T22:27:50.883 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:50 smithi167 conmon[54076]: debug 2022-01-31T22:27:50.832+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.833111+0000) 2022-01-31T22:27:50.884 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:50 smithi167 conmon[60316]: debug 2022-01-31T22:27:50.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.590258+0000) 2022-01-31T22:27:50.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:50 smithi167 conmon[49112]: debug 2022-01-31T22:27:50.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:50.663086+0000) 2022-01-31T22:27:51.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:51 smithi171 conmon[46715]: debug 2022-01-31T22:27:51.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:51.331554+0000) 2022-01-31T22:27:51.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:51 smithi171 conmon[51620]: debug 2022-01-31T22:27:51.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:51.192281+0000) 2022-01-31T22:27:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:51 smithi171 conmon[41853]: debug 2022-01-31T22:27:51.536+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:51.537660+0000) 2022-01-31T22:27:51.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:51 smithi167 conmon[49112]: debug 2022-01-31T22:27:51.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:51.663240+0000) 2022-01-31T22:27:51.884 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:51 smithi167 conmon[54076]: debug 2022-01-31T22:27:51.832+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:51.833240+0000) 2022-01-31T22:27:51.884 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:51 smithi167 conmon[60316]: debug 2022-01-31T22:27:51.589+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:51.590416+0000) 2022-01-31T22:27:52.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:52 smithi171 conmon[46715]: debug 2022-01-31T22:27:52.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:52.331735+0000) 2022-01-31T22:27:52.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:52 smithi171 conmon[51620]: debug 2022-01-31T22:27:52.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:52.192429+0000) 2022-01-31T22:27:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:52 smithi171 conmon[41853]: debug 2022-01-31T22:27:52.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:52.537829+0000) 2022-01-31T22:27:52.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:52 smithi167 conmon[49112]: debug 2022-01-31T22:27:52.662+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:52.663459+0000) 2022-01-31T22:27:52.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:52 smithi167 conmon[54076]: debug 2022-01-31T22:27:52.832+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:52.833396+0000) 2022-01-31T22:27:52.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:52 smithi167 conmon[60316]: debug 2022-01-31T22:27:52.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:52.590617+0000) 2022-01-31T22:27:53.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:53 smithi171 conmon[46715]: debug 2022-01-31T22:27:53.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:53.331955+0000) 2022-01-31T22:27:53.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:53 smithi171 conmon[51620]: debug 2022-01-31T22:27:53.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:53.192620+0000) 2022-01-31T22:27:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:53 smithi171 conmon[41853]: debug 2022-01-31T22:27:53.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:53.537968+0000) 2022-01-31T22:27:53.883 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:53 smithi167 conmon[49112]: debug 2022-01-31T22:27:53.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:53.663614+0000) 2022-01-31T22:27:53.884 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:53 smithi167 conmon[54076]: debug 2022-01-31T22:27:53.833+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:53.833581+0000) 2022-01-31T22:27:53.885 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:53 smithi167 conmon[60316]: debug 2022-01-31T22:27:53.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:53.590834+0000) 2022-01-31T22:27:54.192 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:53 smithi171 conmon[35325]: debug 2022-01-31T22:27:53.857+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:27:54.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:54 smithi171 conmon[46715]: debug 2022-01-31T22:27:54.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:54.332142+0000) 2022-01-31T22:27:54.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:54 smithi171 conmon[51620]: debug 2022-01-31T22:27:54.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:54.192815+0000) 2022-01-31T22:27:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:54 smithi171 conmon[41853]: debug 2022-01-31T22:27:54.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:54.538133+0000) 2022-01-31T22:27:54.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:54 smithi167 conmon[49112]: debug 2022-01-31T22:27:54.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:54.663771+0000) 2022-01-31T22:27:54.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:54 smithi167 conmon[54076]: debug 2022-01-31T22:27:54.833+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:54.833738+0000) 2022-01-31T22:27:54.885 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:54 smithi167 conmon[60316]: debug 2022-01-31T22:27:54.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:54.591031+0000) 2022-01-31T22:27:55.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:55 smithi167 conmon[49112]: debug 2022-01-31T22:27:55.261+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.262189+0000) 2022-01-31T22:27:55.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:55 smithi167 conmon[60316]: debug 2022-01-31T22:27:55.261+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.262369+0000) 2022-01-31T22:27:55.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:55 smithi167 conmon[54076]: debug 2022-01-31T22:27:55.262+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.262626+0000) 2022-01-31T22:27:55.496 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[35325]: debug 2022-01-31T22:27:55.288+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211589 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:27:55.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[41853]: debug 2022-01-31T22:27:55.262+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.263314+0000) 2022-01-31T22:27:55.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[46715]: debug 2022-01-31T22:27:55.260+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.261437+0000) 2022-01-31T22:27:55.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[46715]: debug 2022-01-31T22:27:55.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.332337+0000) 2022-01-31T22:27:55.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[51620]: debug 2022-01-31T22:27:55.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.192999+0000) 2022-01-31T22:27:55.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[51620]: debug 2022-01-31T22:27:55.262+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.263190+0000) 2022-01-31T22:27:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:55 smithi171 conmon[41853]: debug 2022-01-31T22:27:55.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.538336+0000) 2022-01-31T22:27:55.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:55 smithi167 conmon[49112]: debug 2022-01-31T22:27:55.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.663968+0000) 2022-01-31T22:27:55.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:55 smithi167 conmon[54076]: debug 2022-01-31T22:27:55.833+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.833897+0000) 2022-01-31T22:27:55.885 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:55 smithi167 conmon[60316]: debug 2022-01-31T22:27:55.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:55.591202+0000) 2022-01-31T22:27:56.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:56 smithi171 conmon[46715]: debug 2022-01-31T22:27:56.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:56.332470+0000) 2022-01-31T22:27:56.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:56 smithi171 conmon[51620]: debug 2022-01-31T22:27:56.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:56.193188+0000) 2022-01-31T22:27:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:56 smithi171 conmon[41853]: debug 2022-01-31T22:27:56.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:56.538490+0000) 2022-01-31T22:27:56.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:56 smithi167 conmon[49112]: debug 2022-01-31T22:27:56.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:56.664134+0000) 2022-01-31T22:27:56.885 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:56 smithi167 conmon[60316]: debug 2022-01-31T22:27:56.590+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:56.591361+0000) 2022-01-31T22:27:56.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:56 smithi167 conmon[54076]: debug 2022-01-31T22:27:56.833+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:56.834047+0000) 2022-01-31T22:27:57.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:57 smithi171 conmon[46715]: debug 2022-01-31T22:27:57.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:57.332599+0000) 2022-01-31T22:27:57.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:57 smithi171 conmon[51620]: debug 2022-01-31T22:27:57.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:57.193328+0000) 2022-01-31T22:27:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:57 smithi171 conmon[41853]: debug 2022-01-31T22:27:57.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:57.538670+0000) 2022-01-31T22:27:57.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:57 smithi167 conmon[49112]: debug 2022-01-31T22:27:57.663+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:57.664332+0000) 2022-01-31T22:27:57.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:57 smithi167 conmon[54076]: debug 2022-01-31T22:27:57.833+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:57.834259+0000) 2022-01-31T22:27:57.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:57 smithi167 conmon[60316]: debug 2022-01-31T22:27:57.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:57.591523+0000) 2022-01-31T22:27:58.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:58 smithi171 conmon[46715]: debug 2022-01-31T22:27:58.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:58.332793+0000) 2022-01-31T22:27:58.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:58 smithi171 conmon[51620]: debug 2022-01-31T22:27:58.193+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:58.193486+0000) 2022-01-31T22:27:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:58 smithi171 conmon[41853]: debug 2022-01-31T22:27:58.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:58.538825+0000) 2022-01-31T22:27:58.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:58 smithi167 conmon[49112]: debug 2022-01-31T22:27:58.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:58.664494+0000) 2022-01-31T22:27:58.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:58 smithi167 conmon[54076]: debug 2022-01-31T22:27:58.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:58.834437+0000) 2022-01-31T22:27:58.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:58 smithi167 conmon[60316]: debug 2022-01-31T22:27:58.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:58.591750+0000) 2022-01-31T22:27:59.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:27:59 smithi171 conmon[46715]: debug 2022-01-31T22:27:59.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:59.332975+0000) 2022-01-31T22:27:59.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:27:59 smithi171 conmon[51620]: debug 2022-01-31T22:27:59.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:59.193631+0000) 2022-01-31T22:27:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:27:59 smithi171 conmon[41853]: debug 2022-01-31T22:27:59.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:59.538952+0000) 2022-01-31T22:27:59.884 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:27:59 smithi167 conmon[49112]: debug 2022-01-31T22:27:59.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:59.664749+0000) 2022-01-31T22:27:59.885 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:27:59 smithi167 conmon[60316]: debug 2022-01-31T22:27:59.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:59.591892+0000) 2022-01-31T22:27:59.886 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:27:59 smithi167 conmon[54076]: debug 2022-01-31T22:27:59.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:27:59.834565+0000) 2022-01-31T22:28:00.419 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[54076]: debug 2022-01-31T22:28:00.291+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.292355+0000) 2022-01-31T22:28:00.419 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[60316]: debug 2022-01-31T22:28:00.291+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.291802+0000) 2022-01-31T22:28:00.420 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[49112]: debug 2022-01-31T22:28:00.420 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[49112]: 2022-01-31T22:28:00.298+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.299260+0000) 2022-01-31T22:28:00.497 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[35325]: debug 2022-01-31T22:28:00.312+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211699 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:00.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[46715]: debug 2022-01-31T22:28:00.289+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.290845+0000) 2022-01-31T22:28:00.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[46715]: debug 2022-01-31T22:28:00.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.333136+0000) 2022-01-31T22:28:00.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[41853]: debug 2022-01-31T22:28:00.291+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.292859+0000) 2022-01-31T22:28:00.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[51620]: debug 2022-01-31T22:28:00.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.193786+0000) 2022-01-31T22:28:00.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[51620]: debug 2022-01-31T22:28:00.290+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.292023+0000) 2022-01-31T22:28:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:00 smithi171 conmon[41853]: debug 2022-01-31T22:28:00.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.539107+0000) 2022-01-31T22:28:00.885 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[49112]: debug 2022-01-31T22:28:00.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.664973+0000) 2022-01-31T22:28:00.885 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[54076]: debug 2022-01-31T22:28:00.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.834688+0000) 2022-01-31T22:28:00.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:00 smithi167 conmon[60316]: debug 2022-01-31T22:28:00.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:00.592101+0000) 2022-01-31T22:28:01.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:01 smithi171 conmon[46715]: debug 2022-01-31T22:28:01.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:01.333271+0000) 2022-01-31T22:28:01.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:01 smithi171 conmon[51620]: debug 2022-01-31T22:28:01.192+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:01.193979+0000) 2022-01-31T22:28:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:01 smithi171 conmon[41853]: debug 2022-01-31T22:28:01.537+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:01.539262+0000) 2022-01-31T22:28:01.885 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:01 smithi167 conmon[49112]: debug 2022-01-31T22:28:01.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:01.665138+0000) 2022-01-31T22:28:01.886 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:01 smithi167 conmon[54076]: debug 2022-01-31T22:28:01.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:01.834845+0000) 2022-01-31T22:28:01.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:01 smithi167 conmon[60316]: debug 2022-01-31T22:28:01.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:01.592198+0000) 2022-01-31T22:28:02.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:02 smithi171 conmon[46715]: debug 2022-01-31T22:28:02.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:02.333434+0000) 2022-01-31T22:28:02.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:02 smithi171 conmon[51620]: debug 2022-01-31T22:28:02.193+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:02.194151+0000) 2022-01-31T22:28:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:02 smithi171 conmon[41853]: debug 2022-01-31T22:28:02.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:02.539439+0000) 2022-01-31T22:28:02.885 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:02 smithi167 conmon[49112]: debug 2022-01-31T22:28:02.664+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:02.665313+0000) 2022-01-31T22:28:02.886 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:02 smithi167 conmon[54076]: debug 2022-01-31T22:28:02.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:02.835034+0000) 2022-01-31T22:28:02.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:02 smithi167 conmon[60316]: debug 2022-01-31T22:28:02.591+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:02.592322+0000) 2022-01-31T22:28:03.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:03 smithi171 conmon[46715]: debug 2022-01-31T22:28:03.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:03.333609+0000) 2022-01-31T22:28:03.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:03 smithi171 conmon[51620]: debug 2022-01-31T22:28:03.193+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:03.194341+0000) 2022-01-31T22:28:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:03 smithi171 conmon[41853]: debug 2022-01-31T22:28:03.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:03.539634+0000) 2022-01-31T22:28:03.885 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:03 smithi167 conmon[49112]: debug 2022-01-31T22:28:03.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:03.665494+0000) 2022-01-31T22:28:03.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:03 smithi167 conmon[60316]: debug 2022-01-31T22:28:03.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:03.592525+0000) 2022-01-31T22:28:03.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:03 smithi167 conmon[54076]: debug 2022-01-31T22:28:03.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:03.835226+0000) 2022-01-31T22:28:04.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:04 smithi171 conmon[51620]: debug 2022-01-31T22:28:04.193+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:04.194561+0000) 2022-01-31T22:28:04.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:04 smithi171 conmon[46715]: debug 2022-01-31T22:28:04.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:04.333811+0000) 2022-01-31T22:28:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:04 smithi171 conmon[41853]: debug 2022-01-31T22:28:04.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:04.539824+0000) 2022-01-31T22:28:04.886 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:04 smithi167 conmon[49112]: debug 2022-01-31T22:28:04.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:04.665737+0000) 2022-01-31T22:28:04.886 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:04 smithi167 conmon[60316]: debug 2022-01-31T22:28:04.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:04.592710+0000) 2022-01-31T22:28:04.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:04 smithi167 conmon[54076]: debug 2022-01-31T22:28:04.834+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:04.835409+0000) 2022-01-31T22:28:05.419 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:05 smithi167 conmon[49112]: debug 2022-01-31T22:28:05.317+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.317828+0000) 2022-01-31T22:28:05.420 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:05 smithi167 conmon[54076]: debug 2022-01-31T22:28:05.316+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.316999+0000) 2022-01-31T22:28:05.420 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:05 smithi167 conmon[60316]: debug 2022-01-31T22:28:05.316+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.316624+0000) 2022-01-31T22:28:05.498 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[35325]: debug 2022-01-31T22:28:05.336+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211811 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:05.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[41853]: debug 2022-01-31T22:28:05.315+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.316886+0000) 2022-01-31T22:28:05.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[46715]: debug 2022-01-31T22:28:05.314+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.316383+0000) 2022-01-31T22:28:05.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[46715]: debug 2022-01-31T22:28:05.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.333953+0000) 2022-01-31T22:28:05.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[51620]: debug 2022-01-31T22:28:05.193+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.194792+0000) 2022-01-31T22:28:05.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[51620]: debug 2022-01-31T22:28:05.315+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.316981+0000) 2022-01-31T22:28:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:05 smithi171 conmon[41853]: debug 2022-01-31T22:28:05.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.539972+0000) 2022-01-31T22:28:05.886 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:05 smithi167 conmon[49112]: debug 2022-01-31T22:28:05.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.665907+0000) 2022-01-31T22:28:05.887 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:05 smithi167 conmon[60316]: debug 2022-01-31T22:28:05.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.592823+0000) 2022-01-31T22:28:05.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:05 smithi167 conmon[54076]: debug 2022-01-31T22:28:05.835+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:05.835621+0000) 2022-01-31T22:28:06.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:06 smithi171 conmon[46715]: debug 2022-01-31T22:28:06.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:06.334104+0000) 2022-01-31T22:28:06.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:06 smithi171 conmon[51620]: debug 2022-01-31T22:28:06.194+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:06.194977+0000) 2022-01-31T22:28:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:06 smithi171 conmon[41853]: debug 2022-01-31T22:28:06.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:06.540091+0000) 2022-01-31T22:28:06.886 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:06 smithi167 conmon[49112]: debug 2022-01-31T22:28:06.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:06.666008+0000) 2022-01-31T22:28:06.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:06 smithi167 conmon[54076]: debug 2022-01-31T22:28:06.835+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:06.835768+0000) 2022-01-31T22:28:06.887 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:06 smithi167 conmon[60316]: debug 2022-01-31T22:28:06.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:06.592974+0000) 2022-01-31T22:28:07.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:07 smithi171 conmon[46715]: debug 2022-01-31T22:28:07.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:07.334322+0000) 2022-01-31T22:28:07.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:07 smithi171 conmon[51620]: debug 2022-01-31T22:28:07.194+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:07.195142+0000) 2022-01-31T22:28:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:07 smithi171 conmon[41853]: debug 2022-01-31T22:28:07.538+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:07.540237+0000) 2022-01-31T22:28:07.886 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:07 smithi167 conmon[49112]: debug 2022-01-31T22:28:07.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:07.666173+0000) 2022-01-31T22:28:07.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:07 smithi167 conmon[54076]: debug 2022-01-31T22:28:07.835+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:07.835926+0000) 2022-01-31T22:28:07.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:07 smithi167 conmon[60316]: debug 2022-01-31T22:28:07.592+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:07.593178+0000) 2022-01-31T22:28:08.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:08 smithi171 conmon[46715]: debug 2022-01-31T22:28:08.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:08.334481+0000) 2022-01-31T22:28:08.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:08 smithi171 conmon[51620]: debug 2022-01-31T22:28:08.194+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:08.195348+0000) 2022-01-31T22:28:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:08 smithi171 conmon[41853]: debug 2022-01-31T22:28:08.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:08.540381+0000) 2022-01-31T22:28:08.943 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:08 smithi167 conmon[49112]: debug 2022-01-31T22:28:08.665+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:08.666382+0000) 2022-01-31T22:28:08.944 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:08 smithi167 conmon[54076]: debug 2022-01-31T22:28:08.835+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:08.836106+0000) 2022-01-31T22:28:08.944 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:08 smithi167 conmon[60316]: debug 2022-01-31T22:28:08.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:08.593378+0000) 2022-01-31T22:28:09.195 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:08 smithi171 conmon[35325]: debug 2022-01-31T22:28:08.857+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:28:09.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:09 smithi171 conmon[46715]: debug 2022-01-31T22:28:09.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:09.334630+0000) 2022-01-31T22:28:09.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:09 smithi171 conmon[51620]: debug 2022-01-31T22:28:09.194+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:09.195523+0000) 2022-01-31T22:28:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:09 smithi171 conmon[41853]: debug 2022-01-31T22:28:09.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:09.540506+0000) 2022-01-31T22:28:09.888 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:09 smithi167 conmon[49112]: debug 2022-01-31T22:28:09.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:09.666585+0000) 2022-01-31T22:28:09.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:09 smithi167 conmon[54076]: debug 2022-01-31T22:28:09.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:09.836269+0000) 2022-01-31T22:28:09.890 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:09 smithi167 conmon[60316]: debug 2022-01-31T22:28:09.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:09.593513+0000) 2022-01-31T22:28:10.499 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[35325]: debug 2022-01-31T22:28:10.362+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 211921 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:10.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[41853]: debug 2022-01-31T22:28:10.339+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.341475+0000) 2022-01-31T22:28:10.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[46715]: debug 2022-01-31T22:28:10.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.334791+0000) 2022-01-31T22:28:10.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[46715]: debug 2022-01-31T22:28:10.339+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.341300+0000) 2022-01-31T22:28:10.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[51620]: debug 2022-01-31T22:28:10.194+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.195729+0000) 2022-01-31T22:28:10.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[51620]: debug 2022-01-31T22:28:10.340+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.342121+0000) 2022-01-31T22:28:10.593 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:10 smithi167 conmon[49112]: debug 2022-01-31T22:28:10.341+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.342145+0000) 2022-01-31T22:28:10.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:10 smithi167 conmon[54076]: debug 2022-01-31T22:28:10.340+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.341095+0000) 2022-01-31T22:28:10.594 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:10 smithi167 conmon[60316]: debug 2022-01-31T22:28:10.340+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.341183+0000) 2022-01-31T22:28:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:10 smithi171 conmon[41853]: debug 2022-01-31T22:28:10.539+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.540621+0000) 2022-01-31T22:28:10.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:10 smithi167 conmon[49112]: debug 2022-01-31T22:28:10.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.666788+0000) 2022-01-31T22:28:10.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:10 smithi167 conmon[54076]: debug 2022-01-31T22:28:10.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.836516+0000) 2022-01-31T22:28:10.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:10 smithi167 conmon[60316]: debug 2022-01-31T22:28:10.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:10.593646+0000) 2022-01-31T22:28:11.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:11 smithi171 conmon[46715]: debug 2022-01-31T22:28:11.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:11.334977+0000) 2022-01-31T22:28:11.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:11 smithi171 conmon[51620]: debug 2022-01-31T22:28:11.195+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:11.195929+0000) 2022-01-31T22:28:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:11 smithi171 conmon[41853]: debug 2022-01-31T22:28:11.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:11.540767+0000) 2022-01-31T22:28:11.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:11 smithi167 conmon[49112]: debug 2022-01-31T22:28:11.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:11.666964+0000) 2022-01-31T22:28:11.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:11 smithi167 conmon[54076]: debug 2022-01-31T22:28:11.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:11.836691+0000) 2022-01-31T22:28:11.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:11 smithi167 conmon[60316]: debug 2022-01-31T22:28:11.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:11.593737+0000) 2022-01-31T22:28:12.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:12 smithi171 conmon[46715]: debug 2022-01-31T22:28:12.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:12.335134+0000) 2022-01-31T22:28:12.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:12 smithi171 conmon[51620]: debug 2022-01-31T22:28:12.195+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:12.196081+0000) 2022-01-31T22:28:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:12 smithi171 conmon[41853]: debug 2022-01-31T22:28:12.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:12.540957+0000) 2022-01-31T22:28:12.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:12 smithi167 conmon[49112]: debug 2022-01-31T22:28:12.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:12.667145+0000) 2022-01-31T22:28:12.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:12 smithi167 conmon[60316]: debug 2022-01-31T22:28:12.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:12.593895+0000) 2022-01-31T22:28:12.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:12 smithi167 conmon[54076]: debug 2022-01-31T22:28:12.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:12.836894+0000) 2022-01-31T22:28:13.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:13 smithi171 conmon[46715]: debug 2022-01-31T22:28:13.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:13.335320+0000) 2022-01-31T22:28:13.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:13 smithi171 conmon[51620]: debug 2022-01-31T22:28:13.195+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:13.196260+0000) 2022-01-31T22:28:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:13 smithi171 conmon[41853]: debug 2022-01-31T22:28:13.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:13.541153+0000) 2022-01-31T22:28:13.887 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:13 smithi167 conmon[49112]: debug 2022-01-31T22:28:13.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:13.667245+0000) 2022-01-31T22:28:13.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:13 smithi167 conmon[54076]: debug 2022-01-31T22:28:13.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:13.837064+0000) 2022-01-31T22:28:13.889 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:13 smithi167 conmon[60316]: debug 2022-01-31T22:28:13.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:13.594054+0000) 2022-01-31T22:28:14.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:14 smithi171 conmon[46715]: debug 2022-01-31T22:28:14.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:14.335490+0000) 2022-01-31T22:28:14.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:14 smithi171 conmon[51620]: debug 2022-01-31T22:28:14.195+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:14.196419+0000) 2022-01-31T22:28:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:14 smithi171 conmon[41853]: debug 2022-01-31T22:28:14.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:14.541364+0000) 2022-01-31T22:28:14.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:14 smithi167 conmon[54076]: debug 2022-01-31T22:28:14.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:14.837248+0000) 2022-01-31T22:28:14.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:14 smithi167 conmon[60316]: debug 2022-01-31T22:28:14.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:14.594267+0000) 2022-01-31T22:28:14.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:14 smithi167 conmon[49112]: debug 2022-01-31T22:28:14.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:14.667384+0000) 2022-01-31T22:28:15.500 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[35325]: debug 2022-01-31T22:28:15.386+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212033 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:15.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[41853]: debug 2022-01-31T22:28:15.365+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.366426+0000) 2022-01-31T22:28:15.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[51620]: debug 2022-01-31T22:28:15.195+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.196585+0000) 2022-01-31T22:28:15.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[51620]: debug 2022-01-31T22:28:15.365+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.366692+0000) 2022-01-31T22:28:15.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[46715]: debug 2022-01-31T22:28:15.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.335637+0000) 2022-01-31T22:28:15.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[46715]: debug 2022-01-31T22:28:15.366+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.367415+0000) 2022-01-31T22:28:15.594 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:15 smithi167 conmon[49112]: debug 2022-01-31T22:28:15.364+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.366382+0000) 2022-01-31T22:28:15.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:15 smithi167 conmon[54076]: debug 2022-01-31T22:28:15.365+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.366528+0000) 2022-01-31T22:28:15.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:15 smithi167 conmon[60316]: debug 2022-01-31T22:28:15.364+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.366240+0000) 2022-01-31T22:28:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:15 smithi171 conmon[41853]: debug 2022-01-31T22:28:15.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.541504+0000) 2022-01-31T22:28:15.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:15 smithi167 conmon[54076]: debug 2022-01-31T22:28:15.835+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.837402+0000) 2022-01-31T22:28:15.888 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:15 smithi167 conmon[49112]: debug 2022-01-31T22:28:15.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.667570+0000) 2022-01-31T22:28:15.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:15 smithi167 conmon[60316]: debug 2022-01-31T22:28:15.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:15.594425+0000) 2022-01-31T22:28:16.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:16 smithi171 conmon[46715]: debug 2022-01-31T22:28:16.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:16.335794+0000) 2022-01-31T22:28:16.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:16 smithi171 conmon[51620]: debug 2022-01-31T22:28:16.196+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:16.196783+0000) 2022-01-31T22:28:16.837 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:16 smithi167 conmon[49112]: debug 2022-01-31T22:28:16.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:16.667710+0000) 2022-01-31T22:28:16.837 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:16 smithi167 conmon[60316]: debug 2022-01-31T22:28:16.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:16.594510+0000) 2022-01-31T22:28:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:16 smithi171 conmon[41853]: debug 2022-01-31T22:28:16.540+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:16.541643+0000) 2022-01-31T22:28:17.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:16 smithi167 conmon[54076]: debug 2022-01-31T22:28:16.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:16.837555+0000) 2022-01-31T22:28:17.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:17 smithi171 conmon[46715]: debug 2022-01-31T22:28:17.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:17.335945+0000) 2022-01-31T22:28:17.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:17 smithi171 conmon[51620]: debug 2022-01-31T22:28:17.196+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:17.196958+0000) 2022-01-31T22:28:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:17 smithi171 conmon[41853]: debug 2022-01-31T22:28:17.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:17.541846+0000) 2022-01-31T22:28:17.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:17 smithi167 conmon[60316]: debug 2022-01-31T22:28:17.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:17.594679+0000) 2022-01-31T22:28:17.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:17 smithi167 conmon[49112]: debug 2022-01-31T22:28:17.666+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:17.667908+0000) 2022-01-31T22:28:17.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:17 smithi167 conmon[54076]: debug 2022-01-31T22:28:17.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:17.837706+0000) 2022-01-31T22:28:18.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:18 smithi171 conmon[46715]: debug 2022-01-31T22:28:18.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:18.336124+0000) 2022-01-31T22:28:18.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:18 smithi171 conmon[51620]: debug 2022-01-31T22:28:18.196+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:18.197173+0000) 2022-01-31T22:28:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:18 smithi171 conmon[41853]: debug 2022-01-31T22:28:18.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:18.542004+0000) 2022-01-31T22:28:18.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:18 smithi167 conmon[60316]: debug 2022-01-31T22:28:18.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:18.594782+0000) 2022-01-31T22:28:18.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:18 smithi167 conmon[49112]: debug 2022-01-31T22:28:18.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:18.668072+0000) 2022-01-31T22:28:18.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:18 smithi167 conmon[54076]: debug 2022-01-31T22:28:18.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:18.837892+0000) 2022-01-31T22:28:19.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:19 smithi171 conmon[46715]: debug 2022-01-31T22:28:19.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:19.336279+0000) 2022-01-31T22:28:19.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:19 smithi171 conmon[51620]: debug 2022-01-31T22:28:19.196+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:19.197382+0000) 2022-01-31T22:28:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:19 smithi171 conmon[41853]: debug 2022-01-31T22:28:19.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:19.542122+0000) 2022-01-31T22:28:19.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:19 smithi167 conmon[54076]: debug 2022-01-31T22:28:19.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:19.838054+0000) 2022-01-31T22:28:19.889 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:19 smithi167 conmon[60316]: debug 2022-01-31T22:28:19.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:19.595041+0000) 2022-01-31T22:28:19.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:19 smithi167 conmon[49112]: debug 2022-01-31T22:28:19.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:19.668291+0000) 2022-01-31T22:28:20.501 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[35325]: debug 2022-01-31T22:28:20.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212144 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:20.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[41853]: debug 2022-01-31T22:28:20.389+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.389802+0000) 2022-01-31T22:28:20.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[46715]: debug 2022-01-31T22:28:20.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.336463+0000) 2022-01-31T22:28:20.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[46715]: debug 2022-01-31T22:28:20.390+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.390904+0000) 2022-01-31T22:28:20.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[51620]: debug 2022-01-31T22:28:20.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.197606+0000) 2022-01-31T22:28:20.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[51620]: debug 2022-01-31T22:28:20.390+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.391335+0000) 2022-01-31T22:28:20.595 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:20 smithi167 conmon[49112]: debug 2022-01-31T22:28:20.389+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.390490+0000) 2022-01-31T22:28:20.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:20 smithi167 conmon[54076]: debug 2022-01-31T22:28:20.389+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.390765+0000) 2022-01-31T22:28:20.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:20 smithi167 conmon[60316]: debug 2022-01-31T22:28:20.389+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.390842+0000) 2022-01-31T22:28:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:20 smithi171 conmon[41853]: debug 2022-01-31T22:28:20.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.542329+0000) 2022-01-31T22:28:20.888 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:20 smithi167 conmon[49112]: debug 2022-01-31T22:28:20.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.668454+0000) 2022-01-31T22:28:20.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:20 smithi167 conmon[54076]: debug 2022-01-31T22:28:20.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.838214+0000) 2022-01-31T22:28:20.890 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:20 smithi167 conmon[60316]: debug 2022-01-31T22:28:20.593+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:20.595260+0000) 2022-01-31T22:28:21.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:21 smithi171 conmon[46715]: debug 2022-01-31T22:28:21.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:21.336653+0000) 2022-01-31T22:28:21.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:21 smithi171 conmon[51620]: debug 2022-01-31T22:28:21.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:21.197768+0000) 2022-01-31T22:28:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:21 smithi171 conmon[41853]: debug 2022-01-31T22:28:21.541+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:21.542513+0000) 2022-01-31T22:28:21.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:21 smithi167 conmon[49112]: debug 2022-01-31T22:28:21.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:21.668534+0000) 2022-01-31T22:28:21.890 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:21 smithi167 conmon[54076]: debug 2022-01-31T22:28:21.836+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:21.838371+0000) 2022-01-31T22:28:21.890 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:21 smithi167 conmon[60316]: debug 2022-01-31T22:28:21.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:21.595434+0000) 2022-01-31T22:28:22.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:22 smithi171 conmon[46715]: debug 2022-01-31T22:28:22.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:22.336836+0000) 2022-01-31T22:28:22.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:22 smithi171 conmon[51620]: debug 2022-01-31T22:28:22.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:22.197934+0000) 2022-01-31T22:28:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:22 smithi171 conmon[41853]: debug 2022-01-31T22:28:22.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:22.542697+0000) 2022-01-31T22:28:22.888 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:22 smithi167 conmon[54076]: debug 2022-01-31T22:28:22.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:22.838547+0000) 2022-01-31T22:28:22.889 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:22 smithi167 conmon[60316]: debug 2022-01-31T22:28:22.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:22.595614+0000) 2022-01-31T22:28:22.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:22 smithi167 conmon[49112]: debug 2022-01-31T22:28:22.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:22.668634+0000) 2022-01-31T22:28:23.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:23 smithi171 conmon[46715]: debug 2022-01-31T22:28:23.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:23.337038+0000) 2022-01-31T22:28:23.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:23 smithi171 conmon[51620]: debug 2022-01-31T22:28:23.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:23.198089+0000) 2022-01-31T22:28:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:23 smithi171 conmon[41853]: debug 2022-01-31T22:28:23.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:23.542933+0000) 2022-01-31T22:28:23.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:23 smithi167 conmon[54076]: debug 2022-01-31T22:28:23.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:23.838713+0000) 2022-01-31T22:28:23.890 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:23 smithi167 conmon[60316]: debug 2022-01-31T22:28:23.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:23.595760+0000) 2022-01-31T22:28:23.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:23 smithi167 conmon[49112]: debug 2022-01-31T22:28:23.667+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:23.668789+0000) 2022-01-31T22:28:24.198 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:23 smithi171 conmon[35325]: debug 2022-01-31T22:28:23.858+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:28:24.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:24 smithi171 conmon[46715]: debug 2022-01-31T22:28:24.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:24.337219+0000) 2022-01-31T22:28:24.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:24 smithi171 conmon[51620]: debug 2022-01-31T22:28:24.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:24.198244+0000) 2022-01-31T22:28:24.793 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:24 smithi171 conmon[41853]: debug 2022-01-31T22:28:24.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:24.543085+0000) 2022-01-31T22:28:24.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:24 smithi167 conmon[49112]: debug 2022-01-31T22:28:24.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:24.668983+0000) 2022-01-31T22:28:24.890 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:24 smithi167 conmon[54076]: debug 2022-01-31T22:28:24.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:24.838882+0000) 2022-01-31T22:28:24.890 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:24 smithi167 conmon[60316]: debug 2022-01-31T22:28:24.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:24.595955+0000) 2022-01-31T22:28:25.502 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[35325]: debug 2022-01-31T22:28:25.435+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212255 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:25.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[41853]: debug 2022-01-31T22:28:25.414+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.414957+0000) 2022-01-31T22:28:25.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[51620]: debug 2022-01-31T22:28:25.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.198423+0000) 2022-01-31T22:28:25.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[51620]: debug 2022-01-31T22:28:25.415+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.416589+0000) 2022-01-31T22:28:25.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[46715]: debug 2022-01-31T22:28:25.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.337843+0000) 2022-01-31T22:28:25.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[46715]: debug 2022-01-31T22:28:25.414+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.414972+0000) 2022-01-31T22:28:25.595 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:25 smithi167 conmon[49112]: debug 2022-01-31T22:28:25.413+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.415235+0000) 2022-01-31T22:28:25.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:25 smithi167 conmon[54076]: debug 2022-01-31T22:28:25.413+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.414997+0000) 2022-01-31T22:28:25.597 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:25 smithi167 conmon[60316]: debug 2022-01-31T22:28:25.414+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.416030+0000) 2022-01-31T22:28:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:25 smithi171 conmon[41853]: debug 2022-01-31T22:28:25.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.543262+0000) 2022-01-31T22:28:25.889 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:25 smithi167 conmon[60316]: debug 2022-01-31T22:28:25.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.596164+0000) 2022-01-31T22:28:25.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:25 smithi167 conmon[49112]: debug 2022-01-31T22:28:25.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.669244+0000) 2022-01-31T22:28:25.890 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:25 smithi167 conmon[54076]: debug 2022-01-31T22:28:25.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:25.839070+0000) 2022-01-31T22:28:26.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:26 smithi171 conmon[46715]: debug 2022-01-31T22:28:26.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:26.338046+0000) 2022-01-31T22:28:26.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:26 smithi171 conmon[51620]: debug 2022-01-31T22:28:26.197+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:26.198640+0000) 2022-01-31T22:28:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:26 smithi171 conmon[41853]: debug 2022-01-31T22:28:26.542+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:26.543392+0000) 2022-01-31T22:28:26.889 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:26 smithi167 conmon[49112]: debug 2022-01-31T22:28:26.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:26.669440+0000) 2022-01-31T22:28:26.890 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:26 smithi167 conmon[54076]: debug 2022-01-31T22:28:26.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:26.839180+0000) 2022-01-31T22:28:26.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:26 smithi167 conmon[60316]: debug 2022-01-31T22:28:26.594+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:26.596345+0000) 2022-01-31T22:28:27.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:27 smithi171 conmon[46715]: debug 2022-01-31T22:28:27.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:27.338203+0000) 2022-01-31T22:28:27.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:27 smithi171 conmon[51620]: debug 2022-01-31T22:28:27.198+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:27.198826+0000) 2022-01-31T22:28:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:27 smithi171 conmon[41853]: debug 2022-01-31T22:28:27.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:27.543588+0000) 2022-01-31T22:28:27.889 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:27 smithi167 conmon[54076]: debug 2022-01-31T22:28:27.837+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:27.839362+0000) 2022-01-31T22:28:27.890 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:27 smithi167 conmon[60316]: debug 2022-01-31T22:28:27.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:27.596531+0000) 2022-01-31T22:28:27.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:27 smithi167 conmon[49112]: debug 2022-01-31T22:28:27.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:27.669593+0000) 2022-01-31T22:28:28.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:28 smithi171 conmon[46715]: debug 2022-01-31T22:28:28.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:28.338361+0000) 2022-01-31T22:28:28.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:28 smithi171 conmon[51620]: debug 2022-01-31T22:28:28.198+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:28.199039+0000) 2022-01-31T22:28:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:28 smithi171 conmon[41853]: debug 2022-01-31T22:28:28.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:28.543771+0000) 2022-01-31T22:28:28.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:28 smithi167 conmon[49112]: debug 2022-01-31T22:28:28.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:28.669784+0000) 2022-01-31T22:28:28.890 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:28 smithi167 conmon[54076]: debug 2022-01-31T22:28:28.838+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:28.839519+0000) 2022-01-31T22:28:28.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:28 smithi167 conmon[60316]: debug 2022-01-31T22:28:28.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:28.596691+0000) 2022-01-31T22:28:29.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:29 smithi171 conmon[46715]: debug 2022-01-31T22:28:29.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:29.338550+0000) 2022-01-31T22:28:29.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:29 smithi171 conmon[51620]: debug 2022-01-31T22:28:29.198+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:29.199241+0000) 2022-01-31T22:28:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:29 smithi171 conmon[41853]: debug 2022-01-31T22:28:29.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:29.543966+0000) 2022-01-31T22:28:29.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:29 smithi167 conmon[49112]: debug 2022-01-31T22:28:29.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:29.669963+0000) 2022-01-31T22:28:29.891 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:29 smithi167 conmon[54076]: debug 2022-01-31T22:28:29.838+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:29.839685+0000) 2022-01-31T22:28:29.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:29 smithi167 conmon[60316]: debug 2022-01-31T22:28:29.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:29.596858+0000) 2022-01-31T22:28:30.461 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[41853]: debug 2022-01-31T22:28:30.437+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.438188+0000) 2022-01-31T22:28:30.461 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[46715]: debug 2022-01-31T22:28:30.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.338735+0000) 2022-01-31T22:28:30.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[46715]: debug 2022-01-31T22:28:30.438+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.439148+0000) 2022-01-31T22:28:30.462 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[51620]: debug 2022-01-31T22:28:30.198+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.199411+0000) 2022-01-31T22:28:30.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[51620]: debug 2022-01-31T22:28:30.439+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.440229+0000) 2022-01-31T22:28:30.596 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:30 smithi167 conmon[49112]: debug 2022-01-31T22:28:30.438+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.440027+0000) 2022-01-31T22:28:30.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:30 smithi167 conmon[54076]: debug 2022-01-31T22:28:30.437+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.439197+0000) 2022-01-31T22:28:30.598 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:30 smithi167 conmon[60316]: debug 2022-01-31T22:28:30.437+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.439297+0000) 2022-01-31T22:28:30.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[35325]: debug 2022-01-31T22:28:30.460+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212368 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:30.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:30 smithi171 conmon[41853]: debug 2022-01-31T22:28:30.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.544112+0000) 2022-01-31T22:28:30.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:30 smithi167 conmon[49112]: debug 2022-01-31T22:28:30.668+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.670146+0000) 2022-01-31T22:28:30.890 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:30 smithi167 conmon[54076]: debug 2022-01-31T22:28:30.838+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.839845+0000) 2022-01-31T22:28:30.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:30 smithi167 conmon[60316]: debug 2022-01-31T22:28:30.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:30.597104+0000) 2022-01-31T22:28:31.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:31 smithi171 conmon[46715]: debug 2022-01-31T22:28:31.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:31.338943+0000) 2022-01-31T22:28:31.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:31 smithi171 conmon[51620]: debug 2022-01-31T22:28:31.198+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:31.199578+0000) 2022-01-31T22:28:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:31 smithi171 conmon[41853]: debug 2022-01-31T22:28:31.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:31.544297+0000) 2022-01-31T22:28:31.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:31 smithi167 conmon[49112]: debug 2022-01-31T22:28:31.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:31.670299+0000) 2022-01-31T22:28:31.891 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:31 smithi167 conmon[54076]: debug 2022-01-31T22:28:31.838+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:31.840010+0000) 2022-01-31T22:28:31.892 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:31 smithi167 conmon[60316]: debug 2022-01-31T22:28:31.595+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:31.597297+0000) 2022-01-31T22:28:32.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:32 smithi171 conmon[46715]: debug 2022-01-31T22:28:32.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:32.339105+0000) 2022-01-31T22:28:32.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:32 smithi171 conmon[51620]: debug 2022-01-31T22:28:32.199+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:32.199741+0000) 2022-01-31T22:28:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:32 smithi171 conmon[41853]: debug 2022-01-31T22:28:32.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:32.544480+0000) 2022-01-31T22:28:32.890 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:32 smithi167 conmon[49112]: debug 2022-01-31T22:28:32.669+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:32.670526+0000) 2022-01-31T22:28:32.891 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:32 smithi167 conmon[54076]: debug 2022-01-31T22:28:32.839+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:32.840202+0000) 2022-01-31T22:28:32.892 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:32 smithi167 conmon[60316]: debug 2022-01-31T22:28:32.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:32.597470+0000) 2022-01-31T22:28:33.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:33 smithi171 conmon[46715]: debug 2022-01-31T22:28:33.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:33.339295+0000) 2022-01-31T22:28:33.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:33 smithi171 conmon[51620]: debug 2022-01-31T22:28:33.199+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:33.199923+0000) 2022-01-31T22:28:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:33 smithi171 conmon[41853]: debug 2022-01-31T22:28:33.543+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:33.544680+0000) 2022-01-31T22:28:33.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:33 smithi167 conmon[49112]: debug 2022-01-31T22:28:33.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:33.670734+0000) 2022-01-31T22:28:33.891 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:33 smithi167 conmon[54076]: debug 2022-01-31T22:28:33.839+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:33.840375+0000) 2022-01-31T22:28:33.892 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:33 smithi167 conmon[60316]: debug 2022-01-31T22:28:33.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:33.597604+0000) 2022-01-31T22:28:34.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:34 smithi171 conmon[46715]: debug 2022-01-31T22:28:34.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:34.339497+0000) 2022-01-31T22:28:34.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:34 smithi171 conmon[51620]: debug 2022-01-31T22:28:34.199+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:34.200150+0000) 2022-01-31T22:28:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:34 smithi171 conmon[41853]: debug 2022-01-31T22:28:34.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:34.544851+0000) 2022-01-31T22:28:34.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:34 smithi167 conmon[49112]: debug 2022-01-31T22:28:34.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:34.670924+0000) 2022-01-31T22:28:34.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:34 smithi167 conmon[54076]: debug 2022-01-31T22:28:34.839+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:34.840511+0000) 2022-01-31T22:28:34.892 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:34 smithi167 conmon[60316]: debug 2022-01-31T22:28:34.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:34.597816+0000) 2022-01-31T22:28:35.462 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[46715]: debug 2022-01-31T22:28:35.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.339680+0000) 2022-01-31T22:28:35.463 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[51620]: debug 2022-01-31T22:28:35.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.200336+0000) 2022-01-31T22:28:35.597 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:35 smithi167 conmon[49112]: debug 2022-01-31T22:28:35.463+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.464632+0000) 2022-01-31T22:28:35.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:35 smithi167 conmon[54076]: debug 2022-01-31T22:28:35.462+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.463923+0000) 2022-01-31T22:28:35.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:35 smithi167 conmon[60316]: debug 2022-01-31T22:28:35.462+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.464306+0000) 2022-01-31T22:28:35.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[35325]: debug 2022-01-31T22:28:35.485+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212478 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:35.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[46715]: debug 2022-01-31T22:28:35.464+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.464772+0000) 2022-01-31T22:28:35.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[51620]: debug 2022-01-31T22:28:35.462+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.463498+0000) 2022-01-31T22:28:35.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[41853]: debug 2022-01-31T22:28:35.463+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.463765+0000) 2022-01-31T22:28:35.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:35 smithi171 conmon[41853]: debug 2022-01-31T22:28:35.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.544995+0000) 2022-01-31T22:28:35.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:35 smithi167 conmon[60316]: debug 2022-01-31T22:28:35.596+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.597987+0000) 2022-01-31T22:28:35.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:35 smithi167 conmon[49112]: debug 2022-01-31T22:28:35.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.671145+0000) 2022-01-31T22:28:35.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:35 smithi167 conmon[54076]: debug 2022-01-31T22:28:35.839+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:35.840625+0000) 2022-01-31T22:28:36.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:36 smithi171 conmon[46715]: debug 2022-01-31T22:28:36.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:36.339864+0000) 2022-01-31T22:28:36.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:36 smithi171 conmon[51620]: debug 2022-01-31T22:28:36.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:36.200541+0000) 2022-01-31T22:28:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:36 smithi171 conmon[41853]: debug 2022-01-31T22:28:36.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:36.545116+0000) 2022-01-31T22:28:36.891 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:36 smithi167 conmon[60316]: debug 2022-01-31T22:28:36.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:36.598116+0000) 2022-01-31T22:28:36.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:36 smithi167 conmon[49112]: debug 2022-01-31T22:28:36.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:36.671307+0000) 2022-01-31T22:28:36.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:36 smithi167 conmon[54076]: debug 2022-01-31T22:28:36.840+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:36.840780+0000) 2022-01-31T22:28:37.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:37 smithi171 conmon[46715]: debug 2022-01-31T22:28:37.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:37.339949+0000) 2022-01-31T22:28:37.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:37 smithi171 conmon[51620]: debug 2022-01-31T22:28:37.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:37.200697+0000) 2022-01-31T22:28:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:37 smithi171 conmon[41853]: debug 2022-01-31T22:28:37.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:37.545296+0000) 2022-01-31T22:28:37.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:37 smithi167 conmon[49112]: debug 2022-01-31T22:28:37.670+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:37.671496+0000) 2022-01-31T22:28:37.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:37 smithi167 conmon[54076]: debug 2022-01-31T22:28:37.839+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:37.840975+0000) 2022-01-31T22:28:37.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:37 smithi167 conmon[60316]: debug 2022-01-31T22:28:37.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:37.598285+0000) 2022-01-31T22:28:38.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:38 smithi171 conmon[46715]: debug 2022-01-31T22:28:38.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:38.340109+0000) 2022-01-31T22:28:38.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:38 smithi171 conmon[51620]: debug 2022-01-31T22:28:38.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:38.200862+0000) 2022-01-31T22:28:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:38 smithi171 conmon[41853]: debug 2022-01-31T22:28:38.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:38.545424+0000) 2022-01-31T22:28:38.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:38 smithi167 conmon[49112]: debug 2022-01-31T22:28:38.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:38.671654+0000) 2022-01-31T22:28:38.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:38 smithi167 conmon[54076]: debug 2022-01-31T22:28:38.840+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:38.841139+0000) 2022-01-31T22:28:38.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:38 smithi167 conmon[60316]: debug 2022-01-31T22:28:38.597+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:38.598474+0000) 2022-01-31T22:28:39.200 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:38 smithi171 conmon[35325]: debug 2022-01-31T22:28:38.859+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:28:39.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:39 smithi171 conmon[46715]: debug 2022-01-31T22:28:39.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:39.340207+0000) 2022-01-31T22:28:39.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:39 smithi171 conmon[51620]: debug 2022-01-31T22:28:39.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:39.201047+0000) 2022-01-31T22:28:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:39 smithi171 conmon[41853]: debug 2022-01-31T22:28:39.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:39.545571+0000) 2022-01-31T22:28:39.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:39 smithi167 conmon[49112]: debug 2022-01-31T22:28:39.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:39.671834+0000) 2022-01-31T22:28:39.892 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:39 smithi167 conmon[54076]: debug 2022-01-31T22:28:39.840+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:39.841289+0000) 2022-01-31T22:28:39.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:39 smithi167 conmon[60316]: debug 2022-01-31T22:28:39.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:39.598630+0000) 2022-01-31T22:28:40.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[46715]: debug 2022-01-31T22:28:40.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.340363+0000) 2022-01-31T22:28:40.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[51620]: debug 2022-01-31T22:28:40.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.201299+0000) 2022-01-31T22:28:40.598 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:40 smithi167 conmon[49112]: debug 2022-01-31T22:28:40.489+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.489729+0000) 2022-01-31T22:28:40.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:40 smithi167 conmon[54076]: debug 2022-01-31T22:28:40.488+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.489391+0000) 2022-01-31T22:28:40.599 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:40 smithi167 conmon[60316]: debug 2022-01-31T22:28:40.489+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.489906+0000) 2022-01-31T22:28:40.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[46715]: debug 2022-01-31T22:28:40.489+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.490455+0000) 2022-01-31T22:28:40.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[51620]: debug 2022-01-31T22:28:40.488+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.489145+0000) 2022-01-31T22:28:40.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[35325]: debug 2022-01-31T22:28:40.510+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212606 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:40.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[41853]: debug 2022-01-31T22:28:40.489+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.489836+0000) 2022-01-31T22:28:40.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:40 smithi171 conmon[41853]: debug 2022-01-31T22:28:40.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.545739+0000) 2022-01-31T22:28:40.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:40 smithi167 conmon[49112]: debug 2022-01-31T22:28:40.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.672025+0000) 2022-01-31T22:28:40.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:40 smithi167 conmon[54076]: debug 2022-01-31T22:28:40.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.841535+0000) 2022-01-31T22:28:40.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:40 smithi167 conmon[60316]: debug 2022-01-31T22:28:40.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:40.598786+0000) 2022-01-31T22:28:41.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:41 smithi171 conmon[46715]: debug 2022-01-31T22:28:41.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:41.340547+0000) 2022-01-31T22:28:41.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:41 smithi171 conmon[51620]: debug 2022-01-31T22:28:41.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:41.201500+0000) 2022-01-31T22:28:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:41 smithi171 conmon[41853]: debug 2022-01-31T22:28:41.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:41.545855+0000) 2022-01-31T22:28:41.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:41 smithi167 conmon[49112]: debug 2022-01-31T22:28:41.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:41.672197+0000) 2022-01-31T22:28:41.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:41 smithi167 conmon[54076]: debug 2022-01-31T22:28:41.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:41.841612+0000) 2022-01-31T22:28:41.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:41 smithi167 conmon[60316]: debug 2022-01-31T22:28:41.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:41.598871+0000) 2022-01-31T22:28:42.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:42 smithi171 conmon[46715]: debug 2022-01-31T22:28:42.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:42.340666+0000) 2022-01-31T22:28:42.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:42 smithi171 conmon[51620]: debug 2022-01-31T22:28:42.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:42.201677+0000) 2022-01-31T22:28:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:42 smithi171 conmon[41853]: debug 2022-01-31T22:28:42.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:42.546025+0000) 2022-01-31T22:28:42.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:42 smithi167 conmon[49112]: debug 2022-01-31T22:28:42.671+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:42.672385+0000) 2022-01-31T22:28:42.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:42 smithi167 conmon[54076]: debug 2022-01-31T22:28:42.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:42.841774+0000) 2022-01-31T22:28:42.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:42 smithi167 conmon[60316]: debug 2022-01-31T22:28:42.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:42.599032+0000) 2022-01-31T22:28:43.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:43 smithi171 conmon[46715]: debug 2022-01-31T22:28:43.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:43.340873+0000) 2022-01-31T22:28:43.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:43 smithi171 conmon[51620]: debug 2022-01-31T22:28:43.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:43.201830+0000) 2022-01-31T22:28:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:43 smithi171 conmon[41853]: debug 2022-01-31T22:28:43.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:43.546240+0000) 2022-01-31T22:28:43.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:43 smithi167 conmon[49112]: debug 2022-01-31T22:28:43.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:43.672619+0000) 2022-01-31T22:28:43.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:43 smithi167 conmon[54076]: debug 2022-01-31T22:28:43.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:43.841932+0000) 2022-01-31T22:28:43.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:43 smithi167 conmon[60316]: debug 2022-01-31T22:28:43.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:43.599235+0000) 2022-01-31T22:28:44.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:44 smithi171 conmon[46715]: debug 2022-01-31T22:28:44.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:44.341071+0000) 2022-01-31T22:28:44.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:44 smithi171 conmon[51620]: debug 2022-01-31T22:28:44.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:44.202026+0000) 2022-01-31T22:28:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:44 smithi171 conmon[41853]: debug 2022-01-31T22:28:44.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:44.546448+0000) 2022-01-31T22:28:44.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:44 smithi167 conmon[49112]: debug 2022-01-31T22:28:44.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:44.672784+0000) 2022-01-31T22:28:44.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:44 smithi167 conmon[54076]: debug 2022-01-31T22:28:44.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:44.842117+0000) 2022-01-31T22:28:44.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:44 smithi167 conmon[60316]: debug 2022-01-31T22:28:44.598+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:44.599456+0000) 2022-01-31T22:28:45.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[46715]: debug 2022-01-31T22:28:45.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.341276+0000) 2022-01-31T22:28:45.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[51620]: debug 2022-01-31T22:28:45.202+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.202210+0000) 2022-01-31T22:28:45.599 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:45 smithi167 conmon[49112]: debug 2022-01-31T22:28:45.513+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.513492+0000) 2022-01-31T22:28:45.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:45 smithi167 conmon[54076]: debug 2022-01-31T22:28:45.513+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.514214+0000) 2022-01-31T22:28:45.600 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:45 smithi167 conmon[60316]: debug 2022-01-31T22:28:45.514+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.514865+0000) 2022-01-31T22:28:45.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[35325]: debug 2022-01-31T22:28:45.534+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212716 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:45.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[46715]: debug 2022-01-31T22:28:45.513+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.513875+0000) 2022-01-31T22:28:45.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[51620]: debug 2022-01-31T22:28:45.513+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.514384+0000) 2022-01-31T22:28:45.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[41853]: debug 2022-01-31T22:28:45.512+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.513740+0000) 2022-01-31T22:28:45.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:45 smithi171 conmon[41853]: debug 2022-01-31T22:28:45.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.546598+0000) 2022-01-31T22:28:45.892 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:45 smithi167 conmon[49112]: debug 2022-01-31T22:28:45.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.672957+0000) 2022-01-31T22:28:45.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:45 smithi167 conmon[54076]: debug 2022-01-31T22:28:45.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.842335+0000) 2022-01-31T22:28:45.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:45 smithi167 conmon[60316]: debug 2022-01-31T22:28:45.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:45.599669+0000) 2022-01-31T22:28:46.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:46 smithi171 conmon[46715]: debug 2022-01-31T22:28:46.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:46.341463+0000) 2022-01-31T22:28:46.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:46 smithi171 conmon[51620]: debug 2022-01-31T22:28:46.200+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:46.202395+0000) 2022-01-31T22:28:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:46 smithi171 conmon[41853]: debug 2022-01-31T22:28:46.544+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:46.546729+0000) 2022-01-31T22:28:46.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:46 smithi167 conmon[49112]: debug 2022-01-31T22:28:46.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:46.673094+0000) 2022-01-31T22:28:46.894 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:46 smithi167 conmon[54076]: debug 2022-01-31T22:28:46.841+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:46.842480+0000) 2022-01-31T22:28:46.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:46 smithi167 conmon[60316]: debug 2022-01-31T22:28:46.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:46.599820+0000) 2022-01-31T22:28:47.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:47 smithi171 conmon[46715]: debug 2022-01-31T22:28:47.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:47.341627+0000) 2022-01-31T22:28:47.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:47 smithi171 conmon[51620]: debug 2022-01-31T22:28:47.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:47.202537+0000) 2022-01-31T22:28:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:47 smithi171 conmon[41853]: debug 2022-01-31T22:28:47.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:47.546923+0000) 2022-01-31T22:28:47.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:47 smithi167 conmon[49112]: debug 2022-01-31T22:28:47.672+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:47.673293+0000) 2022-01-31T22:28:47.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:47 smithi167 conmon[60316]: debug 2022-01-31T22:28:47.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:47.600034+0000) 2022-01-31T22:28:47.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:47 smithi167 conmon[54076]: debug 2022-01-31T22:28:47.842+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:47.842685+0000) 2022-01-31T22:28:48.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:48 smithi171 conmon[46715]: debug 2022-01-31T22:28:48.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:48.341815+0000) 2022-01-31T22:28:48.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:48 smithi171 conmon[51620]: debug 2022-01-31T22:28:48.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:48.202690+0000) 2022-01-31T22:28:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:48 smithi171 conmon[41853]: debug 2022-01-31T22:28:48.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:48.547117+0000) 2022-01-31T22:28:48.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:48 smithi167 conmon[49112]: debug 2022-01-31T22:28:48.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:48.673445+0000) 2022-01-31T22:28:48.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:48 smithi167 conmon[60316]: debug 2022-01-31T22:28:48.599+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:48.600194+0000) 2022-01-31T22:28:48.894 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:48 smithi167 conmon[54076]: debug 2022-01-31T22:28:48.842+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:48.842872+0000) 2022-01-31T22:28:49.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:49 smithi171 conmon[46715]: debug 2022-01-31T22:28:49.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:49.341943+0000) 2022-01-31T22:28:49.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:49 smithi171 conmon[51620]: debug 2022-01-31T22:28:49.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:49.202868+0000) 2022-01-31T22:28:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:49 smithi171 conmon[41853]: debug 2022-01-31T22:28:49.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:49.547236+0000) 2022-01-31T22:28:49.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:49 smithi167 conmon[49112]: debug 2022-01-31T22:28:49.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:49.673644+0000) 2022-01-31T22:28:49.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:49 smithi167 conmon[60316]: debug 2022-01-31T22:28:49.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:49.600379+0000) 2022-01-31T22:28:49.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:49 smithi167 conmon[54076]: debug 2022-01-31T22:28:49.842+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:49.843059+0000) 2022-01-31T22:28:50.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[46715]: debug 2022-01-31T22:28:50.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.342122+0000) 2022-01-31T22:28:50.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[51620]: debug 2022-01-31T22:28:50.202+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.203094+0000) 2022-01-31T22:28:50.600 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[49112]: debug 2022-01-31T22:28:50.536+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.537475+0000) 2022-01-31T22:28:50.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[54076]: debug 2022-01-31T22:28:50.537+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.537637+0000) 2022-01-31T22:28:50.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[60316]: debug 2022-01-31T22:28:50.601 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[60316]: 2022-01-31T22:28:50.538+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.538634+0000) 2022-01-31T22:28:50.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[35325]: debug 2022-01-31T22:28:50.557+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212828 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:50.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[46715]: debug 2022-01-31T22:28:50.535+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.537698+0000) 2022-01-31T22:28:50.860 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[51620]: debug 2022-01-31T22:28:50.536+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.538180+0000) 2022-01-31T22:28:50.861 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[41853]: debug 2022-01-31T22:28:50.537+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.539040+0000) 2022-01-31T22:28:50.861 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:50 smithi171 conmon[41853]: debug 2022-01-31T22:28:50.545+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.547371+0000) 2022-01-31T22:28:50.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[49112]: debug 2022-01-31T22:28:50.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.673844+0000) 2022-01-31T22:28:50.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[54076]: debug 2022-01-31T22:28:50.842+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.843267+0000) 2022-01-31T22:28:50.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:50 smithi167 conmon[60316]: debug 2022-01-31T22:28:50.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:50.600551+0000) 2022-01-31T22:28:51.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:51 smithi171 conmon[46715]: debug 2022-01-31T22:28:51.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:51.342320+0000) 2022-01-31T22:28:51.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:51 smithi171 conmon[51620]: debug 2022-01-31T22:28:51.202+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:51.203272+0000) 2022-01-31T22:28:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:51 smithi171 conmon[41853]: debug 2022-01-31T22:28:51.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:51.547532+0000) 2022-01-31T22:28:51.894 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:51 smithi167 conmon[54076]: debug 2022-01-31T22:28:51.843+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:51.843423+0000) 2022-01-31T22:28:51.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:51 smithi167 conmon[60316]: debug 2022-01-31T22:28:51.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:51.600708+0000) 2022-01-31T22:28:51.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:51 smithi167 conmon[49112]: debug 2022-01-31T22:28:51.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:51.674005+0000) 2022-01-31T22:28:52.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:52 smithi171 conmon[46715]: debug 2022-01-31T22:28:52.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:52.342473+0000) 2022-01-31T22:28:52.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:52 smithi171 conmon[51620]: debug 2022-01-31T22:28:52.202+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:52.203436+0000) 2022-01-31T22:28:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:52 smithi171 conmon[41853]: debug 2022-01-31T22:28:52.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:52.547716+0000) 2022-01-31T22:28:52.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:52 smithi167 conmon[49112]: debug 2022-01-31T22:28:52.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:52.674146+0000) 2022-01-31T22:28:52.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:52 smithi167 conmon[54076]: debug 2022-01-31T22:28:52.843+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:52.843577+0000) 2022-01-31T22:28:52.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:52 smithi167 conmon[60316]: debug 2022-01-31T22:28:52.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:52.600873+0000) 2022-01-31T22:28:53.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:53 smithi171 conmon[46715]: debug 2022-01-31T22:28:53.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:53.342601+0000) 2022-01-31T22:28:53.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:53 smithi171 conmon[51620]: debug 2022-01-31T22:28:53.201+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:53.203610+0000) 2022-01-31T22:28:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:53 smithi171 conmon[41853]: debug 2022-01-31T22:28:53.546+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:53.547937+0000) 2022-01-31T22:28:53.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:53 smithi167 conmon[49112]: debug 2022-01-31T22:28:53.673+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:53.674351+0000) 2022-01-31T22:28:53.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:53 smithi167 conmon[54076]: debug 2022-01-31T22:28:53.843+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:53.843713+0000) 2022-01-31T22:28:53.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:53 smithi167 conmon[60316]: debug 2022-01-31T22:28:53.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:53.601069+0000) 2022-01-31T22:28:54.203 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:53 smithi171 conmon[35325]: debug 2022-01-31T22:28:53.859+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:28:54.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:54 smithi171 conmon[46715]: debug 2022-01-31T22:28:54.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:54.342827+0000) 2022-01-31T22:28:54.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:54 smithi171 conmon[51620]: debug 2022-01-31T22:28:54.202+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:54.203840+0000) 2022-01-31T22:28:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:54 smithi171 conmon[41853]: debug 2022-01-31T22:28:54.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:54.548131+0000) 2022-01-31T22:28:54.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:54 smithi167 conmon[60316]: debug 2022-01-31T22:28:54.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:54.601250+0000) 2022-01-31T22:28:54.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:54 smithi167 conmon[49112]: debug 2022-01-31T22:28:54.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:54.674527+0000) 2022-01-31T22:28:54.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:54 smithi167 conmon[54076]: debug 2022-01-31T22:28:54.843+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:54.843914+0000) 2022-01-31T22:28:55.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[46715]: debug 2022-01-31T22:28:55.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.343026+0000) 2022-01-31T22:28:55.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[51620]: debug 2022-01-31T22:28:55.203+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.204044+0000) 2022-01-31T22:28:55.569 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:55 smithi167 conmon[49112]: debug 2022-01-31T22:28:55.561+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.561952+0000) 2022-01-31T22:28:55.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:55 smithi167 conmon[54076]: debug 2022-01-31T22:28:55.562+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.562654+0000) 2022-01-31T22:28:55.570 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:55 smithi167 conmon[60316]: debug 2022-01-31T22:28:55.562+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.563421+0000) 2022-01-31T22:28:55.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:55 smithi167 conmon[49112]: debug 2022-01-31T22:28:55.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.674657+0000) 2022-01-31T22:28:55.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:55 smithi167 conmon[60316]: debug 2022-01-31T22:28:55.600+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.601387+0000) 2022-01-31T22:28:55.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[35325]: debug 2022-01-31T22:28:55.588+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 212939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:28:55.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[46715]: debug 2022-01-31T22:28:55.559+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.561608+0000) 2022-01-31T22:28:55.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[51620]: debug 2022-01-31T22:28:55.560+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.562569+0000) 2022-01-31T22:28:55.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[41853]: debug 2022-01-31T22:28:55.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.548330+0000) 2022-01-31T22:28:55.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:55 smithi171 conmon[41853]: debug 2022-01-31T22:28:55.560+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.561875+0000) 2022-01-31T22:28:56.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:55 smithi167 conmon[54076]: debug 2022-01-31T22:28:55.843+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:55.844130+0000) 2022-01-31T22:28:56.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:56 smithi171 conmon[46715]: debug 2022-01-31T22:28:56.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:56.343178+0000) 2022-01-31T22:28:56.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:56 smithi171 conmon[51620]: debug 2022-01-31T22:28:56.203+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:56.204264+0000) 2022-01-31T22:28:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:56 smithi171 conmon[41853]: debug 2022-01-31T22:28:56.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:56.548549+0000) 2022-01-31T22:28:56.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:56 smithi167 conmon[49112]: debug 2022-01-31T22:28:56.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:56.674797+0000) 2022-01-31T22:28:56.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:56 smithi167 conmon[54076]: debug 2022-01-31T22:28:56.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:56.844269+0000) 2022-01-31T22:28:56.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:56 smithi167 conmon[60316]: debug 2022-01-31T22:28:56.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:56.601522+0000) 2022-01-31T22:28:57.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:57 smithi171 conmon[46715]: debug 2022-01-31T22:28:57.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:57.343341+0000) 2022-01-31T22:28:57.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:57 smithi171 conmon[51620]: debug 2022-01-31T22:28:57.203+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:57.204418+0000) 2022-01-31T22:28:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:57 smithi171 conmon[41853]: debug 2022-01-31T22:28:57.547+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:57.548669+0000) 2022-01-31T22:28:57.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:57 smithi167 conmon[49112]: debug 2022-01-31T22:28:57.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:57.674960+0000) 2022-01-31T22:28:57.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:57 smithi167 conmon[54076]: debug 2022-01-31T22:28:57.843+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:57.844455+0000) 2022-01-31T22:28:57.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:57 smithi167 conmon[60316]: debug 2022-01-31T22:28:57.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:57.601681+0000) 2022-01-31T22:28:58.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:58 smithi171 conmon[46715]: debug 2022-01-31T22:28:58.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:58.343501+0000) 2022-01-31T22:28:58.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:58 smithi171 conmon[51620]: debug 2022-01-31T22:28:58.203+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:58.204618+0000) 2022-01-31T22:28:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:58 smithi171 conmon[41853]: debug 2022-01-31T22:28:58.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:58.548813+0000) 2022-01-31T22:28:58.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:58 smithi167 conmon[49112]: debug 2022-01-31T22:28:58.674+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:58.675075+0000) 2022-01-31T22:28:58.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:58 smithi167 conmon[54076]: debug 2022-01-31T22:28:58.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:58.844665+0000) 2022-01-31T22:28:58.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:58 smithi167 conmon[60316]: debug 2022-01-31T22:28:58.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:58.601871+0000) 2022-01-31T22:28:59.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:28:59 smithi171 conmon[46715]: debug 2022-01-31T22:28:59.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:59.343687+0000) 2022-01-31T22:28:59.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:28:59 smithi171 conmon[51620]: debug 2022-01-31T22:28:59.204+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:59.204814+0000) 2022-01-31T22:28:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:28:59 smithi171 conmon[41853]: debug 2022-01-31T22:28:59.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:59.548966+0000) 2022-01-31T22:28:59.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:28:59 smithi167 conmon[49112]: debug 2022-01-31T22:28:59.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:59.675289+0000) 2022-01-31T22:28:59.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:28:59 smithi167 conmon[60316]: debug 2022-01-31T22:28:59.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:59.602031+0000) 2022-01-31T22:28:59.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:28:59 smithi167 conmon[54076]: debug 2022-01-31T22:28:59.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:28:59.844841+0000) 2022-01-31T22:29:00.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[51620]: debug 2022-01-31T22:29:00.204+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.204975+0000) 2022-01-31T22:29:00.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[46715]: debug 2022-01-31T22:29:00.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.343850+0000) 2022-01-31T22:29:00.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:00 smithi167 conmon[54076]: debug 2022-01-31T22:29:00.593+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.594026+0000) 2022-01-31T22:29:00.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:00 smithi167 conmon[49112]: debug 2022-01-31T22:29:00.599+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.599541+0000) 2022-01-31T22:29:00.845 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:00 smithi167 conmon[49112]: debug 2022-01-31T22:29:00.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.675496+0000) 2022-01-31T22:29:00.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:00 smithi167 conmon[60316]: debug 2022-01-31T22:29:00.593+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.594261+0000) 2022-01-31T22:29:00.846 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:00 smithi167 conmon[60316]: debug 2022-01-31T22:29:00.601+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.602171+0000) 2022-01-31T22:29:00.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[35325]: debug 2022-01-31T22:29:00.613+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213050 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:00.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[46715]: debug 2022-01-31T22:29:00.591+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.591972+0000) 2022-01-31T22:29:00.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[51620]: debug 2022-01-31T22:29:00.591+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.592471+0000) 2022-01-31T22:29:00.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[41853]: debug 2022-01-31T22:29:00.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.549196+0000) 2022-01-31T22:29:00.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:00 smithi171 conmon[41853]: debug 2022-01-31T22:29:00.591+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.592317+0000) 2022-01-31T22:29:01.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:00 smithi167 conmon[54076]: debug 2022-01-31T22:29:00.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:00.845007+0000) 2022-01-31T22:29:01.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:01 smithi171 conmon[46715]: debug 2022-01-31T22:29:01.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:01.343949+0000) 2022-01-31T22:29:01.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:01 smithi171 conmon[51620]: debug 2022-01-31T22:29:01.204+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:01.205184+0000) 2022-01-31T22:29:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:01 smithi171 conmon[41853]: debug 2022-01-31T22:29:01.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:01.549393+0000) 2022-01-31T22:29:01.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:01 smithi167 conmon[49112]: debug 2022-01-31T22:29:01.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:01.675681+0000) 2022-01-31T22:29:01.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:01 smithi167 conmon[54076]: debug 2022-01-31T22:29:01.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:01.845148+0000) 2022-01-31T22:29:01.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:01 smithi167 conmon[60316]: debug 2022-01-31T22:29:01.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:01.602321+0000) 2022-01-31T22:29:02.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:02 smithi171 conmon[46715]: debug 2022-01-31T22:29:02.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:02.344091+0000) 2022-01-31T22:29:02.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:02 smithi171 conmon[51620]: debug 2022-01-31T22:29:02.204+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:02.205335+0000) 2022-01-31T22:29:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:02 smithi171 conmon[41853]: debug 2022-01-31T22:29:02.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:02.549537+0000) 2022-01-31T22:29:02.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:02 smithi167 conmon[49112]: debug 2022-01-31T22:29:02.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:02.675894+0000) 2022-01-31T22:29:02.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:02 smithi167 conmon[54076]: debug 2022-01-31T22:29:02.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:02.845354+0000) 2022-01-31T22:29:02.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:02 smithi167 conmon[60316]: debug 2022-01-31T22:29:02.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:02.602522+0000) 2022-01-31T22:29:03.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:03 smithi171 conmon[46715]: debug 2022-01-31T22:29:03.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:03.344304+0000) 2022-01-31T22:29:03.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:03 smithi171 conmon[51620]: debug 2022-01-31T22:29:03.204+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:03.205548+0000) 2022-01-31T22:29:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:03 smithi171 conmon[41853]: debug 2022-01-31T22:29:03.548+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:03.549682+0000) 2022-01-31T22:29:03.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:03 smithi167 conmon[49112]: debug 2022-01-31T22:29:03.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:03.676086+0000) 2022-01-31T22:29:03.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:03 smithi167 conmon[54076]: debug 2022-01-31T22:29:03.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:03.845555+0000) 2022-01-31T22:29:03.897 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:03 smithi167 conmon[60316]: debug 2022-01-31T22:29:03.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:03.602675+0000) 2022-01-31T22:29:04.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:04 smithi171 conmon[46715]: debug 2022-01-31T22:29:04.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:04.344493+0000) 2022-01-31T22:29:04.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:04 smithi171 conmon[51620]: debug 2022-01-31T22:29:04.205+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:04.205804+0000) 2022-01-31T22:29:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:04 smithi171 conmon[41853]: debug 2022-01-31T22:29:04.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:04.549869+0000) 2022-01-31T22:29:04.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:04 smithi167 conmon[49112]: debug 2022-01-31T22:29:04.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:04.676225+0000) 2022-01-31T22:29:04.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:04 smithi167 conmon[54076]: debug 2022-01-31T22:29:04.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:04.845738+0000) 2022-01-31T22:29:04.897 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:04 smithi167 conmon[60316]: debug 2022-01-31T22:29:04.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:04.602872+0000) 2022-01-31T22:29:05.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[46715]: debug 2022-01-31T22:29:05.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.344696+0000) 2022-01-31T22:29:05.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[51620]: debug 2022-01-31T22:29:05.205+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.205962+0000) 2022-01-31T22:29:05.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[35325]: debug 2022-01-31T22:29:05.637+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213161 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:05.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[46715]: debug 2022-01-31T22:29:05.615+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.616453+0000) 2022-01-31T22:29:05.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[51620]: debug 2022-01-31T22:29:05.615+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.616540+0000) 2022-01-31T22:29:05.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[41853]: debug 2022-01-31T22:29:05.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.550068+0000) 2022-01-31T22:29:05.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:05 smithi171 conmon[41853]: debug 2022-01-31T22:29:05.616+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.617489+0000) 2022-01-31T22:29:05.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:05 smithi167 conmon[49112]: debug 2022-01-31T22:29:05.616+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.617805+0000) 2022-01-31T22:29:05.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:05 smithi167 conmon[49112]: debug 2022-01-31T22:29:05.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.676384+0000) 2022-01-31T22:29:05.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:05 smithi167 conmon[54076]: debug 2022-01-31T22:29:05.616+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.618256+0000) 2022-01-31T22:29:05.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:05 smithi167 conmon[54076]: debug 2022-01-31T22:29:05.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.845901+0000) 2022-01-31T22:29:05.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:05 smithi167 conmon[60316]: debug 2022-01-31T22:29:05.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.603078+0000) 2022-01-31T22:29:05.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:05 smithi167 conmon[60316]: debug 2022-01-31T22:29:05.616+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:05.618428+0000) 2022-01-31T22:29:06.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:06 smithi171 conmon[46715]: debug 2022-01-31T22:29:06.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:06.344850+0000) 2022-01-31T22:29:06.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:06 smithi171 conmon[51620]: debug 2022-01-31T22:29:06.205+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:06.206121+0000) 2022-01-31T22:29:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:06 smithi171 conmon[41853]: debug 2022-01-31T22:29:06.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:06.550192+0000) 2022-01-31T22:29:06.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:06 smithi167 conmon[49112]: debug 2022-01-31T22:29:06.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:06.676554+0000) 2022-01-31T22:29:06.897 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:06 smithi167 conmon[60316]: debug 2022-01-31T22:29:06.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:06.603291+0000) 2022-01-31T22:29:06.898 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:06 smithi167 conmon[54076]: debug 2022-01-31T22:29:06.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:06.846041+0000) 2022-01-31T22:29:07.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:07 smithi171 conmon[46715]: debug 2022-01-31T22:29:07.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:07.344980+0000) 2022-01-31T22:29:07.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:07 smithi171 conmon[51620]: debug 2022-01-31T22:29:07.205+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:07.206278+0000) 2022-01-31T22:29:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:07 smithi171 conmon[41853]: debug 2022-01-31T22:29:07.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:07.550368+0000) 2022-01-31T22:29:07.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:07 smithi167 conmon[49112]: debug 2022-01-31T22:29:07.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:07.676766+0000) 2022-01-31T22:29:07.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:07 smithi167 conmon[54076]: debug 2022-01-31T22:29:07.844+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:07.846262+0000) 2022-01-31T22:29:07.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:07 smithi167 conmon[60316]: debug 2022-01-31T22:29:07.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:07.603468+0000) 2022-01-31T22:29:08.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:08 smithi171 conmon[46715]: debug 2022-01-31T22:29:08.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:08.345101+0000) 2022-01-31T22:29:08.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:08 smithi171 conmon[51620]: debug 2022-01-31T22:29:08.205+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:08.206475+0000) 2022-01-31T22:29:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:08 smithi171 conmon[41853]: debug 2022-01-31T22:29:08.549+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:08.550547+0000) 2022-01-31T22:29:08.896 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:08 smithi167 conmon[54076]: debug 2022-01-31T22:29:08.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:08.846393+0000) 2022-01-31T22:29:08.897 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:08 smithi167 conmon[60316]: debug 2022-01-31T22:29:08.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:08.603689+0000) 2022-01-31T22:29:08.897 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:08 smithi167 conmon[49112]: debug 2022-01-31T22:29:08.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:08.676957+0000) 2022-01-31T22:29:09.206 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:08 smithi171 conmon[35325]: debug 2022-01-31T22:29:08.860+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:29:09.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:09 smithi171 conmon[46715]: debug 2022-01-31T22:29:09.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:09.345266+0000) 2022-01-31T22:29:09.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:09 smithi171 conmon[51620]: debug 2022-01-31T22:29:09.206+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:09.206656+0000) 2022-01-31T22:29:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:09 smithi171 conmon[41853]: debug 2022-01-31T22:29:09.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:09.550714+0000) 2022-01-31T22:29:09.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:09 smithi167 conmon[54076]: debug 2022-01-31T22:29:09.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:09.846587+0000) 2022-01-31T22:29:09.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:09 smithi167 conmon[49112]: debug 2022-01-31T22:29:09.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:09.677175+0000) 2022-01-31T22:29:09.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:09 smithi167 conmon[60316]: debug 2022-01-31T22:29:09.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:09.603807+0000) 2022-01-31T22:29:10.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[46715]: debug 2022-01-31T22:29:10.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.345451+0000) 2022-01-31T22:29:10.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[51620]: debug 2022-01-31T22:29:10.206+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.206831+0000) 2022-01-31T22:29:10.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[35325]: debug 2022-01-31T22:29:10.662+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213273 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:10.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[46715]: debug 2022-01-31T22:29:10.639+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.640564+0000) 2022-01-31T22:29:10.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[51620]: debug 2022-01-31T22:29:10.639+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.640662+0000) 2022-01-31T22:29:10.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[41853]: debug 2022-01-31T22:29:10.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.550911+0000) 2022-01-31T22:29:10.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:10 smithi171 conmon[41853]: debug 2022-01-31T22:29:10.641+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.642280+0000) 2022-01-31T22:29:10.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:10 smithi167 conmon[54076]: debug 2022-01-31T22:29:10.640+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.641937+0000) 2022-01-31T22:29:10.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:10 smithi167 conmon[54076]: debug 2022-01-31T22:29:10.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.846766+0000) 2022-01-31T22:29:10.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:10 smithi167 conmon[60316]: debug 2022-01-31T22:29:10.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.603954+0000) 2022-01-31T22:29:10.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:10 smithi167 conmon[60316]: debug 2022-01-31T22:29:10.640+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.641760+0000) 2022-01-31T22:29:10.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:10 smithi167 conmon[49112]: debug 2022-01-31T22:29:10.640+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.642018+0000) 2022-01-31T22:29:10.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:10 smithi167 conmon[49112]: debug 2022-01-31T22:29:10.675+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:10.677385+0000) 2022-01-31T22:29:11.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:11 smithi171 conmon[46715]: debug 2022-01-31T22:29:11.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:11.345651+0000) 2022-01-31T22:29:11.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:11 smithi171 conmon[51620]: debug 2022-01-31T22:29:11.206+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:11.207014+0000) 2022-01-31T22:29:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:11 smithi171 conmon[41853]: debug 2022-01-31T22:29:11.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:11.551092+0000) 2022-01-31T22:29:11.897 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:11 smithi167 conmon[49112]: debug 2022-01-31T22:29:11.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:11.677544+0000) 2022-01-31T22:29:11.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:11 smithi167 conmon[54076]: debug 2022-01-31T22:29:11.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:11.846915+0000) 2022-01-31T22:29:11.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:11 smithi167 conmon[60316]: debug 2022-01-31T22:29:11.602+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:11.604141+0000) 2022-01-31T22:29:12.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:12 smithi171 conmon[46715]: debug 2022-01-31T22:29:12.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:12.345825+0000) 2022-01-31T22:29:12.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:12 smithi171 conmon[51620]: debug 2022-01-31T22:29:12.206+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:12.207184+0000) 2022-01-31T22:29:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:12 smithi171 conmon[41853]: debug 2022-01-31T22:29:12.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:12.551222+0000) 2022-01-31T22:29:12.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:12 smithi167 conmon[54076]: debug 2022-01-31T22:29:12.846+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:12.847108+0000) 2022-01-31T22:29:12.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:12 smithi167 conmon[60316]: debug 2022-01-31T22:29:12.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:12.604312+0000) 2022-01-31T22:29:12.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:12 smithi167 conmon[49112]: debug 2022-01-31T22:29:12.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:12.677730+0000) 2022-01-31T22:29:13.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:13 smithi171 conmon[46715]: debug 2022-01-31T22:29:13.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:13.346043+0000) 2022-01-31T22:29:13.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:13 smithi171 conmon[51620]: debug 2022-01-31T22:29:13.206+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:13.207380+0000) 2022-01-31T22:29:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:13 smithi171 conmon[41853]: debug 2022-01-31T22:29:13.550+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:13.551423+0000) 2022-01-31T22:29:13.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:13 smithi167 conmon[54076]: debug 2022-01-31T22:29:13.845+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:13.847289+0000) 2022-01-31T22:29:13.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:13 smithi167 conmon[60316]: debug 2022-01-31T22:29:13.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:13.604496+0000) 2022-01-31T22:29:13.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:13 smithi167 conmon[49112]: debug 2022-01-31T22:29:13.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:13.677916+0000) 2022-01-31T22:29:14.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:14 smithi171 conmon[46715]: debug 2022-01-31T22:29:14.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:14.346168+0000) 2022-01-31T22:29:14.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:14 smithi171 conmon[51620]: debug 2022-01-31T22:29:14.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:14.207625+0000) 2022-01-31T22:29:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:14 smithi171 conmon[41853]: debug 2022-01-31T22:29:14.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:14.551611+0000) 2022-01-31T22:29:14.897 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:14 smithi167 conmon[49112]: debug 2022-01-31T22:29:14.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:14.678108+0000) 2022-01-31T22:29:14.898 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:14 smithi167 conmon[54076]: debug 2022-01-31T22:29:14.846+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:14.847420+0000) 2022-01-31T22:29:14.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:14 smithi167 conmon[60316]: debug 2022-01-31T22:29:14.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:14.604709+0000) 2022-01-31T22:29:15.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[51620]: debug 2022-01-31T22:29:15.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.207822+0000) 2022-01-31T22:29:15.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[46715]: debug 2022-01-31T22:29:15.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.346269+0000) 2022-01-31T22:29:15.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[35325]: debug 2022-01-31T22:29:15.687+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213384 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:15.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[46715]: debug 2022-01-31T22:29:15.665+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.666284+0000) 2022-01-31T22:29:15.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[51620]: debug 2022-01-31T22:29:15.664+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.665342+0000) 2022-01-31T22:29:15.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[41853]: debug 2022-01-31T22:29:15.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.551819+0000) 2022-01-31T22:29:15.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:15 smithi171 conmon[41853]: debug 2022-01-31T22:29:15.665+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.666587+0000) 2022-01-31T22:29:15.898 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:15 smithi167 conmon[54076]: debug 2022-01-31T22:29:15.666+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.667609+0000) 2022-01-31T22:29:15.898 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:15 smithi167 conmon[54076]: debug 2022-01-31T22:29:15.846+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.847552+0000) 2022-01-31T22:29:15.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:15 smithi167 conmon[60316]: debug 2022-01-31T22:29:15.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.604895+0000) 2022-01-31T22:29:15.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:15 smithi167 conmon[60316]: debug 2022-01-31T22:29:15.665+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.667344+0000) 2022-01-31T22:29:15.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:15 smithi167 conmon[49112]: debug 2022-01-31T22:29:15.665+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.667027+0000) 2022-01-31T22:29:15.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:15 smithi167 conmon[49112]: debug 2022-01-31T22:29:15.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:15.678313+0000) 2022-01-31T22:29:16.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:16 smithi171 conmon[46715]: debug 2022-01-31T22:29:16.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:16.346425+0000) 2022-01-31T22:29:16.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:16 smithi171 conmon[51620]: debug 2022-01-31T22:29:16.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:16.207989+0000) 2022-01-31T22:29:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:16 smithi171 conmon[41853]: debug 2022-01-31T22:29:16.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:16.552057+0000) 2022-01-31T22:29:16.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:16 smithi167 conmon[49112]: debug 2022-01-31T22:29:16.676+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:16.678480+0000) 2022-01-31T22:29:16.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:16 smithi167 conmon[54076]: debug 2022-01-31T22:29:16.846+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:16.847707+0000) 2022-01-31T22:29:16.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:16 smithi167 conmon[60316]: debug 2022-01-31T22:29:16.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:16.605106+0000) 2022-01-31T22:29:17.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:17 smithi171 conmon[46715]: debug 2022-01-31T22:29:17.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:17.346562+0000) 2022-01-31T22:29:17.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:17 smithi171 conmon[51620]: debug 2022-01-31T22:29:17.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:17.208137+0000) 2022-01-31T22:29:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:17 smithi171 conmon[41853]: debug 2022-01-31T22:29:17.551+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:17.552194+0000) 2022-01-31T22:29:17.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:17 smithi167 conmon[49112]: debug 2022-01-31T22:29:17.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:17.678728+0000) 2022-01-31T22:29:17.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:17 smithi167 conmon[54076]: debug 2022-01-31T22:29:17.846+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:17.847846+0000) 2022-01-31T22:29:17.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:17 smithi167 conmon[60316]: debug 2022-01-31T22:29:17.603+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:17.605302+0000) 2022-01-31T22:29:18.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:18 smithi171 conmon[46715]: debug 2022-01-31T22:29:18.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:18.346719+0000) 2022-01-31T22:29:18.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:18 smithi171 conmon[51620]: debug 2022-01-31T22:29:18.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:18.208345+0000) 2022-01-31T22:29:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:18 smithi171 conmon[41853]: debug 2022-01-31T22:29:18.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:18.552374+0000) 2022-01-31T22:29:18.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:18 smithi167 conmon[49112]: debug 2022-01-31T22:29:18.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:18.678854+0000) 2022-01-31T22:29:18.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:18 smithi167 conmon[54076]: debug 2022-01-31T22:29:18.847+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:18.848056+0000) 2022-01-31T22:29:18.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:18 smithi167 conmon[60316]: debug 2022-01-31T22:29:18.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:18.605484+0000) 2022-01-31T22:29:19.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:19 smithi171 conmon[46715]: debug 2022-01-31T22:29:19.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:19.346915+0000) 2022-01-31T22:29:19.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:19 smithi171 conmon[51620]: debug 2022-01-31T22:29:19.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:19.208462+0000) 2022-01-31T22:29:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:19 smithi171 conmon[41853]: debug 2022-01-31T22:29:19.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:19.552562+0000) 2022-01-31T22:29:19.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:19 smithi167 conmon[49112]: debug 2022-01-31T22:29:19.677+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:19.679032+0000) 2022-01-31T22:29:19.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:19 smithi167 conmon[54076]: debug 2022-01-31T22:29:19.847+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:19.848210+0000) 2022-01-31T22:29:19.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:19 smithi167 conmon[60316]: debug 2022-01-31T22:29:19.604+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:19.605709+0000) 2022-01-31T22:29:20.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[46715]: debug 2022-01-31T22:29:20.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.347050+0000) 2022-01-31T22:29:20.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[51620]: debug 2022-01-31T22:29:20.207+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.208677+0000) 2022-01-31T22:29:20.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[35325]: debug 2022-01-31T22:29:20.711+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213495 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:20.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[41853]: debug 2022-01-31T22:29:20.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.552794+0000) 2022-01-31T22:29:20.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[41853]: debug 2022-01-31T22:29:20.690+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.690924+0000) 2022-01-31T22:29:20.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[46715]: debug 2022-01-31T22:29:20.689+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.690019+0000) 2022-01-31T22:29:20.860 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:20 smithi171 conmon[51620]: debug 2022-01-31T22:29:20.689+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.690093+0000) 2022-01-31T22:29:20.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:20 smithi167 conmon[49112]: debug 2022-01-31T22:29:20.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.679209+0000) 2022-01-31T22:29:20.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:20 smithi167 conmon[49112]: debug 2022-01-31T22:29:20.689+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.690756+0000) 2022-01-31T22:29:20.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:20 smithi167 conmon[54076]: debug 2022-01-31T22:29:20.690+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.691817+0000) 2022-01-31T22:29:20.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:20 smithi167 conmon[54076]: debug 2022-01-31T22:29:20.847+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.848329+0000) 2022-01-31T22:29:20.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:20 smithi167 conmon[60316]: debug 2022-01-31T22:29:20.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.605908+0000) 2022-01-31T22:29:20.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:20 smithi167 conmon[60316]: debug 2022-01-31T22:29:20.690+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:20.692169+0000) 2022-01-31T22:29:21.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:21 smithi171 conmon[46715]: debug 2022-01-31T22:29:21.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:21.347179+0000) 2022-01-31T22:29:21.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:21 smithi171 conmon[51620]: debug 2022-01-31T22:29:21.208+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:21.208842+0000) 2022-01-31T22:29:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:21 smithi171 conmon[41853]: debug 2022-01-31T22:29:21.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:21.552968+0000) 2022-01-31T22:29:21.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:21 smithi167 conmon[49112]: debug 2022-01-31T22:29:21.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:21.679402+0000) 2022-01-31T22:29:21.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:21 smithi167 conmon[54076]: debug 2022-01-31T22:29:21.846+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:21.848450+0000) 2022-01-31T22:29:21.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:21 smithi167 conmon[60316]: debug 2022-01-31T22:29:21.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:21.606096+0000) 2022-01-31T22:29:22.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:22 smithi171 conmon[46715]: debug 2022-01-31T22:29:22.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:22.347325+0000) 2022-01-31T22:29:22.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:22 smithi171 conmon[51620]: debug 2022-01-31T22:29:22.208+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:22.208948+0000) 2022-01-31T22:29:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:22 smithi171 conmon[41853]: debug 2022-01-31T22:29:22.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:22.553088+0000) 2022-01-31T22:29:22.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:22 smithi167 conmon[49112]: debug 2022-01-31T22:29:22.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:22.679580+0000) 2022-01-31T22:29:22.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:22 smithi167 conmon[54076]: debug 2022-01-31T22:29:22.847+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:22.848640+0000) 2022-01-31T22:29:22.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:22 smithi167 conmon[60316]: debug 2022-01-31T22:29:22.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:22.606252+0000) 2022-01-31T22:29:23.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:23 smithi171 conmon[46715]: debug 2022-01-31T22:29:23.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:23.347455+0000) 2022-01-31T22:29:23.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:23 smithi171 conmon[51620]: debug 2022-01-31T22:29:23.208+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:23.209149+0000) 2022-01-31T22:29:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:23 smithi171 conmon[41853]: debug 2022-01-31T22:29:23.552+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:23.553332+0000) 2022-01-31T22:29:23.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:23 smithi167 conmon[49112]: debug 2022-01-31T22:29:23.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:23.679770+0000) 2022-01-31T22:29:23.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:23 smithi167 conmon[60316]: debug 2022-01-31T22:29:23.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:23.606468+0000) 2022-01-31T22:29:23.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:23 smithi167 conmon[54076]: debug 2022-01-31T22:29:23.848+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:23.848814+0000) 2022-01-31T22:29:24.209 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:23 smithi171 conmon[35325]: debug 2022-01-31T22:29:23.861+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:29:24.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:24 smithi171 conmon[46715]: debug 2022-01-31T22:29:24.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:24.347654+0000) 2022-01-31T22:29:24.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:24 smithi171 conmon[51620]: debug 2022-01-31T22:29:24.208+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:24.209368+0000) 2022-01-31T22:29:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:24 smithi171 conmon[41853]: debug 2022-01-31T22:29:24.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:24.553533+0000) 2022-01-31T22:29:24.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:24 smithi167 conmon[49112]: debug 2022-01-31T22:29:24.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:24.679935+0000) 2022-01-31T22:29:24.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:24 smithi167 conmon[54076]: debug 2022-01-31T22:29:24.848+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:24.848977+0000) 2022-01-31T22:29:24.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:24 smithi167 conmon[60316]: debug 2022-01-31T22:29:24.605+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:24.606650+0000) 2022-01-31T22:29:25.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[46715]: debug 2022-01-31T22:29:25.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.347807+0000) 2022-01-31T22:29:25.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[51620]: debug 2022-01-31T22:29:25.209+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.209573+0000) 2022-01-31T22:29:25.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[35325]: debug 2022-01-31T22:29:25.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213606 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:25.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[46715]: debug 2022-01-31T22:29:25.715+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.715981+0000) 2022-01-31T22:29:25.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[51620]: debug 2022-01-31T22:29:25.713+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.714633+0000) 2022-01-31T22:29:25.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[41853]: debug 2022-01-31T22:29:25.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.553737+0000) 2022-01-31T22:29:25.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:25 smithi171 conmon[41853]: debug 2022-01-31T22:29:25.713+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.714277+0000) 2022-01-31T22:29:25.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:25 smithi167 conmon[54076]: debug 2022-01-31T22:29:25.713+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.715394+0000) 2022-01-31T22:29:25.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:25 smithi167 conmon[54076]: debug 2022-01-31T22:29:25.847+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.849184+0000) 2022-01-31T22:29:25.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:25 smithi167 conmon[60316]: debug 2022-01-31T22:29:25.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.606856+0000) 2022-01-31T22:29:25.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:25 smithi167 conmon[60316]: debug 2022-01-31T22:29:25.714+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.716287+0000) 2022-01-31T22:29:25.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:25 smithi167 conmon[49112]: debug 2022-01-31T22:29:25.678+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.680114+0000) 2022-01-31T22:29:25.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:25 smithi167 conmon[49112]: debug 2022-01-31T22:29:25.713+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:25.715479+0000) 2022-01-31T22:29:26.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:26 smithi171 conmon[46715]: debug 2022-01-31T22:29:26.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:26.347996+0000) 2022-01-31T22:29:26.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:26 smithi171 conmon[51620]: debug 2022-01-31T22:29:26.209+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:26.209797+0000) 2022-01-31T22:29:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:26 smithi171 conmon[41853]: debug 2022-01-31T22:29:26.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:26.553950+0000) 2022-01-31T22:29:26.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:26 smithi167 conmon[49112]: debug 2022-01-31T22:29:26.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:26.680311+0000) 2022-01-31T22:29:26.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:26 smithi167 conmon[54076]: debug 2022-01-31T22:29:26.848+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:26.849333+0000) 2022-01-31T22:29:26.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:26 smithi167 conmon[60316]: debug 2022-01-31T22:29:26.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:26.607045+0000) 2022-01-31T22:29:27.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:27 smithi171 conmon[46715]: debug 2022-01-31T22:29:27.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:27.348124+0000) 2022-01-31T22:29:27.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:27 smithi171 conmon[51620]: debug 2022-01-31T22:29:27.209+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:27.209926+0000) 2022-01-31T22:29:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:27 smithi171 conmon[41853]: debug 2022-01-31T22:29:27.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:27.554091+0000) 2022-01-31T22:29:27.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:27 smithi167 conmon[49112]: debug 2022-01-31T22:29:27.679+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:27.680450+0000) 2022-01-31T22:29:27.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:27 smithi167 conmon[54076]: debug 2022-01-31T22:29:27.848+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:27.849491+0000) 2022-01-31T22:29:27.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:27 smithi167 conmon[60316]: debug 2022-01-31T22:29:27.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:27.607216+0000) 2022-01-31T22:29:28.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:28 smithi171 conmon[46715]: debug 2022-01-31T22:29:28.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:28.348321+0000) 2022-01-31T22:29:28.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:28 smithi171 conmon[51620]: debug 2022-01-31T22:29:28.209+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:28.210100+0000) 2022-01-31T22:29:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:28 smithi171 conmon[41853]: debug 2022-01-31T22:29:28.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:28.554251+0000) 2022-01-31T22:29:28.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:28 smithi167 conmon[49112]: debug 2022-01-31T22:29:28.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:28.680588+0000) 2022-01-31T22:29:28.900 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:28 smithi167 conmon[54076]: debug 2022-01-31T22:29:28.849+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:28.849701+0000) 2022-01-31T22:29:28.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:28 smithi167 conmon[60316]: debug 2022-01-31T22:29:28.606+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:28.607412+0000) 2022-01-31T22:29:29.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:29 smithi171 conmon[46715]: debug 2022-01-31T22:29:29.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:29.348487+0000) 2022-01-31T22:29:29.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:29 smithi171 conmon[51620]: debug 2022-01-31T22:29:29.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:29.210319+0000) 2022-01-31T22:29:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:29 smithi171 conmon[41853]: debug 2022-01-31T22:29:29.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:29.554373+0000) 2022-01-31T22:29:29.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:29 smithi167 conmon[49112]: debug 2022-01-31T22:29:29.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:29.680792+0000) 2022-01-31T22:29:29.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:29 smithi167 conmon[54076]: debug 2022-01-31T22:29:29.849+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:29.849896+0000) 2022-01-31T22:29:29.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:29 smithi167 conmon[60316]: debug 2022-01-31T22:29:29.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:29.607616+0000) 2022-01-31T22:29:30.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[46715]: debug 2022-01-31T22:29:30.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.348628+0000) 2022-01-31T22:29:30.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[51620]: debug 2022-01-31T22:29:30.209+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.210538+0000) 2022-01-31T22:29:30.778 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[46715]: debug 2022-01-31T22:29:30.755+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.756727+0000) 2022-01-31T22:29:30.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[51620]: debug 2022-01-31T22:29:30.754+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.755182+0000) 2022-01-31T22:29:30.779 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[41853]: debug 2022-01-31T22:29:30.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.554589+0000) 2022-01-31T22:29:30.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[41853]: debug 2022-01-31T22:29:30.754+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.755654+0000) 2022-01-31T22:29:30.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:30 smithi167 conmon[49112]: debug 2022-01-31T22:29:30.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.680981+0000) 2022-01-31T22:29:30.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:30 smithi167 conmon[49112]: debug 2022-01-31T22:29:30.756+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.757453+0000) 2022-01-31T22:29:30.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:30 smithi167 conmon[54076]: debug 2022-01-31T22:29:30.756+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.756863+0000) 2022-01-31T22:29:30.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:30 smithi167 conmon[54076]: debug 2022-01-31T22:29:30.849+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.850059+0000) 2022-01-31T22:29:30.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:30 smithi167 conmon[60316]: debug 2022-01-31T22:29:30.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.607779+0000) 2022-01-31T22:29:30.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:30 smithi167 conmon[60316]: debug 2022-01-31T22:29:30.756+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:30.757006+0000) 2022-01-31T22:29:31.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:30 smithi171 conmon[35325]: debug 2022-01-31T22:29:30.777+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213717 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:31.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:31 smithi171 conmon[46715]: debug 2022-01-31T22:29:31.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:31.348794+0000) 2022-01-31T22:29:31.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:31 smithi171 conmon[51620]: debug 2022-01-31T22:29:31.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:31.210737+0000) 2022-01-31T22:29:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:31 smithi171 conmon[41853]: debug 2022-01-31T22:29:31.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:31.554772+0000) 2022-01-31T22:29:31.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:31 smithi167 conmon[49112]: debug 2022-01-31T22:29:31.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:31.681165+0000) 2022-01-31T22:29:31.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:31 smithi167 conmon[54076]: debug 2022-01-31T22:29:31.849+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:31.850222+0000) 2022-01-31T22:29:31.901 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:31 smithi167 conmon[60316]: debug 2022-01-31T22:29:31.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:31.607943+0000) 2022-01-31T22:29:32.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:32 smithi171 conmon[46715]: debug 2022-01-31T22:29:32.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:32.348908+0000) 2022-01-31T22:29:32.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:32 smithi171 conmon[51620]: debug 2022-01-31T22:29:32.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:32.210869+0000) 2022-01-31T22:29:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:32 smithi171 conmon[41853]: debug 2022-01-31T22:29:32.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:32.554921+0000) 2022-01-31T22:29:32.900 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:32 smithi167 conmon[49112]: debug 2022-01-31T22:29:32.680+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:32.681333+0000) 2022-01-31T22:29:32.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:32 smithi167 conmon[54076]: debug 2022-01-31T22:29:32.849+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:32.850414+0000) 2022-01-31T22:29:32.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:32 smithi167 conmon[60316]: debug 2022-01-31T22:29:32.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:32.608026+0000) 2022-01-31T22:29:33.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:33 smithi171 conmon[46715]: debug 2022-01-31T22:29:33.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:33.349066+0000) 2022-01-31T22:29:33.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:33 smithi171 conmon[51620]: debug 2022-01-31T22:29:33.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:33.211072+0000) 2022-01-31T22:29:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:33 smithi171 conmon[41853]: debug 2022-01-31T22:29:33.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:33.555114+0000) 2022-01-31T22:29:33.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:33 smithi167 conmon[49112]: debug 2022-01-31T22:29:33.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:33.681542+0000) 2022-01-31T22:29:33.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:33 smithi167 conmon[54076]: debug 2022-01-31T22:29:33.850+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:33.850600+0000) 2022-01-31T22:29:33.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:33 smithi167 conmon[60316]: debug 2022-01-31T22:29:33.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:33.608214+0000) 2022-01-31T22:29:34.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:34 smithi171 conmon[46715]: debug 2022-01-31T22:29:34.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:34.349264+0000) 2022-01-31T22:29:34.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:34 smithi171 conmon[51620]: debug 2022-01-31T22:29:34.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:34.211284+0000) 2022-01-31T22:29:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:34 smithi171 conmon[41853]: debug 2022-01-31T22:29:34.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:34.555367+0000) 2022-01-31T22:29:34.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:34 smithi167 conmon[49112]: debug 2022-01-31T22:29:34.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:34.681710+0000) 2022-01-31T22:29:34.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:34 smithi167 conmon[54076]: debug 2022-01-31T22:29:34.850+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:34.850807+0000) 2022-01-31T22:29:34.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:34 smithi167 conmon[60316]: debug 2022-01-31T22:29:34.607+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:34.608405+0000) 2022-01-31T22:29:35.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[46715]: debug 2022-01-31T22:29:35.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.349474+0000) 2022-01-31T22:29:35.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[51620]: debug 2022-01-31T22:29:35.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.211509+0000) 2022-01-31T22:29:35.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[41853]: debug 2022-01-31T22:29:35.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.555495+0000) 2022-01-31T22:29:35.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:35 smithi167 conmon[49112]: debug 2022-01-31T22:29:35.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.681852+0000) 2022-01-31T22:29:35.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:35 smithi167 conmon[49112]: debug 2022-01-31T22:29:35.781+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.782406+0000) 2022-01-31T22:29:35.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:35 smithi167 conmon[54076]: debug 2022-01-31T22:29:35.782+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.782724+0000) 2022-01-31T22:29:35.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:35 smithi167 conmon[54076]: debug 2022-01-31T22:29:35.850+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.850984+0000) 2022-01-31T22:29:35.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:35 smithi167 conmon[60316]: debug 2022-01-31T22:29:35.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.608628+0000) 2022-01-31T22:29:35.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:35 smithi167 conmon[60316]: debug 2022-01-31T22:29:35.781+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.781844+0000) 2022-01-31T22:29:36.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[35325]: debug 2022-01-31T22:29:35.801+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213828 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:36.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[41853]: debug 2022-01-31T22:29:35.779+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.780774+0000) 2022-01-31T22:29:36.109 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[46715]: debug 2022-01-31T22:29:35.779+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.781475+0000) 2022-01-31T22:29:36.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:35 smithi171 conmon[51620]: debug 2022-01-31T22:29:35.780+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:35.781879+0000) 2022-01-31T22:29:36.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:36 smithi171 conmon[46715]: debug 2022-01-31T22:29:36.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:36.349659+0000) 2022-01-31T22:29:36.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:36 smithi171 conmon[51620]: debug 2022-01-31T22:29:36.209+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:36.211663+0000) 2022-01-31T22:29:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:36 smithi171 conmon[41853]: debug 2022-01-31T22:29:36.553+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:36.555646+0000) 2022-01-31T22:29:36.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:36 smithi167 conmon[49112]: debug 2022-01-31T22:29:36.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:36.682018+0000) 2022-01-31T22:29:36.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:36 smithi167 conmon[54076]: debug 2022-01-31T22:29:36.850+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:36.851141+0000) 2022-01-31T22:29:36.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:36 smithi167 conmon[60316]: debug 2022-01-31T22:29:36.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:36.608821+0000) 2022-01-31T22:29:37.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:37 smithi171 conmon[46715]: debug 2022-01-31T22:29:37.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:37.349834+0000) 2022-01-31T22:29:37.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:37 smithi171 conmon[51620]: debug 2022-01-31T22:29:37.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:37.211829+0000) 2022-01-31T22:29:37.860 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:37 smithi171 conmon[41853]: debug 2022-01-31T22:29:37.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:37.555794+0000) 2022-01-31T22:29:37.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:37 smithi167 conmon[49112]: debug 2022-01-31T22:29:37.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:37.682170+0000) 2022-01-31T22:29:37.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:37 smithi167 conmon[54076]: debug 2022-01-31T22:29:37.850+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:37.851298+0000) 2022-01-31T22:29:37.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:37 smithi167 conmon[60316]: debug 2022-01-31T22:29:37.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:37.608980+0000) 2022-01-31T22:29:38.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:38 smithi171 conmon[46715]: debug 2022-01-31T22:29:38.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:38.349996+0000) 2022-01-31T22:29:38.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:38 smithi171 conmon[51620]: debug 2022-01-31T22:29:38.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:38.211958+0000) 2022-01-31T22:29:38.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:38 smithi171 conmon[41853]: debug 2022-01-31T22:29:38.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:38.555980+0000) 2022-01-31T22:29:38.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:38 smithi167 conmon[49112]: debug 2022-01-31T22:29:38.681+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:38.682354+0000) 2022-01-31T22:29:38.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:38 smithi167 conmon[54076]: debug 2022-01-31T22:29:38.851+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:38.851480+0000) 2022-01-31T22:29:38.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:38 smithi167 conmon[60316]: debug 2022-01-31T22:29:38.608+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:38.609155+0000) 2022-01-31T22:29:39.211 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:38 smithi171 conmon[35325]: debug 2022-01-31T22:29:38.861+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:29:39.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:39 smithi171 conmon[46715]: debug 2022-01-31T22:29:39.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:39.350174+0000) 2022-01-31T22:29:39.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:39 smithi171 conmon[51620]: debug 2022-01-31T22:29:39.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:39.212080+0000) 2022-01-31T22:29:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:39 smithi171 conmon[41853]: debug 2022-01-31T22:29:39.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:39.556207+0000) 2022-01-31T22:29:39.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:39 smithi167 conmon[49112]: debug 2022-01-31T22:29:39.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:39.682556+0000) 2022-01-31T22:29:39.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:39 smithi167 conmon[54076]: debug 2022-01-31T22:29:39.851+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:39.851682+0000) 2022-01-31T22:29:39.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:39 smithi167 conmon[60316]: debug 2022-01-31T22:29:39.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:39.609348+0000) 2022-01-31T22:29:40.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[46715]: debug 2022-01-31T22:29:40.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.350442+0000) 2022-01-31T22:29:40.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[51620]: debug 2022-01-31T22:29:40.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.212314+0000) 2022-01-31T22:29:40.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[41853]: debug 2022-01-31T22:29:40.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.556407+0000) 2022-01-31T22:29:40.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:40 smithi167 conmon[54076]: debug 2022-01-31T22:29:40.807+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.807750+0000) 2022-01-31T22:29:40.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:40 smithi167 conmon[54076]: debug 2022-01-31T22:29:40.851+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.851836+0000) 2022-01-31T22:29:40.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:40 smithi167 conmon[60316]: debug 2022-01-31T22:29:40.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.609580+0000) 2022-01-31T22:29:40.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:40 smithi167 conmon[60316]: debug 2022-01-31T22:29:40.806+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.806894+0000) 2022-01-31T22:29:40.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:40 smithi167 conmon[49112]: debug 2022-01-31T22:29:40.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.682832+0000) 2022-01-31T22:29:40.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:40 smithi167 conmon[49112]: debug 2022-01-31T22:29:40.807+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.807593+0000) 2022-01-31T22:29:41.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[35325]: debug 2022-01-31T22:29:40.826+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 213939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:41.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[41853]: debug 2022-01-31T22:29:40.804+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.805975+0000) 2022-01-31T22:29:41.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[46715]: debug 2022-01-31T22:29:40.804+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.806100+0000) 2022-01-31T22:29:41.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:40 smithi171 conmon[51620]: debug 2022-01-31T22:29:40.805+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:40.807026+0000) 2022-01-31T22:29:41.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:41 smithi171 conmon[46715]: debug 2022-01-31T22:29:41.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:41.350616+0000) 2022-01-31T22:29:41.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:41 smithi171 conmon[51620]: debug 2022-01-31T22:29:41.210+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:41.212408+0000) 2022-01-31T22:29:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:41 smithi171 conmon[41853]: debug 2022-01-31T22:29:41.554+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:41.556542+0000) 2022-01-31T22:29:41.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:41 smithi167 conmon[49112]: debug 2022-01-31T22:29:41.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:41.683016+0000) 2022-01-31T22:29:41.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:41 smithi167 conmon[54076]: debug 2022-01-31T22:29:41.851+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:41.852013+0000) 2022-01-31T22:29:41.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:41 smithi167 conmon[60316]: debug 2022-01-31T22:29:41.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:41.609710+0000) 2022-01-31T22:29:42.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:42 smithi171 conmon[46715]: debug 2022-01-31T22:29:42.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:42.350773+0000) 2022-01-31T22:29:42.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:42 smithi171 conmon[51620]: debug 2022-01-31T22:29:42.211+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:42.212545+0000) 2022-01-31T22:29:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:42 smithi171 conmon[41853]: debug 2022-01-31T22:29:42.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:42.556670+0000) 2022-01-31T22:29:42.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:42 smithi167 conmon[49112]: debug 2022-01-31T22:29:42.682+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:42.683201+0000) 2022-01-31T22:29:42.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:42 smithi167 conmon[54076]: debug 2022-01-31T22:29:42.851+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:42.852144+0000) 2022-01-31T22:29:42.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:42 smithi167 conmon[60316]: debug 2022-01-31T22:29:42.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:42.609813+0000) 2022-01-31T22:29:43.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:43 smithi171 conmon[46715]: debug 2022-01-31T22:29:43.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:43.350911+0000) 2022-01-31T22:29:43.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:43 smithi171 conmon[51620]: debug 2022-01-31T22:29:43.211+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:43.212738+0000) 2022-01-31T22:29:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:43 smithi171 conmon[41853]: debug 2022-01-31T22:29:43.555+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:43.556831+0000) 2022-01-31T22:29:43.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:43 smithi167 conmon[49112]: debug 2022-01-31T22:29:43.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:43.683388+0000) 2022-01-31T22:29:43.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:43 smithi167 conmon[60316]: debug 2022-01-31T22:29:43.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:43.610005+0000) 2022-01-31T22:29:43.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:43 smithi167 conmon[54076]: debug 2022-01-31T22:29:43.851+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:43.852327+0000) 2022-01-31T22:29:44.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:44 smithi171 conmon[46715]: debug 2022-01-31T22:29:44.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:44.351082+0000) 2022-01-31T22:29:44.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:44 smithi171 conmon[51620]: debug 2022-01-31T22:29:44.212+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:44.212943+0000) 2022-01-31T22:29:44.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:44 smithi171 conmon[41853]: debug 2022-01-31T22:29:44.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:44.557034+0000) 2022-01-31T22:29:44.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:44 smithi167 conmon[49112]: debug 2022-01-31T22:29:44.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:44.683626+0000) 2022-01-31T22:29:44.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:44 smithi167 conmon[54076]: debug 2022-01-31T22:29:44.852+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:44.852543+0000) 2022-01-31T22:29:44.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:44 smithi167 conmon[60316]: debug 2022-01-31T22:29:44.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:44.610203+0000) 2022-01-31T22:29:45.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[46715]: debug 2022-01-31T22:29:45.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.351272+0000) 2022-01-31T22:29:45.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[51620]: debug 2022-01-31T22:29:45.212+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.213086+0000) 2022-01-31T22:29:45.829 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[41853]: debug 2022-01-31T22:29:45.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.557184+0000) 2022-01-31T22:29:45.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:45 smithi167 conmon[49112]: debug 2022-01-31T22:29:45.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.683859+0000) 2022-01-31T22:29:45.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:45 smithi167 conmon[49112]: debug 2022-01-31T22:29:45.830+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.831381+0000) 2022-01-31T22:29:45.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:45 smithi167 conmon[60316]: debug 2022-01-31T22:29:45.609+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.610416+0000) 2022-01-31T22:29:45.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:45 smithi167 conmon[60316]: debug 2022-01-31T22:29:45.831+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.831753+0000) 2022-01-31T22:29:45.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:45 smithi167 conmon[54076]: debug 2022-01-31T22:29:45.830+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.831296+0000) 2022-01-31T22:29:45.908 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:45 smithi167 conmon[54076]: debug 2022-01-31T22:29:45.852+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.852695+0000) 2022-01-31T22:29:46.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[35325]: debug 2022-01-31T22:29:45.852+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214052 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:46.226 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[41853]: debug 2022-01-31T22:29:45.829+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.830254+0000) 2022-01-31T22:29:46.227 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[46715]: debug 2022-01-31T22:29:45.829+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.829900+0000) 2022-01-31T22:29:46.229 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:45 smithi171 conmon[51620]: debug 2022-01-31T22:29:45.829+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:45.830200+0000) 2022-01-31T22:29:46.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:46 smithi171 conmon[51620]: debug 2022-01-31T22:29:46.212+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:46.213286+0000) 2022-01-31T22:29:46.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:46 smithi171 conmon[46715]: debug 2022-01-31T22:29:46.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:46.351441+0000) 2022-01-31T22:29:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:46 smithi171 conmon[41853]: debug 2022-01-31T22:29:46.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:46.557348+0000) 2022-01-31T22:29:46.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:46 smithi167 conmon[49112]: debug 2022-01-31T22:29:46.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:46.684038+0000) 2022-01-31T22:29:46.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:46 smithi167 conmon[60316]: debug 2022-01-31T22:29:46.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:46.610590+0000) 2022-01-31T22:29:46.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:46 smithi167 conmon[54076]: debug 2022-01-31T22:29:46.852+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:46.852833+0000) 2022-01-31T22:29:47.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:47 smithi171 conmon[46715]: debug 2022-01-31T22:29:47.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:47.351524+0000) 2022-01-31T22:29:47.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:47 smithi171 conmon[51620]: debug 2022-01-31T22:29:47.212+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:47.213423+0000) 2022-01-31T22:29:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:47 smithi171 conmon[41853]: debug 2022-01-31T22:29:47.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:47.557498+0000) 2022-01-31T22:29:47.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:47 smithi167 conmon[49112]: debug 2022-01-31T22:29:47.683+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:47.684176+0000) 2022-01-31T22:29:47.905 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:47 smithi167 conmon[54076]: debug 2022-01-31T22:29:47.852+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:47.852979+0000) 2022-01-31T22:29:47.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:47 smithi167 conmon[60316]: debug 2022-01-31T22:29:47.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:47.610754+0000) 2022-01-31T22:29:48.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:48 smithi171 conmon[51620]: debug 2022-01-31T22:29:48.212+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:48.213591+0000) 2022-01-31T22:29:48.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:48 smithi171 conmon[46715]: debug 2022-01-31T22:29:48.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:48.351719+0000) 2022-01-31T22:29:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:48 smithi171 conmon[41853]: debug 2022-01-31T22:29:48.556+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:48.557603+0000) 2022-01-31T22:29:48.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:48 smithi167 conmon[49112]: debug 2022-01-31T22:29:48.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:48.684361+0000) 2022-01-31T22:29:48.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:48 smithi167 conmon[60316]: debug 2022-01-31T22:29:48.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:48.610979+0000) 2022-01-31T22:29:48.905 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:48 smithi167 conmon[54076]: debug 2022-01-31T22:29:48.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:48.853162+0000) 2022-01-31T22:29:49.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:49 smithi171 conmon[46715]: debug 2022-01-31T22:29:49.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:49.351912+0000) 2022-01-31T22:29:49.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:49 smithi171 conmon[51620]: debug 2022-01-31T22:29:49.213+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:49.213806+0000) 2022-01-31T22:29:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:49 smithi171 conmon[41853]: debug 2022-01-31T22:29:49.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:49.557747+0000) 2022-01-31T22:29:49.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:49 smithi167 conmon[49112]: debug 2022-01-31T22:29:49.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:49.684562+0000) 2022-01-31T22:29:49.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:49 smithi167 conmon[54076]: debug 2022-01-31T22:29:49.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:49.853349+0000) 2022-01-31T22:29:49.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:49 smithi167 conmon[60316]: debug 2022-01-31T22:29:49.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:49.611193+0000) 2022-01-31T22:29:50.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[46715]: debug 2022-01-31T22:29:50.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.352025+0000) 2022-01-31T22:29:50.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[51620]: debug 2022-01-31T22:29:50.213+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.214011+0000) 2022-01-31T22:29:50.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[41853]: debug 2022-01-31T22:29:50.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.557912+0000) 2022-01-31T22:29:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[41853]: debug 2022-01-31T22:29:50.853+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.854483+0000) 2022-01-31T22:29:50.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:50 smithi167 conmon[49112]: debug 2022-01-31T22:29:50.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.684729+0000) 2022-01-31T22:29:50.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:50 smithi167 conmon[49112]: debug 2022-01-31T22:29:50.856+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.856542+0000) 2022-01-31T22:29:50.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:50 smithi167 conmon[60316]: debug 2022-01-31T22:29:50.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.611358+0000) 2022-01-31T22:29:50.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:50 smithi167 conmon[60316]: debug 2022-01-31T22:29:50.856+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.856755+0000) 2022-01-31T22:29:50.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:50 smithi167 conmon[54076]: debug 2022-01-31T22:29:50.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.853542+0000) 2022-01-31T22:29:50.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:50 smithi167 conmon[54076]: debug 2022-01-31T22:29:50.856+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.856620+0000) 2022-01-31T22:29:51.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[51620]: debug 2022-01-31T22:29:50.854+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.855280+0000) 2022-01-31T22:29:51.108 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[35325]: debug 2022-01-31T22:29:50.876+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214163 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:51.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:50 smithi171 conmon[46715]: debug 2022-01-31T22:29:50.854+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:50.855156+0000) 2022-01-31T22:29:51.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:51 smithi171 conmon[46715]: debug 2022-01-31T22:29:51.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:51.352223+0000) 2022-01-31T22:29:51.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:51 smithi171 conmon[51620]: debug 2022-01-31T22:29:51.213+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:51.214217+0000) 2022-01-31T22:29:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:51 smithi171 conmon[41853]: debug 2022-01-31T22:29:51.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:51.558060+0000) 2022-01-31T22:29:51.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:51 smithi167 conmon[49112]: debug 2022-01-31T22:29:51.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:51.684951+0000) 2022-01-31T22:29:51.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:51 smithi167 conmon[54076]: debug 2022-01-31T22:29:51.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:51.853723+0000) 2022-01-31T22:29:51.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:51 smithi167 conmon[60316]: debug 2022-01-31T22:29:51.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:51.611525+0000) 2022-01-31T22:29:52.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:52 smithi171 conmon[46715]: debug 2022-01-31T22:29:52.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:52.352365+0000) 2022-01-31T22:29:52.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:52 smithi171 conmon[51620]: debug 2022-01-31T22:29:52.213+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:52.214359+0000) 2022-01-31T22:29:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:52 smithi171 conmon[41853]: debug 2022-01-31T22:29:52.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:52.558247+0000) 2022-01-31T22:29:52.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:52 smithi167 conmon[49112]: debug 2022-01-31T22:29:52.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:52.685152+0000) 2022-01-31T22:29:52.905 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:52 smithi167 conmon[54076]: debug 2022-01-31T22:29:52.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:52.853837+0000) 2022-01-31T22:29:52.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:52 smithi167 conmon[60316]: debug 2022-01-31T22:29:52.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:52.611655+0000) 2022-01-31T22:29:53.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:53 smithi171 conmon[46715]: debug 2022-01-31T22:29:53.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:53.352560+0000) 2022-01-31T22:29:53.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:53 smithi171 conmon[51620]: debug 2022-01-31T22:29:53.213+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:53.214564+0000) 2022-01-31T22:29:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:53 smithi171 conmon[41853]: debug 2022-01-31T22:29:53.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:53.558458+0000) 2022-01-31T22:29:53.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:53 smithi167 conmon[49112]: debug 2022-01-31T22:29:53.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:53.685337+0000) 2022-01-31T22:29:53.905 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:53 smithi167 conmon[54076]: debug 2022-01-31T22:29:53.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:53.854020+0000) 2022-01-31T22:29:53.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:53 smithi167 conmon[60316]: debug 2022-01-31T22:29:53.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:53.611829+0000) 2022-01-31T22:29:54.214 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:53 smithi171 conmon[35325]: debug 2022-01-31T22:29:53.862+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:29:54.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:54 smithi171 conmon[46715]: debug 2022-01-31T22:29:54.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:54.352789+0000) 2022-01-31T22:29:54.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:54 smithi171 conmon[51620]: debug 2022-01-31T22:29:54.214+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:54.214772+0000) 2022-01-31T22:29:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:54 smithi171 conmon[41853]: debug 2022-01-31T22:29:54.557+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:54.558642+0000) 2022-01-31T22:29:54.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:54 smithi167 conmon[49112]: debug 2022-01-31T22:29:54.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:54.685527+0000) 2022-01-31T22:29:54.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:54 smithi167 conmon[54076]: debug 2022-01-31T22:29:54.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:54.854172+0000) 2022-01-31T22:29:54.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:54 smithi167 conmon[60316]: debug 2022-01-31T22:29:54.610+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:54.612017+0000) 2022-01-31T22:29:55.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[46715]: debug 2022-01-31T22:29:55.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.352957+0000) 2022-01-31T22:29:55.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[51620]: debug 2022-01-31T22:29:55.214+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.214975+0000) 2022-01-31T22:29:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[41853]: debug 2022-01-31T22:29:55.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.558832+0000) 2022-01-31T22:29:55.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:55 smithi167 conmon[49112]: debug 2022-01-31T22:29:55.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.685689+0000) 2022-01-31T22:29:55.880 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:55 smithi167 conmon[54076]: debug 2022-01-31T22:29:55.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.854380+0000) 2022-01-31T22:29:55.881 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:55 smithi167 conmon[60316]: debug 2022-01-31T22:29:55.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.612208+0000) 2022-01-31T22:29:56.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:55 smithi167 conmon[49112]: debug 2022-01-31T22:29:55.879+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.880680+0000) 2022-01-31T22:29:56.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:55 smithi167 conmon[54076]: debug 2022-01-31T22:29:55.878+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.880322+0000) 2022-01-31T22:29:56.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:55 smithi167 conmon[60316]: debug 2022-01-31T22:29:55.879+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.880537+0000) 2022-01-31T22:29:56.215 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[35325]: debug 2022-01-31T22:29:55.908+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214274 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:29:56.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[41853]: debug 2022-01-31T22:29:55.878+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.879046+0000) 2022-01-31T22:29:56.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[46715]: debug 2022-01-31T22:29:55.878+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.879531+0000) 2022-01-31T22:29:56.217 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:55 smithi171 conmon[51620]: debug 2022-01-31T22:29:55.878+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:55.879132+0000) 2022-01-31T22:29:56.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:56 smithi171 conmon[46715]: debug 2022-01-31T22:29:56.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:56.353136+0000) 2022-01-31T22:29:56.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:56 smithi171 conmon[51620]: debug 2022-01-31T22:29:56.214+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:56.215190+0000) 2022-01-31T22:29:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:56 smithi171 conmon[41853]: debug 2022-01-31T22:29:56.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:56.559034+0000) 2022-01-31T22:29:56.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:56 smithi167 conmon[49112]: debug 2022-01-31T22:29:56.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:56.685881+0000) 2022-01-31T22:29:56.905 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:56 smithi167 conmon[54076]: debug 2022-01-31T22:29:56.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:56.854521+0000) 2022-01-31T22:29:56.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:56 smithi167 conmon[60316]: debug 2022-01-31T22:29:56.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:56.612364+0000) 2022-01-31T22:29:57.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:57 smithi171 conmon[46715]: debug 2022-01-31T22:29:57.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:57.353277+0000) 2022-01-31T22:29:57.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:57 smithi171 conmon[51620]: debug 2022-01-31T22:29:57.214+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:57.215356+0000) 2022-01-31T22:29:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:57 smithi171 conmon[41853]: debug 2022-01-31T22:29:57.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:57.559168+0000) 2022-01-31T22:29:57.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:57 smithi167 conmon[49112]: debug 2022-01-31T22:29:57.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:57.686037+0000) 2022-01-31T22:29:57.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:57 smithi167 conmon[54076]: debug 2022-01-31T22:29:57.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:57.854634+0000) 2022-01-31T22:29:57.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:57 smithi167 conmon[60316]: debug 2022-01-31T22:29:57.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:57.612528+0000) 2022-01-31T22:29:58.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:58 smithi171 conmon[46715]: debug 2022-01-31T22:29:58.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:58.353469+0000) 2022-01-31T22:29:58.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:58 smithi171 conmon[51620]: debug 2022-01-31T22:29:58.215+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:58.215555+0000) 2022-01-31T22:29:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:58 smithi171 conmon[41853]: debug 2022-01-31T22:29:58.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:58.559337+0000) 2022-01-31T22:29:58.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:58 smithi167 conmon[49112]: debug 2022-01-31T22:29:58.684+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:58.686228+0000) 2022-01-31T22:29:58.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:58 smithi167 conmon[54076]: debug 2022-01-31T22:29:58.853+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:58.854795+0000) 2022-01-31T22:29:58.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:58 smithi167 conmon[60316]: debug 2022-01-31T22:29:58.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:58.612680+0000) 2022-01-31T22:29:59.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:29:59 smithi171 conmon[46715]: debug 2022-01-31T22:29:59.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:59.353658+0000) 2022-01-31T22:29:59.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:29:59 smithi171 conmon[51620]: debug 2022-01-31T22:29:59.215+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:59.215781+0000) 2022-01-31T22:29:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:29:59 smithi171 conmon[41853]: debug 2022-01-31T22:29:59.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:59.559502+0000) 2022-01-31T22:29:59.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:29:59 smithi167 conmon[49112]: debug 2022-01-31T22:29:59.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:59.686432+0000) 2022-01-31T22:29:59.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:29:59 smithi167 conmon[54076]: debug 2022-01-31T22:29:59.854+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:59.854982+0000) 2022-01-31T22:29:59.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:29:59 smithi167 conmon[60316]: debug 2022-01-31T22:29:59.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:29:59.612851+0000) 2022-01-31T22:30:00.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[46715]: debug 2022-01-31T22:30:00.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.353762+0000) 2022-01-31T22:30:00.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[51620]: debug 2022-01-31T22:30:00.215+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.215993+0000) 2022-01-31T22:30:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[41853]: debug 2022-01-31T22:30:00.558+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.559708+0000) 2022-01-31T22:30:00.905 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:00 smithi167 conmon[49112]: debug 2022-01-31T22:30:00.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.686595+0000) 2022-01-31T22:30:00.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:00 smithi167 conmon[60316]: debug 2022-01-31T22:30:00.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.613036+0000) 2022-01-31T22:30:00.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:00 smithi167 conmon[54076]: debug 2022-01-31T22:30:00.854+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.855204+0000) 2022-01-31T22:30:01.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:00 smithi167 conmon[49112]: debug 2022-01-31T22:30:00.910+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.912458+0000) 2022-01-31T22:30:01.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:00 smithi167 conmon[54076]: debug 2022-01-31T22:30:00.910+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.911784+0000) 2022-01-31T22:30:01.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:00 smithi167 conmon[60316]: debug 2022-01-31T22:30:00.910+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.912349+0000) 2022-01-31T22:30:01.216 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[35325]: debug 2022-01-31T22:30:00.933+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214385 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:01.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[46715]: debug 2022-01-31T22:30:00.909+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.910735+0000) 2022-01-31T22:30:01.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[51620]: debug 2022-01-31T22:30:00.910+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.910800+0000) 2022-01-31T22:30:01.218 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:00 smithi171 conmon[41853]: debug 2022-01-31T22:30:00.910+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:00.911426+0000) 2022-01-31T22:30:01.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:01 smithi171 conmon[51620]: debug 2022-01-31T22:30:01.215+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:01.216133+0000) 2022-01-31T22:30:01.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:01 smithi171 conmon[46715]: debug 2022-01-31T22:30:01.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:01.353935+0000) 2022-01-31T22:30:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:01 smithi171 conmon[41853]: debug 2022-01-31T22:30:01.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:01.559905+0000) 2022-01-31T22:30:01.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:01 smithi167 conmon[60316]: debug 2022-01-31T22:30:01.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:01.613253+0000) 2022-01-31T22:30:01.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:01 smithi167 conmon[49112]: debug 2022-01-31T22:30:01.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:01.686803+0000) 2022-01-31T22:30:01.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:01 smithi167 conmon[54076]: debug 2022-01-31T22:30:01.854+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:01.855426+0000) 2022-01-31T22:30:02.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:02 smithi171 conmon[46715]: debug 2022-01-31T22:30:02.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:02.354093+0000) 2022-01-31T22:30:02.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:02 smithi171 conmon[51620]: debug 2022-01-31T22:30:02.215+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:02.216304+0000) 2022-01-31T22:30:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:02 smithi171 conmon[41853]: debug 2022-01-31T22:30:02.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:02.560023+0000) 2022-01-31T22:30:02.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:02 smithi167 conmon[49112]: debug 2022-01-31T22:30:02.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:02.687003+0000) 2022-01-31T22:30:02.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:02 smithi167 conmon[54076]: debug 2022-01-31T22:30:02.854+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:02.855599+0000) 2022-01-31T22:30:02.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:02 smithi167 conmon[60316]: debug 2022-01-31T22:30:02.611+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:02.613439+0000) 2022-01-31T22:30:03.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:03 smithi171 conmon[46715]: debug 2022-01-31T22:30:03.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:03.354286+0000) 2022-01-31T22:30:03.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:03 smithi171 conmon[51620]: debug 2022-01-31T22:30:03.216+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:03.216505+0000) 2022-01-31T22:30:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:03 smithi171 conmon[41853]: debug 2022-01-31T22:30:03.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:03.560215+0000) 2022-01-31T22:30:03.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:03 smithi167 conmon[49112]: debug 2022-01-31T22:30:03.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:03.687210+0000) 2022-01-31T22:30:03.906 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:03 smithi167 conmon[54076]: debug 2022-01-31T22:30:03.854+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:03.855790+0000) 2022-01-31T22:30:03.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:03 smithi167 conmon[60316]: debug 2022-01-31T22:30:03.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:03.613565+0000) 2022-01-31T22:30:04.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:04 smithi171 conmon[46715]: debug 2022-01-31T22:30:04.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:04.354435+0000) 2022-01-31T22:30:04.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:04 smithi171 conmon[51620]: debug 2022-01-31T22:30:04.216+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:04.216735+0000) 2022-01-31T22:30:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:04 smithi171 conmon[41853]: debug 2022-01-31T22:30:04.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:04.560372+0000) 2022-01-31T22:30:04.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:04 smithi167 conmon[49112]: debug 2022-01-31T22:30:04.685+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:04.687417+0000) 2022-01-31T22:30:04.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:04 smithi167 conmon[60316]: debug 2022-01-31T22:30:04.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:04.613749+0000) 2022-01-31T22:30:04.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:04 smithi167 conmon[54076]: debug 2022-01-31T22:30:04.854+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:04.856011+0000) 2022-01-31T22:30:05.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[46715]: debug 2022-01-31T22:30:05.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.354604+0000) 2022-01-31T22:30:05.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[51620]: debug 2022-01-31T22:30:05.216+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.216946+0000) 2022-01-31T22:30:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[41853]: debug 2022-01-31T22:30:05.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.560529+0000) 2022-01-31T22:30:05.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:05 smithi167 conmon[49112]: debug 2022-01-31T22:30:05.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.687627+0000) 2022-01-31T22:30:05.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:05 smithi167 conmon[60316]: debug 2022-01-31T22:30:05.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.613969+0000) 2022-01-31T22:30:05.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:05 smithi167 conmon[54076]: debug 2022-01-31T22:30:05.855+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.856198+0000) 2022-01-31T22:30:06.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:05 smithi167 conmon[49112]: debug 2022-01-31T22:30:05.936+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.937783+0000) 2022-01-31T22:30:06.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:05 smithi167 conmon[54076]: debug 2022-01-31T22:30:05.935+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.937153+0000) 2022-01-31T22:30:06.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:05 smithi167 conmon[60316]: debug 2022-01-31T22:30:05.935+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.937340+0000) 2022-01-31T22:30:06.216 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[35325]: debug 2022-01-31T22:30:05.958+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214498 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:06.217 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[41853]: debug 2022-01-31T22:30:05.936+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.937100+0000) 2022-01-31T22:30:06.217 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[46715]: debug 2022-01-31T22:30:05.935+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.936686+0000) 2022-01-31T22:30:06.218 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:05 smithi171 conmon[51620]: debug 2022-01-31T22:30:05.936+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:05.937490+0000) 2022-01-31T22:30:06.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:06 smithi171 conmon[46715]: debug 2022-01-31T22:30:06.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:06.354834+0000) 2022-01-31T22:30:06.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:06 smithi171 conmon[51620]: debug 2022-01-31T22:30:06.216+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:06.217114+0000) 2022-01-31T22:30:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:06 smithi171 conmon[41853]: debug 2022-01-31T22:30:06.559+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:06.560708+0000) 2022-01-31T22:30:06.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:06 smithi167 conmon[49112]: debug 2022-01-31T22:30:06.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:06.687784+0000) 2022-01-31T22:30:06.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:06 smithi167 conmon[54076]: debug 2022-01-31T22:30:06.855+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:06.856385+0000) 2022-01-31T22:30:06.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:06 smithi167 conmon[60316]: debug 2022-01-31T22:30:06.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:06.614174+0000) 2022-01-31T22:30:07.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:07 smithi171 conmon[46715]: debug 2022-01-31T22:30:07.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:07.355000+0000) 2022-01-31T22:30:07.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:07 smithi171 conmon[51620]: debug 2022-01-31T22:30:07.216+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:07.217301+0000) 2022-01-31T22:30:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:07 smithi171 conmon[41853]: debug 2022-01-31T22:30:07.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:07.560835+0000) 2022-01-31T22:30:07.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:07 smithi167 conmon[49112]: debug 2022-01-31T22:30:07.686+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:07.687896+0000) 2022-01-31T22:30:07.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:07 smithi167 conmon[54076]: debug 2022-01-31T22:30:07.855+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:07.856575+0000) 2022-01-31T22:30:07.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:07 smithi167 conmon[60316]: debug 2022-01-31T22:30:07.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:07.614298+0000) 2022-01-31T22:30:08.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:08 smithi171 conmon[46715]: debug 2022-01-31T22:30:08.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:08.355098+0000) 2022-01-31T22:30:08.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:08 smithi171 conmon[51620]: debug 2022-01-31T22:30:08.216+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:08.217479+0000) 2022-01-31T22:30:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:08 smithi171 conmon[41853]: debug 2022-01-31T22:30:08.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:08.560964+0000) 2022-01-31T22:30:08.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:08 smithi167 conmon[49112]: debug 2022-01-31T22:30:08.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:08.688090+0000) 2022-01-31T22:30:08.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:08 smithi167 conmon[54076]: debug 2022-01-31T22:30:08.855+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:08.856755+0000) 2022-01-31T22:30:08.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:08 smithi167 conmon[60316]: debug 2022-01-31T22:30:08.612+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:08.614485+0000) 2022-01-31T22:30:09.217 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:08 smithi171 conmon[35325]: debug 2022-01-31T22:30:08.863+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:30:09.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:09 smithi171 conmon[46715]: debug 2022-01-31T22:30:09.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:09.355229+0000) 2022-01-31T22:30:09.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:09 smithi171 conmon[51620]: debug 2022-01-31T22:30:09.217+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:09.217675+0000) 2022-01-31T22:30:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:09 smithi171 conmon[41853]: debug 2022-01-31T22:30:09.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:09.561147+0000) 2022-01-31T22:30:09.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:09 smithi167 conmon[49112]: debug 2022-01-31T22:30:09.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:09.688282+0000) 2022-01-31T22:30:09.908 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:09 smithi167 conmon[54076]: debug 2022-01-31T22:30:09.856+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:09.856938+0000) 2022-01-31T22:30:09.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:09 smithi167 conmon[60316]: debug 2022-01-31T22:30:09.613+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:09.614646+0000) 2022-01-31T22:30:10.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[46715]: debug 2022-01-31T22:30:10.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.355345+0000) 2022-01-31T22:30:10.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[51620]: debug 2022-01-31T22:30:10.217+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.217886+0000) 2022-01-31T22:30:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[41853]: debug 2022-01-31T22:30:10.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.561379+0000) 2022-01-31T22:30:10.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:10 smithi167 conmon[49112]: debug 2022-01-31T22:30:10.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.688491+0000) 2022-01-31T22:30:10.908 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:10 smithi167 conmon[54076]: debug 2022-01-31T22:30:10.856+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.857157+0000) 2022-01-31T22:30:10.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:10 smithi167 conmon[60316]: debug 2022-01-31T22:30:10.613+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.614743+0000) 2022-01-31T22:30:11.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:10 smithi167 conmon[49112]: debug 2022-01-31T22:30:10.961+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.962564+0000) 2022-01-31T22:30:11.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:10 smithi167 conmon[54076]: debug 2022-01-31T22:30:10.959+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.961128+0000) 2022-01-31T22:30:11.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:10 smithi167 conmon[60316]: debug 2022-01-31T22:30:10.960+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.962080+0000) 2022-01-31T22:30:11.217 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[35325]: debug 2022-01-31T22:30:10.982+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214611 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:11.218 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[41853]: debug 2022-01-31T22:30:10.960+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.961279+0000) 2022-01-31T22:30:11.219 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[46715]: debug 2022-01-31T22:30:10.960+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.961079+0000) 2022-01-31T22:30:11.219 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:10 smithi171 conmon[51620]: debug 2022-01-31T22:30:10.961+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:10.962264+0000) 2022-01-31T22:30:11.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:11 smithi171 conmon[46715]: debug 2022-01-31T22:30:11.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:11.355530+0000) 2022-01-31T22:30:11.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:11 smithi171 conmon[51620]: debug 2022-01-31T22:30:11.217+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:11.218065+0000) 2022-01-31T22:30:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:11 smithi171 conmon[41853]: debug 2022-01-31T22:30:11.560+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:11.561551+0000) 2022-01-31T22:30:11.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:11 smithi167 conmon[49112]: debug 2022-01-31T22:30:11.687+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:11.688678+0000) 2022-01-31T22:30:11.908 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:11 smithi167 conmon[54076]: debug 2022-01-31T22:30:11.856+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:11.857347+0000) 2022-01-31T22:30:11.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:11 smithi167 conmon[60316]: debug 2022-01-31T22:30:11.613+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:11.614868+0000) 2022-01-31T22:30:12.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:12 smithi171 conmon[46715]: debug 2022-01-31T22:30:12.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:12.355644+0000) 2022-01-31T22:30:12.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:12 smithi171 conmon[51620]: debug 2022-01-31T22:30:12.217+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:12.218222+0000) 2022-01-31T22:30:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:12 smithi171 conmon[41853]: debug 2022-01-31T22:30:12.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:12.561734+0000) 2022-01-31T22:30:12.907 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:12 smithi167 conmon[49112]: debug 2022-01-31T22:30:12.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:12.688834+0000) 2022-01-31T22:30:12.908 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:12 smithi167 conmon[54076]: debug 2022-01-31T22:30:12.856+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:12.857538+0000) 2022-01-31T22:30:12.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:12 smithi167 conmon[60316]: debug 2022-01-31T22:30:12.613+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:12.615028+0000) 2022-01-31T22:30:13.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:13 smithi171 conmon[46715]: debug 2022-01-31T22:30:13.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:13.355853+0000) 2022-01-31T22:30:13.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:13 smithi171 conmon[51620]: debug 2022-01-31T22:30:13.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:13.218405+0000) 2022-01-31T22:30:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:13 smithi171 conmon[41853]: debug 2022-01-31T22:30:13.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:13.561901+0000) 2022-01-31T22:30:13.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:13 smithi167 conmon[49112]: debug 2022-01-31T22:30:13.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:13.689044+0000) 2022-01-31T22:30:13.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:13 smithi167 conmon[60316]: debug 2022-01-31T22:30:13.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:13.615173+0000) 2022-01-31T22:30:13.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:13 smithi167 conmon[54076]: debug 2022-01-31T22:30:13.857+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:13.857696+0000) 2022-01-31T22:30:14.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:14 smithi171 conmon[46715]: debug 2022-01-31T22:30:14.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:14.356057+0000) 2022-01-31T22:30:14.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:14 smithi171 conmon[51620]: debug 2022-01-31T22:30:14.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:14.218614+0000) 2022-01-31T22:30:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:14 smithi171 conmon[41853]: debug 2022-01-31T22:30:14.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:14.562025+0000) 2022-01-31T22:30:14.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:14 smithi167 conmon[49112]: debug 2022-01-31T22:30:14.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:14.689228+0000) 2022-01-31T22:30:14.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:14 smithi167 conmon[60316]: debug 2022-01-31T22:30:14.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:14.615365+0000) 2022-01-31T22:30:14.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:14 smithi167 conmon[54076]: debug 2022-01-31T22:30:14.857+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:14.857885+0000) 2022-01-31T22:30:15.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:15 smithi171 conmon[46715]: debug 2022-01-31T22:30:15.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.356248+0000) 2022-01-31T22:30:15.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:15 smithi171 conmon[51620]: debug 2022-01-31T22:30:15.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.218789+0000) 2022-01-31T22:30:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:15 smithi171 conmon[41853]: debug 2022-01-31T22:30:15.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.562164+0000) 2022-01-31T22:30:15.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:15 smithi167 conmon[49112]: debug 2022-01-31T22:30:15.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.689412+0000) 2022-01-31T22:30:15.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:15 smithi167 conmon[60316]: debug 2022-01-31T22:30:15.614+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.615518+0000) 2022-01-31T22:30:15.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:15 smithi167 conmon[54076]: debug 2022-01-31T22:30:15.857+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.858089+0000) 2022-01-31T22:30:16.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:15 smithi167 conmon[49112]: debug 2022-01-31T22:30:15.985+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.986510+0000) 2022-01-31T22:30:16.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:15 smithi167 conmon[54076]: debug 2022-01-31T22:30:15.984+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.986155+0000) 2022-01-31T22:30:16.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:15 smithi167 conmon[60316]: debug 2022-01-31T22:30:15.984+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.986331+0000) 2022-01-31T22:30:16.327 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:16 smithi171 conmon[35325]: debug 2022-01-31T22:30:16.006+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214722 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:16.328 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:15 smithi171 conmon[41853]: debug 2022-01-31T22:30:15.984+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.984775+0000) 2022-01-31T22:30:16.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:15 smithi171 conmon[46715]: debug 2022-01-31T22:30:15.984+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.984986+0000) 2022-01-31T22:30:16.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:15 smithi171 conmon[51620]: debug 2022-01-31T22:30:15.985+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:15.986560+0000) 2022-01-31T22:30:16.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:16 smithi171 conmon[51620]: debug 2022-01-31T22:30:16.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:16.218958+0000) 2022-01-31T22:30:16.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:16 smithi171 conmon[41853]: debug 2022-01-31T22:30:16.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:16.562355+0000) 2022-01-31T22:30:16.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:16 smithi171 conmon[46715]: debug 2022-01-31T22:30:16.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:16.356405+0000) 2022-01-31T22:30:16.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:16 smithi167 conmon[49112]: debug 2022-01-31T22:30:16.688+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:16.689582+0000) 2022-01-31T22:30:16.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:16 smithi167 conmon[60316]: debug 2022-01-31T22:30:16.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:16.615682+0000) 2022-01-31T22:30:16.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:16 smithi167 conmon[54076]: debug 2022-01-31T22:30:16.857+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:16.858302+0000) 2022-01-31T22:30:17.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:17 smithi171 conmon[46715]: debug 2022-01-31T22:30:17.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:17.356552+0000) 2022-01-31T22:30:17.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:17 smithi171 conmon[51620]: debug 2022-01-31T22:30:17.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:17.219122+0000) 2022-01-31T22:30:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:17 smithi171 conmon[41853]: debug 2022-01-31T22:30:17.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:17.562493+0000) 2022-01-31T22:30:17.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:17 smithi167 conmon[49112]: debug 2022-01-31T22:30:17.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:17.689711+0000) 2022-01-31T22:30:17.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:17 smithi167 conmon[54076]: debug 2022-01-31T22:30:17.857+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:17.858468+0000) 2022-01-31T22:30:17.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:17 smithi167 conmon[60316]: debug 2022-01-31T22:30:17.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:17.615816+0000) 2022-01-31T22:30:18.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:18 smithi171 conmon[46715]: debug 2022-01-31T22:30:18.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:18.356688+0000) 2022-01-31T22:30:18.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:18 smithi171 conmon[51620]: debug 2022-01-31T22:30:18.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:18.219347+0000) 2022-01-31T22:30:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:18 smithi171 conmon[41853]: debug 2022-01-31T22:30:18.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:18.562675+0000) 2022-01-31T22:30:18.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:18 smithi167 conmon[49112]: debug 2022-01-31T22:30:18.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:18.689854+0000) 2022-01-31T22:30:18.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:18 smithi167 conmon[54076]: debug 2022-01-31T22:30:18.858+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:18.858624+0000) 2022-01-31T22:30:18.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:18 smithi167 conmon[60316]: debug 2022-01-31T22:30:18.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:18.615981+0000) 2022-01-31T22:30:19.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:19 smithi171 conmon[46715]: debug 2022-01-31T22:30:19.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:19.356845+0000) 2022-01-31T22:30:19.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:19 smithi171 conmon[51620]: debug 2022-01-31T22:30:19.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:19.219502+0000) 2022-01-31T22:30:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:19 smithi171 conmon[41853]: debug 2022-01-31T22:30:19.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:19.562816+0000) 2022-01-31T22:30:19.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:19 smithi167 conmon[49112]: debug 2022-01-31T22:30:19.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:19.690044+0000) 2022-01-31T22:30:19.909 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:19 smithi167 conmon[54076]: debug 2022-01-31T22:30:19.858+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:19.858800+0000) 2022-01-31T22:30:19.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:19 smithi167 conmon[60316]: debug 2022-01-31T22:30:19.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:19.616116+0000) 2022-01-31T22:30:20.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:20 smithi171 conmon[46715]: debug 2022-01-31T22:30:20.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:20.357031+0000) 2022-01-31T22:30:20.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:20 smithi171 conmon[51620]: debug 2022-01-31T22:30:20.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:20.219685+0000) 2022-01-31T22:30:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:20 smithi171 conmon[41853]: debug 2022-01-31T22:30:20.561+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:20.562966+0000) 2022-01-31T22:30:20.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:20 smithi167 conmon[49112]: debug 2022-01-31T22:30:20.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:20.690270+0000) 2022-01-31T22:30:20.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:20 smithi167 conmon[54076]: debug 2022-01-31T22:30:20.858+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:20.859002+0000) 2022-01-31T22:30:20.911 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:20 smithi167 conmon[60316]: debug 2022-01-31T22:30:20.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:20.616280+0000) 2022-01-31T22:30:21.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:21 smithi167 conmon[49112]: debug 2022-01-31T22:30:21.010+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.010688+0000) 2022-01-31T22:30:21.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:21 smithi167 conmon[54076]: debug 2022-01-31T22:30:21.009+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.010019+0000) 2022-01-31T22:30:21.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:21 smithi167 conmon[60316]: debug 2022-01-31T22:30:21.010+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.011075+0000) 2022-01-31T22:30:21.328 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[35325]: debug 2022-01-31T22:30:21.031+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214833 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:21.328 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[41853]: debug 2022-01-31T22:30:21.007+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.009281+0000) 2022-01-31T22:30:21.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[46715]: debug 2022-01-31T22:30:21.007+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.009344+0000) 2022-01-31T22:30:21.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[51620]: debug 2022-01-31T22:30:21.009+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.011381+0000) 2022-01-31T22:30:21.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[51620]: debug 2022-01-31T22:30:21.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.219871+0000) 2022-01-31T22:30:21.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[41853]: debug 2022-01-31T22:30:21.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.563133+0000) 2022-01-31T22:30:21.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:21 smithi171 conmon[46715]: debug 2022-01-31T22:30:21.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.357189+0000) 2022-01-31T22:30:21.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:21 smithi167 conmon[49112]: debug 2022-01-31T22:30:21.689+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.690473+0000) 2022-01-31T22:30:21.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:21 smithi167 conmon[54076]: debug 2022-01-31T22:30:21.858+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.859178+0000) 2022-01-31T22:30:21.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:21 smithi167 conmon[60316]: debug 2022-01-31T22:30:21.615+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:21.616457+0000) 2022-01-31T22:30:22.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:22 smithi171 conmon[46715]: debug 2022-01-31T22:30:22.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:22.357345+0000) 2022-01-31T22:30:22.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:22 smithi171 conmon[51620]: debug 2022-01-31T22:30:22.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:22.220031+0000) 2022-01-31T22:30:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:22 smithi171 conmon[41853]: debug 2022-01-31T22:30:22.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:22.563301+0000) 2022-01-31T22:30:22.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:22 smithi167 conmon[49112]: debug 2022-01-31T22:30:22.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:22.690629+0000) 2022-01-31T22:30:22.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:22 smithi167 conmon[54076]: debug 2022-01-31T22:30:22.858+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:22.859321+0000) 2022-01-31T22:30:22.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:22 smithi167 conmon[60316]: debug 2022-01-31T22:30:22.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:22.616597+0000) 2022-01-31T22:30:23.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:23 smithi171 conmon[46715]: debug 2022-01-31T22:30:23.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:23.357534+0000) 2022-01-31T22:30:23.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:23 smithi171 conmon[51620]: debug 2022-01-31T22:30:23.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:23.220223+0000) 2022-01-31T22:30:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:23 smithi171 conmon[41853]: debug 2022-01-31T22:30:23.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:23.563503+0000) 2022-01-31T22:30:23.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:23 smithi167 conmon[49112]: debug 2022-01-31T22:30:23.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:23.690810+0000) 2022-01-31T22:30:23.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:23 smithi167 conmon[54076]: debug 2022-01-31T22:30:23.858+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:23.859449+0000) 2022-01-31T22:30:23.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:23 smithi167 conmon[60316]: debug 2022-01-31T22:30:23.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:23.616775+0000) 2022-01-31T22:30:24.220 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:23 smithi171 conmon[35325]: debug 2022-01-31T22:30:23.863+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:30:24.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:24 smithi171 conmon[46715]: debug 2022-01-31T22:30:24.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:24.357731+0000) 2022-01-31T22:30:24.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:24 smithi171 conmon[51620]: debug 2022-01-31T22:30:24.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:24.220409+0000) 2022-01-31T22:30:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:24 smithi171 conmon[41853]: debug 2022-01-31T22:30:24.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:24.563658+0000) 2022-01-31T22:30:24.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:24 smithi167 conmon[49112]: debug 2022-01-31T22:30:24.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:24.691006+0000) 2022-01-31T22:30:24.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:24 smithi167 conmon[60316]: debug 2022-01-31T22:30:24.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:24.616972+0000) 2022-01-31T22:30:25.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:24 smithi167 conmon[54076]: debug 2022-01-31T22:30:24.859+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:24.859633+0000) 2022-01-31T22:30:25.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:25 smithi171 conmon[46715]: debug 2022-01-31T22:30:25.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:25.357937+0000) 2022-01-31T22:30:25.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:25 smithi171 conmon[51620]: debug 2022-01-31T22:30:25.218+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:25.220567+0000) 2022-01-31T22:30:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:25 smithi171 conmon[41853]: debug 2022-01-31T22:30:25.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:25.563791+0000) 2022-01-31T22:30:25.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:25 smithi167 conmon[49112]: debug 2022-01-31T22:30:25.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:25.691200+0000) 2022-01-31T22:30:25.859 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:25 smithi167 conmon[60316]: debug 2022-01-31T22:30:25.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:25.617205+0000) 2022-01-31T22:30:26.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:26 smithi167 conmon[49112]: debug 2022-01-31T22:30:26.035+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.035628+0000) 2022-01-31T22:30:26.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:26 smithi167 conmon[60316]: debug 2022-01-31T22:30:26.035+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.036235+0000) 2022-01-31T22:30:26.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:25 smithi167 conmon[54076]: debug 2022-01-31T22:30:25.859+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:25.859828+0000) 2022-01-31T22:30:26.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:26 smithi167 conmon[54076]: debug 2022-01-31T22:30:26.034+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.034908+0000) 2022-01-31T22:30:26.328 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[35325]: debug 2022-01-31T22:30:26.055+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 214944 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:26.329 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[41853]: debug 2022-01-31T22:30:26.033+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.035097+0000) 2022-01-31T22:30:26.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[46715]: debug 2022-01-31T22:30:26.033+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.035678+0000) 2022-01-31T22:30:26.330 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[51620]: debug 2022-01-31T22:30:26.034+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.036589+0000) 2022-01-31T22:30:26.331 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[51620]: debug 2022-01-31T22:30:26.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.220729+0000) 2022-01-31T22:30:26.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[46715]: debug 2022-01-31T22:30:26.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.358159+0000) 2022-01-31T22:30:26.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:26 smithi171 conmon[41853]: debug 2022-01-31T22:30:26.562+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.563909+0000) 2022-01-31T22:30:26.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:26 smithi167 conmon[49112]: debug 2022-01-31T22:30:26.690+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.691293+0000) 2022-01-31T22:30:26.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:26 smithi167 conmon[60316]: debug 2022-01-31T22:30:26.616+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.617395+0000) 2022-01-31T22:30:27.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:26 smithi167 conmon[54076]: debug 2022-01-31T22:30:26.859+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:26.860029+0000) 2022-01-31T22:30:27.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:27 smithi171 conmon[46715]: debug 2022-01-31T22:30:27.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:27.358317+0000) 2022-01-31T22:30:27.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:27 smithi171 conmon[51620]: debug 2022-01-31T22:30:27.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:27.220858+0000) 2022-01-31T22:30:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:27 smithi171 conmon[41853]: debug 2022-01-31T22:30:27.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:27.564061+0000) 2022-01-31T22:30:27.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:27 smithi167 conmon[49112]: debug 2022-01-31T22:30:27.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:27.691442+0000) 2022-01-31T22:30:27.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:27 smithi167 conmon[60316]: debug 2022-01-31T22:30:27.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:27.617570+0000) 2022-01-31T22:30:28.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:27 smithi167 conmon[54076]: debug 2022-01-31T22:30:27.859+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:27.860191+0000) 2022-01-31T22:30:28.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:28 smithi171 conmon[46715]: debug 2022-01-31T22:30:28.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:28.358845+0000) 2022-01-31T22:30:28.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:28 smithi171 conmon[51620]: debug 2022-01-31T22:30:28.219+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:28.221011+0000) 2022-01-31T22:30:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:28 smithi171 conmon[41853]: debug 2022-01-31T22:30:28.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:28.564241+0000) 2022-01-31T22:30:28.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:28 smithi167 conmon[49112]: debug 2022-01-31T22:30:28.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:28.691624+0000) 2022-01-31T22:30:28.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:28 smithi167 conmon[60316]: debug 2022-01-31T22:30:28.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:28.617750+0000) 2022-01-31T22:30:29.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:28 smithi167 conmon[54076]: debug 2022-01-31T22:30:28.859+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:28.860362+0000) 2022-01-31T22:30:29.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:29 smithi171 conmon[46715]: debug 2022-01-31T22:30:29.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:29.358964+0000) 2022-01-31T22:30:29.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:29 smithi171 conmon[51620]: debug 2022-01-31T22:30:29.220+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:29.221162+0000) 2022-01-31T22:30:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:29 smithi171 conmon[41853]: debug 2022-01-31T22:30:29.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:29.564403+0000) 2022-01-31T22:30:29.860 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:29 smithi167 conmon[49112]: debug 2022-01-31T22:30:29.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:29.691830+0000) 2022-01-31T22:30:29.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:29 smithi167 conmon[60316]: debug 2022-01-31T22:30:29.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:29.617920+0000) 2022-01-31T22:30:30.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:29 smithi167 conmon[54076]: debug 2022-01-31T22:30:29.860+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:29.860532+0000) 2022-01-31T22:30:30.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:30 smithi171 conmon[46715]: debug 2022-01-31T22:30:30.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:30.359128+0000) 2022-01-31T22:30:30.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:30 smithi171 conmon[51620]: debug 2022-01-31T22:30:30.220+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:30.221416+0000) 2022-01-31T22:30:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:30 smithi171 conmon[41853]: debug 2022-01-31T22:30:30.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:30.564534+0000) 2022-01-31T22:30:30.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:30 smithi167 conmon[49112]: debug 2022-01-31T22:30:30.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:30.692022+0000) 2022-01-31T22:30:30.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:30 smithi167 conmon[54076]: debug 2022-01-31T22:30:30.860+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:30.860691+0000) 2022-01-31T22:30:30.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:30 smithi167 conmon[60316]: debug 2022-01-31T22:30:30.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:30.618088+0000) 2022-01-31T22:30:31.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:31 smithi167 conmon[49112]: debug 2022-01-31T22:30:31.059+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.060278+0000) 2022-01-31T22:30:31.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:31 smithi167 conmon[54076]: debug 2022-01-31T22:30:31.059+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.060053+0000) 2022-01-31T22:30:31.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:31 smithi167 conmon[60316]: debug 2022-01-31T22:30:31.060+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.060706+0000) 2022-01-31T22:30:31.330 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[35325]: debug 2022-01-31T22:30:31.080+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215055 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:31.331 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[41853]: debug 2022-01-31T22:30:31.059+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.061003+0000) 2022-01-31T22:30:31.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[46715]: debug 2022-01-31T22:30:31.057+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.059672+0000) 2022-01-31T22:30:31.333 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[51620]: debug 2022-01-31T22:30:31.058+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.060419+0000) 2022-01-31T22:30:31.333 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[51620]: debug 2022-01-31T22:30:31.220+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.221595+0000) 2022-01-31T22:30:31.607 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[41853]: debug 2022-01-31T22:30:31.563+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.564702+0000) 2022-01-31T22:30:31.608 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:31 smithi171 conmon[46715]: debug 2022-01-31T22:30:31.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.359300+0000) 2022-01-31T22:30:31.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:31 smithi167 conmon[49112]: debug 2022-01-31T22:30:31.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.692232+0000) 2022-01-31T22:30:31.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:31 smithi167 conmon[54076]: debug 2022-01-31T22:30:31.860+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.860815+0000) 2022-01-31T22:30:31.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:31 smithi167 conmon[60316]: debug 2022-01-31T22:30:31.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:31.618221+0000) 2022-01-31T22:30:32.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:32 smithi171 conmon[46715]: debug 2022-01-31T22:30:32.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:32.359476+0000) 2022-01-31T22:30:32.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:32 smithi171 conmon[51620]: debug 2022-01-31T22:30:32.221+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:32.221765+0000) 2022-01-31T22:30:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:32 smithi171 conmon[41853]: debug 2022-01-31T22:30:32.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:32.564863+0000) 2022-01-31T22:30:32.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:32 smithi167 conmon[49112]: debug 2022-01-31T22:30:32.691+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:32.692397+0000) 2022-01-31T22:30:32.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:32 smithi167 conmon[54076]: debug 2022-01-31T22:30:32.860+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:32.860921+0000) 2022-01-31T22:30:32.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:32 smithi167 conmon[60316]: debug 2022-01-31T22:30:32.617+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:32.618312+0000) 2022-01-31T22:30:33.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:33 smithi171 conmon[46715]: debug 2022-01-31T22:30:33.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:33.359685+0000) 2022-01-31T22:30:33.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:33 smithi171 conmon[51620]: debug 2022-01-31T22:30:33.221+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:33.221954+0000) 2022-01-31T22:30:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:33 smithi171 conmon[41853]: debug 2022-01-31T22:30:33.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:33.565069+0000) 2022-01-31T22:30:33.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:33 smithi167 conmon[49112]: debug 2022-01-31T22:30:33.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:33.692561+0000) 2022-01-31T22:30:33.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:33 smithi167 conmon[54076]: debug 2022-01-31T22:30:33.860+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:33.861099+0000) 2022-01-31T22:30:33.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:33 smithi167 conmon[60316]: debug 2022-01-31T22:30:33.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:33.618525+0000) 2022-01-31T22:30:34.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:34 smithi171 conmon[46715]: debug 2022-01-31T22:30:34.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:34.359886+0000) 2022-01-31T22:30:34.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:34 smithi171 conmon[51620]: debug 2022-01-31T22:30:34.221+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:34.222142+0000) 2022-01-31T22:30:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:34 smithi171 conmon[41853]: debug 2022-01-31T22:30:34.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:34.565249+0000) 2022-01-31T22:30:34.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:34 smithi167 conmon[49112]: debug 2022-01-31T22:30:34.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:34.692718+0000) 2022-01-31T22:30:34.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:34 smithi167 conmon[54076]: debug 2022-01-31T22:30:34.860+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:34.861256+0000) 2022-01-31T22:30:34.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:34 smithi167 conmon[60316]: debug 2022-01-31T22:30:34.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:34.618733+0000) 2022-01-31T22:30:35.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:35 smithi171 conmon[46715]: debug 2022-01-31T22:30:35.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:35.360055+0000) 2022-01-31T22:30:35.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:35 smithi171 conmon[51620]: debug 2022-01-31T22:30:35.221+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:35.222323+0000) 2022-01-31T22:30:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:35 smithi171 conmon[41853]: debug 2022-01-31T22:30:35.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:35.565433+0000) 2022-01-31T22:30:35.911 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:35 smithi167 conmon[54076]: debug 2022-01-31T22:30:35.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:35.861381+0000) 2022-01-31T22:30:35.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:35 smithi167 conmon[60316]: debug 2022-01-31T22:30:35.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:35.618937+0000) 2022-01-31T22:30:35.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:35 smithi167 conmon[49112]: debug 2022-01-31T22:30:35.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:35.692878+0000) 2022-01-31T22:30:36.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:36 smithi167 conmon[49112]: debug 2022-01-31T22:30:36.084+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.084864+0000) 2022-01-31T22:30:36.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:36 smithi167 conmon[54076]: debug 2022-01-31T22:30:36.083+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.084457+0000) 2022-01-31T22:30:36.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:36 smithi167 conmon[60316]: debug 2022-01-31T22:30:36.085+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.085539+0000) 2022-01-31T22:30:36.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[35325]: debug 2022-01-31T22:30:36.106+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215166 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:36.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[41853]: debug 2022-01-31T22:30:36.084+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.085578+0000) 2022-01-31T22:30:36.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[46715]: debug 2022-01-31T22:30:36.084+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.085201+0000) 2022-01-31T22:30:36.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[51620]: debug 2022-01-31T22:30:36.084+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.085473+0000) 2022-01-31T22:30:36.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[51620]: debug 2022-01-31T22:30:36.221+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.222507+0000) 2022-01-31T22:30:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[41853]: debug 2022-01-31T22:30:36.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.565560+0000) 2022-01-31T22:30:36.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:36 smithi171 conmon[46715]: debug 2022-01-31T22:30:36.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.360162+0000) 2022-01-31T22:30:36.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:36 smithi167 conmon[49112]: debug 2022-01-31T22:30:36.692+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.693057+0000) 2022-01-31T22:30:36.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:36 smithi167 conmon[54076]: debug 2022-01-31T22:30:36.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.861536+0000) 2022-01-31T22:30:36.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:36 smithi167 conmon[60316]: debug 2022-01-31T22:30:36.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:36.619156+0000) 2022-01-31T22:30:37.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:37 smithi171 conmon[46715]: debug 2022-01-31T22:30:37.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:37.360270+0000) 2022-01-31T22:30:37.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:37 smithi171 conmon[51620]: debug 2022-01-31T22:30:37.222+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:37.222670+0000) 2022-01-31T22:30:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:37 smithi171 conmon[41853]: debug 2022-01-31T22:30:37.564+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:37.565687+0000) 2022-01-31T22:30:37.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:37 smithi167 conmon[49112]: debug 2022-01-31T22:30:37.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:37.693215+0000) 2022-01-31T22:30:37.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:37 smithi167 conmon[54076]: debug 2022-01-31T22:30:37.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:37.861696+0000) 2022-01-31T22:30:37.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:37 smithi167 conmon[60316]: debug 2022-01-31T22:30:37.618+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:37.619319+0000) 2022-01-31T22:30:38.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:38 smithi171 conmon[46715]: debug 2022-01-31T22:30:38.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:38.360466+0000) 2022-01-31T22:30:38.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:38 smithi171 conmon[51620]: debug 2022-01-31T22:30:38.222+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:38.222866+0000) 2022-01-31T22:30:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:38 smithi171 conmon[41853]: debug 2022-01-31T22:30:38.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:38.565844+0000) 2022-01-31T22:30:38.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:38 smithi167 conmon[49112]: debug 2022-01-31T22:30:38.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:38.693344+0000) 2022-01-31T22:30:38.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:38 smithi167 conmon[54076]: debug 2022-01-31T22:30:38.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:38.861826+0000) 2022-01-31T22:30:38.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:38 smithi167 conmon[60316]: debug 2022-01-31T22:30:38.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:38.619527+0000) 2022-01-31T22:30:39.222 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:38 smithi171 conmon[35325]: debug 2022-01-31T22:30:38.864+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:30:39.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:39 smithi171 conmon[51620]: debug 2022-01-31T22:30:39.222+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:39.223064+0000) 2022-01-31T22:30:39.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:39 smithi171 conmon[46715]: debug 2022-01-31T22:30:39.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:39.360688+0000) 2022-01-31T22:30:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:39 smithi171 conmon[41853]: debug 2022-01-31T22:30:39.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:39.566018+0000) 2022-01-31T22:30:39.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:39 smithi167 conmon[49112]: debug 2022-01-31T22:30:39.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:39.693577+0000) 2022-01-31T22:30:39.913 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:39 smithi167 conmon[54076]: debug 2022-01-31T22:30:39.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:39.862013+0000) 2022-01-31T22:30:39.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:39 smithi167 conmon[60316]: debug 2022-01-31T22:30:39.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:39.619687+0000) 2022-01-31T22:30:40.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:40 smithi171 conmon[46715]: debug 2022-01-31T22:30:40.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:40.360876+0000) 2022-01-31T22:30:40.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:40 smithi171 conmon[51620]: debug 2022-01-31T22:30:40.222+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:40.223251+0000) 2022-01-31T22:30:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:40 smithi171 conmon[41853]: debug 2022-01-31T22:30:40.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:40.566168+0000) 2022-01-31T22:30:40.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:40 smithi167 conmon[49112]: debug 2022-01-31T22:30:40.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:40.693763+0000) 2022-01-31T22:30:40.913 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:40 smithi167 conmon[54076]: debug 2022-01-31T22:30:40.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:40.862189+0000) 2022-01-31T22:30:40.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:40 smithi167 conmon[60316]: debug 2022-01-31T22:30:40.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:40.619873+0000) 2022-01-31T22:30:41.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:41 smithi167 conmon[49112]: debug 2022-01-31T22:30:41.110+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.110875+0000) 2022-01-31T22:30:41.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:41 smithi167 conmon[54076]: debug 2022-01-31T22:30:41.109+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.110060+0000) 2022-01-31T22:30:41.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:41 smithi167 conmon[60316]: debug 2022-01-31T22:30:41.110+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.111290+0000) 2022-01-31T22:30:41.360 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[35325]: debug 2022-01-31T22:30:41.131+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215277 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:41.361 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[41853]: debug 2022-01-31T22:30:41.108+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.109733+0000) 2022-01-31T22:30:41.362 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[46715]: debug 2022-01-31T22:30:41.109+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.110028+0000) 2022-01-31T22:30:41.363 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[51620]: debug 2022-01-31T22:30:41.108+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.109235+0000) 2022-01-31T22:30:41.364 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[51620]: debug 2022-01-31T22:30:41.222+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.223464+0000) 2022-01-31T22:30:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[41853]: debug 2022-01-31T22:30:41.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.566342+0000) 2022-01-31T22:30:41.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:41 smithi171 conmon[46715]: debug 2022-01-31T22:30:41.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.361095+0000) 2022-01-31T22:30:41.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:41 smithi167 conmon[49112]: debug 2022-01-31T22:30:41.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.693930+0000) 2022-01-31T22:30:41.913 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:41 smithi167 conmon[54076]: debug 2022-01-31T22:30:41.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.862322+0000) 2022-01-31T22:30:41.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:41 smithi167 conmon[60316]: debug 2022-01-31T22:30:41.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:41.620033+0000) 2022-01-31T22:30:42.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:42 smithi171 conmon[46715]: debug 2022-01-31T22:30:42.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:42.361240+0000) 2022-01-31T22:30:42.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:42 smithi171 conmon[51620]: debug 2022-01-31T22:30:42.223+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:42.223619+0000) 2022-01-31T22:30:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:42 smithi171 conmon[41853]: debug 2022-01-31T22:30:42.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:42.566497+0000) 2022-01-31T22:30:42.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:42 smithi167 conmon[49112]: debug 2022-01-31T22:30:42.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:42.694024+0000) 2022-01-31T22:30:42.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:42 smithi167 conmon[60316]: debug 2022-01-31T22:30:42.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:42.620183+0000) 2022-01-31T22:30:42.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:42 smithi167 conmon[54076]: debug 2022-01-31T22:30:42.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:42.862457+0000) 2022-01-31T22:30:43.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:43 smithi171 conmon[46715]: debug 2022-01-31T22:30:43.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:43.361476+0000) 2022-01-31T22:30:43.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:43 smithi171 conmon[51620]: debug 2022-01-31T22:30:43.223+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:43.223821+0000) 2022-01-31T22:30:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:43 smithi171 conmon[41853]: debug 2022-01-31T22:30:43.565+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:43.566632+0000) 2022-01-31T22:30:43.913 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:43 smithi167 conmon[54076]: debug 2022-01-31T22:30:43.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:43.862590+0000) 2022-01-31T22:30:43.913 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:43 smithi167 conmon[60316]: debug 2022-01-31T22:30:43.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:43.620390+0000) 2022-01-31T22:30:43.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:43 smithi167 conmon[49112]: debug 2022-01-31T22:30:43.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:43.694179+0000) 2022-01-31T22:30:44.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:44 smithi171 conmon[46715]: debug 2022-01-31T22:30:44.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:44.361666+0000) 2022-01-31T22:30:44.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:44 smithi171 conmon[51620]: debug 2022-01-31T22:30:44.223+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:44.224043+0000) 2022-01-31T22:30:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:44 smithi171 conmon[41853]: debug 2022-01-31T22:30:44.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:44.566785+0000) 2022-01-31T22:30:44.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:44 smithi167 conmon[49112]: debug 2022-01-31T22:30:44.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:44.694325+0000) 2022-01-31T22:30:44.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:44 smithi167 conmon[54076]: debug 2022-01-31T22:30:44.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:44.862747+0000) 2022-01-31T22:30:44.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:44 smithi167 conmon[60316]: debug 2022-01-31T22:30:44.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:44.620619+0000) 2022-01-31T22:30:45.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:45 smithi171 conmon[46715]: debug 2022-01-31T22:30:45.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:45.361854+0000) 2022-01-31T22:30:45.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:45 smithi171 conmon[51620]: debug 2022-01-31T22:30:45.223+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:45.224276+0000) 2022-01-31T22:30:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:45 smithi171 conmon[41853]: debug 2022-01-31T22:30:45.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:45.566884+0000) 2022-01-31T22:30:45.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:45 smithi167 conmon[49112]: debug 2022-01-31T22:30:45.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:45.694512+0000) 2022-01-31T22:30:45.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:45 smithi167 conmon[54076]: debug 2022-01-31T22:30:45.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:45.862918+0000) 2022-01-31T22:30:45.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:45 smithi167 conmon[60316]: debug 2022-01-31T22:30:45.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:45.620831+0000) 2022-01-31T22:30:46.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:46 smithi167 conmon[49112]: debug 2022-01-31T22:30:46.133+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.135317+0000) 2022-01-31T22:30:46.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:46 smithi167 conmon[54076]: debug 2022-01-31T22:30:46.134+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.135640+0000) 2022-01-31T22:30:46.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:46 smithi167 conmon[60316]: debug 2022-01-31T22:30:46.134+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.135914+0000) 2022-01-31T22:30:46.526 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[35325]: debug 2022-01-31T22:30:46.157+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215390 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:46.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[41853]: debug 2022-01-31T22:30:46.134+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.135276+0000) 2022-01-31T22:30:46.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[46715]: debug 2022-01-31T22:30:46.135+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.135823+0000) 2022-01-31T22:30:46.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[46715]: debug 2022-01-31T22:30:46.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.362039+0000) 2022-01-31T22:30:46.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[51620]: debug 2022-01-31T22:30:46.133+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.134586+0000) 2022-01-31T22:30:46.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[51620]: debug 2022-01-31T22:30:46.223+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.224487+0000) 2022-01-31T22:30:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:46 smithi171 conmon[41853]: debug 2022-01-31T22:30:46.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.566994+0000) 2022-01-31T22:30:46.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:46 smithi167 conmon[49112]: debug 2022-01-31T22:30:46.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.694674+0000) 2022-01-31T22:30:46.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:46 smithi167 conmon[54076]: debug 2022-01-31T22:30:46.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.863058+0000) 2022-01-31T22:30:46.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:46 smithi167 conmon[60316]: debug 2022-01-31T22:30:46.619+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:46.621050+0000) 2022-01-31T22:30:47.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:47 smithi171 conmon[46715]: debug 2022-01-31T22:30:47.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:47.362227+0000) 2022-01-31T22:30:47.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:47 smithi171 conmon[51620]: debug 2022-01-31T22:30:47.223+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:47.224587+0000) 2022-01-31T22:30:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:47 smithi171 conmon[41853]: debug 2022-01-31T22:30:47.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:47.567096+0000) 2022-01-31T22:30:47.914 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:47 smithi167 conmon[54076]: debug 2022-01-31T22:30:47.861+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:47.863243+0000) 2022-01-31T22:30:47.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:47 smithi167 conmon[49112]: debug 2022-01-31T22:30:47.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:47.694848+0000) 2022-01-31T22:30:47.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:47 smithi167 conmon[60316]: debug 2022-01-31T22:30:47.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:47.621205+0000) 2022-01-31T22:30:48.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:48 smithi171 conmon[46715]: debug 2022-01-31T22:30:48.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:48.362427+0000) 2022-01-31T22:30:48.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:48 smithi171 conmon[51620]: debug 2022-01-31T22:30:48.224+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:48.224745+0000) 2022-01-31T22:30:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:48 smithi171 conmon[41853]: debug 2022-01-31T22:30:48.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:48.567278+0000) 2022-01-31T22:30:48.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:48 smithi167 conmon[49112]: debug 2022-01-31T22:30:48.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:48.695036+0000) 2022-01-31T22:30:48.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:48 smithi167 conmon[54076]: debug 2022-01-31T22:30:48.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:48.863451+0000) 2022-01-31T22:30:48.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:48 smithi167 conmon[60316]: debug 2022-01-31T22:30:48.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:48.621405+0000) 2022-01-31T22:30:49.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:49 smithi171 conmon[46715]: debug 2022-01-31T22:30:49.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:49.362628+0000) 2022-01-31T22:30:49.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:49 smithi171 conmon[51620]: debug 2022-01-31T22:30:49.224+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:49.224906+0000) 2022-01-31T22:30:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:49 smithi171 conmon[41853]: debug 2022-01-31T22:30:49.566+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:49.567426+0000) 2022-01-31T22:30:49.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:49 smithi167 conmon[49112]: debug 2022-01-31T22:30:49.693+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:49.695235+0000) 2022-01-31T22:30:49.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:49 smithi167 conmon[54076]: debug 2022-01-31T22:30:49.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:49.863684+0000) 2022-01-31T22:30:49.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:49 smithi167 conmon[60316]: debug 2022-01-31T22:30:49.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:49.621622+0000) 2022-01-31T22:30:50.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:50 smithi171 conmon[46715]: debug 2022-01-31T22:30:50.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:50.362816+0000) 2022-01-31T22:30:50.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:50 smithi171 conmon[51620]: debug 2022-01-31T22:30:50.224+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:50.225124+0000) 2022-01-31T22:30:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:50 smithi171 conmon[41853]: debug 2022-01-31T22:30:50.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:50.567614+0000) 2022-01-31T22:30:50.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:50 smithi167 conmon[49112]: debug 2022-01-31T22:30:50.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:50.695422+0000) 2022-01-31T22:30:50.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:50 smithi167 conmon[54076]: debug 2022-01-31T22:30:50.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:50.863907+0000) 2022-01-31T22:30:50.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:50 smithi167 conmon[60316]: debug 2022-01-31T22:30:50.620+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:50.621771+0000) 2022-01-31T22:30:51.164 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:51 smithi167 conmon[49112]: debug 2022-01-31T22:30:51.158+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.160129+0000) 2022-01-31T22:30:51.165 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:51 smithi167 conmon[54076]: debug 2022-01-31T22:30:51.158+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.159869+0000) 2022-01-31T22:30:51.166 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:51 smithi167 conmon[60316]: debug 2022-01-31T22:30:51.159+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.160937+0000) 2022-01-31T22:30:51.527 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[35325]: debug 2022-01-31T22:30:51.181+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215500 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:51.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[41853]: debug 2022-01-31T22:30:51.159+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.160190+0000) 2022-01-31T22:30:51.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[51620]: debug 2022-01-31T22:30:51.158+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.159390+0000) 2022-01-31T22:30:51.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[51620]: debug 2022-01-31T22:30:51.224+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.225276+0000) 2022-01-31T22:30:51.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[46715]: debug 2022-01-31T22:30:51.159+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.160083+0000) 2022-01-31T22:30:51.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[46715]: debug 2022-01-31T22:30:51.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.363003+0000) 2022-01-31T22:30:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:51 smithi171 conmon[41853]: debug 2022-01-31T22:30:51.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.567806+0000) 2022-01-31T22:30:51.863 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:51 smithi167 conmon[49112]: debug 2022-01-31T22:30:51.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.695604+0000) 2022-01-31T22:30:51.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:51 smithi167 conmon[60316]: debug 2022-01-31T22:30:51.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.621949+0000) 2022-01-31T22:30:52.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:51 smithi167 conmon[54076]: debug 2022-01-31T22:30:51.863+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:51.864094+0000) 2022-01-31T22:30:52.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:52 smithi171 conmon[46715]: debug 2022-01-31T22:30:52.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:52.363142+0000) 2022-01-31T22:30:52.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:52 smithi171 conmon[51620]: debug 2022-01-31T22:30:52.224+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:52.225435+0000) 2022-01-31T22:30:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:52 smithi171 conmon[41853]: debug 2022-01-31T22:30:52.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:52.567965+0000) 2022-01-31T22:30:52.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:52 smithi167 conmon[49112]: debug 2022-01-31T22:30:52.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:52.695756+0000) 2022-01-31T22:30:52.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:52 smithi167 conmon[54076]: debug 2022-01-31T22:30:52.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:52.864268+0000) 2022-01-31T22:30:52.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:52 smithi167 conmon[60316]: debug 2022-01-31T22:30:52.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:52.622134+0000) 2022-01-31T22:30:53.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:53 smithi171 conmon[46715]: debug 2022-01-31T22:30:53.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:53.363348+0000) 2022-01-31T22:30:53.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:53 smithi171 conmon[51620]: debug 2022-01-31T22:30:53.224+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:53.225637+0000) 2022-01-31T22:30:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:53 smithi171 conmon[41853]: debug 2022-01-31T22:30:53.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:53.568099+0000) 2022-01-31T22:30:53.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:53 smithi167 conmon[49112]: debug 2022-01-31T22:30:53.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:53.695894+0000) 2022-01-31T22:30:53.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:53 smithi167 conmon[54076]: debug 2022-01-31T22:30:53.862+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:53.864392+0000) 2022-01-31T22:30:53.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:53 smithi167 conmon[60316]: debug 2022-01-31T22:30:53.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:53.622307+0000) 2022-01-31T22:30:54.225 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:53 smithi171 conmon[35325]: debug 2022-01-31T22:30:53.865+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:30:54.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:54 smithi171 conmon[46715]: debug 2022-01-31T22:30:54.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:54.363533+0000) 2022-01-31T22:30:54.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:54 smithi171 conmon[51620]: debug 2022-01-31T22:30:54.225+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:54.225842+0000) 2022-01-31T22:30:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:54 smithi171 conmon[41853]: debug 2022-01-31T22:30:54.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:54.568283+0000) 2022-01-31T22:30:54.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:54 smithi167 conmon[49112]: debug 2022-01-31T22:30:54.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:54.696102+0000) 2022-01-31T22:30:54.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:54 smithi167 conmon[54076]: debug 2022-01-31T22:30:54.863+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:54.864550+0000) 2022-01-31T22:30:54.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:54 smithi167 conmon[60316]: debug 2022-01-31T22:30:54.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:54.622495+0000) 2022-01-31T22:30:55.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:55 smithi171 conmon[46715]: debug 2022-01-31T22:30:55.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:55.363695+0000) 2022-01-31T22:30:55.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:55 smithi171 conmon[51620]: debug 2022-01-31T22:30:55.225+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:55.226048+0000) 2022-01-31T22:30:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:55 smithi171 conmon[41853]: debug 2022-01-31T22:30:55.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:55.568384+0000) 2022-01-31T22:30:55.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:55 smithi167 conmon[49112]: debug 2022-01-31T22:30:55.694+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:55.696336+0000) 2022-01-31T22:30:55.916 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:55 smithi167 conmon[54076]: debug 2022-01-31T22:30:55.863+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:55.864707+0000) 2022-01-31T22:30:55.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:55 smithi167 conmon[60316]: debug 2022-01-31T22:30:55.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:55.622696+0000) 2022-01-31T22:30:56.454 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:56 smithi167 conmon[49112]: debug 2022-01-31T22:30:56.183+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.184978+0000) 2022-01-31T22:30:56.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:56 smithi167 conmon[54076]: debug 2022-01-31T22:30:56.184+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.185811+0000) 2022-01-31T22:30:56.455 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:56 smithi167 conmon[60316]: debug 2022-01-31T22:30:56.184+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.185949+0000) 2022-01-31T22:30:56.528 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[35325]: debug 2022-01-31T22:30:56.204+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215613 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:30:56.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[41853]: debug 2022-01-31T22:30:56.185+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.186056+0000) 2022-01-31T22:30:56.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[51620]: debug 2022-01-31T22:30:56.184+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.185273+0000) 2022-01-31T22:30:56.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[51620]: debug 2022-01-31T22:30:56.225+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.226238+0000) 2022-01-31T22:30:56.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[46715]: debug 2022-01-31T22:30:56.184+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.184988+0000) 2022-01-31T22:30:56.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[46715]: debug 2022-01-31T22:30:56.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.363886+0000) 2022-01-31T22:30:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:56 smithi171 conmon[41853]: debug 2022-01-31T22:30:56.567+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.568570+0000) 2022-01-31T22:30:56.864 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:56 smithi167 conmon[49112]: debug 2022-01-31T22:30:56.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.696581+0000) 2022-01-31T22:30:56.864 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:56 smithi167 conmon[60316]: debug 2022-01-31T22:30:56.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.622880+0000) 2022-01-31T22:30:57.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:56 smithi167 conmon[54076]: debug 2022-01-31T22:30:56.864+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:56.864898+0000) 2022-01-31T22:30:57.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:57 smithi171 conmon[46715]: debug 2022-01-31T22:30:57.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:57.364070+0000) 2022-01-31T22:30:57.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:57 smithi171 conmon[51620]: debug 2022-01-31T22:30:57.226+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:57.226404+0000) 2022-01-31T22:30:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:57 smithi171 conmon[41853]: debug 2022-01-31T22:30:57.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:57.568739+0000) 2022-01-31T22:30:57.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:57 smithi167 conmon[49112]: debug 2022-01-31T22:30:57.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:57.696677+0000) 2022-01-31T22:30:57.916 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:57 smithi167 conmon[54076]: debug 2022-01-31T22:30:57.864+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:57.865044+0000) 2022-01-31T22:30:57.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:57 smithi167 conmon[60316]: debug 2022-01-31T22:30:57.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:57.623048+0000) 2022-01-31T22:30:58.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:58 smithi171 conmon[46715]: debug 2022-01-31T22:30:58.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:58.364274+0000) 2022-01-31T22:30:58.529 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:58 smithi171 conmon[51620]: debug 2022-01-31T22:30:58.226+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:58.226596+0000) 2022-01-31T22:30:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:58 smithi171 conmon[41853]: debug 2022-01-31T22:30:58.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:58.568883+0000) 2022-01-31T22:30:58.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:58 smithi167 conmon[49112]: debug 2022-01-31T22:30:58.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:58.696781+0000) 2022-01-31T22:30:58.916 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:58 smithi167 conmon[54076]: debug 2022-01-31T22:30:58.864+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:58.865260+0000) 2022-01-31T22:30:58.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:58 smithi167 conmon[60316]: debug 2022-01-31T22:30:58.621+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:58.623264+0000) 2022-01-31T22:30:59.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:30:59 smithi171 conmon[46715]: debug 2022-01-31T22:30:59.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:59.364475+0000) 2022-01-31T22:30:59.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:30:59 smithi171 conmon[51620]: debug 2022-01-31T22:30:59.226+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:59.226811+0000) 2022-01-31T22:30:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:30:59 smithi171 conmon[41853]: debug 2022-01-31T22:30:59.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:59.569035+0000) 2022-01-31T22:30:59.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:30:59 smithi167 conmon[49112]: debug 2022-01-31T22:30:59.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:59.696921+0000) 2022-01-31T22:30:59.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:30:59 smithi167 conmon[54076]: debug 2022-01-31T22:30:59.864+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:59.865516+0000) 2022-01-31T22:30:59.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:30:59 smithi167 conmon[60316]: debug 2022-01-31T22:30:59.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:30:59.623476+0000) 2022-01-31T22:31:00.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:00 smithi171 conmon[46715]: debug 2022-01-31T22:31:00.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:00.364645+0000) 2022-01-31T22:31:00.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:00 smithi171 conmon[51620]: debug 2022-01-31T22:31:00.226+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:00.226988+0000) 2022-01-31T22:31:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:00 smithi171 conmon[41853]: debug 2022-01-31T22:31:00.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:00.569207+0000) 2022-01-31T22:31:00.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:00 smithi167 conmon[49112]: debug 2022-01-31T22:31:00.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:00.697130+0000) 2022-01-31T22:31:00.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:00 smithi167 conmon[54076]: debug 2022-01-31T22:31:00.865+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:00.865737+0000) 2022-01-31T22:31:00.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:00 smithi167 conmon[60316]: debug 2022-01-31T22:31:00.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:00.623681+0000) 2022-01-31T22:31:01.529 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[35325]: debug 2022-01-31T22:31:01.235+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215723 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:01.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[41853]: debug 2022-01-31T22:31:01.207+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.208202+0000) 2022-01-31T22:31:01.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[46715]: debug 2022-01-31T22:31:01.207+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.207784+0000) 2022-01-31T22:31:01.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[46715]: debug 2022-01-31T22:31:01.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.364822+0000) 2022-01-31T22:31:01.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[51620]: debug 2022-01-31T22:31:01.208+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.208860+0000) 2022-01-31T22:31:01.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[51620]: debug 2022-01-31T22:31:01.226+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.227145+0000) 2022-01-31T22:31:01.623 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:01 smithi167 conmon[49112]: debug 2022-01-31T22:31:01.206+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.207897+0000) 2022-01-31T22:31:01.624 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:01 smithi167 conmon[54076]: debug 2022-01-31T22:31:01.206+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.207789+0000) 2022-01-31T22:31:01.625 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:01 smithi167 conmon[60316]: debug 2022-01-31T22:31:01.214+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.215569+0000) 2022-01-31T22:31:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:01 smithi171 conmon[41853]: debug 2022-01-31T22:31:01.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.569325+0000) 2022-01-31T22:31:01.916 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:01 smithi167 conmon[54076]: debug 2022-01-31T22:31:01.865+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.865925+0000) 2022-01-31T22:31:01.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:01 smithi167 conmon[60316]: debug 2022-01-31T22:31:01.622+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.623802+0000) 2022-01-31T22:31:01.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:01 smithi167 conmon[49112]: debug 2022-01-31T22:31:01.695+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:01.697319+0000) 2022-01-31T22:31:02.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:02 smithi171 conmon[46715]: debug 2022-01-31T22:31:02.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:02.364970+0000) 2022-01-31T22:31:02.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:02 smithi171 conmon[51620]: debug 2022-01-31T22:31:02.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:02.227293+0000) 2022-01-31T22:31:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:02 smithi171 conmon[41853]: debug 2022-01-31T22:31:02.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:02.569461+0000) 2022-01-31T22:31:02.916 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:02 smithi167 conmon[54076]: debug 2022-01-31T22:31:02.864+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:02.866059+0000) 2022-01-31T22:31:02.917 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:02 smithi167 conmon[60316]: debug 2022-01-31T22:31:02.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:02.623956+0000) 2022-01-31T22:31:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:02 smithi167 conmon[49112]: debug 2022-01-31T22:31:02.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:02.697488+0000) 2022-01-31T22:31:03.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:03 smithi171 conmon[46715]: debug 2022-01-31T22:31:03.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:03.365118+0000) 2022-01-31T22:31:03.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:03 smithi171 conmon[51620]: debug 2022-01-31T22:31:03.226+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:03.227480+0000) 2022-01-31T22:31:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:03 smithi171 conmon[41853]: debug 2022-01-31T22:31:03.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:03.569683+0000) 2022-01-31T22:31:03.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:03 smithi167 conmon[49112]: debug 2022-01-31T22:31:03.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:03.697704+0000) 2022-01-31T22:31:03.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:03 smithi167 conmon[54076]: debug 2022-01-31T22:31:03.865+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:03.866243+0000) 2022-01-31T22:31:03.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:03 smithi167 conmon[60316]: debug 2022-01-31T22:31:03.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:03.624118+0000) 2022-01-31T22:31:04.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:04 smithi171 conmon[46715]: debug 2022-01-31T22:31:04.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:04.365342+0000) 2022-01-31T22:31:04.530 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:04 smithi171 conmon[51620]: debug 2022-01-31T22:31:04.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:04.227664+0000) 2022-01-31T22:31:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:04 smithi171 conmon[41853]: debug 2022-01-31T22:31:04.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:04.569816+0000) 2022-01-31T22:31:04.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:04 smithi167 conmon[49112]: debug 2022-01-31T22:31:04.696+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:04.697831+0000) 2022-01-31T22:31:04.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:04 smithi167 conmon[54076]: debug 2022-01-31T22:31:04.865+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:04.866387+0000) 2022-01-31T22:31:04.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:04 smithi167 conmon[60316]: debug 2022-01-31T22:31:04.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:04.624296+0000) 2022-01-31T22:31:05.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:05 smithi171 conmon[46715]: debug 2022-01-31T22:31:05.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:05.365531+0000) 2022-01-31T22:31:05.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:05 smithi171 conmon[51620]: debug 2022-01-31T22:31:05.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:05.227842+0000) 2022-01-31T22:31:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:05 smithi171 conmon[41853]: debug 2022-01-31T22:31:05.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:05.570020+0000) 2022-01-31T22:31:05.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:05 smithi167 conmon[49112]: debug 2022-01-31T22:31:05.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:05.698047+0000) 2022-01-31T22:31:05.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:05 smithi167 conmon[60316]: debug 2022-01-31T22:31:05.623+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:05.624493+0000) 2022-01-31T22:31:05.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:05 smithi167 conmon[54076]: debug 2022-01-31T22:31:05.865+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:05.866602+0000) 2022-01-31T22:31:06.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[41853]: debug 2022-01-31T22:31:06.240+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.240890+0000) 2022-01-31T22:31:06.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[46715]: debug 2022-01-31T22:31:06.238+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.239181+0000) 2022-01-31T22:31:06.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[46715]: debug 2022-01-31T22:31:06.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.365734+0000) 2022-01-31T22:31:06.532 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[35325]: debug 2022-01-31T22:31:06.261+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215836 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:06.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[51620]: debug 2022-01-31T22:31:06.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.228016+0000) 2022-01-31T22:31:06.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[51620]: debug 2022-01-31T22:31:06.239+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.239797+0000) 2022-01-31T22:31:06.624 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:06 smithi167 conmon[49112]: debug 2022-01-31T22:31:06.238+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.239653+0000) 2022-01-31T22:31:06.625 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:06 smithi167 conmon[54076]: debug 2022-01-31T22:31:06.238+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.239898+0000) 2022-01-31T22:31:06.625 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:06 smithi167 conmon[60316]: debug 2022-01-31T22:31:06.239+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.240856+0000) 2022-01-31T22:31:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:06 smithi171 conmon[41853]: debug 2022-01-31T22:31:06.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.570141+0000) 2022-01-31T22:31:06.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:06 smithi167 conmon[49112]: debug 2022-01-31T22:31:06.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.698232+0000) 2022-01-31T22:31:06.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:06 smithi167 conmon[54076]: debug 2022-01-31T22:31:06.866+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.866738+0000) 2022-01-31T22:31:06.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:06 smithi167 conmon[60316]: debug 2022-01-31T22:31:06.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:06.624685+0000) 2022-01-31T22:31:07.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:07 smithi171 conmon[46715]: debug 2022-01-31T22:31:07.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:07.365906+0000) 2022-01-31T22:31:07.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:07 smithi171 conmon[51620]: debug 2022-01-31T22:31:07.228+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:07.228215+0000) 2022-01-31T22:31:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:07 smithi171 conmon[41853]: debug 2022-01-31T22:31:07.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:07.570257+0000) 2022-01-31T22:31:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:07 smithi167 conmon[49112]: debug 2022-01-31T22:31:07.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:07.698360+0000) 2022-01-31T22:31:07.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:07 smithi167 conmon[54076]: debug 2022-01-31T22:31:07.866+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:07.866838+0000) 2022-01-31T22:31:07.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:07 smithi167 conmon[60316]: debug 2022-01-31T22:31:07.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:07.624812+0000) 2022-01-31T22:31:08.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:08 smithi171 conmon[46715]: debug 2022-01-31T22:31:08.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:08.366093+0000) 2022-01-31T22:31:08.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:08 smithi171 conmon[51620]: debug 2022-01-31T22:31:08.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:08.228363+0000) 2022-01-31T22:31:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:08 smithi171 conmon[41853]: debug 2022-01-31T22:31:08.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:08.570456+0000) 2022-01-31T22:31:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:08 smithi167 conmon[49112]: debug 2022-01-31T22:31:08.697+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:08.698543+0000) 2022-01-31T22:31:08.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:08 smithi167 conmon[54076]: debug 2022-01-31T22:31:08.865+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:08.866967+0000) 2022-01-31T22:31:08.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:08 smithi167 conmon[60316]: debug 2022-01-31T22:31:08.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:08.624990+0000) 2022-01-31T22:31:09.228 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:08 smithi171 conmon[35325]: debug 2022-01-31T22:31:08.865+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:31:09.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:09 smithi171 conmon[46715]: debug 2022-01-31T22:31:09.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:09.366279+0000) 2022-01-31T22:31:09.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:09 smithi171 conmon[51620]: debug 2022-01-31T22:31:09.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:09.228565+0000) 2022-01-31T22:31:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:09 smithi171 conmon[41853]: debug 2022-01-31T22:31:09.568+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:09.570610+0000) 2022-01-31T22:31:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:09 smithi167 conmon[49112]: debug 2022-01-31T22:31:09.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:09.698714+0000) 2022-01-31T22:31:09.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:09 smithi167 conmon[60316]: debug 2022-01-31T22:31:09.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:09.625191+0000) 2022-01-31T22:31:09.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:09 smithi167 conmon[54076]: debug 2022-01-31T22:31:09.866+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:09.867148+0000) 2022-01-31T22:31:10.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:10 smithi171 conmon[46715]: debug 2022-01-31T22:31:10.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:10.366484+0000) 2022-01-31T22:31:10.531 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:10 smithi171 conmon[51620]: debug 2022-01-31T22:31:10.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:10.228794+0000) 2022-01-31T22:31:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:10 smithi171 conmon[41853]: debug 2022-01-31T22:31:10.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:10.570786+0000) 2022-01-31T22:31:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:10 smithi167 conmon[49112]: debug 2022-01-31T22:31:10.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:10.698874+0000) 2022-01-31T22:31:10.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:10 smithi167 conmon[60316]: debug 2022-01-31T22:31:10.624+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:10.625378+0000) 2022-01-31T22:31:10.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:10 smithi167 conmon[54076]: debug 2022-01-31T22:31:10.866+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:10.867307+0000) 2022-01-31T22:31:11.531 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[35325]: debug 2022-01-31T22:31:11.284+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 215946 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:11.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[41853]: debug 2022-01-31T22:31:11.262+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.263818+0000) 2022-01-31T22:31:11.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[46715]: debug 2022-01-31T22:31:11.262+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.264289+0000) 2022-01-31T22:31:11.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[46715]: debug 2022-01-31T22:31:11.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.366702+0000) 2022-01-31T22:31:11.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[51620]: debug 2022-01-31T22:31:11.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.228969+0000) 2022-01-31T22:31:11.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[51620]: debug 2022-01-31T22:31:11.263+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.264956+0000) 2022-01-31T22:31:11.625 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:11 smithi167 conmon[49112]: debug 2022-01-31T22:31:11.264+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.265193+0000) 2022-01-31T22:31:11.626 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:11 smithi167 conmon[54076]: debug 2022-01-31T22:31:11.265+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.265898+0000) 2022-01-31T22:31:11.627 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:11 smithi167 conmon[60316]: debug 2022-01-31T22:31:11.264+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.264753+0000) 2022-01-31T22:31:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:11 smithi171 conmon[41853]: debug 2022-01-31T22:31:11.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.570911+0000) 2022-01-31T22:31:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:11 smithi167 conmon[49112]: debug 2022-01-31T22:31:11.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.699101+0000) 2022-01-31T22:31:11.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:11 smithi167 conmon[60316]: debug 2022-01-31T22:31:11.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.625536+0000) 2022-01-31T22:31:11.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:11 smithi167 conmon[54076]: debug 2022-01-31T22:31:11.867+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:11.867511+0000) 2022-01-31T22:31:12.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:12 smithi171 conmon[46715]: debug 2022-01-31T22:31:12.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:12.366856+0000) 2022-01-31T22:31:12.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:12 smithi171 conmon[51620]: debug 2022-01-31T22:31:12.228+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:12.229137+0000) 2022-01-31T22:31:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:12 smithi171 conmon[41853]: debug 2022-01-31T22:31:12.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:12.571067+0000) 2022-01-31T22:31:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:12 smithi167 conmon[49112]: debug 2022-01-31T22:31:12.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:12.699233+0000) 2022-01-31T22:31:12.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:12 smithi167 conmon[54076]: debug 2022-01-31T22:31:12.867+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:12.867673+0000) 2022-01-31T22:31:12.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:12 smithi167 conmon[60316]: debug 2022-01-31T22:31:12.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:12.625686+0000) 2022-01-31T22:31:13.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:13 smithi171 conmon[46715]: debug 2022-01-31T22:31:13.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:13.367056+0000) 2022-01-31T22:31:13.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:13 smithi171 conmon[51620]: debug 2022-01-31T22:31:13.228+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:13.229298+0000) 2022-01-31T22:31:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:13 smithi171 conmon[41853]: debug 2022-01-31T22:31:13.569+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:13.571295+0000) 2022-01-31T22:31:13.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:13 smithi167 conmon[49112]: debug 2022-01-31T22:31:13.698+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:13.699394+0000) 2022-01-31T22:31:13.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:13 smithi167 conmon[54076]: debug 2022-01-31T22:31:13.867+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:13.867855+0000) 2022-01-31T22:31:13.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:13 smithi167 conmon[60316]: debug 2022-01-31T22:31:13.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:13.625851+0000) 2022-01-31T22:31:14.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:14 smithi171 conmon[46715]: debug 2022-01-31T22:31:14.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:14.367277+0000) 2022-01-31T22:31:14.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:14 smithi171 conmon[51620]: debug 2022-01-31T22:31:14.227+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:14.229485+0000) 2022-01-31T22:31:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:14 smithi171 conmon[41853]: debug 2022-01-31T22:31:14.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:14.571476+0000) 2022-01-31T22:31:14.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:14 smithi167 conmon[49112]: debug 2022-01-31T22:31:14.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:14.699592+0000) 2022-01-31T22:31:14.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:14 smithi167 conmon[54076]: debug 2022-01-31T22:31:14.867+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:14.868052+0000) 2022-01-31T22:31:14.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:14 smithi167 conmon[60316]: debug 2022-01-31T22:31:14.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:14.626064+0000) 2022-01-31T22:31:15.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:15 smithi171 conmon[46715]: debug 2022-01-31T22:31:15.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:15.367484+0000) 2022-01-31T22:31:15.532 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:15 smithi171 conmon[51620]: debug 2022-01-31T22:31:15.228+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:15.229669+0000) 2022-01-31T22:31:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:15 smithi171 conmon[41853]: debug 2022-01-31T22:31:15.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:15.571651+0000) 2022-01-31T22:31:15.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:15 smithi167 conmon[54076]: debug 2022-01-31T22:31:15.867+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:15.868228+0000) 2022-01-31T22:31:15.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:15 smithi167 conmon[49112]: debug 2022-01-31T22:31:15.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:15.699775+0000) 2022-01-31T22:31:15.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:15 smithi167 conmon[60316]: debug 2022-01-31T22:31:15.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:15.626256+0000) 2022-01-31T22:31:16.532 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[35325]: debug 2022-01-31T22:31:16.309+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216059 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:16.533 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[41853]: debug 2022-01-31T22:31:16.287+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.288844+0000) 2022-01-31T22:31:16.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[51620]: debug 2022-01-31T22:31:16.229+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.229886+0000) 2022-01-31T22:31:16.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[51620]: debug 2022-01-31T22:31:16.287+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.289439+0000) 2022-01-31T22:31:16.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[46715]: debug 2022-01-31T22:31:16.286+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.288648+0000) 2022-01-31T22:31:16.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[46715]: debug 2022-01-31T22:31:16.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.367673+0000) 2022-01-31T22:31:16.625 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:16 smithi167 conmon[49112]: debug 2022-01-31T22:31:16.289+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.290096+0000) 2022-01-31T22:31:16.626 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:16 smithi167 conmon[54076]: debug 2022-01-31T22:31:16.289+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.290223+0000) 2022-01-31T22:31:16.627 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:16 smithi167 conmon[60316]: debug 2022-01-31T22:31:16.289+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.289988+0000) 2022-01-31T22:31:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:16 smithi171 conmon[41853]: debug 2022-01-31T22:31:16.570+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.571784+0000) 2022-01-31T22:31:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:16 smithi167 conmon[49112]: debug 2022-01-31T22:31:16.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.699992+0000) 2022-01-31T22:31:16.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:16 smithi167 conmon[54076]: debug 2022-01-31T22:31:16.867+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.868374+0000) 2022-01-31T22:31:16.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:16 smithi167 conmon[60316]: debug 2022-01-31T22:31:16.625+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:16.626381+0000) 2022-01-31T22:31:17.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:17 smithi171 conmon[46715]: debug 2022-01-31T22:31:17.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:17.367816+0000) 2022-01-31T22:31:17.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:17 smithi171 conmon[51620]: debug 2022-01-31T22:31:17.229+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:17.230063+0000) 2022-01-31T22:31:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:17 smithi171 conmon[41853]: debug 2022-01-31T22:31:17.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:17.571869+0000) 2022-01-31T22:31:17.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:17 smithi167 conmon[49112]: debug 2022-01-31T22:31:17.699+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:17.700141+0000) 2022-01-31T22:31:17.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:17 smithi167 conmon[54076]: debug 2022-01-31T22:31:17.868+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:17.868546+0000) 2022-01-31T22:31:17.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:17 smithi167 conmon[60316]: debug 2022-01-31T22:31:17.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:17.626524+0000) 2022-01-31T22:31:18.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:18 smithi171 conmon[46715]: debug 2022-01-31T22:31:18.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:18.367948+0000) 2022-01-31T22:31:18.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:18 smithi171 conmon[51620]: debug 2022-01-31T22:31:18.229+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:18.230223+0000) 2022-01-31T22:31:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:18 smithi171 conmon[41853]: debug 2022-01-31T22:31:18.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:18.572025+0000) 2022-01-31T22:31:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:18 smithi167 conmon[49112]: debug 2022-01-31T22:31:18.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:18.700348+0000) 2022-01-31T22:31:18.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:18 smithi167 conmon[54076]: debug 2022-01-31T22:31:18.868+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:18.868709+0000) 2022-01-31T22:31:18.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:18 smithi167 conmon[60316]: debug 2022-01-31T22:31:18.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:18.626705+0000) 2022-01-31T22:31:19.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:19 smithi171 conmon[46715]: debug 2022-01-31T22:31:19.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:19.368113+0000) 2022-01-31T22:31:19.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:19 smithi171 conmon[51620]: debug 2022-01-31T22:31:19.229+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:19.230421+0000) 2022-01-31T22:31:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:19 smithi171 conmon[41853]: debug 2022-01-31T22:31:19.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:19.572158+0000) 2022-01-31T22:31:19.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:19 smithi167 conmon[49112]: debug 2022-01-31T22:31:19.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:19.700538+0000) 2022-01-31T22:31:19.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:19 smithi167 conmon[54076]: debug 2022-01-31T22:31:19.868+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:19.868894+0000) 2022-01-31T22:31:19.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:19 smithi167 conmon[60316]: debug 2022-01-31T22:31:19.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:19.626888+0000) 2022-01-31T22:31:20.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:20 smithi171 conmon[46715]: debug 2022-01-31T22:31:20.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:20.368303+0000) 2022-01-31T22:31:20.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:20 smithi171 conmon[51620]: debug 2022-01-31T22:31:20.229+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:20.230655+0000) 2022-01-31T22:31:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:20 smithi171 conmon[41853]: debug 2022-01-31T22:31:20.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:20.572313+0000) 2022-01-31T22:31:20.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:20 smithi167 conmon[49112]: debug 2022-01-31T22:31:20.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:20.700781+0000) 2022-01-31T22:31:20.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:20 smithi167 conmon[54076]: debug 2022-01-31T22:31:20.868+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:20.869072+0000) 2022-01-31T22:31:20.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:20 smithi167 conmon[60316]: debug 2022-01-31T22:31:20.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:20.627076+0000) 2022-01-31T22:31:21.533 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[35325]: debug 2022-01-31T22:31:21.335+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216170 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:21.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[41853]: debug 2022-01-31T22:31:21.313+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.313832+0000) 2022-01-31T22:31:21.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[46715]: debug 2022-01-31T22:31:21.313+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.313904+0000) 2022-01-31T22:31:21.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[46715]: debug 2022-01-31T22:31:21.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.368478+0000) 2022-01-31T22:31:21.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[51620]: debug 2022-01-31T22:31:21.230+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.230831+0000) 2022-01-31T22:31:21.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[51620]: debug 2022-01-31T22:31:21.314+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.314886+0000) 2022-01-31T22:31:21.627 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:21 smithi167 conmon[49112]: debug 2022-01-31T22:31:21.314+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.315083+0000) 2022-01-31T22:31:21.627 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:21 smithi167 conmon[54076]: debug 2022-01-31T22:31:21.313+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.314112+0000) 2022-01-31T22:31:21.628 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:21 smithi167 conmon[60316]: debug 2022-01-31T22:31:21.315+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.315625+0000) 2022-01-31T22:31:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:21 smithi171 conmon[41853]: debug 2022-01-31T22:31:21.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.572511+0000) 2022-01-31T22:31:21.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:21 smithi167 conmon[54076]: debug 2022-01-31T22:31:21.868+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.869255+0000) 2022-01-31T22:31:21.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:21 smithi167 conmon[60316]: debug 2022-01-31T22:31:21.626+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.627234+0000) 2022-01-31T22:31:21.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:21 smithi167 conmon[49112]: debug 2022-01-31T22:31:21.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:21.700982+0000) 2022-01-31T22:31:22.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:22 smithi171 conmon[46715]: debug 2022-01-31T22:31:22.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:22.368656+0000) 2022-01-31T22:31:22.533 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:22 smithi171 conmon[51620]: debug 2022-01-31T22:31:22.230+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:22.230989+0000) 2022-01-31T22:31:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:22 smithi171 conmon[41853]: debug 2022-01-31T22:31:22.571+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:22.572620+0000) 2022-01-31T22:31:22.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:22 smithi167 conmon[49112]: debug 2022-01-31T22:31:22.700+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:22.701128+0000) 2022-01-31T22:31:22.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:22 smithi167 conmon[54076]: debug 2022-01-31T22:31:22.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:22.869383+0000) 2022-01-31T22:31:22.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:22 smithi167 conmon[60316]: debug 2022-01-31T22:31:22.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:22.627452+0000) 2022-01-31T22:31:23.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:23 smithi171 conmon[46715]: debug 2022-01-31T22:31:23.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:23.368857+0000) 2022-01-31T22:31:23.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:23 smithi171 conmon[51620]: debug 2022-01-31T22:31:23.230+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:23.231158+0000) 2022-01-31T22:31:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:23 smithi171 conmon[41853]: debug 2022-01-31T22:31:23.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:23.572799+0000) 2022-01-31T22:31:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:23 smithi167 conmon[49112]: debug 2022-01-31T22:31:23.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:23.701337+0000) 2022-01-31T22:31:23.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:23 smithi167 conmon[54076]: debug 2022-01-31T22:31:23.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:23.869519+0000) 2022-01-31T22:31:23.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:23 smithi167 conmon[60316]: debug 2022-01-31T22:31:23.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:23.627680+0000) 2022-01-31T22:31:24.231 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:23 smithi171 conmon[35325]: debug 2022-01-31T22:31:23.866+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:31:24.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:24 smithi171 conmon[46715]: debug 2022-01-31T22:31:24.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:24.369062+0000) 2022-01-31T22:31:24.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:24 smithi171 conmon[51620]: debug 2022-01-31T22:31:24.230+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:24.231319+0000) 2022-01-31T22:31:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:24 smithi171 conmon[41853]: debug 2022-01-31T22:31:24.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:24.573005+0000) 2022-01-31T22:31:24.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:24 smithi167 conmon[49112]: debug 2022-01-31T22:31:24.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:24.701540+0000) 2022-01-31T22:31:24.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:24 smithi167 conmon[54076]: debug 2022-01-31T22:31:24.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:24.869735+0000) 2022-01-31T22:31:24.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:24 smithi167 conmon[60316]: debug 2022-01-31T22:31:24.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:24.627847+0000) 2022-01-31T22:31:25.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:25 smithi171 conmon[46715]: debug 2022-01-31T22:31:25.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:25.369272+0000) 2022-01-31T22:31:25.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:25 smithi171 conmon[51620]: debug 2022-01-31T22:31:25.230+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:25.231520+0000) 2022-01-31T22:31:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:25 smithi171 conmon[41853]: debug 2022-01-31T22:31:25.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:25.573147+0000) 2022-01-31T22:31:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:25 smithi167 conmon[49112]: debug 2022-01-31T22:31:25.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:25.701676+0000) 2022-01-31T22:31:25.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:25 smithi167 conmon[54076]: debug 2022-01-31T22:31:25.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:25.869892+0000) 2022-01-31T22:31:25.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:25 smithi167 conmon[60316]: debug 2022-01-31T22:31:25.627+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:25.628028+0000) 2022-01-31T22:31:26.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[41853]: debug 2022-01-31T22:31:26.337+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.338306+0000) 2022-01-31T22:31:26.534 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[51620]: debug 2022-01-31T22:31:26.231+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.231737+0000) 2022-01-31T22:31:26.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[51620]: debug 2022-01-31T22:31:26.339+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.340270+0000) 2022-01-31T22:31:26.535 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[35325]: debug 2022-01-31T22:31:26.359+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216278 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:26.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[46715]: debug 2022-01-31T22:31:26.338+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.339097+0000) 2022-01-31T22:31:26.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[46715]: debug 2022-01-31T22:31:26.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.369424+0000) 2022-01-31T22:31:26.627 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:26 smithi167 conmon[49112]: debug 2022-01-31T22:31:26.339+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.339605+0000) 2022-01-31T22:31:26.628 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:26 smithi167 conmon[54076]: debug 2022-01-31T22:31:26.338+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.338954+0000) 2022-01-31T22:31:26.628 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:26 smithi167 conmon[60316]: debug 2022-01-31T22:31:26.339+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.339744+0000) 2022-01-31T22:31:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:26 smithi171 conmon[41853]: debug 2022-01-31T22:31:26.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.573325+0000) 2022-01-31T22:31:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:26 smithi167 conmon[49112]: debug 2022-01-31T22:31:26.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.701850+0000) 2022-01-31T22:31:26.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:26 smithi167 conmon[54076]: debug 2022-01-31T22:31:26.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.870049+0000) 2022-01-31T22:31:26.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:26 smithi167 conmon[60316]: debug 2022-01-31T22:31:26.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:26.628231+0000) 2022-01-31T22:31:27.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:27 smithi171 conmon[46715]: debug 2022-01-31T22:31:27.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:27.369589+0000) 2022-01-31T22:31:27.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:27 smithi171 conmon[51620]: debug 2022-01-31T22:31:27.231+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:27.231857+0000) 2022-01-31T22:31:27.809 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:27 smithi171 conmon[41853]: debug 2022-01-31T22:31:27.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:27.573498+0000) 2022-01-31T22:31:27.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:27 smithi167 conmon[49112]: debug 2022-01-31T22:31:27.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:27.702025+0000) 2022-01-31T22:31:27.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:27 smithi167 conmon[54076]: debug 2022-01-31T22:31:27.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:27.870202+0000) 2022-01-31T22:31:27.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:27 smithi167 conmon[60316]: debug 2022-01-31T22:31:27.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:27.628382+0000) 2022-01-31T22:31:28.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:28 smithi171 conmon[51620]: debug 2022-01-31T22:31:28.231+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:28.232019+0000) 2022-01-31T22:31:28.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:28 smithi171 conmon[46715]: debug 2022-01-31T22:31:28.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:28.369788+0000) 2022-01-31T22:31:28.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:28 smithi171 conmon[41853]: debug 2022-01-31T22:31:28.572+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:28.573630+0000) 2022-01-31T22:31:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:28 smithi167 conmon[49112]: debug 2022-01-31T22:31:28.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:28.702197+0000) 2022-01-31T22:31:28.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:28 smithi167 conmon[54076]: debug 2022-01-31T22:31:28.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:28.870330+0000) 2022-01-31T22:31:28.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:28 smithi167 conmon[60316]: debug 2022-01-31T22:31:28.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:28.628560+0000) 2022-01-31T22:31:29.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:29 smithi171 conmon[46715]: debug 2022-01-31T22:31:29.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:29.369977+0000) 2022-01-31T22:31:29.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:29 smithi171 conmon[51620]: debug 2022-01-31T22:31:29.231+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:29.232215+0000) 2022-01-31T22:31:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:29 smithi171 conmon[41853]: debug 2022-01-31T22:31:29.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:29.573785+0000) 2022-01-31T22:31:29.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:29 smithi167 conmon[49112]: debug 2022-01-31T22:31:29.701+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:29.702412+0000) 2022-01-31T22:31:29.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:29 smithi167 conmon[54076]: debug 2022-01-31T22:31:29.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:29.870523+0000) 2022-01-31T22:31:29.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:29 smithi167 conmon[60316]: debug 2022-01-31T22:31:29.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:29.628724+0000) 2022-01-31T22:31:30.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:30 smithi171 conmon[46715]: debug 2022-01-31T22:31:30.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:30.370163+0000) 2022-01-31T22:31:30.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:30 smithi171 conmon[51620]: debug 2022-01-31T22:31:30.231+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:30.232413+0000) 2022-01-31T22:31:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:30 smithi171 conmon[41853]: debug 2022-01-31T22:31:30.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:30.573979+0000) 2022-01-31T22:31:30.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:30 smithi167 conmon[49112]: debug 2022-01-31T22:31:30.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:30.702619+0000) 2022-01-31T22:31:30.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:30 smithi167 conmon[54076]: debug 2022-01-31T22:31:30.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:30.870725+0000) 2022-01-31T22:31:30.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:30 smithi167 conmon[60316]: debug 2022-01-31T22:31:30.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:30.628852+0000) 2022-01-31T22:31:31.534 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[35325]: debug 2022-01-31T22:31:31.384+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216388 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:31.535 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[41853]: debug 2022-01-31T22:31:31.361+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.362656+0000) 2022-01-31T22:31:31.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[46715]: debug 2022-01-31T22:31:31.362+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.363190+0000) 2022-01-31T22:31:31.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[46715]: debug 2022-01-31T22:31:31.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.370323+0000) 2022-01-31T22:31:31.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[51620]: debug 2022-01-31T22:31:31.232+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.232585+0000) 2022-01-31T22:31:31.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[51620]: debug 2022-01-31T22:31:31.362+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.363698+0000) 2022-01-31T22:31:31.628 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:31 smithi167 conmon[49112]: debug 2022-01-31T22:31:31.364+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.364873+0000) 2022-01-31T22:31:31.629 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:31 smithi167 conmon[54076]: debug 2022-01-31T22:31:31.363+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.364074+0000) 2022-01-31T22:31:31.630 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:31 smithi167 conmon[60316]: debug 2022-01-31T22:31:31.369+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.369475+0000) 2022-01-31T22:31:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:31 smithi171 conmon[41853]: debug 2022-01-31T22:31:31.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.574114+0000) 2022-01-31T22:31:31.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:31 smithi167 conmon[49112]: debug 2022-01-31T22:31:31.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.702810+0000) 2022-01-31T22:31:31.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:31 smithi167 conmon[54076]: debug 2022-01-31T22:31:31.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.870870+0000) 2022-01-31T22:31:31.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:31 smithi167 conmon[60316]: debug 2022-01-31T22:31:31.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:31.629031+0000) 2022-01-31T22:31:32.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:32 smithi171 conmon[46715]: debug 2022-01-31T22:31:32.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:32.370489+0000) 2022-01-31T22:31:32.535 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:32 smithi171 conmon[51620]: debug 2022-01-31T22:31:32.232+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:32.232733+0000) 2022-01-31T22:31:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:32 smithi171 conmon[41853]: debug 2022-01-31T22:31:32.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:32.574230+0000) 2022-01-31T22:31:32.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:32 smithi167 conmon[49112]: debug 2022-01-31T22:31:32.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:32.702969+0000) 2022-01-31T22:31:32.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:32 smithi167 conmon[54076]: debug 2022-01-31T22:31:32.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:32.871027+0000) 2022-01-31T22:31:32.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:32 smithi167 conmon[60316]: debug 2022-01-31T22:31:32.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:32.629186+0000) 2022-01-31T22:31:33.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:33 smithi171 conmon[46715]: debug 2022-01-31T22:31:33.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:33.370675+0000) 2022-01-31T22:31:33.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:33 smithi171 conmon[51620]: debug 2022-01-31T22:31:33.232+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:33.232910+0000) 2022-01-31T22:31:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:33 smithi171 conmon[41853]: debug 2022-01-31T22:31:33.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:33.574407+0000) 2022-01-31T22:31:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:33 smithi167 conmon[49112]: debug 2022-01-31T22:31:33.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:33.703159+0000) 2022-01-31T22:31:33.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:33 smithi167 conmon[54076]: debug 2022-01-31T22:31:33.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:33.871222+0000) 2022-01-31T22:31:33.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:33 smithi167 conmon[60316]: debug 2022-01-31T22:31:33.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:33.629371+0000) 2022-01-31T22:31:34.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:34 smithi171 conmon[46715]: debug 2022-01-31T22:31:34.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:34.370920+0000) 2022-01-31T22:31:34.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:34 smithi171 conmon[51620]: debug 2022-01-31T22:31:34.232+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:34.233122+0000) 2022-01-31T22:31:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:34 smithi171 conmon[41853]: debug 2022-01-31T22:31:34.573+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:34.574559+0000) 2022-01-31T22:31:34.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:34 smithi167 conmon[49112]: debug 2022-01-31T22:31:34.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:34.703347+0000) 2022-01-31T22:31:34.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:34 smithi167 conmon[54076]: debug 2022-01-31T22:31:34.869+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:34.871386+0000) 2022-01-31T22:31:34.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:34 smithi167 conmon[60316]: debug 2022-01-31T22:31:34.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:34.629581+0000) 2022-01-31T22:31:35.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:35 smithi171 conmon[46715]: debug 2022-01-31T22:31:35.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:35.371106+0000) 2022-01-31T22:31:35.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:35 smithi171 conmon[51620]: debug 2022-01-31T22:31:35.232+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:35.233284+0000) 2022-01-31T22:31:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:35 smithi171 conmon[41853]: debug 2022-01-31T22:31:35.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:35.574715+0000) 2022-01-31T22:31:35.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:35 smithi167 conmon[49112]: debug 2022-01-31T22:31:35.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:35.703535+0000) 2022-01-31T22:31:35.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:35 smithi167 conmon[54076]: debug 2022-01-31T22:31:35.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:35.871582+0000) 2022-01-31T22:31:35.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:35 smithi167 conmon[60316]: debug 2022-01-31T22:31:35.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:35.629769+0000) 2022-01-31T22:31:36.535 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[35325]: debug 2022-01-31T22:31:36.408+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216501 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:36.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[41853]: debug 2022-01-31T22:31:36.386+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.387374+0000) 2022-01-31T22:31:36.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[46715]: debug 2022-01-31T22:31:36.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.371292+0000) 2022-01-31T22:31:36.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[46715]: debug 2022-01-31T22:31:36.385+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.386690+0000) 2022-01-31T22:31:36.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[51620]: debug 2022-01-31T22:31:36.232+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.233424+0000) 2022-01-31T22:31:36.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[51620]: debug 2022-01-31T22:31:36.387+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.387869+0000) 2022-01-31T22:31:36.629 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:36 smithi167 conmon[49112]: debug 2022-01-31T22:31:36.386+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.388253+0000) 2022-01-31T22:31:36.630 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:36 smithi167 conmon[54076]: debug 2022-01-31T22:31:36.386+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.387830+0000) 2022-01-31T22:31:36.631 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:36 smithi167 conmon[60316]: debug 2022-01-31T22:31:36.386+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.387719+0000) 2022-01-31T22:31:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:36 smithi171 conmon[41853]: debug 2022-01-31T22:31:36.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.574884+0000) 2022-01-31T22:31:36.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:36 smithi167 conmon[49112]: debug 2022-01-31T22:31:36.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.703722+0000) 2022-01-31T22:31:36.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:36 smithi167 conmon[60316]: debug 2022-01-31T22:31:36.628+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.629961+0000) 2022-01-31T22:31:36.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:36 smithi167 conmon[54076]: debug 2022-01-31T22:31:36.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:36.871797+0000) 2022-01-31T22:31:37.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:37 smithi171 conmon[46715]: debug 2022-01-31T22:31:37.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:37.371481+0000) 2022-01-31T22:31:37.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:37 smithi171 conmon[51620]: debug 2022-01-31T22:31:37.233+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:37.233599+0000) 2022-01-31T22:31:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:37 smithi171 conmon[41853]: debug 2022-01-31T22:31:37.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:37.575001+0000) 2022-01-31T22:31:37.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:37 smithi167 conmon[60316]: debug 2022-01-31T22:31:37.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:37.630124+0000) 2022-01-31T22:31:37.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:37 smithi167 conmon[49112]: debug 2022-01-31T22:31:37.702+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:37.703896+0000) 2022-01-31T22:31:37.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:37 smithi167 conmon[54076]: debug 2022-01-31T22:31:37.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:37.871946+0000) 2022-01-31T22:31:38.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:38 smithi171 conmon[46715]: debug 2022-01-31T22:31:38.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:38.371652+0000) 2022-01-31T22:31:38.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:38 smithi171 conmon[51620]: debug 2022-01-31T22:31:38.233+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:38.233770+0000) 2022-01-31T22:31:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:38 smithi171 conmon[41853]: debug 2022-01-31T22:31:38.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:38.575235+0000) 2022-01-31T22:31:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:38 smithi167 conmon[49112]: debug 2022-01-31T22:31:38.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:38.704105+0000) 2022-01-31T22:31:38.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:38 smithi167 conmon[54076]: debug 2022-01-31T22:31:38.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:38.872118+0000) 2022-01-31T22:31:38.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:38 smithi167 conmon[60316]: debug 2022-01-31T22:31:38.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:38.630265+0000) 2022-01-31T22:31:39.233 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:38 smithi171 conmon[35325]: debug 2022-01-31T22:31:38.867+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:31:39.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:39 smithi171 conmon[46715]: debug 2022-01-31T22:31:39.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:39.371857+0000) 2022-01-31T22:31:39.536 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:39 smithi171 conmon[51620]: debug 2022-01-31T22:31:39.233+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:39.233969+0000) 2022-01-31T22:31:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:39 smithi171 conmon[41853]: debug 2022-01-31T22:31:39.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:39.575395+0000) 2022-01-31T22:31:39.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:39 smithi167 conmon[49112]: debug 2022-01-31T22:31:39.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:39.704306+0000) 2022-01-31T22:31:39.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:39 smithi167 conmon[54076]: debug 2022-01-31T22:31:39.870+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:39.872244+0000) 2022-01-31T22:31:39.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:39 smithi167 conmon[60316]: debug 2022-01-31T22:31:39.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:39.630425+0000) 2022-01-31T22:31:40.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:40 smithi171 conmon[46715]: debug 2022-01-31T22:31:40.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:40.372040+0000) 2022-01-31T22:31:40.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:40 smithi171 conmon[51620]: debug 2022-01-31T22:31:40.233+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:40.234163+0000) 2022-01-31T22:31:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:40 smithi171 conmon[41853]: debug 2022-01-31T22:31:40.574+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:40.575578+0000) 2022-01-31T22:31:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:40 smithi167 conmon[49112]: debug 2022-01-31T22:31:40.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:40.704482+0000) 2022-01-31T22:31:40.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:40 smithi167 conmon[54076]: debug 2022-01-31T22:31:40.871+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:40.872407+0000) 2022-01-31T22:31:40.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:40 smithi167 conmon[60316]: debug 2022-01-31T22:31:40.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:40.630614+0000) 2022-01-31T22:31:41.536 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[35325]: debug 2022-01-31T22:31:41.433+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216625 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:41.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[41853]: debug 2022-01-31T22:31:41.411+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.411833+0000) 2022-01-31T22:31:41.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[46715]: debug 2022-01-31T22:31:41.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.372232+0000) 2022-01-31T22:31:41.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[46715]: debug 2022-01-31T22:31:41.410+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.411102+0000) 2022-01-31T22:31:41.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[51620]: debug 2022-01-31T22:31:41.233+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.234359+0000) 2022-01-31T22:31:41.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[51620]: debug 2022-01-31T22:31:41.411+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.412110+0000) 2022-01-31T22:31:41.630 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:41 smithi167 conmon[49112]: debug 2022-01-31T22:31:41.411+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.412515+0000) 2022-01-31T22:31:41.631 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:41 smithi167 conmon[54076]: debug 2022-01-31T22:31:41.410+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.412425+0000) 2022-01-31T22:31:41.631 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:41 smithi167 conmon[60316]: debug 2022-01-31T22:31:41.410+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.412314+0000) 2022-01-31T22:31:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:41 smithi171 conmon[41853]: debug 2022-01-31T22:31:41.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.575704+0000) 2022-01-31T22:31:41.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:41 smithi167 conmon[49112]: debug 2022-01-31T22:31:41.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.704711+0000) 2022-01-31T22:31:41.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:41 smithi167 conmon[54076]: debug 2022-01-31T22:31:41.871+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.872604+0000) 2022-01-31T22:31:41.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:41 smithi167 conmon[60316]: debug 2022-01-31T22:31:41.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:41.630811+0000) 2022-01-31T22:31:42.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:42 smithi171 conmon[46715]: debug 2022-01-31T22:31:42.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:42.372384+0000) 2022-01-31T22:31:42.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:42 smithi171 conmon[51620]: debug 2022-01-31T22:31:42.233+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:42.234488+0000) 2022-01-31T22:31:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:42 smithi171 conmon[41853]: debug 2022-01-31T22:31:42.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:42.575846+0000) 2022-01-31T22:31:42.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:42 smithi167 conmon[49112]: debug 2022-01-31T22:31:42.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:42.704852+0000) 2022-01-31T22:31:42.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:42 smithi167 conmon[54076]: debug 2022-01-31T22:31:42.871+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:42.872758+0000) 2022-01-31T22:31:42.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:42 smithi167 conmon[60316]: debug 2022-01-31T22:31:42.629+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:42.630981+0000) 2022-01-31T22:31:43.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:43 smithi171 conmon[46715]: debug 2022-01-31T22:31:43.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:43.372541+0000) 2022-01-31T22:31:43.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:43 smithi171 conmon[51620]: debug 2022-01-31T22:31:43.234+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:43.234620+0000) 2022-01-31T22:31:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:43 smithi171 conmon[41853]: debug 2022-01-31T22:31:43.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:43.575975+0000) 2022-01-31T22:31:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:43 smithi167 conmon[49112]: debug 2022-01-31T22:31:43.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:43.705037+0000) 2022-01-31T22:31:43.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:43 smithi167 conmon[54076]: debug 2022-01-31T22:31:43.872+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:43.872945+0000) 2022-01-31T22:31:43.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:43 smithi167 conmon[60316]: debug 2022-01-31T22:31:43.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:43.631149+0000) 2022-01-31T22:31:44.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:44 smithi171 conmon[46715]: debug 2022-01-31T22:31:44.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:44.372716+0000) 2022-01-31T22:31:44.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:44 smithi171 conmon[51620]: debug 2022-01-31T22:31:44.234+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:44.234815+0000) 2022-01-31T22:31:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:44 smithi171 conmon[41853]: debug 2022-01-31T22:31:44.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:44.576171+0000) 2022-01-31T22:31:44.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:44 smithi167 conmon[54076]: debug 2022-01-31T22:31:44.872+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:44.873137+0000) 2022-01-31T22:31:44.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:44 smithi167 conmon[60316]: debug 2022-01-31T22:31:44.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:44.631361+0000) 2022-01-31T22:31:44.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:44 smithi167 conmon[49112]: debug 2022-01-31T22:31:44.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:44.705255+0000) 2022-01-31T22:31:45.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:45 smithi171 conmon[46715]: debug 2022-01-31T22:31:45.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:45.372819+0000) 2022-01-31T22:31:45.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:45 smithi171 conmon[51620]: debug 2022-01-31T22:31:45.234+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:45.235027+0000) 2022-01-31T22:31:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:45 smithi171 conmon[41853]: debug 2022-01-31T22:31:45.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:45.576302+0000) 2022-01-31T22:31:45.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:45 smithi167 conmon[49112]: debug 2022-01-31T22:31:45.703+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:45.705412+0000) 2022-01-31T22:31:45.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:45 smithi167 conmon[54076]: debug 2022-01-31T22:31:45.872+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:45.873307+0000) 2022-01-31T22:31:45.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:45 smithi167 conmon[60316]: debug 2022-01-31T22:31:45.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:45.631541+0000) 2022-01-31T22:31:46.537 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[35325]: debug 2022-01-31T22:31:46.459+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216738 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:46.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[41853]: debug 2022-01-31T22:31:46.436+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.437394+0000) 2022-01-31T22:31:46.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[46715]: debug 2022-01-31T22:31:46.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.372976+0000) 2022-01-31T22:31:46.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[46715]: debug 2022-01-31T22:31:46.435+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.436254+0000) 2022-01-31T22:31:46.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[51620]: debug 2022-01-31T22:31:46.234+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.235221+0000) 2022-01-31T22:31:46.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[51620]: debug 2022-01-31T22:31:46.435+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.436391+0000) 2022-01-31T22:31:46.631 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:46 smithi167 conmon[49112]: debug 2022-01-31T22:31:46.436+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.438185+0000) 2022-01-31T22:31:46.632 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:46 smithi167 conmon[54076]: debug 2022-01-31T22:31:46.435+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.436655+0000) 2022-01-31T22:31:46.632 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:46 smithi167 conmon[60316]: debug 2022-01-31T22:31:46.435+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.437226+0000) 2022-01-31T22:31:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:46 smithi171 conmon[41853]: debug 2022-01-31T22:31:46.575+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.576453+0000) 2022-01-31T22:31:46.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:46 smithi167 conmon[49112]: debug 2022-01-31T22:31:46.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.705587+0000) 2022-01-31T22:31:46.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:46 smithi167 conmon[54076]: debug 2022-01-31T22:31:46.871+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.873455+0000) 2022-01-31T22:31:46.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:46 smithi167 conmon[60316]: debug 2022-01-31T22:31:46.630+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:46.631699+0000) 2022-01-31T22:31:47.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:47 smithi171 conmon[46715]: debug 2022-01-31T22:31:47.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:47.373152+0000) 2022-01-31T22:31:47.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:47 smithi171 conmon[51620]: debug 2022-01-31T22:31:47.234+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:47.235431+0000) 2022-01-31T22:31:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:47 smithi171 conmon[41853]: debug 2022-01-31T22:31:47.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:47.576604+0000) 2022-01-31T22:31:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:47 smithi167 conmon[49112]: debug 2022-01-31T22:31:47.704+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:47.705739+0000) 2022-01-31T22:31:47.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:47 smithi167 conmon[54076]: debug 2022-01-31T22:31:47.872+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:47.873540+0000) 2022-01-31T22:31:47.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:47 smithi167 conmon[60316]: debug 2022-01-31T22:31:47.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:47.631826+0000) 2022-01-31T22:31:48.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:48 smithi171 conmon[46715]: debug 2022-01-31T22:31:48.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:48.373309+0000) 2022-01-31T22:31:48.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:48 smithi171 conmon[51620]: debug 2022-01-31T22:31:48.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:48.235600+0000) 2022-01-31T22:31:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:48 smithi171 conmon[41853]: debug 2022-01-31T22:31:48.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:48.576788+0000) 2022-01-31T22:31:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:48 smithi167 conmon[49112]: debug 2022-01-31T22:31:48.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:48.705924+0000) 2022-01-31T22:31:48.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:48 smithi167 conmon[54076]: debug 2022-01-31T22:31:48.872+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:48.873697+0000) 2022-01-31T22:31:48.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:48 smithi167 conmon[60316]: debug 2022-01-31T22:31:48.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:48.631961+0000) 2022-01-31T22:31:49.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:49 smithi171 conmon[46715]: debug 2022-01-31T22:31:49.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:49.373519+0000) 2022-01-31T22:31:49.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:49 smithi171 conmon[51620]: debug 2022-01-31T22:31:49.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:49.235802+0000) 2022-01-31T22:31:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:49 smithi171 conmon[41853]: debug 2022-01-31T22:31:49.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:49.576939+0000) 2022-01-31T22:31:49.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:49 smithi167 conmon[49112]: debug 2022-01-31T22:31:49.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:49.706136+0000) 2022-01-31T22:31:49.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:49 smithi167 conmon[54076]: debug 2022-01-31T22:31:49.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:49.873892+0000) 2022-01-31T22:31:49.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:49 smithi167 conmon[60316]: debug 2022-01-31T22:31:49.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:49.632145+0000) 2022-01-31T22:31:50.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:50 smithi171 conmon[46715]: debug 2022-01-31T22:31:50.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:50.373645+0000) 2022-01-31T22:31:50.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:50 smithi171 conmon[51620]: debug 2022-01-31T22:31:50.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:50.235986+0000) 2022-01-31T22:31:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:50 smithi171 conmon[41853]: debug 2022-01-31T22:31:50.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:50.577145+0000) 2022-01-31T22:31:50.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:50 smithi167 conmon[49112]: debug 2022-01-31T22:31:50.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:50.706343+0000) 2022-01-31T22:31:50.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:50 smithi167 conmon[54076]: debug 2022-01-31T22:31:50.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:50.874093+0000) 2022-01-31T22:31:50.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:50 smithi167 conmon[60316]: debug 2022-01-31T22:31:50.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:50.632350+0000) 2022-01-31T22:31:51.538 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[35325]: debug 2022-01-31T22:31:51.484+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216848 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:51.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[41853]: debug 2022-01-31T22:31:51.462+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.463575+0000) 2022-01-31T22:31:51.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[46715]: debug 2022-01-31T22:31:51.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.373823+0000) 2022-01-31T22:31:51.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[46715]: debug 2022-01-31T22:31:51.460+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.461671+0000) 2022-01-31T22:31:51.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[51620]: debug 2022-01-31T22:31:51.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.236146+0000) 2022-01-31T22:31:51.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[51620]: debug 2022-01-31T22:31:51.461+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.462281+0000) 2022-01-31T22:31:51.632 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:51 smithi167 conmon[49112]: debug 2022-01-31T22:31:51.461+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.463290+0000) 2022-01-31T22:31:51.632 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:51 smithi167 conmon[54076]: debug 2022-01-31T22:31:51.461+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.463331+0000) 2022-01-31T22:31:51.633 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:51 smithi167 conmon[60316]: debug 2022-01-31T22:31:51.461+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.463051+0000) 2022-01-31T22:31:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:51 smithi171 conmon[41853]: debug 2022-01-31T22:31:51.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.577333+0000) 2022-01-31T22:31:51.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:51 smithi167 conmon[60316]: debug 2022-01-31T22:31:51.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.632568+0000) 2022-01-31T22:31:51.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:51 smithi167 conmon[54076]: debug 2022-01-31T22:31:51.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.874254+0000) 2022-01-31T22:31:51.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:51 smithi167 conmon[49112]: debug 2022-01-31T22:31:51.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:51.706531+0000) 2022-01-31T22:31:52.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:52 smithi171 conmon[46715]: debug 2022-01-31T22:31:52.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:52.374008+0000) 2022-01-31T22:31:52.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:52 smithi171 conmon[51620]: debug 2022-01-31T22:31:52.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:52.236337+0000) 2022-01-31T22:31:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:52 smithi171 conmon[41853]: debug 2022-01-31T22:31:52.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:52.577490+0000) 2022-01-31T22:31:52.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:52 smithi167 conmon[60316]: debug 2022-01-31T22:31:52.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:52.632746+0000) 2022-01-31T22:31:52.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:52 smithi167 conmon[49112]: debug 2022-01-31T22:31:52.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:52.706689+0000) 2022-01-31T22:31:52.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:52 smithi167 conmon[54076]: debug 2022-01-31T22:31:52.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:52.874390+0000) 2022-01-31T22:31:53.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:53 smithi171 conmon[46715]: debug 2022-01-31T22:31:53.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:53.374161+0000) 2022-01-31T22:31:53.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:53 smithi171 conmon[51620]: debug 2022-01-31T22:31:53.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:53.236470+0000) 2022-01-31T22:31:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:53 smithi171 conmon[41853]: debug 2022-01-31T22:31:53.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:53.577705+0000) 2022-01-31T22:31:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:53 smithi167 conmon[49112]: debug 2022-01-31T22:31:53.705+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:53.706847+0000) 2022-01-31T22:31:53.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:53 smithi167 conmon[54076]: debug 2022-01-31T22:31:53.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:53.874585+0000) 2022-01-31T22:31:53.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:53 smithi167 conmon[60316]: debug 2022-01-31T22:31:53.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:53.632937+0000) 2022-01-31T22:31:54.236 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:53 smithi171 conmon[35325]: debug 2022-01-31T22:31:53.868+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:31:54.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:54 smithi171 conmon[46715]: debug 2022-01-31T22:31:54.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:54.374296+0000) 2022-01-31T22:31:54.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:54 smithi171 conmon[51620]: debug 2022-01-31T22:31:54.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:54.236648+0000) 2022-01-31T22:31:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:54 smithi171 conmon[41853]: debug 2022-01-31T22:31:54.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:54.577934+0000) 2022-01-31T22:31:54.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:54 smithi167 conmon[60316]: debug 2022-01-31T22:31:54.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:54.633105+0000) 2022-01-31T22:31:54.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:54 smithi167 conmon[49112]: debug 2022-01-31T22:31:54.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:54.707011+0000) 2022-01-31T22:31:54.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:54 smithi167 conmon[54076]: debug 2022-01-31T22:31:54.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:54.874732+0000) 2022-01-31T22:31:55.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:55 smithi171 conmon[46715]: debug 2022-01-31T22:31:55.372+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:55.374471+0000) 2022-01-31T22:31:55.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:55 smithi171 conmon[51620]: debug 2022-01-31T22:31:55.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:55.236778+0000) 2022-01-31T22:31:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:55 smithi171 conmon[41853]: debug 2022-01-31T22:31:55.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:55.578081+0000) 2022-01-31T22:31:55.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:55 smithi167 conmon[60316]: debug 2022-01-31T22:31:55.631+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:55.633285+0000) 2022-01-31T22:31:55.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:55 smithi167 conmon[49112]: debug 2022-01-31T22:31:55.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:55.707207+0000) 2022-01-31T22:31:55.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:55 smithi167 conmon[54076]: debug 2022-01-31T22:31:55.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:55.874892+0000) 2022-01-31T22:31:56.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[46715]: debug 2022-01-31T22:31:56.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.374655+0000) 2022-01-31T22:31:56.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[51620]: debug 2022-01-31T22:31:56.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.236944+0000) 2022-01-31T22:31:56.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[51620]: debug 2022-01-31T22:31:56.484+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.486460+0000) 2022-01-31T22:31:56.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[41853]: debug 2022-01-31T22:31:56.485+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.487139+0000) 2022-01-31T22:31:56.633 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:56 smithi167 conmon[49112]: debug 2022-01-31T22:31:56.485+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.487424+0000) 2022-01-31T22:31:56.633 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:56 smithi167 conmon[54076]: debug 2022-01-31T22:31:56.486+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.487844+0000) 2022-01-31T22:31:56.634 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:56 smithi167 conmon[60316]: debug 2022-01-31T22:31:56.487+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.488687+0000) 2022-01-31T22:31:56.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[35325]: debug 2022-01-31T22:31:56.508+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 216961 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:31:56.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[46715]: debug 2022-01-31T22:31:56.485+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.487446+0000) 2022-01-31T22:31:56.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:56 smithi171 conmon[41853]: debug 2022-01-31T22:31:56.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.578286+0000) 2022-01-31T22:31:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:56 smithi167 conmon[49112]: debug 2022-01-31T22:31:56.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.707403+0000) 2022-01-31T22:31:56.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:56 smithi167 conmon[54076]: debug 2022-01-31T22:31:56.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.875089+0000) 2022-01-31T22:31:56.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:56 smithi167 conmon[60316]: debug 2022-01-31T22:31:56.632+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:56.633497+0000) 2022-01-31T22:31:57.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:57 smithi171 conmon[46715]: debug 2022-01-31T22:31:57.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:57.374844+0000) 2022-01-31T22:31:57.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:57 smithi171 conmon[51620]: debug 2022-01-31T22:31:57.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:57.237104+0000) 2022-01-31T22:31:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:57 smithi171 conmon[41853]: debug 2022-01-31T22:31:57.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:57.578433+0000) 2022-01-31T22:31:57.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:57 smithi167 conmon[49112]: debug 2022-01-31T22:31:57.706+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:57.707555+0000) 2022-01-31T22:31:57.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:57 smithi167 conmon[54076]: debug 2022-01-31T22:31:57.873+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:57.875229+0000) 2022-01-31T22:31:57.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:57 smithi167 conmon[60316]: debug 2022-01-31T22:31:57.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:57.633681+0000) 2022-01-31T22:31:58.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:58 smithi171 conmon[46715]: debug 2022-01-31T22:31:58.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:58.374987+0000) 2022-01-31T22:31:58.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:58 smithi171 conmon[51620]: debug 2022-01-31T22:31:58.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:58.237248+0000) 2022-01-31T22:31:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:58 smithi171 conmon[41853]: debug 2022-01-31T22:31:58.576+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:58.578632+0000) 2022-01-31T22:31:58.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:58 smithi167 conmon[49112]: debug 2022-01-31T22:31:58.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:58.707743+0000) 2022-01-31T22:31:58.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:58 smithi167 conmon[54076]: debug 2022-01-31T22:31:58.874+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:58.875387+0000) 2022-01-31T22:31:58.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:58 smithi167 conmon[60316]: debug 2022-01-31T22:31:58.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:58.633862+0000) 2022-01-31T22:31:59.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:31:59 smithi171 conmon[46715]: debug 2022-01-31T22:31:59.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:59.375169+0000) 2022-01-31T22:31:59.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:31:59 smithi171 conmon[51620]: debug 2022-01-31T22:31:59.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:59.237432+0000) 2022-01-31T22:31:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:31:59 smithi171 conmon[41853]: debug 2022-01-31T22:31:59.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:59.578832+0000) 2022-01-31T22:31:59.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:31:59 smithi167 conmon[49112]: debug 2022-01-31T22:31:59.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:59.708010+0000) 2022-01-31T22:31:59.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:31:59 smithi167 conmon[54076]: debug 2022-01-31T22:31:59.875+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:59.875607+0000) 2022-01-31T22:31:59.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:31:59 smithi167 conmon[60316]: debug 2022-01-31T22:31:59.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:31:59.634019+0000) 2022-01-31T22:32:00.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:00 smithi171 conmon[46715]: debug 2022-01-31T22:32:00.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:00.375357+0000) 2022-01-31T22:32:00.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:00 smithi171 conmon[51620]: debug 2022-01-31T22:32:00.235+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:00.237633+0000) 2022-01-31T22:32:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:00 smithi171 conmon[41853]: debug 2022-01-31T22:32:00.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:00.578933+0000) 2022-01-31T22:32:00.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:00 smithi167 conmon[49112]: debug 2022-01-31T22:32:00.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:00.708197+0000) 2022-01-31T22:32:00.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:00 smithi167 conmon[54076]: debug 2022-01-31T22:32:00.875+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:00.875810+0000) 2022-01-31T22:32:00.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:00 smithi167 conmon[60316]: debug 2022-01-31T22:32:00.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:00.634195+0000) 2022-01-31T22:32:01.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[46715]: debug 2022-01-31T22:32:01.373+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.375549+0000) 2022-01-31T22:32:01.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[51620]: debug 2022-01-31T22:32:01.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.237830+0000) 2022-01-31T22:32:01.634 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:01 smithi167 conmon[49112]: debug 2022-01-31T22:32:01.512+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.512607+0000) 2022-01-31T22:32:01.634 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:01 smithi167 conmon[54076]: debug 2022-01-31T22:32:01.519+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.519895+0000) 2022-01-31T22:32:01.635 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:01 smithi167 conmon[60316]: debug 2022-01-31T22:32:01.511+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.512420+0000) 2022-01-31T22:32:01.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[46715]: debug 2022-01-31T22:32:01.510+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.512574+0000) 2022-01-31T22:32:01.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[51620]: debug 2022-01-31T22:32:01.509+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.511712+0000) 2022-01-31T22:32:01.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[35325]: debug 2022-01-31T22:32:01.548+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217071 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:01.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[41853]: debug 2022-01-31T22:32:01.510+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.512323+0000) 2022-01-31T22:32:01.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:01 smithi171 conmon[41853]: debug 2022-01-31T22:32:01.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.579036+0000) 2022-01-31T22:32:01.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:01 smithi167 conmon[49112]: debug 2022-01-31T22:32:01.707+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.708359+0000) 2022-01-31T22:32:01.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:01 smithi167 conmon[54076]: debug 2022-01-31T22:32:01.875+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.875997+0000) 2022-01-31T22:32:01.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:01 smithi167 conmon[60316]: debug 2022-01-31T22:32:01.633+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:01.634399+0000) 2022-01-31T22:32:02.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:02 smithi171 conmon[46715]: debug 2022-01-31T22:32:02.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:02.375688+0000) 2022-01-31T22:32:02.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:02 smithi171 conmon[51620]: debug 2022-01-31T22:32:02.236+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:02.237960+0000) 2022-01-31T22:32:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:02 smithi171 conmon[41853]: debug 2022-01-31T22:32:02.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:02.579181+0000) 2022-01-31T22:32:02.886 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:02 smithi167 conmon[49112]: debug 2022-01-31T22:32:02.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:02.708531+0000) 2022-01-31T22:32:02.887 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:02 smithi167 conmon[54076]: debug 2022-01-31T22:32:02.875+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:02.876173+0000) 2022-01-31T22:32:02.888 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:02 smithi167 conmon[60316]: debug 2022-01-31T22:32:02.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:02.634552+0000) 2022-01-31T22:32:03.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:03 smithi171 conmon[46715]: debug 2022-01-31T22:32:03.374+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:03.375836+0000) 2022-01-31T22:32:03.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:03 smithi171 conmon[51620]: debug 2022-01-31T22:32:03.237+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:03.238126+0000) 2022-01-31T22:32:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:03 smithi171 conmon[41853]: debug 2022-01-31T22:32:03.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:03.579393+0000) 2022-01-31T22:32:03.891 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:03 smithi167 conmon[49112]: debug 2022-01-31T22:32:03.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:03.708692+0000) 2022-01-31T22:32:03.892 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:03 smithi167 conmon[60316]: debug 2022-01-31T22:32:03.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:03.634735+0000) 2022-01-31T22:32:03.893 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:03 smithi167 conmon[54076]: debug 2022-01-31T22:32:03.875+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:03.876361+0000) 2022-01-31T22:32:04.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:04 smithi171 conmon[46715]: debug 2022-01-31T22:32:04.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:04.376033+0000) 2022-01-31T22:32:04.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:04 smithi171 conmon[51620]: debug 2022-01-31T22:32:04.237+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:04.238326+0000) 2022-01-31T22:32:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:04 smithi171 conmon[41853]: debug 2022-01-31T22:32:04.577+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:04.579522+0000) 2022-01-31T22:32:04.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:04 smithi167 conmon[49112]: debug 2022-01-31T22:32:04.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:04.708921+0000) 2022-01-31T22:32:04.895 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:04 smithi167 conmon[54076]: debug 2022-01-31T22:32:04.876+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:04.876497+0000) 2022-01-31T22:32:04.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:04 smithi167 conmon[60316]: debug 2022-01-31T22:32:04.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:04.634930+0000) 2022-01-31T22:32:05.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:05 smithi171 conmon[46715]: debug 2022-01-31T22:32:05.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:05.376220+0000) 2022-01-31T22:32:05.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:05 smithi171 conmon[51620]: debug 2022-01-31T22:32:05.237+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:05.238476+0000) 2022-01-31T22:32:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:05 smithi171 conmon[41853]: debug 2022-01-31T22:32:05.578+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:05.579677+0000) 2022-01-31T22:32:05.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:05 smithi167 conmon[49112]: debug 2022-01-31T22:32:05.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:05.709111+0000) 2022-01-31T22:32:05.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:05 smithi167 conmon[54076]: debug 2022-01-31T22:32:05.876+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:05.876678+0000) 2022-01-31T22:32:05.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:05 smithi167 conmon[60316]: debug 2022-01-31T22:32:05.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:05.635141+0000) 2022-01-31T22:32:06.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[46715]: debug 2022-01-31T22:32:06.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.376403+0000) 2022-01-31T22:32:06.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[51620]: debug 2022-01-31T22:32:06.237+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.238642+0000) 2022-01-31T22:32:06.635 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:06 smithi167 conmon[49112]: debug 2022-01-31T22:32:06.551+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.551964+0000) 2022-01-31T22:32:06.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:06 smithi167 conmon[54076]: debug 2022-01-31T22:32:06.552+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.553301+0000) 2022-01-31T22:32:06.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:06 smithi167 conmon[60316]: debug 2022-01-31T22:32:06.552+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.552991+0000) 2022-01-31T22:32:06.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[46715]: debug 2022-01-31T22:32:06.551+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.552092+0000) 2022-01-31T22:32:06.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[51620]: debug 2022-01-31T22:32:06.552+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.552952+0000) 2022-01-31T22:32:06.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[35325]: debug 2022-01-31T22:32:06.573+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217185 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:06.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[41853]: debug 2022-01-31T22:32:06.551+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.552377+0000) 2022-01-31T22:32:06.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:06 smithi171 conmon[41853]: debug 2022-01-31T22:32:06.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.579779+0000) 2022-01-31T22:32:06.898 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:06 smithi167 conmon[49112]: debug 2022-01-31T22:32:06.708+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.709311+0000) 2022-01-31T22:32:06.899 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:06 smithi167 conmon[54076]: debug 2022-01-31T22:32:06.876+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.876882+0000) 2022-01-31T22:32:06.900 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:06 smithi167 conmon[60316]: debug 2022-01-31T22:32:06.634+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:06.635320+0000) 2022-01-31T22:32:07.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:07 smithi171 conmon[46715]: debug 2022-01-31T22:32:07.375+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:07.376628+0000) 2022-01-31T22:32:07.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:07 smithi171 conmon[51620]: debug 2022-01-31T22:32:07.238+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:07.238788+0000) 2022-01-31T22:32:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:07 smithi171 conmon[41853]: debug 2022-01-31T22:32:07.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:07.579922+0000) 2022-01-31T22:32:07.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:07 smithi167 conmon[49112]: debug 2022-01-31T22:32:07.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:07.709482+0000) 2022-01-31T22:32:07.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:07 smithi167 conmon[54076]: debug 2022-01-31T22:32:07.876+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:07.877059+0000) 2022-01-31T22:32:07.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:07 smithi167 conmon[60316]: debug 2022-01-31T22:32:07.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:07.635497+0000) 2022-01-31T22:32:08.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:08 smithi171 conmon[46715]: debug 2022-01-31T22:32:08.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:08.376786+0000) 2022-01-31T22:32:08.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:08 smithi171 conmon[51620]: debug 2022-01-31T22:32:08.238+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:08.238918+0000) 2022-01-31T22:32:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:08 smithi171 conmon[41853]: debug 2022-01-31T22:32:08.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:08.580109+0000) 2022-01-31T22:32:08.906 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:08 smithi167 conmon[49112]: debug 2022-01-31T22:32:08.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:08.709678+0000) 2022-01-31T22:32:08.907 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:08 smithi167 conmon[54076]: debug 2022-01-31T22:32:08.876+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:08.877230+0000) 2022-01-31T22:32:08.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:08 smithi167 conmon[60316]: debug 2022-01-31T22:32:08.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:08.635712+0000) 2022-01-31T22:32:09.238 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:08 smithi171 conmon[35325]: debug 2022-01-31T22:32:08.868+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:32:09.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:09 smithi171 conmon[46715]: debug 2022-01-31T22:32:09.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:09.376929+0000) 2022-01-31T22:32:09.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:09 smithi171 conmon[51620]: debug 2022-01-31T22:32:09.238+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:09.239069+0000) 2022-01-31T22:32:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:09 smithi171 conmon[41853]: debug 2022-01-31T22:32:09.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:09.580313+0000) 2022-01-31T22:32:09.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:09 smithi167 conmon[49112]: debug 2022-01-31T22:32:09.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:09.709844+0000) 2022-01-31T22:32:09.911 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:09 smithi167 conmon[54076]: debug 2022-01-31T22:32:09.877+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:09.877414+0000) 2022-01-31T22:32:09.911 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:09 smithi167 conmon[60316]: debug 2022-01-31T22:32:09.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:09.635912+0000) 2022-01-31T22:32:10.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:10 smithi171 conmon[46715]: debug 2022-01-31T22:32:10.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:10.377058+0000) 2022-01-31T22:32:10.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:10 smithi171 conmon[51620]: debug 2022-01-31T22:32:10.238+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:10.239241+0000) 2022-01-31T22:32:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:10 smithi171 conmon[41853]: debug 2022-01-31T22:32:10.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:10.580515+0000) 2022-01-31T22:32:10.913 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:10 smithi167 conmon[54076]: debug 2022-01-31T22:32:10.877+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:10.877657+0000) 2022-01-31T22:32:10.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:10 smithi167 conmon[60316]: debug 2022-01-31T22:32:10.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:10.636113+0000) 2022-01-31T22:32:10.915 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:10 smithi167 conmon[49112]: debug 2022-01-31T22:32:10.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:10.710006+0000) 2022-01-31T22:32:11.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[46715]: debug 2022-01-31T22:32:11.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.377224+0000) 2022-01-31T22:32:11.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[51620]: debug 2022-01-31T22:32:11.238+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.239413+0000) 2022-01-31T22:32:11.635 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:11 smithi167 conmon[49112]: debug 2022-01-31T22:32:11.576+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.577007+0000) 2022-01-31T22:32:11.636 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:11 smithi167 conmon[54076]: debug 2022-01-31T22:32:11.577+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.578263+0000) 2022-01-31T22:32:11.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:11 smithi167 conmon[60316]: debug 2022-01-31T22:32:11.576+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.577342+0000) 2022-01-31T22:32:11.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[35325]: debug 2022-01-31T22:32:11.598+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217294 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:11.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[46715]: debug 2022-01-31T22:32:11.577+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.577916+0000) 2022-01-31T22:32:11.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[51620]: debug 2022-01-31T22:32:11.575+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.576681+0000) 2022-01-31T22:32:11.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[41853]: debug 2022-01-31T22:32:11.576+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.576993+0000) 2022-01-31T22:32:11.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:11 smithi171 conmon[41853]: debug 2022-01-31T22:32:11.579+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.580634+0000) 2022-01-31T22:32:11.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:11 smithi167 conmon[49112]: debug 2022-01-31T22:32:11.709+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.710180+0000) 2022-01-31T22:32:11.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:11 smithi167 conmon[60316]: debug 2022-01-31T22:32:11.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.636340+0000) 2022-01-31T22:32:11.915 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:11 smithi167 conmon[54076]: debug 2022-01-31T22:32:11.877+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:11.877843+0000) 2022-01-31T22:32:12.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:12 smithi171 conmon[46715]: debug 2022-01-31T22:32:12.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:12.377381+0000) 2022-01-31T22:32:12.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:12 smithi171 conmon[51620]: debug 2022-01-31T22:32:12.238+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:12.239605+0000) 2022-01-31T22:32:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:12 smithi171 conmon[41853]: debug 2022-01-31T22:32:12.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:12.580786+0000) 2022-01-31T22:32:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:12 smithi167 conmon[49112]: debug 2022-01-31T22:32:12.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:12.710345+0000) 2022-01-31T22:32:12.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:12 smithi167 conmon[54076]: debug 2022-01-31T22:32:12.877+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:12.877988+0000) 2022-01-31T22:32:12.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:12 smithi167 conmon[60316]: debug 2022-01-31T22:32:12.635+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:12.636467+0000) 2022-01-31T22:32:13.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:13 smithi171 conmon[46715]: debug 2022-01-31T22:32:13.376+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:13.377537+0000) 2022-01-31T22:32:13.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:13 smithi171 conmon[51620]: debug 2022-01-31T22:32:13.239+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:13.239746+0000) 2022-01-31T22:32:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:13 smithi171 conmon[41853]: debug 2022-01-31T22:32:13.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:13.580959+0000) 2022-01-31T22:32:13.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:13 smithi167 conmon[49112]: debug 2022-01-31T22:32:13.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:13.710483+0000) 2022-01-31T22:32:13.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:13 smithi167 conmon[54076]: debug 2022-01-31T22:32:13.877+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:13.878145+0000) 2022-01-31T22:32:13.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:13 smithi167 conmon[60316]: debug 2022-01-31T22:32:13.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:13.636633+0000) 2022-01-31T22:32:14.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:14 smithi171 conmon[46715]: debug 2022-01-31T22:32:14.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:14.377736+0000) 2022-01-31T22:32:14.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:14 smithi171 conmon[51620]: debug 2022-01-31T22:32:14.239+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:14.239880+0000) 2022-01-31T22:32:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:14 smithi171 conmon[41853]: debug 2022-01-31T22:32:14.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:14.581121+0000) 2022-01-31T22:32:14.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:14 smithi167 conmon[49112]: debug 2022-01-31T22:32:14.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:14.710703+0000) 2022-01-31T22:32:14.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:14 smithi167 conmon[54076]: debug 2022-01-31T22:32:14.877+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:14.878333+0000) 2022-01-31T22:32:14.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:14 smithi167 conmon[60316]: debug 2022-01-31T22:32:14.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:14.636832+0000) 2022-01-31T22:32:15.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:15 smithi171 conmon[46715]: debug 2022-01-31T22:32:15.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:15.377937+0000) 2022-01-31T22:32:15.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:15 smithi171 conmon[51620]: debug 2022-01-31T22:32:15.239+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:15.240061+0000) 2022-01-31T22:32:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:15 smithi171 conmon[41853]: debug 2022-01-31T22:32:15.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:15.581246+0000) 2022-01-31T22:32:15.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:15 smithi167 conmon[49112]: debug 2022-01-31T22:32:15.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:15.711007+0000) 2022-01-31T22:32:15.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:15 smithi167 conmon[54076]: debug 2022-01-31T22:32:15.878+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:15.878456+0000) 2022-01-31T22:32:15.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:15 smithi167 conmon[60316]: debug 2022-01-31T22:32:15.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:15.637036+0000) 2022-01-31T22:32:16.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[46715]: debug 2022-01-31T22:32:16.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.378120+0000) 2022-01-31T22:32:16.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[51620]: debug 2022-01-31T22:32:16.239+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.240231+0000) 2022-01-31T22:32:16.636 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:16 smithi167 conmon[49112]: debug 2022-01-31T22:32:16.602+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.602728+0000) 2022-01-31T22:32:16.637 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:16 smithi167 conmon[54076]: debug 2022-01-31T22:32:16.602+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.603279+0000) 2022-01-31T22:32:16.637 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:16 smithi167 conmon[60316]: debug 2022-01-31T22:32:16.602+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.602988+0000) 2022-01-31T22:32:16.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[35325]: debug 2022-01-31T22:32:16.624+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217407 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:16.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[46715]: debug 2022-01-31T22:32:16.602+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.603425+0000) 2022-01-31T22:32:16.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[51620]: debug 2022-01-31T22:32:16.601+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.602674+0000) 2022-01-31T22:32:16.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[41853]: debug 2022-01-31T22:32:16.580+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.581390+0000) 2022-01-31T22:32:16.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:16 smithi171 conmon[41853]: debug 2022-01-31T22:32:16.601+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.601853+0000) 2022-01-31T22:32:16.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:16 smithi167 conmon[54076]: debug 2022-01-31T22:32:16.878+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.878670+0000) 2022-01-31T22:32:16.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:16 smithi167 conmon[49112]: debug 2022-01-31T22:32:16.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.711156+0000) 2022-01-31T22:32:16.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:16 smithi167 conmon[60316]: debug 2022-01-31T22:32:16.636+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:16.637193+0000) 2022-01-31T22:32:17.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:17 smithi171 conmon[46715]: debug 2022-01-31T22:32:17.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:17.378249+0000) 2022-01-31T22:32:17.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:17 smithi171 conmon[51620]: debug 2022-01-31T22:32:17.239+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:17.240425+0000) 2022-01-31T22:32:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:17 smithi171 conmon[41853]: debug 2022-01-31T22:32:17.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:17.581582+0000) 2022-01-31T22:32:17.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:17 smithi167 conmon[49112]: debug 2022-01-31T22:32:17.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:17.711236+0000) 2022-01-31T22:32:17.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:17 smithi167 conmon[54076]: debug 2022-01-31T22:32:17.878+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:17.878782+0000) 2022-01-31T22:32:17.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:17 smithi167 conmon[60316]: debug 2022-01-31T22:32:17.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:17.637340+0000) 2022-01-31T22:32:18.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:18 smithi171 conmon[46715]: debug 2022-01-31T22:32:18.377+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:18.378396+0000) 2022-01-31T22:32:18.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:18 smithi171 conmon[51620]: debug 2022-01-31T22:32:18.240+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:18.240594+0000) 2022-01-31T22:32:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:18 smithi171 conmon[41853]: debug 2022-01-31T22:32:18.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:18.581778+0000) 2022-01-31T22:32:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:18 smithi167 conmon[49112]: debug 2022-01-31T22:32:18.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:18.711413+0000) 2022-01-31T22:32:18.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:18 smithi167 conmon[54076]: debug 2022-01-31T22:32:18.878+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:18.878973+0000) 2022-01-31T22:32:18.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:18 smithi167 conmon[60316]: debug 2022-01-31T22:32:18.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:18.637566+0000) 2022-01-31T22:32:19.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:19 smithi171 conmon[46715]: debug 2022-01-31T22:32:19.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:19.378583+0000) 2022-01-31T22:32:19.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:19 smithi171 conmon[51620]: debug 2022-01-31T22:32:19.240+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:19.240792+0000) 2022-01-31T22:32:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:19 smithi171 conmon[41853]: debug 2022-01-31T22:32:19.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:19.581897+0000) 2022-01-31T22:32:19.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:19 smithi167 conmon[49112]: debug 2022-01-31T22:32:19.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:19.711537+0000) 2022-01-31T22:32:19.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:19 smithi167 conmon[54076]: debug 2022-01-31T22:32:19.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:19.879157+0000) 2022-01-31T22:32:19.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:19 smithi167 conmon[60316]: debug 2022-01-31T22:32:19.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:19.637683+0000) 2022-01-31T22:32:20.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:20 smithi171 conmon[46715]: debug 2022-01-31T22:32:20.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:20.378784+0000) 2022-01-31T22:32:20.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:20 smithi171 conmon[51620]: debug 2022-01-31T22:32:20.240+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:20.240989+0000) 2022-01-31T22:32:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:20 smithi171 conmon[41853]: debug 2022-01-31T22:32:20.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:20.582058+0000) 2022-01-31T22:32:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:20 smithi167 conmon[49112]: debug 2022-01-31T22:32:20.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:20.711747+0000) 2022-01-31T22:32:20.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:20 smithi167 conmon[54076]: debug 2022-01-31T22:32:20.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:20.879361+0000) 2022-01-31T22:32:20.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:20 smithi167 conmon[60316]: debug 2022-01-31T22:32:20.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:20.637847+0000) 2022-01-31T22:32:21.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[46715]: debug 2022-01-31T22:32:21.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.378938+0000) 2022-01-31T22:32:21.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[51620]: debug 2022-01-31T22:32:21.240+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.241363+0000) 2022-01-31T22:32:21.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[46715]: debug 2022-01-31T22:32:21.627+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.628385+0000) 2022-01-31T22:32:21.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[35325]: debug 2022-01-31T22:32:21.650+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217516 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:21.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[51620]: debug 2022-01-31T22:32:21.626+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.627413+0000) 2022-01-31T22:32:21.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[41853]: debug 2022-01-31T22:32:21.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.582223+0000) 2022-01-31T22:32:21.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:21 smithi171 conmon[41853]: debug 2022-01-31T22:32:21.627+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.628371+0000) 2022-01-31T22:32:21.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:21 smithi167 conmon[54076]: debug 2022-01-31T22:32:21.628+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.628585+0000) 2022-01-31T22:32:21.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:21 smithi167 conmon[49112]: debug 2022-01-31T22:32:21.627+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.627656+0000) 2022-01-31T22:32:21.880 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:21 smithi167 conmon[49112]: debug 2022-01-31T22:32:21.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.711908+0000) 2022-01-31T22:32:21.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:21 smithi167 conmon[60316]: debug 2022-01-31T22:32:21.627+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.628402+0000) 2022-01-31T22:32:21.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:21 smithi167 conmon[60316]: debug 2022-01-31T22:32:21.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.638057+0000) 2022-01-31T22:32:22.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:21 smithi167 conmon[54076]: debug 2022-01-31T22:32:21.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:21.879546+0000) 2022-01-31T22:32:22.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:22 smithi171 conmon[46715]: debug 2022-01-31T22:32:22.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:22.379074+0000) 2022-01-31T22:32:22.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:22 smithi171 conmon[51620]: debug 2022-01-31T22:32:22.240+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:22.241466+0000) 2022-01-31T22:32:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:22 smithi171 conmon[41853]: debug 2022-01-31T22:32:22.581+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:22.582370+0000) 2022-01-31T22:32:22.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:22 smithi167 conmon[49112]: debug 2022-01-31T22:32:22.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:22.712082+0000) 2022-01-31T22:32:22.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:22 smithi167 conmon[60316]: debug 2022-01-31T22:32:22.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:22.638234+0000) 2022-01-31T22:32:22.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:22 smithi167 conmon[54076]: debug 2022-01-31T22:32:22.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:22.879656+0000) 2022-01-31T22:32:23.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:23 smithi171 conmon[46715]: debug 2022-01-31T22:32:23.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:23.379217+0000) 2022-01-31T22:32:23.544 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:23 smithi171 conmon[51620]: debug 2022-01-31T22:32:23.241+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:23.241575+0000) 2022-01-31T22:32:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:23 smithi171 conmon[41853]: debug 2022-01-31T22:32:23.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:23.582552+0000) 2022-01-31T22:32:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:23 smithi167 conmon[49112]: debug 2022-01-31T22:32:23.710+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:23.712252+0000) 2022-01-31T22:32:23.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:23 smithi167 conmon[54076]: debug 2022-01-31T22:32:23.878+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:23.879833+0000) 2022-01-31T22:32:23.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:23 smithi167 conmon[60316]: debug 2022-01-31T22:32:23.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:23.638433+0000) 2022-01-31T22:32:24.241 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:23 smithi171 conmon[35325]: debug 2022-01-31T22:32:23.869+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:32:24.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:24 smithi171 conmon[46715]: debug 2022-01-31T22:32:24.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:24.379456+0000) 2022-01-31T22:32:24.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:24 smithi171 conmon[51620]: debug 2022-01-31T22:32:24.241+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:24.241718+0000) 2022-01-31T22:32:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:24 smithi171 conmon[41853]: debug 2022-01-31T22:32:24.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:24.582778+0000) 2022-01-31T22:32:24.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:24 smithi167 conmon[49112]: debug 2022-01-31T22:32:24.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:24.712413+0000) 2022-01-31T22:32:24.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:24 smithi167 conmon[54076]: debug 2022-01-31T22:32:24.878+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:24.879984+0000) 2022-01-31T22:32:24.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:24 smithi167 conmon[60316]: debug 2022-01-31T22:32:24.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:24.638612+0000) 2022-01-31T22:32:25.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:25 smithi171 conmon[46715]: debug 2022-01-31T22:32:25.378+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:25.379681+0000) 2022-01-31T22:32:25.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:25 smithi171 conmon[51620]: debug 2022-01-31T22:32:25.241+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:25.241841+0000) 2022-01-31T22:32:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:25 smithi171 conmon[41853]: debug 2022-01-31T22:32:25.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:25.582969+0000) 2022-01-31T22:32:25.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:25 smithi167 conmon[49112]: debug 2022-01-31T22:32:25.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:25.712575+0000) 2022-01-31T22:32:25.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:25 smithi167 conmon[54076]: debug 2022-01-31T22:32:25.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:25.880175+0000) 2022-01-31T22:32:25.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:25 smithi167 conmon[60316]: debug 2022-01-31T22:32:25.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:25.638716+0000) 2022-01-31T22:32:26.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[46715]: debug 2022-01-31T22:32:26.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.379827+0000) 2022-01-31T22:32:26.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[51620]: debug 2022-01-31T22:32:26.241+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.241982+0000) 2022-01-31T22:32:26.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[35325]: debug 2022-01-31T22:32:26.675+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217629 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:26.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[46715]: debug 2022-01-31T22:32:26.653+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.654223+0000) 2022-01-31T22:32:26.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[51620]: debug 2022-01-31T22:32:26.652+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.652852+0000) 2022-01-31T22:32:26.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[41853]: debug 2022-01-31T22:32:26.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.583089+0000) 2022-01-31T22:32:26.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:26 smithi171 conmon[41853]: debug 2022-01-31T22:32:26.653+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.654507+0000) 2022-01-31T22:32:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:26 smithi167 conmon[49112]: debug 2022-01-31T22:32:26.652+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.653905+0000) 2022-01-31T22:32:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:26 smithi167 conmon[49112]: debug 2022-01-31T22:32:26.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.712712+0000) 2022-01-31T22:32:26.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:26 smithi167 conmon[54076]: debug 2022-01-31T22:32:26.652+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.654074+0000) 2022-01-31T22:32:26.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:26 smithi167 conmon[54076]: debug 2022-01-31T22:32:26.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.880373+0000) 2022-01-31T22:32:26.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:26 smithi167 conmon[60316]: debug 2022-01-31T22:32:26.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.638884+0000) 2022-01-31T22:32:26.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:26 smithi167 conmon[60316]: debug 2022-01-31T22:32:26.653+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:26.654605+0000) 2022-01-31T22:32:27.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:27 smithi171 conmon[46715]: debug 2022-01-31T22:32:27.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:27.379984+0000) 2022-01-31T22:32:27.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:27 smithi171 conmon[51620]: debug 2022-01-31T22:32:27.241+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:27.242130+0000) 2022-01-31T22:32:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:27 smithi171 conmon[41853]: debug 2022-01-31T22:32:27.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:27.583253+0000) 2022-01-31T22:32:27.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:27 smithi167 conmon[49112]: debug 2022-01-31T22:32:27.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:27.712814+0000) 2022-01-31T22:32:27.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:27 smithi167 conmon[54076]: debug 2022-01-31T22:32:27.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:27.880493+0000) 2022-01-31T22:32:27.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:27 smithi167 conmon[60316]: debug 2022-01-31T22:32:27.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:27.639037+0000) 2022-01-31T22:32:28.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:28 smithi171 conmon[46715]: debug 2022-01-31T22:32:28.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:28.380150+0000) 2022-01-31T22:32:28.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:28 smithi171 conmon[51620]: debug 2022-01-31T22:32:28.242+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:28.242323+0000) 2022-01-31T22:32:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:28 smithi171 conmon[41853]: debug 2022-01-31T22:32:28.582+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:28.583407+0000) 2022-01-31T22:32:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:28 smithi167 conmon[49112]: debug 2022-01-31T22:32:28.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:28.712984+0000) 2022-01-31T22:32:28.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:28 smithi167 conmon[60316]: debug 2022-01-31T22:32:28.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:28.639093+0000) 2022-01-31T22:32:28.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:28 smithi167 conmon[54076]: debug 2022-01-31T22:32:28.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:28.880689+0000) 2022-01-31T22:32:29.352 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:29 smithi171 conmon[51620]: debug 2022-01-31T22:32:29.241+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:29.242467+0000) 2022-01-31T22:32:29.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:29 smithi171 conmon[46715]: debug 2022-01-31T22:32:29.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:29.380299+0000) 2022-01-31T22:32:29.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:29 smithi171 conmon[41853]: debug 2022-01-31T22:32:29.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:29.583595+0000) 2022-01-31T22:32:29.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:29 smithi167 conmon[49112]: debug 2022-01-31T22:32:29.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:29.713137+0000) 2022-01-31T22:32:29.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:29 smithi167 conmon[54076]: debug 2022-01-31T22:32:29.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:29.880898+0000) 2022-01-31T22:32:29.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:29 smithi167 conmon[60316]: debug 2022-01-31T22:32:29.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:29.639265+0000) 2022-01-31T22:32:30.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:30 smithi171 conmon[46715]: debug 2022-01-31T22:32:30.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:30.380395+0000) 2022-01-31T22:32:30.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:30 smithi171 conmon[51620]: debug 2022-01-31T22:32:30.242+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:30.242650+0000) 2022-01-31T22:32:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:30 smithi171 conmon[41853]: debug 2022-01-31T22:32:30.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:30.583787+0000) 2022-01-31T22:32:30.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:30 smithi167 conmon[49112]: debug 2022-01-31T22:32:30.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:30.713316+0000) 2022-01-31T22:32:30.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:30 smithi167 conmon[54076]: debug 2022-01-31T22:32:30.880+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:30.881119+0000) 2022-01-31T22:32:30.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:30 smithi167 conmon[60316]: debug 2022-01-31T22:32:30.637+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:30.639480+0000) 2022-01-31T22:32:31.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[46715]: debug 2022-01-31T22:32:31.379+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.380606+0000) 2022-01-31T22:32:31.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[51620]: debug 2022-01-31T22:32:31.242+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.242878+0000) 2022-01-31T22:32:31.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[35325]: debug 2022-01-31T22:32:31.700+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217738 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:31.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[46715]: debug 2022-01-31T22:32:31.678+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.679174+0000) 2022-01-31T22:32:31.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[51620]: debug 2022-01-31T22:32:31.677+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.677894+0000) 2022-01-31T22:32:31.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[41853]: debug 2022-01-31T22:32:31.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.583915+0000) 2022-01-31T22:32:31.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:31 smithi171 conmon[41853]: debug 2022-01-31T22:32:31.677+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.678406+0000) 2022-01-31T22:32:31.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:31 smithi167 conmon[54076]: debug 2022-01-31T22:32:31.677+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.678754+0000) 2022-01-31T22:32:31.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:31 smithi167 conmon[54076]: debug 2022-01-31T22:32:31.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.881322+0000) 2022-01-31T22:32:31.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:31 smithi167 conmon[60316]: debug 2022-01-31T22:32:31.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.639618+0000) 2022-01-31T22:32:31.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:31 smithi167 conmon[60316]: debug 2022-01-31T22:32:31.678+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.679506+0000) 2022-01-31T22:32:31.921 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:31 smithi167 conmon[49112]: debug 2022-01-31T22:32:31.676+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.678247+0000) 2022-01-31T22:32:31.921 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:31 smithi167 conmon[49112]: debug 2022-01-31T22:32:31.711+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:31.713450+0000) 2022-01-31T22:32:32.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:32 smithi171 conmon[46715]: debug 2022-01-31T22:32:32.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:32.380816+0000) 2022-01-31T22:32:32.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:32 smithi171 conmon[51620]: debug 2022-01-31T22:32:32.242+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:32.243050+0000) 2022-01-31T22:32:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:32 smithi171 conmon[41853]: debug 2022-01-31T22:32:32.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:32.584086+0000) 2022-01-31T22:32:32.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:32 smithi167 conmon[49112]: debug 2022-01-31T22:32:32.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:32.713586+0000) 2022-01-31T22:32:32.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:32 smithi167 conmon[54076]: debug 2022-01-31T22:32:32.879+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:32.881476+0000) 2022-01-31T22:32:32.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:32 smithi167 conmon[60316]: debug 2022-01-31T22:32:32.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:32.639736+0000) 2022-01-31T22:32:33.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:33 smithi171 conmon[46715]: debug 2022-01-31T22:32:33.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:33.381017+0000) 2022-01-31T22:32:33.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:33 smithi171 conmon[51620]: debug 2022-01-31T22:32:33.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:33.243216+0000) 2022-01-31T22:32:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:33 smithi171 conmon[41853]: debug 2022-01-31T22:32:33.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:33.584251+0000) 2022-01-31T22:32:33.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:33 smithi167 conmon[60316]: debug 2022-01-31T22:32:33.638+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:33.639852+0000) 2022-01-31T22:32:33.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:33 smithi167 conmon[49112]: debug 2022-01-31T22:32:33.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:33.713727+0000) 2022-01-31T22:32:33.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:33 smithi167 conmon[54076]: debug 2022-01-31T22:32:33.880+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:33.881683+0000) 2022-01-31T22:32:34.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:34 smithi171 conmon[46715]: debug 2022-01-31T22:32:34.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:34.381178+0000) 2022-01-31T22:32:34.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:34 smithi171 conmon[51620]: debug 2022-01-31T22:32:34.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:34.243435+0000) 2022-01-31T22:32:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:34 smithi171 conmon[41853]: debug 2022-01-31T22:32:34.583+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:34.584434+0000) 2022-01-31T22:32:34.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:34 smithi167 conmon[60316]: debug 2022-01-31T22:32:34.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:34.640044+0000) 2022-01-31T22:32:34.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:34 smithi167 conmon[49112]: debug 2022-01-31T22:32:34.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:34.713891+0000) 2022-01-31T22:32:34.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:34 smithi167 conmon[54076]: debug 2022-01-31T22:32:34.881+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:34.881845+0000) 2022-01-31T22:32:35.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:35 smithi171 conmon[46715]: debug 2022-01-31T22:32:35.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:35.381316+0000) 2022-01-31T22:32:35.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:35 smithi171 conmon[51620]: debug 2022-01-31T22:32:35.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:35.243652+0000) 2022-01-31T22:32:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:35 smithi171 conmon[41853]: debug 2022-01-31T22:32:35.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:35.584633+0000) 2022-01-31T22:32:35.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:35 smithi167 conmon[49112]: debug 2022-01-31T22:32:35.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:35.714073+0000) 2022-01-31T22:32:35.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:35 smithi167 conmon[54076]: debug 2022-01-31T22:32:35.881+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:35.882031+0000) 2022-01-31T22:32:35.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:35 smithi167 conmon[60316]: debug 2022-01-31T22:32:35.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:35.640250+0000) 2022-01-31T22:32:36.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[46715]: debug 2022-01-31T22:32:36.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.381469+0000) 2022-01-31T22:32:36.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[51620]: debug 2022-01-31T22:32:36.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.243865+0000) 2022-01-31T22:32:36.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[35325]: debug 2022-01-31T22:32:36.725+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217852 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:36.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[46715]: debug 2022-01-31T22:32:36.705+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.705809+0000) 2022-01-31T22:32:36.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[51620]: debug 2022-01-31T22:32:36.703+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.704214+0000) 2022-01-31T22:32:36.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[41853]: debug 2022-01-31T22:32:36.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.584746+0000) 2022-01-31T22:32:36.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:36 smithi171 conmon[41853]: debug 2022-01-31T22:32:36.702+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.703661+0000) 2022-01-31T22:32:36.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:36 smithi167 conmon[54076]: debug 2022-01-31T22:32:36.702+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.704318+0000) 2022-01-31T22:32:36.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:36 smithi167 conmon[54076]: debug 2022-01-31T22:32:36.880+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.882193+0000) 2022-01-31T22:32:36.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:36 smithi167 conmon[49112]: debug 2022-01-31T22:32:36.703+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.704728+0000) 2022-01-31T22:32:36.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:36 smithi167 conmon[49112]: debug 2022-01-31T22:32:36.712+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.714259+0000) 2022-01-31T22:32:36.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:36 smithi167 conmon[60316]: debug 2022-01-31T22:32:36.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.640425+0000) 2022-01-31T22:32:36.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:36 smithi167 conmon[60316]: debug 2022-01-31T22:32:36.703+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:36.705067+0000) 2022-01-31T22:32:37.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:37 smithi171 conmon[46715]: debug 2022-01-31T22:32:37.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:37.381646+0000) 2022-01-31T22:32:37.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:37 smithi171 conmon[51620]: debug 2022-01-31T22:32:37.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:37.244032+0000) 2022-01-31T22:32:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:37 smithi171 conmon[41853]: debug 2022-01-31T22:32:37.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:37.584940+0000) 2022-01-31T22:32:37.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:37 smithi167 conmon[49112]: debug 2022-01-31T22:32:37.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:37.714401+0000) 2022-01-31T22:32:37.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:37 smithi167 conmon[54076]: debug 2022-01-31T22:32:37.881+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:37.882376+0000) 2022-01-31T22:32:37.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:37 smithi167 conmon[60316]: debug 2022-01-31T22:32:37.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:37.640611+0000) 2022-01-31T22:32:38.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:38 smithi171 conmon[51620]: debug 2022-01-31T22:32:38.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:38.244141+0000) 2022-01-31T22:32:38.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:38 smithi171 conmon[46715]: debug 2022-01-31T22:32:38.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:38.381825+0000) 2022-01-31T22:32:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:38 smithi171 conmon[41853]: debug 2022-01-31T22:32:38.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:38.585063+0000) 2022-01-31T22:32:38.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:38 smithi167 conmon[49112]: debug 2022-01-31T22:32:38.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:38.714589+0000) 2022-01-31T22:32:38.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:38 smithi167 conmon[54076]: debug 2022-01-31T22:32:38.881+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:38.882596+0000) 2022-01-31T22:32:38.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:38 smithi167 conmon[60316]: debug 2022-01-31T22:32:38.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:38.640724+0000) 2022-01-31T22:32:39.243 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:38 smithi171 conmon[35325]: debug 2022-01-31T22:32:38.870+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:32:39.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:39 smithi171 conmon[51620]: debug 2022-01-31T22:32:39.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:39.244278+0000) 2022-01-31T22:32:39.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:39 smithi171 conmon[46715]: debug 2022-01-31T22:32:39.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:39.382010+0000) 2022-01-31T22:32:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:39 smithi171 conmon[41853]: debug 2022-01-31T22:32:39.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:39.585249+0000) 2022-01-31T22:32:39.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:39 smithi167 conmon[49112]: debug 2022-01-31T22:32:39.713+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:39.714763+0000) 2022-01-31T22:32:39.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:39 smithi167 conmon[54076]: debug 2022-01-31T22:32:39.881+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:39.882752+0000) 2022-01-31T22:32:39.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:39 smithi167 conmon[60316]: debug 2022-01-31T22:32:39.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:39.640950+0000) 2022-01-31T22:32:40.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:40 smithi171 conmon[46715]: debug 2022-01-31T22:32:40.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:40.382168+0000) 2022-01-31T22:32:40.547 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:40 smithi171 conmon[51620]: debug 2022-01-31T22:32:40.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:40.244443+0000) 2022-01-31T22:32:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:40 smithi171 conmon[41853]: debug 2022-01-31T22:32:40.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:40.585430+0000) 2022-01-31T22:32:40.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:40 smithi167 conmon[49112]: debug 2022-01-31T22:32:40.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:40.714948+0000) 2022-01-31T22:32:40.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:40 smithi167 conmon[54076]: debug 2022-01-31T22:32:40.882+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:40.883010+0000) 2022-01-31T22:32:40.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:40 smithi167 conmon[60316]: debug 2022-01-31T22:32:40.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:40.641114+0000) 2022-01-31T22:32:41.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[46715]: debug 2022-01-31T22:32:41.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.382415+0000) 2022-01-31T22:32:41.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[51620]: debug 2022-01-31T22:32:41.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.244841+0000) 2022-01-31T22:32:41.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[35325]: debug 2022-01-31T22:32:41.749+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 217961 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:41.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[41853]: debug 2022-01-31T22:32:41.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.585597+0000) 2022-01-31T22:32:41.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[41853]: debug 2022-01-31T22:32:41.727+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.728995+0000) 2022-01-31T22:32:41.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[46715]: debug 2022-01-31T22:32:41.728+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.730315+0000) 2022-01-31T22:32:41.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:41 smithi171 conmon[51620]: debug 2022-01-31T22:32:41.726+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.728262+0000) 2022-01-31T22:32:41.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:41 smithi167 conmon[49112]: debug 2022-01-31T22:32:41.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.715204+0000) 2022-01-31T22:32:41.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:41 smithi167 conmon[49112]: debug 2022-01-31T22:32:41.728+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.729708+0000) 2022-01-31T22:32:41.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:41 smithi167 conmon[54076]: debug 2022-01-31T22:32:41.728+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.729571+0000) 2022-01-31T22:32:41.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:41 smithi167 conmon[54076]: debug 2022-01-31T22:32:41.882+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.883191+0000) 2022-01-31T22:32:41.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:41 smithi167 conmon[60316]: debug 2022-01-31T22:32:41.639+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.641395+0000) 2022-01-31T22:32:41.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:41 smithi167 conmon[60316]: debug 2022-01-31T22:32:41.727+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:41.729036+0000) 2022-01-31T22:32:42.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:42 smithi171 conmon[46715]: debug 2022-01-31T22:32:42.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:42.382531+0000) 2022-01-31T22:32:42.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:42 smithi171 conmon[51620]: debug 2022-01-31T22:32:42.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:42.245051+0000) 2022-01-31T22:32:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:42 smithi171 conmon[41853]: debug 2022-01-31T22:32:42.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:42.585785+0000) 2022-01-31T22:32:42.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:42 smithi167 conmon[49112]: debug 2022-01-31T22:32:42.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:42.715367+0000) 2022-01-31T22:32:42.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:42 smithi167 conmon[54076]: debug 2022-01-31T22:32:42.882+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:42.883336+0000) 2022-01-31T22:32:42.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:42 smithi167 conmon[60316]: debug 2022-01-31T22:32:42.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:42.641598+0000) 2022-01-31T22:32:43.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:43 smithi171 conmon[46715]: debug 2022-01-31T22:32:43.380+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:43.382725+0000) 2022-01-31T22:32:43.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:43 smithi171 conmon[51620]: debug 2022-01-31T22:32:43.244+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:43.245191+0000) 2022-01-31T22:32:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:43 smithi171 conmon[41853]: debug 2022-01-31T22:32:43.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:43.585919+0000) 2022-01-31T22:32:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:43 smithi167 conmon[49112]: debug 2022-01-31T22:32:43.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:43.715544+0000) 2022-01-31T22:32:43.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:43 smithi167 conmon[54076]: debug 2022-01-31T22:32:43.882+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:43.883541+0000) 2022-01-31T22:32:43.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:43 smithi167 conmon[60316]: debug 2022-01-31T22:32:43.640+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:43.641739+0000) 2022-01-31T22:32:44.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:44 smithi171 conmon[46715]: debug 2022-01-31T22:32:44.381+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:44.382972+0000) 2022-01-31T22:32:44.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:44 smithi171 conmon[51620]: debug 2022-01-31T22:32:44.244+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:44.245426+0000) 2022-01-31T22:32:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:44 smithi171 conmon[41853]: debug 2022-01-31T22:32:44.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:44.586073+0000) 2022-01-31T22:32:44.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:44 smithi167 conmon[60316]: debug 2022-01-31T22:32:44.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:44.641931+0000) 2022-01-31T22:32:44.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:44 smithi167 conmon[49112]: debug 2022-01-31T22:32:44.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:44.715738+0000) 2022-01-31T22:32:44.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:44 smithi167 conmon[54076]: debug 2022-01-31T22:32:44.883+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:44.883744+0000) 2022-01-31T22:32:45.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:45 smithi171 conmon[46715]: debug 2022-01-31T22:32:45.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:45.383198+0000) 2022-01-31T22:32:45.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:45 smithi171 conmon[51620]: debug 2022-01-31T22:32:45.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:45.245586+0000) 2022-01-31T22:32:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:45 smithi171 conmon[41853]: debug 2022-01-31T22:32:45.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:45.586269+0000) 2022-01-31T22:32:45.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:45 smithi167 conmon[49112]: debug 2022-01-31T22:32:45.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:45.715944+0000) 2022-01-31T22:32:45.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:45 smithi167 conmon[54076]: debug 2022-01-31T22:32:45.883+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:45.883945+0000) 2022-01-31T22:32:45.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:45 smithi167 conmon[60316]: debug 2022-01-31T22:32:45.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:45.642135+0000) 2022-01-31T22:32:46.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[46715]: debug 2022-01-31T22:32:46.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.383333+0000) 2022-01-31T22:32:46.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[51620]: debug 2022-01-31T22:32:46.243+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.245737+0000) 2022-01-31T22:32:46.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[35325]: debug 2022-01-31T22:32:46.774+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218074 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:46.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[46715]: debug 2022-01-31T22:32:46.753+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.755551+0000) 2022-01-31T22:32:46.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[51620]: debug 2022-01-31T22:32:46.752+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.753957+0000) 2022-01-31T22:32:46.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[41853]: debug 2022-01-31T22:32:46.584+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.586430+0000) 2022-01-31T22:32:46.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:46 smithi171 conmon[41853]: debug 2022-01-31T22:32:46.751+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.753531+0000) 2022-01-31T22:32:46.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:46 smithi167 conmon[54076]: debug 2022-01-31T22:32:46.753+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.754855+0000) 2022-01-31T22:32:46.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:46 smithi167 conmon[54076]: debug 2022-01-31T22:32:46.883+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.884156+0000) 2022-01-31T22:32:46.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:46 smithi167 conmon[49112]: debug 2022-01-31T22:32:46.714+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.716144+0000) 2022-01-31T22:32:46.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:46 smithi167 conmon[49112]: debug 2022-01-31T22:32:46.751+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.753460+0000) 2022-01-31T22:32:46.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:46 smithi167 conmon[60316]: debug 2022-01-31T22:32:46.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.642291+0000) 2022-01-31T22:32:46.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:46 smithi167 conmon[60316]: debug 2022-01-31T22:32:46.752+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:46.754007+0000) 2022-01-31T22:32:47.548 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:47 smithi171 conmon[51620]: debug 2022-01-31T22:32:47.244+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:47.245852+0000) 2022-01-31T22:32:47.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:47 smithi171 conmon[46715]: debug 2022-01-31T22:32:47.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:47.383490+0000) 2022-01-31T22:32:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:47 smithi171 conmon[41853]: debug 2022-01-31T22:32:47.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:47.586588+0000) 2022-01-31T22:32:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:47 smithi167 conmon[49112]: debug 2022-01-31T22:32:47.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:47.716333+0000) 2022-01-31T22:32:47.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:47 smithi167 conmon[54076]: debug 2022-01-31T22:32:47.883+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:47.884327+0000) 2022-01-31T22:32:47.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:47 smithi167 conmon[60316]: debug 2022-01-31T22:32:47.641+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:47.642493+0000) 2022-01-31T22:32:48.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:48 smithi171 conmon[46715]: debug 2022-01-31T22:32:48.382+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:48.383670+0000) 2022-01-31T22:32:48.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:48 smithi171 conmon[51620]: debug 2022-01-31T22:32:48.244+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:48.245941+0000) 2022-01-31T22:32:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:48 smithi171 conmon[41853]: debug 2022-01-31T22:32:48.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:48.586741+0000) 2022-01-31T22:32:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:48 smithi167 conmon[49112]: debug 2022-01-31T22:32:48.715+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:48.716472+0000) 2022-01-31T22:32:48.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:48 smithi167 conmon[54076]: debug 2022-01-31T22:32:48.884+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:48.884523+0000) 2022-01-31T22:32:48.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:48 smithi167 conmon[60316]: debug 2022-01-31T22:32:48.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:48.642642+0000) 2022-01-31T22:32:49.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:49 smithi171 conmon[46715]: debug 2022-01-31T22:32:49.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:49.383878+0000) 2022-01-31T22:32:49.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:49 smithi171 conmon[51620]: debug 2022-01-31T22:32:49.245+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:49.246081+0000) 2022-01-31T22:32:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:49 smithi171 conmon[41853]: debug 2022-01-31T22:32:49.585+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:49.586906+0000) 2022-01-31T22:32:49.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:49 smithi167 conmon[60316]: debug 2022-01-31T22:32:49.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:49.642837+0000) 2022-01-31T22:32:49.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:49 smithi167 conmon[49112]: debug 2022-01-31T22:32:49.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:49.716654+0000) 2022-01-31T22:32:49.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:49 smithi167 conmon[54076]: debug 2022-01-31T22:32:49.884+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:49.884701+0000) 2022-01-31T22:32:50.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:50 smithi171 conmon[46715]: debug 2022-01-31T22:32:50.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:50.384062+0000) 2022-01-31T22:32:50.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:50 smithi171 conmon[51620]: debug 2022-01-31T22:32:50.244+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:50.246266+0000) 2022-01-31T22:32:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:50 smithi171 conmon[41853]: debug 2022-01-31T22:32:50.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:50.587097+0000) 2022-01-31T22:32:50.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:50 smithi167 conmon[49112]: debug 2022-01-31T22:32:50.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:50.716804+0000) 2022-01-31T22:32:50.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:50 smithi167 conmon[54076]: debug 2022-01-31T22:32:50.884+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:50.884911+0000) 2022-01-31T22:32:50.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:50 smithi167 conmon[60316]: debug 2022-01-31T22:32:50.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:50.643003+0000) 2022-01-31T22:32:51.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[46715]: debug 2022-01-31T22:32:51.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.384205+0000) 2022-01-31T22:32:51.549 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[51620]: debug 2022-01-31T22:32:51.244+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.246423+0000) 2022-01-31T22:32:51.801 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[41853]: debug 2022-01-31T22:32:51.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.587276+0000) 2022-01-31T22:32:51.802 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[41853]: debug 2022-01-31T22:32:51.776+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.778039+0000) 2022-01-31T22:32:51.802 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[46715]: debug 2022-01-31T22:32:51.779+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.780776+0000) 2022-01-31T22:32:51.803 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[51620]: debug 2022-01-31T22:32:51.777+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.779325+0000) 2022-01-31T22:32:51.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:51 smithi167 conmon[60316]: debug 2022-01-31T22:32:51.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.643165+0000) 2022-01-31T22:32:51.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:51 smithi167 conmon[60316]: debug 2022-01-31T22:32:51.778+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.779376+0000) 2022-01-31T22:32:51.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:51 smithi167 conmon[49112]: debug 2022-01-31T22:32:51.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.716986+0000) 2022-01-31T22:32:51.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:51 smithi167 conmon[49112]: debug 2022-01-31T22:32:51.778+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.779243+0000) 2022-01-31T22:32:51.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:51 smithi167 conmon[54076]: debug 2022-01-31T22:32:51.779+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.779756+0000) 2022-01-31T22:32:51.921 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:51 smithi167 conmon[54076]: debug 2022-01-31T22:32:51.884+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:51.885095+0000) 2022-01-31T22:32:52.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:51 smithi171 conmon[35325]: debug 2022-01-31T22:32:51.799+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218183 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:52.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:52 smithi171 conmon[46715]: debug 2022-01-31T22:32:52.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:52.384386+0000) 2022-01-31T22:32:52.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:52 smithi171 conmon[51620]: debug 2022-01-31T22:32:52.245+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:52.246636+0000) 2022-01-31T22:32:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:52 smithi171 conmon[41853]: debug 2022-01-31T22:32:52.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:52.587478+0000) 2022-01-31T22:32:52.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:52 smithi167 conmon[49112]: debug 2022-01-31T22:32:52.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:52.717115+0000) 2022-01-31T22:32:52.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:52 smithi167 conmon[54076]: debug 2022-01-31T22:32:52.884+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:52.885249+0000) 2022-01-31T22:32:52.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:52 smithi167 conmon[60316]: debug 2022-01-31T22:32:52.642+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:52.643353+0000) 2022-01-31T22:32:53.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:53 smithi171 conmon[46715]: debug 2022-01-31T22:32:53.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:53.384529+0000) 2022-01-31T22:32:53.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:53 smithi171 conmon[51620]: debug 2022-01-31T22:32:53.246+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:53.246779+0000) 2022-01-31T22:32:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:53 smithi171 conmon[41853]: debug 2022-01-31T22:32:53.586+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:53.587651+0000) 2022-01-31T22:32:53.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:53 smithi167 conmon[60316]: debug 2022-01-31T22:32:53.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:53.643538+0000) 2022-01-31T22:32:53.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:53 smithi167 conmon[49112]: debug 2022-01-31T22:32:53.716+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:53.717279+0000) 2022-01-31T22:32:53.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:53 smithi167 conmon[54076]: debug 2022-01-31T22:32:53.884+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:53.885450+0000) 2022-01-31T22:32:54.246 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:53 smithi171 conmon[35325]: debug 2022-01-31T22:32:53.870+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:32:54.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:54 smithi171 conmon[46715]: debug 2022-01-31T22:32:54.383+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:54.384695+0000) 2022-01-31T22:32:54.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:54 smithi171 conmon[51620]: debug 2022-01-31T22:32:54.246+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:54.246881+0000) 2022-01-31T22:32:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:54 smithi171 conmon[41853]: debug 2022-01-31T22:32:54.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:54.587831+0000) 2022-01-31T22:32:54.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:54 smithi167 conmon[49112]: debug 2022-01-31T22:32:54.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:54.717481+0000) 2022-01-31T22:32:54.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:54 smithi167 conmon[54076]: debug 2022-01-31T22:32:54.885+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:54.885642+0000) 2022-01-31T22:32:54.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:54 smithi167 conmon[60316]: debug 2022-01-31T22:32:54.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:54.643681+0000) 2022-01-31T22:32:55.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:55 smithi171 conmon[46715]: debug 2022-01-31T22:32:55.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:55.384843+0000) 2022-01-31T22:32:55.610 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:55 smithi171 conmon[51620]: debug 2022-01-31T22:32:55.246+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:55.247041+0000) 2022-01-31T22:32:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:55 smithi171 conmon[41853]: debug 2022-01-31T22:32:55.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:55.588009+0000) 2022-01-31T22:32:55.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:55 smithi167 conmon[60316]: debug 2022-01-31T22:32:55.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:55.643840+0000) 2022-01-31T22:32:55.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:55 smithi167 conmon[49112]: debug 2022-01-31T22:32:55.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:55.717699+0000) 2022-01-31T22:32:55.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:55 smithi167 conmon[54076]: debug 2022-01-31T22:32:55.885+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:55.885825+0000) 2022-01-31T22:32:56.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[46715]: debug 2022-01-31T22:32:56.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.385005+0000) 2022-01-31T22:32:56.550 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[51620]: debug 2022-01-31T22:32:56.246+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.247231+0000) 2022-01-31T22:32:56.803 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[41853]: debug 2022-01-31T22:32:56.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.588156+0000) 2022-01-31T22:32:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:56 smithi167 conmon[49112]: debug 2022-01-31T22:32:56.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.717894+0000) 2022-01-31T22:32:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:56 smithi167 conmon[49112]: debug 2022-01-31T22:32:56.804+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.804923+0000) 2022-01-31T22:32:56.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:56 smithi167 conmon[54076]: debug 2022-01-31T22:32:56.804+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.805144+0000) 2022-01-31T22:32:56.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:56 smithi167 conmon[54076]: debug 2022-01-31T22:32:56.885+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.885989+0000) 2022-01-31T22:32:56.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:56 smithi167 conmon[60316]: debug 2022-01-31T22:32:56.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.644007+0000) 2022-01-31T22:32:56.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:56 smithi167 conmon[60316]: debug 2022-01-31T22:32:56.804+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.804827+0000) 2022-01-31T22:32:57.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[35325]: debug 2022-01-31T22:32:56.825+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218296 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:32:57.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[41853]: debug 2022-01-31T22:32:56.802+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.803320+0000) 2022-01-31T22:32:57.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[46715]: debug 2022-01-31T22:32:56.803+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.804250+0000) 2022-01-31T22:32:57.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:56 smithi171 conmon[51620]: debug 2022-01-31T22:32:56.804+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:56.805188+0000) 2022-01-31T22:32:57.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:57 smithi171 conmon[46715]: debug 2022-01-31T22:32:57.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:57.385197+0000) 2022-01-31T22:32:57.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:57 smithi171 conmon[51620]: debug 2022-01-31T22:32:57.246+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:57.247416+0000) 2022-01-31T22:32:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:57 smithi171 conmon[41853]: debug 2022-01-31T22:32:57.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:57.588295+0000) 2022-01-31T22:32:57.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:57 smithi167 conmon[60316]: debug 2022-01-31T22:32:57.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:57.644201+0000) 2022-01-31T22:32:57.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:57 smithi167 conmon[49112]: debug 2022-01-31T22:32:57.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:57.718084+0000) 2022-01-31T22:32:57.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:57 smithi167 conmon[54076]: debug 2022-01-31T22:32:57.885+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:57.886144+0000) 2022-01-31T22:32:58.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:58 smithi171 conmon[46715]: debug 2022-01-31T22:32:58.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:58.385367+0000) 2022-01-31T22:32:58.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:58 smithi171 conmon[51620]: debug 2022-01-31T22:32:58.246+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:58.247584+0000) 2022-01-31T22:32:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:58 smithi171 conmon[41853]: debug 2022-01-31T22:32:58.587+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:58.588454+0000) 2022-01-31T22:32:58.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:58 smithi167 conmon[60316]: debug 2022-01-31T22:32:58.643+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:58.644353+0000) 2022-01-31T22:32:58.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:58 smithi167 conmon[49112]: debug 2022-01-31T22:32:58.717+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:58.718254+0000) 2022-01-31T22:32:58.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:58 smithi167 conmon[54076]: debug 2022-01-31T22:32:58.886+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:58.886332+0000) 2022-01-31T22:32:59.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:32:59 smithi171 conmon[46715]: debug 2022-01-31T22:32:59.384+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:59.385522+0000) 2022-01-31T22:32:59.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:32:59 smithi171 conmon[51620]: debug 2022-01-31T22:32:59.247+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:59.247798+0000) 2022-01-31T22:32:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:32:59 smithi171 conmon[41853]: debug 2022-01-31T22:32:59.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:59.588642+0000) 2022-01-31T22:32:59.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:32:59 smithi167 conmon[49112]: debug 2022-01-31T22:32:59.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:59.718460+0000) 2022-01-31T22:32:59.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:32:59 smithi167 conmon[54076]: debug 2022-01-31T22:32:59.886+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:59.886533+0000) 2022-01-31T22:32:59.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:32:59 smithi167 conmon[60316]: debug 2022-01-31T22:32:59.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:32:59.644543+0000) 2022-01-31T22:33:00.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:00 smithi171 conmon[46715]: debug 2022-01-31T22:33:00.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:00.385739+0000) 2022-01-31T22:33:00.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:00 smithi171 conmon[51620]: debug 2022-01-31T22:33:00.247+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:00.248011+0000) 2022-01-31T22:33:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:00 smithi171 conmon[41853]: debug 2022-01-31T22:33:00.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:00.588852+0000) 2022-01-31T22:33:00.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:00 smithi167 conmon[60316]: debug 2022-01-31T22:33:00.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:00.644733+0000) 2022-01-31T22:33:00.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:00 smithi167 conmon[49112]: debug 2022-01-31T22:33:00.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:00.718686+0000) 2022-01-31T22:33:00.921 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:00 smithi167 conmon[54076]: debug 2022-01-31T22:33:00.886+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:00.886700+0000) 2022-01-31T22:33:01.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[46715]: debug 2022-01-31T22:33:01.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.385923+0000) 2022-01-31T22:33:01.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[51620]: debug 2022-01-31T22:33:01.247+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.248200+0000) 2022-01-31T22:33:01.828 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[41853]: debug 2022-01-31T22:33:01.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.588972+0000) 2022-01-31T22:33:01.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:01 smithi167 conmon[60316]: debug 2022-01-31T22:33:01.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.644895+0000) 2022-01-31T22:33:01.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:01 smithi167 conmon[60316]: debug 2022-01-31T22:33:01.835+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.835581+0000) 2022-01-31T22:33:01.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:01 smithi167 conmon[49112]: debug 2022-01-31T22:33:01.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.718879+0000) 2022-01-31T22:33:01.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:01 smithi167 conmon[49112]: debug 2022-01-31T22:33:01.827+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.829345+0000) 2022-01-31T22:33:01.921 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:01 smithi167 conmon[54076]: debug 2022-01-31T22:33:01.835+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.835973+0000) 2022-01-31T22:33:01.921 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:01 smithi167 conmon[54076]: debug 2022-01-31T22:33:01.886+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.886823+0000) 2022-01-31T22:33:02.107 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[35325]: debug 2022-01-31T22:33:01.865+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218405 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:02.108 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[41853]: debug 2022-01-31T22:33:01.828+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.828747+0000) 2022-01-31T22:33:02.108 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[46715]: debug 2022-01-31T22:33:01.828+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.829012+0000) 2022-01-31T22:33:02.109 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:01 smithi171 conmon[51620]: debug 2022-01-31T22:33:01.829+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:01.829782+0000) 2022-01-31T22:33:02.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:02 smithi171 conmon[46715]: debug 2022-01-31T22:33:02.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:02.386081+0000) 2022-01-31T22:33:02.551 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:02 smithi171 conmon[51620]: debug 2022-01-31T22:33:02.247+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:02.248397+0000) 2022-01-31T22:33:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:02 smithi171 conmon[41853]: debug 2022-01-31T22:33:02.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:02.589155+0000) 2022-01-31T22:33:02.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:02 smithi167 conmon[60316]: debug 2022-01-31T22:33:02.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:02.645032+0000) 2022-01-31T22:33:02.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:02 smithi167 conmon[49112]: debug 2022-01-31T22:33:02.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:02.719083+0000) 2022-01-31T22:33:02.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:02 smithi167 conmon[54076]: debug 2022-01-31T22:33:02.886+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:02.886961+0000) 2022-01-31T22:33:03.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:03 smithi171 conmon[46715]: debug 2022-01-31T22:33:03.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:03.386221+0000) 2022-01-31T22:33:03.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:03 smithi171 conmon[51620]: debug 2022-01-31T22:33:03.247+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:03.248531+0000) 2022-01-31T22:33:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:03 smithi171 conmon[41853]: debug 2022-01-31T22:33:03.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:03.589326+0000) 2022-01-31T22:33:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:03 smithi167 conmon[49112]: debug 2022-01-31T22:33:03.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:03.719263+0000) 2022-01-31T22:33:03.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:03 smithi167 conmon[54076]: debug 2022-01-31T22:33:03.886+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:03.887144+0000) 2022-01-31T22:33:03.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:03 smithi167 conmon[60316]: debug 2022-01-31T22:33:03.644+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:03.645177+0000) 2022-01-31T22:33:04.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:04 smithi171 conmon[46715]: debug 2022-01-31T22:33:04.385+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:04.386413+0000) 2022-01-31T22:33:04.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:04 smithi171 conmon[51620]: debug 2022-01-31T22:33:04.248+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:04.248716+0000) 2022-01-31T22:33:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:04 smithi171 conmon[41853]: debug 2022-01-31T22:33:04.588+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:04.589531+0000) 2022-01-31T22:33:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:04 smithi167 conmon[49112]: debug 2022-01-31T22:33:04.718+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:04.719467+0000) 2022-01-31T22:33:04.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:04 smithi167 conmon[54076]: debug 2022-01-31T22:33:04.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:04.887348+0000) 2022-01-31T22:33:04.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:04 smithi167 conmon[60316]: debug 2022-01-31T22:33:04.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:04.645361+0000) 2022-01-31T22:33:05.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:05 smithi171 conmon[46715]: debug 2022-01-31T22:33:05.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:05.386630+0000) 2022-01-31T22:33:05.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:05 smithi171 conmon[51620]: debug 2022-01-31T22:33:05.248+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:05.248940+0000) 2022-01-31T22:33:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:05 smithi171 conmon[41853]: debug 2022-01-31T22:33:05.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:05.589739+0000) 2022-01-31T22:33:05.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:05 smithi167 conmon[49112]: debug 2022-01-31T22:33:05.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:05.719719+0000) 2022-01-31T22:33:05.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:05 smithi167 conmon[54076]: debug 2022-01-31T22:33:05.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:05.887530+0000) 2022-01-31T22:33:05.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:05 smithi167 conmon[60316]: debug 2022-01-31T22:33:05.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:05.645535+0000) 2022-01-31T22:33:06.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[46715]: debug 2022-01-31T22:33:06.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.386815+0000) 2022-01-31T22:33:06.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[51620]: debug 2022-01-31T22:33:06.248+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.249141+0000) 2022-01-31T22:33:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[41853]: debug 2022-01-31T22:33:06.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.589954+0000) 2022-01-31T22:33:06.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:06 smithi167 conmon[49112]: debug 2022-01-31T22:33:06.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.719850+0000) 2022-01-31T22:33:06.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:06 smithi167 conmon[49112]: debug 2022-01-31T22:33:06.869+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.870253+0000) 2022-01-31T22:33:06.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:06 smithi167 conmon[60316]: debug 2022-01-31T22:33:06.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.645738+0000) 2022-01-31T22:33:06.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:06 smithi167 conmon[60316]: debug 2022-01-31T22:33:06.870+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.870726+0000) 2022-01-31T22:33:06.922 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:06 smithi167 conmon[54076]: debug 2022-01-31T22:33:06.870+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.870712+0000) 2022-01-31T22:33:06.922 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:06 smithi167 conmon[54076]: debug 2022-01-31T22:33:06.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.887634+0000) 2022-01-31T22:33:07.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[35325]: debug 2022-01-31T22:33:06.890+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218520 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:07.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[41853]: debug 2022-01-31T22:33:06.867+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.868532+0000) 2022-01-31T22:33:07.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[46715]: debug 2022-01-31T22:33:06.868+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.869056+0000) 2022-01-31T22:33:07.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:06 smithi171 conmon[51620]: debug 2022-01-31T22:33:06.869+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:06.869852+0000) 2022-01-31T22:33:07.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:07 smithi171 conmon[46715]: debug 2022-01-31T22:33:07.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:07.387044+0000) 2022-01-31T22:33:07.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:07 smithi171 conmon[51620]: debug 2022-01-31T22:33:07.248+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:07.249345+0000) 2022-01-31T22:33:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:07 smithi171 conmon[41853]: debug 2022-01-31T22:33:07.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:07.590083+0000) 2022-01-31T22:33:07.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:07 smithi167 conmon[49112]: debug 2022-01-31T22:33:07.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:07.720007+0000) 2022-01-31T22:33:07.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:07 smithi167 conmon[54076]: debug 2022-01-31T22:33:07.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:07.887808+0000) 2022-01-31T22:33:07.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:07 smithi167 conmon[60316]: debug 2022-01-31T22:33:07.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:07.645977+0000) 2022-01-31T22:33:08.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:08 smithi171 conmon[46715]: debug 2022-01-31T22:33:08.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:08.387163+0000) 2022-01-31T22:33:08.552 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:08 smithi171 conmon[51620]: debug 2022-01-31T22:33:08.248+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:08.249497+0000) 2022-01-31T22:33:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:08 smithi171 conmon[41853]: debug 2022-01-31T22:33:08.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:08.590240+0000) 2022-01-31T22:33:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:08 smithi167 conmon[49112]: debug 2022-01-31T22:33:08.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:08.720177+0000) 2022-01-31T22:33:08.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:08 smithi167 conmon[60316]: debug 2022-01-31T22:33:08.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:08.646132+0000) 2022-01-31T22:33:08.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:08 smithi167 conmon[54076]: debug 2022-01-31T22:33:08.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:08.888012+0000) 2022-01-31T22:33:09.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:08 smithi171 conmon[35325]: debug 2022-01-31T22:33:08.871+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:33:09.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:09 smithi171 conmon[46715]: debug 2022-01-31T22:33:09.386+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:09.387317+0000) 2022-01-31T22:33:09.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:09 smithi171 conmon[51620]: debug 2022-01-31T22:33:09.249+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:09.249696+0000) 2022-01-31T22:33:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:09 smithi171 conmon[41853]: debug 2022-01-31T22:33:09.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:09.590422+0000) 2022-01-31T22:33:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:09 smithi167 conmon[49112]: debug 2022-01-31T22:33:09.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:09.720351+0000) 2022-01-31T22:33:09.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:09 smithi167 conmon[54076]: debug 2022-01-31T22:33:09.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:09.888219+0000) 2022-01-31T22:33:09.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:09 smithi167 conmon[60316]: debug 2022-01-31T22:33:09.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:09.646319+0000) 2022-01-31T22:33:10.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:10 smithi171 conmon[46715]: debug 2022-01-31T22:33:10.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:10.387494+0000) 2022-01-31T22:33:10.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:10 smithi171 conmon[51620]: debug 2022-01-31T22:33:10.249+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:10.249921+0000) 2022-01-31T22:33:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:10 smithi171 conmon[41853]: debug 2022-01-31T22:33:10.589+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:10.590611+0000) 2022-01-31T22:33:10.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:10 smithi167 conmon[60316]: debug 2022-01-31T22:33:10.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:10.646530+0000) 2022-01-31T22:33:10.957 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:10 smithi167 conmon[49112]: debug 2022-01-31T22:33:10.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:10.720578+0000) 2022-01-31T22:33:10.958 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:10 smithi167 conmon[54076]: debug 2022-01-31T22:33:10.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:10.888441+0000) 2022-01-31T22:33:11.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[46715]: debug 2022-01-31T22:33:11.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.387682+0000) 2022-01-31T22:33:11.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[51620]: debug 2022-01-31T22:33:11.249+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.250110+0000) 2022-01-31T22:33:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[41853]: debug 2022-01-31T22:33:11.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.590772+0000) 2022-01-31T22:33:11.897 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:11 smithi167 conmon[49112]: debug 2022-01-31T22:33:11.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.720765+0000) 2022-01-31T22:33:11.897 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:11 smithi167 conmon[49112]: debug 2022-01-31T22:33:11.894+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.895070+0000) 2022-01-31T22:33:11.898 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:11 smithi167 conmon[54076]: debug 2022-01-31T22:33:11.888+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.888640+0000) 2022-01-31T22:33:11.898 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:11 smithi167 conmon[54076]: debug 2022-01-31T22:33:11.893+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.894098+0000) 2022-01-31T22:33:11.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:11 smithi167 conmon[60316]: debug 2022-01-31T22:33:11.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.646692+0000) 2022-01-31T22:33:11.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:11 smithi167 conmon[60316]: debug 2022-01-31T22:33:11.894+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.895451+0000) 2022-01-31T22:33:12.249 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[41853]: debug 2022-01-31T22:33:11.893+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.893795+0000) 2022-01-31T22:33:12.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[46715]: debug 2022-01-31T22:33:11.894+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.894943+0000) 2022-01-31T22:33:12.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[51620]: debug 2022-01-31T22:33:11.893+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:11.894434+0000) 2022-01-31T22:33:12.252 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:11 smithi171 conmon[35325]: debug 2022-01-31T22:33:11.915+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218629 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:12.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:12 smithi171 conmon[46715]: debug 2022-01-31T22:33:12.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:12.387847+0000) 2022-01-31T22:33:12.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:12 smithi171 conmon[51620]: debug 2022-01-31T22:33:12.249+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:12.250297+0000) 2022-01-31T22:33:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:12 smithi171 conmon[41853]: debug 2022-01-31T22:33:12.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:12.590929+0000) 2022-01-31T22:33:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:12 smithi167 conmon[49112]: debug 2022-01-31T22:33:12.719+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:12.720930+0000) 2022-01-31T22:33:12.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:12 smithi167 conmon[54076]: debug 2022-01-31T22:33:12.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:12.888783+0000) 2022-01-31T22:33:12.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:12 smithi167 conmon[60316]: debug 2022-01-31T22:33:12.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:12.646852+0000) 2022-01-31T22:33:13.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:13 smithi171 conmon[46715]: debug 2022-01-31T22:33:13.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:13.387969+0000) 2022-01-31T22:33:13.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:13 smithi171 conmon[51620]: debug 2022-01-31T22:33:13.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:13.250446+0000) 2022-01-31T22:33:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:13 smithi171 conmon[41853]: debug 2022-01-31T22:33:13.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:13.591095+0000) 2022-01-31T22:33:13.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:13 smithi167 conmon[49112]: debug 2022-01-31T22:33:13.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:13.721068+0000) 2022-01-31T22:33:13.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:13 smithi167 conmon[54076]: debug 2022-01-31T22:33:13.887+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:13.888971+0000) 2022-01-31T22:33:13.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:13 smithi167 conmon[60316]: debug 2022-01-31T22:33:13.645+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:13.647021+0000) 2022-01-31T22:33:14.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:14 smithi171 conmon[51620]: debug 2022-01-31T22:33:14.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:14.250640+0000) 2022-01-31T22:33:14.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:14 smithi171 conmon[46715]: debug 2022-01-31T22:33:14.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:14.388155+0000) 2022-01-31T22:33:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:14 smithi171 conmon[41853]: debug 2022-01-31T22:33:14.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:14.591302+0000) 2022-01-31T22:33:14.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:14 smithi167 conmon[49112]: debug 2022-01-31T22:33:14.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:14.721224+0000) 2022-01-31T22:33:14.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:14 smithi167 conmon[54076]: debug 2022-01-31T22:33:14.888+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:14.889142+0000) 2022-01-31T22:33:14.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:14 smithi167 conmon[60316]: debug 2022-01-31T22:33:14.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:14.647226+0000) 2022-01-31T22:33:15.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:15 smithi171 conmon[46715]: debug 2022-01-31T22:33:15.387+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:15.388363+0000) 2022-01-31T22:33:15.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:15 smithi171 conmon[51620]: debug 2022-01-31T22:33:15.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:15.250821+0000) 2022-01-31T22:33:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:15 smithi171 conmon[41853]: debug 2022-01-31T22:33:15.590+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:15.591507+0000) 2022-01-31T22:33:15.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:15 smithi167 conmon[49112]: debug 2022-01-31T22:33:15.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:15.721387+0000) 2022-01-31T22:33:15.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:15 smithi167 conmon[54076]: debug 2022-01-31T22:33:15.888+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:15.889301+0000) 2022-01-31T22:33:15.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:15 smithi167 conmon[60316]: debug 2022-01-31T22:33:15.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:15.647419+0000) 2022-01-31T22:33:16.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[51620]: debug 2022-01-31T22:33:16.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.251025+0000) 2022-01-31T22:33:16.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[46715]: debug 2022-01-31T22:33:16.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.388542+0000) 2022-01-31T22:33:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[41853]: debug 2022-01-31T22:33:16.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.591693+0000) 2022-01-31T22:33:16.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:16 smithi167 conmon[49112]: debug 2022-01-31T22:33:16.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.721594+0000) 2022-01-31T22:33:16.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:16 smithi167 conmon[54076]: debug 2022-01-31T22:33:16.888+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.889495+0000) 2022-01-31T22:33:16.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:16 smithi167 conmon[60316]: debug 2022-01-31T22:33:16.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.647606+0000) 2022-01-31T22:33:17.168 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:16 smithi167 conmon[49112]: debug 2022-01-31T22:33:16.918+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.919801+0000) 2022-01-31T22:33:17.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:16 smithi167 conmon[54076]: debug 2022-01-31T22:33:16.918+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.919600+0000) 2022-01-31T22:33:17.170 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:16 smithi167 conmon[60316]: debug 2022-01-31T22:33:16.918+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.919933+0000) 2022-01-31T22:33:17.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[41853]: debug 2022-01-31T22:33:16.918+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.918811+0000) 2022-01-31T22:33:17.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[46715]: debug 2022-01-31T22:33:16.918+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.919674+0000) 2022-01-31T22:33:17.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[51620]: debug 2022-01-31T22:33:16.918+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:16.919425+0000) 2022-01-31T22:33:17.253 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:16 smithi171 conmon[35325]: debug 2022-01-31T22:33:16.940+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218742 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:17.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:17 smithi171 conmon[46715]: debug 2022-01-31T22:33:17.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:17.388700+0000) 2022-01-31T22:33:17.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:17 smithi171 conmon[51620]: debug 2022-01-31T22:33:17.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:17.251208+0000) 2022-01-31T22:33:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:17 smithi171 conmon[41853]: debug 2022-01-31T22:33:17.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:17.591879+0000) 2022-01-31T22:33:17.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:17 smithi167 conmon[49112]: debug 2022-01-31T22:33:17.720+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:17.721821+0000) 2022-01-31T22:33:17.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:17 smithi167 conmon[54076]: debug 2022-01-31T22:33:17.888+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:17.889702+0000) 2022-01-31T22:33:17.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:17 smithi167 conmon[60316]: debug 2022-01-31T22:33:17.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:17.647767+0000) 2022-01-31T22:33:18.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:18 smithi171 conmon[46715]: debug 2022-01-31T22:33:18.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:18.388839+0000) 2022-01-31T22:33:18.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:18 smithi171 conmon[51620]: debug 2022-01-31T22:33:18.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:18.251371+0000) 2022-01-31T22:33:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:18 smithi171 conmon[41853]: debug 2022-01-31T22:33:18.591+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:18.592051+0000) 2022-01-31T22:33:18.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:18 smithi167 conmon[49112]: debug 2022-01-31T22:33:18.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:18.721976+0000) 2022-01-31T22:33:18.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:18 smithi167 conmon[54076]: debug 2022-01-31T22:33:18.888+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:18.889844+0000) 2022-01-31T22:33:18.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:18 smithi167 conmon[60316]: debug 2022-01-31T22:33:18.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:18.647961+0000) 2022-01-31T22:33:19.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:19 smithi171 conmon[46715]: debug 2022-01-31T22:33:19.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:19.389025+0000) 2022-01-31T22:33:19.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:19 smithi171 conmon[51620]: debug 2022-01-31T22:33:19.250+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:19.251580+0000) 2022-01-31T22:33:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:19 smithi171 conmon[41853]: debug 2022-01-31T22:33:19.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:19.592230+0000) 2022-01-31T22:33:19.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:19 smithi167 conmon[60316]: debug 2022-01-31T22:33:19.646+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:19.648169+0000) 2022-01-31T22:33:19.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:19 smithi167 conmon[49112]: debug 2022-01-31T22:33:19.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:19.722173+0000) 2022-01-31T22:33:19.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:19 smithi167 conmon[54076]: debug 2022-01-31T22:33:19.889+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:19.890056+0000) 2022-01-31T22:33:20.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:20 smithi171 conmon[46715]: debug 2022-01-31T22:33:20.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:20.389212+0000) 2022-01-31T22:33:20.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:20 smithi171 conmon[51620]: debug 2022-01-31T22:33:20.251+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:20.251749+0000) 2022-01-31T22:33:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:20 smithi171 conmon[41853]: debug 2022-01-31T22:33:20.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:20.592429+0000) 2022-01-31T22:33:20.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:20 smithi167 conmon[49112]: debug 2022-01-31T22:33:20.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:20.722369+0000) 2022-01-31T22:33:20.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:20 smithi167 conmon[54076]: debug 2022-01-31T22:33:20.889+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:20.890272+0000) 2022-01-31T22:33:20.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:20 smithi167 conmon[60316]: debug 2022-01-31T22:33:20.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:20.648402+0000) 2022-01-31T22:33:21.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[46715]: debug 2022-01-31T22:33:21.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.389397+0000) 2022-01-31T22:33:21.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[51620]: debug 2022-01-31T22:33:21.251+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.251965+0000) 2022-01-31T22:33:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[41853]: debug 2022-01-31T22:33:21.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.592611+0000) 2022-01-31T22:33:21.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:21 smithi167 conmon[49112]: debug 2022-01-31T22:33:21.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.722532+0000) 2022-01-31T22:33:21.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:21 smithi167 conmon[54076]: debug 2022-01-31T22:33:21.889+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.890457+0000) 2022-01-31T22:33:21.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:21 smithi167 conmon[60316]: debug 2022-01-31T22:33:21.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.648606+0000) 2022-01-31T22:33:22.251 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[35325]: debug 2022-01-31T22:33:21.966+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218851 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:22.252 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[41853]: debug 2022-01-31T22:33:21.942+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.943713+0000) 2022-01-31T22:33:22.253 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[46715]: debug 2022-01-31T22:33:21.944+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.945296+0000) 2022-01-31T22:33:22.253 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:21 smithi171 conmon[51620]: debug 2022-01-31T22:33:21.944+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.945484+0000) 2022-01-31T22:33:22.371 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:21 smithi167 conmon[49112]: debug 2022-01-31T22:33:21.943+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.944826+0000) 2022-01-31T22:33:22.371 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:21 smithi167 conmon[54076]: debug 2022-01-31T22:33:21.944+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.945555+0000) 2022-01-31T22:33:22.372 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:21 smithi167 conmon[60316]: debug 2022-01-31T22:33:21.943+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:21.945092+0000) 2022-01-31T22:33:22.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:22 smithi171 conmon[46715]: debug 2022-01-31T22:33:22.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:22.389610+0000) 2022-01-31T22:33:22.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:22 smithi171 conmon[51620]: debug 2022-01-31T22:33:22.251+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:22.252160+0000) 2022-01-31T22:33:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:22 smithi171 conmon[41853]: debug 2022-01-31T22:33:22.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:22.592792+0000) 2022-01-31T22:33:22.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:22 smithi167 conmon[54076]: debug 2022-01-31T22:33:22.889+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:22.890626+0000) 2022-01-31T22:33:22.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:22 smithi167 conmon[49112]: debug 2022-01-31T22:33:22.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:22.722711+0000) 2022-01-31T22:33:22.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:22 smithi167 conmon[60316]: debug 2022-01-31T22:33:22.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:22.648771+0000) 2022-01-31T22:33:23.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:23 smithi171 conmon[46715]: debug 2022-01-31T22:33:23.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:23.389766+0000) 2022-01-31T22:33:23.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:23 smithi171 conmon[51620]: debug 2022-01-31T22:33:23.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:23.252319+0000) 2022-01-31T22:33:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:23 smithi171 conmon[41853]: debug 2022-01-31T22:33:23.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:23.592942+0000) 2022-01-31T22:33:23.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:23 smithi167 conmon[54076]: debug 2022-01-31T22:33:23.889+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:23.890731+0000) 2022-01-31T22:33:23.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:23 smithi167 conmon[49112]: debug 2022-01-31T22:33:23.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:23.722874+0000) 2022-01-31T22:33:23.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:23 smithi167 conmon[60316]: debug 2022-01-31T22:33:23.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:23.648905+0000) 2022-01-31T22:33:24.252 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:23 smithi171 conmon[35325]: debug 2022-01-31T22:33:23.872+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:33:24.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:24 smithi171 conmon[46715]: debug 2022-01-31T22:33:24.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:24.389990+0000) 2022-01-31T22:33:24.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:24 smithi171 conmon[51620]: debug 2022-01-31T22:33:24.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:24.252531+0000) 2022-01-31T22:33:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:24 smithi171 conmon[41853]: debug 2022-01-31T22:33:24.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:24.593089+0000) 2022-01-31T22:33:24.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:24 smithi167 conmon[60316]: debug 2022-01-31T22:33:24.647+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:24.649093+0000) 2022-01-31T22:33:24.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:24 smithi167 conmon[49112]: debug 2022-01-31T22:33:24.721+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:24.723037+0000) 2022-01-31T22:33:24.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:24 smithi167 conmon[54076]: debug 2022-01-31T22:33:24.890+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:24.890878+0000) 2022-01-31T22:33:25.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:25 smithi171 conmon[46715]: debug 2022-01-31T22:33:25.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:25.390146+0000) 2022-01-31T22:33:25.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:25 smithi171 conmon[51620]: debug 2022-01-31T22:33:25.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:25.252732+0000) 2022-01-31T22:33:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:25 smithi171 conmon[41853]: debug 2022-01-31T22:33:25.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:25.593258+0000) 2022-01-31T22:33:25.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:25 smithi167 conmon[54076]: debug 2022-01-31T22:33:25.890+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:25.891061+0000) 2022-01-31T22:33:25.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:25 smithi167 conmon[49112]: debug 2022-01-31T22:33:25.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:25.723223+0000) 2022-01-31T22:33:25.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:25 smithi167 conmon[60316]: debug 2022-01-31T22:33:25.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:25.649312+0000) 2022-01-31T22:33:26.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[46715]: debug 2022-01-31T22:33:26.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.390335+0000) 2022-01-31T22:33:26.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[51620]: debug 2022-01-31T22:33:26.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.252933+0000) 2022-01-31T22:33:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[41853]: debug 2022-01-31T22:33:26.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.593494+0000) 2022-01-31T22:33:26.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:26 smithi167 conmon[54076]: debug 2022-01-31T22:33:26.890+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.891214+0000) 2022-01-31T22:33:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:26 smithi167 conmon[49112]: debug 2022-01-31T22:33:26.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.723411+0000) 2022-01-31T22:33:26.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:26 smithi167 conmon[60316]: debug 2022-01-31T22:33:26.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.649506+0000) 2022-01-31T22:33:27.252 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[35325]: debug 2022-01-31T22:33:26.991+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 218964 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:27.253 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[41853]: debug 2022-01-31T22:33:26.968+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.969176+0000) 2022-01-31T22:33:27.254 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[46715]: debug 2022-01-31T22:33:26.970+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.970769+0000) 2022-01-31T22:33:27.254 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:26 smithi171 conmon[51620]: debug 2022-01-31T22:33:26.969+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.970292+0000) 2022-01-31T22:33:27.371 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:26 smithi167 conmon[49112]: debug 2022-01-31T22:33:26.968+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.970210+0000) 2022-01-31T22:33:27.372 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:26 smithi167 conmon[54076]: debug 2022-01-31T22:33:26.969+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.971386+0000) 2022-01-31T22:33:27.373 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:26 smithi167 conmon[60316]: debug 2022-01-31T22:33:26.969+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:26.970771+0000) 2022-01-31T22:33:27.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:27 smithi171 conmon[46715]: debug 2022-01-31T22:33:27.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:27.390524+0000) 2022-01-31T22:33:27.555 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:27 smithi171 conmon[51620]: debug 2022-01-31T22:33:27.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:27.253089+0000) 2022-01-31T22:33:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:27 smithi171 conmon[41853]: debug 2022-01-31T22:33:27.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:27.593673+0000) 2022-01-31T22:33:27.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:27 smithi167 conmon[60316]: debug 2022-01-31T22:33:27.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:27.649665+0000) 2022-01-31T22:33:27.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:27 smithi167 conmon[49112]: debug 2022-01-31T22:33:27.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:27.723596+0000) 2022-01-31T22:33:27.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:27 smithi167 conmon[54076]: debug 2022-01-31T22:33:27.890+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:27.891391+0000) 2022-01-31T22:33:28.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:28 smithi171 conmon[46715]: debug 2022-01-31T22:33:28.388+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:28.390684+0000) 2022-01-31T22:33:28.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:28 smithi171 conmon[51620]: debug 2022-01-31T22:33:28.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:28.253251+0000) 2022-01-31T22:33:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:28 smithi171 conmon[41853]: debug 2022-01-31T22:33:28.592+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:28.593844+0000) 2022-01-31T22:33:28.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:28 smithi167 conmon[49112]: debug 2022-01-31T22:33:28.722+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:28.723725+0000) 2022-01-31T22:33:28.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:28 smithi167 conmon[54076]: debug 2022-01-31T22:33:28.890+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:28.891548+0000) 2022-01-31T22:33:28.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:28 smithi167 conmon[60316]: debug 2022-01-31T22:33:28.648+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:28.649820+0000) 2022-01-31T22:33:29.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:29 smithi171 conmon[46715]: debug 2022-01-31T22:33:29.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:29.390841+0000) 2022-01-31T22:33:29.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:29 smithi171 conmon[51620]: debug 2022-01-31T22:33:29.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:29.253423+0000) 2022-01-31T22:33:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:29 smithi171 conmon[41853]: debug 2022-01-31T22:33:29.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:29.594023+0000) 2022-01-31T22:33:29.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:29 smithi167 conmon[60316]: debug 2022-01-31T22:33:29.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:29.650008+0000) 2022-01-31T22:33:29.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:29 smithi167 conmon[49112]: debug 2022-01-31T22:33:29.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:29.723888+0000) 2022-01-31T22:33:29.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:29 smithi167 conmon[54076]: debug 2022-01-31T22:33:29.891+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:29.891732+0000) 2022-01-31T22:33:30.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:30 smithi171 conmon[46715]: debug 2022-01-31T22:33:30.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:30.390973+0000) 2022-01-31T22:33:30.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:30 smithi171 conmon[51620]: debug 2022-01-31T22:33:30.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:30.253628+0000) 2022-01-31T22:33:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:30 smithi171 conmon[41853]: debug 2022-01-31T22:33:30.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:30.594196+0000) 2022-01-31T22:33:30.916 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:30 smithi167 conmon[60316]: debug 2022-01-31T22:33:30.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:30.650172+0000) 2022-01-31T22:33:30.916 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:30 smithi167 conmon[49112]: debug 2022-01-31T22:33:30.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:30.724072+0000) 2022-01-31T22:33:30.917 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:30 smithi167 conmon[54076]: debug 2022-01-31T22:33:30.891+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:30.891927+0000) 2022-01-31T22:33:31.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:31 smithi171 conmon[46715]: debug 2022-01-31T22:33:31.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.391153+0000) 2022-01-31T22:33:31.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:31 smithi171 conmon[51620]: debug 2022-01-31T22:33:31.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.253826+0000) 2022-01-31T22:33:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:31 smithi171 conmon[41853]: debug 2022-01-31T22:33:31.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.594414+0000) 2022-01-31T22:33:31.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:31 smithi167 conmon[60316]: debug 2022-01-31T22:33:31.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.650351+0000) 2022-01-31T22:33:31.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:31 smithi167 conmon[49112]: debug 2022-01-31T22:33:31.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.724258+0000) 2022-01-31T22:33:31.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:31 smithi167 conmon[54076]: debug 2022-01-31T22:33:31.890+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.892130+0000) 2022-01-31T22:33:32.253 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:32 smithi171 conmon[35325]: debug 2022-01-31T22:33:32.015+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219075 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:32.254 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:31 smithi171 conmon[41853]: debug 2022-01-31T22:33:31.993+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.995513+0000) 2022-01-31T22:33:32.255 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:31 smithi171 conmon[46715]: debug 2022-01-31T22:33:31.994+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.996045+0000) 2022-01-31T22:33:32.255 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:31 smithi171 conmon[51620]: debug 2022-01-31T22:33:31.993+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.994881+0000) 2022-01-31T22:33:32.317 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:31 smithi167 conmon[49112]: debug 2022-01-31T22:33:31.994+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.995705+0000) 2022-01-31T22:33:32.317 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:31 smithi167 conmon[54076]: debug 2022-01-31T22:33:31.994+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.996446+0000) 2022-01-31T22:33:32.318 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:31 smithi167 conmon[60316]: debug 2022-01-31T22:33:31.994+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:31.995519+0000) 2022-01-31T22:33:32.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:32 smithi171 conmon[46715]: debug 2022-01-31T22:33:32.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:32.391317+0000) 2022-01-31T22:33:32.556 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:32 smithi171 conmon[51620]: debug 2022-01-31T22:33:32.253+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:32.254031+0000) 2022-01-31T22:33:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:32 smithi171 conmon[41853]: debug 2022-01-31T22:33:32.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:32.594601+0000) 2022-01-31T22:33:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:32 smithi167 conmon[49112]: debug 2022-01-31T22:33:32.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:32.724443+0000) 2022-01-31T22:33:32.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:32 smithi167 conmon[54076]: debug 2022-01-31T22:33:32.891+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:32.892278+0000) 2022-01-31T22:33:32.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:32 smithi167 conmon[60316]: debug 2022-01-31T22:33:32.649+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:32.650544+0000) 2022-01-31T22:33:33.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:33 smithi171 conmon[46715]: debug 2022-01-31T22:33:33.389+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:33.391487+0000) 2022-01-31T22:33:33.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:33 smithi171 conmon[51620]: debug 2022-01-31T22:33:33.252+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:33.254204+0000) 2022-01-31T22:33:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:33 smithi171 conmon[41853]: debug 2022-01-31T22:33:33.593+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:33.594778+0000) 2022-01-31T22:33:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:33 smithi167 conmon[49112]: debug 2022-01-31T22:33:33.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:33.724596+0000) 2022-01-31T22:33:33.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:33 smithi167 conmon[54076]: debug 2022-01-31T22:33:33.891+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:33.892415+0000) 2022-01-31T22:33:33.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:33 smithi167 conmon[60316]: debug 2022-01-31T22:33:33.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:33.650699+0000) 2022-01-31T22:33:34.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:34 smithi171 conmon[46715]: debug 2022-01-31T22:33:34.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:34.391671+0000) 2022-01-31T22:33:34.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:34 smithi171 conmon[51620]: debug 2022-01-31T22:33:34.253+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:34.254376+0000) 2022-01-31T22:33:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:34 smithi171 conmon[41853]: debug 2022-01-31T22:33:34.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:34.594934+0000) 2022-01-31T22:33:34.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:34 smithi167 conmon[49112]: debug 2022-01-31T22:33:34.723+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:34.724775+0000) 2022-01-31T22:33:34.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:34 smithi167 conmon[54076]: debug 2022-01-31T22:33:34.892+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:34.892608+0000) 2022-01-31T22:33:34.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:34 smithi167 conmon[60316]: debug 2022-01-31T22:33:34.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:34.650908+0000) 2022-01-31T22:33:35.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:35 smithi171 conmon[46715]: debug 2022-01-31T22:33:35.390+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:35.391862+0000) 2022-01-31T22:33:35.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:35 smithi171 conmon[51620]: debug 2022-01-31T22:33:35.253+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:35.254590+0000) 2022-01-31T22:33:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:35 smithi171 conmon[41853]: debug 2022-01-31T22:33:35.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:35.595114+0000) 2022-01-31T22:33:35.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:35 smithi167 conmon[49112]: debug 2022-01-31T22:33:35.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:35.724952+0000) 2022-01-31T22:33:35.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:35 smithi167 conmon[54076]: debug 2022-01-31T22:33:35.892+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:35.892802+0000) 2022-01-31T22:33:35.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:35 smithi167 conmon[60316]: debug 2022-01-31T22:33:35.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:35.651102+0000) 2022-01-31T22:33:36.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:36 smithi171 conmon[46715]: debug 2022-01-31T22:33:36.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:36.392049+0000) 2022-01-31T22:33:36.557 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:36 smithi171 conmon[51620]: debug 2022-01-31T22:33:36.253+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:36.254748+0000) 2022-01-31T22:33:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:36 smithi171 conmon[41853]: debug 2022-01-31T22:33:36.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:36.595300+0000) 2022-01-31T22:33:36.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:36 smithi167 conmon[60316]: debug 2022-01-31T22:33:36.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:36.651286+0000) 2022-01-31T22:33:36.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:36 smithi167 conmon[49112]: debug 2022-01-31T22:33:36.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:36.725139+0000) 2022-01-31T22:33:36.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:36 smithi167 conmon[54076]: debug 2022-01-31T22:33:36.892+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:36.892964+0000) 2022-01-31T22:33:37.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[35325]: debug 2022-01-31T22:33:37.041+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219189 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:37.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[46715]: debug 2022-01-31T22:33:37.018+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.019985+0000) 2022-01-31T22:33:37.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[41853]: debug 2022-01-31T22:33:37.017+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.019390+0000) 2022-01-31T22:33:37.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[51620]: debug 2022-01-31T22:33:37.017+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.019088+0000) 2022-01-31T22:33:37.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[51620]: debug 2022-01-31T22:33:37.254+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.254921+0000) 2022-01-31T22:33:37.373 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:37 smithi167 conmon[49112]: debug 2022-01-31T22:33:37.017+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.019346+0000) 2022-01-31T22:33:37.374 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:37 smithi167 conmon[54076]: debug 2022-01-31T22:33:37.019+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.020709+0000) 2022-01-31T22:33:37.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:37 smithi167 conmon[60316]: debug 2022-01-31T22:33:37.019+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.020594+0000) 2022-01-31T22:33:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[41853]: debug 2022-01-31T22:33:37.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.595465+0000) 2022-01-31T22:33:37.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:37 smithi171 conmon[46715]: debug 2022-01-31T22:33:37.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.392218+0000) 2022-01-31T22:33:37.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:37 smithi167 conmon[60316]: debug 2022-01-31T22:33:37.650+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.651471+0000) 2022-01-31T22:33:37.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:37 smithi167 conmon[49112]: debug 2022-01-31T22:33:37.724+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.725327+0000) 2022-01-31T22:33:37.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:37 smithi167 conmon[54076]: debug 2022-01-31T22:33:37.892+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:37.893120+0000) 2022-01-31T22:33:38.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:38 smithi171 conmon[46715]: debug 2022-01-31T22:33:38.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:38.392373+0000) 2022-01-31T22:33:38.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:38 smithi171 conmon[51620]: debug 2022-01-31T22:33:38.254+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:38.255081+0000) 2022-01-31T22:33:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:38 smithi171 conmon[41853]: debug 2022-01-31T22:33:38.594+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:38.595591+0000) 2022-01-31T22:33:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:38 smithi167 conmon[49112]: debug 2022-01-31T22:33:38.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:38.725520+0000) 2022-01-31T22:33:38.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:38 smithi167 conmon[54076]: debug 2022-01-31T22:33:38.892+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:38.893230+0000) 2022-01-31T22:33:38.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:38 smithi167 conmon[60316]: debug 2022-01-31T22:33:38.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:38.651643+0000) 2022-01-31T22:33:39.255 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:38 smithi171 conmon[35325]: debug 2022-01-31T22:33:38.872+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:33:39.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:39 smithi171 conmon[46715]: debug 2022-01-31T22:33:39.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:39.392534+0000) 2022-01-31T22:33:39.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:39 smithi171 conmon[51620]: debug 2022-01-31T22:33:39.254+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:39.255268+0000) 2022-01-31T22:33:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:39 smithi171 conmon[41853]: debug 2022-01-31T22:33:39.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:39.595746+0000) 2022-01-31T22:33:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:39 smithi167 conmon[49112]: debug 2022-01-31T22:33:39.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:39.725747+0000) 2022-01-31T22:33:39.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:39 smithi167 conmon[54076]: debug 2022-01-31T22:33:39.892+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:39.893455+0000) 2022-01-31T22:33:39.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:39 smithi167 conmon[60316]: debug 2022-01-31T22:33:39.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:39.651806+0000) 2022-01-31T22:33:40.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:40 smithi171 conmon[46715]: debug 2022-01-31T22:33:40.391+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:40.392723+0000) 2022-01-31T22:33:40.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:40 smithi171 conmon[51620]: debug 2022-01-31T22:33:40.254+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:40.255472+0000) 2022-01-31T22:33:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:40 smithi171 conmon[41853]: debug 2022-01-31T22:33:40.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:40.595947+0000) 2022-01-31T22:33:40.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:40 smithi167 conmon[49112]: debug 2022-01-31T22:33:40.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:40.725936+0000) 2022-01-31T22:33:40.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:40 smithi167 conmon[54076]: debug 2022-01-31T22:33:40.893+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:40.893589+0000) 2022-01-31T22:33:40.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:40 smithi167 conmon[60316]: debug 2022-01-31T22:33:40.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:40.652001+0000) 2022-01-31T22:33:41.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:41 smithi171 conmon[46715]: debug 2022-01-31T22:33:41.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:41.392937+0000) 2022-01-31T22:33:41.558 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:41 smithi171 conmon[51620]: debug 2022-01-31T22:33:41.254+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:41.255649+0000) 2022-01-31T22:33:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:41 smithi171 conmon[41853]: debug 2022-01-31T22:33:41.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:41.596075+0000) 2022-01-31T22:33:41.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:41 smithi167 conmon[49112]: debug 2022-01-31T22:33:41.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:41.726166+0000) 2022-01-31T22:33:41.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:41 smithi167 conmon[54076]: debug 2022-01-31T22:33:41.893+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:41.893731+0000) 2022-01-31T22:33:41.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:41 smithi167 conmon[60316]: debug 2022-01-31T22:33:41.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:41.652212+0000) 2022-01-31T22:33:42.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[35325]: debug 2022-01-31T22:33:42.067+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219298 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:42.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[41853]: debug 2022-01-31T22:33:42.044+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.045282+0000) 2022-01-31T22:33:42.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[46715]: debug 2022-01-31T22:33:42.044+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.045634+0000) 2022-01-31T22:33:42.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[51620]: debug 2022-01-31T22:33:42.043+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.044707+0000) 2022-01-31T22:33:42.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[51620]: debug 2022-01-31T22:33:42.255+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.255851+0000) 2022-01-31T22:33:42.374 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:42 smithi167 conmon[49112]: debug 2022-01-31T22:33:42.044+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.045253+0000) 2022-01-31T22:33:42.375 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:42 smithi167 conmon[54076]: debug 2022-01-31T22:33:42.046+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.046901+0000) 2022-01-31T22:33:42.375 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:42 smithi167 conmon[60316]: debug 2022-01-31T22:33:42.045+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.046339+0000) 2022-01-31T22:33:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[41853]: debug 2022-01-31T22:33:42.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.596255+0000) 2022-01-31T22:33:42.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:42 smithi171 conmon[46715]: debug 2022-01-31T22:33:42.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.393158+0000) 2022-01-31T22:33:42.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:42 smithi167 conmon[49112]: debug 2022-01-31T22:33:42.725+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.726368+0000) 2022-01-31T22:33:42.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:42 smithi167 conmon[54076]: debug 2022-01-31T22:33:42.893+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.893850+0000) 2022-01-31T22:33:42.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:42 smithi167 conmon[60316]: debug 2022-01-31T22:33:42.651+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:42.652406+0000) 2022-01-31T22:33:43.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:43 smithi171 conmon[46715]: debug 2022-01-31T22:33:43.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:43.393336+0000) 2022-01-31T22:33:43.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:43 smithi171 conmon[51620]: debug 2022-01-31T22:33:43.255+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:43.255977+0000) 2022-01-31T22:33:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:43 smithi171 conmon[41853]: debug 2022-01-31T22:33:43.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:43.596439+0000) 2022-01-31T22:33:43.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:43 smithi167 conmon[49112]: debug 2022-01-31T22:33:43.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:43.726536+0000) 2022-01-31T22:33:43.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:43 smithi167 conmon[54076]: debug 2022-01-31T22:33:43.893+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:43.893996+0000) 2022-01-31T22:33:43.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:43 smithi167 conmon[60316]: debug 2022-01-31T22:33:43.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:43.652577+0000) 2022-01-31T22:33:44.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:44 smithi171 conmon[46715]: debug 2022-01-31T22:33:44.392+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:44.393509+0000) 2022-01-31T22:33:44.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:44 smithi171 conmon[51620]: debug 2022-01-31T22:33:44.255+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:44.256111+0000) 2022-01-31T22:33:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:44 smithi171 conmon[41853]: debug 2022-01-31T22:33:44.595+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:44.596614+0000) 2022-01-31T22:33:44.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:44 smithi167 conmon[60316]: debug 2022-01-31T22:33:44.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:44.652811+0000) 2022-01-31T22:33:44.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:44 smithi167 conmon[49112]: debug 2022-01-31T22:33:44.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:44.726714+0000) 2022-01-31T22:33:45.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:44 smithi167 conmon[54076]: debug 2022-01-31T22:33:44.893+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:44.894205+0000) 2022-01-31T22:33:45.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:45 smithi171 conmon[46715]: debug 2022-01-31T22:33:45.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:45.393683+0000) 2022-01-31T22:33:45.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:45 smithi171 conmon[51620]: debug 2022-01-31T22:33:45.255+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:45.256330+0000) 2022-01-31T22:33:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:45 smithi171 conmon[41853]: debug 2022-01-31T22:33:45.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:45.596779+0000) 2022-01-31T22:33:45.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:45 smithi167 conmon[49112]: debug 2022-01-31T22:33:45.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:45.726901+0000) 2022-01-31T22:33:45.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:45 smithi167 conmon[60316]: debug 2022-01-31T22:33:45.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:45.652995+0000) 2022-01-31T22:33:46.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:45 smithi167 conmon[54076]: debug 2022-01-31T22:33:45.893+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:45.894387+0000) 2022-01-31T22:33:46.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:46 smithi171 conmon[46715]: debug 2022-01-31T22:33:46.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:46.393845+0000) 2022-01-31T22:33:46.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:46 smithi171 conmon[51620]: debug 2022-01-31T22:33:46.255+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:46.256535+0000) 2022-01-31T22:33:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:46 smithi171 conmon[41853]: debug 2022-01-31T22:33:46.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:46.596954+0000) 2022-01-31T22:33:46.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:46 smithi167 conmon[49112]: debug 2022-01-31T22:33:46.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:46.727070+0000) 2022-01-31T22:33:46.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:46 smithi167 conmon[60316]: debug 2022-01-31T22:33:46.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:46.653147+0000) 2022-01-31T22:33:47.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:47 smithi167 conmon[49112]: debug 2022-01-31T22:33:47.070+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.070786+0000) 2022-01-31T22:33:47.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:47 smithi167 conmon[60316]: debug 2022-01-31T22:33:47.070+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.070903+0000) 2022-01-31T22:33:47.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:46 smithi167 conmon[54076]: debug 2022-01-31T22:33:46.894+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:46.894553+0000) 2022-01-31T22:33:47.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:47 smithi167 conmon[54076]: debug 2022-01-31T22:33:47.069+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.070342+0000) 2022-01-31T22:33:47.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[35325]: debug 2022-01-31T22:33:47.092+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219411 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:47.358 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[41853]: debug 2022-01-31T22:33:47.070+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.070937+0000) 2022-01-31T22:33:47.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[46715]: debug 2022-01-31T22:33:47.068+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.069627+0000) 2022-01-31T22:33:47.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[51620]: debug 2022-01-31T22:33:47.069+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.070252+0000) 2022-01-31T22:33:47.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[51620]: debug 2022-01-31T22:33:47.256+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.256733+0000) 2022-01-31T22:33:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[41853]: debug 2022-01-31T22:33:47.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.597128+0000) 2022-01-31T22:33:47.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:47 smithi171 conmon[46715]: debug 2022-01-31T22:33:47.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.393989+0000) 2022-01-31T22:33:47.893 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:47 smithi167 conmon[60316]: debug 2022-01-31T22:33:47.652+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.653247+0000) 2022-01-31T22:33:47.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:47 smithi167 conmon[49112]: debug 2022-01-31T22:33:47.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.727227+0000) 2022-01-31T22:33:48.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:47 smithi167 conmon[54076]: debug 2022-01-31T22:33:47.894+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:47.894679+0000) 2022-01-31T22:33:48.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:48 smithi171 conmon[46715]: debug 2022-01-31T22:33:48.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:48.394127+0000) 2022-01-31T22:33:48.559 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:48 smithi171 conmon[51620]: debug 2022-01-31T22:33:48.256+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:48.256917+0000) 2022-01-31T22:33:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:48 smithi171 conmon[41853]: debug 2022-01-31T22:33:48.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:48.597297+0000) 2022-01-31T22:33:48.894 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:48 smithi167 conmon[60316]: debug 2022-01-31T22:33:48.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:48.653389+0000) 2022-01-31T22:33:48.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:48 smithi167 conmon[49112]: debug 2022-01-31T22:33:48.726+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:48.727447+0000) 2022-01-31T22:33:49.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:48 smithi167 conmon[54076]: debug 2022-01-31T22:33:48.894+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:48.894832+0000) 2022-01-31T22:33:49.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:49 smithi171 conmon[46715]: debug 2022-01-31T22:33:49.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:49.394337+0000) 2022-01-31T22:33:49.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:49 smithi171 conmon[51620]: debug 2022-01-31T22:33:49.256+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:49.257116+0000) 2022-01-31T22:33:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:49 smithi171 conmon[41853]: debug 2022-01-31T22:33:49.596+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:49.597493+0000) 2022-01-31T22:33:49.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:49 smithi167 conmon[49112]: debug 2022-01-31T22:33:49.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:49.727633+0000) 2022-01-31T22:33:49.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:49 smithi167 conmon[60316]: debug 2022-01-31T22:33:49.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:49.653572+0000) 2022-01-31T22:33:50.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:49 smithi167 conmon[54076]: debug 2022-01-31T22:33:49.894+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:49.894955+0000) 2022-01-31T22:33:50.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:50 smithi171 conmon[46715]: debug 2022-01-31T22:33:50.393+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:50.394531+0000) 2022-01-31T22:33:50.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:50 smithi171 conmon[51620]: debug 2022-01-31T22:33:50.256+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:50.257287+0000) 2022-01-31T22:33:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:50 smithi171 conmon[41853]: debug 2022-01-31T22:33:50.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:50.597674+0000) 2022-01-31T22:33:50.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:50 smithi167 conmon[49112]: debug 2022-01-31T22:33:50.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:50.727760+0000) 2022-01-31T22:33:50.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:50 smithi167 conmon[60316]: debug 2022-01-31T22:33:50.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:50.653754+0000) 2022-01-31T22:33:51.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:50 smithi167 conmon[54076]: debug 2022-01-31T22:33:50.894+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:50.895142+0000) 2022-01-31T22:33:51.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:51 smithi171 conmon[46715]: debug 2022-01-31T22:33:51.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:51.394737+0000) 2022-01-31T22:33:51.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:51 smithi171 conmon[51620]: debug 2022-01-31T22:33:51.256+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:51.257497+0000) 2022-01-31T22:33:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:51 smithi171 conmon[41853]: debug 2022-01-31T22:33:51.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:51.597860+0000) 2022-01-31T22:33:51.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:51 smithi167 conmon[49112]: debug 2022-01-31T22:33:51.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:51.727911+0000) 2022-01-31T22:33:51.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:51 smithi167 conmon[60316]: debug 2022-01-31T22:33:51.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:51.653961+0000) 2022-01-31T22:33:52.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:52 smithi167 conmon[49112]: debug 2022-01-31T22:33:52.095+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.095966+0000) 2022-01-31T22:33:52.169 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:52 smithi167 conmon[60316]: debug 2022-01-31T22:33:52.095+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.096313+0000) 2022-01-31T22:33:52.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:51 smithi167 conmon[54076]: debug 2022-01-31T22:33:51.895+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:51.895342+0000) 2022-01-31T22:33:52.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:52 smithi167 conmon[54076]: debug 2022-01-31T22:33:52.095+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.095565+0000) 2022-01-31T22:33:52.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[35325]: debug 2022-01-31T22:33:52.117+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219520 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:52.357 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[41853]: debug 2022-01-31T22:33:52.095+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.096087+0000) 2022-01-31T22:33:52.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[46715]: debug 2022-01-31T22:33:52.094+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.095442+0000) 2022-01-31T22:33:52.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[51620]: debug 2022-01-31T22:33:52.094+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.095062+0000) 2022-01-31T22:33:52.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[51620]: debug 2022-01-31T22:33:52.257+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.257692+0000) 2022-01-31T22:33:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[41853]: debug 2022-01-31T22:33:52.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.598053+0000) 2022-01-31T22:33:52.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:52 smithi171 conmon[46715]: debug 2022-01-31T22:33:52.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.394962+0000) 2022-01-31T22:33:52.894 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:52 smithi167 conmon[49112]: debug 2022-01-31T22:33:52.727+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.728093+0000) 2022-01-31T22:33:52.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:52 smithi167 conmon[60316]: debug 2022-01-31T22:33:52.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.654141+0000) 2022-01-31T22:33:53.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:52 smithi167 conmon[54076]: debug 2022-01-31T22:33:52.895+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:52.895514+0000) 2022-01-31T22:33:53.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:53 smithi171 conmon[46715]: debug 2022-01-31T22:33:53.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:53.395116+0000) 2022-01-31T22:33:53.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:53 smithi171 conmon[51620]: debug 2022-01-31T22:33:53.257+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:53.257867+0000) 2022-01-31T22:33:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:53 smithi171 conmon[41853]: debug 2022-01-31T22:33:53.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:53.598241+0000) 2022-01-31T22:33:53.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:53 smithi167 conmon[49112]: debug 2022-01-31T22:33:53.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:53.728250+0000) 2022-01-31T22:33:53.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:53 smithi167 conmon[60316]: debug 2022-01-31T22:33:53.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:53.654254+0000) 2022-01-31T22:33:54.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:53 smithi167 conmon[54076]: debug 2022-01-31T22:33:53.895+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:53.895598+0000) 2022-01-31T22:33:54.257 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:53 smithi171 conmon[35325]: debug 2022-01-31T22:33:53.873+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:33:54.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:54 smithi171 conmon[46715]: debug 2022-01-31T22:33:54.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:54.395319+0000) 2022-01-31T22:33:54.560 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:54 smithi171 conmon[51620]: debug 2022-01-31T22:33:54.257+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:54.258069+0000) 2022-01-31T22:33:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:54 smithi171 conmon[41853]: debug 2022-01-31T22:33:54.597+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:54.598395+0000) 2022-01-31T22:33:54.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:54 smithi167 conmon[49112]: debug 2022-01-31T22:33:54.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:54.728427+0000) 2022-01-31T22:33:54.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:54 smithi167 conmon[60316]: debug 2022-01-31T22:33:54.653+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:54.654441+0000) 2022-01-31T22:33:55.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:54 smithi167 conmon[54076]: debug 2022-01-31T22:33:54.895+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:54.895797+0000) 2022-01-31T22:33:55.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:55 smithi171 conmon[46715]: debug 2022-01-31T22:33:55.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:55.395508+0000) 2022-01-31T22:33:55.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:55 smithi171 conmon[51620]: debug 2022-01-31T22:33:55.257+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:55.258268+0000) 2022-01-31T22:33:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:55 smithi171 conmon[41853]: debug 2022-01-31T22:33:55.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:55.598570+0000) 2022-01-31T22:33:55.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:55 smithi167 conmon[49112]: debug 2022-01-31T22:33:55.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:55.728611+0000) 2022-01-31T22:33:55.895 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:55 smithi167 conmon[60316]: debug 2022-01-31T22:33:55.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:55.654674+0000) 2022-01-31T22:33:56.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:55 smithi167 conmon[54076]: debug 2022-01-31T22:33:55.895+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:55.896023+0000) 2022-01-31T22:33:56.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:56 smithi171 conmon[46715]: debug 2022-01-31T22:33:56.394+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:56.395712+0000) 2022-01-31T22:33:56.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:56 smithi171 conmon[51620]: debug 2022-01-31T22:33:56.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:56.258463+0000) 2022-01-31T22:33:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:56 smithi171 conmon[41853]: debug 2022-01-31T22:33:56.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:56.598772+0000) 2022-01-31T22:33:56.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:56 smithi167 conmon[49112]: debug 2022-01-31T22:33:56.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:56.728797+0000) 2022-01-31T22:33:56.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:56 smithi167 conmon[60316]: debug 2022-01-31T22:33:56.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:56.654867+0000) 2022-01-31T22:33:57.169 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:57 smithi167 conmon[49112]: debug 2022-01-31T22:33:57.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.120208+0000) 2022-01-31T22:33:57.169 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:56 smithi167 conmon[54076]: debug 2022-01-31T22:33:56.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:56.896203+0000) 2022-01-31T22:33:57.170 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:57 smithi167 conmon[54076]: debug 2022-01-31T22:33:57.119+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.120341+0000) 2022-01-31T22:33:57.171 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:57 smithi167 conmon[60316]: debug 2022-01-31T22:33:57.121+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.121826+0000) 2022-01-31T22:33:57.370 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[35325]: debug 2022-01-31T22:33:57.142+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219634 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:33:57.371 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[41853]: debug 2022-01-31T22:33:57.120+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.121500+0000) 2022-01-31T22:33:57.371 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[46715]: debug 2022-01-31T22:33:57.119+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.120534+0000) 2022-01-31T22:33:57.372 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[51620]: debug 2022-01-31T22:33:57.120+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.121053+0000) 2022-01-31T22:33:57.372 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[51620]: debug 2022-01-31T22:33:57.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.258622+0000) 2022-01-31T22:33:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[41853]: debug 2022-01-31T22:33:57.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.598965+0000) 2022-01-31T22:33:57.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:57 smithi171 conmon[46715]: debug 2022-01-31T22:33:57.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.395859+0000) 2022-01-31T22:33:57.895 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:57 smithi167 conmon[49112]: debug 2022-01-31T22:33:57.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.728985+0000) 2022-01-31T22:33:57.896 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:57 smithi167 conmon[60316]: debug 2022-01-31T22:33:57.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.655075+0000) 2022-01-31T22:33:58.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:57 smithi167 conmon[54076]: debug 2022-01-31T22:33:57.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:57.896372+0000) 2022-01-31T22:33:58.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:58 smithi171 conmon[46715]: debug 2022-01-31T22:33:58.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:58.395967+0000) 2022-01-31T22:33:58.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:58 smithi171 conmon[51620]: debug 2022-01-31T22:33:58.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:58.258802+0000) 2022-01-31T22:33:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:58 smithi171 conmon[41853]: debug 2022-01-31T22:33:58.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:58.599094+0000) 2022-01-31T22:33:58.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:58 smithi167 conmon[49112]: debug 2022-01-31T22:33:58.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:58.729131+0000) 2022-01-31T22:33:58.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:58 smithi167 conmon[54076]: debug 2022-01-31T22:33:58.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:58.896546+0000) 2022-01-31T22:33:58.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:58 smithi167 conmon[60316]: debug 2022-01-31T22:33:58.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:58.655232+0000) 2022-01-31T22:33:59.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:33:59 smithi171 conmon[46715]: debug 2022-01-31T22:33:59.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:59.396122+0000) 2022-01-31T22:33:59.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:33:59 smithi171 conmon[51620]: debug 2022-01-31T22:33:59.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:59.258943+0000) 2022-01-31T22:33:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:33:59 smithi171 conmon[41853]: debug 2022-01-31T22:33:59.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:59.599282+0000) 2022-01-31T22:33:59.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:33:59 smithi167 conmon[54076]: debug 2022-01-31T22:33:59.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:59.896750+0000) 2022-01-31T22:33:59.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:33:59 smithi167 conmon[49112]: debug 2022-01-31T22:33:59.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:59.729321+0000) 2022-01-31T22:33:59.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:33:59 smithi167 conmon[60316]: debug 2022-01-31T22:33:59.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:33:59.655444+0000) 2022-01-31T22:34:00.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:00 smithi171 conmon[46715]: debug 2022-01-31T22:34:00.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:00.396269+0000) 2022-01-31T22:34:00.561 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:00 smithi171 conmon[51620]: debug 2022-01-31T22:34:00.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:00.259077+0000) 2022-01-31T22:34:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:00 smithi171 conmon[41853]: debug 2022-01-31T22:34:00.598+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:00.599473+0000) 2022-01-31T22:34:00.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:00 smithi167 conmon[49112]: debug 2022-01-31T22:34:00.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:00.729537+0000) 2022-01-31T22:34:00.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:00 smithi167 conmon[54076]: debug 2022-01-31T22:34:00.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:00.896951+0000) 2022-01-31T22:34:00.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:00 smithi167 conmon[60316]: debug 2022-01-31T22:34:00.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:00.655610+0000) 2022-01-31T22:34:01.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:01 smithi171 conmon[46715]: debug 2022-01-31T22:34:01.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:01.396455+0000) 2022-01-31T22:34:01.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:01 smithi171 conmon[51620]: debug 2022-01-31T22:34:01.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:01.259173+0000) 2022-01-31T22:34:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:01 smithi171 conmon[41853]: debug 2022-01-31T22:34:01.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:01.599663+0000) 2022-01-31T22:34:01.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:01 smithi167 conmon[49112]: debug 2022-01-31T22:34:01.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:01.729724+0000) 2022-01-31T22:34:01.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:01 smithi167 conmon[54076]: debug 2022-01-31T22:34:01.895+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:01.897161+0000) 2022-01-31T22:34:01.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:01 smithi167 conmon[60316]: debug 2022-01-31T22:34:01.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:01.655818+0000) 2022-01-31T22:34:02.249 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:02 smithi167 conmon[49112]: debug 2022-01-31T22:34:02.144+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.145552+0000) 2022-01-31T22:34:02.250 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:02 smithi167 conmon[54076]: debug 2022-01-31T22:34:02.145+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.146765+0000) 2022-01-31T22:34:02.251 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:02 smithi167 conmon[60316]: debug 2022-01-31T22:34:02.145+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.146965+0000) 2022-01-31T22:34:02.396 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[35325]: debug 2022-01-31T22:34:02.173+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219744 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:02.396 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[41853]: debug 2022-01-31T22:34:02.146+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.147155+0000) 2022-01-31T22:34:02.397 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[46715]: debug 2022-01-31T22:34:02.144+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.145370+0000) 2022-01-31T22:34:02.398 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[51620]: debug 2022-01-31T22:34:02.145+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.146187+0000) 2022-01-31T22:34:02.398 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[51620]: debug 2022-01-31T22:34:02.259+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.259328+0000) 2022-01-31T22:34:02.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[46715]: debug 2022-01-31T22:34:02.395+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.396661+0000) 2022-01-31T22:34:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:02 smithi171 conmon[41853]: debug 2022-01-31T22:34:02.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.599860+0000) 2022-01-31T22:34:02.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:02 smithi167 conmon[49112]: debug 2022-01-31T22:34:02.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.729895+0000) 2022-01-31T22:34:02.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:02 smithi167 conmon[60316]: debug 2022-01-31T22:34:02.654+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.656034+0000) 2022-01-31T22:34:02.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:02 smithi167 conmon[54076]: debug 2022-01-31T22:34:02.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:02.897346+0000) 2022-01-31T22:34:03.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:03 smithi171 conmon[46715]: debug 2022-01-31T22:34:03.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:03.396833+0000) 2022-01-31T22:34:03.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:03 smithi171 conmon[51620]: debug 2022-01-31T22:34:03.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:03.259437+0000) 2022-01-31T22:34:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:03 smithi171 conmon[41853]: debug 2022-01-31T22:34:03.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:03.600019+0000) 2022-01-31T22:34:03.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:03 smithi167 conmon[49112]: debug 2022-01-31T22:34:03.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:03.730051+0000) 2022-01-31T22:34:03.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:03 smithi167 conmon[54076]: debug 2022-01-31T22:34:03.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:03.897537+0000) 2022-01-31T22:34:03.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:03 smithi167 conmon[60316]: debug 2022-01-31T22:34:03.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:03.656190+0000) 2022-01-31T22:34:04.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:04 smithi171 conmon[46715]: debug 2022-01-31T22:34:04.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:04.397022+0000) 2022-01-31T22:34:04.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:04 smithi171 conmon[51620]: debug 2022-01-31T22:34:04.258+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:04.259629+0000) 2022-01-31T22:34:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:04 smithi171 conmon[41853]: debug 2022-01-31T22:34:04.599+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:04.600171+0000) 2022-01-31T22:34:04.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:04 smithi167 conmon[60316]: debug 2022-01-31T22:34:04.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:04.656393+0000) 2022-01-31T22:34:04.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:04 smithi167 conmon[49112]: debug 2022-01-31T22:34:04.728+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:04.730251+0000) 2022-01-31T22:34:04.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:04 smithi167 conmon[54076]: debug 2022-01-31T22:34:04.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:04.897725+0000) 2022-01-31T22:34:05.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:05 smithi171 conmon[46715]: debug 2022-01-31T22:34:05.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:05.397152+0000) 2022-01-31T22:34:05.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:05 smithi171 conmon[51620]: debug 2022-01-31T22:34:05.259+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:05.259813+0000) 2022-01-31T22:34:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:05 smithi171 conmon[41853]: debug 2022-01-31T22:34:05.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:05.600320+0000) 2022-01-31T22:34:05.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:05 smithi167 conmon[49112]: debug 2022-01-31T22:34:05.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:05.730439+0000) 2022-01-31T22:34:05.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:05 smithi167 conmon[54076]: debug 2022-01-31T22:34:05.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:05.897858+0000) 2022-01-31T22:34:05.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:05 smithi167 conmon[60316]: debug 2022-01-31T22:34:05.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:05.656580+0000) 2022-01-31T22:34:06.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:06 smithi171 conmon[46715]: debug 2022-01-31T22:34:06.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:06.397348+0000) 2022-01-31T22:34:06.562 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:06 smithi171 conmon[51620]: debug 2022-01-31T22:34:06.259+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:06.260004+0000) 2022-01-31T22:34:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:06 smithi171 conmon[41853]: debug 2022-01-31T22:34:06.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:06.600479+0000) 2022-01-31T22:34:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:06 smithi167 conmon[49112]: debug 2022-01-31T22:34:06.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:06.730619+0000) 2022-01-31T22:34:06.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:06 smithi167 conmon[60316]: debug 2022-01-31T22:34:06.655+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:06.656775+0000) 2022-01-31T22:34:06.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:06 smithi167 conmon[54076]: debug 2022-01-31T22:34:06.897+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:06.898037+0000) 2022-01-31T22:34:07.265 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:07 smithi167 conmon[49112]: debug 2022-01-31T22:34:07.176+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.177991+0000) 2022-01-31T22:34:07.266 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:07 smithi167 conmon[54076]: debug 2022-01-31T22:34:07.175+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.177067+0000) 2022-01-31T22:34:07.267 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:07 smithi167 conmon[60316]: debug 2022-01-31T22:34:07.176+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.178334+0000) 2022-01-31T22:34:07.562 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[35325]: debug 2022-01-31T22:34:07.199+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219858 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:07.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[41853]: debug 2022-01-31T22:34:07.176+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.177720+0000) 2022-01-31T22:34:07.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[46715]: debug 2022-01-31T22:34:07.175+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.176276+0000) 2022-01-31T22:34:07.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[46715]: debug 2022-01-31T22:34:07.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.397527+0000) 2022-01-31T22:34:07.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[51620]: debug 2022-01-31T22:34:07.177+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.178425+0000) 2022-01-31T22:34:07.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[51620]: debug 2022-01-31T22:34:07.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.260191+0000) 2022-01-31T22:34:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:07 smithi171 conmon[41853]: debug 2022-01-31T22:34:07.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.600632+0000) 2022-01-31T22:34:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:07 smithi167 conmon[49112]: debug 2022-01-31T22:34:07.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.730781+0000) 2022-01-31T22:34:07.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:07 smithi167 conmon[54076]: debug 2022-01-31T22:34:07.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.898220+0000) 2022-01-31T22:34:07.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:07 smithi167 conmon[60316]: debug 2022-01-31T22:34:07.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:07.656936+0000) 2022-01-31T22:34:08.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:08 smithi171 conmon[46715]: debug 2022-01-31T22:34:08.396+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:08.397653+0000) 2022-01-31T22:34:08.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:08 smithi171 conmon[51620]: debug 2022-01-31T22:34:08.259+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:08.260346+0000) 2022-01-31T22:34:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:08 smithi171 conmon[41853]: debug 2022-01-31T22:34:08.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:08.600821+0000) 2022-01-31T22:34:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:08 smithi167 conmon[49112]: debug 2022-01-31T22:34:08.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:08.730953+0000) 2022-01-31T22:34:08.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:08 smithi167 conmon[54076]: debug 2022-01-31T22:34:08.896+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:08.898402+0000) 2022-01-31T22:34:08.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:08 smithi167 conmon[60316]: debug 2022-01-31T22:34:08.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:08.657090+0000) 2022-01-31T22:34:09.260 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:08 smithi171 conmon[35325]: debug 2022-01-31T22:34:08.874+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:34:09.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:09 smithi171 conmon[46715]: debug 2022-01-31T22:34:09.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:09.397760+0000) 2022-01-31T22:34:09.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:09 smithi171 conmon[51620]: debug 2022-01-31T22:34:09.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:09.260505+0000) 2022-01-31T22:34:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:09 smithi171 conmon[41853]: debug 2022-01-31T22:34:09.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:09.601022+0000) 2022-01-31T22:34:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:09 smithi167 conmon[49112]: debug 2022-01-31T22:34:09.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:09.731126+0000) 2022-01-31T22:34:09.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:09 smithi167 conmon[54076]: debug 2022-01-31T22:34:09.897+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:09.898554+0000) 2022-01-31T22:34:09.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:09 smithi167 conmon[60316]: debug 2022-01-31T22:34:09.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:09.657280+0000) 2022-01-31T22:34:10.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:10 smithi171 conmon[46715]: debug 2022-01-31T22:34:10.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:10.397919+0000) 2022-01-31T22:34:10.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:10 smithi171 conmon[51620]: debug 2022-01-31T22:34:10.259+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:10.260694+0000) 2022-01-31T22:34:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:10 smithi171 conmon[41853]: debug 2022-01-31T22:34:10.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:10.601250+0000) 2022-01-31T22:34:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:10 smithi167 conmon[49112]: debug 2022-01-31T22:34:10.729+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:10.731318+0000) 2022-01-31T22:34:10.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:10 smithi167 conmon[54076]: debug 2022-01-31T22:34:10.897+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:10.898727+0000) 2022-01-31T22:34:10.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:10 smithi167 conmon[60316]: debug 2022-01-31T22:34:10.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:10.657477+0000) 2022-01-31T22:34:11.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:11 smithi171 conmon[51620]: debug 2022-01-31T22:34:11.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:11.260860+0000) 2022-01-31T22:34:11.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:11 smithi171 conmon[46715]: debug 2022-01-31T22:34:11.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:11.398129+0000) 2022-01-31T22:34:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:11 smithi171 conmon[41853]: debug 2022-01-31T22:34:11.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:11.601460+0000) 2022-01-31T22:34:11.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:11 smithi167 conmon[49112]: debug 2022-01-31T22:34:11.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:11.731474+0000) 2022-01-31T22:34:11.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:11 smithi167 conmon[54076]: debug 2022-01-31T22:34:11.898+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:11.898926+0000) 2022-01-31T22:34:11.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:11 smithi167 conmon[60316]: debug 2022-01-31T22:34:11.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:11.657649+0000) 2022-01-31T22:34:12.489 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:12 smithi167 conmon[49112]: debug 2022-01-31T22:34:12.202+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.204057+0000) 2022-01-31T22:34:12.490 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:12 smithi167 conmon[54076]: debug 2022-01-31T22:34:12.201+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.203165+0000) 2022-01-31T22:34:12.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:12 smithi167 conmon[60316]: debug 2022-01-31T22:34:12.202+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.203645+0000) 2022-01-31T22:34:12.563 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[35325]: debug 2022-01-31T22:34:12.225+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 219968 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:12.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[41853]: debug 2022-01-31T22:34:12.202+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.203148+0000) 2022-01-31T22:34:12.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[46715]: debug 2022-01-31T22:34:12.201+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.202544+0000) 2022-01-31T22:34:12.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[46715]: debug 2022-01-31T22:34:12.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.398251+0000) 2022-01-31T22:34:12.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[51620]: debug 2022-01-31T22:34:12.203+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.204153+0000) 2022-01-31T22:34:12.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[51620]: debug 2022-01-31T22:34:12.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.261042+0000) 2022-01-31T22:34:12.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:12 smithi171 conmon[41853]: debug 2022-01-31T22:34:12.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.601674+0000) 2022-01-31T22:34:12.898 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:12 smithi167 conmon[60316]: debug 2022-01-31T22:34:12.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.657819+0000) 2022-01-31T22:34:12.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:12 smithi167 conmon[49112]: debug 2022-01-31T22:34:12.730+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.731658+0000) 2022-01-31T22:34:13.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:12 smithi167 conmon[54076]: debug 2022-01-31T22:34:12.897+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:12.899102+0000) 2022-01-31T22:34:13.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:13 smithi171 conmon[46715]: debug 2022-01-31T22:34:13.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:13.398421+0000) 2022-01-31T22:34:13.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:13 smithi171 conmon[51620]: debug 2022-01-31T22:34:13.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:13.261214+0000) 2022-01-31T22:34:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:13 smithi171 conmon[41853]: debug 2022-01-31T22:34:13.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:13.601825+0000) 2022-01-31T22:34:13.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:13 smithi167 conmon[54076]: debug 2022-01-31T22:34:13.897+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:13.899255+0000) 2022-01-31T22:34:13.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:13 smithi167 conmon[49112]: debug 2022-01-31T22:34:13.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:13.731785+0000) 2022-01-31T22:34:13.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:13 smithi167 conmon[60316]: debug 2022-01-31T22:34:13.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:13.657965+0000) 2022-01-31T22:34:14.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:14 smithi171 conmon[51620]: debug 2022-01-31T22:34:14.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:14.261403+0000) 2022-01-31T22:34:14.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:14 smithi171 conmon[46715]: debug 2022-01-31T22:34:14.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:14.398615+0000) 2022-01-31T22:34:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:14 smithi171 conmon[41853]: debug 2022-01-31T22:34:14.600+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:14.601998+0000) 2022-01-31T22:34:14.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:14 smithi167 conmon[54076]: debug 2022-01-31T22:34:14.898+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:14.899459+0000) 2022-01-31T22:34:14.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:14 smithi167 conmon[60316]: debug 2022-01-31T22:34:14.656+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:14.658194+0000) 2022-01-31T22:34:14.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:14 smithi167 conmon[49112]: debug 2022-01-31T22:34:14.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:14.731883+0000) 2022-01-31T22:34:15.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:15 smithi171 conmon[46715]: debug 2022-01-31T22:34:15.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:15.398810+0000) 2022-01-31T22:34:15.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:15 smithi171 conmon[51620]: debug 2022-01-31T22:34:15.259+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:15.261605+0000) 2022-01-31T22:34:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:15 smithi171 conmon[41853]: debug 2022-01-31T22:34:15.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:15.602187+0000) 2022-01-31T22:34:15.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:15 smithi167 conmon[54076]: debug 2022-01-31T22:34:15.898+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:15.899670+0000) 2022-01-31T22:34:15.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:15 smithi167 conmon[60316]: debug 2022-01-31T22:34:15.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:15.658384+0000) 2022-01-31T22:34:15.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:15 smithi167 conmon[49112]: debug 2022-01-31T22:34:15.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:15.732043+0000) 2022-01-31T22:34:16.563 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:16 smithi171 conmon[51620]: debug 2022-01-31T22:34:16.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:16.261802+0000) 2022-01-31T22:34:16.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:16 smithi171 conmon[46715]: debug 2022-01-31T22:34:16.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:16.398995+0000) 2022-01-31T22:34:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:16 smithi171 conmon[41853]: debug 2022-01-31T22:34:16.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:16.602369+0000) 2022-01-31T22:34:16.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:16 smithi167 conmon[54076]: debug 2022-01-31T22:34:16.899+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:16.899837+0000) 2022-01-31T22:34:16.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:16 smithi167 conmon[49112]: debug 2022-01-31T22:34:16.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:16.732231+0000) 2022-01-31T22:34:16.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:16 smithi167 conmon[60316]: debug 2022-01-31T22:34:16.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:16.658567+0000) 2022-01-31T22:34:17.490 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:17 smithi167 conmon[49112]: debug 2022-01-31T22:34:17.227+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.229370+0000) 2022-01-31T22:34:17.491 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:17 smithi167 conmon[54076]: debug 2022-01-31T22:34:17.228+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.230039+0000) 2022-01-31T22:34:17.491 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:17 smithi167 conmon[60316]: debug 2022-01-31T22:34:17.228+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.229884+0000) 2022-01-31T22:34:17.563 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[35325]: debug 2022-01-31T22:34:17.250+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220082 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:17.564 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[41853]: debug 2022-01-31T22:34:17.228+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.230201+0000) 2022-01-31T22:34:17.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[46715]: debug 2022-01-31T22:34:17.227+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.228960+0000) 2022-01-31T22:34:17.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[46715]: debug 2022-01-31T22:34:17.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.399153+0000) 2022-01-31T22:34:17.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[51620]: debug 2022-01-31T22:34:17.228+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.230692+0000) 2022-01-31T22:34:17.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[51620]: debug 2022-01-31T22:34:17.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.261896+0000) 2022-01-31T22:34:17.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:17 smithi171 conmon[41853]: debug 2022-01-31T22:34:17.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.602552+0000) 2022-01-31T22:34:17.899 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:17 smithi167 conmon[49112]: debug 2022-01-31T22:34:17.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.732416+0000) 2022-01-31T22:34:17.899 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:17 smithi167 conmon[60316]: debug 2022-01-31T22:34:17.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.658712+0000) 2022-01-31T22:34:18.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:17 smithi167 conmon[54076]: debug 2022-01-31T22:34:17.899+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:17.900025+0000) 2022-01-31T22:34:18.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:18 smithi171 conmon[46715]: debug 2022-01-31T22:34:18.397+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:18.399310+0000) 2022-01-31T22:34:18.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:18 smithi171 conmon[51620]: debug 2022-01-31T22:34:18.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:18.261997+0000) 2022-01-31T22:34:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:18 smithi171 conmon[41853]: debug 2022-01-31T22:34:18.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:18.602719+0000) 2022-01-31T22:34:18.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:18 smithi167 conmon[54076]: debug 2022-01-31T22:34:18.898+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:18.900197+0000) 2022-01-31T22:34:18.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:18 smithi167 conmon[60316]: debug 2022-01-31T22:34:18.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:18.658863+0000) 2022-01-31T22:34:18.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:18 smithi167 conmon[49112]: debug 2022-01-31T22:34:18.731+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:18.732578+0000) 2022-01-31T22:34:19.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:19 smithi171 conmon[46715]: debug 2022-01-31T22:34:19.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:19.399494+0000) 2022-01-31T22:34:19.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:19 smithi171 conmon[51620]: debug 2022-01-31T22:34:19.261+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:19.262115+0000) 2022-01-31T22:34:19.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:19 smithi171 conmon[41853]: debug 2022-01-31T22:34:19.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:19.602929+0000) 2022-01-31T22:34:19.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:19 smithi167 conmon[49112]: debug 2022-01-31T22:34:19.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:19.732755+0000) 2022-01-31T22:34:19.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:19 smithi167 conmon[54076]: debug 2022-01-31T22:34:19.899+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:19.900359+0000) 2022-01-31T22:34:19.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:19 smithi167 conmon[60316]: debug 2022-01-31T22:34:19.658+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:19.659057+0000) 2022-01-31T22:34:20.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:20 smithi171 conmon[46715]: debug 2022-01-31T22:34:20.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:20.399651+0000) 2022-01-31T22:34:20.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:20 smithi171 conmon[51620]: debug 2022-01-31T22:34:20.261+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:20.262314+0000) 2022-01-31T22:34:20.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:20 smithi171 conmon[41853]: debug 2022-01-31T22:34:20.601+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:20.603090+0000) 2022-01-31T22:34:20.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:20 smithi167 conmon[49112]: debug 2022-01-31T22:34:20.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:20.732916+0000) 2022-01-31T22:34:20.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:20 smithi167 conmon[54076]: debug 2022-01-31T22:34:20.899+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:20.900517+0000) 2022-01-31T22:34:20.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:20 smithi167 conmon[60316]: debug 2022-01-31T22:34:20.658+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:20.659223+0000) 2022-01-31T22:34:21.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:21 smithi171 conmon[46715]: debug 2022-01-31T22:34:21.398+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:21.399809+0000) 2022-01-31T22:34:21.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:21 smithi171 conmon[51620]: debug 2022-01-31T22:34:21.261+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:21.262506+0000) 2022-01-31T22:34:21.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:21 smithi171 conmon[41853]: debug 2022-01-31T22:34:21.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:21.603306+0000) 2022-01-31T22:34:21.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:21 smithi167 conmon[49112]: debug 2022-01-31T22:34:21.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:21.733074+0000) 2022-01-31T22:34:21.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:21 smithi167 conmon[54076]: debug 2022-01-31T22:34:21.900+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:21.900720+0000) 2022-01-31T22:34:21.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:21 smithi167 conmon[60316]: debug 2022-01-31T22:34:21.657+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:21.659422+0000) 2022-01-31T22:34:22.564 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[35325]: debug 2022-01-31T22:34:22.275+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:22.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[41853]: debug 2022-01-31T22:34:22.253+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.255137+0000) 2022-01-31T22:34:22.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[46715]: debug 2022-01-31T22:34:22.251+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.253658+0000) 2022-01-31T22:34:22.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[46715]: debug 2022-01-31T22:34:22.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.400000+0000) 2022-01-31T22:34:22.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[51620]: debug 2022-01-31T22:34:22.253+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.254906+0000) 2022-01-31T22:34:22.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[51620]: debug 2022-01-31T22:34:22.260+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.262668+0000) 2022-01-31T22:34:22.659 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:22 smithi167 conmon[49112]: debug 2022-01-31T22:34:22.253+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.254843+0000) 2022-01-31T22:34:22.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:22 smithi167 conmon[54076]: debug 2022-01-31T22:34:22.253+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.254944+0000) 2022-01-31T22:34:22.660 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:22 smithi167 conmon[60316]: debug 2022-01-31T22:34:22.253+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.254578+0000) 2022-01-31T22:34:22.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:22 smithi171 conmon[41853]: debug 2022-01-31T22:34:22.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.603493+0000) 2022-01-31T22:34:22.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:22 smithi167 conmon[54076]: debug 2022-01-31T22:34:22.900+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.900905+0000) 2022-01-31T22:34:22.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:22 smithi167 conmon[49112]: debug 2022-01-31T22:34:22.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.733278+0000) 2022-01-31T22:34:22.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:22 smithi167 conmon[60316]: debug 2022-01-31T22:34:22.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:22.659682+0000) 2022-01-31T22:34:23.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:23 smithi171 conmon[46715]: debug 2022-01-31T22:34:23.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:23.400140+0000) 2022-01-31T22:34:23.565 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:23 smithi171 conmon[51620]: debug 2022-01-31T22:34:23.261+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:23.262791+0000) 2022-01-31T22:34:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:23 smithi171 conmon[41853]: debug 2022-01-31T22:34:23.602+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:23.603661+0000) 2022-01-31T22:34:23.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:23 smithi167 conmon[49112]: debug 2022-01-31T22:34:23.732+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:23.733449+0000) 2022-01-31T22:34:23.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:23 smithi167 conmon[54076]: debug 2022-01-31T22:34:23.899+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:23.901067+0000) 2022-01-31T22:34:23.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:23 smithi167 conmon[60316]: debug 2022-01-31T22:34:23.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:23.659846+0000) 2022-01-31T22:34:24.262 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:23 smithi171 conmon[35325]: debug 2022-01-31T22:34:23.874+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:34:24.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:24 smithi171 conmon[46715]: debug 2022-01-31T22:34:24.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:24.400365+0000) 2022-01-31T22:34:24.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:24 smithi171 conmon[51620]: debug 2022-01-31T22:34:24.262+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:24.262913+0000) 2022-01-31T22:34:24.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:24 smithi171 conmon[41853]: debug 2022-01-31T22:34:24.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:24.603843+0000) 2022-01-31T22:34:24.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:24 smithi167 conmon[54076]: debug 2022-01-31T22:34:24.900+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:24.901210+0000) 2022-01-31T22:34:24.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:24 smithi167 conmon[60316]: debug 2022-01-31T22:34:24.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:24.660054+0000) 2022-01-31T22:34:24.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:24 smithi167 conmon[49112]: debug 2022-01-31T22:34:24.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:24.733620+0000) 2022-01-31T22:34:25.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:25 smithi171 conmon[46715]: debug 2022-01-31T22:34:25.399+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:25.400535+0000) 2022-01-31T22:34:25.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:25 smithi171 conmon[51620]: debug 2022-01-31T22:34:25.262+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:25.263066+0000) 2022-01-31T22:34:25.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:25 smithi171 conmon[41853]: debug 2022-01-31T22:34:25.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:25.604025+0000) 2022-01-31T22:34:25.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:25 smithi167 conmon[49112]: debug 2022-01-31T22:34:25.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:25.733784+0000) 2022-01-31T22:34:25.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:25 smithi167 conmon[54076]: debug 2022-01-31T22:34:25.900+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:25.901376+0000) 2022-01-31T22:34:25.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:25 smithi167 conmon[60316]: debug 2022-01-31T22:34:25.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:25.660225+0000) 2022-01-31T22:34:26.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:26 smithi171 conmon[46715]: debug 2022-01-31T22:34:26.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:26.400778+0000) 2022-01-31T22:34:26.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:26 smithi171 conmon[51620]: debug 2022-01-31T22:34:26.262+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:26.263240+0000) 2022-01-31T22:34:26.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:26 smithi171 conmon[41853]: debug 2022-01-31T22:34:26.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:26.604212+0000) 2022-01-31T22:34:26.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:26 smithi167 conmon[60316]: debug 2022-01-31T22:34:26.659+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:26.660433+0000) 2022-01-31T22:34:26.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:26 smithi167 conmon[49112]: debug 2022-01-31T22:34:26.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:26.733970+0000) 2022-01-31T22:34:26.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:26 smithi167 conmon[54076]: debug 2022-01-31T22:34:26.901+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:26.901567+0000) 2022-01-31T22:34:27.565 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[35325]: debug 2022-01-31T22:34:27.301+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220306 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:27.566 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[41853]: debug 2022-01-31T22:34:27.279+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.280239+0000) 2022-01-31T22:34:27.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[51620]: debug 2022-01-31T22:34:27.262+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.263447+0000) 2022-01-31T22:34:27.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[51620]: debug 2022-01-31T22:34:27.279+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.280556+0000) 2022-01-31T22:34:27.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[46715]: debug 2022-01-31T22:34:27.278+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.278996+0000) 2022-01-31T22:34:27.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[46715]: debug 2022-01-31T22:34:27.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.400950+0000) 2022-01-31T22:34:27.660 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:27 smithi167 conmon[49112]: debug 2022-01-31T22:34:27.279+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.280043+0000) 2022-01-31T22:34:27.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:27 smithi167 conmon[54076]: debug 2022-01-31T22:34:27.279+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.279951+0000) 2022-01-31T22:34:27.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:27 smithi167 conmon[60316]: debug 2022-01-31T22:34:27.279+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.279637+0000) 2022-01-31T22:34:27.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:27 smithi171 conmon[41853]: debug 2022-01-31T22:34:27.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.604389+0000) 2022-01-31T22:34:27.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:27 smithi167 conmon[54076]: debug 2022-01-31T22:34:27.901+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.901751+0000) 2022-01-31T22:34:27.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:27 smithi167 conmon[49112]: debug 2022-01-31T22:34:27.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.734156+0000) 2022-01-31T22:34:27.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:27 smithi167 conmon[60316]: debug 2022-01-31T22:34:27.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:27.660616+0000) 2022-01-31T22:34:28.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:28 smithi171 conmon[46715]: debug 2022-01-31T22:34:28.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:28.401068+0000) 2022-01-31T22:34:28.566 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:28 smithi171 conmon[51620]: debug 2022-01-31T22:34:28.262+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:28.263673+0000) 2022-01-31T22:34:28.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:28 smithi171 conmon[41853]: debug 2022-01-31T22:34:28.603+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:28.604537+0000) 2022-01-31T22:34:28.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:28 smithi167 conmon[54076]: debug 2022-01-31T22:34:28.901+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:28.901890+0000) 2022-01-31T22:34:28.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:28 smithi167 conmon[49112]: debug 2022-01-31T22:34:28.733+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:28.734299+0000) 2022-01-31T22:34:28.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:28 smithi167 conmon[60316]: debug 2022-01-31T22:34:28.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:28.660729+0000) 2022-01-31T22:34:29.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:29 smithi171 conmon[46715]: debug 2022-01-31T22:34:29.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:29.401278+0000) 2022-01-31T22:34:29.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:29 smithi171 conmon[51620]: debug 2022-01-31T22:34:29.263+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:29.263845+0000) 2022-01-31T22:34:29.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:29 smithi171 conmon[41853]: debug 2022-01-31T22:34:29.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:29.604743+0000) 2022-01-31T22:34:29.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:29 smithi167 conmon[49112]: debug 2022-01-31T22:34:29.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:29.734513+0000) 2022-01-31T22:34:29.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:29 smithi167 conmon[54076]: debug 2022-01-31T22:34:29.901+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:29.902105+0000) 2022-01-31T22:34:29.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:29 smithi167 conmon[60316]: debug 2022-01-31T22:34:29.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:29.660902+0000) 2022-01-31T22:34:30.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:30 smithi171 conmon[46715]: debug 2022-01-31T22:34:30.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:30.401491+0000) 2022-01-31T22:34:30.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:30 smithi171 conmon[51620]: debug 2022-01-31T22:34:30.263+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:30.264062+0000) 2022-01-31T22:34:30.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:30 smithi171 conmon[41853]: debug 2022-01-31T22:34:30.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:30.604930+0000) 2022-01-31T22:34:30.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:30 smithi167 conmon[49112]: debug 2022-01-31T22:34:30.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:30.734688+0000) 2022-01-31T22:34:30.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:30 smithi167 conmon[60316]: debug 2022-01-31T22:34:30.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:30.661111+0000) 2022-01-31T22:34:30.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:30 smithi167 conmon[54076]: debug 2022-01-31T22:34:30.901+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:30.902335+0000) 2022-01-31T22:34:31.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:31 smithi171 conmon[46715]: debug 2022-01-31T22:34:31.400+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:31.401632+0000) 2022-01-31T22:34:31.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:31 smithi171 conmon[51620]: debug 2022-01-31T22:34:31.263+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:31.264261+0000) 2022-01-31T22:34:31.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:31 smithi171 conmon[41853]: debug 2022-01-31T22:34:31.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:31.605065+0000) 2022-01-31T22:34:31.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:31 smithi167 conmon[60316]: debug 2022-01-31T22:34:31.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:31.661295+0000) 2022-01-31T22:34:31.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:31 smithi167 conmon[49112]: debug 2022-01-31T22:34:31.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:31.734895+0000) 2022-01-31T22:34:31.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:31 smithi167 conmon[54076]: debug 2022-01-31T22:34:31.901+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:31.902458+0000) 2022-01-31T22:34:32.567 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[35325]: debug 2022-01-31T22:34:32.332+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220416 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:32.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[41853]: debug 2022-01-31T22:34:32.304+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.305612+0000) 2022-01-31T22:34:32.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[46715]: debug 2022-01-31T22:34:32.304+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.305667+0000) 2022-01-31T22:34:32.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[46715]: debug 2022-01-31T22:34:32.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.401869+0000) 2022-01-31T22:34:32.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[51620]: debug 2022-01-31T22:34:32.263+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.264469+0000) 2022-01-31T22:34:32.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[51620]: debug 2022-01-31T22:34:32.305+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.306449+0000) 2022-01-31T22:34:32.661 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:32 smithi167 conmon[49112]: debug 2022-01-31T22:34:32.304+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.305188+0000) 2022-01-31T22:34:32.661 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:32 smithi167 conmon[54076]: debug 2022-01-31T22:34:32.304+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.304925+0000) 2022-01-31T22:34:32.662 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:32 smithi167 conmon[60316]: debug 2022-01-31T22:34:32.305+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.305580+0000) 2022-01-31T22:34:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:32 smithi171 conmon[41853]: debug 2022-01-31T22:34:32.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.605226+0000) 2022-01-31T22:34:32.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:32 smithi167 conmon[49112]: debug 2022-01-31T22:34:32.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.735100+0000) 2022-01-31T22:34:32.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:32 smithi167 conmon[54076]: debug 2022-01-31T22:34:32.902+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.902617+0000) 2022-01-31T22:34:32.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:32 smithi167 conmon[60316]: debug 2022-01-31T22:34:32.660+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:32.661513+0000) 2022-01-31T22:34:33.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:33 smithi171 conmon[46715]: debug 2022-01-31T22:34:33.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:33.401985+0000) 2022-01-31T22:34:33.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:33 smithi171 conmon[51620]: debug 2022-01-31T22:34:33.263+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:33.264685+0000) 2022-01-31T22:34:33.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:33 smithi171 conmon[41853]: debug 2022-01-31T22:34:33.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:33.605361+0000) 2022-01-31T22:34:33.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:33 smithi167 conmon[49112]: debug 2022-01-31T22:34:33.734+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:33.735285+0000) 2022-01-31T22:34:33.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:33 smithi167 conmon[54076]: debug 2022-01-31T22:34:33.902+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:33.902793+0000) 2022-01-31T22:34:33.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:33 smithi167 conmon[60316]: debug 2022-01-31T22:34:33.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:33.661610+0000) 2022-01-31T22:34:34.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:34 smithi171 conmon[46715]: debug 2022-01-31T22:34:34.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:34.402165+0000) 2022-01-31T22:34:34.567 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:34 smithi171 conmon[51620]: debug 2022-01-31T22:34:34.264+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:34.264849+0000) 2022-01-31T22:34:34.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:34 smithi171 conmon[41853]: debug 2022-01-31T22:34:34.604+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:34.605575+0000) 2022-01-31T22:34:34.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:34 smithi167 conmon[49112]: debug 2022-01-31T22:34:34.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:34.735492+0000) 2022-01-31T22:34:34.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:34 smithi167 conmon[54076]: debug 2022-01-31T22:34:34.902+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:34.902956+0000) 2022-01-31T22:34:34.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:34 smithi167 conmon[60316]: debug 2022-01-31T22:34:34.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:34.661757+0000) 2022-01-31T22:34:35.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:35 smithi171 conmon[46715]: debug 2022-01-31T22:34:35.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:35.402342+0000) 2022-01-31T22:34:35.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:35 smithi171 conmon[51620]: debug 2022-01-31T22:34:35.264+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:35.265037+0000) 2022-01-31T22:34:35.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:35 smithi171 conmon[41853]: debug 2022-01-31T22:34:35.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:35.605790+0000) 2022-01-31T22:34:35.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:35 smithi167 conmon[54076]: debug 2022-01-31T22:34:35.902+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:35.903115+0000) 2022-01-31T22:34:35.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:35 smithi167 conmon[49112]: debug 2022-01-31T22:34:35.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:35.735643+0000) 2022-01-31T22:34:35.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:35 smithi167 conmon[60316]: debug 2022-01-31T22:34:35.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:35.661928+0000) 2022-01-31T22:34:36.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:36 smithi171 conmon[46715]: debug 2022-01-31T22:34:36.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:36.402509+0000) 2022-01-31T22:34:36.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:36 smithi171 conmon[51620]: debug 2022-01-31T22:34:36.264+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:36.265200+0000) 2022-01-31T22:34:36.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:36 smithi171 conmon[41853]: debug 2022-01-31T22:34:36.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:36.605997+0000) 2022-01-31T22:34:36.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:36 smithi167 conmon[60316]: debug 2022-01-31T22:34:36.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:36.662085+0000) 2022-01-31T22:34:36.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:36 smithi167 conmon[49112]: debug 2022-01-31T22:34:36.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:36.735823+0000) 2022-01-31T22:34:36.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:36 smithi167 conmon[54076]: debug 2022-01-31T22:34:36.902+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:36.903268+0000) 2022-01-31T22:34:37.567 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[35325]: debug 2022-01-31T22:34:37.357+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220530 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:37.568 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[41853]: debug 2022-01-31T22:34:37.334+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.335264+0000) 2022-01-31T22:34:37.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[46715]: debug 2022-01-31T22:34:37.334+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.335129+0000) 2022-01-31T22:34:37.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[46715]: debug 2022-01-31T22:34:37.401+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.402683+0000) 2022-01-31T22:34:37.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[51620]: debug 2022-01-31T22:34:37.264+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.265405+0000) 2022-01-31T22:34:37.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[51620]: debug 2022-01-31T22:34:37.335+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.336335+0000) 2022-01-31T22:34:37.661 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:37 smithi167 conmon[49112]: debug 2022-01-31T22:34:37.336+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.336800+0000) 2022-01-31T22:34:37.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:37 smithi167 conmon[54076]: debug 2022-01-31T22:34:37.335+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.336280+0000) 2022-01-31T22:34:37.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:37 smithi167 conmon[60316]: debug 2022-01-31T22:34:37.335+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.336061+0000) 2022-01-31T22:34:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:37 smithi171 conmon[41853]: debug 2022-01-31T22:34:37.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.606184+0000) 2022-01-31T22:34:37.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:37 smithi167 conmon[54076]: debug 2022-01-31T22:34:37.902+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.903402+0000) 2022-01-31T22:34:37.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:37 smithi167 conmon[49112]: debug 2022-01-31T22:34:37.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.736002+0000) 2022-01-31T22:34:37.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:37 smithi167 conmon[60316]: debug 2022-01-31T22:34:37.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:37.662278+0000) 2022-01-31T22:34:38.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:38 smithi171 conmon[46715]: debug 2022-01-31T22:34:38.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:38.402814+0000) 2022-01-31T22:34:38.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:38 smithi171 conmon[51620]: debug 2022-01-31T22:34:38.264+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:38.265588+0000) 2022-01-31T22:34:38.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:38 smithi171 conmon[41853]: debug 2022-01-31T22:34:38.605+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:38.606336+0000) 2022-01-31T22:34:38.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:38 smithi167 conmon[49112]: debug 2022-01-31T22:34:38.735+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:38.736130+0000) 2022-01-31T22:34:38.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:38 smithi167 conmon[54076]: debug 2022-01-31T22:34:38.903+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:38.903572+0000) 2022-01-31T22:34:38.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:38 smithi167 conmon[60316]: debug 2022-01-31T22:34:38.661+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:38.662459+0000) 2022-01-31T22:34:39.265 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:38 smithi171 conmon[35325]: debug 2022-01-31T22:34:38.875+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:34:39.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:39 smithi171 conmon[46715]: debug 2022-01-31T22:34:39.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:39.402957+0000) 2022-01-31T22:34:39.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:39 smithi171 conmon[51620]: debug 2022-01-31T22:34:39.265+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:39.265743+0000) 2022-01-31T22:34:39.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:39 smithi171 conmon[41853]: debug 2022-01-31T22:34:39.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:39.606544+0000) 2022-01-31T22:34:39.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:39 smithi167 conmon[49112]: debug 2022-01-31T22:34:39.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:39.736321+0000) 2022-01-31T22:34:39.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:39 smithi167 conmon[54076]: debug 2022-01-31T22:34:39.903+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:39.903725+0000) 2022-01-31T22:34:39.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:39 smithi167 conmon[60316]: debug 2022-01-31T22:34:39.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:39.662680+0000) 2022-01-31T22:34:40.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:40 smithi171 conmon[46715]: debug 2022-01-31T22:34:40.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:40.403114+0000) 2022-01-31T22:34:40.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:40 smithi171 conmon[51620]: debug 2022-01-31T22:34:40.265+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:40.265990+0000) 2022-01-31T22:34:40.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:40 smithi171 conmon[41853]: debug 2022-01-31T22:34:40.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:40.606747+0000) 2022-01-31T22:34:40.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:40 smithi167 conmon[49112]: debug 2022-01-31T22:34:40.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:40.736496+0000) 2022-01-31T22:34:40.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:40 smithi167 conmon[54076]: debug 2022-01-31T22:34:40.903+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:40.903949+0000) 2022-01-31T22:34:40.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:40 smithi167 conmon[60316]: debug 2022-01-31T22:34:40.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:40.662847+0000) 2022-01-31T22:34:41.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:41 smithi171 conmon[46715]: debug 2022-01-31T22:34:41.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:41.403296+0000) 2022-01-31T22:34:41.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:41 smithi171 conmon[51620]: debug 2022-01-31T22:34:41.265+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:41.266192+0000) 2022-01-31T22:34:41.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:41 smithi171 conmon[41853]: debug 2022-01-31T22:34:41.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:41.606939+0000) 2022-01-31T22:34:41.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:41 smithi167 conmon[49112]: debug 2022-01-31T22:34:41.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:41.736698+0000) 2022-01-31T22:34:41.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:41 smithi167 conmon[54076]: debug 2022-01-31T22:34:41.903+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:41.904105+0000) 2022-01-31T22:34:41.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:41 smithi167 conmon[60316]: debug 2022-01-31T22:34:41.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:41.663014+0000) 2022-01-31T22:34:42.568 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[35325]: debug 2022-01-31T22:34:42.383+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220657 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:42.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[41853]: debug 2022-01-31T22:34:42.359+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.360526+0000) 2022-01-31T22:34:42.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[46715]: debug 2022-01-31T22:34:42.359+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.360093+0000) 2022-01-31T22:34:42.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[46715]: debug 2022-01-31T22:34:42.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.403498+0000) 2022-01-31T22:34:42.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[51620]: debug 2022-01-31T22:34:42.265+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.266405+0000) 2022-01-31T22:34:42.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[51620]: debug 2022-01-31T22:34:42.361+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.362100+0000) 2022-01-31T22:34:42.662 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:42 smithi167 conmon[49112]: debug 2022-01-31T22:34:42.361+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.361858+0000) 2022-01-31T22:34:42.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:42 smithi167 conmon[54076]: debug 2022-01-31T22:34:42.360+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.361307+0000) 2022-01-31T22:34:42.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:42 smithi167 conmon[60316]: debug 2022-01-31T22:34:42.361+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.361881+0000) 2022-01-31T22:34:42.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:42 smithi171 conmon[41853]: debug 2022-01-31T22:34:42.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.607112+0000) 2022-01-31T22:34:42.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:42 smithi167 conmon[49112]: debug 2022-01-31T22:34:42.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.736848+0000) 2022-01-31T22:34:42.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:42 smithi167 conmon[54076]: debug 2022-01-31T22:34:42.903+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.904264+0000) 2022-01-31T22:34:42.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:42 smithi167 conmon[60316]: debug 2022-01-31T22:34:42.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:42.663226+0000) 2022-01-31T22:34:43.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:43 smithi171 conmon[46715]: debug 2022-01-31T22:34:43.402+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:43.403656+0000) 2022-01-31T22:34:43.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:43 smithi171 conmon[51620]: debug 2022-01-31T22:34:43.265+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:43.266595+0000) 2022-01-31T22:34:43.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:43 smithi171 conmon[41853]: debug 2022-01-31T22:34:43.606+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:43.607277+0000) 2022-01-31T22:34:43.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:43 smithi167 conmon[49112]: debug 2022-01-31T22:34:43.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:43.737004+0000) 2022-01-31T22:34:43.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:43 smithi167 conmon[54076]: debug 2022-01-31T22:34:43.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:43.904411+0000) 2022-01-31T22:34:43.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:43 smithi167 conmon[60316]: debug 2022-01-31T22:34:43.662+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:43.663397+0000) 2022-01-31T22:34:44.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:44 smithi171 conmon[46715]: debug 2022-01-31T22:34:44.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:44.403866+0000) 2022-01-31T22:34:44.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:44 smithi171 conmon[51620]: debug 2022-01-31T22:34:44.266+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:44.266743+0000) 2022-01-31T22:34:44.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:44 smithi171 conmon[41853]: debug 2022-01-31T22:34:44.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:44.607441+0000) 2022-01-31T22:34:44.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:44 smithi167 conmon[54076]: debug 2022-01-31T22:34:44.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:44.904575+0000) 2022-01-31T22:34:44.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:44 smithi167 conmon[60316]: debug 2022-01-31T22:34:44.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:44.663602+0000) 2022-01-31T22:34:44.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:44 smithi167 conmon[49112]: debug 2022-01-31T22:34:44.736+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:44.737192+0000) 2022-01-31T22:34:45.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:45 smithi171 conmon[46715]: debug 2022-01-31T22:34:45.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:45.404051+0000) 2022-01-31T22:34:45.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:45 smithi171 conmon[51620]: debug 2022-01-31T22:34:45.266+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:45.266958+0000) 2022-01-31T22:34:45.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:45 smithi171 conmon[41853]: debug 2022-01-31T22:34:45.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:45.607594+0000) 2022-01-31T22:34:45.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:45 smithi167 conmon[60316]: debug 2022-01-31T22:34:45.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:45.663804+0000) 2022-01-31T22:34:45.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:45 smithi167 conmon[49112]: debug 2022-01-31T22:34:45.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:45.737381+0000) 2022-01-31T22:34:45.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:45 smithi167 conmon[54076]: debug 2022-01-31T22:34:45.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:45.904748+0000) 2022-01-31T22:34:46.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:46 smithi171 conmon[46715]: debug 2022-01-31T22:34:46.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:46.404230+0000) 2022-01-31T22:34:46.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:46 smithi171 conmon[51620]: debug 2022-01-31T22:34:46.266+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:46.267163+0000) 2022-01-31T22:34:46.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:46 smithi171 conmon[41853]: debug 2022-01-31T22:34:46.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:46.607854+0000) 2022-01-31T22:34:46.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:46 smithi167 conmon[49112]: debug 2022-01-31T22:34:46.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:46.737605+0000) 2022-01-31T22:34:46.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:46 smithi167 conmon[54076]: debug 2022-01-31T22:34:46.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:46.904900+0000) 2022-01-31T22:34:46.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:46 smithi167 conmon[60316]: debug 2022-01-31T22:34:46.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:46.663990+0000) 2022-01-31T22:34:47.386 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:47 smithi167 conmon[60316]: debug 2022-01-31T22:34:47.386+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.386682+0000) 2022-01-31T22:34:47.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[35325]: debug 2022-01-31T22:34:47.407+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220770 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:47.570 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[41853]: debug 2022-01-31T22:34:47.385+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.386159+0000) 2022-01-31T22:34:47.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[46715]: debug 2022-01-31T22:34:47.384+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.385446+0000) 2022-01-31T22:34:47.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[46715]: debug 2022-01-31T22:34:47.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.404401+0000) 2022-01-31T22:34:47.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[51620]: debug 2022-01-31T22:34:47.267+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.267382+0000) 2022-01-31T22:34:47.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[51620]: debug 2022-01-31T22:34:47.385+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.386479+0000) 2022-01-31T22:34:47.663 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:47 smithi167 conmon[49112]: debug 2022-01-31T22:34:47.386+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.387006+0000) 2022-01-31T22:34:47.664 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:47 smithi167 conmon[54076]: debug 2022-01-31T22:34:47.386+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.387128+0000) 2022-01-31T22:34:47.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:47 smithi171 conmon[41853]: debug 2022-01-31T22:34:47.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.608063+0000) 2022-01-31T22:34:47.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:47 smithi167 conmon[54076]: debug 2022-01-31T22:34:47.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.905106+0000) 2022-01-31T22:34:47.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:47 smithi167 conmon[49112]: debug 2022-01-31T22:34:47.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.737761+0000) 2022-01-31T22:34:47.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:47 smithi167 conmon[60316]: debug 2022-01-31T22:34:47.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:47.664177+0000) 2022-01-31T22:34:48.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:48 smithi171 conmon[46715]: debug 2022-01-31T22:34:48.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:48.404555+0000) 2022-01-31T22:34:48.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:48 smithi171 conmon[51620]: debug 2022-01-31T22:34:48.266+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:48.267548+0000) 2022-01-31T22:34:48.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:48 smithi171 conmon[41853]: debug 2022-01-31T22:34:48.607+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:48.608156+0000) 2022-01-31T22:34:48.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:48 smithi167 conmon[49112]: debug 2022-01-31T22:34:48.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:48.737914+0000) 2022-01-31T22:34:48.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:48 smithi167 conmon[54076]: debug 2022-01-31T22:34:48.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:48.905266+0000) 2022-01-31T22:34:48.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:48 smithi167 conmon[60316]: debug 2022-01-31T22:34:48.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:48.664355+0000) 2022-01-31T22:34:49.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:49 smithi171 conmon[46715]: debug 2022-01-31T22:34:49.403+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:49.404681+0000) 2022-01-31T22:34:49.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:49 smithi171 conmon[51620]: debug 2022-01-31T22:34:49.267+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:49.267665+0000) 2022-01-31T22:34:49.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:49 smithi171 conmon[41853]: debug 2022-01-31T22:34:49.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:49.608304+0000) 2022-01-31T22:34:49.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:49 smithi167 conmon[49112]: debug 2022-01-31T22:34:49.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:49.738111+0000) 2022-01-31T22:34:49.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:49 smithi167 conmon[54076]: debug 2022-01-31T22:34:49.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:49.905445+0000) 2022-01-31T22:34:49.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:49 smithi167 conmon[60316]: debug 2022-01-31T22:34:49.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:49.664580+0000) 2022-01-31T22:34:50.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:50 smithi171 conmon[46715]: debug 2022-01-31T22:34:50.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:50.404862+0000) 2022-01-31T22:34:50.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:50 smithi171 conmon[51620]: debug 2022-01-31T22:34:50.267+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:50.267871+0000) 2022-01-31T22:34:50.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:50 smithi171 conmon[41853]: debug 2022-01-31T22:34:50.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:50.608498+0000) 2022-01-31T22:34:50.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:50 smithi167 conmon[49112]: debug 2022-01-31T22:34:50.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:50.738282+0000) 2022-01-31T22:34:50.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:50 smithi167 conmon[54076]: debug 2022-01-31T22:34:50.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:50.905651+0000) 2022-01-31T22:34:50.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:50 smithi167 conmon[60316]: debug 2022-01-31T22:34:50.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:50.664780+0000) 2022-01-31T22:34:51.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:51 smithi171 conmon[46715]: debug 2022-01-31T22:34:51.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:51.405041+0000) 2022-01-31T22:34:51.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:51 smithi171 conmon[51620]: debug 2022-01-31T22:34:51.267+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:51.268042+0000) 2022-01-31T22:34:51.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:51 smithi171 conmon[41853]: debug 2022-01-31T22:34:51.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:51.608707+0000) 2022-01-31T22:34:51.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:51 smithi167 conmon[54076]: debug 2022-01-31T22:34:51.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:51.905830+0000) 2022-01-31T22:34:51.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:51 smithi167 conmon[49112]: debug 2022-01-31T22:34:51.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:51.738411+0000) 2022-01-31T22:34:51.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:51 smithi167 conmon[60316]: debug 2022-01-31T22:34:51.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:51.664961+0000) 2022-01-31T22:34:52.570 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[35325]: debug 2022-01-31T22:34:52.433+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220881 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:52.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[41853]: debug 2022-01-31T22:34:52.410+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.411098+0000) 2022-01-31T22:34:52.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[46715]: debug 2022-01-31T22:34:52.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.405274+0000) 2022-01-31T22:34:52.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[46715]: debug 2022-01-31T22:34:52.410+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.411399+0000) 2022-01-31T22:34:52.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[51620]: debug 2022-01-31T22:34:52.267+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.268226+0000) 2022-01-31T22:34:52.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[51620]: debug 2022-01-31T22:34:52.410+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.410859+0000) 2022-01-31T22:34:52.664 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:52 smithi167 conmon[49112]: debug 2022-01-31T22:34:52.411+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.412751+0000) 2022-01-31T22:34:52.665 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:52 smithi167 conmon[54076]: debug 2022-01-31T22:34:52.411+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.412783+0000) 2022-01-31T22:34:52.665 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:52 smithi167 conmon[60316]: debug 2022-01-31T22:34:52.411+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.412598+0000) 2022-01-31T22:34:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:52 smithi171 conmon[41853]: debug 2022-01-31T22:34:52.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.608860+0000) 2022-01-31T22:34:52.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:52 smithi167 conmon[54076]: debug 2022-01-31T22:34:52.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.906037+0000) 2022-01-31T22:34:52.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:52 smithi167 conmon[49112]: debug 2022-01-31T22:34:52.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.738566+0000) 2022-01-31T22:34:52.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:52 smithi167 conmon[60316]: debug 2022-01-31T22:34:52.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:52.665141+0000) 2022-01-31T22:34:53.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:53 smithi171 conmon[46715]: debug 2022-01-31T22:34:53.404+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:53.405503+0000) 2022-01-31T22:34:53.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:53 smithi171 conmon[51620]: debug 2022-01-31T22:34:53.267+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:53.268372+0000) 2022-01-31T22:34:53.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:53 smithi171 conmon[41853]: debug 2022-01-31T22:34:53.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:53.609005+0000) 2022-01-31T22:34:53.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:53 smithi167 conmon[49112]: debug 2022-01-31T22:34:53.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:53.738694+0000) 2022-01-31T22:34:53.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:53 smithi167 conmon[54076]: debug 2022-01-31T22:34:53.904+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:53.906208+0000) 2022-01-31T22:34:53.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:53 smithi167 conmon[60316]: debug 2022-01-31T22:34:53.663+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:53.665292+0000) 2022-01-31T22:34:54.268 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:53 smithi171 conmon[35325]: debug 2022-01-31T22:34:53.876+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:34:54.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:54 smithi171 conmon[51620]: debug 2022-01-31T22:34:54.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:54.268563+0000) 2022-01-31T22:34:54.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:54 smithi171 conmon[46715]: debug 2022-01-31T22:34:54.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:54.405680+0000) 2022-01-31T22:34:54.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:54 smithi171 conmon[41853]: debug 2022-01-31T22:34:54.608+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:54.609137+0000) 2022-01-31T22:34:54.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:54 smithi167 conmon[54076]: debug 2022-01-31T22:34:54.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:54.906395+0000) 2022-01-31T22:34:54.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:54 smithi167 conmon[49112]: debug 2022-01-31T22:34:54.737+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:54.738849+0000) 2022-01-31T22:34:54.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:54 smithi167 conmon[60316]: debug 2022-01-31T22:34:54.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:54.665472+0000) 2022-01-31T22:34:55.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:55 smithi171 conmon[46715]: debug 2022-01-31T22:34:55.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:55.405843+0000) 2022-01-31T22:34:55.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:55 smithi171 conmon[51620]: debug 2022-01-31T22:34:55.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:55.268689+0000) 2022-01-31T22:34:55.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:55 smithi171 conmon[41853]: debug 2022-01-31T22:34:55.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:55.609355+0000) 2022-01-31T22:34:55.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:55 smithi167 conmon[49112]: debug 2022-01-31T22:34:55.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:55.739034+0000) 2022-01-31T22:34:55.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:55 smithi167 conmon[54076]: debug 2022-01-31T22:34:55.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:55.906551+0000) 2022-01-31T22:34:55.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:55 smithi167 conmon[60316]: debug 2022-01-31T22:34:55.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:55.665658+0000) 2022-01-31T22:34:56.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:56 smithi171 conmon[46715]: debug 2022-01-31T22:34:56.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:56.406037+0000) 2022-01-31T22:34:56.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:56 smithi171 conmon[51620]: debug 2022-01-31T22:34:56.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:56.268777+0000) 2022-01-31T22:34:56.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:56 smithi171 conmon[41853]: debug 2022-01-31T22:34:56.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:56.609568+0000) 2022-01-31T22:34:56.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:56 smithi167 conmon[54076]: debug 2022-01-31T22:34:56.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:56.906713+0000) 2022-01-31T22:34:56.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:56 smithi167 conmon[49112]: debug 2022-01-31T22:34:56.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:56.739195+0000) 2022-01-31T22:34:56.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:56 smithi167 conmon[60316]: debug 2022-01-31T22:34:56.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:56.665834+0000) 2022-01-31T22:34:57.571 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[35325]: debug 2022-01-31T22:34:57.459+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 220993 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:34:57.572 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[41853]: debug 2022-01-31T22:34:57.436+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.436896+0000) 2022-01-31T22:34:57.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[46715]: debug 2022-01-31T22:34:57.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.406257+0000) 2022-01-31T22:34:57.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[46715]: debug 2022-01-31T22:34:57.435+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.436461+0000) 2022-01-31T22:34:57.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[51620]: debug 2022-01-31T22:34:57.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.268941+0000) 2022-01-31T22:34:57.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[51620]: debug 2022-01-31T22:34:57.436+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.437240+0000) 2022-01-31T22:34:57.665 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:57 smithi167 conmon[49112]: debug 2022-01-31T22:34:57.436+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.437922+0000) 2022-01-31T22:34:57.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:57 smithi167 conmon[54076]: debug 2022-01-31T22:34:57.436+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.438350+0000) 2022-01-31T22:34:57.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:57 smithi167 conmon[60316]: debug 2022-01-31T22:34:57.437+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.438489+0000) 2022-01-31T22:34:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:57 smithi171 conmon[41853]: debug 2022-01-31T22:34:57.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.609787+0000) 2022-01-31T22:34:57.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:57 smithi167 conmon[54076]: debug 2022-01-31T22:34:57.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.906872+0000) 2022-01-31T22:34:57.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:57 smithi167 conmon[60316]: debug 2022-01-31T22:34:57.664+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.666046+0000) 2022-01-31T22:34:57.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:57 smithi167 conmon[49112]: debug 2022-01-31T22:34:57.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:57.739350+0000) 2022-01-31T22:34:58.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:58 smithi171 conmon[46715]: debug 2022-01-31T22:34:58.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:58.406462+0000) 2022-01-31T22:34:58.571 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:58 smithi171 conmon[51620]: debug 2022-01-31T22:34:58.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:58.269112+0000) 2022-01-31T22:34:58.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:58 smithi171 conmon[41853]: debug 2022-01-31T22:34:58.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:58.609947+0000) 2022-01-31T22:34:58.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:58 smithi167 conmon[54076]: debug 2022-01-31T22:34:58.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:58.906986+0000) 2022-01-31T22:34:58.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:58 smithi167 conmon[60316]: debug 2022-01-31T22:34:58.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:58.666199+0000) 2022-01-31T22:34:58.920 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:58 smithi167 conmon[49112]: debug 2022-01-31T22:34:58.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:58.739512+0000) 2022-01-31T22:34:59.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:34:59 smithi171 conmon[46715]: debug 2022-01-31T22:34:59.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:59.406631+0000) 2022-01-31T22:34:59.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:34:59 smithi171 conmon[51620]: debug 2022-01-31T22:34:59.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:59.269286+0000) 2022-01-31T22:34:59.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:34:59 smithi171 conmon[41853]: debug 2022-01-31T22:34:59.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:59.610146+0000) 2022-01-31T22:34:59.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:34:59 smithi167 conmon[49112]: debug 2022-01-31T22:34:59.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:59.739678+0000) 2022-01-31T22:34:59.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:34:59 smithi167 conmon[54076]: debug 2022-01-31T22:34:59.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:59.907173+0000) 2022-01-31T22:34:59.921 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:34:59 smithi167 conmon[60316]: debug 2022-01-31T22:34:59.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:34:59.666410+0000) 2022-01-31T22:35:00.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:00 smithi171 conmon[46715]: debug 2022-01-31T22:35:00.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:00.406805+0000) 2022-01-31T22:35:00.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:00 smithi171 conmon[51620]: debug 2022-01-31T22:35:00.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:00.269530+0000) 2022-01-31T22:35:00.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:00 smithi171 conmon[41853]: debug 2022-01-31T22:35:00.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:00.610346+0000) 2022-01-31T22:35:00.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:00 smithi167 conmon[49112]: debug 2022-01-31T22:35:00.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:00.739864+0000) 2022-01-31T22:35:00.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:00 smithi167 conmon[54076]: debug 2022-01-31T22:35:00.905+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:00.907380+0000) 2022-01-31T22:35:00.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:00 smithi167 conmon[60316]: debug 2022-01-31T22:35:00.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:00.666596+0000) 2022-01-31T22:35:01.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:01 smithi171 conmon[46715]: debug 2022-01-31T22:35:01.405+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:01.406967+0000) 2022-01-31T22:35:01.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:01 smithi171 conmon[51620]: debug 2022-01-31T22:35:01.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:01.269748+0000) 2022-01-31T22:35:01.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:01 smithi171 conmon[41853]: debug 2022-01-31T22:35:01.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:01.610503+0000) 2022-01-31T22:35:01.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:01 smithi167 conmon[49112]: debug 2022-01-31T22:35:01.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:01.740048+0000) 2022-01-31T22:35:01.920 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:01 smithi167 conmon[54076]: debug 2022-01-31T22:35:01.906+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:01.907583+0000) 2022-01-31T22:35:01.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:01 smithi167 conmon[60316]: debug 2022-01-31T22:35:01.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:01.666789+0000) 2022-01-31T22:35:02.572 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[35325]: debug 2022-01-31T22:35:02.498+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221104 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:02.572 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[41853]: debug 2022-01-31T22:35:02.459+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.462316+0000) 2022-01-31T22:35:02.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[51620]: debug 2022-01-31T22:35:02.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.269898+0000) 2022-01-31T22:35:02.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[51620]: debug 2022-01-31T22:35:02.459+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.462654+0000) 2022-01-31T22:35:02.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[46715]: debug 2022-01-31T22:35:02.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.407153+0000) 2022-01-31T22:35:02.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[46715]: debug 2022-01-31T22:35:02.459+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.463457+0000) 2022-01-31T22:35:02.666 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:02 smithi167 conmon[49112]: debug 2022-01-31T22:35:02.468+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.470160+0000) 2022-01-31T22:35:02.667 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:02 smithi167 conmon[54076]: debug 2022-01-31T22:35:02.461+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.463301+0000) 2022-01-31T22:35:02.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:02 smithi167 conmon[60316]: debug 2022-01-31T22:35:02.468+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.470250+0000) 2022-01-31T22:35:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:02 smithi171 conmon[41853]: debug 2022-01-31T22:35:02.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.610746+0000) 2022-01-31T22:35:02.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:02 smithi167 conmon[49112]: debug 2022-01-31T22:35:02.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.740228+0000) 2022-01-31T22:35:02.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:02 smithi167 conmon[54076]: debug 2022-01-31T22:35:02.906+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.907773+0000) 2022-01-31T22:35:02.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:02 smithi167 conmon[60316]: debug 2022-01-31T22:35:02.665+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:02.666980+0000) 2022-01-31T22:35:03.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:03 smithi171 conmon[46715]: debug 2022-01-31T22:35:03.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:03.407339+0000) 2022-01-31T22:35:03.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:03 smithi171 conmon[51620]: debug 2022-01-31T22:35:03.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:03.270086+0000) 2022-01-31T22:35:03.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:03 smithi171 conmon[41853]: debug 2022-01-31T22:35:03.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:03.610936+0000) 2022-01-31T22:35:03.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:03 smithi167 conmon[49112]: debug 2022-01-31T22:35:03.738+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:03.740385+0000) 2022-01-31T22:35:03.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:03 smithi167 conmon[54076]: debug 2022-01-31T22:35:03.906+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:03.907896+0000) 2022-01-31T22:35:03.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:03 smithi167 conmon[60316]: debug 2022-01-31T22:35:03.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:03.667121+0000) 2022-01-31T22:35:04.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:04 smithi171 conmon[46715]: debug 2022-01-31T22:35:04.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:04.407513+0000) 2022-01-31T22:35:04.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:04 smithi171 conmon[51620]: debug 2022-01-31T22:35:04.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:04.270265+0000) 2022-01-31T22:35:04.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:04 smithi171 conmon[41853]: debug 2022-01-31T22:35:04.609+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:04.611060+0000) 2022-01-31T22:35:04.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:04 smithi167 conmon[49112]: debug 2022-01-31T22:35:04.739+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:04.740574+0000) 2022-01-31T22:35:04.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:04 smithi167 conmon[54076]: debug 2022-01-31T22:35:04.906+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:04.908075+0000) 2022-01-31T22:35:04.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:04 smithi167 conmon[60316]: debug 2022-01-31T22:35:04.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:04.667356+0000) 2022-01-31T22:35:05.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:05 smithi171 conmon[46715]: debug 2022-01-31T22:35:05.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:05.407702+0000) 2022-01-31T22:35:05.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:05 smithi171 conmon[51620]: debug 2022-01-31T22:35:05.268+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:05.270442+0000) 2022-01-31T22:35:05.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:05 smithi171 conmon[41853]: debug 2022-01-31T22:35:05.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:05.611218+0000) 2022-01-31T22:35:05.919 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:05 smithi167 conmon[49112]: debug 2022-01-31T22:35:05.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:05.740783+0000) 2022-01-31T22:35:05.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:05 smithi167 conmon[54076]: debug 2022-01-31T22:35:05.907+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:05.908269+0000) 2022-01-31T22:35:05.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:05 smithi167 conmon[60316]: debug 2022-01-31T22:35:05.666+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:05.667537+0000) 2022-01-31T22:35:06.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:06 smithi171 conmon[46715]: debug 2022-01-31T22:35:06.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:06.407913+0000) 2022-01-31T22:35:06.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:06 smithi171 conmon[51620]: debug 2022-01-31T22:35:06.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:06.270604+0000) 2022-01-31T22:35:06.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:06 smithi171 conmon[41853]: debug 2022-01-31T22:35:06.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:06.611421+0000) 2022-01-31T22:35:06.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:06 smithi167 conmon[49112]: debug 2022-01-31T22:35:06.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:06.740976+0000) 2022-01-31T22:35:06.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:06 smithi167 conmon[54076]: debug 2022-01-31T22:35:06.907+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:06.908477+0000) 2022-01-31T22:35:06.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:06 smithi167 conmon[60316]: debug 2022-01-31T22:35:06.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:06.667731+0000) 2022-01-31T22:35:07.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[41853]: debug 2022-01-31T22:35:07.501+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.502907+0000) 2022-01-31T22:35:07.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[46715]: debug 2022-01-31T22:35:07.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.408098+0000) 2022-01-31T22:35:07.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[46715]: debug 2022-01-31T22:35:07.501+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.503706+0000) 2022-01-31T22:35:07.527 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[51620]: debug 2022-01-31T22:35:07.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.270803+0000) 2022-01-31T22:35:07.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[51620]: debug 2022-01-31T22:35:07.501+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.502917+0000) 2022-01-31T22:35:07.667 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:07 smithi167 conmon[49112]: debug 2022-01-31T22:35:07.502+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.504305+0000) 2022-01-31T22:35:07.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:07 smithi167 conmon[54076]: debug 2022-01-31T22:35:07.508+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.510134+0000) 2022-01-31T22:35:07.668 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:07 smithi167 conmon[60316]: debug 2022-01-31T22:35:07.502+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.503970+0000) 2022-01-31T22:35:07.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[41853]: debug 2022-01-31T22:35:07.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.611633+0000) 2022-01-31T22:35:07.858 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:07 smithi171 conmon[35325]: debug 2022-01-31T22:35:07.524+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221216 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:07.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:07 smithi167 conmon[49112]: debug 2022-01-31T22:35:07.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.741136+0000) 2022-01-31T22:35:07.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:07 smithi167 conmon[60316]: debug 2022-01-31T22:35:07.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.667864+0000) 2022-01-31T22:35:07.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:07 smithi167 conmon[54076]: debug 2022-01-31T22:35:07.907+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:07.908628+0000) 2022-01-31T22:35:08.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:08 smithi171 conmon[46715]: debug 2022-01-31T22:35:08.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:08.408263+0000) 2022-01-31T22:35:08.573 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:08 smithi171 conmon[51620]: debug 2022-01-31T22:35:08.270+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:08.270950+0000) 2022-01-31T22:35:08.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:08 smithi171 conmon[41853]: debug 2022-01-31T22:35:08.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:08.611765+0000) 2022-01-31T22:35:08.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:08 smithi167 conmon[49112]: debug 2022-01-31T22:35:08.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:08.741276+0000) 2022-01-31T22:35:08.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:08 smithi167 conmon[54076]: debug 2022-01-31T22:35:08.907+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:08.908763+0000) 2022-01-31T22:35:08.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:08 smithi167 conmon[60316]: debug 2022-01-31T22:35:08.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:08.668009+0000) 2022-01-31T22:35:09.270 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:08 smithi171 conmon[35325]: debug 2022-01-31T22:35:08.876+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:35:09.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:09 smithi171 conmon[46715]: debug 2022-01-31T22:35:09.406+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:09.408419+0000) 2022-01-31T22:35:09.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:09 smithi171 conmon[51620]: debug 2022-01-31T22:35:09.269+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:09.271071+0000) 2022-01-31T22:35:09.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:09 smithi171 conmon[41853]: debug 2022-01-31T22:35:09.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:09.611922+0000) 2022-01-31T22:35:09.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:09 smithi167 conmon[49112]: debug 2022-01-31T22:35:09.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:09.741450+0000) 2022-01-31T22:35:09.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:09 smithi167 conmon[54076]: debug 2022-01-31T22:35:09.908+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:09.908928+0000) 2022-01-31T22:35:09.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:09 smithi167 conmon[60316]: debug 2022-01-31T22:35:09.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:09.668194+0000) 2022-01-31T22:35:10.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:10 smithi171 conmon[46715]: debug 2022-01-31T22:35:10.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:10.408587+0000) 2022-01-31T22:35:10.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:10 smithi171 conmon[51620]: debug 2022-01-31T22:35:10.270+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:10.271320+0000) 2022-01-31T22:35:10.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:10 smithi171 conmon[41853]: debug 2022-01-31T22:35:10.610+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:10.612127+0000) 2022-01-31T22:35:10.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:10 smithi167 conmon[49112]: debug 2022-01-31T22:35:10.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:10.741574+0000) 2022-01-31T22:35:10.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:10 smithi167 conmon[54076]: debug 2022-01-31T22:35:10.907+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:10.909134+0000) 2022-01-31T22:35:10.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:10 smithi167 conmon[60316]: debug 2022-01-31T22:35:10.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:10.668399+0000) 2022-01-31T22:35:11.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:11 smithi171 conmon[46715]: debug 2022-01-31T22:35:11.407+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:11.408797+0000) 2022-01-31T22:35:11.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:11 smithi171 conmon[51620]: debug 2022-01-31T22:35:11.270+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:11.271524+0000) 2022-01-31T22:35:11.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:11 smithi171 conmon[41853]: debug 2022-01-31T22:35:11.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:11.612336+0000) 2022-01-31T22:35:11.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:11 smithi167 conmon[49112]: debug 2022-01-31T22:35:11.740+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:11.741715+0000) 2022-01-31T22:35:11.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:11 smithi167 conmon[54076]: debug 2022-01-31T22:35:11.908+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:11.909309+0000) 2022-01-31T22:35:11.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:11 smithi167 conmon[60316]: debug 2022-01-31T22:35:11.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:11.668587+0000) 2022-01-31T22:35:12.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[46715]: debug 2022-01-31T22:35:12.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.408929+0000) 2022-01-31T22:35:12.528 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[51620]: debug 2022-01-31T22:35:12.270+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.271717+0000) 2022-01-31T22:35:12.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[41853]: debug 2022-01-31T22:35:12.527+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.528183+0000) 2022-01-31T22:35:12.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:12 smithi167 conmon[49112]: debug 2022-01-31T22:35:12.527+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.529416+0000) 2022-01-31T22:35:12.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:12 smithi167 conmon[54076]: debug 2022-01-31T22:35:12.528+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.529975+0000) 2022-01-31T22:35:12.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:12 smithi167 conmon[60316]: debug 2022-01-31T22:35:12.529+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.530764+0000) 2022-01-31T22:35:12.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[35325]: debug 2022-01-31T22:35:12.551+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221328 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:12.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[46715]: debug 2022-01-31T22:35:12.529+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.530063+0000) 2022-01-31T22:35:12.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[51620]: debug 2022-01-31T22:35:12.528+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.529731+0000) 2022-01-31T22:35:12.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:12 smithi171 conmon[41853]: debug 2022-01-31T22:35:12.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.612490+0000) 2022-01-31T22:35:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:12 smithi167 conmon[49112]: debug 2022-01-31T22:35:12.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.741908+0000) 2022-01-31T22:35:12.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:12 smithi167 conmon[54076]: debug 2022-01-31T22:35:12.908+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.909474+0000) 2022-01-31T22:35:12.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:12 smithi167 conmon[60316]: debug 2022-01-31T22:35:12.667+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:12.668751+0000) 2022-01-31T22:35:13.573 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:13 smithi171 conmon[46715]: debug 2022-01-31T22:35:13.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:13.409051+0000) 2022-01-31T22:35:13.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:13 smithi171 conmon[51620]: debug 2022-01-31T22:35:13.271+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:13.271928+0000) 2022-01-31T22:35:13.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:13 smithi171 conmon[41853]: debug 2022-01-31T22:35:13.611+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:13.612650+0000) 2022-01-31T22:35:13.924 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:13 smithi167 conmon[49112]: debug 2022-01-31T22:35:13.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:13.742085+0000) 2022-01-31T22:35:13.925 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:13 smithi167 conmon[54076]: debug 2022-01-31T22:35:13.909+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:13.909640+0000) 2022-01-31T22:35:13.925 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:13 smithi167 conmon[60316]: debug 2022-01-31T22:35:13.668+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:13.668912+0000) 2022-01-31T22:35:14.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:14 smithi171 conmon[46715]: debug 2022-01-31T22:35:14.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:14.409225+0000) 2022-01-31T22:35:14.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:14 smithi171 conmon[51620]: debug 2022-01-31T22:35:14.271+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:14.272055+0000) 2022-01-31T22:35:14.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:14 smithi171 conmon[41853]: debug 2022-01-31T22:35:14.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:14.612834+0000) 2022-01-31T22:35:14.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:14 smithi167 conmon[49112]: debug 2022-01-31T22:35:14.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:14.742271+0000) 2022-01-31T22:35:14.919 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:14 smithi167 conmon[54076]: debug 2022-01-31T22:35:14.909+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:14.909753+0000) 2022-01-31T22:35:14.920 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:14 smithi167 conmon[60316]: debug 2022-01-31T22:35:14.668+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:14.669113+0000) 2022-01-31T22:35:15.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:15 smithi171 conmon[46715]: debug 2022-01-31T22:35:15.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:15.409406+0000) 2022-01-31T22:35:15.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:15 smithi171 conmon[51620]: debug 2022-01-31T22:35:15.271+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:15.272259+0000) 2022-01-31T22:35:15.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:15 smithi171 conmon[41853]: debug 2022-01-31T22:35:15.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:15.613018+0000) 2022-01-31T22:35:15.961 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:15 smithi167 conmon[49112]: debug 2022-01-31T22:35:15.741+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:15.742432+0000) 2022-01-31T22:35:15.962 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:15 smithi167 conmon[54076]: debug 2022-01-31T22:35:15.909+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:15.909918+0000) 2022-01-31T22:35:15.962 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:15 smithi167 conmon[60316]: debug 2022-01-31T22:35:15.668+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:15.669322+0000) 2022-01-31T22:35:16.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:16 smithi171 conmon[46715]: debug 2022-01-31T22:35:16.408+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:16.409632+0000) 2022-01-31T22:35:16.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:16 smithi171 conmon[51620]: debug 2022-01-31T22:35:16.271+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:16.272462+0000) 2022-01-31T22:35:16.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:16 smithi171 conmon[41853]: debug 2022-01-31T22:35:16.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:16.613167+0000) 2022-01-31T22:35:16.961 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:16 smithi167 conmon[49112]: debug 2022-01-31T22:35:16.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:16.742594+0000) 2022-01-31T22:35:16.962 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:16 smithi167 conmon[60316]: debug 2022-01-31T22:35:16.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:16.669543+0000) 2022-01-31T22:35:16.962 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:16 smithi167 conmon[54076]: debug 2022-01-31T22:35:16.909+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:16.910122+0000) 2022-01-31T22:35:17.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[46715]: debug 2022-01-31T22:35:17.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.409823+0000) 2022-01-31T22:35:17.554 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[51620]: debug 2022-01-31T22:35:17.271+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.272611+0000) 2022-01-31T22:35:17.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[41853]: debug 2022-01-31T22:35:17.553+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.553847+0000) 2022-01-31T22:35:17.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:17 smithi167 conmon[49112]: debug 2022-01-31T22:35:17.554+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.554754+0000) 2022-01-31T22:35:17.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:17 smithi167 conmon[54076]: debug 2022-01-31T22:35:17.554+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.555288+0000) 2022-01-31T22:35:17.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:17 smithi167 conmon[60316]: debug 2022-01-31T22:35:17.555+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.555679+0000) 2022-01-31T22:35:17.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[46715]: debug 2022-01-31T22:35:17.554+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.554821+0000) 2022-01-31T22:35:17.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[35325]: debug 2022-01-31T22:35:17.576+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221440 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:17.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[41853]: debug 2022-01-31T22:35:17.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.613339+0000) 2022-01-31T22:35:17.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:17 smithi171 conmon[51620]: debug 2022-01-31T22:35:17.554+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.555003+0000) 2022-01-31T22:35:17.961 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:17 smithi167 conmon[60316]: debug 2022-01-31T22:35:17.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.669702+0000) 2022-01-31T22:35:17.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:17 smithi167 conmon[49112]: debug 2022-01-31T22:35:17.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.742735+0000) 2022-01-31T22:35:17.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:17 smithi167 conmon[54076]: debug 2022-01-31T22:35:17.909+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:17.910309+0000) 2022-01-31T22:35:18.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:18 smithi171 conmon[46715]: debug 2022-01-31T22:35:18.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:18.409968+0000) 2022-01-31T22:35:18.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:18 smithi171 conmon[51620]: debug 2022-01-31T22:35:18.272+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:18.272793+0000) 2022-01-31T22:35:18.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:18 smithi171 conmon[41853]: debug 2022-01-31T22:35:18.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:18.613502+0000) 2022-01-31T22:35:18.961 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:18 smithi167 conmon[54076]: debug 2022-01-31T22:35:18.909+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:18.910472+0000) 2022-01-31T22:35:18.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:18 smithi167 conmon[49112]: debug 2022-01-31T22:35:18.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:18.742883+0000) 2022-01-31T22:35:18.962 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:18 smithi167 conmon[60316]: debug 2022-01-31T22:35:18.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:18.669802+0000) 2022-01-31T22:35:19.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:19 smithi171 conmon[46715]: debug 2022-01-31T22:35:19.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:19.410133+0000) 2022-01-31T22:35:19.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:19 smithi171 conmon[51620]: debug 2022-01-31T22:35:19.272+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:19.272937+0000) 2022-01-31T22:35:19.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:19 smithi171 conmon[41853]: debug 2022-01-31T22:35:19.612+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:19.613652+0000) 2022-01-31T22:35:19.961 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:19 smithi167 conmon[49112]: debug 2022-01-31T22:35:19.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:19.743077+0000) 2022-01-31T22:35:19.962 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:19 smithi167 conmon[54076]: debug 2022-01-31T22:35:19.910+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:19.910652+0000) 2022-01-31T22:35:19.963 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:19 smithi167 conmon[60316]: debug 2022-01-31T22:35:19.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:19.669938+0000) 2022-01-31T22:35:20.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:20 smithi171 conmon[46715]: debug 2022-01-31T22:35:20.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:20.410338+0000) 2022-01-31T22:35:20.575 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:20 smithi171 conmon[51620]: debug 2022-01-31T22:35:20.272+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:20.273102+0000) 2022-01-31T22:35:20.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:20 smithi171 conmon[41853]: debug 2022-01-31T22:35:20.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:20.613820+0000) 2022-01-31T22:35:20.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:20 smithi167 conmon[49112]: debug 2022-01-31T22:35:20.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:20.743236+0000) 2022-01-31T22:35:20.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:20 smithi167 conmon[54076]: debug 2022-01-31T22:35:20.910+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:20.910855+0000) 2022-01-31T22:35:20.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:20 smithi167 conmon[60316]: debug 2022-01-31T22:35:20.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:20.670081+0000) 2022-01-31T22:35:21.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:21 smithi171 conmon[46715]: debug 2022-01-31T22:35:21.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:21.410523+0000) 2022-01-31T22:35:21.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:21 smithi171 conmon[51620]: debug 2022-01-31T22:35:21.272+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:21.273265+0000) 2022-01-31T22:35:21.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:21 smithi171 conmon[41853]: debug 2022-01-31T22:35:21.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:21.614009+0000) 2022-01-31T22:35:21.961 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:21 smithi167 conmon[49112]: debug 2022-01-31T22:35:21.742+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:21.743395+0000) 2022-01-31T22:35:21.962 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:21 smithi167 conmon[54076]: debug 2022-01-31T22:35:21.910+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:21.911007+0000) 2022-01-31T22:35:21.963 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:21 smithi167 conmon[60316]: debug 2022-01-31T22:35:21.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:21.670214+0000) 2022-01-31T22:35:22.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[46715]: debug 2022-01-31T22:35:22.409+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.410726+0000) 2022-01-31T22:35:22.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[51620]: debug 2022-01-31T22:35:22.272+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.273466+0000) 2022-01-31T22:35:22.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:22 smithi167 conmon[49112]: debug 2022-01-31T22:35:22.579+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.580099+0000) 2022-01-31T22:35:22.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:22 smithi167 conmon[54076]: debug 2022-01-31T22:35:22.579+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.579968+0000) 2022-01-31T22:35:22.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:22 smithi167 conmon[60316]: debug 2022-01-31T22:35:22.580+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.580562+0000) 2022-01-31T22:35:22.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[35325]: debug 2022-01-31T22:35:22.601+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221553 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:22.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[46715]: debug 2022-01-31T22:35:22.578+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.579000+0000) 2022-01-31T22:35:22.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[51620]: debug 2022-01-31T22:35:22.579+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.579857+0000) 2022-01-31T22:35:22.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[41853]: debug 2022-01-31T22:35:22.578+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.579167+0000) 2022-01-31T22:35:22.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:22 smithi171 conmon[41853]: debug 2022-01-31T22:35:22.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.614212+0000) 2022-01-31T22:35:22.962 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:22 smithi167 conmon[60316]: debug 2022-01-31T22:35:22.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.670351+0000) 2022-01-31T22:35:22.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:22 smithi167 conmon[49112]: debug 2022-01-31T22:35:22.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.743576+0000) 2022-01-31T22:35:22.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:22 smithi167 conmon[54076]: debug 2022-01-31T22:35:22.910+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:22.911143+0000) 2022-01-31T22:35:23.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:23 smithi171 conmon[46715]: debug 2022-01-31T22:35:23.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:23.410910+0000) 2022-01-31T22:35:23.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:23 smithi171 conmon[51620]: debug 2022-01-31T22:35:23.272+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:23.273635+0000) 2022-01-31T22:35:23.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:23 smithi171 conmon[41853]: debug 2022-01-31T22:35:23.613+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:23.614342+0000) 2022-01-31T22:35:23.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:23 smithi167 conmon[49112]: debug 2022-01-31T22:35:23.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:23.743749+0000) 2022-01-31T22:35:23.963 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:23 smithi167 conmon[60316]: debug 2022-01-31T22:35:23.669+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:23.670468+0000) 2022-01-31T22:35:23.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:23 smithi167 conmon[54076]: debug 2022-01-31T22:35:23.910+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:23.911290+0000) 2022-01-31T22:35:24.273 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:23 smithi171 conmon[35325]: debug 2022-01-31T22:35:23.877+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:35:24.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:24 smithi171 conmon[46715]: debug 2022-01-31T22:35:24.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:24.411080+0000) 2022-01-31T22:35:24.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:24 smithi171 conmon[51620]: debug 2022-01-31T22:35:24.273+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:24.273807+0000) 2022-01-31T22:35:24.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:24 smithi171 conmon[41853]: debug 2022-01-31T22:35:24.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:24.614539+0000) 2022-01-31T22:35:24.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:24 smithi167 conmon[49112]: debug 2022-01-31T22:35:24.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:24.743936+0000) 2022-01-31T22:35:24.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:24 smithi167 conmon[54076]: debug 2022-01-31T22:35:24.910+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:24.911415+0000) 2022-01-31T22:35:24.963 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:24 smithi167 conmon[60316]: debug 2022-01-31T22:35:24.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:24.670561+0000) 2022-01-31T22:35:25.575 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:25 smithi171 conmon[46715]: debug 2022-01-31T22:35:25.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:25.411272+0000) 2022-01-31T22:35:25.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:25 smithi171 conmon[51620]: debug 2022-01-31T22:35:25.273+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:25.273960+0000) 2022-01-31T22:35:25.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:25 smithi171 conmon[41853]: debug 2022-01-31T22:35:25.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:25.614723+0000) 2022-01-31T22:35:25.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:25 smithi167 conmon[49112]: debug 2022-01-31T22:35:25.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:25.744070+0000) 2022-01-31T22:35:25.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:25 smithi167 conmon[54076]: debug 2022-01-31T22:35:25.911+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:25.911595+0000) 2022-01-31T22:35:25.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:25 smithi167 conmon[60316]: debug 2022-01-31T22:35:25.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:25.670704+0000) 2022-01-31T22:35:26.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:26 smithi171 conmon[46715]: debug 2022-01-31T22:35:26.410+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:26.411478+0000) 2022-01-31T22:35:26.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:26 smithi171 conmon[51620]: debug 2022-01-31T22:35:26.273+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:26.274194+0000) 2022-01-31T22:35:26.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:26 smithi171 conmon[41853]: debug 2022-01-31T22:35:26.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:26.614959+0000) 2022-01-31T22:35:26.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:26 smithi167 conmon[49112]: debug 2022-01-31T22:35:26.743+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:26.744228+0000) 2022-01-31T22:35:26.963 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:26 smithi167 conmon[60316]: debug 2022-01-31T22:35:26.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:26.670820+0000) 2022-01-31T22:35:26.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:26 smithi167 conmon[54076]: debug 2022-01-31T22:35:26.911+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:26.911815+0000) 2022-01-31T22:35:27.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[46715]: debug 2022-01-31T22:35:27.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.411632+0000) 2022-01-31T22:35:27.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[51620]: debug 2022-01-31T22:35:27.273+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.274386+0000) 2022-01-31T22:35:27.668 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:27 smithi167 conmon[54076]: debug 2022-01-31T22:35:27.604+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.605191+0000) 2022-01-31T22:35:27.669 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:27 smithi167 conmon[49112]: debug 2022-01-31T22:35:27.604+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.605363+0000) 2022-01-31T22:35:27.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:27 smithi167 conmon[60316]: debug 2022-01-31T22:35:27.605+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.605620+0000) 2022-01-31T22:35:27.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[35325]: debug 2022-01-31T22:35:27.627+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221664 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:27.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[46715]: debug 2022-01-31T22:35:27.603+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.604512+0000) 2022-01-31T22:35:27.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[51620]: debug 2022-01-31T22:35:27.604+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.604779+0000) 2022-01-31T22:35:27.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[41853]: debug 2022-01-31T22:35:27.605+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.605874+0000) 2022-01-31T22:35:27.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:27 smithi171 conmon[41853]: debug 2022-01-31T22:35:27.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.615124+0000) 2022-01-31T22:35:27.962 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:27 smithi167 conmon[49112]: debug 2022-01-31T22:35:27.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.744423+0000) 2022-01-31T22:35:27.963 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:27 smithi167 conmon[54076]: debug 2022-01-31T22:35:27.911+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.911982+0000) 2022-01-31T22:35:27.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:27 smithi167 conmon[60316]: debug 2022-01-31T22:35:27.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:27.670969+0000) 2022-01-31T22:35:28.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:28 smithi171 conmon[46715]: debug 2022-01-31T22:35:28.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:28.411827+0000) 2022-01-31T22:35:28.576 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:28 smithi171 conmon[51620]: debug 2022-01-31T22:35:28.273+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:28.274596+0000) 2022-01-31T22:35:28.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:28 smithi171 conmon[41853]: debug 2022-01-31T22:35:28.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:28.615279+0000) 2022-01-31T22:35:28.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:28 smithi167 conmon[49112]: debug 2022-01-31T22:35:28.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:28.744597+0000) 2022-01-31T22:35:28.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:28 smithi167 conmon[54076]: debug 2022-01-31T22:35:28.911+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:28.912142+0000) 2022-01-31T22:35:28.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:28 smithi167 conmon[60316]: debug 2022-01-31T22:35:28.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:28.671143+0000) 2022-01-31T22:35:29.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:29 smithi171 conmon[46715]: debug 2022-01-31T22:35:29.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:29.411963+0000) 2022-01-31T22:35:29.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:29 smithi171 conmon[51620]: debug 2022-01-31T22:35:29.274+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:29.274791+0000) 2022-01-31T22:35:29.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:29 smithi171 conmon[41853]: debug 2022-01-31T22:35:29.614+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:29.615437+0000) 2022-01-31T22:35:29.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:29 smithi167 conmon[49112]: debug 2022-01-31T22:35:29.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:29.744790+0000) 2022-01-31T22:35:29.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:29 smithi167 conmon[54076]: debug 2022-01-31T22:35:29.911+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:29.912274+0000) 2022-01-31T22:35:29.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:29 smithi167 conmon[60316]: debug 2022-01-31T22:35:29.670+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:29.671356+0000) 2022-01-31T22:35:30.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:30 smithi171 conmon[46715]: debug 2022-01-31T22:35:30.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:30.412064+0000) 2022-01-31T22:35:30.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:30 smithi171 conmon[51620]: debug 2022-01-31T22:35:30.274+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:30.274944+0000) 2022-01-31T22:35:30.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:30 smithi171 conmon[41853]: debug 2022-01-31T22:35:30.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:30.615613+0000) 2022-01-31T22:35:30.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:30 smithi167 conmon[49112]: debug 2022-01-31T22:35:30.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:30.744951+0000) 2022-01-31T22:35:30.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:30 smithi167 conmon[54076]: debug 2022-01-31T22:35:30.912+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:30.912440+0000) 2022-01-31T22:35:30.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:30 smithi167 conmon[60316]: debug 2022-01-31T22:35:30.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:30.671560+0000) 2022-01-31T22:35:31.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:31 smithi171 conmon[46715]: debug 2022-01-31T22:35:31.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:31.412196+0000) 2022-01-31T22:35:31.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:31 smithi171 conmon[51620]: debug 2022-01-31T22:35:31.274+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:31.275119+0000) 2022-01-31T22:35:31.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:31 smithi171 conmon[41853]: debug 2022-01-31T22:35:31.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:31.615790+0000) 2022-01-31T22:35:31.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:31 smithi167 conmon[49112]: debug 2022-01-31T22:35:31.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:31.745119+0000) 2022-01-31T22:35:31.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:31 smithi167 conmon[54076]: debug 2022-01-31T22:35:31.912+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:31.912658+0000) 2022-01-31T22:35:31.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:31 smithi167 conmon[60316]: debug 2022-01-31T22:35:31.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:31.671690+0000) 2022-01-31T22:35:32.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[46715]: debug 2022-01-31T22:35:32.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.412384+0000) 2022-01-31T22:35:32.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[51620]: debug 2022-01-31T22:35:32.274+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.275285+0000) 2022-01-31T22:35:32.668 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:32 smithi167 conmon[49112]: debug 2022-01-31T22:35:32.631+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.631896+0000) 2022-01-31T22:35:32.669 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:32 smithi167 conmon[54076]: debug 2022-01-31T22:35:32.631+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.631565+0000) 2022-01-31T22:35:32.669 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:32 smithi167 conmon[60316]: debug 2022-01-31T22:35:32.631+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.631729+0000) 2022-01-31T22:35:32.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[35325]: debug 2022-01-31T22:35:32.653+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221776 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[41853]: debug 2022-01-31T22:35:32.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.615935+0000) 2022-01-31T22:35:32.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[41853]: debug 2022-01-31T22:35:32.630+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.631444+0000) 2022-01-31T22:35:32.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[46715]: debug 2022-01-31T22:35:32.629+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.630479+0000) 2022-01-31T22:35:32.859 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:32 smithi171 conmon[51620]: debug 2022-01-31T22:35:32.629+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.630671+0000) 2022-01-31T22:35:32.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:32 smithi167 conmon[49112]: debug 2022-01-31T22:35:32.744+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.745328+0000) 2022-01-31T22:35:32.964 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:32 smithi167 conmon[60316]: debug 2022-01-31T22:35:32.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.671818+0000) 2022-01-31T22:35:32.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:32 smithi167 conmon[54076]: debug 2022-01-31T22:35:32.912+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:32.912839+0000) 2022-01-31T22:35:33.576 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:33 smithi171 conmon[46715]: debug 2022-01-31T22:35:33.411+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:33.412579+0000) 2022-01-31T22:35:33.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:33 smithi171 conmon[51620]: debug 2022-01-31T22:35:33.274+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:33.275496+0000) 2022-01-31T22:35:33.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:33 smithi171 conmon[41853]: debug 2022-01-31T22:35:33.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:33.616111+0000) 2022-01-31T22:35:33.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:33 smithi167 conmon[49112]: debug 2022-01-31T22:35:33.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:33.745516+0000) 2022-01-31T22:35:33.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:33 smithi167 conmon[54076]: debug 2022-01-31T22:35:33.912+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:33.913011+0000) 2022-01-31T22:35:33.965 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:33 smithi167 conmon[60316]: debug 2022-01-31T22:35:33.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:33.671963+0000) 2022-01-31T22:35:34.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:34 smithi171 conmon[46715]: debug 2022-01-31T22:35:34.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:34.412771+0000) 2022-01-31T22:35:34.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:34 smithi171 conmon[51620]: debug 2022-01-31T22:35:34.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:34.275674+0000) 2022-01-31T22:35:34.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:34 smithi171 conmon[41853]: debug 2022-01-31T22:35:34.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:34.616273+0000) 2022-01-31T22:35:34.963 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:34 smithi167 conmon[49112]: debug 2022-01-31T22:35:34.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:34.745672+0000) 2022-01-31T22:35:34.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:34 smithi167 conmon[54076]: debug 2022-01-31T22:35:34.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:34.913205+0000) 2022-01-31T22:35:34.965 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:34 smithi167 conmon[60316]: debug 2022-01-31T22:35:34.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:34.672140+0000) 2022-01-31T22:35:35.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:35 smithi171 conmon[46715]: debug 2022-01-31T22:35:35.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:35.412963+0000) 2022-01-31T22:35:35.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:35 smithi171 conmon[51620]: debug 2022-01-31T22:35:35.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:35.275855+0000) 2022-01-31T22:35:35.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:35 smithi171 conmon[41853]: debug 2022-01-31T22:35:35.615+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:35.616494+0000) 2022-01-31T22:35:35.964 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:35 smithi167 conmon[49112]: debug 2022-01-31T22:35:35.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:35.745855+0000) 2022-01-31T22:35:35.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:35 smithi167 conmon[54076]: debug 2022-01-31T22:35:35.912+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:35.913388+0000) 2022-01-31T22:35:35.965 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:35 smithi167 conmon[60316]: debug 2022-01-31T22:35:35.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:35.672349+0000) 2022-01-31T22:35:36.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:36 smithi171 conmon[46715]: debug 2022-01-31T22:35:36.412+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:36.413159+0000) 2022-01-31T22:35:36.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:36 smithi171 conmon[51620]: debug 2022-01-31T22:35:36.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:36.276058+0000) 2022-01-31T22:35:36.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:36 smithi171 conmon[41853]: debug 2022-01-31T22:35:36.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:36.616683+0000) 2022-01-31T22:35:36.964 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:36 smithi167 conmon[49112]: debug 2022-01-31T22:35:36.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:36.746040+0000) 2022-01-31T22:35:36.964 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:36 smithi167 conmon[54076]: debug 2022-01-31T22:35:36.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:36.913606+0000) 2022-01-31T22:35:36.965 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:36 smithi167 conmon[60316]: debug 2022-01-31T22:35:36.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:36.672557+0000) 2022-01-31T22:35:37.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[46715]: debug 2022-01-31T22:35:37.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.413311+0000) 2022-01-31T22:35:37.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[51620]: debug 2022-01-31T22:35:37.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.276211+0000) 2022-01-31T22:35:37.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[35325]: debug 2022-01-31T22:35:37.678+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221887 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[41853]: debug 2022-01-31T22:35:37.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.616801+0000) 2022-01-31T22:35:37.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[41853]: debug 2022-01-31T22:35:37.657+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.658302+0000) 2022-01-31T22:35:37.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[46715]: debug 2022-01-31T22:35:37.656+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.656916+0000) 2022-01-31T22:35:37.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:37 smithi171 conmon[51620]: debug 2022-01-31T22:35:37.656+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.656784+0000) 2022-01-31T22:35:37.912 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:37 smithi167 conmon[54076]: debug 2022-01-31T22:35:37.657+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.657912+0000) 2022-01-31T22:35:37.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:37 smithi167 conmon[49112]: debug 2022-01-31T22:35:37.658+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.658658+0000) 2022-01-31T22:35:37.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:37 smithi167 conmon[49112]: debug 2022-01-31T22:35:37.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.746232+0000) 2022-01-31T22:35:37.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:37 smithi167 conmon[60316]: debug 2022-01-31T22:35:37.657+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.658410+0000) 2022-01-31T22:35:37.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:37 smithi167 conmon[60316]: debug 2022-01-31T22:35:37.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.672729+0000) 2022-01-31T22:35:38.168 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:37 smithi167 conmon[54076]: debug 2022-01-31T22:35:37.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:37.913756+0000) 2022-01-31T22:35:38.577 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:38 smithi171 conmon[51620]: debug 2022-01-31T22:35:38.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:38.276427+0000) 2022-01-31T22:35:38.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:38 smithi171 conmon[46715]: debug 2022-01-31T22:35:38.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:38.413496+0000) 2022-01-31T22:35:38.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:38 smithi171 conmon[41853]: debug 2022-01-31T22:35:38.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:38.616959+0000) 2022-01-31T22:35:38.964 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:38 smithi167 conmon[49112]: debug 2022-01-31T22:35:38.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:38.746369+0000) 2022-01-31T22:35:38.965 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:38 smithi167 conmon[54076]: debug 2022-01-31T22:35:38.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:38.913882+0000) 2022-01-31T22:35:38.965 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:38 smithi167 conmon[60316]: debug 2022-01-31T22:35:38.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:38.672846+0000) 2022-01-31T22:35:39.275 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:38 smithi171 conmon[35325]: debug 2022-01-31T22:35:38.878+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:35:39.577 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:39 smithi171 conmon[46715]: debug 2022-01-31T22:35:39.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:39.413663+0000) 2022-01-31T22:35:39.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:39 smithi171 conmon[51620]: debug 2022-01-31T22:35:39.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:39.276555+0000) 2022-01-31T22:35:39.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:39 smithi171 conmon[41853]: debug 2022-01-31T22:35:39.616+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:39.617068+0000) 2022-01-31T22:35:39.964 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:39 smithi167 conmon[49112]: debug 2022-01-31T22:35:39.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:39.746542+0000) 2022-01-31T22:35:39.965 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:39 smithi167 conmon[54076]: debug 2022-01-31T22:35:39.914+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:39.914053+0000) 2022-01-31T22:35:39.965 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:39 smithi167 conmon[60316]: debug 2022-01-31T22:35:39.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:39.673029+0000) 2022-01-31T22:35:40.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:40 smithi171 conmon[46715]: debug 2022-01-31T22:35:40.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:40.413844+0000) 2022-01-31T22:35:40.578 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:40 smithi171 conmon[51620]: debug 2022-01-31T22:35:40.275+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:40.276725+0000) 2022-01-31T22:35:40.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:40 smithi171 conmon[41853]: debug 2022-01-31T22:35:40.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:40.617235+0000) 2022-01-31T22:35:40.964 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:40 smithi167 conmon[49112]: debug 2022-01-31T22:35:40.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:40.746673+0000) 2022-01-31T22:35:40.965 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:40 smithi167 conmon[54076]: debug 2022-01-31T22:35:40.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:40.914255+0000) 2022-01-31T22:35:40.966 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:40 smithi167 conmon[60316]: debug 2022-01-31T22:35:40.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:40.673234+0000) 2022-01-31T22:35:41.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:41 smithi171 conmon[46715]: debug 2022-01-31T22:35:41.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:41.413991+0000) 2022-01-31T22:35:41.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:41 smithi171 conmon[51620]: debug 2022-01-31T22:35:41.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:41.276913+0000) 2022-01-31T22:35:41.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:41 smithi171 conmon[41853]: debug 2022-01-31T22:35:41.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:41.617390+0000) 2022-01-31T22:35:41.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:41 smithi167 conmon[49112]: debug 2022-01-31T22:35:41.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:41.746870+0000) 2022-01-31T22:35:41.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:41 smithi167 conmon[54076]: debug 2022-01-31T22:35:41.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:41.914480+0000) 2022-01-31T22:35:41.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:41 smithi167 conmon[60316]: debug 2022-01-31T22:35:41.671+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:41.673444+0000) 2022-01-31T22:35:42.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[46715]: debug 2022-01-31T22:35:42.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.414173+0000) 2022-01-31T22:35:42.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[51620]: debug 2022-01-31T22:35:42.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.277106+0000) 2022-01-31T22:35:42.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[35325]: debug 2022-01-31T22:35:42.704+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 221999 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:42.857 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[46715]: debug 2022-01-31T22:35:42.681+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.682023+0000) 2022-01-31T22:35:42.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[51620]: debug 2022-01-31T22:35:42.681+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.681851+0000) 2022-01-31T22:35:42.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[41853]: debug 2022-01-31T22:35:42.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.617637+0000) 2022-01-31T22:35:42.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:42 smithi171 conmon[41853]: debug 2022-01-31T22:35:42.681+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.682458+0000) 2022-01-31T22:35:42.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:42 smithi167 conmon[49112]: debug 2022-01-31T22:35:42.681+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.682890+0000) 2022-01-31T22:35:42.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:42 smithi167 conmon[49112]: debug 2022-01-31T22:35:42.745+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.747066+0000) 2022-01-31T22:35:42.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:42 smithi167 conmon[54076]: debug 2022-01-31T22:35:42.681+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.683086+0000) 2022-01-31T22:35:42.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:42 smithi167 conmon[54076]: debug 2022-01-31T22:35:42.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.914623+0000) 2022-01-31T22:35:42.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:42 smithi167 conmon[60316]: debug 2022-01-31T22:35:42.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.673655+0000) 2022-01-31T22:35:42.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:42 smithi167 conmon[60316]: debug 2022-01-31T22:35:42.681+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:42.683200+0000) 2022-01-31T22:35:43.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:43 smithi171 conmon[46715]: debug 2022-01-31T22:35:43.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:43.414335+0000) 2022-01-31T22:35:43.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:43 smithi171 conmon[51620]: debug 2022-01-31T22:35:43.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:43.277231+0000) 2022-01-31T22:35:43.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:43 smithi171 conmon[41853]: debug 2022-01-31T22:35:43.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:43.617836+0000) 2022-01-31T22:35:43.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:43 smithi167 conmon[49112]: debug 2022-01-31T22:35:43.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:43.747223+0000) 2022-01-31T22:35:43.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:43 smithi167 conmon[54076]: debug 2022-01-31T22:35:43.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:43.914773+0000) 2022-01-31T22:35:43.966 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:43 smithi167 conmon[60316]: debug 2022-01-31T22:35:43.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:43.673763+0000) 2022-01-31T22:35:44.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:44 smithi171 conmon[46715]: debug 2022-01-31T22:35:44.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:44.414513+0000) 2022-01-31T22:35:44.627 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:44 smithi171 conmon[51620]: debug 2022-01-31T22:35:44.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:44.277373+0000) 2022-01-31T22:35:44.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:44 smithi171 conmon[41853]: debug 2022-01-31T22:35:44.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:44.617989+0000) 2022-01-31T22:35:44.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:44 smithi167 conmon[49112]: debug 2022-01-31T22:35:44.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:44.747403+0000) 2022-01-31T22:35:44.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:44 smithi167 conmon[54076]: debug 2022-01-31T22:35:44.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:44.914948+0000) 2022-01-31T22:35:44.966 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:44 smithi167 conmon[60316]: debug 2022-01-31T22:35:44.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:44.673911+0000) 2022-01-31T22:35:45.578 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:45 smithi171 conmon[46715]: debug 2022-01-31T22:35:45.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:45.414699+0000) 2022-01-31T22:35:45.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:45 smithi171 conmon[51620]: debug 2022-01-31T22:35:45.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:45.277562+0000) 2022-01-31T22:35:45.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:45 smithi171 conmon[41853]: debug 2022-01-31T22:35:45.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:45.618165+0000) 2022-01-31T22:35:45.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:45 smithi167 conmon[49112]: debug 2022-01-31T22:35:45.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:45.747556+0000) 2022-01-31T22:35:45.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:45 smithi167 conmon[54076]: debug 2022-01-31T22:35:45.914+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:45.915103+0000) 2022-01-31T22:35:45.966 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:45 smithi167 conmon[60316]: debug 2022-01-31T22:35:45.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:45.674114+0000) 2022-01-31T22:35:46.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:46 smithi171 conmon[46715]: debug 2022-01-31T22:35:46.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:46.414898+0000) 2022-01-31T22:35:46.579 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:46 smithi171 conmon[51620]: debug 2022-01-31T22:35:46.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:46.277711+0000) 2022-01-31T22:35:46.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:46 smithi171 conmon[41853]: debug 2022-01-31T22:35:46.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:46.618354+0000) 2022-01-31T22:35:46.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:46 smithi167 conmon[49112]: debug 2022-01-31T22:35:46.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:46.747735+0000) 2022-01-31T22:35:46.966 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:46 smithi167 conmon[54076]: debug 2022-01-31T22:35:46.913+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:46.915266+0000) 2022-01-31T22:35:46.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:46 smithi167 conmon[60316]: debug 2022-01-31T22:35:46.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:46.674298+0000) 2022-01-31T22:35:47.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[46715]: debug 2022-01-31T22:35:47.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.415050+0000) 2022-01-31T22:35:47.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[51620]: debug 2022-01-31T22:35:47.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.277904+0000) 2022-01-31T22:35:47.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[51620]: debug 2022-01-31T22:35:47.705+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.707077+0000) 2022-01-31T22:35:47.857 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[35325]: debug 2022-01-31T22:35:47.729+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222110 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:47.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[41853]: debug 2022-01-31T22:35:47.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.618539+0000) 2022-01-31T22:35:47.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[41853]: debug 2022-01-31T22:35:47.706+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.707941+0000) 2022-01-31T22:35:47.859 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:47 smithi171 conmon[46715]: debug 2022-01-31T22:35:47.705+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.707725+0000) 2022-01-31T22:35:47.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:47 smithi167 conmon[60316]: debug 2022-01-31T22:35:47.672+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.674455+0000) 2022-01-31T22:35:47.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:47 smithi167 conmon[60316]: debug 2022-01-31T22:35:47.707+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.708554+0000) 2022-01-31T22:35:47.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:47 smithi167 conmon[49112]: debug 2022-01-31T22:35:47.706+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.708277+0000) 2022-01-31T22:35:47.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:47 smithi167 conmon[49112]: debug 2022-01-31T22:35:47.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.747945+0000) 2022-01-31T22:35:47.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:47 smithi167 conmon[54076]: debug 2022-01-31T22:35:47.707+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.709177+0000) 2022-01-31T22:35:47.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:47 smithi167 conmon[54076]: debug 2022-01-31T22:35:47.914+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:47.915424+0000) 2022-01-31T22:35:48.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:48 smithi171 conmon[46715]: debug 2022-01-31T22:35:48.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:48.415265+0000) 2022-01-31T22:35:48.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:48 smithi171 conmon[51620]: debug 2022-01-31T22:35:48.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:48.278039+0000) 2022-01-31T22:35:48.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:48 smithi171 conmon[41853]: debug 2022-01-31T22:35:48.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:48.618704+0000) 2022-01-31T22:35:48.965 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:48 smithi167 conmon[49112]: debug 2022-01-31T22:35:48.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:48.748090+0000) 2022-01-31T22:35:48.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:48 smithi167 conmon[54076]: debug 2022-01-31T22:35:48.914+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:48.915576+0000) 2022-01-31T22:35:48.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:48 smithi167 conmon[60316]: debug 2022-01-31T22:35:48.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:48.674632+0000) 2022-01-31T22:35:49.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:49 smithi171 conmon[46715]: debug 2022-01-31T22:35:49.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:49.415419+0000) 2022-01-31T22:35:49.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:49 smithi171 conmon[51620]: debug 2022-01-31T22:35:49.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:49.278180+0000) 2022-01-31T22:35:49.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:49 smithi171 conmon[41853]: debug 2022-01-31T22:35:49.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:49.618851+0000) 2022-01-31T22:35:49.966 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:49 smithi167 conmon[49112]: debug 2022-01-31T22:35:49.746+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:49.748309+0000) 2022-01-31T22:35:49.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:49 smithi167 conmon[54076]: debug 2022-01-31T22:35:49.914+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:49.915795+0000) 2022-01-31T22:35:49.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:49 smithi167 conmon[60316]: debug 2022-01-31T22:35:49.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:49.674738+0000) 2022-01-31T22:35:50.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:50 smithi171 conmon[46715]: debug 2022-01-31T22:35:50.413+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:50.415575+0000) 2022-01-31T22:35:50.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:50 smithi171 conmon[51620]: debug 2022-01-31T22:35:50.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:50.278402+0000) 2022-01-31T22:35:50.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:50 smithi171 conmon[41853]: debug 2022-01-31T22:35:50.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:50.619045+0000) 2022-01-31T22:35:50.966 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:50 smithi167 conmon[49112]: debug 2022-01-31T22:35:50.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:50.748496+0000) 2022-01-31T22:35:50.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:50 smithi167 conmon[54076]: debug 2022-01-31T22:35:50.915+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:50.915992+0000) 2022-01-31T22:35:50.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:50 smithi167 conmon[60316]: debug 2022-01-31T22:35:50.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:50.674880+0000) 2022-01-31T22:35:51.579 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:51 smithi171 conmon[46715]: debug 2022-01-31T22:35:51.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:51.415749+0000) 2022-01-31T22:35:51.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:51 smithi171 conmon[51620]: debug 2022-01-31T22:35:51.276+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:51.278554+0000) 2022-01-31T22:35:51.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:51 smithi171 conmon[41853]: debug 2022-01-31T22:35:51.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:51.619242+0000) 2022-01-31T22:35:51.966 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:51 smithi167 conmon[49112]: debug 2022-01-31T22:35:51.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:51.748683+0000) 2022-01-31T22:35:51.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:51 smithi167 conmon[54076]: debug 2022-01-31T22:35:51.915+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:51.916147+0000) 2022-01-31T22:35:51.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:51 smithi167 conmon[60316]: debug 2022-01-31T22:35:51.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:51.675035+0000) 2022-01-31T22:35:52.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[46715]: debug 2022-01-31T22:35:52.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.415959+0000) 2022-01-31T22:35:52.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[51620]: debug 2022-01-31T22:35:52.277+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.278774+0000) 2022-01-31T22:35:52.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[35325]: debug 2022-01-31T22:35:52.755+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222222 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:52.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[46715]: debug 2022-01-31T22:35:52.732+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.733762+0000) 2022-01-31T22:35:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[41853]: debug 2022-01-31T22:35:52.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.619430+0000) 2022-01-31T22:35:52.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[41853]: debug 2022-01-31T22:35:52.731+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.733011+0000) 2022-01-31T22:35:52.858 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:52 smithi171 conmon[51620]: debug 2022-01-31T22:35:52.731+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.733489+0000) 2022-01-31T22:35:52.966 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:52 smithi167 conmon[49112]: debug 2022-01-31T22:35:52.733+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.734882+0000) 2022-01-31T22:35:52.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:52 smithi167 conmon[49112]: debug 2022-01-31T22:35:52.747+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.748801+0000) 2022-01-31T22:35:52.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:52 smithi167 conmon[54076]: debug 2022-01-31T22:35:52.732+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.734411+0000) 2022-01-31T22:35:52.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:52 smithi167 conmon[54076]: debug 2022-01-31T22:35:52.915+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.916346+0000) 2022-01-31T22:35:52.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:52 smithi167 conmon[60316]: debug 2022-01-31T22:35:52.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.675223+0000) 2022-01-31T22:35:52.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:52 smithi167 conmon[60316]: debug 2022-01-31T22:35:52.733+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:52.735011+0000) 2022-01-31T22:35:53.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:53 smithi171 conmon[46715]: debug 2022-01-31T22:35:53.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:53.416144+0000) 2022-01-31T22:35:53.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:53 smithi171 conmon[51620]: debug 2022-01-31T22:35:53.277+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:53.278950+0000) 2022-01-31T22:35:53.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:53 smithi171 conmon[41853]: debug 2022-01-31T22:35:53.617+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:53.619630+0000) 2022-01-31T22:35:53.966 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:53 smithi167 conmon[49112]: debug 2022-01-31T22:35:53.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:53.748963+0000) 2022-01-31T22:35:53.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:53 smithi167 conmon[54076]: debug 2022-01-31T22:35:53.914+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:53.916454+0000) 2022-01-31T22:35:53.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:53 smithi167 conmon[60316]: debug 2022-01-31T22:35:53.673+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:53.675408+0000) 2022-01-31T22:35:54.278 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:53 smithi171 conmon[35325]: debug 2022-01-31T22:35:53.878+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:35:54.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:54 smithi171 conmon[46715]: debug 2022-01-31T22:35:54.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:54.416342+0000) 2022-01-31T22:35:54.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:54 smithi171 conmon[51620]: debug 2022-01-31T22:35:54.277+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:54.279082+0000) 2022-01-31T22:35:54.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:54 smithi171 conmon[41853]: debug 2022-01-31T22:35:54.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:54.619796+0000) 2022-01-31T22:35:54.966 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:54 smithi167 conmon[60316]: debug 2022-01-31T22:35:54.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:54.675617+0000) 2022-01-31T22:35:54.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:54 smithi167 conmon[49112]: debug 2022-01-31T22:35:54.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:54.749150+0000) 2022-01-31T22:35:54.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:54 smithi167 conmon[54076]: debug 2022-01-31T22:35:54.915+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:54.916658+0000) 2022-01-31T22:35:55.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:55 smithi171 conmon[46715]: debug 2022-01-31T22:35:55.414+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:55.416543+0000) 2022-01-31T22:35:55.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:55 smithi171 conmon[51620]: debug 2022-01-31T22:35:55.277+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:55.279284+0000) 2022-01-31T22:35:55.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:55 smithi171 conmon[41853]: debug 2022-01-31T22:35:55.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:55.619943+0000) 2022-01-31T22:35:55.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:55 smithi167 conmon[54076]: debug 2022-01-31T22:35:55.915+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:55.916806+0000) 2022-01-31T22:35:55.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:55 smithi167 conmon[60316]: debug 2022-01-31T22:35:55.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:55.675756+0000) 2022-01-31T22:35:55.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:55 smithi167 conmon[49112]: debug 2022-01-31T22:35:55.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:55.749308+0000) 2022-01-31T22:35:56.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:56 smithi171 conmon[46715]: debug 2022-01-31T22:35:56.415+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:56.416731+0000) 2022-01-31T22:35:56.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:56 smithi171 conmon[51620]: debug 2022-01-31T22:35:56.277+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:56.279467+0000) 2022-01-31T22:35:56.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:56 smithi171 conmon[41853]: debug 2022-01-31T22:35:56.618+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:56.620083+0000) 2022-01-31T22:35:56.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:56 smithi167 conmon[60316]: debug 2022-01-31T22:35:56.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:56.675973+0000) 2022-01-31T22:35:56.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:56 smithi167 conmon[49112]: debug 2022-01-31T22:35:56.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:56.749455+0000) 2022-01-31T22:35:56.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:56 smithi167 conmon[54076]: debug 2022-01-31T22:35:56.916+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:56.916988+0000) 2022-01-31T22:35:57.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[46715]: debug 2022-01-31T22:35:57.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.416936+0000) 2022-01-31T22:35:57.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[51620]: debug 2022-01-31T22:35:57.278+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.279620+0000) 2022-01-31T22:35:57.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[35325]: debug 2022-01-31T22:35:57.779+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222334 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:35:57.856 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[51620]: debug 2022-01-31T22:35:57.757+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.759509+0000) 2022-01-31T22:35:57.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[41853]: debug 2022-01-31T22:35:57.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.620213+0000) 2022-01-31T22:35:57.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[41853]: debug 2022-01-31T22:35:57.757+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.759253+0000) 2022-01-31T22:35:57.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[46715]: debug 2022-01-31T22:35:57.858 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:57 smithi171 conmon[46715]: 2022-01-31T22:35:57.758+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.760419+0000) 2022-01-31T22:35:57.967 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:57 smithi167 conmon[54076]: debug 2022-01-31T22:35:57.758+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.760127+0000) 2022-01-31T22:35:57.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:57 smithi167 conmon[54076]: debug 2022-01-31T22:35:57.916+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.917184+0000) 2022-01-31T22:35:57.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:57 smithi167 conmon[60316]: debug 2022-01-31T22:35:57.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.676104+0000) 2022-01-31T22:35:57.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:57 smithi167 conmon[60316]: debug 2022-01-31T22:35:57.759+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.760681+0000) 2022-01-31T22:35:57.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:57 smithi167 conmon[49112]: debug 2022-01-31T22:35:57.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.749614+0000) 2022-01-31T22:35:57.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:57 smithi167 conmon[49112]: debug 2022-01-31T22:35:57.759+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:57.760950+0000) 2022-01-31T22:35:58.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:58 smithi171 conmon[46715]: debug 2022-01-31T22:35:58.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:58.417134+0000) 2022-01-31T22:35:58.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:58 smithi171 conmon[51620]: debug 2022-01-31T22:35:58.278+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:58.279748+0000) 2022-01-31T22:35:58.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:58 smithi171 conmon[41853]: debug 2022-01-31T22:35:58.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:58.620364+0000) 2022-01-31T22:35:58.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:58 smithi167 conmon[49112]: debug 2022-01-31T22:35:58.748+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:58.749750+0000) 2022-01-31T22:35:58.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:58 smithi167 conmon[54076]: debug 2022-01-31T22:35:58.916+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:58.917353+0000) 2022-01-31T22:35:58.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:58 smithi167 conmon[60316]: debug 2022-01-31T22:35:58.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:58.676304+0000) 2022-01-31T22:35:59.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:35:59 smithi171 conmon[46715]: debug 2022-01-31T22:35:59.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:59.417262+0000) 2022-01-31T22:35:59.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:35:59 smithi171 conmon[51620]: debug 2022-01-31T22:35:59.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:59.279863+0000) 2022-01-31T22:35:59.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:35:59 smithi171 conmon[41853]: debug 2022-01-31T22:35:59.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:59.620515+0000) 2022-01-31T22:35:59.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:35:59 smithi167 conmon[49112]: debug 2022-01-31T22:35:59.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:59.749926+0000) 2022-01-31T22:35:59.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:35:59 smithi167 conmon[54076]: debug 2022-01-31T22:35:59.916+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:59.917514+0000) 2022-01-31T22:35:59.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:35:59 smithi167 conmon[60316]: debug 2022-01-31T22:35:59.674+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:35:59.676456+0000) 2022-01-31T22:36:00.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:00 smithi171 conmon[46715]: debug 2022-01-31T22:36:00.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:00.417467+0000) 2022-01-31T22:36:00.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:00 smithi171 conmon[51620]: debug 2022-01-31T22:36:00.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:00.279994+0000) 2022-01-31T22:36:00.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:00 smithi171 conmon[41853]: debug 2022-01-31T22:36:00.619+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:00.620696+0000) 2022-01-31T22:36:00.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:00 smithi167 conmon[49112]: debug 2022-01-31T22:36:00.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:00.750110+0000) 2022-01-31T22:36:00.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:00 smithi167 conmon[54076]: debug 2022-01-31T22:36:00.917+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:00.917678+0000) 2022-01-31T22:36:00.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:00 smithi167 conmon[60316]: debug 2022-01-31T22:36:00.675+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:00.676626+0000) 2022-01-31T22:36:01.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:01 smithi171 conmon[46715]: debug 2022-01-31T22:36:01.416+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:01.417667+0000) 2022-01-31T22:36:01.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:01 smithi171 conmon[51620]: debug 2022-01-31T22:36:01.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:01.280089+0000) 2022-01-31T22:36:01.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:01 smithi171 conmon[41853]: debug 2022-01-31T22:36:01.620+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:01.620916+0000) 2022-01-31T22:36:01.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:01 smithi167 conmon[49112]: debug 2022-01-31T22:36:01.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:01.750292+0000) 2022-01-31T22:36:01.968 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:01 smithi167 conmon[54076]: debug 2022-01-31T22:36:01.916+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:01.917803+0000) 2022-01-31T22:36:01.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:01 smithi167 conmon[60316]: debug 2022-01-31T22:36:01.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:01.676738+0000) 2022-01-31T22:36:02.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[46715]: debug 2022-01-31T22:36:02.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.417821+0000) 2022-01-31T22:36:02.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[51620]: debug 2022-01-31T22:36:02.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.280226+0000) 2022-01-31T22:36:02.856 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[35325]: debug 2022-01-31T22:36:02.820+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222446 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:02.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[46715]: debug 2022-01-31T22:36:02.783+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.784350+0000) 2022-01-31T22:36:02.857 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[51620]: debug 2022-01-31T22:36:02.782+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.783353+0000) 2022-01-31T22:36:02.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[41853]: debug 2022-01-31T22:36:02.620+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.621072+0000) 2022-01-31T22:36:02.858 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:02 smithi171 conmon[41853]: debug 2022-01-31T22:36:02.782+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.783139+0000) 2022-01-31T22:36:02.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:02 smithi167 conmon[49112]: debug 2022-01-31T22:36:02.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.750461+0000) 2022-01-31T22:36:02.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:02 smithi167 conmon[49112]: debug 2022-01-31T22:36:02.788+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.790252+0000) 2022-01-31T22:36:02.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:02 smithi167 conmon[54076]: debug 2022-01-31T22:36:02.781+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.783316+0000) 2022-01-31T22:36:02.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:02 smithi167 conmon[54076]: debug 2022-01-31T22:36:02.916+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.917977+0000) 2022-01-31T22:36:02.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:02 smithi167 conmon[60316]: debug 2022-01-31T22:36:02.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.676874+0000) 2022-01-31T22:36:02.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:02 smithi167 conmon[60316]: debug 2022-01-31T22:36:02.782+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:02.783674+0000) 2022-01-31T22:36:03.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:03 smithi171 conmon[46715]: debug 2022-01-31T22:36:03.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:03.418014+0000) 2022-01-31T22:36:03.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:03 smithi171 conmon[51620]: debug 2022-01-31T22:36:03.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:03.280452+0000) 2022-01-31T22:36:03.856 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:03 smithi171 conmon[41853]: debug 2022-01-31T22:36:03.620+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:03.621249+0000) 2022-01-31T22:36:03.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:03 smithi167 conmon[49112]: debug 2022-01-31T22:36:03.749+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:03.750586+0000) 2022-01-31T22:36:03.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:03 smithi167 conmon[54076]: debug 2022-01-31T22:36:03.917+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:03.918120+0000) 2022-01-31T22:36:03.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:03 smithi167 conmon[60316]: debug 2022-01-31T22:36:03.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:03.677033+0000) 2022-01-31T22:36:04.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:04 smithi171 conmon[46715]: debug 2022-01-31T22:36:04.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:04.418195+0000) 2022-01-31T22:36:04.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:04 smithi171 conmon[51620]: debug 2022-01-31T22:36:04.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:04.280600+0000) 2022-01-31T22:36:04.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:04 smithi171 conmon[41853]: debug 2022-01-31T22:36:04.620+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:04.621404+0000) 2022-01-31T22:36:04.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:04 smithi167 conmon[49112]: debug 2022-01-31T22:36:04.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:04.750756+0000) 2022-01-31T22:36:04.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:04 smithi167 conmon[54076]: debug 2022-01-31T22:36:04.917+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:04.918308+0000) 2022-01-31T22:36:04.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:04 smithi167 conmon[60316]: debug 2022-01-31T22:36:04.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:04.677213+0000) 2022-01-31T22:36:05.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:05 smithi171 conmon[46715]: debug 2022-01-31T22:36:05.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:05.418397+0000) 2022-01-31T22:36:05.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:05 smithi171 conmon[51620]: debug 2022-01-31T22:36:05.279+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:05.280719+0000) 2022-01-31T22:36:05.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:05 smithi171 conmon[41853]: debug 2022-01-31T22:36:05.620+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:05.621590+0000) 2022-01-31T22:36:05.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:05 smithi167 conmon[49112]: debug 2022-01-31T22:36:05.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:05.750973+0000) 2022-01-31T22:36:05.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:05 smithi167 conmon[54076]: debug 2022-01-31T22:36:05.918+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:05.918524+0000) 2022-01-31T22:36:05.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:05 smithi167 conmon[60316]: debug 2022-01-31T22:36:05.676+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:05.677405+0000) 2022-01-31T22:36:06.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:06 smithi171 conmon[46715]: debug 2022-01-31T22:36:06.417+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:06.418585+0000) 2022-01-31T22:36:06.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:06 smithi171 conmon[51620]: debug 2022-01-31T22:36:06.280+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:06.280861+0000) 2022-01-31T22:36:06.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:06 smithi171 conmon[41853]: debug 2022-01-31T22:36:06.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:06.621795+0000) 2022-01-31T22:36:06.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:06 smithi167 conmon[49112]: debug 2022-01-31T22:36:06.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:06.751164+0000) 2022-01-31T22:36:06.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:06 smithi167 conmon[54076]: debug 2022-01-31T22:36:06.918+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:06.918707+0000) 2022-01-31T22:36:06.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:06 smithi167 conmon[60316]: debug 2022-01-31T22:36:06.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:06.677604+0000) 2022-01-31T22:36:07.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[46715]: debug 2022-01-31T22:36:07.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.418798+0000) 2022-01-31T22:36:07.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[51620]: debug 2022-01-31T22:36:07.280+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.280994+0000) 2022-01-31T22:36:07.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[46715]: debug 2022-01-31T22:36:07.824+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.825370+0000) 2022-01-31T22:36:07.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[51620]: debug 2022-01-31T22:36:07.823+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.824331+0000) 2022-01-31T22:36:07.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[41853]: debug 2022-01-31T22:36:07.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.621935+0000) 2022-01-31T22:36:07.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[41853]: debug 2022-01-31T22:36:07.822+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.823477+0000) 2022-01-31T22:36:07.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:07 smithi167 conmon[49112]: debug 2022-01-31T22:36:07.750+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.751345+0000) 2022-01-31T22:36:07.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:07 smithi167 conmon[49112]: debug 2022-01-31T22:36:07.824+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.825355+0000) 2022-01-31T22:36:07.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:07 smithi167 conmon[60316]: debug 2022-01-31T22:36:07.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.677787+0000) 2022-01-31T22:36:07.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:07 smithi167 conmon[60316]: debug 2022-01-31T22:36:07.824+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.824701+0000) 2022-01-31T22:36:07.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:07 smithi167 conmon[54076]: debug 2022-01-31T22:36:07.824+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.824562+0000) 2022-01-31T22:36:07.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:07 smithi167 conmon[54076]: debug 2022-01-31T22:36:07.918+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:07.918840+0000) 2022-01-31T22:36:08.106 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:07 smithi171 conmon[35325]: debug 2022-01-31T22:36:07.846+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222557 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:08.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:08 smithi171 conmon[46715]: debug 2022-01-31T22:36:08.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:08.418927+0000) 2022-01-31T22:36:08.582 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:08 smithi171 conmon[51620]: debug 2022-01-31T22:36:08.280+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:08.281111+0000) 2022-01-31T22:36:08.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:08 smithi171 conmon[41853]: debug 2022-01-31T22:36:08.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:08.622101+0000) 2022-01-31T22:36:08.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:08 smithi167 conmon[49112]: debug 2022-01-31T22:36:08.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:08.751539+0000) 2022-01-31T22:36:08.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:08 smithi167 conmon[54076]: debug 2022-01-31T22:36:08.918+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:08.918924+0000) 2022-01-31T22:36:08.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:08 smithi167 conmon[60316]: debug 2022-01-31T22:36:08.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:08.677985+0000) 2022-01-31T22:36:09.279 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:08 smithi171 conmon[35325]: debug 2022-01-31T22:36:08.879+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:36:09.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:09 smithi171 conmon[46715]: debug 2022-01-31T22:36:09.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:09.419064+0000) 2022-01-31T22:36:09.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:09 smithi171 conmon[51620]: debug 2022-01-31T22:36:09.280+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:09.281291+0000) 2022-01-31T22:36:09.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:09 smithi171 conmon[41853]: debug 2022-01-31T22:36:09.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:09.622236+0000) 2022-01-31T22:36:09.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:09 smithi167 conmon[49112]: debug 2022-01-31T22:36:09.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:09.751705+0000) 2022-01-31T22:36:09.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:09 smithi167 conmon[60316]: debug 2022-01-31T22:36:09.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:09.678153+0000) 2022-01-31T22:36:09.970 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:09 smithi167 conmon[54076]: debug 2022-01-31T22:36:09.918+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:09.919089+0000) 2022-01-31T22:36:10.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:10 smithi171 conmon[46715]: debug 2022-01-31T22:36:10.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:10.419250+0000) 2022-01-31T22:36:10.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:10 smithi171 conmon[51620]: debug 2022-01-31T22:36:10.280+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:10.281502+0000) 2022-01-31T22:36:10.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:10 smithi171 conmon[41853]: debug 2022-01-31T22:36:10.621+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:10.622445+0000) 2022-01-31T22:36:10.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:10 smithi167 conmon[49112]: debug 2022-01-31T22:36:10.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:10.751879+0000) 2022-01-31T22:36:10.970 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:10 smithi167 conmon[54076]: debug 2022-01-31T22:36:10.918+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:10.919273+0000) 2022-01-31T22:36:10.971 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:10 smithi167 conmon[60316]: debug 2022-01-31T22:36:10.677+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:10.678364+0000) 2022-01-31T22:36:11.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:11 smithi171 conmon[46715]: debug 2022-01-31T22:36:11.418+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:11.419420+0000) 2022-01-31T22:36:11.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:11 smithi171 conmon[51620]: debug 2022-01-31T22:36:11.281+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:11.281699+0000) 2022-01-31T22:36:11.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:11 smithi171 conmon[41853]: debug 2022-01-31T22:36:11.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:11.622628+0000) 2022-01-31T22:36:11.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:11 smithi167 conmon[54076]: debug 2022-01-31T22:36:11.919+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:11.919460+0000) 2022-01-31T22:36:11.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:11 smithi167 conmon[60316]: debug 2022-01-31T22:36:11.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:11.678555+0000) 2022-01-31T22:36:11.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:11 smithi167 conmon[49112]: debug 2022-01-31T22:36:11.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:11.752035+0000) 2022-01-31T22:36:12.582 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[46715]: debug 2022-01-31T22:36:12.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.419626+0000) 2022-01-31T22:36:12.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[51620]: debug 2022-01-31T22:36:12.281+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.281870+0000) 2022-01-31T22:36:12.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[41853]: debug 2022-01-31T22:36:12.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.622811+0000) 2022-01-31T22:36:12.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:12 smithi167 conmon[49112]: debug 2022-01-31T22:36:12.751+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.752229+0000) 2022-01-31T22:36:12.918 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:12 smithi167 conmon[49112]: debug 2022-01-31T22:36:12.850+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.851044+0000) 2022-01-31T22:36:12.918 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:12 smithi167 conmon[54076]: debug 2022-01-31T22:36:12.849+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.850110+0000) 2022-01-31T22:36:12.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:12 smithi167 conmon[60316]: debug 2022-01-31T22:36:12.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.678754+0000) 2022-01-31T22:36:12.919 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:12 smithi167 conmon[60316]: debug 2022-01-31T22:36:12.849+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.850235+0000) 2022-01-31T22:36:13.105 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[35325]: debug 2022-01-31T22:36:12.871+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222669 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:13.106 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[41853]: debug 2022-01-31T22:36:12.849+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.850258+0000) 2022-01-31T22:36:13.107 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[46715]: debug 2022-01-31T22:36:12.849+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.850715+0000) 2022-01-31T22:36:13.107 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:12 smithi171 conmon[51620]: debug 2022-01-31T22:36:12.848+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.849323+0000) 2022-01-31T22:36:13.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:12 smithi167 conmon[54076]: debug 2022-01-31T22:36:12.919+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:12.919721+0000) 2022-01-31T22:36:13.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:13 smithi171 conmon[46715]: debug 2022-01-31T22:36:13.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:13.419829+0000) 2022-01-31T22:36:13.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:13 smithi171 conmon[51620]: debug 2022-01-31T22:36:13.281+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:13.282040+0000) 2022-01-31T22:36:13.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:13 smithi171 conmon[41853]: debug 2022-01-31T22:36:13.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:13.622979+0000) 2022-01-31T22:36:13.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:13 smithi167 conmon[49112]: debug 2022-01-31T22:36:13.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:13.752411+0000) 2022-01-31T22:36:13.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:13 smithi167 conmon[60316]: debug 2022-01-31T22:36:13.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:13.678916+0000) 2022-01-31T22:36:14.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:13 smithi167 conmon[54076]: debug 2022-01-31T22:36:13.919+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:13.919837+0000) 2022-01-31T22:36:14.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:14 smithi171 conmon[46715]: debug 2022-01-31T22:36:14.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:14.420007+0000) 2022-01-31T22:36:14.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:14 smithi171 conmon[51620]: debug 2022-01-31T22:36:14.281+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:14.282189+0000) 2022-01-31T22:36:14.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:14 smithi171 conmon[41853]: debug 2022-01-31T22:36:14.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:14.623135+0000) 2022-01-31T22:36:14.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:14 smithi167 conmon[49112]: debug 2022-01-31T22:36:14.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:14.752568+0000) 2022-01-31T22:36:14.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:14 smithi167 conmon[60316]: debug 2022-01-31T22:36:14.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:14.679098+0000) 2022-01-31T22:36:15.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:14 smithi167 conmon[54076]: debug 2022-01-31T22:36:14.919+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:14.919963+0000) 2022-01-31T22:36:15.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:15 smithi171 conmon[46715]: debug 2022-01-31T22:36:15.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:15.420203+0000) 2022-01-31T22:36:15.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:15 smithi171 conmon[51620]: debug 2022-01-31T22:36:15.281+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:15.282343+0000) 2022-01-31T22:36:15.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:15 smithi171 conmon[41853]: debug 2022-01-31T22:36:15.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:15.623346+0000) 2022-01-31T22:36:15.917 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:15 smithi167 conmon[49112]: debug 2022-01-31T22:36:15.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:15.752754+0000) 2022-01-31T22:36:15.918 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:15 smithi167 conmon[60316]: debug 2022-01-31T22:36:15.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:15.679294+0000) 2022-01-31T22:36:16.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:15 smithi167 conmon[54076]: debug 2022-01-31T22:36:15.919+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:15.920066+0000) 2022-01-31T22:36:16.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:16 smithi171 conmon[46715]: debug 2022-01-31T22:36:16.419+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:16.420392+0000) 2022-01-31T22:36:16.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:16 smithi171 conmon[51620]: debug 2022-01-31T22:36:16.281+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:16.282522+0000) 2022-01-31T22:36:16.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:16 smithi171 conmon[41853]: debug 2022-01-31T22:36:16.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:16.623531+0000) 2022-01-31T22:36:16.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:16 smithi167 conmon[49112]: debug 2022-01-31T22:36:16.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:16.752911+0000) 2022-01-31T22:36:16.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:16 smithi167 conmon[54076]: debug 2022-01-31T22:36:16.919+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:16.920201+0000) 2022-01-31T22:36:16.971 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:16 smithi167 conmon[60316]: debug 2022-01-31T22:36:16.678+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:16.679475+0000) 2022-01-31T22:36:17.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[46715]: debug 2022-01-31T22:36:17.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.420579+0000) 2022-01-31T22:36:17.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[51620]: debug 2022-01-31T22:36:17.282+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.282746+0000) 2022-01-31T22:36:17.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[41853]: debug 2022-01-31T22:36:17.622+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.623627+0000) 2022-01-31T22:36:17.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:17 smithi167 conmon[49112]: debug 2022-01-31T22:36:17.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.753119+0000) 2022-01-31T22:36:17.971 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:17 smithi167 conmon[49112]: debug 2022-01-31T22:36:17.874+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.875310+0000) 2022-01-31T22:36:17.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:17 smithi167 conmon[54076]: debug 2022-01-31T22:36:17.875+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.875822+0000) 2022-01-31T22:36:17.972 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:17 smithi167 conmon[54076]: debug 2022-01-31T22:36:17.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.920365+0000) 2022-01-31T22:36:17.972 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:17 smithi167 conmon[60316]: debug 2022-01-31T22:36:17.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.679619+0000) 2022-01-31T22:36:17.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:17 smithi167 conmon[60316]: debug 2022-01-31T22:36:17.874+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.875442+0000) 2022-01-31T22:36:18.280 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[35325]: debug 2022-01-31T22:36:17.898+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222780 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:18.281 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[41853]: debug 2022-01-31T22:36:17.874+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.875586+0000) 2022-01-31T22:36:18.282 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[46715]: debug 2022-01-31T22:36:17.875+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.876502+0000) 2022-01-31T22:36:18.282 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:17 smithi171 conmon[51620]: debug 2022-01-31T22:36:17.873+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:17.874352+0000) 2022-01-31T22:36:18.583 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:18 smithi171 conmon[51620]: debug 2022-01-31T22:36:18.282+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:18.282858+0000) 2022-01-31T22:36:18.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:18 smithi171 conmon[46715]: debug 2022-01-31T22:36:18.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:18.420778+0000) 2022-01-31T22:36:18.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:18 smithi171 conmon[41853]: debug 2022-01-31T22:36:18.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:18.623765+0000) 2022-01-31T22:36:18.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:18 smithi167 conmon[49112]: debug 2022-01-31T22:36:18.752+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:18.753309+0000) 2022-01-31T22:36:18.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:18 smithi167 conmon[54076]: debug 2022-01-31T22:36:18.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:18.920536+0000) 2022-01-31T22:36:18.971 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:18 smithi167 conmon[60316]: debug 2022-01-31T22:36:18.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:18.679732+0000) 2022-01-31T22:36:19.583 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:19 smithi171 conmon[46715]: debug 2022-01-31T22:36:19.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:19.420932+0000) 2022-01-31T22:36:19.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:19 smithi171 conmon[51620]: debug 2022-01-31T22:36:19.282+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:19.283018+0000) 2022-01-31T22:36:19.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:19 smithi171 conmon[41853]: debug 2022-01-31T22:36:19.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:19.623908+0000) 2022-01-31T22:36:19.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:19 smithi167 conmon[49112]: debug 2022-01-31T22:36:19.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:19.753535+0000) 2022-01-31T22:36:19.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:19 smithi167 conmon[54076]: debug 2022-01-31T22:36:19.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:19.920682+0000) 2022-01-31T22:36:19.972 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:19 smithi167 conmon[60316]: debug 2022-01-31T22:36:19.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:19.679848+0000) 2022-01-31T22:36:20.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:20 smithi171 conmon[46715]: debug 2022-01-31T22:36:20.420+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:20.421135+0000) 2022-01-31T22:36:20.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:20 smithi171 conmon[51620]: debug 2022-01-31T22:36:20.282+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:20.283206+0000) 2022-01-31T22:36:20.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:20 smithi171 conmon[41853]: debug 2022-01-31T22:36:20.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:20.624084+0000) 2022-01-31T22:36:20.971 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:20 smithi167 conmon[49112]: debug 2022-01-31T22:36:20.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:20.753695+0000) 2022-01-31T22:36:20.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:20 smithi167 conmon[54076]: debug 2022-01-31T22:36:20.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:20.920899+0000) 2022-01-31T22:36:20.972 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:20 smithi167 conmon[60316]: debug 2022-01-31T22:36:20.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:20.680040+0000) 2022-01-31T22:36:21.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:21 smithi171 conmon[46715]: debug 2022-01-31T22:36:21.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:21.421322+0000) 2022-01-31T22:36:21.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:21 smithi171 conmon[51620]: debug 2022-01-31T22:36:21.282+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:21.283369+0000) 2022-01-31T22:36:21.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:21 smithi171 conmon[41853]: debug 2022-01-31T22:36:21.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:21.624210+0000) 2022-01-31T22:36:21.971 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:21 smithi167 conmon[49112]: debug 2022-01-31T22:36:21.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:21.753852+0000) 2022-01-31T22:36:21.971 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:21 smithi167 conmon[54076]: debug 2022-01-31T22:36:21.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:21.921058+0000) 2022-01-31T22:36:21.972 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:21 smithi167 conmon[60316]: debug 2022-01-31T22:36:21.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:21.680213+0000) 2022-01-31T22:36:22.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[46715]: debug 2022-01-31T22:36:22.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.421534+0000) 2022-01-31T22:36:22.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[51620]: debug 2022-01-31T22:36:22.283+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.283545+0000) 2022-01-31T22:36:22.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[41853]: debug 2022-01-31T22:36:22.623+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.624311+0000) 2022-01-31T22:36:22.971 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:22 smithi167 conmon[49112]: debug 2022-01-31T22:36:22.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.754006+0000) 2022-01-31T22:36:22.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:22 smithi167 conmon[49112]: debug 2022-01-31T22:36:22.901+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.902173+0000) 2022-01-31T22:36:22.972 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:22 smithi167 conmon[60316]: debug 2022-01-31T22:36:22.679+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.680421+0000) 2022-01-31T22:36:22.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:22 smithi167 conmon[60316]: debug 2022-01-31T22:36:22.902+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.902665+0000) 2022-01-31T22:36:22.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:22 smithi167 conmon[54076]: debug 2022-01-31T22:36:22.901+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.902331+0000) 2022-01-31T22:36:22.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:22 smithi167 conmon[54076]: debug 2022-01-31T22:36:22.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.921214+0000) 2022-01-31T22:36:23.281 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[35325]: debug 2022-01-31T22:36:22.923+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 222892 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:23.282 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[41853]: debug 2022-01-31T22:36:22.900+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.901186+0000) 2022-01-31T22:36:23.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[46715]: debug 2022-01-31T22:36:22.901+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.902691+0000) 2022-01-31T22:36:23.283 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:22 smithi171 conmon[51620]: debug 2022-01-31T22:36:22.900+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:22.900863+0000) 2022-01-31T22:36:23.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:23 smithi171 conmon[46715]: debug 2022-01-31T22:36:23.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:23.421722+0000) 2022-01-31T22:36:23.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:23 smithi171 conmon[51620]: debug 2022-01-31T22:36:23.283+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:23.283805+0000) 2022-01-31T22:36:23.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:23 smithi171 conmon[41853]: debug 2022-01-31T22:36:23.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:23.624482+0000) 2022-01-31T22:36:23.971 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:23 smithi167 conmon[49112]: debug 2022-01-31T22:36:23.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:23.754186+0000) 2022-01-31T22:36:23.974 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:23 smithi167 conmon[60316]: debug 2022-01-31T22:36:23.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:23.680583+0000) 2022-01-31T22:36:23.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:23 smithi167 conmon[54076]: debug 2022-01-31T22:36:23.920+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:23.921360+0000) 2022-01-31T22:36:24.281 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:23 smithi171 conmon[35325]: debug 2022-01-31T22:36:23.880+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:36:24.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:24 smithi171 conmon[46715]: debug 2022-01-31T22:36:24.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:24.421927+0000) 2022-01-31T22:36:24.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:24 smithi171 conmon[51620]: debug 2022-01-31T22:36:24.283+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:24.284006+0000) 2022-01-31T22:36:24.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:24 smithi171 conmon[41853]: debug 2022-01-31T22:36:24.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:24.624648+0000) 2022-01-31T22:36:24.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:24 smithi167 conmon[49112]: debug 2022-01-31T22:36:24.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:24.754314+0000) 2022-01-31T22:36:24.972 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:24 smithi167 conmon[54076]: debug 2022-01-31T22:36:24.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:24.921544+0000) 2022-01-31T22:36:24.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:24 smithi167 conmon[60316]: debug 2022-01-31T22:36:24.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:24.680766+0000) 2022-01-31T22:36:25.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:25 smithi171 conmon[51620]: debug 2022-01-31T22:36:25.283+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:25.284137+0000) 2022-01-31T22:36:25.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:25 smithi171 conmon[46715]: debug 2022-01-31T22:36:25.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:25.422114+0000) 2022-01-31T22:36:25.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:25 smithi171 conmon[41853]: debug 2022-01-31T22:36:25.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:25.624821+0000) 2022-01-31T22:36:25.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:25 smithi167 conmon[49112]: debug 2022-01-31T22:36:25.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:25.754470+0000) 2022-01-31T22:36:25.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:25 smithi167 conmon[54076]: debug 2022-01-31T22:36:25.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:25.921726+0000) 2022-01-31T22:36:25.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:25 smithi167 conmon[60316]: debug 2022-01-31T22:36:25.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:25.680957+0000) 2022-01-31T22:36:26.584 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:26 smithi171 conmon[46715]: debug 2022-01-31T22:36:26.421+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:26.422276+0000) 2022-01-31T22:36:26.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:26 smithi171 conmon[51620]: debug 2022-01-31T22:36:26.283+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:26.284325+0000) 2022-01-31T22:36:26.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:26 smithi171 conmon[41853]: debug 2022-01-31T22:36:26.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:26.624940+0000) 2022-01-31T22:36:26.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:26 smithi167 conmon[49112]: debug 2022-01-31T22:36:26.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:26.754648+0000) 2022-01-31T22:36:26.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:26 smithi167 conmon[54076]: debug 2022-01-31T22:36:26.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:26.921884+0000) 2022-01-31T22:36:26.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:26 smithi167 conmon[60316]: debug 2022-01-31T22:36:26.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:26.681166+0000) 2022-01-31T22:36:27.584 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[51620]: debug 2022-01-31T22:36:27.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.284520+0000) 2022-01-31T22:36:27.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[46715]: debug 2022-01-31T22:36:27.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.422442+0000) 2022-01-31T22:36:27.859 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[41853]: debug 2022-01-31T22:36:27.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.625135+0000) 2022-01-31T22:36:27.972 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:27 smithi167 conmon[54076]: debug 2022-01-31T22:36:27.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.922033+0000) 2022-01-31T22:36:27.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:27 smithi167 conmon[54076]: debug 2022-01-31T22:36:27.927+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.928433+0000) 2022-01-31T22:36:27.974 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:27 smithi167 conmon[49112]: debug 2022-01-31T22:36:27.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.754898+0000) 2022-01-31T22:36:27.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:27 smithi167 conmon[49112]: debug 2022-01-31T22:36:27.926+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.927359+0000) 2022-01-31T22:36:27.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:27 smithi167 conmon[60316]: debug 2022-01-31T22:36:27.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.681370+0000) 2022-01-31T22:36:27.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:27 smithi167 conmon[60316]: debug 2022-01-31T22:36:27.927+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.927851+0000) 2022-01-31T22:36:28.282 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[35325]: debug 2022-01-31T22:36:27.950+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223005 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:28.283 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[41853]: debug 2022-01-31T22:36:27.925+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.926303+0000) 2022-01-31T22:36:28.283 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[46715]: debug 2022-01-31T22:36:27.926+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.927423+0000) 2022-01-31T22:36:28.284 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:27 smithi171 conmon[51620]: debug 2022-01-31T22:36:27.926+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:27.926792+0000) 2022-01-31T22:36:28.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:28 smithi171 conmon[46715]: debug 2022-01-31T22:36:28.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:28.422625+0000) 2022-01-31T22:36:28.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:28 smithi171 conmon[51620]: debug 2022-01-31T22:36:28.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:28.284688+0000) 2022-01-31T22:36:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:28 smithi171 conmon[41853]: debug 2022-01-31T22:36:28.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:28.625310+0000) 2022-01-31T22:36:28.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:28 smithi167 conmon[49112]: debug 2022-01-31T22:36:28.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:28.755138+0000) 2022-01-31T22:36:28.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:28 smithi167 conmon[54076]: debug 2022-01-31T22:36:28.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:28.922216+0000) 2022-01-31T22:36:28.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:28 smithi167 conmon[60316]: debug 2022-01-31T22:36:28.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:28.681579+0000) 2022-01-31T22:36:29.585 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:29 smithi171 conmon[51620]: debug 2022-01-31T22:36:29.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:29.284872+0000) 2022-01-31T22:36:29.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:29 smithi171 conmon[46715]: debug 2022-01-31T22:36:29.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:29.422774+0000) 2022-01-31T22:36:29.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:29 smithi171 conmon[41853]: debug 2022-01-31T22:36:29.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:29.625495+0000) 2022-01-31T22:36:29.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:29 smithi167 conmon[49112]: debug 2022-01-31T22:36:29.753+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:29.755340+0000) 2022-01-31T22:36:29.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:29 smithi167 conmon[54076]: debug 2022-01-31T22:36:29.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:29.922352+0000) 2022-01-31T22:36:29.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:29 smithi167 conmon[60316]: debug 2022-01-31T22:36:29.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:29.681711+0000) 2022-01-31T22:36:30.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:30 smithi171 conmon[46715]: debug 2022-01-31T22:36:30.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:30.422928+0000) 2022-01-31T22:36:30.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:30 smithi171 conmon[51620]: debug 2022-01-31T22:36:30.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:30.285078+0000) 2022-01-31T22:36:30.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:30 smithi171 conmon[41853]: debug 2022-01-31T22:36:30.624+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:30.625717+0000) 2022-01-31T22:36:30.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:30 smithi167 conmon[49112]: debug 2022-01-31T22:36:30.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:30.755539+0000) 2022-01-31T22:36:30.973 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:30 smithi167 conmon[54076]: debug 2022-01-31T22:36:30.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:30.922531+0000) 2022-01-31T22:36:30.974 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:30 smithi167 conmon[60316]: debug 2022-01-31T22:36:30.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:30.681913+0000) 2022-01-31T22:36:31.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:31 smithi171 conmon[46715]: debug 2022-01-31T22:36:31.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:31.423114+0000) 2022-01-31T22:36:31.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:31 smithi171 conmon[51620]: debug 2022-01-31T22:36:31.285+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:31.285284+0000) 2022-01-31T22:36:31.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:31 smithi171 conmon[41853]: debug 2022-01-31T22:36:31.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:31.625872+0000) 2022-01-31T22:36:31.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:31 smithi167 conmon[49112]: debug 2022-01-31T22:36:31.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:31.755725+0000) 2022-01-31T22:36:31.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:31 smithi167 conmon[54076]: debug 2022-01-31T22:36:31.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:31.922723+0000) 2022-01-31T22:36:31.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:31 smithi167 conmon[60316]: debug 2022-01-31T22:36:31.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:31.682104+0000) 2022-01-31T22:36:32.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[46715]: debug 2022-01-31T22:36:32.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.423297+0000) 2022-01-31T22:36:32.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[51620]: debug 2022-01-31T22:36:32.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.285499+0000) 2022-01-31T22:36:32.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[41853]: debug 2022-01-31T22:36:32.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.626019+0000) 2022-01-31T22:36:32.951 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:32 smithi167 conmon[49112]: debug 2022-01-31T22:36:32.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.755933+0000) 2022-01-31T22:36:32.952 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:32 smithi167 conmon[54076]: debug 2022-01-31T22:36:32.921+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.922931+0000) 2022-01-31T22:36:32.953 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:32 smithi167 conmon[60316]: debug 2022-01-31T22:36:32.680+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.682313+0000) 2022-01-31T22:36:33.256 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[35325]: debug 2022-01-31T22:36:32.976+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223117 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:33.257 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[41853]: debug 2022-01-31T22:36:32.953+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.953896+0000) 2022-01-31T22:36:33.257 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[46715]: debug 2022-01-31T22:36:32.953+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.954277+0000) 2022-01-31T22:36:33.258 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:32 smithi171 conmon[51620]: debug 2022-01-31T22:36:32.952+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.953676+0000) 2022-01-31T22:36:33.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:32 smithi167 conmon[49112]: debug 2022-01-31T22:36:32.953+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.955131+0000) 2022-01-31T22:36:33.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:32 smithi167 conmon[54076]: debug 2022-01-31T22:36:32.953+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.955249+0000) 2022-01-31T22:36:33.412 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:32 smithi167 conmon[60316]: debug 2022-01-31T22:36:32.953+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:32.954578+0000) 2022-01-31T22:36:33.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:33 smithi171 conmon[46715]: debug 2022-01-31T22:36:33.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:33.423506+0000) 2022-01-31T22:36:33.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:33 smithi171 conmon[51620]: debug 2022-01-31T22:36:33.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:33.285706+0000) 2022-01-31T22:36:33.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:33 smithi171 conmon[41853]: debug 2022-01-31T22:36:33.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:33.626197+0000) 2022-01-31T22:36:33.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:33 smithi167 conmon[49112]: debug 2022-01-31T22:36:33.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:33.756141+0000) 2022-01-31T22:36:33.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:33 smithi167 conmon[54076]: debug 2022-01-31T22:36:33.922+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:33.923152+0000) 2022-01-31T22:36:33.974 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:33 smithi167 conmon[60316]: debug 2022-01-31T22:36:33.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:33.682517+0000) 2022-01-31T22:36:34.585 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:34 smithi171 conmon[46715]: debug 2022-01-31T22:36:34.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:34.423699+0000) 2022-01-31T22:36:34.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:34 smithi171 conmon[51620]: debug 2022-01-31T22:36:34.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:34.285864+0000) 2022-01-31T22:36:34.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:34 smithi171 conmon[41853]: debug 2022-01-31T22:36:34.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:34.626324+0000) 2022-01-31T22:36:34.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:34 smithi167 conmon[49112]: debug 2022-01-31T22:36:34.754+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:34.756307+0000) 2022-01-31T22:36:34.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:34 smithi167 conmon[54076]: debug 2022-01-31T22:36:34.922+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:34.923296+0000) 2022-01-31T22:36:34.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:34 smithi167 conmon[60316]: debug 2022-01-31T22:36:34.681+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:34.682726+0000) 2022-01-31T22:36:35.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:35 smithi171 conmon[46715]: debug 2022-01-31T22:36:35.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:35.423912+0000) 2022-01-31T22:36:35.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:35 smithi171 conmon[51620]: debug 2022-01-31T22:36:35.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:35.286027+0000) 2022-01-31T22:36:35.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:35 smithi171 conmon[41853]: debug 2022-01-31T22:36:35.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:35.626424+0000) 2022-01-31T22:36:35.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:35 smithi167 conmon[49112]: debug 2022-01-31T22:36:35.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:35.756466+0000) 2022-01-31T22:36:35.974 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:35 smithi167 conmon[60316]: debug 2022-01-31T22:36:35.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:35.682943+0000) 2022-01-31T22:36:35.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:35 smithi167 conmon[54076]: debug 2022-01-31T22:36:35.922+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:35.923485+0000) 2022-01-31T22:36:36.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:36 smithi171 conmon[46715]: debug 2022-01-31T22:36:36.422+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:36.424063+0000) 2022-01-31T22:36:36.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:36 smithi171 conmon[51620]: debug 2022-01-31T22:36:36.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:36.286158+0000) 2022-01-31T22:36:36.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:36 smithi171 conmon[41853]: debug 2022-01-31T22:36:36.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:36.626553+0000) 2022-01-31T22:36:36.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:36 smithi167 conmon[49112]: debug 2022-01-31T22:36:36.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:36.756667+0000) 2022-01-31T22:36:36.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:36 smithi167 conmon[54076]: debug 2022-01-31T22:36:36.922+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:36.923653+0000) 2022-01-31T22:36:36.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:36 smithi167 conmon[60316]: debug 2022-01-31T22:36:36.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:36.683132+0000) 2022-01-31T22:36:37.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:37 smithi171 conmon[46715]: debug 2022-01-31T22:36:37.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.424223+0000) 2022-01-31T22:36:37.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:37 smithi171 conmon[51620]: debug 2022-01-31T22:36:37.284+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.286271+0000) 2022-01-31T22:36:37.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:37 smithi171 conmon[41853]: debug 2022-01-31T22:36:37.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.626650+0000) 2022-01-31T22:36:37.974 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:37 smithi167 conmon[49112]: debug 2022-01-31T22:36:37.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.756824+0000) 2022-01-31T22:36:37.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:37 smithi167 conmon[54076]: debug 2022-01-31T22:36:37.922+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.923847+0000) 2022-01-31T22:36:37.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:37 smithi167 conmon[60316]: debug 2022-01-31T22:36:37.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.683319+0000) 2022-01-31T22:36:38.284 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:38 smithi171 conmon[35325]: debug 2022-01-31T22:36:38.000+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223228 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:38.284 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:37 smithi171 conmon[41853]: debug 2022-01-31T22:36:37.977+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.979001+0000) 2022-01-31T22:36:38.285 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:37 smithi171 conmon[46715]: debug 2022-01-31T22:36:37.977+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.979109+0000) 2022-01-31T22:36:38.286 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:37 smithi171 conmon[51620]: debug 2022-01-31T22:36:37.977+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.979353+0000) 2022-01-31T22:36:38.361 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:37 smithi167 conmon[49112]: debug 2022-01-31T22:36:37.979+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.980669+0000) 2022-01-31T22:36:38.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:37 smithi167 conmon[54076]: debug 2022-01-31T22:36:37.979+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.980759+0000) 2022-01-31T22:36:38.363 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:37 smithi167 conmon[60316]: debug 2022-01-31T22:36:37.977+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:37.979011+0000) 2022-01-31T22:36:38.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:38 smithi171 conmon[51620]: debug 2022-01-31T22:36:38.285+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:38.286442+0000) 2022-01-31T22:36:38.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:38 smithi171 conmon[46715]: debug 2022-01-31T22:36:38.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:38.424404+0000) 2022-01-31T22:36:38.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:38 smithi171 conmon[41853]: debug 2022-01-31T22:36:38.625+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:38.626816+0000) 2022-01-31T22:36:38.974 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:38 smithi167 conmon[49112]: debug 2022-01-31T22:36:38.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:38.756982+0000) 2022-01-31T22:36:38.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:38 smithi167 conmon[54076]: debug 2022-01-31T22:36:38.922+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:38.924033+0000) 2022-01-31T22:36:38.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:38 smithi167 conmon[60316]: debug 2022-01-31T22:36:38.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:38.683543+0000) 2022-01-31T22:36:39.284 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:38 smithi171 conmon[35325]: debug 2022-01-31T22:36:38.880+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:36:39.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:39 smithi171 conmon[46715]: debug 2022-01-31T22:36:39.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:39.424550+0000) 2022-01-31T22:36:39.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:39 smithi171 conmon[51620]: debug 2022-01-31T22:36:39.285+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:39.286553+0000) 2022-01-31T22:36:39.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:39 smithi171 conmon[41853]: debug 2022-01-31T22:36:39.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:39.626987+0000) 2022-01-31T22:36:39.974 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:39 smithi167 conmon[49112]: debug 2022-01-31T22:36:39.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:39.757156+0000) 2022-01-31T22:36:39.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:39 smithi167 conmon[54076]: debug 2022-01-31T22:36:39.923+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:39.924178+0000) 2022-01-31T22:36:39.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:39 smithi167 conmon[60316]: debug 2022-01-31T22:36:39.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:39.683652+0000) 2022-01-31T22:36:40.586 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:40 smithi171 conmon[46715]: debug 2022-01-31T22:36:40.423+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:40.424774+0000) 2022-01-31T22:36:40.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:40 smithi171 conmon[51620]: debug 2022-01-31T22:36:40.285+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:40.286766+0000) 2022-01-31T22:36:40.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:40 smithi171 conmon[41853]: debug 2022-01-31T22:36:40.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:40.627181+0000) 2022-01-31T22:36:40.974 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:40 smithi167 conmon[49112]: debug 2022-01-31T22:36:40.755+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:40.757285+0000) 2022-01-31T22:36:40.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:40 smithi167 conmon[54076]: debug 2022-01-31T22:36:40.923+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:40.924397+0000) 2022-01-31T22:36:40.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:40 smithi167 conmon[60316]: debug 2022-01-31T22:36:40.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:40.683791+0000) 2022-01-31T22:36:41.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:41 smithi171 conmon[46715]: debug 2022-01-31T22:36:41.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:41.424936+0000) 2022-01-31T22:36:41.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:41 smithi171 conmon[51620]: debug 2022-01-31T22:36:41.286+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:41.286993+0000) 2022-01-31T22:36:41.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:41 smithi171 conmon[41853]: debug 2022-01-31T22:36:41.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:41.627382+0000) 2022-01-31T22:36:41.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:41 smithi167 conmon[49112]: debug 2022-01-31T22:36:41.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:41.757476+0000) 2022-01-31T22:36:41.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:41 smithi167 conmon[54076]: debug 2022-01-31T22:36:41.923+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:41.924532+0000) 2022-01-31T22:36:41.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:41 smithi167 conmon[60316]: debug 2022-01-31T22:36:41.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:41.683988+0000) 2022-01-31T22:36:42.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:42 smithi171 conmon[46715]: debug 2022-01-31T22:36:42.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:42.425120+0000) 2022-01-31T22:36:42.587 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:42 smithi171 conmon[51620]: debug 2022-01-31T22:36:42.286+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:42.287161+0000) 2022-01-31T22:36:42.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:42 smithi171 conmon[41853]: debug 2022-01-31T22:36:42.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:42.627590+0000) 2022-01-31T22:36:42.974 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:42 smithi167 conmon[49112]: debug 2022-01-31T22:36:42.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:42.757693+0000) 2022-01-31T22:36:42.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:42 smithi167 conmon[54076]: debug 2022-01-31T22:36:42.923+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:42.924716+0000) 2022-01-31T22:36:42.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:42 smithi167 conmon[60316]: debug 2022-01-31T22:36:42.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:42.684212+0000) 2022-01-31T22:36:43.284 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[35325]: debug 2022-01-31T22:36:43.025+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223340 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:43.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[41853]: debug 2022-01-31T22:36:43.002+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.004097+0000) 2022-01-31T22:36:43.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[46715]: debug 2022-01-31T22:36:43.002+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.003902+0000) 2022-01-31T22:36:43.286 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[51620]: debug 2022-01-31T22:36:43.002+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.004592+0000) 2022-01-31T22:36:43.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:43 smithi167 conmon[49112]: debug 2022-01-31T22:36:43.003+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.004670+0000) 2022-01-31T22:36:43.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:43 smithi167 conmon[54076]: debug 2022-01-31T22:36:43.003+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.004519+0000) 2022-01-31T22:36:43.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:43 smithi167 conmon[60316]: debug 2022-01-31T22:36:43.003+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.005259+0000) 2022-01-31T22:36:43.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[46715]: debug 2022-01-31T22:36:43.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.425313+0000) 2022-01-31T22:36:43.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[51620]: debug 2022-01-31T22:36:43.285+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.287330+0000) 2022-01-31T22:36:43.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:43 smithi171 conmon[41853]: debug 2022-01-31T22:36:43.626+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.627811+0000) 2022-01-31T22:36:43.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:43 smithi167 conmon[49112]: debug 2022-01-31T22:36:43.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.757889+0000) 2022-01-31T22:36:43.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:43 smithi167 conmon[54076]: debug 2022-01-31T22:36:43.923+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.924834+0000) 2022-01-31T22:36:43.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:43 smithi167 conmon[60316]: debug 2022-01-31T22:36:43.682+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:43.684402+0000) 2022-01-31T22:36:44.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:44 smithi171 conmon[46715]: debug 2022-01-31T22:36:44.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:44.425490+0000) 2022-01-31T22:36:44.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:44 smithi171 conmon[51620]: debug 2022-01-31T22:36:44.286+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:44.287495+0000) 2022-01-31T22:36:44.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:44 smithi171 conmon[41853]: debug 2022-01-31T22:36:44.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:44.627991+0000) 2022-01-31T22:36:44.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:44 smithi167 conmon[49112]: debug 2022-01-31T22:36:44.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:44.758029+0000) 2022-01-31T22:36:44.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:44 smithi167 conmon[54076]: debug 2022-01-31T22:36:44.924+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:44.924952+0000) 2022-01-31T22:36:44.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:44 smithi167 conmon[60316]: debug 2022-01-31T22:36:44.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:44.684542+0000) 2022-01-31T22:36:45.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:45 smithi171 conmon[46715]: debug 2022-01-31T22:36:45.424+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:45.425669+0000) 2022-01-31T22:36:45.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:45 smithi171 conmon[51620]: debug 2022-01-31T22:36:45.286+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:45.287695+0000) 2022-01-31T22:36:45.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:45 smithi171 conmon[41853]: debug 2022-01-31T22:36:45.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:45.628126+0000) 2022-01-31T22:36:45.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:45 smithi167 conmon[49112]: debug 2022-01-31T22:36:45.756+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:45.758231+0000) 2022-01-31T22:36:45.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:45 smithi167 conmon[54076]: debug 2022-01-31T22:36:45.924+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:45.925137+0000) 2022-01-31T22:36:45.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:45 smithi167 conmon[60316]: debug 2022-01-31T22:36:45.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:45.684704+0000) 2022-01-31T22:36:46.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:46 smithi171 conmon[46715]: debug 2022-01-31T22:36:46.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:46.425830+0000) 2022-01-31T22:36:46.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:46 smithi171 conmon[51620]: debug 2022-01-31T22:36:46.287+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:46.287975+0000) 2022-01-31T22:36:46.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:46 smithi171 conmon[41853]: debug 2022-01-31T22:36:46.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:46.628289+0000) 2022-01-31T22:36:46.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:46 smithi167 conmon[49112]: debug 2022-01-31T22:36:46.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:46.758412+0000) 2022-01-31T22:36:46.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:46 smithi167 conmon[54076]: debug 2022-01-31T22:36:46.923+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:46.925339+0000) 2022-01-31T22:36:46.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:46 smithi167 conmon[60316]: debug 2022-01-31T22:36:46.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:46.684882+0000) 2022-01-31T22:36:47.587 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:47 smithi171 conmon[46715]: debug 2022-01-31T22:36:47.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:47.426020+0000) 2022-01-31T22:36:47.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:47 smithi171 conmon[51620]: debug 2022-01-31T22:36:47.287+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:47.288130+0000) 2022-01-31T22:36:47.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:47 smithi171 conmon[41853]: debug 2022-01-31T22:36:47.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:47.628482+0000) 2022-01-31T22:36:47.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:47 smithi167 conmon[49112]: debug 2022-01-31T22:36:47.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:47.758603+0000) 2022-01-31T22:36:47.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:47 smithi167 conmon[54076]: debug 2022-01-31T22:36:47.924+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:47.925542+0000) 2022-01-31T22:36:47.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:47 smithi167 conmon[60316]: debug 2022-01-31T22:36:47.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:47.685031+0000) 2022-01-31T22:36:48.285 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[35325]: debug 2022-01-31T22:36:48.052+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223452 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:48.286 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[41853]: debug 2022-01-31T22:36:48.028+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.029674+0000) 2022-01-31T22:36:48.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[46715]: debug 2022-01-31T22:36:48.028+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.029116+0000) 2022-01-31T22:36:48.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[51620]: debug 2022-01-31T22:36:48.028+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.029366+0000) 2022-01-31T22:36:48.413 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:48 smithi167 conmon[49112]: debug 2022-01-31T22:36:48.028+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.030346+0000) 2022-01-31T22:36:48.414 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:48 smithi167 conmon[54076]: debug 2022-01-31T22:36:48.029+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.030796+0000) 2022-01-31T22:36:48.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:48 smithi167 conmon[60316]: debug 2022-01-31T22:36:48.028+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.029592+0000) 2022-01-31T22:36:48.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[46715]: debug 2022-01-31T22:36:48.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.426203+0000) 2022-01-31T22:36:48.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[51620]: debug 2022-01-31T22:36:48.287+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.288308+0000) 2022-01-31T22:36:48.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:48 smithi171 conmon[41853]: debug 2022-01-31T22:36:48.627+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.628711+0000) 2022-01-31T22:36:48.975 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:48 smithi167 conmon[54076]: debug 2022-01-31T22:36:48.925+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.925699+0000) 2022-01-31T22:36:48.976 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:48 smithi167 conmon[60316]: debug 2022-01-31T22:36:48.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.685146+0000) 2022-01-31T22:36:48.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:48 smithi167 conmon[49112]: debug 2022-01-31T22:36:48.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:48.758821+0000) 2022-01-31T22:36:49.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:49 smithi171 conmon[46715]: debug 2022-01-31T22:36:49.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:49.426362+0000) 2022-01-31T22:36:49.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:49 smithi171 conmon[51620]: debug 2022-01-31T22:36:49.287+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:49.288440+0000) 2022-01-31T22:36:49.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:49 smithi171 conmon[41853]: debug 2022-01-31T22:36:49.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:49.628868+0000) 2022-01-31T22:36:49.975 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:49 smithi167 conmon[49112]: debug 2022-01-31T22:36:49.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:49.758972+0000) 2022-01-31T22:36:49.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:49 smithi167 conmon[54076]: debug 2022-01-31T22:36:49.925+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:49.925866+0000) 2022-01-31T22:36:49.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:49 smithi167 conmon[60316]: debug 2022-01-31T22:36:49.683+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:49.685329+0000) 2022-01-31T22:36:50.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:50 smithi171 conmon[46715]: debug 2022-01-31T22:36:50.425+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:50.426540+0000) 2022-01-31T22:36:50.588 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:50 smithi171 conmon[51620]: debug 2022-01-31T22:36:50.287+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:50.288614+0000) 2022-01-31T22:36:50.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:50 smithi171 conmon[41853]: debug 2022-01-31T22:36:50.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:50.629050+0000) 2022-01-31T22:36:50.976 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:50 smithi167 conmon[49112]: debug 2022-01-31T22:36:50.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:50.759136+0000) 2022-01-31T22:36:50.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:50 smithi167 conmon[54076]: debug 2022-01-31T22:36:50.925+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:50.926071+0000) 2022-01-31T22:36:50.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:50 smithi167 conmon[60316]: debug 2022-01-31T22:36:50.684+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:50.685492+0000) 2022-01-31T22:36:51.588 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:51 smithi171 conmon[46715]: debug 2022-01-31T22:36:51.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:51.426760+0000) 2022-01-31T22:36:51.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:51 smithi171 conmon[51620]: debug 2022-01-31T22:36:51.288+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:51.288794+0000) 2022-01-31T22:36:51.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:51 smithi171 conmon[41853]: debug 2022-01-31T22:36:51.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:51.629204+0000) 2022-01-31T22:36:51.976 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:51 smithi167 conmon[49112]: debug 2022-01-31T22:36:51.757+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:51.759254+0000) 2022-01-31T22:36:51.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:51 smithi167 conmon[54076]: debug 2022-01-31T22:36:51.925+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:51.926244+0000) 2022-01-31T22:36:51.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:51 smithi167 conmon[60316]: debug 2022-01-31T22:36:51.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:51.685668+0000) 2022-01-31T22:36:52.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:52 smithi171 conmon[46715]: debug 2022-01-31T22:36:52.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:52.426907+0000) 2022-01-31T22:36:52.589 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:52 smithi171 conmon[51620]: debug 2022-01-31T22:36:52.288+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:52.288956+0000) 2022-01-31T22:36:52.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:52 smithi171 conmon[41853]: debug 2022-01-31T22:36:52.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:52.629350+0000) 2022-01-31T22:36:52.976 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:52 smithi167 conmon[49112]: debug 2022-01-31T22:36:52.758+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:52.759438+0000) 2022-01-31T22:36:52.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:52 smithi167 conmon[54076]: debug 2022-01-31T22:36:52.925+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:52.926397+0000) 2022-01-31T22:36:52.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:52 smithi167 conmon[60316]: debug 2022-01-31T22:36:52.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:52.685827+0000) 2022-01-31T22:36:53.354 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[35325]: debug 2022-01-31T22:36:53.078+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223562 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:53.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[41853]: debug 2022-01-31T22:36:53.053+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.054601+0000) 2022-01-31T22:36:53.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[46715]: debug 2022-01-31T22:36:53.054+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.055089+0000) 2022-01-31T22:36:53.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[51620]: debug 2022-01-31T22:36:53.053+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.054679+0000) 2022-01-31T22:36:53.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[51620]: debug 2022-01-31T22:36:53.288+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.289097+0000) 2022-01-31T22:36:53.414 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:53 smithi167 conmon[49112]: debug 2022-01-31T22:36:53.054+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.056456+0000) 2022-01-31T22:36:53.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:53 smithi167 conmon[54076]: debug 2022-01-31T22:36:53.055+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.056581+0000) 2022-01-31T22:36:53.415 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:53 smithi167 conmon[60316]: debug 2022-01-31T22:36:53.054+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.055745+0000) 2022-01-31T22:36:53.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[41853]: debug 2022-01-31T22:36:53.628+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.629544+0000) 2022-01-31T22:36:53.856 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[46715]: debug 2022-01-31T22:36:53.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.427118+0000) 2022-01-31T22:36:53.976 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:53 smithi167 conmon[54076]: debug 2022-01-31T22:36:53.926+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.926528+0000) 2022-01-31T22:36:53.977 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:53 smithi167 conmon[60316]: debug 2022-01-31T22:36:53.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.685987+0000) 2022-01-31T22:36:53.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:53 smithi167 conmon[49112]: debug 2022-01-31T22:36:53.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:53.759626+0000) 2022-01-31T22:36:54.286 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:53 smithi171 conmon[35325]: debug 2022-01-31T22:36:53.881+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:36:54.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:54 smithi171 conmon[46715]: debug 2022-01-31T22:36:54.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:54.427291+0000) 2022-01-31T22:36:54.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:54 smithi171 conmon[51620]: debug 2022-01-31T22:36:54.288+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:54.289217+0000) 2022-01-31T22:36:54.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:54 smithi171 conmon[41853]: debug 2022-01-31T22:36:54.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:54.629696+0000) 2022-01-31T22:36:54.976 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:54 smithi167 conmon[49112]: debug 2022-01-31T22:36:54.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:54.759734+0000) 2022-01-31T22:36:54.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:54 smithi167 conmon[54076]: debug 2022-01-31T22:36:54.926+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:54.926699+0000) 2022-01-31T22:36:54.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:54 smithi167 conmon[60316]: debug 2022-01-31T22:36:54.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:54.686146+0000) 2022-01-31T22:36:55.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:55 smithi171 conmon[46715]: debug 2022-01-31T22:36:55.426+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:55.427453+0000) 2022-01-31T22:36:55.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:55 smithi171 conmon[51620]: debug 2022-01-31T22:36:55.288+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:55.289391+0000) 2022-01-31T22:36:55.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:55 smithi171 conmon[41853]: debug 2022-01-31T22:36:55.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:55.629917+0000) 2022-01-31T22:36:55.976 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:55 smithi167 conmon[49112]: debug 2022-01-31T22:36:55.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:55.759918+0000) 2022-01-31T22:36:55.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:55 smithi167 conmon[54076]: debug 2022-01-31T22:36:55.926+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:55.926878+0000) 2022-01-31T22:36:55.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:55 smithi167 conmon[60316]: debug 2022-01-31T22:36:55.685+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:55.686340+0000) 2022-01-31T22:36:56.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:56 smithi171 conmon[46715]: debug 2022-01-31T22:36:56.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:56.427642+0000) 2022-01-31T22:36:56.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:56 smithi171 conmon[51620]: debug 2022-01-31T22:36:56.288+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:56.289581+0000) 2022-01-31T22:36:56.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:56 smithi171 conmon[41853]: debug 2022-01-31T22:36:56.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:56.630106+0000) 2022-01-31T22:36:56.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:56 smithi167 conmon[49112]: debug 2022-01-31T22:36:56.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:56.760112+0000) 2022-01-31T22:36:56.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:56 smithi167 conmon[54076]: debug 2022-01-31T22:36:56.926+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:56.927012+0000) 2022-01-31T22:36:56.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:56 smithi167 conmon[60316]: debug 2022-01-31T22:36:56.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:56.686530+0000) 2022-01-31T22:36:57.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:57 smithi171 conmon[46715]: debug 2022-01-31T22:36:57.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:57.427828+0000) 2022-01-31T22:36:57.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:57 smithi171 conmon[51620]: debug 2022-01-31T22:36:57.289+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:57.289804+0000) 2022-01-31T22:36:57.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:57 smithi171 conmon[41853]: debug 2022-01-31T22:36:57.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:57.630267+0000) 2022-01-31T22:36:57.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:57 smithi167 conmon[54076]: debug 2022-01-31T22:36:57.926+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:57.927179+0000) 2022-01-31T22:36:57.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:57 smithi167 conmon[60316]: debug 2022-01-31T22:36:57.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:57.686690+0000) 2022-01-31T22:36:57.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:57 smithi167 conmon[49112]: debug 2022-01-31T22:36:57.759+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:57.760317+0000) 2022-01-31T22:36:58.359 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[35325]: debug 2022-01-31T22:36:58.103+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223674 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:36:58.360 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[41853]: debug 2022-01-31T22:36:58.080+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.080938+0000) 2022-01-31T22:36:58.360 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[46715]: debug 2022-01-31T22:36:58.080+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.081258+0000) 2022-01-31T22:36:58.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[51620]: debug 2022-01-31T22:36:58.080+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.081052+0000) 2022-01-31T22:36:58.361 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[51620]: debug 2022-01-31T22:36:58.289+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.289972+0000) 2022-01-31T22:36:58.414 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:58 smithi167 conmon[49112]: debug 2022-01-31T22:36:58.081+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.082397+0000) 2022-01-31T22:36:58.415 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:58 smithi167 conmon[54076]: debug 2022-01-31T22:36:58.081+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.081833+0000) 2022-01-31T22:36:58.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:58 smithi167 conmon[60316]: debug 2022-01-31T22:36:58.082+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.083426+0000) 2022-01-31T22:36:58.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[41853]: debug 2022-01-31T22:36:58.629+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.630487+0000) 2022-01-31T22:36:58.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:58 smithi171 conmon[46715]: debug 2022-01-31T22:36:58.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.427984+0000) 2022-01-31T22:36:58.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:58 smithi167 conmon[49112]: debug 2022-01-31T22:36:58.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.760536+0000) 2022-01-31T22:36:58.978 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:58 smithi167 conmon[54076]: debug 2022-01-31T22:36:58.926+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.927370+0000) 2022-01-31T22:36:58.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:58 smithi167 conmon[60316]: debug 2022-01-31T22:36:58.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:58.686806+0000) 2022-01-31T22:36:59.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:36:59 smithi171 conmon[46715]: debug 2022-01-31T22:36:59.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:59.428145+0000) 2022-01-31T22:36:59.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:36:59 smithi171 conmon[51620]: debug 2022-01-31T22:36:59.289+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:59.290114+0000) 2022-01-31T22:36:59.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:36:59 smithi171 conmon[41853]: debug 2022-01-31T22:36:59.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:59.630657+0000) 2022-01-31T22:36:59.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:36:59 smithi167 conmon[49112]: debug 2022-01-31T22:36:59.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:59.760706+0000) 2022-01-31T22:36:59.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:36:59 smithi167 conmon[54076]: debug 2022-01-31T22:36:59.927+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:59.927531+0000) 2022-01-31T22:36:59.979 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:36:59 smithi167 conmon[60316]: debug 2022-01-31T22:36:59.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:36:59.686926+0000) 2022-01-31T22:37:00.589 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:00 smithi171 conmon[46715]: debug 2022-01-31T22:37:00.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:00.428331+0000) 2022-01-31T22:37:00.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:00 smithi171 conmon[51620]: debug 2022-01-31T22:37:00.289+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:00.290307+0000) 2022-01-31T22:37:00.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:00 smithi171 conmon[41853]: debug 2022-01-31T22:37:00.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:00.630855+0000) 2022-01-31T22:37:00.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:00 smithi167 conmon[54076]: debug 2022-01-31T22:37:00.927+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:00.927728+0000) 2022-01-31T22:37:00.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:00 smithi167 conmon[60316]: debug 2022-01-31T22:37:00.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:00.687086+0000) 2022-01-31T22:37:00.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:00 smithi167 conmon[49112]: debug 2022-01-31T22:37:00.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:00.760894+0000) 2022-01-31T22:37:01.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:01 smithi171 conmon[46715]: debug 2022-01-31T22:37:01.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:01.428515+0000) 2022-01-31T22:37:01.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:01 smithi171 conmon[51620]: debug 2022-01-31T22:37:01.290+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:01.290520+0000) 2022-01-31T22:37:01.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:01 smithi171 conmon[41853]: debug 2022-01-31T22:37:01.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:01.631024+0000) 2022-01-31T22:37:01.977 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:01 smithi167 conmon[54076]: debug 2022-01-31T22:37:01.927+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:01.927884+0000) 2022-01-31T22:37:01.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:01 smithi167 conmon[60316]: debug 2022-01-31T22:37:01.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:01.687300+0000) 2022-01-31T22:37:01.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:01 smithi167 conmon[49112]: debug 2022-01-31T22:37:01.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:01.761077+0000) 2022-01-31T22:37:02.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:02 smithi171 conmon[46715]: debug 2022-01-31T22:37:02.427+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:02.428719+0000) 2022-01-31T22:37:02.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:02 smithi171 conmon[51620]: debug 2022-01-31T22:37:02.289+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:02.290703+0000) 2022-01-31T22:37:02.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:02 smithi171 conmon[41853]: debug 2022-01-31T22:37:02.630+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:02.631178+0000) 2022-01-31T22:37:02.978 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:02 smithi167 conmon[54076]: debug 2022-01-31T22:37:02.927+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:02.928040+0000) 2022-01-31T22:37:02.978 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:02 smithi167 conmon[60316]: debug 2022-01-31T22:37:02.686+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:02.687425+0000) 2022-01-31T22:37:02.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:02 smithi167 conmon[49112]: debug 2022-01-31T22:37:02.760+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:02.761233+0000) 2022-01-31T22:37:03.354 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[35325]: debug 2022-01-31T22:37:03.134+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223785 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:03.355 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[41853]: debug 2022-01-31T22:37:03.105+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.106650+0000) 2022-01-31T22:37:03.356 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[46715]: debug 2022-01-31T22:37:03.105+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.105919+0000) 2022-01-31T22:37:03.356 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[51620]: debug 2022-01-31T22:37:03.105+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.106376+0000) 2022-01-31T22:37:03.357 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[51620]: debug 2022-01-31T22:37:03.290+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.290894+0000) 2022-01-31T22:37:03.415 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:03 smithi167 conmon[49112]: debug 2022-01-31T22:37:03.107+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.107506+0000) 2022-01-31T22:37:03.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:03 smithi167 conmon[54076]: debug 2022-01-31T22:37:03.107+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.107699+0000) 2022-01-31T22:37:03.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:03 smithi167 conmon[60316]: debug 2022-01-31T22:37:03.107+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.107998+0000) 2022-01-31T22:37:03.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[41853]: debug 2022-01-31T22:37:03.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.631410+0000) 2022-01-31T22:37:03.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:03 smithi171 conmon[46715]: debug 2022-01-31T22:37:03.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.428930+0000) 2022-01-31T22:37:03.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:03 smithi167 conmon[49112]: debug 2022-01-31T22:37:03.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.761388+0000) 2022-01-31T22:37:03.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:03 smithi167 conmon[54076]: debug 2022-01-31T22:37:03.927+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.928223+0000) 2022-01-31T22:37:03.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:03 smithi167 conmon[60316]: debug 2022-01-31T22:37:03.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:03.687625+0000) 2022-01-31T22:37:04.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:04 smithi171 conmon[46715]: debug 2022-01-31T22:37:04.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:04.429075+0000) 2022-01-31T22:37:04.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:04 smithi171 conmon[51620]: debug 2022-01-31T22:37:04.290+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:04.291087+0000) 2022-01-31T22:37:04.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:04 smithi171 conmon[41853]: debug 2022-01-31T22:37:04.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:04.631577+0000) 2022-01-31T22:37:04.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:04 smithi167 conmon[49112]: debug 2022-01-31T22:37:04.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:04.761567+0000) 2022-01-31T22:37:04.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:04 smithi167 conmon[54076]: debug 2022-01-31T22:37:04.928+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:04.928376+0000) 2022-01-31T22:37:04.979 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:04 smithi167 conmon[60316]: debug 2022-01-31T22:37:04.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:04.687833+0000) 2022-01-31T22:37:05.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:05 smithi171 conmon[46715]: debug 2022-01-31T22:37:05.428+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:05.429261+0000) 2022-01-31T22:37:05.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:05 smithi171 conmon[51620]: debug 2022-01-31T22:37:05.290+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:05.291288+0000) 2022-01-31T22:37:05.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:05 smithi171 conmon[41853]: debug 2022-01-31T22:37:05.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:05.631778+0000) 2022-01-31T22:37:05.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:05 smithi167 conmon[49112]: debug 2022-01-31T22:37:05.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:05.761701+0000) 2022-01-31T22:37:05.979 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:05 smithi167 conmon[60316]: debug 2022-01-31T22:37:05.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:05.688049+0000) 2022-01-31T22:37:05.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:05 smithi167 conmon[54076]: debug 2022-01-31T22:37:05.928+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:05.928575+0000) 2022-01-31T22:37:06.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:06 smithi171 conmon[46715]: debug 2022-01-31T22:37:06.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:06.429449+0000) 2022-01-31T22:37:06.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:06 smithi171 conmon[51620]: debug 2022-01-31T22:37:06.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:06.291481+0000) 2022-01-31T22:37:06.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:06 smithi171 conmon[41853]: debug 2022-01-31T22:37:06.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:06.631963+0000) 2022-01-31T22:37:06.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:06 smithi167 conmon[49112]: debug 2022-01-31T22:37:06.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:06.761883+0000) 2022-01-31T22:37:06.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:06 smithi167 conmon[54076]: debug 2022-01-31T22:37:06.928+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:06.928722+0000) 2022-01-31T22:37:06.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:06 smithi167 conmon[60316]: debug 2022-01-31T22:37:06.687+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:06.688261+0000) 2022-01-31T22:37:07.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:07 smithi171 conmon[46715]: debug 2022-01-31T22:37:07.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:07.429636+0000) 2022-01-31T22:37:07.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:07 smithi171 conmon[51620]: debug 2022-01-31T22:37:07.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:07.291663+0000) 2022-01-31T22:37:07.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:07 smithi171 conmon[41853]: debug 2022-01-31T22:37:07.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:07.632121+0000) 2022-01-31T22:37:07.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:07 smithi167 conmon[49112]: debug 2022-01-31T22:37:07.761+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:07.762094+0000) 2022-01-31T22:37:07.979 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:07 smithi167 conmon[60316]: debug 2022-01-31T22:37:07.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:07.688404+0000) 2022-01-31T22:37:07.980 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:07 smithi167 conmon[54076]: debug 2022-01-31T22:37:07.928+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:07.928935+0000) 2022-01-31T22:37:08.405 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[35325]: debug 2022-01-31T22:37:08.160+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 223898 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:08.405 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[41853]: debug 2022-01-31T22:37:08.137+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.138097+0000) 2022-01-31T22:37:08.406 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[46715]: debug 2022-01-31T22:37:08.137+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.137781+0000) 2022-01-31T22:37:08.407 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[51620]: debug 2022-01-31T22:37:08.138+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.138849+0000) 2022-01-31T22:37:08.407 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[51620]: debug 2022-01-31T22:37:08.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.291874+0000) 2022-01-31T22:37:08.416 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:08 smithi167 conmon[49112]: debug 2022-01-31T22:37:08.143+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.144158+0000) 2022-01-31T22:37:08.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:08 smithi167 conmon[54076]: debug 2022-01-31T22:37:08.135+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.139173+0000) 2022-01-31T22:37:08.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:08 smithi167 conmon[60316]: debug 2022-01-31T22:37:08.135+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.138278+0000) 2022-01-31T22:37:08.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[41853]: debug 2022-01-31T22:37:08.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.632257+0000) 2022-01-31T22:37:08.855 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[46715]: debug 2022-01-31T22:37:08.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.429833+0000) 2022-01-31T22:37:08.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:08 smithi167 conmon[49112]: debug 2022-01-31T22:37:08.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.762300+0000) 2022-01-31T22:37:08.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:08 smithi167 conmon[54076]: debug 2022-01-31T22:37:08.928+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.929074+0000) 2022-01-31T22:37:08.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:08 smithi167 conmon[60316]: debug 2022-01-31T22:37:08.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:08.688602+0000) 2022-01-31T22:37:09.288 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:08 smithi171 conmon[35325]: debug 2022-01-31T22:37:08.882+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:37:09.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:09 smithi171 conmon[46715]: debug 2022-01-31T22:37:09.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:09.429960+0000) 2022-01-31T22:37:09.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:09 smithi171 conmon[51620]: debug 2022-01-31T22:37:09.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:09.292027+0000) 2022-01-31T22:37:09.685 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:09 smithi167 conmon[60316]: debug 2022-01-31T22:37:09.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:09.688756+0000) 2022-01-31T22:37:09.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:09 smithi171 conmon[41853]: debug 2022-01-31T22:37:09.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:09.632365+0000) 2022-01-31T22:37:09.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:09 smithi167 conmon[49112]: debug 2022-01-31T22:37:09.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:09.762518+0000) 2022-01-31T22:37:09.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:09 smithi167 conmon[54076]: debug 2022-01-31T22:37:09.928+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:09.929245+0000) 2022-01-31T22:37:10.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:10 smithi171 conmon[46715]: debug 2022-01-31T22:37:10.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:10.430139+0000) 2022-01-31T22:37:10.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:10 smithi171 conmon[51620]: debug 2022-01-31T22:37:10.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:10.292191+0000) 2022-01-31T22:37:10.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:10 smithi171 conmon[41853]: debug 2022-01-31T22:37:10.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:10.632513+0000) 2022-01-31T22:37:10.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:10 smithi167 conmon[49112]: debug 2022-01-31T22:37:10.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:10.762727+0000) 2022-01-31T22:37:10.980 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:10 smithi167 conmon[54076]: debug 2022-01-31T22:37:10.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:10.929491+0000) 2022-01-31T22:37:10.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:10 smithi167 conmon[60316]: debug 2022-01-31T22:37:10.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:10.688864+0000) 2022-01-31T22:37:11.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:11 smithi171 conmon[46715]: debug 2022-01-31T22:37:11.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:11.430265+0000) 2022-01-31T22:37:11.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:11 smithi171 conmon[51620]: debug 2022-01-31T22:37:11.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:11.292344+0000) 2022-01-31T22:37:11.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:11 smithi171 conmon[41853]: debug 2022-01-31T22:37:11.631+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:11.632717+0000) 2022-01-31T22:37:11.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:11 smithi167 conmon[49112]: debug 2022-01-31T22:37:11.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:11.762892+0000) 2022-01-31T22:37:11.980 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:11 smithi167 conmon[54076]: debug 2022-01-31T22:37:11.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:11.929695+0000) 2022-01-31T22:37:11.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:11 smithi167 conmon[60316]: debug 2022-01-31T22:37:11.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:11.689029+0000) 2022-01-31T22:37:12.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:12 smithi171 conmon[46715]: debug 2022-01-31T22:37:12.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:12.430430+0000) 2022-01-31T22:37:12.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:12 smithi171 conmon[51620]: debug 2022-01-31T22:37:12.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:12.292514+0000) 2022-01-31T22:37:12.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:12 smithi171 conmon[41853]: debug 2022-01-31T22:37:12.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:12.632871+0000) 2022-01-31T22:37:12.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:12 smithi167 conmon[49112]: debug 2022-01-31T22:37:12.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:12.763077+0000) 2022-01-31T22:37:12.980 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:12 smithi167 conmon[54076]: debug 2022-01-31T22:37:12.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:12.929880+0000) 2022-01-31T22:37:12.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:12 smithi167 conmon[60316]: debug 2022-01-31T22:37:12.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:12.689204+0000) 2022-01-31T22:37:13.416 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:13 smithi167 conmon[49112]: debug 2022-01-31T22:37:13.164+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.164923+0000) 2022-01-31T22:37:13.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:13 smithi167 conmon[54076]: debug 2022-01-31T22:37:13.164+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.164607+0000) 2022-01-31T22:37:13.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:13 smithi167 conmon[60316]: debug 2022-01-31T22:37:13.163+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.164287+0000) 2022-01-31T22:37:13.428 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[35325]: debug 2022-01-31T22:37:13.185+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224009 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:13.428 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[41853]: debug 2022-01-31T22:37:13.163+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.164592+0000) 2022-01-31T22:37:13.429 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[46715]: debug 2022-01-31T22:37:13.163+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.164186+0000) 2022-01-31T22:37:13.429 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[51620]: debug 2022-01-31T22:37:13.163+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.164387+0000) 2022-01-31T22:37:13.430 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[51620]: debug 2022-01-31T22:37:13.291+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.292700+0000) 2022-01-31T22:37:13.854 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[46715]: debug 2022-01-31T22:37:13.429+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.430670+0000) 2022-01-31T22:37:13.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:13 smithi171 conmon[41853]: debug 2022-01-31T22:37:13.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.633011+0000) 2022-01-31T22:37:13.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:13 smithi167 conmon[49112]: debug 2022-01-31T22:37:13.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.763277+0000) 2022-01-31T22:37:13.980 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:13 smithi167 conmon[54076]: debug 2022-01-31T22:37:13.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.930089+0000) 2022-01-31T22:37:13.981 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:13 smithi167 conmon[60316]: debug 2022-01-31T22:37:13.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:13.689389+0000) 2022-01-31T22:37:14.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:14 smithi171 conmon[46715]: debug 2022-01-31T22:37:14.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:14.430866+0000) 2022-01-31T22:37:14.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:14 smithi171 conmon[51620]: debug 2022-01-31T22:37:14.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:14.292954+0000) 2022-01-31T22:37:14.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:14 smithi171 conmon[41853]: debug 2022-01-31T22:37:14.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:14.633173+0000) 2022-01-31T22:37:14.979 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:14 smithi167 conmon[49112]: debug 2022-01-31T22:37:14.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:14.763419+0000) 2022-01-31T22:37:14.980 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:14 smithi167 conmon[54076]: debug 2022-01-31T22:37:14.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:14.930291+0000) 2022-01-31T22:37:14.981 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:14 smithi167 conmon[60316]: debug 2022-01-31T22:37:14.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:14.689566+0000) 2022-01-31T22:37:15.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:15 smithi171 conmon[46715]: debug 2022-01-31T22:37:15.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:15.431022+0000) 2022-01-31T22:37:15.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:15 smithi171 conmon[51620]: debug 2022-01-31T22:37:15.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:15.293120+0000) 2022-01-31T22:37:15.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:15 smithi171 conmon[41853]: debug 2022-01-31T22:37:15.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:15.633301+0000) 2022-01-31T22:37:15.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:15 smithi167 conmon[60316]: debug 2022-01-31T22:37:15.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:15.689670+0000) 2022-01-31T22:37:15.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:15 smithi167 conmon[49112]: debug 2022-01-31T22:37:15.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:15.763556+0000) 2022-01-31T22:37:15.981 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:15 smithi167 conmon[54076]: debug 2022-01-31T22:37:15.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:15.930480+0000) 2022-01-31T22:37:16.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:16 smithi171 conmon[46715]: debug 2022-01-31T22:37:16.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:16.431213+0000) 2022-01-31T22:37:16.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:16 smithi171 conmon[51620]: debug 2022-01-31T22:37:16.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:16.293283+0000) 2022-01-31T22:37:16.857 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:16 smithi171 conmon[41853]: debug 2022-01-31T22:37:16.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:16.633463+0000) 2022-01-31T22:37:16.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:16 smithi167 conmon[60316]: debug 2022-01-31T22:37:16.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:16.689829+0000) 2022-01-31T22:37:16.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:16 smithi167 conmon[49112]: debug 2022-01-31T22:37:16.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:16.763735+0000) 2022-01-31T22:37:16.981 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:16 smithi167 conmon[54076]: debug 2022-01-31T22:37:16.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:16.930683+0000) 2022-01-31T22:37:17.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:17 smithi171 conmon[46715]: debug 2022-01-31T22:37:17.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:17.431372+0000) 2022-01-31T22:37:17.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:17 smithi171 conmon[51620]: debug 2022-01-31T22:37:17.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:17.293482+0000) 2022-01-31T22:37:17.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:17 smithi171 conmon[41853]: debug 2022-01-31T22:37:17.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:17.633642+0000) 2022-01-31T22:37:17.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:17 smithi167 conmon[60316]: debug 2022-01-31T22:37:17.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:17.689954+0000) 2022-01-31T22:37:17.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:17 smithi167 conmon[49112]: debug 2022-01-31T22:37:17.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:17.763918+0000) 2022-01-31T22:37:17.981 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:17 smithi167 conmon[54076]: debug 2022-01-31T22:37:17.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:17.930876+0000) 2022-01-31T22:37:18.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[54076]: debug 2022-01-31T22:37:18.188+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.189673+0000) 2022-01-31T22:37:18.416 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[60316]: debug 2022-01-31T22:37:18.188+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.189908+0000) 2022-01-31T22:37:18.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[49112]: debug 2022-01-31T22:37:18.417 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[49112]: 2022-01-31T22:37:18.188+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.190399+0000) 2022-01-31T22:37:18.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[35325]: debug 2022-01-31T22:37:18.211+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224121 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:18.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[41853]: debug 2022-01-31T22:37:18.188+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.189098+0000) 2022-01-31T22:37:18.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[46715]: debug 2022-01-31T22:37:18.187+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.188709+0000) 2022-01-31T22:37:18.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[46715]: debug 2022-01-31T22:37:18.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.431574+0000) 2022-01-31T22:37:18.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[51620]: debug 2022-01-31T22:37:18.189+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.190454+0000) 2022-01-31T22:37:18.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[51620]: debug 2022-01-31T22:37:18.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.293630+0000) 2022-01-31T22:37:18.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:18 smithi171 conmon[41853]: debug 2022-01-31T22:37:18.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.633827+0000) 2022-01-31T22:37:18.980 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[49112]: debug 2022-01-31T22:37:18.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.764133+0000) 2022-01-31T22:37:18.981 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[54076]: debug 2022-01-31T22:37:18.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.931034+0000) 2022-01-31T22:37:18.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:18 smithi167 conmon[60316]: debug 2022-01-31T22:37:18.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:18.690070+0000) 2022-01-31T22:37:19.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:19 smithi171 conmon[46715]: debug 2022-01-31T22:37:19.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:19.431723+0000) 2022-01-31T22:37:19.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:19 smithi171 conmon[51620]: debug 2022-01-31T22:37:19.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:19.293826+0000) 2022-01-31T22:37:19.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:19 smithi171 conmon[41853]: debug 2022-01-31T22:37:19.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:19.633995+0000) 2022-01-31T22:37:19.980 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:19 smithi167 conmon[49112]: debug 2022-01-31T22:37:19.762+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:19.764281+0000) 2022-01-31T22:37:19.981 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:19 smithi167 conmon[54076]: debug 2022-01-31T22:37:19.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:19.931192+0000) 2022-01-31T22:37:19.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:19 smithi167 conmon[60316]: debug 2022-01-31T22:37:19.688+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:19.690248+0000) 2022-01-31T22:37:20.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:20 smithi171 conmon[46715]: debug 2022-01-31T22:37:20.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:20.431884+0000) 2022-01-31T22:37:20.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:20 smithi171 conmon[51620]: debug 2022-01-31T22:37:20.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:20.294018+0000) 2022-01-31T22:37:20.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:20 smithi171 conmon[41853]: debug 2022-01-31T22:37:20.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:20.634227+0000) 2022-01-31T22:37:20.980 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:20 smithi167 conmon[49112]: debug 2022-01-31T22:37:20.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:20.764512+0000) 2022-01-31T22:37:20.981 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:20 smithi167 conmon[54076]: debug 2022-01-31T22:37:20.929+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:20.931413+0000) 2022-01-31T22:37:20.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:20 smithi167 conmon[60316]: debug 2022-01-31T22:37:20.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:20.690430+0000) 2022-01-31T22:37:21.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:21 smithi171 conmon[46715]: debug 2022-01-31T22:37:21.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:21.432072+0000) 2022-01-31T22:37:21.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:21 smithi171 conmon[51620]: debug 2022-01-31T22:37:21.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:21.294223+0000) 2022-01-31T22:37:21.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:21 smithi171 conmon[41853]: debug 2022-01-31T22:37:21.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:21.634423+0000) 2022-01-31T22:37:21.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:21 smithi167 conmon[49112]: debug 2022-01-31T22:37:21.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:21.764744+0000) 2022-01-31T22:37:21.982 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:21 smithi167 conmon[54076]: debug 2022-01-31T22:37:21.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:21.931629+0000) 2022-01-31T22:37:21.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:21 smithi167 conmon[60316]: debug 2022-01-31T22:37:21.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:21.690549+0000) 2022-01-31T22:37:22.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:22 smithi171 conmon[46715]: debug 2022-01-31T22:37:22.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:22.432245+0000) 2022-01-31T22:37:22.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:22 smithi171 conmon[51620]: debug 2022-01-31T22:37:22.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:22.294454+0000) 2022-01-31T22:37:22.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:22 smithi171 conmon[41853]: debug 2022-01-31T22:37:22.632+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:22.634605+0000) 2022-01-31T22:37:22.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:22 smithi167 conmon[49112]: debug 2022-01-31T22:37:22.763+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:22.764931+0000) 2022-01-31T22:37:22.982 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:22 smithi167 conmon[54076]: debug 2022-01-31T22:37:22.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:22.931763+0000) 2022-01-31T22:37:22.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:22 smithi167 conmon[60316]: debug 2022-01-31T22:37:22.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:22.690737+0000) 2022-01-31T22:37:23.415 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:23 smithi167 conmon[49112]: debug 2022-01-31T22:37:23.214+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.215601+0000) 2022-01-31T22:37:23.416 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:23 smithi167 conmon[54076]: debug 2022-01-31T22:37:23.214+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.216185+0000) 2022-01-31T22:37:23.417 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:23 smithi167 conmon[60316]: debug 2022-01-31T22:37:23.213+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.215399+0000) 2022-01-31T22:37:23.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[35325]: debug 2022-01-31T22:37:23.236+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224233 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:23.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[41853]: debug 2022-01-31T22:37:23.213+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.214789+0000) 2022-01-31T22:37:23.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[46715]: debug 2022-01-31T22:37:23.212+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.214688+0000) 2022-01-31T22:37:23.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[46715]: debug 2022-01-31T22:37:23.430+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.432484+0000) 2022-01-31T22:37:23.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[51620]: debug 2022-01-31T22:37:23.214+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.216231+0000) 2022-01-31T22:37:23.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[51620]: debug 2022-01-31T22:37:23.292+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.294697+0000) 2022-01-31T22:37:23.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[41853]: debug 2022-01-31T22:37:23.633+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.634846+0000) 2022-01-31T22:37:23.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:23 smithi167 conmon[49112]: debug 2022-01-31T22:37:23.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.765122+0000) 2022-01-31T22:37:23.982 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:23 smithi167 conmon[54076]: debug 2022-01-31T22:37:23.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.931869+0000) 2022-01-31T22:37:23.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:23 smithi167 conmon[60316]: debug 2022-01-31T22:37:23.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:23.690948+0000) 2022-01-31T22:37:24.291 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:23 smithi171 conmon[35325]: debug 2022-01-31T22:37:23.882+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:37:24.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:24 smithi171 conmon[46715]: debug 2022-01-31T22:37:24.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:24.432682+0000) 2022-01-31T22:37:24.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:24 smithi171 conmon[51620]: debug 2022-01-31T22:37:24.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:24.294836+0000) 2022-01-31T22:37:24.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:24 smithi171 conmon[41853]: debug 2022-01-31T22:37:24.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:24.635005+0000) 2022-01-31T22:37:24.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:24 smithi167 conmon[49112]: debug 2022-01-31T22:37:24.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:24.765291+0000) 2022-01-31T22:37:24.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:24 smithi167 conmon[60316]: debug 2022-01-31T22:37:24.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:24.691122+0000) 2022-01-31T22:37:24.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:24 smithi167 conmon[54076]: debug 2022-01-31T22:37:24.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:24.932032+0000) 2022-01-31T22:37:25.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:25 smithi171 conmon[46715]: debug 2022-01-31T22:37:25.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:25.432897+0000) 2022-01-31T22:37:25.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:25 smithi171 conmon[51620]: debug 2022-01-31T22:37:25.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:25.294949+0000) 2022-01-31T22:37:25.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:25 smithi171 conmon[41853]: debug 2022-01-31T22:37:25.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:25.635190+0000) 2022-01-31T22:37:25.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:25 smithi167 conmon[49112]: debug 2022-01-31T22:37:25.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:25.765471+0000) 2022-01-31T22:37:25.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:25 smithi167 conmon[60316]: debug 2022-01-31T22:37:25.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:25.691222+0000) 2022-01-31T22:37:25.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:25 smithi167 conmon[54076]: debug 2022-01-31T22:37:25.930+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:25.932237+0000) 2022-01-31T22:37:26.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:26 smithi171 conmon[46715]: debug 2022-01-31T22:37:26.432+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:26.433081+0000) 2022-01-31T22:37:26.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:26 smithi171 conmon[51620]: debug 2022-01-31T22:37:26.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:26.295145+0000) 2022-01-31T22:37:26.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:26 smithi171 conmon[41853]: debug 2022-01-31T22:37:26.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:26.635371+0000) 2022-01-31T22:37:26.981 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:26 smithi167 conmon[49112]: debug 2022-01-31T22:37:26.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:26.765654+0000) 2022-01-31T22:37:26.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:26 smithi167 conmon[60316]: debug 2022-01-31T22:37:26.689+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:26.691317+0000) 2022-01-31T22:37:26.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:26 smithi167 conmon[54076]: debug 2022-01-31T22:37:26.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:26.932404+0000) 2022-01-31T22:37:27.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:27 smithi171 conmon[46715]: debug 2022-01-31T22:37:27.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:27.433273+0000) 2022-01-31T22:37:27.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:27 smithi171 conmon[51620]: debug 2022-01-31T22:37:27.294+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:27.295318+0000) 2022-01-31T22:37:27.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:27 smithi171 conmon[41853]: debug 2022-01-31T22:37:27.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:27.635559+0000) 2022-01-31T22:37:27.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:27 smithi167 conmon[49112]: debug 2022-01-31T22:37:27.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:27.765838+0000) 2022-01-31T22:37:27.982 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:27 smithi167 conmon[54076]: debug 2022-01-31T22:37:27.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:27.932611+0000) 2022-01-31T22:37:27.983 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:27 smithi167 conmon[60316]: debug 2022-01-31T22:37:27.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:27.691524+0000) 2022-01-31T22:37:28.522 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:28 smithi167 conmon[49112]: debug 2022-01-31T22:37:28.239+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.241069+0000) 2022-01-31T22:37:28.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:28 smithi167 conmon[54076]: debug 2022-01-31T22:37:28.240+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.241510+0000) 2022-01-31T22:37:28.523 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:28 smithi167 conmon[60316]: debug 2022-01-31T22:37:28.238+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.240379+0000) 2022-01-31T22:37:28.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[35325]: debug 2022-01-31T22:37:28.261+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224345 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:28.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[41853]: debug 2022-01-31T22:37:28.239+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.241038+0000) 2022-01-31T22:37:28.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[46715]: debug 2022-01-31T22:37:28.238+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.240070+0000) 2022-01-31T22:37:28.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[46715]: debug 2022-01-31T22:37:28.431+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.433383+0000) 2022-01-31T22:37:28.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[51620]: debug 2022-01-31T22:37:28.239+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.241276+0000) 2022-01-31T22:37:28.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[51620]: debug 2022-01-31T22:37:28.293+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.295540+0000) 2022-01-31T22:37:28.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:28 smithi171 conmon[41853]: debug 2022-01-31T22:37:28.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.635774+0000) 2022-01-31T22:37:28.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:28 smithi167 conmon[49112]: debug 2022-01-31T22:37:28.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.766018+0000) 2022-01-31T22:37:28.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:28 smithi167 conmon[60316]: debug 2022-01-31T22:37:28.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.691710+0000) 2022-01-31T22:37:29.165 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:28 smithi167 conmon[54076]: debug 2022-01-31T22:37:28.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:28.932759+0000) 2022-01-31T22:37:29.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:29 smithi171 conmon[46715]: debug 2022-01-31T22:37:29.432+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:29.433533+0000) 2022-01-31T22:37:29.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:29 smithi171 conmon[51620]: debug 2022-01-31T22:37:29.294+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:29.295706+0000) 2022-01-31T22:37:29.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:29 smithi171 conmon[41853]: debug 2022-01-31T22:37:29.634+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:29.635902+0000) 2022-01-31T22:37:29.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:29 smithi167 conmon[49112]: debug 2022-01-31T22:37:29.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:29.766186+0000) 2022-01-31T22:37:29.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:29 smithi167 conmon[54076]: debug 2022-01-31T22:37:29.932+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:29.932876+0000) 2022-01-31T22:37:29.983 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:29 smithi167 conmon[60316]: debug 2022-01-31T22:37:29.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:29.691861+0000) 2022-01-31T22:37:30.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:30 smithi171 conmon[46715]: debug 2022-01-31T22:37:30.432+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:30.433687+0000) 2022-01-31T22:37:30.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:30 smithi171 conmon[51620]: debug 2022-01-31T22:37:30.294+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:30.295937+0000) 2022-01-31T22:37:30.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:30 smithi171 conmon[41853]: debug 2022-01-31T22:37:30.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:30.636094+0000) 2022-01-31T22:37:30.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:30 smithi167 conmon[49112]: debug 2022-01-31T22:37:30.764+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:30.766360+0000) 2022-01-31T22:37:30.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:30 smithi167 conmon[54076]: debug 2022-01-31T22:37:30.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:30.933082+0000) 2022-01-31T22:37:30.983 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:30 smithi167 conmon[60316]: debug 2022-01-31T22:37:30.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:30.692051+0000) 2022-01-31T22:37:31.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:31 smithi171 conmon[46715]: debug 2022-01-31T22:37:31.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:31.433866+0000) 2022-01-31T22:37:31.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:31 smithi171 conmon[51620]: debug 2022-01-31T22:37:31.295+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:31.296140+0000) 2022-01-31T22:37:31.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:31 smithi171 conmon[41853]: debug 2022-01-31T22:37:31.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:31.636280+0000) 2022-01-31T22:37:31.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:31 smithi167 conmon[49112]: debug 2022-01-31T22:37:31.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:31.766541+0000) 2022-01-31T22:37:31.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:31 smithi167 conmon[54076]: debug 2022-01-31T22:37:31.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:31.933197+0000) 2022-01-31T22:37:31.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:31 smithi167 conmon[60316]: debug 2022-01-31T22:37:31.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:31.692206+0000) 2022-01-31T22:37:32.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:32 smithi171 conmon[46715]: debug 2022-01-31T22:37:32.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:32.434048+0000) 2022-01-31T22:37:32.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:32 smithi171 conmon[51620]: debug 2022-01-31T22:37:32.295+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:32.296388+0000) 2022-01-31T22:37:32.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:32 smithi171 conmon[41853]: debug 2022-01-31T22:37:32.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:32.636452+0000) 2022-01-31T22:37:32.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:32 smithi167 conmon[49112]: debug 2022-01-31T22:37:32.765+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:32.766658+0000) 2022-01-31T22:37:32.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:32 smithi167 conmon[54076]: debug 2022-01-31T22:37:32.931+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:32.933413+0000) 2022-01-31T22:37:32.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:32 smithi167 conmon[60316]: debug 2022-01-31T22:37:32.690+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:32.692384+0000) 2022-01-31T22:37:33.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:33 smithi167 conmon[54076]: debug 2022-01-31T22:37:33.264+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.266397+0000) 2022-01-31T22:37:33.523 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:33 smithi167 conmon[60316]: debug 2022-01-31T22:37:33.264+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.265727+0000) 2022-01-31T22:37:33.524 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:33 smithi167 conmon[49112]: debug 2022-01-31T22:37:33.264+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.266199+0000) 2022-01-31T22:37:33.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[35325]: debug 2022-01-31T22:37:33.289+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224457 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:33.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[41853]: debug 2022-01-31T22:37:33.264+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.265393+0000) 2022-01-31T22:37:33.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[46715]: debug 2022-01-31T22:37:33.265+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.266666+0000) 2022-01-31T22:37:33.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[46715]: debug 2022-01-31T22:37:33.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.434223+0000) 2022-01-31T22:37:33.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[51620]: debug 2022-01-31T22:37:33.265+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.265834+0000) 2022-01-31T22:37:33.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[51620]: debug 2022-01-31T22:37:33.295+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.296538+0000) 2022-01-31T22:37:33.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:33 smithi171 conmon[41853]: debug 2022-01-31T22:37:33.635+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.636657+0000) 2022-01-31T22:37:33.911 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:33 smithi167 conmon[49112]: debug 2022-01-31T22:37:33.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.766861+0000) 2022-01-31T22:37:33.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:33 smithi167 conmon[60316]: debug 2022-01-31T22:37:33.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.692591+0000) 2022-01-31T22:37:34.165 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:33 smithi167 conmon[54076]: debug 2022-01-31T22:37:33.932+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:33.933553+0000) 2022-01-31T22:37:34.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:34 smithi171 conmon[46715]: debug 2022-01-31T22:37:34.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:34.434392+0000) 2022-01-31T22:37:34.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:34 smithi171 conmon[51620]: debug 2022-01-31T22:37:34.295+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:34.296658+0000) 2022-01-31T22:37:34.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:34 smithi171 conmon[41853]: debug 2022-01-31T22:37:34.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:34.636810+0000) 2022-01-31T22:37:34.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:34 smithi167 conmon[49112]: debug 2022-01-31T22:37:34.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:34.767014+0000) 2022-01-31T22:37:34.984 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:34 smithi167 conmon[54076]: debug 2022-01-31T22:37:34.932+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:34.933639+0000) 2022-01-31T22:37:34.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:34 smithi167 conmon[60316]: debug 2022-01-31T22:37:34.691+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:34.692731+0000) 2022-01-31T22:37:35.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:35 smithi171 conmon[46715]: debug 2022-01-31T22:37:35.433+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:35.434566+0000) 2022-01-31T22:37:35.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:35 smithi171 conmon[51620]: debug 2022-01-31T22:37:35.296+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:35.296862+0000) 2022-01-31T22:37:35.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:35 smithi171 conmon[41853]: debug 2022-01-31T22:37:35.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:35.636931+0000) 2022-01-31T22:37:35.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:35 smithi167 conmon[49112]: debug 2022-01-31T22:37:35.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:35.767196+0000) 2022-01-31T22:37:35.984 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:35 smithi167 conmon[54076]: debug 2022-01-31T22:37:35.933+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:35.933833+0000) 2022-01-31T22:37:35.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:35 smithi167 conmon[60316]: debug 2022-01-31T22:37:35.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:35.692933+0000) 2022-01-31T22:37:36.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:36 smithi171 conmon[46715]: debug 2022-01-31T22:37:36.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:36.434732+0000) 2022-01-31T22:37:36.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:36 smithi171 conmon[51620]: debug 2022-01-31T22:37:36.296+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:36.297041+0000) 2022-01-31T22:37:36.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:36 smithi171 conmon[41853]: debug 2022-01-31T22:37:36.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:36.637128+0000) 2022-01-31T22:37:36.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:36 smithi167 conmon[49112]: debug 2022-01-31T22:37:36.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:36.767351+0000) 2022-01-31T22:37:36.984 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:36 smithi167 conmon[54076]: debug 2022-01-31T22:37:36.932+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:36.934023+0000) 2022-01-31T22:37:36.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:36 smithi167 conmon[60316]: debug 2022-01-31T22:37:36.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:36.693093+0000) 2022-01-31T22:37:37.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:37 smithi171 conmon[46715]: debug 2022-01-31T22:37:37.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:37.434942+0000) 2022-01-31T22:37:37.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:37 smithi171 conmon[51620]: debug 2022-01-31T22:37:37.296+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:37.297247+0000) 2022-01-31T22:37:37.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:37 smithi171 conmon[41853]: debug 2022-01-31T22:37:37.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:37.637296+0000) 2022-01-31T22:37:37.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:37 smithi167 conmon[49112]: debug 2022-01-31T22:37:37.766+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:37.767523+0000) 2022-01-31T22:37:37.984 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:37 smithi167 conmon[54076]: debug 2022-01-31T22:37:37.933+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:37.934198+0000) 2022-01-31T22:37:37.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:37 smithi167 conmon[60316]: debug 2022-01-31T22:37:37.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:37.693291+0000) 2022-01-31T22:37:38.595 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[35325]: debug 2022-01-31T22:37:38.315+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224569 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:38.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[41853]: debug 2022-01-31T22:37:38.291+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.292215+0000) 2022-01-31T22:37:38.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[51620]: debug 2022-01-31T22:37:38.291+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.292308+0000) 2022-01-31T22:37:38.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[51620]: debug 2022-01-31T22:37:38.296+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.297417+0000) 2022-01-31T22:37:38.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[46715]: debug 2022-01-31T22:37:38.291+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.291871+0000) 2022-01-31T22:37:38.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[46715]: debug 2022-01-31T22:37:38.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.435132+0000) 2022-01-31T22:37:38.665 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:38 smithi167 conmon[49112]: debug 2022-01-31T22:37:38.292+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.293849+0000) 2022-01-31T22:37:38.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:38 smithi167 conmon[54076]: debug 2022-01-31T22:37:38.292+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.293743+0000) 2022-01-31T22:37:38.667 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:38 smithi167 conmon[60316]: debug 2022-01-31T22:37:38.291+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.292843+0000) 2022-01-31T22:37:38.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[41853]: debug 2022-01-31T22:37:38.636+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.637489+0000) 2022-01-31T22:37:38.939 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:38 smithi167 conmon[49112]: debug 2022-01-31T22:37:38.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.767685+0000) 2022-01-31T22:37:38.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:38 smithi167 conmon[60316]: debug 2022-01-31T22:37:38.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.693497+0000) 2022-01-31T22:37:38.940 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:38 smithi167 conmon[54076]: debug 2022-01-31T22:37:38.932+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:38.934380+0000) 2022-01-31T22:37:39.293 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:38 smithi171 conmon[35325]: debug 2022-01-31T22:37:38.883+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:37:39.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:39 smithi171 conmon[46715]: debug 2022-01-31T22:37:39.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:39.435290+0000) 2022-01-31T22:37:39.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:39 smithi171 conmon[51620]: debug 2022-01-31T22:37:39.296+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:39.297545+0000) 2022-01-31T22:37:39.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:39 smithi171 conmon[41853]: debug 2022-01-31T22:37:39.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:39.637673+0000) 2022-01-31T22:37:39.943 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:39 smithi167 conmon[49112]: debug 2022-01-31T22:37:39.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:39.767881+0000) 2022-01-31T22:37:39.944 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:39 smithi167 conmon[54076]: debug 2022-01-31T22:37:39.933+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:39.934521+0000) 2022-01-31T22:37:39.945 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:39 smithi167 conmon[60316]: debug 2022-01-31T22:37:39.692+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:39.693648+0000) 2022-01-31T22:37:40.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:40 smithi171 conmon[46715]: debug 2022-01-31T22:37:40.434+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:40.435483+0000) 2022-01-31T22:37:40.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:40 smithi171 conmon[51620]: debug 2022-01-31T22:37:40.297+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:40.297749+0000) 2022-01-31T22:37:40.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:40 smithi171 conmon[41853]: debug 2022-01-31T22:37:40.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:40.637853+0000) 2022-01-31T22:37:40.946 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:40 smithi167 conmon[49112]: debug 2022-01-31T22:37:40.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:40.768007+0000) 2022-01-31T22:37:40.947 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:40 smithi167 conmon[54076]: debug 2022-01-31T22:37:40.934+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:40.934647+0000) 2022-01-31T22:37:40.948 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:40 smithi167 conmon[60316]: debug 2022-01-31T22:37:40.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:40.693801+0000) 2022-01-31T22:37:41.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:41 smithi171 conmon[46715]: debug 2022-01-31T22:37:41.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:41.435670+0000) 2022-01-31T22:37:41.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:41 smithi171 conmon[51620]: debug 2022-01-31T22:37:41.297+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:41.297918+0000) 2022-01-31T22:37:41.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:41 smithi171 conmon[41853]: debug 2022-01-31T22:37:41.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:41.638009+0000) 2022-01-31T22:37:41.950 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:41 smithi167 conmon[49112]: debug 2022-01-31T22:37:41.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:41.768203+0000) 2022-01-31T22:37:41.951 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:41 smithi167 conmon[54076]: debug 2022-01-31T22:37:41.934+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:41.934802+0000) 2022-01-31T22:37:41.952 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:41 smithi167 conmon[60316]: debug 2022-01-31T22:37:41.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:41.693988+0000) 2022-01-31T22:37:42.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:42 smithi171 conmon[46715]: debug 2022-01-31T22:37:42.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:42.435867+0000) 2022-01-31T22:37:42.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:42 smithi171 conmon[51620]: debug 2022-01-31T22:37:42.297+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:42.298124+0000) 2022-01-31T22:37:42.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:42 smithi171 conmon[41853]: debug 2022-01-31T22:37:42.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:42.638194+0000) 2022-01-31T22:37:42.954 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:42 smithi167 conmon[49112]: debug 2022-01-31T22:37:42.767+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:42.768408+0000) 2022-01-31T22:37:42.955 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:42 smithi167 conmon[54076]: debug 2022-01-31T22:37:42.934+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:42.934970+0000) 2022-01-31T22:37:42.955 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:42 smithi167 conmon[60316]: debug 2022-01-31T22:37:42.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:42.694122+0000) 2022-01-31T22:37:43.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[41853]: debug 2022-01-31T22:37:43.317+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.318137+0000) 2022-01-31T22:37:43.596 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[35325]: debug 2022-01-31T22:37:43.341+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224696 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:43.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[46715]: debug 2022-01-31T22:37:43.317+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.317913+0000) 2022-01-31T22:37:43.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[46715]: debug 2022-01-31T22:37:43.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.436012+0000) 2022-01-31T22:37:43.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[51620]: debug 2022-01-31T22:37:43.297+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.298330+0000) 2022-01-31T22:37:43.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[51620]: debug 2022-01-31T22:37:43.317+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.318251+0000) 2022-01-31T22:37:43.665 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:43 smithi167 conmon[60316]: debug 2022-01-31T22:37:43.317+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.318111+0000) 2022-01-31T22:37:43.666 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:43 smithi167 conmon[49112]: debug 2022-01-31T22:37:43.319+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.319689+0000) 2022-01-31T22:37:43.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:43 smithi167 conmon[54076]: debug 2022-01-31T22:37:43.318+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.319203+0000) 2022-01-31T22:37:43.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:43 smithi171 conmon[41853]: debug 2022-01-31T22:37:43.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.638385+0000) 2022-01-31T22:37:43.955 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:43 smithi167 conmon[49112]: debug 2022-01-31T22:37:43.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.768598+0000) 2022-01-31T22:37:43.956 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:43 smithi167 conmon[54076]: debug 2022-01-31T22:37:43.934+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.935144+0000) 2022-01-31T22:37:43.956 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:43 smithi167 conmon[60316]: debug 2022-01-31T22:37:43.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:43.694286+0000) 2022-01-31T22:37:44.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:44 smithi171 conmon[46715]: debug 2022-01-31T22:37:44.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:44.436176+0000) 2022-01-31T22:37:44.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:44 smithi171 conmon[51620]: debug 2022-01-31T22:37:44.297+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:44.298518+0000) 2022-01-31T22:37:44.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:44 smithi171 conmon[41853]: debug 2022-01-31T22:37:44.637+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:44.638555+0000) 2022-01-31T22:37:44.959 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:44 smithi167 conmon[49112]: debug 2022-01-31T22:37:44.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:44.768764+0000) 2022-01-31T22:37:44.960 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:44 smithi167 conmon[54076]: debug 2022-01-31T22:37:44.934+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:44.935317+0000) 2022-01-31T22:37:44.961 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:44 smithi167 conmon[60316]: debug 2022-01-31T22:37:44.693+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:44.694439+0000) 2022-01-31T22:37:45.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:45 smithi171 conmon[46715]: debug 2022-01-31T22:37:45.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:45.436328+0000) 2022-01-31T22:37:45.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:45 smithi171 conmon[51620]: debug 2022-01-31T22:37:45.298+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:45.298674+0000) 2022-01-31T22:37:45.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:45 smithi171 conmon[41853]: debug 2022-01-31T22:37:45.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:45.638758+0000) 2022-01-31T22:37:45.964 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:45 smithi167 conmon[49112]: debug 2022-01-31T22:37:45.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:45.768954+0000) 2022-01-31T22:37:45.965 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:45 smithi167 conmon[54076]: debug 2022-01-31T22:37:45.935+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:45.935535+0000) 2022-01-31T22:37:45.966 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:45 smithi167 conmon[60316]: debug 2022-01-31T22:37:45.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:45.694600+0000) 2022-01-31T22:37:46.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:46 smithi171 conmon[46715]: debug 2022-01-31T22:37:46.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:46.436484+0000) 2022-01-31T22:37:46.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:46 smithi171 conmon[51620]: debug 2022-01-31T22:37:46.298+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:46.298858+0000) 2022-01-31T22:37:46.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:46 smithi171 conmon[41853]: debug 2022-01-31T22:37:46.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:46.638931+0000) 2022-01-31T22:37:46.969 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:46 smithi167 conmon[49112]: debug 2022-01-31T22:37:46.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:46.769105+0000) 2022-01-31T22:37:46.969 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:46 smithi167 conmon[54076]: debug 2022-01-31T22:37:46.935+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:46.935723+0000) 2022-01-31T22:37:46.970 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:46 smithi167 conmon[60316]: debug 2022-01-31T22:37:46.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:46.694795+0000) 2022-01-31T22:37:47.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:47 smithi171 conmon[46715]: debug 2022-01-31T22:37:47.435+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:47.436689+0000) 2022-01-31T22:37:47.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:47 smithi171 conmon[51620]: debug 2022-01-31T22:37:47.298+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:47.299044+0000) 2022-01-31T22:37:47.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:47 smithi171 conmon[41853]: debug 2022-01-31T22:37:47.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:47.639094+0000) 2022-01-31T22:37:47.973 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:47 smithi167 conmon[49112]: debug 2022-01-31T22:37:47.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:47.769203+0000) 2022-01-31T22:37:47.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:47 smithi167 conmon[54076]: debug 2022-01-31T22:37:47.935+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:47.935906+0000) 2022-01-31T22:37:47.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:47 smithi167 conmon[60316]: debug 2022-01-31T22:37:47.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:47.694915+0000) 2022-01-31T22:37:48.596 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[35325]: debug 2022-01-31T22:37:48.367+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224808 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:48.597 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[41853]: debug 2022-01-31T22:37:48.343+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.344444+0000) 2022-01-31T22:37:48.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[46715]: debug 2022-01-31T22:37:48.343+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.344388+0000) 2022-01-31T22:37:48.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[46715]: debug 2022-01-31T22:37:48.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.436841+0000) 2022-01-31T22:37:48.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[51620]: debug 2022-01-31T22:37:48.298+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.299248+0000) 2022-01-31T22:37:48.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[51620]: debug 2022-01-31T22:37:48.344+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.345494+0000) 2022-01-31T22:37:48.665 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:48 smithi167 conmon[49112]: debug 2022-01-31T22:37:48.345+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.345708+0000) 2022-01-31T22:37:48.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:48 smithi167 conmon[54076]: debug 2022-01-31T22:37:48.345+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.345906+0000) 2022-01-31T22:37:48.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:48 smithi167 conmon[60316]: debug 2022-01-31T22:37:48.345+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.346465+0000) 2022-01-31T22:37:48.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:48 smithi171 conmon[41853]: debug 2022-01-31T22:37:48.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.639239+0000) 2022-01-31T22:37:48.974 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:48 smithi167 conmon[54076]: debug 2022-01-31T22:37:48.935+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.936058+0000) 2022-01-31T22:37:48.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:48 smithi167 conmon[60316]: debug 2022-01-31T22:37:48.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.695077+0000) 2022-01-31T22:37:48.976 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:48 smithi167 conmon[49112]: debug 2022-01-31T22:37:48.768+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:48.769317+0000) 2022-01-31T22:37:49.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:49 smithi171 conmon[46715]: debug 2022-01-31T22:37:49.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:49.436992+0000) 2022-01-31T22:37:49.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:49 smithi171 conmon[51620]: debug 2022-01-31T22:37:49.298+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:49.299487+0000) 2022-01-31T22:37:49.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:49 smithi171 conmon[41853]: debug 2022-01-31T22:37:49.638+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:49.639389+0000) 2022-01-31T22:37:49.978 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:49 smithi167 conmon[49112]: debug 2022-01-31T22:37:49.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:49.769493+0000) 2022-01-31T22:37:49.979 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:49 smithi167 conmon[54076]: debug 2022-01-31T22:37:49.935+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:49.936217+0000) 2022-01-31T22:37:49.980 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:49 smithi167 conmon[60316]: debug 2022-01-31T22:37:49.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:49.695262+0000) 2022-01-31T22:37:50.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:50 smithi171 conmon[46715]: debug 2022-01-31T22:37:50.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:50.437169+0000) 2022-01-31T22:37:50.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:50 smithi171 conmon[51620]: debug 2022-01-31T22:37:50.298+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:50.299674+0000) 2022-01-31T22:37:50.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:50 smithi171 conmon[41853]: debug 2022-01-31T22:37:50.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:50.639580+0000) 2022-01-31T22:37:50.982 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:50 smithi167 conmon[49112]: debug 2022-01-31T22:37:50.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:50.769685+0000) 2022-01-31T22:37:50.983 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:50 smithi167 conmon[54076]: debug 2022-01-31T22:37:50.936+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:50.936419+0000) 2022-01-31T22:37:50.983 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:50 smithi167 conmon[60316]: debug 2022-01-31T22:37:50.694+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:50.695448+0000) 2022-01-31T22:37:51.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:51 smithi171 conmon[46715]: debug 2022-01-31T22:37:51.436+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:51.437386+0000) 2022-01-31T22:37:51.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:51 smithi171 conmon[51620]: debug 2022-01-31T22:37:51.299+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:51.299859+0000) 2022-01-31T22:37:51.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:51 smithi171 conmon[41853]: debug 2022-01-31T22:37:51.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:51.639771+0000) 2022-01-31T22:37:51.985 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:51 smithi167 conmon[54076]: debug 2022-01-31T22:37:51.936+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:51.936604+0000) 2022-01-31T22:37:51.986 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:51 smithi167 conmon[60316]: debug 2022-01-31T22:37:51.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:51.695648+0000) 2022-01-31T22:37:51.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:51 smithi167 conmon[49112]: debug 2022-01-31T22:37:51.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:51.769825+0000) 2022-01-31T22:37:52.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:52 smithi171 conmon[46715]: debug 2022-01-31T22:37:52.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:52.437542+0000) 2022-01-31T22:37:52.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:52 smithi171 conmon[51620]: debug 2022-01-31T22:37:52.299+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:52.299988+0000) 2022-01-31T22:37:52.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:52 smithi171 conmon[41853]: debug 2022-01-31T22:37:52.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:52.639945+0000) 2022-01-31T22:37:52.985 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:52 smithi167 conmon[54076]: debug 2022-01-31T22:37:52.936+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:52.936740+0000) 2022-01-31T22:37:52.986 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:52 smithi167 conmon[60316]: debug 2022-01-31T22:37:52.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:52.695834+0000) 2022-01-31T22:37:52.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:52 smithi167 conmon[49112]: debug 2022-01-31T22:37:52.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:52.769972+0000) 2022-01-31T22:37:53.597 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[41853]: debug 2022-01-31T22:37:53.370+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.370914+0000) 2022-01-31T22:37:53.598 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[35325]: debug 2022-01-31T22:37:53.394+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 224920 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:53.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[46715]: debug 2022-01-31T22:37:53.370+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.371255+0000) 2022-01-31T22:37:53.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[46715]: debug 2022-01-31T22:37:53.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.437746+0000) 2022-01-31T22:37:53.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[51620]: debug 2022-01-31T22:37:53.299+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.300150+0000) 2022-01-31T22:37:53.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[51620]: debug 2022-01-31T22:37:53.370+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.370743+0000) 2022-01-31T22:37:53.665 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:53 smithi167 conmon[49112]: debug 2022-01-31T22:37:53.371+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.372392+0000) 2022-01-31T22:37:53.666 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:53 smithi167 conmon[54076]: debug 2022-01-31T22:37:53.371+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.372184+0000) 2022-01-31T22:37:53.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:53 smithi167 conmon[60316]: debug 2022-01-31T22:37:53.372+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.373040+0000) 2022-01-31T22:37:53.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[41853]: debug 2022-01-31T22:37:53.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.640122+0000) 2022-01-31T22:37:53.985 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:53 smithi167 conmon[49112]: debug 2022-01-31T22:37:53.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.770139+0000) 2022-01-31T22:37:53.986 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:53 smithi167 conmon[54076]: debug 2022-01-31T22:37:53.936+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.936892+0000) 2022-01-31T22:37:53.987 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:53 smithi167 conmon[60316]: debug 2022-01-31T22:37:53.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:53.695994+0000) 2022-01-31T22:37:54.296 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:53 smithi171 conmon[35325]: debug 2022-01-31T22:37:53.884+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:37:54.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:54 smithi171 conmon[46715]: debug 2022-01-31T22:37:54.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:54.437949+0000) 2022-01-31T22:37:54.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:54 smithi171 conmon[51620]: debug 2022-01-31T22:37:54.299+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:54.300359+0000) 2022-01-31T22:37:54.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:54 smithi171 conmon[41853]: debug 2022-01-31T22:37:54.639+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:54.640276+0000) 2022-01-31T22:37:54.985 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:54 smithi167 conmon[54076]: debug 2022-01-31T22:37:54.936+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:54.937047+0000) 2022-01-31T22:37:54.986 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:54 smithi167 conmon[60316]: debug 2022-01-31T22:37:54.695+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:54.696144+0000) 2022-01-31T22:37:54.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:54 smithi167 conmon[49112]: debug 2022-01-31T22:37:54.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:54.770275+0000) 2022-01-31T22:37:55.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:55 smithi171 conmon[46715]: debug 2022-01-31T22:37:55.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:55.438137+0000) 2022-01-31T22:37:55.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:55 smithi171 conmon[51620]: debug 2022-01-31T22:37:55.299+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:55.300557+0000) 2022-01-31T22:37:55.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:55 smithi171 conmon[41853]: debug 2022-01-31T22:37:55.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:55.640431+0000) 2022-01-31T22:37:55.985 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:55 smithi167 conmon[49112]: debug 2022-01-31T22:37:55.769+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:55.770479+0000) 2022-01-31T22:37:55.986 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:55 smithi167 conmon[54076]: debug 2022-01-31T22:37:55.936+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:55.937265+0000) 2022-01-31T22:37:55.987 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:55 smithi167 conmon[60316]: debug 2022-01-31T22:37:55.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:55.696325+0000) 2022-01-31T22:37:56.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:56 smithi171 conmon[46715]: debug 2022-01-31T22:37:56.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:56.438317+0000) 2022-01-31T22:37:56.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:56 smithi171 conmon[51620]: debug 2022-01-31T22:37:56.300+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:56.300768+0000) 2022-01-31T22:37:56.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:56 smithi171 conmon[41853]: debug 2022-01-31T22:37:56.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:56.640618+0000) 2022-01-31T22:37:56.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:56 smithi167 conmon[49112]: debug 2022-01-31T22:37:56.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:56.770706+0000) 2022-01-31T22:37:56.987 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:56 smithi167 conmon[54076]: debug 2022-01-31T22:37:56.937+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:56.937473+0000) 2022-01-31T22:37:56.987 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:56 smithi167 conmon[60316]: debug 2022-01-31T22:37:56.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:56.696536+0000) 2022-01-31T22:37:57.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:57 smithi171 conmon[46715]: debug 2022-01-31T22:37:57.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:57.438481+0000) 2022-01-31T22:37:57.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:57 smithi171 conmon[51620]: debug 2022-01-31T22:37:57.300+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:57.300945+0000) 2022-01-31T22:37:57.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:57 smithi171 conmon[41853]: debug 2022-01-31T22:37:57.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:57.640800+0000) 2022-01-31T22:37:57.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:57 smithi167 conmon[49112]: debug 2022-01-31T22:37:57.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:57.770893+0000) 2022-01-31T22:37:57.987 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:57 smithi167 conmon[54076]: debug 2022-01-31T22:37:57.937+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:57.937679+0000) 2022-01-31T22:37:57.988 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:57 smithi167 conmon[60316]: debug 2022-01-31T22:37:57.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:57.696670+0000) 2022-01-31T22:37:58.598 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[35325]: debug 2022-01-31T22:37:58.419+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225034 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:37:58.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[41853]: debug 2022-01-31T22:37:58.396+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.396980+0000) 2022-01-31T22:37:58.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[46715]: debug 2022-01-31T22:37:58.397+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.397875+0000) 2022-01-31T22:37:58.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[46715]: debug 2022-01-31T22:37:58.437+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.438665+0000) 2022-01-31T22:37:58.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[51620]: debug 2022-01-31T22:37:58.300+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.301063+0000) 2022-01-31T22:37:58.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[51620]: debug 2022-01-31T22:37:58.395+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.396674+0000) 2022-01-31T22:37:58.664 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:58 smithi167 conmon[49112]: debug 2022-01-31T22:37:58.397+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.397808+0000) 2022-01-31T22:37:58.665 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:58 smithi167 conmon[54076]: debug 2022-01-31T22:37:58.397+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.398459+0000) 2022-01-31T22:37:58.666 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:58 smithi167 conmon[60316]: debug 2022-01-31T22:37:58.398+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.398625+0000) 2022-01-31T22:37:58.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:58 smithi171 conmon[41853]: debug 2022-01-31T22:37:58.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.640965+0000) 2022-01-31T22:37:58.986 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:58 smithi167 conmon[60316]: debug 2022-01-31T22:37:58.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.696845+0000) 2022-01-31T22:37:58.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:58 smithi167 conmon[49112]: debug 2022-01-31T22:37:58.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.771087+0000) 2022-01-31T22:37:58.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:58 smithi167 conmon[54076]: debug 2022-01-31T22:37:58.937+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:58.937839+0000) 2022-01-31T22:37:59.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:37:59 smithi171 conmon[46715]: debug 2022-01-31T22:37:59.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:59.438809+0000) 2022-01-31T22:37:59.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:37:59 smithi171 conmon[51620]: debug 2022-01-31T22:37:59.300+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:59.301206+0000) 2022-01-31T22:37:59.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:37:59 smithi171 conmon[41853]: debug 2022-01-31T22:37:59.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:59.641080+0000) 2022-01-31T22:37:59.986 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:37:59 smithi167 conmon[49112]: debug 2022-01-31T22:37:59.770+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:59.771301+0000) 2022-01-31T22:37:59.987 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:37:59 smithi167 conmon[54076]: debug 2022-01-31T22:37:59.937+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:59.938008+0000) 2022-01-31T22:37:59.988 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:37:59 smithi167 conmon[60316]: debug 2022-01-31T22:37:59.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:37:59.696939+0000) 2022-01-31T22:38:00.598 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:00 smithi171 conmon[46715]: debug 2022-01-31T22:38:00.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:00.438966+0000) 2022-01-31T22:38:00.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:00 smithi171 conmon[51620]: debug 2022-01-31T22:38:00.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:00.301365+0000) 2022-01-31T22:38:00.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:00 smithi171 conmon[41853]: debug 2022-01-31T22:38:00.640+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:00.641276+0000) 2022-01-31T22:38:00.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:00 smithi167 conmon[49112]: debug 2022-01-31T22:38:00.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:00.771447+0000) 2022-01-31T22:38:00.988 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:00 smithi167 conmon[54076]: debug 2022-01-31T22:38:00.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:00.938203+0000) 2022-01-31T22:38:00.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:00 smithi167 conmon[60316]: debug 2022-01-31T22:38:00.696+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:00.697156+0000) 2022-01-31T22:38:01.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:01 smithi171 conmon[46715]: debug 2022-01-31T22:38:01.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:01.439157+0000) 2022-01-31T22:38:01.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:01 smithi171 conmon[51620]: debug 2022-01-31T22:38:01.300+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:01.301518+0000) 2022-01-31T22:38:01.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:01 smithi171 conmon[41853]: debug 2022-01-31T22:38:01.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:01.641476+0000) 2022-01-31T22:38:01.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:01 smithi167 conmon[49112]: debug 2022-01-31T22:38:01.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:01.771635+0000) 2022-01-31T22:38:01.987 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:01 smithi167 conmon[54076]: debug 2022-01-31T22:38:01.937+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:01.938385+0000) 2022-01-31T22:38:01.988 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:01 smithi167 conmon[60316]: debug 2022-01-31T22:38:01.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:01.697360+0000) 2022-01-31T22:38:02.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:02 smithi171 conmon[46715]: debug 2022-01-31T22:38:02.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:02.439308+0000) 2022-01-31T22:38:02.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:02 smithi171 conmon[51620]: debug 2022-01-31T22:38:02.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:02.301690+0000) 2022-01-31T22:38:02.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:02 smithi171 conmon[41853]: debug 2022-01-31T22:38:02.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:02.641635+0000) 2022-01-31T22:38:02.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:02 smithi167 conmon[49112]: debug 2022-01-31T22:38:02.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:02.771786+0000) 2022-01-31T22:38:02.988 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:02 smithi167 conmon[54076]: debug 2022-01-31T22:38:02.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:02.938515+0000) 2022-01-31T22:38:02.988 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:02 smithi167 conmon[60316]: debug 2022-01-31T22:38:02.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:02.697580+0000) 2022-01-31T22:38:03.599 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[35325]: debug 2022-01-31T22:38:03.445+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225144 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:03.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[41853]: debug 2022-01-31T22:38:03.423+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.423917+0000) 2022-01-31T22:38:03.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[46715]: debug 2022-01-31T22:38:03.422+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.423023+0000) 2022-01-31T22:38:03.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[46715]: debug 2022-01-31T22:38:03.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.439456+0000) 2022-01-31T22:38:03.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[51620]: debug 2022-01-31T22:38:03.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.301853+0000) 2022-01-31T22:38:03.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[51620]: debug 2022-01-31T22:38:03.422+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.423176+0000) 2022-01-31T22:38:03.693 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:03 smithi167 conmon[49112]: debug 2022-01-31T22:38:03.423+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.423652+0000) 2022-01-31T22:38:03.694 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:03 smithi167 conmon[54076]: debug 2022-01-31T22:38:03.423+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.424275+0000) 2022-01-31T22:38:03.694 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:03 smithi167 conmon[60316]: debug 2022-01-31T22:38:03.423+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.424165+0000) 2022-01-31T22:38:03.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:03 smithi171 conmon[41853]: debug 2022-01-31T22:38:03.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.641818+0000) 2022-01-31T22:38:03.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:03 smithi167 conmon[49112]: debug 2022-01-31T22:38:03.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.771971+0000) 2022-01-31T22:38:03.988 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:03 smithi167 conmon[54076]: debug 2022-01-31T22:38:03.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.938670+0000) 2022-01-31T22:38:03.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:03 smithi167 conmon[60316]: debug 2022-01-31T22:38:03.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:03.697737+0000) 2022-01-31T22:38:04.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:04 smithi171 conmon[46715]: debug 2022-01-31T22:38:04.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:04.439676+0000) 2022-01-31T22:38:04.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:04 smithi171 conmon[51620]: debug 2022-01-31T22:38:04.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:04.302051+0000) 2022-01-31T22:38:04.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:04 smithi171 conmon[41853]: debug 2022-01-31T22:38:04.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:04.641959+0000) 2022-01-31T22:38:04.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:04 smithi167 conmon[49112]: debug 2022-01-31T22:38:04.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:04.772127+0000) 2022-01-31T22:38:04.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:04 smithi167 conmon[54076]: debug 2022-01-31T22:38:04.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:04.938821+0000) 2022-01-31T22:38:04.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:04 smithi167 conmon[60316]: debug 2022-01-31T22:38:04.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:04.697879+0000) 2022-01-31T22:38:05.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:05 smithi171 conmon[46715]: debug 2022-01-31T22:38:05.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:05.439926+0000) 2022-01-31T22:38:05.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:05 smithi171 conmon[51620]: debug 2022-01-31T22:38:05.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:05.302226+0000) 2022-01-31T22:38:05.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:05 smithi171 conmon[41853]: debug 2022-01-31T22:38:05.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:05.642164+0000) 2022-01-31T22:38:05.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:05 smithi167 conmon[49112]: debug 2022-01-31T22:38:05.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:05.772322+0000) 2022-01-31T22:38:05.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:05 smithi167 conmon[60316]: debug 2022-01-31T22:38:05.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:05.698043+0000) 2022-01-31T22:38:05.990 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:05 smithi167 conmon[54076]: debug 2022-01-31T22:38:05.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:05.938995+0000) 2022-01-31T22:38:06.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:06 smithi171 conmon[46715]: debug 2022-01-31T22:38:06.440+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:06.440128+0000) 2022-01-31T22:38:06.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:06 smithi171 conmon[51620]: debug 2022-01-31T22:38:06.302+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:06.302401+0000) 2022-01-31T22:38:06.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:06 smithi171 conmon[41853]: debug 2022-01-31T22:38:06.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:06.642353+0000) 2022-01-31T22:38:06.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:06 smithi167 conmon[49112]: debug 2022-01-31T22:38:06.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:06.772543+0000) 2022-01-31T22:38:06.988 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:06 smithi167 conmon[60316]: debug 2022-01-31T22:38:06.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:06.698238+0000) 2022-01-31T22:38:06.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:06 smithi167 conmon[54076]: debug 2022-01-31T22:38:06.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:06.939181+0000) 2022-01-31T22:38:07.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:07 smithi171 conmon[46715]: debug 2022-01-31T22:38:07.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:07.440242+0000) 2022-01-31T22:38:07.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:07 smithi171 conmon[51620]: debug 2022-01-31T22:38:07.302+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:07.302578+0000) 2022-01-31T22:38:07.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:07 smithi171 conmon[41853]: debug 2022-01-31T22:38:07.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:07.642522+0000) 2022-01-31T22:38:07.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:07 smithi167 conmon[49112]: debug 2022-01-31T22:38:07.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:07.772740+0000) 2022-01-31T22:38:07.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:07 smithi167 conmon[54076]: debug 2022-01-31T22:38:07.937+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:07.939401+0000) 2022-01-31T22:38:07.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:07 smithi167 conmon[60316]: debug 2022-01-31T22:38:07.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:07.698443+0000) 2022-01-31T22:38:08.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[35325]: debug 2022-01-31T22:38:08.485+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225257 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:08.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[41853]: debug 2022-01-31T22:38:08.447+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.449478+0000) 2022-01-31T22:38:08.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[51620]: debug 2022-01-31T22:38:08.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.302771+0000) 2022-01-31T22:38:08.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[51620]: debug 2022-01-31T22:38:08.447+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.449052+0000) 2022-01-31T22:38:08.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[46715]: debug 2022-01-31T22:38:08.438+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.440444+0000) 2022-01-31T22:38:08.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[46715]: debug 2022-01-31T22:38:08.446+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.448713+0000) 2022-01-31T22:38:08.694 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:08 smithi167 conmon[49112]: debug 2022-01-31T22:38:08.455+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.457048+0000) 2022-01-31T22:38:08.695 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:08 smithi167 conmon[54076]: debug 2022-01-31T22:38:08.447+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.448604+0000) 2022-01-31T22:38:08.695 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:08 smithi167 conmon[60316]: debug 2022-01-31T22:38:08.456+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.457625+0000) 2022-01-31T22:38:08.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[41853]: debug 2022-01-31T22:38:08.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.642723+0000) 2022-01-31T22:38:08.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:08 smithi167 conmon[49112]: debug 2022-01-31T22:38:08.771+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.772912+0000) 2022-01-31T22:38:08.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:08 smithi167 conmon[54076]: debug 2022-01-31T22:38:08.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.939602+0000) 2022-01-31T22:38:08.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:08 smithi167 conmon[60316]: debug 2022-01-31T22:38:08.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:08.698641+0000) 2022-01-31T22:38:09.298 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:08 smithi171 conmon[35325]: debug 2022-01-31T22:38:08.884+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:38:09.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:09 smithi171 conmon[46715]: debug 2022-01-31T22:38:09.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:09.440626+0000) 2022-01-31T22:38:09.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:09 smithi171 conmon[51620]: debug 2022-01-31T22:38:09.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:09.302941+0000) 2022-01-31T22:38:09.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:09 smithi171 conmon[41853]: debug 2022-01-31T22:38:09.641+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:09.642855+0000) 2022-01-31T22:38:09.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:09 smithi167 conmon[49112]: debug 2022-01-31T22:38:09.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:09.773078+0000) 2022-01-31T22:38:09.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:09 smithi167 conmon[54076]: debug 2022-01-31T22:38:09.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:09.939760+0000) 2022-01-31T22:38:09.989 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:09 smithi167 conmon[60316]: debug 2022-01-31T22:38:09.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:09.698779+0000) 2022-01-31T22:38:10.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:10 smithi171 conmon[46715]: debug 2022-01-31T22:38:10.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:10.440792+0000) 2022-01-31T22:38:10.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:10 smithi171 conmon[51620]: debug 2022-01-31T22:38:10.302+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:10.303100+0000) 2022-01-31T22:38:10.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:10 smithi171 conmon[41853]: debug 2022-01-31T22:38:10.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:10.643062+0000) 2022-01-31T22:38:10.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:10 smithi167 conmon[49112]: debug 2022-01-31T22:38:10.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:10.773276+0000) 2022-01-31T22:38:10.990 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:10 smithi167 conmon[54076]: debug 2022-01-31T22:38:10.939+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:10.939982+0000) 2022-01-31T22:38:10.990 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:10 smithi167 conmon[60316]: debug 2022-01-31T22:38:10.697+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:10.698958+0000) 2022-01-31T22:38:11.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:11 smithi171 conmon[46715]: debug 2022-01-31T22:38:11.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:11.440949+0000) 2022-01-31T22:38:11.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:11 smithi171 conmon[51620]: debug 2022-01-31T22:38:11.302+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:11.303287+0000) 2022-01-31T22:38:11.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:11 smithi171 conmon[41853]: debug 2022-01-31T22:38:11.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:11.643273+0000) 2022-01-31T22:38:11.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:11 smithi167 conmon[49112]: debug 2022-01-31T22:38:11.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:11.773466+0000) 2022-01-31T22:38:11.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:11 smithi167 conmon[54076]: debug 2022-01-31T22:38:11.939+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:11.940167+0000) 2022-01-31T22:38:11.990 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:11 smithi167 conmon[60316]: debug 2022-01-31T22:38:11.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:11.699147+0000) 2022-01-31T22:38:12.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:12 smithi171 conmon[46715]: debug 2022-01-31T22:38:12.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:12.441116+0000) 2022-01-31T22:38:12.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:12 smithi171 conmon[51620]: debug 2022-01-31T22:38:12.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:12.303447+0000) 2022-01-31T22:38:12.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:12 smithi171 conmon[41853]: debug 2022-01-31T22:38:12.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:12.643478+0000) 2022-01-31T22:38:12.988 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:12 smithi167 conmon[54076]: debug 2022-01-31T22:38:12.938+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:12.940381+0000) 2022-01-31T22:38:12.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:12 smithi167 conmon[49112]: debug 2022-01-31T22:38:12.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:12.773709+0000) 2022-01-31T22:38:12.990 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:12 smithi167 conmon[60316]: debug 2022-01-31T22:38:12.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:12.699336+0000) 2022-01-31T22:38:13.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[35325]: debug 2022-01-31T22:38:13.512+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225367 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:13.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[41853]: debug 2022-01-31T22:38:13.487+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.489424+0000) 2022-01-31T22:38:13.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[46715]: debug 2022-01-31T22:38:13.440+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.441292+0000) 2022-01-31T22:38:13.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[46715]: debug 2022-01-31T22:38:13.488+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.490135+0000) 2022-01-31T22:38:13.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[51620]: debug 2022-01-31T22:38:13.301+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.303642+0000) 2022-01-31T22:38:13.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[51620]: debug 2022-01-31T22:38:13.489+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.490792+0000) 2022-01-31T22:38:13.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:13 smithi167 conmon[49112]: debug 2022-01-31T22:38:13.489+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.491480+0000) 2022-01-31T22:38:13.696 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:13 smithi167 conmon[54076]: debug 2022-01-31T22:38:13.488+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.490130+0000) 2022-01-31T22:38:13.697 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:13 smithi167 conmon[60316]: debug 2022-01-31T22:38:13.489+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.490674+0000) 2022-01-31T22:38:13.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:13 smithi171 conmon[41853]: debug 2022-01-31T22:38:13.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.643672+0000) 2022-01-31T22:38:13.988 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:13 smithi167 conmon[49112]: debug 2022-01-31T22:38:13.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.773896+0000) 2022-01-31T22:38:13.989 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:13 smithi167 conmon[54076]: debug 2022-01-31T22:38:13.939+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.940566+0000) 2022-01-31T22:38:13.990 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:13 smithi167 conmon[60316]: debug 2022-01-31T22:38:13.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:13.699556+0000) 2022-01-31T22:38:14.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:14 smithi171 conmon[46715]: debug 2022-01-31T22:38:14.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:14.441505+0000) 2022-01-31T22:38:14.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:14 smithi171 conmon[51620]: debug 2022-01-31T22:38:14.302+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:14.303794+0000) 2022-01-31T22:38:14.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:14 smithi171 conmon[41853]: debug 2022-01-31T22:38:14.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:14.643833+0000) 2022-01-31T22:38:14.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:14 smithi167 conmon[49112]: debug 2022-01-31T22:38:14.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:14.774049+0000) 2022-01-31T22:38:14.990 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:14 smithi167 conmon[54076]: debug 2022-01-31T22:38:14.939+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:14.940681+0000) 2022-01-31T22:38:14.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:14 smithi167 conmon[60316]: debug 2022-01-31T22:38:14.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:14.699738+0000) 2022-01-31T22:38:15.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:15 smithi171 conmon[46715]: debug 2022-01-31T22:38:15.439+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:15.441665+0000) 2022-01-31T22:38:15.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:15 smithi171 conmon[51620]: debug 2022-01-31T22:38:15.303+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:15.303936+0000) 2022-01-31T22:38:15.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:15 smithi171 conmon[41853]: debug 2022-01-31T22:38:15.642+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:15.644008+0000) 2022-01-31T22:38:15.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:15 smithi167 conmon[49112]: debug 2022-01-31T22:38:15.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:15.774249+0000) 2022-01-31T22:38:15.990 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:15 smithi167 conmon[54076]: debug 2022-01-31T22:38:15.939+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:15.940849+0000) 2022-01-31T22:38:15.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:15 smithi167 conmon[60316]: debug 2022-01-31T22:38:15.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:15.699922+0000) 2022-01-31T22:38:16.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:16 smithi171 conmon[46715]: debug 2022-01-31T22:38:16.441+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:16.441880+0000) 2022-01-31T22:38:16.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:16 smithi171 conmon[51620]: debug 2022-01-31T22:38:16.302+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:16.304053+0000) 2022-01-31T22:38:16.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:16 smithi171 conmon[41853]: debug 2022-01-31T22:38:16.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:16.644159+0000) 2022-01-31T22:38:16.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:16 smithi167 conmon[49112]: debug 2022-01-31T22:38:16.772+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:16.774414+0000) 2022-01-31T22:38:16.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:16 smithi167 conmon[54076]: debug 2022-01-31T22:38:16.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:16.941050+0000) 2022-01-31T22:38:16.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:16 smithi167 conmon[60316]: debug 2022-01-31T22:38:16.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:16.700078+0000) 2022-01-31T22:38:17.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:17 smithi171 conmon[51620]: debug 2022-01-31T22:38:17.303+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:17.304240+0000) 2022-01-31T22:38:17.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:17 smithi171 conmon[46715]: debug 2022-01-31T22:38:17.441+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:17.442041+0000) 2022-01-31T22:38:17.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:17 smithi171 conmon[41853]: debug 2022-01-31T22:38:17.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:17.644359+0000) 2022-01-31T22:38:17.989 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:17 smithi167 conmon[49112]: debug 2022-01-31T22:38:17.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:17.774605+0000) 2022-01-31T22:38:17.990 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:17 smithi167 conmon[54076]: debug 2022-01-31T22:38:17.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:17.941224+0000) 2022-01-31T22:38:17.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:17 smithi167 conmon[60316]: debug 2022-01-31T22:38:17.698+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:17.700282+0000) 2022-01-31T22:38:18.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[46715]: debug 2022-01-31T22:38:18.441+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.442209+0000) 2022-01-31T22:38:18.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[46715]: debug 2022-01-31T22:38:18.515+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.517346+0000) 2022-01-31T22:38:18.603 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[35325]: debug 2022-01-31T22:38:18.538+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225480 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:18.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[41853]: debug 2022-01-31T22:38:18.514+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.515872+0000) 2022-01-31T22:38:18.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[51620]: debug 2022-01-31T22:38:18.303+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.304429+0000) 2022-01-31T22:38:18.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[51620]: debug 2022-01-31T22:38:18.514+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.516287+0000) 2022-01-31T22:38:18.695 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:18 smithi167 conmon[49112]: debug 2022-01-31T22:38:18.515+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.517072+0000) 2022-01-31T22:38:18.697 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:18 smithi167 conmon[54076]: debug 2022-01-31T22:38:18.515+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.516726+0000) 2022-01-31T22:38:18.697 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:18 smithi167 conmon[60316]: debug 2022-01-31T22:38:18.515+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.517331+0000) 2022-01-31T22:38:18.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:18 smithi171 conmon[41853]: debug 2022-01-31T22:38:18.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.644561+0000) 2022-01-31T22:38:18.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:18 smithi167 conmon[49112]: debug 2022-01-31T22:38:18.773+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.774784+0000) 2022-01-31T22:38:18.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:18 smithi167 conmon[54076]: debug 2022-01-31T22:38:18.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.941357+0000) 2022-01-31T22:38:18.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:18 smithi167 conmon[60316]: debug 2022-01-31T22:38:18.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:18.700470+0000) 2022-01-31T22:38:19.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:19 smithi171 conmon[46715]: debug 2022-01-31T22:38:19.441+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:19.442413+0000) 2022-01-31T22:38:19.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:19 smithi171 conmon[51620]: debug 2022-01-31T22:38:19.303+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:19.304613+0000) 2022-01-31T22:38:19.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:19 smithi171 conmon[41853]: debug 2022-01-31T22:38:19.643+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:19.644690+0000) 2022-01-31T22:38:19.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:19 smithi167 conmon[49112]: debug 2022-01-31T22:38:19.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:19.774950+0000) 2022-01-31T22:38:19.990 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:19 smithi167 conmon[54076]: debug 2022-01-31T22:38:19.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:19.941543+0000) 2022-01-31T22:38:19.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:19 smithi167 conmon[60316]: debug 2022-01-31T22:38:19.699+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:19.700666+0000) 2022-01-31T22:38:20.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:20 smithi171 conmon[46715]: debug 2022-01-31T22:38:20.441+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:20.442575+0000) 2022-01-31T22:38:20.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:20 smithi171 conmon[51620]: debug 2022-01-31T22:38:20.304+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:20.304794+0000) 2022-01-31T22:38:20.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:20 smithi171 conmon[41853]: debug 2022-01-31T22:38:20.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:20.644846+0000) 2022-01-31T22:38:20.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:20 smithi167 conmon[49112]: debug 2022-01-31T22:38:20.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:20.775110+0000) 2022-01-31T22:38:20.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:20 smithi167 conmon[54076]: debug 2022-01-31T22:38:20.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:20.941740+0000) 2022-01-31T22:38:20.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:20 smithi167 conmon[60316]: debug 2022-01-31T22:38:20.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:20.700862+0000) 2022-01-31T22:38:21.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:21 smithi171 conmon[46715]: debug 2022-01-31T22:38:21.442+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:21.442789+0000) 2022-01-31T22:38:21.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:21 smithi171 conmon[51620]: debug 2022-01-31T22:38:21.304+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:21.305060+0000) 2022-01-31T22:38:21.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:21 smithi171 conmon[41853]: debug 2022-01-31T22:38:21.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:21.645028+0000) 2022-01-31T22:38:21.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:21 smithi167 conmon[49112]: debug 2022-01-31T22:38:21.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:21.775299+0000) 2022-01-31T22:38:21.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:21 smithi167 conmon[54076]: debug 2022-01-31T22:38:21.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:21.941921+0000) 2022-01-31T22:38:21.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:21 smithi167 conmon[60316]: debug 2022-01-31T22:38:21.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:21.701027+0000) 2022-01-31T22:38:22.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:22 smithi171 conmon[46715]: debug 2022-01-31T22:38:22.442+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:22.442985+0000) 2022-01-31T22:38:22.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:22 smithi171 conmon[51620]: debug 2022-01-31T22:38:22.304+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:22.305195+0000) 2022-01-31T22:38:22.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:22 smithi171 conmon[41853]: debug 2022-01-31T22:38:22.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:22.645243+0000) 2022-01-31T22:38:22.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:22 smithi167 conmon[49112]: debug 2022-01-31T22:38:22.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:22.775533+0000) 2022-01-31T22:38:22.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:22 smithi167 conmon[54076]: debug 2022-01-31T22:38:22.941+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:22.942133+0000) 2022-01-31T22:38:22.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:22 smithi167 conmon[60316]: debug 2022-01-31T22:38:22.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:22.701212+0000) 2022-01-31T22:38:23.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[41853]: debug 2022-01-31T22:38:23.541+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.542342+0000) 2022-01-31T22:38:23.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[46715]: debug 2022-01-31T22:38:23.442+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.443146+0000) 2022-01-31T22:38:23.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[46715]: debug 2022-01-31T22:38:23.542+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.543091+0000) 2022-01-31T22:38:23.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[51620]: debug 2022-01-31T22:38:23.304+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.305330+0000) 2022-01-31T22:38:23.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[51620]: debug 2022-01-31T22:38:23.542+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.543345+0000) 2022-01-31T22:38:23.696 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:23 smithi167 conmon[54076]: debug 2022-01-31T22:38:23.542+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.543834+0000) 2022-01-31T22:38:23.697 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:23 smithi167 conmon[60316]: debug 2022-01-31T22:38:23.542+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.543768+0000) 2022-01-31T22:38:23.698 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:23 smithi167 conmon[49112]: debug 2022-01-31T22:38:23.541+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.542705+0000) 2022-01-31T22:38:23.852 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[35325]: debug 2022-01-31T22:38:23.566+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225590 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:23.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[41853]: debug 2022-01-31T22:38:23.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.645432+0000) 2022-01-31T22:38:23.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:23 smithi167 conmon[49112]: debug 2022-01-31T22:38:23.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.775743+0000) 2022-01-31T22:38:23.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:23 smithi167 conmon[54076]: debug 2022-01-31T22:38:23.940+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.942335+0000) 2022-01-31T22:38:23.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:23 smithi167 conmon[60316]: debug 2022-01-31T22:38:23.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:23.701422+0000) 2022-01-31T22:38:24.300 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:23 smithi171 conmon[35325]: debug 2022-01-31T22:38:23.885+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:38:24.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:24 smithi171 conmon[46715]: debug 2022-01-31T22:38:24.442+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:24.443343+0000) 2022-01-31T22:38:24.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:24 smithi171 conmon[51620]: debug 2022-01-31T22:38:24.304+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:24.305536+0000) 2022-01-31T22:38:24.852 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:24 smithi171 conmon[41853]: debug 2022-01-31T22:38:24.644+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:24.645606+0000) 2022-01-31T22:38:24.990 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:24 smithi167 conmon[49112]: debug 2022-01-31T22:38:24.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:24.775888+0000) 2022-01-31T22:38:24.991 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:24 smithi167 conmon[60316]: debug 2022-01-31T22:38:24.700+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:24.701601+0000) 2022-01-31T22:38:24.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:24 smithi167 conmon[54076]: debug 2022-01-31T22:38:24.941+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:24.942459+0000) 2022-01-31T22:38:25.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:25 smithi171 conmon[46715]: debug 2022-01-31T22:38:25.442+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:25.443528+0000) 2022-01-31T22:38:25.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:25 smithi171 conmon[51620]: debug 2022-01-31T22:38:25.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:25.305700+0000) 2022-01-31T22:38:25.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:25 smithi167 conmon[49112]: debug 2022-01-31T22:38:25.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:25.776075+0000) 2022-01-31T22:38:25.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:25 smithi167 conmon[54076]: debug 2022-01-31T22:38:25.941+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:25.942594+0000) 2022-01-31T22:38:25.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:25 smithi167 conmon[60316]: debug 2022-01-31T22:38:25.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:25.701786+0000) 2022-01-31T22:38:26.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:25 smithi171 conmon[41853]: debug 2022-01-31T22:38:25.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:25.645785+0000) 2022-01-31T22:38:26.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:26 smithi171 conmon[51620]: debug 2022-01-31T22:38:26.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:26.305843+0000) 2022-01-31T22:38:26.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:26 smithi171 conmon[46715]: debug 2022-01-31T22:38:26.443+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:26.443716+0000) 2022-01-31T22:38:26.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:26 smithi167 conmon[49112]: debug 2022-01-31T22:38:26.774+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:26.776257+0000) 2022-01-31T22:38:26.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:26 smithi167 conmon[54076]: debug 2022-01-31T22:38:26.942+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:26.942773+0000) 2022-01-31T22:38:26.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:26 smithi167 conmon[60316]: debug 2022-01-31T22:38:26.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:26.701945+0000) 2022-01-31T22:38:27.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:26 smithi171 conmon[41853]: debug 2022-01-31T22:38:26.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:26.645955+0000) 2022-01-31T22:38:27.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:27 smithi171 conmon[46715]: debug 2022-01-31T22:38:27.443+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:27.443922+0000) 2022-01-31T22:38:27.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:27 smithi171 conmon[51620]: debug 2022-01-31T22:38:27.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:27.305991+0000) 2022-01-31T22:38:27.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:27 smithi167 conmon[49112]: debug 2022-01-31T22:38:27.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:27.776443+0000) 2022-01-31T22:38:27.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:27 smithi167 conmon[54076]: debug 2022-01-31T22:38:27.942+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:27.942985+0000) 2022-01-31T22:38:27.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:27 smithi167 conmon[60316]: debug 2022-01-31T22:38:27.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:27.702131+0000) 2022-01-31T22:38:28.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:27 smithi171 conmon[41853]: debug 2022-01-31T22:38:27.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:27.646148+0000) 2022-01-31T22:38:28.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[46715]: debug 2022-01-31T22:38:28.443+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.444114+0000) 2022-01-31T22:38:28.564 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[51620]: debug 2022-01-31T22:38:28.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.306143+0000) 2022-01-31T22:38:28.697 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:28 smithi167 conmon[49112]: debug 2022-01-31T22:38:28.568+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.569619+0000) 2022-01-31T22:38:28.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:28 smithi167 conmon[54076]: debug 2022-01-31T22:38:28.568+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.569520+0000) 2022-01-31T22:38:28.699 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:28 smithi167 conmon[60316]: debug 2022-01-31T22:38:28.569+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.571011+0000) 2022-01-31T22:38:28.852 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[35325]: debug 2022-01-31T22:38:28.592+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225704 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:28.853 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[46715]: debug 2022-01-31T22:38:28.569+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.570737+0000) 2022-01-31T22:38:28.854 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[51620]: debug 2022-01-31T22:38:28.568+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.569279+0000) 2022-01-31T22:38:28.854 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[41853]: debug 2022-01-31T22:38:28.568+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.568984+0000) 2022-01-31T22:38:28.855 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:28 smithi171 conmon[41853]: debug 2022-01-31T22:38:28.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.646310+0000) 2022-01-31T22:38:28.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:28 smithi167 conmon[49112]: debug 2022-01-31T22:38:28.775+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.776619+0000) 2022-01-31T22:38:28.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:28 smithi167 conmon[60316]: debug 2022-01-31T22:38:28.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.702327+0000) 2022-01-31T22:38:28.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:28 smithi167 conmon[54076]: debug 2022-01-31T22:38:28.942+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:28.943191+0000) 2022-01-31T22:38:29.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:29 smithi171 conmon[46715]: debug 2022-01-31T22:38:29.443+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:29.444311+0000) 2022-01-31T22:38:29.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:29 smithi171 conmon[51620]: debug 2022-01-31T22:38:29.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:29.306331+0000) 2022-01-31T22:38:29.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:29 smithi167 conmon[49112]: debug 2022-01-31T22:38:29.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:29.776798+0000) 2022-01-31T22:38:29.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:29 smithi167 conmon[60316]: debug 2022-01-31T22:38:29.701+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:29.702438+0000) 2022-01-31T22:38:29.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:29 smithi167 conmon[54076]: debug 2022-01-31T22:38:29.942+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:29.943336+0000) 2022-01-31T22:38:30.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:29 smithi171 conmon[41853]: debug 2022-01-31T22:38:29.645+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:29.646444+0000) 2022-01-31T22:38:30.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:30 smithi171 conmon[46715]: debug 2022-01-31T22:38:30.443+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:30.444469+0000) 2022-01-31T22:38:30.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:30 smithi171 conmon[51620]: debug 2022-01-31T22:38:30.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:30.306527+0000) 2022-01-31T22:38:30.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:30 smithi167 conmon[54076]: debug 2022-01-31T22:38:30.943+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:30.943550+0000) 2022-01-31T22:38:30.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:30 smithi167 conmon[60316]: debug 2022-01-31T22:38:30.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:30.702614+0000) 2022-01-31T22:38:30.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:30 smithi167 conmon[49112]: debug 2022-01-31T22:38:30.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:30.777007+0000) 2022-01-31T22:38:31.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:30 smithi171 conmon[41853]: debug 2022-01-31T22:38:30.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:30.646599+0000) 2022-01-31T22:38:31.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:31 smithi171 conmon[46715]: debug 2022-01-31T22:38:31.443+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:31.444595+0000) 2022-01-31T22:38:31.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:31 smithi171 conmon[51620]: debug 2022-01-31T22:38:31.305+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:31.306698+0000) 2022-01-31T22:38:31.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:31 smithi167 conmon[49112]: debug 2022-01-31T22:38:31.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:31.777166+0000) 2022-01-31T22:38:31.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:31 smithi167 conmon[54076]: debug 2022-01-31T22:38:31.943+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:31.943737+0000) 2022-01-31T22:38:31.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:31 smithi167 conmon[60316]: debug 2022-01-31T22:38:31.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:31.702806+0000) 2022-01-31T22:38:32.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:31 smithi171 conmon[41853]: debug 2022-01-31T22:38:31.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:31.646796+0000) 2022-01-31T22:38:32.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:32 smithi171 conmon[46715]: debug 2022-01-31T22:38:32.444+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:32.444749+0000) 2022-01-31T22:38:32.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:32 smithi171 conmon[51620]: debug 2022-01-31T22:38:32.306+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:32.306921+0000) 2022-01-31T22:38:32.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:32 smithi167 conmon[49112]: debug 2022-01-31T22:38:32.776+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:32.777330+0000) 2022-01-31T22:38:32.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:32 smithi167 conmon[54076]: debug 2022-01-31T22:38:32.943+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:32.943891+0000) 2022-01-31T22:38:32.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:32 smithi167 conmon[60316]: debug 2022-01-31T22:38:32.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:32.703012+0000) 2022-01-31T22:38:33.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:32 smithi171 conmon[41853]: debug 2022-01-31T22:38:32.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:32.647000+0000) 2022-01-31T22:38:33.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[46715]: debug 2022-01-31T22:38:33.444+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.444958+0000) 2022-01-31T22:38:33.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[51620]: debug 2022-01-31T22:38:33.306+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.307117+0000) 2022-01-31T22:38:33.698 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:33 smithi167 conmon[54076]: debug 2022-01-31T22:38:33.594+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.595288+0000) 2022-01-31T22:38:33.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:33 smithi167 conmon[60316]: debug 2022-01-31T22:38:33.596+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.597057+0000) 2022-01-31T22:38:33.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:33 smithi167 conmon[49112]: debug 2022-01-31T22:38:33.595+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.595507+0000) 2022-01-31T22:38:33.969 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[46715]: debug 2022-01-31T22:38:33.596+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.596808+0000) 2022-01-31T22:38:33.970 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[35325]: debug 2022-01-31T22:38:33.618+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225814 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:33.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[51620]: debug 2022-01-31T22:38:33.595+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.596270+0000) 2022-01-31T22:38:33.971 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[41853]: debug 2022-01-31T22:38:33.595+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.596376+0000) 2022-01-31T22:38:33.972 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:33 smithi171 conmon[41853]: debug 2022-01-31T22:38:33.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.647190+0000) 2022-01-31T22:38:33.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:33 smithi167 conmon[49112]: debug 2022-01-31T22:38:33.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.777488+0000) 2022-01-31T22:38:33.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:33 smithi167 conmon[54076]: debug 2022-01-31T22:38:33.943+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.944097+0000) 2022-01-31T22:38:33.994 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:33 smithi167 conmon[60316]: debug 2022-01-31T22:38:33.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:33.703201+0000) 2022-01-31T22:38:34.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:34 smithi171 conmon[46715]: debug 2022-01-31T22:38:34.444+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:34.445179+0000) 2022-01-31T22:38:34.581 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:34 smithi171 conmon[51620]: debug 2022-01-31T22:38:34.306+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:34.307286+0000) 2022-01-31T22:38:34.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:34 smithi167 conmon[49112]: debug 2022-01-31T22:38:34.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:34.777580+0000) 2022-01-31T22:38:34.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:34 smithi167 conmon[54076]: debug 2022-01-31T22:38:34.943+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:34.944237+0000) 2022-01-31T22:38:34.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:34 smithi167 conmon[60316]: debug 2022-01-31T22:38:34.702+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:34.703383+0000) 2022-01-31T22:38:35.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:34 smithi171 conmon[41853]: debug 2022-01-31T22:38:34.646+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:34.647293+0000) 2022-01-31T22:38:35.420 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:35 smithi171 conmon[51620]: debug 2022-01-31T22:38:35.306+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:35.307443+0000) 2022-01-31T22:38:35.852 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:35 smithi171 conmon[46715]: debug 2022-01-31T22:38:35.444+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:35.445341+0000) 2022-01-31T22:38:35.853 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:35 smithi171 conmon[41853]: debug 2022-01-31T22:38:35.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:35.647464+0000) 2022-01-31T22:38:35.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:35 smithi167 conmon[49112]: debug 2022-01-31T22:38:35.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:35.777745+0000) 2022-01-31T22:38:35.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:35 smithi167 conmon[54076]: debug 2022-01-31T22:38:35.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:35.944446+0000) 2022-01-31T22:38:35.994 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:35 smithi167 conmon[60316]: debug 2022-01-31T22:38:35.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:35.703593+0000) 2022-01-31T22:38:36.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:36 smithi171 conmon[46715]: debug 2022-01-31T22:38:36.445+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:36.445501+0000) 2022-01-31T22:38:36.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:36 smithi171 conmon[51620]: debug 2022-01-31T22:38:36.307+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:36.307623+0000) 2022-01-31T22:38:36.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:36 smithi167 conmon[49112]: debug 2022-01-31T22:38:36.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:36.777945+0000) 2022-01-31T22:38:36.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:36 smithi167 conmon[54076]: debug 2022-01-31T22:38:36.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:36.944623+0000) 2022-01-31T22:38:36.994 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:36 smithi167 conmon[60316]: debug 2022-01-31T22:38:36.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:36.703756+0000) 2022-01-31T22:38:37.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:36 smithi171 conmon[41853]: debug 2022-01-31T22:38:36.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:36.647645+0000) 2022-01-31T22:38:37.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:37 smithi171 conmon[46715]: debug 2022-01-31T22:38:37.445+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:37.445688+0000) 2022-01-31T22:38:37.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:37 smithi171 conmon[51620]: debug 2022-01-31T22:38:37.307+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:37.307839+0000) 2022-01-31T22:38:37.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:37 smithi167 conmon[54076]: debug 2022-01-31T22:38:37.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:37.944792+0000) 2022-01-31T22:38:37.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:37 smithi167 conmon[49112]: debug 2022-01-31T22:38:37.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:37.778134+0000) 2022-01-31T22:38:37.994 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:37 smithi167 conmon[60316]: debug 2022-01-31T22:38:37.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:37.703913+0000) 2022-01-31T22:38:38.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:37 smithi171 conmon[41853]: debug 2022-01-31T22:38:37.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:37.647871+0000) 2022-01-31T22:38:38.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[46715]: debug 2022-01-31T22:38:38.445+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.445874+0000) 2022-01-31T22:38:38.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[51620]: debug 2022-01-31T22:38:38.307+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.307990+0000) 2022-01-31T22:38:38.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:38 smithi167 conmon[49112]: debug 2022-01-31T22:38:38.622+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.622783+0000) 2022-01-31T22:38:38.700 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:38 smithi167 conmon[54076]: debug 2022-01-31T22:38:38.621+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.622126+0000) 2022-01-31T22:38:38.701 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:38 smithi167 conmon[60316]: debug 2022-01-31T22:38:38.622+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.623081+0000) 2022-01-31T22:38:38.882 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[35325]: debug 2022-01-31T22:38:38.644+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 225929 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:38.882 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[46715]: debug 2022-01-31T22:38:38.622+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.622865+0000) 2022-01-31T22:38:38.883 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[51620]: debug 2022-01-31T22:38:38.621+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.622409+0000) 2022-01-31T22:38:38.884 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[41853]: debug 2022-01-31T22:38:38.621+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.622582+0000) 2022-01-31T22:38:38.884 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[41853]: debug 2022-01-31T22:38:38.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.648026+0000) 2022-01-31T22:38:38.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:38 smithi167 conmon[49112]: debug 2022-01-31T22:38:38.777+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.778313+0000) 2022-01-31T22:38:38.994 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:38 smithi167 conmon[54076]: debug 2022-01-31T22:38:38.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.944971+0000) 2022-01-31T22:38:38.994 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:38 smithi167 conmon[60316]: debug 2022-01-31T22:38:38.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:38.704118+0000) 2022-01-31T22:38:39.303 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:38 smithi171 conmon[35325]: debug 2022-01-31T22:38:38.886+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:38:39.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:39 smithi171 conmon[46715]: debug 2022-01-31T22:38:39.445+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:39.446065+0000) 2022-01-31T22:38:39.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:39 smithi171 conmon[51620]: debug 2022-01-31T22:38:39.307+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:39.308172+0000) 2022-01-31T22:38:39.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:39 smithi167 conmon[49112]: debug 2022-01-31T22:38:39.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:39.778440+0000) 2022-01-31T22:38:39.994 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:39 smithi167 conmon[54076]: debug 2022-01-31T22:38:39.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:39.945109+0000) 2022-01-31T22:38:39.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:39 smithi167 conmon[60316]: debug 2022-01-31T22:38:39.703+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:39.704230+0000) 2022-01-31T22:38:40.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:39 smithi171 conmon[41853]: debug 2022-01-31T22:38:39.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:39.648188+0000) 2022-01-31T22:38:40.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:40 smithi171 conmon[46715]: debug 2022-01-31T22:38:40.445+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:40.446233+0000) 2022-01-31T22:38:40.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:40 smithi171 conmon[51620]: debug 2022-01-31T22:38:40.308+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:40.308354+0000) 2022-01-31T22:38:40.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:40 smithi167 conmon[49112]: debug 2022-01-31T22:38:40.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:40.778618+0000) 2022-01-31T22:38:40.994 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:40 smithi167 conmon[54076]: debug 2022-01-31T22:38:40.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:40.945325+0000) 2022-01-31T22:38:40.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:40 smithi167 conmon[60316]: debug 2022-01-31T22:38:40.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:40.704404+0000) 2022-01-31T22:38:41.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:40 smithi171 conmon[41853]: debug 2022-01-31T22:38:40.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:40.648353+0000) 2022-01-31T22:38:41.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:41 smithi171 conmon[46715]: debug 2022-01-31T22:38:41.445+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:41.446422+0000) 2022-01-31T22:38:41.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:41 smithi171 conmon[51620]: debug 2022-01-31T22:38:41.307+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:41.308524+0000) 2022-01-31T22:38:41.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:41 smithi167 conmon[49112]: debug 2022-01-31T22:38:41.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:41.778814+0000) 2022-01-31T22:38:41.994 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:41 smithi167 conmon[54076]: debug 2022-01-31T22:38:41.944+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:41.945468+0000) 2022-01-31T22:38:41.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:41 smithi167 conmon[60316]: debug 2022-01-31T22:38:41.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:41.704608+0000) 2022-01-31T22:38:42.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:41 smithi171 conmon[41853]: debug 2022-01-31T22:38:41.647+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:41.648575+0000) 2022-01-31T22:38:42.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:42 smithi171 conmon[46715]: debug 2022-01-31T22:38:42.446+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:42.446580+0000) 2022-01-31T22:38:42.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:42 smithi171 conmon[51620]: debug 2022-01-31T22:38:42.308+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:42.308691+0000) 2022-01-31T22:38:42.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:42 smithi167 conmon[49112]: debug 2022-01-31T22:38:42.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:42.779053+0000) 2022-01-31T22:38:42.994 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:42 smithi167 conmon[54076]: debug 2022-01-31T22:38:42.945+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:42.945677+0000) 2022-01-31T22:38:42.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:42 smithi167 conmon[60316]: debug 2022-01-31T22:38:42.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:42.704843+0000) 2022-01-31T22:38:43.102 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:42 smithi171 conmon[41853]: debug 2022-01-31T22:38:42.648+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:42.648794+0000) 2022-01-31T22:38:43.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:43 smithi171 conmon[46715]: debug 2022-01-31T22:38:43.446+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.446847+0000) 2022-01-31T22:38:43.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:43 smithi171 conmon[51620]: debug 2022-01-31T22:38:43.308+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.308906+0000) 2022-01-31T22:38:43.699 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:43 smithi167 conmon[49112]: debug 2022-01-31T22:38:43.648+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.649054+0000) 2022-01-31T22:38:43.700 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:43 smithi167 conmon[54076]: debug 2022-01-31T22:38:43.647+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.648362+0000) 2022-01-31T22:38:43.701 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:43 smithi167 conmon[60316]: debug 2022-01-31T22:38:43.649+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.649682+0000) 2022-01-31T22:38:43.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:43 smithi167 conmon[49112]: debug 2022-01-31T22:38:43.778+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.779195+0000) 2022-01-31T22:38:43.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:43 smithi167 conmon[54076]: debug 2022-01-31T22:38:43.945+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.945819+0000) 2022-01-31T22:38:43.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:43 smithi167 conmon[60316]: debug 2022-01-31T22:38:43.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.705023+0000) 2022-01-31T22:38:44.102 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:43 smithi171 conmon[35325]: debug 2022-01-31T22:38:43.670+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226038 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:44.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:43 smithi171 conmon[41853]: debug 2022-01-31T22:38:43.649+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.650040+0000) 2022-01-31T22:38:44.104 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:43 smithi171 conmon[46715]: debug 2022-01-31T22:38:43.647+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.647775+0000) 2022-01-31T22:38:44.104 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:43 smithi171 conmon[51620]: debug 2022-01-31T22:38:43.647+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:43.648581+0000) 2022-01-31T22:38:44.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:44 smithi171 conmon[46715]: debug 2022-01-31T22:38:44.446+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:44.447028+0000) 2022-01-31T22:38:44.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:44 smithi171 conmon[51620]: debug 2022-01-31T22:38:44.308+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:44.309075+0000) 2022-01-31T22:38:44.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:44 smithi167 conmon[49112]: debug 2022-01-31T22:38:44.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:44.779350+0000) 2022-01-31T22:38:44.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:44 smithi167 conmon[54076]: debug 2022-01-31T22:38:44.945+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:44.945963+0000) 2022-01-31T22:38:44.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:44 smithi167 conmon[60316]: debug 2022-01-31T22:38:44.704+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:44.705216+0000) 2022-01-31T22:38:45.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:45 smithi171 conmon[46715]: debug 2022-01-31T22:38:45.446+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:45.447206+0000) 2022-01-31T22:38:45.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:45 smithi171 conmon[51620]: debug 2022-01-31T22:38:45.308+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:45.309229+0000) 2022-01-31T22:38:45.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:45 smithi167 conmon[49112]: debug 2022-01-31T22:38:45.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:45.779534+0000) 2022-01-31T22:38:45.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:45 smithi167 conmon[54076]: debug 2022-01-31T22:38:45.945+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:45.946178+0000) 2022-01-31T22:38:45.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:45 smithi167 conmon[60316]: debug 2022-01-31T22:38:45.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:45.705406+0000) 2022-01-31T22:38:46.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:46 smithi171 conmon[51620]: debug 2022-01-31T22:38:46.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:46.309419+0000) 2022-01-31T22:38:46.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:46 smithi171 conmon[46715]: debug 2022-01-31T22:38:46.446+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:46.447394+0000) 2022-01-31T22:38:46.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:46 smithi167 conmon[49112]: debug 2022-01-31T22:38:46.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:46.779687+0000) 2022-01-31T22:38:46.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:46 smithi167 conmon[54076]: debug 2022-01-31T22:38:46.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:46.946366+0000) 2022-01-31T22:38:46.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:46 smithi167 conmon[60316]: debug 2022-01-31T22:38:46.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:46.705578+0000) 2022-01-31T22:38:47.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:47 smithi171 conmon[46715]: debug 2022-01-31T22:38:47.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:47.447594+0000) 2022-01-31T22:38:47.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:47 smithi171 conmon[51620]: debug 2022-01-31T22:38:47.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:47.309635+0000) 2022-01-31T22:38:47.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:47 smithi167 conmon[49112]: debug 2022-01-31T22:38:47.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:47.779855+0000) 2022-01-31T22:38:47.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:47 smithi167 conmon[54076]: debug 2022-01-31T22:38:47.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:47.946547+0000) 2022-01-31T22:38:47.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:47 smithi167 conmon[60316]: debug 2022-01-31T22:38:47.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:47.705742+0000) 2022-01-31T22:38:48.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:48 smithi171 conmon[51620]: debug 2022-01-31T22:38:48.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.309819+0000) 2022-01-31T22:38:48.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:48 smithi171 conmon[46715]: debug 2022-01-31T22:38:48.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.447774+0000) 2022-01-31T22:38:48.700 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:48 smithi167 conmon[49112]: debug 2022-01-31T22:38:48.674+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.674832+0000) 2022-01-31T22:38:48.701 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:48 smithi167 conmon[54076]: debug 2022-01-31T22:38:48.674+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.674531+0000) 2022-01-31T22:38:48.702 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:48 smithi167 conmon[60316]: debug 2022-01-31T22:38:48.674+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.674895+0000) 2022-01-31T22:38:48.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:48 smithi167 conmon[49112]: debug 2022-01-31T22:38:48.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.780006+0000) 2022-01-31T22:38:48.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:48 smithi167 conmon[60316]: debug 2022-01-31T22:38:48.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.705873+0000) 2022-01-31T22:38:48.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:48 smithi167 conmon[54076]: debug 2022-01-31T22:38:48.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.946689+0000) 2022-01-31T22:38:49.102 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:48 smithi171 conmon[35325]: debug 2022-01-31T22:38:48.696+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226152 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:49.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:48 smithi171 conmon[41853]: debug 2022-01-31T22:38:48.674+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.675156+0000) 2022-01-31T22:38:49.103 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:48 smithi171 conmon[46715]: debug 2022-01-31T22:38:48.673+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.674064+0000) 2022-01-31T22:38:49.104 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:48 smithi171 conmon[51620]: debug 2022-01-31T22:38:48.673+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:48.673745+0000) 2022-01-31T22:38:49.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:49 smithi171 conmon[46715]: debug 2022-01-31T22:38:49.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:49.447950+0000) 2022-01-31T22:38:49.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:49 smithi171 conmon[51620]: debug 2022-01-31T22:38:49.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:49.310002+0000) 2022-01-31T22:38:49.994 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:49 smithi167 conmon[49112]: debug 2022-01-31T22:38:49.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:49.780177+0000) 2022-01-31T22:38:49.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:49 smithi167 conmon[60316]: debug 2022-01-31T22:38:49.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:49.705996+0000) 2022-01-31T22:38:49.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:49 smithi167 conmon[54076]: debug 2022-01-31T22:38:49.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:49.946845+0000) 2022-01-31T22:38:50.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:50 smithi171 conmon[41853]: debug 2022-01-31T22:38:50.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:50.399233+0000) 2022-01-31T22:38:50.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:50 smithi171 conmon[46715]: debug 2022-01-31T22:38:50.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:50.448088+0000) 2022-01-31T22:38:50.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:50 smithi171 conmon[51620]: debug 2022-01-31T22:38:50.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:50.310165+0000) 2022-01-31T22:38:50.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:50 smithi167 conmon[49112]: debug 2022-01-31T22:38:50.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:50.780369+0000) 2022-01-31T22:38:50.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:50 smithi167 conmon[54076]: debug 2022-01-31T22:38:50.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:50.947007+0000) 2022-01-31T22:38:50.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:50 smithi167 conmon[60316]: debug 2022-01-31T22:38:50.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:50.706200+0000) 2022-01-31T22:38:51.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:51 smithi171 conmon[41853]: debug 2022-01-31T22:38:51.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:51.399356+0000) 2022-01-31T22:38:51.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:51 smithi171 conmon[46715]: debug 2022-01-31T22:38:51.448+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:51.448277+0000) 2022-01-31T22:38:51.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:51 smithi171 conmon[51620]: debug 2022-01-31T22:38:51.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:51.310352+0000) 2022-01-31T22:38:51.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:51 smithi167 conmon[49112]: debug 2022-01-31T22:38:51.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:51.780556+0000) 2022-01-31T22:38:51.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:51 smithi167 conmon[54076]: debug 2022-01-31T22:38:51.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:51.947186+0000) 2022-01-31T22:38:51.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:51 smithi167 conmon[60316]: debug 2022-01-31T22:38:51.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:51.706387+0000) 2022-01-31T22:38:52.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:52 smithi171 conmon[41853]: debug 2022-01-31T22:38:52.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:52.399523+0000) 2022-01-31T22:38:52.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:52 smithi171 conmon[46715]: debug 2022-01-31T22:38:52.448+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:52.448483+0000) 2022-01-31T22:38:52.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:52 smithi171 conmon[51620]: debug 2022-01-31T22:38:52.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:52.310558+0000) 2022-01-31T22:38:52.995 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:52 smithi167 conmon[60316]: debug 2022-01-31T22:38:52.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:52.706587+0000) 2022-01-31T22:38:52.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:52 smithi167 conmon[49112]: debug 2022-01-31T22:38:52.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:52.780746+0000) 2022-01-31T22:38:52.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:52 smithi167 conmon[54076]: debug 2022-01-31T22:38:52.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:52.947387+0000) 2022-01-31T22:38:53.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[41853]: debug 2022-01-31T22:38:53.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.399743+0000) 2022-01-31T22:38:53.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[46715]: debug 2022-01-31T22:38:53.448+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.448671+0000) 2022-01-31T22:38:53.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[51620]: debug 2022-01-31T22:38:53.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.310719+0000) 2022-01-31T22:38:53.882 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[35325]: debug 2022-01-31T22:38:53.722+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226261 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:53.883 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[41853]: debug 2022-01-31T22:38:53.700+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.700719+0000) 2022-01-31T22:38:53.884 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[46715]: debug 2022-01-31T22:38:53.698+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.699365+0000) 2022-01-31T22:38:53.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[51620]: debug 2022-01-31T22:38:53.699+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.699782+0000) 2022-01-31T22:38:53.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:53 smithi167 conmon[54076]: debug 2022-01-31T22:38:53.700+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.700757+0000) 2022-01-31T22:38:53.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:53 smithi167 conmon[54076]: debug 2022-01-31T22:38:53.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.947582+0000) 2022-01-31T22:38:53.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:53 smithi167 conmon[49112]: debug 2022-01-31T22:38:53.700+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.700999+0000) 2022-01-31T22:38:53.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:53 smithi167 conmon[49112]: debug 2022-01-31T22:38:53.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.780924+0000) 2022-01-31T22:38:53.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:53 smithi167 conmon[60316]: debug 2022-01-31T22:38:53.699+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.699696+0000) 2022-01-31T22:38:53.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:53 smithi167 conmon[60316]: debug 2022-01-31T22:38:53.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:53.706704+0000) 2022-01-31T22:38:54.305 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:53 smithi171 conmon[35325]: debug 2022-01-31T22:38:53.887+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:38:54.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:54 smithi171 conmon[41853]: debug 2022-01-31T22:38:54.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:54.399919+0000) 2022-01-31T22:38:54.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:54 smithi171 conmon[46715]: debug 2022-01-31T22:38:54.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:54.448865+0000) 2022-01-31T22:38:54.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:54 smithi171 conmon[51620]: debug 2022-01-31T22:38:54.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:54.310886+0000) 2022-01-31T22:38:54.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:54 smithi167 conmon[49112]: debug 2022-01-31T22:38:54.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:54.781068+0000) 2022-01-31T22:38:54.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:54 smithi167 conmon[54076]: debug 2022-01-31T22:38:54.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:54.947745+0000) 2022-01-31T22:38:54.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:54 smithi167 conmon[60316]: debug 2022-01-31T22:38:54.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:54.706843+0000) 2022-01-31T22:38:55.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:55 smithi171 conmon[41853]: debug 2022-01-31T22:38:55.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:55.400089+0000) 2022-01-31T22:38:55.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:55 smithi171 conmon[46715]: debug 2022-01-31T22:38:55.448+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:55.449041+0000) 2022-01-31T22:38:55.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:55 smithi171 conmon[51620]: debug 2022-01-31T22:38:55.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:55.311027+0000) 2022-01-31T22:38:55.995 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:55 smithi167 conmon[54076]: debug 2022-01-31T22:38:55.946+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:55.947922+0000) 2022-01-31T22:38:55.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:55 smithi167 conmon[60316]: debug 2022-01-31T22:38:55.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:55.707011+0000) 2022-01-31T22:38:55.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:55 smithi167 conmon[49112]: debug 2022-01-31T22:38:55.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:55.781238+0000) 2022-01-31T22:38:56.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:56 smithi171 conmon[41853]: debug 2022-01-31T22:38:56.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:56.400226+0000) 2022-01-31T22:38:56.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:56 smithi171 conmon[46715]: debug 2022-01-31T22:38:56.448+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:56.449195+0000) 2022-01-31T22:38:56.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:56 smithi171 conmon[51620]: debug 2022-01-31T22:38:56.309+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:56.311239+0000) 2022-01-31T22:38:56.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:56 smithi167 conmon[54076]: debug 2022-01-31T22:38:56.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:56.948114+0000) 2022-01-31T22:38:56.996 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:56 smithi167 conmon[60316]: debug 2022-01-31T22:38:56.705+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:56.707209+0000) 2022-01-31T22:38:56.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:56 smithi167 conmon[49112]: debug 2022-01-31T22:38:56.779+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:56.781426+0000) 2022-01-31T22:38:57.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:57 smithi171 conmon[41853]: debug 2022-01-31T22:38:57.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:57.400397+0000) 2022-01-31T22:38:57.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:57 smithi171 conmon[46715]: debug 2022-01-31T22:38:57.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:57.449394+0000) 2022-01-31T22:38:57.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:57 smithi171 conmon[51620]: debug 2022-01-31T22:38:57.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:57.311456+0000) 2022-01-31T22:38:57.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:57 smithi167 conmon[54076]: debug 2022-01-31T22:38:57.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:57.948300+0000) 2022-01-31T22:38:57.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:57 smithi167 conmon[60316]: debug 2022-01-31T22:38:57.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:57.707384+0000) 2022-01-31T22:38:57.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:57 smithi167 conmon[49112]: debug 2022-01-31T22:38:57.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:57.781567+0000) 2022-01-31T22:38:58.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[41853]: debug 2022-01-31T22:38:58.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.400601+0000) 2022-01-31T22:38:58.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[46715]: debug 2022-01-31T22:38:58.447+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.449599+0000) 2022-01-31T22:38:58.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[51620]: debug 2022-01-31T22:38:58.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.311668+0000) 2022-01-31T22:38:58.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:58 smithi167 conmon[49112]: debug 2022-01-31T22:38:58.724+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.726452+0000) 2022-01-31T22:38:58.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:58 smithi167 conmon[49112]: debug 2022-01-31T22:38:58.780+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.781781+0000) 2022-01-31T22:38:58.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:58 smithi167 conmon[60316]: debug 2022-01-31T22:38:58.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.707549+0000) 2022-01-31T22:38:58.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:58 smithi167 conmon[60316]: debug 2022-01-31T22:38:58.725+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.727274+0000) 2022-01-31T22:38:58.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:58 smithi167 conmon[54076]: debug 2022-01-31T22:38:58.724+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.726267+0000) 2022-01-31T22:38:58.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:58 smithi167 conmon[54076]: debug 2022-01-31T22:38:58.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.948491+0000) 2022-01-31T22:38:59.102 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[46715]: debug 2022-01-31T22:38:58.723+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.725703+0000) 2022-01-31T22:38:59.103 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[35325]: debug 2022-01-31T22:38:58.748+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226375 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:38:59.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[41853]: debug 2022-01-31T22:38:58.725+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.727099+0000) 2022-01-31T22:38:59.105 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:58 smithi171 conmon[51620]: debug 2022-01-31T22:38:58.724+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:58.725784+0000) 2022-01-31T22:38:59.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:38:59 smithi171 conmon[41853]: debug 2022-01-31T22:38:59.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:59.400811+0000) 2022-01-31T22:38:59.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:38:59 smithi171 conmon[46715]: debug 2022-01-31T22:38:59.448+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:59.449768+0000) 2022-01-31T22:38:59.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:38:59 smithi171 conmon[51620]: debug 2022-01-31T22:38:59.310+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:59.311852+0000) 2022-01-31T22:38:59.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:38:59 smithi167 conmon[49112]: debug 2022-01-31T22:38:59.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:59.781943+0000) 2022-01-31T22:38:59.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:38:59 smithi167 conmon[54076]: debug 2022-01-31T22:38:59.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:59.948659+0000) 2022-01-31T22:38:59.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:38:59 smithi167 conmon[60316]: debug 2022-01-31T22:38:59.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:38:59.707789+0000) 2022-01-31T22:39:00.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:00 smithi171 conmon[41853]: debug 2022-01-31T22:39:00.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:00.400944+0000) 2022-01-31T22:39:00.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:00 smithi171 conmon[46715]: debug 2022-01-31T22:39:00.449+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:00.449906+0000) 2022-01-31T22:39:00.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:00 smithi171 conmon[51620]: debug 2022-01-31T22:39:00.311+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:00.312033+0000) 2022-01-31T22:39:00.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:00 smithi167 conmon[49112]: debug 2022-01-31T22:39:00.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:00.782101+0000) 2022-01-31T22:39:00.997 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:00 smithi167 conmon[54076]: debug 2022-01-31T22:39:00.947+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:00.948801+0000) 2022-01-31T22:39:00.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:00 smithi167 conmon[60316]: debug 2022-01-31T22:39:00.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:00.707943+0000) 2022-01-31T22:39:01.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:01 smithi171 conmon[41853]: debug 2022-01-31T22:39:01.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:01.401073+0000) 2022-01-31T22:39:01.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:01 smithi171 conmon[46715]: debug 2022-01-31T22:39:01.449+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:01.450092+0000) 2022-01-31T22:39:01.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:01 smithi171 conmon[51620]: debug 2022-01-31T22:39:01.311+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:01.312254+0000) 2022-01-31T22:39:01.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:01 smithi167 conmon[49112]: debug 2022-01-31T22:39:01.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:01.782321+0000) 2022-01-31T22:39:01.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:01 smithi167 conmon[54076]: debug 2022-01-31T22:39:01.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:01.948987+0000) 2022-01-31T22:39:01.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:01 smithi167 conmon[60316]: debug 2022-01-31T22:39:01.706+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:01.708135+0000) 2022-01-31T22:39:02.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:02 smithi171 conmon[41853]: debug 2022-01-31T22:39:02.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:02.401216+0000) 2022-01-31T22:39:02.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:02 smithi171 conmon[46715]: debug 2022-01-31T22:39:02.449+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:02.450273+0000) 2022-01-31T22:39:02.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:02 smithi171 conmon[51620]: debug 2022-01-31T22:39:02.311+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:02.312486+0000) 2022-01-31T22:39:02.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:02 smithi167 conmon[49112]: debug 2022-01-31T22:39:02.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:02.782519+0000) 2022-01-31T22:39:02.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:02 smithi167 conmon[54076]: debug 2022-01-31T22:39:02.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:02.949150+0000) 2022-01-31T22:39:02.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:02 smithi167 conmon[60316]: debug 2022-01-31T22:39:02.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:02.708346+0000) 2022-01-31T22:39:03.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[41853]: debug 2022-01-31T22:39:03.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.401411+0000) 2022-01-31T22:39:03.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[46715]: debug 2022-01-31T22:39:03.449+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.450461+0000) 2022-01-31T22:39:03.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[51620]: debug 2022-01-31T22:39:03.311+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.312704+0000) 2022-01-31T22:39:03.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:03 smithi167 conmon[49112]: debug 2022-01-31T22:39:03.750+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.751954+0000) 2022-01-31T22:39:03.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:03 smithi167 conmon[49112]: debug 2022-01-31T22:39:03.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.782665+0000) 2022-01-31T22:39:03.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:03 smithi167 conmon[54076]: debug 2022-01-31T22:39:03.750+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.752303+0000) 2022-01-31T22:39:03.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:03 smithi167 conmon[54076]: debug 2022-01-31T22:39:03.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.949316+0000) 2022-01-31T22:39:03.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:03 smithi167 conmon[60316]: debug 2022-01-31T22:39:03.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.708533+0000) 2022-01-31T22:39:04.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:03 smithi167 conmon[60316]: debug 2022-01-31T22:39:03.752+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.753774+0000) 2022-01-31T22:39:04.102 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[35325]: debug 2022-01-31T22:39:03.774+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226484 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:04.104 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[41853]: debug 2022-01-31T22:39:03.750+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.752446+0000) 2022-01-31T22:39:04.105 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[46715]: debug 2022-01-31T22:39:03.751+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.753492+0000) 2022-01-31T22:39:04.106 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:03 smithi171 conmon[51620]: debug 2022-01-31T22:39:03.751+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:03.753380+0000) 2022-01-31T22:39:04.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:04 smithi171 conmon[51620]: debug 2022-01-31T22:39:04.312+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:04.312852+0000) 2022-01-31T22:39:04.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:04 smithi171 conmon[41853]: debug 2022-01-31T22:39:04.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:04.401596+0000) 2022-01-31T22:39:04.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:04 smithi171 conmon[46715]: debug 2022-01-31T22:39:04.449+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:04.450662+0000) 2022-01-31T22:39:04.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:04 smithi167 conmon[49112]: debug 2022-01-31T22:39:04.781+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:04.782822+0000) 2022-01-31T22:39:04.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:04 smithi167 conmon[54076]: debug 2022-01-31T22:39:04.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:04.949469+0000) 2022-01-31T22:39:04.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:04 smithi167 conmon[60316]: debug 2022-01-31T22:39:04.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:04.708678+0000) 2022-01-31T22:39:05.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:05 smithi171 conmon[51620]: debug 2022-01-31T22:39:05.312+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:05.313012+0000) 2022-01-31T22:39:05.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:05 smithi171 conmon[41853]: debug 2022-01-31T22:39:05.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:05.401759+0000) 2022-01-31T22:39:05.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:05 smithi171 conmon[46715]: debug 2022-01-31T22:39:05.450+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:05.450806+0000) 2022-01-31T22:39:05.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:05 smithi167 conmon[49112]: debug 2022-01-31T22:39:05.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:05.783011+0000) 2022-01-31T22:39:05.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:05 smithi167 conmon[54076]: debug 2022-01-31T22:39:05.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:05.949669+0000) 2022-01-31T22:39:05.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:05 smithi167 conmon[60316]: debug 2022-01-31T22:39:05.707+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:05.708808+0000) 2022-01-31T22:39:06.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:06 smithi171 conmon[41853]: debug 2022-01-31T22:39:06.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:06.401948+0000) 2022-01-31T22:39:06.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:06 smithi171 conmon[46715]: debug 2022-01-31T22:39:06.450+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:06.450976+0000) 2022-01-31T22:39:06.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:06 smithi171 conmon[51620]: debug 2022-01-31T22:39:06.312+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:06.313253+0000) 2022-01-31T22:39:06.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:06 smithi167 conmon[60316]: debug 2022-01-31T22:39:06.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:06.709026+0000) 2022-01-31T22:39:06.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:06 smithi167 conmon[49112]: debug 2022-01-31T22:39:06.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:06.783202+0000) 2022-01-31T22:39:07.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:06 smithi167 conmon[54076]: debug 2022-01-31T22:39:06.949+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:06.949888+0000) 2022-01-31T22:39:07.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:07 smithi171 conmon[41853]: debug 2022-01-31T22:39:07.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:07.402111+0000) 2022-01-31T22:39:07.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:07 smithi171 conmon[51620]: debug 2022-01-31T22:39:07.312+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:07.313474+0000) 2022-01-31T22:39:07.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:07 smithi171 conmon[46715]: debug 2022-01-31T22:39:07.450+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:07.451089+0000) 2022-01-31T22:39:07.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:07 smithi167 conmon[49112]: debug 2022-01-31T22:39:07.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:07.783408+0000) 2022-01-31T22:39:07.998 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:07 smithi167 conmon[60316]: debug 2022-01-31T22:39:07.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:07.709221+0000) 2022-01-31T22:39:07.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:07 smithi167 conmon[54076]: debug 2022-01-31T22:39:07.949+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:07.950075+0000) 2022-01-31T22:39:08.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[41853]: debug 2022-01-31T22:39:08.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.402325+0000) 2022-01-31T22:39:08.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[46715]: debug 2022-01-31T22:39:08.450+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.451270+0000) 2022-01-31T22:39:08.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[51620]: debug 2022-01-31T22:39:08.312+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.313638+0000) 2022-01-31T22:39:08.882 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[35325]: debug 2022-01-31T22:39:08.816+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:08.883 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[41853]: debug 2022-01-31T22:39:08.778+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.779734+0000) 2022-01-31T22:39:08.883 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[46715]: debug 2022-01-31T22:39:08.778+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.779152+0000) 2022-01-31T22:39:08.884 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[51620]: debug 2022-01-31T22:39:08.779+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.779774+0000) 2022-01-31T22:39:08.997 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:08 smithi167 conmon[49112]: debug 2022-01-31T22:39:08.777+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.779380+0000) 2022-01-31T22:39:08.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:08 smithi167 conmon[49112]: debug 2022-01-31T22:39:08.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.783551+0000) 2022-01-31T22:39:08.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:08 smithi167 conmon[54076]: debug 2022-01-31T22:39:08.777+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.778577+0000) 2022-01-31T22:39:08.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:08 smithi167 conmon[54076]: debug 2022-01-31T22:39:08.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.950275+0000) 2022-01-31T22:39:09.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:08 smithi167 conmon[60316]: debug 2022-01-31T22:39:08.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.709409+0000) 2022-01-31T22:39:09.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:08 smithi167 conmon[60316]: debug 2022-01-31T22:39:08.778+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:08.779818+0000) 2022-01-31T22:39:09.308 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:08 smithi171 conmon[35325]: debug 2022-01-31T22:39:08.887+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:39:09.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:09 smithi171 conmon[41853]: debug 2022-01-31T22:39:09.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:09.402471+0000) 2022-01-31T22:39:09.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:09 smithi171 conmon[51620]: debug 2022-01-31T22:39:09.313+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:09.313856+0000) 2022-01-31T22:39:09.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:09 smithi171 conmon[46715]: debug 2022-01-31T22:39:09.450+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:09.451459+0000) 2022-01-31T22:39:09.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:09 smithi167 conmon[49112]: debug 2022-01-31T22:39:09.782+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:09.783675+0000) 2022-01-31T22:39:09.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:09 smithi167 conmon[54076]: debug 2022-01-31T22:39:09.948+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:09.950439+0000) 2022-01-31T22:39:09.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:09 smithi167 conmon[60316]: debug 2022-01-31T22:39:09.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:09.709601+0000) 2022-01-31T22:39:10.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:10 smithi171 conmon[41853]: debug 2022-01-31T22:39:10.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:10.402597+0000) 2022-01-31T22:39:10.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:10 smithi171 conmon[46715]: debug 2022-01-31T22:39:10.450+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:10.451630+0000) 2022-01-31T22:39:10.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:10 smithi171 conmon[51620]: debug 2022-01-31T22:39:10.313+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:10.313999+0000) 2022-01-31T22:39:10.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:10 smithi167 conmon[49112]: debug 2022-01-31T22:39:10.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:10.783861+0000) 2022-01-31T22:39:10.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:10 smithi167 conmon[60316]: debug 2022-01-31T22:39:10.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:10.709702+0000) 2022-01-31T22:39:11.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:10 smithi167 conmon[54076]: debug 2022-01-31T22:39:10.949+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:10.950645+0000) 2022-01-31T22:39:11.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:11 smithi171 conmon[41853]: debug 2022-01-31T22:39:11.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:11.402772+0000) 2022-01-31T22:39:11.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:11 smithi171 conmon[46715]: debug 2022-01-31T22:39:11.451+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:11.451821+0000) 2022-01-31T22:39:11.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:11 smithi171 conmon[51620]: debug 2022-01-31T22:39:11.313+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:11.314201+0000) 2022-01-31T22:39:11.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:11 smithi167 conmon[49112]: debug 2022-01-31T22:39:11.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:11.784087+0000) 2022-01-31T22:39:11.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:11 smithi167 conmon[54076]: debug 2022-01-31T22:39:11.950+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:11.950827+0000) 2022-01-31T22:39:11.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:11 smithi167 conmon[60316]: debug 2022-01-31T22:39:11.709+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:11.709873+0000) 2022-01-31T22:39:12.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:12 smithi171 conmon[41853]: debug 2022-01-31T22:39:12.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:12.402935+0000) 2022-01-31T22:39:12.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:12 smithi171 conmon[46715]: debug 2022-01-31T22:39:12.451+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:12.451999+0000) 2022-01-31T22:39:12.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:12 smithi171 conmon[51620]: debug 2022-01-31T22:39:12.313+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:12.314407+0000) 2022-01-31T22:39:12.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:12 smithi167 conmon[49112]: debug 2022-01-31T22:39:12.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:12.784264+0000) 2022-01-31T22:39:12.999 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:12 smithi167 conmon[54076]: debug 2022-01-31T22:39:12.950+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:12.950991+0000) 2022-01-31T22:39:13.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:12 smithi167 conmon[60316]: debug 2022-01-31T22:39:12.709+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:12.710004+0000) 2022-01-31T22:39:13.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[41853]: debug 2022-01-31T22:39:13.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.403080+0000) 2022-01-31T22:39:13.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[46715]: debug 2022-01-31T22:39:13.451+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.452185+0000) 2022-01-31T22:39:13.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[51620]: debug 2022-01-31T22:39:13.313+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.314634+0000) 2022-01-31T22:39:13.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:13 smithi167 conmon[49112]: debug 2022-01-31T22:39:13.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.784421+0000) 2022-01-31T22:39:13.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:13 smithi167 conmon[49112]: debug 2022-01-31T22:39:13.819+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.820777+0000) 2022-01-31T22:39:14.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:13 smithi167 conmon[54076]: debug 2022-01-31T22:39:13.817+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.819483+0000) 2022-01-31T22:39:14.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:13 smithi167 conmon[54076]: debug 2022-01-31T22:39:13.950+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.951176+0000) 2022-01-31T22:39:14.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:13 smithi167 conmon[60316]: debug 2022-01-31T22:39:13.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.710147+0000) 2022-01-31T22:39:14.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:13 smithi167 conmon[60316]: debug 2022-01-31T22:39:13.818+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.820114+0000) 2022-01-31T22:39:14.102 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[35325]: debug 2022-01-31T22:39:13.842+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226708 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:14.103 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[41853]: debug 2022-01-31T22:39:13.819+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.819945+0000) 2022-01-31T22:39:14.104 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[46715]: debug 2022-01-31T22:39:13.820+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.821138+0000) 2022-01-31T22:39:14.104 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:13 smithi171 conmon[51620]: debug 2022-01-31T22:39:13.819+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:13.820420+0000) 2022-01-31T22:39:14.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:14 smithi171 conmon[41853]: debug 2022-01-31T22:39:14.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:14.403287+0000) 2022-01-31T22:39:14.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:14 smithi171 conmon[46715]: debug 2022-01-31T22:39:14.451+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:14.452359+0000) 2022-01-31T22:39:14.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:14 smithi171 conmon[51620]: debug 2022-01-31T22:39:14.314+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:14.314778+0000) 2022-01-31T22:39:14.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:14 smithi167 conmon[49112]: debug 2022-01-31T22:39:14.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:14.784605+0000) 2022-01-31T22:39:15.118 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:14 smithi167 conmon[60316]: debug 2022-01-31T22:39:14.708+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:14.710356+0000) 2022-01-31T22:39:15.252 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:14 smithi167 conmon[54076]: debug 2022-01-31T22:39:14.949+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:14.951314+0000) 2022-01-31T22:39:15.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:15 smithi171 conmon[41853]: debug 2022-01-31T22:39:15.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:15.403424+0000) 2022-01-31T22:39:15.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:15 smithi171 conmon[46715]: debug 2022-01-31T22:39:15.451+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:15.452496+0000) 2022-01-31T22:39:15.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:15 smithi171 conmon[51620]: debug 2022-01-31T22:39:15.314+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:15.314954+0000) 2022-01-31T22:39:15.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:15 smithi167 conmon[49112]: debug 2022-01-31T22:39:15.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:15.784763+0000) 2022-01-31T22:39:15.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:15 smithi167 conmon[60316]: debug 2022-01-31T22:39:15.709+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:15.710538+0000) 2022-01-31T22:39:16.256 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:15 smithi167 conmon[54076]: debug 2022-01-31T22:39:15.950+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:15.951475+0000) 2022-01-31T22:39:16.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:16 smithi171 conmon[41853]: debug 2022-01-31T22:39:16.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:16.403607+0000) 2022-01-31T22:39:16.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:16 smithi171 conmon[46715]: debug 2022-01-31T22:39:16.452+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:16.452714+0000) 2022-01-31T22:39:16.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:16 smithi171 conmon[51620]: debug 2022-01-31T22:39:16.314+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:16.315165+0000) 2022-01-31T22:39:16.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:16 smithi167 conmon[49112]: debug 2022-01-31T22:39:16.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:16.784950+0000) 2022-01-31T22:39:16.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:16 smithi167 conmon[60316]: debug 2022-01-31T22:39:16.709+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:16.710696+0000) 2022-01-31T22:39:17.260 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:16 smithi167 conmon[54076]: debug 2022-01-31T22:39:16.951+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:16.951692+0000) 2022-01-31T22:39:17.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:17 smithi171 conmon[46715]: debug 2022-01-31T22:39:17.452+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:17.452913+0000) 2022-01-31T22:39:17.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:17 smithi171 conmon[41853]: debug 2022-01-31T22:39:17.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:17.403807+0000) 2022-01-31T22:39:17.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:17 smithi171 conmon[51620]: debug 2022-01-31T22:39:17.314+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:17.315391+0000) 2022-01-31T22:39:17.914 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:17 smithi167 conmon[49112]: debug 2022-01-31T22:39:17.783+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:17.785170+0000) 2022-01-31T22:39:17.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:17 smithi167 conmon[60316]: debug 2022-01-31T22:39:17.710+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:17.710877+0000) 2022-01-31T22:39:18.264 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:17 smithi167 conmon[54076]: debug 2022-01-31T22:39:17.951+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:17.951869+0000) 2022-01-31T22:39:18.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[46715]: debug 2022-01-31T22:39:18.452+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.453107+0000) 2022-01-31T22:39:18.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[41853]: debug 2022-01-31T22:39:18.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.403974+0000) 2022-01-31T22:39:18.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[51620]: debug 2022-01-31T22:39:18.314+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.315567+0000) 2022-01-31T22:39:18.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:18 smithi167 conmon[49112]: debug 2022-01-31T22:39:18.784+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.785325+0000) 2022-01-31T22:39:18.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:18 smithi167 conmon[60316]: debug 2022-01-31T22:39:18.710+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.711059+0000) 2022-01-31T22:39:18.864 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[41853]: debug 2022-01-31T22:39:18.845+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.846205+0000) 2022-01-31T22:39:18.865 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[46715]: debug 2022-01-31T22:39:18.846+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.846827+0000) 2022-01-31T22:39:18.866 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[51620]: debug 2022-01-31T22:39:18.845+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.846330+0000) 2022-01-31T22:39:19.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:18 smithi167 conmon[49112]: debug 2022-01-31T22:39:18.845+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.847293+0000) 2022-01-31T22:39:19.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:18 smithi167 conmon[60316]: debug 2022-01-31T22:39:18.844+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.845624+0000) 2022-01-31T22:39:19.153 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:18 smithi167 conmon[54076]: debug 2022-01-31T22:39:18.845+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.847132+0000) 2022-01-31T22:39:19.153 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:18 smithi167 conmon[54076]: debug 2022-01-31T22:39:18.951+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:18.952089+0000) 2022-01-31T22:39:19.310 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:18 smithi171 conmon[35325]: debug 2022-01-31T22:39:18.868+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226818 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:19.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:19 smithi171 conmon[41853]: debug 2022-01-31T22:39:19.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:19.404125+0000) 2022-01-31T22:39:19.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:19 smithi171 conmon[46715]: debug 2022-01-31T22:39:19.452+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:19.453287+0000) 2022-01-31T22:39:19.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:19 smithi171 conmon[51620]: debug 2022-01-31T22:39:19.315+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:19.315774+0000) 2022-01-31T22:39:19.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:19 smithi167 conmon[49112]: debug 2022-01-31T22:39:19.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:19.785533+0000) 2022-01-31T22:39:20.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:19 smithi167 conmon[54076]: debug 2022-01-31T22:39:19.951+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:19.952204+0000) 2022-01-31T22:39:20.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:19 smithi167 conmon[60316]: debug 2022-01-31T22:39:19.710+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:19.711238+0000) 2022-01-31T22:39:20.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:20 smithi171 conmon[41853]: debug 2022-01-31T22:39:20.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:20.404220+0000) 2022-01-31T22:39:20.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:20 smithi171 conmon[46715]: debug 2022-01-31T22:39:20.452+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:20.453399+0000) 2022-01-31T22:39:20.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:20 smithi171 conmon[51620]: debug 2022-01-31T22:39:20.315+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:20.315924+0000) 2022-01-31T22:39:20.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:20 smithi167 conmon[49112]: debug 2022-01-31T22:39:20.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:20.785684+0000) 2022-01-31T22:39:20.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:20 smithi167 conmon[60316]: debug 2022-01-31T22:39:20.710+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:20.711450+0000) 2022-01-31T22:39:21.272 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:20 smithi167 conmon[54076]: debug 2022-01-31T22:39:20.951+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:20.952360+0000) 2022-01-31T22:39:21.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:21 smithi171 conmon[41853]: debug 2022-01-31T22:39:21.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:21.404365+0000) 2022-01-31T22:39:21.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:21 smithi171 conmon[46715]: debug 2022-01-31T22:39:21.453+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:21.453552+0000) 2022-01-31T22:39:21.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:21 smithi171 conmon[51620]: debug 2022-01-31T22:39:21.315+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:21.316088+0000) 2022-01-31T22:39:21.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:21 smithi167 conmon[49112]: debug 2022-01-31T22:39:21.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:21.785874+0000) 2022-01-31T22:39:21.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:21 smithi167 conmon[60316]: debug 2022-01-31T22:39:21.711+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:21.711623+0000) 2022-01-31T22:39:22.276 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:21 smithi167 conmon[54076]: debug 2022-01-31T22:39:21.952+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:21.952538+0000) 2022-01-31T22:39:22.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:22 smithi171 conmon[41853]: debug 2022-01-31T22:39:22.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:22.404551+0000) 2022-01-31T22:39:22.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:22 smithi171 conmon[46715]: debug 2022-01-31T22:39:22.453+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:22.453736+0000) 2022-01-31T22:39:22.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:22 smithi171 conmon[51620]: debug 2022-01-31T22:39:22.315+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:22.316274+0000) 2022-01-31T22:39:22.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:22 smithi167 conmon[49112]: debug 2022-01-31T22:39:22.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:22.786059+0000) 2022-01-31T22:39:22.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:22 smithi167 conmon[60316]: debug 2022-01-31T22:39:22.711+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:22.711745+0000) 2022-01-31T22:39:23.280 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:22 smithi167 conmon[54076]: debug 2022-01-31T22:39:22.952+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:22.952695+0000) 2022-01-31T22:39:23.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[41853]: debug 2022-01-31T22:39:23.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.404727+0000) 2022-01-31T22:39:23.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[46715]: debug 2022-01-31T22:39:23.453+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.453930+0000) 2022-01-31T22:39:23.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[51620]: debug 2022-01-31T22:39:23.316+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.316493+0000) 2022-01-31T22:39:23.865 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:23 smithi167 conmon[49112]: debug 2022-01-31T22:39:23.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.786278+0000) 2022-01-31T22:39:23.866 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:23 smithi167 conmon[60316]: debug 2022-01-31T22:39:23.711+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.711933+0000) 2022-01-31T22:39:24.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:23 smithi167 conmon[49112]: debug 2022-01-31T22:39:23.872+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.873247+0000) 2022-01-31T22:39:24.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:23 smithi167 conmon[54076]: debug 2022-01-31T22:39:23.872+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.873112+0000) 2022-01-31T22:39:24.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:23 smithi167 conmon[54076]: debug 2022-01-31T22:39:23.952+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.952835+0000) 2022-01-31T22:39:24.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:23 smithi167 conmon[60316]: debug 2022-01-31T22:39:23.872+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.872527+0000) 2022-01-31T22:39:24.310 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[35325]: debug 2022-01-31T22:39:23.888+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:39:24.311 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[35325]: debug 2022-01-31T22:39:23.894+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 226927 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:24.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[41853]: debug 2022-01-31T22:39:23.872+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.872808+0000) 2022-01-31T22:39:24.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[46715]: debug 2022-01-31T22:39:23.872+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.873472+0000) 2022-01-31T22:39:24.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:23 smithi171 conmon[51620]: debug 2022-01-31T22:39:23.871+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:23.872308+0000) 2022-01-31T22:39:24.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:24 smithi171 conmon[51620]: debug 2022-01-31T22:39:24.316+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:24.316665+0000) 2022-01-31T22:39:24.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:24 smithi171 conmon[46715]: debug 2022-01-31T22:39:24.453+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:24.454133+0000) 2022-01-31T22:39:24.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:24 smithi171 conmon[41853]: debug 2022-01-31T22:39:24.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:24.404928+0000) 2022-01-31T22:39:25.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:24 smithi167 conmon[49112]: debug 2022-01-31T22:39:24.785+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:24.786425+0000) 2022-01-31T22:39:25.001 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:24 smithi167 conmon[54076]: debug 2022-01-31T22:39:24.952+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:24.952982+0000) 2022-01-31T22:39:25.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:24 smithi167 conmon[60316]: debug 2022-01-31T22:39:24.711+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:24.712123+0000) 2022-01-31T22:39:25.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:25 smithi171 conmon[46715]: debug 2022-01-31T22:39:25.453+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:25.454297+0000) 2022-01-31T22:39:25.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:25 smithi171 conmon[51620]: debug 2022-01-31T22:39:25.316+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:25.316809+0000) 2022-01-31T22:39:25.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:25 smithi171 conmon[41853]: debug 2022-01-31T22:39:25.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:25.405079+0000) 2022-01-31T22:39:25.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:25 smithi167 conmon[49112]: debug 2022-01-31T22:39:25.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:25.786579+0000) 2022-01-31T22:39:25.914 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:25 smithi167 conmon[60316]: debug 2022-01-31T22:39:25.711+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:25.712265+0000) 2022-01-31T22:39:26.289 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:25 smithi167 conmon[54076]: debug 2022-01-31T22:39:25.952+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:25.953137+0000) 2022-01-31T22:39:26.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:26 smithi171 conmon[46715]: debug 2022-01-31T22:39:26.454+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:26.454451+0000) 2022-01-31T22:39:26.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:26 smithi171 conmon[51620]: debug 2022-01-31T22:39:26.316+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:26.317025+0000) 2022-01-31T22:39:26.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:26 smithi171 conmon[41853]: debug 2022-01-31T22:39:26.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:26.405232+0000) 2022-01-31T22:39:26.913 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:26 smithi167 conmon[49112]: debug 2022-01-31T22:39:26.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:26.786767+0000) 2022-01-31T22:39:26.915 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:26 smithi167 conmon[60316]: debug 2022-01-31T22:39:26.712+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:26.712455+0000) 2022-01-31T22:39:27.294 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:26 smithi167 conmon[54076]: debug 2022-01-31T22:39:26.952+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:26.953315+0000) 2022-01-31T22:39:27.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:27 smithi171 conmon[46715]: debug 2022-01-31T22:39:27.453+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:27.454651+0000) 2022-01-31T22:39:27.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:27 smithi171 conmon[51620]: debug 2022-01-31T22:39:27.316+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:27.317222+0000) 2022-01-31T22:39:27.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:27 smithi171 conmon[41853]: debug 2022-01-31T22:39:27.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:27.405367+0000) 2022-01-31T22:39:28.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:27 smithi167 conmon[49112]: debug 2022-01-31T22:39:27.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:27.786956+0000) 2022-01-31T22:39:28.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:27 smithi167 conmon[54076]: debug 2022-01-31T22:39:27.953+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:27.953467+0000) 2022-01-31T22:39:28.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:27 smithi167 conmon[60316]: debug 2022-01-31T22:39:27.712+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:27.712655+0000) 2022-01-31T22:39:28.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[51620]: debug 2022-01-31T22:39:28.316+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.317419+0000) 2022-01-31T22:39:28.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[41853]: debug 2022-01-31T22:39:28.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.405520+0000) 2022-01-31T22:39:28.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[46715]: debug 2022-01-31T22:39:28.454+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.454832+0000) 2022-01-31T22:39:29.001 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:28 smithi167 conmon[54076]: debug 2022-01-31T22:39:28.897+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.898191+0000) 2022-01-31T22:39:29.001 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:28 smithi167 conmon[54076]: debug 2022-01-31T22:39:28.953+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.953624+0000) 2022-01-31T22:39:29.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:28 smithi167 conmon[60316]: debug 2022-01-31T22:39:28.712+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.712863+0000) 2022-01-31T22:39:29.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:28 smithi167 conmon[60316]: debug 2022-01-31T22:39:28.897+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.898392+0000) 2022-01-31T22:39:29.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:28 smithi167 conmon[49112]: debug 2022-01-31T22:39:28.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.787142+0000) 2022-01-31T22:39:29.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:28 smithi167 conmon[49112]: debug 2022-01-31T22:39:28.898+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.899168+0000) 2022-01-31T22:39:29.311 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[35325]: debug 2022-01-31T22:39:28.921+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227041 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:29.312 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[41853]: debug 2022-01-31T22:39:28.899+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.899744+0000) 2022-01-31T22:39:29.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[46715]: debug 2022-01-31T22:39:28.897+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.898396+0000) 2022-01-31T22:39:29.313 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:28 smithi171 conmon[51620]: debug 2022-01-31T22:39:28.896+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:28.897555+0000) 2022-01-31T22:39:29.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:29 smithi171 conmon[46715]: debug 2022-01-31T22:39:29.454+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:29.455053+0000) 2022-01-31T22:39:29.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:29 smithi171 conmon[51620]: debug 2022-01-31T22:39:29.317+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:29.317615+0000) 2022-01-31T22:39:29.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:29 smithi171 conmon[41853]: debug 2022-01-31T22:39:29.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:29.405716+0000) 2022-01-31T22:39:30.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:29 smithi167 conmon[60316]: debug 2022-01-31T22:39:29.712+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:29.713051+0000) 2022-01-31T22:39:30.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:29 smithi167 conmon[49112]: debug 2022-01-31T22:39:29.786+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:29.787334+0000) 2022-01-31T22:39:30.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:29 smithi167 conmon[54076]: debug 2022-01-31T22:39:29.953+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:29.953788+0000) 2022-01-31T22:39:30.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:30 smithi171 conmon[41853]: debug 2022-01-31T22:39:30.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:30.405860+0000) 2022-01-31T22:39:30.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:30 smithi171 conmon[46715]: debug 2022-01-31T22:39:30.454+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:30.455181+0000) 2022-01-31T22:39:30.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:30 smithi171 conmon[51620]: debug 2022-01-31T22:39:30.317+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:30.317767+0000) 2022-01-31T22:39:31.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:30 smithi167 conmon[49112]: debug 2022-01-31T22:39:30.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:30.787471+0000) 2022-01-31T22:39:31.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:30 smithi167 conmon[54076]: debug 2022-01-31T22:39:30.953+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:30.953962+0000) 2022-01-31T22:39:31.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:30 smithi167 conmon[60316]: debug 2022-01-31T22:39:30.712+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:30.713202+0000) 2022-01-31T22:39:31.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:31 smithi171 conmon[41853]: debug 2022-01-31T22:39:31.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:31.406048+0000) 2022-01-31T22:39:31.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:31 smithi171 conmon[46715]: debug 2022-01-31T22:39:31.455+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:31.455346+0000) 2022-01-31T22:39:31.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:31 smithi171 conmon[51620]: debug 2022-01-31T22:39:31.317+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:31.317928+0000) 2022-01-31T22:39:32.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:31 smithi167 conmon[49112]: debug 2022-01-31T22:39:31.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:31.787680+0000) 2022-01-31T22:39:32.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:31 smithi167 conmon[54076]: debug 2022-01-31T22:39:31.953+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:31.954130+0000) 2022-01-31T22:39:32.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:31 smithi167 conmon[60316]: debug 2022-01-31T22:39:31.712+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:31.713403+0000) 2022-01-31T22:39:32.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:32 smithi171 conmon[41853]: debug 2022-01-31T22:39:32.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:32.406227+0000) 2022-01-31T22:39:32.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:32 smithi171 conmon[46715]: debug 2022-01-31T22:39:32.455+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:32.455530+0000) 2022-01-31T22:39:32.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:32 smithi171 conmon[51620]: debug 2022-01-31T22:39:32.317+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:32.318060+0000) 2022-01-31T22:39:33.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:32 smithi167 conmon[49112]: debug 2022-01-31T22:39:32.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:32.787870+0000) 2022-01-31T22:39:33.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:32 smithi167 conmon[54076]: debug 2022-01-31T22:39:32.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:32.954285+0000) 2022-01-31T22:39:33.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:32 smithi167 conmon[60316]: debug 2022-01-31T22:39:32.713+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:32.713601+0000) 2022-01-31T22:39:33.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[51620]: debug 2022-01-31T22:39:33.318+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.318296+0000) 2022-01-31T22:39:33.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[41853]: debug 2022-01-31T22:39:33.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.406431+0000) 2022-01-31T22:39:33.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[46715]: debug 2022-01-31T22:39:33.455+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.455736+0000) 2022-01-31T22:39:34.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:33 smithi167 conmon[54076]: debug 2022-01-31T22:39:33.924+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.925416+0000) 2022-01-31T22:39:34.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:33 smithi167 conmon[54076]: debug 2022-01-31T22:39:33.953+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.954422+0000) 2022-01-31T22:39:34.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:33 smithi167 conmon[60316]: debug 2022-01-31T22:39:33.713+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.713787+0000) 2022-01-31T22:39:34.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:33 smithi167 conmon[60316]: debug 2022-01-31T22:39:33.924+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.925023+0000) 2022-01-31T22:39:34.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:33 smithi167 conmon[49112]: debug 2022-01-31T22:39:33.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.788050+0000) 2022-01-31T22:39:34.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:33 smithi167 conmon[49112]: debug 2022-01-31T22:39:33.923+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.924262+0000) 2022-01-31T22:39:34.312 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[35325]: debug 2022-01-31T22:39:33.947+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227150 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:34.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[46715]: debug 2022-01-31T22:39:33.923+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.924387+0000) 2022-01-31T22:39:34.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[51620]: debug 2022-01-31T22:39:33.922+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.923717+0000) 2022-01-31T22:39:34.314 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[41853]: debug 2022-01-31T22:39:33.924+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:33.925680+0000) 2022-01-31T22:39:34.315 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:33 smithi171 conmon[41853]: 2022-01-31T22:39:34.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:34 smithi171 conmon[41853]: debug 2022-01-31T22:39:34.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:34.406640+0000) 2022-01-31T22:39:34.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:34 smithi171 conmon[46715]: debug 2022-01-31T22:39:34.455+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:34.455981+0000) 2022-01-31T22:39:34.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:34 smithi171 conmon[51620]: debug 2022-01-31T22:39:34.318+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:34.318516+0000) 2022-01-31T22:39:35.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:34 smithi167 conmon[49112]: debug 2022-01-31T22:39:34.787+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:34.788209+0000) 2022-01-31T22:39:35.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:34 smithi167 conmon[54076]: debug 2022-01-31T22:39:34.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:34.954566+0000) 2022-01-31T22:39:35.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:34 smithi167 conmon[60316]: debug 2022-01-31T22:39:34.713+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:34.714003+0000) 2022-01-31T22:39:35.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:35 smithi171 conmon[41853]: debug 2022-01-31T22:39:35.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:35.406819+0000) 2022-01-31T22:39:35.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:35 smithi171 conmon[46715]: debug 2022-01-31T22:39:35.455+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:35.456150+0000) 2022-01-31T22:39:35.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:35 smithi171 conmon[51620]: debug 2022-01-31T22:39:35.317+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:35.318704+0000) 2022-01-31T22:39:36.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:35 smithi167 conmon[54076]: debug 2022-01-31T22:39:35.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:35.954695+0000) 2022-01-31T22:39:36.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:35 smithi167 conmon[49112]: debug 2022-01-31T22:39:35.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:35.788369+0000) 2022-01-31T22:39:36.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:35 smithi167 conmon[60316]: debug 2022-01-31T22:39:35.713+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:35.714143+0000) 2022-01-31T22:39:36.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:36 smithi171 conmon[46715]: debug 2022-01-31T22:39:36.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:36.456339+0000) 2022-01-31T22:39:36.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:36 smithi171 conmon[41853]: debug 2022-01-31T22:39:36.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:36.406993+0000) 2022-01-31T22:39:36.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:36 smithi171 conmon[51620]: debug 2022-01-31T22:39:36.318+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:36.318941+0000) 2022-01-31T22:39:36.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:36 smithi167 conmon[49112]: debug 2022-01-31T22:39:36.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:36.788537+0000) 2022-01-31T22:39:36.992 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:36 smithi167 conmon[54076]: debug 2022-01-31T22:39:36.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:36.954812+0000) 2022-01-31T22:39:36.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:36 smithi167 conmon[60316]: debug 2022-01-31T22:39:36.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:36.714341+0000) 2022-01-31T22:39:37.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:37 smithi171 conmon[41853]: debug 2022-01-31T22:39:37.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:37.407187+0000) 2022-01-31T22:39:37.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:37 smithi171 conmon[46715]: debug 2022-01-31T22:39:37.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:37.456531+0000) 2022-01-31T22:39:37.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:37 smithi171 conmon[51620]: debug 2022-01-31T22:39:37.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:37.319136+0000) 2022-01-31T22:39:38.002 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:37 smithi167 conmon[54076]: debug 2022-01-31T22:39:37.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:37.954986+0000) 2022-01-31T22:39:38.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:37 smithi167 conmon[60316]: debug 2022-01-31T22:39:37.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:37.714514+0000) 2022-01-31T22:39:38.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:37 smithi167 conmon[49112]: debug 2022-01-31T22:39:37.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:37.788729+0000) 2022-01-31T22:39:38.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[46715]: debug 2022-01-31T22:39:38.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.456767+0000) 2022-01-31T22:39:38.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[41853]: debug 2022-01-31T22:39:38.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.407369+0000) 2022-01-31T22:39:38.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[51620]: debug 2022-01-31T22:39:38.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.319315+0000) 2022-01-31T22:39:39.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:38 smithi167 conmon[49112]: debug 2022-01-31T22:39:38.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.788921+0000) 2022-01-31T22:39:39.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:38 smithi167 conmon[49112]: debug 2022-01-31T22:39:38.950+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.950596+0000) 2022-01-31T22:39:39.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:38 smithi167 conmon[54076]: debug 2022-01-31T22:39:38.950+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.950832+0000) 2022-01-31T22:39:39.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:38 smithi167 conmon[54076]: debug 2022-01-31T22:39:38.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.955097+0000) 2022-01-31T22:39:39.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:38 smithi167 conmon[60316]: debug 2022-01-31T22:39:38.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.714652+0000) 2022-01-31T22:39:39.007 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:38 smithi167 conmon[60316]: debug 2022-01-31T22:39:38.949+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.950176+0000) 2022-01-31T22:39:39.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[46715]: debug 2022-01-31T22:39:38.949+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.950161+0000) 2022-01-31T22:39:39.314 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[51620]: debug 2022-01-31T22:39:38.949+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.949903+0000) 2022-01-31T22:39:39.314 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[35325]: debug 2022-01-31T22:39:38.889+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:39:39.315 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[35325]: debug 2022-01-31T22:39:38.973+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227265 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:39.315 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:38 smithi171 conmon[41853]: debug 2022-01-31T22:39:38.949+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:38.950676+0000) 2022-01-31T22:39:39.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:39 smithi171 conmon[46715]: debug 2022-01-31T22:39:39.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:39.456955+0000) 2022-01-31T22:39:39.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:39 smithi171 conmon[51620]: debug 2022-01-31T22:39:39.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:39.319518+0000) 2022-01-31T22:39:39.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:39 smithi171 conmon[41853]: debug 2022-01-31T22:39:39.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:39.407546+0000) 2022-01-31T22:39:40.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:39 smithi167 conmon[49112]: debug 2022-01-31T22:39:39.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:39.789119+0000) 2022-01-31T22:39:40.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:39 smithi167 conmon[54076]: debug 2022-01-31T22:39:39.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:39.955263+0000) 2022-01-31T22:39:40.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:39 smithi167 conmon[60316]: debug 2022-01-31T22:39:39.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:39.714824+0000) 2022-01-31T22:39:40.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:40 smithi171 conmon[41853]: debug 2022-01-31T22:39:40.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:40.407658+0000) 2022-01-31T22:39:40.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:40 smithi171 conmon[46715]: debug 2022-01-31T22:39:40.457+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:40.457107+0000) 2022-01-31T22:39:40.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:40 smithi171 conmon[51620]: debug 2022-01-31T22:39:40.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:40.319704+0000) 2022-01-31T22:39:41.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:40 smithi167 conmon[49112]: debug 2022-01-31T22:39:40.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:40.789270+0000) 2022-01-31T22:39:41.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:40 smithi167 conmon[54076]: debug 2022-01-31T22:39:40.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:40.955482+0000) 2022-01-31T22:39:41.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:40 smithi167 conmon[60316]: debug 2022-01-31T22:39:40.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:40.714958+0000) 2022-01-31T22:39:41.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:41 smithi171 conmon[41853]: debug 2022-01-31T22:39:41.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:41.407812+0000) 2022-01-31T22:39:41.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:41 smithi171 conmon[46715]: debug 2022-01-31T22:39:41.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:41.457269+0000) 2022-01-31T22:39:41.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:41 smithi171 conmon[51620]: debug 2022-01-31T22:39:41.318+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:41.319878+0000) 2022-01-31T22:39:42.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:41 smithi167 conmon[49112]: debug 2022-01-31T22:39:41.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:41.789436+0000) 2022-01-31T22:39:42.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:41 smithi167 conmon[54076]: debug 2022-01-31T22:39:41.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:41.955712+0000) 2022-01-31T22:39:42.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:41 smithi167 conmon[60316]: debug 2022-01-31T22:39:41.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:41.715139+0000) 2022-01-31T22:39:42.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:42 smithi171 conmon[41853]: debug 2022-01-31T22:39:42.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:42.407966+0000) 2022-01-31T22:39:42.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:42 smithi171 conmon[46715]: debug 2022-01-31T22:39:42.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:42.457425+0000) 2022-01-31T22:39:42.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:42 smithi171 conmon[51620]: debug 2022-01-31T22:39:42.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:42.320068+0000) 2022-01-31T22:39:43.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:42 smithi167 conmon[49112]: debug 2022-01-31T22:39:42.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:42.789660+0000) 2022-01-31T22:39:43.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:42 smithi167 conmon[54076]: debug 2022-01-31T22:39:42.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:42.955903+0000) 2022-01-31T22:39:43.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:42 smithi167 conmon[60316]: debug 2022-01-31T22:39:42.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:42.715341+0000) 2022-01-31T22:39:43.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:43 smithi171 conmon[41853]: debug 2022-01-31T22:39:43.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.408123+0000) 2022-01-31T22:39:43.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:43 smithi171 conmon[46715]: debug 2022-01-31T22:39:43.455+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.457603+0000) 2022-01-31T22:39:43.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:43 smithi171 conmon[51620]: debug 2022-01-31T22:39:43.318+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.320241+0000) 2022-01-31T22:39:43.970 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:43 smithi167 conmon[49112]: debug 2022-01-31T22:39:43.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.789870+0000) 2022-01-31T22:39:43.971 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:43 smithi167 conmon[60316]: debug 2022-01-31T22:39:43.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.715537+0000) 2022-01-31T22:39:43.972 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:43 smithi167 conmon[54076]: debug 2022-01-31T22:39:43.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.956093+0000) 2022-01-31T22:39:43.972 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:43 smithi167 conmon[54076]: debug 2022-01-31T22:39:43.975+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.976653+0000) 2022-01-31T22:39:44.314 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:44 smithi171 conmon[35325]: debug 2022-01-31T22:39:43.998+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227376 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:44.315 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:43 smithi171 conmon[41853]: debug 2022-01-31T22:39:43.973+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.975674+0000) 2022-01-31T22:39:44.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:43 smithi171 conmon[46715]: debug 2022-01-31T22:39:43.973+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.975583+0000) 2022-01-31T22:39:44.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:43 smithi171 conmon[51620]: debug 2022-01-31T22:39:43.974+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.976735+0000) 2022-01-31T22:39:44.342 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:43 smithi167 conmon[49112]: debug 2022-01-31T22:39:43.975+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.977292+0000) 2022-01-31T22:39:44.343 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:43 smithi167 conmon[60316]: debug 2022-01-31T22:39:43.976+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:43.977888+0000) 2022-01-31T22:39:44.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:44 smithi171 conmon[46715]: debug 2022-01-31T22:39:44.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:44.457804+0000) 2022-01-31T22:39:44.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:44 smithi171 conmon[51620]: debug 2022-01-31T22:39:44.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:44.320422+0000) 2022-01-31T22:39:44.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:44 smithi171 conmon[41853]: debug 2022-01-31T22:39:44.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:44.408305+0000) 2022-01-31T22:39:45.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:44 smithi167 conmon[49112]: debug 2022-01-31T22:39:44.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:44.790029+0000) 2022-01-31T22:39:45.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:44 smithi167 conmon[54076]: debug 2022-01-31T22:39:44.954+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:44.956302+0000) 2022-01-31T22:39:45.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:44 smithi167 conmon[60316]: debug 2022-01-31T22:39:44.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:44.715708+0000) 2022-01-31T22:39:45.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:45 smithi171 conmon[41853]: debug 2022-01-31T22:39:45.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:45.408455+0000) 2022-01-31T22:39:46.025 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:45 smithi171 conmon[46715]: debug 2022-01-31T22:39:45.457+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:45.457942+0000) 2022-01-31T22:39:46.026 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:45 smithi171 conmon[51620]: debug 2022-01-31T22:39:45.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:45.320582+0000) 2022-01-31T22:39:46.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:45 smithi167 conmon[49112]: debug 2022-01-31T22:39:45.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:45.790183+0000) 2022-01-31T22:39:46.027 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:45 smithi167 conmon[54076]: debug 2022-01-31T22:39:45.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:45.956461+0000) 2022-01-31T22:39:46.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:45 smithi167 conmon[60316]: debug 2022-01-31T22:39:45.714+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:45.715837+0000) 2022-01-31T22:39:46.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:46 smithi171 conmon[41853]: debug 2022-01-31T22:39:46.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:46.408659+0000) 2022-01-31T22:39:46.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:46 smithi171 conmon[46715]: debug 2022-01-31T22:39:46.457+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:46.458133+0000) 2022-01-31T22:39:46.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:46 smithi171 conmon[51620]: debug 2022-01-31T22:39:46.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:46.320846+0000) 2022-01-31T22:39:47.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:46 smithi167 conmon[49112]: debug 2022-01-31T22:39:46.788+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:46.790380+0000) 2022-01-31T22:39:47.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:46 smithi167 conmon[54076]: debug 2022-01-31T22:39:46.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:46.956667+0000) 2022-01-31T22:39:47.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:46 smithi167 conmon[60316]: debug 2022-01-31T22:39:46.715+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:46.715993+0000) 2022-01-31T22:39:47.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:47 smithi171 conmon[46715]: debug 2022-01-31T22:39:47.457+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:47.458324+0000) 2022-01-31T22:39:47.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:47 smithi171 conmon[41853]: debug 2022-01-31T22:39:47.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:47.408843+0000) 2022-01-31T22:39:47.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:47 smithi171 conmon[51620]: debug 2022-01-31T22:39:47.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:47.321100+0000) 2022-01-31T22:39:48.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:47 smithi167 conmon[49112]: debug 2022-01-31T22:39:47.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:47.790523+0000) 2022-01-31T22:39:48.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:47 smithi167 conmon[54076]: debug 2022-01-31T22:39:47.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:47.956872+0000) 2022-01-31T22:39:48.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:47 smithi167 conmon[60316]: debug 2022-01-31T22:39:47.715+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:47.716160+0000) 2022-01-31T22:39:48.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:48 smithi171 conmon[46715]: debug 2022-01-31T22:39:48.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:48.458523+0000) 2022-01-31T22:39:48.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:48 smithi171 conmon[41853]: debug 2022-01-31T22:39:48.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:48.409032+0000) 2022-01-31T22:39:48.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:48 smithi171 conmon[51620]: debug 2022-01-31T22:39:48.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:48.321230+0000) 2022-01-31T22:39:48.995 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[41853]: debug 2022-01-31T22:39:49.000+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.001886+0000) 2022-01-31T22:39:48.996 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:48 smithi167 conmon[49112]: debug 2022-01-31T22:39:48.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:48.790656+0000) 2022-01-31T22:39:48.998 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:48 smithi167 conmon[54076]: debug 2022-01-31T22:39:48.955+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:48.957050+0000) 2022-01-31T22:39:48.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:48 smithi167 conmon[60316]: debug 2022-01-31T22:39:48.715+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:48.716300+0000) 2022-01-31T22:39:49.315 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[35325]: debug 2022-01-31T22:39:49.023+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227490 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:49.315 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[46715]: debug 2022-01-31T22:39:49.000+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.002506+0000) 2022-01-31T22:39:49.316 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[51620]: debug 2022-01-31T22:39:49.002+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.003866+0000) 2022-01-31T22:39:49.365 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:49 smithi167 conmon[49112]: debug 2022-01-31T22:39:49.002+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.003733+0000) 2022-01-31T22:39:49.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:49 smithi167 conmon[54076]: debug 2022-01-31T22:39:49.000+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.002466+0000) 2022-01-31T22:39:49.367 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:49 smithi167 conmon[60316]: debug 2022-01-31T22:39:49.002+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.004249+0000) 2022-01-31T22:39:49.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[46715]: debug 2022-01-31T22:39:49.456+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.458699+0000) 2022-01-31T22:39:49.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[51620]: debug 2022-01-31T22:39:49.320+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.321427+0000) 2022-01-31T22:39:49.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:49 smithi171 conmon[41853]: debug 2022-01-31T22:39:49.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.409214+0000) 2022-01-31T22:39:50.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:49 smithi167 conmon[49112]: debug 2022-01-31T22:39:49.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.790830+0000) 2022-01-31T22:39:50.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:49 smithi167 conmon[54076]: debug 2022-01-31T22:39:49.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.957247+0000) 2022-01-31T22:39:50.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:49 smithi167 conmon[60316]: debug 2022-01-31T22:39:49.715+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:49.716529+0000) 2022-01-31T22:39:50.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:50 smithi171 conmon[46715]: debug 2022-01-31T22:39:50.458+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:50.458850+0000) 2022-01-31T22:39:50.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:50 smithi171 conmon[51620]: debug 2022-01-31T22:39:50.319+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:50.321615+0000) 2022-01-31T22:39:50.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:50 smithi171 conmon[41853]: debug 2022-01-31T22:39:50.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:50.409381+0000) 2022-01-31T22:39:51.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:50 smithi167 conmon[49112]: debug 2022-01-31T22:39:50.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:50.790999+0000) 2022-01-31T22:39:51.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:50 smithi167 conmon[54076]: debug 2022-01-31T22:39:50.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:50.957425+0000) 2022-01-31T22:39:51.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:50 smithi167 conmon[60316]: debug 2022-01-31T22:39:50.715+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:50.716682+0000) 2022-01-31T22:39:51.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:51 smithi171 conmon[41853]: debug 2022-01-31T22:39:51.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:51.409585+0000) 2022-01-31T22:39:51.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:51 smithi171 conmon[46715]: debug 2022-01-31T22:39:51.458+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:51.459004+0000) 2022-01-31T22:39:51.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:51 smithi171 conmon[51620]: debug 2022-01-31T22:39:51.321+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:51.321845+0000) 2022-01-31T22:39:52.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:51 smithi167 conmon[49112]: debug 2022-01-31T22:39:51.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:51.791186+0000) 2022-01-31T22:39:52.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:51 smithi167 conmon[54076]: debug 2022-01-31T22:39:51.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:51.957609+0000) 2022-01-31T22:39:52.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:51 smithi167 conmon[60316]: debug 2022-01-31T22:39:51.715+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:51.716862+0000) 2022-01-31T22:39:52.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:52 smithi171 conmon[41853]: debug 2022-01-31T22:39:52.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:52.409793+0000) 2022-01-31T22:39:52.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:52 smithi171 conmon[46715]: debug 2022-01-31T22:39:52.458+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:52.459192+0000) 2022-01-31T22:39:52.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:52 smithi171 conmon[51620]: debug 2022-01-31T22:39:52.321+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:52.322014+0000) 2022-01-31T22:39:53.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:52 smithi167 conmon[49112]: debug 2022-01-31T22:39:52.789+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:52.791345+0000) 2022-01-31T22:39:53.006 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:52 smithi167 conmon[54076]: debug 2022-01-31T22:39:52.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:52.957769+0000) 2022-01-31T22:39:53.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:52 smithi167 conmon[60316]: debug 2022-01-31T22:39:52.716+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:52.717029+0000) 2022-01-31T22:39:53.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:53 smithi171 conmon[41853]: debug 2022-01-31T22:39:53.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:53.409954+0000) 2022-01-31T22:39:53.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:53 smithi171 conmon[46715]: debug 2022-01-31T22:39:53.458+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:53.459364+0000) 2022-01-31T22:39:53.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:53 smithi171 conmon[51620]: debug 2022-01-31T22:39:53.321+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:53.322197+0000) 2022-01-31T22:39:54.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:53 smithi167 conmon[49112]: debug 2022-01-31T22:39:53.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:53.791544+0000) 2022-01-31T22:39:54.006 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:53 smithi167 conmon[54076]: debug 2022-01-31T22:39:53.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:53.957924+0000) 2022-01-31T22:39:54.007 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:53 smithi167 conmon[60316]: debug 2022-01-31T22:39:53.716+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:53.717192+0000) 2022-01-31T22:39:54.316 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:53 smithi171 conmon[35325]: debug 2022-01-31T22:39:53.889+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:39:54.317 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[35325]: debug 2022-01-31T22:39:54.050+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227599 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:54.318 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[41853]: debug 2022-01-31T22:39:54.027+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.028382+0000) 2022-01-31T22:39:54.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[46715]: debug 2022-01-31T22:39:54.027+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.027990+0000) 2022-01-31T22:39:54.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[51620]: debug 2022-01-31T22:39:54.028+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.029279+0000) 2022-01-31T22:39:54.381 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:54 smithi167 conmon[49112]: debug 2022-01-31T22:39:54.028+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.029556+0000) 2022-01-31T22:39:54.382 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:54 smithi167 conmon[54076]: debug 2022-01-31T22:39:54.026+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.027608+0000) 2022-01-31T22:39:54.383 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:54 smithi167 conmon[60316]: debug 2022-01-31T22:39:54.027+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.029224+0000) 2022-01-31T22:39:54.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[51620]: debug 2022-01-31T22:39:54.321+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.322353+0000) 2022-01-31T22:39:54.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[41853]: debug 2022-01-31T22:39:54.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.410157+0000) 2022-01-31T22:39:54.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:54 smithi171 conmon[46715]: debug 2022-01-31T22:39:54.458+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.459543+0000) 2022-01-31T22:39:55.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:54 smithi167 conmon[49112]: debug 2022-01-31T22:39:54.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.791736+0000) 2022-01-31T22:39:55.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:54 smithi167 conmon[60316]: debug 2022-01-31T22:39:54.716+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.717375+0000) 2022-01-31T22:39:55.006 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:54 smithi167 conmon[54076]: debug 2022-01-31T22:39:54.957+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:54.958076+0000) 2022-01-31T22:39:55.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:55 smithi171 conmon[46715]: debug 2022-01-31T22:39:55.458+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:55.459688+0000) 2022-01-31T22:39:55.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:55 smithi171 conmon[41853]: debug 2022-01-31T22:39:55.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:55.410288+0000) 2022-01-31T22:39:55.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:55 smithi171 conmon[51620]: debug 2022-01-31T22:39:55.321+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:55.322534+0000) 2022-01-31T22:39:56.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:55 smithi167 conmon[49112]: debug 2022-01-31T22:39:55.790+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:55.791860+0000) 2022-01-31T22:39:56.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:55 smithi167 conmon[60316]: debug 2022-01-31T22:39:55.716+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:55.717536+0000) 2022-01-31T22:39:56.007 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:55 smithi167 conmon[54076]: debug 2022-01-31T22:39:55.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:55.958213+0000) 2022-01-31T22:39:56.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:56 smithi171 conmon[46715]: debug 2022-01-31T22:39:56.459+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:56.459882+0000) 2022-01-31T22:39:56.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:56 smithi171 conmon[41853]: debug 2022-01-31T22:39:56.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:56.410441+0000) 2022-01-31T22:39:56.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:56 smithi171 conmon[51620]: debug 2022-01-31T22:39:56.322+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:56.322735+0000) 2022-01-31T22:39:57.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:56 smithi167 conmon[49112]: debug 2022-01-31T22:39:56.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:56.792026+0000) 2022-01-31T22:39:57.007 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:56 smithi167 conmon[54076]: debug 2022-01-31T22:39:56.956+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:56.958429+0000) 2022-01-31T22:39:57.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:56 smithi167 conmon[60316]: debug 2022-01-31T22:39:56.716+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:56.717740+0000) 2022-01-31T22:39:57.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:57 smithi171 conmon[46715]: debug 2022-01-31T22:39:57.459+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:57.460058+0000) 2022-01-31T22:39:57.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:57 smithi171 conmon[51620]: debug 2022-01-31T22:39:57.322+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:57.322923+0000) 2022-01-31T22:39:57.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:57 smithi171 conmon[41853]: debug 2022-01-31T22:39:57.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:57.410594+0000) 2022-01-31T22:39:58.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:57 smithi167 conmon[49112]: debug 2022-01-31T22:39:57.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:57.792210+0000) 2022-01-31T22:39:58.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:57 smithi167 conmon[60316]: debug 2022-01-31T22:39:57.716+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:57.717870+0000) 2022-01-31T22:39:58.007 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:57 smithi167 conmon[54076]: debug 2022-01-31T22:39:57.957+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:57.958616+0000) 2022-01-31T22:39:58.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:58 smithi171 conmon[46715]: debug 2022-01-31T22:39:58.459+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:58.460246+0000) 2022-01-31T22:39:58.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:58 smithi171 conmon[41853]: debug 2022-01-31T22:39:58.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:58.410797+0000) 2022-01-31T22:39:58.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:58 smithi171 conmon[51620]: debug 2022-01-31T22:39:58.322+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:58.323142+0000) 2022-01-31T22:39:59.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:58 smithi167 conmon[49112]: debug 2022-01-31T22:39:58.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:58.792369+0000) 2022-01-31T22:39:59.007 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:58 smithi167 conmon[54076]: debug 2022-01-31T22:39:58.958+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:58.958806+0000) 2022-01-31T22:39:59.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:58 smithi167 conmon[60316]: debug 2022-01-31T22:39:58.717+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:58.718040+0000) 2022-01-31T22:39:59.317 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[35325]: debug 2022-01-31T22:39:59.077+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227713 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:39:59.317 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[41853]: debug 2022-01-31T22:39:59.054+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.055149+0000) 2022-01-31T22:39:59.318 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[46715]: debug 2022-01-31T22:39:59.052+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.053468+0000) 2022-01-31T22:39:59.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[51620]: debug 2022-01-31T22:39:59.053+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.054245+0000) 2022-01-31T22:39:59.397 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:59 smithi167 conmon[49112]: debug 2022-01-31T22:39:59.054+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.055554+0000) 2022-01-31T22:39:59.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:59 smithi167 conmon[54076]: debug 2022-01-31T22:39:59.052+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.053795+0000) 2022-01-31T22:39:59.398 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:59 smithi167 conmon[60316]: debug 2022-01-31T22:39:59.053+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.055466+0000) 2022-01-31T22:39:59.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[41853]: debug 2022-01-31T22:39:59.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.410925+0000) 2022-01-31T22:39:59.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[51620]: debug 2022-01-31T22:39:59.322+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.323374+0000) 2022-01-31T22:39:59.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:39:59 smithi171 conmon[46715]: debug 2022-01-31T22:39:59.459+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.460401+0000) 2022-01-31T22:40:00.006 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:39:59 smithi167 conmon[54076]: debug 2022-01-31T22:39:59.958+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.958968+0000) 2022-01-31T22:40:00.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:39:59 smithi167 conmon[49112]: debug 2022-01-31T22:39:59.791+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.792565+0000) 2022-01-31T22:40:00.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:39:59 smithi167 conmon[60316]: debug 2022-01-31T22:39:59.717+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:39:59.718160+0000) 2022-01-31T22:40:00.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:00 smithi171 conmon[41853]: debug 2022-01-31T22:40:00.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:00.411062+0000) 2022-01-31T22:40:00.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:00 smithi171 conmon[51620]: debug 2022-01-31T22:40:00.322+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:00.323565+0000) 2022-01-31T22:40:00.604 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:00 smithi171 conmon[46715]: debug 2022-01-31T22:40:00.459+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:00.460543+0000) 2022-01-31T22:40:01.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:00 smithi167 conmon[49112]: debug 2022-01-31T22:40:00.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:00.792716+0000) 2022-01-31T22:40:01.007 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:00 smithi167 conmon[54076]: debug 2022-01-31T22:40:00.958+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:00.959168+0000) 2022-01-31T22:40:01.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:00 smithi167 conmon[60316]: debug 2022-01-31T22:40:00.717+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:00.718319+0000) 2022-01-31T22:40:01.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:01 smithi171 conmon[41853]: debug 2022-01-31T22:40:01.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:01.411242+0000) 2022-01-31T22:40:01.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:01 smithi171 conmon[46715]: debug 2022-01-31T22:40:01.460+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:01.460731+0000) 2022-01-31T22:40:01.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:01 smithi171 conmon[51620]: debug 2022-01-31T22:40:01.323+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:01.323794+0000) 2022-01-31T22:40:02.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:01 smithi167 conmon[49112]: debug 2022-01-31T22:40:01.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:01.792917+0000) 2022-01-31T22:40:02.007 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:01 smithi167 conmon[60316]: debug 2022-01-31T22:40:01.717+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:01.718526+0000) 2022-01-31T22:40:02.007 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:01 smithi167 conmon[54076]: debug 2022-01-31T22:40:01.957+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:01.959356+0000) 2022-01-31T22:40:02.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:02 smithi171 conmon[41853]: debug 2022-01-31T22:40:02.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:02.411411+0000) 2022-01-31T22:40:02.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:02 smithi171 conmon[51620]: debug 2022-01-31T22:40:02.323+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:02.324012+0000) 2022-01-31T22:40:02.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:02 smithi171 conmon[46715]: debug 2022-01-31T22:40:02.460+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:02.460912+0000) 2022-01-31T22:40:03.006 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:02 smithi167 conmon[49112]: debug 2022-01-31T22:40:02.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:02.793076+0000) 2022-01-31T22:40:03.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:02 smithi167 conmon[54076]: debug 2022-01-31T22:40:02.958+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:02.959560+0000) 2022-01-31T22:40:03.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:02 smithi167 conmon[60316]: debug 2022-01-31T22:40:02.718+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:02.718750+0000) 2022-01-31T22:40:03.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:03 smithi171 conmon[41853]: debug 2022-01-31T22:40:03.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:03.411633+0000) 2022-01-31T22:40:03.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:03 smithi171 conmon[51620]: debug 2022-01-31T22:40:03.323+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:03.324225+0000) 2022-01-31T22:40:03.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:03 smithi171 conmon[46715]: debug 2022-01-31T22:40:03.460+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:03.461098+0000) 2022-01-31T22:40:04.006 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:03 smithi167 conmon[54076]: debug 2022-01-31T22:40:03.958+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:03.959722+0000) 2022-01-31T22:40:04.007 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:03 smithi167 conmon[60316]: debug 2022-01-31T22:40:03.718+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:03.718975+0000) 2022-01-31T22:40:04.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:03 smithi167 conmon[49112]: debug 2022-01-31T22:40:03.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:03.793290+0000) 2022-01-31T22:40:04.351 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[35325]: debug 2022-01-31T22:40:04.103+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227823 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:04.352 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[41853]: debug 2022-01-31T22:40:04.079+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.080408+0000) 2022-01-31T22:40:04.353 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[46715]: debug 2022-01-31T22:40:04.079+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.080048+0000) 2022-01-31T22:40:04.354 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[51620]: debug 2022-01-31T22:40:04.079+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.080104+0000) 2022-01-31T22:40:04.355 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[51620]: debug 2022-01-31T22:40:04.323+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.324400+0000) 2022-01-31T22:40:04.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:04 smithi167 conmon[49112]: debug 2022-01-31T22:40:04.080+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.081637+0000) 2022-01-31T22:40:04.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:04 smithi167 conmon[54076]: debug 2022-01-31T22:40:04.079+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.080689+0000) 2022-01-31T22:40:04.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:04 smithi167 conmon[60316]: debug 2022-01-31T22:40:04.080+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.081757+0000) 2022-01-31T22:40:04.851 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[41853]: debug 2022-01-31T22:40:04.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.411793+0000) 2022-01-31T22:40:04.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:04 smithi171 conmon[46715]: debug 2022-01-31T22:40:04.460+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.461313+0000) 2022-01-31T22:40:05.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:04 smithi167 conmon[49112]: debug 2022-01-31T22:40:04.792+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.793482+0000) 2022-01-31T22:40:05.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:04 smithi167 conmon[60316]: debug 2022-01-31T22:40:04.717+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.719173+0000) 2022-01-31T22:40:05.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:04 smithi167 conmon[54076]: debug 2022-01-31T22:40:04.958+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:04.959923+0000) 2022-01-31T22:40:05.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:05 smithi171 conmon[41853]: debug 2022-01-31T22:40:05.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:05.411971+0000) 2022-01-31T22:40:05.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:05 smithi171 conmon[46715]: debug 2022-01-31T22:40:05.460+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:05.461507+0000) 2022-01-31T22:40:05.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:05 smithi171 conmon[51620]: debug 2022-01-31T22:40:05.323+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:05.324591+0000) 2022-01-31T22:40:06.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:05 smithi167 conmon[49112]: debug 2022-01-31T22:40:05.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:05.793654+0000) 2022-01-31T22:40:06.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:05 smithi167 conmon[54076]: debug 2022-01-31T22:40:05.959+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:05.960066+0000) 2022-01-31T22:40:06.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:05 smithi167 conmon[60316]: debug 2022-01-31T22:40:05.717+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:05.719311+0000) 2022-01-31T22:40:06.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:06 smithi171 conmon[41853]: debug 2022-01-31T22:40:06.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:06.412114+0000) 2022-01-31T22:40:06.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:06 smithi171 conmon[46715]: debug 2022-01-31T22:40:06.460+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:06.461665+0000) 2022-01-31T22:40:06.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:06 smithi171 conmon[51620]: debug 2022-01-31T22:40:06.324+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:06.324834+0000) 2022-01-31T22:40:07.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:06 smithi167 conmon[49112]: debug 2022-01-31T22:40:06.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:06.793812+0000) 2022-01-31T22:40:07.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:06 smithi167 conmon[54076]: debug 2022-01-31T22:40:06.959+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:06.960246+0000) 2022-01-31T22:40:07.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:06 smithi167 conmon[60316]: debug 2022-01-31T22:40:06.719+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:06.719546+0000) 2022-01-31T22:40:07.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:07 smithi171 conmon[46715]: debug 2022-01-31T22:40:07.461+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:07.461860+0000) 2022-01-31T22:40:07.604 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:07 smithi171 conmon[41853]: debug 2022-01-31T22:40:07.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:07.412316+0000) 2022-01-31T22:40:07.604 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:07 smithi171 conmon[51620]: debug 2022-01-31T22:40:07.324+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:07.325044+0000) 2022-01-31T22:40:08.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:07 smithi167 conmon[49112]: debug 2022-01-31T22:40:07.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:07.794038+0000) 2022-01-31T22:40:08.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:07 smithi167 conmon[54076]: debug 2022-01-31T22:40:07.959+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:07.960428+0000) 2022-01-31T22:40:08.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:07 smithi167 conmon[60316]: debug 2022-01-31T22:40:07.719+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:07.719712+0000) 2022-01-31T22:40:08.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:08 smithi171 conmon[46715]: debug 2022-01-31T22:40:08.461+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:08.462050+0000) 2022-01-31T22:40:08.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:08 smithi171 conmon[41853]: debug 2022-01-31T22:40:08.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:08.412493+0000) 2022-01-31T22:40:08.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:08 smithi171 conmon[51620]: debug 2022-01-31T22:40:08.324+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:08.325258+0000) 2022-01-31T22:40:09.007 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:08 smithi167 conmon[60316]: debug 2022-01-31T22:40:08.719+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:08.719859+0000) 2022-01-31T22:40:09.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:08 smithi167 conmon[49112]: debug 2022-01-31T22:40:08.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:08.794197+0000) 2022-01-31T22:40:09.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:08 smithi167 conmon[54076]: debug 2022-01-31T22:40:08.960+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:08.960609+0000) 2022-01-31T22:40:09.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[41853]: debug 2022-01-31T22:40:09.106+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.107662+0000) 2022-01-31T22:40:09.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[46715]: debug 2022-01-31T22:40:09.105+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.106497+0000) 2022-01-31T22:40:09.320 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[51620]: debug 2022-01-31T22:40:09.105+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.106289+0000) 2022-01-31T22:40:09.321 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:08 smithi171 conmon[35325]: debug 2022-01-31T22:40:08.890+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:40:09.321 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[35325]: debug 2022-01-31T22:40:09.135+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 227939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:09.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:09 smithi167 conmon[49112]: debug 2022-01-31T22:40:09.113+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.113953+0000) 2022-01-31T22:40:09.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:09 smithi167 conmon[54076]: debug 2022-01-31T22:40:09.106+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.107085+0000) 2022-01-31T22:40:09.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:09 smithi167 conmon[60316]: debug 2022-01-31T22:40:09.113+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.113844+0000) 2022-01-31T22:40:09.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[41853]: debug 2022-01-31T22:40:09.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.412677+0000) 2022-01-31T22:40:09.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[46715]: debug 2022-01-31T22:40:09.461+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.462269+0000) 2022-01-31T22:40:09.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:09 smithi171 conmon[51620]: debug 2022-01-31T22:40:09.324+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.325431+0000) 2022-01-31T22:40:10.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:09 smithi167 conmon[49112]: debug 2022-01-31T22:40:09.793+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.794364+0000) 2022-01-31T22:40:10.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:09 smithi167 conmon[54076]: debug 2022-01-31T22:40:09.960+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.960792+0000) 2022-01-31T22:40:10.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:09 smithi167 conmon[60316]: debug 2022-01-31T22:40:09.719+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:09.720039+0000) 2022-01-31T22:40:10.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:10 smithi171 conmon[41853]: debug 2022-01-31T22:40:10.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:10.412831+0000) 2022-01-31T22:40:10.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:10 smithi171 conmon[46715]: debug 2022-01-31T22:40:10.461+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:10.462448+0000) 2022-01-31T22:40:10.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:10 smithi171 conmon[51620]: debug 2022-01-31T22:40:10.325+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:10.325607+0000) 2022-01-31T22:40:11.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:10 smithi167 conmon[49112]: debug 2022-01-31T22:40:10.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:10.794559+0000) 2022-01-31T22:40:11.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:10 smithi167 conmon[54076]: debug 2022-01-31T22:40:10.960+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:10.960946+0000) 2022-01-31T22:40:11.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:10 smithi167 conmon[60316]: debug 2022-01-31T22:40:10.719+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:10.720171+0000) 2022-01-31T22:40:11.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:11 smithi171 conmon[41853]: debug 2022-01-31T22:40:11.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:11.413002+0000) 2022-01-31T22:40:11.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:11 smithi171 conmon[46715]: debug 2022-01-31T22:40:11.462+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:11.462632+0000) 2022-01-31T22:40:11.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:11 smithi171 conmon[51620]: debug 2022-01-31T22:40:11.325+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:11.325844+0000) 2022-01-31T22:40:12.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:11 smithi167 conmon[49112]: debug 2022-01-31T22:40:11.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:11.794742+0000) 2022-01-31T22:40:12.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:11 smithi167 conmon[54076]: debug 2022-01-31T22:40:11.960+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:11.961118+0000) 2022-01-31T22:40:12.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:11 smithi167 conmon[60316]: debug 2022-01-31T22:40:11.719+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:11.720331+0000) 2022-01-31T22:40:12.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:12 smithi171 conmon[41853]: debug 2022-01-31T22:40:12.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:12.413210+0000) 2022-01-31T22:40:12.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:12 smithi171 conmon[46715]: debug 2022-01-31T22:40:12.462+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:12.462853+0000) 2022-01-31T22:40:12.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:12 smithi171 conmon[51620]: debug 2022-01-31T22:40:12.325+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:12.326050+0000) 2022-01-31T22:40:13.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:12 smithi167 conmon[49112]: debug 2022-01-31T22:40:12.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:12.794939+0000) 2022-01-31T22:40:13.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:12 smithi167 conmon[54076]: debug 2022-01-31T22:40:12.960+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:12.961329+0000) 2022-01-31T22:40:13.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:12 smithi167 conmon[60316]: debug 2022-01-31T22:40:12.720+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:12.720541+0000) 2022-01-31T22:40:13.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:13 smithi171 conmon[41853]: debug 2022-01-31T22:40:13.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:13.413396+0000) 2022-01-31T22:40:13.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:13 smithi171 conmon[46715]: debug 2022-01-31T22:40:13.462+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:13.463054+0000) 2022-01-31T22:40:13.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:13 smithi171 conmon[51620]: debug 2022-01-31T22:40:13.325+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:13.326245+0000) 2022-01-31T22:40:14.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:13 smithi167 conmon[49112]: debug 2022-01-31T22:40:13.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:13.795125+0000) 2022-01-31T22:40:14.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:13 smithi167 conmon[54076]: debug 2022-01-31T22:40:13.961+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:13.961544+0000) 2022-01-31T22:40:14.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:13 smithi167 conmon[60316]: debug 2022-01-31T22:40:13.720+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:13.720744+0000) 2022-01-31T22:40:14.407 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[35325]: debug 2022-01-31T22:40:14.161+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228049 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:14.407 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[41853]: debug 2022-01-31T22:40:14.139+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.140397+0000) 2022-01-31T22:40:14.408 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[46715]: debug 2022-01-31T22:40:14.138+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.139180+0000) 2022-01-31T22:40:14.409 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[51620]: debug 2022-01-31T22:40:14.138+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.139712+0000) 2022-01-31T22:40:14.409 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[51620]: debug 2022-01-31T22:40:14.326+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.326388+0000) 2022-01-31T22:40:14.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:14 smithi167 conmon[49112]: debug 2022-01-31T22:40:14.139+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.140061+0000) 2022-01-31T22:40:14.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:14 smithi167 conmon[54076]: debug 2022-01-31T22:40:14.139+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.139922+0000) 2022-01-31T22:40:14.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:14 smithi167 conmon[60316]: debug 2022-01-31T22:40:14.139+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.139766+0000) 2022-01-31T22:40:14.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[41853]: debug 2022-01-31T22:40:14.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.413584+0000) 2022-01-31T22:40:14.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:14 smithi171 conmon[46715]: debug 2022-01-31T22:40:14.462+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.463298+0000) 2022-01-31T22:40:15.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:14 smithi167 conmon[49112]: debug 2022-01-31T22:40:14.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.795312+0000) 2022-01-31T22:40:15.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:14 smithi167 conmon[54076]: debug 2022-01-31T22:40:14.961+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.961717+0000) 2022-01-31T22:40:15.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:14 smithi167 conmon[60316]: debug 2022-01-31T22:40:14.720+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:14.720961+0000) 2022-01-31T22:40:15.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:15 smithi171 conmon[41853]: debug 2022-01-31T22:40:15.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:15.413742+0000) 2022-01-31T22:40:15.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:15 smithi171 conmon[51620]: debug 2022-01-31T22:40:15.325+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:15.326512+0000) 2022-01-31T22:40:15.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:15 smithi171 conmon[46715]: debug 2022-01-31T22:40:15.462+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:15.463437+0000) 2022-01-31T22:40:16.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:15 smithi167 conmon[49112]: debug 2022-01-31T22:40:15.794+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:15.795470+0000) 2022-01-31T22:40:16.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:15 smithi167 conmon[54076]: debug 2022-01-31T22:40:15.961+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:15.961816+0000) 2022-01-31T22:40:16.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:15 smithi167 conmon[60316]: debug 2022-01-31T22:40:15.720+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:15.721098+0000) 2022-01-31T22:40:16.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:16 smithi171 conmon[41853]: debug 2022-01-31T22:40:16.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:16.413931+0000) 2022-01-31T22:40:16.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:16 smithi171 conmon[46715]: debug 2022-01-31T22:40:16.462+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:16.463603+0000) 2022-01-31T22:40:16.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:16 smithi171 conmon[51620]: debug 2022-01-31T22:40:16.326+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:16.326727+0000) 2022-01-31T22:40:17.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:16 smithi167 conmon[49112]: debug 2022-01-31T22:40:16.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:16.795659+0000) 2022-01-31T22:40:17.010 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:16 smithi167 conmon[54076]: debug 2022-01-31T22:40:16.961+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:16.961973+0000) 2022-01-31T22:40:17.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:16 smithi167 conmon[60316]: debug 2022-01-31T22:40:16.720+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:16.721307+0000) 2022-01-31T22:40:17.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:17 smithi171 conmon[41853]: debug 2022-01-31T22:40:17.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:17.414082+0000) 2022-01-31T22:40:17.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:17 smithi171 conmon[46715]: debug 2022-01-31T22:40:17.463+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:17.463790+0000) 2022-01-31T22:40:17.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:17 smithi171 conmon[51620]: debug 2022-01-31T22:40:17.326+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:17.326982+0000) 2022-01-31T22:40:18.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:17 smithi167 conmon[54076]: debug 2022-01-31T22:40:17.961+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:17.962127+0000) 2022-01-31T22:40:18.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:17 smithi167 conmon[49112]: debug 2022-01-31T22:40:17.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:17.795810+0000) 2022-01-31T22:40:18.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:17 smithi167 conmon[60316]: debug 2022-01-31T22:40:17.721+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:17.721470+0000) 2022-01-31T22:40:18.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:18 smithi171 conmon[41853]: debug 2022-01-31T22:40:18.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:18.414254+0000) 2022-01-31T22:40:18.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:18 smithi171 conmon[51620]: debug 2022-01-31T22:40:18.326+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:18.327196+0000) 2022-01-31T22:40:18.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:18 smithi171 conmon[46715]: debug 2022-01-31T22:40:18.463+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:18.463939+0000) 2022-01-31T22:40:19.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:18 smithi167 conmon[54076]: debug 2022-01-31T22:40:18.961+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:18.962342+0000) 2022-01-31T22:40:19.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:18 smithi167 conmon[49112]: debug 2022-01-31T22:40:18.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:18.796050+0000) 2022-01-31T22:40:19.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:18 smithi167 conmon[60316]: debug 2022-01-31T22:40:18.721+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:18.721671+0000) 2022-01-31T22:40:19.407 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[35325]: debug 2022-01-31T22:40:19.187+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:19.408 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[41853]: debug 2022-01-31T22:40:19.165+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.165933+0000) 2022-01-31T22:40:19.409 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[46715]: debug 2022-01-31T22:40:19.164+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.164784+0000) 2022-01-31T22:40:19.409 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[51620]: debug 2022-01-31T22:40:19.164+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.164897+0000) 2022-01-31T22:40:19.410 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[51620]: debug 2022-01-31T22:40:19.327+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.327411+0000) 2022-01-31T22:40:19.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:19 smithi167 conmon[49112]: debug 2022-01-31T22:40:19.165+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.166128+0000) 2022-01-31T22:40:19.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:19 smithi167 conmon[54076]: debug 2022-01-31T22:40:19.165+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.165762+0000) 2022-01-31T22:40:19.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:19 smithi167 conmon[60316]: debug 2022-01-31T22:40:19.165+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.165578+0000) 2022-01-31T22:40:19.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[41853]: debug 2022-01-31T22:40:19.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.414434+0000) 2022-01-31T22:40:19.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:19 smithi171 conmon[46715]: debug 2022-01-31T22:40:19.463+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.464130+0000) 2022-01-31T22:40:20.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:19 smithi167 conmon[49112]: debug 2022-01-31T22:40:19.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.796279+0000) 2022-01-31T22:40:20.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:19 smithi167 conmon[54076]: debug 2022-01-31T22:40:19.962+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.962522+0000) 2022-01-31T22:40:20.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:19 smithi167 conmon[60316]: debug 2022-01-31T22:40:19.721+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:19.721845+0000) 2022-01-31T22:40:20.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:20 smithi171 conmon[41853]: debug 2022-01-31T22:40:20.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:20.414558+0000) 2022-01-31T22:40:20.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:20 smithi171 conmon[46715]: debug 2022-01-31T22:40:20.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:20.464283+0000) 2022-01-31T22:40:20.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:20 smithi171 conmon[51620]: debug 2022-01-31T22:40:20.326+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:20.327583+0000) 2022-01-31T22:40:21.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:20 smithi167 conmon[49112]: debug 2022-01-31T22:40:20.795+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:20.796443+0000) 2022-01-31T22:40:21.010 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:20 smithi167 conmon[54076]: debug 2022-01-31T22:40:20.962+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:20.962671+0000) 2022-01-31T22:40:21.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:20 smithi167 conmon[60316]: debug 2022-01-31T22:40:20.721+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:20.721981+0000) 2022-01-31T22:40:21.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:21 smithi171 conmon[46715]: debug 2022-01-31T22:40:21.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:21.464438+0000) 2022-01-31T22:40:21.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:21 smithi171 conmon[51620]: debug 2022-01-31T22:40:21.327+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:21.327811+0000) 2022-01-31T22:40:21.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:21 smithi171 conmon[41853]: debug 2022-01-31T22:40:21.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:21.414739+0000) 2022-01-31T22:40:22.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:21 smithi167 conmon[49112]: debug 2022-01-31T22:40:21.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:21.796639+0000) 2022-01-31T22:40:22.010 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:21 smithi167 conmon[54076]: debug 2022-01-31T22:40:21.962+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:21.962886+0000) 2022-01-31T22:40:22.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:21 smithi167 conmon[60316]: debug 2022-01-31T22:40:21.721+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:21.722173+0000) 2022-01-31T22:40:22.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:22 smithi171 conmon[41853]: debug 2022-01-31T22:40:22.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:22.414949+0000) 2022-01-31T22:40:22.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:22 smithi171 conmon[46715]: debug 2022-01-31T22:40:22.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:22.464592+0000) 2022-01-31T22:40:22.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:22 smithi171 conmon[51620]: debug 2022-01-31T22:40:22.327+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:22.327983+0000) 2022-01-31T22:40:23.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:22 smithi167 conmon[49112]: debug 2022-01-31T22:40:22.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:22.796793+0000) 2022-01-31T22:40:23.010 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:22 smithi167 conmon[54076]: debug 2022-01-31T22:40:22.962+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:22.963013+0000) 2022-01-31T22:40:23.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:22 smithi167 conmon[60316]: debug 2022-01-31T22:40:22.722+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:22.722395+0000) 2022-01-31T22:40:23.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:23 smithi171 conmon[46715]: debug 2022-01-31T22:40:23.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:23.464790+0000) 2022-01-31T22:40:23.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:23 smithi171 conmon[41853]: debug 2022-01-31T22:40:23.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:23.415097+0000) 2022-01-31T22:40:23.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:23 smithi171 conmon[51620]: debug 2022-01-31T22:40:23.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:23.328200+0000) 2022-01-31T22:40:24.009 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:23 smithi167 conmon[49112]: debug 2022-01-31T22:40:23.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:23.797022+0000) 2022-01-31T22:40:24.010 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:23 smithi167 conmon[54076]: debug 2022-01-31T22:40:23.962+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:23.963188+0000) 2022-01-31T22:40:24.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:23 smithi167 conmon[60316]: debug 2022-01-31T22:40:23.722+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:23.722589+0000) 2022-01-31T22:40:24.183 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:23 smithi171 conmon[35325]: debug 2022-01-31T22:40:23.891+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:40:24.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:24 smithi167 conmon[49112]: debug 2022-01-31T22:40:24.191+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.191621+0000) 2022-01-31T22:40:24.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:24 smithi167 conmon[54076]: debug 2022-01-31T22:40:24.190+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.191217+0000) 2022-01-31T22:40:24.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:24 smithi167 conmon[60316]: debug 2022-01-31T22:40:24.190+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.191480+0000) 2022-01-31T22:40:24.440 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[35325]: debug 2022-01-31T22:40:24.213+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228273 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:24.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[46715]: debug 2022-01-31T22:40:24.190+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.191008+0000) 2022-01-31T22:40:24.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[51620]: debug 2022-01-31T22:40:24.189+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.190498+0000) 2022-01-31T22:40:24.442 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[51620]: debug 2022-01-31T22:40:24.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.328374+0000) 2022-01-31T22:40:24.443 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[41853]: debug 2022-01-31T22:40:24.191+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.192686+0000) 2022-01-31T22:40:24.443 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[41853]: debug 2022-01-31T22:40:24.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.415325+0000) 2022-01-31T22:40:24.850 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:24 smithi171 conmon[46715]: debug 2022-01-31T22:40:24.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.464973+0000) 2022-01-31T22:40:25.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:24 smithi167 conmon[49112]: debug 2022-01-31T22:40:24.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.797228+0000) 2022-01-31T22:40:25.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:24 smithi167 conmon[54076]: debug 2022-01-31T22:40:24.962+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.963391+0000) 2022-01-31T22:40:25.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:24 smithi167 conmon[60316]: debug 2022-01-31T22:40:24.722+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:24.722767+0000) 2022-01-31T22:40:25.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:25 smithi171 conmon[46715]: debug 2022-01-31T22:40:25.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:25.465135+0000) 2022-01-31T22:40:25.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:25 smithi171 conmon[51620]: debug 2022-01-31T22:40:25.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:25.328520+0000) 2022-01-31T22:40:25.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:25 smithi171 conmon[41853]: debug 2022-01-31T22:40:25.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:25.415470+0000) 2022-01-31T22:40:26.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:25 smithi167 conmon[49112]: debug 2022-01-31T22:40:25.796+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:25.797398+0000) 2022-01-31T22:40:26.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:25 smithi167 conmon[54076]: debug 2022-01-31T22:40:25.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:25.963551+0000) 2022-01-31T22:40:26.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:25 smithi167 conmon[60316]: debug 2022-01-31T22:40:25.722+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:25.722922+0000) 2022-01-31T22:40:26.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:26 smithi171 conmon[41853]: debug 2022-01-31T22:40:26.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:26.415670+0000) 2022-01-31T22:40:26.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:26 smithi171 conmon[46715]: debug 2022-01-31T22:40:26.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:26.465319+0000) 2022-01-31T22:40:26.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:26 smithi171 conmon[51620]: debug 2022-01-31T22:40:26.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:26.328776+0000) 2022-01-31T22:40:27.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:26 smithi167 conmon[49112]: debug 2022-01-31T22:40:26.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:26.797579+0000) 2022-01-31T22:40:27.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:26 smithi167 conmon[54076]: debug 2022-01-31T22:40:26.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:26.963713+0000) 2022-01-31T22:40:27.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:26 smithi167 conmon[60316]: debug 2022-01-31T22:40:26.722+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:26.723108+0000) 2022-01-31T22:40:27.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:27 smithi171 conmon[51620]: debug 2022-01-31T22:40:27.327+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:27.328995+0000) 2022-01-31T22:40:27.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:27 smithi171 conmon[41853]: debug 2022-01-31T22:40:27.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:27.415867+0000) 2022-01-31T22:40:27.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:27 smithi171 conmon[46715]: debug 2022-01-31T22:40:27.463+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:27.465523+0000) 2022-01-31T22:40:28.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:27 smithi167 conmon[49112]: debug 2022-01-31T22:40:27.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:27.797737+0000) 2022-01-31T22:40:28.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:27 smithi167 conmon[54076]: debug 2022-01-31T22:40:27.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:27.963832+0000) 2022-01-31T22:40:28.011 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:27 smithi167 conmon[60316]: debug 2022-01-31T22:40:27.722+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:27.723316+0000) 2022-01-31T22:40:28.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:28 smithi171 conmon[51620]: debug 2022-01-31T22:40:28.327+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:28.329188+0000) 2022-01-31T22:40:28.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:28 smithi171 conmon[41853]: debug 2022-01-31T22:40:28.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:28.416019+0000) 2022-01-31T22:40:28.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:28 smithi171 conmon[46715]: debug 2022-01-31T22:40:28.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:28.465737+0000) 2022-01-31T22:40:29.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:28 smithi167 conmon[49112]: debug 2022-01-31T22:40:28.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:28.797896+0000) 2022-01-31T22:40:29.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:28 smithi167 conmon[54076]: debug 2022-01-31T22:40:28.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:28.963986+0000) 2022-01-31T22:40:29.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:28 smithi167 conmon[60316]: debug 2022-01-31T22:40:28.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:28.723540+0000) 2022-01-31T22:40:29.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[49112]: debug 2022-01-31T22:40:29.217+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.218066+0000) 2022-01-31T22:40:29.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[54076]: debug 2022-01-31T22:40:29.216+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.217441+0000) 2022-01-31T22:40:29.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[60316]: debug 2022-01-31T22:40:29.414 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[60316]: 2022-01-31T22:40:29.216+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.217028+0000) 2022-01-31T22:40:29.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[35325]: debug 2022-01-31T22:40:29.239+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228386 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:29.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[41853]: debug 2022-01-31T22:40:29.217+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.218883+0000) 2022-01-31T22:40:29.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[41853]: debug 2022-01-31T22:40:29.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.416172+0000) 2022-01-31T22:40:29.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[51620]: debug 2022-01-31T22:40:29.214+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.216686+0000) 2022-01-31T22:40:29.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[51620]: debug 2022-01-31T22:40:29.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.329331+0000) 2022-01-31T22:40:29.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[46715]: debug 2022-01-31T22:40:29.215+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.217456+0000) 2022-01-31T22:40:29.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:29 smithi171 conmon[46715]: debug 2022-01-31T22:40:29.464+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.465959+0000) 2022-01-31T22:40:30.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[49112]: debug 2022-01-31T22:40:29.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.798127+0000) 2022-01-31T22:40:30.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[54076]: debug 2022-01-31T22:40:29.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.964143+0000) 2022-01-31T22:40:30.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:29 smithi167 conmon[60316]: debug 2022-01-31T22:40:29.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:29.723768+0000) 2022-01-31T22:40:30.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:30 smithi171 conmon[46715]: debug 2022-01-31T22:40:30.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:30.466106+0000) 2022-01-31T22:40:30.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:30 smithi171 conmon[41853]: debug 2022-01-31T22:40:30.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:30.416283+0000) 2022-01-31T22:40:30.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:30 smithi171 conmon[51620]: debug 2022-01-31T22:40:30.327+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:30.329509+0000) 2022-01-31T22:40:31.010 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:30 smithi167 conmon[49112]: debug 2022-01-31T22:40:30.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:30.798286+0000) 2022-01-31T22:40:31.011 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:30 smithi167 conmon[54076]: debug 2022-01-31T22:40:30.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:30.964274+0000) 2022-01-31T22:40:31.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:30 smithi167 conmon[60316]: debug 2022-01-31T22:40:30.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:30.723963+0000) 2022-01-31T22:40:31.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:31 smithi171 conmon[46715]: debug 2022-01-31T22:40:31.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:31.466268+0000) 2022-01-31T22:40:31.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:31 smithi171 conmon[41853]: debug 2022-01-31T22:40:31.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:31.416404+0000) 2022-01-31T22:40:31.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:31 smithi171 conmon[51620]: debug 2022-01-31T22:40:31.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:31.329715+0000) 2022-01-31T22:40:32.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:31 smithi167 conmon[49112]: debug 2022-01-31T22:40:31.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:31.798470+0000) 2022-01-31T22:40:32.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:31 smithi167 conmon[54076]: debug 2022-01-31T22:40:31.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:31.964468+0000) 2022-01-31T22:40:32.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:31 smithi167 conmon[60316]: debug 2022-01-31T22:40:31.724+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:31.724164+0000) 2022-01-31T22:40:32.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:32 smithi171 conmon[46715]: debug 2022-01-31T22:40:32.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:32.466472+0000) 2022-01-31T22:40:32.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:32 smithi171 conmon[51620]: debug 2022-01-31T22:40:32.329+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:32.329981+0000) 2022-01-31T22:40:32.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:32 smithi171 conmon[41853]: debug 2022-01-31T22:40:32.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:32.416585+0000) 2022-01-31T22:40:33.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:32 smithi167 conmon[49112]: debug 2022-01-31T22:40:32.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:32.798580+0000) 2022-01-31T22:40:33.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:32 smithi167 conmon[54076]: debug 2022-01-31T22:40:32.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:32.964676+0000) 2022-01-31T22:40:33.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:32 smithi167 conmon[60316]: debug 2022-01-31T22:40:32.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:32.724379+0000) 2022-01-31T22:40:33.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:33 smithi171 conmon[46715]: debug 2022-01-31T22:40:33.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:33.466630+0000) 2022-01-31T22:40:33.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:33 smithi171 conmon[41853]: debug 2022-01-31T22:40:33.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:33.416791+0000) 2022-01-31T22:40:33.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:33 smithi171 conmon[51620]: debug 2022-01-31T22:40:33.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:33.330168+0000) 2022-01-31T22:40:34.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:33 smithi167 conmon[49112]: debug 2022-01-31T22:40:33.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:33.798739+0000) 2022-01-31T22:40:34.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:33 smithi167 conmon[54076]: debug 2022-01-31T22:40:33.963+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:33.964883+0000) 2022-01-31T22:40:34.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:33 smithi167 conmon[60316]: debug 2022-01-31T22:40:33.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:33.724603+0000) 2022-01-31T22:40:34.412 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:34 smithi167 conmon[49112]: debug 2022-01-31T22:40:34.241+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.243033+0000) 2022-01-31T22:40:34.413 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:34 smithi167 conmon[54076]: debug 2022-01-31T22:40:34.241+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.243441+0000) 2022-01-31T22:40:34.413 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:34 smithi167 conmon[60316]: debug 2022-01-31T22:40:34.241+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.242871+0000) 2022-01-31T22:40:34.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[35325]: debug 2022-01-31T22:40:34.265+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228496 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:34.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[41853]: debug 2022-01-31T22:40:34.243+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.244746+0000) 2022-01-31T22:40:34.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[41853]: debug 2022-01-31T22:40:34.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.416933+0000) 2022-01-31T22:40:34.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[46715]: debug 2022-01-31T22:40:34.242+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.243802+0000) 2022-01-31T22:40:34.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[46715]: debug 2022-01-31T22:40:34.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.466874+0000) 2022-01-31T22:40:34.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[51620]: debug 2022-01-31T22:40:34.241+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.243665+0000) 2022-01-31T22:40:34.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:34 smithi171 conmon[51620]: debug 2022-01-31T22:40:34.329+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.330304+0000) 2022-01-31T22:40:35.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:34 smithi167 conmon[49112]: debug 2022-01-31T22:40:34.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.798929+0000) 2022-01-31T22:40:35.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:34 smithi167 conmon[54076]: debug 2022-01-31T22:40:34.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.965051+0000) 2022-01-31T22:40:35.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:34 smithi167 conmon[60316]: debug 2022-01-31T22:40:34.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:34.724794+0000) 2022-01-31T22:40:35.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:35 smithi171 conmon[51620]: debug 2022-01-31T22:40:35.328+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:35.330427+0000) 2022-01-31T22:40:35.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:35 smithi171 conmon[41853]: debug 2022-01-31T22:40:35.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:35.417083+0000) 2022-01-31T22:40:35.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:35 smithi171 conmon[46715]: debug 2022-01-31T22:40:35.466+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:35.467026+0000) 2022-01-31T22:40:36.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:35 smithi167 conmon[49112]: debug 2022-01-31T22:40:35.797+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:35.799088+0000) 2022-01-31T22:40:36.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:35 smithi167 conmon[54076]: debug 2022-01-31T22:40:35.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:35.965234+0000) 2022-01-31T22:40:36.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:35 smithi167 conmon[60316]: debug 2022-01-31T22:40:35.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:35.724915+0000) 2022-01-31T22:40:36.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:36 smithi171 conmon[41853]: debug 2022-01-31T22:40:36.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:36.417282+0000) 2022-01-31T22:40:36.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:36 smithi171 conmon[46715]: debug 2022-01-31T22:40:36.465+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:36.467188+0000) 2022-01-31T22:40:36.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:36 smithi171 conmon[51620]: debug 2022-01-31T22:40:36.329+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:36.330603+0000) 2022-01-31T22:40:37.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:36 smithi167 conmon[49112]: debug 2022-01-31T22:40:36.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:36.799284+0000) 2022-01-31T22:40:37.013 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:36 smithi167 conmon[54076]: debug 2022-01-31T22:40:36.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:36.965394+0000) 2022-01-31T22:40:37.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:36 smithi167 conmon[60316]: debug 2022-01-31T22:40:36.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:36.725101+0000) 2022-01-31T22:40:37.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:37 smithi171 conmon[41853]: debug 2022-01-31T22:40:37.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:37.417456+0000) 2022-01-31T22:40:37.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:37 smithi171 conmon[46715]: debug 2022-01-31T22:40:37.466+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:37.467374+0000) 2022-01-31T22:40:37.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:37 smithi171 conmon[51620]: debug 2022-01-31T22:40:37.330+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:37.330824+0000) 2022-01-31T22:40:38.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:37 smithi167 conmon[49112]: debug 2022-01-31T22:40:37.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:37.799458+0000) 2022-01-31T22:40:38.013 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:37 smithi167 conmon[54076]: debug 2022-01-31T22:40:37.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:37.965602+0000) 2022-01-31T22:40:38.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:37 smithi167 conmon[60316]: debug 2022-01-31T22:40:37.723+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:37.725313+0000) 2022-01-31T22:40:38.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:38 smithi171 conmon[41853]: debug 2022-01-31T22:40:38.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:38.417599+0000) 2022-01-31T22:40:38.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:38 smithi171 conmon[46715]: debug 2022-01-31T22:40:38.466+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:38.467528+0000) 2022-01-31T22:40:38.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:38 smithi171 conmon[51620]: debug 2022-01-31T22:40:38.330+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:38.331011+0000) 2022-01-31T22:40:39.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:38 smithi167 conmon[49112]: debug 2022-01-31T22:40:38.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:38.799661+0000) 2022-01-31T22:40:39.013 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:38 smithi167 conmon[54076]: debug 2022-01-31T22:40:38.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:38.965787+0000) 2022-01-31T22:40:39.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:38 smithi167 conmon[60316]: debug 2022-01-31T22:40:38.724+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:38.725521+0000) 2022-01-31T22:40:39.261 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:38 smithi171 conmon[35325]: debug 2022-01-31T22:40:38.891+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:40:39.262 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[51620]: debug 2022-01-31T22:40:39.268+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.268808+0000) 2022-01-31T22:40:39.263 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:39 smithi167 conmon[60316]: debug 2022-01-31T22:40:39.268+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.269540+0000) 2022-01-31T22:40:39.528 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:39 smithi167 conmon[49112]: debug 2022-01-31T22:40:39.268+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.270271+0000) 2022-01-31T22:40:39.529 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:39 smithi167 conmon[54076]: debug 2022-01-31T22:40:39.268+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.270243+0000) 2022-01-31T22:40:39.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[35325]: debug 2022-01-31T22:40:39.292+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228610 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:39.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[51620]: debug 2022-01-31T22:40:39.330+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.331216+0000) 2022-01-31T22:40:39.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[41853]: debug 2022-01-31T22:40:39.270+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.270874+0000) 2022-01-31T22:40:39.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[41853]: debug 2022-01-31T22:40:39.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.417781+0000) 2022-01-31T22:40:39.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[46715]: debug 2022-01-31T22:40:39.269+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.270447+0000) 2022-01-31T22:40:39.603 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:39 smithi171 conmon[46715]: debug 2022-01-31T22:40:39.466+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.467703+0000) 2022-01-31T22:40:39.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:39 smithi167 conmon[60316]: debug 2022-01-31T22:40:39.724+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.725737+0000) 2022-01-31T22:40:40.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:39 smithi167 conmon[49112]: debug 2022-01-31T22:40:39.798+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.799829+0000) 2022-01-31T22:40:40.163 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:39 smithi167 conmon[54076]: debug 2022-01-31T22:40:39.964+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:39.965916+0000) 2022-01-31T22:40:40.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:40 smithi171 conmon[41853]: debug 2022-01-31T22:40:40.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:40.417912+0000) 2022-01-31T22:40:40.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:40 smithi171 conmon[46715]: debug 2022-01-31T22:40:40.467+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:40.467840+0000) 2022-01-31T22:40:40.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:40 smithi171 conmon[51620]: debug 2022-01-31T22:40:40.330+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:40.331429+0000) 2022-01-31T22:40:41.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:40 smithi167 conmon[49112]: debug 2022-01-31T22:40:40.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:40.799956+0000) 2022-01-31T22:40:41.013 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:40 smithi167 conmon[54076]: debug 2022-01-31T22:40:40.965+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:40.966109+0000) 2022-01-31T22:40:41.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:40 smithi167 conmon[60316]: debug 2022-01-31T22:40:40.724+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:40.725889+0000) 2022-01-31T22:40:41.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:41 smithi171 conmon[46715]: debug 2022-01-31T22:40:41.467+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:41.467942+0000) 2022-01-31T22:40:41.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:41 smithi171 conmon[41853]: debug 2022-01-31T22:40:41.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:41.418067+0000) 2022-01-31T22:40:41.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:41 smithi171 conmon[51620]: debug 2022-01-31T22:40:41.330+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:41.331607+0000) 2022-01-31T22:40:42.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:41 smithi167 conmon[49112]: debug 2022-01-31T22:40:41.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:41.800110+0000) 2022-01-31T22:40:42.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:41 smithi167 conmon[60316]: debug 2022-01-31T22:40:41.724+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:41.726101+0000) 2022-01-31T22:40:42.014 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:41 smithi167 conmon[54076]: debug 2022-01-31T22:40:41.965+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:41.966318+0000) 2022-01-31T22:40:42.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:42 smithi171 conmon[41853]: debug 2022-01-31T22:40:42.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:42.418275+0000) 2022-01-31T22:40:42.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:42 smithi171 conmon[46715]: debug 2022-01-31T22:40:42.467+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:42.468056+0000) 2022-01-31T22:40:42.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:42 smithi171 conmon[51620]: debug 2022-01-31T22:40:42.331+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:42.331792+0000) 2022-01-31T22:40:43.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:42 smithi167 conmon[49112]: debug 2022-01-31T22:40:42.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:42.800270+0000) 2022-01-31T22:40:43.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:42 smithi167 conmon[60316]: debug 2022-01-31T22:40:42.725+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:42.726332+0000) 2022-01-31T22:40:43.014 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:42 smithi167 conmon[54076]: debug 2022-01-31T22:40:42.965+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:42.966515+0000) 2022-01-31T22:40:43.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:43 smithi171 conmon[41853]: debug 2022-01-31T22:40:43.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:43.418431+0000) 2022-01-31T22:40:43.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:43 smithi171 conmon[46715]: debug 2022-01-31T22:40:43.467+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:43.468239+0000) 2022-01-31T22:40:43.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:43 smithi171 conmon[51620]: debug 2022-01-31T22:40:43.331+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:43.331946+0000) 2022-01-31T22:40:44.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:43 smithi167 conmon[49112]: debug 2022-01-31T22:40:43.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:43.800459+0000) 2022-01-31T22:40:44.014 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:43 smithi167 conmon[54076]: debug 2022-01-31T22:40:43.965+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:43.966646+0000) 2022-01-31T22:40:44.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:43 smithi167 conmon[60316]: debug 2022-01-31T22:40:43.725+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:43.726525+0000) 2022-01-31T22:40:44.544 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:44 smithi167 conmon[49112]: debug 2022-01-31T22:40:44.294+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.296391+0000) 2022-01-31T22:40:44.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:44 smithi167 conmon[54076]: debug 2022-01-31T22:40:44.295+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.296863+0000) 2022-01-31T22:40:44.546 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:44 smithi167 conmon[60316]: debug 2022-01-31T22:40:44.295+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.296549+0000) 2022-01-31T22:40:44.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[35325]: debug 2022-01-31T22:40:44.318+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228735 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:44.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[46715]: debug 2022-01-31T22:40:44.294+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.295664+0000) 2022-01-31T22:40:44.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[46715]: debug 2022-01-31T22:40:44.467+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.468431+0000) 2022-01-31T22:40:44.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[41853]: debug 2022-01-31T22:40:44.295+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.296580+0000) 2022-01-31T22:40:44.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[41853]: debug 2022-01-31T22:40:44.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.418583+0000) 2022-01-31T22:40:44.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[51620]: debug 2022-01-31T22:40:44.295+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.296096+0000) 2022-01-31T22:40:44.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:44 smithi171 conmon[51620]: debug 2022-01-31T22:40:44.331+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.332097+0000) 2022-01-31T22:40:44.912 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:44 smithi167 conmon[60316]: debug 2022-01-31T22:40:44.725+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.726716+0000) 2022-01-31T22:40:44.912 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:44 smithi167 conmon[49112]: debug 2022-01-31T22:40:44.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.800688+0000) 2022-01-31T22:40:45.333 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:44 smithi167 conmon[54076]: debug 2022-01-31T22:40:44.966+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:44.966803+0000) 2022-01-31T22:40:45.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:45 smithi171 conmon[46715]: debug 2022-01-31T22:40:45.467+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:45.468583+0000) 2022-01-31T22:40:45.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:45 smithi171 conmon[51620]: debug 2022-01-31T22:40:45.331+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:45.332197+0000) 2022-01-31T22:40:45.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:45 smithi171 conmon[41853]: debug 2022-01-31T22:40:45.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:45.418730+0000) 2022-01-31T22:40:46.013 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:45 smithi167 conmon[49112]: debug 2022-01-31T22:40:45.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:45.800839+0000) 2022-01-31T22:40:46.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:45 smithi167 conmon[60316]: debug 2022-01-31T22:40:45.726+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:45.726873+0000) 2022-01-31T22:40:46.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:45 smithi167 conmon[54076]: debug 2022-01-31T22:40:45.966+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:45.966968+0000) 2022-01-31T22:40:46.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:46 smithi171 conmon[46715]: debug 2022-01-31T22:40:46.468+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:46.468788+0000) 2022-01-31T22:40:46.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:46 smithi171 conmon[41853]: debug 2022-01-31T22:40:46.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:46.418934+0000) 2022-01-31T22:40:46.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:46 smithi171 conmon[51620]: debug 2022-01-31T22:40:46.331+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:46.332384+0000) 2022-01-31T22:40:47.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:46 smithi167 conmon[49112]: debug 2022-01-31T22:40:46.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:46.801024+0000) 2022-01-31T22:40:47.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:46 smithi167 conmon[60316]: debug 2022-01-31T22:40:46.726+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:46.727059+0000) 2022-01-31T22:40:47.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:46 smithi167 conmon[54076]: debug 2022-01-31T22:40:46.966+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:46.967126+0000) 2022-01-31T22:40:47.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:47 smithi171 conmon[46715]: debug 2022-01-31T22:40:47.468+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:47.468964+0000) 2022-01-31T22:40:47.739 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:47 smithi171 conmon[41853]: debug 2022-01-31T22:40:47.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:47.419129+0000) 2022-01-31T22:40:47.739 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:47 smithi171 conmon[51620]: debug 2022-01-31T22:40:47.331+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:47.332592+0000) 2022-01-31T22:40:48.014 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:47 smithi167 conmon[54076]: debug 2022-01-31T22:40:47.966+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:47.967317+0000) 2022-01-31T22:40:48.015 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:47 smithi167 conmon[60316]: debug 2022-01-31T22:40:47.725+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:47.727269+0000) 2022-01-31T22:40:48.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:47 smithi167 conmon[49112]: debug 2022-01-31T22:40:47.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:47.801212+0000) 2022-01-31T22:40:48.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:48 smithi171 conmon[41853]: debug 2022-01-31T22:40:48.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:48.419277+0000) 2022-01-31T22:40:48.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:48 smithi171 conmon[46715]: debug 2022-01-31T22:40:48.468+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:48.469113+0000) 2022-01-31T22:40:48.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:48 smithi171 conmon[51620]: debug 2022-01-31T22:40:48.332+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:48.332776+0000) 2022-01-31T22:40:49.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:48 smithi167 conmon[60316]: debug 2022-01-31T22:40:48.726+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:48.727438+0000) 2022-01-31T22:40:49.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:48 smithi167 conmon[49112]: debug 2022-01-31T22:40:48.799+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:48.801391+0000) 2022-01-31T22:40:49.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:48 smithi167 conmon[54076]: debug 2022-01-31T22:40:48.966+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:48.967539+0000) 2022-01-31T22:40:49.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[35325]: debug 2022-01-31T22:40:49.344+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228844 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:49.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[41853]: debug 2022-01-31T22:40:49.322+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.322773+0000) 2022-01-31T22:40:49.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[41853]: debug 2022-01-31T22:40:49.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.419446+0000) 2022-01-31T22:40:49.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[46715]: debug 2022-01-31T22:40:49.321+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.322001+0000) 2022-01-31T22:40:49.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[46715]: debug 2022-01-31T22:40:49.468+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.469321+0000) 2022-01-31T22:40:49.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[51620]: debug 2022-01-31T22:40:49.320+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.321731+0000) 2022-01-31T22:40:49.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:49 smithi171 conmon[51620]: debug 2022-01-31T22:40:49.332+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.332940+0000) 2022-01-31T22:40:49.662 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:49 smithi167 conmon[49112]: debug 2022-01-31T22:40:49.321+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.323487+0000) 2022-01-31T22:40:49.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:49 smithi167 conmon[54076]: debug 2022-01-31T22:40:49.320+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.321915+0000) 2022-01-31T22:40:49.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:49 smithi167 conmon[60316]: debug 2022-01-31T22:40:49.321+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.322632+0000) 2022-01-31T22:40:50.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:49 smithi167 conmon[49112]: debug 2022-01-31T22:40:49.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.801562+0000) 2022-01-31T22:40:50.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:49 smithi167 conmon[54076]: debug 2022-01-31T22:40:49.967+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.967716+0000) 2022-01-31T22:40:50.015 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:49 smithi167 conmon[60316]: debug 2022-01-31T22:40:49.726+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:49.727560+0000) 2022-01-31T22:40:50.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:50 smithi171 conmon[41853]: debug 2022-01-31T22:40:50.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:50.419586+0000) 2022-01-31T22:40:50.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:50 smithi171 conmon[46715]: debug 2022-01-31T22:40:50.468+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:50.469456+0000) 2022-01-31T22:40:50.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:50 smithi171 conmon[51620]: debug 2022-01-31T22:40:50.332+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:50.333096+0000) 2022-01-31T22:40:51.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:50 smithi167 conmon[49112]: debug 2022-01-31T22:40:50.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:50.801693+0000) 2022-01-31T22:40:51.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:50 smithi167 conmon[54076]: debug 2022-01-31T22:40:50.967+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:50.967870+0000) 2022-01-31T22:40:51.015 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:50 smithi167 conmon[60316]: debug 2022-01-31T22:40:50.727+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:50.727714+0000) 2022-01-31T22:40:51.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:51 smithi171 conmon[46715]: debug 2022-01-31T22:40:51.468+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:51.469660+0000) 2022-01-31T22:40:51.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:51 smithi171 conmon[41853]: debug 2022-01-31T22:40:51.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:51.419742+0000) 2022-01-31T22:40:51.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:51 smithi171 conmon[51620]: debug 2022-01-31T22:40:51.332+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:51.333288+0000) 2022-01-31T22:40:52.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:51 smithi167 conmon[49112]: debug 2022-01-31T22:40:51.800+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:51.801838+0000) 2022-01-31T22:40:52.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:51 smithi167 conmon[54076]: debug 2022-01-31T22:40:51.966+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:51.968050+0000) 2022-01-31T22:40:52.015 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:51 smithi167 conmon[60316]: debug 2022-01-31T22:40:51.727+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:51.727886+0000) 2022-01-31T22:40:52.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:52 smithi171 conmon[41853]: debug 2022-01-31T22:40:52.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:52.419906+0000) 2022-01-31T22:40:52.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:52 smithi171 conmon[46715]: debug 2022-01-31T22:40:52.469+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:52.469824+0000) 2022-01-31T22:40:52.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:52 smithi171 conmon[51620]: debug 2022-01-31T22:40:52.332+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:52.333505+0000) 2022-01-31T22:40:53.014 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:52 smithi167 conmon[49112]: debug 2022-01-31T22:40:52.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:52.801995+0000) 2022-01-31T22:40:53.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:52 smithi167 conmon[54076]: debug 2022-01-31T22:40:52.967+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:52.968228+0000) 2022-01-31T22:40:53.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:52 smithi167 conmon[60316]: debug 2022-01-31T22:40:52.727+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:52.728041+0000) 2022-01-31T22:40:53.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:53 smithi171 conmon[41853]: debug 2022-01-31T22:40:53.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:53.420048+0000) 2022-01-31T22:40:53.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:53 smithi171 conmon[46715]: debug 2022-01-31T22:40:53.469+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:53.470020+0000) 2022-01-31T22:40:53.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:53 smithi171 conmon[51620]: debug 2022-01-31T22:40:53.332+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:53.333683+0000) 2022-01-31T22:40:54.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:53 smithi167 conmon[49112]: debug 2022-01-31T22:40:53.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:53.802181+0000) 2022-01-31T22:40:54.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:53 smithi167 conmon[54076]: debug 2022-01-31T22:40:53.967+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:53.968411+0000) 2022-01-31T22:40:54.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:53 smithi167 conmon[60316]: debug 2022-01-31T22:40:53.727+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:53.728270+0000) 2022-01-31T22:40:54.326 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:53 smithi171 conmon[35325]: debug 2022-01-31T22:40:53.892+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:40:54.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[41853]: debug 2022-01-31T22:40:54.348+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.349678+0000) 2022-01-31T22:40:54.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[41853]: debug 2022-01-31T22:40:54.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.420157+0000) 2022-01-31T22:40:54.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[46715]: debug 2022-01-31T22:40:54.346+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.347412+0000) 2022-01-31T22:40:54.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[46715]: debug 2022-01-31T22:40:54.469+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.470212+0000) 2022-01-31T22:40:54.602 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[35325]: debug 2022-01-31T22:40:54.370+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 228954 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:54.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[51620]: debug 2022-01-31T22:40:54.333+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.333855+0000) 2022-01-31T22:40:54.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:54 smithi171 conmon[51620]: debug 2022-01-31T22:40:54.347+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.347859+0000) 2022-01-31T22:40:54.662 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:54 smithi167 conmon[60316]: debug 2022-01-31T22:40:54.348+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.350123+0000) 2022-01-31T22:40:54.663 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:54 smithi167 conmon[49112]: debug 2022-01-31T22:40:54.347+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.349296+0000) 2022-01-31T22:40:54.663 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:54 smithi167 conmon[54076]: debug 2022-01-31T22:40:54.347+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.348992+0000) 2022-01-31T22:40:55.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:54 smithi167 conmon[49112]: debug 2022-01-31T22:40:54.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.802361+0000) 2022-01-31T22:40:55.015 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:54 smithi167 conmon[54076]: debug 2022-01-31T22:40:54.968+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.968578+0000) 2022-01-31T22:40:55.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:54 smithi167 conmon[60316]: debug 2022-01-31T22:40:54.727+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:54.728516+0000) 2022-01-31T22:40:55.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:55 smithi171 conmon[41853]: debug 2022-01-31T22:40:55.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:55.420303+0000) 2022-01-31T22:40:55.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:55 smithi171 conmon[46715]: debug 2022-01-31T22:40:55.469+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:55.470405+0000) 2022-01-31T22:40:55.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:55 smithi171 conmon[51620]: debug 2022-01-31T22:40:55.333+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:55.334067+0000) 2022-01-31T22:40:56.015 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:55 smithi167 conmon[60316]: debug 2022-01-31T22:40:55.728+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:55.728674+0000) 2022-01-31T22:40:56.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:55 smithi167 conmon[49112]: debug 2022-01-31T22:40:55.801+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:55.802531+0000) 2022-01-31T22:40:56.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:55 smithi167 conmon[54076]: debug 2022-01-31T22:40:55.968+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:55.968749+0000) 2022-01-31T22:40:56.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:56 smithi171 conmon[41853]: debug 2022-01-31T22:40:56.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:56.420510+0000) 2022-01-31T22:40:56.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:56 smithi171 conmon[46715]: debug 2022-01-31T22:40:56.470+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:56.470560+0000) 2022-01-31T22:40:56.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:56 smithi171 conmon[51620]: debug 2022-01-31T22:40:56.333+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:56.334214+0000) 2022-01-31T22:40:57.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:56 smithi167 conmon[49112]: debug 2022-01-31T22:40:56.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:56.802718+0000) 2022-01-31T22:40:57.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:56 smithi167 conmon[54076]: debug 2022-01-31T22:40:56.968+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:56.968932+0000) 2022-01-31T22:40:57.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:56 smithi167 conmon[60316]: debug 2022-01-31T22:40:56.728+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:56.728821+0000) 2022-01-31T22:40:57.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:57 smithi171 conmon[41853]: debug 2022-01-31T22:40:57.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:57.420730+0000) 2022-01-31T22:40:57.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:57 smithi171 conmon[46715]: debug 2022-01-31T22:40:57.470+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:57.470745+0000) 2022-01-31T22:40:57.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:57 smithi171 conmon[51620]: debug 2022-01-31T22:40:57.333+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:57.334451+0000) 2022-01-31T22:40:58.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:57 smithi167 conmon[49112]: debug 2022-01-31T22:40:57.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:57.802914+0000) 2022-01-31T22:40:58.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:57 smithi167 conmon[60316]: debug 2022-01-31T22:40:57.728+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:57.729033+0000) 2022-01-31T22:40:58.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:57 smithi167 conmon[54076]: debug 2022-01-31T22:40:57.968+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:57.969115+0000) 2022-01-31T22:40:58.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:58 smithi171 conmon[51620]: debug 2022-01-31T22:40:58.334+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:58.334649+0000) 2022-01-31T22:40:58.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:58 smithi171 conmon[41853]: debug 2022-01-31T22:40:58.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:58.420946+0000) 2022-01-31T22:40:58.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:58 smithi171 conmon[46715]: debug 2022-01-31T22:40:58.470+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:58.470901+0000) 2022-01-31T22:40:59.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:58 smithi167 conmon[49112]: debug 2022-01-31T22:40:58.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:58.803099+0000) 2022-01-31T22:40:59.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:58 smithi167 conmon[54076]: debug 2022-01-31T22:40:58.968+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:58.969302+0000) 2022-01-31T22:40:59.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:58 smithi167 conmon[60316]: debug 2022-01-31T22:40:58.728+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:58.729225+0000) 2022-01-31T22:40:59.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[35325]: debug 2022-01-31T22:40:59.396+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229068 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:40:59.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[41853]: debug 2022-01-31T22:40:59.374+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.375210+0000) 2022-01-31T22:40:59.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[41853]: debug 2022-01-31T22:40:59.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.421121+0000) 2022-01-31T22:40:59.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[46715]: debug 2022-01-31T22:40:59.372+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.373232+0000) 2022-01-31T22:40:59.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[46715]: debug 2022-01-31T22:40:59.470+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.471020+0000) 2022-01-31T22:40:59.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[51620]: debug 2022-01-31T22:40:59.334+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.334874+0000) 2022-01-31T22:40:59.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:40:59 smithi171 conmon[51620]: debug 2022-01-31T22:40:59.374+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.375177+0000) 2022-01-31T22:40:59.662 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:59 smithi167 conmon[49112]: debug 2022-01-31T22:40:59.374+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.374819+0000) 2022-01-31T22:40:59.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:59 smithi167 conmon[54076]: debug 2022-01-31T22:40:59.374+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.374756+0000) 2022-01-31T22:40:59.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:59 smithi167 conmon[60316]: debug 2022-01-31T22:40:59.374+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.374893+0000) 2022-01-31T22:41:00.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:40:59 smithi167 conmon[54076]: debug 2022-01-31T22:40:59.969+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.969486+0000) 2022-01-31T22:41:00.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:40:59 smithi167 conmon[60316]: debug 2022-01-31T22:40:59.728+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.729421+0000) 2022-01-31T22:41:00.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:40:59 smithi167 conmon[49112]: debug 2022-01-31T22:40:59.802+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:40:59.803316+0000) 2022-01-31T22:41:00.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:00 smithi171 conmon[46715]: debug 2022-01-31T22:41:00.470+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:00.471136+0000) 2022-01-31T22:41:00.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:00 smithi171 conmon[41853]: debug 2022-01-31T22:41:00.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:00.421264+0000) 2022-01-31T22:41:00.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:00 smithi171 conmon[51620]: debug 2022-01-31T22:41:00.334+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:00.335111+0000) 2022-01-31T22:41:01.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:00 smithi167 conmon[49112]: debug 2022-01-31T22:41:00.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:00.803490+0000) 2022-01-31T22:41:01.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:00 smithi167 conmon[54076]: debug 2022-01-31T22:41:00.969+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:00.969623+0000) 2022-01-31T22:41:01.017 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:00 smithi167 conmon[60316]: debug 2022-01-31T22:41:00.729+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:00.729575+0000) 2022-01-31T22:41:01.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:01 smithi171 conmon[41853]: debug 2022-01-31T22:41:01.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:01.421452+0000) 2022-01-31T22:41:01.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:01 smithi171 conmon[46715]: debug 2022-01-31T22:41:01.470+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:01.471351+0000) 2022-01-31T22:41:01.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:01 smithi171 conmon[51620]: debug 2022-01-31T22:41:01.334+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:01.335303+0000) 2022-01-31T22:41:02.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:01 smithi167 conmon[49112]: debug 2022-01-31T22:41:01.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:01.803685+0000) 2022-01-31T22:41:02.017 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:01 smithi167 conmon[54076]: debug 2022-01-31T22:41:01.969+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:01.969773+0000) 2022-01-31T22:41:02.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:01 smithi167 conmon[60316]: debug 2022-01-31T22:41:01.729+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:01.729793+0000) 2022-01-31T22:41:02.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:02 smithi171 conmon[46715]: debug 2022-01-31T22:41:02.471+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:02.471538+0000) 2022-01-31T22:41:02.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:02 smithi171 conmon[51620]: debug 2022-01-31T22:41:02.335+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:02.335517+0000) 2022-01-31T22:41:02.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:02 smithi171 conmon[41853]: debug 2022-01-31T22:41:02.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:02.421586+0000) 2022-01-31T22:41:03.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:02 smithi167 conmon[49112]: debug 2022-01-31T22:41:02.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:02.803876+0000) 2022-01-31T22:41:03.017 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:02 smithi167 conmon[54076]: debug 2022-01-31T22:41:02.969+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:02.969927+0000) 2022-01-31T22:41:03.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:02 smithi167 conmon[60316]: debug 2022-01-31T22:41:02.729+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:02.729958+0000) 2022-01-31T22:41:03.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:03 smithi171 conmon[41853]: debug 2022-01-31T22:41:03.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:03.421808+0000) 2022-01-31T22:41:03.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:03 smithi171 conmon[46715]: debug 2022-01-31T22:41:03.471+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:03.471718+0000) 2022-01-31T22:41:03.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:03 smithi171 conmon[51620]: debug 2022-01-31T22:41:03.335+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:03.335723+0000) 2022-01-31T22:41:04.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:03 smithi167 conmon[49112]: debug 2022-01-31T22:41:03.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:03.804080+0000) 2022-01-31T22:41:04.017 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:03 smithi167 conmon[54076]: debug 2022-01-31T22:41:03.969+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:03.970081+0000) 2022-01-31T22:41:04.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:03 smithi167 conmon[60316]: debug 2022-01-31T22:41:03.729+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:03.730147+0000) 2022-01-31T22:41:04.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[35325]: debug 2022-01-31T22:41:04.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229179 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:04.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[46715]: debug 2022-01-31T22:41:04.398+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.398906+0000) 2022-01-31T22:41:04.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[46715]: debug 2022-01-31T22:41:04.471+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.471876+0000) 2022-01-31T22:41:04.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[51620]: debug 2022-01-31T22:41:04.335+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.335886+0000) 2022-01-31T22:41:04.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[51620]: debug 2022-01-31T22:41:04.399+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.399908+0000) 2022-01-31T22:41:04.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[41853]: debug 2022-01-31T22:41:04.399+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.400227+0000) 2022-01-31T22:41:04.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:04 smithi171 conmon[41853]: debug 2022-01-31T22:41:04.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.421954+0000) 2022-01-31T22:41:04.661 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:04 smithi167 conmon[49112]: debug 2022-01-31T22:41:04.398+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.399456+0000) 2022-01-31T22:41:04.662 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:04 smithi167 conmon[54076]: debug 2022-01-31T22:41:04.398+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.399418+0000) 2022-01-31T22:41:04.663 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:04 smithi167 conmon[60316]: debug 2022-01-31T22:41:04.399+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.400411+0000) 2022-01-31T22:41:05.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:04 smithi167 conmon[49112]: debug 2022-01-31T22:41:04.803+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.804274+0000) 2022-01-31T22:41:05.017 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:04 smithi167 conmon[54076]: debug 2022-01-31T22:41:04.969+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.970283+0000) 2022-01-31T22:41:05.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:04 smithi167 conmon[60316]: debug 2022-01-31T22:41:04.729+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:04.730365+0000) 2022-01-31T22:41:05.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:05 smithi171 conmon[41853]: debug 2022-01-31T22:41:05.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:05.422098+0000) 2022-01-31T22:41:05.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:05 smithi171 conmon[46715]: debug 2022-01-31T22:41:05.471+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:05.472034+0000) 2022-01-31T22:41:05.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:05 smithi171 conmon[51620]: debug 2022-01-31T22:41:05.335+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:05.336069+0000) 2022-01-31T22:41:06.016 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:05 smithi167 conmon[49112]: debug 2022-01-31T22:41:05.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:05.804428+0000) 2022-01-31T22:41:06.017 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:05 smithi167 conmon[54076]: debug 2022-01-31T22:41:05.970+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:05.970429+0000) 2022-01-31T22:41:06.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:05 smithi167 conmon[60316]: debug 2022-01-31T22:41:05.730+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:05.730496+0000) 2022-01-31T22:41:06.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:06 smithi171 conmon[46715]: debug 2022-01-31T22:41:06.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:06.472236+0000) 2022-01-31T22:41:06.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:06 smithi171 conmon[41853]: debug 2022-01-31T22:41:06.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:06.422258+0000) 2022-01-31T22:41:06.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:06 smithi171 conmon[51620]: debug 2022-01-31T22:41:06.336+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:06.336190+0000) 2022-01-31T22:41:07.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:06 smithi167 conmon[49112]: debug 2022-01-31T22:41:06.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:06.804616+0000) 2022-01-31T22:41:07.017 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:06 smithi167 conmon[60316]: debug 2022-01-31T22:41:06.730+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:06.730704+0000) 2022-01-31T22:41:07.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:06 smithi167 conmon[54076]: debug 2022-01-31T22:41:06.970+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:06.970528+0000) 2022-01-31T22:41:07.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:07 smithi171 conmon[46715]: debug 2022-01-31T22:41:07.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:07.472418+0000) 2022-01-31T22:41:07.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:07 smithi171 conmon[51620]: debug 2022-01-31T22:41:07.335+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:07.336385+0000) 2022-01-31T22:41:07.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:07 smithi171 conmon[41853]: debug 2022-01-31T22:41:07.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:07.422425+0000) 2022-01-31T22:41:08.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:07 smithi167 conmon[49112]: debug 2022-01-31T22:41:07.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:07.804825+0000) 2022-01-31T22:41:08.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:07 smithi167 conmon[60316]: debug 2022-01-31T22:41:07.730+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:07.730899+0000) 2022-01-31T22:41:08.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:07 smithi167 conmon[54076]: debug 2022-01-31T22:41:07.970+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:07.970682+0000) 2022-01-31T22:41:08.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:08 smithi171 conmon[46715]: debug 2022-01-31T22:41:08.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:08.472576+0000) 2022-01-31T22:41:08.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:08 smithi171 conmon[51620]: debug 2022-01-31T22:41:08.335+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:08.336610+0000) 2022-01-31T22:41:08.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:08 smithi171 conmon[41853]: debug 2022-01-31T22:41:08.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:08.422603+0000) 2022-01-31T22:41:09.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:08 smithi167 conmon[49112]: debug 2022-01-31T22:41:08.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:08.804996+0000) 2022-01-31T22:41:09.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:08 smithi167 conmon[54076]: debug 2022-01-31T22:41:08.970+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:08.970832+0000) 2022-01-31T22:41:09.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:08 smithi167 conmon[60316]: debug 2022-01-31T22:41:08.730+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:08.731070+0000) 2022-01-31T22:41:09.329 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:08 smithi171 conmon[35325]: debug 2022-01-31T22:41:08.893+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:41:09.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[35325]: debug 2022-01-31T22:41:09.454+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229292 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:09.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[46715]: debug 2022-01-31T22:41:09.424+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.425285+0000) 2022-01-31T22:41:09.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[46715]: debug 2022-01-31T22:41:09.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.472738+0000) 2022-01-31T22:41:09.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[51620]: debug 2022-01-31T22:41:09.336+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.336849+0000) 2022-01-31T22:41:09.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[51620]: debug 2022-01-31T22:41:09.425+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.425908+0000) 2022-01-31T22:41:09.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[41853]: debug 2022-01-31T22:41:09.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.422760+0000) 2022-01-31T22:41:09.603 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:09 smithi171 conmon[41853]: debug 2022-01-31T22:41:09.425+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.426548+0000) 2022-01-31T22:41:09.723 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[49112]: debug 2022-01-31T22:41:09.425+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.425696+0000) 2022-01-31T22:41:09.724 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[60316]: debug 2022-01-31T22:41:09.425+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.426408+0000) 2022-01-31T22:41:09.725 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[54076]: debug 2022-01-31T22:41:09.424+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.425469+0000) 2022-01-31T22:41:09.725 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[54076]: 2022-01-31T22:41:10.017 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[60316]: debug 2022-01-31T22:41:09.730+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.731257+0000) 2022-01-31T22:41:10.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[49112]: debug 2022-01-31T22:41:09.804+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.805159+0000) 2022-01-31T22:41:10.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:09 smithi167 conmon[54076]: debug 2022-01-31T22:41:09.970+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:09.971007+0000) 2022-01-31T22:41:10.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:10 smithi171 conmon[41853]: debug 2022-01-31T22:41:10.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:10.422890+0000) 2022-01-31T22:41:10.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:10 smithi171 conmon[46715]: debug 2022-01-31T22:41:10.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:10.472903+0000) 2022-01-31T22:41:10.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:10 smithi171 conmon[51620]: debug 2022-01-31T22:41:10.336+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:10.337028+0000) 2022-01-31T22:41:11.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:10 smithi167 conmon[49112]: debug 2022-01-31T22:41:10.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:10.805315+0000) 2022-01-31T22:41:11.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:10 smithi167 conmon[54076]: debug 2022-01-31T22:41:10.970+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:10.971217+0000) 2022-01-31T22:41:11.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:10 smithi167 conmon[60316]: debug 2022-01-31T22:41:10.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:10.731418+0000) 2022-01-31T22:41:11.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:11 smithi171 conmon[51620]: debug 2022-01-31T22:41:11.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:11.337193+0000) 2022-01-31T22:41:11.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:11 smithi171 conmon[41853]: debug 2022-01-31T22:41:11.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:11.423036+0000) 2022-01-31T22:41:11.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:11 smithi171 conmon[46715]: debug 2022-01-31T22:41:11.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:11.473073+0000) 2022-01-31T22:41:12.017 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:11 smithi167 conmon[49112]: debug 2022-01-31T22:41:11.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:11.805518+0000) 2022-01-31T22:41:12.018 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:11 smithi167 conmon[54076]: debug 2022-01-31T22:41:11.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:11.971420+0000) 2022-01-31T22:41:12.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:11 smithi167 conmon[60316]: debug 2022-01-31T22:41:11.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:11.731574+0000) 2022-01-31T22:41:12.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:12 smithi171 conmon[51620]: debug 2022-01-31T22:41:12.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:12.337402+0000) 2022-01-31T22:41:12.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:12 smithi171 conmon[41853]: debug 2022-01-31T22:41:12.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:12.423161+0000) 2022-01-31T22:41:12.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:12 smithi171 conmon[46715]: debug 2022-01-31T22:41:12.473+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:12.473254+0000) 2022-01-31T22:41:13.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:12 smithi167 conmon[49112]: debug 2022-01-31T22:41:12.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:12.805672+0000) 2022-01-31T22:41:13.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:12 smithi167 conmon[54076]: debug 2022-01-31T22:41:12.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:12.971594+0000) 2022-01-31T22:41:13.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:12 smithi167 conmon[60316]: debug 2022-01-31T22:41:12.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:12.731750+0000) 2022-01-31T22:41:13.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:13 smithi171 conmon[41853]: debug 2022-01-31T22:41:13.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:13.423322+0000) 2022-01-31T22:41:13.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:13 smithi171 conmon[46715]: debug 2022-01-31T22:41:13.473+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:13.473448+0000) 2022-01-31T22:41:13.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:13 smithi171 conmon[51620]: debug 2022-01-31T22:41:13.336+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:13.337617+0000) 2022-01-31T22:41:14.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:13 smithi167 conmon[49112]: debug 2022-01-31T22:41:13.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:13.805856+0000) 2022-01-31T22:41:14.019 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:13 smithi167 conmon[54076]: debug 2022-01-31T22:41:13.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:13.971721+0000) 2022-01-31T22:41:14.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:13 smithi167 conmon[60316]: debug 2022-01-31T22:41:13.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:13.731957+0000) 2022-01-31T22:41:14.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[35325]: debug 2022-01-31T22:41:14.480+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229404 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:14.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[41853]: debug 2022-01-31T22:41:14.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.423498+0000) 2022-01-31T22:41:14.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[41853]: debug 2022-01-31T22:41:14.456+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.457814+0000) 2022-01-31T22:41:14.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[46715]: debug 2022-01-31T22:41:14.457+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.458897+0000) 2022-01-31T22:41:14.602 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[46715]: debug 2022-01-31T22:41:14.471+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.473648+0000) 2022-01-31T22:41:14.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[51620]: debug 2022-01-31T22:41:14.336+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.337815+0000) 2022-01-31T22:41:14.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:14 smithi171 conmon[51620]: debug 2022-01-31T22:41:14.456+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.458132+0000) 2022-01-31T22:41:14.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:14 smithi167 conmon[49112]: debug 2022-01-31T22:41:14.458+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.459353+0000) 2022-01-31T22:41:14.725 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:14 smithi167 conmon[54076]: debug 2022-01-31T22:41:14.458+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.459436+0000) 2022-01-31T22:41:14.725 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:14 smithi167 conmon[60316]: debug 2022-01-31T22:41:14.458+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.459076+0000) 2022-01-31T22:41:15.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:14 smithi167 conmon[49112]: debug 2022-01-31T22:41:14.805+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.806011+0000) 2022-01-31T22:41:15.019 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:14 smithi167 conmon[54076]: debug 2022-01-31T22:41:14.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.971898+0000) 2022-01-31T22:41:15.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:14 smithi167 conmon[60316]: debug 2022-01-31T22:41:14.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:14.732186+0000) 2022-01-31T22:41:15.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:15 smithi171 conmon[41853]: debug 2022-01-31T22:41:15.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:15.423671+0000) 2022-01-31T22:41:15.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:15 smithi171 conmon[46715]: debug 2022-01-31T22:41:15.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:15.473830+0000) 2022-01-31T22:41:15.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:15 smithi171 conmon[51620]: debug 2022-01-31T22:41:15.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:15.338000+0000) 2022-01-31T22:41:16.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:15 smithi167 conmon[49112]: debug 2022-01-31T22:41:15.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:15.806164+0000) 2022-01-31T22:41:16.019 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:15 smithi167 conmon[54076]: debug 2022-01-31T22:41:15.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:15.972036+0000) 2022-01-31T22:41:16.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:15 smithi167 conmon[60316]: debug 2022-01-31T22:41:15.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:15.732328+0000) 2022-01-31T22:41:16.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:16 smithi171 conmon[51620]: debug 2022-01-31T22:41:16.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:16.338178+0000) 2022-01-31T22:41:16.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:16 smithi171 conmon[41853]: debug 2022-01-31T22:41:16.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:16.423807+0000) 2022-01-31T22:41:16.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:16 smithi171 conmon[46715]: debug 2022-01-31T22:41:16.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:16.474012+0000) 2022-01-31T22:41:17.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:16 smithi167 conmon[49112]: debug 2022-01-31T22:41:16.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:16.806369+0000) 2022-01-31T22:41:17.019 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:16 smithi167 conmon[54076]: debug 2022-01-31T22:41:16.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:16.972195+0000) 2022-01-31T22:41:17.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:16 smithi167 conmon[60316]: debug 2022-01-31T22:41:16.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:16.732489+0000) 2022-01-31T22:41:17.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:17 smithi171 conmon[41853]: debug 2022-01-31T22:41:17.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:17.423929+0000) 2022-01-31T22:41:17.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:17 smithi171 conmon[46715]: debug 2022-01-31T22:41:17.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:17.474168+0000) 2022-01-31T22:41:17.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:17 smithi171 conmon[51620]: debug 2022-01-31T22:41:17.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:17.338407+0000) 2022-01-31T22:41:18.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:17 smithi167 conmon[49112]: debug 2022-01-31T22:41:17.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:17.806606+0000) 2022-01-31T22:41:18.019 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:17 smithi167 conmon[54076]: debug 2022-01-31T22:41:17.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:17.972392+0000) 2022-01-31T22:41:18.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:17 smithi167 conmon[60316]: debug 2022-01-31T22:41:17.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:17.732684+0000) 2022-01-31T22:41:18.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:18 smithi171 conmon[46715]: debug 2022-01-31T22:41:18.473+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:18.474304+0000) 2022-01-31T22:41:18.600 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:18 smithi171 conmon[41853]: debug 2022-01-31T22:41:18.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:18.424072+0000) 2022-01-31T22:41:18.601 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:18 smithi171 conmon[51620]: debug 2022-01-31T22:41:18.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:18.338591+0000) 2022-01-31T22:41:19.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:18 smithi167 conmon[60316]: debug 2022-01-31T22:41:18.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:18.732833+0000) 2022-01-31T22:41:19.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:18 smithi167 conmon[49112]: debug 2022-01-31T22:41:18.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:18.806793+0000) 2022-01-31T22:41:19.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:18 smithi167 conmon[54076]: debug 2022-01-31T22:41:18.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:18.972588+0000) 2022-01-31T22:41:19.600 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[35325]: debug 2022-01-31T22:41:19.505+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229516 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:19.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[46715]: debug 2022-01-31T22:41:19.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.474437+0000) 2022-01-31T22:41:19.601 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[46715]: debug 2022-01-31T22:41:19.483+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.485645+0000) 2022-01-31T22:41:19.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[41853]: debug 2022-01-31T22:41:19.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.424243+0000) 2022-01-31T22:41:19.602 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[41853]: debug 2022-01-31T22:41:19.483+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.484811+0000) 2022-01-31T22:41:19.602 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[51620]: debug 2022-01-31T22:41:19.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.338812+0000) 2022-01-31T22:41:19.603 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:19 smithi171 conmon[51620]: debug 2022-01-31T22:41:19.482+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.484429+0000) 2022-01-31T22:41:19.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:19 smithi167 conmon[49112]: debug 2022-01-31T22:41:19.484+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.485358+0000) 2022-01-31T22:41:19.726 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:19 smithi167 conmon[54076]: debug 2022-01-31T22:41:19.484+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.485467+0000) 2022-01-31T22:41:19.726 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:19 smithi167 conmon[60316]: debug 2022-01-31T22:41:19.485+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.485535+0000) 2022-01-31T22:41:20.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:19 smithi167 conmon[60316]: debug 2022-01-31T22:41:19.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.733021+0000) 2022-01-31T22:41:20.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:19 smithi167 conmon[49112]: debug 2022-01-31T22:41:19.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.807013+0000) 2022-01-31T22:41:20.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:19 smithi167 conmon[54076]: debug 2022-01-31T22:41:19.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:19.972747+0000) 2022-01-31T22:41:20.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:20 smithi171 conmon[46715]: debug 2022-01-31T22:41:20.473+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:20.474586+0000) 2022-01-31T22:41:20.600 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:20 smithi171 conmon[51620]: debug 2022-01-31T22:41:20.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:20.338989+0000) 2022-01-31T22:41:20.601 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:20 smithi171 conmon[41853]: debug 2022-01-31T22:41:20.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:20.424361+0000) 2022-01-31T22:41:21.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:20 smithi167 conmon[49112]: debug 2022-01-31T22:41:20.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:20.807308+0000) 2022-01-31T22:41:21.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:20 smithi167 conmon[54076]: debug 2022-01-31T22:41:20.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:20.972932+0000) 2022-01-31T22:41:21.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:20 smithi167 conmon[60316]: debug 2022-01-31T22:41:20.731+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:20.733154+0000) 2022-01-31T22:41:21.467 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:21 smithi171 conmon[41853]: debug 2022-01-31T22:41:21.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:21.424517+0000) 2022-01-31T22:41:21.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:21 smithi171 conmon[51620]: debug 2022-01-31T22:41:21.337+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:21.339131+0000) 2022-01-31T22:41:21.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:21 smithi171 conmon[46715]: debug 2022-01-31T22:41:21.472+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:21.474724+0000) 2022-01-31T22:41:22.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:21 smithi167 conmon[49112]: debug 2022-01-31T22:41:21.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:21.807459+0000) 2022-01-31T22:41:22.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:21 smithi167 conmon[54076]: debug 2022-01-31T22:41:21.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:21.973140+0000) 2022-01-31T22:41:22.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:21 smithi167 conmon[60316]: debug 2022-01-31T22:41:21.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:21.733367+0000) 2022-01-31T22:41:22.467 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:22 smithi171 conmon[41853]: debug 2022-01-31T22:41:22.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:22.424709+0000) 2022-01-31T22:41:22.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:22 smithi171 conmon[51620]: debug 2022-01-31T22:41:22.338+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:22.339323+0000) 2022-01-31T22:41:22.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:22 smithi171 conmon[46715]: debug 2022-01-31T22:41:22.473+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:22.474897+0000) 2022-01-31T22:41:23.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:22 smithi167 conmon[49112]: debug 2022-01-31T22:41:22.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:22.807693+0000) 2022-01-31T22:41:23.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:22 smithi167 conmon[54076]: debug 2022-01-31T22:41:22.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:22.973323+0000) 2022-01-31T22:41:23.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:22 smithi167 conmon[60316]: debug 2022-01-31T22:41:22.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:22.733580+0000) 2022-01-31T22:41:23.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:23 smithi171 conmon[51620]: debug 2022-01-31T22:41:23.338+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:23.339526+0000) 2022-01-31T22:41:23.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:23 smithi171 conmon[41853]: debug 2022-01-31T22:41:23.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:23.424827+0000) 2022-01-31T22:41:23.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:23 smithi171 conmon[46715]: debug 2022-01-31T22:41:23.474+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:23.475074+0000) 2022-01-31T22:41:24.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:23 smithi167 conmon[49112]: debug 2022-01-31T22:41:23.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:23.807879+0000) 2022-01-31T22:41:24.020 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:23 smithi167 conmon[54076]: debug 2022-01-31T22:41:23.971+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:23.973461+0000) 2022-01-31T22:41:24.021 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:23 smithi167 conmon[60316]: debug 2022-01-31T22:41:23.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:23.733770+0000) 2022-01-31T22:41:24.200 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:23 smithi171 conmon[35325]: debug 2022-01-31T22:41:23.893+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:41:24.467 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[41853]: debug 2022-01-31T22:41:24.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.425012+0000) 2022-01-31T22:41:24.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[51620]: debug 2022-01-31T22:41:24.338+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.339701+0000) 2022-01-31T22:41:24.726 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:24 smithi167 conmon[60316]: debug 2022-01-31T22:41:24.508+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.509743+0000) 2022-01-31T22:41:24.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:24 smithi167 conmon[49112]: debug 2022-01-31T22:41:24.507+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.509362+0000) 2022-01-31T22:41:24.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:24 smithi167 conmon[54076]: debug 2022-01-31T22:41:24.508+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.509649+0000) 2022-01-31T22:41:24.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[35325]: debug 2022-01-31T22:41:24.531+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229627 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:24.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[41853]: debug 2022-01-31T22:41:24.507+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.509197+0000) 2022-01-31T22:41:24.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[51620]: debug 2022-01-31T22:41:24.507+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.508939+0000) 2022-01-31T22:41:24.852 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[46715]: debug 2022-01-31T22:41:24.474+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.475275+0000) 2022-01-31T22:41:24.852 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:24 smithi171 conmon[46715]: debug 2022-01-31T22:41:24.507+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.509439+0000) 2022-01-31T22:41:25.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:24 smithi167 conmon[60316]: debug 2022-01-31T22:41:24.733+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.733992+0000) 2022-01-31T22:41:25.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:24 smithi167 conmon[49112]: debug 2022-01-31T22:41:24.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.808063+0000) 2022-01-31T22:41:25.021 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:24 smithi167 conmon[54076]: debug 2022-01-31T22:41:24.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:24.973644+0000) 2022-01-31T22:41:25.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:25 smithi171 conmon[51620]: debug 2022-01-31T22:41:25.339+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:25.339935+0000) 2022-01-31T22:41:25.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:25 smithi171 conmon[41853]: debug 2022-01-31T22:41:25.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:25.425191+0000) 2022-01-31T22:41:25.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:25 smithi171 conmon[46715]: debug 2022-01-31T22:41:25.474+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:25.475440+0000) 2022-01-31T22:41:26.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:25 smithi167 conmon[60316]: debug 2022-01-31T22:41:25.732+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:25.734184+0000) 2022-01-31T22:41:26.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:25 smithi167 conmon[49112]: debug 2022-01-31T22:41:25.806+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:25.808247+0000) 2022-01-31T22:41:26.021 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:25 smithi167 conmon[54076]: debug 2022-01-31T22:41:25.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:25.973835+0000) 2022-01-31T22:41:26.467 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:26 smithi171 conmon[41853]: debug 2022-01-31T22:41:26.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:26.425316+0000) 2022-01-31T22:41:26.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:26 smithi171 conmon[51620]: debug 2022-01-31T22:41:26.339+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:26.340109+0000) 2022-01-31T22:41:26.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:26 smithi171 conmon[46715]: debug 2022-01-31T22:41:26.474+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:26.475582+0000) 2022-01-31T22:41:27.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:26 smithi167 conmon[60316]: debug 2022-01-31T22:41:26.733+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:26.734366+0000) 2022-01-31T22:41:27.020 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:26 smithi167 conmon[49112]: debug 2022-01-31T22:41:26.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:26.808408+0000) 2022-01-31T22:41:27.021 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:26 smithi167 conmon[54076]: debug 2022-01-31T22:41:26.972+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:26.974050+0000) 2022-01-31T22:41:27.467 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:27 smithi171 conmon[41853]: debug 2022-01-31T22:41:27.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:27.425507+0000) 2022-01-31T22:41:27.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:27 smithi171 conmon[51620]: debug 2022-01-31T22:41:27.339+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:27.340333+0000) 2022-01-31T22:41:27.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:27 smithi171 conmon[46715]: debug 2022-01-31T22:41:27.474+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:27.475717+0000) 2022-01-31T22:41:28.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:27 smithi167 conmon[60316]: debug 2022-01-31T22:41:27.733+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:27.734522+0000) 2022-01-31T22:41:28.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:27 smithi167 conmon[49112]: debug 2022-01-31T22:41:27.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:27.808571+0000) 2022-01-31T22:41:28.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:27 smithi167 conmon[54076]: debug 2022-01-31T22:41:27.973+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:27.974235+0000) 2022-01-31T22:41:28.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:28 smithi171 conmon[41853]: debug 2022-01-31T22:41:28.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:28.425665+0000) 2022-01-31T22:41:28.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:28 smithi171 conmon[51620]: debug 2022-01-31T22:41:28.339+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:28.340543+0000) 2022-01-31T22:41:28.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:28 smithi171 conmon[46715]: debug 2022-01-31T22:41:28.475+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:28.475840+0000) 2022-01-31T22:41:29.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:28 smithi167 conmon[60316]: debug 2022-01-31T22:41:28.733+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:28.734683+0000) 2022-01-31T22:41:29.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:28 smithi167 conmon[49112]: debug 2022-01-31T22:41:28.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:28.808720+0000) 2022-01-31T22:41:29.021 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:28 smithi167 conmon[54076]: debug 2022-01-31T22:41:28.973+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:28.974436+0000) 2022-01-31T22:41:29.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[41853]: debug 2022-01-31T22:41:29.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.425848+0000) 2022-01-31T22:41:29.468 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[51620]: debug 2022-01-31T22:41:29.340+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.340783+0000) 2022-01-31T22:41:29.726 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:29 smithi167 conmon[49112]: debug 2022-01-31T22:41:29.535+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.536509+0000) 2022-01-31T22:41:29.727 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:29 smithi167 conmon[54076]: debug 2022-01-31T22:41:29.534+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.535532+0000) 2022-01-31T22:41:29.728 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:29 smithi167 conmon[60316]: debug 2022-01-31T22:41:29.533+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.534722+0000) 2022-01-31T22:41:29.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[35325]: debug 2022-01-31T22:41:29.558+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229739 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:29.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[41853]: debug 2022-01-31T22:41:29.535+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.536347+0000) 2022-01-31T22:41:29.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[51620]: debug 2022-01-31T22:41:29.534+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.534835+0000) 2022-01-31T22:41:29.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[46715]: debug 2022-01-31T22:41:29.475+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.476028+0000) 2022-01-31T22:41:29.852 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:29 smithi171 conmon[46715]: debug 2022-01-31T22:41:29.534+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.535391+0000) 2022-01-31T22:41:30.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:29 smithi167 conmon[60316]: debug 2022-01-31T22:41:29.733+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.734843+0000) 2022-01-31T22:41:30.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:29 smithi167 conmon[49112]: debug 2022-01-31T22:41:29.807+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.808909+0000) 2022-01-31T22:41:30.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:29 smithi167 conmon[54076]: debug 2022-01-31T22:41:29.973+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:29.974594+0000) 2022-01-31T22:41:30.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:30 smithi171 conmon[41853]: debug 2022-01-31T22:41:30.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:30.425984+0000) 2022-01-31T22:41:30.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:30 smithi171 conmon[51620]: debug 2022-01-31T22:41:30.340+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:30.340960+0000) 2022-01-31T22:41:30.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:30 smithi171 conmon[46715]: debug 2022-01-31T22:41:30.475+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:30.476206+0000) 2022-01-31T22:41:31.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:30 smithi167 conmon[60316]: debug 2022-01-31T22:41:30.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:30.734977+0000) 2022-01-31T22:41:31.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:30 smithi167 conmon[49112]: debug 2022-01-31T22:41:30.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:30.809062+0000) 2022-01-31T22:41:31.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:30 smithi167 conmon[54076]: debug 2022-01-31T22:41:30.973+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:30.974718+0000) 2022-01-31T22:41:31.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:31 smithi171 conmon[41853]: debug 2022-01-31T22:41:31.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:31.426135+0000) 2022-01-31T22:41:31.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:31 smithi171 conmon[51620]: debug 2022-01-31T22:41:31.340+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:31.341136+0000) 2022-01-31T22:41:31.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:31 smithi171 conmon[46715]: debug 2022-01-31T22:41:31.475+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:31.476328+0000) 2022-01-31T22:41:32.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:31 smithi167 conmon[60316]: debug 2022-01-31T22:41:31.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:31.735136+0000) 2022-01-31T22:41:32.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:31 smithi167 conmon[49112]: debug 2022-01-31T22:41:31.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:31.809249+0000) 2022-01-31T22:41:32.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:31 smithi167 conmon[54076]: debug 2022-01-31T22:41:31.973+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:31.974889+0000) 2022-01-31T22:41:32.468 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:32 smithi171 conmon[41853]: debug 2022-01-31T22:41:32.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:32.426314+0000) 2022-01-31T22:41:32.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:32 smithi171 conmon[51620]: debug 2022-01-31T22:41:32.340+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:32.341369+0000) 2022-01-31T22:41:32.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:32 smithi171 conmon[46715]: debug 2022-01-31T22:41:32.475+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:32.476545+0000) 2022-01-31T22:41:33.021 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:32 smithi167 conmon[60316]: debug 2022-01-31T22:41:32.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:32.735334+0000) 2022-01-31T22:41:33.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:32 smithi167 conmon[49112]: debug 2022-01-31T22:41:32.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:32.809449+0000) 2022-01-31T22:41:33.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:32 smithi167 conmon[54076]: debug 2022-01-31T22:41:32.974+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:32.975065+0000) 2022-01-31T22:41:33.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:33 smithi171 conmon[41853]: debug 2022-01-31T22:41:33.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:33.426480+0000) 2022-01-31T22:41:33.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:33 smithi171 conmon[51620]: debug 2022-01-31T22:41:33.340+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:33.341596+0000) 2022-01-31T22:41:33.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:33 smithi171 conmon[46715]: debug 2022-01-31T22:41:33.475+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:33.476695+0000) 2022-01-31T22:41:34.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:33 smithi167 conmon[49112]: debug 2022-01-31T22:41:33.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:33.809644+0000) 2022-01-31T22:41:34.022 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:33 smithi167 conmon[60316]: debug 2022-01-31T22:41:33.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:33.735508+0000) 2022-01-31T22:41:34.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:33 smithi167 conmon[54076]: debug 2022-01-31T22:41:33.973+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:33.975233+0000) 2022-01-31T22:41:34.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[41853]: debug 2022-01-31T22:41:34.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.426706+0000) 2022-01-31T22:41:34.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[51620]: debug 2022-01-31T22:41:34.341+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.341791+0000) 2022-01-31T22:41:34.709 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:34 smithi167 conmon[49112]: debug 2022-01-31T22:41:34.561+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.563160+0000) 2022-01-31T22:41:34.710 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:34 smithi167 conmon[54076]: debug 2022-01-31T22:41:34.561+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.562555+0000) 2022-01-31T22:41:34.711 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:34 smithi167 conmon[60316]: debug 2022-01-31T22:41:34.560+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.562384+0000) 2022-01-31T22:41:34.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[35325]: debug 2022-01-31T22:41:34.585+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229850 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:34.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[41853]: debug 2022-01-31T22:41:34.561+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.561844+0000) 2022-01-31T22:41:34.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[51620]: debug 2022-01-31T22:41:34.560+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.561310+0000) 2022-01-31T22:41:34.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[46715]: debug 2022-01-31T22:41:34.476+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.476897+0000) 2022-01-31T22:41:34.852 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:34 smithi171 conmon[46715]: debug 2022-01-31T22:41:34.561+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.562115+0000) 2022-01-31T22:41:34.967 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:34 smithi167 conmon[60316]: debug 2022-01-31T22:41:34.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.735583+0000) 2022-01-31T22:41:34.968 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:34 smithi167 conmon[49112]: debug 2022-01-31T22:41:34.808+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.809742+0000) 2022-01-31T22:41:35.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:34 smithi167 conmon[54076]: debug 2022-01-31T22:41:34.974+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:34.975371+0000) 2022-01-31T22:41:35.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:35 smithi171 conmon[41853]: debug 2022-01-31T22:41:35.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:35.426903+0000) 2022-01-31T22:41:35.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:35 smithi171 conmon[51620]: debug 2022-01-31T22:41:35.341+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:35.341950+0000) 2022-01-31T22:41:35.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:35 smithi171 conmon[46715]: debug 2022-01-31T22:41:35.476+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:35.477083+0000) 2022-01-31T22:41:35.967 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:35 smithi167 conmon[49112]: debug 2022-01-31T22:41:35.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:35.809883+0000) 2022-01-31T22:41:35.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:35 smithi167 conmon[60316]: debug 2022-01-31T22:41:35.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:35.735711+0000) 2022-01-31T22:41:36.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:35 smithi167 conmon[54076]: debug 2022-01-31T22:41:35.974+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:35.975559+0000) 2022-01-31T22:41:36.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:36 smithi171 conmon[41853]: debug 2022-01-31T22:41:36.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:36.427008+0000) 2022-01-31T22:41:36.469 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:36 smithi171 conmon[51620]: debug 2022-01-31T22:41:36.341+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:36.342080+0000) 2022-01-31T22:41:36.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:36 smithi171 conmon[46715]: debug 2022-01-31T22:41:36.476+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:36.477194+0000) 2022-01-31T22:41:37.021 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:36 smithi167 conmon[49112]: debug 2022-01-31T22:41:36.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:36.810076+0000) 2022-01-31T22:41:37.022 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:36 smithi167 conmon[54076]: debug 2022-01-31T22:41:36.975+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:36.975748+0000) 2022-01-31T22:41:37.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:36 smithi167 conmon[60316]: debug 2022-01-31T22:41:36.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:36.735833+0000) 2022-01-31T22:41:37.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:37 smithi171 conmon[41853]: debug 2022-01-31T22:41:37.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:37.427207+0000) 2022-01-31T22:41:37.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:37 smithi171 conmon[51620]: debug 2022-01-31T22:41:37.341+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:37.342281+0000) 2022-01-31T22:41:37.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:37 smithi171 conmon[46715]: debug 2022-01-31T22:41:37.476+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:37.477335+0000) 2022-01-31T22:41:38.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:37 smithi167 conmon[49112]: debug 2022-01-31T22:41:37.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:37.810237+0000) 2022-01-31T22:41:38.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:37 smithi167 conmon[54076]: debug 2022-01-31T22:41:37.975+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:37.975933+0000) 2022-01-31T22:41:38.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:37 smithi167 conmon[60316]: debug 2022-01-31T22:41:37.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:37.735951+0000) 2022-01-31T22:41:38.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:38 smithi171 conmon[41853]: debug 2022-01-31T22:41:38.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:38.427407+0000) 2022-01-31T22:41:38.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:38 smithi171 conmon[51620]: debug 2022-01-31T22:41:38.341+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:38.342491+0000) 2022-01-31T22:41:38.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:38 smithi171 conmon[46715]: debug 2022-01-31T22:41:38.477+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:38.477543+0000) 2022-01-31T22:41:39.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:38 smithi167 conmon[49112]: debug 2022-01-31T22:41:38.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:38.810399+0000) 2022-01-31T22:41:39.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:38 smithi167 conmon[54076]: debug 2022-01-31T22:41:38.975+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:38.976070+0000) 2022-01-31T22:41:39.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:38 smithi167 conmon[60316]: debug 2022-01-31T22:41:38.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:38.736096+0000) 2022-01-31T22:41:39.203 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:38 smithi171 conmon[35325]: debug 2022-01-31T22:41:38.894+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:41:39.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[51620]: debug 2022-01-31T22:41:39.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.342688+0000) 2022-01-31T22:41:39.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[41853]: debug 2022-01-31T22:41:39.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.427590+0000) 2022-01-31T22:41:39.727 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:39 smithi167 conmon[49112]: debug 2022-01-31T22:41:39.588+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.589546+0000) 2022-01-31T22:41:39.728 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:39 smithi167 conmon[54076]: debug 2022-01-31T22:41:39.588+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.589719+0000) 2022-01-31T22:41:39.729 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:39 smithi167 conmon[60316]: debug 2022-01-31T22:41:39.588+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.589753+0000) 2022-01-31T22:41:39.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[35325]: debug 2022-01-31T22:41:39.611+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 229962 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:39.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[41853]: debug 2022-01-31T22:41:39.587+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.588129+0000) 2022-01-31T22:41:39.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[51620]: debug 2022-01-31T22:41:39.587+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.588404+0000) 2022-01-31T22:41:39.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[46715]: debug 2022-01-31T22:41:39.477+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.477741+0000) 2022-01-31T22:41:39.852 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:39 smithi171 conmon[46715]: debug 2022-01-31T22:41:39.589+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.590223+0000) 2022-01-31T22:41:40.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:39 smithi167 conmon[49112]: debug 2022-01-31T22:41:39.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.810615+0000) 2022-01-31T22:41:40.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:39 smithi167 conmon[60316]: debug 2022-01-31T22:41:39.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.736224+0000) 2022-01-31T22:41:40.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:39 smithi167 conmon[54076]: debug 2022-01-31T22:41:39.975+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:39.976259+0000) 2022-01-31T22:41:40.470 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:40 smithi171 conmon[41853]: debug 2022-01-31T22:41:40.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:40.427791+0000) 2022-01-31T22:41:40.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:40 smithi171 conmon[51620]: debug 2022-01-31T22:41:40.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:40.342857+0000) 2022-01-31T22:41:40.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:40 smithi171 conmon[46715]: debug 2022-01-31T22:41:40.477+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:40.477904+0000) 2022-01-31T22:41:41.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:40 smithi167 conmon[49112]: debug 2022-01-31T22:41:40.809+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:40.810766+0000) 2022-01-31T22:41:41.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:40 smithi167 conmon[60316]: debug 2022-01-31T22:41:40.734+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:40.736375+0000) 2022-01-31T22:41:41.024 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:40 smithi167 conmon[54076]: debug 2022-01-31T22:41:40.974+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:40.976429+0000) 2022-01-31T22:41:41.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:41 smithi171 conmon[51620]: debug 2022-01-31T22:41:41.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:41.343035+0000) 2022-01-31T22:41:41.470 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:41 smithi171 conmon[41853]: debug 2022-01-31T22:41:41.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:41.427920+0000) 2022-01-31T22:41:41.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:41 smithi171 conmon[46715]: debug 2022-01-31T22:41:41.477+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:41.478026+0000) 2022-01-31T22:41:42.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:41 smithi167 conmon[49112]: debug 2022-01-31T22:41:41.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:41.810953+0000) 2022-01-31T22:41:42.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:41 smithi167 conmon[60316]: debug 2022-01-31T22:41:41.735+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:41.736560+0000) 2022-01-31T22:41:42.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:41 smithi167 conmon[54076]: debug 2022-01-31T22:41:41.975+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:41.976587+0000) 2022-01-31T22:41:42.470 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:42 smithi171 conmon[41853]: debug 2022-01-31T22:41:42.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:42.428054+0000) 2022-01-31T22:41:42.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:42 smithi171 conmon[51620]: debug 2022-01-31T22:41:42.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:42.343201+0000) 2022-01-31T22:41:42.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:42 smithi171 conmon[46715]: debug 2022-01-31T22:41:42.477+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:42.478200+0000) 2022-01-31T22:41:43.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:42 smithi167 conmon[49112]: debug 2022-01-31T22:41:42.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:42.811106+0000) 2022-01-31T22:41:43.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:42 smithi167 conmon[54076]: debug 2022-01-31T22:41:42.976+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:42.976777+0000) 2022-01-31T22:41:43.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:42 smithi167 conmon[60316]: debug 2022-01-31T22:41:42.736+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:42.736739+0000) 2022-01-31T22:41:43.470 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:43 smithi171 conmon[51620]: debug 2022-01-31T22:41:43.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:43.343394+0000) 2022-01-31T22:41:43.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:43 smithi171 conmon[41853]: debug 2022-01-31T22:41:43.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:43.428222+0000) 2022-01-31T22:41:43.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:43 smithi171 conmon[46715]: debug 2022-01-31T22:41:43.477+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:43.478364+0000) 2022-01-31T22:41:44.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:43 smithi167 conmon[49112]: debug 2022-01-31T22:41:43.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:43.811315+0000) 2022-01-31T22:41:44.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:43 smithi167 conmon[54076]: debug 2022-01-31T22:41:43.975+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:43.977006+0000) 2022-01-31T22:41:44.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:43 smithi167 conmon[60316]: debug 2022-01-31T22:41:43.736+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:43.736974+0000) 2022-01-31T22:41:44.470 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[41853]: debug 2022-01-31T22:41:44.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.428513+0000) 2022-01-31T22:41:44.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[51620]: debug 2022-01-31T22:41:44.342+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.343661+0000) 2022-01-31T22:41:44.729 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:44 smithi167 conmon[49112]: debug 2022-01-31T22:41:44.615+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.615680+0000) 2022-01-31T22:41:44.729 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:44 smithi167 conmon[54076]: debug 2022-01-31T22:41:44.615+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.616089+0000) 2022-01-31T22:41:44.730 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:44 smithi167 conmon[60316]: debug 2022-01-31T22:41:44.615+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.616411+0000) 2022-01-31T22:41:44.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[41853]: debug 2022-01-31T22:41:44.613+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.614653+0000) 2022-01-31T22:41:44.850 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[35325]: debug 2022-01-31T22:41:44.637+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230074 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:44.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[46715]: debug 2022-01-31T22:41:44.478+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.478629+0000) 2022-01-31T22:41:44.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[46715]: debug 2022-01-31T22:41:44.615+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.616306+0000) 2022-01-31T22:41:44.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:44 smithi171 conmon[51620]: debug 2022-01-31T22:41:44.614+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.615354+0000) 2022-01-31T22:41:45.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:44 smithi167 conmon[49112]: debug 2022-01-31T22:41:44.810+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.811463+0000) 2022-01-31T22:41:45.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:44 smithi167 conmon[54076]: debug 2022-01-31T22:41:44.976+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.977143+0000) 2022-01-31T22:41:45.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:44 smithi167 conmon[60316]: debug 2022-01-31T22:41:44.736+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:44.737181+0000) 2022-01-31T22:41:45.470 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:45 smithi171 conmon[41853]: debug 2022-01-31T22:41:45.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:45.428708+0000) 2022-01-31T22:41:45.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:45 smithi171 conmon[51620]: debug 2022-01-31T22:41:45.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:45.343845+0000) 2022-01-31T22:41:45.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:45 smithi171 conmon[46715]: debug 2022-01-31T22:41:45.478+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:45.478777+0000) 2022-01-31T22:41:46.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:45 smithi167 conmon[49112]: debug 2022-01-31T22:41:45.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:45.811613+0000) 2022-01-31T22:41:46.024 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:45 smithi167 conmon[54076]: debug 2022-01-31T22:41:45.976+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:45.977275+0000) 2022-01-31T22:41:46.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:45 smithi167 conmon[60316]: debug 2022-01-31T22:41:45.736+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:45.737341+0000) 2022-01-31T22:41:46.470 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:46 smithi171 conmon[41853]: debug 2022-01-31T22:41:46.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:46.428874+0000) 2022-01-31T22:41:46.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:46 smithi171 conmon[51620]: debug 2022-01-31T22:41:46.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:46.343988+0000) 2022-01-31T22:41:46.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:46 smithi171 conmon[46715]: debug 2022-01-31T22:41:46.478+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:46.478943+0000) 2022-01-31T22:41:47.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:46 smithi167 conmon[49112]: debug 2022-01-31T22:41:46.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:46.811778+0000) 2022-01-31T22:41:47.024 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:46 smithi167 conmon[54076]: debug 2022-01-31T22:41:46.976+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:46.977452+0000) 2022-01-31T22:41:47.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:46 smithi167 conmon[60316]: debug 2022-01-31T22:41:46.737+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:46.737519+0000) 2022-01-31T22:41:47.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:47 smithi171 conmon[41853]: debug 2022-01-31T22:41:47.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:47.429052+0000) 2022-01-31T22:41:47.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:47 smithi171 conmon[51620]: debug 2022-01-31T22:41:47.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:47.344150+0000) 2022-01-31T22:41:47.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:47 smithi171 conmon[46715]: debug 2022-01-31T22:41:47.478+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:47.479101+0000) 2022-01-31T22:41:48.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:47 smithi167 conmon[54076]: debug 2022-01-31T22:41:47.977+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:47.977616+0000) 2022-01-31T22:41:48.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:47 smithi167 conmon[60316]: debug 2022-01-31T22:41:47.737+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:47.737712+0000) 2022-01-31T22:41:48.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:47 smithi167 conmon[49112]: debug 2022-01-31T22:41:47.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:47.811973+0000) 2022-01-31T22:41:48.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:48 smithi171 conmon[41853]: debug 2022-01-31T22:41:48.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:48.429156+0000) 2022-01-31T22:41:48.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:48 smithi171 conmon[51620]: debug 2022-01-31T22:41:48.343+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:48.344316+0000) 2022-01-31T22:41:48.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:48 smithi171 conmon[46715]: debug 2022-01-31T22:41:48.478+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:48.479233+0000) 2022-01-31T22:41:49.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:48 smithi167 conmon[54076]: debug 2022-01-31T22:41:48.977+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:48.977772+0000) 2022-01-31T22:41:49.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:48 smithi167 conmon[49112]: debug 2022-01-31T22:41:48.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:48.812127+0000) 2022-01-31T22:41:49.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:48 smithi167 conmon[60316]: debug 2022-01-31T22:41:48.737+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:48.737891+0000) 2022-01-31T22:41:49.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[41853]: debug 2022-01-31T22:41:49.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.429274+0000) 2022-01-31T22:41:49.471 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[51620]: debug 2022-01-31T22:41:49.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.344556+0000) 2022-01-31T22:41:49.730 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:49 smithi167 conmon[54076]: debug 2022-01-31T22:41:49.641+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.641900+0000) 2022-01-31T22:41:49.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:49 smithi167 conmon[49112]: debug 2022-01-31T22:41:49.640+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.641353+0000) 2022-01-31T22:41:49.731 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:49 smithi167 conmon[60316]: debug 2022-01-31T22:41:49.641+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.641995+0000) 2022-01-31T22:41:49.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[35325]: debug 2022-01-31T22:41:49.662+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230186 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:49.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[41853]: debug 2022-01-31T22:41:49.640+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.640890+0000) 2022-01-31T22:41:49.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[46715]: debug 2022-01-31T22:41:49.478+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.479361+0000) 2022-01-31T22:41:49.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[46715]: debug 2022-01-31T22:41:49.640+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.641067+0000) 2022-01-31T22:41:49.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:49 smithi171 conmon[51620]: debug 2022-01-31T22:41:49.639+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.640268+0000) 2022-01-31T22:41:50.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:49 smithi167 conmon[49112]: debug 2022-01-31T22:41:49.811+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.812311+0000) 2022-01-31T22:41:50.024 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:49 smithi167 conmon[54076]: debug 2022-01-31T22:41:49.977+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.977961+0000) 2022-01-31T22:41:50.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:49 smithi167 conmon[60316]: debug 2022-01-31T22:41:49.737+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:49.738081+0000) 2022-01-31T22:41:50.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:50 smithi171 conmon[41853]: debug 2022-01-31T22:41:50.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:50.429389+0000) 2022-01-31T22:41:50.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:50 smithi171 conmon[51620]: debug 2022-01-31T22:41:50.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:50.344820+0000) 2022-01-31T22:41:50.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:50 smithi171 conmon[46715]: debug 2022-01-31T22:41:50.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:50.479524+0000) 2022-01-31T22:41:51.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:50 smithi167 conmon[49112]: debug 2022-01-31T22:41:50.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:50.812464+0000) 2022-01-31T22:41:51.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:50 smithi167 conmon[54076]: debug 2022-01-31T22:41:50.977+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:50.978174+0000) 2022-01-31T22:41:51.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:50 smithi167 conmon[60316]: debug 2022-01-31T22:41:50.737+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:50.738222+0000) 2022-01-31T22:41:51.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:51 smithi171 conmon[41853]: debug 2022-01-31T22:41:51.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:51.429551+0000) 2022-01-31T22:41:51.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:51 smithi171 conmon[51620]: debug 2022-01-31T22:41:51.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:51.344954+0000) 2022-01-31T22:41:51.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:51 smithi171 conmon[46715]: debug 2022-01-31T22:41:51.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:51.479680+0000) 2022-01-31T22:41:52.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:51 smithi167 conmon[49112]: debug 2022-01-31T22:41:51.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:51.812675+0000) 2022-01-31T22:41:52.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:51 smithi167 conmon[54076]: debug 2022-01-31T22:41:51.977+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:51.978377+0000) 2022-01-31T22:41:52.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:51 smithi167 conmon[60316]: debug 2022-01-31T22:41:51.738+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:51.738400+0000) 2022-01-31T22:41:52.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:52 smithi171 conmon[41853]: debug 2022-01-31T22:41:52.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:52.429734+0000) 2022-01-31T22:41:52.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:52 smithi171 conmon[51620]: debug 2022-01-31T22:41:52.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:52.345119+0000) 2022-01-31T22:41:52.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:52 smithi171 conmon[46715]: debug 2022-01-31T22:41:52.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:52.479860+0000) 2022-01-31T22:41:53.024 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:52 smithi167 conmon[54076]: debug 2022-01-31T22:41:52.978+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:52.978566+0000) 2022-01-31T22:41:53.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:52 smithi167 conmon[60316]: debug 2022-01-31T22:41:52.738+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:52.738591+0000) 2022-01-31T22:41:53.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:52 smithi167 conmon[49112]: debug 2022-01-31T22:41:52.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:52.812853+0000) 2022-01-31T22:41:53.471 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:53 smithi171 conmon[41853]: debug 2022-01-31T22:41:53.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:53.429868+0000) 2022-01-31T22:41:53.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:53 smithi171 conmon[51620]: debug 2022-01-31T22:41:53.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:53.345339+0000) 2022-01-31T22:41:53.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:53 smithi171 conmon[46715]: debug 2022-01-31T22:41:53.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:53.480029+0000) 2022-01-31T22:41:54.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:53 smithi167 conmon[49112]: debug 2022-01-31T22:41:53.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:53.813018+0000) 2022-01-31T22:41:54.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:53 smithi167 conmon[54076]: debug 2022-01-31T22:41:53.978+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:53.978698+0000) 2022-01-31T22:41:54.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:53 smithi167 conmon[60316]: debug 2022-01-31T22:41:53.738+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:53.738774+0000) 2022-01-31T22:41:54.206 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:53 smithi171 conmon[35325]: debug 2022-01-31T22:41:53.895+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:41:54.472 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[41853]: debug 2022-01-31T22:41:54.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.430053+0000) 2022-01-31T22:41:54.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[51620]: debug 2022-01-31T22:41:54.344+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.345531+0000) 2022-01-31T22:41:54.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:54 smithi167 conmon[49112]: debug 2022-01-31T22:41:54.666+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.666636+0000) 2022-01-31T22:41:54.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:54 smithi167 conmon[54076]: debug 2022-01-31T22:41:54.667+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.667668+0000) 2022-01-31T22:41:54.732 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:54 smithi167 conmon[60316]: debug 2022-01-31T22:41:54.666+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.666845+0000) 2022-01-31T22:41:54.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[41853]: debug 2022-01-31T22:41:54.665+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.665786+0000) 2022-01-31T22:41:54.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[35325]: debug 2022-01-31T22:41:54.688+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230299 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:54.850 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[46715]: debug 2022-01-31T22:41:54.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.480241+0000) 2022-01-31T22:41:54.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[46715]: debug 2022-01-31T22:41:54.666+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.667156+0000) 2022-01-31T22:41:54.851 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:54 smithi171 conmon[51620]: debug 2022-01-31T22:41:54.665+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.665852+0000) 2022-01-31T22:41:55.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:54 smithi167 conmon[54076]: debug 2022-01-31T22:41:54.978+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.978863+0000) 2022-01-31T22:41:55.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:54 smithi167 conmon[49112]: debug 2022-01-31T22:41:54.812+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.813190+0000) 2022-01-31T22:41:55.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:54 smithi167 conmon[60316]: debug 2022-01-31T22:41:54.738+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:54.738974+0000) 2022-01-31T22:41:55.472 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:55 smithi171 conmon[51620]: debug 2022-01-31T22:41:55.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:55.345738+0000) 2022-01-31T22:41:55.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:55 smithi171 conmon[41853]: debug 2022-01-31T22:41:55.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:55.430241+0000) 2022-01-31T22:41:55.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:55 smithi171 conmon[46715]: debug 2022-01-31T22:41:55.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:55.480426+0000) 2022-01-31T22:41:56.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:55 smithi167 conmon[49112]: debug 2022-01-31T22:41:55.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:55.813328+0000) 2022-01-31T22:41:56.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:55 smithi167 conmon[54076]: debug 2022-01-31T22:41:55.978+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:55.979022+0000) 2022-01-31T22:41:56.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:55 smithi167 conmon[60316]: debug 2022-01-31T22:41:55.738+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:55.739090+0000) 2022-01-31T22:41:56.472 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:56 smithi171 conmon[41853]: debug 2022-01-31T22:41:56.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:56.430323+0000) 2022-01-31T22:41:56.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:56 smithi171 conmon[51620]: debug 2022-01-31T22:41:56.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:56.345902+0000) 2022-01-31T22:41:56.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:56 smithi171 conmon[46715]: debug 2022-01-31T22:41:56.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:56.480578+0000) 2022-01-31T22:41:57.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:56 smithi167 conmon[49112]: debug 2022-01-31T22:41:56.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:56.813530+0000) 2022-01-31T22:41:57.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:56 smithi167 conmon[54076]: debug 2022-01-31T22:41:56.978+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:56.979153+0000) 2022-01-31T22:41:57.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:56 smithi167 conmon[60316]: debug 2022-01-31T22:41:56.739+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:56.739300+0000) 2022-01-31T22:41:57.472 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:57 smithi171 conmon[41853]: debug 2022-01-31T22:41:57.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:57.430416+0000) 2022-01-31T22:41:57.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:57 smithi171 conmon[51620]: debug 2022-01-31T22:41:57.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:57.346037+0000) 2022-01-31T22:41:57.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:57 smithi171 conmon[46715]: debug 2022-01-31T22:41:57.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:57.480774+0000) 2022-01-31T22:41:58.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:57 smithi167 conmon[49112]: debug 2022-01-31T22:41:57.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:57.813676+0000) 2022-01-31T22:41:58.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:57 smithi167 conmon[54076]: debug 2022-01-31T22:41:57.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:57.979318+0000) 2022-01-31T22:41:58.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:57 smithi167 conmon[60316]: debug 2022-01-31T22:41:57.739+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:57.739434+0000) 2022-01-31T22:41:58.472 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:58 smithi171 conmon[41853]: debug 2022-01-31T22:41:58.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:58.430523+0000) 2022-01-31T22:41:58.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:58 smithi171 conmon[51620]: debug 2022-01-31T22:41:58.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:58.346209+0000) 2022-01-31T22:41:58.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:58 smithi171 conmon[46715]: debug 2022-01-31T22:41:58.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:58.480927+0000) 2022-01-31T22:41:59.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:58 smithi167 conmon[49112]: debug 2022-01-31T22:41:58.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:58.813830+0000) 2022-01-31T22:41:59.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:58 smithi167 conmon[54076]: debug 2022-01-31T22:41:58.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:58.979542+0000) 2022-01-31T22:41:59.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:58 smithi167 conmon[60316]: debug 2022-01-31T22:41:58.739+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:58.739587+0000) 2022-01-31T22:41:59.472 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[41853]: debug 2022-01-31T22:41:59.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.430658+0000) 2022-01-31T22:41:59.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[51620]: debug 2022-01-31T22:41:59.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.346393+0000) 2022-01-31T22:41:59.731 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:59 smithi167 conmon[49112]: debug 2022-01-31T22:41:59.691+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.692280+0000) 2022-01-31T22:41:59.732 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:59 smithi167 conmon[54076]: debug 2022-01-31T22:41:59.692+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.693323+0000) 2022-01-31T22:41:59.733 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:59 smithi167 conmon[60316]: debug 2022-01-31T22:41:59.691+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.691994+0000) 2022-01-31T22:41:59.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[35325]: debug 2022-01-31T22:41:59.714+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230410 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:41:59.850 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[41853]: debug 2022-01-31T22:41:59.691+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.691834+0000) 2022-01-31T22:41:59.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[51620]: debug 2022-01-31T22:41:59.690+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.691592+0000) 2022-01-31T22:41:59.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[46715]: debug 2022-01-31T22:41:59.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.481129+0000) 2022-01-31T22:41:59.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:41:59 smithi171 conmon[46715]: debug 2022-01-31T22:41:59.692+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.692894+0000) 2022-01-31T22:42:00.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:41:59 smithi167 conmon[49112]: debug 2022-01-31T22:41:59.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.814058+0000) 2022-01-31T22:42:00.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:41:59 smithi167 conmon[54076]: debug 2022-01-31T22:41:59.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.979761+0000) 2022-01-31T22:42:00.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:41:59 smithi167 conmon[60316]: debug 2022-01-31T22:41:59.739+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:41:59.739816+0000) 2022-01-31T22:42:00.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:00 smithi171 conmon[41853]: debug 2022-01-31T22:42:00.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:00.430856+0000) 2022-01-31T22:42:00.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:00 smithi171 conmon[51620]: debug 2022-01-31T22:42:00.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:00.346609+0000) 2022-01-31T22:42:00.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:00 smithi171 conmon[46715]: debug 2022-01-31T22:42:00.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:00.481273+0000) 2022-01-31T22:42:01.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:00 smithi167 conmon[49112]: debug 2022-01-31T22:42:00.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:00.814226+0000) 2022-01-31T22:42:01.027 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:00 smithi167 conmon[54076]: debug 2022-01-31T22:42:00.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:00.979910+0000) 2022-01-31T22:42:01.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:00 smithi167 conmon[60316]: debug 2022-01-31T22:42:00.739+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:00.739975+0000) 2022-01-31T22:42:01.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:01 smithi171 conmon[51620]: debug 2022-01-31T22:42:01.345+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:01.346796+0000) 2022-01-31T22:42:01.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:01 smithi171 conmon[41853]: debug 2022-01-31T22:42:01.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:01.430982+0000) 2022-01-31T22:42:01.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:01 smithi171 conmon[46715]: debug 2022-01-31T22:42:01.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:01.481373+0000) 2022-01-31T22:42:02.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:01 smithi167 conmon[49112]: debug 2022-01-31T22:42:01.813+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:01.814353+0000) 2022-01-31T22:42:02.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:01 smithi167 conmon[54076]: debug 2022-01-31T22:42:01.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:01.980066+0000) 2022-01-31T22:42:02.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:01 smithi167 conmon[60316]: debug 2022-01-31T22:42:01.739+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:01.740148+0000) 2022-01-31T22:42:02.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:02 smithi171 conmon[41853]: debug 2022-01-31T22:42:02.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:02.431184+0000) 2022-01-31T22:42:02.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:02 smithi171 conmon[51620]: debug 2022-01-31T22:42:02.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:02.347028+0000) 2022-01-31T22:42:02.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:02 smithi171 conmon[46715]: debug 2022-01-31T22:42:02.479+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:02.481564+0000) 2022-01-31T22:42:03.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:02 smithi167 conmon[49112]: debug 2022-01-31T22:42:02.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:02.814517+0000) 2022-01-31T22:42:03.027 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:02 smithi167 conmon[54076]: debug 2022-01-31T22:42:02.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:02.980207+0000) 2022-01-31T22:42:03.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:02 smithi167 conmon[60316]: debug 2022-01-31T22:42:02.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:02.740357+0000) 2022-01-31T22:42:03.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:03 smithi171 conmon[41853]: debug 2022-01-31T22:42:03.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:03.431395+0000) 2022-01-31T22:42:03.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:03 smithi171 conmon[51620]: debug 2022-01-31T22:42:03.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:03.347222+0000) 2022-01-31T22:42:03.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:03 smithi171 conmon[46715]: debug 2022-01-31T22:42:03.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:03.481717+0000) 2022-01-31T22:42:04.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:03 smithi167 conmon[54076]: debug 2022-01-31T22:42:03.979+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:03.980375+0000) 2022-01-31T22:42:04.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:03 smithi167 conmon[60316]: debug 2022-01-31T22:42:03.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:03.740560+0000) 2022-01-31T22:42:04.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:03 smithi167 conmon[49112]: debug 2022-01-31T22:42:03.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:03.814684+0000) 2022-01-31T22:42:04.473 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[51620]: debug 2022-01-31T22:42:04.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.347413+0000) 2022-01-31T22:42:04.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[41853]: debug 2022-01-31T22:42:04.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.431582+0000) 2022-01-31T22:42:04.714 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:04 smithi167 conmon[49112]: debug 2022-01-31T22:42:04.717+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.718377+0000) 2022-01-31T22:42:04.715 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:04 smithi167 conmon[60316]: debug 2022-01-31T22:42:04.718+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.718784+0000) 2022-01-31T22:42:04.716 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:04 smithi167 conmon[54076]: debug 2022-01-31T22:42:04.718+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.719052+0000) 2022-01-31T22:42:04.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[51620]: debug 2022-01-31T22:42:04.716+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.717845+0000) 2022-01-31T22:42:04.735 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[41853]: debug 2022-01-31T22:42:04.715+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.717607+0000) 2022-01-31T22:42:04.736 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[46715]: debug 2022-01-31T22:42:04.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.481938+0000) 2022-01-31T22:42:04.736 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[46715]: debug 2022-01-31T22:42:04.716+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.718658+0000) 2022-01-31T22:42:04.972 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:04 smithi167 conmon[49112]: debug 2022-01-31T22:42:04.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.814897+0000) 2022-01-31T22:42:04.973 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:04 smithi167 conmon[60316]: debug 2022-01-31T22:42:04.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.740703+0000) 2022-01-31T22:42:05.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:04 smithi171 conmon[35325]: debug 2022-01-31T22:42:04.740+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230522 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:05.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:04 smithi167 conmon[54076]: debug 2022-01-31T22:42:04.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:04.980579+0000) 2022-01-31T22:42:05.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:05 smithi171 conmon[41853]: debug 2022-01-31T22:42:05.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:05.431775+0000) 2022-01-31T22:42:05.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:05 smithi171 conmon[51620]: debug 2022-01-31T22:42:05.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:05.347615+0000) 2022-01-31T22:42:05.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:05 smithi171 conmon[46715]: debug 2022-01-31T22:42:05.481+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:05.482123+0000) 2022-01-31T22:42:06.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:05 smithi167 conmon[54076]: debug 2022-01-31T22:42:05.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:05.980728+0000) 2022-01-31T22:42:06.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:05 smithi167 conmon[49112]: debug 2022-01-31T22:42:05.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:05.815037+0000) 2022-01-31T22:42:06.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:05 smithi167 conmon[60316]: debug 2022-01-31T22:42:05.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:05.740849+0000) 2022-01-31T22:42:06.473 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:06 smithi171 conmon[41853]: debug 2022-01-31T22:42:06.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:06.431896+0000) 2022-01-31T22:42:06.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:06 smithi171 conmon[51620]: debug 2022-01-31T22:42:06.346+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:06.347764+0000) 2022-01-31T22:42:06.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:06 smithi171 conmon[46715]: debug 2022-01-31T22:42:06.481+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:06.482261+0000) 2022-01-31T22:42:07.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:06 smithi167 conmon[49112]: debug 2022-01-31T22:42:06.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:06.815189+0000) 2022-01-31T22:42:07.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:06 smithi167 conmon[54076]: debug 2022-01-31T22:42:06.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:06.980941+0000) 2022-01-31T22:42:07.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:06 smithi167 conmon[60316]: debug 2022-01-31T22:42:06.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:06.741036+0000) 2022-01-31T22:42:07.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:07 smithi171 conmon[41853]: debug 2022-01-31T22:42:07.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:07.432040+0000) 2022-01-31T22:42:07.474 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:07 smithi171 conmon[51620]: debug 2022-01-31T22:42:07.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:07.347962+0000) 2022-01-31T22:42:07.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:07 smithi171 conmon[46715]: debug 2022-01-31T22:42:07.481+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:07.482414+0000) 2022-01-31T22:42:08.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:07 smithi167 conmon[49112]: debug 2022-01-31T22:42:07.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:07.815376+0000) 2022-01-31T22:42:08.027 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:07 smithi167 conmon[54076]: debug 2022-01-31T22:42:07.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:07.981104+0000) 2022-01-31T22:42:08.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:07 smithi167 conmon[60316]: debug 2022-01-31T22:42:07.741+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:07.741197+0000) 2022-01-31T22:42:08.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:08 smithi171 conmon[41853]: debug 2022-01-31T22:42:08.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:08.432190+0000) 2022-01-31T22:42:08.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:08 smithi171 conmon[51620]: debug 2022-01-31T22:42:08.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:08.348174+0000) 2022-01-31T22:42:08.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:08 smithi171 conmon[46715]: debug 2022-01-31T22:42:08.480+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:08.482595+0000) 2022-01-31T22:42:09.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:08 smithi167 conmon[49112]: debug 2022-01-31T22:42:08.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:08.815531+0000) 2022-01-31T22:42:09.027 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:08 smithi167 conmon[54076]: debug 2022-01-31T22:42:08.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:08.981290+0000) 2022-01-31T22:42:09.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:08 smithi167 conmon[60316]: debug 2022-01-31T22:42:08.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:08.741380+0000) 2022-01-31T22:42:09.209 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:08 smithi171 conmon[35325]: debug 2022-01-31T22:42:08.895+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:42:09.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[41853]: debug 2022-01-31T22:42:09.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.432419+0000) 2022-01-31T22:42:09.607 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[51620]: debug 2022-01-31T22:42:09.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.348343+0000) 2022-01-31T22:42:09.736 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[46715]: debug 2022-01-31T22:42:09.481+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.482787+0000) 2022-01-31T22:42:10.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:09 smithi167 conmon[49112]: debug 2022-01-31T22:42:09.743+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.744586+0000) 2022-01-31T22:42:10.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:09 smithi167 conmon[49112]: debug 2022-01-31T22:42:09.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.815674+0000) 2022-01-31T22:42:10.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:09 smithi167 conmon[54076]: debug 2022-01-31T22:42:09.744+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.746370+0000) 2022-01-31T22:42:10.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:09 smithi167 conmon[54076]: debug 2022-01-31T22:42:09.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.981494+0000) 2022-01-31T22:42:10.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:09 smithi167 conmon[60316]: debug 2022-01-31T22:42:09.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.741543+0000) 2022-01-31T22:42:10.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:09 smithi167 conmon[60316]: debug 2022-01-31T22:42:09.743+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.745474+0000) 2022-01-31T22:42:10.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[35325]: debug 2022-01-31T22:42:09.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230633 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:10.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[41853]: debug 2022-01-31T22:42:09.743+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.745295+0000) 2022-01-31T22:42:10.100 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[46715]: debug 2022-01-31T22:42:09.743+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.745561+0000) 2022-01-31T22:42:10.101 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:09 smithi171 conmon[51620]: debug 2022-01-31T22:42:09.743+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:09.744776+0000) 2022-01-31T22:42:10.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:10 smithi171 conmon[41853]: debug 2022-01-31T22:42:10.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:10.432569+0000) 2022-01-31T22:42:10.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:10 smithi171 conmon[51620]: debug 2022-01-31T22:42:10.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:10.348576+0000) 2022-01-31T22:42:10.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:10 smithi171 conmon[46715]: debug 2022-01-31T22:42:10.482+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:10.482969+0000) 2022-01-31T22:42:11.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:10 smithi167 conmon[49112]: debug 2022-01-31T22:42:10.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:10.815798+0000) 2022-01-31T22:42:11.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:10 smithi167 conmon[54076]: debug 2022-01-31T22:42:10.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:10.981678+0000) 2022-01-31T22:42:11.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:10 smithi167 conmon[60316]: debug 2022-01-31T22:42:10.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:10.741751+0000) 2022-01-31T22:42:11.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:11 smithi171 conmon[41853]: debug 2022-01-31T22:42:11.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:11.432761+0000) 2022-01-31T22:42:11.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:11 smithi171 conmon[51620]: debug 2022-01-31T22:42:11.347+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:11.348785+0000) 2022-01-31T22:42:11.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:11 smithi171 conmon[46715]: debug 2022-01-31T22:42:11.482+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:11.483082+0000) 2022-01-31T22:42:12.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:11 smithi167 conmon[49112]: debug 2022-01-31T22:42:11.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:11.815960+0000) 2022-01-31T22:42:12.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:11 smithi167 conmon[54076]: debug 2022-01-31T22:42:11.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:11.981880+0000) 2022-01-31T22:42:12.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:11 smithi167 conmon[60316]: debug 2022-01-31T22:42:11.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:11.741891+0000) 2022-01-31T22:42:12.474 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:12 smithi171 conmon[41853]: debug 2022-01-31T22:42:12.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:12.432912+0000) 2022-01-31T22:42:12.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:12 smithi171 conmon[51620]: debug 2022-01-31T22:42:12.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:12.348988+0000) 2022-01-31T22:42:12.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:12 smithi171 conmon[46715]: debug 2022-01-31T22:42:12.482+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:12.483227+0000) 2022-01-31T22:42:13.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:12 smithi167 conmon[49112]: debug 2022-01-31T22:42:12.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:12.816150+0000) 2022-01-31T22:42:13.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:12 smithi167 conmon[54076]: debug 2022-01-31T22:42:12.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:12.982034+0000) 2022-01-31T22:42:13.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:12 smithi167 conmon[60316]: debug 2022-01-31T22:42:12.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:12.742095+0000) 2022-01-31T22:42:13.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:13 smithi171 conmon[41853]: debug 2022-01-31T22:42:13.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:13.433089+0000) 2022-01-31T22:42:13.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:13 smithi171 conmon[51620]: debug 2022-01-31T22:42:13.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:13.349231+0000) 2022-01-31T22:42:13.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:13 smithi171 conmon[46715]: debug 2022-01-31T22:42:13.482+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:13.483405+0000) 2022-01-31T22:42:14.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:13 smithi167 conmon[49112]: debug 2022-01-31T22:42:13.814+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:13.816335+0000) 2022-01-31T22:42:14.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:13 smithi167 conmon[54076]: debug 2022-01-31T22:42:13.980+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:13.982249+0000) 2022-01-31T22:42:14.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:13 smithi167 conmon[60316]: debug 2022-01-31T22:42:13.740+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:13.742319+0000) 2022-01-31T22:42:14.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[41853]: debug 2022-01-31T22:42:14.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.433271+0000) 2022-01-31T22:42:14.475 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[51620]: debug 2022-01-31T22:42:14.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.349444+0000) 2022-01-31T22:42:14.777 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[46715]: debug 2022-01-31T22:42:14.482+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.483558+0000) 2022-01-31T22:42:15.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:14 smithi167 conmon[49112]: debug 2022-01-31T22:42:14.786+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.787565+0000) 2022-01-31T22:42:15.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:14 smithi167 conmon[49112]: debug 2022-01-31T22:42:14.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.816949+0000) 2022-01-31T22:42:15.029 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:14 smithi167 conmon[54076]: debug 2022-01-31T22:42:14.786+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.787722+0000) 2022-01-31T22:42:15.029 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:14 smithi167 conmon[54076]: debug 2022-01-31T22:42:14.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.982431+0000) 2022-01-31T22:42:15.030 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:14 smithi167 conmon[60316]: debug 2022-01-31T22:42:14.741+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.742518+0000) 2022-01-31T22:42:15.030 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:14 smithi167 conmon[60316]: debug 2022-01-31T22:42:14.785+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.786727+0000) 2022-01-31T22:42:15.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[35325]: debug 2022-01-31T22:42:14.809+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230745 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:15.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[41853]: debug 2022-01-31T22:42:14.785+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.786245+0000) 2022-01-31T22:42:15.100 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[46715]: debug 2022-01-31T22:42:14.786+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.786758+0000) 2022-01-31T22:42:15.100 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:14 smithi171 conmon[51620]: debug 2022-01-31T22:42:14.785+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:14.786386+0000) 2022-01-31T22:42:15.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:15 smithi171 conmon[41853]: debug 2022-01-31T22:42:15.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:15.433466+0000) 2022-01-31T22:42:15.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:15 smithi171 conmon[51620]: debug 2022-01-31T22:42:15.348+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:15.349663+0000) 2022-01-31T22:42:15.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:15 smithi171 conmon[46715]: debug 2022-01-31T22:42:15.482+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:15.483683+0000) 2022-01-31T22:42:16.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:15 smithi167 conmon[49112]: debug 2022-01-31T22:42:15.815+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:15.817168+0000) 2022-01-31T22:42:16.028 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:15 smithi167 conmon[54076]: debug 2022-01-31T22:42:15.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:15.982542+0000) 2022-01-31T22:42:16.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:15 smithi167 conmon[60316]: debug 2022-01-31T22:42:15.741+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:15.742701+0000) 2022-01-31T22:42:16.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:16 smithi171 conmon[41853]: debug 2022-01-31T22:42:16.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:16.433609+0000) 2022-01-31T22:42:16.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:16 smithi171 conmon[51620]: debug 2022-01-31T22:42:16.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:16.349830+0000) 2022-01-31T22:42:16.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:16 smithi171 conmon[46715]: debug 2022-01-31T22:42:16.483+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:16.483845+0000) 2022-01-31T22:42:17.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:16 smithi167 conmon[49112]: debug 2022-01-31T22:42:16.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:16.817324+0000) 2022-01-31T22:42:17.029 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:16 smithi167 conmon[54076]: debug 2022-01-31T22:42:16.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:16.982751+0000) 2022-01-31T22:42:17.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:16 smithi167 conmon[60316]: debug 2022-01-31T22:42:16.741+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:16.742817+0000) 2022-01-31T22:42:17.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:17 smithi171 conmon[41853]: debug 2022-01-31T22:42:17.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:17.433811+0000) 2022-01-31T22:42:17.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:17 smithi171 conmon[51620]: debug 2022-01-31T22:42:17.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:17.350040+0000) 2022-01-31T22:42:17.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:17 smithi171 conmon[46715]: debug 2022-01-31T22:42:17.483+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:17.484061+0000) 2022-01-31T22:42:18.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:17 smithi167 conmon[49112]: debug 2022-01-31T22:42:17.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:17.817520+0000) 2022-01-31T22:42:18.029 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:17 smithi167 conmon[54076]: debug 2022-01-31T22:42:17.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:17.982910+0000) 2022-01-31T22:42:18.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:17 smithi167 conmon[60316]: debug 2022-01-31T22:42:17.742+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:17.743028+0000) 2022-01-31T22:42:18.475 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:18 smithi171 conmon[41853]: debug 2022-01-31T22:42:18.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:18.433998+0000) 2022-01-31T22:42:18.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:18 smithi171 conmon[51620]: debug 2022-01-31T22:42:18.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:18.350277+0000) 2022-01-31T22:42:18.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:18 smithi171 conmon[46715]: debug 2022-01-31T22:42:18.483+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:18.484284+0000) 2022-01-31T22:42:19.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:18 smithi167 conmon[49112]: debug 2022-01-31T22:42:18.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:18.817699+0000) 2022-01-31T22:42:19.029 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:18 smithi167 conmon[54076]: debug 2022-01-31T22:42:18.982+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:18.983068+0000) 2022-01-31T22:42:19.030 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:18 smithi167 conmon[60316]: debug 2022-01-31T22:42:18.742+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:18.743198+0000) 2022-01-31T22:42:19.476 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[41853]: debug 2022-01-31T22:42:19.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.434158+0000) 2022-01-31T22:42:19.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[51620]: debug 2022-01-31T22:42:19.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.350489+0000) 2022-01-31T22:42:19.803 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[46715]: debug 2022-01-31T22:42:19.483+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.484463+0000) 2022-01-31T22:42:20.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:19 smithi167 conmon[49112]: debug 2022-01-31T22:42:19.811+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.813048+0000) 2022-01-31T22:42:20.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:19 smithi167 conmon[49112]: debug 2022-01-31T22:42:19.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.817833+0000) 2022-01-31T22:42:20.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:19 smithi167 conmon[54076]: debug 2022-01-31T22:42:19.811+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.813233+0000) 2022-01-31T22:42:20.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:19 smithi167 conmon[54076]: debug 2022-01-31T22:42:19.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.983275+0000) 2022-01-31T22:42:20.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:19 smithi167 conmon[60316]: debug 2022-01-31T22:42:19.741+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.743386+0000) 2022-01-31T22:42:20.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:19 smithi167 conmon[60316]: debug 2022-01-31T22:42:19.811+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.812668+0000) 2022-01-31T22:42:20.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[35325]: debug 2022-01-31T22:42:19.835+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230856 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:20.099 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[41853]: debug 2022-01-31T22:42:19.811+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.811908+0000) 2022-01-31T22:42:20.100 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[46715]: debug 2022-01-31T22:42:19.811+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.812210+0000) 2022-01-31T22:42:20.101 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:19 smithi171 conmon[51620]: debug 2022-01-31T22:42:19.812+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:19.813417+0000) 2022-01-31T22:42:20.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:20 smithi171 conmon[51620]: debug 2022-01-31T22:42:20.349+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:20.350726+0000) 2022-01-31T22:42:20.476 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:20 smithi171 conmon[41853]: debug 2022-01-31T22:42:20.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:20.434338+0000) 2022-01-31T22:42:20.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:20 smithi171 conmon[46715]: debug 2022-01-31T22:42:20.484+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:20.484664+0000) 2022-01-31T22:42:21.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:20 smithi167 conmon[49112]: debug 2022-01-31T22:42:20.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:20.817977+0000) 2022-01-31T22:42:21.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:20 smithi167 conmon[54076]: debug 2022-01-31T22:42:20.981+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:20.983482+0000) 2022-01-31T22:42:21.030 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:20 smithi167 conmon[60316]: debug 2022-01-31T22:42:20.742+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:20.743571+0000) 2022-01-31T22:42:21.476 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:21 smithi171 conmon[41853]: debug 2022-01-31T22:42:21.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:21.434490+0000) 2022-01-31T22:42:21.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:21 smithi171 conmon[51620]: debug 2022-01-31T22:42:21.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:21.350919+0000) 2022-01-31T22:42:21.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:21 smithi171 conmon[46715]: debug 2022-01-31T22:42:21.484+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:21.484819+0000) 2022-01-31T22:42:22.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:21 smithi167 conmon[49112]: debug 2022-01-31T22:42:21.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:21.818176+0000) 2022-01-31T22:42:22.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:21 smithi167 conmon[54076]: debug 2022-01-31T22:42:21.982+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:21.983649+0000) 2022-01-31T22:42:22.030 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:21 smithi167 conmon[60316]: debug 2022-01-31T22:42:21.742+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:21.743673+0000) 2022-01-31T22:42:22.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:22 smithi171 conmon[51620]: debug 2022-01-31T22:42:22.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:22.351084+0000) 2022-01-31T22:42:22.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:22 smithi171 conmon[41853]: debug 2022-01-31T22:42:22.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:22.434645+0000) 2022-01-31T22:42:22.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:22 smithi171 conmon[46715]: debug 2022-01-31T22:42:22.484+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:22.484998+0000) 2022-01-31T22:42:23.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:22 smithi167 conmon[49112]: debug 2022-01-31T22:42:22.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:22.818338+0000) 2022-01-31T22:42:23.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:22 smithi167 conmon[54076]: debug 2022-01-31T22:42:22.982+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:22.983795+0000) 2022-01-31T22:42:23.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:22 smithi167 conmon[60316]: debug 2022-01-31T22:42:22.743+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:22.743899+0000) 2022-01-31T22:42:23.476 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:23 smithi171 conmon[41853]: debug 2022-01-31T22:42:23.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:23.434845+0000) 2022-01-31T22:42:23.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:23 smithi171 conmon[51620]: debug 2022-01-31T22:42:23.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:23.351256+0000) 2022-01-31T22:42:23.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:23 smithi171 conmon[46715]: debug 2022-01-31T22:42:23.484+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:23.485170+0000) 2022-01-31T22:42:24.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:23 smithi167 conmon[49112]: debug 2022-01-31T22:42:23.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:23.818542+0000) 2022-01-31T22:42:24.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:23 smithi167 conmon[54076]: debug 2022-01-31T22:42:23.982+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:23.983980+0000) 2022-01-31T22:42:24.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:23 smithi167 conmon[60316]: debug 2022-01-31T22:42:23.743+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:23.744118+0000) 2022-01-31T22:42:24.212 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:23 smithi171 conmon[35325]: debug 2022-01-31T22:42:23.896+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:42:24.476 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[51620]: debug 2022-01-31T22:42:24.350+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.351465+0000) 2022-01-31T22:42:24.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[41853]: debug 2022-01-31T22:42:24.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.435043+0000) 2022-01-31T22:42:24.830 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[46715]: debug 2022-01-31T22:42:24.484+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.485359+0000) 2022-01-31T22:42:25.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:24 smithi167 conmon[54076]: debug 2022-01-31T22:42:24.838+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.839881+0000) 2022-01-31T22:42:25.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:24 smithi167 conmon[54076]: debug 2022-01-31T22:42:24.982+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.984199+0000) 2022-01-31T22:42:25.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:24 smithi167 conmon[49112]: debug 2022-01-31T22:42:24.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.818725+0000) 2022-01-31T22:42:25.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:24 smithi167 conmon[49112]: debug 2022-01-31T22:42:24.838+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.840135+0000) 2022-01-31T22:42:25.032 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:24 smithi167 conmon[60316]: debug 2022-01-31T22:42:24.743+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.744322+0000) 2022-01-31T22:42:25.032 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:24 smithi167 conmon[60316]: debug 2022-01-31T22:42:24.837+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.839146+0000) 2022-01-31T22:42:25.099 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[35325]: debug 2022-01-31T22:42:24.861+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 230969 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:25.100 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[41853]: debug 2022-01-31T22:42:24.838+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.839018+0000) 2022-01-31T22:42:25.101 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[46715]: debug 2022-01-31T22:42:24.838+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.839468+0000) 2022-01-31T22:42:25.101 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:24 smithi171 conmon[51620]: debug 2022-01-31T22:42:24.837+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:24.838629+0000) 2022-01-31T22:42:25.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:25 smithi171 conmon[41853]: debug 2022-01-31T22:42:25.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:25.435231+0000) 2022-01-31T22:42:25.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:25 smithi171 conmon[51620]: debug 2022-01-31T22:42:25.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:25.351677+0000) 2022-01-31T22:42:25.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:25 smithi171 conmon[46715]: debug 2022-01-31T22:42:25.485+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:25.485514+0000) 2022-01-31T22:42:26.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:25 smithi167 conmon[49112]: debug 2022-01-31T22:42:25.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:25.818877+0000) 2022-01-31T22:42:26.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:25 smithi167 conmon[54076]: debug 2022-01-31T22:42:25.982+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:25.984470+0000) 2022-01-31T22:42:26.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:25 smithi167 conmon[60316]: debug 2022-01-31T22:42:25.743+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:25.744521+0000) 2022-01-31T22:42:26.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:26 smithi171 conmon[41853]: debug 2022-01-31T22:42:26.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:26.435386+0000) 2022-01-31T22:42:26.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:26 smithi171 conmon[51620]: debug 2022-01-31T22:42:26.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:26.351845+0000) 2022-01-31T22:42:26.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:26 smithi171 conmon[46715]: debug 2022-01-31T22:42:26.485+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:26.485643+0000) 2022-01-31T22:42:27.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:26 smithi167 conmon[49112]: debug 2022-01-31T22:42:26.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:26.819026+0000) 2022-01-31T22:42:27.030 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:26 smithi167 conmon[54076]: debug 2022-01-31T22:42:26.984+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:26.984664+0000) 2022-01-31T22:42:27.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:26 smithi167 conmon[60316]: debug 2022-01-31T22:42:26.743+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:26.744663+0000) 2022-01-31T22:42:27.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:27 smithi171 conmon[41853]: debug 2022-01-31T22:42:27.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:27.435547+0000) 2022-01-31T22:42:27.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:27 smithi171 conmon[51620]: debug 2022-01-31T22:42:27.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:27.352031+0000) 2022-01-31T22:42:27.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:27 smithi171 conmon[46715]: debug 2022-01-31T22:42:27.485+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:27.485805+0000) 2022-01-31T22:42:28.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:27 smithi167 conmon[49112]: debug 2022-01-31T22:42:27.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:27.819178+0000) 2022-01-31T22:42:28.031 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:27 smithi167 conmon[54076]: debug 2022-01-31T22:42:27.984+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:27.984857+0000) 2022-01-31T22:42:28.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:27 smithi167 conmon[60316]: debug 2022-01-31T22:42:27.744+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:27.744848+0000) 2022-01-31T22:42:28.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:28 smithi171 conmon[51620]: debug 2022-01-31T22:42:28.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:28.352184+0000) 2022-01-31T22:42:28.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:28 smithi171 conmon[41853]: debug 2022-01-31T22:42:28.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:28.435705+0000) 2022-01-31T22:42:28.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:28 smithi171 conmon[46715]: debug 2022-01-31T22:42:28.485+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:28.486024+0000) 2022-01-31T22:42:29.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:28 smithi167 conmon[49112]: debug 2022-01-31T22:42:28.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:28.819335+0000) 2022-01-31T22:42:29.031 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:28 smithi167 conmon[54076]: debug 2022-01-31T22:42:28.983+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:28.985045+0000) 2022-01-31T22:42:29.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:28 smithi167 conmon[60316]: debug 2022-01-31T22:42:28.744+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:28.745027+0000) 2022-01-31T22:42:29.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[41853]: debug 2022-01-31T22:42:29.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.435950+0000) 2022-01-31T22:42:29.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[51620]: debug 2022-01-31T22:42:29.351+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.352356+0000) 2022-01-31T22:42:29.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[46715]: debug 2022-01-31T22:42:29.485+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.486189+0000) 2022-01-31T22:42:30.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:29 smithi167 conmon[49112]: debug 2022-01-31T22:42:29.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.819525+0000) 2022-01-31T22:42:30.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:29 smithi167 conmon[49112]: debug 2022-01-31T22:42:29.865+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.866544+0000) 2022-01-31T22:42:30.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:29 smithi167 conmon[54076]: debug 2022-01-31T22:42:29.865+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.866763+0000) 2022-01-31T22:42:30.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:29 smithi167 conmon[54076]: debug 2022-01-31T22:42:29.984+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.985199+0000) 2022-01-31T22:42:30.032 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:29 smithi167 conmon[60316]: debug 2022-01-31T22:42:29.743+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.745202+0000) 2022-01-31T22:42:30.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:29 smithi167 conmon[60316]: debug 2022-01-31T22:42:29.864+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.866128+0000) 2022-01-31T22:42:30.213 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[35325]: debug 2022-01-31T22:42:29.887+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231080 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:30.214 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[51620]: debug 2022-01-31T22:42:29.864+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.864953+0000) 2022-01-31T22:42:30.214 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[41853]: debug 2022-01-31T22:42:29.864+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.865477+0000) 2022-01-31T22:42:30.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:29 smithi171 conmon[46715]: debug 2022-01-31T22:42:29.865+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:29.865739+0000) 2022-01-31T22:42:30.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:30 smithi171 conmon[41853]: debug 2022-01-31T22:42:30.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:30.436090+0000) 2022-01-31T22:42:30.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:30 smithi171 conmon[51620]: debug 2022-01-31T22:42:30.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:30.352592+0000) 2022-01-31T22:42:30.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:30 smithi171 conmon[46715]: debug 2022-01-31T22:42:30.485+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:30.486369+0000) 2022-01-31T22:42:31.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:30 smithi167 conmon[49112]: debug 2022-01-31T22:42:30.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:30.819654+0000) 2022-01-31T22:42:31.031 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:30 smithi167 conmon[54076]: debug 2022-01-31T22:42:30.984+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:30.985386+0000) 2022-01-31T22:42:31.032 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:30 smithi167 conmon[60316]: debug 2022-01-31T22:42:30.744+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:30.745389+0000) 2022-01-31T22:42:31.477 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:31 smithi171 conmon[41853]: debug 2022-01-31T22:42:31.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:31.436202+0000) 2022-01-31T22:42:31.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:31 smithi171 conmon[51620]: debug 2022-01-31T22:42:31.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:31.352782+0000) 2022-01-31T22:42:31.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:31 smithi171 conmon[46715]: debug 2022-01-31T22:42:31.486+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:31.486506+0000) 2022-01-31T22:42:32.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:31 smithi167 conmon[49112]: debug 2022-01-31T22:42:31.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:31.819810+0000) 2022-01-31T22:42:32.031 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:31 smithi167 conmon[54076]: debug 2022-01-31T22:42:31.985+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:31.985560+0000) 2022-01-31T22:42:32.032 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:31 smithi167 conmon[60316]: debug 2022-01-31T22:42:31.744+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:31.745545+0000) 2022-01-31T22:42:32.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:32 smithi171 conmon[41853]: debug 2022-01-31T22:42:32.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:32.436376+0000) 2022-01-31T22:42:32.478 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:32 smithi171 conmon[51620]: debug 2022-01-31T22:42:32.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:32.352997+0000) 2022-01-31T22:42:32.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:32 smithi171 conmon[46715]: debug 2022-01-31T22:42:32.486+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:32.486722+0000) 2022-01-31T22:42:33.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:32 smithi167 conmon[49112]: debug 2022-01-31T22:42:32.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:32.820011+0000) 2022-01-31T22:42:33.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:32 smithi167 conmon[54076]: debug 2022-01-31T22:42:32.985+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:32.985720+0000) 2022-01-31T22:42:33.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:32 smithi167 conmon[60316]: debug 2022-01-31T22:42:32.745+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:32.745748+0000) 2022-01-31T22:42:33.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:33 smithi171 conmon[41853]: debug 2022-01-31T22:42:33.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:33.436491+0000) 2022-01-31T22:42:33.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:33 smithi171 conmon[51620]: debug 2022-01-31T22:42:33.352+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:33.353210+0000) 2022-01-31T22:42:33.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:33 smithi171 conmon[46715]: debug 2022-01-31T22:42:33.486+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:33.486910+0000) 2022-01-31T22:42:34.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:33 smithi167 conmon[49112]: debug 2022-01-31T22:42:33.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:33.820226+0000) 2022-01-31T22:42:34.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:33 smithi167 conmon[54076]: debug 2022-01-31T22:42:33.985+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:33.985852+0000) 2022-01-31T22:42:34.032 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:33 smithi167 conmon[60316]: debug 2022-01-31T22:42:33.745+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:33.745906+0000) 2022-01-31T22:42:34.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[41853]: debug 2022-01-31T22:42:34.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.436687+0000) 2022-01-31T22:42:34.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[51620]: debug 2022-01-31T22:42:34.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.353400+0000) 2022-01-31T22:42:34.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[46715]: debug 2022-01-31T22:42:34.486+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.487103+0000) 2022-01-31T22:42:35.031 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:34 smithi167 conmon[54076]: debug 2022-01-31T22:42:34.891+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.891670+0000) 2022-01-31T22:42:35.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:34 smithi167 conmon[54076]: debug 2022-01-31T22:42:34.985+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.986049+0000) 2022-01-31T22:42:35.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:34 smithi167 conmon[49112]: debug 2022-01-31T22:42:34.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.820394+0000) 2022-01-31T22:42:35.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:34 smithi167 conmon[49112]: debug 2022-01-31T22:42:34.890+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.890808+0000) 2022-01-31T22:42:35.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:34 smithi167 conmon[60316]: debug 2022-01-31T22:42:34.745+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.746075+0000) 2022-01-31T22:42:35.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:34 smithi167 conmon[60316]: debug 2022-01-31T22:42:34.891+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.891968+0000) 2022-01-31T22:42:35.214 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[35325]: debug 2022-01-31T22:42:34.913+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:35.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[41853]: debug 2022-01-31T22:42:34.889+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.890548+0000) 2022-01-31T22:42:35.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[46715]: debug 2022-01-31T22:42:34.890+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.891317+0000) 2022-01-31T22:42:35.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:34 smithi171 conmon[51620]: debug 2022-01-31T22:42:34.890+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:34.891536+0000) 2022-01-31T22:42:35.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:35 smithi171 conmon[41853]: debug 2022-01-31T22:42:35.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:35.436843+0000) 2022-01-31T22:42:35.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:35 smithi171 conmon[51620]: debug 2022-01-31T22:42:35.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:35.353575+0000) 2022-01-31T22:42:35.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:35 smithi171 conmon[46715]: debug 2022-01-31T22:42:35.486+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:35.487256+0000) 2022-01-31T22:42:36.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:35 smithi167 conmon[49112]: debug 2022-01-31T22:42:35.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:35.820524+0000) 2022-01-31T22:42:36.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:35 smithi167 conmon[54076]: debug 2022-01-31T22:42:35.985+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:35.986159+0000) 2022-01-31T22:42:36.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:35 smithi167 conmon[60316]: debug 2022-01-31T22:42:35.745+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:35.746261+0000) 2022-01-31T22:42:36.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:36 smithi171 conmon[41853]: debug 2022-01-31T22:42:36.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:36.436992+0000) 2022-01-31T22:42:36.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:36 smithi171 conmon[51620]: debug 2022-01-31T22:42:36.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:36.353784+0000) 2022-01-31T22:42:36.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:36 smithi171 conmon[46715]: debug 2022-01-31T22:42:36.487+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:36.487415+0000) 2022-01-31T22:42:37.031 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:36 smithi167 conmon[49112]: debug 2022-01-31T22:42:36.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:36.820605+0000) 2022-01-31T22:42:37.032 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:36 smithi167 conmon[54076]: debug 2022-01-31T22:42:36.985+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:36.986345+0000) 2022-01-31T22:42:37.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:36 smithi167 conmon[60316]: debug 2022-01-31T22:42:36.745+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:36.746425+0000) 2022-01-31T22:42:37.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:37 smithi171 conmon[41853]: debug 2022-01-31T22:42:37.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:37.437174+0000) 2022-01-31T22:42:37.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:37 smithi171 conmon[51620]: debug 2022-01-31T22:42:37.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:37.353953+0000) 2022-01-31T22:42:37.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:37 smithi171 conmon[46715]: debug 2022-01-31T22:42:37.487+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:37.487619+0000) 2022-01-31T22:42:38.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:37 smithi167 conmon[49112]: debug 2022-01-31T22:42:37.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:37.820774+0000) 2022-01-31T22:42:38.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:37 smithi167 conmon[54076]: debug 2022-01-31T22:42:37.986+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:37.986567+0000) 2022-01-31T22:42:38.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:37 smithi167 conmon[60316]: debug 2022-01-31T22:42:37.746+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:37.746614+0000) 2022-01-31T22:42:38.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:38 smithi171 conmon[41853]: debug 2022-01-31T22:42:38.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:38.437356+0000) 2022-01-31T22:42:38.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:38 smithi171 conmon[51620]: debug 2022-01-31T22:42:38.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:38.354132+0000) 2022-01-31T22:42:38.801 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:38 smithi171 conmon[46715]: debug 2022-01-31T22:42:38.487+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:38.487793+0000) 2022-01-31T22:42:39.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:38 smithi167 conmon[49112]: debug 2022-01-31T22:42:38.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:38.820970+0000) 2022-01-31T22:42:39.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:38 smithi167 conmon[54076]: debug 2022-01-31T22:42:38.986+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:38.986713+0000) 2022-01-31T22:42:39.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:38 smithi167 conmon[60316]: debug 2022-01-31T22:42:38.746+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:38.746797+0000) 2022-01-31T22:42:39.214 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:38 smithi171 conmon[35325]: debug 2022-01-31T22:42:38.897+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:42:39.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[41853]: debug 2022-01-31T22:42:39.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.437569+0000) 2022-01-31T22:42:39.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[51620]: debug 2022-01-31T22:42:39.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.354348+0000) 2022-01-31T22:42:39.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[46715]: debug 2022-01-31T22:42:39.487+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.487929+0000) 2022-01-31T22:42:40.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:39 smithi167 conmon[49112]: debug 2022-01-31T22:42:39.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.821183+0000) 2022-01-31T22:42:40.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:39 smithi167 conmon[49112]: debug 2022-01-31T22:42:39.917+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.917612+0000) 2022-01-31T22:42:40.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:39 smithi167 conmon[54076]: debug 2022-01-31T22:42:39.918+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.918900+0000) 2022-01-31T22:42:40.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:39 smithi167 conmon[54076]: debug 2022-01-31T22:42:39.986+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.986873+0000) 2022-01-31T22:42:40.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:39 smithi167 conmon[60316]: debug 2022-01-31T22:42:39.746+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.746988+0000) 2022-01-31T22:42:40.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:39 smithi167 conmon[60316]: debug 2022-01-31T22:42:39.917+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.918018+0000) 2022-01-31T22:42:40.214 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[35325]: debug 2022-01-31T22:42:39.939+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231303 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:40.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[41853]: debug 2022-01-31T22:42:39.915+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.916211+0000) 2022-01-31T22:42:40.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[46715]: debug 2022-01-31T22:42:39.915+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.916323+0000) 2022-01-31T22:42:40.216 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:39 smithi171 conmon[51620]: debug 2022-01-31T22:42:39.915+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:39.916616+0000) 2022-01-31T22:42:40.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:40 smithi171 conmon[41853]: debug 2022-01-31T22:42:40.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:40.437726+0000) 2022-01-31T22:42:40.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:40 smithi171 conmon[51620]: debug 2022-01-31T22:42:40.353+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:40.354576+0000) 2022-01-31T22:42:40.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:40 smithi171 conmon[46715]: debug 2022-01-31T22:42:40.487+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:40.488097+0000) 2022-01-31T22:42:41.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:40 smithi167 conmon[49112]: debug 2022-01-31T22:42:40.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:40.821366+0000) 2022-01-31T22:42:41.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:40 smithi167 conmon[54076]: debug 2022-01-31T22:42:40.986+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:40.987063+0000) 2022-01-31T22:42:41.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:40 smithi167 conmon[60316]: debug 2022-01-31T22:42:40.746+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:40.747177+0000) 2022-01-31T22:42:41.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:41 smithi171 conmon[41853]: debug 2022-01-31T22:42:41.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:41.437896+0000) 2022-01-31T22:42:41.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:41 smithi171 conmon[51620]: debug 2022-01-31T22:42:41.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:41.354748+0000) 2022-01-31T22:42:41.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:41 smithi171 conmon[46715]: debug 2022-01-31T22:42:41.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:41.488211+0000) 2022-01-31T22:42:42.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:41 smithi167 conmon[49112]: debug 2022-01-31T22:42:41.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:41.821525+0000) 2022-01-31T22:42:42.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:41 smithi167 conmon[54076]: debug 2022-01-31T22:42:41.986+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:41.987265+0000) 2022-01-31T22:42:42.033 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:41 smithi167 conmon[60316]: debug 2022-01-31T22:42:41.746+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:41.747332+0000) 2022-01-31T22:42:42.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:42 smithi171 conmon[41853]: debug 2022-01-31T22:42:42.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:42.438095+0000) 2022-01-31T22:42:42.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:42 smithi171 conmon[51620]: debug 2022-01-31T22:42:42.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:42.354945+0000) 2022-01-31T22:42:42.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:42 smithi171 conmon[46715]: debug 2022-01-31T22:42:42.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:42.488363+0000) 2022-01-31T22:42:43.032 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:42 smithi167 conmon[49112]: debug 2022-01-31T22:42:42.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:42.821653+0000) 2022-01-31T22:42:43.033 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:42 smithi167 conmon[54076]: debug 2022-01-31T22:42:42.986+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:42.987426+0000) 2022-01-31T22:42:43.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:42 smithi167 conmon[60316]: debug 2022-01-31T22:42:42.747+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:42.747533+0000) 2022-01-31T22:42:43.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:43 smithi171 conmon[41853]: debug 2022-01-31T22:42:43.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:43.438260+0000) 2022-01-31T22:42:43.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:43 smithi171 conmon[51620]: debug 2022-01-31T22:42:43.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:43.355146+0000) 2022-01-31T22:42:43.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:43 smithi171 conmon[46715]: debug 2022-01-31T22:42:43.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:43.488550+0000) 2022-01-31T22:42:44.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:43 smithi167 conmon[49112]: debug 2022-01-31T22:42:43.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:43.821772+0000) 2022-01-31T22:42:44.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:43 smithi167 conmon[54076]: debug 2022-01-31T22:42:43.987+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:43.987617+0000) 2022-01-31T22:42:44.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:43 smithi167 conmon[60316]: debug 2022-01-31T22:42:43.747+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:43.747743+0000) 2022-01-31T22:42:44.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[51620]: debug 2022-01-31T22:42:44.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.355321+0000) 2022-01-31T22:42:44.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[41853]: debug 2022-01-31T22:42:44.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.438436+0000) 2022-01-31T22:42:44.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[46715]: debug 2022-01-31T22:42:44.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.488785+0000) 2022-01-31T22:42:45.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:44 smithi167 conmon[49112]: debug 2022-01-31T22:42:44.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.821956+0000) 2022-01-31T22:42:45.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:44 smithi167 conmon[49112]: debug 2022-01-31T22:42:44.941+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.942365+0000) 2022-01-31T22:42:45.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:44 smithi167 conmon[54076]: debug 2022-01-31T22:42:44.943+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.943506+0000) 2022-01-31T22:42:45.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:44 smithi167 conmon[54076]: debug 2022-01-31T22:42:44.987+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.987841+0000) 2022-01-31T22:42:45.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:44 smithi167 conmon[60316]: debug 2022-01-31T22:42:44.747+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.747921+0000) 2022-01-31T22:42:45.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:44 smithi167 conmon[60316]: debug 2022-01-31T22:42:44.942+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.942855+0000) 2022-01-31T22:42:45.215 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[41853]: debug 2022-01-31T22:42:44.942+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.943198+0000) 2022-01-31T22:42:45.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[46715]: debug 2022-01-31T22:42:44.942+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.943258+0000) 2022-01-31T22:42:45.217 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[35325]: debug 2022-01-31T22:42:44.966+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231415 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:45.217 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:44 smithi171 conmon[51620]: debug 2022-01-31T22:42:44.942+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:44.943701+0000) 2022-01-31T22:42:45.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:45 smithi171 conmon[51620]: debug 2022-01-31T22:42:45.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:45.355517+0000) 2022-01-31T22:42:45.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:45 smithi171 conmon[41853]: debug 2022-01-31T22:42:45.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:45.438607+0000) 2022-01-31T22:42:45.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:45 smithi171 conmon[46715]: debug 2022-01-31T22:42:45.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:45.488950+0000) 2022-01-31T22:42:46.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:45 smithi167 conmon[49112]: debug 2022-01-31T22:42:45.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:45.822166+0000) 2022-01-31T22:42:46.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:45 smithi167 conmon[54076]: debug 2022-01-31T22:42:45.987+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:45.988001+0000) 2022-01-31T22:42:46.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:45 smithi167 conmon[60316]: debug 2022-01-31T22:42:45.747+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:45.748063+0000) 2022-01-31T22:42:46.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:46 smithi171 conmon[51620]: debug 2022-01-31T22:42:46.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:46.355686+0000) 2022-01-31T22:42:46.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:46 smithi171 conmon[41853]: debug 2022-01-31T22:42:46.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:46.438777+0000) 2022-01-31T22:42:46.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:46 smithi171 conmon[46715]: debug 2022-01-31T22:42:46.489+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:46.489112+0000) 2022-01-31T22:42:47.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:46 smithi167 conmon[49112]: debug 2022-01-31T22:42:46.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:46.822307+0000) 2022-01-31T22:42:47.034 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:46 smithi167 conmon[60316]: debug 2022-01-31T22:42:46.747+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:46.748220+0000) 2022-01-31T22:42:47.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:46 smithi167 conmon[54076]: debug 2022-01-31T22:42:46.987+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:46.988214+0000) 2022-01-31T22:42:47.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:47 smithi171 conmon[51620]: debug 2022-01-31T22:42:47.354+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:47.355940+0000) 2022-01-31T22:42:47.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:47 smithi171 conmon[41853]: debug 2022-01-31T22:42:47.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:47.438967+0000) 2022-01-31T22:42:47.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:47 smithi171 conmon[46715]: debug 2022-01-31T22:42:47.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:47.489306+0000) 2022-01-31T22:42:48.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:47 smithi167 conmon[49112]: debug 2022-01-31T22:42:47.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:47.822493+0000) 2022-01-31T22:42:48.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:47 smithi167 conmon[54076]: debug 2022-01-31T22:42:47.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:47.988396+0000) 2022-01-31T22:42:48.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:47 smithi167 conmon[60316]: debug 2022-01-31T22:42:47.747+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:47.748401+0000) 2022-01-31T22:42:48.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:48 smithi171 conmon[41853]: debug 2022-01-31T22:42:48.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:48.439110+0000) 2022-01-31T22:42:48.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:48 smithi171 conmon[51620]: debug 2022-01-31T22:42:48.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:48.356177+0000) 2022-01-31T22:42:48.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:48 smithi171 conmon[46715]: debug 2022-01-31T22:42:48.487+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:48.489523+0000) 2022-01-31T22:42:49.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:48 smithi167 conmon[49112]: debug 2022-01-31T22:42:48.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:48.822694+0000) 2022-01-31T22:42:49.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:48 smithi167 conmon[54076]: debug 2022-01-31T22:42:48.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:48.988596+0000) 2022-01-31T22:42:49.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:48 smithi167 conmon[60316]: debug 2022-01-31T22:42:48.748+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:48.748582+0000) 2022-01-31T22:42:49.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[51620]: debug 2022-01-31T22:42:49.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.356375+0000) 2022-01-31T22:42:49.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[41853]: debug 2022-01-31T22:42:49.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.439282+0000) 2022-01-31T22:42:49.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[46715]: debug 2022-01-31T22:42:49.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.489735+0000) 2022-01-31T22:42:50.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:49 smithi167 conmon[49112]: debug 2022-01-31T22:42:49.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.822924+0000) 2022-01-31T22:42:50.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:49 smithi167 conmon[49112]: debug 2022-01-31T22:42:49.969+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.970175+0000) 2022-01-31T22:42:50.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:49 smithi167 conmon[54076]: debug 2022-01-31T22:42:49.969+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.970330+0000) 2022-01-31T22:42:50.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:49 smithi167 conmon[54076]: debug 2022-01-31T22:42:49.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.988729+0000) 2022-01-31T22:42:50.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:49 smithi167 conmon[60316]: debug 2022-01-31T22:42:49.748+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.748758+0000) 2022-01-31T22:42:50.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:49 smithi167 conmon[60316]: debug 2022-01-31T22:42:49.970+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.970736+0000) 2022-01-31T22:42:50.216 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[35325]: debug 2022-01-31T22:42:49.991+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231526 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:50.389 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[41853]: debug 2022-01-31T22:42:49.968+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.970358+0000) 2022-01-31T22:42:50.390 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[46715]: debug 2022-01-31T22:42:49.968+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.970631+0000) 2022-01-31T22:42:50.390 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:49 smithi171 conmon[51620]: debug 2022-01-31T22:42:49.969+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:49.970902+0000) 2022-01-31T22:42:50.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:50 smithi171 conmon[41853]: debug 2022-01-31T22:42:50.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:50.439486+0000) 2022-01-31T22:42:50.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:50 smithi171 conmon[51620]: debug 2022-01-31T22:42:50.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:50.356576+0000) 2022-01-31T22:42:50.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:50 smithi171 conmon[46715]: debug 2022-01-31T22:42:50.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:50.489891+0000) 2022-01-31T22:42:51.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:50 smithi167 conmon[49112]: debug 2022-01-31T22:42:50.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:50.823113+0000) 2022-01-31T22:42:51.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:50 smithi167 conmon[54076]: debug 2022-01-31T22:42:50.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:50.988876+0000) 2022-01-31T22:42:51.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:50 smithi167 conmon[60316]: debug 2022-01-31T22:42:50.748+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:50.748955+0000) 2022-01-31T22:42:51.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:51 smithi171 conmon[41853]: debug 2022-01-31T22:42:51.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:51.439631+0000) 2022-01-31T22:42:51.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:51 smithi171 conmon[51620]: debug 2022-01-31T22:42:51.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:51.356749+0000) 2022-01-31T22:42:51.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:51 smithi171 conmon[46715]: debug 2022-01-31T22:42:51.488+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:51.490001+0000) 2022-01-31T22:42:52.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:51 smithi167 conmon[49112]: debug 2022-01-31T22:42:51.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:51.823280+0000) 2022-01-31T22:42:52.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:51 smithi167 conmon[54076]: debug 2022-01-31T22:42:51.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:51.989054+0000) 2022-01-31T22:42:52.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:51 smithi167 conmon[60316]: debug 2022-01-31T22:42:51.748+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:51.749091+0000) 2022-01-31T22:42:52.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:52 smithi171 conmon[41853]: debug 2022-01-31T22:42:52.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:52.439808+0000) 2022-01-31T22:42:52.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:52 smithi171 conmon[51620]: debug 2022-01-31T22:42:52.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:52.356952+0000) 2022-01-31T22:42:52.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:52 smithi171 conmon[46715]: debug 2022-01-31T22:42:52.489+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:52.490165+0000) 2022-01-31T22:42:53.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:52 smithi167 conmon[49112]: debug 2022-01-31T22:42:52.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:52.823464+0000) 2022-01-31T22:42:53.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:52 smithi167 conmon[54076]: debug 2022-01-31T22:42:52.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:52.989257+0000) 2022-01-31T22:42:53.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:52 smithi167 conmon[60316]: debug 2022-01-31T22:42:52.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:52.749233+0000) 2022-01-31T22:42:53.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:53 smithi171 conmon[41853]: debug 2022-01-31T22:42:53.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:53.439989+0000) 2022-01-31T22:42:53.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:53 smithi171 conmon[51620]: debug 2022-01-31T22:42:53.355+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:53.357183+0000) 2022-01-31T22:42:53.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:53 smithi171 conmon[46715]: debug 2022-01-31T22:42:53.489+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:53.490347+0000) 2022-01-31T22:42:54.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:53 smithi167 conmon[49112]: debug 2022-01-31T22:42:53.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:53.823628+0000) 2022-01-31T22:42:54.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:53 smithi167 conmon[54076]: debug 2022-01-31T22:42:53.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:53.989392+0000) 2022-01-31T22:42:54.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:53 smithi167 conmon[60316]: debug 2022-01-31T22:42:53.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:53.749388+0000) 2022-01-31T22:42:54.217 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:53 smithi171 conmon[35325]: debug 2022-01-31T22:42:53.897+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:42:54.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:54 smithi171 conmon[51620]: debug 2022-01-31T22:42:54.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.357393+0000) 2022-01-31T22:42:54.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:54 smithi171 conmon[41853]: debug 2022-01-31T22:42:54.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.440196+0000) 2022-01-31T22:42:54.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:54 smithi171 conmon[46715]: debug 2022-01-31T22:42:54.489+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.490529+0000) 2022-01-31T22:42:54.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:54 smithi167 conmon[54076]: debug 2022-01-31T22:42:54.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.989604+0000) 2022-01-31T22:42:54.991 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:54 smithi167 conmon[54076]: debug 2022-01-31T22:42:54.996+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.996914+0000) 2022-01-31T22:42:54.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:54 smithi167 conmon[49112]: debug 2022-01-31T22:42:54.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.823805+0000) 2022-01-31T22:42:54.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:54 smithi167 conmon[49112]: debug 2022-01-31T22:42:54.995+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.996014+0000) 2022-01-31T22:42:54.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:54 smithi167 conmon[60316]: debug 2022-01-31T22:42:54.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.749662+0000) 2022-01-31T22:42:54.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:54 smithi167 conmon[60316]: debug 2022-01-31T22:42:54.994+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.995340+0000) 2022-01-31T22:42:55.348 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:42:55 smithi171 conmon[35325]: debug 2022-01-31T22:42:55.018+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231640 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:42:55.349 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:54 smithi171 conmon[41853]: debug 2022-01-31T22:42:54.993+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.995686+0000) 2022-01-31T22:42:55.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:54 smithi171 conmon[46715]: debug 2022-01-31T22:42:54.994+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.996329+0000) 2022-01-31T22:42:55.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:54 smithi171 conmon[51620]: debug 2022-01-31T22:42:54.994+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:54.995856+0000) 2022-01-31T22:42:55.645 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:55 smithi171 conmon[41853]: debug 2022-01-31T22:42:55.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:55.440350+0000) 2022-01-31T22:42:55.646 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:55 smithi171 conmon[46715]: debug 2022-01-31T22:42:55.489+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:55.490713+0000) 2022-01-31T22:42:55.646 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:55 smithi171 conmon[51620]: debug 2022-01-31T22:42:55.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:55.357600+0000) 2022-01-31T22:42:56.035 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:55 smithi167 conmon[49112]: debug 2022-01-31T22:42:55.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:55.823992+0000) 2022-01-31T22:42:56.035 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:55 smithi167 conmon[54076]: debug 2022-01-31T22:42:55.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:55.989735+0000) 2022-01-31T22:42:56.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:55 smithi167 conmon[60316]: debug 2022-01-31T22:42:55.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:55.749856+0000) 2022-01-31T22:42:56.481 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:56 smithi171 conmon[41853]: debug 2022-01-31T22:42:56.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:56.440500+0000) 2022-01-31T22:42:56.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:56 smithi171 conmon[51620]: debug 2022-01-31T22:42:56.356+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:56.357790+0000) 2022-01-31T22:42:56.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:56 smithi171 conmon[46715]: debug 2022-01-31T22:42:56.490+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:56.490864+0000) 2022-01-31T22:42:57.035 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:56 smithi167 conmon[49112]: debug 2022-01-31T22:42:56.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:56.824164+0000) 2022-01-31T22:42:57.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:56 smithi167 conmon[60316]: debug 2022-01-31T22:42:56.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:56.750015+0000) 2022-01-31T22:42:57.036 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:56 smithi167 conmon[54076]: debug 2022-01-31T22:42:56.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:56.989855+0000) 2022-01-31T22:42:57.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:57 smithi171 conmon[41853]: debug 2022-01-31T22:42:57.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:57.440681+0000) 2022-01-31T22:42:57.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:57 smithi171 conmon[51620]: debug 2022-01-31T22:42:57.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:57.357943+0000) 2022-01-31T22:42:57.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:57 smithi171 conmon[46715]: debug 2022-01-31T22:42:57.490+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:57.491020+0000) 2022-01-31T22:42:58.035 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:57 smithi167 conmon[49112]: debug 2022-01-31T22:42:57.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:57.824347+0000) 2022-01-31T22:42:58.036 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:57 smithi167 conmon[54076]: debug 2022-01-31T22:42:57.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:57.990024+0000) 2022-01-31T22:42:58.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:57 smithi167 conmon[60316]: debug 2022-01-31T22:42:57.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:57.750166+0000) 2022-01-31T22:42:58.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:58 smithi171 conmon[41853]: debug 2022-01-31T22:42:58.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:58.440831+0000) 2022-01-31T22:42:58.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:58 smithi171 conmon[51620]: debug 2022-01-31T22:42:58.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:58.358131+0000) 2022-01-31T22:42:58.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:58 smithi171 conmon[46715]: debug 2022-01-31T22:42:58.490+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:58.491210+0000) 2022-01-31T22:42:59.035 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:58 smithi167 conmon[49112]: debug 2022-01-31T22:42:58.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:58.824552+0000) 2022-01-31T22:42:59.036 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:58 smithi167 conmon[60316]: debug 2022-01-31T22:42:58.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:58.750327+0000) 2022-01-31T22:42:59.036 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:58 smithi167 conmon[54076]: debug 2022-01-31T22:42:58.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:58.990222+0000) 2022-01-31T22:42:59.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:42:59 smithi171 conmon[41853]: debug 2022-01-31T22:42:59.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:59.441012+0000) 2022-01-31T22:42:59.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:42:59 smithi171 conmon[51620]: debug 2022-01-31T22:42:59.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:59.358360+0000) 2022-01-31T22:42:59.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:42:59 smithi171 conmon[46715]: debug 2022-01-31T22:42:59.490+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:59.491424+0000) 2022-01-31T22:43:00.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:42:59 smithi167 conmon[49112]: debug 2022-01-31T22:42:59.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:59.824758+0000) 2022-01-31T22:43:00.013 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:42:59 smithi167 conmon[54076]: debug 2022-01-31T22:42:59.988+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:59.990430+0000) 2022-01-31T22:43:00.014 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:42:59 smithi167 conmon[60316]: debug 2022-01-31T22:42:59.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:42:59.750513+0000) 2022-01-31T22:43:00.348 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[35325]: debug 2022-01-31T22:43:00.045+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231751 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:00.349 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[41853]: debug 2022-01-31T22:43:00.021+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.022117+0000) 2022-01-31T22:43:00.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[46715]: debug 2022-01-31T22:43:00.021+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.022621+0000) 2022-01-31T22:43:00.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[51620]: debug 2022-01-31T22:43:00.021+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.022719+0000) 2022-01-31T22:43:00.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:00 smithi167 conmon[49112]: debug 2022-01-31T22:43:00.021+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.022819+0000) 2022-01-31T22:43:00.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:00 smithi167 conmon[54076]: debug 2022-01-31T22:43:00.022+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.023642+0000) 2022-01-31T22:43:00.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:00 smithi167 conmon[60316]: debug 2022-01-31T22:43:00.020+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.022471+0000) 2022-01-31T22:43:00.645 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[41853]: debug 2022-01-31T22:43:00.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.441202+0000) 2022-01-31T22:43:00.646 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[46715]: debug 2022-01-31T22:43:00.490+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.491627+0000) 2022-01-31T22:43:00.647 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:00 smithi171 conmon[51620]: debug 2022-01-31T22:43:00.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.358522+0000) 2022-01-31T22:43:00.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:00 smithi167 conmon[49112]: debug 2022-01-31T22:43:00.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.824944+0000) 2022-01-31T22:43:00.993 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:00 smithi167 conmon[54076]: debug 2022-01-31T22:43:00.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.990600+0000) 2022-01-31T22:43:00.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:00 smithi167 conmon[60316]: debug 2022-01-31T22:43:00.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:00.750648+0000) 2022-01-31T22:43:01.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:01 smithi171 conmon[41853]: debug 2022-01-31T22:43:01.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:01.441367+0000) 2022-01-31T22:43:01.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:01 smithi171 conmon[51620]: debug 2022-01-31T22:43:01.357+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:01.358675+0000) 2022-01-31T22:43:01.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:01 smithi171 conmon[46715]: debug 2022-01-31T22:43:01.491+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:01.491788+0000) 2022-01-31T22:43:01.995 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:01 smithi167 conmon[49112]: debug 2022-01-31T22:43:01.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:01.825086+0000) 2022-01-31T22:43:01.996 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:01 smithi167 conmon[54076]: debug 2022-01-31T22:43:01.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:01.990731+0000) 2022-01-31T22:43:01.997 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:01 smithi167 conmon[60316]: debug 2022-01-31T22:43:01.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:01.750751+0000) 2022-01-31T22:43:02.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:02 smithi171 conmon[41853]: debug 2022-01-31T22:43:02.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:02.441570+0000) 2022-01-31T22:43:02.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:02 smithi171 conmon[51620]: debug 2022-01-31T22:43:02.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:02.358915+0000) 2022-01-31T22:43:02.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:02 smithi171 conmon[46715]: debug 2022-01-31T22:43:02.491+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:02.491999+0000) 2022-01-31T22:43:02.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:02 smithi167 conmon[49112]: debug 2022-01-31T22:43:02.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:02.825272+0000) 2022-01-31T22:43:03.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:02 smithi167 conmon[54076]: debug 2022-01-31T22:43:02.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:02.990908+0000) 2022-01-31T22:43:03.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:02 smithi167 conmon[60316]: debug 2022-01-31T22:43:02.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:02.750945+0000) 2022-01-31T22:43:03.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:03 smithi171 conmon[41853]: debug 2022-01-31T22:43:03.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:03.441806+0000) 2022-01-31T22:43:03.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:03 smithi171 conmon[51620]: debug 2022-01-31T22:43:03.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:03.359122+0000) 2022-01-31T22:43:03.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:03 smithi171 conmon[46715]: debug 2022-01-31T22:43:03.491+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:03.492201+0000) 2022-01-31T22:43:04.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:03 smithi167 conmon[49112]: debug 2022-01-31T22:43:03.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:03.825455+0000) 2022-01-31T22:43:04.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:03 smithi167 conmon[54076]: debug 2022-01-31T22:43:03.989+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:03.991059+0000) 2022-01-31T22:43:04.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:03 smithi167 conmon[60316]: debug 2022-01-31T22:43:03.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:03.751102+0000) 2022-01-31T22:43:04.483 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:04 smithi171 conmon[51620]: debug 2022-01-31T22:43:04.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:04.359305+0000) 2022-01-31T22:43:04.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:04 smithi171 conmon[41853]: debug 2022-01-31T22:43:04.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:04.441986+0000) 2022-01-31T22:43:04.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:04 smithi171 conmon[46715]: debug 2022-01-31T22:43:04.491+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:04.492377+0000) 2022-01-31T22:43:05.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:04 smithi167 conmon[49112]: debug 2022-01-31T22:43:04.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:04.825611+0000) 2022-01-31T22:43:05.008 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:04 smithi167 conmon[54076]: debug 2022-01-31T22:43:04.990+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:04.991234+0000) 2022-01-31T22:43:05.009 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:04 smithi167 conmon[60316]: debug 2022-01-31T22:43:04.749+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:04.751298+0000) 2022-01-31T22:43:05.348 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[35325]: debug 2022-01-31T22:43:05.071+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231863 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:05.349 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[41853]: debug 2022-01-31T22:43:05.047+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.048444+0000) 2022-01-31T22:43:05.350 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[46715]: debug 2022-01-31T22:43:05.048+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.048942+0000) 2022-01-31T22:43:05.351 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[51620]: debug 2022-01-31T22:43:05.047+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.048036+0000) 2022-01-31T22:43:05.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:05 smithi167 conmon[49112]: debug 2022-01-31T22:43:05.047+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.048542+0000) 2022-01-31T22:43:05.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:05 smithi167 conmon[54076]: debug 2022-01-31T22:43:05.048+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.049638+0000) 2022-01-31T22:43:05.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:05 smithi167 conmon[60316]: debug 2022-01-31T22:43:05.048+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.050037+0000) 2022-01-31T22:43:05.646 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[41853]: debug 2022-01-31T22:43:05.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.442172+0000) 2022-01-31T22:43:05.647 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[46715]: debug 2022-01-31T22:43:05.491+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.492558+0000) 2022-01-31T22:43:05.648 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:05 smithi171 conmon[51620]: debug 2022-01-31T22:43:05.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.359468+0000) 2022-01-31T22:43:06.008 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:05 smithi167 conmon[49112]: debug 2022-01-31T22:43:05.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.825756+0000) 2022-01-31T22:43:06.009 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:05 smithi167 conmon[54076]: debug 2022-01-31T22:43:05.990+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.991398+0000) 2022-01-31T22:43:06.010 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:05 smithi167 conmon[60316]: debug 2022-01-31T22:43:05.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:05.751514+0000) 2022-01-31T22:43:06.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:06 smithi171 conmon[41853]: debug 2022-01-31T22:43:06.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:06.442335+0000) 2022-01-31T22:43:06.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:06 smithi171 conmon[51620]: debug 2022-01-31T22:43:06.358+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:06.359616+0000) 2022-01-31T22:43:06.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:06 smithi171 conmon[46715]: debug 2022-01-31T22:43:06.492+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:06.492696+0000) 2022-01-31T22:43:07.012 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:06 smithi167 conmon[49112]: debug 2022-01-31T22:43:06.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:06.825910+0000) 2022-01-31T22:43:07.012 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:06 smithi167 conmon[54076]: debug 2022-01-31T22:43:06.990+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:06.991556+0000) 2022-01-31T22:43:07.013 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:06 smithi167 conmon[60316]: debug 2022-01-31T22:43:06.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:06.751641+0000) 2022-01-31T22:43:07.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:07 smithi171 conmon[41853]: debug 2022-01-31T22:43:07.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:07.442464+0000) 2022-01-31T22:43:07.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:07 smithi171 conmon[51620]: debug 2022-01-31T22:43:07.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:07.359803+0000) 2022-01-31T22:43:07.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:07 smithi171 conmon[46715]: debug 2022-01-31T22:43:07.492+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:07.492861+0000) 2022-01-31T22:43:08.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:07 smithi167 conmon[49112]: debug 2022-01-31T22:43:07.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:07.826063+0000) 2022-01-31T22:43:08.016 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:07 smithi167 conmon[54076]: debug 2022-01-31T22:43:07.990+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:07.991733+0000) 2022-01-31T22:43:08.017 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:07 smithi167 conmon[60316]: debug 2022-01-31T22:43:07.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:07.751777+0000) 2022-01-31T22:43:08.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:08 smithi171 conmon[41853]: debug 2022-01-31T22:43:08.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:08.442637+0000) 2022-01-31T22:43:08.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:08 smithi171 conmon[51620]: debug 2022-01-31T22:43:08.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:08.360015+0000) 2022-01-31T22:43:08.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:08 smithi171 conmon[46715]: debug 2022-01-31T22:43:08.492+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:08.493042+0000) 2022-01-31T22:43:09.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:08 smithi167 conmon[49112]: debug 2022-01-31T22:43:08.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:08.826253+0000) 2022-01-31T22:43:09.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:08 smithi167 conmon[60316]: debug 2022-01-31T22:43:08.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:08.751939+0000) 2022-01-31T22:43:09.021 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:08 smithi167 conmon[54076]: debug 2022-01-31T22:43:08.991+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:08.991914+0000) 2022-01-31T22:43:09.219 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:08 smithi171 conmon[35325]: debug 2022-01-31T22:43:08.898+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:43:09.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:09 smithi171 conmon[41853]: debug 2022-01-31T22:43:09.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:09.442820+0000) 2022-01-31T22:43:09.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:09 smithi171 conmon[51620]: debug 2022-01-31T22:43:09.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:09.360239+0000) 2022-01-31T22:43:09.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:09 smithi171 conmon[46715]: debug 2022-01-31T22:43:09.492+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:09.493232+0000) 2022-01-31T22:43:10.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:09 smithi167 conmon[49112]: debug 2022-01-31T22:43:09.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:09.826422+0000) 2022-01-31T22:43:10.024 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:09 smithi167 conmon[54076]: debug 2022-01-31T22:43:09.991+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:09.992073+0000) 2022-01-31T22:43:10.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:09 smithi167 conmon[60316]: debug 2022-01-31T22:43:09.751+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:09.752102+0000) 2022-01-31T22:43:10.348 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[46715]: debug 2022-01-31T22:43:10.075+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.075834+0000) 2022-01-31T22:43:10.349 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[35325]: debug 2022-01-31T22:43:10.113+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 231975 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:10.349 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[41853]: debug 2022-01-31T22:43:10.074+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.074781+0000) 2022-01-31T22:43:10.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[51620]: debug 2022-01-31T22:43:10.073+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.074619+0000) 2022-01-31T22:43:10.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:10 smithi167 conmon[54076]: debug 2022-01-31T22:43:10.080+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.082114+0000) 2022-01-31T22:43:10.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:10 smithi167 conmon[60316]: debug 2022-01-31T22:43:10.080+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.081804+0000) 2022-01-31T22:43:10.411 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:10 smithi167 conmon[49112]: debug 2022-01-31T22:43:10.074+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.075494+0000) 2022-01-31T22:43:10.647 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[46715]: debug 2022-01-31T22:43:10.492+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.493433+0000) 2022-01-31T22:43:10.648 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[51620]: debug 2022-01-31T22:43:10.359+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.360436+0000) 2022-01-31T22:43:10.648 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:10 smithi171 conmon[41853]: debug 2022-01-31T22:43:10.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.442998+0000) 2022-01-31T22:43:11.025 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:10 smithi167 conmon[49112]: debug 2022-01-31T22:43:10.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.826617+0000) 2022-01-31T22:43:11.025 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:10 smithi167 conmon[54076]: debug 2022-01-31T22:43:10.991+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.992251+0000) 2022-01-31T22:43:11.026 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:10 smithi167 conmon[60316]: debug 2022-01-31T22:43:10.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:10.752280+0000) 2022-01-31T22:43:11.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:11 smithi171 conmon[41853]: debug 2022-01-31T22:43:11.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:11.443158+0000) 2022-01-31T22:43:11.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:11 smithi171 conmon[51620]: debug 2022-01-31T22:43:11.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:11.360604+0000) 2022-01-31T22:43:11.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:11 smithi171 conmon[46715]: debug 2022-01-31T22:43:11.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:11.493595+0000) 2022-01-31T22:43:12.028 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:11 smithi167 conmon[49112]: debug 2022-01-31T22:43:11.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:11.826763+0000) 2022-01-31T22:43:12.029 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:11 smithi167 conmon[54076]: debug 2022-01-31T22:43:11.990+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:11.992411+0000) 2022-01-31T22:43:12.030 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:11 smithi167 conmon[60316]: debug 2022-01-31T22:43:11.750+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:11.752456+0000) 2022-01-31T22:43:12.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:12 smithi171 conmon[41853]: debug 2022-01-31T22:43:12.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:12.443364+0000) 2022-01-31T22:43:12.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:12 smithi171 conmon[51620]: debug 2022-01-31T22:43:12.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:12.360833+0000) 2022-01-31T22:43:12.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:12 smithi171 conmon[46715]: debug 2022-01-31T22:43:12.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:12.493769+0000) 2022-01-31T22:43:13.033 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:12 smithi167 conmon[49112]: debug 2022-01-31T22:43:12.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:12.826970+0000) 2022-01-31T22:43:13.034 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:12 smithi167 conmon[54076]: debug 2022-01-31T22:43:12.991+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:12.992568+0000) 2022-01-31T22:43:13.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:12 smithi167 conmon[60316]: debug 2022-01-31T22:43:12.751+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:12.752644+0000) 2022-01-31T22:43:13.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:13 smithi171 conmon[41853]: debug 2022-01-31T22:43:13.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:13.443582+0000) 2022-01-31T22:43:13.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:13 smithi171 conmon[51620]: debug 2022-01-31T22:43:13.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:13.361017+0000) 2022-01-31T22:43:13.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:13 smithi171 conmon[46715]: debug 2022-01-31T22:43:13.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:13.493952+0000) 2022-01-31T22:43:14.037 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:13 smithi167 conmon[49112]: debug 2022-01-31T22:43:13.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:13.827151+0000) 2022-01-31T22:43:14.037 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:13 smithi167 conmon[54076]: debug 2022-01-31T22:43:13.991+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:13.992714+0000) 2022-01-31T22:43:14.038 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:13 smithi167 conmon[60316]: debug 2022-01-31T22:43:13.752+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:13.752828+0000) 2022-01-31T22:43:14.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:14 smithi171 conmon[41853]: debug 2022-01-31T22:43:14.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:14.443800+0000) 2022-01-31T22:43:14.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:14 smithi171 conmon[51620]: debug 2022-01-31T22:43:14.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:14.361166+0000) 2022-01-31T22:43:14.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:14 smithi171 conmon[46715]: debug 2022-01-31T22:43:14.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:14.494160+0000) 2022-01-31T22:43:15.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:14 smithi167 conmon[49112]: debug 2022-01-31T22:43:14.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:14.827328+0000) 2022-01-31T22:43:15.039 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:14 smithi167 conmon[60316]: debug 2022-01-31T22:43:14.752+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:14.753054+0000) 2022-01-31T22:43:15.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:14 smithi167 conmon[54076]: debug 2022-01-31T22:43:14.992+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:14.992907+0000) 2022-01-31T22:43:15.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:15 smithi167 conmon[60316]: debug 2022-01-31T22:43:15.116+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.117912+0000) 2022-01-31T22:43:15.411 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:15 smithi167 conmon[49112]: debug 2022-01-31T22:43:15.116+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.117769+0000) 2022-01-31T22:43:15.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:15 smithi167 conmon[54076]: debug 2022-01-31T22:43:15.116+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.118283+0000) 2022-01-31T22:43:15.434 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[35325]: debug 2022-01-31T22:43:15.140+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232087 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:15.435 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[41853]: debug 2022-01-31T22:43:15.116+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.116901+0000) 2022-01-31T22:43:15.435 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[46715]: debug 2022-01-31T22:43:15.116+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.117294+0000) 2022-01-31T22:43:15.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[51620]: debug 2022-01-31T22:43:15.116+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.116921+0000) 2022-01-31T22:43:15.436 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[51620]: debug 2022-01-31T22:43:15.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.361303+0000) 2022-01-31T22:43:15.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[41853]: debug 2022-01-31T22:43:15.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.443990+0000) 2022-01-31T22:43:15.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:15 smithi171 conmon[46715]: debug 2022-01-31T22:43:15.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.494363+0000) 2022-01-31T22:43:16.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:15 smithi167 conmon[49112]: debug 2022-01-31T22:43:15.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.827530+0000) 2022-01-31T22:43:16.039 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:15 smithi167 conmon[60316]: debug 2022-01-31T22:43:15.752+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.753263+0000) 2022-01-31T22:43:16.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:15 smithi167 conmon[54076]: debug 2022-01-31T22:43:15.992+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:15.993101+0000) 2022-01-31T22:43:16.485 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:16 smithi171 conmon[41853]: debug 2022-01-31T22:43:16.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:16.444119+0000) 2022-01-31T22:43:16.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:16 smithi171 conmon[51620]: debug 2022-01-31T22:43:16.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:16.361475+0000) 2022-01-31T22:43:16.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:16 smithi171 conmon[46715]: debug 2022-01-31T22:43:16.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:16.494509+0000) 2022-01-31T22:43:17.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:16 smithi167 conmon[49112]: debug 2022-01-31T22:43:16.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:16.827682+0000) 2022-01-31T22:43:17.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:16 smithi167 conmon[54076]: debug 2022-01-31T22:43:16.991+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:16.993260+0000) 2022-01-31T22:43:17.039 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:16 smithi167 conmon[60316]: debug 2022-01-31T22:43:16.752+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:16.753439+0000) 2022-01-31T22:43:17.485 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:17 smithi171 conmon[41853]: debug 2022-01-31T22:43:17.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:17.444289+0000) 2022-01-31T22:43:17.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:17 smithi171 conmon[51620]: debug 2022-01-31T22:43:17.360+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:17.361707+0000) 2022-01-31T22:43:17.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:17 smithi171 conmon[46715]: debug 2022-01-31T22:43:17.493+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:17.494691+0000) 2022-01-31T22:43:18.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:17 smithi167 conmon[49112]: debug 2022-01-31T22:43:17.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:17.827866+0000) 2022-01-31T22:43:18.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:17 smithi167 conmon[54076]: debug 2022-01-31T22:43:17.992+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:17.993436+0000) 2022-01-31T22:43:18.039 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:17 smithi167 conmon[60316]: debug 2022-01-31T22:43:17.753+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:17.753654+0000) 2022-01-31T22:43:18.485 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:18 smithi171 conmon[41853]: debug 2022-01-31T22:43:18.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:18.444471+0000) 2022-01-31T22:43:18.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:18 smithi171 conmon[51620]: debug 2022-01-31T22:43:18.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:18.361962+0000) 2022-01-31T22:43:18.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:18 smithi171 conmon[46715]: debug 2022-01-31T22:43:18.494+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:18.494866+0000) 2022-01-31T22:43:19.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:18 smithi167 conmon[49112]: debug 2022-01-31T22:43:18.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:18.828058+0000) 2022-01-31T22:43:19.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:18 smithi167 conmon[54076]: debug 2022-01-31T22:43:18.993+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:18.993619+0000) 2022-01-31T22:43:19.039 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:18 smithi167 conmon[60316]: debug 2022-01-31T22:43:18.752+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:18.753831+0000) 2022-01-31T22:43:19.485 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:19 smithi171 conmon[51620]: debug 2022-01-31T22:43:19.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:19.362183+0000) 2022-01-31T22:43:19.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:19 smithi171 conmon[41853]: debug 2022-01-31T22:43:19.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:19.444616+0000) 2022-01-31T22:43:19.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:19 smithi171 conmon[46715]: debug 2022-01-31T22:43:19.494+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:19.495042+0000) 2022-01-31T22:43:20.038 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:19 smithi167 conmon[49112]: debug 2022-01-31T22:43:19.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:19.828249+0000) 2022-01-31T22:43:20.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:19 smithi167 conmon[54076]: debug 2022-01-31T22:43:19.993+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:19.993782+0000) 2022-01-31T22:43:20.040 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:19 smithi167 conmon[60316]: debug 2022-01-31T22:43:19.753+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:19.753950+0000) 2022-01-31T22:43:20.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:20 smithi167 conmon[49112]: debug 2022-01-31T22:43:20.142+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.144142+0000) 2022-01-31T22:43:20.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:20 smithi167 conmon[54076]: debug 2022-01-31T22:43:20.142+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.143711+0000) 2022-01-31T22:43:20.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:20 smithi167 conmon[60316]: debug 2022-01-31T22:43:20.142+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.144445+0000) 2022-01-31T22:43:20.435 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[35325]: debug 2022-01-31T22:43:20.166+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232197 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:20.436 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[41853]: debug 2022-01-31T22:43:20.141+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.142668+0000) 2022-01-31T22:43:20.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[46715]: debug 2022-01-31T22:43:20.143+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.144397+0000) 2022-01-31T22:43:20.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[51620]: debug 2022-01-31T22:43:20.142+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.143329+0000) 2022-01-31T22:43:20.437 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[51620]: debug 2022-01-31T22:43:20.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.362360+0000) 2022-01-31T22:43:20.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[41853]: debug 2022-01-31T22:43:20.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.444851+0000) 2022-01-31T22:43:20.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:20 smithi171 conmon[46715]: debug 2022-01-31T22:43:20.494+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.495258+0000) 2022-01-31T22:43:21.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:20 smithi167 conmon[49112]: debug 2022-01-31T22:43:20.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.828438+0000) 2022-01-31T22:43:21.039 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:20 smithi167 conmon[54076]: debug 2022-01-31T22:43:20.993+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.993899+0000) 2022-01-31T22:43:21.040 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:20 smithi167 conmon[60316]: debug 2022-01-31T22:43:20.753+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:20.754107+0000) 2022-01-31T22:43:21.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:21 smithi171 conmon[41853]: debug 2022-01-31T22:43:21.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:21.444989+0000) 2022-01-31T22:43:21.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:21 smithi171 conmon[51620]: debug 2022-01-31T22:43:21.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:21.362509+0000) 2022-01-31T22:43:21.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:21 smithi171 conmon[46715]: debug 2022-01-31T22:43:21.494+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:21.495401+0000) 2022-01-31T22:43:22.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:21 smithi167 conmon[49112]: debug 2022-01-31T22:43:21.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:21.828590+0000) 2022-01-31T22:43:22.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:21 smithi167 conmon[54076]: debug 2022-01-31T22:43:21.993+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:21.994050+0000) 2022-01-31T22:43:22.040 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:21 smithi167 conmon[60316]: debug 2022-01-31T22:43:21.753+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:21.754243+0000) 2022-01-31T22:43:22.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:22 smithi171 conmon[41853]: debug 2022-01-31T22:43:22.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:22.445167+0000) 2022-01-31T22:43:22.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:22 smithi171 conmon[51620]: debug 2022-01-31T22:43:22.361+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:22.362715+0000) 2022-01-31T22:43:22.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:22 smithi171 conmon[46715]: debug 2022-01-31T22:43:22.494+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:22.495506+0000) 2022-01-31T22:43:23.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:22 smithi167 conmon[49112]: debug 2022-01-31T22:43:22.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:22.828692+0000) 2022-01-31T22:43:23.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:22 smithi167 conmon[54076]: debug 2022-01-31T22:43:22.993+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:22.994244+0000) 2022-01-31T22:43:23.040 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:22 smithi167 conmon[60316]: debug 2022-01-31T22:43:22.753+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:22.754428+0000) 2022-01-31T22:43:23.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:23 smithi171 conmon[41853]: debug 2022-01-31T22:43:23.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:23.445353+0000) 2022-01-31T22:43:23.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:23 smithi171 conmon[51620]: debug 2022-01-31T22:43:23.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:23.362909+0000) 2022-01-31T22:43:23.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:23 smithi171 conmon[46715]: debug 2022-01-31T22:43:23.495+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:23.495657+0000) 2022-01-31T22:43:24.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:23 smithi167 conmon[54076]: debug 2022-01-31T22:43:23.993+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:23.994429+0000) 2022-01-31T22:43:24.040 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:23 smithi167 conmon[60316]: debug 2022-01-31T22:43:23.754+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:23.754587+0000) 2022-01-31T22:43:24.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:23 smithi167 conmon[49112]: debug 2022-01-31T22:43:23.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:23.828844+0000) 2022-01-31T22:43:24.222 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:23 smithi171 conmon[35325]: debug 2022-01-31T22:43:23.899+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:43:24.486 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:24 smithi171 conmon[51620]: debug 2022-01-31T22:43:24.362+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:24.363148+0000) 2022-01-31T22:43:24.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:24 smithi171 conmon[41853]: debug 2022-01-31T22:43:24.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:24.445548+0000) 2022-01-31T22:43:24.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:24 smithi171 conmon[46715]: debug 2022-01-31T22:43:24.495+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:24.495845+0000) 2022-01-31T22:43:25.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:24 smithi167 conmon[49112]: debug 2022-01-31T22:43:24.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:24.829006+0000) 2022-01-31T22:43:25.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:24 smithi167 conmon[54076]: debug 2022-01-31T22:43:24.994+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:24.994634+0000) 2022-01-31T22:43:25.040 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:24 smithi167 conmon[60316]: debug 2022-01-31T22:43:24.754+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:24.754782+0000) 2022-01-31T22:43:25.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:25 smithi167 conmon[49112]: debug 2022-01-31T22:43:25.169+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.170043+0000) 2022-01-31T22:43:25.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:25 smithi167 conmon[54076]: debug 2022-01-31T22:43:25.169+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.169757+0000) 2022-01-31T22:43:25.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:25 smithi167 conmon[60316]: debug 2022-01-31T22:43:25.170+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.171085+0000) 2022-01-31T22:43:25.436 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[35325]: debug 2022-01-31T22:43:25.193+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232308 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:25.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[46715]: debug 2022-01-31T22:43:25.170+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.170838+0000) 2022-01-31T22:43:25.437 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[41853]: debug 2022-01-31T22:43:25.169+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.170537+0000) 2022-01-31T22:43:25.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[51620]: debug 2022-01-31T22:43:25.168+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.169295+0000) 2022-01-31T22:43:25.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[51620]: debug 2022-01-31T22:43:25.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.363363+0000) 2022-01-31T22:43:25.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[46715]: debug 2022-01-31T22:43:25.495+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.495961+0000) 2022-01-31T22:43:25.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:25 smithi171 conmon[41853]: debug 2022-01-31T22:43:25.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.445741+0000) 2022-01-31T22:43:26.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:25 smithi167 conmon[49112]: debug 2022-01-31T22:43:25.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.829188+0000) 2022-01-31T22:43:26.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:25 smithi167 conmon[54076]: debug 2022-01-31T22:43:25.994+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.994841+0000) 2022-01-31T22:43:26.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:25 smithi167 conmon[60316]: debug 2022-01-31T22:43:25.754+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:25.754958+0000) 2022-01-31T22:43:26.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:26 smithi171 conmon[41853]: debug 2022-01-31T22:43:26.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:26.445914+0000) 2022-01-31T22:43:26.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:26 smithi171 conmon[51620]: debug 2022-01-31T22:43:26.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:26.363524+0000) 2022-01-31T22:43:26.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:26 smithi171 conmon[46715]: debug 2022-01-31T22:43:26.495+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:26.496067+0000) 2022-01-31T22:43:27.039 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:26 smithi167 conmon[49112]: debug 2022-01-31T22:43:26.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:26.829343+0000) 2022-01-31T22:43:27.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:26 smithi167 conmon[54076]: debug 2022-01-31T22:43:26.994+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:26.995013+0000) 2022-01-31T22:43:27.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:26 smithi167 conmon[60316]: debug 2022-01-31T22:43:26.754+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:26.755120+0000) 2022-01-31T22:43:27.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:27 smithi171 conmon[41853]: debug 2022-01-31T22:43:27.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:27.446108+0000) 2022-01-31T22:43:27.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:27 smithi171 conmon[51620]: debug 2022-01-31T22:43:27.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:27.363738+0000) 2022-01-31T22:43:27.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:27 smithi171 conmon[46715]: debug 2022-01-31T22:43:27.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:27.496243+0000) 2022-01-31T22:43:28.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:27 smithi167 conmon[49112]: debug 2022-01-31T22:43:27.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:27.829541+0000) 2022-01-31T22:43:28.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:27 smithi167 conmon[54076]: debug 2022-01-31T22:43:27.994+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:27.995179+0000) 2022-01-31T22:43:28.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:27 smithi167 conmon[60316]: debug 2022-01-31T22:43:27.754+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:27.755324+0000) 2022-01-31T22:43:28.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:28 smithi171 conmon[41853]: debug 2022-01-31T22:43:28.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:28.446235+0000) 2022-01-31T22:43:28.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:28 smithi171 conmon[51620]: debug 2022-01-31T22:43:28.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:28.363935+0000) 2022-01-31T22:43:28.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:28 smithi171 conmon[46715]: debug 2022-01-31T22:43:28.495+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:28.496418+0000) 2022-01-31T22:43:29.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:28 smithi167 conmon[49112]: debug 2022-01-31T22:43:28.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:28.829708+0000) 2022-01-31T22:43:29.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:28 smithi167 conmon[54076]: debug 2022-01-31T22:43:28.995+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:28.995380+0000) 2022-01-31T22:43:29.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:28 smithi167 conmon[60316]: debug 2022-01-31T22:43:28.755+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:28.755545+0000) 2022-01-31T22:43:29.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:29 smithi171 conmon[41853]: debug 2022-01-31T22:43:29.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:29.446422+0000) 2022-01-31T22:43:29.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:29 smithi171 conmon[51620]: debug 2022-01-31T22:43:29.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:29.364126+0000) 2022-01-31T22:43:29.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:29 smithi171 conmon[46715]: debug 2022-01-31T22:43:29.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:29.496596+0000) 2022-01-31T22:43:30.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:29 smithi167 conmon[49112]: debug 2022-01-31T22:43:29.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:29.829855+0000) 2022-01-31T22:43:30.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:29 smithi167 conmon[54076]: debug 2022-01-31T22:43:29.995+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:29.995566+0000) 2022-01-31T22:43:30.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:29 smithi167 conmon[60316]: debug 2022-01-31T22:43:29.755+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:29.755734+0000) 2022-01-31T22:43:30.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:30 smithi167 conmon[49112]: debug 2022-01-31T22:43:30.196+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.197185+0000) 2022-01-31T22:43:30.411 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:30 smithi167 conmon[54076]: debug 2022-01-31T22:43:30.197+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.197507+0000) 2022-01-31T22:43:30.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:30 smithi167 conmon[60316]: debug 2022-01-31T22:43:30.196+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.196829+0000) 2022-01-31T22:43:30.437 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[35325]: debug 2022-01-31T22:43:30.218+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232420 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:30.437 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[41853]: debug 2022-01-31T22:43:30.197+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.197749+0000) 2022-01-31T22:43:30.438 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[46715]: debug 2022-01-31T22:43:30.196+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.197009+0000) 2022-01-31T22:43:30.438 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[51620]: debug 2022-01-31T22:43:30.196+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.196765+0000) 2022-01-31T22:43:30.439 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[51620]: debug 2022-01-31T22:43:30.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.364338+0000) 2022-01-31T22:43:30.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[46715]: debug 2022-01-31T22:43:30.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.496723+0000) 2022-01-31T22:43:30.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:30 smithi171 conmon[41853]: debug 2022-01-31T22:43:30.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.446574+0000) 2022-01-31T22:43:31.040 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:30 smithi167 conmon[54076]: debug 2022-01-31T22:43:30.995+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.995736+0000) 2022-01-31T22:43:31.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:30 smithi167 conmon[60316]: debug 2022-01-31T22:43:30.755+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.755901+0000) 2022-01-31T22:43:31.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:30 smithi167 conmon[49112]: debug 2022-01-31T22:43:30.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:30.829983+0000) 2022-01-31T22:43:31.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:31 smithi171 conmon[41853]: debug 2022-01-31T22:43:31.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:31.446710+0000) 2022-01-31T22:43:31.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:31 smithi171 conmon[51620]: debug 2022-01-31T22:43:31.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:31.364500+0000) 2022-01-31T22:43:31.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:31 smithi171 conmon[46715]: debug 2022-01-31T22:43:31.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:31.496857+0000) 2022-01-31T22:43:32.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:31 smithi167 conmon[49112]: debug 2022-01-31T22:43:31.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:31.830135+0000) 2022-01-31T22:43:32.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:31 smithi167 conmon[54076]: debug 2022-01-31T22:43:31.995+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:31.995902+0000) 2022-01-31T22:43:32.041 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:31 smithi167 conmon[60316]: debug 2022-01-31T22:43:31.755+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:31.756034+0000) 2022-01-31T22:43:32.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:32 smithi171 conmon[51620]: debug 2022-01-31T22:43:32.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:32.364681+0000) 2022-01-31T22:43:32.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:32 smithi171 conmon[41853]: debug 2022-01-31T22:43:32.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:32.446916+0000) 2022-01-31T22:43:32.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:32 smithi171 conmon[46715]: debug 2022-01-31T22:43:32.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:32.497042+0000) 2022-01-31T22:43:33.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:32 smithi167 conmon[49112]: debug 2022-01-31T22:43:32.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:32.830316+0000) 2022-01-31T22:43:33.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:32 smithi167 conmon[60316]: debug 2022-01-31T22:43:32.755+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:32.756227+0000) 2022-01-31T22:43:33.042 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:32 smithi167 conmon[54076]: debug 2022-01-31T22:43:32.995+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:32.996088+0000) 2022-01-31T22:43:33.487 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:33 smithi171 conmon[51620]: debug 2022-01-31T22:43:33.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:33.364774+0000) 2022-01-31T22:43:33.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:33 smithi171 conmon[41853]: debug 2022-01-31T22:43:33.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:33.447100+0000) 2022-01-31T22:43:33.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:33 smithi171 conmon[46715]: debug 2022-01-31T22:43:33.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:33.497223+0000) 2022-01-31T22:43:34.040 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:33 smithi167 conmon[49112]: debug 2022-01-31T22:43:33.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:33.830533+0000) 2022-01-31T22:43:34.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:33 smithi167 conmon[54076]: debug 2022-01-31T22:43:33.995+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:33.996272+0000) 2022-01-31T22:43:34.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:33 smithi167 conmon[60316]: debug 2022-01-31T22:43:33.755+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:33.756411+0000) 2022-01-31T22:43:34.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:34 smithi171 conmon[41853]: debug 2022-01-31T22:43:34.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:34.447322+0000) 2022-01-31T22:43:34.488 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:34 smithi171 conmon[51620]: debug 2022-01-31T22:43:34.363+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:34.364974+0000) 2022-01-31T22:43:34.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:34 smithi171 conmon[46715]: debug 2022-01-31T22:43:34.495+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:34.497379+0000) 2022-01-31T22:43:35.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:34 smithi167 conmon[49112]: debug 2022-01-31T22:43:34.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:34.830723+0000) 2022-01-31T22:43:35.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:34 smithi167 conmon[54076]: debug 2022-01-31T22:43:34.996+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:34.996444+0000) 2022-01-31T22:43:35.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:34 smithi167 conmon[60316]: debug 2022-01-31T22:43:34.756+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:34.756577+0000) 2022-01-31T22:43:35.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:35 smithi167 conmon[49112]: debug 2022-01-31T22:43:35.223+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.224090+0000) 2022-01-31T22:43:35.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:35 smithi167 conmon[54076]: debug 2022-01-31T22:43:35.223+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.223656+0000) 2022-01-31T22:43:35.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:35 smithi167 conmon[60316]: debug 2022-01-31T22:43:35.222+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.223160+0000) 2022-01-31T22:43:35.488 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[35325]: debug 2022-01-31T22:43:35.244+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232531 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:35.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[46715]: debug 2022-01-31T22:43:35.220+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.221772+0000) 2022-01-31T22:43:35.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[41853]: debug 2022-01-31T22:43:35.220+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.222231+0000) 2022-01-31T22:43:35.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[41853]: debug 2022-01-31T22:43:35.445+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.447548+0000) 2022-01-31T22:43:35.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[51620]: debug 2022-01-31T22:43:35.221+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.223363+0000) 2022-01-31T22:43:35.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[51620]: debug 2022-01-31T22:43:35.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.365115+0000) 2022-01-31T22:43:35.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:35 smithi171 conmon[46715]: debug 2022-01-31T22:43:35.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.497537+0000) 2022-01-31T22:43:36.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:35 smithi167 conmon[54076]: debug 2022-01-31T22:43:35.996+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.996601+0000) 2022-01-31T22:43:36.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:35 smithi167 conmon[49112]: debug 2022-01-31T22:43:35.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.830885+0000) 2022-01-31T22:43:36.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:35 smithi167 conmon[60316]: debug 2022-01-31T22:43:35.756+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:35.756761+0000) 2022-01-31T22:43:36.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:36 smithi171 conmon[41853]: debug 2022-01-31T22:43:36.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:36.447694+0000) 2022-01-31T22:43:36.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:36 smithi171 conmon[51620]: debug 2022-01-31T22:43:36.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:36.365296+0000) 2022-01-31T22:43:36.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:36 smithi171 conmon[46715]: debug 2022-01-31T22:43:36.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:36.497676+0000) 2022-01-31T22:43:37.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:36 smithi167 conmon[49112]: debug 2022-01-31T22:43:36.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:36.831000+0000) 2022-01-31T22:43:37.042 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:36 smithi167 conmon[54076]: debug 2022-01-31T22:43:36.996+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:36.996752+0000) 2022-01-31T22:43:37.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:36 smithi167 conmon[60316]: debug 2022-01-31T22:43:36.756+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:36.756885+0000) 2022-01-31T22:43:37.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:37 smithi171 conmon[41853]: debug 2022-01-31T22:43:37.446+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:37.447868+0000) 2022-01-31T22:43:37.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:37 smithi171 conmon[51620]: debug 2022-01-31T22:43:37.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:37.365514+0000) 2022-01-31T22:43:37.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:37 smithi171 conmon[46715]: debug 2022-01-31T22:43:37.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:37.497892+0000) 2022-01-31T22:43:38.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:37 smithi167 conmon[49112]: debug 2022-01-31T22:43:37.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:37.831176+0000) 2022-01-31T22:43:38.042 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:37 smithi167 conmon[54076]: debug 2022-01-31T22:43:37.996+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:37.996938+0000) 2022-01-31T22:43:38.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:37 smithi167 conmon[60316]: debug 2022-01-31T22:43:37.756+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:37.757041+0000) 2022-01-31T22:43:38.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:38 smithi171 conmon[41853]: debug 2022-01-31T22:43:38.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:38.448035+0000) 2022-01-31T22:43:38.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:38 smithi171 conmon[51620]: debug 2022-01-31T22:43:38.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:38.365700+0000) 2022-01-31T22:43:38.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:38 smithi171 conmon[46715]: debug 2022-01-31T22:43:38.497+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:38.498076+0000) 2022-01-31T22:43:39.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:38 smithi167 conmon[49112]: debug 2022-01-31T22:43:38.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:38.831352+0000) 2022-01-31T22:43:39.042 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:38 smithi167 conmon[54076]: debug 2022-01-31T22:43:38.996+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:38.997116+0000) 2022-01-31T22:43:39.043 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:38 smithi167 conmon[60316]: debug 2022-01-31T22:43:38.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:38.757228+0000) 2022-01-31T22:43:39.225 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:38 smithi171 conmon[35325]: debug 2022-01-31T22:43:38.899+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:43:39.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:39 smithi171 conmon[41853]: debug 2022-01-31T22:43:39.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:39.448194+0000) 2022-01-31T22:43:39.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:39 smithi171 conmon[51620]: debug 2022-01-31T22:43:39.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:39.365833+0000) 2022-01-31T22:43:39.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:39 smithi171 conmon[46715]: debug 2022-01-31T22:43:39.497+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:39.498272+0000) 2022-01-31T22:43:40.041 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:39 smithi167 conmon[49112]: debug 2022-01-31T22:43:39.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:39.831535+0000) 2022-01-31T22:43:40.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:39 smithi167 conmon[60316]: debug 2022-01-31T22:43:39.756+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:39.757410+0000) 2022-01-31T22:43:40.043 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:39 smithi167 conmon[54076]: debug 2022-01-31T22:43:39.996+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:39.997266+0000) 2022-01-31T22:43:40.410 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:40 smithi167 conmon[49112]: debug 2022-01-31T22:43:40.250+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.250643+0000) 2022-01-31T22:43:40.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:40 smithi167 conmon[54076]: debug 2022-01-31T22:43:40.248+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.249275+0000) 2022-01-31T22:43:40.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:40 smithi167 conmon[60316]: debug 2022-01-31T22:43:40.248+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.249120+0000) 2022-01-31T22:43:40.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[46715]: debug 2022-01-31T22:43:40.247+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.248879+0000) 2022-01-31T22:43:40.489 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[35325]: debug 2022-01-31T22:43:40.274+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232644 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:40.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[41853]: debug 2022-01-31T22:43:40.246+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.248429+0000) 2022-01-31T22:43:40.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[41853]: debug 2022-01-31T22:43:40.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.448385+0000) 2022-01-31T22:43:40.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[51620]: debug 2022-01-31T22:43:40.248+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.250230+0000) 2022-01-31T22:43:40.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[51620]: debug 2022-01-31T22:43:40.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.366009+0000) 2022-01-31T22:43:40.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:40 smithi171 conmon[46715]: debug 2022-01-31T22:43:40.496+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.498437+0000) 2022-01-31T22:43:40.987 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:40 smithi167 conmon[49112]: debug 2022-01-31T22:43:40.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.831668+0000) 2022-01-31T22:43:40.988 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:40 smithi167 conmon[60316]: debug 2022-01-31T22:43:40.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.757600+0000) 2022-01-31T22:43:41.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:40 smithi167 conmon[54076]: debug 2022-01-31T22:43:40.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:40.997483+0000) 2022-01-31T22:43:41.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:41 smithi171 conmon[41853]: debug 2022-01-31T22:43:41.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:41.448528+0000) 2022-01-31T22:43:41.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:41 smithi171 conmon[51620]: debug 2022-01-31T22:43:41.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:41.366112+0000) 2022-01-31T22:43:41.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:41 smithi171 conmon[46715]: debug 2022-01-31T22:43:41.497+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:41.498589+0000) 2022-01-31T22:43:42.042 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:41 smithi167 conmon[49112]: debug 2022-01-31T22:43:41.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:41.831816+0000) 2022-01-31T22:43:42.043 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:41 smithi167 conmon[54076]: debug 2022-01-31T22:43:41.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:41.997702+0000) 2022-01-31T22:43:42.043 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:41 smithi167 conmon[60316]: debug 2022-01-31T22:43:41.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:41.757714+0000) 2022-01-31T22:43:42.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:42 smithi171 conmon[41853]: debug 2022-01-31T22:43:42.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:42.448677+0000) 2022-01-31T22:43:42.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:42 smithi171 conmon[51620]: debug 2022-01-31T22:43:42.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:42.366216+0000) 2022-01-31T22:43:42.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:42 smithi171 conmon[46715]: debug 2022-01-31T22:43:42.497+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:42.498771+0000) 2022-01-31T22:43:43.042 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:42 smithi167 conmon[49112]: debug 2022-01-31T22:43:42.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:42.831998+0000) 2022-01-31T22:43:43.043 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:42 smithi167 conmon[54076]: debug 2022-01-31T22:43:42.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:42.997882+0000) 2022-01-31T22:43:43.043 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:42 smithi167 conmon[60316]: debug 2022-01-31T22:43:42.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:42.757879+0000) 2022-01-31T22:43:43.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:43 smithi171 conmon[41853]: debug 2022-01-31T22:43:43.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:43.448852+0000) 2022-01-31T22:43:43.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:43 smithi171 conmon[51620]: debug 2022-01-31T22:43:43.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:43.366382+0000) 2022-01-31T22:43:43.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:43 smithi171 conmon[46715]: debug 2022-01-31T22:43:43.498+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:43.498898+0000) 2022-01-31T22:43:44.042 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:43 smithi167 conmon[49112]: debug 2022-01-31T22:43:43.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:43.832166+0000) 2022-01-31T22:43:44.043 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:43 smithi167 conmon[54076]: debug 2022-01-31T22:43:43.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:43.998061+0000) 2022-01-31T22:43:44.044 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:43 smithi167 conmon[60316]: debug 2022-01-31T22:43:43.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:43.758064+0000) 2022-01-31T22:43:44.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:44 smithi171 conmon[41853]: debug 2022-01-31T22:43:44.447+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:44.449009+0000) 2022-01-31T22:43:44.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:44 smithi171 conmon[51620]: debug 2022-01-31T22:43:44.364+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:44.366522+0000) 2022-01-31T22:43:44.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:44 smithi171 conmon[46715]: debug 2022-01-31T22:43:44.498+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:44.499096+0000) 2022-01-31T22:43:45.042 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:44 smithi167 conmon[49112]: debug 2022-01-31T22:43:44.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:44.832321+0000) 2022-01-31T22:43:45.043 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:44 smithi167 conmon[54076]: debug 2022-01-31T22:43:44.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:44.998271+0000) 2022-01-31T22:43:45.043 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:44 smithi167 conmon[60316]: debug 2022-01-31T22:43:44.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:44.758248+0000) 2022-01-31T22:43:45.409 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:45 smithi167 conmon[49112]: debug 2022-01-31T22:43:45.278+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.280038+0000) 2022-01-31T22:43:45.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:45 smithi167 conmon[54076]: debug 2022-01-31T22:43:45.278+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.280096+0000) 2022-01-31T22:43:45.411 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:45 smithi167 conmon[60316]: debug 2022-01-31T22:43:45.277+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.278999+0000) 2022-01-31T22:43:45.489 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[35325]: debug 2022-01-31T22:43:45.302+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232770 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:45.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[46715]: debug 2022-01-31T22:43:45.278+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.278840+0000) 2022-01-31T22:43:45.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[41853]: debug 2022-01-31T22:43:45.277+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.278081+0000) 2022-01-31T22:43:45.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[41853]: debug 2022-01-31T22:43:45.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.449154+0000) 2022-01-31T22:43:45.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[51620]: debug 2022-01-31T22:43:45.278+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.279620+0000) 2022-01-31T22:43:45.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[51620]: debug 2022-01-31T22:43:45.365+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.366676+0000) 2022-01-31T22:43:45.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:45 smithi171 conmon[46715]: debug 2022-01-31T22:43:45.498+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.499294+0000) 2022-01-31T22:43:46.042 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:45 smithi167 conmon[49112]: debug 2022-01-31T22:43:45.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.832531+0000) 2022-01-31T22:43:46.043 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:45 smithi167 conmon[60316]: debug 2022-01-31T22:43:45.756+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.758420+0000) 2022-01-31T22:43:46.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:45 smithi167 conmon[54076]: debug 2022-01-31T22:43:45.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:45.998456+0000) 2022-01-31T22:43:46.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:46 smithi171 conmon[41853]: debug 2022-01-31T22:43:46.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:46.449273+0000) 2022-01-31T22:43:46.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:46 smithi171 conmon[51620]: debug 2022-01-31T22:43:46.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:46.366829+0000) 2022-01-31T22:43:46.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:46 smithi171 conmon[46715]: debug 2022-01-31T22:43:46.498+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:46.499430+0000) 2022-01-31T22:43:47.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:46 smithi167 conmon[49112]: debug 2022-01-31T22:43:46.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:46.832702+0000) 2022-01-31T22:43:47.043 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:46 smithi167 conmon[60316]: debug 2022-01-31T22:43:46.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:46.758606+0000) 2022-01-31T22:43:47.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:46 smithi167 conmon[54076]: debug 2022-01-31T22:43:46.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:46.998624+0000) 2022-01-31T22:43:47.489 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:47 smithi171 conmon[51620]: debug 2022-01-31T22:43:47.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:47.366974+0000) 2022-01-31T22:43:47.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:47 smithi171 conmon[41853]: debug 2022-01-31T22:43:47.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:47.449429+0000) 2022-01-31T22:43:47.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:47 smithi171 conmon[46715]: debug 2022-01-31T22:43:47.498+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:47.499584+0000) 2022-01-31T22:43:48.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:47 smithi167 conmon[49112]: debug 2022-01-31T22:43:47.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:47.832827+0000) 2022-01-31T22:43:48.044 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:47 smithi167 conmon[60316]: debug 2022-01-31T22:43:47.757+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:47.758799+0000) 2022-01-31T22:43:48.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:47 smithi167 conmon[54076]: debug 2022-01-31T22:43:47.997+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:47.998782+0000) 2022-01-31T22:43:48.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:48 smithi171 conmon[51620]: debug 2022-01-31T22:43:48.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:48.367162+0000) 2022-01-31T22:43:48.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:48 smithi171 conmon[41853]: debug 2022-01-31T22:43:48.448+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:48.449576+0000) 2022-01-31T22:43:48.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:48 smithi171 conmon[46715]: debug 2022-01-31T22:43:48.498+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:48.499699+0000) 2022-01-31T22:43:49.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:48 smithi167 conmon[49112]: debug 2022-01-31T22:43:48.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:48.832921+0000) 2022-01-31T22:43:49.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:48 smithi167 conmon[54076]: debug 2022-01-31T22:43:48.998+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:48.998967+0000) 2022-01-31T22:43:49.044 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:48 smithi167 conmon[60316]: debug 2022-01-31T22:43:48.758+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:48.758981+0000) 2022-01-31T22:43:49.490 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:49 smithi171 conmon[51620]: debug 2022-01-31T22:43:49.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:49.367316+0000) 2022-01-31T22:43:49.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:49 smithi171 conmon[41853]: debug 2022-01-31T22:43:49.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:49.449799+0000) 2022-01-31T22:43:49.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:49 smithi171 conmon[46715]: debug 2022-01-31T22:43:49.499+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:49.500002+0000) 2022-01-31T22:43:50.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:49 smithi167 conmon[49112]: debug 2022-01-31T22:43:49.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:49.833094+0000) 2022-01-31T22:43:50.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:49 smithi167 conmon[54076]: debug 2022-01-31T22:43:49.998+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:49.999147+0000) 2022-01-31T22:43:50.045 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:49 smithi167 conmon[60316]: debug 2022-01-31T22:43:49.758+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:49.759116+0000) 2022-01-31T22:43:50.490 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[35325]: debug 2022-01-31T22:43:50.329+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232882 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:50.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[41853]: debug 2022-01-31T22:43:50.306+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.307325+0000) 2022-01-31T22:43:50.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[41853]: debug 2022-01-31T22:43:50.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.449950+0000) 2022-01-31T22:43:50.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[46715]: debug 2022-01-31T22:43:50.305+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.306473+0000) 2022-01-31T22:43:50.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[51620]: debug 2022-01-31T22:43:50.306+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.307160+0000) 2022-01-31T22:43:50.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[51620]: debug 2022-01-31T22:43:50.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.367493+0000) 2022-01-31T22:43:50.587 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:50 smithi167 conmon[49112]: debug 2022-01-31T22:43:50.306+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.307731+0000) 2022-01-31T22:43:50.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:50 smithi167 conmon[54076]: debug 2022-01-31T22:43:50.307+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.308606+0000) 2022-01-31T22:43:50.588 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:50 smithi167 conmon[60316]: debug 2022-01-31T22:43:50.306+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.307649+0000) 2022-01-31T22:43:50.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:50 smithi171 conmon[46715]: debug 2022-01-31T22:43:50.499+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.500157+0000) 2022-01-31T22:43:50.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:50 smithi167 conmon[49112]: debug 2022-01-31T22:43:50.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.833246+0000) 2022-01-31T22:43:50.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:50 smithi167 conmon[60316]: debug 2022-01-31T22:43:50.758+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.759268+0000) 2022-01-31T22:43:51.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:50 smithi167 conmon[54076]: debug 2022-01-31T22:43:50.998+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:50.999333+0000) 2022-01-31T22:43:51.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:51 smithi171 conmon[41853]: debug 2022-01-31T22:43:51.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:51.450095+0000) 2022-01-31T22:43:51.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:51 smithi171 conmon[51620]: debug 2022-01-31T22:43:51.366+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:51.367585+0000) 2022-01-31T22:43:51.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:51 smithi171 conmon[46715]: debug 2022-01-31T22:43:51.499+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:51.500251+0000) 2022-01-31T22:43:52.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:51 smithi167 conmon[49112]: debug 2022-01-31T22:43:51.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:51.833400+0000) 2022-01-31T22:43:52.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:51 smithi167 conmon[54076]: debug 2022-01-31T22:43:51.998+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:51.999488+0000) 2022-01-31T22:43:52.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:51 smithi167 conmon[60316]: debug 2022-01-31T22:43:51.758+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:51.759455+0000) 2022-01-31T22:43:52.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:52 smithi171 conmon[41853]: debug 2022-01-31T22:43:52.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:52.450254+0000) 2022-01-31T22:43:52.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:52 smithi171 conmon[51620]: debug 2022-01-31T22:43:52.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:52.367744+0000) 2022-01-31T22:43:52.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:52 smithi171 conmon[46715]: debug 2022-01-31T22:43:52.499+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:52.500412+0000) 2022-01-31T22:43:53.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:52 smithi167 conmon[49112]: debug 2022-01-31T22:43:52.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:52.833559+0000) 2022-01-31T22:43:53.044 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:52 smithi167 conmon[54076]: debug 2022-01-31T22:43:52.998+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:52.999652+0000) 2022-01-31T22:43:53.045 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:52 smithi167 conmon[60316]: debug 2022-01-31T22:43:52.758+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:52.759665+0000) 2022-01-31T22:43:53.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:53 smithi171 conmon[41853]: debug 2022-01-31T22:43:53.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:53.450399+0000) 2022-01-31T22:43:53.491 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:53 smithi171 conmon[51620]: debug 2022-01-31T22:43:53.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:53.367969+0000) 2022-01-31T22:43:53.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:53 smithi171 conmon[46715]: debug 2022-01-31T22:43:53.499+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:53.500562+0000) 2022-01-31T22:43:54.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:53 smithi167 conmon[49112]: debug 2022-01-31T22:43:53.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:53.833751+0000) 2022-01-31T22:43:54.045 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:53 smithi167 conmon[54076]: debug 2022-01-31T22:43:53.998+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:53.999832+0000) 2022-01-31T22:43:54.045 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:53 smithi167 conmon[60316]: debug 2022-01-31T22:43:53.758+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:53.759842+0000) 2022-01-31T22:43:54.227 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:53 smithi171 conmon[35325]: debug 2022-01-31T22:43:53.900+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:43:54.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:54 smithi171 conmon[41853]: debug 2022-01-31T22:43:54.449+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:54.450575+0000) 2022-01-31T22:43:54.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:54 smithi171 conmon[51620]: debug 2022-01-31T22:43:54.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:54.368170+0000) 2022-01-31T22:43:54.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:54 smithi171 conmon[46715]: debug 2022-01-31T22:43:54.500+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:54.500780+0000) 2022-01-31T22:43:55.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:54 smithi167 conmon[49112]: debug 2022-01-31T22:43:54.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:54.833906+0000) 2022-01-31T22:43:55.045 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:55 smithi167 conmon[54076]: debug 2022-01-31T22:43:54.999+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:54.999978+0000) 2022-01-31T22:43:55.045 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:54 smithi167 conmon[60316]: debug 2022-01-31T22:43:54.759+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:54.760076+0000) 2022-01-31T22:43:55.491 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[35325]: debug 2022-01-31T22:43:55.369+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 232994 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:43:55.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[46715]: debug 2022-01-31T22:43:55.332+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.333303+0000) 2022-01-31T22:43:55.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[41853]: debug 2022-01-31T22:43:55.336+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.337194+0000) 2022-01-31T22:43:55.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[41853]: debug 2022-01-31T22:43:55.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.450734+0000) 2022-01-31T22:43:55.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[51620]: debug 2022-01-31T22:43:55.336+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.336857+0000) 2022-01-31T22:43:55.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[51620]: debug 2022-01-31T22:43:55.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.368322+0000) 2022-01-31T22:43:55.588 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:55 smithi167 conmon[49112]: debug 2022-01-31T22:43:55.335+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.336581+0000) 2022-01-31T22:43:55.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:55 smithi167 conmon[60316]: debug 2022-01-31T22:43:55.334+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.336126+0000) 2022-01-31T22:43:55.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:55 smithi167 conmon[54076]: debug 2022-01-31T22:43:55.336+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.337627+0000) 2022-01-31T22:43:55.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:55 smithi171 conmon[46715]: debug 2022-01-31T22:43:55.500+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.500937+0000) 2022-01-31T22:43:55.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:55 smithi167 conmon[60316]: debug 2022-01-31T22:43:55.759+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.760240+0000) 2022-01-31T22:43:55.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:55 smithi167 conmon[49112]: debug 2022-01-31T22:43:55.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:55.834015+0000) 2022-01-31T22:43:56.307 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:56 smithi167 conmon[54076]: debug 2022-01-31T22:43:55.999+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:56.000191+0000) 2022-01-31T22:43:56.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:56 smithi171 conmon[41853]: debug 2022-01-31T22:43:56.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:56.450839+0000) 2022-01-31T22:43:56.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:56 smithi171 conmon[51620]: debug 2022-01-31T22:43:56.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:56.368489+0000) 2022-01-31T22:43:56.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:56 smithi171 conmon[46715]: debug 2022-01-31T22:43:56.500+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:56.501037+0000) 2022-01-31T22:43:57.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:56 smithi167 conmon[49112]: debug 2022-01-31T22:43:56.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:56.834184+0000) 2022-01-31T22:43:57.045 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:57 smithi167 conmon[54076]: debug 2022-01-31T22:43:56.999+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:57.000318+0000) 2022-01-31T22:43:57.046 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:56 smithi167 conmon[60316]: debug 2022-01-31T22:43:56.759+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:56.760391+0000) 2022-01-31T22:43:57.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:57 smithi171 conmon[41853]: debug 2022-01-31T22:43:57.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:57.450986+0000) 2022-01-31T22:43:57.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:57 smithi171 conmon[51620]: debug 2022-01-31T22:43:57.367+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:57.368647+0000) 2022-01-31T22:43:57.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:57 smithi171 conmon[46715]: debug 2022-01-31T22:43:57.500+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:57.501188+0000) 2022-01-31T22:43:58.044 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:57 smithi167 conmon[49112]: debug 2022-01-31T22:43:57.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:57.834358+0000) 2022-01-31T22:43:58.045 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:58 smithi167 conmon[54076]: debug 2022-01-31T22:43:57.999+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:58.000465+0000) 2022-01-31T22:43:58.046 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:57 smithi167 conmon[60316]: debug 2022-01-31T22:43:57.759+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:57.760566+0000) 2022-01-31T22:43:58.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:58 smithi171 conmon[41853]: debug 2022-01-31T22:43:58.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:58.451130+0000) 2022-01-31T22:43:58.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:58 smithi171 conmon[51620]: debug 2022-01-31T22:43:58.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:58.368795+0000) 2022-01-31T22:43:58.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:58 smithi171 conmon[46715]: debug 2022-01-31T22:43:58.500+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:58.501404+0000) 2022-01-31T22:43:59.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:58 smithi167 conmon[49112]: debug 2022-01-31T22:43:58.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:58.834536+0000) 2022-01-31T22:43:59.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:43:59 smithi167 conmon[54076]: debug 2022-01-31T22:43:58.999+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:59.000646+0000) 2022-01-31T22:43:59.047 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:58 smithi167 conmon[60316]: debug 2022-01-31T22:43:58.759+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:58.760715+0000) 2022-01-31T22:43:59.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:43:59 smithi171 conmon[41853]: debug 2022-01-31T22:43:59.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:59.451305+0000) 2022-01-31T22:43:59.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:43:59 smithi171 conmon[51620]: debug 2022-01-31T22:43:59.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:59.368927+0000) 2022-01-31T22:43:59.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:43:59 smithi171 conmon[46715]: debug 2022-01-31T22:43:59.500+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:59.501603+0000) 2022-01-31T22:44:00.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:43:59 smithi167 conmon[49112]: debug 2022-01-31T22:43:59.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:59.834711+0000) 2022-01-31T22:44:00.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:00 smithi167 conmon[54076]: debug 2022-01-31T22:44:00.000+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.000829+0000) 2022-01-31T22:44:00.048 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:43:59 smithi167 conmon[60316]: debug 2022-01-31T22:43:59.759+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:43:59.760862+0000) 2022-01-31T22:44:00.492 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[35325]: debug 2022-01-31T22:44:00.396+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233106 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:00.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[41853]: debug 2022-01-31T22:44:00.372+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.373330+0000) 2022-01-31T22:44:00.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[41853]: debug 2022-01-31T22:44:00.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.451482+0000) 2022-01-31T22:44:00.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[46715]: debug 2022-01-31T22:44:00.371+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.372526+0000) 2022-01-31T22:44:00.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[51620]: debug 2022-01-31T22:44:00.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.369102+0000) 2022-01-31T22:44:00.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[51620]: debug 2022-01-31T22:44:00.373+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.374043+0000) 2022-01-31T22:44:00.659 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:00 smithi167 conmon[49112]: debug 2022-01-31T22:44:00.371+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.372610+0000) 2022-01-31T22:44:00.660 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:00 smithi167 conmon[54076]: debug 2022-01-31T22:44:00.372+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.373866+0000) 2022-01-31T22:44:00.661 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:00 smithi167 conmon[60316]: debug 2022-01-31T22:44:00.371+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.373408+0000) 2022-01-31T22:44:00.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:00 smithi171 conmon[46715]: debug 2022-01-31T22:44:00.501+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.501760+0000) 2022-01-31T22:44:00.991 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:00 smithi167 conmon[49112]: debug 2022-01-31T22:44:00.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.834810+0000) 2022-01-31T22:44:00.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:00 smithi167 conmon[60316]: debug 2022-01-31T22:44:00.760+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:00.761048+0000) 2022-01-31T22:44:01.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:01 smithi167 conmon[54076]: debug 2022-01-31T22:44:01.000+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:01.000998+0000) 2022-01-31T22:44:01.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:01 smithi171 conmon[51620]: debug 2022-01-31T22:44:01.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:01.369288+0000) 2022-01-31T22:44:01.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:01 smithi171 conmon[41853]: debug 2022-01-31T22:44:01.450+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:01.451601+0000) 2022-01-31T22:44:01.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:01 smithi171 conmon[46715]: debug 2022-01-31T22:44:01.501+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:01.501869+0000) 2022-01-31T22:44:02.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:01 smithi167 conmon[49112]: debug 2022-01-31T22:44:01.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:01.834895+0000) 2022-01-31T22:44:02.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:02 smithi167 conmon[54076]: debug 2022-01-31T22:44:01.999+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:02.001167+0000) 2022-01-31T22:44:02.046 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:01 smithi167 conmon[60316]: debug 2022-01-31T22:44:01.760+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:01.761186+0000) 2022-01-31T22:44:02.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:02 smithi171 conmon[41853]: debug 2022-01-31T22:44:02.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:02.451776+0000) 2022-01-31T22:44:02.492 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:02 smithi171 conmon[51620]: debug 2022-01-31T22:44:02.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:02.369434+0000) 2022-01-31T22:44:02.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:02 smithi171 conmon[46715]: debug 2022-01-31T22:44:02.501+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:02.502024+0000) 2022-01-31T22:44:03.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:02 smithi167 conmon[49112]: debug 2022-01-31T22:44:02.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:02.835067+0000) 2022-01-31T22:44:03.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:03 smithi167 conmon[54076]: debug 2022-01-31T22:44:03.000+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:03.001377+0000) 2022-01-31T22:44:03.047 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:02 smithi167 conmon[60316]: debug 2022-01-31T22:44:02.760+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:02.761380+0000) 2022-01-31T22:44:03.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:03 smithi171 conmon[41853]: debug 2022-01-31T22:44:03.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:03.451904+0000) 2022-01-31T22:44:03.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:03 smithi171 conmon[51620]: debug 2022-01-31T22:44:03.368+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:03.369605+0000) 2022-01-31T22:44:03.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:03 smithi171 conmon[46715]: debug 2022-01-31T22:44:03.501+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:03.502131+0000) 2022-01-31T22:44:04.046 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:03 smithi167 conmon[49112]: debug 2022-01-31T22:44:03.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:03.835255+0000) 2022-01-31T22:44:04.046 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:03 smithi167 conmon[60316]: debug 2022-01-31T22:44:03.760+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:03.761588+0000) 2022-01-31T22:44:04.047 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:04 smithi167 conmon[54076]: debug 2022-01-31T22:44:04.000+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:04.001575+0000) 2022-01-31T22:44:04.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:04 smithi171 conmon[41853]: debug 2022-01-31T22:44:04.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:04.452081+0000) 2022-01-31T22:44:04.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:04 smithi171 conmon[51620]: debug 2022-01-31T22:44:04.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:04.369831+0000) 2022-01-31T22:44:04.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:04 smithi171 conmon[46715]: debug 2022-01-31T22:44:04.501+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:04.502332+0000) 2022-01-31T22:44:05.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:05 smithi167 conmon[54076]: debug 2022-01-31T22:44:05.001+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.001774+0000) 2022-01-31T22:44:05.046 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:04 smithi167 conmon[60316]: debug 2022-01-31T22:44:04.761+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:04.761756+0000) 2022-01-31T22:44:05.047 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:04 smithi167 conmon[49112]: debug 2022-01-31T22:44:04.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:04.835463+0000) 2022-01-31T22:44:05.492 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[35325]: debug 2022-01-31T22:44:05.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233217 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:05.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[46715]: debug 2022-01-31T22:44:05.397+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.398682+0000) 2022-01-31T22:44:05.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[41853]: debug 2022-01-31T22:44:05.398+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.399601+0000) 2022-01-31T22:44:05.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[41853]: debug 2022-01-31T22:44:05.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.452257+0000) 2022-01-31T22:44:05.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[51620]: debug 2022-01-31T22:44:05.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.369970+0000) 2022-01-31T22:44:05.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[51620]: debug 2022-01-31T22:44:05.399+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.400381+0000) 2022-01-31T22:44:05.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:05 smithi167 conmon[54076]: debug 2022-01-31T22:44:05.398+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.400057+0000) 2022-01-31T22:44:05.660 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:05 smithi167 conmon[49112]: debug 2022-01-31T22:44:05.398+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.399631+0000) 2022-01-31T22:44:05.660 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:05 smithi167 conmon[60316]: debug 2022-01-31T22:44:05.397+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.398935+0000) 2022-01-31T22:44:05.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:05 smithi171 conmon[46715]: debug 2022-01-31T22:44:05.501+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.502515+0000) 2022-01-31T22:44:05.992 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:05 smithi167 conmon[49112]: debug 2022-01-31T22:44:05.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.835690+0000) 2022-01-31T22:44:05.992 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:05 smithi167 conmon[60316]: debug 2022-01-31T22:44:05.760+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:05.761964+0000) 2022-01-31T22:44:06.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:06 smithi167 conmon[54076]: debug 2022-01-31T22:44:06.001+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:06.001982+0000) 2022-01-31T22:44:06.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:06 smithi171 conmon[41853]: debug 2022-01-31T22:44:06.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:06.452420+0000) 2022-01-31T22:44:06.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:06 smithi171 conmon[51620]: debug 2022-01-31T22:44:06.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:06.370158+0000) 2022-01-31T22:44:06.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:06 smithi171 conmon[46715]: debug 2022-01-31T22:44:06.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:06.502663+0000) 2022-01-31T22:44:07.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:07 smithi167 conmon[54076]: debug 2022-01-31T22:44:07.000+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:07.002172+0000) 2022-01-31T22:44:07.047 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:06 smithi167 conmon[49112]: debug 2022-01-31T22:44:06.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:06.835829+0000) 2022-01-31T22:44:07.047 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:06 smithi167 conmon[60316]: debug 2022-01-31T22:44:06.761+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:06.762152+0000) 2022-01-31T22:44:07.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:07 smithi171 conmon[41853]: debug 2022-01-31T22:44:07.451+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:07.452606+0000) 2022-01-31T22:44:07.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:07 smithi171 conmon[51620]: debug 2022-01-31T22:44:07.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:07.370305+0000) 2022-01-31T22:44:07.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:07 smithi171 conmon[46715]: debug 2022-01-31T22:44:07.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:07.502821+0000) 2022-01-31T22:44:08.046 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:07 smithi167 conmon[49112]: debug 2022-01-31T22:44:07.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:07.836022+0000) 2022-01-31T22:44:08.047 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:08 smithi167 conmon[54076]: debug 2022-01-31T22:44:08.001+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:08.002403+0000) 2022-01-31T22:44:08.048 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:07 smithi167 conmon[60316]: debug 2022-01-31T22:44:07.761+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:07.762359+0000) 2022-01-31T22:44:08.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:08 smithi171 conmon[41853]: debug 2022-01-31T22:44:08.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:08.452778+0000) 2022-01-31T22:44:08.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:08 smithi171 conmon[51620]: debug 2022-01-31T22:44:08.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:08.370471+0000) 2022-01-31T22:44:08.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:08 smithi171 conmon[46715]: debug 2022-01-31T22:44:08.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:08.502978+0000) 2022-01-31T22:44:09.046 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:08 smithi167 conmon[49112]: debug 2022-01-31T22:44:08.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:08.836125+0000) 2022-01-31T22:44:09.047 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:08 smithi167 conmon[60316]: debug 2022-01-31T22:44:08.761+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:08.762532+0000) 2022-01-31T22:44:09.048 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:09 smithi167 conmon[54076]: debug 2022-01-31T22:44:09.001+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:09.002577+0000) 2022-01-31T22:44:09.230 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:08 smithi171 conmon[35325]: debug 2022-01-31T22:44:08.901+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:44:09.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:09 smithi171 conmon[41853]: debug 2022-01-31T22:44:09.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:09.452887+0000) 2022-01-31T22:44:09.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:09 smithi171 conmon[51620]: debug 2022-01-31T22:44:09.369+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:09.370661+0000) 2022-01-31T22:44:09.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:09 smithi171 conmon[46715]: debug 2022-01-31T22:44:09.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:09.503119+0000) 2022-01-31T22:44:10.047 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:09 smithi167 conmon[49112]: debug 2022-01-31T22:44:09.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:09.836279+0000) 2022-01-31T22:44:10.047 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:10 smithi167 conmon[54076]: debug 2022-01-31T22:44:10.002+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.002768+0000) 2022-01-31T22:44:10.048 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:09 smithi167 conmon[60316]: debug 2022-01-31T22:44:09.762+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:09.762717+0000) 2022-01-31T22:44:10.493 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[35325]: debug 2022-01-31T22:44:10.464+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233331 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:10.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[46715]: debug 2022-01-31T22:44:10.424+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.425307+0000) 2022-01-31T22:44:10.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[41853]: debug 2022-01-31T22:44:10.424+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.425648+0000) 2022-01-31T22:44:10.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[41853]: debug 2022-01-31T22:44:10.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.452999+0000) 2022-01-31T22:44:10.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[51620]: debug 2022-01-31T22:44:10.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.370801+0000) 2022-01-31T22:44:10.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[51620]: debug 2022-01-31T22:44:10.425+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.425844+0000) 2022-01-31T22:44:10.724 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:10 smithi167 conmon[49112]: debug 2022-01-31T22:44:10.424+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.424817+0000) 2022-01-31T22:44:10.724 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:10 smithi167 conmon[54076]: debug 2022-01-31T22:44:10.425+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.425617+0000) 2022-01-31T22:44:10.725 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:10 smithi167 conmon[60316]: debug 2022-01-31T22:44:10.424+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.424955+0000) 2022-01-31T22:44:10.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:10 smithi171 conmon[46715]: debug 2022-01-31T22:44:10.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.503246+0000) 2022-01-31T22:44:10.993 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:10 smithi167 conmon[60316]: debug 2022-01-31T22:44:10.762+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.762935+0000) 2022-01-31T22:44:10.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:10 smithi167 conmon[49112]: debug 2022-01-31T22:44:10.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:10.836454+0000) 2022-01-31T22:44:11.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:11 smithi167 conmon[54076]: debug 2022-01-31T22:44:11.002+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:11.002942+0000) 2022-01-31T22:44:11.493 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:11 smithi171 conmon[51620]: debug 2022-01-31T22:44:11.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:11.370981+0000) 2022-01-31T22:44:11.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:11 smithi171 conmon[41853]: debug 2022-01-31T22:44:11.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:11.453151+0000) 2022-01-31T22:44:11.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:11 smithi171 conmon[46715]: debug 2022-01-31T22:44:11.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:11.503338+0000) 2022-01-31T22:44:12.047 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:12 smithi167 conmon[54076]: debug 2022-01-31T22:44:12.002+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:12.003094+0000) 2022-01-31T22:44:12.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:11 smithi167 conmon[49112]: debug 2022-01-31T22:44:11.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:11.836610+0000) 2022-01-31T22:44:12.048 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:11 smithi167 conmon[60316]: debug 2022-01-31T22:44:11.762+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:11.763111+0000) 2022-01-31T22:44:12.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:12 smithi171 conmon[41853]: debug 2022-01-31T22:44:12.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:12.453295+0000) 2022-01-31T22:44:12.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:12 smithi171 conmon[51620]: debug 2022-01-31T22:44:12.370+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:12.371159+0000) 2022-01-31T22:44:12.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:12 smithi171 conmon[46715]: debug 2022-01-31T22:44:12.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:12.503456+0000) 2022-01-31T22:44:13.047 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:13 smithi167 conmon[54076]: debug 2022-01-31T22:44:13.002+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:13.003271+0000) 2022-01-31T22:44:13.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:12 smithi167 conmon[49112]: debug 2022-01-31T22:44:12.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:12.836772+0000) 2022-01-31T22:44:13.048 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:12 smithi167 conmon[60316]: debug 2022-01-31T22:44:12.762+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:12.763285+0000) 2022-01-31T22:44:13.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:13 smithi171 conmon[41853]: debug 2022-01-31T22:44:13.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:13.453404+0000) 2022-01-31T22:44:13.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:13 smithi171 conmon[51620]: debug 2022-01-31T22:44:13.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:13.371375+0000) 2022-01-31T22:44:13.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:13 smithi171 conmon[46715]: debug 2022-01-31T22:44:13.502+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:13.503628+0000) 2022-01-31T22:44:14.047 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:13 smithi167 conmon[49112]: debug 2022-01-31T22:44:13.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:13.836959+0000) 2022-01-31T22:44:14.048 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:14 smithi167 conmon[54076]: debug 2022-01-31T22:44:14.002+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:14.003434+0000) 2022-01-31T22:44:14.049 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:13 smithi167 conmon[60316]: debug 2022-01-31T22:44:13.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:13.763440+0000) 2022-01-31T22:44:14.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:14 smithi171 conmon[41853]: debug 2022-01-31T22:44:14.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:14.453563+0000) 2022-01-31T22:44:14.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:14 smithi171 conmon[51620]: debug 2022-01-31T22:44:14.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:14.371572+0000) 2022-01-31T22:44:14.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:14 smithi171 conmon[46715]: debug 2022-01-31T22:44:14.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:14.503835+0000) 2022-01-31T22:44:15.047 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:14 smithi167 conmon[49112]: debug 2022-01-31T22:44:14.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:14.837167+0000) 2022-01-31T22:44:15.048 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:15 smithi167 conmon[54076]: debug 2022-01-31T22:44:15.003+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.003639+0000) 2022-01-31T22:44:15.049 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:14 smithi167 conmon[60316]: debug 2022-01-31T22:44:14.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:14.763628+0000) 2022-01-31T22:44:15.494 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[35325]: debug 2022-01-31T22:44:15.490+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233441 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:15.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[41853]: debug 2022-01-31T22:44:15.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.453720+0000) 2022-01-31T22:44:15.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[41853]: debug 2022-01-31T22:44:15.467+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.467826+0000) 2022-01-31T22:44:15.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[46715]: debug 2022-01-31T22:44:15.465+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.466569+0000) 2022-01-31T22:44:15.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[51620]: debug 2022-01-31T22:44:15.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.371807+0000) 2022-01-31T22:44:15.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[51620]: debug 2022-01-31T22:44:15.466+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.466919+0000) 2022-01-31T22:44:15.725 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:15 smithi167 conmon[49112]: debug 2022-01-31T22:44:15.466+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.467233+0000) 2022-01-31T22:44:15.725 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:15 smithi167 conmon[54076]: debug 2022-01-31T22:44:15.467+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.467528+0000) 2022-01-31T22:44:15.726 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:15 smithi167 conmon[60316]: debug 2022-01-31T22:44:15.467+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.467639+0000) 2022-01-31T22:44:15.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:15 smithi171 conmon[46715]: debug 2022-01-31T22:44:15.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.503983+0000) 2022-01-31T22:44:15.993 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:15 smithi167 conmon[49112]: debug 2022-01-31T22:44:15.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.837353+0000) 2022-01-31T22:44:15.994 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:15 smithi167 conmon[60316]: debug 2022-01-31T22:44:15.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:15.763779+0000) 2022-01-31T22:44:16.256 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:16 smithi167 conmon[54076]: debug 2022-01-31T22:44:16.003+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:16.003847+0000) 2022-01-31T22:44:16.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:16 smithi171 conmon[41853]: debug 2022-01-31T22:44:16.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:16.453924+0000) 2022-01-31T22:44:16.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:16 smithi171 conmon[51620]: debug 2022-01-31T22:44:16.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:16.371992+0000) 2022-01-31T22:44:16.827 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:16 smithi167 conmon[60316]: debug 2022-01-31T22:44:16.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:16.763905+0000) 2022-01-31T22:44:16.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:16 smithi171 conmon[46715]: debug 2022-01-31T22:44:16.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:16.504063+0000) 2022-01-31T22:44:17.159 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:17 smithi167 conmon[54076]: debug 2022-01-31T22:44:17.003+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:17.003995+0000) 2022-01-31T22:44:17.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:16 smithi167 conmon[49112]: debug 2022-01-31T22:44:16.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:16.837469+0000) 2022-01-31T22:44:17.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:17 smithi171 conmon[41853]: debug 2022-01-31T22:44:17.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:17.454027+0000) 2022-01-31T22:44:17.494 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:17 smithi171 conmon[51620]: debug 2022-01-31T22:44:17.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:17.372152+0000) 2022-01-31T22:44:17.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:17 smithi171 conmon[46715]: debug 2022-01-31T22:44:17.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:17.504200+0000) 2022-01-31T22:44:18.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:17 smithi167 conmon[49112]: debug 2022-01-31T22:44:17.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:17.837627+0000) 2022-01-31T22:44:18.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:18 smithi167 conmon[54076]: debug 2022-01-31T22:44:18.003+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:18.004181+0000) 2022-01-31T22:44:18.049 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:17 smithi167 conmon[60316]: debug 2022-01-31T22:44:17.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:17.764057+0000) 2022-01-31T22:44:18.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:18 smithi171 conmon[41853]: debug 2022-01-31T22:44:18.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:18.454233+0000) 2022-01-31T22:44:18.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:18 smithi171 conmon[51620]: debug 2022-01-31T22:44:18.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:18.372305+0000) 2022-01-31T22:44:18.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:18 smithi171 conmon[46715]: debug 2022-01-31T22:44:18.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:18.504358+0000) 2022-01-31T22:44:19.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:18 smithi167 conmon[49112]: debug 2022-01-31T22:44:18.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:18.837773+0000) 2022-01-31T22:44:19.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:19 smithi167 conmon[54076]: debug 2022-01-31T22:44:19.004+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:19.004362+0000) 2022-01-31T22:44:19.050 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:18 smithi167 conmon[60316]: debug 2022-01-31T22:44:18.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:18.764259+0000) 2022-01-31T22:44:19.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:19 smithi171 conmon[41853]: debug 2022-01-31T22:44:19.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:19.454419+0000) 2022-01-31T22:44:19.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:19 smithi171 conmon[51620]: debug 2022-01-31T22:44:19.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:19.372505+0000) 2022-01-31T22:44:19.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:19 smithi171 conmon[46715]: debug 2022-01-31T22:44:19.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:19.504565+0000) 2022-01-31T22:44:20.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:19 smithi167 conmon[49112]: debug 2022-01-31T22:44:19.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:19.837942+0000) 2022-01-31T22:44:20.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:20 smithi167 conmon[54076]: debug 2022-01-31T22:44:20.004+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.004572+0000) 2022-01-31T22:44:20.050 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:19 smithi167 conmon[60316]: debug 2022-01-31T22:44:19.763+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:19.764401+0000) 2022-01-31T22:44:20.483 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[41853]: debug 2022-01-31T22:44:20.452+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.454620+0000) 2022-01-31T22:44:20.484 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[51620]: debug 2022-01-31T22:44:20.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.372735+0000) 2022-01-31T22:44:20.754 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:20 smithi167 conmon[49112]: debug 2022-01-31T22:44:20.494+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.494534+0000) 2022-01-31T22:44:20.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:20 smithi167 conmon[54076]: debug 2022-01-31T22:44:20.493+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.494434+0000) 2022-01-31T22:44:20.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:20 smithi167 conmon[60316]: debug 2022-01-31T22:44:20.494+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.494898+0000) 2022-01-31T22:44:20.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[35325]: debug 2022-01-31T22:44:20.516+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233555 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:20.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[41853]: debug 2022-01-31T22:44:20.491+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.493307+0000) 2022-01-31T22:44:20.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[51620]: debug 2022-01-31T22:44:20.491+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.493060+0000) 2022-01-31T22:44:20.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[46715]: debug 2022-01-31T22:44:20.491+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.492973+0000) 2022-01-31T22:44:20.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:20 smithi171 conmon[46715]: debug 2022-01-31T22:44:20.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.504740+0000) 2022-01-31T22:44:21.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:20 smithi167 conmon[49112]: debug 2022-01-31T22:44:20.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.838100+0000) 2022-01-31T22:44:21.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:21 smithi167 conmon[54076]: debug 2022-01-31T22:44:21.004+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:21.004747+0000) 2022-01-31T22:44:21.050 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:20 smithi167 conmon[60316]: debug 2022-01-31T22:44:20.764+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:20.764587+0000) 2022-01-31T22:44:21.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:21 smithi171 conmon[41853]: debug 2022-01-31T22:44:21.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:21.454809+0000) 2022-01-31T22:44:21.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:21 smithi171 conmon[51620]: debug 2022-01-31T22:44:21.371+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:21.372983+0000) 2022-01-31T22:44:21.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:21 smithi171 conmon[46715]: debug 2022-01-31T22:44:21.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:21.504898+0000) 2022-01-31T22:44:22.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:21 smithi167 conmon[49112]: debug 2022-01-31T22:44:21.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:21.838260+0000) 2022-01-31T22:44:22.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:22 smithi167 conmon[54076]: debug 2022-01-31T22:44:22.004+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:22.004909+0000) 2022-01-31T22:44:22.050 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:21 smithi167 conmon[60316]: debug 2022-01-31T22:44:21.764+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:21.764733+0000) 2022-01-31T22:44:22.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:22 smithi171 conmon[41853]: debug 2022-01-31T22:44:22.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:22.454960+0000) 2022-01-31T22:44:22.697 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:22 smithi171 conmon[51620]: debug 2022-01-31T22:44:22.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:22.373144+0000) 2022-01-31T22:44:22.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:22 smithi171 conmon[46715]: debug 2022-01-31T22:44:22.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:22.505040+0000) 2022-01-31T22:44:23.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:22 smithi167 conmon[49112]: debug 2022-01-31T22:44:22.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:22.838437+0000) 2022-01-31T22:44:23.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:23 smithi167 conmon[54076]: debug 2022-01-31T22:44:23.004+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:23.005094+0000) 2022-01-31T22:44:23.050 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:22 smithi167 conmon[60316]: debug 2022-01-31T22:44:22.764+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:22.764948+0000) 2022-01-31T22:44:23.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:23 smithi171 conmon[41853]: debug 2022-01-31T22:44:23.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:23.455095+0000) 2022-01-31T22:44:23.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:23 smithi171 conmon[51620]: debug 2022-01-31T22:44:23.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:23.373350+0000) 2022-01-31T22:44:23.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:23 smithi171 conmon[46715]: debug 2022-01-31T22:44:23.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:23.505220+0000) 2022-01-31T22:44:24.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:23 smithi167 conmon[49112]: debug 2022-01-31T22:44:23.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:23.838615+0000) 2022-01-31T22:44:24.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:24 smithi167 conmon[54076]: debug 2022-01-31T22:44:24.004+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:24.005308+0000) 2022-01-31T22:44:24.051 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:23 smithi167 conmon[60316]: debug 2022-01-31T22:44:23.764+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:23.765172+0000) 2022-01-31T22:44:24.233 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:23 smithi171 conmon[35325]: debug 2022-01-31T22:44:23.901+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:44:24.495 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:24 smithi171 conmon[51620]: debug 2022-01-31T22:44:24.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:24.373511+0000) 2022-01-31T22:44:24.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:24 smithi171 conmon[41853]: debug 2022-01-31T22:44:24.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:24.455291+0000) 2022-01-31T22:44:24.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:24 smithi171 conmon[46715]: debug 2022-01-31T22:44:24.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:24.505418+0000) 2022-01-31T22:44:25.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:24 smithi167 conmon[49112]: debug 2022-01-31T22:44:24.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:24.838780+0000) 2022-01-31T22:44:25.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:25 smithi167 conmon[54076]: debug 2022-01-31T22:44:25.005+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.005538+0000) 2022-01-31T22:44:25.051 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:24 smithi167 conmon[60316]: debug 2022-01-31T22:44:24.764+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:24.765389+0000) 2022-01-31T22:44:25.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[41853]: debug 2022-01-31T22:44:25.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.455502+0000) 2022-01-31T22:44:25.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[51620]: debug 2022-01-31T22:44:25.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.373699+0000) 2022-01-31T22:44:25.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:25 smithi167 conmon[49112]: debug 2022-01-31T22:44:25.520+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.520624+0000) 2022-01-31T22:44:25.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:25 smithi167 conmon[54076]: debug 2022-01-31T22:44:25.519+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.520177+0000) 2022-01-31T22:44:25.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:25 smithi167 conmon[60316]: debug 2022-01-31T22:44:25.521+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.521587+0000) 2022-01-31T22:44:25.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[41853]: debug 2022-01-31T22:44:25.519+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.520942+0000) 2022-01-31T22:44:25.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[51620]: debug 2022-01-31T22:44:25.517+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.519562+0000) 2022-01-31T22:44:25.849 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[35325]: debug 2022-01-31T22:44:25.543+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233666 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:25.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[46715]: debug 2022-01-31T22:44:25.503+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.505606+0000) 2022-01-31T22:44:25.850 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:25 smithi171 conmon[46715]: debug 2022-01-31T22:44:25.518+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.520446+0000) 2022-01-31T22:44:26.049 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:25 smithi167 conmon[60316]: debug 2022-01-31T22:44:25.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.765576+0000) 2022-01-31T22:44:26.050 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:25 smithi167 conmon[49112]: debug 2022-01-31T22:44:25.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:25.838967+0000) 2022-01-31T22:44:26.051 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:26 smithi167 conmon[54076]: debug 2022-01-31T22:44:26.005+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:26.005680+0000) 2022-01-31T22:44:26.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:26 smithi171 conmon[41853]: debug 2022-01-31T22:44:26.453+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:26.455689+0000) 2022-01-31T22:44:26.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:26 smithi171 conmon[51620]: debug 2022-01-31T22:44:26.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:26.373827+0000) 2022-01-31T22:44:26.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:26 smithi171 conmon[46715]: debug 2022-01-31T22:44:26.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:26.505834+0000) 2022-01-31T22:44:27.049 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:26 smithi167 conmon[49112]: debug 2022-01-31T22:44:26.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:26.839125+0000) 2022-01-31T22:44:27.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:27 smithi167 conmon[54076]: debug 2022-01-31T22:44:27.005+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:27.005832+0000) 2022-01-31T22:44:27.051 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:26 smithi167 conmon[60316]: debug 2022-01-31T22:44:26.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:26.765752+0000) 2022-01-31T22:44:27.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:27 smithi171 conmon[41853]: debug 2022-01-31T22:44:27.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:27.455864+0000) 2022-01-31T22:44:27.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:27 smithi171 conmon[51620]: debug 2022-01-31T22:44:27.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:27.373993+0000) 2022-01-31T22:44:27.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:27 smithi171 conmon[46715]: debug 2022-01-31T22:44:27.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:27.505964+0000) 2022-01-31T22:44:28.050 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:27 smithi167 conmon[49112]: debug 2022-01-31T22:44:27.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:27.839310+0000) 2022-01-31T22:44:28.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:28 smithi167 conmon[54076]: debug 2022-01-31T22:44:28.005+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:28.006013+0000) 2022-01-31T22:44:28.051 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:27 smithi167 conmon[60316]: debug 2022-01-31T22:44:27.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:27.765902+0000) 2022-01-31T22:44:28.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:28 smithi171 conmon[41853]: debug 2022-01-31T22:44:28.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:28.456052+0000) 2022-01-31T22:44:28.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:28 smithi171 conmon[51620]: debug 2022-01-31T22:44:28.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:28.374210+0000) 2022-01-31T22:44:28.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:28 smithi171 conmon[46715]: debug 2022-01-31T22:44:28.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:28.506071+0000) 2022-01-31T22:44:29.050 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:29 smithi167 conmon[54076]: debug 2022-01-31T22:44:29.005+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:29.006183+0000) 2022-01-31T22:44:29.051 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:28 smithi167 conmon[49112]: debug 2022-01-31T22:44:28.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:28.839464+0000) 2022-01-31T22:44:29.051 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:28 smithi167 conmon[60316]: debug 2022-01-31T22:44:28.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:28.766099+0000) 2022-01-31T22:44:29.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:29 smithi171 conmon[41853]: debug 2022-01-31T22:44:29.454+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:29.456230+0000) 2022-01-31T22:44:29.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:29 smithi171 conmon[51620]: debug 2022-01-31T22:44:29.372+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:29.374432+0000) 2022-01-31T22:44:29.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:29 smithi171 conmon[46715]: debug 2022-01-31T22:44:29.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:29.506193+0000) 2022-01-31T22:44:29.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:29 smithi167 conmon[49112]: debug 2022-01-31T22:44:29.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:29.839683+0000) 2022-01-31T22:44:29.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:29 smithi167 conmon[60316]: debug 2022-01-31T22:44:29.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:29.766304+0000) 2022-01-31T22:44:30.305 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:30 smithi167 conmon[54076]: debug 2022-01-31T22:44:30.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.006367+0000) 2022-01-31T22:44:30.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[41853]: debug 2022-01-31T22:44:30.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.456364+0000) 2022-01-31T22:44:30.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[51620]: debug 2022-01-31T22:44:30.373+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.374642+0000) 2022-01-31T22:44:30.593 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:30 smithi167 conmon[49112]: debug 2022-01-31T22:44:30.547+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.547529+0000) 2022-01-31T22:44:30.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:30 smithi167 conmon[54076]: debug 2022-01-31T22:44:30.547+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.547691+0000) 2022-01-31T22:44:30.595 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:30 smithi167 conmon[60316]: debug 2022-01-31T22:44:30.548+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.548681+0000) 2022-01-31T22:44:30.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[35325]: debug 2022-01-31T22:44:30.569+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233779 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:30.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[41853]: debug 2022-01-31T22:44:30.546+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.548471+0000) 2022-01-31T22:44:30.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[51620]: debug 2022-01-31T22:44:30.545+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.546872+0000) 2022-01-31T22:44:30.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[46715]: debug 2022-01-31T22:44:30.504+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.506298+0000) 2022-01-31T22:44:30.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:30 smithi171 conmon[46715]: debug 2022-01-31T22:44:30.547+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.548845+0000) 2022-01-31T22:44:30.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:30 smithi167 conmon[60316]: debug 2022-01-31T22:44:30.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.766524+0000) 2022-01-31T22:44:30.910 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:30 smithi167 conmon[49112]: debug 2022-01-31T22:44:30.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:30.839887+0000) 2022-01-31T22:44:31.306 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:31 smithi167 conmon[54076]: debug 2022-01-31T22:44:31.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:31.006577+0000) 2022-01-31T22:44:31.496 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:31 smithi171 conmon[51620]: debug 2022-01-31T22:44:31.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:31.374828+0000) 2022-01-31T22:44:31.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:31 smithi171 conmon[41853]: debug 2022-01-31T22:44:31.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:31.456556+0000) 2022-01-31T22:44:31.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:31 smithi171 conmon[46715]: debug 2022-01-31T22:44:31.505+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:31.506478+0000) 2022-01-31T22:44:31.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:31 smithi167 conmon[49112]: debug 2022-01-31T22:44:31.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:31.840059+0000) 2022-01-31T22:44:31.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:31 smithi167 conmon[60316]: debug 2022-01-31T22:44:31.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:31.766675+0000) 2022-01-31T22:44:32.311 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:32 smithi167 conmon[54076]: debug 2022-01-31T22:44:32.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:32.006749+0000) 2022-01-31T22:44:32.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:32 smithi171 conmon[41853]: debug 2022-01-31T22:44:32.455+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:32.456728+0000) 2022-01-31T22:44:32.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:32 smithi171 conmon[51620]: debug 2022-01-31T22:44:32.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:32.374981+0000) 2022-01-31T22:44:32.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:32 smithi171 conmon[46715]: debug 2022-01-31T22:44:32.505+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:32.506609+0000) 2022-01-31T22:44:32.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:32 smithi167 conmon[49112]: debug 2022-01-31T22:44:32.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:32.840247+0000) 2022-01-31T22:44:32.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:32 smithi167 conmon[60316]: debug 2022-01-31T22:44:32.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:32.766815+0000) 2022-01-31T22:44:33.315 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:33 smithi167 conmon[54076]: debug 2022-01-31T22:44:33.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:33.006932+0000) 2022-01-31T22:44:33.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:33 smithi171 conmon[41853]: debug 2022-01-31T22:44:33.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:33.456890+0000) 2022-01-31T22:44:33.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:33 smithi171 conmon[51620]: debug 2022-01-31T22:44:33.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:33.375190+0000) 2022-01-31T22:44:33.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:33 smithi171 conmon[46715]: debug 2022-01-31T22:44:33.506+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:33.506808+0000) 2022-01-31T22:44:33.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:33 smithi167 conmon[49112]: debug 2022-01-31T22:44:33.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:33.840448+0000) 2022-01-31T22:44:33.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:33 smithi167 conmon[60316]: debug 2022-01-31T22:44:33.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:33.766996+0000) 2022-01-31T22:44:34.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:34 smithi167 conmon[54076]: debug 2022-01-31T22:44:34.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:34.007144+0000) 2022-01-31T22:44:34.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:34 smithi171 conmon[41853]: debug 2022-01-31T22:44:34.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:34.457039+0000) 2022-01-31T22:44:34.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:34 smithi171 conmon[51620]: debug 2022-01-31T22:44:34.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:34.375359+0000) 2022-01-31T22:44:34.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:34 smithi171 conmon[46715]: debug 2022-01-31T22:44:34.506+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:34.506980+0000) 2022-01-31T22:44:34.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:34 smithi167 conmon[49112]: debug 2022-01-31T22:44:34.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:34.840652+0000) 2022-01-31T22:44:34.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:34 smithi167 conmon[60316]: debug 2022-01-31T22:44:34.765+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:34.767166+0000) 2022-01-31T22:44:35.324 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:35 smithi167 conmon[54076]: debug 2022-01-31T22:44:35.005+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.007358+0000) 2022-01-31T22:44:35.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[41853]: debug 2022-01-31T22:44:35.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.457144+0000) 2022-01-31T22:44:35.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[51620]: debug 2022-01-31T22:44:35.374+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.375569+0000) 2022-01-31T22:44:35.595 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:35 smithi167 conmon[49112]: debug 2022-01-31T22:44:35.573+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.574655+0000) 2022-01-31T22:44:35.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:35 smithi167 conmon[54076]: debug 2022-01-31T22:44:35.573+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.574563+0000) 2022-01-31T22:44:35.596 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:35 smithi167 conmon[60316]: debug 2022-01-31T22:44:35.573+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.575016+0000) 2022-01-31T22:44:35.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[35325]: debug 2022-01-31T22:44:35.597+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 233889 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:35.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[41853]: debug 2022-01-31T22:44:35.574+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.575078+0000) 2022-01-31T22:44:35.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[51620]: debug 2022-01-31T22:44:35.572+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.573169+0000) 2022-01-31T22:44:35.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[46715]: debug 2022-01-31T22:44:35.506+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.507140+0000) 2022-01-31T22:44:35.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:35 smithi171 conmon[46715]: debug 2022-01-31T22:44:35.574+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.575098+0000) 2022-01-31T22:44:35.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:35 smithi167 conmon[60316]: debug 2022-01-31T22:44:35.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.767323+0000) 2022-01-31T22:44:35.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:35 smithi167 conmon[49112]: debug 2022-01-31T22:44:35.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:35.840832+0000) 2022-01-31T22:44:36.325 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:36 smithi167 conmon[54076]: debug 2022-01-31T22:44:36.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:36.007517+0000) 2022-01-31T22:44:36.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:36 smithi171 conmon[41853]: debug 2022-01-31T22:44:36.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:36.457239+0000) 2022-01-31T22:44:36.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:36 smithi171 conmon[51620]: debug 2022-01-31T22:44:36.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:36.375787+0000) 2022-01-31T22:44:36.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:36 smithi171 conmon[46715]: debug 2022-01-31T22:44:36.506+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:36.507289+0000) 2022-01-31T22:44:36.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:36 smithi167 conmon[60316]: debug 2022-01-31T22:44:36.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:36.767479+0000) 2022-01-31T22:44:36.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:36 smithi167 conmon[49112]: debug 2022-01-31T22:44:36.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:36.840953+0000) 2022-01-31T22:44:37.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:37 smithi167 conmon[54076]: debug 2022-01-31T22:44:37.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:37.007686+0000) 2022-01-31T22:44:37.497 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:37 smithi171 conmon[51620]: debug 2022-01-31T22:44:37.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:37.375933+0000) 2022-01-31T22:44:37.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:37 smithi171 conmon[41853]: debug 2022-01-31T22:44:37.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:37.457345+0000) 2022-01-31T22:44:37.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:37 smithi171 conmon[46715]: debug 2022-01-31T22:44:37.506+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:37.507465+0000) 2022-01-31T22:44:37.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:37 smithi167 conmon[49112]: debug 2022-01-31T22:44:37.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:37.841106+0000) 2022-01-31T22:44:37.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:37 smithi167 conmon[60316]: debug 2022-01-31T22:44:37.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:37.767655+0000) 2022-01-31T22:44:38.332 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:38 smithi167 conmon[54076]: debug 2022-01-31T22:44:38.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:38.007869+0000) 2022-01-31T22:44:38.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:38 smithi171 conmon[41853]: debug 2022-01-31T22:44:38.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:38.457539+0000) 2022-01-31T22:44:38.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:38 smithi171 conmon[51620]: debug 2022-01-31T22:44:38.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:38.376037+0000) 2022-01-31T22:44:38.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:38 smithi171 conmon[46715]: debug 2022-01-31T22:44:38.507+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:38.507659+0000) 2022-01-31T22:44:38.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:38 smithi167 conmon[49112]: debug 2022-01-31T22:44:38.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:38.841261+0000) 2022-01-31T22:44:38.909 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:38 smithi167 conmon[60316]: debug 2022-01-31T22:44:38.766+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:38.767853+0000) 2022-01-31T22:44:39.236 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:38 smithi171 conmon[35325]: debug 2022-01-31T22:44:38.902+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:44:39.336 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:39 smithi167 conmon[54076]: debug 2022-01-31T22:44:39.006+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:39.008017+0000) 2022-01-31T22:44:39.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:39 smithi171 conmon[41853]: debug 2022-01-31T22:44:39.456+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:39.457721+0000) 2022-01-31T22:44:39.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:39 smithi171 conmon[51620]: debug 2022-01-31T22:44:39.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:39.376173+0000) 2022-01-31T22:44:39.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:39 smithi171 conmon[46715]: debug 2022-01-31T22:44:39.507+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:39.507843+0000) 2022-01-31T22:44:39.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:39 smithi167 conmon[49112]: debug 2022-01-31T22:44:39.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:39.841445+0000) 2022-01-31T22:44:39.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:39 smithi167 conmon[60316]: debug 2022-01-31T22:44:39.767+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:39.768073+0000) 2022-01-31T22:44:40.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:40 smithi167 conmon[54076]: debug 2022-01-31T22:44:40.007+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.008191+0000) 2022-01-31T22:44:40.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[41853]: debug 2022-01-31T22:44:40.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.457849+0000) 2022-01-31T22:44:40.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[51620]: debug 2022-01-31T22:44:40.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.376354+0000) 2022-01-31T22:44:40.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:40 smithi167 conmon[60316]: debug 2022-01-31T22:44:40.598+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.600307+0000) 2022-01-31T22:44:40.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[35325]: debug 2022-01-31T22:44:40.625+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234002 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:40.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[46715]: debug 2022-01-31T22:44:40.507+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.508052+0000) 2022-01-31T22:44:40.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[46715]: debug 2022-01-31T22:44:40.600+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.601256+0000) 2022-01-31T22:44:40.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[51620]: debug 2022-01-31T22:44:40.600+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.600768+0000) 2022-01-31T22:44:40.849 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:40 smithi171 conmon[41853]: debug 2022-01-31T22:44:40.600+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.600829+0000) 2022-01-31T22:44:40.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:40 smithi167 conmon[49112]: debug 2022-01-31T22:44:40.600+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.601756+0000) 2022-01-31T22:44:40.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:40 smithi167 conmon[49112]: debug 2022-01-31T22:44:40.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.841650+0000) 2022-01-31T22:44:40.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:40 smithi167 conmon[54076]: debug 2022-01-31T22:44:40.598+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.600463+0000) 2022-01-31T22:44:40.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:40 smithi167 conmon[60316]: debug 2022-01-31T22:44:40.767+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:40.768262+0000) 2022-01-31T22:44:41.341 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:41 smithi167 conmon[54076]: debug 2022-01-31T22:44:41.007+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:41.008335+0000) 2022-01-31T22:44:41.447 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:41 smithi171 conmon[51620]: debug 2022-01-31T22:44:41.375+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:41.376532+0000) 2022-01-31T22:44:41.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:41 smithi171 conmon[46715]: debug 2022-01-31T22:44:41.507+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:41.508242+0000) 2022-01-31T22:44:41.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:41 smithi171 conmon[41853]: debug 2022-01-31T22:44:41.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:41.457950+0000) 2022-01-31T22:44:41.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:41 smithi167 conmon[49112]: debug 2022-01-31T22:44:41.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:41.841812+0000) 2022-01-31T22:44:41.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:41 smithi167 conmon[60316]: debug 2022-01-31T22:44:41.767+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:41.768434+0000) 2022-01-31T22:44:41.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:41 smithi167 conmon[60316]: 2022-01-31T22:44:42.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:42 smithi167 conmon[54076]: debug 2022-01-31T22:44:42.007+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:42.008481+0000) 2022-01-31T22:44:42.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:42 smithi171 conmon[41853]: debug 2022-01-31T22:44:42.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:42.458031+0000) 2022-01-31T22:44:42.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:42 smithi171 conmon[51620]: debug 2022-01-31T22:44:42.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:42.376676+0000) 2022-01-31T22:44:42.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:42 smithi171 conmon[46715]: debug 2022-01-31T22:44:42.507+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:42.508325+0000) 2022-01-31T22:44:42.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:42 smithi167 conmon[49112]: debug 2022-01-31T22:44:42.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:42.841998+0000) 2022-01-31T22:44:42.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:42 smithi167 conmon[60316]: debug 2022-01-31T22:44:42.767+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:42.768643+0000) 2022-01-31T22:44:43.346 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:43 smithi167 conmon[54076]: debug 2022-01-31T22:44:43.007+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:43.008677+0000) 2022-01-31T22:44:43.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:43 smithi171 conmon[41853]: debug 2022-01-31T22:44:43.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:43.458213+0000) 2022-01-31T22:44:43.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:43 smithi171 conmon[51620]: debug 2022-01-31T22:44:43.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:43.376841+0000) 2022-01-31T22:44:43.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:43 smithi171 conmon[46715]: debug 2022-01-31T22:44:43.507+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:43.508494+0000) 2022-01-31T22:44:43.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:43 smithi167 conmon[49112]: debug 2022-01-31T22:44:43.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:43.842201+0000) 2022-01-31T22:44:43.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:43 smithi167 conmon[60316]: debug 2022-01-31T22:44:43.767+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:43.768810+0000) 2022-01-31T22:44:44.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:44 smithi167 conmon[54076]: debug 2022-01-31T22:44:44.007+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:44.008789+0000) 2022-01-31T22:44:44.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:44 smithi171 conmon[41853]: debug 2022-01-31T22:44:44.457+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:44.458417+0000) 2022-01-31T22:44:44.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:44 smithi171 conmon[51620]: debug 2022-01-31T22:44:44.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:44.377011+0000) 2022-01-31T22:44:44.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:44 smithi171 conmon[46715]: debug 2022-01-31T22:44:44.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:44.508699+0000) 2022-01-31T22:44:44.998 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:44 smithi167 conmon[49112]: debug 2022-01-31T22:44:44.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:44.842463+0000) 2022-01-31T22:44:44.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:44 smithi167 conmon[60316]: debug 2022-01-31T22:44:44.768+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:44.769056+0000) 2022-01-31T22:44:45.330 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:45 smithi167 conmon[54076]: debug 2022-01-31T22:44:45.007+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.009016+0000) 2022-01-31T22:44:45.498 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[51620]: debug 2022-01-31T22:44:45.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.377234+0000) 2022-01-31T22:44:45.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[41853]: debug 2022-01-31T22:44:45.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.458663+0000) 2022-01-31T22:44:45.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[35325]: debug 2022-01-31T22:44:45.651+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:45.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[41853]: debug 2022-01-31T22:44:45.627+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.628159+0000) 2022-01-31T22:44:45.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[51620]: debug 2022-01-31T22:44:45.628+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.629274+0000) 2022-01-31T22:44:45.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[46715]: debug 2022-01-31T22:44:45.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.508956+0000) 2022-01-31T22:44:45.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:45 smithi171 conmon[46715]: debug 2022-01-31T22:44:45.627+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.627834+0000) 2022-01-31T22:44:45.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:45 smithi167 conmon[49112]: debug 2022-01-31T22:44:45.627+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.629162+0000) 2022-01-31T22:44:45.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:45 smithi167 conmon[49112]: debug 2022-01-31T22:44:45.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.842610+0000) 2022-01-31T22:44:45.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:45 smithi167 conmon[54076]: debug 2022-01-31T22:44:45.627+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.628788+0000) 2022-01-31T22:44:45.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:45 smithi167 conmon[60316]: debug 2022-01-31T22:44:45.626+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.627939+0000) 2022-01-31T22:44:45.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:45 smithi167 conmon[60316]: debug 2022-01-31T22:44:45.768+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:45.769226+0000) 2022-01-31T22:44:46.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:46 smithi167 conmon[54076]: debug 2022-01-31T22:44:46.008+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:46.009197+0000) 2022-01-31T22:44:46.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:46 smithi171 conmon[41853]: debug 2022-01-31T22:44:46.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:46.458825+0000) 2022-01-31T22:44:46.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:46 smithi171 conmon[51620]: debug 2022-01-31T22:44:46.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:46.377365+0000) 2022-01-31T22:44:46.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:46 smithi171 conmon[46715]: debug 2022-01-31T22:44:46.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:46.509074+0000) 2022-01-31T22:44:46.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:46 smithi167 conmon[49112]: debug 2022-01-31T22:44:46.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:46.842729+0000) 2022-01-31T22:44:46.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:46 smithi167 conmon[60316]: debug 2022-01-31T22:44:46.768+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:46.769396+0000) 2022-01-31T22:44:47.366 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:47 smithi167 conmon[54076]: debug 2022-01-31T22:44:47.008+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:47.009348+0000) 2022-01-31T22:44:47.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:47 smithi171 conmon[41853]: debug 2022-01-31T22:44:47.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:47.458971+0000) 2022-01-31T22:44:47.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:47 smithi171 conmon[51620]: debug 2022-01-31T22:44:47.376+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:47.377553+0000) 2022-01-31T22:44:47.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:47 smithi171 conmon[46715]: debug 2022-01-31T22:44:47.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:47.509167+0000) 2022-01-31T22:44:47.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:47 smithi167 conmon[49112]: debug 2022-01-31T22:44:47.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:47.842875+0000) 2022-01-31T22:44:47.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:47 smithi167 conmon[60316]: debug 2022-01-31T22:44:47.768+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:47.769622+0000) 2022-01-31T22:44:48.370 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:48 smithi167 conmon[54076]: debug 2022-01-31T22:44:48.008+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:48.009523+0000) 2022-01-31T22:44:48.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:48 smithi171 conmon[41853]: debug 2022-01-31T22:44:48.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:48.459119+0000) 2022-01-31T22:44:48.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:48 smithi171 conmon[51620]: debug 2022-01-31T22:44:48.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:48.377726+0000) 2022-01-31T22:44:48.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:48 smithi171 conmon[46715]: debug 2022-01-31T22:44:48.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:48.509309+0000) 2022-01-31T22:44:48.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:48 smithi167 conmon[49112]: debug 2022-01-31T22:44:48.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:48.842976+0000) 2022-01-31T22:44:48.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:48 smithi167 conmon[60316]: debug 2022-01-31T22:44:48.769+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:48.769804+0000) 2022-01-31T22:44:49.298 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:49 smithi167 conmon[54076]: debug 2022-01-31T22:44:49.008+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:49.009669+0000) 2022-01-31T22:44:49.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:49 smithi171 conmon[41853]: debug 2022-01-31T22:44:49.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:49.459281+0000) 2022-01-31T22:44:49.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:49 smithi171 conmon[51620]: debug 2022-01-31T22:44:49.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:49.377898+0000) 2022-01-31T22:44:49.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:49 smithi171 conmon[46715]: debug 2022-01-31T22:44:49.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:49.509537+0000) 2022-01-31T22:44:49.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:49 smithi167 conmon[49112]: debug 2022-01-31T22:44:49.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:49.843143+0000) 2022-01-31T22:44:49.999 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:49 smithi167 conmon[60316]: debug 2022-01-31T22:44:49.768+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:49.769953+0000) 2022-01-31T22:44:50.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:50 smithi167 conmon[54076]: debug 2022-01-31T22:44:50.009+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.009850+0000) 2022-01-31T22:44:50.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[41853]: debug 2022-01-31T22:44:50.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.459445+0000) 2022-01-31T22:44:50.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[51620]: debug 2022-01-31T22:44:50.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.378062+0000) 2022-01-31T22:44:50.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[35325]: debug 2022-01-31T22:44:50.677+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234225 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:50.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[41853]: debug 2022-01-31T22:44:50.654+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.655179+0000) 2022-01-31T22:44:50.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[51620]: debug 2022-01-31T22:44:50.654+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.654954+0000) 2022-01-31T22:44:50.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[46715]: debug 2022-01-31T22:44:50.508+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.509675+0000) 2022-01-31T22:44:50.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:50 smithi171 conmon[46715]: debug 2022-01-31T22:44:50.653+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.653861+0000) 2022-01-31T22:44:50.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:50 smithi167 conmon[49112]: debug 2022-01-31T22:44:50.654+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.655872+0000) 2022-01-31T22:44:50.909 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:50 smithi167 conmon[49112]: debug 2022-01-31T22:44:50.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.843274+0000) 2022-01-31T22:44:50.910 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:50 smithi167 conmon[54076]: debug 2022-01-31T22:44:50.653+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.655036+0000) 2022-01-31T22:44:50.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:50 smithi167 conmon[60316]: debug 2022-01-31T22:44:50.653+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.655284+0000) 2022-01-31T22:44:50.910 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:50 smithi167 conmon[60316]: debug 2022-01-31T22:44:50.769+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:50.770143+0000) 2022-01-31T22:44:51.379 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:51 smithi167 conmon[54076]: debug 2022-01-31T22:44:51.009+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:51.010066+0000) 2022-01-31T22:44:51.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:51 smithi171 conmon[41853]: debug 2022-01-31T22:44:51.458+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:51.459652+0000) 2022-01-31T22:44:51.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:51 smithi171 conmon[51620]: debug 2022-01-31T22:44:51.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:51.378249+0000) 2022-01-31T22:44:51.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:51 smithi171 conmon[46715]: debug 2022-01-31T22:44:51.509+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:51.509813+0000) 2022-01-31T22:44:51.999 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:51 smithi167 conmon[49112]: debug 2022-01-31T22:44:51.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:51.843432+0000) 2022-01-31T22:44:52.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:51 smithi167 conmon[49112]: 2022-01-31T22:44:52.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:51 smithi167 conmon[60316]: debug 2022-01-31T22:44:51.769+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:51.770296+0000) 2022-01-31T22:44:52.383 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:52 smithi167 conmon[54076]: debug 2022-01-31T22:44:52.009+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:52.010195+0000) 2022-01-31T22:44:52.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:52 smithi171 conmon[41853]: debug 2022-01-31T22:44:52.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:52.459807+0000) 2022-01-31T22:44:52.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:52 smithi171 conmon[51620]: debug 2022-01-31T22:44:52.377+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:52.378433+0000) 2022-01-31T22:44:52.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:52 smithi171 conmon[46715]: debug 2022-01-31T22:44:52.509+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:52.509931+0000) 2022-01-31T22:44:53.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:52 smithi167 conmon[49112]: debug 2022-01-31T22:44:52.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:52.843622+0000) 2022-01-31T22:44:53.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:52 smithi167 conmon[60316]: debug 2022-01-31T22:44:52.769+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:52.770530+0000) 2022-01-31T22:44:53.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:53 smithi167 conmon[54076]: debug 2022-01-31T22:44:53.009+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:53.010369+0000) 2022-01-31T22:44:53.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:53 smithi171 conmon[51620]: debug 2022-01-31T22:44:53.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:53.378617+0000) 2022-01-31T22:44:53.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:53 smithi171 conmon[41853]: debug 2022-01-31T22:44:53.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:53.459932+0000) 2022-01-31T22:44:53.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:53 smithi171 conmon[46715]: debug 2022-01-31T22:44:53.509+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:53.510049+0000) 2022-01-31T22:44:54.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:53 smithi167 conmon[49112]: debug 2022-01-31T22:44:53.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:53.843752+0000) 2022-01-31T22:44:54.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:53 smithi167 conmon[60316]: debug 2022-01-31T22:44:53.770+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:53.770718+0000) 2022-01-31T22:44:54.238 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:53 smithi171 conmon[35325]: debug 2022-01-31T22:44:53.903+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:44:54.390 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:54 smithi167 conmon[54076]: debug 2022-01-31T22:44:54.009+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:54.010604+0000) 2022-01-31T22:44:54.499 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:54 smithi171 conmon[51620]: debug 2022-01-31T22:44:54.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:54.378835+0000) 2022-01-31T22:44:54.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:54 smithi171 conmon[41853]: debug 2022-01-31T22:44:54.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:54.460147+0000) 2022-01-31T22:44:54.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:54 smithi171 conmon[46715]: debug 2022-01-31T22:44:54.509+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:54.510256+0000) 2022-01-31T22:44:55.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:54 smithi167 conmon[49112]: debug 2022-01-31T22:44:54.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:54.843916+0000) 2022-01-31T22:44:55.000 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:54 smithi167 conmon[60316]: debug 2022-01-31T22:44:54.770+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:54.770914+0000) 2022-01-31T22:44:55.394 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:55 smithi167 conmon[54076]: debug 2022-01-31T22:44:55.010+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.010804+0000) 2022-01-31T22:44:55.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[51620]: debug 2022-01-31T22:44:55.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.379024+0000) 2022-01-31T22:44:55.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[41853]: debug 2022-01-31T22:44:55.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.460325+0000) 2022-01-31T22:44:55.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[35325]: debug 2022-01-31T22:44:55.703+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234332 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:44:55.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[41853]: debug 2022-01-31T22:44:55.681+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.682204+0000) 2022-01-31T22:44:55.848 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[51620]: debug 2022-01-31T22:44:55.679+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.680707+0000) 2022-01-31T22:44:55.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[46715]: debug 2022-01-31T22:44:55.509+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.510443+0000) 2022-01-31T22:44:55.849 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:55 smithi171 conmon[46715]: debug 2022-01-31T22:44:55.679+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.680446+0000) 2022-01-31T22:44:56.000 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:55 smithi167 conmon[54076]: debug 2022-01-31T22:44:55.680+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.681778+0000) 2022-01-31T22:44:56.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:55 smithi167 conmon[60316]: debug 2022-01-31T22:44:55.679+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.681247+0000) 2022-01-31T22:44:56.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:55 smithi167 conmon[60316]: debug 2022-01-31T22:44:55.770+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.771079+0000) 2022-01-31T22:44:56.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:55 smithi167 conmon[49112]: debug 2022-01-31T22:44:55.680+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.681974+0000) 2022-01-31T22:44:56.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:55 smithi167 conmon[49112]: debug 2022-01-31T22:44:55.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:55.844121+0000) 2022-01-31T22:44:56.396 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:56 smithi167 conmon[54076]: debug 2022-01-31T22:44:56.010+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:56.010988+0000) 2022-01-31T22:44:56.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:56 smithi171 conmon[41853]: debug 2022-01-31T22:44:56.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:56.460451+0000) 2022-01-31T22:44:56.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:56 smithi171 conmon[51620]: debug 2022-01-31T22:44:56.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:56.379175+0000) 2022-01-31T22:44:56.759 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:56 smithi171 conmon[46715]: debug 2022-01-31T22:44:56.509+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:56.510623+0000) 2022-01-31T22:44:57.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:56 smithi167 conmon[49112]: debug 2022-01-31T22:44:56.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:56.844274+0000) 2022-01-31T22:44:57.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:56 smithi167 conmon[60316]: debug 2022-01-31T22:44:56.770+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:56.771220+0000) 2022-01-31T22:44:57.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:57 smithi167 conmon[54076]: debug 2022-01-31T22:44:57.010+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:57.011140+0000) 2022-01-31T22:44:57.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:57 smithi171 conmon[51620]: debug 2022-01-31T22:44:57.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:57.379266+0000) 2022-01-31T22:44:57.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:57 smithi171 conmon[41853]: debug 2022-01-31T22:44:57.459+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:57.460593+0000) 2022-01-31T22:44:57.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:57 smithi171 conmon[46715]: debug 2022-01-31T22:44:57.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:57.510798+0000) 2022-01-31T22:44:58.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:57 smithi167 conmon[49112]: debug 2022-01-31T22:44:57.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:57.844431+0000) 2022-01-31T22:44:58.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:57 smithi167 conmon[60316]: debug 2022-01-31T22:44:57.770+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:57.771376+0000) 2022-01-31T22:44:58.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:58 smithi167 conmon[54076]: debug 2022-01-31T22:44:58.010+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:58.011327+0000) 2022-01-31T22:44:58.500 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:58 smithi171 conmon[51620]: debug 2022-01-31T22:44:58.378+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:58.379465+0000) 2022-01-31T22:44:58.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:58 smithi171 conmon[41853]: debug 2022-01-31T22:44:58.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:58.460793+0000) 2022-01-31T22:44:58.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:58 smithi171 conmon[46715]: debug 2022-01-31T22:44:58.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:58.510961+0000) 2022-01-31T22:44:59.000 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:58 smithi167 conmon[49112]: debug 2022-01-31T22:44:58.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:58.844586+0000) 2022-01-31T22:44:59.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:58 smithi167 conmon[60316]: debug 2022-01-31T22:44:58.771+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:58.771553+0000) 2022-01-31T22:44:59.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:44:59 smithi167 conmon[54076]: debug 2022-01-31T22:44:59.011+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:59.011491+0000) 2022-01-31T22:44:59.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:44:59 smithi171 conmon[41853]: debug 2022-01-31T22:44:59.460+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:59.460988+0000) 2022-01-31T22:44:59.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:44:59 smithi171 conmon[51620]: debug 2022-01-31T22:44:59.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:59.379650+0000) 2022-01-31T22:44:59.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:44:59 smithi171 conmon[46715]: debug 2022-01-31T22:44:59.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:59.511144+0000) 2022-01-31T22:45:00.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:44:59 smithi167 conmon[49112]: debug 2022-01-31T22:44:59.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:59.844774+0000) 2022-01-31T22:45:00.001 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:44:59 smithi167 conmon[60316]: debug 2022-01-31T22:44:59.771+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:44:59.771657+0000) 2022-01-31T22:45:00.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:00 smithi167 conmon[54076]: debug 2022-01-31T22:45:00.011+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.011605+0000) 2022-01-31T22:45:00.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[41853]: debug 2022-01-31T22:45:00.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.461179+0000) 2022-01-31T22:45:00.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[51620]: debug 2022-01-31T22:45:00.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.379812+0000) 2022-01-31T22:45:00.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[35325]: debug 2022-01-31T22:45:00.730+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234446 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:00.848 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[41853]: debug 2022-01-31T22:45:00.706+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.707501+0000) 2022-01-31T22:45:00.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[51620]: debug 2022-01-31T22:45:00.706+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.706827+0000) 2022-01-31T22:45:00.850 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[46715]: debug 2022-01-31T22:45:00.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.511347+0000) 2022-01-31T22:45:00.851 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:00 smithi171 conmon[46715]: debug 2022-01-31T22:45:00.705+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.706522+0000) 2022-01-31T22:45:01.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:00 smithi167 conmon[49112]: debug 2022-01-31T22:45:00.705+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.706392+0000) 2022-01-31T22:45:01.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:00 smithi167 conmon[49112]: debug 2022-01-31T22:45:00.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.844933+0000) 2022-01-31T22:45:01.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:00 smithi167 conmon[54076]: debug 2022-01-31T22:45:00.705+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.706225+0000) 2022-01-31T22:45:01.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:00 smithi167 conmon[60316]: debug 2022-01-31T22:45:00.706+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.707151+0000) 2022-01-31T22:45:01.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:00 smithi167 conmon[60316]: debug 2022-01-31T22:45:00.771+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:00.771815+0000) 2022-01-31T22:45:01.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:01 smithi167 conmon[54076]: debug 2022-01-31T22:45:01.011+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:01.011746+0000) 2022-01-31T22:45:01.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:01 smithi171 conmon[41853]: debug 2022-01-31T22:45:01.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:01.461393+0000) 2022-01-31T22:45:01.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:01 smithi171 conmon[51620]: debug 2022-01-31T22:45:01.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:01.379996+0000) 2022-01-31T22:45:01.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:01 smithi171 conmon[46715]: debug 2022-01-31T22:45:01.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:01.511525+0000) 2022-01-31T22:45:02.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:01 smithi167 conmon[49112]: debug 2022-01-31T22:45:01.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:01.845058+0000) 2022-01-31T22:45:02.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:01 smithi167 conmon[60316]: debug 2022-01-31T22:45:01.771+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:01.771991+0000) 2022-01-31T22:45:02.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:02 smithi167 conmon[54076]: debug 2022-01-31T22:45:02.011+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:02.011890+0000) 2022-01-31T22:45:02.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:02 smithi171 conmon[51620]: debug 2022-01-31T22:45:02.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:02.380177+0000) 2022-01-31T22:45:02.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:02 smithi171 conmon[41853]: debug 2022-01-31T22:45:02.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:02.461557+0000) 2022-01-31T22:45:02.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:02 smithi171 conmon[46715]: debug 2022-01-31T22:45:02.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:02.511688+0000) 2022-01-31T22:45:03.001 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:02 smithi167 conmon[49112]: debug 2022-01-31T22:45:02.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:02.845251+0000) 2022-01-31T22:45:03.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:02 smithi167 conmon[60316]: debug 2022-01-31T22:45:02.771+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:02.772182+0000) 2022-01-31T22:45:03.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:03 smithi167 conmon[54076]: debug 2022-01-31T22:45:03.011+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:03.012094+0000) 2022-01-31T22:45:03.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:03 smithi171 conmon[41853]: debug 2022-01-31T22:45:03.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:03.461710+0000) 2022-01-31T22:45:03.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:03 smithi171 conmon[51620]: debug 2022-01-31T22:45:03.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:03.380390+0000) 2022-01-31T22:45:03.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:03 smithi171 conmon[46715]: debug 2022-01-31T22:45:03.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:03.511893+0000) 2022-01-31T22:45:04.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:03 smithi167 conmon[49112]: debug 2022-01-31T22:45:03.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:03.845457+0000) 2022-01-31T22:45:04.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:03 smithi167 conmon[60316]: debug 2022-01-31T22:45:03.771+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:03.772378+0000) 2022-01-31T22:45:04.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:04 smithi167 conmon[54076]: debug 2022-01-31T22:45:04.011+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:04.012304+0000) 2022-01-31T22:45:04.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:04 smithi171 conmon[51620]: debug 2022-01-31T22:45:04.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:04.380585+0000) 2022-01-31T22:45:04.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:04 smithi171 conmon[41853]: debug 2022-01-31T22:45:04.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:04.461928+0000) 2022-01-31T22:45:04.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:04 smithi171 conmon[46715]: debug 2022-01-31T22:45:04.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:04.512053+0000) 2022-01-31T22:45:05.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:04 smithi167 conmon[49112]: debug 2022-01-31T22:45:04.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:04.845589+0000) 2022-01-31T22:45:05.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:04 smithi167 conmon[60316]: debug 2022-01-31T22:45:04.772+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:04.772542+0000) 2022-01-31T22:45:05.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:05 smithi167 conmon[54076]: debug 2022-01-31T22:45:05.012+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.012495+0000) 2022-01-31T22:45:05.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[41853]: debug 2022-01-31T22:45:05.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.462129+0000) 2022-01-31T22:45:05.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[51620]: debug 2022-01-31T22:45:05.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.380809+0000) 2022-01-31T22:45:05.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[41853]: debug 2022-01-31T22:45:05.733+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.734305+0000) 2022-01-31T22:45:05.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[35325]: debug 2022-01-31T22:45:05.757+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234556 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:05.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[46715]: debug 2022-01-31T22:45:05.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.512223+0000) 2022-01-31T22:45:05.848 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[46715]: debug 2022-01-31T22:45:05.732+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.733575+0000) 2022-01-31T22:45:05.849 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:05 smithi171 conmon[51620]: debug 2022-01-31T22:45:05.733+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.733817+0000) 2022-01-31T22:45:06.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:05 smithi167 conmon[49112]: debug 2022-01-31T22:45:05.732+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.733388+0000) 2022-01-31T22:45:06.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:05 smithi167 conmon[49112]: debug 2022-01-31T22:45:05.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.845756+0000) 2022-01-31T22:45:06.003 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:05 smithi167 conmon[54076]: debug 2022-01-31T22:45:05.733+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.733494+0000) 2022-01-31T22:45:06.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:05 smithi167 conmon[60316]: debug 2022-01-31T22:45:05.733+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.733884+0000) 2022-01-31T22:45:06.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:05 smithi167 conmon[60316]: debug 2022-01-31T22:45:05.772+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:05.772727+0000) 2022-01-31T22:45:06.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:06 smithi167 conmon[54076]: debug 2022-01-31T22:45:06.012+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:06.012703+0000) 2022-01-31T22:45:06.501 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:06 smithi171 conmon[51620]: debug 2022-01-31T22:45:06.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:06.380981+0000) 2022-01-31T22:45:06.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:06 smithi171 conmon[41853]: debug 2022-01-31T22:45:06.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:06.462302+0000) 2022-01-31T22:45:06.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:06 smithi171 conmon[46715]: debug 2022-01-31T22:45:06.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:06.512364+0000) 2022-01-31T22:45:07.002 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:06 smithi167 conmon[60316]: debug 2022-01-31T22:45:06.772+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:06.772875+0000) 2022-01-31T22:45:07.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:06 smithi167 conmon[49112]: debug 2022-01-31T22:45:06.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:06.845916+0000) 2022-01-31T22:45:07.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:07 smithi167 conmon[54076]: debug 2022-01-31T22:45:07.012+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:07.012838+0000) 2022-01-31T22:45:07.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:07 smithi171 conmon[41853]: debug 2022-01-31T22:45:07.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:07.462455+0000) 2022-01-31T22:45:07.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:07 smithi171 conmon[51620]: debug 2022-01-31T22:45:07.379+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:07.381115+0000) 2022-01-31T22:45:07.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:07 smithi171 conmon[46715]: debug 2022-01-31T22:45:07.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:07.512475+0000) 2022-01-31T22:45:08.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:07 smithi167 conmon[49112]: debug 2022-01-31T22:45:07.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:07.846094+0000) 2022-01-31T22:45:08.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:07 smithi167 conmon[60316]: debug 2022-01-31T22:45:07.772+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:07.773026+0000) 2022-01-31T22:45:08.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:08 smithi167 conmon[54076]: debug 2022-01-31T22:45:08.012+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:08.012989+0000) 2022-01-31T22:45:08.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:08 smithi171 conmon[41853]: debug 2022-01-31T22:45:08.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:08.462616+0000) 2022-01-31T22:45:08.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:08 smithi171 conmon[51620]: debug 2022-01-31T22:45:08.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:08.381308+0000) 2022-01-31T22:45:08.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:08 smithi171 conmon[46715]: debug 2022-01-31T22:45:08.510+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:08.512634+0000) 2022-01-31T22:45:09.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:08 smithi167 conmon[49112]: debug 2022-01-31T22:45:08.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:08.846234+0000) 2022-01-31T22:45:09.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:08 smithi167 conmon[60316]: debug 2022-01-31T22:45:08.772+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:08.773212+0000) 2022-01-31T22:45:09.241 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:08 smithi171 conmon[35325]: debug 2022-01-31T22:45:08.903+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:45:09.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:09 smithi167 conmon[54076]: debug 2022-01-31T22:45:09.012+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:09.013210+0000) 2022-01-31T22:45:09.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:09 smithi171 conmon[41853]: debug 2022-01-31T22:45:09.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:09.462768+0000) 2022-01-31T22:45:09.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:09 smithi171 conmon[51620]: debug 2022-01-31T22:45:09.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:09.381522+0000) 2022-01-31T22:45:09.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:09 smithi171 conmon[46715]: debug 2022-01-31T22:45:09.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:09.512827+0000) 2022-01-31T22:45:10.002 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:09 smithi167 conmon[49112]: debug 2022-01-31T22:45:09.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:09.846438+0000) 2022-01-31T22:45:10.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:09 smithi167 conmon[60316]: debug 2022-01-31T22:45:09.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:09.773415+0000) 2022-01-31T22:45:10.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:10 smithi167 conmon[54076]: debug 2022-01-31T22:45:10.012+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.013409+0000) 2022-01-31T22:45:10.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[41853]: debug 2022-01-31T22:45:10.461+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.462878+0000) 2022-01-31T22:45:10.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[51620]: debug 2022-01-31T22:45:10.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.381702+0000) 2022-01-31T22:45:10.756 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[51620]: debug 2022-01-31T22:45:10.758+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.760684+0000) 2022-01-31T22:45:10.757 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[41853]: debug 2022-01-31T22:45:10.758+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.760081+0000) 2022-01-31T22:45:10.757 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[46715]: debug 2022-01-31T22:45:10.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.512980+0000) 2022-01-31T22:45:10.758 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[46715]: debug 2022-01-31T22:45:10.759+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.761277+0000) 2022-01-31T22:45:11.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:10 smithi167 conmon[49112]: debug 2022-01-31T22:45:10.760+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.761106+0000) 2022-01-31T22:45:11.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:10 smithi167 conmon[49112]: debug 2022-01-31T22:45:10.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.846586+0000) 2022-01-31T22:45:11.004 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:10 smithi167 conmon[54076]: debug 2022-01-31T22:45:10.767+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.768395+0000) 2022-01-31T22:45:11.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:10 smithi167 conmon[60316]: debug 2022-01-31T22:45:10.760+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.760987+0000) 2022-01-31T22:45:11.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:10 smithi167 conmon[60316]: debug 2022-01-31T22:45:10.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:10.773595+0000) 2022-01-31T22:45:11.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:10 smithi171 conmon[35325]: debug 2022-01-31T22:45:10.788+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234670 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:11.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:11 smithi167 conmon[54076]: debug 2022-01-31T22:45:11.013+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:11.013585+0000) 2022-01-31T22:45:11.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:11 smithi171 conmon[41853]: debug 2022-01-31T22:45:11.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:11.463058+0000) 2022-01-31T22:45:11.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:11 smithi171 conmon[51620]: debug 2022-01-31T22:45:11.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:11.381903+0000) 2022-01-31T22:45:11.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:11 smithi171 conmon[46715]: debug 2022-01-31T22:45:11.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:11.513152+0000) 2022-01-31T22:45:12.003 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:11 smithi167 conmon[60316]: debug 2022-01-31T22:45:11.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:11.773736+0000) 2022-01-31T22:45:12.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:11 smithi167 conmon[49112]: debug 2022-01-31T22:45:11.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:11.846739+0000) 2022-01-31T22:45:12.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:12 smithi167 conmon[54076]: debug 2022-01-31T22:45:12.013+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:12.013710+0000) 2022-01-31T22:45:12.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:12 smithi171 conmon[51620]: debug 2022-01-31T22:45:12.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:12.382069+0000) 2022-01-31T22:45:12.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:12 smithi171 conmon[41853]: debug 2022-01-31T22:45:12.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:12.463214+0000) 2022-01-31T22:45:12.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:12 smithi171 conmon[46715]: debug 2022-01-31T22:45:12.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:12.513325+0000) 2022-01-31T22:45:13.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:12 smithi167 conmon[49112]: debug 2022-01-31T22:45:12.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:12.846896+0000) 2022-01-31T22:45:13.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:12 smithi167 conmon[60316]: debug 2022-01-31T22:45:12.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:12.773860+0000) 2022-01-31T22:45:13.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:13 smithi167 conmon[54076]: debug 2022-01-31T22:45:13.013+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:13.013892+0000) 2022-01-31T22:45:13.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:13 smithi171 conmon[41853]: debug 2022-01-31T22:45:13.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:13.463405+0000) 2022-01-31T22:45:13.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:13 smithi171 conmon[51620]: debug 2022-01-31T22:45:13.380+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:13.382245+0000) 2022-01-31T22:45:13.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:13 smithi171 conmon[46715]: debug 2022-01-31T22:45:13.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:13.513458+0000) 2022-01-31T22:45:14.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:13 smithi167 conmon[49112]: debug 2022-01-31T22:45:13.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:13.847088+0000) 2022-01-31T22:45:14.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:13 smithi167 conmon[60316]: debug 2022-01-31T22:45:13.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:13.774043+0000) 2022-01-31T22:45:14.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:14 smithi167 conmon[54076]: debug 2022-01-31T22:45:14.013+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:14.014086+0000) 2022-01-31T22:45:14.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:14 smithi171 conmon[41853]: debug 2022-01-31T22:45:14.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:14.463592+0000) 2022-01-31T22:45:14.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:14 smithi171 conmon[51620]: debug 2022-01-31T22:45:14.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:14.382408+0000) 2022-01-31T22:45:14.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:14 smithi171 conmon[46715]: debug 2022-01-31T22:45:14.511+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:14.513663+0000) 2022-01-31T22:45:15.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:14 smithi167 conmon[49112]: debug 2022-01-31T22:45:14.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:14.847287+0000) 2022-01-31T22:45:15.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:14 smithi167 conmon[60316]: debug 2022-01-31T22:45:14.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:14.774150+0000) 2022-01-31T22:45:15.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:15 smithi167 conmon[54076]: debug 2022-01-31T22:45:15.013+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.014306+0000) 2022-01-31T22:45:15.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[51620]: debug 2022-01-31T22:45:15.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.382606+0000) 2022-01-31T22:45:15.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[41853]: debug 2022-01-31T22:45:15.462+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.463736+0000) 2022-01-31T22:45:15.782 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[46715]: debug 2022-01-31T22:45:15.512+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.513849+0000) 2022-01-31T22:45:16.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:15 smithi167 conmon[49112]: debug 2022-01-31T22:45:15.793+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.793510+0000) 2022-01-31T22:45:16.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:15 smithi167 conmon[49112]: debug 2022-01-31T22:45:15.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.847450+0000) 2022-01-31T22:45:16.005 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:15 smithi167 conmon[54076]: debug 2022-01-31T22:45:15.793+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.793717+0000) 2022-01-31T22:45:16.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:15 smithi167 conmon[60316]: debug 2022-01-31T22:45:15.773+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.774332+0000) 2022-01-31T22:45:16.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:15 smithi167 conmon[60316]: debug 2022-01-31T22:45:15.792+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.793347+0000) 2022-01-31T22:45:16.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[46715]: debug 2022-01-31T22:45:15.792+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.794279+0000) 2022-01-31T22:45:16.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[51620]: debug 2022-01-31T22:45:15.791+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.792914+0000) 2022-01-31T22:45:16.098 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[35325]: debug 2022-01-31T22:45:15.814+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234779 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:16.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:15 smithi171 conmon[41853]: debug 2022-01-31T22:45:15.791+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:15.793448+0000) 2022-01-31T22:45:16.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:16 smithi167 conmon[54076]: debug 2022-01-31T22:45:16.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:16.014538+0000) 2022-01-31T22:45:16.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:16 smithi171 conmon[41853]: debug 2022-01-31T22:45:16.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:16.463856+0000) 2022-01-31T22:45:16.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:16 smithi171 conmon[51620]: debug 2022-01-31T22:45:16.381+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:16.382791+0000) 2022-01-31T22:45:16.847 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:16 smithi171 conmon[46715]: debug 2022-01-31T22:45:16.512+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:16.514006+0000) 2022-01-31T22:45:17.003 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:16 smithi167 conmon[49112]: debug 2022-01-31T22:45:16.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:16.847608+0000) 2022-01-31T22:45:17.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:16 smithi167 conmon[60316]: debug 2022-01-31T22:45:16.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:16.774540+0000) 2022-01-31T22:45:17.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:17 smithi167 conmon[54076]: debug 2022-01-31T22:45:17.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:17.014728+0000) 2022-01-31T22:45:17.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:17 smithi171 conmon[41853]: debug 2022-01-31T22:45:17.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:17.464009+0000) 2022-01-31T22:45:17.503 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:17 smithi171 conmon[51620]: debug 2022-01-31T22:45:17.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:17.382934+0000) 2022-01-31T22:45:17.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:17 smithi171 conmon[46715]: debug 2022-01-31T22:45:17.513+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:17.514165+0000) 2022-01-31T22:45:18.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:17 smithi167 conmon[49112]: debug 2022-01-31T22:45:17.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:17.847813+0000) 2022-01-31T22:45:18.004 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:17 smithi167 conmon[60316]: debug 2022-01-31T22:45:17.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:17.774662+0000) 2022-01-31T22:45:18.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:18 smithi167 conmon[54076]: debug 2022-01-31T22:45:18.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:18.014919+0000) 2022-01-31T22:45:18.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:18 smithi171 conmon[41853]: debug 2022-01-31T22:45:18.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:18.464193+0000) 2022-01-31T22:45:18.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:18 smithi171 conmon[51620]: debug 2022-01-31T22:45:18.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:18.383122+0000) 2022-01-31T22:45:18.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:18 smithi171 conmon[46715]: debug 2022-01-31T22:45:18.513+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:18.514346+0000) 2022-01-31T22:45:19.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:18 smithi167 conmon[49112]: debug 2022-01-31T22:45:18.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:18.848002+0000) 2022-01-31T22:45:19.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:18 smithi167 conmon[60316]: debug 2022-01-31T22:45:18.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:18.774823+0000) 2022-01-31T22:45:19.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:19 smithi167 conmon[54076]: debug 2022-01-31T22:45:19.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:19.015105+0000) 2022-01-31T22:45:19.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:19 smithi171 conmon[41853]: debug 2022-01-31T22:45:19.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:19.464374+0000) 2022-01-31T22:45:19.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:19 smithi171 conmon[51620]: debug 2022-01-31T22:45:19.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:19.383318+0000) 2022-01-31T22:45:19.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:19 smithi171 conmon[46715]: debug 2022-01-31T22:45:19.513+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:19.514545+0000) 2022-01-31T22:45:20.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:19 smithi167 conmon[49112]: debug 2022-01-31T22:45:19.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:19.848168+0000) 2022-01-31T22:45:20.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:19 smithi167 conmon[60316]: debug 2022-01-31T22:45:19.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:19.774992+0000) 2022-01-31T22:45:20.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:20 smithi167 conmon[54076]: debug 2022-01-31T22:45:20.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.015261+0000) 2022-01-31T22:45:20.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[41853]: debug 2022-01-31T22:45:20.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.464560+0000) 2022-01-31T22:45:20.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[51620]: debug 2022-01-31T22:45:20.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.383519+0000) 2022-01-31T22:45:20.807 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[46715]: debug 2022-01-31T22:45:20.513+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.514723+0000) 2022-01-31T22:45:21.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:20 smithi167 conmon[49112]: debug 2022-01-31T22:45:20.819+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.819906+0000) 2022-01-31T22:45:21.005 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:20 smithi167 conmon[49112]: debug 2022-01-31T22:45:20.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.848303+0000) 2022-01-31T22:45:21.006 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:20 smithi167 conmon[54076]: debug 2022-01-31T22:45:20.818+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.819302+0000) 2022-01-31T22:45:21.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:20 smithi167 conmon[60316]: debug 2022-01-31T22:45:20.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.775208+0000) 2022-01-31T22:45:21.007 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:20 smithi167 conmon[60316]: debug 2022-01-31T22:45:20.819+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.819565+0000) 2022-01-31T22:45:21.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[35325]: debug 2022-01-31T22:45:20.842+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 234893 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:21.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[41853]: debug 2022-01-31T22:45:20.818+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.819487+0000) 2022-01-31T22:45:21.098 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[46715]: debug 2022-01-31T22:45:20.818+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.819183+0000) 2022-01-31T22:45:21.098 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:20 smithi171 conmon[51620]: debug 2022-01-31T22:45:20.818+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:20.818981+0000) 2022-01-31T22:45:21.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:21 smithi167 conmon[54076]: debug 2022-01-31T22:45:21.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:21.015438+0000) 2022-01-31T22:45:21.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:21 smithi171 conmon[41853]: debug 2022-01-31T22:45:21.463+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:21.464707+0000) 2022-01-31T22:45:21.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:21 smithi171 conmon[51620]: debug 2022-01-31T22:45:21.382+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:21.383697+0000) 2022-01-31T22:45:21.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:21 smithi171 conmon[46715]: debug 2022-01-31T22:45:21.514+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:21.514891+0000) 2022-01-31T22:45:22.004 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:21 smithi167 conmon[49112]: debug 2022-01-31T22:45:21.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:21.848477+0000) 2022-01-31T22:45:22.005 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:21 smithi167 conmon[60316]: debug 2022-01-31T22:45:21.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:21.775368+0000) 2022-01-31T22:45:22.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:22 smithi167 conmon[54076]: debug 2022-01-31T22:45:22.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:22.015565+0000) 2022-01-31T22:45:22.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:22 smithi171 conmon[41853]: debug 2022-01-31T22:45:22.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:22.464858+0000) 2022-01-31T22:45:22.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:22 smithi171 conmon[51620]: debug 2022-01-31T22:45:22.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:22.383850+0000) 2022-01-31T22:45:22.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:22 smithi171 conmon[46715]: debug 2022-01-31T22:45:22.514+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:22.515025+0000) 2022-01-31T22:45:23.059 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:22 smithi167 conmon[49112]: debug 2022-01-31T22:45:22.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:22.848687+0000) 2022-01-31T22:45:23.059 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:23 smithi167 conmon[54076]: debug 2022-01-31T22:45:23.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:23.015741+0000) 2022-01-31T22:45:23.060 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:22 smithi167 conmon[60316]: debug 2022-01-31T22:45:22.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:22.775510+0000) 2022-01-31T22:45:23.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:23 smithi171 conmon[41853]: debug 2022-01-31T22:45:23.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:23.464994+0000) 2022-01-31T22:45:23.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:23 smithi171 conmon[51620]: debug 2022-01-31T22:45:23.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:23.384057+0000) 2022-01-31T22:45:23.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:23 smithi171 conmon[46715]: debug 2022-01-31T22:45:23.514+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:23.515201+0000) 2022-01-31T22:45:24.059 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:23 smithi167 conmon[49112]: debug 2022-01-31T22:45:23.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:23.848828+0000) 2022-01-31T22:45:24.060 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:24 smithi167 conmon[54076]: debug 2022-01-31T22:45:24.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:24.015864+0000) 2022-01-31T22:45:24.060 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:23 smithi167 conmon[60316]: debug 2022-01-31T22:45:23.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:23.775620+0000) 2022-01-31T22:45:24.244 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:23 smithi171 conmon[35325]: debug 2022-01-31T22:45:23.904+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:45:24.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:24 smithi171 conmon[41853]: debug 2022-01-31T22:45:24.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:24.465183+0000) 2022-01-31T22:45:24.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:24 smithi171 conmon[51620]: debug 2022-01-31T22:45:24.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:24.384259+0000) 2022-01-31T22:45:24.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:24 smithi171 conmon[46715]: debug 2022-01-31T22:45:24.514+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:24.515382+0000) 2022-01-31T22:45:25.059 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:25 smithi167 conmon[54076]: debug 2022-01-31T22:45:25.014+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.016035+0000) 2022-01-31T22:45:25.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:24 smithi167 conmon[49112]: debug 2022-01-31T22:45:24.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:24.849024+0000) 2022-01-31T22:45:25.060 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:24 smithi167 conmon[60316]: debug 2022-01-31T22:45:24.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:24.775774+0000) 2022-01-31T22:45:25.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[51620]: debug 2022-01-31T22:45:25.383+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.384474+0000) 2022-01-31T22:45:25.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[41853]: debug 2022-01-31T22:45:25.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.465306+0000) 2022-01-31T22:45:25.835 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[46715]: debug 2022-01-31T22:45:25.514+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.515555+0000) 2022-01-31T22:45:26.059 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:25 smithi167 conmon[54076]: debug 2022-01-31T22:45:25.845+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.846972+0000) 2022-01-31T22:45:26.060 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:26 smithi167 conmon[54076]: debug 2022-01-31T22:45:26.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:26.016162+0000) 2022-01-31T22:45:26.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:25 smithi167 conmon[49112]: debug 2022-01-31T22:45:25.845+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.846736+0000) 2022-01-31T22:45:26.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:25 smithi167 conmon[49112]: debug 2022-01-31T22:45:25.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.849124+0000) 2022-01-31T22:45:26.061 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:25 smithi167 conmon[60316]: debug 2022-01-31T22:45:25.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.775958+0000) 2022-01-31T22:45:26.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:25 smithi167 conmon[60316]: debug 2022-01-31T22:45:25.845+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.846511+0000) 2022-01-31T22:45:26.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[35325]: debug 2022-01-31T22:45:25.869+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235002 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:26.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[41853]: debug 2022-01-31T22:45:25.845+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.846228+0000) 2022-01-31T22:45:26.098 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[46715]: debug 2022-01-31T22:45:25.845+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.845937+0000) 2022-01-31T22:45:26.098 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:25 smithi171 conmon[51620]: debug 2022-01-31T22:45:25.845+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:25.845828+0000) 2022-01-31T22:45:26.504 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:26 smithi171 conmon[51620]: debug 2022-01-31T22:45:26.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:26.384699+0000) 2022-01-31T22:45:26.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:26 smithi171 conmon[41853]: debug 2022-01-31T22:45:26.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:26.465494+0000) 2022-01-31T22:45:26.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:26 smithi171 conmon[46715]: debug 2022-01-31T22:45:26.514+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:26.515700+0000) 2022-01-31T22:45:27.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:26 smithi167 conmon[49112]: debug 2022-01-31T22:45:26.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:26.849265+0000) 2022-01-31T22:45:27.060 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:27 smithi167 conmon[54076]: debug 2022-01-31T22:45:27.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:27.016340+0000) 2022-01-31T22:45:27.061 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:26 smithi167 conmon[60316]: debug 2022-01-31T22:45:26.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:26.776116+0000) 2022-01-31T22:45:27.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:27 smithi171 conmon[51620]: debug 2022-01-31T22:45:27.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:27.384883+0000) 2022-01-31T22:45:27.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:27 smithi171 conmon[41853]: debug 2022-01-31T22:45:27.464+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:27.465667+0000) 2022-01-31T22:45:27.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:27 smithi171 conmon[46715]: debug 2022-01-31T22:45:27.515+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:27.515834+0000) 2022-01-31T22:45:28.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:27 smithi167 conmon[49112]: debug 2022-01-31T22:45:27.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:27.849471+0000) 2022-01-31T22:45:28.060 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:28 smithi167 conmon[54076]: debug 2022-01-31T22:45:28.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:28.016547+0000) 2022-01-31T22:45:28.061 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:27 smithi167 conmon[60316]: debug 2022-01-31T22:45:27.774+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:27.776273+0000) 2022-01-31T22:45:28.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:28 smithi171 conmon[41853]: debug 2022-01-31T22:45:28.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:28.465840+0000) 2022-01-31T22:45:28.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:28 smithi171 conmon[51620]: debug 2022-01-31T22:45:28.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:28.385078+0000) 2022-01-31T22:45:28.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:28 smithi171 conmon[46715]: debug 2022-01-31T22:45:28.515+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:28.515966+0000) 2022-01-31T22:45:29.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:28 smithi167 conmon[49112]: debug 2022-01-31T22:45:28.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:28.849677+0000) 2022-01-31T22:45:29.060 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:29 smithi167 conmon[54076]: debug 2022-01-31T22:45:29.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:29.016786+0000) 2022-01-31T22:45:29.061 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:28 smithi167 conmon[60316]: debug 2022-01-31T22:45:28.775+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:28.776474+0000) 2022-01-31T22:45:29.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:29 smithi171 conmon[41853]: debug 2022-01-31T22:45:29.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:29.466043+0000) 2022-01-31T22:45:29.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:29 smithi171 conmon[51620]: debug 2022-01-31T22:45:29.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:29.385258+0000) 2022-01-31T22:45:29.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:29 smithi171 conmon[46715]: debug 2022-01-31T22:45:29.515+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:29.516072+0000) 2022-01-31T22:45:30.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:29 smithi167 conmon[49112]: debug 2022-01-31T22:45:29.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:29.849894+0000) 2022-01-31T22:45:30.061 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:30 smithi167 conmon[54076]: debug 2022-01-31T22:45:30.016+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.016955+0000) 2022-01-31T22:45:30.061 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:29 smithi167 conmon[60316]: debug 2022-01-31T22:45:29.775+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:29.776667+0000) 2022-01-31T22:45:30.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[51620]: debug 2022-01-31T22:45:30.384+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.385474+0000) 2022-01-31T22:45:30.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[41853]: debug 2022-01-31T22:45:30.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.466244+0000) 2022-01-31T22:45:30.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[46715]: debug 2022-01-31T22:45:30.515+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.516240+0000) 2022-01-31T22:45:31.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:30 smithi167 conmon[49112]: debug 2022-01-31T22:45:30.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.850077+0000) 2022-01-31T22:45:31.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:30 smithi167 conmon[49112]: debug 2022-01-31T22:45:30.872+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.873642+0000) 2022-01-31T22:45:31.061 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:30 smithi167 conmon[54076]: debug 2022-01-31T22:45:30.871+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.873388+0000) 2022-01-31T22:45:31.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:31 smithi167 conmon[54076]: debug 2022-01-31T22:45:31.016+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:31.017191+0000) 2022-01-31T22:45:31.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:30 smithi167 conmon[60316]: debug 2022-01-31T22:45:30.776+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.776862+0000) 2022-01-31T22:45:31.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:30 smithi167 conmon[60316]: debug 2022-01-31T22:45:30.872+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.873769+0000) 2022-01-31T22:45:31.245 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[35325]: debug 2022-01-31T22:45:30.896+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235116 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:31.246 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[41853]: debug 2022-01-31T22:45:30.871+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.872406+0000) 2022-01-31T22:45:31.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[46715]: debug 2022-01-31T22:45:30.871+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.872739+0000) 2022-01-31T22:45:31.247 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:30 smithi171 conmon[51620]: debug 2022-01-31T22:45:30.872+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:30.872854+0000) 2022-01-31T22:45:31.505 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:31 smithi171 conmon[51620]: debug 2022-01-31T22:45:31.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:31.385659+0000) 2022-01-31T22:45:31.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:31 smithi171 conmon[41853]: debug 2022-01-31T22:45:31.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:31.466402+0000) 2022-01-31T22:45:31.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:31 smithi171 conmon[46715]: debug 2022-01-31T22:45:31.515+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:31.516426+0000) 2022-01-31T22:45:32.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:31 smithi167 conmon[49112]: debug 2022-01-31T22:45:31.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:31.850246+0000) 2022-01-31T22:45:32.061 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:32 smithi167 conmon[54076]: debug 2022-01-31T22:45:32.015+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:32.017367+0000) 2022-01-31T22:45:32.061 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:31 smithi167 conmon[60316]: debug 2022-01-31T22:45:31.776+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:31.777071+0000) 2022-01-31T22:45:32.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:32 smithi171 conmon[41853]: debug 2022-01-31T22:45:32.465+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:32.466560+0000) 2022-01-31T22:45:32.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:32 smithi171 conmon[51620]: debug 2022-01-31T22:45:32.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:32.385840+0000) 2022-01-31T22:45:32.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:32 smithi171 conmon[46715]: debug 2022-01-31T22:45:32.516+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:32.516567+0000) 2022-01-31T22:45:33.060 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:32 smithi167 conmon[49112]: debug 2022-01-31T22:45:32.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:32.850414+0000) 2022-01-31T22:45:33.061 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:33 smithi167 conmon[54076]: debug 2022-01-31T22:45:33.016+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:33.017578+0000) 2022-01-31T22:45:33.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:32 smithi167 conmon[60316]: debug 2022-01-31T22:45:32.775+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:32.777255+0000) 2022-01-31T22:45:33.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:33 smithi171 conmon[41853]: debug 2022-01-31T22:45:33.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:33.466718+0000) 2022-01-31T22:45:33.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:33 smithi171 conmon[51620]: debug 2022-01-31T22:45:33.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:33.386046+0000) 2022-01-31T22:45:33.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:33 smithi171 conmon[46715]: debug 2022-01-31T22:45:33.516+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:33.516791+0000) 2022-01-31T22:45:34.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:33 smithi167 conmon[49112]: debug 2022-01-31T22:45:33.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:33.850618+0000) 2022-01-31T22:45:34.061 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:34 smithi167 conmon[54076]: debug 2022-01-31T22:45:34.016+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:34.017778+0000) 2022-01-31T22:45:34.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:33 smithi167 conmon[60316]: debug 2022-01-31T22:45:33.776+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:33.777451+0000) 2022-01-31T22:45:34.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:34 smithi171 conmon[41853]: debug 2022-01-31T22:45:34.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:34.466930+0000) 2022-01-31T22:45:34.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:34 smithi171 conmon[51620]: debug 2022-01-31T22:45:34.385+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:34.386284+0000) 2022-01-31T22:45:34.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:34 smithi171 conmon[46715]: debug 2022-01-31T22:45:34.516+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:34.517012+0000) 2022-01-31T22:45:35.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:34 smithi167 conmon[49112]: debug 2022-01-31T22:45:34.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:34.850782+0000) 2022-01-31T22:45:35.061 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:35 smithi167 conmon[54076]: debug 2022-01-31T22:45:35.017+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.017969+0000) 2022-01-31T22:45:35.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:34 smithi167 conmon[60316]: debug 2022-01-31T22:45:34.776+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:34.777688+0000) 2022-01-31T22:45:35.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[41853]: debug 2022-01-31T22:45:35.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.467080+0000) 2022-01-31T22:45:35.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[51620]: debug 2022-01-31T22:45:35.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.386488+0000) 2022-01-31T22:45:35.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[46715]: debug 2022-01-31T22:45:35.516+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.517136+0000) 2022-01-31T22:45:36.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:35 smithi167 conmon[49112]: debug 2022-01-31T22:45:35.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.850991+0000) 2022-01-31T22:45:36.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:35 smithi167 conmon[49112]: debug 2022-01-31T22:45:35.898+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.899812+0000) 2022-01-31T22:45:36.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:35 smithi167 conmon[54076]: debug 2022-01-31T22:45:35.899+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.901148+0000) 2022-01-31T22:45:36.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:36 smithi167 conmon[54076]: debug 2022-01-31T22:45:36.017+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:36.018160+0000) 2022-01-31T22:45:36.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:35 smithi167 conmon[60316]: debug 2022-01-31T22:45:35.777+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.777875+0000) 2022-01-31T22:45:36.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:35 smithi167 conmon[60316]: debug 2022-01-31T22:45:35.900+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.901509+0000) 2022-01-31T22:45:36.247 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[35325]: debug 2022-01-31T22:45:35.922+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235226 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:36.247 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[41853]: debug 2022-01-31T22:45:35.899+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.900191+0000) 2022-01-31T22:45:36.248 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[46715]: debug 2022-01-31T22:45:35.898+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.899703+0000) 2022-01-31T22:45:36.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:35 smithi171 conmon[51620]: debug 2022-01-31T22:45:35.900+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:35.901233+0000) 2022-01-31T22:45:36.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:36 smithi171 conmon[41853]: debug 2022-01-31T22:45:36.466+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:36.467189+0000) 2022-01-31T22:45:36.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:36 smithi171 conmon[51620]: debug 2022-01-31T22:45:36.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:36.386688+0000) 2022-01-31T22:45:36.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:36 smithi171 conmon[46715]: debug 2022-01-31T22:45:36.516+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:36.517287+0000) 2022-01-31T22:45:37.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:36 smithi167 conmon[49112]: debug 2022-01-31T22:45:36.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:36.851171+0000) 2022-01-31T22:45:37.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:36 smithi167 conmon[60316]: debug 2022-01-31T22:45:36.777+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:36.778005+0000) 2022-01-31T22:45:37.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:37 smithi167 conmon[54076]: debug 2022-01-31T22:45:37.017+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:37.018333+0000) 2022-01-31T22:45:37.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:37 smithi171 conmon[41853]: debug 2022-01-31T22:45:37.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:37.467379+0000) 2022-01-31T22:45:37.507 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:37 smithi171 conmon[51620]: debug 2022-01-31T22:45:37.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:37.386856+0000) 2022-01-31T22:45:37.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:37 smithi171 conmon[46715]: debug 2022-01-31T22:45:37.517+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:37.517443+0000) 2022-01-31T22:45:38.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:37 smithi167 conmon[49112]: debug 2022-01-31T22:45:37.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:37.851298+0000) 2022-01-31T22:45:38.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:38 smithi167 conmon[54076]: debug 2022-01-31T22:45:38.017+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:38.018492+0000) 2022-01-31T22:45:38.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:37 smithi167 conmon[60316]: debug 2022-01-31T22:45:37.777+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:37.778147+0000) 2022-01-31T22:45:38.506 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:38 smithi171 conmon[51620]: debug 2022-01-31T22:45:38.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:38.387052+0000) 2022-01-31T22:45:38.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:38 smithi171 conmon[41853]: debug 2022-01-31T22:45:38.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:38.467549+0000) 2022-01-31T22:45:38.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:38 smithi171 conmon[46715]: debug 2022-01-31T22:45:38.516+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:38.517634+0000) 2022-01-31T22:45:39.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:38 smithi167 conmon[49112]: debug 2022-01-31T22:45:38.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:38.851418+0000) 2022-01-31T22:45:39.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:39 smithi167 conmon[54076]: debug 2022-01-31T22:45:39.017+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:39.018651+0000) 2022-01-31T22:45:39.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:38 smithi167 conmon[60316]: debug 2022-01-31T22:45:38.777+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:38.778338+0000) 2022-01-31T22:45:39.247 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:38 smithi171 conmon[35325]: debug 2022-01-31T22:45:38.905+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:45:39.585 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:39 smithi171 conmon[41853]: debug 2022-01-31T22:45:39.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:39.467701+0000) 2022-01-31T22:45:39.586 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:39 smithi171 conmon[51620]: debug 2022-01-31T22:45:39.386+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:39.387263+0000) 2022-01-31T22:45:39.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:39 smithi171 conmon[46715]: debug 2022-01-31T22:45:39.517+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:39.517787+0000) 2022-01-31T22:45:40.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:39 smithi167 conmon[49112]: debug 2022-01-31T22:45:39.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:39.851602+0000) 2022-01-31T22:45:40.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:40 smithi167 conmon[54076]: debug 2022-01-31T22:45:40.017+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.018753+0000) 2022-01-31T22:45:40.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:39 smithi167 conmon[60316]: debug 2022-01-31T22:45:39.777+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:39.778529+0000) 2022-01-31T22:45:40.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[41853]: debug 2022-01-31T22:45:40.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.467847+0000) 2022-01-31T22:45:40.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[51620]: debug 2022-01-31T22:45:40.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.387509+0000) 2022-01-31T22:45:40.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[46715]: debug 2022-01-31T22:45:40.517+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.517975+0000) 2022-01-31T22:45:41.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:40 smithi167 conmon[49112]: debug 2022-01-31T22:45:40.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.851720+0000) 2022-01-31T22:45:41.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:40 smithi167 conmon[49112]: debug 2022-01-31T22:45:40.924+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.925818+0000) 2022-01-31T22:45:41.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:40 smithi167 conmon[54076]: debug 2022-01-31T22:45:40.925+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.927215+0000) 2022-01-31T22:45:41.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:41 smithi167 conmon[54076]: debug 2022-01-31T22:45:41.018+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:41.018974+0000) 2022-01-31T22:45:41.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:40 smithi167 conmon[60316]: debug 2022-01-31T22:45:40.777+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.778647+0000) 2022-01-31T22:45:41.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:40 smithi167 conmon[60316]: debug 2022-01-31T22:45:40.925+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.927063+0000) 2022-01-31T22:45:41.247 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[35325]: debug 2022-01-31T22:45:40.949+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235341 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:41.248 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[41853]: debug 2022-01-31T22:45:40.925+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.926168+0000) 2022-01-31T22:45:41.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[46715]: debug 2022-01-31T22:45:40.925+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.926601+0000) 2022-01-31T22:45:41.249 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:40 smithi171 conmon[51620]: debug 2022-01-31T22:45:40.926+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:40.927342+0000) 2022-01-31T22:45:41.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:41 smithi171 conmon[41853]: debug 2022-01-31T22:45:41.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:41.467946+0000) 2022-01-31T22:45:41.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:41 smithi171 conmon[51620]: debug 2022-01-31T22:45:41.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:41.387697+0000) 2022-01-31T22:45:41.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:41 smithi171 conmon[46715]: debug 2022-01-31T22:45:41.517+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:41.518185+0000) 2022-01-31T22:45:42.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:41 smithi167 conmon[49112]: debug 2022-01-31T22:45:41.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:41.851893+0000) 2022-01-31T22:45:42.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:42 smithi167 conmon[54076]: debug 2022-01-31T22:45:42.018+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:42.019132+0000) 2022-01-31T22:45:42.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:41 smithi167 conmon[60316]: debug 2022-01-31T22:45:41.778+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:41.778809+0000) 2022-01-31T22:45:42.672 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:42 smithi171 conmon[41853]: debug 2022-01-31T22:45:42.467+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:42.468081+0000) 2022-01-31T22:45:42.673 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:42 smithi171 conmon[46715]: debug 2022-01-31T22:45:42.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:42.518340+0000) 2022-01-31T22:45:42.673 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:42 smithi171 conmon[51620]: debug 2022-01-31T22:45:42.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:42.387887+0000) 2022-01-31T22:45:43.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:42 smithi167 conmon[49112]: debug 2022-01-31T22:45:42.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:42.852080+0000) 2022-01-31T22:45:43.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:42 smithi167 conmon[60316]: debug 2022-01-31T22:45:42.778+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:42.778960+0000) 2022-01-31T22:45:43.064 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:43 smithi167 conmon[54076]: debug 2022-01-31T22:45:43.018+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:43.019262+0000) 2022-01-31T22:45:43.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:43 smithi171 conmon[41853]: debug 2022-01-31T22:45:43.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:43.468267+0000) 2022-01-31T22:45:43.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:43 smithi171 conmon[51620]: debug 2022-01-31T22:45:43.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:43.388032+0000) 2022-01-31T22:45:43.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:43 smithi171 conmon[46715]: debug 2022-01-31T22:45:43.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:43.518523+0000) 2022-01-31T22:45:44.062 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:43 smithi167 conmon[49112]: debug 2022-01-31T22:45:43.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:43.852243+0000) 2022-01-31T22:45:44.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:44 smithi167 conmon[54076]: debug 2022-01-31T22:45:44.018+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:44.019447+0000) 2022-01-31T22:45:44.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:43 smithi167 conmon[60316]: debug 2022-01-31T22:45:43.778+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:43.779151+0000) 2022-01-31T22:45:44.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:44 smithi171 conmon[41853]: debug 2022-01-31T22:45:44.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:44.468459+0000) 2022-01-31T22:45:44.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:44 smithi171 conmon[51620]: debug 2022-01-31T22:45:44.387+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:44.388208+0000) 2022-01-31T22:45:44.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:44 smithi171 conmon[46715]: debug 2022-01-31T22:45:44.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:44.518742+0000) 2022-01-31T22:45:45.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:45 smithi167 conmon[54076]: debug 2022-01-31T22:45:45.019+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.019670+0000) 2022-01-31T22:45:45.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:44 smithi167 conmon[49112]: debug 2022-01-31T22:45:44.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:44.852407+0000) 2022-01-31T22:45:45.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:44 smithi167 conmon[60316]: debug 2022-01-31T22:45:44.778+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:44.779365+0000) 2022-01-31T22:45:45.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[41853]: debug 2022-01-31T22:45:45.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.468630+0000) 2022-01-31T22:45:45.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[51620]: debug 2022-01-31T22:45:45.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.388399+0000) 2022-01-31T22:45:45.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[46715]: debug 2022-01-31T22:45:45.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.518918+0000) 2022-01-31T22:45:46.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:45 smithi167 conmon[49112]: debug 2022-01-31T22:45:45.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.852587+0000) 2022-01-31T22:45:46.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:45 smithi167 conmon[49112]: debug 2022-01-31T22:45:45.951+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.953034+0000) 2022-01-31T22:45:46.064 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:45 smithi167 conmon[54076]: debug 2022-01-31T22:45:45.952+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.953852+0000) 2022-01-31T22:45:46.064 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:46 smithi167 conmon[54076]: debug 2022-01-31T22:45:46.019+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:46.019825+0000) 2022-01-31T22:45:46.065 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:45 smithi167 conmon[60316]: debug 2022-01-31T22:45:45.779+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.779576+0000) 2022-01-31T22:45:46.065 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:45 smithi167 conmon[60316]: debug 2022-01-31T22:45:45.952+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.953749+0000) 2022-01-31T22:45:46.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[35325]: debug 2022-01-31T22:45:45.976+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235450 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:46.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[41853]: debug 2022-01-31T22:45:45.952+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.953241+0000) 2022-01-31T22:45:46.250 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[46715]: debug 2022-01-31T22:45:45.952+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.953071+0000) 2022-01-31T22:45:46.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:45 smithi171 conmon[51620]: debug 2022-01-31T22:45:45.952+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:45.953054+0000) 2022-01-31T22:45:46.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:46 smithi171 conmon[51620]: debug 2022-01-31T22:45:46.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:46.388584+0000) 2022-01-31T22:45:46.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:46 smithi171 conmon[41853]: debug 2022-01-31T22:45:46.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:46.468784+0000) 2022-01-31T22:45:46.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:46 smithi171 conmon[46715]: debug 2022-01-31T22:45:46.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:46.519086+0000) 2022-01-31T22:45:47.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:47 smithi167 conmon[54076]: debug 2022-01-31T22:45:47.019+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:47.019960+0000) 2022-01-31T22:45:47.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:46 smithi167 conmon[60316]: debug 2022-01-31T22:45:46.779+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:46.779723+0000) 2022-01-31T22:45:47.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:46 smithi167 conmon[49112]: debug 2022-01-31T22:45:46.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:46.852743+0000) 2022-01-31T22:45:47.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:47 smithi171 conmon[41853]: debug 2022-01-31T22:45:47.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:47.468939+0000) 2022-01-31T22:45:47.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:47 smithi171 conmon[51620]: debug 2022-01-31T22:45:47.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:47.388714+0000) 2022-01-31T22:45:47.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:47 smithi171 conmon[46715]: debug 2022-01-31T22:45:47.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:47.519234+0000) 2022-01-31T22:45:48.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:47 smithi167 conmon[49112]: debug 2022-01-31T22:45:47.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:47.852895+0000) 2022-01-31T22:45:48.064 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:48 smithi167 conmon[54076]: debug 2022-01-31T22:45:48.019+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:48.020124+0000) 2022-01-31T22:45:48.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:47 smithi167 conmon[60316]: debug 2022-01-31T22:45:47.779+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:47.779804+0000) 2022-01-31T22:45:48.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:48 smithi171 conmon[41853]: debug 2022-01-31T22:45:48.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:48.469150+0000) 2022-01-31T22:45:48.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:48 smithi171 conmon[51620]: debug 2022-01-31T22:45:48.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:48.388843+0000) 2022-01-31T22:45:48.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:48 smithi171 conmon[46715]: debug 2022-01-31T22:45:48.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:48.519436+0000) 2022-01-31T22:45:49.063 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:49 smithi167 conmon[54076]: debug 2022-01-31T22:45:49.019+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:49.020314+0000) 2022-01-31T22:45:49.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:48 smithi167 conmon[60316]: debug 2022-01-31T22:45:48.779+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:48.779985+0000) 2022-01-31T22:45:49.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:48 smithi167 conmon[49112]: debug 2022-01-31T22:45:48.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:48.853086+0000) 2022-01-31T22:45:49.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:49 smithi171 conmon[41853]: debug 2022-01-31T22:45:49.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:49.469322+0000) 2022-01-31T22:45:49.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:49 smithi171 conmon[51620]: debug 2022-01-31T22:45:49.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:49.389135+0000) 2022-01-31T22:45:49.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:49 smithi171 conmon[46715]: debug 2022-01-31T22:45:49.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:49.519624+0000) 2022-01-31T22:45:50.063 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:49 smithi167 conmon[60316]: debug 2022-01-31T22:45:49.779+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:49.780189+0000) 2022-01-31T22:45:50.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:49 smithi167 conmon[49112]: debug 2022-01-31T22:45:49.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:49.853283+0000) 2022-01-31T22:45:50.064 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:50 smithi167 conmon[54076]: debug 2022-01-31T22:45:50.020+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.020550+0000) 2022-01-31T22:45:50.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:50 smithi171 conmon[41853]: debug 2022-01-31T22:45:50.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.469479+0000) 2022-01-31T22:45:50.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:50 smithi171 conmon[51620]: debug 2022-01-31T22:45:50.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.389885+0000) 2022-01-31T22:45:50.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:50 smithi171 conmon[46715]: debug 2022-01-31T22:45:50.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.519838+0000) 2022-01-31T22:45:51.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:50 smithi167 conmon[60316]: debug 2022-01-31T22:45:50.779+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.780390+0000) 2022-01-31T22:45:51.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:50 smithi167 conmon[60316]: debug 2022-01-31T22:45:50.979+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.980361+0000) 2022-01-31T22:45:51.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:50 smithi167 conmon[49112]: debug 2022-01-31T22:45:50.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.853430+0000) 2022-01-31T22:45:51.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:50 smithi167 conmon[49112]: debug 2022-01-31T22:45:50.979+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.980306+0000) 2022-01-31T22:45:51.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:50 smithi167 conmon[54076]: debug 2022-01-31T22:45:50.980+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.981093+0000) 2022-01-31T22:45:51.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:51 smithi167 conmon[54076]: debug 2022-01-31T22:45:51.020+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:51.020714+0000) 2022-01-31T22:45:51.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:51 smithi171 conmon[35325]: debug 2022-01-31T22:45:51.003+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235564 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:51.250 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:50 smithi171 conmon[41853]: debug 2022-01-31T22:45:50.979+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.980687+0000) 2022-01-31T22:45:51.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:50 smithi171 conmon[46715]: debug 2022-01-31T22:45:50.979+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.979787+0000) 2022-01-31T22:45:51.251 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:50 smithi171 conmon[51620]: debug 2022-01-31T22:45:50.979+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:50.979833+0000) 2022-01-31T22:45:51.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:51 smithi171 conmon[41853]: debug 2022-01-31T22:45:51.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:51.469656+0000) 2022-01-31T22:45:51.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:51 smithi171 conmon[51620]: debug 2022-01-31T22:45:51.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:51.390019+0000) 2022-01-31T22:45:51.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:51 smithi171 conmon[46715]: debug 2022-01-31T22:45:51.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:51.520056+0000) 2022-01-31T22:45:52.063 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:51 smithi167 conmon[49112]: debug 2022-01-31T22:45:51.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:51.853599+0000) 2022-01-31T22:45:52.064 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:51 smithi167 conmon[60316]: debug 2022-01-31T22:45:51.780+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:51.780583+0000) 2022-01-31T22:45:52.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:52 smithi167 conmon[54076]: debug 2022-01-31T22:45:52.020+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:52.020836+0000) 2022-01-31T22:45:52.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:52 smithi171 conmon[51620]: debug 2022-01-31T22:45:52.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:52.390125+0000) 2022-01-31T22:45:52.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:52 smithi171 conmon[41853]: debug 2022-01-31T22:45:52.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:52.469823+0000) 2022-01-31T22:45:52.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:52 smithi171 conmon[46715]: debug 2022-01-31T22:45:52.520+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:52.520202+0000) 2022-01-31T22:45:53.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:52 smithi167 conmon[49112]: debug 2022-01-31T22:45:52.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:52.853713+0000) 2022-01-31T22:45:53.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:53 smithi167 conmon[54076]: debug 2022-01-31T22:45:53.020+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:53.021028+0000) 2022-01-31T22:45:53.065 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:52 smithi167 conmon[60316]: debug 2022-01-31T22:45:52.780+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:52.780699+0000) 2022-01-31T22:45:53.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:53 smithi171 conmon[41853]: debug 2022-01-31T22:45:53.468+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:53.470042+0000) 2022-01-31T22:45:53.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:53 smithi171 conmon[51620]: debug 2022-01-31T22:45:53.388+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:53.390235+0000) 2022-01-31T22:45:53.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:53 smithi171 conmon[46715]: debug 2022-01-31T22:45:53.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:53.520382+0000) 2022-01-31T22:45:54.064 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:54 smithi167 conmon[54076]: debug 2022-01-31T22:45:54.020+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:54.021213+0000) 2022-01-31T22:45:54.065 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:53 smithi167 conmon[60316]: debug 2022-01-31T22:45:53.780+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:53.780885+0000) 2022-01-31T22:45:54.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:53 smithi167 conmon[49112]: debug 2022-01-31T22:45:53.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:53.853857+0000) 2022-01-31T22:45:54.249 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:53 smithi171 conmon[35325]: debug 2022-01-31T22:45:53.905+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:45:54.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:54 smithi171 conmon[51620]: debug 2022-01-31T22:45:54.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:54.390350+0000) 2022-01-31T22:45:54.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:54 smithi171 conmon[41853]: debug 2022-01-31T22:45:54.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:54.470201+0000) 2022-01-31T22:45:54.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:54 smithi171 conmon[46715]: debug 2022-01-31T22:45:54.518+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:54.520547+0000) 2022-01-31T22:45:55.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:54 smithi167 conmon[49112]: debug 2022-01-31T22:45:54.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:54.854023+0000) 2022-01-31T22:45:55.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:55 smithi167 conmon[54076]: debug 2022-01-31T22:45:55.021+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:55.021380+0000) 2022-01-31T22:45:55.065 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:54 smithi167 conmon[60316]: debug 2022-01-31T22:45:54.780+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:54.781085+0000) 2022-01-31T22:45:55.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:55 smithi171 conmon[51620]: debug 2022-01-31T22:45:55.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:55.390467+0000) 2022-01-31T22:45:55.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:55 smithi171 conmon[41853]: debug 2022-01-31T22:45:55.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:55.470353+0000) 2022-01-31T22:45:55.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:55 smithi171 conmon[46715]: debug 2022-01-31T22:45:55.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:55.520721+0000) 2022-01-31T22:45:56.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:55 smithi167 conmon[49112]: debug 2022-01-31T22:45:55.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:55.854224+0000) 2022-01-31T22:45:56.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:56 smithi167 conmon[49112]: debug 2022-01-31T22:45:56.007+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.007751+0000) 2022-01-31T22:45:56.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:56 smithi167 conmon[54076]: debug 2022-01-31T22:45:56.006+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.006761+0000) 2022-01-31T22:45:56.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:56 smithi167 conmon[54076]: debug 2022-01-31T22:45:56.021+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.021592+0000) 2022-01-31T22:45:56.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:55 smithi167 conmon[60316]: debug 2022-01-31T22:45:55.780+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:55.781280+0000) 2022-01-31T22:45:56.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:56 smithi167 conmon[60316]: debug 2022-01-31T22:45:56.006+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.007036+0000) 2022-01-31T22:45:56.250 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[35325]: debug 2022-01-31T22:45:56.029+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235673 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:45:56.251 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[41853]: debug 2022-01-31T22:45:56.006+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.008042+0000) 2022-01-31T22:45:56.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[46715]: debug 2022-01-31T22:45:56.005+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.007044+0000) 2022-01-31T22:45:56.252 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[51620]: debug 2022-01-31T22:45:56.005+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.006883+0000) 2022-01-31T22:45:56.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[51620]: debug 2022-01-31T22:45:56.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.390613+0000) 2022-01-31T22:45:56.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[41853]: debug 2022-01-31T22:45:56.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.470518+0000) 2022-01-31T22:45:56.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:56 smithi171 conmon[46715]: debug 2022-01-31T22:45:56.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.520901+0000) 2022-01-31T22:45:57.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:56 smithi167 conmon[49112]: debug 2022-01-31T22:45:56.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.854403+0000) 2022-01-31T22:45:57.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:57 smithi167 conmon[54076]: debug 2022-01-31T22:45:57.021+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:57.021752+0000) 2022-01-31T22:45:57.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:56 smithi167 conmon[60316]: debug 2022-01-31T22:45:56.780+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:56.781433+0000) 2022-01-31T22:45:57.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:57 smithi171 conmon[51620]: debug 2022-01-31T22:45:57.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:57.390732+0000) 2022-01-31T22:45:57.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:57 smithi171 conmon[41853]: debug 2022-01-31T22:45:57.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:57.470624+0000) 2022-01-31T22:45:57.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:57 smithi171 conmon[46715]: debug 2022-01-31T22:45:57.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:57.521040+0000) 2022-01-31T22:45:58.064 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:57 smithi167 conmon[49112]: debug 2022-01-31T22:45:57.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:57.854543+0000) 2022-01-31T22:45:58.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:58 smithi167 conmon[54076]: debug 2022-01-31T22:45:58.021+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:58.021925+0000) 2022-01-31T22:45:58.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:57 smithi167 conmon[60316]: debug 2022-01-31T22:45:57.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:57.781590+0000) 2022-01-31T22:45:58.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:58 smithi171 conmon[41853]: debug 2022-01-31T22:45:58.469+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:58.470816+0000) 2022-01-31T22:45:58.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:58 smithi171 conmon[51620]: debug 2022-01-31T22:45:58.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:58.390921+0000) 2022-01-31T22:45:58.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:58 smithi171 conmon[46715]: debug 2022-01-31T22:45:58.519+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:58.521220+0000) 2022-01-31T22:45:59.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:58 smithi167 conmon[49112]: debug 2022-01-31T22:45:58.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:58.854700+0000) 2022-01-31T22:45:59.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:45:59 smithi167 conmon[54076]: debug 2022-01-31T22:45:59.021+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:59.022088+0000) 2022-01-31T22:45:59.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:58 smithi167 conmon[60316]: debug 2022-01-31T22:45:58.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:58.781799+0000) 2022-01-31T22:45:59.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:45:59 smithi171 conmon[41853]: debug 2022-01-31T22:45:59.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:59.471002+0000) 2022-01-31T22:45:59.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:45:59 smithi171 conmon[51620]: debug 2022-01-31T22:45:59.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:59.391083+0000) 2022-01-31T22:45:59.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:45:59 smithi171 conmon[46715]: debug 2022-01-31T22:45:59.520+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:59.521335+0000) 2022-01-31T22:46:00.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:45:59 smithi167 conmon[49112]: debug 2022-01-31T22:45:59.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:59.854897+0000) 2022-01-31T22:46:00.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:00 smithi167 conmon[54076]: debug 2022-01-31T22:46:00.021+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:00.022245+0000) 2022-01-31T22:46:00.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:45:59 smithi167 conmon[60316]: debug 2022-01-31T22:45:59.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:45:59.781933+0000) 2022-01-31T22:46:00.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:00 smithi171 conmon[51620]: debug 2022-01-31T22:46:00.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:00.391286+0000) 2022-01-31T22:46:00.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:00 smithi171 conmon[41853]: debug 2022-01-31T22:46:00.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:00.471173+0000) 2022-01-31T22:46:00.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:00 smithi171 conmon[46715]: debug 2022-01-31T22:46:00.520+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:00.521465+0000) 2022-01-31T22:46:01.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:00 smithi167 conmon[49112]: debug 2022-01-31T22:46:00.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:00.855127+0000) 2022-01-31T22:46:01.023 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:01 smithi167 conmon[54076]: debug 2022-01-31T22:46:01.022+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.022442+0000) 2022-01-31T22:46:01.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:00 smithi167 conmon[60316]: debug 2022-01-31T22:46:00.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:00.782066+0000) 2022-01-31T22:46:01.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[46715]: debug 2022-01-31T22:46:01.031+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.033258+0000) 2022-01-31T22:46:01.347 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[35325]: debug 2022-01-31T22:46:01.056+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235789 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:01.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[41853]: debug 2022-01-31T22:46:01.032+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.034619+0000) 2022-01-31T22:46:01.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[51620]: debug 2022-01-31T22:46:01.032+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.034135+0000) 2022-01-31T22:46:01.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:01 smithi167 conmon[49112]: debug 2022-01-31T22:46:01.032+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.033454+0000) 2022-01-31T22:46:01.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:01 smithi167 conmon[54076]: debug 2022-01-31T22:46:01.032+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.033219+0000) 2022-01-31T22:46:01.410 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:01 smithi167 conmon[60316]: debug 2022-01-31T22:46:01.033+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.034180+0000) 2022-01-31T22:46:01.675 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[41853]: debug 2022-01-31T22:46:01.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.471345+0000) 2022-01-31T22:46:01.675 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[46715]: debug 2022-01-31T22:46:01.520+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.521574+0000) 2022-01-31T22:46:01.676 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:01 smithi171 conmon[51620]: debug 2022-01-31T22:46:01.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.391417+0000) 2022-01-31T22:46:02.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:01 smithi167 conmon[49112]: debug 2022-01-31T22:46:01.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.855313+0000) 2022-01-31T22:46:02.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:02 smithi167 conmon[54076]: debug 2022-01-31T22:46:02.022+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:02.022605+0000) 2022-01-31T22:46:02.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:01 smithi167 conmon[60316]: debug 2022-01-31T22:46:01.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:01.782224+0000) 2022-01-31T22:46:02.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:02 smithi171 conmon[51620]: debug 2022-01-31T22:46:02.389+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:02.391523+0000) 2022-01-31T22:46:02.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:02 smithi171 conmon[41853]: debug 2022-01-31T22:46:02.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:02.471512+0000) 2022-01-31T22:46:02.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:02 smithi171 conmon[46715]: debug 2022-01-31T22:46:02.520+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:02.521806+0000) 2022-01-31T22:46:03.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:02 smithi167 conmon[49112]: debug 2022-01-31T22:46:02.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:02.855451+0000) 2022-01-31T22:46:03.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:03 smithi167 conmon[54076]: debug 2022-01-31T22:46:03.022+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:03.022775+0000) 2022-01-31T22:46:03.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:02 smithi167 conmon[60316]: debug 2022-01-31T22:46:02.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:02.782362+0000) 2022-01-31T22:46:03.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:03 smithi171 conmon[41853]: debug 2022-01-31T22:46:03.470+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:03.471732+0000) 2022-01-31T22:46:03.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:03 smithi171 conmon[51620]: debug 2022-01-31T22:46:03.390+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:03.391671+0000) 2022-01-31T22:46:03.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:03 smithi171 conmon[46715]: debug 2022-01-31T22:46:03.520+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:03.521955+0000) 2022-01-31T22:46:04.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:03 smithi167 conmon[49112]: debug 2022-01-31T22:46:03.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:03.855612+0000) 2022-01-31T22:46:04.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:04 smithi167 conmon[54076]: debug 2022-01-31T22:46:04.022+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:04.022944+0000) 2022-01-31T22:46:04.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:03 smithi167 conmon[60316]: debug 2022-01-31T22:46:03.781+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:03.782485+0000) 2022-01-31T22:46:04.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:04 smithi171 conmon[41853]: debug 2022-01-31T22:46:04.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:04.472002+0000) 2022-01-31T22:46:04.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:04 smithi171 conmon[51620]: debug 2022-01-31T22:46:04.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:04.391854+0000) 2022-01-31T22:46:04.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:04 smithi171 conmon[46715]: debug 2022-01-31T22:46:04.521+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:04.522121+0000) 2022-01-31T22:46:05.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:04 smithi167 conmon[49112]: debug 2022-01-31T22:46:04.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:04.855812+0000) 2022-01-31T22:46:05.067 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:05 smithi167 conmon[54076]: debug 2022-01-31T22:46:05.022+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:05.023082+0000) 2022-01-31T22:46:05.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:04 smithi167 conmon[60316]: debug 2022-01-31T22:46:04.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:04.782659+0000) 2022-01-31T22:46:05.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:05 smithi171 conmon[41853]: debug 2022-01-31T22:46:05.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:05.472174+0000) 2022-01-31T22:46:05.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:05 smithi171 conmon[51620]: debug 2022-01-31T22:46:05.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:05.392071+0000) 2022-01-31T22:46:05.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:05 smithi171 conmon[46715]: debug 2022-01-31T22:46:05.521+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:05.522234+0000) 2022-01-31T22:46:06.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:05 smithi167 conmon[49112]: debug 2022-01-31T22:46:05.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:05.856035+0000) 2022-01-31T22:46:06.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[54076]: debug 2022-01-31T22:46:06.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.023241+0000) 2022-01-31T22:46:06.050 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:05 smithi167 conmon[60316]: debug 2022-01-31T22:46:05.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:05.782786+0000) 2022-01-31T22:46:06.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[35325]: debug 2022-01-31T22:46:06.084+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 235898 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:06.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[41853]: debug 2022-01-31T22:46:06.060+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.061434+0000) 2022-01-31T22:46:06.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[46715]: debug 2022-01-31T22:46:06.058+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.059705+0000) 2022-01-31T22:46:06.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[51620]: debug 2022-01-31T22:46:06.061+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.061745+0000) 2022-01-31T22:46:06.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[49112]: debug 2022-01-31T22:46:06.060+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.061162+0000) 2022-01-31T22:46:06.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[60316]: debug 2022-01-31T22:46:06.060+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.061273+0000) 2022-01-31T22:46:06.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[54076]: debug 2022-01-31T22:46:06.410 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[54076]: 2022-01-31T22:46:06.060+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.060870+0000) 2022-01-31T22:46:06.676 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[41853]: debug 2022-01-31T22:46:06.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.472349+0000) 2022-01-31T22:46:06.676 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[46715]: debug 2022-01-31T22:46:06.521+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.522421+0000) 2022-01-31T22:46:06.677 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:06 smithi171 conmon[51620]: debug 2022-01-31T22:46:06.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.392282+0000) 2022-01-31T22:46:07.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[49112]: debug 2022-01-31T22:46:06.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.856190+0000) 2022-01-31T22:46:07.067 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:07 smithi167 conmon[54076]: debug 2022-01-31T22:46:07.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:07.023400+0000) 2022-01-31T22:46:07.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:06 smithi167 conmon[60316]: debug 2022-01-31T22:46:06.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:06.782948+0000) 2022-01-31T22:46:07.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:07 smithi171 conmon[41853]: debug 2022-01-31T22:46:07.471+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:07.472494+0000) 2022-01-31T22:46:07.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:07 smithi171 conmon[51620]: debug 2022-01-31T22:46:07.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:07.392411+0000) 2022-01-31T22:46:07.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:07 smithi171 conmon[46715]: debug 2022-01-31T22:46:07.521+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:07.522548+0000) 2022-01-31T22:46:08.066 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:07 smithi167 conmon[49112]: debug 2022-01-31T22:46:07.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:07.856369+0000) 2022-01-31T22:46:08.067 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:08 smithi167 conmon[54076]: debug 2022-01-31T22:46:08.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:08.023569+0000) 2022-01-31T22:46:08.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:07 smithi167 conmon[60316]: debug 2022-01-31T22:46:07.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:07.783074+0000) 2022-01-31T22:46:08.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:08 smithi171 conmon[41853]: debug 2022-01-31T22:46:08.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:08.472710+0000) 2022-01-31T22:46:08.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:08 smithi171 conmon[51620]: debug 2022-01-31T22:46:08.391+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:08.392592+0000) 2022-01-31T22:46:08.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:08 smithi171 conmon[46715]: debug 2022-01-31T22:46:08.522+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:08.522717+0000) 2022-01-31T22:46:09.066 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:09 smithi167 conmon[54076]: debug 2022-01-31T22:46:09.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:09.023779+0000) 2022-01-31T22:46:09.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:08 smithi167 conmon[60316]: debug 2022-01-31T22:46:08.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:08.783251+0000) 2022-01-31T22:46:09.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:08 smithi167 conmon[49112]: debug 2022-01-31T22:46:08.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:08.856530+0000) 2022-01-31T22:46:09.252 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:08 smithi171 conmon[35325]: debug 2022-01-31T22:46:08.906+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:46:09.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:09 smithi171 conmon[41853]: debug 2022-01-31T22:46:09.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:09.472878+0000) 2022-01-31T22:46:09.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:09 smithi171 conmon[51620]: debug 2022-01-31T22:46:09.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:09.392786+0000) 2022-01-31T22:46:09.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:09 smithi171 conmon[46715]: debug 2022-01-31T22:46:09.522+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:09.522945+0000) 2022-01-31T22:46:10.067 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:09 smithi167 conmon[60316]: debug 2022-01-31T22:46:09.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:09.783449+0000) 2022-01-31T22:46:10.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:09 smithi167 conmon[49112]: debug 2022-01-31T22:46:09.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:09.856709+0000) 2022-01-31T22:46:10.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:10 smithi167 conmon[54076]: debug 2022-01-31T22:46:10.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:10.023983+0000) 2022-01-31T22:46:10.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:10 smithi171 conmon[41853]: debug 2022-01-31T22:46:10.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:10.473032+0000) 2022-01-31T22:46:10.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:10 smithi171 conmon[51620]: debug 2022-01-31T22:46:10.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:10.392983+0000) 2022-01-31T22:46:10.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:10 smithi171 conmon[46715]: debug 2022-01-31T22:46:10.522+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:10.523146+0000) 2022-01-31T22:46:11.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:10 smithi167 conmon[49112]: debug 2022-01-31T22:46:10.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:10.856867+0000) 2022-01-31T22:46:11.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:11 smithi167 conmon[54076]: debug 2022-01-31T22:46:11.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.024156+0000) 2022-01-31T22:46:11.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:10 smithi167 conmon[60316]: debug 2022-01-31T22:46:10.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:10.783671+0000) 2022-01-31T22:46:11.346 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[35325]: debug 2022-01-31T22:46:11.111+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236012 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:11.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[46715]: debug 2022-01-31T22:46:11.086+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.086995+0000) 2022-01-31T22:46:11.348 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[41853]: debug 2022-01-31T22:46:11.087+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.088278+0000) 2022-01-31T22:46:11.348 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[51620]: debug 2022-01-31T22:46:11.087+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.088116+0000) 2022-01-31T22:46:11.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:11 smithi167 conmon[49112]: debug 2022-01-31T22:46:11.087+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.088947+0000) 2022-01-31T22:46:11.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:11 smithi167 conmon[54076]: debug 2022-01-31T22:46:11.087+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.088584+0000) 2022-01-31T22:46:11.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:11 smithi167 conmon[60316]: debug 2022-01-31T22:46:11.086+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.088146+0000) 2022-01-31T22:46:11.676 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[46715]: debug 2022-01-31T22:46:11.522+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.523326+0000) 2022-01-31T22:46:11.677 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[41853]: debug 2022-01-31T22:46:11.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.473190+0000) 2022-01-31T22:46:11.678 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:11 smithi171 conmon[51620]: debug 2022-01-31T22:46:11.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.393189+0000) 2022-01-31T22:46:12.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:11 smithi167 conmon[49112]: debug 2022-01-31T22:46:11.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.857050+0000) 2022-01-31T22:46:12.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:11 smithi167 conmon[60316]: debug 2022-01-31T22:46:11.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:11.783886+0000) 2022-01-31T22:46:12.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:12 smithi167 conmon[54076]: debug 2022-01-31T22:46:12.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:12.024317+0000) 2022-01-31T22:46:12.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:12 smithi171 conmon[51620]: debug 2022-01-31T22:46:12.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:12.393353+0000) 2022-01-31T22:46:12.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:12 smithi171 conmon[41853]: debug 2022-01-31T22:46:12.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:12.473346+0000) 2022-01-31T22:46:12.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:12 smithi171 conmon[46715]: debug 2022-01-31T22:46:12.522+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:12.523461+0000) 2022-01-31T22:46:13.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:12 smithi167 conmon[49112]: debug 2022-01-31T22:46:12.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:12.857162+0000) 2022-01-31T22:46:13.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:12 smithi167 conmon[60316]: debug 2022-01-31T22:46:12.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:12.784073+0000) 2022-01-31T22:46:13.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:13 smithi167 conmon[54076]: debug 2022-01-31T22:46:13.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:13.024487+0000) 2022-01-31T22:46:13.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:13 smithi171 conmon[41853]: debug 2022-01-31T22:46:13.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:13.473518+0000) 2022-01-31T22:46:13.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:13 smithi171 conmon[51620]: debug 2022-01-31T22:46:13.392+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:13.393552+0000) 2022-01-31T22:46:13.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:13 smithi171 conmon[46715]: debug 2022-01-31T22:46:13.523+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:13.523640+0000) 2022-01-31T22:46:14.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:13 smithi167 conmon[49112]: debug 2022-01-31T22:46:13.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:13.857362+0000) 2022-01-31T22:46:14.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:14 smithi167 conmon[54076]: debug 2022-01-31T22:46:14.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:14.024736+0000) 2022-01-31T22:46:14.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:13 smithi167 conmon[60316]: debug 2022-01-31T22:46:13.782+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:13.784300+0000) 2022-01-31T22:46:14.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:14 smithi171 conmon[41853]: debug 2022-01-31T22:46:14.472+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:14.473699+0000) 2022-01-31T22:46:14.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:14 smithi171 conmon[51620]: debug 2022-01-31T22:46:14.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:14.393740+0000) 2022-01-31T22:46:14.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:14 smithi171 conmon[46715]: debug 2022-01-31T22:46:14.523+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:14.523843+0000) 2022-01-31T22:46:15.067 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:15 smithi167 conmon[54076]: debug 2022-01-31T22:46:15.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:15.024938+0000) 2022-01-31T22:46:15.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:14 smithi167 conmon[60316]: debug 2022-01-31T22:46:14.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:14.784506+0000) 2022-01-31T22:46:15.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:14 smithi167 conmon[49112]: debug 2022-01-31T22:46:14.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:14.857533+0000) 2022-01-31T22:46:15.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:15 smithi171 conmon[41853]: debug 2022-01-31T22:46:15.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:15.473804+0000) 2022-01-31T22:46:15.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:15 smithi171 conmon[51620]: debug 2022-01-31T22:46:15.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:15.393944+0000) 2022-01-31T22:46:15.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:15 smithi171 conmon[46715]: debug 2022-01-31T22:46:15.523+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:15.524032+0000) 2022-01-31T22:46:16.067 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:16 smithi167 conmon[54076]: debug 2022-01-31T22:46:16.024+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.025127+0000) 2022-01-31T22:46:16.069 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:15 smithi167 conmon[60316]: debug 2022-01-31T22:46:15.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:15.784641+0000) 2022-01-31T22:46:16.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:15 smithi167 conmon[49112]: debug 2022-01-31T22:46:15.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:15.857723+0000) 2022-01-31T22:46:16.383 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[35325]: debug 2022-01-31T22:46:16.153+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236122 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:16.383 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[41853]: debug 2022-01-31T22:46:16.113+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.114311+0000) 2022-01-31T22:46:16.384 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[46715]: debug 2022-01-31T22:46:16.114+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.114894+0000) 2022-01-31T22:46:16.385 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[51620]: debug 2022-01-31T22:46:16.113+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.114073+0000) 2022-01-31T22:46:16.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:16 smithi167 conmon[54076]: debug 2022-01-31T22:46:16.120+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.122069+0000) 2022-01-31T22:46:16.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:16 smithi167 conmon[60316]: debug 2022-01-31T22:46:16.114+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.115510+0000) 2022-01-31T22:46:16.409 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:16 smithi167 conmon[49112]: debug 2022-01-31T22:46:16.120+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.122260+0000) 2022-01-31T22:46:16.677 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[46715]: debug 2022-01-31T22:46:16.523+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.524196+0000) 2022-01-31T22:46:16.678 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[41853]: debug 2022-01-31T22:46:16.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.473937+0000) 2022-01-31T22:46:16.678 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:16 smithi171 conmon[51620]: debug 2022-01-31T22:46:16.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.394163+0000) 2022-01-31T22:46:17.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:17 smithi167 conmon[54076]: debug 2022-01-31T22:46:17.023+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:17.025298+0000) 2022-01-31T22:46:17.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:16 smithi167 conmon[60316]: debug 2022-01-31T22:46:16.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.784811+0000) 2022-01-31T22:46:17.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:16 smithi167 conmon[49112]: debug 2022-01-31T22:46:16.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:16.857903+0000) 2022-01-31T22:46:17.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:17 smithi171 conmon[41853]: debug 2022-01-31T22:46:17.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:17.474115+0000) 2022-01-31T22:46:17.513 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:17 smithi171 conmon[51620]: debug 2022-01-31T22:46:17.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:17.394312+0000) 2022-01-31T22:46:17.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:17 smithi171 conmon[46715]: debug 2022-01-31T22:46:17.523+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:17.524342+0000) 2022-01-31T22:46:18.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:17 smithi167 conmon[49112]: debug 2022-01-31T22:46:17.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:17.858061+0000) 2022-01-31T22:46:18.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:17 smithi167 conmon[60316]: debug 2022-01-31T22:46:17.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:17.784921+0000) 2022-01-31T22:46:18.069 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:18 smithi167 conmon[54076]: debug 2022-01-31T22:46:18.024+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:18.025433+0000) 2022-01-31T22:46:18.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:18 smithi171 conmon[41853]: debug 2022-01-31T22:46:18.473+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:18.474327+0000) 2022-01-31T22:46:18.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:18 smithi171 conmon[51620]: debug 2022-01-31T22:46:18.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:18.394469+0000) 2022-01-31T22:46:18.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:18 smithi171 conmon[46715]: debug 2022-01-31T22:46:18.523+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:18.524526+0000) 2022-01-31T22:46:19.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:18 smithi167 conmon[60316]: debug 2022-01-31T22:46:18.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:18.785046+0000) 2022-01-31T22:46:19.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:18 smithi167 conmon[49112]: debug 2022-01-31T22:46:18.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:18.858261+0000) 2022-01-31T22:46:19.069 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:19 smithi167 conmon[54076]: debug 2022-01-31T22:46:19.024+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:19.025601+0000) 2022-01-31T22:46:19.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:19 smithi171 conmon[41853]: debug 2022-01-31T22:46:19.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:19.474510+0000) 2022-01-31T22:46:19.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:19 smithi171 conmon[51620]: debug 2022-01-31T22:46:19.393+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:19.394603+0000) 2022-01-31T22:46:19.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:19 smithi171 conmon[46715]: debug 2022-01-31T22:46:19.524+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:19.524747+0000) 2022-01-31T22:46:20.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:19 smithi167 conmon[49112]: debug 2022-01-31T22:46:19.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:19.858468+0000) 2022-01-31T22:46:20.069 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:20 smithi167 conmon[54076]: debug 2022-01-31T22:46:20.024+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:20.025797+0000) 2022-01-31T22:46:20.069 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:19 smithi167 conmon[60316]: debug 2022-01-31T22:46:19.783+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:19.785241+0000) 2022-01-31T22:46:20.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:20 smithi171 conmon[41853]: debug 2022-01-31T22:46:20.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:20.474703+0000) 2022-01-31T22:46:20.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:20 smithi171 conmon[51620]: debug 2022-01-31T22:46:20.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:20.394798+0000) 2022-01-31T22:46:20.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:20 smithi171 conmon[46715]: debug 2022-01-31T22:46:20.524+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:20.524944+0000) 2022-01-31T22:46:21.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:20 smithi167 conmon[60316]: debug 2022-01-31T22:46:20.784+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:20.785422+0000) 2022-01-31T22:46:21.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:20 smithi167 conmon[49112]: debug 2022-01-31T22:46:20.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:20.858727+0000) 2022-01-31T22:46:21.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:21 smithi167 conmon[54076]: debug 2022-01-31T22:46:21.025+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.025999+0000) 2022-01-31T22:46:21.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:21 smithi167 conmon[49112]: debug 2022-01-31T22:46:21.156+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.157532+0000) 2022-01-31T22:46:21.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:21 smithi167 conmon[54076]: debug 2022-01-31T22:46:21.155+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.156939+0000) 2022-01-31T22:46:21.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:21 smithi167 conmon[60316]: debug 2022-01-31T22:46:21.155+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.156814+0000) 2022-01-31T22:46:21.463 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[35325]: debug 2022-01-31T22:46:21.180+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236236 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:21.464 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[41853]: debug 2022-01-31T22:46:21.157+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.157876+0000) 2022-01-31T22:46:21.464 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[46715]: debug 2022-01-31T22:46:21.156+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.157024+0000) 2022-01-31T22:46:21.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[51620]: debug 2022-01-31T22:46:21.155+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.155814+0000) 2022-01-31T22:46:21.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[51620]: debug 2022-01-31T22:46:21.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.394941+0000) 2022-01-31T22:46:21.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[46715]: debug 2022-01-31T22:46:21.524+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.525095+0000) 2022-01-31T22:46:21.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:21 smithi171 conmon[41853]: debug 2022-01-31T22:46:21.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.474773+0000) 2022-01-31T22:46:22.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:21 smithi167 conmon[60316]: debug 2022-01-31T22:46:21.784+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.785602+0000) 2022-01-31T22:46:22.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:21 smithi167 conmon[49112]: debug 2022-01-31T22:46:21.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:21.858879+0000) 2022-01-31T22:46:22.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:22 smithi167 conmon[54076]: debug 2022-01-31T22:46:22.025+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:22.026202+0000) 2022-01-31T22:46:22.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:22 smithi171 conmon[41853]: debug 2022-01-31T22:46:22.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:22.474932+0000) 2022-01-31T22:46:22.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:22 smithi171 conmon[51620]: debug 2022-01-31T22:46:22.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:22.395103+0000) 2022-01-31T22:46:22.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:22 smithi171 conmon[46715]: debug 2022-01-31T22:46:22.524+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:22.525241+0000) 2022-01-31T22:46:23.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:23 smithi167 conmon[54076]: debug 2022-01-31T22:46:23.025+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:23.026326+0000) 2022-01-31T22:46:23.069 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:22 smithi167 conmon[60316]: debug 2022-01-31T22:46:22.784+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:22.785751+0000) 2022-01-31T22:46:23.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:22 smithi167 conmon[49112]: debug 2022-01-31T22:46:22.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:22.859028+0000) 2022-01-31T22:46:23.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:23 smithi171 conmon[41853]: debug 2022-01-31T22:46:23.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:23.475111+0000) 2022-01-31T22:46:23.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:23 smithi171 conmon[51620]: debug 2022-01-31T22:46:23.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:23.395296+0000) 2022-01-31T22:46:23.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:23 smithi171 conmon[46715]: debug 2022-01-31T22:46:23.524+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:23.525420+0000) 2022-01-31T22:46:24.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:23 smithi167 conmon[49112]: debug 2022-01-31T22:46:23.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:23.859207+0000) 2022-01-31T22:46:24.069 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:24 smithi167 conmon[54076]: debug 2022-01-31T22:46:24.025+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:24.026479+0000) 2022-01-31T22:46:24.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:23 smithi167 conmon[60316]: debug 2022-01-31T22:46:23.785+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:23.785954+0000) 2022-01-31T22:46:24.255 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:23 smithi171 conmon[35325]: debug 2022-01-31T22:46:23.907+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:46:24.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:24 smithi171 conmon[41853]: debug 2022-01-31T22:46:24.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:24.475308+0000) 2022-01-31T22:46:24.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:24 smithi171 conmon[51620]: debug 2022-01-31T22:46:24.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:24.395511+0000) 2022-01-31T22:46:24.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:24 smithi171 conmon[46715]: debug 2022-01-31T22:46:24.525+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:24.525606+0000) 2022-01-31T22:46:25.069 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:25 smithi167 conmon[54076]: debug 2022-01-31T22:46:25.025+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:25.026703+0000) 2022-01-31T22:46:25.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:24 smithi167 conmon[60316]: debug 2022-01-31T22:46:24.785+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:24.786154+0000) 2022-01-31T22:46:25.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:24 smithi167 conmon[49112]: debug 2022-01-31T22:46:24.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:24.859393+0000) 2022-01-31T22:46:25.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:25 smithi171 conmon[51620]: debug 2022-01-31T22:46:25.394+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:25.395723+0000) 2022-01-31T22:46:25.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:25 smithi171 conmon[41853]: debug 2022-01-31T22:46:25.474+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:25.475493+0000) 2022-01-31T22:46:25.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:25 smithi171 conmon[46715]: debug 2022-01-31T22:46:25.525+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:25.525807+0000) 2022-01-31T22:46:26.069 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:26 smithi167 conmon[54076]: debug 2022-01-31T22:46:26.025+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.026898+0000) 2022-01-31T22:46:26.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:25 smithi167 conmon[60316]: debug 2022-01-31T22:46:25.784+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:25.786333+0000) 2022-01-31T22:46:26.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:25 smithi167 conmon[49112]: debug 2022-01-31T22:46:25.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:25.859585+0000) 2022-01-31T22:46:26.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:26 smithi167 conmon[49112]: debug 2022-01-31T22:46:26.182+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.184138+0000) 2022-01-31T22:46:26.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:26 smithi167 conmon[54076]: debug 2022-01-31T22:46:26.181+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.183024+0000) 2022-01-31T22:46:26.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:26 smithi167 conmon[60316]: debug 2022-01-31T22:46:26.182+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.183871+0000) 2022-01-31T22:46:26.464 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[35325]: debug 2022-01-31T22:46:26.207+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236348 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:26.464 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[41853]: debug 2022-01-31T22:46:26.183+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.184102+0000) 2022-01-31T22:46:26.465 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[46715]: debug 2022-01-31T22:46:26.183+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.184521+0000) 2022-01-31T22:46:26.465 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[51620]: debug 2022-01-31T22:46:26.182+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.183165+0000) 2022-01-31T22:46:26.466 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[51620]: debug 2022-01-31T22:46:26.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.395936+0000) 2022-01-31T22:46:26.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[46715]: debug 2022-01-31T22:46:26.525+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.525982+0000) 2022-01-31T22:46:26.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:26 smithi171 conmon[41853]: debug 2022-01-31T22:46:26.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.475679+0000) 2022-01-31T22:46:27.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:26 smithi167 conmon[49112]: debug 2022-01-31T22:46:26.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.859764+0000) 2022-01-31T22:46:27.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:27 smithi167 conmon[54076]: debug 2022-01-31T22:46:27.026+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:27.027028+0000) 2022-01-31T22:46:27.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:26 smithi167 conmon[60316]: debug 2022-01-31T22:46:26.785+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:26.786538+0000) 2022-01-31T22:46:27.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:27 smithi171 conmon[41853]: debug 2022-01-31T22:46:27.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:27.475807+0000) 2022-01-31T22:46:27.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:27 smithi171 conmon[51620]: debug 2022-01-31T22:46:27.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:27.396089+0000) 2022-01-31T22:46:27.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:27 smithi171 conmon[46715]: debug 2022-01-31T22:46:27.525+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:27.526088+0000) 2022-01-31T22:46:28.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:27 smithi167 conmon[49112]: debug 2022-01-31T22:46:27.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:27.859907+0000) 2022-01-31T22:46:28.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:28 smithi167 conmon[54076]: debug 2022-01-31T22:46:28.026+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:28.027177+0000) 2022-01-31T22:46:28.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:27 smithi167 conmon[60316]: debug 2022-01-31T22:46:27.785+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:27.786658+0000) 2022-01-31T22:46:28.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:28 smithi171 conmon[41853]: debug 2022-01-31T22:46:28.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:28.476030+0000) 2022-01-31T22:46:28.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:28 smithi171 conmon[51620]: debug 2022-01-31T22:46:28.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:28.396313+0000) 2022-01-31T22:46:28.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:28 smithi171 conmon[46715]: debug 2022-01-31T22:46:28.525+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:28.526267+0000) 2022-01-31T22:46:29.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:28 smithi167 conmon[49112]: debug 2022-01-31T22:46:28.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:28.860097+0000) 2022-01-31T22:46:29.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:29 smithi167 conmon[54076]: debug 2022-01-31T22:46:29.026+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:29.027306+0000) 2022-01-31T22:46:29.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:28 smithi167 conmon[60316]: debug 2022-01-31T22:46:28.785+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:28.786816+0000) 2022-01-31T22:46:29.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:29 smithi171 conmon[51620]: debug 2022-01-31T22:46:29.395+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:29.396475+0000) 2022-01-31T22:46:29.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:29 smithi171 conmon[41853]: debug 2022-01-31T22:46:29.475+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:29.476170+0000) 2022-01-31T22:46:29.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:29 smithi171 conmon[46715]: debug 2022-01-31T22:46:29.525+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:29.526451+0000) 2022-01-31T22:46:30.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:29 smithi167 conmon[49112]: debug 2022-01-31T22:46:29.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:29.860288+0000) 2022-01-31T22:46:30.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:30 smithi167 conmon[54076]: debug 2022-01-31T22:46:30.026+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:30.027483+0000) 2022-01-31T22:46:30.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:29 smithi167 conmon[60316]: debug 2022-01-31T22:46:29.786+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:29.787017+0000) 2022-01-31T22:46:30.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:30 smithi171 conmon[41853]: debug 2022-01-31T22:46:30.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:30.476368+0000) 2022-01-31T22:46:30.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:30 smithi171 conmon[51620]: debug 2022-01-31T22:46:30.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:30.396696+0000) 2022-01-31T22:46:30.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:30 smithi171 conmon[46715]: debug 2022-01-31T22:46:30.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:30.526630+0000) 2022-01-31T22:46:31.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:30 smithi167 conmon[49112]: debug 2022-01-31T22:46:30.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:30.860481+0000) 2022-01-31T22:46:31.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:31 smithi167 conmon[54076]: debug 2022-01-31T22:46:31.027+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.027701+0000) 2022-01-31T22:46:31.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:30 smithi167 conmon[60316]: debug 2022-01-31T22:46:30.786+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:30.787205+0000) 2022-01-31T22:46:31.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:31 smithi167 conmon[49112]: debug 2022-01-31T22:46:31.208+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.210109+0000) 2022-01-31T22:46:31.409 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:31 smithi167 conmon[54076]: debug 2022-01-31T22:46:31.209+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.210618+0000) 2022-01-31T22:46:31.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:31 smithi167 conmon[60316]: debug 2022-01-31T22:46:31.209+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.210963+0000) 2022-01-31T22:46:31.465 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[35325]: debug 2022-01-31T22:46:31.234+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:31.465 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[41853]: debug 2022-01-31T22:46:31.210+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.211158+0000) 2022-01-31T22:46:31.466 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[46715]: debug 2022-01-31T22:46:31.210+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.210805+0000) 2022-01-31T22:46:31.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[51620]: debug 2022-01-31T22:46:31.209+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.210068+0000) 2022-01-31T22:46:31.467 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[51620]: debug 2022-01-31T22:46:31.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.396876+0000) 2022-01-31T22:46:31.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[46715]: debug 2022-01-31T22:46:31.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.526817+0000) 2022-01-31T22:46:31.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:31 smithi171 conmon[41853]: debug 2022-01-31T22:46:31.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.476492+0000) 2022-01-31T22:46:32.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:32 smithi167 conmon[54076]: debug 2022-01-31T22:46:32.027+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:32.027863+0000) 2022-01-31T22:46:32.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:31 smithi167 conmon[49112]: debug 2022-01-31T22:46:31.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.860686+0000) 2022-01-31T22:46:32.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:31 smithi167 conmon[60316]: debug 2022-01-31T22:46:31.786+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:31.787399+0000) 2022-01-31T22:46:32.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:32 smithi171 conmon[41853]: debug 2022-01-31T22:46:32.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:32.476625+0000) 2022-01-31T22:46:32.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:32 smithi171 conmon[51620]: debug 2022-01-31T22:46:32.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:32.397028+0000) 2022-01-31T22:46:32.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:32 smithi171 conmon[46715]: debug 2022-01-31T22:46:32.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:32.526909+0000) 2022-01-31T22:46:33.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:32 smithi167 conmon[49112]: debug 2022-01-31T22:46:32.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:32.860780+0000) 2022-01-31T22:46:33.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:33 smithi167 conmon[54076]: debug 2022-01-31T22:46:33.027+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:33.028011+0000) 2022-01-31T22:46:33.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:32 smithi167 conmon[60316]: debug 2022-01-31T22:46:32.786+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:32.787538+0000) 2022-01-31T22:46:33.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:33 smithi171 conmon[41853]: debug 2022-01-31T22:46:33.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:33.476809+0000) 2022-01-31T22:46:33.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:33 smithi171 conmon[51620]: debug 2022-01-31T22:46:33.396+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:33.397128+0000) 2022-01-31T22:46:33.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:33 smithi171 conmon[46715]: debug 2022-01-31T22:46:33.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:33.527023+0000) 2022-01-31T22:46:34.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:33 smithi167 conmon[49112]: debug 2022-01-31T22:46:33.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:33.860964+0000) 2022-01-31T22:46:34.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:34 smithi167 conmon[54076]: debug 2022-01-31T22:46:34.027+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:34.028217+0000) 2022-01-31T22:46:34.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:33 smithi167 conmon[60316]: debug 2022-01-31T22:46:33.787+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:33.787746+0000) 2022-01-31T22:46:34.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:34 smithi171 conmon[51620]: debug 2022-01-31T22:46:34.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:34.397270+0000) 2022-01-31T22:46:34.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:34 smithi171 conmon[41853]: debug 2022-01-31T22:46:34.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:34.477010+0000) 2022-01-31T22:46:34.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:34 smithi171 conmon[46715]: debug 2022-01-31T22:46:34.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:34.527176+0000) 2022-01-31T22:46:35.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:34 smithi167 conmon[49112]: debug 2022-01-31T22:46:34.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:34.861170+0000) 2022-01-31T22:46:35.072 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:35 smithi167 conmon[54076]: debug 2022-01-31T22:46:35.027+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:35.028371+0000) 2022-01-31T22:46:35.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:34 smithi167 conmon[60316]: debug 2022-01-31T22:46:34.787+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:34.787941+0000) 2022-01-31T22:46:35.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:35 smithi171 conmon[51620]: debug 2022-01-31T22:46:35.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:35.397431+0000) 2022-01-31T22:46:35.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:35 smithi171 conmon[41853]: debug 2022-01-31T22:46:35.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:35.477198+0000) 2022-01-31T22:46:35.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:35 smithi171 conmon[46715]: debug 2022-01-31T22:46:35.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:35.527318+0000) 2022-01-31T22:46:36.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:35 smithi167 conmon[49112]: debug 2022-01-31T22:46:35.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:35.861352+0000) 2022-01-31T22:46:36.072 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:36 smithi167 conmon[54076]: debug 2022-01-31T22:46:36.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.028529+0000) 2022-01-31T22:46:36.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:35 smithi167 conmon[60316]: debug 2022-01-31T22:46:35.787+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:35.788098+0000) 2022-01-31T22:46:36.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:36 smithi167 conmon[49112]: debug 2022-01-31T22:46:36.237+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.238128+0000) 2022-01-31T22:46:36.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:36 smithi167 conmon[54076]: debug 2022-01-31T22:46:36.237+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.238416+0000) 2022-01-31T22:46:36.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:36 smithi167 conmon[60316]: debug 2022-01-31T22:46:36.239+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.239680+0000) 2022-01-31T22:46:36.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[41853]: debug 2022-01-31T22:46:36.237+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.238436+0000) 2022-01-31T22:46:36.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[41853]: debug 2022-01-31T22:46:36.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.477348+0000) 2022-01-31T22:46:36.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[46715]: debug 2022-01-31T22:46:36.237+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.238316+0000) 2022-01-31T22:46:36.518 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[35325]: debug 2022-01-31T22:46:36.262+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236571 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:36.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[51620]: debug 2022-01-31T22:46:36.237+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.237801+0000) 2022-01-31T22:46:36.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[51620]: debug 2022-01-31T22:46:36.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.397621+0000) 2022-01-31T22:46:36.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:36 smithi171 conmon[46715]: debug 2022-01-31T22:46:36.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.527459+0000) 2022-01-31T22:46:37.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:36 smithi167 conmon[49112]: debug 2022-01-31T22:46:36.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.861531+0000) 2022-01-31T22:46:37.072 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:37 smithi167 conmon[54076]: debug 2022-01-31T22:46:37.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:37.028668+0000) 2022-01-31T22:46:37.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:36 smithi167 conmon[60316]: debug 2022-01-31T22:46:36.787+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:36.788259+0000) 2022-01-31T22:46:37.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:37 smithi171 conmon[41853]: debug 2022-01-31T22:46:37.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:37.477496+0000) 2022-01-31T22:46:37.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:37 smithi171 conmon[51620]: debug 2022-01-31T22:46:37.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:37.397773+0000) 2022-01-31T22:46:37.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:37 smithi171 conmon[46715]: debug 2022-01-31T22:46:37.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:37.527574+0000) 2022-01-31T22:46:38.071 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:37 smithi167 conmon[49112]: debug 2022-01-31T22:46:37.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:37.861676+0000) 2022-01-31T22:46:38.072 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:38 smithi167 conmon[54076]: debug 2022-01-31T22:46:38.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:38.028782+0000) 2022-01-31T22:46:38.073 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:37 smithi167 conmon[60316]: debug 2022-01-31T22:46:37.787+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:37.788419+0000) 2022-01-31T22:46:38.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:38 smithi171 conmon[41853]: debug 2022-01-31T22:46:38.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:38.477679+0000) 2022-01-31T22:46:38.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:38 smithi171 conmon[51620]: debug 2022-01-31T22:46:38.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:38.397955+0000) 2022-01-31T22:46:38.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:38 smithi171 conmon[46715]: debug 2022-01-31T22:46:38.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:38.527796+0000) 2022-01-31T22:46:39.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:38 smithi167 conmon[49112]: debug 2022-01-31T22:46:38.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:38.861895+0000) 2022-01-31T22:46:39.073 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:39 smithi167 conmon[54076]: debug 2022-01-31T22:46:39.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:39.028881+0000) 2022-01-31T22:46:39.074 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:38 smithi167 conmon[60316]: debug 2022-01-31T22:46:38.788+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:38.788631+0000) 2022-01-31T22:46:39.258 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:38 smithi171 conmon[35325]: debug 2022-01-31T22:46:38.908+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:46:39.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:39 smithi171 conmon[41853]: debug 2022-01-31T22:46:39.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:39.477895+0000) 2022-01-31T22:46:39.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:39 smithi171 conmon[51620]: debug 2022-01-31T22:46:39.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:39.398165+0000) 2022-01-31T22:46:39.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:39 smithi171 conmon[46715]: debug 2022-01-31T22:46:39.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:39.527988+0000) 2022-01-31T22:46:40.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:39 smithi167 conmon[49112]: debug 2022-01-31T22:46:39.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:39.862078+0000) 2022-01-31T22:46:40.072 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:40 smithi167 conmon[54076]: debug 2022-01-31T22:46:40.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:40.029047+0000) 2022-01-31T22:46:40.073 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:39 smithi167 conmon[60316]: debug 2022-01-31T22:46:39.788+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:39.788791+0000) 2022-01-31T22:46:40.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:40 smithi171 conmon[41853]: debug 2022-01-31T22:46:40.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:40.478087+0000) 2022-01-31T22:46:40.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:40 smithi171 conmon[51620]: debug 2022-01-31T22:46:40.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:40.398383+0000) 2022-01-31T22:46:40.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:40 smithi171 conmon[46715]: debug 2022-01-31T22:46:40.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:40.528181+0000) 2022-01-31T22:46:41.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:40 smithi167 conmon[49112]: debug 2022-01-31T22:46:40.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:40.862269+0000) 2022-01-31T22:46:41.073 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:41 smithi167 conmon[54076]: debug 2022-01-31T22:46:41.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.029265+0000) 2022-01-31T22:46:41.073 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:40 smithi167 conmon[60316]: debug 2022-01-31T22:46:40.788+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:40.788904+0000) 2022-01-31T22:46:41.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:41 smithi167 conmon[49112]: debug 2022-01-31T22:46:41.265+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.266131+0000) 2022-01-31T22:46:41.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:41 smithi167 conmon[54076]: debug 2022-01-31T22:46:41.265+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.265847+0000) 2022-01-31T22:46:41.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:41 smithi167 conmon[60316]: debug 2022-01-31T22:46:41.265+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.266269+0000) 2022-01-31T22:46:41.516 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[35325]: debug 2022-01-31T22:46:41.287+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236684 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:41.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[46715]: debug 2022-01-31T22:46:41.263+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.264955+0000) 2022-01-31T22:46:41.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[41853]: debug 2022-01-31T22:46:41.264+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.266522+0000) 2022-01-31T22:46:41.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[41853]: debug 2022-01-31T22:46:41.476+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.478225+0000) 2022-01-31T22:46:41.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[51620]: debug 2022-01-31T22:46:41.263+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.265392+0000) 2022-01-31T22:46:41.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[51620]: debug 2022-01-31T22:46:41.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.398581+0000) 2022-01-31T22:46:41.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:41 smithi171 conmon[46715]: debug 2022-01-31T22:46:41.526+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.528288+0000) 2022-01-31T22:46:42.072 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:41 smithi167 conmon[49112]: debug 2022-01-31T22:46:41.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.862458+0000) 2022-01-31T22:46:42.073 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:42 smithi167 conmon[54076]: debug 2022-01-31T22:46:42.028+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:42.029467+0000) 2022-01-31T22:46:42.073 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:41 smithi167 conmon[60316]: debug 2022-01-31T22:46:41.788+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:41.789112+0000) 2022-01-31T22:46:42.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:42 smithi171 conmon[41853]: debug 2022-01-31T22:46:42.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:42.478332+0000) 2022-01-31T22:46:42.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:42 smithi171 conmon[51620]: debug 2022-01-31T22:46:42.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:42.398749+0000) 2022-01-31T22:46:42.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:42 smithi171 conmon[46715]: debug 2022-01-31T22:46:42.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:42.528436+0000) 2022-01-31T22:46:43.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:42 smithi167 conmon[49112]: debug 2022-01-31T22:46:42.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:42.862613+0000) 2022-01-31T22:46:43.018 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:42 smithi167 conmon[60316]: debug 2022-01-31T22:46:42.788+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:42.789286+0000) 2022-01-31T22:46:43.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:43 smithi167 conmon[54076]: debug 2022-01-31T22:46:43.029+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:43.029608+0000) 2022-01-31T22:46:43.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:43 smithi171 conmon[41853]: debug 2022-01-31T22:46:43.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:43.478506+0000) 2022-01-31T22:46:43.517 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:43 smithi171 conmon[51620]: debug 2022-01-31T22:46:43.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:43.398959+0000) 2022-01-31T22:46:43.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:43 smithi171 conmon[46715]: debug 2022-01-31T22:46:43.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:43.528630+0000) 2022-01-31T22:46:44.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:43 smithi167 conmon[49112]: debug 2022-01-31T22:46:43.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:43.862772+0000) 2022-01-31T22:46:44.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:43 smithi167 conmon[60316]: debug 2022-01-31T22:46:43.788+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:43.789475+0000) 2022-01-31T22:46:44.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:44 smithi167 conmon[54076]: debug 2022-01-31T22:46:44.029+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:44.029737+0000) 2022-01-31T22:46:44.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:44 smithi171 conmon[41853]: debug 2022-01-31T22:46:44.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:44.478686+0000) 2022-01-31T22:46:44.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:44 smithi171 conmon[51620]: debug 2022-01-31T22:46:44.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:44.399154+0000) 2022-01-31T22:46:44.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:44 smithi171 conmon[46715]: debug 2022-01-31T22:46:44.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:44.528828+0000) 2022-01-31T22:46:45.018 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:44 smithi167 conmon[49112]: debug 2022-01-31T22:46:44.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:44.862969+0000) 2022-01-31T22:46:45.019 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:44 smithi167 conmon[60316]: debug 2022-01-31T22:46:44.789+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:44.789674+0000) 2022-01-31T22:46:45.329 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:45 smithi167 conmon[54076]: debug 2022-01-31T22:46:45.029+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:45.029929+0000) 2022-01-31T22:46:45.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:45 smithi171 conmon[41853]: debug 2022-01-31T22:46:45.477+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:45.478842+0000) 2022-01-31T22:46:45.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:45 smithi171 conmon[51620]: debug 2022-01-31T22:46:45.397+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:45.399387+0000) 2022-01-31T22:46:45.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:45 smithi171 conmon[46715]: debug 2022-01-31T22:46:45.528+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:45.529016+0000) 2022-01-31T22:46:46.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:45 smithi167 conmon[49112]: debug 2022-01-31T22:46:45.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:45.863176+0000) 2022-01-31T22:46:46.073 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:46 smithi167 conmon[54076]: debug 2022-01-31T22:46:46.029+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.030089+0000) 2022-01-31T22:46:46.074 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:45 smithi167 conmon[60316]: debug 2022-01-31T22:46:45.789+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:45.789803+0000) 2022-01-31T22:46:46.408 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:46 smithi167 conmon[49112]: debug 2022-01-31T22:46:46.292+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.293212+0000) 2022-01-31T22:46:46.408 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:46 smithi167 conmon[54076]: debug 2022-01-31T22:46:46.292+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.292596+0000) 2022-01-31T22:46:46.409 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:46 smithi167 conmon[60316]: debug 2022-01-31T22:46:46.292+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.293308+0000) 2022-01-31T22:46:46.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[46715]: debug 2022-01-31T22:46:46.289+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.291465+0000) 2022-01-31T22:46:46.518 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[35325]: debug 2022-01-31T22:46:46.315+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236811 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:46.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[41853]: debug 2022-01-31T22:46:46.291+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.292760+0000) 2022-01-31T22:46:46.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[41853]: debug 2022-01-31T22:46:46.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.479021+0000) 2022-01-31T22:46:46.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[51620]: debug 2022-01-31T22:46:46.291+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.293303+0000) 2022-01-31T22:46:46.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[51620]: debug 2022-01-31T22:46:46.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.399590+0000) 2022-01-31T22:46:46.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:46 smithi171 conmon[46715]: debug 2022-01-31T22:46:46.527+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.529167+0000) 2022-01-31T22:46:47.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:46 smithi167 conmon[49112]: debug 2022-01-31T22:46:46.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.863326+0000) 2022-01-31T22:46:47.074 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:47 smithi167 conmon[54076]: debug 2022-01-31T22:46:47.029+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:47.030272+0000) 2022-01-31T22:46:47.074 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:46 smithi167 conmon[60316]: debug 2022-01-31T22:46:46.789+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:46.789996+0000) 2022-01-31T22:46:47.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:47 smithi171 conmon[41853]: debug 2022-01-31T22:46:47.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:47.479148+0000) 2022-01-31T22:46:47.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:47 smithi171 conmon[51620]: debug 2022-01-31T22:46:47.398+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:47.399769+0000) 2022-01-31T22:46:47.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:47 smithi171 conmon[46715]: debug 2022-01-31T22:46:47.528+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:47.529285+0000) 2022-01-31T22:46:48.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:47 smithi167 conmon[49112]: debug 2022-01-31T22:46:47.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:47.863520+0000) 2022-01-31T22:46:48.074 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:48 smithi167 conmon[54076]: debug 2022-01-31T22:46:48.030+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:48.030419+0000) 2022-01-31T22:46:48.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:47 smithi167 conmon[60316]: debug 2022-01-31T22:46:47.789+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:47.790097+0000) 2022-01-31T22:46:48.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:48 smithi171 conmon[51620]: debug 2022-01-31T22:46:48.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:48.399969+0000) 2022-01-31T22:46:48.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:48 smithi171 conmon[41853]: debug 2022-01-31T22:46:48.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:48.479331+0000) 2022-01-31T22:46:48.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:48 smithi171 conmon[46715]: debug 2022-01-31T22:46:48.528+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:48.529453+0000) 2022-01-31T22:46:49.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:48 smithi167 conmon[49112]: debug 2022-01-31T22:46:48.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:48.863652+0000) 2022-01-31T22:46:49.074 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:49 smithi167 conmon[54076]: debug 2022-01-31T22:46:49.030+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:49.030582+0000) 2022-01-31T22:46:49.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:48 smithi167 conmon[60316]: debug 2022-01-31T22:46:48.789+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:48.790271+0000) 2022-01-31T22:46:49.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:49 smithi171 conmon[41853]: debug 2022-01-31T22:46:49.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:49.479460+0000) 2022-01-31T22:46:49.518 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:49 smithi171 conmon[51620]: debug 2022-01-31T22:46:49.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:49.400190+0000) 2022-01-31T22:46:49.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:49 smithi171 conmon[46715]: debug 2022-01-31T22:46:49.528+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:49.529608+0000) 2022-01-31T22:46:50.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:49 smithi167 conmon[49112]: debug 2022-01-31T22:46:49.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:49.863780+0000) 2022-01-31T22:46:50.074 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:50 smithi167 conmon[54076]: debug 2022-01-31T22:46:50.030+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:50.030765+0000) 2022-01-31T22:46:50.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:49 smithi167 conmon[60316]: debug 2022-01-31T22:46:49.789+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:49.790460+0000) 2022-01-31T22:46:50.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:50 smithi171 conmon[41853]: debug 2022-01-31T22:46:50.478+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:50.479638+0000) 2022-01-31T22:46:50.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:50 smithi171 conmon[51620]: debug 2022-01-31T22:46:50.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:50.400413+0000) 2022-01-31T22:46:50.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:50 smithi171 conmon[46715]: debug 2022-01-31T22:46:50.528+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:50.529789+0000) 2022-01-31T22:46:51.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:50 smithi167 conmon[49112]: debug 2022-01-31T22:46:50.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:50.863916+0000) 2022-01-31T22:46:51.074 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:51 smithi167 conmon[54076]: debug 2022-01-31T22:46:51.030+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.031018+0000) 2022-01-31T22:46:51.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:50 smithi167 conmon[60316]: debug 2022-01-31T22:46:50.790+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:50.790707+0000) 2022-01-31T22:46:51.345 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:51 smithi167 conmon[49112]: debug 2022-01-31T22:46:51.320+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.321039+0000) 2022-01-31T22:46:51.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:51 smithi167 conmon[54076]: debug 2022-01-31T22:46:51.319+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.319889+0000) 2022-01-31T22:46:51.346 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:51 smithi167 conmon[60316]: debug 2022-01-31T22:46:51.319+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.320046+0000) 2022-01-31T22:46:51.518 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[35325]: debug 2022-01-31T22:46:51.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 236924 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:51.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[46715]: debug 2022-01-31T22:46:51.318+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.319123+0000) 2022-01-31T22:46:51.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[41853]: debug 2022-01-31T22:46:51.319+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.319845+0000) 2022-01-31T22:46:51.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[41853]: debug 2022-01-31T22:46:51.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.479825+0000) 2022-01-31T22:46:51.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[51620]: debug 2022-01-31T22:46:51.319+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.320354+0000) 2022-01-31T22:46:51.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[51620]: debug 2022-01-31T22:46:51.399+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.400573+0000) 2022-01-31T22:46:51.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:51 smithi171 conmon[46715]: debug 2022-01-31T22:46:51.529+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.529977+0000) 2022-01-31T22:46:52.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:51 smithi167 conmon[49112]: debug 2022-01-31T22:46:51.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.864079+0000) 2022-01-31T22:46:52.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:52 smithi167 conmon[54076]: debug 2022-01-31T22:46:52.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:52.031226+0000) 2022-01-31T22:46:52.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:51 smithi167 conmon[60316]: debug 2022-01-31T22:46:51.790+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:51.790884+0000) 2022-01-31T22:46:52.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:52 smithi171 conmon[41853]: debug 2022-01-31T22:46:52.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:52.479967+0000) 2022-01-31T22:46:52.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:52 smithi171 conmon[51620]: debug 2022-01-31T22:46:52.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:52.400733+0000) 2022-01-31T22:46:52.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:52 smithi171 conmon[46715]: debug 2022-01-31T22:46:52.529+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:52.530154+0000) 2022-01-31T22:46:53.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:52 smithi167 conmon[49112]: debug 2022-01-31T22:46:52.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:52.864169+0000) 2022-01-31T22:46:53.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:53 smithi167 conmon[54076]: debug 2022-01-31T22:46:53.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:53.031415+0000) 2022-01-31T22:46:53.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:52 smithi167 conmon[60316]: debug 2022-01-31T22:46:52.790+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:52.791033+0000) 2022-01-31T22:46:53.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:53 smithi171 conmon[41853]: debug 2022-01-31T22:46:53.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:53.480186+0000) 2022-01-31T22:46:53.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:53 smithi171 conmon[51620]: debug 2022-01-31T22:46:53.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:53.400919+0000) 2022-01-31T22:46:53.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:53 smithi171 conmon[46715]: debug 2022-01-31T22:46:53.529+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:53.530305+0000) 2022-01-31T22:46:54.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:53 smithi167 conmon[49112]: debug 2022-01-31T22:46:53.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:53.864373+0000) 2022-01-31T22:46:54.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:54 smithi167 conmon[54076]: debug 2022-01-31T22:46:54.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:54.031596+0000) 2022-01-31T22:46:54.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:53 smithi167 conmon[60316]: debug 2022-01-31T22:46:53.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:53.791230+0000) 2022-01-31T22:46:54.261 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:53 smithi171 conmon[35325]: debug 2022-01-31T22:46:53.908+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:46:54.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:54 smithi171 conmon[51620]: debug 2022-01-31T22:46:54.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:54.401097+0000) 2022-01-31T22:46:54.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:54 smithi171 conmon[41853]: debug 2022-01-31T22:46:54.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:54.480340+0000) 2022-01-31T22:46:54.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:54 smithi171 conmon[46715]: debug 2022-01-31T22:46:54.529+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:54.530487+0000) 2022-01-31T22:46:55.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:54 smithi167 conmon[49112]: debug 2022-01-31T22:46:54.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:54.864556+0000) 2022-01-31T22:46:55.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:55 smithi167 conmon[54076]: debug 2022-01-31T22:46:55.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:55.031792+0000) 2022-01-31T22:46:55.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:54 smithi167 conmon[60316]: debug 2022-01-31T22:46:54.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:54.791429+0000) 2022-01-31T22:46:55.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:55 smithi171 conmon[41853]: debug 2022-01-31T22:46:55.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:55.480502+0000) 2022-01-31T22:46:55.519 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:55 smithi171 conmon[51620]: debug 2022-01-31T22:46:55.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:55.401266+0000) 2022-01-31T22:46:55.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:55 smithi171 conmon[46715]: debug 2022-01-31T22:46:55.529+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:55.530656+0000) 2022-01-31T22:46:56.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:55 smithi167 conmon[49112]: debug 2022-01-31T22:46:55.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:55.864731+0000) 2022-01-31T22:46:56.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:56 smithi167 conmon[54076]: debug 2022-01-31T22:46:56.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.031943+0000) 2022-01-31T22:46:56.076 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:55 smithi167 conmon[60316]: debug 2022-01-31T22:46:55.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:55.791639+0000) 2022-01-31T22:46:56.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[46715]: debug 2022-01-31T22:46:56.345+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.346387+0000) 2022-01-31T22:46:56.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[35325]: debug 2022-01-31T22:46:56.369+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237035 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:46:56.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[41853]: debug 2022-01-31T22:46:56.344+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.345601+0000) 2022-01-31T22:46:56.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[41853]: debug 2022-01-31T22:46:56.479+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.480644+0000) 2022-01-31T22:46:56.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[51620]: debug 2022-01-31T22:46:56.346+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.347185+0000) 2022-01-31T22:46:56.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[51620]: debug 2022-01-31T22:46:56.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.401479+0000) 2022-01-31T22:46:56.618 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:56 smithi167 conmon[49112]: debug 2022-01-31T22:46:56.346+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.346933+0000) 2022-01-31T22:46:56.619 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:56 smithi167 conmon[54076]: debug 2022-01-31T22:46:56.347+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.347668+0000) 2022-01-31T22:46:56.620 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:56 smithi167 conmon[60316]: debug 2022-01-31T22:46:56.345+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.346219+0000) 2022-01-31T22:46:56.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:56 smithi171 conmon[46715]: debug 2022-01-31T22:46:56.530+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.530784+0000) 2022-01-31T22:46:56.908 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:56 smithi167 conmon[60316]: debug 2022-01-31T22:46:56.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.791842+0000) 2022-01-31T22:46:56.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:56 smithi167 conmon[49112]: debug 2022-01-31T22:46:56.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:56.864915+0000) 2022-01-31T22:46:57.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:57 smithi167 conmon[54076]: debug 2022-01-31T22:46:57.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:57.032117+0000) 2022-01-31T22:46:57.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:57 smithi171 conmon[41853]: debug 2022-01-31T22:46:57.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:57.480789+0000) 2022-01-31T22:46:57.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:57 smithi171 conmon[51620]: debug 2022-01-31T22:46:57.400+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:57.401654+0000) 2022-01-31T22:46:57.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:57 smithi171 conmon[46715]: debug 2022-01-31T22:46:57.530+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:57.530929+0000) 2022-01-31T22:46:58.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:57 smithi167 conmon[49112]: debug 2022-01-31T22:46:57.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:57.865086+0000) 2022-01-31T22:46:58.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:58 smithi167 conmon[54076]: debug 2022-01-31T22:46:58.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:58.032272+0000) 2022-01-31T22:46:58.076 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:57 smithi167 conmon[60316]: debug 2022-01-31T22:46:57.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:57.792032+0000) 2022-01-31T22:46:58.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:58 smithi171 conmon[41853]: debug 2022-01-31T22:46:58.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:58.480978+0000) 2022-01-31T22:46:58.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:58 smithi171 conmon[51620]: debug 2022-01-31T22:46:58.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:58.401858+0000) 2022-01-31T22:46:58.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:58 smithi171 conmon[46715]: debug 2022-01-31T22:46:58.530+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:58.531075+0000) 2022-01-31T22:46:59.075 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:58 smithi167 conmon[49112]: debug 2022-01-31T22:46:58.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:58.865295+0000) 2022-01-31T22:46:59.076 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:46:59 smithi167 conmon[54076]: debug 2022-01-31T22:46:59.030+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:59.032453+0000) 2022-01-31T22:46:59.076 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:58 smithi167 conmon[60316]: debug 2022-01-31T22:46:58.790+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:58.792217+0000) 2022-01-31T22:46:59.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:46:59 smithi171 conmon[41853]: debug 2022-01-31T22:46:59.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:59.481174+0000) 2022-01-31T22:46:59.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:46:59 smithi171 conmon[51620]: debug 2022-01-31T22:46:59.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:59.402052+0000) 2022-01-31T22:46:59.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:46:59 smithi171 conmon[46715]: debug 2022-01-31T22:46:59.530+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:59.531259+0000) 2022-01-31T22:47:00.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:00 smithi167 conmon[54076]: debug 2022-01-31T22:47:00.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:00.032658+0000) 2022-01-31T22:47:00.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:46:59 smithi167 conmon[49112]: debug 2022-01-31T22:46:59.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:59.865496+0000) 2022-01-31T22:47:00.076 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:46:59 smithi167 conmon[60316]: debug 2022-01-31T22:46:59.790+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:46:59.792427+0000) 2022-01-31T22:47:00.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:00 smithi171 conmon[41853]: debug 2022-01-31T22:47:00.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:00.481316+0000) 2022-01-31T22:47:00.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:00 smithi171 conmon[51620]: debug 2022-01-31T22:47:00.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:00.402243+0000) 2022-01-31T22:47:00.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:00 smithi171 conmon[46715]: debug 2022-01-31T22:47:00.530+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:00.531439+0000) 2022-01-31T22:47:01.076 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:01 smithi167 conmon[54076]: debug 2022-01-31T22:47:01.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.032831+0000) 2022-01-31T22:47:01.400 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:00 smithi167 conmon[49112]: debug 2022-01-31T22:47:00.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:00.865695+0000) 2022-01-31T22:47:01.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:00 smithi167 conmon[60316]: debug 2022-01-31T22:47:00.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:00.792612+0000) 2022-01-31T22:47:01.520 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[35325]: debug 2022-01-31T22:47:01.395+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237148 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:01.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[46715]: debug 2022-01-31T22:47:01.371+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.371925+0000) 2022-01-31T22:47:01.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[41853]: debug 2022-01-31T22:47:01.371+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.372171+0000) 2022-01-31T22:47:01.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[41853]: debug 2022-01-31T22:47:01.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.481425+0000) 2022-01-31T22:47:01.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[51620]: debug 2022-01-31T22:47:01.372+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.373324+0000) 2022-01-31T22:47:01.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[51620]: debug 2022-01-31T22:47:01.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.402398+0000) 2022-01-31T22:47:01.619 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:01 smithi167 conmon[49112]: debug 2022-01-31T22:47:01.371+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.372576+0000) 2022-01-31T22:47:01.620 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:01 smithi167 conmon[54076]: debug 2022-01-31T22:47:01.371+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.373152+0000) 2022-01-31T22:47:01.621 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:01 smithi167 conmon[60316]: debug 2022-01-31T22:47:01.371+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.373219+0000) 2022-01-31T22:47:01.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:01 smithi171 conmon[46715]: debug 2022-01-31T22:47:01.530+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.531591+0000) 2022-01-31T22:47:01.907 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:01 smithi167 conmon[60316]: debug 2022-01-31T22:47:01.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.792798+0000) 2022-01-31T22:47:01.908 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:01 smithi167 conmon[49112]: debug 2022-01-31T22:47:01.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:01.865897+0000) 2022-01-31T22:47:02.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:02 smithi167 conmon[54076]: debug 2022-01-31T22:47:02.031+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:02.033032+0000) 2022-01-31T22:47:02.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:02 smithi171 conmon[41853]: debug 2022-01-31T22:47:02.480+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:02.481587+0000) 2022-01-31T22:47:02.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:02 smithi171 conmon[51620]: debug 2022-01-31T22:47:02.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:02.402522+0000) 2022-01-31T22:47:02.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:02 smithi171 conmon[46715]: debug 2022-01-31T22:47:02.531+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:02.531747+0000) 2022-01-31T22:47:03.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:02 smithi167 conmon[49112]: debug 2022-01-31T22:47:02.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:02.866009+0000) 2022-01-31T22:47:03.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:03 smithi167 conmon[54076]: debug 2022-01-31T22:47:03.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:03.033183+0000) 2022-01-31T22:47:03.077 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:02 smithi167 conmon[60316]: debug 2022-01-31T22:47:02.792+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:02.792984+0000) 2022-01-31T22:47:03.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:03 smithi171 conmon[41853]: debug 2022-01-31T22:47:03.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:03.481787+0000) 2022-01-31T22:47:03.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:03 smithi171 conmon[51620]: debug 2022-01-31T22:47:03.401+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:03.402702+0000) 2022-01-31T22:47:03.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:03 smithi171 conmon[46715]: debug 2022-01-31T22:47:03.531+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:03.531924+0000) 2022-01-31T22:47:04.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:03 smithi167 conmon[49112]: debug 2022-01-31T22:47:03.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:03.866189+0000) 2022-01-31T22:47:04.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:04 smithi167 conmon[54076]: debug 2022-01-31T22:47:04.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:04.033340+0000) 2022-01-31T22:47:04.077 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:03 smithi167 conmon[60316]: debug 2022-01-31T22:47:03.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:03.793189+0000) 2022-01-31T22:47:04.520 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:04 smithi171 conmon[51620]: debug 2022-01-31T22:47:04.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:04.402900+0000) 2022-01-31T22:47:04.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:04 smithi171 conmon[41853]: debug 2022-01-31T22:47:04.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:04.481953+0000) 2022-01-31T22:47:04.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:04 smithi171 conmon[46715]: debug 2022-01-31T22:47:04.531+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:04.532116+0000) 2022-01-31T22:47:05.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:04 smithi167 conmon[49112]: debug 2022-01-31T22:47:04.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:04.866379+0000) 2022-01-31T22:47:05.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:05 smithi167 conmon[54076]: debug 2022-01-31T22:47:05.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:05.033562+0000) 2022-01-31T22:47:05.077 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:04 smithi167 conmon[60316]: debug 2022-01-31T22:47:04.791+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:04.793407+0000) 2022-01-31T22:47:05.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:05 smithi171 conmon[41853]: debug 2022-01-31T22:47:05.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:05.482138+0000) 2022-01-31T22:47:05.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:05 smithi171 conmon[51620]: debug 2022-01-31T22:47:05.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:05.403103+0000) 2022-01-31T22:47:05.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:05 smithi171 conmon[46715]: debug 2022-01-31T22:47:05.531+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:05.532293+0000) 2022-01-31T22:47:06.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:05 smithi167 conmon[49112]: debug 2022-01-31T22:47:05.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:05.866554+0000) 2022-01-31T22:47:06.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:06 smithi167 conmon[54076]: debug 2022-01-31T22:47:06.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.033718+0000) 2022-01-31T22:47:06.078 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:05 smithi167 conmon[60316]: debug 2022-01-31T22:47:05.792+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:05.793596+0000) 2022-01-31T22:47:06.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[46715]: debug 2022-01-31T22:47:06.398+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.398795+0000) 2022-01-31T22:47:06.521 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[35325]: debug 2022-01-31T22:47:06.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237258 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:06.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[41853]: debug 2022-01-31T22:47:06.397+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.398644+0000) 2022-01-31T22:47:06.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[41853]: debug 2022-01-31T22:47:06.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.482324+0000) 2022-01-31T22:47:06.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[51620]: debug 2022-01-31T22:47:06.398+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.399312+0000) 2022-01-31T22:47:06.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[51620]: debug 2022-01-31T22:47:06.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.403268+0000) 2022-01-31T22:47:06.658 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:06 smithi167 conmon[49112]: debug 2022-01-31T22:47:06.398+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.399678+0000) 2022-01-31T22:47:06.658 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:06 smithi167 conmon[60316]: debug 2022-01-31T22:47:06.398+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.399921+0000) 2022-01-31T22:47:06.659 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:06 smithi167 conmon[54076]: debug 2022-01-31T22:47:06.397+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.398938+0000) 2022-01-31T22:47:06.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:06 smithi171 conmon[46715]: debug 2022-01-31T22:47:06.531+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.532454+0000) 2022-01-31T22:47:07.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:06 smithi167 conmon[49112]: debug 2022-01-31T22:47:06.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.866678+0000) 2022-01-31T22:47:07.022 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:06 smithi167 conmon[60316]: debug 2022-01-31T22:47:06.792+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:06.793777+0000) 2022-01-31T22:47:07.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:07 smithi167 conmon[54076]: debug 2022-01-31T22:47:07.033+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:07.033919+0000) 2022-01-31T22:47:07.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:07 smithi171 conmon[41853]: debug 2022-01-31T22:47:07.481+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:07.482449+0000) 2022-01-31T22:47:07.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:07 smithi171 conmon[51620]: debug 2022-01-31T22:47:07.402+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:07.403427+0000) 2022-01-31T22:47:07.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:07 smithi171 conmon[46715]: debug 2022-01-31T22:47:07.532+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:07.532595+0000) 2022-01-31T22:47:08.076 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:07 smithi167 conmon[49112]: debug 2022-01-31T22:47:07.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:07.866810+0000) 2022-01-31T22:47:08.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:08 smithi167 conmon[54076]: debug 2022-01-31T22:47:08.033+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:08.034070+0000) 2022-01-31T22:47:08.078 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:07 smithi167 conmon[60316]: debug 2022-01-31T22:47:07.792+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:07.793902+0000) 2022-01-31T22:47:08.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:08 smithi171 conmon[41853]: debug 2022-01-31T22:47:08.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:08.482636+0000) 2022-01-31T22:47:08.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:08 smithi171 conmon[51620]: debug 2022-01-31T22:47:08.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:08.403607+0000) 2022-01-31T22:47:08.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:08 smithi171 conmon[46715]: debug 2022-01-31T22:47:08.532+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:08.532769+0000) 2022-01-31T22:47:09.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:08 smithi167 conmon[49112]: debug 2022-01-31T22:47:08.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:08.866938+0000) 2022-01-31T22:47:09.078 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:09 smithi167 conmon[54076]: debug 2022-01-31T22:47:09.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:09.034251+0000) 2022-01-31T22:47:09.078 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:08 smithi167 conmon[60316]: debug 2022-01-31T22:47:08.792+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:08.794031+0000) 2022-01-31T22:47:09.263 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:08 smithi171 conmon[35325]: debug 2022-01-31T22:47:08.909+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:47:09.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:09 smithi171 conmon[41853]: debug 2022-01-31T22:47:09.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:09.482865+0000) 2022-01-31T22:47:09.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:09 smithi171 conmon[51620]: debug 2022-01-31T22:47:09.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:09.403796+0000) 2022-01-31T22:47:09.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:09 smithi171 conmon[46715]: debug 2022-01-31T22:47:09.532+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:09.532981+0000) 2022-01-31T22:47:10.077 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:10 smithi167 conmon[54076]: debug 2022-01-31T22:47:10.032+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:10.034427+0000) 2022-01-31T22:47:10.078 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:09 smithi167 conmon[60316]: debug 2022-01-31T22:47:09.792+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:09.794245+0000) 2022-01-31T22:47:10.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:09 smithi167 conmon[49112]: debug 2022-01-31T22:47:09.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:09.867070+0000) 2022-01-31T22:47:10.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:10 smithi171 conmon[51620]: debug 2022-01-31T22:47:10.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:10.403969+0000) 2022-01-31T22:47:10.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:10 smithi171 conmon[41853]: debug 2022-01-31T22:47:10.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:10.483047+0000) 2022-01-31T22:47:10.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:10 smithi171 conmon[46715]: debug 2022-01-31T22:47:10.532+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:10.533156+0000) 2022-01-31T22:47:11.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:10 smithi167 conmon[49112]: debug 2022-01-31T22:47:10.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:10.867255+0000) 2022-01-31T22:47:11.078 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:11 smithi167 conmon[54076]: debug 2022-01-31T22:47:11.033+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.034656+0000) 2022-01-31T22:47:11.078 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:10 smithi167 conmon[60316]: debug 2022-01-31T22:47:10.793+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:10.794378+0000) 2022-01-31T22:47:11.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[46715]: debug 2022-01-31T22:47:11.425+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.425886+0000) 2022-01-31T22:47:11.522 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[35325]: debug 2022-01-31T22:47:11.450+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237372 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:11.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[41853]: debug 2022-01-31T22:47:11.424+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.425532+0000) 2022-01-31T22:47:11.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[41853]: debug 2022-01-31T22:47:11.482+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.483231+0000) 2022-01-31T22:47:11.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[51620]: debug 2022-01-31T22:47:11.403+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.404195+0000) 2022-01-31T22:47:11.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[51620]: debug 2022-01-31T22:47:11.424+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.425368+0000) 2022-01-31T22:47:11.754 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:11 smithi167 conmon[49112]: debug 2022-01-31T22:47:11.424+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.425950+0000) 2022-01-31T22:47:11.755 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:11 smithi167 conmon[60316]: debug 2022-01-31T22:47:11.424+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.426473+0000) 2022-01-31T22:47:11.755 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:11 smithi167 conmon[54076]: debug 2022-01-31T22:47:11.424+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.426244+0000) 2022-01-31T22:47:11.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:11 smithi171 conmon[46715]: debug 2022-01-31T22:47:11.532+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.533395+0000) 2022-01-31T22:47:12.023 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:11 smithi167 conmon[49112]: debug 2022-01-31T22:47:11.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.867426+0000) 2022-01-31T22:47:12.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:11 smithi167 conmon[60316]: debug 2022-01-31T22:47:11.793+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:11.794545+0000) 2022-01-31T22:47:12.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:12 smithi167 conmon[54076]: debug 2022-01-31T22:47:12.034+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:12.034824+0000) 2022-01-31T22:47:12.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:12 smithi171 conmon[41853]: debug 2022-01-31T22:47:12.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:12.483385+0000) 2022-01-31T22:47:12.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:12 smithi171 conmon[51620]: debug 2022-01-31T22:47:12.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:12.404378+0000) 2022-01-31T22:47:12.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:12 smithi171 conmon[46715]: debug 2022-01-31T22:47:12.533+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:12.533554+0000) 2022-01-31T22:47:13.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:12 smithi167 conmon[49112]: debug 2022-01-31T22:47:12.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:12.867568+0000) 2022-01-31T22:47:13.078 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:13 smithi167 conmon[54076]: debug 2022-01-31T22:47:13.033+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:13.034954+0000) 2022-01-31T22:47:13.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:12 smithi167 conmon[60316]: debug 2022-01-31T22:47:12.794+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:12.794749+0000) 2022-01-31T22:47:13.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:13 smithi171 conmon[41853]: debug 2022-01-31T22:47:13.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:13.483537+0000) 2022-01-31T22:47:13.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:13 smithi171 conmon[51620]: debug 2022-01-31T22:47:13.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:13.404576+0000) 2022-01-31T22:47:13.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:13 smithi171 conmon[46715]: debug 2022-01-31T22:47:13.533+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:13.533725+0000) 2022-01-31T22:47:14.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:13 smithi167 conmon[49112]: debug 2022-01-31T22:47:13.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:13.867789+0000) 2022-01-31T22:47:14.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:14 smithi167 conmon[54076]: debug 2022-01-31T22:47:14.034+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:14.035154+0000) 2022-01-31T22:47:14.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:13 smithi167 conmon[60316]: debug 2022-01-31T22:47:13.794+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:13.794957+0000) 2022-01-31T22:47:14.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:14 smithi171 conmon[41853]: debug 2022-01-31T22:47:14.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:14.483735+0000) 2022-01-31T22:47:14.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:14 smithi171 conmon[51620]: debug 2022-01-31T22:47:14.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:14.404797+0000) 2022-01-31T22:47:14.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:14 smithi171 conmon[46715]: debug 2022-01-31T22:47:14.533+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:14.533958+0000) 2022-01-31T22:47:15.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:14 smithi167 conmon[49112]: debug 2022-01-31T22:47:14.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:14.867973+0000) 2022-01-31T22:47:15.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:15 smithi167 conmon[54076]: debug 2022-01-31T22:47:15.033+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:15.035348+0000) 2022-01-31T22:47:15.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:14 smithi167 conmon[60316]: debug 2022-01-31T22:47:14.793+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:14.795207+0000) 2022-01-31T22:47:15.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:15 smithi171 conmon[41853]: debug 2022-01-31T22:47:15.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:15.483928+0000) 2022-01-31T22:47:15.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:15 smithi171 conmon[51620]: debug 2022-01-31T22:47:15.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:15.404964+0000) 2022-01-31T22:47:15.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:15 smithi171 conmon[46715]: debug 2022-01-31T22:47:15.533+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:15.534120+0000) 2022-01-31T22:47:16.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:15 smithi167 conmon[49112]: debug 2022-01-31T22:47:15.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:15.868121+0000) 2022-01-31T22:47:16.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:16 smithi167 conmon[54076]: debug 2022-01-31T22:47:16.034+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.035569+0000) 2022-01-31T22:47:16.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:15 smithi167 conmon[60316]: debug 2022-01-31T22:47:15.793+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:15.795410+0000) 2022-01-31T22:47:16.522 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[35325]: debug 2022-01-31T22:47:16.482+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237484 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:16.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[46715]: debug 2022-01-31T22:47:16.452+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.453069+0000) 2022-01-31T22:47:16.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[41853]: debug 2022-01-31T22:47:16.452+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.453553+0000) 2022-01-31T22:47:16.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[41853]: debug 2022-01-31T22:47:16.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.484050+0000) 2022-01-31T22:47:16.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[51620]: debug 2022-01-31T22:47:16.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.405151+0000) 2022-01-31T22:47:16.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[51620]: debug 2022-01-31T22:47:16.452+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.453162+0000) 2022-01-31T22:47:16.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:16 smithi167 conmon[49112]: debug 2022-01-31T22:47:16.458+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.459835+0000) 2022-01-31T22:47:16.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:16 smithi167 conmon[54076]: debug 2022-01-31T22:47:16.450+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.454490+0000) 2022-01-31T22:47:16.756 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:16 smithi167 conmon[60316]: debug 2022-01-31T22:47:16.450+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.453776+0000) 2022-01-31T22:47:16.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:16 smithi171 conmon[46715]: debug 2022-01-31T22:47:16.533+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.534271+0000) 2022-01-31T22:47:17.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:16 smithi167 conmon[49112]: debug 2022-01-31T22:47:16.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.868276+0000) 2022-01-31T22:47:17.024 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:16 smithi167 conmon[60316]: debug 2022-01-31T22:47:16.794+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:16.795569+0000) 2022-01-31T22:47:17.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:17 smithi167 conmon[54076]: debug 2022-01-31T22:47:17.035+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:17.035757+0000) 2022-01-31T22:47:17.522 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:17 smithi171 conmon[51620]: debug 2022-01-31T22:47:17.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:17.405259+0000) 2022-01-31T22:47:17.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:17 smithi171 conmon[41853]: debug 2022-01-31T22:47:17.483+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:17.484217+0000) 2022-01-31T22:47:17.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:17 smithi171 conmon[46715]: debug 2022-01-31T22:47:17.534+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:17.534445+0000) 2022-01-31T22:47:18.078 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:17 smithi167 conmon[49112]: debug 2022-01-31T22:47:17.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:17.868416+0000) 2022-01-31T22:47:18.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:18 smithi167 conmon[54076]: debug 2022-01-31T22:47:18.035+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:18.035907+0000) 2022-01-31T22:47:18.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:17 smithi167 conmon[60316]: debug 2022-01-31T22:47:17.795+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:17.795683+0000) 2022-01-31T22:47:18.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:18 smithi171 conmon[41853]: debug 2022-01-31T22:47:18.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:18.484422+0000) 2022-01-31T22:47:18.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:18 smithi171 conmon[51620]: debug 2022-01-31T22:47:18.404+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:18.405453+0000) 2022-01-31T22:47:18.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:18 smithi171 conmon[46715]: debug 2022-01-31T22:47:18.534+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:18.534663+0000) 2022-01-31T22:47:19.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:18 smithi167 conmon[49112]: debug 2022-01-31T22:47:18.867+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:18.868601+0000) 2022-01-31T22:47:19.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:19 smithi167 conmon[54076]: debug 2022-01-31T22:47:19.035+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:19.036079+0000) 2022-01-31T22:47:19.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:18 smithi167 conmon[60316]: debug 2022-01-31T22:47:18.794+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:18.795815+0000) 2022-01-31T22:47:19.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:19 smithi171 conmon[41853]: debug 2022-01-31T22:47:19.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:19.484628+0000) 2022-01-31T22:47:19.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:19 smithi171 conmon[51620]: debug 2022-01-31T22:47:19.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:19.405656+0000) 2022-01-31T22:47:19.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:19 smithi171 conmon[46715]: debug 2022-01-31T22:47:19.534+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:19.534822+0000) 2022-01-31T22:47:20.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:19 smithi167 conmon[49112]: debug 2022-01-31T22:47:19.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:19.868804+0000) 2022-01-31T22:47:20.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:20 smithi167 conmon[54076]: debug 2022-01-31T22:47:20.035+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:20.036266+0000) 2022-01-31T22:47:20.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:19 smithi167 conmon[60316]: debug 2022-01-31T22:47:19.795+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:19.795976+0000) 2022-01-31T22:47:20.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:20 smithi171 conmon[41853]: debug 2022-01-31T22:47:20.484+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:20.484829+0000) 2022-01-31T22:47:20.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:20 smithi171 conmon[51620]: debug 2022-01-31T22:47:20.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:20.405858+0000) 2022-01-31T22:47:20.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:20 smithi171 conmon[46715]: debug 2022-01-31T22:47:20.534+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:20.534952+0000) 2022-01-31T22:47:21.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:20 smithi167 conmon[49112]: debug 2022-01-31T22:47:20.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:20.868931+0000) 2022-01-31T22:47:21.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:21 smithi167 conmon[54076]: debug 2022-01-31T22:47:21.035+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.036463+0000) 2022-01-31T22:47:21.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:20 smithi167 conmon[60316]: debug 2022-01-31T22:47:20.795+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:20.796177+0000) 2022-01-31T22:47:21.523 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:21 smithi171 conmon[35325]: debug 2022-01-31T22:47:21.509+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237596 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:21.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:21 smithi171 conmon[46715]: debug 2022-01-31T22:47:21.484+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.485281+0000) 2022-01-31T22:47:21.525 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:21 smithi171 conmon[41853]: debug 2022-01-31T22:47:21.486+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.487232+0000) 2022-01-31T22:47:21.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:21 smithi171 conmon[51620]: debug 2022-01-31T22:47:21.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.406058+0000) 2022-01-31T22:47:21.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:21 smithi171 conmon[51620]: debug 2022-01-31T22:47:21.485+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.486384+0000) 2022-01-31T22:47:21.755 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:21 smithi167 conmon[49112]: debug 2022-01-31T22:47:21.485+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.487039+0000) 2022-01-31T22:47:21.756 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:21 smithi167 conmon[54076]: debug 2022-01-31T22:47:21.484+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.485537+0000) 2022-01-31T22:47:21.757 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:21 smithi167 conmon[60316]: debug 2022-01-31T22:47:21.484+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.486255+0000) 2022-01-31T22:47:21.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:21 smithi171 conmon[46715]: debug 2022-01-31T22:47:21.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.535175+0000) 2022-01-31T22:47:22.024 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:21 smithi167 conmon[49112]: debug 2022-01-31T22:47:21.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.869061+0000) 2022-01-31T22:47:22.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:21 smithi167 conmon[60316]: debug 2022-01-31T22:47:21.795+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:21.796355+0000) 2022-01-31T22:47:22.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:22 smithi167 conmon[54076]: debug 2022-01-31T22:47:22.036+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:22.036650+0000) 2022-01-31T22:47:22.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:22 smithi171 conmon[51620]: debug 2022-01-31T22:47:22.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:22.406274+0000) 2022-01-31T22:47:22.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:22 smithi171 conmon[46715]: debug 2022-01-31T22:47:22.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:22.535322+0000) 2022-01-31T22:47:23.079 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:23 smithi167 conmon[54076]: debug 2022-01-31T22:47:23.036+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:23.036797+0000) 2022-01-31T22:47:23.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:22 smithi167 conmon[49112]: debug 2022-01-31T22:47:22.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:22.869219+0000) 2022-01-31T22:47:23.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:22 smithi167 conmon[60316]: debug 2022-01-31T22:47:22.796+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:22.796524+0000) 2022-01-31T22:47:23.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:23 smithi171 conmon[51620]: debug 2022-01-31T22:47:23.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:23.406449+0000) 2022-01-31T22:47:23.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:23 smithi171 conmon[46715]: debug 2022-01-31T22:47:23.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:23.535532+0000) 2022-01-31T22:47:24.079 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:23 smithi167 conmon[49112]: debug 2022-01-31T22:47:23.868+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:23.869403+0000) 2022-01-31T22:47:24.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:24 smithi167 conmon[54076]: debug 2022-01-31T22:47:24.036+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:24.037021+0000) 2022-01-31T22:47:24.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:23 smithi167 conmon[60316]: debug 2022-01-31T22:47:23.796+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:23.796735+0000) 2022-01-31T22:47:24.267 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:23 smithi171 conmon[35325]: debug 2022-01-31T22:47:23.910+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:47:24.523 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:24 smithi171 conmon[51620]: debug 2022-01-31T22:47:24.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:24.406586+0000) 2022-01-31T22:47:24.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:24 smithi171 conmon[46715]: debug 2022-01-31T22:47:24.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:24.535722+0000) 2022-01-31T22:47:25.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:24 smithi167 conmon[49112]: debug 2022-01-31T22:47:24.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:24.869587+0000) 2022-01-31T22:47:25.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:25 smithi167 conmon[54076]: debug 2022-01-31T22:47:25.036+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:25.037189+0000) 2022-01-31T22:47:25.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:24 smithi167 conmon[60316]: debug 2022-01-31T22:47:24.796+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:24.796904+0000) 2022-01-31T22:47:25.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:25 smithi171 conmon[51620]: debug 2022-01-31T22:47:25.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:25.406794+0000) 2022-01-31T22:47:25.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:25 smithi171 conmon[46715]: debug 2022-01-31T22:47:25.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:25.535938+0000) 2022-01-31T22:47:26.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:25 smithi167 conmon[49112]: debug 2022-01-31T22:47:25.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:25.869776+0000) 2022-01-31T22:47:26.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:26 smithi167 conmon[54076]: debug 2022-01-31T22:47:26.036+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.037373+0000) 2022-01-31T22:47:26.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:25 smithi167 conmon[60316]: debug 2022-01-31T22:47:25.796+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:25.797093+0000) 2022-01-31T22:47:26.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:26 smithi171 conmon[46715]: debug 2022-01-31T22:47:26.510+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.512065+0000) 2022-01-31T22:47:26.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:26 smithi171 conmon[51620]: debug 2022-01-31T22:47:26.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.406984+0000) 2022-01-31T22:47:26.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:26 smithi171 conmon[51620]: debug 2022-01-31T22:47:26.511+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.513324+0000) 2022-01-31T22:47:26.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:26 smithi171 conmon[41853]: debug 2022-01-31T22:47:26.510+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.512461+0000) 2022-01-31T22:47:26.756 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:26 smithi167 conmon[49112]: debug 2022-01-31T22:47:26.512+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.513449+0000) 2022-01-31T22:47:26.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:26 smithi167 conmon[54076]: debug 2022-01-31T22:47:26.512+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.512664+0000) 2022-01-31T22:47:26.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:26 smithi167 conmon[60316]: debug 2022-01-31T22:47:26.511+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.512450+0000) 2022-01-31T22:47:26.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:26 smithi171 conmon[35325]: debug 2022-01-31T22:47:26.534+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237708 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:26.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:26 smithi171 conmon[46715]: debug 2022-01-31T22:47:26.534+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.536101+0000) 2022-01-31T22:47:27.025 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:26 smithi167 conmon[60316]: debug 2022-01-31T22:47:26.796+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.797323+0000) 2022-01-31T22:47:27.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:26 smithi167 conmon[49112]: debug 2022-01-31T22:47:26.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:26.869954+0000) 2022-01-31T22:47:27.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:27 smithi167 conmon[54076]: debug 2022-01-31T22:47:27.037+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:27.037580+0000) 2022-01-31T22:47:27.524 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:27 smithi171 conmon[51620]: debug 2022-01-31T22:47:27.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:27.407159+0000) 2022-01-31T22:47:27.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:27 smithi171 conmon[46715]: debug 2022-01-31T22:47:27.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:27.536280+0000) 2022-01-31T22:47:28.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:27 smithi167 conmon[49112]: debug 2022-01-31T22:47:27.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:27.870095+0000) 2022-01-31T22:47:28.081 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:28 smithi167 conmon[54076]: debug 2022-01-31T22:47:28.037+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:28.037735+0000) 2022-01-31T22:47:28.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:27 smithi167 conmon[60316]: debug 2022-01-31T22:47:27.797+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:27.797466+0000) 2022-01-31T22:47:28.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:28 smithi171 conmon[41853]: debug 2022-01-31T22:47:28.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:28.235249+0000) 2022-01-31T22:47:28.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:28 smithi171 conmon[51620]: debug 2022-01-31T22:47:28.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:28.407288+0000) 2022-01-31T22:47:28.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:28 smithi171 conmon[46715]: debug 2022-01-31T22:47:28.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:28.536442+0000) 2022-01-31T22:47:29.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:29 smithi167 conmon[54076]: debug 2022-01-31T22:47:29.037+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:29.037920+0000) 2022-01-31T22:47:29.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:28 smithi167 conmon[60316]: debug 2022-01-31T22:47:28.797+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:28.797625+0000) 2022-01-31T22:47:29.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:28 smithi167 conmon[49112]: debug 2022-01-31T22:47:28.869+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:28.870277+0000) 2022-01-31T22:47:29.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:29 smithi171 conmon[41853]: debug 2022-01-31T22:47:29.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:29.235434+0000) 2022-01-31T22:47:29.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:29 smithi171 conmon[51620]: debug 2022-01-31T22:47:29.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:29.407477+0000) 2022-01-31T22:47:29.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:29 smithi171 conmon[46715]: debug 2022-01-31T22:47:29.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:29.536616+0000) 2022-01-31T22:47:30.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:30 smithi167 conmon[54076]: debug 2022-01-31T22:47:30.037+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:30.038131+0000) 2022-01-31T22:47:30.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:29 smithi167 conmon[60316]: debug 2022-01-31T22:47:29.797+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:29.797800+0000) 2022-01-31T22:47:30.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:29 smithi167 conmon[49112]: debug 2022-01-31T22:47:29.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:29.870462+0000) 2022-01-31T22:47:30.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:30 smithi171 conmon[41853]: debug 2022-01-31T22:47:30.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:30.235636+0000) 2022-01-31T22:47:30.525 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:30 smithi171 conmon[51620]: debug 2022-01-31T22:47:30.405+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:30.407679+0000) 2022-01-31T22:47:30.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:30 smithi171 conmon[46715]: debug 2022-01-31T22:47:30.535+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:30.536780+0000) 2022-01-31T22:47:31.080 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:31 smithi167 conmon[54076]: debug 2022-01-31T22:47:31.037+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.038321+0000) 2022-01-31T22:47:31.081 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:30 smithi167 conmon[60316]: debug 2022-01-31T22:47:30.797+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:30.797982+0000) 2022-01-31T22:47:31.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:30 smithi167 conmon[49112]: debug 2022-01-31T22:47:30.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:30.870684+0000) 2022-01-31T22:47:31.526 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:31 smithi171 conmon[51620]: debug 2022-01-31T22:47:31.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.407873+0000) 2022-01-31T22:47:31.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:31 smithi171 conmon[41853]: debug 2022-01-31T22:47:31.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.235830+0000) 2022-01-31T22:47:31.786 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:31 smithi167 conmon[49112]: debug 2022-01-31T22:47:31.538+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.539048+0000) 2022-01-31T22:47:31.787 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:31 smithi167 conmon[54076]: debug 2022-01-31T22:47:31.537+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.538189+0000) 2022-01-31T22:47:31.787 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:31 smithi167 conmon[60316]: debug 2022-01-31T22:47:31.538+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.538526+0000) 2022-01-31T22:47:31.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:31 smithi171 conmon[35325]: debug 2022-01-31T22:47:31.561+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237822 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:31.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:31 smithi171 conmon[46715]: debug 2022-01-31T22:47:31.538+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.539780+0000) 2022-01-31T22:47:31.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:31 smithi171 conmon[51620]: debug 2022-01-31T22:47:31.536+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.538324+0000) 2022-01-31T22:47:31.847 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:31 smithi171 conmon[41853]: debug 2022-01-31T22:47:31.537+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.538855+0000) 2022-01-31T22:47:32.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:31 smithi167 conmon[49112]: debug 2022-01-31T22:47:31.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.870871+0000) 2022-01-31T22:47:32.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:32 smithi167 conmon[54076]: debug 2022-01-31T22:47:32.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:32.038530+0000) 2022-01-31T22:47:32.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:31 smithi167 conmon[60316]: debug 2022-01-31T22:47:31.797+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:31.798188+0000) 2022-01-31T22:47:32.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:32 smithi171 conmon[51620]: debug 2022-01-31T22:47:32.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:32.408058+0000) 2022-01-31T22:47:32.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:32 smithi171 conmon[41853]: debug 2022-01-31T22:47:32.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:32.236012+0000) 2022-01-31T22:47:33.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:32 smithi167 conmon[49112]: debug 2022-01-31T22:47:32.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:32.871022+0000) 2022-01-31T22:47:33.081 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:33 smithi167 conmon[54076]: debug 2022-01-31T22:47:33.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:33.038628+0000) 2022-01-31T22:47:33.082 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:32 smithi167 conmon[60316]: debug 2022-01-31T22:47:32.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:32.798360+0000) 2022-01-31T22:47:33.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:33 smithi171 conmon[41853]: debug 2022-01-31T22:47:33.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:33.236157+0000) 2022-01-31T22:47:33.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:33 smithi171 conmon[51620]: debug 2022-01-31T22:47:33.406+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:33.408203+0000) 2022-01-31T22:47:34.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:33 smithi167 conmon[49112]: debug 2022-01-31T22:47:33.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:33.871212+0000) 2022-01-31T22:47:34.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:34 smithi167 conmon[54076]: debug 2022-01-31T22:47:34.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:34.038735+0000) 2022-01-31T22:47:34.082 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:33 smithi167 conmon[60316]: debug 2022-01-31T22:47:33.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:33.798555+0000) 2022-01-31T22:47:34.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:34 smithi171 conmon[51620]: debug 2022-01-31T22:47:34.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:34.408392+0000) 2022-01-31T22:47:34.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:34 smithi171 conmon[41853]: debug 2022-01-31T22:47:34.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:34.236361+0000) 2022-01-31T22:47:35.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:34 smithi167 conmon[49112]: debug 2022-01-31T22:47:34.870+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:34.871388+0000) 2022-01-31T22:47:35.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:35 smithi167 conmon[54076]: debug 2022-01-31T22:47:35.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:35.038898+0000) 2022-01-31T22:47:35.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:34 smithi167 conmon[60316]: debug 2022-01-31T22:47:34.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:34.798712+0000) 2022-01-31T22:47:35.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:35 smithi171 conmon[51620]: debug 2022-01-31T22:47:35.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:35.408587+0000) 2022-01-31T22:47:35.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:35 smithi171 conmon[41853]: debug 2022-01-31T22:47:35.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:35.236523+0000) 2022-01-31T22:47:36.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:35 smithi167 conmon[49112]: debug 2022-01-31T22:47:35.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:35.871592+0000) 2022-01-31T22:47:36.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:36 smithi167 conmon[54076]: debug 2022-01-31T22:47:36.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.039073+0000) 2022-01-31T22:47:36.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:35 smithi167 conmon[60316]: debug 2022-01-31T22:47:35.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:35.798914+0000) 2022-01-31T22:47:36.553 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:36 smithi171 conmon[51620]: debug 2022-01-31T22:47:36.407+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.408799+0000) 2022-01-31T22:47:36.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:36 smithi171 conmon[41853]: debug 2022-01-31T22:47:36.235+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.236673+0000) 2022-01-31T22:47:36.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:36 smithi167 conmon[49112]: debug 2022-01-31T22:47:36.565+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.565853+0000) 2022-01-31T22:47:36.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:36 smithi167 conmon[54076]: debug 2022-01-31T22:47:36.566+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.566615+0000) 2022-01-31T22:47:36.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:36 smithi167 conmon[60316]: debug 2022-01-31T22:47:36.565+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.565943+0000) 2022-01-31T22:47:36.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:36 smithi171 conmon[35325]: debug 2022-01-31T22:47:36.589+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 237933 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:36.846 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:36 smithi171 conmon[41853]: debug 2022-01-31T22:47:36.563+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.565561+0000) 2022-01-31T22:47:36.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:36 smithi171 conmon[46715]: debug 2022-01-31T22:47:36.565+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.566876+0000) 2022-01-31T22:47:36.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:36 smithi171 conmon[51620]: debug 2022-01-31T22:47:36.563+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.565705+0000) 2022-01-31T22:47:37.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:36 smithi167 conmon[49112]: debug 2022-01-31T22:47:36.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.871746+0000) 2022-01-31T22:47:37.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:37 smithi167 conmon[54076]: debug 2022-01-31T22:47:37.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:37.039182+0000) 2022-01-31T22:47:37.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:36 smithi167 conmon[60316]: debug 2022-01-31T22:47:36.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:36.799097+0000) 2022-01-31T22:47:37.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:37 smithi171 conmon[41853]: debug 2022-01-31T22:47:37.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:37.236907+0000) 2022-01-31T22:47:37.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:37 smithi171 conmon[51620]: debug 2022-01-31T22:47:37.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:37.408955+0000) 2022-01-31T22:47:38.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:37 smithi167 conmon[49112]: debug 2022-01-31T22:47:37.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:37.871874+0000) 2022-01-31T22:47:38.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:38 smithi167 conmon[54076]: debug 2022-01-31T22:47:38.038+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:38.039366+0000) 2022-01-31T22:47:38.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:37 smithi167 conmon[60316]: debug 2022-01-31T22:47:37.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:37.799260+0000) 2022-01-31T22:47:38.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:38 smithi171 conmon[41853]: debug 2022-01-31T22:47:38.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:38.237072+0000) 2022-01-31T22:47:38.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:38 smithi171 conmon[46715]: debug 2022-01-31T22:47:38.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:38.287386+0000) 2022-01-31T22:47:38.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:38 smithi171 conmon[51620]: debug 2022-01-31T22:47:38.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:38.409096+0000) 2022-01-31T22:47:39.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:38 smithi167 conmon[49112]: debug 2022-01-31T22:47:38.871+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:38.872064+0000) 2022-01-31T22:47:39.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:39 smithi167 conmon[54076]: debug 2022-01-31T22:47:39.039+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:39.039557+0000) 2022-01-31T22:47:39.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:38 smithi167 conmon[60316]: debug 2022-01-31T22:47:38.798+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:38.799461+0000) 2022-01-31T22:47:39.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:38 smithi171 conmon[35325]: debug 2022-01-31T22:47:38.910+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:47:39.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:39 smithi171 conmon[41853]: debug 2022-01-31T22:47:39.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:39.237243+0000) 2022-01-31T22:47:39.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:39 smithi171 conmon[46715]: debug 2022-01-31T22:47:39.286+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:39.287631+0000) 2022-01-31T22:47:39.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:39 smithi171 conmon[51620]: debug 2022-01-31T22:47:39.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:39.409276+0000) 2022-01-31T22:47:40.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:39 smithi167 conmon[49112]: debug 2022-01-31T22:47:39.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:39.872263+0000) 2022-01-31T22:47:40.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:40 smithi167 conmon[54076]: debug 2022-01-31T22:47:40.039+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:40.039749+0000) 2022-01-31T22:47:40.083 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:39 smithi167 conmon[60316]: debug 2022-01-31T22:47:39.799+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:39.799647+0000) 2022-01-31T22:47:40.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:40 smithi171 conmon[41853]: debug 2022-01-31T22:47:40.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:40.237417+0000) 2022-01-31T22:47:40.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:40 smithi171 conmon[46715]: debug 2022-01-31T22:47:40.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:40.287843+0000) 2022-01-31T22:47:40.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:40 smithi171 conmon[51620]: debug 2022-01-31T22:47:40.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:40.409501+0000) 2022-01-31T22:47:41.082 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:40 smithi167 conmon[49112]: debug 2022-01-31T22:47:40.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:40.872418+0000) 2022-01-31T22:47:41.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:41 smithi167 conmon[54076]: debug 2022-01-31T22:47:41.039+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.039950+0000) 2022-01-31T22:47:41.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:40 smithi167 conmon[60316]: debug 2022-01-31T22:47:40.799+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:40.799820+0000) 2022-01-31T22:47:41.580 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[46715]: debug 2022-01-31T22:47:41.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.287977+0000) 2022-01-31T22:47:41.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[51620]: debug 2022-01-31T22:47:41.408+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.409693+0000) 2022-01-31T22:47:41.581 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[41853]: debug 2022-01-31T22:47:41.236+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.237603+0000) 2022-01-31T22:47:41.788 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:41 smithi167 conmon[49112]: debug 2022-01-31T22:47:41.592+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.592734+0000) 2022-01-31T22:47:41.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:41 smithi167 conmon[54076]: debug 2022-01-31T22:47:41.592+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.593456+0000) 2022-01-31T22:47:41.789 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:41 smithi167 conmon[60316]: debug 2022-01-31T22:47:41.592+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.592919+0000) 2022-01-31T22:47:41.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[41853]: debug 2022-01-31T22:47:41.591+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.592540+0000) 2022-01-31T22:47:41.846 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[46715]: debug 2022-01-31T22:47:41.592+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.593219+0000) 2022-01-31T22:47:41.847 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[51620]: debug 2022-01-31T22:47:41.592+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.593262+0000) 2022-01-31T22:47:41.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:41 smithi171 conmon[35325]: debug 2022-01-31T22:47:41.616+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238045 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:42.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:41 smithi167 conmon[49112]: debug 2022-01-31T22:47:41.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.872594+0000) 2022-01-31T22:47:42.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:42 smithi167 conmon[54076]: debug 2022-01-31T22:47:42.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:42.040140+0000) 2022-01-31T22:47:42.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:41 smithi167 conmon[60316]: debug 2022-01-31T22:47:41.799+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:41.800033+0000) 2022-01-31T22:47:42.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:42 smithi171 conmon[41853]: debug 2022-01-31T22:47:42.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:42.237767+0000) 2022-01-31T22:47:42.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:42 smithi171 conmon[46715]: debug 2022-01-31T22:47:42.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:42.288104+0000) 2022-01-31T22:47:42.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:42 smithi171 conmon[51620]: debug 2022-01-31T22:47:42.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:42.409911+0000) 2022-01-31T22:47:43.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:42 smithi167 conmon[49112]: debug 2022-01-31T22:47:42.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:42.872773+0000) 2022-01-31T22:47:43.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:43 smithi167 conmon[54076]: debug 2022-01-31T22:47:43.039+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:43.040293+0000) 2022-01-31T22:47:43.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:42 smithi167 conmon[60316]: debug 2022-01-31T22:47:42.799+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:42.800202+0000) 2022-01-31T22:47:43.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:43 smithi171 conmon[41853]: debug 2022-01-31T22:47:43.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:43.237924+0000) 2022-01-31T22:47:43.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:43 smithi171 conmon[46715]: debug 2022-01-31T22:47:43.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:43.288255+0000) 2022-01-31T22:47:43.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:43 smithi171 conmon[51620]: debug 2022-01-31T22:47:43.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:43.410088+0000) 2022-01-31T22:47:44.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:43 smithi167 conmon[49112]: debug 2022-01-31T22:47:43.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:43.872975+0000) 2022-01-31T22:47:44.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:44 smithi167 conmon[54076]: debug 2022-01-31T22:47:44.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:44.040476+0000) 2022-01-31T22:47:44.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:43 smithi167 conmon[60316]: debug 2022-01-31T22:47:43.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:43.800408+0000) 2022-01-31T22:47:44.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:44 smithi171 conmon[41853]: debug 2022-01-31T22:47:44.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:44.238084+0000) 2022-01-31T22:47:44.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:44 smithi171 conmon[46715]: debug 2022-01-31T22:47:44.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:44.288466+0000) 2022-01-31T22:47:44.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:44 smithi171 conmon[51620]: debug 2022-01-31T22:47:44.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:44.410250+0000) 2022-01-31T22:47:45.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:44 smithi167 conmon[49112]: debug 2022-01-31T22:47:44.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:44.873124+0000) 2022-01-31T22:47:45.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:45 smithi167 conmon[54076]: debug 2022-01-31T22:47:45.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:45.040693+0000) 2022-01-31T22:47:45.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:44 smithi167 conmon[60316]: debug 2022-01-31T22:47:44.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:44.800563+0000) 2022-01-31T22:47:45.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:45 smithi171 conmon[41853]: debug 2022-01-31T22:47:45.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:45.238268+0000) 2022-01-31T22:47:45.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:45 smithi171 conmon[46715]: debug 2022-01-31T22:47:45.287+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:45.288700+0000) 2022-01-31T22:47:45.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:45 smithi171 conmon[51620]: debug 2022-01-31T22:47:45.409+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:45.410458+0000) 2022-01-31T22:47:46.029 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:45 smithi167 conmon[49112]: debug 2022-01-31T22:47:45.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:45.873372+0000) 2022-01-31T22:47:46.029 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:45 smithi167 conmon[60316]: debug 2022-01-31T22:47:45.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:45.800800+0000) 2022-01-31T22:47:46.320 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:46 smithi167 conmon[54076]: debug 2022-01-31T22:47:46.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.040969+0000) 2022-01-31T22:47:46.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[41853]: debug 2022-01-31T22:47:46.237+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.238475+0000) 2022-01-31T22:47:46.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[46715]: debug 2022-01-31T22:47:46.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.288932+0000) 2022-01-31T22:47:46.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[51620]: debug 2022-01-31T22:47:46.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.410729+0000) 2022-01-31T22:47:46.789 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:46 smithi167 conmon[49112]: debug 2022-01-31T22:47:46.620+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.621979+0000) 2022-01-31T22:47:46.789 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:46 smithi167 conmon[54076]: debug 2022-01-31T22:47:46.618+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.620091+0000) 2022-01-31T22:47:46.790 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:46 smithi167 conmon[60316]: debug 2022-01-31T22:47:46.619+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.621012+0000) 2022-01-31T22:47:47.083 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:47 smithi167 conmon[54076]: debug 2022-01-31T22:47:47.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:47.041108+0000) 2022-01-31T22:47:47.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:46 smithi167 conmon[60316]: debug 2022-01-31T22:47:46.799+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.800947+0000) 2022-01-31T22:47:47.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:46 smithi167 conmon[49112]: debug 2022-01-31T22:47:46.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.873553+0000) 2022-01-31T22:47:47.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[35325]: debug 2022-01-31T22:47:46.644+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238156 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:47.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[41853]: debug 2022-01-31T22:47:46.618+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.619664+0000) 2022-01-31T22:47:47.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[46715]: debug 2022-01-31T22:47:46.619+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.620242+0000) 2022-01-31T22:47:47.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:46 smithi171 conmon[51620]: debug 2022-01-31T22:47:46.621+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:46.622337+0000) 2022-01-31T22:47:47.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:47 smithi171 conmon[41853]: debug 2022-01-31T22:47:47.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:47.238639+0000) 2022-01-31T22:47:47.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:47 smithi171 conmon[46715]: debug 2022-01-31T22:47:47.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:47.289022+0000) 2022-01-31T22:47:47.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:47 smithi171 conmon[51620]: debug 2022-01-31T22:47:47.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:47.410918+0000) 2022-01-31T22:47:48.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:47 smithi167 conmon[49112]: debug 2022-01-31T22:47:47.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:47.873704+0000) 2022-01-31T22:47:48.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:48 smithi167 conmon[54076]: debug 2022-01-31T22:47:48.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:48.041284+0000) 2022-01-31T22:47:48.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:47 smithi167 conmon[60316]: debug 2022-01-31T22:47:47.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:47.801097+0000) 2022-01-31T22:47:48.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:48 smithi171 conmon[41853]: debug 2022-01-31T22:47:48.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:48.238809+0000) 2022-01-31T22:47:48.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:48 smithi171 conmon[46715]: debug 2022-01-31T22:47:48.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:48.289158+0000) 2022-01-31T22:47:48.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:48 smithi171 conmon[51620]: debug 2022-01-31T22:47:48.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:48.411065+0000) 2022-01-31T22:47:49.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:48 smithi167 conmon[49112]: debug 2022-01-31T22:47:48.872+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:48.873919+0000) 2022-01-31T22:47:49.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:49 smithi167 conmon[54076]: debug 2022-01-31T22:47:49.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:49.041490+0000) 2022-01-31T22:47:49.085 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:48 smithi167 conmon[60316]: debug 2022-01-31T22:47:48.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:48.801289+0000) 2022-01-31T22:47:49.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:49 smithi171 conmon[41853]: debug 2022-01-31T22:47:49.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:49.239013+0000) 2022-01-31T22:47:49.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:49 smithi171 conmon[46715]: debug 2022-01-31T22:47:49.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:49.289365+0000) 2022-01-31T22:47:49.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:49 smithi171 conmon[51620]: debug 2022-01-31T22:47:49.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:49.411258+0000) 2022-01-31T22:47:50.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:49 smithi167 conmon[49112]: debug 2022-01-31T22:47:49.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:49.874115+0000) 2022-01-31T22:47:50.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:50 smithi167 conmon[54076]: debug 2022-01-31T22:47:50.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:50.041684+0000) 2022-01-31T22:47:50.085 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:49 smithi167 conmon[60316]: debug 2022-01-31T22:47:49.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:49.801532+0000) 2022-01-31T22:47:50.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:50 smithi171 conmon[41853]: debug 2022-01-31T22:47:50.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:50.239178+0000) 2022-01-31T22:47:50.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:50 smithi171 conmon[46715]: debug 2022-01-31T22:47:50.288+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:50.289583+0000) 2022-01-31T22:47:50.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:50 smithi171 conmon[51620]: debug 2022-01-31T22:47:50.410+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:50.411442+0000) 2022-01-31T22:47:51.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:50 smithi167 conmon[49112]: debug 2022-01-31T22:47:50.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:50.874317+0000) 2022-01-31T22:47:51.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:51 smithi167 conmon[54076]: debug 2022-01-31T22:47:51.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.041866+0000) 2022-01-31T22:47:51.085 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:50 smithi167 conmon[60316]: debug 2022-01-31T22:47:50.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:50.801750+0000) 2022-01-31T22:47:51.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[41853]: debug 2022-01-31T22:47:51.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.239370+0000) 2022-01-31T22:47:51.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[46715]: debug 2022-01-31T22:47:51.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.289786+0000) 2022-01-31T22:47:51.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[51620]: debug 2022-01-31T22:47:51.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.411652+0000) 2022-01-31T22:47:51.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:51 smithi167 conmon[49112]: debug 2022-01-31T22:47:51.646+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.648031+0000) 2022-01-31T22:47:51.790 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:51 smithi167 conmon[54076]: debug 2022-01-31T22:47:51.646+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.647716+0000) 2022-01-31T22:47:51.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:51 smithi167 conmon[60316]: debug 2022-01-31T22:47:51.645+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.647295+0000) 2022-01-31T22:47:52.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:52 smithi167 conmon[54076]: debug 2022-01-31T22:47:52.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:52.042060+0000) 2022-01-31T22:47:52.085 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:51 smithi167 conmon[60316]: debug 2022-01-31T22:47:51.801+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.801917+0000) 2022-01-31T22:47:52.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:51 smithi167 conmon[49112]: debug 2022-01-31T22:47:51.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.874480+0000) 2022-01-31T22:47:52.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[35325]: debug 2022-01-31T22:47:51.671+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238268 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:52.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[46715]: debug 2022-01-31T22:47:51.646+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.647602+0000) 2022-01-31T22:47:52.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[51620]: debug 2022-01-31T22:47:51.647+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.647968+0000) 2022-01-31T22:47:52.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:51 smithi171 conmon[41853]: debug 2022-01-31T22:47:51.647+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:51.648230+0000) 2022-01-31T22:47:52.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:52 smithi171 conmon[41853]: debug 2022-01-31T22:47:52.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:52.239546+0000) 2022-01-31T22:47:52.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:52 smithi171 conmon[46715]: debug 2022-01-31T22:47:52.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:52.289886+0000) 2022-01-31T22:47:52.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:52 smithi171 conmon[51620]: debug 2022-01-31T22:47:52.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:52.411836+0000) 2022-01-31T22:47:53.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:52 smithi167 conmon[49112]: debug 2022-01-31T22:47:52.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:52.874606+0000) 2022-01-31T22:47:53.085 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:53 smithi167 conmon[54076]: debug 2022-01-31T22:47:53.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:53.042197+0000) 2022-01-31T22:47:53.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:52 smithi167 conmon[60316]: debug 2022-01-31T22:47:52.801+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:52.802063+0000) 2022-01-31T22:47:53.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:53 smithi171 conmon[41853]: debug 2022-01-31T22:47:53.238+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:53.239712+0000) 2022-01-31T22:47:53.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:53 smithi171 conmon[46715]: debug 2022-01-31T22:47:53.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:53.289979+0000) 2022-01-31T22:47:53.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:53 smithi171 conmon[51620]: debug 2022-01-31T22:47:53.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:53.411999+0000) 2022-01-31T22:47:54.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:53 smithi167 conmon[49112]: debug 2022-01-31T22:47:53.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:53.874769+0000) 2022-01-31T22:47:54.085 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:54 smithi167 conmon[54076]: debug 2022-01-31T22:47:54.040+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:54.042414+0000) 2022-01-31T22:47:54.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:53 smithi167 conmon[60316]: debug 2022-01-31T22:47:53.801+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:53.802279+0000) 2022-01-31T22:47:54.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:53 smithi171 conmon[35325]: debug 2022-01-31T22:47:53.911+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:47:54.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:54 smithi171 conmon[41853]: debug 2022-01-31T22:47:54.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:54.239939+0000) 2022-01-31T22:47:54.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:54 smithi171 conmon[46715]: debug 2022-01-31T22:47:54.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:54.290107+0000) 2022-01-31T22:47:54.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:54 smithi171 conmon[51620]: debug 2022-01-31T22:47:54.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:54.412184+0000) 2022-01-31T22:47:55.084 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:54 smithi167 conmon[49112]: debug 2022-01-31T22:47:54.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:54.874911+0000) 2022-01-31T22:47:55.085 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:55 smithi167 conmon[54076]: debug 2022-01-31T22:47:55.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:55.042558+0000) 2022-01-31T22:47:55.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:54 smithi167 conmon[60316]: debug 2022-01-31T22:47:54.800+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:54.802418+0000) 2022-01-31T22:47:55.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:55 smithi171 conmon[41853]: debug 2022-01-31T22:47:55.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:55.240108+0000) 2022-01-31T22:47:55.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:55 smithi171 conmon[46715]: debug 2022-01-31T22:47:55.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:55.290319+0000) 2022-01-31T22:47:55.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:55 smithi171 conmon[51620]: debug 2022-01-31T22:47:55.411+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:55.412373+0000) 2022-01-31T22:47:56.084 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:56 smithi167 conmon[54076]: debug 2022-01-31T22:47:56.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.042690+0000) 2022-01-31T22:47:56.085 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:55 smithi167 conmon[60316]: debug 2022-01-31T22:47:55.801+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:55.802525+0000) 2022-01-31T22:47:56.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:55 smithi167 conmon[49112]: debug 2022-01-31T22:47:55.873+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:55.875072+0000) 2022-01-31T22:47:56.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[41853]: debug 2022-01-31T22:47:56.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.240304+0000) 2022-01-31T22:47:56.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[46715]: debug 2022-01-31T22:47:56.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.290502+0000) 2022-01-31T22:47:56.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[51620]: debug 2022-01-31T22:47:56.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.412595+0000) 2022-01-31T22:47:56.790 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:56 smithi167 conmon[49112]: debug 2022-01-31T22:47:56.673+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.675124+0000) 2022-01-31T22:47:56.791 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:56 smithi167 conmon[60316]: debug 2022-01-31T22:47:56.673+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.674966+0000) 2022-01-31T22:47:56.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:56 smithi167 conmon[54076]: debug 2022-01-31T22:47:56.674+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.675863+0000) 2022-01-31T22:47:57.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:56 smithi167 conmon[49112]: debug 2022-01-31T22:47:56.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.875229+0000) 2022-01-31T22:47:57.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:56 smithi167 conmon[60316]: debug 2022-01-31T22:47:56.801+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.802845+0000) 2022-01-31T22:47:57.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:57 smithi167 conmon[54076]: debug 2022-01-31T22:47:57.042+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:57.042880+0000) 2022-01-31T22:47:57.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[46715]: debug 2022-01-31T22:47:56.673+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.674452+0000) 2022-01-31T22:47:57.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[51620]: debug 2022-01-31T22:47:56.674+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.675529+0000) 2022-01-31T22:47:57.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[35325]: debug 2022-01-31T22:47:56.698+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238379 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:47:57.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:56 smithi171 conmon[41853]: debug 2022-01-31T22:47:56.675+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:56.675977+0000) 2022-01-31T22:47:57.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:57 smithi171 conmon[46715]: debug 2022-01-31T22:47:57.289+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:57.290704+0000) 2022-01-31T22:47:57.703 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:57 smithi171 conmon[41853]: debug 2022-01-31T22:47:57.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:57.240508+0000) 2022-01-31T22:47:57.703 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:57 smithi171 conmon[51620]: debug 2022-01-31T22:47:57.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:57.412801+0000) 2022-01-31T22:47:58.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:57 smithi167 conmon[49112]: debug 2022-01-31T22:47:57.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:57.875380+0000) 2022-01-31T22:47:58.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:58 smithi167 conmon[54076]: debug 2022-01-31T22:47:58.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:58.043053+0000) 2022-01-31T22:47:58.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:57 smithi167 conmon[60316]: debug 2022-01-31T22:47:57.802+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:57.803017+0000) 2022-01-31T22:47:58.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:58 smithi171 conmon[46715]: debug 2022-01-31T22:47:58.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:58.290836+0000) 2022-01-31T22:47:58.521 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:58 smithi171 conmon[51620]: debug 2022-01-31T22:47:58.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:58.412986+0000) 2022-01-31T22:47:58.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:58 smithi171 conmon[41853]: debug 2022-01-31T22:47:58.239+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:58.240682+0000) 2022-01-31T22:47:59.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:58 smithi167 conmon[49112]: debug 2022-01-31T22:47:58.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:58.875544+0000) 2022-01-31T22:47:59.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:47:59 smithi167 conmon[54076]: debug 2022-01-31T22:47:59.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:59.043172+0000) 2022-01-31T22:47:59.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:58 smithi167 conmon[60316]: debug 2022-01-31T22:47:58.802+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:58.803223+0000) 2022-01-31T22:47:59.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:47:59 smithi171 conmon[41853]: debug 2022-01-31T22:47:59.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:59.240898+0000) 2022-01-31T22:47:59.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:47:59 smithi171 conmon[46715]: debug 2022-01-31T22:47:59.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:59.291003+0000) 2022-01-31T22:47:59.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:47:59 smithi171 conmon[51620]: debug 2022-01-31T22:47:59.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:59.413097+0000) 2022-01-31T22:48:00.085 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:47:59 smithi167 conmon[49112]: debug 2022-01-31T22:47:59.874+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:59.875730+0000) 2022-01-31T22:48:00.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:00 smithi167 conmon[54076]: debug 2022-01-31T22:48:00.041+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:00.043310+0000) 2022-01-31T22:48:00.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:47:59 smithi167 conmon[60316]: debug 2022-01-31T22:47:59.802+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:47:59.803409+0000) 2022-01-31T22:48:00.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:00 smithi171 conmon[46715]: debug 2022-01-31T22:48:00.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:00.291123+0000) 2022-01-31T22:48:00.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:00 smithi171 conmon[41853]: debug 2022-01-31T22:48:00.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:00.241056+0000) 2022-01-31T22:48:00.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:00 smithi171 conmon[51620]: debug 2022-01-31T22:48:00.412+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:00.413263+0000) 2022-01-31T22:48:01.085 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:01 smithi167 conmon[54076]: debug 2022-01-31T22:48:01.042+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.043521+0000) 2022-01-31T22:48:01.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:00 smithi167 conmon[60316]: debug 2022-01-31T22:48:00.802+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:00.803645+0000) 2022-01-31T22:48:01.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:00 smithi167 conmon[49112]: debug 2022-01-31T22:48:00.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:00.875919+0000) 2022-01-31T22:48:01.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[41853]: debug 2022-01-31T22:48:01.240+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.241224+0000) 2022-01-31T22:48:01.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[46715]: debug 2022-01-31T22:48:01.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.291305+0000) 2022-01-31T22:48:01.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[51620]: debug 2022-01-31T22:48:01.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.413459+0000) 2022-01-31T22:48:01.791 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:01 smithi167 conmon[49112]: debug 2022-01-31T22:48:01.700+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.701547+0000) 2022-01-31T22:48:01.792 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:01 smithi167 conmon[54076]: debug 2022-01-31T22:48:01.700+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.701773+0000) 2022-01-31T22:48:01.792 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:01 smithi167 conmon[60316]: debug 2022-01-31T22:48:01.700+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.702032+0000) 2022-01-31T22:48:02.085 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:02 smithi167 conmon[54076]: debug 2022-01-31T22:48:02.042+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:02.043665+0000) 2022-01-31T22:48:02.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:01 smithi167 conmon[49112]: debug 2022-01-31T22:48:01.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.876130+0000) 2022-01-31T22:48:02.087 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:01 smithi167 conmon[60316]: debug 2022-01-31T22:48:01.803+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.803840+0000) 2022-01-31T22:48:02.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[46715]: debug 2022-01-31T22:48:01.700+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.700914+0000) 2022-01-31T22:48:02.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[51620]: debug 2022-01-31T22:48:01.700+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.701458+0000) 2022-01-31T22:48:02.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[35325]: debug 2022-01-31T22:48:01.725+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238492 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:02.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:01 smithi171 conmon[41853]: debug 2022-01-31T22:48:01.702+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:01.703137+0000) 2022-01-31T22:48:02.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:02 smithi171 conmon[41853]: debug 2022-01-31T22:48:02.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:02.241400+0000) 2022-01-31T22:48:02.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:02 smithi171 conmon[46715]: debug 2022-01-31T22:48:02.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:02.291477+0000) 2022-01-31T22:48:02.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:02 smithi171 conmon[51620]: debug 2022-01-31T22:48:02.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:02.413672+0000) 2022-01-31T22:48:03.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:03 smithi167 conmon[54076]: debug 2022-01-31T22:48:03.043+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:03.043779+0000) 2022-01-31T22:48:03.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:02 smithi167 conmon[49112]: debug 2022-01-31T22:48:02.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:02.876297+0000) 2022-01-31T22:48:03.087 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:02 smithi167 conmon[60316]: debug 2022-01-31T22:48:02.803+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:02.803982+0000) 2022-01-31T22:48:03.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:03 smithi171 conmon[41853]: debug 2022-01-31T22:48:03.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:03.241583+0000) 2022-01-31T22:48:03.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:03 smithi171 conmon[46715]: debug 2022-01-31T22:48:03.290+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:03.291660+0000) 2022-01-31T22:48:03.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:03 smithi171 conmon[51620]: debug 2022-01-31T22:48:03.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:03.413830+0000) 2022-01-31T22:48:04.086 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:03 smithi167 conmon[49112]: debug 2022-01-31T22:48:03.875+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:03.876519+0000) 2022-01-31T22:48:04.086 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:03 smithi167 conmon[60316]: debug 2022-01-31T22:48:03.803+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:03.804175+0000) 2022-01-31T22:48:04.087 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:04 smithi167 conmon[54076]: debug 2022-01-31T22:48:04.043+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:04.043933+0000) 2022-01-31T22:48:04.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:04 smithi171 conmon[46715]: debug 2022-01-31T22:48:04.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:04.291834+0000) 2022-01-31T22:48:04.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:04 smithi171 conmon[51620]: debug 2022-01-31T22:48:04.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:04.414020+0000) 2022-01-31T22:48:04.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:04 smithi171 conmon[41853]: debug 2022-01-31T22:48:04.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:04.241779+0000) 2022-01-31T22:48:05.044 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:04 smithi167 conmon[60316]: debug 2022-01-31T22:48:04.803+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:04.804371+0000) 2022-01-31T22:48:05.045 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:04 smithi167 conmon[49112]: debug 2022-01-31T22:48:04.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:04.876670+0000) 2022-01-31T22:48:05.046 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:05 smithi167 conmon[54076]: debug 2022-01-31T22:48:05.043+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:05.044129+0000) 2022-01-31T22:48:05.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:05 smithi171 conmon[41853]: debug 2022-01-31T22:48:05.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:05.241965+0000) 2022-01-31T22:48:05.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:05 smithi171 conmon[46715]: debug 2022-01-31T22:48:05.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:05.291935+0000) 2022-01-31T22:48:05.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:05 smithi171 conmon[51620]: debug 2022-01-31T22:48:05.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:05.414223+0000) 2022-01-31T22:48:06.048 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:05 smithi167 conmon[49112]: debug 2022-01-31T22:48:05.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:05.876828+0000) 2022-01-31T22:48:06.049 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:06 smithi167 conmon[54076]: debug 2022-01-31T22:48:06.043+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.044339+0000) 2022-01-31T22:48:06.049 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:05 smithi167 conmon[60316]: debug 2022-01-31T22:48:05.803+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:05.804579+0000) 2022-01-31T22:48:06.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[41853]: debug 2022-01-31T22:48:06.241+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.242138+0000) 2022-01-31T22:48:06.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[46715]: debug 2022-01-31T22:48:06.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.292038+0000) 2022-01-31T22:48:06.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[51620]: debug 2022-01-31T22:48:06.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.414384+0000) 2022-01-31T22:48:06.792 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:06 smithi167 conmon[49112]: debug 2022-01-31T22:48:06.727+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.728986+0000) 2022-01-31T22:48:06.793 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:06 smithi167 conmon[54076]: debug 2022-01-31T22:48:06.728+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.729637+0000) 2022-01-31T22:48:06.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:06 smithi167 conmon[60316]: debug 2022-01-31T22:48:06.728+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.729712+0000) 2022-01-31T22:48:07.050 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:06 smithi167 conmon[49112]: debug 2022-01-31T22:48:06.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.876959+0000) 2022-01-31T22:48:07.051 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:07 smithi167 conmon[54076]: debug 2022-01-31T22:48:07.043+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:07.044526+0000) 2022-01-31T22:48:07.051 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:06 smithi167 conmon[60316]: debug 2022-01-31T22:48:06.804+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.804745+0000) 2022-01-31T22:48:07.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[35325]: debug 2022-01-31T22:48:06.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238604 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:07.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[41853]: debug 2022-01-31T22:48:06.729+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.730011+0000) 2022-01-31T22:48:07.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[46715]: debug 2022-01-31T22:48:06.728+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.728790+0000) 2022-01-31T22:48:07.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:06 smithi171 conmon[51620]: debug 2022-01-31T22:48:06.728+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:06.729238+0000) 2022-01-31T22:48:07.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:07 smithi171 conmon[46715]: debug 2022-01-31T22:48:07.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:07.292183+0000) 2022-01-31T22:48:07.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:07 smithi171 conmon[41853]: debug 2022-01-31T22:48:07.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:07.242323+0000) 2022-01-31T22:48:07.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:07 smithi171 conmon[51620]: debug 2022-01-31T22:48:07.413+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:07.414590+0000) 2022-01-31T22:48:08.054 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:07 smithi167 conmon[49112]: debug 2022-01-31T22:48:07.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:07.877104+0000) 2022-01-31T22:48:08.054 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:08 smithi167 conmon[54076]: debug 2022-01-31T22:48:08.044+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:08.044669+0000) 2022-01-31T22:48:08.055 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:07 smithi167 conmon[60316]: debug 2022-01-31T22:48:07.804+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:07.804891+0000) 2022-01-31T22:48:08.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:08 smithi171 conmon[41853]: debug 2022-01-31T22:48:08.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:08.242459+0000) 2022-01-31T22:48:08.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:08 smithi171 conmon[46715]: debug 2022-01-31T22:48:08.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:08.292355+0000) 2022-01-31T22:48:08.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:08 smithi171 conmon[51620]: debug 2022-01-31T22:48:08.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:08.414784+0000) 2022-01-31T22:48:09.057 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:08 smithi167 conmon[49112]: debug 2022-01-31T22:48:08.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:08.877312+0000) 2022-01-31T22:48:09.058 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:09 smithi167 conmon[54076]: debug 2022-01-31T22:48:09.044+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:09.044852+0000) 2022-01-31T22:48:09.059 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:08 smithi167 conmon[60316]: debug 2022-01-31T22:48:08.804+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:08.805130+0000) 2022-01-31T22:48:09.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:08 smithi171 conmon[35325]: debug 2022-01-31T22:48:08.912+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:48:09.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:09 smithi171 conmon[41853]: debug 2022-01-31T22:48:09.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:09.242644+0000) 2022-01-31T22:48:09.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:09 smithi171 conmon[46715]: debug 2022-01-31T22:48:09.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:09.292552+0000) 2022-01-31T22:48:09.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:09 smithi171 conmon[51620]: debug 2022-01-31T22:48:09.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:09.414971+0000) 2022-01-31T22:48:10.061 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:09 smithi167 conmon[49112]: debug 2022-01-31T22:48:09.876+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:09.877529+0000) 2022-01-31T22:48:10.062 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:10 smithi167 conmon[54076]: debug 2022-01-31T22:48:10.044+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:10.045007+0000) 2022-01-31T22:48:10.062 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:09 smithi167 conmon[60316]: debug 2022-01-31T22:48:09.804+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:09.805351+0000) 2022-01-31T22:48:10.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:10 smithi171 conmon[41853]: debug 2022-01-31T22:48:10.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:10.242798+0000) 2022-01-31T22:48:10.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:10 smithi171 conmon[46715]: debug 2022-01-31T22:48:10.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:10.292733+0000) 2022-01-31T22:48:10.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:10 smithi171 conmon[51620]: debug 2022-01-31T22:48:10.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:10.415174+0000) 2022-01-31T22:48:11.065 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:10 smithi167 conmon[49112]: debug 2022-01-31T22:48:10.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:10.877716+0000) 2022-01-31T22:48:11.065 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:11 smithi167 conmon[54076]: debug 2022-01-31T22:48:11.044+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.045249+0000) 2022-01-31T22:48:11.066 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:10 smithi167 conmon[60316]: debug 2022-01-31T22:48:10.805+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:10.805559+0000) 2022-01-31T22:48:11.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[41853]: debug 2022-01-31T22:48:11.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.242969+0000) 2022-01-31T22:48:11.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[46715]: debug 2022-01-31T22:48:11.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.292911+0000) 2022-01-31T22:48:11.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[51620]: debug 2022-01-31T22:48:11.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.415358+0000) 2022-01-31T22:48:11.793 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:11 smithi167 conmon[49112]: debug 2022-01-31T22:48:11.755+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.755879+0000) 2022-01-31T22:48:11.794 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:11 smithi167 conmon[54076]: debug 2022-01-31T22:48:11.756+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.756983+0000) 2022-01-31T22:48:11.794 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:11 smithi167 conmon[60316]: debug 2022-01-31T22:48:11.755+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.755970+0000) 2022-01-31T22:48:12.067 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:11 smithi167 conmon[49112]: debug 2022-01-31T22:48:11.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.877924+0000) 2022-01-31T22:48:12.068 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:12 smithi167 conmon[54076]: debug 2022-01-31T22:48:12.044+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:12.045461+0000) 2022-01-31T22:48:12.068 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:11 smithi167 conmon[60316]: debug 2022-01-31T22:48:11.805+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.805723+0000) 2022-01-31T22:48:12.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[46715]: debug 2022-01-31T22:48:11.754+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.755455+0000) 2022-01-31T22:48:12.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[51620]: debug 2022-01-31T22:48:11.756+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.756769+0000) 2022-01-31T22:48:12.097 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[35325]: debug 2022-01-31T22:48:11.777+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238716 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:12.098 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:11 smithi171 conmon[41853]: debug 2022-01-31T22:48:11.755+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:11.756134+0000) 2022-01-31T22:48:12.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:12 smithi171 conmon[41853]: debug 2022-01-31T22:48:12.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:12.243154+0000) 2022-01-31T22:48:12.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:12 smithi171 conmon[46715]: debug 2022-01-31T22:48:12.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:12.293087+0000) 2022-01-31T22:48:12.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:12 smithi171 conmon[51620]: debug 2022-01-31T22:48:12.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:12.415518+0000) 2022-01-31T22:48:13.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:12 smithi167 conmon[49112]: debug 2022-01-31T22:48:12.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:12.878054+0000) 2022-01-31T22:48:13.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:13 smithi167 conmon[54076]: debug 2022-01-31T22:48:13.045+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:13.045578+0000) 2022-01-31T22:48:13.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:12 smithi167 conmon[60316]: debug 2022-01-31T22:48:12.805+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:12.805896+0000) 2022-01-31T22:48:13.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:13 smithi171 conmon[41853]: debug 2022-01-31T22:48:13.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:13.243295+0000) 2022-01-31T22:48:13.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:13 smithi171 conmon[51620]: debug 2022-01-31T22:48:13.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:13.415655+0000) 2022-01-31T22:48:13.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:13 smithi171 conmon[46715]: debug 2022-01-31T22:48:13.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:13.293262+0000) 2022-01-31T22:48:14.074 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:13 smithi167 conmon[49112]: debug 2022-01-31T22:48:13.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:13.878218+0000) 2022-01-31T22:48:14.075 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:14 smithi167 conmon[54076]: debug 2022-01-31T22:48:14.045+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:14.045747+0000) 2022-01-31T22:48:14.075 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:13 smithi167 conmon[60316]: debug 2022-01-31T22:48:13.805+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:13.806115+0000) 2022-01-31T22:48:14.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:14 smithi171 conmon[41853]: debug 2022-01-31T22:48:14.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:14.243466+0000) 2022-01-31T22:48:14.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:14 smithi171 conmon[46715]: debug 2022-01-31T22:48:14.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:14.293502+0000) 2022-01-31T22:48:14.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:14 smithi171 conmon[51620]: debug 2022-01-31T22:48:14.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:14.415847+0000) 2022-01-31T22:48:15.077 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:14 smithi167 conmon[49112]: debug 2022-01-31T22:48:14.877+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:14.878403+0000) 2022-01-31T22:48:15.078 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:15 smithi167 conmon[54076]: debug 2022-01-31T22:48:15.045+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:15.045970+0000) 2022-01-31T22:48:15.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:14 smithi167 conmon[60316]: debug 2022-01-31T22:48:14.805+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:14.806312+0000) 2022-01-31T22:48:15.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:15 smithi171 conmon[41853]: debug 2022-01-31T22:48:15.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:15.243603+0000) 2022-01-31T22:48:15.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:15 smithi171 conmon[46715]: debug 2022-01-31T22:48:15.291+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:15.293628+0000) 2022-01-31T22:48:15.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:15 smithi171 conmon[51620]: debug 2022-01-31T22:48:15.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:15.415967+0000) 2022-01-31T22:48:16.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:15 smithi167 conmon[49112]: debug 2022-01-31T22:48:15.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:15.878589+0000) 2022-01-31T22:48:16.082 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:16 smithi167 conmon[54076]: debug 2022-01-31T22:48:16.045+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.046172+0000) 2022-01-31T22:48:16.082 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:15 smithi167 conmon[60316]: debug 2022-01-31T22:48:15.805+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:15.806466+0000) 2022-01-31T22:48:16.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[41853]: debug 2022-01-31T22:48:16.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.243763+0000) 2022-01-31T22:48:16.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[46715]: debug 2022-01-31T22:48:16.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.293820+0000) 2022-01-31T22:48:16.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[51620]: debug 2022-01-31T22:48:16.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.416098+0000) 2022-01-31T22:48:16.778 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:16 smithi167 conmon[49112]: debug 2022-01-31T22:48:16.781+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.781840+0000) 2022-01-31T22:48:16.779 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:16 smithi167 conmon[54076]: debug 2022-01-31T22:48:16.782+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.783151+0000) 2022-01-31T22:48:16.780 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:16 smithi167 conmon[60316]: debug 2022-01-31T22:48:16.781+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.781767+0000) 2022-01-31T22:48:17.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:16 smithi167 conmon[49112]: debug 2022-01-31T22:48:16.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.878744+0000) 2022-01-31T22:48:17.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:16 smithi167 conmon[60316]: debug 2022-01-31T22:48:16.806+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.806639+0000) 2022-01-31T22:48:17.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[35325]: debug 2022-01-31T22:48:16.810+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238828 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:17.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[41853]: debug 2022-01-31T22:48:16.781+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.782795+0000) 2022-01-31T22:48:17.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[46715]: debug 2022-01-31T22:48:16.780+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.782593+0000) 2022-01-31T22:48:17.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:16 smithi171 conmon[51620]: debug 2022-01-31T22:48:16.781+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:16.783159+0000) 2022-01-31T22:48:17.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:17 smithi167 conmon[54076]: debug 2022-01-31T22:48:17.045+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:17.046368+0000) 2022-01-31T22:48:17.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:17 smithi171 conmon[41853]: debug 2022-01-31T22:48:17.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:17.243903+0000) 2022-01-31T22:48:17.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:17 smithi171 conmon[46715]: debug 2022-01-31T22:48:17.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:17.293902+0000) 2022-01-31T22:48:17.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:17 smithi171 conmon[51620]: debug 2022-01-31T22:48:17.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:17.416289+0000) 2022-01-31T22:48:18.086 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:18 smithi167 conmon[54076]: debug 2022-01-31T22:48:18.045+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:18.046473+0000) 2022-01-31T22:48:18.087 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:17 smithi167 conmon[60316]: debug 2022-01-31T22:48:17.806+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:17.806806+0000) 2022-01-31T22:48:18.087 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:17 smithi167 conmon[49112]: debug 2022-01-31T22:48:17.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:17.878876+0000) 2022-01-31T22:48:18.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:18 smithi171 conmon[41853]: debug 2022-01-31T22:48:18.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:18.244025+0000) 2022-01-31T22:48:18.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:18 smithi171 conmon[46715]: debug 2022-01-31T22:48:18.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:18.294064+0000) 2022-01-31T22:48:18.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:18 smithi171 conmon[51620]: debug 2022-01-31T22:48:18.414+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:18.416495+0000) 2022-01-31T22:48:19.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:18 smithi167 conmon[49112]: debug 2022-01-31T22:48:18.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:18.879021+0000) 2022-01-31T22:48:19.089 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:19 smithi167 conmon[54076]: debug 2022-01-31T22:48:19.046+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:19.046663+0000) 2022-01-31T22:48:19.089 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:18 smithi167 conmon[60316]: debug 2022-01-31T22:48:18.806+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:18.806984+0000) 2022-01-31T22:48:19.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:19 smithi171 conmon[41853]: debug 2022-01-31T22:48:19.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:19.244220+0000) 2022-01-31T22:48:19.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:19 smithi171 conmon[46715]: debug 2022-01-31T22:48:19.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:19.294276+0000) 2022-01-31T22:48:19.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:19 smithi171 conmon[51620]: debug 2022-01-31T22:48:19.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:19.416672+0000) 2022-01-31T22:48:20.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:19 smithi167 conmon[49112]: debug 2022-01-31T22:48:19.878+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:19.879159+0000) 2022-01-31T22:48:20.089 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:20 smithi167 conmon[54076]: debug 2022-01-31T22:48:20.046+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:20.046825+0000) 2022-01-31T22:48:20.090 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:19 smithi167 conmon[60316]: debug 2022-01-31T22:48:19.806+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:19.807140+0000) 2022-01-31T22:48:20.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:20 smithi171 conmon[41853]: debug 2022-01-31T22:48:20.242+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:20.244353+0000) 2022-01-31T22:48:20.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:20 smithi171 conmon[46715]: debug 2022-01-31T22:48:20.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:20.294465+0000) 2022-01-31T22:48:20.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:20 smithi171 conmon[51620]: debug 2022-01-31T22:48:20.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:20.416875+0000) 2022-01-31T22:48:21.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:20 smithi167 conmon[49112]: debug 2022-01-31T22:48:20.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:20.879350+0000) 2022-01-31T22:48:21.089 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:21 smithi167 conmon[54076]: debug 2022-01-31T22:48:21.046+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.047002+0000) 2022-01-31T22:48:21.090 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:20 smithi167 conmon[60316]: debug 2022-01-31T22:48:20.806+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:20.807324+0000) 2022-01-31T22:48:21.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[41853]: debug 2022-01-31T22:48:21.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.244505+0000) 2022-01-31T22:48:21.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[46715]: debug 2022-01-31T22:48:21.292+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.294646+0000) 2022-01-31T22:48:21.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[51620]: debug 2022-01-31T22:48:21.415+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.417025+0000) 2022-01-31T22:48:22.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:21 smithi167 conmon[49112]: debug 2022-01-31T22:48:21.815+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.815717+0000) 2022-01-31T22:48:22.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:21 smithi167 conmon[49112]: debug 2022-01-31T22:48:21.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.879535+0000) 2022-01-31T22:48:22.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:21 smithi167 conmon[54076]: debug 2022-01-31T22:48:21.814+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.815454+0000) 2022-01-31T22:48:22.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:22 smithi167 conmon[54076]: debug 2022-01-31T22:48:22.046+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:22.047212+0000) 2022-01-31T22:48:22.090 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:21 smithi167 conmon[60316]: debug 2022-01-31T22:48:21.807+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.807530+0000) 2022-01-31T22:48:22.091 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:21 smithi167 conmon[60316]: debug 2022-01-31T22:48:21.815+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.815997+0000) 2022-01-31T22:48:22.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[41853]: debug 2022-01-31T22:48:21.812+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.814651+0000) 2022-01-31T22:48:22.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[46715]: debug 2022-01-31T22:48:21.813+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.815065+0000) 2022-01-31T22:48:22.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[51620]: debug 2022-01-31T22:48:21.813+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:21.815167+0000) 2022-01-31T22:48:22.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:21 smithi171 conmon[35325]: debug 2022-01-31T22:48:21.838+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 238939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:22.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:22 smithi171 conmon[51620]: debug 2022-01-31T22:48:22.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:22.417195+0000) 2022-01-31T22:48:22.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:22 smithi171 conmon[41853]: debug 2022-01-31T22:48:22.243+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:22.244705+0000) 2022-01-31T22:48:22.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:22 smithi171 conmon[46715]: debug 2022-01-31T22:48:22.293+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:22.294808+0000) 2022-01-31T22:48:23.088 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:23 smithi167 conmon[54076]: debug 2022-01-31T22:48:23.047+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:23.047371+0000) 2022-01-31T22:48:23.089 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:22 smithi167 conmon[60316]: debug 2022-01-31T22:48:22.807+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:22.807660+0000) 2022-01-31T22:48:23.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:22 smithi167 conmon[49112]: debug 2022-01-31T22:48:22.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:22.879689+0000) 2022-01-31T22:48:23.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:23 smithi171 conmon[46715]: debug 2022-01-31T22:48:23.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:23.294920+0000) 2022-01-31T22:48:23.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:23 smithi171 conmon[41853]: debug 2022-01-31T22:48:23.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:23.244876+0000) 2022-01-31T22:48:23.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:23 smithi171 conmon[51620]: debug 2022-01-31T22:48:23.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:23.417346+0000) 2022-01-31T22:48:24.089 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:24 smithi167 conmon[54076]: debug 2022-01-31T22:48:24.047+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:24.047576+0000) 2022-01-31T22:48:24.090 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:23 smithi167 conmon[60316]: debug 2022-01-31T22:48:23.807+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:23.807807+0000) 2022-01-31T22:48:24.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:23 smithi167 conmon[49112]: debug 2022-01-31T22:48:23.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:23.879852+0000) 2022-01-31T22:48:24.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:23 smithi171 conmon[35325]: debug 2022-01-31T22:48:23.912+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:48:24.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:24 smithi171 conmon[46715]: debug 2022-01-31T22:48:24.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:24.295100+0000) 2022-01-31T22:48:24.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:24 smithi171 conmon[41853]: debug 2022-01-31T22:48:24.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:24.245062+0000) 2022-01-31T22:48:24.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:24 smithi171 conmon[51620]: debug 2022-01-31T22:48:24.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:24.417532+0000) 2022-01-31T22:48:25.089 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:25 smithi167 conmon[54076]: debug 2022-01-31T22:48:25.047+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:25.047806+0000) 2022-01-31T22:48:25.089 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:24 smithi167 conmon[60316]: debug 2022-01-31T22:48:24.807+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:24.807978+0000) 2022-01-31T22:48:25.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:24 smithi167 conmon[49112]: debug 2022-01-31T22:48:24.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:24.880036+0000) 2022-01-31T22:48:25.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:25 smithi171 conmon[46715]: debug 2022-01-31T22:48:25.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:25.295274+0000) 2022-01-31T22:48:25.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:25 smithi171 conmon[41853]: debug 2022-01-31T22:48:25.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:25.245266+0000) 2022-01-31T22:48:25.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:25 smithi171 conmon[51620]: debug 2022-01-31T22:48:25.416+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:25.417733+0000) 2022-01-31T22:48:26.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:25 smithi167 conmon[49112]: debug 2022-01-31T22:48:25.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:25.880158+0000) 2022-01-31T22:48:26.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:26 smithi167 conmon[54076]: debug 2022-01-31T22:48:26.047+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.048003+0000) 2022-01-31T22:48:26.091 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:25 smithi167 conmon[60316]: debug 2022-01-31T22:48:25.807+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:25.808136+0000) 2022-01-31T22:48:26.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[41853]: debug 2022-01-31T22:48:26.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.245529+0000) 2022-01-31T22:48:26.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[46715]: debug 2022-01-31T22:48:26.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.295431+0000) 2022-01-31T22:48:26.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[51620]: debug 2022-01-31T22:48:26.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.417951+0000) 2022-01-31T22:48:27.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:26 smithi167 conmon[49112]: debug 2022-01-31T22:48:26.842+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.843265+0000) 2022-01-31T22:48:27.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:26 smithi167 conmon[49112]: debug 2022-01-31T22:48:26.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.880278+0000) 2022-01-31T22:48:27.091 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:26 smithi167 conmon[54076]: debug 2022-01-31T22:48:26.841+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.842104+0000) 2022-01-31T22:48:27.091 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:27 smithi167 conmon[54076]: debug 2022-01-31T22:48:27.047+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:27.048163+0000) 2022-01-31T22:48:27.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:26 smithi167 conmon[60316]: debug 2022-01-31T22:48:26.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.808300+0000) 2022-01-31T22:48:27.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:26 smithi167 conmon[60316]: debug 2022-01-31T22:48:26.841+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.842300+0000) 2022-01-31T22:48:27.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[35325]: debug 2022-01-31T22:48:26.867+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239051 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:27.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[41853]: debug 2022-01-31T22:48:26.841+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.842698+0000) 2022-01-31T22:48:27.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[46715]: debug 2022-01-31T22:48:26.841+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.841786+0000) 2022-01-31T22:48:27.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:26 smithi171 conmon[51620]: debug 2022-01-31T22:48:26.842+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:26.842785+0000) 2022-01-31T22:48:27.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:27 smithi171 conmon[41853]: debug 2022-01-31T22:48:27.244+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:27.245707+0000) 2022-01-31T22:48:27.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:27 smithi171 conmon[46715]: debug 2022-01-31T22:48:27.294+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:27.295602+0000) 2022-01-31T22:48:27.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:27 smithi171 conmon[51620]: debug 2022-01-31T22:48:27.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:27.418154+0000) 2022-01-31T22:48:28.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:27 smithi167 conmon[49112]: debug 2022-01-31T22:48:27.879+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:27.880377+0000) 2022-01-31T22:48:28.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:28 smithi167 conmon[54076]: debug 2022-01-31T22:48:28.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:28.048311+0000) 2022-01-31T22:48:28.091 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:27 smithi167 conmon[60316]: debug 2022-01-31T22:48:27.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:27.808483+0000) 2022-01-31T22:48:28.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:28 smithi171 conmon[41853]: debug 2022-01-31T22:48:28.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:28.245872+0000) 2022-01-31T22:48:28.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:28 smithi171 conmon[46715]: debug 2022-01-31T22:48:28.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:28.295789+0000) 2022-01-31T22:48:28.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:28 smithi171 conmon[51620]: debug 2022-01-31T22:48:28.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:28.418281+0000) 2022-01-31T22:48:29.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:28 smithi167 conmon[49112]: debug 2022-01-31T22:48:28.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:28.880589+0000) 2022-01-31T22:48:29.091 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:28 smithi167 conmon[60316]: debug 2022-01-31T22:48:28.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:28.808644+0000) 2022-01-31T22:48:29.091 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:29 smithi167 conmon[54076]: debug 2022-01-31T22:48:29.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:29.048477+0000) 2022-01-31T22:48:29.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:29 smithi171 conmon[41853]: debug 2022-01-31T22:48:29.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:29.246020+0000) 2022-01-31T22:48:29.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:29 smithi171 conmon[46715]: debug 2022-01-31T22:48:29.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:29.295957+0000) 2022-01-31T22:48:29.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:29 smithi171 conmon[51620]: debug 2022-01-31T22:48:29.417+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:29.418491+0000) 2022-01-31T22:48:30.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:29 smithi167 conmon[49112]: debug 2022-01-31T22:48:29.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:29.880799+0000) 2022-01-31T22:48:30.090 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:30 smithi167 conmon[54076]: debug 2022-01-31T22:48:30.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:30.048665+0000) 2022-01-31T22:48:30.091 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:29 smithi167 conmon[60316]: debug 2022-01-31T22:48:29.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:29.808809+0000) 2022-01-31T22:48:30.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:30 smithi171 conmon[41853]: debug 2022-01-31T22:48:30.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:30.246187+0000) 2022-01-31T22:48:30.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:30 smithi171 conmon[46715]: debug 2022-01-31T22:48:30.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:30.296090+0000) 2022-01-31T22:48:30.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:30 smithi171 conmon[51620]: debug 2022-01-31T22:48:30.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:30.418670+0000) 2022-01-31T22:48:31.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:30 smithi167 conmon[49112]: debug 2022-01-31T22:48:30.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:30.880937+0000) 2022-01-31T22:48:31.091 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:30 smithi167 conmon[60316]: debug 2022-01-31T22:48:30.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:30.808993+0000) 2022-01-31T22:48:31.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:31 smithi167 conmon[54076]: debug 2022-01-31T22:48:31.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.048827+0000) 2022-01-31T22:48:31.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[46715]: debug 2022-01-31T22:48:31.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.296305+0000) 2022-01-31T22:48:31.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[41853]: debug 2022-01-31T22:48:31.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.246323+0000) 2022-01-31T22:48:31.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[51620]: debug 2022-01-31T22:48:31.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.418858+0000) 2022-01-31T22:48:32.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:31 smithi167 conmon[49112]: debug 2022-01-31T22:48:31.870+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.871217+0000) 2022-01-31T22:48:32.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:31 smithi167 conmon[49112]: debug 2022-01-31T22:48:31.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.881100+0000) 2022-01-31T22:48:32.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:31 smithi167 conmon[54076]: debug 2022-01-31T22:48:31.869+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.870010+0000) 2022-01-31T22:48:32.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:32 smithi167 conmon[54076]: debug 2022-01-31T22:48:32.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:32.049046+0000) 2022-01-31T22:48:32.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:31 smithi167 conmon[60316]: debug 2022-01-31T22:48:31.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.809203+0000) 2022-01-31T22:48:32.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:31 smithi167 conmon[60316]: debug 2022-01-31T22:48:31.870+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.870764+0000) 2022-01-31T22:48:32.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[35325]: debug 2022-01-31T22:48:31.894+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:32.097 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[41853]: debug 2022-01-31T22:48:31.869+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.869955+0000) 2022-01-31T22:48:32.098 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[46715]: debug 2022-01-31T22:48:31.869+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.870581+0000) 2022-01-31T22:48:32.099 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:31 smithi171 conmon[51620]: debug 2022-01-31T22:48:31.869+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:31.870195+0000) 2022-01-31T22:48:32.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:32 smithi171 conmon[41853]: debug 2022-01-31T22:48:32.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:32.246459+0000) 2022-01-31T22:48:32.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:32 smithi171 conmon[46715]: debug 2022-01-31T22:48:32.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:32.296427+0000) 2022-01-31T22:48:32.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:32 smithi171 conmon[51620]: debug 2022-01-31T22:48:32.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:32.419031+0000) 2022-01-31T22:48:33.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:32 smithi167 conmon[49112]: debug 2022-01-31T22:48:32.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:32.881268+0000) 2022-01-31T22:48:33.091 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:33 smithi167 conmon[54076]: debug 2022-01-31T22:48:33.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:33.049231+0000) 2022-01-31T22:48:33.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:32 smithi167 conmon[60316]: debug 2022-01-31T22:48:32.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:32.809398+0000) 2022-01-31T22:48:33.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:33 smithi171 conmon[41853]: debug 2022-01-31T22:48:33.245+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:33.246604+0000) 2022-01-31T22:48:33.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:33 smithi171 conmon[46715]: debug 2022-01-31T22:48:33.295+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:33.296556+0000) 2022-01-31T22:48:33.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:33 smithi171 conmon[51620]: debug 2022-01-31T22:48:33.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:33.419183+0000) 2022-01-31T22:48:34.090 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:33 smithi167 conmon[49112]: debug 2022-01-31T22:48:33.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:33.881425+0000) 2022-01-31T22:48:34.091 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:34 smithi167 conmon[54076]: debug 2022-01-31T22:48:34.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:34.049421+0000) 2022-01-31T22:48:34.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:33 smithi167 conmon[60316]: debug 2022-01-31T22:48:33.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:33.809571+0000) 2022-01-31T22:48:34.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:34 smithi171 conmon[46715]: debug 2022-01-31T22:48:34.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:34.296784+0000) 2022-01-31T22:48:34.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:34 smithi171 conmon[41853]: debug 2022-01-31T22:48:34.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:34.246775+0000) 2022-01-31T22:48:34.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:34 smithi171 conmon[51620]: debug 2022-01-31T22:48:34.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:34.419346+0000) 2022-01-31T22:48:35.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:34 smithi167 conmon[49112]: debug 2022-01-31T22:48:34.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:34.881652+0000) 2022-01-31T22:48:35.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:35 smithi167 conmon[54076]: debug 2022-01-31T22:48:35.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:35.049606+0000) 2022-01-31T22:48:35.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:34 smithi167 conmon[60316]: debug 2022-01-31T22:48:34.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:34.809759+0000) 2022-01-31T22:48:35.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:35 smithi171 conmon[41853]: debug 2022-01-31T22:48:35.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:35.246913+0000) 2022-01-31T22:48:35.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:35 smithi171 conmon[46715]: debug 2022-01-31T22:48:35.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:35.296963+0000) 2022-01-31T22:48:35.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:35 smithi171 conmon[51620]: debug 2022-01-31T22:48:35.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:35.419464+0000) 2022-01-31T22:48:36.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:35 smithi167 conmon[49112]: debug 2022-01-31T22:48:35.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:35.881760+0000) 2022-01-31T22:48:36.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:36 smithi167 conmon[54076]: debug 2022-01-31T22:48:36.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.049785+0000) 2022-01-31T22:48:36.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:35 smithi167 conmon[60316]: debug 2022-01-31T22:48:35.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:35.809894+0000) 2022-01-31T22:48:36.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[51620]: debug 2022-01-31T22:48:36.418+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.419640+0000) 2022-01-31T22:48:36.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[41853]: debug 2022-01-31T22:48:36.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.247093+0000) 2022-01-31T22:48:36.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[46715]: debug 2022-01-31T22:48:36.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.297130+0000) 2022-01-31T22:48:37.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:36 smithi167 conmon[49112]: debug 2022-01-31T22:48:36.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.881928+0000) 2022-01-31T22:48:37.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:36 smithi167 conmon[49112]: debug 2022-01-31T22:48:36.897+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.899220+0000) 2022-01-31T22:48:37.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:36 smithi167 conmon[54076]: debug 2022-01-31T22:48:36.896+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.898031+0000) 2022-01-31T22:48:37.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:37 smithi167 conmon[54076]: debug 2022-01-31T22:48:37.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:37.049998+0000) 2022-01-31T22:48:37.093 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:36 smithi167 conmon[60316]: debug 2022-01-31T22:48:36.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.810077+0000) 2022-01-31T22:48:37.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:36 smithi167 conmon[60316]: debug 2022-01-31T22:48:36.896+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.897804+0000) 2022-01-31T22:48:37.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[35325]: debug 2022-01-31T22:48:36.921+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239275 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:37.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[41853]: debug 2022-01-31T22:48:36.897+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.897967+0000) 2022-01-31T22:48:37.096 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[46715]: debug 2022-01-31T22:48:36.896+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.897552+0000) 2022-01-31T22:48:37.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:36 smithi171 conmon[51620]: debug 2022-01-31T22:48:36.897+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:36.898083+0000) 2022-01-31T22:48:37.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:37 smithi171 conmon[41853]: debug 2022-01-31T22:48:37.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:37.247271+0000) 2022-01-31T22:48:37.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:37 smithi171 conmon[46715]: debug 2022-01-31T22:48:37.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:37.297289+0000) 2022-01-31T22:48:37.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:37 smithi171 conmon[51620]: debug 2022-01-31T22:48:37.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:37.419838+0000) 2022-01-31T22:48:38.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:37 smithi167 conmon[49112]: debug 2022-01-31T22:48:37.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:37.882134+0000) 2022-01-31T22:48:38.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:38 smithi167 conmon[54076]: debug 2022-01-31T22:48:38.048+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:38.050146+0000) 2022-01-31T22:48:38.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:37 smithi167 conmon[60316]: debug 2022-01-31T22:48:37.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:37.810244+0000) 2022-01-31T22:48:38.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:38 smithi171 conmon[41853]: debug 2022-01-31T22:48:38.246+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:38.247433+0000) 2022-01-31T22:48:38.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:38 smithi171 conmon[46715]: debug 2022-01-31T22:48:38.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:38.297459+0000) 2022-01-31T22:48:38.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:38 smithi171 conmon[51620]: debug 2022-01-31T22:48:38.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:38.419937+0000) 2022-01-31T22:48:39.091 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:38 smithi167 conmon[49112]: debug 2022-01-31T22:48:38.880+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:38.882294+0000) 2022-01-31T22:48:39.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:39 smithi167 conmon[54076]: debug 2022-01-31T22:48:39.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:39.050307+0000) 2022-01-31T22:48:39.093 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:38 smithi167 conmon[60316]: debug 2022-01-31T22:48:38.808+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:38.810415+0000) 2022-01-31T22:48:39.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:38 smithi171 conmon[35325]: debug 2022-01-31T22:48:38.913+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:48:39.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:39 smithi171 conmon[41853]: debug 2022-01-31T22:48:39.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:39.247613+0000) 2022-01-31T22:48:39.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:39 smithi171 conmon[46715]: debug 2022-01-31T22:48:39.296+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:39.297700+0000) 2022-01-31T22:48:39.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:39 smithi171 conmon[51620]: debug 2022-01-31T22:48:39.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:39.420116+0000) 2022-01-31T22:48:40.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:39 smithi167 conmon[49112]: debug 2022-01-31T22:48:39.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:39.882475+0000) 2022-01-31T22:48:40.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:39 smithi167 conmon[60316]: debug 2022-01-31T22:48:39.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:39.810605+0000) 2022-01-31T22:48:40.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:40 smithi167 conmon[54076]: debug 2022-01-31T22:48:40.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:40.050493+0000) 2022-01-31T22:48:40.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:40 smithi171 conmon[41853]: debug 2022-01-31T22:48:40.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:40.247799+0000) 2022-01-31T22:48:40.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:40 smithi171 conmon[46715]: debug 2022-01-31T22:48:40.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:40.297868+0000) 2022-01-31T22:48:40.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:40 smithi171 conmon[51620]: debug 2022-01-31T22:48:40.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:40.420316+0000) 2022-01-31T22:48:41.092 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:41 smithi167 conmon[54076]: debug 2022-01-31T22:48:41.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.050684+0000) 2022-01-31T22:48:41.092 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:40 smithi167 conmon[60316]: debug 2022-01-31T22:48:40.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:40.810794+0000) 2022-01-31T22:48:41.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:40 smithi167 conmon[49112]: debug 2022-01-31T22:48:40.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:40.882607+0000) 2022-01-31T22:48:41.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[41853]: debug 2022-01-31T22:48:41.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.247956+0000) 2022-01-31T22:48:41.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[46715]: debug 2022-01-31T22:48:41.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.298021+0000) 2022-01-31T22:48:41.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[51620]: debug 2022-01-31T22:48:41.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.420501+0000) 2022-01-31T22:48:42.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:41 smithi167 conmon[49112]: debug 2022-01-31T22:48:41.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.882749+0000) 2022-01-31T22:48:42.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:41 smithi167 conmon[49112]: debug 2022-01-31T22:48:41.922+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.924056+0000) 2022-01-31T22:48:42.093 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:41 smithi167 conmon[60316]: debug 2022-01-31T22:48:41.809+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.810982+0000) 2022-01-31T22:48:42.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:41 smithi167 conmon[60316]: debug 2022-01-31T22:48:41.923+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.924947+0000) 2022-01-31T22:48:42.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:41 smithi167 conmon[54076]: debug 2022-01-31T22:48:41.922+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.923925+0000) 2022-01-31T22:48:42.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:42 smithi167 conmon[54076]: debug 2022-01-31T22:48:42.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:42.050846+0000) 2022-01-31T22:48:42.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[35325]: debug 2022-01-31T22:48:41.948+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239386 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:42.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[41853]: debug 2022-01-31T22:48:41.924+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.925376+0000) 2022-01-31T22:48:42.097 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[46715]: debug 2022-01-31T22:48:41.924+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.925131+0000) 2022-01-31T22:48:42.097 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:41 smithi171 conmon[51620]: debug 2022-01-31T22:48:41.924+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:41.924902+0000) 2022-01-31T22:48:42.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:42 smithi171 conmon[41853]: debug 2022-01-31T22:48:42.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:42.248084+0000) 2022-01-31T22:48:42.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:42 smithi171 conmon[46715]: debug 2022-01-31T22:48:42.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:42.298155+0000) 2022-01-31T22:48:42.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:42 smithi171 conmon[51620]: debug 2022-01-31T22:48:42.419+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:42.420685+0000) 2022-01-31T22:48:43.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:42 smithi167 conmon[49112]: debug 2022-01-31T22:48:42.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:42.882961+0000) 2022-01-31T22:48:43.093 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:42 smithi167 conmon[60316]: debug 2022-01-31T22:48:42.810+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:42.811223+0000) 2022-01-31T22:48:43.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:43 smithi167 conmon[54076]: debug 2022-01-31T22:48:43.050+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:43.050997+0000) 2022-01-31T22:48:43.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:43 smithi171 conmon[41853]: debug 2022-01-31T22:48:43.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:43.248245+0000) 2022-01-31T22:48:43.574 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:43 smithi171 conmon[46715]: debug 2022-01-31T22:48:43.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:43.298324+0000) 2022-01-31T22:48:43.574 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:43 smithi171 conmon[51620]: debug 2022-01-31T22:48:43.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:43.420834+0000) 2022-01-31T22:48:44.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:43 smithi167 conmon[49112]: debug 2022-01-31T22:48:43.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:43.883055+0000) 2022-01-31T22:48:44.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:44 smithi167 conmon[54076]: debug 2022-01-31T22:48:44.049+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:44.051184+0000) 2022-01-31T22:48:44.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:43 smithi167 conmon[60316]: debug 2022-01-31T22:48:43.810+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:43.811401+0000) 2022-01-31T22:48:44.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:44 smithi171 conmon[41853]: debug 2022-01-31T22:48:44.247+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:44.248413+0000) 2022-01-31T22:48:44.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:44 smithi171 conmon[46715]: debug 2022-01-31T22:48:44.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:44.298448+0000) 2022-01-31T22:48:44.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:44 smithi171 conmon[51620]: debug 2022-01-31T22:48:44.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:44.420982+0000) 2022-01-31T22:48:45.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:44 smithi167 conmon[49112]: debug 2022-01-31T22:48:44.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:44.883224+0000) 2022-01-31T22:48:45.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:45 smithi167 conmon[54076]: debug 2022-01-31T22:48:45.050+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:45.051370+0000) 2022-01-31T22:48:45.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:44 smithi167 conmon[60316]: debug 2022-01-31T22:48:44.810+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:44.811619+0000) 2022-01-31T22:48:45.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:45 smithi171 conmon[46715]: debug 2022-01-31T22:48:45.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:45.298610+0000) 2022-01-31T22:48:45.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:45 smithi171 conmon[51620]: debug 2022-01-31T22:48:45.420+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:45.421164+0000) 2022-01-31T22:48:45.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:45 smithi171 conmon[41853]: debug 2022-01-31T22:48:45.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:45.248564+0000) 2022-01-31T22:48:46.093 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:46 smithi167 conmon[54076]: debug 2022-01-31T22:48:46.050+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.051566+0000) 2022-01-31T22:48:46.094 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:45 smithi167 conmon[60316]: debug 2022-01-31T22:48:45.810+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:45.811767+0000) 2022-01-31T22:48:46.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:45 smithi167 conmon[49112]: debug 2022-01-31T22:48:45.881+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:45.883433+0000) 2022-01-31T22:48:46.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[46715]: debug 2022-01-31T22:48:46.297+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.298749+0000) 2022-01-31T22:48:46.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[51620]: debug 2022-01-31T22:48:46.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.421397+0000) 2022-01-31T22:48:46.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[41853]: debug 2022-01-31T22:48:46.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.248737+0000) 2022-01-31T22:48:46.964 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[41853]: debug 2022-01-31T22:48:46.951+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.952374+0000) 2022-01-31T22:48:46.965 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[46715]: debug 2022-01-31T22:48:46.951+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.952639+0000) 2022-01-31T22:48:46.966 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[51620]: debug 2022-01-31T22:48:46.951+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.951884+0000) 2022-01-31T22:48:47.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:46 smithi167 conmon[49112]: debug 2022-01-31T22:48:46.882+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.883668+0000) 2022-01-31T22:48:47.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:46 smithi167 conmon[49112]: debug 2022-01-31T22:48:46.951+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.953310+0000) 2022-01-31T22:48:47.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:46 smithi167 conmon[54076]: debug 2022-01-31T22:48:46.950+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.952025+0000) 2022-01-31T22:48:47.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:47 smithi167 conmon[54076]: debug 2022-01-31T22:48:47.050+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:47.051775+0000) 2022-01-31T22:48:47.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:46 smithi167 conmon[60316]: debug 2022-01-31T22:48:46.811+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.811948+0000) 2022-01-31T22:48:47.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:46 smithi167 conmon[60316]: debug 2022-01-31T22:48:46.950+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:46.951902+0000) 2022-01-31T22:48:47.236 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:46 smithi171 conmon[35325]: debug 2022-01-31T22:48:46.976+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239499 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:47.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:47 smithi171 conmon[46715]: debug 2022-01-31T22:48:47.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:47.299169+0000) 2022-01-31T22:48:47.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:47 smithi171 conmon[41853]: debug 2022-01-31T22:48:47.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:47.248904+0000) 2022-01-31T22:48:47.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:47 smithi171 conmon[51620]: debug 2022-01-31T22:48:47.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:47.421575+0000) 2022-01-31T22:48:48.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:47 smithi167 conmon[49112]: debug 2022-01-31T22:48:47.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:47.883852+0000) 2022-01-31T22:48:48.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:48 smithi167 conmon[54076]: debug 2022-01-31T22:48:48.051+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:48.051928+0000) 2022-01-31T22:48:48.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:47 smithi167 conmon[60316]: debug 2022-01-31T22:48:47.811+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:47.812129+0000) 2022-01-31T22:48:48.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:48 smithi171 conmon[41853]: debug 2022-01-31T22:48:48.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:48.249010+0000) 2022-01-31T22:48:48.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:48 smithi171 conmon[46715]: debug 2022-01-31T22:48:48.298+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:48.299613+0000) 2022-01-31T22:48:48.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:48 smithi171 conmon[51620]: debug 2022-01-31T22:48:48.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:48.421747+0000) 2022-01-31T22:48:49.093 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:48 smithi167 conmon[49112]: debug 2022-01-31T22:48:48.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:48.884001+0000) 2022-01-31T22:48:49.094 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:49 smithi167 conmon[54076]: debug 2022-01-31T22:48:49.051+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:49.052116+0000) 2022-01-31T22:48:49.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:48 smithi167 conmon[60316]: debug 2022-01-31T22:48:48.810+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:48.812264+0000) 2022-01-31T22:48:49.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:49 smithi171 conmon[41853]: debug 2022-01-31T22:48:49.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:49.249188+0000) 2022-01-31T22:48:49.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:49 smithi171 conmon[46715]: debug 2022-01-31T22:48:49.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:49.299847+0000) 2022-01-31T22:48:49.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:49 smithi171 conmon[51620]: debug 2022-01-31T22:48:49.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:49.421923+0000) 2022-01-31T22:48:50.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:49 smithi167 conmon[49112]: debug 2022-01-31T22:48:49.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:49.884202+0000) 2022-01-31T22:48:50.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:50 smithi167 conmon[54076]: debug 2022-01-31T22:48:50.051+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:50.052312+0000) 2022-01-31T22:48:50.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:49 smithi167 conmon[60316]: debug 2022-01-31T22:48:49.811+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:49.812478+0000) 2022-01-31T22:48:50.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:50 smithi171 conmon[41853]: debug 2022-01-31T22:48:50.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:50.249348+0000) 2022-01-31T22:48:50.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:50 smithi171 conmon[46715]: debug 2022-01-31T22:48:50.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:50.299948+0000) 2022-01-31T22:48:50.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:50 smithi171 conmon[51620]: debug 2022-01-31T22:48:50.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:50.422091+0000) 2022-01-31T22:48:51.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:50 smithi167 conmon[49112]: debug 2022-01-31T22:48:50.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:50.884406+0000) 2022-01-31T22:48:51.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:51 smithi167 conmon[54076]: debug 2022-01-31T22:48:51.051+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.052535+0000) 2022-01-31T22:48:51.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:50 smithi167 conmon[60316]: debug 2022-01-31T22:48:50.811+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:50.812694+0000) 2022-01-31T22:48:51.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:51 smithi171 conmon[41853]: debug 2022-01-31T22:48:51.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.249479+0000) 2022-01-31T22:48:51.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:51 smithi171 conmon[46715]: debug 2022-01-31T22:48:51.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.300063+0000) 2022-01-31T22:48:51.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:51 smithi171 conmon[51620]: debug 2022-01-31T22:48:51.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.422282+0000) 2022-01-31T22:48:52.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:51 smithi167 conmon[49112]: debug 2022-01-31T22:48:51.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.884583+0000) 2022-01-31T22:48:52.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:51 smithi167 conmon[49112]: debug 2022-01-31T22:48:51.978+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.980001+0000) 2022-01-31T22:48:52.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:51 smithi167 conmon[54076]: debug 2022-01-31T22:48:51.977+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.979231+0000) 2022-01-31T22:48:52.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:52 smithi167 conmon[54076]: debug 2022-01-31T22:48:52.051+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:52.052678+0000) 2022-01-31T22:48:52.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:51 smithi167 conmon[60316]: debug 2022-01-31T22:48:51.812+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.812879+0000) 2022-01-31T22:48:52.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:51 smithi167 conmon[60316]: debug 2022-01-31T22:48:51.978+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.979497+0000) 2022-01-31T22:48:52.237 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:52 smithi171 conmon[35325]: debug 2022-01-31T22:48:52.003+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239610 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:52.238 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:51 smithi171 conmon[46715]: debug 2022-01-31T22:48:51.978+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.979723+0000) 2022-01-31T22:48:52.238 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:51 smithi171 conmon[51620]: debug 2022-01-31T22:48:51.978+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.979196+0000) 2022-01-31T22:48:52.239 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:51 smithi171 conmon[41853]: debug 2022-01-31T22:48:51.979+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:51.980133+0000) 2022-01-31T22:48:52.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:52 smithi171 conmon[46715]: debug 2022-01-31T22:48:52.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:52.300216+0000) 2022-01-31T22:48:52.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:52 smithi171 conmon[41853]: debug 2022-01-31T22:48:52.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:52.249646+0000) 2022-01-31T22:48:52.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:52 smithi171 conmon[51620]: debug 2022-01-31T22:48:52.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:52.422467+0000) 2022-01-31T22:48:53.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:52 smithi167 conmon[49112]: debug 2022-01-31T22:48:52.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:52.884767+0000) 2022-01-31T22:48:53.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:53 smithi167 conmon[54076]: debug 2022-01-31T22:48:53.052+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:53.052840+0000) 2022-01-31T22:48:53.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:52 smithi167 conmon[60316]: debug 2022-01-31T22:48:52.812+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:52.813043+0000) 2022-01-31T22:48:53.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:53 smithi171 conmon[41853]: debug 2022-01-31T22:48:53.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:53.249824+0000) 2022-01-31T22:48:53.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:53 smithi171 conmon[46715]: debug 2022-01-31T22:48:53.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:53.300359+0000) 2022-01-31T22:48:53.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:53 smithi171 conmon[51620]: debug 2022-01-31T22:48:53.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:53.422629+0000) 2022-01-31T22:48:54.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:53 smithi167 conmon[49112]: debug 2022-01-31T22:48:53.883+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:53.884907+0000) 2022-01-31T22:48:54.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:54 smithi167 conmon[54076]: debug 2022-01-31T22:48:54.051+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:54.052990+0000) 2022-01-31T22:48:54.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:53 smithi167 conmon[60316]: debug 2022-01-31T22:48:53.811+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:53.813185+0000) 2022-01-31T22:48:54.096 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:53 smithi171 conmon[35325]: debug 2022-01-31T22:48:53.914+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:48:54.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:54 smithi171 conmon[41853]: debug 2022-01-31T22:48:54.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:54.249994+0000) 2022-01-31T22:48:54.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:54 smithi171 conmon[46715]: debug 2022-01-31T22:48:54.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:54.300551+0000) 2022-01-31T22:48:54.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:54 smithi171 conmon[51620]: debug 2022-01-31T22:48:54.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:54.422815+0000) 2022-01-31T22:48:55.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:54 smithi167 conmon[49112]: debug 2022-01-31T22:48:54.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:54.885066+0000) 2022-01-31T22:48:55.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:55 smithi167 conmon[54076]: debug 2022-01-31T22:48:55.052+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:55.053212+0000) 2022-01-31T22:48:55.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:54 smithi167 conmon[60316]: debug 2022-01-31T22:48:54.812+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:54.813330+0000) 2022-01-31T22:48:55.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:55 smithi171 conmon[46715]: debug 2022-01-31T22:48:55.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:55.300730+0000) 2022-01-31T22:48:55.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:55 smithi171 conmon[51620]: debug 2022-01-31T22:48:55.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:55.423008+0000) 2022-01-31T22:48:55.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:55 smithi171 conmon[41853]: debug 2022-01-31T22:48:55.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:55.250147+0000) 2022-01-31T22:48:56.094 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:55 smithi167 conmon[49112]: debug 2022-01-31T22:48:55.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:55.885248+0000) 2022-01-31T22:48:56.095 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:55 smithi167 conmon[60316]: debug 2022-01-31T22:48:55.812+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:55.813534+0000) 2022-01-31T22:48:56.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:56 smithi167 conmon[54076]: debug 2022-01-31T22:48:56.052+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:56.053390+0000) 2022-01-31T22:48:56.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:56 smithi171 conmon[41853]: debug 2022-01-31T22:48:56.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:56.250310+0000) 2022-01-31T22:48:56.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:56 smithi171 conmon[46715]: debug 2022-01-31T22:48:56.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:56.300906+0000) 2022-01-31T22:48:56.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:56 smithi171 conmon[51620]: debug 2022-01-31T22:48:56.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:56.423199+0000) 2022-01-31T22:48:57.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:56 smithi167 conmon[49112]: debug 2022-01-31T22:48:56.884+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:56.885424+0000) 2022-01-31T22:48:57.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:57 smithi167 conmon[49112]: debug 2022-01-31T22:48:57.006+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.007615+0000) 2022-01-31T22:48:57.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:57 smithi167 conmon[54076]: debug 2022-01-31T22:48:57.004+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.006395+0000) 2022-01-31T22:48:57.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:57 smithi167 conmon[54076]: debug 2022-01-31T22:48:57.053+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.053613+0000) 2022-01-31T22:48:57.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:56 smithi167 conmon[60316]: debug 2022-01-31T22:48:56.812+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:56.813669+0000) 2022-01-31T22:48:57.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:57 smithi167 conmon[60316]: debug 2022-01-31T22:48:57.006+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.007556+0000) 2022-01-31T22:48:57.284 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[35325]: debug 2022-01-31T22:48:57.030+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239722 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:48:57.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[41853]: debug 2022-01-31T22:48:57.007+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.007824+0000) 2022-01-31T22:48:57.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[41853]: debug 2022-01-31T22:48:57.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.250455+0000) 2022-01-31T22:48:57.285 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[46715]: debug 2022-01-31T22:48:57.006+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.006902+0000) 2022-01-31T22:48:57.286 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[51620]: debug 2022-01-31T22:48:57.006+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.007353+0000) 2022-01-31T22:48:57.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[51620]: debug 2022-01-31T22:48:57.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.423430+0000) 2022-01-31T22:48:57.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:57 smithi171 conmon[46715]: debug 2022-01-31T22:48:57.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.301100+0000) 2022-01-31T22:48:58.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:57 smithi167 conmon[49112]: debug 2022-01-31T22:48:57.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.885579+0000) 2022-01-31T22:48:58.095 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:58 smithi167 conmon[54076]: debug 2022-01-31T22:48:58.052+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:58.053716+0000) 2022-01-31T22:48:58.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:57 smithi167 conmon[60316]: debug 2022-01-31T22:48:57.812+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:57.813792+0000) 2022-01-31T22:48:58.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:58 smithi171 conmon[46715]: debug 2022-01-31T22:48:58.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:58.301240+0000) 2022-01-31T22:48:58.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:58 smithi171 conmon[51620]: debug 2022-01-31T22:48:58.421+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:58.423561+0000) 2022-01-31T22:48:58.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:58 smithi171 conmon[41853]: debug 2022-01-31T22:48:58.248+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:58.250596+0000) 2022-01-31T22:48:59.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:58 smithi167 conmon[49112]: debug 2022-01-31T22:48:58.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:58.885748+0000) 2022-01-31T22:48:59.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:58 smithi167 conmon[60316]: debug 2022-01-31T22:48:58.813+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:58.813945+0000) 2022-01-31T22:48:59.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:48:59 smithi167 conmon[54076]: debug 2022-01-31T22:48:59.053+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:59.053872+0000) 2022-01-31T22:48:59.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:48:59 smithi171 conmon[46715]: debug 2022-01-31T22:48:59.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:59.301413+0000) 2022-01-31T22:48:59.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:48:59 smithi171 conmon[41853]: debug 2022-01-31T22:48:59.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:59.250785+0000) 2022-01-31T22:48:59.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:48:59 smithi171 conmon[51620]: debug 2022-01-31T22:48:59.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:59.423775+0000) 2022-01-31T22:49:00.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:48:59 smithi167 conmon[49112]: debug 2022-01-31T22:48:59.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:59.885966+0000) 2022-01-31T22:49:00.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:00 smithi167 conmon[54076]: debug 2022-01-31T22:49:00.053+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:00.054063+0000) 2022-01-31T22:49:00.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:48:59 smithi167 conmon[60316]: debug 2022-01-31T22:48:59.813+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:48:59.814105+0000) 2022-01-31T22:49:00.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:00 smithi171 conmon[46715]: debug 2022-01-31T22:49:00.299+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:00.301592+0000) 2022-01-31T22:49:00.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:00 smithi171 conmon[41853]: debug 2022-01-31T22:49:00.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:00.250945+0000) 2022-01-31T22:49:00.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:00 smithi171 conmon[51620]: debug 2022-01-31T22:49:00.422+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:00.423940+0000) 2022-01-31T22:49:01.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:00 smithi167 conmon[49112]: debug 2022-01-31T22:49:00.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:00.886149+0000) 2022-01-31T22:49:01.096 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:00 smithi167 conmon[60316]: debug 2022-01-31T22:49:00.813+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:00.814284+0000) 2022-01-31T22:49:01.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:01 smithi167 conmon[54076]: debug 2022-01-31T22:49:01.053+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:01.054245+0000) 2022-01-31T22:49:01.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:01 smithi171 conmon[46715]: debug 2022-01-31T22:49:01.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:01.301800+0000) 2022-01-31T22:49:01.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:01 smithi171 conmon[41853]: debug 2022-01-31T22:49:01.249+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:01.251119+0000) 2022-01-31T22:49:01.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:01 smithi171 conmon[51620]: debug 2022-01-31T22:49:01.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:01.424142+0000) 2022-01-31T22:49:02.026 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:02 smithi167 conmon[54076]: debug 2022-01-31T22:49:02.033+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.034020+0000) 2022-01-31T22:49:02.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:01 smithi167 conmon[49112]: debug 2022-01-31T22:49:01.885+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:01.886352+0000) 2022-01-31T22:49:02.027 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:02 smithi167 conmon[49112]: debug 2022-01-31T22:49:02.034+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.035192+0000) 2022-01-31T22:49:02.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:01 smithi167 conmon[60316]: debug 2022-01-31T22:49:01.814+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:01.814523+0000) 2022-01-31T22:49:02.028 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:02 smithi167 conmon[60316]: debug 2022-01-31T22:49:02.034+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.034586+0000) 2022-01-31T22:49:02.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[41853]: debug 2022-01-31T22:49:02.033+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.034928+0000) 2022-01-31T22:49:02.285 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[41853]: debug 2022-01-31T22:49:02.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.251263+0000) 2022-01-31T22:49:02.286 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[46715]: debug 2022-01-31T22:49:02.032+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.033955+0000) 2022-01-31T22:49:02.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[51620]: debug 2022-01-31T22:49:02.032+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.033858+0000) 2022-01-31T22:49:02.287 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[35325]: debug 2022-01-31T22:49:02.056+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239833 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:02.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:02 smithi167 conmon[54076]: debug 2022-01-31T22:49:02.053+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.054448+0000) 2022-01-31T22:49:02.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[46715]: debug 2022-01-31T22:49:02.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.301946+0000) 2022-01-31T22:49:02.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:02 smithi171 conmon[51620]: debug 2022-01-31T22:49:02.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.424352+0000) 2022-01-31T22:49:03.095 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:02 smithi167 conmon[49112]: debug 2022-01-31T22:49:02.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.886539+0000) 2022-01-31T22:49:03.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:03 smithi167 conmon[54076]: debug 2022-01-31T22:49:03.054+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:03.054587+0000) 2022-01-31T22:49:03.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:02 smithi167 conmon[60316]: debug 2022-01-31T22:49:02.814+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:02.814716+0000) 2022-01-31T22:49:03.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:03 smithi171 conmon[46715]: debug 2022-01-31T22:49:03.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:03.302064+0000) 2022-01-31T22:49:03.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:03 smithi171 conmon[51620]: debug 2022-01-31T22:49:03.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:03.424513+0000) 2022-01-31T22:49:03.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:03 smithi171 conmon[41853]: debug 2022-01-31T22:49:03.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:03.251432+0000) 2022-01-31T22:49:04.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:03 smithi167 conmon[49112]: debug 2022-01-31T22:49:03.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:03.886679+0000) 2022-01-31T22:49:04.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:04 smithi167 conmon[54076]: debug 2022-01-31T22:49:04.054+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:04.054777+0000) 2022-01-31T22:49:04.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:03 smithi167 conmon[60316]: debug 2022-01-31T22:49:03.814+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:03.814876+0000) 2022-01-31T22:49:04.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:04 smithi171 conmon[41853]: debug 2022-01-31T22:49:04.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:04.251578+0000) 2022-01-31T22:49:04.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:04 smithi171 conmon[46715]: debug 2022-01-31T22:49:04.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:04.302220+0000) 2022-01-31T22:49:04.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:04 smithi171 conmon[51620]: debug 2022-01-31T22:49:04.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:04.424704+0000) 2022-01-31T22:49:05.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:04 smithi167 conmon[49112]: debug 2022-01-31T22:49:04.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:04.886883+0000) 2022-01-31T22:49:05.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:05 smithi167 conmon[54076]: debug 2022-01-31T22:49:05.054+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:05.054996+0000) 2022-01-31T22:49:05.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:04 smithi167 conmon[60316]: debug 2022-01-31T22:49:04.814+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:04.815016+0000) 2022-01-31T22:49:05.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:05 smithi171 conmon[46715]: debug 2022-01-31T22:49:05.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:05.302399+0000) 2022-01-31T22:49:05.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:05 smithi171 conmon[41853]: debug 2022-01-31T22:49:05.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:05.251718+0000) 2022-01-31T22:49:05.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:05 smithi171 conmon[51620]: debug 2022-01-31T22:49:05.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:05.424918+0000) 2022-01-31T22:49:06.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:05 smithi167 conmon[49112]: debug 2022-01-31T22:49:05.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:05.887066+0000) 2022-01-31T22:49:06.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:06 smithi167 conmon[54076]: debug 2022-01-31T22:49:06.054+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:06.055181+0000) 2022-01-31T22:49:06.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:05 smithi167 conmon[60316]: debug 2022-01-31T22:49:05.814+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:05.815189+0000) 2022-01-31T22:49:06.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:06 smithi171 conmon[46715]: debug 2022-01-31T22:49:06.300+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:06.302606+0000) 2022-01-31T22:49:06.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:06 smithi171 conmon[41853]: debug 2022-01-31T22:49:06.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:06.251880+0000) 2022-01-31T22:49:06.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:06 smithi171 conmon[51620]: debug 2022-01-31T22:49:06.423+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:06.425105+0000) 2022-01-31T22:49:07.053 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:07 smithi167 conmon[54076]: debug 2022-01-31T22:49:07.054+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.055369+0000) 2022-01-31T22:49:07.053 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:07 smithi167 conmon[54076]: debug 2022-01-31T22:49:07.060+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.061417+0000) 2022-01-31T22:49:07.054 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:06 smithi167 conmon[60316]: debug 2022-01-31T22:49:06.815+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:06.815392+0000) 2022-01-31T22:49:07.054 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:07 smithi167 conmon[60316]: debug 2022-01-31T22:49:07.060+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.061271+0000) 2022-01-31T22:49:07.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:06 smithi167 conmon[49112]: debug 2022-01-31T22:49:06.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:06.887248+0000) 2022-01-31T22:49:07.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:07 smithi167 conmon[49112]: debug 2022-01-31T22:49:07.061+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.062098+0000) 2022-01-31T22:49:07.344 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[35325]: debug 2022-01-31T22:49:07.081+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 239945 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:07.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[46715]: debug 2022-01-31T22:49:07.059+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.061528+0000) 2022-01-31T22:49:07.345 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[46715]: debug 2022-01-31T22:49:07.301+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.302790+0000) 2022-01-31T22:49:07.346 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[51620]: debug 2022-01-31T22:49:07.058+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.060357+0000) 2022-01-31T22:49:07.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[41853]: debug 2022-01-31T22:49:07.060+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.061837+0000) 2022-01-31T22:49:07.347 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[41853]: debug 2022-01-31T22:49:07.250+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.252019+0000) 2022-01-31T22:49:07.704 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:07 smithi171 conmon[51620]: debug 2022-01-31T22:49:07.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.425312+0000) 2022-01-31T22:49:08.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:07 smithi167 conmon[49112]: debug 2022-01-31T22:49:07.886+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.887418+0000) 2022-01-31T22:49:08.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:08 smithi167 conmon[54076]: debug 2022-01-31T22:49:08.055+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:08.055532+0000) 2022-01-31T22:49:08.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:07 smithi167 conmon[60316]: debug 2022-01-31T22:49:07.815+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:07.815563+0000) 2022-01-31T22:49:08.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:08 smithi171 conmon[46715]: debug 2022-01-31T22:49:08.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:08.302909+0000) 2022-01-31T22:49:08.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:08 smithi171 conmon[51620]: debug 2022-01-31T22:49:08.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:08.425498+0000) 2022-01-31T22:49:08.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:08 smithi171 conmon[41853]: debug 2022-01-31T22:49:08.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:08.252147+0000) 2022-01-31T22:49:09.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:08 smithi171 conmon[35325]: debug 2022-01-31T22:49:08.914+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:49:09.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:08 smithi167 conmon[49112]: debug 2022-01-31T22:49:08.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:08.887566+0000) 2022-01-31T22:49:09.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:09 smithi167 conmon[54076]: debug 2022-01-31T22:49:09.055+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:09.055684+0000) 2022-01-31T22:49:09.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:08 smithi167 conmon[60316]: debug 2022-01-31T22:49:08.815+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:08.815725+0000) 2022-01-31T22:49:09.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:09 smithi171 conmon[46715]: debug 2022-01-31T22:49:09.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:09.303102+0000) 2022-01-31T22:49:09.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:09 smithi171 conmon[41853]: debug 2022-01-31T22:49:09.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:09.252320+0000) 2022-01-31T22:49:09.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:09 smithi171 conmon[51620]: debug 2022-01-31T22:49:09.424+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:09.425689+0000) 2022-01-31T22:49:10.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:09 smithi167 conmon[49112]: debug 2022-01-31T22:49:09.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:09.887710+0000) 2022-01-31T22:49:10.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:09 smithi167 conmon[60316]: debug 2022-01-31T22:49:09.815+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:09.815886+0000) 2022-01-31T22:49:10.098 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:10 smithi167 conmon[54076]: debug 2022-01-31T22:49:10.055+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:10.055902+0000) 2022-01-31T22:49:10.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:10 smithi171 conmon[41853]: debug 2022-01-31T22:49:10.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:10.252413+0000) 2022-01-31T22:49:10.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:10 smithi171 conmon[46715]: debug 2022-01-31T22:49:10.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:10.303306+0000) 2022-01-31T22:49:10.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:10 smithi171 conmon[51620]: debug 2022-01-31T22:49:10.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:10.425836+0000) 2022-01-31T22:49:11.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:11 smithi167 conmon[54076]: debug 2022-01-31T22:49:11.055+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:11.056085+0000) 2022-01-31T22:49:11.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:10 smithi167 conmon[60316]: debug 2022-01-31T22:49:10.815+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:10.816036+0000) 2022-01-31T22:49:11.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:10 smithi167 conmon[49112]: debug 2022-01-31T22:49:10.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:10.887823+0000) 2022-01-31T22:49:11.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:11 smithi171 conmon[46715]: debug 2022-01-31T22:49:11.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:11.303543+0000) 2022-01-31T22:49:11.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:11 smithi171 conmon[41853]: debug 2022-01-31T22:49:11.251+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:11.252600+0000) 2022-01-31T22:49:11.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:11 smithi171 conmon[51620]: debug 2022-01-31T22:49:11.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:11.426007+0000) 2022-01-31T22:49:12.072 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:12 smithi167 conmon[54076]: debug 2022-01-31T22:49:12.055+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.056245+0000) 2022-01-31T22:49:12.073 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:11 smithi167 conmon[60316]: debug 2022-01-31T22:49:11.815+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:11.816189+0000) 2022-01-31T22:49:12.073 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:11 smithi167 conmon[49112]: debug 2022-01-31T22:49:11.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:11.888025+0000) 2022-01-31T22:49:12.344 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[35325]: debug 2022-01-31T22:49:12.109+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240058 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:12.345 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[51620]: debug 2022-01-31T22:49:12.084+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.085041+0000) 2022-01-31T22:49:12.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[41853]: debug 2022-01-31T22:49:12.085+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.086483+0000) 2022-01-31T22:49:12.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[41853]: debug 2022-01-31T22:49:12.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.252795+0000) 2022-01-31T22:49:12.346 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[46715]: debug 2022-01-31T22:49:12.084+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.084906+0000) 2022-01-31T22:49:12.347 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[46715]: debug 2022-01-31T22:49:12.302+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.303705+0000) 2022-01-31T22:49:12.406 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:12 smithi167 conmon[49112]: debug 2022-01-31T22:49:12.086+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.086836+0000) 2022-01-31T22:49:12.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:12 smithi167 conmon[54076]: debug 2022-01-31T22:49:12.084+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.085455+0000) 2022-01-31T22:49:12.407 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:12 smithi167 conmon[60316]: debug 2022-01-31T22:49:12.085+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.086244+0000) 2022-01-31T22:49:12.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:12 smithi171 conmon[51620]: debug 2022-01-31T22:49:12.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.426205+0000) 2022-01-31T22:49:13.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:12 smithi167 conmon[49112]: debug 2022-01-31T22:49:12.887+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.888213+0000) 2022-01-31T22:49:13.098 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:13 smithi167 conmon[54076]: debug 2022-01-31T22:49:13.055+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:13.056419+0000) 2022-01-31T22:49:13.099 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:12 smithi167 conmon[60316]: debug 2022-01-31T22:49:12.816+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:12.816375+0000) 2022-01-31T22:49:13.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:13 smithi171 conmon[41853]: debug 2022-01-31T22:49:13.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:13.252940+0000) 2022-01-31T22:49:13.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:13 smithi171 conmon[46715]: debug 2022-01-31T22:49:13.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:13.303851+0000) 2022-01-31T22:49:13.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:13 smithi171 conmon[51620]: debug 2022-01-31T22:49:13.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:13.426390+0000) 2022-01-31T22:49:14.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:13 smithi167 conmon[49112]: debug 2022-01-31T22:49:13.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:13.888371+0000) 2022-01-31T22:49:14.098 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:14 smithi167 conmon[54076]: debug 2022-01-31T22:49:14.056+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:14.056543+0000) 2022-01-31T22:49:14.099 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:13 smithi167 conmon[60316]: debug 2022-01-31T22:49:13.816+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:13.816554+0000) 2022-01-31T22:49:14.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:14 smithi171 conmon[41853]: debug 2022-01-31T22:49:14.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:14.253141+0000) 2022-01-31T22:49:14.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:14 smithi171 conmon[46715]: debug 2022-01-31T22:49:14.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:14.303988+0000) 2022-01-31T22:49:14.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:14 smithi171 conmon[51620]: debug 2022-01-31T22:49:14.425+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:14.426573+0000) 2022-01-31T22:49:15.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:14 smithi167 conmon[49112]: debug 2022-01-31T22:49:14.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:14.888547+0000) 2022-01-31T22:49:15.098 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:15 smithi167 conmon[54076]: debug 2022-01-31T22:49:15.056+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:15.056722+0000) 2022-01-31T22:49:15.099 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:14 smithi167 conmon[60316]: debug 2022-01-31T22:49:14.816+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:14.816726+0000) 2022-01-31T22:49:15.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:15 smithi171 conmon[41853]: debug 2022-01-31T22:49:15.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:15.253300+0000) 2022-01-31T22:49:15.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:15 smithi171 conmon[46715]: debug 2022-01-31T22:49:15.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:15.304166+0000) 2022-01-31T22:49:15.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:15 smithi171 conmon[51620]: debug 2022-01-31T22:49:15.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:15.426788+0000) 2022-01-31T22:49:16.098 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:15 smithi167 conmon[49112]: debug 2022-01-31T22:49:15.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:15.888735+0000) 2022-01-31T22:49:16.099 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:16 smithi167 conmon[54076]: debug 2022-01-31T22:49:16.056+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:16.056924+0000) 2022-01-31T22:49:16.099 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:15 smithi167 conmon[60316]: debug 2022-01-31T22:49:15.816+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:15.816871+0000) 2022-01-31T22:49:16.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:16 smithi171 conmon[41853]: debug 2022-01-31T22:49:16.252+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:16.253511+0000) 2022-01-31T22:49:16.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:16 smithi171 conmon[46715]: debug 2022-01-31T22:49:16.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:16.304313+0000) 2022-01-31T22:49:16.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:16 smithi171 conmon[51620]: debug 2022-01-31T22:49:16.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:16.426937+0000) 2022-01-31T22:49:17.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:16 smithi167 conmon[49112]: debug 2022-01-31T22:49:16.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:16.888893+0000) 2022-01-31T22:49:17.100 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:17 smithi167 conmon[54076]: debug 2022-01-31T22:49:17.056+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.057106+0000) 2022-01-31T22:49:17.101 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:16 smithi167 conmon[60316]: debug 2022-01-31T22:49:16.816+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:16.817051+0000) 2022-01-31T22:49:17.407 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:17 smithi167 conmon[49112]: debug 2022-01-31T22:49:17.112+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.113473+0000) 2022-01-31T22:49:17.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:17 smithi167 conmon[54076]: debug 2022-01-31T22:49:17.112+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.113171+0000) 2022-01-31T22:49:17.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:17 smithi167 conmon[60316]: debug 2022-01-31T22:49:17.112+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.112972+0000) 2022-01-31T22:49:17.414 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[35325]: debug 2022-01-31T22:49:17.143+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240170 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:17.414 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[51620]: debug 2022-01-31T22:49:17.112+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.113455+0000) 2022-01-31T22:49:17.415 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[41853]: debug 2022-01-31T22:49:17.113+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.113863+0000) 2022-01-31T22:49:17.415 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[41853]: debug 2022-01-31T22:49:17.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.253713+0000) 2022-01-31T22:49:17.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[46715]: debug 2022-01-31T22:49:17.112+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.112901+0000) 2022-01-31T22:49:17.416 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[46715]: debug 2022-01-31T22:49:17.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.304484+0000) 2022-01-31T22:49:17.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:17 smithi171 conmon[51620]: debug 2022-01-31T22:49:17.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.427124+0000) 2022-01-31T22:49:18.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:18 smithi167 conmon[54076]: debug 2022-01-31T22:49:18.056+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:18.057281+0000) 2022-01-31T22:49:18.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:17 smithi167 conmon[49112]: debug 2022-01-31T22:49:17.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.889076+0000) 2022-01-31T22:49:18.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:17 smithi167 conmon[60316]: debug 2022-01-31T22:49:17.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:17.817246+0000) 2022-01-31T22:49:18.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:18 smithi171 conmon[41853]: debug 2022-01-31T22:49:18.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:18.253867+0000) 2022-01-31T22:49:18.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:18 smithi171 conmon[46715]: debug 2022-01-31T22:49:18.303+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:18.304659+0000) 2022-01-31T22:49:18.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:18 smithi171 conmon[51620]: debug 2022-01-31T22:49:18.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:18.427294+0000) 2022-01-31T22:49:19.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:19 smithi167 conmon[54076]: debug 2022-01-31T22:49:19.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:19.057430+0000) 2022-01-31T22:49:19.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:18 smithi167 conmon[49112]: debug 2022-01-31T22:49:18.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:18.889225+0000) 2022-01-31T22:49:19.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:18 smithi167 conmon[60316]: debug 2022-01-31T22:49:18.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:18.817375+0000) 2022-01-31T22:49:19.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:19 smithi171 conmon[41853]: debug 2022-01-31T22:49:19.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:19.254056+0000) 2022-01-31T22:49:19.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:19 smithi171 conmon[46715]: debug 2022-01-31T22:49:19.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:19.304861+0000) 2022-01-31T22:49:19.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:19 smithi171 conmon[51620]: debug 2022-01-31T22:49:19.426+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:19.427458+0000) 2022-01-31T22:49:20.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:19 smithi167 conmon[49112]: debug 2022-01-31T22:49:19.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:19.889458+0000) 2022-01-31T22:49:20.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:20 smithi167 conmon[54076]: debug 2022-01-31T22:49:20.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:20.057614+0000) 2022-01-31T22:49:20.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:19 smithi167 conmon[60316]: debug 2022-01-31T22:49:19.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:19.817559+0000) 2022-01-31T22:49:20.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:20 smithi171 conmon[41853]: debug 2022-01-31T22:49:20.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:20.254221+0000) 2022-01-31T22:49:20.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:20 smithi171 conmon[46715]: debug 2022-01-31T22:49:20.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:20.305009+0000) 2022-01-31T22:49:20.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:20 smithi171 conmon[51620]: debug 2022-01-31T22:49:20.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:20.427655+0000) 2022-01-31T22:49:21.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:20 smithi167 conmon[49112]: debug 2022-01-31T22:49:20.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:20.889672+0000) 2022-01-31T22:49:21.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:21 smithi167 conmon[54076]: debug 2022-01-31T22:49:21.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:21.057798+0000) 2022-01-31T22:49:21.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:20 smithi167 conmon[60316]: debug 2022-01-31T22:49:20.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:20.817751+0000) 2022-01-31T22:49:21.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:21 smithi171 conmon[41853]: debug 2022-01-31T22:49:21.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:21.254394+0000) 2022-01-31T22:49:21.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:21 smithi171 conmon[46715]: debug 2022-01-31T22:49:21.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:21.305173+0000) 2022-01-31T22:49:21.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:21 smithi171 conmon[51620]: debug 2022-01-31T22:49:21.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:21.427828+0000) 2022-01-31T22:49:22.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:21 smithi167 conmon[49112]: debug 2022-01-31T22:49:21.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:21.889878+0000) 2022-01-31T22:49:22.134 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:22 smithi167 conmon[54076]: debug 2022-01-31T22:49:22.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.057978+0000) 2022-01-31T22:49:22.135 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:21 smithi167 conmon[60316]: debug 2022-01-31T22:49:21.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:21.817932+0000) 2022-01-31T22:49:22.406 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:22 smithi167 conmon[49112]: debug 2022-01-31T22:49:22.147+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.147667+0000) 2022-01-31T22:49:22.407 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:22 smithi167 conmon[54076]: debug 2022-01-31T22:49:22.146+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.146494+0000) 2022-01-31T22:49:22.408 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:22 smithi167 conmon[60316]: debug 2022-01-31T22:49:22.147+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.147741+0000) 2022-01-31T22:49:22.415 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[35325]: debug 2022-01-31T22:49:22.170+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240282 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:22.415 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[41853]: debug 2022-01-31T22:49:22.147+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.147941+0000) 2022-01-31T22:49:22.416 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[41853]: debug 2022-01-31T22:49:22.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.254566+0000) 2022-01-31T22:49:22.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[51620]: debug 2022-01-31T22:49:22.147+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.148191+0000) 2022-01-31T22:49:22.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[46715]: debug 2022-01-31T22:49:22.145+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.146584+0000) 2022-01-31T22:49:22.417 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[46715]: debug 2022-01-31T22:49:22.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.305333+0000) 2022-01-31T22:49:22.706 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:22 smithi171 conmon[51620]: debug 2022-01-31T22:49:22.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.428022+0000) 2022-01-31T22:49:23.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:22 smithi167 conmon[49112]: debug 2022-01-31T22:49:22.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.890061+0000) 2022-01-31T22:49:23.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:22 smithi167 conmon[60316]: debug 2022-01-31T22:49:22.818+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:22.818114+0000) 2022-01-31T22:49:23.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:23 smithi167 conmon[54076]: debug 2022-01-31T22:49:23.058+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:23.058163+0000) 2022-01-31T22:49:23.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:23 smithi171 conmon[41853]: debug 2022-01-31T22:49:23.253+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:23.254684+0000) 2022-01-31T22:49:23.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:23 smithi171 conmon[46715]: debug 2022-01-31T22:49:23.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:23.305503+0000) 2022-01-31T22:49:23.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:23 smithi171 conmon[51620]: debug 2022-01-31T22:49:23.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:23.428193+0000) 2022-01-31T22:49:24.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:23 smithi171 conmon[35325]: debug 2022-01-31T22:49:23.915+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:49:24.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:23 smithi167 conmon[49112]: debug 2022-01-31T22:49:23.888+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:23.890234+0000) 2022-01-31T22:49:24.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:24 smithi167 conmon[54076]: debug 2022-01-31T22:49:24.056+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:24.058334+0000) 2022-01-31T22:49:24.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:23 smithi167 conmon[60316]: debug 2022-01-31T22:49:23.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:23.818258+0000) 2022-01-31T22:49:24.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:24 smithi171 conmon[41853]: debug 2022-01-31T22:49:24.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:24.254772+0000) 2022-01-31T22:49:24.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:24 smithi171 conmon[51620]: debug 2022-01-31T22:49:24.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:24.428394+0000) 2022-01-31T22:49:24.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:24 smithi171 conmon[46715]: debug 2022-01-31T22:49:24.304+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:24.305714+0000) 2022-01-31T22:49:25.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:24 smithi167 conmon[49112]: debug 2022-01-31T22:49:24.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:24.890414+0000) 2022-01-31T22:49:25.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:25 smithi167 conmon[54076]: debug 2022-01-31T22:49:25.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:25.058575+0000) 2022-01-31T22:49:25.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:24 smithi167 conmon[60316]: debug 2022-01-31T22:49:24.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:24.818462+0000) 2022-01-31T22:49:25.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:25 smithi171 conmon[41853]: debug 2022-01-31T22:49:25.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:25.254916+0000) 2022-01-31T22:49:25.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:25 smithi171 conmon[46715]: debug 2022-01-31T22:49:25.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:25.305882+0000) 2022-01-31T22:49:25.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:25 smithi171 conmon[51620]: debug 2022-01-31T22:49:25.427+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:25.428600+0000) 2022-01-31T22:49:26.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:25 smithi167 conmon[49112]: debug 2022-01-31T22:49:25.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:25.890593+0000) 2022-01-31T22:49:26.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:26 smithi167 conmon[54076]: debug 2022-01-31T22:49:26.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:26.058779+0000) 2022-01-31T22:49:26.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:25 smithi167 conmon[60316]: debug 2022-01-31T22:49:25.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:25.818630+0000) 2022-01-31T22:49:26.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:26 smithi171 conmon[41853]: debug 2022-01-31T22:49:26.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:26.255038+0000) 2022-01-31T22:49:26.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:26 smithi171 conmon[46715]: debug 2022-01-31T22:49:26.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:26.306016+0000) 2022-01-31T22:49:26.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:26 smithi171 conmon[51620]: debug 2022-01-31T22:49:26.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:26.428778+0000) 2022-01-31T22:49:27.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:26 smithi167 conmon[49112]: debug 2022-01-31T22:49:26.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:26.890750+0000) 2022-01-31T22:49:27.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:27 smithi167 conmon[54076]: debug 2022-01-31T22:49:27.057+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.058941+0000) 2022-01-31T22:49:27.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:26 smithi167 conmon[60316]: debug 2022-01-31T22:49:26.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:26.818839+0000) 2022-01-31T22:49:27.416 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[35325]: debug 2022-01-31T22:49:27.197+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240389 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:27.416 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[51620]: debug 2022-01-31T22:49:27.173+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.174711+0000) 2022-01-31T22:49:27.417 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[41853]: debug 2022-01-31T22:49:27.173+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.174450+0000) 2022-01-31T22:49:27.417 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[41853]: debug 2022-01-31T22:49:27.254+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.255252+0000) 2022-01-31T22:49:27.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[46715]: debug 2022-01-31T22:49:27.173+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.174543+0000) 2022-01-31T22:49:27.418 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[46715]: debug 2022-01-31T22:49:27.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.306154+0000) 2022-01-31T22:49:27.539 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:27 smithi167 conmon[49112]: debug 2022-01-31T22:49:27.172+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.174077+0000) 2022-01-31T22:49:27.540 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:27 smithi167 conmon[54076]: debug 2022-01-31T22:49:27.172+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.173892+0000) 2022-01-31T22:49:27.540 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:27 smithi167 conmon[60316]: debug 2022-01-31T22:49:27.172+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.174169+0000) 2022-01-31T22:49:27.707 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:27 smithi171 conmon[51620]: debug 2022-01-31T22:49:27.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.428912+0000) 2022-01-31T22:49:28.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:27 smithi167 conmon[49112]: debug 2022-01-31T22:49:27.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.890940+0000) 2022-01-31T22:49:28.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:28 smithi167 conmon[54076]: debug 2022-01-31T22:49:28.058+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:28.059126+0000) 2022-01-31T22:49:28.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:27 smithi167 conmon[60316]: debug 2022-01-31T22:49:27.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:27.819018+0000) 2022-01-31T22:49:28.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:28 smithi171 conmon[41853]: debug 2022-01-31T22:49:28.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:28.255420+0000) 2022-01-31T22:49:28.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:28 smithi171 conmon[46715]: debug 2022-01-31T22:49:28.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:28.306323+0000) 2022-01-31T22:49:28.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:28 smithi171 conmon[51620]: debug 2022-01-31T22:49:28.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:28.429059+0000) 2022-01-31T22:49:29.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:28 smithi167 conmon[49112]: debug 2022-01-31T22:49:28.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:28.891054+0000) 2022-01-31T22:49:29.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:28 smithi167 conmon[60316]: debug 2022-01-31T22:49:28.818+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:28.819162+0000) 2022-01-31T22:49:29.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:29 smithi167 conmon[54076]: debug 2022-01-31T22:49:29.058+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:29.059279+0000) 2022-01-31T22:49:29.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:29 smithi171 conmon[41853]: debug 2022-01-31T22:49:29.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:29.255614+0000) 2022-01-31T22:49:29.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:29 smithi171 conmon[46715]: debug 2022-01-31T22:49:29.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:29.306510+0000) 2022-01-31T22:49:29.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:29 smithi171 conmon[51620]: debug 2022-01-31T22:49:29.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:29.429245+0000) 2022-01-31T22:49:30.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:29 smithi167 conmon[49112]: debug 2022-01-31T22:49:29.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:29.891203+0000) 2022-01-31T22:49:30.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:30 smithi167 conmon[54076]: debug 2022-01-31T22:49:30.058+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:30.059484+0000) 2022-01-31T22:49:30.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:29 smithi167 conmon[60316]: debug 2022-01-31T22:49:29.817+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:29.819364+0000) 2022-01-31T22:49:30.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:30 smithi171 conmon[41853]: debug 2022-01-31T22:49:30.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:30.255797+0000) 2022-01-31T22:49:30.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:30 smithi171 conmon[46715]: debug 2022-01-31T22:49:30.305+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:30.306717+0000) 2022-01-31T22:49:30.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:30 smithi171 conmon[51620]: debug 2022-01-31T22:49:30.428+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:30.429404+0000) 2022-01-31T22:49:31.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:30 smithi167 conmon[49112]: debug 2022-01-31T22:49:30.889+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:30.891402+0000) 2022-01-31T22:49:31.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:31 smithi167 conmon[54076]: debug 2022-01-31T22:49:31.058+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:31.059709+0000) 2022-01-31T22:49:31.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:30 smithi167 conmon[60316]: debug 2022-01-31T22:49:30.818+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:30.819588+0000) 2022-01-31T22:49:31.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:31 smithi171 conmon[41853]: debug 2022-01-31T22:49:31.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:31.255993+0000) 2022-01-31T22:49:31.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:31 smithi171 conmon[46715]: debug 2022-01-31T22:49:31.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:31.306862+0000) 2022-01-31T22:49:31.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:31 smithi171 conmon[51620]: debug 2022-01-31T22:49:31.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:31.429558+0000) 2022-01-31T22:49:32.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:31 smithi167 conmon[49112]: debug 2022-01-31T22:49:31.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:31.891603+0000) 2022-01-31T22:49:32.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:31 smithi167 conmon[60316]: debug 2022-01-31T22:49:31.818+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:31.819790+0000) 2022-01-31T22:49:32.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:32 smithi167 conmon[54076]: debug 2022-01-31T22:49:32.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.059929+0000) 2022-01-31T22:49:32.540 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:32 smithi167 conmon[49112]: debug 2022-01-31T22:49:32.199+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.201149+0000) 2022-01-31T22:49:32.541 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:32 smithi167 conmon[54076]: debug 2022-01-31T22:49:32.199+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.200841+0000) 2022-01-31T22:49:32.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:32 smithi167 conmon[60316]: debug 2022-01-31T22:49:32.199+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.200729+0000) 2022-01-31T22:49:32.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[35325]: debug 2022-01-31T22:49:32.225+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240502 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:32.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[41853]: debug 2022-01-31T22:49:32.200+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.201496+0000) 2022-01-31T22:49:32.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[41853]: debug 2022-01-31T22:49:32.255+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.256177+0000) 2022-01-31T22:49:32.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[46715]: debug 2022-01-31T22:49:32.199+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.200587+0000) 2022-01-31T22:49:32.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[46715]: debug 2022-01-31T22:49:32.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.306980+0000) 2022-01-31T22:49:32.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[51620]: debug 2022-01-31T22:49:32.201+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.201930+0000) 2022-01-31T22:49:32.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:32 smithi171 conmon[51620]: debug 2022-01-31T22:49:32.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.429741+0000) 2022-01-31T22:49:33.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:32 smithi167 conmon[60316]: debug 2022-01-31T22:49:32.819+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.819929+0000) 2022-01-31T22:49:33.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:32 smithi167 conmon[49112]: debug 2022-01-31T22:49:32.890+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:32.891817+0000) 2022-01-31T22:49:33.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:33 smithi167 conmon[54076]: debug 2022-01-31T22:49:33.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:33.060081+0000) 2022-01-31T22:49:33.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:33 smithi171 conmon[41853]: debug 2022-01-31T22:49:33.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:33.256323+0000) 2022-01-31T22:49:33.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:33 smithi171 conmon[46715]: debug 2022-01-31T22:49:33.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:33.307081+0000) 2022-01-31T22:49:33.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:33 smithi171 conmon[51620]: debug 2022-01-31T22:49:33.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:33.429911+0000) 2022-01-31T22:49:34.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:33 smithi167 conmon[49112]: debug 2022-01-31T22:49:33.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:33.892001+0000) 2022-01-31T22:49:34.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:33 smithi167 conmon[60316]: debug 2022-01-31T22:49:33.819+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:33.820092+0000) 2022-01-31T22:49:34.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:34 smithi167 conmon[54076]: debug 2022-01-31T22:49:34.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:34.060239+0000) 2022-01-31T22:49:34.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:34 smithi171 conmon[41853]: debug 2022-01-31T22:49:34.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:34.256482+0000) 2022-01-31T22:49:34.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:34 smithi171 conmon[46715]: debug 2022-01-31T22:49:34.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:34.307292+0000) 2022-01-31T22:49:34.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:34 smithi171 conmon[51620]: debug 2022-01-31T22:49:34.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:34.430095+0000) 2022-01-31T22:49:35.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:34 smithi167 conmon[49112]: debug 2022-01-31T22:49:34.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:34.892192+0000) 2022-01-31T22:49:35.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:34 smithi167 conmon[60316]: debug 2022-01-31T22:49:34.819+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:34.820284+0000) 2022-01-31T22:49:35.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:35 smithi167 conmon[54076]: debug 2022-01-31T22:49:35.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:35.060441+0000) 2022-01-31T22:49:35.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:35 smithi171 conmon[41853]: debug 2022-01-31T22:49:35.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:35.256669+0000) 2022-01-31T22:49:35.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:35 smithi171 conmon[46715]: debug 2022-01-31T22:49:35.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:35.307475+0000) 2022-01-31T22:49:35.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:35 smithi171 conmon[51620]: debug 2022-01-31T22:49:35.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:35.430231+0000) 2022-01-31T22:49:36.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:35 smithi167 conmon[49112]: debug 2022-01-31T22:49:35.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:35.892348+0000) 2022-01-31T22:49:36.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:36 smithi167 conmon[54076]: debug 2022-01-31T22:49:36.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:36.060566+0000) 2022-01-31T22:49:36.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:35 smithi167 conmon[60316]: debug 2022-01-31T22:49:35.818+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:35.820419+0000) 2022-01-31T22:49:36.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:36 smithi171 conmon[51620]: debug 2022-01-31T22:49:36.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:36.430405+0000) 2022-01-31T22:49:36.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:36 smithi171 conmon[41853]: debug 2022-01-31T22:49:36.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:36.256843+0000) 2022-01-31T22:49:36.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:36 smithi171 conmon[46715]: debug 2022-01-31T22:49:36.306+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:36.307680+0000) 2022-01-31T22:49:37.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:36 smithi167 conmon[49112]: debug 2022-01-31T22:49:36.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:36.892521+0000) 2022-01-31T22:49:37.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:37 smithi167 conmon[54076]: debug 2022-01-31T22:49:37.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.060716+0000) 2022-01-31T22:49:37.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:36 smithi167 conmon[60316]: debug 2022-01-31T22:49:36.819+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:36.820595+0000) 2022-01-31T22:49:37.541 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:37 smithi167 conmon[49112]: debug 2022-01-31T22:49:37.226+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.228156+0000) 2022-01-31T22:49:37.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:37 smithi167 conmon[54076]: debug 2022-01-31T22:49:37.226+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.228380+0000) 2022-01-31T22:49:37.542 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:37 smithi167 conmon[60316]: debug 2022-01-31T22:49:37.227+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.228640+0000) 2022-01-31T22:49:37.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[35325]: debug 2022-01-31T22:49:37.253+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240609 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:37.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[41853]: debug 2022-01-31T22:49:37.229+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.229971+0000) 2022-01-31T22:49:37.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[41853]: debug 2022-01-31T22:49:37.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.256979+0000) 2022-01-31T22:49:37.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[46715]: debug 2022-01-31T22:49:37.228+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.228838+0000) 2022-01-31T22:49:37.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[46715]: debug 2022-01-31T22:49:37.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.307837+0000) 2022-01-31T22:49:37.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[51620]: debug 2022-01-31T22:49:37.228+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.229547+0000) 2022-01-31T22:49:37.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:37 smithi171 conmon[51620]: debug 2022-01-31T22:49:37.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.430572+0000) 2022-01-31T22:49:38.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:37 smithi167 conmon[49112]: debug 2022-01-31T22:49:37.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.892655+0000) 2022-01-31T22:49:38.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:38 smithi167 conmon[54076]: debug 2022-01-31T22:49:38.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:38.060866+0000) 2022-01-31T22:49:38.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:37 smithi167 conmon[60316]: debug 2022-01-31T22:49:37.820+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:37.820803+0000) 2022-01-31T22:49:38.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:38 smithi171 conmon[41853]: debug 2022-01-31T22:49:38.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:38.257129+0000) 2022-01-31T22:49:38.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:38 smithi171 conmon[46715]: debug 2022-01-31T22:49:38.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:38.307972+0000) 2022-01-31T22:49:38.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:38 smithi171 conmon[51620]: debug 2022-01-31T22:49:38.429+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:38.430705+0000) 2022-01-31T22:49:39.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:38 smithi171 conmon[35325]: debug 2022-01-31T22:49:38.916+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:49:39.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:38 smithi167 conmon[49112]: debug 2022-01-31T22:49:38.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:38.892738+0000) 2022-01-31T22:49:39.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:39 smithi167 conmon[54076]: debug 2022-01-31T22:49:39.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:39.061039+0000) 2022-01-31T22:49:39.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:38 smithi167 conmon[60316]: debug 2022-01-31T22:49:38.819+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:38.820975+0000) 2022-01-31T22:49:39.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:39 smithi171 conmon[51620]: debug 2022-01-31T22:49:39.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:39.430903+0000) 2022-01-31T22:49:39.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:39 smithi171 conmon[41853]: debug 2022-01-31T22:49:39.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:39.257309+0000) 2022-01-31T22:49:39.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:39 smithi171 conmon[46715]: debug 2022-01-31T22:49:39.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:39.308181+0000) 2022-01-31T22:49:40.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:39 smithi167 conmon[49112]: debug 2022-01-31T22:49:39.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:39.892936+0000) 2022-01-31T22:49:40.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:40 smithi167 conmon[54076]: debug 2022-01-31T22:49:40.059+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:40.061210+0000) 2022-01-31T22:49:40.159 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:39 smithi167 conmon[60316]: debug 2022-01-31T22:49:39.820+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:39.821142+0000) 2022-01-31T22:49:40.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:40 smithi171 conmon[46715]: debug 2022-01-31T22:49:40.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:40.308366+0000) 2022-01-31T22:49:40.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:40 smithi171 conmon[51620]: debug 2022-01-31T22:49:40.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:40.431100+0000) 2022-01-31T22:49:40.597 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:40 smithi171 conmon[41853]: debug 2022-01-31T22:49:40.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:40.257465+0000) 2022-01-31T22:49:41.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:40 smithi167 conmon[49112]: debug 2022-01-31T22:49:40.891+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:40.893140+0000) 2022-01-31T22:49:41.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:41 smithi167 conmon[54076]: debug 2022-01-31T22:49:41.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:41.061407+0000) 2022-01-31T22:49:41.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:40 smithi167 conmon[60316]: debug 2022-01-31T22:49:40.820+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:40.821376+0000) 2022-01-31T22:49:41.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:41 smithi171 conmon[46715]: debug 2022-01-31T22:49:41.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:41.308531+0000) 2022-01-31T22:49:41.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:41 smithi171 conmon[51620]: debug 2022-01-31T22:49:41.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:41.431314+0000) 2022-01-31T22:49:41.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:41 smithi171 conmon[41853]: debug 2022-01-31T22:49:41.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:41.257647+0000) 2022-01-31T22:49:42.160 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:41 smithi167 conmon[49112]: debug 2022-01-31T22:49:41.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:41.893325+0000) 2022-01-31T22:49:42.161 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:41 smithi167 conmon[60316]: debug 2022-01-31T22:49:41.820+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:41.821546+0000) 2022-01-31T22:49:42.161 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:42 smithi167 conmon[54076]: debug 2022-01-31T22:49:42.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.061609+0000) 2022-01-31T22:49:42.542 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:42 smithi167 conmon[49112]: debug 2022-01-31T22:49:42.254+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.256228+0000) 2022-01-31T22:49:42.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:42 smithi167 conmon[54076]: debug 2022-01-31T22:49:42.255+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.256945+0000) 2022-01-31T22:49:42.543 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:42 smithi167 conmon[60316]: debug 2022-01-31T22:49:42.255+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.257392+0000) 2022-01-31T22:49:42.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[35325]: debug 2022-01-31T22:49:42.281+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240721 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:42.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[41853]: debug 2022-01-31T22:49:42.256+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.257277+0000) 2022-01-31T22:49:42.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[41853]: debug 2022-01-31T22:49:42.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.257855+0000) 2022-01-31T22:49:42.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[46715]: debug 2022-01-31T22:49:42.256+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.256927+0000) 2022-01-31T22:49:42.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[46715]: debug 2022-01-31T22:49:42.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.308653+0000) 2022-01-31T22:49:42.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[51620]: debug 2022-01-31T22:49:42.256+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.257576+0000) 2022-01-31T22:49:42.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:42 smithi171 conmon[51620]: debug 2022-01-31T22:49:42.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.431519+0000) 2022-01-31T22:49:43.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:42 smithi167 conmon[49112]: debug 2022-01-31T22:49:42.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.893543+0000) 2022-01-31T22:49:43.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:43 smithi167 conmon[54076]: debug 2022-01-31T22:49:43.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:43.061767+0000) 2022-01-31T22:49:43.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:42 smithi167 conmon[60316]: debug 2022-01-31T22:49:42.821+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:42.821727+0000) 2022-01-31T22:49:43.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:43 smithi171 conmon[41853]: debug 2022-01-31T22:49:43.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:43.257971+0000) 2022-01-31T22:49:43.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:43 smithi171 conmon[46715]: debug 2022-01-31T22:49:43.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:43.308844+0000) 2022-01-31T22:49:43.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:43 smithi171 conmon[51620]: debug 2022-01-31T22:49:43.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:43.431697+0000) 2022-01-31T22:49:44.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:43 smithi167 conmon[49112]: debug 2022-01-31T22:49:43.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:43.893693+0000) 2022-01-31T22:49:44.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:44 smithi167 conmon[54076]: debug 2022-01-31T22:49:44.061+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:44.061907+0000) 2022-01-31T22:49:44.159 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:43 smithi167 conmon[60316]: debug 2022-01-31T22:49:43.820+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:43.821830+0000) 2022-01-31T22:49:44.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:44 smithi171 conmon[41853]: debug 2022-01-31T22:49:44.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:44.258134+0000) 2022-01-31T22:49:44.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:44 smithi171 conmon[46715]: debug 2022-01-31T22:49:44.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:44.308953+0000) 2022-01-31T22:49:44.502 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:44 smithi171 conmon[51620]: debug 2022-01-31T22:49:44.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:44.431926+0000) 2022-01-31T22:49:45.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:44 smithi167 conmon[49112]: debug 2022-01-31T22:49:44.892+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:44.893849+0000) 2022-01-31T22:49:45.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:45 smithi167 conmon[54076]: debug 2022-01-31T22:49:45.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:45.062116+0000) 2022-01-31T22:49:45.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:44 smithi167 conmon[60316]: debug 2022-01-31T22:49:44.820+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:44.822020+0000) 2022-01-31T22:49:45.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:45 smithi171 conmon[41853]: debug 2022-01-31T22:49:45.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:45.258294+0000) 2022-01-31T22:49:45.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:45 smithi171 conmon[46715]: debug 2022-01-31T22:49:45.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:45.309078+0000) 2022-01-31T22:49:45.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:45 smithi171 conmon[51620]: debug 2022-01-31T22:49:45.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:45.432072+0000) 2022-01-31T22:49:46.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:46 smithi167 conmon[54076]: debug 2022-01-31T22:49:46.061+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:46.062273+0000) 2022-01-31T22:49:46.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:45 smithi167 conmon[60316]: debug 2022-01-31T22:49:45.821+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:45.822182+0000) 2022-01-31T22:49:46.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:45 smithi167 conmon[49112]: debug 2022-01-31T22:49:45.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:45.894050+0000) 2022-01-31T22:49:46.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:46 smithi171 conmon[41853]: debug 2022-01-31T22:49:46.256+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:46.258509+0000) 2022-01-31T22:49:46.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:46 smithi171 conmon[46715]: debug 2022-01-31T22:49:46.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:46.309222+0000) 2022-01-31T22:49:46.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:46 smithi171 conmon[51620]: debug 2022-01-31T22:49:46.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:46.432230+0000) 2022-01-31T22:49:47.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:46 smithi167 conmon[49112]: debug 2022-01-31T22:49:46.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:46.894255+0000) 2022-01-31T22:49:47.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:47 smithi167 conmon[54076]: debug 2022-01-31T22:49:47.060+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.062447+0000) 2022-01-31T22:49:47.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:46 smithi167 conmon[60316]: debug 2022-01-31T22:49:46.821+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:46.822312+0000) 2022-01-31T22:49:47.542 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:47 smithi167 conmon[49112]: debug 2022-01-31T22:49:47.283+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.284954+0000) 2022-01-31T22:49:47.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:47 smithi167 conmon[54076]: debug 2022-01-31T22:49:47.283+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.285441+0000) 2022-01-31T22:49:47.544 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:47 smithi167 conmon[60316]: debug 2022-01-31T22:49:47.284+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.285643+0000) 2022-01-31T22:49:47.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[35325]: debug 2022-01-31T22:49:47.307+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240848 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:47.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[41853]: debug 2022-01-31T22:49:47.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.258672+0000) 2022-01-31T22:49:47.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[41853]: debug 2022-01-31T22:49:47.283+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.285446+0000) 2022-01-31T22:49:47.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[46715]: debug 2022-01-31T22:49:47.283+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.284933+0000) 2022-01-31T22:49:47.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[46715]: debug 2022-01-31T22:49:47.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.309375+0000) 2022-01-31T22:49:47.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[51620]: debug 2022-01-31T22:49:47.283+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.285307+0000) 2022-01-31T22:49:47.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:47 smithi171 conmon[51620]: debug 2022-01-31T22:49:47.430+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.432394+0000) 2022-01-31T22:49:48.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:47 smithi167 conmon[60316]: debug 2022-01-31T22:49:47.821+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.822430+0000) 2022-01-31T22:49:48.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:47 smithi167 conmon[49112]: debug 2022-01-31T22:49:47.893+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:47.894443+0000) 2022-01-31T22:49:48.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:48 smithi167 conmon[54076]: debug 2022-01-31T22:49:48.062+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:48.062553+0000) 2022-01-31T22:49:48.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:48 smithi171 conmon[46715]: debug 2022-01-31T22:49:48.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:48.309483+0000) 2022-01-31T22:49:48.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:48 smithi171 conmon[51620]: debug 2022-01-31T22:49:48.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:48.432580+0000) 2022-01-31T22:49:48.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:48 smithi171 conmon[41853]: debug 2022-01-31T22:49:48.257+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:48.258810+0000) 2022-01-31T22:49:49.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:48 smithi167 conmon[60316]: debug 2022-01-31T22:49:48.822+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:48.822586+0000) 2022-01-31T22:49:49.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:48 smithi167 conmon[49112]: debug 2022-01-31T22:49:48.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:48.894604+0000) 2022-01-31T22:49:49.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:49 smithi167 conmon[54076]: debug 2022-01-31T22:49:49.062+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:49.062724+0000) 2022-01-31T22:49:49.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:49 smithi171 conmon[41853]: debug 2022-01-31T22:49:49.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:49.258939+0000) 2022-01-31T22:49:49.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:49 smithi171 conmon[46715]: debug 2022-01-31T22:49:49.307+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:49.309640+0000) 2022-01-31T22:49:49.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:49 smithi171 conmon[51620]: debug 2022-01-31T22:49:49.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:49.432790+0000) 2022-01-31T22:49:50.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:49 smithi167 conmon[49112]: debug 2022-01-31T22:49:49.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:49.894827+0000) 2022-01-31T22:49:50.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:50 smithi167 conmon[54076]: debug 2022-01-31T22:49:50.062+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:50.062905+0000) 2022-01-31T22:49:50.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:49 smithi167 conmon[60316]: debug 2022-01-31T22:49:49.822+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:49.822775+0000) 2022-01-31T22:49:50.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:50 smithi171 conmon[41853]: debug 2022-01-31T22:49:50.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:50.259070+0000) 2022-01-31T22:49:50.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:50 smithi171 conmon[46715]: debug 2022-01-31T22:49:50.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:50.309799+0000) 2022-01-31T22:49:50.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:50 smithi171 conmon[51620]: debug 2022-01-31T22:49:50.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:50.432957+0000) 2022-01-31T22:49:51.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:50 smithi167 conmon[60316]: debug 2022-01-31T22:49:50.822+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:50.822936+0000) 2022-01-31T22:49:51.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:50 smithi167 conmon[49112]: debug 2022-01-31T22:49:50.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:50.894983+0000) 2022-01-31T22:49:51.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:51 smithi167 conmon[54076]: debug 2022-01-31T22:49:51.062+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:51.063098+0000) 2022-01-31T22:49:51.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:51 smithi171 conmon[46715]: debug 2022-01-31T22:49:51.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:51.309932+0000) 2022-01-31T22:49:51.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:51 smithi171 conmon[51620]: debug 2022-01-31T22:49:51.431+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:51.433136+0000) 2022-01-31T22:49:51.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:51 smithi171 conmon[41853]: debug 2022-01-31T22:49:51.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:51.259211+0000) 2022-01-31T22:49:52.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:51 smithi167 conmon[60316]: debug 2022-01-31T22:49:51.822+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:51.823112+0000) 2022-01-31T22:49:52.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:51 smithi167 conmon[49112]: debug 2022-01-31T22:49:51.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:51.895131+0000) 2022-01-31T22:49:52.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:52 smithi167 conmon[54076]: debug 2022-01-31T22:49:52.062+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.063270+0000) 2022-01-31T22:49:52.543 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:52 smithi167 conmon[49112]: debug 2022-01-31T22:49:52.312+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.312525+0000) 2022-01-31T22:49:52.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:52 smithi167 conmon[54076]: debug 2022-01-31T22:49:52.311+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.312382+0000) 2022-01-31T22:49:52.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:52 smithi167 conmon[60316]: debug 2022-01-31T22:49:52.312+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.312959+0000) 2022-01-31T22:49:52.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[46715]: debug 2022-01-31T22:49:52.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.310089+0000) 2022-01-31T22:49:52.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[46715]: debug 2022-01-31T22:49:52.310+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.312555+0000) 2022-01-31T22:49:52.595 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[35325]: debug 2022-01-31T22:49:52.341+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 240961 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:52.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[41853]: debug 2022-01-31T22:49:52.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.259423+0000) 2022-01-31T22:49:52.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[41853]: debug 2022-01-31T22:49:52.310+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.312039+0000) 2022-01-31T22:49:52.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[51620]: debug 2022-01-31T22:49:52.311+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.312902+0000) 2022-01-31T22:49:52.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:52 smithi171 conmon[51620]: debug 2022-01-31T22:49:52.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.433305+0000) 2022-01-31T22:49:53.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:52 smithi167 conmon[49112]: debug 2022-01-31T22:49:52.894+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.895312+0000) 2022-01-31T22:49:53.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:53 smithi167 conmon[54076]: debug 2022-01-31T22:49:53.063+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:53.063459+0000) 2022-01-31T22:49:53.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:52 smithi167 conmon[60316]: debug 2022-01-31T22:49:52.822+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:52.823295+0000) 2022-01-31T22:49:53.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:53 smithi171 conmon[41853]: debug 2022-01-31T22:49:53.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:53.259605+0000) 2022-01-31T22:49:53.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:53 smithi171 conmon[46715]: debug 2022-01-31T22:49:53.308+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:53.310296+0000) 2022-01-31T22:49:53.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:53 smithi171 conmon[51620]: debug 2022-01-31T22:49:53.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:53.433496+0000) 2022-01-31T22:49:54.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:53 smithi171 conmon[35325]: debug 2022-01-31T22:49:53.916+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:49:54.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:53 smithi167 conmon[49112]: debug 2022-01-31T22:49:53.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:53.895526+0000) 2022-01-31T22:49:54.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:54 smithi167 conmon[54076]: debug 2022-01-31T22:49:54.063+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:54.063572+0000) 2022-01-31T22:49:54.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:53 smithi167 conmon[60316]: debug 2022-01-31T22:49:53.823+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:53.823486+0000) 2022-01-31T22:49:54.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:54 smithi171 conmon[41853]: debug 2022-01-31T22:49:54.258+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:54.259767+0000) 2022-01-31T22:49:54.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:54 smithi171 conmon[46715]: debug 2022-01-31T22:49:54.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:54.310433+0000) 2022-01-31T22:49:54.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:54 smithi171 conmon[51620]: debug 2022-01-31T22:49:54.432+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:54.433702+0000) 2022-01-31T22:49:55.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:54 smithi167 conmon[49112]: debug 2022-01-31T22:49:54.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:54.895680+0000) 2022-01-31T22:49:55.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:55 smithi167 conmon[54076]: debug 2022-01-31T22:49:55.063+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:55.063698+0000) 2022-01-31T22:49:55.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:54 smithi167 conmon[60316]: debug 2022-01-31T22:49:54.823+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:54.823656+0000) 2022-01-31T22:49:55.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:55 smithi171 conmon[41853]: debug 2022-01-31T22:49:55.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:55.259899+0000) 2022-01-31T22:49:55.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:55 smithi171 conmon[46715]: debug 2022-01-31T22:49:55.309+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:55.310640+0000) 2022-01-31T22:49:55.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:55 smithi171 conmon[51620]: debug 2022-01-31T22:49:55.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:55.433899+0000) 2022-01-31T22:49:56.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:55 smithi167 conmon[49112]: debug 2022-01-31T22:49:55.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:55.895843+0000) 2022-01-31T22:49:56.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:56 smithi167 conmon[54076]: debug 2022-01-31T22:49:56.063+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:56.063885+0000) 2022-01-31T22:49:56.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:55 smithi167 conmon[60316]: debug 2022-01-31T22:49:55.823+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:55.823846+0000) 2022-01-31T22:49:56.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:56 smithi171 conmon[46715]: debug 2022-01-31T22:49:56.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:56.310829+0000) 2022-01-31T22:49:56.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:56 smithi171 conmon[51620]: debug 2022-01-31T22:49:56.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:56.434069+0000) 2022-01-31T22:49:56.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:56 smithi171 conmon[41853]: debug 2022-01-31T22:49:56.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:56.260033+0000) 2022-01-31T22:49:57.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:56 smithi167 conmon[49112]: debug 2022-01-31T22:49:56.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:56.896040+0000) 2022-01-31T22:49:57.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:57 smithi167 conmon[54076]: debug 2022-01-31T22:49:57.063+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.064066+0000) 2022-01-31T22:49:57.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:56 smithi167 conmon[60316]: debug 2022-01-31T22:49:56.823+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:56.824025+0000) 2022-01-31T22:49:57.544 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:57 smithi167 conmon[49112]: debug 2022-01-31T22:49:57.346+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.346633+0000) 2022-01-31T22:49:57.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:57 smithi167 conmon[54076]: debug 2022-01-31T22:49:57.345+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.345589+0000) 2022-01-31T22:49:57.545 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:57 smithi167 conmon[60316]: debug 2022-01-31T22:49:57.346+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.346724+0000) 2022-01-31T22:49:57.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[35325]: debug 2022-01-31T22:49:57.369+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241072 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:49:57.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[41853]: debug 2022-01-31T22:49:57.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.260187+0000) 2022-01-31T22:49:57.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[41853]: debug 2022-01-31T22:49:57.344+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.345590+0000) 2022-01-31T22:49:57.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[46715]: debug 2022-01-31T22:49:57.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.311027+0000) 2022-01-31T22:49:57.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[46715]: debug 2022-01-31T22:49:57.346+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.347497+0000) 2022-01-31T22:49:57.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[51620]: debug 2022-01-31T22:49:57.345+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.346173+0000) 2022-01-31T22:49:57.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:57 smithi171 conmon[51620]: debug 2022-01-31T22:49:57.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.434214+0000) 2022-01-31T22:49:58.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:57 smithi167 conmon[49112]: debug 2022-01-31T22:49:57.895+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.896195+0000) 2022-01-31T22:49:58.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:57 smithi167 conmon[60316]: debug 2022-01-31T22:49:57.823+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:57.824166+0000) 2022-01-31T22:49:58.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:58 smithi167 conmon[54076]: debug 2022-01-31T22:49:58.063+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:58.064196+0000) 2022-01-31T22:49:58.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:58 smithi171 conmon[41853]: debug 2022-01-31T22:49:58.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:58.260390+0000) 2022-01-31T22:49:58.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:58 smithi171 conmon[46715]: debug 2022-01-31T22:49:58.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:58.311155+0000) 2022-01-31T22:49:58.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:58 smithi171 conmon[51620]: debug 2022-01-31T22:49:58.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:58.434368+0000) 2022-01-31T22:49:59.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:58 smithi167 conmon[49112]: debug 2022-01-31T22:49:58.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:58.896352+0000) 2022-01-31T22:49:59.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:49:59 smithi167 conmon[54076]: debug 2022-01-31T22:49:59.064+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:59.064345+0000) 2022-01-31T22:49:59.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:58 smithi167 conmon[60316]: debug 2022-01-31T22:49:58.823+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:58.824315+0000) 2022-01-31T22:49:59.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:49:59 smithi171 conmon[46715]: debug 2022-01-31T22:49:59.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:59.311321+0000) 2022-01-31T22:49:59.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:49:59 smithi171 conmon[51620]: debug 2022-01-31T22:49:59.433+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:59.434564+0000) 2022-01-31T22:49:59.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:49:59 smithi171 conmon[41853]: debug 2022-01-31T22:49:59.259+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:59.260572+0000) 2022-01-31T22:50:00.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:49:59 smithi167 conmon[49112]: debug 2022-01-31T22:49:59.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:59.896555+0000) 2022-01-31T22:50:00.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:00 smithi167 conmon[54076]: debug 2022-01-31T22:50:00.064+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:00.064531+0000) 2022-01-31T22:50:00.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:49:59 smithi167 conmon[60316]: debug 2022-01-31T22:49:59.824+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:49:59.824476+0000) 2022-01-31T22:50:00.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:00 smithi171 conmon[41853]: debug 2022-01-31T22:50:00.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:00.260768+0000) 2022-01-31T22:50:00.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:00 smithi171 conmon[46715]: debug 2022-01-31T22:50:00.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:00.311487+0000) 2022-01-31T22:50:00.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:00 smithi171 conmon[51620]: debug 2022-01-31T22:50:00.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:00.434783+0000) 2022-01-31T22:50:01.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:00 smithi167 conmon[49112]: debug 2022-01-31T22:50:00.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:00.896736+0000) 2022-01-31T22:50:01.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:01 smithi167 conmon[54076]: debug 2022-01-31T22:50:01.064+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:01.064699+0000) 2022-01-31T22:50:01.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:00 smithi167 conmon[60316]: debug 2022-01-31T22:50:01.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:00 smithi167 conmon[60316]: 2022-01-31T22:50:00.824+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:00.824680+0000) 2022-01-31T22:50:01.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:01 smithi171 conmon[46715]: debug 2022-01-31T22:50:01.310+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:01.311638+0000) 2022-01-31T22:50:01.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:01 smithi171 conmon[51620]: debug 2022-01-31T22:50:01.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:01.434977+0000) 2022-01-31T22:50:01.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:01 smithi171 conmon[41853]: debug 2022-01-31T22:50:01.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:01.260953+0000) 2022-01-31T22:50:02.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:01 smithi167 conmon[60316]: debug 2022-01-31T22:50:01.824+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:01.824867+0000) 2022-01-31T22:50:02.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:01 smithi167 conmon[49112]: debug 2022-01-31T22:50:01.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:01.896897+0000) 2022-01-31T22:50:02.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:02 smithi167 conmon[54076]: debug 2022-01-31T22:50:02.064+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.064861+0000) 2022-01-31T22:50:02.545 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:02 smithi167 conmon[49112]: debug 2022-01-31T22:50:02.373+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.373519+0000) 2022-01-31T22:50:02.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:02 smithi167 conmon[54076]: debug 2022-01-31T22:50:02.372+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.373087+0000) 2022-01-31T22:50:02.547 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:02 smithi167 conmon[60316]: debug 2022-01-31T22:50:02.373+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.373763+0000) 2022-01-31T22:50:02.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[35325]: debug 2022-01-31T22:50:02.397+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241184 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:02.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[46715]: debug 2022-01-31T22:50:02.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.311834+0000) 2022-01-31T22:50:02.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[46715]: debug 2022-01-31T22:50:02.373+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.374259+0000) 2022-01-31T22:50:02.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[41853]: debug 2022-01-31T22:50:02.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.261142+0000) 2022-01-31T22:50:02.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[41853]: debug 2022-01-31T22:50:02.372+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.372767+0000) 2022-01-31T22:50:02.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[51620]: debug 2022-01-31T22:50:02.372+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.373240+0000) 2022-01-31T22:50:02.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:02 smithi171 conmon[51620]: debug 2022-01-31T22:50:02.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.435189+0000) 2022-01-31T22:50:03.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:02 smithi167 conmon[49112]: debug 2022-01-31T22:50:02.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.897083+0000) 2022-01-31T22:50:03.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:03 smithi167 conmon[54076]: debug 2022-01-31T22:50:03.064+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:03.065016+0000) 2022-01-31T22:50:03.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:02 smithi167 conmon[60316]: debug 2022-01-31T22:50:02.824+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:02.825021+0000) 2022-01-31T22:50:03.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:03 smithi171 conmon[41853]: debug 2022-01-31T22:50:03.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:03.261301+0000) 2022-01-31T22:50:03.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:03 smithi171 conmon[46715]: debug 2022-01-31T22:50:03.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:03.311941+0000) 2022-01-31T22:50:03.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:03 smithi171 conmon[51620]: debug 2022-01-31T22:50:03.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:03.435359+0000) 2022-01-31T22:50:04.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:03 smithi167 conmon[49112]: debug 2022-01-31T22:50:03.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:03.897228+0000) 2022-01-31T22:50:04.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:04 smithi167 conmon[54076]: debug 2022-01-31T22:50:04.064+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:04.065180+0000) 2022-01-31T22:50:04.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:03 smithi167 conmon[60316]: debug 2022-01-31T22:50:03.824+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:03.825169+0000) 2022-01-31T22:50:04.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:04 smithi171 conmon[46715]: debug 2022-01-31T22:50:04.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:04.312044+0000) 2022-01-31T22:50:04.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:04 smithi171 conmon[51620]: debug 2022-01-31T22:50:04.434+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:04.435578+0000) 2022-01-31T22:50:04.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:04 smithi171 conmon[41853]: debug 2022-01-31T22:50:04.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:04.261396+0000) 2022-01-31T22:50:05.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:04 smithi167 conmon[49112]: debug 2022-01-31T22:50:04.896+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:04.897415+0000) 2022-01-31T22:50:05.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:04 smithi167 conmon[60316]: debug 2022-01-31T22:50:04.824+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:04.825370+0000) 2022-01-31T22:50:05.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:05 smithi167 conmon[54076]: debug 2022-01-31T22:50:05.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:05.065379+0000) 2022-01-31T22:50:05.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:05 smithi171 conmon[46715]: debug 2022-01-31T22:50:05.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:05.312166+0000) 2022-01-31T22:50:05.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:05 smithi171 conmon[51620]: debug 2022-01-31T22:50:05.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:05.435787+0000) 2022-01-31T22:50:05.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:05 smithi171 conmon[41853]: debug 2022-01-31T22:50:05.260+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:05.261556+0000) 2022-01-31T22:50:06.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:05 smithi167 conmon[49112]: debug 2022-01-31T22:50:05.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:05.897582+0000) 2022-01-31T22:50:06.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:06 smithi167 conmon[54076]: debug 2022-01-31T22:50:06.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:06.065575+0000) 2022-01-31T22:50:06.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:05 smithi167 conmon[60316]: debug 2022-01-31T22:50:05.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:05.825575+0000) 2022-01-31T22:50:06.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:06 smithi171 conmon[41853]: debug 2022-01-31T22:50:06.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:06.261715+0000) 2022-01-31T22:50:06.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:06 smithi171 conmon[46715]: debug 2022-01-31T22:50:06.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:06.312303+0000) 2022-01-31T22:50:06.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:06 smithi171 conmon[51620]: debug 2022-01-31T22:50:06.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:06.436008+0000) 2022-01-31T22:50:07.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:06 smithi167 conmon[49112]: debug 2022-01-31T22:50:06.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:06.897730+0000) 2022-01-31T22:50:07.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:07 smithi167 conmon[54076]: debug 2022-01-31T22:50:07.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.065749+0000) 2022-01-31T22:50:07.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:06 smithi167 conmon[60316]: debug 2022-01-31T22:50:06.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:06.825727+0000) 2022-01-31T22:50:07.546 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:07 smithi167 conmon[49112]: debug 2022-01-31T22:50:07.400+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.401186+0000) 2022-01-31T22:50:07.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:07 smithi167 conmon[54076]: debug 2022-01-31T22:50:07.401+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.401925+0000) 2022-01-31T22:50:07.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:07 smithi167 conmon[60316]: debug 2022-01-31T22:50:07.401+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.401884+0000) 2022-01-31T22:50:07.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[35325]: debug 2022-01-31T22:50:07.424+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241296 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:07.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[41853]: debug 2022-01-31T22:50:07.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.261873+0000) 2022-01-31T22:50:07.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[41853]: debug 2022-01-31T22:50:07.399+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.400001+0000) 2022-01-31T22:50:07.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[46715]: debug 2022-01-31T22:50:07.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.312515+0000) 2022-01-31T22:50:07.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[46715]: debug 2022-01-31T22:50:07.400+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.400952+0000) 2022-01-31T22:50:07.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[51620]: debug 2022-01-31T22:50:07.399+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.400120+0000) 2022-01-31T22:50:07.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:07 smithi171 conmon[51620]: debug 2022-01-31T22:50:07.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.436139+0000) 2022-01-31T22:50:08.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:07 smithi167 conmon[60316]: debug 2022-01-31T22:50:07.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.825872+0000) 2022-01-31T22:50:08.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:07 smithi167 conmon[49112]: debug 2022-01-31T22:50:07.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:07.897917+0000) 2022-01-31T22:50:08.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:08 smithi167 conmon[54076]: debug 2022-01-31T22:50:08.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:08.065941+0000) 2022-01-31T22:50:08.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:08 smithi171 conmon[41853]: debug 2022-01-31T22:50:08.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:08.262059+0000) 2022-01-31T22:50:08.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:08 smithi171 conmon[46715]: debug 2022-01-31T22:50:08.311+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:08.312729+0000) 2022-01-31T22:50:08.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:08 smithi171 conmon[51620]: debug 2022-01-31T22:50:08.435+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:08.436312+0000) 2022-01-31T22:50:09.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:08 smithi171 conmon[35325]: debug 2022-01-31T22:50:08.917+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:50:09.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:08 smithi167 conmon[49112]: debug 2022-01-31T22:50:08.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:08.898101+0000) 2022-01-31T22:50:09.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:09 smithi167 conmon[54076]: debug 2022-01-31T22:50:09.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:09.066095+0000) 2022-01-31T22:50:09.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:08 smithi167 conmon[60316]: debug 2022-01-31T22:50:08.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:08.826050+0000) 2022-01-31T22:50:09.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:09 smithi171 conmon[41853]: debug 2022-01-31T22:50:09.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:09.262234+0000) 2022-01-31T22:50:09.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:09 smithi171 conmon[46715]: debug 2022-01-31T22:50:09.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:09.312861+0000) 2022-01-31T22:50:09.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:09 smithi171 conmon[51620]: debug 2022-01-31T22:50:09.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:09.436496+0000) 2022-01-31T22:50:10.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:09 smithi167 conmon[49112]: debug 2022-01-31T22:50:09.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:09.898286+0000) 2022-01-31T22:50:10.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:10 smithi167 conmon[54076]: debug 2022-01-31T22:50:10.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:10.066267+0000) 2022-01-31T22:50:10.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:09 smithi167 conmon[60316]: debug 2022-01-31T22:50:09.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:09.826233+0000) 2022-01-31T22:50:10.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:10 smithi171 conmon[41853]: debug 2022-01-31T22:50:10.261+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:10.262407+0000) 2022-01-31T22:50:10.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:10 smithi171 conmon[46715]: debug 2022-01-31T22:50:10.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:10.312989+0000) 2022-01-31T22:50:10.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:10 smithi171 conmon[51620]: debug 2022-01-31T22:50:10.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:10.436676+0000) 2022-01-31T22:50:11.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:10 smithi167 conmon[49112]: debug 2022-01-31T22:50:10.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:10.898446+0000) 2022-01-31T22:50:11.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:11 smithi167 conmon[54076]: debug 2022-01-31T22:50:11.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:11.066434+0000) 2022-01-31T22:50:11.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:10 smithi167 conmon[60316]: debug 2022-01-31T22:50:10.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:10.826396+0000) 2022-01-31T22:50:11.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:11 smithi171 conmon[41853]: debug 2022-01-31T22:50:11.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:11.262596+0000) 2022-01-31T22:50:11.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:11 smithi171 conmon[46715]: debug 2022-01-31T22:50:11.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:11.313143+0000) 2022-01-31T22:50:11.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:11 smithi171 conmon[51620]: debug 2022-01-31T22:50:11.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:11.436852+0000) 2022-01-31T22:50:12.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:11 smithi167 conmon[49112]: debug 2022-01-31T22:50:11.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:11.898603+0000) 2022-01-31T22:50:12.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:12 smithi167 conmon[54076]: debug 2022-01-31T22:50:12.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.066620+0000) 2022-01-31T22:50:12.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:11 smithi167 conmon[60316]: debug 2022-01-31T22:50:11.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:11.826559+0000) 2022-01-31T22:50:12.547 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:12 smithi167 conmon[49112]: debug 2022-01-31T22:50:12.426+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.428129+0000) 2022-01-31T22:50:12.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:12 smithi167 conmon[54076]: debug 2022-01-31T22:50:12.428+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.429593+0000) 2022-01-31T22:50:12.548 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:12 smithi167 conmon[60316]: debug 2022-01-31T22:50:12.427+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.428928+0000) 2022-01-31T22:50:12.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[35325]: debug 2022-01-31T22:50:12.451+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241409 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:12.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[51620]: debug 2022-01-31T22:50:12.427+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.427762+0000) 2022-01-31T22:50:12.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[51620]: debug 2022-01-31T22:50:12.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.437003+0000) 2022-01-31T22:50:12.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[41853]: debug 2022-01-31T22:50:12.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.262793+0000) 2022-01-31T22:50:12.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[41853]: debug 2022-01-31T22:50:12.426+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.427462+0000) 2022-01-31T22:50:12.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[46715]: debug 2022-01-31T22:50:12.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.313343+0000) 2022-01-31T22:50:12.597 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:12 smithi171 conmon[46715]: debug 2022-01-31T22:50:12.427+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.428133+0000) 2022-01-31T22:50:13.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:12 smithi167 conmon[49112]: debug 2022-01-31T22:50:12.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.898755+0000) 2022-01-31T22:50:13.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:13 smithi167 conmon[54076]: debug 2022-01-31T22:50:13.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:13.066737+0000) 2022-01-31T22:50:13.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:12 smithi167 conmon[60316]: debug 2022-01-31T22:50:12.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:12.826681+0000) 2022-01-31T22:50:13.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:13 smithi171 conmon[46715]: debug 2022-01-31T22:50:13.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:13.313476+0000) 2022-01-31T22:50:13.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:13 smithi171 conmon[51620]: debug 2022-01-31T22:50:13.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:13.437161+0000) 2022-01-31T22:50:13.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:13 smithi171 conmon[41853]: debug 2022-01-31T22:50:13.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:13.262981+0000) 2022-01-31T22:50:14.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:13 smithi167 conmon[49112]: debug 2022-01-31T22:50:13.897+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:13.898923+0000) 2022-01-31T22:50:14.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:14 smithi167 conmon[54076]: debug 2022-01-31T22:50:14.065+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:14.066910+0000) 2022-01-31T22:50:14.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:13 smithi167 conmon[60316]: debug 2022-01-31T22:50:13.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:13.826806+0000) 2022-01-31T22:50:14.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:14 smithi171 conmon[41853]: debug 2022-01-31T22:50:14.262+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:14.263187+0000) 2022-01-31T22:50:14.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:14 smithi171 conmon[46715]: debug 2022-01-31T22:50:14.312+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:14.313668+0000) 2022-01-31T22:50:14.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:14 smithi171 conmon[51620]: debug 2022-01-31T22:50:14.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:14.437357+0000) 2022-01-31T22:50:15.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:14 smithi167 conmon[49112]: debug 2022-01-31T22:50:14.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:14.899134+0000) 2022-01-31T22:50:15.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:15 smithi167 conmon[54076]: debug 2022-01-31T22:50:15.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:15.067134+0000) 2022-01-31T22:50:15.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:14 smithi167 conmon[60316]: debug 2022-01-31T22:50:14.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:14.827006+0000) 2022-01-31T22:50:15.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:15 smithi171 conmon[46715]: debug 2022-01-31T22:50:15.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:15.313817+0000) 2022-01-31T22:50:15.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:15 smithi171 conmon[41853]: debug 2022-01-31T22:50:15.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:15.263378+0000) 2022-01-31T22:50:15.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:15 smithi171 conmon[51620]: debug 2022-01-31T22:50:15.436+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:15.437585+0000) 2022-01-31T22:50:16.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:15 smithi167 conmon[49112]: debug 2022-01-31T22:50:15.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:15.899321+0000) 2022-01-31T22:50:16.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:16 smithi167 conmon[54076]: debug 2022-01-31T22:50:16.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:16.067339+0000) 2022-01-31T22:50:16.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:15 smithi167 conmon[60316]: debug 2022-01-31T22:50:15.825+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:15.827166+0000) 2022-01-31T22:50:16.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:16 smithi171 conmon[46715]: debug 2022-01-31T22:50:16.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:16.313942+0000) 2022-01-31T22:50:16.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:16 smithi171 conmon[41853]: debug 2022-01-31T22:50:16.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:16.263565+0000) 2022-01-31T22:50:16.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:16 smithi171 conmon[51620]: debug 2022-01-31T22:50:16.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:16.437805+0000) 2022-01-31T22:50:17.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:16 smithi167 conmon[49112]: debug 2022-01-31T22:50:16.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:16.899518+0000) 2022-01-31T22:50:17.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:16 smithi167 conmon[60316]: debug 2022-01-31T22:50:16.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:16.827323+0000) 2022-01-31T22:50:17.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:17 smithi167 conmon[54076]: debug 2022-01-31T22:50:17.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.067473+0000) 2022-01-31T22:50:17.548 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:17 smithi167 conmon[49112]: debug 2022-01-31T22:50:17.452+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.454440+0000) 2022-01-31T22:50:17.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:17 smithi167 conmon[54076]: debug 2022-01-31T22:50:17.460+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.462021+0000) 2022-01-31T22:50:17.549 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:17 smithi167 conmon[60316]: debug 2022-01-31T22:50:17.460+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.461759+0000) 2022-01-31T22:50:17.594 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[35325]: debug 2022-01-31T22:50:17.494+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241520 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:17.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[41853]: debug 2022-01-31T22:50:17.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.263772+0000) 2022-01-31T22:50:17.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[41853]: debug 2022-01-31T22:50:17.454+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.455006+0000) 2022-01-31T22:50:17.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[46715]: debug 2022-01-31T22:50:17.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.314120+0000) 2022-01-31T22:50:17.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[46715]: debug 2022-01-31T22:50:17.455+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.455889+0000) 2022-01-31T22:50:17.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[51620]: debug 2022-01-31T22:50:17.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.437976+0000) 2022-01-31T22:50:17.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:17 smithi171 conmon[51620]: debug 2022-01-31T22:50:17.453+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.454723+0000) 2022-01-31T22:50:18.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:17 smithi167 conmon[60316]: debug 2022-01-31T22:50:17.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.827532+0000) 2022-01-31T22:50:18.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:17 smithi167 conmon[49112]: debug 2022-01-31T22:50:17.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:17.899699+0000) 2022-01-31T22:50:18.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:18 smithi167 conmon[54076]: debug 2022-01-31T22:50:18.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:18.067610+0000) 2022-01-31T22:50:18.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:18 smithi171 conmon[41853]: debug 2022-01-31T22:50:18.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:18.263926+0000) 2022-01-31T22:50:18.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:18 smithi171 conmon[46715]: debug 2022-01-31T22:50:18.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:18.314280+0000) 2022-01-31T22:50:18.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:18 smithi171 conmon[51620]: debug 2022-01-31T22:50:18.437+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:18.438150+0000) 2022-01-31T22:50:19.054 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:18 smithi167 conmon[49112]: debug 2022-01-31T22:50:18.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:18.899876+0000) 2022-01-31T22:50:19.055 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:18 smithi167 conmon[60316]: debug 2022-01-31T22:50:18.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:18.827682+0000) 2022-01-31T22:50:19.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:19 smithi167 conmon[54076]: debug 2022-01-31T22:50:19.066+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:19.067793+0000) 2022-01-31T22:50:19.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:19 smithi171 conmon[41853]: debug 2022-01-31T22:50:19.263+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:19.264105+0000) 2022-01-31T22:50:19.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:19 smithi171 conmon[46715]: debug 2022-01-31T22:50:19.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:19.314452+0000) 2022-01-31T22:50:19.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:19 smithi171 conmon[51620]: debug 2022-01-31T22:50:19.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:19.438300+0000) 2022-01-31T22:50:20.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:19 smithi167 conmon[49112]: debug 2022-01-31T22:50:19.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:19.900074+0000) 2022-01-31T22:50:20.055 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:19 smithi167 conmon[60316]: debug 2022-01-31T22:50:19.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:19.827871+0000) 2022-01-31T22:50:20.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:20 smithi167 conmon[54076]: debug 2022-01-31T22:50:20.067+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:20.067978+0000) 2022-01-31T22:50:20.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:20 smithi171 conmon[41853]: debug 2022-01-31T22:50:20.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:20.264259+0000) 2022-01-31T22:50:20.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:20 smithi171 conmon[46715]: debug 2022-01-31T22:50:20.313+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:20.314634+0000) 2022-01-31T22:50:20.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:20 smithi171 conmon[51620]: debug 2022-01-31T22:50:20.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:20.438510+0000) 2022-01-31T22:50:21.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:20 smithi167 conmon[49112]: debug 2022-01-31T22:50:20.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:20.900265+0000) 2022-01-31T22:50:21.055 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:20 smithi167 conmon[60316]: debug 2022-01-31T22:50:20.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:20.828080+0000) 2022-01-31T22:50:21.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:21 smithi167 conmon[54076]: debug 2022-01-31T22:50:21.067+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:21.068172+0000) 2022-01-31T22:50:21.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:21 smithi171 conmon[41853]: debug 2022-01-31T22:50:21.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:21.264446+0000) 2022-01-31T22:50:21.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:21 smithi171 conmon[51620]: debug 2022-01-31T22:50:21.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:21.438687+0000) 2022-01-31T22:50:21.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:21 smithi171 conmon[46715]: debug 2022-01-31T22:50:21.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:21.314834+0000) 2022-01-31T22:50:22.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:21 smithi167 conmon[49112]: debug 2022-01-31T22:50:21.898+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:21.900468+0000) 2022-01-31T22:50:22.056 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:21 smithi167 conmon[60316]: debug 2022-01-31T22:50:21.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:21.828288+0000) 2022-01-31T22:50:22.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:22 smithi167 conmon[54076]: debug 2022-01-31T22:50:22.067+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.068354+0000) 2022-01-31T22:50:22.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[41853]: debug 2022-01-31T22:50:22.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.264644+0000) 2022-01-31T22:50:22.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[41853]: debug 2022-01-31T22:50:22.497+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.498042+0000) 2022-01-31T22:50:22.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[46715]: debug 2022-01-31T22:50:22.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.314991+0000) 2022-01-31T22:50:22.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[46715]: debug 2022-01-31T22:50:22.497+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.498214+0000) 2022-01-31T22:50:22.511 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[51620]: debug 2022-01-31T22:50:22.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.438862+0000) 2022-01-31T22:50:22.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[51620]: debug 2022-01-31T22:50:22.496+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.497658+0000) 2022-01-31T22:50:22.787 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:22 smithi167 conmon[49112]: debug 2022-01-31T22:50:22.496+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.497774+0000) 2022-01-31T22:50:22.788 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:22 smithi167 conmon[60316]: debug 2022-01-31T22:50:22.497+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.499034+0000) 2022-01-31T22:50:22.788 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:22 smithi167 conmon[54076]: debug 2022-01-31T22:50:22.496+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.497905+0000) 2022-01-31T22:50:22.844 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:22 smithi171 conmon[35325]: debug 2022-01-31T22:50:22.522+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241634 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:23.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:22 smithi167 conmon[49112]: debug 2022-01-31T22:50:22.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.900695+0000) 2022-01-31T22:50:23.056 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:22 smithi167 conmon[60316]: debug 2022-01-31T22:50:22.826+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:22.828475+0000) 2022-01-31T22:50:23.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:23 smithi167 conmon[54076]: debug 2022-01-31T22:50:23.067+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:23.068542+0000) 2022-01-31T22:50:23.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:23 smithi171 conmon[41853]: debug 2022-01-31T22:50:23.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:23.264833+0000) 2022-01-31T22:50:23.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:23 smithi171 conmon[46715]: debug 2022-01-31T22:50:23.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:23.315207+0000) 2022-01-31T22:50:23.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:23 smithi171 conmon[51620]: debug 2022-01-31T22:50:23.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:23.439077+0000) 2022-01-31T22:50:24.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:23 smithi167 conmon[49112]: debug 2022-01-31T22:50:23.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:23.900854+0000) 2022-01-31T22:50:24.056 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:23 smithi167 conmon[60316]: debug 2022-01-31T22:50:23.827+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:23.828583+0000) 2022-01-31T22:50:24.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:23 smithi171 conmon[35325]: debug 2022-01-31T22:50:23.918+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:50:24.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:24 smithi167 conmon[54076]: debug 2022-01-31T22:50:24.067+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:24.068667+0000) 2022-01-31T22:50:24.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:24 smithi171 conmon[41853]: debug 2022-01-31T22:50:24.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:24.265002+0000) 2022-01-31T22:50:24.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:24 smithi171 conmon[46715]: debug 2022-01-31T22:50:24.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:24.315387+0000) 2022-01-31T22:50:24.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:24 smithi171 conmon[51620]: debug 2022-01-31T22:50:24.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:24.439209+0000) 2022-01-31T22:50:25.055 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:24 smithi167 conmon[49112]: debug 2022-01-31T22:50:24.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:24.900988+0000) 2022-01-31T22:50:25.056 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:24 smithi167 conmon[60316]: debug 2022-01-31T22:50:24.827+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:24.828767+0000) 2022-01-31T22:50:25.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:25 smithi167 conmon[54076]: debug 2022-01-31T22:50:25.068+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:25.068823+0000) 2022-01-31T22:50:25.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:25 smithi171 conmon[41853]: debug 2022-01-31T22:50:25.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:25.265210+0000) 2022-01-31T22:50:25.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:25 smithi171 conmon[46715]: debug 2022-01-31T22:50:25.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:25.315709+0000) 2022-01-31T22:50:25.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:25 smithi171 conmon[51620]: debug 2022-01-31T22:50:25.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:25.439396+0000) 2022-01-31T22:50:26.056 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:25 smithi167 conmon[49112]: debug 2022-01-31T22:50:25.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:25.901114+0000) 2022-01-31T22:50:26.056 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:25 smithi167 conmon[60316]: debug 2022-01-31T22:50:25.827+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:25.828975+0000) 2022-01-31T22:50:26.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:26 smithi167 conmon[54076]: debug 2022-01-31T22:50:26.067+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:26.069014+0000) 2022-01-31T22:50:26.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:26 smithi171 conmon[46715]: debug 2022-01-31T22:50:26.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:26.315888+0000) 2022-01-31T22:50:26.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:26 smithi171 conmon[41853]: debug 2022-01-31T22:50:26.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:26.265412+0000) 2022-01-31T22:50:26.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:26 smithi171 conmon[51620]: debug 2022-01-31T22:50:26.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:26.439596+0000) 2022-01-31T22:50:27.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:26 smithi167 conmon[60316]: debug 2022-01-31T22:50:26.828+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:26.829181+0000) 2022-01-31T22:50:27.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:26 smithi167 conmon[49112]: debug 2022-01-31T22:50:26.899+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:26.901289+0000) 2022-01-31T22:50:27.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:27 smithi167 conmon[54076]: debug 2022-01-31T22:50:27.068+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.069201+0000) 2022-01-31T22:50:27.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[46715]: debug 2022-01-31T22:50:27.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.316042+0000) 2022-01-31T22:50:27.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[51620]: debug 2022-01-31T22:50:27.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.439762+0000) 2022-01-31T22:50:27.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[41853]: debug 2022-01-31T22:50:27.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.265589+0000) 2022-01-31T22:50:27.513 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:27 smithi167 conmon[49112]: debug 2022-01-31T22:50:27.524+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.525945+0000) 2022-01-31T22:50:27.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:27 smithi167 conmon[54076]: debug 2022-01-31T22:50:27.523+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.525216+0000) 2022-01-31T22:50:27.514 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:27 smithi167 conmon[60316]: debug 2022-01-31T22:50:27.524+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.526151+0000) 2022-01-31T22:50:27.844 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[35325]: debug 2022-01-31T22:50:27.549+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241744 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:27.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[46715]: debug 2022-01-31T22:50:27.525+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.526145+0000) 2022-01-31T22:50:27.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[51620]: debug 2022-01-31T22:50:27.524+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.524839+0000) 2022-01-31T22:50:27.846 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:27 smithi171 conmon[41853]: debug 2022-01-31T22:50:27.525+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.526732+0000) 2022-01-31T22:50:28.056 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:27 smithi167 conmon[49112]: debug 2022-01-31T22:50:27.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.901490+0000) 2022-01-31T22:50:28.056 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:27 smithi167 conmon[60316]: debug 2022-01-31T22:50:27.828+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:27.829337+0000) 2022-01-31T22:50:28.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:28 smithi167 conmon[54076]: debug 2022-01-31T22:50:28.068+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:28.069387+0000) 2022-01-31T22:50:28.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:28 smithi171 conmon[41853]: debug 2022-01-31T22:50:28.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:28.265804+0000) 2022-01-31T22:50:28.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:28 smithi171 conmon[46715]: debug 2022-01-31T22:50:28.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:28.316189+0000) 2022-01-31T22:50:28.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:28 smithi171 conmon[51620]: debug 2022-01-31T22:50:28.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:28.439965+0000) 2022-01-31T22:50:29.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:28 smithi167 conmon[49112]: debug 2022-01-31T22:50:28.900+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:28.901608+0000) 2022-01-31T22:50:29.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:28 smithi167 conmon[60316]: debug 2022-01-31T22:50:28.828+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:28.829524+0000) 2022-01-31T22:50:29.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:29 smithi167 conmon[54076]: debug 2022-01-31T22:50:29.068+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:29.069599+0000) 2022-01-31T22:50:29.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:29 smithi171 conmon[41853]: debug 2022-01-31T22:50:29.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:29.265969+0000) 2022-01-31T22:50:29.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:29 smithi171 conmon[46715]: debug 2022-01-31T22:50:29.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:29.316301+0000) 2022-01-31T22:50:29.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:29 smithi171 conmon[51620]: debug 2022-01-31T22:50:29.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:29.440083+0000) 2022-01-31T22:50:30.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:29 smithi167 conmon[49112]: debug 2022-01-31T22:50:29.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:29.901804+0000) 2022-01-31T22:50:30.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:30 smithi167 conmon[54076]: debug 2022-01-31T22:50:30.069+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:30.069764+0000) 2022-01-31T22:50:30.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:29 smithi167 conmon[60316]: debug 2022-01-31T22:50:29.829+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:29.829685+0000) 2022-01-31T22:50:30.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:30 smithi171 conmon[51620]: debug 2022-01-31T22:50:30.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:30.440300+0000) 2022-01-31T22:50:30.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:30 smithi171 conmon[41853]: debug 2022-01-31T22:50:30.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:30.266138+0000) 2022-01-31T22:50:30.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:30 smithi171 conmon[46715]: debug 2022-01-31T22:50:30.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:30.316498+0000) 2022-01-31T22:50:31.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:30 smithi167 conmon[49112]: debug 2022-01-31T22:50:30.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:30.901994+0000) 2022-01-31T22:50:31.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:31 smithi167 conmon[54076]: debug 2022-01-31T22:50:31.069+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:31.069967+0000) 2022-01-31T22:50:31.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:30 smithi167 conmon[60316]: debug 2022-01-31T22:50:30.829+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:30.829849+0000) 2022-01-31T22:50:31.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:31 smithi171 conmon[51620]: debug 2022-01-31T22:50:31.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:31.440490+0000) 2022-01-31T22:50:31.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:31 smithi171 conmon[41853]: debug 2022-01-31T22:50:31.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:31.266318+0000) 2022-01-31T22:50:31.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:31 smithi171 conmon[46715]: debug 2022-01-31T22:50:31.314+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:31.316674+0000) 2022-01-31T22:50:32.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:31 smithi167 conmon[49112]: debug 2022-01-31T22:50:31.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:31.902152+0000) 2022-01-31T22:50:32.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:31 smithi167 conmon[60316]: debug 2022-01-31T22:50:31.829+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:31.830017+0000) 2022-01-31T22:50:32.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:32 smithi167 conmon[54076]: debug 2022-01-31T22:50:32.069+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.070155+0000) 2022-01-31T22:50:32.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[41853]: debug 2022-01-31T22:50:32.264+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.266532+0000) 2022-01-31T22:50:32.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[46715]: debug 2022-01-31T22:50:32.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.316838+0000) 2022-01-31T22:50:32.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[51620]: debug 2022-01-31T22:50:32.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.440612+0000) 2022-01-31T22:50:32.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:32 smithi167 conmon[49112]: debug 2022-01-31T22:50:32.551+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.553398+0000) 2022-01-31T22:50:32.818 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:32 smithi167 conmon[54076]: debug 2022-01-31T22:50:32.551+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.553344+0000) 2022-01-31T22:50:32.818 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:32 smithi167 conmon[60316]: debug 2022-01-31T22:50:32.552+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.553603+0000) 2022-01-31T22:50:32.844 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[41853]: debug 2022-01-31T22:50:32.552+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.554539+0000) 2022-01-31T22:50:32.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[46715]: debug 2022-01-31T22:50:32.552+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.554006+0000) 2022-01-31T22:50:32.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[51620]: debug 2022-01-31T22:50:32.550+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.552447+0000) 2022-01-31T22:50:32.845 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:32 smithi171 conmon[35325]: debug 2022-01-31T22:50:32.576+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241858 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:33.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:32 smithi167 conmon[49112]: debug 2022-01-31T22:50:32.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.902331+0000) 2022-01-31T22:50:33.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:33 smithi167 conmon[54076]: debug 2022-01-31T22:50:33.068+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:33.070310+0000) 2022-01-31T22:50:33.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:32 smithi167 conmon[60316]: debug 2022-01-31T22:50:32.829+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:32.830176+0000) 2022-01-31T22:50:33.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:33 smithi171 conmon[41853]: debug 2022-01-31T22:50:33.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:33.266725+0000) 2022-01-31T22:50:33.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:33 smithi171 conmon[46715]: debug 2022-01-31T22:50:33.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:33.316972+0000) 2022-01-31T22:50:33.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:33 smithi171 conmon[51620]: debug 2022-01-31T22:50:33.438+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:33.440732+0000) 2022-01-31T22:50:34.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:34 smithi167 conmon[54076]: debug 2022-01-31T22:50:34.069+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:34.070509+0000) 2022-01-31T22:50:34.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:33 smithi167 conmon[49112]: debug 2022-01-31T22:50:33.901+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:33.902522+0000) 2022-01-31T22:50:34.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:33 smithi167 conmon[60316]: debug 2022-01-31T22:50:33.828+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:33.830347+0000) 2022-01-31T22:50:34.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:34 smithi171 conmon[51620]: debug 2022-01-31T22:50:34.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:34.440871+0000) 2022-01-31T22:50:34.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:34 smithi171 conmon[41853]: debug 2022-01-31T22:50:34.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:34.266940+0000) 2022-01-31T22:50:34.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:34 smithi171 conmon[46715]: debug 2022-01-31T22:50:34.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:34.317097+0000) 2022-01-31T22:50:35.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:34 smithi167 conmon[49112]: debug 2022-01-31T22:50:34.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:34.902710+0000) 2022-01-31T22:50:35.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:35 smithi167 conmon[54076]: debug 2022-01-31T22:50:35.069+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:35.070655+0000) 2022-01-31T22:50:35.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:34 smithi167 conmon[60316]: debug 2022-01-31T22:50:34.829+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:34.830538+0000) 2022-01-31T22:50:35.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:35 smithi171 conmon[41853]: debug 2022-01-31T22:50:35.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:35.267094+0000) 2022-01-31T22:50:35.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:35 smithi171 conmon[46715]: debug 2022-01-31T22:50:35.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:35.317234+0000) 2022-01-31T22:50:35.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:35 smithi171 conmon[51620]: debug 2022-01-31T22:50:35.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:35.441021+0000) 2022-01-31T22:50:36.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:35 smithi167 conmon[49112]: debug 2022-01-31T22:50:35.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:35.902917+0000) 2022-01-31T22:50:36.209 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:36 smithi167 conmon[54076]: debug 2022-01-31T22:50:36.070+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:36.070830+0000) 2022-01-31T22:50:36.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:35 smithi167 conmon[60316]: debug 2022-01-31T22:50:35.829+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:35.830662+0000) 2022-01-31T22:50:36.598 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:36 smithi171 conmon[41853]: debug 2022-01-31T22:50:36.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:36.267276+0000) 2022-01-31T22:50:36.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:36 smithi171 conmon[46715]: debug 2022-01-31T22:50:36.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:36.317404+0000) 2022-01-31T22:50:36.599 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:36 smithi171 conmon[51620]: debug 2022-01-31T22:50:36.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:36.441226+0000) 2022-01-31T22:50:37.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:36 smithi167 conmon[49112]: debug 2022-01-31T22:50:36.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:36.903073+0000) 2022-01-31T22:50:37.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[54076]: debug 2022-01-31T22:50:37.070+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.071006+0000) 2022-01-31T22:50:37.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:36 smithi167 conmon[60316]: debug 2022-01-31T22:50:36.830+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:36.830816+0000) 2022-01-31T22:50:37.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[46715]: debug 2022-01-31T22:50:37.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.317553+0000) 2022-01-31T22:50:37.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[51620]: debug 2022-01-31T22:50:37.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.441381+0000) 2022-01-31T22:50:37.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[41853]: debug 2022-01-31T22:50:37.265+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.267398+0000) 2022-01-31T22:50:37.817 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[49112]: debug 2022-01-31T22:50:37.579+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.580398+0000) 2022-01-31T22:50:37.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[49112]: 2022-01-31T22:50:37.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[54076]: debug 2022-01-31T22:50:37.581+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.581919+0000) 2022-01-31T22:50:37.819 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[60316]: debug 2022-01-31T22:50:37.580+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.580710+0000) 2022-01-31T22:50:37.844 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[41853]: debug 2022-01-31T22:50:37.579+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.581334+0000) 2022-01-31T22:50:37.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[46715]: debug 2022-01-31T22:50:37.578+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.580102+0000) 2022-01-31T22:50:37.846 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[51620]: debug 2022-01-31T22:50:37.579+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.580926+0000) 2022-01-31T22:50:37.847 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:37 smithi171 conmon[35325]: debug 2022-01-31T22:50:37.603+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 241969 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:38.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[49112]: debug 2022-01-31T22:50:37.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.903276+0000) 2022-01-31T22:50:38.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:38 smithi167 conmon[54076]: debug 2022-01-31T22:50:38.070+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:38.071158+0000) 2022-01-31T22:50:38.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:37 smithi167 conmon[60316]: debug 2022-01-31T22:50:37.830+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:37.830987+0000) 2022-01-31T22:50:38.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:38 smithi171 conmon[46715]: debug 2022-01-31T22:50:38.315+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:38.317729+0000) 2022-01-31T22:50:38.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:38 smithi171 conmon[51620]: debug 2022-01-31T22:50:38.440+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:38.441581+0000) 2022-01-31T22:50:38.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:38 smithi171 conmon[41853]: debug 2022-01-31T22:50:38.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:38.267585+0000) 2022-01-31T22:50:39.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:38 smithi171 conmon[35325]: debug 2022-01-31T22:50:38.918+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:50:39.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:38 smithi167 conmon[49112]: debug 2022-01-31T22:50:38.902+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:38.903441+0000) 2022-01-31T22:50:39.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:39 smithi167 conmon[54076]: debug 2022-01-31T22:50:39.070+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:39.071323+0000) 2022-01-31T22:50:39.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:38 smithi167 conmon[60316]: debug 2022-01-31T22:50:38.830+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:38.831134+0000) 2022-01-31T22:50:39.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:39 smithi171 conmon[46715]: debug 2022-01-31T22:50:39.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:39.317919+0000) 2022-01-31T22:50:39.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:39 smithi171 conmon[41853]: debug 2022-01-31T22:50:39.266+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:39.267771+0000) 2022-01-31T22:50:39.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:39 smithi171 conmon[51620]: debug 2022-01-31T22:50:39.439+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:39.441716+0000) 2022-01-31T22:50:40.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:39 smithi167 conmon[60316]: debug 2022-01-31T22:50:39.830+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:39.831300+0000) 2022-01-31T22:50:40.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:39 smithi167 conmon[49112]: debug 2022-01-31T22:50:39.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:39.903661+0000) 2022-01-31T22:50:40.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:40 smithi167 conmon[54076]: debug 2022-01-31T22:50:40.071+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:40.071525+0000) 2022-01-31T22:50:40.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:40 smithi171 conmon[41853]: debug 2022-01-31T22:50:40.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:40.267920+0000) 2022-01-31T22:50:40.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:40 smithi171 conmon[46715]: debug 2022-01-31T22:50:40.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:40.318067+0000) 2022-01-31T22:50:40.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:40 smithi171 conmon[51620]: debug 2022-01-31T22:50:40.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:40.441916+0000) 2022-01-31T22:50:41.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:40 smithi167 conmon[49112]: debug 2022-01-31T22:50:40.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:40.903861+0000) 2022-01-31T22:50:41.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:41 smithi167 conmon[54076]: debug 2022-01-31T22:50:41.071+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:41.071718+0000) 2022-01-31T22:50:41.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:40 smithi167 conmon[60316]: debug 2022-01-31T22:50:40.830+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:40.831460+0000) 2022-01-31T22:50:41.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:41 smithi171 conmon[41853]: debug 2022-01-31T22:50:41.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:41.268095+0000) 2022-01-31T22:50:41.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:41 smithi171 conmon[46715]: debug 2022-01-31T22:50:41.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:41.318247+0000) 2022-01-31T22:50:41.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:41 smithi171 conmon[51620]: debug 2022-01-31T22:50:41.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:41.442103+0000) 2022-01-31T22:50:42.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:41 smithi167 conmon[49112]: debug 2022-01-31T22:50:41.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:41.904027+0000) 2022-01-31T22:50:42.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:42 smithi167 conmon[54076]: debug 2022-01-31T22:50:42.071+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.071900+0000) 2022-01-31T22:50:42.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:41 smithi167 conmon[60316]: debug 2022-01-31T22:50:41.831+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:41.831665+0000) 2022-01-31T22:50:42.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[46715]: debug 2022-01-31T22:50:42.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.318448+0000) 2022-01-31T22:50:42.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[51620]: debug 2022-01-31T22:50:42.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.442271+0000) 2022-01-31T22:50:42.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[41853]: debug 2022-01-31T22:50:42.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.268244+0000) 2022-01-31T22:50:42.818 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:42 smithi167 conmon[49112]: debug 2022-01-31T22:50:42.608+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.608717+0000) 2022-01-31T22:50:42.819 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:42 smithi167 conmon[54076]: debug 2022-01-31T22:50:42.608+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.608984+0000) 2022-01-31T22:50:42.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:42 smithi167 conmon[60316]: debug 2022-01-31T22:50:42.607+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.607718+0000) 2022-01-31T22:50:42.844 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[35325]: debug 2022-01-31T22:50:42.632+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242082 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:42.844 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[41853]: debug 2022-01-31T22:50:42.608+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.609169+0000) 2022-01-31T22:50:42.845 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[46715]: debug 2022-01-31T22:50:42.606+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.607139+0000) 2022-01-31T22:50:42.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:42 smithi171 conmon[51620]: debug 2022-01-31T22:50:42.607+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.608329+0000) 2022-01-31T22:50:43.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:42 smithi167 conmon[49112]: debug 2022-01-31T22:50:42.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.904190+0000) 2022-01-31T22:50:43.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:43 smithi167 conmon[54076]: debug 2022-01-31T22:50:43.071+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:43.072087+0000) 2022-01-31T22:50:43.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:42 smithi167 conmon[60316]: debug 2022-01-31T22:50:42.831+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:42.831835+0000) 2022-01-31T22:50:43.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:43 smithi171 conmon[51620]: debug 2022-01-31T22:50:43.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:43.442450+0000) 2022-01-31T22:50:43.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:43 smithi171 conmon[41853]: debug 2022-01-31T22:50:43.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:43.268433+0000) 2022-01-31T22:50:43.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:43 smithi171 conmon[46715]: debug 2022-01-31T22:50:43.317+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:43.318636+0000) 2022-01-31T22:50:44.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:43 smithi167 conmon[49112]: debug 2022-01-31T22:50:43.903+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:43.904353+0000) 2022-01-31T22:50:44.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:44 smithi167 conmon[54076]: debug 2022-01-31T22:50:44.071+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:44.072283+0000) 2022-01-31T22:50:44.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:43 smithi167 conmon[60316]: debug 2022-01-31T22:50:43.831+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:43.831987+0000) 2022-01-31T22:50:44.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:44 smithi171 conmon[41853]: debug 2022-01-31T22:50:44.267+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:44.268569+0000) 2022-01-31T22:50:44.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:44 smithi171 conmon[46715]: debug 2022-01-31T22:50:44.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:44.318820+0000) 2022-01-31T22:50:44.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:44 smithi171 conmon[51620]: debug 2022-01-31T22:50:44.441+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:44.442607+0000) 2022-01-31T22:50:45.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:44 smithi167 conmon[49112]: debug 2022-01-31T22:50:44.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:44.904543+0000) 2022-01-31T22:50:45.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:45 smithi167 conmon[54076]: debug 2022-01-31T22:50:45.072+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:45.072530+0000) 2022-01-31T22:50:45.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:44 smithi167 conmon[60316]: debug 2022-01-31T22:50:44.831+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:44.832206+0000) 2022-01-31T22:50:45.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:45 smithi171 conmon[41853]: debug 2022-01-31T22:50:45.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:45.268782+0000) 2022-01-31T22:50:45.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:45 smithi171 conmon[46715]: debug 2022-01-31T22:50:45.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:45.318940+0000) 2022-01-31T22:50:45.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:45 smithi171 conmon[51620]: debug 2022-01-31T22:50:45.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:45.442818+0000) 2022-01-31T22:50:46.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:45 smithi167 conmon[49112]: debug 2022-01-31T22:50:45.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:45.904735+0000) 2022-01-31T22:50:46.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:46 smithi167 conmon[54076]: debug 2022-01-31T22:50:46.072+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:46.072694+0000) 2022-01-31T22:50:46.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:45 smithi167 conmon[60316]: debug 2022-01-31T22:50:45.832+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:45.832359+0000) 2022-01-31T22:50:46.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:46 smithi171 conmon[41853]: debug 2022-01-31T22:50:46.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:46.268929+0000) 2022-01-31T22:50:46.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:46 smithi171 conmon[46715]: debug 2022-01-31T22:50:46.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:46.319091+0000) 2022-01-31T22:50:46.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:46 smithi171 conmon[51620]: debug 2022-01-31T22:50:46.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:46.442960+0000) 2022-01-31T22:50:47.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:46 smithi167 conmon[49112]: debug 2022-01-31T22:50:46.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:46.904994+0000) 2022-01-31T22:50:47.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:47 smithi167 conmon[54076]: debug 2022-01-31T22:50:47.072+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.072883+0000) 2022-01-31T22:50:47.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:46 smithi167 conmon[60316]: debug 2022-01-31T22:50:46.832+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:46.832561+0000) 2022-01-31T22:50:47.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[41853]: debug 2022-01-31T22:50:47.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.269162+0000) 2022-01-31T22:50:47.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[46715]: debug 2022-01-31T22:50:47.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.319354+0000) 2022-01-31T22:50:47.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[51620]: debug 2022-01-31T22:50:47.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.443228+0000) 2022-01-31T22:50:47.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:47 smithi167 conmon[49112]: debug 2022-01-31T22:50:47.635+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.636418+0000) 2022-01-31T22:50:47.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:47 smithi167 conmon[54076]: debug 2022-01-31T22:50:47.635+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.636167+0000) 2022-01-31T22:50:47.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:47 smithi167 conmon[60316]: debug 2022-01-31T22:50:47.636+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.636606+0000) 2022-01-31T22:50:48.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[35325]: debug 2022-01-31T22:50:47.659+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242192 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:48.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[41853]: debug 2022-01-31T22:50:47.635+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.635963+0000) 2022-01-31T22:50:48.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[46715]: debug 2022-01-31T22:50:47.633+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.634442+0000) 2022-01-31T22:50:48.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:47 smithi171 conmon[51620]: debug 2022-01-31T22:50:47.634+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.635481+0000) 2022-01-31T22:50:48.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:47 smithi167 conmon[49112]: debug 2022-01-31T22:50:47.904+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.905145+0000) 2022-01-31T22:50:48.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:48 smithi167 conmon[54076]: debug 2022-01-31T22:50:48.072+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:48.073064+0000) 2022-01-31T22:50:48.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:47 smithi167 conmon[60316]: debug 2022-01-31T22:50:47.832+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:47.832726+0000) 2022-01-31T22:50:48.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:48 smithi171 conmon[46715]: debug 2022-01-31T22:50:48.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:48.319518+0000) 2022-01-31T22:50:48.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:48 smithi171 conmon[41853]: debug 2022-01-31T22:50:48.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:48.269300+0000) 2022-01-31T22:50:48.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:48 smithi171 conmon[51620]: debug 2022-01-31T22:50:48.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:48.443384+0000) 2022-01-31T22:50:49.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:48 smithi167 conmon[49112]: debug 2022-01-31T22:50:48.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:48.905296+0000) 2022-01-31T22:50:49.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:49 smithi167 conmon[54076]: debug 2022-01-31T22:50:49.072+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:49.073210+0000) 2022-01-31T22:50:49.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:48 smithi167 conmon[60316]: debug 2022-01-31T22:50:48.832+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:48.832909+0000) 2022-01-31T22:50:49.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:49 smithi171 conmon[41853]: debug 2022-01-31T22:50:49.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:49.269454+0000) 2022-01-31T22:50:49.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:49 smithi171 conmon[46715]: debug 2022-01-31T22:50:49.318+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:49.319717+0000) 2022-01-31T22:50:49.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:49 smithi171 conmon[51620]: debug 2022-01-31T22:50:49.442+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:49.443546+0000) 2022-01-31T22:50:50.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:49 smithi167 conmon[49112]: debug 2022-01-31T22:50:49.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:49.905469+0000) 2022-01-31T22:50:50.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:50 smithi167 conmon[54076]: debug 2022-01-31T22:50:50.073+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:50.073425+0000) 2022-01-31T22:50:50.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:49 smithi167 conmon[60316]: debug 2022-01-31T22:50:49.832+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:49.833095+0000) 2022-01-31T22:50:50.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:50 smithi171 conmon[41853]: debug 2022-01-31T22:50:50.268+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:50.269656+0000) 2022-01-31T22:50:50.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:50 smithi171 conmon[46715]: debug 2022-01-31T22:50:50.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:50.319891+0000) 2022-01-31T22:50:50.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:50 smithi171 conmon[51620]: debug 2022-01-31T22:50:50.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:50.443765+0000) 2022-01-31T22:50:51.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:50 smithi167 conmon[49112]: debug 2022-01-31T22:50:50.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:50.905637+0000) 2022-01-31T22:50:51.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:51 smithi167 conmon[54076]: debug 2022-01-31T22:50:51.073+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:51.073579+0000) 2022-01-31T22:50:51.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:50 smithi167 conmon[60316]: debug 2022-01-31T22:50:50.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:50.833253+0000) 2022-01-31T22:50:51.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:51 smithi171 conmon[41853]: debug 2022-01-31T22:50:51.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:51.269845+0000) 2022-01-31T22:50:51.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:51 smithi171 conmon[46715]: debug 2022-01-31T22:50:51.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:51.320012+0000) 2022-01-31T22:50:51.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:51 smithi171 conmon[51620]: debug 2022-01-31T22:50:51.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:51.443956+0000) 2022-01-31T22:50:52.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:51 smithi167 conmon[49112]: debug 2022-01-31T22:50:51.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:51.905825+0000) 2022-01-31T22:50:52.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:52 smithi167 conmon[54076]: debug 2022-01-31T22:50:52.073+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.073771+0000) 2022-01-31T22:50:52.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:51 smithi167 conmon[60316]: debug 2022-01-31T22:50:51.832+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:51.833444+0000) 2022-01-31T22:50:52.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[41853]: debug 2022-01-31T22:50:52.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.270020+0000) 2022-01-31T22:50:52.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[46715]: debug 2022-01-31T22:50:52.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.320181+0000) 2022-01-31T22:50:52.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[51620]: debug 2022-01-31T22:50:52.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.444117+0000) 2022-01-31T22:50:52.820 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:52 smithi167 conmon[49112]: debug 2022-01-31T22:50:52.662+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.662569+0000) 2022-01-31T22:50:52.821 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:52 smithi167 conmon[54076]: debug 2022-01-31T22:50:52.662+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.662795+0000) 2022-01-31T22:50:52.821 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:52 smithi167 conmon[60316]: debug 2022-01-31T22:50:52.662+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.662900+0000) 2022-01-31T22:50:53.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[35325]: debug 2022-01-31T22:50:52.685+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242305 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:53.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[41853]: debug 2022-01-31T22:50:52.662+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.663340+0000) 2022-01-31T22:50:53.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[46715]: debug 2022-01-31T22:50:52.661+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.661897+0000) 2022-01-31T22:50:53.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:52 smithi171 conmon[51620]: debug 2022-01-31T22:50:52.661+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.662449+0000) 2022-01-31T22:50:53.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:53 smithi167 conmon[54076]: debug 2022-01-31T22:50:53.073+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:53.073898+0000) 2022-01-31T22:50:53.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:52 smithi167 conmon[60316]: debug 2022-01-31T22:50:52.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.833618+0000) 2022-01-31T22:50:53.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:52 smithi167 conmon[49112]: debug 2022-01-31T22:50:52.905+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:52.906040+0000) 2022-01-31T22:50:53.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:53 smithi171 conmon[41853]: debug 2022-01-31T22:50:53.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:53.270176+0000) 2022-01-31T22:50:53.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:53 smithi171 conmon[46715]: debug 2022-01-31T22:50:53.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:53.320368+0000) 2022-01-31T22:50:53.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:53 smithi171 conmon[51620]: debug 2022-01-31T22:50:53.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:53.444289+0000) 2022-01-31T22:50:54.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:53 smithi171 conmon[35325]: debug 2022-01-31T22:50:53.919+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:50:54.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:54 smithi167 conmon[54076]: debug 2022-01-31T22:50:54.073+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:54.074080+0000) 2022-01-31T22:50:54.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:53 smithi167 conmon[60316]: debug 2022-01-31T22:50:53.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:53.833728+0000) 2022-01-31T22:50:54.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:53 smithi167 conmon[49112]: debug 2022-01-31T22:50:53.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:53.906212+0000) 2022-01-31T22:50:54.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:54 smithi171 conmon[41853]: debug 2022-01-31T22:50:54.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:54.270267+0000) 2022-01-31T22:50:54.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:54 smithi171 conmon[46715]: debug 2022-01-31T22:50:54.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:54.320515+0000) 2022-01-31T22:50:54.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:54 smithi171 conmon[51620]: debug 2022-01-31T22:50:54.443+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:54.444441+0000) 2022-01-31T22:50:55.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:54 smithi167 conmon[49112]: debug 2022-01-31T22:50:54.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:54.906414+0000) 2022-01-31T22:50:55.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:55 smithi167 conmon[54076]: debug 2022-01-31T22:50:55.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:55.074273+0000) 2022-01-31T22:50:55.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:54 smithi167 conmon[60316]: debug 2022-01-31T22:50:54.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:54.833913+0000) 2022-01-31T22:50:55.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:55 smithi171 conmon[41853]: debug 2022-01-31T22:50:55.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:55.270447+0000) 2022-01-31T22:50:55.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:55 smithi171 conmon[46715]: debug 2022-01-31T22:50:55.319+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:55.320699+0000) 2022-01-31T22:50:55.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:55 smithi171 conmon[51620]: debug 2022-01-31T22:50:55.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:55.444621+0000) 2022-01-31T22:50:56.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:55 smithi167 conmon[49112]: debug 2022-01-31T22:50:55.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:55.906578+0000) 2022-01-31T22:50:56.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:55 smithi167 conmon[60316]: debug 2022-01-31T22:50:55.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:55.834144+0000) 2022-01-31T22:50:56.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:56 smithi167 conmon[54076]: debug 2022-01-31T22:50:56.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:56.074436+0000) 2022-01-31T22:50:56.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:56 smithi171 conmon[41853]: debug 2022-01-31T22:50:56.269+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:56.270611+0000) 2022-01-31T22:50:56.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:56 smithi171 conmon[46715]: debug 2022-01-31T22:50:56.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:56.320828+0000) 2022-01-31T22:50:56.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:56 smithi171 conmon[51620]: debug 2022-01-31T22:50:56.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:56.444806+0000) 2022-01-31T22:50:57.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:56 smithi167 conmon[49112]: debug 2022-01-31T22:50:56.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:56.906782+0000) 2022-01-31T22:50:57.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:57 smithi167 conmon[54076]: debug 2022-01-31T22:50:57.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.074578+0000) 2022-01-31T22:50:57.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:56 smithi167 conmon[60316]: debug 2022-01-31T22:50:56.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:56.834353+0000) 2022-01-31T22:50:57.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[41853]: debug 2022-01-31T22:50:57.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.270811+0000) 2022-01-31T22:50:57.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[46715]: debug 2022-01-31T22:50:57.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.320945+0000) 2022-01-31T22:50:57.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[51620]: debug 2022-01-31T22:50:57.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.444968+0000) 2022-01-31T22:50:57.821 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:57 smithi167 conmon[49112]: debug 2022-01-31T22:50:57.689+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.689936+0000) 2022-01-31T22:50:57.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:57 smithi167 conmon[54076]: debug 2022-01-31T22:50:57.689+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.690043+0000) 2022-01-31T22:50:57.822 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:57 smithi167 conmon[60316]: debug 2022-01-31T22:50:57.689+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.689623+0000) 2022-01-31T22:50:58.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[35325]: debug 2022-01-31T22:50:57.713+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242415 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:50:58.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[41853]: debug 2022-01-31T22:50:57.689+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.690496+0000) 2022-01-31T22:50:58.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[46715]: debug 2022-01-31T22:50:57.688+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.688869+0000) 2022-01-31T22:50:58.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:57 smithi171 conmon[51620]: debug 2022-01-31T22:50:57.689+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.690273+0000) 2022-01-31T22:50:58.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:57 smithi167 conmon[49112]: debug 2022-01-31T22:50:57.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.906994+0000) 2022-01-31T22:50:58.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:58 smithi167 conmon[54076]: debug 2022-01-31T22:50:58.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:58.074724+0000) 2022-01-31T22:50:58.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:57 smithi167 conmon[60316]: debug 2022-01-31T22:50:57.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:57.834483+0000) 2022-01-31T22:50:58.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:58 smithi171 conmon[41853]: debug 2022-01-31T22:50:58.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:58.270970+0000) 2022-01-31T22:50:58.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:58 smithi171 conmon[46715]: debug 2022-01-31T22:50:58.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:58.321101+0000) 2022-01-31T22:50:58.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:58 smithi171 conmon[51620]: debug 2022-01-31T22:50:58.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:58.445150+0000) 2022-01-31T22:50:59.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:58 smithi167 conmon[49112]: debug 2022-01-31T22:50:58.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:58.907153+0000) 2022-01-31T22:50:59.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:58 smithi167 conmon[60316]: debug 2022-01-31T22:50:58.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:58.834614+0000) 2022-01-31T22:50:59.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:50:59 smithi167 conmon[54076]: debug 2022-01-31T22:50:59.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:59.074870+0000) 2022-01-31T22:50:59.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:50:59 smithi171 conmon[46715]: debug 2022-01-31T22:50:59.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:59.321238+0000) 2022-01-31T22:50:59.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:50:59 smithi171 conmon[51620]: debug 2022-01-31T22:50:59.444+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:59.445300+0000) 2022-01-31T22:50:59.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:50:59 smithi171 conmon[41853]: debug 2022-01-31T22:50:59.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:59.271172+0000) 2022-01-31T22:51:00.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:50:59 smithi167 conmon[49112]: debug 2022-01-31T22:50:59.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:59.907331+0000) 2022-01-31T22:51:00.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:00 smithi167 conmon[54076]: debug 2022-01-31T22:51:00.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:00.075057+0000) 2022-01-31T22:51:00.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:50:59 smithi167 conmon[60316]: debug 2022-01-31T22:50:59.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:50:59.834795+0000) 2022-01-31T22:51:00.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:00 smithi171 conmon[46715]: debug 2022-01-31T22:51:00.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:00.321396+0000) 2022-01-31T22:51:00.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:00 smithi171 conmon[51620]: debug 2022-01-31T22:51:00.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:00.445500+0000) 2022-01-31T22:51:00.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:00 smithi171 conmon[41853]: debug 2022-01-31T22:51:00.270+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:00.271353+0000) 2022-01-31T22:51:01.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:00 smithi167 conmon[49112]: debug 2022-01-31T22:51:00.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:00.907528+0000) 2022-01-31T22:51:01.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:01 smithi167 conmon[54076]: debug 2022-01-31T22:51:01.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:01.075268+0000) 2022-01-31T22:51:01.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:00 smithi167 conmon[60316]: debug 2022-01-31T22:51:00.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:00.834972+0000) 2022-01-31T22:51:01.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:01 smithi171 conmon[46715]: debug 2022-01-31T22:51:01.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:01.321584+0000) 2022-01-31T22:51:01.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:01 smithi171 conmon[51620]: debug 2022-01-31T22:51:01.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:01.445679+0000) 2022-01-31T22:51:01.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:01 smithi171 conmon[41853]: debug 2022-01-31T22:51:01.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:01.271523+0000) 2022-01-31T22:51:02.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:01 smithi167 conmon[49112]: debug 2022-01-31T22:51:01.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:01.907683+0000) 2022-01-31T22:51:02.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:02 smithi167 conmon[54076]: debug 2022-01-31T22:51:02.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.075427+0000) 2022-01-31T22:51:02.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:01 smithi167 conmon[60316]: debug 2022-01-31T22:51:01.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:01.835149+0000) 2022-01-31T22:51:02.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[46715]: debug 2022-01-31T22:51:02.320+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.321688+0000) 2022-01-31T22:51:02.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[51620]: debug 2022-01-31T22:51:02.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.445830+0000) 2022-01-31T22:51:02.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[41853]: debug 2022-01-31T22:51:02.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.271737+0000) 2022-01-31T22:51:02.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:02 smithi167 conmon[49112]: debug 2022-01-31T22:51:02.716+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.718314+0000) 2022-01-31T22:51:02.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:02 smithi167 conmon[54076]: debug 2022-01-31T22:51:02.715+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.716767+0000) 2022-01-31T22:51:02.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:02 smithi167 conmon[60316]: debug 2022-01-31T22:51:02.716+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.717618+0000) 2022-01-31T22:51:03.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[35325]: debug 2022-01-31T22:51:02.741+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242528 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:03.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[41853]: debug 2022-01-31T22:51:02.716+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.717282+0000) 2022-01-31T22:51:03.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[46715]: debug 2022-01-31T22:51:02.716+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.716767+0000) 2022-01-31T22:51:03.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:02 smithi171 conmon[51620]: debug 2022-01-31T22:51:02.717+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.718094+0000) 2022-01-31T22:51:03.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:02 smithi167 conmon[49112]: debug 2022-01-31T22:51:02.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.907865+0000) 2022-01-31T22:51:03.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:03 smithi167 conmon[54076]: debug 2022-01-31T22:51:03.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:03.075601+0000) 2022-01-31T22:51:03.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:02 smithi167 conmon[60316]: debug 2022-01-31T22:51:02.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:02.835335+0000) 2022-01-31T22:51:03.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:03 smithi171 conmon[46715]: debug 2022-01-31T22:51:03.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:03.321830+0000) 2022-01-31T22:51:03.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:03 smithi171 conmon[51620]: debug 2022-01-31T22:51:03.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:03.445979+0000) 2022-01-31T22:51:03.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:03 smithi171 conmon[41853]: debug 2022-01-31T22:51:03.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:03.271918+0000) 2022-01-31T22:51:04.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:03 smithi167 conmon[49112]: debug 2022-01-31T22:51:03.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:03.908025+0000) 2022-01-31T22:51:04.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:03 smithi167 conmon[60316]: debug 2022-01-31T22:51:03.833+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:03.835489+0000) 2022-01-31T22:51:04.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:04 smithi167 conmon[54076]: debug 2022-01-31T22:51:04.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:04.075747+0000) 2022-01-31T22:51:04.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:04 smithi171 conmon[41853]: debug 2022-01-31T22:51:04.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:04.272068+0000) 2022-01-31T22:51:04.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:04 smithi171 conmon[46715]: debug 2022-01-31T22:51:04.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:04.321937+0000) 2022-01-31T22:51:04.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:04 smithi171 conmon[51620]: debug 2022-01-31T22:51:04.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:04.446141+0000) 2022-01-31T22:51:05.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:04 smithi167 conmon[49112]: debug 2022-01-31T22:51:04.906+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:04.908203+0000) 2022-01-31T22:51:05.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:05 smithi167 conmon[54076]: debug 2022-01-31T22:51:05.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:05.075942+0000) 2022-01-31T22:51:05.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:04 smithi167 conmon[60316]: debug 2022-01-31T22:51:04.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:04.835682+0000) 2022-01-31T22:51:05.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:05 smithi171 conmon[51620]: debug 2022-01-31T22:51:05.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:05.446314+0000) 2022-01-31T22:51:05.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:05 smithi171 conmon[41853]: debug 2022-01-31T22:51:05.271+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:05.272271+0000) 2022-01-31T22:51:05.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:05 smithi171 conmon[46715]: debug 2022-01-31T22:51:05.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:05.322041+0000) 2022-01-31T22:51:06.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:05 smithi167 conmon[49112]: debug 2022-01-31T22:51:05.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:05.908363+0000) 2022-01-31T22:51:06.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:06 smithi167 conmon[54076]: debug 2022-01-31T22:51:06.075+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:06.076101+0000) 2022-01-31T22:51:06.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:05 smithi167 conmon[60316]: debug 2022-01-31T22:51:05.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:05.835874+0000) 2022-01-31T22:51:06.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:06 smithi171 conmon[41853]: debug 2022-01-31T22:51:06.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:06.272404+0000) 2022-01-31T22:51:06.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:06 smithi171 conmon[46715]: debug 2022-01-31T22:51:06.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:06.322190+0000) 2022-01-31T22:51:06.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:06 smithi171 conmon[51620]: debug 2022-01-31T22:51:06.445+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:06.446517+0000) 2022-01-31T22:51:07.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:06 smithi167 conmon[49112]: debug 2022-01-31T22:51:06.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:06.908542+0000) 2022-01-31T22:51:07.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:07 smithi167 conmon[54076]: debug 2022-01-31T22:51:07.074+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.076304+0000) 2022-01-31T22:51:07.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:06 smithi167 conmon[60316]: debug 2022-01-31T22:51:06.835+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:06.836058+0000) 2022-01-31T22:51:07.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[41853]: debug 2022-01-31T22:51:07.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.272582+0000) 2022-01-31T22:51:07.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[46715]: debug 2022-01-31T22:51:07.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.322345+0000) 2022-01-31T22:51:07.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[51620]: debug 2022-01-31T22:51:07.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.446748+0000) 2022-01-31T22:51:07.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:07 smithi167 conmon[49112]: debug 2022-01-31T22:51:07.743+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.744809+0000) 2022-01-31T22:51:07.823 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:07 smithi167 conmon[54076]: debug 2022-01-31T22:51:07.743+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.744650+0000) 2022-01-31T22:51:07.824 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:07 smithi167 conmon[60316]: debug 2022-01-31T22:51:07.742+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.744452+0000) 2022-01-31T22:51:08.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[35325]: debug 2022-01-31T22:51:07.768+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242639 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:08.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[41853]: debug 2022-01-31T22:51:07.743+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.744638+0000) 2022-01-31T22:51:08.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[46715]: debug 2022-01-31T22:51:07.743+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.744409+0000) 2022-01-31T22:51:08.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:07 smithi171 conmon[51620]: debug 2022-01-31T22:51:07.744+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.744949+0000) 2022-01-31T22:51:08.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:07 smithi167 conmon[49112]: debug 2022-01-31T22:51:07.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.908705+0000) 2022-01-31T22:51:08.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:08 smithi167 conmon[54076]: debug 2022-01-31T22:51:08.075+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:08.076523+0000) 2022-01-31T22:51:08.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:07 smithi167 conmon[60316]: debug 2022-01-31T22:51:07.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:07.836219+0000) 2022-01-31T22:51:08.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:08 smithi171 conmon[41853]: debug 2022-01-31T22:51:08.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:08.272794+0000) 2022-01-31T22:51:08.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:08 smithi171 conmon[46715]: debug 2022-01-31T22:51:08.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:08.322502+0000) 2022-01-31T22:51:08.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:08 smithi171 conmon[51620]: debug 2022-01-31T22:51:08.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:08.446969+0000) 2022-01-31T22:51:09.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:08 smithi171 conmon[35325]: debug 2022-01-31T22:51:08.920+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:51:09.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:08 smithi167 conmon[49112]: debug 2022-01-31T22:51:08.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:08.908852+0000) 2022-01-31T22:51:09.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:09 smithi167 conmon[54076]: debug 2022-01-31T22:51:09.075+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:09.076674+0000) 2022-01-31T22:51:09.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:08 smithi167 conmon[60316]: debug 2022-01-31T22:51:08.834+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:08.836369+0000) 2022-01-31T22:51:09.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:09 smithi171 conmon[41853]: debug 2022-01-31T22:51:09.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:09.272963+0000) 2022-01-31T22:51:09.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:09 smithi171 conmon[46715]: debug 2022-01-31T22:51:09.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:09.322632+0000) 2022-01-31T22:51:09.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:09 smithi171 conmon[51620]: debug 2022-01-31T22:51:09.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:09.447152+0000) 2022-01-31T22:51:10.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:09 smithi167 conmon[49112]: debug 2022-01-31T22:51:09.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:09.908981+0000) 2022-01-31T22:51:10.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:10 smithi167 conmon[54076]: debug 2022-01-31T22:51:10.076+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:10.076888+0000) 2022-01-31T22:51:10.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:09 smithi167 conmon[60316]: debug 2022-01-31T22:51:09.835+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:09.836584+0000) 2022-01-31T22:51:10.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:10 smithi171 conmon[41853]: debug 2022-01-31T22:51:10.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:10.273147+0000) 2022-01-31T22:51:10.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:10 smithi171 conmon[46715]: debug 2022-01-31T22:51:10.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:10.322725+0000) 2022-01-31T22:51:10.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:10 smithi171 conmon[51620]: debug 2022-01-31T22:51:10.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:10.447338+0000) 2022-01-31T22:51:11.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:10 smithi167 conmon[60316]: debug 2022-01-31T22:51:10.835+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:10.836810+0000) 2022-01-31T22:51:11.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:10 smithi167 conmon[49112]: debug 2022-01-31T22:51:10.907+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:10.909163+0000) 2022-01-31T22:51:11.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:11 smithi167 conmon[54076]: debug 2022-01-31T22:51:11.076+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:11.077107+0000) 2022-01-31T22:51:11.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:11 smithi171 conmon[41853]: debug 2022-01-31T22:51:11.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:11.273313+0000) 2022-01-31T22:51:11.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:11 smithi171 conmon[46715]: debug 2022-01-31T22:51:11.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:11.322850+0000) 2022-01-31T22:51:11.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:11 smithi171 conmon[51620]: debug 2022-01-31T22:51:11.446+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:11.447470+0000) 2022-01-31T22:51:12.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:11 smithi167 conmon[49112]: debug 2022-01-31T22:51:11.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:11.909354+0000) 2022-01-31T22:51:12.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:12 smithi167 conmon[54076]: debug 2022-01-31T22:51:12.076+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.077255+0000) 2022-01-31T22:51:12.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:11 smithi167 conmon[60316]: debug 2022-01-31T22:51:11.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:11.836992+0000) 2022-01-31T22:51:12.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[41853]: debug 2022-01-31T22:51:12.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.273477+0000) 2022-01-31T22:51:12.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[46715]: debug 2022-01-31T22:51:12.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.322956+0000) 2022-01-31T22:51:12.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[51620]: debug 2022-01-31T22:51:12.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.447676+0000) 2022-01-31T22:51:12.823 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:12 smithi167 conmon[49112]: debug 2022-01-31T22:51:12.770+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.771522+0000) 2022-01-31T22:51:12.824 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:12 smithi167 conmon[54076]: debug 2022-01-31T22:51:12.771+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.772642+0000) 2022-01-31T22:51:12.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:12 smithi167 conmon[60316]: debug 2022-01-31T22:51:12.770+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.771656+0000) 2022-01-31T22:51:13.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[46715]: debug 2022-01-31T22:51:12.772+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.772900+0000) 2022-01-31T22:51:13.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[51620]: debug 2022-01-31T22:51:12.772+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.772774+0000) 2022-01-31T22:51:13.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[35325]: debug 2022-01-31T22:51:12.796+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242753 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:13.096 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:12 smithi171 conmon[41853]: debug 2022-01-31T22:51:12.771+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.772592+0000) 2022-01-31T22:51:13.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:12 smithi167 conmon[49112]: debug 2022-01-31T22:51:12.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.909604+0000) 2022-01-31T22:51:13.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:13 smithi167 conmon[54076]: debug 2022-01-31T22:51:13.075+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:13.077451+0000) 2022-01-31T22:51:13.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:12 smithi167 conmon[60316]: debug 2022-01-31T22:51:12.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:12.837143+0000) 2022-01-31T22:51:13.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:13 smithi171 conmon[46715]: debug 2022-01-31T22:51:13.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:13.323071+0000) 2022-01-31T22:51:13.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:13 smithi171 conmon[51620]: debug 2022-01-31T22:51:13.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:13.447893+0000) 2022-01-31T22:51:13.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:13 smithi171 conmon[41853]: debug 2022-01-31T22:51:13.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:13.273603+0000) 2022-01-31T22:51:14.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:13 smithi167 conmon[49112]: debug 2022-01-31T22:51:13.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:13.909728+0000) 2022-01-31T22:51:14.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:14 smithi167 conmon[54076]: debug 2022-01-31T22:51:14.076+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:14.077644+0000) 2022-01-31T22:51:14.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:13 smithi167 conmon[60316]: debug 2022-01-31T22:51:13.835+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:13.837316+0000) 2022-01-31T22:51:14.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:14 smithi171 conmon[46715]: debug 2022-01-31T22:51:14.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:14.323173+0000) 2022-01-31T22:51:14.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:14 smithi171 conmon[41853]: debug 2022-01-31T22:51:14.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:14.273776+0000) 2022-01-31T22:51:14.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:14 smithi171 conmon[51620]: debug 2022-01-31T22:51:14.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:14.448077+0000) 2022-01-31T22:51:15.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:14 smithi167 conmon[49112]: debug 2022-01-31T22:51:14.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:14.909926+0000) 2022-01-31T22:51:15.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:15 smithi167 conmon[54076]: debug 2022-01-31T22:51:15.077+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:15.077799+0000) 2022-01-31T22:51:15.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:14 smithi167 conmon[60316]: debug 2022-01-31T22:51:14.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:14.837538+0000) 2022-01-31T22:51:15.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:15 smithi171 conmon[46715]: debug 2022-01-31T22:51:15.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:15.323276+0000) 2022-01-31T22:51:15.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:15 smithi171 conmon[41853]: debug 2022-01-31T22:51:15.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:15.273959+0000) 2022-01-31T22:51:15.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:15 smithi171 conmon[51620]: debug 2022-01-31T22:51:15.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:15.448246+0000) 2022-01-31T22:51:16.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:15 smithi167 conmon[49112]: debug 2022-01-31T22:51:15.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:15.910084+0000) 2022-01-31T22:51:16.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:16 smithi167 conmon[54076]: debug 2022-01-31T22:51:16.077+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:16.077960+0000) 2022-01-31T22:51:16.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:15 smithi167 conmon[60316]: debug 2022-01-31T22:51:15.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:15.837720+0000) 2022-01-31T22:51:16.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:16 smithi171 conmon[46715]: debug 2022-01-31T22:51:16.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:16.323399+0000) 2022-01-31T22:51:16.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:16 smithi171 conmon[41853]: debug 2022-01-31T22:51:16.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:16.274135+0000) 2022-01-31T22:51:16.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:16 smithi171 conmon[51620]: debug 2022-01-31T22:51:16.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:16.448437+0000) 2022-01-31T22:51:17.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:16 smithi167 conmon[49112]: debug 2022-01-31T22:51:16.908+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:16.910298+0000) 2022-01-31T22:51:17.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:17 smithi167 conmon[54076]: debug 2022-01-31T22:51:17.076+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.078191+0000) 2022-01-31T22:51:17.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:16 smithi167 conmon[60316]: debug 2022-01-31T22:51:16.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:16.837837+0000) 2022-01-31T22:51:17.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[46715]: debug 2022-01-31T22:51:17.321+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.323562+0000) 2022-01-31T22:51:17.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[41853]: debug 2022-01-31T22:51:17.272+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.274254+0000) 2022-01-31T22:51:17.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[51620]: debug 2022-01-31T22:51:17.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.448628+0000) 2022-01-31T22:51:17.824 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:17 smithi167 conmon[49112]: debug 2022-01-31T22:51:17.797+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.799476+0000) 2022-01-31T22:51:17.825 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:17 smithi167 conmon[54076]: debug 2022-01-31T22:51:17.797+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.799159+0000) 2022-01-31T22:51:17.825 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:17 smithi167 conmon[60316]: debug 2022-01-31T22:51:17.805+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.807322+0000) 2022-01-31T22:51:18.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[35325]: debug 2022-01-31T22:51:17.837+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242862 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:18.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[41853]: debug 2022-01-31T22:51:17.798+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.800264+0000) 2022-01-31T22:51:18.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[46715]: debug 2022-01-31T22:51:17.798+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.799996+0000) 2022-01-31T22:51:18.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:17 smithi171 conmon[51620]: debug 2022-01-31T22:51:17.797+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.799649+0000) 2022-01-31T22:51:18.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:17 smithi167 conmon[49112]: debug 2022-01-31T22:51:17.909+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.910522+0000) 2022-01-31T22:51:18.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:18 smithi167 conmon[54076]: debug 2022-01-31T22:51:18.077+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:18.078375+0000) 2022-01-31T22:51:18.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:17 smithi167 conmon[60316]: debug 2022-01-31T22:51:17.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:17.838017+0000) 2022-01-31T22:51:18.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:18 smithi171 conmon[41853]: debug 2022-01-31T22:51:18.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:18.274395+0000) 2022-01-31T22:51:18.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:18 smithi171 conmon[46715]: debug 2022-01-31T22:51:18.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:18.323976+0000) 2022-01-31T22:51:18.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:18 smithi171 conmon[51620]: debug 2022-01-31T22:51:18.447+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:18.448822+0000) 2022-01-31T22:51:19.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:19 smithi167 conmon[54076]: debug 2022-01-31T22:51:19.077+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:19.078524+0000) 2022-01-31T22:51:19.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:18 smithi167 conmon[49112]: debug 2022-01-31T22:51:18.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:18.910682+0000) 2022-01-31T22:51:19.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:18 smithi167 conmon[60316]: debug 2022-01-31T22:51:18.836+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:18.838170+0000) 2022-01-31T22:51:19.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:19 smithi171 conmon[41853]: debug 2022-01-31T22:51:19.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:19.274557+0000) 2022-01-31T22:51:19.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:19 smithi171 conmon[46715]: debug 2022-01-31T22:51:19.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:19.324084+0000) 2022-01-31T22:51:19.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:19 smithi171 conmon[51620]: debug 2022-01-31T22:51:19.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:19.448972+0000) 2022-01-31T22:51:20.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:20 smithi167 conmon[54076]: debug 2022-01-31T22:51:20.078+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:20.078700+0000) 2022-01-31T22:51:20.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:19 smithi167 conmon[49112]: debug 2022-01-31T22:51:19.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:19.910904+0000) 2022-01-31T22:51:20.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:19 smithi167 conmon[60316]: debug 2022-01-31T22:51:19.837+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:19.838336+0000) 2022-01-31T22:51:20.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:20 smithi171 conmon[46715]: debug 2022-01-31T22:51:20.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:20.324189+0000) 2022-01-31T22:51:20.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:20 smithi171 conmon[41853]: debug 2022-01-31T22:51:20.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:20.274740+0000) 2022-01-31T22:51:20.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:20 smithi171 conmon[51620]: debug 2022-01-31T22:51:20.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:20.449169+0000) 2022-01-31T22:51:21.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:21 smithi167 conmon[54076]: debug 2022-01-31T22:51:21.078+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:21.078864+0000) 2022-01-31T22:51:21.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:20 smithi167 conmon[49112]: debug 2022-01-31T22:51:20.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:20.911090+0000) 2022-01-31T22:51:21.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:20 smithi167 conmon[60316]: debug 2022-01-31T22:51:20.837+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:20.838512+0000) 2022-01-31T22:51:21.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:21 smithi171 conmon[46715]: debug 2022-01-31T22:51:21.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:21.324365+0000) 2022-01-31T22:51:21.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:21 smithi171 conmon[41853]: debug 2022-01-31T22:51:21.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:21.274922+0000) 2022-01-31T22:51:21.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:21 smithi171 conmon[51620]: debug 2022-01-31T22:51:21.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:21.449357+0000) 2022-01-31T22:51:22.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:22 smithi167 conmon[54076]: debug 2022-01-31T22:51:22.078+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.079071+0000) 2022-01-31T22:51:22.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:21 smithi167 conmon[49112]: debug 2022-01-31T22:51:21.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:21.911276+0000) 2022-01-31T22:51:22.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:21 smithi167 conmon[60316]: debug 2022-01-31T22:51:21.837+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:21.838632+0000) 2022-01-31T22:51:22.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[41853]: debug 2022-01-31T22:51:22.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.275090+0000) 2022-01-31T22:51:22.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[46715]: debug 2022-01-31T22:51:22.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.324510+0000) 2022-01-31T22:51:22.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[51620]: debug 2022-01-31T22:51:22.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.449560+0000) 2022-01-31T22:51:23.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[35325]: debug 2022-01-31T22:51:22.864+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 242972 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:23.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[41853]: debug 2022-01-31T22:51:22.840+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.842497+0000) 2022-01-31T22:51:23.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[46715]: debug 2022-01-31T22:51:22.840+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.842132+0000) 2022-01-31T22:51:23.096 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:22 smithi171 conmon[51620]: debug 2022-01-31T22:51:22.839+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.841603+0000) 2022-01-31T22:51:23.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:22 smithi167 conmon[49112]: debug 2022-01-31T22:51:22.840+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.841781+0000) 2022-01-31T22:51:23.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:22 smithi167 conmon[49112]: debug 2022-01-31T22:51:22.910+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.911479+0000) 2022-01-31T22:51:23.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:22 smithi167 conmon[54076]: debug 2022-01-31T22:51:22.840+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.841584+0000) 2022-01-31T22:51:23.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:23 smithi167 conmon[54076]: debug 2022-01-31T22:51:23.078+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:23.079256+0000) 2022-01-31T22:51:23.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:22 smithi167 conmon[60316]: debug 2022-01-31T22:51:22.837+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.838770+0000) 2022-01-31T22:51:23.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:22 smithi167 conmon[60316]: debug 2022-01-31T22:51:22.841+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:22.842952+0000) 2022-01-31T22:51:23.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:23 smithi171 conmon[41853]: debug 2022-01-31T22:51:23.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:23.275271+0000) 2022-01-31T22:51:23.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:23 smithi171 conmon[46715]: debug 2022-01-31T22:51:23.322+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:23.324653+0000) 2022-01-31T22:51:23.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:23 smithi171 conmon[51620]: debug 2022-01-31T22:51:23.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:23.449772+0000) 2022-01-31T22:51:24.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:23 smithi171 conmon[35325]: debug 2022-01-31T22:51:23.920+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:51:24.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:23 smithi167 conmon[49112]: debug 2022-01-31T22:51:23.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:23.911635+0000) 2022-01-31T22:51:24.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:24 smithi167 conmon[54076]: debug 2022-01-31T22:51:24.078+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:24.079413+0000) 2022-01-31T22:51:24.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:23 smithi167 conmon[60316]: debug 2022-01-31T22:51:23.837+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:23.838922+0000) 2022-01-31T22:51:24.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:24 smithi171 conmon[41853]: debug 2022-01-31T22:51:24.273+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:24.275426+0000) 2022-01-31T22:51:24.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:24 smithi171 conmon[46715]: debug 2022-01-31T22:51:24.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:24.324795+0000) 2022-01-31T22:51:24.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:24 smithi171 conmon[51620]: debug 2022-01-31T22:51:24.448+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:24.449907+0000) 2022-01-31T22:51:25.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:24 smithi167 conmon[49112]: debug 2022-01-31T22:51:24.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:24.911847+0000) 2022-01-31T22:51:25.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:25 smithi167 conmon[54076]: debug 2022-01-31T22:51:25.079+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:25.079622+0000) 2022-01-31T22:51:25.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:24 smithi167 conmon[60316]: debug 2022-01-31T22:51:24.837+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:24.839117+0000) 2022-01-31T22:51:25.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:25 smithi171 conmon[41853]: debug 2022-01-31T22:51:25.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:25.275572+0000) 2022-01-31T22:51:25.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:25 smithi171 conmon[46715]: debug 2022-01-31T22:51:25.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:25.324926+0000) 2022-01-31T22:51:25.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:25 smithi171 conmon[51620]: debug 2022-01-31T22:51:25.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:25.450037+0000) 2022-01-31T22:51:26.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:25 smithi167 conmon[49112]: debug 2022-01-31T22:51:25.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:25.912036+0000) 2022-01-31T22:51:26.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:26 smithi167 conmon[54076]: debug 2022-01-31T22:51:26.079+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:26.079827+0000) 2022-01-31T22:51:26.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:25 smithi167 conmon[60316]: debug 2022-01-31T22:51:25.838+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:25.839327+0000) 2022-01-31T22:51:26.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:26 smithi171 conmon[41853]: debug 2022-01-31T22:51:26.274+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:26.275747+0000) 2022-01-31T22:51:26.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:26 smithi171 conmon[46715]: debug 2022-01-31T22:51:26.323+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:26.325057+0000) 2022-01-31T22:51:26.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:26 smithi171 conmon[51620]: debug 2022-01-31T22:51:26.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:26.450236+0000) 2022-01-31T22:51:27.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:26 smithi167 conmon[49112]: debug 2022-01-31T22:51:26.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:26.912212+0000) 2022-01-31T22:51:27.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:26 smithi167 conmon[60316]: debug 2022-01-31T22:51:26.839+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:26.839547+0000) 2022-01-31T22:51:27.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:27 smithi167 conmon[54076]: debug 2022-01-31T22:51:27.079+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.080030+0000) 2022-01-31T22:51:27.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[41853]: debug 2022-01-31T22:51:27.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.275934+0000) 2022-01-31T22:51:27.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[46715]: debug 2022-01-31T22:51:27.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.325214+0000) 2022-01-31T22:51:27.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[51620]: debug 2022-01-31T22:51:27.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.450396+0000) 2022-01-31T22:51:28.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[35325]: debug 2022-01-31T22:51:27.893+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243081 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:28.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[41853]: debug 2022-01-31T22:51:27.867+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.868589+0000) 2022-01-31T22:51:28.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[46715]: debug 2022-01-31T22:51:27.868+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.868997+0000) 2022-01-31T22:51:28.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:27 smithi171 conmon[51620]: debug 2022-01-31T22:51:27.869+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.869806+0000) 2022-01-31T22:51:28.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:27 smithi167 conmon[49112]: debug 2022-01-31T22:51:27.869+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.869782+0000) 2022-01-31T22:51:28.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:27 smithi167 conmon[49112]: debug 2022-01-31T22:51:27.911+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.912366+0000) 2022-01-31T22:51:28.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:27 smithi167 conmon[54076]: debug 2022-01-31T22:51:27.868+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.868744+0000) 2022-01-31T22:51:28.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:28 smithi167 conmon[54076]: debug 2022-01-31T22:51:28.079+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:28.080187+0000) 2022-01-31T22:51:28.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:27 smithi167 conmon[60316]: debug 2022-01-31T22:51:27.839+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.839746+0000) 2022-01-31T22:51:28.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:27 smithi167 conmon[60316]: debug 2022-01-31T22:51:27.868+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:27.869172+0000) 2022-01-31T22:51:28.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:28 smithi171 conmon[41853]: debug 2022-01-31T22:51:28.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:28.276080+0000) 2022-01-31T22:51:28.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:28 smithi171 conmon[46715]: debug 2022-01-31T22:51:28.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:28.325424+0000) 2022-01-31T22:51:28.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:28 smithi171 conmon[51620]: debug 2022-01-31T22:51:28.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:28.450589+0000) 2022-01-31T22:51:29.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:28 smithi167 conmon[49112]: debug 2022-01-31T22:51:28.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:28.912521+0000) 2022-01-31T22:51:29.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:29 smithi167 conmon[54076]: debug 2022-01-31T22:51:29.079+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:29.080332+0000) 2022-01-31T22:51:29.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:28 smithi167 conmon[60316]: debug 2022-01-31T22:51:28.839+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:28.839930+0000) 2022-01-31T22:51:29.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:29 smithi171 conmon[41853]: debug 2022-01-31T22:51:29.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:29.276234+0000) 2022-01-31T22:51:29.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:29 smithi171 conmon[46715]: debug 2022-01-31T22:51:29.324+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:29.325611+0000) 2022-01-31T22:51:29.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:29 smithi171 conmon[51620]: debug 2022-01-31T22:51:29.449+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:29.450722+0000) 2022-01-31T22:51:30.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:29 smithi167 conmon[49112]: debug 2022-01-31T22:51:29.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:29.912633+0000) 2022-01-31T22:51:30.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:30 smithi167 conmon[54076]: debug 2022-01-31T22:51:30.080+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:30.080516+0000) 2022-01-31T22:51:30.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:29 smithi167 conmon[60316]: debug 2022-01-31T22:51:29.839+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:29.840086+0000) 2022-01-31T22:51:30.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:30 smithi171 conmon[41853]: debug 2022-01-31T22:51:30.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:30.276421+0000) 2022-01-31T22:51:30.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:30 smithi171 conmon[46715]: debug 2022-01-31T22:51:30.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:30.325810+0000) 2022-01-31T22:51:30.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:30 smithi171 conmon[51620]: debug 2022-01-31T22:51:30.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:30.450910+0000) 2022-01-31T22:51:31.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:30 smithi167 conmon[49112]: debug 2022-01-31T22:51:30.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:30.912806+0000) 2022-01-31T22:51:31.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:30 smithi167 conmon[60316]: debug 2022-01-31T22:51:30.839+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:30.840242+0000) 2022-01-31T22:51:31.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:31 smithi167 conmon[54076]: debug 2022-01-31T22:51:31.080+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:31.080691+0000) 2022-01-31T22:51:31.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:31 smithi171 conmon[41853]: debug 2022-01-31T22:51:31.275+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:31.276599+0000) 2022-01-31T22:51:31.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:31 smithi171 conmon[46715]: debug 2022-01-31T22:51:31.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:31.326001+0000) 2022-01-31T22:51:31.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:31 smithi171 conmon[51620]: debug 2022-01-31T22:51:31.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:31.451104+0000) 2022-01-31T22:51:32.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:31 smithi167 conmon[49112]: debug 2022-01-31T22:51:31.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:31.912988+0000) 2022-01-31T22:51:32.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:32 smithi167 conmon[54076]: debug 2022-01-31T22:51:32.080+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.080870+0000) 2022-01-31T22:51:32.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:31 smithi167 conmon[60316]: debug 2022-01-31T22:51:31.839+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:31.840402+0000) 2022-01-31T22:51:32.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[41853]: debug 2022-01-31T22:51:32.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.276767+0000) 2022-01-31T22:51:32.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[46715]: debug 2022-01-31T22:51:32.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.326171+0000) 2022-01-31T22:51:32.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[51620]: debug 2022-01-31T22:51:32.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.451300+0000) 2022-01-31T22:51:33.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[35325]: debug 2022-01-31T22:51:32.921+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243195 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:33.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[41853]: debug 2022-01-31T22:51:32.895+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.896510+0000) 2022-01-31T22:51:33.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[46715]: debug 2022-01-31T22:51:32.896+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.897417+0000) 2022-01-31T22:51:33.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:32 smithi171 conmon[51620]: debug 2022-01-31T22:51:32.896+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.897337+0000) 2022-01-31T22:51:33.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:32 smithi167 conmon[54076]: debug 2022-01-31T22:51:32.897+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.897975+0000) 2022-01-31T22:51:33.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:33 smithi167 conmon[54076]: debug 2022-01-31T22:51:33.080+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:33.081028+0000) 2022-01-31T22:51:33.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:32 smithi167 conmon[49112]: debug 2022-01-31T22:51:32.896+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.896733+0000) 2022-01-31T22:51:33.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:32 smithi167 conmon[49112]: debug 2022-01-31T22:51:32.912+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.913195+0000) 2022-01-31T22:51:33.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:32 smithi167 conmon[60316]: debug 2022-01-31T22:51:32.840+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.840614+0000) 2022-01-31T22:51:33.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:32 smithi167 conmon[60316]: debug 2022-01-31T22:51:32.896+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:32.897346+0000) 2022-01-31T22:51:33.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:33 smithi171 conmon[41853]: debug 2022-01-31T22:51:33.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:33.276878+0000) 2022-01-31T22:51:33.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:33 smithi171 conmon[46715]: debug 2022-01-31T22:51:33.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:33.326381+0000) 2022-01-31T22:51:33.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:33 smithi171 conmon[51620]: debug 2022-01-31T22:51:33.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:33.451499+0000) 2022-01-31T22:51:34.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:33 smithi167 conmon[49112]: debug 2022-01-31T22:51:33.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:33.913361+0000) 2022-01-31T22:51:34.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:34 smithi167 conmon[54076]: debug 2022-01-31T22:51:34.080+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:34.081207+0000) 2022-01-31T22:51:34.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:33 smithi167 conmon[60316]: debug 2022-01-31T22:51:33.840+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:33.840800+0000) 2022-01-31T22:51:34.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:34 smithi171 conmon[41853]: debug 2022-01-31T22:51:34.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:34.277030+0000) 2022-01-31T22:51:34.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:34 smithi171 conmon[46715]: debug 2022-01-31T22:51:34.325+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:34.326494+0000) 2022-01-31T22:51:34.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:34 smithi171 conmon[51620]: debug 2022-01-31T22:51:34.450+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:34.451599+0000) 2022-01-31T22:51:35.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:34 smithi167 conmon[49112]: debug 2022-01-31T22:51:34.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:34.913575+0000) 2022-01-31T22:51:35.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:35 smithi167 conmon[54076]: debug 2022-01-31T22:51:35.080+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:35.081424+0000) 2022-01-31T22:51:35.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:34 smithi167 conmon[60316]: debug 2022-01-31T22:51:34.840+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:34.840977+0000) 2022-01-31T22:51:35.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:35 smithi171 conmon[41853]: debug 2022-01-31T22:51:35.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:35.277197+0000) 2022-01-31T22:51:35.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:35 smithi171 conmon[46715]: debug 2022-01-31T22:51:35.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:35.326623+0000) 2022-01-31T22:51:35.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:35 smithi171 conmon[51620]: debug 2022-01-31T22:51:35.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:35.451803+0000) 2022-01-31T22:51:36.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:35 smithi167 conmon[49112]: debug 2022-01-31T22:51:35.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:35.913718+0000) 2022-01-31T22:51:36.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:35 smithi167 conmon[60316]: debug 2022-01-31T22:51:35.840+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:35.841175+0000) 2022-01-31T22:51:36.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:36 smithi167 conmon[54076]: debug 2022-01-31T22:51:36.081+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:36.081520+0000) 2022-01-31T22:51:36.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:36 smithi171 conmon[41853]: debug 2022-01-31T22:51:36.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:36.277359+0000) 2022-01-31T22:51:36.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:36 smithi171 conmon[51620]: debug 2022-01-31T22:51:36.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:36.451964+0000) 2022-01-31T22:51:36.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:36 smithi171 conmon[46715]: debug 2022-01-31T22:51:36.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:36.326810+0000) 2022-01-31T22:51:37.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:36 smithi167 conmon[49112]: debug 2022-01-31T22:51:36.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:36.913868+0000) 2022-01-31T22:51:37.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:36 smithi167 conmon[60316]: debug 2022-01-31T22:51:36.840+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:36.841379+0000) 2022-01-31T22:51:37.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:37 smithi167 conmon[54076]: debug 2022-01-31T22:51:37.081+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.081604+0000) 2022-01-31T22:51:37.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[41853]: debug 2022-01-31T22:51:37.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.277540+0000) 2022-01-31T22:51:37.613 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[46715]: debug 2022-01-31T22:51:37.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.326996+0000) 2022-01-31T22:51:37.614 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[51620]: debug 2022-01-31T22:51:37.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.452148+0000) 2022-01-31T22:51:38.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[35325]: debug 2022-01-31T22:51:37.948+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243304 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:38.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[41853]: debug 2022-01-31T22:51:37.923+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.924361+0000) 2022-01-31T22:51:38.095 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[46715]: debug 2022-01-31T22:51:37.925+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.925850+0000) 2022-01-31T22:51:38.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:37 smithi171 conmon[51620]: debug 2022-01-31T22:51:37.924+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.925234+0000) 2022-01-31T22:51:38.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:37 smithi167 conmon[49112]: debug 2022-01-31T22:51:37.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.914013+0000) 2022-01-31T22:51:38.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:37 smithi167 conmon[49112]: debug 2022-01-31T22:51:37.925+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.925726+0000) 2022-01-31T22:51:38.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:37 smithi167 conmon[54076]: debug 2022-01-31T22:51:37.925+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.926245+0000) 2022-01-31T22:51:38.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:38 smithi167 conmon[54076]: debug 2022-01-31T22:51:38.081+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:38.081730+0000) 2022-01-31T22:51:38.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:37 smithi167 conmon[60316]: debug 2022-01-31T22:51:37.841+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.841538+0000) 2022-01-31T22:51:38.159 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:37 smithi167 conmon[60316]: debug 2022-01-31T22:51:37.924+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:37.924955+0000) 2022-01-31T22:51:38.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:38 smithi171 conmon[41853]: debug 2022-01-31T22:51:38.276+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:38.277692+0000) 2022-01-31T22:51:38.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:38 smithi171 conmon[46715]: debug 2022-01-31T22:51:38.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:38.327178+0000) 2022-01-31T22:51:38.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:38 smithi171 conmon[51620]: debug 2022-01-31T22:51:38.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:38.452336+0000) 2022-01-31T22:51:39.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:38 smithi171 conmon[35325]: debug 2022-01-31T22:51:38.921+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:51:39.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:38 smithi167 conmon[49112]: debug 2022-01-31T22:51:38.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:38.914184+0000) 2022-01-31T22:51:39.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:39 smithi167 conmon[54076]: debug 2022-01-31T22:51:39.081+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:39.081875+0000) 2022-01-31T22:51:39.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:38 smithi167 conmon[60316]: debug 2022-01-31T22:51:38.841+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:38.841724+0000) 2022-01-31T22:51:39.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:39 smithi171 conmon[41853]: debug 2022-01-31T22:51:39.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:39.277849+0000) 2022-01-31T22:51:39.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:39 smithi171 conmon[51620]: debug 2022-01-31T22:51:39.451+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:39.452492+0000) 2022-01-31T22:51:39.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:39 smithi171 conmon[46715]: debug 2022-01-31T22:51:39.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:39.327328+0000) 2022-01-31T22:51:40.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:39 smithi167 conmon[49112]: debug 2022-01-31T22:51:39.913+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:39.914378+0000) 2022-01-31T22:51:40.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:40 smithi167 conmon[54076]: debug 2022-01-31T22:51:40.081+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:40.082062+0000) 2022-01-31T22:51:40.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:39 smithi167 conmon[60316]: debug 2022-01-31T22:51:39.841+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:39.841868+0000) 2022-01-31T22:51:40.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:40 smithi171 conmon[41853]: debug 2022-01-31T22:51:40.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:40.278027+0000) 2022-01-31T22:51:40.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:40 smithi171 conmon[46715]: debug 2022-01-31T22:51:40.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:40.327480+0000) 2022-01-31T22:51:40.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:40 smithi171 conmon[51620]: debug 2022-01-31T22:51:40.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:40.452694+0000) 2022-01-31T22:51:41.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:40 smithi167 conmon[49112]: debug 2022-01-31T22:51:40.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:40.914532+0000) 2022-01-31T22:51:41.069 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:40 smithi167 conmon[60316]: debug 2022-01-31T22:51:40.841+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:40.842038+0000) 2022-01-31T22:51:41.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:41 smithi167 conmon[54076]: debug 2022-01-31T22:51:41.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:41.082260+0000) 2022-01-31T22:51:41.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:41 smithi171 conmon[41853]: debug 2022-01-31T22:51:41.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:41.278208+0000) 2022-01-31T22:51:41.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:41 smithi171 conmon[46715]: debug 2022-01-31T22:51:41.326+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:41.327686+0000) 2022-01-31T22:51:41.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:41 smithi171 conmon[51620]: debug 2022-01-31T22:51:41.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:41.452899+0000) 2022-01-31T22:51:42.068 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:41 smithi167 conmon[49112]: debug 2022-01-31T22:51:41.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:41.914697+0000) 2022-01-31T22:51:42.069 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:41 smithi167 conmon[60316]: debug 2022-01-31T22:51:41.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:41.842242+0000) 2022-01-31T22:51:42.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:42 smithi167 conmon[54076]: debug 2022-01-31T22:51:42.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.082468+0000) 2022-01-31T22:51:42.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[41853]: debug 2022-01-31T22:51:42.277+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.278380+0000) 2022-01-31T22:51:42.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[51620]: debug 2022-01-31T22:51:42.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.453089+0000) 2022-01-31T22:51:42.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[46715]: debug 2022-01-31T22:51:42.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.327803+0000) 2022-01-31T22:51:43.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:42 smithi167 conmon[49112]: debug 2022-01-31T22:51:42.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.914902+0000) 2022-01-31T22:51:43.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:42 smithi167 conmon[49112]: debug 2022-01-31T22:51:42.951+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.952217+0000) 2022-01-31T22:51:43.070 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:42 smithi167 conmon[54076]: debug 2022-01-31T22:51:42.952+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.952627+0000) 2022-01-31T22:51:43.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:42 smithi167 conmon[60316]: debug 2022-01-31T22:51:42.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.842383+0000) 2022-01-31T22:51:43.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:42 smithi167 conmon[60316]: debug 2022-01-31T22:51:42.950+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.951347+0000) 2022-01-31T22:51:43.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[35325]: debug 2022-01-31T22:51:42.976+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243418 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:43.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[41853]: debug 2022-01-31T22:51:42.951+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.951999+0000) 2022-01-31T22:51:43.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[46715]: debug 2022-01-31T22:51:42.951+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.951905+0000) 2022-01-31T22:51:43.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:42 smithi171 conmon[51620]: debug 2022-01-31T22:51:42.951+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:42.951764+0000) 2022-01-31T22:51:43.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:43 smithi167 conmon[54076]: debug 2022-01-31T22:51:43.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:43.082646+0000) 2022-01-31T22:51:43.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:43 smithi171 conmon[41853]: debug 2022-01-31T22:51:43.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:43.278547+0000) 2022-01-31T22:51:43.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:43 smithi171 conmon[46715]: debug 2022-01-31T22:51:43.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:43.327908+0000) 2022-01-31T22:51:43.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:43 smithi171 conmon[51620]: debug 2022-01-31T22:51:43.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:43.453274+0000) 2022-01-31T22:51:44.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:43 smithi167 conmon[49112]: debug 2022-01-31T22:51:43.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:43.915091+0000) 2022-01-31T22:51:44.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:43 smithi167 conmon[60316]: debug 2022-01-31T22:51:43.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:43.842568+0000) 2022-01-31T22:51:44.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:44 smithi167 conmon[54076]: debug 2022-01-31T22:51:44.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:44.082804+0000) 2022-01-31T22:51:44.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:44 smithi171 conmon[41853]: debug 2022-01-31T22:51:44.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:44.278712+0000) 2022-01-31T22:51:44.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:44 smithi171 conmon[46715]: debug 2022-01-31T22:51:44.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:44.327992+0000) 2022-01-31T22:51:44.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:44 smithi171 conmon[51620]: debug 2022-01-31T22:51:44.452+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:44.453418+0000) 2022-01-31T22:51:45.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:44 smithi167 conmon[49112]: debug 2022-01-31T22:51:44.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:44.915228+0000) 2022-01-31T22:51:45.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:44 smithi167 conmon[60316]: debug 2022-01-31T22:51:44.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:44.842756+0000) 2022-01-31T22:51:45.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:45 smithi167 conmon[54076]: debug 2022-01-31T22:51:45.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:45.083003+0000) 2022-01-31T22:51:45.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:45 smithi171 conmon[46715]: debug 2022-01-31T22:51:45.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:45.328125+0000) 2022-01-31T22:51:45.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:45 smithi171 conmon[41853]: debug 2022-01-31T22:51:45.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:45.278883+0000) 2022-01-31T22:51:45.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:45 smithi171 conmon[51620]: debug 2022-01-31T22:51:45.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:45.453571+0000) 2022-01-31T22:51:46.069 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:45 smithi167 conmon[49112]: debug 2022-01-31T22:51:45.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:45.915424+0000) 2022-01-31T22:51:46.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:45 smithi167 conmon[60316]: debug 2022-01-31T22:51:45.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:45.842925+0000) 2022-01-31T22:51:46.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:46 smithi167 conmon[54076]: debug 2022-01-31T22:51:46.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:46.083183+0000) 2022-01-31T22:51:46.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:46 smithi171 conmon[46715]: debug 2022-01-31T22:51:46.327+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:46.328277+0000) 2022-01-31T22:51:46.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:46 smithi171 conmon[41853]: debug 2022-01-31T22:51:46.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:46.279045+0000) 2022-01-31T22:51:46.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:46 smithi171 conmon[51620]: debug 2022-01-31T22:51:46.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:46.453770+0000) 2022-01-31T22:51:47.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:46 smithi167 conmon[49112]: debug 2022-01-31T22:51:46.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:46.915614+0000) 2022-01-31T22:51:47.070 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:46 smithi167 conmon[60316]: debug 2022-01-31T22:51:46.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:46.843089+0000) 2022-01-31T22:51:47.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:47 smithi167 conmon[54076]: debug 2022-01-31T22:51:47.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.083346+0000) 2022-01-31T22:51:47.440 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:47 smithi171 conmon[46715]: debug 2022-01-31T22:51:47.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.328467+0000) 2022-01-31T22:51:47.441 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:47 smithi171 conmon[41853]: debug 2022-01-31T22:51:47.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.279154+0000) 2022-01-31T22:51:47.730 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:47 smithi171 conmon[51620]: debug 2022-01-31T22:51:47.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.453967+0000) 2022-01-31T22:51:47.991 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:47 smithi171 conmon[46715]: debug 2022-01-31T22:51:47.978+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.979337+0000) 2022-01-31T22:51:47.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:47 smithi171 conmon[51620]: debug 2022-01-31T22:51:47.978+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.979446+0000) 2022-01-31T22:51:47.992 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:47 smithi171 conmon[41853]: debug 2022-01-31T22:51:47.980+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.980735+0000) 2022-01-31T22:51:48.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:47 smithi167 conmon[49112]: debug 2022-01-31T22:51:47.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.915743+0000) 2022-01-31T22:51:48.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:47 smithi167 conmon[49112]: debug 2022-01-31T22:51:47.980+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.981135+0000) 2022-01-31T22:51:48.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:47 smithi167 conmon[60316]: debug 2022-01-31T22:51:47.843+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.843277+0000) 2022-01-31T22:51:48.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:47 smithi167 conmon[60316]: debug 2022-01-31T22:51:47.979+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.979763+0000) 2022-01-31T22:51:48.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:47 smithi167 conmon[54076]: debug 2022-01-31T22:51:47.980+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:47.980958+0000) 2022-01-31T22:51:48.158 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:48 smithi167 conmon[54076]: debug 2022-01-31T22:51:48.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:48.083537+0000) 2022-01-31T22:51:48.265 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:48 smithi171 conmon[35325]: debug 2022-01-31T22:51:48.004+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243529 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:48.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:48 smithi171 conmon[46715]: debug 2022-01-31T22:51:48.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:48.328634+0000) 2022-01-31T22:51:48.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:48 smithi171 conmon[41853]: debug 2022-01-31T22:51:48.278+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:48.279343+0000) 2022-01-31T22:51:48.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:48 smithi171 conmon[51620]: debug 2022-01-31T22:51:48.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:48.454249+0000) 2022-01-31T22:51:49.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:48 smithi167 conmon[49112]: debug 2022-01-31T22:51:48.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:48.915933+0000) 2022-01-31T22:51:49.143 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:49 smithi167 conmon[54076]: debug 2022-01-31T22:51:49.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:49.083744+0000) 2022-01-31T22:51:49.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:48 smithi167 conmon[60316]: debug 2022-01-31T22:51:48.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:48.843479+0000) 2022-01-31T22:51:49.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:49 smithi171 conmon[41853]: debug 2022-01-31T22:51:49.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:49.279525+0000) 2022-01-31T22:51:49.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:49 smithi171 conmon[46715]: debug 2022-01-31T22:51:49.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:49.328816+0000) 2022-01-31T22:51:49.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:49 smithi171 conmon[51620]: debug 2022-01-31T22:51:49.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:49.454421+0000) 2022-01-31T22:51:50.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:50 smithi167 conmon[54076]: debug 2022-01-31T22:51:50.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:50.083905+0000) 2022-01-31T22:51:50.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:49 smithi167 conmon[49112]: debug 2022-01-31T22:51:49.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:49.916108+0000) 2022-01-31T22:51:50.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:49 smithi167 conmon[60316]: debug 2022-01-31T22:51:49.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:49.843617+0000) 2022-01-31T22:51:50.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:50 smithi171 conmon[41853]: debug 2022-01-31T22:51:50.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:50.279722+0000) 2022-01-31T22:51:50.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:50 smithi171 conmon[46715]: debug 2022-01-31T22:51:50.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:50.328965+0000) 2022-01-31T22:51:50.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:50 smithi171 conmon[51620]: debug 2022-01-31T22:51:50.453+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:50.454617+0000) 2022-01-31T22:51:51.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:50 smithi167 conmon[49112]: debug 2022-01-31T22:51:50.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:50.916271+0000) 2022-01-31T22:51:51.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:51 smithi167 conmon[54076]: debug 2022-01-31T22:51:51.082+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:51.084132+0000) 2022-01-31T22:51:51.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:50 smithi167 conmon[60316]: debug 2022-01-31T22:51:50.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:50.843789+0000) 2022-01-31T22:51:51.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:51 smithi171 conmon[41853]: debug 2022-01-31T22:51:51.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:51.279946+0000) 2022-01-31T22:51:51.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:51 smithi171 conmon[46715]: debug 2022-01-31T22:51:51.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:51.329174+0000) 2022-01-31T22:51:51.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:51 smithi171 conmon[51620]: debug 2022-01-31T22:51:51.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:51.454815+0000) 2022-01-31T22:51:52.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:51 smithi167 conmon[49112]: debug 2022-01-31T22:51:51.914+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:51.916482+0000) 2022-01-31T22:51:52.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:52 smithi167 conmon[54076]: debug 2022-01-31T22:51:52.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:52.084322+0000) 2022-01-31T22:51:52.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:51 smithi167 conmon[60316]: debug 2022-01-31T22:51:51.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:51.843966+0000) 2022-01-31T22:51:52.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:52 smithi171 conmon[46715]: debug 2022-01-31T22:51:52.328+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:52.329337+0000) 2022-01-31T22:51:52.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:52 smithi171 conmon[41853]: debug 2022-01-31T22:51:52.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:52.280131+0000) 2022-01-31T22:51:52.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:52 smithi171 conmon[51620]: debug 2022-01-31T22:51:52.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:52.455004+0000) 2022-01-31T22:51:52.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[51620]: debug 2022-01-31T22:51:53.005+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.006591+0000) 2022-01-31T22:51:53.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:52 smithi167 conmon[49112]: debug 2022-01-31T22:51:52.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:52.916655+0000) 2022-01-31T22:51:53.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:53 smithi167 conmon[49112]: debug 2022-01-31T22:51:53.006+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.007923+0000) 2022-01-31T22:51:53.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:53 smithi167 conmon[54076]: debug 2022-01-31T22:51:53.007+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.008634+0000) 2022-01-31T22:51:53.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:53 smithi167 conmon[54076]: debug 2022-01-31T22:51:53.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.084514+0000) 2022-01-31T22:51:53.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:52 smithi167 conmon[60316]: debug 2022-01-31T22:51:52.843+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:52.844189+0000) 2022-01-31T22:51:53.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:53 smithi167 conmon[60316]: debug 2022-01-31T22:51:53.006+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.007632+0000) 2022-01-31T22:51:53.266 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[35325]: debug 2022-01-31T22:51:53.032+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243644 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:53.267 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[41853]: debug 2022-01-31T22:51:53.008+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.008910+0000) 2022-01-31T22:51:53.267 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[46715]: debug 2022-01-31T22:51:53.005+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.006683+0000) 2022-01-31T22:51:53.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[41853]: debug 2022-01-31T22:51:53.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.280313+0000) 2022-01-31T22:51:53.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[46715]: debug 2022-01-31T22:51:53.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.329519+0000) 2022-01-31T22:51:53.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[51620]: debug 2022-01-31T22:51:53.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.455194+0000) 2022-01-31T22:51:54.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:53 smithi171 conmon[35325]: debug 2022-01-31T22:51:53.922+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:51:54.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:53 smithi167 conmon[49112]: debug 2022-01-31T22:51:53.915+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.916794+0000) 2022-01-31T22:51:54.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:54 smithi167 conmon[54076]: debug 2022-01-31T22:51:54.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:54.084664+0000) 2022-01-31T22:51:54.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:53 smithi167 conmon[60316]: debug 2022-01-31T22:51:53.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:53.844365+0000) 2022-01-31T22:51:54.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:54 smithi171 conmon[41853]: debug 2022-01-31T22:51:54.279+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:54.280458+0000) 2022-01-31T22:51:54.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:54 smithi171 conmon[51620]: debug 2022-01-31T22:51:54.454+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:54.455352+0000) 2022-01-31T22:51:54.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:54 smithi171 conmon[46715]: debug 2022-01-31T22:51:54.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:54.329680+0000) 2022-01-31T22:51:55.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:54 smithi167 conmon[49112]: debug 2022-01-31T22:51:54.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:54.916940+0000) 2022-01-31T22:51:55.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:54 smithi167 conmon[60316]: debug 2022-01-31T22:51:54.842+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:54.844483+0000) 2022-01-31T22:51:55.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:55 smithi167 conmon[54076]: debug 2022-01-31T22:51:55.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:55.084855+0000) 2022-01-31T22:51:55.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:55 smithi171 conmon[51620]: debug 2022-01-31T22:51:55.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:55.455539+0000) 2022-01-31T22:51:55.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:55 smithi171 conmon[41853]: debug 2022-01-31T22:51:55.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:55.280627+0000) 2022-01-31T22:51:55.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:55 smithi171 conmon[46715]: debug 2022-01-31T22:51:55.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:55.329874+0000) 2022-01-31T22:51:56.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:55 smithi167 conmon[49112]: debug 2022-01-31T22:51:55.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:55.917147+0000) 2022-01-31T22:51:56.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:56 smithi167 conmon[54076]: debug 2022-01-31T22:51:56.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:56.085012+0000) 2022-01-31T22:51:56.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:55 smithi167 conmon[60316]: debug 2022-01-31T22:51:55.843+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:55.844652+0000) 2022-01-31T22:51:56.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:56 smithi171 conmon[41853]: debug 2022-01-31T22:51:56.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:56.280811+0000) 2022-01-31T22:51:56.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:56 smithi171 conmon[51620]: debug 2022-01-31T22:51:56.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:56.455744+0000) 2022-01-31T22:51:56.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:56 smithi171 conmon[46715]: debug 2022-01-31T22:51:56.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:56.330061+0000) 2022-01-31T22:51:57.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:56 smithi167 conmon[49112]: debug 2022-01-31T22:51:56.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:56.917296+0000) 2022-01-31T22:51:57.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:57 smithi167 conmon[54076]: debug 2022-01-31T22:51:57.083+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:57.085174+0000) 2022-01-31T22:51:57.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:56 smithi167 conmon[60316]: debug 2022-01-31T22:51:56.843+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:56.844763+0000) 2022-01-31T22:51:57.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:57 smithi171 conmon[41853]: debug 2022-01-31T22:51:57.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:57.280993+0000) 2022-01-31T22:51:57.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:57 smithi171 conmon[46715]: debug 2022-01-31T22:51:57.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:57.330265+0000) 2022-01-31T22:51:57.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:57 smithi171 conmon[51620]: debug 2022-01-31T22:51:57.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:57.455948+0000) 2022-01-31T22:51:58.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:58 smithi167 conmon[54076]: debug 2022-01-31T22:51:58.035+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.036545+0000) 2022-01-31T22:51:58.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:58 smithi167 conmon[54076]: debug 2022-01-31T22:51:58.084+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.085301+0000) 2022-01-31T22:51:58.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:57 smithi167 conmon[49112]: debug 2022-01-31T22:51:57.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:57.917497+0000) 2022-01-31T22:51:58.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:58 smithi167 conmon[49112]: debug 2022-01-31T22:51:58.034+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.035737+0000) 2022-01-31T22:51:58.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:57 smithi167 conmon[60316]: debug 2022-01-31T22:51:57.844+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:57.844953+0000) 2022-01-31T22:51:58.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:58 smithi167 conmon[60316]: debug 2022-01-31T22:51:58.035+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.037012+0000) 2022-01-31T22:51:58.316 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[35325]: debug 2022-01-31T22:51:58.059+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243753 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:51:58.317 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[46715]: debug 2022-01-31T22:51:58.034+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.035576+0000) 2022-01-31T22:51:58.317 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[51620]: debug 2022-01-31T22:51:58.034+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.035735+0000) 2022-01-31T22:51:58.318 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[41853]: debug 2022-01-31T22:51:58.036+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.036926+0000) 2022-01-31T22:51:58.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[41853]: debug 2022-01-31T22:51:58.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.281181+0000) 2022-01-31T22:51:58.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[51620]: debug 2022-01-31T22:51:58.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.456091+0000) 2022-01-31T22:51:58.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:58 smithi171 conmon[46715]: debug 2022-01-31T22:51:58.329+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.330392+0000) 2022-01-31T22:51:59.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:58 smithi167 conmon[49112]: debug 2022-01-31T22:51:58.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.917699+0000) 2022-01-31T22:51:59.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:51:59 smithi167 conmon[54076]: debug 2022-01-31T22:51:59.084+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:59.085433+0000) 2022-01-31T22:51:59.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:58 smithi167 conmon[60316]: debug 2022-01-31T22:51:58.843+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:58.845159+0000) 2022-01-31T22:51:59.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:51:59 smithi171 conmon[51620]: debug 2022-01-31T22:51:59.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:59.456241+0000) 2022-01-31T22:51:59.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:51:59 smithi171 conmon[41853]: debug 2022-01-31T22:51:59.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:59.281365+0000) 2022-01-31T22:51:59.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:51:59 smithi171 conmon[46715]: debug 2022-01-31T22:51:59.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:59.330545+0000) 2022-01-31T22:52:00.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:51:59 smithi167 conmon[49112]: debug 2022-01-31T22:51:59.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:59.917839+0000) 2022-01-31T22:52:00.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:51:59 smithi167 conmon[60316]: debug 2022-01-31T22:51:59.843+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:51:59.845311+0000) 2022-01-31T22:52:00.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:00 smithi167 conmon[54076]: debug 2022-01-31T22:52:00.084+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:00.085555+0000) 2022-01-31T22:52:00.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:00 smithi171 conmon[41853]: debug 2022-01-31T22:52:00.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:00.281510+0000) 2022-01-31T22:52:00.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:00 smithi171 conmon[46715]: debug 2022-01-31T22:52:00.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:00.330764+0000) 2022-01-31T22:52:00.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:00 smithi171 conmon[51620]: debug 2022-01-31T22:52:00.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:00.456434+0000) 2022-01-31T22:52:01.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:00 smithi167 conmon[60316]: debug 2022-01-31T22:52:00.844+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:00.845515+0000) 2022-01-31T22:52:01.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:00 smithi167 conmon[49112]: debug 2022-01-31T22:52:00.916+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:00.917993+0000) 2022-01-31T22:52:01.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:01 smithi167 conmon[54076]: debug 2022-01-31T22:52:01.084+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:01.085711+0000) 2022-01-31T22:52:01.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:01 smithi171 conmon[46715]: debug 2022-01-31T22:52:01.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:01.330909+0000) 2022-01-31T22:52:01.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:01 smithi171 conmon[51620]: debug 2022-01-31T22:52:01.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:01.456615+0000) 2022-01-31T22:52:01.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:01 smithi171 conmon[41853]: debug 2022-01-31T22:52:01.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:01.281661+0000) 2022-01-31T22:52:02.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:01 smithi167 conmon[49112]: debug 2022-01-31T22:52:01.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:01.918188+0000) 2022-01-31T22:52:02.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:01 smithi167 conmon[60316]: debug 2022-01-31T22:52:01.844+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:01.845675+0000) 2022-01-31T22:52:02.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:02 smithi167 conmon[54076]: debug 2022-01-31T22:52:02.085+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:02.085897+0000) 2022-01-31T22:52:02.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:02 smithi171 conmon[41853]: debug 2022-01-31T22:52:02.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:02.281827+0000) 2022-01-31T22:52:02.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:02 smithi171 conmon[46715]: debug 2022-01-31T22:52:02.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:02.331091+0000) 2022-01-31T22:52:02.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:02 smithi171 conmon[51620]: debug 2022-01-31T22:52:02.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:02.456829+0000) 2022-01-31T22:52:03.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:02 smithi167 conmon[49112]: debug 2022-01-31T22:52:02.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:02.918332+0000) 2022-01-31T22:52:03.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:03 smithi167 conmon[49112]: debug 2022-01-31T22:52:03.060+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.062491+0000) 2022-01-31T22:52:03.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:03 smithi167 conmon[54076]: debug 2022-01-31T22:52:03.062+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.063755+0000) 2022-01-31T22:52:03.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:03 smithi167 conmon[54076]: debug 2022-01-31T22:52:03.084+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.086032+0000) 2022-01-31T22:52:03.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:02 smithi167 conmon[60316]: debug 2022-01-31T22:52:02.845+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:02.845860+0000) 2022-01-31T22:52:03.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:03 smithi167 conmon[60316]: debug 2022-01-31T22:52:03.061+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.063229+0000) 2022-01-31T22:52:03.317 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[35325]: debug 2022-01-31T22:52:03.085+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243867 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:03.318 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[41853]: debug 2022-01-31T22:52:03.061+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.063533+0000) 2022-01-31T22:52:03.318 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[41853]: debug 2022-01-31T22:52:03.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.281946+0000) 2022-01-31T22:52:03.319 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[46715]: debug 2022-01-31T22:52:03.060+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.062055+0000) 2022-01-31T22:52:03.319 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[51620]: debug 2022-01-31T22:52:03.060+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.062133+0000) 2022-01-31T22:52:03.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[46715]: debug 2022-01-31T22:52:03.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.331252+0000) 2022-01-31T22:52:03.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:03 smithi171 conmon[51620]: debug 2022-01-31T22:52:03.455+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.457006+0000) 2022-01-31T22:52:04.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:03 smithi167 conmon[49112]: debug 2022-01-31T22:52:03.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.918528+0000) 2022-01-31T22:52:04.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:04 smithi167 conmon[54076]: debug 2022-01-31T22:52:04.085+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:04.086165+0000) 2022-01-31T22:52:04.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:03 smithi167 conmon[60316]: debug 2022-01-31T22:52:03.844+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:03.846027+0000) 2022-01-31T22:52:04.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:04 smithi171 conmon[41853]: debug 2022-01-31T22:52:04.280+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:04.282098+0000) 2022-01-31T22:52:04.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:04 smithi171 conmon[46715]: debug 2022-01-31T22:52:04.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:04.331410+0000) 2022-01-31T22:52:04.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:04 smithi171 conmon[51620]: debug 2022-01-31T22:52:04.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:04.457160+0000) 2022-01-31T22:52:05.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:04 smithi167 conmon[49112]: debug 2022-01-31T22:52:04.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:04.918684+0000) 2022-01-31T22:52:05.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:05 smithi167 conmon[54076]: debug 2022-01-31T22:52:05.085+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:05.086339+0000) 2022-01-31T22:52:05.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:04 smithi167 conmon[60316]: debug 2022-01-31T22:52:04.845+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:04.846208+0000) 2022-01-31T22:52:05.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:05 smithi171 conmon[41853]: debug 2022-01-31T22:52:05.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:05.282201+0000) 2022-01-31T22:52:05.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:05 smithi171 conmon[46715]: debug 2022-01-31T22:52:05.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:05.331594+0000) 2022-01-31T22:52:05.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:05 smithi171 conmon[51620]: debug 2022-01-31T22:52:05.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:05.457313+0000) 2022-01-31T22:52:06.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:05 smithi167 conmon[49112]: debug 2022-01-31T22:52:05.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:05.918916+0000) 2022-01-31T22:52:06.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:06 smithi167 conmon[54076]: debug 2022-01-31T22:52:06.085+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:06.086536+0000) 2022-01-31T22:52:06.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:05 smithi167 conmon[60316]: debug 2022-01-31T22:52:05.845+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:05.846395+0000) 2022-01-31T22:52:06.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:06 smithi171 conmon[41853]: debug 2022-01-31T22:52:06.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:06.282354+0000) 2022-01-31T22:52:06.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:06 smithi171 conmon[46715]: debug 2022-01-31T22:52:06.330+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:06.331783+0000) 2022-01-31T22:52:06.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:06 smithi171 conmon[51620]: debug 2022-01-31T22:52:06.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:06.457520+0000) 2022-01-31T22:52:07.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:06 smithi167 conmon[49112]: debug 2022-01-31T22:52:06.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:06.919107+0000) 2022-01-31T22:52:07.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:07 smithi167 conmon[54076]: debug 2022-01-31T22:52:07.085+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:07.086686+0000) 2022-01-31T22:52:07.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:06 smithi167 conmon[60316]: debug 2022-01-31T22:52:06.845+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:06.846598+0000) 2022-01-31T22:52:07.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:07 smithi171 conmon[41853]: debug 2022-01-31T22:52:07.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:07.282511+0000) 2022-01-31T22:52:07.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:07 smithi171 conmon[46715]: debug 2022-01-31T22:52:07.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:07.331936+0000) 2022-01-31T22:52:07.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:07 smithi171 conmon[51620]: debug 2022-01-31T22:52:07.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:07.457694+0000) 2022-01-31T22:52:08.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:07 smithi167 conmon[49112]: debug 2022-01-31T22:52:07.918+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:07.919317+0000) 2022-01-31T22:52:08.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:08 smithi167 conmon[49112]: debug 2022-01-31T22:52:08.088+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.089777+0000) 2022-01-31T22:52:08.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:08 smithi167 conmon[54076]: debug 2022-01-31T22:52:08.086+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.086917+0000) 2022-01-31T22:52:08.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:08 smithi167 conmon[54076]: debug 2022-01-31T22:52:08.090+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.091613+0000) 2022-01-31T22:52:08.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:07 smithi167 conmon[60316]: debug 2022-01-31T22:52:07.846+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:07.846810+0000) 2022-01-31T22:52:08.158 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:08 smithi167 conmon[60316]: debug 2022-01-31T22:52:08.089+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.091207+0000) 2022-01-31T22:52:08.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[51620]: debug 2022-01-31T22:52:08.088+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.090208+0000) 2022-01-31T22:52:08.389 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[35325]: debug 2022-01-31T22:52:08.112+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 243976 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:08.390 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[41853]: debug 2022-01-31T22:52:08.089+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.091170+0000) 2022-01-31T22:52:08.391 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[41853]: debug 2022-01-31T22:52:08.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.282654+0000) 2022-01-31T22:52:08.391 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[46715]: debug 2022-01-31T22:52:08.087+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.089613+0000) 2022-01-31T22:52:08.392 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[46715]: debug 2022-01-31T22:52:08.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.332104+0000) 2022-01-31T22:52:08.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[51620]: debug 2022-01-31T22:52:08.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.457899+0000) 2022-01-31T22:52:09.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:08 smithi171 conmon[35325]: debug 2022-01-31T22:52:08.922+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:52:09.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:08 smithi167 conmon[49112]: debug 2022-01-31T22:52:08.917+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.919441+0000) 2022-01-31T22:52:09.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:09 smithi167 conmon[54076]: debug 2022-01-31T22:52:09.086+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:09.087112+0000) 2022-01-31T22:52:09.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:08 smithi167 conmon[60316]: debug 2022-01-31T22:52:08.845+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:08.846959+0000) 2022-01-31T22:52:09.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:09 smithi171 conmon[41853]: debug 2022-01-31T22:52:09.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:09.282824+0000) 2022-01-31T22:52:09.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:09 smithi171 conmon[46715]: debug 2022-01-31T22:52:09.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:09.332251+0000) 2022-01-31T22:52:09.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:09 smithi171 conmon[51620]: debug 2022-01-31T22:52:09.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:09.458005+0000) 2022-01-31T22:52:10.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:09 smithi167 conmon[49112]: debug 2022-01-31T22:52:09.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:09.919614+0000) 2022-01-31T22:52:10.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:10 smithi167 conmon[54076]: debug 2022-01-31T22:52:10.086+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:10.087253+0000) 2022-01-31T22:52:10.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:09 smithi167 conmon[60316]: debug 2022-01-31T22:52:09.846+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:09.847113+0000) 2022-01-31T22:52:10.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:10 smithi171 conmon[41853]: debug 2022-01-31T22:52:10.282+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:10.282990+0000) 2022-01-31T22:52:10.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:10 smithi171 conmon[46715]: debug 2022-01-31T22:52:10.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:10.332448+0000) 2022-01-31T22:52:10.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:10 smithi171 conmon[51620]: debug 2022-01-31T22:52:10.456+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:10.458169+0000) 2022-01-31T22:52:11.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:10 smithi167 conmon[49112]: debug 2022-01-31T22:52:10.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:10.919773+0000) 2022-01-31T22:52:11.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:11 smithi167 conmon[54076]: debug 2022-01-31T22:52:11.086+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:11.087419+0000) 2022-01-31T22:52:11.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:10 smithi167 conmon[60316]: debug 2022-01-31T22:52:10.846+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:10.847300+0000) 2022-01-31T22:52:11.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:11 smithi171 conmon[41853]: debug 2022-01-31T22:52:11.281+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:11.283095+0000) 2022-01-31T22:52:11.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:11 smithi171 conmon[46715]: debug 2022-01-31T22:52:11.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:11.332620+0000) 2022-01-31T22:52:11.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:11 smithi171 conmon[51620]: debug 2022-01-31T22:52:11.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:11.458340+0000) 2022-01-31T22:52:12.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:11 smithi167 conmon[49112]: debug 2022-01-31T22:52:11.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:11.919965+0000) 2022-01-31T22:52:12.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:12 smithi167 conmon[54076]: debug 2022-01-31T22:52:12.087+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:12.087622+0000) 2022-01-31T22:52:12.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:11 smithi167 conmon[60316]: debug 2022-01-31T22:52:11.846+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:11.847456+0000) 2022-01-31T22:52:12.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:12 smithi171 conmon[41853]: debug 2022-01-31T22:52:12.282+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:12.283254+0000) 2022-01-31T22:52:12.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:12 smithi171 conmon[46715]: debug 2022-01-31T22:52:12.331+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:12.332783+0000) 2022-01-31T22:52:12.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:12 smithi171 conmon[51620]: debug 2022-01-31T22:52:12.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:12.458527+0000) 2022-01-31T22:52:13.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:12 smithi167 conmon[49112]: debug 2022-01-31T22:52:12.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:12.920171+0000) 2022-01-31T22:52:13.104 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:13 smithi167 conmon[54076]: debug 2022-01-31T22:52:13.087+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.087813+0000) 2022-01-31T22:52:13.104 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:12 smithi167 conmon[60316]: debug 2022-01-31T22:52:12.847+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:12.847609+0000) 2022-01-31T22:52:13.104 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:13 smithi167 conmon[60316]: debug 2022-01-31T22:52:13.115+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.117245+0000) 2022-01-31T22:52:13.405 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:13 smithi167 conmon[49112]: debug 2022-01-31T22:52:13.116+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.117715+0000) 2022-01-31T22:52:13.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:13 smithi167 conmon[54076]: debug 2022-01-31T22:52:13.116+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.117963+0000) 2022-01-31T22:52:13.444 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[35325]: debug 2022-01-31T22:52:13.139+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244090 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:13.445 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[51620]: debug 2022-01-31T22:52:13.114+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.116293+0000) 2022-01-31T22:52:13.446 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[41853]: debug 2022-01-31T22:52:13.116+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.118733+0000) 2022-01-31T22:52:13.446 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[41853]: debug 2022-01-31T22:52:13.282+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.283428+0000) 2022-01-31T22:52:13.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[46715]: debug 2022-01-31T22:52:13.114+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.116660+0000) 2022-01-31T22:52:13.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[46715]: debug 2022-01-31T22:52:13.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.332895+0000) 2022-01-31T22:52:13.734 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:13 smithi171 conmon[51620]: debug 2022-01-31T22:52:13.457+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.458705+0000) 2022-01-31T22:52:14.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:13 smithi167 conmon[49112]: debug 2022-01-31T22:52:13.919+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.920356+0000) 2022-01-31T22:52:14.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:14 smithi167 conmon[54076]: debug 2022-01-31T22:52:14.087+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:14.087967+0000) 2022-01-31T22:52:14.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:13 smithi167 conmon[60316]: debug 2022-01-31T22:52:13.847+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:13.847808+0000) 2022-01-31T22:52:14.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:14 smithi171 conmon[41853]: debug 2022-01-31T22:52:14.282+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:14.283568+0000) 2022-01-31T22:52:14.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:14 smithi171 conmon[46715]: debug 2022-01-31T22:52:14.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:14.333021+0000) 2022-01-31T22:52:14.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:14 smithi171 conmon[51620]: debug 2022-01-31T22:52:14.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:14.458855+0000) 2022-01-31T22:52:15.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:14 smithi167 conmon[60316]: debug 2022-01-31T22:52:14.847+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:14.847990+0000) 2022-01-31T22:52:15.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:14 smithi167 conmon[49112]: debug 2022-01-31T22:52:14.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:14.920543+0000) 2022-01-31T22:52:15.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:15 smithi167 conmon[54076]: debug 2022-01-31T22:52:15.087+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:15.088110+0000) 2022-01-31T22:52:15.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:15 smithi171 conmon[41853]: debug 2022-01-31T22:52:15.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:15.283735+0000) 2022-01-31T22:52:15.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:15 smithi171 conmon[46715]: debug 2022-01-31T22:52:15.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:15.333181+0000) 2022-01-31T22:52:15.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:15 smithi171 conmon[51620]: debug 2022-01-31T22:52:15.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:15.459033+0000) 2022-01-31T22:52:16.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:15 smithi167 conmon[60316]: debug 2022-01-31T22:52:15.847+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:15.848181+0000) 2022-01-31T22:52:16.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:15 smithi167 conmon[49112]: debug 2022-01-31T22:52:15.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:15.920766+0000) 2022-01-31T22:52:16.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:16 smithi167 conmon[54076]: debug 2022-01-31T22:52:16.087+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:16.088254+0000) 2022-01-31T22:52:16.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:16 smithi171 conmon[41853]: debug 2022-01-31T22:52:16.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:16.283953+0000) 2022-01-31T22:52:16.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:16 smithi171 conmon[46715]: debug 2022-01-31T22:52:16.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:16.333368+0000) 2022-01-31T22:52:16.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:16 smithi171 conmon[51620]: debug 2022-01-31T22:52:16.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:16.459223+0000) 2022-01-31T22:52:17.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:16 smithi167 conmon[60316]: debug 2022-01-31T22:52:16.847+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:16.848388+0000) 2022-01-31T22:52:17.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:16 smithi167 conmon[49112]: debug 2022-01-31T22:52:16.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:16.920954+0000) 2022-01-31T22:52:17.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:17 smithi167 conmon[54076]: debug 2022-01-31T22:52:17.087+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:17.088461+0000) 2022-01-31T22:52:17.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:17 smithi171 conmon[41853]: debug 2022-01-31T22:52:17.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:17.284114+0000) 2022-01-31T22:52:17.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:17 smithi171 conmon[46715]: debug 2022-01-31T22:52:17.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:17.333551+0000) 2022-01-31T22:52:17.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:17 smithi171 conmon[51620]: debug 2022-01-31T22:52:17.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:17.459407+0000) 2022-01-31T22:52:18.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:17 smithi167 conmon[49112]: debug 2022-01-31T22:52:17.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:17.921142+0000) 2022-01-31T22:52:18.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:17 smithi167 conmon[60316]: debug 2022-01-31T22:52:17.848+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:17.848551+0000) 2022-01-31T22:52:18.131 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:18 smithi167 conmon[54076]: debug 2022-01-31T22:52:18.088+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.088617+0000) 2022-01-31T22:52:18.405 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:18 smithi167 conmon[49112]: debug 2022-01-31T22:52:18.143+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.143980+0000) 2022-01-31T22:52:18.406 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:18 smithi167 conmon[54076]: debug 2022-01-31T22:52:18.143+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.144246+0000) 2022-01-31T22:52:18.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:18 smithi167 conmon[60316]: debug 2022-01-31T22:52:18.144+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.144811+0000) 2022-01-31T22:52:18.445 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[35325]: debug 2022-01-31T22:52:18.173+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244202 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:18.446 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[51620]: debug 2022-01-31T22:52:18.143+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.144593+0000) 2022-01-31T22:52:18.446 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[41853]: debug 2022-01-31T22:52:18.144+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.145010+0000) 2022-01-31T22:52:18.447 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[41853]: debug 2022-01-31T22:52:18.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.284321+0000) 2022-01-31T22:52:18.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[46715]: debug 2022-01-31T22:52:18.142+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.143506+0000) 2022-01-31T22:52:18.448 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[46715]: debug 2022-01-31T22:52:18.332+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.333700+0000) 2022-01-31T22:52:18.735 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:18 smithi171 conmon[51620]: debug 2022-01-31T22:52:18.458+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.459589+0000) 2022-01-31T22:52:19.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:18 smithi167 conmon[49112]: debug 2022-01-31T22:52:18.920+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.921330+0000) 2022-01-31T22:52:19.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:19 smithi167 conmon[54076]: debug 2022-01-31T22:52:19.088+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:19.088804+0000) 2022-01-31T22:52:19.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:18 smithi167 conmon[60316]: debug 2022-01-31T22:52:18.848+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:18.848753+0000) 2022-01-31T22:52:19.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:19 smithi171 conmon[41853]: debug 2022-01-31T22:52:19.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:19.284462+0000) 2022-01-31T22:52:19.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:19 smithi171 conmon[46715]: debug 2022-01-31T22:52:19.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:19.333854+0000) 2022-01-31T22:52:19.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:19 smithi171 conmon[51620]: debug 2022-01-31T22:52:19.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:19.459747+0000) 2022-01-31T22:52:20.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:19 smithi167 conmon[49112]: debug 2022-01-31T22:52:19.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:19.921527+0000) 2022-01-31T22:52:20.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:20 smithi167 conmon[54076]: debug 2022-01-31T22:52:20.088+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:20.088982+0000) 2022-01-31T22:52:20.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:19 smithi167 conmon[60316]: debug 2022-01-31T22:52:19.848+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:19.848884+0000) 2022-01-31T22:52:20.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:20 smithi171 conmon[41853]: debug 2022-01-31T22:52:20.283+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:20.284615+0000) 2022-01-31T22:52:20.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:20 smithi171 conmon[46715]: debug 2022-01-31T22:52:20.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:20.334013+0000) 2022-01-31T22:52:20.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:20 smithi171 conmon[51620]: debug 2022-01-31T22:52:20.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:20.459956+0000) 2022-01-31T22:52:21.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:20 smithi167 conmon[49112]: debug 2022-01-31T22:52:20.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:20.921744+0000) 2022-01-31T22:52:21.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:20 smithi167 conmon[60316]: debug 2022-01-31T22:52:20.848+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:20.849039+0000) 2022-01-31T22:52:21.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:21 smithi167 conmon[54076]: debug 2022-01-31T22:52:21.088+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:21.089197+0000) 2022-01-31T22:52:21.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:21 smithi171 conmon[41853]: debug 2022-01-31T22:52:21.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:21.284801+0000) 2022-01-31T22:52:21.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:21 smithi171 conmon[46715]: debug 2022-01-31T22:52:21.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:21.334168+0000) 2022-01-31T22:52:21.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:21 smithi171 conmon[51620]: debug 2022-01-31T22:52:21.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:21.460130+0000) 2022-01-31T22:52:22.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:21 smithi167 conmon[49112]: debug 2022-01-31T22:52:21.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:21.921906+0000) 2022-01-31T22:52:22.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:21 smithi167 conmon[60316]: debug 2022-01-31T22:52:21.848+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:21.849228+0000) 2022-01-31T22:52:22.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:22 smithi167 conmon[54076]: debug 2022-01-31T22:52:22.089+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:22.089396+0000) 2022-01-31T22:52:22.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:22 smithi171 conmon[41853]: debug 2022-01-31T22:52:22.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:22.284992+0000) 2022-01-31T22:52:22.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:22 smithi171 conmon[46715]: debug 2022-01-31T22:52:22.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:22.334336+0000) 2022-01-31T22:52:22.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:22 smithi171 conmon[51620]: debug 2022-01-31T22:52:22.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:22.460349+0000) 2022-01-31T22:52:23.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:22 smithi167 conmon[49112]: debug 2022-01-31T22:52:22.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:22.922085+0000) 2022-01-31T22:52:23.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:23 smithi167 conmon[54076]: debug 2022-01-31T22:52:23.089+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.089585+0000) 2022-01-31T22:52:23.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:22 smithi167 conmon[60316]: debug 2022-01-31T22:52:22.849+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:22.849420+0000) 2022-01-31T22:52:23.446 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[35325]: debug 2022-01-31T22:52:23.201+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244315 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:23.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[46715]: debug 2022-01-31T22:52:23.175+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.176668+0000) 2022-01-31T22:52:23.447 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[46715]: debug 2022-01-31T22:52:23.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.334505+0000) 2022-01-31T22:52:23.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[51620]: debug 2022-01-31T22:52:23.175+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.176250+0000) 2022-01-31T22:52:23.448 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[41853]: debug 2022-01-31T22:52:23.177+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.178165+0000) 2022-01-31T22:52:23.449 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[41853]: debug 2022-01-31T22:52:23.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.285156+0000) 2022-01-31T22:52:23.570 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:23 smithi167 conmon[49112]: debug 2022-01-31T22:52:23.183+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.183814+0000) 2022-01-31T22:52:23.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:23 smithi167 conmon[54076]: debug 2022-01-31T22:52:23.177+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.177620+0000) 2022-01-31T22:52:23.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:23 smithi167 conmon[60316]: debug 2022-01-31T22:52:23.176+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.176747+0000) 2022-01-31T22:52:23.736 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[51620]: debug 2022-01-31T22:52:23.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.460538+0000) 2022-01-31T22:52:24.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:23 smithi171 conmon[35325]: debug 2022-01-31T22:52:23.923+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:52:24.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:23 smithi167 conmon[49112]: debug 2022-01-31T22:52:23.921+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.922300+0000) 2022-01-31T22:52:24.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:24 smithi167 conmon[54076]: debug 2022-01-31T22:52:24.089+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:24.089799+0000) 2022-01-31T22:52:24.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:23 smithi167 conmon[60316]: debug 2022-01-31T22:52:23.849+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:23.849604+0000) 2022-01-31T22:52:24.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:24 smithi171 conmon[41853]: debug 2022-01-31T22:52:24.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:24.285303+0000) 2022-01-31T22:52:24.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:24 smithi171 conmon[46715]: debug 2022-01-31T22:52:24.333+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:24.334645+0000) 2022-01-31T22:52:24.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:24 smithi171 conmon[51620]: debug 2022-01-31T22:52:24.459+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:24.460697+0000) 2022-01-31T22:52:25.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:25 smithi167 conmon[54076]: debug 2022-01-31T22:52:25.089+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:25.089955+0000) 2022-01-31T22:52:25.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:24 smithi167 conmon[60316]: debug 2022-01-31T22:52:24.849+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:24.849762+0000) 2022-01-31T22:52:25.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:24 smithi167 conmon[49112]: debug 2022-01-31T22:52:24.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:24.922451+0000) 2022-01-31T22:52:25.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:25 smithi171 conmon[41853]: debug 2022-01-31T22:52:25.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:25.285494+0000) 2022-01-31T22:52:25.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:25 smithi171 conmon[46715]: debug 2022-01-31T22:52:25.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:25.334833+0000) 2022-01-31T22:52:25.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:25 smithi171 conmon[51620]: debug 2022-01-31T22:52:25.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:25.460910+0000) 2022-01-31T22:52:26.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:26 smithi167 conmon[54076]: debug 2022-01-31T22:52:26.089+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:26.090139+0000) 2022-01-31T22:52:26.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:25 smithi167 conmon[60316]: debug 2022-01-31T22:52:25.849+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:25.849861+0000) 2022-01-31T22:52:26.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:25 smithi167 conmon[49112]: debug 2022-01-31T22:52:25.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:25.922653+0000) 2022-01-31T22:52:26.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:26 smithi171 conmon[41853]: debug 2022-01-31T22:52:26.284+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:26.285695+0000) 2022-01-31T22:52:26.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:26 smithi171 conmon[51620]: debug 2022-01-31T22:52:26.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:26.461103+0000) 2022-01-31T22:52:26.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:26 smithi171 conmon[46715]: debug 2022-01-31T22:52:26.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:26.335017+0000) 2022-01-31T22:52:27.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:27 smithi167 conmon[54076]: debug 2022-01-31T22:52:27.090+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:27.090279+0000) 2022-01-31T22:52:27.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:26 smithi167 conmon[49112]: debug 2022-01-31T22:52:26.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:26.922851+0000) 2022-01-31T22:52:27.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:26 smithi167 conmon[60316]: debug 2022-01-31T22:52:26.849+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:26.850054+0000) 2022-01-31T22:52:27.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:27 smithi171 conmon[41853]: debug 2022-01-31T22:52:27.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:27.285900+0000) 2022-01-31T22:52:27.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:27 smithi171 conmon[46715]: debug 2022-01-31T22:52:27.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:27.335199+0000) 2022-01-31T22:52:27.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:27 smithi171 conmon[51620]: debug 2022-01-31T22:52:27.460+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:27.461289+0000) 2022-01-31T22:52:28.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:28 smithi167 conmon[54076]: debug 2022-01-31T22:52:28.090+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.090451+0000) 2022-01-31T22:52:28.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:27 smithi167 conmon[60316]: debug 2022-01-31T22:52:27.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:27.850241+0000) 2022-01-31T22:52:28.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:27 smithi167 conmon[49112]: debug 2022-01-31T22:52:27.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:27.923067+0000) 2022-01-31T22:52:28.447 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[35325]: debug 2022-01-31T22:52:28.229+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244425 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:28.448 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[51620]: debug 2022-01-31T22:52:28.203+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.204571+0000) 2022-01-31T22:52:28.448 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[41853]: debug 2022-01-31T22:52:28.205+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.205894+0000) 2022-01-31T22:52:28.449 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[41853]: debug 2022-01-31T22:52:28.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.286054+0000) 2022-01-31T22:52:28.449 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[46715]: debug 2022-01-31T22:52:28.204+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.205572+0000) 2022-01-31T22:52:28.450 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[46715]: debug 2022-01-31T22:52:28.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.335374+0000) 2022-01-31T22:52:28.571 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:28 smithi167 conmon[49112]: debug 2022-01-31T22:52:28.205+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.205726+0000) 2022-01-31T22:52:28.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:28 smithi167 conmon[60316]: debug 2022-01-31T22:52:28.203+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.204071+0000) 2022-01-31T22:52:28.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:28 smithi167 conmon[54076]: debug 2022-01-31T22:52:28.205+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.205686+0000) 2022-01-31T22:52:28.737 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:28 smithi171 conmon[51620]: debug 2022-01-31T22:52:28.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.461469+0000) 2022-01-31T22:52:29.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:28 smithi167 conmon[60316]: debug 2022-01-31T22:52:28.849+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.850446+0000) 2022-01-31T22:52:29.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:28 smithi167 conmon[49112]: debug 2022-01-31T22:52:28.922+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:28.923267+0000) 2022-01-31T22:52:29.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:29 smithi167 conmon[54076]: debug 2022-01-31T22:52:29.090+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:29.090703+0000) 2022-01-31T22:52:29.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:29 smithi171 conmon[41853]: debug 2022-01-31T22:52:29.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:29.286206+0000) 2022-01-31T22:52:29.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:29 smithi171 conmon[46715]: debug 2022-01-31T22:52:29.334+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:29.335543+0000) 2022-01-31T22:52:29.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:29 smithi171 conmon[51620]: debug 2022-01-31T22:52:29.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:29.461621+0000) 2022-01-31T22:52:30.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:30 smithi167 conmon[54076]: debug 2022-01-31T22:52:30.090+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:30.090840+0000) 2022-01-31T22:52:30.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:29 smithi167 conmon[60316]: debug 2022-01-31T22:52:29.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:29.850626+0000) 2022-01-31T22:52:30.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:29 smithi167 conmon[49112]: debug 2022-01-31T22:52:29.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:29.923422+0000) 2022-01-31T22:52:30.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:30 smithi171 conmon[41853]: debug 2022-01-31T22:52:30.285+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:30.286398+0000) 2022-01-31T22:52:30.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:30 smithi171 conmon[46715]: debug 2022-01-31T22:52:30.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:30.335701+0000) 2022-01-31T22:52:30.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:30 smithi171 conmon[51620]: debug 2022-01-31T22:52:30.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:30.461835+0000) 2022-01-31T22:52:31.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:30 smithi167 conmon[49112]: debug 2022-01-31T22:52:30.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:30.923662+0000) 2022-01-31T22:52:31.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:31 smithi167 conmon[54076]: debug 2022-01-31T22:52:31.090+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:31.091009+0000) 2022-01-31T22:52:31.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:30 smithi167 conmon[60316]: debug 2022-01-31T22:52:30.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:30.850788+0000) 2022-01-31T22:52:31.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:31 smithi171 conmon[41853]: debug 2022-01-31T22:52:31.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:31.286579+0000) 2022-01-31T22:52:31.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:31 smithi171 conmon[46715]: debug 2022-01-31T22:52:31.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:31.335918+0000) 2022-01-31T22:52:31.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:31 smithi171 conmon[51620]: debug 2022-01-31T22:52:31.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:31.462037+0000) 2022-01-31T22:52:32.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:31 smithi167 conmon[49112]: debug 2022-01-31T22:52:31.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:31.923788+0000) 2022-01-31T22:52:32.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:32 smithi167 conmon[54076]: debug 2022-01-31T22:52:32.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:32.091175+0000) 2022-01-31T22:52:32.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:31 smithi167 conmon[60316]: debug 2022-01-31T22:52:31.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:31.850922+0000) 2022-01-31T22:52:32.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:32 smithi171 conmon[41853]: debug 2022-01-31T22:52:32.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:32.286767+0000) 2022-01-31T22:52:32.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:32 smithi171 conmon[46715]: debug 2022-01-31T22:52:32.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:32.336135+0000) 2022-01-31T22:52:32.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:32 smithi171 conmon[51620]: debug 2022-01-31T22:52:32.461+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:32.462196+0000) 2022-01-31T22:52:33.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:33 smithi167 conmon[54076]: debug 2022-01-31T22:52:33.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.091370+0000) 2022-01-31T22:52:33.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:32 smithi167 conmon[49112]: debug 2022-01-31T22:52:32.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:32.923976+0000) 2022-01-31T22:52:33.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:32 smithi167 conmon[60316]: debug 2022-01-31T22:52:32.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:32.851077+0000) 2022-01-31T22:52:33.571 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:33 smithi167 conmon[49112]: debug 2022-01-31T22:52:33.233+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.233743+0000) 2022-01-31T22:52:33.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:33 smithi167 conmon[54076]: debug 2022-01-31T22:52:33.231+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.232338+0000) 2022-01-31T22:52:33.572 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:33 smithi167 conmon[60316]: debug 2022-01-31T22:52:33.232+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.232882+0000) 2022-01-31T22:52:33.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[35325]: debug 2022-01-31T22:52:33.257+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244538 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:33.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[41853]: debug 2022-01-31T22:52:33.233+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.234025+0000) 2022-01-31T22:52:33.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[41853]: debug 2022-01-31T22:52:33.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.286932+0000) 2022-01-31T22:52:33.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[46715]: debug 2022-01-31T22:52:33.231+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.232242+0000) 2022-01-31T22:52:33.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[46715]: debug 2022-01-31T22:52:33.335+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.336332+0000) 2022-01-31T22:52:33.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[51620]: debug 2022-01-31T22:52:33.231+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.232740+0000) 2022-01-31T22:52:33.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:33 smithi171 conmon[51620]: debug 2022-01-31T22:52:33.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.462416+0000) 2022-01-31T22:52:34.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:33 smithi167 conmon[49112]: debug 2022-01-31T22:52:33.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.924182+0000) 2022-01-31T22:52:34.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:34 smithi167 conmon[54076]: debug 2022-01-31T22:52:34.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:34.091561+0000) 2022-01-31T22:52:34.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:33 smithi167 conmon[60316]: debug 2022-01-31T22:52:33.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:33.851249+0000) 2022-01-31T22:52:34.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:34 smithi171 conmon[41853]: debug 2022-01-31T22:52:34.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:34.287076+0000) 2022-01-31T22:52:34.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:34 smithi171 conmon[46715]: debug 2022-01-31T22:52:34.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:34.336521+0000) 2022-01-31T22:52:34.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:34 smithi171 conmon[51620]: debug 2022-01-31T22:52:34.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:34.462571+0000) 2022-01-31T22:52:35.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:34 smithi167 conmon[49112]: debug 2022-01-31T22:52:34.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:34.924337+0000) 2022-01-31T22:52:35.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:35 smithi167 conmon[54076]: debug 2022-01-31T22:52:35.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:35.091756+0000) 2022-01-31T22:52:35.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:34 smithi167 conmon[60316]: debug 2022-01-31T22:52:34.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:34.851406+0000) 2022-01-31T22:52:35.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:35 smithi171 conmon[41853]: debug 2022-01-31T22:52:35.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:35.287255+0000) 2022-01-31T22:52:35.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:35 smithi171 conmon[46715]: debug 2022-01-31T22:52:35.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:35.336734+0000) 2022-01-31T22:52:35.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:35 smithi171 conmon[51620]: debug 2022-01-31T22:52:35.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:35.462785+0000) 2022-01-31T22:52:36.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:36 smithi167 conmon[54076]: debug 2022-01-31T22:52:36.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:36.091964+0000) 2022-01-31T22:52:36.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:35 smithi167 conmon[49112]: debug 2022-01-31T22:52:35.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:35.924470+0000) 2022-01-31T22:52:36.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:35 smithi167 conmon[60316]: debug 2022-01-31T22:52:35.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:35.851561+0000) 2022-01-31T22:52:36.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:36 smithi171 conmon[41853]: debug 2022-01-31T22:52:36.286+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:36.287434+0000) 2022-01-31T22:52:36.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:36 smithi171 conmon[46715]: debug 2022-01-31T22:52:36.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:36.336953+0000) 2022-01-31T22:52:36.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:36 smithi171 conmon[51620]: debug 2022-01-31T22:52:36.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:36.462970+0000) 2022-01-31T22:52:37.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:37 smithi167 conmon[54076]: debug 2022-01-31T22:52:37.092+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:37.092121+0000) 2022-01-31T22:52:37.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:36 smithi167 conmon[49112]: debug 2022-01-31T22:52:36.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:36.924646+0000) 2022-01-31T22:52:37.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:36 smithi167 conmon[60316]: debug 2022-01-31T22:52:36.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:36.851773+0000) 2022-01-31T22:52:37.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:37 smithi171 conmon[41853]: debug 2022-01-31T22:52:37.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:37.287614+0000) 2022-01-31T22:52:37.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:37 smithi171 conmon[46715]: debug 2022-01-31T22:52:37.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:37.337151+0000) 2022-01-31T22:52:37.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:37 smithi171 conmon[51620]: debug 2022-01-31T22:52:37.462+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:37.463152+0000) 2022-01-31T22:52:38.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:37 smithi167 conmon[49112]: debug 2022-01-31T22:52:37.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:37.924860+0000) 2022-01-31T22:52:38.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:38 smithi167 conmon[54076]: debug 2022-01-31T22:52:38.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.092320+0000) 2022-01-31T22:52:38.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:37 smithi167 conmon[60316]: debug 2022-01-31T22:52:37.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:37.851911+0000) 2022-01-31T22:52:38.572 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:38 smithi167 conmon[49112]: debug 2022-01-31T22:52:38.258+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.260293+0000) 2022-01-31T22:52:38.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:38 smithi167 conmon[54076]: debug 2022-01-31T22:52:38.258+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.259841+0000) 2022-01-31T22:52:38.573 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:38 smithi167 conmon[60316]: debug 2022-01-31T22:52:38.259+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.260594+0000) 2022-01-31T22:52:38.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[35325]: debug 2022-01-31T22:52:38.284+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244649 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:38.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[46715]: debug 2022-01-31T22:52:38.259+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.260016+0000) 2022-01-31T22:52:38.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[46715]: debug 2022-01-31T22:52:38.336+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.337348+0000) 2022-01-31T22:52:38.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[41853]: debug 2022-01-31T22:52:38.259+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.260132+0000) 2022-01-31T22:52:38.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[41853]: debug 2022-01-31T22:52:38.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.287767+0000) 2022-01-31T22:52:38.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[51620]: debug 2022-01-31T22:52:38.259+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.260595+0000) 2022-01-31T22:52:38.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[51620]: debug 2022-01-31T22:52:38.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.463336+0000) 2022-01-31T22:52:39.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:38 smithi171 conmon[35325]: debug 2022-01-31T22:52:38.924+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:52:39.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:38 smithi167 conmon[49112]: debug 2022-01-31T22:52:38.923+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.924994+0000) 2022-01-31T22:52:39.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:39 smithi167 conmon[54076]: debug 2022-01-31T22:52:39.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:39.092532+0000) 2022-01-31T22:52:39.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:38 smithi167 conmon[60316]: debug 2022-01-31T22:52:38.850+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:38.852098+0000) 2022-01-31T22:52:39.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:39 smithi171 conmon[46715]: debug 2022-01-31T22:52:39.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:39.337545+0000) 2022-01-31T22:52:39.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:39 smithi171 conmon[41853]: debug 2022-01-31T22:52:39.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:39.287958+0000) 2022-01-31T22:52:39.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:39 smithi171 conmon[51620]: debug 2022-01-31T22:52:39.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:39.463527+0000) 2022-01-31T22:52:40.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:39 smithi167 conmon[49112]: debug 2022-01-31T22:52:39.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:39.925173+0000) 2022-01-31T22:52:40.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:40 smithi167 conmon[54076]: debug 2022-01-31T22:52:40.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:40.092684+0000) 2022-01-31T22:52:40.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:39 smithi167 conmon[60316]: debug 2022-01-31T22:52:39.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:39.852199+0000) 2022-01-31T22:52:40.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:40 smithi171 conmon[41853]: debug 2022-01-31T22:52:40.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:40.288128+0000) 2022-01-31T22:52:40.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:40 smithi171 conmon[46715]: debug 2022-01-31T22:52:40.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:40.337784+0000) 2022-01-31T22:52:40.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:40 smithi171 conmon[51620]: debug 2022-01-31T22:52:40.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:40.463730+0000) 2022-01-31T22:52:41.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:40 smithi167 conmon[49112]: debug 2022-01-31T22:52:40.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:40.925373+0000) 2022-01-31T22:52:41.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:41 smithi167 conmon[54076]: debug 2022-01-31T22:52:41.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:41.092942+0000) 2022-01-31T22:52:41.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:40 smithi167 conmon[60316]: debug 2022-01-31T22:52:40.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:40.852411+0000) 2022-01-31T22:52:41.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:41 smithi171 conmon[46715]: debug 2022-01-31T22:52:41.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:41.337975+0000) 2022-01-31T22:52:41.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:41 smithi171 conmon[41853]: debug 2022-01-31T22:52:41.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:41.288267+0000) 2022-01-31T22:52:41.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:41 smithi171 conmon[51620]: debug 2022-01-31T22:52:41.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:41.463962+0000) 2022-01-31T22:52:42.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:42 smithi167 conmon[54076]: debug 2022-01-31T22:52:42.091+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:42.093119+0000) 2022-01-31T22:52:42.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:41 smithi167 conmon[49112]: debug 2022-01-31T22:52:41.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:41.925536+0000) 2022-01-31T22:52:42.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:41 smithi167 conmon[60316]: debug 2022-01-31T22:52:41.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:41.852598+0000) 2022-01-31T22:52:42.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:42 smithi171 conmon[46715]: debug 2022-01-31T22:52:42.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:42.338157+0000) 2022-01-31T22:52:42.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:42 smithi171 conmon[41853]: debug 2022-01-31T22:52:42.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:42.288418+0000) 2022-01-31T22:52:42.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:42 smithi171 conmon[51620]: debug 2022-01-31T22:52:42.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:42.464194+0000) 2022-01-31T22:52:43.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:43 smithi167 conmon[54076]: debug 2022-01-31T22:52:43.092+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.093301+0000) 2022-01-31T22:52:43.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:42 smithi167 conmon[60316]: debug 2022-01-31T22:52:42.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:42.852799+0000) 2022-01-31T22:52:43.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:42 smithi167 conmon[49112]: debug 2022-01-31T22:52:42.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:42.925724+0000) 2022-01-31T22:52:43.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:43 smithi167 conmon[54076]: debug 2022-01-31T22:52:43.286+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.288154+0000) 2022-01-31T22:52:43.574 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:43 smithi167 conmon[60316]: debug 2022-01-31T22:52:43.286+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.288350+0000) 2022-01-31T22:52:43.574 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:43 smithi167 conmon[49112]: debug 2022-01-31T22:52:43.287+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.288529+0000) 2022-01-31T22:52:43.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[35325]: debug 2022-01-31T22:52:43.312+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244761 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:43.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[51620]: debug 2022-01-31T22:52:43.288+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.289001+0000) 2022-01-31T22:52:43.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[51620]: debug 2022-01-31T22:52:43.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.464399+0000) 2022-01-31T22:52:43.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[41853]: debug 2022-01-31T22:52:43.287+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.288107+0000) 2022-01-31T22:52:43.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[41853]: debug 2022-01-31T22:52:43.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.288524+0000) 2022-01-31T22:52:43.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[46715]: debug 2022-01-31T22:52:43.286+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.287508+0000) 2022-01-31T22:52:43.600 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:43 smithi171 conmon[46715]: debug 2022-01-31T22:52:43.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.338355+0000) 2022-01-31T22:52:44.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:43 smithi167 conmon[49112]: debug 2022-01-31T22:52:43.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.925847+0000) 2022-01-31T22:52:44.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:44 smithi167 conmon[54076]: debug 2022-01-31T22:52:44.092+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:44.093487+0000) 2022-01-31T22:52:44.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:43 smithi167 conmon[60316]: debug 2022-01-31T22:52:43.851+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:43.853007+0000) 2022-01-31T22:52:44.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:44 smithi171 conmon[51620]: debug 2022-01-31T22:52:44.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:44.464544+0000) 2022-01-31T22:52:44.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:44 smithi171 conmon[41853]: debug 2022-01-31T22:52:44.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:44.288643+0000) 2022-01-31T22:52:44.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:44 smithi171 conmon[46715]: debug 2022-01-31T22:52:44.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:44.338535+0000) 2022-01-31T22:52:45.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:44 smithi167 conmon[49112]: debug 2022-01-31T22:52:44.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:44.926004+0000) 2022-01-31T22:52:45.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:45 smithi167 conmon[54076]: debug 2022-01-31T22:52:45.092+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:45.093690+0000) 2022-01-31T22:52:45.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:44 smithi167 conmon[60316]: debug 2022-01-31T22:52:44.852+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:44.853184+0000) 2022-01-31T22:52:45.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:45 smithi171 conmon[41853]: debug 2022-01-31T22:52:45.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:45.288811+0000) 2022-01-31T22:52:45.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:45 smithi171 conmon[46715]: debug 2022-01-31T22:52:45.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:45.338691+0000) 2022-01-31T22:52:45.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:45 smithi171 conmon[51620]: debug 2022-01-31T22:52:45.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:45.464663+0000) 2022-01-31T22:52:46.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:45 smithi167 conmon[49112]: debug 2022-01-31T22:52:45.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:45.926166+0000) 2022-01-31T22:52:46.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:46 smithi167 conmon[54076]: debug 2022-01-31T22:52:46.092+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:46.093818+0000) 2022-01-31T22:52:46.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:45 smithi167 conmon[60316]: debug 2022-01-31T22:52:45.852+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:45.853384+0000) 2022-01-31T22:52:46.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:46 smithi171 conmon[46715]: debug 2022-01-31T22:52:46.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:46.338872+0000) 2022-01-31T22:52:46.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:46 smithi171 conmon[51620]: debug 2022-01-31T22:52:46.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:46.464796+0000) 2022-01-31T22:52:46.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:46 smithi171 conmon[41853]: debug 2022-01-31T22:52:46.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:46.288961+0000) 2022-01-31T22:52:47.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:46 smithi167 conmon[49112]: debug 2022-01-31T22:52:46.924+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:46.926353+0000) 2022-01-31T22:52:47.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:47 smithi167 conmon[54076]: debug 2022-01-31T22:52:47.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:47.093969+0000) 2022-01-31T22:52:47.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:46 smithi167 conmon[60316]: debug 2022-01-31T22:52:46.852+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:46.853567+0000) 2022-01-31T22:52:47.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:47 smithi171 conmon[51620]: debug 2022-01-31T22:52:47.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:47.465002+0000) 2022-01-31T22:52:47.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:47 smithi171 conmon[41853]: debug 2022-01-31T22:52:47.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:47.289092+0000) 2022-01-31T22:52:47.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:47 smithi171 conmon[46715]: debug 2022-01-31T22:52:47.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:47.339041+0000) 2022-01-31T22:52:48.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:47 smithi167 conmon[49112]: debug 2022-01-31T22:52:47.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:47.926545+0000) 2022-01-31T22:52:48.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:48 smithi167 conmon[54076]: debug 2022-01-31T22:52:48.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.094182+0000) 2022-01-31T22:52:48.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:47 smithi167 conmon[60316]: debug 2022-01-31T22:52:47.852+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:47.853751+0000) 2022-01-31T22:52:48.574 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:48 smithi167 conmon[49112]: debug 2022-01-31T22:52:48.313+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.315312+0000) 2022-01-31T22:52:48.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:48 smithi167 conmon[54076]: debug 2022-01-31T22:52:48.314+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.315632+0000) 2022-01-31T22:52:48.575 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:48 smithi167 conmon[60316]: debug 2022-01-31T22:52:48.314+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.315700+0000) 2022-01-31T22:52:48.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[35325]: debug 2022-01-31T22:52:48.339+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244877 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:48.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[41853]: debug 2022-01-31T22:52:48.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.289258+0000) 2022-01-31T22:52:48.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[41853]: debug 2022-01-31T22:52:48.313+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.315224+0000) 2022-01-31T22:52:48.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[46715]: debug 2022-01-31T22:52:48.312+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.314673+0000) 2022-01-31T22:52:48.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[46715]: debug 2022-01-31T22:52:48.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.339178+0000) 2022-01-31T22:52:48.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[51620]: debug 2022-01-31T22:52:48.314+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.316297+0000) 2022-01-31T22:52:48.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:48 smithi171 conmon[51620]: debug 2022-01-31T22:52:48.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.465130+0000) 2022-01-31T22:52:49.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:48 smithi167 conmon[49112]: debug 2022-01-31T22:52:48.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.926700+0000) 2022-01-31T22:52:49.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:49 smithi167 conmon[54076]: debug 2022-01-31T22:52:49.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:49.094348+0000) 2022-01-31T22:52:49.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:48 smithi167 conmon[60316]: debug 2022-01-31T22:52:48.852+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:48.853891+0000) 2022-01-31T22:52:49.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:49 smithi171 conmon[41853]: debug 2022-01-31T22:52:49.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:49.289448+0000) 2022-01-31T22:52:49.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:49 smithi171 conmon[46715]: debug 2022-01-31T22:52:49.337+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:49.339390+0000) 2022-01-31T22:52:49.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:49 smithi171 conmon[51620]: debug 2022-01-31T22:52:49.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:49.465269+0000) 2022-01-31T22:52:50.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:50 smithi167 conmon[54076]: debug 2022-01-31T22:52:50.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:50.094472+0000) 2022-01-31T22:52:50.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:49 smithi167 conmon[49112]: debug 2022-01-31T22:52:49.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:49.926849+0000) 2022-01-31T22:52:50.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:49 smithi167 conmon[60316]: debug 2022-01-31T22:52:49.853+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:49.854048+0000) 2022-01-31T22:52:50.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:50 smithi171 conmon[41853]: debug 2022-01-31T22:52:50.287+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:50.289590+0000) 2022-01-31T22:52:50.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:50 smithi171 conmon[46715]: debug 2022-01-31T22:52:50.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:50.339539+0000) 2022-01-31T22:52:50.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:50 smithi171 conmon[51620]: debug 2022-01-31T22:52:50.463+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:50.465478+0000) 2022-01-31T22:52:51.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:51 smithi167 conmon[54076]: debug 2022-01-31T22:52:51.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:51.094698+0000) 2022-01-31T22:52:51.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:50 smithi167 conmon[60316]: debug 2022-01-31T22:52:50.852+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:50.854199+0000) 2022-01-31T22:52:51.158 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:50 smithi167 conmon[49112]: debug 2022-01-31T22:52:50.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:50.926981+0000) 2022-01-31T22:52:51.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:51 smithi171 conmon[46715]: debug 2022-01-31T22:52:51.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:51.339710+0000) 2022-01-31T22:52:51.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:51 smithi171 conmon[41853]: debug 2022-01-31T22:52:51.288+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:51.289815+0000) 2022-01-31T22:52:51.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:51 smithi171 conmon[51620]: debug 2022-01-31T22:52:51.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:51.465643+0000) 2022-01-31T22:52:52.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:51 smithi167 conmon[60316]: debug 2022-01-31T22:52:51.853+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:51.854392+0000) 2022-01-31T22:52:52.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:51 smithi167 conmon[49112]: debug 2022-01-31T22:52:51.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:51.927142+0000) 2022-01-31T22:52:52.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:52 smithi167 conmon[54076]: debug 2022-01-31T22:52:52.094+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:52.094890+0000) 2022-01-31T22:52:52.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:52 smithi171 conmon[46715]: debug 2022-01-31T22:52:52.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:52.339919+0000) 2022-01-31T22:52:52.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:52 smithi171 conmon[41853]: debug 2022-01-31T22:52:52.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:52.290055+0000) 2022-01-31T22:52:52.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:52 smithi171 conmon[51620]: debug 2022-01-31T22:52:52.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:52.465862+0000) 2022-01-31T22:52:53.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:52 smithi167 conmon[49112]: debug 2022-01-31T22:52:52.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:52.927303+0000) 2022-01-31T22:52:53.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:53 smithi167 conmon[54076]: debug 2022-01-31T22:52:53.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.095073+0000) 2022-01-31T22:52:53.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:52 smithi167 conmon[60316]: debug 2022-01-31T22:52:52.853+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:52.854584+0000) 2022-01-31T22:52:53.576 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:53 smithi167 conmon[49112]: debug 2022-01-31T22:52:53.342+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.343713+0000) 2022-01-31T22:52:53.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:53 smithi167 conmon[54076]: debug 2022-01-31T22:52:53.342+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.344210+0000) 2022-01-31T22:52:53.577 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:53 smithi167 conmon[60316]: debug 2022-01-31T22:52:53.343+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.344919+0000) 2022-01-31T22:52:53.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[35325]: debug 2022-01-31T22:52:53.367+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 244989 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:53.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[41853]: debug 2022-01-31T22:52:53.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.290239+0000) 2022-01-31T22:52:53.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[41853]: debug 2022-01-31T22:52:53.340+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.342356+0000) 2022-01-31T22:52:53.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[46715]: debug 2022-01-31T22:52:53.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.340137+0000) 2022-01-31T22:52:53.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[46715]: debug 2022-01-31T22:52:53.340+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.342269+0000) 2022-01-31T22:52:53.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[51620]: debug 2022-01-31T22:52:53.341+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.343411+0000) 2022-01-31T22:52:53.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[51620]: debug 2022-01-31T22:52:53.465+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.466060+0000) 2022-01-31T22:52:54.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:53 smithi171 conmon[35325]: debug 2022-01-31T22:52:53.924+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:52:54.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:53 smithi167 conmon[49112]: debug 2022-01-31T22:52:53.925+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.927422+0000) 2022-01-31T22:52:54.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:54 smithi167 conmon[54076]: debug 2022-01-31T22:52:54.094+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:54.095236+0000) 2022-01-31T22:52:54.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:53 smithi167 conmon[60316]: debug 2022-01-31T22:52:53.854+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:53.854747+0000) 2022-01-31T22:52:54.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:54 smithi171 conmon[41853]: debug 2022-01-31T22:52:54.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:54.290367+0000) 2022-01-31T22:52:54.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:54 smithi171 conmon[46715]: debug 2022-01-31T22:52:54.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:54.340288+0000) 2022-01-31T22:52:54.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:54 smithi171 conmon[51620]: debug 2022-01-31T22:52:54.465+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:54.466221+0000) 2022-01-31T22:52:55.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:54 smithi167 conmon[60316]: debug 2022-01-31T22:52:54.853+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:54.854902+0000) 2022-01-31T22:52:55.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:54 smithi167 conmon[49112]: debug 2022-01-31T22:52:54.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:54.927532+0000) 2022-01-31T22:52:55.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:55 smithi167 conmon[54076]: debug 2022-01-31T22:52:55.093+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:55.095388+0000) 2022-01-31T22:52:55.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:55 smithi171 conmon[41853]: debug 2022-01-31T22:52:55.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:55.290535+0000) 2022-01-31T22:52:55.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:55 smithi171 conmon[46715]: debug 2022-01-31T22:52:55.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:55.340370+0000) 2022-01-31T22:52:55.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:55 smithi171 conmon[51620]: debug 2022-01-31T22:52:55.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:55.466396+0000) 2022-01-31T22:52:56.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:55 smithi167 conmon[49112]: debug 2022-01-31T22:52:55.926+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:55.927716+0000) 2022-01-31T22:52:56.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:56 smithi167 conmon[54076]: debug 2022-01-31T22:52:56.094+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:56.095573+0000) 2022-01-31T22:52:56.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:55 smithi167 conmon[60316]: debug 2022-01-31T22:52:55.853+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:55.855032+0000) 2022-01-31T22:52:56.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:56 smithi171 conmon[41853]: debug 2022-01-31T22:52:56.289+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:56.290730+0000) 2022-01-31T22:52:56.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:56 smithi171 conmon[51620]: debug 2022-01-31T22:52:56.464+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:56.466579+0000) 2022-01-31T22:52:56.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:56 smithi171 conmon[46715]: debug 2022-01-31T22:52:56.338+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:56.340551+0000) 2022-01-31T22:52:57.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:56 smithi167 conmon[49112]: debug 2022-01-31T22:52:56.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:56.927881+0000) 2022-01-31T22:52:57.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:57 smithi167 conmon[54076]: debug 2022-01-31T22:52:57.095+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:57.095774+0000) 2022-01-31T22:52:57.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:56 smithi167 conmon[60316]: debug 2022-01-31T22:52:56.854+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:56.855191+0000) 2022-01-31T22:52:57.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:57 smithi171 conmon[41853]: debug 2022-01-31T22:52:57.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:57.290888+0000) 2022-01-31T22:52:57.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:57 smithi171 conmon[51620]: debug 2022-01-31T22:52:57.465+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:57.466789+0000) 2022-01-31T22:52:57.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:57 smithi171 conmon[46715]: debug 2022-01-31T22:52:57.339+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:57.340728+0000) 2022-01-31T22:52:58.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:58 smithi167 conmon[54076]: debug 2022-01-31T22:52:58.095+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.095977+0000) 2022-01-31T22:52:58.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:57 smithi167 conmon[49112]: debug 2022-01-31T22:52:57.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:57.928067+0000) 2022-01-31T22:52:58.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:57 smithi167 conmon[60316]: debug 2022-01-31T22:52:57.853+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:57.855371+0000) 2022-01-31T22:52:58.577 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:58 smithi167 conmon[49112]: debug 2022-01-31T22:52:58.370+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.371888+0000) 2022-01-31T22:52:58.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:58 smithi167 conmon[54076]: debug 2022-01-31T22:52:58.370+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.372281+0000) 2022-01-31T22:52:58.578 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:58 smithi167 conmon[60316]: debug 2022-01-31T22:52:58.370+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.371665+0000) 2022-01-31T22:52:58.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[35325]: debug 2022-01-31T22:52:58.394+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245100 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:52:58.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[41853]: debug 2022-01-31T22:52:58.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.291058+0000) 2022-01-31T22:52:58.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[41853]: debug 2022-01-31T22:52:58.368+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.370397+0000) 2022-01-31T22:52:58.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[46715]: debug 2022-01-31T22:52:58.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.340885+0000) 2022-01-31T22:52:58.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[46715]: debug 2022-01-31T22:52:58.369+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.370936+0000) 2022-01-31T22:52:58.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[51620]: debug 2022-01-31T22:52:58.370+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.371744+0000) 2022-01-31T22:52:58.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:58 smithi171 conmon[51620]: debug 2022-01-31T22:52:58.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.467014+0000) 2022-01-31T22:52:59.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:52:59 smithi167 conmon[54076]: debug 2022-01-31T22:52:59.095+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:59.096144+0000) 2022-01-31T22:52:59.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:58 smithi167 conmon[49112]: debug 2022-01-31T22:52:58.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.928254+0000) 2022-01-31T22:52:59.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:58 smithi167 conmon[60316]: debug 2022-01-31T22:52:58.854+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:58.855546+0000) 2022-01-31T22:52:59.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:52:59 smithi171 conmon[41853]: debug 2022-01-31T22:52:59.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:59.291265+0000) 2022-01-31T22:52:59.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:52:59 smithi171 conmon[46715]: debug 2022-01-31T22:52:59.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:59.341063+0000) 2022-01-31T22:52:59.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:52:59 smithi171 conmon[51620]: debug 2022-01-31T22:52:59.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:59.467140+0000) 2022-01-31T22:53:00.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:00 smithi167 conmon[54076]: debug 2022-01-31T22:53:00.095+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:00.096289+0000) 2022-01-31T22:53:00.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:52:59 smithi167 conmon[49112]: debug 2022-01-31T22:52:59.927+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:59.928430+0000) 2022-01-31T22:53:00.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:52:59 smithi167 conmon[60316]: debug 2022-01-31T22:52:59.855+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:52:59.855709+0000) 2022-01-31T22:53:00.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:00 smithi171 conmon[41853]: debug 2022-01-31T22:53:00.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:00.291407+0000) 2022-01-31T22:53:00.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:00 smithi171 conmon[46715]: debug 2022-01-31T22:53:00.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:00.341186+0000) 2022-01-31T22:53:00.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:00 smithi171 conmon[51620]: debug 2022-01-31T22:53:00.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:00.467322+0000) 2022-01-31T22:53:01.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:00 smithi167 conmon[49112]: debug 2022-01-31T22:53:00.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:00.928562+0000) 2022-01-31T22:53:01.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:01 smithi167 conmon[54076]: debug 2022-01-31T22:53:01.095+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:01.096448+0000) 2022-01-31T22:53:01.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:00 smithi167 conmon[60316]: debug 2022-01-31T22:53:00.855+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:00.855896+0000) 2022-01-31T22:53:01.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:01 smithi171 conmon[41853]: debug 2022-01-31T22:53:01.290+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:01.291609+0000) 2022-01-31T22:53:01.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:01 smithi171 conmon[46715]: debug 2022-01-31T22:53:01.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:01.341382+0000) 2022-01-31T22:53:01.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:01 smithi171 conmon[51620]: debug 2022-01-31T22:53:01.466+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:01.467544+0000) 2022-01-31T22:53:02.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:01 smithi167 conmon[49112]: debug 2022-01-31T22:53:01.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:01.928721+0000) 2022-01-31T22:53:02.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:02 smithi167 conmon[54076]: debug 2022-01-31T22:53:02.096+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:02.096693+0000) 2022-01-31T22:53:02.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:01 smithi167 conmon[60316]: debug 2022-01-31T22:53:01.855+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:01.856088+0000) 2022-01-31T22:53:02.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:02 smithi171 conmon[41853]: debug 2022-01-31T22:53:02.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:02.291796+0000) 2022-01-31T22:53:02.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:02 smithi171 conmon[46715]: debug 2022-01-31T22:53:02.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:02.341540+0000) 2022-01-31T22:53:02.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:02 smithi171 conmon[51620]: debug 2022-01-31T22:53:02.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:02.467771+0000) 2022-01-31T22:53:03.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:03 smithi167 conmon[54076]: debug 2022-01-31T22:53:03.096+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.096884+0000) 2022-01-31T22:53:03.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:02 smithi167 conmon[60316]: debug 2022-01-31T22:53:02.855+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:02.856227+0000) 2022-01-31T22:53:03.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:02 smithi167 conmon[49112]: debug 2022-01-31T22:53:02.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:02.928920+0000) 2022-01-31T22:53:03.578 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:03 smithi167 conmon[49112]: debug 2022-01-31T22:53:03.399+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.399547+0000) 2022-01-31T22:53:03.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:03 smithi167 conmon[54076]: debug 2022-01-31T22:53:03.399+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.399701+0000) 2022-01-31T22:53:03.579 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:03 smithi167 conmon[60316]: debug 2022-01-31T22:53:03.398+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.398759+0000) 2022-01-31T22:53:03.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[35325]: debug 2022-01-31T22:53:03.423+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245213 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:03.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[41853]: debug 2022-01-31T22:53:03.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.291947+0000) 2022-01-31T22:53:03.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[41853]: debug 2022-01-31T22:53:03.397+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.397861+0000) 2022-01-31T22:53:03.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[51620]: debug 2022-01-31T22:53:03.398+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.399382+0000) 2022-01-31T22:53:03.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[51620]: debug 2022-01-31T22:53:03.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.467920+0000) 2022-01-31T22:53:03.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[46715]: debug 2022-01-31T22:53:03.340+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.341689+0000) 2022-01-31T22:53:03.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:03 smithi171 conmon[46715]: debug 2022-01-31T22:53:03.397+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.398354+0000) 2022-01-31T22:53:04.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:04 smithi167 conmon[54076]: debug 2022-01-31T22:53:04.096+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:04.097074+0000) 2022-01-31T22:53:04.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:03 smithi167 conmon[60316]: debug 2022-01-31T22:53:03.855+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.856416+0000) 2022-01-31T22:53:04.157 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:03 smithi167 conmon[49112]: debug 2022-01-31T22:53:03.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:03.929082+0000) 2022-01-31T22:53:04.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:04 smithi171 conmon[46715]: debug 2022-01-31T22:53:04.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:04.341913+0000) 2022-01-31T22:53:04.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:04 smithi171 conmon[51620]: debug 2022-01-31T22:53:04.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:04.468076+0000) 2022-01-31T22:53:04.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:04 smithi171 conmon[41853]: debug 2022-01-31T22:53:04.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:04.292079+0000) 2022-01-31T22:53:05.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:04 smithi167 conmon[49112]: debug 2022-01-31T22:53:04.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:04.929183+0000) 2022-01-31T22:53:05.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:05 smithi167 conmon[54076]: debug 2022-01-31T22:53:05.096+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:05.097271+0000) 2022-01-31T22:53:05.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:04 smithi167 conmon[60316]: debug 2022-01-31T22:53:04.856+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:04.856587+0000) 2022-01-31T22:53:05.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:05 smithi171 conmon[41853]: debug 2022-01-31T22:53:05.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:05.292256+0000) 2022-01-31T22:53:05.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:05 smithi171 conmon[46715]: debug 2022-01-31T22:53:05.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:05.342054+0000) 2022-01-31T22:53:05.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:05 smithi171 conmon[51620]: debug 2022-01-31T22:53:05.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:05.468245+0000) 2022-01-31T22:53:06.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:05 smithi167 conmon[60316]: debug 2022-01-31T22:53:05.856+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:05.856733+0000) 2022-01-31T22:53:06.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:05 smithi167 conmon[49112]: debug 2022-01-31T22:53:05.928+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:05.929346+0000) 2022-01-31T22:53:06.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:06 smithi167 conmon[54076]: debug 2022-01-31T22:53:06.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:06.097457+0000) 2022-01-31T22:53:06.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:06 smithi171 conmon[41853]: debug 2022-01-31T22:53:06.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:06.292462+0000) 2022-01-31T22:53:06.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:06 smithi171 conmon[46715]: debug 2022-01-31T22:53:06.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:06.342276+0000) 2022-01-31T22:53:06.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:06 smithi171 conmon[51620]: debug 2022-01-31T22:53:06.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:06.468470+0000) 2022-01-31T22:53:07.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:06 smithi167 conmon[49112]: debug 2022-01-31T22:53:06.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:06.929478+0000) 2022-01-31T22:53:07.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:06 smithi167 conmon[60316]: debug 2022-01-31T22:53:06.856+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:06.856947+0000) 2022-01-31T22:53:07.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:07 smithi167 conmon[54076]: debug 2022-01-31T22:53:07.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:07.097605+0000) 2022-01-31T22:53:07.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:07 smithi171 conmon[46715]: debug 2022-01-31T22:53:07.341+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:07.342455+0000) 2022-01-31T22:53:07.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:07 smithi171 conmon[41853]: debug 2022-01-31T22:53:07.291+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:07.292634+0000) 2022-01-31T22:53:07.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:07 smithi171 conmon[51620]: debug 2022-01-31T22:53:07.467+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:07.468692+0000) 2022-01-31T22:53:08.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:07 smithi167 conmon[49112]: debug 2022-01-31T22:53:07.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:07.929680+0000) 2022-01-31T22:53:08.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:07 smithi167 conmon[60316]: debug 2022-01-31T22:53:07.856+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:07.857122+0000) 2022-01-31T22:53:08.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:08 smithi167 conmon[54076]: debug 2022-01-31T22:53:08.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.097763+0000) 2022-01-31T22:53:08.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[35325]: debug 2022-01-31T22:53:08.451+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245325 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:08.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[41853]: debug 2022-01-31T22:53:08.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.292807+0000) 2022-01-31T22:53:08.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[41853]: debug 2022-01-31T22:53:08.427+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.428086+0000) 2022-01-31T22:53:08.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[46715]: debug 2022-01-31T22:53:08.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.342614+0000) 2022-01-31T22:53:08.596 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[46715]: debug 2022-01-31T22:53:08.426+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.427112+0000) 2022-01-31T22:53:08.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[51620]: debug 2022-01-31T22:53:08.426+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.427320+0000) 2022-01-31T22:53:08.597 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[51620]: debug 2022-01-31T22:53:08.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.468864+0000) 2022-01-31T22:53:08.685 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:08 smithi167 conmon[49112]: debug 2022-01-31T22:53:08.427+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.428099+0000) 2022-01-31T22:53:08.686 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:08 smithi167 conmon[54076]: debug 2022-01-31T22:53:08.425+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.426484+0000) 2022-01-31T22:53:08.687 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:08 smithi167 conmon[60316]: debug 2022-01-31T22:53:08.426+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.426899+0000) 2022-01-31T22:53:09.083 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:08 smithi167 conmon[49112]: debug 2022-01-31T22:53:08.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.929813+0000) 2022-01-31T22:53:09.084 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:08 smithi167 conmon[60316]: debug 2022-01-31T22:53:08.856+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:08.857309+0000) 2022-01-31T22:53:09.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:08 smithi171 conmon[35325]: debug 2022-01-31T22:53:08.925+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:53:09.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:09 smithi167 conmon[54076]: debug 2022-01-31T22:53:09.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:09.097916+0000) 2022-01-31T22:53:09.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:09 smithi171 conmon[41853]: debug 2022-01-31T22:53:09.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:09.292990+0000) 2022-01-31T22:53:09.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:09 smithi171 conmon[46715]: debug 2022-01-31T22:53:09.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:09.342827+0000) 2022-01-31T22:53:09.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:09 smithi171 conmon[51620]: debug 2022-01-31T22:53:09.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:09.469026+0000) 2022-01-31T22:53:10.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:09 smithi167 conmon[49112]: debug 2022-01-31T22:53:09.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:09.929958+0000) 2022-01-31T22:53:10.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:10 smithi167 conmon[54076]: debug 2022-01-31T22:53:10.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:10.098054+0000) 2022-01-31T22:53:10.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:09 smithi167 conmon[60316]: debug 2022-01-31T22:53:09.856+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:09.857433+0000) 2022-01-31T22:53:10.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:10 smithi171 conmon[41853]: debug 2022-01-31T22:53:10.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:10.293111+0000) 2022-01-31T22:53:10.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:10 smithi171 conmon[46715]: debug 2022-01-31T22:53:10.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:10.342986+0000) 2022-01-31T22:53:10.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:10 smithi171 conmon[51620]: debug 2022-01-31T22:53:10.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:10.469245+0000) 2022-01-31T22:53:11.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:10 smithi167 conmon[49112]: debug 2022-01-31T22:53:10.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:10.930131+0000) 2022-01-31T22:53:11.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:11 smithi167 conmon[54076]: debug 2022-01-31T22:53:11.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:11.098227+0000) 2022-01-31T22:53:11.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:10 smithi167 conmon[60316]: debug 2022-01-31T22:53:10.857+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:10.857609+0000) 2022-01-31T22:53:11.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:11 smithi171 conmon[46715]: debug 2022-01-31T22:53:11.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:11.343179+0000) 2022-01-31T22:53:11.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:11 smithi171 conmon[41853]: debug 2022-01-31T22:53:11.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:11.293246+0000) 2022-01-31T22:53:11.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:11 smithi171 conmon[51620]: debug 2022-01-31T22:53:11.468+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:11.469468+0000) 2022-01-31T22:53:12.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:11 smithi167 conmon[49112]: debug 2022-01-31T22:53:11.929+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:11.930310+0000) 2022-01-31T22:53:12.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:12 smithi167 conmon[54076]: debug 2022-01-31T22:53:12.097+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:12.098421+0000) 2022-01-31T22:53:12.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:11 smithi167 conmon[60316]: debug 2022-01-31T22:53:11.857+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:11.857798+0000) 2022-01-31T22:53:12.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:12 smithi171 conmon[41853]: debug 2022-01-31T22:53:12.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:12.293371+0000) 2022-01-31T22:53:12.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:12 smithi171 conmon[46715]: debug 2022-01-31T22:53:12.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:12.343385+0000) 2022-01-31T22:53:12.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:12 smithi171 conmon[51620]: debug 2022-01-31T22:53:12.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:12.469653+0000) 2022-01-31T22:53:13.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:12 smithi167 conmon[49112]: debug 2022-01-31T22:53:12.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:12.930491+0000) 2022-01-31T22:53:13.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:13 smithi167 conmon[54076]: debug 2022-01-31T22:53:13.098+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.098582+0000) 2022-01-31T22:53:13.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:12 smithi167 conmon[60316]: debug 2022-01-31T22:53:12.857+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:12.857923+0000) 2022-01-31T22:53:13.579 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:13 smithi167 conmon[49112]: debug 2022-01-31T22:53:13.454+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.455274+0000) 2022-01-31T22:53:13.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:13 smithi167 conmon[54076]: debug 2022-01-31T22:53:13.455+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.455954+0000) 2022-01-31T22:53:13.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:13 smithi167 conmon[60316]: debug 2022-01-31T22:53:13.455+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.455694+0000) 2022-01-31T22:53:13.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[35325]: debug 2022-01-31T22:53:13.478+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245437 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:13.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[41853]: debug 2022-01-31T22:53:13.292+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.293556+0000) 2022-01-31T22:53:13.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[41853]: debug 2022-01-31T22:53:13.454+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.455115+0000) 2022-01-31T22:53:13.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[46715]: debug 2022-01-31T22:53:13.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.343490+0000) 2022-01-31T22:53:13.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[46715]: debug 2022-01-31T22:53:13.454+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.454878+0000) 2022-01-31T22:53:13.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[51620]: debug 2022-01-31T22:53:13.455+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.456116+0000) 2022-01-31T22:53:13.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:13 smithi171 conmon[51620]: debug 2022-01-31T22:53:13.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.469821+0000) 2022-01-31T22:53:14.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:13 smithi167 conmon[49112]: debug 2022-01-31T22:53:13.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.930674+0000) 2022-01-31T22:53:14.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:14 smithi167 conmon[54076]: debug 2022-01-31T22:53:14.098+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:14.098712+0000) 2022-01-31T22:53:14.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:13 smithi167 conmon[60316]: debug 2022-01-31T22:53:13.857+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:13.858127+0000) 2022-01-31T22:53:14.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:14 smithi171 conmon[46715]: debug 2022-01-31T22:53:14.342+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:14.343618+0000) 2022-01-31T22:53:14.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:14 smithi171 conmon[51620]: debug 2022-01-31T22:53:14.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:14.469976+0000) 2022-01-31T22:53:14.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:14 smithi171 conmon[41853]: debug 2022-01-31T22:53:14.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:14.293803+0000) 2022-01-31T22:53:15.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:14 smithi167 conmon[49112]: debug 2022-01-31T22:53:14.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:14.930782+0000) 2022-01-31T22:53:15.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:15 smithi167 conmon[54076]: debug 2022-01-31T22:53:15.098+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:15.098845+0000) 2022-01-31T22:53:15.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:14 smithi167 conmon[60316]: debug 2022-01-31T22:53:14.857+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:14.858246+0000) 2022-01-31T22:53:15.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:15 smithi171 conmon[41853]: debug 2022-01-31T22:53:15.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:15.293967+0000) 2022-01-31T22:53:15.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:15 smithi171 conmon[46715]: debug 2022-01-31T22:53:15.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:15.343784+0000) 2022-01-31T22:53:15.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:15 smithi171 conmon[51620]: debug 2022-01-31T22:53:15.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:15.470116+0000) 2022-01-31T22:53:16.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:15 smithi167 conmon[49112]: debug 2022-01-31T22:53:15.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:15.930949+0000) 2022-01-31T22:53:16.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:16 smithi167 conmon[54076]: debug 2022-01-31T22:53:16.098+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:16.099048+0000) 2022-01-31T22:53:16.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:15 smithi167 conmon[60316]: debug 2022-01-31T22:53:15.858+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:15.858402+0000) 2022-01-31T22:53:16.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:16 smithi171 conmon[41853]: debug 2022-01-31T22:53:16.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:16.294155+0000) 2022-01-31T22:53:16.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:16 smithi171 conmon[46715]: debug 2022-01-31T22:53:16.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:16.343931+0000) 2022-01-31T22:53:16.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:16 smithi171 conmon[51620]: debug 2022-01-31T22:53:16.469+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:16.470303+0000) 2022-01-31T22:53:17.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:16 smithi167 conmon[49112]: debug 2022-01-31T22:53:16.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:16.931110+0000) 2022-01-31T22:53:17.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:17 smithi167 conmon[54076]: debug 2022-01-31T22:53:17.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:17.099229+0000) 2022-01-31T22:53:17.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:16 smithi167 conmon[60316]: debug 2022-01-31T22:53:16.858+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:16.858572+0000) 2022-01-31T22:53:17.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:17 smithi171 conmon[41853]: debug 2022-01-31T22:53:17.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:17.294298+0000) 2022-01-31T22:53:17.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:17 smithi171 conmon[46715]: debug 2022-01-31T22:53:17.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:17.344133+0000) 2022-01-31T22:53:17.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:17 smithi171 conmon[51620]: debug 2022-01-31T22:53:17.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:17.470531+0000) 2022-01-31T22:53:18.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:17 smithi167 conmon[49112]: debug 2022-01-31T22:53:17.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:17.931266+0000) 2022-01-31T22:53:18.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:18 smithi167 conmon[54076]: debug 2022-01-31T22:53:18.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.099374+0000) 2022-01-31T22:53:18.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:17 smithi167 conmon[60316]: debug 2022-01-31T22:53:17.858+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:17.858725+0000) 2022-01-31T22:53:18.580 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:18 smithi167 conmon[49112]: debug 2022-01-31T22:53:18.482+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.483274+0000) 2022-01-31T22:53:18.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:18 smithi167 conmon[54076]: debug 2022-01-31T22:53:18.482+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.483455+0000) 2022-01-31T22:53:18.581 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:18 smithi167 conmon[60316]: debug 2022-01-31T22:53:18.481+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.482265+0000) 2022-01-31T22:53:18.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[35325]: debug 2022-01-31T22:53:18.506+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:18.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[41853]: debug 2022-01-31T22:53:18.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.294394+0000) 2022-01-31T22:53:18.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[41853]: debug 2022-01-31T22:53:18.481+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.481787+0000) 2022-01-31T22:53:18.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[46715]: debug 2022-01-31T22:53:18.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.344317+0000) 2022-01-31T22:53:18.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[46715]: debug 2022-01-31T22:53:18.481+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.482636+0000) 2022-01-31T22:53:18.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[51620]: debug 2022-01-31T22:53:18.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.470711+0000) 2022-01-31T22:53:18.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:18 smithi171 conmon[51620]: debug 2022-01-31T22:53:18.481+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.482032+0000) 2022-01-31T22:53:19.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:18 smithi167 conmon[49112]: debug 2022-01-31T22:53:18.930+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.931463+0000) 2022-01-31T22:53:19.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:19 smithi167 conmon[54076]: debug 2022-01-31T22:53:19.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:19.099537+0000) 2022-01-31T22:53:19.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:18 smithi167 conmon[60316]: debug 2022-01-31T22:53:18.858+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:18.858925+0000) 2022-01-31T22:53:19.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:19 smithi171 conmon[41853]: debug 2022-01-31T22:53:19.293+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:19.294578+0000) 2022-01-31T22:53:19.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:19 smithi171 conmon[46715]: debug 2022-01-31T22:53:19.343+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:19.344512+0000) 2022-01-31T22:53:19.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:19 smithi171 conmon[51620]: debug 2022-01-31T22:53:19.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:19.470883+0000) 2022-01-31T22:53:20.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:19 smithi167 conmon[49112]: debug 2022-01-31T22:53:19.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:19.931624+0000) 2022-01-31T22:53:20.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:20 smithi167 conmon[54076]: debug 2022-01-31T22:53:20.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:20.099673+0000) 2022-01-31T22:53:20.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:19 smithi167 conmon[60316]: debug 2022-01-31T22:53:19.858+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:19.859080+0000) 2022-01-31T22:53:20.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:20 smithi171 conmon[46715]: debug 2022-01-31T22:53:20.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:20.344663+0000) 2022-01-31T22:53:20.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:20 smithi171 conmon[51620]: debug 2022-01-31T22:53:20.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:20.471024+0000) 2022-01-31T22:53:20.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:20 smithi171 conmon[41853]: debug 2022-01-31T22:53:20.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:20.294730+0000) 2022-01-31T22:53:21.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:20 smithi167 conmon[49112]: debug 2022-01-31T22:53:20.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:20.931758+0000) 2022-01-31T22:53:21.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:21 smithi167 conmon[54076]: debug 2022-01-31T22:53:21.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:21.099874+0000) 2022-01-31T22:53:21.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:20 smithi167 conmon[60316]: debug 2022-01-31T22:53:20.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:20.859264+0000) 2022-01-31T22:53:21.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:21 smithi171 conmon[41853]: debug 2022-01-31T22:53:21.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:21.294931+0000) 2022-01-31T22:53:21.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:21 smithi171 conmon[46715]: debug 2022-01-31T22:53:21.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:21.344824+0000) 2022-01-31T22:53:21.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:21 smithi171 conmon[51620]: debug 2022-01-31T22:53:21.470+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:21.471231+0000) 2022-01-31T22:53:22.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:21 smithi167 conmon[49112]: debug 2022-01-31T22:53:21.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:21.931913+0000) 2022-01-31T22:53:22.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:22 smithi167 conmon[54076]: debug 2022-01-31T22:53:22.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:22.100050+0000) 2022-01-31T22:53:22.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:21 smithi167 conmon[60316]: debug 2022-01-31T22:53:21.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:21.859443+0000) 2022-01-31T22:53:22.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:22 smithi171 conmon[41853]: debug 2022-01-31T22:53:22.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:22.295079+0000) 2022-01-31T22:53:22.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:22 smithi171 conmon[46715]: debug 2022-01-31T22:53:22.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:22.344980+0000) 2022-01-31T22:53:22.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:22 smithi171 conmon[51620]: debug 2022-01-31T22:53:22.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:22.471463+0000) 2022-01-31T22:53:23.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:22 smithi167 conmon[49112]: debug 2022-01-31T22:53:22.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:22.932109+0000) 2022-01-31T22:53:23.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:23 smithi167 conmon[54076]: debug 2022-01-31T22:53:23.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.100227+0000) 2022-01-31T22:53:23.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:22 smithi167 conmon[60316]: debug 2022-01-31T22:53:22.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:22.859557+0000) 2022-01-31T22:53:23.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[41853]: debug 2022-01-31T22:53:23.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.295243+0000) 2022-01-31T22:53:23.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[41853]: debug 2022-01-31T22:53:23.509+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.510039+0000) 2022-01-31T22:53:23.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[46715]: debug 2022-01-31T22:53:23.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.345171+0000) 2022-01-31T22:53:23.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[46715]: debug 2022-01-31T22:53:23.510+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.511056+0000) 2022-01-31T22:53:23.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[51620]: debug 2022-01-31T22:53:23.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.471704+0000) 2022-01-31T22:53:23.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[51620]: debug 2022-01-31T22:53:23.509+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.510564+0000) 2022-01-31T22:53:23.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:23 smithi167 conmon[49112]: debug 2022-01-31T22:53:23.508+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.511129+0000) 2022-01-31T22:53:23.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:23 smithi167 conmon[54076]: debug 2022-01-31T22:53:23.508+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.510570+0000) 2022-01-31T22:53:23.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:23 smithi167 conmon[60316]: debug 2022-01-31T22:53:23.508+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.510980+0000) 2022-01-31T22:53:23.843 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[35325]: debug 2022-01-31T22:53:23.549+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245663 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:24.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:23 smithi167 conmon[49112]: debug 2022-01-31T22:53:23.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.932322+0000) 2022-01-31T22:53:24.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:24 smithi167 conmon[54076]: debug 2022-01-31T22:53:24.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:24.100411+0000) 2022-01-31T22:53:24.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:23 smithi167 conmon[60316]: debug 2022-01-31T22:53:23.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:23.859732+0000) 2022-01-31T22:53:24.280 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:23 smithi171 conmon[35325]: debug 2022-01-31T22:53:23.926+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:53:24.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:24 smithi171 conmon[41853]: debug 2022-01-31T22:53:24.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:24.295338+0000) 2022-01-31T22:53:24.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:24 smithi171 conmon[46715]: debug 2022-01-31T22:53:24.344+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:24.345278+0000) 2022-01-31T22:53:24.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:24 smithi171 conmon[51620]: debug 2022-01-31T22:53:24.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:24.471926+0000) 2022-01-31T22:53:25.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:24 smithi167 conmon[49112]: debug 2022-01-31T22:53:24.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:24.932460+0000) 2022-01-31T22:53:25.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:25 smithi167 conmon[54076]: debug 2022-01-31T22:53:25.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:25.100571+0000) 2022-01-31T22:53:25.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:24 smithi167 conmon[60316]: debug 2022-01-31T22:53:24.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:24.859920+0000) 2022-01-31T22:53:25.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:25 smithi171 conmon[46715]: debug 2022-01-31T22:53:25.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:25.345432+0000) 2022-01-31T22:53:25.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:25 smithi171 conmon[51620]: debug 2022-01-31T22:53:25.471+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:25.472066+0000) 2022-01-31T22:53:25.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:25 smithi171 conmon[41853]: debug 2022-01-31T22:53:25.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:25.295498+0000) 2022-01-31T22:53:26.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:25 smithi167 conmon[49112]: debug 2022-01-31T22:53:25.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:25.932596+0000) 2022-01-31T22:53:26.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:26 smithi167 conmon[54076]: debug 2022-01-31T22:53:26.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:26.100790+0000) 2022-01-31T22:53:26.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:25 smithi167 conmon[60316]: debug 2022-01-31T22:53:25.858+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:25.860084+0000) 2022-01-31T22:53:26.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:26 smithi171 conmon[41853]: debug 2022-01-31T22:53:26.294+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:26.295648+0000) 2022-01-31T22:53:26.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:26 smithi171 conmon[46715]: debug 2022-01-31T22:53:26.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:26.345592+0000) 2022-01-31T22:53:26.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:26 smithi171 conmon[51620]: debug 2022-01-31T22:53:26.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:26.472275+0000) 2022-01-31T22:53:27.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:26 smithi167 conmon[49112]: debug 2022-01-31T22:53:26.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:26.932718+0000) 2022-01-31T22:53:27.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:27 smithi167 conmon[54076]: debug 2022-01-31T22:53:27.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:27.100977+0000) 2022-01-31T22:53:27.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:26 smithi167 conmon[60316]: debug 2022-01-31T22:53:26.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:26.860274+0000) 2022-01-31T22:53:27.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:27 smithi171 conmon[41853]: debug 2022-01-31T22:53:27.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:27.295848+0000) 2022-01-31T22:53:27.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:27 smithi171 conmon[46715]: debug 2022-01-31T22:53:27.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:27.345790+0000) 2022-01-31T22:53:27.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:27 smithi171 conmon[51620]: debug 2022-01-31T22:53:27.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:27.472454+0000) 2022-01-31T22:53:28.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:27 smithi167 conmon[49112]: debug 2022-01-31T22:53:27.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:27.932912+0000) 2022-01-31T22:53:28.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:28 smithi167 conmon[54076]: debug 2022-01-31T22:53:28.099+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.101200+0000) 2022-01-31T22:53:28.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:27 smithi167 conmon[60316]: debug 2022-01-31T22:53:27.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:27.860441+0000) 2022-01-31T22:53:28.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[41853]: debug 2022-01-31T22:53:28.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.296024+0000) 2022-01-31T22:53:28.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[46715]: debug 2022-01-31T22:53:28.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.345966+0000) 2022-01-31T22:53:28.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[51620]: debug 2022-01-31T22:53:28.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.472626+0000) 2022-01-31T22:53:28.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:28 smithi167 conmon[49112]: debug 2022-01-31T22:53:28.551+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.553357+0000) 2022-01-31T22:53:28.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:28 smithi167 conmon[54076]: debug 2022-01-31T22:53:28.552+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.553590+0000) 2022-01-31T22:53:28.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:28 smithi167 conmon[60316]: debug 2022-01-31T22:53:28.552+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.553829+0000) 2022-01-31T22:53:28.843 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[35325]: debug 2022-01-31T22:53:28.577+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245774 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:28.844 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[41853]: debug 2022-01-31T22:53:28.551+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.552295+0000) 2022-01-31T22:53:28.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[46715]: debug 2022-01-31T22:53:28.552+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.552803+0000) 2022-01-31T22:53:28.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:28 smithi171 conmon[51620]: debug 2022-01-31T22:53:28.553+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.554061+0000) 2022-01-31T22:53:29.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:28 smithi167 conmon[49112]: debug 2022-01-31T22:53:28.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.933112+0000) 2022-01-31T22:53:29.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:29 smithi167 conmon[54076]: debug 2022-01-31T22:53:29.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:29.101386+0000) 2022-01-31T22:53:29.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:28 smithi167 conmon[60316]: debug 2022-01-31T22:53:28.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:28.860597+0000) 2022-01-31T22:53:29.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:29 smithi171 conmon[41853]: debug 2022-01-31T22:53:29.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:29.296208+0000) 2022-01-31T22:53:29.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:29 smithi171 conmon[46715]: debug 2022-01-31T22:53:29.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:29.346164+0000) 2022-01-31T22:53:29.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:29 smithi171 conmon[51620]: debug 2022-01-31T22:53:29.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:29.472850+0000) 2022-01-31T22:53:30.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:29 smithi167 conmon[49112]: debug 2022-01-31T22:53:29.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:29.933227+0000) 2022-01-31T22:53:30.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:30 smithi167 conmon[54076]: debug 2022-01-31T22:53:30.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:30.101547+0000) 2022-01-31T22:53:30.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:29 smithi167 conmon[60316]: debug 2022-01-31T22:53:29.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:29.860734+0000) 2022-01-31T22:53:30.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:30 smithi171 conmon[41853]: debug 2022-01-31T22:53:30.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:30.296285+0000) 2022-01-31T22:53:30.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:30 smithi171 conmon[46715]: debug 2022-01-31T22:53:30.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:30.346348+0000) 2022-01-31T22:53:30.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:30 smithi171 conmon[51620]: debug 2022-01-31T22:53:30.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:30.473023+0000) 2022-01-31T22:53:31.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:30 smithi167 conmon[49112]: debug 2022-01-31T22:53:30.931+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:30.933325+0000) 2022-01-31T22:53:31.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:30 smithi167 conmon[60316]: debug 2022-01-31T22:53:30.860+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:30.860940+0000) 2022-01-31T22:53:31.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:31 smithi167 conmon[54076]: debug 2022-01-31T22:53:31.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:31.101688+0000) 2022-01-31T22:53:31.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:31 smithi171 conmon[41853]: debug 2022-01-31T22:53:31.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:31.296473+0000) 2022-01-31T22:53:31.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:31 smithi171 conmon[46715]: debug 2022-01-31T22:53:31.345+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:31.346565+0000) 2022-01-31T22:53:31.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:31 smithi171 conmon[51620]: debug 2022-01-31T22:53:31.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:31.473233+0000) 2022-01-31T22:53:32.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:31 smithi167 conmon[49112]: debug 2022-01-31T22:53:31.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:31.933524+0000) 2022-01-31T22:53:32.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:31 smithi167 conmon[60316]: debug 2022-01-31T22:53:31.859+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:31.861150+0000) 2022-01-31T22:53:32.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:32 smithi167 conmon[54076]: debug 2022-01-31T22:53:32.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:32.101849+0000) 2022-01-31T22:53:32.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:32 smithi171 conmon[51620]: debug 2022-01-31T22:53:32.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:32.473465+0000) 2022-01-31T22:53:32.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:32 smithi171 conmon[41853]: debug 2022-01-31T22:53:32.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:32.296641+0000) 2022-01-31T22:53:32.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:32 smithi171 conmon[46715]: debug 2022-01-31T22:53:32.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:32.346747+0000) 2022-01-31T22:53:33.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:32 smithi167 conmon[49112]: debug 2022-01-31T22:53:32.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:32.933719+0000) 2022-01-31T22:53:33.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:33 smithi167 conmon[54076]: debug 2022-01-31T22:53:33.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.102025+0000) 2022-01-31T22:53:33.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:32 smithi167 conmon[60316]: debug 2022-01-31T22:53:32.860+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:32.861338+0000) 2022-01-31T22:53:33.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[46715]: debug 2022-01-31T22:53:33.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.346946+0000) 2022-01-31T22:53:33.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[41853]: debug 2022-01-31T22:53:33.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.296872+0000) 2022-01-31T22:53:33.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[51620]: debug 2022-01-31T22:53:33.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.473651+0000) 2022-01-31T22:53:33.819 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:33 smithi167 conmon[49112]: debug 2022-01-31T22:53:33.580+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.581830+0000) 2022-01-31T22:53:33.820 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:33 smithi167 conmon[54076]: debug 2022-01-31T22:53:33.580+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.582159+0000) 2022-01-31T22:53:33.820 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:33 smithi167 conmon[60316]: debug 2022-01-31T22:53:33.581+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.582496+0000) 2022-01-31T22:53:33.843 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[35325]: debug 2022-01-31T22:53:33.605+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 245888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:33.844 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[46715]: debug 2022-01-31T22:53:33.579+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.580394+0000) 2022-01-31T22:53:33.845 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[51620]: debug 2022-01-31T22:53:33.580+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.581701+0000) 2022-01-31T22:53:33.845 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:33 smithi171 conmon[41853]: debug 2022-01-31T22:53:33.581+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.582387+0000) 2022-01-31T22:53:34.088 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:33 smithi167 conmon[49112]: debug 2022-01-31T22:53:33.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.933856+0000) 2022-01-31T22:53:34.088 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:33 smithi167 conmon[60316]: debug 2022-01-31T22:53:33.860+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:33.861537+0000) 2022-01-31T22:53:34.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:34 smithi167 conmon[54076]: debug 2022-01-31T22:53:34.100+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:34.102210+0000) 2022-01-31T22:53:34.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:34 smithi171 conmon[41853]: debug 2022-01-31T22:53:34.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:34.296978+0000) 2022-01-31T22:53:34.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:34 smithi171 conmon[46715]: debug 2022-01-31T22:53:34.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:34.347084+0000) 2022-01-31T22:53:34.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:34 smithi171 conmon[51620]: debug 2022-01-31T22:53:34.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:34.473826+0000) 2022-01-31T22:53:35.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:34 smithi167 conmon[49112]: debug 2022-01-31T22:53:34.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:34.933965+0000) 2022-01-31T22:53:35.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:35 smithi167 conmon[54076]: debug 2022-01-31T22:53:35.101+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:35.102354+0000) 2022-01-31T22:53:35.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:34 smithi167 conmon[60316]: debug 2022-01-31T22:53:34.860+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:34.861736+0000) 2022-01-31T22:53:35.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:35 smithi171 conmon[46715]: debug 2022-01-31T22:53:35.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:35.347235+0000) 2022-01-31T22:53:35.596 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:35 smithi171 conmon[41853]: debug 2022-01-31T22:53:35.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:35.297123+0000) 2022-01-31T22:53:35.598 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:35 smithi171 conmon[51620]: debug 2022-01-31T22:53:35.472+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:35.473987+0000) 2022-01-31T22:53:36.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:35 smithi167 conmon[49112]: debug 2022-01-31T22:53:35.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:35.934139+0000) 2022-01-31T22:53:36.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:36 smithi167 conmon[54076]: debug 2022-01-31T22:53:36.101+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:36.102521+0000) 2022-01-31T22:53:36.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:35 smithi167 conmon[60316]: debug 2022-01-31T22:53:35.861+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:35.861923+0000) 2022-01-31T22:53:36.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:36 smithi171 conmon[46715]: debug 2022-01-31T22:53:36.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:36.347423+0000) 2022-01-31T22:53:36.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:36 smithi171 conmon[41853]: debug 2022-01-31T22:53:36.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:36.297282+0000) 2022-01-31T22:53:36.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:36 smithi171 conmon[51620]: debug 2022-01-31T22:53:36.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:36.474171+0000) 2022-01-31T22:53:37.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:36 smithi167 conmon[49112]: debug 2022-01-31T22:53:36.932+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:36.934320+0000) 2022-01-31T22:53:37.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:37 smithi167 conmon[54076]: debug 2022-01-31T22:53:37.101+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:37.102721+0000) 2022-01-31T22:53:37.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:36 smithi167 conmon[60316]: debug 2022-01-31T22:53:36.861+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:36.862084+0000) 2022-01-31T22:53:37.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:37 smithi171 conmon[46715]: debug 2022-01-31T22:53:37.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:37.347609+0000) 2022-01-31T22:53:37.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:37 smithi171 conmon[41853]: debug 2022-01-31T22:53:37.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:37.297465+0000) 2022-01-31T22:53:37.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:37 smithi171 conmon[51620]: debug 2022-01-31T22:53:37.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:37.474382+0000) 2022-01-31T22:53:38.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:37 smithi167 conmon[49112]: debug 2022-01-31T22:53:37.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:37.934522+0000) 2022-01-31T22:53:38.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:38 smithi167 conmon[54076]: debug 2022-01-31T22:53:38.101+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.102899+0000) 2022-01-31T22:53:38.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:37 smithi167 conmon[60316]: debug 2022-01-31T22:53:37.860+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:37.862213+0000) 2022-01-31T22:53:38.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[46715]: debug 2022-01-31T22:53:38.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.347789+0000) 2022-01-31T22:53:38.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[51620]: debug 2022-01-31T22:53:38.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.474589+0000) 2022-01-31T22:53:38.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[41853]: debug 2022-01-31T22:53:38.295+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.297635+0000) 2022-01-31T22:53:38.848 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:38 smithi167 conmon[49112]: debug 2022-01-31T22:53:38.608+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.609583+0000) 2022-01-31T22:53:38.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:38 smithi167 conmon[54076]: debug 2022-01-31T22:53:38.608+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.610297+0000) 2022-01-31T22:53:38.849 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:38 smithi167 conmon[60316]: debug 2022-01-31T22:53:38.608+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.610004+0000) 2022-01-31T22:53:38.914 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[41853]: debug 2022-01-31T22:53:38.608+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.610439+0000) 2022-01-31T22:53:38.914 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[46715]: debug 2022-01-31T22:53:38.607+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.608808+0000) 2022-01-31T22:53:38.915 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[51620]: debug 2022-01-31T22:53:38.607+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.609237+0000) 2022-01-31T22:53:38.915 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[35325]: debug 2022-01-31T22:53:38.632+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246000 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:39.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:38 smithi167 conmon[49112]: debug 2022-01-31T22:53:38.933+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.934669+0000) 2022-01-31T22:53:39.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:39 smithi167 conmon[54076]: debug 2022-01-31T22:53:39.102+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:39.103053+0000) 2022-01-31T22:53:39.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:38 smithi167 conmon[60316]: debug 2022-01-31T22:53:38.860+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:38.862400+0000) 2022-01-31T22:53:39.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:38 smithi171 conmon[35325]: debug 2022-01-31T22:53:38.926+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:53:39.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:39 smithi171 conmon[41853]: debug 2022-01-31T22:53:39.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:39.297811+0000) 2022-01-31T22:53:39.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:39 smithi171 conmon[46715]: debug 2022-01-31T22:53:39.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:39.347959+0000) 2022-01-31T22:53:39.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:39 smithi171 conmon[51620]: debug 2022-01-31T22:53:39.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:39.474774+0000) 2022-01-31T22:53:40.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:39 smithi167 conmon[49112]: debug 2022-01-31T22:53:39.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:39.934868+0000) 2022-01-31T22:53:40.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:40 smithi167 conmon[54076]: debug 2022-01-31T22:53:40.102+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:40.103214+0000) 2022-01-31T22:53:40.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:39 smithi167 conmon[60316]: debug 2022-01-31T22:53:39.861+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:39.862560+0000) 2022-01-31T22:53:40.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:40 smithi171 conmon[41853]: debug 2022-01-31T22:53:40.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:40.297953+0000) 2022-01-31T22:53:40.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:40 smithi171 conmon[46715]: debug 2022-01-31T22:53:40.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:40.348110+0000) 2022-01-31T22:53:40.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:40 smithi171 conmon[51620]: debug 2022-01-31T22:53:40.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:40.474921+0000) 2022-01-31T22:53:41.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:40 smithi167 conmon[49112]: debug 2022-01-31T22:53:40.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:40.935054+0000) 2022-01-31T22:53:41.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:41 smithi167 conmon[54076]: debug 2022-01-31T22:53:41.102+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:41.103409+0000) 2022-01-31T22:53:41.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:40 smithi167 conmon[60316]: debug 2022-01-31T22:53:40.861+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:40.862675+0000) 2022-01-31T22:53:41.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:41 smithi171 conmon[41853]: debug 2022-01-31T22:53:41.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:41.298351+0000) 2022-01-31T22:53:41.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:41 smithi171 conmon[46715]: debug 2022-01-31T22:53:41.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:41.348260+0000) 2022-01-31T22:53:41.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:41 smithi171 conmon[51620]: debug 2022-01-31T22:53:41.473+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:41.475085+0000) 2022-01-31T22:53:42.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:41 smithi167 conmon[60316]: debug 2022-01-31T22:53:41.862+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:41.862879+0000) 2022-01-31T22:53:42.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:41 smithi167 conmon[49112]: debug 2022-01-31T22:53:41.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:41.935239+0000) 2022-01-31T22:53:42.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:42 smithi167 conmon[54076]: debug 2022-01-31T22:53:42.102+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:42.103570+0000) 2022-01-31T22:53:42.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:42 smithi171 conmon[46715]: debug 2022-01-31T22:53:42.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:42.348430+0000) 2022-01-31T22:53:42.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:42 smithi171 conmon[41853]: debug 2022-01-31T22:53:42.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:42.298516+0000) 2022-01-31T22:53:42.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:42 smithi171 conmon[51620]: debug 2022-01-31T22:53:42.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:42.475291+0000) 2022-01-31T22:53:43.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:42 smithi167 conmon[49112]: debug 2022-01-31T22:53:42.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:42.935449+0000) 2022-01-31T22:53:43.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:43 smithi167 conmon[54076]: debug 2022-01-31T22:53:43.103+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.103758+0000) 2022-01-31T22:53:43.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:42 smithi167 conmon[60316]: debug 2022-01-31T22:53:42.862+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:42.863031+0000) 2022-01-31T22:53:43.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[46715]: debug 2022-01-31T22:53:43.346+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.348609+0000) 2022-01-31T22:53:43.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[41853]: debug 2022-01-31T22:53:43.296+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.298696+0000) 2022-01-31T22:53:43.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[51620]: debug 2022-01-31T22:53:43.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.475515+0000) 2022-01-31T22:53:43.849 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:43 smithi167 conmon[54076]: debug 2022-01-31T22:53:43.636+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.637779+0000) 2022-01-31T22:53:43.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:43 smithi167 conmon[60316]: debug 2022-01-31T22:53:43.636+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.637707+0000) 2022-01-31T22:53:43.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:43 smithi167 conmon[49112]: debug 2022-01-31T22:53:43.636+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.637602+0000) 2022-01-31T22:53:44.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[41853]: debug 2022-01-31T22:53:43.636+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.638189+0000) 2022-01-31T22:53:44.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[46715]: debug 2022-01-31T22:53:43.634+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.636279+0000) 2022-01-31T22:53:44.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[51620]: debug 2022-01-31T22:53:43.635+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.636913+0000) 2022-01-31T22:53:44.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:43 smithi171 conmon[35325]: debug 2022-01-31T22:53:43.659+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246112 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:44.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:44 smithi167 conmon[54076]: debug 2022-01-31T22:53:44.103+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:44.103922+0000) 2022-01-31T22:53:44.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:43 smithi167 conmon[60316]: debug 2022-01-31T22:53:43.861+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.863217+0000) 2022-01-31T22:53:44.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:43 smithi167 conmon[49112]: debug 2022-01-31T22:53:43.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:43.935579+0000) 2022-01-31T22:53:44.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:44 smithi171 conmon[51620]: debug 2022-01-31T22:53:44.474+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:44.475727+0000) 2022-01-31T22:53:44.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:44 smithi171 conmon[41853]: debug 2022-01-31T22:53:44.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:44.298858+0000) 2022-01-31T22:53:44.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:44 smithi171 conmon[46715]: debug 2022-01-31T22:53:44.347+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:44.348791+0000) 2022-01-31T22:53:45.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:44 smithi167 conmon[49112]: debug 2022-01-31T22:53:44.934+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:44.935700+0000) 2022-01-31T22:53:45.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:44 smithi167 conmon[60316]: debug 2022-01-31T22:53:44.862+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:44.863401+0000) 2022-01-31T22:53:45.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:45 smithi167 conmon[54076]: debug 2022-01-31T22:53:45.103+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:45.104087+0000) 2022-01-31T22:53:45.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:45 smithi171 conmon[41853]: debug 2022-01-31T22:53:45.297+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:45.298960+0000) 2022-01-31T22:53:45.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:45 smithi171 conmon[46715]: debug 2022-01-31T22:53:45.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:45.348914+0000) 2022-01-31T22:53:45.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:45 smithi171 conmon[51620]: debug 2022-01-31T22:53:45.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:45.475911+0000) 2022-01-31T22:53:46.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:45 smithi167 conmon[49112]: debug 2022-01-31T22:53:45.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:45.935885+0000) 2022-01-31T22:53:46.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:46 smithi167 conmon[54076]: debug 2022-01-31T22:53:46.103+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:46.104307+0000) 2022-01-31T22:53:46.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:45 smithi167 conmon[60316]: debug 2022-01-31T22:53:45.862+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:45.863587+0000) 2022-01-31T22:53:46.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:46 smithi171 conmon[41853]: debug 2022-01-31T22:53:46.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:46.299116+0000) 2022-01-31T22:53:46.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:46 smithi171 conmon[46715]: debug 2022-01-31T22:53:46.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:46.349069+0000) 2022-01-31T22:53:46.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:46 smithi171 conmon[51620]: debug 2022-01-31T22:53:46.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:46.476113+0000) 2022-01-31T22:53:47.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:46 smithi167 conmon[49112]: debug 2022-01-31T22:53:46.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:46.936032+0000) 2022-01-31T22:53:47.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:47 smithi167 conmon[54076]: debug 2022-01-31T22:53:47.103+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:47.104470+0000) 2022-01-31T22:53:47.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:46 smithi167 conmon[60316]: debug 2022-01-31T22:53:46.863+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:46.863781+0000) 2022-01-31T22:53:47.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:47 smithi171 conmon[41853]: debug 2022-01-31T22:53:47.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:47.299261+0000) 2022-01-31T22:53:47.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:47 smithi171 conmon[46715]: debug 2022-01-31T22:53:47.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:47.349250+0000) 2022-01-31T22:53:47.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:47 smithi171 conmon[51620]: debug 2022-01-31T22:53:47.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:47.476298+0000) 2022-01-31T22:53:48.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:47 smithi167 conmon[49112]: debug 2022-01-31T22:53:47.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:47.936251+0000) 2022-01-31T22:53:48.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:48 smithi167 conmon[54076]: debug 2022-01-31T22:53:48.104+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.104733+0000) 2022-01-31T22:53:48.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:47 smithi167 conmon[60316]: debug 2022-01-31T22:53:47.863+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:47.863974+0000) 2022-01-31T22:53:48.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[41853]: debug 2022-01-31T22:53:48.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.299509+0000) 2022-01-31T22:53:48.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[46715]: debug 2022-01-31T22:53:48.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.349471+0000) 2022-01-31T22:53:48.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[51620]: debug 2022-01-31T22:53:48.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.476519+0000) 2022-01-31T22:53:48.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:48 smithi167 conmon[49112]: debug 2022-01-31T22:53:48.662+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.664038+0000) 2022-01-31T22:53:48.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:48 smithi167 conmon[54076]: debug 2022-01-31T22:53:48.663+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.665048+0000) 2022-01-31T22:53:48.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:48 smithi167 conmon[60316]: debug 2022-01-31T22:53:48.663+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.665213+0000) 2022-01-31T22:53:49.003 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[35325]: debug 2022-01-31T22:53:48.688+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246223 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:49.004 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[46715]: debug 2022-01-31T22:53:48.663+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.664513+0000) 2022-01-31T22:53:49.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[51620]: debug 2022-01-31T22:53:48.662+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.663029+0000) 2022-01-31T22:53:49.005 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:48 smithi171 conmon[41853]: debug 2022-01-31T22:53:48.663+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.664587+0000) 2022-01-31T22:53:49.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:48 smithi167 conmon[49112]: debug 2022-01-31T22:53:48.935+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.936448+0000) 2022-01-31T22:53:49.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:49 smithi167 conmon[54076]: debug 2022-01-31T22:53:49.104+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:49.104916+0000) 2022-01-31T22:53:49.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:48 smithi167 conmon[60316]: debug 2022-01-31T22:53:48.863+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:48.864151+0000) 2022-01-31T22:53:49.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:49 smithi171 conmon[46715]: debug 2022-01-31T22:53:49.348+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:49.349613+0000) 2022-01-31T22:53:49.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:49 smithi171 conmon[51620]: debug 2022-01-31T22:53:49.475+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:49.476646+0000) 2022-01-31T22:53:49.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:49 smithi171 conmon[41853]: debug 2022-01-31T22:53:49.298+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:49.299622+0000) 2022-01-31T22:53:50.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:49 smithi167 conmon[49112]: debug 2022-01-31T22:53:49.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:49.936623+0000) 2022-01-31T22:53:50.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:50 smithi167 conmon[54076]: debug 2022-01-31T22:53:50.104+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:50.105055+0000) 2022-01-31T22:53:50.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:49 smithi167 conmon[60316]: debug 2022-01-31T22:53:49.863+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:49.864301+0000) 2022-01-31T22:53:50.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:50 smithi171 conmon[41853]: debug 2022-01-31T22:53:50.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:50.299758+0000) 2022-01-31T22:53:50.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:50 smithi171 conmon[46715]: debug 2022-01-31T22:53:50.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:50.349781+0000) 2022-01-31T22:53:50.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:50 smithi171 conmon[51620]: debug 2022-01-31T22:53:50.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:50.476818+0000) 2022-01-31T22:53:51.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:50 smithi167 conmon[49112]: debug 2022-01-31T22:53:50.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:50.936848+0000) 2022-01-31T22:53:51.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:51 smithi167 conmon[54076]: debug 2022-01-31T22:53:51.104+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:51.105250+0000) 2022-01-31T22:53:51.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:50 smithi167 conmon[60316]: debug 2022-01-31T22:53:50.863+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:50.864468+0000) 2022-01-31T22:53:51.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:51 smithi171 conmon[46715]: debug 2022-01-31T22:53:51.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:51.349932+0000) 2022-01-31T22:53:51.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:51 smithi171 conmon[41853]: debug 2022-01-31T22:53:51.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:51.299908+0000) 2022-01-31T22:53:51.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:51 smithi171 conmon[51620]: debug 2022-01-31T22:53:51.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:51.476985+0000) 2022-01-31T22:53:52.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:51 smithi167 conmon[49112]: debug 2022-01-31T22:53:51.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:51.936989+0000) 2022-01-31T22:53:52.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:52 smithi167 conmon[54076]: debug 2022-01-31T22:53:52.104+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:52.105432+0000) 2022-01-31T22:53:52.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:51 smithi167 conmon[60316]: debug 2022-01-31T22:53:51.864+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:51.864656+0000) 2022-01-31T22:53:52.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:52 smithi171 conmon[46715]: debug 2022-01-31T22:53:52.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:52.350092+0000) 2022-01-31T22:53:52.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:52 smithi171 conmon[41853]: debug 2022-01-31T22:53:52.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:52.300074+0000) 2022-01-31T22:53:52.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:52 smithi171 conmon[51620]: debug 2022-01-31T22:53:52.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:52.477115+0000) 2022-01-31T22:53:53.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:52 smithi167 conmon[49112]: debug 2022-01-31T22:53:52.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:52.937124+0000) 2022-01-31T22:53:53.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:53 smithi167 conmon[54076]: debug 2022-01-31T22:53:53.105+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.105615+0000) 2022-01-31T22:53:53.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:52 smithi167 conmon[60316]: debug 2022-01-31T22:53:52.864+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:52.864817+0000) 2022-01-31T22:53:53.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[46715]: debug 2022-01-31T22:53:53.349+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.350389+0000) 2022-01-31T22:53:53.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[41853]: debug 2022-01-31T22:53:53.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.300231+0000) 2022-01-31T22:53:53.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[51620]: debug 2022-01-31T22:53:53.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.477311+0000) 2022-01-31T22:53:53.850 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:53 smithi167 conmon[49112]: debug 2022-01-31T22:53:53.690+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.691151+0000) 2022-01-31T22:53:53.851 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:53 smithi167 conmon[54076]: debug 2022-01-31T22:53:53.690+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.691412+0000) 2022-01-31T22:53:53.852 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:53 smithi167 conmon[60316]: debug 2022-01-31T22:53:53.691+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.691624+0000) 2022-01-31T22:53:54.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[41853]: debug 2022-01-31T22:53:53.690+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.690968+0000) 2022-01-31T22:53:54.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[46715]: debug 2022-01-31T22:53:53.691+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.692678+0000) 2022-01-31T22:53:54.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[51620]: debug 2022-01-31T22:53:53.690+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.690876+0000) 2022-01-31T22:53:54.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[35325]: debug 2022-01-31T22:53:53.715+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246335 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:54.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:53 smithi171 conmon[35325]: debug 2022-01-31T22:53:53.927+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:53:54.159 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:53 smithi167 conmon[49112]: debug 2022-01-31T22:53:53.936+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.937345+0000) 2022-01-31T22:53:54.160 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:54 smithi167 conmon[54076]: debug 2022-01-31T22:53:54.105+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:54.105823+0000) 2022-01-31T22:53:54.161 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:53 smithi167 conmon[60316]: debug 2022-01-31T22:53:53.864+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:53.865028+0000) 2022-01-31T22:53:54.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:54 smithi171 conmon[46715]: debug 2022-01-31T22:53:54.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:54.351167+0000) 2022-01-31T22:53:54.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:54 smithi171 conmon[41853]: debug 2022-01-31T22:53:54.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:54.300379+0000) 2022-01-31T22:53:54.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:54 smithi171 conmon[51620]: debug 2022-01-31T22:53:54.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:54.477436+0000) 2022-01-31T22:53:55.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:54 smithi167 conmon[49112]: debug 2022-01-31T22:53:54.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:54.937514+0000) 2022-01-31T22:53:55.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:55 smithi167 conmon[54076]: debug 2022-01-31T22:53:55.105+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:55.105976+0000) 2022-01-31T22:53:55.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:54 smithi167 conmon[60316]: debug 2022-01-31T22:53:54.864+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:54.865194+0000) 2022-01-31T22:53:55.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:55 smithi171 conmon[46715]: debug 2022-01-31T22:53:55.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:55.351267+0000) 2022-01-31T22:53:55.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:55 smithi171 conmon[41853]: debug 2022-01-31T22:53:55.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:55.300514+0000) 2022-01-31T22:53:55.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:55 smithi171 conmon[51620]: debug 2022-01-31T22:53:55.476+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:55.477619+0000) 2022-01-31T22:53:56.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:55 smithi167 conmon[49112]: debug 2022-01-31T22:53:55.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:55.937700+0000) 2022-01-31T22:53:56.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:56 smithi167 conmon[54076]: debug 2022-01-31T22:53:56.105+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:56.106159+0000) 2022-01-31T22:53:56.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:55 smithi167 conmon[60316]: debug 2022-01-31T22:53:55.864+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:55.865380+0000) 2022-01-31T22:53:56.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:56 smithi171 conmon[46715]: debug 2022-01-31T22:53:56.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:56.351377+0000) 2022-01-31T22:53:56.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:56 smithi171 conmon[41853]: debug 2022-01-31T22:53:56.299+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:56.300706+0000) 2022-01-31T22:53:56.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:56 smithi171 conmon[51620]: debug 2022-01-31T22:53:56.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:56.477809+0000) 2022-01-31T22:53:57.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:56 smithi167 conmon[49112]: debug 2022-01-31T22:53:56.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:56.937881+0000) 2022-01-31T22:53:57.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:57 smithi167 conmon[54076]: debug 2022-01-31T22:53:57.105+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:57.106343+0000) 2022-01-31T22:53:57.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:56 smithi167 conmon[60316]: debug 2022-01-31T22:53:56.865+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:56.865566+0000) 2022-01-31T22:53:57.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:57 smithi171 conmon[46715]: debug 2022-01-31T22:53:57.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:57.351535+0000) 2022-01-31T22:53:57.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:57 smithi171 conmon[41853]: debug 2022-01-31T22:53:57.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:57.300852+0000) 2022-01-31T22:53:57.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:57 smithi171 conmon[51620]: debug 2022-01-31T22:53:57.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:57.478016+0000) 2022-01-31T22:53:58.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:57 smithi167 conmon[49112]: debug 2022-01-31T22:53:57.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:57.938050+0000) 2022-01-31T22:53:58.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:58 smithi167 conmon[54076]: debug 2022-01-31T22:53:58.106+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.106512+0000) 2022-01-31T22:53:58.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:57 smithi167 conmon[60316]: debug 2022-01-31T22:53:57.865+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:57.865729+0000) 2022-01-31T22:53:58.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[41853]: debug 2022-01-31T22:53:58.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.300957+0000) 2022-01-31T22:53:58.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[46715]: debug 2022-01-31T22:53:58.350+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.351692+0000) 2022-01-31T22:53:58.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[51620]: debug 2022-01-31T22:53:58.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.478187+0000) 2022-01-31T22:53:58.851 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:58 smithi167 conmon[60316]: debug 2022-01-31T22:53:58.718+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.719351+0000) 2022-01-31T22:53:58.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:58 smithi167 conmon[49112]: debug 2022-01-31T22:53:58.718+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.718725+0000) 2022-01-31T22:53:58.852 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:58 smithi167 conmon[54076]: debug 2022-01-31T22:53:58.717+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.718445+0000) 2022-01-31T22:53:59.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[35325]: debug 2022-01-31T22:53:58.744+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246446 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:53:59.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[46715]: debug 2022-01-31T22:53:58.717+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.718217+0000) 2022-01-31T22:53:59.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[51620]: debug 2022-01-31T22:53:58.717+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.718052+0000) 2022-01-31T22:53:59.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:58 smithi171 conmon[41853]: debug 2022-01-31T22:53:58.717+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.717949+0000) 2022-01-31T22:53:59.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:58 smithi167 conmon[49112]: debug 2022-01-31T22:53:58.937+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.938228+0000) 2022-01-31T22:53:59.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:58 smithi167 conmon[60316]: debug 2022-01-31T22:53:58.865+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:58.865841+0000) 2022-01-31T22:53:59.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:53:59 smithi167 conmon[54076]: debug 2022-01-31T22:53:59.106+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:59.106700+0000) 2022-01-31T22:53:59.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:53:59 smithi171 conmon[46715]: debug 2022-01-31T22:53:59.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:59.351938+0000) 2022-01-31T22:53:59.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:53:59 smithi171 conmon[41853]: debug 2022-01-31T22:53:59.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:59.301138+0000) 2022-01-31T22:53:59.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:53:59 smithi171 conmon[51620]: debug 2022-01-31T22:53:59.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:59.478394+0000) 2022-01-31T22:54:00.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:53:59 smithi167 conmon[49112]: debug 2022-01-31T22:53:59.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:59.938420+0000) 2022-01-31T22:54:00.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:53:59 smithi167 conmon[60316]: debug 2022-01-31T22:53:59.865+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:53:59.865974+0000) 2022-01-31T22:54:00.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:00 smithi167 conmon[54076]: debug 2022-01-31T22:54:00.106+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:00.106838+0000) 2022-01-31T22:54:00.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:00 smithi171 conmon[41853]: debug 2022-01-31T22:54:00.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:00.301257+0000) 2022-01-31T22:54:00.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:00 smithi171 conmon[46715]: debug 2022-01-31T22:54:00.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:00.352103+0000) 2022-01-31T22:54:00.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:00 smithi171 conmon[51620]: debug 2022-01-31T22:54:00.477+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:00.478587+0000) 2022-01-31T22:54:01.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:00 smithi167 conmon[49112]: debug 2022-01-31T22:54:00.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:00.938587+0000) 2022-01-31T22:54:01.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:00 smithi167 conmon[60316]: debug 2022-01-31T22:54:00.865+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:00.866128+0000) 2022-01-31T22:54:01.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:01 smithi167 conmon[54076]: debug 2022-01-31T22:54:01.106+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:01.107038+0000) 2022-01-31T22:54:01.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:01 smithi171 conmon[46715]: debug 2022-01-31T22:54:01.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:01.352304+0000) 2022-01-31T22:54:01.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:01 smithi171 conmon[41853]: debug 2022-01-31T22:54:01.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:01.301452+0000) 2022-01-31T22:54:01.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:01 smithi171 conmon[51620]: debug 2022-01-31T22:54:01.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:01.478798+0000) 2022-01-31T22:54:02.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:01 smithi167 conmon[49112]: debug 2022-01-31T22:54:01.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:01.938780+0000) 2022-01-31T22:54:02.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:02 smithi167 conmon[54076]: debug 2022-01-31T22:54:02.106+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:02.107223+0000) 2022-01-31T22:54:02.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:01 smithi167 conmon[60316]: debug 2022-01-31T22:54:01.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:01.866293+0000) 2022-01-31T22:54:02.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:02 smithi171 conmon[41853]: debug 2022-01-31T22:54:02.300+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:02.301623+0000) 2022-01-31T22:54:02.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:02 smithi171 conmon[46715]: debug 2022-01-31T22:54:02.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:02.352485+0000) 2022-01-31T22:54:02.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:02 smithi171 conmon[51620]: debug 2022-01-31T22:54:02.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:02.478928+0000) 2022-01-31T22:54:03.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:02 smithi167 conmon[49112]: debug 2022-01-31T22:54:02.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:02.938962+0000) 2022-01-31T22:54:03.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:02 smithi167 conmon[60316]: debug 2022-01-31T22:54:02.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:02.866478+0000) 2022-01-31T22:54:03.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:03 smithi167 conmon[54076]: debug 2022-01-31T22:54:03.107+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.107423+0000) 2022-01-31T22:54:03.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[41853]: debug 2022-01-31T22:54:03.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.301774+0000) 2022-01-31T22:54:03.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[46715]: debug 2022-01-31T22:54:03.351+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.352670+0000) 2022-01-31T22:54:03.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[51620]: debug 2022-01-31T22:54:03.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.479084+0000) 2022-01-31T22:54:03.852 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:03 smithi167 conmon[49112]: debug 2022-01-31T22:54:03.746+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.747482+0000) 2022-01-31T22:54:03.853 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:03 smithi167 conmon[54076]: debug 2022-01-31T22:54:03.746+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.747210+0000) 2022-01-31T22:54:03.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:03 smithi167 conmon[60316]: debug 2022-01-31T22:54:03.747+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.747759+0000) 2022-01-31T22:54:04.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[35325]: debug 2022-01-31T22:54:03.772+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246558 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:04.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[41853]: debug 2022-01-31T22:54:03.746+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.746852+0000) 2022-01-31T22:54:04.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[46715]: debug 2022-01-31T22:54:03.747+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.747871+0000) 2022-01-31T22:54:04.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:03 smithi171 conmon[51620]: debug 2022-01-31T22:54:03.746+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.747039+0000) 2022-01-31T22:54:04.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:03 smithi167 conmon[49112]: debug 2022-01-31T22:54:03.938+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.939149+0000) 2022-01-31T22:54:04.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:04 smithi167 conmon[54076]: debug 2022-01-31T22:54:04.107+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:04.107659+0000) 2022-01-31T22:54:04.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:03 smithi167 conmon[60316]: debug 2022-01-31T22:54:03.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:03.866600+0000) 2022-01-31T22:54:04.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:04 smithi171 conmon[41853]: debug 2022-01-31T22:54:04.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:04.301957+0000) 2022-01-31T22:54:04.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:04 smithi171 conmon[46715]: debug 2022-01-31T22:54:04.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:04.352920+0000) 2022-01-31T22:54:04.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:04 smithi171 conmon[51620]: debug 2022-01-31T22:54:04.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:04.479218+0000) 2022-01-31T22:54:05.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:04 smithi167 conmon[49112]: debug 2022-01-31T22:54:04.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:04.939301+0000) 2022-01-31T22:54:05.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:04 smithi167 conmon[60316]: debug 2022-01-31T22:54:04.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:04.866778+0000) 2022-01-31T22:54:05.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:05 smithi167 conmon[54076]: debug 2022-01-31T22:54:05.107+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:05.107801+0000) 2022-01-31T22:54:05.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:05 smithi171 conmon[41853]: debug 2022-01-31T22:54:05.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:05.302085+0000) 2022-01-31T22:54:05.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:05 smithi171 conmon[46715]: debug 2022-01-31T22:54:05.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:05.353074+0000) 2022-01-31T22:54:05.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:05 smithi171 conmon[51620]: debug 2022-01-31T22:54:05.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:05.479342+0000) 2022-01-31T22:54:06.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:05 smithi167 conmon[49112]: debug 2022-01-31T22:54:05.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:05.939518+0000) 2022-01-31T22:54:06.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:05 smithi167 conmon[60316]: debug 2022-01-31T22:54:05.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:05.867002+0000) 2022-01-31T22:54:06.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:06 smithi167 conmon[54076]: debug 2022-01-31T22:54:06.107+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:06.107997+0000) 2022-01-31T22:54:06.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:06 smithi171 conmon[41853]: debug 2022-01-31T22:54:06.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:06.302259+0000) 2022-01-31T22:54:06.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:06 smithi171 conmon[46715]: debug 2022-01-31T22:54:06.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:06.353249+0000) 2022-01-31T22:54:06.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:06 smithi171 conmon[51620]: debug 2022-01-31T22:54:06.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:06.479528+0000) 2022-01-31T22:54:07.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:06 smithi167 conmon[49112]: debug 2022-01-31T22:54:06.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:06.939678+0000) 2022-01-31T22:54:07.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:07 smithi167 conmon[54076]: debug 2022-01-31T22:54:07.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:07.108188+0000) 2022-01-31T22:54:07.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:06 smithi167 conmon[60316]: debug 2022-01-31T22:54:06.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:06.867197+0000) 2022-01-31T22:54:07.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:07 smithi171 conmon[41853]: debug 2022-01-31T22:54:07.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:07.302402+0000) 2022-01-31T22:54:07.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:07 smithi171 conmon[46715]: debug 2022-01-31T22:54:07.352+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:07.353398+0000) 2022-01-31T22:54:07.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:07 smithi171 conmon[51620]: debug 2022-01-31T22:54:07.478+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:07.479732+0000) 2022-01-31T22:54:08.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:07 smithi167 conmon[49112]: debug 2022-01-31T22:54:07.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:07.939845+0000) 2022-01-31T22:54:08.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:08 smithi167 conmon[54076]: debug 2022-01-31T22:54:08.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.108392+0000) 2022-01-31T22:54:08.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:07 smithi167 conmon[60316]: debug 2022-01-31T22:54:07.866+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:07.867404+0000) 2022-01-31T22:54:08.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[41853]: debug 2022-01-31T22:54:08.301+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.302541+0000) 2022-01-31T22:54:08.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[46715]: debug 2022-01-31T22:54:08.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.353588+0000) 2022-01-31T22:54:08.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[51620]: debug 2022-01-31T22:54:08.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.479906+0000) 2022-01-31T22:54:08.853 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:08 smithi167 conmon[49112]: debug 2022-01-31T22:54:08.775+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.775940+0000) 2022-01-31T22:54:08.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:08 smithi167 conmon[54076]: debug 2022-01-31T22:54:08.774+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.775427+0000) 2022-01-31T22:54:08.854 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:08 smithi167 conmon[60316]: debug 2022-01-31T22:54:08.775+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.776243+0000) 2022-01-31T22:54:09.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[46715]: debug 2022-01-31T22:54:08.775+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.776245+0000) 2022-01-31T22:54:09.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[35325]: debug 2022-01-31T22:54:08.800+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246670 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:09.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[35325]: debug 2022-01-31T22:54:08.928+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:54:09.095 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[41853]: debug 2022-01-31T22:54:08.774+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.775122+0000) 2022-01-31T22:54:09.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:08 smithi171 conmon[51620]: debug 2022-01-31T22:54:08.775+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.775820+0000) 2022-01-31T22:54:09.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:08 smithi167 conmon[49112]: debug 2022-01-31T22:54:08.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.940052+0000) 2022-01-31T22:54:09.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:09 smithi167 conmon[54076]: debug 2022-01-31T22:54:09.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:09.108597+0000) 2022-01-31T22:54:09.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:08 smithi167 conmon[60316]: debug 2022-01-31T22:54:08.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:08.867572+0000) 2022-01-31T22:54:09.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:09 smithi171 conmon[41853]: debug 2022-01-31T22:54:09.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:09.302729+0000) 2022-01-31T22:54:09.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:09 smithi171 conmon[46715]: debug 2022-01-31T22:54:09.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:09.353773+0000) 2022-01-31T22:54:09.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:09 smithi171 conmon[51620]: debug 2022-01-31T22:54:09.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:09.480032+0000) 2022-01-31T22:54:10.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:09 smithi167 conmon[49112]: debug 2022-01-31T22:54:09.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:09.940219+0000) 2022-01-31T22:54:10.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:10 smithi167 conmon[54076]: debug 2022-01-31T22:54:10.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:10.108780+0000) 2022-01-31T22:54:10.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:09 smithi167 conmon[60316]: debug 2022-01-31T22:54:09.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:09.867692+0000) 2022-01-31T22:54:10.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:10 smithi171 conmon[41853]: debug 2022-01-31T22:54:10.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:10.302920+0000) 2022-01-31T22:54:10.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:10 smithi171 conmon[46715]: debug 2022-01-31T22:54:10.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:10.353946+0000) 2022-01-31T22:54:10.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:10 smithi171 conmon[51620]: debug 2022-01-31T22:54:10.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:10.480165+0000) 2022-01-31T22:54:11.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:10 smithi167 conmon[49112]: debug 2022-01-31T22:54:10.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:10.940396+0000) 2022-01-31T22:54:11.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:11 smithi167 conmon[54076]: debug 2022-01-31T22:54:11.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:11.108985+0000) 2022-01-31T22:54:11.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:10 smithi167 conmon[60316]: debug 2022-01-31T22:54:10.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:10.867881+0000) 2022-01-31T22:54:11.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:11 smithi171 conmon[41853]: debug 2022-01-31T22:54:11.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:11.303077+0000) 2022-01-31T22:54:11.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:11 smithi171 conmon[46715]: debug 2022-01-31T22:54:11.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:11.354093+0000) 2022-01-31T22:54:11.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:11 smithi171 conmon[51620]: debug 2022-01-31T22:54:11.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:11.480347+0000) 2022-01-31T22:54:12.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:11 smithi167 conmon[49112]: debug 2022-01-31T22:54:11.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:11.940599+0000) 2022-01-31T22:54:12.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:12 smithi167 conmon[54076]: debug 2022-01-31T22:54:12.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:12.109180+0000) 2022-01-31T22:54:12.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:11 smithi167 conmon[60316]: debug 2022-01-31T22:54:11.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:11.868068+0000) 2022-01-31T22:54:12.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:12 smithi171 conmon[41853]: debug 2022-01-31T22:54:12.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:12.303208+0000) 2022-01-31T22:54:12.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:12 smithi171 conmon[46715]: debug 2022-01-31T22:54:12.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:12.354272+0000) 2022-01-31T22:54:12.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:12 smithi171 conmon[51620]: debug 2022-01-31T22:54:12.479+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:12.480553+0000) 2022-01-31T22:54:13.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:12 smithi167 conmon[49112]: debug 2022-01-31T22:54:12.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:12.940791+0000) 2022-01-31T22:54:13.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:13 smithi167 conmon[54076]: debug 2022-01-31T22:54:13.109+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.109337+0000) 2022-01-31T22:54:13.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:12 smithi167 conmon[60316]: debug 2022-01-31T22:54:12.868+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:12.868254+0000) 2022-01-31T22:54:13.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[46715]: debug 2022-01-31T22:54:13.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.354424+0000) 2022-01-31T22:54:13.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[41853]: debug 2022-01-31T22:54:13.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.303356+0000) 2022-01-31T22:54:13.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[51620]: debug 2022-01-31T22:54:13.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.480781+0000) 2022-01-31T22:54:13.854 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:13 smithi167 conmon[54076]: debug 2022-01-31T22:54:13.802+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.803190+0000) 2022-01-31T22:54:13.855 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:13 smithi167 conmon[49112]: debug 2022-01-31T22:54:13.803+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.803646+0000) 2022-01-31T22:54:13.855 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:13 smithi167 conmon[60316]: debug 2022-01-31T22:54:13.803+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.803988+0000) 2022-01-31T22:54:14.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[35325]: debug 2022-01-31T22:54:13.828+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246782 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:14.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[41853]: debug 2022-01-31T22:54:13.802+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.803178+0000) 2022-01-31T22:54:14.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[46715]: debug 2022-01-31T22:54:13.802+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.803548+0000) 2022-01-31T22:54:14.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:13 smithi171 conmon[51620]: debug 2022-01-31T22:54:13.802+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.803640+0000) 2022-01-31T22:54:14.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:13 smithi167 conmon[49112]: debug 2022-01-31T22:54:13.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.940965+0000) 2022-01-31T22:54:14.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:14 smithi167 conmon[54076]: debug 2022-01-31T22:54:14.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:14.109541+0000) 2022-01-31T22:54:14.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:13 smithi167 conmon[60316]: debug 2022-01-31T22:54:13.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:13.868400+0000) 2022-01-31T22:54:14.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:14 smithi171 conmon[46715]: debug 2022-01-31T22:54:14.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:14.354543+0000) 2022-01-31T22:54:14.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:14 smithi171 conmon[41853]: debug 2022-01-31T22:54:14.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:14.303491+0000) 2022-01-31T22:54:14.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:14 smithi171 conmon[51620]: debug 2022-01-31T22:54:14.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:14.480954+0000) 2022-01-31T22:54:15.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:14 smithi167 conmon[49112]: debug 2022-01-31T22:54:14.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:14.941086+0000) 2022-01-31T22:54:15.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:15 smithi167 conmon[54076]: debug 2022-01-31T22:54:15.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:15.109658+0000) 2022-01-31T22:54:15.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:14 smithi167 conmon[60316]: debug 2022-01-31T22:54:14.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:14.868597+0000) 2022-01-31T22:54:15.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:15 smithi171 conmon[46715]: debug 2022-01-31T22:54:15.353+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:15.354637+0000) 2022-01-31T22:54:15.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:15 smithi171 conmon[41853]: debug 2022-01-31T22:54:15.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:15.303588+0000) 2022-01-31T22:54:15.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:15 smithi171 conmon[51620]: debug 2022-01-31T22:54:15.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:15.481102+0000) 2022-01-31T22:54:16.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:15 smithi167 conmon[49112]: debug 2022-01-31T22:54:15.939+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:15.941268+0000) 2022-01-31T22:54:16.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:16 smithi167 conmon[54076]: debug 2022-01-31T22:54:16.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:16.109803+0000) 2022-01-31T22:54:16.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:15 smithi167 conmon[60316]: debug 2022-01-31T22:54:15.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:15.868725+0000) 2022-01-31T22:54:16.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:16 smithi171 conmon[46715]: debug 2022-01-31T22:54:16.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:16.354790+0000) 2022-01-31T22:54:16.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:16 smithi171 conmon[41853]: debug 2022-01-31T22:54:16.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:16.303714+0000) 2022-01-31T22:54:16.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:16 smithi171 conmon[51620]: debug 2022-01-31T22:54:16.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:16.481301+0000) 2022-01-31T22:54:17.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:16 smithi167 conmon[49112]: debug 2022-01-31T22:54:16.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:16.941462+0000) 2022-01-31T22:54:17.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:17 smithi167 conmon[54076]: debug 2022-01-31T22:54:17.109+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:17.109990+0000) 2022-01-31T22:54:17.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:16 smithi167 conmon[60316]: debug 2022-01-31T22:54:16.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:16.868886+0000) 2022-01-31T22:54:17.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:17 smithi171 conmon[46715]: debug 2022-01-31T22:54:17.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:17.354973+0000) 2022-01-31T22:54:17.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:17 smithi171 conmon[41853]: debug 2022-01-31T22:54:17.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:17.303854+0000) 2022-01-31T22:54:17.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:17 smithi171 conmon[51620]: debug 2022-01-31T22:54:17.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:17.481495+0000) 2022-01-31T22:54:18.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:17 smithi167 conmon[49112]: debug 2022-01-31T22:54:17.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:17.941615+0000) 2022-01-31T22:54:18.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:18 smithi167 conmon[54076]: debug 2022-01-31T22:54:18.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.110147+0000) 2022-01-31T22:54:18.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:17 smithi167 conmon[60316]: debug 2022-01-31T22:54:17.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:17.869023+0000) 2022-01-31T22:54:18.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[41853]: debug 2022-01-31T22:54:18.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.303993+0000) 2022-01-31T22:54:18.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[46715]: debug 2022-01-31T22:54:18.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.355145+0000) 2022-01-31T22:54:18.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[51620]: debug 2022-01-31T22:54:18.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.481699+0000) 2022-01-31T22:54:18.854 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:18 smithi167 conmon[49112]: debug 2022-01-31T22:54:18.830+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.831989+0000) 2022-01-31T22:54:18.855 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:18 smithi167 conmon[54076]: debug 2022-01-31T22:54:18.831+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.832592+0000) 2022-01-31T22:54:18.856 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:18 smithi167 conmon[60316]: debug 2022-01-31T22:54:18.830+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.832343+0000) 2022-01-31T22:54:19.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[35325]: debug 2022-01-31T22:54:18.856+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 246894 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:19.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[41853]: debug 2022-01-31T22:54:18.830+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.831467+0000) 2022-01-31T22:54:19.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[46715]: debug 2022-01-31T22:54:18.831+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.831796+0000) 2022-01-31T22:54:19.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:18 smithi171 conmon[51620]: debug 2022-01-31T22:54:18.831+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.832372+0000) 2022-01-31T22:54:19.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:18 smithi167 conmon[49112]: debug 2022-01-31T22:54:18.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.941771+0000) 2022-01-31T22:54:19.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:19 smithi167 conmon[54076]: debug 2022-01-31T22:54:19.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:19.110270+0000) 2022-01-31T22:54:19.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:18 smithi167 conmon[60316]: debug 2022-01-31T22:54:18.867+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:18.869135+0000) 2022-01-31T22:54:19.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:19 smithi171 conmon[46715]: debug 2022-01-31T22:54:19.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:19.355311+0000) 2022-01-31T22:54:19.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:19 smithi171 conmon[51620]: debug 2022-01-31T22:54:19.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:19.481870+0000) 2022-01-31T22:54:19.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:19 smithi171 conmon[41853]: debug 2022-01-31T22:54:19.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:19.304135+0000) 2022-01-31T22:54:20.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:19 smithi167 conmon[49112]: debug 2022-01-31T22:54:19.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:19.941977+0000) 2022-01-31T22:54:20.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:20 smithi167 conmon[54076]: debug 2022-01-31T22:54:20.108+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:20.110400+0000) 2022-01-31T22:54:20.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:19 smithi167 conmon[60316]: debug 2022-01-31T22:54:19.868+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:19.869283+0000) 2022-01-31T22:54:20.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:20 smithi171 conmon[41853]: debug 2022-01-31T22:54:20.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:20.304232+0000) 2022-01-31T22:54:20.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:20 smithi171 conmon[46715]: debug 2022-01-31T22:54:20.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:20.355464+0000) 2022-01-31T22:54:20.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:20 smithi171 conmon[51620]: debug 2022-01-31T22:54:20.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:20.482047+0000) 2022-01-31T22:54:21.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:20 smithi167 conmon[49112]: debug 2022-01-31T22:54:20.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:20.942177+0000) 2022-01-31T22:54:21.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:21 smithi167 conmon[54076]: debug 2022-01-31T22:54:21.109+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:21.110586+0000) 2022-01-31T22:54:21.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:20 smithi167 conmon[60316]: debug 2022-01-31T22:54:20.868+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:20.869468+0000) 2022-01-31T22:54:21.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:21 smithi171 conmon[41853]: debug 2022-01-31T22:54:21.302+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:21.304434+0000) 2022-01-31T22:54:21.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:21 smithi171 conmon[46715]: debug 2022-01-31T22:54:21.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:21.355645+0000) 2022-01-31T22:54:21.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:21 smithi171 conmon[51620]: debug 2022-01-31T22:54:21.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:21.482224+0000) 2022-01-31T22:54:22.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:21 smithi167 conmon[49112]: debug 2022-01-31T22:54:21.940+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:21.942344+0000) 2022-01-31T22:54:22.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:22 smithi167 conmon[54076]: debug 2022-01-31T22:54:22.109+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:22.110733+0000) 2022-01-31T22:54:22.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:21 smithi167 conmon[60316]: debug 2022-01-31T22:54:21.868+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:21.869680+0000) 2022-01-31T22:54:22.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:22 smithi171 conmon[41853]: debug 2022-01-31T22:54:22.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:22.304578+0000) 2022-01-31T22:54:22.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:22 smithi171 conmon[46715]: debug 2022-01-31T22:54:22.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:22.355837+0000) 2022-01-31T22:54:22.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:22 smithi171 conmon[51620]: debug 2022-01-31T22:54:22.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:22.482420+0000) 2022-01-31T22:54:23.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:22 smithi167 conmon[49112]: debug 2022-01-31T22:54:22.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:22.942511+0000) 2022-01-31T22:54:23.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:23 smithi167 conmon[54076]: debug 2022-01-31T22:54:23.109+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.110899+0000) 2022-01-31T22:54:23.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:22 smithi167 conmon[60316]: debug 2022-01-31T22:54:22.868+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:22.869859+0000) 2022-01-31T22:54:23.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[46715]: debug 2022-01-31T22:54:23.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.356010+0000) 2022-01-31T22:54:23.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[41853]: debug 2022-01-31T22:54:23.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.304784+0000) 2022-01-31T22:54:23.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[51620]: debug 2022-01-31T22:54:23.480+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.482640+0000) 2022-01-31T22:54:24.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[41853]: debug 2022-01-31T22:54:23.857+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.859048+0000) 2022-01-31T22:54:24.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[46715]: debug 2022-01-31T22:54:23.858+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.860074+0000) 2022-01-31T22:54:24.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[51620]: debug 2022-01-31T22:54:23.858+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.860422+0000) 2022-01-31T22:54:24.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[35325]: debug 2022-01-31T22:54:23.889+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247005 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:24.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:23 smithi171 conmon[35325]: debug 2022-01-31T22:54:23.928+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:54:24.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:23 smithi167 conmon[49112]: debug 2022-01-31T22:54:23.857+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.859440+0000) 2022-01-31T22:54:24.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:23 smithi167 conmon[49112]: debug 2022-01-31T22:54:23.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.942641+0000) 2022-01-31T22:54:24.097 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:23 smithi167 conmon[54076]: debug 2022-01-31T22:54:23.858+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.859682+0000) 2022-01-31T22:54:24.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:23 smithi167 conmon[60316]: debug 2022-01-31T22:54:23.858+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.860070+0000) 2022-01-31T22:54:24.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:23 smithi167 conmon[60316]: debug 2022-01-31T22:54:23.868+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:23.870016+0000) 2022-01-31T22:54:24.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:24 smithi167 conmon[54076]: debug 2022-01-31T22:54:24.110+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:24.111101+0000) 2022-01-31T22:54:24.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:24 smithi171 conmon[46715]: debug 2022-01-31T22:54:24.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:24.356226+0000) 2022-01-31T22:54:24.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:24 smithi171 conmon[41853]: debug 2022-01-31T22:54:24.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:24.304985+0000) 2022-01-31T22:54:24.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:24 smithi171 conmon[51620]: debug 2022-01-31T22:54:24.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:24.482829+0000) 2022-01-31T22:54:25.112 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:24 smithi167 conmon[49112]: debug 2022-01-31T22:54:24.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:24.942753+0000) 2022-01-31T22:54:25.112 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:25 smithi167 conmon[54076]: debug 2022-01-31T22:54:25.110+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:25.111255+0000) 2022-01-31T22:54:25.113 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:24 smithi167 conmon[60316]: debug 2022-01-31T22:54:24.869+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:24.870198+0000) 2022-01-31T22:54:25.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:25 smithi171 conmon[41853]: debug 2022-01-31T22:54:25.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:25.305131+0000) 2022-01-31T22:54:25.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:25 smithi171 conmon[46715]: debug 2022-01-31T22:54:25.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:25.356396+0000) 2022-01-31T22:54:25.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:25 smithi171 conmon[51620]: debug 2022-01-31T22:54:25.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:25.482996+0000) 2022-01-31T22:54:26.115 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:25 smithi167 conmon[49112]: debug 2022-01-31T22:54:25.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:25.942938+0000) 2022-01-31T22:54:26.116 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:26 smithi167 conmon[54076]: debug 2022-01-31T22:54:26.110+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:26.111405+0000) 2022-01-31T22:54:26.117 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:25 smithi167 conmon[60316]: debug 2022-01-31T22:54:25.869+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:25.870396+0000) 2022-01-31T22:54:26.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:26 smithi171 conmon[41853]: debug 2022-01-31T22:54:26.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:26.305367+0000) 2022-01-31T22:54:26.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:26 smithi171 conmon[46715]: debug 2022-01-31T22:54:26.354+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:26.356622+0000) 2022-01-31T22:54:26.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:26 smithi171 conmon[51620]: debug 2022-01-31T22:54:26.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:26.483197+0000) 2022-01-31T22:54:27.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:26 smithi167 conmon[49112]: debug 2022-01-31T22:54:26.941+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:26.943111+0000) 2022-01-31T22:54:27.120 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:27 smithi167 conmon[54076]: debug 2022-01-31T22:54:27.110+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:27.111570+0000) 2022-01-31T22:54:27.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:26 smithi167 conmon[60316]: debug 2022-01-31T22:54:26.869+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:26.870605+0000) 2022-01-31T22:54:27.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:27 smithi171 conmon[41853]: debug 2022-01-31T22:54:27.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:27.305512+0000) 2022-01-31T22:54:27.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:27 smithi171 conmon[46715]: debug 2022-01-31T22:54:27.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:27.356780+0000) 2022-01-31T22:54:27.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:27 smithi171 conmon[51620]: debug 2022-01-31T22:54:27.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:27.483407+0000) 2022-01-31T22:54:28.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:27 smithi167 conmon[49112]: debug 2022-01-31T22:54:27.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:27.943302+0000) 2022-01-31T22:54:28.123 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:28 smithi167 conmon[54076]: debug 2022-01-31T22:54:28.111+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.111765+0000) 2022-01-31T22:54:28.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:27 smithi167 conmon[60316]: debug 2022-01-31T22:54:27.870+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:27.870794+0000) 2022-01-31T22:54:28.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[46715]: debug 2022-01-31T22:54:28.355+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.356914+0000) 2022-01-31T22:54:28.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[41853]: debug 2022-01-31T22:54:28.303+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.305700+0000) 2022-01-31T22:54:28.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[51620]: debug 2022-01-31T22:54:28.481+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.483606+0000) 2022-01-31T22:54:29.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[35325]: debug 2022-01-31T22:54:28.916+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247117 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:29.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[46715]: debug 2022-01-31T22:54:28.892+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.894291+0000) 2022-01-31T22:54:29.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[51620]: debug 2022-01-31T22:54:28.891+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.893553+0000) 2022-01-31T22:54:29.094 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:28 smithi171 conmon[41853]: debug 2022-01-31T22:54:28.891+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.893184+0000) 2022-01-31T22:54:29.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:28 smithi167 conmon[49112]: debug 2022-01-31T22:54:28.891+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.893373+0000) 2022-01-31T22:54:29.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:28 smithi167 conmon[49112]: debug 2022-01-31T22:54:28.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.943497+0000) 2022-01-31T22:54:29.126 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:28 smithi167 conmon[54076]: debug 2022-01-31T22:54:28.892+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.894369+0000) 2022-01-31T22:54:29.126 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:29 smithi167 conmon[54076]: debug 2022-01-31T22:54:29.111+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:29.111956+0000) 2022-01-31T22:54:29.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:28 smithi167 conmon[60316]: debug 2022-01-31T22:54:28.869+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.870924+0000) 2022-01-31T22:54:29.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:28 smithi167 conmon[60316]: debug 2022-01-31T22:54:28.892+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:28.893739+0000) 2022-01-31T22:54:29.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:29 smithi171 conmon[41853]: debug 2022-01-31T22:54:29.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:29.305886+0000) 2022-01-31T22:54:29.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:29 smithi171 conmon[46715]: debug 2022-01-31T22:54:29.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:29.357078+0000) 2022-01-31T22:54:29.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:29 smithi171 conmon[51620]: debug 2022-01-31T22:54:29.482+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:29.483832+0000) 2022-01-31T22:54:30.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:29 smithi167 conmon[49112]: debug 2022-01-31T22:54:29.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:29.943713+0000) 2022-01-31T22:54:30.130 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:30 smithi167 conmon[54076]: debug 2022-01-31T22:54:30.110+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:30.112079+0000) 2022-01-31T22:54:30.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:29 smithi167 conmon[60316]: debug 2022-01-31T22:54:29.870+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:29.871083+0000) 2022-01-31T22:54:30.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:30 smithi171 conmon[46715]: debug 2022-01-31T22:54:30.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:30.357231+0000) 2022-01-31T22:54:30.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:30 smithi171 conmon[41853]: debug 2022-01-31T22:54:30.304+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:30.305978+0000) 2022-01-31T22:54:30.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:30 smithi171 conmon[51620]: debug 2022-01-31T22:54:30.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:30.483958+0000) 2022-01-31T22:54:31.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:30 smithi167 conmon[49112]: debug 2022-01-31T22:54:30.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:30.943889+0000) 2022-01-31T22:54:31.134 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:31 smithi167 conmon[54076]: debug 2022-01-31T22:54:31.110+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:31.112244+0000) 2022-01-31T22:54:31.135 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:30 smithi167 conmon[60316]: debug 2022-01-31T22:54:30.870+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:30.871222+0000) 2022-01-31T22:54:31.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:31 smithi171 conmon[46715]: debug 2022-01-31T22:54:31.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:31.357423+0000) 2022-01-31T22:54:31.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:31 smithi171 conmon[41853]: debug 2022-01-31T22:54:31.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:31.306094+0000) 2022-01-31T22:54:31.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:31 smithi171 conmon[51620]: debug 2022-01-31T22:54:31.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:31.484108+0000) 2022-01-31T22:54:32.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:31 smithi167 conmon[49112]: debug 2022-01-31T22:54:31.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:31.944077+0000) 2022-01-31T22:54:32.139 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:32 smithi167 conmon[54076]: debug 2022-01-31T22:54:32.111+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:32.112428+0000) 2022-01-31T22:54:32.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:31 smithi167 conmon[60316]: debug 2022-01-31T22:54:31.870+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:31.871388+0000) 2022-01-31T22:54:32.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:32 smithi171 conmon[46715]: debug 2022-01-31T22:54:32.356+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:32.357610+0000) 2022-01-31T22:54:32.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:32 smithi171 conmon[41853]: debug 2022-01-31T22:54:32.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:32.306298+0000) 2022-01-31T22:54:32.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:32 smithi171 conmon[51620]: debug 2022-01-31T22:54:32.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:32.484311+0000) 2022-01-31T22:54:33.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:32 smithi167 conmon[49112]: debug 2022-01-31T22:54:32.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:32.944234+0000) 2022-01-31T22:54:33.143 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:33 smithi167 conmon[54076]: debug 2022-01-31T22:54:33.111+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.112579+0000) 2022-01-31T22:54:33.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:32 smithi167 conmon[60316]: debug 2022-01-31T22:54:32.870+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:32.871601+0000) 2022-01-31T22:54:33.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[46715]: debug 2022-01-31T22:54:33.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.357799+0000) 2022-01-31T22:54:33.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[41853]: debug 2022-01-31T22:54:33.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.306461+0000) 2022-01-31T22:54:33.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[51620]: debug 2022-01-31T22:54:33.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.484508+0000) 2022-01-31T22:54:34.093 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[35325]: debug 2022-01-31T22:54:33.945+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247228 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:34.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[41853]: debug 2022-01-31T22:54:33.920+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.921140+0000) 2022-01-31T22:54:34.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[46715]: debug 2022-01-31T22:54:33.921+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.922055+0000) 2022-01-31T22:54:34.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:33 smithi171 conmon[51620]: debug 2022-01-31T22:54:33.919+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.920403+0000) 2022-01-31T22:54:34.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:33 smithi167 conmon[49112]: debug 2022-01-31T22:54:33.919+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.921087+0000) 2022-01-31T22:54:34.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:33 smithi167 conmon[49112]: debug 2022-01-31T22:54:33.942+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.944409+0000) 2022-01-31T22:54:34.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:33 smithi167 conmon[60316]: debug 2022-01-31T22:54:33.871+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.871779+0000) 2022-01-31T22:54:34.147 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:33 smithi167 conmon[60316]: debug 2022-01-31T22:54:33.919+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.920597+0000) 2022-01-31T22:54:34.148 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:33 smithi167 conmon[54076]: debug 2022-01-31T22:54:33.920+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:33.921543+0000) 2022-01-31T22:54:34.148 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:34 smithi167 conmon[54076]: debug 2022-01-31T22:54:34.112+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:34.112776+0000) 2022-01-31T22:54:34.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:34 smithi171 conmon[41853]: debug 2022-01-31T22:54:34.305+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:34.306647+0000) 2022-01-31T22:54:34.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:34 smithi171 conmon[46715]: debug 2022-01-31T22:54:34.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:34.357974+0000) 2022-01-31T22:54:34.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:34 smithi171 conmon[51620]: debug 2022-01-31T22:54:34.483+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:34.484682+0000) 2022-01-31T22:54:35.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:34 smithi167 conmon[49112]: debug 2022-01-31T22:54:34.943+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:34.944545+0000) 2022-01-31T22:54:35.150 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:35 smithi167 conmon[54076]: debug 2022-01-31T22:54:35.112+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:35.112919+0000) 2022-01-31T22:54:35.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:34 smithi167 conmon[60316]: debug 2022-01-31T22:54:34.870+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:34.872021+0000) 2022-01-31T22:54:35.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:35 smithi171 conmon[41853]: debug 2022-01-31T22:54:35.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:35.306824+0000) 2022-01-31T22:54:35.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:35 smithi171 conmon[46715]: debug 2022-01-31T22:54:35.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:35.358103+0000) 2022-01-31T22:54:35.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:35 smithi171 conmon[51620]: debug 2022-01-31T22:54:35.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:35.484796+0000) 2022-01-31T22:54:36.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:35 smithi167 conmon[49112]: debug 2022-01-31T22:54:35.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:35.944705+0000) 2022-01-31T22:54:36.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:36 smithi167 conmon[54076]: debug 2022-01-31T22:54:36.112+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:36.113048+0000) 2022-01-31T22:54:36.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:35 smithi167 conmon[60316]: debug 2022-01-31T22:54:35.871+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:35.872175+0000) 2022-01-31T22:54:36.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:36 smithi171 conmon[41853]: debug 2022-01-31T22:54:36.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:36.306994+0000) 2022-01-31T22:54:36.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:36 smithi171 conmon[46715]: debug 2022-01-31T22:54:36.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:36.358282+0000) 2022-01-31T22:54:36.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:36 smithi171 conmon[51620]: debug 2022-01-31T22:54:36.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:36.484980+0000) 2022-01-31T22:54:37.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:37 smithi167 conmon[54076]: debug 2022-01-31T22:54:37.112+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:37.113241+0000) 2022-01-31T22:54:37.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:36 smithi167 conmon[49112]: debug 2022-01-31T22:54:36.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:36.944877+0000) 2022-01-31T22:54:37.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:36 smithi167 conmon[60316]: debug 2022-01-31T22:54:36.871+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:36.872336+0000) 2022-01-31T22:54:37.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:37 smithi171 conmon[46715]: debug 2022-01-31T22:54:37.357+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:37.358470+0000) 2022-01-31T22:54:37.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:37 smithi171 conmon[41853]: debug 2022-01-31T22:54:37.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:37.307161+0000) 2022-01-31T22:54:37.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:37 smithi171 conmon[51620]: debug 2022-01-31T22:54:37.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:37.485117+0000) 2022-01-31T22:54:38.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:38 smithi167 conmon[54076]: debug 2022-01-31T22:54:38.112+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.113430+0000) 2022-01-31T22:54:38.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:37 smithi167 conmon[49112]: debug 2022-01-31T22:54:37.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:37.945061+0000) 2022-01-31T22:54:38.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:37 smithi167 conmon[60316]: debug 2022-01-31T22:54:37.871+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:37.872522+0000) 2022-01-31T22:54:38.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[41853]: debug 2022-01-31T22:54:38.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.307355+0000) 2022-01-31T22:54:38.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[46715]: debug 2022-01-31T22:54:38.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.358682+0000) 2022-01-31T22:54:38.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[51620]: debug 2022-01-31T22:54:38.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.485267+0000) 2022-01-31T22:54:39.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[41853]: debug 2022-01-31T22:54:38.948+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.949499+0000) 2022-01-31T22:54:39.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[46715]: debug 2022-01-31T22:54:38.949+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.950000+0000) 2022-01-31T22:54:39.094 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[35325]: debug 2022-01-31T22:54:38.929+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:54:39.095 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[35325]: debug 2022-01-31T22:54:38.973+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247340 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:39.095 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:38 smithi171 conmon[51620]: debug 2022-01-31T22:54:38.947+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.948343+0000) 2022-01-31T22:54:39.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:38 smithi167 conmon[49112]: debug 2022-01-31T22:54:38.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.945216+0000) 2022-01-31T22:54:39.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:38 smithi167 conmon[49112]: debug 2022-01-31T22:54:38.947+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.948127+0000) 2022-01-31T22:54:39.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:38 smithi167 conmon[54076]: debug 2022-01-31T22:54:38.948+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.949006+0000) 2022-01-31T22:54:39.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:39 smithi167 conmon[54076]: debug 2022-01-31T22:54:39.113+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:39.113577+0000) 2022-01-31T22:54:39.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:38 smithi167 conmon[60316]: debug 2022-01-31T22:54:38.872+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.872734+0000) 2022-01-31T22:54:39.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:38 smithi167 conmon[60316]: debug 2022-01-31T22:54:38.948+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:38.948583+0000) 2022-01-31T22:54:39.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:39 smithi171 conmon[41853]: debug 2022-01-31T22:54:39.306+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:39.307567+0000) 2022-01-31T22:54:39.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:39 smithi171 conmon[46715]: debug 2022-01-31T22:54:39.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:39.358828+0000) 2022-01-31T22:54:39.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:39 smithi171 conmon[51620]: debug 2022-01-31T22:54:39.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:39.485451+0000) 2022-01-31T22:54:40.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:40 smithi167 conmon[54076]: debug 2022-01-31T22:54:40.113+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:40.113723+0000) 2022-01-31T22:54:40.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:39 smithi167 conmon[49112]: debug 2022-01-31T22:54:39.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:39.945396+0000) 2022-01-31T22:54:40.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:39 smithi167 conmon[60316]: debug 2022-01-31T22:54:39.872+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:39.872915+0000) 2022-01-31T22:54:40.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:40 smithi171 conmon[41853]: debug 2022-01-31T22:54:40.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:40.307739+0000) 2022-01-31T22:54:40.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:40 smithi171 conmon[46715]: debug 2022-01-31T22:54:40.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:40.358956+0000) 2022-01-31T22:54:40.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:40 smithi171 conmon[51620]: debug 2022-01-31T22:54:40.484+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:40.485662+0000) 2022-01-31T22:54:41.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:41 smithi167 conmon[54076]: debug 2022-01-31T22:54:41.113+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:41.113924+0000) 2022-01-31T22:54:41.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:40 smithi167 conmon[49112]: debug 2022-01-31T22:54:40.944+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:40.945489+0000) 2022-01-31T22:54:41.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:40 smithi167 conmon[60316]: debug 2022-01-31T22:54:40.872+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:40.873095+0000) 2022-01-31T22:54:41.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:41 smithi171 conmon[41853]: debug 2022-01-31T22:54:41.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:41.307951+0000) 2022-01-31T22:54:41.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:41 smithi171 conmon[46715]: debug 2022-01-31T22:54:41.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:41.359139+0000) 2022-01-31T22:54:41.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:41 smithi171 conmon[51620]: debug 2022-01-31T22:54:41.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:41.485869+0000) 2022-01-31T22:54:42.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:41 smithi167 conmon[49112]: debug 2022-01-31T22:54:41.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:41.945648+0000) 2022-01-31T22:54:42.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:42 smithi167 conmon[54076]: debug 2022-01-31T22:54:42.113+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:42.114095+0000) 2022-01-31T22:54:42.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:41 smithi167 conmon[60316]: debug 2022-01-31T22:54:41.872+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:41.873309+0000) 2022-01-31T22:54:42.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:42 smithi171 conmon[41853]: debug 2022-01-31T22:54:42.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:42.308158+0000) 2022-01-31T22:54:42.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:42 smithi171 conmon[46715]: debug 2022-01-31T22:54:42.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:42.359294+0000) 2022-01-31T22:54:42.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:42 smithi171 conmon[51620]: debug 2022-01-31T22:54:42.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:42.486038+0000) 2022-01-31T22:54:43.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:42 smithi167 conmon[49112]: debug 2022-01-31T22:54:42.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:42.945780+0000) 2022-01-31T22:54:43.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:43 smithi167 conmon[54076]: debug 2022-01-31T22:54:43.113+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.114291+0000) 2022-01-31T22:54:43.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:42 smithi167 conmon[60316]: debug 2022-01-31T22:54:42.873+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:42.873531+0000) 2022-01-31T22:54:43.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:43 smithi171 conmon[41853]: debug 2022-01-31T22:54:43.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.308358+0000) 2022-01-31T22:54:43.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:43 smithi171 conmon[46715]: debug 2022-01-31T22:54:43.358+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.359424+0000) 2022-01-31T22:54:43.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:43 smithi171 conmon[51620]: debug 2022-01-31T22:54:43.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.486192+0000) 2022-01-31T22:54:44.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:44 smithi171 conmon[35325]: debug 2022-01-31T22:54:44.001+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247451 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:44.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:43 smithi171 conmon[41853]: debug 2022-01-31T22:54:43.976+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.977102+0000) 2022-01-31T22:54:44.094 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:43 smithi171 conmon[46715]: debug 2022-01-31T22:54:43.975+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.976184+0000) 2022-01-31T22:54:44.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:43 smithi171 conmon[51620]: debug 2022-01-31T22:54:43.975+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.976261+0000) 2022-01-31T22:54:44.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:43 smithi167 conmon[49112]: debug 2022-01-31T22:54:43.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.945997+0000) 2022-01-31T22:54:44.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:43 smithi167 conmon[49112]: debug 2022-01-31T22:54:43.976+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.977079+0000) 2022-01-31T22:54:44.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:43 smithi167 conmon[54076]: debug 2022-01-31T22:54:43.976+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.976711+0000) 2022-01-31T22:54:44.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:44 smithi167 conmon[54076]: debug 2022-01-31T22:54:44.113+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:44.114461+0000) 2022-01-31T22:54:44.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:43 smithi167 conmon[60316]: debug 2022-01-31T22:54:43.873+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.873700+0000) 2022-01-31T22:54:44.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:43 smithi167 conmon[60316]: debug 2022-01-31T22:54:43.977+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:43.978006+0000) 2022-01-31T22:54:44.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:44 smithi171 conmon[51620]: debug 2022-01-31T22:54:44.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:44.486372+0000) 2022-01-31T22:54:44.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:44 smithi171 conmon[41853]: debug 2022-01-31T22:54:44.307+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:44.308518+0000) 2022-01-31T22:54:44.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:44 smithi171 conmon[46715]: debug 2022-01-31T22:54:44.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:44.359602+0000) 2022-01-31T22:54:45.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:44 smithi167 conmon[60316]: debug 2022-01-31T22:54:44.873+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:44.873931+0000) 2022-01-31T22:54:45.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:44 smithi167 conmon[49112]: debug 2022-01-31T22:54:44.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:44.946181+0000) 2022-01-31T22:54:45.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:45 smithi167 conmon[54076]: debug 2022-01-31T22:54:45.114+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:45.114606+0000) 2022-01-31T22:54:45.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:45 smithi171 conmon[41853]: debug 2022-01-31T22:54:45.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:45.308699+0000) 2022-01-31T22:54:45.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:45 smithi171 conmon[46715]: debug 2022-01-31T22:54:45.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:45.359748+0000) 2022-01-31T22:54:45.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:45 smithi171 conmon[51620]: debug 2022-01-31T22:54:45.485+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:45.486542+0000) 2022-01-31T22:54:46.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:45 smithi167 conmon[49112]: debug 2022-01-31T22:54:45.945+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:45.946362+0000) 2022-01-31T22:54:46.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:46 smithi167 conmon[54076]: debug 2022-01-31T22:54:46.114+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:46.114786+0000) 2022-01-31T22:54:46.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:45 smithi167 conmon[60316]: debug 2022-01-31T22:54:45.873+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:45.874085+0000) 2022-01-31T22:54:46.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:46 smithi171 conmon[41853]: debug 2022-01-31T22:54:46.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:46.308877+0000) 2022-01-31T22:54:46.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:46 smithi171 conmon[46715]: debug 2022-01-31T22:54:46.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:46.359905+0000) 2022-01-31T22:54:46.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:46 smithi171 conmon[51620]: debug 2022-01-31T22:54:46.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:46.486727+0000) 2022-01-31T22:54:47.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:46 smithi167 conmon[49112]: debug 2022-01-31T22:54:46.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:46.946557+0000) 2022-01-31T22:54:47.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:47 smithi167 conmon[54076]: debug 2022-01-31T22:54:47.114+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:47.114972+0000) 2022-01-31T22:54:47.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:46 smithi167 conmon[60316]: debug 2022-01-31T22:54:46.873+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:46.874274+0000) 2022-01-31T22:54:47.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:47 smithi171 conmon[41853]: debug 2022-01-31T22:54:47.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:47.309043+0000) 2022-01-31T22:54:47.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:47 smithi171 conmon[46715]: debug 2022-01-31T22:54:47.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:47.360105+0000) 2022-01-31T22:54:47.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:47 smithi171 conmon[51620]: debug 2022-01-31T22:54:47.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:47.486890+0000) 2022-01-31T22:54:48.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:47 smithi167 conmon[49112]: debug 2022-01-31T22:54:47.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:47.946713+0000) 2022-01-31T22:54:48.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:48 smithi167 conmon[54076]: debug 2022-01-31T22:54:48.114+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:48.115156+0000) 2022-01-31T22:54:48.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:47 smithi167 conmon[60316]: debug 2022-01-31T22:54:47.873+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:47.874479+0000) 2022-01-31T22:54:48.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:48 smithi171 conmon[41853]: debug 2022-01-31T22:54:48.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:48.309233+0000) 2022-01-31T22:54:48.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:48 smithi171 conmon[46715]: debug 2022-01-31T22:54:48.359+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:48.360265+0000) 2022-01-31T22:54:48.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:48 smithi171 conmon[51620]: debug 2022-01-31T22:54:48.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:48.487013+0000) 2022-01-31T22:54:49.015 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[41853]: debug 2022-01-31T22:54:49.003+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.004097+0000) 2022-01-31T22:54:49.015 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[46715]: debug 2022-01-31T22:54:49.003+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.004478+0000) 2022-01-31T22:54:49.016 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[51620]: debug 2022-01-31T22:54:49.004+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.004742+0000) 2022-01-31T22:54:49.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:48 smithi167 conmon[49112]: debug 2022-01-31T22:54:48.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:48.946924+0000) 2022-01-31T22:54:49.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:49 smithi167 conmon[49112]: debug 2022-01-31T22:54:49.005+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.005563+0000) 2022-01-31T22:54:49.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:49 smithi167 conmon[54076]: debug 2022-01-31T22:54:49.004+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.004562+0000) 2022-01-31T22:54:49.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:49 smithi167 conmon[54076]: debug 2022-01-31T22:54:49.114+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.115341+0000) 2022-01-31T22:54:49.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:48 smithi167 conmon[60316]: debug 2022-01-31T22:54:48.874+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:48.874648+0000) 2022-01-31T22:54:49.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:49 smithi167 conmon[60316]: debug 2022-01-31T22:54:49.005+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.005541+0000) 2022-01-31T22:54:49.295 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[35325]: debug 2022-01-31T22:54:49.028+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247565 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:49.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[46715]: debug 2022-01-31T22:54:49.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.360468+0000) 2022-01-31T22:54:49.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[41853]: debug 2022-01-31T22:54:49.308+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.309423+0000) 2022-01-31T22:54:49.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:49 smithi171 conmon[51620]: debug 2022-01-31T22:54:49.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.487123+0000) 2022-01-31T22:54:50.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:49 smithi167 conmon[49112]: debug 2022-01-31T22:54:49.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.947107+0000) 2022-01-31T22:54:50.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:50 smithi167 conmon[54076]: debug 2022-01-31T22:54:50.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:50.115482+0000) 2022-01-31T22:54:50.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:49 smithi167 conmon[60316]: debug 2022-01-31T22:54:49.874+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:49.874863+0000) 2022-01-31T22:54:50.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:50 smithi171 conmon[46715]: debug 2022-01-31T22:54:50.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:50.360618+0000) 2022-01-31T22:54:50.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:50 smithi171 conmon[51620]: debug 2022-01-31T22:54:50.486+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:50.487341+0000) 2022-01-31T22:54:50.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:50 smithi171 conmon[41853]: debug 2022-01-31T22:54:50.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:50.309567+0000) 2022-01-31T22:54:51.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:51 smithi167 conmon[54076]: debug 2022-01-31T22:54:51.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:51.115665+0000) 2022-01-31T22:54:51.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:50 smithi167 conmon[49112]: debug 2022-01-31T22:54:50.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:50.947269+0000) 2022-01-31T22:54:51.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:50 smithi167 conmon[60316]: debug 2022-01-31T22:54:50.874+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:50.875055+0000) 2022-01-31T22:54:51.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:51 smithi171 conmon[41853]: debug 2022-01-31T22:54:51.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:51.309768+0000) 2022-01-31T22:54:51.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:51 smithi171 conmon[46715]: debug 2022-01-31T22:54:51.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:51.360809+0000) 2022-01-31T22:54:51.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:51 smithi171 conmon[51620]: debug 2022-01-31T22:54:51.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:51.488050+0000) 2022-01-31T22:54:52.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:51 smithi167 conmon[49112]: debug 2022-01-31T22:54:51.946+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:51.947422+0000) 2022-01-31T22:54:52.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:52 smithi167 conmon[54076]: debug 2022-01-31T22:54:52.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:52.115870+0000) 2022-01-31T22:54:52.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:51 smithi167 conmon[60316]: debug 2022-01-31T22:54:51.874+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:51.875241+0000) 2022-01-31T22:54:52.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:52 smithi171 conmon[41853]: debug 2022-01-31T22:54:52.309+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:52.309925+0000) 2022-01-31T22:54:52.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:52 smithi171 conmon[46715]: debug 2022-01-31T22:54:52.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:52.360970+0000) 2022-01-31T22:54:52.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:52 smithi171 conmon[51620]: debug 2022-01-31T22:54:52.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:52.488171+0000) 2022-01-31T22:54:53.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:52 smithi167 conmon[49112]: debug 2022-01-31T22:54:52.947+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:52.947615+0000) 2022-01-31T22:54:53.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:53 smithi167 conmon[54076]: debug 2022-01-31T22:54:53.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:53.116048+0000) 2022-01-31T22:54:53.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:52 smithi167 conmon[60316]: debug 2022-01-31T22:54:52.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:52.875401+0000) 2022-01-31T22:54:53.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:53 smithi171 conmon[41853]: debug 2022-01-31T22:54:53.312+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:53.313037+0000) 2022-01-31T22:54:53.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:53 smithi171 conmon[46715]: debug 2022-01-31T22:54:53.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:53.361165+0000) 2022-01-31T22:54:53.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:53 smithi171 conmon[51620]: debug 2022-01-31T22:54:53.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:53.488271+0000) 2022-01-31T22:54:54.017 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:53 smithi171 conmon[35325]: debug 2022-01-31T22:54:53.930+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:54:54.018 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[41853]: debug 2022-01-31T22:54:54.030+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.031490+0000) 2022-01-31T22:54:54.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:53 smithi167 conmon[49112]: debug 2022-01-31T22:54:53.947+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:53.947771+0000) 2022-01-31T22:54:54.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:54 smithi167 conmon[49112]: debug 2022-01-31T22:54:54.034+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.034874+0000) 2022-01-31T22:54:54.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:54 smithi167 conmon[54076]: debug 2022-01-31T22:54:54.032+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.032967+0000) 2022-01-31T22:54:54.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:54 smithi167 conmon[54076]: debug 2022-01-31T22:54:54.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.116219+0000) 2022-01-31T22:54:54.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:53 smithi167 conmon[60316]: debug 2022-01-31T22:54:53.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:53.875581+0000) 2022-01-31T22:54:54.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:54 smithi167 conmon[60316]: debug 2022-01-31T22:54:54.032+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.032691+0000) 2022-01-31T22:54:54.299 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[35325]: debug 2022-01-31T22:54:54.057+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247678 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:54.299 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[46715]: debug 2022-01-31T22:54:54.033+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.033989+0000) 2022-01-31T22:54:54.300 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[51620]: debug 2022-01-31T22:54:54.031+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.031772+0000) 2022-01-31T22:54:54.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[51620]: debug 2022-01-31T22:54:54.487+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.488367+0000) 2022-01-31T22:54:54.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[41853]: debug 2022-01-31T22:54:54.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.313818+0000) 2022-01-31T22:54:54.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:54 smithi171 conmon[46715]: debug 2022-01-31T22:54:54.360+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.361369+0000) 2022-01-31T22:54:55.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:54 smithi167 conmon[49112]: debug 2022-01-31T22:54:54.947+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.947942+0000) 2022-01-31T22:54:55.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:55 smithi167 conmon[54076]: debug 2022-01-31T22:54:55.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:55.116326+0000) 2022-01-31T22:54:55.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:54 smithi167 conmon[60316]: debug 2022-01-31T22:54:54.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:54.875770+0000) 2022-01-31T22:54:55.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:55 smithi171 conmon[51620]: debug 2022-01-31T22:54:55.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:55.488484+0000) 2022-01-31T22:54:55.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:55 smithi171 conmon[41853]: debug 2022-01-31T22:54:55.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:55.313946+0000) 2022-01-31T22:54:55.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:55 smithi171 conmon[46715]: debug 2022-01-31T22:54:55.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:55.361543+0000) 2022-01-31T22:54:56.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:55 smithi167 conmon[49112]: debug 2022-01-31T22:54:55.947+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:55.948088+0000) 2022-01-31T22:54:56.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:56 smithi167 conmon[54076]: debug 2022-01-31T22:54:56.115+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:56.116479+0000) 2022-01-31T22:54:56.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:55 smithi167 conmon[60316]: debug 2022-01-31T22:54:55.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:55.875924+0000) 2022-01-31T22:54:56.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:56 smithi171 conmon[51620]: debug 2022-01-31T22:54:56.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:56.488608+0000) 2022-01-31T22:54:56.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:56 smithi171 conmon[41853]: debug 2022-01-31T22:54:56.313+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:56.314085+0000) 2022-01-31T22:54:56.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:56 smithi171 conmon[46715]: debug 2022-01-31T22:54:56.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:56.361703+0000) 2022-01-31T22:54:57.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:56 smithi167 conmon[49112]: debug 2022-01-31T22:54:56.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:56.948271+0000) 2022-01-31T22:54:57.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:57 smithi167 conmon[54076]: debug 2022-01-31T22:54:57.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:57.116624+0000) 2022-01-31T22:54:57.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:56 smithi167 conmon[60316]: debug 2022-01-31T22:54:56.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:56.876127+0000) 2022-01-31T22:54:57.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:57 smithi171 conmon[41853]: debug 2022-01-31T22:54:57.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:57.314289+0000) 2022-01-31T22:54:57.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:57 smithi171 conmon[46715]: debug 2022-01-31T22:54:57.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:57.361880+0000) 2022-01-31T22:54:57.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:57 smithi171 conmon[51620]: debug 2022-01-31T22:54:57.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:57.488776+0000) 2022-01-31T22:54:58.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:58 smithi167 conmon[54076]: debug 2022-01-31T22:54:58.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:58.116729+0000) 2022-01-31T22:54:58.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:57 smithi167 conmon[49112]: debug 2022-01-31T22:54:57.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:57.948450+0000) 2022-01-31T22:54:58.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:57 smithi167 conmon[60316]: debug 2022-01-31T22:54:57.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:57.876291+0000) 2022-01-31T22:54:58.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:58 smithi171 conmon[41853]: debug 2022-01-31T22:54:58.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:58.314459+0000) 2022-01-31T22:54:58.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:58 smithi171 conmon[46715]: debug 2022-01-31T22:54:58.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:58.362005+0000) 2022-01-31T22:54:58.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:58 smithi171 conmon[51620]: debug 2022-01-31T22:54:58.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:58.488990+0000) 2022-01-31T22:54:59.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:58 smithi167 conmon[49112]: debug 2022-01-31T22:54:58.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:58.948652+0000) 2022-01-31T22:54:59.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:59 smithi167 conmon[49112]: debug 2022-01-31T22:54:59.060+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.061241+0000) 2022-01-31T22:54:59.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:58 smithi167 conmon[60316]: debug 2022-01-31T22:54:58.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:58.876440+0000) 2022-01-31T22:54:59.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:59 smithi167 conmon[60316]: debug 2022-01-31T22:54:59.060+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.060633+0000) 2022-01-31T22:54:59.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:59 smithi167 conmon[54076]: debug 2022-01-31T22:54:59.060+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.060510+0000) 2022-01-31T22:54:59.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:54:59 smithi167 conmon[54076]: debug 2022-01-31T22:54:59.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.116889+0000) 2022-01-31T22:54:59.300 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[41853]: debug 2022-01-31T22:54:59.058+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.059726+0000) 2022-01-31T22:54:59.300 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[46715]: debug 2022-01-31T22:54:59.060+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.061591+0000) 2022-01-31T22:54:59.301 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[51620]: debug 2022-01-31T22:54:59.059+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.060509+0000) 2022-01-31T22:54:59.301 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[35325]: debug 2022-01-31T22:54:59.084+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247790 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:54:59.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[41853]: debug 2022-01-31T22:54:59.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.314660+0000) 2022-01-31T22:54:59.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[46715]: debug 2022-01-31T22:54:59.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.362221+0000) 2022-01-31T22:54:59.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:54:59 smithi171 conmon[51620]: debug 2022-01-31T22:54:59.488+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.489162+0000) 2022-01-31T22:55:00.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:54:59 smithi167 conmon[49112]: debug 2022-01-31T22:54:59.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.948823+0000) 2022-01-31T22:55:00.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:00 smithi167 conmon[54076]: debug 2022-01-31T22:55:00.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:00.117062+0000) 2022-01-31T22:55:00.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:54:59 smithi167 conmon[60316]: debug 2022-01-31T22:54:59.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:54:59.876684+0000) 2022-01-31T22:55:00.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:00 smithi171 conmon[41853]: debug 2022-01-31T22:55:00.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:00.314812+0000) 2022-01-31T22:55:00.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:00 smithi171 conmon[46715]: debug 2022-01-31T22:55:00.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:00.362354+0000) 2022-01-31T22:55:00.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:00 smithi171 conmon[51620]: debug 2022-01-31T22:55:00.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:00.489327+0000) 2022-01-31T22:55:01.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:00 smithi167 conmon[49112]: debug 2022-01-31T22:55:00.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:00.948987+0000) 2022-01-31T22:55:01.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:01 smithi167 conmon[54076]: debug 2022-01-31T22:55:01.117+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:01.117278+0000) 2022-01-31T22:55:01.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:00 smithi167 conmon[60316]: debug 2022-01-31T22:55:00.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:00.876839+0000) 2022-01-31T22:55:01.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:01 smithi171 conmon[46715]: debug 2022-01-31T22:55:01.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:01.362512+0000) 2022-01-31T22:55:01.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:01 smithi171 conmon[41853]: debug 2022-01-31T22:55:01.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:01.315014+0000) 2022-01-31T22:55:01.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:01 smithi171 conmon[51620]: debug 2022-01-31T22:55:01.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:01.489541+0000) 2022-01-31T22:55:02.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:02 smithi167 conmon[54076]: debug 2022-01-31T22:55:02.117+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:02.117484+0000) 2022-01-31T22:55:02.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:01 smithi167 conmon[49112]: debug 2022-01-31T22:55:01.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:01.949189+0000) 2022-01-31T22:55:02.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:01 smithi167 conmon[60316]: debug 2022-01-31T22:55:01.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:01.876994+0000) 2022-01-31T22:55:02.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:02 smithi171 conmon[41853]: debug 2022-01-31T22:55:02.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:02.315212+0000) 2022-01-31T22:55:02.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:02 smithi171 conmon[46715]: debug 2022-01-31T22:55:02.361+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:02.362670+0000) 2022-01-31T22:55:02.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:02 smithi171 conmon[51620]: debug 2022-01-31T22:55:02.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:02.489747+0000) 2022-01-31T22:55:03.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:02 smithi167 conmon[49112]: debug 2022-01-31T22:55:02.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:02.949397+0000) 2022-01-31T22:55:03.442 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:03 smithi167 conmon[54076]: debug 2022-01-31T22:55:03.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:03.117666+0000) 2022-01-31T22:55:03.443 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:02 smithi167 conmon[60316]: debug 2022-01-31T22:55:02.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:02.877199+0000) 2022-01-31T22:55:03.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:03 smithi171 conmon[41853]: debug 2022-01-31T22:55:03.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:03.315346+0000) 2022-01-31T22:55:03.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:03 smithi171 conmon[46715]: debug 2022-01-31T22:55:03.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:03.362810+0000) 2022-01-31T22:55:03.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:03 smithi171 conmon[51620]: debug 2022-01-31T22:55:03.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:03.489981+0000) 2022-01-31T22:55:04.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:04 smithi167 conmon[54076]: debug 2022-01-31T22:55:04.086+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.087678+0000) 2022-01-31T22:55:04.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:04 smithi167 conmon[54076]: debug 2022-01-31T22:55:04.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.117777+0000) 2022-01-31T22:55:04.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:03 smithi167 conmon[49112]: debug 2022-01-31T22:55:03.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:03.949600+0000) 2022-01-31T22:55:04.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:04 smithi167 conmon[49112]: debug 2022-01-31T22:55:04.087+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.088897+0000) 2022-01-31T22:55:04.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:03 smithi167 conmon[60316]: debug 2022-01-31T22:55:03.875+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:03.877379+0000) 2022-01-31T22:55:04.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:04 smithi167 conmon[60316]: debug 2022-01-31T22:55:04.086+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.087954+0000) 2022-01-31T22:55:04.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[35325]: debug 2022-01-31T22:55:04.112+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 247901 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:04.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[46715]: debug 2022-01-31T22:55:04.088+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.089101+0000) 2022-01-31T22:55:04.344 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[51620]: debug 2022-01-31T22:55:04.087+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.088006+0000) 2022-01-31T22:55:04.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[41853]: debug 2022-01-31T22:55:04.086+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.087449+0000) 2022-01-31T22:55:04.345 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[41853]: debug 2022-01-31T22:55:04.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.315520+0000) 2022-01-31T22:55:04.763 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[46715]: debug 2022-01-31T22:55:04.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.362932+0000) 2022-01-31T22:55:04.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:04 smithi171 conmon[51620]: debug 2022-01-31T22:55:04.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.490177+0000) 2022-01-31T22:55:05.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:05 smithi167 conmon[54076]: debug 2022-01-31T22:55:05.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:05.117908+0000) 2022-01-31T22:55:05.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:04 smithi167 conmon[49112]: debug 2022-01-31T22:55:04.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.949758+0000) 2022-01-31T22:55:05.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:04 smithi167 conmon[60316]: debug 2022-01-31T22:55:04.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:04.877554+0000) 2022-01-31T22:55:05.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:05 smithi171 conmon[46715]: debug 2022-01-31T22:55:05.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:05.363133+0000) 2022-01-31T22:55:05.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:05 smithi171 conmon[41853]: debug 2022-01-31T22:55:05.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:05.315672+0000) 2022-01-31T22:55:05.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:05 smithi171 conmon[51620]: debug 2022-01-31T22:55:05.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:05.490342+0000) 2022-01-31T22:55:06.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:05 smithi167 conmon[49112]: debug 2022-01-31T22:55:05.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:05.949874+0000) 2022-01-31T22:55:06.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:06 smithi167 conmon[54076]: debug 2022-01-31T22:55:06.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:06.118084+0000) 2022-01-31T22:55:06.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:05 smithi167 conmon[60316]: debug 2022-01-31T22:55:05.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:05.877692+0000) 2022-01-31T22:55:06.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:06 smithi171 conmon[46715]: debug 2022-01-31T22:55:06.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:06.363332+0000) 2022-01-31T22:55:06.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:06 smithi171 conmon[41853]: debug 2022-01-31T22:55:06.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:06.315810+0000) 2022-01-31T22:55:06.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:06 smithi171 conmon[51620]: debug 2022-01-31T22:55:06.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:06.490552+0000) 2022-01-31T22:55:07.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:07 smithi167 conmon[54076]: debug 2022-01-31T22:55:07.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:07.118251+0000) 2022-01-31T22:55:07.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:06 smithi167 conmon[49112]: debug 2022-01-31T22:55:06.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:06.950056+0000) 2022-01-31T22:55:07.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:06 smithi167 conmon[60316]: debug 2022-01-31T22:55:06.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:06.877858+0000) 2022-01-31T22:55:07.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:07 smithi171 conmon[41853]: debug 2022-01-31T22:55:07.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:07.315970+0000) 2022-01-31T22:55:07.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:07 smithi171 conmon[46715]: debug 2022-01-31T22:55:07.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:07.363497+0000) 2022-01-31T22:55:07.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:07 smithi171 conmon[51620]: debug 2022-01-31T22:55:07.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:07.490782+0000) 2022-01-31T22:55:08.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:07 smithi167 conmon[49112]: debug 2022-01-31T22:55:07.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:07.950271+0000) 2022-01-31T22:55:08.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:08 smithi167 conmon[54076]: debug 2022-01-31T22:55:08.116+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:08.118412+0000) 2022-01-31T22:55:08.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:07 smithi167 conmon[60316]: debug 2022-01-31T22:55:07.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:07.878032+0000) 2022-01-31T22:55:08.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:08 smithi171 conmon[41853]: debug 2022-01-31T22:55:08.314+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:08.316112+0000) 2022-01-31T22:55:08.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:08 smithi171 conmon[51620]: debug 2022-01-31T22:55:08.489+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:08.490995+0000) 2022-01-31T22:55:08.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:08 smithi171 conmon[46715]: debug 2022-01-31T22:55:08.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:08.363664+0000) 2022-01-31T22:55:09.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:08 smithi171 conmon[35325]: debug 2022-01-31T22:55:08.930+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:55:09.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:09 smithi167 conmon[54076]: debug 2022-01-31T22:55:09.115+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.116869+0000) 2022-01-31T22:55:09.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:09 smithi167 conmon[54076]: debug 2022-01-31T22:55:09.117+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.118522+0000) 2022-01-31T22:55:09.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:08 smithi167 conmon[49112]: debug 2022-01-31T22:55:08.948+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:08.950425+0000) 2022-01-31T22:55:09.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:09 smithi167 conmon[49112]: debug 2022-01-31T22:55:09.115+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.116770+0000) 2022-01-31T22:55:09.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:08 smithi167 conmon[60316]: debug 2022-01-31T22:55:08.876+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:08.878162+0000) 2022-01-31T22:55:09.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:09 smithi167 conmon[60316]: debug 2022-01-31T22:55:09.114+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.115836+0000) 2022-01-31T22:55:09.476 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[35325]: debug 2022-01-31T22:55:09.139+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248013 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:09.477 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[51620]: debug 2022-01-31T22:55:09.114+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.116622+0000) 2022-01-31T22:55:09.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[41853]: debug 2022-01-31T22:55:09.114+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.116218+0000) 2022-01-31T22:55:09.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[41853]: debug 2022-01-31T22:55:09.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.316352+0000) 2022-01-31T22:55:09.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[46715]: debug 2022-01-31T22:55:09.115+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.117482+0000) 2022-01-31T22:55:09.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[46715]: debug 2022-01-31T22:55:09.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.363865+0000) 2022-01-31T22:55:09.764 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:09 smithi171 conmon[51620]: debug 2022-01-31T22:55:09.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.491238+0000) 2022-01-31T22:55:10.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:10 smithi167 conmon[54076]: debug 2022-01-31T22:55:10.117+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:10.118698+0000) 2022-01-31T22:55:10.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:09 smithi167 conmon[49112]: debug 2022-01-31T22:55:09.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.950638+0000) 2022-01-31T22:55:10.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:09 smithi167 conmon[60316]: debug 2022-01-31T22:55:09.877+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:09.878336+0000) 2022-01-31T22:55:10.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:10 smithi171 conmon[41853]: debug 2022-01-31T22:55:10.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:10.316502+0000) 2022-01-31T22:55:10.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:10 smithi171 conmon[46715]: debug 2022-01-31T22:55:10.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:10.364013+0000) 2022-01-31T22:55:10.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:10 smithi171 conmon[51620]: debug 2022-01-31T22:55:10.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:10.491413+0000) 2022-01-31T22:55:11.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:10 smithi167 conmon[49112]: debug 2022-01-31T22:55:10.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:10.950800+0000) 2022-01-31T22:55:11.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:10 smithi167 conmon[60316]: debug 2022-01-31T22:55:10.877+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:10.878496+0000) 2022-01-31T22:55:11.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:11 smithi167 conmon[54076]: debug 2022-01-31T22:55:11.118+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:11.118896+0000) 2022-01-31T22:55:11.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:11 smithi171 conmon[41853]: debug 2022-01-31T22:55:11.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:11.316663+0000) 2022-01-31T22:55:11.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:11 smithi171 conmon[46715]: debug 2022-01-31T22:55:11.363+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:11.364179+0000) 2022-01-31T22:55:11.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:11 smithi171 conmon[51620]: debug 2022-01-31T22:55:11.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:11.491598+0000) 2022-01-31T22:55:12.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:12 smithi167 conmon[54076]: debug 2022-01-31T22:55:12.118+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:12.119108+0000) 2022-01-31T22:55:12.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:11 smithi167 conmon[49112]: debug 2022-01-31T22:55:11.949+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:11.951000+0000) 2022-01-31T22:55:12.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:11 smithi167 conmon[60316]: debug 2022-01-31T22:55:11.877+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:11.878698+0000) 2022-01-31T22:55:12.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:12 smithi171 conmon[41853]: debug 2022-01-31T22:55:12.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:12.316818+0000) 2022-01-31T22:55:12.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:12 smithi171 conmon[51620]: debug 2022-01-31T22:55:12.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:12.491815+0000) 2022-01-31T22:55:12.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:12 smithi171 conmon[46715]: debug 2022-01-31T22:55:12.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:12.364373+0000) 2022-01-31T22:55:13.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:12 smithi167 conmon[49112]: debug 2022-01-31T22:55:12.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:12.951185+0000) 2022-01-31T22:55:13.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:13 smithi167 conmon[54076]: debug 2022-01-31T22:55:13.118+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:13.119299+0000) 2022-01-31T22:55:13.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:12 smithi167 conmon[60316]: debug 2022-01-31T22:55:12.878+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:12.878886+0000) 2022-01-31T22:55:13.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:13 smithi171 conmon[41853]: debug 2022-01-31T22:55:13.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:13.317020+0000) 2022-01-31T22:55:13.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:13 smithi171 conmon[46715]: debug 2022-01-31T22:55:13.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:13.364554+0000) 2022-01-31T22:55:13.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:13 smithi171 conmon[51620]: debug 2022-01-31T22:55:13.490+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:13.492015+0000) 2022-01-31T22:55:14.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:13 smithi167 conmon[49112]: debug 2022-01-31T22:55:13.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:13.951367+0000) 2022-01-31T22:55:14.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:13 smithi167 conmon[60316]: debug 2022-01-31T22:55:13.877+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:13.879089+0000) 2022-01-31T22:55:14.131 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:14 smithi167 conmon[54076]: debug 2022-01-31T22:55:14.118+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.119485+0000) 2022-01-31T22:55:14.404 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:14 smithi167 conmon[49112]: debug 2022-01-31T22:55:14.143+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.144943+0000) 2022-01-31T22:55:14.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:14 smithi167 conmon[54076]: debug 2022-01-31T22:55:14.143+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.145008+0000) 2022-01-31T22:55:14.406 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:14 smithi167 conmon[60316]: debug 2022-01-31T22:55:14.142+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.144401+0000) 2022-01-31T22:55:14.477 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[35325]: debug 2022-01-31T22:55:14.167+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248125 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:14.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[41853]: debug 2022-01-31T22:55:14.141+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.143503+0000) 2022-01-31T22:55:14.478 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[41853]: debug 2022-01-31T22:55:14.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.317169+0000) 2022-01-31T22:55:14.478 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[46715]: debug 2022-01-31T22:55:14.141+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.143385+0000) 2022-01-31T22:55:14.479 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[46715]: debug 2022-01-31T22:55:14.362+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.364703+0000) 2022-01-31T22:55:14.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[51620]: debug 2022-01-31T22:55:14.142+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.143837+0000) 2022-01-31T22:55:14.480 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:14 smithi171 conmon[51620]: debug 2022-01-31T22:55:14.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.492226+0000) 2022-01-31T22:55:15.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:15 smithi167 conmon[54076]: debug 2022-01-31T22:55:15.118+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:15.119708+0000) 2022-01-31T22:55:15.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:14 smithi167 conmon[49112]: debug 2022-01-31T22:55:14.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.951543+0000) 2022-01-31T22:55:15.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:14 smithi167 conmon[60316]: debug 2022-01-31T22:55:14.878+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:14.879302+0000) 2022-01-31T22:55:15.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:15 smithi171 conmon[41853]: debug 2022-01-31T22:55:15.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:15.317274+0000) 2022-01-31T22:55:15.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:15 smithi171 conmon[46715]: debug 2022-01-31T22:55:15.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:15.364881+0000) 2022-01-31T22:55:15.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:15 smithi171 conmon[51620]: debug 2022-01-31T22:55:15.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:15.492391+0000) 2022-01-31T22:55:16.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:15 smithi167 conmon[49112]: debug 2022-01-31T22:55:15.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:15.951680+0000) 2022-01-31T22:55:16.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:16 smithi167 conmon[54076]: debug 2022-01-31T22:55:16.119+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:16.119861+0000) 2022-01-31T22:55:16.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:15 smithi167 conmon[60316]: debug 2022-01-31T22:55:15.877+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:15.879417+0000) 2022-01-31T22:55:16.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:16 smithi171 conmon[46715]: debug 2022-01-31T22:55:16.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:16.365088+0000) 2022-01-31T22:55:16.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:16 smithi171 conmon[41853]: debug 2022-01-31T22:55:16.315+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:16.317434+0000) 2022-01-31T22:55:16.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:16 smithi171 conmon[51620]: debug 2022-01-31T22:55:16.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:16.492602+0000) 2022-01-31T22:55:17.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:16 smithi167 conmon[49112]: debug 2022-01-31T22:55:16.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:16.951778+0000) 2022-01-31T22:55:17.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:17 smithi167 conmon[54076]: debug 2022-01-31T22:55:17.119+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:17.120065+0000) 2022-01-31T22:55:17.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:16 smithi167 conmon[60316]: debug 2022-01-31T22:55:16.878+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:16.879598+0000) 2022-01-31T22:55:17.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:17 smithi171 conmon[41853]: debug 2022-01-31T22:55:17.316+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:17.317618+0000) 2022-01-31T22:55:17.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:17 smithi171 conmon[46715]: debug 2022-01-31T22:55:17.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:17.365281+0000) 2022-01-31T22:55:17.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:17 smithi171 conmon[51620]: debug 2022-01-31T22:55:17.491+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:17.492803+0000) 2022-01-31T22:55:18.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:18 smithi167 conmon[54076]: debug 2022-01-31T22:55:18.119+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:18.120261+0000) 2022-01-31T22:55:18.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:17 smithi167 conmon[49112]: debug 2022-01-31T22:55:17.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:17.951959+0000) 2022-01-31T22:55:18.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:17 smithi167 conmon[60316]: debug 2022-01-31T22:55:17.879+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:17.879818+0000) 2022-01-31T22:55:18.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:18 smithi171 conmon[41853]: debug 2022-01-31T22:55:18.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:18.317823+0000) 2022-01-31T22:55:18.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:18 smithi171 conmon[46715]: debug 2022-01-31T22:55:18.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:18.365518+0000) 2022-01-31T22:55:18.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:18 smithi171 conmon[51620]: debug 2022-01-31T22:55:18.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:18.492989+0000) 2022-01-31T22:55:19.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:19 smithi167 conmon[54076]: debug 2022-01-31T22:55:19.119+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.120457+0000) 2022-01-31T22:55:19.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:18 smithi167 conmon[49112]: debug 2022-01-31T22:55:18.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:18.952108+0000) 2022-01-31T22:55:19.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:18 smithi167 conmon[60316]: debug 2022-01-31T22:55:18.878+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:18.880019+0000) 2022-01-31T22:55:19.478 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[35325]: debug 2022-01-31T22:55:19.197+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248238 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:19.479 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[51620]: debug 2022-01-31T22:55:19.171+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.172382+0000) 2022-01-31T22:55:19.479 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[41853]: debug 2022-01-31T22:55:19.171+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.172601+0000) 2022-01-31T22:55:19.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[41853]: debug 2022-01-31T22:55:19.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.318001+0000) 2022-01-31T22:55:19.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[46715]: debug 2022-01-31T22:55:19.170+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.171164+0000) 2022-01-31T22:55:19.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[46715]: debug 2022-01-31T22:55:19.364+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.365713+0000) 2022-01-31T22:55:19.602 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:19 smithi167 conmon[49112]: debug 2022-01-31T22:55:19.170+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.172182+0000) 2022-01-31T22:55:19.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:19 smithi167 conmon[54076]: debug 2022-01-31T22:55:19.171+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.173401+0000) 2022-01-31T22:55:19.603 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:19 smithi167 conmon[60316]: debug 2022-01-31T22:55:19.170+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.172311+0000) 2022-01-31T22:55:19.765 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:19 smithi171 conmon[51620]: debug 2022-01-31T22:55:19.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.493214+0000) 2022-01-31T22:55:20.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:19 smithi167 conmon[49112]: debug 2022-01-31T22:55:19.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.952245+0000) 2022-01-31T22:55:20.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:20 smithi167 conmon[54076]: debug 2022-01-31T22:55:20.119+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:20.120606+0000) 2022-01-31T22:55:20.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:19 smithi167 conmon[60316]: debug 2022-01-31T22:55:19.879+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:19.880214+0000) 2022-01-31T22:55:20.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:20 smithi171 conmon[51620]: debug 2022-01-31T22:55:20.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:20.493392+0000) 2022-01-31T22:55:20.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:20 smithi171 conmon[41853]: debug 2022-01-31T22:55:20.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:20.318167+0000) 2022-01-31T22:55:20.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:20 smithi171 conmon[46715]: debug 2022-01-31T22:55:20.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:20.365900+0000) 2022-01-31T22:55:21.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:21 smithi167 conmon[54076]: debug 2022-01-31T22:55:21.119+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:21.120755+0000) 2022-01-31T22:55:21.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:20 smithi167 conmon[60316]: debug 2022-01-31T22:55:20.879+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:20.880366+0000) 2022-01-31T22:55:21.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:20 smithi167 conmon[49112]: debug 2022-01-31T22:55:20.950+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:20.952414+0000) 2022-01-31T22:55:21.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:21 smithi171 conmon[41853]: debug 2022-01-31T22:55:21.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:21.318325+0000) 2022-01-31T22:55:21.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:21 smithi171 conmon[46715]: debug 2022-01-31T22:55:21.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:21.366019+0000) 2022-01-31T22:55:21.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:21 smithi171 conmon[51620]: debug 2022-01-31T22:55:21.492+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:21.493612+0000) 2022-01-31T22:55:22.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:21 smithi167 conmon[49112]: debug 2022-01-31T22:55:21.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:21.952594+0000) 2022-01-31T22:55:22.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:22 smithi167 conmon[54076]: debug 2022-01-31T22:55:22.120+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:22.120911+0000) 2022-01-31T22:55:22.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:21 smithi167 conmon[60316]: debug 2022-01-31T22:55:21.879+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:21.880549+0000) 2022-01-31T22:55:22.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:22 smithi171 conmon[41853]: debug 2022-01-31T22:55:22.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:22.318485+0000) 2022-01-31T22:55:22.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:22 smithi171 conmon[46715]: debug 2022-01-31T22:55:22.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:22.366179+0000) 2022-01-31T22:55:22.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:22 smithi171 conmon[51620]: debug 2022-01-31T22:55:22.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:22.493797+0000) 2022-01-31T22:55:23.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:22 smithi167 conmon[60316]: debug 2022-01-31T22:55:22.879+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:22.880696+0000) 2022-01-31T22:55:23.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:22 smithi167 conmon[49112]: debug 2022-01-31T22:55:22.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:22.952747+0000) 2022-01-31T22:55:23.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:23 smithi167 conmon[54076]: debug 2022-01-31T22:55:23.120+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:23.121102+0000) 2022-01-31T22:55:23.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:23 smithi171 conmon[41853]: debug 2022-01-31T22:55:23.317+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:23.318637+0000) 2022-01-31T22:55:23.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:23 smithi171 conmon[46715]: debug 2022-01-31T22:55:23.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:23.366360+0000) 2022-01-31T22:55:23.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:23 smithi171 conmon[51620]: debug 2022-01-31T22:55:23.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:23.493965+0000) 2022-01-31T22:55:24.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:23 smithi171 conmon[35325]: debug 2022-01-31T22:55:23.931+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:55:24.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:23 smithi167 conmon[49112]: debug 2022-01-31T22:55:23.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:23.952915+0000) 2022-01-31T22:55:24.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:24 smithi167 conmon[54076]: debug 2022-01-31T22:55:24.120+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.121290+0000) 2022-01-31T22:55:24.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:23 smithi167 conmon[60316]: debug 2022-01-31T22:55:23.879+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:23.880829+0000) 2022-01-31T22:55:24.479 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[35325]: debug 2022-01-31T22:55:24.230+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248350 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:24.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[41853]: debug 2022-01-31T22:55:24.199+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.200370+0000) 2022-01-31T22:55:24.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[41853]: debug 2022-01-31T22:55:24.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.318810+0000) 2022-01-31T22:55:24.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[46715]: debug 2022-01-31T22:55:24.199+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.200218+0000) 2022-01-31T22:55:24.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[46715]: debug 2022-01-31T22:55:24.365+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.366535+0000) 2022-01-31T22:55:24.482 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[51620]: debug 2022-01-31T22:55:24.200+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.200741+0000) 2022-01-31T22:55:24.603 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:24 smithi167 conmon[49112]: debug 2022-01-31T22:55:24.198+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.199790+0000) 2022-01-31T22:55:24.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:24 smithi167 conmon[54076]: debug 2022-01-31T22:55:24.199+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.201054+0000) 2022-01-31T22:55:24.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:24 smithi167 conmon[60316]: debug 2022-01-31T22:55:24.199+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.200620+0000) 2022-01-31T22:55:24.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:24 smithi171 conmon[51620]: debug 2022-01-31T22:55:24.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.494147+0000) 2022-01-31T22:55:25.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:24 smithi167 conmon[49112]: debug 2022-01-31T22:55:24.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.953088+0000) 2022-01-31T22:55:25.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:24 smithi167 conmon[60316]: debug 2022-01-31T22:55:24.880+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:24.881018+0000) 2022-01-31T22:55:25.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:25 smithi167 conmon[54076]: debug 2022-01-31T22:55:25.120+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:25.121488+0000) 2022-01-31T22:55:25.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:25 smithi171 conmon[41853]: debug 2022-01-31T22:55:25.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:25.318954+0000) 2022-01-31T22:55:25.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:25 smithi171 conmon[46715]: debug 2022-01-31T22:55:25.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:25.366679+0000) 2022-01-31T22:55:25.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:25 smithi171 conmon[51620]: debug 2022-01-31T22:55:25.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:25.494242+0000) 2022-01-31T22:55:26.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:25 smithi167 conmon[49112]: debug 2022-01-31T22:55:25.952+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:25.953243+0000) 2022-01-31T22:55:26.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:26 smithi167 conmon[54076]: debug 2022-01-31T22:55:26.121+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:26.121642+0000) 2022-01-31T22:55:26.157 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:25 smithi167 conmon[60316]: debug 2022-01-31T22:55:25.880+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:25.881174+0000) 2022-01-31T22:55:26.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:26 smithi171 conmon[41853]: debug 2022-01-31T22:55:26.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:26.319184+0000) 2022-01-31T22:55:26.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:26 smithi171 conmon[46715]: debug 2022-01-31T22:55:26.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:26.366871+0000) 2022-01-31T22:55:26.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:26 smithi171 conmon[51620]: debug 2022-01-31T22:55:26.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:26.494422+0000) 2022-01-31T22:55:27.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:26 smithi167 conmon[49112]: debug 2022-01-31T22:55:26.951+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:26.953395+0000) 2022-01-31T22:55:27.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:27 smithi167 conmon[54076]: debug 2022-01-31T22:55:27.121+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:27.121801+0000) 2022-01-31T22:55:27.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:26 smithi167 conmon[60316]: debug 2022-01-31T22:55:26.880+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:26.881397+0000) 2022-01-31T22:55:27.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:27 smithi171 conmon[41853]: debug 2022-01-31T22:55:27.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:27.319389+0000) 2022-01-31T22:55:27.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:27 smithi171 conmon[46715]: debug 2022-01-31T22:55:27.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:27.366988+0000) 2022-01-31T22:55:27.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:27 smithi171 conmon[51620]: debug 2022-01-31T22:55:27.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:27.494576+0000) 2022-01-31T22:55:28.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:27 smithi167 conmon[49112]: debug 2022-01-31T22:55:27.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:27.953569+0000) 2022-01-31T22:55:28.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:28 smithi167 conmon[54076]: debug 2022-01-31T22:55:28.121+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:28.122038+0000) 2022-01-31T22:55:28.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:27 smithi167 conmon[60316]: debug 2022-01-31T22:55:27.881+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:27.881579+0000) 2022-01-31T22:55:28.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:28 smithi171 conmon[41853]: debug 2022-01-31T22:55:28.318+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:28.319568+0000) 2022-01-31T22:55:28.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:28 smithi171 conmon[51620]: debug 2022-01-31T22:55:28.493+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:28.494703+0000) 2022-01-31T22:55:28.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:28 smithi171 conmon[46715]: debug 2022-01-31T22:55:28.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:28.367085+0000) 2022-01-31T22:55:29.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:28 smithi167 conmon[49112]: debug 2022-01-31T22:55:28.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:28.953717+0000) 2022-01-31T22:55:29.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:28 smithi167 conmon[60316]: debug 2022-01-31T22:55:28.881+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:28.881703+0000) 2022-01-31T22:55:29.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:29 smithi167 conmon[54076]: debug 2022-01-31T22:55:29.121+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.122231+0000) 2022-01-31T22:55:29.480 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[35325]: debug 2022-01-31T22:55:29.258+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248461 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:29.480 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[46715]: debug 2022-01-31T22:55:29.233+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.234497+0000) 2022-01-31T22:55:29.481 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[46715]: debug 2022-01-31T22:55:29.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.367232+0000) 2022-01-31T22:55:29.481 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[51620]: debug 2022-01-31T22:55:29.234+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.234962+0000) 2022-01-31T22:55:29.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[41853]: debug 2022-01-31T22:55:29.234+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.234906+0000) 2022-01-31T22:55:29.482 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[41853]: debug 2022-01-31T22:55:29.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.319736+0000) 2022-01-31T22:55:29.604 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:29 smithi167 conmon[49112]: debug 2022-01-31T22:55:29.233+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.233977+0000) 2022-01-31T22:55:29.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:29 smithi167 conmon[54076]: debug 2022-01-31T22:55:29.233+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.234415+0000) 2022-01-31T22:55:29.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:29 smithi167 conmon[60316]: debug 2022-01-31T22:55:29.234+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.234995+0000) 2022-01-31T22:55:29.767 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:29 smithi171 conmon[51620]: debug 2022-01-31T22:55:29.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.494845+0000) 2022-01-31T22:55:30.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:30 smithi167 conmon[54076]: debug 2022-01-31T22:55:30.121+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:30.122420+0000) 2022-01-31T22:55:30.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:29 smithi167 conmon[49112]: debug 2022-01-31T22:55:29.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.953919+0000) 2022-01-31T22:55:30.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:29 smithi167 conmon[60316]: debug 2022-01-31T22:55:29.881+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:29.881900+0000) 2022-01-31T22:55:30.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:30 smithi171 conmon[41853]: debug 2022-01-31T22:55:30.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:30.319878+0000) 2022-01-31T22:55:30.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:30 smithi171 conmon[46715]: debug 2022-01-31T22:55:30.366+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:30.367367+0000) 2022-01-31T22:55:30.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:30 smithi171 conmon[51620]: debug 2022-01-31T22:55:30.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:30.494981+0000) 2022-01-31T22:55:31.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:30 smithi167 conmon[49112]: debug 2022-01-31T22:55:30.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:30.954045+0000) 2022-01-31T22:55:31.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:31 smithi167 conmon[54076]: debug 2022-01-31T22:55:31.122+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:31.122608+0000) 2022-01-31T22:55:31.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:30 smithi167 conmon[60316]: debug 2022-01-31T22:55:30.881+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:30.882027+0000) 2022-01-31T22:55:31.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:31 smithi171 conmon[51620]: debug 2022-01-31T22:55:31.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:31.495136+0000) 2022-01-31T22:55:31.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:31 smithi171 conmon[41853]: debug 2022-01-31T22:55:31.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:31.320060+0000) 2022-01-31T22:55:31.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:31 smithi171 conmon[46715]: debug 2022-01-31T22:55:31.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:31.367532+0000) 2022-01-31T22:55:32.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:31 smithi167 conmon[49112]: debug 2022-01-31T22:55:31.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:31.954221+0000) 2022-01-31T22:55:32.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:32 smithi167 conmon[54076]: debug 2022-01-31T22:55:32.122+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:32.122751+0000) 2022-01-31T22:55:32.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:31 smithi167 conmon[60316]: debug 2022-01-31T22:55:31.881+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:31.882189+0000) 2022-01-31T22:55:32.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:32 smithi171 conmon[51620]: debug 2022-01-31T22:55:32.494+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:32.495341+0000) 2022-01-31T22:55:32.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:32 smithi171 conmon[41853]: debug 2022-01-31T22:55:32.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:32.320212+0000) 2022-01-31T22:55:32.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:32 smithi171 conmon[46715]: debug 2022-01-31T22:55:32.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:32.367684+0000) 2022-01-31T22:55:33.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:32 smithi167 conmon[60316]: debug 2022-01-31T22:55:32.881+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:32.882373+0000) 2022-01-31T22:55:33.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:32 smithi167 conmon[49112]: debug 2022-01-31T22:55:32.953+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:32.954405+0000) 2022-01-31T22:55:33.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:33 smithi167 conmon[54076]: debug 2022-01-31T22:55:33.122+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:33.122904+0000) 2022-01-31T22:55:33.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:33 smithi171 conmon[51620]: debug 2022-01-31T22:55:33.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:33.495551+0000) 2022-01-31T22:55:33.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:33 smithi171 conmon[41853]: debug 2022-01-31T22:55:33.319+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:33.320379+0000) 2022-01-31T22:55:33.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:33 smithi171 conmon[46715]: debug 2022-01-31T22:55:33.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:33.367866+0000) 2022-01-31T22:55:34.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:33 smithi167 conmon[49112]: debug 2022-01-31T22:55:33.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:33.954621+0000) 2022-01-31T22:55:34.192 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:34 smithi167 conmon[54076]: debug 2022-01-31T22:55:34.122+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.123071+0000) 2022-01-31T22:55:34.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:33 smithi167 conmon[60316]: debug 2022-01-31T22:55:33.882+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:33.882531+0000) 2022-01-31T22:55:34.593 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[35325]: debug 2022-01-31T22:55:34.286+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248573 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:34.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[41853]: debug 2022-01-31T22:55:34.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[41853]: 2022-01-31T22:55:34.261+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.261958+0000) 2022-01-31T22:55:34.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[41853]: debug 2022-01-31T22:55:34.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.320564+0000) 2022-01-31T22:55:34.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[46715]: debug 2022-01-31T22:55:34.262+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.262797+0000) 2022-01-31T22:55:34.595 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[46715]: debug 2022-01-31T22:55:34.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.368028+0000) 2022-01-31T22:55:34.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[51620]: debug 2022-01-31T22:55:34.260+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.261730+0000) 2022-01-31T22:55:34.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:34 smithi171 conmon[51620]: debug 2022-01-31T22:55:34.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.495774+0000) 2022-01-31T22:55:34.604 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:34 smithi167 conmon[49112]: debug 2022-01-31T22:55:34.262+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.262757+0000) 2022-01-31T22:55:34.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:34 smithi167 conmon[54076]: debug 2022-01-31T22:55:34.261+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.262151+0000) 2022-01-31T22:55:34.605 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:34 smithi167 conmon[60316]: debug 2022-01-31T22:55:34.260+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.261346+0000) 2022-01-31T22:55:35.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:34 smithi167 conmon[49112]: debug 2022-01-31T22:55:34.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.954773+0000) 2022-01-31T22:55:35.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:35 smithi167 conmon[54076]: debug 2022-01-31T22:55:35.122+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:35.123288+0000) 2022-01-31T22:55:35.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:34 smithi167 conmon[60316]: debug 2022-01-31T22:55:34.882+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:34.882705+0000) 2022-01-31T22:55:35.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:35 smithi171 conmon[51620]: debug 2022-01-31T22:55:35.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:35.495931+0000) 2022-01-31T22:55:35.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:35 smithi171 conmon[41853]: debug 2022-01-31T22:55:35.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:35.320748+0000) 2022-01-31T22:55:35.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:35 smithi171 conmon[46715]: debug 2022-01-31T22:55:35.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:35.368179+0000) 2022-01-31T22:55:36.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:36 smithi167 conmon[54076]: debug 2022-01-31T22:55:36.122+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:36.123466+0000) 2022-01-31T22:55:36.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:35 smithi167 conmon[49112]: debug 2022-01-31T22:55:35.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:35.954907+0000) 2022-01-31T22:55:36.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:35 smithi167 conmon[60316]: debug 2022-01-31T22:55:35.882+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:35.882858+0000) 2022-01-31T22:55:36.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:36 smithi171 conmon[41853]: debug 2022-01-31T22:55:36.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:36.320923+0000) 2022-01-31T22:55:36.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:36 smithi171 conmon[46715]: debug 2022-01-31T22:55:36.367+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:36.368356+0000) 2022-01-31T22:55:36.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:36 smithi171 conmon[51620]: debug 2022-01-31T22:55:36.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:36.496150+0000) 2022-01-31T22:55:37.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:36 smithi167 conmon[49112]: debug 2022-01-31T22:55:36.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:36.955061+0000) 2022-01-31T22:55:37.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:37 smithi167 conmon[54076]: debug 2022-01-31T22:55:37.123+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:37.123676+0000) 2022-01-31T22:55:37.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:36 smithi167 conmon[60316]: debug 2022-01-31T22:55:36.882+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:36.883028+0000) 2022-01-31T22:55:37.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:37 smithi171 conmon[46715]: debug 2022-01-31T22:55:37.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:37.368538+0000) 2022-01-31T22:55:37.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:37 smithi171 conmon[51620]: debug 2022-01-31T22:55:37.495+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:37.496371+0000) 2022-01-31T22:55:37.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:37 smithi171 conmon[41853]: debug 2022-01-31T22:55:37.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:37.321098+0000) 2022-01-31T22:55:38.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:37 smithi167 conmon[49112]: debug 2022-01-31T22:55:37.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:37.955227+0000) 2022-01-31T22:55:38.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:38 smithi167 conmon[54076]: debug 2022-01-31T22:55:38.123+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:38.123838+0000) 2022-01-31T22:55:38.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:37 smithi167 conmon[60316]: debug 2022-01-31T22:55:37.882+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:37.883233+0000) 2022-01-31T22:55:38.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:38 smithi171 conmon[41853]: debug 2022-01-31T22:55:38.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:38.321300+0000) 2022-01-31T22:55:38.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:38 smithi171 conmon[46715]: debug 2022-01-31T22:55:38.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:38.368721+0000) 2022-01-31T22:55:38.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:38 smithi171 conmon[51620]: debug 2022-01-31T22:55:38.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:38.496587+0000) 2022-01-31T22:55:39.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:38 smithi171 conmon[35325]: debug 2022-01-31T22:55:38.932+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:55:39.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:38 smithi167 conmon[49112]: debug 2022-01-31T22:55:38.954+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:38.955366+0000) 2022-01-31T22:55:39.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:38 smithi167 conmon[60316]: debug 2022-01-31T22:55:38.882+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:38.883410+0000) 2022-01-31T22:55:39.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:39 smithi167 conmon[54076]: debug 2022-01-31T22:55:39.123+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.123995+0000) 2022-01-31T22:55:39.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[35325]: debug 2022-01-31T22:55:39.314+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:39.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[46715]: debug 2022-01-31T22:55:39.289+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.290494+0000) 2022-01-31T22:55:39.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[46715]: debug 2022-01-31T22:55:39.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.368965+0000) 2022-01-31T22:55:39.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[51620]: debug 2022-01-31T22:55:39.289+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.290062+0000) 2022-01-31T22:55:39.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[51620]: debug 2022-01-31T22:55:39.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.496799+0000) 2022-01-31T22:55:39.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[41853]: debug 2022-01-31T22:55:39.288+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.289173+0000) 2022-01-31T22:55:39.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:39 smithi171 conmon[41853]: debug 2022-01-31T22:55:39.320+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.321443+0000) 2022-01-31T22:55:39.605 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:39 smithi167 conmon[49112]: debug 2022-01-31T22:55:39.290+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.290564+0000) 2022-01-31T22:55:39.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:39 smithi167 conmon[54076]: debug 2022-01-31T22:55:39.290+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.290887+0000) 2022-01-31T22:55:39.606 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:39 smithi167 conmon[60316]: debug 2022-01-31T22:55:39.290+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.290783+0000) 2022-01-31T22:55:40.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:39 smithi167 conmon[49112]: debug 2022-01-31T22:55:39.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.955555+0000) 2022-01-31T22:55:40.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:40 smithi167 conmon[54076]: debug 2022-01-31T22:55:40.123+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:40.124186+0000) 2022-01-31T22:55:40.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:39 smithi167 conmon[60316]: debug 2022-01-31T22:55:39.883+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:39.883584+0000) 2022-01-31T22:55:40.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:40 smithi171 conmon[46715]: debug 2022-01-31T22:55:40.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:40.369143+0000) 2022-01-31T22:55:40.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:40 smithi171 conmon[51620]: debug 2022-01-31T22:55:40.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:40.497011+0000) 2022-01-31T22:55:40.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:40 smithi171 conmon[41853]: debug 2022-01-31T22:55:40.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:40.321581+0000) 2022-01-31T22:55:41.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:40 smithi167 conmon[49112]: debug 2022-01-31T22:55:40.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:40.955668+0000) 2022-01-31T22:55:41.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:41 smithi167 conmon[54076]: debug 2022-01-31T22:55:41.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:41.124396+0000) 2022-01-31T22:55:41.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:40 smithi167 conmon[60316]: debug 2022-01-31T22:55:40.883+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:40.883738+0000) 2022-01-31T22:55:41.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:41 smithi171 conmon[41853]: debug 2022-01-31T22:55:41.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:41.321786+0000) 2022-01-31T22:55:41.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:41 smithi171 conmon[46715]: debug 2022-01-31T22:55:41.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:41.369269+0000) 2022-01-31T22:55:41.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:41 smithi171 conmon[51620]: debug 2022-01-31T22:55:41.496+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:41.497222+0000) 2022-01-31T22:55:42.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:42 smithi167 conmon[54076]: debug 2022-01-31T22:55:42.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:42.124592+0000) 2022-01-31T22:55:42.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:41 smithi167 conmon[49112]: debug 2022-01-31T22:55:41.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:41.955824+0000) 2022-01-31T22:55:42.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:41 smithi167 conmon[60316]: debug 2022-01-31T22:55:41.883+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:41.883927+0000) 2022-01-31T22:55:42.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:42 smithi171 conmon[41853]: debug 2022-01-31T22:55:42.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:42.321972+0000) 2022-01-31T22:55:42.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:42 smithi171 conmon[46715]: debug 2022-01-31T22:55:42.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:42.369424+0000) 2022-01-31T22:55:42.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:42 smithi171 conmon[51620]: debug 2022-01-31T22:55:42.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:42.497435+0000) 2022-01-31T22:55:43.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:43 smithi167 conmon[54076]: debug 2022-01-31T22:55:43.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:43.124766+0000) 2022-01-31T22:55:43.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:42 smithi167 conmon[60316]: debug 2022-01-31T22:55:42.883+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:42.884107+0000) 2022-01-31T22:55:43.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:42 smithi167 conmon[49112]: debug 2022-01-31T22:55:42.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:42.956006+0000) 2022-01-31T22:55:43.592 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:43 smithi171 conmon[46715]: debug 2022-01-31T22:55:43.368+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:43.369631+0000) 2022-01-31T22:55:43.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:43 smithi171 conmon[51620]: debug 2022-01-31T22:55:43.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:43.497631+0000) 2022-01-31T22:55:43.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:43 smithi171 conmon[41853]: debug 2022-01-31T22:55:43.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:43.322148+0000) 2022-01-31T22:55:44.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:43 smithi167 conmon[49112]: debug 2022-01-31T22:55:43.955+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:43.956165+0000) 2022-01-31T22:55:44.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:44 smithi167 conmon[54076]: debug 2022-01-31T22:55:44.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.124960+0000) 2022-01-31T22:55:44.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:43 smithi167 conmon[60316]: debug 2022-01-31T22:55:43.883+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:43.884287+0000) 2022-01-31T22:55:44.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[35325]: debug 2022-01-31T22:55:44.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248797 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:44.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[46715]: debug 2022-01-31T22:55:44.318+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.319037+0000) 2022-01-31T22:55:44.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[46715]: debug 2022-01-31T22:55:44.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.369825+0000) 2022-01-31T22:55:44.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[41853]: debug 2022-01-31T22:55:44.316+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.317368+0000) 2022-01-31T22:55:44.595 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[41853]: debug 2022-01-31T22:55:44.321+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.322312+0000) 2022-01-31T22:55:44.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[51620]: debug 2022-01-31T22:55:44.316+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.317444+0000) 2022-01-31T22:55:44.596 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:44 smithi171 conmon[51620]: debug 2022-01-31T22:55:44.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.497841+0000) 2022-01-31T22:55:44.606 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:44 smithi167 conmon[49112]: debug 2022-01-31T22:55:44.318+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.319275+0000) 2022-01-31T22:55:44.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:44 smithi167 conmon[54076]: debug 2022-01-31T22:55:44.319+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.319562+0000) 2022-01-31T22:55:44.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:44 smithi167 conmon[60316]: debug 2022-01-31T22:55:44.318+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.319084+0000) 2022-01-31T22:55:45.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:45 smithi167 conmon[54076]: debug 2022-01-31T22:55:45.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:45.125181+0000) 2022-01-31T22:55:45.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:44 smithi167 conmon[49112]: debug 2022-01-31T22:55:44.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.956373+0000) 2022-01-31T22:55:45.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:44 smithi167 conmon[60316]: debug 2022-01-31T22:55:44.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:44.884519+0000) 2022-01-31T22:55:45.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:45 smithi171 conmon[41853]: debug 2022-01-31T22:55:45.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:45.322473+0000) 2022-01-31T22:55:45.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:45 smithi171 conmon[46715]: debug 2022-01-31T22:55:45.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:45.369929+0000) 2022-01-31T22:55:45.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:45 smithi171 conmon[51620]: debug 2022-01-31T22:55:45.497+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:45.497988+0000) 2022-01-31T22:55:46.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:46 smithi167 conmon[54076]: debug 2022-01-31T22:55:46.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:46.125332+0000) 2022-01-31T22:55:46.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:45 smithi167 conmon[60316]: debug 2022-01-31T22:55:45.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:45.884668+0000) 2022-01-31T22:55:46.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:45 smithi167 conmon[49112]: debug 2022-01-31T22:55:45.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:45.956542+0000) 2022-01-31T22:55:46.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:46 smithi171 conmon[51620]: debug 2022-01-31T22:55:46.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:46.498203+0000) 2022-01-31T22:55:46.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:46 smithi171 conmon[41853]: debug 2022-01-31T22:55:46.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:46.322658+0000) 2022-01-31T22:55:46.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:46 smithi171 conmon[46715]: debug 2022-01-31T22:55:46.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:46.370081+0000) 2022-01-31T22:55:47.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:46 smithi167 conmon[60316]: debug 2022-01-31T22:55:46.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:46.884907+0000) 2022-01-31T22:55:47.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:46 smithi167 conmon[49112]: debug 2022-01-31T22:55:46.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:46.956700+0000) 2022-01-31T22:55:47.161 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:47 smithi167 conmon[54076]: debug 2022-01-31T22:55:47.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:47.125527+0000) 2022-01-31T22:55:47.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:47 smithi171 conmon[41853]: debug 2022-01-31T22:55:47.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:47.322857+0000) 2022-01-31T22:55:47.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:47 smithi171 conmon[46715]: debug 2022-01-31T22:55:47.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:47.370275+0000) 2022-01-31T22:55:47.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:47 smithi171 conmon[51620]: debug 2022-01-31T22:55:47.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:47.498382+0000) 2022-01-31T22:55:48.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:47 smithi167 conmon[60316]: debug 2022-01-31T22:55:47.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:47.885060+0000) 2022-01-31T22:55:48.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:47 smithi167 conmon[49112]: debug 2022-01-31T22:55:47.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:47.956890+0000) 2022-01-31T22:55:48.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:48 smithi167 conmon[54076]: debug 2022-01-31T22:55:48.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:48.125729+0000) 2022-01-31T22:55:48.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:48 smithi171 conmon[41853]: debug 2022-01-31T22:55:48.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:48.323041+0000) 2022-01-31T22:55:48.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:48 smithi171 conmon[46715]: debug 2022-01-31T22:55:48.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:48.370458+0000) 2022-01-31T22:55:48.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:48 smithi171 conmon[51620]: debug 2022-01-31T22:55:48.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:48.498603+0000) 2022-01-31T22:55:49.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:48 smithi167 conmon[60316]: debug 2022-01-31T22:55:48.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:48.885206+0000) 2022-01-31T22:55:49.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:48 smithi167 conmon[49112]: debug 2022-01-31T22:55:48.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:48.957075+0000) 2022-01-31T22:55:49.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:49 smithi167 conmon[54076]: debug 2022-01-31T22:55:49.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.125938+0000) 2022-01-31T22:55:49.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[35325]: debug 2022-01-31T22:55:49.369+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 248924 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:49.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[41853]: debug 2022-01-31T22:55:49.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.323234+0000) 2022-01-31T22:55:49.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[41853]: debug 2022-01-31T22:55:49.345+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.345973+0000) 2022-01-31T22:55:49.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[46715]: debug 2022-01-31T22:55:49.346+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.347729+0000) 2022-01-31T22:55:49.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[46715]: debug 2022-01-31T22:55:49.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.370599+0000) 2022-01-31T22:55:49.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[51620]: debug 2022-01-31T22:55:49.346+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.347309+0000) 2022-01-31T22:55:49.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:49 smithi171 conmon[51620]: debug 2022-01-31T22:55:49.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.498848+0000) 2022-01-31T22:55:49.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:49 smithi167 conmon[54076]: debug 2022-01-31T22:55:49.346+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.347131+0000) 2022-01-31T22:55:49.607 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:49 smithi167 conmon[60316]: debug 2022-01-31T22:55:49.346+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.347485+0000) 2022-01-31T22:55:49.608 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:49 smithi167 conmon[49112]: debug 2022-01-31T22:55:49.347+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.348027+0000) 2022-01-31T22:55:50.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:49 smithi167 conmon[60316]: debug 2022-01-31T22:55:49.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.885416+0000) 2022-01-31T22:55:50.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:49 smithi167 conmon[49112]: debug 2022-01-31T22:55:49.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:49.957289+0000) 2022-01-31T22:55:50.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:50 smithi167 conmon[54076]: debug 2022-01-31T22:55:50.126+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:50.126068+0000) 2022-01-31T22:55:50.579 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:50 smithi171 conmon[41853]: debug 2022-01-31T22:55:50.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:50.323417+0000) 2022-01-31T22:55:50.580 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:50 smithi171 conmon[51620]: debug 2022-01-31T22:55:50.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:50.498997+0000) 2022-01-31T22:55:50.581 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:50 smithi171 conmon[46715]: debug 2022-01-31T22:55:50.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:50.370764+0000) 2022-01-31T22:55:51.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:50 smithi167 conmon[60316]: debug 2022-01-31T22:55:50.885+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:50.885575+0000) 2022-01-31T22:55:51.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:50 smithi167 conmon[49112]: debug 2022-01-31T22:55:50.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:50.957437+0000) 2022-01-31T22:55:51.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:51 smithi167 conmon[54076]: debug 2022-01-31T22:55:51.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:51.126181+0000) 2022-01-31T22:55:51.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:51 smithi171 conmon[41853]: debug 2022-01-31T22:55:51.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:51.323576+0000) 2022-01-31T22:55:51.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:51 smithi171 conmon[46715]: debug 2022-01-31T22:55:51.370+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:51.370922+0000) 2022-01-31T22:55:51.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:51 smithi171 conmon[51620]: debug 2022-01-31T22:55:51.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:51.499193+0000) 2022-01-31T22:55:52.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:51 smithi167 conmon[49112]: debug 2022-01-31T22:55:51.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:51.957638+0000) 2022-01-31T22:55:52.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:51 smithi167 conmon[60316]: debug 2022-01-31T22:55:51.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:51.885772+0000) 2022-01-31T22:55:52.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:52 smithi167 conmon[54076]: debug 2022-01-31T22:55:52.124+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:52.126336+0000) 2022-01-31T22:55:52.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:52 smithi171 conmon[51620]: debug 2022-01-31T22:55:52.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:52.499421+0000) 2022-01-31T22:55:52.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:52 smithi171 conmon[41853]: debug 2022-01-31T22:55:52.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:52 smithi171 conmon[41853]: 2022-01-31T22:55:52.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:52.323738+0000) 2022-01-31T22:55:52.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:52 smithi171 conmon[46715]: debug 2022-01-31T22:55:52.371+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:52.371085+0000) 2022-01-31T22:55:53.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:52 smithi167 conmon[60316]: debug 2022-01-31T22:55:52.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:52.885963+0000) 2022-01-31T22:55:53.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:52 smithi167 conmon[49112]: debug 2022-01-31T22:55:52.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:52.957854+0000) 2022-01-31T22:55:53.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:53 smithi167 conmon[54076]: debug 2022-01-31T22:55:53.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:53.126486+0000) 2022-01-31T22:55:53.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:53 smithi171 conmon[41853]: debug 2022-01-31T22:55:53.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:53.323917+0000) 2022-01-31T22:55:53.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:53 smithi171 conmon[46715]: debug 2022-01-31T22:55:53.369+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:53.371245+0000) 2022-01-31T22:55:53.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:53 smithi171 conmon[51620]: debug 2022-01-31T22:55:53.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:53.499612+0000) 2022-01-31T22:55:54.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:53 smithi171 conmon[35325]: debug 2022-01-31T22:55:53.932+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:55:54.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:53 smithi167 conmon[49112]: debug 2022-01-31T22:55:53.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:53.957965+0000) 2022-01-31T22:55:54.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:54 smithi167 conmon[54076]: debug 2022-01-31T22:55:54.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.126686+0000) 2022-01-31T22:55:54.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:53 smithi167 conmon[60316]: debug 2022-01-31T22:55:53.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:53.886113+0000) 2022-01-31T22:55:54.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:54 smithi171 conmon[35325]: debug 2022-01-31T22:55:54.395+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249036 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:54.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:54 smithi171 conmon[41853]: debug 2022-01-31T22:55:54.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.324041+0000) 2022-01-31T22:55:54.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:54 smithi171 conmon[41853]: debug 2022-01-31T22:55:54.370+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.371865+0000) 2022-01-31T22:55:54.594 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:54 smithi171 conmon[46715]: debug 2022-01-31T22:55:54.371+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.373038+0000) 2022-01-31T22:55:54.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:54 smithi171 conmon[51620]: debug 2022-01-31T22:55:54.371+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.372772+0000) 2022-01-31T22:55:54.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:54 smithi171 conmon[51620]: debug 2022-01-31T22:55:54.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.499770+0000) 2022-01-31T22:55:54.608 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:54 smithi167 conmon[49112]: debug 2022-01-31T22:55:54.371+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.373065+0000) 2022-01-31T22:55:54.608 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:54 smithi167 conmon[54076]: debug 2022-01-31T22:55:54.371+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.373153+0000) 2022-01-31T22:55:54.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:54 smithi167 conmon[60316]: debug 2022-01-31T22:55:54.370+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.372249+0000) 2022-01-31T22:55:55.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:55 smithi167 conmon[54076]: debug 2022-01-31T22:55:55.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:55.126881+0000) 2022-01-31T22:55:55.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:54 smithi167 conmon[60316]: debug 2022-01-31T22:55:54.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.886281+0000) 2022-01-31T22:55:55.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:54 smithi167 conmon[49112]: debug 2022-01-31T22:55:54.956+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:54.958169+0000) 2022-01-31T22:55:55.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:55 smithi171 conmon[41853]: debug 2022-01-31T22:55:55.322+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:55.324197+0000) 2022-01-31T22:55:55.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:55 smithi171 conmon[51620]: debug 2022-01-31T22:55:55.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:55.499922+0000) 2022-01-31T22:55:56.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:55 smithi167 conmon[49112]: debug 2022-01-31T22:55:55.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:55.958351+0000) 2022-01-31T22:55:56.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:56 smithi167 conmon[54076]: debug 2022-01-31T22:55:56.125+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:56.126964+0000) 2022-01-31T22:55:56.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:55 smithi167 conmon[60316]: debug 2022-01-31T22:55:55.884+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:55.886458+0000) 2022-01-31T22:55:56.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:56 smithi171 conmon[41853]: debug 2022-01-31T22:55:56.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:56.324370+0000) 2022-01-31T22:55:56.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:56 smithi171 conmon[51620]: debug 2022-01-31T22:55:56.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:56.500154+0000) 2022-01-31T22:55:57.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:57 smithi167 conmon[54076]: debug 2022-01-31T22:55:57.126+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:57.127121+0000) 2022-01-31T22:55:57.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:56 smithi167 conmon[49112]: debug 2022-01-31T22:55:56.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:56.958539+0000) 2022-01-31T22:55:57.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:56 smithi167 conmon[60316]: debug 2022-01-31T22:55:56.885+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:56.886646+0000) 2022-01-31T22:55:57.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:57 smithi171 conmon[41853]: debug 2022-01-31T22:55:57.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:57.324585+0000) 2022-01-31T22:55:57.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:57 smithi171 conmon[51620]: debug 2022-01-31T22:55:57.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:57.500294+0000) 2022-01-31T22:55:58.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:58 smithi167 conmon[54076]: debug 2022-01-31T22:55:58.126+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:58.127324+0000) 2022-01-31T22:55:58.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:57 smithi167 conmon[49112]: debug 2022-01-31T22:55:57.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:57.958724+0000) 2022-01-31T22:55:58.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:57 smithi167 conmon[60316]: debug 2022-01-31T22:55:57.885+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:57.886809+0000) 2022-01-31T22:55:58.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:58 smithi171 conmon[41853]: debug 2022-01-31T22:55:58.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:58.324794+0000) 2022-01-31T22:55:58.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:58 smithi171 conmon[51620]: debug 2022-01-31T22:55:58.498+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:58.500483+0000) 2022-01-31T22:55:59.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:58 smithi167 conmon[49112]: debug 2022-01-31T22:55:58.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:58.958872+0000) 2022-01-31T22:55:59.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:59 smithi167 conmon[54076]: debug 2022-01-31T22:55:59.126+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.127537+0000) 2022-01-31T22:55:59.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:58 smithi167 conmon[60316]: debug 2022-01-31T22:55:58.885+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:58.886942+0000) 2022-01-31T22:55:59.592 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:55:59 smithi171 conmon[35325]: debug 2022-01-31T22:55:59.423+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249148 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:55:59.593 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:55:59 smithi171 conmon[46715]: debug 2022-01-31T22:55:59.398+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.400597+0000) 2022-01-31T22:55:59.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:59 smithi171 conmon[41853]: debug 2022-01-31T22:55:59.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.324992+0000) 2022-01-31T22:55:59.594 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:55:59 smithi171 conmon[41853]: debug 2022-01-31T22:55:59.397+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.399105+0000) 2022-01-31T22:55:59.594 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:59 smithi171 conmon[51620]: debug 2022-01-31T22:55:59.398+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.400030+0000) 2022-01-31T22:55:59.595 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:55:59 smithi171 conmon[51620]: debug 2022-01-31T22:55:59.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.500649+0000) 2022-01-31T22:55:59.608 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:59 smithi167 conmon[49112]: debug 2022-01-31T22:55:59.398+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.400280+0000) 2022-01-31T22:55:59.609 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:55:59 smithi167 conmon[54076]: debug 2022-01-31T22:55:59.398+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.400091+0000) 2022-01-31T22:55:59.609 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:59 smithi167 conmon[60316]: debug 2022-01-31T22:55:59.398+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.399561+0000) 2022-01-31T22:56:00.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:55:59 smithi167 conmon[49112]: debug 2022-01-31T22:55:59.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.959051+0000) 2022-01-31T22:56:00.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:00 smithi167 conmon[54076]: debug 2022-01-31T22:56:00.126+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:00.127694+0000) 2022-01-31T22:56:00.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:55:59 smithi167 conmon[60316]: debug 2022-01-31T22:55:59.886+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:55:59.887046+0000) 2022-01-31T22:56:00.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:00 smithi171 conmon[41853]: debug 2022-01-31T22:56:00.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:00.325180+0000) 2022-01-31T22:56:00.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:00 smithi171 conmon[51620]: debug 2022-01-31T22:56:00.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:00.500800+0000) 2022-01-31T22:56:01.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:00 smithi167 conmon[49112]: debug 2022-01-31T22:56:00.957+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:00.959221+0000) 2022-01-31T22:56:01.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:01 smithi167 conmon[54076]: debug 2022-01-31T22:56:01.126+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:01.127899+0000) 2022-01-31T22:56:01.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:00 smithi167 conmon[60316]: debug 2022-01-31T22:56:00.885+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:00.887160+0000) 2022-01-31T22:56:01.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:01 smithi171 conmon[46715]: debug 2022-01-31T22:56:01.119+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:01.121644+0000) 2022-01-31T22:56:01.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:01 smithi171 conmon[41853]: debug 2022-01-31T22:56:01.323+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:01.325324+0000) 2022-01-31T22:56:01.772 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:01 smithi171 conmon[51620]: debug 2022-01-31T22:56:01.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:01.500973+0000) 2022-01-31T22:56:02.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:01 smithi167 conmon[60316]: debug 2022-01-31T22:56:01.886+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:01.887297+0000) 2022-01-31T22:56:02.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:01 smithi167 conmon[49112]: debug 2022-01-31T22:56:01.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:01.959400+0000) 2022-01-31T22:56:02.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:02 smithi167 conmon[54076]: debug 2022-01-31T22:56:02.127+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:02.128084+0000) 2022-01-31T22:56:02.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:02 smithi171 conmon[41853]: debug 2022-01-31T22:56:02.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:02.325498+0000) 2022-01-31T22:56:02.486 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:02 smithi171 conmon[46715]: debug 2022-01-31T22:56:02.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:02.121836+0000) 2022-01-31T22:56:02.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:02 smithi171 conmon[51620]: debug 2022-01-31T22:56:02.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:02.501189+0000) 2022-01-31T22:56:03.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:02 smithi167 conmon[60316]: debug 2022-01-31T22:56:02.886+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:02.887508+0000) 2022-01-31T22:56:03.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:02 smithi167 conmon[49112]: debug 2022-01-31T22:56:02.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:02.959610+0000) 2022-01-31T22:56:03.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:03 smithi167 conmon[54076]: debug 2022-01-31T22:56:03.127+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:03.128247+0000) 2022-01-31T22:56:03.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:03 smithi171 conmon[41853]: debug 2022-01-31T22:56:03.324+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:03.325708+0000) 2022-01-31T22:56:03.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:03 smithi171 conmon[46715]: debug 2022-01-31T22:56:03.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:03.122021+0000) 2022-01-31T22:56:03.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:03 smithi171 conmon[51620]: debug 2022-01-31T22:56:03.499+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:03.501408+0000) 2022-01-31T22:56:04.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:03 smithi167 conmon[49112]: debug 2022-01-31T22:56:03.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:03.959810+0000) 2022-01-31T22:56:04.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:04 smithi167 conmon[54076]: debug 2022-01-31T22:56:04.127+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.128437+0000) 2022-01-31T22:56:04.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:03 smithi167 conmon[60316]: debug 2022-01-31T22:56:03.886+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:03.887672+0000) 2022-01-31T22:56:04.412 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[46715]: debug 2022-01-31T22:56:04.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.122207+0000) 2022-01-31T22:56:04.412 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[41853]: debug 2022-01-31T22:56:04.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.325941+0000) 2022-01-31T22:56:04.609 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:04 smithi167 conmon[49112]: debug 2022-01-31T22:56:04.427+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.428562+0000) 2022-01-31T22:56:04.610 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:04 smithi167 conmon[60316]: debug 2022-01-31T22:56:04.427+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.428744+0000) 2022-01-31T22:56:04.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:04 smithi167 conmon[54076]: debug 2022-01-31T22:56:04.426+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.428269+0000) 2022-01-31T22:56:04.773 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[41853]: debug 2022-01-31T22:56:04.426+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.427501+0000) 2022-01-31T22:56:04.774 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[46715]: debug 2022-01-31T22:56:04.427+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.428360+0000) 2022-01-31T22:56:04.775 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[35325]: debug 2022-01-31T22:56:04.453+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249260 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:04.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[51620]: debug 2022-01-31T22:56:04.427+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.427870+0000) 2022-01-31T22:56:04.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:04 smithi171 conmon[51620]: debug 2022-01-31T22:56:04.500+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.501640+0000) 2022-01-31T22:56:05.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:04 smithi167 conmon[49112]: debug 2022-01-31T22:56:04.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.960001+0000) 2022-01-31T22:56:05.252 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:05 smithi167 conmon[54076]: debug 2022-01-31T22:56:05.127+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:05.128643+0000) 2022-01-31T22:56:05.252 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:04 smithi167 conmon[60316]: debug 2022-01-31T22:56:04.887+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:04.887840+0000) 2022-01-31T22:56:05.486 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:05 smithi171 conmon[41853]: debug 2022-01-31T22:56:05.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:05.326117+0000) 2022-01-31T22:56:05.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:05 smithi171 conmon[46715]: debug 2022-01-31T22:56:05.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:05.122408+0000) 2022-01-31T22:56:05.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:05 smithi171 conmon[51620]: debug 2022-01-31T22:56:05.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:05.501792+0000) 2022-01-31T22:56:06.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:05 smithi167 conmon[49112]: debug 2022-01-31T22:56:05.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:05.960167+0000) 2022-01-31T22:56:06.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:05 smithi167 conmon[60316]: debug 2022-01-31T22:56:05.887+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:05.887949+0000) 2022-01-31T22:56:06.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:06 smithi167 conmon[54076]: debug 2022-01-31T22:56:06.128+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:06.128796+0000) 2022-01-31T22:56:06.451 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:06 smithi171 conmon[46715]: debug 2022-01-31T22:56:06.121+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:06.122584+0000) 2022-01-31T22:56:06.452 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:06 smithi171 conmon[41853]: debug 2022-01-31T22:56:06.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:06.326286+0000) 2022-01-31T22:56:06.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:06 smithi171 conmon[51620]: debug 2022-01-31T22:56:06.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:06.502010+0000) 2022-01-31T22:56:07.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:06 smithi167 conmon[60316]: debug 2022-01-31T22:56:06.887+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:06.888125+0000) 2022-01-31T22:56:07.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:06 smithi167 conmon[49112]: debug 2022-01-31T22:56:06.958+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:06.960371+0000) 2022-01-31T22:56:07.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:07 smithi167 conmon[54076]: debug 2022-01-31T22:56:07.128+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:07.128985+0000) 2022-01-31T22:56:07.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:07 smithi171 conmon[41853]: debug 2022-01-31T22:56:07.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:07.326440+0000) 2022-01-31T22:56:07.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:07 smithi171 conmon[46715]: debug 2022-01-31T22:56:07.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:07.122771+0000) 2022-01-31T22:56:07.773 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:07 smithi171 conmon[51620]: debug 2022-01-31T22:56:07.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:07.502216+0000) 2022-01-31T22:56:08.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:07 smithi167 conmon[49112]: debug 2022-01-31T22:56:07.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:07.960528+0000) 2022-01-31T22:56:08.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:08 smithi167 conmon[54076]: debug 2022-01-31T22:56:08.127+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:08.129192+0000) 2022-01-31T22:56:08.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:07 smithi167 conmon[60316]: debug 2022-01-31T22:56:07.886+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:07.888296+0000) 2022-01-31T22:56:08.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:08 smithi171 conmon[41853]: debug 2022-01-31T22:56:08.325+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:08.326584+0000) 2022-01-31T22:56:08.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:08 smithi171 conmon[46715]: debug 2022-01-31T22:56:08.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:08.122950+0000) 2022-01-31T22:56:08.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:08 smithi171 conmon[51620]: debug 2022-01-31T22:56:08.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:08.502406+0000) 2022-01-31T22:56:09.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:08 smithi171 conmon[35325]: debug 2022-01-31T22:56:08.933+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:56:09.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:08 smithi167 conmon[49112]: debug 2022-01-31T22:56:08.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:08.960663+0000) 2022-01-31T22:56:09.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:09 smithi167 conmon[54076]: debug 2022-01-31T22:56:09.128+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.129325+0000) 2022-01-31T22:56:09.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:08 smithi167 conmon[60316]: debug 2022-01-31T22:56:08.887+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:08.888486+0000) 2022-01-31T22:56:09.440 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[41853]: debug 2022-01-31T22:56:09.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.326799+0000) 2022-01-31T22:56:09.441 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[46715]: debug 2022-01-31T22:56:09.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.123162+0000) 2022-01-31T22:56:09.610 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:09 smithi167 conmon[49112]: debug 2022-01-31T22:56:09.456+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.457946+0000) 2022-01-31T22:56:09.611 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:09 smithi167 conmon[54076]: debug 2022-01-31T22:56:09.456+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.457802+0000) 2022-01-31T22:56:09.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:09 smithi167 conmon[60316]: debug 2022-01-31T22:56:09.456+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.458274+0000) 2022-01-31T22:56:09.774 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[35325]: debug 2022-01-31T22:56:09.481+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249372 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:09.775 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[41853]: debug 2022-01-31T22:56:09.455+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.456644+0000) 2022-01-31T22:56:09.776 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[46715]: debug 2022-01-31T22:56:09.456+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.456781+0000) 2022-01-31T22:56:09.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[51620]: debug 2022-01-31T22:56:09.455+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.456161+0000) 2022-01-31T22:56:09.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:09 smithi171 conmon[51620]: debug 2022-01-31T22:56:09.501+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.502570+0000) 2022-01-31T22:56:10.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:09 smithi167 conmon[49112]: debug 2022-01-31T22:56:09.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.960866+0000) 2022-01-31T22:56:10.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:10 smithi167 conmon[54076]: debug 2022-01-31T22:56:10.128+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:10.129529+0000) 2022-01-31T22:56:10.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:09 smithi167 conmon[60316]: debug 2022-01-31T22:56:09.888+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:09.888675+0000) 2022-01-31T22:56:10.487 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:10 smithi171 conmon[41853]: debug 2022-01-31T22:56:10.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:10.327005+0000) 2022-01-31T22:56:10.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:10 smithi171 conmon[46715]: debug 2022-01-31T22:56:10.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:10.123277+0000) 2022-01-31T22:56:10.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:10 smithi171 conmon[51620]: debug 2022-01-31T22:56:10.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:10.502777+0000) 2022-01-31T22:56:11.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:10 smithi167 conmon[49112]: debug 2022-01-31T22:56:10.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:10.961012+0000) 2022-01-31T22:56:11.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:11 smithi167 conmon[54076]: debug 2022-01-31T22:56:11.129+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:11.129718+0000) 2022-01-31T22:56:11.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:10 smithi167 conmon[60316]: debug 2022-01-31T22:56:10.888+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:10.888813+0000) 2022-01-31T22:56:11.487 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:11 smithi171 conmon[46715]: debug 2022-01-31T22:56:11.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:11.123422+0000) 2022-01-31T22:56:11.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:11 smithi171 conmon[41853]: debug 2022-01-31T22:56:11.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:11.327118+0000) 2022-01-31T22:56:11.774 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:11 smithi171 conmon[51620]: debug 2022-01-31T22:56:11.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:11.502943+0000) 2022-01-31T22:56:12.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:11 smithi167 conmon[49112]: debug 2022-01-31T22:56:11.959+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:11.961191+0000) 2022-01-31T22:56:12.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:12 smithi167 conmon[54076]: debug 2022-01-31T22:56:12.129+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:12.129924+0000) 2022-01-31T22:56:12.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:11 smithi167 conmon[60316]: debug 2022-01-31T22:56:11.888+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:11.889028+0000) 2022-01-31T22:56:12.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:12 smithi171 conmon[41853]: debug 2022-01-31T22:56:12.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:12.327328+0000) 2022-01-31T22:56:12.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:12 smithi171 conmon[46715]: debug 2022-01-31T22:56:12.122+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:12.123621+0000) 2022-01-31T22:56:12.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:12 smithi171 conmon[51620]: debug 2022-01-31T22:56:12.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:12.503107+0000) 2022-01-31T22:56:13.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:12 smithi167 conmon[49112]: debug 2022-01-31T22:56:12.960+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:12.961399+0000) 2022-01-31T22:56:13.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:12 smithi167 conmon[60316]: debug 2022-01-31T22:56:12.888+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:12.889216+0000) 2022-01-31T22:56:13.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:13 smithi167 conmon[54076]: debug 2022-01-31T22:56:13.129+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:13.130077+0000) 2022-01-31T22:56:13.488 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:13 smithi171 conmon[41853]: debug 2022-01-31T22:56:13.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:13.327490+0000) 2022-01-31T22:56:13.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:13 smithi171 conmon[46715]: debug 2022-01-31T22:56:13.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:13.123822+0000) 2022-01-31T22:56:13.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:13 smithi171 conmon[51620]: debug 2022-01-31T22:56:13.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:13.503318+0000) 2022-01-31T22:56:14.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:13 smithi167 conmon[49112]: debug 2022-01-31T22:56:13.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:13.961604+0000) 2022-01-31T22:56:14.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:14 smithi167 conmon[54076]: debug 2022-01-31T22:56:14.129+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.130242+0000) 2022-01-31T22:56:14.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:13 smithi167 conmon[60316]: debug 2022-01-31T22:56:13.888+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:13.889417+0000) 2022-01-31T22:56:14.468 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[46715]: debug 2022-01-31T22:56:14.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.123937+0000) 2022-01-31T22:56:14.469 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[41853]: debug 2022-01-31T22:56:14.326+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.327693+0000) 2022-01-31T22:56:14.611 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:14 smithi167 conmon[49112]: debug 2022-01-31T22:56:14.484+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.485527+0000) 2022-01-31T22:56:14.612 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:14 smithi167 conmon[54076]: debug 2022-01-31T22:56:14.483+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.485306+0000) 2022-01-31T22:56:14.612 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:14 smithi167 conmon[60316]: debug 2022-01-31T22:56:14.483+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.485287+0000) 2022-01-31T22:56:14.775 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[35325]: debug 2022-01-31T22:56:14.509+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249485 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:14.776 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[41853]: debug 2022-01-31T22:56:14.483+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.484297+0000) 2022-01-31T22:56:14.777 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[46715]: debug 2022-01-31T22:56:14.482+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.483618+0000) 2022-01-31T22:56:14.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[51620]: debug 2022-01-31T22:56:14.484+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.485391+0000) 2022-01-31T22:56:14.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:14 smithi171 conmon[51620]: debug 2022-01-31T22:56:14.502+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.503535+0000) 2022-01-31T22:56:15.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:14 smithi167 conmon[49112]: debug 2022-01-31T22:56:14.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.961792+0000) 2022-01-31T22:56:15.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:15 smithi167 conmon[54076]: debug 2022-01-31T22:56:15.129+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:15.130448+0000) 2022-01-31T22:56:15.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:14 smithi167 conmon[60316]: debug 2022-01-31T22:56:14.889+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:14.889621+0000) 2022-01-31T22:56:15.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:15 smithi171 conmon[46715]: debug 2022-01-31T22:56:15.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:15.124124+0000) 2022-01-31T22:56:15.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:15 smithi171 conmon[41853]: debug 2022-01-31T22:56:15.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:15.327892+0000) 2022-01-31T22:56:15.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:15 smithi171 conmon[51620]: debug 2022-01-31T22:56:15.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:15.503692+0000) 2022-01-31T22:56:16.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:15 smithi167 conmon[60316]: debug 2022-01-31T22:56:15.889+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:15.889795+0000) 2022-01-31T22:56:16.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:15 smithi167 conmon[49112]: debug 2022-01-31T22:56:15.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:15.961963+0000) 2022-01-31T22:56:16.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:16 smithi167 conmon[54076]: debug 2022-01-31T22:56:16.130+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:16.130521+0000) 2022-01-31T22:56:16.488 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:16 smithi171 conmon[46715]: debug 2022-01-31T22:56:16.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:16.124286+0000) 2022-01-31T22:56:16.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:16 smithi171 conmon[41853]: debug 2022-01-31T22:56:16.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:16.327991+0000) 2022-01-31T22:56:16.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:16 smithi171 conmon[51620]: debug 2022-01-31T22:56:16.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:16.503857+0000) 2022-01-31T22:56:17.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:16 smithi167 conmon[49112]: debug 2022-01-31T22:56:16.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:16.962151+0000) 2022-01-31T22:56:17.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:17 smithi167 conmon[54076]: debug 2022-01-31T22:56:17.130+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:17.130682+0000) 2022-01-31T22:56:17.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:16 smithi167 conmon[60316]: debug 2022-01-31T22:56:16.889+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:16.889979+0000) 2022-01-31T22:56:17.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:17 smithi171 conmon[41853]: debug 2022-01-31T22:56:17.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:17.328096+0000) 2022-01-31T22:56:17.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:17 smithi171 conmon[46715]: debug 2022-01-31T22:56:17.123+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:17.124476+0000) 2022-01-31T22:56:17.775 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:17 smithi171 conmon[51620]: debug 2022-01-31T22:56:17.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:17.504045+0000) 2022-01-31T22:56:18.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:17 smithi167 conmon[60316]: debug 2022-01-31T22:56:17.889+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:17.890212+0000) 2022-01-31T22:56:18.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:17 smithi167 conmon[49112]: debug 2022-01-31T22:56:17.961+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:17.962359+0000) 2022-01-31T22:56:18.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:18 smithi167 conmon[54076]: debug 2022-01-31T22:56:18.130+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:18.130871+0000) 2022-01-31T22:56:18.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:18 smithi171 conmon[41853]: debug 2022-01-31T22:56:18.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:18.328211+0000) 2022-01-31T22:56:18.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:18 smithi171 conmon[46715]: debug 2022-01-31T22:56:18.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:18.124660+0000) 2022-01-31T22:56:18.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:18 smithi171 conmon[51620]: debug 2022-01-31T22:56:18.503+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:18.504263+0000) 2022-01-31T22:56:19.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:18 smithi167 conmon[49112]: debug 2022-01-31T22:56:18.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:18.962543+0000) 2022-01-31T22:56:19.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:19 smithi167 conmon[54076]: debug 2022-01-31T22:56:19.130+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.131058+0000) 2022-01-31T22:56:19.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:18 smithi167 conmon[60316]: debug 2022-01-31T22:56:18.889+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:18.890416+0000) 2022-01-31T22:56:19.489 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[46715]: debug 2022-01-31T22:56:19.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.124822+0000) 2022-01-31T22:56:19.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[41853]: debug 2022-01-31T22:56:19.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.328403+0000) 2022-01-31T22:56:19.612 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:19 smithi167 conmon[49112]: debug 2022-01-31T22:56:19.511+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.512302+0000) 2022-01-31T22:56:19.613 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:19 smithi167 conmon[54076]: debug 2022-01-31T22:56:19.512+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.513061+0000) 2022-01-31T22:56:19.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:19 smithi167 conmon[60316]: debug 2022-01-31T22:56:19.512+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.512689+0000) 2022-01-31T22:56:19.776 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[35325]: debug 2022-01-31T22:56:19.537+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249595 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:19.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[41853]: debug 2022-01-31T22:56:19.511+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.512627+0000) 2022-01-31T22:56:19.778 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[46715]: debug 2022-01-31T22:56:19.511+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.512121+0000) 2022-01-31T22:56:19.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[51620]: debug 2022-01-31T22:56:19.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.504472+0000) 2022-01-31T22:56:19.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:19 smithi171 conmon[51620]: debug 2022-01-31T22:56:19.512+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.513537+0000) 2022-01-31T22:56:20.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:19 smithi167 conmon[49112]: debug 2022-01-31T22:56:19.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.962649+0000) 2022-01-31T22:56:20.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:20 smithi167 conmon[54076]: debug 2022-01-31T22:56:20.130+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:20.131188+0000) 2022-01-31T22:56:20.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:19 smithi167 conmon[60316]: debug 2022-01-31T22:56:19.890+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:19.890594+0000) 2022-01-31T22:56:20.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:20 smithi171 conmon[41853]: debug 2022-01-31T22:56:20.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:20.328569+0000) 2022-01-31T22:56:20.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:20 smithi171 conmon[46715]: debug 2022-01-31T22:56:20.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:20.124922+0000) 2022-01-31T22:56:20.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:20 smithi171 conmon[51620]: debug 2022-01-31T22:56:20.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:20.504658+0000) 2022-01-31T22:56:21.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:20 smithi167 conmon[49112]: debug 2022-01-31T22:56:20.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:20.962784+0000) 2022-01-31T22:56:21.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:21 smithi167 conmon[54076]: debug 2022-01-31T22:56:21.130+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:21.131363+0000) 2022-01-31T22:56:21.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:20 smithi167 conmon[60316]: debug 2022-01-31T22:56:20.890+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:20.890742+0000) 2022-01-31T22:56:21.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:21 smithi171 conmon[41853]: debug 2022-01-31T22:56:21.327+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:21.328721+0000) 2022-01-31T22:56:21.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:21 smithi171 conmon[46715]: debug 2022-01-31T22:56:21.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:21.125054+0000) 2022-01-31T22:56:21.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:21 smithi171 conmon[51620]: debug 2022-01-31T22:56:21.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:21.504813+0000) 2022-01-31T22:56:22.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:21 smithi167 conmon[49112]: debug 2022-01-31T22:56:21.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:21.962939+0000) 2022-01-31T22:56:22.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:22 smithi167 conmon[54076]: debug 2022-01-31T22:56:22.131+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:22.131516+0000) 2022-01-31T22:56:22.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:21 smithi167 conmon[60316]: debug 2022-01-31T22:56:21.890+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:21.890933+0000) 2022-01-31T22:56:22.489 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:22 smithi171 conmon[41853]: debug 2022-01-31T22:56:22.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:22.328848+0000) 2022-01-31T22:56:22.490 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:22 smithi171 conmon[46715]: debug 2022-01-31T22:56:22.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:22.125237+0000) 2022-01-31T22:56:22.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:22 smithi171 conmon[51620]: debug 2022-01-31T22:56:22.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:22.504987+0000) 2022-01-31T22:56:23.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:22 smithi167 conmon[60316]: debug 2022-01-31T22:56:22.890+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:22.891121+0000) 2022-01-31T22:56:23.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:22 smithi167 conmon[49112]: debug 2022-01-31T22:56:22.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:22.963063+0000) 2022-01-31T22:56:23.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:23 smithi167 conmon[54076]: debug 2022-01-31T22:56:23.131+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:23.131644+0000) 2022-01-31T22:56:23.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:23 smithi171 conmon[41853]: debug 2022-01-31T22:56:23.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:23.329053+0000) 2022-01-31T22:56:23.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:23 smithi171 conmon[46715]: debug 2022-01-31T22:56:23.124+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:23.125422+0000) 2022-01-31T22:56:23.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:23 smithi171 conmon[51620]: debug 2022-01-31T22:56:23.504+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:23.505220+0000) 2022-01-31T22:56:24.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:23 smithi171 conmon[35325]: debug 2022-01-31T22:56:23.934+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:56:24.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:23 smithi167 conmon[49112]: debug 2022-01-31T22:56:23.962+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:23.963234+0000) 2022-01-31T22:56:24.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:24 smithi167 conmon[54076]: debug 2022-01-31T22:56:24.131+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.131816+0000) 2022-01-31T22:56:24.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:23 smithi167 conmon[60316]: debug 2022-01-31T22:56:23.890+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:23.891266+0000) 2022-01-31T22:56:24.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[41853]: debug 2022-01-31T22:56:24.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.329171+0000) 2022-01-31T22:56:24.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[46715]: debug 2022-01-31T22:56:24.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.125600+0000) 2022-01-31T22:56:24.613 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:24 smithi167 conmon[49112]: debug 2022-01-31T22:56:24.540+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.540791+0000) 2022-01-31T22:56:24.614 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:24 smithi167 conmon[54076]: debug 2022-01-31T22:56:24.541+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.541683+0000) 2022-01-31T22:56:24.614 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:24 smithi167 conmon[60316]: debug 2022-01-31T22:56:24.547+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.548204+0000) 2022-01-31T22:56:24.777 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[35325]: debug 2022-01-31T22:56:24.572+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249709 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:24.777 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[41853]: debug 2022-01-31T22:56:24.539+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.540465+0000) 2022-01-31T22:56:24.778 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[46715]: debug 2022-01-31T22:56:24.539+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.540461+0000) 2022-01-31T22:56:24.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[51620]: debug 2022-01-31T22:56:24.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.505432+0000) 2022-01-31T22:56:24.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:24 smithi171 conmon[51620]: debug 2022-01-31T22:56:24.541+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.541919+0000) 2022-01-31T22:56:25.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:24 smithi167 conmon[49112]: debug 2022-01-31T22:56:24.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.963370+0000) 2022-01-31T22:56:25.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:25 smithi167 conmon[54076]: debug 2022-01-31T22:56:25.131+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:25.132030+0000) 2022-01-31T22:56:25.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:24 smithi167 conmon[60316]: debug 2022-01-31T22:56:24.891+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:24.891495+0000) 2022-01-31T22:56:25.490 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:25 smithi171 conmon[41853]: debug 2022-01-31T22:56:25.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:25.329377+0000) 2022-01-31T22:56:25.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:25 smithi171 conmon[46715]: debug 2022-01-31T22:56:25.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:25.125792+0000) 2022-01-31T22:56:25.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:25 smithi171 conmon[51620]: debug 2022-01-31T22:56:25.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:25.505639+0000) 2022-01-31T22:56:26.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:25 smithi167 conmon[49112]: debug 2022-01-31T22:56:25.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:25.963546+0000) 2022-01-31T22:56:26.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:26 smithi167 conmon[54076]: debug 2022-01-31T22:56:26.131+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:26.132185+0000) 2022-01-31T22:56:26.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:25 smithi167 conmon[60316]: debug 2022-01-31T22:56:25.891+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:25.891683+0000) 2022-01-31T22:56:26.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:26 smithi171 conmon[46715]: debug 2022-01-31T22:56:26.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:26.125953+0000) 2022-01-31T22:56:26.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:26 smithi171 conmon[41853]: debug 2022-01-31T22:56:26.328+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:26.329545+0000) 2022-01-31T22:56:26.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:26 smithi171 conmon[51620]: debug 2022-01-31T22:56:26.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:26.505804+0000) 2022-01-31T22:56:27.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:26 smithi167 conmon[49112]: debug 2022-01-31T22:56:26.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:26.963737+0000) 2022-01-31T22:56:27.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:27 smithi167 conmon[54076]: debug 2022-01-31T22:56:27.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:27.132374+0000) 2022-01-31T22:56:27.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:26 smithi167 conmon[60316]: debug 2022-01-31T22:56:26.891+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:26.891883+0000) 2022-01-31T22:56:27.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:27 smithi171 conmon[41853]: debug 2022-01-31T22:56:27.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:27.329727+0000) 2022-01-31T22:56:27.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:27 smithi171 conmon[46715]: debug 2022-01-31T22:56:27.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:27.126148+0000) 2022-01-31T22:56:27.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:27 smithi171 conmon[51620]: debug 2022-01-31T22:56:27.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:27.506012+0000) 2022-01-31T22:56:28.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:27 smithi167 conmon[49112]: debug 2022-01-31T22:56:27.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:27.963918+0000) 2022-01-31T22:56:28.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:28 smithi167 conmon[54076]: debug 2022-01-31T22:56:28.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:28.132562+0000) 2022-01-31T22:56:28.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:27 smithi167 conmon[60316]: debug 2022-01-31T22:56:27.891+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:27.892068+0000) 2022-01-31T22:56:28.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:28 smithi171 conmon[41853]: debug 2022-01-31T22:56:28.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:28.329902+0000) 2022-01-31T22:56:28.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:28 smithi171 conmon[46715]: debug 2022-01-31T22:56:28.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:28.126386+0000) 2022-01-31T22:56:28.777 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:28 smithi171 conmon[51620]: debug 2022-01-31T22:56:28.505+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:28.506246+0000) 2022-01-31T22:56:29.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:28 smithi167 conmon[49112]: debug 2022-01-31T22:56:28.963+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:28.964114+0000) 2022-01-31T22:56:29.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:29 smithi167 conmon[54076]: debug 2022-01-31T22:56:29.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.132766+0000) 2022-01-31T22:56:29.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:28 smithi167 conmon[60316]: debug 2022-01-31T22:56:28.891+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:28.892226+0000) 2022-01-31T22:56:29.491 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[46715]: debug 2022-01-31T22:56:29.125+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.126610+0000) 2022-01-31T22:56:29.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[41853]: debug 2022-01-31T22:56:29.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.330060+0000) 2022-01-31T22:56:29.614 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:29 smithi167 conmon[49112]: debug 2022-01-31T22:56:29.574+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.575049+0000) 2022-01-31T22:56:29.614 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:29 smithi167 conmon[54076]: debug 2022-01-31T22:56:29.575+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.576093+0000) 2022-01-31T22:56:29.615 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:29 smithi167 conmon[60316]: debug 2022-01-31T22:56:29.575+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.575858+0000) 2022-01-31T22:56:29.778 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[35325]: debug 2022-01-31T22:56:29.600+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249820 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:29.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[41853]: debug 2022-01-31T22:56:29.574+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.575226+0000) 2022-01-31T22:56:29.779 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[46715]: debug 2022-01-31T22:56:29.574+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.574826+0000) 2022-01-31T22:56:29.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[51620]: debug 2022-01-31T22:56:29.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.506462+0000) 2022-01-31T22:56:29.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:29 smithi171 conmon[51620]: debug 2022-01-31T22:56:29.575+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.576077+0000) 2022-01-31T22:56:30.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:29 smithi167 conmon[49112]: debug 2022-01-31T22:56:29.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.964334+0000) 2022-01-31T22:56:30.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:30 smithi167 conmon[54076]: debug 2022-01-31T22:56:30.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:30.132972+0000) 2022-01-31T22:56:30.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:29 smithi167 conmon[60316]: debug 2022-01-31T22:56:29.891+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:29.892425+0000) 2022-01-31T22:56:30.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:30 smithi171 conmon[41853]: debug 2022-01-31T22:56:30.329+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:30.330249+0000) 2022-01-31T22:56:30.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:30 smithi171 conmon[46715]: debug 2022-01-31T22:56:30.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:30.126839+0000) 2022-01-31T22:56:30.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:30 smithi171 conmon[51620]: debug 2022-01-31T22:56:30.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:30.506680+0000) 2022-01-31T22:56:31.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:30 smithi167 conmon[49112]: debug 2022-01-31T22:56:30.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:30.964492+0000) 2022-01-31T22:56:31.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:31 smithi167 conmon[54076]: debug 2022-01-31T22:56:31.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:31.133088+0000) 2022-01-31T22:56:31.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:30 smithi167 conmon[60316]: debug 2022-01-31T22:56:30.892+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:30.892573+0000) 2022-01-31T22:56:31.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:31 smithi171 conmon[41853]: debug 2022-01-31T22:56:31.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:31.330351+0000) 2022-01-31T22:56:31.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:31 smithi171 conmon[46715]: debug 2022-01-31T22:56:31.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:31.126959+0000) 2022-01-31T22:56:31.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:31 smithi171 conmon[51620]: debug 2022-01-31T22:56:31.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:31.506837+0000) 2022-01-31T22:56:32.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:31 smithi167 conmon[60316]: debug 2022-01-31T22:56:31.892+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:31.892753+0000) 2022-01-31T22:56:32.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:31 smithi167 conmon[49112]: debug 2022-01-31T22:56:31.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:31.964699+0000) 2022-01-31T22:56:32.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:32 smithi167 conmon[54076]: debug 2022-01-31T22:56:32.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:32.133265+0000) 2022-01-31T22:56:32.491 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:32 smithi171 conmon[41853]: debug 2022-01-31T22:56:32.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:32.330492+0000) 2022-01-31T22:56:32.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:32 smithi171 conmon[46715]: debug 2022-01-31T22:56:32.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:32.127159+0000) 2022-01-31T22:56:32.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:32 smithi171 conmon[51620]: debug 2022-01-31T22:56:32.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:32.507045+0000) 2022-01-31T22:56:33.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:32 smithi167 conmon[49112]: debug 2022-01-31T22:56:32.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:32.964884+0000) 2022-01-31T22:56:33.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:33 smithi167 conmon[54076]: debug 2022-01-31T22:56:33.132+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:33.133472+0000) 2022-01-31T22:56:33.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:32 smithi167 conmon[60316]: debug 2022-01-31T22:56:32.892+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:32.892941+0000) 2022-01-31T22:56:33.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:33 smithi171 conmon[41853]: debug 2022-01-31T22:56:33.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:33.330660+0000) 2022-01-31T22:56:33.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:33 smithi171 conmon[46715]: debug 2022-01-31T22:56:33.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:33.127337+0000) 2022-01-31T22:56:33.778 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:33 smithi171 conmon[51620]: debug 2022-01-31T22:56:33.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:33.507168+0000) 2022-01-31T22:56:34.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:33 smithi167 conmon[49112]: debug 2022-01-31T22:56:33.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:33.965039+0000) 2022-01-31T22:56:34.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:34 smithi167 conmon[54076]: debug 2022-01-31T22:56:34.133+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.133682+0000) 2022-01-31T22:56:34.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:33 smithi167 conmon[60316]: debug 2022-01-31T22:56:33.892+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:33.893112+0000) 2022-01-31T22:56:34.492 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[46715]: debug 2022-01-31T22:56:34.126+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.127552+0000) 2022-01-31T22:56:34.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[41853]: debug 2022-01-31T22:56:34.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.330849+0000) 2022-01-31T22:56:34.778 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[41853]: debug 2022-01-31T22:56:34.602+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.603731+0000) 2022-01-31T22:56:34.779 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[46715]: debug 2022-01-31T22:56:34.602+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.603488+0000) 2022-01-31T22:56:34.780 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[35325]: debug 2022-01-31T22:56:34.628+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 249934 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:34.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[51620]: debug 2022-01-31T22:56:34.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.507391+0000) 2022-01-31T22:56:34.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:34 smithi171 conmon[51620]: debug 2022-01-31T22:56:34.603+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.604588+0000) 2022-01-31T22:56:34.849 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:34 smithi167 conmon[49112]: debug 2022-01-31T22:56:34.604+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.605105+0000) 2022-01-31T22:56:34.850 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:34 smithi167 conmon[54076]: debug 2022-01-31T22:56:34.603+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.604371+0000) 2022-01-31T22:56:34.850 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:34 smithi167 conmon[60316]: debug 2022-01-31T22:56:34.603+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.604269+0000) 2022-01-31T22:56:35.118 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:34 smithi167 conmon[60316]: debug 2022-01-31T22:56:34.892+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.893332+0000) 2022-01-31T22:56:35.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:34 smithi167 conmon[49112]: debug 2022-01-31T22:56:34.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:34.965238+0000) 2022-01-31T22:56:35.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:35 smithi167 conmon[54076]: debug 2022-01-31T22:56:35.133+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:35.133887+0000) 2022-01-31T22:56:35.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:35 smithi171 conmon[41853]: debug 2022-01-31T22:56:35.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:35.331035+0000) 2022-01-31T22:56:35.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:35 smithi171 conmon[46715]: debug 2022-01-31T22:56:35.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:35.127769+0000) 2022-01-31T22:56:35.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:35 smithi171 conmon[51620]: debug 2022-01-31T22:56:35.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:35.507588+0000) 2022-01-31T22:56:36.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:35 smithi167 conmon[49112]: debug 2022-01-31T22:56:35.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:35.965375+0000) 2022-01-31T22:56:36.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:36 smithi167 conmon[54076]: debug 2022-01-31T22:56:36.133+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:36.134082+0000) 2022-01-31T22:56:36.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:35 smithi167 conmon[60316]: debug 2022-01-31T22:56:35.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:35.893538+0000) 2022-01-31T22:56:36.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:36 smithi171 conmon[41853]: debug 2022-01-31T22:56:36.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:36.331175+0000) 2022-01-31T22:56:36.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:36 smithi171 conmon[46715]: debug 2022-01-31T22:56:36.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:36.127923+0000) 2022-01-31T22:56:36.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:36 smithi171 conmon[51620]: debug 2022-01-31T22:56:36.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:36.507792+0000) 2022-01-31T22:56:37.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:36 smithi167 conmon[49112]: debug 2022-01-31T22:56:36.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:36.965601+0000) 2022-01-31T22:56:37.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:37 smithi167 conmon[54076]: debug 2022-01-31T22:56:37.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:37.134249+0000) 2022-01-31T22:56:37.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:36 smithi167 conmon[60316]: debug 2022-01-31T22:56:36.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:36.893727+0000) 2022-01-31T22:56:37.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:37 smithi171 conmon[41853]: debug 2022-01-31T22:56:37.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:37.331272+0000) 2022-01-31T22:56:37.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:37 smithi171 conmon[46715]: debug 2022-01-31T22:56:37.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:37.128074+0000) 2022-01-31T22:56:37.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:37 smithi171 conmon[51620]: debug 2022-01-31T22:56:37.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:37.508014+0000) 2022-01-31T22:56:38.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:37 smithi167 conmon[49112]: debug 2022-01-31T22:56:37.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:37.965803+0000) 2022-01-31T22:56:38.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:38 smithi167 conmon[54076]: debug 2022-01-31T22:56:38.133+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:38.134495+0000) 2022-01-31T22:56:38.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:37 smithi167 conmon[60316]: debug 2022-01-31T22:56:37.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:37.893887+0000) 2022-01-31T22:56:38.492 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:38 smithi171 conmon[41853]: debug 2022-01-31T22:56:38.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:38.331455+0000) 2022-01-31T22:56:38.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:38 smithi171 conmon[46715]: debug 2022-01-31T22:56:38.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:38.128211+0000) 2022-01-31T22:56:38.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:38 smithi171 conmon[51620]: debug 2022-01-31T22:56:38.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:38.508174+0000) 2022-01-31T22:56:39.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:38 smithi171 conmon[35325]: debug 2022-01-31T22:56:38.934+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:56:39.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:38 smithi167 conmon[49112]: debug 2022-01-31T22:56:38.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:38.965985+0000) 2022-01-31T22:56:39.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:39 smithi167 conmon[54076]: debug 2022-01-31T22:56:39.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.134676+0000) 2022-01-31T22:56:39.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:38 smithi167 conmon[60316]: debug 2022-01-31T22:56:38.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:38.894003+0000) 2022-01-31T22:56:39.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[41853]: debug 2022-01-31T22:56:39.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.331647+0000) 2022-01-31T22:56:39.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[46715]: debug 2022-01-31T22:56:39.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.128362+0000) 2022-01-31T22:56:39.779 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[35325]: debug 2022-01-31T22:56:39.654+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250046 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:39.780 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[41853]: debug 2022-01-31T22:56:39.629+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.631670+0000) 2022-01-31T22:56:39.781 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[46715]: debug 2022-01-31T22:56:39.629+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.631014+0000) 2022-01-31T22:56:39.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[51620]: debug 2022-01-31T22:56:39.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.508359+0000) 2022-01-31T22:56:39.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:39 smithi171 conmon[51620]: debug 2022-01-31T22:56:39.629+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.631268+0000) 2022-01-31T22:56:39.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:39 smithi167 conmon[49112]: debug 2022-01-31T22:56:39.631+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.633049+0000) 2022-01-31T22:56:39.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:39 smithi167 conmon[54076]: debug 2022-01-31T22:56:39.630+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.632243+0000) 2022-01-31T22:56:39.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:39 smithi167 conmon[60316]: debug 2022-01-31T22:56:39.630+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.632271+0000) 2022-01-31T22:56:40.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:39 smithi167 conmon[49112]: debug 2022-01-31T22:56:39.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.966165+0000) 2022-01-31T22:56:40.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:40 smithi167 conmon[54076]: debug 2022-01-31T22:56:40.133+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:40.134833+0000) 2022-01-31T22:56:40.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:39 smithi167 conmon[60316]: debug 2022-01-31T22:56:39.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:39.894195+0000) 2022-01-31T22:56:40.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:40 smithi171 conmon[46715]: debug 2022-01-31T22:56:40.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:40.128533+0000) 2022-01-31T22:56:40.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:40 smithi171 conmon[41853]: debug 2022-01-31T22:56:40.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:40.331799+0000) 2022-01-31T22:56:40.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:40 smithi171 conmon[51620]: debug 2022-01-31T22:56:40.506+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:40.508546+0000) 2022-01-31T22:56:41.119 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:40 smithi167 conmon[49112]: debug 2022-01-31T22:56:40.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:40.966311+0000) 2022-01-31T22:56:41.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:40 smithi167 conmon[60316]: debug 2022-01-31T22:56:40.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:40.894362+0000) 2022-01-31T22:56:41.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:41 smithi167 conmon[54076]: debug 2022-01-31T22:56:41.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:41.135029+0000) 2022-01-31T22:56:41.493 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:41 smithi171 conmon[46715]: debug 2022-01-31T22:56:41.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:41.128664+0000) 2022-01-31T22:56:41.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:41 smithi171 conmon[41853]: debug 2022-01-31T22:56:41.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:41.331891+0000) 2022-01-31T22:56:41.779 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:41 smithi171 conmon[51620]: debug 2022-01-31T22:56:41.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:41.508694+0000) 2022-01-31T22:56:42.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:41 smithi167 conmon[60316]: debug 2022-01-31T22:56:41.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:41.894556+0000) 2022-01-31T22:56:42.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:41 smithi167 conmon[49112]: debug 2022-01-31T22:56:41.964+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:41.966475+0000) 2022-01-31T22:56:42.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:42 smithi167 conmon[54076]: debug 2022-01-31T22:56:42.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:42.135184+0000) 2022-01-31T22:56:42.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:42 smithi171 conmon[41853]: debug 2022-01-31T22:56:42.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:42.332042+0000) 2022-01-31T22:56:42.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:42 smithi171 conmon[46715]: debug 2022-01-31T22:56:42.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:42.128890+0000) 2022-01-31T22:56:42.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:42 smithi171 conmon[51620]: debug 2022-01-31T22:56:42.507+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:42.508900+0000) 2022-01-31T22:56:43.120 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:42 smithi167 conmon[60316]: debug 2022-01-31T22:56:42.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:42.894697+0000) 2022-01-31T22:56:43.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:42 smithi167 conmon[49112]: debug 2022-01-31T22:56:42.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:42.966630+0000) 2022-01-31T22:56:43.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:43 smithi167 conmon[54076]: debug 2022-01-31T22:56:43.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:43.135374+0000) 2022-01-31T22:56:43.493 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:43 smithi171 conmon[41853]: debug 2022-01-31T22:56:43.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:43.332230+0000) 2022-01-31T22:56:43.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:43 smithi171 conmon[46715]: debug 2022-01-31T22:56:43.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:43.129025+0000) 2022-01-31T22:56:43.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:43 smithi171 conmon[51620]: debug 2022-01-31T22:56:43.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:43.509132+0000) 2022-01-31T22:56:44.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:43 smithi167 conmon[49112]: debug 2022-01-31T22:56:43.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:43.966757+0000) 2022-01-31T22:56:44.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:43 smithi167 conmon[60316]: debug 2022-01-31T22:56:43.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:43.894859+0000) 2022-01-31T22:56:44.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:44 smithi167 conmon[54076]: debug 2022-01-31T22:56:44.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.135591+0000) 2022-01-31T22:56:44.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[41853]: debug 2022-01-31T22:56:44.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.332401+0000) 2022-01-31T22:56:44.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[46715]: debug 2022-01-31T22:56:44.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.129216+0000) 2022-01-31T22:56:44.780 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[35325]: debug 2022-01-31T22:56:44.683+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250159 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:44.781 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[41853]: debug 2022-01-31T22:56:44.656+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.658525+0000) 2022-01-31T22:56:44.781 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[46715]: debug 2022-01-31T22:56:44.656+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.658397+0000) 2022-01-31T22:56:44.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[51620]: debug 2022-01-31T22:56:44.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.509340+0000) 2022-01-31T22:56:44.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:44 smithi171 conmon[51620]: debug 2022-01-31T22:56:44.657+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.659040+0000) 2022-01-31T22:56:44.904 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:44 smithi167 conmon[49112]: debug 2022-01-31T22:56:44.658+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.659867+0000) 2022-01-31T22:56:44.905 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:44 smithi167 conmon[54076]: debug 2022-01-31T22:56:44.657+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.659222+0000) 2022-01-31T22:56:44.905 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:44 smithi167 conmon[60316]: debug 2022-01-31T22:56:44.658+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.660079+0000) 2022-01-31T22:56:44.906 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:44 smithi167 conmon[60316]: debug 2022-01-31T22:56:44.893+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.895066+0000) 2022-01-31T22:56:45.326 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:44 smithi167 conmon[49112]: debug 2022-01-31T22:56:44.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:44.966907+0000) 2022-01-31T22:56:45.327 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:45 smithi167 conmon[54076]: debug 2022-01-31T22:56:45.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:45.135783+0000) 2022-01-31T22:56:45.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:45 smithi171 conmon[41853]: debug 2022-01-31T22:56:45.330+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:45.332555+0000) 2022-01-31T22:56:45.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:45 smithi171 conmon[46715]: debug 2022-01-31T22:56:45.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:45.129427+0000) 2022-01-31T22:56:45.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:45 smithi171 conmon[51620]: debug 2022-01-31T22:56:45.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:45.509551+0000) 2022-01-31T22:56:46.120 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:45 smithi167 conmon[49112]: debug 2022-01-31T22:56:45.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:45.967075+0000) 2022-01-31T22:56:46.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:45 smithi167 conmon[60316]: debug 2022-01-31T22:56:45.894+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:45.895221+0000) 2022-01-31T22:56:46.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:46 smithi167 conmon[54076]: debug 2022-01-31T22:56:46.134+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:46.135911+0000) 2022-01-31T22:56:46.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:46 smithi171 conmon[46715]: debug 2022-01-31T22:56:46.127+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:46.129613+0000) 2022-01-31T22:56:46.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:46 smithi171 conmon[41853]: debug 2022-01-31T22:56:46.331+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:46.332697+0000) 2022-01-31T22:56:46.780 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:46 smithi171 conmon[51620]: debug 2022-01-31T22:56:46.508+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:46.509719+0000) 2022-01-31T22:56:47.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:46 smithi167 conmon[49112]: debug 2022-01-31T22:56:46.965+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:46.967255+0000) 2022-01-31T22:56:47.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:46 smithi167 conmon[60316]: debug 2022-01-31T22:56:46.894+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:46.895408+0000) 2022-01-31T22:56:47.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:47 smithi167 conmon[54076]: debug 2022-01-31T22:56:47.135+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:47.136076+0000) 2022-01-31T22:56:47.494 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:47 smithi171 conmon[41853]: debug 2022-01-31T22:56:47.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:47.332904+0000) 2022-01-31T22:56:47.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:47 smithi171 conmon[46715]: debug 2022-01-31T22:56:47.128+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:47.129818+0000) 2022-01-31T22:56:47.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:47 smithi171 conmon[51620]: debug 2022-01-31T22:56:47.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:47.509954+0000) 2022-01-31T22:56:48.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:47 smithi167 conmon[49112]: debug 2022-01-31T22:56:47.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:47.967454+0000) 2022-01-31T22:56:48.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:47 smithi167 conmon[60316]: debug 2022-01-31T22:56:47.894+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:47.895564+0000) 2022-01-31T22:56:48.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:48 smithi167 conmon[54076]: debug 2022-01-31T22:56:48.135+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:48.136266+0000) 2022-01-31T22:56:48.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:48 smithi171 conmon[41853]: debug 2022-01-31T22:56:48.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:48.333121+0000) 2022-01-31T22:56:48.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:48 smithi171 conmon[46715]: debug 2022-01-31T22:56:48.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:48.130009+0000) 2022-01-31T22:56:48.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:48 smithi171 conmon[51620]: debug 2022-01-31T22:56:48.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:48.510125+0000) 2022-01-31T22:56:49.121 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:48 smithi167 conmon[49112]: debug 2022-01-31T22:56:48.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:48.967739+0000) 2022-01-31T22:56:49.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:48 smithi167 conmon[60316]: debug 2022-01-31T22:56:48.894+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:48.895829+0000) 2022-01-31T22:56:49.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:49 smithi167 conmon[54076]: debug 2022-01-31T22:56:49.135+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.136554+0000) 2022-01-31T22:56:49.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[46715]: debug 2022-01-31T22:56:49.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.130243+0000) 2022-01-31T22:56:49.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[41853]: debug 2022-01-31T22:56:49.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.333399+0000) 2022-01-31T22:56:49.781 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[41853]: debug 2022-01-31T22:56:49.686+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.687990+0000) 2022-01-31T22:56:49.782 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[46715]: debug 2022-01-31T22:56:49.685+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.687110+0000) 2022-01-31T22:56:49.782 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[35325]: debug 2022-01-31T22:56:49.711+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250269 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:49.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[51620]: debug 2022-01-31T22:56:49.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.510370+0000) 2022-01-31T22:56:49.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:49 smithi171 conmon[51620]: debug 2022-01-31T22:56:49.686+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.688130+0000) 2022-01-31T22:56:49.925 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:49 smithi167 conmon[49112]: debug 2022-01-31T22:56:49.687+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.689121+0000) 2022-01-31T22:56:49.926 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:49 smithi167 conmon[54076]: debug 2022-01-31T22:56:49.686+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.687531+0000) 2022-01-31T22:56:49.926 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:49 smithi167 conmon[60316]: debug 2022-01-31T22:56:49.686+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.688384+0000) 2022-01-31T22:56:49.927 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:49 smithi167 conmon[60316]: debug 2022-01-31T22:56:49.895+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.896000+0000) 2022-01-31T22:56:50.327 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:49 smithi167 conmon[49112]: debug 2022-01-31T22:56:49.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:49.967905+0000) 2022-01-31T22:56:50.328 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:50 smithi167 conmon[54076]: debug 2022-01-31T22:56:50.135+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:50.136655+0000) 2022-01-31T22:56:50.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:50 smithi171 conmon[41853]: debug 2022-01-31T22:56:50.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:50.333559+0000) 2022-01-31T22:56:50.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:50 smithi171 conmon[46715]: debug 2022-01-31T22:56:50.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:50.130427+0000) 2022-01-31T22:56:50.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:50 smithi171 conmon[51620]: debug 2022-01-31T22:56:50.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:50.510517+0000) 2022-01-31T22:56:51.121 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:50 smithi167 conmon[60316]: debug 2022-01-31T22:56:50.895+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:50.896148+0000) 2022-01-31T22:56:51.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:50 smithi167 conmon[49112]: debug 2022-01-31T22:56:50.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:50.968066+0000) 2022-01-31T22:56:51.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:51 smithi167 conmon[54076]: debug 2022-01-31T22:56:51.136+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:51.136861+0000) 2022-01-31T22:56:51.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:51 smithi171 conmon[46715]: debug 2022-01-31T22:56:51.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:51.130553+0000) 2022-01-31T22:56:51.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:51 smithi171 conmon[41853]: debug 2022-01-31T22:56:51.332+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:51.333712+0000) 2022-01-31T22:56:51.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:51 smithi171 conmon[51620]: debug 2022-01-31T22:56:51.509+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:51.510637+0000) 2022-01-31T22:56:52.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:51 smithi167 conmon[49112]: debug 2022-01-31T22:56:51.966+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:51.968242+0000) 2022-01-31T22:56:52.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:51 smithi167 conmon[60316]: debug 2022-01-31T22:56:51.894+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:51.896336+0000) 2022-01-31T22:56:52.373 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:52 smithi167 conmon[54076]: debug 2022-01-31T22:56:52.136+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:52.137035+0000) 2022-01-31T22:56:52.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:52 smithi171 conmon[46715]: debug 2022-01-31T22:56:52.129+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:52.130722+0000) 2022-01-31T22:56:52.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:52 smithi171 conmon[41853]: debug 2022-01-31T22:56:52.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:52.334449+0000) 2022-01-31T22:56:52.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:52 smithi171 conmon[51620]: debug 2022-01-31T22:56:52.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:52.510801+0000) 2022-01-31T22:56:53.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:52 smithi167 conmon[49112]: debug 2022-01-31T22:56:52.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:52.968387+0000) 2022-01-31T22:56:53.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:52 smithi167 conmon[60316]: debug 2022-01-31T22:56:52.895+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:52.896543+0000) 2022-01-31T22:56:53.377 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:53 smithi167 conmon[54076]: debug 2022-01-31T22:56:53.136+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:53.137223+0000) 2022-01-31T22:56:53.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:53 smithi171 conmon[41853]: debug 2022-01-31T22:56:53.333+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:53.334556+0000) 2022-01-31T22:56:53.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:53 smithi171 conmon[46715]: debug 2022-01-31T22:56:53.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:53.130929+0000) 2022-01-31T22:56:53.781 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:53 smithi171 conmon[51620]: debug 2022-01-31T22:56:53.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:53.510970+0000) 2022-01-31T22:56:54.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:53 smithi171 conmon[35325]: debug 2022-01-31T22:56:53.935+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:56:54.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:53 smithi167 conmon[49112]: debug 2022-01-31T22:56:53.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:53.968539+0000) 2022-01-31T22:56:54.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:53 smithi167 conmon[60316]: debug 2022-01-31T22:56:53.895+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:53.896672+0000) 2022-01-31T22:56:54.381 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:54 smithi167 conmon[54076]: debug 2022-01-31T22:56:54.136+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.137387+0000) 2022-01-31T22:56:54.495 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[46715]: debug 2022-01-31T22:56:54.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.131105+0000) 2022-01-31T22:56:54.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[41853]: debug 2022-01-31T22:56:54.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.334694+0000) 2022-01-31T22:56:54.782 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[35325]: debug 2022-01-31T22:56:54.741+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250382 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:56:54.783 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[41853]: debug 2022-01-31T22:56:54.714+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.715417+0000) 2022-01-31T22:56:54.783 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[46715]: debug 2022-01-31T22:56:54.714+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.715283+0000) 2022-01-31T22:56:54.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[51620]: debug 2022-01-31T22:56:54.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.511188+0000) 2022-01-31T22:56:54.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:54 smithi171 conmon[51620]: debug 2022-01-31T22:56:54.714+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.715589+0000) 2022-01-31T22:56:54.953 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:54 smithi167 conmon[49112]: debug 2022-01-31T22:56:54.714+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.716393+0000) 2022-01-31T22:56:54.954 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:54 smithi167 conmon[54076]: debug 2022-01-31T22:56:54.715+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.717037+0000) 2022-01-31T22:56:54.955 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:54 smithi167 conmon[60316]: debug 2022-01-31T22:56:54.714+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.716453+0000) 2022-01-31T22:56:54.955 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:54 smithi167 conmon[60316]: debug 2022-01-31T22:56:54.895+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.896800+0000) 2022-01-31T22:56:55.309 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:54 smithi167 conmon[49112]: debug 2022-01-31T22:56:54.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:54.968731+0000) 2022-01-31T22:56:55.309 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:55 smithi167 conmon[54076]: debug 2022-01-31T22:56:55.136+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:55.137584+0000) 2022-01-31T22:56:55.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:55 smithi171 conmon[41853]: debug 2022-01-31T22:56:55.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:55.334853+0000) 2022-01-31T22:56:55.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:55 smithi171 conmon[46715]: debug 2022-01-31T22:56:55.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:55.131267+0000) 2022-01-31T22:56:55.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:55 smithi171 conmon[51620]: debug 2022-01-31T22:56:55.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:55.511407+0000) 2022-01-31T22:56:56.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:55 smithi167 conmon[49112]: debug 2022-01-31T22:56:55.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:55.968888+0000) 2022-01-31T22:56:56.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:55 smithi167 conmon[60316]: debug 2022-01-31T22:56:55.896+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:55.896941+0000) 2022-01-31T22:56:56.387 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:56 smithi167 conmon[54076]: debug 2022-01-31T22:56:56.137+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:56.137731+0000) 2022-01-31T22:56:56.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:56 smithi171 conmon[41853]: debug 2022-01-31T22:56:56.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:56.335028+0000) 2022-01-31T22:56:56.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:56 smithi171 conmon[46715]: debug 2022-01-31T22:56:56.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:56.131382+0000) 2022-01-31T22:56:56.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:56 smithi171 conmon[51620]: debug 2022-01-31T22:56:56.510+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:56.511571+0000) 2022-01-31T22:56:57.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:56 smithi167 conmon[49112]: debug 2022-01-31T22:56:56.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:56.969045+0000) 2022-01-31T22:56:57.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:56 smithi167 conmon[60316]: debug 2022-01-31T22:56:56.895+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:56.897189+0000) 2022-01-31T22:56:57.391 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:57 smithi167 conmon[54076]: debug 2022-01-31T22:56:57.137+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:57.137912+0000) 2022-01-31T22:56:57.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:57 smithi171 conmon[41853]: debug 2022-01-31T22:56:57.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:57.335234+0000) 2022-01-31T22:56:57.496 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:57 smithi171 conmon[46715]: debug 2022-01-31T22:56:57.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:57.131547+0000) 2022-01-31T22:56:57.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:57 smithi171 conmon[51620]: debug 2022-01-31T22:56:57.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:57.511793+0000) 2022-01-31T22:56:58.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:57 smithi167 conmon[49112]: debug 2022-01-31T22:56:57.967+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:57.969248+0000) 2022-01-31T22:56:58.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:57 smithi167 conmon[60316]: debug 2022-01-31T22:56:57.896+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:57.897356+0000) 2022-01-31T22:56:58.395 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:58 smithi167 conmon[54076]: debug 2022-01-31T22:56:58.137+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:58.138070+0000) 2022-01-31T22:56:58.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:58 smithi171 conmon[41853]: debug 2022-01-31T22:56:58.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:58.335443+0000) 2022-01-31T22:56:58.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:58 smithi171 conmon[46715]: debug 2022-01-31T22:56:58.130+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:58.131765+0000) 2022-01-31T22:56:58.782 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:58 smithi171 conmon[51620]: debug 2022-01-31T22:56:58.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:58.511990+0000) 2022-01-31T22:56:59.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:58 smithi167 conmon[49112]: debug 2022-01-31T22:56:58.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:58.969411+0000) 2022-01-31T22:56:59.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:58 smithi167 conmon[60316]: debug 2022-01-31T22:56:58.896+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:58.897525+0000) 2022-01-31T22:56:59.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:59 smithi167 conmon[54076]: debug 2022-01-31T22:56:59.137+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.138259+0000) 2022-01-31T22:56:59.496 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[41853]: debug 2022-01-31T22:56:59.334+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.335649+0000) 2022-01-31T22:56:59.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[46715]: debug 2022-01-31T22:56:59.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.131954+0000) 2022-01-31T22:56:59.755 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[46715]: debug 2022-01-31T22:56:59.743+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.744493+0000) 2022-01-31T22:56:59.756 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[41853]: debug 2022-01-31T22:56:59.744+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.745294+0000) 2022-01-31T22:56:59.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[51620]: debug 2022-01-31T22:56:59.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.512181+0000) 2022-01-31T22:56:59.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[51620]: debug 2022-01-31T22:56:59.744+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.744857+0000) 2022-01-31T22:57:00.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:56:59 smithi171 conmon[35325]: debug 2022-01-31T22:56:59.770+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250492 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:00.123 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:56:59 smithi167 conmon[54076]: debug 2022-01-31T22:56:59.744+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.746324+0000) 2022-01-31T22:57:00.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:59 smithi167 conmon[60316]: debug 2022-01-31T22:56:59.744+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.745694+0000) 2022-01-31T22:57:00.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:56:59 smithi167 conmon[60316]: debug 2022-01-31T22:56:59.897+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.897655+0000) 2022-01-31T22:57:00.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:59 smithi167 conmon[49112]: debug 2022-01-31T22:56:59.743+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.745259+0000) 2022-01-31T22:57:00.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:56:59 smithi167 conmon[49112]: debug 2022-01-31T22:56:59.968+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:56:59.969600+0000) 2022-01-31T22:57:00.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:00 smithi167 conmon[54076]: debug 2022-01-31T22:57:00.136+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:00.138387+0000) 2022-01-31T22:57:00.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:00 smithi171 conmon[41853]: debug 2022-01-31T22:57:00.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:00.335820+0000) 2022-01-31T22:57:00.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:00 smithi171 conmon[46715]: debug 2022-01-31T22:57:00.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:00.132134+0000) 2022-01-31T22:57:00.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:00 smithi171 conmon[51620]: debug 2022-01-31T22:57:00.511+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:00.512400+0000) 2022-01-31T22:57:01.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:00 smithi167 conmon[49112]: debug 2022-01-31T22:57:00.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:00.969696+0000) 2022-01-31T22:57:01.123 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:00 smithi167 conmon[60316]: debug 2022-01-31T22:57:00.897+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:00.897794+0000) 2022-01-31T22:57:01.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:01 smithi167 conmon[54076]: debug 2022-01-31T22:57:01.137+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:01.138531+0000) 2022-01-31T22:57:01.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:01 smithi171 conmon[46715]: debug 2022-01-31T22:57:01.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:01.132304+0000) 2022-01-31T22:57:01.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:01 smithi171 conmon[41853]: debug 2022-01-31T22:57:01.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:01.335971+0000) 2022-01-31T22:57:01.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:01 smithi171 conmon[51620]: debug 2022-01-31T22:57:01.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:01.512564+0000) 2022-01-31T22:57:02.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:01 smithi167 conmon[49112]: debug 2022-01-31T22:57:01.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:01.969895+0000) 2022-01-31T22:57:02.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:01 smithi167 conmon[60316]: debug 2022-01-31T22:57:01.897+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:01.897980+0000) 2022-01-31T22:57:02.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:02 smithi167 conmon[54076]: debug 2022-01-31T22:57:02.138+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:02.138685+0000) 2022-01-31T22:57:02.497 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:02 smithi171 conmon[46715]: debug 2022-01-31T22:57:02.131+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:02.132506+0000) 2022-01-31T22:57:02.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:02 smithi171 conmon[41853]: debug 2022-01-31T22:57:02.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:02.336093+0000) 2022-01-31T22:57:02.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:02 smithi171 conmon[51620]: debug 2022-01-31T22:57:02.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:02.512777+0000) 2022-01-31T22:57:03.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:02 smithi167 conmon[49112]: debug 2022-01-31T22:57:02.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:02.970079+0000) 2022-01-31T22:57:03.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:02 smithi167 conmon[60316]: debug 2022-01-31T22:57:02.897+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:02.898141+0000) 2022-01-31T22:57:03.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:03 smithi167 conmon[54076]: debug 2022-01-31T22:57:03.138+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:03.138883+0000) 2022-01-31T22:57:03.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:03 smithi171 conmon[41853]: debug 2022-01-31T22:57:03.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:03.336262+0000) 2022-01-31T22:57:03.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:03 smithi171 conmon[46715]: debug 2022-01-31T22:57:03.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:03.132683+0000) 2022-01-31T22:57:03.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:03 smithi171 conmon[51620]: debug 2022-01-31T22:57:03.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:03.512987+0000) 2022-01-31T22:57:04.123 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:03 smithi167 conmon[49112]: debug 2022-01-31T22:57:03.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:03.970241+0000) 2022-01-31T22:57:04.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:03 smithi167 conmon[60316]: debug 2022-01-31T22:57:03.897+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:03.898350+0000) 2022-01-31T22:57:04.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:04 smithi167 conmon[54076]: debug 2022-01-31T22:57:04.138+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.139090+0000) 2022-01-31T22:57:04.497 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[41853]: debug 2022-01-31T22:57:04.335+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.336458+0000) 2022-01-31T22:57:04.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[46715]: debug 2022-01-31T22:57:04.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.132898+0000) 2022-01-31T22:57:04.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[51620]: debug 2022-01-31T22:57:04.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.513208+0000) 2022-01-31T22:57:05.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[35325]: debug 2022-01-31T22:57:04.798+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250605 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:05.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[41853]: debug 2022-01-31T22:57:04.773+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.774172+0000) 2022-01-31T22:57:05.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[46715]: debug 2022-01-31T22:57:04.772+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.773224+0000) 2022-01-31T22:57:05.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:04 smithi171 conmon[51620]: debug 2022-01-31T22:57:04.772+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.772942+0000) 2022-01-31T22:57:05.123 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:04 smithi167 conmon[54076]: debug 2022-01-31T22:57:04.773+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.774312+0000) 2022-01-31T22:57:05.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:04 smithi167 conmon[49112]: debug 2022-01-31T22:57:04.772+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.772901+0000) 2022-01-31T22:57:05.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:04 smithi167 conmon[49112]: debug 2022-01-31T22:57:04.969+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.970394+0000) 2022-01-31T22:57:05.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:04 smithi167 conmon[60316]: debug 2022-01-31T22:57:04.772+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.773223+0000) 2022-01-31T22:57:05.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:04 smithi167 conmon[60316]: debug 2022-01-31T22:57:04.898+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:04.898550+0000) 2022-01-31T22:57:05.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:05 smithi167 conmon[54076]: debug 2022-01-31T22:57:05.138+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:05.139256+0000) 2022-01-31T22:57:05.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:05 smithi171 conmon[46715]: debug 2022-01-31T22:57:05.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:05.133036+0000) 2022-01-31T22:57:05.498 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:05 smithi171 conmon[41853]: debug 2022-01-31T22:57:05.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:05.336636+0000) 2022-01-31T22:57:05.783 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:05 smithi171 conmon[51620]: debug 2022-01-31T22:57:05.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:05.513427+0000) 2022-01-31T22:57:06.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:05 smithi167 conmon[49112]: debug 2022-01-31T22:57:05.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:05.970571+0000) 2022-01-31T22:57:06.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:05 smithi167 conmon[60316]: debug 2022-01-31T22:57:05.898+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:05.898736+0000) 2022-01-31T22:57:06.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:06 smithi171 conmon[46715]: debug 2022-01-31T22:57:06.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:06.133121+0000) 2022-01-31T22:57:06.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:06 smithi167 conmon[54076]: debug 2022-01-31T22:57:06.138+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:06.139422+0000) 2022-01-31T22:57:06.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:06 smithi171 conmon[41853]: debug 2022-01-31T22:57:06.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:06.336778+0000) 2022-01-31T22:57:06.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:06 smithi171 conmon[51620]: debug 2022-01-31T22:57:06.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:06.513555+0000) 2022-01-31T22:57:07.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:06 smithi167 conmon[49112]: debug 2022-01-31T22:57:06.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:06.970685+0000) 2022-01-31T22:57:07.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:06 smithi167 conmon[60316]: debug 2022-01-31T22:57:06.898+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:06.898890+0000) 2022-01-31T22:57:07.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:07 smithi171 conmon[41853]: debug 2022-01-31T22:57:07.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:07.336913+0000) 2022-01-31T22:57:07.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:07 smithi171 conmon[46715]: debug 2022-01-31T22:57:07.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:07.133239+0000) 2022-01-31T22:57:07.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:07 smithi167 conmon[54076]: debug 2022-01-31T22:57:07.139+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:07.139555+0000) 2022-01-31T22:57:07.784 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:07 smithi171 conmon[51620]: debug 2022-01-31T22:57:07.512+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:07.513706+0000) 2022-01-31T22:57:08.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:07 smithi167 conmon[49112]: debug 2022-01-31T22:57:07.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:07.970890+0000) 2022-01-31T22:57:08.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:07 smithi167 conmon[60316]: debug 2022-01-31T22:57:07.898+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:07.899046+0000) 2022-01-31T22:57:08.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:08 smithi171 conmon[46715]: debug 2022-01-31T22:57:08.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:08.133383+0000) 2022-01-31T22:57:08.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:08 smithi167 conmon[54076]: debug 2022-01-31T22:57:08.139+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:08.139689+0000) 2022-01-31T22:57:08.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:08 smithi171 conmon[41853]: debug 2022-01-31T22:57:08.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:08.337071+0000) 2022-01-31T22:57:08.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:08 smithi171 conmon[51620]: debug 2022-01-31T22:57:08.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:08.513799+0000) 2022-01-31T22:57:09.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:08 smithi167 conmon[49112]: debug 2022-01-31T22:57:08.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:08.971084+0000) 2022-01-31T22:57:09.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:08 smithi167 conmon[60316]: debug 2022-01-31T22:57:08.898+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:08.899173+0000) 2022-01-31T22:57:09.322 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:08 smithi171 conmon[35325]: debug 2022-01-31T22:57:08.936+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:57:09.322 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[46715]: debug 2022-01-31T22:57:09.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.133520+0000) 2022-01-31T22:57:09.389 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:09 smithi167 conmon[54076]: debug 2022-01-31T22:57:09.139+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.139843+0000) 2022-01-31T22:57:09.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[41853]: debug 2022-01-31T22:57:09.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.337237+0000) 2022-01-31T22:57:09.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[51620]: debug 2022-01-31T22:57:09.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.513936+0000) 2022-01-31T22:57:10.085 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[35325]: debug 2022-01-31T22:57:09.823+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250715 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:10.086 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[41853]: debug 2022-01-31T22:57:09.800+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.801125+0000) 2022-01-31T22:57:10.086 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[46715]: debug 2022-01-31T22:57:09.799+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.800709+0000) 2022-01-31T22:57:10.087 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:09 smithi171 conmon[51620]: debug 2022-01-31T22:57:09.799+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.800618+0000) 2022-01-31T22:57:10.124 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:09 smithi167 conmon[49112]: debug 2022-01-31T22:57:09.801+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.801740+0000) 2022-01-31T22:57:10.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:09 smithi167 conmon[49112]: debug 2022-01-31T22:57:09.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.971301+0000) 2022-01-31T22:57:10.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:09 smithi167 conmon[54076]: debug 2022-01-31T22:57:09.801+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.801646+0000) 2022-01-31T22:57:10.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:09 smithi167 conmon[60316]: debug 2022-01-31T22:57:09.800+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.801302+0000) 2022-01-31T22:57:10.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:09 smithi167 conmon[60316]: debug 2022-01-31T22:57:09.898+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:09.899351+0000) 2022-01-31T22:57:10.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:10 smithi171 conmon[41853]: debug 2022-01-31T22:57:10.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:10.337326+0000) 2022-01-31T22:57:10.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:10 smithi171 conmon[46715]: debug 2022-01-31T22:57:10.132+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:10.133687+0000) 2022-01-31T22:57:10.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:10 smithi167 conmon[54076]: debug 2022-01-31T22:57:10.139+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:10.140006+0000) 2022-01-31T22:57:10.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:10 smithi171 conmon[51620]: debug 2022-01-31T22:57:10.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:10.514102+0000) 2022-01-31T22:57:11.124 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:10 smithi167 conmon[60316]: debug 2022-01-31T22:57:10.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:10.899486+0000) 2022-01-31T22:57:11.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:10 smithi167 conmon[49112]: debug 2022-01-31T22:57:10.970+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:10.971444+0000) 2022-01-31T22:57:11.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:11 smithi167 conmon[54076]: debug 2022-01-31T22:57:11.139+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:11.140195+0000) 2022-01-31T22:57:11.498 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:11 smithi171 conmon[46715]: debug 2022-01-31T22:57:11.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:11.133813+0000) 2022-01-31T22:57:11.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:11 smithi171 conmon[41853]: debug 2022-01-31T22:57:11.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:11.337489+0000) 2022-01-31T22:57:11.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:11 smithi171 conmon[51620]: debug 2022-01-31T22:57:11.513+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:11.514253+0000) 2022-01-31T22:57:12.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:11 smithi167 conmon[49112]: debug 2022-01-31T22:57:11.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:11.971648+0000) 2022-01-31T22:57:12.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:11 smithi167 conmon[60316]: debug 2022-01-31T22:57:11.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:11.899666+0000) 2022-01-31T22:57:12.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:12 smithi167 conmon[54076]: debug 2022-01-31T22:57:12.140+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:12.140388+0000) 2022-01-31T22:57:12.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:12 smithi171 conmon[46715]: debug 2022-01-31T22:57:12.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:12.133955+0000) 2022-01-31T22:57:12.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:12 smithi171 conmon[41853]: debug 2022-01-31T22:57:12.336+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:12.337690+0000) 2022-01-31T22:57:12.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:12 smithi171 conmon[51620]: debug 2022-01-31T22:57:12.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:12.514434+0000) 2022-01-31T22:57:13.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:12 smithi167 conmon[49112]: debug 2022-01-31T22:57:12.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:12.971822+0000) 2022-01-31T22:57:13.125 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:12 smithi167 conmon[60316]: debug 2022-01-31T22:57:12.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:12.899825+0000) 2022-01-31T22:57:13.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:13 smithi167 conmon[54076]: debug 2022-01-31T22:57:13.140+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:13.140569+0000) 2022-01-31T22:57:13.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:13 smithi171 conmon[41853]: debug 2022-01-31T22:57:13.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:13.337830+0000) 2022-01-31T22:57:13.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:13 smithi171 conmon[46715]: debug 2022-01-31T22:57:13.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:13.134124+0000) 2022-01-31T22:57:13.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:13 smithi171 conmon[51620]: debug 2022-01-31T22:57:13.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:13.514635+0000) 2022-01-31T22:57:14.125 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:13 smithi167 conmon[49112]: debug 2022-01-31T22:57:13.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:13.971969+0000) 2022-01-31T22:57:14.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:13 smithi167 conmon[60316]: debug 2022-01-31T22:57:13.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:13.899985+0000) 2022-01-31T22:57:14.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:14 smithi167 conmon[54076]: debug 2022-01-31T22:57:14.140+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.140749+0000) 2022-01-31T22:57:14.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[46715]: debug 2022-01-31T22:57:14.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.134307+0000) 2022-01-31T22:57:14.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[41853]: debug 2022-01-31T22:57:14.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.338023+0000) 2022-01-31T22:57:14.811 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[51620]: debug 2022-01-31T22:57:14.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.514849+0000) 2022-01-31T22:57:14.812 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[41853]: debug 2022-01-31T22:57:14.825+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.826738+0000) 2022-01-31T22:57:15.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[35325]: debug 2022-01-31T22:57:14.852+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250829 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:15.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[46715]: debug 2022-01-31T22:57:14.825+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.826362+0000) 2022-01-31T22:57:15.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:14 smithi171 conmon[51620]: debug 2022-01-31T22:57:14.825+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.826509+0000) 2022-01-31T22:57:15.125 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:14 smithi167 conmon[54076]: debug 2022-01-31T22:57:14.826+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.827388+0000) 2022-01-31T22:57:15.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:14 smithi167 conmon[49112]: debug 2022-01-31T22:57:14.826+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.826571+0000) 2022-01-31T22:57:15.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:14 smithi167 conmon[49112]: debug 2022-01-31T22:57:14.971+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.972147+0000) 2022-01-31T22:57:15.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:14 smithi167 conmon[60316]: debug 2022-01-31T22:57:14.826+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.827189+0000) 2022-01-31T22:57:15.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:14 smithi167 conmon[60316]: debug 2022-01-31T22:57:14.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:14.900113+0000) 2022-01-31T22:57:15.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:15 smithi167 conmon[54076]: debug 2022-01-31T22:57:15.140+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:15.140952+0000) 2022-01-31T22:57:15.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:15 smithi171 conmon[46715]: debug 2022-01-31T22:57:15.133+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:15.134495+0000) 2022-01-31T22:57:15.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:15 smithi171 conmon[41853]: debug 2022-01-31T22:57:15.337+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:15.338209+0000) 2022-01-31T22:57:15.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:15 smithi171 conmon[51620]: debug 2022-01-31T22:57:15.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:15.515025+0000) 2022-01-31T22:57:16.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:15 smithi167 conmon[49112]: debug 2022-01-31T22:57:15.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:15.972316+0000) 2022-01-31T22:57:16.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:15 smithi167 conmon[60316]: debug 2022-01-31T22:57:15.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:15.900259+0000) 2022-01-31T22:57:16.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:16 smithi167 conmon[54076]: debug 2022-01-31T22:57:16.140+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:16.141111+0000) 2022-01-31T22:57:16.499 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:16 smithi171 conmon[46715]: debug 2022-01-31T22:57:16.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:16.134627+0000) 2022-01-31T22:57:16.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:16 smithi171 conmon[41853]: debug 2022-01-31T22:57:16.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:16.338400+0000) 2022-01-31T22:57:16.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:16 smithi171 conmon[51620]: debug 2022-01-31T22:57:16.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:16.515157+0000) 2022-01-31T22:57:17.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:16 smithi167 conmon[49112]: debug 2022-01-31T22:57:16.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:16.972519+0000) 2022-01-31T22:57:17.126 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:16 smithi167 conmon[60316]: debug 2022-01-31T22:57:16.899+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:16.900449+0000) 2022-01-31T22:57:17.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:17 smithi167 conmon[54076]: debug 2022-01-31T22:57:17.141+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:17.141296+0000) 2022-01-31T22:57:17.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:17 smithi171 conmon[41853]: debug 2022-01-31T22:57:17.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:17.338617+0000) 2022-01-31T22:57:17.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:17 smithi171 conmon[46715]: debug 2022-01-31T22:57:17.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:17.134852+0000) 2022-01-31T22:57:17.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:17 smithi171 conmon[51620]: debug 2022-01-31T22:57:17.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:17.515314+0000) 2022-01-31T22:57:18.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:17 smithi167 conmon[49112]: debug 2022-01-31T22:57:17.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:17.972676+0000) 2022-01-31T22:57:18.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:17 smithi167 conmon[60316]: debug 2022-01-31T22:57:17.900+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:17.900647+0000) 2022-01-31T22:57:18.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:18 smithi167 conmon[54076]: debug 2022-01-31T22:57:18.141+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:18.141520+0000) 2022-01-31T22:57:18.499 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:18 smithi171 conmon[41853]: debug 2022-01-31T22:57:18.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:18.338742+0000) 2022-01-31T22:57:18.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:18 smithi171 conmon[46715]: debug 2022-01-31T22:57:18.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:18.135034+0000) 2022-01-31T22:57:18.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:18 smithi171 conmon[51620]: debug 2022-01-31T22:57:18.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:18.515517+0000) 2022-01-31T22:57:19.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:18 smithi167 conmon[49112]: debug 2022-01-31T22:57:18.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:18.972794+0000) 2022-01-31T22:57:19.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:18 smithi167 conmon[60316]: debug 2022-01-31T22:57:18.900+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:18.900745+0000) 2022-01-31T22:57:19.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:19 smithi167 conmon[54076]: debug 2022-01-31T22:57:19.141+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.141706+0000) 2022-01-31T22:57:19.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[46715]: debug 2022-01-31T22:57:19.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.135242+0000) 2022-01-31T22:57:19.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[41853]: debug 2022-01-31T22:57:19.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.338897+0000) 2022-01-31T22:57:19.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[51620]: debug 2022-01-31T22:57:19.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.515693+0000) 2022-01-31T22:57:20.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[35325]: debug 2022-01-31T22:57:19.880+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 250939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:20.093 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[41853]: debug 2022-01-31T22:57:19.855+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.856519+0000) 2022-01-31T22:57:20.093 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[46715]: debug 2022-01-31T22:57:19.855+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.856447+0000) 2022-01-31T22:57:20.094 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:19 smithi171 conmon[51620]: debug 2022-01-31T22:57:19.853+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.854557+0000) 2022-01-31T22:57:20.126 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:19 smithi167 conmon[54076]: debug 2022-01-31T22:57:19.855+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.855948+0000) 2022-01-31T22:57:20.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:19 smithi167 conmon[49112]: debug 2022-01-31T22:57:19.854+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.855245+0000) 2022-01-31T22:57:20.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:19 smithi167 conmon[49112]: debug 2022-01-31T22:57:19.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.972992+0000) 2022-01-31T22:57:20.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:19 smithi167 conmon[60316]: debug 2022-01-31T22:57:19.856+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.856538+0000) 2022-01-31T22:57:20.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:19 smithi167 conmon[60316]: debug 2022-01-31T22:57:19.900+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:19.900950+0000) 2022-01-31T22:57:20.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:20 smithi167 conmon[54076]: debug 2022-01-31T22:57:20.141+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:20.141875+0000) 2022-01-31T22:57:20.500 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:20 smithi171 conmon[41853]: debug 2022-01-31T22:57:20.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:20.339116+0000) 2022-01-31T22:57:20.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:20 smithi171 conmon[46715]: debug 2022-01-31T22:57:20.134+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:20.135424+0000) 2022-01-31T22:57:20.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:20 smithi171 conmon[51620]: debug 2022-01-31T22:57:20.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:20.515919+0000) 2022-01-31T22:57:21.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:20 smithi167 conmon[49112]: debug 2022-01-31T22:57:20.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:20.973171+0000) 2022-01-31T22:57:21.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:20 smithi167 conmon[60316]: debug 2022-01-31T22:57:20.900+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:20.901120+0000) 2022-01-31T22:57:21.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:21 smithi167 conmon[54076]: debug 2022-01-31T22:57:21.141+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:21.142072+0000) 2022-01-31T22:57:21.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:21 smithi171 conmon[46715]: debug 2022-01-31T22:57:21.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:21.135576+0000) 2022-01-31T22:57:21.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:21 smithi171 conmon[41853]: debug 2022-01-31T22:57:21.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:21.339273+0000) 2022-01-31T22:57:21.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:21 smithi171 conmon[51620]: debug 2022-01-31T22:57:21.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:21.516084+0000) 2022-01-31T22:57:22.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:21 smithi167 conmon[49112]: debug 2022-01-31T22:57:21.972+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:21.973396+0000) 2022-01-31T22:57:22.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:21 smithi167 conmon[60316]: debug 2022-01-31T22:57:21.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:21.901309+0000) 2022-01-31T22:57:22.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:22 smithi167 conmon[54076]: debug 2022-01-31T22:57:22.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:22.142257+0000) 2022-01-31T22:57:22.500 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:22 smithi171 conmon[46715]: debug 2022-01-31T22:57:22.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:22.135765+0000) 2022-01-31T22:57:22.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:22 smithi171 conmon[41853]: debug 2022-01-31T22:57:22.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:22.339460+0000) 2022-01-31T22:57:22.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:22 smithi171 conmon[51620]: debug 2022-01-31T22:57:22.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:22.516223+0000) 2022-01-31T22:57:23.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:22 smithi167 conmon[49112]: debug 2022-01-31T22:57:22.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:22.973598+0000) 2022-01-31T22:57:23.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:22 smithi167 conmon[60316]: debug 2022-01-31T22:57:22.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:22.901531+0000) 2022-01-31T22:57:23.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:23 smithi167 conmon[54076]: debug 2022-01-31T22:57:23.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:23.142462+0000) 2022-01-31T22:57:23.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:23 smithi171 conmon[41853]: debug 2022-01-31T22:57:23.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:23.339686+0000) 2022-01-31T22:57:23.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:23 smithi171 conmon[46715]: debug 2022-01-31T22:57:23.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:23.135912+0000) 2022-01-31T22:57:23.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:23 smithi171 conmon[51620]: debug 2022-01-31T22:57:23.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:23.516380+0000) 2022-01-31T22:57:24.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:23 smithi167 conmon[49112]: debug 2022-01-31T22:57:23.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:23.973856+0000) 2022-01-31T22:57:24.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:23 smithi167 conmon[60316]: debug 2022-01-31T22:57:23.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:23.901706+0000) 2022-01-31T22:57:24.324 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:23 smithi171 conmon[35325]: debug 2022-01-31T22:57:23.937+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:57:24.326 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[46715]: debug 2022-01-31T22:57:24.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.136107+0000) 2022-01-31T22:57:24.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:24 smithi167 conmon[54076]: debug 2022-01-31T22:57:24.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.142710+0000) 2022-01-31T22:57:24.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[51620]: debug 2022-01-31T22:57:24.514+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.516548+0000) 2022-01-31T22:57:24.593 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[41853]: debug 2022-01-31T22:57:24.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.339900+0000) 2022-01-31T22:57:25.121 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[35325]: debug 2022-01-31T22:57:24.922+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251053 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:25.121 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[41853]: debug 2022-01-31T22:57:24.882+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.884484+0000) 2022-01-31T22:57:25.122 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[46715]: debug 2022-01-31T22:57:24.882+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.884531+0000) 2022-01-31T22:57:25.123 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:24 smithi171 conmon[51620]: debug 2022-01-31T22:57:24.881+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.883189+0000) 2022-01-31T22:57:25.127 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:24 smithi167 conmon[54076]: debug 2022-01-31T22:57:24.882+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.883787+0000) 2022-01-31T22:57:25.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:24 smithi167 conmon[49112]: debug 2022-01-31T22:57:24.882+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.883597+0000) 2022-01-31T22:57:25.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:24 smithi167 conmon[49112]: debug 2022-01-31T22:57:24.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.974059+0000) 2022-01-31T22:57:25.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:24 smithi167 conmon[60316]: debug 2022-01-31T22:57:24.882+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.884596+0000) 2022-01-31T22:57:25.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:24 smithi167 conmon[60316]: debug 2022-01-31T22:57:24.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:24.901924+0000) 2022-01-31T22:57:25.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:25 smithi167 conmon[54076]: debug 2022-01-31T22:57:25.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:25.142910+0000) 2022-01-31T22:57:25.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:25 smithi171 conmon[41853]: debug 2022-01-31T22:57:25.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:25.340063+0000) 2022-01-31T22:57:25.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:25 smithi171 conmon[46715]: debug 2022-01-31T22:57:25.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:25.136314+0000) 2022-01-31T22:57:25.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:25 smithi171 conmon[51620]: debug 2022-01-31T22:57:25.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:25.516651+0000) 2022-01-31T22:57:26.127 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:25 smithi167 conmon[49112]: debug 2022-01-31T22:57:25.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:25.974187+0000) 2022-01-31T22:57:26.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:25 smithi167 conmon[60316]: debug 2022-01-31T22:57:25.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:25.902086+0000) 2022-01-31T22:57:26.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:26 smithi167 conmon[54076]: debug 2022-01-31T22:57:26.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:26.143063+0000) 2022-01-31T22:57:26.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:26 smithi171 conmon[41853]: debug 2022-01-31T22:57:26.338+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:26.340141+0000) 2022-01-31T22:57:26.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:26 smithi171 conmon[46715]: debug 2022-01-31T22:57:26.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:26.136464+0000) 2022-01-31T22:57:26.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:26 smithi171 conmon[51620]: debug 2022-01-31T22:57:26.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:26.516794+0000) 2022-01-31T22:57:27.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:26 smithi167 conmon[60316]: debug 2022-01-31T22:57:26.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:26.902253+0000) 2022-01-31T22:57:27.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:26 smithi167 conmon[49112]: debug 2022-01-31T22:57:26.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:26.974351+0000) 2022-01-31T22:57:27.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:27 smithi167 conmon[54076]: debug 2022-01-31T22:57:27.143+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:27.143248+0000) 2022-01-31T22:57:27.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:27 smithi171 conmon[41853]: debug 2022-01-31T22:57:27.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:27.340308+0000) 2022-01-31T22:57:27.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:27 smithi171 conmon[46715]: debug 2022-01-31T22:57:27.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:27.136648+0000) 2022-01-31T22:57:27.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:27 smithi171 conmon[51620]: debug 2022-01-31T22:57:27.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:27.516935+0000) 2022-01-31T22:57:28.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:27 smithi167 conmon[60316]: debug 2022-01-31T22:57:27.900+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:27.902404+0000) 2022-01-31T22:57:28.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:27 smithi167 conmon[49112]: debug 2022-01-31T22:57:27.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:27.974556+0000) 2022-01-31T22:57:28.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:28 smithi167 conmon[54076]: debug 2022-01-31T22:57:28.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:28.143432+0000) 2022-01-31T22:57:28.501 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:28 smithi171 conmon[41853]: debug 2022-01-31T22:57:28.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:28.340495+0000) 2022-01-31T22:57:28.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:28 smithi171 conmon[46715]: debug 2022-01-31T22:57:28.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:28.136824+0000) 2022-01-31T22:57:28.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:28 smithi171 conmon[51620]: debug 2022-01-31T22:57:28.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:28.517061+0000) 2022-01-31T22:57:29.128 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:28 smithi167 conmon[60316]: debug 2022-01-31T22:57:28.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:28.902609+0000) 2022-01-31T22:57:29.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:28 smithi167 conmon[49112]: debug 2022-01-31T22:57:28.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:28.974690+0000) 2022-01-31T22:57:29.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:29 smithi167 conmon[54076]: debug 2022-01-31T22:57:29.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.143619+0000) 2022-01-31T22:57:29.501 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[46715]: debug 2022-01-31T22:57:29.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.137005+0000) 2022-01-31T22:57:29.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[41853]: debug 2022-01-31T22:57:29.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.340698+0000) 2022-01-31T22:57:29.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[51620]: debug 2022-01-31T22:57:29.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.517246+0000) 2022-01-31T22:57:30.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:29 smithi167 conmon[49112]: debug 2022-01-31T22:57:29.925+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.927286+0000) 2022-01-31T22:57:30.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:29 smithi167 conmon[49112]: debug 2022-01-31T22:57:29.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.974854+0000) 2022-01-31T22:57:30.130 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:29 smithi167 conmon[54076]: debug 2022-01-31T22:57:29.926+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.928275+0000) 2022-01-31T22:57:30.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:29 smithi167 conmon[60316]: debug 2022-01-31T22:57:29.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.902791+0000) 2022-01-31T22:57:30.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:29 smithi167 conmon[60316]: debug 2022-01-31T22:57:29.926+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.927602+0000) 2022-01-31T22:57:30.325 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[35325]: debug 2022-01-31T22:57:29.950+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:30.326 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[41853]: debug 2022-01-31T22:57:29.925+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.927321+0000) 2022-01-31T22:57:30.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[46715]: debug 2022-01-31T22:57:29.925+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.927193+0000) 2022-01-31T22:57:30.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:30 smithi171 conmon[46715]: debug 2022-01-31T22:57:30.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:30.137172+0000) 2022-01-31T22:57:30.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[51620]: debug 2022-01-31T22:57:30.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:29 smithi171 conmon[51620]: 2022-01-31T22:57:29.925+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:29.926926+0000) 2022-01-31T22:57:30.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:30 smithi167 conmon[54076]: debug 2022-01-31T22:57:30.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:30.143778+0000) 2022-01-31T22:57:30.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:30 smithi171 conmon[41853]: debug 2022-01-31T22:57:30.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:30.340906+0000) 2022-01-31T22:57:30.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:30 smithi171 conmon[51620]: debug 2022-01-31T22:57:30.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:30.517441+0000) 2022-01-31T22:57:31.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:30 smithi167 conmon[49112]: debug 2022-01-31T22:57:30.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:30.975023+0000) 2022-01-31T22:57:31.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:30 smithi167 conmon[60316]: debug 2022-01-31T22:57:30.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:30.902957+0000) 2022-01-31T22:57:31.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:31 smithi167 conmon[54076]: debug 2022-01-31T22:57:31.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:31.143980+0000) 2022-01-31T22:57:31.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:31 smithi171 conmon[41853]: debug 2022-01-31T22:57:31.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:31.341059+0000) 2022-01-31T22:57:31.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:31 smithi171 conmon[46715]: debug 2022-01-31T22:57:31.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:31.137357+0000) 2022-01-31T22:57:31.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:31 smithi171 conmon[51620]: debug 2022-01-31T22:57:31.515+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:31.517615+0000) 2022-01-31T22:57:32.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:31 smithi167 conmon[49112]: debug 2022-01-31T22:57:31.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:31.975222+0000) 2022-01-31T22:57:32.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:31 smithi167 conmon[60316]: debug 2022-01-31T22:57:31.901+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:31.903120+0000) 2022-01-31T22:57:32.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:32 smithi167 conmon[54076]: debug 2022-01-31T22:57:32.143+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:32.144166+0000) 2022-01-31T22:57:32.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:32 smithi171 conmon[41853]: debug 2022-01-31T22:57:32.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:32.341267+0000) 2022-01-31T22:57:32.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:32 smithi171 conmon[46715]: debug 2022-01-31T22:57:32.136+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:32.137535+0000) 2022-01-31T22:57:32.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:32 smithi171 conmon[51620]: debug 2022-01-31T22:57:32.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:32.517786+0000) 2022-01-31T22:57:33.128 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:32 smithi167 conmon[49112]: debug 2022-01-31T22:57:32.973+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:32.975433+0000) 2022-01-31T22:57:33.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:32 smithi167 conmon[60316]: debug 2022-01-31T22:57:32.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:32.903316+0000) 2022-01-31T22:57:33.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:33 smithi167 conmon[54076]: debug 2022-01-31T22:57:33.142+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:33.144348+0000) 2022-01-31T22:57:33.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:33 smithi171 conmon[46715]: debug 2022-01-31T22:57:33.135+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:33.137717+0000) 2022-01-31T22:57:33.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:33 smithi171 conmon[41853]: debug 2022-01-31T22:57:33.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:33.341440+0000) 2022-01-31T22:57:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:33 smithi171 conmon[51620]: debug 2022-01-31T22:57:33.516+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:33.517897+0000) 2022-01-31T22:57:34.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:33 smithi167 conmon[49112]: debug 2022-01-31T22:57:33.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:33.975530+0000) 2022-01-31T22:57:34.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:33 smithi167 conmon[60316]: debug 2022-01-31T22:57:33.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:33.903458+0000) 2022-01-31T22:57:34.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:34 smithi167 conmon[54076]: debug 2022-01-31T22:57:34.143+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.144533+0000) 2022-01-31T22:57:34.502 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[41853]: debug 2022-01-31T22:57:34.339+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.341567+0000) 2022-01-31T22:57:34.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[46715]: debug 2022-01-31T22:57:34.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.137915+0000) 2022-01-31T22:57:34.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[51620]: debug 2022-01-31T22:57:34.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.518063+0000) 2022-01-31T22:57:35.129 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:34 smithi167 conmon[54076]: debug 2022-01-31T22:57:34.954+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.955707+0000) 2022-01-31T22:57:35.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:34 smithi167 conmon[49112]: debug 2022-01-31T22:57:34.954+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.955976+0000) 2022-01-31T22:57:35.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:34 smithi167 conmon[49112]: debug 2022-01-31T22:57:34.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.975673+0000) 2022-01-31T22:57:35.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:34 smithi167 conmon[60316]: debug 2022-01-31T22:57:34.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.903686+0000) 2022-01-31T22:57:35.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:34 smithi167 conmon[60316]: debug 2022-01-31T22:57:34.954+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.955880+0000) 2022-01-31T22:57:35.326 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[35325]: debug 2022-01-31T22:57:34.979+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251276 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:35.327 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[41853]: debug 2022-01-31T22:57:34.952+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.954508+0000) 2022-01-31T22:57:35.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[46715]: debug 2022-01-31T22:57:34.954+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.956428+0000) 2022-01-31T22:57:35.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:35 smithi171 conmon[46715]: debug 2022-01-31T22:57:35.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:35.138107+0000) 2022-01-31T22:57:35.328 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:34 smithi171 conmon[51620]: debug 2022-01-31T22:57:34.953+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:34.955419+0000) 2022-01-31T22:57:35.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:35 smithi167 conmon[54076]: debug 2022-01-31T22:57:35.143+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:35.144754+0000) 2022-01-31T22:57:35.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:35 smithi171 conmon[41853]: debug 2022-01-31T22:57:35.340+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:35.341738+0000) 2022-01-31T22:57:35.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:35 smithi171 conmon[51620]: debug 2022-01-31T22:57:35.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:35.518260+0000) 2022-01-31T22:57:36.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:35 smithi167 conmon[49112]: debug 2022-01-31T22:57:35.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:35.975842+0000) 2022-01-31T22:57:36.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:35 smithi167 conmon[60316]: debug 2022-01-31T22:57:35.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:35.903871+0000) 2022-01-31T22:57:36.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:36 smithi167 conmon[54076]: debug 2022-01-31T22:57:36.144+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:36.144916+0000) 2022-01-31T22:57:36.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:36 smithi171 conmon[46715]: debug 2022-01-31T22:57:36.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:36.138203+0000) 2022-01-31T22:57:36.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:36 smithi171 conmon[41853]: debug 2022-01-31T22:57:36.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:36.341932+0000) 2022-01-31T22:57:36.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:36 smithi171 conmon[51620]: debug 2022-01-31T22:57:36.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:36.518375+0000) 2022-01-31T22:57:37.129 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:36 smithi167 conmon[60316]: debug 2022-01-31T22:57:36.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:36.904068+0000) 2022-01-31T22:57:37.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:36 smithi167 conmon[49112]: debug 2022-01-31T22:57:36.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:36.976047+0000) 2022-01-31T22:57:37.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:37 smithi167 conmon[54076]: debug 2022-01-31T22:57:37.144+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:37.145106+0000) 2022-01-31T22:57:37.502 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:37 smithi171 conmon[46715]: debug 2022-01-31T22:57:37.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:37.138376+0000) 2022-01-31T22:57:37.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:37 smithi171 conmon[41853]: debug 2022-01-31T22:57:37.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:37.342121+0000) 2022-01-31T22:57:37.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:37 smithi171 conmon[51620]: debug 2022-01-31T22:57:37.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:37.518494+0000) 2022-01-31T22:57:38.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:37 smithi167 conmon[49112]: debug 2022-01-31T22:57:37.975+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:37.976229+0000) 2022-01-31T22:57:38.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:37 smithi167 conmon[60316]: debug 2022-01-31T22:57:37.903+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:37.904227+0000) 2022-01-31T22:57:38.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:38 smithi167 conmon[54076]: debug 2022-01-31T22:57:38.144+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:38.145294+0000) 2022-01-31T22:57:38.503 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:38 smithi171 conmon[46715]: debug 2022-01-31T22:57:38.137+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:38.138581+0000) 2022-01-31T22:57:38.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:38 smithi171 conmon[41853]: debug 2022-01-31T22:57:38.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:38.342367+0000) 2022-01-31T22:57:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:38 smithi171 conmon[51620]: debug 2022-01-31T22:57:38.517+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:38.518704+0000) 2022-01-31T22:57:39.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:38 smithi167 conmon[49112]: debug 2022-01-31T22:57:38.974+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:38.976464+0000) 2022-01-31T22:57:39.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:38 smithi167 conmon[60316]: debug 2022-01-31T22:57:38.902+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:38.904436+0000) 2022-01-31T22:57:39.327 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:39 smithi171 conmon[46715]: debug 2022-01-31T22:57:39.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.138767+0000) 2022-01-31T22:57:39.328 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:38 smithi171 conmon[35325]: debug 2022-01-31T22:57:38.937+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:57:39.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:39 smithi167 conmon[54076]: debug 2022-01-31T22:57:39.144+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.145476+0000) 2022-01-31T22:57:39.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:39 smithi171 conmon[41853]: debug 2022-01-31T22:57:39.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.342534+0000) 2022-01-31T22:57:39.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:39 smithi171 conmon[51620]: debug 2022-01-31T22:57:39.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.518895+0000) 2022-01-31T22:57:40.130 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:39 smithi167 conmon[54076]: debug 2022-01-31T22:57:39.982+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.984363+0000) 2022-01-31T22:57:40.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:39 smithi167 conmon[49112]: debug 2022-01-31T22:57:39.975+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.976712+0000) 2022-01-31T22:57:40.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:39 smithi167 conmon[49112]: debug 2022-01-31T22:57:39.982+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.983882+0000) 2022-01-31T22:57:40.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:39 smithi167 conmon[60316]: debug 2022-01-31T22:57:39.903+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.904652+0000) 2022-01-31T22:57:40.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:39 smithi167 conmon[60316]: debug 2022-01-31T22:57:39.982+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.984068+0000) 2022-01-31T22:57:40.327 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:40 smithi171 conmon[35325]: debug 2022-01-31T22:57:40.008+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251385 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:40.328 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:39 smithi171 conmon[41853]: debug 2022-01-31T22:57:39.982+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.983586+0000) 2022-01-31T22:57:40.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:39 smithi171 conmon[46715]: debug 2022-01-31T22:57:39.983+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.983835+0000) 2022-01-31T22:57:40.329 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:40 smithi171 conmon[46715]: debug 2022-01-31T22:57:40.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:40.138883+0000) 2022-01-31T22:57:40.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:39 smithi171 conmon[51620]: debug 2022-01-31T22:57:39.983+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:39.984619+0000) 2022-01-31T22:57:40.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:40 smithi167 conmon[54076]: debug 2022-01-31T22:57:40.144+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:40.145686+0000) 2022-01-31T22:57:40.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:40 smithi171 conmon[41853]: debug 2022-01-31T22:57:40.341+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:40.342677+0000) 2022-01-31T22:57:40.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:40 smithi171 conmon[51620]: debug 2022-01-31T22:57:40.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:40.519019+0000) 2022-01-31T22:57:41.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:40 smithi167 conmon[49112]: debug 2022-01-31T22:57:40.976+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:40.976925+0000) 2022-01-31T22:57:41.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:40 smithi167 conmon[60316]: debug 2022-01-31T22:57:40.904+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:40.904818+0000) 2022-01-31T22:57:41.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:41 smithi167 conmon[54076]: debug 2022-01-31T22:57:41.145+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:41.145871+0000) 2022-01-31T22:57:41.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:41 smithi171 conmon[41853]: debug 2022-01-31T22:57:41.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:41.342828+0000) 2022-01-31T22:57:41.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:41 smithi171 conmon[46715]: debug 2022-01-31T22:57:41.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:41.139037+0000) 2022-01-31T22:57:41.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:41 smithi171 conmon[51620]: debug 2022-01-31T22:57:41.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:41.519120+0000) 2022-01-31T22:57:42.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:41 smithi167 conmon[49112]: debug 2022-01-31T22:57:41.976+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:41.977077+0000) 2022-01-31T22:57:42.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:41 smithi167 conmon[60316]: debug 2022-01-31T22:57:41.904+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:41.904978+0000) 2022-01-31T22:57:42.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:42 smithi167 conmon[54076]: debug 2022-01-31T22:57:42.145+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:42.146072+0000) 2022-01-31T22:57:42.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:42 smithi171 conmon[46715]: debug 2022-01-31T22:57:42.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:42.139236+0000) 2022-01-31T22:57:42.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:42 smithi171 conmon[41853]: debug 2022-01-31T22:57:42.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:42.342955+0000) 2022-01-31T22:57:42.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:42 smithi171 conmon[51620]: debug 2022-01-31T22:57:42.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:42.519318+0000) 2022-01-31T22:57:43.130 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:42 smithi167 conmon[49112]: debug 2022-01-31T22:57:42.976+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:42.977260+0000) 2022-01-31T22:57:43.131 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:42 smithi167 conmon[60316]: debug 2022-01-31T22:57:42.903+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:42.905124+0000) 2022-01-31T22:57:43.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:43 smithi167 conmon[54076]: debug 2022-01-31T22:57:43.145+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:43.146281+0000) 2022-01-31T22:57:43.503 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:43 smithi171 conmon[41853]: debug 2022-01-31T22:57:43.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:43.343135+0000) 2022-01-31T22:57:43.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:43 smithi171 conmon[46715]: debug 2022-01-31T22:57:43.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:43.139413+0000) 2022-01-31T22:57:43.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:43 smithi171 conmon[51620]: debug 2022-01-31T22:57:43.518+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:43.519492+0000) 2022-01-31T22:57:44.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:43 smithi167 conmon[49112]: debug 2022-01-31T22:57:43.975+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:43.977395+0000) 2022-01-31T22:57:44.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:43 smithi167 conmon[60316]: debug 2022-01-31T22:57:43.904+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:43.905331+0000) 2022-01-31T22:57:44.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:44 smithi167 conmon[54076]: debug 2022-01-31T22:57:44.145+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:44.146470+0000) 2022-01-31T22:57:44.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:44 smithi171 conmon[41853]: debug 2022-01-31T22:57:44.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:44.343309+0000) 2022-01-31T22:57:44.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:44 smithi171 conmon[46715]: debug 2022-01-31T22:57:44.138+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:44.139584+0000) 2022-01-31T22:57:44.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:44 smithi171 conmon[51620]: debug 2022-01-31T22:57:44.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:44.519661+0000) 2022-01-31T22:57:45.131 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:45 smithi167 conmon[54076]: debug 2022-01-31T22:57:45.011+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.012589+0000) 2022-01-31T22:57:45.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:44 smithi167 conmon[49112]: debug 2022-01-31T22:57:44.976+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:44.977564+0000) 2022-01-31T22:57:45.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:45 smithi167 conmon[49112]: debug 2022-01-31T22:57:45.010+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.012378+0000) 2022-01-31T22:57:45.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:44 smithi167 conmon[60316]: debug 2022-01-31T22:57:44.904+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:44.905556+0000) 2022-01-31T22:57:45.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:45 smithi167 conmon[60316]: debug 2022-01-31T22:57:45.010+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.012115+0000) 2022-01-31T22:57:45.328 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[35325]: debug 2022-01-31T22:57:45.037+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251499 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:45.328 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[41853]: debug 2022-01-31T22:57:45.010+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.011591+0000) 2022-01-31T22:57:45.329 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[51620]: debug 2022-01-31T22:57:45.012+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.013245+0000) 2022-01-31T22:57:45.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[46715]: debug 2022-01-31T22:57:45.011+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.012388+0000) 2022-01-31T22:57:45.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[46715]: debug 2022-01-31T22:57:45.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.139789+0000) 2022-01-31T22:57:45.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:45 smithi167 conmon[54076]: debug 2022-01-31T22:57:45.146+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.146712+0000) 2022-01-31T22:57:45.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[41853]: debug 2022-01-31T22:57:45.342+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.343519+0000) 2022-01-31T22:57:45.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:45 smithi171 conmon[51620]: debug 2022-01-31T22:57:45.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.519868+0000) 2022-01-31T22:57:46.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:45 smithi167 conmon[49112]: debug 2022-01-31T22:57:45.977+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.977755+0000) 2022-01-31T22:57:46.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:45 smithi167 conmon[60316]: debug 2022-01-31T22:57:45.905+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:45.905747+0000) 2022-01-31T22:57:46.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:46 smithi167 conmon[54076]: debug 2022-01-31T22:57:46.146+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:46.146883+0000) 2022-01-31T22:57:46.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:46 smithi171 conmon[46715]: debug 2022-01-31T22:57:46.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:46.139980+0000) 2022-01-31T22:57:46.504 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:46 smithi171 conmon[41853]: debug 2022-01-31T22:57:46.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:46.343661+0000) 2022-01-31T22:57:46.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:46 smithi171 conmon[51620]: debug 2022-01-31T22:57:46.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:46.520009+0000) 2022-01-31T22:57:47.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:46 smithi167 conmon[49112]: debug 2022-01-31T22:57:46.977+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:46.977913+0000) 2022-01-31T22:57:47.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:46 smithi167 conmon[60316]: debug 2022-01-31T22:57:46.905+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:46.905904+0000) 2022-01-31T22:57:47.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:47 smithi167 conmon[54076]: debug 2022-01-31T22:57:47.146+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:47.147049+0000) 2022-01-31T22:57:47.504 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:47 smithi171 conmon[46715]: debug 2022-01-31T22:57:47.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:47.140104+0000) 2022-01-31T22:57:47.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:47 smithi171 conmon[41853]: debug 2022-01-31T22:57:47.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:47.343834+0000) 2022-01-31T22:57:47.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:47 smithi171 conmon[51620]: debug 2022-01-31T22:57:47.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:47.520177+0000) 2022-01-31T22:57:48.131 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:47 smithi167 conmon[49112]: debug 2022-01-31T22:57:47.977+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:47.978135+0000) 2022-01-31T22:57:48.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:47 smithi167 conmon[60316]: debug 2022-01-31T22:57:47.904+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:47.906066+0000) 2022-01-31T22:57:48.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:48 smithi167 conmon[54076]: debug 2022-01-31T22:57:48.146+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:48.147239+0000) 2022-01-31T22:57:48.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:48 smithi171 conmon[46715]: debug 2022-01-31T22:57:48.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:48.140242+0000) 2022-01-31T22:57:48.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:48 smithi171 conmon[41853]: debug 2022-01-31T22:57:48.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:48.344029+0000) 2022-01-31T22:57:48.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:48 smithi171 conmon[51620]: debug 2022-01-31T22:57:48.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:48.520354+0000) 2022-01-31T22:57:49.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:48 smithi167 conmon[49112]: debug 2022-01-31T22:57:48.977+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:48.978341+0000) 2022-01-31T22:57:49.132 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:48 smithi167 conmon[60316]: debug 2022-01-31T22:57:48.904+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:48.906272+0000) 2022-01-31T22:57:49.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:49 smithi167 conmon[54076]: debug 2022-01-31T22:57:49.145+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:49.147444+0000) 2022-01-31T22:57:49.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:49 smithi171 conmon[41853]: debug 2022-01-31T22:57:49.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:49.344233+0000) 2022-01-31T22:57:49.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:49 smithi171 conmon[46715]: debug 2022-01-31T22:57:49.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:49.140433+0000) 2022-01-31T22:57:49.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:49 smithi171 conmon[51620]: debug 2022-01-31T22:57:49.519+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:49.520509+0000) 2022-01-31T22:57:50.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:49 smithi167 conmon[49112]: debug 2022-01-31T22:57:49.977+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:49.978541+0000) 2022-01-31T22:57:50.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:50 smithi167 conmon[49112]: debug 2022-01-31T22:57:50.039+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.040672+0000) 2022-01-31T22:57:50.133 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:50 smithi167 conmon[54076]: debug 2022-01-31T22:57:50.039+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.041417+0000) 2022-01-31T22:57:50.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:49 smithi167 conmon[60316]: debug 2022-01-31T22:57:49.905+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:49.906511+0000) 2022-01-31T22:57:50.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:50 smithi167 conmon[60316]: debug 2022-01-31T22:57:50.039+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.040885+0000) 2022-01-31T22:57:50.329 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[35325]: debug 2022-01-31T22:57:50.064+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251610 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:50.329 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[41853]: debug 2022-01-31T22:57:50.039+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.040034+0000) 2022-01-31T22:57:50.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[46715]: debug 2022-01-31T22:57:50.040+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.040932+0000) 2022-01-31T22:57:50.330 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[46715]: debug 2022-01-31T22:57:50.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.140584+0000) 2022-01-31T22:57:50.331 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[51620]: debug 2022-01-31T22:57:50.040+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.041238+0000) 2022-01-31T22:57:50.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:50 smithi167 conmon[54076]: debug 2022-01-31T22:57:50.147+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.147596+0000) 2022-01-31T22:57:50.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[41853]: debug 2022-01-31T22:57:50.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.344417+0000) 2022-01-31T22:57:50.593 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:50 smithi171 conmon[51620]: debug 2022-01-31T22:57:50.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.520679+0000) 2022-01-31T22:57:51.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:50 smithi167 conmon[49112]: debug 2022-01-31T22:57:50.978+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.978732+0000) 2022-01-31T22:57:51.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:50 smithi167 conmon[60316]: debug 2022-01-31T22:57:50.906+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:50.906697+0000) 2022-01-31T22:57:51.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:51 smithi167 conmon[54076]: debug 2022-01-31T22:57:51.147+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:51.147800+0000) 2022-01-31T22:57:51.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:51 smithi171 conmon[41853]: debug 2022-01-31T22:57:51.343+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:51.344569+0000) 2022-01-31T22:57:51.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:51 smithi171 conmon[46715]: debug 2022-01-31T22:57:51.139+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:51.140718+0000) 2022-01-31T22:57:51.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:51 smithi171 conmon[51620]: debug 2022-01-31T22:57:51.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:51.520797+0000) 2022-01-31T22:57:52.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:51 smithi167 conmon[49112]: debug 2022-01-31T22:57:51.978+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:51.978879+0000) 2022-01-31T22:57:52.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:51 smithi167 conmon[60316]: debug 2022-01-31T22:57:51.906+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:51.906872+0000) 2022-01-31T22:57:52.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:52 smithi167 conmon[54076]: debug 2022-01-31T22:57:52.147+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:52.147995+0000) 2022-01-31T22:57:52.434 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:52 smithi171 conmon[46715]: debug 2022-01-31T22:57:52.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:52.140908+0000) 2022-01-31T22:57:52.434 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:52 smithi171 conmon[41853]: debug 2022-01-31T22:57:52.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:52.344789+0000) 2022-01-31T22:57:52.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:52 smithi171 conmon[51620]: debug 2022-01-31T22:57:52.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:52.520957+0000) 2022-01-31T22:57:53.132 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:52 smithi167 conmon[49112]: debug 2022-01-31T22:57:52.978+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:52.979078+0000) 2022-01-31T22:57:53.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:52 smithi167 conmon[60316]: debug 2022-01-31T22:57:52.906+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:52.907033+0000) 2022-01-31T22:57:53.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:53 smithi167 conmon[54076]: debug 2022-01-31T22:57:53.147+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:53.148195+0000) 2022-01-31T22:57:53.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:53 smithi171 conmon[46715]: debug 2022-01-31T22:57:53.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:53.141108+0000) 2022-01-31T22:57:53.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:53 smithi171 conmon[41853]: debug 2022-01-31T22:57:53.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:53.344919+0000) 2022-01-31T22:57:53.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:53 smithi171 conmon[51620]: debug 2022-01-31T22:57:53.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:53.521143+0000) 2022-01-31T22:57:54.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:53 smithi167 conmon[49112]: debug 2022-01-31T22:57:53.978+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:53.979207+0000) 2022-01-31T22:57:54.133 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:53 smithi167 conmon[60316]: debug 2022-01-31T22:57:53.906+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:53.907199+0000) 2022-01-31T22:57:54.291 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:54 smithi171 conmon[46715]: debug 2022-01-31T22:57:54.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:54.141284+0000) 2022-01-31T22:57:54.291 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:53 smithi171 conmon[35325]: debug 2022-01-31T22:57:53.938+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:57:54.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:54 smithi167 conmon[54076]: debug 2022-01-31T22:57:54.147+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:54.148405+0000) 2022-01-31T22:57:54.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:54 smithi171 conmon[41853]: debug 2022-01-31T22:57:54.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:54.345078+0000) 2022-01-31T22:57:54.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:54 smithi171 conmon[51620]: debug 2022-01-31T22:57:54.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:54.521327+0000) 2022-01-31T22:57:55.052 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:54 smithi167 conmon[49112]: debug 2022-01-31T22:57:54.978+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:54.979388+0000) 2022-01-31T22:57:55.053 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:54 smithi167 conmon[60316]: debug 2022-01-31T22:57:54.906+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:54.907388+0000) 2022-01-31T22:57:55.330 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[35325]: debug 2022-01-31T22:57:55.093+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251724 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:57:55.330 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[41853]: debug 2022-01-31T22:57:55.067+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.068243+0000) 2022-01-31T22:57:55.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[46715]: debug 2022-01-31T22:57:55.067+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.067999+0000) 2022-01-31T22:57:55.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[46715]: debug 2022-01-31T22:57:55.140+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.141470+0000) 2022-01-31T22:57:55.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[51620]: debug 2022-01-31T22:57:55.067+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.068610+0000) 2022-01-31T22:57:55.339 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:55 smithi167 conmon[49112]: debug 2022-01-31T22:57:55.067+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.068330+0000) 2022-01-31T22:57:55.339 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:55 smithi167 conmon[60316]: debug 2022-01-31T22:57:55.067+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.068417+0000) 2022-01-31T22:57:55.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:55 smithi167 conmon[54076]: debug 2022-01-31T22:57:55.068+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.069485+0000) 2022-01-31T22:57:55.340 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:55 smithi167 conmon[54076]: debug 2022-01-31T22:57:55.148+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.148574+0000) 2022-01-31T22:57:55.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[41853]: debug 2022-01-31T22:57:55.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.345260+0000) 2022-01-31T22:57:55.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:55 smithi171 conmon[51620]: debug 2022-01-31T22:57:55.520+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.521504+0000) 2022-01-31T22:57:56.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:55 smithi167 conmon[49112]: debug 2022-01-31T22:57:55.979+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.979592+0000) 2022-01-31T22:57:56.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:55 smithi167 conmon[60316]: debug 2022-01-31T22:57:55.907+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:55.907520+0000) 2022-01-31T22:57:56.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:56 smithi167 conmon[54076]: debug 2022-01-31T22:57:56.148+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:56.148721+0000) 2022-01-31T22:57:56.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:56 smithi171 conmon[46715]: debug 2022-01-31T22:57:56.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:56.141617+0000) 2022-01-31T22:57:56.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:56 smithi171 conmon[41853]: debug 2022-01-31T22:57:56.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:56.345400+0000) 2022-01-31T22:57:56.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:56 smithi171 conmon[51620]: debug 2022-01-31T22:57:56.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:56.521678+0000) 2022-01-31T22:57:57.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:56 smithi167 conmon[49112]: debug 2022-01-31T22:57:56.979+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:56.979744+0000) 2022-01-31T22:57:57.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:56 smithi167 conmon[60316]: debug 2022-01-31T22:57:56.907+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:56.907664+0000) 2022-01-31T22:57:57.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:57 smithi167 conmon[54076]: debug 2022-01-31T22:57:57.148+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:57.148904+0000) 2022-01-31T22:57:57.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:57 smithi171 conmon[46715]: debug 2022-01-31T22:57:57.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:57.141798+0000) 2022-01-31T22:57:57.506 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:57 smithi171 conmon[41853]: debug 2022-01-31T22:57:57.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:57.345580+0000) 2022-01-31T22:57:57.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:57 smithi171 conmon[51620]: debug 2022-01-31T22:57:57.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:57.521868+0000) 2022-01-31T22:57:58.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:57 smithi167 conmon[49112]: debug 2022-01-31T22:57:57.979+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:57.979973+0000) 2022-01-31T22:57:58.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:57 smithi167 conmon[60316]: debug 2022-01-31T22:57:57.907+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:57.907850+0000) 2022-01-31T22:57:58.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:58 smithi167 conmon[54076]: debug 2022-01-31T22:57:58.148+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:58.149081+0000) 2022-01-31T22:57:58.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:58 smithi171 conmon[46715]: debug 2022-01-31T22:57:58.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:58.141964+0000) 2022-01-31T22:57:58.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:58 smithi171 conmon[41853]: debug 2022-01-31T22:57:58.344+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:58.345707+0000) 2022-01-31T22:57:58.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:58 smithi171 conmon[51620]: debug 2022-01-31T22:57:58.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:58.522053+0000) 2022-01-31T22:57:59.133 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:58 smithi167 conmon[49112]: debug 2022-01-31T22:57:58.979+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:58.980193+0000) 2022-01-31T22:57:59.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:58 smithi167 conmon[60316]: debug 2022-01-31T22:57:58.907+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:58.908007+0000) 2022-01-31T22:57:59.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:57:59 smithi167 conmon[54076]: debug 2022-01-31T22:57:59.148+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:59.149221+0000) 2022-01-31T22:57:59.506 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:57:59 smithi171 conmon[46715]: debug 2022-01-31T22:57:59.141+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:59.142114+0000) 2022-01-31T22:57:59.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:57:59 smithi171 conmon[41853]: debug 2022-01-31T22:57:59.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:59.345865+0000) 2022-01-31T22:57:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:57:59 smithi171 conmon[51620]: debug 2022-01-31T22:57:59.521+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:59.522238+0000) 2022-01-31T22:58:00.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:57:59 smithi167 conmon[49112]: debug 2022-01-31T22:57:59.979+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:59.980383+0000) 2022-01-31T22:58:00.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[49112]: debug 2022-01-31T22:58:00.096+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.097466+0000) 2022-01-31T22:58:00.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:57:59 smithi167 conmon[60316]: debug 2022-01-31T22:57:59.907+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:57:59.908165+0000) 2022-01-31T22:58:00.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[60316]: debug 2022-01-31T22:58:00.095+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.096077+0000) 2022-01-31T22:58:00.156 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[54076]: debug 2022-01-31T22:58:00.097+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.097627+0000) 2022-01-31T22:58:00.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[54076]: 2022-01-31T22:58:00.157 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[54076]: debug 2022-01-31T22:58:00.149+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.149378+0000) 2022-01-31T22:58:00.330 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[35325]: debug 2022-01-31T22:58:00.121+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251825 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:00.331 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[41853]: debug 2022-01-31T22:58:00.096+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.097332+0000) 2022-01-31T22:58:00.332 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[51620]: debug 2022-01-31T22:58:00.097+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.097880+0000) 2022-01-31T22:58:00.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[46715]: debug 2022-01-31T22:58:00.095+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.096001+0000) 2022-01-31T22:58:00.332 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[46715]: debug 2022-01-31T22:58:00.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.142272+0000) 2022-01-31T22:58:00.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[41853]: debug 2022-01-31T22:58:00.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.346013+0000) 2022-01-31T22:58:00.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:00 smithi171 conmon[51620]: debug 2022-01-31T22:58:00.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.522429+0000) 2022-01-31T22:58:01.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[49112]: debug 2022-01-31T22:58:00.980+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.980556+0000) 2022-01-31T22:58:01.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:00 smithi167 conmon[60316]: debug 2022-01-31T22:58:00.908+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:00.908391+0000) 2022-01-31T22:58:01.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:01 smithi167 conmon[54076]: debug 2022-01-31T22:58:01.149+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:01.149553+0000) 2022-01-31T22:58:01.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:01 smithi171 conmon[46715]: debug 2022-01-31T22:58:01.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:01.142412+0000) 2022-01-31T22:58:01.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:01 smithi171 conmon[41853]: debug 2022-01-31T22:58:01.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:01.346108+0000) 2022-01-31T22:58:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:01 smithi171 conmon[51620]: debug 2022-01-31T22:58:01.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:01.522581+0000) 2022-01-31T22:58:02.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:01 smithi167 conmon[49112]: debug 2022-01-31T22:58:01.980+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:01.980705+0000) 2022-01-31T22:58:02.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:01 smithi167 conmon[60316]: debug 2022-01-31T22:58:01.908+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:01.908562+0000) 2022-01-31T22:58:02.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:02 smithi167 conmon[54076]: debug 2022-01-31T22:58:02.149+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:02.149707+0000) 2022-01-31T22:58:02.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:02 smithi171 conmon[46715]: debug 2022-01-31T22:58:02.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:02.142608+0000) 2022-01-31T22:58:02.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:02 smithi171 conmon[41853]: debug 2022-01-31T22:58:02.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:02.346249+0000) 2022-01-31T22:58:02.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:02 smithi171 conmon[51620]: debug 2022-01-31T22:58:02.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:02.522769+0000) 2022-01-31T22:58:03.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:03 smithi167 conmon[54076]: debug 2022-01-31T22:58:03.149+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:03.149898+0000) 2022-01-31T22:58:03.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:02 smithi167 conmon[60316]: debug 2022-01-31T22:58:02.908+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:02.908747+0000) 2022-01-31T22:58:03.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:02 smithi167 conmon[49112]: debug 2022-01-31T22:58:02.980+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:02.980860+0000) 2022-01-31T22:58:03.507 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:03 smithi171 conmon[41853]: debug 2022-01-31T22:58:03.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:03.346408+0000) 2022-01-31T22:58:03.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:03 smithi171 conmon[46715]: debug 2022-01-31T22:58:03.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:03.142784+0000) 2022-01-31T22:58:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:03 smithi171 conmon[51620]: debug 2022-01-31T22:58:03.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:03.522942+0000) 2022-01-31T22:58:04.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:03 smithi167 conmon[49112]: debug 2022-01-31T22:58:03.980+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:03.981068+0000) 2022-01-31T22:58:04.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:04 smithi167 conmon[54076]: debug 2022-01-31T22:58:04.149+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:04.150104+0000) 2022-01-31T22:58:04.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:03 smithi167 conmon[60316]: debug 2022-01-31T22:58:03.908+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:03.908921+0000) 2022-01-31T22:58:04.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:04 smithi171 conmon[46715]: debug 2022-01-31T22:58:04.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:04.142939+0000) 2022-01-31T22:58:04.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:04 smithi171 conmon[41853]: debug 2022-01-31T22:58:04.345+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:04.346598+0000) 2022-01-31T22:58:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:04 smithi171 conmon[51620]: debug 2022-01-31T22:58:04.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:04.523140+0000) 2022-01-31T22:58:05.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:04 smithi167 conmon[49112]: debug 2022-01-31T22:58:04.980+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:04.981240+0000) 2022-01-31T22:58:05.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:05 smithi167 conmon[49112]: debug 2022-01-31T22:58:05.124+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.124703+0000) 2022-01-31T22:58:05.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:05 smithi167 conmon[54076]: debug 2022-01-31T22:58:05.125+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.125696+0000) 2022-01-31T22:58:05.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:05 smithi167 conmon[54076]: debug 2022-01-31T22:58:05.149+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.150328+0000) 2022-01-31T22:58:05.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:04 smithi167 conmon[60316]: debug 2022-01-31T22:58:04.908+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:04.909135+0000) 2022-01-31T22:58:05.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:05 smithi167 conmon[60316]: debug 2022-01-31T22:58:05.124+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.125104+0000) 2022-01-31T22:58:05.507 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[35325]: debug 2022-01-31T22:58:05.151+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 251939 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:05.508 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[51620]: debug 2022-01-31T22:58:05.124+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.125566+0000) 2022-01-31T22:58:05.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[41853]: debug 2022-01-31T22:58:05.124+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.125700+0000) 2022-01-31T22:58:05.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[41853]: debug 2022-01-31T22:58:05.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.346749+0000) 2022-01-31T22:58:05.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[46715]: debug 2022-01-31T22:58:05.123+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.124405+0000) 2022-01-31T22:58:05.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[46715]: debug 2022-01-31T22:58:05.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.143060+0000) 2022-01-31T22:58:05.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:05 smithi171 conmon[51620]: debug 2022-01-31T22:58:05.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.523368+0000) 2022-01-31T22:58:06.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:05 smithi167 conmon[49112]: debug 2022-01-31T22:58:05.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.981396+0000) 2022-01-31T22:58:06.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:06 smithi167 conmon[54076]: debug 2022-01-31T22:58:06.150+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:06.150471+0000) 2022-01-31T22:58:06.160 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:05 smithi167 conmon[60316]: debug 2022-01-31T22:58:05.909+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:05.909319+0000) 2022-01-31T22:58:06.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:06 smithi171 conmon[46715]: debug 2022-01-31T22:58:06.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:06.143230+0000) 2022-01-31T22:58:06.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:06 smithi171 conmon[41853]: debug 2022-01-31T22:58:06.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:06.346900+0000) 2022-01-31T22:58:06.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:06 smithi171 conmon[51620]: debug 2022-01-31T22:58:06.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:06.523539+0000) 2022-01-31T22:58:07.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:06 smithi167 conmon[49112]: debug 2022-01-31T22:58:06.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:06.981561+0000) 2022-01-31T22:58:07.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:06 smithi167 conmon[60316]: debug 2022-01-31T22:58:06.909+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:06.909458+0000) 2022-01-31T22:58:07.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:07 smithi167 conmon[54076]: debug 2022-01-31T22:58:07.150+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:07.150647+0000) 2022-01-31T22:58:07.507 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:07 smithi171 conmon[46715]: debug 2022-01-31T22:58:07.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:07.143410+0000) 2022-01-31T22:58:07.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:07 smithi171 conmon[41853]: debug 2022-01-31T22:58:07.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:07.347105+0000) 2022-01-31T22:58:07.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:07 smithi171 conmon[51620]: debug 2022-01-31T22:58:07.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:07.523725+0000) 2022-01-31T22:58:07.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:07 smithi167 conmon[60316]: debug 2022-01-31T22:58:07.909+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:07.909655+0000) 2022-01-31T22:58:08.341 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:07 smithi167 conmon[49112]: debug 2022-01-31T22:58:07.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:07.981709+0000) 2022-01-31T22:58:08.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:08 smithi167 conmon[54076]: debug 2022-01-31T22:58:08.150+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:08.150811+0000) 2022-01-31T22:58:08.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:08 smithi171 conmon[41853]: debug 2022-01-31T22:58:08.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:08.347313+0000) 2022-01-31T22:58:08.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:08 smithi171 conmon[46715]: debug 2022-01-31T22:58:08.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:08.143589+0000) 2022-01-31T22:58:08.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:08 smithi171 conmon[51620]: debug 2022-01-31T22:58:08.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:08.523935+0000) 2022-01-31T22:58:08.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:08 smithi167 conmon[60316]: debug 2022-01-31T22:58:08.909+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:08.909807+0000) 2022-01-31T22:58:09.331 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:09 smithi171 conmon[46715]: debug 2022-01-31T22:58:09.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:09.143746+0000) 2022-01-31T22:58:09.332 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:08 smithi171 conmon[35325]: debug 2022-01-31T22:58:08.939+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:58:09.341 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:08 smithi167 conmon[49112]: debug 2022-01-31T22:58:08.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:08.981845+0000) 2022-01-31T22:58:09.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:09 smithi167 conmon[54076]: debug 2022-01-31T22:58:09.150+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:09.150968+0000) 2022-01-31T22:58:09.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:09 smithi171 conmon[41853]: debug 2022-01-31T22:58:09.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:09.347483+0000) 2022-01-31T22:58:09.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:09 smithi171 conmon[51620]: debug 2022-01-31T22:58:09.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:09.524071+0000) 2022-01-31T22:58:09.939 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:09 smithi167 conmon[60316]: debug 2022-01-31T22:58:09.909+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:09.910001+0000) 2022-01-31T22:58:10.317 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:09 smithi167 conmon[49112]: debug 2022-01-31T22:58:09.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:09.982067+0000) 2022-01-31T22:58:10.318 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:10 smithi167 conmon[49112]: debug 2022-01-31T22:58:10.153+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.153782+0000) 2022-01-31T22:58:10.318 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:10 smithi167 conmon[54076]: debug 2022-01-31T22:58:10.150+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.151093+0000) 2022-01-31T22:58:10.319 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:10 smithi167 conmon[54076]: debug 2022-01-31T22:58:10.154+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.154901+0000) 2022-01-31T22:58:10.319 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:10 smithi167 conmon[60316]: debug 2022-01-31T22:58:10.153+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.154069+0000) 2022-01-31T22:58:10.508 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[35325]: debug 2022-01-31T22:58:10.179+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252049 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:10.509 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[51620]: debug 2022-01-31T22:58:10.153+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.154478+0000) 2022-01-31T22:58:10.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[41853]: debug 2022-01-31T22:58:10.153+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.154675+0000) 2022-01-31T22:58:10.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[41853]: debug 2022-01-31T22:58:10.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.347613+0000) 2022-01-31T22:58:10.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[46715]: debug 2022-01-31T22:58:10.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.143930+0000) 2022-01-31T22:58:10.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[46715]: debug 2022-01-31T22:58:10.152+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.153293+0000) 2022-01-31T22:58:10.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:10 smithi171 conmon[51620]: debug 2022-01-31T22:58:10.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.524301+0000) 2022-01-31T22:58:10.939 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:10 smithi167 conmon[60316]: debug 2022-01-31T22:58:10.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.910213+0000) 2022-01-31T22:58:11.342 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:10 smithi167 conmon[49112]: debug 2022-01-31T22:58:10.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:10.982271+0000) 2022-01-31T22:58:11.342 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:11 smithi167 conmon[54076]: debug 2022-01-31T22:58:11.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:11.151298+0000) 2022-01-31T22:58:11.508 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:11 smithi171 conmon[46715]: debug 2022-01-31T22:58:11.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:11.144108+0000) 2022-01-31T22:58:11.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:11 smithi171 conmon[41853]: debug 2022-01-31T22:58:11.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:11.347756+0000) 2022-01-31T22:58:11.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:11 smithi171 conmon[51620]: debug 2022-01-31T22:58:11.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:11.524458+0000) 2022-01-31T22:58:11.939 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:11 smithi167 conmon[60316]: debug 2022-01-31T22:58:11.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:11.910387+0000) 2022-01-31T22:58:12.342 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:11 smithi167 conmon[49112]: debug 2022-01-31T22:58:11.981+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:11.982412+0000) 2022-01-31T22:58:12.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:12 smithi167 conmon[54076]: debug 2022-01-31T22:58:12.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:12.151452+0000) 2022-01-31T22:58:12.508 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:12 smithi171 conmon[41853]: debug 2022-01-31T22:58:12.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:12.347892+0000) 2022-01-31T22:58:12.509 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:12 smithi171 conmon[46715]: debug 2022-01-31T22:58:12.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:12.144282+0000) 2022-01-31T22:58:12.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:12 smithi171 conmon[51620]: debug 2022-01-31T22:58:12.522+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:12.524672+0000) 2022-01-31T22:58:12.939 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:12 smithi167 conmon[60316]: debug 2022-01-31T22:58:12.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:12.910560+0000) 2022-01-31T22:58:13.342 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:12 smithi167 conmon[49112]: debug 2022-01-31T22:58:12.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:12.982570+0000) 2022-01-31T22:58:13.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:13 smithi167 conmon[54076]: debug 2022-01-31T22:58:13.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:13.151662+0000) 2022-01-31T22:58:13.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:13 smithi171 conmon[41853]: debug 2022-01-31T22:58:13.346+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:13.347989+0000) 2022-01-31T22:58:13.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:13 smithi171 conmon[46715]: debug 2022-01-31T22:58:13.142+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:13.144553+0000) 2022-01-31T22:58:13.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:13 smithi171 conmon[51620]: debug 2022-01-31T22:58:13.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:13.524834+0000) 2022-01-31T22:58:13.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:13 smithi167 conmon[60316]: debug 2022-01-31T22:58:13.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:13.910680+0000) 2022-01-31T22:58:14.342 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:13 smithi167 conmon[49112]: debug 2022-01-31T22:58:13.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:13.982697+0000) 2022-01-31T22:58:14.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:14 smithi167 conmon[54076]: debug 2022-01-31T22:58:14.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:14.151849+0000) 2022-01-31T22:58:14.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:14 smithi171 conmon[41853]: debug 2022-01-31T22:58:14.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:14.348155+0000) 2022-01-31T22:58:14.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:14 smithi171 conmon[46715]: debug 2022-01-31T22:58:14.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:14.144699+0000) 2022-01-31T22:58:14.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:14 smithi171 conmon[51620]: debug 2022-01-31T22:58:14.523+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:14.525012+0000) 2022-01-31T22:58:14.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:14 smithi167 conmon[60316]: debug 2022-01-31T22:58:14.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:14.910880+0000) 2022-01-31T22:58:15.343 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:14 smithi167 conmon[49112]: debug 2022-01-31T22:58:14.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:14.982873+0000) 2022-01-31T22:58:15.343 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:15 smithi167 conmon[49112]: debug 2022-01-31T22:58:15.181+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.182050+0000) 2022-01-31T22:58:15.344 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:15 smithi167 conmon[60316]: debug 2022-01-31T22:58:15.182+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.182709+0000) 2022-01-31T22:58:15.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:15 smithi167 conmon[54076]: debug 2022-01-31T22:58:15.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.152032+0000) 2022-01-31T22:58:15.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:15 smithi167 conmon[54076]: debug 2022-01-31T22:58:15.181+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.182324+0000) 2022-01-31T22:58:15.509 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[35325]: debug 2022-01-31T22:58:15.207+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:15.510 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[51620]: debug 2022-01-31T22:58:15.181+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.183163+0000) 2022-01-31T22:58:15.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[41853]: debug 2022-01-31T22:58:15.179+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.181740+0000) 2022-01-31T22:58:15.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[41853]: debug 2022-01-31T22:58:15.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.348347+0000) 2022-01-31T22:58:15.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[46715]: debug 2022-01-31T22:58:15.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.144856+0000) 2022-01-31T22:58:15.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[46715]: debug 2022-01-31T22:58:15.180+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.181859+0000) 2022-01-31T22:58:15.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:15 smithi171 conmon[51620]: debug 2022-01-31T22:58:15.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.525194+0000) 2022-01-31T22:58:15.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:15 smithi167 conmon[60316]: debug 2022-01-31T22:58:15.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.910991+0000) 2022-01-31T22:58:16.343 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:15 smithi167 conmon[49112]: debug 2022-01-31T22:58:15.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:15.983059+0000) 2022-01-31T22:58:16.343 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:16 smithi167 conmon[54076]: debug 2022-01-31T22:58:16.152+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:16.152218+0000) 2022-01-31T22:58:16.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:16 smithi171 conmon[41853]: debug 2022-01-31T22:58:16.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:16.348486+0000) 2022-01-31T22:58:16.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:16 smithi171 conmon[46715]: debug 2022-01-31T22:58:16.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:16.145019+0000) 2022-01-31T22:58:16.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:16 smithi171 conmon[51620]: debug 2022-01-31T22:58:16.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:16.525346+0000) 2022-01-31T22:58:16.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:16 smithi167 conmon[60316]: debug 2022-01-31T22:58:16.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:16.911143+0000) 2022-01-31T22:58:17.343 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:16 smithi167 conmon[49112]: debug 2022-01-31T22:58:16.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:16.983263+0000) 2022-01-31T22:58:17.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:17 smithi167 conmon[54076]: debug 2022-01-31T22:58:17.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:17.152378+0000) 2022-01-31T22:58:17.509 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:17 smithi171 conmon[41853]: debug 2022-01-31T22:58:17.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:17.348670+0000) 2022-01-31T22:58:17.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:17 smithi171 conmon[46715]: debug 2022-01-31T22:58:17.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:17.145177+0000) 2022-01-31T22:58:17.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:17 smithi171 conmon[51620]: debug 2022-01-31T22:58:17.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:17.525534+0000) 2022-01-31T22:58:17.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:17 smithi167 conmon[60316]: debug 2022-01-31T22:58:17.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:17.911328+0000) 2022-01-31T22:58:18.343 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:17 smithi167 conmon[49112]: debug 2022-01-31T22:58:17.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:17.983446+0000) 2022-01-31T22:58:18.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:18 smithi167 conmon[54076]: debug 2022-01-31T22:58:18.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:18.152555+0000) 2022-01-31T22:58:18.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:18 smithi171 conmon[41853]: debug 2022-01-31T22:58:18.347+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:18.348832+0000) 2022-01-31T22:58:18.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:18 smithi171 conmon[46715]: debug 2022-01-31T22:58:18.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:18.145353+0000) 2022-01-31T22:58:18.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:18 smithi171 conmon[51620]: debug 2022-01-31T22:58:18.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:18.525709+0000) 2022-01-31T22:58:18.940 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:18 smithi167 conmon[60316]: debug 2022-01-31T22:58:18.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:18.911516+0000) 2022-01-31T22:58:19.343 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:18 smithi167 conmon[49112]: debug 2022-01-31T22:58:18.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:18.983621+0000) 2022-01-31T22:58:19.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:19 smithi167 conmon[54076]: debug 2022-01-31T22:58:19.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:19.152730+0000) 2022-01-31T22:58:19.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:19 smithi171 conmon[46715]: debug 2022-01-31T22:58:19.144+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:19.145536+0000) 2022-01-31T22:58:19.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:19 smithi171 conmon[41853]: debug 2022-01-31T22:58:19.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:19.348999+0000) 2022-01-31T22:58:19.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:19 smithi171 conmon[51620]: debug 2022-01-31T22:58:19.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:19.525835+0000) 2022-01-31T22:58:19.949 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:19 smithi167 conmon[60316]: debug 2022-01-31T22:58:19.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:19.911716+0000) 2022-01-31T22:58:20.200 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:20 smithi167 conmon[60316]: debug 2022-01-31T22:58:20.209+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.211331+0000) 2022-01-31T22:58:20.201 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:19 smithi167 conmon[49112]: debug 2022-01-31T22:58:19.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:19.983824+0000) 2022-01-31T22:58:20.202 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:20 smithi167 conmon[49112]: debug 2022-01-31T22:58:20.210+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.212290+0000) 2022-01-31T22:58:20.202 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:20 smithi167 conmon[54076]: debug 2022-01-31T22:58:20.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.152885+0000) 2022-01-31T22:58:20.202 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:20 smithi167 conmon[54076]: debug 2022-01-31T22:58:20.211+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.212987+0000) 2022-01-31T22:58:20.510 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[35325]: debug 2022-01-31T22:58:20.235+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252274 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:20.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[41853]: debug 2022-01-31T22:58:20.209+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.211620+0000) 2022-01-31T22:58:20.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[41853]: debug 2022-01-31T22:58:20.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.349196+0000) 2022-01-31T22:58:20.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[46715]: debug 2022-01-31T22:58:20.143+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.145716+0000) 2022-01-31T22:58:20.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[46715]: debug 2022-01-31T22:58:20.209+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.210856+0000) 2022-01-31T22:58:20.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[51620]: debug 2022-01-31T22:58:20.209+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.210748+0000) 2022-01-31T22:58:20.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:20 smithi171 conmon[51620]: debug 2022-01-31T22:58:20.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.526035+0000) 2022-01-31T22:58:20.949 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:20 smithi167 conmon[60316]: debug 2022-01-31T22:58:20.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.911901+0000) 2022-01-31T22:58:21.344 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:20 smithi167 conmon[49112]: debug 2022-01-31T22:58:20.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:20.983991+0000) 2022-01-31T22:58:21.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:21 smithi167 conmon[54076]: debug 2022-01-31T22:58:21.151+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:21.153073+0000) 2022-01-31T22:58:21.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:21 smithi171 conmon[46715]: debug 2022-01-31T22:58:21.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:21.145921+0000) 2022-01-31T22:58:21.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:21 smithi171 conmon[41853]: debug 2022-01-31T22:58:21.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:21.349326+0000) 2022-01-31T22:58:21.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:21 smithi171 conmon[51620]: debug 2022-01-31T22:58:21.524+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:21.526220+0000) 2022-01-31T22:58:21.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:21 smithi167 conmon[60316]: debug 2022-01-31T22:58:21.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:21.912042+0000) 2022-01-31T22:58:22.344 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:21 smithi167 conmon[49112]: debug 2022-01-31T22:58:21.983+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:21.984129+0000) 2022-01-31T22:58:22.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:22 smithi167 conmon[54076]: debug 2022-01-31T22:58:22.152+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:22.153251+0000) 2022-01-31T22:58:22.510 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:22 smithi171 conmon[46715]: debug 2022-01-31T22:58:22.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:22.146094+0000) 2022-01-31T22:58:22.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:22 smithi171 conmon[41853]: debug 2022-01-31T22:58:22.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:22.349517+0000) 2022-01-31T22:58:22.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:22 smithi171 conmon[51620]: debug 2022-01-31T22:58:22.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:22.526425+0000) 2022-01-31T22:58:22.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:22 smithi167 conmon[60316]: debug 2022-01-31T22:58:22.911+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:22.912219+0000) 2022-01-31T22:58:23.344 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:22 smithi167 conmon[49112]: debug 2022-01-31T22:58:22.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:22.984317+0000) 2022-01-31T22:58:23.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:23 smithi167 conmon[54076]: debug 2022-01-31T22:58:23.152+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:23.153423+0000) 2022-01-31T22:58:23.510 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:23 smithi171 conmon[41853]: debug 2022-01-31T22:58:23.348+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:23.349697+0000) 2022-01-31T22:58:23.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:23 smithi171 conmon[46715]: debug 2022-01-31T22:58:23.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:23.146218+0000) 2022-01-31T22:58:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:23 smithi171 conmon[51620]: debug 2022-01-31T22:58:23.525+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:23.526645+0000) 2022-01-31T22:58:23.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:23 smithi167 conmon[60316]: debug 2022-01-31T22:58:23.910+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:23.912430+0000) 2022-01-31T22:58:24.334 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:24 smithi171 conmon[46715]: debug 2022-01-31T22:58:24.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:24.146399+0000) 2022-01-31T22:58:24.334 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:23 smithi171 conmon[35325]: debug 2022-01-31T22:58:23.939+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:58:24.344 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:23 smithi167 conmon[49112]: debug 2022-01-31T22:58:23.982+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:23.984452+0000) 2022-01-31T22:58:24.344 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:24 smithi167 conmon[54076]: debug 2022-01-31T22:58:24.152+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:24.153632+0000) 2022-01-31T22:58:24.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:24 smithi171 conmon[41853]: debug 2022-01-31T22:58:24.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:24.349908+0000) 2022-01-31T22:58:24.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:24 smithi171 conmon[51620]: debug 2022-01-31T22:58:24.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:24.526843+0000) 2022-01-31T22:58:24.968 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:24 smithi167 conmon[60316]: debug 2022-01-31T22:58:24.911+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:24.912646+0000) 2022-01-31T22:58:25.223 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:24 smithi167 conmon[49112]: debug 2022-01-31T22:58:24.983+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:24.984645+0000) 2022-01-31T22:58:25.224 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[54076]: debug 2022-01-31T22:58:25.152+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.153794+0000) 2022-01-31T22:58:25.511 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[35325]: debug 2022-01-31T22:58:25.270+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252387 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:25.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[46715]: debug 2022-01-31T22:58:25.145+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.146583+0000) 2022-01-31T22:58:25.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[46715]: debug 2022-01-31T22:58:25.238+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.238951+0000) 2022-01-31T22:58:25.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[51620]: debug 2022-01-31T22:58:25.239+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.239890+0000) 2022-01-31T22:58:25.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[41853]: debug 2022-01-31T22:58:25.239+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.239984+0000) 2022-01-31T22:58:25.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[41853]: debug 2022-01-31T22:58:25.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.350089+0000) 2022-01-31T22:58:25.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[54076]: debug 2022-01-31T22:58:25.238+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.240093+0000) 2022-01-31T22:58:25.654 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[60316]: debug 2022-01-31T22:58:25.239+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.240625+0000) 2022-01-31T22:58:25.654 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[49112]: debug 2022-01-31T22:58:25.245+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.247128+0000) 2022-01-31T22:58:25.655 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[49112]: 2022-01-31T22:58:25.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:25 smithi171 conmon[51620]: debug 2022-01-31T22:58:25.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.527030+0000) 2022-01-31T22:58:25.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[60316]: debug 2022-01-31T22:58:25.911+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.912826+0000) 2022-01-31T22:58:26.344 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:25 smithi167 conmon[49112]: debug 2022-01-31T22:58:25.983+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:25.984832+0000) 2022-01-31T22:58:26.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:26 smithi167 conmon[54076]: debug 2022-01-31T22:58:26.153+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:26.153982+0000) 2022-01-31T22:58:26.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:26 smithi171 conmon[41853]: debug 2022-01-31T22:58:26.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:26.350222+0000) 2022-01-31T22:58:26.511 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:26 smithi171 conmon[46715]: debug 2022-01-31T22:58:26.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:26.146799+0000) 2022-01-31T22:58:26.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:26 smithi171 conmon[51620]: debug 2022-01-31T22:58:26.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:26.527184+0000) 2022-01-31T22:58:26.969 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:26 smithi167 conmon[60316]: debug 2022-01-31T22:58:26.911+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:26.912982+0000) 2022-01-31T22:58:27.345 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:26 smithi167 conmon[49112]: debug 2022-01-31T22:58:26.983+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:26.984988+0000) 2022-01-31T22:58:27.345 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:27 smithi167 conmon[54076]: debug 2022-01-31T22:58:27.153+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:27.154131+0000) 2022-01-31T22:58:27.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:27 smithi171 conmon[41853]: debug 2022-01-31T22:58:27.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:27.350377+0000) 2022-01-31T22:58:27.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:27 smithi171 conmon[46715]: debug 2022-01-31T22:58:27.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:27.146935+0000) 2022-01-31T22:58:27.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:27 smithi171 conmon[51620]: debug 2022-01-31T22:58:27.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:27.527361+0000) 2022-01-31T22:58:28.138 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:27 smithi167 conmon[49112]: debug 2022-01-31T22:58:27.983+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:27.985214+0000) 2022-01-31T22:58:28.139 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:27 smithi167 conmon[60316]: debug 2022-01-31T22:58:27.912+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:27.913184+0000) 2022-01-31T22:58:28.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:28 smithi167 conmon[54076]: debug 2022-01-31T22:58:28.152+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:28.154316+0000) 2022-01-31T22:58:28.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:28 smithi171 conmon[41853]: debug 2022-01-31T22:58:28.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:28.350537+0000) 2022-01-31T22:58:28.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:28 smithi171 conmon[46715]: debug 2022-01-31T22:58:28.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:28.147113+0000) 2022-01-31T22:58:28.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:28 smithi171 conmon[51620]: debug 2022-01-31T22:58:28.526+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:28.527549+0000) 2022-01-31T22:58:29.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:28 smithi167 conmon[49112]: debug 2022-01-31T22:58:28.984+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:28.985418+0000) 2022-01-31T22:58:29.139 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:28 smithi167 conmon[60316]: debug 2022-01-31T22:58:28.912+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:28.913365+0000) 2022-01-31T22:58:29.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:29 smithi167 conmon[54076]: debug 2022-01-31T22:58:29.153+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:29.154490+0000) 2022-01-31T22:58:29.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:29 smithi171 conmon[41853]: debug 2022-01-31T22:58:29.349+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:29.350718+0000) 2022-01-31T22:58:29.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:29 smithi171 conmon[46715]: debug 2022-01-31T22:58:29.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:29.147292+0000) 2022-01-31T22:58:29.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:29 smithi171 conmon[51620]: debug 2022-01-31T22:58:29.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:29.527800+0000) 2022-01-31T22:58:30.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:29 smithi167 conmon[49112]: debug 2022-01-31T22:58:29.984+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:29.985637+0000) 2022-01-31T22:58:30.139 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:29 smithi167 conmon[60316]: debug 2022-01-31T22:58:29.912+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:29.913587+0000) 2022-01-31T22:58:30.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:30 smithi167 conmon[49112]: debug 2022-01-31T22:58:30.273+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.274962+0000) 2022-01-31T22:58:30.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:30 smithi167 conmon[54076]: debug 2022-01-31T22:58:30.153+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.154702+0000) 2022-01-31T22:58:30.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:30 smithi167 conmon[54076]: debug 2022-01-31T22:58:30.273+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.274588+0000) 2022-01-31T22:58:30.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:30 smithi167 conmon[60316]: debug 2022-01-31T22:58:30.273+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.275276+0000) 2022-01-31T22:58:30.512 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[51620]: debug 2022-01-31T22:58:30.272+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.273587+0000) 2022-01-31T22:58:30.512 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[35325]: debug 2022-01-31T22:58:30.298+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252496 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:30.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[41853]: debug 2022-01-31T22:58:30.273+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.273879+0000) 2022-01-31T22:58:30.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[41853]: debug 2022-01-31T22:58:30.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.350926+0000) 2022-01-31T22:58:30.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[46715]: debug 2022-01-31T22:58:30.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.147427+0000) 2022-01-31T22:58:30.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[46715]: debug 2022-01-31T22:58:30.274+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.274817+0000) 2022-01-31T22:58:30.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:30 smithi171 conmon[51620]: debug 2022-01-31T22:58:30.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.527927+0000) 2022-01-31T22:58:31.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:30 smithi167 conmon[49112]: debug 2022-01-31T22:58:30.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.985846+0000) 2022-01-31T22:58:31.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:30 smithi167 conmon[60316]: debug 2022-01-31T22:58:30.912+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:30.913723+0000) 2022-01-31T22:58:31.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:31 smithi167 conmon[54076]: debug 2022-01-31T22:58:31.153+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:31.154857+0000) 2022-01-31T22:58:31.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:31 smithi171 conmon[41853]: debug 2022-01-31T22:58:31.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:31.351089+0000) 2022-01-31T22:58:31.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:31 smithi171 conmon[46715]: debug 2022-01-31T22:58:31.146+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:31.147578+0000) 2022-01-31T22:58:31.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:31 smithi171 conmon[51620]: debug 2022-01-31T22:58:31.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:31.528083+0000) 2022-01-31T22:58:32.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:31 smithi167 conmon[49112]: debug 2022-01-31T22:58:31.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:31.986010+0000) 2022-01-31T22:58:32.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:31 smithi167 conmon[60316]: debug 2022-01-31T22:58:31.912+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:31.913855+0000) 2022-01-31T22:58:32.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:32 smithi167 conmon[54076]: debug 2022-01-31T22:58:32.154+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:32.155010+0000) 2022-01-31T22:58:32.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:32 smithi171 conmon[46715]: debug 2022-01-31T22:58:32.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:32.147749+0000) 2022-01-31T22:58:32.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:32 smithi171 conmon[41853]: debug 2022-01-31T22:58:32.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:32.351297+0000) 2022-01-31T22:58:32.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:32 smithi171 conmon[51620]: debug 2022-01-31T22:58:32.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:32.528262+0000) 2022-01-31T22:58:33.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:32 smithi167 conmon[49112]: debug 2022-01-31T22:58:32.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:32.986199+0000) 2022-01-31T22:58:33.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:32 smithi167 conmon[60316]: debug 2022-01-31T22:58:32.913+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:32.914042+0000) 2022-01-31T22:58:33.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:33 smithi167 conmon[54076]: debug 2022-01-31T22:58:33.154+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:33.155204+0000) 2022-01-31T22:58:33.512 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:33 smithi171 conmon[41853]: debug 2022-01-31T22:58:33.350+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:33.351481+0000) 2022-01-31T22:58:33.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:33 smithi171 conmon[46715]: debug 2022-01-31T22:58:33.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:33.147909+0000) 2022-01-31T22:58:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:33 smithi171 conmon[51620]: debug 2022-01-31T22:58:33.527+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:33.528455+0000) 2022-01-31T22:58:34.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:33 smithi167 conmon[49112]: debug 2022-01-31T22:58:33.984+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:33.986326+0000) 2022-01-31T22:58:34.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:33 smithi167 conmon[60316]: debug 2022-01-31T22:58:33.912+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:33.914245+0000) 2022-01-31T22:58:34.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:34 smithi167 conmon[54076]: debug 2022-01-31T22:58:34.154+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:34.155368+0000) 2022-01-31T22:58:34.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:34 smithi171 conmon[46715]: debug 2022-01-31T22:58:34.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:34.148086+0000) 2022-01-31T22:58:34.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:34 smithi171 conmon[41853]: debug 2022-01-31T22:58:34.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:34.351671+0000) 2022-01-31T22:58:34.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:34 smithi171 conmon[51620]: debug 2022-01-31T22:58:34.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:34.528615+0000) 2022-01-31T22:58:35.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:34 smithi167 conmon[49112]: debug 2022-01-31T22:58:34.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:34.986489+0000) 2022-01-31T22:58:35.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:34 smithi167 conmon[60316]: debug 2022-01-31T22:58:34.913+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:34.914441+0000) 2022-01-31T22:58:35.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:35 smithi167 conmon[49112]: debug 2022-01-31T22:58:35.300+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.301618+0000) 2022-01-31T22:58:35.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:35 smithi167 conmon[54076]: debug 2022-01-31T22:58:35.154+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.155541+0000) 2022-01-31T22:58:35.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:35 smithi167 conmon[54076]: debug 2022-01-31T22:58:35.301+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.302676+0000) 2022-01-31T22:58:35.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:35 smithi167 conmon[60316]: debug 2022-01-31T22:58:35.301+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.303055+0000) 2022-01-31T22:58:35.513 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[35325]: debug 2022-01-31T22:58:35.327+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252600 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:35.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[41853]: debug 2022-01-31T22:58:35.300+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.301110+0000) 2022-01-31T22:58:35.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[41853]: debug 2022-01-31T22:58:35.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.351877+0000) 2022-01-31T22:58:35.514 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[51620]: debug 2022-01-31T22:58:35.300+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.301373+0000) 2022-01-31T22:58:35.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[46715]: debug 2022-01-31T22:58:35.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.148290+0000) 2022-01-31T22:58:35.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[46715]: debug 2022-01-31T22:58:35.301+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.302635+0000) 2022-01-31T22:58:35.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:35 smithi171 conmon[51620]: debug 2022-01-31T22:58:35.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.528785+0000) 2022-01-31T22:58:36.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:35 smithi167 conmon[49112]: debug 2022-01-31T22:58:35.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.986654+0000) 2022-01-31T22:58:36.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:35 smithi167 conmon[60316]: debug 2022-01-31T22:58:35.914+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:35.914675+0000) 2022-01-31T22:58:36.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:36 smithi167 conmon[54076]: debug 2022-01-31T22:58:36.155+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:36.155743+0000) 2022-01-31T22:58:36.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:36 smithi171 conmon[46715]: debug 2022-01-31T22:58:36.147+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:36.148491+0000) 2022-01-31T22:58:36.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:36 smithi171 conmon[41853]: debug 2022-01-31T22:58:36.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:36.352022+0000) 2022-01-31T22:58:36.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:36 smithi171 conmon[51620]: debug 2022-01-31T22:58:36.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:36.528954+0000) 2022-01-31T22:58:37.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:36 smithi167 conmon[49112]: debug 2022-01-31T22:58:36.986+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:36.986797+0000) 2022-01-31T22:58:37.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:36 smithi167 conmon[60316]: debug 2022-01-31T22:58:36.914+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:36.914825+0000) 2022-01-31T22:58:37.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:37 smithi167 conmon[54076]: debug 2022-01-31T22:58:37.154+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:37.155824+0000) 2022-01-31T22:58:37.513 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:37 smithi171 conmon[41853]: debug 2022-01-31T22:58:37.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:37.352189+0000) 2022-01-31T22:58:37.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:37 smithi171 conmon[46715]: debug 2022-01-31T22:58:37.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:37.148645+0000) 2022-01-31T22:58:37.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:37 smithi171 conmon[51620]: debug 2022-01-31T22:58:37.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:37.529093+0000) 2022-01-31T22:58:38.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:37 smithi167 conmon[49112]: debug 2022-01-31T22:58:37.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:37.986894+0000) 2022-01-31T22:58:38.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:37 smithi167 conmon[60316]: debug 2022-01-31T22:58:37.914+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:37.914983+0000) 2022-01-31T22:58:38.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:38 smithi167 conmon[54076]: debug 2022-01-31T22:58:38.155+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:38.155983+0000) 2022-01-31T22:58:38.513 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:38 smithi171 conmon[46715]: debug 2022-01-31T22:58:38.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:38.148836+0000) 2022-01-31T22:58:38.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:38 smithi171 conmon[41853]: debug 2022-01-31T22:58:38.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:38.352291+0000) 2022-01-31T22:58:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:38 smithi171 conmon[51620]: debug 2022-01-31T22:58:38.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:38.529305+0000) 2022-01-31T22:58:39.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:38 smithi167 conmon[49112]: debug 2022-01-31T22:58:38.985+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:38.987010+0000) 2022-01-31T22:58:39.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:38 smithi167 conmon[60316]: debug 2022-01-31T22:58:38.913+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:38.915162+0000) 2022-01-31T22:58:39.336 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:38 smithi171 conmon[35325]: debug 2022-01-31T22:58:38.940+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:58:39.337 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:39 smithi171 conmon[46715]: debug 2022-01-31T22:58:39.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:39.148974+0000) 2022-01-31T22:58:39.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:39 smithi167 conmon[54076]: debug 2022-01-31T22:58:39.155+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:39.156144+0000) 2022-01-31T22:58:39.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:39 smithi171 conmon[41853]: debug 2022-01-31T22:58:39.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:39.352484+0000) 2022-01-31T22:58:39.592 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:39 smithi171 conmon[51620]: debug 2022-01-31T22:58:39.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:39.529462+0000) 2022-01-31T22:58:40.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:39 smithi167 conmon[49112]: debug 2022-01-31T22:58:39.986+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:39.987202+0000) 2022-01-31T22:58:40.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:39 smithi167 conmon[60316]: debug 2022-01-31T22:58:39.914+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:39.915316+0000) 2022-01-31T22:58:40.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:40 smithi167 conmon[49112]: debug 2022-01-31T22:58:40.328+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.330437+0000) 2022-01-31T22:58:40.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:40 smithi167 conmon[54076]: debug 2022-01-31T22:58:40.155+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.156304+0000) 2022-01-31T22:58:40.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:40 smithi167 conmon[54076]: debug 2022-01-31T22:58:40.330+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.331574+0000) 2022-01-31T22:58:40.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:40 smithi167 conmon[60316]: debug 2022-01-31T22:58:40.330+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.331734+0000) 2022-01-31T22:58:40.514 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[35325]: debug 2022-01-31T22:58:40.355+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252709 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:40.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[46715]: debug 2022-01-31T22:58:40.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.149157+0000) 2022-01-31T22:58:40.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[46715]: debug 2022-01-31T22:58:40.329+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.330069+0000) 2022-01-31T22:58:40.515 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[51620]: debug 2022-01-31T22:58:40.330+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.330773+0000) 2022-01-31T22:58:40.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[41853]: debug 2022-01-31T22:58:40.330+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.330842+0000) 2022-01-31T22:58:40.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[41853]: debug 2022-01-31T22:58:40.351+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.352652+0000) 2022-01-31T22:58:40.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:40 smithi171 conmon[51620]: debug 2022-01-31T22:58:40.528+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.529656+0000) 2022-01-31T22:58:41.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:40 smithi167 conmon[49112]: debug 2022-01-31T22:58:40.986+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.987394+0000) 2022-01-31T22:58:41.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:40 smithi167 conmon[60316]: debug 2022-01-31T22:58:40.914+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:40.915536+0000) 2022-01-31T22:58:41.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:41 smithi167 conmon[54076]: debug 2022-01-31T22:58:41.155+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:41.156537+0000) 2022-01-31T22:58:41.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:41 smithi171 conmon[46715]: debug 2022-01-31T22:58:41.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:41.149277+0000) 2022-01-31T22:58:41.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:41 smithi171 conmon[41853]: debug 2022-01-31T22:58:41.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:41.352854+0000) 2022-01-31T22:58:41.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:41 smithi171 conmon[51620]: debug 2022-01-31T22:58:41.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:41.529814+0000) 2022-01-31T22:58:42.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:41 smithi167 conmon[49112]: debug 2022-01-31T22:58:41.987+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:41.987534+0000) 2022-01-31T22:58:42.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:41 smithi167 conmon[60316]: debug 2022-01-31T22:58:41.915+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:41.915687+0000) 2022-01-31T22:58:42.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:42 smithi167 conmon[54076]: debug 2022-01-31T22:58:42.156+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:42.156695+0000) 2022-01-31T22:58:42.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:42 smithi171 conmon[46715]: debug 2022-01-31T22:58:42.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:42.149452+0000) 2022-01-31T22:58:42.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:42 smithi171 conmon[41853]: debug 2022-01-31T22:58:42.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:42.353026+0000) 2022-01-31T22:58:42.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:42 smithi171 conmon[51620]: debug 2022-01-31T22:58:42.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:42.530007+0000) 2022-01-31T22:58:43.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:42 smithi167 conmon[49112]: debug 2022-01-31T22:58:42.987+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:42.987666+0000) 2022-01-31T22:58:43.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:42 smithi167 conmon[60316]: debug 2022-01-31T22:58:42.915+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:42.915836+0000) 2022-01-31T22:58:43.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:43 smithi167 conmon[54076]: debug 2022-01-31T22:58:43.156+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:43.156902+0000) 2022-01-31T22:58:43.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:43 smithi171 conmon[46715]: debug 2022-01-31T22:58:43.148+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:43.149564+0000) 2022-01-31T22:58:43.514 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:43 smithi171 conmon[41853]: debug 2022-01-31T22:58:43.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:43.353201+0000) 2022-01-31T22:58:43.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:43 smithi171 conmon[51620]: debug 2022-01-31T22:58:43.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:43.530165+0000) 2022-01-31T22:58:44.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:43 smithi167 conmon[49112]: debug 2022-01-31T22:58:43.987+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:43.987766+0000) 2022-01-31T22:58:44.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:43 smithi167 conmon[60316]: debug 2022-01-31T22:58:43.915+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:43.916031+0000) 2022-01-31T22:58:44.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:44 smithi167 conmon[54076]: debug 2022-01-31T22:58:44.156+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:44.157094+0000) 2022-01-31T22:58:44.514 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:44 smithi171 conmon[46715]: debug 2022-01-31T22:58:44.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:44.149769+0000) 2022-01-31T22:58:44.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:44 smithi171 conmon[41853]: debug 2022-01-31T22:58:44.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:44.353359+0000) 2022-01-31T22:58:44.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:44 smithi171 conmon[51620]: debug 2022-01-31T22:58:44.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:44.530280+0000) 2022-01-31T22:58:45.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:44 smithi167 conmon[49112]: debug 2022-01-31T22:58:44.987+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:44.987915+0000) 2022-01-31T22:58:45.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:44 smithi167 conmon[60316]: debug 2022-01-31T22:58:44.915+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:44.916228+0000) 2022-01-31T22:58:45.404 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:45 smithi167 conmon[49112]: debug 2022-01-31T22:58:45.359+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.360045+0000) 2022-01-31T22:58:45.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:45 smithi167 conmon[54076]: debug 2022-01-31T22:58:45.156+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.157320+0000) 2022-01-31T22:58:45.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:45 smithi167 conmon[54076]: debug 2022-01-31T22:58:45.359+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.359701+0000) 2022-01-31T22:58:45.405 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:45 smithi167 conmon[60316]: debug 2022-01-31T22:58:45.359+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.359799+0000) 2022-01-31T22:58:45.514 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[35325]: debug 2022-01-31T22:58:45.383+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252818 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:45.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[46715]: debug 2022-01-31T22:58:45.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.149958+0000) 2022-01-31T22:58:45.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[46715]: debug 2022-01-31T22:58:45.357+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.358504+0000) 2022-01-31T22:58:45.516 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[51620]: debug 2022-01-31T22:58:45.358+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.358801+0000) 2022-01-31T22:58:45.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[41853]: debug 2022-01-31T22:58:45.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.353514+0000) 2022-01-31T22:58:45.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[41853]: debug 2022-01-31T22:58:45.360+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.361131+0000) 2022-01-31T22:58:45.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:45 smithi171 conmon[51620]: debug 2022-01-31T22:58:45.529+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.530451+0000) 2022-01-31T22:58:46.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:45 smithi167 conmon[49112]: debug 2022-01-31T22:58:45.987+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.988093+0000) 2022-01-31T22:58:46.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:45 smithi167 conmon[60316]: debug 2022-01-31T22:58:45.915+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:45.916439+0000) 2022-01-31T22:58:46.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:46 smithi167 conmon[54076]: debug 2022-01-31T22:58:46.157+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:46.157485+0000) 2022-01-31T22:58:46.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:46 smithi171 conmon[41853]: debug 2022-01-31T22:58:46.352+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:46.353730+0000) 2022-01-31T22:58:46.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:46 smithi171 conmon[46715]: debug 2022-01-31T22:58:46.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:46.150132+0000) 2022-01-31T22:58:46.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:46 smithi171 conmon[51620]: debug 2022-01-31T22:58:46.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:46.530609+0000) 2022-01-31T22:58:47.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:46 smithi167 conmon[49112]: debug 2022-01-31T22:58:46.987+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:46.988240+0000) 2022-01-31T22:58:47.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:46 smithi167 conmon[60316]: debug 2022-01-31T22:58:46.916+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:46.916614+0000) 2022-01-31T22:58:47.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:47 smithi167 conmon[54076]: debug 2022-01-31T22:58:47.157+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:47.157657+0000) 2022-01-31T22:58:47.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:47 smithi171 conmon[46715]: debug 2022-01-31T22:58:47.149+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:47.150260+0000) 2022-01-31T22:58:47.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:47 smithi171 conmon[41853]: debug 2022-01-31T22:58:47.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:47.353854+0000) 2022-01-31T22:58:47.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:47 smithi171 conmon[51620]: debug 2022-01-31T22:58:47.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:47.530774+0000) 2022-01-31T22:58:48.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:47 smithi167 conmon[49112]: debug 2022-01-31T22:58:47.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:47.988421+0000) 2022-01-31T22:58:48.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:47 smithi167 conmon[60316]: debug 2022-01-31T22:58:47.916+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:47.916756+0000) 2022-01-31T22:58:48.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:48 smithi167 conmon[54076]: debug 2022-01-31T22:58:48.157+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:48.157787+0000) 2022-01-31T22:58:48.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:48 smithi171 conmon[41853]: debug 2022-01-31T22:58:48.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:48.354038+0000) 2022-01-31T22:58:48.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:48 smithi171 conmon[46715]: debug 2022-01-31T22:58:48.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:48.150420+0000) 2022-01-31T22:58:48.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:48 smithi171 conmon[51620]: debug 2022-01-31T22:58:48.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:48.530873+0000) 2022-01-31T22:58:49.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:48 smithi167 conmon[49112]: debug 2022-01-31T22:58:48.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:48.988586+0000) 2022-01-31T22:58:49.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:48 smithi167 conmon[60316]: debug 2022-01-31T22:58:48.916+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:48.916903+0000) 2022-01-31T22:58:49.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:49 smithi167 conmon[54076]: debug 2022-01-31T22:58:49.157+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:49.157983+0000) 2022-01-31T22:58:49.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:49 smithi171 conmon[46715]: debug 2022-01-31T22:58:49.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:49.150626+0000) 2022-01-31T22:58:49.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:49 smithi171 conmon[41853]: debug 2022-01-31T22:58:49.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:49.354202+0000) 2022-01-31T22:58:49.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:49 smithi171 conmon[51620]: debug 2022-01-31T22:58:49.530+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:49.531026+0000) 2022-01-31T22:58:50.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:49 smithi167 conmon[60316]: debug 2022-01-31T22:58:49.916+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:49.917066+0000) 2022-01-31T22:58:50.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:49 smithi167 conmon[49112]: debug 2022-01-31T22:58:49.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:49.988779+0000) 2022-01-31T22:58:50.396 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[46715]: debug 2022-01-31T22:58:50.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.150795+0000) 2022-01-31T22:58:50.396 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[46715]: debug 2022-01-31T22:58:50.385+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.386483+0000) 2022-01-31T22:58:50.397 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[51620]: debug 2022-01-31T22:58:50.386+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.387051+0000) 2022-01-31T22:58:50.397 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[41853]: debug 2022-01-31T22:58:50.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.354392+0000) 2022-01-31T22:58:50.398 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[41853]: debug 2022-01-31T22:58:50.386+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.386835+0000) 2022-01-31T22:58:50.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:50 smithi167 conmon[49112]: debug 2022-01-31T22:58:50.387+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.387762+0000) 2022-01-31T22:58:50.404 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:50 smithi167 conmon[60316]: debug 2022-01-31T22:58:50.386+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.387411+0000) 2022-01-31T22:58:50.404 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:50 smithi167 conmon[54076]: debug 2022-01-31T22:58:50.157+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.158149+0000) 2022-01-31T22:58:50.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:50 smithi167 conmon[54076]: debug 2022-01-31T22:58:50.386+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.387056+0000) 2022-01-31T22:58:50.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[51620]: debug 2022-01-31T22:58:50.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.531260+0000) 2022-01-31T22:58:50.842 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:50 smithi171 conmon[35325]: debug 2022-01-31T22:58:50.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 252944 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:51.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:50 smithi167 conmon[49112]: debug 2022-01-31T22:58:50.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.988980+0000) 2022-01-31T22:58:51.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:50 smithi167 conmon[60316]: debug 2022-01-31T22:58:50.916+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:50.917255+0000) 2022-01-31T22:58:51.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:51 smithi167 conmon[54076]: debug 2022-01-31T22:58:51.157+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:51.158365+0000) 2022-01-31T22:58:51.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:51 smithi171 conmon[46715]: debug 2022-01-31T22:58:51.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:51.150981+0000) 2022-01-31T22:58:51.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:51 smithi171 conmon[41853]: debug 2022-01-31T22:58:51.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:51.354517+0000) 2022-01-31T22:58:51.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:51 smithi171 conmon[51620]: debug 2022-01-31T22:58:51.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:51.531409+0000) 2022-01-31T22:58:52.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:51 smithi167 conmon[49112]: debug 2022-01-31T22:58:51.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:51.989115+0000) 2022-01-31T22:58:52.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:51 smithi167 conmon[60316]: debug 2022-01-31T22:58:51.916+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:51.917403+0000) 2022-01-31T22:58:52.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:52 smithi167 conmon[54076]: debug 2022-01-31T22:58:52.158+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:52.158533+0000) 2022-01-31T22:58:52.515 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:52 smithi171 conmon[46715]: debug 2022-01-31T22:58:52.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:52.151144+0000) 2022-01-31T22:58:52.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:52 smithi171 conmon[41853]: debug 2022-01-31T22:58:52.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:52.354681+0000) 2022-01-31T22:58:52.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:52 smithi171 conmon[51620]: debug 2022-01-31T22:58:52.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:52.531558+0000) 2022-01-31T22:58:53.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:52 smithi167 conmon[49112]: debug 2022-01-31T22:58:52.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:52.989287+0000) 2022-01-31T22:58:53.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:52 smithi167 conmon[60316]: debug 2022-01-31T22:58:52.917+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:52.917597+0000) 2022-01-31T22:58:53.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:53 smithi167 conmon[54076]: debug 2022-01-31T22:58:53.158+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:53.158731+0000) 2022-01-31T22:58:53.515 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:53 smithi171 conmon[41853]: debug 2022-01-31T22:58:53.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:53.354909+0000) 2022-01-31T22:58:53.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:53 smithi171 conmon[46715]: debug 2022-01-31T22:58:53.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:53.151350+0000) 2022-01-31T22:58:53.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:53 smithi171 conmon[51620]: debug 2022-01-31T22:58:53.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:53.531731+0000) 2022-01-31T22:58:54.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:53 smithi167 conmon[49112]: debug 2022-01-31T22:58:53.988+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:53.989416+0000) 2022-01-31T22:58:54.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:53 smithi167 conmon[60316]: debug 2022-01-31T22:58:53.917+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:53.917748+0000) 2022-01-31T22:58:54.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:53 smithi171 conmon[35325]: debug 2022-01-31T22:58:53.941+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:58:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:54 smithi171 conmon[46715]: debug 2022-01-31T22:58:54.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:54.151511+0000) 2022-01-31T22:58:54.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:54 smithi167 conmon[54076]: debug 2022-01-31T22:58:54.158+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:54.158896+0000) 2022-01-31T22:58:54.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:54 smithi171 conmon[51620]: debug 2022-01-31T22:58:54.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:54.531925+0000) 2022-01-31T22:58:54.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:54 smithi171 conmon[41853]: debug 2022-01-31T22:58:54.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:54.355116+0000) 2022-01-31T22:58:55.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:54 smithi167 conmon[49112]: debug 2022-01-31T22:58:54.989+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:54.989583+0000) 2022-01-31T22:58:55.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:54 smithi167 conmon[60316]: debug 2022-01-31T22:58:54.917+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:54.917936+0000) 2022-01-31T22:58:55.398 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:55 smithi167 conmon[54076]: debug 2022-01-31T22:58:55.158+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.159071+0000) 2022-01-31T22:58:55.399 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[46715]: debug 2022-01-31T22:58:55.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.151715+0000) 2022-01-31T22:58:55.400 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[41853]: debug 2022-01-31T22:58:55.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.355274+0000) 2022-01-31T22:58:55.654 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:55 smithi167 conmon[49112]: debug 2022-01-31T22:58:55.414+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.414835+0000) 2022-01-31T22:58:55.655 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:55 smithi167 conmon[54076]: debug 2022-01-31T22:58:55.413+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.414392+0000) 2022-01-31T22:58:55.656 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:55 smithi167 conmon[60316]: debug 2022-01-31T22:58:55.415+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.415537+0000) 2022-01-31T22:58:55.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[35325]: debug 2022-01-31T22:58:55.438+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253058 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:58:55.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[41853]: debug 2022-01-31T22:58:55.413+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.414666+0000) 2022-01-31T22:58:55.843 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[46715]: debug 2022-01-31T22:58:55.414+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.415490+0000) 2022-01-31T22:58:55.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[51620]: debug 2022-01-31T22:58:55.413+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.413952+0000) 2022-01-31T22:58:55.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:55 smithi171 conmon[51620]: debug 2022-01-31T22:58:55.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.532076+0000) 2022-01-31T22:58:56.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:55 smithi167 conmon[49112]: debug 2022-01-31T22:58:55.989+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.989710+0000) 2022-01-31T22:58:56.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:55 smithi167 conmon[60316]: debug 2022-01-31T22:58:55.917+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:55.918110+0000) 2022-01-31T22:58:56.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:56 smithi167 conmon[54076]: debug 2022-01-31T22:58:56.158+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:56.159247+0000) 2022-01-31T22:58:56.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:56 smithi171 conmon[46715]: debug 2022-01-31T22:58:56.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:56.151892+0000) 2022-01-31T22:58:56.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:56 smithi171 conmon[41853]: debug 2022-01-31T22:58:56.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:56.355451+0000) 2022-01-31T22:58:56.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:56 smithi171 conmon[51620]: debug 2022-01-31T22:58:56.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:56.532220+0000) 2022-01-31T22:58:57.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:56 smithi167 conmon[49112]: debug 2022-01-31T22:58:56.989+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:56.989872+0000) 2022-01-31T22:58:57.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:56 smithi167 conmon[60316]: debug 2022-01-31T22:58:56.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:56.918304+0000) 2022-01-31T22:58:57.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:57 smithi167 conmon[54076]: debug 2022-01-31T22:58:57.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:57.159405+0000) 2022-01-31T22:58:57.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:57 smithi171 conmon[46715]: debug 2022-01-31T22:58:57.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:57.152029+0000) 2022-01-31T22:58:57.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:57 smithi171 conmon[41853]: debug 2022-01-31T22:58:57.353+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:57.355613+0000) 2022-01-31T22:58:57.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:57 smithi171 conmon[51620]: debug 2022-01-31T22:58:57.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:57.532405+0000) 2022-01-31T22:58:58.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:57 smithi167 conmon[49112]: debug 2022-01-31T22:58:57.989+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:57.990021+0000) 2022-01-31T22:58:58.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:57 smithi167 conmon[60316]: debug 2022-01-31T22:58:57.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:57.918538+0000) 2022-01-31T22:58:58.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:58 smithi167 conmon[54076]: debug 2022-01-31T22:58:58.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:58.159591+0000) 2022-01-31T22:58:58.516 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:58 smithi171 conmon[41853]: debug 2022-01-31T22:58:58.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:58.355826+0000) 2022-01-31T22:58:58.517 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:58 smithi171 conmon[46715]: debug 2022-01-31T22:58:58.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:58.152223+0000) 2022-01-31T22:58:58.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:58 smithi171 conmon[51620]: debug 2022-01-31T22:58:58.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:58.532564+0000) 2022-01-31T22:58:59.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:58 smithi167 conmon[49112]: debug 2022-01-31T22:58:58.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:58.990187+0000) 2022-01-31T22:58:59.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:58 smithi167 conmon[60316]: debug 2022-01-31T22:58:58.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:58.918752+0000) 2022-01-31T22:58:59.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:58:59 smithi167 conmon[54076]: debug 2022-01-31T22:58:59.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:59.159767+0000) 2022-01-31T22:58:59.516 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:58:59 smithi171 conmon[46715]: debug 2022-01-31T22:58:59.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:59.152449+0000) 2022-01-31T22:58:59.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:58:59 smithi171 conmon[41853]: debug 2022-01-31T22:58:59.354+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:59.355993+0000) 2022-01-31T22:58:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:58:59 smithi171 conmon[51620]: debug 2022-01-31T22:58:59.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:59.532722+0000) 2022-01-31T22:59:00.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:58:59 smithi167 conmon[60316]: debug 2022-01-31T22:58:59.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:59.918947+0000) 2022-01-31T22:59:00.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:58:59 smithi167 conmon[49112]: debug 2022-01-31T22:58:59.989+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:58:59.990405+0000) 2022-01-31T22:59:00.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:00 smithi167 conmon[54076]: debug 2022-01-31T22:59:00.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.159936+0000) 2022-01-31T22:59:00.425 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[46715]: debug 2022-01-31T22:59:00.150+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.152651+0000) 2022-01-31T22:59:00.426 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[51620]: debug 2022-01-31T22:59:00.439+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.441277+0000) 2022-01-31T22:59:00.426 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[41853]: debug 2022-01-31T22:59:00.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.356141+0000) 2022-01-31T22:59:00.427 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[41853]: debug 2022-01-31T22:59:00.439+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.441135+0000) 2022-01-31T22:59:00.747 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:00 smithi167 conmon[49112]: debug 2022-01-31T22:59:00.440+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.441453+0000) 2022-01-31T22:59:00.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:00 smithi167 conmon[54076]: debug 2022-01-31T22:59:00.440+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.441359+0000) 2022-01-31T22:59:00.748 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:00 smithi167 conmon[60316]: debug 2022-01-31T22:59:00.441+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.442251+0000) 2022-01-31T22:59:00.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[35325]: debug 2022-01-31T22:59:00.466+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253170 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:00.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[46715]: debug 2022-01-31T22:59:00.441+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.443128+0000) 2022-01-31T22:59:00.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:00 smithi171 conmon[51620]: debug 2022-01-31T22:59:00.531+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.532936+0000) 2022-01-31T22:59:01.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:00 smithi167 conmon[49112]: debug 2022-01-31T22:59:00.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.990598+0000) 2022-01-31T22:59:01.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:00 smithi167 conmon[60316]: debug 2022-01-31T22:59:00.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:00.919083+0000) 2022-01-31T22:59:01.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:01 smithi167 conmon[54076]: debug 2022-01-31T22:59:01.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:01.160113+0000) 2022-01-31T22:59:01.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:01 smithi171 conmon[41853]: debug 2022-01-31T22:59:01.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:01.356331+0000) 2022-01-31T22:59:01.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:01 smithi171 conmon[46715]: debug 2022-01-31T22:59:01.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:01.152877+0000) 2022-01-31T22:59:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:01 smithi171 conmon[51620]: debug 2022-01-31T22:59:01.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:01.533101+0000) 2022-01-31T22:59:02.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:01 smithi167 conmon[49112]: debug 2022-01-31T22:59:01.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:01.990748+0000) 2022-01-31T22:59:02.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:01 smithi167 conmon[60316]: debug 2022-01-31T22:59:01.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:01.919218+0000) 2022-01-31T22:59:02.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:02 smithi167 conmon[54076]: debug 2022-01-31T22:59:02.160+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:02.160266+0000) 2022-01-31T22:59:02.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:02 smithi171 conmon[41853]: debug 2022-01-31T22:59:02.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:02.356502+0000) 2022-01-31T22:59:02.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:02 smithi171 conmon[46715]: debug 2022-01-31T22:59:02.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:02.153034+0000) 2022-01-31T22:59:02.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:02 smithi171 conmon[51620]: debug 2022-01-31T22:59:02.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:02.533251+0000) 2022-01-31T22:59:03.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:02 smithi167 conmon[49112]: debug 2022-01-31T22:59:02.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:02.990946+0000) 2022-01-31T22:59:03.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:02 smithi167 conmon[60316]: debug 2022-01-31T22:59:02.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:02.919410+0000) 2022-01-31T22:59:03.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:03 smithi167 conmon[54076]: debug 2022-01-31T22:59:03.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:03.160449+0000) 2022-01-31T22:59:03.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:03 smithi171 conmon[41853]: debug 2022-01-31T22:59:03.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:03.356671+0000) 2022-01-31T22:59:03.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:03 smithi171 conmon[46715]: debug 2022-01-31T22:59:03.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:03.153218+0000) 2022-01-31T22:59:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:03 smithi171 conmon[51620]: debug 2022-01-31T22:59:03.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:03.533423+0000) 2022-01-31T22:59:04.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:03 smithi167 conmon[49112]: debug 2022-01-31T22:59:03.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:03.991052+0000) 2022-01-31T22:59:04.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:03 smithi167 conmon[60316]: debug 2022-01-31T22:59:03.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:03.919586+0000) 2022-01-31T22:59:04.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:04 smithi167 conmon[54076]: debug 2022-01-31T22:59:04.160+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:04.160670+0000) 2022-01-31T22:59:04.517 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:04 smithi171 conmon[41853]: debug 2022-01-31T22:59:04.355+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:04.356833+0000) 2022-01-31T22:59:04.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:04 smithi171 conmon[46715]: debug 2022-01-31T22:59:04.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:04.153472+0000) 2022-01-31T22:59:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:04 smithi171 conmon[51620]: debug 2022-01-31T22:59:04.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:04.533596+0000) 2022-01-31T22:59:05.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:04 smithi167 conmon[49112]: debug 2022-01-31T22:59:04.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:04.991281+0000) 2022-01-31T22:59:05.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:04 smithi167 conmon[60316]: debug 2022-01-31T22:59:04.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:04.919790+0000) 2022-01-31T22:59:05.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:05 smithi167 conmon[54076]: debug 2022-01-31T22:59:05.160+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.160903+0000) 2022-01-31T22:59:05.454 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[46715]: debug 2022-01-31T22:59:05.151+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.153705+0000) 2022-01-31T22:59:05.455 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[41853]: debug 2022-01-31T22:59:05.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.357022+0000) 2022-01-31T22:59:05.455 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[41853]: debug 2022-01-31T22:59:05.468+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.470532+0000) 2022-01-31T22:59:05.748 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:05 smithi167 conmon[49112]: debug 2022-01-31T22:59:05.470+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.470713+0000) 2022-01-31T22:59:05.748 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:05 smithi167 conmon[54076]: debug 2022-01-31T22:59:05.470+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.471293+0000) 2022-01-31T22:59:05.749 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:05 smithi167 conmon[60316]: debug 2022-01-31T22:59:05.471+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.472168+0000) 2022-01-31T22:59:05.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[35325]: debug 2022-01-31T22:59:05.495+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253282 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:05.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[46715]: debug 2022-01-31T22:59:05.470+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.472644+0000) 2022-01-31T22:59:05.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[51620]: debug 2022-01-31T22:59:05.470+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.472318+0000) 2022-01-31T22:59:05.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:05 smithi171 conmon[51620]: debug 2022-01-31T22:59:05.532+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.533793+0000) 2022-01-31T22:59:06.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:05 smithi167 conmon[49112]: debug 2022-01-31T22:59:05.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.991464+0000) 2022-01-31T22:59:06.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:05 smithi167 conmon[60316]: debug 2022-01-31T22:59:05.918+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:05.919979+0000) 2022-01-31T22:59:06.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:06 smithi167 conmon[54076]: debug 2022-01-31T22:59:06.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:06.161096+0000) 2022-01-31T22:59:06.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:06 smithi171 conmon[46715]: debug 2022-01-31T22:59:06.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:06.153955+0000) 2022-01-31T22:59:06.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:06 smithi171 conmon[41853]: debug 2022-01-31T22:59:06.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:06.357222+0000) 2022-01-31T22:59:06.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:06 smithi171 conmon[51620]: debug 2022-01-31T22:59:06.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:06.533957+0000) 2022-01-31T22:59:07.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:06 smithi167 conmon[49112]: debug 2022-01-31T22:59:06.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:06.991611+0000) 2022-01-31T22:59:07.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:06 smithi167 conmon[60316]: debug 2022-01-31T22:59:06.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:06.920134+0000) 2022-01-31T22:59:07.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:07 smithi167 conmon[54076]: debug 2022-01-31T22:59:07.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:07.161251+0000) 2022-01-31T22:59:07.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:07 smithi171 conmon[41853]: debug 2022-01-31T22:59:07.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:07.357373+0000) 2022-01-31T22:59:07.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:07 smithi171 conmon[46715]: debug 2022-01-31T22:59:07.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:07.154128+0000) 2022-01-31T22:59:07.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:07 smithi171 conmon[51620]: debug 2022-01-31T22:59:07.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:07.534095+0000) 2022-01-31T22:59:08.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:07 smithi167 conmon[49112]: debug 2022-01-31T22:59:07.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:07.991806+0000) 2022-01-31T22:59:08.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:07 smithi167 conmon[60316]: debug 2022-01-31T22:59:07.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:07.920342+0000) 2022-01-31T22:59:08.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:08 smithi167 conmon[54076]: debug 2022-01-31T22:59:08.159+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:08.161443+0000) 2022-01-31T22:59:08.518 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:08 smithi171 conmon[41853]: debug 2022-01-31T22:59:08.356+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:08.357576+0000) 2022-01-31T22:59:08.518 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:08 smithi171 conmon[46715]: debug 2022-01-31T22:59:08.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:08.154343+0000) 2022-01-31T22:59:08.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:08 smithi171 conmon[51620]: debug 2022-01-31T22:59:08.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:08.534276+0000) 2022-01-31T22:59:09.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:08 smithi167 conmon[49112]: debug 2022-01-31T22:59:08.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:08.991975+0000) 2022-01-31T22:59:09.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:08 smithi167 conmon[60316]: debug 2022-01-31T22:59:08.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:08.920550+0000) 2022-01-31T22:59:09.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:08 smithi171 conmon[35325]: debug 2022-01-31T22:59:08.941+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:59:09.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:09 smithi171 conmon[46715]: debug 2022-01-31T22:59:09.153+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:09.154566+0000) 2022-01-31T22:59:09.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:09 smithi167 conmon[54076]: debug 2022-01-31T22:59:09.160+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:09.161650+0000) 2022-01-31T22:59:09.748 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:09 smithi171 conmon[51620]: debug 2022-01-31T22:59:09.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:09.534463+0000) 2022-01-31T22:59:09.749 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:09 smithi171 conmon[41853]: debug 2022-01-31T22:59:09.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:09.357818+0000) 2022-01-31T22:59:10.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:09 smithi167 conmon[49112]: debug 2022-01-31T22:59:09.991+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:09.992173+0000) 2022-01-31T22:59:10.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:09 smithi167 conmon[60316]: debug 2022-01-31T22:59:09.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:09.920738+0000) 2022-01-31T22:59:10.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:10 smithi167 conmon[54076]: debug 2022-01-31T22:59:10.160+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.161771+0000) 2022-01-31T22:59:10.483 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[46715]: debug 2022-01-31T22:59:10.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.154785+0000) 2022-01-31T22:59:10.484 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[41853]: debug 2022-01-31T22:59:10.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.358031+0000) 2022-01-31T22:59:10.749 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:10 smithi167 conmon[49112]: debug 2022-01-31T22:59:10.499+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.500535+0000) 2022-01-31T22:59:10.749 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:10 smithi167 conmon[54076]: debug 2022-01-31T22:59:10.498+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.500052+0000) 2022-01-31T22:59:10.750 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:10 smithi167 conmon[60316]: debug 2022-01-31T22:59:10.498+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.499526+0000) 2022-01-31T22:59:10.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[35325]: debug 2022-01-31T22:59:10.524+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253393 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:10.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[41853]: debug 2022-01-31T22:59:10.499+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.499910+0000) 2022-01-31T22:59:10.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[46715]: debug 2022-01-31T22:59:10.500+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.500778+0000) 2022-01-31T22:59:10.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[51620]: debug 2022-01-31T22:59:10.500+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.501001+0000) 2022-01-31T22:59:10.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:10 smithi171 conmon[51620]: debug 2022-01-31T22:59:10.533+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.534624+0000) 2022-01-31T22:59:11.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:10 smithi167 conmon[49112]: debug 2022-01-31T22:59:10.990+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.992357+0000) 2022-01-31T22:59:11.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:10 smithi167 conmon[60316]: debug 2022-01-31T22:59:10.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:10.920929+0000) 2022-01-31T22:59:11.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:11 smithi167 conmon[54076]: debug 2022-01-31T22:59:11.160+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:11.161915+0000) 2022-01-31T22:59:11.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:11 smithi171 conmon[46715]: debug 2022-01-31T22:59:11.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:11.155033+0000) 2022-01-31T22:59:11.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:11 smithi171 conmon[41853]: debug 2022-01-31T22:59:11.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:11.358208+0000) 2022-01-31T22:59:11.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:11 smithi171 conmon[51620]: debug 2022-01-31T22:59:11.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:11.534730+0000) 2022-01-31T22:59:12.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:11 smithi167 conmon[49112]: debug 2022-01-31T22:59:11.991+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:11.992525+0000) 2022-01-31T22:59:12.147 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:11 smithi167 conmon[60316]: debug 2022-01-31T22:59:11.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:11.921089+0000) 2022-01-31T22:59:12.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:12 smithi167 conmon[54076]: debug 2022-01-31T22:59:12.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:12.162074+0000) 2022-01-31T22:59:12.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:12 smithi171 conmon[46715]: debug 2022-01-31T22:59:12.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:12.155229+0000) 2022-01-31T22:59:12.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:12 smithi171 conmon[41853]: debug 2022-01-31T22:59:12.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:12.358326+0000) 2022-01-31T22:59:12.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:12 smithi171 conmon[51620]: debug 2022-01-31T22:59:12.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:12.534971+0000) 2022-01-31T22:59:13.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:12 smithi167 conmon[60316]: debug 2022-01-31T22:59:12.920+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:12.921253+0000) 2022-01-31T22:59:13.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:12 smithi167 conmon[49112]: debug 2022-01-31T22:59:12.991+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:12.992728+0000) 2022-01-31T22:59:13.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:13 smithi167 conmon[54076]: debug 2022-01-31T22:59:13.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:13.162269+0000) 2022-01-31T22:59:13.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:13 smithi171 conmon[46715]: debug 2022-01-31T22:59:13.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:13.155422+0000) 2022-01-31T22:59:13.519 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:13 smithi171 conmon[41853]: debug 2022-01-31T22:59:13.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:13.358479+0000) 2022-01-31T22:59:13.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:13 smithi171 conmon[51620]: debug 2022-01-31T22:59:13.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:13.535154+0000) 2022-01-31T22:59:14.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:13 smithi167 conmon[60316]: debug 2022-01-31T22:59:13.919+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:13.921409+0000) 2022-01-31T22:59:14.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:13 smithi167 conmon[49112]: debug 2022-01-31T22:59:13.991+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:13.992882+0000) 2022-01-31T22:59:14.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:14 smithi167 conmon[54076]: debug 2022-01-31T22:59:14.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:14.162470+0000) 2022-01-31T22:59:14.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:14 smithi171 conmon[46715]: debug 2022-01-31T22:59:14.154+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:14.155630+0000) 2022-01-31T22:59:14.599 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:14 smithi171 conmon[41853]: debug 2022-01-31T22:59:14.357+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:14.358643+0000) 2022-01-31T22:59:14.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:14 smithi171 conmon[51620]: debug 2022-01-31T22:59:14.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:14.535321+0000) 2022-01-31T22:59:15.146 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:14 smithi167 conmon[49112]: debug 2022-01-31T22:59:14.992+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:14.993073+0000) 2022-01-31T22:59:15.147 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:14 smithi167 conmon[60316]: debug 2022-01-31T22:59:14.920+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:14.921587+0000) 2022-01-31T22:59:15.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:15 smithi167 conmon[54076]: debug 2022-01-31T22:59:15.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.162669+0000) 2022-01-31T22:59:15.511 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[41853]: debug 2022-01-31T22:59:15.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.358852+0000) 2022-01-31T22:59:15.512 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[46715]: debug 2022-01-31T22:59:15.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.155826+0000) 2022-01-31T22:59:15.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[35325]: debug 2022-01-31T22:59:15.553+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253501 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:15.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[41853]: debug 2022-01-31T22:59:15.526+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.527470+0000) 2022-01-31T22:59:15.843 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[46715]: debug 2022-01-31T22:59:15.528+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.528778+0000) 2022-01-31T22:59:15.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[51620]: debug 2022-01-31T22:59:15.528+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.529175+0000) 2022-01-31T22:59:15.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:15 smithi171 conmon[51620]: debug 2022-01-31T22:59:15.534+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.535518+0000) 2022-01-31T22:59:15.876 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:15 smithi167 conmon[49112]: debug 2022-01-31T22:59:15.528+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.529494+0000) 2022-01-31T22:59:15.877 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:15 smithi167 conmon[54076]: debug 2022-01-31T22:59:15.526+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.527631+0000) 2022-01-31T22:59:15.877 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:15 smithi167 conmon[60316]: debug 2022-01-31T22:59:15.527+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.528970+0000) 2022-01-31T22:59:16.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:15 smithi167 conmon[49112]: debug 2022-01-31T22:59:15.992+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.993266+0000) 2022-01-31T22:59:16.147 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:15 smithi167 conmon[60316]: debug 2022-01-31T22:59:15.920+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:15.921711+0000) 2022-01-31T22:59:16.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:16 smithi167 conmon[54076]: debug 2022-01-31T22:59:16.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:16.162788+0000) 2022-01-31T22:59:16.519 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:16 smithi171 conmon[46715]: debug 2022-01-31T22:59:16.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:16.156019+0000) 2022-01-31T22:59:16.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:16 smithi171 conmon[41853]: debug 2022-01-31T22:59:16.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:16.359033+0000) 2022-01-31T22:59:16.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:16 smithi171 conmon[51620]: debug 2022-01-31T22:59:16.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:16.535671+0000) 2022-01-31T22:59:17.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:16 smithi167 conmon[49112]: debug 2022-01-31T22:59:16.991+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:16.993430+0000) 2022-01-31T22:59:17.147 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:16 smithi167 conmon[60316]: debug 2022-01-31T22:59:16.920+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:16.921829+0000) 2022-01-31T22:59:17.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:17 smithi167 conmon[54076]: debug 2022-01-31T22:59:17.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:17.162905+0000) 2022-01-31T22:59:17.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:17 smithi171 conmon[46715]: debug 2022-01-31T22:59:17.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:17.156186+0000) 2022-01-31T22:59:17.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:17 smithi171 conmon[41853]: debug 2022-01-31T22:59:17.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:17.359208+0000) 2022-01-31T22:59:17.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:17 smithi171 conmon[51620]: debug 2022-01-31T22:59:17.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:17.535836+0000) 2022-01-31T22:59:18.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:17 smithi167 conmon[49112]: debug 2022-01-31T22:59:17.992+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:17.993617+0000) 2022-01-31T22:59:18.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:17 smithi167 conmon[60316]: debug 2022-01-31T22:59:17.920+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:17.921970+0000) 2022-01-31T22:59:18.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:18 smithi167 conmon[54076]: debug 2022-01-31T22:59:18.162+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:18.163005+0000) 2022-01-31T22:59:18.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:18 smithi171 conmon[46715]: debug 2022-01-31T22:59:18.155+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:18.156429+0000) 2022-01-31T22:59:18.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:18 smithi171 conmon[41853]: debug 2022-01-31T22:59:18.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:18.359361+0000) 2022-01-31T22:59:18.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:18 smithi171 conmon[51620]: debug 2022-01-31T22:59:18.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:18.536056+0000) 2022-01-31T22:59:19.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:18 smithi167 conmon[49112]: debug 2022-01-31T22:59:18.993+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:18.993822+0000) 2022-01-31T22:59:19.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:18 smithi167 conmon[60316]: debug 2022-01-31T22:59:18.920+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:18.922104+0000) 2022-01-31T22:59:19.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:19 smithi167 conmon[54076]: debug 2022-01-31T22:59:19.161+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:19.163181+0000) 2022-01-31T22:59:19.520 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:19 smithi171 conmon[46715]: debug 2022-01-31T22:59:19.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:19.156632+0000) 2022-01-31T22:59:19.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:19 smithi171 conmon[41853]: debug 2022-01-31T22:59:19.358+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:19.359558+0000) 2022-01-31T22:59:19.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:19 smithi171 conmon[51620]: debug 2022-01-31T22:59:19.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:19.536245+0000) 2022-01-31T22:59:20.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:19 smithi167 conmon[49112]: debug 2022-01-31T22:59:19.992+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:19.994016+0000) 2022-01-31T22:59:20.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:19 smithi167 conmon[60316]: debug 2022-01-31T22:59:19.921+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:19.922264+0000) 2022-01-31T22:59:20.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:20 smithi167 conmon[54076]: debug 2022-01-31T22:59:20.162+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.163342+0000) 2022-01-31T22:59:20.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[41853]: debug 2022-01-31T22:59:20.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.359719+0000) 2022-01-31T22:59:20.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[46715]: debug 2022-01-31T22:59:20.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.156860+0000) 2022-01-31T22:59:20.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[35325]: debug 2022-01-31T22:59:20.582+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253612 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:20.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[41853]: debug 2022-01-31T22:59:20.555+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.556234+0000) 2022-01-31T22:59:20.843 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[46715]: debug 2022-01-31T22:59:20.555+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.556412+0000) 2022-01-31T22:59:20.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[51620]: debug 2022-01-31T22:59:20.535+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.536447+0000) 2022-01-31T22:59:20.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:20 smithi171 conmon[51620]: debug 2022-01-31T22:59:20.556+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.557570+0000) 2022-01-31T22:59:20.877 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:20 smithi167 conmon[49112]: debug 2022-01-31T22:59:20.556+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.557751+0000) 2022-01-31T22:59:20.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:20 smithi167 conmon[54076]: debug 2022-01-31T22:59:20.555+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.557008+0000) 2022-01-31T22:59:20.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:20 smithi167 conmon[60316]: debug 2022-01-31T22:59:20.556+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.557521+0000) 2022-01-31T22:59:21.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:20 smithi167 conmon[49112]: debug 2022-01-31T22:59:20.993+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.994197+0000) 2022-01-31T22:59:21.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:20 smithi167 conmon[60316]: debug 2022-01-31T22:59:20.921+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:20.922458+0000) 2022-01-31T22:59:21.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:21 smithi167 conmon[54076]: debug 2022-01-31T22:59:21.162+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:21.163534+0000) 2022-01-31T22:59:21.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:21 smithi171 conmon[41853]: debug 2022-01-31T22:59:21.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:21.359911+0000) 2022-01-31T22:59:21.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:21 smithi171 conmon[46715]: debug 2022-01-31T22:59:21.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:21.157089+0000) 2022-01-31T22:59:21.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:21 smithi171 conmon[51620]: debug 2022-01-31T22:59:21.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:21.536623+0000) 2022-01-31T22:59:22.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:21 smithi167 conmon[49112]: debug 2022-01-31T22:59:21.993+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:21.994375+0000) 2022-01-31T22:59:22.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:21 smithi167 conmon[60316]: debug 2022-01-31T22:59:21.921+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:21.922629+0000) 2022-01-31T22:59:22.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:22 smithi167 conmon[54076]: debug 2022-01-31T22:59:22.162+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:22.163688+0000) 2022-01-31T22:59:22.520 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:22 smithi171 conmon[41853]: debug 2022-01-31T22:59:22.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:22.360073+0000) 2022-01-31T22:59:22.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:22 smithi171 conmon[46715]: debug 2022-01-31T22:59:22.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:22.157298+0000) 2022-01-31T22:59:22.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:22 smithi171 conmon[51620]: debug 2022-01-31T22:59:22.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:22.536794+0000) 2022-01-31T22:59:23.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:22 smithi167 conmon[49112]: debug 2022-01-31T22:59:22.993+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:22.994564+0000) 2022-01-31T22:59:23.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:22 smithi167 conmon[60316]: debug 2022-01-31T22:59:22.922+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:22.922815+0000) 2022-01-31T22:59:23.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:23 smithi167 conmon[54076]: debug 2022-01-31T22:59:23.162+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:23.163814+0000) 2022-01-31T22:59:23.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:23 smithi171 conmon[41853]: debug 2022-01-31T22:59:23.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:23.360274+0000) 2022-01-31T22:59:23.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:23 smithi171 conmon[46715]: debug 2022-01-31T22:59:23.156+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:23.157481+0000) 2022-01-31T22:59:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:23 smithi171 conmon[51620]: debug 2022-01-31T22:59:23.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:23.537002+0000) 2022-01-31T22:59:24.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:23 smithi167 conmon[49112]: debug 2022-01-31T22:59:23.993+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:23.994720+0000) 2022-01-31T22:59:24.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:23 smithi167 conmon[60316]: debug 2022-01-31T22:59:23.921+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:23.923022+0000) 2022-01-31T22:59:24.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:23 smithi171 conmon[35325]: debug 2022-01-31T22:59:23.943+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:59:24.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:24 smithi171 conmon[46715]: debug 2022-01-31T22:59:24.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:24.157708+0000) 2022-01-31T22:59:24.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:24 smithi167 conmon[54076]: debug 2022-01-31T22:59:24.162+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:24.163988+0000) 2022-01-31T22:59:24.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:24 smithi171 conmon[41853]: debug 2022-01-31T22:59:24.359+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:24.360479+0000) 2022-01-31T22:59:24.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:24 smithi171 conmon[51620]: debug 2022-01-31T22:59:24.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:24.537215+0000) 2022-01-31T22:59:25.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:24 smithi167 conmon[49112]: debug 2022-01-31T22:59:24.993+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:24.994915+0000) 2022-01-31T22:59:25.149 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:24 smithi167 conmon[60316]: debug 2022-01-31T22:59:24.921+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:24.923258+0000) 2022-01-31T22:59:25.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:25 smithi167 conmon[54076]: debug 2022-01-31T22:59:25.163+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.164122+0000) 2022-01-31T22:59:25.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[41853]: debug 2022-01-31T22:59:25.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.360662+0000) 2022-01-31T22:59:25.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[46715]: debug 2022-01-31T22:59:25.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.157909+0000) 2022-01-31T22:59:25.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[35325]: debug 2022-01-31T22:59:25.625+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253725 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:25.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[46715]: debug 2022-01-31T22:59:25.584+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.585329+0000) 2022-01-31T22:59:25.843 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[41853]: debug 2022-01-31T22:59:25.585+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.585741+0000) 2022-01-31T22:59:25.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[51620]: debug 2022-01-31T22:59:25.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.537432+0000) 2022-01-31T22:59:25.844 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:25 smithi171 conmon[51620]: debug 2022-01-31T22:59:25.585+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.586444+0000) 2022-01-31T22:59:25.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:25 smithi167 conmon[49112]: debug 2022-01-31T22:59:25.584+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.586135+0000) 2022-01-31T22:59:25.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:25 smithi167 conmon[54076]: debug 2022-01-31T22:59:25.584+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.585558+0000) 2022-01-31T22:59:25.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:25 smithi167 conmon[60316]: debug 2022-01-31T22:59:25.584+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.586434+0000) 2022-01-31T22:59:26.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:25 smithi167 conmon[49112]: debug 2022-01-31T22:59:25.994+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.995102+0000) 2022-01-31T22:59:26.148 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:25 smithi167 conmon[60316]: debug 2022-01-31T22:59:25.922+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:25.923440+0000) 2022-01-31T22:59:26.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:26 smithi167 conmon[54076]: debug 2022-01-31T22:59:26.163+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:26.164274+0000) 2022-01-31T22:59:26.521 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:26 smithi171 conmon[41853]: debug 2022-01-31T22:59:26.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:26.360814+0000) 2022-01-31T22:59:26.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:26 smithi171 conmon[46715]: debug 2022-01-31T22:59:26.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:26.158144+0000) 2022-01-31T22:59:26.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:26 smithi171 conmon[51620]: debug 2022-01-31T22:59:26.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:26.537588+0000) 2022-01-31T22:59:27.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:26 smithi167 conmon[49112]: debug 2022-01-31T22:59:26.994+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:26.995256+0000) 2022-01-31T22:59:27.149 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:26 smithi167 conmon[60316]: debug 2022-01-31T22:59:26.922+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:26.923608+0000) 2022-01-31T22:59:27.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:27 smithi167 conmon[54076]: debug 2022-01-31T22:59:27.163+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:27.164444+0000) 2022-01-31T22:59:27.521 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:27 smithi171 conmon[46715]: debug 2022-01-31T22:59:27.157+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:27.158289+0000) 2022-01-31T22:59:27.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:27 smithi171 conmon[41853]: debug 2022-01-31T22:59:27.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:27.360940+0000) 2022-01-31T22:59:27.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:27 smithi171 conmon[51620]: debug 2022-01-31T22:59:27.536+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:27.537714+0000) 2022-01-31T22:59:28.148 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:27 smithi167 conmon[49112]: debug 2022-01-31T22:59:27.994+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:27.995439+0000) 2022-01-31T22:59:28.149 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:27 smithi167 conmon[60316]: debug 2022-01-31T22:59:27.923+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:27.923794+0000) 2022-01-31T22:59:28.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:28 smithi167 conmon[54076]: debug 2022-01-31T22:59:28.164+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:28.164642+0000) 2022-01-31T22:59:28.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:28 smithi171 conmon[46715]: debug 2022-01-31T22:59:28.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:28.158489+0000) 2022-01-31T22:59:28.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:28 smithi171 conmon[41853]: debug 2022-01-31T22:59:28.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:28.361109+0000) 2022-01-31T22:59:28.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:28 smithi171 conmon[51620]: debug 2022-01-31T22:59:28.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:28.537860+0000) 2022-01-31T22:59:29.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:28 smithi167 conmon[49112]: debug 2022-01-31T22:59:28.995+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:28.995624+0000) 2022-01-31T22:59:29.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:28 smithi167 conmon[60316]: debug 2022-01-31T22:59:28.923+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:28.923974+0000) 2022-01-31T22:59:29.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:29 smithi167 conmon[54076]: debug 2022-01-31T22:59:29.164+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:29.164825+0000) 2022-01-31T22:59:29.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:29 smithi171 conmon[46715]: debug 2022-01-31T22:59:29.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:29.158679+0000) 2022-01-31T22:59:29.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:29 smithi171 conmon[41853]: debug 2022-01-31T22:59:29.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:29.361320+0000) 2022-01-31T22:59:29.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:29 smithi171 conmon[51620]: debug 2022-01-31T22:59:29.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:29.538027+0000) 2022-01-31T22:59:30.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:29 smithi167 conmon[49112]: debug 2022-01-31T22:59:29.995+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:29.995795+0000) 2022-01-31T22:59:30.149 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:29 smithi167 conmon[60316]: debug 2022-01-31T22:59:29.923+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:29.924165+0000) 2022-01-31T22:59:30.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:30 smithi167 conmon[54076]: debug 2022-01-31T22:59:30.164+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.165012+0000) 2022-01-31T22:59:30.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[46715]: debug 2022-01-31T22:59:30.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.158902+0000) 2022-01-31T22:59:30.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[41853]: debug 2022-01-31T22:59:30.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.361440+0000) 2022-01-31T22:59:30.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[35325]: debug 2022-01-31T22:59:30.654+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253837 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:30.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[41853]: debug 2022-01-31T22:59:30.627+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.628714+0000) 2022-01-31T22:59:30.843 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[46715]: debug 2022-01-31T22:59:30.627+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.628565+0000) 2022-01-31T22:59:30.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[51620]: debug 2022-01-31T22:59:30.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.538204+0000) 2022-01-31T22:59:30.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:30 smithi171 conmon[51620]: debug 2022-01-31T22:59:30.628+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.629516+0000) 2022-01-31T22:59:30.879 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:30 smithi167 conmon[49112]: debug 2022-01-31T22:59:30.628+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.629101+0000) 2022-01-31T22:59:30.879 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:30 smithi167 conmon[54076]: debug 2022-01-31T22:59:30.628+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.629029+0000) 2022-01-31T22:59:30.880 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:30 smithi167 conmon[60316]: debug 2022-01-31T22:59:30.628+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.629417+0000) 2022-01-31T22:59:31.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:30 smithi167 conmon[49112]: debug 2022-01-31T22:59:30.995+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.995980+0000) 2022-01-31T22:59:31.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:30 smithi167 conmon[60316]: debug 2022-01-31T22:59:30.923+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:30.924375+0000) 2022-01-31T22:59:31.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:31 smithi167 conmon[54076]: debug 2022-01-31T22:59:31.164+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:31.165250+0000) 2022-01-31T22:59:31.522 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:31 smithi171 conmon[41853]: debug 2022-01-31T22:59:31.360+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:31.361687+0000) 2022-01-31T22:59:31.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:31 smithi171 conmon[46715]: debug 2022-01-31T22:59:31.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:31.159112+0000) 2022-01-31T22:59:31.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:31 smithi171 conmon[51620]: debug 2022-01-31T22:59:31.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:31.538363+0000) 2022-01-31T22:59:32.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:31 smithi167 conmon[49112]: debug 2022-01-31T22:59:31.995+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:31.996132+0000) 2022-01-31T22:59:32.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:31 smithi167 conmon[60316]: debug 2022-01-31T22:59:31.924+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:31.924534+0000) 2022-01-31T22:59:32.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:32 smithi167 conmon[54076]: debug 2022-01-31T22:59:32.164+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:32.165406+0000) 2022-01-31T22:59:32.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:32 smithi171 conmon[46715]: debug 2022-01-31T22:59:32.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:32.159331+0000) 2022-01-31T22:59:32.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:32 smithi171 conmon[41853]: debug 2022-01-31T22:59:32.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:32.361835+0000) 2022-01-31T22:59:32.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:32 smithi171 conmon[51620]: debug 2022-01-31T22:59:32.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:32.538504+0000) 2022-01-31T22:59:33.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:32 smithi167 conmon[49112]: debug 2022-01-31T22:59:32.995+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:32.996264+0000) 2022-01-31T22:59:33.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:32 smithi167 conmon[60316]: debug 2022-01-31T22:59:32.924+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:32.924719+0000) 2022-01-31T22:59:33.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:33 smithi167 conmon[54076]: debug 2022-01-31T22:59:33.165+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:33.165531+0000) 2022-01-31T22:59:33.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:33 smithi171 conmon[46715]: debug 2022-01-31T22:59:33.158+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:33.159491+0000) 2022-01-31T22:59:33.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:33 smithi171 conmon[41853]: debug 2022-01-31T22:59:33.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:33.361936+0000) 2022-01-31T22:59:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:33 smithi171 conmon[51620]: debug 2022-01-31T22:59:33.537+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:33.538708+0000) 2022-01-31T22:59:34.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:33 smithi167 conmon[49112]: debug 2022-01-31T22:59:33.995+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:33.996389+0000) 2022-01-31T22:59:34.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:33 smithi167 conmon[60316]: debug 2022-01-31T22:59:33.924+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:33.924873+0000) 2022-01-31T22:59:34.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:34 smithi167 conmon[54076]: debug 2022-01-31T22:59:34.165+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:34.165724+0000) 2022-01-31T22:59:34.522 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:34 smithi171 conmon[46715]: debug 2022-01-31T22:59:34.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:34.159675+0000) 2022-01-31T22:59:34.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:34 smithi171 conmon[41853]: debug 2022-01-31T22:59:34.361+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:34.362093+0000) 2022-01-31T22:59:34.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:34 smithi171 conmon[51620]: debug 2022-01-31T22:59:34.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:34.538891+0000) 2022-01-31T22:59:35.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:34 smithi167 conmon[49112]: debug 2022-01-31T22:59:34.996+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:34.996591+0000) 2022-01-31T22:59:35.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:34 smithi167 conmon[60316]: debug 2022-01-31T22:59:34.924+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:34.925069+0000) 2022-01-31T22:59:35.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:35 smithi167 conmon[54076]: debug 2022-01-31T22:59:35.165+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.165913+0000) 2022-01-31T22:59:35.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[46715]: debug 2022-01-31T22:59:35.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.159805+0000) 2022-01-31T22:59:35.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[41853]: debug 2022-01-31T22:59:35.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.362279+0000) 2022-01-31T22:59:35.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[35325]: debug 2022-01-31T22:59:35.683+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 253950 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:35.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[41853]: debug 2022-01-31T22:59:35.656+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.657616+0000) 2022-01-31T22:59:35.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[46715]: debug 2022-01-31T22:59:35.655+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.656632+0000) 2022-01-31T22:59:35.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[51620]: debug 2022-01-31T22:59:35.538+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.539068+0000) 2022-01-31T22:59:35.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:35 smithi171 conmon[51620]: debug 2022-01-31T22:59:35.656+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.656813+0000) 2022-01-31T22:59:35.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:35 smithi167 conmon[49112]: debug 2022-01-31T22:59:35.657+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.657852+0000) 2022-01-31T22:59:35.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:35 smithi167 conmon[54076]: debug 2022-01-31T22:59:35.657+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.657645+0000) 2022-01-31T22:59:35.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:35 smithi167 conmon[60316]: debug 2022-01-31T22:59:35.657+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.658149+0000) 2022-01-31T22:59:36.356 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:35 smithi167 conmon[49112]: debug 2022-01-31T22:59:35.996+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.996739+0000) 2022-01-31T22:59:36.357 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:36 smithi167 conmon[54076]: debug 2022-01-31T22:59:36.165+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:36.166102+0000) 2022-01-31T22:59:36.358 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:35 smithi167 conmon[60316]: debug 2022-01-31T22:59:35.924+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:35.925284+0000) 2022-01-31T22:59:36.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:36 smithi171 conmon[41853]: debug 2022-01-31T22:59:36.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:36.362418+0000) 2022-01-31T22:59:36.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:36 smithi171 conmon[46715]: debug 2022-01-31T22:59:36.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:36.160006+0000) 2022-01-31T22:59:36.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:36 smithi171 conmon[51620]: debug 2022-01-31T22:59:36.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:36.539220+0000) 2022-01-31T22:59:37.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:36 smithi167 conmon[49112]: debug 2022-01-31T22:59:36.996+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:36.996863+0000) 2022-01-31T22:59:37.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:36 smithi167 conmon[60316]: debug 2022-01-31T22:59:36.925+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:36.925458+0000) 2022-01-31T22:59:37.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:37 smithi167 conmon[54076]: debug 2022-01-31T22:59:37.165+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:37.166253+0000) 2022-01-31T22:59:37.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:37 smithi171 conmon[41853]: debug 2022-01-31T22:59:37.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:37.362563+0000) 2022-01-31T22:59:37.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:37 smithi171 conmon[46715]: debug 2022-01-31T22:59:37.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:37.160163+0000) 2022-01-31T22:59:37.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:37 smithi171 conmon[51620]: debug 2022-01-31T22:59:37.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:37.539377+0000) 2022-01-31T22:59:38.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:37 smithi167 conmon[49112]: debug 2022-01-31T22:59:37.996+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:37.997061+0000) 2022-01-31T22:59:38.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:37 smithi167 conmon[60316]: debug 2022-01-31T22:59:37.925+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:37.925679+0000) 2022-01-31T22:59:38.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:38 smithi167 conmon[54076]: debug 2022-01-31T22:59:38.166+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:38.166441+0000) 2022-01-31T22:59:38.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:38 smithi171 conmon[46715]: debug 2022-01-31T22:59:38.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:38.160338+0000) 2022-01-31T22:59:38.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:38 smithi171 conmon[41853]: debug 2022-01-31T22:59:38.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:38.362749+0000) 2022-01-31T22:59:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:38 smithi171 conmon[51620]: debug 2022-01-31T22:59:38.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:38.539581+0000) 2022-01-31T22:59:39.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:38 smithi167 conmon[60316]: debug 2022-01-31T22:59:38.925+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:38.925818+0000) 2022-01-31T22:59:39.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:38 smithi167 conmon[49112]: debug 2022-01-31T22:59:38.996+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:38.997214+0000) 2022-01-31T22:59:39.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:38 smithi171 conmon[35325]: debug 2022-01-31T22:59:38.944+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:59:39.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:39 smithi171 conmon[46715]: debug 2022-01-31T22:59:39.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:39.160546+0000) 2022-01-31T22:59:39.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:39 smithi167 conmon[54076]: debug 2022-01-31T22:59:39.166+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:39.166654+0000) 2022-01-31T22:59:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:39 smithi171 conmon[51620]: debug 2022-01-31T22:59:39.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:39.539779+0000) 2022-01-31T22:59:39.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:39 smithi171 conmon[41853]: debug 2022-01-31T22:59:39.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:39.362913+0000) 2022-01-31T22:59:40.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:39 smithi167 conmon[49112]: debug 2022-01-31T22:59:39.996+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:39.997401+0000) 2022-01-31T22:59:40.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:39 smithi167 conmon[60316]: debug 2022-01-31T22:59:39.925+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:39.925952+0000) 2022-01-31T22:59:40.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:40 smithi167 conmon[54076]: debug 2022-01-31T22:59:40.166+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.166843+0000) 2022-01-31T22:59:40.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[46715]: debug 2022-01-31T22:59:40.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.160799+0000) 2022-01-31T22:59:40.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[41853]: debug 2022-01-31T22:59:40.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.363018+0000) 2022-01-31T22:59:40.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[35325]: debug 2022-01-31T22:59:40.711+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254063 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:40.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[41853]: debug 2022-01-31T22:59:40.685+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.686303+0000) 2022-01-31T22:59:40.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[46715]: debug 2022-01-31T22:59:40.685+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.686009+0000) 2022-01-31T22:59:40.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[51620]: debug 2022-01-31T22:59:40.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.539942+0000) 2022-01-31T22:59:40.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:40 smithi171 conmon[51620]: debug 2022-01-31T22:59:40.686+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.687318+0000) 2022-01-31T22:59:40.954 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:40 smithi167 conmon[54076]: debug 2022-01-31T22:59:40.687+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.687655+0000) 2022-01-31T22:59:40.955 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:40 smithi167 conmon[60316]: debug 2022-01-31T22:59:40.686+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.686559+0000) 2022-01-31T22:59:40.956 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:40 smithi167 conmon[60316]: debug 2022-01-31T22:59:40.925+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.926115+0000) 2022-01-31T22:59:40.956 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:40 smithi167 conmon[49112]: debug 2022-01-31T22:59:40.686+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.687447+0000) 2022-01-31T22:59:41.357 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:40 smithi167 conmon[49112]: debug 2022-01-31T22:59:40.997+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:40.997607+0000) 2022-01-31T22:59:41.358 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:41 smithi167 conmon[54076]: debug 2022-01-31T22:59:41.166+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:41.167056+0000) 2022-01-31T22:59:41.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:41 smithi171 conmon[41853]: debug 2022-01-31T22:59:41.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:41.363193+0000) 2022-01-31T22:59:41.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:41 smithi171 conmon[46715]: debug 2022-01-31T22:59:41.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:41.160974+0000) 2022-01-31T22:59:41.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:41 smithi171 conmon[51620]: debug 2022-01-31T22:59:41.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:41.540068+0000) 2022-01-31T22:59:42.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:41 smithi167 conmon[49112]: debug 2022-01-31T22:59:41.997+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:41.997745+0000) 2022-01-31T22:59:42.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:41 smithi167 conmon[60316]: debug 2022-01-31T22:59:41.925+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:41.926270+0000) 2022-01-31T22:59:42.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:42 smithi167 conmon[54076]: debug 2022-01-31T22:59:42.166+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:42.167213+0000) 2022-01-31T22:59:42.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:42 smithi171 conmon[41853]: debug 2022-01-31T22:59:42.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:42.363350+0000) 2022-01-31T22:59:42.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:42 smithi171 conmon[46715]: debug 2022-01-31T22:59:42.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:42.161160+0000) 2022-01-31T22:59:42.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:42 smithi171 conmon[51620]: debug 2022-01-31T22:59:42.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:42.540216+0000) 2022-01-31T22:59:43.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:42 smithi167 conmon[49112]: debug 2022-01-31T22:59:42.997+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:42.997954+0000) 2022-01-31T22:59:43.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:42 smithi167 conmon[60316]: debug 2022-01-31T22:59:42.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:42.926496+0000) 2022-01-31T22:59:43.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:43 smithi167 conmon[54076]: debug 2022-01-31T22:59:43.167+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:43.167402+0000) 2022-01-31T22:59:43.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:43 smithi171 conmon[41853]: debug 2022-01-31T22:59:43.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:43.363524+0000) 2022-01-31T22:59:43.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:43 smithi171 conmon[46715]: debug 2022-01-31T22:59:43.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:43.161298+0000) 2022-01-31T22:59:43.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:43 smithi171 conmon[51620]: debug 2022-01-31T22:59:43.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:43.540347+0000) 2022-01-31T22:59:44.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:43 smithi167 conmon[49112]: debug 2022-01-31T22:59:43.997+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:43.998105+0000) 2022-01-31T22:59:44.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:43 smithi167 conmon[60316]: debug 2022-01-31T22:59:43.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:43.926697+0000) 2022-01-31T22:59:44.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:44 smithi167 conmon[54076]: debug 2022-01-31T22:59:44.167+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:44.167569+0000) 2022-01-31T22:59:44.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:44 smithi171 conmon[46715]: debug 2022-01-31T22:59:44.159+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:44.161493+0000) 2022-01-31T22:59:44.525 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:44 smithi171 conmon[41853]: debug 2022-01-31T22:59:44.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:44.363654+0000) 2022-01-31T22:59:44.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:44 smithi171 conmon[51620]: debug 2022-01-31T22:59:44.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:44.540491+0000) 2022-01-31T22:59:45.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:44 smithi167 conmon[49112]: debug 2022-01-31T22:59:44.997+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:44.998322+0000) 2022-01-31T22:59:45.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:44 smithi167 conmon[60316]: debug 2022-01-31T22:59:44.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:44.926825+0000) 2022-01-31T22:59:45.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:45 smithi167 conmon[54076]: debug 2022-01-31T22:59:45.167+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.167772+0000) 2022-01-31T22:59:45.524 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[46715]: debug 2022-01-31T22:59:45.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.161670+0000) 2022-01-31T22:59:45.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[41853]: debug 2022-01-31T22:59:45.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.363805+0000) 2022-01-31T22:59:45.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[35325]: debug 2022-01-31T22:59:45.740+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254175 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:45.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[41853]: debug 2022-01-31T22:59:45.712+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.714689+0000) 2022-01-31T22:59:45.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[46715]: debug 2022-01-31T22:59:45.713+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.715628+0000) 2022-01-31T22:59:45.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[51620]: debug 2022-01-31T22:59:45.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.540605+0000) 2022-01-31T22:59:45.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:45 smithi171 conmon[51620]: debug 2022-01-31T22:59:45.713+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.714843+0000) 2022-01-31T22:59:45.955 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:45 smithi167 conmon[49112]: debug 2022-01-31T22:59:45.714+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.715356+0000) 2022-01-31T22:59:45.956 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:45 smithi167 conmon[54076]: debug 2022-01-31T22:59:45.715+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.715519+0000) 2022-01-31T22:59:45.956 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:45 smithi167 conmon[60316]: debug 2022-01-31T22:59:45.715+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.715904+0000) 2022-01-31T22:59:45.957 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:45 smithi167 conmon[60316]: debug 2022-01-31T22:59:45.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.927022+0000) 2022-01-31T22:59:46.358 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:45 smithi167 conmon[49112]: debug 2022-01-31T22:59:45.998+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:45.998529+0000) 2022-01-31T22:59:46.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:46 smithi167 conmon[54076]: debug 2022-01-31T22:59:46.167+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:46.167994+0000) 2022-01-31T22:59:46.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:46 smithi171 conmon[41853]: debug 2022-01-31T22:59:46.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:46.363955+0000) 2022-01-31T22:59:46.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:46 smithi171 conmon[46715]: debug 2022-01-31T22:59:46.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:46.161857+0000) 2022-01-31T22:59:46.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:46 smithi171 conmon[51620]: debug 2022-01-31T22:59:46.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:46.540724+0000) 2022-01-31T22:59:47.007 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:46 smithi167 conmon[49112]: debug 2022-01-31T22:59:46.998+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:46.998592+0000) 2022-01-31T22:59:47.008 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:46 smithi167 conmon[60316]: debug 2022-01-31T22:59:46.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:46.927170+0000) 2022-01-31T22:59:47.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:47 smithi167 conmon[54076]: debug 2022-01-31T22:59:47.167+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:47.168154+0000) 2022-01-31T22:59:47.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:47 smithi171 conmon[46715]: debug 2022-01-31T22:59:47.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:47.162006+0000) 2022-01-31T22:59:47.525 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:47 smithi171 conmon[41853]: debug 2022-01-31T22:59:47.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:47.364098+0000) 2022-01-31T22:59:47.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:47 smithi171 conmon[51620]: debug 2022-01-31T22:59:47.539+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:47.540821+0000) 2022-01-31T22:59:48.011 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:47 smithi167 conmon[49112]: debug 2022-01-31T22:59:47.998+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:47.998770+0000) 2022-01-31T22:59:48.012 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:47 smithi167 conmon[60316]: debug 2022-01-31T22:59:47.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:47.927315+0000) 2022-01-31T22:59:48.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:48 smithi167 conmon[54076]: debug 2022-01-31T22:59:48.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:48.168343+0000) 2022-01-31T22:59:48.524 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:48 smithi171 conmon[41853]: debug 2022-01-31T22:59:48.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:48.364287+0000) 2022-01-31T22:59:48.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:48 smithi171 conmon[46715]: debug 2022-01-31T22:59:48.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:48.162233+0000) 2022-01-31T22:59:48.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:48 smithi171 conmon[51620]: debug 2022-01-31T22:59:48.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:48.540998+0000) 2022-01-31T22:59:49.015 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:48 smithi167 conmon[49112]: debug 2022-01-31T22:59:48.998+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:48.998955+0000) 2022-01-31T22:59:49.016 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:48 smithi167 conmon[60316]: debug 2022-01-31T22:59:48.926+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:48.927472+0000) 2022-01-31T22:59:49.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:49 smithi167 conmon[54076]: debug 2022-01-31T22:59:49.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:49.168567+0000) 2022-01-31T22:59:49.525 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:49 smithi171 conmon[41853]: debug 2022-01-31T22:59:49.362+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:49.364502+0000) 2022-01-31T22:59:49.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:49 smithi171 conmon[46715]: debug 2022-01-31T22:59:49.160+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:49.162428+0000) 2022-01-31T22:59:49.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:49 smithi171 conmon[51620]: debug 2022-01-31T22:59:49.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:49.541139+0000) 2022-01-31T22:59:50.019 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:49 smithi167 conmon[49112]: debug 2022-01-31T22:59:49.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:49.999214+0000) 2022-01-31T22:59:50.020 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:49 smithi167 conmon[60316]: debug 2022-01-31T22:59:49.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:49.927791+0000) 2022-01-31T22:59:50.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:50 smithi167 conmon[54076]: debug 2022-01-31T22:59:50.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.168816+0000) 2022-01-31T22:59:50.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[46715]: debug 2022-01-31T22:59:50.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.162690+0000) 2022-01-31T22:59:50.525 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[41853]: debug 2022-01-31T22:59:50.363+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.364784+0000) 2022-01-31T22:59:50.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[35325]: debug 2022-01-31T22:59:50.768+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254286 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:50.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[41853]: debug 2022-01-31T22:59:50.743+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.744918+0000) 2022-01-31T22:59:50.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[46715]: debug 2022-01-31T22:59:50.744+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.745931+0000) 2022-01-31T22:59:50.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[51620]: debug 2022-01-31T22:59:50.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.541395+0000) 2022-01-31T22:59:50.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:50 smithi171 conmon[51620]: debug 2022-01-31T22:59:50.743+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.744788+0000) 2022-01-31T22:59:50.983 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:50 smithi167 conmon[49112]: debug 2022-01-31T22:59:50.743+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.744462+0000) 2022-01-31T22:59:50.984 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:50 smithi167 conmon[54076]: debug 2022-01-31T22:59:50.745+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.745671+0000) 2022-01-31T22:59:50.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:50 smithi167 conmon[60316]: debug 2022-01-31T22:59:50.744+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.745481+0000) 2022-01-31T22:59:50.984 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:50 smithi167 conmon[60316]: debug 2022-01-31T22:59:50.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.927994+0000) 2022-01-31T22:59:51.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:51 smithi167 conmon[54076]: debug 2022-01-31T22:59:51.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:51.168983+0000) 2022-01-31T22:59:51.360 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:50 smithi167 conmon[49112]: debug 2022-01-31T22:59:50.998+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:50.999374+0000) 2022-01-31T22:59:51.525 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:51 smithi171 conmon[41853]: debug 2022-01-31T22:59:51.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:51.364941+0000) 2022-01-31T22:59:51.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:51 smithi171 conmon[46715]: debug 2022-01-31T22:59:51.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:51.162868+0000) 2022-01-31T22:59:51.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:51 smithi171 conmon[51620]: debug 2022-01-31T22:59:51.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:51.541554+0000) 2022-01-31T22:59:52.026 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:51 smithi167 conmon[49112]: debug 2022-01-31T22:59:51.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:51.999520+0000) 2022-01-31T22:59:52.027 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:51 smithi167 conmon[60316]: debug 2022-01-31T22:59:51.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:51.928162+0000) 2022-01-31T22:59:52.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:52 smithi167 conmon[54076]: debug 2022-01-31T22:59:52.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:52.169110+0000) 2022-01-31T22:59:52.525 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:52 smithi171 conmon[46715]: debug 2022-01-31T22:59:52.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:52.163040+0000) 2022-01-31T22:59:52.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:52 smithi171 conmon[41853]: debug 2022-01-31T22:59:52.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:52.365086+0000) 2022-01-31T22:59:52.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:52 smithi171 conmon[51620]: debug 2022-01-31T22:59:52.540+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:52.541702+0000) 2022-01-31T22:59:53.030 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:52 smithi167 conmon[49112]: debug 2022-01-31T22:59:52.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:52.999676+0000) 2022-01-31T22:59:53.031 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:52 smithi167 conmon[60316]: debug 2022-01-31T22:59:52.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:52.928292+0000) 2022-01-31T22:59:53.359 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:53 smithi167 conmon[54076]: debug 2022-01-31T22:59:53.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:53.169304+0000) 2022-01-31T22:59:53.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:53 smithi171 conmon[41853]: debug 2022-01-31T22:59:53.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:53.365248+0000) 2022-01-31T22:59:53.526 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:53 smithi171 conmon[46715]: debug 2022-01-31T22:59:53.161+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:53.163211+0000) 2022-01-31T22:59:53.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:53 smithi171 conmon[51620]: debug 2022-01-31T22:59:53.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:53.541928+0000) 2022-01-31T22:59:54.034 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:54 smithi167 conmon[49112]: debug 2022-01-31T22:59:53.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:53.999854+0000) 2022-01-31T22:59:54.035 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:53 smithi167 conmon[60316]: debug 2022-01-31T22:59:53.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:53.928447+0000) 2022-01-31T22:59:54.289 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:53 smithi171 conmon[35325]: debug 2022-01-31T22:59:53.944+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T22:59:54.290 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:54 smithi171 conmon[46715]: debug 2022-01-31T22:59:54.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:54.163383+0000) 2022-01-31T22:59:54.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:54 smithi167 conmon[54076]: debug 2022-01-31T22:59:54.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:54.169540+0000) 2022-01-31T22:59:54.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:54 smithi171 conmon[41853]: debug 2022-01-31T22:59:54.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:54.365406+0000) 2022-01-31T22:59:54.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:54 smithi171 conmon[51620]: debug 2022-01-31T22:59:54.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:54.542110+0000) 2022-01-31T22:59:55.037 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:55 smithi167 conmon[49112]: debug 2022-01-31T22:59:54.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.000044+0000) 2022-01-31T22:59:55.038 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:54 smithi167 conmon[60316]: debug 2022-01-31T22:59:54.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:54.928593+0000) 2022-01-31T22:59:55.349 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[46715]: debug 2022-01-31T22:59:55.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.163534+0000) 2022-01-31T22:59:55.360 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:55 smithi167 conmon[54076]: debug 2022-01-31T22:59:55.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.169737+0000) 2022-01-31T22:59:55.756 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[41853]: debug 2022-01-31T22:59:55.364+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.365587+0000) 2022-01-31T22:59:55.757 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[51620]: debug 2022-01-31T22:59:55.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.542311+0000) 2022-01-31T22:59:56.041 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:55 smithi167 conmon[54076]: debug 2022-01-31T22:59:55.773+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.774556+0000) 2022-01-31T22:59:56.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:55 smithi167 conmon[60316]: debug 2022-01-31T22:59:55.773+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.774756+0000) 2022-01-31T22:59:56.042 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:55 smithi167 conmon[60316]: debug 2022-01-31T22:59:55.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.928753+0000) 2022-01-31T22:59:56.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:55 smithi167 conmon[49112]: debug 2022-01-31T22:59:55.771+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.772741+0000) 2022-01-31T22:59:56.043 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:56 smithi167 conmon[49112]: debug 2022-01-31T22:59:55.998+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:56.000188+0000) 2022-01-31T22:59:56.091 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[35325]: debug 2022-01-31T22:59:55.798+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254398 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T22:59:56.092 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[41853]: debug 2022-01-31T22:59:55.773+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.773905+0000) 2022-01-31T22:59:56.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[46715]: debug 2022-01-31T22:59:55.773+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.774495+0000) 2022-01-31T22:59:56.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:55 smithi171 conmon[51620]: debug 2022-01-31T22:59:55.771+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:55.772660+0000) 2022-01-31T22:59:56.300 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:56 smithi167 conmon[54076]: debug 2022-01-31T22:59:56.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:56.169918+0000) 2022-01-31T22:59:56.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:56 smithi171 conmon[41853]: debug 2022-01-31T22:59:56.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:56.365959+0000) 2022-01-31T22:59:56.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:56 smithi171 conmon[46715]: debug 2022-01-31T22:59:56.162+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:56.163671+0000) 2022-01-31T22:59:56.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:56 smithi171 conmon[51620]: debug 2022-01-31T22:59:56.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:56.542462+0000) 2022-01-31T22:59:57.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:57 smithi167 conmon[49112]: debug 2022-01-31T22:59:56.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:57.000339+0000) 2022-01-31T22:59:57.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:56 smithi167 conmon[60316]: debug 2022-01-31T22:59:56.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:56.928894+0000) 2022-01-31T22:59:57.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:57 smithi171 conmon[41853]: debug 2022-01-31T22:59:57.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:57.366428+0000) 2022-01-31T22:59:57.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:57 smithi171 conmon[46715]: debug 2022-01-31T22:59:57.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:57.163873+0000) 2022-01-31T22:59:57.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:57 smithi167 conmon[54076]: debug 2022-01-31T22:59:57.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:57.170017+0000) 2022-01-31T22:59:57.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:57 smithi171 conmon[51620]: debug 2022-01-31T22:59:57.541+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:57.542597+0000) 2022-01-31T22:59:58.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:58 smithi167 conmon[49112]: debug 2022-01-31T22:59:57.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:58.000548+0000) 2022-01-31T22:59:58.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:57 smithi167 conmon[60316]: debug 2022-01-31T22:59:57.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:57.929082+0000) 2022-01-31T22:59:58.526 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:58 smithi171 conmon[41853]: debug 2022-01-31T22:59:58.365+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:58.366656+0000) 2022-01-31T22:59:58.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:58 smithi171 conmon[46715]: debug 2022-01-31T22:59:58.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:58.164030+0000) 2022-01-31T22:59:58.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:58 smithi167 conmon[54076]: debug 2022-01-31T22:59:58.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:58.170127+0000) 2022-01-31T22:59:58.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:58 smithi171 conmon[51620]: debug 2022-01-31T22:59:58.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:58.542765+0000) 2022-01-31T22:59:59.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 22:59:59 smithi167 conmon[49112]: debug 2022-01-31T22:59:58.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:59.000733+0000) 2022-01-31T22:59:59.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:58 smithi167 conmon[60316]: debug 2022-01-31T22:59:58.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:58.929277+0000) 2022-01-31T22:59:59.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 22:59:59 smithi171 conmon[46715]: debug 2022-01-31T22:59:59.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:59.164150+0000) 2022-01-31T22:59:59.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 22:59:59 smithi171 conmon[41853]: debug 2022-01-31T22:59:59.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:59.366876+0000) 2022-01-31T22:59:59.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 22:59:59 smithi167 conmon[54076]: debug 2022-01-31T22:59:59.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:59.170250+0000) 2022-01-31T22:59:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 22:59:59 smithi171 conmon[51620]: debug 2022-01-31T22:59:59.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:59.542930+0000) 2022-01-31T23:00:00.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:00 smithi167 conmon[49112]: debug 2022-01-31T22:59:59.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.000878+0000) 2022-01-31T23:00:00.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 22:59:59 smithi167 conmon[60316]: debug 2022-01-31T22:59:59.927+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T21:59:59.929456+0000) 2022-01-31T23:00:00.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[41853]: debug 2022-01-31T23:00:00.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.367082+0000) 2022-01-31T23:00:00.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[46715]: debug 2022-01-31T23:00:00.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.164348+0000) 2022-01-31T23:00:00.653 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:00 smithi167 conmon[54076]: debug 2022-01-31T23:00:00.168+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.170369+0000) 2022-01-31T23:00:00.785 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[51620]: debug 2022-01-31T23:00:00.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.543103+0000) 2022-01-31T23:00:01.058 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:00 smithi167 conmon[49112]: debug 2022-01-31T23:00:00.799+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.801481+0000) 2022-01-31T23:00:01.059 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:01 smithi167 conmon[49112]: debug 2022-01-31T23:00:01.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:01.001018+0000) 2022-01-31T23:00:01.059 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:00 smithi167 conmon[54076]: debug 2022-01-31T23:00:00.801+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.803079+0000) 2022-01-31T23:00:01.060 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:00 smithi167 conmon[60316]: debug 2022-01-31T23:00:00.800+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.802175+0000) 2022-01-31T23:00:01.060 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:00 smithi167 conmon[60316]: debug 2022-01-31T23:00:00.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.929699+0000) 2022-01-31T23:00:01.091 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[35325]: debug 2022-01-31T23:00:00.826+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254509 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:01.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[41853]: debug 2022-01-31T23:00:00.801+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.802035+0000) 2022-01-31T23:00:01.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[46715]: debug 2022-01-31T23:00:00.802+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.803218+0000) 2022-01-31T23:00:01.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:00 smithi171 conmon[51620]: debug 2022-01-31T23:00:00.801+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:00.801908+0000) 2022-01-31T23:00:01.361 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:01 smithi167 conmon[54076]: debug 2022-01-31T23:00:01.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:01.170545+0000) 2022-01-31T23:00:01.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:01 smithi171 conmon[41853]: debug 2022-01-31T23:00:01.366+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:01.367301+0000) 2022-01-31T23:00:01.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:01 smithi171 conmon[46715]: debug 2022-01-31T23:00:01.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:01.164531+0000) 2022-01-31T23:00:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:01 smithi171 conmon[51620]: debug 2022-01-31T23:00:01.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:01.543293+0000) 2022-01-31T23:00:02.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:01 smithi167 conmon[60316]: debug 2022-01-31T23:00:01.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:01.929853+0000) 2022-01-31T23:00:02.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:02 smithi167 conmon[49112]: debug 2022-01-31T23:00:01.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:02.001105+0000) 2022-01-31T23:00:02.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:02 smithi167 conmon[54076]: debug 2022-01-31T23:00:02.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:02.170629+0000) 2022-01-31T23:00:02.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:02 smithi171 conmon[41853]: debug 2022-01-31T23:00:02.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:02.368884+0000) 2022-01-31T23:00:02.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:02 smithi171 conmon[46715]: debug 2022-01-31T23:00:02.163+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:02.164678+0000) 2022-01-31T23:00:02.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:02 smithi171 conmon[51620]: debug 2022-01-31T23:00:02.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:02.543469+0000) 2022-01-31T23:00:03.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:03 smithi167 conmon[49112]: debug 2022-01-31T23:00:02.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:03.001218+0000) 2022-01-31T23:00:03.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:02 smithi167 conmon[60316]: debug 2022-01-31T23:00:02.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:02.930002+0000) 2022-01-31T23:00:03.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:03 smithi167 conmon[54076]: debug 2022-01-31T23:00:03.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:03.170751+0000) 2022-01-31T23:00:03.527 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:03 smithi171 conmon[41853]: debug 2022-01-31T23:00:03.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:03.369038+0000) 2022-01-31T23:00:03.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:03 smithi171 conmon[46715]: debug 2022-01-31T23:00:03.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:03.164844+0000) 2022-01-31T23:00:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:03 smithi171 conmon[51620]: debug 2022-01-31T23:00:03.542+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:03.543649+0000) 2022-01-31T23:00:04.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:04 smithi167 conmon[49112]: debug 2022-01-31T23:00:03.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:04.001314+0000) 2022-01-31T23:00:04.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:03 smithi167 conmon[60316]: debug 2022-01-31T23:00:03.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:03.930207+0000) 2022-01-31T23:00:04.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:04 smithi167 conmon[54076]: debug 2022-01-31T23:00:04.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:04.170882+0000) 2022-01-31T23:00:04.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:04 smithi171 conmon[46715]: debug 2022-01-31T23:00:04.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:04.165033+0000) 2022-01-31T23:00:04.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:04 smithi171 conmon[41853]: debug 2022-01-31T23:00:04.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:04.369218+0000) 2022-01-31T23:00:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:04 smithi171 conmon[51620]: debug 2022-01-31T23:00:04.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:04.543847+0000) 2022-01-31T23:00:05.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:05 smithi167 conmon[49112]: debug 2022-01-31T23:00:04.999+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.001434+0000) 2022-01-31T23:00:05.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:04 smithi167 conmon[60316]: debug 2022-01-31T23:00:04.928+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:04.930407+0000) 2022-01-31T23:00:05.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:05 smithi167 conmon[54076]: debug 2022-01-31T23:00:05.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.171005+0000) 2022-01-31T23:00:05.527 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[46715]: debug 2022-01-31T23:00:05.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.165128+0000) 2022-01-31T23:00:05.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[41853]: debug 2022-01-31T23:00:05.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.369405+0000) 2022-01-31T23:00:05.813 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[51620]: debug 2022-01-31T23:00:05.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.544026+0000) 2022-01-31T23:00:06.078 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:05 smithi167 conmon[54076]: debug 2022-01-31T23:00:05.828+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.830259+0000) 2022-01-31T23:00:06.079 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:05 smithi167 conmon[60316]: debug 2022-01-31T23:00:05.830+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.831490+0000) 2022-01-31T23:00:06.080 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:05 smithi167 conmon[60316]: debug 2022-01-31T23:00:05.929+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.930643+0000) 2022-01-31T23:00:06.080 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:05 smithi167 conmon[49112]: debug 2022-01-31T23:00:05.828+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.829706+0000) 2022-01-31T23:00:06.081 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:06 smithi167 conmon[49112]: debug 2022-01-31T23:00:06.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:06.001627+0000) 2022-01-31T23:00:06.091 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[35325]: debug 2022-01-31T23:00:05.856+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254621 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:06.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[41853]: debug 2022-01-31T23:00:05.829+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.830604+0000) 2022-01-31T23:00:06.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[46715]: debug 2022-01-31T23:00:05.830+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.831011+0000) 2022-01-31T23:00:06.093 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:05 smithi171 conmon[51620]: debug 2022-01-31T23:00:05.828+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:05.829569+0000) 2022-01-31T23:00:06.362 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:06 smithi167 conmon[54076]: debug 2022-01-31T23:00:06.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:06.171170+0000) 2022-01-31T23:00:06.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:06 smithi171 conmon[41853]: debug 2022-01-31T23:00:06.368+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:06.369608+0000) 2022-01-31T23:00:06.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:06 smithi171 conmon[46715]: debug 2022-01-31T23:00:06.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:06.165258+0000) 2022-01-31T23:00:06.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:06 smithi171 conmon[51620]: debug 2022-01-31T23:00:06.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:06.544164+0000) 2022-01-31T23:00:07.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:07 smithi167 conmon[49112]: debug 2022-01-31T23:00:07.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:07.001739+0000) 2022-01-31T23:00:07.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:06 smithi167 conmon[60316]: debug 2022-01-31T23:00:06.930+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:06.930803+0000) 2022-01-31T23:00:07.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:07 smithi167 conmon[54076]: debug 2022-01-31T23:00:07.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:07.171295+0000) 2022-01-31T23:00:07.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:07 smithi171 conmon[46715]: debug 2022-01-31T23:00:07.164+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:07.165406+0000) 2022-01-31T23:00:07.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:07 smithi171 conmon[41853]: debug 2022-01-31T23:00:07.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:07.369778+0000) 2022-01-31T23:00:07.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:07 smithi171 conmon[51620]: debug 2022-01-31T23:00:07.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:07.544330+0000) 2022-01-31T23:00:08.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:08 smithi167 conmon[49112]: debug 2022-01-31T23:00:08.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:08.001876+0000) 2022-01-31T23:00:08.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:07 smithi167 conmon[60316]: debug 2022-01-31T23:00:07.929+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:07.930936+0000) 2022-01-31T23:00:08.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:08 smithi167 conmon[54076]: debug 2022-01-31T23:00:08.169+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:08.171430+0000) 2022-01-31T23:00:08.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:08 smithi171 conmon[46715]: debug 2022-01-31T23:00:08.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:08.165541+0000) 2022-01-31T23:00:08.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:08 smithi171 conmon[41853]: debug 2022-01-31T23:00:08.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:08.369905+0000) 2022-01-31T23:00:08.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:08 smithi171 conmon[51620]: debug 2022-01-31T23:00:08.543+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:08.544499+0000) 2022-01-31T23:00:09.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:09 smithi167 conmon[49112]: debug 2022-01-31T23:00:09.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:09.002063+0000) 2022-01-31T23:00:09.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:08 smithi167 conmon[60316]: debug 2022-01-31T23:00:08.929+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:08.931108+0000) 2022-01-31T23:00:09.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:08 smithi171 conmon[35325]: debug 2022-01-31T23:00:08.945+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:00:09.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:09 smithi171 conmon[46715]: debug 2022-01-31T23:00:09.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:09.165692+0000) 2022-01-31T23:00:09.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:09 smithi167 conmon[54076]: debug 2022-01-31T23:00:09.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:09.171553+0000) 2022-01-31T23:00:09.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:09 smithi171 conmon[41853]: debug 2022-01-31T23:00:09.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:09.370049+0000) 2022-01-31T23:00:09.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:09 smithi171 conmon[51620]: debug 2022-01-31T23:00:09.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:09.544611+0000) 2022-01-31T23:00:10.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:10 smithi167 conmon[49112]: debug 2022-01-31T23:00:10.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.002216+0000) 2022-01-31T23:00:10.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:09 smithi167 conmon[60316]: debug 2022-01-31T23:00:09.930+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:09.931263+0000) 2022-01-31T23:00:10.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:10 smithi167 conmon[54076]: debug 2022-01-31T23:00:10.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.171660+0000) 2022-01-31T23:00:10.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[41853]: debug 2022-01-31T23:00:10.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.370215+0000) 2022-01-31T23:00:10.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[46715]: debug 2022-01-31T23:00:10.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.165896+0000) 2022-01-31T23:00:10.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[51620]: debug 2022-01-31T23:00:10.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.544786+0000) 2022-01-31T23:00:11.096 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:10 smithi167 conmon[54076]: debug 2022-01-31T23:00:10.857+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.859285+0000) 2022-01-31T23:00:11.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:10 smithi167 conmon[49112]: debug 2022-01-31T23:00:10.857+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.859471+0000) 2022-01-31T23:00:11.097 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:11 smithi167 conmon[49112]: debug 2022-01-31T23:00:11.000+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:11.002417+0000) 2022-01-31T23:00:11.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:10 smithi167 conmon[60316]: debug 2022-01-31T23:00:10.859+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.860681+0000) 2022-01-31T23:00:11.098 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:10 smithi167 conmon[60316]: debug 2022-01-31T23:00:10.929+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.931464+0000) 2022-01-31T23:00:11.150 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[35325]: debug 2022-01-31T23:00:10.885+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254735 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:11.151 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[41853]: debug 2022-01-31T23:00:10.859+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.860178+0000) 2022-01-31T23:00:11.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[46715]: debug 2022-01-31T23:00:10.858+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.859671+0000) 2022-01-31T23:00:11.152 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:10 smithi171 conmon[51620]: debug 2022-01-31T23:00:10.859+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:10.859972+0000) 2022-01-31T23:00:11.363 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:11 smithi167 conmon[54076]: debug 2022-01-31T23:00:11.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:11.171817+0000) 2022-01-31T23:00:11.528 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:11 smithi171 conmon[41853]: debug 2022-01-31T23:00:11.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:11.370371+0000) 2022-01-31T23:00:11.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:11 smithi171 conmon[46715]: debug 2022-01-31T23:00:11.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:11.166064+0000) 2022-01-31T23:00:11.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:11 smithi171 conmon[51620]: debug 2022-01-31T23:00:11.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:11.544966+0000) 2022-01-31T23:00:12.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:12 smithi167 conmon[49112]: debug 2022-01-31T23:00:12.001+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:12.002563+0000) 2022-01-31T23:00:12.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:11 smithi167 conmon[60316]: debug 2022-01-31T23:00:11.930+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:11.931624+0000) 2022-01-31T23:00:12.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:12 smithi167 conmon[54076]: debug 2022-01-31T23:00:12.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:12.171982+0000) 2022-01-31T23:00:12.528 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:12 smithi171 conmon[46715]: debug 2022-01-31T23:00:12.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:12.166294+0000) 2022-01-31T23:00:12.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:12 smithi171 conmon[41853]: debug 2022-01-31T23:00:12.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:12.370501+0000) 2022-01-31T23:00:12.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:12 smithi171 conmon[51620]: debug 2022-01-31T23:00:12.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:12.545107+0000) 2022-01-31T23:00:13.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:13 smithi167 conmon[49112]: debug 2022-01-31T23:00:13.002+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:13.002716+0000) 2022-01-31T23:00:13.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:12 smithi167 conmon[60316]: debug 2022-01-31T23:00:12.930+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:12.931785+0000) 2022-01-31T23:00:13.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:13 smithi167 conmon[54076]: debug 2022-01-31T23:00:13.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:13.172141+0000) 2022-01-31T23:00:13.531 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:13 smithi171 conmon[41853]: debug 2022-01-31T23:00:13.369+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:13.370654+0000) 2022-01-31T23:00:13.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:13 smithi171 conmon[46715]: debug 2022-01-31T23:00:13.165+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:13.166541+0000) 2022-01-31T23:00:13.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:13 smithi171 conmon[51620]: debug 2022-01-31T23:00:13.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:13.545296+0000) 2022-01-31T23:00:14.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:14 smithi167 conmon[49112]: debug 2022-01-31T23:00:14.001+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:14.002884+0000) 2022-01-31T23:00:14.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:13 smithi167 conmon[60316]: debug 2022-01-31T23:00:13.931+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:13.931967+0000) 2022-01-31T23:00:14.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:14 smithi167 conmon[54076]: debug 2022-01-31T23:00:14.170+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:14.172329+0000) 2022-01-31T23:00:14.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:14 smithi171 conmon[46715]: debug 2022-01-31T23:00:14.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:14.166770+0000) 2022-01-31T23:00:14.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:14 smithi171 conmon[41853]: debug 2022-01-31T23:00:14.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:14.370829+0000) 2022-01-31T23:00:14.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:14 smithi171 conmon[51620]: debug 2022-01-31T23:00:14.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:14.545444+0000) 2022-01-31T23:00:15.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:14 smithi167 conmon[60316]: debug 2022-01-31T23:00:14.930+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:14.932132+0000) 2022-01-31T23:00:15.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:15 smithi167 conmon[49112]: debug 2022-01-31T23:00:15.002+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.003083+0000) 2022-01-31T23:00:15.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:15 smithi167 conmon[54076]: debug 2022-01-31T23:00:15.171+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.172520+0000) 2022-01-31T23:00:15.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[46715]: debug 2022-01-31T23:00:15.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.166950+0000) 2022-01-31T23:00:15.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[41853]: debug 2022-01-31T23:00:15.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.371036+0000) 2022-01-31T23:00:15.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[51620]: debug 2022-01-31T23:00:15.544+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.545574+0000) 2022-01-31T23:00:16.151 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[35325]: debug 2022-01-31T23:00:15.914+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254847 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:16.151 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[41853]: debug 2022-01-31T23:00:15.887+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.887981+0000) 2022-01-31T23:00:16.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[46715]: debug 2022-01-31T23:00:15.887+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.888554+0000) 2022-01-31T23:00:16.153 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:15 smithi171 conmon[51620]: debug 2022-01-31T23:00:15.887+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.887790+0000) 2022-01-31T23:00:16.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:15 smithi167 conmon[49112]: debug 2022-01-31T23:00:15.887+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.888698+0000) 2022-01-31T23:00:16.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:16 smithi167 conmon[49112]: debug 2022-01-31T23:00:16.002+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:16.003243+0000) 2022-01-31T23:00:16.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:15 smithi167 conmon[54076]: debug 2022-01-31T23:00:15.886+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.888420+0000) 2022-01-31T23:00:16.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:15 smithi167 conmon[60316]: debug 2022-01-31T23:00:15.887+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.888757+0000) 2022-01-31T23:00:16.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:15 smithi167 conmon[60316]: debug 2022-01-31T23:00:15.930+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:15.932317+0000) 2022-01-31T23:00:16.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:16 smithi167 conmon[54076]: debug 2022-01-31T23:00:16.171+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:16.172650+0000) 2022-01-31T23:00:16.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:16 smithi171 conmon[46715]: debug 2022-01-31T23:00:16.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:16.167139+0000) 2022-01-31T23:00:16.533 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:16 smithi171 conmon[41853]: debug 2022-01-31T23:00:16.370+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:16.371213+0000) 2022-01-31T23:00:16.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:16 smithi171 conmon[51620]: debug 2022-01-31T23:00:16.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:16.545730+0000) 2022-01-31T23:00:17.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:16 smithi167 conmon[60316]: debug 2022-01-31T23:00:16.931+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:16.932485+0000) 2022-01-31T23:00:17.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:17 smithi167 conmon[49112]: debug 2022-01-31T23:00:17.001+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:17.003356+0000) 2022-01-31T23:00:17.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:17 smithi167 conmon[54076]: debug 2022-01-31T23:00:17.171+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:17.172752+0000) 2022-01-31T23:00:17.529 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:17 smithi171 conmon[46715]: debug 2022-01-31T23:00:17.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:17.167238+0000) 2022-01-31T23:00:17.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:17 smithi171 conmon[41853]: debug 2022-01-31T23:00:17.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:17.371344+0000) 2022-01-31T23:00:17.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:17 smithi171 conmon[51620]: debug 2022-01-31T23:00:17.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:17.545976+0000) 2022-01-31T23:00:18.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:18 smithi167 conmon[49112]: debug 2022-01-31T23:00:18.001+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:18.003468+0000) 2022-01-31T23:00:18.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:17 smithi167 conmon[60316]: debug 2022-01-31T23:00:17.932+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:17.932690+0000) 2022-01-31T23:00:18.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:18 smithi167 conmon[54076]: debug 2022-01-31T23:00:18.172+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:18.172944+0000) 2022-01-31T23:00:18.529 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:18 smithi171 conmon[41853]: debug 2022-01-31T23:00:18.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:18.371528+0000) 2022-01-31T23:00:18.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:18 smithi171 conmon[46715]: debug 2022-01-31T23:00:18.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:18.167349+0000) 2022-01-31T23:00:18.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:18 smithi171 conmon[51620]: debug 2022-01-31T23:00:18.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:18.546094+0000) 2022-01-31T23:00:19.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:18 smithi167 conmon[60316]: debug 2022-01-31T23:00:18.932+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:18.932879+0000) 2022-01-31T23:00:19.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:19 smithi167 conmon[49112]: debug 2022-01-31T23:00:19.003+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:19.003669+0000) 2022-01-31T23:00:19.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:19 smithi167 conmon[54076]: debug 2022-01-31T23:00:19.172+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:19.173139+0000) 2022-01-31T23:00:19.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:19 smithi171 conmon[46715]: debug 2022-01-31T23:00:19.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:19.167558+0000) 2022-01-31T23:00:19.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:19 smithi171 conmon[41853]: debug 2022-01-31T23:00:19.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:19.371716+0000) 2022-01-31T23:00:19.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:19 smithi171 conmon[51620]: debug 2022-01-31T23:00:19.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:19.546245+0000) 2022-01-31T23:00:20.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:19 smithi167 conmon[60316]: debug 2022-01-31T23:00:19.932+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:19.933089+0000) 2022-01-31T23:00:20.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:20 smithi167 conmon[49112]: debug 2022-01-31T23:00:20.003+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.003807+0000) 2022-01-31T23:00:20.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:20 smithi167 conmon[54076]: debug 2022-01-31T23:00:20.172+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.173348+0000) 2022-01-31T23:00:20.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[41853]: debug 2022-01-31T23:00:20.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.371873+0000) 2022-01-31T23:00:20.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[46715]: debug 2022-01-31T23:00:20.166+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.167713+0000) 2022-01-31T23:00:20.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[51620]: debug 2022-01-31T23:00:20.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.546439+0000) 2022-01-31T23:00:21.151 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[35325]: debug 2022-01-31T23:00:20.943+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 254959 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:21.152 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[41853]: debug 2022-01-31T23:00:20.916+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.917302+0000) 2022-01-31T23:00:21.153 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[46715]: debug 2022-01-31T23:00:20.916+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.917017+0000) 2022-01-31T23:00:21.153 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[51620]: debug 2022-01-31T23:00:21.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:20 smithi171 conmon[51620]: 2022-01-31T23:00:20.915+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.916653+0000) 2022-01-31T23:00:21.155 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:20 smithi167 conmon[54076]: debug 2022-01-31T23:00:20.917+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.918254+0000) 2022-01-31T23:00:21.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:20 smithi167 conmon[49112]: debug 2022-01-31T23:00:20.917+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.918181+0000) 2022-01-31T23:00:21.156 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:21 smithi167 conmon[49112]: debug 2022-01-31T23:00:21.003+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:21.003996+0000) 2022-01-31T23:00:21.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:20 smithi167 conmon[60316]: debug 2022-01-31T23:00:20.916+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.917358+0000) 2022-01-31T23:00:21.156 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:20 smithi167 conmon[60316]: debug 2022-01-31T23:00:20.932+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:20.933221+0000) 2022-01-31T23:00:21.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:21 smithi167 conmon[54076]: debug 2022-01-31T23:00:21.173+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:21.173563+0000) 2022-01-31T23:00:21.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:21 smithi171 conmon[41853]: debug 2022-01-31T23:00:21.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:21.372062+0000) 2022-01-31T23:00:21.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:21 smithi171 conmon[46715]: debug 2022-01-31T23:00:21.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:21.167856+0000) 2022-01-31T23:00:21.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:21 smithi171 conmon[51620]: debug 2022-01-31T23:00:21.545+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:21.546591+0000) 2022-01-31T23:00:22.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:22 smithi167 conmon[49112]: debug 2022-01-31T23:00:22.003+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:22.004176+0000) 2022-01-31T23:00:22.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:21 smithi167 conmon[60316]: debug 2022-01-31T23:00:21.932+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:21.933367+0000) 2022-01-31T23:00:22.510 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:22 smithi167 conmon[54076]: debug 2022-01-31T23:00:22.173+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:22.173653+0000) 2022-01-31T23:00:22.530 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:22 smithi171 conmon[41853]: debug 2022-01-31T23:00:22.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:22.372211+0000) 2022-01-31T23:00:22.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:22 smithi171 conmon[46715]: debug 2022-01-31T23:00:22.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:22.168070+0000) 2022-01-31T23:00:22.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:22 smithi171 conmon[51620]: debug 2022-01-31T23:00:22.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:22.546741+0000) 2022-01-31T23:00:23.139 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:23 smithi167 conmon[49112]: debug 2022-01-31T23:00:23.003+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:23.004381+0000) 2022-01-31T23:00:23.140 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:22 smithi167 conmon[60316]: debug 2022-01-31T23:00:22.933+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:22.933544+0000) 2022-01-31T23:00:23.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:23 smithi167 conmon[54076]: debug 2022-01-31T23:00:23.173+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:23.173814+0000) 2022-01-31T23:00:23.530 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:23 smithi171 conmon[46715]: debug 2022-01-31T23:00:23.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:23.168254+0000) 2022-01-31T23:00:23.531 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:23 smithi171 conmon[41853]: debug 2022-01-31T23:00:23.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:23.372442+0000) 2022-01-31T23:00:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:23 smithi171 conmon[51620]: debug 2022-01-31T23:00:23.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:23.546936+0000) 2022-01-31T23:00:24.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:24 smithi167 conmon[49112]: debug 2022-01-31T23:00:24.004+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:24.004569+0000) 2022-01-31T23:00:24.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:23 smithi167 conmon[60316]: debug 2022-01-31T23:00:23.933+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:23.933754+0000) 2022-01-31T23:00:24.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:23 smithi171 conmon[35325]: debug 2022-01-31T23:00:23.946+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:00:24.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:24 smithi171 conmon[46715]: debug 2022-01-31T23:00:24.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:24.168396+0000) 2022-01-31T23:00:24.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:24 smithi167 conmon[54076]: debug 2022-01-31T23:00:24.173+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:24.174028+0000) 2022-01-31T23:00:24.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:24 smithi171 conmon[41853]: debug 2022-01-31T23:00:24.371+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:24.372652+0000) 2022-01-31T23:00:24.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:24 smithi171 conmon[51620]: debug 2022-01-31T23:00:24.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:24.547132+0000) 2022-01-31T23:00:25.147 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:25 smithi167 conmon[49112]: debug 2022-01-31T23:00:25.004+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.004754+0000) 2022-01-31T23:00:25.147 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:24 smithi167 conmon[60316]: debug 2022-01-31T23:00:24.933+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:24.933893+0000) 2022-01-31T23:00:25.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:25 smithi167 conmon[54076]: debug 2022-01-31T23:00:25.173+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.174184+0000) 2022-01-31T23:00:25.531 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[41853]: debug 2022-01-31T23:00:25.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.372855+0000) 2022-01-31T23:00:25.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[46715]: debug 2022-01-31T23:00:25.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.168606+0000) 2022-01-31T23:00:25.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[51620]: debug 2022-01-31T23:00:25.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.547285+0000) 2022-01-31T23:00:26.150 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:25 smithi167 conmon[54076]: debug 2022-01-31T23:00:25.946+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.947099+0000) 2022-01-31T23:00:26.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:25 smithi167 conmon[60316]: debug 2022-01-31T23:00:25.933+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.934023+0000) 2022-01-31T23:00:26.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:25 smithi167 conmon[60316]: debug 2022-01-31T23:00:25.945+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.946004+0000) 2022-01-31T23:00:26.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:25 smithi167 conmon[49112]: debug 2022-01-31T23:00:25.945+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.946307+0000) 2022-01-31T23:00:26.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:26 smithi167 conmon[49112]: debug 2022-01-31T23:00:26.004+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:26.004913+0000) 2022-01-31T23:00:26.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[46715]: debug 2022-01-31T23:00:25.944+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.946353+0000) 2022-01-31T23:00:26.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:26 smithi171 conmon[46715]: debug 2022-01-31T23:00:26.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:26.168801+0000) 2022-01-31T23:00:26.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[51620]: debug 2022-01-31T23:00:25.944+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.945473+0000) 2022-01-31T23:00:26.343 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[35325]: debug 2022-01-31T23:00:25.986+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255070 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:26.343 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:25 smithi171 conmon[41853]: debug 2022-01-31T23:00:25.944+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:25.946807+0000) 2022-01-31T23:00:26.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:26 smithi167 conmon[54076]: debug 2022-01-31T23:00:26.173+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:26.174392+0000) 2022-01-31T23:00:26.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:26 smithi171 conmon[41853]: debug 2022-01-31T23:00:26.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:26.373022+0000) 2022-01-31T23:00:26.850 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:26 smithi171 conmon[51620]: debug 2022-01-31T23:00:26.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:26.547471+0000) 2022-01-31T23:00:27.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:27 smithi167 conmon[49112]: debug 2022-01-31T23:00:27.004+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:27.005088+0000) 2022-01-31T23:00:27.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:26 smithi167 conmon[60316]: debug 2022-01-31T23:00:26.933+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:26.934207+0000) 2022-01-31T23:00:27.405 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:27 smithi167 conmon[54076]: debug 2022-01-31T23:00:27.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:27.174522+0000) 2022-01-31T23:00:27.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:27 smithi171 conmon[46715]: debug 2022-01-31T23:00:27.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:27.168904+0000) 2022-01-31T23:00:27.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:27 smithi171 conmon[41853]: debug 2022-01-31T23:00:27.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:27.373178+0000) 2022-01-31T23:00:27.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:27 smithi171 conmon[51620]: debug 2022-01-31T23:00:27.546+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:27.547613+0000) 2022-01-31T23:00:28.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:27 smithi167 conmon[60316]: debug 2022-01-31T23:00:27.933+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:27.934375+0000) 2022-01-31T23:00:28.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:28 smithi167 conmon[49112]: debug 2022-01-31T23:00:28.004+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:28.005288+0000) 2022-01-31T23:00:28.511 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:28 smithi167 conmon[54076]: debug 2022-01-31T23:00:28.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:28.174633+0000) 2022-01-31T23:00:28.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:28 smithi171 conmon[46715]: debug 2022-01-31T23:00:28.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:28.169066+0000) 2022-01-31T23:00:28.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:28 smithi171 conmon[41853]: debug 2022-01-31T23:00:28.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:28.373358+0000) 2022-01-31T23:00:28.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:28 smithi171 conmon[51620]: debug 2022-01-31T23:00:28.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:28.547811+0000) 2022-01-31T23:00:29.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:29 smithi167 conmon[49112]: debug 2022-01-31T23:00:29.005+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:29.005493+0000) 2022-01-31T23:00:29.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:28 smithi167 conmon[60316]: debug 2022-01-31T23:00:28.934+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:28.934540+0000) 2022-01-31T23:00:29.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:29 smithi167 conmon[54076]: debug 2022-01-31T23:00:29.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:29.174750+0000) 2022-01-31T23:00:29.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:29 smithi171 conmon[46715]: debug 2022-01-31T23:00:29.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:29.169260+0000) 2022-01-31T23:00:29.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:29 smithi171 conmon[41853]: debug 2022-01-31T23:00:29.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:29.373551+0000) 2022-01-31T23:00:29.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:29 smithi171 conmon[51620]: debug 2022-01-31T23:00:29.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:29.547966+0000) 2022-01-31T23:00:30.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:30 smithi167 conmon[49112]: debug 2022-01-31T23:00:30.005+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.005705+0000) 2022-01-31T23:00:30.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:29 smithi167 conmon[60316]: debug 2022-01-31T23:00:29.934+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:29.934644+0000) 2022-01-31T23:00:30.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:30 smithi167 conmon[54076]: debug 2022-01-31T23:00:30.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.174922+0000) 2022-01-31T23:00:30.531 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:30 smithi171 conmon[46715]: debug 2022-01-31T23:00:30.167+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.169448+0000) 2022-01-31T23:00:30.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:30 smithi171 conmon[41853]: debug 2022-01-31T23:00:30.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.373742+0000) 2022-01-31T23:00:30.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:30 smithi171 conmon[51620]: debug 2022-01-31T23:00:30.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.548163+0000) 2022-01-31T23:00:31.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:30 smithi167 conmon[49112]: debug 2022-01-31T23:00:30.989+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.989638+0000) 2022-01-31T23:00:31.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:31 smithi167 conmon[49112]: debug 2022-01-31T23:00:31.005+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:31.005837+0000) 2022-01-31T23:00:31.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:30 smithi167 conmon[54076]: debug 2022-01-31T23:00:30.989+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.989850+0000) 2022-01-31T23:00:31.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:30 smithi167 conmon[60316]: debug 2022-01-31T23:00:30.934+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.934779+0000) 2022-01-31T23:00:31.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:30 smithi167 conmon[60316]: debug 2022-01-31T23:00:30.995+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.995992+0000) 2022-01-31T23:00:31.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:30 smithi171 conmon[46715]: debug 2022-01-31T23:00:30.988+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.990170+0000) 2022-01-31T23:00:31.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:31 smithi171 conmon[46715]: debug 2022-01-31T23:00:31.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:31.169632+0000) 2022-01-31T23:00:31.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:31 smithi171 conmon[35325]: debug 2022-01-31T23:00:31.013+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255182 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:31.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:30 smithi171 conmon[41853]: debug 2022-01-31T23:00:30.988+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.990296+0000) 2022-01-31T23:00:31.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:30 smithi171 conmon[51620]: debug 2022-01-31T23:00:30.987+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:30.989153+0000) 2022-01-31T23:00:31.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:31 smithi167 conmon[54076]: debug 2022-01-31T23:00:31.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:31.175103+0000) 2022-01-31T23:00:31.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:31 smithi171 conmon[51620]: debug 2022-01-31T23:00:31.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:31.548318+0000) 2022-01-31T23:00:31.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:31 smithi171 conmon[41853]: debug 2022-01-31T23:00:31.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:31.373848+0000) 2022-01-31T23:00:32.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:32 smithi167 conmon[49112]: debug 2022-01-31T23:00:32.005+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:32.006003+0000) 2022-01-31T23:00:32.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:31 smithi167 conmon[60316]: debug 2022-01-31T23:00:31.934+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:31.934927+0000) 2022-01-31T23:00:32.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:32 smithi167 conmon[54076]: debug 2022-01-31T23:00:32.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:32.175278+0000) 2022-01-31T23:00:32.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:32 smithi171 conmon[46715]: debug 2022-01-31T23:00:32.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:32.169791+0000) 2022-01-31T23:00:32.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:32 smithi171 conmon[41853]: debug 2022-01-31T23:00:32.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:32.374020+0000) 2022-01-31T23:00:32.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:32 smithi171 conmon[51620]: debug 2022-01-31T23:00:32.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:32.548500+0000) 2022-01-31T23:00:33.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:33 smithi167 conmon[49112]: debug 2022-01-31T23:00:33.005+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:33.006193+0000) 2022-01-31T23:00:33.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:32 smithi167 conmon[60316]: debug 2022-01-31T23:00:32.934+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:32.935022+0000) 2022-01-31T23:00:33.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:33 smithi167 conmon[54076]: debug 2022-01-31T23:00:33.174+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:33.175487+0000) 2022-01-31T23:00:33.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:33 smithi171 conmon[41853]: debug 2022-01-31T23:00:33.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:33.374189+0000) 2022-01-31T23:00:33.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:33 smithi171 conmon[46715]: debug 2022-01-31T23:00:33.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:33.169944+0000) 2022-01-31T23:00:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:33 smithi171 conmon[51620]: debug 2022-01-31T23:00:33.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:33.548656+0000) 2022-01-31T23:00:34.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:34 smithi167 conmon[49112]: debug 2022-01-31T23:00:34.005+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:34.006374+0000) 2022-01-31T23:00:34.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:33 smithi167 conmon[60316]: debug 2022-01-31T23:00:33.934+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:33.935206+0000) 2022-01-31T23:00:34.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:34 smithi167 conmon[54076]: debug 2022-01-31T23:00:34.175+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:34.175693+0000) 2022-01-31T23:00:34.532 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:34 smithi171 conmon[41853]: debug 2022-01-31T23:00:34.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:34.374373+0000) 2022-01-31T23:00:34.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:34 smithi171 conmon[46715]: debug 2022-01-31T23:00:34.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:34.170157+0000) 2022-01-31T23:00:34.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:34 smithi171 conmon[51620]: debug 2022-01-31T23:00:34.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:34.548839+0000) 2022-01-31T23:00:35.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:34 smithi167 conmon[60316]: debug 2022-01-31T23:00:34.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:34.935422+0000) 2022-01-31T23:00:35.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:35 smithi167 conmon[49112]: debug 2022-01-31T23:00:35.006+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:35.006536+0000) 2022-01-31T23:00:35.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:35 smithi167 conmon[54076]: debug 2022-01-31T23:00:35.175+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:35.175877+0000) 2022-01-31T23:00:35.532 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:35 smithi171 conmon[46715]: debug 2022-01-31T23:00:35.168+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:35.170341+0000) 2022-01-31T23:00:35.533 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:35 smithi171 conmon[41853]: debug 2022-01-31T23:00:35.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:35.374551+0000) 2022-01-31T23:00:35.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:35 smithi171 conmon[51620]: debug 2022-01-31T23:00:35.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:35.549059+0000) 2022-01-31T23:00:36.153 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:36 smithi167 conmon[54076]: debug 2022-01-31T23:00:36.017+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.018469+0000) 2022-01-31T23:00:36.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:36 smithi167 conmon[49112]: debug 2022-01-31T23:00:36.006+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.006695+0000) 2022-01-31T23:00:36.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:36 smithi167 conmon[49112]: debug 2022-01-31T23:00:36.017+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.017638+0000) 2022-01-31T23:00:36.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:35 smithi167 conmon[60316]: debug 2022-01-31T23:00:35.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:35.935583+0000) 2022-01-31T23:00:36.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:36 smithi167 conmon[60316]: debug 2022-01-31T23:00:36.017+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.018254+0000) 2022-01-31T23:00:36.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[41853]: debug 2022-01-31T23:00:36.015+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.017305+0000) 2022-01-31T23:00:36.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[51620]: debug 2022-01-31T23:00:36.016+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.018118+0000) 2022-01-31T23:00:36.342 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[35325]: debug 2022-01-31T23:00:36.041+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255293 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:36.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[46715]: debug 2022-01-31T23:00:36.016+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.018184+0000) 2022-01-31T23:00:36.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[46715]: debug 2022-01-31T23:00:36.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.170484+0000) 2022-01-31T23:00:36.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:36 smithi167 conmon[54076]: debug 2022-01-31T23:00:36.175+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.176021+0000) 2022-01-31T23:00:36.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[41853]: debug 2022-01-31T23:00:36.372+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.374682+0000) 2022-01-31T23:00:36.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:36 smithi171 conmon[51620]: debug 2022-01-31T23:00:36.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.549214+0000) 2022-01-31T23:00:37.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:37 smithi167 conmon[49112]: debug 2022-01-31T23:00:37.006+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:37.006845+0000) 2022-01-31T23:00:37.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:36 smithi167 conmon[60316]: debug 2022-01-31T23:00:36.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:36.935713+0000) 2022-01-31T23:00:37.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:37 smithi167 conmon[54076]: debug 2022-01-31T23:00:37.175+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:37.176199+0000) 2022-01-31T23:00:37.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:37 smithi171 conmon[46715]: debug 2022-01-31T23:00:37.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:37.170595+0000) 2022-01-31T23:00:37.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:37 smithi171 conmon[41853]: debug 2022-01-31T23:00:37.373+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:37.374842+0000) 2022-01-31T23:00:37.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:37 smithi171 conmon[51620]: debug 2022-01-31T23:00:37.547+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:37.549364+0000) 2022-01-31T23:00:38.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:38 smithi167 conmon[49112]: debug 2022-01-31T23:00:38.006+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:38.007043+0000) 2022-01-31T23:00:38.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:37 smithi167 conmon[60316]: debug 2022-01-31T23:00:37.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:37.935852+0000) 2022-01-31T23:00:38.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:38 smithi167 conmon[54076]: debug 2022-01-31T23:00:38.175+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:38.176361+0000) 2022-01-31T23:00:38.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:38 smithi171 conmon[46715]: debug 2022-01-31T23:00:38.169+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:38.170847+0000) 2022-01-31T23:00:38.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:38 smithi171 conmon[41853]: debug 2022-01-31T23:00:38.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:38.375024+0000) 2022-01-31T23:00:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:38 smithi171 conmon[51620]: debug 2022-01-31T23:00:38.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:38.549561+0000) 2022-01-31T23:00:39.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:39 smithi167 conmon[49112]: debug 2022-01-31T23:00:39.006+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:39.007218+0000) 2022-01-31T23:00:39.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:38 smithi167 conmon[60316]: debug 2022-01-31T23:00:38.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:38.936031+0000) 2022-01-31T23:00:39.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:38 smithi171 conmon[35325]: debug 2022-01-31T23:00:38.946+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:00:39.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:39 smithi171 conmon[46715]: debug 2022-01-31T23:00:39.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:39.170968+0000) 2022-01-31T23:00:39.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:39 smithi167 conmon[54076]: debug 2022-01-31T23:00:39.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:39.176572+0000) 2022-01-31T23:00:39.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:39 smithi171 conmon[41853]: debug 2022-01-31T23:00:39.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:39.375245+0000) 2022-01-31T23:00:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:39 smithi171 conmon[51620]: debug 2022-01-31T23:00:39.548+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:39.549745+0000) 2022-01-31T23:00:40.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:40 smithi167 conmon[49112]: debug 2022-01-31T23:00:40.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:40.007374+0000) 2022-01-31T23:00:40.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:39 smithi167 conmon[60316]: debug 2022-01-31T23:00:39.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:39.936195+0000) 2022-01-31T23:00:40.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:40 smithi167 conmon[54076]: debug 2022-01-31T23:00:40.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:40.176736+0000) 2022-01-31T23:00:40.533 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:40 smithi171 conmon[46715]: debug 2022-01-31T23:00:40.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:40.171101+0000) 2022-01-31T23:00:40.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:40 smithi171 conmon[41853]: debug 2022-01-31T23:00:40.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:40.375419+0000) 2022-01-31T23:00:40.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:40 smithi171 conmon[51620]: debug 2022-01-31T23:00:40.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:40.549991+0000) 2022-01-31T23:00:41.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:41 smithi167 conmon[49112]: debug 2022-01-31T23:00:41.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.007594+0000) 2022-01-31T23:00:41.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:41 smithi167 conmon[49112]: debug 2022-01-31T23:00:41.045+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.045590+0000) 2022-01-31T23:00:41.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:41 smithi167 conmon[54076]: debug 2022-01-31T23:00:41.045+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.046166+0000) 2022-01-31T23:00:41.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:40 smithi167 conmon[60316]: debug 2022-01-31T23:00:40.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:40.936381+0000) 2022-01-31T23:00:41.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:41 smithi167 conmon[60316]: debug 2022-01-31T23:00:41.045+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.046387+0000) 2022-01-31T23:00:41.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[35325]: debug 2022-01-31T23:00:41.070+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255405 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:41.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[41853]: debug 2022-01-31T23:00:41.043+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.045000+0000) 2022-01-31T23:00:41.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[51620]: debug 2022-01-31T23:00:41.044+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.046731+0000) 2022-01-31T23:00:41.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[51620]: 2022-01-31T23:00:41.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[46715]: debug 2022-01-31T23:00:41.044+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.045848+0000) 2022-01-31T23:00:41.344 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[46715]: debug 2022-01-31T23:00:41.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.171273+0000) 2022-01-31T23:00:41.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:41 smithi167 conmon[54076]: debug 2022-01-31T23:00:41.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.176869+0000) 2022-01-31T23:00:41.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[51620]: debug 2022-01-31T23:00:41.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.550200+0000) 2022-01-31T23:00:41.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:41 smithi171 conmon[41853]: debug 2022-01-31T23:00:41.374+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.375640+0000) 2022-01-31T23:00:42.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:42 smithi167 conmon[49112]: debug 2022-01-31T23:00:42.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:42.007750+0000) 2022-01-31T23:00:42.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:41 smithi167 conmon[60316]: debug 2022-01-31T23:00:41.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:41.936586+0000) 2022-01-31T23:00:42.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:42 smithi167 conmon[54076]: debug 2022-01-31T23:00:42.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:42.177011+0000) 2022-01-31T23:00:42.533 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:42 smithi171 conmon[41853]: debug 2022-01-31T23:00:42.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:42.375807+0000) 2022-01-31T23:00:42.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:42 smithi171 conmon[46715]: debug 2022-01-31T23:00:42.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:42.171386+0000) 2022-01-31T23:00:42.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:42 smithi171 conmon[51620]: debug 2022-01-31T23:00:42.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:42.550381+0000) 2022-01-31T23:00:43.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:43 smithi167 conmon[49112]: debug 2022-01-31T23:00:43.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:43.007865+0000) 2022-01-31T23:00:43.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:42 smithi167 conmon[60316]: debug 2022-01-31T23:00:42.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:42.936682+0000) 2022-01-31T23:00:43.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:43 smithi167 conmon[54076]: debug 2022-01-31T23:00:43.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:43.177186+0000) 2022-01-31T23:00:43.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:43 smithi171 conmon[41853]: debug 2022-01-31T23:00:43.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:43.375927+0000) 2022-01-31T23:00:43.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:43 smithi171 conmon[46715]: debug 2022-01-31T23:00:43.170+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:43.171566+0000) 2022-01-31T23:00:43.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:43 smithi171 conmon[51620]: debug 2022-01-31T23:00:43.549+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:43.550589+0000) 2022-01-31T23:00:44.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:44 smithi167 conmon[49112]: debug 2022-01-31T23:00:44.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:44.008016+0000) 2022-01-31T23:00:44.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:43 smithi167 conmon[60316]: debug 2022-01-31T23:00:43.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:43.936818+0000) 2022-01-31T23:00:44.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:44 smithi167 conmon[54076]: debug 2022-01-31T23:00:44.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:44.177393+0000) 2022-01-31T23:00:44.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:44 smithi171 conmon[41853]: debug 2022-01-31T23:00:44.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:44.376083+0000) 2022-01-31T23:00:44.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:44 smithi171 conmon[46715]: debug 2022-01-31T23:00:44.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:44.171774+0000) 2022-01-31T23:00:44.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:44 smithi171 conmon[51620]: debug 2022-01-31T23:00:44.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:44.550802+0000) 2022-01-31T23:00:45.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:45 smithi167 conmon[49112]: debug 2022-01-31T23:00:45.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:45.008196+0000) 2022-01-31T23:00:45.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:44 smithi167 conmon[60316]: debug 2022-01-31T23:00:44.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:44.937026+0000) 2022-01-31T23:00:45.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:45 smithi167 conmon[54076]: debug 2022-01-31T23:00:45.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:45.177602+0000) 2022-01-31T23:00:45.534 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:45 smithi171 conmon[41853]: debug 2022-01-31T23:00:45.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:45.376260+0000) 2022-01-31T23:00:45.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:45 smithi171 conmon[46715]: debug 2022-01-31T23:00:45.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:45.171959+0000) 2022-01-31T23:00:45.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:45 smithi171 conmon[51620]: debug 2022-01-31T23:00:45.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:45.551010+0000) 2022-01-31T23:00:46.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:46 smithi167 conmon[49112]: debug 2022-01-31T23:00:46.006+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.008421+0000) 2022-01-31T23:00:46.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:46 smithi167 conmon[49112]: debug 2022-01-31T23:00:46.073+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.074989+0000) 2022-01-31T23:00:46.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:46 smithi167 conmon[54076]: debug 2022-01-31T23:00:46.073+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.075275+0000) 2022-01-31T23:00:46.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:45 smithi167 conmon[60316]: debug 2022-01-31T23:00:45.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:45.937236+0000) 2022-01-31T23:00:46.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:46 smithi167 conmon[60316]: debug 2022-01-31T23:00:46.073+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.074606+0000) 2022-01-31T23:00:46.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[46715]: debug 2022-01-31T23:00:46.073+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.073905+0000) 2022-01-31T23:00:46.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[46715]: debug 2022-01-31T23:00:46.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.172134+0000) 2022-01-31T23:00:46.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[51620]: debug 2022-01-31T23:00:46.074+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.074929+0000) 2022-01-31T23:00:46.343 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[35325]: debug 2022-01-31T23:00:46.100+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255517 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:46.344 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[41853]: debug 2022-01-31T23:00:46.074+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.075245+0000) 2022-01-31T23:00:46.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:46 smithi167 conmon[54076]: debug 2022-01-31T23:00:46.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.177739+0000) 2022-01-31T23:00:46.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[51620]: debug 2022-01-31T23:00:46.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.551171+0000) 2022-01-31T23:00:46.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:46 smithi171 conmon[41853]: debug 2022-01-31T23:00:46.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.376423+0000) 2022-01-31T23:00:47.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:47 smithi167 conmon[49112]: debug 2022-01-31T23:00:47.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:47.008558+0000) 2022-01-31T23:00:47.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:46 smithi167 conmon[60316]: debug 2022-01-31T23:00:46.935+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:46.937440+0000) 2022-01-31T23:00:47.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:47 smithi167 conmon[54076]: debug 2022-01-31T23:00:47.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:47.177846+0000) 2022-01-31T23:00:47.534 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:47 smithi171 conmon[46715]: debug 2022-01-31T23:00:47.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:47.172302+0000) 2022-01-31T23:00:47.535 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:47 smithi171 conmon[41853]: debug 2022-01-31T23:00:47.375+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:47.376575+0000) 2022-01-31T23:00:47.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:47 smithi171 conmon[51620]: debug 2022-01-31T23:00:47.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:47.551296+0000) 2022-01-31T23:00:48.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:48 smithi167 conmon[49112]: debug 2022-01-31T23:00:48.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:48.008657+0000) 2022-01-31T23:00:48.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:47 smithi167 conmon[60316]: debug 2022-01-31T23:00:47.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:47.937579+0000) 2022-01-31T23:00:48.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:48 smithi167 conmon[54076]: debug 2022-01-31T23:00:48.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:48.178033+0000) 2022-01-31T23:00:48.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:48 smithi171 conmon[46715]: debug 2022-01-31T23:00:48.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:48.172533+0000) 2022-01-31T23:00:48.535 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:48 smithi171 conmon[41853]: debug 2022-01-31T23:00:48.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:48.376778+0000) 2022-01-31T23:00:48.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:48 smithi171 conmon[51620]: debug 2022-01-31T23:00:48.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:48.551487+0000) 2022-01-31T23:00:49.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:49 smithi167 conmon[49112]: debug 2022-01-31T23:00:49.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:49.008840+0000) 2022-01-31T23:00:49.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:48 smithi167 conmon[60316]: debug 2022-01-31T23:00:48.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:48.937756+0000) 2022-01-31T23:00:49.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:49 smithi167 conmon[54076]: debug 2022-01-31T23:00:49.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:49.178231+0000) 2022-01-31T23:00:49.535 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:49 smithi171 conmon[41853]: debug 2022-01-31T23:00:49.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:49.376951+0000) 2022-01-31T23:00:49.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:49 smithi171 conmon[46715]: debug 2022-01-31T23:00:49.171+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:49.172735+0000) 2022-01-31T23:00:49.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:49 smithi171 conmon[51620]: debug 2022-01-31T23:00:49.550+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:49.551669+0000) 2022-01-31T23:00:50.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:50 smithi167 conmon[49112]: debug 2022-01-31T23:00:50.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:50.008989+0000) 2022-01-31T23:00:50.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:49 smithi167 conmon[60316]: debug 2022-01-31T23:00:49.937+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:49.937942+0000) 2022-01-31T23:00:50.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:50 smithi167 conmon[54076]: debug 2022-01-31T23:00:50.176+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:50.178463+0000) 2022-01-31T23:00:50.535 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:50 smithi171 conmon[41853]: debug 2022-01-31T23:00:50.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:50.377095+0000) 2022-01-31T23:00:50.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:50 smithi171 conmon[46715]: debug 2022-01-31T23:00:50.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:50.172861+0000) 2022-01-31T23:00:50.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:50 smithi171 conmon[51620]: debug 2022-01-31T23:00:50.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:50.551862+0000) 2022-01-31T23:00:51.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:51 smithi167 conmon[49112]: debug 2022-01-31T23:00:51.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.009165+0000) 2022-01-31T23:00:51.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:51 smithi167 conmon[49112]: debug 2022-01-31T23:00:51.103+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.105103+0000) 2022-01-31T23:00:51.154 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:51 smithi167 conmon[54076]: debug 2022-01-31T23:00:51.102+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.103897+0000) 2022-01-31T23:00:51.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:50 smithi167 conmon[60316]: debug 2022-01-31T23:00:50.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:50.938150+0000) 2022-01-31T23:00:51.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:51 smithi167 conmon[60316]: debug 2022-01-31T23:00:51.102+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.104024+0000) 2022-01-31T23:00:51.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[51620]: debug 2022-01-31T23:00:51.103+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.103831+0000) 2022-01-31T23:00:51.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[35325]: debug 2022-01-31T23:00:51.129+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255630 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:51.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[41853]: debug 2022-01-31T23:00:51.103+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.104719+0000) 2022-01-31T23:00:51.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[46715]: debug 2022-01-31T23:00:51.102+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.102883+0000) 2022-01-31T23:00:51.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[46715]: debug 2022-01-31T23:00:51.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.173040+0000) 2022-01-31T23:00:51.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:51 smithi167 conmon[54076]: debug 2022-01-31T23:00:51.177+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.178614+0000) 2022-01-31T23:00:51.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[41853]: debug 2022-01-31T23:00:51.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.377249+0000) 2022-01-31T23:00:51.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:51 smithi171 conmon[51620]: debug 2022-01-31T23:00:51.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.552033+0000) 2022-01-31T23:00:52.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:52 smithi167 conmon[49112]: debug 2022-01-31T23:00:52.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:52.009351+0000) 2022-01-31T23:00:52.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:51 smithi167 conmon[60316]: debug 2022-01-31T23:00:51.936+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:51.938355+0000) 2022-01-31T23:00:52.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:52 smithi167 conmon[54076]: debug 2022-01-31T23:00:52.177+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:52.178732+0000) 2022-01-31T23:00:52.535 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:52 smithi171 conmon[46715]: debug 2022-01-31T23:00:52.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:52.173164+0000) 2022-01-31T23:00:52.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:52 smithi171 conmon[41853]: debug 2022-01-31T23:00:52.376+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:52.377419+0000) 2022-01-31T23:00:52.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:52 smithi171 conmon[51620]: debug 2022-01-31T23:00:52.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:52.552215+0000) 2022-01-31T23:00:53.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:53 smithi167 conmon[49112]: debug 2022-01-31T23:00:53.007+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:53.009478+0000) 2022-01-31T23:00:53.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:52 smithi167 conmon[60316]: debug 2022-01-31T23:00:52.937+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:52.938524+0000) 2022-01-31T23:00:53.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:53 smithi167 conmon[54076]: debug 2022-01-31T23:00:53.177+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:53.178912+0000) 2022-01-31T23:00:53.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:53 smithi171 conmon[41853]: debug 2022-01-31T23:00:53.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:53.377607+0000) 2022-01-31T23:00:53.536 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:53 smithi171 conmon[46715]: debug 2022-01-31T23:00:53.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:53.173388+0000) 2022-01-31T23:00:53.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:53 smithi171 conmon[51620]: debug 2022-01-31T23:00:53.551+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:53.552413+0000) 2022-01-31T23:00:54.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:54 smithi167 conmon[49112]: debug 2022-01-31T23:00:54.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:54.009642+0000) 2022-01-31T23:00:54.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:53 smithi167 conmon[60316]: debug 2022-01-31T23:00:53.937+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:53.938706+0000) 2022-01-31T23:00:54.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:53 smithi171 conmon[35325]: debug 2022-01-31T23:00:53.947+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:00:54.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:54 smithi171 conmon[46715]: debug 2022-01-31T23:00:54.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:54.173572+0000) 2022-01-31T23:00:54.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:54 smithi167 conmon[54076]: debug 2022-01-31T23:00:54.177+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:54.179085+0000) 2022-01-31T23:00:54.705 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:54 smithi171 conmon[51620]: debug 2022-01-31T23:00:54.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:54.552597+0000) 2022-01-31T23:00:54.706 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:54 smithi171 conmon[41853]: debug 2022-01-31T23:00:54.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:54.377792+0000) 2022-01-31T23:00:55.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:55 smithi167 conmon[49112]: debug 2022-01-31T23:00:55.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:55.009792+0000) 2022-01-31T23:00:55.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:54 smithi167 conmon[60316]: debug 2022-01-31T23:00:54.937+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:54.938916+0000) 2022-01-31T23:00:55.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:55 smithi171 conmon[46715]: debug 2022-01-31T23:00:55.172+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:55.173707+0000) 2022-01-31T23:00:55.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:55 smithi167 conmon[54076]: debug 2022-01-31T23:00:55.177+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:55.179198+0000) 2022-01-31T23:00:55.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:55 smithi171 conmon[41853]: debug 2022-01-31T23:00:55.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:55.378018+0000) 2022-01-31T23:00:55.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:55 smithi171 conmon[51620]: debug 2022-01-31T23:00:55.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:55.552807+0000) 2022-01-31T23:00:56.153 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:56 smithi167 conmon[54076]: debug 2022-01-31T23:00:56.131+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.132862+0000) 2022-01-31T23:00:56.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:55 smithi167 conmon[60316]: debug 2022-01-31T23:00:55.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:55.939138+0000) 2022-01-31T23:00:56.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:56 smithi167 conmon[60316]: debug 2022-01-31T23:00:56.131+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.132929+0000) 2022-01-31T23:00:56.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:56 smithi167 conmon[49112]: debug 2022-01-31T23:00:56.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.009941+0000) 2022-01-31T23:00:56.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:56 smithi167 conmon[49112]: debug 2022-01-31T23:00:56.131+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.133119+0000) 2022-01-31T23:00:56.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:56 smithi167 conmon[54076]: debug 2022-01-31T23:00:56.177+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.179409+0000) 2022-01-31T23:00:56.536 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[35325]: debug 2022-01-31T23:00:56.157+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255742 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:00:56.537 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[51620]: debug 2022-01-31T23:00:56.132+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.133620+0000) 2022-01-31T23:00:56.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[41853]: debug 2022-01-31T23:00:56.132+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.133304+0000) 2022-01-31T23:00:56.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[41853]: debug 2022-01-31T23:00:56.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.378231+0000) 2022-01-31T23:00:56.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[46715]: debug 2022-01-31T23:00:56.131+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.131800+0000) 2022-01-31T23:00:56.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[46715]: debug 2022-01-31T23:00:56.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.173858+0000) 2022-01-31T23:00:56.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:56 smithi171 conmon[51620]: debug 2022-01-31T23:00:56.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.552974+0000) 2022-01-31T23:00:57.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:56 smithi167 conmon[60316]: debug 2022-01-31T23:00:56.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:56.939276+0000) 2022-01-31T23:00:57.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:57 smithi167 conmon[49112]: debug 2022-01-31T23:00:57.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:57.010068+0000) 2022-01-31T23:00:57.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:57 smithi167 conmon[54076]: debug 2022-01-31T23:00:57.178+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:57.179553+0000) 2022-01-31T23:00:57.598 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:57 smithi171 conmon[41853]: debug 2022-01-31T23:00:57.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:57.378402+0000) 2022-01-31T23:00:57.599 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:57 smithi171 conmon[46715]: debug 2022-01-31T23:00:57.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:57.173953+0000) 2022-01-31T23:00:57.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:57 smithi171 conmon[51620]: debug 2022-01-31T23:00:57.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:57.553128+0000) 2022-01-31T23:00:58.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:58 smithi167 conmon[49112]: debug 2022-01-31T23:00:58.009+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:58.010229+0000) 2022-01-31T23:00:58.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:57 smithi167 conmon[60316]: debug 2022-01-31T23:00:57.937+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:57.939434+0000) 2022-01-31T23:00:58.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:58 smithi167 conmon[54076]: debug 2022-01-31T23:00:58.178+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:58.179679+0000) 2022-01-31T23:00:58.536 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:58 smithi171 conmon[41853]: debug 2022-01-31T23:00:58.377+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:58.378608+0000) 2022-01-31T23:00:58.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:58 smithi171 conmon[46715]: debug 2022-01-31T23:00:58.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:58.174149+0000) 2022-01-31T23:00:58.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:58 smithi171 conmon[51620]: debug 2022-01-31T23:00:58.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:58.553324+0000) 2022-01-31T23:00:59.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:00:59 smithi167 conmon[49112]: debug 2022-01-31T23:00:59.008+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:59.010407+0000) 2022-01-31T23:00:59.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:58 smithi167 conmon[60316]: debug 2022-01-31T23:00:58.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:58.939614+0000) 2022-01-31T23:00:59.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:00:59 smithi167 conmon[54076]: debug 2022-01-31T23:00:59.178+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:59.179835+0000) 2022-01-31T23:00:59.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:00:59 smithi171 conmon[41853]: debug 2022-01-31T23:00:59.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:59.378805+0000) 2022-01-31T23:00:59.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:00:59 smithi171 conmon[46715]: debug 2022-01-31T23:00:59.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:59.174359+0000) 2022-01-31T23:00:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:00:59 smithi171 conmon[51620]: debug 2022-01-31T23:00:59.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:59.553505+0000) 2022-01-31T23:01:00.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:00 smithi167 conmon[49112]: debug 2022-01-31T23:01:00.009+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:00.010616+0000) 2022-01-31T23:01:00.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:00:59 smithi167 conmon[60316]: debug 2022-01-31T23:00:59.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:00:59.939791+0000) 2022-01-31T23:01:00.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:00 smithi167 conmon[54076]: debug 2022-01-31T23:01:00.178+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:00.180028+0000) 2022-01-31T23:01:00.537 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:00 smithi171 conmon[41853]: debug 2022-01-31T23:01:00.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:00.378967+0000) 2022-01-31T23:01:00.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:00 smithi171 conmon[46715]: debug 2022-01-31T23:01:00.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:00.174463+0000) 2022-01-31T23:01:00.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:00 smithi171 conmon[51620]: debug 2022-01-31T23:01:00.552+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:00.553697+0000) 2022-01-31T23:01:01.144 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:01 smithi167 conmon[49112]: debug 2022-01-31T23:01:01.010+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.010803+0000) 2022-01-31T23:01:01.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:01 smithi167 conmon[49112]: debug 2022-01-31T23:01:01.159+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.160901+0000) 2022-01-31T23:01:01.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:00 smithi167 conmon[60316]: debug 2022-01-31T23:01:00.939+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:00.939947+0000) 2022-01-31T23:01:01.146 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:01 smithi167 conmon[60316]: debug 2022-01-31T23:01:01.158+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.160203+0000) 2022-01-31T23:01:01.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:01 smithi167 conmon[54076]: debug 2022-01-31T23:01:01.160+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.161654+0000) 2022-01-31T23:01:01.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:01 smithi167 conmon[54076]: debug 2022-01-31T23:01:01.178+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.180165+0000) 2022-01-31T23:01:01.537 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[35325]: debug 2022-01-31T23:01:01.185+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255853 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:01.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[41853]: debug 2022-01-31T23:01:01.161+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.161897+0000) 2022-01-31T23:01:01.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[41853]: debug 2022-01-31T23:01:01.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.379063+0000) 2022-01-31T23:01:01.539 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[51620]: debug 2022-01-31T23:01:01.160+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.161606+0000) 2022-01-31T23:01:01.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[46715]: debug 2022-01-31T23:01:01.159+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.160014+0000) 2022-01-31T23:01:01.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[46715]: debug 2022-01-31T23:01:01.173+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.174652+0000) 2022-01-31T23:01:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:01 smithi171 conmon[51620]: debug 2022-01-31T23:01:01.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.553889+0000) 2022-01-31T23:01:02.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:02 smithi167 conmon[49112]: debug 2022-01-31T23:01:02.009+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:02.010962+0000) 2022-01-31T23:01:02.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:01 smithi167 conmon[60316]: debug 2022-01-31T23:01:01.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:01.940135+0000) 2022-01-31T23:01:02.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:02 smithi167 conmon[54076]: debug 2022-01-31T23:01:02.178+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:02.180327+0000) 2022-01-31T23:01:02.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:02 smithi171 conmon[46715]: debug 2022-01-31T23:01:02.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:02.174825+0000) 2022-01-31T23:01:02.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:02 smithi171 conmon[41853]: debug 2022-01-31T23:01:02.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:02.379227+0000) 2022-01-31T23:01:02.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:02 smithi171 conmon[51620]: debug 2022-01-31T23:01:02.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:02.554019+0000) 2022-01-31T23:01:03.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:03 smithi167 conmon[49112]: debug 2022-01-31T23:01:03.009+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:03.011123+0000) 2022-01-31T23:01:03.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:02 smithi167 conmon[60316]: debug 2022-01-31T23:01:02.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:02.940275+0000) 2022-01-31T23:01:03.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:03 smithi167 conmon[54076]: debug 2022-01-31T23:01:03.179+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:03.180545+0000) 2022-01-31T23:01:03.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:03 smithi171 conmon[46715]: debug 2022-01-31T23:01:03.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:03.175033+0000) 2022-01-31T23:01:03.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:03 smithi171 conmon[41853]: debug 2022-01-31T23:01:03.378+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:03.379428+0000) 2022-01-31T23:01:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:03 smithi171 conmon[51620]: debug 2022-01-31T23:01:03.553+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:03.554198+0000) 2022-01-31T23:01:04.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:04 smithi167 conmon[49112]: debug 2022-01-31T23:01:04.009+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:04.011305+0000) 2022-01-31T23:01:04.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:03 smithi167 conmon[60316]: debug 2022-01-31T23:01:03.938+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:03.940457+0000) 2022-01-31T23:01:04.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:04 smithi167 conmon[54076]: debug 2022-01-31T23:01:04.179+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:04.180698+0000) 2022-01-31T23:01:04.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:04 smithi171 conmon[46715]: debug 2022-01-31T23:01:04.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:04.175183+0000) 2022-01-31T23:01:04.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:04 smithi171 conmon[41853]: debug 2022-01-31T23:01:04.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:04.379639+0000) 2022-01-31T23:01:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:04 smithi171 conmon[51620]: debug 2022-01-31T23:01:04.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:04.554360+0000) 2022-01-31T23:01:05.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:05 smithi167 conmon[49112]: debug 2022-01-31T23:01:05.009+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:05.011443+0000) 2022-01-31T23:01:05.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:04 smithi167 conmon[60316]: debug 2022-01-31T23:01:04.939+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:04.940622+0000) 2022-01-31T23:01:05.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:05 smithi167 conmon[54076]: debug 2022-01-31T23:01:05.179+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:05.180860+0000) 2022-01-31T23:01:05.537 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:05 smithi171 conmon[46715]: debug 2022-01-31T23:01:05.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:05.175328+0000) 2022-01-31T23:01:05.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:05 smithi171 conmon[41853]: debug 2022-01-31T23:01:05.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:05.379792+0000) 2022-01-31T23:01:05.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:05 smithi171 conmon[51620]: debug 2022-01-31T23:01:05.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:05.554526+0000) 2022-01-31T23:01:06.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:05 smithi167 conmon[60316]: debug 2022-01-31T23:01:05.939+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:05.940742+0000) 2022-01-31T23:01:06.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:06 smithi167 conmon[49112]: debug 2022-01-31T23:01:06.010+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.011597+0000) 2022-01-31T23:01:06.519 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:06 smithi167 conmon[49112]: debug 2022-01-31T23:01:06.187+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.188978+0000) 2022-01-31T23:01:06.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:06 smithi167 conmon[54076]: debug 2022-01-31T23:01:06.179+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.181047+0000) 2022-01-31T23:01:06.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:06 smithi167 conmon[54076]: debug 2022-01-31T23:01:06.187+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.189463+0000) 2022-01-31T23:01:06.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:06 smithi167 conmon[60316]: debug 2022-01-31T23:01:06.187+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.188863+0000) 2022-01-31T23:01:06.538 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[35325]: debug 2022-01-31T23:01:06.214+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 255965 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:06.538 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[51620]: debug 2022-01-31T23:01:06.189+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.190530+0000) 2022-01-31T23:01:06.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[41853]: debug 2022-01-31T23:01:06.189+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.190170+0000) 2022-01-31T23:01:06.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[41853]: debug 2022-01-31T23:01:06.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.380010+0000) 2022-01-31T23:01:06.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[46715]: debug 2022-01-31T23:01:06.174+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.175508+0000) 2022-01-31T23:01:06.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[46715]: debug 2022-01-31T23:01:06.187+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.188245+0000) 2022-01-31T23:01:06.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:06 smithi171 conmon[51620]: debug 2022-01-31T23:01:06.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.554695+0000) 2022-01-31T23:01:07.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:07 smithi167 conmon[49112]: debug 2022-01-31T23:01:07.011+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:07.011773+0000) 2022-01-31T23:01:07.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:06 smithi167 conmon[60316]: debug 2022-01-31T23:01:06.940+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:06.940938+0000) 2022-01-31T23:01:07.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:07 smithi167 conmon[54076]: debug 2022-01-31T23:01:07.180+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:07.181251+0000) 2022-01-31T23:01:07.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:07 smithi171 conmon[41853]: debug 2022-01-31T23:01:07.379+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:07.380159+0000) 2022-01-31T23:01:07.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:07 smithi171 conmon[46715]: debug 2022-01-31T23:01:07.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:07.175717+0000) 2022-01-31T23:01:07.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:07 smithi171 conmon[51620]: debug 2022-01-31T23:01:07.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:07.554847+0000) 2022-01-31T23:01:08.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:08 smithi167 conmon[49112]: debug 2022-01-31T23:01:08.010+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:08.011875+0000) 2022-01-31T23:01:08.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:07 smithi167 conmon[60316]: debug 2022-01-31T23:01:07.940+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:07.941111+0000) 2022-01-31T23:01:08.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:08 smithi167 conmon[54076]: debug 2022-01-31T23:01:08.180+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:08.181426+0000) 2022-01-31T23:01:08.538 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:08 smithi171 conmon[46715]: debug 2022-01-31T23:01:08.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:08.175878+0000) 2022-01-31T23:01:08.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:08 smithi171 conmon[41853]: debug 2022-01-31T23:01:08.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:08.380351+0000) 2022-01-31T23:01:08.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:08 smithi171 conmon[51620]: debug 2022-01-31T23:01:08.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:08.555037+0000) 2022-01-31T23:01:09.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:09 smithi167 conmon[49112]: debug 2022-01-31T23:01:09.010+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:09.012051+0000) 2022-01-31T23:01:09.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:08 smithi167 conmon[60316]: debug 2022-01-31T23:01:08.940+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:08.941326+0000) 2022-01-31T23:01:09.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:08 smithi171 conmon[35325]: debug 2022-01-31T23:01:08.948+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:01:09.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:09 smithi171 conmon[46715]: debug 2022-01-31T23:01:09.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:09.176054+0000) 2022-01-31T23:01:09.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:09 smithi167 conmon[54076]: debug 2022-01-31T23:01:09.181+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:09.181553+0000) 2022-01-31T23:01:09.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:09 smithi171 conmon[41853]: debug 2022-01-31T23:01:09.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:09.380544+0000) 2022-01-31T23:01:09.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:09 smithi171 conmon[51620]: debug 2022-01-31T23:01:09.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:09.555199+0000) 2022-01-31T23:01:10.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:10 smithi167 conmon[49112]: debug 2022-01-31T23:01:10.011+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:10.012253+0000) 2022-01-31T23:01:10.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:09 smithi167 conmon[60316]: debug 2022-01-31T23:01:09.941+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:09.941487+0000) 2022-01-31T23:01:10.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:10 smithi167 conmon[54076]: debug 2022-01-31T23:01:10.181+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:10.181695+0000) 2022-01-31T23:01:10.538 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:10 smithi171 conmon[41853]: debug 2022-01-31T23:01:10.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:10.380730+0000) 2022-01-31T23:01:10.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:10 smithi171 conmon[46715]: debug 2022-01-31T23:01:10.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:10.176259+0000) 2022-01-31T23:01:10.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:10 smithi171 conmon[51620]: debug 2022-01-31T23:01:10.554+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:10.555398+0000) 2022-01-31T23:01:11.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:11 smithi167 conmon[49112]: debug 2022-01-31T23:01:11.011+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.012381+0000) 2022-01-31T23:01:11.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:10 smithi167 conmon[60316]: debug 2022-01-31T23:01:10.941+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:10.941660+0000) 2022-01-31T23:01:11.520 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:11 smithi167 conmon[49112]: debug 2022-01-31T23:01:11.217+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.217651+0000) 2022-01-31T23:01:11.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:11 smithi167 conmon[54076]: debug 2022-01-31T23:01:11.181+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.181872+0000) 2022-01-31T23:01:11.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:11 smithi167 conmon[54076]: debug 2022-01-31T23:01:11.218+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.218701+0000) 2022-01-31T23:01:11.522 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:11 smithi167 conmon[60316]: debug 2022-01-31T23:01:11.218+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.218944+0000) 2022-01-31T23:01:11.539 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[35325]: debug 2022-01-31T23:01:11.243+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256076 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:11.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[46715]: debug 2022-01-31T23:01:11.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.176434+0000) 2022-01-31T23:01:11.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[46715]: debug 2022-01-31T23:01:11.216+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.217465+0000) 2022-01-31T23:01:11.540 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[51620]: debug 2022-01-31T23:01:11.217+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.218570+0000) 2022-01-31T23:01:11.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[41853]: debug 2022-01-31T23:01:11.218+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.218920+0000) 2022-01-31T23:01:11.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[41853]: debug 2022-01-31T23:01:11.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.380845+0000) 2022-01-31T23:01:11.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:11 smithi171 conmon[51620]: debug 2022-01-31T23:01:11.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.555588+0000) 2022-01-31T23:01:12.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:12 smithi167 conmon[49112]: debug 2022-01-31T23:01:12.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:12.012550+0000) 2022-01-31T23:01:12.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:11 smithi167 conmon[60316]: debug 2022-01-31T23:01:11.941+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:11.941849+0000) 2022-01-31T23:01:12.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:12 smithi167 conmon[54076]: debug 2022-01-31T23:01:12.181+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:12.181970+0000) 2022-01-31T23:01:12.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:12 smithi171 conmon[46715]: debug 2022-01-31T23:01:12.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:12.176628+0000) 2022-01-31T23:01:12.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:12 smithi171 conmon[41853]: debug 2022-01-31T23:01:12.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:12.380990+0000) 2022-01-31T23:01:12.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:12 smithi171 conmon[51620]: debug 2022-01-31T23:01:12.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:12.555746+0000) 2022-01-31T23:01:13.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:13 smithi167 conmon[49112]: debug 2022-01-31T23:01:13.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:13.012699+0000) 2022-01-31T23:01:13.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:12 smithi167 conmon[60316]: debug 2022-01-31T23:01:12.941+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:12.942032+0000) 2022-01-31T23:01:13.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:13 smithi167 conmon[54076]: debug 2022-01-31T23:01:13.181+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:13.182147+0000) 2022-01-31T23:01:13.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:13 smithi171 conmon[41853]: debug 2022-01-31T23:01:13.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:13.381184+0000) 2022-01-31T23:01:13.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:13 smithi171 conmon[46715]: debug 2022-01-31T23:01:13.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:13.176815+0000) 2022-01-31T23:01:13.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:13 smithi171 conmon[51620]: debug 2022-01-31T23:01:13.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:13.555945+0000) 2022-01-31T23:01:14.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:14 smithi167 conmon[49112]: debug 2022-01-31T23:01:14.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:14.012850+0000) 2022-01-31T23:01:14.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:13 smithi167 conmon[60316]: debug 2022-01-31T23:01:13.941+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:13.942243+0000) 2022-01-31T23:01:14.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:14 smithi167 conmon[54076]: debug 2022-01-31T23:01:14.181+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:14.182355+0000) 2022-01-31T23:01:14.539 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:14 smithi171 conmon[41853]: debug 2022-01-31T23:01:14.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:14.381351+0000) 2022-01-31T23:01:14.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:14 smithi171 conmon[46715]: debug 2022-01-31T23:01:14.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:14.176957+0000) 2022-01-31T23:01:14.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:14 smithi171 conmon[51620]: debug 2022-01-31T23:01:14.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:14.556122+0000) 2022-01-31T23:01:15.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:15 smithi167 conmon[49112]: debug 2022-01-31T23:01:15.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:15.013056+0000) 2022-01-31T23:01:15.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:14 smithi167 conmon[60316]: debug 2022-01-31T23:01:14.941+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:14.942428+0000) 2022-01-31T23:01:15.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:15 smithi167 conmon[54076]: debug 2022-01-31T23:01:15.182+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:15.182586+0000) 2022-01-31T23:01:15.539 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:15 smithi171 conmon[46715]: debug 2022-01-31T23:01:15.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:15.177121+0000) 2022-01-31T23:01:15.540 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:15 smithi171 conmon[41853]: debug 2022-01-31T23:01:15.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:15.381499+0000) 2022-01-31T23:01:15.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:15 smithi171 conmon[51620]: debug 2022-01-31T23:01:15.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:15.556312+0000) 2022-01-31T23:01:16.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:15 smithi167 conmon[60316]: debug 2022-01-31T23:01:15.942+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:15.942591+0000) 2022-01-31T23:01:16.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:16 smithi167 conmon[49112]: debug 2022-01-31T23:01:16.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.013218+0000) 2022-01-31T23:01:16.520 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:16 smithi167 conmon[49112]: debug 2022-01-31T23:01:16.246+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.247013+0000) 2022-01-31T23:01:16.521 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:16 smithi167 conmon[60316]: debug 2022-01-31T23:01:16.247+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.247530+0000) 2022-01-31T23:01:16.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:16 smithi167 conmon[54076]: debug 2022-01-31T23:01:16.182+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.182741+0000) 2022-01-31T23:01:16.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:16 smithi167 conmon[54076]: debug 2022-01-31T23:01:16.246+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.247359+0000) 2022-01-31T23:01:16.539 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[35325]: debug 2022-01-31T23:01:16.272+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256188 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:16.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[46715]: debug 2022-01-31T23:01:16.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.177305+0000) 2022-01-31T23:01:16.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[46715]: debug 2022-01-31T23:01:16.246+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.246780+0000) 2022-01-31T23:01:16.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[41853]: debug 2022-01-31T23:01:16.245+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.246573+0000) 2022-01-31T23:01:16.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[41853]: debug 2022-01-31T23:01:16.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.381685+0000) 2022-01-31T23:01:16.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[51620]: debug 2022-01-31T23:01:16.247+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.248088+0000) 2022-01-31T23:01:16.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:16 smithi171 conmon[51620]: debug 2022-01-31T23:01:16.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.556492+0000) 2022-01-31T23:01:17.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:17 smithi167 conmon[49112]: debug 2022-01-31T23:01:17.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:17.013364+0000) 2022-01-31T23:01:17.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:16 smithi167 conmon[60316]: debug 2022-01-31T23:01:16.942+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:16.942763+0000) 2022-01-31T23:01:17.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:17 smithi167 conmon[54076]: debug 2022-01-31T23:01:17.182+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:17.182923+0000) 2022-01-31T23:01:17.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:17 smithi171 conmon[46715]: debug 2022-01-31T23:01:17.175+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:17.177524+0000) 2022-01-31T23:01:17.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:17 smithi171 conmon[41853]: debug 2022-01-31T23:01:17.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:17.381862+0000) 2022-01-31T23:01:17.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:17 smithi171 conmon[51620]: debug 2022-01-31T23:01:17.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:17.556628+0000) 2022-01-31T23:01:18.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:18 smithi167 conmon[49112]: debug 2022-01-31T23:01:18.012+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:18.013518+0000) 2022-01-31T23:01:18.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:17 smithi167 conmon[60316]: debug 2022-01-31T23:01:17.942+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:17.942913+0000) 2022-01-31T23:01:18.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:18 smithi167 conmon[54076]: debug 2022-01-31T23:01:18.182+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:18.183071+0000) 2022-01-31T23:01:18.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:18 smithi171 conmon[46715]: debug 2022-01-31T23:01:18.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:18.177692+0000) 2022-01-31T23:01:18.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:18 smithi171 conmon[41853]: debug 2022-01-31T23:01:18.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:18.382050+0000) 2022-01-31T23:01:18.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:18 smithi171 conmon[51620]: debug 2022-01-31T23:01:18.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:18.556812+0000) 2022-01-31T23:01:19.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:19 smithi167 conmon[49112]: debug 2022-01-31T23:01:19.013+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:19.013705+0000) 2022-01-31T23:01:19.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:18 smithi167 conmon[60316]: debug 2022-01-31T23:01:18.942+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:18.943094+0000) 2022-01-31T23:01:19.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:19 smithi167 conmon[54076]: debug 2022-01-31T23:01:19.182+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:19.183244+0000) 2022-01-31T23:01:19.540 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:19 smithi171 conmon[41853]: debug 2022-01-31T23:01:19.380+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:19.382272+0000) 2022-01-31T23:01:19.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:19 smithi171 conmon[46715]: debug 2022-01-31T23:01:19.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:19.177896+0000) 2022-01-31T23:01:19.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:19 smithi171 conmon[51620]: debug 2022-01-31T23:01:19.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:19.556995+0000) 2022-01-31T23:01:20.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:19 smithi167 conmon[60316]: debug 2022-01-31T23:01:19.942+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:19.943329+0000) 2022-01-31T23:01:20.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:20 smithi167 conmon[49112]: debug 2022-01-31T23:01:20.013+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:20.013866+0000) 2022-01-31T23:01:20.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:20 smithi167 conmon[54076]: debug 2022-01-31T23:01:20.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:20.183425+0000) 2022-01-31T23:01:20.540 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:20 smithi171 conmon[41853]: debug 2022-01-31T23:01:20.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:20.382459+0000) 2022-01-31T23:01:20.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:20 smithi171 conmon[46715]: debug 2022-01-31T23:01:20.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:20.178108+0000) 2022-01-31T23:01:20.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:20 smithi171 conmon[51620]: debug 2022-01-31T23:01:20.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:20.557188+0000) 2022-01-31T23:01:21.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:20 smithi167 conmon[60316]: debug 2022-01-31T23:01:20.943+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:20.943520+0000) 2022-01-31T23:01:21.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:21 smithi167 conmon[49112]: debug 2022-01-31T23:01:21.013+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.014064+0000) 2022-01-31T23:01:21.521 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:21 smithi167 conmon[49112]: debug 2022-01-31T23:01:21.275+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.276095+0000) 2022-01-31T23:01:21.522 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:21 smithi167 conmon[60316]: debug 2022-01-31T23:01:21.276+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.276898+0000) 2022-01-31T23:01:21.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:21 smithi167 conmon[54076]: debug 2022-01-31T23:01:21.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.183589+0000) 2022-01-31T23:01:21.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:21 smithi167 conmon[54076]: debug 2022-01-31T23:01:21.275+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.276300+0000) 2022-01-31T23:01:21.540 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[35325]: debug 2022-01-31T23:01:21.300+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256301 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:21.541 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[51620]: debug 2022-01-31T23:01:21.275+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.277075+0000) 2022-01-31T23:01:21.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[41853]: debug 2022-01-31T23:01:21.273+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.275628+0000) 2022-01-31T23:01:21.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[41853]: debug 2022-01-31T23:01:21.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.382615+0000) 2022-01-31T23:01:21.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[46715]: debug 2022-01-31T23:01:21.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.178347+0000) 2022-01-31T23:01:21.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[46715]: debug 2022-01-31T23:01:21.273+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.275455+0000) 2022-01-31T23:01:21.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:21 smithi171 conmon[51620]: debug 2022-01-31T23:01:21.555+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.557342+0000) 2022-01-31T23:01:22.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:21 smithi167 conmon[60316]: debug 2022-01-31T23:01:21.943+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:21.943677+0000) 2022-01-31T23:01:22.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:22 smithi167 conmon[49112]: debug 2022-01-31T23:01:22.013+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:22.014274+0000) 2022-01-31T23:01:22.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:22 smithi167 conmon[54076]: debug 2022-01-31T23:01:22.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:22.183774+0000) 2022-01-31T23:01:22.540 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:22 smithi171 conmon[46715]: debug 2022-01-31T23:01:22.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:22.178550+0000) 2022-01-31T23:01:22.541 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:22 smithi171 conmon[41853]: debug 2022-01-31T23:01:22.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:22.382781+0000) 2022-01-31T23:01:22.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:22 smithi171 conmon[51620]: debug 2022-01-31T23:01:22.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:22.557514+0000) 2022-01-31T23:01:23.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:23 smithi167 conmon[49112]: debug 2022-01-31T23:01:23.013+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:23.014414+0000) 2022-01-31T23:01:23.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:22 smithi167 conmon[60316]: debug 2022-01-31T23:01:22.943+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:22.943829+0000) 2022-01-31T23:01:23.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:23 smithi167 conmon[54076]: debug 2022-01-31T23:01:23.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:23.183875+0000) 2022-01-31T23:01:23.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:23 smithi171 conmon[46715]: debug 2022-01-31T23:01:23.176+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:23.178665+0000) 2022-01-31T23:01:23.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:23 smithi171 conmon[41853]: debug 2022-01-31T23:01:23.381+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:23.382917+0000) 2022-01-31T23:01:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:23 smithi171 conmon[51620]: debug 2022-01-31T23:01:23.556+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:23.557707+0000) 2022-01-31T23:01:24.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:24 smithi167 conmon[49112]: debug 2022-01-31T23:01:24.014+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:24.014569+0000) 2022-01-31T23:01:24.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:23 smithi167 conmon[60316]: debug 2022-01-31T23:01:23.943+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:23.943979+0000) 2022-01-31T23:01:24.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:23 smithi171 conmon[35325]: debug 2022-01-31T23:01:23.948+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:01:24.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:24 smithi171 conmon[46715]: debug 2022-01-31T23:01:24.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:24.178840+0000) 2022-01-31T23:01:24.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:24 smithi167 conmon[54076]: debug 2022-01-31T23:01:24.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:24.184040+0000) 2022-01-31T23:01:24.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:24 smithi171 conmon[41853]: debug 2022-01-31T23:01:24.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:24.383094+0000) 2022-01-31T23:01:24.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:24 smithi171 conmon[51620]: debug 2022-01-31T23:01:24.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:24.557923+0000) 2022-01-31T23:01:25.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:25 smithi167 conmon[49112]: debug 2022-01-31T23:01:25.014+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:25.014718+0000) 2022-01-31T23:01:25.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:24 smithi167 conmon[60316]: debug 2022-01-31T23:01:24.943+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:24.944211+0000) 2022-01-31T23:01:25.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:25 smithi167 conmon[54076]: debug 2022-01-31T23:01:25.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:25.184178+0000) 2022-01-31T23:01:25.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:25 smithi171 conmon[46715]: debug 2022-01-31T23:01:25.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:25.178997+0000) 2022-01-31T23:01:25.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:25 smithi171 conmon[41853]: debug 2022-01-31T23:01:25.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:25.383252+0000) 2022-01-31T23:01:25.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:25 smithi171 conmon[51620]: debug 2022-01-31T23:01:25.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:25.558106+0000) 2022-01-31T23:01:26.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:25 smithi167 conmon[60316]: debug 2022-01-31T23:01:25.943+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:25.944437+0000) 2022-01-31T23:01:26.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:26 smithi167 conmon[49112]: debug 2022-01-31T23:01:26.014+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.014838+0000) 2022-01-31T23:01:26.522 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:26 smithi167 conmon[49112]: debug 2022-01-31T23:01:26.304+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.305182+0000) 2022-01-31T23:01:26.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:26 smithi167 conmon[54076]: debug 2022-01-31T23:01:26.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.184279+0000) 2022-01-31T23:01:26.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:26 smithi167 conmon[54076]: debug 2022-01-31T23:01:26.310+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.311087+0000) 2022-01-31T23:01:26.524 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:26 smithi167 conmon[60316]: debug 2022-01-31T23:01:26.305+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.305701+0000) 2022-01-31T23:01:26.541 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[35325]: debug 2022-01-31T23:01:26.334+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256414 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:26.542 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[51620]: debug 2022-01-31T23:01:26.303+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.305066+0000) 2022-01-31T23:01:26.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[41853]: debug 2022-01-31T23:01:26.302+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.304536+0000) 2022-01-31T23:01:26.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[41853]: debug 2022-01-31T23:01:26.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.383452+0000) 2022-01-31T23:01:26.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[46715]: debug 2022-01-31T23:01:26.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.179225+0000) 2022-01-31T23:01:26.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[46715]: debug 2022-01-31T23:01:26.302+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.304567+0000) 2022-01-31T23:01:26.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:26 smithi171 conmon[51620]: debug 2022-01-31T23:01:26.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.558271+0000) 2022-01-31T23:01:27.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:26 smithi167 conmon[60316]: debug 2022-01-31T23:01:26.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:26.944580+0000) 2022-01-31T23:01:27.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:27 smithi167 conmon[49112]: debug 2022-01-31T23:01:27.014+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:27.015025+0000) 2022-01-31T23:01:27.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:27 smithi167 conmon[54076]: debug 2022-01-31T23:01:27.183+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:27.184459+0000) 2022-01-31T23:01:27.541 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:27 smithi171 conmon[46715]: debug 2022-01-31T23:01:27.177+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:27.179456+0000) 2022-01-31T23:01:27.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:27 smithi171 conmon[41853]: debug 2022-01-31T23:01:27.382+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:27.383603+0000) 2022-01-31T23:01:27.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:27 smithi171 conmon[51620]: debug 2022-01-31T23:01:27.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:27.558424+0000) 2022-01-31T23:01:28.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:27 smithi167 conmon[60316]: debug 2022-01-31T23:01:27.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:27.944714+0000) 2022-01-31T23:01:28.339 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:28 smithi167 conmon[49112]: debug 2022-01-31T23:01:28.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:28.015142+0000) 2022-01-31T23:01:28.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:28 smithi167 conmon[54076]: debug 2022-01-31T23:01:28.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:28.184615+0000) 2022-01-31T23:01:28.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:28 smithi171 conmon[41853]: debug 2022-01-31T23:01:28.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:28.383819+0000) 2022-01-31T23:01:28.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:28 smithi171 conmon[46715]: debug 2022-01-31T23:01:28.178+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:28.179653+0000) 2022-01-31T23:01:28.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:28 smithi171 conmon[51620]: debug 2022-01-31T23:01:28.557+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:28.558621+0000) 2022-01-31T23:01:29.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:28 smithi167 conmon[60316]: debug 2022-01-31T23:01:28.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:28.944836+0000) 2022-01-31T23:01:29.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:29 smithi167 conmon[49112]: debug 2022-01-31T23:01:29.014+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:29.015318+0000) 2022-01-31T23:01:29.522 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:29 smithi167 conmon[54076]: debug 2022-01-31T23:01:29.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:29.184821+0000) 2022-01-31T23:01:29.542 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:29 smithi171 conmon[46715]: debug 2022-01-31T23:01:29.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:29.179854+0000) 2022-01-31T23:01:29.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:29 smithi171 conmon[41853]: debug 2022-01-31T23:01:29.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:29.384001+0000) 2022-01-31T23:01:29.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:29 smithi171 conmon[51620]: debug 2022-01-31T23:01:29.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:29.558839+0000) 2022-01-31T23:01:30.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:29 smithi167 conmon[60316]: debug 2022-01-31T23:01:29.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:29.944975+0000) 2022-01-31T23:01:30.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:30 smithi167 conmon[49112]: debug 2022-01-31T23:01:30.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:30.015548+0000) 2022-01-31T23:01:30.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:30 smithi167 conmon[54076]: debug 2022-01-31T23:01:30.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:30.185005+0000) 2022-01-31T23:01:30.542 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:30 smithi171 conmon[41853]: debug 2022-01-31T23:01:30.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:30.384207+0000) 2022-01-31T23:01:30.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:30 smithi171 conmon[46715]: debug 2022-01-31T23:01:30.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:30.179999+0000) 2022-01-31T23:01:30.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:30 smithi171 conmon[51620]: debug 2022-01-31T23:01:30.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:30.559005+0000) 2022-01-31T23:01:31.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:30 smithi167 conmon[60316]: debug 2022-01-31T23:01:30.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:30.945111+0000) 2022-01-31T23:01:31.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:31 smithi167 conmon[49112]: debug 2022-01-31T23:01:31.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.015720+0000) 2022-01-31T23:01:31.523 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:31 smithi167 conmon[49112]: debug 2022-01-31T23:01:31.338+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.339478+0000) 2022-01-31T23:01:31.524 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:31 smithi167 conmon[60316]: debug 2022-01-31T23:01:31.338+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.339278+0000) 2022-01-31T23:01:31.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:31 smithi167 conmon[54076]: debug 2022-01-31T23:01:31.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.185233+0000) 2022-01-31T23:01:31.525 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:31 smithi167 conmon[54076]: debug 2022-01-31T23:01:31.339+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.340021+0000) 2022-01-31T23:01:31.542 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[35325]: debug 2022-01-31T23:01:31.365+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256525 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:31.543 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[51620]: debug 2022-01-31T23:01:31.337+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.338571+0000) 2022-01-31T23:01:31.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[41853]: debug 2022-01-31T23:01:31.338+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.339448+0000) 2022-01-31T23:01:31.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[41853]: debug 2022-01-31T23:01:31.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.384385+0000) 2022-01-31T23:01:31.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[46715]: debug 2022-01-31T23:01:31.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.180169+0000) 2022-01-31T23:01:31.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[46715]: debug 2022-01-31T23:01:31.337+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.338465+0000) 2022-01-31T23:01:31.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:31 smithi171 conmon[51620]: debug 2022-01-31T23:01:31.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.559184+0000) 2022-01-31T23:01:32.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:32 smithi167 conmon[49112]: debug 2022-01-31T23:01:32.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:32.015820+0000) 2022-01-31T23:01:32.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:31 smithi167 conmon[60316]: debug 2022-01-31T23:01:31.945+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:31.945266+0000) 2022-01-31T23:01:32.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:32 smithi167 conmon[54076]: debug 2022-01-31T23:01:32.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:32.185407+0000) 2022-01-31T23:01:32.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:32 smithi171 conmon[41853]: debug 2022-01-31T23:01:32.383+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:32.384542+0000) 2022-01-31T23:01:32.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:32 smithi171 conmon[46715]: debug 2022-01-31T23:01:32.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:32.180366+0000) 2022-01-31T23:01:32.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:32 smithi171 conmon[51620]: debug 2022-01-31T23:01:32.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:32.559336+0000) 2022-01-31T23:01:33.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:33 smithi167 conmon[49112]: debug 2022-01-31T23:01:33.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:33.015993+0000) 2022-01-31T23:01:33.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:32 smithi167 conmon[60316]: debug 2022-01-31T23:01:32.945+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:32.945451+0000) 2022-01-31T23:01:33.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:33 smithi167 conmon[54076]: debug 2022-01-31T23:01:33.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:33.185586+0000) 2022-01-31T23:01:33.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:33 smithi171 conmon[41853]: debug 2022-01-31T23:01:33.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:33.384744+0000) 2022-01-31T23:01:33.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:33 smithi171 conmon[46715]: debug 2022-01-31T23:01:33.179+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:33.180543+0000) 2022-01-31T23:01:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:33 smithi171 conmon[51620]: debug 2022-01-31T23:01:33.558+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:33.559543+0000) 2022-01-31T23:01:34.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:34 smithi167 conmon[49112]: debug 2022-01-31T23:01:34.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:34.016196+0000) 2022-01-31T23:01:34.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:33 smithi167 conmon[60316]: debug 2022-01-31T23:01:33.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:33.945609+0000) 2022-01-31T23:01:34.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:34 smithi167 conmon[54076]: debug 2022-01-31T23:01:34.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:34.185750+0000) 2022-01-31T23:01:34.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:34 smithi171 conmon[41853]: debug 2022-01-31T23:01:34.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:34.384930+0000) 2022-01-31T23:01:34.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:34 smithi171 conmon[46715]: debug 2022-01-31T23:01:34.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:34.180772+0000) 2022-01-31T23:01:34.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:34 smithi171 conmon[51620]: debug 2022-01-31T23:01:34.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:34.559733+0000) 2022-01-31T23:01:35.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:35 smithi167 conmon[49112]: debug 2022-01-31T23:01:35.014+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:35.016382+0000) 2022-01-31T23:01:35.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:34 smithi167 conmon[60316]: debug 2022-01-31T23:01:34.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:34.945787+0000) 2022-01-31T23:01:35.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:35 smithi167 conmon[54076]: debug 2022-01-31T23:01:35.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:35.185914+0000) 2022-01-31T23:01:35.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:35 smithi171 conmon[41853]: debug 2022-01-31T23:01:35.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:35.385081+0000) 2022-01-31T23:01:35.543 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:35 smithi171 conmon[46715]: debug 2022-01-31T23:01:35.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:35.180951+0000) 2022-01-31T23:01:35.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:35 smithi171 conmon[51620]: debug 2022-01-31T23:01:35.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:35.559954+0000) 2022-01-31T23:01:36.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:36 smithi167 conmon[49112]: debug 2022-01-31T23:01:36.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.016572+0000) 2022-01-31T23:01:36.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:35 smithi167 conmon[60316]: debug 2022-01-31T23:01:35.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:35.945966+0000) 2022-01-31T23:01:36.524 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:36 smithi167 conmon[49112]: debug 2022-01-31T23:01:36.368+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.369938+0000) 2022-01-31T23:01:36.525 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:36 smithi167 conmon[60316]: debug 2022-01-31T23:01:36.368+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.369661+0000) 2022-01-31T23:01:36.525 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:36 smithi167 conmon[54076]: debug 2022-01-31T23:01:36.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.186117+0000) 2022-01-31T23:01:36.526 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:36 smithi167 conmon[54076]: debug 2022-01-31T23:01:36.368+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.370038+0000) 2022-01-31T23:01:36.543 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[35325]: debug 2022-01-31T23:01:36.392+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256637 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:36.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[46715]: debug 2022-01-31T23:01:36.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.181143+0000) 2022-01-31T23:01:36.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[46715]: debug 2022-01-31T23:01:36.367+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.368268+0000) 2022-01-31T23:01:36.545 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[51620]: debug 2022-01-31T23:01:36.368+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.369016+0000) 2022-01-31T23:01:36.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[41853]: debug 2022-01-31T23:01:36.368+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.369440+0000) 2022-01-31T23:01:36.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[41853]: debug 2022-01-31T23:01:36.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.385239+0000) 2022-01-31T23:01:36.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:36 smithi171 conmon[51620]: debug 2022-01-31T23:01:36.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.560149+0000) 2022-01-31T23:01:37.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:37 smithi167 conmon[49112]: debug 2022-01-31T23:01:37.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:37.016731+0000) 2022-01-31T23:01:37.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:36 smithi167 conmon[60316]: debug 2022-01-31T23:01:36.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:36.946134+0000) 2022-01-31T23:01:37.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:37 smithi167 conmon[54076]: debug 2022-01-31T23:01:37.184+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:37.186289+0000) 2022-01-31T23:01:37.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:37 smithi171 conmon[41853]: debug 2022-01-31T23:01:37.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:37.385442+0000) 2022-01-31T23:01:37.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:37 smithi171 conmon[46715]: debug 2022-01-31T23:01:37.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:37.181350+0000) 2022-01-31T23:01:37.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:37 smithi171 conmon[51620]: debug 2022-01-31T23:01:37.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:37.560281+0000) 2022-01-31T23:01:38.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:38 smithi167 conmon[49112]: debug 2022-01-31T23:01:38.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:38.016832+0000) 2022-01-31T23:01:38.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:37 smithi167 conmon[60316]: debug 2022-01-31T23:01:37.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:37.946289+0000) 2022-01-31T23:01:38.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:38 smithi167 conmon[54076]: debug 2022-01-31T23:01:38.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:38.186434+0000) 2022-01-31T23:01:38.543 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:38 smithi171 conmon[41853]: debug 2022-01-31T23:01:38.384+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:38.385623+0000) 2022-01-31T23:01:38.544 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:38 smithi171 conmon[46715]: debug 2022-01-31T23:01:38.180+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:38.181509+0000) 2022-01-31T23:01:38.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:38 smithi171 conmon[51620]: debug 2022-01-31T23:01:38.559+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:38.560466+0000) 2022-01-31T23:01:39.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:39 smithi167 conmon[49112]: debug 2022-01-31T23:01:39.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:39.016972+0000) 2022-01-31T23:01:39.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:38 smithi167 conmon[60316]: debug 2022-01-31T23:01:38.944+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:38.946513+0000) 2022-01-31T23:01:39.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:38 smithi171 conmon[35325]: debug 2022-01-31T23:01:38.949+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:01:39.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:39 smithi171 conmon[46715]: debug 2022-01-31T23:01:39.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:39.181691+0000) 2022-01-31T23:01:39.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:39 smithi167 conmon[54076]: debug 2022-01-31T23:01:39.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:39.186587+0000) 2022-01-31T23:01:39.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:39 smithi171 conmon[41853]: debug 2022-01-31T23:01:39.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:39.385803+0000) 2022-01-31T23:01:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:39 smithi171 conmon[51620]: debug 2022-01-31T23:01:39.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:39.560655+0000) 2022-01-31T23:01:40.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:40 smithi167 conmon[49112]: debug 2022-01-31T23:01:40.016+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:40.017177+0000) 2022-01-31T23:01:40.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:39 smithi167 conmon[60316]: debug 2022-01-31T23:01:39.945+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:39.946708+0000) 2022-01-31T23:01:40.421 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:40 smithi167 conmon[54076]: debug 2022-01-31T23:01:40.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:40.186743+0000) 2022-01-31T23:01:40.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:40 smithi171 conmon[41853]: debug 2022-01-31T23:01:40.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:40.385991+0000) 2022-01-31T23:01:40.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:40 smithi171 conmon[46715]: debug 2022-01-31T23:01:40.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:40.181901+0000) 2022-01-31T23:01:40.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:40 smithi171 conmon[51620]: debug 2022-01-31T23:01:40.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:40.560867+0000) 2022-01-31T23:01:41.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:41 smithi167 conmon[49112]: debug 2022-01-31T23:01:41.015+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.017283+0000) 2022-01-31T23:01:41.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:40 smithi167 conmon[60316]: debug 2022-01-31T23:01:40.945+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:40.946868+0000) 2022-01-31T23:01:41.424 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:41 smithi167 conmon[49112]: debug 2022-01-31T23:01:41.395+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.396863+0000) 2022-01-31T23:01:41.425 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:41 smithi167 conmon[60316]: debug 2022-01-31T23:01:41.395+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.397372+0000) 2022-01-31T23:01:41.426 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:41 smithi167 conmon[54076]: debug 2022-01-31T23:01:41.186+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.186915+0000) 2022-01-31T23:01:41.426 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:41 smithi167 conmon[54076]: debug 2022-01-31T23:01:41.396+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.397552+0000) 2022-01-31T23:01:41.544 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[35325]: debug 2022-01-31T23:01:41.422+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256748 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:41.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[46715]: debug 2022-01-31T23:01:41.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.182109+0000) 2022-01-31T23:01:41.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[46715]: debug 2022-01-31T23:01:41.394+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.395471+0000) 2022-01-31T23:01:41.546 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[51620]: debug 2022-01-31T23:01:41.394+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.395486+0000) 2022-01-31T23:01:41.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[41853]: debug 2022-01-31T23:01:41.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.386146+0000) 2022-01-31T23:01:41.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[41853]: debug 2022-01-31T23:01:41.395+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.396569+0000) 2022-01-31T23:01:41.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:41 smithi171 conmon[51620]: debug 2022-01-31T23:01:41.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.561014+0000) 2022-01-31T23:01:42.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:42 smithi167 conmon[49112]: debug 2022-01-31T23:01:42.016+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:42.017416+0000) 2022-01-31T23:01:42.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:41 smithi167 conmon[60316]: debug 2022-01-31T23:01:41.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:41.947054+0000) 2022-01-31T23:01:42.428 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:42 smithi167 conmon[54076]: debug 2022-01-31T23:01:42.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:42.187120+0000) 2022-01-31T23:01:42.544 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:42 smithi171 conmon[41853]: debug 2022-01-31T23:01:42.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:42.386321+0000) 2022-01-31T23:01:42.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:42 smithi171 conmon[46715]: debug 2022-01-31T23:01:42.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:42.182326+0000) 2022-01-31T23:01:42.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:42 smithi171 conmon[51620]: debug 2022-01-31T23:01:42.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:42.561195+0000) 2022-01-31T23:01:43.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:43 smithi167 conmon[49112]: debug 2022-01-31T23:01:43.016+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:43.017556+0000) 2022-01-31T23:01:43.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:42 smithi167 conmon[60316]: debug 2022-01-31T23:01:42.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:42.947214+0000) 2022-01-31T23:01:43.431 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:43 smithi167 conmon[54076]: debug 2022-01-31T23:01:43.185+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:43.187293+0000) 2022-01-31T23:01:43.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:43 smithi171 conmon[46715]: debug 2022-01-31T23:01:43.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:43.182450+0000) 2022-01-31T23:01:43.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:43 smithi171 conmon[41853]: debug 2022-01-31T23:01:43.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:43.386494+0000) 2022-01-31T23:01:43.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:43 smithi171 conmon[51620]: debug 2022-01-31T23:01:43.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:43.561395+0000) 2022-01-31T23:01:44.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:44 smithi167 conmon[49112]: debug 2022-01-31T23:01:44.016+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:44.017721+0000) 2022-01-31T23:01:44.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:43 smithi167 conmon[60316]: debug 2022-01-31T23:01:43.945+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:43.947422+0000) 2022-01-31T23:01:44.435 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:44 smithi167 conmon[54076]: debug 2022-01-31T23:01:44.186+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:44.187532+0000) 2022-01-31T23:01:44.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:44 smithi171 conmon[41853]: debug 2022-01-31T23:01:44.385+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:44.386699+0000) 2022-01-31T23:01:44.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:44 smithi171 conmon[46715]: debug 2022-01-31T23:01:44.181+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:44.182639+0000) 2022-01-31T23:01:44.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:44 smithi171 conmon[51620]: debug 2022-01-31T23:01:44.560+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:44.561586+0000) 2022-01-31T23:01:45.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:45 smithi167 conmon[49112]: debug 2022-01-31T23:01:45.017+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:45.017906+0000) 2022-01-31T23:01:45.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:44 smithi167 conmon[60316]: debug 2022-01-31T23:01:44.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:44.947597+0000) 2022-01-31T23:01:45.439 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:45 smithi167 conmon[54076]: debug 2022-01-31T23:01:45.186+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:45.187747+0000) 2022-01-31T23:01:45.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:45 smithi171 conmon[46715]: debug 2022-01-31T23:01:45.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:45.182846+0000) 2022-01-31T23:01:45.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:45 smithi171 conmon[41853]: debug 2022-01-31T23:01:45.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:45.386930+0000) 2022-01-31T23:01:45.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:45 smithi171 conmon[51620]: debug 2022-01-31T23:01:45.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:45.561789+0000) 2022-01-31T23:01:46.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:46 smithi167 conmon[49112]: debug 2022-01-31T23:01:46.016+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.018106+0000) 2022-01-31T23:01:46.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:45 smithi167 conmon[60316]: debug 2022-01-31T23:01:45.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:45.947793+0000) 2022-01-31T23:01:46.435 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[51620]: debug 2022-01-31T23:01:46.424+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.424998+0000) 2022-01-31T23:01:46.436 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[41853]: debug 2022-01-31T23:01:46.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.387114+0000) 2022-01-31T23:01:46.436 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[41853]: debug 2022-01-31T23:01:46.424+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.424806+0000) 2022-01-31T23:01:46.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[46715]: debug 2022-01-31T23:01:46.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.182997+0000) 2022-01-31T23:01:46.437 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[46715]: debug 2022-01-31T23:01:46.424+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.425053+0000) 2022-01-31T23:01:46.442 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:46 smithi167 conmon[49112]: debug 2022-01-31T23:01:46.424+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.426042+0000) 2022-01-31T23:01:46.443 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:46 smithi167 conmon[54076]: debug 2022-01-31T23:01:46.186+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.187894+0000) 2022-01-31T23:01:46.443 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:46 smithi167 conmon[54076]: debug 2022-01-31T23:01:46.424+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.426181+0000) 2022-01-31T23:01:46.443 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:46 smithi167 conmon[60316]: debug 2022-01-31T23:01:46.423+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.425191+0000) 2022-01-31T23:01:46.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[35325]: debug 2022-01-31T23:01:46.450+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256860 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:46.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:46 smithi171 conmon[51620]: debug 2022-01-31T23:01:46.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.561969+0000) 2022-01-31T23:01:46.975 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:46 smithi167 conmon[60316]: debug 2022-01-31T23:01:46.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:46.947995+0000) 2022-01-31T23:01:47.403 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:47 smithi167 conmon[49112]: debug 2022-01-31T23:01:47.016+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:47.018314+0000) 2022-01-31T23:01:47.403 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:47 smithi167 conmon[54076]: debug 2022-01-31T23:01:47.186+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:47.188089+0000) 2022-01-31T23:01:47.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:47 smithi171 conmon[41853]: debug 2022-01-31T23:01:47.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:47.387293+0000) 2022-01-31T23:01:47.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:47 smithi171 conmon[46715]: debug 2022-01-31T23:01:47.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:47.183191+0000) 2022-01-31T23:01:47.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:47 smithi171 conmon[51620]: debug 2022-01-31T23:01:47.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:47.562133+0000) 2022-01-31T23:01:48.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:48 smithi167 conmon[49112]: debug 2022-01-31T23:01:48.017+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:48.018485+0000) 2022-01-31T23:01:48.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:47 smithi167 conmon[60316]: debug 2022-01-31T23:01:47.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:47.948131+0000) 2022-01-31T23:01:48.449 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:48 smithi167 conmon[54076]: debug 2022-01-31T23:01:48.187+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:48.188245+0000) 2022-01-31T23:01:48.545 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:48 smithi171 conmon[46715]: debug 2022-01-31T23:01:48.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:48.183352+0000) 2022-01-31T23:01:48.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:48 smithi171 conmon[41853]: debug 2022-01-31T23:01:48.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:48.387445+0000) 2022-01-31T23:01:48.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:48 smithi171 conmon[51620]: debug 2022-01-31T23:01:48.561+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:48.562369+0000) 2022-01-31T23:01:49.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:49 smithi167 conmon[49112]: debug 2022-01-31T23:01:49.017+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:49.018706+0000) 2022-01-31T23:01:49.172 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:48 smithi167 conmon[60316]: debug 2022-01-31T23:01:48.946+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:48.948306+0000) 2022-01-31T23:01:49.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:49 smithi167 conmon[54076]: debug 2022-01-31T23:01:49.187+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:49.188437+0000) 2022-01-31T23:01:49.545 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:49 smithi171 conmon[41853]: debug 2022-01-31T23:01:49.386+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:49.387602+0000) 2022-01-31T23:01:49.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:49 smithi171 conmon[46715]: debug 2022-01-31T23:01:49.182+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:49.183569+0000) 2022-01-31T23:01:49.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:49 smithi171 conmon[51620]: debug 2022-01-31T23:01:49.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:49.562596+0000) 2022-01-31T23:01:50.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:50 smithi167 conmon[49112]: debug 2022-01-31T23:01:50.017+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:50.018864+0000) 2022-01-31T23:01:50.172 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:49 smithi167 conmon[60316]: debug 2022-01-31T23:01:49.947+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:49.948524+0000) 2022-01-31T23:01:50.456 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:50 smithi167 conmon[54076]: debug 2022-01-31T23:01:50.187+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:50.188622+0000) 2022-01-31T23:01:50.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:50 smithi171 conmon[41853]: debug 2022-01-31T23:01:50.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:50.387793+0000) 2022-01-31T23:01:50.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:50 smithi171 conmon[46715]: debug 2022-01-31T23:01:50.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:50.183744+0000) 2022-01-31T23:01:50.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:50 smithi171 conmon[51620]: debug 2022-01-31T23:01:50.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:50.562767+0000) 2022-01-31T23:01:51.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:51 smithi167 conmon[49112]: debug 2022-01-31T23:01:51.018+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.019047+0000) 2022-01-31T23:01:51.173 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:50 smithi167 conmon[60316]: debug 2022-01-31T23:01:50.947+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:50.948705+0000) 2022-01-31T23:01:51.436 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[46715]: debug 2022-01-31T23:01:51.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.183874+0000) 2022-01-31T23:01:51.437 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[41853]: debug 2022-01-31T23:01:51.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.387974+0000) 2022-01-31T23:01:51.438 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:51 smithi167 conmon[54076]: debug 2022-01-31T23:01:51.187+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.188764+0000) 2022-01-31T23:01:51.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[35325]: debug 2022-01-31T23:01:51.479+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 256987 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:51.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[41853]: debug 2022-01-31T23:01:51.455+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.455773+0000) 2022-01-31T23:01:51.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[46715]: debug 2022-01-31T23:01:51.452+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.453439+0000) 2022-01-31T23:01:51.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[51620]: debug 2022-01-31T23:01:51.453+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.453991+0000) 2022-01-31T23:01:51.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:51 smithi171 conmon[51620]: debug 2022-01-31T23:01:51.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.562918+0000) 2022-01-31T23:01:51.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:51 smithi167 conmon[49112]: debug 2022-01-31T23:01:51.454+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.455619+0000) 2022-01-31T23:01:51.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:51 smithi167 conmon[60316]: debug 2022-01-31T23:01:51.452+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.454389+0000) 2022-01-31T23:01:51.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:51 smithi167 conmon[54076]: debug 2022-01-31T23:01:51.455+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.456520+0000) 2022-01-31T23:01:52.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:52 smithi167 conmon[49112]: debug 2022-01-31T23:01:52.018+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:52.019245+0000) 2022-01-31T23:01:52.172 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:51 smithi167 conmon[60316]: debug 2022-01-31T23:01:51.948+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:51.948914+0000) 2022-01-31T23:01:52.463 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:52 smithi167 conmon[54076]: debug 2022-01-31T23:01:52.187+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:52.188891+0000) 2022-01-31T23:01:52.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:52 smithi171 conmon[41853]: debug 2022-01-31T23:01:52.387+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:52.388110+0000) 2022-01-31T23:01:52.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:52 smithi171 conmon[46715]: debug 2022-01-31T23:01:52.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:52.184011+0000) 2022-01-31T23:01:52.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:52 smithi171 conmon[51620]: debug 2022-01-31T23:01:52.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:52.563100+0000) 2022-01-31T23:01:53.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:53 smithi167 conmon[49112]: debug 2022-01-31T23:01:53.017+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:53.019434+0000) 2022-01-31T23:01:53.173 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:52 smithi167 conmon[60316]: debug 2022-01-31T23:01:52.947+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:52.949019+0000) 2022-01-31T23:01:53.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:53 smithi167 conmon[54076]: debug 2022-01-31T23:01:53.188+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:53.189072+0000) 2022-01-31T23:01:53.546 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:53 smithi171 conmon[46715]: debug 2022-01-31T23:01:53.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:53.184150+0000) 2022-01-31T23:01:53.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:53 smithi171 conmon[41853]: debug 2022-01-31T23:01:53.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:53.388256+0000) 2022-01-31T23:01:53.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:53 smithi171 conmon[51620]: debug 2022-01-31T23:01:53.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:53.563310+0000) 2022-01-31T23:01:54.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:54 smithi167 conmon[49112]: debug 2022-01-31T23:01:54.018+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:54.019618+0000) 2022-01-31T23:01:54.173 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:53 smithi167 conmon[60316]: debug 2022-01-31T23:01:53.947+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:53.949194+0000) 2022-01-31T23:01:54.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:53 smithi171 conmon[35325]: debug 2022-01-31T23:01:53.950+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:01:54.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:54 smithi171 conmon[46715]: debug 2022-01-31T23:01:54.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:54.184342+0000) 2022-01-31T23:01:54.470 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:54 smithi167 conmon[54076]: debug 2022-01-31T23:01:54.188+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:54.189286+0000) 2022-01-31T23:01:54.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:54 smithi171 conmon[41853]: debug 2022-01-31T23:01:54.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:54.388416+0000) 2022-01-31T23:01:54.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:54 smithi171 conmon[51620]: debug 2022-01-31T23:01:54.562+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:54.563496+0000) 2022-01-31T23:01:55.172 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:55 smithi167 conmon[49112]: debug 2022-01-31T23:01:55.019+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:55.019814+0000) 2022-01-31T23:01:55.173 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:54 smithi167 conmon[60316]: debug 2022-01-31T23:01:54.947+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:54.949412+0000) 2022-01-31T23:01:55.474 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:55 smithi167 conmon[54076]: debug 2022-01-31T23:01:55.188+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:55.189495+0000) 2022-01-31T23:01:55.546 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:55 smithi171 conmon[41853]: debug 2022-01-31T23:01:55.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:55.388597+0000) 2022-01-31T23:01:55.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:55 smithi171 conmon[46715]: debug 2022-01-31T23:01:55.183+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:55.184544+0000) 2022-01-31T23:01:55.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:55 smithi171 conmon[51620]: debug 2022-01-31T23:01:55.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:55.563649+0000) 2022-01-31T23:01:56.173 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:56 smithi167 conmon[49112]: debug 2022-01-31T23:01:56.019+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.020024+0000) 2022-01-31T23:01:56.173 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:55 smithi167 conmon[60316]: debug 2022-01-31T23:01:55.948+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:55.949521+0000) 2022-01-31T23:01:56.441 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[41853]: debug 2022-01-31T23:01:56.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.388772+0000) 2022-01-31T23:01:56.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[46715]: debug 2022-01-31T23:01:56.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.184765+0000) 2022-01-31T23:01:56.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:56 smithi167 conmon[54076]: debug 2022-01-31T23:01:56.189+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.189668+0000) 2022-01-31T23:01:56.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[46715]: debug 2022-01-31T23:01:56.481+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.482110+0000) 2022-01-31T23:01:56.842 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[35325]: debug 2022-01-31T23:01:56.508+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257100 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:01:56.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[41853]: debug 2022-01-31T23:01:56.482+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.483472+0000) 2022-01-31T23:01:56.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[51620]: debug 2022-01-31T23:01:56.481+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.482009+0000) 2022-01-31T23:01:56.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:56 smithi171 conmon[51620]: debug 2022-01-31T23:01:56.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.563806+0000) 2022-01-31T23:01:56.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:56 smithi167 conmon[49112]: debug 2022-01-31T23:01:56.482+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.483626+0000) 2022-01-31T23:01:56.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:56 smithi167 conmon[54076]: debug 2022-01-31T23:01:56.482+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.483573+0000) 2022-01-31T23:01:56.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:56 smithi167 conmon[60316]: debug 2022-01-31T23:01:56.481+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.483047+0000) 2022-01-31T23:01:57.173 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:57 smithi167 conmon[49112]: debug 2022-01-31T23:01:57.019+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:57.020180+0000) 2022-01-31T23:01:57.174 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:56 smithi167 conmon[60316]: debug 2022-01-31T23:01:56.948+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:56.949644+0000) 2022-01-31T23:01:57.481 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:57 smithi167 conmon[54076]: debug 2022-01-31T23:01:57.189+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:57.189853+0000) 2022-01-31T23:01:57.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:57 smithi171 conmon[41853]: debug 2022-01-31T23:01:57.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:57.388958+0000) 2022-01-31T23:01:57.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:57 smithi171 conmon[46715]: debug 2022-01-31T23:01:57.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:57.184931+0000) 2022-01-31T23:01:57.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:57 smithi171 conmon[51620]: debug 2022-01-31T23:01:57.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:57.563970+0000) 2022-01-31T23:01:58.173 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:58 smithi167 conmon[49112]: debug 2022-01-31T23:01:58.019+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:58.020308+0000) 2022-01-31T23:01:58.174 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:57 smithi167 conmon[60316]: debug 2022-01-31T23:01:57.949+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:57.949754+0000) 2022-01-31T23:01:58.485 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:58 smithi167 conmon[54076]: debug 2022-01-31T23:01:58.189+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:58.190010+0000) 2022-01-31T23:01:58.547 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:58 smithi171 conmon[41853]: debug 2022-01-31T23:01:58.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:58.389116+0000) 2022-01-31T23:01:58.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:58 smithi171 conmon[46715]: debug 2022-01-31T23:01:58.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:58.185050+0000) 2022-01-31T23:01:58.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:58 smithi171 conmon[51620]: debug 2022-01-31T23:01:58.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:58.564192+0000) 2022-01-31T23:01:59.173 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:01:59 smithi167 conmon[49112]: debug 2022-01-31T23:01:59.019+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:59.020469+0000) 2022-01-31T23:01:59.743 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:58 smithi167 conmon[60316]: debug 2022-01-31T23:01:58.949+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:58.949978+0000) 2022-01-31T23:01:59.744 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:01:59 smithi167 conmon[54076]: debug 2022-01-31T23:01:59.189+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:59.190113+0000) 2022-01-31T23:01:59.745 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:01:59 smithi171 conmon[41853]: debug 2022-01-31T23:01:59.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:59.389317+0000) 2022-01-31T23:01:59.745 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:01:59 smithi171 conmon[46715]: debug 2022-01-31T23:01:59.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:59.185239+0000) 2022-01-31T23:01:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:01:59 smithi171 conmon[51620]: debug 2022-01-31T23:01:59.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:59.564400+0000) 2022-01-31T23:02:00.173 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:00 smithi167 conmon[49112]: debug 2022-01-31T23:02:00.020+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:00.020674+0000) 2022-01-31T23:02:00.174 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:01:59 smithi167 conmon[60316]: debug 2022-01-31T23:01:59.949+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:01:59.950173+0000) 2022-01-31T23:02:00.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:00 smithi167 conmon[54076]: debug 2022-01-31T23:02:00.189+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:00.190235+0000) 2022-01-31T23:02:00.547 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:00 smithi171 conmon[46715]: debug 2022-01-31T23:02:00.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:00.185405+0000) 2022-01-31T23:02:00.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:00 smithi171 conmon[41853]: debug 2022-01-31T23:02:00.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:00.389509+0000) 2022-01-31T23:02:00.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:00 smithi171 conmon[51620]: debug 2022-01-31T23:02:00.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:00.564602+0000) 2022-01-31T23:02:01.173 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:01 smithi167 conmon[49112]: debug 2022-01-31T23:02:01.020+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.020799+0000) 2022-01-31T23:02:01.174 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:00 smithi167 conmon[60316]: debug 2022-01-31T23:02:00.949+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:00.950381+0000) 2022-01-31T23:02:01.494 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[46715]: debug 2022-01-31T23:02:01.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.185579+0000) 2022-01-31T23:02:01.495 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[41853]: debug 2022-01-31T23:02:01.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.389693+0000) 2022-01-31T23:02:01.496 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:01 smithi167 conmon[54076]: debug 2022-01-31T23:02:01.189+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.190457+0000) 2022-01-31T23:02:01.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[35325]: debug 2022-01-31T23:02:01.537+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257206 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:01.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[41853]: debug 2022-01-31T23:02:01.511+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.511914+0000) 2022-01-31T23:02:01.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[46715]: debug 2022-01-31T23:02:01.510+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.511493+0000) 2022-01-31T23:02:01.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[51620]: debug 2022-01-31T23:02:01.511+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.512190+0000) 2022-01-31T23:02:01.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:01 smithi171 conmon[51620]: debug 2022-01-31T23:02:01.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.564805+0000) 2022-01-31T23:02:01.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:01 smithi167 conmon[49112]: debug 2022-01-31T23:02:01.512+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.512561+0000) 2022-01-31T23:02:01.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:01 smithi167 conmon[54076]: debug 2022-01-31T23:02:01.512+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.513470+0000) 2022-01-31T23:02:01.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:01 smithi167 conmon[60316]: debug 2022-01-31T23:02:01.511+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.512266+0000) 2022-01-31T23:02:02.174 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:02 smithi167 conmon[49112]: debug 2022-01-31T23:02:02.020+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:02.020960+0000) 2022-01-31T23:02:02.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:01 smithi167 conmon[60316]: debug 2022-01-31T23:02:01.950+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:01.950574+0000) 2022-01-31T23:02:02.528 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:02 smithi167 conmon[54076]: debug 2022-01-31T23:02:02.190+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:02.190628+0000) 2022-01-31T23:02:02.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:02 smithi171 conmon[41853]: debug 2022-01-31T23:02:02.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:02.389873+0000) 2022-01-31T23:02:02.548 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:02 smithi171 conmon[46715]: debug 2022-01-31T23:02:02.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:02.185792+0000) 2022-01-31T23:02:02.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:02 smithi171 conmon[51620]: debug 2022-01-31T23:02:02.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:02.564982+0000) 2022-01-31T23:02:03.174 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:03 smithi167 conmon[49112]: debug 2022-01-31T23:02:03.020+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:03.021128+0000) 2022-01-31T23:02:03.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:02 smithi167 conmon[60316]: debug 2022-01-31T23:02:02.950+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:02.950670+0000) 2022-01-31T23:02:03.528 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:03 smithi167 conmon[54076]: debug 2022-01-31T23:02:03.190+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:03.190745+0000) 2022-01-31T23:02:03.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:03 smithi171 conmon[41853]: debug 2022-01-31T23:02:03.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:03.390001+0000) 2022-01-31T23:02:03.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:03 smithi171 conmon[46715]: debug 2022-01-31T23:02:03.184+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:03.185906+0000) 2022-01-31T23:02:03.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:03 smithi171 conmon[51620]: debug 2022-01-31T23:02:03.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:03.565202+0000) 2022-01-31T23:02:04.174 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:04 smithi167 conmon[49112]: debug 2022-01-31T23:02:04.020+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:04.021329+0000) 2022-01-31T23:02:04.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:03 smithi167 conmon[60316]: debug 2022-01-31T23:02:03.950+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:03.950815+0000) 2022-01-31T23:02:04.528 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:04 smithi167 conmon[54076]: debug 2022-01-31T23:02:04.190+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:04.190965+0000) 2022-01-31T23:02:04.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:04 smithi171 conmon[41853]: debug 2022-01-31T23:02:04.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:04.390174+0000) 2022-01-31T23:02:04.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:04 smithi171 conmon[46715]: debug 2022-01-31T23:02:04.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:04.186088+0000) 2022-01-31T23:02:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:04 smithi171 conmon[51620]: debug 2022-01-31T23:02:04.563+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:04.565390+0000) 2022-01-31T23:02:05.174 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:05 smithi167 conmon[49112]: debug 2022-01-31T23:02:05.021+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:05.021558+0000) 2022-01-31T23:02:05.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:04 smithi167 conmon[60316]: debug 2022-01-31T23:02:04.950+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:04.951036+0000) 2022-01-31T23:02:05.529 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:05 smithi167 conmon[54076]: debug 2022-01-31T23:02:05.190+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:05.191124+0000) 2022-01-31T23:02:05.548 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:05 smithi171 conmon[41853]: debug 2022-01-31T23:02:05.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:05.390391+0000) 2022-01-31T23:02:05.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:05 smithi171 conmon[46715]: debug 2022-01-31T23:02:05.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:05.186269+0000) 2022-01-31T23:02:05.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:05 smithi171 conmon[51620]: debug 2022-01-31T23:02:05.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:05.565602+0000) 2022-01-31T23:02:06.174 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:06 smithi167 conmon[49112]: debug 2022-01-31T23:02:06.021+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.021732+0000) 2022-01-31T23:02:06.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:05 smithi167 conmon[60316]: debug 2022-01-31T23:02:05.950+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:05.951175+0000) 2022-01-31T23:02:06.523 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[41853]: debug 2022-01-31T23:02:06.388+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.390610+0000) 2022-01-31T23:02:06.523 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[46715]: debug 2022-01-31T23:02:06.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.186446+0000) 2022-01-31T23:02:06.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:06 smithi167 conmon[54076]: debug 2022-01-31T23:02:06.190+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.191281+0000) 2022-01-31T23:02:06.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[35325]: debug 2022-01-31T23:02:06.565+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257319 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:06.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[46715]: debug 2022-01-31T23:02:06.538+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.539852+0000) 2022-01-31T23:02:06.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[41853]: debug 2022-01-31T23:02:06.539+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.541098+0000) 2022-01-31T23:02:06.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[51620]: debug 2022-01-31T23:02:06.540+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.541898+0000) 2022-01-31T23:02:06.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:06 smithi171 conmon[51620]: debug 2022-01-31T23:02:06.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.565818+0000) 2022-01-31T23:02:06.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:06 smithi167 conmon[49112]: debug 2022-01-31T23:02:06.541+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.541925+0000) 2022-01-31T23:02:06.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:06 smithi167 conmon[54076]: debug 2022-01-31T23:02:06.541+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.541692+0000) 2022-01-31T23:02:06.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:06 smithi167 conmon[60316]: debug 2022-01-31T23:02:06.540+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.541102+0000) 2022-01-31T23:02:07.174 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:07 smithi167 conmon[49112]: debug 2022-01-31T23:02:07.021+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:07.021914+0000) 2022-01-31T23:02:07.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:06 smithi167 conmon[60316]: debug 2022-01-31T23:02:06.950+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:06.951362+0000) 2022-01-31T23:02:07.529 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:07 smithi167 conmon[54076]: debug 2022-01-31T23:02:07.190+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:07.191458+0000) 2022-01-31T23:02:07.549 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:07 smithi171 conmon[41853]: debug 2022-01-31T23:02:07.389+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:07.390788+0000) 2022-01-31T23:02:07.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:07 smithi171 conmon[46715]: debug 2022-01-31T23:02:07.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:07.186624+0000) 2022-01-31T23:02:07.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:07 smithi171 conmon[51620]: debug 2022-01-31T23:02:07.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:07.565919+0000) 2022-01-31T23:02:08.175 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:08 smithi167 conmon[49112]: debug 2022-01-31T23:02:08.021+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:08.022078+0000) 2022-01-31T23:02:08.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:07 smithi167 conmon[60316]: debug 2022-01-31T23:02:07.951+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:07.951518+0000) 2022-01-31T23:02:08.529 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:08 smithi167 conmon[54076]: debug 2022-01-31T23:02:08.191+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:08.191616+0000) 2022-01-31T23:02:08.549 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:08 smithi171 conmon[41853]: debug 2022-01-31T23:02:08.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:08.390975+0000) 2022-01-31T23:02:08.549 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:08 smithi171 conmon[46715]: debug 2022-01-31T23:02:08.185+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:08.186784+0000) 2022-01-31T23:02:08.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:08 smithi171 conmon[51620]: debug 2022-01-31T23:02:08.564+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:08.566036+0000) 2022-01-31T23:02:09.175 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:08 smithi167 conmon[60316]: debug 2022-01-31T23:02:08.951+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:08.951623+0000) 2022-01-31T23:02:09.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:09 smithi167 conmon[49112]: debug 2022-01-31T23:02:09.021+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:09.022261+0000) 2022-01-31T23:02:09.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:08 smithi171 conmon[35325]: debug 2022-01-31T23:02:08.950+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:02:09.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:09 smithi171 conmon[46715]: debug 2022-01-31T23:02:09.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:09.186939+0000) 2022-01-31T23:02:09.529 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:09 smithi167 conmon[54076]: debug 2022-01-31T23:02:09.191+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:09.191807+0000) 2022-01-31T23:02:09.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:09 smithi171 conmon[41853]: debug 2022-01-31T23:02:09.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:09.391116+0000) 2022-01-31T23:02:09.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:09 smithi171 conmon[51620]: debug 2022-01-31T23:02:09.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:09.566214+0000) 2022-01-31T23:02:10.175 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:10 smithi167 conmon[49112]: debug 2022-01-31T23:02:10.021+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:10.022483+0000) 2022-01-31T23:02:10.176 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:09 smithi167 conmon[60316]: debug 2022-01-31T23:02:09.951+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:09.951792+0000) 2022-01-31T23:02:10.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:10 smithi167 conmon[54076]: debug 2022-01-31T23:02:10.191+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:10.191992+0000) 2022-01-31T23:02:10.549 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:10 smithi171 conmon[41853]: debug 2022-01-31T23:02:10.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:10.391299+0000) 2022-01-31T23:02:10.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:10 smithi171 conmon[46715]: debug 2022-01-31T23:02:10.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:10.187091+0000) 2022-01-31T23:02:10.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:10 smithi171 conmon[51620]: debug 2022-01-31T23:02:10.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:10.566398+0000) 2022-01-31T23:02:11.175 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:11 smithi167 conmon[49112]: debug 2022-01-31T23:02:11.022+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.022578+0000) 2022-01-31T23:02:11.176 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:10 smithi167 conmon[60316]: debug 2022-01-31T23:02:10.951+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:10.951977+0000) 2022-01-31T23:02:11.505 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[41853]: debug 2022-01-31T23:02:11.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.391524+0000) 2022-01-31T23:02:11.505 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[46715]: debug 2022-01-31T23:02:11.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.187243+0000) 2022-01-31T23:02:11.529 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:11 smithi167 conmon[54076]: debug 2022-01-31T23:02:11.191+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.192185+0000) 2022-01-31T23:02:11.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[35325]: debug 2022-01-31T23:02:11.594+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257429 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:11.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[41853]: debug 2022-01-31T23:02:11.567+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.569640+0000) 2022-01-31T23:02:11.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[46715]: debug 2022-01-31T23:02:11.567+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.569556+0000) 2022-01-31T23:02:11.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[51620]: debug 2022-01-31T23:02:11.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.566584+0000) 2022-01-31T23:02:11.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:11 smithi171 conmon[51620]: debug 2022-01-31T23:02:11.568+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.570264+0000) 2022-01-31T23:02:11.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:11 smithi167 conmon[49112]: debug 2022-01-31T23:02:11.569+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.570264+0000) 2022-01-31T23:02:11.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:11 smithi167 conmon[54076]: debug 2022-01-31T23:02:11.570+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.570593+0000) 2022-01-31T23:02:11.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:11 smithi167 conmon[60316]: debug 2022-01-31T23:02:11.568+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.569208+0000) 2022-01-31T23:02:12.175 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:12 smithi167 conmon[49112]: debug 2022-01-31T23:02:12.022+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:12.022735+0000) 2022-01-31T23:02:12.176 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:11 smithi167 conmon[60316]: debug 2022-01-31T23:02:11.951+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:11.952184+0000) 2022-01-31T23:02:12.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:12 smithi167 conmon[54076]: debug 2022-01-31T23:02:12.191+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:12.192347+0000) 2022-01-31T23:02:12.549 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:12 smithi171 conmon[41853]: debug 2022-01-31T23:02:12.390+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:12.391681+0000) 2022-01-31T23:02:12.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:12 smithi171 conmon[46715]: debug 2022-01-31T23:02:12.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:12.187426+0000) 2022-01-31T23:02:12.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:12 smithi171 conmon[51620]: debug 2022-01-31T23:02:12.565+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:12.566769+0000) 2022-01-31T23:02:13.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:13 smithi167 conmon[49112]: debug 2022-01-31T23:02:13.022+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:13.022935+0000) 2022-01-31T23:02:13.176 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:12 smithi167 conmon[60316]: debug 2022-01-31T23:02:12.951+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:12.952358+0000) 2022-01-31T23:02:13.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:13 smithi167 conmon[54076]: debug 2022-01-31T23:02:13.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:13.192525+0000) 2022-01-31T23:02:13.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:13 smithi171 conmon[46715]: debug 2022-01-31T23:02:13.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:13.187548+0000) 2022-01-31T23:02:13.550 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:13 smithi171 conmon[41853]: debug 2022-01-31T23:02:13.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:13.391858+0000) 2022-01-31T23:02:13.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:13 smithi171 conmon[51620]: debug 2022-01-31T23:02:13.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:13.566905+0000) 2022-01-31T23:02:14.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:14 smithi167 conmon[49112]: debug 2022-01-31T23:02:14.022+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:14.023124+0000) 2022-01-31T23:02:14.176 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:13 smithi167 conmon[60316]: debug 2022-01-31T23:02:13.952+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:13.952558+0000) 2022-01-31T23:02:14.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:14 smithi167 conmon[54076]: debug 2022-01-31T23:02:14.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:14.192673+0000) 2022-01-31T23:02:14.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:14 smithi171 conmon[46715]: debug 2022-01-31T23:02:14.186+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:14.187712+0000) 2022-01-31T23:02:14.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:14 smithi171 conmon[41853]: debug 2022-01-31T23:02:14.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:14.392042+0000) 2022-01-31T23:02:14.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:14 smithi171 conmon[51620]: debug 2022-01-31T23:02:14.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:14.567090+0000) 2022-01-31T23:02:15.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:15 smithi167 conmon[49112]: debug 2022-01-31T23:02:15.022+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:15.023346+0000) 2022-01-31T23:02:15.176 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:14 smithi167 conmon[60316]: debug 2022-01-31T23:02:14.952+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:14.952759+0000) 2022-01-31T23:02:15.530 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:15 smithi167 conmon[54076]: debug 2022-01-31T23:02:15.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:15.192875+0000) 2022-01-31T23:02:15.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:15 smithi171 conmon[46715]: debug 2022-01-31T23:02:15.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:15.187866+0000) 2022-01-31T23:02:15.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:15 smithi171 conmon[41853]: debug 2022-01-31T23:02:15.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:15.392248+0000) 2022-01-31T23:02:15.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:15 smithi171 conmon[51620]: debug 2022-01-31T23:02:15.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:15.567263+0000) 2022-01-31T23:02:16.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:16 smithi167 conmon[49112]: debug 2022-01-31T23:02:16.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.023551+0000) 2022-01-31T23:02:16.177 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:15 smithi167 conmon[60316]: debug 2022-01-31T23:02:15.952+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:15.952965+0000) 2022-01-31T23:02:16.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:16 smithi167 conmon[54076]: debug 2022-01-31T23:02:16.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.193034+0000) 2022-01-31T23:02:16.550 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[46715]: debug 2022-01-31T23:02:16.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.188054+0000) 2022-01-31T23:02:16.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[41853]: debug 2022-01-31T23:02:16.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.392429+0000) 2022-01-31T23:02:16.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[35325]: debug 2022-01-31T23:02:16.623+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257542 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:16.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[41853]: debug 2022-01-31T23:02:16.598+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.599365+0000) 2022-01-31T23:02:16.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[46715]: debug 2022-01-31T23:02:16.598+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.598952+0000) 2022-01-31T23:02:16.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[51620]: debug 2022-01-31T23:02:16.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.567438+0000) 2022-01-31T23:02:16.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:16 smithi171 conmon[51620]: debug 2022-01-31T23:02:16.598+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.599570+0000) 2022-01-31T23:02:16.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:16 smithi167 conmon[54076]: debug 2022-01-31T23:02:16.598+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.599362+0000) 2022-01-31T23:02:16.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:16 smithi167 conmon[49112]: debug 2022-01-31T23:02:16.598+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.598839+0000) 2022-01-31T23:02:16.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:16 smithi167 conmon[60316]: debug 2022-01-31T23:02:16.598+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.598715+0000) 2022-01-31T23:02:17.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:17 smithi167 conmon[49112]: debug 2022-01-31T23:02:17.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:17.023691+0000) 2022-01-31T23:02:17.177 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:16 smithi167 conmon[60316]: debug 2022-01-31T23:02:16.952+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:16.953167+0000) 2022-01-31T23:02:17.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:17 smithi167 conmon[54076]: debug 2022-01-31T23:02:17.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:17.193181+0000) 2022-01-31T23:02:17.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:17 smithi171 conmon[41853]: debug 2022-01-31T23:02:17.391+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:17.392586+0000) 2022-01-31T23:02:17.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:17 smithi171 conmon[46715]: debug 2022-01-31T23:02:17.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:17.188236+0000) 2022-01-31T23:02:17.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:17 smithi171 conmon[51620]: debug 2022-01-31T23:02:17.566+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:17.567626+0000) 2022-01-31T23:02:17.982 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:17 smithi167 conmon[60316]: debug 2022-01-31T23:02:17.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:17.953333+0000) 2022-01-31T23:02:18.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:18 smithi167 conmon[49112]: debug 2022-01-31T23:02:18.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:18.023843+0000) 2022-01-31T23:02:18.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:18 smithi167 conmon[54076]: debug 2022-01-31T23:02:18.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:18.193367+0000) 2022-01-31T23:02:18.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:18 smithi171 conmon[46715]: debug 2022-01-31T23:02:18.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:18.188410+0000) 2022-01-31T23:02:18.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:18 smithi171 conmon[41853]: debug 2022-01-31T23:02:18.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:18.392758+0000) 2022-01-31T23:02:18.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:18 smithi171 conmon[51620]: debug 2022-01-31T23:02:18.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:18.567794+0000) 2022-01-31T23:02:19.176 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:19 smithi167 conmon[49112]: debug 2022-01-31T23:02:19.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:19.024025+0000) 2022-01-31T23:02:19.177 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:18 smithi167 conmon[60316]: debug 2022-01-31T23:02:18.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:18.953555+0000) 2022-01-31T23:02:19.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:19 smithi167 conmon[54076]: debug 2022-01-31T23:02:19.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:19.193555+0000) 2022-01-31T23:02:19.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:19 smithi171 conmon[46715]: debug 2022-01-31T23:02:19.187+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:19.188633+0000) 2022-01-31T23:02:19.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:19 smithi171 conmon[41853]: debug 2022-01-31T23:02:19.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:19.392957+0000) 2022-01-31T23:02:19.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:19 smithi171 conmon[51620]: debug 2022-01-31T23:02:19.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:19.568005+0000) 2022-01-31T23:02:20.181 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:20 smithi167 conmon[49112]: debug 2022-01-31T23:02:20.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:20.024201+0000) 2022-01-31T23:02:20.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:19 smithi167 conmon[60316]: debug 2022-01-31T23:02:19.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:19.953689+0000) 2022-01-31T23:02:20.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:20 smithi167 conmon[54076]: debug 2022-01-31T23:02:20.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:20.193674+0000) 2022-01-31T23:02:20.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:20 smithi171 conmon[46715]: debug 2022-01-31T23:02:20.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:20.188836+0000) 2022-01-31T23:02:20.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:20 smithi171 conmon[41853]: debug 2022-01-31T23:02:20.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:20.393094+0000) 2022-01-31T23:02:20.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:20 smithi171 conmon[51620]: debug 2022-01-31T23:02:20.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:20.568144+0000) 2022-01-31T23:02:21.177 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[49112]: debug 2022-01-31T23:02:21.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.024386+0000) 2022-01-31T23:02:21.178 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:20 smithi167 conmon[60316]: debug 2022-01-31T23:02:20.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:20.953873+0000) 2022-01-31T23:02:21.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[54076]: debug 2022-01-31T23:02:21.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.193893+0000) 2022-01-31T23:02:21.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[46715]: debug 2022-01-31T23:02:21.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.189020+0000) 2022-01-31T23:02:21.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[41853]: debug 2022-01-31T23:02:21.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.393298+0000) 2022-01-31T23:02:21.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[35325]: debug 2022-01-31T23:02:21.652+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257653 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:21.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[41853]: debug 2022-01-31T23:02:21.626+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.627584+0000) 2022-01-31T23:02:21.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[46715]: debug 2022-01-31T23:02:21.625+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.626714+0000) 2022-01-31T23:02:21.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[51620]: debug 2022-01-31T23:02:21.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.568287+0000) 2022-01-31T23:02:21.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:21 smithi171 conmon[51620]: debug 2022-01-31T23:02:21.627+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.628437+0000) 2022-01-31T23:02:21.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[49112]: debug 2022-01-31T23:02:21.626+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.626751+0000) 2022-01-31T23:02:21.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[54076]: debug 2022-01-31T23:02:21.627+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.627957+0000) 2022-01-31T23:02:21.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[60316]: debug 2022-01-31T23:02:21.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[60316]: 2022-01-31T23:02:21.626+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.627075+0000) 2022-01-31T23:02:22.177 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:21 smithi167 conmon[60316]: debug 2022-01-31T23:02:21.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:21.954030+0000) 2022-01-31T23:02:22.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:22 smithi167 conmon[49112]: debug 2022-01-31T23:02:22.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:22.024592+0000) 2022-01-31T23:02:22.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:22 smithi167 conmon[54076]: debug 2022-01-31T23:02:22.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:22.194069+0000) 2022-01-31T23:02:22.551 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:22 smithi171 conmon[41853]: debug 2022-01-31T23:02:22.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:22.393519+0000) 2022-01-31T23:02:22.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:22 smithi171 conmon[46715]: debug 2022-01-31T23:02:22.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:22.189219+0000) 2022-01-31T23:02:22.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:22 smithi171 conmon[51620]: debug 2022-01-31T23:02:22.567+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:22.568480+0000) 2022-01-31T23:02:23.177 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:23 smithi167 conmon[49112]: debug 2022-01-31T23:02:23.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:23.024718+0000) 2022-01-31T23:02:23.178 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:22 smithi167 conmon[60316]: debug 2022-01-31T23:02:22.954+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:22.954183+0000) 2022-01-31T23:02:23.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:23 smithi167 conmon[54076]: debug 2022-01-31T23:02:23.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:23.194216+0000) 2022-01-31T23:02:23.551 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:23 smithi171 conmon[46715]: debug 2022-01-31T23:02:23.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:23.189372+0000) 2022-01-31T23:02:23.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:23 smithi171 conmon[41853]: debug 2022-01-31T23:02:23.392+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:23.393680+0000) 2022-01-31T23:02:23.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:23 smithi171 conmon[51620]: debug 2022-01-31T23:02:23.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:23.568644+0000) 2022-01-31T23:02:24.177 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:24 smithi167 conmon[49112]: debug 2022-01-31T23:02:24.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:24.024839+0000) 2022-01-31T23:02:24.178 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:23 smithi167 conmon[60316]: debug 2022-01-31T23:02:23.952+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:23.954370+0000) 2022-01-31T23:02:24.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:23 smithi171 conmon[35325]: debug 2022-01-31T23:02:23.951+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:02:24.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:24 smithi171 conmon[46715]: debug 2022-01-31T23:02:24.188+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:24.189580+0000) 2022-01-31T23:02:24.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:24 smithi167 conmon[54076]: debug 2022-01-31T23:02:24.192+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:24.194365+0000) 2022-01-31T23:02:24.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:24 smithi171 conmon[41853]: debug 2022-01-31T23:02:24.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:24.393877+0000) 2022-01-31T23:02:24.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:24 smithi171 conmon[51620]: debug 2022-01-31T23:02:24.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:24.568835+0000) 2022-01-31T23:02:25.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:25 smithi167 conmon[49112]: debug 2022-01-31T23:02:25.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:25.024998+0000) 2022-01-31T23:02:25.178 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:24 smithi167 conmon[60316]: debug 2022-01-31T23:02:24.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:24.954520+0000) 2022-01-31T23:02:25.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:25 smithi167 conmon[54076]: debug 2022-01-31T23:02:25.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:25.194533+0000) 2022-01-31T23:02:25.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:25 smithi171 conmon[46715]: debug 2022-01-31T23:02:25.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:25.189794+0000) 2022-01-31T23:02:25.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:25 smithi171 conmon[41853]: debug 2022-01-31T23:02:25.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:25.394034+0000) 2022-01-31T23:02:25.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:25 smithi171 conmon[51620]: debug 2022-01-31T23:02:25.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:25.568963+0000) 2022-01-31T23:02:26.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:26 smithi167 conmon[49112]: debug 2022-01-31T23:02:26.023+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.025196+0000) 2022-01-31T23:02:26.178 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:25 smithi167 conmon[60316]: debug 2022-01-31T23:02:25.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:25.954645+0000) 2022-01-31T23:02:26.532 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:26 smithi167 conmon[54076]: debug 2022-01-31T23:02:26.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.194737+0000) 2022-01-31T23:02:26.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[46715]: debug 2022-01-31T23:02:26.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.189994+0000) 2022-01-31T23:02:26.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[41853]: debug 2022-01-31T23:02:26.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.394188+0000) 2022-01-31T23:02:26.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[35325]: debug 2022-01-31T23:02:26.696+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257767 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:26.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[41853]: debug 2022-01-31T23:02:26.655+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.656365+0000) 2022-01-31T23:02:26.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[46715]: debug 2022-01-31T23:02:26.655+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.655762+0000) 2022-01-31T23:02:26.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[51620]: debug 2022-01-31T23:02:26.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.569113+0000) 2022-01-31T23:02:26.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:26 smithi171 conmon[51620]: debug 2022-01-31T23:02:26.655+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.656298+0000) 2022-01-31T23:02:26.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:26 smithi167 conmon[49112]: debug 2022-01-31T23:02:26.653+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.654971+0000) 2022-01-31T23:02:26.904 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:26 smithi167 conmon[54076]: debug 2022-01-31T23:02:26.653+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.654814+0000) 2022-01-31T23:02:26.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:26 smithi167 conmon[60316]: debug 2022-01-31T23:02:26.654+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.655833+0000) 2022-01-31T23:02:27.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:27 smithi167 conmon[49112]: debug 2022-01-31T23:02:27.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:27.025344+0000) 2022-01-31T23:02:27.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:26 smithi167 conmon[60316]: debug 2022-01-31T23:02:26.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:26.954834+0000) 2022-01-31T23:02:27.533 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:27 smithi167 conmon[54076]: debug 2022-01-31T23:02:27.193+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:27.194923+0000) 2022-01-31T23:02:27.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:27 smithi171 conmon[41853]: debug 2022-01-31T23:02:27.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:27.394376+0000) 2022-01-31T23:02:27.552 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:27 smithi171 conmon[46715]: debug 2022-01-31T23:02:27.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:27.190186+0000) 2022-01-31T23:02:27.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:27 smithi171 conmon[51620]: debug 2022-01-31T23:02:27.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:27.569303+0000) 2022-01-31T23:02:28.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:28 smithi167 conmon[49112]: debug 2022-01-31T23:02:28.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:28.025473+0000) 2022-01-31T23:02:28.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:27 smithi167 conmon[60316]: debug 2022-01-31T23:02:27.954+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:27.954980+0000) 2022-01-31T23:02:28.533 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:28 smithi167 conmon[54076]: debug 2022-01-31T23:02:28.194+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:28.195101+0000) 2022-01-31T23:02:28.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:28 smithi171 conmon[41853]: debug 2022-01-31T23:02:28.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:28.394560+0000) 2022-01-31T23:02:28.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:28 smithi171 conmon[46715]: debug 2022-01-31T23:02:28.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:28.190316+0000) 2022-01-31T23:02:28.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:28 smithi171 conmon[51620]: debug 2022-01-31T23:02:28.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:28.569473+0000) 2022-01-31T23:02:29.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:29 smithi167 conmon[49112]: debug 2022-01-31T23:02:29.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:29.025616+0000) 2022-01-31T23:02:29.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:28 smithi167 conmon[60316]: debug 2022-01-31T23:02:28.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:28.955141+0000) 2022-01-31T23:02:29.533 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:29 smithi167 conmon[54076]: debug 2022-01-31T23:02:29.194+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:29.195282+0000) 2022-01-31T23:02:29.552 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:29 smithi171 conmon[41853]: debug 2022-01-31T23:02:29.393+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:29.394716+0000) 2022-01-31T23:02:29.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:29 smithi171 conmon[46715]: debug 2022-01-31T23:02:29.189+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:29.190466+0000) 2022-01-31T23:02:29.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:29 smithi171 conmon[51620]: debug 2022-01-31T23:02:29.568+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:29.569674+0000) 2022-01-31T23:02:30.178 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:30 smithi167 conmon[49112]: debug 2022-01-31T23:02:30.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:30.025779+0000) 2022-01-31T23:02:30.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:29 smithi167 conmon[60316]: debug 2022-01-31T23:02:29.953+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:29.955347+0000) 2022-01-31T23:02:30.533 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:30 smithi167 conmon[54076]: debug 2022-01-31T23:02:30.194+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:30.195437+0000) 2022-01-31T23:02:30.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:30 smithi171 conmon[46715]: debug 2022-01-31T23:02:30.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:30.190641+0000) 2022-01-31T23:02:30.553 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:30 smithi171 conmon[41853]: debug 2022-01-31T23:02:30.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:30.394875+0000) 2022-01-31T23:02:30.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:30 smithi171 conmon[51620]: debug 2022-01-31T23:02:30.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:30.569859+0000) 2022-01-31T23:02:31.179 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:31 smithi167 conmon[49112]: debug 2022-01-31T23:02:31.025+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.025965+0000) 2022-01-31T23:02:31.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:30 smithi167 conmon[60316]: debug 2022-01-31T23:02:30.954+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:30.955530+0000) 2022-01-31T23:02:31.533 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:31 smithi167 conmon[54076]: debug 2022-01-31T23:02:31.194+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.195589+0000) 2022-01-31T23:02:31.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[46715]: debug 2022-01-31T23:02:31.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.190789+0000) 2022-01-31T23:02:31.553 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[41853]: debug 2022-01-31T23:02:31.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.395053+0000) 2022-01-31T23:02:31.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[35325]: debug 2022-01-31T23:02:31.724+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257879 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:31.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[41853]: debug 2022-01-31T23:02:31.698+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.699402+0000) 2022-01-31T23:02:31.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[46715]: debug 2022-01-31T23:02:31.699+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.699829+0000) 2022-01-31T23:02:31.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[51620]: debug 2022-01-31T23:02:31.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.570012+0000) 2022-01-31T23:02:31.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:31 smithi171 conmon[51620]: debug 2022-01-31T23:02:31.699+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.700599+0000) 2022-01-31T23:02:31.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:31 smithi167 conmon[49112]: debug 2022-01-31T23:02:31.698+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.700029+0000) 2022-01-31T23:02:31.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:31 smithi167 conmon[54076]: debug 2022-01-31T23:02:31.698+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.700265+0000) 2022-01-31T23:02:31.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:31 smithi167 conmon[60316]: debug 2022-01-31T23:02:31.704+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.706192+0000) 2022-01-31T23:02:32.179 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:32 smithi167 conmon[49112]: debug 2022-01-31T23:02:32.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:32.026146+0000) 2022-01-31T23:02:32.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:31 smithi167 conmon[60316]: debug 2022-01-31T23:02:31.954+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:31.955693+0000) 2022-01-31T23:02:32.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:32 smithi167 conmon[54076]: debug 2022-01-31T23:02:32.194+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:32.195689+0000) 2022-01-31T23:02:32.553 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:32 smithi171 conmon[41853]: debug 2022-01-31T23:02:32.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:32.395261+0000) 2022-01-31T23:02:32.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:32 smithi171 conmon[46715]: debug 2022-01-31T23:02:32.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:32.191006+0000) 2022-01-31T23:02:32.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:32 smithi171 conmon[51620]: debug 2022-01-31T23:02:32.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:32.570167+0000) 2022-01-31T23:02:33.179 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:33 smithi167 conmon[49112]: debug 2022-01-31T23:02:33.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:33.026295+0000) 2022-01-31T23:02:33.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:32 smithi167 conmon[60316]: debug 2022-01-31T23:02:32.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:32.955861+0000) 2022-01-31T23:02:33.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:33 smithi167 conmon[54076]: debug 2022-01-31T23:02:33.194+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:33.195812+0000) 2022-01-31T23:02:33.553 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:33 smithi171 conmon[41853]: debug 2022-01-31T23:02:33.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:33.395418+0000) 2022-01-31T23:02:33.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:33 smithi171 conmon[46715]: debug 2022-01-31T23:02:33.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:33.191164+0000) 2022-01-31T23:02:33.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:33 smithi171 conmon[51620]: debug 2022-01-31T23:02:33.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:33.570330+0000) 2022-01-31T23:02:34.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:33 smithi167 conmon[60316]: debug 2022-01-31T23:02:33.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:33.956019+0000) 2022-01-31T23:02:34.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:34 smithi167 conmon[49112]: debug 2022-01-31T23:02:34.024+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:34.026474+0000) 2022-01-31T23:02:34.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:34 smithi167 conmon[54076]: debug 2022-01-31T23:02:34.195+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:34.196000+0000) 2022-01-31T23:02:34.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:34 smithi171 conmon[46715]: debug 2022-01-31T23:02:34.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:34.191351+0000) 2022-01-31T23:02:34.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:34 smithi171 conmon[41853]: debug 2022-01-31T23:02:34.394+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:34.395597+0000) 2022-01-31T23:02:34.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:34 smithi171 conmon[51620]: debug 2022-01-31T23:02:34.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:34.570510+0000) 2022-01-31T23:02:35.179 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:35 smithi167 conmon[49112]: debug 2022-01-31T23:02:35.025+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:35.026698+0000) 2022-01-31T23:02:35.180 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:34 smithi167 conmon[60316]: debug 2022-01-31T23:02:34.954+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:34.956199+0000) 2022-01-31T23:02:35.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:35 smithi167 conmon[54076]: debug 2022-01-31T23:02:35.195+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:35.196193+0000) 2022-01-31T23:02:35.553 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:35 smithi171 conmon[46715]: debug 2022-01-31T23:02:35.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:35.191529+0000) 2022-01-31T23:02:35.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:35 smithi171 conmon[41853]: debug 2022-01-31T23:02:35.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:35.395793+0000) 2022-01-31T23:02:35.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:35 smithi171 conmon[51620]: debug 2022-01-31T23:02:35.569+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:35.570653+0000) 2022-01-31T23:02:36.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:35 smithi167 conmon[60316]: debug 2022-01-31T23:02:35.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:35.956377+0000) 2022-01-31T23:02:36.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:36 smithi167 conmon[49112]: debug 2022-01-31T23:02:36.025+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.026849+0000) 2022-01-31T23:02:36.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:36 smithi167 conmon[54076]: debug 2022-01-31T23:02:36.195+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.196396+0000) 2022-01-31T23:02:36.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[46715]: debug 2022-01-31T23:02:36.190+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.191697+0000) 2022-01-31T23:02:36.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[41853]: debug 2022-01-31T23:02:36.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.395963+0000) 2022-01-31T23:02:36.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[35325]: debug 2022-01-31T23:02:36.753+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 257992 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:36.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[46715]: debug 2022-01-31T23:02:36.726+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.727492+0000) 2022-01-31T23:02:36.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[41853]: debug 2022-01-31T23:02:36.727+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.727774+0000) 2022-01-31T23:02:36.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[51620]: debug 2022-01-31T23:02:36.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.570797+0000) 2022-01-31T23:02:36.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:36 smithi171 conmon[51620]: debug 2022-01-31T23:02:36.728+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.729628+0000) 2022-01-31T23:02:36.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:36 smithi167 conmon[49112]: debug 2022-01-31T23:02:36.726+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.728368+0000) 2022-01-31T23:02:36.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:36 smithi167 conmon[54076]: debug 2022-01-31T23:02:36.726+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.728474+0000) 2022-01-31T23:02:36.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:36 smithi167 conmon[60316]: debug 2022-01-31T23:02:36.727+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.729338+0000) 2022-01-31T23:02:37.179 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:36 smithi167 conmon[60316]: debug 2022-01-31T23:02:36.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:36.956584+0000) 2022-01-31T23:02:37.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:37 smithi167 conmon[49112]: debug 2022-01-31T23:02:37.026+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:37.027032+0000) 2022-01-31T23:02:37.534 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:37 smithi167 conmon[54076]: debug 2022-01-31T23:02:37.195+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:37.196577+0000) 2022-01-31T23:02:37.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:37 smithi171 conmon[41853]: debug 2022-01-31T23:02:37.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:37.396084+0000) 2022-01-31T23:02:37.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:37 smithi171 conmon[46715]: debug 2022-01-31T23:02:37.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:37.191921+0000) 2022-01-31T23:02:37.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:37 smithi171 conmon[51620]: debug 2022-01-31T23:02:37.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:37.570925+0000) 2022-01-31T23:02:38.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:38 smithi167 conmon[49112]: debug 2022-01-31T23:02:38.025+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:38.027191+0000) 2022-01-31T23:02:38.180 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:37 smithi167 conmon[60316]: debug 2022-01-31T23:02:37.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:37.956731+0000) 2022-01-31T23:02:38.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:38 smithi167 conmon[54076]: debug 2022-01-31T23:02:38.195+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:38.196721+0000) 2022-01-31T23:02:38.554 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:38 smithi171 conmon[46715]: debug 2022-01-31T23:02:38.191+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:38.192075+0000) 2022-01-31T23:02:38.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:38 smithi171 conmon[41853]: debug 2022-01-31T23:02:38.395+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:38.396261+0000) 2022-01-31T23:02:38.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:38 smithi171 conmon[51620]: debug 2022-01-31T23:02:38.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:38.571027+0000) 2022-01-31T23:02:39.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:39 smithi167 conmon[49112]: debug 2022-01-31T23:02:39.025+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:39.027360+0000) 2022-01-31T23:02:39.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:38 smithi167 conmon[60316]: debug 2022-01-31T23:02:38.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:38.956883+0000) 2022-01-31T23:02:39.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:38 smithi171 conmon[35325]: debug 2022-01-31T23:02:38.952+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:02:39.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:39 smithi171 conmon[46715]: debug 2022-01-31T23:02:39.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:39.192305+0000) 2022-01-31T23:02:39.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:39 smithi167 conmon[54076]: debug 2022-01-31T23:02:39.196+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:39.196913+0000) 2022-01-31T23:02:39.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:39 smithi171 conmon[41853]: debug 2022-01-31T23:02:39.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:39.396477+0000) 2022-01-31T23:02:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:39 smithi171 conmon[51620]: debug 2022-01-31T23:02:39.570+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:39.571237+0000) 2022-01-31T23:02:40.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:40 smithi167 conmon[49112]: debug 2022-01-31T23:02:40.026+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:40.027570+0000) 2022-01-31T23:02:40.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:39 smithi167 conmon[60316]: debug 2022-01-31T23:02:39.956+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:39.957067+0000) 2022-01-31T23:02:40.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:40 smithi167 conmon[54076]: debug 2022-01-31T23:02:40.196+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:40.197153+0000) 2022-01-31T23:02:40.554 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:40 smithi171 conmon[41853]: debug 2022-01-31T23:02:40.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:40.396664+0000) 2022-01-31T23:02:40.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:40 smithi171 conmon[46715]: debug 2022-01-31T23:02:40.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:40.192454+0000) 2022-01-31T23:02:40.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:40 smithi171 conmon[51620]: debug 2022-01-31T23:02:40.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:40.571465+0000) 2022-01-31T23:02:41.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:41 smithi167 conmon[49112]: debug 2022-01-31T23:02:41.026+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.027702+0000) 2022-01-31T23:02:41.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:40 smithi167 conmon[60316]: debug 2022-01-31T23:02:40.955+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:40.957223+0000) 2022-01-31T23:02:41.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:41 smithi167 conmon[54076]: debug 2022-01-31T23:02:41.196+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.197367+0000) 2022-01-31T23:02:41.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[46715]: debug 2022-01-31T23:02:41.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.192606+0000) 2022-01-31T23:02:41.555 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[41853]: debug 2022-01-31T23:02:41.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.396822+0000) 2022-01-31T23:02:41.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[35325]: debug 2022-01-31T23:02:41.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258103 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:41.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[41853]: debug 2022-01-31T23:02:41.755+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.756240+0000) 2022-01-31T23:02:41.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[46715]: debug 2022-01-31T23:02:41.754+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.755680+0000) 2022-01-31T23:02:41.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[51620]: debug 2022-01-31T23:02:41.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.571652+0000) 2022-01-31T23:02:41.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:41 smithi171 conmon[51620]: debug 2022-01-31T23:02:41.756+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.757171+0000) 2022-01-31T23:02:41.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:41 smithi167 conmon[49112]: debug 2022-01-31T23:02:41.755+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.756512+0000) 2022-01-31T23:02:41.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:41 smithi167 conmon[54076]: debug 2022-01-31T23:02:41.756+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.757733+0000) 2022-01-31T23:02:41.904 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:41 smithi167 conmon[60316]: debug 2022-01-31T23:02:41.755+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.757184+0000) 2022-01-31T23:02:42.180 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:42 smithi167 conmon[49112]: debug 2022-01-31T23:02:42.027+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:42.027857+0000) 2022-01-31T23:02:42.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:41 smithi167 conmon[60316]: debug 2022-01-31T23:02:41.956+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:41.957404+0000) 2022-01-31T23:02:42.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:42 smithi167 conmon[54076]: debug 2022-01-31T23:02:42.196+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:42.197548+0000) 2022-01-31T23:02:42.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:42 smithi171 conmon[46715]: debug 2022-01-31T23:02:42.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:42.192843+0000) 2022-01-31T23:02:42.555 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:42 smithi171 conmon[41853]: debug 2022-01-31T23:02:42.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:42.396947+0000) 2022-01-31T23:02:42.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:42 smithi171 conmon[51620]: debug 2022-01-31T23:02:42.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:42.571841+0000) 2022-01-31T23:02:43.181 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:43 smithi167 conmon[49112]: debug 2022-01-31T23:02:43.027+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:43.027975+0000) 2022-01-31T23:02:43.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:42 smithi167 conmon[60316]: debug 2022-01-31T23:02:42.956+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:42.957533+0000) 2022-01-31T23:02:43.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:43 smithi167 conmon[54076]: debug 2022-01-31T23:02:43.197+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:43.197692+0000) 2022-01-31T23:02:43.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:43 smithi171 conmon[46715]: debug 2022-01-31T23:02:43.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:43.193006+0000) 2022-01-31T23:02:43.555 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:43 smithi171 conmon[41853]: debug 2022-01-31T23:02:43.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:43.397078+0000) 2022-01-31T23:02:43.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:43 smithi171 conmon[51620]: debug 2022-01-31T23:02:43.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:43.571981+0000) 2022-01-31T23:02:44.181 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:44 smithi167 conmon[49112]: debug 2022-01-31T23:02:44.026+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:44.028157+0000) 2022-01-31T23:02:44.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:43 smithi167 conmon[60316]: debug 2022-01-31T23:02:43.957+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:43.957719+0000) 2022-01-31T23:02:44.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:44 smithi167 conmon[54076]: debug 2022-01-31T23:02:44.196+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:44.197846+0000) 2022-01-31T23:02:44.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:44 smithi171 conmon[46715]: debug 2022-01-31T23:02:44.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:44.193170+0000) 2022-01-31T23:02:44.555 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:44 smithi171 conmon[41853]: debug 2022-01-31T23:02:44.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:44.397311+0000) 2022-01-31T23:02:44.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:44 smithi171 conmon[51620]: debug 2022-01-31T23:02:44.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:44.572167+0000) 2022-01-31T23:02:45.181 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:45 smithi167 conmon[49112]: debug 2022-01-31T23:02:45.027+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:45.028371+0000) 2022-01-31T23:02:45.182 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:44 smithi167 conmon[60316]: debug 2022-01-31T23:02:44.956+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:44.957880+0000) 2022-01-31T23:02:45.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:45 smithi167 conmon[54076]: debug 2022-01-31T23:02:45.197+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:45.198036+0000) 2022-01-31T23:02:45.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:45 smithi171 conmon[46715]: debug 2022-01-31T23:02:45.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:45.193318+0000) 2022-01-31T23:02:45.555 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:45 smithi171 conmon[41853]: debug 2022-01-31T23:02:45.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:45.397475+0000) 2022-01-31T23:02:45.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:45 smithi171 conmon[51620]: debug 2022-01-31T23:02:45.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:45.572264+0000) 2022-01-31T23:02:46.181 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:46 smithi167 conmon[49112]: debug 2022-01-31T23:02:46.027+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.028579+0000) 2022-01-31T23:02:46.182 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:45 smithi167 conmon[60316]: debug 2022-01-31T23:02:45.956+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:45.958051+0000) 2022-01-31T23:02:46.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:46 smithi167 conmon[54076]: debug 2022-01-31T23:02:46.197+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.198188+0000) 2022-01-31T23:02:46.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[46715]: debug 2022-01-31T23:02:46.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.193465+0000) 2022-01-31T23:02:46.556 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[41853]: debug 2022-01-31T23:02:46.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.397660+0000) 2022-01-31T23:02:46.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[35325]: debug 2022-01-31T23:02:46.811+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258215 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:46.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[41853]: debug 2022-01-31T23:02:46.784+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.785412+0000) 2022-01-31T23:02:46.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[46715]: debug 2022-01-31T23:02:46.784+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.785507+0000) 2022-01-31T23:02:46.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[51620]: debug 2022-01-31T23:02:46.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.572447+0000) 2022-01-31T23:02:46.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:46 smithi171 conmon[51620]: debug 2022-01-31T23:02:46.785+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.786208+0000) 2022-01-31T23:02:46.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:46 smithi167 conmon[49112]: debug 2022-01-31T23:02:46.783+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.785166+0000) 2022-01-31T23:02:46.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:46 smithi167 conmon[54076]: debug 2022-01-31T23:02:46.785+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.786780+0000) 2022-01-31T23:02:46.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:46 smithi167 conmon[60316]: debug 2022-01-31T23:02:46.784+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.785808+0000) 2022-01-31T23:02:47.181 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:47 smithi167 conmon[49112]: debug 2022-01-31T23:02:47.028+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:47.028703+0000) 2022-01-31T23:02:47.182 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:46 smithi167 conmon[60316]: debug 2022-01-31T23:02:46.957+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:46.958174+0000) 2022-01-31T23:02:47.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:47 smithi167 conmon[54076]: debug 2022-01-31T23:02:47.197+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:47.198336+0000) 2022-01-31T23:02:47.555 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:47 smithi171 conmon[41853]: debug 2022-01-31T23:02:47.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:47.397853+0000) 2022-01-31T23:02:47.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:47 smithi171 conmon[46715]: debug 2022-01-31T23:02:47.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:47.193657+0000) 2022-01-31T23:02:47.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:47 smithi171 conmon[51620]: debug 2022-01-31T23:02:47.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:47.572626+0000) 2022-01-31T23:02:48.181 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:47 smithi167 conmon[60316]: debug 2022-01-31T23:02:47.957+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:47.958317+0000) 2022-01-31T23:02:48.182 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:48 smithi167 conmon[49112]: debug 2022-01-31T23:02:48.028+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:48.028866+0000) 2022-01-31T23:02:48.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:48 smithi167 conmon[54076]: debug 2022-01-31T23:02:48.198+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:48.198489+0000) 2022-01-31T23:02:48.555 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:48 smithi171 conmon[46715]: debug 2022-01-31T23:02:48.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:48.193817+0000) 2022-01-31T23:02:48.556 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:48 smithi171 conmon[41853]: debug 2022-01-31T23:02:48.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:48.398009+0000) 2022-01-31T23:02:48.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:48 smithi171 conmon[51620]: debug 2022-01-31T23:02:48.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:48.572791+0000) 2022-01-31T23:02:49.182 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:49 smithi167 conmon[49112]: debug 2022-01-31T23:02:49.028+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:49.029042+0000) 2022-01-31T23:02:49.182 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:48 smithi167 conmon[60316]: debug 2022-01-31T23:02:48.957+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:48.958442+0000) 2022-01-31T23:02:49.536 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:49 smithi167 conmon[54076]: debug 2022-01-31T23:02:49.198+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:49.198689+0000) 2022-01-31T23:02:49.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:49 smithi171 conmon[46715]: debug 2022-01-31T23:02:49.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:49.193996+0000) 2022-01-31T23:02:49.556 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:49 smithi171 conmon[41853]: debug 2022-01-31T23:02:49.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:49.398215+0000) 2022-01-31T23:02:49.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:49 smithi171 conmon[51620]: debug 2022-01-31T23:02:49.571+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:49.572991+0000) 2022-01-31T23:02:50.182 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:50 smithi167 conmon[49112]: debug 2022-01-31T23:02:50.028+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:50.029258+0000) 2022-01-31T23:02:50.183 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:49 smithi167 conmon[60316]: debug 2022-01-31T23:02:49.958+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:49.958626+0000) 2022-01-31T23:02:50.537 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:50 smithi167 conmon[54076]: debug 2022-01-31T23:02:50.198+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:50.198891+0000) 2022-01-31T23:02:50.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:50 smithi171 conmon[46715]: debug 2022-01-31T23:02:50.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:50.194189+0000) 2022-01-31T23:02:50.556 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:50 smithi171 conmon[41853]: debug 2022-01-31T23:02:50.396+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:50.398439+0000) 2022-01-31T23:02:50.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:50 smithi171 conmon[51620]: debug 2022-01-31T23:02:50.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:50.573185+0000) 2022-01-31T23:02:51.182 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:51 smithi167 conmon[49112]: debug 2022-01-31T23:02:51.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.029521+0000) 2022-01-31T23:02:51.183 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:50 smithi167 conmon[60316]: debug 2022-01-31T23:02:50.958+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:50.958840+0000) 2022-01-31T23:02:51.537 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:51 smithi167 conmon[54076]: debug 2022-01-31T23:02:51.198+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.199103+0000) 2022-01-31T23:02:51.556 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[46715]: debug 2022-01-31T23:02:51.192+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.194452+0000) 2022-01-31T23:02:51.557 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[41853]: debug 2022-01-31T23:02:51.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.398679+0000) 2022-01-31T23:02:51.824 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[46715]: debug 2022-01-31T23:02:51.813+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.814807+0000) 2022-01-31T23:02:51.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[51620]: debug 2022-01-31T23:02:51.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.573457+0000) 2022-01-31T23:02:51.825 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[51620]: debug 2022-01-31T23:02:51.813+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.815388+0000) 2022-01-31T23:02:51.826 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[41853]: debug 2022-01-31T23:02:51.812+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.814649+0000) 2022-01-31T23:02:52.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:51 smithi171 conmon[35325]: debug 2022-01-31T23:02:51.839+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258326 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:52.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:51 smithi167 conmon[49112]: debug 2022-01-31T23:02:51.813+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.814385+0000) 2022-01-31T23:02:52.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:52 smithi167 conmon[49112]: debug 2022-01-31T23:02:52.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:52.029620+0000) 2022-01-31T23:02:52.141 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:51 smithi167 conmon[54076]: debug 2022-01-31T23:02:51.814+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.815096+0000) 2022-01-31T23:02:52.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:51 smithi167 conmon[60316]: debug 2022-01-31T23:02:51.815+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.815679+0000) 2022-01-31T23:02:52.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:51 smithi167 conmon[60316]: debug 2022-01-31T23:02:51.958+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:51.958993+0000) 2022-01-31T23:02:52.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:52 smithi167 conmon[54076]: debug 2022-01-31T23:02:52.198+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:52.199241+0000) 2022-01-31T23:02:52.556 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:52 smithi171 conmon[41853]: debug 2022-01-31T23:02:52.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:52.398839+0000) 2022-01-31T23:02:52.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:52 smithi171 conmon[46715]: debug 2022-01-31T23:02:52.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:52.194573+0000) 2022-01-31T23:02:52.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:52 smithi171 conmon[51620]: debug 2022-01-31T23:02:52.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:52.573597+0000) 2022-01-31T23:02:53.182 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:53 smithi167 conmon[49112]: debug 2022-01-31T23:02:53.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:53.029725+0000) 2022-01-31T23:02:53.183 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:52 smithi167 conmon[60316]: debug 2022-01-31T23:02:52.958+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:52.959107+0000) 2022-01-31T23:02:53.537 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:53 smithi167 conmon[54076]: debug 2022-01-31T23:02:53.198+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:53.199413+0000) 2022-01-31T23:02:53.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:53 smithi171 conmon[46715]: debug 2022-01-31T23:02:53.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:53.194730+0000) 2022-01-31T23:02:53.557 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:53 smithi171 conmon[41853]: debug 2022-01-31T23:02:53.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:53.399013+0000) 2022-01-31T23:02:53.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:53 smithi171 conmon[51620]: debug 2022-01-31T23:02:53.572+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:53.573739+0000) 2022-01-31T23:02:54.182 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:54 smithi167 conmon[49112]: debug 2022-01-31T23:02:54.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:54.029921+0000) 2022-01-31T23:02:54.183 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:53 smithi167 conmon[60316]: debug 2022-01-31T23:02:53.958+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:53.959309+0000) 2022-01-31T23:02:54.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:53 smithi171 conmon[35325]: debug 2022-01-31T23:02:53.952+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:02:54.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:54 smithi171 conmon[46715]: debug 2022-01-31T23:02:54.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:54.194902+0000) 2022-01-31T23:02:54.537 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:54 smithi167 conmon[54076]: debug 2022-01-31T23:02:54.199+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:54.199550+0000) 2022-01-31T23:02:54.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:54 smithi171 conmon[41853]: debug 2022-01-31T23:02:54.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:54.399144+0000) 2022-01-31T23:02:54.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:54 smithi171 conmon[51620]: debug 2022-01-31T23:02:54.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:54.573957+0000) 2022-01-31T23:02:55.183 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:55 smithi167 conmon[49112]: debug 2022-01-31T23:02:55.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:55.030109+0000) 2022-01-31T23:02:55.183 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:54 smithi167 conmon[60316]: debug 2022-01-31T23:02:54.958+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:54.959443+0000) 2022-01-31T23:02:55.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:55 smithi167 conmon[54076]: debug 2022-01-31T23:02:55.199+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:55.199721+0000) 2022-01-31T23:02:55.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:55 smithi171 conmon[46715]: debug 2022-01-31T23:02:55.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:55.195062+0000) 2022-01-31T23:02:55.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:55 smithi171 conmon[41853]: debug 2022-01-31T23:02:55.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:55.399299+0000) 2022-01-31T23:02:55.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:55 smithi171 conmon[51620]: debug 2022-01-31T23:02:55.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:55.574143+0000) 2022-01-31T23:02:56.183 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:56 smithi167 conmon[49112]: debug 2022-01-31T23:02:56.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.030293+0000) 2022-01-31T23:02:56.184 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:55 smithi167 conmon[60316]: debug 2022-01-31T23:02:55.959+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:55.959613+0000) 2022-01-31T23:02:56.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:56 smithi167 conmon[54076]: debug 2022-01-31T23:02:56.199+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.199886+0000) 2022-01-31T23:02:56.557 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[46715]: debug 2022-01-31T23:02:56.193+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.195223+0000) 2022-01-31T23:02:56.557 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[41853]: debug 2022-01-31T23:02:56.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.399490+0000) 2022-01-31T23:02:56.826 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[51620]: debug 2022-01-31T23:02:56.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.574327+0000) 2022-01-31T23:02:57.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[35325]: debug 2022-01-31T23:02:56.874+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258438 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:02:57.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[41853]: debug 2022-01-31T23:02:56.841+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.843719+0000) 2022-01-31T23:02:57.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[46715]: debug 2022-01-31T23:02:56.842+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.844133+0000) 2022-01-31T23:02:57.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:56 smithi171 conmon[51620]: debug 2022-01-31T23:02:56.841+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.843271+0000) 2022-01-31T23:02:57.140 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:56 smithi167 conmon[49112]: debug 2022-01-31T23:02:56.843+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.844074+0000) 2022-01-31T23:02:57.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:57 smithi167 conmon[49112]: debug 2022-01-31T23:02:57.029+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:57.030469+0000) 2022-01-31T23:02:57.141 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:56 smithi167 conmon[54076]: debug 2022-01-31T23:02:56.842+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.843094+0000) 2022-01-31T23:02:57.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:56 smithi167 conmon[60316]: debug 2022-01-31T23:02:56.843+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.844021+0000) 2022-01-31T23:02:57.142 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:56 smithi167 conmon[60316]: debug 2022-01-31T23:02:56.959+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:56.959771+0000) 2022-01-31T23:02:57.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:57 smithi167 conmon[54076]: debug 2022-01-31T23:02:57.199+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:57.200045+0000) 2022-01-31T23:02:57.557 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:57 smithi171 conmon[41853]: debug 2022-01-31T23:02:57.397+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:57.399652+0000) 2022-01-31T23:02:57.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:57 smithi171 conmon[46715]: debug 2022-01-31T23:02:57.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:57.195377+0000) 2022-01-31T23:02:57.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:57 smithi171 conmon[51620]: debug 2022-01-31T23:02:57.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:57.574493+0000) 2022-01-31T23:02:58.183 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:58 smithi167 conmon[49112]: debug 2022-01-31T23:02:58.030+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:58.030617+0000) 2022-01-31T23:02:58.184 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:57 smithi167 conmon[60316]: debug 2022-01-31T23:02:57.959+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:57.959947+0000) 2022-01-31T23:02:58.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:58 smithi167 conmon[54076]: debug 2022-01-31T23:02:58.199+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:58.200219+0000) 2022-01-31T23:02:58.557 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:58 smithi171 conmon[41853]: debug 2022-01-31T23:02:58.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:58.399831+0000) 2022-01-31T23:02:58.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:58 smithi171 conmon[46715]: debug 2022-01-31T23:02:58.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:58.195525+0000) 2022-01-31T23:02:58.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:58 smithi171 conmon[51620]: debug 2022-01-31T23:02:58.573+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:58.574652+0000) 2022-01-31T23:02:59.183 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:02:59 smithi167 conmon[49112]: debug 2022-01-31T23:02:59.030+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:59.030796+0000) 2022-01-31T23:02:59.184 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:58 smithi167 conmon[60316]: debug 2022-01-31T23:02:58.959+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:58.960073+0000) 2022-01-31T23:02:59.437 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:02:59 smithi167 conmon[54076]: debug 2022-01-31T23:02:59.200+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:59.200408+0000) 2022-01-31T23:02:59.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:02:59 smithi171 conmon[46715]: debug 2022-01-31T23:02:59.194+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:59.195725+0000) 2022-01-31T23:02:59.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:02:59 smithi171 conmon[41853]: debug 2022-01-31T23:02:59.398+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:59.399999+0000) 2022-01-31T23:02:59.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:02:59 smithi171 conmon[51620]: debug 2022-01-31T23:02:59.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:59.574839+0000) 2022-01-31T23:03:00.183 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:02:59 smithi167 conmon[60316]: debug 2022-01-31T23:02:59.959+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:02:59.960236+0000) 2022-01-31T23:03:00.184 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:00 smithi167 conmon[49112]: debug 2022-01-31T23:03:00.030+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:00.030966+0000) 2022-01-31T23:03:00.442 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:00 smithi167 conmon[54076]: debug 2022-01-31T23:03:00.200+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:00.200606+0000) 2022-01-31T23:03:00.558 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:00 smithi171 conmon[46715]: debug 2022-01-31T23:03:00.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:00.195872+0000) 2022-01-31T23:03:00.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:00 smithi171 conmon[41853]: debug 2022-01-31T23:03:00.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:00.400184+0000) 2022-01-31T23:03:00.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:00 smithi171 conmon[51620]: debug 2022-01-31T23:03:00.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:00.575003+0000) 2022-01-31T23:03:01.184 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:01 smithi167 conmon[49112]: debug 2022-01-31T23:03:01.030+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.031165+0000) 2022-01-31T23:03:01.184 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:00 smithi167 conmon[60316]: debug 2022-01-31T23:03:00.959+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:00.960365+0000) 2022-01-31T23:03:01.447 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:01 smithi167 conmon[54076]: debug 2022-01-31T23:03:01.200+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.200784+0000) 2022-01-31T23:03:01.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[41853]: debug 2022-01-31T23:03:01.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.400366+0000) 2022-01-31T23:03:01.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[46715]: debug 2022-01-31T23:03:01.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.196018+0000) 2022-01-31T23:03:01.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[51620]: debug 2022-01-31T23:03:01.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.575182+0000) 2022-01-31T23:03:02.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:01 smithi167 conmon[49112]: debug 2022-01-31T23:03:01.880+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.881167+0000) 2022-01-31T23:03:02.142 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:02 smithi167 conmon[49112]: debug 2022-01-31T23:03:02.030+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:02.031373+0000) 2022-01-31T23:03:02.142 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:01 smithi167 conmon[54076]: debug 2022-01-31T23:03:01.878+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.879307+0000) 2022-01-31T23:03:02.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:01 smithi167 conmon[60316]: debug 2022-01-31T23:03:01.879+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.879515+0000) 2022-01-31T23:03:02.143 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:01 smithi167 conmon[60316]: debug 2022-01-31T23:03:01.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.960543+0000) 2022-01-31T23:03:02.179 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[35325]: debug 2022-01-31T23:03:01.904+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258549 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:02.179 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[46715]: debug 2022-01-31T23:03:01.877+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.878714+0000) 2022-01-31T23:03:02.180 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[51620]: debug 2022-01-31T23:03:01.878+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.878929+0000) 2022-01-31T23:03:02.180 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:01 smithi171 conmon[41853]: debug 2022-01-31T23:03:01.878+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:01.879456+0000) 2022-01-31T23:03:02.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:02 smithi167 conmon[54076]: debug 2022-01-31T23:03:02.200+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:02.200989+0000) 2022-01-31T23:03:02.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:02 smithi171 conmon[41853]: debug 2022-01-31T23:03:02.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:02.400551+0000) 2022-01-31T23:03:02.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:02 smithi171 conmon[46715]: debug 2022-01-31T23:03:02.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:02.196177+0000) 2022-01-31T23:03:02.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:02 smithi171 conmon[51620]: debug 2022-01-31T23:03:02.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:02.575387+0000) 2022-01-31T23:03:03.184 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:03 smithi167 conmon[49112]: debug 2022-01-31T23:03:03.031+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:03.031549+0000) 2022-01-31T23:03:03.185 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:02 smithi167 conmon[60316]: debug 2022-01-31T23:03:02.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:02.960643+0000) 2022-01-31T23:03:03.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:03 smithi167 conmon[54076]: debug 2022-01-31T23:03:03.200+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:03.201139+0000) 2022-01-31T23:03:03.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:03 smithi171 conmon[41853]: debug 2022-01-31T23:03:03.399+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:03.400719+0000) 2022-01-31T23:03:03.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:03 smithi171 conmon[46715]: debug 2022-01-31T23:03:03.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:03.196330+0000) 2022-01-31T23:03:03.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:03 smithi171 conmon[51620]: debug 2022-01-31T23:03:03.574+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:03.575568+0000) 2022-01-31T23:03:04.184 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:04 smithi167 conmon[49112]: debug 2022-01-31T23:03:04.031+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:04.031675+0000) 2022-01-31T23:03:04.185 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:03 smithi167 conmon[60316]: debug 2022-01-31T23:03:03.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:03.960812+0000) 2022-01-31T23:03:04.458 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:04 smithi167 conmon[54076]: debug 2022-01-31T23:03:04.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:04.201328+0000) 2022-01-31T23:03:04.559 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:04 smithi171 conmon[41853]: debug 2022-01-31T23:03:04.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:04.400868+0000) 2022-01-31T23:03:04.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:04 smithi171 conmon[46715]: debug 2022-01-31T23:03:04.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:04.196539+0000) 2022-01-31T23:03:04.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:04 smithi171 conmon[51620]: debug 2022-01-31T23:03:04.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:04.575811+0000) 2022-01-31T23:03:05.184 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:05 smithi167 conmon[49112]: debug 2022-01-31T23:03:05.031+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:05.031859+0000) 2022-01-31T23:03:05.185 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:04 smithi167 conmon[60316]: debug 2022-01-31T23:03:04.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:04.960959+0000) 2022-01-31T23:03:05.462 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:05 smithi167 conmon[54076]: debug 2022-01-31T23:03:05.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:05.201536+0000) 2022-01-31T23:03:05.558 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:05 smithi171 conmon[41853]: debug 2022-01-31T23:03:05.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:05.401081+0000) 2022-01-31T23:03:05.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:05 smithi171 conmon[46715]: debug 2022-01-31T23:03:05.195+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:05.196694+0000) 2022-01-31T23:03:05.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:05 smithi171 conmon[51620]: debug 2022-01-31T23:03:05.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:05.576005+0000) 2022-01-31T23:03:06.184 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:06 smithi167 conmon[49112]: debug 2022-01-31T23:03:06.031+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.032020+0000) 2022-01-31T23:03:06.185 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:05 smithi167 conmon[60316]: debug 2022-01-31T23:03:05.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:05.961146+0000) 2022-01-31T23:03:06.466 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:06 smithi167 conmon[54076]: debug 2022-01-31T23:03:06.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.201726+0000) 2022-01-31T23:03:06.559 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[41853]: debug 2022-01-31T23:03:06.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.401285+0000) 2022-01-31T23:03:06.559 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[46715]: debug 2022-01-31T23:03:06.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.196926+0000) 2022-01-31T23:03:06.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[51620]: debug 2022-01-31T23:03:06.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.576199+0000) 2022-01-31T23:03:07.142 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:06 smithi167 conmon[54076]: debug 2022-01-31T23:03:06.907+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.908326+0000) 2022-01-31T23:03:07.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:06 smithi167 conmon[49112]: debug 2022-01-31T23:03:06.908+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.909447+0000) 2022-01-31T23:03:07.143 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:07 smithi167 conmon[49112]: debug 2022-01-31T23:03:07.031+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:07.032205+0000) 2022-01-31T23:03:07.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:06 smithi167 conmon[60316]: debug 2022-01-31T23:03:06.908+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.908564+0000) 2022-01-31T23:03:07.144 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:06 smithi167 conmon[60316]: debug 2022-01-31T23:03:06.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.961277+0000) 2022-01-31T23:03:07.180 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[35325]: debug 2022-01-31T23:03:06.934+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258664 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:07.180 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[41853]: debug 2022-01-31T23:03:06.908+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.909575+0000) 2022-01-31T23:03:07.181 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[46715]: debug 2022-01-31T23:03:06.906+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.907649+0000) 2022-01-31T23:03:07.181 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:06 smithi171 conmon[51620]: debug 2022-01-31T23:03:06.907+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:06.907964+0000) 2022-01-31T23:03:07.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:07 smithi167 conmon[54076]: debug 2022-01-31T23:03:07.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:07.201913+0000) 2022-01-31T23:03:07.559 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:07 smithi171 conmon[41853]: debug 2022-01-31T23:03:07.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:07.401485+0000) 2022-01-31T23:03:07.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:07 smithi171 conmon[46715]: debug 2022-01-31T23:03:07.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:07.197087+0000) 2022-01-31T23:03:07.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:07 smithi171 conmon[51620]: debug 2022-01-31T23:03:07.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:07.576392+0000) 2022-01-31T23:03:08.185 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:08 smithi167 conmon[49112]: debug 2022-01-31T23:03:08.031+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:08.032379+0000) 2022-01-31T23:03:08.186 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:07 smithi167 conmon[60316]: debug 2022-01-31T23:03:07.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:07.961422+0000) 2022-01-31T23:03:08.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:08 smithi167 conmon[54076]: debug 2022-01-31T23:03:08.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:08.202078+0000) 2022-01-31T23:03:08.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:08 smithi171 conmon[41853]: debug 2022-01-31T23:03:08.400+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:08.401618+0000) 2022-01-31T23:03:08.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:08 smithi171 conmon[46715]: debug 2022-01-31T23:03:08.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:08.197216+0000) 2022-01-31T23:03:08.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:08 smithi171 conmon[51620]: debug 2022-01-31T23:03:08.575+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:08.576605+0000) 2022-01-31T23:03:09.185 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:08 smithi167 conmon[60316]: debug 2022-01-31T23:03:08.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:08.961523+0000) 2022-01-31T23:03:09.186 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:09 smithi167 conmon[49112]: debug 2022-01-31T23:03:09.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:09.032615+0000) 2022-01-31T23:03:09.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:08 smithi171 conmon[35325]: debug 2022-01-31T23:03:08.953+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:03:09.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:09 smithi171 conmon[46715]: debug 2022-01-31T23:03:09.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:09.197361+0000) 2022-01-31T23:03:09.477 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:09 smithi167 conmon[54076]: debug 2022-01-31T23:03:09.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:09.202277+0000) 2022-01-31T23:03:09.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:09 smithi171 conmon[41853]: debug 2022-01-31T23:03:09.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:09.401810+0000) 2022-01-31T23:03:09.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:09 smithi171 conmon[51620]: debug 2022-01-31T23:03:09.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:09.576793+0000) 2022-01-31T23:03:10.185 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:10 smithi167 conmon[49112]: debug 2022-01-31T23:03:10.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:10.032817+0000) 2022-01-31T23:03:10.186 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:09 smithi167 conmon[60316]: debug 2022-01-31T23:03:09.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:09.961681+0000) 2022-01-31T23:03:10.480 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:10 smithi167 conmon[54076]: debug 2022-01-31T23:03:10.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:10.202437+0000) 2022-01-31T23:03:10.560 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:10 smithi171 conmon[41853]: debug 2022-01-31T23:03:10.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:10.401931+0000) 2022-01-31T23:03:10.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:10 smithi171 conmon[46715]: debug 2022-01-31T23:03:10.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:10.197517+0000) 2022-01-31T23:03:10.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:10 smithi171 conmon[51620]: debug 2022-01-31T23:03:10.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:10.576988+0000) 2022-01-31T23:03:11.185 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:10 smithi167 conmon[60316]: debug 2022-01-31T23:03:10.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:10.961788+0000) 2022-01-31T23:03:11.186 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:11 smithi167 conmon[49112]: debug 2022-01-31T23:03:11.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.033019+0000) 2022-01-31T23:03:11.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:11 smithi167 conmon[54076]: debug 2022-01-31T23:03:11.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.202651+0000) 2022-01-31T23:03:11.560 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[41853]: debug 2022-01-31T23:03:11.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.402080+0000) 2022-01-31T23:03:11.560 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[46715]: debug 2022-01-31T23:03:11.196+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.197686+0000) 2022-01-31T23:03:11.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[51620]: debug 2022-01-31T23:03:11.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.577230+0000) 2022-01-31T23:03:12.181 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[35325]: debug 2022-01-31T23:03:11.962+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258775 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:12.181 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[41853]: debug 2022-01-31T23:03:11.937+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.938260+0000) 2022-01-31T23:03:12.182 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[46715]: debug 2022-01-31T23:03:11.936+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.937262+0000) 2022-01-31T23:03:12.183 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:11 smithi171 conmon[51620]: debug 2022-01-31T23:03:11.936+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.937043+0000) 2022-01-31T23:03:12.185 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:11 smithi167 conmon[54076]: debug 2022-01-31T23:03:11.937+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.938373+0000) 2022-01-31T23:03:12.186 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:11 smithi167 conmon[60316]: debug 2022-01-31T23:03:11.937+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.938137+0000) 2022-01-31T23:03:12.186 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:11 smithi167 conmon[60316]: debug 2022-01-31T23:03:11.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.961957+0000) 2022-01-31T23:03:12.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:11 smithi167 conmon[49112]: debug 2022-01-31T23:03:11.938+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:11.939097+0000) 2022-01-31T23:03:12.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:12 smithi167 conmon[49112]: debug 2022-01-31T23:03:12.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:12.033198+0000) 2022-01-31T23:03:12.432 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:12 smithi171 conmon[41853]: debug 2022-01-31T23:03:12.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:12.402275+0000) 2022-01-31T23:03:12.433 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:12 smithi171 conmon[46715]: debug 2022-01-31T23:03:12.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:12.197873+0000) 2022-01-31T23:03:12.488 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:12 smithi167 conmon[54076]: debug 2022-01-31T23:03:12.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:12.202826+0000) 2022-01-31T23:03:12.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:12 smithi171 conmon[51620]: debug 2022-01-31T23:03:12.576+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:12.577418+0000) 2022-01-31T23:03:13.186 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:13 smithi167 conmon[49112]: debug 2022-01-31T23:03:13.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:13.033386+0000) 2022-01-31T23:03:13.187 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:12 smithi167 conmon[60316]: debug 2022-01-31T23:03:12.960+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:12.962106+0000) 2022-01-31T23:03:13.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:13 smithi167 conmon[54076]: debug 2022-01-31T23:03:13.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:13.202979+0000) 2022-01-31T23:03:13.560 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:13 smithi171 conmon[41853]: debug 2022-01-31T23:03:13.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:13.402449+0000) 2022-01-31T23:03:13.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:13 smithi171 conmon[46715]: debug 2022-01-31T23:03:13.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:13.198013+0000) 2022-01-31T23:03:13.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:13 smithi171 conmon[51620]: debug 2022-01-31T23:03:13.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:13.577570+0000) 2022-01-31T23:03:14.186 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:14 smithi167 conmon[49112]: debug 2022-01-31T23:03:14.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:14.033580+0000) 2022-01-31T23:03:14.187 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:13 smithi167 conmon[60316]: debug 2022-01-31T23:03:13.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:13.962326+0000) 2022-01-31T23:03:14.444 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:14 smithi167 conmon[54076]: debug 2022-01-31T23:03:14.201+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:14.203131+0000) 2022-01-31T23:03:14.561 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:14 smithi171 conmon[41853]: debug 2022-01-31T23:03:14.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:14.402574+0000) 2022-01-31T23:03:14.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:14 smithi171 conmon[46715]: debug 2022-01-31T23:03:14.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:14.198201+0000) 2022-01-31T23:03:14.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:14 smithi171 conmon[51620]: debug 2022-01-31T23:03:14.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:14.577791+0000) 2022-01-31T23:03:15.186 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:15 smithi167 conmon[49112]: debug 2022-01-31T23:03:15.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:15.033772+0000) 2022-01-31T23:03:15.187 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:14 smithi167 conmon[60316]: debug 2022-01-31T23:03:14.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:14.962534+0000) 2022-01-31T23:03:15.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:15 smithi167 conmon[54076]: debug 2022-01-31T23:03:15.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:15.203311+0000) 2022-01-31T23:03:15.561 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:15 smithi171 conmon[41853]: debug 2022-01-31T23:03:15.401+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:15.402681+0000) 2022-01-31T23:03:15.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:15 smithi171 conmon[46715]: debug 2022-01-31T23:03:15.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:15.198362+0000) 2022-01-31T23:03:15.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:15 smithi171 conmon[51620]: debug 2022-01-31T23:03:15.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:15.578010+0000) 2022-01-31T23:03:16.186 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:16 smithi167 conmon[49112]: debug 2022-01-31T23:03:16.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.033933+0000) 2022-01-31T23:03:16.187 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:15 smithi167 conmon[60316]: debug 2022-01-31T23:03:15.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:15.962732+0000) 2022-01-31T23:03:16.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:16 smithi167 conmon[54076]: debug 2022-01-31T23:03:16.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.203548+0000) 2022-01-31T23:03:16.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[46715]: debug 2022-01-31T23:03:16.197+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.198506+0000) 2022-01-31T23:03:16.561 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[41853]: debug 2022-01-31T23:03:16.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.402872+0000) 2022-01-31T23:03:16.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[51620]: debug 2022-01-31T23:03:16.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.578208+0000) 2022-01-31T23:03:17.187 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:16 smithi167 conmon[54076]: debug 2022-01-31T23:03:16.964+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.965990+0000) 2022-01-31T23:03:17.187 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:16 smithi167 conmon[60316]: debug 2022-01-31T23:03:16.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.962891+0000) 2022-01-31T23:03:17.188 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:16 smithi167 conmon[60316]: debug 2022-01-31T23:03:16.964+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.966188+0000) 2022-01-31T23:03:17.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:16 smithi167 conmon[49112]: debug 2022-01-31T23:03:16.964+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.966336+0000) 2022-01-31T23:03:17.189 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:17 smithi167 conmon[49112]: debug 2022-01-31T23:03:17.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:17.034111+0000) 2022-01-31T23:03:17.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[51620]: debug 2022-01-31T23:03:16.965+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.966620+0000) 2022-01-31T23:03:17.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[35325]: debug 2022-01-31T23:03:16.991+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:17.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[41853]: debug 2022-01-31T23:03:16.965+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.966510+0000) 2022-01-31T23:03:17.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:16 smithi171 conmon[46715]: debug 2022-01-31T23:03:16.964+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:16.965070+0000) 2022-01-31T23:03:17.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:17 smithi171 conmon[46715]: debug 2022-01-31T23:03:17.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:17.198702+0000) 2022-01-31T23:03:17.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:17 smithi167 conmon[54076]: debug 2022-01-31T23:03:17.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:17.203733+0000) 2022-01-31T23:03:17.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:17 smithi171 conmon[41853]: debug 2022-01-31T23:03:17.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:17.403054+0000) 2022-01-31T23:03:17.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:17 smithi171 conmon[51620]: debug 2022-01-31T23:03:17.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:17.578342+0000) 2022-01-31T23:03:18.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:18 smithi167 conmon[49112]: debug 2022-01-31T23:03:18.033+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:18.034256+0000) 2022-01-31T23:03:18.188 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:17 smithi167 conmon[60316]: debug 2022-01-31T23:03:17.961+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:17.963051+0000) 2022-01-31T23:03:18.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:18 smithi167 conmon[54076]: debug 2022-01-31T23:03:18.202+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:18.203885+0000) 2022-01-31T23:03:18.561 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:18 smithi171 conmon[41853]: debug 2022-01-31T23:03:18.402+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:18.403218+0000) 2022-01-31T23:03:18.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:18 smithi171 conmon[46715]: debug 2022-01-31T23:03:18.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:18.198866+0000) 2022-01-31T23:03:18.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:18 smithi171 conmon[51620]: debug 2022-01-31T23:03:18.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:18.578487+0000) 2022-01-31T23:03:19.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:19 smithi167 conmon[49112]: debug 2022-01-31T23:03:19.032+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:19.034432+0000) 2022-01-31T23:03:19.188 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:18 smithi167 conmon[60316]: debug 2022-01-31T23:03:18.962+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:18.963192+0000) 2022-01-31T23:03:19.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:19 smithi167 conmon[54076]: debug 2022-01-31T23:03:19.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:19.204091+0000) 2022-01-31T23:03:19.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:19 smithi171 conmon[46715]: debug 2022-01-31T23:03:19.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:19.199058+0000) 2022-01-31T23:03:19.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:19 smithi171 conmon[41853]: debug 2022-01-31T23:03:19.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:19.403451+0000) 2022-01-31T23:03:19.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:19 smithi171 conmon[51620]: debug 2022-01-31T23:03:19.577+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:19.578699+0000) 2022-01-31T23:03:20.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:20 smithi167 conmon[49112]: debug 2022-01-31T23:03:20.033+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:20.034664+0000) 2022-01-31T23:03:20.188 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:19 smithi167 conmon[60316]: debug 2022-01-31T23:03:19.962+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:19.963377+0000) 2022-01-31T23:03:20.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:20 smithi167 conmon[54076]: debug 2022-01-31T23:03:20.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:20.204284+0000) 2022-01-31T23:03:20.561 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:20 smithi171 conmon[46715]: debug 2022-01-31T23:03:20.198+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:20.199222+0000) 2022-01-31T23:03:20.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:20 smithi171 conmon[41853]: debug 2022-01-31T23:03:20.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:20.403667+0000) 2022-01-31T23:03:20.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:20 smithi171 conmon[51620]: debug 2022-01-31T23:03:20.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:20.578872+0000) 2022-01-31T23:03:21.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:21 smithi167 conmon[49112]: debug 2022-01-31T23:03:21.034+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.034883+0000) 2022-01-31T23:03:21.188 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:20 smithi167 conmon[60316]: debug 2022-01-31T23:03:20.962+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:20.963611+0000) 2022-01-31T23:03:21.523 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:21 smithi167 conmon[54076]: debug 2022-01-31T23:03:21.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.204472+0000) 2022-01-31T23:03:21.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:21 smithi171 conmon[46715]: debug 2022-01-31T23:03:21.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.199401+0000) 2022-01-31T23:03:21.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:21 smithi171 conmon[41853]: debug 2022-01-31T23:03:21.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.403850+0000) 2022-01-31T23:03:21.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:21 smithi171 conmon[51620]: debug 2022-01-31T23:03:21.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.579063+0000) 2022-01-31T23:03:22.187 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:21 smithi167 conmon[49112]: debug 2022-01-31T23:03:21.994+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.996349+0000) 2022-01-31T23:03:22.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:22 smithi167 conmon[49112]: debug 2022-01-31T23:03:22.033+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:22.035041+0000) 2022-01-31T23:03:22.189 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:21 smithi167 conmon[54076]: debug 2022-01-31T23:03:21.993+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.994911+0000) 2022-01-31T23:03:22.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:21 smithi167 conmon[60316]: debug 2022-01-31T23:03:21.962+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.963837+0000) 2022-01-31T23:03:22.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:21 smithi167 conmon[60316]: debug 2022-01-31T23:03:21.994+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.995983+0000) 2022-01-31T23:03:22.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:22 smithi171 conmon[35325]: debug 2022-01-31T23:03:22.020+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 258998 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:22.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:21 smithi171 conmon[41853]: debug 2022-01-31T23:03:21.995+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.995931+0000) 2022-01-31T23:03:22.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:21 smithi171 conmon[51620]: debug 2022-01-31T23:03:21.994+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.995438+0000) 2022-01-31T23:03:22.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:21 smithi171 conmon[46715]: debug 2022-01-31T23:03:21.994+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:21.995186+0000) 2022-01-31T23:03:22.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:22 smithi171 conmon[46715]: debug 2022-01-31T23:03:22.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:22.199532+0000) 2022-01-31T23:03:22.527 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:22 smithi167 conmon[54076]: debug 2022-01-31T23:03:22.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:22.204687+0000) 2022-01-31T23:03:22.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:22 smithi171 conmon[41853]: debug 2022-01-31T23:03:22.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:22.404033+0000) 2022-01-31T23:03:22.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:22 smithi171 conmon[51620]: debug 2022-01-31T23:03:22.578+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:22.579250+0000) 2022-01-31T23:03:23.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:23 smithi167 conmon[49112]: debug 2022-01-31T23:03:23.033+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:23.035180+0000) 2022-01-31T23:03:23.188 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:22 smithi167 conmon[60316]: debug 2022-01-31T23:03:22.963+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:22.963989+0000) 2022-01-31T23:03:23.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:23 smithi167 conmon[54076]: debug 2022-01-31T23:03:23.203+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:23.204832+0000) 2022-01-31T23:03:23.562 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:23 smithi171 conmon[46715]: debug 2022-01-31T23:03:23.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:23.199676+0000) 2022-01-31T23:03:23.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:23 smithi171 conmon[41853]: debug 2022-01-31T23:03:23.403+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:23.404185+0000) 2022-01-31T23:03:23.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:23 smithi171 conmon[51620]: debug 2022-01-31T23:03:23.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:23.579398+0000) 2022-01-31T23:03:24.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:24 smithi167 conmon[49112]: debug 2022-01-31T23:03:24.033+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:24.035403+0000) 2022-01-31T23:03:24.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:23 smithi167 conmon[60316]: debug 2022-01-31T23:03:23.962+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:23.964158+0000) 2022-01-31T23:03:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:24 smithi171 conmon[46715]: debug 2022-01-31T23:03:24.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:24.199906+0000) 2022-01-31T23:03:24.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:23 smithi171 conmon[35325]: debug 2022-01-31T23:03:23.954+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:03:24.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:24 smithi167 conmon[54076]: debug 2022-01-31T23:03:24.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:24.205017+0000) 2022-01-31T23:03:24.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:24 smithi171 conmon[41853]: debug 2022-01-31T23:03:24.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:24.404373+0000) 2022-01-31T23:03:24.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:24 smithi171 conmon[51620]: debug 2022-01-31T23:03:24.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:24.579571+0000) 2022-01-31T23:03:25.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:25 smithi167 conmon[49112]: debug 2022-01-31T23:03:25.034+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:25.035611+0000) 2022-01-31T23:03:25.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:24 smithi167 conmon[60316]: debug 2022-01-31T23:03:24.963+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:24.964330+0000) 2022-01-31T23:03:25.539 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:25 smithi167 conmon[54076]: debug 2022-01-31T23:03:25.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:25.205202+0000) 2022-01-31T23:03:25.562 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:25 smithi171 conmon[41853]: debug 2022-01-31T23:03:25.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:25.404565+0000) 2022-01-31T23:03:25.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:25 smithi171 conmon[46715]: debug 2022-01-31T23:03:25.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:25.200130+0000) 2022-01-31T23:03:25.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:25 smithi171 conmon[51620]: debug 2022-01-31T23:03:25.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:25.579776+0000) 2022-01-31T23:03:26.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:26 smithi167 conmon[49112]: debug 2022-01-31T23:03:26.034+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:26.035790+0000) 2022-01-31T23:03:26.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:25 smithi167 conmon[60316]: debug 2022-01-31T23:03:25.963+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:25.964490+0000) 2022-01-31T23:03:26.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:26 smithi167 conmon[54076]: debug 2022-01-31T23:03:26.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:26.205378+0000) 2022-01-31T23:03:26.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:26 smithi171 conmon[41853]: debug 2022-01-31T23:03:26.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:26.404761+0000) 2022-01-31T23:03:26.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:26 smithi171 conmon[46715]: debug 2022-01-31T23:03:26.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:26.200340+0000) 2022-01-31T23:03:26.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:26 smithi171 conmon[51620]: debug 2022-01-31T23:03:26.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:26.579932+0000) 2022-01-31T23:03:27.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:27 smithi167 conmon[49112]: debug 2022-01-31T23:03:27.022+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.024387+0000) 2022-01-31T23:03:27.189 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:27 smithi167 conmon[49112]: debug 2022-01-31T23:03:27.034+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.035963+0000) 2022-01-31T23:03:27.190 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:27 smithi167 conmon[54076]: debug 2022-01-31T23:03:27.022+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.024008+0000) 2022-01-31T23:03:27.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:26 smithi167 conmon[60316]: debug 2022-01-31T23:03:26.963+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:26.964701+0000) 2022-01-31T23:03:27.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:27 smithi167 conmon[60316]: debug 2022-01-31T23:03:27.023+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.024511+0000) 2022-01-31T23:03:27.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[35325]: debug 2022-01-31T23:03:27.050+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259111 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:27.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[41853]: debug 2022-01-31T23:03:27.024+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.025476+0000) 2022-01-31T23:03:27.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[46715]: debug 2022-01-31T23:03:27.023+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.023889+0000) 2022-01-31T23:03:27.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[46715]: debug 2022-01-31T23:03:27.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.200531+0000) 2022-01-31T23:03:27.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[51620]: debug 2022-01-31T23:03:27.024+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.024802+0000) 2022-01-31T23:03:27.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:27 smithi167 conmon[54076]: debug 2022-01-31T23:03:27.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.205577+0000) 2022-01-31T23:03:27.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[41853]: debug 2022-01-31T23:03:27.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.404921+0000) 2022-01-31T23:03:27.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:27 smithi171 conmon[51620]: debug 2022-01-31T23:03:27.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.580117+0000) 2022-01-31T23:03:28.188 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:28 smithi167 conmon[49112]: debug 2022-01-31T23:03:28.035+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:28.036117+0000) 2022-01-31T23:03:28.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:27 smithi167 conmon[60316]: debug 2022-01-31T23:03:27.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:27.964888+0000) 2022-01-31T23:03:28.543 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:28 smithi167 conmon[54076]: debug 2022-01-31T23:03:28.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:28.205698+0000) 2022-01-31T23:03:28.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:28 smithi171 conmon[41853]: debug 2022-01-31T23:03:28.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:28.405069+0000) 2022-01-31T23:03:28.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:28 smithi171 conmon[46715]: debug 2022-01-31T23:03:28.199+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:28.200717+0000) 2022-01-31T23:03:28.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:28 smithi171 conmon[51620]: debug 2022-01-31T23:03:28.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:28.580269+0000) 2022-01-31T23:03:29.189 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:29 smithi167 conmon[49112]: debug 2022-01-31T23:03:29.034+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:29.036331+0000) 2022-01-31T23:03:29.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:28 smithi167 conmon[60316]: debug 2022-01-31T23:03:28.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:28.965041+0000) 2022-01-31T23:03:29.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:29 smithi167 conmon[54076]: debug 2022-01-31T23:03:29.204+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:29.205875+0000) 2022-01-31T23:03:29.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:29 smithi171 conmon[41853]: debug 2022-01-31T23:03:29.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:29.405283+0000) 2022-01-31T23:03:29.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:29 smithi171 conmon[46715]: debug 2022-01-31T23:03:29.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:29.200952+0000) 2022-01-31T23:03:29.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:29 smithi171 conmon[51620]: debug 2022-01-31T23:03:29.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:29.580449+0000) 2022-01-31T23:03:30.189 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:30 smithi167 conmon[49112]: debug 2022-01-31T23:03:30.035+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:30.036525+0000) 2022-01-31T23:03:30.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:29 smithi167 conmon[60316]: debug 2022-01-31T23:03:29.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:29.965232+0000) 2022-01-31T23:03:30.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:30 smithi167 conmon[54076]: debug 2022-01-31T23:03:30.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:30.206065+0000) 2022-01-31T23:03:30.563 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:30 smithi171 conmon[46715]: debug 2022-01-31T23:03:30.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:30.201166+0000) 2022-01-31T23:03:30.564 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:30 smithi171 conmon[41853]: debug 2022-01-31T23:03:30.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:30.405464+0000) 2022-01-31T23:03:30.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:30 smithi171 conmon[51620]: debug 2022-01-31T23:03:30.579+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:30.580604+0000) 2022-01-31T23:03:31.189 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:31 smithi167 conmon[49112]: debug 2022-01-31T23:03:31.036+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:31.036745+0000) 2022-01-31T23:03:31.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:30 smithi167 conmon[60316]: debug 2022-01-31T23:03:30.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:30.965410+0000) 2022-01-31T23:03:31.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:31 smithi167 conmon[54076]: debug 2022-01-31T23:03:31.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:31.206268+0000) 2022-01-31T23:03:31.563 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:31 smithi171 conmon[41853]: debug 2022-01-31T23:03:31.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:31.405634+0000) 2022-01-31T23:03:31.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:31 smithi171 conmon[46715]: debug 2022-01-31T23:03:31.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:31.201354+0000) 2022-01-31T23:03:31.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:31 smithi171 conmon[51620]: debug 2022-01-31T23:03:31.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:31.580812+0000) 2022-01-31T23:03:32.189 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:32 smithi167 conmon[54076]: debug 2022-01-31T23:03:32.051+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.053411+0000) 2022-01-31T23:03:32.190 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:32 smithi167 conmon[49112]: debug 2022-01-31T23:03:32.035+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.036899+0000) 2022-01-31T23:03:32.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:32 smithi167 conmon[49112]: debug 2022-01-31T23:03:32.052+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.053727+0000) 2022-01-31T23:03:32.191 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:31 smithi167 conmon[60316]: debug 2022-01-31T23:03:31.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:31.965595+0000) 2022-01-31T23:03:32.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:32 smithi167 conmon[60316]: debug 2022-01-31T23:03:32.051+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.053294+0000) 2022-01-31T23:03:32.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[35325]: debug 2022-01-31T23:03:32.086+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259221 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:32.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[41853]: debug 2022-01-31T23:03:32.054+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.054757+0000) 2022-01-31T23:03:32.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[46715]: debug 2022-01-31T23:03:32.053+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.054071+0000) 2022-01-31T23:03:32.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[46715]: debug 2022-01-31T23:03:32.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.201534+0000) 2022-01-31T23:03:32.343 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[51620]: debug 2022-01-31T23:03:32.053+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.054420+0000) 2022-01-31T23:03:32.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:32 smithi167 conmon[54076]: debug 2022-01-31T23:03:32.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.206488+0000) 2022-01-31T23:03:32.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[41853]: debug 2022-01-31T23:03:32.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.405850+0000) 2022-01-31T23:03:32.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:32 smithi171 conmon[51620]: debug 2022-01-31T23:03:32.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.581013+0000) 2022-01-31T23:03:33.189 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:33 smithi167 conmon[49112]: debug 2022-01-31T23:03:33.036+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:33.037086+0000) 2022-01-31T23:03:33.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:32 smithi167 conmon[60316]: debug 2022-01-31T23:03:32.965+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:32.965783+0000) 2022-01-31T23:03:33.544 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:33 smithi167 conmon[54076]: debug 2022-01-31T23:03:33.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:33.206674+0000) 2022-01-31T23:03:33.564 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:33 smithi171 conmon[41853]: debug 2022-01-31T23:03:33.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:33.405951+0000) 2022-01-31T23:03:33.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:33 smithi171 conmon[46715]: debug 2022-01-31T23:03:33.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:33.201669+0000) 2022-01-31T23:03:33.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:33 smithi171 conmon[51620]: debug 2022-01-31T23:03:33.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:33.581173+0000) 2022-01-31T23:03:34.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:33 smithi167 conmon[60316]: debug 2022-01-31T23:03:33.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:33.965917+0000) 2022-01-31T23:03:34.190 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:34 smithi167 conmon[49112]: debug 2022-01-31T23:03:34.036+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:34.037214+0000) 2022-01-31T23:03:34.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:34 smithi167 conmon[54076]: debug 2022-01-31T23:03:34.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:34.206840+0000) 2022-01-31T23:03:34.564 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:34 smithi171 conmon[41853]: debug 2022-01-31T23:03:34.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:34.406103+0000) 2022-01-31T23:03:34.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:34 smithi171 conmon[46715]: debug 2022-01-31T23:03:34.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:34.201829+0000) 2022-01-31T23:03:34.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:34 smithi171 conmon[51620]: debug 2022-01-31T23:03:34.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:34.581371+0000) 2022-01-31T23:03:35.189 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:34 smithi167 conmon[60316]: debug 2022-01-31T23:03:34.964+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:34.966140+0000) 2022-01-31T23:03:35.190 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:35 smithi167 conmon[49112]: debug 2022-01-31T23:03:35.036+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:35.037339+0000) 2022-01-31T23:03:35.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:35 smithi167 conmon[54076]: debug 2022-01-31T23:03:35.205+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:35.206970+0000) 2022-01-31T23:03:35.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:35 smithi171 conmon[46715]: debug 2022-01-31T23:03:35.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:35.202050+0000) 2022-01-31T23:03:35.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:35 smithi171 conmon[41853]: debug 2022-01-31T23:03:35.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:35.406279+0000) 2022-01-31T23:03:35.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:35 smithi171 conmon[51620]: debug 2022-01-31T23:03:35.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:35.581575+0000) 2022-01-31T23:03:36.190 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:36 smithi167 conmon[49112]: debug 2022-01-31T23:03:36.036+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:36.037552+0000) 2022-01-31T23:03:36.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:35 smithi167 conmon[60316]: debug 2022-01-31T23:03:35.965+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:35.966314+0000) 2022-01-31T23:03:36.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:36 smithi167 conmon[54076]: debug 2022-01-31T23:03:36.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:36.207144+0000) 2022-01-31T23:03:36.564 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:36 smithi171 conmon[46715]: debug 2022-01-31T23:03:36.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:36.202217+0000) 2022-01-31T23:03:36.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:36 smithi171 conmon[41853]: debug 2022-01-31T23:03:36.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:36.406480+0000) 2022-01-31T23:03:36.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:36 smithi171 conmon[51620]: debug 2022-01-31T23:03:36.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:36.581791+0000) 2022-01-31T23:03:37.190 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:37 smithi167 conmon[49112]: debug 2022-01-31T23:03:37.037+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.037733+0000) 2022-01-31T23:03:37.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:37 smithi167 conmon[49112]: debug 2022-01-31T23:03:37.088+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.089628+0000) 2022-01-31T23:03:37.191 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:37 smithi167 conmon[54076]: debug 2022-01-31T23:03:37.087+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.089478+0000) 2022-01-31T23:03:37.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:36 smithi167 conmon[60316]: debug 2022-01-31T23:03:36.965+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:36.966538+0000) 2022-01-31T23:03:37.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:37 smithi167 conmon[60316]: debug 2022-01-31T23:03:37.089+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.090580+0000) 2022-01-31T23:03:37.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[35325]: debug 2022-01-31T23:03:37.114+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259334 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:37.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[41853]: debug 2022-01-31T23:03:37.089+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.091255+0000) 2022-01-31T23:03:37.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[51620]: debug 2022-01-31T23:03:37.089+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.090843+0000) 2022-01-31T23:03:37.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[46715]: debug 2022-01-31T23:03:37.088+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.090550+0000) 2022-01-31T23:03:37.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[46715]: debug 2022-01-31T23:03:37.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.202421+0000) 2022-01-31T23:03:37.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:37 smithi167 conmon[54076]: debug 2022-01-31T23:03:37.206+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.207334+0000) 2022-01-31T23:03:37.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[41853]: debug 2022-01-31T23:03:37.404+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.406606+0000) 2022-01-31T23:03:37.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:37 smithi171 conmon[51620]: debug 2022-01-31T23:03:37.580+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.581951+0000) 2022-01-31T23:03:38.190 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:38 smithi167 conmon[49112]: debug 2022-01-31T23:03:38.037+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:38.037929+0000) 2022-01-31T23:03:38.191 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:37 smithi167 conmon[60316]: debug 2022-01-31T23:03:37.966+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:37.966725+0000) 2022-01-31T23:03:38.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:38 smithi167 conmon[54076]: debug 2022-01-31T23:03:38.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:38.207529+0000) 2022-01-31T23:03:38.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:38 smithi171 conmon[46715]: debug 2022-01-31T23:03:38.200+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:38.202591+0000) 2022-01-31T23:03:38.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:38 smithi171 conmon[41853]: debug 2022-01-31T23:03:38.405+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:38.406746+0000) 2022-01-31T23:03:38.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:38 smithi171 conmon[51620]: debug 2022-01-31T23:03:38.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:38.582080+0000) 2022-01-31T23:03:39.190 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:38 smithi167 conmon[60316]: debug 2022-01-31T23:03:38.966+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:38.966870+0000) 2022-01-31T23:03:39.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:39 smithi167 conmon[49112]: debug 2022-01-31T23:03:39.037+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:39.038082+0000) 2022-01-31T23:03:39.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:38 smithi171 conmon[35325]: debug 2022-01-31T23:03:38.954+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:03:39.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:39 smithi171 conmon[46715]: debug 2022-01-31T23:03:39.201+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:39.202816+0000) 2022-01-31T23:03:39.545 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:39 smithi167 conmon[54076]: debug 2022-01-31T23:03:39.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:39.207733+0000) 2022-01-31T23:03:39.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:39 smithi171 conmon[41853]: debug 2022-01-31T23:03:39.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:39.406971+0000) 2022-01-31T23:03:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:39 smithi171 conmon[51620]: debug 2022-01-31T23:03:39.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:39.582234+0000) 2022-01-31T23:03:40.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:40 smithi167 conmon[49112]: debug 2022-01-31T23:03:40.037+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:40.038321+0000) 2022-01-31T23:03:40.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:39 smithi167 conmon[60316]: debug 2022-01-31T23:03:39.966+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:39.967052+0000) 2022-01-31T23:03:40.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:40 smithi167 conmon[54076]: debug 2022-01-31T23:03:40.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:40.207912+0000) 2022-01-31T23:03:40.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:40 smithi171 conmon[46715]: debug 2022-01-31T23:03:40.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:40.203049+0000) 2022-01-31T23:03:40.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:40 smithi171 conmon[41853]: debug 2022-01-31T23:03:40.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:40.407132+0000) 2022-01-31T23:03:40.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:40 smithi171 conmon[51620]: debug 2022-01-31T23:03:40.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:40.582387+0000) 2022-01-31T23:03:41.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:41 smithi167 conmon[49112]: debug 2022-01-31T23:03:41.038+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:41.038524+0000) 2022-01-31T23:03:41.191 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:40 smithi167 conmon[60316]: debug 2022-01-31T23:03:40.966+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:40.967267+0000) 2022-01-31T23:03:41.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:41 smithi167 conmon[54076]: debug 2022-01-31T23:03:41.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:41.208100+0000) 2022-01-31T23:03:41.565 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:41 smithi171 conmon[41853]: debug 2022-01-31T23:03:41.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:41.407292+0000) 2022-01-31T23:03:41.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:41 smithi171 conmon[46715]: debug 2022-01-31T23:03:41.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:41.203276+0000) 2022-01-31T23:03:41.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:41 smithi171 conmon[51620]: debug 2022-01-31T23:03:41.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:41.582542+0000) 2022-01-31T23:03:42.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:42 smithi167 conmon[49112]: debug 2022-01-31T23:03:42.038+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.038706+0000) 2022-01-31T23:03:42.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:42 smithi167 conmon[49112]: debug 2022-01-31T23:03:42.119+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.119742+0000) 2022-01-31T23:03:42.192 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:42 smithi167 conmon[54076]: debug 2022-01-31T23:03:42.119+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.119608+0000) 2022-01-31T23:03:42.193 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:41 smithi167 conmon[60316]: debug 2022-01-31T23:03:41.967+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:41.967465+0000) 2022-01-31T23:03:42.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:42 smithi167 conmon[60316]: debug 2022-01-31T23:03:42.118+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.118884+0000) 2022-01-31T23:03:42.390 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[35325]: debug 2022-01-31T23:03:42.143+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259445 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:42.391 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[41853]: debug 2022-01-31T23:03:42.117+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.119202+0000) 2022-01-31T23:03:42.392 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[51620]: debug 2022-01-31T23:03:42.116+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.118276+0000) 2022-01-31T23:03:42.392 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[46715]: debug 2022-01-31T23:03:42.117+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.119293+0000) 2022-01-31T23:03:42.392 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[46715]: debug 2022-01-31T23:03:42.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.203485+0000) 2022-01-31T23:03:42.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:42 smithi167 conmon[54076]: debug 2022-01-31T23:03:42.207+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.208286+0000) 2022-01-31T23:03:42.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[41853]: debug 2022-01-31T23:03:42.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.407471+0000) 2022-01-31T23:03:42.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:42 smithi171 conmon[51620]: debug 2022-01-31T23:03:42.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.582715+0000) 2022-01-31T23:03:43.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:43 smithi167 conmon[49112]: debug 2022-01-31T23:03:43.038+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:43.038857+0000) 2022-01-31T23:03:43.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:42 smithi167 conmon[60316]: debug 2022-01-31T23:03:42.967+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:42.967591+0000) 2022-01-31T23:03:43.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:43 smithi167 conmon[54076]: debug 2022-01-31T23:03:43.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:43.208436+0000) 2022-01-31T23:03:43.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:43 smithi171 conmon[46715]: debug 2022-01-31T23:03:43.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:43.203599+0000) 2022-01-31T23:03:43.566 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:43 smithi171 conmon[41853]: debug 2022-01-31T23:03:43.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:43.407619+0000) 2022-01-31T23:03:43.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:43 smithi171 conmon[51620]: debug 2022-01-31T23:03:43.581+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:43.582866+0000) 2022-01-31T23:03:44.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:44 smithi167 conmon[49112]: debug 2022-01-31T23:03:44.038+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:44.038986+0000) 2022-01-31T23:03:44.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:43 smithi167 conmon[60316]: debug 2022-01-31T23:03:43.967+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:43.967737+0000) 2022-01-31T23:03:44.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:44 smithi167 conmon[54076]: debug 2022-01-31T23:03:44.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:44.208596+0000) 2022-01-31T23:03:44.565 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:44 smithi171 conmon[46715]: debug 2022-01-31T23:03:44.202+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:44.203793+0000) 2022-01-31T23:03:44.566 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:44 smithi171 conmon[41853]: debug 2022-01-31T23:03:44.406+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:44.407823+0000) 2022-01-31T23:03:44.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:44 smithi171 conmon[51620]: debug 2022-01-31T23:03:44.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:44.583047+0000) 2022-01-31T23:03:45.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:45 smithi167 conmon[49112]: debug 2022-01-31T23:03:45.038+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:45.039166+0000) 2022-01-31T23:03:45.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:44 smithi167 conmon[60316]: debug 2022-01-31T23:03:44.967+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:44.967968+0000) 2022-01-31T23:03:45.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:45 smithi167 conmon[54076]: debug 2022-01-31T23:03:45.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:45.208750+0000) 2022-01-31T23:03:45.566 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:45 smithi171 conmon[41853]: debug 2022-01-31T23:03:45.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:45.408001+0000) 2022-01-31T23:03:45.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:45 smithi171 conmon[46715]: debug 2022-01-31T23:03:45.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:45.204008+0000) 2022-01-31T23:03:45.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:45 smithi171 conmon[51620]: debug 2022-01-31T23:03:45.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:45.583167+0000) 2022-01-31T23:03:46.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:46 smithi167 conmon[49112]: debug 2022-01-31T23:03:46.038+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:46.039370+0000) 2022-01-31T23:03:46.192 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:45 smithi167 conmon[60316]: debug 2022-01-31T23:03:45.967+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:45.968157+0000) 2022-01-31T23:03:46.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:46 smithi167 conmon[54076]: debug 2022-01-31T23:03:46.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:46.208911+0000) 2022-01-31T23:03:46.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:46 smithi171 conmon[46715]: debug 2022-01-31T23:03:46.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:46.204205+0000) 2022-01-31T23:03:46.566 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:46 smithi171 conmon[41853]: debug 2022-01-31T23:03:46.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:46.408186+0000) 2022-01-31T23:03:46.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:46 smithi171 conmon[51620]: debug 2022-01-31T23:03:46.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:46.583324+0000) 2022-01-31T23:03:47.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:47 smithi167 conmon[49112]: debug 2022-01-31T23:03:47.039+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.039554+0000) 2022-01-31T23:03:47.193 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:47 smithi167 conmon[49112]: debug 2022-01-31T23:03:47.148+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.148536+0000) 2022-01-31T23:03:47.193 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:47 smithi167 conmon[54076]: debug 2022-01-31T23:03:47.147+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.148025+0000) 2022-01-31T23:03:47.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:46 smithi167 conmon[60316]: debug 2022-01-31T23:03:46.967+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:46.968358+0000) 2022-01-31T23:03:47.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:47 smithi167 conmon[60316]: debug 2022-01-31T23:03:47.147+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.148381+0000) 2022-01-31T23:03:47.391 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[35325]: debug 2022-01-31T23:03:47.173+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259554 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:47.392 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[41853]: debug 2022-01-31T23:03:47.146+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.147808+0000) 2022-01-31T23:03:47.393 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[51620]: debug 2022-01-31T23:03:47.145+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.147021+0000) 2022-01-31T23:03:47.393 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[46715]: debug 2022-01-31T23:03:47.146+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.148259+0000) 2022-01-31T23:03:47.394 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[46715]: debug 2022-01-31T23:03:47.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.204409+0000) 2022-01-31T23:03:47.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:47 smithi167 conmon[54076]: debug 2022-01-31T23:03:47.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.209102+0000) 2022-01-31T23:03:47.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[41853]: debug 2022-01-31T23:03:47.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.408386+0000) 2022-01-31T23:03:47.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:47 smithi171 conmon[51620]: debug 2022-01-31T23:03:47.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.583503+0000) 2022-01-31T23:03:48.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:48 smithi167 conmon[49112]: debug 2022-01-31T23:03:48.039+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:48.039706+0000) 2022-01-31T23:03:48.193 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:47 smithi167 conmon[60316]: debug 2022-01-31T23:03:47.968+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:47.968574+0000) 2022-01-31T23:03:48.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:48 smithi167 conmon[54076]: debug 2022-01-31T23:03:48.208+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:48.209204+0000) 2022-01-31T23:03:48.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:48 smithi171 conmon[46715]: debug 2022-01-31T23:03:48.203+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:48.204579+0000) 2022-01-31T23:03:48.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:48 smithi171 conmon[41853]: debug 2022-01-31T23:03:48.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:48.408536+0000) 2022-01-31T23:03:48.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:48 smithi171 conmon[51620]: debug 2022-01-31T23:03:48.582+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:48.583641+0000) 2022-01-31T23:03:49.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:49 smithi167 conmon[49112]: debug 2022-01-31T23:03:49.039+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:49.039849+0000) 2022-01-31T23:03:49.193 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:48 smithi167 conmon[60316]: debug 2022-01-31T23:03:48.968+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:48.968730+0000) 2022-01-31T23:03:49.547 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:49 smithi167 conmon[54076]: debug 2022-01-31T23:03:49.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:49.209368+0000) 2022-01-31T23:03:49.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:49 smithi171 conmon[46715]: debug 2022-01-31T23:03:49.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:49.204805+0000) 2022-01-31T23:03:49.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:49 smithi171 conmon[41853]: debug 2022-01-31T23:03:49.407+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:49.408729+0000) 2022-01-31T23:03:49.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:49 smithi171 conmon[51620]: debug 2022-01-31T23:03:49.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:49.583795+0000) 2022-01-31T23:03:50.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:50 smithi167 conmon[49112]: debug 2022-01-31T23:03:50.039+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:50.040059+0000) 2022-01-31T23:03:50.193 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:49 smithi167 conmon[60316]: debug 2022-01-31T23:03:49.968+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:49.968916+0000) 2022-01-31T23:03:50.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:50 smithi167 conmon[54076]: debug 2022-01-31T23:03:50.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:50.209551+0000) 2022-01-31T23:03:50.566 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:50 smithi171 conmon[46715]: debug 2022-01-31T23:03:50.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:50.205013+0000) 2022-01-31T23:03:50.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:50 smithi171 conmon[41853]: debug 2022-01-31T23:03:50.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:50.408945+0000) 2022-01-31T23:03:50.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:50 smithi171 conmon[51620]: debug 2022-01-31T23:03:50.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:50.584007+0000) 2022-01-31T23:03:51.192 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:51 smithi167 conmon[49112]: debug 2022-01-31T23:03:51.039+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:51.040231+0000) 2022-01-31T23:03:51.193 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:50 smithi167 conmon[60316]: debug 2022-01-31T23:03:50.968+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:50.969105+0000) 2022-01-31T23:03:51.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:51 smithi167 conmon[54076]: debug 2022-01-31T23:03:51.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:51.209708+0000) 2022-01-31T23:03:51.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:51 smithi171 conmon[46715]: debug 2022-01-31T23:03:51.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:51.205223+0000) 2022-01-31T23:03:51.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:51 smithi171 conmon[41853]: debug 2022-01-31T23:03:51.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:51.409109+0000) 2022-01-31T23:03:51.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:51 smithi171 conmon[51620]: debug 2022-01-31T23:03:51.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:51.584177+0000) 2022-01-31T23:03:52.194 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:52 smithi167 conmon[49112]: debug 2022-01-31T23:03:52.040+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.040363+0000) 2022-01-31T23:03:52.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:52 smithi167 conmon[49112]: debug 2022-01-31T23:03:52.177+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.177774+0000) 2022-01-31T23:03:52.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:51 smithi167 conmon[60316]: debug 2022-01-31T23:03:51.969+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:51.969290+0000) 2022-01-31T23:03:52.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:52 smithi167 conmon[60316]: debug 2022-01-31T23:03:52.177+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.178214+0000) 2022-01-31T23:03:52.196 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:52 smithi167 conmon[54076]: debug 2022-01-31T23:03:52.177+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.177989+0000) 2022-01-31T23:03:52.196 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:52 smithi167 conmon[54076]: debug 2022-01-31T23:03:52.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.209867+0000) 2022-01-31T23:03:52.567 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[35325]: debug 2022-01-31T23:03:52.204+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259665 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:52.568 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[51620]: debug 2022-01-31T23:03:52.176+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.176885+0000) 2022-01-31T23:03:52.568 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[41853]: debug 2022-01-31T23:03:52.176+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.177024+0000) 2022-01-31T23:03:52.569 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[41853]: debug 2022-01-31T23:03:52.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.409259+0000) 2022-01-31T23:03:52.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[46715]: debug 2022-01-31T23:03:52.177+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.177749+0000) 2022-01-31T23:03:52.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[46715]: debug 2022-01-31T23:03:52.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.205409+0000) 2022-01-31T23:03:52.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:52 smithi171 conmon[51620]: debug 2022-01-31T23:03:52.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.584371+0000) 2022-01-31T23:03:53.193 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:53 smithi167 conmon[49112]: debug 2022-01-31T23:03:53.040+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:53.040524+0000) 2022-01-31T23:03:53.193 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:52 smithi167 conmon[60316]: debug 2022-01-31T23:03:52.969+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:52.969454+0000) 2022-01-31T23:03:53.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:53 smithi167 conmon[54076]: debug 2022-01-31T23:03:53.209+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:53.210033+0000) 2022-01-31T23:03:53.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:53 smithi171 conmon[41853]: debug 2022-01-31T23:03:53.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:53.409400+0000) 2022-01-31T23:03:53.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:53 smithi171 conmon[46715]: debug 2022-01-31T23:03:53.204+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:53.205587+0000) 2022-01-31T23:03:53.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:53 smithi171 conmon[51620]: debug 2022-01-31T23:03:53.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:53.584524+0000) 2022-01-31T23:03:54.193 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:54 smithi167 conmon[49112]: debug 2022-01-31T23:03:54.040+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:54.040690+0000) 2022-01-31T23:03:54.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:53 smithi167 conmon[60316]: debug 2022-01-31T23:03:53.969+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:53.969608+0000) 2022-01-31T23:03:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:54 smithi171 conmon[46715]: debug 2022-01-31T23:03:54.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:54.205795+0000) 2022-01-31T23:03:54.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:53 smithi171 conmon[35325]: debug 2022-01-31T23:03:53.955+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:03:54.548 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:54 smithi167 conmon[54076]: debug 2022-01-31T23:03:54.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:54.210243+0000) 2022-01-31T23:03:54.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:54 smithi171 conmon[41853]: debug 2022-01-31T23:03:54.408+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:54.409590+0000) 2022-01-31T23:03:54.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:54 smithi171 conmon[51620]: debug 2022-01-31T23:03:54.583+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:54.584719+0000) 2022-01-31T23:03:55.193 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:55 smithi167 conmon[49112]: debug 2022-01-31T23:03:55.040+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:55.040847+0000) 2022-01-31T23:03:55.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:54 smithi167 conmon[60316]: debug 2022-01-31T23:03:54.969+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:54.969802+0000) 2022-01-31T23:03:55.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:55 smithi167 conmon[54076]: debug 2022-01-31T23:03:55.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:55.210466+0000) 2022-01-31T23:03:55.567 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:55 smithi171 conmon[41853]: debug 2022-01-31T23:03:55.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:55.409786+0000) 2022-01-31T23:03:55.568 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:55 smithi171 conmon[46715]: debug 2022-01-31T23:03:55.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:55.205950+0000) 2022-01-31T23:03:55.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:55 smithi171 conmon[51620]: debug 2022-01-31T23:03:55.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:55.584931+0000) 2022-01-31T23:03:56.193 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:56 smithi167 conmon[49112]: debug 2022-01-31T23:03:56.040+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:56.041010+0000) 2022-01-31T23:03:56.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:55 smithi167 conmon[60316]: debug 2022-01-31T23:03:55.969+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:55.969972+0000) 2022-01-31T23:03:56.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:56 smithi167 conmon[54076]: debug 2022-01-31T23:03:56.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:56.210697+0000) 2022-01-31T23:03:56.567 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:56 smithi171 conmon[46715]: debug 2022-01-31T23:03:56.205+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:56.206154+0000) 2022-01-31T23:03:56.568 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:56 smithi171 conmon[41853]: debug 2022-01-31T23:03:56.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:56.409961+0000) 2022-01-31T23:03:56.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:56 smithi171 conmon[51620]: debug 2022-01-31T23:03:56.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:56.585121+0000) 2022-01-31T23:03:57.191 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:57 smithi167 conmon[49112]: debug 2022-01-31T23:03:57.040+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.041191+0000) 2022-01-31T23:03:57.191 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:56 smithi167 conmon[60316]: debug 2022-01-31T23:03:56.969+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:56.970162+0000) 2022-01-31T23:03:57.442 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:57 smithi167 conmon[49112]: debug 2022-01-31T23:03:57.208+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.208552+0000) 2022-01-31T23:03:57.443 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:57 smithi167 conmon[60316]: debug 2022-01-31T23:03:57.207+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.208004+0000) 2022-01-31T23:03:57.443 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:57 smithi167 conmon[54076]: debug 2022-01-31T23:03:57.208+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.209046+0000) 2022-01-31T23:03:57.444 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:57 smithi167 conmon[54076]: debug 2022-01-31T23:03:57.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.210824+0000) 2022-01-31T23:03:57.568 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:03:57 smithi171 conmon[35325]: debug 2022-01-31T23:03:57.234+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259778 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:03:57.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:03:57 smithi171 conmon[46715]: debug 2022-01-31T23:03:57.208+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.209636+0000) 2022-01-31T23:03:57.569 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:57 smithi171 conmon[51620]: debug 2022-01-31T23:03:57.207+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.208425+0000) 2022-01-31T23:03:57.570 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:57 smithi171 conmon[41853]: debug 2022-01-31T23:03:57.207+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.208589+0000) 2022-01-31T23:03:57.570 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:57 smithi171 conmon[41853]: debug 2022-01-31T23:03:57.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.410139+0000) 2022-01-31T23:03:57.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:57 smithi171 conmon[51620]: debug 2022-01-31T23:03:57.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.585334+0000) 2022-01-31T23:03:58.193 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:58 smithi167 conmon[49112]: debug 2022-01-31T23:03:58.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:58.041373+0000) 2022-01-31T23:03:58.194 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:57 smithi167 conmon[60316]: debug 2022-01-31T23:03:57.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:57.970355+0000) 2022-01-31T23:03:58.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:58 smithi167 conmon[54076]: debug 2022-01-31T23:03:58.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:58.210985+0000) 2022-01-31T23:03:58.792 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:58 smithi171 conmon[41853]: debug 2022-01-31T23:03:58.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:58.410292+0000) 2022-01-31T23:03:58.793 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:58 smithi171 conmon[51620]: debug 2022-01-31T23:03:58.584+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:58.585496+0000) 2022-01-31T23:03:59.194 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:03:59 smithi167 conmon[49112]: debug 2022-01-31T23:03:59.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:59.041543+0000) 2022-01-31T23:03:59.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:58 smithi167 conmon[60316]: debug 2022-01-31T23:03:58.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:58.970521+0000) 2022-01-31T23:03:59.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:03:59 smithi167 conmon[54076]: debug 2022-01-31T23:03:59.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:59.211181+0000) 2022-01-31T23:03:59.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:03:59 smithi171 conmon[41853]: debug 2022-01-31T23:03:59.409+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:59.410485+0000) 2022-01-31T23:03:59.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:03:59 smithi171 conmon[51620]: debug 2022-01-31T23:03:59.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:59.585706+0000) 2022-01-31T23:04:00.194 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:00 smithi167 conmon[49112]: debug 2022-01-31T23:04:00.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:00.041750+0000) 2022-01-31T23:04:00.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:03:59 smithi167 conmon[60316]: debug 2022-01-31T23:03:59.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:03:59.970722+0000) 2022-01-31T23:04:00.549 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:00 smithi167 conmon[54076]: debug 2022-01-31T23:04:00.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:00.211359+0000) 2022-01-31T23:04:00.568 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:00 smithi171 conmon[41853]: debug 2022-01-31T23:04:00.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:00.410659+0000) 2022-01-31T23:04:00.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:00 smithi171 conmon[51620]: debug 2022-01-31T23:04:00.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:00.585885+0000) 2022-01-31T23:04:01.194 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:01 smithi167 conmon[49112]: debug 2022-01-31T23:04:01.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:01.041925+0000) 2022-01-31T23:04:01.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:00 smithi167 conmon[60316]: debug 2022-01-31T23:04:00.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:00.970903+0000) 2022-01-31T23:04:01.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:01 smithi167 conmon[54076]: debug 2022-01-31T23:04:01.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:01.211598+0000) 2022-01-31T23:04:01.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:01 smithi171 conmon[41853]: debug 2022-01-31T23:04:01.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:01.410813+0000) 2022-01-31T23:04:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:01 smithi171 conmon[51620]: debug 2022-01-31T23:04:01.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:01.586020+0000) 2022-01-31T23:04:02.194 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:02 smithi167 conmon[49112]: debug 2022-01-31T23:04:02.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.042077+0000) 2022-01-31T23:04:02.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:01 smithi167 conmon[60316]: debug 2022-01-31T23:04:01.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:01.971124+0000) 2022-01-31T23:04:02.550 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:02 smithi167 conmon[49112]: debug 2022-01-31T23:04:02.238+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.238815+0000) 2022-01-31T23:04:02.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:02 smithi167 conmon[54076]: debug 2022-01-31T23:04:02.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.211803+0000) 2022-01-31T23:04:02.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:02 smithi167 conmon[54076]: debug 2022-01-31T23:04:02.237+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.238014+0000) 2022-01-31T23:04:02.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:02 smithi167 conmon[60316]: debug 2022-01-31T23:04:02.237+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.237622+0000) 2022-01-31T23:04:02.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:02 smithi171 conmon[35325]: debug 2022-01-31T23:04:02.265+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259888 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:02.569 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:02 smithi171 conmon[46715]: debug 2022-01-31T23:04:02.237+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.238417+0000) 2022-01-31T23:04:02.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:02 smithi171 conmon[51620]: debug 2022-01-31T23:04:02.236+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.237474+0000) 2022-01-31T23:04:02.570 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:02 smithi171 conmon[41853]: debug 2022-01-31T23:04:02.237+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.237941+0000) 2022-01-31T23:04:02.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:02 smithi171 conmon[41853]: debug 2022-01-31T23:04:02.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.410970+0000) 2022-01-31T23:04:02.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:02 smithi171 conmon[51620]: debug 2022-01-31T23:04:02.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.586131+0000) 2022-01-31T23:04:03.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:03 smithi167 conmon[49112]: debug 2022-01-31T23:04:03.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:03.042232+0000) 2022-01-31T23:04:03.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:02 smithi167 conmon[60316]: debug 2022-01-31T23:04:02.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:02.971307+0000) 2022-01-31T23:04:03.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:03 smithi167 conmon[54076]: debug 2022-01-31T23:04:03.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:03.212009+0000) 2022-01-31T23:04:03.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:03 smithi171 conmon[41853]: debug 2022-01-31T23:04:03.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:03.411133+0000) 2022-01-31T23:04:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:03 smithi171 conmon[51620]: debug 2022-01-31T23:04:03.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:03.586291+0000) 2022-01-31T23:04:04.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:04 smithi167 conmon[49112]: debug 2022-01-31T23:04:04.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:04.042378+0000) 2022-01-31T23:04:04.195 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:03 smithi167 conmon[60316]: debug 2022-01-31T23:04:03.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:03.971451+0000) 2022-01-31T23:04:04.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:03 smithi171 conmon[46715]: debug 2022-01-31T23:04:03.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:03.956743+0000) 2022-01-31T23:04:04.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:04 smithi167 conmon[54076]: debug 2022-01-31T23:04:04.210+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:04.212131+0000) 2022-01-31T23:04:04.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:04 smithi171 conmon[41853]: debug 2022-01-31T23:04:04.410+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:04.411289+0000) 2022-01-31T23:04:04.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:04 smithi171 conmon[51620]: debug 2022-01-31T23:04:04.585+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:04.586496+0000) 2022-01-31T23:04:05.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:05 smithi167 conmon[49112]: debug 2022-01-31T23:04:05.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:05.042579+0000) 2022-01-31T23:04:05.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:04 smithi167 conmon[60316]: debug 2022-01-31T23:04:04.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:04.971686+0000) 2022-01-31T23:04:05.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:04 smithi171 conmon[46715]: debug 2022-01-31T23:04:04.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:04.956910+0000) 2022-01-31T23:04:05.550 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:05 smithi167 conmon[54076]: debug 2022-01-31T23:04:05.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:05.212327+0000) 2022-01-31T23:04:05.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:05 smithi171 conmon[41853]: debug 2022-01-31T23:04:05.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:05.411529+0000) 2022-01-31T23:04:05.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:05 smithi171 conmon[51620]: debug 2022-01-31T23:04:05.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:05.586690+0000) 2022-01-31T23:04:06.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:06 smithi167 conmon[49112]: debug 2022-01-31T23:04:06.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:06.042780+0000) 2022-01-31T23:04:06.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:05 smithi167 conmon[60316]: debug 2022-01-31T23:04:05.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:05.971870+0000) 2022-01-31T23:04:06.321 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:05 smithi171 conmon[46715]: debug 2022-01-31T23:04:05.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:05.957058+0000) 2022-01-31T23:04:06.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:06 smithi167 conmon[54076]: debug 2022-01-31T23:04:06.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:06.212557+0000) 2022-01-31T23:04:06.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:06 smithi171 conmon[41853]: debug 2022-01-31T23:04:06.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:06.411678+0000) 2022-01-31T23:04:06.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:06 smithi171 conmon[51620]: debug 2022-01-31T23:04:06.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:06.586842+0000) 2022-01-31T23:04:07.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:07 smithi167 conmon[49112]: debug 2022-01-31T23:04:07.042+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.042961+0000) 2022-01-31T23:04:07.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:06 smithi167 conmon[60316]: debug 2022-01-31T23:04:06.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:06.972033+0000) 2022-01-31T23:04:07.251 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:06 smithi171 conmon[46715]: debug 2022-01-31T23:04:06.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:06.957213+0000) 2022-01-31T23:04:07.551 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:07 smithi167 conmon[49112]: debug 2022-01-31T23:04:07.268+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.269513+0000) 2022-01-31T23:04:07.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:07 smithi167 conmon[60316]: debug 2022-01-31T23:04:07.266+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.268072+0000) 2022-01-31T23:04:07.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:07 smithi167 conmon[54076]: debug 2022-01-31T23:04:07.211+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.212738+0000) 2022-01-31T23:04:07.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:07 smithi167 conmon[54076]: debug 2022-01-31T23:04:07.267+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.268932+0000) 2022-01-31T23:04:07.569 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[35325]: debug 2022-01-31T23:04:07.295+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 259991 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:07.570 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[46715]: debug 2022-01-31T23:04:07.268+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.269724+0000) 2022-01-31T23:04:07.570 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[51620]: debug 2022-01-31T23:04:07.268+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.268935+0000) 2022-01-31T23:04:07.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[41853]: debug 2022-01-31T23:04:07.267+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.268330+0000) 2022-01-31T23:04:07.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[41853]: debug 2022-01-31T23:04:07.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.411938+0000) 2022-01-31T23:04:07.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[51620]: debug 2022-01-31T23:04:07.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.587024+0000) 2022-01-31T23:04:08.195 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:08 smithi167 conmon[49112]: debug 2022-01-31T23:04:08.041+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:08.043137+0000) 2022-01-31T23:04:08.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:07 smithi167 conmon[60316]: debug 2022-01-31T23:04:07.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.972222+0000) 2022-01-31T23:04:08.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:07 smithi171 conmon[46715]: debug 2022-01-31T23:04:07.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:07.957393+0000) 2022-01-31T23:04:08.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:08 smithi167 conmon[54076]: debug 2022-01-31T23:04:08.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:08.212949+0000) 2022-01-31T23:04:08.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:08 smithi171 conmon[51620]: debug 2022-01-31T23:04:08.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:08.587184+0000) 2022-01-31T23:04:08.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:08 smithi171 conmon[41853]: debug 2022-01-31T23:04:08.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:08.412087+0000) 2022-01-31T23:04:09.196 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:09 smithi167 conmon[49112]: debug 2022-01-31T23:04:09.042+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:09.043295+0000) 2022-01-31T23:04:09.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:08 smithi167 conmon[60316]: debug 2022-01-31T23:04:08.970+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:08.972389+0000) 2022-01-31T23:04:09.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:08 smithi171 conmon[46715]: debug 2022-01-31T23:04:08.956+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:08.957505+0000) 2022-01-31T23:04:09.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:08 smithi171 conmon[35325]: debug 2022-01-31T23:04:08.956+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:04:09.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:09 smithi167 conmon[54076]: debug 2022-01-31T23:04:09.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:09.213105+0000) 2022-01-31T23:04:09.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:09 smithi171 conmon[41853]: debug 2022-01-31T23:04:09.411+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:09.412284+0000) 2022-01-31T23:04:09.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:09 smithi171 conmon[51620]: debug 2022-01-31T23:04:09.586+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:09.587374+0000) 2022-01-31T23:04:10.196 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:10 smithi167 conmon[49112]: debug 2022-01-31T23:04:10.042+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:10.043505+0000) 2022-01-31T23:04:10.196 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:09 smithi167 conmon[60316]: debug 2022-01-31T23:04:09.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:09.972572+0000) 2022-01-31T23:04:10.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:09 smithi171 conmon[46715]: debug 2022-01-31T23:04:09.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:09.957672+0000) 2022-01-31T23:04:10.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:10 smithi167 conmon[54076]: debug 2022-01-31T23:04:10.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:10.213284+0000) 2022-01-31T23:04:10.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:10 smithi171 conmon[51620]: debug 2022-01-31T23:04:10.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:10.587553+0000) 2022-01-31T23:04:10.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:10 smithi171 conmon[41853]: debug 2022-01-31T23:04:10.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:10.412461+0000) 2022-01-31T23:04:11.196 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:11 smithi167 conmon[49112]: debug 2022-01-31T23:04:11.042+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:11.043638+0000) 2022-01-31T23:04:11.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:10 smithi167 conmon[60316]: debug 2022-01-31T23:04:10.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:10.972750+0000) 2022-01-31T23:04:11.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:10 smithi171 conmon[46715]: debug 2022-01-31T23:04:10.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:10.957856+0000) 2022-01-31T23:04:11.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:11 smithi167 conmon[54076]: debug 2022-01-31T23:04:11.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:11.213466+0000) 2022-01-31T23:04:11.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:11 smithi171 conmon[41853]: debug 2022-01-31T23:04:11.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:11.412584+0000) 2022-01-31T23:04:11.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:11 smithi171 conmon[51620]: debug 2022-01-31T23:04:11.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:11.587767+0000) 2022-01-31T23:04:12.196 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:12 smithi167 conmon[49112]: debug 2022-01-31T23:04:12.042+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.043826+0000) 2022-01-31T23:04:12.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:11 smithi167 conmon[60316]: debug 2022-01-31T23:04:11.972+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:11.972908+0000) 2022-01-31T23:04:12.281 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:11 smithi171 conmon[46715]: debug 2022-01-31T23:04:11.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:11.958042+0000) 2022-01-31T23:04:12.457 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[41853]: debug 2022-01-31T23:04:12.297+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.298321+0000) 2022-01-31T23:04:12.552 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:12 smithi167 conmon[49112]: debug 2022-01-31T23:04:12.298+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.300030+0000) 2022-01-31T23:04:12.552 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:12 smithi167 conmon[60316]: debug 2022-01-31T23:04:12.297+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.299195+0000) 2022-01-31T23:04:12.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:12 smithi167 conmon[54076]: debug 2022-01-31T23:04:12.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.213618+0000) 2022-01-31T23:04:12.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:12 smithi167 conmon[54076]: debug 2022-01-31T23:04:12.298+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.300133+0000) 2022-01-31T23:04:12.570 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[35325]: debug 2022-01-31T23:04:12.326+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260105 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:12.571 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[41853]: debug 2022-01-31T23:04:12.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.412767+0000) 2022-01-31T23:04:12.571 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[46715]: debug 2022-01-31T23:04:12.299+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.300106+0000) 2022-01-31T23:04:12.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[51620]: debug 2022-01-31T23:04:12.298+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.299527+0000) 2022-01-31T23:04:12.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[51620]: debug 2022-01-31T23:04:12.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.587960+0000) 2022-01-31T23:04:13.196 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:13 smithi167 conmon[49112]: debug 2022-01-31T23:04:13.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:13.043992+0000) 2022-01-31T23:04:13.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:12 smithi167 conmon[60316]: debug 2022-01-31T23:04:12.972+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.973091+0000) 2022-01-31T23:04:13.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:12 smithi171 conmon[46715]: debug 2022-01-31T23:04:12.957+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:12.958201+0000) 2022-01-31T23:04:13.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:13 smithi167 conmon[54076]: debug 2022-01-31T23:04:13.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:13.213729+0000) 2022-01-31T23:04:13.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:13 smithi171 conmon[51620]: debug 2022-01-31T23:04:13.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:13.588089+0000) 2022-01-31T23:04:13.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:13 smithi171 conmon[41853]: debug 2022-01-31T23:04:13.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:13.412899+0000) 2022-01-31T23:04:14.196 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:14 smithi167 conmon[49112]: debug 2022-01-31T23:04:14.042+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:14.044124+0000) 2022-01-31T23:04:14.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:13 smithi167 conmon[60316]: debug 2022-01-31T23:04:13.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:13.973203+0000) 2022-01-31T23:04:14.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:13 smithi171 conmon[46715]: debug 2022-01-31T23:04:13.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:13.958356+0000) 2022-01-31T23:04:14.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:14 smithi167 conmon[54076]: debug 2022-01-31T23:04:14.213+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:14.213893+0000) 2022-01-31T23:04:14.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:14 smithi171 conmon[41853]: debug 2022-01-31T23:04:14.412+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:14.413018+0000) 2022-01-31T23:04:14.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:14 smithi171 conmon[51620]: debug 2022-01-31T23:04:14.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:14.588302+0000) 2022-01-31T23:04:15.197 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:15 smithi167 conmon[49112]: debug 2022-01-31T23:04:15.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:15.044327+0000) 2022-01-31T23:04:15.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:14 smithi167 conmon[60316]: debug 2022-01-31T23:04:14.971+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:14.973406+0000) 2022-01-31T23:04:15.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:14 smithi171 conmon[46715]: debug 2022-01-31T23:04:14.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:14.958585+0000) 2022-01-31T23:04:15.552 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:15 smithi167 conmon[54076]: debug 2022-01-31T23:04:15.213+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:15.214082+0000) 2022-01-31T23:04:15.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:15 smithi171 conmon[41853]: debug 2022-01-31T23:04:15.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:15.413195+0000) 2022-01-31T23:04:15.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:15 smithi171 conmon[51620]: debug 2022-01-31T23:04:15.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:15.588486+0000) 2022-01-31T23:04:16.197 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:16 smithi167 conmon[49112]: debug 2022-01-31T23:04:16.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:16.044527+0000) 2022-01-31T23:04:16.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:15 smithi167 conmon[60316]: debug 2022-01-31T23:04:15.972+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:15.973581+0000) 2022-01-31T23:04:16.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:15 smithi171 conmon[46715]: debug 2022-01-31T23:04:15.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:15.958734+0000) 2022-01-31T23:04:16.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:16 smithi167 conmon[54076]: debug 2022-01-31T23:04:16.212+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:16.214286+0000) 2022-01-31T23:04:16.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:16 smithi171 conmon[41853]: debug 2022-01-31T23:04:16.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:16.413348+0000) 2022-01-31T23:04:16.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:16 smithi171 conmon[51620]: debug 2022-01-31T23:04:16.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:16.588675+0000) 2022-01-31T23:04:17.197 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:17 smithi167 conmon[49112]: debug 2022-01-31T23:04:17.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.044682+0000) 2022-01-31T23:04:17.197 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:16 smithi167 conmon[60316]: debug 2022-01-31T23:04:16.972+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:16.973739+0000) 2022-01-31T23:04:17.311 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:16 smithi171 conmon[46715]: debug 2022-01-31T23:04:16.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:16.958972+0000) 2022-01-31T23:04:17.553 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:17 smithi167 conmon[49112]: debug 2022-01-31T23:04:17.329+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.331295+0000) 2022-01-31T23:04:17.554 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:17 smithi167 conmon[60316]: debug 2022-01-31T23:04:17.327+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.329315+0000) 2022-01-31T23:04:17.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:17 smithi167 conmon[54076]: debug 2022-01-31T23:04:17.213+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.214444+0000) 2022-01-31T23:04:17.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:17 smithi167 conmon[54076]: debug 2022-01-31T23:04:17.328+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.329948+0000) 2022-01-31T23:04:17.571 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[35325]: debug 2022-01-31T23:04:17.356+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260217 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:17.572 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[46715]: debug 2022-01-31T23:04:17.329+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.330225+0000) 2022-01-31T23:04:17.572 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[51620]: debug 2022-01-31T23:04:17.329+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.330430+0000) 2022-01-31T23:04:17.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[41853]: debug 2022-01-31T23:04:17.327+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.328652+0000) 2022-01-31T23:04:17.573 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[41853]: debug 2022-01-31T23:04:17.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.413531+0000) 2022-01-31T23:04:17.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[51620]: debug 2022-01-31T23:04:17.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.588856+0000) 2022-01-31T23:04:18.197 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:18 smithi167 conmon[49112]: debug 2022-01-31T23:04:18.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:18.044856+0000) 2022-01-31T23:04:18.198 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:17 smithi167 conmon[60316]: debug 2022-01-31T23:04:17.973+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.973933+0000) 2022-01-31T23:04:18.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:17 smithi171 conmon[46715]: debug 2022-01-31T23:04:17.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:17.959172+0000) 2022-01-31T23:04:18.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:18 smithi167 conmon[54076]: debug 2022-01-31T23:04:18.213+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:18.214580+0000) 2022-01-31T23:04:18.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:18 smithi171 conmon[41853]: debug 2022-01-31T23:04:18.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:18.413667+0000) 2022-01-31T23:04:18.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:18 smithi171 conmon[51620]: debug 2022-01-31T23:04:18.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:18.588968+0000) 2022-01-31T23:04:19.197 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:19 smithi167 conmon[49112]: debug 2022-01-31T23:04:19.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:19.045032+0000) 2022-01-31T23:04:19.198 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:18 smithi167 conmon[60316]: debug 2022-01-31T23:04:18.973+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:18.974147+0000) 2022-01-31T23:04:19.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:18 smithi171 conmon[46715]: debug 2022-01-31T23:04:18.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:18.959309+0000) 2022-01-31T23:04:19.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:19 smithi167 conmon[54076]: debug 2022-01-31T23:04:19.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:19.214701+0000) 2022-01-31T23:04:19.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:19 smithi171 conmon[41853]: debug 2022-01-31T23:04:19.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:19.413865+0000) 2022-01-31T23:04:19.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:19 smithi171 conmon[51620]: debug 2022-01-31T23:04:19.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:19.589167+0000) 2022-01-31T23:04:20.197 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:20 smithi167 conmon[49112]: debug 2022-01-31T23:04:20.043+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:20.045211+0000) 2022-01-31T23:04:20.198 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:19 smithi167 conmon[60316]: debug 2022-01-31T23:04:19.973+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:19.974301+0000) 2022-01-31T23:04:20.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:19 smithi171 conmon[46715]: debug 2022-01-31T23:04:19.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:19.959502+0000) 2022-01-31T23:04:20.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:20 smithi167 conmon[54076]: debug 2022-01-31T23:04:20.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:20.214865+0000) 2022-01-31T23:04:20.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:20 smithi171 conmon[41853]: debug 2022-01-31T23:04:20.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:20.414014+0000) 2022-01-31T23:04:20.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:20 smithi171 conmon[51620]: debug 2022-01-31T23:04:20.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:20.589361+0000) 2022-01-31T23:04:21.198 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:21 smithi167 conmon[49112]: debug 2022-01-31T23:04:21.044+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:21.045416+0000) 2022-01-31T23:04:21.198 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:20 smithi167 conmon[60316]: debug 2022-01-31T23:04:20.973+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:20.974481+0000) 2022-01-31T23:04:21.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:20 smithi171 conmon[46715]: debug 2022-01-31T23:04:20.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:20.959695+0000) 2022-01-31T23:04:21.553 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:21 smithi167 conmon[54076]: debug 2022-01-31T23:04:21.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:21.215038+0000) 2022-01-31T23:04:21.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:21 smithi171 conmon[51620]: debug 2022-01-31T23:04:21.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:21.589513+0000) 2022-01-31T23:04:21.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:21 smithi171 conmon[41853]: debug 2022-01-31T23:04:21.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:21.414197+0000) 2022-01-31T23:04:22.198 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:21 smithi167 conmon[60316]: debug 2022-01-31T23:04:21.974+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:21.974680+0000) 2022-01-31T23:04:22.198 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:22 smithi167 conmon[49112]: debug 2022-01-31T23:04:22.044+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.045602+0000) 2022-01-31T23:04:22.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:21 smithi171 conmon[46715]: debug 2022-01-31T23:04:21.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:21.959904+0000) 2022-01-31T23:04:22.554 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:22 smithi167 conmon[49112]: debug 2022-01-31T23:04:22.358+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.360262+0000) 2022-01-31T23:04:22.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:22 smithi167 conmon[54076]: debug 2022-01-31T23:04:22.213+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.215194+0000) 2022-01-31T23:04:22.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:22 smithi167 conmon[54076]: debug 2022-01-31T23:04:22.358+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.359979+0000) 2022-01-31T23:04:22.555 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:22 smithi167 conmon[60316]: debug 2022-01-31T23:04:22.359+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.360566+0000) 2022-01-31T23:04:22.606 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[35325]: debug 2022-01-31T23:04:22.385+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260329 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:22.607 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[46715]: debug 2022-01-31T23:04:22.358+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.360437+0000) 2022-01-31T23:04:22.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[41853]: debug 2022-01-31T23:04:22.357+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.359490+0000) 2022-01-31T23:04:22.608 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[41853]: debug 2022-01-31T23:04:22.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.414400+0000) 2022-01-31T23:04:22.608 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[51620]: debug 2022-01-31T23:04:22.357+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.359326+0000) 2022-01-31T23:04:22.609 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[51620]: debug 2022-01-31T23:04:22.587+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.589721+0000) 2022-01-31T23:04:23.198 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:23 smithi167 conmon[49112]: debug 2022-01-31T23:04:23.044+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:23.045789+0000) 2022-01-31T23:04:23.199 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:22 smithi167 conmon[60316]: debug 2022-01-31T23:04:22.974+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.974868+0000) 2022-01-31T23:04:23.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:22 smithi171 conmon[46715]: debug 2022-01-31T23:04:22.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:22.960062+0000) 2022-01-31T23:04:23.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:23 smithi167 conmon[54076]: debug 2022-01-31T23:04:23.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:23.215373+0000) 2022-01-31T23:04:23.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:23 smithi171 conmon[41853]: debug 2022-01-31T23:04:23.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:23.414549+0000) 2022-01-31T23:04:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:23 smithi171 conmon[51620]: debug 2022-01-31T23:04:23.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:23.589868+0000) 2022-01-31T23:04:24.198 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:24 smithi167 conmon[49112]: debug 2022-01-31T23:04:24.044+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:24.045921+0000) 2022-01-31T23:04:24.199 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:23 smithi167 conmon[60316]: debug 2022-01-31T23:04:23.973+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:23.974979+0000) 2022-01-31T23:04:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:23 smithi171 conmon[46715]: debug 2022-01-31T23:04:23.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:23.960184+0000) 2022-01-31T23:04:24.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:23 smithi171 conmon[35325]: debug 2022-01-31T23:04:23.956+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:04:24.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:24 smithi167 conmon[54076]: debug 2022-01-31T23:04:24.214+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:24.215534+0000) 2022-01-31T23:04:24.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:24 smithi171 conmon[41853]: debug 2022-01-31T23:04:24.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:24.414713+0000) 2022-01-31T23:04:24.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:24 smithi171 conmon[51620]: debug 2022-01-31T23:04:24.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:24.590068+0000) 2022-01-31T23:04:25.198 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:25 smithi167 conmon[49112]: debug 2022-01-31T23:04:25.045+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:25.046123+0000) 2022-01-31T23:04:25.199 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:24 smithi167 conmon[60316]: debug 2022-01-31T23:04:24.974+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:24.975169+0000) 2022-01-31T23:04:25.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:24 smithi171 conmon[46715]: debug 2022-01-31T23:04:24.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:24.960371+0000) 2022-01-31T23:04:25.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:25 smithi167 conmon[54076]: debug 2022-01-31T23:04:25.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:25.215764+0000) 2022-01-31T23:04:25.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:25 smithi171 conmon[41853]: debug 2022-01-31T23:04:25.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:25.414850+0000) 2022-01-31T23:04:25.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:25 smithi171 conmon[51620]: debug 2022-01-31T23:04:25.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:25.590289+0000) 2022-01-31T23:04:26.198 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:26 smithi167 conmon[49112]: debug 2022-01-31T23:04:26.045+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:26.046305+0000) 2022-01-31T23:04:26.199 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:25 smithi167 conmon[60316]: debug 2022-01-31T23:04:25.974+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:25.975292+0000) 2022-01-31T23:04:26.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:25 smithi171 conmon[46715]: debug 2022-01-31T23:04:25.958+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:25.960502+0000) 2022-01-31T23:04:26.554 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:26 smithi167 conmon[54076]: debug 2022-01-31T23:04:26.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:26.215922+0000) 2022-01-31T23:04:26.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:26 smithi171 conmon[41853]: debug 2022-01-31T23:04:26.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:26.415004+0000) 2022-01-31T23:04:26.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:26 smithi171 conmon[51620]: debug 2022-01-31T23:04:26.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:26.590498+0000) 2022-01-31T23:04:27.199 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:27 smithi167 conmon[49112]: debug 2022-01-31T23:04:27.046+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.046555+0000) 2022-01-31T23:04:27.199 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:26 smithi167 conmon[60316]: debug 2022-01-31T23:04:26.975+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:26.975517+0000) 2022-01-31T23:04:27.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:26 smithi171 conmon[46715]: debug 2022-01-31T23:04:26.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:26.960696+0000) 2022-01-31T23:04:27.555 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:27 smithi167 conmon[49112]: debug 2022-01-31T23:04:27.388+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.389453+0000) 2022-01-31T23:04:27.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:27 smithi167 conmon[54076]: debug 2022-01-31T23:04:27.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.216103+0000) 2022-01-31T23:04:27.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:27 smithi167 conmon[54076]: debug 2022-01-31T23:04:27.389+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.389829+0000) 2022-01-31T23:04:27.557 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:27 smithi167 conmon[60316]: debug 2022-01-31T23:04:27.389+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.390241+0000) 2022-01-31T23:04:27.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[35325]: debug 2022-01-31T23:04:27.415+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260441 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:27.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[46715]: debug 2022-01-31T23:04:27.388+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.389950+0000) 2022-01-31T23:04:27.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[41853]: debug 2022-01-31T23:04:27.387+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.389344+0000) 2022-01-31T23:04:27.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[41853]: debug 2022-01-31T23:04:27.413+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.415143+0000) 2022-01-31T23:04:27.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[51620]: debug 2022-01-31T23:04:27.387+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.389359+0000) 2022-01-31T23:04:27.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[51620]: debug 2022-01-31T23:04:27.588+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.590706+0000) 2022-01-31T23:04:28.199 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:28 smithi167 conmon[49112]: debug 2022-01-31T23:04:28.046+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:28.046659+0000) 2022-01-31T23:04:28.199 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:27 smithi167 conmon[60316]: debug 2022-01-31T23:04:27.975+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.975669+0000) 2022-01-31T23:04:28.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:27 smithi171 conmon[46715]: debug 2022-01-31T23:04:27.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:27.960909+0000) 2022-01-31T23:04:28.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:28 smithi167 conmon[54076]: debug 2022-01-31T23:04:28.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:28.216296+0000) 2022-01-31T23:04:28.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:28 smithi171 conmon[41853]: debug 2022-01-31T23:04:28.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:28.415299+0000) 2022-01-31T23:04:28.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:28 smithi171 conmon[51620]: debug 2022-01-31T23:04:28.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:28.590862+0000) 2022-01-31T23:04:29.200 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:29 smithi167 conmon[49112]: debug 2022-01-31T23:04:29.046+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:29.046790+0000) 2022-01-31T23:04:29.201 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:28 smithi167 conmon[60316]: debug 2022-01-31T23:04:28.975+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:28.975825+0000) 2022-01-31T23:04:29.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:28 smithi171 conmon[46715]: debug 2022-01-31T23:04:28.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:28.961031+0000) 2022-01-31T23:04:29.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:29 smithi167 conmon[54076]: debug 2022-01-31T23:04:29.215+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:29.216445+0000) 2022-01-31T23:04:29.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:29 smithi171 conmon[41853]: debug 2022-01-31T23:04:29.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:29.415490+0000) 2022-01-31T23:04:29.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:29 smithi171 conmon[51620]: debug 2022-01-31T23:04:29.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:29.590971+0000) 2022-01-31T23:04:30.199 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:30 smithi167 conmon[49112]: debug 2022-01-31T23:04:30.046+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:30.046975+0000) 2022-01-31T23:04:30.200 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:29 smithi167 conmon[60316]: debug 2022-01-31T23:04:29.975+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:29.976013+0000) 2022-01-31T23:04:30.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:29 smithi171 conmon[46715]: debug 2022-01-31T23:04:29.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:29.961193+0000) 2022-01-31T23:04:30.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:30 smithi167 conmon[54076]: debug 2022-01-31T23:04:30.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:30.216605+0000) 2022-01-31T23:04:30.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:30 smithi171 conmon[51620]: debug 2022-01-31T23:04:30.589+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:30.591098+0000) 2022-01-31T23:04:30.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:30 smithi171 conmon[41853]: debug 2022-01-31T23:04:30.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:30.415719+0000) 2022-01-31T23:04:31.199 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:31 smithi167 conmon[49112]: debug 2022-01-31T23:04:31.046+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:31.047163+0000) 2022-01-31T23:04:31.200 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:30 smithi167 conmon[60316]: debug 2022-01-31T23:04:30.975+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:30.976170+0000) 2022-01-31T23:04:31.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:30 smithi171 conmon[46715]: debug 2022-01-31T23:04:30.959+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:30.961388+0000) 2022-01-31T23:04:31.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:31 smithi167 conmon[54076]: debug 2022-01-31T23:04:31.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:31.216771+0000) 2022-01-31T23:04:31.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:31 smithi171 conmon[41853]: debug 2022-01-31T23:04:31.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:31.415921+0000) 2022-01-31T23:04:31.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:31 smithi171 conmon[51620]: debug 2022-01-31T23:04:31.590+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:31.591260+0000) 2022-01-31T23:04:32.199 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:32 smithi167 conmon[49112]: debug 2022-01-31T23:04:32.046+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.047324+0000) 2022-01-31T23:04:32.200 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:31 smithi167 conmon[60316]: debug 2022-01-31T23:04:31.975+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:31.976359+0000) 2022-01-31T23:04:32.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:31 smithi171 conmon[46715]: debug 2022-01-31T23:04:31.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:31.961533+0000) 2022-01-31T23:04:32.556 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:32 smithi167 conmon[49112]: debug 2022-01-31T23:04:32.426+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.426745+0000) 2022-01-31T23:04:32.556 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:32 smithi167 conmon[60316]: debug 2022-01-31T23:04:32.419+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.419612+0000) 2022-01-31T23:04:32.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:32 smithi167 conmon[54076]: debug 2022-01-31T23:04:32.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.216935+0000) 2022-01-31T23:04:32.557 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:32 smithi167 conmon[54076]: debug 2022-01-31T23:04:32.419+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.419977+0000) 2022-01-31T23:04:32.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[35325]: debug 2022-01-31T23:04:32.451+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260552 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:32.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[41853]: debug 2022-01-31T23:04:32.414+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.416037+0000) 2022-01-31T23:04:32.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[41853]: debug 2022-01-31T23:04:32.418+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.420312+0000) 2022-01-31T23:04:32.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[46715]: debug 2022-01-31T23:04:32.419+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.420972+0000) 2022-01-31T23:04:32.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[51620]: debug 2022-01-31T23:04:32.417+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.419517+0000) 2022-01-31T23:04:32.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[51620]: debug 2022-01-31T23:04:32.590+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.591450+0000) 2022-01-31T23:04:33.199 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:33 smithi167 conmon[49112]: debug 2022-01-31T23:04:33.047+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:33.047532+0000) 2022-01-31T23:04:33.200 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:32 smithi167 conmon[60316]: debug 2022-01-31T23:04:32.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.976578+0000) 2022-01-31T23:04:33.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:32 smithi171 conmon[46715]: debug 2022-01-31T23:04:32.960+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:32.961740+0000) 2022-01-31T23:04:33.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:33 smithi167 conmon[54076]: debug 2022-01-31T23:04:33.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:33.217095+0000) 2022-01-31T23:04:33.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:33 smithi171 conmon[41853]: debug 2022-01-31T23:04:33.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:33.416189+0000) 2022-01-31T23:04:33.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:33 smithi171 conmon[51620]: debug 2022-01-31T23:04:33.590+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:33.591613+0000) 2022-01-31T23:04:34.200 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:34 smithi167 conmon[49112]: debug 2022-01-31T23:04:34.047+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:34.047651+0000) 2022-01-31T23:04:34.200 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:33 smithi167 conmon[60316]: debug 2022-01-31T23:04:33.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:33.976680+0000) 2022-01-31T23:04:34.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:33 smithi171 conmon[46715]: debug 2022-01-31T23:04:33.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:33.961904+0000) 2022-01-31T23:04:34.556 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:34 smithi167 conmon[54076]: debug 2022-01-31T23:04:34.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:34.217241+0000) 2022-01-31T23:04:34.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:34 smithi171 conmon[41853]: debug 2022-01-31T23:04:34.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:34.416335+0000) 2022-01-31T23:04:34.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:34 smithi171 conmon[51620]: debug 2022-01-31T23:04:34.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:34.591811+0000) 2022-01-31T23:04:35.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:35 smithi167 conmon[49112]: debug 2022-01-31T23:04:35.047+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:35.047825+0000) 2022-01-31T23:04:35.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:34 smithi167 conmon[60316]: debug 2022-01-31T23:04:34.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:34.976876+0000) 2022-01-31T23:04:35.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:34 smithi171 conmon[46715]: debug 2022-01-31T23:04:34.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:34.962124+0000) 2022-01-31T23:04:35.448 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:35 smithi167 conmon[54076]: debug 2022-01-31T23:04:35.216+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:35.217377+0000) 2022-01-31T23:04:35.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:35 smithi171 conmon[41853]: debug 2022-01-31T23:04:35.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:35.416495+0000) 2022-01-31T23:04:35.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:35 smithi171 conmon[51620]: debug 2022-01-31T23:04:35.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:35.591996+0000) 2022-01-31T23:04:36.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:36 smithi167 conmon[49112]: debug 2022-01-31T23:04:36.047+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:36.048008+0000) 2022-01-31T23:04:36.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:35 smithi167 conmon[60316]: debug 2022-01-31T23:04:35.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:35.976982+0000) 2022-01-31T23:04:36.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:35 smithi171 conmon[46715]: debug 2022-01-31T23:04:35.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:35.962352+0000) 2022-01-31T23:04:36.448 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:36 smithi167 conmon[54076]: debug 2022-01-31T23:04:36.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:36.217536+0000) 2022-01-31T23:04:36.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:36 smithi171 conmon[41853]: debug 2022-01-31T23:04:36.415+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:36.416648+0000) 2022-01-31T23:04:36.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:36 smithi171 conmon[51620]: debug 2022-01-31T23:04:36.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:36.592192+0000) 2022-01-31T23:04:37.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:37 smithi167 conmon[49112]: debug 2022-01-31T23:04:37.047+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.048171+0000) 2022-01-31T23:04:37.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:36 smithi167 conmon[60316]: debug 2022-01-31T23:04:36.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:36.977141+0000) 2022-01-31T23:04:37.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:36 smithi171 conmon[46715]: debug 2022-01-31T23:04:36.961+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:36.962557+0000) 2022-01-31T23:04:37.438 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:37 smithi167 conmon[54076]: debug 2022-01-31T23:04:37.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.217676+0000) 2022-01-31T23:04:37.822 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:37 smithi167 conmon[49112]: debug 2022-01-31T23:04:37.456+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.456553+0000) 2022-01-31T23:04:37.822 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:37 smithi167 conmon[54076]: debug 2022-01-31T23:04:37.454+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.455187+0000) 2022-01-31T23:04:37.823 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:37 smithi167 conmon[60316]: debug 2022-01-31T23:04:37.455+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.456439+0000) 2022-01-31T23:04:37.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[35325]: debug 2022-01-31T23:04:37.481+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260664 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:38.152 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[46715]: debug 2022-01-31T23:04:37.454+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.455107+0000) 2022-01-31T23:04:38.153 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[41853]: debug 2022-01-31T23:04:37.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.416806+0000) 2022-01-31T23:04:38.153 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[41853]: debug 2022-01-31T23:04:37.454+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.455554+0000) 2022-01-31T23:04:38.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[51620]: debug 2022-01-31T23:04:37.454+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.455669+0000) 2022-01-31T23:04:38.154 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[51620]: debug 2022-01-31T23:04:37.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.592392+0000) 2022-01-31T23:04:38.155 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:38 smithi167 conmon[49112]: debug 2022-01-31T23:04:38.048+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:38.048359+0000) 2022-01-31T23:04:38.155 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:37 smithi167 conmon[60316]: debug 2022-01-31T23:04:37.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.977327+0000) 2022-01-31T23:04:38.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:37 smithi171 conmon[46715]: debug 2022-01-31T23:04:37.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:37.962800+0000) 2022-01-31T23:04:38.449 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:38 smithi167 conmon[54076]: debug 2022-01-31T23:04:38.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:38.217841+0000) 2022-01-31T23:04:38.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:38 smithi171 conmon[41853]: debug 2022-01-31T23:04:38.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:38.416960+0000) 2022-01-31T23:04:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:38 smithi171 conmon[51620]: debug 2022-01-31T23:04:38.591+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:38.592558+0000) 2022-01-31T23:04:39.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:39 smithi167 conmon[49112]: debug 2022-01-31T23:04:39.048+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:39.048511+0000) 2022-01-31T23:04:39.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:38 smithi167 conmon[60316]: debug 2022-01-31T23:04:38.976+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:38.977464+0000) 2022-01-31T23:04:39.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:38 smithi171 conmon[35325]: debug 2022-01-31T23:04:38.957+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:04:39.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:38 smithi171 conmon[46715]: debug 2022-01-31T23:04:38.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:38.962900+0000) 2022-01-31T23:04:39.449 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:39 smithi167 conmon[54076]: debug 2022-01-31T23:04:39.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:39.218008+0000) 2022-01-31T23:04:39.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:39 smithi171 conmon[41853]: debug 2022-01-31T23:04:39.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:39.417127+0000) 2022-01-31T23:04:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:39 smithi171 conmon[51620]: debug 2022-01-31T23:04:39.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:39.592783+0000) 2022-01-31T23:04:40.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:39 smithi167 conmon[60316]: debug 2022-01-31T23:04:39.977+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:39.977659+0000) 2022-01-31T23:04:40.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:40 smithi167 conmon[49112]: debug 2022-01-31T23:04:40.048+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:40.048716+0000) 2022-01-31T23:04:40.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:39 smithi171 conmon[46715]: debug 2022-01-31T23:04:39.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:39.963063+0000) 2022-01-31T23:04:40.449 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:40 smithi167 conmon[54076]: debug 2022-01-31T23:04:40.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:40.218191+0000) 2022-01-31T23:04:40.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:40 smithi171 conmon[51620]: debug 2022-01-31T23:04:40.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:40.593002+0000) 2022-01-31T23:04:40.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:40 smithi171 conmon[41853]: debug 2022-01-31T23:04:40.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:40.417316+0000) 2022-01-31T23:04:41.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:41 smithi167 conmon[49112]: debug 2022-01-31T23:04:41.048+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:41.048914+0000) 2022-01-31T23:04:41.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:40 smithi167 conmon[60316]: debug 2022-01-31T23:04:40.977+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:40.977846+0000) 2022-01-31T23:04:41.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:40 smithi171 conmon[46715]: debug 2022-01-31T23:04:40.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:40.963248+0000) 2022-01-31T23:04:41.449 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:41 smithi167 conmon[54076]: debug 2022-01-31T23:04:41.217+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:41.218421+0000) 2022-01-31T23:04:41.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:41 smithi171 conmon[51620]: debug 2022-01-31T23:04:41.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:41.593159+0000) 2022-01-31T23:04:41.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:41 smithi171 conmon[41853]: debug 2022-01-31T23:04:41.416+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:41.417503+0000) 2022-01-31T23:04:42.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:41 smithi167 conmon[60316]: debug 2022-01-31T23:04:41.977+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:41.978076+0000) 2022-01-31T23:04:42.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:42 smithi167 conmon[49112]: debug 2022-01-31T23:04:42.048+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.049107+0000) 2022-01-31T23:04:42.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:41 smithi171 conmon[46715]: debug 2022-01-31T23:04:41.962+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:41.963454+0000) 2022-01-31T23:04:42.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:42 smithi167 conmon[54076]: debug 2022-01-31T23:04:42.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.218608+0000) 2022-01-31T23:04:42.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[35325]: debug 2022-01-31T23:04:42.511+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260776 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:42.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[41853]: debug 2022-01-31T23:04:42.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.417732+0000) 2022-01-31T23:04:42.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[41853]: debug 2022-01-31T23:04:42.483+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.484124+0000) 2022-01-31T23:04:42.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[46715]: debug 2022-01-31T23:04:42.483+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.484297+0000) 2022-01-31T23:04:42.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[51620]: debug 2022-01-31T23:04:42.484+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.485033+0000) 2022-01-31T23:04:42.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[51620]: debug 2022-01-31T23:04:42.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.593348+0000) 2022-01-31T23:04:42.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:42 smithi167 conmon[49112]: debug 2022-01-31T23:04:42.485+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.485966+0000) 2022-01-31T23:04:42.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:42 smithi167 conmon[54076]: debug 2022-01-31T23:04:42.485+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.485536+0000) 2022-01-31T23:04:42.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:42 smithi167 conmon[60316]: debug 2022-01-31T23:04:42.485+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.485830+0000) 2022-01-31T23:04:43.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:42 smithi167 conmon[60316]: debug 2022-01-31T23:04:42.977+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.978227+0000) 2022-01-31T23:04:43.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:43 smithi167 conmon[49112]: debug 2022-01-31T23:04:43.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:43.049302+0000) 2022-01-31T23:04:43.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:42 smithi171 conmon[46715]: debug 2022-01-31T23:04:42.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:42.963602+0000) 2022-01-31T23:04:43.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:43 smithi167 conmon[54076]: debug 2022-01-31T23:04:43.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:43.218775+0000) 2022-01-31T23:04:43.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:43 smithi171 conmon[41853]: debug 2022-01-31T23:04:43.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:43.417974+0000) 2022-01-31T23:04:43.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:43 smithi171 conmon[51620]: debug 2022-01-31T23:04:43.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:43.593543+0000) 2022-01-31T23:04:44.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:44 smithi167 conmon[49112]: debug 2022-01-31T23:04:44.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:44.049477+0000) 2022-01-31T23:04:44.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:43 smithi167 conmon[60316]: debug 2022-01-31T23:04:43.977+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:43.978331+0000) 2022-01-31T23:04:44.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:43 smithi171 conmon[46715]: debug 2022-01-31T23:04:43.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:43.963742+0000) 2022-01-31T23:04:44.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:44 smithi167 conmon[54076]: debug 2022-01-31T23:04:44.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:44.218901+0000) 2022-01-31T23:04:44.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:44 smithi171 conmon[51620]: debug 2022-01-31T23:04:44.592+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:44.593715+0000) 2022-01-31T23:04:44.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:44 smithi171 conmon[41853]: debug 2022-01-31T23:04:44.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:44.418086+0000) 2022-01-31T23:04:45.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:45 smithi167 conmon[49112]: debug 2022-01-31T23:04:45.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:45.049697+0000) 2022-01-31T23:04:45.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:44 smithi167 conmon[60316]: debug 2022-01-31T23:04:44.978+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:44.978557+0000) 2022-01-31T23:04:45.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:44 smithi171 conmon[46715]: debug 2022-01-31T23:04:44.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:44.963973+0000) 2022-01-31T23:04:45.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:45 smithi167 conmon[54076]: debug 2022-01-31T23:04:45.218+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:45.219099+0000) 2022-01-31T23:04:45.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:45 smithi171 conmon[41853]: debug 2022-01-31T23:04:45.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:45.418214+0000) 2022-01-31T23:04:45.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:45 smithi171 conmon[51620]: debug 2022-01-31T23:04:45.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:45.593850+0000) 2022-01-31T23:04:46.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:46 smithi167 conmon[49112]: debug 2022-01-31T23:04:46.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:46.049879+0000) 2022-01-31T23:04:46.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:45 smithi167 conmon[60316]: debug 2022-01-31T23:04:45.978+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:45.978763+0000) 2022-01-31T23:04:46.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:45 smithi171 conmon[46715]: debug 2022-01-31T23:04:45.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:45.964144+0000) 2022-01-31T23:04:46.450 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:46 smithi167 conmon[54076]: debug 2022-01-31T23:04:46.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:46.219307+0000) 2022-01-31T23:04:46.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:46 smithi171 conmon[41853]: debug 2022-01-31T23:04:46.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:46.418371+0000) 2022-01-31T23:04:46.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:46 smithi171 conmon[51620]: debug 2022-01-31T23:04:46.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:46.593980+0000) 2022-01-31T23:04:47.006 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:46 smithi167 conmon[60316]: debug 2022-01-31T23:04:46.978+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:46.978983+0000) 2022-01-31T23:04:47.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:46 smithi171 conmon[46715]: debug 2022-01-31T23:04:46.963+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:46.964332+0000) 2022-01-31T23:04:47.402 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:47 smithi167 conmon[49112]: debug 2022-01-31T23:04:47.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.050090+0000) 2022-01-31T23:04:47.402 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:47 smithi167 conmon[54076]: debug 2022-01-31T23:04:47.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.219526+0000) 2022-01-31T23:04:47.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[35325]: debug 2022-01-31T23:04:47.542+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 260887 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:47.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[46715]: debug 2022-01-31T23:04:47.513+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.514278+0000) 2022-01-31T23:04:47.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[51620]: debug 2022-01-31T23:04:47.514+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.515188+0000) 2022-01-31T23:04:47.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[51620]: debug 2022-01-31T23:04:47.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.594191+0000) 2022-01-31T23:04:47.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[41853]: debug 2022-01-31T23:04:47.417+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.418546+0000) 2022-01-31T23:04:47.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[41853]: debug 2022-01-31T23:04:47.513+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.514642+0000) 2022-01-31T23:04:47.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:47 smithi167 conmon[49112]: debug 2022-01-31T23:04:47.515+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.515591+0000) 2022-01-31T23:04:47.860 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:47 smithi167 conmon[54076]: debug 2022-01-31T23:04:47.515+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.516131+0000) 2022-01-31T23:04:47.860 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:47 smithi167 conmon[60316]: debug 2022-01-31T23:04:47.515+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.515776+0000) 2022-01-31T23:04:48.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:48 smithi167 conmon[49112]: debug 2022-01-31T23:04:48.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:48.050297+0000) 2022-01-31T23:04:48.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:47 smithi167 conmon[60316]: debug 2022-01-31T23:04:47.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.979188+0000) 2022-01-31T23:04:48.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:47 smithi171 conmon[46715]: debug 2022-01-31T23:04:47.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:47.964535+0000) 2022-01-31T23:04:48.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:48 smithi167 conmon[54076]: debug 2022-01-31T23:04:48.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:48.219679+0000) 2022-01-31T23:04:48.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:48 smithi171 conmon[41853]: debug 2022-01-31T23:04:48.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:48.418768+0000) 2022-01-31T23:04:48.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:48 smithi171 conmon[51620]: debug 2022-01-31T23:04:48.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:48.594353+0000) 2022-01-31T23:04:49.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:49 smithi167 conmon[49112]: debug 2022-01-31T23:04:49.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:49.050469+0000) 2022-01-31T23:04:49.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:48 smithi167 conmon[60316]: debug 2022-01-31T23:04:48.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:48.979323+0000) 2022-01-31T23:04:49.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:48 smithi171 conmon[46715]: debug 2022-01-31T23:04:48.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:48.964673+0000) 2022-01-31T23:04:49.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:49 smithi167 conmon[54076]: debug 2022-01-31T23:04:49.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:49.219826+0000) 2022-01-31T23:04:49.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:49 smithi171 conmon[41853]: debug 2022-01-31T23:04:49.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:49.418927+0000) 2022-01-31T23:04:49.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:49 smithi171 conmon[51620]: debug 2022-01-31T23:04:49.593+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:49.594553+0000) 2022-01-31T23:04:50.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:50 smithi167 conmon[49112]: debug 2022-01-31T23:04:50.050+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:50.050675+0000) 2022-01-31T23:04:50.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:49 smithi167 conmon[60316]: debug 2022-01-31T23:04:49.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:49.979523+0000) 2022-01-31T23:04:50.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:49 smithi171 conmon[46715]: debug 2022-01-31T23:04:49.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:49.964842+0000) 2022-01-31T23:04:50.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:50 smithi167 conmon[54076]: debug 2022-01-31T23:04:50.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:50.219948+0000) 2022-01-31T23:04:50.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:50 smithi171 conmon[51620]: debug 2022-01-31T23:04:50.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:50.594766+0000) 2022-01-31T23:04:50.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:50 smithi171 conmon[41853]: debug 2022-01-31T23:04:50.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:50.419112+0000) 2022-01-31T23:04:51.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:51 smithi167 conmon[49112]: debug 2022-01-31T23:04:51.050+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:51.050859+0000) 2022-01-31T23:04:51.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:50 smithi167 conmon[60316]: debug 2022-01-31T23:04:50.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:50.979673+0000) 2022-01-31T23:04:51.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:50 smithi171 conmon[46715]: debug 2022-01-31T23:04:50.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:50.965056+0000) 2022-01-31T23:04:51.451 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:51 smithi167 conmon[54076]: debug 2022-01-31T23:04:51.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:51.220175+0000) 2022-01-31T23:04:51.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:51 smithi171 conmon[41853]: debug 2022-01-31T23:04:51.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:51.419291+0000) 2022-01-31T23:04:51.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:51 smithi171 conmon[51620]: debug 2022-01-31T23:04:51.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:51.594915+0000) 2022-01-31T23:04:52.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:52 smithi167 conmon[49112]: debug 2022-01-31T23:04:52.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.051029+0000) 2022-01-31T23:04:52.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:51 smithi167 conmon[60316]: debug 2022-01-31T23:04:51.978+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:51.979862+0000) 2022-01-31T23:04:52.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:51 smithi171 conmon[46715]: debug 2022-01-31T23:04:51.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:51.965225+0000) 2022-01-31T23:04:52.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:52 smithi167 conmon[54076]: debug 2022-01-31T23:04:52.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.220339+0000) 2022-01-31T23:04:52.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[35325]: debug 2022-01-31T23:04:52.572+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261015 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:52.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[46715]: debug 2022-01-31T23:04:52.545+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.546501+0000) 2022-01-31T23:04:52.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[41853]: debug 2022-01-31T23:04:52.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.419426+0000) 2022-01-31T23:04:52.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[41853]: debug 2022-01-31T23:04:52.544+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.545156+0000) 2022-01-31T23:04:52.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[51620]: debug 2022-01-31T23:04:52.545+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.546044+0000) 2022-01-31T23:04:52.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[51620]: debug 2022-01-31T23:04:52.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.595047+0000) 2022-01-31T23:04:52.878 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:52 smithi167 conmon[49112]: debug 2022-01-31T23:04:52.544+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.545591+0000) 2022-01-31T23:04:52.878 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:52 smithi167 conmon[54076]: debug 2022-01-31T23:04:52.545+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.546717+0000) 2022-01-31T23:04:52.879 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:52 smithi167 conmon[60316]: debug 2022-01-31T23:04:52.544+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.546164+0000) 2022-01-31T23:04:53.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:53 smithi167 conmon[49112]: debug 2022-01-31T23:04:53.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:53.051209+0000) 2022-01-31T23:04:53.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:52 smithi167 conmon[60316]: debug 2022-01-31T23:04:52.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.980055+0000) 2022-01-31T23:04:53.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:52 smithi171 conmon[46715]: debug 2022-01-31T23:04:52.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:52.965445+0000) 2022-01-31T23:04:53.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:53 smithi167 conmon[54076]: debug 2022-01-31T23:04:53.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:53.220528+0000) 2022-01-31T23:04:53.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:53 smithi171 conmon[41853]: debug 2022-01-31T23:04:53.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:53.419524+0000) 2022-01-31T23:04:53.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:53 smithi171 conmon[51620]: debug 2022-01-31T23:04:53.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:53.595188+0000) 2022-01-31T23:04:54.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:54 smithi167 conmon[49112]: debug 2022-01-31T23:04:54.049+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:54.051380+0000) 2022-01-31T23:04:54.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:53 smithi167 conmon[60316]: debug 2022-01-31T23:04:53.978+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:53.980202+0000) 2022-01-31T23:04:54.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:53 smithi171 conmon[35325]: debug 2022-01-31T23:04:53.958+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:04:54.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:53 smithi171 conmon[46715]: debug 2022-01-31T23:04:53.964+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:53.965566+0000) 2022-01-31T23:04:54.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:54 smithi167 conmon[54076]: debug 2022-01-31T23:04:54.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:54.220695+0000) 2022-01-31T23:04:54.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:54 smithi171 conmon[41853]: debug 2022-01-31T23:04:54.418+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:54.419680+0000) 2022-01-31T23:04:54.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:54 smithi171 conmon[51620]: debug 2022-01-31T23:04:54.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:54.595369+0000) 2022-01-31T23:04:55.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:55 smithi167 conmon[49112]: debug 2022-01-31T23:04:55.050+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:55.051602+0000) 2022-01-31T23:04:55.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:54 smithi167 conmon[60316]: debug 2022-01-31T23:04:54.978+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:54.980421+0000) 2022-01-31T23:04:55.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:54 smithi171 conmon[46715]: debug 2022-01-31T23:04:54.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:54.965814+0000) 2022-01-31T23:04:55.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:55 smithi167 conmon[54076]: debug 2022-01-31T23:04:55.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:55.220881+0000) 2022-01-31T23:04:55.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:55 smithi171 conmon[41853]: debug 2022-01-31T23:04:55.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:55.419825+0000) 2022-01-31T23:04:55.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:55 smithi171 conmon[51620]: debug 2022-01-31T23:04:55.594+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:55.595571+0000) 2022-01-31T23:04:56.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:56 smithi167 conmon[49112]: debug 2022-01-31T23:04:56.050+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:56.051764+0000) 2022-01-31T23:04:56.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:55 smithi167 conmon[60316]: debug 2022-01-31T23:04:55.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:55.980607+0000) 2022-01-31T23:04:56.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:55 smithi171 conmon[46715]: debug 2022-01-31T23:04:55.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:55.965970+0000) 2022-01-31T23:04:56.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:56 smithi167 conmon[54076]: debug 2022-01-31T23:04:56.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:56.221083+0000) 2022-01-31T23:04:56.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:56 smithi171 conmon[41853]: debug 2022-01-31T23:04:56.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:56.419956+0000) 2022-01-31T23:04:56.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:56 smithi171 conmon[51620]: debug 2022-01-31T23:04:56.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:56.595764+0000) 2022-01-31T23:04:57.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:57 smithi167 conmon[49112]: debug 2022-01-31T23:04:57.050+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.051927+0000) 2022-01-31T23:04:57.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:56 smithi167 conmon[60316]: debug 2022-01-31T23:04:56.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:56.980791+0000) 2022-01-31T23:04:57.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:56 smithi171 conmon[46715]: debug 2022-01-31T23:04:56.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:56.966158+0000) 2022-01-31T23:04:57.452 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:57 smithi167 conmon[54076]: debug 2022-01-31T23:04:57.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.221268+0000) 2022-01-31T23:04:57.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[35325]: debug 2022-01-31T23:04:57.603+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261126 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:04:57.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[46715]: debug 2022-01-31T23:04:57.575+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.576389+0000) 2022-01-31T23:04:57.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[41853]: debug 2022-01-31T23:04:57.419+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.420125+0000) 2022-01-31T23:04:57.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[41853]: debug 2022-01-31T23:04:57.575+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.575972+0000) 2022-01-31T23:04:57.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[51620]: debug 2022-01-31T23:04:57.576+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.576849+0000) 2022-01-31T23:04:57.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[51620]: debug 2022-01-31T23:04:57.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.595902+0000) 2022-01-31T23:04:57.896 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:57 smithi167 conmon[49112]: debug 2022-01-31T23:04:57.575+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.576703+0000) 2022-01-31T23:04:57.897 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:57 smithi167 conmon[54076]: debug 2022-01-31T23:04:57.575+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.577249+0000) 2022-01-31T23:04:57.897 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:57 smithi167 conmon[60316]: debug 2022-01-31T23:04:57.575+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.577109+0000) 2022-01-31T23:04:58.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:58 smithi167 conmon[49112]: debug 2022-01-31T23:04:58.050+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:58.052132+0000) 2022-01-31T23:04:58.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:57 smithi167 conmon[60316]: debug 2022-01-31T23:04:57.980+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.980965+0000) 2022-01-31T23:04:58.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:57 smithi171 conmon[46715]: debug 2022-01-31T23:04:57.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:57.966359+0000) 2022-01-31T23:04:58.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:58 smithi167 conmon[54076]: debug 2022-01-31T23:04:58.219+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:58.221458+0000) 2022-01-31T23:04:58.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:58 smithi171 conmon[41853]: debug 2022-01-31T23:04:58.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:58.420309+0000) 2022-01-31T23:04:58.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:58 smithi171 conmon[51620]: debug 2022-01-31T23:04:58.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:58.596050+0000) 2022-01-31T23:04:59.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:04:59 smithi167 conmon[49112]: debug 2022-01-31T23:04:59.051+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:59.052284+0000) 2022-01-31T23:04:59.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:58 smithi167 conmon[60316]: debug 2022-01-31T23:04:58.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:58.981146+0000) 2022-01-31T23:04:59.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:58 smithi171 conmon[46715]: debug 2022-01-31T23:04:58.965+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:58.966546+0000) 2022-01-31T23:04:59.453 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:04:59 smithi167 conmon[54076]: debug 2022-01-31T23:04:59.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:59.221608+0000) 2022-01-31T23:04:59.720 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:04:59 smithi171 conmon[41853]: debug 2022-01-31T23:04:59.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:59.420487+0000) 2022-01-31T23:04:59.721 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:04:59 smithi171 conmon[51620]: debug 2022-01-31T23:04:59.595+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:59.596220+0000) 2022-01-31T23:05:00.161 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:00 smithi167 conmon[49112]: debug 2022-01-31T23:05:00.051+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:00.052524+0000) 2022-01-31T23:05:00.162 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:04:59 smithi167 conmon[60316]: debug 2022-01-31T23:04:59.979+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:59.981360+0000) 2022-01-31T23:05:00.249 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:04:59 smithi171 conmon[46715]: debug 2022-01-31T23:04:59.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:04:59.966765+0000) 2022-01-31T23:05:00.427 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:00 smithi167 conmon[54076]: debug 2022-01-31T23:05:00.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:00.221796+0000) 2022-01-31T23:05:00.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:00 smithi171 conmon[41853]: debug 2022-01-31T23:05:00.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:00.420698+0000) 2022-01-31T23:05:00.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:00 smithi171 conmon[51620]: debug 2022-01-31T23:05:00.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:00.596415+0000) 2022-01-31T23:05:01.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:01 smithi167 conmon[49112]: debug 2022-01-31T23:05:01.051+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:01.052687+0000) 2022-01-31T23:05:01.162 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:00 smithi167 conmon[60316]: debug 2022-01-31T23:05:00.980+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:00.981549+0000) 2022-01-31T23:05:01.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:00 smithi171 conmon[46715]: debug 2022-01-31T23:05:00.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:00.966984+0000) 2022-01-31T23:05:01.455 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:01 smithi167 conmon[54076]: debug 2022-01-31T23:05:01.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:01.222001+0000) 2022-01-31T23:05:01.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:01 smithi171 conmon[41853]: debug 2022-01-31T23:05:01.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:01.420874+0000) 2022-01-31T23:05:01.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:01 smithi171 conmon[51620]: debug 2022-01-31T23:05:01.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:01.596602+0000) 2022-01-31T23:05:02.162 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:02 smithi167 conmon[49112]: debug 2022-01-31T23:05:02.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.052846+0000) 2022-01-31T23:05:02.163 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:01 smithi167 conmon[60316]: debug 2022-01-31T23:05:01.980+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:01.981781+0000) 2022-01-31T23:05:02.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:01 smithi171 conmon[46715]: debug 2022-01-31T23:05:01.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:01.967183+0000) 2022-01-31T23:05:02.454 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:02 smithi167 conmon[54076]: debug 2022-01-31T23:05:02.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.222149+0000) 2022-01-31T23:05:02.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[35325]: debug 2022-01-31T23:05:02.634+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261238 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:02.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[41853]: debug 2022-01-31T23:05:02.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.421059+0000) 2022-01-31T23:05:02.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[41853]: debug 2022-01-31T23:05:02.605+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.606508+0000) 2022-01-31T23:05:02.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[46715]: debug 2022-01-31T23:05:02.606+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.606939+0000) 2022-01-31T23:05:02.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[51620]: debug 2022-01-31T23:05:02.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.596808+0000) 2022-01-31T23:05:02.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[51620]: debug 2022-01-31T23:05:02.607+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.608589+0000) 2022-01-31T23:05:02.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:02 smithi167 conmon[49112]: debug 2022-01-31T23:05:02.606+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.607884+0000) 2022-01-31T23:05:02.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:02 smithi167 conmon[54076]: debug 2022-01-31T23:05:02.607+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.608669+0000) 2022-01-31T23:05:02.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:02 smithi167 conmon[60316]: debug 2022-01-31T23:05:02.606+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.608285+0000) 2022-01-31T23:05:03.205 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:03 smithi167 conmon[49112]: debug 2022-01-31T23:05:03.051+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:03.053018+0000) 2022-01-31T23:05:03.205 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:02 smithi167 conmon[60316]: debug 2022-01-31T23:05:02.981+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.981969+0000) 2022-01-31T23:05:03.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:02 smithi171 conmon[46715]: debug 2022-01-31T23:05:02.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:02.967424+0000) 2022-01-31T23:05:03.561 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:03 smithi167 conmon[54076]: debug 2022-01-31T23:05:03.220+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:03.222329+0000) 2022-01-31T23:05:03.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:03 smithi171 conmon[41853]: debug 2022-01-31T23:05:03.420+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:03.421236+0000) 2022-01-31T23:05:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:03 smithi171 conmon[51620]: debug 2022-01-31T23:05:03.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:03.596953+0000) 2022-01-31T23:05:04.205 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:04 smithi167 conmon[49112]: debug 2022-01-31T23:05:04.051+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:04.053168+0000) 2022-01-31T23:05:04.206 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:03 smithi167 conmon[60316]: debug 2022-01-31T23:05:03.980+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:03.982086+0000) 2022-01-31T23:05:04.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:03 smithi171 conmon[46715]: debug 2022-01-31T23:05:03.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:03.967526+0000) 2022-01-31T23:05:04.561 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:04 smithi167 conmon[54076]: debug 2022-01-31T23:05:04.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:04.222533+0000) 2022-01-31T23:05:04.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:04 smithi171 conmon[41853]: debug 2022-01-31T23:05:04.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:04.421362+0000) 2022-01-31T23:05:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:04 smithi171 conmon[51620]: debug 2022-01-31T23:05:04.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:04.597104+0000) 2022-01-31T23:05:05.205 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:05 smithi167 conmon[49112]: debug 2022-01-31T23:05:05.051+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:05.053389+0000) 2022-01-31T23:05:05.206 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:04 smithi167 conmon[60316]: debug 2022-01-31T23:05:04.980+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:04.982334+0000) 2022-01-31T23:05:05.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:04 smithi171 conmon[46715]: debug 2022-01-31T23:05:04.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:04.967717+0000) 2022-01-31T23:05:05.562 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:05 smithi167 conmon[54076]: debug 2022-01-31T23:05:05.221+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:05.222735+0000) 2022-01-31T23:05:05.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:05 smithi171 conmon[41853]: debug 2022-01-31T23:05:05.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:05.421575+0000) 2022-01-31T23:05:05.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:05 smithi171 conmon[51620]: debug 2022-01-31T23:05:05.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:05.597264+0000) 2022-01-31T23:05:06.205 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:06 smithi167 conmon[49112]: debug 2022-01-31T23:05:06.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:06.053595+0000) 2022-01-31T23:05:06.206 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:05 smithi167 conmon[60316]: debug 2022-01-31T23:05:05.981+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:05.982495+0000) 2022-01-31T23:05:06.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:05 smithi171 conmon[46715]: debug 2022-01-31T23:05:05.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:05.967904+0000) 2022-01-31T23:05:06.561 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:06 smithi167 conmon[54076]: debug 2022-01-31T23:05:06.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:06.222939+0000) 2022-01-31T23:05:06.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:06 smithi171 conmon[41853]: debug 2022-01-31T23:05:06.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:06.421723+0000) 2022-01-31T23:05:06.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:06 smithi171 conmon[51620]: debug 2022-01-31T23:05:06.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:06.597423+0000) 2022-01-31T23:05:07.205 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:07 smithi167 conmon[49112]: debug 2022-01-31T23:05:07.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.053716+0000) 2022-01-31T23:05:07.206 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:06 smithi167 conmon[60316]: debug 2022-01-31T23:05:06.981+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:06.982659+0000) 2022-01-31T23:05:07.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:06 smithi171 conmon[46715]: debug 2022-01-31T23:05:06.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:06.968098+0000) 2022-01-31T23:05:07.562 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:07 smithi167 conmon[54076]: debug 2022-01-31T23:05:07.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.223116+0000) 2022-01-31T23:05:07.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[35325]: debug 2022-01-31T23:05:07.664+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261349 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:07.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[41853]: debug 2022-01-31T23:05:07.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.421803+0000) 2022-01-31T23:05:07.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[41853]: debug 2022-01-31T23:05:07.636+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.637042+0000) 2022-01-31T23:05:07.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[46715]: debug 2022-01-31T23:05:07.635+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.636579+0000) 2022-01-31T23:05:07.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[51620]: debug 2022-01-31T23:05:07.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.597613+0000) 2022-01-31T23:05:07.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[51620]: debug 2022-01-31T23:05:07.636+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.636875+0000) 2022-01-31T23:05:07.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:07 smithi167 conmon[49112]: debug 2022-01-31T23:05:07.636+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.637534+0000) 2022-01-31T23:05:07.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:07 smithi167 conmon[54076]: debug 2022-01-31T23:05:07.636+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.637760+0000) 2022-01-31T23:05:07.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:07 smithi167 conmon[60316]: debug 2022-01-31T23:05:07.636+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.637712+0000) 2022-01-31T23:05:08.206 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:08 smithi167 conmon[49112]: debug 2022-01-31T23:05:08.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:08.053862+0000) 2022-01-31T23:05:08.206 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:07 smithi167 conmon[60316]: debug 2022-01-31T23:05:07.982+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.982877+0000) 2022-01-31T23:05:08.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:07 smithi171 conmon[46715]: debug 2022-01-31T23:05:07.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:07.968305+0000) 2022-01-31T23:05:08.562 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:08 smithi167 conmon[54076]: debug 2022-01-31T23:05:08.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:08.223307+0000) 2022-01-31T23:05:08.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:08 smithi171 conmon[41853]: debug 2022-01-31T23:05:08.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:08.421997+0000) 2022-01-31T23:05:08.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:08 smithi171 conmon[51620]: debug 2022-01-31T23:05:08.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:08.597828+0000) 2022-01-31T23:05:09.206 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:09 smithi167 conmon[49112]: debug 2022-01-31T23:05:09.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:09.053995+0000) 2022-01-31T23:05:09.207 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:08 smithi167 conmon[60316]: debug 2022-01-31T23:05:08.981+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:08.983004+0000) 2022-01-31T23:05:09.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:08 smithi171 conmon[35325]: debug 2022-01-31T23:05:08.958+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:05:09.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:08 smithi171 conmon[46715]: debug 2022-01-31T23:05:08.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:08.968472+0000) 2022-01-31T23:05:09.562 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:09 smithi167 conmon[54076]: debug 2022-01-31T23:05:09.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:09.223492+0000) 2022-01-31T23:05:09.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:09 smithi171 conmon[41853]: debug 2022-01-31T23:05:09.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:09.422176+0000) 2022-01-31T23:05:09.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:09 smithi171 conmon[51620]: debug 2022-01-31T23:05:09.596+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:09.598003+0000) 2022-01-31T23:05:10.206 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:10 smithi167 conmon[49112]: debug 2022-01-31T23:05:10.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:10.054162+0000) 2022-01-31T23:05:10.207 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:09 smithi167 conmon[60316]: debug 2022-01-31T23:05:09.981+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:09.983181+0000) 2022-01-31T23:05:10.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:09 smithi171 conmon[46715]: debug 2022-01-31T23:05:09.966+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:09.968676+0000) 2022-01-31T23:05:10.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:10 smithi167 conmon[54076]: debug 2022-01-31T23:05:10.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:10.223696+0000) 2022-01-31T23:05:10.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:10 smithi171 conmon[41853]: debug 2022-01-31T23:05:10.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:10.422390+0000) 2022-01-31T23:05:10.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:10 smithi171 conmon[51620]: debug 2022-01-31T23:05:10.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:10.598193+0000) 2022-01-31T23:05:11.206 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:11 smithi167 conmon[49112]: debug 2022-01-31T23:05:11.052+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:11.054343+0000) 2022-01-31T23:05:11.207 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:10 smithi167 conmon[60316]: debug 2022-01-31T23:05:10.982+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:10.983371+0000) 2022-01-31T23:05:11.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:10 smithi171 conmon[46715]: debug 2022-01-31T23:05:10.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:10.968778+0000) 2022-01-31T23:05:11.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:11 smithi167 conmon[54076]: debug 2022-01-31T23:05:11.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:11.223833+0000) 2022-01-31T23:05:11.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:11 smithi171 conmon[41853]: debug 2022-01-31T23:05:11.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:11.422542+0000) 2022-01-31T23:05:11.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:11 smithi171 conmon[51620]: debug 2022-01-31T23:05:11.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:11.598396+0000) 2022-01-31T23:05:12.206 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:12 smithi167 conmon[49112]: debug 2022-01-31T23:05:12.053+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.054552+0000) 2022-01-31T23:05:12.207 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:11 smithi167 conmon[60316]: debug 2022-01-31T23:05:11.982+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:11.983586+0000) 2022-01-31T23:05:12.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:11 smithi171 conmon[46715]: debug 2022-01-31T23:05:11.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:11.968887+0000) 2022-01-31T23:05:12.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:12 smithi167 conmon[54076]: debug 2022-01-31T23:05:12.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.223989+0000) 2022-01-31T23:05:12.676 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[46715]: debug 2022-01-31T23:05:12.665+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.667142+0000) 2022-01-31T23:05:12.676 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[51620]: debug 2022-01-31T23:05:12.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.598551+0000) 2022-01-31T23:05:12.677 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[51620]: debug 2022-01-31T23:05:12.665+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.667142+0000) 2022-01-31T23:05:12.677 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[41853]: debug 2022-01-31T23:05:12.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.422721+0000) 2022-01-31T23:05:12.678 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[41853]: debug 2022-01-31T23:05:12.665+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.667491+0000) 2022-01-31T23:05:12.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:12 smithi167 conmon[49112]: debug 2022-01-31T23:05:12.667+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.668777+0000) 2022-01-31T23:05:12.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:12 smithi167 conmon[54076]: debug 2022-01-31T23:05:12.666+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.667631+0000) 2022-01-31T23:05:12.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:12 smithi167 conmon[60316]: debug 2022-01-31T23:05:12.665+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.666962+0000) 2022-01-31T23:05:12.951 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[35325]: debug 2022-01-31T23:05:12.691+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261462 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:13.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:13 smithi167 conmon[49112]: debug 2022-01-31T23:05:13.053+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:13.054729+0000) 2022-01-31T23:05:13.207 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:12 smithi167 conmon[60316]: debug 2022-01-31T23:05:12.982+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.983716+0000) 2022-01-31T23:05:13.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:12 smithi171 conmon[46715]: debug 2022-01-31T23:05:12.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:12.969069+0000) 2022-01-31T23:05:13.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:13 smithi167 conmon[54076]: debug 2022-01-31T23:05:13.222+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:13.224197+0000) 2022-01-31T23:05:13.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:13 smithi171 conmon[41853]: debug 2022-01-31T23:05:13.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:13.422922+0000) 2022-01-31T23:05:14.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:13 smithi171 conmon[51620]: debug 2022-01-31T23:05:13.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:13.598687+0000) 2022-01-31T23:05:14.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:14 smithi167 conmon[49112]: debug 2022-01-31T23:05:14.054+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:14.054872+0000) 2022-01-31T23:05:14.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:13 smithi167 conmon[60316]: debug 2022-01-31T23:05:13.982+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:13.983847+0000) 2022-01-31T23:05:14.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:13 smithi171 conmon[46715]: debug 2022-01-31T23:05:13.967+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:13.969195+0000) 2022-01-31T23:05:14.563 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:14 smithi167 conmon[54076]: debug 2022-01-31T23:05:14.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:14.224348+0000) 2022-01-31T23:05:14.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:14 smithi171 conmon[41853]: debug 2022-01-31T23:05:14.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:14.423070+0000) 2022-01-31T23:05:14.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:14 smithi171 conmon[51620]: debug 2022-01-31T23:05:14.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:14.598819+0000) 2022-01-31T23:05:15.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:15 smithi167 conmon[49112]: debug 2022-01-31T23:05:15.054+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:15.055066+0000) 2022-01-31T23:05:15.207 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:14 smithi167 conmon[60316]: debug 2022-01-31T23:05:14.982+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:14.984032+0000) 2022-01-31T23:05:15.313 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:14 smithi171 conmon[46715]: debug 2022-01-31T23:05:14.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:14.969426+0000) 2022-01-31T23:05:15.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:15 smithi167 conmon[54076]: debug 2022-01-31T23:05:15.223+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:15.224531+0000) 2022-01-31T23:05:15.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:15 smithi171 conmon[41853]: debug 2022-01-31T23:05:15.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:15.423260+0000) 2022-01-31T23:05:15.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:15 smithi171 conmon[51620]: debug 2022-01-31T23:05:15.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:15.599030+0000) 2022-01-31T23:05:16.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:16 smithi167 conmon[49112]: debug 2022-01-31T23:05:16.054+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:16.055235+0000) 2022-01-31T23:05:16.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:15 smithi167 conmon[60316]: debug 2022-01-31T23:05:15.983+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:15.984163+0000) 2022-01-31T23:05:16.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:15 smithi171 conmon[46715]: debug 2022-01-31T23:05:15.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:15.969582+0000) 2022-01-31T23:05:16.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:16 smithi167 conmon[54076]: debug 2022-01-31T23:05:16.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:16.224721+0000) 2022-01-31T23:05:16.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:16 smithi171 conmon[41853]: debug 2022-01-31T23:05:16.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:16.423441+0000) 2022-01-31T23:05:16.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:16 smithi171 conmon[51620]: debug 2022-01-31T23:05:16.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:16.599219+0000) 2022-01-31T23:05:17.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:17 smithi167 conmon[49112]: debug 2022-01-31T23:05:17.054+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.055413+0000) 2022-01-31T23:05:17.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:16 smithi167 conmon[60316]: debug 2022-01-31T23:05:16.983+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:16.984313+0000) 2022-01-31T23:05:17.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:16 smithi171 conmon[46715]: debug 2022-01-31T23:05:16.968+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:16.969781+0000) 2022-01-31T23:05:17.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:17 smithi167 conmon[54076]: debug 2022-01-31T23:05:17.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.224877+0000) 2022-01-31T23:05:17.677 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[51620]: debug 2022-01-31T23:05:17.597+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.599439+0000) 2022-01-31T23:05:17.678 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[41853]: debug 2022-01-31T23:05:17.421+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.423596+0000) 2022-01-31T23:05:17.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:17 smithi167 conmon[49112]: debug 2022-01-31T23:05:17.696+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.696574+0000) 2022-01-31T23:05:17.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:17 smithi167 conmon[54076]: debug 2022-01-31T23:05:17.696+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.697387+0000) 2022-01-31T23:05:17.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:17 smithi167 conmon[60316]: debug 2022-01-31T23:05:17.695+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.695684+0000) 2022-01-31T23:05:17.952 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[35325]: debug 2022-01-31T23:05:17.721+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261573 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:17.953 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[41853]: debug 2022-01-31T23:05:17.695+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.696799+0000) 2022-01-31T23:05:17.953 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[46715]: debug 2022-01-31T23:05:17.693+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.695205+0000) 2022-01-31T23:05:17.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[51620]: debug 2022-01-31T23:05:17.694+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.695982+0000) 2022-01-31T23:05:18.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:18 smithi167 conmon[49112]: debug 2022-01-31T23:05:18.055+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:18.055601+0000) 2022-01-31T23:05:18.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:17 smithi167 conmon[60316]: debug 2022-01-31T23:05:17.983+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.984467+0000) 2022-01-31T23:05:18.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:17 smithi171 conmon[46715]: debug 2022-01-31T23:05:17.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:17.970032+0000) 2022-01-31T23:05:18.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:18 smithi167 conmon[54076]: debug 2022-01-31T23:05:18.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:18.224998+0000) 2022-01-31T23:05:18.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:18 smithi171 conmon[41853]: debug 2022-01-31T23:05:18.422+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:18.423784+0000) 2022-01-31T23:05:18.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:18 smithi171 conmon[51620]: debug 2022-01-31T23:05:18.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:18.599646+0000) 2022-01-31T23:05:19.207 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:19 smithi167 conmon[49112]: debug 2022-01-31T23:05:19.055+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:19.055763+0000) 2022-01-31T23:05:19.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:18 smithi167 conmon[60316]: debug 2022-01-31T23:05:18.984+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:18.984656+0000) 2022-01-31T23:05:19.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:18 smithi171 conmon[46715]: debug 2022-01-31T23:05:18.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:18.970175+0000) 2022-01-31T23:05:19.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:19 smithi167 conmon[54076]: debug 2022-01-31T23:05:19.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:19.225138+0000) 2022-01-31T23:05:19.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:19 smithi171 conmon[41853]: debug 2022-01-31T23:05:19.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:19.423957+0000) 2022-01-31T23:05:19.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:19 smithi171 conmon[51620]: debug 2022-01-31T23:05:19.598+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:19.599828+0000) 2022-01-31T23:05:20.208 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:20 smithi167 conmon[49112]: debug 2022-01-31T23:05:20.055+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:20.055931+0000) 2022-01-31T23:05:20.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:19 smithi167 conmon[60316]: debug 2022-01-31T23:05:19.984+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:19.984859+0000) 2022-01-31T23:05:20.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:19 smithi171 conmon[46715]: debug 2022-01-31T23:05:19.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:19.970389+0000) 2022-01-31T23:05:20.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:20 smithi167 conmon[54076]: debug 2022-01-31T23:05:20.224+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:20.225348+0000) 2022-01-31T23:05:20.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:20 smithi171 conmon[41853]: debug 2022-01-31T23:05:20.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:20.424118+0000) 2022-01-31T23:05:20.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:20 smithi171 conmon[51620]: debug 2022-01-31T23:05:20.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:20.599987+0000) 2022-01-31T23:05:21.208 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:21 smithi167 conmon[49112]: debug 2022-01-31T23:05:21.055+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:21.056143+0000) 2022-01-31T23:05:21.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:20 smithi167 conmon[60316]: debug 2022-01-31T23:05:20.984+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:20.985046+0000) 2022-01-31T23:05:21.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:20 smithi171 conmon[46715]: debug 2022-01-31T23:05:20.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:20.970576+0000) 2022-01-31T23:05:21.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:21 smithi167 conmon[54076]: debug 2022-01-31T23:05:21.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:21.225566+0000) 2022-01-31T23:05:21.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:21 smithi171 conmon[41853]: debug 2022-01-31T23:05:21.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:21.424258+0000) 2022-01-31T23:05:21.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:21 smithi171 conmon[51620]: debug 2022-01-31T23:05:21.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:21.600152+0000) 2022-01-31T23:05:22.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:21 smithi167 conmon[60316]: debug 2022-01-31T23:05:21.984+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:21.985256+0000) 2022-01-31T23:05:22.208 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:22 smithi167 conmon[49112]: debug 2022-01-31T23:05:22.055+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.056357+0000) 2022-01-31T23:05:22.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:21 smithi171 conmon[46715]: debug 2022-01-31T23:05:21.969+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:21.970731+0000) 2022-01-31T23:05:22.565 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:22 smithi167 conmon[54076]: debug 2022-01-31T23:05:22.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.225696+0000) 2022-01-31T23:05:22.707 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[41853]: debug 2022-01-31T23:05:22.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.424408+0000) 2022-01-31T23:05:22.708 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[51620]: debug 2022-01-31T23:05:22.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.600352+0000) 2022-01-31T23:05:22.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:22 smithi167 conmon[49112]: debug 2022-01-31T23:05:22.724+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.725423+0000) 2022-01-31T23:05:22.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:22 smithi167 conmon[54076]: debug 2022-01-31T23:05:22.726+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.726744+0000) 2022-01-31T23:05:22.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:22 smithi167 conmon[60316]: debug 2022-01-31T23:05:22.725+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.726263+0000) 2022-01-31T23:05:23.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[41853]: debug 2022-01-31T23:05:22.725+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.726195+0000) 2022-01-31T23:05:23.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[46715]: debug 2022-01-31T23:05:22.724+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.725697+0000) 2022-01-31T23:05:23.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[46715]: debug 2022-01-31T23:05:22.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.970896+0000) 2022-01-31T23:05:23.092 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[35325]: debug 2022-01-31T23:05:22.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:23.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:22 smithi171 conmon[51620]: debug 2022-01-31T23:05:22.724+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.724907+0000) 2022-01-31T23:05:23.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:22 smithi167 conmon[60316]: debug 2022-01-31T23:05:22.984+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:22.985458+0000) 2022-01-31T23:05:23.209 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:23 smithi167 conmon[49112]: debug 2022-01-31T23:05:23.056+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:23.056573+0000) 2022-01-31T23:05:23.565 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:23 smithi167 conmon[54076]: debug 2022-01-31T23:05:23.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:23.225859+0000) 2022-01-31T23:05:23.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:23 smithi171 conmon[41853]: debug 2022-01-31T23:05:23.423+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:23.424562+0000) 2022-01-31T23:05:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:23 smithi171 conmon[51620]: debug 2022-01-31T23:05:23.599+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:23.600544+0000) 2022-01-31T23:05:24.208 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:24 smithi167 conmon[49112]: debug 2022-01-31T23:05:24.056+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:24.056720+0000) 2022-01-31T23:05:24.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:23 smithi167 conmon[60316]: debug 2022-01-31T23:05:23.985+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:23.985579+0000) 2022-01-31T23:05:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:23 smithi171 conmon[46715]: debug 2022-01-31T23:05:23.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:23.970991+0000) 2022-01-31T23:05:24.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:23 smithi171 conmon[35325]: debug 2022-01-31T23:05:23.959+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:05:24.565 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:24 smithi167 conmon[54076]: debug 2022-01-31T23:05:24.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:24.226035+0000) 2022-01-31T23:05:24.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:24 smithi171 conmon[41853]: debug 2022-01-31T23:05:24.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:24.424706+0000) 2022-01-31T23:05:24.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:24 smithi171 conmon[51620]: debug 2022-01-31T23:05:24.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:24.600792+0000) 2022-01-31T23:05:25.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:24 smithi167 conmon[60316]: debug 2022-01-31T23:05:24.985+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:24.985713+0000) 2022-01-31T23:05:25.209 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:25 smithi167 conmon[49112]: debug 2022-01-31T23:05:25.056+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:25.056913+0000) 2022-01-31T23:05:25.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:24 smithi171 conmon[46715]: debug 2022-01-31T23:05:24.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:24.971184+0000) 2022-01-31T23:05:25.565 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:25 smithi167 conmon[54076]: debug 2022-01-31T23:05:25.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:25.226248+0000) 2022-01-31T23:05:25.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:25 smithi171 conmon[41853]: debug 2022-01-31T23:05:25.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:25.424834+0000) 2022-01-31T23:05:25.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:25 smithi171 conmon[51620]: debug 2022-01-31T23:05:25.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:25.600927+0000) 2022-01-31T23:05:26.209 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:26 smithi167 conmon[49112]: debug 2022-01-31T23:05:26.056+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:26.057106+0000) 2022-01-31T23:05:26.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:25 smithi167 conmon[60316]: debug 2022-01-31T23:05:25.985+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:25.985897+0000) 2022-01-31T23:05:26.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:25 smithi171 conmon[46715]: debug 2022-01-31T23:05:25.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:25.971326+0000) 2022-01-31T23:05:26.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:26 smithi167 conmon[54076]: debug 2022-01-31T23:05:26.225+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:26.226444+0000) 2022-01-31T23:05:26.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:26 smithi171 conmon[41853]: debug 2022-01-31T23:05:26.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:26.425032+0000) 2022-01-31T23:05:26.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:26 smithi171 conmon[51620]: debug 2022-01-31T23:05:26.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:26.601059+0000) 2022-01-31T23:05:27.209 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:27 smithi167 conmon[49112]: debug 2022-01-31T23:05:27.056+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.057286+0000) 2022-01-31T23:05:27.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:26 smithi167 conmon[60316]: debug 2022-01-31T23:05:26.985+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:26.986085+0000) 2022-01-31T23:05:27.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:26 smithi171 conmon[46715]: debug 2022-01-31T23:05:26.970+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:26.971488+0000) 2022-01-31T23:05:27.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:27 smithi167 conmon[54076]: debug 2022-01-31T23:05:27.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.226624+0000) 2022-01-31T23:05:27.738 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[51620]: debug 2022-01-31T23:05:27.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.601212+0000) 2022-01-31T23:05:27.738 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[41853]: debug 2022-01-31T23:05:27.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.425191+0000) 2022-01-31T23:05:27.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:27 smithi167 conmon[49112]: debug 2022-01-31T23:05:27.755+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.755564+0000) 2022-01-31T23:05:27.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:27 smithi167 conmon[54076]: debug 2022-01-31T23:05:27.756+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.756695+0000) 2022-01-31T23:05:27.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:27 smithi167 conmon[60316]: debug 2022-01-31T23:05:27.756+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.757123+0000) 2022-01-31T23:05:28.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[35325]: debug 2022-01-31T23:05:27.782+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261798 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:28.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[41853]: debug 2022-01-31T23:05:27.755+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.756711+0000) 2022-01-31T23:05:28.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[46715]: debug 2022-01-31T23:05:27.755+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.756293+0000) 2022-01-31T23:05:28.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[46715]: debug 2022-01-31T23:05:27.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.971644+0000) 2022-01-31T23:05:28.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:27 smithi171 conmon[51620]: debug 2022-01-31T23:05:27.754+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.755353+0000) 2022-01-31T23:05:28.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:27 smithi167 conmon[60316]: debug 2022-01-31T23:05:27.985+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:27.986287+0000) 2022-01-31T23:05:28.210 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:28 smithi167 conmon[49112]: debug 2022-01-31T23:05:28.056+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:28.057441+0000) 2022-01-31T23:05:28.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:28 smithi167 conmon[54076]: debug 2022-01-31T23:05:28.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:28.226809+0000) 2022-01-31T23:05:28.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:28 smithi171 conmon[41853]: debug 2022-01-31T23:05:28.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:28.425336+0000) 2022-01-31T23:05:28.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:28 smithi171 conmon[51620]: debug 2022-01-31T23:05:28.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:28.601355+0000) 2022-01-31T23:05:29.209 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:29 smithi167 conmon[49112]: debug 2022-01-31T23:05:29.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:29.057537+0000) 2022-01-31T23:05:29.210 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:28 smithi167 conmon[60316]: debug 2022-01-31T23:05:28.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:28.986475+0000) 2022-01-31T23:05:29.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:28 smithi171 conmon[46715]: debug 2022-01-31T23:05:28.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:28.971820+0000) 2022-01-31T23:05:29.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:29 smithi167 conmon[54076]: debug 2022-01-31T23:05:29.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:29.226953+0000) 2022-01-31T23:05:29.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:29 smithi171 conmon[41853]: debug 2022-01-31T23:05:29.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:29.425489+0000) 2022-01-31T23:05:29.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:29 smithi171 conmon[51620]: debug 2022-01-31T23:05:29.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:29.601444+0000) 2022-01-31T23:05:30.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:29 smithi167 conmon[60316]: debug 2022-01-31T23:05:29.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:29.986671+0000) 2022-01-31T23:05:30.210 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:30 smithi167 conmon[49112]: debug 2022-01-31T23:05:30.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:30.057673+0000) 2022-01-31T23:05:30.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:29 smithi171 conmon[46715]: debug 2022-01-31T23:05:29.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:29.972047+0000) 2022-01-31T23:05:30.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:30 smithi167 conmon[54076]: debug 2022-01-31T23:05:30.226+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:30.227102+0000) 2022-01-31T23:05:30.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:30 smithi171 conmon[41853]: debug 2022-01-31T23:05:30.424+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:30.425679+0000) 2022-01-31T23:05:30.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:30 smithi171 conmon[51620]: debug 2022-01-31T23:05:30.600+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:30.601631+0000) 2022-01-31T23:05:31.209 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:31 smithi167 conmon[49112]: debug 2022-01-31T23:05:31.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:31.057840+0000) 2022-01-31T23:05:31.210 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:30 smithi167 conmon[60316]: debug 2022-01-31T23:05:30.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:30.986848+0000) 2022-01-31T23:05:31.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:30 smithi171 conmon[46715]: debug 2022-01-31T23:05:30.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:30.972250+0000) 2022-01-31T23:05:31.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:31 smithi167 conmon[54076]: debug 2022-01-31T23:05:31.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:31.227326+0000) 2022-01-31T23:05:31.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:31 smithi171 conmon[41853]: debug 2022-01-31T23:05:31.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:31.425873+0000) 2022-01-31T23:05:31.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:31 smithi171 conmon[51620]: debug 2022-01-31T23:05:31.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:31.601833+0000) 2022-01-31T23:05:32.210 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:32 smithi167 conmon[49112]: debug 2022-01-31T23:05:32.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.057989+0000) 2022-01-31T23:05:32.211 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:31 smithi167 conmon[60316]: debug 2022-01-31T23:05:31.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:31.987006+0000) 2022-01-31T23:05:32.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:31 smithi171 conmon[46715]: debug 2022-01-31T23:05:31.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:31.972458+0000) 2022-01-31T23:05:32.566 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:32 smithi167 conmon[54076]: debug 2022-01-31T23:05:32.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.227533+0000) 2022-01-31T23:05:32.768 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[41853]: debug 2022-01-31T23:05:32.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.426055+0000) 2022-01-31T23:05:32.768 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[51620]: debug 2022-01-31T23:05:32.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.601982+0000) 2022-01-31T23:05:32.769 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[51620]: debug 2022-01-31T23:05:32.784+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.785282+0000) 2022-01-31T23:05:32.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:32 smithi167 conmon[49112]: debug 2022-01-31T23:05:32.809+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.809537+0000) 2022-01-31T23:05:32.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:32 smithi167 conmon[54076]: debug 2022-01-31T23:05:32.786+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.787345+0000) 2022-01-31T23:05:32.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:32 smithi167 conmon[60316]: debug 2022-01-31T23:05:32.786+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.786991+0000) 2022-01-31T23:05:33.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[35325]: debug 2022-01-31T23:05:32.818+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 261911 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:33.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[41853]: debug 2022-01-31T23:05:32.786+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.786788+0000) 2022-01-31T23:05:33.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[46715]: debug 2022-01-31T23:05:32.786+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.787066+0000) 2022-01-31T23:05:33.092 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:32 smithi171 conmon[46715]: debug 2022-01-31T23:05:32.971+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.972616+0000) 2022-01-31T23:05:33.210 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:32 smithi167 conmon[60316]: debug 2022-01-31T23:05:32.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:32.987152+0000) 2022-01-31T23:05:33.211 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:33 smithi167 conmon[49112]: debug 2022-01-31T23:05:33.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:33.058158+0000) 2022-01-31T23:05:33.567 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:33 smithi167 conmon[54076]: debug 2022-01-31T23:05:33.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:33.227681+0000) 2022-01-31T23:05:33.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:33 smithi171 conmon[41853]: debug 2022-01-31T23:05:33.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:33.426241+0000) 2022-01-31T23:05:33.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:33 smithi171 conmon[51620]: debug 2022-01-31T23:05:33.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:33.602133+0000) 2022-01-31T23:05:34.210 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:34 smithi167 conmon[49112]: debug 2022-01-31T23:05:34.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:34.058300+0000) 2022-01-31T23:05:34.211 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:33 smithi167 conmon[60316]: debug 2022-01-31T23:05:33.986+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:33.987268+0000) 2022-01-31T23:05:34.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:33 smithi171 conmon[46715]: debug 2022-01-31T23:05:33.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:33.972795+0000) 2022-01-31T23:05:34.567 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:34 smithi167 conmon[54076]: debug 2022-01-31T23:05:34.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:34.227803+0000) 2022-01-31T23:05:34.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:34 smithi171 conmon[51620]: debug 2022-01-31T23:05:34.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:34.602267+0000) 2022-01-31T23:05:34.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:34 smithi171 conmon[41853]: debug 2022-01-31T23:05:34.425+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:34.426391+0000) 2022-01-31T23:05:35.210 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:35 smithi167 conmon[49112]: debug 2022-01-31T23:05:35.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:35.058436+0000) 2022-01-31T23:05:35.211 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:34 smithi167 conmon[60316]: debug 2022-01-31T23:05:34.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:34.987453+0000) 2022-01-31T23:05:35.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:34 smithi171 conmon[46715]: debug 2022-01-31T23:05:34.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:34.973009+0000) 2022-01-31T23:05:35.567 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:35 smithi167 conmon[54076]: debug 2022-01-31T23:05:35.227+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:35.227992+0000) 2022-01-31T23:05:35.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:35 smithi171 conmon[51620]: debug 2022-01-31T23:05:35.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:35.602471+0000) 2022-01-31T23:05:35.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:35 smithi171 conmon[41853]: debug 2022-01-31T23:05:35.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:35.426577+0000) 2022-01-31T23:05:36.210 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:36 smithi167 conmon[49112]: debug 2022-01-31T23:05:36.058+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:36.058543+0000) 2022-01-31T23:05:36.211 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:35 smithi167 conmon[60316]: debug 2022-01-31T23:05:35.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:35.987621+0000) 2022-01-31T23:05:36.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:35 smithi171 conmon[46715]: debug 2022-01-31T23:05:35.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:35.973197+0000) 2022-01-31T23:05:36.567 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:36 smithi167 conmon[54076]: debug 2022-01-31T23:05:36.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:36.228194+0000) 2022-01-31T23:05:36.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:36 smithi171 conmon[41853]: debug 2022-01-31T23:05:36.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:36.426793+0000) 2022-01-31T23:05:36.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:36 smithi171 conmon[51620]: debug 2022-01-31T23:05:36.601+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:36.602634+0000) 2022-01-31T23:05:37.211 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:37 smithi167 conmon[49112]: debug 2022-01-31T23:05:37.058+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.058704+0000) 2022-01-31T23:05:37.212 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:36 smithi167 conmon[60316]: debug 2022-01-31T23:05:36.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:36.987770+0000) 2022-01-31T23:05:37.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:36 smithi171 conmon[46715]: debug 2022-01-31T23:05:36.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:36.973348+0000) 2022-01-31T23:05:37.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:37 smithi167 conmon[54076]: debug 2022-01-31T23:05:37.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.228380+0000) 2022-01-31T23:05:37.805 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[41853]: debug 2022-01-31T23:05:37.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.426985+0000) 2022-01-31T23:05:37.806 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[46715]: debug 2022-01-31T23:05:37.821+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.822098+0000) 2022-01-31T23:05:37.806 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[51620]: debug 2022-01-31T23:05:37.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.602837+0000) 2022-01-31T23:05:37.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:37 smithi167 conmon[49112]: debug 2022-01-31T23:05:37.822+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.823430+0000) 2022-01-31T23:05:37.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:37 smithi167 conmon[54076]: debug 2022-01-31T23:05:37.823+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.823563+0000) 2022-01-31T23:05:37.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:37 smithi167 conmon[60316]: debug 2022-01-31T23:05:37.822+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.822617+0000) 2022-01-31T23:05:38.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[35325]: debug 2022-01-31T23:05:37.849+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262022 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:38.090 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[41853]: debug 2022-01-31T23:05:37.821+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.822305+0000) 2022-01-31T23:05:38.091 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[51620]: debug 2022-01-31T23:05:37.822+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.823395+0000) 2022-01-31T23:05:38.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:37 smithi171 conmon[46715]: debug 2022-01-31T23:05:37.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.973534+0000) 2022-01-31T23:05:38.211 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:38 smithi167 conmon[49112]: debug 2022-01-31T23:05:38.058+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:38.058881+0000) 2022-01-31T23:05:38.211 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:37 smithi167 conmon[60316]: debug 2022-01-31T23:05:37.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:37.987962+0000) 2022-01-31T23:05:38.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:38 smithi167 conmon[54076]: debug 2022-01-31T23:05:38.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:38.228580+0000) 2022-01-31T23:05:38.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:38 smithi171 conmon[41853]: debug 2022-01-31T23:05:38.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:38.427161+0000) 2022-01-31T23:05:38.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:38 smithi171 conmon[51620]: debug 2022-01-31T23:05:38.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:38.603056+0000) 2022-01-31T23:05:39.211 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:38 smithi167 conmon[60316]: debug 2022-01-31T23:05:38.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:38.988140+0000) 2022-01-31T23:05:39.212 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:39 smithi167 conmon[49112]: debug 2022-01-31T23:05:39.058+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:39.059029+0000) 2022-01-31T23:05:39.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:38 smithi171 conmon[46715]: debug 2022-01-31T23:05:38.972+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:38.973682+0000) 2022-01-31T23:05:39.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:38 smithi171 conmon[35325]: debug 2022-01-31T23:05:38.960+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:05:39.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:39 smithi167 conmon[54076]: debug 2022-01-31T23:05:39.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:39.228745+0000) 2022-01-31T23:05:39.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:39 smithi171 conmon[41853]: debug 2022-01-31T23:05:39.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:39.427325+0000) 2022-01-31T23:05:39.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:39 smithi171 conmon[51620]: debug 2022-01-31T23:05:39.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:39.603218+0000) 2022-01-31T23:05:40.211 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:40 smithi167 conmon[49112]: debug 2022-01-31T23:05:40.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:40.059253+0000) 2022-01-31T23:05:40.212 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:39 smithi167 conmon[60316]: debug 2022-01-31T23:05:39.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:39.988328+0000) 2022-01-31T23:05:40.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:39 smithi171 conmon[46715]: debug 2022-01-31T23:05:39.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:39.973871+0000) 2022-01-31T23:05:40.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:40 smithi167 conmon[54076]: debug 2022-01-31T23:05:40.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:40.228912+0000) 2022-01-31T23:05:40.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:40 smithi171 conmon[51620]: debug 2022-01-31T23:05:40.602+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:40.603473+0000) 2022-01-31T23:05:40.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:40 smithi171 conmon[41853]: debug 2022-01-31T23:05:40.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:40.427520+0000) 2022-01-31T23:05:41.211 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:41 smithi167 conmon[49112]: debug 2022-01-31T23:05:41.057+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:41.059432+0000) 2022-01-31T23:05:41.212 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:40 smithi167 conmon[60316]: debug 2022-01-31T23:05:40.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:40.988526+0000) 2022-01-31T23:05:41.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:40 smithi171 conmon[46715]: debug 2022-01-31T23:05:40.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:40.974054+0000) 2022-01-31T23:05:41.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:41 smithi167 conmon[54076]: debug 2022-01-31T23:05:41.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:41.229081+0000) 2022-01-31T23:05:41.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:41 smithi171 conmon[51620]: debug 2022-01-31T23:05:41.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:41.603665+0000) 2022-01-31T23:05:41.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:41 smithi171 conmon[41853]: debug 2022-01-31T23:05:41.426+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:41.427715+0000) 2022-01-31T23:05:42.211 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:42 smithi167 conmon[49112]: debug 2022-01-31T23:05:42.058+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.059625+0000) 2022-01-31T23:05:42.212 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:41 smithi167 conmon[60316]: debug 2022-01-31T23:05:41.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:41.988707+0000) 2022-01-31T23:05:42.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:41 smithi171 conmon[46715]: debug 2022-01-31T23:05:41.973+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:41.974242+0000) 2022-01-31T23:05:42.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:42 smithi167 conmon[54076]: debug 2022-01-31T23:05:42.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.229287+0000) 2022-01-31T23:05:42.835 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[51620]: debug 2022-01-31T23:05:42.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.603852+0000) 2022-01-31T23:05:42.835 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[41853]: debug 2022-01-31T23:05:42.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.427876+0000) 2022-01-31T23:05:43.090 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[35325]: debug 2022-01-31T23:05:42.880+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262134 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:43.091 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[41853]: debug 2022-01-31T23:05:42.852+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.853214+0000) 2022-01-31T23:05:43.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[46715]: debug 2022-01-31T23:05:42.851+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.852443+0000) 2022-01-31T23:05:43.091 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[46715]: debug 2022-01-31T23:05:42.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.974458+0000) 2022-01-31T23:05:43.092 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:42 smithi171 conmon[51620]: debug 2022-01-31T23:05:42.851+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.852644+0000) 2022-01-31T23:05:43.151 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:42 smithi167 conmon[54076]: debug 2022-01-31T23:05:42.852+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.853667+0000) 2022-01-31T23:05:43.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:42 smithi167 conmon[49112]: debug 2022-01-31T23:05:42.851+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.853385+0000) 2022-01-31T23:05:43.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:43 smithi167 conmon[49112]: debug 2022-01-31T23:05:43.058+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:43.059816+0000) 2022-01-31T23:05:43.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:42 smithi167 conmon[60316]: debug 2022-01-31T23:05:42.852+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.853534+0000) 2022-01-31T23:05:43.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:42 smithi167 conmon[60316]: debug 2022-01-31T23:05:42.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:42.988872+0000) 2022-01-31T23:05:43.461 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:43 smithi167 conmon[54076]: debug 2022-01-31T23:05:43.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:43.229474+0000) 2022-01-31T23:05:43.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:43 smithi171 conmon[41853]: debug 2022-01-31T23:05:43.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:43.428056+0000) 2022-01-31T23:05:43.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:43 smithi171 conmon[51620]: debug 2022-01-31T23:05:43.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:43.604064+0000) 2022-01-31T23:05:44.212 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:44 smithi167 conmon[49112]: debug 2022-01-31T23:05:44.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:44.059969+0000) 2022-01-31T23:05:44.213 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:43 smithi167 conmon[60316]: debug 2022-01-31T23:05:43.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:43.988960+0000) 2022-01-31T23:05:44.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:43 smithi171 conmon[46715]: debug 2022-01-31T23:05:43.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:43.974613+0000) 2022-01-31T23:05:44.569 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:44 smithi167 conmon[54076]: debug 2022-01-31T23:05:44.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:44.229659+0000) 2022-01-31T23:05:44.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:44 smithi171 conmon[41853]: debug 2022-01-31T23:05:44.427+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:44.428188+0000) 2022-01-31T23:05:44.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:44 smithi171 conmon[51620]: debug 2022-01-31T23:05:44.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:44.604209+0000) 2022-01-31T23:05:45.212 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:45 smithi167 conmon[49112]: debug 2022-01-31T23:05:45.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:45.060165+0000) 2022-01-31T23:05:45.213 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:44 smithi167 conmon[60316]: debug 2022-01-31T23:05:44.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:44.989139+0000) 2022-01-31T23:05:45.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:44 smithi171 conmon[46715]: debug 2022-01-31T23:05:44.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:44.974818+0000) 2022-01-31T23:05:45.569 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:45 smithi167 conmon[54076]: debug 2022-01-31T23:05:45.228+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:45.229831+0000) 2022-01-31T23:05:45.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:45 smithi171 conmon[51620]: debug 2022-01-31T23:05:45.603+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:45.604358+0000) 2022-01-31T23:05:45.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:45 smithi171 conmon[41853]: debug 2022-01-31T23:05:45.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:45.428352+0000) 2022-01-31T23:05:46.212 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:46 smithi167 conmon[49112]: debug 2022-01-31T23:05:46.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:46.060370+0000) 2022-01-31T23:05:46.213 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:45 smithi167 conmon[60316]: debug 2022-01-31T23:05:45.987+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:45.989343+0000) 2022-01-31T23:05:46.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:45 smithi171 conmon[46715]: debug 2022-01-31T23:05:45.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:45.975001+0000) 2022-01-31T23:05:46.569 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:46 smithi167 conmon[54076]: debug 2022-01-31T23:05:46.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:46.230016+0000) 2022-01-31T23:05:46.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:46 smithi171 conmon[41853]: debug 2022-01-31T23:05:46.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:46.428492+0000) 2022-01-31T23:05:46.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:46 smithi171 conmon[51620]: debug 2022-01-31T23:05:46.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:46.604519+0000) 2022-01-31T23:05:47.212 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:47 smithi167 conmon[49112]: debug 2022-01-31T23:05:47.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.060527+0000) 2022-01-31T23:05:47.213 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:46 smithi167 conmon[60316]: debug 2022-01-31T23:05:46.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:46.989566+0000) 2022-01-31T23:05:47.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:46 smithi171 conmon[46715]: debug 2022-01-31T23:05:46.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:46.975188+0000) 2022-01-31T23:05:47.569 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:47 smithi167 conmon[54076]: debug 2022-01-31T23:05:47.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.230214+0000) 2022-01-31T23:05:47.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[41853]: debug 2022-01-31T23:05:47.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.428647+0000) 2022-01-31T23:05:47.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[51620]: debug 2022-01-31T23:05:47.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.604711+0000) 2022-01-31T23:05:48.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:47 smithi167 conmon[49112]: debug 2022-01-31T23:05:47.883+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.884676+0000) 2022-01-31T23:05:48.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:48 smithi167 conmon[49112]: debug 2022-01-31T23:05:48.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:48.060626+0000) 2022-01-31T23:05:48.153 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:47 smithi167 conmon[54076]: debug 2022-01-31T23:05:47.882+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.883921+0000) 2022-01-31T23:05:48.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:47 smithi167 conmon[60316]: debug 2022-01-31T23:05:47.881+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.883436+0000) 2022-01-31T23:05:48.154 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:47 smithi167 conmon[60316]: debug 2022-01-31T23:05:47.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.989723+0000) 2022-01-31T23:05:48.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[51620]: debug 2022-01-31T23:05:47.882+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.883115+0000) 2022-01-31T23:05:48.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[35325]: debug 2022-01-31T23:05:47.910+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262245 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:48.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[41853]: debug 2022-01-31T23:05:47.882+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.883312+0000) 2022-01-31T23:05:48.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[46715]: debug 2022-01-31T23:05:47.883+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.883997+0000) 2022-01-31T23:05:48.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:47 smithi171 conmon[46715]: debug 2022-01-31T23:05:47.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:47.975351+0000) 2022-01-31T23:05:48.461 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:48 smithi167 conmon[54076]: debug 2022-01-31T23:05:48.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:48.230392+0000) 2022-01-31T23:05:48.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:48 smithi171 conmon[41853]: debug 2022-01-31T23:05:48.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:48.428832+0000) 2022-01-31T23:05:48.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:48 smithi171 conmon[51620]: debug 2022-01-31T23:05:48.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:48.604871+0000) 2022-01-31T23:05:49.213 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:49 smithi167 conmon[49112]: debug 2022-01-31T23:05:49.059+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:49.060788+0000) 2022-01-31T23:05:49.213 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:48 smithi167 conmon[60316]: debug 2022-01-31T23:05:48.988+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:48.989843+0000) 2022-01-31T23:05:49.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:48 smithi171 conmon[46715]: debug 2022-01-31T23:05:48.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:48.975488+0000) 2022-01-31T23:05:49.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:49 smithi167 conmon[54076]: debug 2022-01-31T23:05:49.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:49.230529+0000) 2022-01-31T23:05:49.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:49 smithi171 conmon[51620]: debug 2022-01-31T23:05:49.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:49.605026+0000) 2022-01-31T23:05:49.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:49 smithi171 conmon[41853]: debug 2022-01-31T23:05:49.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:49.429004+0000) 2022-01-31T23:05:50.213 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:50 smithi167 conmon[49112]: debug 2022-01-31T23:05:50.060+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:50.060989+0000) 2022-01-31T23:05:50.213 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:49 smithi167 conmon[60316]: debug 2022-01-31T23:05:49.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:49.990060+0000) 2022-01-31T23:05:50.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:49 smithi171 conmon[46715]: debug 2022-01-31T23:05:49.974+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:49.975664+0000) 2022-01-31T23:05:50.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:50 smithi167 conmon[54076]: debug 2022-01-31T23:05:50.229+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:50.230686+0000) 2022-01-31T23:05:50.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:50 smithi171 conmon[51620]: debug 2022-01-31T23:05:50.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:50.605222+0000) 2022-01-31T23:05:50.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:50 smithi171 conmon[41853]: debug 2022-01-31T23:05:50.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:50.429164+0000) 2022-01-31T23:05:51.213 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:51 smithi167 conmon[49112]: debug 2022-01-31T23:05:51.060+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:51.061170+0000) 2022-01-31T23:05:51.214 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:50 smithi167 conmon[60316]: debug 2022-01-31T23:05:50.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:50.990262+0000) 2022-01-31T23:05:51.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:50 smithi171 conmon[46715]: debug 2022-01-31T23:05:50.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:50.975821+0000) 2022-01-31T23:05:51.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:51 smithi167 conmon[54076]: debug 2022-01-31T23:05:51.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:51.230930+0000) 2022-01-31T23:05:51.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:51 smithi171 conmon[41853]: debug 2022-01-31T23:05:51.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:51.429282+0000) 2022-01-31T23:05:51.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:51 smithi171 conmon[51620]: debug 2022-01-31T23:05:51.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:51.605407+0000) 2022-01-31T23:05:52.213 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:52 smithi167 conmon[49112]: debug 2022-01-31T23:05:52.060+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.061341+0000) 2022-01-31T23:05:52.214 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:51 smithi167 conmon[60316]: debug 2022-01-31T23:05:51.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:51.990481+0000) 2022-01-31T23:05:52.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:51 smithi171 conmon[46715]: debug 2022-01-31T23:05:51.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:51.976069+0000) 2022-01-31T23:05:52.570 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:52 smithi167 conmon[54076]: debug 2022-01-31T23:05:52.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.231205+0000) 2022-01-31T23:05:52.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[41853]: debug 2022-01-31T23:05:52.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.429525+0000) 2022-01-31T23:05:52.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[51620]: debug 2022-01-31T23:05:52.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.605613+0000) 2022-01-31T23:05:53.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:52 smithi167 conmon[54076]: debug 2022-01-31T23:05:52.913+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.915107+0000) 2022-01-31T23:05:53.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:52 smithi167 conmon[60316]: debug 2022-01-31T23:05:52.913+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.914632+0000) 2022-01-31T23:05:53.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:52 smithi167 conmon[60316]: debug 2022-01-31T23:05:52.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.990674+0000) 2022-01-31T23:05:53.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:52 smithi167 conmon[49112]: debug 2022-01-31T23:05:52.914+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.915880+0000) 2022-01-31T23:05:53.154 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:53 smithi167 conmon[49112]: debug 2022-01-31T23:05:53.060+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:53.061526+0000) 2022-01-31T23:05:53.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[35325]: debug 2022-01-31T23:05:52.941+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262358 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:53.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[41853]: debug 2022-01-31T23:05:52.912+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.913409+0000) 2022-01-31T23:05:53.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[46715]: debug 2022-01-31T23:05:52.913+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.913792+0000) 2022-01-31T23:05:53.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[46715]: debug 2022-01-31T23:05:52.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.976261+0000) 2022-01-31T23:05:53.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:52 smithi171 conmon[51620]: debug 2022-01-31T23:05:52.913+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:52.914027+0000) 2022-01-31T23:05:53.462 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:53 smithi167 conmon[54076]: debug 2022-01-31T23:05:53.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:53.231368+0000) 2022-01-31T23:05:53.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:53 smithi171 conmon[51620]: debug 2022-01-31T23:05:53.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:53.605790+0000) 2022-01-31T23:05:53.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:53 smithi171 conmon[41853]: debug 2022-01-31T23:05:53.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:53.429704+0000) 2022-01-31T23:05:54.214 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:54 smithi167 conmon[49112]: debug 2022-01-31T23:05:54.060+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:54.061628+0000) 2022-01-31T23:05:54.215 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:53 smithi167 conmon[60316]: debug 2022-01-31T23:05:53.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:53.990791+0000) 2022-01-31T23:05:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:53 smithi171 conmon[46715]: debug 2022-01-31T23:05:53.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:53.976452+0000) 2022-01-31T23:05:54.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:53 smithi171 conmon[35325]: debug 2022-01-31T23:05:53.961+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:05:54.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:54 smithi167 conmon[54076]: debug 2022-01-31T23:05:54.230+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:54.231558+0000) 2022-01-31T23:05:54.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:54 smithi171 conmon[51620]: debug 2022-01-31T23:05:54.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:54.605932+0000) 2022-01-31T23:05:54.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:54 smithi171 conmon[41853]: debug 2022-01-31T23:05:54.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:54.429874+0000) 2022-01-31T23:05:55.214 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:55 smithi167 conmon[49112]: debug 2022-01-31T23:05:55.061+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:55.061805+0000) 2022-01-31T23:05:55.214 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:54 smithi167 conmon[60316]: debug 2022-01-31T23:05:54.989+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:54.990927+0000) 2022-01-31T23:05:55.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:54 smithi171 conmon[46715]: debug 2022-01-31T23:05:54.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:54.976662+0000) 2022-01-31T23:05:55.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:55 smithi167 conmon[54076]: debug 2022-01-31T23:05:55.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:55.231749+0000) 2022-01-31T23:05:55.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:55 smithi171 conmon[41853]: debug 2022-01-31T23:05:55.428+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:55.429974+0000) 2022-01-31T23:05:55.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:55 smithi171 conmon[51620]: debug 2022-01-31T23:05:55.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:55.606122+0000) 2022-01-31T23:05:56.214 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:56 smithi167 conmon[49112]: debug 2022-01-31T23:05:56.061+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:56.061992+0000) 2022-01-31T23:05:56.215 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:55 smithi167 conmon[60316]: debug 2022-01-31T23:05:55.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:55.991111+0000) 2022-01-31T23:05:56.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:55 smithi171 conmon[46715]: debug 2022-01-31T23:05:55.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:55.976856+0000) 2022-01-31T23:05:56.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:56 smithi167 conmon[54076]: debug 2022-01-31T23:05:56.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:56.231922+0000) 2022-01-31T23:05:56.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:56 smithi171 conmon[41853]: debug 2022-01-31T23:05:56.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:56.430129+0000) 2022-01-31T23:05:56.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:56 smithi171 conmon[51620]: debug 2022-01-31T23:05:56.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:56.606265+0000) 2022-01-31T23:05:57.214 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:57 smithi167 conmon[49112]: debug 2022-01-31T23:05:57.061+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.062173+0000) 2022-01-31T23:05:57.215 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:56 smithi167 conmon[60316]: debug 2022-01-31T23:05:56.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:56.991293+0000) 2022-01-31T23:05:57.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:56 smithi171 conmon[46715]: debug 2022-01-31T23:05:56.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:56.977025+0000) 2022-01-31T23:05:57.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:57 smithi167 conmon[54076]: debug 2022-01-31T23:05:57.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.232128+0000) 2022-01-31T23:05:57.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[41853]: debug 2022-01-31T23:05:57.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.430286+0000) 2022-01-31T23:05:57.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[51620]: debug 2022-01-31T23:05:57.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.606431+0000) 2022-01-31T23:05:58.215 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:57 smithi167 conmon[54076]: debug 2022-01-31T23:05:57.943+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.945184+0000) 2022-01-31T23:05:58.215 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:57 smithi167 conmon[49112]: debug 2022-01-31T23:05:57.944+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.945657+0000) 2022-01-31T23:05:58.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:58 smithi167 conmon[49112]: debug 2022-01-31T23:05:58.061+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:58.062357+0000) 2022-01-31T23:05:58.216 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:57 smithi167 conmon[60316]: debug 2022-01-31T23:05:57.943+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.944828+0000) 2022-01-31T23:05:58.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:57 smithi167 conmon[60316]: debug 2022-01-31T23:05:57.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.991475+0000) 2022-01-31T23:05:58.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[41853]: debug 2022-01-31T23:05:57.942+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.943990+0000) 2022-01-31T23:05:58.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[51620]: debug 2022-01-31T23:05:57.942+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.944093+0000) 2022-01-31T23:05:58.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[35325]: debug 2022-01-31T23:05:57.970+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262469 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:05:58.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[46715]: debug 2022-01-31T23:05:57.942+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.944340+0000) 2022-01-31T23:05:58.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:57 smithi171 conmon[46715]: debug 2022-01-31T23:05:57.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:57.977164+0000) 2022-01-31T23:05:58.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:58 smithi167 conmon[54076]: debug 2022-01-31T23:05:58.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:58.232290+0000) 2022-01-31T23:05:58.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:58 smithi171 conmon[41853]: debug 2022-01-31T23:05:58.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:58.430479+0000) 2022-01-31T23:05:58.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:58 smithi171 conmon[51620]: debug 2022-01-31T23:05:58.604+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:58.606632+0000) 2022-01-31T23:05:59.214 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:05:59 smithi167 conmon[49112]: debug 2022-01-31T23:05:59.061+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:59.062522+0000) 2022-01-31T23:05:59.215 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:58 smithi167 conmon[60316]: debug 2022-01-31T23:05:58.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:58.991590+0000) 2022-01-31T23:05:59.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:58 smithi171 conmon[46715]: debug 2022-01-31T23:05:58.975+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:58.977327+0000) 2022-01-31T23:05:59.571 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:05:59 smithi167 conmon[54076]: debug 2022-01-31T23:05:59.231+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:59.232434+0000) 2022-01-31T23:05:59.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:05:59 smithi171 conmon[41853]: debug 2022-01-31T23:05:59.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:59.430674+0000) 2022-01-31T23:05:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:05:59 smithi171 conmon[51620]: debug 2022-01-31T23:05:59.605+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:59.606782+0000) 2022-01-31T23:06:00.215 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:00 smithi167 conmon[49112]: debug 2022-01-31T23:06:00.062+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:00.062733+0000) 2022-01-31T23:06:00.215 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:05:59 smithi167 conmon[60316]: debug 2022-01-31T23:05:59.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:59.991752+0000) 2022-01-31T23:06:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:05:59 smithi171 conmon[46715]: debug 2022-01-31T23:05:59.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:05:59.977513+0000) 2022-01-31T23:06:00.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:00 smithi167 conmon[54076]: debug 2022-01-31T23:06:00.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:00.232695+0000) 2022-01-31T23:06:00.623 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:00 smithi171 conmon[41853]: debug 2022-01-31T23:06:00.429+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:00.430856+0000) 2022-01-31T23:06:00.623 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:00 smithi171 conmon[51620]: debug 2022-01-31T23:06:00.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:00.606961+0000) 2022-01-31T23:06:01.215 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:01 smithi167 conmon[49112]: debug 2022-01-31T23:06:01.062+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:01.062912+0000) 2022-01-31T23:06:01.216 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:00 smithi167 conmon[60316]: debug 2022-01-31T23:06:00.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:00.991910+0000) 2022-01-31T23:06:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:00 smithi171 conmon[46715]: debug 2022-01-31T23:06:00.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:00.977693+0000) 2022-01-31T23:06:01.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:01 smithi167 conmon[54076]: debug 2022-01-31T23:06:01.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:01.232917+0000) 2022-01-31T23:06:01.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:01 smithi171 conmon[41853]: debug 2022-01-31T23:06:01.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:01.431037+0000) 2022-01-31T23:06:01.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:01 smithi171 conmon[51620]: debug 2022-01-31T23:06:01.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:01.607094+0000) 2022-01-31T23:06:02.215 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:02 smithi167 conmon[49112]: debug 2022-01-31T23:06:02.062+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.063066+0000) 2022-01-31T23:06:02.216 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:01 smithi167 conmon[60316]: debug 2022-01-31T23:06:01.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:01.992101+0000) 2022-01-31T23:06:02.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:01 smithi171 conmon[46715]: debug 2022-01-31T23:06:01.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:01.977877+0000) 2022-01-31T23:06:02.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:02 smithi167 conmon[54076]: debug 2022-01-31T23:06:02.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.233108+0000) 2022-01-31T23:06:02.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:02 smithi171 conmon[41853]: debug 2022-01-31T23:06:02.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.431224+0000) 2022-01-31T23:06:02.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:02 smithi171 conmon[51620]: debug 2022-01-31T23:06:02.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.607248+0000) 2022-01-31T23:06:03.215 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:02 smithi167 conmon[49112]: debug 2022-01-31T23:06:02.974+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.975586+0000) 2022-01-31T23:06:03.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:03 smithi167 conmon[49112]: debug 2022-01-31T23:06:03.061+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:03.063267+0000) 2022-01-31T23:06:03.217 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:02 smithi167 conmon[54076]: debug 2022-01-31T23:06:02.973+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.975395+0000) 2022-01-31T23:06:03.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:02 smithi167 conmon[60316]: debug 2022-01-31T23:06:02.973+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.974820+0000) 2022-01-31T23:06:03.218 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:02 smithi167 conmon[60316]: debug 2022-01-31T23:06:02.991+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.992290+0000) 2022-01-31T23:06:03.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:03 smithi171 conmon[35325]: debug 2022-01-31T23:06:03.001+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262575 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:03.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:02 smithi171 conmon[41853]: debug 2022-01-31T23:06:02.972+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.974434+0000) 2022-01-31T23:06:03.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:02 smithi171 conmon[51620]: debug 2022-01-31T23:06:02.972+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.974177+0000) 2022-01-31T23:06:03.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:02 smithi171 conmon[46715]: debug 2022-01-31T23:06:02.972+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.974514+0000) 2022-01-31T23:06:03.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:02 smithi171 conmon[46715]: debug 2022-01-31T23:06:02.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:02.978009+0000) 2022-01-31T23:06:03.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:03 smithi167 conmon[54076]: debug 2022-01-31T23:06:03.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:03.233305+0000) 2022-01-31T23:06:03.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:03 smithi171 conmon[41853]: debug 2022-01-31T23:06:03.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:03.431409+0000) 2022-01-31T23:06:03.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:03 smithi171 conmon[51620]: debug 2022-01-31T23:06:03.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:03.607430+0000) 2022-01-31T23:06:04.215 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:04 smithi167 conmon[49112]: debug 2022-01-31T23:06:04.062+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:04.063443+0000) 2022-01-31T23:06:04.216 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:03 smithi167 conmon[60316]: debug 2022-01-31T23:06:03.990+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:03.992405+0000) 2022-01-31T23:06:04.336 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:03 smithi171 conmon[46715]: debug 2022-01-31T23:06:03.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:03.978132+0000) 2022-01-31T23:06:04.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:04 smithi167 conmon[54076]: debug 2022-01-31T23:06:04.232+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:04.233413+0000) 2022-01-31T23:06:04.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:04 smithi171 conmon[51620]: debug 2022-01-31T23:06:04.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:04.607565+0000) 2022-01-31T23:06:04.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:04 smithi171 conmon[41853]: debug 2022-01-31T23:06:04.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:04.431560+0000) 2022-01-31T23:06:05.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:05 smithi167 conmon[49112]: debug 2022-01-31T23:06:05.063+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:05.063666+0000) 2022-01-31T23:06:05.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:04 smithi167 conmon[60316]: debug 2022-01-31T23:06:04.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:04.992637+0000) 2022-01-31T23:06:05.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:04 smithi171 conmon[46715]: debug 2022-01-31T23:06:04.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:04.978321+0000) 2022-01-31T23:06:05.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:05 smithi167 conmon[54076]: debug 2022-01-31T23:06:05.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:05.233588+0000) 2022-01-31T23:06:05.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:05 smithi171 conmon[51620]: debug 2022-01-31T23:06:05.606+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:05.607766+0000) 2022-01-31T23:06:05.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:05 smithi171 conmon[41853]: debug 2022-01-31T23:06:05.430+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:05.431743+0000) 2022-01-31T23:06:06.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:06 smithi167 conmon[49112]: debug 2022-01-31T23:06:06.063+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:06.063831+0000) 2022-01-31T23:06:06.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:05 smithi167 conmon[60316]: debug 2022-01-31T23:06:05.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:05.992865+0000) 2022-01-31T23:06:06.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:05 smithi171 conmon[46715]: debug 2022-01-31T23:06:05.976+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:05.978501+0000) 2022-01-31T23:06:06.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:06 smithi167 conmon[54076]: debug 2022-01-31T23:06:06.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:06.233777+0000) 2022-01-31T23:06:06.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:06 smithi171 conmon[41853]: debug 2022-01-31T23:06:06.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:06.431978+0000) 2022-01-31T23:06:06.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:06 smithi171 conmon[51620]: debug 2022-01-31T23:06:06.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:06.607950+0000) 2022-01-31T23:06:07.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:07 smithi167 conmon[49112]: debug 2022-01-31T23:06:07.063+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:07.064029+0000) 2022-01-31T23:06:07.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:06 smithi167 conmon[60316]: debug 2022-01-31T23:06:06.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:06.993050+0000) 2022-01-31T23:06:07.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:06 smithi171 conmon[46715]: debug 2022-01-31T23:06:06.977+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:06.978659+0000) 2022-01-31T23:06:07.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:07 smithi167 conmon[54076]: debug 2022-01-31T23:06:07.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:07.233965+0000) 2022-01-31T23:06:07.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:07 smithi171 conmon[41853]: debug 2022-01-31T23:06:07.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:07.432181+0000) 2022-01-31T23:06:07.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:07 smithi171 conmon[51620]: debug 2022-01-31T23:06:07.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:07.608076+0000) 2022-01-31T23:06:08.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:08 smithi167 conmon[49112]: debug 2022-01-31T23:06:08.006+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.006577+0000) 2022-01-31T23:06:08.217 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:08 smithi167 conmon[49112]: debug 2022-01-31T23:06:08.063+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.064186+0000) 2022-01-31T23:06:08.218 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:08 smithi167 conmon[54076]: debug 2022-01-31T23:06:08.005+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.005820+0000) 2022-01-31T23:06:08.218 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:07 smithi167 conmon[60316]: debug 2022-01-31T23:06:07.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:07.993233+0000) 2022-01-31T23:06:08.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:08 smithi167 conmon[60316]: debug 2022-01-31T23:06:08.004+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.005110+0000) 2022-01-31T23:06:08.304 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[35325]: debug 2022-01-31T23:06:08.033+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262685 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:08.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[41853]: debug 2022-01-31T23:06:08.004+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.004742+0000) 2022-01-31T23:06:08.305 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[51620]: debug 2022-01-31T23:06:08.004+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.005303+0000) 2022-01-31T23:06:08.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:07 smithi171 conmon[46715]: debug 2022-01-31T23:06:07.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:07.978828+0000) 2022-01-31T23:06:08.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[46715]: debug 2022-01-31T23:06:08.004+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.005408+0000) 2022-01-31T23:06:08.573 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:08 smithi167 conmon[54076]: debug 2022-01-31T23:06:08.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.234167+0000) 2022-01-31T23:06:08.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[41853]: debug 2022-01-31T23:06:08.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.432384+0000) 2022-01-31T23:06:08.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[51620]: debug 2022-01-31T23:06:08.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.608280+0000) 2022-01-31T23:06:09.216 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:09 smithi167 conmon[49112]: debug 2022-01-31T23:06:09.063+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:09.064323+0000) 2022-01-31T23:06:09.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:08 smithi167 conmon[60316]: debug 2022-01-31T23:06:08.992+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.993363+0000) 2022-01-31T23:06:09.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[46715]: debug 2022-01-31T23:06:08.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:08.978992+0000) 2022-01-31T23:06:09.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:08 smithi171 conmon[35325]: debug 2022-01-31T23:06:08.961+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:06:09.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:09 smithi167 conmon[54076]: debug 2022-01-31T23:06:09.233+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:09.234294+0000) 2022-01-31T23:06:09.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:09 smithi171 conmon[41853]: debug 2022-01-31T23:06:09.431+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:09.432566+0000) 2022-01-31T23:06:09.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:09 smithi171 conmon[51620]: debug 2022-01-31T23:06:09.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:09.608435+0000) 2022-01-31T23:06:10.217 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:10 smithi167 conmon[49112]: debug 2022-01-31T23:06:10.064+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:10.064541+0000) 2022-01-31T23:06:10.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:09 smithi167 conmon[60316]: debug 2022-01-31T23:06:09.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:09.993505+0000) 2022-01-31T23:06:10.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:09 smithi171 conmon[46715]: debug 2022-01-31T23:06:09.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:09.979129+0000) 2022-01-31T23:06:10.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:10 smithi167 conmon[54076]: debug 2022-01-31T23:06:10.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:10.234459+0000) 2022-01-31T23:06:10.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:10 smithi171 conmon[41853]: debug 2022-01-31T23:06:10.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:10.432760+0000) 2022-01-31T23:06:10.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:10 smithi171 conmon[51620]: debug 2022-01-31T23:06:10.607+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:10.608611+0000) 2022-01-31T23:06:11.217 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:11 smithi167 conmon[49112]: debug 2022-01-31T23:06:11.064+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:11.064687+0000) 2022-01-31T23:06:11.217 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:10 smithi167 conmon[60316]: debug 2022-01-31T23:06:10.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:10.993662+0000) 2022-01-31T23:06:11.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:10 smithi171 conmon[46715]: debug 2022-01-31T23:06:10.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:10.979306+0000) 2022-01-31T23:06:11.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:11 smithi167 conmon[54076]: debug 2022-01-31T23:06:11.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:11.234622+0000) 2022-01-31T23:06:11.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:11 smithi171 conmon[41853]: debug 2022-01-31T23:06:11.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:11.432959+0000) 2022-01-31T23:06:11.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:11 smithi171 conmon[51620]: debug 2022-01-31T23:06:11.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:11.608780+0000) 2022-01-31T23:06:12.217 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:12 smithi167 conmon[49112]: debug 2022-01-31T23:06:12.064+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:12.064846+0000) 2022-01-31T23:06:12.218 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:11 smithi167 conmon[60316]: debug 2022-01-31T23:06:11.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:11.993822+0000) 2022-01-31T23:06:12.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:11 smithi171 conmon[46715]: debug 2022-01-31T23:06:11.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:11.979437+0000) 2022-01-31T23:06:12.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:12 smithi167 conmon[54076]: debug 2022-01-31T23:06:12.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:12.234802+0000) 2022-01-31T23:06:12.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:12 smithi171 conmon[41853]: debug 2022-01-31T23:06:12.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:12.433166+0000) 2022-01-31T23:06:12.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:12 smithi171 conmon[51620]: debug 2022-01-31T23:06:12.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:12.608955+0000) 2022-01-31T23:06:13.217 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:13 smithi167 conmon[49112]: debug 2022-01-31T23:06:13.037+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.038034+0000) 2022-01-31T23:06:13.218 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:13 smithi167 conmon[49112]: debug 2022-01-31T23:06:13.064+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.064985+0000) 2022-01-31T23:06:13.219 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:13 smithi167 conmon[54076]: debug 2022-01-31T23:06:13.036+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.037253+0000) 2022-01-31T23:06:13.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:12 smithi167 conmon[60316]: debug 2022-01-31T23:06:12.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:12.993977+0000) 2022-01-31T23:06:13.220 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:13 smithi167 conmon[60316]: debug 2022-01-31T23:06:13.036+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.036733+0000) 2022-01-31T23:06:13.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[35325]: debug 2022-01-31T23:06:13.064+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262798 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:13.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[41853]: debug 2022-01-31T23:06:13.035+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.036478+0000) 2022-01-31T23:06:13.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[51620]: debug 2022-01-31T23:06:13.035+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.036221+0000) 2022-01-31T23:06:13.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:12 smithi171 conmon[46715]: debug 2022-01-31T23:06:12.978+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:12.979581+0000) 2022-01-31T23:06:13.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[46715]: debug 2022-01-31T23:06:13.036+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.037378+0000) 2022-01-31T23:06:13.574 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:13 smithi167 conmon[54076]: debug 2022-01-31T23:06:13.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.234985+0000) 2022-01-31T23:06:13.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[41853]: debug 2022-01-31T23:06:13.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.433291+0000) 2022-01-31T23:06:13.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[51620]: debug 2022-01-31T23:06:13.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.609160+0000) 2022-01-31T23:06:14.217 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:14 smithi167 conmon[49112]: debug 2022-01-31T23:06:14.064+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:14.065128+0000) 2022-01-31T23:06:14.218 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:13 smithi167 conmon[60316]: debug 2022-01-31T23:06:13.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.994115+0000) 2022-01-31T23:06:14.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:13 smithi171 conmon[46715]: debug 2022-01-31T23:06:13.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:13.979740+0000) 2022-01-31T23:06:14.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:14 smithi167 conmon[54076]: debug 2022-01-31T23:06:14.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:14.235155+0000) 2022-01-31T23:06:14.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:14 smithi171 conmon[41853]: debug 2022-01-31T23:06:14.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:14.433425+0000) 2022-01-31T23:06:14.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:14 smithi171 conmon[51620]: debug 2022-01-31T23:06:14.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:14.609286+0000) 2022-01-31T23:06:15.218 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:15 smithi167 conmon[49112]: debug 2022-01-31T23:06:15.064+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:15.065311+0000) 2022-01-31T23:06:15.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:14 smithi167 conmon[60316]: debug 2022-01-31T23:06:14.993+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:14.994305+0000) 2022-01-31T23:06:15.590 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:15 smithi167 conmon[54076]: debug 2022-01-31T23:06:15.234+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:15.235316+0000) 2022-01-31T23:06:15.591 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:14 smithi171 conmon[46715]: debug 2022-01-31T23:06:14.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:14.979898+0000) 2022-01-31T23:06:15.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:15 smithi171 conmon[41853]: debug 2022-01-31T23:06:15.432+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:15.433619+0000) 2022-01-31T23:06:15.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:15 smithi171 conmon[51620]: debug 2022-01-31T23:06:15.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:15.609456+0000) 2022-01-31T23:06:16.218 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:16 smithi167 conmon[49112]: debug 2022-01-31T23:06:16.065+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:16.065542+0000) 2022-01-31T23:06:16.218 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:15 smithi167 conmon[60316]: debug 2022-01-31T23:06:15.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:15.994519+0000) 2022-01-31T23:06:16.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:15 smithi171 conmon[46715]: debug 2022-01-31T23:06:15.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:15.980067+0000) 2022-01-31T23:06:16.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:16 smithi167 conmon[54076]: debug 2022-01-31T23:06:16.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:16.235518+0000) 2022-01-31T23:06:16.752 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:16 smithi171 conmon[41853]: debug 2022-01-31T23:06:16.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:16.433806+0000) 2022-01-31T23:06:16.753 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:16 smithi171 conmon[51620]: debug 2022-01-31T23:06:16.608+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:16.609639+0000) 2022-01-31T23:06:17.218 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:17 smithi167 conmon[49112]: debug 2022-01-31T23:06:17.065+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:17.065738+0000) 2022-01-31T23:06:17.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:16 smithi167 conmon[60316]: debug 2022-01-31T23:06:16.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:16.994655+0000) 2022-01-31T23:06:17.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:16 smithi171 conmon[46715]: debug 2022-01-31T23:06:16.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:16.980224+0000) 2022-01-31T23:06:17.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:17 smithi167 conmon[54076]: debug 2022-01-31T23:06:17.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:17.235706+0000) 2022-01-31T23:06:17.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:17 smithi171 conmon[41853]: debug 2022-01-31T23:06:17.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:17.433986+0000) 2022-01-31T23:06:17.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:17 smithi171 conmon[51620]: debug 2022-01-31T23:06:17.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:17.609847+0000) 2022-01-31T23:06:18.200 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:18 smithi167 conmon[49112]: debug 2022-01-31T23:06:18.068+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.068662+0000) 2022-01-31T23:06:18.201 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:18 smithi167 conmon[54076]: debug 2022-01-31T23:06:18.067+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.067879+0000) 2022-01-31T23:06:18.201 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:17 smithi167 conmon[60316]: debug 2022-01-31T23:06:17.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:17.994781+0000) 2022-01-31T23:06:18.202 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:18 smithi167 conmon[60316]: debug 2022-01-31T23:06:18.067+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.067909+0000) 2022-01-31T23:06:18.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[35325]: debug 2022-01-31T23:06:18.094+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 262908 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:18.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[41853]: debug 2022-01-31T23:06:18.067+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.067793+0000) 2022-01-31T23:06:18.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[51620]: debug 2022-01-31T23:06:18.066+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.067638+0000) 2022-01-31T23:06:18.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:17 smithi171 conmon[46715]: debug 2022-01-31T23:06:17.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:17.980382+0000) 2022-01-31T23:06:18.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[46715]: debug 2022-01-31T23:06:18.067+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.068449+0000) 2022-01-31T23:06:18.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:18 smithi167 conmon[54076]: debug 2022-01-31T23:06:18.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.235871+0000) 2022-01-31T23:06:18.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[41853]: debug 2022-01-31T23:06:18.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.434175+0000) 2022-01-31T23:06:18.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[51620]: debug 2022-01-31T23:06:18.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.610035+0000) 2022-01-31T23:06:19.204 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:18 smithi167 conmon[60316]: debug 2022-01-31T23:06:18.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.994912+0000) 2022-01-31T23:06:19.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:18 smithi171 conmon[46715]: debug 2022-01-31T23:06:18.979+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:18.980534+0000) 2022-01-31T23:06:19.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:19 smithi167 conmon[54076]: debug 2022-01-31T23:06:19.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:19.236015+0000) 2022-01-31T23:06:19.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:19 smithi171 conmon[41853]: debug 2022-01-31T23:06:19.433+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:19.434362+0000) 2022-01-31T23:06:19.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:19 smithi171 conmon[51620]: debug 2022-01-31T23:06:19.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:19.610146+0000) 2022-01-31T23:06:20.208 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:19 smithi167 conmon[60316]: debug 2022-01-31T23:06:19.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:19.995059+0000) 2022-01-31T23:06:20.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:19 smithi171 conmon[46715]: debug 2022-01-31T23:06:19.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:19.980690+0000) 2022-01-31T23:06:20.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:20 smithi167 conmon[54076]: debug 2022-01-31T23:06:20.235+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:20.236303+0000) 2022-01-31T23:06:20.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:20 smithi171 conmon[41853]: debug 2022-01-31T23:06:20.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:20.434546+0000) 2022-01-31T23:06:20.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:20 smithi171 conmon[51620]: debug 2022-01-31T23:06:20.609+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:20.610314+0000) 2022-01-31T23:06:21.212 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:20 smithi167 conmon[60316]: debug 2022-01-31T23:06:20.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:20.995244+0000) 2022-01-31T23:06:21.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:20 smithi171 conmon[46715]: debug 2022-01-31T23:06:20.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:20.980880+0000) 2022-01-31T23:06:21.467 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:21 smithi167 conmon[54076]: debug 2022-01-31T23:06:21.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:21.236546+0000) 2022-01-31T23:06:21.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:21 smithi171 conmon[51620]: debug 2022-01-31T23:06:21.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:21.610496+0000) 2022-01-31T23:06:21.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:21 smithi171 conmon[41853]: debug 2022-01-31T23:06:21.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:21.434728+0000) 2022-01-31T23:06:22.216 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:21 smithi167 conmon[60316]: debug 2022-01-31T23:06:21.994+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:21.995465+0000) 2022-01-31T23:06:22.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:21 smithi171 conmon[46715]: debug 2022-01-31T23:06:21.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:21.981054+0000) 2022-01-31T23:06:22.468 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:22 smithi167 conmon[54076]: debug 2022-01-31T23:06:22.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:22.236650+0000) 2022-01-31T23:06:22.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:22 smithi171 conmon[51620]: debug 2022-01-31T23:06:22.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:22.610675+0000) 2022-01-31T23:06:22.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:22 smithi171 conmon[41853]: debug 2022-01-31T23:06:22.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:22.434938+0000) 2022-01-31T23:06:23.219 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:23 smithi167 conmon[49112]: debug 2022-01-31T23:06:23.098+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.099063+0000) 2022-01-31T23:06:23.220 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:23 smithi167 conmon[54076]: debug 2022-01-31T23:06:23.097+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.098283+0000) 2022-01-31T23:06:23.220 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:22 smithi167 conmon[60316]: debug 2022-01-31T23:06:22.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:22.995683+0000) 2022-01-31T23:06:23.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:23 smithi167 conmon[60316]: debug 2022-01-31T23:06:23.097+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.097582+0000) 2022-01-31T23:06:23.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[35325]: debug 2022-01-31T23:06:23.125+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263021 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:23.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[41853]: debug 2022-01-31T23:06:23.097+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.098094+0000) 2022-01-31T23:06:23.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[51620]: debug 2022-01-31T23:06:23.097+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.097947+0000) 2022-01-31T23:06:23.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:22 smithi171 conmon[46715]: debug 2022-01-31T23:06:22.980+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:22.981249+0000) 2022-01-31T23:06:23.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[46715]: debug 2022-01-31T23:06:23.098+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.098744+0000) 2022-01-31T23:06:23.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:23 smithi167 conmon[54076]: debug 2022-01-31T23:06:23.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.236855+0000) 2022-01-31T23:06:23.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[41853]: debug 2022-01-31T23:06:23.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.435121+0000) 2022-01-31T23:06:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[51620]: debug 2022-01-31T23:06:23.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.610842+0000) 2022-01-31T23:06:24.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:23 smithi167 conmon[60316]: debug 2022-01-31T23:06:23.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.995830+0000) 2022-01-31T23:06:24.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[35325]: debug 2022-01-31T23:06:23.963+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:06:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:23 smithi171 conmon[46715]: debug 2022-01-31T23:06:23.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:23.981457+0000) 2022-01-31T23:06:24.576 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:24 smithi167 conmon[54076]: debug 2022-01-31T23:06:24.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:24.236988+0000) 2022-01-31T23:06:24.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:24 smithi171 conmon[51620]: debug 2022-01-31T23:06:24.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:24.610956+0000) 2022-01-31T23:06:24.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:24 smithi171 conmon[41853]: debug 2022-01-31T23:06:24.434+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:24.435276+0000) 2022-01-31T23:06:24.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:24 smithi167 conmon[49112]: debug 2022-01-31T23:06:24.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:24.816430+0000) 2022-01-31T23:06:25.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:24 smithi167 conmon[60316]: debug 2022-01-31T23:06:24.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:24.996058+0000) 2022-01-31T23:06:25.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:24 smithi171 conmon[46715]: debug 2022-01-31T23:06:24.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:24.981654+0000) 2022-01-31T23:06:25.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:25 smithi167 conmon[54076]: debug 2022-01-31T23:06:25.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:25.237150+0000) 2022-01-31T23:06:25.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:25 smithi171 conmon[41853]: debug 2022-01-31T23:06:25.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:25.435463+0000) 2022-01-31T23:06:25.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:25 smithi171 conmon[51620]: debug 2022-01-31T23:06:25.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:25.611137+0000) 2022-01-31T23:06:25.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:25 smithi167 conmon[49112]: debug 2022-01-31T23:06:25.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:25.816660+0000) 2022-01-31T23:06:26.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:25 smithi167 conmon[60316]: debug 2022-01-31T23:06:25.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:25.996231+0000) 2022-01-31T23:06:26.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:25 smithi171 conmon[46715]: debug 2022-01-31T23:06:25.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:25.981838+0000) 2022-01-31T23:06:26.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:26 smithi167 conmon[54076]: debug 2022-01-31T23:06:26.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:26.237340+0000) 2022-01-31T23:06:26.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:26 smithi171 conmon[51620]: debug 2022-01-31T23:06:26.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:26.611315+0000) 2022-01-31T23:06:26.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:26 smithi171 conmon[41853]: debug 2022-01-31T23:06:26.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:26.435657+0000) 2022-01-31T23:06:26.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:26 smithi167 conmon[49112]: debug 2022-01-31T23:06:26.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:26.816842+0000) 2022-01-31T23:06:27.219 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:26 smithi167 conmon[60316]: debug 2022-01-31T23:06:26.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:26.996415+0000) 2022-01-31T23:06:27.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:26 smithi171 conmon[46715]: debug 2022-01-31T23:06:26.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:26.981998+0000) 2022-01-31T23:06:27.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:27 smithi167 conmon[54076]: debug 2022-01-31T23:06:27.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:27.237539+0000) 2022-01-31T23:06:27.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:27 smithi171 conmon[41853]: debug 2022-01-31T23:06:27.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:27.435816+0000) 2022-01-31T23:06:27.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:27 smithi171 conmon[51620]: debug 2022-01-31T23:06:27.610+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:27.611524+0000) 2022-01-31T23:06:27.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:27 smithi167 conmon[49112]: debug 2022-01-31T23:06:27.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:27.817028+0000) 2022-01-31T23:06:28.220 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:28 smithi167 conmon[49112]: debug 2022-01-31T23:06:28.130+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.130759+0000) 2022-01-31T23:06:28.221 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:28 smithi167 conmon[54076]: debug 2022-01-31T23:06:28.128+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.129071+0000) 2022-01-31T23:06:28.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:27 smithi167 conmon[60316]: debug 2022-01-31T23:06:27.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:27.996598+0000) 2022-01-31T23:06:28.222 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:28 smithi167 conmon[60316]: debug 2022-01-31T23:06:28.128+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.129288+0000) 2022-01-31T23:06:28.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[51620]: debug 2022-01-31T23:06:28.129+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.129833+0000) 2022-01-31T23:06:28.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[35325]: debug 2022-01-31T23:06:28.155+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263131 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:28.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[41853]: debug 2022-01-31T23:06:28.127+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.128232+0000) 2022-01-31T23:06:28.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:27 smithi171 conmon[46715]: debug 2022-01-31T23:06:27.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:27.982130+0000) 2022-01-31T23:06:28.343 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[46715]: debug 2022-01-31T23:06:28.128+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.129222+0000) 2022-01-31T23:06:28.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:28 smithi167 conmon[54076]: debug 2022-01-31T23:06:28.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.237725+0000) 2022-01-31T23:06:28.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[41853]: debug 2022-01-31T23:06:28.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.435994+0000) 2022-01-31T23:06:28.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[51620]: debug 2022-01-31T23:06:28.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.611710+0000) 2022-01-31T23:06:28.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:28 smithi167 conmon[49112]: debug 2022-01-31T23:06:28.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.817203+0000) 2022-01-31T23:06:29.220 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:28 smithi167 conmon[60316]: debug 2022-01-31T23:06:28.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.996780+0000) 2022-01-31T23:06:29.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:28 smithi171 conmon[46715]: debug 2022-01-31T23:06:28.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:28.982297+0000) 2022-01-31T23:06:29.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:29 smithi167 conmon[54076]: debug 2022-01-31T23:06:29.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:29.237882+0000) 2022-01-31T23:06:29.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:29 smithi171 conmon[41853]: debug 2022-01-31T23:06:29.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:29.436173+0000) 2022-01-31T23:06:29.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:29 smithi171 conmon[51620]: debug 2022-01-31T23:06:29.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:29.611777+0000) 2022-01-31T23:06:29.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:29 smithi167 conmon[49112]: debug 2022-01-31T23:06:29.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:29.817356+0000) 2022-01-31T23:06:30.220 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:29 smithi167 conmon[60316]: debug 2022-01-31T23:06:29.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:29.996911+0000) 2022-01-31T23:06:30.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:29 smithi171 conmon[46715]: debug 2022-01-31T23:06:29.981+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:29.982499+0000) 2022-01-31T23:06:30.577 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:30 smithi167 conmon[54076]: debug 2022-01-31T23:06:30.236+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:30.238098+0000) 2022-01-31T23:06:30.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:30 smithi171 conmon[41853]: debug 2022-01-31T23:06:30.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:30.436329+0000) 2022-01-31T23:06:30.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:30 smithi171 conmon[51620]: debug 2022-01-31T23:06:30.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:30.611914+0000) 2022-01-31T23:06:30.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:30 smithi167 conmon[49112]: debug 2022-01-31T23:06:30.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:30.817505+0000) 2022-01-31T23:06:31.220 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:30 smithi167 conmon[60316]: debug 2022-01-31T23:06:30.995+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:30.997069+0000) 2022-01-31T23:06:31.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:30 smithi171 conmon[46715]: debug 2022-01-31T23:06:30.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:30.982710+0000) 2022-01-31T23:06:31.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:31 smithi167 conmon[54076]: debug 2022-01-31T23:06:31.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:31.238310+0000) 2022-01-31T23:06:31.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:31 smithi171 conmon[41853]: debug 2022-01-31T23:06:31.435+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:31.436483+0000) 2022-01-31T23:06:31.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:31 smithi171 conmon[51620]: debug 2022-01-31T23:06:31.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:31.612076+0000) 2022-01-31T23:06:31.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:31 smithi167 conmon[49112]: debug 2022-01-31T23:06:31.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:31.817629+0000) 2022-01-31T23:06:32.220 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:31 smithi167 conmon[60316]: debug 2022-01-31T23:06:31.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:31.997255+0000) 2022-01-31T23:06:32.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:31 smithi171 conmon[46715]: debug 2022-01-31T23:06:31.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:31.982914+0000) 2022-01-31T23:06:32.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:32 smithi167 conmon[54076]: debug 2022-01-31T23:06:32.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:32.238483+0000) 2022-01-31T23:06:32.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:32 smithi171 conmon[41853]: debug 2022-01-31T23:06:32.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:32.436671+0000) 2022-01-31T23:06:32.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:32 smithi171 conmon[51620]: debug 2022-01-31T23:06:32.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:32.612231+0000) 2022-01-31T23:06:32.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:32 smithi167 conmon[49112]: debug 2022-01-31T23:06:32.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:32.817797+0000) 2022-01-31T23:06:33.221 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:33 smithi167 conmon[49112]: debug 2022-01-31T23:06:33.169+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.171245+0000) 2022-01-31T23:06:33.221 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:33 smithi167 conmon[54076]: debug 2022-01-31T23:06:33.167+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.169148+0000) 2022-01-31T23:06:33.222 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:32 smithi167 conmon[60316]: debug 2022-01-31T23:06:32.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:32.997445+0000) 2022-01-31T23:06:33.222 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:33 smithi167 conmon[60316]: debug 2022-01-31T23:06:33.165+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.166984+0000) 2022-01-31T23:06:33.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[35325]: debug 2022-01-31T23:06:33.192+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263245 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:33.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[41853]: debug 2022-01-31T23:06:33.159+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.160161+0000) 2022-01-31T23:06:33.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[51620]: debug 2022-01-31T23:06:33.158+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.159447+0000) 2022-01-31T23:06:33.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:32 smithi171 conmon[46715]: debug 2022-01-31T23:06:32.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:32.983068+0000) 2022-01-31T23:06:33.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[46715]: debug 2022-01-31T23:06:33.158+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.159324+0000) 2022-01-31T23:06:33.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:33 smithi167 conmon[54076]: debug 2022-01-31T23:06:33.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.238682+0000) 2022-01-31T23:06:33.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[41853]: debug 2022-01-31T23:06:33.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.436887+0000) 2022-01-31T23:06:33.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[51620]: debug 2022-01-31T23:06:33.611+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.612430+0000) 2022-01-31T23:06:33.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:33 smithi167 conmon[49112]: debug 2022-01-31T23:06:33.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.817988+0000) 2022-01-31T23:06:34.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:33 smithi167 conmon[60316]: debug 2022-01-31T23:06:33.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.997611+0000) 2022-01-31T23:06:34.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:33 smithi171 conmon[46715]: debug 2022-01-31T23:06:33.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:33.983240+0000) 2022-01-31T23:06:34.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:34 smithi167 conmon[54076]: debug 2022-01-31T23:06:34.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:34.238788+0000) 2022-01-31T23:06:34.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:34 smithi171 conmon[41853]: debug 2022-01-31T23:06:34.436+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:34.437039+0000) 2022-01-31T23:06:34.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:34 smithi171 conmon[51620]: debug 2022-01-31T23:06:34.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:34.612581+0000) 2022-01-31T23:06:34.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:34 smithi167 conmon[49112]: debug 2022-01-31T23:06:34.816+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:34.818167+0000) 2022-01-31T23:06:35.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:34 smithi167 conmon[60316]: debug 2022-01-31T23:06:34.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:34.997798+0000) 2022-01-31T23:06:35.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:34 smithi171 conmon[46715]: debug 2022-01-31T23:06:34.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:34.983461+0000) 2022-01-31T23:06:35.578 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:35 smithi167 conmon[54076]: debug 2022-01-31T23:06:35.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:35.238959+0000) 2022-01-31T23:06:35.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:35 smithi171 conmon[41853]: debug 2022-01-31T23:06:35.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:35.437198+0000) 2022-01-31T23:06:35.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:35 smithi171 conmon[51620]: debug 2022-01-31T23:06:35.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:35.612783+0000) 2022-01-31T23:06:35.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:35 smithi167 conmon[49112]: debug 2022-01-31T23:06:35.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:35.818368+0000) 2022-01-31T23:06:36.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:35 smithi167 conmon[60316]: debug 2022-01-31T23:06:35.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:35.997976+0000) 2022-01-31T23:06:36.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:35 smithi171 conmon[46715]: debug 2022-01-31T23:06:35.982+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:35.983687+0000) 2022-01-31T23:06:36.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:36 smithi167 conmon[54076]: debug 2022-01-31T23:06:36.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:36.239129+0000) 2022-01-31T23:06:36.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:36 smithi171 conmon[41853]: debug 2022-01-31T23:06:36.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:36.437358+0000) 2022-01-31T23:06:36.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:36 smithi171 conmon[51620]: debug 2022-01-31T23:06:36.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:36.612979+0000) 2022-01-31T23:06:36.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:36 smithi167 conmon[49112]: debug 2022-01-31T23:06:36.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:36.818562+0000) 2022-01-31T23:06:37.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:36 smithi167 conmon[60316]: debug 2022-01-31T23:06:36.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:36.998141+0000) 2022-01-31T23:06:37.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:36 smithi171 conmon[46715]: debug 2022-01-31T23:06:36.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:36.983846+0000) 2022-01-31T23:06:37.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:37 smithi167 conmon[54076]: debug 2022-01-31T23:06:37.237+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:37.239309+0000) 2022-01-31T23:06:37.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:37 smithi171 conmon[41853]: debug 2022-01-31T23:06:37.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:37.437545+0000) 2022-01-31T23:06:37.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:37 smithi171 conmon[51620]: debug 2022-01-31T23:06:37.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:37.613155+0000) 2022-01-31T23:06:37.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:37 smithi167 conmon[49112]: debug 2022-01-31T23:06:37.817+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:37.818700+0000) 2022-01-31T23:06:38.223 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:38 smithi167 conmon[49112]: debug 2022-01-31T23:06:38.195+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.196810+0000) 2022-01-31T23:06:38.224 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:38 smithi167 conmon[54076]: debug 2022-01-31T23:06:38.195+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.196481+0000) 2022-01-31T23:06:38.224 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:37 smithi167 conmon[60316]: debug 2022-01-31T23:06:37.996+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:37.998322+0000) 2022-01-31T23:06:38.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:38 smithi167 conmon[60316]: debug 2022-01-31T23:06:38.195+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.197178+0000) 2022-01-31T23:06:38.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[35325]: debug 2022-01-31T23:06:38.223+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263356 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:38.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[41853]: debug 2022-01-31T23:06:38.196+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.197155+0000) 2022-01-31T23:06:38.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:37 smithi171 conmon[46715]: debug 2022-01-31T23:06:37.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:37.984012+0000) 2022-01-31T23:06:38.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[46715]: debug 2022-01-31T23:06:38.194+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.195427+0000) 2022-01-31T23:06:38.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[51620]: debug 2022-01-31T23:06:38.195+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.195836+0000) 2022-01-31T23:06:38.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:38 smithi167 conmon[54076]: debug 2022-01-31T23:06:38.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.239510+0000) 2022-01-31T23:06:38.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[51620]: debug 2022-01-31T23:06:38.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.613302+0000) 2022-01-31T23:06:38.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[41853]: debug 2022-01-31T23:06:38.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.437727+0000) 2022-01-31T23:06:38.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:38 smithi167 conmon[49112]: debug 2022-01-31T23:06:38.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.818891+0000) 2022-01-31T23:06:39.221 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:38 smithi167 conmon[60316]: debug 2022-01-31T23:06:38.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.998511+0000) 2022-01-31T23:06:39.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[35325]: debug 2022-01-31T23:06:38.964+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:06:39.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:38 smithi171 conmon[46715]: debug 2022-01-31T23:06:38.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:38.984180+0000) 2022-01-31T23:06:39.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:39 smithi167 conmon[54076]: debug 2022-01-31T23:06:39.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:39.239670+0000) 2022-01-31T23:06:39.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:39 smithi171 conmon[41853]: debug 2022-01-31T23:06:39.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:39.437938+0000) 2022-01-31T23:06:39.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:39 smithi171 conmon[51620]: debug 2022-01-31T23:06:39.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:39.613431+0000) 2022-01-31T23:06:39.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:39 smithi167 conmon[49112]: debug 2022-01-31T23:06:39.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:39.819063+0000) 2022-01-31T23:06:40.222 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:39 smithi167 conmon[60316]: debug 2022-01-31T23:06:39.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:39.998658+0000) 2022-01-31T23:06:40.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:39 smithi171 conmon[46715]: debug 2022-01-31T23:06:39.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:39.984367+0000) 2022-01-31T23:06:40.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:40 smithi167 conmon[54076]: debug 2022-01-31T23:06:40.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:40.239775+0000) 2022-01-31T23:06:40.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:40 smithi171 conmon[41853]: debug 2022-01-31T23:06:40.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:40.438122+0000) 2022-01-31T23:06:40.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:40 smithi171 conmon[51620]: debug 2022-01-31T23:06:40.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:40.613621+0000) 2022-01-31T23:06:40.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:40 smithi167 conmon[49112]: debug 2022-01-31T23:06:40.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:40.819245+0000) 2022-01-31T23:06:41.222 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:40 smithi167 conmon[60316]: debug 2022-01-31T23:06:40.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:40.998853+0000) 2022-01-31T23:06:41.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:40 smithi171 conmon[46715]: debug 2022-01-31T23:06:40.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:40.984570+0000) 2022-01-31T23:06:41.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:41 smithi167 conmon[54076]: debug 2022-01-31T23:06:41.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:41.239957+0000) 2022-01-31T23:06:41.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:41 smithi171 conmon[51620]: debug 2022-01-31T23:06:41.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:41.613810+0000) 2022-01-31T23:06:41.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:41 smithi171 conmon[41853]: debug 2022-01-31T23:06:41.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:41.438281+0000) 2022-01-31T23:06:41.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:41 smithi167 conmon[49112]: debug 2022-01-31T23:06:41.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:41.819403+0000) 2022-01-31T23:06:42.222 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:41 smithi167 conmon[60316]: debug 2022-01-31T23:06:41.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:41.999037+0000) 2022-01-31T23:06:42.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:41 smithi171 conmon[46715]: debug 2022-01-31T23:06:41.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:41.984763+0000) 2022-01-31T23:06:42.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:42 smithi167 conmon[54076]: debug 2022-01-31T23:06:42.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:42.240131+0000) 2022-01-31T23:06:42.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:42 smithi171 conmon[41853]: debug 2022-01-31T23:06:42.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:42.438471+0000) 2022-01-31T23:06:42.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:42 smithi171 conmon[51620]: debug 2022-01-31T23:06:42.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:42.613995+0000) 2022-01-31T23:06:42.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:42 smithi167 conmon[49112]: debug 2022-01-31T23:06:42.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:42.819585+0000) 2022-01-31T23:06:43.209 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:42 smithi167 conmon[60316]: debug 2022-01-31T23:06:42.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:42.999196+0000) 2022-01-31T23:06:43.236 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[51620]: debug 2022-01-31T23:06:43.224+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.226630+0000) 2022-01-31T23:06:43.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:42 smithi171 conmon[46715]: debug 2022-01-31T23:06:42.983+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:42.984914+0000) 2022-01-31T23:06:43.237 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[46715]: debug 2022-01-31T23:06:43.224+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.226386+0000) 2022-01-31T23:06:43.238 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[41853]: debug 2022-01-31T23:06:43.226+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.227836+0000) 2022-01-31T23:06:43.471 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:43 smithi167 conmon[60316]: debug 2022-01-31T23:06:43.225+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.226858+0000) 2022-01-31T23:06:43.472 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:43 smithi167 conmon[49112]: debug 2022-01-31T23:06:43.226+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.227666+0000) 2022-01-31T23:06:43.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:43 smithi167 conmon[54076]: debug 2022-01-31T23:06:43.225+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.227328+0000) 2022-01-31T23:06:43.473 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:43 smithi167 conmon[54076]: debug 2022-01-31T23:06:43.238+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.240285+0000) 2022-01-31T23:06:43.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[35325]: debug 2022-01-31T23:06:43.252+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263469 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:43.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[41853]: debug 2022-01-31T23:06:43.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.438623+0000) 2022-01-31T23:06:43.967 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[51620]: debug 2022-01-31T23:06:43.612+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.614172+0000) 2022-01-31T23:06:44.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:43 smithi167 conmon[49112]: debug 2022-01-31T23:06:43.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.819797+0000) 2022-01-31T23:06:44.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:43 smithi167 conmon[60316]: debug 2022-01-31T23:06:43.997+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.999378+0000) 2022-01-31T23:06:44.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:43 smithi171 conmon[46715]: debug 2022-01-31T23:06:43.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:43.985091+0000) 2022-01-31T23:06:44.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:44 smithi167 conmon[54076]: debug 2022-01-31T23:06:44.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:44.240458+0000) 2022-01-31T23:06:44.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:44 smithi171 conmon[51620]: debug 2022-01-31T23:06:44.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:44.614305+0000) 2022-01-31T23:06:44.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:44 smithi171 conmon[41853]: debug 2022-01-31T23:06:44.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:44.438787+0000) 2022-01-31T23:06:44.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:44 smithi167 conmon[49112]: debug 2022-01-31T23:06:44.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:44.819950+0000) 2022-01-31T23:06:45.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:44 smithi167 conmon[60316]: debug 2022-01-31T23:06:44.998+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:44.999550+0000) 2022-01-31T23:06:45.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:44 smithi171 conmon[46715]: debug 2022-01-31T23:06:44.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:44.985287+0000) 2022-01-31T23:06:45.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:45 smithi167 conmon[54076]: debug 2022-01-31T23:06:45.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:45.240629+0000) 2022-01-31T23:06:45.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:45 smithi171 conmon[41853]: debug 2022-01-31T23:06:45.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:45.438979+0000) 2022-01-31T23:06:45.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:45 smithi171 conmon[51620]: debug 2022-01-31T23:06:45.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:45.614485+0000) 2022-01-31T23:06:45.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:45 smithi167 conmon[49112]: debug 2022-01-31T23:06:45.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:45.820125+0000) 2022-01-31T23:06:46.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:45 smithi167 conmon[60316]: debug 2022-01-31T23:06:45.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:45.999754+0000) 2022-01-31T23:06:46.352 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:45 smithi171 conmon[46715]: debug 2022-01-31T23:06:45.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:45.985482+0000) 2022-01-31T23:06:46.580 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:46 smithi167 conmon[54076]: debug 2022-01-31T23:06:46.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:46.240781+0000) 2022-01-31T23:06:46.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:46 smithi171 conmon[41853]: debug 2022-01-31T23:06:46.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:46.439131+0000) 2022-01-31T23:06:46.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:46 smithi171 conmon[51620]: debug 2022-01-31T23:06:46.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:46.614669+0000) 2022-01-31T23:06:46.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:46 smithi167 conmon[49112]: debug 2022-01-31T23:06:46.818+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:46.820283+0000) 2022-01-31T23:06:47.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:47 smithi167 conmon[60316]: debug 2022-01-31T23:06:46.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:46.999916+0000) 2022-01-31T23:06:47.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:46 smithi171 conmon[46715]: debug 2022-01-31T23:06:46.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:46.985675+0000) 2022-01-31T23:06:47.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:47 smithi167 conmon[54076]: debug 2022-01-31T23:06:47.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:47.240918+0000) 2022-01-31T23:06:47.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:47 smithi171 conmon[51620]: debug 2022-01-31T23:06:47.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:47.614857+0000) 2022-01-31T23:06:47.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:47 smithi171 conmon[41853]: debug 2022-01-31T23:06:47.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:47.439303+0000) 2022-01-31T23:06:47.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:47 smithi167 conmon[49112]: debug 2022-01-31T23:06:47.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:47.820477+0000) 2022-01-31T23:06:48.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:48 smithi167 conmon[60316]: debug 2022-01-31T23:06:47.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.000065+0000) 2022-01-31T23:06:48.238 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:47 smithi171 conmon[46715]: debug 2022-01-31T23:06:47.984+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:47.985867+0000) 2022-01-31T23:06:48.581 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:48 smithi167 conmon[49112]: debug 2022-01-31T23:06:48.255+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.257382+0000) 2022-01-31T23:06:48.582 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:48 smithi167 conmon[60316]: debug 2022-01-31T23:06:48.255+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.257146+0000) 2022-01-31T23:06:48.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:48 smithi167 conmon[54076]: debug 2022-01-31T23:06:48.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.241119+0000) 2022-01-31T23:06:48.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:48 smithi167 conmon[54076]: debug 2022-01-31T23:06:48.255+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.257256+0000) 2022-01-31T23:06:48.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[35325]: debug 2022-01-31T23:06:48.282+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263580 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:48.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[46715]: debug 2022-01-31T23:06:48.254+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.256285+0000) 2022-01-31T23:06:48.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[51620]: debug 2022-01-31T23:06:48.254+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.256425+0000) 2022-01-31T23:06:48.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[41853]: debug 2022-01-31T23:06:48.256+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.257810+0000) 2022-01-31T23:06:48.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[41853]: debug 2022-01-31T23:06:48.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.439486+0000) 2022-01-31T23:06:48.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:48 smithi167 conmon[49112]: debug 2022-01-31T23:06:48.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.820700+0000) 2022-01-31T23:06:48.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[51620]: debug 2022-01-31T23:06:48.613+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.615008+0000) 2022-01-31T23:06:49.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:49 smithi167 conmon[60316]: debug 2022-01-31T23:06:48.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:49.000260+0000) 2022-01-31T23:06:49.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:48 smithi171 conmon[46715]: debug 2022-01-31T23:06:48.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:48.986079+0000) 2022-01-31T23:06:49.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:49 smithi167 conmon[54076]: debug 2022-01-31T23:06:49.239+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:49.241326+0000) 2022-01-31T23:06:49.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:49 smithi171 conmon[51620]: debug 2022-01-31T23:06:49.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:49.615182+0000) 2022-01-31T23:06:49.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:49 smithi171 conmon[41853]: debug 2022-01-31T23:06:49.437+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:49.439649+0000) 2022-01-31T23:06:49.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:49 smithi167 conmon[49112]: debug 2022-01-31T23:06:49.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:49.820853+0000) 2022-01-31T23:06:50.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:50 smithi167 conmon[60316]: debug 2022-01-31T23:06:49.999+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:50.000411+0000) 2022-01-31T23:06:50.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:49 smithi171 conmon[46715]: debug 2022-01-31T23:06:49.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:49.986238+0000) 2022-01-31T23:06:50.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:50 smithi167 conmon[54076]: debug 2022-01-31T23:06:50.240+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:50.241472+0000) 2022-01-31T23:06:50.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:50 smithi171 conmon[41853]: debug 2022-01-31T23:06:50.438+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:50.439839+0000) 2022-01-31T23:06:50.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:50 smithi171 conmon[51620]: debug 2022-01-31T23:06:50.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:50.615406+0000) 2022-01-31T23:06:50.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:50 smithi167 conmon[49112]: debug 2022-01-31T23:06:50.819+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:50.821042+0000) 2022-01-31T23:06:51.223 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:51 smithi167 conmon[60316]: debug 2022-01-31T23:06:51.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:51.000619+0000) 2022-01-31T23:06:51.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:50 smithi171 conmon[46715]: debug 2022-01-31T23:06:50.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:50.986439+0000) 2022-01-31T23:06:51.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:51 smithi167 conmon[54076]: debug 2022-01-31T23:06:51.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:51.241657+0000) 2022-01-31T23:06:51.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:51 smithi171 conmon[41853]: debug 2022-01-31T23:06:51.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:51.440036+0000) 2022-01-31T23:06:51.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:51 smithi171 conmon[51620]: debug 2022-01-31T23:06:51.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:51.615583+0000) 2022-01-31T23:06:51.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:51 smithi167 conmon[49112]: debug 2022-01-31T23:06:51.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:51.821156+0000) 2022-01-31T23:06:52.224 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:52 smithi167 conmon[60316]: debug 2022-01-31T23:06:52.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:52.000766+0000) 2022-01-31T23:06:52.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:51 smithi171 conmon[46715]: debug 2022-01-31T23:06:51.985+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:51.986630+0000) 2022-01-31T23:06:52.581 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:52 smithi167 conmon[54076]: debug 2022-01-31T23:06:52.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:52.241828+0000) 2022-01-31T23:06:52.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:52 smithi171 conmon[41853]: debug 2022-01-31T23:06:52.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:52.440209+0000) 2022-01-31T23:06:52.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:52 smithi171 conmon[51620]: debug 2022-01-31T23:06:52.614+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:52.615770+0000) 2022-01-31T23:06:52.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:52 smithi167 conmon[49112]: debug 2022-01-31T23:06:52.820+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:52.821371+0000) 2022-01-31T23:06:53.224 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:53 smithi167 conmon[60316]: debug 2022-01-31T23:06:53.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.000918+0000) 2022-01-31T23:06:53.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:52 smithi171 conmon[46715]: debug 2022-01-31T23:06:52.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:52.986829+0000) 2022-01-31T23:06:53.269 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[46715]: debug 2022-01-31T23:06:53.285+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.286145+0000) 2022-01-31T23:06:53.582 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:53 smithi167 conmon[49112]: debug 2022-01-31T23:06:53.285+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.286827+0000) 2022-01-31T23:06:53.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:53 smithi167 conmon[54076]: debug 2022-01-31T23:06:53.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.241985+0000) 2022-01-31T23:06:53.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:53 smithi167 conmon[54076]: debug 2022-01-31T23:06:53.286+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.287919+0000) 2022-01-31T23:06:53.584 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:53 smithi167 conmon[60316]: debug 2022-01-31T23:06:53.286+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.287661+0000) 2022-01-31T23:06:53.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[35325]: debug 2022-01-31T23:06:53.313+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263692 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:53.590 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[51620]: debug 2022-01-31T23:06:53.285+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.286421+0000) 2022-01-31T23:06:53.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[41853]: debug 2022-01-31T23:06:53.287+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.287815+0000) 2022-01-31T23:06:53.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[41853]: debug 2022-01-31T23:06:53.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.440393+0000) 2022-01-31T23:06:53.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:53 smithi167 conmon[49112]: debug 2022-01-31T23:06:53.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.821529+0000) 2022-01-31T23:06:53.947 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[51620]: debug 2022-01-31T23:06:53.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.615902+0000) 2022-01-31T23:06:54.224 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:54 smithi167 conmon[60316]: debug 2022-01-31T23:06:54.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:54.001066+0000) 2022-01-31T23:06:54.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[35325]: debug 2022-01-31T23:06:53.964+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:06:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:53 smithi171 conmon[46715]: debug 2022-01-31T23:06:53.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:53.986968+0000) 2022-01-31T23:06:54.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:54 smithi167 conmon[54076]: debug 2022-01-31T23:06:54.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:54.242190+0000) 2022-01-31T23:06:54.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:54 smithi171 conmon[41853]: debug 2022-01-31T23:06:54.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:54.440485+0000) 2022-01-31T23:06:54.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:54 smithi171 conmon[51620]: debug 2022-01-31T23:06:54.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:54.616062+0000) 2022-01-31T23:06:54.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:54 smithi167 conmon[49112]: debug 2022-01-31T23:06:54.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:54.821614+0000) 2022-01-31T23:06:55.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:55 smithi167 conmon[60316]: debug 2022-01-31T23:06:55.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:55.001237+0000) 2022-01-31T23:06:55.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:54 smithi171 conmon[46715]: debug 2022-01-31T23:06:54.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:54.987171+0000) 2022-01-31T23:06:55.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:55 smithi167 conmon[54076]: debug 2022-01-31T23:06:55.241+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:55.242342+0000) 2022-01-31T23:06:55.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:55 smithi171 conmon[41853]: debug 2022-01-31T23:06:55.439+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:55.440642+0000) 2022-01-31T23:06:55.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:55 smithi171 conmon[51620]: debug 2022-01-31T23:06:55.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:55.616251+0000) 2022-01-31T23:06:55.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:55 smithi167 conmon[49112]: debug 2022-01-31T23:06:55.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:55.821789+0000) 2022-01-31T23:06:56.224 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:56 smithi167 conmon[60316]: debug 2022-01-31T23:06:56.000+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:56.001360+0000) 2022-01-31T23:06:56.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:55 smithi171 conmon[46715]: debug 2022-01-31T23:06:55.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:55.987376+0000) 2022-01-31T23:06:56.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:56 smithi167 conmon[54076]: debug 2022-01-31T23:06:56.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:56.242540+0000) 2022-01-31T23:06:56.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:56 smithi171 conmon[41853]: debug 2022-01-31T23:06:56.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:56.440816+0000) 2022-01-31T23:06:56.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:56 smithi171 conmon[51620]: debug 2022-01-31T23:06:56.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:56.616416+0000) 2022-01-31T23:06:56.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:56 smithi167 conmon[49112]: debug 2022-01-31T23:06:56.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:56.821972+0000) 2022-01-31T23:06:57.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:57 smithi167 conmon[60316]: debug 2022-01-31T23:06:57.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:57.001508+0000) 2022-01-31T23:06:57.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:56 smithi171 conmon[46715]: debug 2022-01-31T23:06:56.986+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:56.987595+0000) 2022-01-31T23:06:57.582 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:57 smithi167 conmon[54076]: debug 2022-01-31T23:06:57.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:57.242721+0000) 2022-01-31T23:06:57.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:57 smithi171 conmon[41853]: debug 2022-01-31T23:06:57.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:57.440992+0000) 2022-01-31T23:06:57.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:57 smithi171 conmon[51620]: debug 2022-01-31T23:06:57.615+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:57.616612+0000) 2022-01-31T23:06:57.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:57 smithi167 conmon[49112]: debug 2022-01-31T23:06:57.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:57.822142+0000) 2022-01-31T23:06:58.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:58 smithi167 conmon[60316]: debug 2022-01-31T23:06:58.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.001652+0000) 2022-01-31T23:06:58.298 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:57 smithi171 conmon[46715]: debug 2022-01-31T23:06:57.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:57.987820+0000) 2022-01-31T23:06:58.583 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:58 smithi167 conmon[49112]: debug 2022-01-31T23:06:58.315+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.316372+0000) 2022-01-31T23:06:58.583 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:58 smithi167 conmon[60316]: debug 2022-01-31T23:06:58.316+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.317062+0000) 2022-01-31T23:06:58.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:58 smithi167 conmon[54076]: debug 2022-01-31T23:06:58.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.242908+0000) 2022-01-31T23:06:58.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:58 smithi167 conmon[54076]: debug 2022-01-31T23:06:58.316+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.316525+0000) 2022-01-31T23:06:58.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[35325]: debug 2022-01-31T23:06:58.343+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263803 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:06:58.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[46715]: debug 2022-01-31T23:06:58.315+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.315866+0000) 2022-01-31T23:06:58.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[41853]: debug 2022-01-31T23:06:58.316+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.317291+0000) 2022-01-31T23:06:58.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[41853]: debug 2022-01-31T23:06:58.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.441184+0000) 2022-01-31T23:06:58.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[51620]: debug 2022-01-31T23:06:58.314+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.315684+0000) 2022-01-31T23:06:58.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:58 smithi167 conmon[49112]: debug 2022-01-31T23:06:58.821+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.822348+0000) 2022-01-31T23:06:58.970 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[51620]: debug 2022-01-31T23:06:58.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.616826+0000) 2022-01-31T23:06:59.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:06:59 smithi167 conmon[60316]: debug 2022-01-31T23:06:59.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:59.001813+0000) 2022-01-31T23:06:59.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:58 smithi171 conmon[46715]: debug 2022-01-31T23:06:58.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:58.988029+0000) 2022-01-31T23:06:59.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:06:59 smithi167 conmon[54076]: debug 2022-01-31T23:06:59.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:59.243102+0000) 2022-01-31T23:06:59.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:06:59 smithi171 conmon[41853]: debug 2022-01-31T23:06:59.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:59.441357+0000) 2022-01-31T23:06:59.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:06:59 smithi171 conmon[51620]: debug 2022-01-31T23:06:59.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:59.616967+0000) 2022-01-31T23:06:59.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:06:59 smithi167 conmon[49112]: debug 2022-01-31T23:06:59.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:59.822488+0000) 2022-01-31T23:07:00.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:00 smithi167 conmon[60316]: debug 2022-01-31T23:07:00.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:00.001962+0000) 2022-01-31T23:07:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:06:59 smithi171 conmon[46715]: debug 2022-01-31T23:06:59.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:06:59.988145+0000) 2022-01-31T23:07:00.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:00 smithi167 conmon[54076]: debug 2022-01-31T23:07:00.242+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:00.243261+0000) 2022-01-31T23:07:00.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:00 smithi171 conmon[41853]: debug 2022-01-31T23:07:00.440+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:00.441536+0000) 2022-01-31T23:07:00.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:00 smithi171 conmon[51620]: debug 2022-01-31T23:07:00.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:00.617161+0000) 2022-01-31T23:07:00.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:00 smithi167 conmon[49112]: debug 2022-01-31T23:07:00.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:00.822651+0000) 2022-01-31T23:07:01.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:01 smithi167 conmon[60316]: debug 2022-01-31T23:07:01.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:01.002144+0000) 2022-01-31T23:07:01.296 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:00 smithi171 conmon[46715]: debug 2022-01-31T23:07:00.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:00.988319+0000) 2022-01-31T23:07:01.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:01 smithi167 conmon[54076]: debug 2022-01-31T23:07:01.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:01.243491+0000) 2022-01-31T23:07:01.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:01 smithi171 conmon[41853]: debug 2022-01-31T23:07:01.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:01.441666+0000) 2022-01-31T23:07:01.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:01 smithi171 conmon[51620]: debug 2022-01-31T23:07:01.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:01.617368+0000) 2022-01-31T23:07:01.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:01 smithi167 conmon[49112]: debug 2022-01-31T23:07:01.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:01.822829+0000) 2022-01-31T23:07:02.225 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:02 smithi167 conmon[60316]: debug 2022-01-31T23:07:02.001+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:02.002304+0000) 2022-01-31T23:07:02.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:01 smithi171 conmon[46715]: debug 2022-01-31T23:07:01.987+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:01.988518+0000) 2022-01-31T23:07:02.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:02 smithi167 conmon[54076]: debug 2022-01-31T23:07:02.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:02.243582+0000) 2022-01-31T23:07:02.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:02 smithi171 conmon[41853]: debug 2022-01-31T23:07:02.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:02.441841+0000) 2022-01-31T23:07:02.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:02 smithi171 conmon[51620]: debug 2022-01-31T23:07:02.616+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:02.617556+0000) 2022-01-31T23:07:02.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:02 smithi167 conmon[49112]: debug 2022-01-31T23:07:02.822+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:02.823150+0000) 2022-01-31T23:07:03.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:03 smithi167 conmon[60316]: debug 2022-01-31T23:07:03.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.002442+0000) 2022-01-31T23:07:03.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:02 smithi171 conmon[46715]: debug 2022-01-31T23:07:02.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:02.988713+0000) 2022-01-31T23:07:03.570 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:03 smithi167 conmon[49112]: debug 2022-01-31T23:07:03.346+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.346871+0000) 2022-01-31T23:07:03.571 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:03 smithi167 conmon[60316]: debug 2022-01-31T23:07:03.347+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.347541+0000) 2022-01-31T23:07:03.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:03 smithi167 conmon[54076]: debug 2022-01-31T23:07:03.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.243754+0000) 2022-01-31T23:07:03.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:03 smithi167 conmon[54076]: debug 2022-01-31T23:07:03.347+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.347640+0000) 2022-01-31T23:07:03.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[35325]: debug 2022-01-31T23:07:03.380+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 263918 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:03.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[46715]: debug 2022-01-31T23:07:03.345+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.346607+0000) 2022-01-31T23:07:03.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[51620]: debug 2022-01-31T23:07:03.346+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.347132+0000) 2022-01-31T23:07:03.591 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[41853]: debug 2022-01-31T23:07:03.347+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.348294+0000) 2022-01-31T23:07:03.592 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[41853]: debug 2022-01-31T23:07:03.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.441999+0000) 2022-01-31T23:07:03.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:03 smithi167 conmon[49112]: debug 2022-01-31T23:07:03.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.823993+0000) 2022-01-31T23:07:03.971 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[51620]: debug 2022-01-31T23:07:03.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.617792+0000) 2022-01-31T23:07:04.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:04 smithi167 conmon[60316]: debug 2022-01-31T23:07:04.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:04.002603+0000) 2022-01-31T23:07:04.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:03 smithi171 conmon[46715]: debug 2022-01-31T23:07:03.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:03.988863+0000) 2022-01-31T23:07:04.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:04 smithi167 conmon[54076]: debug 2022-01-31T23:07:04.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:04.243920+0000) 2022-01-31T23:07:04.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:04 smithi171 conmon[41853]: debug 2022-01-31T23:07:04.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:04.442142+0000) 2022-01-31T23:07:04.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:04 smithi171 conmon[51620]: debug 2022-01-31T23:07:04.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:04.617933+0000) 2022-01-31T23:07:04.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:04 smithi167 conmon[49112]: debug 2022-01-31T23:07:04.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:04.824148+0000) 2022-01-31T23:07:05.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:05 smithi167 conmon[60316]: debug 2022-01-31T23:07:05.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:05.002749+0000) 2022-01-31T23:07:05.328 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:04 smithi171 conmon[46715]: debug 2022-01-31T23:07:04.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:04.989037+0000) 2022-01-31T23:07:05.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:05 smithi167 conmon[54076]: debug 2022-01-31T23:07:05.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:05.244094+0000) 2022-01-31T23:07:05.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:05 smithi171 conmon[41853]: debug 2022-01-31T23:07:05.441+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:05.442330+0000) 2022-01-31T23:07:05.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:05 smithi171 conmon[51620]: debug 2022-01-31T23:07:05.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:05.618117+0000) 2022-01-31T23:07:05.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:05 smithi167 conmon[49112]: debug 2022-01-31T23:07:05.823+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:05.824378+0000) 2022-01-31T23:07:06.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:06 smithi167 conmon[60316]: debug 2022-01-31T23:07:06.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:06.002909+0000) 2022-01-31T23:07:06.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:05 smithi171 conmon[46715]: debug 2022-01-31T23:07:05.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:05.989218+0000) 2022-01-31T23:07:06.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:06 smithi167 conmon[54076]: debug 2022-01-31T23:07:06.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:06.244292+0000) 2022-01-31T23:07:06.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:06 smithi171 conmon[41853]: debug 2022-01-31T23:07:06.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:06.442540+0000) 2022-01-31T23:07:06.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:06 smithi171 conmon[51620]: debug 2022-01-31T23:07:06.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:06.618297+0000) 2022-01-31T23:07:06.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:06 smithi167 conmon[49112]: debug 2022-01-31T23:07:06.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:06.824554+0000) 2022-01-31T23:07:07.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:07 smithi167 conmon[60316]: debug 2022-01-31T23:07:07.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:07.003061+0000) 2022-01-31T23:07:07.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:06 smithi171 conmon[46715]: debug 2022-01-31T23:07:06.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:06.989314+0000) 2022-01-31T23:07:07.584 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:07 smithi167 conmon[54076]: debug 2022-01-31T23:07:07.243+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:07.244464+0000) 2022-01-31T23:07:07.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:07 smithi171 conmon[41853]: debug 2022-01-31T23:07:07.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:07.442728+0000) 2022-01-31T23:07:07.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:07 smithi171 conmon[51620]: debug 2022-01-31T23:07:07.617+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:07.618451+0000) 2022-01-31T23:07:07.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:07 smithi167 conmon[49112]: debug 2022-01-31T23:07:07.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:07.824702+0000) 2022-01-31T23:07:08.226 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:08 smithi167 conmon[60316]: debug 2022-01-31T23:07:08.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.003243+0000) 2022-01-31T23:07:08.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:07 smithi171 conmon[46715]: debug 2022-01-31T23:07:07.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:07.989411+0000) 2022-01-31T23:07:08.584 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:08 smithi167 conmon[49112]: debug 2022-01-31T23:07:08.383+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.383634+0000) 2022-01-31T23:07:08.585 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:08 smithi167 conmon[60316]: debug 2022-01-31T23:07:08.384+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.384515+0000) 2022-01-31T23:07:08.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:08 smithi167 conmon[54076]: debug 2022-01-31T23:07:08.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.244695+0000) 2022-01-31T23:07:08.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:08 smithi167 conmon[54076]: debug 2022-01-31T23:07:08.383+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.384314+0000) 2022-01-31T23:07:08.634 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[46715]: debug 2022-01-31T23:07:08.382+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.383116+0000) 2022-01-31T23:07:08.635 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[35325]: debug 2022-01-31T23:07:08.411+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264030 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:08.635 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[41853]: debug 2022-01-31T23:07:08.384+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.384975+0000) 2022-01-31T23:07:08.636 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[41853]: debug 2022-01-31T23:07:08.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.442865+0000) 2022-01-31T23:07:08.636 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[51620]: debug 2022-01-31T23:07:08.384+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.385317+0000) 2022-01-31T23:07:08.636 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[51620]: debug 2022-01-31T23:07:08.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.618620+0000) 2022-01-31T23:07:08.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:08 smithi167 conmon[49112]: debug 2022-01-31T23:07:08.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.824899+0000) 2022-01-31T23:07:09.227 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:09 smithi167 conmon[60316]: debug 2022-01-31T23:07:09.002+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:09.003368+0000) 2022-01-31T23:07:09.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[35325]: debug 2022-01-31T23:07:08.965+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:07:09.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:08 smithi171 conmon[46715]: debug 2022-01-31T23:07:08.988+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:08.989595+0000) 2022-01-31T23:07:09.585 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:09 smithi167 conmon[54076]: debug 2022-01-31T23:07:09.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:09.244886+0000) 2022-01-31T23:07:09.635 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:09 smithi171 conmon[41853]: debug 2022-01-31T23:07:09.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:09.443033+0000) 2022-01-31T23:07:09.635 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:09 smithi171 conmon[51620]: debug 2022-01-31T23:07:09.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:09.618788+0000) 2022-01-31T23:07:09.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:09 smithi167 conmon[49112]: debug 2022-01-31T23:07:09.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:09.825051+0000) 2022-01-31T23:07:10.227 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:10 smithi167 conmon[60316]: debug 2022-01-31T23:07:10.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:10.003533+0000) 2022-01-31T23:07:10.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:09 smithi171 conmon[46715]: debug 2022-01-31T23:07:09.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:09.989723+0000) 2022-01-31T23:07:10.585 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:10 smithi167 conmon[54076]: debug 2022-01-31T23:07:10.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:10.245070+0000) 2022-01-31T23:07:10.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:10 smithi171 conmon[41853]: debug 2022-01-31T23:07:10.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:10.443192+0000) 2022-01-31T23:07:10.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:10 smithi171 conmon[51620]: debug 2022-01-31T23:07:10.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:10.618986+0000) 2022-01-31T23:07:10.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:10 smithi167 conmon[49112]: debug 2022-01-31T23:07:10.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:10.825235+0000) 2022-01-31T23:07:11.227 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:11 smithi167 conmon[60316]: debug 2022-01-31T23:07:11.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:11.003710+0000) 2022-01-31T23:07:11.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:10 smithi171 conmon[46715]: debug 2022-01-31T23:07:10.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:10.989912+0000) 2022-01-31T23:07:11.585 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:11 smithi167 conmon[54076]: debug 2022-01-31T23:07:11.244+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:11.245261+0000) 2022-01-31T23:07:11.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:11 smithi171 conmon[41853]: debug 2022-01-31T23:07:11.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:11.443374+0000) 2022-01-31T23:07:11.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:11 smithi171 conmon[51620]: debug 2022-01-31T23:07:11.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:11.619182+0000) 2022-01-31T23:07:11.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:11 smithi167 conmon[49112]: debug 2022-01-31T23:07:11.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:11.825401+0000) 2022-01-31T23:07:12.227 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:12 smithi167 conmon[60316]: debug 2022-01-31T23:07:12.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:12.003854+0000) 2022-01-31T23:07:12.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:11 smithi171 conmon[46715]: debug 2022-01-31T23:07:11.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:11.990061+0000) 2022-01-31T23:07:12.585 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:12 smithi167 conmon[54076]: debug 2022-01-31T23:07:12.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:12.245455+0000) 2022-01-31T23:07:12.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:12 smithi171 conmon[41853]: debug 2022-01-31T23:07:12.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:12.443557+0000) 2022-01-31T23:07:12.840 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:12 smithi171 conmon[51620]: debug 2022-01-31T23:07:12.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:12.619345+0000) 2022-01-31T23:07:12.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:12 smithi167 conmon[49112]: debug 2022-01-31T23:07:12.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:12.825562+0000) 2022-01-31T23:07:13.227 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:13 smithi167 conmon[60316]: debug 2022-01-31T23:07:13.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.004050+0000) 2022-01-31T23:07:13.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:12 smithi171 conmon[46715]: debug 2022-01-31T23:07:12.989+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:12.990250+0000) 2022-01-31T23:07:13.586 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:13 smithi167 conmon[49112]: debug 2022-01-31T23:07:13.414+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.414885+0000) 2022-01-31T23:07:13.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:13 smithi167 conmon[60316]: debug 2022-01-31T23:07:13.414+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.415121+0000) 2022-01-31T23:07:13.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:13 smithi167 conmon[54076]: debug 2022-01-31T23:07:13.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.245611+0000) 2022-01-31T23:07:13.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:13 smithi167 conmon[54076]: debug 2022-01-31T23:07:13.413+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.414429+0000) 2022-01-31T23:07:13.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[35325]: debug 2022-01-31T23:07:13.441+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264142 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:13.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[46715]: debug 2022-01-31T23:07:13.413+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.414462+0000) 2022-01-31T23:07:13.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[41853]: debug 2022-01-31T23:07:13.415+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.416159+0000) 2022-01-31T23:07:13.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[41853]: debug 2022-01-31T23:07:13.442+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.443692+0000) 2022-01-31T23:07:13.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[51620]: debug 2022-01-31T23:07:13.414+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.415611+0000) 2022-01-31T23:07:13.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[51620]: debug 2022-01-31T23:07:13.618+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.619538+0000) 2022-01-31T23:07:13.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:13 smithi167 conmon[49112]: debug 2022-01-31T23:07:13.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.825748+0000) 2022-01-31T23:07:14.228 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:14 smithi167 conmon[60316]: debug 2022-01-31T23:07:14.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:14.004201+0000) 2022-01-31T23:07:14.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:13 smithi171 conmon[46715]: debug 2022-01-31T23:07:13.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:13.990436+0000) 2022-01-31T23:07:14.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:14 smithi167 conmon[54076]: debug 2022-01-31T23:07:14.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:14.245798+0000) 2022-01-31T23:07:14.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:14 smithi171 conmon[51620]: debug 2022-01-31T23:07:14.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:14.619690+0000) 2022-01-31T23:07:14.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:14 smithi171 conmon[41853]: debug 2022-01-31T23:07:14.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:14.443826+0000) 2022-01-31T23:07:14.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:14 smithi167 conmon[49112]: debug 2022-01-31T23:07:14.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:14.825921+0000) 2022-01-31T23:07:15.228 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:15 smithi167 conmon[60316]: debug 2022-01-31T23:07:15.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:15.004365+0000) 2022-01-31T23:07:15.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:14 smithi171 conmon[46715]: debug 2022-01-31T23:07:14.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:14.990641+0000) 2022-01-31T23:07:15.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:15 smithi167 conmon[54076]: debug 2022-01-31T23:07:15.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:15.245955+0000) 2022-01-31T23:07:15.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:15 smithi171 conmon[51620]: debug 2022-01-31T23:07:15.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:15.619877+0000) 2022-01-31T23:07:15.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:15 smithi171 conmon[41853]: debug 2022-01-31T23:07:15.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:15.444013+0000) 2022-01-31T23:07:15.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:15 smithi167 conmon[49112]: debug 2022-01-31T23:07:15.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:15.826112+0000) 2022-01-31T23:07:16.228 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:16 smithi167 conmon[60316]: debug 2022-01-31T23:07:16.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:16.004584+0000) 2022-01-31T23:07:16.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:15 smithi171 conmon[46715]: debug 2022-01-31T23:07:15.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:15.990813+0000) 2022-01-31T23:07:16.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:16 smithi167 conmon[54076]: debug 2022-01-31T23:07:16.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:16.246114+0000) 2022-01-31T23:07:16.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:16 smithi171 conmon[51620]: debug 2022-01-31T23:07:16.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:16.620063+0000) 2022-01-31T23:07:16.977 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:16 smithi171 conmon[41853]: debug 2022-01-31T23:07:16.443+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:16.444196+0000) 2022-01-31T23:07:16.977 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:16 smithi167 conmon[49112]: debug 2022-01-31T23:07:16.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:16.826275+0000) 2022-01-31T23:07:17.228 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:17 smithi167 conmon[60316]: debug 2022-01-31T23:07:17.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:17.004757+0000) 2022-01-31T23:07:17.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:16 smithi171 conmon[46715]: debug 2022-01-31T23:07:16.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:16.990986+0000) 2022-01-31T23:07:17.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:17 smithi167 conmon[54076]: debug 2022-01-31T23:07:17.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:17.246357+0000) 2022-01-31T23:07:17.679 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:17 smithi171 conmon[51620]: debug 2022-01-31T23:07:17.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:17.620247+0000) 2022-01-31T23:07:17.680 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:17 smithi171 conmon[41853]: debug 2022-01-31T23:07:17.444+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:17.444449+0000) 2022-01-31T23:07:17.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:17 smithi167 conmon[49112]: debug 2022-01-31T23:07:17.824+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:17.826471+0000) 2022-01-31T23:07:18.228 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:18 smithi167 conmon[60316]: debug 2022-01-31T23:07:18.003+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.004959+0000) 2022-01-31T23:07:18.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:17 smithi171 conmon[46715]: debug 2022-01-31T23:07:17.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:17.991180+0000) 2022-01-31T23:07:18.587 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:18 smithi167 conmon[60316]: debug 2022-01-31T23:07:18.443+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.444605+0000) 2022-01-31T23:07:18.588 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:18 smithi167 conmon[49112]: debug 2022-01-31T23:07:18.444+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.445856+0000) 2022-01-31T23:07:18.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:18 smithi167 conmon[54076]: debug 2022-01-31T23:07:18.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.246538+0000) 2022-01-31T23:07:18.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:18 smithi167 conmon[54076]: debug 2022-01-31T23:07:18.444+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.445646+0000) 2022-01-31T23:07:18.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:18 smithi171 conmon[41853]: debug 2022-01-31T23:07:18.445+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.446389+0000) 2022-01-31T23:07:18.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:18 smithi171 conmon[46715]: debug 2022-01-31T23:07:18.444+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.445265+0000) 2022-01-31T23:07:18.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:18 smithi171 conmon[35325]: debug 2022-01-31T23:07:18.473+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264253 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:18.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:18 smithi171 conmon[51620]: debug 2022-01-31T23:07:18.445+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.446058+0000) 2022-01-31T23:07:18.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:18 smithi171 conmon[51620]: debug 2022-01-31T23:07:18.619+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.620427+0000) 2022-01-31T23:07:18.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:18 smithi167 conmon[49112]: debug 2022-01-31T23:07:18.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.826657+0000) 2022-01-31T23:07:19.229 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:19 smithi167 conmon[60316]: debug 2022-01-31T23:07:19.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:19.005137+0000) 2022-01-31T23:07:19.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:18 smithi171 conmon[46715]: debug 2022-01-31T23:07:18.990+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:18.991360+0000) 2022-01-31T23:07:19.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:19 smithi167 conmon[54076]: debug 2022-01-31T23:07:19.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:19.246690+0000) 2022-01-31T23:07:19.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:19 smithi167 conmon[49112]: debug 2022-01-31T23:07:19.825+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:19.826805+0000) 2022-01-31T23:07:19.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:19 smithi171 conmon[51620]: debug 2022-01-31T23:07:19.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:19.620580+0000) 2022-01-31T23:07:20.229 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:20 smithi167 conmon[60316]: debug 2022-01-31T23:07:20.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:20.005289+0000) 2022-01-31T23:07:20.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:19 smithi171 conmon[46715]: debug 2022-01-31T23:07:19.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:19.991496+0000) 2022-01-31T23:07:20.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:20 smithi167 conmon[54076]: debug 2022-01-31T23:07:20.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:20.246865+0000) 2022-01-31T23:07:20.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:20 smithi167 conmon[49112]: debug 2022-01-31T23:07:20.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:20.826972+0000) 2022-01-31T23:07:20.973 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:20 smithi171 conmon[51620]: debug 2022-01-31T23:07:20.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:20.620798+0000) 2022-01-31T23:07:21.229 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:21 smithi167 conmon[60316]: debug 2022-01-31T23:07:21.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:21.005517+0000) 2022-01-31T23:07:21.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:20 smithi171 conmon[46715]: debug 2022-01-31T23:07:20.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:20.991654+0000) 2022-01-31T23:07:21.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:21 smithi167 conmon[54076]: debug 2022-01-31T23:07:21.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:21.247082+0000) 2022-01-31T23:07:21.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:21 smithi167 conmon[49112]: debug 2022-01-31T23:07:21.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:21.827128+0000) 2022-01-31T23:07:21.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:21 smithi171 conmon[51620]: debug 2022-01-31T23:07:21.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:21.620974+0000) 2022-01-31T23:07:22.229 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:22 smithi167 conmon[60316]: debug 2022-01-31T23:07:22.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:22.005727+0000) 2022-01-31T23:07:22.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:21 smithi171 conmon[46715]: debug 2022-01-31T23:07:21.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:21.991835+0000) 2022-01-31T23:07:22.587 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:22 smithi167 conmon[54076]: debug 2022-01-31T23:07:22.245+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:22.247238+0000) 2022-01-31T23:07:22.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:22 smithi167 conmon[49112]: debug 2022-01-31T23:07:22.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:22.827301+0000) 2022-01-31T23:07:22.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:22 smithi171 conmon[51620]: debug 2022-01-31T23:07:22.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:22.621095+0000) 2022-01-31T23:07:23.229 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:23 smithi167 conmon[60316]: debug 2022-01-31T23:07:23.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.005896+0000) 2022-01-31T23:07:23.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:22 smithi171 conmon[46715]: debug 2022-01-31T23:07:22.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:22.992032+0000) 2022-01-31T23:07:23.587 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:23 smithi167 conmon[49112]: debug 2022-01-31T23:07:23.475+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.476545+0000) 2022-01-31T23:07:23.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:23 smithi167 conmon[54076]: debug 2022-01-31T23:07:23.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.247402+0000) 2022-01-31T23:07:23.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:23 smithi167 conmon[54076]: debug 2022-01-31T23:07:23.474+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.476462+0000) 2022-01-31T23:07:23.589 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:23 smithi167 conmon[60316]: debug 2022-01-31T23:07:23.474+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.476078+0000) 2022-01-31T23:07:23.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[35325]: debug 2022-01-31T23:07:23.503+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264365 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:23.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[41853]: debug 2022-01-31T23:07:23.475+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.476662+0000) 2022-01-31T23:07:23.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[46715]: debug 2022-01-31T23:07:23.476+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.477005+0000) 2022-01-31T23:07:23.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[51620]: debug 2022-01-31T23:07:23.475+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.476109+0000) 2022-01-31T23:07:23.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[51620]: debug 2022-01-31T23:07:23.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.621293+0000) 2022-01-31T23:07:23.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:23 smithi167 conmon[49112]: debug 2022-01-31T23:07:23.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.827465+0000) 2022-01-31T23:07:24.229 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:24 smithi167 conmon[60316]: debug 2022-01-31T23:07:24.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:24.006073+0000) 2022-01-31T23:07:24.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[35325]: debug 2022-01-31T23:07:23.966+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:07:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:23 smithi171 conmon[46715]: debug 2022-01-31T23:07:23.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:23.992210+0000) 2022-01-31T23:07:24.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:24 smithi167 conmon[54076]: debug 2022-01-31T23:07:24.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:24.247580+0000) 2022-01-31T23:07:24.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:24 smithi167 conmon[49112]: debug 2022-01-31T23:07:24.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:24.827578+0000) 2022-01-31T23:07:24.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:24 smithi171 conmon[51620]: debug 2022-01-31T23:07:24.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:24.621467+0000) 2022-01-31T23:07:25.230 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:25 smithi167 conmon[60316]: debug 2022-01-31T23:07:25.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:25.006267+0000) 2022-01-31T23:07:25.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:25 smithi171 conmon[41853]: debug 2022-01-31T23:07:25.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:25.194949+0000) 2022-01-31T23:07:25.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:24 smithi171 conmon[46715]: debug 2022-01-31T23:07:24.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:24.992515+0000) 2022-01-31T23:07:25.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:25 smithi167 conmon[54076]: debug 2022-01-31T23:07:25.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:25.247718+0000) 2022-01-31T23:07:25.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:25 smithi167 conmon[49112]: debug 2022-01-31T23:07:25.826+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:25.827788+0000) 2022-01-31T23:07:25.974 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:25 smithi171 conmon[51620]: debug 2022-01-31T23:07:25.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:25.621686+0000) 2022-01-31T23:07:26.230 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:26 smithi167 conmon[60316]: debug 2022-01-31T23:07:26.004+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:26.006445+0000) 2022-01-31T23:07:26.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:26 smithi171 conmon[41853]: debug 2022-01-31T23:07:26.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:26.195186+0000) 2022-01-31T23:07:26.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:25 smithi171 conmon[46715]: debug 2022-01-31T23:07:25.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:25.992662+0000) 2022-01-31T23:07:26.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:26 smithi167 conmon[54076]: debug 2022-01-31T23:07:26.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:26.247878+0000) 2022-01-31T23:07:26.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:26 smithi167 conmon[49112]: debug 2022-01-31T23:07:26.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:26.827969+0000) 2022-01-31T23:07:26.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:26 smithi171 conmon[51620]: debug 2022-01-31T23:07:26.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:26.621830+0000) 2022-01-31T23:07:27.230 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:27 smithi167 conmon[60316]: debug 2022-01-31T23:07:27.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:27.006594+0000) 2022-01-31T23:07:27.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:27 smithi171 conmon[41853]: debug 2022-01-31T23:07:27.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:27.195388+0000) 2022-01-31T23:07:27.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:26 smithi171 conmon[46715]: debug 2022-01-31T23:07:26.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:26.992827+0000) 2022-01-31T23:07:27.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:27 smithi167 conmon[54076]: debug 2022-01-31T23:07:27.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:27.248068+0000) 2022-01-31T23:07:27.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:27 smithi167 conmon[49112]: debug 2022-01-31T23:07:27.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:27.828131+0000) 2022-01-31T23:07:27.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:27 smithi171 conmon[51620]: debug 2022-01-31T23:07:27.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:27.621970+0000) 2022-01-31T23:07:28.230 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:28 smithi167 conmon[60316]: debug 2022-01-31T23:07:28.005+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.006718+0000) 2022-01-31T23:07:28.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:27 smithi171 conmon[46715]: debug 2022-01-31T23:07:27.991+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:27.992988+0000) 2022-01-31T23:07:28.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[41853]: debug 2022-01-31T23:07:28.193+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.195583+0000) 2022-01-31T23:07:28.489 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:28 smithi167 conmon[54076]: debug 2022-01-31T23:07:28.246+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.248196+0000) 2022-01-31T23:07:28.810 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:28 smithi167 conmon[49112]: debug 2022-01-31T23:07:28.505+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.506916+0000) 2022-01-31T23:07:28.811 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:28 smithi167 conmon[54076]: debug 2022-01-31T23:07:28.506+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.507803+0000) 2022-01-31T23:07:28.811 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:28 smithi167 conmon[60316]: debug 2022-01-31T23:07:28.505+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.506856+0000) 2022-01-31T23:07:28.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[41853]: debug 2022-01-31T23:07:28.506+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.507802+0000) 2022-01-31T23:07:28.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[46715]: debug 2022-01-31T23:07:28.505+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.507190+0000) 2022-01-31T23:07:28.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[35325]: debug 2022-01-31T23:07:28.532+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264477 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:28.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[51620]: debug 2022-01-31T23:07:28.504+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.506342+0000) 2022-01-31T23:07:28.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[51620]: debug 2022-01-31T23:07:28.620+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.622141+0000) 2022-01-31T23:07:29.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:28 smithi167 conmon[49112]: debug 2022-01-31T23:07:28.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.828314+0000) 2022-01-31T23:07:29.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:29 smithi167 conmon[60316]: debug 2022-01-31T23:07:29.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:29.006896+0000) 2022-01-31T23:07:29.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:28 smithi171 conmon[46715]: debug 2022-01-31T23:07:28.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:28.993178+0000) 2022-01-31T23:07:29.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:29 smithi171 conmon[41853]: debug 2022-01-31T23:07:29.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:29.195778+0000) 2022-01-31T23:07:29.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:29 smithi167 conmon[54076]: debug 2022-01-31T23:07:29.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:29.248360+0000) 2022-01-31T23:07:29.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:29 smithi167 conmon[49112]: debug 2022-01-31T23:07:29.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:29.828494+0000) 2022-01-31T23:07:29.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:29 smithi171 conmon[51620]: debug 2022-01-31T23:07:29.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:29.622299+0000) 2022-01-31T23:07:30.230 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:30 smithi167 conmon[60316]: debug 2022-01-31T23:07:30.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:30.007057+0000) 2022-01-31T23:07:30.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:30 smithi171 conmon[41853]: debug 2022-01-31T23:07:30.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:30.195925+0000) 2022-01-31T23:07:30.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:29 smithi171 conmon[46715]: debug 2022-01-31T23:07:29.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:29.993332+0000) 2022-01-31T23:07:30.588 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:30 smithi167 conmon[54076]: debug 2022-01-31T23:07:30.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:30.248531+0000) 2022-01-31T23:07:30.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:30 smithi167 conmon[49112]: debug 2022-01-31T23:07:30.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:30.828696+0000) 2022-01-31T23:07:30.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:30 smithi171 conmon[51620]: debug 2022-01-31T23:07:30.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:30.622517+0000) 2022-01-31T23:07:31.231 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:31 smithi167 conmon[60316]: debug 2022-01-31T23:07:31.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:31.007219+0000) 2022-01-31T23:07:31.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:31 smithi171 conmon[41853]: debug 2022-01-31T23:07:31.194+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:31.196065+0000) 2022-01-31T23:07:31.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:30 smithi171 conmon[46715]: debug 2022-01-31T23:07:30.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:30.993520+0000) 2022-01-31T23:07:31.589 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:31 smithi167 conmon[54076]: debug 2022-01-31T23:07:31.247+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:31.248709+0000) 2022-01-31T23:07:31.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:31 smithi167 conmon[49112]: debug 2022-01-31T23:07:31.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:31.828858+0000) 2022-01-31T23:07:31.975 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:31 smithi171 conmon[51620]: debug 2022-01-31T23:07:31.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:31.622711+0000) 2022-01-31T23:07:32.231 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:32 smithi167 conmon[60316]: debug 2022-01-31T23:07:32.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:32.007384+0000) 2022-01-31T23:07:32.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:32 smithi171 conmon[41853]: debug 2022-01-31T23:07:32.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:32.196295+0000) 2022-01-31T23:07:32.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:31 smithi171 conmon[46715]: debug 2022-01-31T23:07:31.992+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:31.993781+0000) 2022-01-31T23:07:32.484 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:32 smithi167 conmon[54076]: debug 2022-01-31T23:07:32.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:32.248906+0000) 2022-01-31T23:07:32.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:32 smithi171 conmon[51620]: debug 2022-01-31T23:07:32.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:32.622954+0000) 2022-01-31T23:07:33.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:32 smithi167 conmon[49112]: debug 2022-01-31T23:07:32.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:32.829019+0000) 2022-01-31T23:07:33.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:33 smithi167 conmon[60316]: debug 2022-01-31T23:07:33.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.007553+0000) 2022-01-31T23:07:33.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[41853]: debug 2022-01-31T23:07:33.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.196496+0000) 2022-01-31T23:07:33.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:32 smithi171 conmon[46715]: debug 2022-01-31T23:07:32.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:32.993987+0000) 2022-01-31T23:07:33.487 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:33 smithi167 conmon[54076]: debug 2022-01-31T23:07:33.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.249098+0000) 2022-01-31T23:07:33.811 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:33 smithi167 conmon[49112]: debug 2022-01-31T23:07:33.535+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.536847+0000) 2022-01-31T23:07:33.812 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:33 smithi167 conmon[54076]: debug 2022-01-31T23:07:33.541+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.543143+0000) 2022-01-31T23:07:33.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:33 smithi167 conmon[60316]: debug 2022-01-31T23:07:33.535+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.537222+0000) 2022-01-31T23:07:33.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[41853]: debug 2022-01-31T23:07:33.535+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.537048+0000) 2022-01-31T23:07:33.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[46715]: debug 2022-01-31T23:07:33.534+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.536095+0000) 2022-01-31T23:07:33.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[35325]: debug 2022-01-31T23:07:33.568+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264588 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[51620]: debug 2022-01-31T23:07:33.534+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.536312+0000) 2022-01-31T23:07:33.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[51620]: debug 2022-01-31T23:07:33.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.623104+0000) 2022-01-31T23:07:34.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:33 smithi167 conmon[49112]: debug 2022-01-31T23:07:33.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.829215+0000) 2022-01-31T23:07:34.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:34 smithi167 conmon[60316]: debug 2022-01-31T23:07:34.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:34.007684+0000) 2022-01-31T23:07:34.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:33 smithi171 conmon[46715]: debug 2022-01-31T23:07:33.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:33.994188+0000) 2022-01-31T23:07:34.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:34 smithi171 conmon[41853]: debug 2022-01-31T23:07:34.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:34.196661+0000) 2022-01-31T23:07:34.492 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:34 smithi167 conmon[54076]: debug 2022-01-31T23:07:34.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:34.249285+0000) 2022-01-31T23:07:34.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:34 smithi171 conmon[51620]: debug 2022-01-31T23:07:34.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:34.623259+0000) 2022-01-31T23:07:35.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:35 smithi167 conmon[60316]: debug 2022-01-31T23:07:35.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:35.007901+0000) 2022-01-31T23:07:35.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:34 smithi167 conmon[49112]: debug 2022-01-31T23:07:34.827+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:34.829386+0000) 2022-01-31T23:07:35.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:35 smithi171 conmon[41853]: debug 2022-01-31T23:07:35.195+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:35.196771+0000) 2022-01-31T23:07:35.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:34 smithi171 conmon[46715]: debug 2022-01-31T23:07:34.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:34.994377+0000) 2022-01-31T23:07:35.495 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:35 smithi167 conmon[54076]: debug 2022-01-31T23:07:35.248+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:35.249442+0000) 2022-01-31T23:07:35.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:35 smithi171 conmon[51620]: debug 2022-01-31T23:07:35.621+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:35.623464+0000) 2022-01-31T23:07:36.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:36 smithi167 conmon[60316]: debug 2022-01-31T23:07:36.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:36.008104+0000) 2022-01-31T23:07:36.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:35 smithi167 conmon[49112]: debug 2022-01-31T23:07:35.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:35.829608+0000) 2022-01-31T23:07:36.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:35 smithi171 conmon[46715]: debug 2022-01-31T23:07:35.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:35.994551+0000) 2022-01-31T23:07:36.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:36 smithi171 conmon[41853]: debug 2022-01-31T23:07:36.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:36.196956+0000) 2022-01-31T23:07:36.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:36 smithi167 conmon[54076]: debug 2022-01-31T23:07:36.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:36.249673+0000) 2022-01-31T23:07:36.976 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:36 smithi171 conmon[51620]: debug 2022-01-31T23:07:36.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:36.623648+0000) 2022-01-31T23:07:37.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:37 smithi167 conmon[60316]: debug 2022-01-31T23:07:37.006+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:37.008277+0000) 2022-01-31T23:07:37.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:36 smithi167 conmon[49112]: debug 2022-01-31T23:07:36.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:36.829769+0000) 2022-01-31T23:07:37.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:37 smithi171 conmon[41853]: debug 2022-01-31T23:07:37.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:37.197137+0000) 2022-01-31T23:07:37.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:36 smithi171 conmon[46715]: debug 2022-01-31T23:07:36.993+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:36.994740+0000) 2022-01-31T23:07:37.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:37 smithi167 conmon[54076]: debug 2022-01-31T23:07:37.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:37.249837+0000) 2022-01-31T23:07:37.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:37 smithi171 conmon[51620]: debug 2022-01-31T23:07:37.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:37.623822+0000) 2022-01-31T23:07:38.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:37 smithi167 conmon[49112]: debug 2022-01-31T23:07:37.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:37.829924+0000) 2022-01-31T23:07:38.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:38 smithi167 conmon[60316]: debug 2022-01-31T23:07:38.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.008440+0000) 2022-01-31T23:07:38.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[41853]: debug 2022-01-31T23:07:38.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.197348+0000) 2022-01-31T23:07:38.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:37 smithi171 conmon[46715]: debug 2022-01-31T23:07:37.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:37.994921+0000) 2022-01-31T23:07:38.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:38 smithi167 conmon[54076]: debug 2022-01-31T23:07:38.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.249992+0000) 2022-01-31T23:07:38.812 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:38 smithi167 conmon[49112]: debug 2022-01-31T23:07:38.570+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.572421+0000) 2022-01-31T23:07:38.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:38 smithi167 conmon[54076]: debug 2022-01-31T23:07:38.571+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.573409+0000) 2022-01-31T23:07:38.813 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:38 smithi167 conmon[60316]: debug 2022-01-31T23:07:38.571+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.572575+0000) 2022-01-31T23:07:38.839 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[41853]: debug 2022-01-31T23:07:38.571+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.573564+0000) 2022-01-31T23:07:38.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[35325]: debug 2022-01-31T23:07:38.599+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264700 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:38.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[46715]: debug 2022-01-31T23:07:38.569+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.571734+0000) 2022-01-31T23:07:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[51620]: debug 2022-01-31T23:07:38.571+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.573021+0000) 2022-01-31T23:07:38.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[51620]: debug 2022-01-31T23:07:38.622+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.623971+0000) 2022-01-31T23:07:39.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:39 smithi167 conmon[60316]: debug 2022-01-31T23:07:39.007+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:39.008548+0000) 2022-01-31T23:07:39.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:38 smithi167 conmon[49112]: debug 2022-01-31T23:07:38.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.830137+0000) 2022-01-31T23:07:39.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[35325]: debug 2022-01-31T23:07:38.966+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:07:39.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:39 smithi171 conmon[41853]: debug 2022-01-31T23:07:39.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:39.197522+0000) 2022-01-31T23:07:39.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:38 smithi171 conmon[46715]: debug 2022-01-31T23:07:38.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:38.995067+0000) 2022-01-31T23:07:39.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:39 smithi167 conmon[54076]: debug 2022-01-31T23:07:39.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:39.250141+0000) 2022-01-31T23:07:39.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:39 smithi171 conmon[51620]: debug 2022-01-31T23:07:39.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:39.624120+0000) 2022-01-31T23:07:40.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:40 smithi167 conmon[60316]: debug 2022-01-31T23:07:40.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:40.008694+0000) 2022-01-31T23:07:40.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:39 smithi167 conmon[49112]: debug 2022-01-31T23:07:39.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:39.830329+0000) 2022-01-31T23:07:40.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:40 smithi171 conmon[41853]: debug 2022-01-31T23:07:40.196+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:40.197676+0000) 2022-01-31T23:07:40.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:39 smithi171 conmon[46715]: debug 2022-01-31T23:07:39.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:39.995205+0000) 2022-01-31T23:07:40.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:40 smithi167 conmon[54076]: debug 2022-01-31T23:07:40.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:40.250303+0000) 2022-01-31T23:07:40.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:40 smithi171 conmon[51620]: debug 2022-01-31T23:07:40.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:40.624314+0000) 2022-01-31T23:07:41.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:40 smithi167 conmon[49112]: debug 2022-01-31T23:07:40.828+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:40.830454+0000) 2022-01-31T23:07:41.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:41 smithi167 conmon[60316]: debug 2022-01-31T23:07:41.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:41.008885+0000) 2022-01-31T23:07:41.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:40 smithi171 conmon[46715]: debug 2022-01-31T23:07:40.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:40.995355+0000) 2022-01-31T23:07:41.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:41 smithi171 conmon[41853]: debug 2022-01-31T23:07:41.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:41.197859+0000) 2022-01-31T23:07:41.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:41 smithi167 conmon[54076]: debug 2022-01-31T23:07:41.249+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:41.250534+0000) 2022-01-31T23:07:41.977 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:41 smithi171 conmon[51620]: debug 2022-01-31T23:07:41.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:41.624473+0000) 2022-01-31T23:07:42.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:41 smithi167 conmon[49112]: debug 2022-01-31T23:07:41.829+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:41.830546+0000) 2022-01-31T23:07:42.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:42 smithi167 conmon[60316]: debug 2022-01-31T23:07:42.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:42.009074+0000) 2022-01-31T23:07:42.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:41 smithi171 conmon[46715]: debug 2022-01-31T23:07:41.994+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:41.995570+0000) 2022-01-31T23:07:42.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:42 smithi171 conmon[41853]: debug 2022-01-31T23:07:42.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:42.198054+0000) 2022-01-31T23:07:42.521 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:42 smithi167 conmon[54076]: debug 2022-01-31T23:07:42.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:42.250741+0000) 2022-01-31T23:07:42.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:42 smithi171 conmon[51620]: debug 2022-01-31T23:07:42.623+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:42.624693+0000) 2022-01-31T23:07:43.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:42 smithi167 conmon[49112]: debug 2022-01-31T23:07:42.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:42.830694+0000) 2022-01-31T23:07:43.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:43 smithi167 conmon[60316]: debug 2022-01-31T23:07:43.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.009237+0000) 2022-01-31T23:07:43.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[41853]: debug 2022-01-31T23:07:43.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.198255+0000) 2022-01-31T23:07:43.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:42 smithi171 conmon[46715]: debug 2022-01-31T23:07:42.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:42.995745+0000) 2022-01-31T23:07:43.524 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:43 smithi167 conmon[54076]: debug 2022-01-31T23:07:43.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.250889+0000) 2022-01-31T23:07:43.813 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:43 smithi167 conmon[49112]: debug 2022-01-31T23:07:43.603+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.603780+0000) 2022-01-31T23:07:43.813 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:43 smithi167 conmon[54076]: debug 2022-01-31T23:07:43.603+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.604362+0000) 2022-01-31T23:07:43.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:43 smithi167 conmon[60316]: debug 2022-01-31T23:07:43.603+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.603569+0000) 2022-01-31T23:07:43.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[35325]: debug 2022-01-31T23:07:43.631+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264811 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:43.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[41853]: debug 2022-01-31T23:07:43.605+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.605852+0000) 2022-01-31T23:07:43.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[46715]: debug 2022-01-31T23:07:43.602+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.603462+0000) 2022-01-31T23:07:43.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[51620]: debug 2022-01-31T23:07:43.604+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.605655+0000) 2022-01-31T23:07:43.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[51620]: debug 2022-01-31T23:07:43.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.624840+0000) 2022-01-31T23:07:44.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:43 smithi167 conmon[49112]: debug 2022-01-31T23:07:43.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.830909+0000) 2022-01-31T23:07:44.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:44 smithi167 conmon[60316]: debug 2022-01-31T23:07:44.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:44.009375+0000) 2022-01-31T23:07:44.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:44 smithi171 conmon[41853]: debug 2022-01-31T23:07:44.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:44.198459+0000) 2022-01-31T23:07:44.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:43 smithi171 conmon[46715]: debug 2022-01-31T23:07:43.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:43.995913+0000) 2022-01-31T23:07:44.528 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:44 smithi167 conmon[54076]: debug 2022-01-31T23:07:44.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:44.251047+0000) 2022-01-31T23:07:44.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:44 smithi171 conmon[51620]: debug 2022-01-31T23:07:44.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:44.624963+0000) 2022-01-31T23:07:45.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:45 smithi167 conmon[60316]: debug 2022-01-31T23:07:45.008+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:45.009489+0000) 2022-01-31T23:07:45.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:44 smithi167 conmon[49112]: debug 2022-01-31T23:07:44.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:44.831037+0000) 2022-01-31T23:07:45.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:44 smithi171 conmon[46715]: debug 2022-01-31T23:07:44.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:44.996091+0000) 2022-01-31T23:07:45.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:45 smithi171 conmon[41853]: debug 2022-01-31T23:07:45.197+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:45.198613+0000) 2022-01-31T23:07:45.531 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:45 smithi167 conmon[54076]: debug 2022-01-31T23:07:45.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:45.251180+0000) 2022-01-31T23:07:45.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:45 smithi171 conmon[51620]: debug 2022-01-31T23:07:45.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:45.625157+0000) 2022-01-31T23:07:46.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:45 smithi167 conmon[49112]: debug 2022-01-31T23:07:45.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:45.831238+0000) 2022-01-31T23:07:46.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:46 smithi167 conmon[60316]: debug 2022-01-31T23:07:46.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:46.009651+0000) 2022-01-31T23:07:46.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:45 smithi171 conmon[46715]: debug 2022-01-31T23:07:45.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:45.996277+0000) 2022-01-31T23:07:46.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:46 smithi171 conmon[41853]: debug 2022-01-31T23:07:46.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:46.198781+0000) 2022-01-31T23:07:46.535 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:46 smithi167 conmon[54076]: debug 2022-01-31T23:07:46.250+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:46.251368+0000) 2022-01-31T23:07:46.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:46 smithi171 conmon[51620]: debug 2022-01-31T23:07:46.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:46.625267+0000) 2022-01-31T23:07:47.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:46 smithi167 conmon[49112]: debug 2022-01-31T23:07:46.830+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:46.831396+0000) 2022-01-31T23:07:47.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:47 smithi167 conmon[60316]: debug 2022-01-31T23:07:47.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:47.009835+0000) 2022-01-31T23:07:47.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:46 smithi171 conmon[46715]: debug 2022-01-31T23:07:46.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:46.996433+0000) 2022-01-31T23:07:47.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:47 smithi171 conmon[41853]: debug 2022-01-31T23:07:47.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:47.198911+0000) 2022-01-31T23:07:47.538 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:47 smithi167 conmon[54076]: debug 2022-01-31T23:07:47.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:47.251542+0000) 2022-01-31T23:07:47.978 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:47 smithi171 conmon[51620]: debug 2022-01-31T23:07:47.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:47.625507+0000) 2022-01-31T23:07:48.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:47 smithi167 conmon[49112]: debug 2022-01-31T23:07:47.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:47.831583+0000) 2022-01-31T23:07:48.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:48 smithi167 conmon[60316]: debug 2022-01-31T23:07:48.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.010006+0000) 2022-01-31T23:07:48.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[41853]: debug 2022-01-31T23:07:48.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.199007+0000) 2022-01-31T23:07:48.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:47 smithi171 conmon[46715]: debug 2022-01-31T23:07:47.995+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:47.996610+0000) 2022-01-31T23:07:48.542 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:48 smithi167 conmon[54076]: debug 2022-01-31T23:07:48.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.251712+0000) 2022-01-31T23:07:48.814 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:48 smithi167 conmon[49112]: debug 2022-01-31T23:07:48.634+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.634617+0000) 2022-01-31T23:07:48.814 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:48 smithi167 conmon[60316]: debug 2022-01-31T23:07:48.634+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.635376+0000) 2022-01-31T23:07:48.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:48 smithi167 conmon[54076]: debug 2022-01-31T23:07:48.634+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.635195+0000) 2022-01-31T23:07:48.979 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[35325]: debug 2022-01-31T23:07:48.661+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 264923 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:48.979 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[41853]: debug 2022-01-31T23:07:48.634+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.635145+0000) 2022-01-31T23:07:48.980 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[46715]: debug 2022-01-31T23:07:48.633+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.634085+0000) 2022-01-31T23:07:48.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[51620]: debug 2022-01-31T23:07:48.624+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.625660+0000) 2022-01-31T23:07:48.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[51620]: debug 2022-01-31T23:07:48.633+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.634215+0000) 2022-01-31T23:07:49.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:48 smithi167 conmon[49112]: debug 2022-01-31T23:07:48.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.831811+0000) 2022-01-31T23:07:49.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:49 smithi167 conmon[60316]: debug 2022-01-31T23:07:49.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:49.010175+0000) 2022-01-31T23:07:49.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:49 smithi171 conmon[41853]: debug 2022-01-31T23:07:49.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:49.199162+0000) 2022-01-31T23:07:49.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:48 smithi171 conmon[46715]: debug 2022-01-31T23:07:48.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:48.996792+0000) 2022-01-31T23:07:49.546 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:49 smithi167 conmon[54076]: debug 2022-01-31T23:07:49.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:49.251893+0000) 2022-01-31T23:07:49.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:49 smithi171 conmon[51620]: debug 2022-01-31T23:07:49.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:49.625869+0000) 2022-01-31T23:07:50.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:49 smithi167 conmon[49112]: debug 2022-01-31T23:07:49.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:49.831963+0000) 2022-01-31T23:07:50.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:50 smithi167 conmon[60316]: debug 2022-01-31T23:07:50.009+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:50.010328+0000) 2022-01-31T23:07:50.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:50 smithi171 conmon[41853]: debug 2022-01-31T23:07:50.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:50.199342+0000) 2022-01-31T23:07:50.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:49 smithi171 conmon[46715]: debug 2022-01-31T23:07:49.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:49.996897+0000) 2022-01-31T23:07:50.551 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:50 smithi167 conmon[54076]: debug 2022-01-31T23:07:50.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:50.252030+0000) 2022-01-31T23:07:50.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:50 smithi171 conmon[51620]: debug 2022-01-31T23:07:50.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:50.626045+0000) 2022-01-31T23:07:51.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:50 smithi167 conmon[49112]: debug 2022-01-31T23:07:50.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:50.832157+0000) 2022-01-31T23:07:51.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:51 smithi167 conmon[60316]: debug 2022-01-31T23:07:51.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:51.010535+0000) 2022-01-31T23:07:51.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:51 smithi171 conmon[41853]: debug 2022-01-31T23:07:51.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:51.199499+0000) 2022-01-31T23:07:51.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:50 smithi171 conmon[46715]: debug 2022-01-31T23:07:50.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:50.997091+0000) 2022-01-31T23:07:51.555 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:51 smithi167 conmon[54076]: debug 2022-01-31T23:07:51.251+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:51.252243+0000) 2022-01-31T23:07:51.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:51 smithi171 conmon[51620]: debug 2022-01-31T23:07:51.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:51.626246+0000) 2022-01-31T23:07:52.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:51 smithi167 conmon[49112]: debug 2022-01-31T23:07:51.831+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:51.832338+0000) 2022-01-31T23:07:52.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:52 smithi167 conmon[60316]: debug 2022-01-31T23:07:52.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:52.010741+0000) 2022-01-31T23:07:52.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:52 smithi171 conmon[41853]: debug 2022-01-31T23:07:52.198+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:52.199702+0000) 2022-01-31T23:07:52.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:51 smithi171 conmon[46715]: debug 2022-01-31T23:07:51.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:51.997257+0000) 2022-01-31T23:07:52.560 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:52 smithi167 conmon[54076]: debug 2022-01-31T23:07:52.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:52.252418+0000) 2022-01-31T23:07:52.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:52 smithi171 conmon[51620]: debug 2022-01-31T23:07:52.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:52.626435+0000) 2022-01-31T23:07:53.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:52 smithi167 conmon[49112]: debug 2022-01-31T23:07:52.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:52.832490+0000) 2022-01-31T23:07:53.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:53 smithi167 conmon[60316]: debug 2022-01-31T23:07:53.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.010942+0000) 2022-01-31T23:07:53.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[41853]: debug 2022-01-31T23:07:53.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.199913+0000) 2022-01-31T23:07:53.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:52 smithi171 conmon[46715]: debug 2022-01-31T23:07:52.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:52.997421+0000) 2022-01-31T23:07:53.564 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:53 smithi167 conmon[54076]: debug 2022-01-31T23:07:53.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.252576+0000) 2022-01-31T23:07:53.815 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:53 smithi167 conmon[49112]: debug 2022-01-31T23:07:53.665+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.665724+0000) 2022-01-31T23:07:53.815 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:53 smithi167 conmon[54076]: debug 2022-01-31T23:07:53.665+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.665988+0000) 2022-01-31T23:07:53.816 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:53 smithi167 conmon[60316]: debug 2022-01-31T23:07:53.666+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.666544+0000) 2022-01-31T23:07:53.950 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[35325]: debug 2022-01-31T23:07:53.692+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265050 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:53.951 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[41853]: debug 2022-01-31T23:07:53.663+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.664613+0000) 2022-01-31T23:07:53.951 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[46715]: debug 2022-01-31T23:07:53.664+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.665005+0000) 2022-01-31T23:07:53.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[51620]: debug 2022-01-31T23:07:53.625+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.626670+0000) 2022-01-31T23:07:53.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[51620]: debug 2022-01-31T23:07:53.664+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.665306+0000) 2022-01-31T23:07:54.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:53 smithi167 conmon[49112]: debug 2022-01-31T23:07:53.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.832668+0000) 2022-01-31T23:07:54.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:54 smithi167 conmon[60316]: debug 2022-01-31T23:07:54.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:54.011110+0000) 2022-01-31T23:07:54.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[35325]: debug 2022-01-31T23:07:53.967+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:07:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:53 smithi171 conmon[46715]: debug 2022-01-31T23:07:53.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:53.997578+0000) 2022-01-31T23:07:54.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:54 smithi171 conmon[41853]: debug 2022-01-31T23:07:54.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:54.200123+0000) 2022-01-31T23:07:54.568 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:54 smithi167 conmon[54076]: debug 2022-01-31T23:07:54.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:54.252765+0000) 2022-01-31T23:07:54.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:54 smithi167 conmon[49112]: debug 2022-01-31T23:07:54.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:54.832824+0000) 2022-01-31T23:07:54.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:54 smithi171 conmon[51620]: debug 2022-01-31T23:07:54.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:54.626893+0000) 2022-01-31T23:07:55.234 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:55 smithi167 conmon[60316]: debug 2022-01-31T23:07:55.010+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:55.011325+0000) 2022-01-31T23:07:55.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:54 smithi171 conmon[46715]: debug 2022-01-31T23:07:54.996+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:54.997704+0000) 2022-01-31T23:07:55.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:55 smithi171 conmon[41853]: debug 2022-01-31T23:07:55.199+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:55.200234+0000) 2022-01-31T23:07:55.572 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:55 smithi167 conmon[54076]: debug 2022-01-31T23:07:55.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:55.252925+0000) 2022-01-31T23:07:55.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:55 smithi167 conmon[49112]: debug 2022-01-31T23:07:55.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:55.833030+0000) 2022-01-31T23:07:55.979 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:55 smithi171 conmon[51620]: debug 2022-01-31T23:07:55.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:55.627062+0000) 2022-01-31T23:07:56.235 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:56 smithi167 conmon[60316]: debug 2022-01-31T23:07:56.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:56.011536+0000) 2022-01-31T23:07:56.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:55 smithi171 conmon[46715]: debug 2022-01-31T23:07:55.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:55.997902+0000) 2022-01-31T23:07:56.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:56 smithi171 conmon[41853]: debug 2022-01-31T23:07:56.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:56.200429+0000) 2022-01-31T23:07:56.575 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:56 smithi167 conmon[54076]: debug 2022-01-31T23:07:56.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:56.253051+0000) 2022-01-31T23:07:56.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:56 smithi167 conmon[49112]: debug 2022-01-31T23:07:56.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:56.833187+0000) 2022-01-31T23:07:56.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:56 smithi171 conmon[51620]: debug 2022-01-31T23:07:56.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:56.627273+0000) 2022-01-31T23:07:57.235 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:57 smithi167 conmon[60316]: debug 2022-01-31T23:07:57.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:57.011690+0000) 2022-01-31T23:07:57.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:57 smithi171 conmon[41853]: debug 2022-01-31T23:07:57.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:57.200610+0000) 2022-01-31T23:07:57.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:56 smithi171 conmon[46715]: debug 2022-01-31T23:07:56.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:56.998068+0000) 2022-01-31T23:07:57.579 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:57 smithi167 conmon[54076]: debug 2022-01-31T23:07:57.252+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:57.253206+0000) 2022-01-31T23:07:57.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:57 smithi167 conmon[49112]: debug 2022-01-31T23:07:57.832+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:57.833380+0000) 2022-01-31T23:07:57.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:57 smithi171 conmon[51620]: debug 2022-01-31T23:07:57.626+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:57.627403+0000) 2022-01-31T23:07:58.235 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:58 smithi167 conmon[60316]: debug 2022-01-31T23:07:58.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.011878+0000) 2022-01-31T23:07:58.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:57 smithi171 conmon[46715]: debug 2022-01-31T23:07:57.997+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:57.998261+0000) 2022-01-31T23:07:58.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[41853]: debug 2022-01-31T23:07:58.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.200820+0000) 2022-01-31T23:07:58.583 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:58 smithi167 conmon[54076]: debug 2022-01-31T23:07:58.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.253408+0000) 2022-01-31T23:07:58.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:58 smithi167 conmon[54076]: debug 2022-01-31T23:07:58.696+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.696790+0000) 2022-01-31T23:07:58.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:58 smithi167 conmon[60316]: debug 2022-01-31T23:07:58.697+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.697859+0000) 2022-01-31T23:07:58.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:58 smithi167 conmon[49112]: debug 2022-01-31T23:07:58.696+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.696697+0000) 2022-01-31T23:07:58.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:58 smithi167 conmon[49112]: debug 2022-01-31T23:07:58.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.833569+0000) 2022-01-31T23:07:58.980 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[35325]: debug 2022-01-31T23:07:58.723+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265162 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:07:58.981 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[46715]: debug 2022-01-31T23:07:58.695+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.696076+0000) 2022-01-31T23:07:58.981 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[41853]: debug 2022-01-31T23:07:58.695+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.695917+0000) 2022-01-31T23:07:58.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[51620]: debug 2022-01-31T23:07:58.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.627558+0000) 2022-01-31T23:07:58.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[51620]: debug 2022-01-31T23:07:58.696+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.697169+0000) 2022-01-31T23:07:59.235 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:07:59 smithi167 conmon[60316]: debug 2022-01-31T23:07:59.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:59.012054+0000) 2022-01-31T23:07:59.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:58 smithi171 conmon[46715]: debug 2022-01-31T23:07:58.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:58.998452+0000) 2022-01-31T23:07:59.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:07:59 smithi171 conmon[41853]: debug 2022-01-31T23:07:59.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:59.200973+0000) 2022-01-31T23:07:59.586 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:07:59 smithi167 conmon[54076]: debug 2022-01-31T23:07:59.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:59.253574+0000) 2022-01-31T23:07:59.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:07:59 smithi167 conmon[49112]: debug 2022-01-31T23:07:59.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:59.833699+0000) 2022-01-31T23:07:59.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:07:59 smithi171 conmon[51620]: debug 2022-01-31T23:07:59.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:59.627790+0000) 2022-01-31T23:08:00.235 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:00 smithi167 conmon[60316]: debug 2022-01-31T23:08:00.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:00.012210+0000) 2022-01-31T23:08:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:07:59 smithi171 conmon[46715]: debug 2022-01-31T23:07:59.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:07:59.998583+0000) 2022-01-31T23:08:00.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:00 smithi171 conmon[41853]: debug 2022-01-31T23:08:00.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:00.201084+0000) 2022-01-31T23:08:00.591 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:00 smithi167 conmon[54076]: debug 2022-01-31T23:08:00.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:00.253720+0000) 2022-01-31T23:08:00.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:00 smithi167 conmon[49112]: debug 2022-01-31T23:08:00.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:00.833835+0000) 2022-01-31T23:08:00.980 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:00 smithi171 conmon[51620]: debug 2022-01-31T23:08:00.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:00.627966+0000) 2022-01-31T23:08:01.235 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:01 smithi167 conmon[60316]: debug 2022-01-31T23:08:01.011+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:01.012399+0000) 2022-01-31T23:08:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:00 smithi171 conmon[46715]: debug 2022-01-31T23:08:00.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:00.998787+0000) 2022-01-31T23:08:01.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:01 smithi171 conmon[41853]: debug 2022-01-31T23:08:01.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:01.201264+0000) 2022-01-31T23:08:01.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:01 smithi167 conmon[54076]: debug 2022-01-31T23:08:01.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:01.253925+0000) 2022-01-31T23:08:01.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:01 smithi167 conmon[49112]: debug 2022-01-31T23:08:01.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:01.834014+0000) 2022-01-31T23:08:01.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:01 smithi171 conmon[51620]: debug 2022-01-31T23:08:01.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:01.628181+0000) 2022-01-31T23:08:02.236 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:02 smithi167 conmon[60316]: debug 2022-01-31T23:08:02.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:02.012580+0000) 2022-01-31T23:08:02.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:02 smithi171 conmon[41853]: debug 2022-01-31T23:08:02.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:02.201483+0000) 2022-01-31T23:08:02.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:01 smithi171 conmon[46715]: debug 2022-01-31T23:08:01.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:01.998948+0000) 2022-01-31T23:08:02.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:02 smithi167 conmon[54076]: debug 2022-01-31T23:08:02.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:02.254122+0000) 2022-01-31T23:08:02.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:02 smithi171 conmon[51620]: debug 2022-01-31T23:08:02.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:02.628348+0000) 2022-01-31T23:08:02.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:02 smithi167 conmon[49112]: debug 2022-01-31T23:08:02.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:02.834171+0000) 2022-01-31T23:08:03.236 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:03 smithi167 conmon[60316]: debug 2022-01-31T23:08:03.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.012785+0000) 2022-01-31T23:08:03.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[41853]: debug 2022-01-31T23:08:03.200+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.201686+0000) 2022-01-31T23:08:03.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:02 smithi171 conmon[46715]: debug 2022-01-31T23:08:02.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:02.999106+0000) 2022-01-31T23:08:03.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:03 smithi167 conmon[54076]: debug 2022-01-31T23:08:03.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.254337+0000) 2022-01-31T23:08:03.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:03 smithi167 conmon[54076]: debug 2022-01-31T23:08:03.726+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.727269+0000) 2022-01-31T23:08:03.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:03 smithi167 conmon[60316]: debug 2022-01-31T23:08:03.727+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.727951+0000) 2022-01-31T23:08:03.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:03 smithi167 conmon[49112]: debug 2022-01-31T23:08:03.725+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.726394+0000) 2022-01-31T23:08:03.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:03 smithi167 conmon[49112]: debug 2022-01-31T23:08:03.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.834357+0000) 2022-01-31T23:08:03.981 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[35325]: debug 2022-01-31T23:08:03.752+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265274 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:03.982 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[41853]: debug 2022-01-31T23:08:03.725+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.725985+0000) 2022-01-31T23:08:03.982 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[46715]: debug 2022-01-31T23:08:03.725+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.725835+0000) 2022-01-31T23:08:03.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[51620]: debug 2022-01-31T23:08:03.627+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.628500+0000) 2022-01-31T23:08:03.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[51620]: debug 2022-01-31T23:08:03.725+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.726621+0000) 2022-01-31T23:08:04.236 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:04 smithi167 conmon[60316]: debug 2022-01-31T23:08:04.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:04 smithi167 conmon[60316]: 2022-01-31T23:08:04.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:04.012957+0000) 2022-01-31T23:08:04.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:03 smithi171 conmon[46715]: debug 2022-01-31T23:08:03.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:03.999325+0000) 2022-01-31T23:08:04.336 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:04 smithi171 conmon[41853]: debug 2022-01-31T23:08:04.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:04.201877+0000) 2022-01-31T23:08:04.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:04 smithi167 conmon[54076]: debug 2022-01-31T23:08:04.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:04.254552+0000) 2022-01-31T23:08:04.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:04 smithi167 conmon[49112]: debug 2022-01-31T23:08:04.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:04.834524+0000) 2022-01-31T23:08:04.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:04 smithi171 conmon[51620]: debug 2022-01-31T23:08:04.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:04.628694+0000) 2022-01-31T23:08:05.236 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:05 smithi167 conmon[60316]: debug 2022-01-31T23:08:05.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:05.013163+0000) 2022-01-31T23:08:05.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:05 smithi171 conmon[41853]: debug 2022-01-31T23:08:05.201+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:05.202041+0000) 2022-01-31T23:08:05.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:04 smithi171 conmon[46715]: debug 2022-01-31T23:08:04.998+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:04.999540+0000) 2022-01-31T23:08:05.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:05 smithi167 conmon[54076]: debug 2022-01-31T23:08:05.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:05.254707+0000) 2022-01-31T23:08:05.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:05 smithi167 conmon[49112]: debug 2022-01-31T23:08:05.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:05.834669+0000) 2022-01-31T23:08:05.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:05 smithi171 conmon[51620]: debug 2022-01-31T23:08:05.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:05.628862+0000) 2022-01-31T23:08:06.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:06 smithi167 conmon[60316]: debug 2022-01-31T23:08:06.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:06.013370+0000) 2022-01-31T23:08:06.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:05 smithi171 conmon[46715]: debug 2022-01-31T23:08:05.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:05.999669+0000) 2022-01-31T23:08:06.321 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:06 smithi171 conmon[41853]: debug 2022-01-31T23:08:06.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:06.202237+0000) 2022-01-31T23:08:06.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:06 smithi167 conmon[54076]: debug 2022-01-31T23:08:06.253+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:06.254900+0000) 2022-01-31T23:08:06.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:06 smithi167 conmon[49112]: debug 2022-01-31T23:08:06.833+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:06.834865+0000) 2022-01-31T23:08:06.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:06 smithi171 conmon[51620]: debug 2022-01-31T23:08:06.628+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:06.629076+0000) 2022-01-31T23:08:07.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:07 smithi167 conmon[60316]: debug 2022-01-31T23:08:07.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:07.013505+0000) 2022-01-31T23:08:07.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:06 smithi171 conmon[46715]: debug 2022-01-31T23:08:06.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:06.999819+0000) 2022-01-31T23:08:07.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:07 smithi171 conmon[41853]: debug 2022-01-31T23:08:07.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:07.202427+0000) 2022-01-31T23:08:07.595 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:07 smithi167 conmon[54076]: debug 2022-01-31T23:08:07.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:07.255079+0000) 2022-01-31T23:08:07.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:07 smithi167 conmon[49112]: debug 2022-01-31T23:08:07.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:07.835025+0000) 2022-01-31T23:08:07.981 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:07 smithi171 conmon[51620]: debug 2022-01-31T23:08:07.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:07.629288+0000) 2022-01-31T23:08:08.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:08 smithi167 conmon[60316]: debug 2022-01-31T23:08:08.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.013688+0000) 2022-01-31T23:08:08.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[46715]: debug 2022-01-31T23:08:07.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:07.999986+0000) 2022-01-31T23:08:08.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[41853]: debug 2022-01-31T23:08:08.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.202612+0000) 2022-01-31T23:08:08.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:08 smithi167 conmon[54076]: debug 2022-01-31T23:08:08.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.255244+0000) 2022-01-31T23:08:08.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:08 smithi167 conmon[49112]: debug 2022-01-31T23:08:08.754+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.756212+0000) 2022-01-31T23:08:08.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:08 smithi167 conmon[49112]: debug 2022-01-31T23:08:08.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.835197+0000) 2022-01-31T23:08:08.903 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:08 smithi167 conmon[54076]: debug 2022-01-31T23:08:08.755+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.757390+0000) 2022-01-31T23:08:08.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:08 smithi167 conmon[60316]: debug 2022-01-31T23:08:08.756+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.757904+0000) 2022-01-31T23:08:08.951 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[35325]: debug 2022-01-31T23:08:08.784+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265389 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:08.951 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[41853]: debug 2022-01-31T23:08:08.756+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.757028+0000) 2022-01-31T23:08:08.952 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[46715]: debug 2022-01-31T23:08:08.754+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.755633+0000) 2022-01-31T23:08:08.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[51620]: debug 2022-01-31T23:08:08.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.629465+0000) 2022-01-31T23:08:08.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[51620]: debug 2022-01-31T23:08:08.756+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:08.757538+0000) 2022-01-31T23:08:09.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:09 smithi167 conmon[60316]: debug 2022-01-31T23:08:09.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:09.013822+0000) 2022-01-31T23:08:09.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:08 smithi171 conmon[35325]: debug 2022-01-31T23:08:08.968+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:08:09.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:09 smithi171 conmon[41853]: debug 2022-01-31T23:08:09.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:09.202794+0000) 2022-01-31T23:08:09.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:09 smithi171 conmon[46715]: debug 2022-01-31T23:08:08.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:09.000107+0000) 2022-01-31T23:08:09.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:09 smithi167 conmon[54076]: debug 2022-01-31T23:08:09.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:09.255442+0000) 2022-01-31T23:08:09.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:09 smithi167 conmon[49112]: debug 2022-01-31T23:08:09.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:09.835318+0000) 2022-01-31T23:08:09.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:09 smithi171 conmon[51620]: debug 2022-01-31T23:08:09.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:09.629703+0000) 2022-01-31T23:08:10.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:10 smithi167 conmon[60316]: debug 2022-01-31T23:08:10.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:10.013899+0000) 2022-01-31T23:08:10.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:10 smithi171 conmon[46715]: debug 2022-01-31T23:08:10.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:10.000259+0000) 2022-01-31T23:08:10.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:10 smithi171 conmon[41853]: debug 2022-01-31T23:08:10.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:10.202927+0000) 2022-01-31T23:08:10.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:10 smithi167 conmon[54076]: debug 2022-01-31T23:08:10.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:10.255575+0000) 2022-01-31T23:08:10.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:10 smithi167 conmon[49112]: debug 2022-01-31T23:08:10.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:10.835482+0000) 2022-01-31T23:08:10.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:10 smithi171 conmon[51620]: debug 2022-01-31T23:08:10.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:10.629866+0000) 2022-01-31T23:08:11.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:11 smithi167 conmon[60316]: debug 2022-01-31T23:08:11.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:11.014081+0000) 2022-01-31T23:08:11.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:11 smithi171 conmon[41853]: debug 2022-01-31T23:08:11.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:11.203113+0000) 2022-01-31T23:08:11.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:11 smithi171 conmon[46715]: debug 2022-01-31T23:08:10.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:11.000436+0000) 2022-01-31T23:08:11.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:11 smithi167 conmon[54076]: debug 2022-01-31T23:08:11.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:11.255767+0000) 2022-01-31T23:08:11.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:11 smithi167 conmon[49112]: debug 2022-01-31T23:08:11.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:11.835686+0000) 2022-01-31T23:08:11.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:11 smithi171 conmon[51620]: debug 2022-01-31T23:08:11.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:11.630042+0000) 2022-01-31T23:08:12.237 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:12 smithi167 conmon[60316]: debug 2022-01-31T23:08:12.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:12.014301+0000) 2022-01-31T23:08:12.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:12 smithi171 conmon[41853]: debug 2022-01-31T23:08:12.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:12.203301+0000) 2022-01-31T23:08:12.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:12 smithi171 conmon[46715]: debug 2022-01-31T23:08:12.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:12.000596+0000) 2022-01-31T23:08:12.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:12 smithi167 conmon[54076]: debug 2022-01-31T23:08:12.254+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:12.255945+0000) 2022-01-31T23:08:12.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:12 smithi167 conmon[49112]: debug 2022-01-31T23:08:12.834+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:12.835871+0000) 2022-01-31T23:08:12.982 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:12 smithi171 conmon[51620]: debug 2022-01-31T23:08:12.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:12.630245+0000) 2022-01-31T23:08:13.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:13 smithi167 conmon[60316]: debug 2022-01-31T23:08:13.012+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.014474+0000) 2022-01-31T23:08:13.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[41853]: debug 2022-01-31T23:08:13.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.203521+0000) 2022-01-31T23:08:13.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[46715]: debug 2022-01-31T23:08:13.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.000740+0000) 2022-01-31T23:08:13.596 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:13 smithi167 conmon[54076]: debug 2022-01-31T23:08:13.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.256079+0000) 2022-01-31T23:08:13.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:13 smithi167 conmon[54076]: debug 2022-01-31T23:08:13.785+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.787406+0000) 2022-01-31T23:08:13.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:13 smithi167 conmon[60316]: debug 2022-01-31T23:08:13.787+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.788572+0000) 2022-01-31T23:08:13.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:13 smithi167 conmon[49112]: debug 2022-01-31T23:08:13.786+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.787953+0000) 2022-01-31T23:08:13.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:13 smithi167 conmon[49112]: debug 2022-01-31T23:08:13.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.836045+0000) 2022-01-31T23:08:13.983 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[35325]: debug 2022-01-31T23:08:13.814+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265500 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:13.983 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[41853]: debug 2022-01-31T23:08:13.787+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.788349+0000) 2022-01-31T23:08:13.984 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[46715]: debug 2022-01-31T23:08:13.785+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.786515+0000) 2022-01-31T23:08:13.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[51620]: debug 2022-01-31T23:08:13.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.630479+0000) 2022-01-31T23:08:13.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:13 smithi171 conmon[51620]: debug 2022-01-31T23:08:13.787+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:13.788102+0000) 2022-01-31T23:08:14.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:14 smithi167 conmon[60316]: debug 2022-01-31T23:08:14.013+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:14.014714+0000) 2022-01-31T23:08:14.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:14 smithi171 conmon[41853]: debug 2022-01-31T23:08:14.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:14.203725+0000) 2022-01-31T23:08:14.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:14 smithi171 conmon[46715]: debug 2022-01-31T23:08:14.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:14.000942+0000) 2022-01-31T23:08:14.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:14 smithi167 conmon[54076]: debug 2022-01-31T23:08:14.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:14.256240+0000) 2022-01-31T23:08:14.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:14 smithi167 conmon[49112]: debug 2022-01-31T23:08:14.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:14.836183+0000) 2022-01-31T23:08:14.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:14 smithi171 conmon[51620]: debug 2022-01-31T23:08:14.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:14.630696+0000) 2022-01-31T23:08:15.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:15 smithi167 conmon[60316]: debug 2022-01-31T23:08:15.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:15.014920+0000) 2022-01-31T23:08:15.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:15 smithi171 conmon[41853]: debug 2022-01-31T23:08:15.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:15.203879+0000) 2022-01-31T23:08:15.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:15 smithi171 conmon[46715]: debug 2022-01-31T23:08:14.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:15.001077+0000) 2022-01-31T23:08:15.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:15 smithi167 conmon[54076]: debug 2022-01-31T23:08:15.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:15.256367+0000) 2022-01-31T23:08:15.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:15 smithi167 conmon[49112]: debug 2022-01-31T23:08:15.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:15.836355+0000) 2022-01-31T23:08:15.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:15 smithi171 conmon[51620]: debug 2022-01-31T23:08:15.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:15.630867+0000) 2022-01-31T23:08:16.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:16 smithi167 conmon[60316]: debug 2022-01-31T23:08:16.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:16.015050+0000) 2022-01-31T23:08:16.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:16 smithi171 conmon[46715]: debug 2022-01-31T23:08:16.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:16.001285+0000) 2022-01-31T23:08:16.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:16 smithi171 conmon[41853]: debug 2022-01-31T23:08:16.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:16.204081+0000) 2022-01-31T23:08:16.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:16 smithi167 conmon[54076]: debug 2022-01-31T23:08:16.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:16.256532+0000) 2022-01-31T23:08:16.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:16 smithi167 conmon[49112]: debug 2022-01-31T23:08:16.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:16.836527+0000) 2022-01-31T23:08:16.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:16 smithi171 conmon[51620]: debug 2022-01-31T23:08:16.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:16.631087+0000) 2022-01-31T23:08:17.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:17 smithi167 conmon[60316]: debug 2022-01-31T23:08:17.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:17.015211+0000) 2022-01-31T23:08:17.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:17 smithi171 conmon[46715]: debug 2022-01-31T23:08:17.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:17.001445+0000) 2022-01-31T23:08:17.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:17 smithi171 conmon[41853]: debug 2022-01-31T23:08:17.202+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:17.204241+0000) 2022-01-31T23:08:17.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:17 smithi167 conmon[54076]: debug 2022-01-31T23:08:17.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:17.256679+0000) 2022-01-31T23:08:17.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:17 smithi167 conmon[49112]: debug 2022-01-31T23:08:17.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:17.836670+0000) 2022-01-31T23:08:17.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:17 smithi171 conmon[51620]: debug 2022-01-31T23:08:17.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:17.631299+0000) 2022-01-31T23:08:18.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:18 smithi167 conmon[60316]: debug 2022-01-31T23:08:18.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.015388+0000) 2022-01-31T23:08:18.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[46715]: debug 2022-01-31T23:08:17.999+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.001651+0000) 2022-01-31T23:08:18.346 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[41853]: debug 2022-01-31T23:08:18.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.204396+0000) 2022-01-31T23:08:18.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:18 smithi167 conmon[54076]: debug 2022-01-31T23:08:18.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.256779+0000) 2022-01-31T23:08:18.901 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:18 smithi167 conmon[54076]: debug 2022-01-31T23:08:18.816+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.818132+0000) 2022-01-31T23:08:18.902 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:18 smithi167 conmon[60316]: debug 2022-01-31T23:08:18.817+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.818865+0000) 2022-01-31T23:08:18.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:18 smithi167 conmon[49112]: debug 2022-01-31T23:08:18.816+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.818431+0000) 2022-01-31T23:08:18.903 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:18 smithi167 conmon[49112]: debug 2022-01-31T23:08:18.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.836798+0000) 2022-01-31T23:08:18.983 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[35325]: debug 2022-01-31T23:08:18.845+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265614 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:18.984 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[41853]: debug 2022-01-31T23:08:18.816+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.817831+0000) 2022-01-31T23:08:18.985 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[46715]: debug 2022-01-31T23:08:18.815+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.817092+0000) 2022-01-31T23:08:18.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[51620]: debug 2022-01-31T23:08:18.629+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.631495+0000) 2022-01-31T23:08:18.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:18 smithi171 conmon[51620]: debug 2022-01-31T23:08:18.815+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:18.817394+0000) 2022-01-31T23:08:19.238 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:19 smithi167 conmon[60316]: debug 2022-01-31T23:08:19.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:19.015572+0000) 2022-01-31T23:08:19.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:19 smithi171 conmon[41853]: debug 2022-01-31T23:08:19.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:19.204583+0000) 2022-01-31T23:08:19.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:19 smithi171 conmon[46715]: debug 2022-01-31T23:08:19.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:19.001872+0000) 2022-01-31T23:08:19.594 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:19 smithi167 conmon[54076]: debug 2022-01-31T23:08:19.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:19.256939+0000) 2022-01-31T23:08:19.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:19 smithi167 conmon[49112]: debug 2022-01-31T23:08:19.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:19.836915+0000) 2022-01-31T23:08:19.983 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:19 smithi171 conmon[51620]: debug 2022-01-31T23:08:19.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:19.631705+0000) 2022-01-31T23:08:20.239 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:20 smithi167 conmon[60316]: debug 2022-01-31T23:08:20.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:20.015707+0000) 2022-01-31T23:08:20.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:20 smithi171 conmon[41853]: debug 2022-01-31T23:08:20.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:20.204739+0000) 2022-01-31T23:08:20.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:20 smithi171 conmon[46715]: debug 2022-01-31T23:08:20.000+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:20.002008+0000) 2022-01-31T23:08:20.597 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:20 smithi167 conmon[54076]: debug 2022-01-31T23:08:20.255+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:20.257109+0000) 2022-01-31T23:08:20.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:20 smithi167 conmon[49112]: debug 2022-01-31T23:08:20.835+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:20.837102+0000) 2022-01-31T23:08:20.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:20 smithi171 conmon[51620]: debug 2022-01-31T23:08:20.630+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:20.631882+0000) 2022-01-31T23:08:21.239 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:21 smithi167 conmon[60316]: debug 2022-01-31T23:08:21.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:21.015875+0000) 2022-01-31T23:08:21.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:21 smithi171 conmon[41853]: debug 2022-01-31T23:08:21.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:21.204968+0000) 2022-01-31T23:08:21.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:21 smithi171 conmon[46715]: debug 2022-01-31T23:08:21.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:21.002211+0000) 2022-01-31T23:08:21.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:21 smithi167 conmon[54076]: debug 2022-01-31T23:08:21.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:21.257284+0000) 2022-01-31T23:08:21.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:21 smithi167 conmon[49112]: debug 2022-01-31T23:08:21.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:21.837259+0000) 2022-01-31T23:08:21.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:21 smithi171 conmon[51620]: debug 2022-01-31T23:08:21.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:21.632084+0000) 2022-01-31T23:08:22.239 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:22 smithi167 conmon[60316]: debug 2022-01-31T23:08:22.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:22.016071+0000) 2022-01-31T23:08:22.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:22 smithi171 conmon[41853]: debug 2022-01-31T23:08:22.203+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:22.205157+0000) 2022-01-31T23:08:22.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:22 smithi171 conmon[46715]: debug 2022-01-31T23:08:22.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:22.002401+0000) 2022-01-31T23:08:22.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:22 smithi167 conmon[54076]: debug 2022-01-31T23:08:22.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:22.257440+0000) 2022-01-31T23:08:22.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:22 smithi167 conmon[49112]: debug 2022-01-31T23:08:22.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:22.837443+0000) 2022-01-31T23:08:22.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:22 smithi171 conmon[51620]: debug 2022-01-31T23:08:22.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:22.632296+0000) 2022-01-31T23:08:23.239 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:23 smithi167 conmon[60316]: debug 2022-01-31T23:08:23.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.016255+0000) 2022-01-31T23:08:23.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[41853]: debug 2022-01-31T23:08:23.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.205394+0000) 2022-01-31T23:08:23.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[46715]: debug 2022-01-31T23:08:23.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.002584+0000) 2022-01-31T23:08:23.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:23 smithi167 conmon[54076]: debug 2022-01-31T23:08:23.256+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.257622+0000) 2022-01-31T23:08:23.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:23 smithi167 conmon[49112]: debug 2022-01-31T23:08:23.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.837668+0000) 2022-01-31T23:08:23.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:23 smithi167 conmon[49112]: debug 2022-01-31T23:08:23.848+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.850207+0000) 2022-01-31T23:08:23.902 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:23 smithi167 conmon[54076]: debug 2022-01-31T23:08:23.848+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.849514+0000) 2022-01-31T23:08:23.903 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:23 smithi167 conmon[60316]: debug 2022-01-31T23:08:23.849+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.850642+0000) 2022-01-31T23:08:23.952 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[35325]: debug 2022-01-31T23:08:23.876+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265721 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:23.952 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[41853]: debug 2022-01-31T23:08:23.847+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.849257+0000) 2022-01-31T23:08:23.953 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[46715]: debug 2022-01-31T23:08:23.847+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.849403+0000) 2022-01-31T23:08:23.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[51620]: debug 2022-01-31T23:08:23.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.632501+0000) 2022-01-31T23:08:23.954 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[51620]: debug 2022-01-31T23:08:23.847+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:23.849652+0000) 2022-01-31T23:08:24.239 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:24 smithi167 conmon[60316]: debug 2022-01-31T23:08:24.014+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:24.016391+0000) 2022-01-31T23:08:24.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:24 smithi171 conmon[41853]: debug 2022-01-31T23:08:24.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:24.205544+0000) 2022-01-31T23:08:24.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:24 smithi171 conmon[46715]: debug 2022-01-31T23:08:24.001+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:24.002786+0000) 2022-01-31T23:08:24.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:23 smithi171 conmon[35325]: debug 2022-01-31T23:08:23.968+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:08:24.598 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:24 smithi167 conmon[54076]: debug 2022-01-31T23:08:24.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:24.257779+0000) 2022-01-31T23:08:24.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:24 smithi167 conmon[49112]: debug 2022-01-31T23:08:24.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:24.837854+0000) 2022-01-31T23:08:24.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:24 smithi171 conmon[51620]: debug 2022-01-31T23:08:24.631+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:24.632688+0000) 2022-01-31T23:08:25.239 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:25 smithi167 conmon[60316]: debug 2022-01-31T23:08:25.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:25.016570+0000) 2022-01-31T23:08:25.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:25 smithi171 conmon[41853]: debug 2022-01-31T23:08:25.204+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:25.205683+0000) 2022-01-31T23:08:25.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:25 smithi171 conmon[46715]: debug 2022-01-31T23:08:25.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:25.002944+0000) 2022-01-31T23:08:25.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:25 smithi167 conmon[54076]: debug 2022-01-31T23:08:25.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:25.257937+0000) 2022-01-31T23:08:25.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:25 smithi167 conmon[49112]: debug 2022-01-31T23:08:25.836+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:25.838054+0000) 2022-01-31T23:08:25.984 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:25 smithi171 conmon[51620]: debug 2022-01-31T23:08:25.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:25.632878+0000) 2022-01-31T23:08:26.240 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:26 smithi167 conmon[60316]: debug 2022-01-31T23:08:26.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:26.016779+0000) 2022-01-31T23:08:26.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:26 smithi171 conmon[46715]: debug 2022-01-31T23:08:26.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:26.003043+0000) 2022-01-31T23:08:26.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:26 smithi171 conmon[41853]: debug 2022-01-31T23:08:26.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:26.205904+0000) 2022-01-31T23:08:26.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:26 smithi167 conmon[54076]: debug 2022-01-31T23:08:26.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:26.258126+0000) 2022-01-31T23:08:26.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:26 smithi167 conmon[49112]: debug 2022-01-31T23:08:26.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:26.838244+0000) 2022-01-31T23:08:26.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:26 smithi171 conmon[51620]: debug 2022-01-31T23:08:26.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:26.633009+0000) 2022-01-31T23:08:27.240 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:27 smithi167 conmon[60316]: debug 2022-01-31T23:08:27.015+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:27.016911+0000) 2022-01-31T23:08:27.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:27 smithi171 conmon[46715]: debug 2022-01-31T23:08:27.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:27.003194+0000) 2022-01-31T23:08:27.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:27 smithi171 conmon[41853]: debug 2022-01-31T23:08:27.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:27.206031+0000) 2022-01-31T23:08:27.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:27 smithi167 conmon[54076]: debug 2022-01-31T23:08:27.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:27.258314+0000) 2022-01-31T23:08:27.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:27 smithi167 conmon[49112]: debug 2022-01-31T23:08:27.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:27.838473+0000) 2022-01-31T23:08:27.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:27 smithi171 conmon[51620]: debug 2022-01-31T23:08:27.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:27.633200+0000) 2022-01-31T23:08:28.240 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:28 smithi167 conmon[60316]: debug 2022-01-31T23:08:28.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.017077+0000) 2022-01-31T23:08:28.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[46715]: debug 2022-01-31T23:08:28.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.003403+0000) 2022-01-31T23:08:28.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[41853]: debug 2022-01-31T23:08:28.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.206146+0000) 2022-01-31T23:08:28.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:28 smithi167 conmon[54076]: debug 2022-01-31T23:08:28.257+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.258518+0000) 2022-01-31T23:08:28.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:28 smithi167 conmon[49112]: debug 2022-01-31T23:08:28.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.838646+0000) 2022-01-31T23:08:28.890 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[41853]: debug 2022-01-31T23:08:28.879+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.880480+0000) 2022-01-31T23:08:28.891 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[46715]: debug 2022-01-31T23:08:28.878+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.879504+0000) 2022-01-31T23:08:28.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[51620]: debug 2022-01-31T23:08:28.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.633433+0000) 2022-01-31T23:08:28.891 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[51620]: debug 2022-01-31T23:08:28.879+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.880598+0000) 2022-01-31T23:08:29.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:28 smithi167 conmon[49112]: debug 2022-01-31T23:08:28.879+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.881157+0000) 2022-01-31T23:08:29.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:28 smithi167 conmon[54076]: debug 2022-01-31T23:08:28.879+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.880664+0000) 2022-01-31T23:08:29.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:28 smithi167 conmon[60316]: debug 2022-01-31T23:08:28.879+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:28.880958+0000) 2022-01-31T23:08:29.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:29 smithi167 conmon[60316]: debug 2022-01-31T23:08:29.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:29.017254+0000) 2022-01-31T23:08:29.188 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:29 smithi171 conmon[46715]: debug 2022-01-31T23:08:29.002+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:29.003582+0000) 2022-01-31T23:08:29.188 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:28 smithi171 conmon[35325]: debug 2022-01-31T23:08:28.907+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265833 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:29.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:29 smithi171 conmon[41853]: debug 2022-01-31T23:08:29.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:29.206337+0000) 2022-01-31T23:08:29.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:29 smithi167 conmon[54076]: debug 2022-01-31T23:08:29.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:29.258703+0000) 2022-01-31T23:08:29.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:29 smithi167 conmon[49112]: debug 2022-01-31T23:08:29.837+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:29.838804+0000) 2022-01-31T23:08:29.985 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:29 smithi171 conmon[51620]: debug 2022-01-31T23:08:29.632+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:29.633663+0000) 2022-01-31T23:08:30.240 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:30 smithi167 conmon[60316]: debug 2022-01-31T23:08:30.016+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:30.017408+0000) 2022-01-31T23:08:30.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:30 smithi171 conmon[41853]: debug 2022-01-31T23:08:30.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:30.206520+0000) 2022-01-31T23:08:30.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:30 smithi171 conmon[46715]: debug 2022-01-31T23:08:30.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:30.003768+0000) 2022-01-31T23:08:30.599 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:30 smithi167 conmon[54076]: debug 2022-01-31T23:08:30.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:30.258811+0000) 2022-01-31T23:08:30.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:30 smithi167 conmon[49112]: debug 2022-01-31T23:08:30.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:30.838971+0000) 2022-01-31T23:08:30.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:30 smithi171 conmon[51620]: debug 2022-01-31T23:08:30.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:30.633840+0000) 2022-01-31T23:08:31.241 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:31 smithi167 conmon[60316]: debug 2022-01-31T23:08:31.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:31.017553+0000) 2022-01-31T23:08:31.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:31 smithi171 conmon[41853]: debug 2022-01-31T23:08:31.205+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:31.206726+0000) 2022-01-31T23:08:31.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:31 smithi171 conmon[46715]: debug 2022-01-31T23:08:31.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:31.003963+0000) 2022-01-31T23:08:31.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:31 smithi167 conmon[54076]: debug 2022-01-31T23:08:31.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:31.258982+0000) 2022-01-31T23:08:31.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:31 smithi167 conmon[49112]: debug 2022-01-31T23:08:31.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:31.839124+0000) 2022-01-31T23:08:31.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:31 smithi171 conmon[51620]: debug 2022-01-31T23:08:31.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:31.634051+0000) 2022-01-31T23:08:32.241 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:32 smithi167 conmon[60316]: debug 2022-01-31T23:08:32.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:32.017742+0000) 2022-01-31T23:08:32.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:32 smithi171 conmon[41853]: debug 2022-01-31T23:08:32.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:32.206905+0000) 2022-01-31T23:08:32.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:32 smithi171 conmon[46715]: debug 2022-01-31T23:08:32.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:32.004170+0000) 2022-01-31T23:08:32.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:32 smithi167 conmon[54076]: debug 2022-01-31T23:08:32.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:32.259140+0000) 2022-01-31T23:08:32.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:32 smithi167 conmon[49112]: debug 2022-01-31T23:08:32.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:32.839299+0000) 2022-01-31T23:08:32.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:32 smithi171 conmon[51620]: debug 2022-01-31T23:08:32.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:32.634267+0000) 2022-01-31T23:08:33.241 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:33 smithi167 conmon[60316]: debug 2022-01-31T23:08:33.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.017895+0000) 2022-01-31T23:08:33.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[41853]: debug 2022-01-31T23:08:33.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.207003+0000) 2022-01-31T23:08:33.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[46715]: debug 2022-01-31T23:08:33.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.004362+0000) 2022-01-31T23:08:33.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:33 smithi167 conmon[54076]: debug 2022-01-31T23:08:33.258+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.259293+0000) 2022-01-31T23:08:33.892 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[51620]: debug 2022-01-31T23:08:33.633+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.634469+0000) 2022-01-31T23:08:33.893 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:33 smithi167 conmon[49112]: debug 2022-01-31T23:08:33.838+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.839461+0000) 2022-01-31T23:08:34.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:33 smithi167 conmon[49112]: debug 2022-01-31T23:08:33.917+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.917743+0000) 2022-01-31T23:08:34.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:33 smithi167 conmon[54076]: debug 2022-01-31T23:08:33.910+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.910943+0000) 2022-01-31T23:08:34.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:33 smithi167 conmon[60316]: debug 2022-01-31T23:08:33.917+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.917820+0000) 2022-01-31T23:08:34.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:34 smithi167 conmon[60316]: debug 2022-01-31T23:08:34.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:34.018072+0000) 2022-01-31T23:08:34.189 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[35325]: debug 2022-01-31T23:08:33.953+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 265944 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:34.189 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[41853]: debug 2022-01-31T23:08:33.911+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.911912+0000) 2022-01-31T23:08:34.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[51620]: debug 2022-01-31T23:08:33.910+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.911034+0000) 2022-01-31T23:08:34.190 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:33 smithi171 conmon[46715]: debug 2022-01-31T23:08:33.909+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:33.910347+0000) 2022-01-31T23:08:34.191 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:34 smithi171 conmon[46715]: debug 2022-01-31T23:08:34.003+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:34.004543+0000) 2022-01-31T23:08:34.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:34 smithi171 conmon[41853]: debug 2022-01-31T23:08:34.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:34.207100+0000) 2022-01-31T23:08:34.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:34 smithi167 conmon[54076]: debug 2022-01-31T23:08:34.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:34.259537+0000) 2022-01-31T23:08:34.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:34 smithi167 conmon[49112]: debug 2022-01-31T23:08:34.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:34.839577+0000) 2022-01-31T23:08:34.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:34 smithi171 conmon[51620]: debug 2022-01-31T23:08:34.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:34.634686+0000) 2022-01-31T23:08:35.241 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:35 smithi167 conmon[60316]: debug 2022-01-31T23:08:35.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:35.018262+0000) 2022-01-31T23:08:35.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:35 smithi171 conmon[46715]: debug 2022-01-31T23:08:35.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:35.004724+0000) 2022-01-31T23:08:35.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:35 smithi171 conmon[41853]: debug 2022-01-31T23:08:35.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:35.207276+0000) 2022-01-31T23:08:35.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:35 smithi167 conmon[54076]: debug 2022-01-31T23:08:35.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:35.259679+0000) 2022-01-31T23:08:35.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:35 smithi167 conmon[49112]: debug 2022-01-31T23:08:35.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:35.839720+0000) 2022-01-31T23:08:35.986 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:35 smithi171 conmon[51620]: debug 2022-01-31T23:08:35.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:35.634877+0000) 2022-01-31T23:08:36.241 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:36 smithi167 conmon[60316]: debug 2022-01-31T23:08:36.017+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:36.018468+0000) 2022-01-31T23:08:36.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:36 smithi171 conmon[41853]: debug 2022-01-31T23:08:36.206+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:36.207473+0000) 2022-01-31T23:08:36.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:36 smithi171 conmon[46715]: debug 2022-01-31T23:08:36.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:36.004930+0000) 2022-01-31T23:08:36.600 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:36 smithi167 conmon[54076]: debug 2022-01-31T23:08:36.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:36.259854+0000) 2022-01-31T23:08:36.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:36 smithi167 conmon[49112]: debug 2022-01-31T23:08:36.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:36.839861+0000) 2022-01-31T23:08:36.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:36 smithi171 conmon[51620]: debug 2022-01-31T23:08:36.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:36.635087+0000) 2022-01-31T23:08:37.242 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:37 smithi167 conmon[60316]: debug 2022-01-31T23:08:37.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:37.018626+0000) 2022-01-31T23:08:37.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:37 smithi171 conmon[46715]: debug 2022-01-31T23:08:37.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:37.005083+0000) 2022-01-31T23:08:37.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:37 smithi171 conmon[41853]: debug 2022-01-31T23:08:37.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:37.207667+0000) 2022-01-31T23:08:37.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:37 smithi167 conmon[54076]: debug 2022-01-31T23:08:37.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:37.260019+0000) 2022-01-31T23:08:37.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:37 smithi167 conmon[49112]: debug 2022-01-31T23:08:37.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:37.840070+0000) 2022-01-31T23:08:37.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:37 smithi171 conmon[51620]: debug 2022-01-31T23:08:37.634+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:37.635337+0000) 2022-01-31T23:08:38.242 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:38 smithi167 conmon[60316]: debug 2022-01-31T23:08:38.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.018788+0000) 2022-01-31T23:08:38.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[41853]: debug 2022-01-31T23:08:38.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.207857+0000) 2022-01-31T23:08:38.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[46715]: debug 2022-01-31T23:08:38.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.005290+0000) 2022-01-31T23:08:38.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:38 smithi167 conmon[54076]: debug 2022-01-31T23:08:38.259+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.260250+0000) 2022-01-31T23:08:38.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:38 smithi167 conmon[49112]: debug 2022-01-31T23:08:38.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.840173+0000) 2022-01-31T23:08:38.937 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[51620]: debug 2022-01-31T23:08:38.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.635543+0000) 2022-01-31T23:08:39.190 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[41853]: debug 2022-01-31T23:08:38.956+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.956806+0000) 2022-01-31T23:08:39.190 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[51620]: debug 2022-01-31T23:08:38.956+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.957432+0000) 2022-01-31T23:08:39.191 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[35325]: debug 2022-01-31T23:08:38.969+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:08:39.191 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[35325]: debug 2022-01-31T23:08:38.983+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266056 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:39.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:38 smithi171 conmon[46715]: debug 2022-01-31T23:08:38.955+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.956047+0000) 2022-01-31T23:08:39.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:39 smithi171 conmon[46715]: debug 2022-01-31T23:08:39.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:39.005475+0000) 2022-01-31T23:08:39.242 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:38 smithi167 conmon[54076]: debug 2022-01-31T23:08:38.956+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.957106+0000) 2022-01-31T23:08:39.243 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:38 smithi167 conmon[49112]: debug 2022-01-31T23:08:38.957+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.957920+0000) 2022-01-31T23:08:39.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:38 smithi167 conmon[60316]: debug 2022-01-31T23:08:38.957+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:38.957737+0000) 2022-01-31T23:08:39.244 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:39 smithi167 conmon[60316]: debug 2022-01-31T23:08:39.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:39.019022+0000) 2022-01-31T23:08:39.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:39 smithi171 conmon[41853]: debug 2022-01-31T23:08:39.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:39.208027+0000) 2022-01-31T23:08:39.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:39 smithi167 conmon[54076]: debug 2022-01-31T23:08:39.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:39.260464+0000) 2022-01-31T23:08:39.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:39 smithi167 conmon[49112]: debug 2022-01-31T23:08:39.839+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:39.840347+0000) 2022-01-31T23:08:39.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:39 smithi171 conmon[51620]: debug 2022-01-31T23:08:39.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:39.635711+0000) 2022-01-31T23:08:40.242 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:40 smithi167 conmon[60316]: debug 2022-01-31T23:08:40.018+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:40.019184+0000) 2022-01-31T23:08:40.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:40 smithi171 conmon[41853]: debug 2022-01-31T23:08:40.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:40.208137+0000) 2022-01-31T23:08:40.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:40 smithi171 conmon[46715]: debug 2022-01-31T23:08:40.004+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:40.005612+0000) 2022-01-31T23:08:40.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:40 smithi167 conmon[54076]: debug 2022-01-31T23:08:40.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:40.260597+0000) 2022-01-31T23:08:40.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:40 smithi167 conmon[49112]: debug 2022-01-31T23:08:40.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:40.840534+0000) 2022-01-31T23:08:40.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:40 smithi171 conmon[51620]: debug 2022-01-31T23:08:40.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:40.635895+0000) 2022-01-31T23:08:41.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:41 smithi167 conmon[60316]: debug 2022-01-31T23:08:41.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:41.019374+0000) 2022-01-31T23:08:41.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:41 smithi171 conmon[46715]: debug 2022-01-31T23:08:41.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:41.005785+0000) 2022-01-31T23:08:41.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:41 smithi171 conmon[41853]: debug 2022-01-31T23:08:41.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:41.208322+0000) 2022-01-31T23:08:41.601 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:41 smithi167 conmon[54076]: debug 2022-01-31T23:08:41.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:41.260795+0000) 2022-01-31T23:08:41.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:41 smithi167 conmon[49112]: debug 2022-01-31T23:08:41.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:41.840741+0000) 2022-01-31T23:08:41.987 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:41 smithi171 conmon[51620]: debug 2022-01-31T23:08:41.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:41.636084+0000) 2022-01-31T23:08:42.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:42 smithi167 conmon[60316]: debug 2022-01-31T23:08:42.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:42.019581+0000) 2022-01-31T23:08:42.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:42 smithi171 conmon[41853]: debug 2022-01-31T23:08:42.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:42.208507+0000) 2022-01-31T23:08:42.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:42 smithi171 conmon[46715]: debug 2022-01-31T23:08:42.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:42.005975+0000) 2022-01-31T23:08:42.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:42 smithi167 conmon[54076]: debug 2022-01-31T23:08:42.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:42.260973+0000) 2022-01-31T23:08:42.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:42 smithi167 conmon[49112]: debug 2022-01-31T23:08:42.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:42.840929+0000) 2022-01-31T23:08:42.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:42 smithi171 conmon[51620]: debug 2022-01-31T23:08:42.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:42.636261+0000) 2022-01-31T23:08:43.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:43 smithi167 conmon[60316]: debug 2022-01-31T23:08:43.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.019768+0000) 2022-01-31T23:08:43.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:43 smithi171 conmon[41853]: debug 2022-01-31T23:08:43.207+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.208664+0000) 2022-01-31T23:08:43.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:43 smithi171 conmon[46715]: debug 2022-01-31T23:08:43.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.006156+0000) 2022-01-31T23:08:43.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:43 smithi167 conmon[54076]: debug 2022-01-31T23:08:43.260+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.261131+0000) 2022-01-31T23:08:43.902 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:43 smithi167 conmon[49112]: debug 2022-01-31T23:08:43.840+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.841118+0000) 2022-01-31T23:08:43.968 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:43 smithi171 conmon[51620]: debug 2022-01-31T23:08:43.635+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.636386+0000) 2022-01-31T23:08:44.243 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:43 smithi167 conmon[49112]: debug 2022-01-31T23:08:43.987+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.988157+0000) 2022-01-31T23:08:44.244 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:43 smithi167 conmon[54076]: debug 2022-01-31T23:08:43.986+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.987289+0000) 2022-01-31T23:08:44.244 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:43 smithi167 conmon[60316]: debug 2022-01-31T23:08:43.986+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.986839+0000) 2022-01-31T23:08:44.245 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:44 smithi167 conmon[60316]: debug 2022-01-31T23:08:44.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:44.019905+0000) 2022-01-31T23:08:44.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:44 smithi171 conmon[35325]: debug 2022-01-31T23:08:44.014+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266163 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:44.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:43 smithi171 conmon[46715]: debug 2022-01-31T23:08:43.985+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.986507+0000) 2022-01-31T23:08:44.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:44 smithi171 conmon[46715]: debug 2022-01-31T23:08:44.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:44.006318+0000) 2022-01-31T23:08:44.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:43 smithi171 conmon[51620]: debug 2022-01-31T23:08:43.986+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.987540+0000) 2022-01-31T23:08:44.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:43 smithi171 conmon[41853]: debug 2022-01-31T23:08:43.985+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:43.986585+0000) 2022-01-31T23:08:44.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:44 smithi171 conmon[41853]: debug 2022-01-31T23:08:44.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:44.208840+0000) 2022-01-31T23:08:44.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:44 smithi167 conmon[54076]: debug 2022-01-31T23:08:44.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:44.261350+0000) 2022-01-31T23:08:44.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:44 smithi167 conmon[49112]: debug 2022-01-31T23:08:44.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:44.841291+0000) 2022-01-31T23:08:44.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:44 smithi171 conmon[51620]: debug 2022-01-31T23:08:44.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:44.636611+0000) 2022-01-31T23:08:45.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:45 smithi167 conmon[60316]: debug 2022-01-31T23:08:45.019+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:45.020073+0000) 2022-01-31T23:08:45.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:45 smithi171 conmon[41853]: debug 2022-01-31T23:08:45.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:45.208960+0000) 2022-01-31T23:08:45.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:45 smithi171 conmon[46715]: debug 2022-01-31T23:08:45.005+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:45.006485+0000) 2022-01-31T23:08:45.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:45 smithi167 conmon[54076]: debug 2022-01-31T23:08:45.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:45.261513+0000) 2022-01-31T23:08:45.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:45 smithi167 conmon[49112]: debug 2022-01-31T23:08:45.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:45.841459+0000) 2022-01-31T23:08:45.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:45 smithi171 conmon[51620]: debug 2022-01-31T23:08:45.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:45.636807+0000) 2022-01-31T23:08:46.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:46 smithi167 conmon[60316]: debug 2022-01-31T23:08:46.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:46.020268+0000) 2022-01-31T23:08:46.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:46 smithi171 conmon[41853]: debug 2022-01-31T23:08:46.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:46.209157+0000) 2022-01-31T23:08:46.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:46 smithi171 conmon[46715]: debug 2022-01-31T23:08:46.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:46.006666+0000) 2022-01-31T23:08:46.602 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:46 smithi167 conmon[54076]: debug 2022-01-31T23:08:46.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:46.261694+0000) 2022-01-31T23:08:46.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:46 smithi167 conmon[49112]: debug 2022-01-31T23:08:46.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:46.841690+0000) 2022-01-31T23:08:46.988 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:46 smithi171 conmon[51620]: debug 2022-01-31T23:08:46.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:46.637046+0000) 2022-01-31T23:08:47.243 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:47 smithi167 conmon[60316]: debug 2022-01-31T23:08:47.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:47.020454+0000) 2022-01-31T23:08:47.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:47 smithi171 conmon[41853]: debug 2022-01-31T23:08:47.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:47.209358+0000) 2022-01-31T23:08:47.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:47 smithi171 conmon[46715]: debug 2022-01-31T23:08:47.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:47.006825+0000) 2022-01-31T23:08:47.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:47 smithi167 conmon[54076]: debug 2022-01-31T23:08:47.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:47.261868+0000) 2022-01-31T23:08:47.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:47 smithi167 conmon[49112]: debug 2022-01-31T23:08:47.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:47.841881+0000) 2022-01-31T23:08:47.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:47 smithi171 conmon[51620]: debug 2022-01-31T23:08:47.636+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:47.637263+0000) 2022-01-31T23:08:48.244 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:48 smithi167 conmon[60316]: debug 2022-01-31T23:08:48.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:48.020652+0000) 2022-01-31T23:08:48.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:48 smithi171 conmon[41853]: debug 2022-01-31T23:08:48.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:48.209564+0000) 2022-01-31T23:08:48.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:48 smithi171 conmon[46715]: debug 2022-01-31T23:08:48.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:48.007008+0000) 2022-01-31T23:08:48.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:48 smithi167 conmon[54076]: debug 2022-01-31T23:08:48.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:48.262074+0000) 2022-01-31T23:08:48.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:48 smithi167 conmon[49112]: debug 2022-01-31T23:08:48.841+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:48.842064+0000) 2022-01-31T23:08:48.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:48 smithi171 conmon[51620]: debug 2022-01-31T23:08:48.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:48.637474+0000) 2022-01-31T23:08:49.244 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:49 smithi167 conmon[49112]: debug 2022-01-31T23:08:49.018+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.019150+0000) 2022-01-31T23:08:49.245 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:49 smithi167 conmon[54076]: debug 2022-01-31T23:08:49.017+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.018253+0000) 2022-01-31T23:08:49.246 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:49 smithi167 conmon[60316]: debug 2022-01-31T23:08:49.018+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.018921+0000) 2022-01-31T23:08:49.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:49 smithi167 conmon[60316]: debug 2022-01-31T23:08:49.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.020783+0000) 2022-01-31T23:08:49.339 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[51620]: debug 2022-01-31T23:08:49.017+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.018632+0000) 2022-01-31T23:08:49.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[35325]: debug 2022-01-31T23:08:49.045+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266277 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:49.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[41853]: debug 2022-01-31T23:08:49.016+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.017714+0000) 2022-01-31T23:08:49.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[41853]: debug 2022-01-31T23:08:49.208+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.209734+0000) 2022-01-31T23:08:49.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[46715]: debug 2022-01-31T23:08:49.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.007188+0000) 2022-01-31T23:08:49.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[46715]: debug 2022-01-31T23:08:49.016+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.017472+0000) 2022-01-31T23:08:49.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:49 smithi167 conmon[54076]: debug 2022-01-31T23:08:49.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.262264+0000) 2022-01-31T23:08:49.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:49 smithi167 conmon[49112]: debug 2022-01-31T23:08:49.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.842231+0000) 2022-01-31T23:08:49.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:49 smithi171 conmon[51620]: debug 2022-01-31T23:08:49.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:49.637691+0000) 2022-01-31T23:08:50.244 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:50 smithi167 conmon[60316]: debug 2022-01-31T23:08:50.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:50.020934+0000) 2022-01-31T23:08:50.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:50 smithi171 conmon[46715]: debug 2022-01-31T23:08:50.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:50.007348+0000) 2022-01-31T23:08:50.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:50 smithi171 conmon[41853]: debug 2022-01-31T23:08:50.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:50.209867+0000) 2022-01-31T23:08:50.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:50 smithi167 conmon[54076]: debug 2022-01-31T23:08:50.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:50.262456+0000) 2022-01-31T23:08:50.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:50 smithi167 conmon[49112]: debug 2022-01-31T23:08:50.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:50.842391+0000) 2022-01-31T23:08:50.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:50 smithi171 conmon[51620]: debug 2022-01-31T23:08:50.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:50.637888+0000) 2022-01-31T23:08:51.244 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:51 smithi167 conmon[60316]: debug 2022-01-31T23:08:51.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:51.021142+0000) 2022-01-31T23:08:51.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:51 smithi171 conmon[41853]: debug 2022-01-31T23:08:51.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:51.210051+0000) 2022-01-31T23:08:51.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:51 smithi171 conmon[46715]: debug 2022-01-31T23:08:51.006+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:51.007522+0000) 2022-01-31T23:08:51.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:51 smithi167 conmon[54076]: debug 2022-01-31T23:08:51.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:51.262698+0000) 2022-01-31T23:08:51.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:51 smithi167 conmon[49112]: debug 2022-01-31T23:08:51.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:51.842579+0000) 2022-01-31T23:08:51.989 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:51 smithi171 conmon[51620]: debug 2022-01-31T23:08:51.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:51.638108+0000) 2022-01-31T23:08:52.244 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:52 smithi167 conmon[60316]: debug 2022-01-31T23:08:52.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:52.021346+0000) 2022-01-31T23:08:52.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:52 smithi171 conmon[46715]: debug 2022-01-31T23:08:52.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:52.007703+0000) 2022-01-31T23:08:52.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:52 smithi171 conmon[41853]: debug 2022-01-31T23:08:52.209+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:52.210188+0000) 2022-01-31T23:08:52.603 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:52 smithi167 conmon[54076]: debug 2022-01-31T23:08:52.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:52.262905+0000) 2022-01-31T23:08:52.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:52 smithi167 conmon[49112]: debug 2022-01-31T23:08:52.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:52.842765+0000) 2022-01-31T23:08:52.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:52 smithi171 conmon[51620]: debug 2022-01-31T23:08:52.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:52.638292+0000) 2022-01-31T23:08:53.245 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:53 smithi167 conmon[60316]: debug 2022-01-31T23:08:53.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:53.021575+0000) 2022-01-31T23:08:53.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:53 smithi171 conmon[41853]: debug 2022-01-31T23:08:53.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:53.210430+0000) 2022-01-31T23:08:53.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:53 smithi171 conmon[46715]: debug 2022-01-31T23:08:53.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:53.007972+0000) 2022-01-31T23:08:53.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:53 smithi167 conmon[54076]: debug 2022-01-31T23:08:53.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:53.263162+0000) 2022-01-31T23:08:53.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:53 smithi167 conmon[49112]: debug 2022-01-31T23:08:53.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:53.843036+0000) 2022-01-31T23:08:53.952 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:53 smithi171 conmon[51620]: debug 2022-01-31T23:08:53.637+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:53.638605+0000) 2022-01-31T23:08:54.245 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:54 smithi167 conmon[49112]: debug 2022-01-31T23:08:54.049+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.049955+0000) 2022-01-31T23:08:54.246 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:54 smithi167 conmon[54076]: debug 2022-01-31T23:08:54.048+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.049313+0000) 2022-01-31T23:08:54.246 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:54 smithi167 conmon[60316]: debug 2022-01-31T23:08:54.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.021715+0000) 2022-01-31T23:08:54.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:54 smithi167 conmon[60316]: debug 2022-01-31T23:08:54.049+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.049787+0000) 2022-01-31T23:08:54.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:53 smithi171 conmon[35325]: debug 2022-01-31T23:08:53.970+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:08:54.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[35325]: debug 2022-01-31T23:08:54.075+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266387 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[46715]: debug 2022-01-31T23:08:54.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.008202+0000) 2022-01-31T23:08:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[46715]: debug 2022-01-31T23:08:54.047+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.047850+0000) 2022-01-31T23:08:54.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[51620]: debug 2022-01-31T23:08:54.047+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.048060+0000) 2022-01-31T23:08:54.341 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[41853]: debug 2022-01-31T23:08:54.048+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.048746+0000) 2022-01-31T23:08:54.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[41853]: debug 2022-01-31T23:08:54.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.210593+0000) 2022-01-31T23:08:54.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:54 smithi167 conmon[54076]: debug 2022-01-31T23:08:54.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.263299+0000) 2022-01-31T23:08:54.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:54 smithi167 conmon[49112]: debug 2022-01-31T23:08:54.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.843215+0000) 2022-01-31T23:08:54.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:54 smithi171 conmon[51620]: debug 2022-01-31T23:08:54.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:54.638779+0000) 2022-01-31T23:08:55.245 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:55 smithi167 conmon[60316]: debug 2022-01-31T23:08:55.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:55.021835+0000) 2022-01-31T23:08:55.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:55 smithi171 conmon[46715]: debug 2022-01-31T23:08:55.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:55.008340+0000) 2022-01-31T23:08:55.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:55 smithi171 conmon[41853]: debug 2022-01-31T23:08:55.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:55.210737+0000) 2022-01-31T23:08:55.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:55 smithi167 conmon[54076]: debug 2022-01-31T23:08:55.261+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:55.263403+0000) 2022-01-31T23:08:55.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:55 smithi167 conmon[49112]: debug 2022-01-31T23:08:55.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:55.843429+0000) 2022-01-31T23:08:55.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:55 smithi171 conmon[51620]: debug 2022-01-31T23:08:55.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:55.638917+0000) 2022-01-31T23:08:56.245 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:56 smithi167 conmon[60316]: debug 2022-01-31T23:08:56.020+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:56.021993+0000) 2022-01-31T23:08:56.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:56 smithi171 conmon[41853]: debug 2022-01-31T23:08:56.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:56.210971+0000) 2022-01-31T23:08:56.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:56 smithi171 conmon[46715]: debug 2022-01-31T23:08:56.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:56.008464+0000) 2022-01-31T23:08:56.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:56 smithi167 conmon[54076]: debug 2022-01-31T23:08:56.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:56.263571+0000) 2022-01-31T23:08:56.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:56 smithi167 conmon[49112]: debug 2022-01-31T23:08:56.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:56.843669+0000) 2022-01-31T23:08:56.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:56 smithi171 conmon[51620]: debug 2022-01-31T23:08:56.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:56.639085+0000) 2022-01-31T23:08:57.245 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:57 smithi167 conmon[60316]: debug 2022-01-31T23:08:57.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:57.022157+0000) 2022-01-31T23:08:57.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:57 smithi171 conmon[46715]: debug 2022-01-31T23:08:57.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:57.008646+0000) 2022-01-31T23:08:57.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:57 smithi171 conmon[41853]: debug 2022-01-31T23:08:57.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:57.211123+0000) 2022-01-31T23:08:57.604 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:57 smithi167 conmon[54076]: debug 2022-01-31T23:08:57.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:57.263759+0000) 2022-01-31T23:08:57.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:57 smithi167 conmon[49112]: debug 2022-01-31T23:08:57.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:57.843860+0000) 2022-01-31T23:08:57.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:57 smithi171 conmon[51620]: debug 2022-01-31T23:08:57.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:57.639298+0000) 2022-01-31T23:08:58.245 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:58 smithi167 conmon[60316]: debug 2022-01-31T23:08:58.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:58.022311+0000) 2022-01-31T23:08:58.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:58 smithi171 conmon[41853]: debug 2022-01-31T23:08:58.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:58.211306+0000) 2022-01-31T23:08:58.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:58 smithi171 conmon[46715]: debug 2022-01-31T23:08:58.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:58.008815+0000) 2022-01-31T23:08:58.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:58 smithi167 conmon[54076]: debug 2022-01-31T23:08:58.262+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:58.263908+0000) 2022-01-31T23:08:58.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:58 smithi167 conmon[49112]: debug 2022-01-31T23:08:58.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:58.844047+0000) 2022-01-31T23:08:58.990 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:58 smithi171 conmon[51620]: debug 2022-01-31T23:08:58.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:58.639536+0000) 2022-01-31T23:08:59.246 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:59 smithi167 conmon[49112]: debug 2022-01-31T23:08:59.078+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.080156+0000) 2022-01-31T23:08:59.356 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:59 smithi167 conmon[54076]: debug 2022-01-31T23:08:59.078+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.079589+0000) 2022-01-31T23:08:59.356 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:59 smithi167 conmon[60316]: debug 2022-01-31T23:08:59.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.022488+0000) 2022-01-31T23:08:59.357 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:08:59 smithi167 conmon[60316]: debug 2022-01-31T23:08:59.077+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.079271+0000) 2022-01-31T23:08:59.357 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[35325]: debug 2022-01-31T23:08:59.106+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266500 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:08:59.358 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[46715]: debug 2022-01-31T23:08:59.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.008979+0000) 2022-01-31T23:08:59.359 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[46715]: debug 2022-01-31T23:08:59.077+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.078662+0000) 2022-01-31T23:08:59.359 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[51620]: debug 2022-01-31T23:08:59.078+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.078967+0000) 2022-01-31T23:08:59.360 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[41853]: debug 2022-01-31T23:08:59.078+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.079280+0000) 2022-01-31T23:08:59.360 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[41853]: debug 2022-01-31T23:08:59.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.211463+0000) 2022-01-31T23:08:59.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:08:59 smithi167 conmon[54076]: debug 2022-01-31T23:08:59.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.264102+0000) 2022-01-31T23:08:59.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:08:59 smithi167 conmon[49112]: debug 2022-01-31T23:08:59.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.844269+0000) 2022-01-31T23:08:59.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:08:59 smithi171 conmon[51620]: debug 2022-01-31T23:08:59.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:08:59.639728+0000) 2022-01-31T23:09:00.246 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:00 smithi167 conmon[60316]: debug 2022-01-31T23:09:00.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:00.022666+0000) 2022-01-31T23:09:00.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:00 smithi171 conmon[41853]: debug 2022-01-31T23:09:00.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:00.211602+0000) 2022-01-31T23:09:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:00 smithi171 conmon[46715]: debug 2022-01-31T23:09:00.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:00.009137+0000) 2022-01-31T23:09:00.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:00 smithi167 conmon[54076]: debug 2022-01-31T23:09:00.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:00.264274+0000) 2022-01-31T23:09:00.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:00 smithi167 conmon[49112]: debug 2022-01-31T23:09:00.842+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:00.844399+0000) 2022-01-31T23:09:00.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:00 smithi171 conmon[51620]: debug 2022-01-31T23:09:00.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:00.639937+0000) 2022-01-31T23:09:01.246 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:01 smithi167 conmon[60316]: debug 2022-01-31T23:09:01.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:01.022792+0000) 2022-01-31T23:09:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:01 smithi171 conmon[46715]: debug 2022-01-31T23:09:01.007+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:01.009293+0000) 2022-01-31T23:09:01.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:01 smithi171 conmon[41853]: debug 2022-01-31T23:09:01.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:01.211828+0000) 2022-01-31T23:09:01.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:01 smithi167 conmon[54076]: debug 2022-01-31T23:09:01.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:01.264464+0000) 2022-01-31T23:09:01.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:01 smithi167 conmon[49112]: debug 2022-01-31T23:09:01.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:01.844534+0000) 2022-01-31T23:09:01.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:01 smithi171 conmon[51620]: debug 2022-01-31T23:09:01.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:01.640174+0000) 2022-01-31T23:09:02.246 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:02 smithi167 conmon[60316]: debug 2022-01-31T23:09:02.021+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:02.022946+0000) 2022-01-31T23:09:02.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:02 smithi171 conmon[46715]: debug 2022-01-31T23:09:02.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:02.009478+0000) 2022-01-31T23:09:02.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:02 smithi171 conmon[41853]: debug 2022-01-31T23:09:02.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:02.212039+0000) 2022-01-31T23:09:02.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:02 smithi167 conmon[54076]: debug 2022-01-31T23:09:02.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:02.264695+0000) 2022-01-31T23:09:02.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:02 smithi167 conmon[49112]: debug 2022-01-31T23:09:02.843+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:02.844692+0000) 2022-01-31T23:09:02.919 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:02 smithi171 conmon[51620]: debug 2022-01-31T23:09:02.638+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:02.640404+0000) 2022-01-31T23:09:03.246 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:03 smithi167 conmon[60316]: debug 2022-01-31T23:09:03.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:03.023136+0000) 2022-01-31T23:09:03.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:03 smithi171 conmon[46715]: debug 2022-01-31T23:09:03.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:03.009682+0000) 2022-01-31T23:09:03.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:03 smithi171 conmon[41853]: debug 2022-01-31T23:09:03.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:03.212206+0000) 2022-01-31T23:09:03.605 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:03 smithi167 conmon[54076]: debug 2022-01-31T23:09:03.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:03.264850+0000) 2022-01-31T23:09:03.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:03 smithi171 conmon[51620]: debug 2022-01-31T23:09:03.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:03.640613+0000) 2022-01-31T23:09:03.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:03 smithi167 conmon[49112]: debug 2022-01-31T23:09:03.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:03.844919+0000) 2022-01-31T23:09:04.247 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:04 smithi167 conmon[49112]: debug 2022-01-31T23:09:04.109+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.110965+0000) 2022-01-31T23:09:04.247 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:04 smithi167 conmon[54076]: debug 2022-01-31T23:09:04.109+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.110840+0000) 2022-01-31T23:09:04.248 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:04 smithi167 conmon[60316]: debug 2022-01-31T23:09:04.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.023337+0000) 2022-01-31T23:09:04.248 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:04 smithi167 conmon[60316]: debug 2022-01-31T23:09:04.108+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.110249+0000) 2022-01-31T23:09:04.303 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[35325]: debug 2022-01-31T23:09:04.137+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266610 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:04.304 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[51620]: debug 2022-01-31T23:09:04.108+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.109888+0000) 2022-01-31T23:09:04.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[41853]: debug 2022-01-31T23:09:04.108+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.110166+0000) 2022-01-31T23:09:04.305 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[41853]: debug 2022-01-31T23:09:04.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.212371+0000) 2022-01-31T23:09:04.305 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[46715]: debug 2022-01-31T23:09:04.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.009861+0000) 2022-01-31T23:09:04.306 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[46715]: debug 2022-01-31T23:09:04.107+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.109321+0000) 2022-01-31T23:09:04.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:04 smithi167 conmon[54076]: debug 2022-01-31T23:09:04.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.265033+0000) 2022-01-31T23:09:04.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:04 smithi167 conmon[49112]: debug 2022-01-31T23:09:04.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.845121+0000) 2022-01-31T23:09:04.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:04 smithi171 conmon[51620]: debug 2022-01-31T23:09:04.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:04.640843+0000) 2022-01-31T23:09:05.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:05 smithi167 conmon[60316]: debug 2022-01-31T23:09:05.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:05.023555+0000) 2022-01-31T23:09:05.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:05 smithi171 conmon[46715]: debug 2022-01-31T23:09:05.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:05.010056+0000) 2022-01-31T23:09:05.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:05 smithi171 conmon[41853]: debug 2022-01-31T23:09:05.210+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:05.212535+0000) 2022-01-31T23:09:05.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:05 smithi167 conmon[54076]: debug 2022-01-31T23:09:05.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:05.265188+0000) 2022-01-31T23:09:05.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:05 smithi167 conmon[49112]: debug 2022-01-31T23:09:05.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:05.845286+0000) 2022-01-31T23:09:05.991 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:05 smithi171 conmon[51620]: debug 2022-01-31T23:09:05.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:05.641027+0000) 2022-01-31T23:09:06.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:06 smithi167 conmon[60316]: debug 2022-01-31T23:09:06.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:06.023693+0000) 2022-01-31T23:09:06.287 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:06 smithi171 conmon[46715]: debug 2022-01-31T23:09:06.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:06.010218+0000) 2022-01-31T23:09:06.288 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:06 smithi171 conmon[41853]: debug 2022-01-31T23:09:06.211+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:06.212785+0000) 2022-01-31T23:09:06.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:06 smithi167 conmon[54076]: debug 2022-01-31T23:09:06.263+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:06.265397+0000) 2022-01-31T23:09:06.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:06 smithi167 conmon[49112]: debug 2022-01-31T23:09:06.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:06.845460+0000) 2022-01-31T23:09:06.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:06 smithi171 conmon[51620]: debug 2022-01-31T23:09:06.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:06.641259+0000) 2022-01-31T23:09:07.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:07 smithi167 conmon[60316]: debug 2022-01-31T23:09:07.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:07.023879+0000) 2022-01-31T23:09:07.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:07 smithi171 conmon[41853]: debug 2022-01-31T23:09:07.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:07.213005+0000) 2022-01-31T23:09:07.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:07 smithi171 conmon[46715]: debug 2022-01-31T23:09:07.008+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:07.010425+0000) 2022-01-31T23:09:07.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:07 smithi167 conmon[54076]: debug 2022-01-31T23:09:07.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:07.265567+0000) 2022-01-31T23:09:07.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:07 smithi167 conmon[49112]: debug 2022-01-31T23:09:07.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:07.845609+0000) 2022-01-31T23:09:07.992 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:07 smithi171 conmon[51620]: debug 2022-01-31T23:09:07.639+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:07.641488+0000) 2022-01-31T23:09:08.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:08 smithi167 conmon[60316]: debug 2022-01-31T23:09:08.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:08.024049+0000) 2022-01-31T23:09:08.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:08 smithi171 conmon[46715]: debug 2022-01-31T23:09:08.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:08.010610+0000) 2022-01-31T23:09:08.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:08 smithi171 conmon[41853]: debug 2022-01-31T23:09:08.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:08.213134+0000) 2022-01-31T23:09:08.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:08 smithi167 conmon[54076]: debug 2022-01-31T23:09:08.264+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:08.265716+0000) 2022-01-31T23:09:08.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:08 smithi167 conmon[49112]: debug 2022-01-31T23:09:08.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:08.845762+0000) 2022-01-31T23:09:08.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:08 smithi171 conmon[51620]: debug 2022-01-31T23:09:08.640+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:08.641696+0000) 2022-01-31T23:09:09.248 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:09 smithi167 conmon[49112]: debug 2022-01-31T23:09:09.140+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.141584+0000) 2022-01-31T23:09:09.248 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:09 smithi167 conmon[54076]: debug 2022-01-31T23:09:09.141+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.142601+0000) 2022-01-31T23:09:09.249 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:09 smithi167 conmon[60316]: debug 2022-01-31T23:09:09.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.024211+0000) 2022-01-31T23:09:09.249 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:09 smithi167 conmon[60316]: debug 2022-01-31T23:09:09.139+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.141431+0000) 2022-01-31T23:09:09.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[46715]: debug 2022-01-31T23:09:09.009+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.010824+0000) 2022-01-31T23:09:09.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[46715]: debug 2022-01-31T23:09:09.139+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.141243+0000) 2022-01-31T23:09:09.340 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[51620]: debug 2022-01-31T23:09:09.140+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.142030+0000) 2022-01-31T23:09:09.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:08 smithi171 conmon[35325]: debug 2022-01-31T23:09:08.970+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:09:09.341 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[35325]: debug 2022-01-31T23:09:09.167+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266723 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:09.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[41853]: debug 2022-01-31T23:09:09.139+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.141084+0000) 2022-01-31T23:09:09.342 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[41853]: debug 2022-01-31T23:09:09.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.213265+0000) 2022-01-31T23:09:09.606 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:09 smithi167 conmon[54076]: debug 2022-01-31T23:09:09.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.265905+0000) 2022-01-31T23:09:09.901 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:09 smithi167 conmon[49112]: debug 2022-01-31T23:09:09.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.845929+0000) 2022-01-31T23:09:09.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:09 smithi171 conmon[51620]: debug 2022-01-31T23:09:09.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:09.641958+0000) 2022-01-31T23:09:10.247 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:10 smithi167 conmon[60316]: debug 2022-01-31T23:09:10.022+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:10.024395+0000) 2022-01-31T23:09:10.319 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:10 smithi171 conmon[41853]: debug 2022-01-31T23:09:10.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:10.213404+0000) 2022-01-31T23:09:10.320 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:10 smithi171 conmon[46715]: debug 2022-01-31T23:09:10.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:10.011014+0000) 2022-01-31T23:09:10.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:10 smithi167 conmon[54076]: debug 2022-01-31T23:09:10.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:10.266057+0000) 2022-01-31T23:09:10.859 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:10 smithi167 conmon[49112]: debug 2022-01-31T23:09:10.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:10.846087+0000) 2022-01-31T23:09:10.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:10 smithi171 conmon[51620]: debug 2022-01-31T23:09:10.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:10.642110+0000) 2022-01-31T23:09:11.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:11 smithi167 conmon[60316]: debug 2022-01-31T23:09:11.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:11.024582+0000) 2022-01-31T23:09:11.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:11 smithi171 conmon[41853]: debug 2022-01-31T23:09:11.212+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:11.213614+0000) 2022-01-31T23:09:11.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:11 smithi171 conmon[46715]: debug 2022-01-31T23:09:11.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:11.011207+0000) 2022-01-31T23:09:11.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:11 smithi167 conmon[54076]: debug 2022-01-31T23:09:11.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:11.266242+0000) 2022-01-31T23:09:11.862 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:11 smithi167 conmon[49112]: debug 2022-01-31T23:09:11.845+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:11.846273+0000) 2022-01-31T23:09:11.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:11 smithi171 conmon[51620]: debug 2022-01-31T23:09:11.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:11.642337+0000) 2022-01-31T23:09:12.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:12 smithi167 conmon[60316]: debug 2022-01-31T23:09:12.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:12.024799+0000) 2022-01-31T23:09:12.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:12 smithi171 conmon[41853]: debug 2022-01-31T23:09:12.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:12.213774+0000) 2022-01-31T23:09:12.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:12 smithi171 conmon[46715]: debug 2022-01-31T23:09:12.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:12.011385+0000) 2022-01-31T23:09:12.607 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:12 smithi167 conmon[54076]: debug 2022-01-31T23:09:12.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:12.266399+0000) 2022-01-31T23:09:12.866 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:12 smithi167 conmon[49112]: debug 2022-01-31T23:09:12.844+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:12.846454+0000) 2022-01-31T23:09:12.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:12 smithi171 conmon[51620]: debug 2022-01-31T23:09:12.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:12.642521+0000) 2022-01-31T23:09:13.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:13 smithi167 conmon[60316]: debug 2022-01-31T23:09:13.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:13.024987+0000) 2022-01-31T23:09:13.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:13 smithi171 conmon[41853]: debug 2022-01-31T23:09:13.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:13.213940+0000) 2022-01-31T23:09:13.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:13 smithi171 conmon[46715]: debug 2022-01-31T23:09:13.010+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:13.011565+0000) 2022-01-31T23:09:13.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:13 smithi167 conmon[54076]: debug 2022-01-31T23:09:13.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:13.266572+0000) 2022-01-31T23:09:13.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:13 smithi171 conmon[51620]: debug 2022-01-31T23:09:13.641+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:13.642685+0000) 2022-01-31T23:09:14.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:13 smithi167 conmon[49112]: debug 2022-01-31T23:09:13.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:13.846664+0000) 2022-01-31T23:09:14.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:14 smithi167 conmon[60316]: debug 2022-01-31T23:09:14.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.025152+0000) 2022-01-31T23:09:14.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[35325]: debug 2022-01-31T23:09:14.199+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266834 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:14.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[41853]: debug 2022-01-31T23:09:14.170+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.170681+0000) 2022-01-31T23:09:14.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[41853]: debug 2022-01-31T23:09:14.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.214083+0000) 2022-01-31T23:09:14.341 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[51620]: debug 2022-01-31T23:09:14.171+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.171765+0000) 2022-01-31T23:09:14.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[46715]: debug 2022-01-31T23:09:14.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.011764+0000) 2022-01-31T23:09:14.342 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[46715]: debug 2022-01-31T23:09:14.170+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.171426+0000) 2022-01-31T23:09:14.498 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:14 smithi167 conmon[49112]: debug 2022-01-31T23:09:14.170+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.171573+0000) 2022-01-31T23:09:14.499 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:14 smithi167 conmon[60316]: debug 2022-01-31T23:09:14.171+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.172611+0000) 2022-01-31T23:09:14.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:14 smithi167 conmon[54076]: debug 2022-01-31T23:09:14.170+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.172245+0000) 2022-01-31T23:09:14.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:14 smithi167 conmon[54076]: debug 2022-01-31T23:09:14.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.266706+0000) 2022-01-31T23:09:14.993 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:14 smithi171 conmon[51620]: debug 2022-01-31T23:09:14.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.642910+0000) 2022-01-31T23:09:15.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:14 smithi167 conmon[49112]: debug 2022-01-31T23:09:14.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:14.846851+0000) 2022-01-31T23:09:15.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:15 smithi167 conmon[60316]: debug 2022-01-31T23:09:15.023+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:15.025365+0000) 2022-01-31T23:09:15.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:15 smithi171 conmon[41853]: debug 2022-01-31T23:09:15.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:15.214229+0000) 2022-01-31T23:09:15.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:15 smithi171 conmon[46715]: debug 2022-01-31T23:09:15.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:15.011959+0000) 2022-01-31T23:09:15.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:15 smithi167 conmon[54076]: debug 2022-01-31T23:09:15.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:15.266853+0000) 2022-01-31T23:09:15.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:15 smithi171 conmon[51620]: debug 2022-01-31T23:09:15.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:15.643077+0000) 2022-01-31T23:09:16.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:15 smithi167 conmon[49112]: debug 2022-01-31T23:09:15.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:15.847029+0000) 2022-01-31T23:09:16.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:16 smithi167 conmon[60316]: debug 2022-01-31T23:09:16.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:16.025493+0000) 2022-01-31T23:09:16.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:16 smithi171 conmon[46715]: debug 2022-01-31T23:09:16.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:16.012105+0000) 2022-01-31T23:09:16.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:16 smithi171 conmon[41853]: debug 2022-01-31T23:09:16.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:16.214404+0000) 2022-01-31T23:09:16.498 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:16 smithi167 conmon[54076]: debug 2022-01-31T23:09:16.265+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:16.267008+0000) 2022-01-31T23:09:16.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:16 smithi171 conmon[51620]: debug 2022-01-31T23:09:16.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:16.643268+0000) 2022-01-31T23:09:17.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:16 smithi167 conmon[49112]: debug 2022-01-31T23:09:16.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:16.847168+0000) 2022-01-31T23:09:17.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:17 smithi167 conmon[60316]: debug 2022-01-31T23:09:17.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:17.025651+0000) 2022-01-31T23:09:17.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:17 smithi171 conmon[41853]: debug 2022-01-31T23:09:17.213+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:17.214558+0000) 2022-01-31T23:09:17.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:17 smithi171 conmon[46715]: debug 2022-01-31T23:09:17.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:17.012312+0000) 2022-01-31T23:09:17.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:17 smithi167 conmon[54076]: debug 2022-01-31T23:09:17.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:17.267197+0000) 2022-01-31T23:09:17.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:17 smithi171 conmon[51620]: debug 2022-01-31T23:09:17.642+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:17.643478+0000) 2022-01-31T23:09:18.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:17 smithi167 conmon[49112]: debug 2022-01-31T23:09:17.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:17.847354+0000) 2022-01-31T23:09:18.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:18 smithi167 conmon[60316]: debug 2022-01-31T23:09:18.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:18.025843+0000) 2022-01-31T23:09:18.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:18 smithi171 conmon[41853]: debug 2022-01-31T23:09:18.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:18.214748+0000) 2022-01-31T23:09:18.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:18 smithi171 conmon[46715]: debug 2022-01-31T23:09:18.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:18.012515+0000) 2022-01-31T23:09:18.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:18 smithi167 conmon[54076]: debug 2022-01-31T23:09:18.266+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:18.267369+0000) 2022-01-31T23:09:18.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:18 smithi171 conmon[51620]: debug 2022-01-31T23:09:18.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:18.643692+0000) 2022-01-31T23:09:19.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:18 smithi167 conmon[49112]: debug 2022-01-31T23:09:18.846+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:18.847527+0000) 2022-01-31T23:09:19.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:19 smithi167 conmon[60316]: debug 2022-01-31T23:09:19.024+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.026083+0000) 2022-01-31T23:09:19.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[35325]: debug 2022-01-31T23:09:19.230+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 266946 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:19.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[41853]: debug 2022-01-31T23:09:19.201+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.202080+0000) 2022-01-31T23:09:19.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[41853]: debug 2022-01-31T23:09:19.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.214910+0000) 2022-01-31T23:09:19.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[46715]: debug 2022-01-31T23:09:19.011+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.012722+0000) 2022-01-31T23:09:19.341 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[46715]: debug 2022-01-31T23:09:19.202+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.202836+0000) 2022-01-31T23:09:19.342 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[51620]: debug 2022-01-31T23:09:19.201+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.202443+0000) 2022-01-31T23:09:19.499 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:19 smithi167 conmon[49112]: debug 2022-01-31T23:09:19.201+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.202955+0000) 2022-01-31T23:09:19.500 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:19 smithi167 conmon[60316]: debug 2022-01-31T23:09:19.201+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.203298+0000) 2022-01-31T23:09:19.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:19 smithi167 conmon[54076]: debug 2022-01-31T23:09:19.201+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.203478+0000) 2022-01-31T23:09:19.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:19 smithi167 conmon[54076]: debug 2022-01-31T23:09:19.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.267555+0000) 2022-01-31T23:09:19.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:19 smithi171 conmon[51620]: debug 2022-01-31T23:09:19.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.643936+0000) 2022-01-31T23:09:20.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:19 smithi167 conmon[49112]: debug 2022-01-31T23:09:19.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:19.847679+0000) 2022-01-31T23:09:20.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:20 smithi167 conmon[60316]: debug 2022-01-31T23:09:20.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:20.026238+0000) 2022-01-31T23:09:20.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:20 smithi171 conmon[41853]: debug 2022-01-31T23:09:20.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:20.215063+0000) 2022-01-31T23:09:20.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:20 smithi171 conmon[46715]: debug 2022-01-31T23:09:20.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:20.012934+0000) 2022-01-31T23:09:20.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:20 smithi167 conmon[54076]: debug 2022-01-31T23:09:20.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:20.267732+0000) 2022-01-31T23:09:20.994 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:20 smithi171 conmon[51620]: debug 2022-01-31T23:09:20.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:20.644094+0000) 2022-01-31T23:09:21.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:20 smithi167 conmon[49112]: debug 2022-01-31T23:09:20.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:20.847838+0000) 2022-01-31T23:09:21.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:21 smithi167 conmon[60316]: debug 2022-01-31T23:09:21.025+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:21.026366+0000) 2022-01-31T23:09:21.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:21 smithi171 conmon[41853]: debug 2022-01-31T23:09:21.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:21.215232+0000) 2022-01-31T23:09:21.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:21 smithi171 conmon[46715]: debug 2022-01-31T23:09:21.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:21.013087+0000) 2022-01-31T23:09:21.499 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:21 smithi167 conmon[54076]: debug 2022-01-31T23:09:21.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:21.267923+0000) 2022-01-31T23:09:21.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:21 smithi171 conmon[51620]: debug 2022-01-31T23:09:21.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:21.644292+0000) 2022-01-31T23:09:22.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:21 smithi167 conmon[49112]: debug 2022-01-31T23:09:21.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:21.848025+0000) 2022-01-31T23:09:22.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:22 smithi167 conmon[60316]: debug 2022-01-31T23:09:22.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:22.026520+0000) 2022-01-31T23:09:22.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:22 smithi171 conmon[46715]: debug 2022-01-31T23:09:22.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:22.013307+0000) 2022-01-31T23:09:22.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:22 smithi171 conmon[41853]: debug 2022-01-31T23:09:22.214+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:22.215371+0000) 2022-01-31T23:09:22.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:22 smithi167 conmon[54076]: debug 2022-01-31T23:09:22.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:22.268086+0000) 2022-01-31T23:09:22.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:22 smithi171 conmon[51620]: debug 2022-01-31T23:09:22.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:22.644476+0000) 2022-01-31T23:09:23.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:22 smithi167 conmon[49112]: debug 2022-01-31T23:09:22.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:22.848230+0000) 2022-01-31T23:09:23.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:23 smithi167 conmon[60316]: debug 2022-01-31T23:09:23.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:23.026621+0000) 2022-01-31T23:09:23.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:23 smithi171 conmon[41853]: debug 2022-01-31T23:09:23.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:23.215554+0000) 2022-01-31T23:09:23.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:23 smithi171 conmon[46715]: debug 2022-01-31T23:09:23.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:23.013493+0000) 2022-01-31T23:09:23.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:23 smithi167 conmon[54076]: debug 2022-01-31T23:09:23.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:23.268244+0000) 2022-01-31T23:09:23.953 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:23 smithi171 conmon[51620]: debug 2022-01-31T23:09:23.643+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:23.644667+0000) 2022-01-31T23:09:24.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:23 smithi167 conmon[49112]: debug 2022-01-31T23:09:23.847+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:23.848443+0000) 2022-01-31T23:09:24.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:24 smithi167 conmon[60316]: debug 2022-01-31T23:09:24.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.026769+0000) 2022-01-31T23:09:24.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[46715]: debug 2022-01-31T23:09:24.012+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.013684+0000) 2022-01-31T23:09:24.215 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:23 smithi171 conmon[35325]: debug 2022-01-31T23:09:23.971+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:09:24.216 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[41853]: debug 2022-01-31T23:09:24.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.215719+0000) 2022-01-31T23:09:24.500 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:24 smithi167 conmon[49112]: debug 2022-01-31T23:09:24.234+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.235041+0000) 2022-01-31T23:09:24.501 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:24 smithi167 conmon[60316]: debug 2022-01-31T23:09:24.233+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.233841+0000) 2022-01-31T23:09:24.502 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:24 smithi167 conmon[54076]: debug 2022-01-31T23:09:24.234+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.234678+0000) 2022-01-31T23:09:24.502 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:24 smithi167 conmon[54076]: debug 2022-01-31T23:09:24.267+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.268382+0000) 2022-01-31T23:09:24.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[35325]: debug 2022-01-31T23:09:24.261+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267055 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:24.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[41853]: debug 2022-01-31T23:09:24.233+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.233793+0000) 2022-01-31T23:09:24.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[46715]: debug 2022-01-31T23:09:24.233+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.234343+0000) 2022-01-31T23:09:24.591 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[51620]: debug 2022-01-31T23:09:24.233+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.234579+0000) 2022-01-31T23:09:24.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:24 smithi171 conmon[51620]: debug 2022-01-31T23:09:24.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.644915+0000) 2022-01-31T23:09:25.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:25 smithi167 conmon[60316]: debug 2022-01-31T23:09:25.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:25.026994+0000) 2022-01-31T23:09:25.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:24 smithi167 conmon[49112]: debug 2022-01-31T23:09:24.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:24.848565+0000) 2022-01-31T23:09:25.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:25 smithi171 conmon[46715]: debug 2022-01-31T23:09:25.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:25.013907+0000) 2022-01-31T23:09:25.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:25 smithi171 conmon[41853]: debug 2022-01-31T23:09:25.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:25.215883+0000) 2022-01-31T23:09:25.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:25 smithi167 conmon[54076]: debug 2022-01-31T23:09:25.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:25.268572+0000) 2022-01-31T23:09:25.995 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:25 smithi171 conmon[51620]: debug 2022-01-31T23:09:25.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:25.645087+0000) 2022-01-31T23:09:26.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:25 smithi167 conmon[49112]: debug 2022-01-31T23:09:25.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:25.848757+0000) 2022-01-31T23:09:26.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:26 smithi167 conmon[60316]: debug 2022-01-31T23:09:26.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:26.027168+0000) 2022-01-31T23:09:26.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:26 smithi171 conmon[41853]: debug 2022-01-31T23:09:26.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:26.216001+0000) 2022-01-31T23:09:26.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:26 smithi171 conmon[46715]: debug 2022-01-31T23:09:26.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:26.014099+0000) 2022-01-31T23:09:26.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:26 smithi167 conmon[54076]: debug 2022-01-31T23:09:26.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:26.268770+0000) 2022-01-31T23:09:26.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:26 smithi171 conmon[51620]: debug 2022-01-31T23:09:26.644+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:26.645301+0000) 2022-01-31T23:09:27.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:27 smithi167 conmon[60316]: debug 2022-01-31T23:09:27.026+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:27.027351+0000) 2022-01-31T23:09:27.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:26 smithi167 conmon[49112]: debug 2022-01-31T23:09:26.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:26.848935+0000) 2022-01-31T23:09:27.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:27 smithi171 conmon[41853]: debug 2022-01-31T23:09:27.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:27.216161+0000) 2022-01-31T23:09:27.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:27 smithi171 conmon[46715]: debug 2022-01-31T23:09:27.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:27.014328+0000) 2022-01-31T23:09:27.500 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:27 smithi167 conmon[54076]: debug 2022-01-31T23:09:27.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:27.268937+0000) 2022-01-31T23:09:27.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:27 smithi171 conmon[51620]: debug 2022-01-31T23:09:27.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:27.645513+0000) 2022-01-31T23:09:28.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:27 smithi167 conmon[49112]: debug 2022-01-31T23:09:27.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:27.849128+0000) 2022-01-31T23:09:28.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:28 smithi167 conmon[60316]: debug 2022-01-31T23:09:28.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:28.027538+0000) 2022-01-31T23:09:28.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:28 smithi171 conmon[46715]: debug 2022-01-31T23:09:28.013+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:28.014548+0000) 2022-01-31T23:09:28.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:28 smithi171 conmon[41853]: debug 2022-01-31T23:09:28.215+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:28.216343+0000) 2022-01-31T23:09:28.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:28 smithi167 conmon[54076]: debug 2022-01-31T23:09:28.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:28.269108+0000) 2022-01-31T23:09:28.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:28 smithi171 conmon[51620]: debug 2022-01-31T23:09:28.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:28.645731+0000) 2022-01-31T23:09:29.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:28 smithi167 conmon[49112]: debug 2022-01-31T23:09:28.848+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:28.849322+0000) 2022-01-31T23:09:29.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:29 smithi167 conmon[60316]: debug 2022-01-31T23:09:29.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.027671+0000) 2022-01-31T23:09:29.246 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[41853]: debug 2022-01-31T23:09:29.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.216537+0000) 2022-01-31T23:09:29.247 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[46715]: debug 2022-01-31T23:09:29.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.014738+0000) 2022-01-31T23:09:29.248 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[51620]: debug 2022-01-31T23:09:29.263+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.264650+0000) 2022-01-31T23:09:29.501 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:29 smithi167 conmon[49112]: debug 2022-01-31T23:09:29.265+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.266421+0000) 2022-01-31T23:09:29.502 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:29 smithi167 conmon[60316]: debug 2022-01-31T23:09:29.267+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.267551+0000) 2022-01-31T23:09:29.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:29 smithi167 conmon[54076]: debug 2022-01-31T23:09:29.265+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.266131+0000) 2022-01-31T23:09:29.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:29 smithi167 conmon[54076]: debug 2022-01-31T23:09:29.268+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.269208+0000) 2022-01-31T23:09:29.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[35325]: debug 2022-01-31T23:09:29.293+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267167 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:29.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[41853]: debug 2022-01-31T23:09:29.264+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.265347+0000) 2022-01-31T23:09:29.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[46715]: debug 2022-01-31T23:09:29.266+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.267123+0000) 2022-01-31T23:09:29.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:29 smithi171 conmon[51620]: debug 2022-01-31T23:09:29.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.645980+0000) 2022-01-31T23:09:30.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:29 smithi167 conmon[49112]: debug 2022-01-31T23:09:29.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:29.849521+0000) 2022-01-31T23:09:30.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:30 smithi167 conmon[60316]: debug 2022-01-31T23:09:30.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:30.027780+0000) 2022-01-31T23:09:30.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:30 smithi171 conmon[41853]: debug 2022-01-31T23:09:30.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:30.216681+0000) 2022-01-31T23:09:30.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:30 smithi171 conmon[46715]: debug 2022-01-31T23:09:30.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:30.014906+0000) 2022-01-31T23:09:30.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:30 smithi167 conmon[54076]: debug 2022-01-31T23:09:30.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:30.269370+0000) 2022-01-31T23:09:30.996 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:30 smithi171 conmon[51620]: debug 2022-01-31T23:09:30.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:30.646154+0000) 2022-01-31T23:09:31.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:30 smithi167 conmon[49112]: debug 2022-01-31T23:09:30.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:30.849682+0000) 2022-01-31T23:09:31.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:31 smithi167 conmon[60316]: debug 2022-01-31T23:09:31.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:31.027928+0000) 2022-01-31T23:09:31.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:31 smithi171 conmon[41853]: debug 2022-01-31T23:09:31.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:31.216822+0000) 2022-01-31T23:09:31.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:31 smithi171 conmon[46715]: debug 2022-01-31T23:09:31.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:31.015069+0000) 2022-01-31T23:09:31.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:31 smithi167 conmon[54076]: debug 2022-01-31T23:09:31.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:31.269572+0000) 2022-01-31T23:09:31.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:31 smithi171 conmon[51620]: debug 2022-01-31T23:09:31.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:31.646329+0000) 2022-01-31T23:09:32.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:31 smithi167 conmon[49112]: debug 2022-01-31T23:09:31.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:31.849887+0000) 2022-01-31T23:09:32.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:32 smithi167 conmon[60316]: debug 2022-01-31T23:09:32.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:32.028088+0000) 2022-01-31T23:09:32.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:32 smithi171 conmon[46715]: debug 2022-01-31T23:09:32.014+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:32.015270+0000) 2022-01-31T23:09:32.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:32 smithi171 conmon[41853]: debug 2022-01-31T23:09:32.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:32.216970+0000) 2022-01-31T23:09:32.501 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:32 smithi167 conmon[54076]: debug 2022-01-31T23:09:32.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:32.269753+0000) 2022-01-31T23:09:32.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:32 smithi171 conmon[51620]: debug 2022-01-31T23:09:32.645+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:32.646480+0000) 2022-01-31T23:09:33.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:32 smithi167 conmon[49112]: debug 2022-01-31T23:09:32.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:32.850053+0000) 2022-01-31T23:09:33.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:33 smithi167 conmon[60316]: debug 2022-01-31T23:09:33.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:33.028286+0000) 2022-01-31T23:09:33.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:33 smithi171 conmon[41853]: debug 2022-01-31T23:09:33.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:33.217155+0000) 2022-01-31T23:09:33.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:33 smithi171 conmon[46715]: debug 2022-01-31T23:09:33.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:33.015462+0000) 2022-01-31T23:09:33.502 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:33 smithi167 conmon[54076]: debug 2022-01-31T23:09:33.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:33.269943+0000) 2022-01-31T23:09:33.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:33 smithi171 conmon[51620]: debug 2022-01-31T23:09:33.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:33.646657+0000) 2022-01-31T23:09:34.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:33 smithi167 conmon[49112]: debug 2022-01-31T23:09:33.849+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:33.850225+0000) 2022-01-31T23:09:34.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:34 smithi167 conmon[60316]: debug 2022-01-31T23:09:34.027+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.028431+0000) 2022-01-31T23:09:34.277 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[46715]: debug 2022-01-31T23:09:34.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.015677+0000) 2022-01-31T23:09:34.278 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[41853]: debug 2022-01-31T23:09:34.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.217347+0000) 2022-01-31T23:09:34.279 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[51620]: debug 2022-01-31T23:09:34.295+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.296423+0000) 2022-01-31T23:09:34.502 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:34 smithi167 conmon[49112]: debug 2022-01-31T23:09:34.303+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.304329+0000) 2022-01-31T23:09:34.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:34 smithi167 conmon[60316]: debug 2022-01-31T23:09:34.303+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.303708+0000) 2022-01-31T23:09:34.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:34 smithi167 conmon[54076]: debug 2022-01-31T23:09:34.269+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.270122+0000) 2022-01-31T23:09:34.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:34 smithi167 conmon[54076]: debug 2022-01-31T23:09:34.303+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.304201+0000) 2022-01-31T23:09:34.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[35325]: debug 2022-01-31T23:09:34.339+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267279 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:34.590 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[41853]: debug 2022-01-31T23:09:34.295+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.296573+0000) 2022-01-31T23:09:34.590 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[46715]: debug 2022-01-31T23:09:34.296+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.297223+0000) 2022-01-31T23:09:34.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:34 smithi171 conmon[51620]: debug 2022-01-31T23:09:34.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.646826+0000) 2022-01-31T23:09:35.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:34 smithi167 conmon[49112]: debug 2022-01-31T23:09:34.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:34.850427+0000) 2022-01-31T23:09:35.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:35 smithi167 conmon[60316]: debug 2022-01-31T23:09:35.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:35.028647+0000) 2022-01-31T23:09:35.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:35 smithi171 conmon[46715]: debug 2022-01-31T23:09:35.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:35.015876+0000) 2022-01-31T23:09:35.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:35 smithi171 conmon[41853]: debug 2022-01-31T23:09:35.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:35.217496+0000) 2022-01-31T23:09:35.502 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:35 smithi167 conmon[54076]: debug 2022-01-31T23:09:35.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:35.270320+0000) 2022-01-31T23:09:35.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:35 smithi171 conmon[51620]: debug 2022-01-31T23:09:35.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:35.646974+0000) 2022-01-31T23:09:36.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:35 smithi167 conmon[49112]: debug 2022-01-31T23:09:35.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:35.850594+0000) 2022-01-31T23:09:36.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:36 smithi167 conmon[60316]: debug 2022-01-31T23:09:36.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:36.028813+0000) 2022-01-31T23:09:36.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:36 smithi171 conmon[46715]: debug 2022-01-31T23:09:36.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:36.016029+0000) 2022-01-31T23:09:36.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:36 smithi171 conmon[41853]: debug 2022-01-31T23:09:36.216+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:36.217688+0000) 2022-01-31T23:09:36.502 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:36 smithi167 conmon[54076]: debug 2022-01-31T23:09:36.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:36.270492+0000) 2022-01-31T23:09:36.997 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:36 smithi171 conmon[51620]: debug 2022-01-31T23:09:36.646+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:36.647182+0000) 2022-01-31T23:09:37.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:36 smithi167 conmon[49112]: debug 2022-01-31T23:09:36.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:36.850778+0000) 2022-01-31T23:09:37.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:37 smithi167 conmon[60316]: debug 2022-01-31T23:09:37.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:37.028987+0000) 2022-01-31T23:09:37.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:37 smithi171 conmon[46715]: debug 2022-01-31T23:09:37.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:37.016188+0000) 2022-01-31T23:09:37.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:37 smithi171 conmon[41853]: debug 2022-01-31T23:09:37.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:37.217844+0000) 2022-01-31T23:09:37.502 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:37 smithi167 conmon[54076]: debug 2022-01-31T23:09:37.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:37.270722+0000) 2022-01-31T23:09:37.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:37 smithi171 conmon[51620]: debug 2022-01-31T23:09:37.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:37.647412+0000) 2022-01-31T23:09:38.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:37 smithi167 conmon[49112]: debug 2022-01-31T23:09:37.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:37.850961+0000) 2022-01-31T23:09:38.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:38 smithi167 conmon[60316]: debug 2022-01-31T23:09:38.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:38.029085+0000) 2022-01-31T23:09:38.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:38 smithi171 conmon[41853]: debug 2022-01-31T23:09:38.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:38.218023+0000) 2022-01-31T23:09:38.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:38 smithi171 conmon[46715]: debug 2022-01-31T23:09:38.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:38.016352+0000) 2022-01-31T23:09:38.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:38 smithi167 conmon[54076]: debug 2022-01-31T23:09:38.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:38.270948+0000) 2022-01-31T23:09:38.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:38 smithi171 conmon[51620]: debug 2022-01-31T23:09:38.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:38.647620+0000) 2022-01-31T23:09:39.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:38 smithi167 conmon[49112]: debug 2022-01-31T23:09:38.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:38.851144+0000) 2022-01-31T23:09:39.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:39 smithi167 conmon[60316]: debug 2022-01-31T23:09:39.028+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.029236+0000) 2022-01-31T23:09:39.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[46715]: debug 2022-01-31T23:09:39.015+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.016522+0000) 2022-01-31T23:09:39.324 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[46715]: debug 2022-01-31T23:09:39.341+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.341759+0000) 2022-01-31T23:09:39.325 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:38 smithi171 conmon[35325]: debug 2022-01-31T23:09:38.972+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:09:39.326 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[41853]: debug 2022-01-31T23:09:39.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.218200+0000) 2022-01-31T23:09:39.326 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[51620]: debug 2022-01-31T23:09:39.341+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.342014+0000) 2022-01-31T23:09:39.503 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:39 smithi167 conmon[49112]: debug 2022-01-31T23:09:39.343+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.344281+0000) 2022-01-31T23:09:39.503 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:39 smithi167 conmon[60316]: debug 2022-01-31T23:09:39.343+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.343774+0000) 2022-01-31T23:09:39.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:39 smithi167 conmon[54076]: debug 2022-01-31T23:09:39.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.271147+0000) 2022-01-31T23:09:39.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:39 smithi167 conmon[54076]: debug 2022-01-31T23:09:39.342+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.343198+0000) 2022-01-31T23:09:39.589 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[35325]: debug 2022-01-31T23:09:39.370+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267390 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:39.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[41853]: debug 2022-01-31T23:09:39.343+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.344157+0000) 2022-01-31T23:09:39.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:39 smithi171 conmon[51620]: debug 2022-01-31T23:09:39.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.647801+0000) 2022-01-31T23:09:40.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:39 smithi167 conmon[49112]: debug 2022-01-31T23:09:39.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:39.851325+0000) 2022-01-31T23:09:40.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:40 smithi167 conmon[60316]: debug 2022-01-31T23:09:40.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:40.029417+0000) 2022-01-31T23:09:40.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:40 smithi171 conmon[41853]: debug 2022-01-31T23:09:40.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:40.218407+0000) 2022-01-31T23:09:40.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:40 smithi171 conmon[46715]: debug 2022-01-31T23:09:40.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:40.016707+0000) 2022-01-31T23:09:40.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:40 smithi167 conmon[54076]: debug 2022-01-31T23:09:40.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:40.271349+0000) 2022-01-31T23:09:40.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:40 smithi171 conmon[51620]: debug 2022-01-31T23:09:40.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:40.647980+0000) 2022-01-31T23:09:41.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:40 smithi167 conmon[49112]: debug 2022-01-31T23:09:40.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:40.851480+0000) 2022-01-31T23:09:41.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:41 smithi167 conmon[60316]: debug 2022-01-31T23:09:41.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:41.029567+0000) 2022-01-31T23:09:41.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:41 smithi171 conmon[41853]: debug 2022-01-31T23:09:41.217+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:41.218584+0000) 2022-01-31T23:09:41.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:41 smithi171 conmon[46715]: debug 2022-01-31T23:09:41.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:41.016859+0000) 2022-01-31T23:09:41.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:41 smithi167 conmon[54076]: debug 2022-01-31T23:09:41.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:41.271561+0000) 2022-01-31T23:09:41.998 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:41 smithi171 conmon[51620]: debug 2022-01-31T23:09:41.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:41.648190+0000) 2022-01-31T23:09:42.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:41 smithi167 conmon[49112]: debug 2022-01-31T23:09:41.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:41.851641+0000) 2022-01-31T23:09:42.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:42 smithi167 conmon[60316]: debug 2022-01-31T23:09:42.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:42.029748+0000) 2022-01-31T23:09:42.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:42 smithi171 conmon[41853]: debug 2022-01-31T23:09:42.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:42.218805+0000) 2022-01-31T23:09:42.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:42 smithi171 conmon[46715]: debug 2022-01-31T23:09:42.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:42.017013+0000) 2022-01-31T23:09:42.503 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:42 smithi167 conmon[54076]: debug 2022-01-31T23:09:42.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:42.271751+0000) 2022-01-31T23:09:42.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:42 smithi171 conmon[51620]: debug 2022-01-31T23:09:42.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:42.648403+0000) 2022-01-31T23:09:43.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:42 smithi167 conmon[49112]: debug 2022-01-31T23:09:42.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:42.851813+0000) 2022-01-31T23:09:43.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:43 smithi167 conmon[60316]: debug 2022-01-31T23:09:43.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:43.029933+0000) 2022-01-31T23:09:43.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:43 smithi171 conmon[41853]: debug 2022-01-31T23:09:43.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:43.218962+0000) 2022-01-31T23:09:43.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:43 smithi171 conmon[46715]: debug 2022-01-31T23:09:43.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:43.017219+0000) 2022-01-31T23:09:43.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:43 smithi167 conmon[54076]: debug 2022-01-31T23:09:43.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:43.271870+0000) 2022-01-31T23:09:43.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:43 smithi171 conmon[51620]: debug 2022-01-31T23:09:43.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:43.648617+0000) 2022-01-31T23:09:44.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:43 smithi167 conmon[49112]: debug 2022-01-31T23:09:43.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:43.851993+0000) 2022-01-31T23:09:44.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:44 smithi167 conmon[60316]: debug 2022-01-31T23:09:44.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.030125+0000) 2022-01-31T23:09:44.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[41853]: debug 2022-01-31T23:09:44.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.219107+0000) 2022-01-31T23:09:44.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[46715]: debug 2022-01-31T23:09:44.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.017412+0000) 2022-01-31T23:09:44.504 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:44 smithi167 conmon[49112]: debug 2022-01-31T23:09:44.373+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.374832+0000) 2022-01-31T23:09:44.505 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:44 smithi167 conmon[60316]: debug 2022-01-31T23:09:44.373+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.375397+0000) 2022-01-31T23:09:44.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:44 smithi167 conmon[54076]: debug 2022-01-31T23:09:44.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.272059+0000) 2022-01-31T23:09:44.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:44 smithi167 conmon[54076]: debug 2022-01-31T23:09:44.372+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.374224+0000) 2022-01-31T23:09:44.630 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[35325]: debug 2022-01-31T23:09:44.401+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267502 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:44.631 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[41853]: debug 2022-01-31T23:09:44.373+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.374584+0000) 2022-01-31T23:09:44.632 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[46715]: debug 2022-01-31T23:09:44.372+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.373370+0000) 2022-01-31T23:09:44.632 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[51620]: debug 2022-01-31T23:09:44.372+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.373707+0000) 2022-01-31T23:09:44.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:44 smithi171 conmon[51620]: debug 2022-01-31T23:09:44.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.648816+0000) 2022-01-31T23:09:45.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:44 smithi167 conmon[49112]: debug 2022-01-31T23:09:44.850+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:44.852197+0000) 2022-01-31T23:09:45.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:45 smithi167 conmon[60316]: debug 2022-01-31T23:09:45.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:45.030330+0000) 2022-01-31T23:09:45.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:45 smithi171 conmon[41853]: debug 2022-01-31T23:09:45.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:45.219259+0000) 2022-01-31T23:09:45.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:45 smithi171 conmon[46715]: debug 2022-01-31T23:09:45.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:45.017586+0000) 2022-01-31T23:09:45.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:45 smithi167 conmon[54076]: debug 2022-01-31T23:09:45.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:45.272246+0000) 2022-01-31T23:09:45.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:45 smithi171 conmon[51620]: debug 2022-01-31T23:09:45.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:45.648971+0000) 2022-01-31T23:09:46.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:45 smithi167 conmon[49112]: debug 2022-01-31T23:09:45.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:45.852404+0000) 2022-01-31T23:09:46.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:46 smithi167 conmon[60316]: debug 2022-01-31T23:09:46.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:46.030464+0000) 2022-01-31T23:09:46.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:46 smithi171 conmon[41853]: debug 2022-01-31T23:09:46.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:46.219452+0000) 2022-01-31T23:09:46.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:46 smithi171 conmon[46715]: debug 2022-01-31T23:09:46.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:46.017731+0000) 2022-01-31T23:09:46.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:46 smithi167 conmon[54076]: debug 2022-01-31T23:09:46.270+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:46.272435+0000) 2022-01-31T23:09:46.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:46 smithi171 conmon[51620]: debug 2022-01-31T23:09:46.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:46.649135+0000) 2022-01-31T23:09:47.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:46 smithi167 conmon[49112]: debug 2022-01-31T23:09:46.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:46.852588+0000) 2022-01-31T23:09:47.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:47 smithi167 conmon[60316]: debug 2022-01-31T23:09:47.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:47.030636+0000) 2022-01-31T23:09:47.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:47 smithi171 conmon[41853]: debug 2022-01-31T23:09:47.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:47.219657+0000) 2022-01-31T23:09:47.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:47 smithi171 conmon[46715]: debug 2022-01-31T23:09:47.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:47.017911+0000) 2022-01-31T23:09:47.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:47 smithi167 conmon[54076]: debug 2022-01-31T23:09:47.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:47.272603+0000) 2022-01-31T23:09:47.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:47 smithi171 conmon[51620]: debug 2022-01-31T23:09:47.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:47.649321+0000) 2022-01-31T23:09:48.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:47 smithi167 conmon[49112]: debug 2022-01-31T23:09:47.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:47.852710+0000) 2022-01-31T23:09:48.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:48 smithi167 conmon[60316]: debug 2022-01-31T23:09:48.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:48.030866+0000) 2022-01-31T23:09:48.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:48 smithi171 conmon[41853]: debug 2022-01-31T23:09:48.218+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:48.219864+0000) 2022-01-31T23:09:48.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:48 smithi171 conmon[46715]: debug 2022-01-31T23:09:48.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:48.018090+0000) 2022-01-31T23:09:48.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:48 smithi167 conmon[54076]: debug 2022-01-31T23:09:48.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:48.272765+0000) 2022-01-31T23:09:48.999 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:48 smithi171 conmon[51620]: debug 2022-01-31T23:09:48.647+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:48.649552+0000) 2022-01-31T23:09:49.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:48 smithi167 conmon[49112]: debug 2022-01-31T23:09:48.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:48.852863+0000) 2022-01-31T23:09:49.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:49 smithi167 conmon[60316]: debug 2022-01-31T23:09:49.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.031079+0000) 2022-01-31T23:09:49.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[41853]: debug 2022-01-31T23:09:49.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.220044+0000) 2022-01-31T23:09:49.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[46715]: debug 2022-01-31T23:09:49.016+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.018290+0000) 2022-01-31T23:09:49.505 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:49 smithi167 conmon[49112]: debug 2022-01-31T23:09:49.404+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.406133+0000) 2022-01-31T23:09:49.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:49 smithi167 conmon[54076]: debug 2022-01-31T23:09:49.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.272981+0000) 2022-01-31T23:09:49.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:49 smithi167 conmon[54076]: debug 2022-01-31T23:09:49.404+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.406405+0000) 2022-01-31T23:09:49.506 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:49 smithi167 conmon[60316]: debug 2022-01-31T23:09:49.404+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.405785+0000) 2022-01-31T23:09:49.661 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[35325]: debug 2022-01-31T23:09:49.432+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267616 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:49.662 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[41853]: debug 2022-01-31T23:09:49.404+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.405869+0000) 2022-01-31T23:09:49.663 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[46715]: debug 2022-01-31T23:09:49.402+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.404701+0000) 2022-01-31T23:09:49.663 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[51620]: debug 2022-01-31T23:09:49.403+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.405173+0000) 2022-01-31T23:09:49.664 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:49 smithi171 conmon[51620]: debug 2022-01-31T23:09:49.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.649783+0000) 2022-01-31T23:09:50.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:49 smithi167 conmon[49112]: debug 2022-01-31T23:09:49.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:49.853084+0000) 2022-01-31T23:09:50.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:50 smithi167 conmon[60316]: debug 2022-01-31T23:09:50.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:50.031256+0000) 2022-01-31T23:09:50.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:50 smithi171 conmon[41853]: debug 2022-01-31T23:09:50.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:50.220237+0000) 2022-01-31T23:09:50.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:50 smithi171 conmon[46715]: debug 2022-01-31T23:09:50.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:50.018506+0000) 2022-01-31T23:09:50.504 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:50 smithi167 conmon[54076]: debug 2022-01-31T23:09:50.271+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:50.273184+0000) 2022-01-31T23:09:51.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:50 smithi171 conmon[51620]: debug 2022-01-31T23:09:50.648+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:50.649929+0000) 2022-01-31T23:09:51.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:51 smithi167 conmon[60316]: debug 2022-01-31T23:09:51.029+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:51.031365+0000) 2022-01-31T23:09:51.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:50 smithi167 conmon[49112]: debug 2022-01-31T23:09:50.851+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:50.853260+0000) 2022-01-31T23:09:51.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:51 smithi171 conmon[46715]: debug 2022-01-31T23:09:51.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:51.018641+0000) 2022-01-31T23:09:51.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:51 smithi171 conmon[41853]: debug 2022-01-31T23:09:51.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:51.220394+0000) 2022-01-31T23:09:51.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:51 smithi167 conmon[54076]: debug 2022-01-31T23:09:51.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:51.273357+0000) 2022-01-31T23:09:52.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:51 smithi171 conmon[51620]: debug 2022-01-31T23:09:51.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:51.650159+0000) 2022-01-31T23:09:52.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:51 smithi167 conmon[49112]: debug 2022-01-31T23:09:51.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:51.853449+0000) 2022-01-31T23:09:52.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:52 smithi167 conmon[60316]: debug 2022-01-31T23:09:52.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:52.031526+0000) 2022-01-31T23:09:52.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:52 smithi171 conmon[41853]: debug 2022-01-31T23:09:52.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:52.220515+0000) 2022-01-31T23:09:52.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:52 smithi171 conmon[46715]: debug 2022-01-31T23:09:52.017+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:52.018820+0000) 2022-01-31T23:09:52.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:52 smithi167 conmon[54076]: debug 2022-01-31T23:09:52.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:52.273539+0000) 2022-01-31T23:09:53.000 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:52 smithi171 conmon[51620]: debug 2022-01-31T23:09:52.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:52.650373+0000) 2022-01-31T23:09:53.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:53 smithi167 conmon[60316]: debug 2022-01-31T23:09:53.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:53.031685+0000) 2022-01-31T23:09:53.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:52 smithi167 conmon[49112]: debug 2022-01-31T23:09:52.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:52.853606+0000) 2022-01-31T23:09:53.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:53 smithi171 conmon[41853]: debug 2022-01-31T23:09:53.219+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:53.220700+0000) 2022-01-31T23:09:53.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:53 smithi171 conmon[46715]: debug 2022-01-31T23:09:53.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:53.019009+0000) 2022-01-31T23:09:53.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:53 smithi167 conmon[54076]: debug 2022-01-31T23:09:53.272+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:53.273659+0000) 2022-01-31T23:09:53.956 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:53 smithi171 conmon[51620]: debug 2022-01-31T23:09:53.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:53.650576+0000) 2022-01-31T23:09:54.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:54 smithi167 conmon[60316]: debug 2022-01-31T23:09:54.030+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.031873+0000) 2022-01-31T23:09:54.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:53 smithi167 conmon[49112]: debug 2022-01-31T23:09:53.852+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:53.853787+0000) 2022-01-31T23:09:54.339 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:53 smithi171 conmon[35325]: debug 2022-01-31T23:09:53.972+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:09:54.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[41853]: debug 2022-01-31T23:09:54.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.220858+0000) 2022-01-31T23:09:54.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[46715]: debug 2022-01-31T23:09:54.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.019161+0000) 2022-01-31T23:09:54.506 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:54 smithi167 conmon[49112]: debug 2022-01-31T23:09:54.436+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.437525+0000) 2022-01-31T23:09:54.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:54 smithi167 conmon[60316]: debug 2022-01-31T23:09:54.437+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.438632+0000) 2022-01-31T23:09:54.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:54 smithi167 conmon[54076]: debug 2022-01-31T23:09:54.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.273883+0000) 2022-01-31T23:09:54.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:54 smithi167 conmon[54076]: debug 2022-01-31T23:09:54.436+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.437944+0000) 2022-01-31T23:09:54.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[35325]: debug 2022-01-31T23:09:54.464+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267729 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:54.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[41853]: debug 2022-01-31T23:09:54.436+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.438479+0000) 2022-01-31T23:09:54.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[46715]: debug 2022-01-31T23:09:54.434+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.436628+0000) 2022-01-31T23:09:54.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[51620]: debug 2022-01-31T23:09:54.435+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.437307+0000) 2022-01-31T23:09:54.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:54 smithi171 conmon[51620]: debug 2022-01-31T23:09:54.649+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.650762+0000) 2022-01-31T23:09:55.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:54 smithi167 conmon[49112]: debug 2022-01-31T23:09:54.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:54.853975+0000) 2022-01-31T23:09:55.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:55 smithi167 conmon[60316]: debug 2022-01-31T23:09:55.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:55.032048+0000) 2022-01-31T23:09:55.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:55 smithi171 conmon[41853]: debug 2022-01-31T23:09:55.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:55.221031+0000) 2022-01-31T23:09:55.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:55 smithi171 conmon[46715]: debug 2022-01-31T23:09:55.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:55.019385+0000) 2022-01-31T23:09:55.505 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:55 smithi167 conmon[54076]: debug 2022-01-31T23:09:55.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:55.274076+0000) 2022-01-31T23:09:56.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:55 smithi171 conmon[51620]: debug 2022-01-31T23:09:55.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:55.650900+0000) 2022-01-31T23:09:56.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:55 smithi167 conmon[49112]: debug 2022-01-31T23:09:55.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:55.854157+0000) 2022-01-31T23:09:56.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:56 smithi167 conmon[60316]: debug 2022-01-31T23:09:56.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:56.032204+0000) 2022-01-31T23:09:56.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:56 smithi171 conmon[46715]: debug 2022-01-31T23:09:56.018+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:56.019575+0000) 2022-01-31T23:09:56.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:56 smithi171 conmon[41853]: debug 2022-01-31T23:09:56.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:56.221234+0000) 2022-01-31T23:09:56.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:56 smithi167 conmon[54076]: debug 2022-01-31T23:09:56.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:56.274225+0000) 2022-01-31T23:09:57.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:56 smithi171 conmon[51620]: debug 2022-01-31T23:09:56.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:56.651118+0000) 2022-01-31T23:09:57.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:56 smithi167 conmon[49112]: debug 2022-01-31T23:09:56.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:56.854338+0000) 2022-01-31T23:09:57.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:57 smithi167 conmon[60316]: debug 2022-01-31T23:09:57.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:57.032402+0000) 2022-01-31T23:09:57.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:57 smithi171 conmon[41853]: debug 2022-01-31T23:09:57.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:57.221408+0000) 2022-01-31T23:09:57.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:57 smithi171 conmon[46715]: debug 2022-01-31T23:09:57.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:57.019801+0000) 2022-01-31T23:09:57.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:57 smithi167 conmon[54076]: debug 2022-01-31T23:09:57.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:57.274411+0000) 2022-01-31T23:09:58.001 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:57 smithi171 conmon[51620]: debug 2022-01-31T23:09:57.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:57.651358+0000) 2022-01-31T23:09:58.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:57 smithi167 conmon[49112]: debug 2022-01-31T23:09:57.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:57.854492+0000) 2022-01-31T23:09:58.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:58 smithi167 conmon[60316]: debug 2022-01-31T23:09:58.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:58.032589+0000) 2022-01-31T23:09:58.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:58 smithi171 conmon[46715]: debug 2022-01-31T23:09:58.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:58.019962+0000) 2022-01-31T23:09:58.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:58 smithi171 conmon[41853]: debug 2022-01-31T23:09:58.220+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:58.221614+0000) 2022-01-31T23:09:58.506 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:58 smithi167 conmon[54076]: debug 2022-01-31T23:09:58.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:58.274573+0000) 2022-01-31T23:09:59.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:58 smithi171 conmon[51620]: debug 2022-01-31T23:09:58.650+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:58.651587+0000) 2022-01-31T23:09:59.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:58 smithi167 conmon[49112]: debug 2022-01-31T23:09:58.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:58.854702+0000) 2022-01-31T23:09:59.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:59 smithi167 conmon[60316]: debug 2022-01-31T23:09:59.031+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.032733+0000) 2022-01-31T23:09:59.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[41853]: debug 2022-01-31T23:09:59.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.221773+0000) 2022-01-31T23:09:59.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[46715]: debug 2022-01-31T23:09:59.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.020153+0000) 2022-01-31T23:09:59.506 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:59 smithi167 conmon[49112]: debug 2022-01-31T23:09:59.467+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.468561+0000) 2022-01-31T23:09:59.507 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:09:59 smithi167 conmon[60316]: debug 2022-01-31T23:09:59.466+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.468477+0000) 2022-01-31T23:09:59.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:59 smithi167 conmon[54076]: debug 2022-01-31T23:09:59.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.274728+0000) 2022-01-31T23:09:59.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:09:59 smithi167 conmon[54076]: debug 2022-01-31T23:09:59.467+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.468821+0000) 2022-01-31T23:09:59.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[35325]: debug 2022-01-31T23:09:59.496+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267840 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:09:59.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[41853]: debug 2022-01-31T23:09:59.469+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.470063+0000) 2022-01-31T23:09:59.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[46715]: debug 2022-01-31T23:09:59.468+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.468783+0000) 2022-01-31T23:09:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[51620]: debug 2022-01-31T23:09:59.466+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.467599+0000) 2022-01-31T23:09:59.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:09:59 smithi171 conmon[51620]: debug 2022-01-31T23:09:59.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.651794+0000) 2022-01-31T23:10:00.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:09:59 smithi167 conmon[49112]: debug 2022-01-31T23:09:59.853+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:09:59.854920+0000) 2022-01-31T23:10:00.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:00 smithi167 conmon[60316]: debug 2022-01-31T23:10:00.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:00.032889+0000) 2022-01-31T23:10:00.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:00 smithi171 conmon[41853]: debug 2022-01-31T23:10:00.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:00.221914+0000) 2022-01-31T23:10:00.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:00 smithi171 conmon[46715]: debug 2022-01-31T23:10:00.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:00.020343+0000) 2022-01-31T23:10:00.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:00 smithi167 conmon[54076]: debug 2022-01-31T23:10:00.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:00.274900+0000) 2022-01-31T23:10:01.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:00 smithi171 conmon[51620]: debug 2022-01-31T23:10:00.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:00.651946+0000) 2022-01-31T23:10:01.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:00 smithi167 conmon[49112]: debug 2022-01-31T23:10:00.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:00.855053+0000) 2022-01-31T23:10:01.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:01 smithi167 conmon[60316]: debug 2022-01-31T23:10:01.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:01.033038+0000) 2022-01-31T23:10:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:01 smithi171 conmon[46715]: debug 2022-01-31T23:10:01.019+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:01.020500+0000) 2022-01-31T23:10:01.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:01 smithi171 conmon[41853]: debug 2022-01-31T23:10:01.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:01.222060+0000) 2022-01-31T23:10:01.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:01 smithi167 conmon[54076]: debug 2022-01-31T23:10:01.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:01.275053+0000) 2022-01-31T23:10:02.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:01 smithi171 conmon[51620]: debug 2022-01-31T23:10:01.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:01.652136+0000) 2022-01-31T23:10:02.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:01 smithi167 conmon[49112]: debug 2022-01-31T23:10:01.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:01.855243+0000) 2022-01-31T23:10:02.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:02 smithi167 conmon[60316]: debug 2022-01-31T23:10:02.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:02.033192+0000) 2022-01-31T23:10:02.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:02 smithi171 conmon[46715]: debug 2022-01-31T23:10:02.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:02.020694+0000) 2022-01-31T23:10:02.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:02 smithi171 conmon[41853]: debug 2022-01-31T23:10:02.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:02.222212+0000) 2022-01-31T23:10:02.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:02 smithi167 conmon[54076]: debug 2022-01-31T23:10:02.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:02.275243+0000) 2022-01-31T23:10:03.002 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:02 smithi171 conmon[51620]: debug 2022-01-31T23:10:02.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:02.652344+0000) 2022-01-31T23:10:03.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:02 smithi167 conmon[49112]: debug 2022-01-31T23:10:02.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:02.855430+0000) 2022-01-31T23:10:03.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:03 smithi167 conmon[60316]: debug 2022-01-31T23:10:03.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:03.033373+0000) 2022-01-31T23:10:03.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:03 smithi171 conmon[41853]: debug 2022-01-31T23:10:03.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:03.222400+0000) 2022-01-31T23:10:03.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:03 smithi171 conmon[46715]: debug 2022-01-31T23:10:03.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:03.020899+0000) 2022-01-31T23:10:03.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:03 smithi167 conmon[54076]: debug 2022-01-31T23:10:03.273+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:03.275428+0000) 2022-01-31T23:10:03.839 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:03 smithi171 conmon[51620]: debug 2022-01-31T23:10:03.651+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:03.652545+0000) 2022-01-31T23:10:04.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:03 smithi167 conmon[49112]: debug 2022-01-31T23:10:03.854+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:03.855613+0000) 2022-01-31T23:10:04.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:04 smithi167 conmon[60316]: debug 2022-01-31T23:10:04.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.033514+0000) 2022-01-31T23:10:04.204 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[46715]: debug 2022-01-31T23:10:04.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.021056+0000) 2022-01-31T23:10:04.480 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[41853]: debug 2022-01-31T23:10:04.221+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.222576+0000) 2022-01-31T23:10:04.507 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:04 smithi167 conmon[49112]: debug 2022-01-31T23:10:04.497+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.499434+0000) 2022-01-31T23:10:04.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:04 smithi167 conmon[54076]: debug 2022-01-31T23:10:04.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.275618+0000) 2022-01-31T23:10:04.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:04 smithi167 conmon[54076]: debug 2022-01-31T23:10:04.497+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.499199+0000) 2022-01-31T23:10:04.509 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:04 smithi167 conmon[60316]: debug 2022-01-31T23:10:04.498+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.500383+0000) 2022-01-31T23:10:04.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[35325]: debug 2022-01-31T23:10:04.527+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 267953 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:04.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[41853]: debug 2022-01-31T23:10:04.499+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.500327+0000) 2022-01-31T23:10:04.840 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[46715]: debug 2022-01-31T23:10:04.499+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.500235+0000) 2022-01-31T23:10:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[51620]: debug 2022-01-31T23:10:04.498+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.499263+0000) 2022-01-31T23:10:04.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:04 smithi171 conmon[51620]: debug 2022-01-31T23:10:04.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.652731+0000) 2022-01-31T23:10:05.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:04 smithi167 conmon[49112]: debug 2022-01-31T23:10:04.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:04.855739+0000) 2022-01-31T23:10:05.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:05 smithi167 conmon[60316]: debug 2022-01-31T23:10:05.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:05.033709+0000) 2022-01-31T23:10:05.295 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:05 smithi171 conmon[46715]: debug 2022-01-31T23:10:05.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:05.021274+0000) 2022-01-31T23:10:05.296 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:05 smithi171 conmon[41853]: debug 2022-01-31T23:10:05.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:05.222795+0000) 2022-01-31T23:10:05.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:05 smithi167 conmon[54076]: debug 2022-01-31T23:10:05.274+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:05.275784+0000) 2022-01-31T23:10:06.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:05 smithi171 conmon[51620]: debug 2022-01-31T23:10:05.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:05.652873+0000) 2022-01-31T23:10:06.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:05 smithi167 conmon[49112]: debug 2022-01-31T23:10:05.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:05.855846+0000) 2022-01-31T23:10:06.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:06 smithi167 conmon[60316]: debug 2022-01-31T23:10:06.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:06.033832+0000) 2022-01-31T23:10:06.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:06 smithi171 conmon[41853]: debug 2022-01-31T23:10:06.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:06.222966+0000) 2022-01-31T23:10:06.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:06 smithi171 conmon[46715]: debug 2022-01-31T23:10:06.020+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:06.021446+0000) 2022-01-31T23:10:06.507 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:06 smithi167 conmon[54076]: debug 2022-01-31T23:10:06.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:06.275946+0000) 2022-01-31T23:10:07.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:06 smithi171 conmon[51620]: debug 2022-01-31T23:10:06.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:06.653082+0000) 2022-01-31T23:10:07.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:06 smithi167 conmon[49112]: debug 2022-01-31T23:10:06.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:06.856008+0000) 2022-01-31T23:10:07.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:07 smithi167 conmon[60316]: debug 2022-01-31T23:10:07.032+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:07.033991+0000) 2022-01-31T23:10:07.279 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:07 smithi171 conmon[41853]: debug 2022-01-31T23:10:07.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:07.223098+0000) 2022-01-31T23:10:07.280 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:07 smithi171 conmon[46715]: debug 2022-01-31T23:10:07.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:07.021654+0000) 2022-01-31T23:10:07.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:07 smithi167 conmon[54076]: debug 2022-01-31T23:10:07.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:07.276166+0000) 2022-01-31T23:10:08.003 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:07 smithi171 conmon[51620]: debug 2022-01-31T23:10:07.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:07.653313+0000) 2022-01-31T23:10:08.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:07 smithi167 conmon[49112]: debug 2022-01-31T23:10:07.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:07.856182+0000) 2022-01-31T23:10:08.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:08 smithi167 conmon[60316]: debug 2022-01-31T23:10:08.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:08.034196+0000) 2022-01-31T23:10:08.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:08 smithi171 conmon[41853]: debug 2022-01-31T23:10:08.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:08.223272+0000) 2022-01-31T23:10:08.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:08 smithi171 conmon[46715]: debug 2022-01-31T23:10:08.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:08.021847+0000) 2022-01-31T23:10:08.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:08 smithi167 conmon[54076]: debug 2022-01-31T23:10:08.275+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:08.276350+0000) 2022-01-31T23:10:08.955 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:08 smithi171 conmon[51620]: debug 2022-01-31T23:10:08.652+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:08.653531+0000) 2022-01-31T23:10:09.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:08 smithi167 conmon[49112]: debug 2022-01-31T23:10:08.855+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:08.856380+0000) 2022-01-31T23:10:09.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:09 smithi167 conmon[60316]: debug 2022-01-31T23:10:09.033+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.034396+0000) 2022-01-31T23:10:09.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[41853]: debug 2022-01-31T23:10:09.222+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.223453+0000) 2022-01-31T23:10:09.340 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:08 smithi171 conmon[35325]: debug 2022-01-31T23:10:08.973+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:10:09.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[46715]: debug 2022-01-31T23:10:09.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.022026+0000) 2022-01-31T23:10:09.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:09 smithi167 conmon[54076]: debug 2022-01-31T23:10:09.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.276555+0000) 2022-01-31T23:10:09.838 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:09 smithi167 conmon[49112]: debug 2022-01-31T23:10:09.530+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.530814+0000) 2022-01-31T23:10:09.839 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:09 smithi167 conmon[54076]: debug 2022-01-31T23:10:09.530+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.530915+0000) 2022-01-31T23:10:09.839 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:09 smithi167 conmon[60316]: debug 2022-01-31T23:10:09.530+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.531330+0000) 2022-01-31T23:10:09.840 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[35325]: debug 2022-01-31T23:10:09.558+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268063 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:09.841 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[41853]: debug 2022-01-31T23:10:09.530+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.531671+0000) 2022-01-31T23:10:09.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[46715]: debug 2022-01-31T23:10:09.529+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.530555+0000) 2022-01-31T23:10:09.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[51620]: debug 2022-01-31T23:10:09.530+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.531478+0000) 2022-01-31T23:10:09.842 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:09 smithi171 conmon[51620]: debug 2022-01-31T23:10:09.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.653795+0000) 2022-01-31T23:10:10.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:09 smithi167 conmon[49112]: debug 2022-01-31T23:10:09.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:09.856574+0000) 2022-01-31T23:10:10.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:10 smithi167 conmon[60316]: debug 2022-01-31T23:10:10.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:10.034595+0000) 2022-01-31T23:10:10.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:10 smithi171 conmon[41853]: debug 2022-01-31T23:10:10.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:10.223638+0000) 2022-01-31T23:10:10.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:10 smithi171 conmon[46715]: debug 2022-01-31T23:10:10.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:10.022209+0000) 2022-01-31T23:10:10.508 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:10 smithi167 conmon[54076]: debug 2022-01-31T23:10:10.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:10.276745+0000) 2022-01-31T23:10:11.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:10 smithi171 conmon[51620]: debug 2022-01-31T23:10:10.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:10.653935+0000) 2022-01-31T23:10:11.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:10 smithi167 conmon[49112]: debug 2022-01-31T23:10:10.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:10.856729+0000) 2022-01-31T23:10:11.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:11 smithi167 conmon[60316]: debug 2022-01-31T23:10:11.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:11.034739+0000) 2022-01-31T23:10:11.311 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:11 smithi171 conmon[41853]: debug 2022-01-31T23:10:11.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:11.223792+0000) 2022-01-31T23:10:11.312 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:11 smithi171 conmon[46715]: debug 2022-01-31T23:10:11.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:11.022344+0000) 2022-01-31T23:10:11.509 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:11 smithi167 conmon[54076]: debug 2022-01-31T23:10:11.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:11.276945+0000) 2022-01-31T23:10:12.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:11 smithi171 conmon[51620]: debug 2022-01-31T23:10:11.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:11.654122+0000) 2022-01-31T23:10:12.089 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:11 smithi167 conmon[49112]: debug 2022-01-31T23:10:11.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:11.856919+0000) 2022-01-31T23:10:12.090 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:12 smithi167 conmon[60316]: debug 2022-01-31T23:10:12.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:12.034894+0000) 2022-01-31T23:10:12.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:12 smithi171 conmon[41853]: debug 2022-01-31T23:10:12.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:12.224008+0000) 2022-01-31T23:10:12.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:12 smithi171 conmon[46715]: debug 2022-01-31T23:10:12.021+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:12.022517+0000) 2022-01-31T23:10:12.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:12 smithi167 conmon[54076]: debug 2022-01-31T23:10:12.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:12.277099+0000) 2022-01-31T23:10:13.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:12 smithi171 conmon[51620]: debug 2022-01-31T23:10:12.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:12.654317+0000) 2022-01-31T23:10:13.092 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:12 smithi167 conmon[49112]: debug 2022-01-31T23:10:12.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:12.857104+0000) 2022-01-31T23:10:13.093 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:13 smithi167 conmon[60316]: debug 2022-01-31T23:10:13.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:13.035052+0000) 2022-01-31T23:10:13.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:13 smithi171 conmon[41853]: debug 2022-01-31T23:10:13.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:13.224186+0000) 2022-01-31T23:10:13.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:13 smithi171 conmon[46715]: debug 2022-01-31T23:10:13.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:13.022707+0000) 2022-01-31T23:10:13.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:13 smithi167 conmon[54076]: debug 2022-01-31T23:10:13.276+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:13.277319+0000) 2022-01-31T23:10:14.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:13 smithi171 conmon[51620]: debug 2022-01-31T23:10:13.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:13.654485+0000) 2022-01-31T23:10:14.096 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:13 smithi167 conmon[49112]: debug 2022-01-31T23:10:13.856+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:13.857296+0000) 2022-01-31T23:10:14.097 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:14 smithi167 conmon[60316]: debug 2022-01-31T23:10:14.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.035240+0000) 2022-01-31T23:10:14.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[41853]: debug 2022-01-31T23:10:14.223+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.224376+0000) 2022-01-31T23:10:14.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[46715]: debug 2022-01-31T23:10:14.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.022920+0000) 2022-01-31T23:10:14.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:14 smithi167 conmon[54076]: debug 2022-01-31T23:10:14.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.277521+0000) 2022-01-31T23:10:14.839 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:14 smithi167 conmon[49112]: debug 2022-01-31T23:10:14.561+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.561793+0000) 2022-01-31T23:10:14.840 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:14 smithi167 conmon[54076]: debug 2022-01-31T23:10:14.560+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.561378+0000) 2022-01-31T23:10:14.840 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:14 smithi167 conmon[60316]: debug 2022-01-31T23:10:14.561+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.562390+0000) 2022-01-31T23:10:14.841 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[35325]: debug 2022-01-31T23:10:14.589+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268176 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:14.842 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[41853]: debug 2022-01-31T23:10:14.560+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.561308+0000) 2022-01-31T23:10:14.842 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[46715]: debug 2022-01-31T23:10:14.560+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.561014+0000) 2022-01-31T23:10:14.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[51620]: debug 2022-01-31T23:10:14.560+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.561740+0000) 2022-01-31T23:10:14.843 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:14 smithi171 conmon[51620]: debug 2022-01-31T23:10:14.653+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.654723+0000) 2022-01-31T23:10:15.100 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:14 smithi167 conmon[49112]: debug 2022-01-31T23:10:14.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:14.857521+0000) 2022-01-31T23:10:15.100 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:15 smithi167 conmon[60316]: debug 2022-01-31T23:10:15.034+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:15.035452+0000) 2022-01-31T23:10:15.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:15 smithi171 conmon[41853]: debug 2022-01-31T23:10:15.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:15.224559+0000) 2022-01-31T23:10:15.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:15 smithi171 conmon[46715]: debug 2022-01-31T23:10:15.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:15.023115+0000) 2022-01-31T23:10:15.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:15 smithi167 conmon[54076]: debug 2022-01-31T23:10:15.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:15.277733+0000) 2022-01-31T23:10:16.004 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:15 smithi171 conmon[51620]: debug 2022-01-31T23:10:15.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:15.654923+0000) 2022-01-31T23:10:16.103 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:15 smithi167 conmon[49112]: debug 2022-01-31T23:10:15.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:15.857663+0000) 2022-01-31T23:10:16.202 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:16 smithi167 conmon[60316]: debug 2022-01-31T23:10:16.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:16.035610+0000) 2022-01-31T23:10:16.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:16 smithi171 conmon[41853]: debug 2022-01-31T23:10:16.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:16.224770+0000) 2022-01-31T23:10:16.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:16 smithi171 conmon[46715]: debug 2022-01-31T23:10:16.022+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:16.023266+0000) 2022-01-31T23:10:16.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:16 smithi167 conmon[54076]: debug 2022-01-31T23:10:16.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:16.277863+0000) 2022-01-31T23:10:17.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:16 smithi171 conmon[51620]: debug 2022-01-31T23:10:16.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:16.655127+0000) 2022-01-31T23:10:17.107 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:16 smithi167 conmon[49112]: debug 2022-01-31T23:10:16.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:16.857856+0000) 2022-01-31T23:10:17.107 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:17 smithi167 conmon[60316]: debug 2022-01-31T23:10:17.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:17.035791+0000) 2022-01-31T23:10:17.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:17 smithi171 conmon[41853]: debug 2022-01-31T23:10:17.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:17.224914+0000) 2022-01-31T23:10:17.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:17 smithi171 conmon[46715]: debug 2022-01-31T23:10:17.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:17.023480+0000) 2022-01-31T23:10:17.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:17 smithi167 conmon[54076]: debug 2022-01-31T23:10:17.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:17.278086+0000) 2022-01-31T23:10:18.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:17 smithi171 conmon[51620]: debug 2022-01-31T23:10:17.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:17.655333+0000) 2022-01-31T23:10:18.110 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:17 smithi167 conmon[49112]: debug 2022-01-31T23:10:17.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:17.858064+0000) 2022-01-31T23:10:18.111 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:18 smithi167 conmon[60316]: debug 2022-01-31T23:10:18.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:18.035934+0000) 2022-01-31T23:10:18.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:18 smithi171 conmon[46715]: debug 2022-01-31T23:10:18.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:18.023671+0000) 2022-01-31T23:10:18.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:18 smithi171 conmon[41853]: debug 2022-01-31T23:10:18.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:18.225092+0000) 2022-01-31T23:10:18.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:18 smithi167 conmon[54076]: debug 2022-01-31T23:10:18.277+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:18.278263+0000) 2022-01-31T23:10:19.005 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:18 smithi171 conmon[51620]: debug 2022-01-31T23:10:18.654+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:18.655516+0000) 2022-01-31T23:10:19.114 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:18 smithi167 conmon[49112]: debug 2022-01-31T23:10:18.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:18.858187+0000) 2022-01-31T23:10:19.115 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:19 smithi167 conmon[60316]: debug 2022-01-31T23:10:19.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.036060+0000) 2022-01-31T23:10:19.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[41853]: debug 2022-01-31T23:10:19.224+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.225294+0000) 2022-01-31T23:10:19.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[46715]: debug 2022-01-31T23:10:19.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.023878+0000) 2022-01-31T23:10:19.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:19 smithi167 conmon[54076]: debug 2022-01-31T23:10:19.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.278446+0000) 2022-01-31T23:10:19.839 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[35325]: debug 2022-01-31T23:10:19.620+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268286 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:19.840 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[41853]: debug 2022-01-31T23:10:19.591+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.591871+0000) 2022-01-31T23:10:19.841 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[46715]: debug 2022-01-31T23:10:19.591+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.592057+0000) 2022-01-31T23:10:19.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[51620]: debug 2022-01-31T23:10:19.592+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.593421+0000) 2022-01-31T23:10:19.841 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:19 smithi171 conmon[51620]: debug 2022-01-31T23:10:19.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.655694+0000) 2022-01-31T23:10:19.842 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:19 smithi167 conmon[49112]: debug 2022-01-31T23:10:19.592+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.593355+0000) 2022-01-31T23:10:19.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:19 smithi167 conmon[54076]: debug 2022-01-31T23:10:19.591+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.592355+0000) 2022-01-31T23:10:19.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:19 smithi167 conmon[60316]: debug 2022-01-31T23:10:19.592+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.593143+0000) 2022-01-31T23:10:20.117 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:19 smithi167 conmon[49112]: debug 2022-01-31T23:10:19.857+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:19.858355+0000) 2022-01-31T23:10:20.118 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:20 smithi167 conmon[60316]: debug 2022-01-31T23:10:20.035+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:20.036224+0000) 2022-01-31T23:10:20.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:20 smithi171 conmon[41853]: debug 2022-01-31T23:10:20.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:20.225481+0000) 2022-01-31T23:10:20.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:20 smithi171 conmon[46715]: debug 2022-01-31T23:10:20.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:20.024074+0000) 2022-01-31T23:10:20.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:20 smithi167 conmon[54076]: debug 2022-01-31T23:10:20.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:20.278601+0000) 2022-01-31T23:10:21.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:20 smithi171 conmon[51620]: debug 2022-01-31T23:10:20.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:20.655876+0000) 2022-01-31T23:10:21.122 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:20 smithi167 conmon[49112]: debug 2022-01-31T23:10:20.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:20.858521+0000) 2022-01-31T23:10:21.122 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:21 smithi167 conmon[60316]: debug 2022-01-31T23:10:21.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:21.036401+0000) 2022-01-31T23:10:21.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:21 smithi171 conmon[41853]: debug 2022-01-31T23:10:21.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:21.225634+0000) 2022-01-31T23:10:21.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:21 smithi171 conmon[46715]: debug 2022-01-31T23:10:21.023+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:21.024249+0000) 2022-01-31T23:10:21.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:21 smithi167 conmon[54076]: debug 2022-01-31T23:10:21.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:21.278795+0000) 2022-01-31T23:10:22.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:21 smithi171 conmon[51620]: debug 2022-01-31T23:10:21.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:21.656090+0000) 2022-01-31T23:10:22.126 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:21 smithi167 conmon[49112]: debug 2022-01-31T23:10:21.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:21.858724+0000) 2022-01-31T23:10:22.127 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:22 smithi167 conmon[60316]: debug 2022-01-31T23:10:22.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:22.036609+0000) 2022-01-31T23:10:22.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:22 smithi171 conmon[41853]: debug 2022-01-31T23:10:22.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:22.225817+0000) 2022-01-31T23:10:22.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:22 smithi171 conmon[46715]: debug 2022-01-31T23:10:22.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:22.024453+0000) 2022-01-31T23:10:22.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:22 smithi167 conmon[54076]: debug 2022-01-31T23:10:22.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:22.278974+0000) 2022-01-31T23:10:23.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:22 smithi171 conmon[51620]: debug 2022-01-31T23:10:22.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:22.656298+0000) 2022-01-31T23:10:23.129 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:22 smithi167 conmon[49112]: debug 2022-01-31T23:10:22.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:22.858887+0000) 2022-01-31T23:10:23.130 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:23 smithi167 conmon[60316]: debug 2022-01-31T23:10:23.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:23.036763+0000) 2022-01-31T23:10:23.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:23 smithi171 conmon[41853]: debug 2022-01-31T23:10:23.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:23.225997+0000) 2022-01-31T23:10:23.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:23 smithi171 conmon[46715]: debug 2022-01-31T23:10:23.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:23.024617+0000) 2022-01-31T23:10:23.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:23 smithi167 conmon[54076]: debug 2022-01-31T23:10:23.278+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:23.279181+0000) 2022-01-31T23:10:23.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:23 smithi171 conmon[51620]: debug 2022-01-31T23:10:23.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:23.656468+0000) 2022-01-31T23:10:24.134 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:23 smithi167 conmon[49112]: debug 2022-01-31T23:10:23.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:23.859044+0000) 2022-01-31T23:10:24.134 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:24 smithi167 conmon[60316]: debug 2022-01-31T23:10:24.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.036891+0000) 2022-01-31T23:10:24.207 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:23 smithi171 conmon[35325]: debug 2022-01-31T23:10:23.974+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:10:24.208 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[46715]: debug 2022-01-31T23:10:24.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.024824+0000) 2022-01-31T23:10:24.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:24 smithi167 conmon[54076]: debug 2022-01-31T23:10:24.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.279375+0000) 2022-01-31T23:10:24.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[41853]: debug 2022-01-31T23:10:24.225+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.226146+0000) 2022-01-31T23:10:24.840 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:24 smithi167 conmon[49112]: debug 2022-01-31T23:10:24.623+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.624432+0000) 2022-01-31T23:10:24.841 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:24 smithi167 conmon[54076]: debug 2022-01-31T23:10:24.623+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.623715+0000) 2022-01-31T23:10:24.842 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:24 smithi167 conmon[60316]: debug 2022-01-31T23:10:24.623+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.624139+0000) 2022-01-31T23:10:25.006 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[41853]: debug 2022-01-31T23:10:24.622+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.623490+0000) 2022-01-31T23:10:25.007 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[46715]: debug 2022-01-31T23:10:24.622+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.623231+0000) 2022-01-31T23:10:25.008 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[35325]: debug 2022-01-31T23:10:24.652+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268400 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:25.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[51620]: debug 2022-01-31T23:10:24.624+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.624960+0000) 2022-01-31T23:10:25.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:24 smithi171 conmon[51620]: debug 2022-01-31T23:10:24.655+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.656632+0000) 2022-01-31T23:10:25.137 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:24 smithi167 conmon[49112]: debug 2022-01-31T23:10:24.858+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:24.859227+0000) 2022-01-31T23:10:25.137 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:25 smithi167 conmon[60316]: debug 2022-01-31T23:10:25.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:25.037074+0000) 2022-01-31T23:10:25.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:25 smithi171 conmon[41853]: debug 2022-01-31T23:10:25.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:25.226343+0000) 2022-01-31T23:10:25.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:25 smithi171 conmon[46715]: debug 2022-01-31T23:10:25.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:25.025029+0000) 2022-01-31T23:10:25.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:25 smithi167 conmon[54076]: debug 2022-01-31T23:10:25.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:25.279573+0000) 2022-01-31T23:10:26.006 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:25 smithi171 conmon[51620]: debug 2022-01-31T23:10:25.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:25.656779+0000) 2022-01-31T23:10:26.141 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:25 smithi167 conmon[49112]: debug 2022-01-31T23:10:25.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:25.859396+0000) 2022-01-31T23:10:26.141 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:26 smithi167 conmon[60316]: debug 2022-01-31T23:10:26.036+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:26.037246+0000) 2022-01-31T23:10:26.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:26 smithi171 conmon[46715]: debug 2022-01-31T23:10:26.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:26.025236+0000) 2022-01-31T23:10:26.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:26 smithi171 conmon[41853]: debug 2022-01-31T23:10:26.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:26.226531+0000) 2022-01-31T23:10:26.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:26 smithi167 conmon[54076]: debug 2022-01-31T23:10:26.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:26.279735+0000) 2022-01-31T23:10:27.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:26 smithi171 conmon[51620]: debug 2022-01-31T23:10:26.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:26.656980+0000) 2022-01-31T23:10:27.145 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:26 smithi167 conmon[49112]: debug 2022-01-31T23:10:26.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:26.859579+0000) 2022-01-31T23:10:27.145 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:27 smithi167 conmon[60316]: debug 2022-01-31T23:10:27.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:27.037439+0000) 2022-01-31T23:10:27.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:27 smithi171 conmon[46715]: debug 2022-01-31T23:10:27.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:27.025469+0000) 2022-01-31T23:10:27.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:27 smithi171 conmon[41853]: debug 2022-01-31T23:10:27.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:27.226704+0000) 2022-01-31T23:10:27.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:27 smithi167 conmon[54076]: debug 2022-01-31T23:10:27.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:27.279902+0000) 2022-01-31T23:10:28.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:27 smithi171 conmon[51620]: debug 2022-01-31T23:10:27.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:27.657117+0000) 2022-01-31T23:10:28.149 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:27 smithi167 conmon[49112]: debug 2022-01-31T23:10:27.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:27.859707+0000) 2022-01-31T23:10:28.149 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:28 smithi167 conmon[60316]: debug 2022-01-31T23:10:28.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:28.037598+0000) 2022-01-31T23:10:28.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:28 smithi171 conmon[46715]: debug 2022-01-31T23:10:28.024+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:28.025683+0000) 2022-01-31T23:10:28.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:28 smithi171 conmon[41853]: debug 2022-01-31T23:10:28.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:28.226925+0000) 2022-01-31T23:10:28.401 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:28 smithi167 conmon[54076]: debug 2022-01-31T23:10:28.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:28.280084+0000) 2022-01-31T23:10:29.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:28 smithi171 conmon[51620]: debug 2022-01-31T23:10:28.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:28.657310+0000) 2022-01-31T23:10:29.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:29 smithi167 conmon[60316]: debug 2022-01-31T23:10:29.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.037759+0000) 2022-01-31T23:10:29.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:28 smithi167 conmon[49112]: debug 2022-01-31T23:10:28.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:28.859865+0000) 2022-01-31T23:10:29.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[46715]: debug 2022-01-31T23:10:29.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.025917+0000) 2022-01-31T23:10:29.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[41853]: debug 2022-01-31T23:10:29.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.227130+0000) 2022-01-31T23:10:29.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:29 smithi167 conmon[54076]: debug 2022-01-31T23:10:29.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.280259+0000) 2022-01-31T23:10:29.841 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:29 smithi167 conmon[49112]: debug 2022-01-31T23:10:29.655+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.656203+0000) 2022-01-31T23:10:29.842 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:29 smithi167 conmon[54076]: debug 2022-01-31T23:10:29.656+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.656522+0000) 2022-01-31T23:10:29.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:29 smithi167 conmon[60316]: debug 2022-01-31T23:10:29.655+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.655595+0000) 2022-01-31T23:10:30.007 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[35325]: debug 2022-01-31T23:10:29.683+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268510 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:30.008 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[41853]: debug 2022-01-31T23:10:29.654+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.654802+0000) 2022-01-31T23:10:30.009 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[46715]: debug 2022-01-31T23:10:29.654+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.655060+0000) 2022-01-31T23:10:30.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[51620]: debug 2022-01-31T23:10:29.654+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.655716+0000) 2022-01-31T23:10:30.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:29 smithi171 conmon[51620]: debug 2022-01-31T23:10:29.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.657456+0000) 2022-01-31T23:10:30.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:29 smithi167 conmon[49112]: debug 2022-01-31T23:10:29.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:29.860094+0000) 2022-01-31T23:10:30.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:30 smithi167 conmon[60316]: debug 2022-01-31T23:10:30.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:30.037885+0000) 2022-01-31T23:10:30.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:30 smithi171 conmon[46715]: debug 2022-01-31T23:10:30.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:30.026073+0000) 2022-01-31T23:10:30.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:30 smithi171 conmon[41853]: debug 2022-01-31T23:10:30.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:30.227317+0000) 2022-01-31T23:10:30.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:30 smithi167 conmon[54076]: debug 2022-01-31T23:10:30.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:30.280439+0000) 2022-01-31T23:10:31.007 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:30 smithi171 conmon[51620]: debug 2022-01-31T23:10:30.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:30.657622+0000) 2022-01-31T23:10:31.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:30 smithi167 conmon[49112]: debug 2022-01-31T23:10:30.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:30.860271+0000) 2022-01-31T23:10:31.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:31 smithi167 conmon[60316]: debug 2022-01-31T23:10:31.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:31.038035+0000) 2022-01-31T23:10:31.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:31 smithi171 conmon[41853]: debug 2022-01-31T23:10:31.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:31.227473+0000) 2022-01-31T23:10:31.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:31 smithi171 conmon[46715]: debug 2022-01-31T23:10:31.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:31.026241+0000) 2022-01-31T23:10:31.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:31 smithi167 conmon[54076]: debug 2022-01-31T23:10:31.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:31.280598+0000) 2022-01-31T23:10:32.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:31 smithi171 conmon[51620]: debug 2022-01-31T23:10:31.656+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:31.657837+0000) 2022-01-31T23:10:32.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:31 smithi167 conmon[49112]: debug 2022-01-31T23:10:31.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:31.860452+0000) 2022-01-31T23:10:32.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:32 smithi167 conmon[60316]: debug 2022-01-31T23:10:32.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:32.038236+0000) 2022-01-31T23:10:32.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:32 smithi171 conmon[41853]: debug 2022-01-31T23:10:32.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:32.227656+0000) 2022-01-31T23:10:32.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:32 smithi171 conmon[46715]: debug 2022-01-31T23:10:32.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:32.026432+0000) 2022-01-31T23:10:32.512 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:32 smithi167 conmon[54076]: debug 2022-01-31T23:10:32.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:32.280775+0000) 2022-01-31T23:10:33.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:32 smithi171 conmon[51620]: debug 2022-01-31T23:10:32.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:32.658071+0000) 2022-01-31T23:10:33.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:32 smithi167 conmon[49112]: debug 2022-01-31T23:10:32.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:32.860686+0000) 2022-01-31T23:10:33.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:33 smithi167 conmon[60316]: debug 2022-01-31T23:10:33.038+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:33.038400+0000) 2022-01-31T23:10:33.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:33 smithi171 conmon[41853]: debug 2022-01-31T23:10:33.226+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:33.227852+0000) 2022-01-31T23:10:33.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:33 smithi171 conmon[46715]: debug 2022-01-31T23:10:33.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:33.026644+0000) 2022-01-31T23:10:33.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:33 smithi167 conmon[54076]: debug 2022-01-31T23:10:33.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:33.280934+0000) 2022-01-31T23:10:34.008 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:33 smithi171 conmon[51620]: debug 2022-01-31T23:10:33.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:33.658236+0000) 2022-01-31T23:10:34.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:33 smithi167 conmon[49112]: debug 2022-01-31T23:10:33.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:33.860868+0000) 2022-01-31T23:10:34.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[60316]: debug 2022-01-31T23:10:34.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.038579+0000) 2022-01-31T23:10:34.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[41853]: debug 2022-01-31T23:10:34.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.228042+0000) 2022-01-31T23:10:34.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[46715]: debug 2022-01-31T23:10:34.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.026841+0000) 2022-01-31T23:10:34.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[54076]: debug 2022-01-31T23:10:34.279+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.281095+0000) 2022-01-31T23:10:34.843 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[54076]: debug 2022-01-31T23:10:34.686+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.687669+0000) 2022-01-31T23:10:34.843 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[60316]: debug 2022-01-31T23:10:34.685+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.687340+0000) 2022-01-31T23:10:34.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[49112]: debug 2022-01-31T23:10:34.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[49112]: 2022-01-31T23:10:34.686+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.687951+0000) 2022-01-31T23:10:35.008 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[35325]: debug 2022-01-31T23:10:34.718+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268623 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:35.009 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[41853]: debug 2022-01-31T23:10:34.683+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.687445+0000) 2022-01-31T23:10:35.010 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[46715]: debug 2022-01-31T23:10:34.683+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.687760+0000) 2022-01-31T23:10:35.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[51620]: debug 2022-01-31T23:10:34.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.658417+0000) 2022-01-31T23:10:35.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:34 smithi171 conmon[51620]: debug 2022-01-31T23:10:34.683+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.686364+0000) 2022-01-31T23:10:35.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:34 smithi167 conmon[49112]: debug 2022-01-31T23:10:34.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:34.861032+0000) 2022-01-31T23:10:35.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:35 smithi167 conmon[60316]: debug 2022-01-31T23:10:35.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:35.038755+0000) 2022-01-31T23:10:35.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:35 smithi171 conmon[41853]: debug 2022-01-31T23:10:35.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:35.228252+0000) 2022-01-31T23:10:35.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:35 smithi171 conmon[46715]: debug 2022-01-31T23:10:35.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:35.027058+0000) 2022-01-31T23:10:35.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:35 smithi167 conmon[54076]: debug 2022-01-31T23:10:35.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:35.281294+0000) 2022-01-31T23:10:36.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:35 smithi171 conmon[51620]: debug 2022-01-31T23:10:35.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:35.658634+0000) 2022-01-31T23:10:36.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:35 smithi167 conmon[49112]: debug 2022-01-31T23:10:35.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:35.861203+0000) 2022-01-31T23:10:36.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:36 smithi167 conmon[60316]: debug 2022-01-31T23:10:36.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:36.038857+0000) 2022-01-31T23:10:36.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:36 smithi171 conmon[41853]: debug 2022-01-31T23:10:36.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:36.228438+0000) 2022-01-31T23:10:36.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:36 smithi171 conmon[46715]: debug 2022-01-31T23:10:36.025+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:36.027233+0000) 2022-01-31T23:10:36.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:36 smithi167 conmon[54076]: debug 2022-01-31T23:10:36.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:36.281456+0000) 2022-01-31T23:10:37.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:36 smithi171 conmon[51620]: debug 2022-01-31T23:10:36.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:36.658822+0000) 2022-01-31T23:10:37.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:37 smithi167 conmon[60316]: debug 2022-01-31T23:10:37.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:37.039046+0000) 2022-01-31T23:10:37.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:36 smithi167 conmon[49112]: debug 2022-01-31T23:10:36.859+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:36.861413+0000) 2022-01-31T23:10:37.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:37 smithi171 conmon[41853]: debug 2022-01-31T23:10:37.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:37.228558+0000) 2022-01-31T23:10:37.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:37 smithi171 conmon[46715]: debug 2022-01-31T23:10:37.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:37.027426+0000) 2022-01-31T23:10:37.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:37 smithi167 conmon[54076]: debug 2022-01-31T23:10:37.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:37.281671+0000) 2022-01-31T23:10:38.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:37 smithi171 conmon[51620]: debug 2022-01-31T23:10:37.657+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:37.659004+0000) 2022-01-31T23:10:38.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:37 smithi167 conmon[49112]: debug 2022-01-31T23:10:37.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:37.861603+0000) 2022-01-31T23:10:38.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:38 smithi167 conmon[60316]: debug 2022-01-31T23:10:38.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:38.039250+0000) 2022-01-31T23:10:38.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:38 smithi171 conmon[46715]: debug 2022-01-31T23:10:38.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:38.027561+0000) 2022-01-31T23:10:38.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:38 smithi171 conmon[41853]: debug 2022-01-31T23:10:38.227+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:38.228733+0000) 2022-01-31T23:10:38.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:38 smithi167 conmon[54076]: debug 2022-01-31T23:10:38.280+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:38.281837+0000) 2022-01-31T23:10:38.957 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:38 smithi171 conmon[51620]: debug 2022-01-31T23:10:38.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:38.659169+0000) 2022-01-31T23:10:39.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:38 smithi167 conmon[49112]: debug 2022-01-31T23:10:38.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:38.861758+0000) 2022-01-31T23:10:39.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:39 smithi167 conmon[60316]: debug 2022-01-31T23:10:39.037+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.039439+0000) 2022-01-31T23:10:39.210 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:38 smithi171 conmon[35325]: debug 2022-01-31T23:10:38.974+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:10:39.211 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[46715]: debug 2022-01-31T23:10:39.026+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.027772+0000) 2022-01-31T23:10:39.513 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:39 smithi167 conmon[54076]: debug 2022-01-31T23:10:39.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.281995+0000) 2022-01-31T23:10:39.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[41853]: debug 2022-01-31T23:10:39.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.228936+0000) 2022-01-31T23:10:39.843 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:39 smithi167 conmon[49112]: debug 2022-01-31T23:10:39.721+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.723062+0000) 2022-01-31T23:10:39.844 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:39 smithi167 conmon[54076]: debug 2022-01-31T23:10:39.721+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.723003+0000) 2022-01-31T23:10:39.844 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:39 smithi167 conmon[60316]: debug 2022-01-31T23:10:39.721+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.722799+0000) 2022-01-31T23:10:40.010 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[35325]: debug 2022-01-31T23:10:39.749+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268733 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:40.010 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[41853]: debug 2022-01-31T23:10:39.722+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.723831+0000) 2022-01-31T23:10:40.011 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[46715]: debug 2022-01-31T23:10:39.721+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.723392+0000) 2022-01-31T23:10:40.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[51620]: debug 2022-01-31T23:10:39.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.659358+0000) 2022-01-31T23:10:40.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:39 smithi171 conmon[51620]: debug 2022-01-31T23:10:39.720+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.722630+0000) 2022-01-31T23:10:40.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:39 smithi167 conmon[49112]: debug 2022-01-31T23:10:39.860+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:39.861920+0000) 2022-01-31T23:10:40.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:40 smithi167 conmon[60316]: debug 2022-01-31T23:10:40.038+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:40.039591+0000) 2022-01-31T23:10:40.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:40 smithi171 conmon[41853]: debug 2022-01-31T23:10:40.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:40.229107+0000) 2022-01-31T23:10:40.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:40 smithi171 conmon[46715]: debug 2022-01-31T23:10:40.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:40.027983+0000) 2022-01-31T23:10:40.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:40 smithi167 conmon[54076]: debug 2022-01-31T23:10:40.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:40.282196+0000) 2022-01-31T23:10:41.009 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:40 smithi171 conmon[51620]: debug 2022-01-31T23:10:40.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:40.659558+0000) 2022-01-31T23:10:41.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:40 smithi167 conmon[49112]: debug 2022-01-31T23:10:40.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:40.862055+0000) 2022-01-31T23:10:41.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:41 smithi167 conmon[60316]: debug 2022-01-31T23:10:41.038+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:41.039703+0000) 2022-01-31T23:10:41.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:41 smithi171 conmon[41853]: debug 2022-01-31T23:10:41.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:41.229252+0000) 2022-01-31T23:10:41.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:41 smithi171 conmon[46715]: debug 2022-01-31T23:10:41.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:41.028140+0000) 2022-01-31T23:10:41.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:41 smithi167 conmon[54076]: debug 2022-01-31T23:10:41.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:41.282394+0000) 2022-01-31T23:10:42.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:41 smithi171 conmon[51620]: debug 2022-01-31T23:10:41.658+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:41.659723+0000) 2022-01-31T23:10:42.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:41 smithi167 conmon[49112]: debug 2022-01-31T23:10:41.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:41.862250+0000) 2022-01-31T23:10:42.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:42 smithi167 conmon[60316]: debug 2022-01-31T23:10:42.038+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:42.039847+0000) 2022-01-31T23:10:42.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:42 smithi171 conmon[46715]: debug 2022-01-31T23:10:42.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:42.028320+0000) 2022-01-31T23:10:42.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:42 smithi171 conmon[41853]: debug 2022-01-31T23:10:42.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:42.229405+0000) 2022-01-31T23:10:42.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:42 smithi167 conmon[54076]: debug 2022-01-31T23:10:42.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:42.282583+0000) 2022-01-31T23:10:43.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:42 smithi171 conmon[51620]: debug 2022-01-31T23:10:42.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:42.659947+0000) 2022-01-31T23:10:43.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:43 smithi167 conmon[60316]: debug 2022-01-31T23:10:43.038+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:43.040051+0000) 2022-01-31T23:10:43.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:42 smithi167 conmon[49112]: debug 2022-01-31T23:10:42.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:42.862439+0000) 2022-01-31T23:10:43.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:43 smithi171 conmon[46715]: debug 2022-01-31T23:10:43.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:43.028510+0000) 2022-01-31T23:10:43.340 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:43 smithi171 conmon[41853]: debug 2022-01-31T23:10:43.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:43.229534+0000) 2022-01-31T23:10:43.514 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:43 smithi167 conmon[54076]: debug 2022-01-31T23:10:43.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:43.282770+0000) 2022-01-31T23:10:44.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:43 smithi171 conmon[51620]: debug 2022-01-31T23:10:43.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:43.660147+0000) 2022-01-31T23:10:44.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:44 smithi167 conmon[60316]: debug 2022-01-31T23:10:44.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.040209+0000) 2022-01-31T23:10:44.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:43 smithi167 conmon[49112]: debug 2022-01-31T23:10:43.861+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:43.862621+0000) 2022-01-31T23:10:44.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[46715]: debug 2022-01-31T23:10:44.027+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.028698+0000) 2022-01-31T23:10:44.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[41853]: debug 2022-01-31T23:10:44.228+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.229670+0000) 2022-01-31T23:10:44.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:44 smithi167 conmon[54076]: debug 2022-01-31T23:10:44.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.282983+0000) 2022-01-31T23:10:44.844 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:44 smithi167 conmon[49112]: debug 2022-01-31T23:10:44.752+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.753910+0000) 2022-01-31T23:10:44.845 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:44 smithi167 conmon[54076]: debug 2022-01-31T23:10:44.752+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.753525+0000) 2022-01-31T23:10:44.845 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:44 smithi167 conmon[60316]: debug 2022-01-31T23:10:44.753+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.755192+0000) 2022-01-31T23:10:45.010 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[35325]: debug 2022-01-31T23:10:44.781+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268847 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:45.011 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[41853]: debug 2022-01-31T23:10:44.753+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.753997+0000) 2022-01-31T23:10:45.012 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[46715]: debug 2022-01-31T23:10:44.753+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.754423+0000) 2022-01-31T23:10:45.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[51620]: debug 2022-01-31T23:10:44.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.660295+0000) 2022-01-31T23:10:45.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:44 smithi171 conmon[51620]: debug 2022-01-31T23:10:44.752+0000 7f84f2e2c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.753735+0000) 2022-01-31T23:10:45.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:45 smithi167 conmon[60316]: debug 2022-01-31T23:10:45.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:45.040456+0000) 2022-01-31T23:10:45.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:44 smithi167 conmon[49112]: debug 2022-01-31T23:10:44.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:44.862839+0000) 2022-01-31T23:10:45.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:45 smithi171 conmon[41853]: debug 2022-01-31T23:10:45.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:45.229830+0000) 2022-01-31T23:10:45.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:45 smithi171 conmon[46715]: debug 2022-01-31T23:10:45.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:45.028891+0000) 2022-01-31T23:10:45.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:45 smithi167 conmon[54076]: debug 2022-01-31T23:10:45.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:45.283170+0000) 2022-01-31T23:10:46.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:45 smithi171 conmon[51620]: debug 2022-01-31T23:10:45.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:45.660470+0000) 2022-01-31T23:10:46.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:45 smithi167 conmon[49112]: debug 2022-01-31T23:10:45.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:45.862992+0000) 2022-01-31T23:10:46.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:46 smithi167 conmon[60316]: debug 2022-01-31T23:10:46.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:46.040613+0000) 2022-01-31T23:10:46.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:46 smithi171 conmon[41853]: debug 2022-01-31T23:10:46.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:46.229983+0000) 2022-01-31T23:10:46.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:46 smithi171 conmon[46715]: debug 2022-01-31T23:10:46.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:46.028994+0000) 2022-01-31T23:10:46.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:46 smithi167 conmon[54076]: debug 2022-01-31T23:10:46.281+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:46.283343+0000) 2022-01-31T23:10:47.010 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:46 smithi171 conmon[51620]: debug 2022-01-31T23:10:46.659+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:46.660641+0000) 2022-01-31T23:10:47.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:47 smithi167 conmon[60316]: debug 2022-01-31T23:10:47.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:47.040776+0000) 2022-01-31T23:10:47.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:46 smithi167 conmon[49112]: debug 2022-01-31T23:10:46.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:46.863152+0000) 2022-01-31T23:10:47.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:47 smithi171 conmon[41853]: debug 2022-01-31T23:10:47.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:47.230136+0000) 2022-01-31T23:10:47.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:47 smithi171 conmon[46715]: debug 2022-01-31T23:10:47.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:47.029184+0000) 2022-01-31T23:10:47.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:47 smithi167 conmon[54076]: debug 2022-01-31T23:10:47.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:47.283552+0000) 2022-01-31T23:10:48.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:47 smithi171 conmon[51620]: debug 2022-01-31T23:10:47.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:47.660843+0000) 2022-01-31T23:10:48.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:48 smithi167 conmon[60316]: debug 2022-01-31T23:10:48.039+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:48.040964+0000) 2022-01-31T23:10:48.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:47 smithi167 conmon[49112]: debug 2022-01-31T23:10:47.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:47.863352+0000) 2022-01-31T23:10:48.338 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:48 smithi171 conmon[41853]: debug 2022-01-31T23:10:48.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:48.230319+0000) 2022-01-31T23:10:48.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:48 smithi171 conmon[46715]: debug 2022-01-31T23:10:48.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:48.029367+0000) 2022-01-31T23:10:48.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:48 smithi167 conmon[54076]: debug 2022-01-31T23:10:48.282+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:48.283738+0000) 2022-01-31T23:10:49.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:48 smithi171 conmon[51620]: debug 2022-01-31T23:10:48.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:48.661075+0000) 2022-01-31T23:10:49.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:48 smithi167 conmon[49112]: debug 2022-01-31T23:10:48.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:48.863557+0000) 2022-01-31T23:10:49.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:49 smithi167 conmon[60316]: debug 2022-01-31T23:10:49.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.041118+0000) 2022-01-31T23:10:49.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[41853]: debug 2022-01-31T23:10:49.229+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.230500+0000) 2022-01-31T23:10:49.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[46715]: debug 2022-01-31T23:10:49.028+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.029578+0000) 2022-01-31T23:10:49.515 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:49 smithi167 conmon[54076]: debug 2022-01-31T23:10:49.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.283934+0000) 2022-01-31T23:10:49.766 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:49 smithi167 conmon[49112]: debug 2022-01-31T23:10:49.782+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.784087+0000) 2022-01-31T23:10:49.766 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:49 smithi167 conmon[54076]: debug 2022-01-31T23:10:49.783+0000 7f5465e1c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.784511+0000) 2022-01-31T23:10:50.011 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[35325]: debug 2022-01-31T23:10:49.811+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 268960 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:50.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[41853]: debug 2022-01-31T23:10:49.784+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.785090+0000) 2022-01-31T23:10:50.012 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[46715]: debug 2022-01-31T23:10:49.784+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.785191+0000) 2022-01-31T23:10:50.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[51620]: debug 2022-01-31T23:10:49.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.661319+0000) 2022-01-31T23:10:50.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:49 smithi171 conmon[51620]: debug 2022-01-31T23:10:49.783+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.784342+0000) 2022-01-31T23:10:50.022 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:49 smithi167 conmon[49112]: debug 2022-01-31T23:10:49.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.863744+0000) 2022-01-31T23:10:50.023 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:49 smithi167 conmon[60316]: debug 2022-01-31T23:10:49.784+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:49.785514+0000) 2022-01-31T23:10:50.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:50 smithi171 conmon[41853]: debug 2022-01-31T23:10:50.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:50.230649+0000) 2022-01-31T23:10:50.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:50 smithi171 conmon[46715]: debug 2022-01-31T23:10:50.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:50.029779+0000) 2022-01-31T23:10:50.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:50 smithi167 conmon[54076]: debug 2022-01-31T23:10:50.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:50.284123+0000) 2022-01-31T23:10:50.401 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:50 smithi167 conmon[60316]: debug 2022-01-31T23:10:50.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:50.041231+0000) 2022-01-31T23:10:51.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:50 smithi171 conmon[51620]: debug 2022-01-31T23:10:50.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:50.661513+0000) 2022-01-31T23:10:51.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:50 smithi167 conmon[49112]: debug 2022-01-31T23:10:50.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:50.863883+0000) 2022-01-31T23:10:51.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:51 smithi167 conmon[60316]: debug 2022-01-31T23:10:51.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:51.041384+0000) 2022-01-31T23:10:51.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:51 smithi171 conmon[41853]: debug 2022-01-31T23:10:51.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:51.230798+0000) 2022-01-31T23:10:51.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:51 smithi171 conmon[46715]: debug 2022-01-31T23:10:51.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:51.029886+0000) 2022-01-31T23:10:51.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:51 smithi167 conmon[54076]: debug 2022-01-31T23:10:51.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:51.284307+0000) 2022-01-31T23:10:52.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:51 smithi171 conmon[51620]: debug 2022-01-31T23:10:51.660+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:51.661699+0000) 2022-01-31T23:10:52.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:51 smithi167 conmon[49112]: debug 2022-01-31T23:10:51.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:51.864037+0000) 2022-01-31T23:10:52.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:52 smithi167 conmon[60316]: debug 2022-01-31T23:10:52.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:52.041542+0000) 2022-01-31T23:10:52.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:52 smithi171 conmon[46715]: debug 2022-01-31T23:10:52.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:52.030048+0000) 2022-01-31T23:10:52.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:52 smithi171 conmon[41853]: debug 2022-01-31T23:10:52.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:52.231024+0000) 2022-01-31T23:10:52.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:52 smithi167 conmon[54076]: debug 2022-01-31T23:10:52.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:52.284495+0000) 2022-01-31T23:10:53.011 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:52 smithi171 conmon[51620]: debug 2022-01-31T23:10:52.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:52.661930+0000) 2022-01-31T23:10:53.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:52 smithi167 conmon[49112]: debug 2022-01-31T23:10:52.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:52.864197+0000) 2022-01-31T23:10:53.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:53 smithi167 conmon[60316]: debug 2022-01-31T23:10:53.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:53.041653+0000) 2022-01-31T23:10:53.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:53 smithi171 conmon[41853]: debug 2022-01-31T23:10:53.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:53.231196+0000) 2022-01-31T23:10:53.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:53 smithi171 conmon[46715]: debug 2022-01-31T23:10:53.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:53.030239+0000) 2022-01-31T23:10:53.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:53 smithi167 conmon[54076]: debug 2022-01-31T23:10:53.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:53.284646+0000) 2022-01-31T23:10:53.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:53 smithi171 conmon[51620]: debug 2022-01-31T23:10:53.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:53.662137+0000) 2022-01-31T23:10:54.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:53 smithi167 conmon[49112]: debug 2022-01-31T23:10:53.862+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:53.864403+0000) 2022-01-31T23:10:54.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:54 smithi167 conmon[60316]: debug 2022-01-31T23:10:54.040+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.041783+0000) 2022-01-31T23:10:54.212 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:53 smithi171 conmon[35325]: debug 2022-01-31T23:10:53.975+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:10:54.213 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[46715]: debug 2022-01-31T23:10:54.029+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.030410+0000) 2022-01-31T23:10:54.516 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:54 smithi167 conmon[54076]: debug 2022-01-31T23:10:54.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.284849+0000) 2022-01-31T23:10:54.588 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[41853]: debug 2022-01-31T23:10:54.230+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.231345+0000) 2022-01-31T23:10:55.012 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[35325]: debug 2022-01-31T23:10:54.842+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 269095 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:10:55.012 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[41853]: debug 2022-01-31T23:10:54.815+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.816160+0000) 2022-01-31T23:10:55.013 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[46715]: debug 2022-01-31T23:10:54.814+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.815298+0000) 2022-01-31T23:10:55.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[51620]: debug 2022-01-31T23:10:54.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.662380+0000) 2022-01-31T23:10:55.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:54 smithi171 conmon[51620]: debug 2022-01-31T23:10:54.814+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.815196+0000) 2022-01-31T23:10:55.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:54 smithi167 conmon[49112]: debug 2022-01-31T23:10:54.812+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.814381+0000) 2022-01-31T23:10:55.070 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:54 smithi167 conmon[49112]: debug 2022-01-31T23:10:54.863+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.864601+0000) 2022-01-31T23:10:55.071 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:54 smithi167 conmon[54076]: debug 2022-01-31T23:10:54.813+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.814680+0000) 2022-01-31T23:10:55.071 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:54 smithi167 conmon[60316]: debug 2022-01-31T23:10:54.814+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:54.815795+0000) 2022-01-31T23:10:55.072 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:55 smithi167 conmon[60316]: debug 2022-01-31T23:10:55.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:55.041929+0000) 2022-01-31T23:10:55.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:55 smithi171 conmon[41853]: debug 2022-01-31T23:10:55.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:55.231528+0000) 2022-01-31T23:10:55.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:55 smithi171 conmon[46715]: debug 2022-01-31T23:10:55.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:55.030620+0000) 2022-01-31T23:10:55.400 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:55 smithi167 conmon[54076]: debug 2022-01-31T23:10:55.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:55.285079+0000) 2022-01-31T23:10:56.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:55 smithi171 conmon[51620]: debug 2022-01-31T23:10:55.661+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:55.662587+0000) 2022-01-31T23:10:56.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:56 smithi167 conmon[60316]: debug 2022-01-31T23:10:56.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:56.042065+0000) 2022-01-31T23:10:56.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:55 smithi167 conmon[49112]: debug 2022-01-31T23:10:55.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:55.864788+0000) 2022-01-31T23:10:56.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:56 smithi171 conmon[41853]: debug 2022-01-31T23:10:56.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:56.231702+0000) 2022-01-31T23:10:56.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:56 smithi171 conmon[46715]: debug 2022-01-31T23:10:56.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:56.030791+0000) 2022-01-31T23:10:56.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:56 smithi167 conmon[54076]: debug 2022-01-31T23:10:56.283+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:56.285248+0000) 2022-01-31T23:10:57.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:56 smithi171 conmon[51620]: debug 2022-01-31T23:10:56.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:56.662771+0000) 2022-01-31T23:10:57.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:56 smithi167 conmon[49112]: debug 2022-01-31T23:10:56.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:56.864982+0000) 2022-01-31T23:10:57.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:57 smithi167 conmon[60316]: debug 2022-01-31T23:10:57.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:57.042229+0000) 2022-01-31T23:10:57.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:57 smithi171 conmon[41853]: debug 2022-01-31T23:10:57.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:57.231862+0000) 2022-01-31T23:10:57.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:57 smithi171 conmon[46715]: debug 2022-01-31T23:10:57.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:57.030988+0000) 2022-01-31T23:10:57.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:57 smithi167 conmon[54076]: debug 2022-01-31T23:10:57.284+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:57.285402+0000) 2022-01-31T23:10:58.012 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:57 smithi171 conmon[51620]: debug 2022-01-31T23:10:57.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:57.663012+0000) 2022-01-31T23:10:58.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:57 smithi167 conmon[49112]: debug 2022-01-31T23:10:57.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:57.865215+0000) 2022-01-31T23:10:58.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:58 smithi167 conmon[60316]: debug 2022-01-31T23:10:58.041+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:58.042390+0000) 2022-01-31T23:10:58.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:58 smithi171 conmon[41853]: debug 2022-01-31T23:10:58.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:58.232044+0000) 2022-01-31T23:10:58.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:58 smithi171 conmon[46715]: debug 2022-01-31T23:10:58.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:58.031176+0000) 2022-01-31T23:10:58.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:58 smithi167 conmon[54076]: debug 2022-01-31T23:10:58.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:58.285559+0000) 2022-01-31T23:10:59.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:58 smithi171 conmon[51620]: debug 2022-01-31T23:10:58.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:58.663205+0000) 2022-01-31T23:10:59.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:58 smithi167 conmon[49112]: debug 2022-01-31T23:10:58.864+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:58.865401+0000) 2022-01-31T23:10:59.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:59 smithi167 conmon[60316]: debug 2022-01-31T23:10:59.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.042573+0000) 2022-01-31T23:10:59.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[41853]: debug 2022-01-31T23:10:59.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.232233+0000) 2022-01-31T23:10:59.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[46715]: debug 2022-01-31T23:10:59.030+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.031377+0000) 2022-01-31T23:10:59.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:59 smithi167 conmon[54076]: debug 2022-01-31T23:10:59.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.285724+0000) 2022-01-31T23:11:00.013 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[35325]: debug 2022-01-31T23:10:59.873+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 269208 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:11:00.014 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[41853]: debug 2022-01-31T23:10:59.846+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.847220+0000) 2022-01-31T23:11:00.014 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[46715]: debug 2022-01-31T23:10:59.845+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.845795+0000) 2022-01-31T23:11:00.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[51620]: debug 2022-01-31T23:10:59.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.663422+0000) 2022-01-31T23:11:00.015 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:10:59 smithi171 conmon[51620]: debug 2022-01-31T23:10:59.845+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.846502+0000) 2022-01-31T23:11:00.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:59 smithi167 conmon[49112]: debug 2022-01-31T23:10:59.844+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.845459+0000) 2022-01-31T23:11:00.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:10:59 smithi167 conmon[49112]: debug 2022-01-31T23:10:59.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.865587+0000) 2022-01-31T23:11:00.152 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:10:59 smithi167 conmon[54076]: debug 2022-01-31T23:10:59.845+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.846298+0000) 2022-01-31T23:11:00.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:10:59 smithi167 conmon[60316]: debug 2022-01-31T23:10:59.846+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:10:59.847129+0000) 2022-01-31T23:11:00.153 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:00 smithi167 conmon[60316]: debug 2022-01-31T23:11:00.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:00.042769+0000) 2022-01-31T23:11:00.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:00 smithi171 conmon[46715]: debug 2022-01-31T23:11:00.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:00.031569+0000) 2022-01-31T23:11:00.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:00 smithi171 conmon[41853]: debug 2022-01-31T23:11:00.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:00.232418+0000) 2022-01-31T23:11:00.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:00 smithi167 conmon[54076]: debug 2022-01-31T23:11:00.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:00.285893+0000) 2022-01-31T23:11:01.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:00 smithi171 conmon[51620]: debug 2022-01-31T23:11:00.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:00.663647+0000) 2022-01-31T23:11:01.150 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:00 smithi167 conmon[49112]: debug 2022-01-31T23:11:00.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:00.865741+0000) 2022-01-31T23:11:01.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:01 smithi167 conmon[60316]: debug 2022-01-31T23:11:01.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:01.042929+0000) 2022-01-31T23:11:01.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:01 smithi171 conmon[46715]: debug 2022-01-31T23:11:01.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:01.031711+0000) 2022-01-31T23:11:01.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:01 smithi171 conmon[41853]: debug 2022-01-31T23:11:01.231+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:01.232593+0000) 2022-01-31T23:11:01.517 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:01 smithi167 conmon[54076]: debug 2022-01-31T23:11:01.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:01.286067+0000) 2022-01-31T23:11:02.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:01 smithi171 conmon[51620]: debug 2022-01-31T23:11:01.662+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:01.663735+0000) 2022-01-31T23:11:02.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:01 smithi167 conmon[49112]: debug 2022-01-31T23:11:01.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:01.865900+0000) 2022-01-31T23:11:02.295 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:02 smithi167 conmon[60316]: debug 2022-01-31T23:11:02.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:02.043139+0000) 2022-01-31T23:11:02.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:02 smithi171 conmon[46715]: debug 2022-01-31T23:11:02.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:02.031864+0000) 2022-01-31T23:11:02.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:02 smithi171 conmon[41853]: debug 2022-01-31T23:11:02.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:02.232791+0000) 2022-01-31T23:11:02.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:02 smithi167 conmon[54076]: debug 2022-01-31T23:11:02.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:02.286224+0000) 2022-01-31T23:11:03.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:02 smithi171 conmon[51620]: debug 2022-01-31T23:11:02.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:02.663920+0000) 2022-01-31T23:11:03.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:02 smithi167 conmon[49112]: debug 2022-01-31T23:11:02.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:02.866077+0000) 2022-01-31T23:11:03.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:03 smithi167 conmon[60316]: debug 2022-01-31T23:11:03.042+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:03.043338+0000) 2022-01-31T23:11:03.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:03 smithi171 conmon[41853]: debug 2022-01-31T23:11:03.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:03.232928+0000) 2022-01-31T23:11:03.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:03 smithi171 conmon[46715]: debug 2022-01-31T23:11:03.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:03.032053+0000) 2022-01-31T23:11:03.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:03 smithi167 conmon[54076]: debug 2022-01-31T23:11:03.285+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:03.286447+0000) 2022-01-31T23:11:04.013 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:03 smithi171 conmon[51620]: debug 2022-01-31T23:11:03.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:03.664108+0000) 2022-01-31T23:11:04.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:03 smithi167 conmon[49112]: debug 2022-01-31T23:11:03.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:03.866264+0000) 2022-01-31T23:11:04.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:04 smithi167 conmon[60316]: debug 2022-01-31T23:11:04.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.043541+0000) 2022-01-31T23:11:04.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[41853]: debug 2022-01-31T23:11:04.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.233083+0000) 2022-01-31T23:11:04.340 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[46715]: debug 2022-01-31T23:11:04.031+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.032207+0000) 2022-01-31T23:11:04.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:04 smithi167 conmon[54076]: debug 2022-01-31T23:11:04.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.286644+0000) 2022-01-31T23:11:04.776 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[51620]: debug 2022-01-31T23:11:04.663+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.664267+0000) 2022-01-31T23:11:05.151 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:04 smithi167 conmon[54076]: debug 2022-01-31T23:11:04.875+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.876178+0000) 2022-01-31T23:11:05.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:04 smithi167 conmon[60316]: debug 2022-01-31T23:11:04.876+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.877300+0000) 2022-01-31T23:11:05.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:05 smithi167 conmon[60316]: debug 2022-01-31T23:11:05.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:05.043734+0000) 2022-01-31T23:11:05.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:04 smithi167 conmon[49112]: debug 2022-01-31T23:11:04.865+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.866428+0000) 2022-01-31T23:11:05.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:04 smithi167 conmon[49112]: debug 2022-01-31T23:11:04.876+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.876634+0000) 2022-01-31T23:11:05.214 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[35325]: debug 2022-01-31T23:11:04.903+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 269321 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:11:05.215 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[51620]: debug 2022-01-31T23:11:04.876+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.877486+0000) 2022-01-31T23:11:05.216 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[41853]: debug 2022-01-31T23:11:04.877+0000 7f940648c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.877742+0000) 2022-01-31T23:11:05.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:04 smithi171 conmon[46715]: debug 2022-01-31T23:11:04.875+0000 7fa6a117f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:04.876303+0000) 2022-01-31T23:11:05.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:05 smithi171 conmon[46715]: debug 2022-01-31T23:11:05.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:05.032397+0000) 2022-01-31T23:11:05.518 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:05 smithi167 conmon[54076]: debug 2022-01-31T23:11:05.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:05.286794+0000) 2022-01-31T23:11:05.589 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:05 smithi171 conmon[41853]: debug 2022-01-31T23:11:05.232+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:05.233246+0000) 2022-01-31T23:11:06.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:05 smithi171 conmon[51620]: debug 2022-01-31T23:11:05.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:05.664464+0000) 2022-01-31T23:11:06.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:06 smithi167 conmon[60316]: debug 2022-01-31T23:11:06.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:06.043826+0000) 2022-01-31T23:11:06.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:05 smithi167 conmon[49112]: debug 2022-01-31T23:11:05.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:05.866694+0000) 2022-01-31T23:11:06.288 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:06 smithi171 conmon[46715]: debug 2022-01-31T23:11:06.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:06.032537+0000) 2022-01-31T23:11:06.288 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:06 smithi171 conmon[41853]: debug 2022-01-31T23:11:06.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:06.233424+0000) 2022-01-31T23:11:06.519 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:06 smithi167 conmon[54076]: debug 2022-01-31T23:11:06.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:06.286925+0000) 2022-01-31T23:11:07.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:06 smithi171 conmon[51620]: debug 2022-01-31T23:11:06.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:06.664632+0000) 2022-01-31T23:11:07.150 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:07 smithi167 conmon[60316]: debug 2022-01-31T23:11:07.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:07.044007+0000) 2022-01-31T23:11:07.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:06 smithi167 conmon[49112]: debug 2022-01-31T23:11:06.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:06.866807+0000) 2022-01-31T23:11:07.339 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:07 smithi171 conmon[41853]: debug 2022-01-31T23:11:07.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:07.233612+0000) 2022-01-31T23:11:07.339 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:07 smithi171 conmon[46715]: debug 2022-01-31T23:11:07.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:07.032727+0000) 2022-01-31T23:11:07.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:07 smithi167 conmon[54076]: debug 2022-01-31T23:11:07.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:07.287068+0000) 2022-01-31T23:11:08.014 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:07 smithi171 conmon[51620]: debug 2022-01-31T23:11:07.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:07.664788+0000) 2022-01-31T23:11:08.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:08 smithi167 conmon[60316]: debug 2022-01-31T23:11:08.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:08.044159+0000) 2022-01-31T23:11:08.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:07 smithi167 conmon[49112]: debug 2022-01-31T23:11:07.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:07.867013+0000) 2022-01-31T23:11:08.335 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:08 smithi171 conmon[41853]: debug 2022-01-31T23:11:08.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:08.233827+0000) 2022-01-31T23:11:08.335 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:08 smithi171 conmon[46715]: debug 2022-01-31T23:11:08.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:08.032907+0000) 2022-01-31T23:11:08.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:08 smithi167 conmon[54076]: debug 2022-01-31T23:11:08.286+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:08.287237+0000) 2022-01-31T23:11:08.958 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:08 smithi171 conmon[51620]: debug 2022-01-31T23:11:08.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:08.664969+0000) 2022-01-31T23:11:09.151 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:08 smithi167 conmon[49112]: debug 2022-01-31T23:11:08.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:08.867203+0000) 2022-01-31T23:11:09.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:09 smithi167 conmon[60316]: debug 2022-01-31T23:11:09.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.044333+0000) 2022-01-31T23:11:09.215 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:11:08 smithi171 conmon[35325]: debug 2022-01-31T23:11:08.976+0000 7fb4801d5700 -1 mon.smithi171@1(probing) e2 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-01-31T23:11:09.216 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[46715]: debug 2022-01-31T23:11:09.032+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.033038+0000) 2022-01-31T23:11:09.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:09 smithi167 conmon[54076]: debug 2022-01-31T23:11:09.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.287392+0000) 2022-01-31T23:11:09.588 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[41853]: debug 2022-01-31T23:11:09.233+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.234032+0000) 2022-01-31T23:11:09.917 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[46715]: debug 2022-01-31T23:11:09.906+0000 7fa6a097e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.906879+0000) 2022-01-31T23:11:09.917 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[41853]: debug 2022-01-31T23:11:09.906+0000 7f9406c8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.907308+0000) 2022-01-31T23:11:09.918 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[51620]: debug 2022-01-31T23:11:09.664+0000 7f84dd34e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.665097+0000) 2022-01-31T23:11:09.918 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[51620]: debug 2022-01-31T23:11:09.906+0000 7f84f362d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.907074+0000) 2022-01-31T23:11:10.151 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:09 smithi167 conmon[54076]: debug 2022-01-31T23:11:09.907+0000 7f546561b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.907708+0000) 2022-01-31T23:11:10.151 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:09 smithi167 conmon[60316]: debug 2022-01-31T23:11:09.906+0000 7f4304d89700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.907183+0000) 2022-01-31T23:11:10.152 INFO:journalctl@ceph.osd.2.smithi167.stdout:Jan 31 23:11:10 smithi167 conmon[60316]: debug 2022-01-31T23:11:10.043+0000 7f42f2ab2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:10.044461+0000) 2022-01-31T23:11:10.152 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:09 smithi167 conmon[49112]: debug 2022-01-31T23:11:09.866+0000 7fc43e412700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.867414+0000) 2022-01-31T23:11:10.153 INFO:journalctl@ceph.osd.0.smithi167.stdout:Jan 31 23:11:09 smithi167 conmon[49112]: debug 2022-01-31T23:11:09.907+0000 7fc4536ef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:09.907940+0000) 2022-01-31T23:11:10.215 INFO:journalctl@ceph.osd.4.smithi171.stdout:Jan 31 23:11:10 smithi171 conmon[46715]: debug 2022-01-31T23:11:10.033+0000 7fa68d6a5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:10.033196+0000) 2022-01-31T23:11:10.216 INFO:journalctl@ceph.mon.smithi171.smithi171.stdout:Jan 31 23:11:09 smithi171 conmon[35325]: debug 2022-01-31T23:11:09.934+0000 7fb4839dc700 -1 mon.smithi171@1(probing) e2 get_health_metrics reporting 269431 slow ops, oldest is log(1 entries from seq 76 at 2022-01-31T19:45:17.671333+0000) 2022-01-31T23:11:10.520 INFO:journalctl@ceph.osd.1.smithi167.stdout:Jan 31 23:11:10 smithi167 conmon[54076]: debug 2022-01-31T23:11:10.287+0000 7f5453344700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:10.287578+0000) 2022-01-31T23:11:10.588 INFO:journalctl@ceph.osd.3.smithi171.stdout:Jan 31 23:11:10 smithi171 conmon[41853]: debug 2022-01-31T23:11:10.234+0000 7f93f41b5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-01-31T22:11:10.234239+0000) 2022-01-31T23:11:10.919 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-01-31T23:11:10.921 DEBUG:teuthology.task.console_log:Killing console logger for smithi167 2022-01-31T23:11:10.922 DEBUG:teuthology.task.console_log:Killing console logger for smithi171 2022-01-31T23:11:10.923 DEBUG:teuthology.exit:Finished running handlers